2024-03-23T22:37:01.969 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-23T22:37:01.970 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} --verbose --name mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi --owner mchangir --archive /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616369 -- /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616369/orig.config.yaml 2024-03-23T22:37:02.021 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-23T22:37:02.120 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616369 branch: wip-mchangir-testing-main-20240318.032620 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} email: mchangir@redhat.com first_in_suite: false job_id: '7616369' kernel: client: branch: testing flavor: default kdb: true sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-mchangir-testing-main-20240318.032620 ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - slow metadata IO - SLOW_OPS - slow request sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported dry_run: true install: ceph: flavor: default sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mchangir-testing-main-20240318.032620 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 owner: mchangir priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3550 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 sleep_before_teardown: 0 subset: 613/1024 suite: fs suite_branch: wip-mchangir-testing-main-20240318.032620 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi012.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIODUQ7FvGYJLWYhbyXWV04x7zFmqpVbLTY6pZadtVVwqbWPx/tw67RhiroOV3Sm3+RL6c1FEedUmAdLsFfty9U= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJGIytuhzMFSdGPBzLZNW38fnbqI7BKyIyhy4VYxsvoJy33yzGIYx+8Vmg7jQjy/DMHOx1w8+B3GPxZbNcmvK7o= smithi203.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL0DH2kuv4+0EuhMnWHVEO7RHu07yj9kLGKpbTXI1CJEPAewj0XPNn3gQUEfKbMwAsoyM1GMdcrnMd1plkCL0Rc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - fs/test_o_trunc.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-22_09:49:57 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-23T22:37:02.120 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa; will attempt to use it 2024-03-23T22:37:02.121 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks 2024-03-23T22:37:02.121 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-23T22:37:02.123 INFO:teuthology.task.internal:Checking packages... 2024-03-23T22:37:02.146 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3be9a5c9bc793e11e2800a8c0c696e8b46742033' 2024-03-23T22:37:02.146 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-23T22:37:02.146 INFO:teuthology.packaging:ref: None 2024-03-23T22:37:02.146 INFO:teuthology.packaging:tag: None 2024-03-23T22:37:02.146 INFO:teuthology.packaging:branch: wip-mchangir-testing-main-20240318.032620 2024-03-23T22:37:02.146 INFO:teuthology.packaging:sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-23T22:37:02.146 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-mchangir-testing-main-20240318.032620 2024-03-23T22:37:02.294 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2145.g3be9a5c9 2024-03-23T22:37:02.295 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-23T22:37:02.302 INFO:teuthology.task.internal:no buildpackages task found 2024-03-23T22:37:02.302 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-23T22:37:02.308 INFO:teuthology.task.internal:Saving configuration 2024-03-23T22:37:02.326 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-23T22:37:02.331 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-23T22:37:02.357 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616369', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 22:31:53.666787', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIODUQ7FvGYJLWYhbyXWV04x7zFmqpVbLTY6pZadtVVwqbWPx/tw67RhiroOV3Sm3+RL6c1FEedUmAdLsFfty9U='} 2024-03-23T22:37:02.378 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616369', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 22:31:53.665044', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJGIytuhzMFSdGPBzLZNW38fnbqI7BKyIyhy4VYxsvoJy33yzGIYx+8Vmg7jQjy/DMHOx1w8+B3GPxZbNcmvK7o='} 2024-03-23T22:37:02.403 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616369', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 22:31:53.663094', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL0DH2kuv4+0EuhMnWHVEO7RHu07yj9kLGKpbTXI1CJEPAewj0XPNn3gQUEfKbMwAsoyM1GMdcrnMd1plkCL0Rc='} 2024-03-23T22:37:02.403 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-23T22:37:02.409 INFO:teuthology.task.internal:roles: ubuntu@smithi012.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-23T22:37:02.409 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-23T22:37:02.409 INFO:teuthology.task.internal:roles: ubuntu@smithi203.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-23T22:37:02.409 INFO:teuthology.run_tasks:Running task console_log... 2024-03-23T22:37:02.511 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f608fdd3f70>, signals=[15]) 2024-03-23T22:37:02.512 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-23T22:37:02.518 INFO:teuthology.task.internal:Opening connections... 2024-03-23T22:37:02.518 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi012.front.sepia.ceph.com 2024-03-23T22:37:02.520 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T22:37:02.625 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2024-03-23T22:37:02.626 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T22:37:02.713 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi203.front.sepia.ceph.com 2024-03-23T22:37:02.714 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T22:37:02.801 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-23T22:37:02.807 DEBUG:teuthology.orchestra.run.smithi012:> uname -m 2024-03-23T22:37:02.825 INFO:teuthology.orchestra.run.smithi012.stdout:x86_64 2024-03-23T22:37:02.825 DEBUG:teuthology.orchestra.run.smithi012:> cat /etc/os-release 2024-03-23T22:37:02.882 INFO:teuthology.orchestra.run.smithi012.stdout:NAME="CentOS Stream" 2024-03-23T22:37:02.883 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION="9" 2024-03-23T22:37:02.883 INFO:teuthology.orchestra.run.smithi012.stdout:ID="centos" 2024-03-23T22:37:02.883 INFO:teuthology.orchestra.run.smithi012.stdout:ID_LIKE="rhel fedora" 2024-03-23T22:37:02.883 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_ID="9" 2024-03-23T22:37:02.883 INFO:teuthology.orchestra.run.smithi012.stdout:PLATFORM_ID="platform:el9" 2024-03-23T22:37:02.883 INFO:teuthology.orchestra.run.smithi012.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-23T22:37:02.883 INFO:teuthology.orchestra.run.smithi012.stdout:ANSI_COLOR="0;31" 2024-03-23T22:37:02.883 INFO:teuthology.orchestra.run.smithi012.stdout:LOGO="fedora-logo-icon" 2024-03-23T22:37:02.883 INFO:teuthology.orchestra.run.smithi012.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-23T22:37:02.883 INFO:teuthology.orchestra.run.smithi012.stdout:HOME_URL="https://centos.org/" 2024-03-23T22:37:02.883 INFO:teuthology.orchestra.run.smithi012.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-23T22:37:02.883 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-23T22:37:02.883 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-23T22:37:02.885 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server 2024-03-23T22:37:02.913 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2024-03-23T22:37:02.931 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2024-03-23T22:37:02.931 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2024-03-23T22:37:02.988 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Stream" 2024-03-23T22:37:02.988 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="9" 2024-03-23T22:37:02.989 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos" 2024-03-23T22:37:02.989 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora" 2024-03-23T22:37:02.989 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="9" 2024-03-23T22:37:02.989 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el9" 2024-03-23T22:37:02.989 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-23T22:37:02.989 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31" 2024-03-23T22:37:02.989 INFO:teuthology.orchestra.run.smithi173.stdout:LOGO="fedora-logo-icon" 2024-03-23T22:37:02.989 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-23T22:37:02.989 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/" 2024-03-23T22:37:02.989 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-23T22:37:02.989 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-23T22:37:02.989 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-23T22:37:02.990 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2024-03-23T22:37:03.021 DEBUG:teuthology.orchestra.run.smithi203:> uname -m 2024-03-23T22:37:03.039 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64 2024-03-23T22:37:03.039 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release 2024-03-23T22:37:03.096 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="CentOS Stream" 2024-03-23T22:37:03.096 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="9" 2024-03-23T22:37:03.096 INFO:teuthology.orchestra.run.smithi203.stdout:ID="centos" 2024-03-23T22:37:03.096 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE="rhel fedora" 2024-03-23T22:37:03.096 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="9" 2024-03-23T22:37:03.096 INFO:teuthology.orchestra.run.smithi203.stdout:PLATFORM_ID="platform:el9" 2024-03-23T22:37:03.096 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-23T22:37:03.096 INFO:teuthology.orchestra.run.smithi203.stdout:ANSI_COLOR="0;31" 2024-03-23T22:37:03.096 INFO:teuthology.orchestra.run.smithi203.stdout:LOGO="fedora-logo-icon" 2024-03-23T22:37:03.097 INFO:teuthology.orchestra.run.smithi203.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-23T22:37:03.097 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://centos.org/" 2024-03-23T22:37:03.097 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-23T22:37:03.097 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-23T22:37:03.097 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-23T22:37:03.097 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server 2024-03-23T22:37:03.121 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-23T22:37:03.132 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-23T22:37:03.143 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-23T22:37:03.143 DEBUG:teuthology.orchestra.run.smithi012:> test '!' -e /home/ubuntu/cephtest 2024-03-23T22:37:03.146 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2024-03-23T22:37:03.148 DEBUG:teuthology.orchestra.run.smithi203:> test '!' -e /home/ubuntu/cephtest 2024-03-23T22:37:03.164 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-23T22:37:03.170 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-23T22:37:03.170 DEBUG:teuthology.orchestra.run.smithi012:> test -z $(ls -A /var/lib/ceph) 2024-03-23T22:37:03.203 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2024-03-23T22:37:03.205 DEBUG:teuthology.orchestra.run.smithi203:> test -z $(ls -A /var/lib/ceph) 2024-03-23T22:37:03.306 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-23T22:37:03.383 INFO:teuthology.run_tasks:Running task kernel... 2024-03-23T22:37:03.399 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'} 2024-03-23T22:37:03.399 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-23T22:37:03.399 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-23T22:37:03.399 DEBUG:teuthology.orchestra.run.smithi012:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-23T22:37:03.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T22:37:03.417 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-23T22:37:03.716 DEBUG:teuthology.task.kernel:sha1 for client.0 is 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T22:37:03.716 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5ec3e6e88b2e2783e09498588bb1f433ae9c8986"... 2024-03-23T22:37:03.716 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2024-03-23T22:37:03.734 INFO:teuthology.orchestra.run.smithi012.stdout:5.14.0-361.el9.x86_64 2024-03-23T22:37:03.734 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T22:37:03.734 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-23T22:37:03.735 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum upgrade -y linux-firmware 2024-03-23T22:37:05.870 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-23T22:37:07.337 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2024-03-23T22:37:08.427 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-23T22:37:10.611 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-03-23T22:37:14.562 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-23T22:37:16.021 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - Extras packages 53 kB/s | 16 kB 00:00 2024-03-23T22:37:18.297 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 9.6 MB/s | 21 MB 00:02 2024-03-23T22:37:23.446 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 48 kB/s | 1.7 kB 00:00 2024-03-23T22:37:25.734 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-23T22:37:25.743 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-23T22:37:25.743 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-03-23T22:37:25.743 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-23T22:37:25.744 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrading: 2024-03-23T22:37:25.744 INFO:teuthology.orchestra.run.smithi012.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-23T22:37:25.744 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:37:25.744 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-03-23T22:37:25.744 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-23T22:37:25.744 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrade 1 Package 2024-03-23T22:37:25.744 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:37:25.744 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 375 M 2024-03-23T22:37:25.744 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-03-23T22:37:32.813 INFO:teuthology.orchestra.run.smithi012.stdout:linux-firmware-20240219-143.el9.noarch.rpm 53 MB/s | 375 MB 00:07 2024-03-23T22:37:32.814 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-23T22:37:32.815 INFO:teuthology.orchestra.run.smithi012.stdout:Total 53 MB/s | 375 MB 00:07 2024-03-23T22:37:32.815 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-03-23T22:37:32.906 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-03-23T22:37:32.906 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-03-23T22:37:35.427 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-03-23T22:37:35.427 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-03-23T22:37:41.892 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-03-23T22:37:42.143 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-23T22:37:42.240 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-23T22:37:43.615 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-23T22:37:43.616 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-23T22:37:44.030 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-23T22:37:44.030 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:37:44.030 INFO:teuthology.orchestra.run.smithi012.stdout:Upgraded: 2024-03-23T22:37:44.030 INFO:teuthology.orchestra.run.smithi012.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-23T22:37:44.030 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:37:44.031 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-23T22:37:44.324 INFO:teuthology.task.kernel:Downloading kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on client.0... 2024-03-23T22:37:44.325 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T22:37:44.475 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/ 2024-03-23T22:37:44.475 DEBUG:teuthology.orchestra.run.smithi012:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-23T22:37:44.478 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi012.front.sepia.ceph.com to complete... 2024-03-23T22:37:58.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23 22:37:58 URL:https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm [336309687/336309687] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-23T22:37:58.006 INFO:teuthology.orchestra.run.smithi012.stderr:FINISHED --2024-03-23 22:37:58-- 2024-03-23T22:37:58.006 INFO:teuthology.orchestra.run.smithi012.stderr:Total wall clock time: 14s 2024-03-23T22:37:58.006 INFO:teuthology.orchestra.run.smithi012.stderr:Downloaded: 1 files, 321M in 13s (24.4 MB/s) 2024-03-23T22:37:58.008 INFO:teuthology.task.kernel:Installing kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on client.0... 2024-03-23T22:37:58.008 DEBUG:teuthology.orchestra.run.smithi012:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-23T22:38:00.035 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying... ######################################## 2024-03-23T22:38:00.093 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing... ######################################## 2024-03-23T22:38:00.177 INFO:teuthology.orchestra.run.smithi012.stdout:Updating / installing... 2024-03-23T22:38:08.023 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-6.8.0_g5ec3e6e88b2e-1 ######################################## 2024-03-23T22:38:24.217 INFO:teuthology.orchestra.run.smithi012.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-23T22:38:47.558 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi012.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-23T22:38:47.559 DEBUG:teuthology.orchestra.run.smithi012:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-23T22:38:47.642 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e 2024-03-23T22:38:47.643 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/System.map 2024-03-23T22:38:47.643 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/config 2024-03-23T22:38:47.643 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel 2024-03-23T22:38:47.643 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch 2024-03-23T22:38:47.643 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86 2024-03-23T22:38:47.643 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto 2024-03-23T22:38:47.643 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-23T22:38:47.643 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-23T22:38:47.643 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-23T22:38:47.643 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-23T22:38:47.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-23T22:38:47.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-23T22:38:47.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-23T22:38:47.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-23T22:38:47.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-23T22:38:47.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-23T22:38:47.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-23T22:38:47.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-23T22:38:47.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-23T22:38:47.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-23T22:38:47.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-23T22:38:47.644 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-23T22:38:47.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-23T22:38:47.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-23T22:38:47.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-23T22:38:47.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-23T22:38:47.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel 2024-03-23T22:38:47.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu 2024-03-23T22:38:47.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce 2024-03-23T22:38:47.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-23T22:38:47.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm 2024-03-23T22:38:47.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-23T22:38:47.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-23T22:38:47.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm.ko 2024-03-23T22:38:47.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto 2024-03-23T22:38:47.645 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ansi_cprng.ko 2024-03-23T22:38:47.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/anubis.ko 2024-03-23T22:38:47.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/arc4.ko 2024-03-23T22:38:47.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx 2024-03-23T22:38:47.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_memcpy.ko 2024-03-23T22:38:47.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_pq.ko 2024-03-23T22:38:47.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-23T22:38:47.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_tx.ko 2024-03-23T22:38:47.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_xor.ko 2024-03-23T22:38:47.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/raid6test.ko 2024-03-23T22:38:47.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authenc.ko 2024-03-23T22:38:47.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authencesn.ko 2024-03-23T22:38:47.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blake2b_generic.ko 2024-03-23T22:38:47.646 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_common.ko 2024-03-23T22:38:47.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_generic.ko 2024-03-23T22:38:47.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/camellia_generic.ko 2024-03-23T22:38:47.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast5_generic.ko 2024-03-23T22:38:47.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast6_generic.ko 2024-03-23T22:38:47.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast_common.ko 2024-03-23T22:38:47.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ccm.ko 2024-03-23T22:38:47.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cmac.ko 2024-03-23T22:38:47.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crc32_generic.ko 2024-03-23T22:38:47.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_engine.ko 2024-03-23T22:38:47.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_user.ko 2024-03-23T22:38:47.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cts.ko 2024-03-23T22:38:47.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/des_generic.ko 2024-03-23T22:38:47.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/dh_generic.ko 2024-03-23T22:38:47.647 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/echainiv.ko 2024-03-23T22:38:47.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/essiv.ko 2024-03-23T22:38:47.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/fcrypt.ko 2024-03-23T22:38:47.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/khazad.ko 2024-03-23T22:38:47.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4.ko 2024-03-23T22:38:47.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4hc.ko 2024-03-23T22:38:47.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/md4.ko 2024-03-23T22:38:47.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/michael_mic.ko 2024-03-23T22:38:47.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcbc.ko 2024-03-23T22:38:47.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcrypt.ko 2024-03-23T22:38:47.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/rmd160.ko 2024-03-23T22:38:47.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/seed.ko 2024-03-23T22:38:47.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/serpent_generic.ko 2024-03-23T22:38:47.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tcrypt.ko 2024-03-23T22:38:47.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tea.ko 2024-03-23T22:38:47.648 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_common.ko 2024-03-23T22:38:47.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_generic.ko 2024-03-23T22:38:47.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/vmac.ko 2024-03-23T22:38:47.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/wp512.ko 2024-03-23T22:38:47.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xcbc.ko 2024-03-23T22:38:47.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xor.ko 2024-03-23T22:38:47.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xxhash_generic.ko 2024-03-23T22:38:47.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers 2024-03-23T22:38:47.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi 2024-03-23T22:38:47.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-23T22:38:47.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_pad.ko 2024-03-23T22:38:47.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/custom_method.ko 2024-03-23T22:38:47.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/ec_sys.ko 2024-03-23T22:38:47.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/platform_profile.ko 2024-03-23T22:38:47.649 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbs.ko 2024-03-23T22:38:47.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbshc.ko 2024-03-23T22:38:47.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/video.ko 2024-03-23T22:38:47.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata 2024-03-23T22:38:47.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/acard-ahci.ko 2024-03-23T22:38:47.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ahci_platform.ko 2024-03-23T22:38:47.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ata_generic.ko 2024-03-23T22:38:47.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/libahci_platform.ko 2024-03-23T22:38:47.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_acpi.ko 2024-03-23T22:38:47.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ali.ko 2024-03-23T22:38:47.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_amd.ko 2024-03-23T22:38:47.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_artop.ko 2024-03-23T22:38:47.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atiixp.ko 2024-03-23T22:38:47.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atp867x.ko 2024-03-23T22:38:47.650 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd640.ko 2024-03-23T22:38:47.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd64x.ko 2024-03-23T22:38:47.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cypress.ko 2024-03-23T22:38:47.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_efar.ko 2024-03-23T22:38:47.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt366.ko 2024-03-23T22:38:47.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt37x.ko 2024-03-23T22:38:47.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-23T22:38:47.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-23T22:38:47.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it8213.ko 2024-03-23T22:38:47.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it821x.ko 2024-03-23T22:38:47.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_jmicron.ko 2024-03-23T22:38:47.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_marvell.ko 2024-03-23T22:38:47.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_mpiix.ko 2024-03-23T22:38:47.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_netcell.ko 2024-03-23T22:38:47.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ninja32.ko 2024-03-23T22:38:47.651 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87410.ko 2024-03-23T22:38:47.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87415.ko 2024-03-23T22:38:47.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_oldpiix.ko 2024-03-23T22:38:47.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_opti.ko 2024-03-23T22:38:47.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_optidma.ko 2024-03-23T22:38:47.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pcmcia.ko 2024-03-23T22:38:47.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-23T22:38:47.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-23T22:38:47.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_piccolo.ko 2024-03-23T22:38:47.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_rdc.ko 2024-03-23T22:38:47.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sch.ko 2024-03-23T22:38:47.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_serverworks.ko 2024-03-23T22:38:47.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sil680.ko 2024-03-23T22:38:47.652 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sis.ko 2024-03-23T22:38:47.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sl82c105.ko 2024-03-23T22:38:47.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_triflex.ko 2024-03-23T22:38:47.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_via.ko 2024-03-23T22:38:47.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pdc_adma.ko 2024-03-23T22:38:47.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_inic162x.ko 2024-03-23T22:38:47.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_mv.ko 2024-03-23T22:38:47.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_nv.ko 2024-03-23T22:38:47.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_promise.ko 2024-03-23T22:38:47.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_qstor.ko 2024-03-23T22:38:47.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil.ko 2024-03-23T22:38:47.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil24.ko 2024-03-23T22:38:47.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sis.ko 2024-03-23T22:38:47.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_svw.ko 2024-03-23T22:38:47.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sx4.ko 2024-03-23T22:38:47.653 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_uli.ko 2024-03-23T22:38:47.654 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_via.ko 2024-03-23T22:38:47.654 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_vsc.ko 2024-03-23T22:38:47.654 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay 2024-03-23T22:38:47.654 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay/ks0108.ko 2024-03-23T22:38:47.654 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base 2024-03-23T22:38:47.654 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap 2024-03-23T22:38:47.654 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-23T22:38:47.654 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-23T22:38:47.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma 2024-03-23T22:38:47.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma/bcma.ko 2024-03-23T22:38:47.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block 2024-03-23T22:38:47.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe 2024-03-23T22:38:47.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe/aoe.ko 2024-03-23T22:38:47.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/brd.ko 2024-03-23T22:38:47.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd 2024-03-23T22:38:47.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd/drbd.ko 2024-03-23T22:38:47.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/floppy.ko 2024-03-23T22:38:47.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/loop.ko 2024-03-23T22:38:47.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx 2024-03-23T22:38:47.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-23T22:38:47.655 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/nbd.ko 2024-03-23T22:38:47.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk 2024-03-23T22:38:47.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk/null_blk.ko 2024-03-23T22:38:47.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/pktcdvd.ko 2024-03-23T22:38:47.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/rbd.ko 2024-03-23T22:38:47.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/virtio_blk.ko 2024-03-23T22:38:47.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback 2024-03-23T22:38:47.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-23T22:38:47.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkfront.ko 2024-03-23T22:38:47.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram 2024-03-23T22:38:47.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram/zram.ko 2024-03-23T22:38:47.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char 2024-03-23T22:38:47.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hangcheck-timer.ko 2024-03-23T22:38:47.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random 2024-03-23T22:38:47.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-23T22:38:47.656 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-23T22:38:47.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-23T22:38:47.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/via-rng.ko 2024-03-23T22:38:47.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-23T22:38:47.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi 2024-03-23T22:38:47.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-23T22:38:47.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-23T22:38:47.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-23T22:38:47.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-23T22:38:47.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-23T22:38:47.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-23T22:38:47.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/lp.ko 2024-03-23T22:38:47.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave 2024-03-23T22:38:47.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave/mwave.ko 2024-03-23T22:38:47.657 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ppdev.ko 2024-03-23T22:38:47.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tlclk.ko 2024-03-23T22:38:47.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm 2024-03-23T22:38:47.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm.ko 2024-03-23T22:38:47.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-23T22:38:47.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-23T22:38:47.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-23T22:38:47.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-23T22:38:47.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-23T22:38:47.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/uv_mmtimer.ko 2024-03-23T22:38:47.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/virtio_console.ko 2024-03-23T22:38:47.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq 2024-03-23T22:38:47.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-23T22:38:47.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-23T22:38:47.658 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-23T22:38:47.659 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-23T22:38:47.659 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-23T22:38:47.659 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-23T22:38:47.659 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto 2024-03-23T22:38:47.659 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp 2024-03-23T22:38:47.659 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-23T22:38:47.659 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp.ko 2024-03-23T22:38:47.659 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel 2024-03-23T22:38:47.659 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat 2024-03-23T22:38:47.659 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common 2024-03-23T22:38:47.659 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-23T22:38:47.659 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-23T22:38:47.659 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-23T22:38:47.659 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-aes.ko 2024-03-23T22:38:47.659 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-sha.ko 2024-03-23T22:38:47.660 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio 2024-03-23T22:38:47.660 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-23T22:38:47.660 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca 2024-03-23T22:38:47.660 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca/dca.ko 2024-03-23T22:38:47.660 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq 2024-03-23T22:38:47.660 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-23T22:38:47.660 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma 2024-03-23T22:38:47.660 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw 2024-03-23T22:38:47.660 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-23T22:38:47.660 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat 2024-03-23T22:38:47.660 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-23T22:38:47.660 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac 2024-03-23T22:38:47.660 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/amd64_edac.ko 2024-03-23T22:38:47.660 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/e752x_edac.ko 2024-03-23T22:38:47.660 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/edac_mce_amd.ko 2024-03-23T22:38:47.660 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3000_edac.ko 2024-03-23T22:38:47.661 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3200_edac.ko 2024-03-23T22:38:47.661 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5100_edac.ko 2024-03-23T22:38:47.661 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5400_edac.ko 2024-03-23T22:38:47.661 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7300_edac.ko 2024-03-23T22:38:47.661 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7core_edac.ko 2024-03-23T22:38:47.661 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i82975x_edac.ko 2024-03-23T22:38:47.661 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/ie31200_edac.ko 2024-03-23T22:38:47.661 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/sb_edac.ko 2024-03-23T22:38:47.661 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/x38_edac.ko 2024-03-23T22:38:47.661 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware 2024-03-23T22:38:47.661 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/edd.ko 2024-03-23T22:38:47.661 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-23T22:38:47.661 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio 2024-03-23T22:38:47.661 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-ich.ko 2024-03-23T22:38:47.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-23T22:38:47.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid 2024-03-23T22:38:47.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-appleir.ko 2024-03-23T22:38:47.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-aureal.ko 2024-03-23T22:38:47.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-axff.ko 2024-03-23T22:38:47.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-dr.ko 2024-03-23T22:38:47.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elecom.ko 2024-03-23T22:38:47.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elo.ko 2024-03-23T22:38:47.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-emsff.ko 2024-03-23T22:38:47.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gaff.ko 2024-03-23T22:38:47.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gt683r.ko 2024-03-23T22:38:47.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gyration.ko 2024-03-23T22:38:47.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-23T22:38:47.662 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-23T22:38:47.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtekff.ko 2024-03-23T22:38:47.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-hyperv.ko 2024-03-23T22:38:47.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-icade.ko 2024-03-23T22:38:47.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-keytouch.ko 2024-03-23T22:38:47.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-kye.ko 2024-03-23T22:38:47.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lcpower.ko 2024-03-23T22:38:47.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-led.ko 2024-03-23T22:38:47.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lenovo.ko 2024-03-23T22:38:47.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-23T22:38:47.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-23T22:38:47.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-multitouch.ko 2024-03-23T22:38:47.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ortek.ko 2024-03-23T22:38:47.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-penmount.ko 2024-03-23T22:38:47.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-petalynx.ko 2024-03-23T22:38:47.663 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-picolcd.ko 2024-03-23T22:38:47.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-pl.ko 2024-03-23T22:38:47.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-plantronics.ko 2024-03-23T22:38:47.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-primax.ko 2024-03-23T22:38:47.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-rmi.ko 2024-03-23T22:38:47.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-23T22:38:47.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-common.ko 2024-03-23T22:38:47.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-23T22:38:47.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-23T22:38:47.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-23T22:38:47.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-23T22:38:47.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-23T22:38:47.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-23T22:38:47.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-23T22:38:47.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-23T22:38:47.664 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-23T22:38:47.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat.ko 2024-03-23T22:38:47.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-saitek.ko 2024-03-23T22:38:47.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-samsung.ko 2024-03-23T22:38:47.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-23T22:38:47.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sjoy.ko 2024-03-23T22:38:47.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sony.ko 2024-03-23T22:38:47.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-speedlink.ko 2024-03-23T22:38:47.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-steelseries.ko 2024-03-23T22:38:47.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sunplus.ko 2024-03-23T22:38:47.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-23T22:38:47.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tivo.ko 2024-03-23T22:38:47.665 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tmff.ko 2024-03-23T22:38:47.666 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-topseed.ko 2024-03-23T22:38:47.666 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-twinhan.ko 2024-03-23T22:38:47.666 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-uclogic.ko 2024-03-23T22:38:47.666 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-waltop.ko 2024-03-23T22:38:47.666 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-wiimote.ko 2024-03-23T22:38:47.666 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-xinmo.ko 2024-03-23T22:38:47.666 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zpff.ko 2024-03-23T22:38:47.666 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zydacron.ko 2024-03-23T22:38:47.666 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/uhid.ko 2024-03-23T22:38:47.666 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/wacom.ko 2024-03-23T22:38:47.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv 2024-03-23T22:38:47.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_balloon.ko 2024-03-23T22:38:47.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_utils.ko 2024-03-23T22:38:47.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_vmbus.ko 2024-03-23T22:38:47.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon 2024-03-23T22:38:47.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru.ko 2024-03-23T22:38:47.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru3.ko 2024-03-23T22:38:47.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-23T22:38:47.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7414.ko 2024-03-23T22:38:47.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7418.ko 2024-03-23T22:38:47.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adc128d818.ko 2024-03-23T22:38:47.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1025.ko 2024-03-23T22:38:47.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1026.ko 2024-03-23T22:38:47.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1029.ko 2024-03-23T22:38:47.667 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1031.ko 2024-03-23T22:38:47.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm9240.ko 2024-03-23T22:38:47.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ads7828.ko 2024-03-23T22:38:47.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7410.ko 2024-03-23T22:38:47.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7411.ko 2024-03-23T22:38:47.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7462.ko 2024-03-23T22:38:47.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7470.ko 2024-03-23T22:38:47.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7475.ko 2024-03-23T22:38:47.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7x10.ko 2024-03-23T22:38:47.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/amc6821.ko 2024-03-23T22:38:47.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/applesmc.ko 2024-03-23T22:38:47.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asb100.ko 2024-03-23T22:38:47.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asc7621.ko 2024-03-23T22:38:47.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-23T22:38:47.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/atxp1.ko 2024-03-23T22:38:47.668 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/coretemp.ko 2024-03-23T22:38:47.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-23T22:38:47.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dme1737.ko 2024-03-23T22:38:47.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds1621.ko 2024-03-23T22:38:47.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds620.ko 2024-03-23T22:38:47.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc1403.ko 2024-03-23T22:38:47.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc6w201.ko 2024-03-23T22:38:47.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71805f.ko 2024-03-23T22:38:47.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71882fg.ko 2024-03-23T22:38:47.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f75375s.ko 2024-03-23T22:38:47.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fam15h_power.ko 2024-03-23T22:38:47.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fschmd.ko 2024-03-23T22:38:47.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g760a.ko 2024-03-23T22:38:47.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g762.ko 2024-03-23T22:38:47.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl518sm.ko 2024-03-23T22:38:47.669 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl520sm.ko 2024-03-23T22:38:47.670 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-23T22:38:47.670 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5500_temp.ko 2024-03-23T22:38:47.670 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5k_amb.ko 2024-03-23T22:38:47.670 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmaem.ko 2024-03-23T22:38:47.670 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmpex.ko 2024-03-23T22:38:47.670 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina209.ko 2024-03-23T22:38:47.670 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina2xx.ko 2024-03-23T22:38:47.670 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/it87.ko 2024-03-23T22:38:47.670 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k10temp.ko 2024-03-23T22:38:47.670 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k8temp.ko 2024-03-23T22:38:47.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lineage-pem.ko 2024-03-23T22:38:47.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm63.ko 2024-03-23T22:38:47.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm73.ko 2024-03-23T22:38:47.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm75.ko 2024-03-23T22:38:47.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm77.ko 2024-03-23T22:38:47.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm78.ko 2024-03-23T22:38:47.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm80.ko 2024-03-23T22:38:47.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm83.ko 2024-03-23T22:38:47.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm85.ko 2024-03-23T22:38:47.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm87.ko 2024-03-23T22:38:47.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm90.ko 2024-03-23T22:38:47.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm92.ko 2024-03-23T22:38:47.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm93.ko 2024-03-23T22:38:47.671 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95234.ko 2024-03-23T22:38:47.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95241.ko 2024-03-23T22:38:47.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95245.ko 2024-03-23T22:38:47.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc2945.ko 2024-03-23T22:38:47.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4151.ko 2024-03-23T22:38:47.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4215.ko 2024-03-23T22:38:47.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4222.ko 2024-03-23T22:38:47.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4245.ko 2024-03-23T22:38:47.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4260.ko 2024-03-23T22:38:47.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4261.ko 2024-03-23T22:38:47.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max16065.ko 2024-03-23T22:38:47.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1619.ko 2024-03-23T22:38:47.672 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1668.ko 2024-03-23T22:38:47.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max197.ko 2024-03-23T22:38:47.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6639.ko 2024-03-23T22:38:47.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6650.ko 2024-03-23T22:38:47.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6697.ko 2024-03-23T22:38:47.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/mcp3021.ko 2024-03-23T22:38:47.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6683.ko 2024-03-23T22:38:47.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775-core.ko 2024-03-23T22:38:47.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775.ko 2024-03-23T22:38:47.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct7802.ko 2024-03-23T22:38:47.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87360.ko 2024-03-23T22:38:47.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87427.ko 2024-03-23T22:38:47.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pcf8591.ko 2024-03-23T22:38:47.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus 2024-03-23T22:38:47.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-23T22:38:47.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-23T22:38:47.673 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-23T22:38:47.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-23T22:38:47.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-23T22:38:47.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-23T22:38:47.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-23T22:38:47.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-23T22:38:47.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-23T22:38:47.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-23T22:38:47.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-23T22:38:47.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-23T22:38:47.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/powr1220.ko 2024-03-23T22:38:47.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5627.ko 2024-03-23T22:38:47.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5636.ko 2024-03-23T22:38:47.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-23T22:38:47.674 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht15.ko 2024-03-23T22:38:47.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht21.ko 2024-03-23T22:38:47.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/shtc1.ko 2024-03-23T22:38:47.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sis5595.ko 2024-03-23T22:38:47.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47b397.ko 2024-03-23T22:38:47.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m1.ko 2024-03-23T22:38:47.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m192.ko 2024-03-23T22:38:47.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/thmc50.ko 2024-03-23T22:38:47.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp102.ko 2024-03-23T22:38:47.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp103.ko 2024-03-23T22:38:47.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp401.ko 2024-03-23T22:38:47.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp421.ko 2024-03-23T22:38:47.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via-cputemp.ko 2024-03-23T22:38:47.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via686a.ko 2024-03-23T22:38:47.675 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt1211.ko 2024-03-23T22:38:47.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt8231.ko 2024-03-23T22:38:47.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627ehf.ko 2024-03-23T22:38:47.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627hf.ko 2024-03-23T22:38:47.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83781d.ko 2024-03-23T22:38:47.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83791d.ko 2024-03-23T22:38:47.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83792d.ko 2024-03-23T22:38:47.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83793.ko 2024-03-23T22:38:47.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83795.ko 2024-03-23T22:38:47.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l785ts.ko 2024-03-23T22:38:47.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l786ng.ko 2024-03-23T22:38:47.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c 2024-03-23T22:38:47.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos 2024-03-23T22:38:47.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-23T22:38:47.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-23T22:38:47.676 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses 2024-03-23T22:38:47.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-23T22:38:47.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-23T22:38:47.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-23T22:38:47.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-23T22:38:47.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-23T22:38:47.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-23T22:38:47.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-23T22:38:47.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-23T22:38:47.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-23T22:38:47.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-23T22:38:47.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-23T22:38:47.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-23T22:38:47.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-23T22:38:47.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-23T22:38:47.677 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-23T22:38:47.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-23T22:38:47.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-23T22:38:47.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-23T22:38:47.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-23T22:38:47.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-23T22:38:47.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-23T22:38:47.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-23T22:38:47.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-23T22:38:47.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-dev.ko 2024-03-23T22:38:47.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-mux.ko 2024-03-23T22:38:47.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-23T22:38:47.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-smbus.ko 2024-03-23T22:38:47.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-stub.ko 2024-03-23T22:38:47.678 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband 2024-03-23T22:38:47.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core 2024-03-23T22:38:47.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-23T22:38:47.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_core.ko 2024-03-23T22:38:47.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-23T22:38:47.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-23T22:38:47.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-23T22:38:47.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-23T22:38:47.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-23T22:38:47.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw 2024-03-23T22:38:47.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4 2024-03-23T22:38:47.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-23T22:38:47.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4 2024-03-23T22:38:47.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-23T22:38:47.679 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5 2024-03-23T22:38:47.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-23T22:38:47.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca 2024-03-23T22:38:47.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-23T22:38:47.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp 2024-03-23T22:38:47.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib 2024-03-23T22:38:47.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-23T22:38:47.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser 2024-03-23T22:38:47.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-23T22:38:47.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert 2024-03-23T22:38:47.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-23T22:38:47.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp 2024-03-23T22:38:47.680 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-23T22:38:47.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt 2024-03-23T22:38:47.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-23T22:38:47.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input 2024-03-23T22:38:47.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport 2024-03-23T22:38:47.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-23T22:38:47.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-23T22:38:47.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/gameport.ko 2024-03-23T22:38:47.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/joydev.ko 2024-03-23T22:38:47.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard 2024-03-23T22:38:47.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-23T22:38:47.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse 2024-03-23T22:38:47.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/appletouch.ko 2024-03-23T22:38:47.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/bcm5974.ko 2024-03-23T22:38:47.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/cyapatp.ko 2024-03-23T22:38:47.681 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-23T22:38:47.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/sermouse.ko 2024-03-23T22:38:47.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-23T22:38:47.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-23T22:38:47.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-23T22:38:47.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4 2024-03-23T22:38:47.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-23T22:38:47.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio 2024-03-23T22:38:47.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/altera_ps2.ko 2024-03-23T22:38:47.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/arc_ps2.ko 2024-03-23T22:38:47.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-23T22:38:47.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/serio_raw.ko 2024-03-23T22:38:47.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/sparse-keymap.ko 2024-03-23T22:38:47.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds 2024-03-23T22:38:47.682 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-blinkm.ko 2024-03-23T22:38:47.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3530.ko 2024-03-23T22:38:47.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lp3944.ko 2024-03-23T22:38:47.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-ss4200.ko 2024-03-23T22:38:47.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger 2024-03-23T22:38:47.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-23T22:38:47.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-23T22:38:47.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-23T22:38:47.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-23T22:38:47.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-23T22:38:47.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-23T22:38:47.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-23T22:38:47.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-23T22:38:47.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-23T22:38:47.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md 2024-03-23T22:38:47.683 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache 2024-03-23T22:38:47.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache/bcache.ko 2024-03-23T22:38:47.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-bio-prison.ko 2024-03-23T22:38:47.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache-smq.ko 2024-03-23T22:38:47.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache.ko 2024-03-23T22:38:47.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-crypt.ko 2024-03-23T22:38:47.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-delay.ko 2024-03-23T22:38:47.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-flakey.ko 2024-03-23T22:38:47.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-log-userspace.ko 2024-03-23T22:38:47.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-multipath.ko 2024-03-23T22:38:47.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-queue-length.ko 2024-03-23T22:38:47.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-raid.ko 2024-03-23T22:38:47.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-round-robin.ko 2024-03-23T22:38:47.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-service-time.ko 2024-03-23T22:38:47.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-switch.ko 2024-03-23T22:38:47.684 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-thin-pool.ko 2024-03-23T22:38:47.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-verity.ko 2024-03-23T22:38:47.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data 2024-03-23T22:38:47.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-23T22:38:47.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid0.ko 2024-03-23T22:38:47.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid1.ko 2024-03-23T22:38:47.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid10.ko 2024-03-23T22:38:47.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid456.ko 2024-03-23T22:38:47.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message 2024-03-23T22:38:47.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion 2024-03-23T22:38:47.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptbase.ko 2024-03-23T22:38:47.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptctl.ko 2024-03-23T22:38:47.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptfc.ko 2024-03-23T22:38:47.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptlan.ko 2024-03-23T22:38:47.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptsas.ko 2024-03-23T22:38:47.685 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptscsih.ko 2024-03-23T22:38:47.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptspi.ko 2024-03-23T22:38:47.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd 2024-03-23T22:38:47.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_ich.ko 2024-03-23T22:38:47.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_sch.ko 2024-03-23T22:38:47.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/mfd-core.ko 2024-03-23T22:38:47.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/sm501.ko 2024-03-23T22:38:47.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/viperboard.ko 2024-03-23T22:38:47.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/vx855.ko 2024-03-23T22:38:47.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/wl1273-core.ko 2024-03-23T22:38:47.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc 2024-03-23T22:38:47.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl 2024-03-23T22:38:47.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-23T22:38:47.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds9802als.ko 2024-03-23T22:38:47.686 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds990x.ko 2024-03-23T22:38:47.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/bh1770glc.ko 2024-03-23T22:38:47.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710 2024-03-23T22:38:47.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710/cb710.ko 2024-03-23T22:38:47.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom 2024-03-23T22:38:47.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/at24.ko 2024-03-23T22:38:47.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-23T22:38:47.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/max6875.ko 2024-03-23T22:38:47.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/enclosure.ko 2024-03-23T22:38:47.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/hpilo.ko 2024-03-23T22:38:47.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29003.ko 2024-03-23T22:38:47.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29020.ko 2024-03-23T22:38:47.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d 2024-03-23T22:38:47.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-23T22:38:47.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-23T22:38:47.687 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei 2024-03-23T22:38:47.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-me.ko 2024-03-23T22:38:47.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-txe.ko 2024-03-23T22:38:47.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei.ko 2024-03-23T22:38:47.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru 2024-03-23T22:38:47.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-23T22:38:47.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp 2024-03-23T22:38:47.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-23T22:38:47.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-23T22:38:47.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-23T22:38:47.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_7xx1.ko 2024-03-23T22:38:47.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_core.ko 2024-03-23T22:38:47.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tsl2550.ko 2024-03-23T22:38:47.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_balloon.ko 2024-03-23T22:38:47.688 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci 2024-03-23T22:38:47.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-23T22:38:47.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net 2024-03-23T22:38:47.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding 2024-03-23T22:38:47.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding/bonding.ko 2024-03-23T22:38:47.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa 2024-03-23T22:38:47.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53 2024-03-23T22:38:47.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-23T22:38:47.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-23T22:38:47.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-23T22:38:47.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx 2024-03-23T22:38:47.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-23T22:38:47.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dummy.ko 2024-03-23T22:38:47.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/eql.ko 2024-03-23T22:38:47.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet 2024-03-23T22:38:47.689 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com 2024-03-23T22:38:47.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-23T22:38:47.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-23T22:38:47.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-23T22:38:47.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-23T22:38:47.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390 2024-03-23T22:38:47.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-23T22:38:47.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-23T22:38:47.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-23T22:38:47.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-23T22:38:47.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec 2024-03-23T22:38:47.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-23T22:38:47.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere 2024-03-23T22:38:47.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-23T22:38:47.690 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon 2024-03-23T22:38:47.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-23T22:38:47.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera 2024-03-23T22:38:47.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-23T22:38:47.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd 2024-03-23T22:38:47.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-23T22:38:47.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-23T22:38:47.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-23T22:38:47.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros 2024-03-23T22:38:47.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx 2024-03-23T22:38:47.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-23T22:38:47.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-23T22:38:47.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-23T22:38:47.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-23T22:38:47.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-23T22:38:47.691 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx 2024-03-23T22:38:47.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-23T22:38:47.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-23T22:38:47.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom 2024-03-23T22:38:47.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-23T22:38:47.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-23T22:38:47.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-23T22:38:47.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-23T22:38:47.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-23T22:38:47.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet 2024-03-23T22:38:47.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-23T22:38:47.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-23T22:38:47.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade 2024-03-23T22:38:47.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna 2024-03-23T22:38:47.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-23T22:38:47.692 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio 2024-03-23T22:38:47.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-23T22:38:47.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-23T22:38:47.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-23T22:38:47.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-23T22:38:47.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-23T22:38:47.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-23T22:38:47.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-23T22:38:47.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-23T22:38:47.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-23T22:38:47.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-23T22:38:47.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco 2024-03-23T22:38:47.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic 2024-03-23T22:38:47.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-23T22:38:47.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec 2024-03-23T22:38:47.693 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip 2024-03-23T22:38:47.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-23T22:38:47.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-23T22:38:47.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-23T22:38:47.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-23T22:38:47.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-23T22:38:47.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-23T22:38:47.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink 2024-03-23T22:38:47.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-23T22:38:47.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-23T22:38:47.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dnet.ko 2024-03-23T22:38:47.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex 2024-03-23T22:38:47.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet 2024-03-23T22:38:47.694 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-23T22:38:47.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ethoc.ko 2024-03-23T22:38:47.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fealnx.ko 2024-03-23T22:38:47.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel 2024-03-23T22:38:47.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-23T22:38:47.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000 2024-03-23T22:38:47.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-23T22:38:47.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e 2024-03-23T22:38:47.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-23T22:38:47.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k 2024-03-23T22:38:47.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-23T22:38:47.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e 2024-03-23T22:38:47.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-23T22:38:47.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf 2024-03-23T22:38:47.695 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-23T22:38:47.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb 2024-03-23T22:38:47.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-23T22:38:47.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf 2024-03-23T22:38:47.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-23T22:38:47.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-23T22:38:47.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-23T22:38:47.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-23T22:38:47.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-23T22:38:47.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/jme.ko 2024-03-23T22:38:47.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell 2024-03-23T22:38:47.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-23T22:38:47.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-23T22:38:47.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-23T22:38:47.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox 2024-03-23T22:38:47.696 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-23T22:38:47.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-23T22:38:47.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-23T22:38:47.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-23T22:38:47.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-23T22:38:47.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-23T22:38:47.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel 2024-03-23T22:38:47.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-23T22:38:47.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom 2024-03-23T22:38:47.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-23T22:38:47.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-23T22:38:47.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi 2024-03-23T22:38:47.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-23T22:38:47.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-23T22:38:47.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion 2024-03-23T22:38:47.697 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-23T22:38:47.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia 2024-03-23T22:38:47.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-23T22:38:47.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines 2024-03-23T22:38:47.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-23T22:38:47.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-23T22:38:47.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic 2024-03-23T22:38:47.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-23T22:38:47.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-23T22:38:47.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-23T22:38:47.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-23T22:38:47.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-23T22:38:47.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc 2024-03-23T22:38:47.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-23T22:38:47.698 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek 2024-03-23T22:38:47.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-23T22:38:47.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-23T22:38:47.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-23T22:38:47.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-23T22:38:47.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker 2024-03-23T22:38:47.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-23T22:38:47.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc 2024-03-23T22:38:47.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-23T22:38:47.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan 2024-03-23T22:38:47.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-23T22:38:47.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis 2024-03-23T22:38:47.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-23T22:38:47.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-23T22:38:47.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc 2024-03-23T22:38:47.699 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-23T22:38:47.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-23T22:38:47.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-23T22:38:47.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-23T22:38:47.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro 2024-03-23T22:38:47.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-23T22:38:47.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-23T22:38:47.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-23T22:38:47.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun 2024-03-23T22:38:47.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-23T22:38:47.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-23T22:38:47.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-23T22:38:47.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-23T22:38:47.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti 2024-03-23T22:38:47.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-23T22:38:47.700 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti 2024-03-23T22:38:47.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-23T22:38:47.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via 2024-03-23T22:38:47.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-23T22:38:47.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-23T22:38:47.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet 2024-03-23T22:38:47.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-23T22:38:47.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-23T22:38:47.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom 2024-03-23T22:38:47.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-23T22:38:47.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/geneve.ko 2024-03-23T22:38:47.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv 2024-03-23T22:38:47.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-23T22:38:47.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154 2024-03-23T22:38:47.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-23T22:38:47.701 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ifb.ko 2024-03-23T22:38:47.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan 2024-03-23T22:38:47.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-23T22:38:47.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvlan.ko 2024-03-23T22:38:47.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvtap.ko 2024-03-23T22:38:47.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio 2024-03-23T22:38:47.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio.ko 2024-03-23T22:38:47.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-23T22:38:47.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-23T22:38:47.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-23T22:38:47.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mii.ko 2024-03-23T22:38:47.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/net_failover.ko 2024-03-23T22:38:47.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/netconsole.ko 2024-03-23T22:38:47.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/nlmon.ko 2024-03-23T22:38:47.702 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs 2024-03-23T22:38:47.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-23T22:38:47.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-23T22:38:47.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy 2024-03-23T22:38:47.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/amd.ko 2024-03-23T22:38:47.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-23T22:38:47.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-23T22:38:47.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-23T22:38:47.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm87xx.ko 2024-03-23T22:38:47.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/broadcom.ko 2024-03-23T22:38:47.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/cicada.ko 2024-03-23T22:38:47.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/davicom.ko 2024-03-23T22:38:47.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/dp83640.ko 2024-03-23T22:38:47.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/et1011c.ko 2024-03-23T22:38:47.703 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/icplus.ko 2024-03-23T22:38:47.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/lxt.ko 2024-03-23T22:38:47.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/marvell.ko 2024-03-23T22:38:47.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/micrel.ko 2024-03-23T22:38:47.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/national.ko 2024-03-23T22:38:47.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/phylink.ko 2024-03-23T22:38:47.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/qsemi.ko 2024-03-23T22:38:47.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/realtek.ko 2024-03-23T22:38:47.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/smsc.ko 2024-03-23T22:38:47.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/ste10Xp.ko 2024-03-23T22:38:47.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/vitesse.ko 2024-03-23T22:38:47.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp 2024-03-23T22:38:47.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-23T22:38:47.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_async.ko 2024-03-23T22:38:47.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-23T22:38:47.704 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-23T22:38:47.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-23T22:38:47.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-23T22:38:47.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppoe.ko 2024-03-23T22:38:47.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppox.ko 2024-03-23T22:38:47.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pptp.ko 2024-03-23T22:38:47.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip 2024-03-23T22:38:47.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slhc.ko 2024-03-23T22:38:47.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slip.ko 2024-03-23T22:38:47.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/sungem_phy.ko 2024-03-23T22:38:47.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tap.ko 2024-03-23T22:38:47.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team 2024-03-23T22:38:47.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team.ko 2024-03-23T22:38:47.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-23T22:38:47.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-23T22:38:47.705 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-23T22:38:47.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_random.ko 2024-03-23T22:38:47.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-23T22:38:47.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tun.ko 2024-03-23T22:38:47.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/veth.ko 2024-03-23T22:38:47.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/virtio_net.ko 2024-03-23T22:38:47.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3 2024-03-23T22:38:47.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-23T22:38:47.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan 2024-03-23T22:38:47.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan/vxlan.ko 2024-03-23T22:38:47.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback 2024-03-23T22:38:47.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-23T22:38:47.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netfront.ko 2024-03-23T22:38:47.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb 2024-03-23T22:38:47.706 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb/ntb.ko 2024-03-23T22:38:47.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme 2024-03-23T22:38:47.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host 2024-03-23T22:38:47.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme-core.ko 2024-03-23T22:38:47.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme.ko 2024-03-23T22:38:47.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport 2024-03-23T22:38:47.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport.ko 2024-03-23T22:38:47.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_cs.ko 2024-03-23T22:38:47.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_pc.ko 2024-03-23T22:38:47.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_serial.ko 2024-03-23T22:38:47.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci 2024-03-23T22:38:47.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug 2024-03-23T22:38:47.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-23T22:38:47.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie 2024-03-23T22:38:47.707 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-23T22:38:47.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/xen-pcifront.ko 2024-03-23T22:38:47.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia 2024-03-23T22:38:47.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/i82092.ko 2024-03-23T22:38:47.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pd6729.ko 2024-03-23T22:38:47.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-23T22:38:47.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl 2024-03-23T22:38:47.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel 2024-03-23T22:38:47.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-23T22:38:47.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform 2024-03-23T22:38:47.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86 2024-03-23T22:38:47.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-23T22:38:47.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acerhdf.ko 2024-03-23T22:38:47.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-23T22:38:47.708 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-23T22:38:47.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-23T22:38:47.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-23T22:38:47.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-23T22:38:47.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-23T22:38:47.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-23T22:38:47.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-23T22:38:47.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-23T22:38:47.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/hdaps.ko 2024-03-23T22:38:47.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel 2024-03-23T22:38:47.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-23T22:38:47.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel_ips.ko 2024-03-23T22:38:47.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-23T22:38:47.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-23T22:38:47.709 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-23T22:38:47.710 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-23T22:38:47.710 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-23T22:38:47.710 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-23T22:38:47.710 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-23T22:38:47.710 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-23T22:38:47.710 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-23T22:38:47.710 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi.ko 2024-03-23T22:38:47.710 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap 2024-03-23T22:38:47.710 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-23T22:38:47.710 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-23T22:38:47.710 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps 2024-03-23T22:38:47.710 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients 2024-03-23T22:38:47.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-23T22:38:47.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-23T22:38:47.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps_parport.ko 2024-03-23T22:38:47.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/pps_core.ko 2024-03-23T22:38:47.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp 2024-03-23T22:38:47.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp.ko 2024-03-23T22:38:47.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp_kvm.ko 2024-03-23T22:38:47.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc 2024-03-23T22:38:47.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-23T22:38:47.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-23T22:38:47.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-23T22:38:47.711 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-23T22:38:47.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-23T22:38:47.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-23T22:38:47.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-23T22:38:47.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-23T22:38:47.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-23T22:38:47.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-23T22:38:47.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-em3027.ko 2024-03-23T22:38:47.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-23T22:38:47.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-23T22:38:47.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-23T22:38:47.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-23T22:38:47.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-23T22:38:47.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-23T22:38:47.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max6900.ko 2024-03-23T22:38:47.712 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-23T22:38:47.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-23T22:38:47.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-23T22:38:47.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-23T22:38:47.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-23T22:38:47.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-23T22:38:47.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-23T22:38:47.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-23T22:38:47.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-23T22:38:47.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-23T22:38:47.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-23T22:38:47.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-23T22:38:47.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-x1205.ko 2024-03-23T22:38:47.713 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi 2024-03-23T22:38:47.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-9xxx.ko 2024-03-23T22:38:47.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-sas.ko 2024-03-23T22:38:47.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-xxxx.ko 2024-03-23T22:38:47.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/BusLogic.ko 2024-03-23T22:38:47.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/a100u2w.ko 2024-03-23T22:38:47.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid 2024-03-23T22:38:47.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-23T22:38:47.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/advansys.ko 2024-03-23T22:38:47.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx 2024-03-23T22:38:47.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-23T22:38:47.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-23T22:38:47.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/am53c974.ko 2024-03-23T22:38:47.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr 2024-03-23T22:38:47.714 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-23T22:38:47.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/atp870u.ko 2024-03-23T22:38:47.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi 2024-03-23T22:38:47.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-23T22:38:47.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa 2024-03-23T22:38:47.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa/bfa.ko 2024-03-23T22:38:47.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc 2024-03-23T22:38:47.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-23T22:38:47.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i 2024-03-23T22:38:47.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-23T22:38:47.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ch.ko 2024-03-23T22:38:47.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor 2024-03-23T22:38:47.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-23T22:38:47.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi 2024-03-23T22:38:47.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-23T22:38:47.715 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-23T22:38:47.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-23T22:38:47.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-23T22:38:47.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-23T22:38:47.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dc395x.ko 2024-03-23T22:38:47.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler 2024-03-23T22:38:47.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-23T22:38:47.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-23T22:38:47.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-23T22:38:47.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-23T22:38:47.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dmx3191d.ko 2024-03-23T22:38:47.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r 2024-03-23T22:38:47.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-23T22:38:47.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esp_scsi.ko 2024-03-23T22:38:47.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe 2024-03-23T22:38:47.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-23T22:38:47.716 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-23T22:38:47.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic 2024-03-23T22:38:47.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic/fnic.ko 2024-03-23T22:38:47.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hpsa.ko 2024-03-23T22:38:47.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hptiop.ko 2024-03-23T22:38:47.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hv_storvsc.ko 2024-03-23T22:38:47.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/initio.ko 2024-03-23T22:38:47.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ipr.ko 2024-03-23T22:38:47.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ips.ko 2024-03-23T22:38:47.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci 2024-03-23T22:38:47.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci/isci.ko 2024-03-23T22:38:47.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-23T22:38:47.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-23T22:38:47.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc 2024-03-23T22:38:47.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc/libfc.ko 2024-03-23T22:38:47.717 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi.ko 2024-03-23T22:38:47.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-23T22:38:47.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas 2024-03-23T22:38:47.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas/libsas.ko 2024-03-23T22:38:47.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc 2024-03-23T22:38:47.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-23T22:38:47.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid 2024-03-23T22:38:47.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid.ko 2024-03-23T22:38:47.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-23T22:38:47.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-23T22:38:47.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-23T22:38:47.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas 2024-03-23T22:38:47.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-23T22:38:47.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas 2024-03-23T22:38:47.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-23T22:38:47.718 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvumi.ko 2024-03-23T22:38:47.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001 2024-03-23T22:38:47.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-23T22:38:47.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pmcraid.ko 2024-03-23T22:38:47.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla1280.ko 2024-03-23T22:38:47.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx 2024-03-23T22:38:47.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-23T22:38:47.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-23T22:38:47.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx 2024-03-23T22:38:47.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-23T22:38:47.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/raid_class.ko 2024-03-23T22:38:47.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_debug.ko 2024-03-23T22:38:47.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-23T22:38:47.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-23T22:38:47.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-23T22:38:47.719 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-23T22:38:47.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-23T22:38:47.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ses.ko 2024-03-23T22:38:47.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/st.ko 2024-03-23T22:38:47.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/stex.ko 2024-03-23T22:38:47.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2 2024-03-23T22:38:47.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-23T22:38:47.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/virtio_scsi.ko 2024-03-23T22:38:47.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-23T22:38:47.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/wd719x.ko 2024-03-23T22:38:47.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb 2024-03-23T22:38:47.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb/ssb.ko 2024-03-23T22:38:47.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target 2024-03-23T22:38:47.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi 2024-03-23T22:38:47.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-23T22:38:47.720 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback 2024-03-23T22:38:47.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-23T22:38:47.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_file.ko 2024-03-23T22:38:47.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_iblock.ko 2024-03-23T22:38:47.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_mod.ko 2024-03-23T22:38:47.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_pscsi.ko 2024-03-23T22:38:47.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_user.ko 2024-03-23T22:38:47.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc 2024-03-23T22:38:47.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-23T22:38:47.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal 2024-03-23T22:38:47.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel 2024-03-23T22:38:47.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal 2024-03-23T22:38:47.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-23T22:38:47.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-23T22:38:47.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-23T22:38:47.721 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-23T22:38:47.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-23T22:38:47.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-23T22:38:47.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-23T22:38:47.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-23T22:38:47.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-23T22:38:47.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-23T22:38:47.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-23T22:38:47.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-23T22:38:47.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-23T22:38:47.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-23T22:38:47.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-23T22:38:47.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-23T22:38:47.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-23T22:38:47.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-23T22:38:47.722 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty 2024-03-23T22:38:47.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless 2024-03-23T22:38:47.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-23T22:38:47.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_gsm.ko 2024-03-23T22:38:47.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_hdlc.ko 2024-03-23T22:38:47.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/nozomi.ko 2024-03-23T22:38:47.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial 2024-03-23T22:38:47.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250 2024-03-23T22:38:47.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-23T22:38:47.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/arc_uart.ko 2024-03-23T22:38:47.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm 2024-03-23T22:38:47.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-23T22:38:47.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs 2024-03-23T22:38:47.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core 2024-03-23T22:38:47.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-23T22:38:47.723 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host 2024-03-23T22:38:47.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-23T22:38:47.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio 2024-03-23T22:38:47.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio.ko 2024-03-23T22:38:47.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_aec.ko 2024-03-23T22:38:47.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_cif.ko 2024-03-23T22:38:47.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_pci_generic.ko 2024-03-23T22:38:47.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_sercos3.ko 2024-03-23T22:38:47.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb 2024-03-23T22:38:47.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class 2024-03-23T22:38:47.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-acm.ko 2024-03-23T22:38:47.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-23T22:38:47.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usblp.ko 2024-03-23T22:38:47.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usbtmc.ko 2024-03-23T22:38:47.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host 2024-03-23T22:38:47.724 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-23T22:38:47.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image 2024-03-23T22:38:47.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/mdc800.ko 2024-03-23T22:38:47.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/microtek.ko 2024-03-23T22:38:47.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc 2024-03-23T22:38:47.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/adutux.ko 2024-03-23T22:38:47.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/appledisplay.ko 2024-03-23T22:38:47.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi26.ko 2024-03-23T22:38:47.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi62.ko 2024-03-23T22:38:47.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ezusb.ko 2024-03-23T22:38:47.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/idmouse.ko 2024-03-23T22:38:47.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/iowarrior.ko 2024-03-23T22:38:47.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-23T22:38:47.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ldusb.ko 2024-03-23T22:38:47.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/legousbtower.ko 2024-03-23T22:38:47.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga 2024-03-23T22:38:47.725 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-23T22:38:47.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-23T22:38:47.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usb3503.ko 2024-03-23T22:38:47.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usblcd.ko 2024-03-23T22:38:47.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-23T22:38:47.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/uss720.ko 2024-03-23T22:38:47.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/yurex.ko 2024-03-23T22:38:47.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy 2024-03-23T22:38:47.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-generic.ko 2024-03-23T22:38:47.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial 2024-03-23T22:38:47.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/aircable.ko 2024-03-23T22:38:47.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ark3116.ko 2024-03-23T22:38:47.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-23T22:38:47.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ch341.ko 2024-03-23T22:38:47.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cp210x.ko 2024-03-23T22:38:47.726 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cyberjack.ko 2024-03-23T22:38:47.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-23T22:38:47.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-23T22:38:47.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/empeg.ko 2024-03-23T22:38:47.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-23T22:38:47.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-23T22:38:47.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-23T22:38:47.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_ti.ko 2024-03-23T22:38:47.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipaq.ko 2024-03-23T22:38:47.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipw.ko 2024-03-23T22:38:47.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ir-usb.ko 2024-03-23T22:38:47.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-23T22:38:47.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan.ko 2024-03-23T22:38:47.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-23T22:38:47.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-23T22:38:47.727 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-23T22:38:47.728 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mct_u232.ko 2024-03-23T22:38:47.728 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7720.ko 2024-03-23T22:38:47.728 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7840.ko 2024-03-23T22:38:47.728 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/navman.ko 2024-03-23T22:38:47.728 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/omninet.ko 2024-03-23T22:38:47.728 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/opticon.ko 2024-03-23T22:38:47.728 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/option.ko 2024-03-23T22:38:47.728 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/oti6858.ko 2024-03-23T22:38:47.728 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/pl2303.ko 2024-03-23T22:38:47.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcaux.ko 2024-03-23T22:38:47.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcserial.ko 2024-03-23T22:38:47.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/quatech2.ko 2024-03-23T22:38:47.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/safe_serial.ko 2024-03-23T22:38:47.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/sierra.ko 2024-03-23T22:38:47.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-23T22:38:47.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ssu100.ko 2024-03-23T22:38:47.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/symbolserial.ko 2024-03-23T22:38:47.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-23T22:38:47.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-23T22:38:47.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_debug.ko 2024-03-23T22:38:47.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-23T22:38:47.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/visor.ko 2024-03-23T22:38:47.729 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/whiteheat.ko 2024-03-23T22:38:47.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-23T22:38:47.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage 2024-03-23T22:38:47.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/uas.ko 2024-03-23T22:38:47.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-23T22:38:47.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-23T22:38:47.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-23T22:38:47.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-23T22:38:47.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-23T22:38:47.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-23T22:38:47.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-23T22:38:47.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-karma.ko 2024-03-23T22:38:47.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-23T22:38:47.730 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-23T22:38:47.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-23T22:38:47.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-23T22:38:47.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-23T22:38:47.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/usb-storage.ko 2024-03-23T22:38:47.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip 2024-03-23T22:38:47.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-23T22:38:47.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-23T22:38:47.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-23T22:38:47.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio 2024-03-23T22:38:47.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci 2024-03-23T22:38:47.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-23T22:38:47.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-23T22:38:47.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio.ko 2024-03-23T22:38:47.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-23T22:38:47.731 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost 2024-03-23T22:38:47.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost.ko 2024-03-23T22:38:47.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-23T22:38:47.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_net.ko 2024-03-23T22:38:47.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_scsi.ko 2024-03-23T22:38:47.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video 2024-03-23T22:38:47.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight 2024-03-23T22:38:47.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/apple_bl.ko 2024-03-23T22:38:47.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lcd.ko 2024-03-23T22:38:47.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-23T22:38:47.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio 2024-03-23T22:38:47.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio.ko 2024-03-23T22:38:47.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_balloon.ko 2024-03-23T22:38:47.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_mmio.ko 2024-03-23T22:38:47.732 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci.ko 2024-03-23T22:38:47.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-23T22:38:47.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-23T22:38:47.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_ring.ko 2024-03-23T22:38:47.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1 2024-03-23T22:38:47.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters 2024-03-23T22:38:47.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2482.ko 2024-03-23T22:38:47.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2490.ko 2024-03-23T22:38:47.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves 2024-03-23T22:38:47.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-23T22:38:47.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-23T22:38:47.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-23T22:38:47.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-23T22:38:47.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-23T22:38:47.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-23T22:38:47.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-23T22:38:47.733 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-23T22:38:47.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-23T22:38:47.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-23T22:38:47.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-23T22:38:47.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/wire.ko 2024-03-23T22:38:47.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog 2024-03-23T22:38:47.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-23T22:38:47.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-23T22:38:47.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-23T22:38:47.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/hpwdt.ko 2024-03-23T22:38:47.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/i6300esb.ko 2024-03-23T22:38:47.734 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-23T22:38:47.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-23T22:38:47.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ib700wdt.ko 2024-03-23T22:38:47.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ibmasr.ko 2024-03-23T22:38:47.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-23T22:38:47.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-23T22:38:47.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it87_wdt.ko 2024-03-23T22:38:47.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/machzwd.ko 2024-03-23T22:38:47.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/nv_tco.ko 2024-03-23T22:38:47.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-23T22:38:47.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-23T22:38:47.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-23T22:38:47.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-23T22:38:47.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/softdog.ko 2024-03-23T22:38:47.735 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-23T22:38:47.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/via_wdt.ko 2024-03-23T22:38:47.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-23T22:38:47.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-23T22:38:47.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-23T22:38:47.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wdt_pci.ko 2024-03-23T22:38:47.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/xen_wdt.ko 2024-03-23T22:38:47.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen 2024-03-23T22:38:47.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-23T22:38:47.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-evtchn.ko 2024-03-23T22:38:47.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntalloc.ko 2024-03-23T22:38:47.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntdev.ko 2024-03-23T22:38:47.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback 2024-03-23T22:38:47.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-23T22:38:47.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-privcmd.ko 2024-03-23T22:38:47.736 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs 2024-03-23T22:38:47.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-23T22:38:47.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs 2024-03-23T22:38:47.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p 2024-03-23T22:38:47.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p/9p.ko 2024-03-23T22:38:47.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs 2024-03-23T22:38:47.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs/affs.ko 2024-03-23T22:38:47.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs 2024-03-23T22:38:47.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs/befs.ko 2024-03-23T22:38:47.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/binfmt_misc.ko 2024-03-23T22:38:47.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs 2024-03-23T22:38:47.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs/btrfs.ko 2024-03-23T22:38:47.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles 2024-03-23T22:38:47.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles/cachefiles.ko 2024-03-23T22:38:47.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph 2024-03-23T22:38:47.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph/ceph.ko 2024-03-23T22:38:47.737 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda 2024-03-23T22:38:47.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda/coda.ko 2024-03-23T22:38:47.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs 2024-03-23T22:38:47.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs/cramfs.ko 2024-03-23T22:38:47.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm 2024-03-23T22:38:47.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm/dlm.ko 2024-03-23T22:38:47.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs 2024-03-23T22:38:47.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-23T22:38:47.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs 2024-03-23T22:38:47.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs/f2fs.ko 2024-03-23T22:38:47.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat 2024-03-23T22:38:47.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/fat.ko 2024-03-23T22:38:47.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/msdos.ko 2024-03-23T22:38:47.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/vfat.ko 2024-03-23T22:38:47.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse 2024-03-23T22:38:47.738 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/cuse.ko 2024-03-23T22:38:47.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/fuse.ko 2024-03-23T22:38:47.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2 2024-03-23T22:38:47.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2/gfs2.ko 2024-03-23T22:38:47.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs 2024-03-23T22:38:47.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs/hfs.ko 2024-03-23T22:38:47.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus 2024-03-23T22:38:47.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus/hfsplus.ko 2024-03-23T22:38:47.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs 2024-03-23T22:38:47.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs/isofs.ko 2024-03-23T22:38:47.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs 2024-03-23T22:38:47.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs/jfs.ko 2024-03-23T22:38:47.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd 2024-03-23T22:38:47.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd/lockd.ko 2024-03-23T22:38:47.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix 2024-03-23T22:38:47.739 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix/minix.ko 2024-03-23T22:38:47.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs 2024-03-23T22:38:47.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs/netfs.ko 2024-03-23T22:38:47.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs 2024-03-23T22:38:47.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout 2024-03-23T22:38:47.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-23T22:38:47.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout 2024-03-23T22:38:47.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-23T22:38:47.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout 2024-03-23T22:38:47.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-23T22:38:47.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfs.ko 2024-03-23T22:38:47.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv3.ko 2024-03-23T22:38:47.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv4.ko 2024-03-23T22:38:47.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common 2024-03-23T22:38:47.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/grace.ko 2024-03-23T22:38:47.740 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/nfs_acl.ko 2024-03-23T22:38:47.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd 2024-03-23T22:38:47.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd/nfsd.ko 2024-03-23T22:38:47.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2 2024-03-23T22:38:47.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2/nilfs2.ko 2024-03-23T22:38:47.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls 2024-03-23T22:38:47.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-celtic.ko 2024-03-23T22:38:47.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-centeuro.ko 2024-03-23T22:38:47.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-croatian.ko 2024-03-23T22:38:47.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-cyrillic.ko 2024-03-23T22:38:47.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-gaelic.ko 2024-03-23T22:38:47.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-greek.ko 2024-03-23T22:38:47.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-iceland.ko 2024-03-23T22:38:47.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-inuit.ko 2024-03-23T22:38:47.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-roman.ko 2024-03-23T22:38:47.741 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-romanian.ko 2024-03-23T22:38:47.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-turkish.ko 2024-03-23T22:38:47.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1250.ko 2024-03-23T22:38:47.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1251.ko 2024-03-23T22:38:47.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1255.ko 2024-03-23T22:38:47.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp737.ko 2024-03-23T22:38:47.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp775.ko 2024-03-23T22:38:47.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp850.ko 2024-03-23T22:38:47.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp852.ko 2024-03-23T22:38:47.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp855.ko 2024-03-23T22:38:47.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp857.ko 2024-03-23T22:38:47.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp860.ko 2024-03-23T22:38:47.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp861.ko 2024-03-23T22:38:47.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp862.ko 2024-03-23T22:38:47.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp863.ko 2024-03-23T22:38:47.742 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp864.ko 2024-03-23T22:38:47.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp865.ko 2024-03-23T22:38:47.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp866.ko 2024-03-23T22:38:47.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp869.ko 2024-03-23T22:38:47.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp874.ko 2024-03-23T22:38:47.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp932.ko 2024-03-23T22:38:47.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp936.ko 2024-03-23T22:38:47.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp949.ko 2024-03-23T22:38:47.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp950.ko 2024-03-23T22:38:47.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_euc-jp.ko 2024-03-23T22:38:47.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-1.ko 2024-03-23T22:38:47.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-13.ko 2024-03-23T22:38:47.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-14.ko 2024-03-23T22:38:47.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-15.ko 2024-03-23T22:38:47.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-2.ko 2024-03-23T22:38:47.743 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-3.ko 2024-03-23T22:38:47.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-4.ko 2024-03-23T22:38:47.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-5.ko 2024-03-23T22:38:47.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-6.ko 2024-03-23T22:38:47.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-7.ko 2024-03-23T22:38:47.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-9.ko 2024-03-23T22:38:47.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-r.ko 2024-03-23T22:38:47.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-ru.ko 2024-03-23T22:38:47.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-u.ko 2024-03-23T22:38:47.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-23T22:38:47.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_utf8.ko 2024-03-23T22:38:47.744 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2 2024-03-23T22:38:47.745 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster 2024-03-23T22:38:47.745 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-23T22:38:47.745 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm 2024-03-23T22:38:47.745 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-23T22:38:47.745 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs 2024-03-23T22:38:47.745 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-23T22:38:47.745 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2.ko 2024-03-23T22:38:47.745 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-23T22:38:47.745 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-23T22:38:47.745 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-23T22:38:47.745 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs 2024-03-23T22:38:47.745 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs/overlay.ko 2024-03-23T22:38:47.745 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore 2024-03-23T22:38:47.745 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore/ramoops.ko 2024-03-23T22:38:47.745 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs 2024-03-23T22:38:47.746 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs/reiserfs.ko 2024-03-23T22:38:47.746 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs 2024-03-23T22:38:47.746 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs/romfs.ko 2024-03-23T22:38:47.746 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb 2024-03-23T22:38:47.746 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client 2024-03-23T22:38:47.746 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client/cifs.ko 2024-03-23T22:38:47.746 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common 2024-03-23T22:38:47.746 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_arc4.ko 2024-03-23T22:38:47.746 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_md4.ko 2024-03-23T22:38:47.746 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs 2024-03-23T22:38:47.746 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs/squashfs.ko 2024-03-23T22:38:47.746 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv 2024-03-23T22:38:47.746 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv/sysv.ko 2024-03-23T22:38:47.746 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf 2024-03-23T22:38:47.747 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf/udf.ko 2024-03-23T22:38:47.747 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs 2024-03-23T22:38:47.747 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs/ufs.ko 2024-03-23T22:38:47.747 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs 2024-03-23T22:38:47.747 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs/xfs.ko 2024-03-23T22:38:47.747 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel 2024-03-23T22:38:47.747 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu 2024-03-23T22:38:47.747 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu/rcutorture.ko 2024-03-23T22:38:47.747 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/torture.ko 2024-03-23T22:38:47.747 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace 2024-03-23T22:38:47.747 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-23T22:38:47.747 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib 2024-03-23T22:38:47.747 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/asn1_encoder.ko 2024-03-23T22:38:47.748 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc-itu-t.ko 2024-03-23T22:38:47.748 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc8.ko 2024-03-23T22:38:47.748 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto 2024-03-23T22:38:47.748 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libarc4.ko 2024-03-23T22:38:47.748 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libdes.ko 2024-03-23T22:38:47.748 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/libcrc32c.ko 2024-03-23T22:38:47.748 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lru_cache.ko 2024-03-23T22:38:47.748 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4 2024-03-23T22:38:47.748 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4_compress.ko 2024-03-23T22:38:47.748 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4hc_compress.ko 2024-03-23T22:38:47.748 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math 2024-03-23T22:38:47.748 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math/cordic.ko 2024-03-23T22:38:47.748 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6 2024-03-23T22:38:47.748 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6/raid6_pq.ko 2024-03-23T22:38:47.748 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon 2024-03-23T22:38:47.749 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-23T22:38:47.749 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_bm.ko 2024-03-23T22:38:47.749 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_fsm.ko 2024-03-23T22:38:47.749 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_kmp.ko 2024-03-23T22:38:47.749 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd 2024-03-23T22:38:47.749 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd/zstd_compress.ko 2024-03-23T22:38:47.749 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm 2024-03-23T22:38:47.749 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm/hwpoison-inject.ko 2024-03-23T22:38:47.749 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net 2024-03-23T22:38:47.749 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan 2024-03-23T22:38:47.749 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/6lowpan.ko 2024-03-23T22:38:47.749 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_dest.ko 2024-03-23T22:38:47.749 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_fragment.ko 2024-03-23T22:38:47.749 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_hop.ko 2024-03-23T22:38:47.749 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-23T22:38:47.750 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_mobility.ko 2024-03-23T22:38:47.750 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_routing.ko 2024-03-23T22:38:47.750 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_udp.ko 2024-03-23T22:38:47.750 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802 2024-03-23T22:38:47.750 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/garp.ko 2024-03-23T22:38:47.750 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/mrp.ko 2024-03-23T22:38:47.750 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/p8022.ko 2024-03-23T22:38:47.750 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/psnap.ko 2024-03-23T22:38:47.750 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/stp.ko 2024-03-23T22:38:47.750 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q 2024-03-23T22:38:47.750 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q/8021q.ko 2024-03-23T22:38:47.750 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p 2024-03-23T22:38:47.750 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet.ko 2024-03-23T22:38:47.750 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_fd.ko 2024-03-23T22:38:47.750 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_rdma.ko 2024-03-23T22:38:47.751 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_virtio.ko 2024-03-23T22:38:47.751 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv 2024-03-23T22:38:47.751 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv/batman-adv.ko 2024-03-23T22:38:47.751 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge 2024-03-23T22:38:47.751 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/br_netfilter.ko 2024-03-23T22:38:47.751 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/bridge.ko 2024-03-23T22:38:47.751 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter 2024-03-23T22:38:47.751 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-23T22:38:47.751 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-23T22:38:47.751 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-23T22:38:47.751 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-23T22:38:47.751 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-23T22:38:47.751 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-23T22:38:47.751 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-23T22:38:47.752 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-23T22:38:47.752 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-23T22:38:47.752 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-23T22:38:47.752 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-23T22:38:47.752 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-23T22:38:47.752 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-23T22:38:47.752 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-23T22:38:47.752 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-23T22:38:47.752 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-23T22:38:47.752 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-23T22:38:47.752 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-23T22:38:47.752 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-23T22:38:47.752 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-23T22:38:47.752 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtables.ko 2024-03-23T22:38:47.752 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-23T22:38:47.753 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-23T22:38:47.753 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-23T22:38:47.753 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph 2024-03-23T22:38:47.753 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph/libceph.ko 2024-03-23T22:38:47.753 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core 2024-03-23T22:38:47.753 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/failover.ko 2024-03-23T22:38:47.753 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/pktgen.ko 2024-03-23T22:38:47.753 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp 2024-03-23T22:38:47.753 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp.ko 2024-03-23T22:38:47.753 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_diag.ko 2024-03-23T22:38:47.753 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv4.ko 2024-03-23T22:38:47.753 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv6.ko 2024-03-23T22:38:47.753 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver 2024-03-23T22:38:47.753 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver/dns_resolver.ko 2024-03-23T22:38:47.753 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa 2024-03-23T22:38:47.753 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/dsa_core.ko 2024-03-23T22:38:47.754 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_brcm.ko 2024-03-23T22:38:47.754 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_dsa.ko 2024-03-23T22:38:47.754 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_none.ko 2024-03-23T22:38:47.754 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_trailer.ko 2024-03-23T22:38:47.754 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154 2024-03-23T22:38:47.754 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan 2024-03-23T22:38:47.754 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-23T22:38:47.754 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154.ko 2024-03-23T22:38:47.754 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-23T22:38:47.754 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4 2024-03-23T22:38:47.754 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ah4.ko 2024-03-23T22:38:47.754 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/esp4.ko 2024-03-23T22:38:47.754 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/fou.ko 2024-03-23T22:38:47.754 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/gre.ko 2024-03-23T22:38:47.755 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/inet_diag.ko 2024-03-23T22:38:47.755 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_gre.ko 2024-03-23T22:38:47.755 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_tunnel.ko 2024-03-23T22:38:47.755 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_vti.ko 2024-03-23T22:38:47.755 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipcomp.ko 2024-03-23T22:38:47.755 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipip.ko 2024-03-23T22:38:47.755 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter 2024-03-23T22:38:47.755 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-23T22:38:47.755 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-23T22:38:47.755 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-23T22:38:47.755 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-23T22:38:47.755 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-23T22:38:47.755 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-23T22:38:47.755 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-23T22:38:47.755 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-23T22:38:47.756 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-23T22:38:47.756 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-23T22:38:47.756 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-23T22:38:47.756 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-23T22:38:47.756 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-23T22:38:47.756 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-23T22:38:47.756 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-23T22:38:47.756 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-23T22:38:47.756 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-23T22:38:47.756 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-23T22:38:47.756 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-23T22:38:47.756 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-23T22:38:47.756 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-23T22:38:47.756 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_bic.ko 2024-03-23T22:38:47.756 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_dctcp.ko 2024-03-23T22:38:47.757 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_diag.ko 2024-03-23T22:38:47.757 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_highspeed.ko 2024-03-23T22:38:47.757 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_htcp.ko 2024-03-23T22:38:47.757 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_hybla.ko 2024-03-23T22:38:47.757 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_illinois.ko 2024-03-23T22:38:47.757 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_lp.ko 2024-03-23T22:38:47.757 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_scalable.ko 2024-03-23T22:38:47.757 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_vegas.ko 2024-03-23T22:38:47.757 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_veno.ko 2024-03-23T22:38:47.757 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_westwood.ko 2024-03-23T22:38:47.757 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_yeah.ko 2024-03-23T22:38:47.757 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tunnel4.ko 2024-03-23T22:38:47.757 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_diag.ko 2024-03-23T22:38:47.757 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_tunnel.ko 2024-03-23T22:38:47.757 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-23T22:38:47.758 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6 2024-03-23T22:38:47.758 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ah6.ko 2024-03-23T22:38:47.758 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/esp6.ko 2024-03-23T22:38:47.758 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/fou6.ko 2024-03-23T22:38:47.758 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_tunnel.ko 2024-03-23T22:38:47.758 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-23T22:38:47.758 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_vti.ko 2024-03-23T22:38:47.758 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ipcomp6.ko 2024-03-23T22:38:47.758 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter 2024-03-23T22:38:47.758 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-23T22:38:47.758 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-23T22:38:47.758 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-23T22:38:47.758 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-23T22:38:47.758 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-23T22:38:47.759 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-23T22:38:47.759 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-23T22:38:47.759 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-23T22:38:47.759 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-23T22:38:47.759 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-23T22:38:47.759 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-23T22:38:47.759 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-23T22:38:47.759 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-23T22:38:47.759 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-23T22:38:47.759 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-23T22:38:47.759 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-23T22:38:47.759 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-23T22:38:47.759 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-23T22:38:47.759 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-23T22:38:47.759 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-23T22:38:47.759 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-23T22:38:47.760 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-23T22:38:47.760 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-23T22:38:47.760 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-23T22:38:47.760 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/sit.ko 2024-03-23T22:38:47.760 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/tunnel6.ko 2024-03-23T22:38:47.760 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-23T22:38:47.760 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key 2024-03-23T22:38:47.760 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key/af_key.ko 2024-03-23T22:38:47.760 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp 2024-03-23T22:38:47.760 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_core.ko 2024-03-23T22:38:47.760 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-23T22:38:47.760 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_eth.ko 2024-03-23T22:38:47.760 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip.ko 2024-03-23T22:38:47.760 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip6.ko 2024-03-23T22:38:47.760 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_netlink.ko 2024-03-23T22:38:47.761 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ppp.ko 2024-03-23T22:38:47.761 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc 2024-03-23T22:38:47.761 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/llc.ko 2024-03-23T22:38:47.761 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154 2024-03-23T22:38:47.761 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154/mac802154.ko 2024-03-23T22:38:47.761 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls 2024-03-23T22:38:47.761 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls/mpls_gso.ko 2024-03-23T22:38:47.761 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter 2024-03-23T22:38:47.761 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset 2024-03-23T22:38:47.761 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set.ko 2024-03-23T22:38:47.761 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-23T22:38:47.761 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-23T22:38:47.761 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-23T22:38:47.761 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-23T22:38:47.761 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-23T22:38:47.761 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-23T22:38:47.762 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-23T22:38:47.762 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-23T22:38:47.762 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-23T22:38:47.762 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-23T22:38:47.762 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-23T22:38:47.762 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-23T22:38:47.762 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-23T22:38:47.762 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-23T22:38:47.762 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-23T22:38:47.762 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs 2024-03-23T22:38:47.762 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-23T22:38:47.762 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-23T22:38:47.762 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-23T22:38:47.762 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-23T22:38:47.763 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-23T22:38:47.763 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-23T22:38:47.763 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-23T22:38:47.763 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-23T22:38:47.763 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-23T22:38:47.763 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-23T22:38:47.763 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-23T22:38:47.763 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-23T22:38:47.763 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-23T22:38:47.763 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-23T22:38:47.763 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conncount.ko 2024-03-23T22:38:47.763 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack.ko 2024-03-23T22:38:47.763 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-23T22:38:47.763 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-23T22:38:47.763 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-23T22:38:47.764 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-23T22:38:47.764 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-23T22:38:47.764 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-23T22:38:47.764 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-23T22:38:47.764 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-23T22:38:47.764 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-23T22:38:47.764 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-23T22:38:47.764 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-23T22:38:47.764 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-23T22:38:47.764 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-23T22:38:47.764 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table.ko 2024-03-23T22:38:47.764 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-23T22:38:47.764 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_log_syslog.ko 2024-03-23T22:38:47.764 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat.ko 2024-03-23T22:38:47.764 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-23T22:38:47.765 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-23T22:38:47.765 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_irc.ko 2024-03-23T22:38:47.765 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_sip.ko 2024-03-23T22:38:47.765 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-23T22:38:47.765 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-23T22:38:47.765 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_tables.ko 2024-03-23T22:38:47.765 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink.ko 2024-03-23T22:38:47.765 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-23T22:38:47.765 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_log.ko 2024-03-23T22:38:47.765 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-23T22:38:47.765 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-23T22:38:47.765 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_chain_nat.ko 2024-03-23T22:38:47.765 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_compat.ko 2024-03-23T22:38:47.765 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_ct.ko 2024-03-23T22:38:47.765 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-23T22:38:47.765 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fib.ko 2024-03-23T22:38:47.766 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-23T22:38:47.766 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_hash.ko 2024-03-23T22:38:47.766 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_limit.ko 2024-03-23T22:38:47.766 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_log.ko 2024-03-23T22:38:47.766 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_masq.ko 2024-03-23T22:38:47.766 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_nat.ko 2024-03-23T22:38:47.766 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_numgen.ko 2024-03-23T22:38:47.766 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_queue.ko 2024-03-23T22:38:47.766 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_quota.ko 2024-03-23T22:38:47.766 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_redir.ko 2024-03-23T22:38:47.767 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject.ko 2024-03-23T22:38:47.767 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject_inet.ko 2024-03-23T22:38:47.767 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_socket.ko 2024-03-23T22:38:47.767 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_tunnel.ko 2024-03-23T22:38:47.767 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_AUDIT.ko 2024-03-23T22:38:47.767 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-23T22:38:47.767 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-23T22:38:47.767 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-23T22:38:47.767 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CT.ko 2024-03-23T22:38:47.767 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_DSCP.ko 2024-03-23T22:38:47.767 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HL.ko 2024-03-23T22:38:47.767 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HMARK.ko 2024-03-23T22:38:47.767 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-23T22:38:47.767 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LED.ko 2024-03-23T22:38:47.767 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LOG.ko 2024-03-23T22:38:47.768 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-23T22:38:47.768 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NETMAP.ko 2024-03-23T22:38:47.768 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFLOG.ko 2024-03-23T22:38:47.768 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-23T22:38:47.768 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_RATEEST.ko 2024-03-23T22:38:47.768 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-23T22:38:47.768 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_SECMARK.ko 2024-03-23T22:38:47.768 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-23T22:38:47.768 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-23T22:38:47.768 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TEE.ko 2024-03-23T22:38:47.768 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TPROXY.ko 2024-03-23T22:38:47.768 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TRACE.ko 2024-03-23T22:38:47.768 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_addrtype.ko 2024-03-23T22:38:47.768 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_bpf.ko 2024-03-23T22:38:47.769 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cgroup.ko 2024-03-23T22:38:47.769 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cluster.ko 2024-03-23T22:38:47.769 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_comment.ko 2024-03-23T22:38:47.769 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connbytes.ko 2024-03-23T22:38:47.769 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlabel.ko 2024-03-23T22:38:47.769 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlimit.ko 2024-03-23T22:38:47.769 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connmark.ko 2024-03-23T22:38:47.769 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_conntrack.ko 2024-03-23T22:38:47.769 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cpu.ko 2024-03-23T22:38:47.769 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dccp.ko 2024-03-23T22:38:47.770 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_devgroup.ko 2024-03-23T22:38:47.770 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dscp.ko 2024-03-23T22:38:47.770 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ecn.ko 2024-03-23T22:38:47.770 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_esp.ko 2024-03-23T22:38:47.770 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hashlimit.ko 2024-03-23T22:38:47.770 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_helper.ko 2024-03-23T22:38:47.770 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hl.ko 2024-03-23T22:38:47.771 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipcomp.ko 2024-03-23T22:38:47.771 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_iprange.ko 2024-03-23T22:38:47.771 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipvs.ko 2024-03-23T22:38:47.771 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_l2tp.ko 2024-03-23T22:38:47.771 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_length.ko 2024-03-23T22:38:47.771 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_limit.ko 2024-03-23T22:38:47.771 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mac.ko 2024-03-23T22:38:47.771 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mark.ko 2024-03-23T22:38:47.771 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_multiport.ko 2024-03-23T22:38:47.771 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nat.ko 2024-03-23T22:38:47.771 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nfacct.ko 2024-03-23T22:38:47.771 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_osf.ko 2024-03-23T22:38:47.771 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_owner.ko 2024-03-23T22:38:47.771 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_physdev.ko 2024-03-23T22:38:47.772 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_pkttype.ko 2024-03-23T22:38:47.772 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_policy.ko 2024-03-23T22:38:47.772 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_quota.ko 2024-03-23T22:38:47.772 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_rateest.ko 2024-03-23T22:38:47.772 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_realm.ko 2024-03-23T22:38:47.772 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_recent.ko 2024-03-23T22:38:47.772 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_sctp.ko 2024-03-23T22:38:47.772 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_set.ko 2024-03-23T22:38:47.772 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_socket.ko 2024-03-23T22:38:47.772 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_state.ko 2024-03-23T22:38:47.772 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_statistic.ko 2024-03-23T22:38:47.772 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_string.ko 2024-03-23T22:38:47.773 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_tcpmss.ko 2024-03-23T22:38:47.773 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_time.ko 2024-03-23T22:38:47.773 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_u32.ko 2024-03-23T22:38:47.773 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink 2024-03-23T22:38:47.773 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink/netlink_diag.ko 2024-03-23T22:38:47.773 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh 2024-03-23T22:38:47.773 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh/nsh.ko 2024-03-23T22:38:47.773 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch 2024-03-23T22:38:47.773 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/openvswitch.ko 2024-03-23T22:38:47.773 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-geneve.ko 2024-03-23T22:38:47.773 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-gre.ko 2024-03-23T22:38:47.773 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-vxlan.ko 2024-03-23T22:38:47.773 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet 2024-03-23T22:38:47.773 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet/af_packet_diag.ko 2024-03-23T22:38:47.773 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds 2024-03-23T22:38:47.774 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds.ko 2024-03-23T22:38:47.774 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_rdma.ko 2024-03-23T22:38:47.774 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_tcp.ko 2024-03-23T22:38:47.774 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched 2024-03-23T22:38:47.774 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_csum.ko 2024-03-23T22:38:47.774 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_gact.ko 2024-03-23T22:38:47.774 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_mirred.ko 2024-03-23T22:38:47.774 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_nat.ko 2024-03-23T22:38:47.774 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_pedit.ko 2024-03-23T22:38:47.774 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_police.ko 2024-03-23T22:38:47.774 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_simple.ko 2024-03-23T22:38:47.774 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_skbedit.ko 2024-03-23T22:38:47.774 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_vlan.ko 2024-03-23T22:38:47.774 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_basic.ko 2024-03-23T22:38:47.775 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_bpf.ko 2024-03-23T22:38:47.775 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_flow.ko 2024-03-23T22:38:47.775 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_fw.ko 2024-03-23T22:38:47.775 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_route.ko 2024-03-23T22:38:47.775 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_u32.ko 2024-03-23T22:38:47.775 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_cmp.ko 2024-03-23T22:38:47.775 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_ipset.ko 2024-03-23T22:38:47.775 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_meta.ko 2024-03-23T22:38:47.775 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_nbyte.ko 2024-03-23T22:38:47.775 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_text.ko 2024-03-23T22:38:47.775 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_u32.ko 2024-03-23T22:38:47.775 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_choke.ko 2024-03-23T22:38:47.775 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_codel.ko 2024-03-23T22:38:47.775 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_drr.ko 2024-03-23T22:38:47.776 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_fq.ko 2024-03-23T22:38:47.776 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_gred.ko 2024-03-23T22:38:47.776 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hfsc.ko 2024-03-23T22:38:47.776 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hhf.ko 2024-03-23T22:38:47.776 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_htb.ko 2024-03-23T22:38:47.776 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_ingress.ko 2024-03-23T22:38:47.776 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio.ko 2024-03-23T22:38:47.776 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio_lib.ko 2024-03-23T22:38:47.776 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_multiq.ko 2024-03-23T22:38:47.776 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_netem.ko 2024-03-23T22:38:47.776 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_pie.ko 2024-03-23T22:38:47.776 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_plug.ko 2024-03-23T22:38:47.776 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_prio.ko 2024-03-23T22:38:47.776 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_qfq.ko 2024-03-23T22:38:47.776 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_red.ko 2024-03-23T22:38:47.777 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfb.ko 2024-03-23T22:38:47.777 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfq.ko 2024-03-23T22:38:47.777 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_tbf.ko 2024-03-23T22:38:47.777 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_teql.ko 2024-03-23T22:38:47.777 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp 2024-03-23T22:38:47.777 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp.ko 2024-03-23T22:38:47.777 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp_diag.ko 2024-03-23T22:38:47.777 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc 2024-03-23T22:38:47.777 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss 2024-03-23T22:38:47.777 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-23T22:38:47.777 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-23T22:38:47.777 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/sunrpc.ko 2024-03-23T22:38:47.777 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma 2024-03-23T22:38:47.777 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-23T22:38:47.777 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc 2024-03-23T22:38:47.778 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/diag.ko 2024-03-23T22:38:47.778 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/tipc.ko 2024-03-23T22:38:47.778 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix 2024-03-23T22:38:47.778 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix/unix_diag.ko 2024-03-23T22:38:47.778 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock 2024-03-23T22:38:47.778 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-23T22:38:47.778 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-23T22:38:47.778 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock.ko 2024-03-23T22:38:47.778 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-23T22:38:47.778 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-23T22:38:47.778 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm 2024-03-23T22:38:47.778 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-23T22:38:47.778 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security 2024-03-23T22:38:47.778 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys 2024-03-23T22:38:47.779 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys 2024-03-23T22:38:47.779 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-23T22:38:47.779 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys 2024-03-23T22:38:47.779 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys/trusted.ko 2024-03-23T22:38:47.779 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt 2024-03-23T22:38:47.779 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib 2024-03-23T22:38:47.779 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib/irqbypass.ko 2024-03-23T22:38:47.779 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias 2024-03-23T22:38:47.779 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias.bin 2024-03-23T22:38:47.779 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin 2024-03-23T22:38:47.779 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.alias.bin 2024-03-23T22:38:47.779 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.bin 2024-03-23T22:38:47.780 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.modinfo 2024-03-23T22:38:47.780 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep 2024-03-23T22:38:47.780 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep.bin 2024-03-23T22:38:47.780 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.devname 2024-03-23T22:38:47.780 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.order 2024-03-23T22:38:47.780 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.softdep 2024-03-23T22:38:47.780 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols 2024-03-23T22:38:47.780 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols.bin 2024-03-23T22:38:47.780 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-23T22:38:47.784 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g5ec3e6e88b2e 2024-03-23T22:38:47.784 DEBUG:teuthology.orchestra.run.smithi012:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-23T22:38:47.807 INFO:teuthology.orchestra.run.smithi012.stdout:postinstall scriptlet (using /bin/sh): 2024-03-23T22:38:47.807 INFO:teuthology.orchestra.run.smithi012.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-23T22:38:47.807 INFO:teuthology.orchestra.run.smithi012.stdout: /usr/bin/kernel-install add 6.8.0-g5ec3e6e88b2e /lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-23T22:38:47.807 INFO:teuthology.orchestra.run.smithi012.stdout:fi 2024-03-23T22:38:47.808 INFO:teuthology.orchestra.run.smithi012.stdout:for file in vmlinuz System.map config; do 2024-03-23T22:38:47.808 INFO:teuthology.orchestra.run.smithi012.stdout: if ! cmp --silent "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e"; then 2024-03-23T22:38:47.808 INFO:teuthology.orchestra.run.smithi012.stdout: cp "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e" 2024-03-23T22:38:47.808 INFO:teuthology.orchestra.run.smithi012.stdout: fi 2024-03-23T22:38:47.808 INFO:teuthology.orchestra.run.smithi012.stdout:done 2024-03-23T22:38:47.808 INFO:teuthology.orchestra.run.smithi012.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-23T22:38:47.808 INFO:teuthology.orchestra.run.smithi012.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-23T22:38:47.808 INFO:teuthology.orchestra.run.smithi012.stdout:new-kernel-pkg --remove 6.8.0-g5ec3e6e88b2e --rminitrd --initrdfile=/boot/initramfs-6.8.0-g5ec3e6e88b2e.img 2024-03-23T22:38:47.808 INFO:teuthology.orchestra.run.smithi012.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-23T22:38:47.808 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-install remove 6.8.0-g5ec3e6e88b2e 2024-03-23T22:38:47.808 INFO:teuthology.orchestra.run.smithi012.stdout:fi 2024-03-23T22:38:47.808 INFO:teuthology.orchestra.run.smithi012.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-23T22:38:47.808 INFO:teuthology.orchestra.run.smithi012.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-23T22:38:47.808 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/update-bootloader --remove 6.8.0-g5ec3e6e88b2e 2024-03-23T22:38:47.808 INFO:teuthology.orchestra.run.smithi012.stdout:fi 2024-03-23T22:38:47.809 DEBUG:teuthology.orchestra.run.smithi012:> sudo rpm -qi grub2-tools 2024-03-23T22:38:47.890 INFO:teuthology.orchestra.run.smithi012.stdout:Name : grub2-tools 2024-03-23T22:38:47.890 INFO:teuthology.orchestra.run.smithi012.stdout:Epoch : 1 2024-03-23T22:38:47.890 INFO:teuthology.orchestra.run.smithi012.stdout:Version : 2.06 2024-03-23T22:38:47.890 INFO:teuthology.orchestra.run.smithi012.stdout:Release : 68.el9 2024-03-23T22:38:47.890 INFO:teuthology.orchestra.run.smithi012.stdout:Architecture: x86_64 2024-03-23T22:38:47.891 INFO:teuthology.orchestra.run.smithi012.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-23T22:38:47.891 INFO:teuthology.orchestra.run.smithi012.stdout:Group : Unspecified 2024-03-23T22:38:47.891 INFO:teuthology.orchestra.run.smithi012.stdout:Size : 8264254 2024-03-23T22:38:47.891 INFO:teuthology.orchestra.run.smithi012.stdout:License : GPLv3+ 2024-03-23T22:38:47.891 INFO:teuthology.orchestra.run.smithi012.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-23T22:38:47.891 INFO:teuthology.orchestra.run.smithi012.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-23T22:38:47.891 INFO:teuthology.orchestra.run.smithi012.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-23T22:38:47.891 INFO:teuthology.orchestra.run.smithi012.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-23T22:38:47.891 INFO:teuthology.orchestra.run.smithi012.stdout:Packager : builder@centos.org 2024-03-23T22:38:47.891 INFO:teuthology.orchestra.run.smithi012.stdout:Vendor : CentOS 2024-03-23T22:38:47.891 INFO:teuthology.orchestra.run.smithi012.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-23T22:38:47.891 INFO:teuthology.orchestra.run.smithi012.stdout:Summary : Support tools for GRUB. 2024-03-23T22:38:47.891 INFO:teuthology.orchestra.run.smithi012.stdout:Description : 2024-03-23T22:38:47.891 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:38:47.891 INFO:teuthology.orchestra.run.smithi012.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-23T22:38:47.892 INFO:teuthology.orchestra.run.smithi012.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-23T22:38:47.892 INFO:teuthology.orchestra.run.smithi012.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-23T22:38:47.892 INFO:teuthology.orchestra.run.smithi012.stdout:hardware devices. 2024-03-23T22:38:47.892 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:38:47.892 INFO:teuthology.orchestra.run.smithi012.stdout:This subpackage provides tools for support of all platforms. 2024-03-23T22:38:47.893 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-23T22:38:47.893 INFO:teuthology.task.kernel:Updating grub on smithi012 to boot 6.8.0-g5ec3e6e88b2e 2024-03-23T22:38:47.893 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-23T22:38:48.526 INFO:teuthology.orchestra.run.smithi012.stderr:Generating grub configuration file ... 2024-03-23T22:38:49.856 INFO:teuthology.orchestra.run.smithi012.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-23T22:38:49.883 INFO:teuthology.orchestra.run.smithi012.stderr:done 2024-03-23T22:38:49.885 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2024-03-23T22:38:49.900 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.jVgWDtiqbw 2024-03-23T22:38:49.901 DEBUG:teuthology.orchestra.run.smithi012:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.jVgWDtiqbw 2024-03-23T22:38:49.973 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.jVgWDtiqbw 2024-03-23T22:38:50.124 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.jVgWDtiqbw is 6KB 2024-03-23T22:38:50.173 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.jVgWDtiqbw 2024-03-23T22:38:50.187 DEBUG:teuthology.orchestra.run.smithi012:> sudo /bin/ls /boot/loader/entries || true 2024-03-23T22:38:50.255 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-23T22:38:50.256 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-23T22:38:50.256 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-23T22:38:50.256 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-6.8.0-g5ec3e6e88b2e.conf 2024-03-23T22:38:50.257 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-g5ec3e6e88b2e 2024-03-23T22:38:50.425 DEBUG:teuthology.orchestra.run.smithi012:> sudo shutdown -r now 2024-03-23T22:39:20.428 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-23T22:39:20.429 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-23T22:39:20.430 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T22:39:38.786 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2024-03-23T22:39:47.787 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-23T22:39:47.788 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T22:40:06.174 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2024-03-23T22:40:18.175 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-23T22:40:18.177 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T22:40:21.246 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2024-03-23T22:40:36.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-23T22:40:36.248 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T22:40:36.639 DEBUG:teuthology.orchestra.run.smithi012:> true 2024-03-23T22:40:37.199 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-23T22:40:37.199 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-g5ec3e6e88b2e"... 2024-03-23T22:40:37.200 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2024-03-23T22:40:37.235 INFO:teuthology.orchestra.run.smithi012.stdout:6.8.0-g5ec3e6e88b2e 2024-03-23T22:40:37.236 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g5ec3e6e88b2e vs 6.8.0-g5ec3e6e88b2e 2024-03-23T22:40:37.236 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-23T22:40:37.236 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-23T22:40:38.237 DEBUG:teuthology.parallel:result is None 2024-03-23T22:40:38.237 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-23T22:40:38.245 INFO:teuthology.task.internal:Creating test directory... 2024-03-23T22:40:38.246 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-23T22:40:38.253 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-23T22:40:38.256 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-23T22:40:38.300 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-23T22:40:38.315 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-23T22:40:38.342 INFO:teuthology.task.internal:Creating archive directory... 2024-03-23T22:40:38.342 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-23T22:40:38.349 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-23T22:40:38.351 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-23T22:40:38.469 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-23T22:40:38.480 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-23T22:40:38.480 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-23T22:40:38.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T22:40:38.496 DEBUG:teuthology.orchestra.run.smithi203:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-23T22:40:38.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T22:40:38.513 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-23T22:40:38.520 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-23T22:40:38.539 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-23T22:40:38.596 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T22:40:38.616 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T22:40:38.621 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T22:40:38.642 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T22:40:38.664 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T22:40:38.736 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T22:40:38.741 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-23T22:40:38.750 INFO:teuthology.task.internal:Configuring sudo... 2024-03-23T22:40:38.750 DEBUG:teuthology.orchestra.run.smithi012:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-23T22:40:38.754 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-23T22:40:38.757 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-23T22:40:38.824 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-23T22:40:38.837 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-23T22:40:38.837 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-23T22:40:38.842 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-23T22:40:38.844 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-23T22:40:38.881 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T22:40:38.998 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T22:40:39.094 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T22:40:39.171 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T22:40:39.254 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-23T22:40:39.254 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-23T22:40:39.321 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T22:40:39.362 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T22:40:39.419 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T22:40:39.503 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T22:40:39.547 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T22:40:39.547 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-23T22:40:39.615 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T22:40:39.657 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T22:40:39.711 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T22:40:39.795 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T22:40:39.839 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-23T22:40:39.839 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-23T22:40:39.908 DEBUG:teuthology.orchestra.run.smithi012:> sudo service rsyslog restart 2024-03-23T22:40:39.919 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2024-03-23T22:40:39.921 DEBUG:teuthology.orchestra.run.smithi203:> sudo service rsyslog restart 2024-03-23T22:40:39.968 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T22:40:39.992 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T22:40:40.083 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T22:40:40.614 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-23T22:40:40.632 INFO:teuthology.task.internal:Starting timer... 2024-03-23T22:40:40.632 INFO:teuthology.run_tasks:Running task pcp... 2024-03-23T22:40:40.658 INFO:teuthology.run_tasks:Running task selinux... 2024-03-23T22:40:40.691 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-23T22:40:40.691 DEBUG:teuthology.orchestra.run.smithi012:> sudo service auditd rotate 2024-03-23T22:40:40.903 INFO:teuthology.orchestra.run.smithi012.stdout:Rotating logs: 2024-03-23T22:40:40.908 DEBUG:teuthology.orchestra.run.smithi173:> sudo service auditd rotate 2024-03-23T22:40:40.970 INFO:teuthology.orchestra.run.smithi173.stdout:Rotating logs: 2024-03-23T22:40:40.972 DEBUG:teuthology.orchestra.run.smithi203:> sudo service auditd rotate 2024-03-23T22:40:41.042 INFO:teuthology.orchestra.run.smithi203.stdout:Rotating logs: 2024-03-23T22:40:41.044 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-23T22:40:41.045 DEBUG:teuthology.orchestra.run.smithi012:> /usr/sbin/getenforce 2024-03-23T22:40:41.132 INFO:teuthology.orchestra.run.smithi012.stdout:Permissive 2024-03-23T22:40:41.133 DEBUG:teuthology.orchestra.run.smithi173:> /usr/sbin/getenforce 2024-03-23T22:40:41.152 INFO:teuthology.orchestra.run.smithi173.stdout:Permissive 2024-03-23T22:40:41.153 DEBUG:teuthology.orchestra.run.smithi203:> /usr/sbin/getenforce 2024-03-23T22:40:41.175 INFO:teuthology.orchestra.run.smithi203.stdout:Permissive 2024-03-23T22:40:41.175 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi012.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi173.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi203.front.sepia.ceph.com': 'permissive'} 2024-03-23T22:40:41.175 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-23T22:40:41.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T22:40:41.321 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-23T22:40:41.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T22:40:41.349 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-23T22:40:41.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T22:40:41.377 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-23T22:40:41.378 DEBUG:teuthology.orchestra.run.smithi012:> sudo /usr/sbin/setenforce permissive 2024-03-23T22:40:41.524 DEBUG:teuthology.orchestra.run.smithi173:> sudo /usr/sbin/setenforce permissive 2024-03-23T22:40:41.554 DEBUG:teuthology.orchestra.run.smithi203:> sudo /usr/sbin/setenforce permissive 2024-03-23T22:40:41.592 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-23T22:40:41.603 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-23T22:40:41.609 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-23T22:40:41.796 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-23T22:40:41.812 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-23T22:40:41.813 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi012.front.sepia.ceph.com,smithi173.front.sepia.ceph.com,smithi203.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-23T22:46:46.252 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi012.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com'), Remote(name='ubuntu@smithi203.front.sepia.ceph.com')] 2024-03-23T22:46:46.253 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-23T22:46:46.255 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T22:46:46.402 DEBUG:teuthology.orchestra.run.smithi012:> true 2024-03-23T22:46:46.555 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-23T22:46:46.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-23T22:46:46.556 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T22:46:46.642 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-03-23T22:46:46.709 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-23T22:46:46.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-23T22:46:46.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T22:46:46.798 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-03-23T22:46:46.866 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-23T22:46:46.866 INFO:teuthology.run_tasks:Running task clock... 2024-03-23T22:46:46.877 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-23T22:46:46.877 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-23T22:46:46.877 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T22:46:46.885 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-23T22:46:46.885 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T22:46:46.887 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-23T22:46:46.887 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T22:46:46.915 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-23T22:46:46.932 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-23T22:46:46.939 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-23T22:46:46.956 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-23T22:46:46.962 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: ntpd: command not found 2024-03-23T22:46:46.976 INFO:teuthology.orchestra.run.smithi173.stdout:506 Cannot talk to daemon 2024-03-23T22:46:46.985 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-23T22:46:46.986 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: ntpd: command not found 2024-03-23T22:46:46.993 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-23T22:46:47.000 INFO:teuthology.orchestra.run.smithi203.stdout:506 Cannot talk to daemon 2024-03-23T22:46:47.011 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-23T22:46:47.016 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-23T22:46:47.033 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-23T22:46:47.041 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-23T22:46:47.056 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-03-23T22:46:47.060 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T22:46:47.060 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-03-23T22:46:47.060 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T22:46:47.060 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T22:46:47.060 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T22:46:47.060 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T22:46:47.077 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-03-23T22:46:47.080 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T22:46:47.080 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-03-23T22:46:47.081 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T22:46:47.081 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T22:46:47.081 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T22:46:47.081 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T22:46:47.145 INFO:teuthology.orchestra.run.smithi012.stderr:sudo: ntpd: command not found 2024-03-23T22:46:47.190 INFO:teuthology.orchestra.run.smithi012.stdout:506 Cannot talk to daemon 2024-03-23T22:46:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-23T22:46:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-23T22:46:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:bash: line 1: ntpq: command not found 2024-03-23T22:46:47.450 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T22:46:47.450 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2024-03-23T22:46:47.450 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T22:46:47.450 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T22:46:47.451 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T22:46:47.451 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T22:46:47.452 INFO:teuthology.run_tasks:Running task install... 2024-03-23T22:46:47.460 DEBUG:teuthology.task.install:project ceph 2024-03-23T22:46:47.460 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}} 2024-03-23T22:46:47.460 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'} 2024-03-23T22:46:47.461 INFO:teuthology.task.install:Using flavor: default 2024-03-23T22:46:47.466 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-23T22:46:47.466 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-23T22:46:47.466 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-23T22:46:47.467 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-23T22:46:47.469 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-23T22:46:47.469 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-23T22:46:47.470 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-23T22:46:47.470 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-23T22:46:47.611 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-23T22:46:47.611 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-23T22:46:47.615 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-23T22:46:47.615 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-23T22:46:47.623 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-23T22:46:47.623 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-23T22:46:47.725 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-23T22:46:47.725 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-23T22:46:47.725 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-23T22:46:47.755 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-23T22:46:47.755 DEBUG:teuthology.orchestra.run.smithi203:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-23T22:46:47.780 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-23T22:46:47.780 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-23T22:46:47.780 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-23T22:46:47.792 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-23T22:46:47.792 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T22:46:47.792 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-23T22:46:47.820 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-23T22:46:47.820 DEBUG:teuthology.orchestra.run.smithi173:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-23T22:46:47.824 DEBUG:teuthology.orchestra.run.smithi203:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-23T22:46:47.877 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-23T22:46:47.877 DEBUG:teuthology.orchestra.run.smithi012:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-23T22:46:47.888 DEBUG:teuthology.orchestra.run.smithi173:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-23T22:46:47.905 DEBUG:teuthology.orchestra.run.smithi203:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-23T22:46:47.968 DEBUG:teuthology.orchestra.run.smithi173:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-23T22:46:47.972 INFO:teuthology.orchestra.run.smithi203.stdout:check_obsoletes = 1 2024-03-23T22:46:47.974 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum clean all 2024-03-23T22:46:48.024 DEBUG:teuthology.orchestra.run.smithi012:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-23T22:46:48.035 INFO:teuthology.orchestra.run.smithi173.stdout:check_obsoletes = 1 2024-03-23T22:46:48.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum clean all 2024-03-23T22:46:48.165 DEBUG:teuthology.orchestra.run.smithi012:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-23T22:46:48.210 INFO:teuthology.orchestra.run.smithi203.stdout:74 files removed 2024-03-23T22:46:48.237 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-23T22:46:48.251 INFO:teuthology.orchestra.run.smithi012.stdout:check_obsoletes = 1 2024-03-23T22:46:48.256 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum clean all 2024-03-23T22:46:48.271 INFO:teuthology.orchestra.run.smithi173.stdout:74 files removed 2024-03-23T22:46:48.298 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-23T22:46:48.632 INFO:teuthology.orchestra.run.smithi012.stdout:74 files removed 2024-03-23T22:46:48.673 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-23T22:46:48.928 INFO:teuthology.orchestra.run.smithi203.stdout:ceph packages for x86_64 208 kB/s | 83 kB 00:00 2024-03-23T22:46:48.983 INFO:teuthology.orchestra.run.smithi173.stdout:ceph packages for x86_64 210 kB/s | 83 kB 00:00 2024-03-23T22:46:49.287 INFO:teuthology.orchestra.run.smithi203.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-23T22:46:49.359 INFO:teuthology.orchestra.run.smithi173.stdout:ceph noarch packages 30 kB/s | 10 kB 00:00 2024-03-23T22:46:49.629 INFO:teuthology.orchestra.run.smithi203.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-03-23T22:46:49.693 INFO:teuthology.orchestra.run.smithi173.stdout:ceph source packages 6.2 kB/s | 1.9 kB 00:00 2024-03-23T22:46:49.737 INFO:teuthology.orchestra.run.smithi012.stdout:ceph packages for x86_64 152 kB/s | 83 kB 00:00 2024-03-23T22:46:50.265 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-23T22:46:50.304 INFO:teuthology.orchestra.run.smithi012.stdout:ceph noarch packages 21 kB/s | 10 kB 00:00 2024-03-23T22:46:50.604 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - AppStream 7.6 MB/s | 6.7 MB 00:00 2024-03-23T22:46:50.821 INFO:teuthology.orchestra.run.smithi012.stdout:ceph source packages 4.3 kB/s | 1.9 kB 00:00 2024-03-23T22:46:51.450 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-23T22:46:51.731 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2024-03-23T22:46:52.037 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - BaseOS 6.0 MB/s | 2.0 MB 00:00 2024-03-23T22:46:52.597 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-23T22:46:53.109 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2024-03-23T22:46:54.674 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-03-23T22:46:59.490 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 6.1 MB 00:01 2024-03-23T22:47:01.175 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-03-23T22:47:01.884 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 35 MB/s | 21 MB 00:00 2024-03-23T22:47:02.311 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - BaseOS 936 kB/s | 8.0 MB 00:08 2024-03-23T22:47:04.289 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-03-23T22:47:06.035 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 601 kB/s | 8.0 MB 00:13 2024-03-23T22:47:06.876 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-03-23T22:47:08.303 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-03-23T22:47:09.173 INFO:teuthology.orchestra.run.smithi203.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-23T22:47:09.174 INFO:teuthology.orchestra.run.smithi203.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-23T22:47:09.188 INFO:teuthology.orchestra.run.smithi203.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-23T22:47:09.189 INFO:teuthology.orchestra.run.smithi203.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-23T22:47:09.190 INFO:teuthology.orchestra.run.smithi203.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-23T22:47:09.192 INFO:teuthology.orchestra.run.smithi203.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-23T22:47:09.194 INFO:teuthology.orchestra.run.smithi203.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-23T22:47:09.196 INFO:teuthology.orchestra.run.smithi203.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-23T22:47:09.197 INFO:teuthology.orchestra.run.smithi203.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-23T22:47:09.277 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-23T22:47:09.288 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-03-23T22:47:09.289 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-23T22:47:09.289 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-03-23T22:47:09.289 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-23T22:47:09.289 INFO:teuthology.orchestra.run.smithi203.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-23T22:47:09.289 INFO:teuthology.orchestra.run.smithi203.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-23T22:47:09.289 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-23T22:47:09.289 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-23T22:47:09.289 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-23T22:47:09.289 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-23T22:47:09.289 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-23T22:47:09.289 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-23T22:47:09.289 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-23T22:47:09.290 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-23T22:47:09.290 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-23T22:47:09.290 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-23T22:47:09.290 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-23T22:47:09.290 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-23T22:47:09.290 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-23T22:47:09.290 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-23T22:47:09.290 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-23T22:47:09.290 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-23T22:47:09.290 INFO:teuthology.orchestra.run.smithi203.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-23T22:47:09.290 INFO:teuthology.orchestra.run.smithi203.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-23T22:47:09.290 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-23T22:47:09.290 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-23T22:47:09.291 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-23T22:47:09.291 INFO:teuthology.orchestra.run.smithi203.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-23T22:47:09.291 INFO:teuthology.orchestra.run.smithi203.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-23T22:47:09.291 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-23T22:47:09.291 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-23T22:47:09.291 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-23T22:47:09.291 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-23T22:47:09.291 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-23T22:47:09.291 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-23T22:47:09.291 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-23T22:47:09.291 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-23T22:47:09.291 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-23T22:47:09.291 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-23T22:47:09.292 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-23T22:47:09.292 INFO:teuthology.orchestra.run.smithi203.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-23T22:47:09.292 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrading: 2024-03-23T22:47:09.292 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-23T22:47:09.292 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-23T22:47:09.292 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-23T22:47:09.292 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-23T22:47:09.292 INFO:teuthology.orchestra.run.smithi203.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-23T22:47:09.292 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-23T22:47:09.292 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-23T22:47:09.292 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-23T22:47:09.292 INFO:teuthology.orchestra.run.smithi203.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-23T22:47:09.292 INFO:teuthology.orchestra.run.smithi203.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-23T22:47:09.293 INFO:teuthology.orchestra.run.smithi203.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-23T22:47:09.293 INFO:teuthology.orchestra.run.smithi203.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-23T22:47:09.293 INFO:teuthology.orchestra.run.smithi203.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-23T22:47:09.293 INFO:teuthology.orchestra.run.smithi203.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-23T22:47:09.293 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-23T22:47:09.293 INFO:teuthology.orchestra.run.smithi203.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-23T22:47:09.293 INFO:teuthology.orchestra.run.smithi203.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-23T22:47:09.293 INFO:teuthology.orchestra.run.smithi203.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-23T22:47:09.293 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-23T22:47:09.293 INFO:teuthology.orchestra.run.smithi203.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-23T22:47:09.293 INFO:teuthology.orchestra.run.smithi203.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-23T22:47:09.293 INFO:teuthology.orchestra.run.smithi203.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-23T22:47:09.293 INFO:teuthology.orchestra.run.smithi203.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-23T22:47:09.293 INFO:teuthology.orchestra.run.smithi203.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-23T22:47:09.294 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-23T22:47:09.294 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-23T22:47:09.294 INFO:teuthology.orchestra.run.smithi203.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-23T22:47:09.294 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-23T22:47:09.294 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-23T22:47:09.294 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-23T22:47:09.294 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-23T22:47:09.294 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-23T22:47:09.294 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-23T22:47:09.294 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-23T22:47:09.294 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-23T22:47:09.294 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-23T22:47:09.294 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-23T22:47:09.294 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-23T22:47:09.295 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-23T22:47:09.295 INFO:teuthology.orchestra.run.smithi203.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-23T22:47:09.295 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-23T22:47:09.295 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-23T22:47:09.295 INFO:teuthology.orchestra.run.smithi203.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-23T22:47:09.295 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-23T22:47:09.295 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-23T22:47:09.295 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-23T22:47:09.295 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-23T22:47:09.295 INFO:teuthology.orchestra.run.smithi203.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-23T22:47:09.295 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-23T22:47:09.295 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-23T22:47:09.295 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-23T22:47:09.295 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-23T22:47:09.296 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-23T22:47:09.296 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-23T22:47:09.296 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-23T22:47:09.296 INFO:teuthology.orchestra.run.smithi203.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-23T22:47:09.296 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-23T22:47:09.296 INFO:teuthology.orchestra.run.smithi203.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-23T22:47:09.296 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-23T22:47:09.296 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-23T22:47:09.296 INFO:teuthology.orchestra.run.smithi203.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-23T22:47:09.296 INFO:teuthology.orchestra.run.smithi203.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-23T22:47:09.296 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-23T22:47:09.296 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-23T22:47:09.296 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-23T22:47:09.296 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-23T22:47:09.296 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-23T22:47:09.297 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-23T22:47:09.297 INFO:teuthology.orchestra.run.smithi203.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-23T22:47:09.297 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-23T22:47:09.297 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-23T22:47:09.297 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-23T22:47:09.297 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-23T22:47:09.297 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-23T22:47:09.297 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-23T22:47:09.297 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-23T22:47:09.297 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-23T22:47:09.297 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-23T22:47:09.297 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-23T22:47:09.297 INFO:teuthology.orchestra.run.smithi203.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-23T22:47:09.297 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-23T22:47:09.297 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-23T22:47:09.297 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-23T22:47:09.298 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-23T22:47:09.298 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-23T22:47:09.298 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-23T22:47:09.298 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-23T22:47:09.298 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-23T22:47:09.298 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-23T22:47:09.298 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-23T22:47:09.298 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-23T22:47:09.298 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-23T22:47:09.298 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-23T22:47:09.298 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-23T22:47:09.298 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-23T22:47:09.298 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-23T22:47:09.298 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-23T22:47:09.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-23T22:47:09.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-23T22:47:09.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-23T22:47:09.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-23T22:47:09.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-23T22:47:09.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-23T22:47:09.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-23T22:47:09.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-23T22:47:09.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-23T22:47:09.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-23T22:47:09.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-23T22:47:09.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-23T22:47:09.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-23T22:47:09.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-23T22:47:09.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-23T22:47:09.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-23T22:47:09.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-23T22:47:09.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-23T22:47:09.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-23T22:47:09.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-23T22:47:09.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-23T22:47:09.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-23T22:47:09.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-23T22:47:09.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-23T22:47:09.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-23T22:47:09.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-23T22:47:09.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-23T22:47:09.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-23T22:47:09.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-23T22:47:09.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-23T22:47:09.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-23T22:47:09.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-23T22:47:09.301 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-23T22:47:09.301 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-23T22:47:09.301 INFO:teuthology.orchestra.run.smithi203.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-23T22:47:09.301 INFO:teuthology.orchestra.run.smithi203.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-23T22:47:09.301 INFO:teuthology.orchestra.run.smithi203.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-23T22:47:09.301 INFO:teuthology.orchestra.run.smithi203.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-23T22:47:09.301 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-23T22:47:09.301 INFO:teuthology.orchestra.run.smithi203.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-23T22:47:09.301 INFO:teuthology.orchestra.run.smithi203.stdout:Installing weak dependencies: 2024-03-23T22:47:09.301 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-23T22:47:09.301 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-23T22:47:09.301 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-23T22:47:09.301 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-03-23T22:47:09.301 INFO:teuthology.orchestra.run.smithi203.stdout:Install 150 Packages 2024-03-23T22:47:09.301 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrade 24 Packages 2024-03-23T22:47:09.302 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-23T22:47:09.302 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 224 M 2024-03-23T22:47:09.302 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-23T22:47:09.554 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.1 MB 00:01 2024-03-23T22:47:09.977 INFO:teuthology.orchestra.run.smithi203.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 40 kB/s | 6.6 kB 00:00 2024-03-23T22:47:10.161 INFO:teuthology.orchestra.run.smithi203.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 4.7 MB/s | 890 kB 00:00 2024-03-23T22:47:10.219 INFO:teuthology.orchestra.run.smithi203.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 2.4 MB/s | 145 kB 00:00 2024-03-23T22:47:10.494 INFO:teuthology.orchestra.run.smithi203.stdout:(4/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 8.4 MB/s | 2.3 MB 00:00 2024-03-23T22:47:10.736 INFO:teuthology.orchestra.run.smithi203.stdout:(5/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 5.8 MB/s | 5.4 MB 00:00 2024-03-23T22:47:10.770 INFO:teuthology.orchestra.run.smithi203.stdout:(6/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 5.4 MB/s | 1.5 MB 00:00 2024-03-23T22:47:11.204 INFO:teuthology.orchestra.run.smithi203.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 10 MB/s | 4.7 MB 00:00 2024-03-23T22:47:11.649 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - Extras packages 21 kB/s | 16 kB 00:00 2024-03-23T22:47:11.979 INFO:teuthology.orchestra.run.smithi203.stdout:(8/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el9 13 MB/s | 9.9 MB 00:00 2024-03-23T22:47:12.054 INFO:teuthology.orchestra.run.smithi203.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 335 kB/s | 25 kB 00:00 2024-03-23T22:47:12.162 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-23T22:47:12.879 INFO:teuthology.orchestra.run.smithi203.stdout:(10/174): ceph-common-19.0.0-2145.g3be9a5c9.el9 6.7 MB/s | 20 MB 00:03 2024-03-23T22:47:13.071 INFO:teuthology.orchestra.run.smithi203.stdout:(11/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 7.3 MB/s | 17 MB 00:02 2024-03-23T22:47:13.096 INFO:teuthology.orchestra.run.smithi203.stdout:(12/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 1.0 MB/s | 220 kB 00:00 2024-03-23T22:47:13.122 INFO:teuthology.orchestra.run.smithi203.stdout:(13/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 541 kB/s | 27 kB 00:00 2024-03-23T22:47:13.155 INFO:teuthology.orchestra.run.smithi203.stdout:(14/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 534 kB/s | 31 kB 00:00 2024-03-23T22:47:13.247 INFO:teuthology.orchestra.run.smithi203.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.7 MB/s | 163 kB 00:00 2024-03-23T22:47:13.273 INFO:teuthology.orchestra.run.smithi203.stdout:(16/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 4.8 MB/s | 746 kB 00:00 2024-03-23T22:47:13.331 INFO:teuthology.orchestra.run.smithi203.stdout:(17/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.5 MB/s | 127 kB 00:00 2024-03-23T22:47:13.390 INFO:teuthology.orchestra.run.smithi203.stdout:(18/174): libradosstriper1-19.0.0-2145.g3be9a5c 4.2 MB/s | 501 kB 00:00 2024-03-23T22:47:13.456 INFO:teuthology.orchestra.run.smithi203.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 680 kB/s | 45 kB 00:00 2024-03-23T22:47:13.532 INFO:teuthology.orchestra.run.smithi203.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.7 MB/s | 131 kB 00:00 2024-03-23T22:47:13.599 INFO:teuthology.orchestra.run.smithi203.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 2.4 MB/s | 162 kB 00:00 2024-03-23T22:47:13.671 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-03-23T22:47:13.682 INFO:teuthology.orchestra.run.smithi203.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 3.8 MB/s | 321 kB 00:00 2024-03-23T22:47:13.723 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-23T22:47:13.758 INFO:teuthology.orchestra.run.smithi203.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 3.9 MB/s | 298 kB 00:00 2024-03-23T22:47:13.833 INFO:teuthology.orchestra.run.smithi203.stdout:(24/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.3 MB/s | 99 kB 00:00 2024-03-23T22:47:13.908 INFO:teuthology.orchestra.run.smithi203.stdout:(25/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-23T22:47:14.400 INFO:teuthology.orchestra.run.smithi203.stdout:(26/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 4.8 MB/s | 5.1 MB 00:01 2024-03-23T22:47:14.459 INFO:teuthology.orchestra.run.smithi203.stdout:(27/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 5.6 MB/s | 3.1 MB 00:00 2024-03-23T22:47:14.484 INFO:teuthology.orchestra.run.smithi203.stdout:(28/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 2.0 MB/s | 170 kB 00:00 2024-03-23T22:47:14.501 INFO:teuthology.orchestra.run.smithi203.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2145.g 809 kB/s | 34 kB 00:00 2024-03-23T22:47:14.551 INFO:teuthology.orchestra.run.smithi203.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 2.0 MB/s | 139 kB 00:00 2024-03-23T22:47:14.818 INFO:teuthology.orchestra.run.smithi203.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 5.7 MB/s | 1.8 MB 00:00 2024-03-23T22:47:14.956 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-03-23T22:47:15.244 INFO:teuthology.orchestra.run.smithi203.stdout:(32/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 15 MB/s | 48 MB 00:03 2024-03-23T22:47:15.269 INFO:teuthology.orchestra.run.smithi203.stdout:(33/174): ceph-mgr-modules-core-19.0.0-2145.g3b 552 kB/s | 248 kB 00:00 2024-03-23T22:47:15.302 INFO:teuthology.orchestra.run.smithi203.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 498 kB/s | 16 kB 00:00 2024-03-23T22:47:15.319 INFO:teuthology.orchestra.run.smithi203.stdout:(35/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 658 kB/s | 49 kB 00:00 2024-03-23T22:47:15.386 INFO:teuthology.orchestra.run.smithi203.stdout:(36/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 3.1 MB/s | 264 kB 00:00 2024-03-23T22:47:15.420 INFO:teuthology.orchestra.run.smithi203.stdout:(37/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 7.3 MB/s | 748 kB 00:00 2024-03-23T22:47:15.437 INFO:teuthology.orchestra.run.smithi203.stdout:(38/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 362 kB/s | 18 kB 00:00 2024-03-23T22:47:15.579 INFO:teuthology.orchestra.run.smithi203.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 4.1 MB/s | 662 kB 00:00 2024-03-23T22:47:15.612 INFO:teuthology.orchestra.run.smithi203.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 5.1 MB/s | 920 kB 00:00 2024-03-23T22:47:15.679 INFO:teuthology.orchestra.run.smithi203.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.0 MB/s | 106 kB 00:00 2024-03-23T22:47:15.713 INFO:teuthology.orchestra.run.smithi203.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 3.0 MB/s | 307 kB 00:00 2024-03-23T22:47:15.730 INFO:teuthology.orchestra.run.smithi203.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 637 kB/s | 32 kB 00:00 2024-03-23T22:47:15.853 INFO:teuthology.orchestra.run.smithi203.stdout:(44/174): ceph-mgr-diskprediction-local-19.0.0- 5.7 MB/s | 7.4 MB 00:01 2024-03-23T22:47:15.914 INFO:teuthology.orchestra.run.smithi203.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 15 MB/s | 3.0 MB 00:00 2024-03-23T22:47:15.939 INFO:teuthology.orchestra.run.smithi203.stdout:(46/174): flexiblas-openblas-openmp-3.0.4-8.el9 82 kB/s | 17 kB 00:00 2024-03-23T22:47:15.964 INFO:teuthology.orchestra.run.smithi203.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 93 kB/s | 10 kB 00:00 2024-03-23T22:47:15.989 INFO:teuthology.orchestra.run.smithi203.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 884 kB/s | 44 kB 00:00 2024-03-23T22:47:16.015 INFO:teuthology.orchestra.run.smithi203.stdout:(49/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 159 kB 00:00 2024-03-23T22:47:16.040 INFO:teuthology.orchestra.run.smithi203.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-23T22:47:16.065 INFO:teuthology.orchestra.run.smithi203.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 7.6 MB/s | 578 kB 00:00 2024-03-23T22:47:16.087 INFO:teuthology.orchestra.run.smithi203.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.0 MB/s | 46 kB 00:00 2024-03-23T22:47:16.112 INFO:teuthology.orchestra.run.smithi203.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.9 MB/s | 282 kB 00:00 2024-03-23T22:47:16.137 INFO:teuthology.orchestra.run.smithi203.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.6 MB/s | 187 kB 00:00 2024-03-23T22:47:16.154 INFO:teuthology.orchestra.run.smithi203.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 831 kB/s | 34 kB 00:00 2024-03-23T22:47:16.179 INFO:teuthology.orchestra.run.smithi203.stdout:(56/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-03-23T22:47:16.263 INFO:teuthology.orchestra.run.smithi203.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-23T22:47:16.305 INFO:teuthology.orchestra.run.smithi203.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 10 MB/s | 1.5 MB 00:00 2024-03-23T22:47:16.330 INFO:teuthology.orchestra.run.smithi203.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 5.4 MB/s | 811 kB 00:00 2024-03-23T22:47:16.355 INFO:teuthology.orchestra.run.smithi203.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.5 MB/s | 134 kB 00:00 2024-03-23T22:47:16.389 INFO:teuthology.orchestra.run.smithi203.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2024-03-23T22:47:16.498 INFO:teuthology.orchestra.run.smithi203.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 34 MB/s | 5.7 MB 00:00 2024-03-23T22:47:16.523 INFO:teuthology.orchestra.run.smithi203.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 468 kB/s | 78 kB 00:00 2024-03-23T22:47:16.615 INFO:teuthology.orchestra.run.smithi203.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 26 MB/s | 5.8 MB 00:00 2024-03-23T22:47:16.649 INFO:teuthology.orchestra.run.smithi203.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 165 kB/s | 24 kB 00:00 2024-03-23T22:47:16.674 INFO:teuthology.orchestra.run.smithi203.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 122 kB/s | 18 kB 00:00 2024-03-23T22:47:16.699 INFO:teuthology.orchestra.run.smithi203.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 144 kB 00:00 2024-03-23T22:47:16.724 INFO:teuthology.orchestra.run.smithi203.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 635 kB/s | 32 kB 00:00 2024-03-23T22:47:16.808 INFO:teuthology.orchestra.run.smithi203.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-23T22:47:16.833 INFO:teuthology.orchestra.run.smithi203.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 217 kB 00:00 2024-03-23T22:47:16.858 INFO:teuthology.orchestra.run.smithi203.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.3 MB/s | 174 kB 00:00 2024-03-23T22:47:16.884 INFO:teuthology.orchestra.run.smithi203.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 922 kB/s | 69 kB 00:00 2024-03-23T22:47:16.909 INFO:teuthology.orchestra.run.smithi203.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 822 kB/s | 41 kB 00:00 2024-03-23T22:47:16.934 INFO:teuthology.orchestra.run.smithi203.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-23T22:47:16.959 INFO:teuthology.orchestra.run.smithi203.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-23T22:47:16.985 INFO:teuthology.orchestra.run.smithi203.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 954 kB/s | 47 kB 00:00 2024-03-23T22:47:17.010 INFO:teuthology.orchestra.run.smithi203.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-03-23T22:47:17.035 INFO:teuthology.orchestra.run.smithi203.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 568 kB/s | 43 kB 00:00 2024-03-23T22:47:17.261 INFO:teuthology.orchestra.run.smithi203.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 174 kB/s | 39 kB 00:00 2024-03-23T22:47:17.286 INFO:teuthology.orchestra.run.smithi203.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 166 kB/s | 41 kB 00:00 2024-03-23T22:47:17.444 INFO:teuthology.orchestra.run.smithi203.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-03-23T22:47:17.469 INFO:teuthology.orchestra.run.smithi203.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-03-23T22:47:17.486 INFO:teuthology.orchestra.run.smithi203.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 282 kB/s | 56 kB 00:00 2024-03-23T22:47:17.503 INFO:teuthology.orchestra.run.smithi203.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-23T22:47:17.520 INFO:teuthology.orchestra.run.smithi203.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 880 kB/s | 44 kB 00:00 2024-03-23T22:47:17.545 INFO:teuthology.orchestra.run.smithi203.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 79 kB 00:00 2024-03-23T22:47:17.571 INFO:teuthology.orchestra.run.smithi203.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-23T22:47:17.604 INFO:teuthology.orchestra.run.smithi203.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-23T22:47:17.629 INFO:teuthology.orchestra.run.smithi203.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-23T22:47:17.663 INFO:teuthology.orchestra.run.smithi203.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 124 kB 00:00 2024-03-23T22:47:17.688 INFO:teuthology.orchestra.run.smithi203.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.1 MB/s | 187 kB 00:00 2024-03-23T22:47:17.830 INFO:teuthology.orchestra.run.smithi203.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 935 kB/s | 187 kB 00:00 2024-03-23T22:47:17.880 INFO:teuthology.orchestra.run.smithi203.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-23T22:47:17.905 INFO:teuthology.orchestra.run.smithi203.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-03-23T22:47:17.939 INFO:teuthology.orchestra.run.smithi203.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.8 MB/s | 803 kB 00:00 2024-03-23T22:47:18.114 INFO:teuthology.orchestra.run.smithi203.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 92 kB/s | 21 kB 00:00 2024-03-23T22:47:18.131 INFO:teuthology.orchestra.run.smithi203.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 113 kB/s | 25 kB 00:00 2024-03-23T22:47:18.215 INFO:teuthology.orchestra.run.smithi203.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-03-23T22:47:18.249 INFO:teuthology.orchestra.run.smithi203.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.1 MB/s | 248 kB 00:00 2024-03-23T22:47:18.299 INFO:teuthology.orchestra.run.smithi203.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.5 MB/s | 216 kB 00:00 2024-03-23T22:47:18.324 INFO:teuthology.orchestra.run.smithi203.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 465 kB/s | 35 kB 00:00 2024-03-23T22:47:18.408 INFO:teuthology.orchestra.run.smithi203.stdout:(102/174): openssl-devel-3.0.7-27.el9.x86_64.rp 8.8 MB/s | 4.1 MB 00:00 2024-03-23T22:47:18.433 INFO:teuthology.orchestra.run.smithi203.stdout:(103/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 333 kB/s | 44 kB 00:00 2024-03-23T22:47:18.458 INFO:teuthology.orchestra.run.smithi203.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 699 kB/s | 94 kB 00:00 2024-03-23T22:47:18.551 INFO:teuthology.orchestra.run.smithi203.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 157 kB/s | 22 kB 00:00 2024-03-23T22:47:18.734 INFO:teuthology.orchestra.run.smithi203.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.8 MB/s | 863 kB 00:00 2024-03-23T22:47:18.759 INFO:teuthology.orchestra.run.smithi203.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.8 MB/s | 548 kB 00:00 2024-03-23T22:47:18.793 INFO:teuthology.orchestra.run.smithi203.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 333 kB/s | 19 kB 00:00 2024-03-23T22:47:18.818 INFO:teuthology.orchestra.run.smithi203.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.1 MB/s | 309 kB 00:00 2024-03-23T22:47:18.859 INFO:teuthology.orchestra.run.smithi203.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 397 kB/s | 26 kB 00:00 2024-03-23T22:47:18.884 INFO:teuthology.orchestra.run.smithi203.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 743 kB/s | 49 kB 00:00 2024-03-23T22:47:18.909 INFO:teuthology.orchestra.run.smithi203.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-23T22:47:18.960 INFO:teuthology.orchestra.run.smithi203.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-03-23T22:47:19.010 INFO:teuthology.orchestra.run.smithi203.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.2 MB/s | 838 kB 00:00 2024-03-23T22:47:19.069 INFO:teuthology.orchestra.run.smithi203.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 743 kB/s | 43 kB 00:00 2024-03-23T22:47:19.102 INFO:teuthology.orchestra.run.smithi203.stdout:(116/174): python3-asyncssh-2.13.2-5.el9.noarch 3.8 MB/s | 548 kB 00:00 2024-03-23T22:47:19.128 INFO:teuthology.orchestra.run.smithi203.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 550 kB/s | 32 kB 00:00 2024-03-23T22:47:19.161 INFO:teuthology.orchestra.run.smithi203.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 256 kB/s | 15 kB 00:00 2024-03-23T22:47:19.186 INFO:teuthology.orchestra.run.smithi203.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-03-23T22:47:19.253 INFO:teuthology.orchestra.run.smithi203.stdout:(120/174): libarrow-9.0.0-10.el9.x86_64.rpm 8.9 MB/s | 4.4 MB 00:00 2024-03-23T22:47:19.279 INFO:teuthology.orchestra.run.smithi203.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-03-23T22:47:19.312 INFO:teuthology.orchestra.run.smithi203.stdout:(122/174): python3-cherrypy-18.6.1-2.el9.noarch 2.3 MB/s | 358 kB 00:00 2024-03-23T22:47:19.329 INFO:teuthology.orchestra.run.smithi203.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 468 kB/s | 35 kB 00:00 2024-03-23T22:47:19.355 INFO:teuthology.orchestra.run.smithi203.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-03-23T22:47:19.380 INFO:teuthology.orchestra.run.smithi203.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-03-23T22:47:19.405 INFO:teuthology.orchestra.run.smithi203.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-03-23T22:47:19.430 INFO:teuthology.orchestra.run.smithi203.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 353 kB/s | 18 kB 00:00 2024-03-23T22:47:19.481 INFO:teuthology.orchestra.run.smithi203.stdout:(128/174): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-03-23T22:47:19.506 INFO:teuthology.orchestra.run.smithi203.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 230 kB/s | 23 kB 00:00 2024-03-23T22:47:19.531 INFO:teuthology.orchestra.run.smithi203.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 193 kB/s | 19 kB 00:00 2024-03-23T22:47:19.557 INFO:teuthology.orchestra.run.smithi203.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 260 kB/s | 20 kB 00:00 2024-03-23T22:47:19.590 INFO:teuthology.orchestra.run.smithi203.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 793 kB/s | 46 kB 00:00 2024-03-23T22:47:19.616 INFO:teuthology.orchestra.run.smithi203.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-03-23T22:47:19.649 INFO:teuthology.orchestra.run.smithi203.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 8.7 MB/s | 1.0 MB 00:00 2024-03-23T22:47:19.674 INFO:teuthology.orchestra.run.smithi203.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 688 kB/s | 58 kB 00:00 2024-03-23T22:47:19.700 INFO:teuthology.orchestra.run.smithi203.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-23T22:47:19.725 INFO:teuthology.orchestra.run.smithi203.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 217 kB/s | 16 kB 00:00 2024-03-23T22:47:19.750 INFO:teuthology.orchestra.run.smithi203.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-23T22:47:19.772 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-23T22:47:19.776 INFO:teuthology.orchestra.run.smithi203.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 373 kB/s | 28 kB 00:00 2024-03-23T22:47:19.801 INFO:teuthology.orchestra.run.smithi203.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 407 kB/s | 31 kB 00:00 2024-03-23T22:47:19.826 INFO:teuthology.orchestra.run.smithi203.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-23T22:47:19.852 INFO:teuthology.orchestra.run.smithi203.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 782 kB/s | 59 kB 00:00 2024-03-23T22:47:19.877 INFO:teuthology.orchestra.run.smithi203.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 472 kB/s | 36 kB 00:00 2024-03-23T22:47:19.902 INFO:teuthology.orchestra.run.smithi203.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 673 kB/s | 51 kB 00:00 2024-03-23T22:47:19.906 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-23T22:47:19.928 INFO:teuthology.orchestra.run.smithi203.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-23T22:47:19.953 INFO:teuthology.orchestra.run.smithi203.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-23T22:47:19.986 INFO:teuthology.orchestra.run.smithi203.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-03-23T22:47:20.012 INFO:teuthology.orchestra.run.smithi203.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 238 kB/s | 20 kB 00:00 2024-03-23T22:47:20.037 INFO:teuthology.orchestra.run.smithi203.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-23T22:47:20.087 INFO:teuthology.orchestra.run.smithi203.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-23T22:47:20.488 INFO:teuthology.orchestra.run.smithi203.stdout:(151/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 7.0 MB/s | 3.1 MB 00:00 2024-03-23T22:47:20.513 INFO:teuthology.orchestra.run.smithi203.stdout:(152/174): xfsdump-3.1.12-4.el9.x86_64.rpm 859 kB/s | 321 kB 00:00 2024-03-23T22:47:20.814 INFO:teuthology.orchestra.run.smithi203.stdout:(153/174): librados2-19.0.0-2145.g3be9a5c9.el9. 4.2 MB/s | 3.4 MB 00:00 2024-03-23T22:47:20.939 INFO:teuthology.orchestra.run.smithi203.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 338 kB/s | 42 kB 00:00 2024-03-23T22:47:21.472 INFO:teuthology.orchestra.run.smithi203.stdout:(155/174): NetworkManager-libnm-1.47.2-1.el9.x8 1.9 MB/s | 1.8 MB 00:00 2024-03-23T22:47:21.498 INFO:teuthology.orchestra.run.smithi203.stdout:(156/174): NetworkManager-tui-1.47.2-1.el9.x86_ 444 kB/s | 248 kB 00:00 2024-03-23T22:47:21.690 INFO:teuthology.orchestra.run.smithi203.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 329 kB/s | 71 kB 00:00 2024-03-23T22:47:21.715 INFO:teuthology.orchestra.run.smithi203.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 168 kB/s | 36 kB 00:00 2024-03-23T22:47:21.765 INFO:teuthology.orchestra.run.smithi203.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.5 MB/s | 192 kB 00:00 2024-03-23T22:47:21.832 INFO:teuthology.orchestra.run.smithi203.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 346 kB/s | 23 kB 00:00 2024-03-23T22:47:21.899 INFO:teuthology.orchestra.run.smithi203.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 108 kB 00:00 2024-03-23T22:47:21.966 INFO:teuthology.orchestra.run.smithi203.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 155 kB 00:00 2024-03-23T22:47:22.016 INFO:teuthology.orchestra.run.smithi203.stdout:(163/174): elfutils-libs-0.190-2.el9.x86_64.rpm 849 kB/s | 256 kB 00:00 2024-03-23T22:47:22.042 INFO:teuthology.orchestra.run.smithi203.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 136 kB 00:00 2024-03-23T22:47:22.075 INFO:teuthology.orchestra.run.smithi203.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.1 MB/s | 64 kB 00:00 2024-03-23T22:47:22.100 INFO:teuthology.orchestra.run.smithi203.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 491 kB/s | 29 kB 00:00 2024-03-23T22:47:22.220 INFO:teuthology.orchestra.run.smithi173.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-23T22:47:22.221 INFO:teuthology.orchestra.run.smithi173.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-23T22:47:22.235 INFO:teuthology.orchestra.run.smithi173.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-23T22:47:22.236 INFO:teuthology.orchestra.run.smithi173.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-23T22:47:22.237 INFO:teuthology.orchestra.run.smithi173.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-23T22:47:22.238 INFO:teuthology.orchestra.run.smithi173.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-23T22:47:22.240 INFO:teuthology.orchestra.run.smithi173.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-23T22:47:22.243 INFO:teuthology.orchestra.run.smithi173.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-23T22:47:22.244 INFO:teuthology.orchestra.run.smithi173.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-23T22:47:22.323 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-23T22:47:22.395 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-03-23T22:47:22.395 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-23T22:47:22.396 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-03-23T22:47:22.396 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-23T22:47:22.396 INFO:teuthology.orchestra.run.smithi173.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-23T22:47:22.396 INFO:teuthology.orchestra.run.smithi173.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-23T22:47:22.396 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-23T22:47:22.396 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-23T22:47:22.396 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-23T22:47:22.396 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-23T22:47:22.396 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-23T22:47:22.396 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-23T22:47:22.396 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-23T22:47:22.396 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-23T22:47:22.397 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-23T22:47:22.397 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-23T22:47:22.397 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-23T22:47:22.397 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-23T22:47:22.397 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-23T22:47:22.397 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-23T22:47:22.397 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-23T22:47:22.397 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-23T22:47:22.397 INFO:teuthology.orchestra.run.smithi173.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-23T22:47:22.397 INFO:teuthology.orchestra.run.smithi173.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-23T22:47:22.397 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-23T22:47:22.397 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-23T22:47:22.397 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-23T22:47:22.397 INFO:teuthology.orchestra.run.smithi173.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-23T22:47:22.397 INFO:teuthology.orchestra.run.smithi173.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-23T22:47:22.398 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-23T22:47:22.398 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-23T22:47:22.398 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-23T22:47:22.398 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-23T22:47:22.398 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-23T22:47:22.398 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-23T22:47:22.398 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-23T22:47:22.398 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-23T22:47:22.398 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-23T22:47:22.398 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-23T22:47:22.398 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-23T22:47:22.399 INFO:teuthology.orchestra.run.smithi173.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-23T22:47:22.399 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrading: 2024-03-23T22:47:22.399 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-23T22:47:22.399 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-23T22:47:22.399 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-23T22:47:22.399 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-23T22:47:22.399 INFO:teuthology.orchestra.run.smithi173.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-23T22:47:22.399 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-23T22:47:22.399 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-23T22:47:22.399 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-23T22:47:22.399 INFO:teuthology.orchestra.run.smithi173.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-23T22:47:22.399 INFO:teuthology.orchestra.run.smithi173.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-23T22:47:22.399 INFO:teuthology.orchestra.run.smithi173.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-23T22:47:22.399 INFO:teuthology.orchestra.run.smithi173.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-23T22:47:22.400 INFO:teuthology.orchestra.run.smithi173.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-23T22:47:22.400 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-23T22:47:22.400 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-23T22:47:22.400 INFO:teuthology.orchestra.run.smithi173.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-23T22:47:22.400 INFO:teuthology.orchestra.run.smithi173.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-23T22:47:22.400 INFO:teuthology.orchestra.run.smithi173.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-23T22:47:22.400 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-23T22:47:22.400 INFO:teuthology.orchestra.run.smithi173.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-23T22:47:22.400 INFO:teuthology.orchestra.run.smithi173.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-23T22:47:22.400 INFO:teuthology.orchestra.run.smithi173.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-23T22:47:22.400 INFO:teuthology.orchestra.run.smithi173.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-23T22:47:22.400 INFO:teuthology.orchestra.run.smithi173.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-23T22:47:22.400 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-23T22:47:22.401 INFO:teuthology.orchestra.run.smithi173.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-23T22:47:22.401 INFO:teuthology.orchestra.run.smithi173.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-23T22:47:22.401 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-23T22:47:22.401 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-23T22:47:22.401 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-23T22:47:22.401 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-23T22:47:22.401 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-23T22:47:22.401 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-23T22:47:22.401 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-23T22:47:22.401 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-23T22:47:22.401 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-23T22:47:22.401 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-23T22:47:22.401 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-23T22:47:22.401 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-23T22:47:22.402 INFO:teuthology.orchestra.run.smithi173.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-23T22:47:22.402 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-23T22:47:22.402 INFO:teuthology.orchestra.run.smithi173.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-23T22:47:22.402 INFO:teuthology.orchestra.run.smithi173.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-23T22:47:22.402 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-23T22:47:22.402 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-23T22:47:22.402 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-23T22:47:22.402 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-23T22:47:22.402 INFO:teuthology.orchestra.run.smithi173.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-23T22:47:22.402 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-23T22:47:22.402 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-23T22:47:22.402 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-23T22:47:22.402 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-23T22:47:22.403 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-23T22:47:22.403 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-23T22:47:22.403 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-23T22:47:22.403 INFO:teuthology.orchestra.run.smithi173.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-23T22:47:22.403 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-23T22:47:22.403 INFO:teuthology.orchestra.run.smithi173.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-23T22:47:22.403 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-23T22:47:22.403 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-23T22:47:22.403 INFO:teuthology.orchestra.run.smithi173.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-23T22:47:22.403 INFO:teuthology.orchestra.run.smithi173.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-23T22:47:22.403 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-23T22:47:22.403 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-23T22:47:22.403 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-23T22:47:22.403 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-23T22:47:22.403 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-23T22:47:22.404 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-23T22:47:22.404 INFO:teuthology.orchestra.run.smithi173.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-23T22:47:22.404 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-23T22:47:22.404 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-23T22:47:22.404 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-23T22:47:22.404 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-23T22:47:22.404 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-23T22:47:22.404 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-23T22:47:22.404 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-23T22:47:22.404 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-23T22:47:22.404 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-23T22:47:22.404 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-23T22:47:22.404 INFO:teuthology.orchestra.run.smithi173.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-23T22:47:22.404 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-23T22:47:22.405 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-23T22:47:22.405 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-23T22:47:22.405 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-23T22:47:22.405 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-23T22:47:22.405 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-23T22:47:22.405 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-23T22:47:22.405 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-23T22:47:22.405 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-23T22:47:22.405 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-23T22:47:22.405 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-23T22:47:22.405 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-23T22:47:22.405 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-23T22:47:22.405 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-23T22:47:22.405 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-23T22:47:22.406 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-23T22:47:22.406 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-23T22:47:22.406 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-23T22:47:22.406 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-23T22:47:22.406 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-23T22:47:22.406 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-23T22:47:22.406 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-23T22:47:22.406 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-23T22:47:22.406 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-23T22:47:22.406 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-23T22:47:22.406 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-23T22:47:22.406 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-23T22:47:22.406 INFO:teuthology.orchestra.run.smithi173.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-23T22:47:22.406 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-23T22:47:22.406 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-23T22:47:22.407 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-23T22:47:22.407 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-23T22:47:22.407 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-23T22:47:22.407 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-23T22:47:22.407 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-23T22:47:22.407 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-23T22:47:22.407 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-23T22:47:22.407 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-23T22:47:22.407 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-23T22:47:22.407 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-23T22:47:22.407 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-23T22:47:22.407 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-23T22:47:22.407 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-23T22:47:22.408 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-23T22:47:22.408 INFO:teuthology.orchestra.run.smithi173.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-23T22:47:22.408 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-23T22:47:22.408 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-23T22:47:22.408 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-23T22:47:22.408 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-23T22:47:22.408 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-23T22:47:22.408 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-23T22:47:22.408 INFO:teuthology.orchestra.run.smithi173.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-23T22:47:22.408 INFO:teuthology.orchestra.run.smithi173.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-23T22:47:22.408 INFO:teuthology.orchestra.run.smithi173.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-23T22:47:22.408 INFO:teuthology.orchestra.run.smithi173.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-23T22:47:22.408 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-23T22:47:22.408 INFO:teuthology.orchestra.run.smithi173.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-23T22:47:22.408 INFO:teuthology.orchestra.run.smithi173.stdout:Installing weak dependencies: 2024-03-23T22:47:22.408 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-23T22:47:22.409 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T22:47:22.409 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-23T22:47:22.409 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-03-23T22:47:22.409 INFO:teuthology.orchestra.run.smithi173.stdout:Install 150 Packages 2024-03-23T22:47:22.409 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrade 24 Packages 2024-03-23T22:47:22.409 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T22:47:22.409 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 224 M 2024-03-23T22:47:22.410 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-23T22:47:22.453 INFO:teuthology.orchestra.run.smithi012.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-23T22:47:22.455 INFO:teuthology.orchestra.run.smithi012.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-23T22:47:22.479 INFO:teuthology.orchestra.run.smithi012.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-23T22:47:22.481 INFO:teuthology.orchestra.run.smithi012.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-23T22:47:22.482 INFO:teuthology.orchestra.run.smithi012.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-23T22:47:22.484 INFO:teuthology.orchestra.run.smithi012.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-23T22:47:22.487 INFO:teuthology.orchestra.run.smithi012.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-23T22:47:22.491 INFO:teuthology.orchestra.run.smithi012.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-23T22:47:22.492 INFO:teuthology.orchestra.run.smithi012.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-23T22:47:22.551 INFO:teuthology.orchestra.run.smithi203.stdout:(167/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 4.8 MB/s | 2.2 MB 00:00 2024-03-23T22:47:22.587 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-23T22:47:22.613 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2024-03-23T22:47:22.613 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-03-23T22:47:22.613 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2024-03-23T22:47:22.613 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-03-23T22:47:22.613 INFO:teuthology.orchestra.run.smithi012.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-23T22:47:22.613 INFO:teuthology.orchestra.run.smithi012.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-23T22:47:22.613 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-23T22:47:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-23T22:47:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-23T22:47:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-23T22:47:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-23T22:47:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-23T22:47:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-23T22:47:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-23T22:47:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-23T22:47:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-23T22:47:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-23T22:47:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-23T22:47:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-23T22:47:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-23T22:47:22.615 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-23T22:47:22.615 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-23T22:47:22.615 INFO:teuthology.orchestra.run.smithi012.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-23T22:47:22.615 INFO:teuthology.orchestra.run.smithi012.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-23T22:47:22.615 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-23T22:47:22.615 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-23T22:47:22.615 INFO:teuthology.orchestra.run.smithi012.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-23T22:47:22.615 INFO:teuthology.orchestra.run.smithi012.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-23T22:47:22.615 INFO:teuthology.orchestra.run.smithi012.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-23T22:47:22.615 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-23T22:47:22.615 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-23T22:47:22.615 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-23T22:47:22.615 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-23T22:47:22.615 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-23T22:47:22.615 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-23T22:47:22.615 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-23T22:47:22.616 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-23T22:47:22.616 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-23T22:47:22.616 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-23T22:47:22.616 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-23T22:47:22.616 INFO:teuthology.orchestra.run.smithi012.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-23T22:47:22.616 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrading: 2024-03-23T22:47:22.616 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-23T22:47:22.616 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-23T22:47:22.616 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-23T22:47:22.616 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-23T22:47:22.616 INFO:teuthology.orchestra.run.smithi012.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-23T22:47:22.616 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-23T22:47:22.616 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-23T22:47:22.616 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-23T22:47:22.617 INFO:teuthology.orchestra.run.smithi012.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-23T22:47:22.617 INFO:teuthology.orchestra.run.smithi012.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-23T22:47:22.617 INFO:teuthology.orchestra.run.smithi012.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-23T22:47:22.617 INFO:teuthology.orchestra.run.smithi012.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-23T22:47:22.617 INFO:teuthology.orchestra.run.smithi012.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-23T22:47:22.617 INFO:teuthology.orchestra.run.smithi012.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-23T22:47:22.617 INFO:teuthology.orchestra.run.smithi012.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-23T22:47:22.617 INFO:teuthology.orchestra.run.smithi012.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-23T22:47:22.617 INFO:teuthology.orchestra.run.smithi012.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-23T22:47:22.617 INFO:teuthology.orchestra.run.smithi012.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-23T22:47:22.617 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-23T22:47:22.617 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-23T22:47:22.617 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-23T22:47:22.617 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-23T22:47:22.618 INFO:teuthology.orchestra.run.smithi012.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-23T22:47:22.618 INFO:teuthology.orchestra.run.smithi012.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-23T22:47:22.618 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-03-23T22:47:22.618 INFO:teuthology.orchestra.run.smithi012.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-23T22:47:22.618 INFO:teuthology.orchestra.run.smithi012.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-23T22:47:22.618 INFO:teuthology.orchestra.run.smithi012.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-23T22:47:22.618 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-23T22:47:22.618 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-23T22:47:22.618 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-23T22:47:22.618 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-23T22:47:22.618 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-23T22:47:22.618 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-23T22:47:22.618 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-23T22:47:22.618 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-23T22:47:22.618 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-23T22:47:22.619 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-23T22:47:22.619 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-23T22:47:22.619 INFO:teuthology.orchestra.run.smithi012.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-23T22:47:22.619 INFO:teuthology.orchestra.run.smithi012.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-23T22:47:22.619 INFO:teuthology.orchestra.run.smithi012.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-23T22:47:22.619 INFO:teuthology.orchestra.run.smithi012.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-23T22:47:22.619 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-23T22:47:22.619 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-23T22:47:22.619 INFO:teuthology.orchestra.run.smithi012.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-23T22:47:22.619 INFO:teuthology.orchestra.run.smithi012.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-23T22:47:22.619 INFO:teuthology.orchestra.run.smithi012.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-23T22:47:22.619 INFO:teuthology.orchestra.run.smithi012.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-23T22:47:22.619 INFO:teuthology.orchestra.run.smithi012.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-23T22:47:22.619 INFO:teuthology.orchestra.run.smithi012.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-23T22:47:22.620 INFO:teuthology.orchestra.run.smithi012.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-23T22:47:22.620 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-23T22:47:22.620 INFO:teuthology.orchestra.run.smithi012.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-23T22:47:22.620 INFO:teuthology.orchestra.run.smithi012.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-23T22:47:22.620 INFO:teuthology.orchestra.run.smithi012.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-23T22:47:22.620 INFO:teuthology.orchestra.run.smithi012.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-23T22:47:22.620 INFO:teuthology.orchestra.run.smithi012.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-23T22:47:22.620 INFO:teuthology.orchestra.run.smithi012.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-23T22:47:22.620 INFO:teuthology.orchestra.run.smithi012.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-23T22:47:22.620 INFO:teuthology.orchestra.run.smithi012.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-23T22:47:22.620 INFO:teuthology.orchestra.run.smithi012.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-23T22:47:22.620 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-23T22:47:22.620 INFO:teuthology.orchestra.run.smithi012.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-23T22:47:22.620 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-23T22:47:22.620 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-23T22:47:22.621 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-23T22:47:22.621 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-23T22:47:22.621 INFO:teuthology.orchestra.run.smithi012.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-23T22:47:22.621 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-23T22:47:22.621 INFO:teuthology.orchestra.run.smithi012.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-23T22:47:22.621 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-23T22:47:22.621 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-23T22:47:22.621 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-23T22:47:22.621 INFO:teuthology.orchestra.run.smithi012.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-23T22:47:22.621 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-23T22:47:22.621 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-23T22:47:22.621 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-23T22:47:22.621 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-23T22:47:22.621 INFO:teuthology.orchestra.run.smithi012.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-23T22:47:22.622 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-23T22:47:22.622 INFO:teuthology.orchestra.run.smithi012.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-23T22:47:22.622 INFO:teuthology.orchestra.run.smithi012.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-23T22:47:22.622 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-23T22:47:22.622 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-23T22:47:22.622 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-23T22:47:22.622 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-23T22:47:22.622 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-23T22:47:22.622 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-23T22:47:22.622 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-23T22:47:22.622 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-23T22:47:22.622 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-23T22:47:22.622 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-23T22:47:22.622 INFO:teuthology.orchestra.run.smithi012.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-23T22:47:22.623 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-23T22:47:22.623 INFO:teuthology.orchestra.run.smithi012.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-23T22:47:22.623 INFO:teuthology.orchestra.run.smithi012.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-23T22:47:22.623 INFO:teuthology.orchestra.run.smithi012.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-23T22:47:22.623 INFO:teuthology.orchestra.run.smithi012.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-23T22:47:22.623 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-23T22:47:22.623 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-23T22:47:22.623 INFO:teuthology.orchestra.run.smithi012.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-23T22:47:22.623 INFO:teuthology.orchestra.run.smithi012.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-23T22:47:22.623 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-23T22:47:22.623 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-23T22:47:22.623 INFO:teuthology.orchestra.run.smithi012.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-23T22:47:22.623 INFO:teuthology.orchestra.run.smithi012.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-23T22:47:22.623 INFO:teuthology.orchestra.run.smithi012.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-23T22:47:22.624 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-23T22:47:22.624 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-23T22:47:22.624 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-23T22:47:22.624 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-23T22:47:22.624 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-23T22:47:22.624 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-23T22:47:22.624 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-23T22:47:22.624 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-23T22:47:22.624 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-23T22:47:22.624 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-23T22:47:22.624 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-23T22:47:22.624 INFO:teuthology.orchestra.run.smithi012.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-23T22:47:22.624 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-23T22:47:22.624 INFO:teuthology.orchestra.run.smithi012.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-23T22:47:22.624 INFO:teuthology.orchestra.run.smithi012.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-23T22:47:22.624 INFO:teuthology.orchestra.run.smithi012.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-23T22:47:22.625 INFO:teuthology.orchestra.run.smithi012.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-23T22:47:22.625 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-23T22:47:22.625 INFO:teuthology.orchestra.run.smithi012.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-23T22:47:22.625 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-23T22:47:22.625 INFO:teuthology.orchestra.run.smithi012.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-23T22:47:22.625 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-23T22:47:22.625 INFO:teuthology.orchestra.run.smithi012.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-23T22:47:22.625 INFO:teuthology.orchestra.run.smithi012.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-23T22:47:22.625 INFO:teuthology.orchestra.run.smithi012.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-23T22:47:22.625 INFO:teuthology.orchestra.run.smithi012.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-23T22:47:22.625 INFO:teuthology.orchestra.run.smithi012.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-23T22:47:22.625 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-23T22:47:22.625 INFO:teuthology.orchestra.run.smithi012.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-23T22:47:22.625 INFO:teuthology.orchestra.run.smithi012.stdout:Installing weak dependencies: 2024-03-23T22:47:22.625 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-23T22:47:22.625 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:47:22.626 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-03-23T22:47:22.626 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2024-03-23T22:47:22.626 INFO:teuthology.orchestra.run.smithi012.stdout:Install 150 Packages 2024-03-23T22:47:22.626 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrade 24 Packages 2024-03-23T22:47:22.626 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:47:22.626 INFO:teuthology.orchestra.run.smithi203.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 2.2 MB/s | 1.2 MB 00:00 2024-03-23T22:47:22.627 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 224 M 2024-03-23T22:47:22.627 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-03-23T22:47:23.216 INFO:teuthology.orchestra.run.smithi173.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 40 kB/s | 6.6 kB 00:00 2024-03-23T22:47:23.391 INFO:teuthology.orchestra.run.smithi173.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 5.0 MB/s | 890 kB 00:00 2024-03-23T22:47:23.450 INFO:teuthology.orchestra.run.smithi173.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 2.4 MB/s | 145 kB 00:00 2024-03-23T22:47:23.459 INFO:teuthology.orchestra.run.smithi203.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 2.5 MB/s | 2.3 MB 00:00 2024-03-23T22:47:23.650 INFO:teuthology.orchestra.run.smithi173.stdout:(4/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 9.0 MB/s | 5.4 MB 00:00 2024-03-23T22:47:23.668 INFO:teuthology.orchestra.run.smithi203.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 149 kB/s | 31 kB 00:00 2024-03-23T22:47:23.695 INFO:teuthology.orchestra.run.smithi203.stdout:(171/174): util-linux-core-2.37.4-18.el9.x86_64 436 kB/s | 465 kB 00:01 2024-03-23T22:47:23.701 INFO:teuthology.orchestra.run.smithi173.stdout:(5/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 9.2 MB/s | 2.3 MB 00:00 2024-03-23T22:47:23.776 INFO:teuthology.orchestra.run.smithi173.stdout:(6/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 12 MB/s | 1.5 MB 00:00 2024-03-23T22:47:23.877 INFO:teuthology.orchestra.run.smithi203.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 406 kB/s | 74 kB 00:00 2024-03-23T22:47:24.151 INFO:teuthology.orchestra.run.smithi173.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 10 MB/s | 4.7 MB 00:00 2024-03-23T22:47:24.238 INFO:teuthology.orchestra.run.smithi012.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 10 kB/s | 6.6 kB 00:00 2024-03-23T22:47:24.857 INFO:teuthology.orchestra.run.smithi012.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.4 MB/s | 890 kB 00:00 2024-03-23T22:47:24.949 INFO:teuthology.orchestra.run.smithi012.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 1.6 MB/s | 145 kB 00:00 2024-03-23T22:47:24.993 INFO:teuthology.orchestra.run.smithi173.stdout:(8/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el9 12 MB/s | 9.9 MB 00:00 2024-03-23T22:47:25.068 INFO:teuthology.orchestra.run.smithi173.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 336 kB/s | 25 kB 00:00 2024-03-23T22:47:25.308 INFO:teuthology.orchestra.run.smithi012.stdout:(4/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 6.5 MB/s | 2.3 MB 00:00 2024-03-23T22:47:25.385 INFO:teuthology.orchestra.run.smithi173.stdout:(10/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 10 MB/s | 17 MB 00:01 2024-03-23T22:47:25.394 INFO:teuthology.orchestra.run.smithi203.stdout:(173/174): NetworkManager-1.47.2-1.el9.x86_64.r 481 kB/s | 2.3 MB 00:04 2024-03-23T22:47:25.527 INFO:teuthology.orchestra.run.smithi173.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 1.5 MB/s | 220 kB 00:00 2024-03-23T22:47:25.549 INFO:teuthology.orchestra.run.smithi012.stdout:(5/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 6.2 MB/s | 1.5 MB 00:00 2024-03-23T22:47:25.577 INFO:teuthology.orchestra.run.smithi173.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 543 kB/s | 27 kB 00:00 2024-03-23T22:47:25.661 INFO:teuthology.orchestra.run.smithi173.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 376 kB/s | 31 kB 00:00 2024-03-23T22:47:25.936 INFO:teuthology.orchestra.run.smithi203.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 485 kB/s | 1.1 MB 00:02 2024-03-23T22:47:25.941 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-23T22:47:25.942 INFO:teuthology.orchestra.run.smithi203.stdout:Total 13 MB/s | 224 MB 00:16 2024-03-23T22:47:25.970 INFO:teuthology.orchestra.run.smithi173.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 2.4 MB/s | 746 kB 00:00 2024-03-23T22:47:26.062 INFO:teuthology.orchestra.run.smithi173.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.7 MB/s | 163 kB 00:00 2024-03-23T22:47:26.137 INFO:teuthology.orchestra.run.smithi173.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.7 MB/s | 127 kB 00:00 2024-03-23T22:47:26.296 INFO:teuthology.orchestra.run.smithi173.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 3.1 MB/s | 501 kB 00:00 2024-03-23T22:47:26.341 INFO:teuthology.orchestra.run.smithi012.stdout:(6/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 5.9 MB/s | 4.7 MB 00:00 2024-03-23T22:47:26.424 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-23T22:47:26.696 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-23T22:47:26.696 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-23T22:47:27.721 INFO:teuthology.orchestra.run.smithi173.stdout:(18/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 3.6 MB/s | 5.1 MB 00:01 2024-03-23T22:47:27.813 INFO:teuthology.orchestra.run.smithi173.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 493 kB/s | 45 kB 00:00 2024-03-23T22:47:27.905 INFO:teuthology.orchestra.run.smithi173.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.4 MB/s | 131 kB 00:00 2024-03-23T22:47:27.989 INFO:teuthology.orchestra.run.smithi173.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 1.9 MB/s | 162 kB 00:00 2024-03-23T22:47:28.164 INFO:teuthology.orchestra.run.smithi173.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 1.8 MB/s | 321 kB 00:00 2024-03-23T22:47:28.256 INFO:teuthology.orchestra.run.smithi173.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 3.2 MB/s | 298 kB 00:00 2024-03-23T22:47:28.381 INFO:teuthology.orchestra.run.smithi173.stdout:(24/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 795 kB/s | 99 kB 00:00 2024-03-23T22:47:28.448 INFO:teuthology.orchestra.run.smithi173.stdout:(25/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.2 MB/s | 85 kB 00:00 2024-03-23T22:47:28.477 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-23T22:47:28.478 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-23T22:47:28.633 INFO:teuthology.orchestra.run.smithi012.stdout:(7/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x86 7.3 MB/s | 17 MB 00:02 2024-03-23T22:47:28.867 INFO:teuthology.orchestra.run.smithi012.stdout:(8/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 1.0 MB/s | 5.4 MB 00:05 2024-03-23T22:47:28.925 INFO:teuthology.orchestra.run.smithi012.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 440 kB/s | 25 kB 00:00 2024-03-23T22:47:29.282 INFO:teuthology.orchestra.run.smithi173.stdout:(26/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 11 MB/s | 48 MB 00:04 2024-03-23T22:47:29.349 INFO:teuthology.orchestra.run.smithi173.stdout:(27/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 3.4 MB/s | 3.1 MB 00:00 2024-03-23T22:47:29.374 INFO:teuthology.orchestra.run.smithi173.stdout:(28/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-23T22:47:29.421 INFO:teuthology.orchestra.run.smithi173.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2145.g 471 kB/s | 34 kB 00:00 2024-03-23T22:47:29.450 INFO:teuthology.orchestra.run.smithi173.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 1.8 MB/s | 139 kB 00:00 2024-03-23T22:47:29.575 INFO:teuthology.orchestra.run.smithi012.stdout:(10/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el 10 MB/s | 9.9 MB 00:00 2024-03-23T22:47:29.692 INFO:teuthology.orchestra.run.smithi173.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 6.6 MB/s | 1.8 MB 00:00 2024-03-23T22:47:29.692 INFO:teuthology.orchestra.run.smithi012.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 1.9 MB/s | 220 kB 00:00 2024-03-23T22:47:29.776 INFO:teuthology.orchestra.run.smithi012.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 326 kB/s | 27 kB 00:00 2024-03-23T22:47:29.817 INFO:teuthology.orchestra.run.smithi173.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 1.9 MB/s | 248 kB 00:00 2024-03-23T22:47:29.826 INFO:teuthology.orchestra.run.smithi012.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 633 kB/s | 31 kB 00:00 2024-03-23T22:47:29.876 INFO:teuthology.orchestra.run.smithi173.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 848 kB/s | 49 kB 00:00 2024-03-23T22:47:29.968 INFO:teuthology.orchestra.run.smithi173.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 180 kB/s | 16 kB 00:00 2024-03-23T22:47:30.001 INFO:teuthology.orchestra.run.smithi012.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 4.2 MB/s | 746 kB 00:00 2024-03-23T22:47:30.060 INFO:teuthology.orchestra.run.smithi173.stdout:(35/174): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-03-23T22:47:30.068 INFO:teuthology.orchestra.run.smithi012.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 2.4 MB/s | 163 kB 00:00 2024-03-23T22:47:30.085 INFO:teuthology.orchestra.run.smithi173.stdout:(36/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 2.2 MB/s | 264 kB 00:00 2024-03-23T22:47:30.135 INFO:teuthology.orchestra.run.smithi173.stdout:(37/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 9.7 MB/s | 748 kB 00:00 2024-03-23T22:47:30.160 INFO:teuthology.orchestra.run.smithi012.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.4 MB/s | 127 kB 00:00 2024-03-23T22:47:30.161 INFO:teuthology.orchestra.run.smithi173.stdout:(38/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 242 kB/s | 18 kB 00:00 2024-03-23T22:47:30.244 INFO:teuthology.orchestra.run.smithi012.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 5.9 MB/s | 501 kB 00:00 2024-03-23T22:47:30.428 INFO:teuthology.orchestra.run.smithi173.stdout:(39/174): ceph-common-19.0.0-2145.g3be9a5c9.el9 2.8 MB/s | 20 MB 00:07 2024-03-23T22:47:30.503 INFO:teuthology.orchestra.run.smithi173.stdout:(40/174): automake-1.16.2-8.el9.noarch.rpm 1.8 MB/s | 662 kB 00:00 2024-03-23T22:47:30.528 INFO:teuthology.orchestra.run.smithi173.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.0 MB/s | 106 kB 00:00 2024-03-23T22:47:30.562 INFO:teuthology.orchestra.run.smithi173.stdout:(42/174): bison-3.7.4-5.el9.x86_64.rpm 2.2 MB/s | 920 kB 00:00 2024-03-23T22:47:30.596 INFO:teuthology.orchestra.run.smithi173.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 3.3 MB/s | 307 kB 00:00 2024-03-23T22:47:30.621 INFO:teuthology.orchestra.run.smithi173.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 345 kB/s | 32 kB 00:00 2024-03-23T22:47:30.647 INFO:teuthology.orchestra.run.smithi173.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 339 kB/s | 17 kB 00:00 2024-03-23T22:47:30.669 INFO:teuthology.orchestra.run.smithi012.stdout:(18/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 12 MB/s | 5.1 MB 00:00 2024-03-23T22:47:30.705 INFO:teuthology.orchestra.run.smithi173.stdout:(46/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 27 MB/s | 3.0 MB 00:00 2024-03-23T22:47:30.733 INFO:teuthology.orchestra.run.smithi173.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 92 kB/s | 10 kB 00:00 2024-03-23T22:47:30.756 INFO:teuthology.orchestra.run.smithi173.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.5 MB/s | 159 kB 00:00 2024-03-23T22:47:30.781 INFO:teuthology.orchestra.run.smithi173.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 588 kB/s | 44 kB 00:00 2024-03-23T22:47:30.815 INFO:teuthology.orchestra.run.smithi173.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.7 MB/s | 578 kB 00:00 2024-03-23T22:47:30.840 INFO:teuthology.orchestra.run.smithi173.stdout:(51/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-23T22:47:30.857 INFO:teuthology.orchestra.run.smithi173.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 615 kB/s | 46 kB 00:00 2024-03-23T22:47:30.882 INFO:teuthology.orchestra.run.smithi173.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.1 MB/s | 282 kB 00:00 2024-03-23T22:47:30.907 INFO:teuthology.orchestra.run.smithi173.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-23T22:47:30.933 INFO:teuthology.orchestra.run.smithi173.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-23T22:47:30.936 INFO:teuthology.orchestra.run.smithi012.stdout:(19/174): ceph-common-19.0.0-2145.g3be9a5c9.el9 2.8 MB/s | 20 MB 00:07 2024-03-23T22:47:30.949 INFO:teuthology.orchestra.run.smithi173.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 517 kB/s | 34 kB 00:00 2024-03-23T22:47:30.953 INFO:teuthology.orchestra.run.smithi012.stdout:(20/174): python3-ceph-argparse-19.0.0-2145.g3b 159 kB/s | 45 kB 00:00 2024-03-23T22:47:31.003 INFO:teuthology.orchestra.run.smithi012.stdout:(21/174): python3-ceph-common-19.0.0-2145.g3be9 1.9 MB/s | 131 kB 00:00 2024-03-23T22:47:31.033 INFO:teuthology.orchestra.run.smithi173.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 15 MB/s | 1.5 MB 00:00 2024-03-23T22:47:31.037 INFO:teuthology.orchestra.run.smithi012.stdout:(22/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 1.9 MB/s | 162 kB 00:00 2024-03-23T22:47:31.109 INFO:teuthology.orchestra.run.smithi173.stdout:(58/174): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2024-03-23T22:47:31.122 INFO:teuthology.orchestra.run.smithi012.stdout:(23/174): python3-rados-19.0.0-2145.g3be9a5c9.e 2.7 MB/s | 321 kB 00:00 2024-03-23T22:47:31.134 INFO:teuthology.orchestra.run.smithi173.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 4.3 MB/s | 811 kB 00:00 2024-03-23T22:47:31.159 INFO:teuthology.orchestra.run.smithi173.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.1 MB/s | 134 kB 00:00 2024-03-23T22:47:31.162 INFO:teuthology.orchestra.run.smithi012.stdout:(24/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 2.3 MB/s | 298 kB 00:00 2024-03-23T22:47:31.196 INFO:teuthology.orchestra.run.smithi012.stdout:(25/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.3 MB/s | 99 kB 00:00 2024-03-23T22:47:31.230 INFO:teuthology.orchestra.run.smithi012.stdout:(26/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.3 MB/s | 85 kB 00:00 2024-03-23T22:47:31.276 INFO:teuthology.orchestra.run.smithi173.stdout:(61/174): postgresql-server-13.11-1.el9.x86_64. 34 MB/s | 5.7 MB 00:00 2024-03-23T22:47:31.301 INFO:teuthology.orchestra.run.smithi173.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 558 kB/s | 78 kB 00:00 2024-03-23T22:47:31.305 INFO:teuthology.orchestra.run.smithi012.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 2.2 MB/s | 170 kB 00:00 2024-03-23T22:47:31.335 INFO:teuthology.orchestra.run.smithi173.stdout:(63/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-03-23T22:47:31.372 INFO:teuthology.orchestra.run.smithi012.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 516 kB/s | 34 kB 00:00 2024-03-23T22:47:31.435 INFO:teuthology.orchestra.run.smithi173.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 36 MB/s | 5.8 MB 00:00 2024-03-23T22:47:31.439 INFO:teuthology.orchestra.run.smithi012.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 2.1 MB/s | 139 kB 00:00 2024-03-23T22:47:31.452 INFO:teuthology.orchestra.run.smithi173.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 166 kB/s | 24 kB 00:00 2024-03-23T22:47:31.469 INFO:teuthology.orchestra.run.smithi173.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 137 kB/s | 18 kB 00:00 2024-03-23T22:47:31.494 INFO:teuthology.orchestra.run.smithi173.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-23T22:47:31.519 INFO:teuthology.orchestra.run.smithi173.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 632 kB/s | 32 kB 00:00 2024-03-23T22:47:31.595 INFO:teuthology.orchestra.run.smithi173.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-23T22:47:31.620 INFO:teuthology.orchestra.run.smithi173.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 217 kB 00:00 2024-03-23T22:47:31.631 INFO:teuthology.orchestra.run.smithi012.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 9.4 MB/s | 1.8 MB 00:00 2024-03-23T22:47:31.645 INFO:teuthology.orchestra.run.smithi173.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.4 MB/s | 174 kB 00:00 2024-03-23T22:47:31.671 INFO:teuthology.orchestra.run.smithi173.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 914 kB/s | 69 kB 00:00 2024-03-23T22:47:31.696 INFO:teuthology.orchestra.run.smithi173.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 821 kB/s | 41 kB 00:00 2024-03-23T22:47:31.721 INFO:teuthology.orchestra.run.smithi173.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-23T22:47:31.731 INFO:teuthology.orchestra.run.smithi012.stdout:(31/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 5.8 MB/s | 3.1 MB 00:00 2024-03-23T22:47:31.746 INFO:teuthology.orchestra.run.smithi173.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-23T22:47:31.772 INFO:teuthology.orchestra.run.smithi173.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-23T22:47:31.797 INFO:teuthology.orchestra.run.smithi173.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 629 kB/s | 47 kB 00:00 2024-03-23T22:47:31.806 INFO:teuthology.orchestra.run.smithi012.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 3.2 MB/s | 248 kB 00:00 2024-03-23T22:47:31.822 INFO:teuthology.orchestra.run.smithi173.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 569 kB/s | 43 kB 00:00 2024-03-23T22:47:31.847 INFO:teuthology.orchestra.run.smithi173.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 774 kB/s | 39 kB 00:00 2024-03-23T22:47:31.873 INFO:teuthology.orchestra.run.smithi012.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 751 kB/s | 49 kB 00:00 2024-03-23T22:47:31.932 INFO:teuthology.orchestra.run.smithi012.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 286 kB/s | 16 kB 00:00 2024-03-23T22:47:31.973 INFO:teuthology.orchestra.run.smithi173.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 272 kB/s | 41 kB 00:00 2024-03-23T22:47:32.015 INFO:teuthology.orchestra.run.smithi173.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.8 MB/s | 300 kB 00:00 2024-03-23T22:47:32.024 INFO:teuthology.orchestra.run.smithi012.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 2.8 MB/s | 264 kB 00:00 2024-03-23T22:47:32.199 INFO:teuthology.orchestra.run.smithi173.stdout:(82/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-03-23T22:47:32.199 INFO:teuthology.orchestra.run.smithi012.stdout:(36/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 4.2 MB/s | 748 kB 00:00 2024-03-23T22:47:32.223 INFO:teuthology.orchestra.run.smithi173.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 227 kB/s | 56 kB 00:00 2024-03-23T22:47:32.249 INFO:teuthology.orchestra.run.smithi173.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 269 kB/s | 63 kB 00:00 2024-03-23T22:47:32.266 INFO:teuthology.orchestra.run.smithi173.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2024-03-23T22:47:32.291 INFO:teuthology.orchestra.run.smithi173.stdout:(86/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 657 kB/s | 44 kB 00:00 2024-03-23T22:47:32.316 INFO:teuthology.orchestra.run.smithi173.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-23T22:47:32.325 INFO:teuthology.orchestra.run.smithi012.stdout:(37/174): ceph-mgr-diskprediction-local-19.0.0- 11 MB/s | 7.4 MB 00:00 2024-03-23T22:47:32.350 INFO:teuthology.orchestra.run.smithi173.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-23T22:47:32.350 INFO:teuthology.orchestra.run.smithi012.stdout:(38/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 122 kB/s | 18 kB 00:00 2024-03-23T22:47:32.367 INFO:teuthology.orchestra.run.smithi173.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 103 kB 00:00 2024-03-23T22:47:32.392 INFO:teuthology.orchestra.run.smithi173.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.7 MB/s | 124 kB 00:00 2024-03-23T22:47:32.418 INFO:teuthology.orchestra.run.smithi173.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.7 MB/s | 187 kB 00:00 2024-03-23T22:47:32.567 INFO:teuthology.orchestra.run.smithi012.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.7 MB/s | 662 kB 00:00 2024-03-23T22:47:32.609 INFO:teuthology.orchestra.run.smithi012.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 3.5 MB/s | 920 kB 00:00 2024-03-23T22:47:32.634 INFO:teuthology.orchestra.run.smithi012.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.7 MB/s | 106 kB 00:00 2024-03-23T22:47:32.651 INFO:teuthology.orchestra.run.smithi173.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 838 kB/s | 195 kB 00:00 2024-03-23T22:47:32.668 INFO:teuthology.orchestra.run.smithi012.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.3 MB/s | 307 kB 00:00 2024-03-23T22:47:32.693 INFO:teuthology.orchestra.run.smithi173.stdout:(93/174): jq-1.6-16.el9.x86_64.rpm 623 kB/s | 187 kB 00:00 2024-03-23T22:47:32.693 INFO:teuthology.orchestra.run.smithi012.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 573 kB/s | 32 kB 00:00 2024-03-23T22:47:32.710 INFO:teuthology.orchestra.run.smithi173.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-03-23T22:47:32.744 INFO:teuthology.orchestra.run.smithi173.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 803 kB 00:00 2024-03-23T22:47:32.769 INFO:teuthology.orchestra.run.smithi173.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 285 kB/s | 21 kB 00:00 2024-03-23T22:47:32.802 INFO:teuthology.orchestra.run.smithi012.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 23 MB/s | 3.0 MB 00:00 2024-03-23T22:47:32.802 INFO:teuthology.orchestra.run.smithi173.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 275 kB/s | 25 kB 00:00 2024-03-23T22:47:32.852 INFO:teuthology.orchestra.run.smithi012.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 126 kB/s | 17 kB 00:00 2024-03-23T22:47:32.878 INFO:teuthology.orchestra.run.smithi173.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-03-23T22:47:32.894 INFO:teuthology.orchestra.run.smithi012.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 248 kB/s | 10 kB 00:00 2024-03-23T22:47:32.903 INFO:teuthology.orchestra.run.smithi173.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-03-23T22:47:32.919 INFO:teuthology.orchestra.run.smithi012.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.4 MB/s | 159 kB 00:00 2024-03-23T22:47:32.926 INFO:teuthology.orchestra.run.smithi173.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 4.4 MB/s | 216 kB 00:00 2024-03-23T22:47:32.951 INFO:teuthology.orchestra.run.smithi173.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 728 kB/s | 35 kB 00:00 2024-03-23T22:47:32.969 INFO:teuthology.orchestra.run.smithi012.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 905 kB/s | 44 kB 00:00 2024-03-23T22:47:32.977 INFO:teuthology.orchestra.run.smithi173.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 882 kB/s | 44 kB 00:00 2024-03-23T22:47:33.002 INFO:teuthology.orchestra.run.smithi173.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-23T22:47:33.011 INFO:teuthology.orchestra.run.smithi012.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.1 MB/s | 661 kB 00:00 2024-03-23T22:47:33.053 INFO:teuthology.orchestra.run.smithi012.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-23T22:47:33.095 INFO:teuthology.orchestra.run.smithi012.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 6.8 MB/s | 578 kB 00:00 2024-03-23T22:47:33.129 INFO:teuthology.orchestra.run.smithi012.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.0 MB/s | 282 kB 00:00 2024-03-23T22:47:33.136 INFO:teuthology.orchestra.run.smithi173.stdout:(104/174): openssl-devel-3.0.7-27.el9.x86_64.rp 11 MB/s | 4.1 MB 00:00 2024-03-23T22:47:33.162 INFO:teuthology.orchestra.run.smithi012.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 2.9 MB/s | 187 kB 00:00 2024-03-23T22:47:33.196 INFO:teuthology.orchestra.run.smithi012.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.3 MB/s | 217 kB 00:00 2024-03-23T22:47:33.213 INFO:teuthology.orchestra.run.smithi012.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 718 kB/s | 34 kB 00:00 2024-03-23T22:47:33.244 INFO:teuthology.orchestra.run.smithi173.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 83 kB/s | 22 kB 00:00 2024-03-23T22:47:33.363 INFO:teuthology.orchestra.run.smithi012.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 29 MB/s | 4.7 MB 00:00 2024-03-23T22:47:33.419 INFO:teuthology.orchestra.run.smithi173.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.0 MB/s | 863 kB 00:00 2024-03-23T22:47:33.438 INFO:teuthology.orchestra.run.smithi012.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 7.6 MB/s | 1.5 MB 00:00 2024-03-23T22:47:33.489 INFO:teuthology.orchestra.run.smithi012.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 6.4 MB/s | 811 kB 00:00 2024-03-23T22:47:33.514 INFO:teuthology.orchestra.run.smithi012.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.0 MB/s | 134 kB 00:00 2024-03-23T22:47:33.561 INFO:teuthology.orchestra.run.smithi173.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.3 MB/s | 548 kB 00:00 2024-03-23T22:47:33.589 INFO:teuthology.orchestra.run.smithi173.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 116 kB/s | 19 kB 00:00 2024-03-23T22:47:33.637 INFO:teuthology.orchestra.run.smithi173.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 786 kB/s | 309 kB 00:00 2024-03-23T22:47:33.673 INFO:teuthology.orchestra.run.smithi012.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 31 MB/s | 5.7 MB 00:00 2024-03-23T22:47:33.706 INFO:teuthology.orchestra.run.smithi012.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.0 MB/s | 1.0 MB 00:00 2024-03-23T22:47:33.773 INFO:teuthology.orchestra.run.smithi012.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.2 MB/s | 78 kB 00:00 2024-03-23T22:47:33.787 INFO:teuthology.orchestra.run.smithi173.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 147 kB/s | 26 kB 00:00 2024-03-23T22:47:33.846 INFO:teuthology.orchestra.run.smithi173.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 263 kB/s | 49 kB 00:00 2024-03-23T22:47:33.907 INFO:teuthology.orchestra.run.smithi012.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 29 MB/s | 5.8 MB 00:00 2024-03-23T22:47:33.929 INFO:teuthology.orchestra.run.smithi173.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 475 kB/s | 67 kB 00:00 2024-03-23T22:47:33.932 INFO:teuthology.orchestra.run.smithi012.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 170 kB/s | 24 kB 00:00 2024-03-23T22:47:34.005 INFO:teuthology.orchestra.run.smithi012.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 251 kB/s | 18 kB 00:00 2024-03-23T22:47:34.024 INFO:teuthology.orchestra.run.smithi012.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 144 kB 00:00 2024-03-23T22:47:34.049 INFO:teuthology.orchestra.run.smithi173.stdout:(113/174): libarrow-9.0.0-10.el9.x86_64.rpm 9.1 MB/s | 4.4 MB 00:00 2024-03-23T22:47:34.074 INFO:teuthology.orchestra.run.smithi173.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 667 kB/s | 151 kB 00:00 2024-03-23T22:47:34.091 INFO:teuthology.orchestra.run.smithi012.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 482 kB/s | 32 kB 00:00 2024-03-23T22:47:34.158 INFO:teuthology.orchestra.run.smithi173.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.6 MB/s | 838 kB 00:00 2024-03-23T22:47:34.208 INFO:teuthology.orchestra.run.smithi012.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 26 MB/s | 4.8 MB 00:00 2024-03-23T22:47:34.233 INFO:teuthology.orchestra.run.smithi012.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 217 kB 00:00 2024-03-23T22:47:34.241 INFO:teuthology.orchestra.run.smithi173.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 302 kB/s | 43 kB 00:00 2024-03-23T22:47:34.266 INFO:teuthology.orchestra.run.smithi173.stdout:(117/174): python3-asyncssh-2.13.2-5.el9.noarch 2.5 MB/s | 548 kB 00:00 2024-03-23T22:47:34.275 INFO:teuthology.orchestra.run.smithi012.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 4.2 MB/s | 174 kB 00:00 2024-03-23T22:47:34.308 INFO:teuthology.orchestra.run.smithi173.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 215 kB/s | 32 kB 00:00 2024-03-23T22:47:34.342 INFO:teuthology.orchestra.run.smithi012.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 641 kB/s | 69 kB 00:00 2024-03-23T22:47:34.376 INFO:teuthology.orchestra.run.smithi012.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 1.3 MB/s | 41 kB 00:00 2024-03-23T22:47:34.392 INFO:teuthology.orchestra.run.smithi173.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 100 kB/s | 15 kB 00:00 2024-03-23T22:47:34.409 INFO:teuthology.orchestra.run.smithi012.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 3.6 MB/s | 238 kB 00:00 2024-03-23T22:47:34.435 INFO:teuthology.orchestra.run.smithi012.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 132 kB 00:00 2024-03-23T22:47:34.468 INFO:teuthology.orchestra.run.smithi012.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 211 kB 00:00 2024-03-23T22:47:34.483 INFO:teuthology.orchestra.run.smithi173.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 887 kB/s | 172 kB 00:00 2024-03-23T22:47:34.494 INFO:teuthology.orchestra.run.smithi012.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 850 kB/s | 47 kB 00:00 2024-03-23T22:47:34.519 INFO:teuthology.orchestra.run.smithi012.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 873 kB/s | 43 kB 00:00 2024-03-23T22:47:34.567 INFO:teuthology.orchestra.run.smithi173.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 1.5 MB/s | 358 kB 00:00 2024-03-23T22:47:34.634 INFO:teuthology.orchestra.run.smithi173.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.2 MB/s | 303 kB 00:00 2024-03-23T22:47:34.693 INFO:teuthology.orchestra.run.smithi173.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 169 kB/s | 35 kB 00:00 2024-03-23T22:47:34.760 INFO:teuthology.orchestra.run.smithi173.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 1.0 MB/s | 203 kB 00:00 2024-03-23T22:47:34.845 INFO:teuthology.orchestra.run.smithi012.stdout:(78/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-03-23T22:47:34.852 INFO:teuthology.orchestra.run.smithi173.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 916 kB/s | 144 kB 00:00 2024-03-23T22:47:34.869 INFO:teuthology.orchestra.run.smithi012.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 119 kB/s | 39 kB 00:00 2024-03-23T22:47:34.877 INFO:teuthology.orchestra.run.smithi173.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 92 kB/s | 11 kB 00:00 2024-03-23T22:47:34.911 INFO:teuthology.orchestra.run.smithi012.stdout:(80/174): socat-1.7.4.1-5.el9.x86_64.rpm 7.3 MB/s | 300 kB 00:00 2024-03-23T22:47:34.937 INFO:teuthology.orchestra.run.smithi012.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 624 kB/s | 41 kB 00:00 2024-03-23T22:47:34.962 INFO:teuthology.orchestra.run.smithi012.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-23T22:47:34.969 INFO:teuthology.orchestra.run.smithi173.stdout:(127/174): python3-grpcio-1.46.7-10.el9.x86_64. 6.1 MB/s | 2.0 MB 00:00 2024-03-23T22:47:34.987 INFO:teuthology.orchestra.run.smithi012.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 63 kB 00:00 2024-03-23T22:47:35.002 INFO:teuthology.orchestra.run.smithi173.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 117 kB/s | 18 kB 00:00 2024-03-23T22:47:35.012 INFO:teuthology.orchestra.run.smithi012.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 904 kB/s | 44 kB 00:00 2024-03-23T22:47:35.029 INFO:teuthology.orchestra.run.smithi012.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.9 MB/s | 79 kB 00:00 2024-03-23T22:47:35.055 INFO:teuthology.orchestra.run.smithi012.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.8 MB/s | 241 kB 00:00 2024-03-23T22:47:35.077 INFO:teuthology.orchestra.run.smithi173.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 131 kB/s | 23 kB 00:00 2024-03-23T22:47:35.136 INFO:teuthology.orchestra.run.smithi012.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-03-23T22:47:35.144 INFO:teuthology.orchestra.run.smithi173.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 111 kB/s | 19 kB 00:00 2024-03-23T22:47:35.172 INFO:teuthology.orchestra.run.smithi012.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 975 kB/s | 103 kB 00:00 2024-03-23T22:47:35.178 INFO:teuthology.orchestra.run.smithi173.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 129 kB/s | 20 kB 00:00 2024-03-23T22:47:35.197 INFO:teuthology.orchestra.run.smithi012.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.1 MB/s | 124 kB 00:00 2024-03-23T22:47:35.231 INFO:teuthology.orchestra.run.smithi012.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 187 kB 00:00 2024-03-23T22:47:35.328 INFO:teuthology.orchestra.run.smithi173.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 4.1 MB/s | 1.0 MB 00:00 2024-03-23T22:47:35.353 INFO:teuthology.orchestra.run.smithi173.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 222 kB/s | 46 kB 00:00 2024-03-23T22:47:35.390 INFO:teuthology.orchestra.run.smithi012.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 990 kB/s | 187 kB 00:00 2024-03-23T22:47:35.395 INFO:teuthology.orchestra.run.smithi173.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 406 kB/s | 79 kB 00:00 2024-03-23T22:47:35.483 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-23T22:47:35.498 INFO:teuthology.orchestra.run.smithi012.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 195 kB 00:00 2024-03-23T22:47:35.504 INFO:teuthology.orchestra.run.smithi173.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 329 kB/s | 58 kB 00:00 2024-03-23T22:47:35.540 INFO:teuthology.orchestra.run.smithi012.stdout:(93/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2024-03-23T22:47:35.571 INFO:teuthology.orchestra.run.smithi173.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-03-23T22:47:35.590 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-23T22:47:35.596 INFO:teuthology.orchestra.run.smithi173.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 93 kB/s | 16 kB 00:00 2024-03-23T22:47:35.657 INFO:teuthology.orchestra.run.smithi012.stdout:(94/174): elfutils-libelf-devel-0.190-2.el9.x86 185 kB/s | 21 kB 00:00 2024-03-23T22:47:35.693 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-23T22:47:35.713 INFO:teuthology.orchestra.run.smithi173.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 431 kB/s | 90 kB 00:00 2024-03-23T22:47:35.749 INFO:teuthology.orchestra.run.smithi012.stdout:(95/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 280 kB/s | 25 kB 00:00 2024-03-23T22:47:35.772 INFO:teuthology.orchestra.run.smithi173.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 141 kB/s | 28 kB 00:00 2024-03-23T22:47:35.774 INFO:teuthology.orchestra.run.smithi012.stdout:(96/174): python3-requests-2.25.1-8.el9.noarch. 454 kB/s | 125 kB 00:00 2024-03-23T22:47:35.797 INFO:teuthology.orchestra.run.smithi173.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 173 kB/s | 31 kB 00:00 2024-03-23T22:47:35.833 INFO:teuthology.orchestra.run.smithi012.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 4.2 MB/s | 245 kB 00:00 2024-03-23T22:47:35.864 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-23T22:47:35.875 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-23T22:47:35.924 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-23T22:47:35.933 INFO:teuthology.orchestra.run.smithi012.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-03-23T22:47:35.956 INFO:teuthology.orchestra.run.smithi173.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 782 kB/s | 188 kB 00:00 2024-03-23T22:47:35.979 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-23T22:47:35.981 INFO:teuthology.orchestra.run.smithi173.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 224 kB/s | 36 kB 00:00 2024-03-23T22:47:36.018 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-23T22:47:36.031 INFO:teuthology.orchestra.run.smithi173.stdout:(143/174): python3-rsa-4.9-2.el9.noarch.rpm 229 kB/s | 59 kB 00:00 2024-03-23T22:47:36.043 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-23T22:47:36.095 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-23T22:47:36.131 INFO:teuthology.orchestra.run.smithi173.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 291 kB/s | 51 kB 00:00 2024-03-23T22:47:36.198 INFO:teuthology.orchestra.run.smithi173.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 626 kB/s | 90 kB 00:00 2024-03-23T22:47:36.282 INFO:teuthology.orchestra.run.smithi173.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 826 kB/s | 230 kB 00:00 2024-03-23T22:47:36.341 INFO:teuthology.orchestra.run.smithi173.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 141 kB/s | 20 kB 00:00 2024-03-23T22:47:36.350 INFO:teuthology.orchestra.run.smithi012.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 521 kB/s | 216 kB 00:00 2024-03-23T22:47:36.374 INFO:teuthology.orchestra.run.smithi173.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 1.7 MB/s | 427 kB 00:00 2024-03-23T22:47:36.424 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-23T22:47:36.449 INFO:teuthology.orchestra.run.smithi173.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-23T22:47:36.459 INFO:teuthology.orchestra.run.smithi012.stdout:(100/174): openssl-devel-3.0.7-27.el9.x86_64.rp 5.8 MB/s | 4.1 MB 00:00 2024-03-23T22:47:36.476 INFO:teuthology.orchestra.run.smithi012.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 280 kB/s | 35 kB 00:00 2024-03-23T22:47:36.494 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-23T22:47:36.526 INFO:teuthology.orchestra.run.smithi012.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 675 kB/s | 44 kB 00:00 2024-03-23T22:47:36.568 INFO:teuthology.orchestra.run.smithi012.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.0 MB/s | 94 kB 00:00 2024-03-23T22:47:36.591 INFO:teuthology.orchestra.run.smithi173.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 6.3 MB/s | 1.6 MB 00:00 2024-03-23T22:47:36.598 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-23T22:47:36.602 INFO:teuthology.orchestra.run.smithi012.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 300 kB/s | 22 kB 00:00 2024-03-23T22:47:36.620 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-23T22:47:36.695 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-23T22:47:36.708 INFO:teuthology.orchestra.run.smithi173.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.7 MB/s | 321 kB 00:00 2024-03-23T22:47:36.710 INFO:teuthology.orchestra.run.smithi012.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 6.0 MB/s | 863 kB 00:00 2024-03-23T22:47:36.724 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-23T22:47:36.760 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-23T22:47:36.917 INFO:teuthology.orchestra.run.smithi173.stdout:(152/174): NetworkManager-1.47.2-1.el9.x86_64.r 11 MB/s | 2.3 MB 00:00 2024-03-23T22:47:36.936 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-23T22:47:36.998 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-23T22:47:37.000 INFO:teuthology.orchestra.run.smithi173.stdout:(153/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 6.0 MB/s | 3.1 MB 00:00 2024-03-23T22:47:37.026 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-23T22:47:37.051 INFO:teuthology.orchestra.run.smithi173.stdout:(154/174): NetworkManager-libnm-1.47.2-1.el9.x8 14 MB/s | 1.8 MB 00:00 2024-03-23T22:47:37.093 INFO:teuthology.orchestra.run.smithi173.stdout:(155/174): NetworkManager-team-1.47.2-1.el9.x86 457 kB/s | 42 kB 00:00 2024-03-23T22:47:37.130 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-23T22:47:37.151 INFO:teuthology.orchestra.run.smithi173.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-03-23T22:47:37.153 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-23T22:47:37.185 INFO:teuthology.orchestra.run.smithi173.stdout:(157/174): NetworkManager-tui-1.47.2-1.el9.x86_ 1.8 MB/s | 248 kB 00:00 2024-03-23T22:47:37.216 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-23T22:47:37.248 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-23T22:47:37.269 INFO:teuthology.orchestra.run.smithi173.stdout:(158/174): librados2-19.0.0-2145.g3be9a5c9.el9. 3.9 MB/s | 3.4 MB 00:00 2024-03-23T22:47:37.279 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-23T22:47:37.286 INFO:teuthology.orchestra.run.smithi173.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 272 kB/s | 36 kB 00:00 2024-03-23T22:47:37.295 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-23T22:47:37.311 INFO:teuthology.orchestra.run.smithi173.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.5 MB/s | 192 kB 00:00 2024-03-23T22:47:37.336 INFO:teuthology.orchestra.run.smithi173.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.7 MB/s | 256 kB 00:00 2024-03-23T22:47:37.378 INFO:teuthology.orchestra.run.smithi173.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 108 kB 00:00 2024-03-23T22:47:37.410 INFO:teuthology.orchestra.run.smithi012.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 679 kB/s | 548 kB 00:00 2024-03-23T22:47:37.429 INFO:teuthology.orchestra.run.smithi173.stdout:(163/174): libacl-2.3.1-4.el9.x86_64.rpm 162 kB/s | 23 kB 00:00 2024-03-23T22:47:37.462 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-23T22:47:37.470 INFO:teuthology.orchestra.run.smithi173.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.9 MB/s | 136 kB 00:00 2024-03-23T22:47:37.486 INFO:teuthology.orchestra.run.smithi012.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 436 kB/s | 309 kB 00:00 2024-03-23T22:47:37.493 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-23T22:47:37.495 INFO:teuthology.orchestra.run.smithi173.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 969 kB/s | 64 kB 00:00 2024-03-23T22:47:37.525 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-23T22:47:37.528 INFO:teuthology.orchestra.run.smithi012.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 168 kB/s | 19 kB 00:00 2024-03-23T22:47:37.537 INFO:teuthology.orchestra.run.smithi173.stdout:(166/174): libfdisk-2.37.4-18.el9.x86_64.rpm 770 kB/s | 155 kB 00:00 2024-03-23T22:47:37.563 INFO:teuthology.orchestra.run.smithi173.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 311 kB/s | 29 kB 00:00 2024-03-23T22:47:37.622 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-23T22:47:37.646 INFO:teuthology.orchestra.run.smithi173.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 20 MB/s | 2.2 MB 00:00 2024-03-23T22:47:37.686 INFO:teuthology.orchestra.run.smithi012.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 165 kB/s | 26 kB 00:00 2024-03-23T22:47:37.721 INFO:teuthology.orchestra.run.smithi173.stdout:(169/174): openssl-3.0.7-27.el9.x86_64.rpm 5.4 MB/s | 1.2 MB 00:00 2024-03-23T22:47:37.728 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-23T22:47:37.781 INFO:teuthology.orchestra.run.smithi173.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 10 MB/s | 2.3 MB 00:00 2024-03-23T22:47:37.788 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-23T22:47:37.803 INFO:teuthology.orchestra.run.smithi012.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 419 kB/s | 49 kB 00:00 2024-03-23T22:47:37.805 INFO:teuthology.orchestra.run.smithi173.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 372 kB/s | 31 kB 00:00 2024-03-23T22:47:37.831 INFO:teuthology.orchestra.run.smithi173.stdout:(172/174): util-linux-core-2.37.4-18.el9.x86_64 2.5 MB/s | 465 kB 00:00 2024-03-23T22:47:37.864 INFO:teuthology.orchestra.run.smithi173.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 13 MB/s | 1.1 MB 00:00 2024-03-23T22:47:37.889 INFO:teuthology.orchestra.run.smithi173.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 887 kB/s | 74 kB 00:00 2024-03-23T22:47:37.895 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-23T22:47:37.896 INFO:teuthology.orchestra.run.smithi173.stdout:Total 14 MB/s | 224 MB 00:15 2024-03-23T22:47:37.920 INFO:teuthology.orchestra.run.smithi012.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 582 kB/s | 67 kB 00:00 2024-03-23T22:47:38.000 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-23T22:47:38.054 INFO:teuthology.orchestra.run.smithi012.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 7.8 MB/s | 4.4 MB 00:00 2024-03-23T22:47:38.198 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-23T22:47:38.344 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-23T22:47:38.413 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-23T22:47:38.524 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-23T22:47:38.597 INFO:teuthology.orchestra.run.smithi012.stdout:(113/174): ceph-test-19.0.0-2145.g3be9a5c9.el9. 4.9 MB/s | 48 MB 00:09 2024-03-23T22:47:38.611 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-23T22:47:38.611 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-23T22:47:38.627 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-23T22:47:38.629 INFO:teuthology.orchestra.run.smithi012.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 213 kB/s | 151 kB 00:00 2024-03-23T22:47:38.679 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-23T22:47:38.688 INFO:teuthology.orchestra.run.smithi012.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.3 MB/s | 838 kB 00:00 2024-03-23T22:47:38.706 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-23T22:47:38.750 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-23T22:47:38.780 INFO:teuthology.orchestra.run.smithi012.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 290 kB/s | 43 kB 00:00 2024-03-23T22:47:38.826 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-23T22:47:38.872 INFO:teuthology.orchestra.run.smithi012.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 284 kB/s | 32 kB 00:00 2024-03-23T22:47:38.897 INFO:teuthology.orchestra.run.smithi012.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 129 kB/s | 15 kB 00:00 2024-03-23T22:47:38.919 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-23T22:47:38.966 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-23T22:47:38.981 INFO:teuthology.orchestra.run.smithi012.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.6 MB/s | 172 kB 00:00 2024-03-23T22:47:39.041 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-23T22:47:39.056 INFO:teuthology.orchestra.run.smithi012.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 2.2 MB/s | 358 kB 00:00 2024-03-23T22:47:39.074 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-23T22:47:39.106 INFO:teuthology.orchestra.run.smithi012.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-03-23T22:47:39.118 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-23T22:47:39.150 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-23T22:47:39.178 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-23T22:47:39.199 INFO:teuthology.orchestra.run.smithi012.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 249 kB/s | 35 kB 00:00 2024-03-23T22:47:39.232 INFO:teuthology.orchestra.run.smithi012.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 1.6 MB/s | 203 kB 00:00 2024-03-23T22:47:39.249 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-23T22:47:39.283 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-23T22:47:39.327 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-23T22:47:39.349 INFO:teuthology.orchestra.run.smithi012.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-03-23T22:47:39.355 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-23T22:47:39.426 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-23T22:47:39.433 INFO:teuthology.orchestra.run.smithi012.stdout:(125/174): python3-asyncssh-2.13.2-5.el9.noarch 657 kB/s | 548 kB 00:00 2024-03-23T22:47:39.478 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-23T22:47:39.553 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-23T22:47:39.558 INFO:teuthology.orchestra.run.smithi012.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 51 kB/s | 11 kB 00:00 2024-03-23T22:47:39.583 INFO:teuthology.orchestra.run.smithi012.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 118 kB/s | 18 kB 00:00 2024-03-23T22:47:39.683 INFO:teuthology.orchestra.run.smithi012.stdout:(128/174): python3-grpcio-1.46.7-10.el9.x86_64. 4.2 MB/s | 2.0 MB 00:00 2024-03-23T22:47:39.689 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-23T22:47:39.709 INFO:teuthology.orchestra.run.smithi012.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 155 kB/s | 23 kB 00:00 2024-03-23T22:47:39.739 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-23T22:47:39.759 INFO:teuthology.orchestra.run.smithi012.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 111 kB/s | 19 kB 00:00 2024-03-23T22:47:39.793 INFO:teuthology.orchestra.run.smithi012.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 182 kB/s | 20 kB 00:00 2024-03-23T22:47:39.901 INFO:teuthology.orchestra.run.smithi012.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 329 kB/s | 46 kB 00:00 2024-03-23T22:47:39.926 INFO:teuthology.orchestra.run.smithi012.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 593 kB/s | 79 kB 00:00 2024-03-23T22:47:39.968 INFO:teuthology.orchestra.run.smithi012.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 4.9 MB/s | 1.0 MB 00:00 2024-03-23T22:47:40.019 INFO:teuthology.orchestra.run.smithi012.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 497 kB/s | 58 kB 00:00 2024-03-23T22:47:40.052 INFO:teuthology.orchestra.run.smithi012.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-03-23T22:47:40.078 INFO:teuthology.orchestra.run.smithi012.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 152 kB/s | 16 kB 00:00 2024-03-23T22:47:40.136 INFO:teuthology.orchestra.run.smithi012.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 773 kB/s | 90 kB 00:00 2024-03-23T22:47:40.170 INFO:teuthology.orchestra.run.smithi012.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 242 kB/s | 28 kB 00:00 2024-03-23T22:47:40.195 INFO:teuthology.orchestra.run.smithi012.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 264 kB/s | 31 kB 00:00 2024-03-23T22:47:40.262 INFO:teuthology.orchestra.run.smithi012.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-03-23T22:47:40.265 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-23T22:47:40.287 INFO:teuthology.orchestra.run.smithi012.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 508 kB/s | 59 kB 00:00 2024-03-23T22:47:40.316 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-23T22:47:40.317 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-23T22:47:40.337 INFO:teuthology.orchestra.run.smithi012.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 252 kB/s | 36 kB 00:00 2024-03-23T22:47:40.344 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-23T22:47:40.379 INFO:teuthology.orchestra.run.smithi012.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 438 kB/s | 51 kB 00:00 2024-03-23T22:47:40.421 INFO:teuthology.orchestra.run.smithi012.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-23T22:47:40.441 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-23T22:47:40.455 INFO:teuthology.orchestra.run.smithi012.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 769 kB/s | 90 kB 00:00 2024-03-23T22:47:40.488 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-23T22:47:40.497 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-23T22:47:40.528 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-23T22:47:40.530 INFO:teuthology.orchestra.run.smithi012.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 185 kB/s | 20 kB 00:00 2024-03-23T22:47:40.564 INFO:teuthology.orchestra.run.smithi012.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.3 MB/s | 427 kB 00:00 2024-03-23T22:47:40.569 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-23T22:47:40.597 INFO:teuthology.orchestra.run.smithi012.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-23T22:47:40.607 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-23T22:47:40.637 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-23T22:47:40.731 INFO:teuthology.orchestra.run.smithi012.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2024-03-23T22:47:40.775 INFO:teuthology.orchestra.run.smithi012.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.4 MB/s | 321 kB 00:00 2024-03-23T22:47:40.897 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-23T22:47:41.042 INFO:teuthology.orchestra.run.smithi012.stdout:(152/174): librados2-19.0.0-2145.g3be9a5c9.el9. 7.1 MB/s | 3.4 MB 00:00 2024-03-23T22:47:41.117 INFO:teuthology.orchestra.run.smithi012.stdout:(153/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 6.1 MB/s | 3.1 MB 00:00 2024-03-23T22:47:41.165 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-23T22:47:41.232 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-23T22:47:41.277 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-23T22:47:41.410 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-23T22:47:41.455 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-23T22:47:41.510 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-23T22:47:41.582 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-23T22:47:41.614 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-23T22:47:41.617 INFO:teuthology.orchestra.run.smithi012.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 84 kB/s | 42 kB 00:00 2024-03-23T22:47:41.651 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-23T22:47:41.697 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-23T22:47:41.733 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-23T22:47:41.785 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-23T22:47:41.827 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-23T22:47:41.965 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-23T22:47:42.017 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-23T22:47:42.047 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-23T22:47:42.089 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-23T22:47:42.185 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-23T22:47:42.231 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-23T22:47:42.271 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-23T22:47:42.296 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-23T22:47:42.321 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-23T22:47:42.395 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-23T22:47:42.524 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-23T22:47:42.634 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-23T22:47:42.672 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-23T22:47:42.729 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-23T22:47:42.784 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-23T22:47:42.973 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-23T22:47:43.026 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-23T22:47:43.157 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-23T22:47:43.191 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-23T22:47:43.223 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-23T22:47:43.255 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-23T22:47:43.274 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-23T22:47:43.309 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-23T22:47:43.348 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-23T22:47:43.376 INFO:teuthology.orchestra.run.smithi012.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 141 kB/s | 248 kB 00:01 2024-03-23T22:47:43.380 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-23T22:47:43.410 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-23T22:47:43.487 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-23T22:47:43.526 INFO:teuthology.orchestra.run.smithi012.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 496 kB/s | 71 kB 00:00 2024-03-23T22:47:43.554 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-23T22:47:43.587 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-23T22:47:43.629 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-23T22:47:43.663 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-23T22:47:43.745 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-23T22:47:43.792 INFO:teuthology.orchestra.run.smithi012.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 138 kB/s | 36 kB 00:00 2024-03-23T22:47:43.836 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-23T22:47:43.878 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-23T22:47:43.909 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-23T22:47:43.957 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-23T22:47:44.056 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-23T22:47:44.112 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-23T22:47:44.502 INFO:teuthology.orchestra.run.smithi012.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 271 kB/s | 192 kB 00:00 2024-03-23T22:47:44.852 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-23T22:47:44.900 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-23T22:47:44.934 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-23T22:47:44.983 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-23T22:47:45.073 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-23T22:47:45.127 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-23T22:47:45.177 INFO:teuthology.orchestra.run.smithi012.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 379 kB/s | 256 kB 00:00 2024-03-23T22:47:45.302 INFO:teuthology.orchestra.run.smithi012.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 186 kB/s | 23 kB 00:00 2024-03-23T22:47:45.306 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-23T22:47:45.341 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-23T22:47:45.420 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-23T22:47:45.469 INFO:teuthology.orchestra.run.smithi012.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 654 kB/s | 108 kB 00:00 2024-03-23T22:47:45.570 INFO:teuthology.orchestra.run.smithi012.stdout:(162/174): NetworkManager-libnm-1.47.2-1.el9.x8 415 kB/s | 1.8 MB 00:04 2024-03-23T22:47:45.678 INFO:teuthology.orchestra.run.smithi012.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 745 kB/s | 155 kB 00:00 2024-03-23T22:47:45.703 INFO:teuthology.orchestra.run.smithi012.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.0 MB/s | 136 kB 00:00 2024-03-23T22:47:45.754 INFO:teuthology.orchestra.run.smithi012.stdout:(165/174): libuuid-2.37.4-18.el9.x86_64.rpm 586 kB/s | 29 kB 00:00 2024-03-23T22:47:45.779 INFO:teuthology.orchestra.run.smithi012.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 641 kB/s | 64 kB 00:00 2024-03-23T22:47:46.071 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-23T22:47:46.183 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-23T22:47:47.273 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-23T22:47:47.363 INFO:teuthology.orchestra.run.smithi012.stdout:(167/174): NetworkManager-1.47.2-1.el9.x86_64.r 358 kB/s | 2.3 MB 00:06 2024-03-23T22:47:47.385 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-23T22:47:47.415 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-23T22:47:47.419 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-23T22:47:47.450 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-23T22:47:47.459 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-23T22:47:47.494 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-23T22:47:47.499 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-23T22:47:47.526 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-23T22:47:47.552 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-23T22:47:47.565 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-23T22:47:47.586 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-23T22:47:47.593 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-23T22:47:47.612 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-23T22:47:47.654 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-23T22:47:47.907 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-23T22:47:47.931 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-23T22:47:47.983 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-23T22:47:48.022 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-23T22:47:48.038 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-23T22:47:48.053 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-23T22:47:48.114 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-23T22:47:48.143 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-23T22:47:48.173 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-23T22:47:48.263 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-23T22:47:48.293 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-23T22:47:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-23T22:47:48.519 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-23T22:47:48.574 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-23T22:47:48.576 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-23T22:47:48.594 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-23T22:47:48.610 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-23T22:47:48.681 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-23T22:47:48.698 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-23T22:47:48.703 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-23T22:47:48.712 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-23T22:47:48.767 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-23T22:47:48.807 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-23T22:47:48.830 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-23T22:47:48.845 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-23T22:47:49.013 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-23T22:47:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-23T22:47:49.076 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-23T22:47:49.180 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-23T22:47:49.238 INFO:teuthology.orchestra.run.smithi012.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 356 kB/s | 1.2 MB 00:03 2024-03-23T22:47:49.262 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-23T22:47:49.305 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-23T22:47:49.374 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-23T22:47:49.440 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-23T22:47:49.650 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-23T22:47:49.760 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-23T22:47:49.848 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-23T22:47:49.878 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-23T22:47:49.905 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-23T22:47:49.940 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-23T22:47:50.006 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-23T22:47:50.059 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-23T22:47:50.098 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-23T22:47:50.162 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-23T22:47:50.197 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-23T22:47:50.240 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-23T22:47:50.273 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-23T22:47:50.301 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-23T22:47:50.362 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-23T22:47:50.389 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-23T22:47:50.441 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-23T22:47:50.470 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-23T22:47:50.538 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-23T22:47:50.584 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-23T22:47:50.657 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-23T22:47:50.788 INFO:teuthology.orchestra.run.smithi012.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 300 kB/s | 465 kB 00:01 2024-03-23T22:47:50.792 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-23T22:47:50.845 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-23T22:47:50.854 INFO:teuthology.orchestra.run.smithi012.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 476 kB/s | 31 kB 00:00 2024-03-23T22:47:50.910 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-23T22:47:50.950 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-23T22:47:51.284 INFO:teuthology.orchestra.run.smithi012.stdout:(171/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 401 kB/s | 2.2 MB 00:05 2024-03-23T22:47:51.368 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-23T22:47:51.449 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-23T22:47:51.490 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-23T22:47:51.519 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-23T22:47:51.529 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-23T22:47:51.560 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-23T22:47:51.568 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-23T22:47:51.600 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-23T22:47:51.612 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-23T22:47:51.612 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-23T22:47:51.612 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-23T22:47:51.613 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-23T22:47:51.641 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-23T22:47:51.649 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-23T22:47:51.687 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-23T22:47:51.705 INFO:teuthology.orchestra.run.smithi012.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 177 kB/s | 74 kB 00:00 2024-03-23T22:47:51.716 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-23T22:47:51.938 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-23T22:47:52.105 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-23T22:47:52.152 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-23T22:47:52.197 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-23T22:47:52.343 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-23T22:47:52.400 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-23T22:47:52.454 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-23T22:47:52.535 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-23T22:47:52.575 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-23T22:47:52.612 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-23T22:47:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-23T22:47:52.695 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-23T22:47:52.738 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-23T22:47:52.779 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-23T22:47:52.915 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-23T22:47:52.960 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-23T22:47:52.990 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-23T22:47:53.105 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-23T22:47:53.320 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-23T22:47:53.357 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-23T22:47:53.396 INFO:teuthology.orchestra.run.smithi012.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 432 kB/s | 1.1 MB 00:02 2024-03-23T22:47:53.397 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-23T22:47:53.423 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-23T22:47:53.447 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-23T22:47:53.519 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-23T22:47:53.640 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-23T22:47:53.748 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-23T22:47:53.789 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-23T22:47:53.837 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-23T22:47:53.893 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-23T22:47:53.980 INFO:teuthology.orchestra.run.smithi012.stdout:(174/174): util-linux-2.37.4-18.el9.x86_64.rpm 353 kB/s | 2.3 MB 00:06 2024-03-23T22:47:53.991 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-23T22:47:53.991 INFO:teuthology.orchestra.run.smithi012.stdout:Total 7.2 MB/s | 224 MB 00:31 2024-03-23T22:47:54.076 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-23T22:47:54.126 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-23T22:47:54.264 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-23T22:47:54.299 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-23T22:47:54.331 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-23T22:47:54.363 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-23T22:47:54.390 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-23T22:47:54.433 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-23T22:47:54.472 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-23T22:47:54.503 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-23T22:47:54.534 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-23T22:47:54.609 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-23T22:47:54.644 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-23T22:47:54.677 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-23T22:47:54.718 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-23T22:47:54.753 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-23T22:47:54.841 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-23T22:47:54.923 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-23T22:47:54.967 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-23T22:47:55.007 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-23T22:47:55.055 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-23T22:47:55.249 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-23T22:47:55.474 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-03-23T22:47:55.481 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-23T22:47:55.875 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-03-23T22:47:55.875 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-03-23T22:47:56.202 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-23T22:47:56.253 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-23T22:47:56.281 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-23T22:47:56.330 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-23T22:47:56.417 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-23T22:47:56.449 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-23T22:47:56.625 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-23T22:47:56.655 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-23T22:47:56.737 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-23T22:47:57.382 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-23T22:47:57.493 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-23T22:47:58.403 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-03-23T22:47:58.404 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-03-23T22:47:58.715 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-23T22:47:58.797 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-23T22:47:58.828 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-23T22:47:58.859 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-23T22:47:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-23T22:47:59.200 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-23T22:47:59.225 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-23T22:47:59.316 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-23T22:47:59.346 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-23T22:47:59.382 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-23T22:47:59.843 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-23T22:47:59.870 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-23T22:47:59.940 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-23T22:47:59.963 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-23T22:48:02.100 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-23T22:48:02.142 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-23T22:48:02.679 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-23T22:48:02.729 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-23T22:48:02.802 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-23T22:48:02.802 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-23T22:48:02.803 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-23T22:48:02.803 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T22:48:02.830 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-23T22:48:03.379 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-23T22:48:03.379 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /sys 2024-03-23T22:48:03.379 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /proc 2024-03-23T22:48:03.379 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /mnt 2024-03-23T22:48:03.379 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /var/tmp 2024-03-23T22:48:03.379 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /home 2024-03-23T22:48:03.379 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /root 2024-03-23T22:48:03.379 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /tmp 2024-03-23T22:48:03.379 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-23T22:48:03.549 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-23T22:48:03.564 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-23T22:48:03.564 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:03.565 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-23T22:48:03.565 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-23T22:48:05.063 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-23T22:48:05.088 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-23T22:48:05.088 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:05.088 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-23T22:48:05.088 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-23T22:48:05.089 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-23T22:48:05.089 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-23T22:48:05.305 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-23T22:48:05.327 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-23T22:48:05.327 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:05.327 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-23T22:48:05.327 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-23T22:48:05.327 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-23T22:48:05.327 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-23T22:48:05.764 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-23T22:48:05.786 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-23T22:48:05.786 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:05.787 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-23T22:48:05.787 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-23T22:48:05.787 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-23T22:48:05.787 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-23T22:48:05.883 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-23T22:48:05.946 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-03-23T22:48:06.027 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-23T22:48:06.066 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-23T22:48:06.106 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-23T22:48:06.149 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-23T22:48:06.225 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-23T22:48:06.238 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-23T22:48:06.256 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-23T22:48:06.289 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-23T22:48:06.349 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-23T22:48:06.349 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-23T22:48:06.379 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-23T22:48:06.400 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-23T22:48:06.438 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-23T22:48:06.533 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-23T22:48:06.614 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-23T22:48:06.647 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-23T22:48:06.960 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-23T22:48:06.998 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-23T22:48:07.157 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-23T22:48:07.182 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-23T22:48:07.305 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-23T22:48:07.341 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-23T22:48:07.415 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-23T22:48:07.604 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-23T22:48:07.631 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-23T22:48:07.732 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-23T22:48:07.747 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-23T22:48:07.804 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-23T22:48:07.829 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-23T22:48:07.893 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-23T22:48:07.925 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-23T22:48:07.948 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-23T22:48:07.948 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:07.948 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-23T22:48:07.948 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-23T22:48:07.948 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-23T22:48:07.948 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-23T22:48:07.977 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-23T22:48:07.992 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-23T22:48:08.004 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-23T22:48:08.005 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-23T22:48:08.050 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-23T22:48:08.088 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-23T22:48:08.135 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-23T22:48:08.167 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-23T22:48:08.230 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-23T22:48:08.518 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-23T22:48:08.585 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-23T22:48:08.634 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-23T22:48:08.926 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-23T22:48:08.962 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-23T22:48:08.984 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-23T22:48:08.985 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-23T22:48:08.985 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:08.985 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-23T22:48:08.985 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-23T22:48:08.985 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-23T22:48:08.985 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-23T22:48:09.037 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-23T22:48:09.059 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-23T22:48:09.060 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:09.060 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-23T22:48:09.060 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-23T22:48:09.105 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-23T22:48:09.127 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-23T22:48:09.127 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:09.127 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-23T22:48:09.127 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-23T22:48:09.127 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-23T22:48:09.127 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-23T22:48:09.158 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-23T22:48:09.247 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-23T22:48:09.439 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-23T22:48:09.447 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-23T22:48:09.461 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-23T22:48:09.462 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:09.462 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-23T22:48:09.462 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T22:48:09.462 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T22:48:09.462 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-23T22:48:09.502 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-23T22:48:09.893 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-23T22:48:10.080 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-23T22:48:10.211 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-23T22:48:10.252 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-23T22:48:10.312 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-23T22:48:10.381 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-23T22:48:10.500 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-23T22:48:10.577 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-23T22:48:10.626 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-23T22:48:10.740 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-23T22:48:10.789 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-23T22:48:10.855 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-23T22:48:10.911 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-23T22:48:10.951 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-23T22:48:11.077 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-23T22:48:11.113 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-23T22:48:11.182 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-23T22:48:11.220 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-23T22:48:11.346 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-23T22:48:11.417 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-23T22:48:11.543 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-23T22:48:11.731 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-23T22:48:11.800 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-23T22:48:12.525 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-23T22:48:12.652 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-23T22:48:12.782 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-23T22:48:12.838 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-23T22:48:12.866 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-23T22:48:12.914 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-23T22:48:12.992 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-23T22:48:13.051 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-23T22:48:13.097 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-23T22:48:13.547 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-23T22:48:13.746 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-23T22:48:13.859 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-23T22:48:13.910 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-23T22:48:13.917 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-23T22:48:13.952 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-23T22:48:13.982 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-23T22:48:14.009 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-23T22:48:14.144 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-23T22:48:14.189 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-23T22:48:14.192 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-23T22:48:14.284 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-23T22:48:14.319 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-23T22:48:14.358 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-23T22:48:14.393 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-23T22:48:14.397 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-23T22:48:14.439 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-23T22:48:14.458 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-23T22:48:14.479 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-23T22:48:14.502 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-23T22:48:14.535 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-23T22:48:14.551 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-23T22:48:14.584 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-23T22:48:14.595 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-23T22:48:14.613 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-23T22:48:14.661 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-23T22:48:14.668 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-23T22:48:14.727 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-23T22:48:14.729 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-23T22:48:14.762 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-23T22:48:14.782 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-23T22:48:14.782 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /sys 2024-03-23T22:48:14.782 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /proc 2024-03-23T22:48:14.782 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /mnt 2024-03-23T22:48:14.783 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /var/tmp 2024-03-23T22:48:14.783 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /home 2024-03-23T22:48:14.783 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /root 2024-03-23T22:48:14.783 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /tmp 2024-03-23T22:48:14.783 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T22:48:14.934 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-23T22:48:14.952 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-23T22:48:14.967 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-23T22:48:14.967 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:14.967 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-23T22:48:14.967 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T22:48:14.992 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-23T22:48:15.030 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-23T22:48:15.088 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-23T22:48:15.233 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-23T22:48:15.278 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-23T22:48:15.325 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-23T22:48:15.361 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-23T22:48:15.387 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-23T22:48:15.393 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-23T22:48:15.428 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-23T22:48:15.447 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-23T22:48:15.557 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-23T22:48:15.744 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-23T22:48:15.890 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-23T22:48:15.937 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-23T22:48:16.002 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-23T22:48:16.102 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-23T22:48:16.282 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-23T22:48:16.358 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-23T22:48:16.366 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-23T22:48:16.407 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-23T22:48:16.416 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-23T22:48:16.446 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-23T22:48:16.471 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-23T22:48:16.471 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:16.472 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-23T22:48:16.472 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-23T22:48:16.472 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-23T22:48:16.472 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T22:48:16.487 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-23T22:48:16.528 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-23T22:48:16.553 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-23T22:48:16.579 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-23T22:48:16.593 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-23T22:48:16.605 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-23T22:48:16.605 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-23T22:48:16.647 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-23T22:48:16.677 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-23T22:48:16.752 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-23T22:48:16.752 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-23T22:48:16.752 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-23T22:48:16.752 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-23T22:48:16.752 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:16.752 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-23T22:48:16.752 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-23T22:48:16.752 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-23T22:48:16.752 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T22:48:16.752 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-23T22:48:16.753 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-23T22:48:16.775 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-23T22:48:16.783 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-23T22:48:16.809 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-23T22:48:16.821 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-23T22:48:16.842 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-23T22:48:16.868 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-23T22:48:16.889 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-23T22:48:16.892 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-23T22:48:16.926 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-23T22:48:16.949 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-23T22:48:16.965 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-23T22:48:16.996 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-23T22:48:16.998 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-23T22:48:17.023 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-23T22:48:17.049 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-23T22:48:17.074 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-23T22:48:17.099 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-23T22:48:17.106 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-23T22:48:17.106 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-23T22:48:17.130 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-23T22:48:17.147 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-23T22:48:17.169 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-23T22:48:17.170 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:17.170 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-23T22:48:17.170 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-23T22:48:17.170 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-23T22:48:17.170 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T22:48:17.204 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-23T22:48:17.253 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-23T22:48:17.257 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-23T22:48:17.286 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-23T22:48:17.287 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-23T22:48:17.319 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-23T22:48:17.363 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-23T22:48:17.369 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-23T22:48:17.393 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-23T22:48:17.425 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-23T22:48:17.453 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-23T22:48:17.487 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-23T22:48:17.516 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-23T22:48:17.550 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-23T22:48:17.643 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-23T22:48:17.659 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-23T22:48:17.689 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-23T22:48:17.709 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-23T22:48:17.721 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-23T22:48:17.770 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-23T22:48:17.780 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-23T22:48:17.804 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-23T22:48:17.849 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-23T22:48:18.004 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-23T22:48:18.037 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-23T22:48:18.070 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-23T22:48:18.071 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-23T22:48:19.036 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-23T22:48:19.072 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-23T22:48:19.092 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-23T22:48:19.151 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-23T22:48:19.173 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-23T22:48:19.185 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-23T22:48:19.243 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-23T22:48:19.334 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-23T22:48:19.357 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-23T22:48:19.365 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-23T22:48:19.389 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-23T22:48:19.389 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:19.389 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-23T22:48:19.389 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-23T22:48:19.389 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-23T22:48:19.389 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T22:48:19.402 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-23T22:48:19.433 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-23T22:48:19.443 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-23T22:48:19.490 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-23T22:48:19.649 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-23T22:48:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-23T22:48:19.778 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-23T22:48:20.364 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-23T22:48:20.387 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-23T22:48:20.409 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-23T22:48:20.409 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:20.409 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-23T22:48:20.409 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-23T22:48:20.409 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-23T22:48:20.410 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T22:48:20.454 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-23T22:48:20.476 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-23T22:48:20.476 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:20.476 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-23T22:48:20.476 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T22:48:20.522 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-23T22:48:20.543 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-23T22:48:20.543 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:20.543 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-23T22:48:20.543 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-23T22:48:20.544 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-23T22:48:20.544 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T22:48:20.610 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-23T22:48:20.758 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-23T22:48:20.847 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-23T22:48:20.869 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-23T22:48:20.870 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:20.870 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-23T22:48:20.870 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T22:48:20.870 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T22:48:20.870 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T22:48:22.387 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-23T22:48:22.478 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-23T22:48:22.514 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-23T22:48:22.520 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-23T22:48:22.520 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-23T22:48:22.520 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-23T22:48:22.520 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-23T22:48:22.520 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-23T22:48:22.520 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-23T22:48:22.520 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-23T22:48:22.520 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-23T22:48:22.520 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-23T22:48:22.520 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-23T22:48:22.521 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-23T22:48:22.521 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-23T22:48:22.521 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-23T22:48:22.521 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-23T22:48:22.521 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-23T22:48:22.521 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-23T22:48:22.521 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-23T22:48:22.521 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-23T22:48:22.521 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-23T22:48:22.521 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-23T22:48:22.521 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-23T22:48:22.521 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-23T22:48:22.521 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-23T22:48:22.521 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-23T22:48:22.521 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-23T22:48:22.521 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-23T22:48:22.522 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-23T22:48:22.522 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-23T22:48:22.522 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-23T22:48:22.522 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-23T22:48:22.522 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-23T22:48:22.536 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-23T22:48:22.536 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-23T22:48:22.536 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-23T22:48:22.536 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-23T22:48:22.536 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-23T22:48:22.536 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-23T22:48:22.536 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-23T22:48:22.537 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-23T22:48:22.537 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-23T22:48:22.537 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-23T22:48:22.537 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-23T22:48:22.537 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-23T22:48:22.537 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-23T22:48:22.537 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-23T22:48:22.537 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-23T22:48:22.537 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-23T22:48:22.537 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-23T22:48:22.537 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-23T22:48:22.537 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-23T22:48:22.537 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-23T22:48:22.537 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-23T22:48:22.537 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-23T22:48:22.537 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-23T22:48:22.538 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-23T22:48:22.538 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-23T22:48:22.538 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-23T22:48:22.538 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-23T22:48:22.538 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-23T22:48:22.538 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-23T22:48:22.538 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-23T22:48:22.538 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-23T22:48:22.538 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-23T22:48:22.538 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-23T22:48:22.538 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-23T22:48:22.538 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-23T22:48:22.538 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-23T22:48:22.538 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-23T22:48:22.538 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-23T22:48:22.539 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-23T22:48:22.539 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-23T22:48:22.539 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-23T22:48:22.539 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-23T22:48:22.539 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-23T22:48:22.539 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-23T22:48:22.539 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-23T22:48:22.539 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-23T22:48:22.539 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-23T22:48:22.539 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-23T22:48:22.539 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-23T22:48:22.539 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-23T22:48:22.539 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-23T22:48:22.539 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-23T22:48:22.539 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-23T22:48:22.540 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-23T22:48:22.540 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-23T22:48:22.540 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-23T22:48:22.540 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-23T22:48:22.540 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-23T22:48:22.540 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-23T22:48:22.540 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-23T22:48:22.540 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-23T22:48:22.540 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-23T22:48:22.540 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-23T22:48:22.540 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-23T22:48:22.540 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-23T22:48:22.540 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-23T22:48:22.540 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-23T22:48:22.541 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-23T22:48:22.541 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-23T22:48:22.541 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-23T22:48:22.541 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-23T22:48:22.541 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-23T22:48:22.541 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-23T22:48:22.541 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-23T22:48:22.541 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-23T22:48:22.541 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-23T22:48:22.541 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-23T22:48:22.541 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-23T22:48:22.541 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-23T22:48:22.541 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-23T22:48:22.541 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-23T22:48:22.541 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-23T22:48:22.541 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-23T22:48:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-23T22:48:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-23T22:48:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-23T22:48:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-23T22:48:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-23T22:48:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-23T22:48:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-23T22:48:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-23T22:48:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-23T22:48:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-23T22:48:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-23T22:48:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-23T22:48:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-23T22:48:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-23T22:48:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-23T22:48:22.543 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-23T22:48:22.543 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-23T22:48:22.543 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-23T22:48:22.543 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-23T22:48:22.543 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-23T22:48:22.543 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-23T22:48:22.543 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-23T22:48:22.543 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-23T22:48:22.543 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-23T22:48:22.543 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-23T22:48:22.543 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-23T22:48:22.543 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-23T22:48:22.543 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-23T22:48:22.543 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-23T22:48:22.543 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-23T22:48:22.543 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-23T22:48:22.543 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-23T22:48:22.544 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-23T22:48:22.544 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-23T22:48:22.544 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-23T22:48:22.544 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-23T22:48:22.544 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-23T22:48:22.544 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-23T22:48:22.544 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-23T22:48:22.544 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-23T22:48:22.544 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-23T22:48:22.544 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-23T22:48:22.544 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-23T22:48:22.544 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-23T22:48:22.544 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-23T22:48:22.544 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-23T22:48:22.544 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-23T22:48:22.545 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-23T22:48:22.545 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-23T22:48:22.545 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-23T22:48:22.545 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-23T22:48:22.545 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-23T22:48:22.545 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-23T22:48:22.545 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-23T22:48:22.545 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-23T22:48:22.545 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-23T22:48:22.545 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-23T22:48:22.545 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-23T22:48:22.545 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-23T22:48:22.545 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-23T22:48:22.545 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-23T22:48:22.545 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-23T22:48:22.545 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-23T22:48:22.546 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-23T22:48:22.546 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-23T22:48:22.546 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-23T22:48:22.546 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-23T22:48:22.546 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-23T22:48:22.546 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-23T22:48:22.546 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-23T22:48:22.546 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-23T22:48:22.546 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-23T22:48:22.546 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-23T22:48:22.546 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-23T22:48:22.546 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-23T22:48:22.546 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-23T22:48:22.546 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-23T22:48:22.546 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-23T22:48:22.547 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-23T22:48:22.547 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-23T22:48:22.547 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-23T22:48:22.547 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-23T22:48:22.547 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-23T22:48:22.547 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-23T22:48:22.547 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-23T22:48:22.578 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-23T22:48:22.970 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-23T22:48:23.000 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-23T22:48:23.123 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-23T22:48:23.157 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-23T22:48:23.212 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-23T22:48:23.754 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-23T22:48:23.787 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-23T22:48:23.878 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-23T22:48:23.905 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-23T22:48:25.319 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-23T22:48:25.370 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-23T22:48:25.398 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-23T22:48:25.425 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-23T22:48:25.605 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-23T22:48:25.724 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-23T22:48:25.754 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-23T22:48:25.754 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-23T22:48:25.754 INFO:teuthology.orchestra.run.smithi203.stdout:Upgraded: 2024-03-23T22:48:25.754 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-23T22:48:25.754 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-23T22:48:25.754 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-23T22:48:25.754 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-23T22:48:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: acl-2.3.1-4.el9.x86_64 2024-03-23T22:48:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-23T22:48:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-23T22:48:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-23T22:48:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-23T22:48:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-23T22:48:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-23T22:48:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-23T22:48:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-23T22:48:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-23T22:48:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-23T22:48:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-23T22:48:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-23T22:48:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-23T22:48:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-23T22:48:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-23T22:48:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-23T22:48:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-23T22:48:25.756 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-23T22:48:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-23T22:48:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: automake-1.16.2-8.el9.noarch 2024-03-23T22:48:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: bison-3.7.4-5.el9.x86_64 2024-03-23T22:48:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-23T22:48:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:48:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:48:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:48:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:48:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:48:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:48:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:48:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:48:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:48:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:48:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-23T22:48:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: flex-2.6.4-9.el9.x86_64 2024-03-23T22:48:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-23T22:48:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-23T22:48:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-23T22:48:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-23T22:48:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-23T22:48:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-23T22:48:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: jq-1.6-16.el9.x86_64 2024-03-23T22:48:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-23T22:48:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-23T22:48:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-23T22:48:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-23T22:48:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-23T22:48:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-23T22:48:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-23T22:48:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-23T22:48:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-23T22:48:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-23T22:48:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-23T22:48:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-23T22:48:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-23T22:48:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-23T22:48:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: lua-5.4.4-4.el9.x86_64 2024-03-23T22:48:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-23T22:48:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-23T22:48:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-23T22:48:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-23T22:48:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-23T22:48:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-23T22:48:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-23T22:48:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-13.11-1.el9.x86_64 2024-03-23T22:48:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-23T22:48:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-23T22:48:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-23T22:48:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-23T22:48:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-23T22:48:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-23T22:48:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-23T22:48:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-23T22:48:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-23T22:48:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-23T22:48:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-23T22:48:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-23T22:48:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-23T22:48:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-23T22:48:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-23T22:48:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-23T22:48:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-23T22:48:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-23T22:48:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-23T22:48:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-23T22:48:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-23T22:48:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-23T22:48:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-23T22:48:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-23T22:48:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-23T22:48:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-23T22:48:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-23T22:48:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-23T22:48:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-23T22:48:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-23T22:48:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-23T22:48:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-23T22:48:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-23T22:48:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-23T22:48:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-23T22:48:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-23T22:48:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-23T22:48:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-23T22:48:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-23T22:48:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-23T22:48:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply-3.11-14.el9.noarch 2024-03-23T22:48:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-23T22:48:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-23T22:48:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-23T22:48:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-23T22:48:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-23T22:48:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-23T22:48:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-23T22:48:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-23T22:48:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-23T22:48:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-23T22:48:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-23T22:48:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-23T22:48:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-23T22:48:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-23T22:48:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-23T22:48:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-23T22:48:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-23T22:48:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-23T22:48:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-23T22:48:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-23T22:48:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-23T22:48:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-23T22:48:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-23T22:48:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-23T22:48:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-23T22:48:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-23T22:48:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-23T22:48:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: re2-1:20211101-3.el9.x86_64 2024-03-23T22:48:25.766 INFO:teuthology.orchestra.run.smithi203.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-23T22:48:25.766 INFO:teuthology.orchestra.run.smithi203.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-23T22:48:25.766 INFO:teuthology.orchestra.run.smithi203.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-23T22:48:25.766 INFO:teuthology.orchestra.run.smithi203.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-23T22:48:25.766 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-23T22:48:25.766 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-23T22:48:25.766 INFO:teuthology.orchestra.run.smithi203.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-23T22:48:25.767 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-23T22:48:25.767 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-23T22:48:25.804 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-23T22:48:25.857 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-23T22:48:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-23T22:48:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-23T22:48:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-23T22:48:25.995 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-23T22:48:26.049 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-23T22:48:26.109 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-23T22:48:26.143 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-23T22:48:26.222 DEBUG:teuthology.parallel:result is None 2024-03-23T22:48:26.601 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-23T22:48:26.643 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-23T22:48:26.737 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-23T22:48:26.774 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-23T22:48:26.793 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-23T22:48:27.306 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-23T22:48:27.347 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-23T22:48:27.502 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-23T22:48:27.502 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-23T22:48:27.502 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-23T22:48:27.502 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:48:27.586 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-23T22:48:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-23T22:48:27.694 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-23T22:48:27.734 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-23T22:48:27.817 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-23T22:48:27.848 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-23T22:48:27.873 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-23T22:48:27.898 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-23T22:48:27.924 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-23T22:48:27.925 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-23T22:48:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-23T22:48:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-23T22:48:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-23T22:48:28.118 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-23T22:48:28.144 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-23T22:48:28.178 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-23T22:48:28.203 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-23T22:48:28.228 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-23T22:48:28.261 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-23T22:48:28.301 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-23T22:48:28.333 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-23T22:48:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-23T22:48:28.383 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-23T22:48:28.408 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-23T22:48:28.434 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-23T22:48:28.440 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-23T22:48:28.440 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-23T22:48:28.620 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-23T22:48:28.620 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-23T22:48:28.703 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-23T22:48:28.728 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-23T22:48:28.760 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-23T22:48:28.787 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-23T22:48:28.812 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-23T22:48:28.877 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-23T22:48:33.876 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-23T22:48:33.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-23T22:48:33.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-23T22:48:33.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-23T22:48:33.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-23T22:48:33.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-23T22:48:33.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-23T22:48:33.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-23T22:48:33.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-23T22:48:33.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-23T22:48:33.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-23T22:48:33.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-23T22:48:33.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-23T22:48:33.878 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-23T22:48:33.878 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-23T22:48:33.878 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-23T22:48:33.878 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-23T22:48:33.878 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-23T22:48:33.878 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-23T22:48:33.878 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-23T22:48:33.878 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-23T22:48:33.878 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-23T22:48:33.878 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-23T22:48:33.878 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-23T22:48:33.879 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-23T22:48:33.879 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-23T22:48:33.879 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-23T22:48:33.879 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-23T22:48:33.879 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-23T22:48:33.879 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-23T22:48:33.879 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-23T22:48:33.879 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-23T22:48:33.879 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-23T22:48:33.879 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-23T22:48:33.879 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-23T22:48:33.879 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-23T22:48:33.879 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-23T22:48:33.880 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-23T22:48:33.880 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-23T22:48:33.880 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-23T22:48:33.880 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-23T22:48:33.880 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-23T22:48:33.880 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-23T22:48:33.880 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-23T22:48:33.880 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-23T22:48:33.880 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-23T22:48:33.880 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-23T22:48:33.880 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-23T22:48:33.880 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-23T22:48:33.880 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-23T22:48:33.881 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-23T22:48:33.881 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-23T22:48:33.881 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-23T22:48:33.881 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-23T22:48:33.881 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-23T22:48:33.881 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-23T22:48:33.881 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-23T22:48:33.881 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-23T22:48:33.881 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-23T22:48:33.881 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-23T22:48:33.881 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-23T22:48:33.881 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-23T22:48:33.882 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-23T22:48:33.882 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-23T22:48:33.882 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-23T22:48:33.882 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-23T22:48:33.882 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-23T22:48:33.882 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-23T22:48:33.882 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-23T22:48:33.882 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-23T22:48:33.882 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-23T22:48:33.882 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-23T22:48:33.882 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-23T22:48:33.882 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-23T22:48:33.883 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-23T22:48:33.883 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-23T22:48:33.883 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-23T22:48:33.883 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-23T22:48:33.883 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-23T22:48:33.883 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-23T22:48:33.883 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-23T22:48:33.883 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-23T22:48:33.883 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-23T22:48:33.883 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-23T22:48:33.883 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-23T22:48:33.883 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-23T22:48:33.883 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-23T22:48:33.884 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-23T22:48:33.884 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-23T22:48:33.884 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-23T22:48:33.884 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-23T22:48:33.884 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-23T22:48:33.884 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-23T22:48:33.884 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-23T22:48:33.884 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-23T22:48:33.884 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-23T22:48:33.884 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-23T22:48:33.884 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-23T22:48:33.884 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-23T22:48:33.884 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-23T22:48:33.885 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-23T22:48:33.885 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-23T22:48:33.885 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-23T22:48:33.885 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-23T22:48:33.885 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-23T22:48:33.885 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-23T22:48:33.885 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-23T22:48:33.885 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-23T22:48:33.885 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-23T22:48:33.885 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-23T22:48:33.885 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-23T22:48:33.885 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-23T22:48:33.886 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-23T22:48:33.886 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-23T22:48:33.886 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-23T22:48:33.886 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-23T22:48:33.886 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-23T22:48:33.886 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-23T22:48:33.886 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-23T22:48:33.886 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-23T22:48:33.886 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-23T22:48:33.886 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-23T22:48:33.886 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-23T22:48:33.886 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-23T22:48:33.886 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-23T22:48:33.887 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-23T22:48:33.887 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-23T22:48:33.887 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-23T22:48:33.887 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-23T22:48:33.887 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-23T22:48:33.887 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-23T22:48:33.887 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-23T22:48:33.887 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-23T22:48:33.887 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-23T22:48:33.887 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-23T22:48:33.887 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-23T22:48:33.887 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-23T22:48:33.888 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-23T22:48:33.888 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-23T22:48:33.888 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-23T22:48:33.888 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-23T22:48:33.888 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-23T22:48:33.888 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-23T22:48:33.888 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-23T22:48:33.888 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-23T22:48:33.888 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-23T22:48:33.888 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-23T22:48:33.888 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-23T22:48:33.888 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-23T22:48:33.888 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-23T22:48:33.889 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-23T22:48:33.889 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-23T22:48:33.889 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-23T22:48:33.889 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-23T22:48:33.889 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-23T22:48:33.889 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-23T22:48:33.889 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-23T22:48:33.889 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-23T22:48:33.889 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-23T22:48:33.889 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-23T22:48:33.889 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-23T22:48:33.889 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-23T22:48:33.890 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-23T22:48:33.890 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-23T22:48:33.890 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-23T22:48:33.890 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-23T22:48:33.890 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-23T22:48:33.890 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-23T22:48:33.890 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-23T22:48:33.890 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-23T22:48:33.890 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-23T22:48:33.890 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-23T22:48:33.890 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-23T22:48:33.890 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-23T22:48:33.891 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-23T22:48:33.891 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-23T22:48:33.891 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-23T22:48:33.891 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-23T22:48:33.891 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-23T22:48:33.891 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-23T22:48:33.891 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-23T22:48:33.891 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-23T22:48:33.891 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-23T22:48:33.891 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-23T22:48:33.891 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-23T22:48:33.891 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-23T22:48:33.891 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-23T22:48:33.892 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-23T22:48:33.892 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-23T22:48:33.892 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-23T22:48:33.892 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-23T22:48:33.892 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-23T22:48:33.892 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-23T22:48:33.892 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-23T22:48:33.892 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-23T22:48:33.892 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-23T22:48:33.893 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-23T22:48:33.893 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-23T22:48:37.042 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-23T22:48:37.042 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T22:48:37.043 INFO:teuthology.orchestra.run.smithi173.stdout:Upgraded: 2024-03-23T22:48:37.043 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-23T22:48:37.043 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-23T22:48:37.043 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-23T22:48:37.043 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-23T22:48:37.043 INFO:teuthology.orchestra.run.smithi173.stdout: acl-2.3.1-4.el9.x86_64 2024-03-23T22:48:37.043 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-23T22:48:37.043 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-23T22:48:37.043 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-23T22:48:37.043 INFO:teuthology.orchestra.run.smithi173.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-23T22:48:37.043 INFO:teuthology.orchestra.run.smithi173.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-23T22:48:37.043 INFO:teuthology.orchestra.run.smithi173.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-23T22:48:37.044 INFO:teuthology.orchestra.run.smithi173.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-23T22:48:37.044 INFO:teuthology.orchestra.run.smithi173.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-23T22:48:37.044 INFO:teuthology.orchestra.run.smithi173.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.044 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.044 INFO:teuthology.orchestra.run.smithi173.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-23T22:48:37.044 INFO:teuthology.orchestra.run.smithi173.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-23T22:48:37.044 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-23T22:48:37.044 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-23T22:48:37.044 INFO:teuthology.orchestra.run.smithi173.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-23T22:48:37.044 INFO:teuthology.orchestra.run.smithi173.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-23T22:48:37.044 INFO:teuthology.orchestra.run.smithi173.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-23T22:48:37.044 INFO:teuthology.orchestra.run.smithi173.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-23T22:48:37.044 INFO:teuthology.orchestra.run.smithi173.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-23T22:48:37.044 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-23T22:48:37.044 INFO:teuthology.orchestra.run.smithi173.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-23T22:48:37.044 INFO:teuthology.orchestra.run.smithi173.stdout: automake-1.16.2-8.el9.noarch 2024-03-23T22:48:37.045 INFO:teuthology.orchestra.run.smithi173.stdout: bison-3.7.4-5.el9.x86_64 2024-03-23T22:48:37.045 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-23T22:48:37.045 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.045 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.045 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.045 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.045 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:48:37.045 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.045 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.045 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.045 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:48:37.045 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:48:37.045 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:48:37.045 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:48:37.046 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:48:37.046 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.046 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.046 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:48:37.046 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.046 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.046 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.046 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:48:37.046 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:48:37.046 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.046 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.046 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:48:37.046 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-23T22:48:37.046 INFO:teuthology.orchestra.run.smithi173.stdout: flex-2.6.4-9.el9.x86_64 2024-03-23T22:48:37.046 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-23T22:48:37.047 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-23T22:48:37.047 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-23T22:48:37.047 INFO:teuthology.orchestra.run.smithi173.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-23T22:48:37.047 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-23T22:48:37.047 INFO:teuthology.orchestra.run.smithi173.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-23T22:48:37.047 INFO:teuthology.orchestra.run.smithi173.stdout: jq-1.6-16.el9.x86_64 2024-03-23T22:48:37.047 INFO:teuthology.orchestra.run.smithi173.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-23T22:48:37.047 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-23T22:48:37.047 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-23T22:48:37.047 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.047 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.047 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.047 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-23T22:48:37.047 INFO:teuthology.orchestra.run.smithi173.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-23T22:48:37.047 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-23T22:48:37.048 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-23T22:48:37.048 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-23T22:48:37.048 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.048 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.048 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-23T22:48:37.048 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.048 INFO:teuthology.orchestra.run.smithi173.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-23T22:48:37.048 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-23T22:48:37.048 INFO:teuthology.orchestra.run.smithi173.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-23T22:48:37.048 INFO:teuthology.orchestra.run.smithi173.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-23T22:48:37.048 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-23T22:48:37.048 INFO:teuthology.orchestra.run.smithi173.stdout: lua-5.4.4-4.el9.x86_64 2024-03-23T22:48:37.048 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-23T22:48:37.048 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-23T22:48:37.049 INFO:teuthology.orchestra.run.smithi173.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-23T22:48:37.049 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-23T22:48:37.049 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-23T22:48:37.049 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-23T22:48:37.049 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-23T22:48:37.049 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-13.11-1.el9.x86_64 2024-03-23T22:48:37.049 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-23T22:48:37.049 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-23T22:48:37.049 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-23T22:48:37.049 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-23T22:48:37.049 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-23T22:48:37.049 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-23T22:48:37.049 INFO:teuthology.orchestra.run.smithi173.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-23T22:48:37.049 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-23T22:48:37.049 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-23T22:48:37.049 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-23T22:48:37.050 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.050 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.050 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.050 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-23T22:48:37.050 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-23T22:48:37.050 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-23T22:48:37.050 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-23T22:48:37.050 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-23T22:48:37.050 INFO:teuthology.orchestra.run.smithi173.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-23T22:48:37.050 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-23T22:48:37.050 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-23T22:48:37.050 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-23T22:48:37.051 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-23T22:48:37.051 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-23T22:48:37.051 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-23T22:48:37.051 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-23T22:48:37.051 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-23T22:48:37.051 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-23T22:48:37.051 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-23T22:48:37.051 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-23T22:48:37.051 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-23T22:48:37.051 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-23T22:48:37.051 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-23T22:48:37.051 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-23T22:48:37.051 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-23T22:48:37.051 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-23T22:48:37.052 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-23T22:48:37.052 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-23T22:48:37.052 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-23T22:48:37.052 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-23T22:48:37.052 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-23T22:48:37.052 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-23T22:48:37.052 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-23T22:48:37.052 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply-3.11-14.el9.noarch 2024-03-23T22:48:37.052 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-23T22:48:37.052 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-23T22:48:37.052 INFO:teuthology.orchestra.run.smithi173.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-23T22:48:37.052 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-23T22:48:37.052 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-23T22:48:37.052 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-23T22:48:37.052 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-23T22:48:37.052 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-23T22:48:37.053 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-23T22:48:37.053 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.053 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.053 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-23T22:48:37.053 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-23T22:48:37.053 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-23T22:48:37.053 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.053 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-23T22:48:37.053 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-23T22:48:37.053 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-23T22:48:37.053 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-23T22:48:37.053 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-23T22:48:37.053 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-23T22:48:37.053 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-23T22:48:37.053 INFO:teuthology.orchestra.run.smithi173.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-23T22:48:37.054 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-23T22:48:37.054 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-23T22:48:37.054 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-23T22:48:37.054 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-23T22:48:37.054 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-23T22:48:37.054 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-23T22:48:37.054 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-23T22:48:37.054 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.054 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.054 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:48:37.054 INFO:teuthology.orchestra.run.smithi173.stdout: re2-1:20211101-3.el9.x86_64 2024-03-23T22:48:37.054 INFO:teuthology.orchestra.run.smithi173.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-23T22:48:37.054 INFO:teuthology.orchestra.run.smithi173.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-23T22:48:37.054 INFO:teuthology.orchestra.run.smithi173.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-23T22:48:37.054 INFO:teuthology.orchestra.run.smithi173.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-23T22:48:37.055 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-23T22:48:37.055 INFO:teuthology.orchestra.run.smithi173.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-23T22:48:37.055 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T22:48:37.055 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-23T22:48:37.517 DEBUG:teuthology.parallel:result is None 2024-03-23T22:48:41.828 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-23T22:48:41.828 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /sys 2024-03-23T22:48:41.828 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /proc 2024-03-23T22:48:41.828 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /mnt 2024-03-23T22:48:41.828 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /var/tmp 2024-03-23T22:48:41.828 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /home 2024-03-23T22:48:41.828 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /root 2024-03-23T22:48:41.828 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /tmp 2024-03-23T22:48:41.828 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:48:42.051 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-23T22:48:42.100 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-23T22:48:42.101 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:42.101 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-23T22:48:42.101 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:48:43.788 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-23T22:48:43.879 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-23T22:48:43.879 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:43.879 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-23T22:48:43.879 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-23T22:48:43.879 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-23T22:48:43.879 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:48:44.146 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-23T22:48:44.215 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-23T22:48:44.215 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:44.215 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-23T22:48:44.215 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-23T22:48:44.215 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-23T22:48:44.215 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:48:44.730 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-23T22:48:44.804 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-23T22:48:44.805 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:44.805 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-23T22:48:44.805 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-23T22:48:44.805 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-23T22:48:44.805 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:48:44.915 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-23T22:48:45.575 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-23T22:48:45.607 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-23T22:48:45.658 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-23T22:48:45.762 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-23T22:48:45.820 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-23T22:48:46.164 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-23T22:48:46.196 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-23T22:48:47.346 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-23T22:48:47.371 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-23T22:48:47.521 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-23T22:48:47.710 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-23T22:48:47.740 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-23T22:48:47.815 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-23T22:48:47.815 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:47.815 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-23T22:48:47.815 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-23T22:48:47.815 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-23T22:48:47.815 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:48:47.874 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-23T22:48:47.913 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-23T22:48:48.002 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-23T22:48:49.022 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-23T22:48:49.059 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-23T22:48:49.132 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-23T22:48:49.133 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:49.133 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-23T22:48:49.133 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-23T22:48:49.133 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-23T22:48:49.133 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:48:49.185 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-23T22:48:49.259 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-23T22:48:49.260 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:49.260 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-23T22:48:49.260 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:48:49.335 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-23T22:48:49.413 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-23T22:48:49.413 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:49.413 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-23T22:48:49.413 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-23T22:48:49.413 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-23T22:48:49.414 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:48:49.827 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-23T22:48:49.899 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-23T22:48:49.899 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T22:48:49.899 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-23T22:48:49.899 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T22:48:49.899 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T22:48:49.900 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:48:55.060 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-23T22:48:55.118 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-23T22:48:55.158 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-23T22:48:55.198 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-23T22:48:55.515 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-23T22:48:55.673 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-23T22:48:55.725 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-23T22:48:55.781 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-23T22:48:55.841 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-23T22:48:55.886 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-23T22:48:55.921 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-23T22:48:55.965 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-23T22:48:56.019 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-23T22:48:56.104 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-23T22:48:56.245 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-23T22:48:56.293 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-23T22:48:57.202 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-23T22:48:57.247 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-23T22:48:57.305 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-23T22:48:58.755 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-23T22:48:58.845 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-23T22:48:58.897 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-23T22:48:59.040 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-23T22:48:59.097 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-23T22:48:59.130 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-23T22:48:59.165 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-23T22:48:59.215 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-23T22:48:59.215 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-23T22:48:59.377 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-23T22:48:59.418 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-23T22:48:59.418 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-23T22:48:59.556 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-23T22:48:59.601 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-23T22:48:59.655 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-23T22:48:59.688 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-23T22:48:59.720 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-23T22:48:59.801 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-23T22:48:59.878 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-23T22:48:59.925 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-23T22:48:59.958 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-23T22:48:59.992 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-23T22:49:00.026 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-23T22:49:00.059 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-23T22:49:00.095 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-23T22:49:00.096 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-23T22:49:00.425 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-23T22:49:00.426 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-23T22:49:00.548 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-23T22:49:00.579 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-23T22:49:00.634 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-23T22:49:00.673 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-23T22:49:00.721 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-23T22:49:00.868 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-23T22:49:01.142 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-23T22:49:01.142 INFO:teuthology.orchestra.run.smithi012.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-23T22:49:01.142 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:49:06.238 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-23T22:49:06.238 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-23T22:49:06.238 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-23T22:49:06.238 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-23T22:49:06.238 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-23T22:49:06.238 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-23T22:49:06.238 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-23T22:49:06.239 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-23T22:49:06.239 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-23T22:49:06.239 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-23T22:49:06.239 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-23T22:49:06.239 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-23T22:49:06.239 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-23T22:49:06.239 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-23T22:49:06.239 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-23T22:49:06.239 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-23T22:49:06.239 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-23T22:49:06.239 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-23T22:49:06.240 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-23T22:49:06.240 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-23T22:49:06.240 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-23T22:49:06.240 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-23T22:49:06.240 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-23T22:49:06.240 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-23T22:49:06.240 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-23T22:49:06.240 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-23T22:49:06.240 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-23T22:49:06.240 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-23T22:49:06.240 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-23T22:49:06.240 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-23T22:49:06.241 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-23T22:49:06.241 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-23T22:49:06.241 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-23T22:49:06.241 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-23T22:49:06.241 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-23T22:49:06.241 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-23T22:49:06.241 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-23T22:49:06.241 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-23T22:49:06.241 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-23T22:49:06.241 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-23T22:49:06.241 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-23T22:49:06.241 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-23T22:49:06.242 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-23T22:49:06.242 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-23T22:49:06.242 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-23T22:49:06.242 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-23T22:49:06.242 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-23T22:49:06.242 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-23T22:49:06.242 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-23T22:49:06.242 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-23T22:49:06.242 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-23T22:49:06.242 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-23T22:49:06.243 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-23T22:49:06.243 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-23T22:49:06.243 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-23T22:49:06.243 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-23T22:49:06.243 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-23T22:49:06.243 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-23T22:49:06.243 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-23T22:49:06.243 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-23T22:49:06.243 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-23T22:49:06.243 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-23T22:49:06.243 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-23T22:49:06.243 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-23T22:49:06.244 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-23T22:49:06.244 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-23T22:49:06.244 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-23T22:49:06.244 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-23T22:49:06.244 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-23T22:49:06.244 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-23T22:49:06.244 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-23T22:49:06.244 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-23T22:49:06.244 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-23T22:49:06.244 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-23T22:49:06.244 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-23T22:49:06.244 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-23T22:49:06.245 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-23T22:49:06.245 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-23T22:49:06.245 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-23T22:49:06.245 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-23T22:49:06.245 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-23T22:49:06.245 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-23T22:49:06.245 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-23T22:49:06.245 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-23T22:49:06.245 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-23T22:49:06.245 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-23T22:49:06.245 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-23T22:49:06.245 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-23T22:49:06.245 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-23T22:49:06.246 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-23T22:49:06.246 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-23T22:49:06.246 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-23T22:49:06.246 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-23T22:49:06.246 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-23T22:49:06.246 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-23T22:49:06.246 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-23T22:49:06.246 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-23T22:49:06.246 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-23T22:49:06.246 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-23T22:49:06.246 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-23T22:49:06.246 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-23T22:49:06.247 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-23T22:49:06.247 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-23T22:49:06.247 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-23T22:49:06.247 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-23T22:49:06.247 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-23T22:49:06.247 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-23T22:49:06.247 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-23T22:49:06.247 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-23T22:49:06.247 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-23T22:49:06.247 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-23T22:49:06.247 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-23T22:49:06.247 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-23T22:49:06.248 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-23T22:49:06.248 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-23T22:49:06.248 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-23T22:49:06.248 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-23T22:49:06.248 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-23T22:49:06.248 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-23T22:49:06.248 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-23T22:49:06.248 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-23T22:49:06.248 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-23T22:49:06.248 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-23T22:49:06.248 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-23T22:49:06.248 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-23T22:49:06.248 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-23T22:49:06.249 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-23T22:49:06.249 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-23T22:49:06.249 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-23T22:49:06.249 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-23T22:49:06.249 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-23T22:49:06.249 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-23T22:49:06.249 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-23T22:49:06.249 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-23T22:49:06.249 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-23T22:49:06.249 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-23T22:49:06.249 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-23T22:49:06.249 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-23T22:49:06.249 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-23T22:49:06.250 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-23T22:49:06.250 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-23T22:49:06.250 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-23T22:49:06.250 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-23T22:49:06.250 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-23T22:49:06.250 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-23T22:49:06.250 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-23T22:49:06.250 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-23T22:49:06.250 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-23T22:49:06.250 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-23T22:49:06.250 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-23T22:49:06.250 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-23T22:49:06.251 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-23T22:49:06.251 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-23T22:49:06.251 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-23T22:49:06.251 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-23T22:49:06.251 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-23T22:49:06.251 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-23T22:49:06.251 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-23T22:49:06.251 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-23T22:49:06.251 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-23T22:49:06.251 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-23T22:49:06.251 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-23T22:49:06.251 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-23T22:49:06.251 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-23T22:49:06.252 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-23T22:49:06.252 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-23T22:49:06.252 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-23T22:49:06.252 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-23T22:49:06.252 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-23T22:49:06.252 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-23T22:49:06.252 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-23T22:49:06.252 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-23T22:49:06.252 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-23T22:49:06.252 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-23T22:49:06.252 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-23T22:49:06.252 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-23T22:49:06.252 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-23T22:49:06.253 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-23T22:49:06.253 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-23T22:49:06.253 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-23T22:49:06.253 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-23T22:49:06.253 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-23T22:49:06.253 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-23T22:49:06.253 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-23T22:49:06.253 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-23T22:49:06.253 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-23T22:49:06.253 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-23T22:49:06.253 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-23T22:49:06.253 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-23T22:49:06.254 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-23T22:49:06.254 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-23T22:49:06.254 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-23T22:49:06.254 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-23T22:49:06.254 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-23T22:49:06.254 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-23T22:49:06.254 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-23T22:49:06.254 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-23T22:49:08.960 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-23T22:49:08.960 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:49:08.960 INFO:teuthology.orchestra.run.smithi012.stdout:Upgraded: 2024-03-23T22:49:08.961 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-23T22:49:08.961 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-23T22:49:08.961 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-23T22:49:08.961 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-23T22:49:08.961 INFO:teuthology.orchestra.run.smithi012.stdout: acl-2.3.1-4.el9.x86_64 2024-03-23T22:49:08.961 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-23T22:49:08.961 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-23T22:49:08.961 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-23T22:49:08.961 INFO:teuthology.orchestra.run.smithi012.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-23T22:49:08.961 INFO:teuthology.orchestra.run.smithi012.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-23T22:49:08.961 INFO:teuthology.orchestra.run.smithi012.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-23T22:49:08.961 INFO:teuthology.orchestra.run.smithi012.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-23T22:49:08.961 INFO:teuthology.orchestra.run.smithi012.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-23T22:49:08.961 INFO:teuthology.orchestra.run.smithi012.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.962 INFO:teuthology.orchestra.run.smithi012.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.962 INFO:teuthology.orchestra.run.smithi012.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-23T22:49:08.962 INFO:teuthology.orchestra.run.smithi012.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-23T22:49:08.962 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-23T22:49:08.962 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-23T22:49:08.962 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-23T22:49:08.962 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-23T22:49:08.962 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-23T22:49:08.962 INFO:teuthology.orchestra.run.smithi012.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-23T22:49:08.962 INFO:teuthology.orchestra.run.smithi012.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-23T22:49:08.962 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-03-23T22:49:08.962 INFO:teuthology.orchestra.run.smithi012.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-23T22:49:08.962 INFO:teuthology.orchestra.run.smithi012.stdout: automake-1.16.2-8.el9.noarch 2024-03-23T22:49:08.963 INFO:teuthology.orchestra.run.smithi012.stdout: bison-3.7.4-5.el9.x86_64 2024-03-23T22:49:08.963 INFO:teuthology.orchestra.run.smithi012.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-23T22:49:08.963 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.963 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.963 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.963 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.963 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:49:08.963 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.963 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.963 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.963 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:49:08.963 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:49:08.963 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:49:08.963 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:49:08.964 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:49:08.964 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.964 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.964 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:49:08.964 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.964 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.964 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.964 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:49:08.964 INFO:teuthology.orchestra.run.smithi012.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:49:08.964 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.964 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.964 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-23T22:49:08.965 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-23T22:49:08.965 INFO:teuthology.orchestra.run.smithi012.stdout: flex-2.6.4-9.el9.x86_64 2024-03-23T22:49:08.965 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-23T22:49:08.965 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-23T22:49:08.965 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-23T22:49:08.965 INFO:teuthology.orchestra.run.smithi012.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-23T22:49:08.965 INFO:teuthology.orchestra.run.smithi012.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-23T22:49:08.965 INFO:teuthology.orchestra.run.smithi012.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-23T22:49:08.965 INFO:teuthology.orchestra.run.smithi012.stdout: jq-1.6-16.el9.x86_64 2024-03-23T22:49:08.965 INFO:teuthology.orchestra.run.smithi012.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-23T22:49:08.965 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-23T22:49:08.965 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-23T22:49:08.965 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.966 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.966 INFO:teuthology.orchestra.run.smithi012.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.966 INFO:teuthology.orchestra.run.smithi012.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-23T22:49:08.966 INFO:teuthology.orchestra.run.smithi012.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-23T22:49:08.966 INFO:teuthology.orchestra.run.smithi012.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-23T22:49:08.966 INFO:teuthology.orchestra.run.smithi012.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-23T22:49:08.966 INFO:teuthology.orchestra.run.smithi012.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-23T22:49:08.966 INFO:teuthology.orchestra.run.smithi012.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.966 INFO:teuthology.orchestra.run.smithi012.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.966 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-23T22:49:08.966 INFO:teuthology.orchestra.run.smithi012.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.966 INFO:teuthology.orchestra.run.smithi012.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-23T22:49:08.966 INFO:teuthology.orchestra.run.smithi012.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-23T22:49:08.967 INFO:teuthology.orchestra.run.smithi012.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-23T22:49:08.967 INFO:teuthology.orchestra.run.smithi012.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-23T22:49:08.967 INFO:teuthology.orchestra.run.smithi012.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-23T22:49:08.967 INFO:teuthology.orchestra.run.smithi012.stdout: lua-5.4.4-4.el9.x86_64 2024-03-23T22:49:08.967 INFO:teuthology.orchestra.run.smithi012.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-23T22:49:08.967 INFO:teuthology.orchestra.run.smithi012.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-23T22:49:08.967 INFO:teuthology.orchestra.run.smithi012.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-23T22:49:08.967 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-23T22:49:08.967 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-23T22:49:08.967 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-23T22:49:08.967 INFO:teuthology.orchestra.run.smithi012.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-23T22:49:08.967 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-13.11-1.el9.x86_64 2024-03-23T22:49:08.967 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-23T22:49:08.967 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-23T22:49:08.967 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-23T22:49:08.967 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-23T22:49:08.968 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-23T22:49:08.968 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-23T22:49:08.968 INFO:teuthology.orchestra.run.smithi012.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-23T22:49:08.968 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-23T22:49:08.968 INFO:teuthology.orchestra.run.smithi012.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-23T22:49:08.968 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-23T22:49:08.968 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.968 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.968 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.968 INFO:teuthology.orchestra.run.smithi012.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-23T22:49:08.968 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-23T22:49:08.968 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-23T22:49:08.968 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-23T22:49:08.968 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-23T22:49:08.968 INFO:teuthology.orchestra.run.smithi012.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-23T22:49:08.969 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-23T22:49:08.969 INFO:teuthology.orchestra.run.smithi012.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-23T22:49:08.969 INFO:teuthology.orchestra.run.smithi012.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-23T22:49:08.969 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-23T22:49:08.969 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-23T22:49:08.969 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-23T22:49:08.969 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-23T22:49:08.969 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-23T22:49:08.969 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-23T22:49:08.969 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-23T22:49:08.969 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-23T22:49:08.969 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-23T22:49:08.969 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-23T22:49:08.969 INFO:teuthology.orchestra.run.smithi012.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-23T22:49:08.969 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-23T22:49:08.970 INFO:teuthology.orchestra.run.smithi012.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-23T22:49:08.970 INFO:teuthology.orchestra.run.smithi012.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-23T22:49:08.970 INFO:teuthology.orchestra.run.smithi012.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-23T22:49:08.970 INFO:teuthology.orchestra.run.smithi012.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-23T22:49:08.970 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-23T22:49:08.970 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-23T22:49:08.970 INFO:teuthology.orchestra.run.smithi012.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-23T22:49:08.970 INFO:teuthology.orchestra.run.smithi012.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-23T22:49:08.970 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-23T22:49:08.970 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ply-3.11-14.el9.noarch 2024-03-23T22:49:08.970 INFO:teuthology.orchestra.run.smithi012.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-23T22:49:08.970 INFO:teuthology.orchestra.run.smithi012.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-23T22:49:08.970 INFO:teuthology.orchestra.run.smithi012.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-23T22:49:08.970 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-23T22:49:08.971 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-23T22:49:08.971 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-23T22:49:08.971 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-23T22:49:08.971 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-23T22:49:08.971 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-23T22:49:08.971 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.971 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.971 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-23T22:49:08.971 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-23T22:49:08.971 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-23T22:49:08.971 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.971 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-23T22:49:08.971 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-23T22:49:08.971 INFO:teuthology.orchestra.run.smithi012.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-23T22:49:08.972 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-23T22:49:08.972 INFO:teuthology.orchestra.run.smithi012.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-23T22:49:08.972 INFO:teuthology.orchestra.run.smithi012.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-23T22:49:08.972 INFO:teuthology.orchestra.run.smithi012.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-23T22:49:08.972 INFO:teuthology.orchestra.run.smithi012.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-23T22:49:08.972 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-23T22:49:08.972 INFO:teuthology.orchestra.run.smithi012.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-23T22:49:08.972 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-23T22:49:08.972 INFO:teuthology.orchestra.run.smithi012.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-23T22:49:08.972 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-23T22:49:08.972 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-23T22:49:08.972 INFO:teuthology.orchestra.run.smithi012.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-23T22:49:08.972 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.972 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.972 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-23T22:49:08.973 INFO:teuthology.orchestra.run.smithi012.stdout: re2-1:20211101-3.el9.x86_64 2024-03-23T22:49:08.973 INFO:teuthology.orchestra.run.smithi012.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-23T22:49:08.973 INFO:teuthology.orchestra.run.smithi012.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-23T22:49:08.973 INFO:teuthology.orchestra.run.smithi012.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-23T22:49:08.973 INFO:teuthology.orchestra.run.smithi012.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-23T22:49:08.973 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-23T22:49:08.973 INFO:teuthology.orchestra.run.smithi012.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-23T22:49:08.973 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:49:08.973 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-23T22:49:09.677 DEBUG:teuthology.parallel:result is None 2024-03-23T22:49:09.678 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-23T22:49:09.906 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-23T22:49:09.962 INFO:teuthology.orchestra.run.smithi012.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-23T22:49:09.963 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-23T22:49:09.963 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-23T22:49:09.965 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-23T22:49:10.104 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-23T22:49:10.129 INFO:teuthology.orchestra.run.smithi173.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-23T22:49:10.129 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-23T22:49:10.130 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-23T22:49:10.131 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-23T22:49:10.346 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-23T22:49:10.372 INFO:teuthology.orchestra.run.smithi203.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-23T22:49:10.372 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-23T22:49:10.372 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-23T22:49:10.374 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-23T22:49:10.374 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-23T22:49:10.374 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-23T22:49:10.465 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T22:49:10.465 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-23T22:49:10.500 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-23T22:49:10.500 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-23T22:49:10.535 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-23T22:49:10.535 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-23T22:49:10.535 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/daemon-helper 2024-03-23T22:49:10.636 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-23T22:49:10.712 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T22:49:10.712 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2024-03-23T22:49:10.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-23T22:49:10.815 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-23T22:49:10.815 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/daemon-helper 2024-03-23T22:49:10.847 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-23T22:49:10.915 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-23T22:49:10.916 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-23T22:49:10.916 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-23T22:49:10.999 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-23T22:49:11.124 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T22:49:11.125 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-23T22:49:11.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-23T22:49:11.220 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-23T22:49:11.221 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-23T22:49:11.247 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-23T22:49:11.315 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-23T22:49:11.315 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-23T22:49:11.315 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/stdin-killer 2024-03-23T22:49:11.403 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-23T22:49:11.517 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T22:49:11.517 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/stdin-killer 2024-03-23T22:49:11.545 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-23T22:49:11.612 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-23T22:49:11.612 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/stdin-killer 2024-03-23T22:49:11.639 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-23T22:49:11.706 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-23T22:49:11.803 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-23T22:49:11.803 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-23T22:49:11.804 INFO:tasks.cephadm:Cluster fsid is 90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T22:49:11.804 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-23T22:49:11.804 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.12', 'mon.b': '172.21.15.173', 'mon.c': '172.21.15.203'} 2024-03-23T22:49:11.804 INFO:tasks.cephadm:First mon is mon.a on smithi012 2024-03-23T22:49:11.804 INFO:tasks.cephadm:First mgr is x 2024-03-23T22:49:11.804 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-23T22:49:11.804 DEBUG:teuthology.orchestra.run.smithi012:> sudo hostname $(hostname -s) 2024-03-23T22:49:11.886 DEBUG:teuthology.orchestra.run.smithi173:> sudo hostname $(hostname -s) 2024-03-23T22:49:11.913 DEBUG:teuthology.orchestra.run.smithi203:> sudo hostname $(hostname -s) 2024-03-23T22:49:11.940 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-23T22:49:11.940 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-23T22:49:12.081 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77877/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2145-g3be9a5c9', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2145.g3be9a5c9'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-22 05:51:42.267384', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-mchangir-testing-main-20240318.032620', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-23T22:49:12.294 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-mchangir-testing-main-20240318.032620, sha1 3be9a5c9bc793e11e2800a8c0c696e8b46742033 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-23T22:49:12.296 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm 2024-03-23T22:49:12.296 DEBUG:teuthology.orchestra.run.smithi012:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-23T22:49:12.780 INFO:teuthology.orchestra.run.smithi012.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 23 22:49 /home/ubuntu/cephtest/cephadm 2024-03-23T22:49:12.780 DEBUG:teuthology.orchestra.run.smithi173:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-23T22:49:13.384 INFO:teuthology.orchestra.run.smithi173.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 23 22:49 /home/ubuntu/cephtest/cephadm 2024-03-23T22:49:13.384 DEBUG:teuthology.orchestra.run.smithi203:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-23T22:49:13.975 INFO:teuthology.orchestra.run.smithi203.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 23 22:49 /home/ubuntu/cephtest/cephadm 2024-03-23T22:49:13.975 DEBUG:teuthology.orchestra.run.smithi012:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-23T22:49:14.026 DEBUG:teuthology.orchestra.run.smithi173:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-23T22:49:14.045 DEBUG:teuthology.orchestra.run.smithi203:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-23T22:49:14.079 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 on all hosts... 2024-03-23T22:49:14.079 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-23T22:49:14.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-23T22:49:14.089 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-23T22:49:14.853 INFO:teuthology.orchestra.run.smithi173.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-23T22:49:14.906 INFO:teuthology.orchestra.run.smithi203.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-23T22:49:15.695 INFO:teuthology.orchestra.run.smithi012.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-23T22:49:42.325 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-23T22:49:42.325 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-23T22:49:42.325 INFO:teuthology.orchestra.run.smithi203.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-23T22:49:42.325 INFO:teuthology.orchestra.run.smithi203.stdout: "repo_digests": [ 2024-03-23T22:49:42.325 INFO:teuthology.orchestra.run.smithi203.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-23T22:49:42.325 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-23T22:49:42.326 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-23T22:49:46.391 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-23T22:49:46.391 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-23T22:49:46.391 INFO:teuthology.orchestra.run.smithi173.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-23T22:49:46.391 INFO:teuthology.orchestra.run.smithi173.stdout: "repo_digests": [ 2024-03-23T22:49:46.391 INFO:teuthology.orchestra.run.smithi173.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-23T22:49:46.391 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-23T22:49:46.392 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-23T22:49:52.763 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T22:49:52.764 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-23T22:49:52.764 INFO:teuthology.orchestra.run.smithi012.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-23T22:49:52.764 INFO:teuthology.orchestra.run.smithi012.stdout: "repo_digests": [ 2024-03-23T22:49:52.764 INFO:teuthology.orchestra.run.smithi012.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-23T22:49:52.764 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-23T22:49:52.764 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T22:49:52.833 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /etc/ceph 2024-03-23T22:49:52.942 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph 2024-03-23T22:49:52.977 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /etc/ceph 2024-03-23T22:49:53.012 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 777 /etc/ceph 2024-03-23T22:49:53.114 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /etc/ceph 2024-03-23T22:49:53.146 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 777 /etc/ceph 2024-03-23T22:49:53.178 INFO:tasks.cephadm:Writing seed config... 2024-03-23T22:49:53.179 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-23T22:49:53.179 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-23T22:49:53.180 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-23T22:49:53.180 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-23T22:49:53.180 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-23T22:49:53.180 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-23T22:49:53.180 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-23T22:49:53.180 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-23T22:49:53.180 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-23T22:49:53.180 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-23T22:49:53.180 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-23T22:49:53.180 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-23T22:49:53.180 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-23T22:49:53.180 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-23T22:49:53.180 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-23T22:49:53.180 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-23T22:49:53.181 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-23T22:49:53.181 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-23T22:49:53.181 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-23T22:49:53.181 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-23T22:49:53.181 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-23T22:49:53.181 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-23T22:49:53.181 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-23T22:49:53.181 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-23T22:49:53.181 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-23T22:49:53.181 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-23T22:49:53.181 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-23T22:49:53.181 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-23T22:49:53.181 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-23T22:49:53.181 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-23T22:49:53.181 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-23T22:49:53.182 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-23T22:49:53.182 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-23T22:49:53.182 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-23T22:49:53.182 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-23T22:49:53.182 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-23T22:49:53.182 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-23T22:49:53.182 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-23T22:49:53.182 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-23T22:49:53.182 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-23T22:49:53.182 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-23T22:49:53.182 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-23T22:49:53.182 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-23T22:49:53.183 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-23T22:49:53.183 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-23T22:49:53.242 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 90f679ae-e967-11ee-95cd-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-23T22:49:53.243 DEBUG:teuthology.orchestra.run.smithi012:mon.a> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mon.a.service 2024-03-23T22:49:53.290 DEBUG:teuthology.orchestra.run.smithi012:mgr.x> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mgr.x.service 2024-03-23T22:49:53.336 INFO:tasks.cephadm:Bootstrapping... 2024-03-23T22:49:53.336 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 -v bootstrap --fsid 90f679ae-e967-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.12 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-23T22:49:53.625 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-23T22:49:53.625 INFO:teuthology.orchestra.run.smithi012.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033', '-v', 'bootstrap', '--fsid', '90f679ae-e967-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.12', '--skip-admin-label'] 2024-03-23T22:49:53.683 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-23T22:49:53.684 INFO:teuthology.orchestra.run.smithi012.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-23T22:49:53.684 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying podman|docker is present... 2024-03-23T22:49:53.747 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-23T22:49:53.747 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying lvm2 is present... 2024-03-23T22:49:53.748 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying time synchronization is in place... 2024-03-23T22:49:53.766 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-23T22:49:53.766 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-23T22:49:53.790 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-23T22:49:53.790 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-03-23T22:49:53.817 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout enabled 2024-03-23T22:49:53.840 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout active 2024-03-23T22:49:53.840 INFO:teuthology.orchestra.run.smithi012.stdout:Unit chronyd.service is enabled and running 2024-03-23T22:49:53.840 INFO:teuthology.orchestra.run.smithi012.stdout:Repeating the final host check... 2024-03-23T22:49:53.898 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-23T22:49:53.898 INFO:teuthology.orchestra.run.smithi012.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-23T22:49:53.898 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl is present 2024-03-23T22:49:53.898 INFO:teuthology.orchestra.run.smithi012.stdout:lvcreate is present 2024-03-23T22:49:53.911 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-23T22:49:53.911 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-23T22:49:53.928 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-23T22:49:53.928 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-03-23T22:49:53.952 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout enabled 2024-03-23T22:49:53.975 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout active 2024-03-23T22:49:53.975 INFO:teuthology.orchestra.run.smithi012.stdout:Unit chronyd.service is enabled and running 2024-03-23T22:49:53.976 INFO:teuthology.orchestra.run.smithi012.stdout:Host looks OK 2024-03-23T22:49:53.976 INFO:teuthology.orchestra.run.smithi012.stdout:Cluster fsid: 90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T22:49:53.976 INFO:teuthology.orchestra.run.smithi012.stdout:Acquiring lock 140196038790592 on /run/cephadm/90f679ae-e967-11ee-95cd-87774f69a715.lock 2024-03-23T22:49:53.976 INFO:teuthology.orchestra.run.smithi012.stdout:Lock 140196038790592 acquired on /run/cephadm/90f679ae-e967-11ee-95cd-87774f69a715.lock 2024-03-23T22:49:53.978 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying IP 172.21.15.12 port 3300 ... 2024-03-23T22:49:53.979 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying IP 172.21.15.12 port 6789 ... 2024-03-23T22:49:53.979 INFO:teuthology.orchestra.run.smithi012.stdout:Base mon IP(s) is [172.21.15.12:3300, 172.21.15.12:6789], mon addrv is [v2:172.21.15.12:3300,v1:172.21.15.12:6789] 2024-03-23T22:49:53.995 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.12 metric 100 2024-03-23T22:49:53.995 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.12 metric 100 2024-03-23T22:49:54.007 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-23T22:49:54.007 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-23T22:49:54.007 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-23T22:49:54.019 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-23T22:49:54.019 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-23T22:49:54.019 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-23T22:49:54.019 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-23T22:49:54.019 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link noprefixroute 2024-03-23T22:49:54.019 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-23T22:49:54.022 INFO:teuthology.orchestra.run.smithi012.stdout:Mon IP `172.21.15.12` is in CIDR network `172.21.0.0/20` 2024-03-23T22:49:54.022 INFO:teuthology.orchestra.run.smithi012.stdout:Mon IP `172.21.15.12` is in CIDR network `172.21.0.0/20` 2024-03-23T22:49:54.022 INFO:teuthology.orchestra.run.smithi012.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-23T22:49:54.025 INFO:teuthology.orchestra.run.smithi012.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-23T22:49:54.027 INFO:teuthology.orchestra.run.smithi012.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-23T22:49:54.761 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stdout c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-23T22:49:54.761 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-23T22:49:54.762 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Getting image source signatures 2024-03-23T22:49:54.762 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Copying blob sha256:4dd83c6ee8921fe17396d77c31e205985ae94c87d3f752465b5f04aaf9ae340a 2024-03-23T22:49:54.762 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-23T22:49:54.762 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Copying config sha256:c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-23T22:49:54.762 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-23T22:49:55.959 INFO:teuthology.orchestra.run.smithi012.stdout:ceph: stdout ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-23T22:49:55.959 INFO:teuthology.orchestra.run.smithi012.stdout:Ceph version: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-23T22:49:55.960 INFO:teuthology.orchestra.run.smithi012.stdout:Extracting ceph user uid/gid from container image... 2024-03-23T22:50:06.570 INFO:teuthology.orchestra.run.smithi012.stdout:stat: stdout 167 167 2024-03-23T22:50:06.570 INFO:teuthology.orchestra.run.smithi012.stdout:Creating initial keys... 2024-03-23T22:50:07.704 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph-authtool: stdout AQCfXP9lql5MARAASdCLLRBerTmE/zXFLo9PZw== 2024-03-23T22:50:08.789 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph-authtool: stdout AQCgXP9l/pBZChAA12bVssfnyqBLmFiUFV9GNw== 2024-03-23T22:50:09.886 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph-authtool: stdout AQChXP9lveslEBAA6ugkviMCkjYZEa7Jg7i1Zg== 2024-03-23T22:50:09.889 INFO:teuthology.orchestra.run.smithi012.stdout:Creating initial monmap... 2024-03-23T22:50:11.375 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-23T22:50:11.375 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-23T22:50:11.375 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T22:50:11.375 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-23T22:50:11.376 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool for a [v2:172.21.15.12:3300,v1:172.21.15.12:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-23T22:50:11.376 INFO:teuthology.orchestra.run.smithi012.stdout:setting min_mon_release = quincy 2024-03-23T22:50:11.376 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: set fsid to 90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T22:50:11.376 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-23T22:50:11.376 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:50:11.376 INFO:teuthology.orchestra.run.smithi012.stdout:Creating mon... 2024-03-23T22:50:12.537 INFO:teuthology.orchestra.run.smithi012.stdout:create mon.a on 2024-03-23T22:50:12.913 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-23T22:50:13.157 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-23T22:50:13.492 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-90f679ae-e967-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-90f679ae-e967-11ee-95cd-87774f69a715.target. 2024-03-23T22:50:13.492 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-90f679ae-e967-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-90f679ae-e967-11ee-95cd-87774f69a715.target. 2024-03-23T22:50:13.874 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-90f679ae-e967-11ee-95cd-87774f69a715@mon.a 2024-03-23T22:50:13.875 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Failed to reset failed state of unit ceph-90f679ae-e967-11ee-95cd-87774f69a715@mon.a.service: Unit ceph-90f679ae-e967-11ee-95cd-87774f69a715@mon.a.service not loaded. 2024-03-23T22:50:14.178 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-90f679ae-e967-11ee-95cd-87774f69a715.target.wants/ceph-90f679ae-e967-11ee-95cd-87774f69a715@mon.a.service → /etc/systemd/system/ceph-90f679ae-e967-11ee-95cd-87774f69a715@.service. 2024-03-23T22:50:14.819 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:14 smithi012 podman[31304]: 2024-03-23 22:50:14.818300248 +0000 UTC m=+0.379674378 container init 683de88c30f5c4152802cefc73401e31035d832ae37f37fe8cb242ba90dfaa51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-23T22:50:14.944 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-23T22:50:14.944 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout disabled 2024-03-23T22:50:14.961 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-23T22:50:14.961 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-03-23T22:50:14.961 INFO:teuthology.orchestra.run.smithi012.stdout:firewalld.service is not enabled 2024-03-23T22:50:14.961 INFO:teuthology.orchestra.run.smithi012.stdout:Not possible to enable service . firewalld.service is not available 2024-03-23T22:50:14.961 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mon to start... 2024-03-23T22:50:14.961 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mon... 2024-03-23T22:50:15.113 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:14 smithi012 podman[31304]: 2024-03-23 22:50:14.829034597 +0000 UTC m=+0.390408696 container start 683de88c30f5c4152802cefc73401e31035d832ae37f37fe8cb242ba90dfaa51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-23T22:50:15.113 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:14 smithi012 ceph-mon[31323]: mkfs 90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T22:50:15.113 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:14 smithi012 ceph-mon[31323]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-23T22:50:15.114 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:14 smithi012 bash[31304]: 683de88c30f5c4152802cefc73401e31035d832ae37f37fe8cb242ba90dfaa51 2024-03-23T22:50:15.114 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:14 smithi012 systemd[1]: Started Ceph mon.a for 90f679ae-e967-11ee-95cd-87774f69a715. 2024-03-23T22:50:16.023 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:15 smithi012 ceph-mon[31323]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-23T22:50:16.023 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:15 smithi012 ceph-mon[31323]: monmap e1: 1 mons at {a=[v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-23T22:50:16.023 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:15 smithi012 ceph-mon[31323]: fsmap 2024-03-23T22:50:16.023 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:15 smithi012 ceph-mon[31323]: osdmap e1: 0 total, 0 up, 0 in 2024-03-23T22:50:16.023 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:15 smithi012 ceph-mon[31323]: mgrmap e1: no daemons active 2024-03-23T22:50:16.023 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:15 smithi012 ceph-mon[31323]: from='client.? 172.21.15.12:0/3237905736' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-23T22:50:16.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout cluster: 2024-03-23T22:50:16.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout id: 90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T22:50:16.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-23T22:50:16.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-23T22:50:16.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout services: 2024-03-23T22:50:16.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.764212s) 2024-03-23T22:50:16.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-23T22:50:16.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-23T22:50:16.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-23T22:50:16.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout data: 2024-03-23T22:50:16.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-23T22:50:16.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-23T22:50:16.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-23T22:50:16.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout pgs: 2024-03-23T22:50:16.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-23T22:50:16.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.575+0000 7f4dda800700 1 Processor -- start 2024-03-23T22:50:16.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.575+0000 7f4dda800700 1 -- start start 2024-03-23T22:50:16.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.576+0000 7f4dda800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dd410e0f0 0x7f4dd410e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:16.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.576+0000 7f4dda800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd410eae0 con 0x7f4dd410e0f0 2024-03-23T22:50:16.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.577+0000 7f4dd3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dd410e0f0 0x7f4dd410e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:16.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.577+0000 7f4dd3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dd410e0f0 0x7f4dd410e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52052/0 (socket says 172.21.15.12:52052) 2024-03-23T22:50:16.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.577+0000 7f4dd3400700 1 -- 172.21.15.12:0/2034846779 learned_addr learned my addr 172.21.15.12:0/2034846779 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:16.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.577+0000 7f4dd3400700 1 -- 172.21.15.12:0/2034846779 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd410f1e0 con 0x7f4dd410e0f0 2024-03-23T22:50:16.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.577+0000 7f4dd3400700 1 --2- 172.21.15.12:0/2034846779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dd410e0f0 0x7f4dd410e510 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4dc40098a0 tx=0x7f4dc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1817578d688d11c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:16.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.578+0000 7f4dd2000700 1 -- 172.21.15.12:0/2034846779 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4dc4004160 con 0x7f4dd410e0f0 2024-03-23T22:50:16.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.578+0000 7f4dd2000700 1 -- 172.21.15.12:0/2034846779 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc4004300 con 0x7f4dd410e0f0 2024-03-23T22:50:16.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.578+0000 7f4dd2000700 1 -- 172.21.15.12:0/2034846779 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4dc40036a0 con 0x7f4dd410e0f0 2024-03-23T22:50:16.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.578+0000 7f4dda800700 1 -- 172.21.15.12:0/2034846779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dd410e0f0 msgr2=0x7f4dd410e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:16.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.578+0000 7f4dda800700 1 --2- 172.21.15.12:0/2034846779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dd410e0f0 0x7f4dd410e510 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4dc40098a0 tx=0x7f4dc4009bb0 comp rx=0 tx=0).stop 2024-03-23T22:50:16.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.578+0000 7f4dda800700 1 -- 172.21.15.12:0/2034846779 shutdown_connections 2024-03-23T22:50:16.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.578+0000 7f4dda800700 1 --2- 172.21.15.12:0/2034846779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dd410e0f0 0x7f4dd410e510 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:16.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.578+0000 7f4dda800700 1 -- 172.21.15.12:0/2034846779 >> 172.21.15.12:0/2034846779 conn(0x7f4dd407d100 msgr2=0x7f4dd407f520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:16.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.578+0000 7f4dda800700 1 -- 172.21.15.12:0/2034846779 shutdown_connections 2024-03-23T22:50:16.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.579+0000 7f4dda800700 1 -- 172.21.15.12:0/2034846779 wait complete. 2024-03-23T22:50:16.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.580+0000 7f4dda800700 1 Processor -- start 2024-03-23T22:50:16.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.580+0000 7f4dda800700 1 -- start start 2024-03-23T22:50:16.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.580+0000 7f4dda800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dd410e0f0 0x7f4dd41a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:16.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.580+0000 7f4dda800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd410f6b0 con 0x7f4dd410e0f0 2024-03-23T22:50:16.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.581+0000 7f4dd3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dd410e0f0 0x7f4dd41a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:16.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.581+0000 7f4dd3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dd410e0f0 0x7f4dd41a1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52058/0 (socket says 172.21.15.12:52058) 2024-03-23T22:50:16.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.581+0000 7f4dd3400700 1 -- 172.21.15.12:0/3237905736 learned_addr learned my addr 172.21.15.12:0/3237905736 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:16.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.581+0000 7f4dd3400700 1 -- 172.21.15.12:0/3237905736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd41a1bd0 con 0x7f4dd410e0f0 2024-03-23T22:50:16.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.581+0000 7f4dd3400700 1 --2- 172.21.15.12:0/3237905736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dd410e0f0 0x7f4dd41a1690 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4000c00 tx=0x7f4dc4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:16.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.582+0000 7f4dcbe00700 1 -- 172.21.15.12:0/3237905736 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4dc4003db0 con 0x7f4dd410e0f0 2024-03-23T22:50:16.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.582+0000 7f4dcbe00700 1 -- 172.21.15.12:0/3237905736 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc4011630 con 0x7f4dd410e0f0 2024-03-23T22:50:16.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.582+0000 7f4dda800700 1 -- 172.21.15.12:0/3237905736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd41a1e60 con 0x7f4dd410e0f0 2024-03-23T22:50:16.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.582+0000 7f4dcbe00700 1 -- 172.21.15.12:0/3237905736 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4dc4011950 con 0x7f4dd410e0f0 2024-03-23T22:50:16.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.582+0000 7f4dda800700 1 -- 172.21.15.12:0/3237905736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd41a4b50 con 0x7f4dd410e0f0 2024-03-23T22:50:16.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.582+0000 7f4dcbe00700 1 -- 172.21.15.12:0/3237905736 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f4dc4011af0 con 0x7f4dd410e0f0 2024-03-23T22:50:16.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.582+0000 7f4dcbe00700 1 -- 172.21.15.12:0/3237905736 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4dc401c450 con 0x7f4dd410e0f0 2024-03-23T22:50:16.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.583+0000 7f4dda800700 1 -- 172.21.15.12:0/3237905736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd40637c0 con 0x7f4dd410e0f0 2024-03-23T22:50:16.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.586+0000 7f4dcbe00700 1 -- 172.21.15.12:0/3237905736 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f4dc4032020 con 0x7f4dd410e0f0 2024-03-23T22:50:16.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.667+0000 7f4dda800700 1 -- 172.21.15.12:0/3237905736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f4dd402dc30 con 0x7f4dd410e0f0 2024-03-23T22:50:16.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.668+0000 7f4dcbe00700 1 -- 172.21.15.12:0/3237905736 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f4dc4004260 con 0x7f4dd410e0f0 2024-03-23T22:50:16.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.669+0000 7f4dda800700 1 -- 172.21.15.12:0/3237905736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dd410e0f0 msgr2=0x7f4dd41a1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:16.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.669+0000 7f4dda800700 1 --2- 172.21.15.12:0/3237905736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dd410e0f0 0x7f4dd41a1690 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4000c00 tx=0x7f4dc4004750 comp rx=0 tx=0).stop 2024-03-23T22:50:16.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.670+0000 7f4dda800700 1 -- 172.21.15.12:0/3237905736 shutdown_connections 2024-03-23T22:50:16.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.670+0000 7f4dda800700 1 --2- 172.21.15.12:0/3237905736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dd410e0f0 0x7f4dd41a1690 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:16.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.670+0000 7f4dda800700 1 -- 172.21.15.12:0/3237905736 >> 172.21.15.12:0/3237905736 conn(0x7f4dd407d100 msgr2=0x7f4dd4198330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:16.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.670+0000 7f4dda800700 1 -- 172.21.15.12:0/3237905736 shutdown_connections 2024-03-23T22:50:16.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:15.670+0000 7f4dda800700 1 -- 172.21.15.12:0/3237905736 wait complete. 2024-03-23T22:50:16.328 INFO:teuthology.orchestra.run.smithi012.stdout:mon is available 2024-03-23T22:50:16.328 INFO:teuthology.orchestra.run.smithi012.stdout:Assimilating anything we can from ceph.conf... 2024-03-23T22:50:17.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:17 smithi012 ceph-mon[31323]: from='client.? 172.21.15.12:0/2366239467' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-23T22:50:17.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:17 smithi012 ceph-mon[31323]: from='client.? 172.21.15.12:0/2366239467' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-23T22:50:17.642 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-23T22:50:17.642 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [global] 2024-03-23T22:50:17.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout fsid = 90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T22:50:17.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.12:3300,v1:172.21.15.12:6789] 2024-03-23T22:50:17.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-23T22:50:17.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-23T22:50:17.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-23T22:50:17.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-23T22:50:17.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-23T22:50:17.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-23T22:50:17.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-23T22:50:17.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-23T22:50:17.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [osd] 2024-03-23T22:50:17.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-23T22:50:17.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-23T22:50:17.644 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-23T22:50:17.644 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-23T22:50:17.644 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-23T22:50:17.644 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.919+0000 7fe767600700 1 Processor -- start 2024-03-23T22:50:17.644 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.920+0000 7fe767600700 1 -- start start 2024-03-23T22:50:17.644 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.920+0000 7fe767600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe76007b2c0 0x7fe76007b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:17.644 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.920+0000 7fe767600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76007cff0 con 0x7fe76007b2c0 2024-03-23T22:50:17.644 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.921+0000 7fe764c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe76007b2c0 0x7fe76007b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:17.644 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.921+0000 7fe764c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe76007b2c0 0x7fe76007b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52074/0 (socket says 172.21.15.12:52074) 2024-03-23T22:50:17.644 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.921+0000 7fe764c00700 1 -- 172.21.15.12:0/333887039 learned_addr learned my addr 172.21.15.12:0/333887039 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:17.644 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.921+0000 7fe764c00700 1 -- 172.21.15.12:0/333887039 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76007bc20 con 0x7fe76007b2c0 2024-03-23T22:50:17.644 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.922+0000 7fe764c00700 1 --2- 172.21.15.12:0/333887039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe76007b2c0 0x7fe76007b6e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe7500098a0 tx=0x7fe750009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=852345d888f24e31 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:17.644 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.922+0000 7fe75fe00700 1 -- 172.21.15.12:0/333887039 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe750004160 con 0x7fe76007b2c0 2024-03-23T22:50:17.644 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.922+0000 7fe75fe00700 1 -- 172.21.15.12:0/333887039 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe750004300 con 0x7fe76007b2c0 2024-03-23T22:50:17.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.922+0000 7fe75fe00700 1 -- 172.21.15.12:0/333887039 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe7500036a0 con 0x7fe76007b2c0 2024-03-23T22:50:17.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.923+0000 7fe767600700 1 -- 172.21.15.12:0/333887039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe76007b2c0 msgr2=0x7fe76007b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:17.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.923+0000 7fe767600700 1 --2- 172.21.15.12:0/333887039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe76007b2c0 0x7fe76007b6e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe7500098a0 tx=0x7fe750009bb0 comp rx=0 tx=0).stop 2024-03-23T22:50:17.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.923+0000 7fe767600700 1 -- 172.21.15.12:0/333887039 shutdown_connections 2024-03-23T22:50:17.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.923+0000 7fe767600700 1 --2- 172.21.15.12:0/333887039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe76007b2c0 0x7fe76007b6e0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:17.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.923+0000 7fe767600700 1 -- 172.21.15.12:0/333887039 >> 172.21.15.12:0/333887039 conn(0x7fe760103000 msgr2=0x7fe760105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:17.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.923+0000 7fe767600700 1 -- 172.21.15.12:0/333887039 shutdown_connections 2024-03-23T22:50:17.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.923+0000 7fe767600700 1 -- 172.21.15.12:0/333887039 wait complete. 2024-03-23T22:50:17.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.924+0000 7fe767600700 1 Processor -- start 2024-03-23T22:50:17.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.924+0000 7fe767600700 1 -- start start 2024-03-23T22:50:17.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.925+0000 7fe767600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe76007b2c0 0x7fe76019d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:17.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.925+0000 7fe767600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76007c150 con 0x7fe76007b2c0 2024-03-23T22:50:17.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.925+0000 7fe764c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe76007b2c0 0x7fe76019d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:17.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.925+0000 7fe764c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe76007b2c0 0x7fe76019d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52088/0 (socket says 172.21.15.12:52088) 2024-03-23T22:50:17.646 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.925+0000 7fe764c00700 1 -- 172.21.15.12:0/2366239467 learned_addr learned my addr 172.21.15.12:0/2366239467 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:17.646 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.925+0000 7fe764c00700 1 -- 172.21.15.12:0/2366239467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76019d950 con 0x7fe76007b2c0 2024-03-23T22:50:17.646 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.926+0000 7fe764c00700 1 --2- 172.21.15.12:0/2366239467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe76007b2c0 0x7fe76019d410 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe750000c00 tx=0x7fe750004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:17.646 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.926+0000 7fe75e000700 1 -- 172.21.15.12:0/2366239467 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe750003db0 con 0x7fe76007b2c0 2024-03-23T22:50:17.646 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.926+0000 7fe75e000700 1 -- 172.21.15.12:0/2366239467 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe750011630 con 0x7fe76007b2c0 2024-03-23T22:50:17.646 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.926+0000 7fe75e000700 1 -- 172.21.15.12:0/2366239467 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe750011950 con 0x7fe76007b2c0 2024-03-23T22:50:17.646 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.926+0000 7fe767600700 1 -- 172.21.15.12:0/2366239467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe76019dbe0 con 0x7fe76007b2c0 2024-03-23T22:50:17.646 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.926+0000 7fe767600700 1 -- 172.21.15.12:0/2366239467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7601a08d0 con 0x7fe76007b2c0 2024-03-23T22:50:17.646 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.927+0000 7fe75e000700 1 -- 172.21.15.12:0/2366239467 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe75001a460 con 0x7fe76007b2c0 2024-03-23T22:50:17.646 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.927+0000 7fe75e000700 1 -- 172.21.15.12:0/2366239467 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe75001b780 con 0x7fe76007b2c0 2024-03-23T22:50:17.647 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.928+0000 7fe767600700 1 -- 172.21.15.12:0/2366239467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76010d820 con 0x7fe76007b2c0 2024-03-23T22:50:17.647 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:16.931+0000 7fe75e000700 1 -- 172.21.15.12:0/2366239467 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fe75001b980 con 0x7fe76007b2c0 2024-03-23T22:50:17.647 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:17.008+0000 7fe767600700 1 -- 172.21.15.12:0/2366239467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fe7600637c0 con 0x7fe76007b2c0 2024-03-23T22:50:17.647 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:17.018+0000 7fe75e000700 1 -- 172.21.15.12:0/2366239467 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fe75001bb60 con 0x7fe76007b2c0 2024-03-23T22:50:17.647 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:17.019+0000 7fe75e000700 1 -- 172.21.15.12:0/2366239467 <== mon.0 v2:172.21.15.12:3300/0 8 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe75001ada0 con 0x7fe76007b2c0 2024-03-23T22:50:17.647 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:17.022+0000 7fe767600700 1 -- 172.21.15.12:0/2366239467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe76007b2c0 msgr2=0x7fe76019d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:17.647 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:17.022+0000 7fe767600700 1 --2- 172.21.15.12:0/2366239467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe76007b2c0 0x7fe76019d410 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe750000c00 tx=0x7fe750004750 comp rx=0 tx=0).stop 2024-03-23T22:50:17.647 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:17.022+0000 7fe767600700 1 -- 172.21.15.12:0/2366239467 shutdown_connections 2024-03-23T22:50:17.647 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:17.022+0000 7fe767600700 1 --2- 172.21.15.12:0/2366239467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe76007b2c0 0x7fe76019d410 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:17.647 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:17.022+0000 7fe767600700 1 -- 172.21.15.12:0/2366239467 >> 172.21.15.12:0/2366239467 conn(0x7fe760103000 msgr2=0x7fe7601052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:17.647 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:17.022+0000 7fe767600700 1 -- 172.21.15.12:0/2366239467 shutdown_connections 2024-03-23T22:50:17.647 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:17.022+0000 7fe767600700 1 -- 172.21.15.12:0/2366239467 wait complete. 2024-03-23T22:50:17.647 INFO:teuthology.orchestra.run.smithi012.stdout:Generating new minimal ceph.conf... 2024-03-23T22:50:18.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:18 smithi012 ceph-mon[31323]: from='client.? 172.21.15.12:0/3868551372' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:50:19.042 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.253+0000 7fe324a00700 1 Processor -- start 2024-03-23T22:50:19.042 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.254+0000 7fe324a00700 1 -- start start 2024-03-23T22:50:19.042 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.254+0000 7fe324a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe320109c80 0x7fe32010a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:19.042 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.254+0000 7fe324a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32010a670 con 0x7fe320109c80 2024-03-23T22:50:19.042 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.255+0000 7fe31de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe320109c80 0x7fe32010a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:19.042 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.255+0000 7fe31de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe320109c80 0x7fe32010a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52090/0 (socket says 172.21.15.12:52090) 2024-03-23T22:50:19.042 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.255+0000 7fe31de00700 1 -- 172.21.15.12:0/1551558374 learned_addr learned my addr 172.21.15.12:0/1551558374 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:19.043 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.255+0000 7fe31de00700 1 -- 172.21.15.12:0/1551558374 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32010ade0 con 0x7fe320109c80 2024-03-23T22:50:19.043 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.256+0000 7fe31de00700 1 --2- 172.21.15.12:0/1551558374 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe320109c80 0x7fe32010a0a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe3100098a0 tx=0x7fe310009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e8b7b8074853819 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:19.043 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.256+0000 7fe31ca00700 1 -- 172.21.15.12:0/1551558374 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe310004160 con 0x7fe320109c80 2024-03-23T22:50:19.043 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.256+0000 7fe31ca00700 1 -- 172.21.15.12:0/1551558374 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe310004300 con 0x7fe320109c80 2024-03-23T22:50:19.043 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.256+0000 7fe31ca00700 1 -- 172.21.15.12:0/1551558374 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe310011b50 con 0x7fe320109c80 2024-03-23T22:50:19.043 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.256+0000 7fe324a00700 1 -- 172.21.15.12:0/1551558374 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe320109c80 msgr2=0x7fe32010a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:19.043 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.256+0000 7fe324a00700 1 --2- 172.21.15.12:0/1551558374 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe320109c80 0x7fe32010a0a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe3100098a0 tx=0x7fe310009bb0 comp rx=0 tx=0).stop 2024-03-23T22:50:19.043 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.257+0000 7fe324a00700 1 -- 172.21.15.12:0/1551558374 shutdown_connections 2024-03-23T22:50:19.043 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.257+0000 7fe324a00700 1 --2- 172.21.15.12:0/1551558374 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe320109c80 0x7fe32010a0a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:19.043 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.257+0000 7fe324a00700 1 -- 172.21.15.12:0/1551558374 >> 172.21.15.12:0/1551558374 conn(0x7fe3201054a0 msgr2=0x7fe320107890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:19.043 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.257+0000 7fe324a00700 1 -- 172.21.15.12:0/1551558374 shutdown_connections 2024-03-23T22:50:19.043 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.257+0000 7fe324a00700 1 -- 172.21.15.12:0/1551558374 wait complete. 2024-03-23T22:50:19.043 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.258+0000 7fe324a00700 1 Processor -- start 2024-03-23T22:50:19.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.258+0000 7fe324a00700 1 -- start start 2024-03-23T22:50:19.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.258+0000 7fe324a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe320109c80 0x7fe32019dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:19.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.258+0000 7fe324a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32010e190 con 0x7fe320109c80 2024-03-23T22:50:19.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.259+0000 7fe31de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe320109c80 0x7fe32019dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:19.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.259+0000 7fe31de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe320109c80 0x7fe32019dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52092/0 (socket says 172.21.15.12:52092) 2024-03-23T22:50:19.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.259+0000 7fe31de00700 1 -- 172.21.15.12:0/3868551372 learned_addr learned my addr 172.21.15.12:0/3868551372 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:19.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.259+0000 7fe31de00700 1 -- 172.21.15.12:0/3868551372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32019e160 con 0x7fe320109c80 2024-03-23T22:50:19.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.259+0000 7fe31de00700 1 --2- 172.21.15.12:0/3868551372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe320109c80 0x7fe32019dc20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe310011850 tx=0x7fe310011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:19.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.260+0000 7fe316a00700 1 -- 172.21.15.12:0/3868551372 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe310003af0 con 0x7fe320109c80 2024-03-23T22:50:19.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.260+0000 7fe316a00700 1 -- 172.21.15.12:0/3868551372 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe310020070 con 0x7fe320109c80 2024-03-23T22:50:19.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.260+0000 7fe324a00700 1 -- 172.21.15.12:0/3868551372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe32019e3f0 con 0x7fe320109c80 2024-03-23T22:50:19.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.260+0000 7fe316a00700 1 -- 172.21.15.12:0/3868551372 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3100195a0 con 0x7fe320109c80 2024-03-23T22:50:19.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.260+0000 7fe324a00700 1 -- 172.21.15.12:0/3868551372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3201a10e0 con 0x7fe320109c80 2024-03-23T22:50:19.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.260+0000 7fe316a00700 1 -- 172.21.15.12:0/3868551372 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe310019be0 con 0x7fe320109c80 2024-03-23T22:50:19.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.260+0000 7fe316a00700 1 -- 172.21.15.12:0/3868551372 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe310011e00 con 0x7fe320109c80 2024-03-23T22:50:19.045 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.261+0000 7fe324a00700 1 -- 172.21.15.12:0/3868551372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3200637c0 con 0x7fe320109c80 2024-03-23T22:50:19.045 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.264+0000 7fe316a00700 1 -- 172.21.15.12:0/3868551372 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fe310003c90 con 0x7fe320109c80 2024-03-23T22:50:19.045 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.341+0000 7fe324a00700 1 -- 172.21.15.12:0/3868551372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fe3201a12b0 con 0x7fe320109c80 2024-03-23T22:50:19.045 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.342+0000 7fe316a00700 1 -- 172.21.15.12:0/3868551372 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fe310025460 con 0x7fe320109c80 2024-03-23T22:50:19.045 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.344+0000 7fe324a00700 1 -- 172.21.15.12:0/3868551372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe320109c80 msgr2=0x7fe32019dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:19.045 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.344+0000 7fe324a00700 1 --2- 172.21.15.12:0/3868551372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe320109c80 0x7fe32019dc20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe310011850 tx=0x7fe310011c10 comp rx=0 tx=0).stop 2024-03-23T22:50:19.045 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.344+0000 7fe324a00700 1 -- 172.21.15.12:0/3868551372 shutdown_connections 2024-03-23T22:50:19.045 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.344+0000 7fe324a00700 1 --2- 172.21.15.12:0/3868551372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe320109c80 0x7fe32019dc20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:19.045 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.344+0000 7fe324a00700 1 -- 172.21.15.12:0/3868551372 >> 172.21.15.12:0/3868551372 conn(0x7fe3201054a0 msgr2=0x7fe320107000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:19.045 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.344+0000 7fe324a00700 1 -- 172.21.15.12:0/3868551372 shutdown_connections 2024-03-23T22:50:19.045 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:18.344+0000 7fe324a00700 1 -- 172.21.15.12:0/3868551372 wait complete. 2024-03-23T22:50:19.045 INFO:teuthology.orchestra.run.smithi012.stdout:Restarting the monitor... 2024-03-23T22:50:19.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:19 smithi012 systemd[1]: Stopping Ceph mon.a for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T22:50:19.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:19 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-a[31318]: 2024-03-23T22:50:19.507+0000 7f4a02400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T22:50:19.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:19 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-a[31318]: 2024-03-23T22:50:19.507+0000 7f4a02400700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-23T22:50:19.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:19 smithi012 podman[31568]: 2024-03-23 22:50:19.571246932 +0000 UTC m=+0.177941086 container died 683de88c30f5c4152802cefc73401e31035d832ae37f37fe8cb242ba90dfaa51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-a, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-23T22:50:20.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:19 smithi012 podman[31568]: 2024-03-23 22:50:19.943859066 +0000 UTC m=+0.550553221 container cleanup 683de88c30f5c4152802cefc73401e31035d832ae37f37fe8cb242ba90dfaa51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-23T22:50:20.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:19 smithi012 bash[31568]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-a 2024-03-23T22:50:20.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:20 smithi012 podman[31582]: 2024-03-23 22:50:20.14982929 +0000 UTC m=+0.563889042 container remove 683de88c30f5c4152802cefc73401e31035d832ae37f37fe8cb242ba90dfaa51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-a, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-23T22:50:20.995 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:20 smithi012 systemd[1]: ceph-90f679ae-e967-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-23T22:50:20.995 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:20 smithi012 systemd[1]: Stopped Ceph mon.a for 90f679ae-e967-11ee-95cd-87774f69a715. 2024-03-23T22:50:20.995 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:20 smithi012 systemd[1]: ceph-90f679ae-e967-11ee-95cd-87774f69a715@mon.a.service: Consumed 1.896s CPU time. 2024-03-23T22:50:20.995 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:20 smithi012 systemd[1]: Starting Ceph mon.a for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T22:50:21.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:20 smithi012 podman[31676]: 2024-03-23 22:50:20.902116679 +0000 UTC m=+0.036983108 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-23T22:50:21.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 podman[31676]: 2024-03-23 22:50:21.015960509 +0000 UTC m=+0.150826966 container create bdb2aed276e79f0d904555aa6918d03e2fd780229072949a875ee8254c37765b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319) 2024-03-23T22:50:21.945 INFO:teuthology.orchestra.run.smithi012.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-23T22:50:22.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 podman[31676]: 2024-03-23 22:50:21.799698161 +0000 UTC m=+0.934564621 container init bdb2aed276e79f0d904555aa6918d03e2fd780229072949a875ee8254c37765b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-23T22:50:22.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 podman[31676]: 2024-03-23 22:50:21.810960442 +0000 UTC m=+0.945826869 container start bdb2aed276e79f0d904555aa6918d03e2fd780229072949a875ee8254c37765b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-23T22:50:22.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: set uid:gid to 167:167 (ceph:ceph) 2024-03-23T22:50:22.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev), process ceph-mon, pid 2 2024-03-23T22:50:22.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: pidfile_write: ignore empty --pid-file 2024-03-23T22:50:22.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: load: jerasure load: lrc 2024-03-23T22:50:22.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: RocksDB version: 7.9.2 2024-03-23T22:50:22.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Git sha 0 2024-03-23T22:50:22.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Compile date 2024-03-22 04:58:55 2024-03-23T22:50:22.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: DB SUMMARY 2024-03-23T22:50:22.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: DB Session ID: VG2A8UHDC12QUX7BMSIO 2024-03-23T22:50:22.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: CURRENT file: CURRENT 2024-03-23T22:50:22.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: IDENTITY file: IDENTITY 2024-03-23T22:50:22.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-23T22:50:22.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-23T22:50:22.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 99600 ; 2024-03-23T22:50:22.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.error_if_exists: 0 2024-03-23T22:50:22.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.create_if_missing: 0 2024-03-23T22:50:22.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.paranoid_checks: 1 2024-03-23T22:50:22.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-23T22:50:22.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-23T22:50:22.057 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-23T22:50:22.057 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.env: 0x562478b6eb20 2024-03-23T22:50:22.057 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.fs: PosixFileSystem 2024-03-23T22:50:22.057 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.info_log: 0x562479c8d960 2024-03-23T22:50:22.057 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_file_opening_threads: 16 2024-03-23T22:50:22.058 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.statistics: (nil) 2024-03-23T22:50:22.058 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.use_fsync: 0 2024-03-23T22:50:22.058 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_log_file_size: 0 2024-03-23T22:50:22.058 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-23T22:50:22.058 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-23T22:50:22.058 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.keep_log_file_num: 1000 2024-03-23T22:50:22.058 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.recycle_log_file_num: 0 2024-03-23T22:50:22.058 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.allow_fallocate: 1 2024-03-23T22:50:22.059 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.allow_mmap_reads: 0 2024-03-23T22:50:22.059 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.allow_mmap_writes: 0 2024-03-23T22:50:22.059 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.use_direct_reads: 0 2024-03-23T22:50:22.059 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-23T22:50:22.059 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.create_missing_column_families: 0 2024-03-23T22:50:22.059 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.db_log_dir: 2024-03-23T22:50:22.059 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.wal_dir: 2024-03-23T22:50:22.059 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-23T22:50:22.059 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-23T22:50:22.059 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-23T22:50:22.059 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-23T22:50:22.059 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-23T22:50:22.060 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-23T22:50:22.060 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.advise_random_on_open: 1 2024-03-23T22:50:22.060 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.db_write_buffer_size: 0 2024-03-23T22:50:22.060 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.write_buffer_manager: 0x562479c601e0 2024-03-23T22:50:22.060 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-23T22:50:22.060 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-23T22:50:22.060 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-23T22:50:22.060 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.rate_limiter: (nil) 2024-03-23T22:50:22.060 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-23T22:50:22.060 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.wal_recovery_mode: 2 2024-03-23T22:50:22.061 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.enable_thread_tracking: 0 2024-03-23T22:50:22.061 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.enable_pipelined_write: 0 2024-03-23T22:50:22.061 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.unordered_write: 0 2024-03-23T22:50:22.061 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-23T22:50:22.061 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-23T22:50:22.061 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-23T22:50:22.061 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-23T22:50:22.061 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.row_cache: None 2024-03-23T22:50:22.061 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.wal_filter: None 2024-03-23T22:50:22.061 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-23T22:50:22.062 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.allow_ingest_behind: 0 2024-03-23T22:50:22.062 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.two_write_queues: 0 2024-03-23T22:50:22.062 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.manual_wal_flush: 0 2024-03-23T22:50:22.062 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.wal_compression: 0 2024-03-23T22:50:22.062 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.atomic_flush: 0 2024-03-23T22:50:22.062 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-23T22:50:22.062 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-23T22:50:22.062 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-23T22:50:22.062 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.log_readahead_size: 0 2024-03-23T22:50:22.062 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-23T22:50:22.062 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.best_efforts_recovery: 0 2024-03-23T22:50:22.062 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-23T22:50:22.063 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-23T22:50:22.063 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.allow_data_in_errors: 0 2024-03-23T22:50:22.063 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.db_host_id: __hostname__ 2024-03-23T22:50:22.063 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-23T22:50:22.063 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_background_jobs: 2 2024-03-23T22:50:22.063 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_background_compactions: -1 2024-03-23T22:50:22.063 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_subcompactions: 1 2024-03-23T22:50:22.063 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-23T22:50:22.063 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-23T22:50:22.063 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-23T22:50:22.063 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_total_wal_size: 0 2024-03-23T22:50:22.063 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-23T22:50:22.064 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-23T22:50:22.064 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-23T22:50:22.064 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-23T22:50:22.064 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_open_files: -1 2024-03-23T22:50:22.064 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.bytes_per_sync: 0 2024-03-23T22:50:22.064 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-23T22:50:22.064 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-23T22:50:22.064 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.compaction_readahead_size: 0 2024-03-23T22:50:22.064 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_background_flushes: -1 2024-03-23T22:50:22.064 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Compression algorithms supported: 2024-03-23T22:50:22.064 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-23T22:50:22.064 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: kZSTD supported: 0 2024-03-23T22:50:22.064 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: kXpressCompression supported: 0 2024-03-23T22:50:22.065 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: kLZ4HCCompression supported: 1 2024-03-23T22:50:22.065 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: kZlibCompression supported: 1 2024-03-23T22:50:22.065 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: kSnappyCompression supported: 1 2024-03-23T22:50:22.065 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: kLZ4Compression supported: 1 2024-03-23T22:50:22.065 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: kBZip2Compression supported: 0 2024-03-23T22:50:22.065 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-23T22:50:22.065 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-23T22:50:22.065 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-23T22:50:22.065 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-23T22:50:22.065 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-23T22:50:22.065 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.merge_operator: 2024-03-23T22:50:22.066 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.compaction_filter: None 2024-03-23T22:50:22.066 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.compaction_filter_factory: None 2024-03-23T22:50:22.066 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.sst_partitioner_factory: None 2024-03-23T22:50:22.066 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-23T22:50:22.066 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-23T22:50:22.066 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562479c8da60) 2024-03-23T22:50:22.066 INFO:journalctl@ceph.mon.a.smithi012.stdout: cache_index_and_filter_blocks: 1 2024-03-23T22:50:22.066 INFO:journalctl@ceph.mon.a.smithi012.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-23T22:50:22.066 INFO:journalctl@ceph.mon.a.smithi012.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-23T22:50:22.066 INFO:journalctl@ceph.mon.a.smithi012.stdout: pin_top_level_index_and_filter: 1 2024-03-23T22:50:22.066 INFO:journalctl@ceph.mon.a.smithi012.stdout: index_type: 0 2024-03-23T22:50:22.067 INFO:journalctl@ceph.mon.a.smithi012.stdout: data_block_index_type: 0 2024-03-23T22:50:22.067 INFO:journalctl@ceph.mon.a.smithi012.stdout: index_shortening: 1 2024-03-23T22:50:22.067 INFO:journalctl@ceph.mon.a.smithi012.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-23T22:50:22.067 INFO:journalctl@ceph.mon.a.smithi012.stdout: checksum: 4 2024-03-23T22:50:22.067 INFO:journalctl@ceph.mon.a.smithi012.stdout: no_block_cache: 0 2024-03-23T22:50:22.067 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_cache: 0x562479c93090 2024-03-23T22:50:22.067 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_cache_name: BinnedLRUCache 2024-03-23T22:50:22.067 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_cache_options: 2024-03-23T22:50:22.067 INFO:journalctl@ceph.mon.a.smithi012.stdout: capacity : 536870912 2024-03-23T22:50:22.067 INFO:journalctl@ceph.mon.a.smithi012.stdout: num_shard_bits : 4 2024-03-23T22:50:22.067 INFO:journalctl@ceph.mon.a.smithi012.stdout: strict_capacity_limit : 0 2024-03-23T22:50:22.068 INFO:journalctl@ceph.mon.a.smithi012.stdout: high_pri_pool_ratio: 0.000 2024-03-23T22:50:22.068 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_cache_compressed: (nil) 2024-03-23T22:50:22.068 INFO:journalctl@ceph.mon.a.smithi012.stdout: persistent_cache: (nil) 2024-03-23T22:50:22.068 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_size: 4096 2024-03-23T22:50:22.068 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_size_deviation: 10 2024-03-23T22:50:22.068 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_restart_interval: 16 2024-03-23T22:50:22.068 INFO:journalctl@ceph.mon.a.smithi012.stdout: index_block_restart_interval: 1 2024-03-23T22:50:22.068 INFO:journalctl@ceph.mon.a.smithi012.stdout: metadata_block_size: 4096 2024-03-23T22:50:22.068 INFO:journalctl@ceph.mon.a.smithi012.stdout: partition_filters: 0 2024-03-23T22:50:22.068 INFO:journalctl@ceph.mon.a.smithi012.stdout: use_delta_encoding: 1 2024-03-23T22:50:22.068 INFO:journalctl@ceph.mon.a.smithi012.stdout: filter_policy: bloomfilter 2024-03-23T22:50:22.068 INFO:journalctl@ceph.mon.a.smithi012.stdout: whole_key_filtering: 1 2024-03-23T22:50:22.068 INFO:journalctl@ceph.mon.a.smithi012.stdout: verify_compression: 0 2024-03-23T22:50:22.069 INFO:journalctl@ceph.mon.a.smithi012.stdout: read_amp_bytes_per_bit: 0 2024-03-23T22:50:22.069 INFO:journalctl@ceph.mon.a.smithi012.stdout: format_version: 5 2024-03-23T22:50:22.069 INFO:journalctl@ceph.mon.a.smithi012.stdout: enable_index_compression: 1 2024-03-23T22:50:22.069 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_align: 0 2024-03-23T22:50:22.069 INFO:journalctl@ceph.mon.a.smithi012.stdout: max_auto_readahead_size: 262144 2024-03-23T22:50:22.069 INFO:journalctl@ceph.mon.a.smithi012.stdout: prepopulate_block_cache: 0 2024-03-23T22:50:22.069 INFO:journalctl@ceph.mon.a.smithi012.stdout: initial_auto_readahead_size: 8192 2024-03-23T22:50:22.069 INFO:journalctl@ceph.mon.a.smithi012.stdout: num_file_reads_for_auto_readahead: 2 2024-03-23T22:50:22.070 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.write_buffer_size: 33554432 2024-03-23T22:50:22.070 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_write_buffer_number: 2 2024-03-23T22:50:22.070 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.compression: NoCompression 2024-03-23T22:50:22.070 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.bottommost_compression: Disabled 2024-03-23T22:50:22.071 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.prefix_extractor: nullptr 2024-03-23T22:50:22.071 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-23T22:50:22.071 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.num_levels: 7 2024-03-23T22:50:22.071 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-23T22:50:22.071 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-23T22:50:22.071 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-23T22:50:22.071 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-23T22:50:22.071 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-23T22:50:22.071 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-23T22:50:22.071 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-23T22:50:22.071 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-23T22:50:22.071 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-23T22:50:22.071 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-23T22:50:22.072 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-23T22:50:22.072 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-23T22:50:22.072 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-23T22:50:22.072 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.compression_opts.level: 32767 2024-03-23T22:50:22.072 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.compression_opts.strategy: 0 2024-03-23T22:50:22.072 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-23T22:50:22.072 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-23T22:50:22.072 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-23T22:50:22.072 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-23T22:50:22.072 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.compression_opts.enabled: false 2024-03-23T22:50:22.072 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-23T22:50:22.072 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-23T22:50:22.072 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-23T22:50:22.073 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-23T22:50:22.073 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.target_file_size_base: 67108864 2024-03-23T22:50:22.073 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-23T22:50:22.073 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-23T22:50:22.073 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-23T22:50:22.073 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-23T22:50:22.073 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-23T22:50:22.073 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-23T22:50:22.073 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-23T22:50:22.073 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-23T22:50:22.073 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-23T22:50:22.073 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-23T22:50:22.073 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-23T22:50:22.073 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-23T22:50:22.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-23T22:50:22.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-23T22:50:22.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.arena_block_size: 1048576 2024-03-23T22:50:22.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-23T22:50:22.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-23T22:50:22.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.disable_auto_compactions: 0 2024-03-23T22:50:22.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-23T22:50:22.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-23T22:50:22.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-23T22:50:22.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-23T22:50:22.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-23T22:50:22.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-23T22:50:22.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-23T22:50:22.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-23T22:50:22.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-23T22:50:22.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-23T22:50:22.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.table_properties_collectors: 2024-03-23T22:50:22.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.inplace_update_support: 0 2024-03-23T22:50:22.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-23T22:50:22.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-23T22:50:22.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-23T22:50:22.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-23T22:50:22.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.bloom_locality: 0 2024-03-23T22:50:22.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.max_successive_merges: 0 2024-03-23T22:50:22.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-23T22:50:22.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.paranoid_file_checks: 0 2024-03-23T22:50:22.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.force_consistency_checks: 1 2024-03-23T22:50:22.076 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.report_bg_io_stats: 0 2024-03-23T22:50:22.076 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.ttl: 2592000 2024-03-23T22:50:22.076 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-23T22:50:22.076 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-23T22:50:22.076 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-23T22:50:22.076 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.enable_blob_files: false 2024-03-23T22:50:22.076 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.min_blob_size: 0 2024-03-23T22:50:22.076 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.blob_file_size: 268435456 2024-03-23T22:50:22.076 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-23T22:50:22.076 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-23T22:50:22.076 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-23T22:50:22.076 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-23T22:50:22.076 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-23T22:50:22.076 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.blob_file_starting_level: 0 2024-03-23T22:50:22.077 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-23T22:50:22.077 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-23T22:50:22.077 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-23T22:50:22.077 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5cfc8f60-5ce2-445a-b83a-b1bbf127a89a 2024-03-23T22:50:22.077 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711234221872316, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-23T22:50:22.077 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-23T22:50:22.077 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711234221874347, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 94838, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 92831, "index_size": 289, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13660, "raw_average_key_size": 50, "raw_value_size": 85391, "raw_average_value_size": 312, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711234221, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5cfc8f60-5ce2-445a-b83a-b1bbf127a89a", "db_session_id": "VG2A8UHDC12QUX7BMSIO", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-23T22:50:22.077 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711234221874536, "job": 1, "event": "recovery_finished"} 2024-03-23T22:50:22.077 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-23T22:50:22.077 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-23T22:50:22.077 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562479d86000 2024-03-23T22:50:22.077 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: DB pointer 0x562479d70000 2024-03-23T22:50:22.077 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-23T22:50:22.078 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-23T22:50:22.078 INFO:journalctl@ceph.mon.a.smithi012.stdout: ** DB Stats ** 2024-03-23T22:50:22.078 INFO:journalctl@ceph.mon.a.smithi012.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-23T22:50:22.078 INFO:journalctl@ceph.mon.a.smithi012.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-23T22:50:22.078 INFO:journalctl@ceph.mon.a.smithi012.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-23T22:50:22.078 INFO:journalctl@ceph.mon.a.smithi012.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-23T22:50:22.078 INFO:journalctl@ceph.mon.a.smithi012.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-23T22:50:22.078 INFO:journalctl@ceph.mon.a.smithi012.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-23T22:50:22.078 INFO:journalctl@ceph.mon.a.smithi012.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-23T22:50:22.078 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-03-23T22:50:22.078 INFO:journalctl@ceph.mon.a.smithi012.stdout: ** Compaction Stats [default] ** 2024-03-23T22:50:22.078 INFO:journalctl@ceph.mon.a.smithi012.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-23T22:50:22.079 INFO:journalctl@ceph.mon.a.smithi012.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-23T22:50:22.079 INFO:journalctl@ceph.mon.a.smithi012.stdout: L0 2/0 94.45 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 67.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-23T22:50:22.079 INFO:journalctl@ceph.mon.a.smithi012.stdout: Sum 2/0 94.45 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 67.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-23T22:50:22.079 INFO:journalctl@ceph.mon.a.smithi012.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 67.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-23T22:50:22.079 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-03-23T22:50:22.079 INFO:journalctl@ceph.mon.a.smithi012.stdout: ** Compaction Stats [default] ** 2024-03-23T22:50:22.079 INFO:journalctl@ceph.mon.a.smithi012.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-23T22:50:22.079 INFO:journalctl@ceph.mon.a.smithi012.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-23T22:50:22.079 INFO:journalctl@ceph.mon.a.smithi012.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 67.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-23T22:50:22.079 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-03-23T22:50:22.079 INFO:journalctl@ceph.mon.a.smithi012.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-23T22:50:22.079 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-03-23T22:50:22.079 INFO:journalctl@ceph.mon.a.smithi012.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-23T22:50:22.080 INFO:journalctl@ceph.mon.a.smithi012.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-23T22:50:22.080 INFO:journalctl@ceph.mon.a.smithi012.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-23T22:50:22.080 INFO:journalctl@ceph.mon.a.smithi012.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-23T22:50:22.080 INFO:journalctl@ceph.mon.a.smithi012.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-23T22:50:22.080 INFO:journalctl@ceph.mon.a.smithi012.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-23T22:50:22.080 INFO:journalctl@ceph.mon.a.smithi012.stdout: Cumulative compaction: 0.00 GB write, 12.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-23T22:50:22.080 INFO:journalctl@ceph.mon.a.smithi012.stdout: Interval compaction: 0.00 GB write, 12.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-23T22:50:22.080 INFO:journalctl@ceph.mon.a.smithi012.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-23T22:50:22.080 INFO:journalctl@ceph.mon.a.smithi012.stdout: Block cache BinnedLRUCache@0x562479c93090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-03-23T22:50:22.080 INFO:journalctl@ceph.mon.a.smithi012.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-23T22:50:22.080 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-03-23T22:50:22.080 INFO:journalctl@ceph.mon.a.smithi012.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-23T22:50:22.080 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: starting mon.a rank 0 at public addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] at bind addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T22:50:22.080 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: mon.a@-1(???) e1 preinit fsid 90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T22:50:22.081 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: mon.a@-1(???).mds e1 new map 2024-03-23T22:50:22.081 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: mon.a@-1(???).mds e1 print_map 2024-03-23T22:50:22.081 INFO:journalctl@ceph.mon.a.smithi012.stdout: e1 2024-03-23T22:50:22.081 INFO:journalctl@ceph.mon.a.smithi012.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T22:50:22.081 INFO:journalctl@ceph.mon.a.smithi012.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-23T22:50:22.081 INFO:journalctl@ceph.mon.a.smithi012.stdout: legacy client fscid: -1 2024-03-23T22:50:22.081 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-03-23T22:50:22.081 INFO:journalctl@ceph.mon.a.smithi012.stdout: No filesystems configured 2024-03-23T22:50:22.081 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-23T22:50:22.081 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-23T22:50:22.081 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-23T22:50:22.081 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-23T22:50:22.082 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-23T22:50:22.082 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 bash[31676]: bdb2aed276e79f0d904555aa6918d03e2fd780229072949a875ee8254c37765b 2024-03-23T22:50:22.082 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 systemd[1]: Started Ceph mon.a for 90f679ae-e967-11ee-95cd-87774f69a715. 2024-03-23T22:50:22.082 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-23T22:50:22.082 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: monmap e1: 1 mons at {a=[v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-23T22:50:22.082 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: fsmap 2024-03-23T22:50:22.082 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: osdmap e1: 0 total, 0 up, 0 in 2024-03-23T22:50:22.082 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:21 smithi012 ceph-mon[31695]: mgrmap e1: no daemons active 2024-03-23T22:50:23.380 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.626+0000 7f34fd600700 1 Processor -- start 2024-03-23T22:50:23.380 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.626+0000 7f34fd600700 1 -- start start 2024-03-23T22:50:23.380 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.627+0000 7f34fd600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34f81084c0 0x7f34f810a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:23.382 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.627+0000 7f34fd600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f807ae30 con 0x7f34f81084c0 2024-03-23T22:50:23.382 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.627+0000 7f34f6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34f81084c0 0x7f34f810a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:23.382 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.627+0000 7f34f6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34f81084c0 0x7f34f810a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52096/0 (socket says 172.21.15.12:52096) 2024-03-23T22:50:23.382 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.627+0000 7f34f6a00700 1 -- 172.21.15.12:0/2084911831 learned_addr learned my addr 172.21.15.12:0/2084911831 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:23.382 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.628+0000 7f34f6a00700 1 -- 172.21.15.12:0/2084911831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f810adf0 con 0x7f34f81084c0 2024-03-23T22:50:23.382 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.629+0000 7f34f6a00700 1 --2- 172.21.15.12:0/2084911831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34f81084c0 0x7f34f810a8b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f34e8009e80 tx=0x7f34e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2feca255717961ac server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:23.382 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.629+0000 7f34f5600700 1 -- 172.21.15.12:0/2084911831 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34e8017070 con 0x7f34f81084c0 2024-03-23T22:50:23.382 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.629+0000 7f34f5600700 1 -- 172.21.15.12:0/2084911831 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f34e800b8c0 con 0x7f34f81084c0 2024-03-23T22:50:23.382 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.629+0000 7f34f5600700 1 -- 172.21.15.12:0/2084911831 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34e80045c0 con 0x7f34f81084c0 2024-03-23T22:50:23.382 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.630+0000 7f34fd600700 1 -- 172.21.15.12:0/2084911831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34f81084c0 msgr2=0x7f34f810a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:23.382 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.630+0000 7f34fd600700 1 --2- 172.21.15.12:0/2084911831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34f81084c0 0x7f34f810a8b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f34e8009e80 tx=0x7f34e800b330 comp rx=0 tx=0).stop 2024-03-23T22:50:23.382 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.630+0000 7f34fd600700 1 -- 172.21.15.12:0/2084911831 shutdown_connections 2024-03-23T22:50:23.383 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.630+0000 7f34fd600700 1 --2- 172.21.15.12:0/2084911831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34f81084c0 0x7f34f810a8b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:23.383 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.630+0000 7f34fd600700 1 -- 172.21.15.12:0/2084911831 >> 172.21.15.12:0/2084911831 conn(0x7f34f8101f70 msgr2=0x7f34f8104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:23.383 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.630+0000 7f34fd600700 1 -- 172.21.15.12:0/2084911831 shutdown_connections 2024-03-23T22:50:23.383 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.630+0000 7f34fd600700 1 -- 172.21.15.12:0/2084911831 wait complete. 2024-03-23T22:50:23.383 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.631+0000 7f34fd600700 1 Processor -- start 2024-03-23T22:50:23.383 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.631+0000 7f34fd600700 1 -- start start 2024-03-23T22:50:23.385 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.632+0000 7f34fd600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34f81084c0 0x7f34f819db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:23.385 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.632+0000 7f34fd600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f810bdc0 con 0x7f34f81084c0 2024-03-23T22:50:23.385 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.632+0000 7f34f6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34f81084c0 0x7f34f819db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:23.385 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.632+0000 7f34f6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34f81084c0 0x7f34f819db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52104/0 (socket says 172.21.15.12:52104) 2024-03-23T22:50:23.385 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.632+0000 7f34f6a00700 1 -- 172.21.15.12:0/547476142 learned_addr learned my addr 172.21.15.12:0/547476142 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:23.385 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.632+0000 7f34f6a00700 1 -- 172.21.15.12:0/547476142 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f819e060 con 0x7f34f81084c0 2024-03-23T22:50:23.385 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.633+0000 7f34f6a00700 1 --2- 172.21.15.12:0/547476142 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34f81084c0 0x7f34f819db20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f34e8004350 tx=0x7f34e800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:23.386 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.633+0000 7f34ef400700 1 -- 172.21.15.12:0/547476142 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34e8020020 con 0x7f34f81084c0 2024-03-23T22:50:23.386 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.633+0000 7f34ef400700 1 -- 172.21.15.12:0/547476142 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f34e8003c10 con 0x7f34f81084c0 2024-03-23T22:50:23.386 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.633+0000 7f34fd600700 1 -- 172.21.15.12:0/547476142 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f819e2f0 con 0x7f34f81084c0 2024-03-23T22:50:23.386 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.633+0000 7f34ef400700 1 -- 172.21.15.12:0/547476142 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34e8017040 con 0x7f34f81084c0 2024-03-23T22:50:23.386 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.633+0000 7f34fd600700 1 -- 172.21.15.12:0/547476142 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f81a0fe0 con 0x7f34f81084c0 2024-03-23T22:50:23.386 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.634+0000 7f34ef400700 1 -- 172.21.15.12:0/547476142 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f34e8023020 con 0x7f34f81084c0 2024-03-23T22:50:23.386 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.634+0000 7f34ef400700 1 -- 172.21.15.12:0/547476142 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f34e800f050 con 0x7f34f81084c0 2024-03-23T22:50:23.386 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.635+0000 7f34fd600700 1 -- 172.21.15.12:0/547476142 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f80637c0 con 0x7f34f81084c0 2024-03-23T22:50:23.386 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.638+0000 7f34ef400700 1 -- 172.21.15.12:0/547476142 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f34e802a020 con 0x7f34f81084c0 2024-03-23T22:50:23.386 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.715+0000 7f34fd600700 1 -- 172.21.15.12:0/547476142 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f34f81a11b0 con 0x7f34f81084c0 2024-03-23T22:50:23.386 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.718+0000 7f34ef400700 1 -- 172.21.15.12:0/547476142 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f34e801ccd0 con 0x7f34f81084c0 2024-03-23T22:50:23.386 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.720+0000 7f34fd600700 1 -- 172.21.15.12:0/547476142 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34f81084c0 msgr2=0x7f34f819db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:23.386 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.720+0000 7f34fd600700 1 --2- 172.21.15.12:0/547476142 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34f81084c0 0x7f34f819db20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f34e8004350 tx=0x7f34e800bd50 comp rx=0 tx=0).stop 2024-03-23T22:50:23.386 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.720+0000 7f34fd600700 1 -- 172.21.15.12:0/547476142 shutdown_connections 2024-03-23T22:50:23.386 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.720+0000 7f34fd600700 1 --2- 172.21.15.12:0/547476142 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34f81084c0 0x7f34f819db20 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:23.387 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.720+0000 7f34fd600700 1 -- 172.21.15.12:0/547476142 >> 172.21.15.12:0/547476142 conn(0x7f34f8101f70 msgr2=0x7f34f8102990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:23.387 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.720+0000 7f34fd600700 1 -- 172.21.15.12:0/547476142 shutdown_connections 2024-03-23T22:50:23.387 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:22.720+0000 7f34fd600700 1 -- 172.21.15.12:0/547476142 wait complete. 2024-03-23T22:50:23.387 INFO:teuthology.orchestra.run.smithi012.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-23T22:50:23.412 INFO:teuthology.orchestra.run.smithi012.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-23T22:50:23.412 INFO:teuthology.orchestra.run.smithi012.stdout:Creating mgr... 2024-03-23T22:50:23.413 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-23T22:50:23.415 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-23T22:50:23.859 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-90f679ae-e967-11ee-95cd-87774f69a715@mgr.x 2024-03-23T22:50:23.859 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Failed to reset failed state of unit ceph-90f679ae-e967-11ee-95cd-87774f69a715@mgr.x.service: Unit ceph-90f679ae-e967-11ee-95cd-87774f69a715@mgr.x.service not loaded. 2024-03-23T22:50:24.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:23 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/547476142' entity='client.admin' 2024-03-23T22:50:24.134 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-90f679ae-e967-11ee-95cd-87774f69a715.target.wants/ceph-90f679ae-e967-11ee-95cd-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-90f679ae-e967-11ee-95cd-87774f69a715@.service. 2024-03-23T22:50:24.805 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:24 smithi012 podman[31907]: 2024-03-23 22:50:24.43957953 +0000 UTC m=+0.032429855 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-23T22:50:24.916 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-23T22:50:24.916 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout disabled 2024-03-23T22:50:24.931 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-23T22:50:24.931 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-03-23T22:50:24.931 INFO:teuthology.orchestra.run.smithi012.stdout:firewalld.service is not enabled 2024-03-23T22:50:24.932 INFO:teuthology.orchestra.run.smithi012.stdout:Not possible to enable service . firewalld.service is not available 2024-03-23T22:50:24.950 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-23T22:50:24.950 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout disabled 2024-03-23T22:50:24.965 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-23T22:50:24.965 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-03-23T22:50:24.965 INFO:teuthology.orchestra.run.smithi012.stdout:firewalld.service is not enabled 2024-03-23T22:50:24.965 INFO:teuthology.orchestra.run.smithi012.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-23T22:50:24.965 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mgr to start... 2024-03-23T22:50:24.965 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mgr... 2024-03-23T22:50:25.116 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:24 smithi012 podman[31907]: 2024-03-23 22:50:24.810288986 +0000 UTC m=+0.403139337 container init 1b29f43ea2ee610ca75b9576dcf67eda7f1fe9916c35cecf067898434254804e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-23T22:50:25.116 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:24 smithi012 podman[31907]: 2024-03-23 22:50:24.81806503 +0000 UTC m=+0.410915366 container start 1b29f43ea2ee610ca75b9576dcf67eda7f1fe9916c35cecf067898434254804e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-23T22:50:25.116 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:24 smithi012 bash[31907]: 1b29f43ea2ee610ca75b9576dcf67eda7f1fe9916c35cecf067898434254804e 2024-03-23T22:50:25.116 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:24 smithi012 systemd[1]: Started Ceph mgr.x for 90f679ae-e967-11ee-95cd-87774f69a715. 2024-03-23T22:50:25.630 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:25 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:25.611+0000 7fbba2896200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-23T22:50:25.986 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:25 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:25.745+0000 7fbba2896200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-23T22:50:25.986 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:25 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3853792001' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-23T22:50:26.309 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-23T22:50:26.312 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-03-23T22:50:26.312 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsid": "90f679ae-e967-11ee-95cd-87774f69a715", 2024-03-23T22:50:26.312 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "health": { 2024-03-23T22:50:26.312 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-23T22:50:26.312 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-23T22:50:26.312 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-23T22:50:26.312 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:26.312 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-23T22:50:26.312 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-23T22:50:26.312 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 0 2024-03-23T22:50:26.312 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-23T22:50:26.313 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-23T22:50:26.313 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "a" 2024-03-23T22:50:26.313 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-23T22:50:26.313 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-23T22:50:26.313 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-23T22:50:26.313 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-23T22:50:26.313 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-23T22:50:26.313 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-23T22:50:26.313 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:26.313 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-23T22:50:26.313 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-23T22:50:26.313 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-23T22:50:26.313 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-23T22:50:26.313 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-23T22:50:26.313 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-23T22:50:26.314 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-23T22:50:26.314 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-23T22:50:26.314 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:26.314 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-23T22:50:26.321 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-23T22:50:26.321 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-23T22:50:26.321 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-23T22:50:26.321 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-23T22:50:26.321 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-23T22:50:26.321 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-23T22:50:26.321 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-23T22:50:26.321 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-23T22:50:26.321 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:26.321 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-23T22:50:26.321 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-23T22:50:26.321 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-23T22:50:26.321 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-23T22:50:26.321 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:26.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-23T22:50:26.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-23T22:50:26.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-23T22:50:26.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-23T22:50:26.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-23T22:50:26.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-23T22:50:26.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "restful" 2024-03-23T22:50:26.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-23T22:50:26.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-23T22:50:26.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:26.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-23T22:50:26.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-23T22:50:26.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modified": "2024-03-23T22:50:14.907565+0000", 2024-03-23T22:50:26.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-23T22:50:26.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:26.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-23T22:50:26.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-03-23T22:50:26.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.534+0000 7f1bd0c00700 1 Processor -- start 2024-03-23T22:50:26.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.534+0000 7f1bd0c00700 1 -- start start 2024-03-23T22:50:26.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.534+0000 7f1bd0c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bcc07b2e0 0x7f1bcc07b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:26.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.534+0000 7f1bd0c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc07d010 con 0x7f1bcc07b2e0 2024-03-23T22:50:26.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.535+0000 7f1bc9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bcc07b2e0 0x7f1bcc07b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:26.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.535+0000 7f1bc9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bcc07b2e0 0x7f1bcc07b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39884/0 (socket says 172.21.15.12:39884) 2024-03-23T22:50:26.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.535+0000 7f1bc9e00700 1 -- 172.21.15.12:0/388025535 learned_addr learned my addr 172.21.15.12:0/388025535 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:26.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.536+0000 7f1bc9e00700 1 -- 172.21.15.12:0/388025535 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bcc07bc40 con 0x7f1bcc07b2e0 2024-03-23T22:50:26.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.536+0000 7f1bc9e00700 1 --2- 172.21.15.12:0/388025535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bcc07b2e0 0x7f1bcc07b700 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc0098a0 tx=0x7f1bbc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afb734b58af433e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:26.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.537+0000 7f1bc8a00700 1 -- 172.21.15.12:0/388025535 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1bbc004160 con 0x7f1bcc07b2e0 2024-03-23T22:50:26.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.537+0000 7f1bc8a00700 1 -- 172.21.15.12:0/388025535 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1bbc004300 con 0x7f1bcc07b2e0 2024-03-23T22:50:26.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.537+0000 7f1bc8a00700 1 -- 172.21.15.12:0/388025535 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1bbc011b50 con 0x7f1bcc07b2e0 2024-03-23T22:50:26.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.538+0000 7f1bd0c00700 1 -- 172.21.15.12:0/388025535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bcc07b2e0 msgr2=0x7f1bcc07b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:26.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.538+0000 7f1bd0c00700 1 --2- 172.21.15.12:0/388025535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bcc07b2e0 0x7f1bcc07b700 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc0098a0 tx=0x7f1bbc009bb0 comp rx=0 tx=0).stop 2024-03-23T22:50:26.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.538+0000 7f1bd0c00700 1 -- 172.21.15.12:0/388025535 shutdown_connections 2024-03-23T22:50:26.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.538+0000 7f1bd0c00700 1 --2- 172.21.15.12:0/388025535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bcc07b2e0 0x7f1bcc07b700 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:26.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.538+0000 7f1bd0c00700 1 -- 172.21.15.12:0/388025535 >> 172.21.15.12:0/388025535 conn(0x7f1bcc103000 msgr2=0x7f1bcc105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:26.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.538+0000 7f1bd0c00700 1 -- 172.21.15.12:0/388025535 shutdown_connections 2024-03-23T22:50:26.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.539+0000 7f1bd0c00700 1 -- 172.21.15.12:0/388025535 wait complete. 2024-03-23T22:50:26.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.540+0000 7f1bd0c00700 1 Processor -- start 2024-03-23T22:50:26.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.540+0000 7f1bd0c00700 1 -- start start 2024-03-23T22:50:26.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.540+0000 7f1bd0c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bcc07b2e0 0x7f1bcc19dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:26.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.540+0000 7f1bd0c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc10e110 con 0x7f1bcc07b2e0 2024-03-23T22:50:26.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.540+0000 7f1bc9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bcc07b2e0 0x7f1bcc19dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:26.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.540+0000 7f1bc9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bcc07b2e0 0x7f1bcc19dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39898/0 (socket says 172.21.15.12:39898) 2024-03-23T22:50:26.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.540+0000 7f1bc9e00700 1 -- 172.21.15.12:0/3853792001 learned_addr learned my addr 172.21.15.12:0/3853792001 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:26.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.541+0000 7f1bc9e00700 1 -- 172.21.15.12:0/3853792001 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bcc19e230 con 0x7f1bcc07b2e0 2024-03-23T22:50:26.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.541+0000 7f1bc9e00700 1 --2- 172.21.15.12:0/3853792001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bcc07b2e0 0x7f1bcc19dcf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc011b60 tx=0x7f1bbc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:26.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.541+0000 7f1bc2a00700 1 -- 172.21.15.12:0/3853792001 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1bbc003970 con 0x7f1bcc07b2e0 2024-03-23T22:50:26.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.542+0000 7f1bc2a00700 1 -- 172.21.15.12:0/3853792001 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1bbc003b10 con 0x7f1bcc07b2e0 2024-03-23T22:50:26.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.542+0000 7f1bd0c00700 1 -- 172.21.15.12:0/3853792001 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bcc19e4c0 con 0x7f1bcc07b2e0 2024-03-23T22:50:26.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.542+0000 7f1bc2a00700 1 -- 172.21.15.12:0/3853792001 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1bbc024610 con 0x7f1bcc07b2e0 2024-03-23T22:50:26.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.542+0000 7f1bd0c00700 1 -- 172.21.15.12:0/3853792001 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bcc1a11a0 con 0x7f1bcc07b2e0 2024-03-23T22:50:26.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.542+0000 7f1bc2a00700 1 -- 172.21.15.12:0/3853792001 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f1bbc024c50 con 0x7f1bcc07b2e0 2024-03-23T22:50:26.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.542+0000 7f1bc2a00700 1 -- 172.21.15.12:0/3853792001 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1bbc011e00 con 0x7f1bcc07b2e0 2024-03-23T22:50:26.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.543+0000 7f1bd0c00700 1 -- 172.21.15.12:0/3853792001 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bac005360 con 0x7f1bcc07b2e0 2024-03-23T22:50:26.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.547+0000 7f1bc2a00700 1 -- 172.21.15.12:0/3853792001 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f1bbc01a9f0 con 0x7f1bcc07b2e0 2024-03-23T22:50:26.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.627+0000 7f1bd0c00700 1 -- 172.21.15.12:0/3853792001 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1bac005920 con 0x7f1bcc07b2e0 2024-03-23T22:50:26.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.628+0000 7f1bc2a00700 1 -- 172.21.15.12:0/3853792001 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f1bbc01a430 con 0x7f1bcc07b2e0 2024-03-23T22:50:26.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.630+0000 7f1bd0c00700 1 -- 172.21.15.12:0/3853792001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bcc07b2e0 msgr2=0x7f1bcc19dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:26.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.630+0000 7f1bd0c00700 1 --2- 172.21.15.12:0/3853792001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bcc07b2e0 0x7f1bcc19dcf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc011b60 tx=0x7f1bbc011c40 comp rx=0 tx=0).stop 2024-03-23T22:50:26.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.630+0000 7f1bd0c00700 1 -- 172.21.15.12:0/3853792001 shutdown_connections 2024-03-23T22:50:26.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.630+0000 7f1bd0c00700 1 --2- 172.21.15.12:0/3853792001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bcc07b2e0 0x7f1bcc19dcf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:26.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.630+0000 7f1bd0c00700 1 -- 172.21.15.12:0/3853792001 >> 172.21.15.12:0/3853792001 conn(0x7f1bcc103000 msgr2=0x7f1bcc103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:26.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.630+0000 7f1bd0c00700 1 -- 172.21.15.12:0/3853792001 shutdown_connections 2024-03-23T22:50:26.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:25.631+0000 7f1bd0c00700 1 -- 172.21.15.12:0/3853792001 wait complete. 2024-03-23T22:50:26.326 INFO:teuthology.orchestra.run.smithi012.stdout:mgr not available, waiting (1/15)... 2024-03-23T22:50:26.899 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:26 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:26.583+0000 7fbba2896200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-23T22:50:26.900 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:26 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:26.709+0000 7fbba2896200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-23T22:50:26.900 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:26 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:26.898+0000 7fbba2896200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-23T22:50:27.303 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:27 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:27.027+0000 7fbba2896200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-23T22:50:27.304 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:27 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:27.139+0000 7fbba2896200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-23T22:50:27.756 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:27 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:27.492+0000 7fbba2896200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-23T22:50:27.756 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:27 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:27.639+0000 7fbba2896200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-23T22:50:28.054 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:27 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:27.755+0000 7fbba2896200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-23T22:50:28.054 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:27 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:27.882+0000 7fbba2896200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-23T22:50:28.449 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:28 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:28.103+0000 7fbba2896200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-23T22:50:29.272 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:28 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3859139535' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-23T22:50:29.553 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:29 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:29.493+0000 7fbba2896200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-23T22:50:29.653 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-23T22:50:29.653 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-03-23T22:50:29.653 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsid": "90f679ae-e967-11ee-95cd-87774f69a715", 2024-03-23T22:50:29.653 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "health": { 2024-03-23T22:50:29.653 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-23T22:50:29.653 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-23T22:50:29.653 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-23T22:50:29.653 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:29.653 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-23T22:50:29.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-23T22:50:29.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 0 2024-03-23T22:50:29.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-23T22:50:29.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-23T22:50:29.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "a" 2024-03-23T22:50:29.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-23T22:50:29.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-03-23T22:50:29.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-23T22:50:29.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-23T22:50:29.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-23T22:50:29.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-23T22:50:29.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:29.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-23T22:50:29.655 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-23T22:50:29.655 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-23T22:50:29.655 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-23T22:50:29.655 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-23T22:50:29.663 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-23T22:50:29.663 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-23T22:50:29.663 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-23T22:50:29.663 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:29.663 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-23T22:50:29.663 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-23T22:50:29.663 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-23T22:50:29.664 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-23T22:50:29.664 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-23T22:50:29.664 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-23T22:50:29.664 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-23T22:50:29.664 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-23T22:50:29.664 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-23T22:50:29.664 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:29.664 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-23T22:50:29.664 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-23T22:50:29.664 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-23T22:50:29.664 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-23T22:50:29.664 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:29.664 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-23T22:50:29.664 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-23T22:50:29.665 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-23T22:50:29.665 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-23T22:50:29.665 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-23T22:50:29.665 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-23T22:50:29.665 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "restful" 2024-03-23T22:50:29.665 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-23T22:50:29.665 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-23T22:50:29.665 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:29.665 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-23T22:50:29.665 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-23T22:50:29.665 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modified": "2024-03-23T22:50:14.907565+0000", 2024-03-23T22:50:29.665 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-23T22:50:29.665 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:29.666 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-23T22:50:29.666 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-03-23T22:50:29.666 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.822+0000 7fd5f3a00700 1 Processor -- start 2024-03-23T22:50:29.666 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.822+0000 7fd5f3a00700 1 -- start start 2024-03-23T22:50:29.666 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.823+0000 7fd5f3a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5ec108470 0x7fd5ec10a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:29.666 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.823+0000 7fd5f3a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec07ae70 con 0x7fd5ec108470 2024-03-23T22:50:29.666 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.823+0000 7fd5f1000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5ec108470 0x7fd5ec10a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:29.666 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.823+0000 7fd5f1000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5ec108470 0x7fd5ec10a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39900/0 (socket says 172.21.15.12:39900) 2024-03-23T22:50:29.666 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.823+0000 7fd5f1000700 1 -- 172.21.15.12:0/3848730347 learned_addr learned my addr 172.21.15.12:0/3848730347 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:29.666 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.823+0000 7fd5f1000700 1 -- 172.21.15.12:0/3848730347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ec10ada0 con 0x7fd5ec108470 2024-03-23T22:50:29.666 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.824+0000 7fd5f1000700 1 --2- 172.21.15.12:0/3848730347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5ec108470 0x7fd5ec10a860 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd5e00098a0 tx=0x7fd5e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ca3c67395e61f28 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:29.666 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.824+0000 7fd5eb400700 1 -- 172.21.15.12:0/3848730347 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5e0004160 con 0x7fd5ec108470 2024-03-23T22:50:29.666 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.824+0000 7fd5eb400700 1 -- 172.21.15.12:0/3848730347 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd5e0004300 con 0x7fd5ec108470 2024-03-23T22:50:29.667 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.824+0000 7fd5eb400700 1 -- 172.21.15.12:0/3848730347 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5e0011b50 con 0x7fd5ec108470 2024-03-23T22:50:29.667 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.825+0000 7fd5f3a00700 1 -- 172.21.15.12:0/3848730347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5ec108470 msgr2=0x7fd5ec10a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:29.667 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.825+0000 7fd5f3a00700 1 --2- 172.21.15.12:0/3848730347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5ec108470 0x7fd5ec10a860 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd5e00098a0 tx=0x7fd5e0009bb0 comp rx=0 tx=0).stop 2024-03-23T22:50:29.667 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.825+0000 7fd5f3a00700 1 -- 172.21.15.12:0/3848730347 shutdown_connections 2024-03-23T22:50:29.667 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.825+0000 7fd5f3a00700 1 --2- 172.21.15.12:0/3848730347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5ec108470 0x7fd5ec10a860 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:29.667 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.825+0000 7fd5f3a00700 1 -- 172.21.15.12:0/3848730347 >> 172.21.15.12:0/3848730347 conn(0x7fd5ec101f40 msgr2=0x7fd5ec104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:29.667 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.825+0000 7fd5f3a00700 1 -- 172.21.15.12:0/3848730347 shutdown_connections 2024-03-23T22:50:29.667 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.825+0000 7fd5f3a00700 1 -- 172.21.15.12:0/3848730347 wait complete. 2024-03-23T22:50:29.667 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.826+0000 7fd5f3a00700 1 Processor -- start 2024-03-23T22:50:29.667 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.826+0000 7fd5f3a00700 1 -- start start 2024-03-23T22:50:29.667 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.827+0000 7fd5f3a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5ec108470 0x7fd5ec19db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:29.667 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.827+0000 7fd5f3a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec10bd60 con 0x7fd5ec108470 2024-03-23T22:50:29.667 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.827+0000 7fd5f1000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5ec108470 0x7fd5ec19db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:29.667 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.827+0000 7fd5f1000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5ec108470 0x7fd5ec19db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39904/0 (socket says 172.21.15.12:39904) 2024-03-23T22:50:29.668 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.827+0000 7fd5f1000700 1 -- 172.21.15.12:0/3859139535 learned_addr learned my addr 172.21.15.12:0/3859139535 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:29.668 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.827+0000 7fd5f1000700 1 -- 172.21.15.12:0/3859139535 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ec19e0a0 con 0x7fd5ec108470 2024-03-23T22:50:29.668 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.827+0000 7fd5f1000700 1 --2- 172.21.15.12:0/3859139535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5ec108470 0x7fd5ec19db60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0011b60 tx=0x7fd5e0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:29.668 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.828+0000 7fd5e9600700 1 -- 172.21.15.12:0/3859139535 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5e0003b20 con 0x7fd5ec108470 2024-03-23T22:50:29.668 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.828+0000 7fd5e9600700 1 -- 172.21.15.12:0/3859139535 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd5e0003cc0 con 0x7fd5ec108470 2024-03-23T22:50:29.668 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.828+0000 7fd5f3a00700 1 -- 172.21.15.12:0/3859139535 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5ec19e390 con 0x7fd5ec108470 2024-03-23T22:50:29.668 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.828+0000 7fd5e9600700 1 -- 172.21.15.12:0/3859139535 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5e0024840 con 0x7fd5ec108470 2024-03-23T22:50:29.668 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.828+0000 7fd5f3a00700 1 -- 172.21.15.12:0/3859139535 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5ec1a1020 con 0x7fd5ec108470 2024-03-23T22:50:29.668 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.828+0000 7fd5e9600700 1 -- 172.21.15.12:0/3859139535 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fd5e0022020 con 0x7fd5ec108470 2024-03-23T22:50:29.668 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.828+0000 7fd5e9600700 1 -- 172.21.15.12:0/3859139535 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd5e0011e00 con 0x7fd5ec108470 2024-03-23T22:50:29.668 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.829+0000 7fd5dea00700 1 -- 172.21.15.12:0/3859139535 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5c8005330 con 0x7fd5ec108470 2024-03-23T22:50:29.668 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.832+0000 7fd5e9600700 1 -- 172.21.15.12:0/3859139535 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fd5e001a430 con 0x7fd5ec108470 2024-03-23T22:50:29.668 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.912+0000 7fd5dea00700 1 -- 172.21.15.12:0/3859139535 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd5c80058f0 con 0x7fd5ec108470 2024-03-23T22:50:29.668 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.913+0000 7fd5e9600700 1 -- 172.21.15.12:0/3859139535 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fd5e001a610 con 0x7fd5ec108470 2024-03-23T22:50:29.669 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.914+0000 7fd5dea00700 1 -- 172.21.15.12:0/3859139535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5ec108470 msgr2=0x7fd5ec19db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:29.669 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.914+0000 7fd5dea00700 1 --2- 172.21.15.12:0/3859139535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5ec108470 0x7fd5ec19db60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0011b60 tx=0x7fd5e0011c40 comp rx=0 tx=0).stop 2024-03-23T22:50:29.669 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.914+0000 7fd5dea00700 1 -- 172.21.15.12:0/3859139535 shutdown_connections 2024-03-23T22:50:29.669 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.914+0000 7fd5dea00700 1 --2- 172.21.15.12:0/3859139535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5ec108470 0x7fd5ec19db60 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:29.669 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.914+0000 7fd5dea00700 1 -- 172.21.15.12:0/3859139535 >> 172.21.15.12:0/3859139535 conn(0x7fd5ec101f40 msgr2=0x7fd5ec102ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:29.669 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.914+0000 7fd5dea00700 1 -- 172.21.15.12:0/3859139535 shutdown_connections 2024-03-23T22:50:29.669 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:28.914+0000 7fd5dea00700 1 -- 172.21.15.12:0/3859139535 wait complete. 2024-03-23T22:50:29.669 INFO:teuthology.orchestra.run.smithi012.stdout:mgr not available, waiting (2/15)... 2024-03-23T22:50:30.053 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:29 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:29.673+0000 7fbba2896200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-23T22:50:30.485 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:30 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:30.100+0000 7fbba2896200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-23T22:50:30.485 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:30 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:30.221+0000 7fbba2896200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-23T22:50:30.736 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:30 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:30.484+0000 7fbba2896200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-23T22:50:31.053 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:30 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:30.735+0000 7fbba2896200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-23T22:50:31.442 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:31 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:31.119+0000 7fbba2896200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-23T22:50:31.756 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:31 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:31.441+0000 7fbba2896200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-23T22:50:32.041 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:31 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:31.756+0000 7fbba2896200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-23T22:50:32.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:32 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3763380687' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-23T22:50:33.027 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:32 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:32.846+0000 7fbba2896200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-23T22:50:33.041 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-23T22:50:33.041 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-03-23T22:50:33.041 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsid": "90f679ae-e967-11ee-95cd-87774f69a715", 2024-03-23T22:50:33.041 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "health": { 2024-03-23T22:50:33.041 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-23T22:50:33.043 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-23T22:50:33.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-23T22:50:33.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:33.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-23T22:50:33.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-23T22:50:33.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 0 2024-03-23T22:50:33.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-23T22:50:33.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-23T22:50:33.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "a" 2024-03-23T22:50:33.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-23T22:50:33.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-23T22:50:33.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-23T22:50:33.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-23T22:50:33.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-23T22:50:33.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-23T22:50:33.044 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:33.045 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-23T22:50:33.048 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-23T22:50:33.048 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-23T22:50:33.048 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-23T22:50:33.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-23T22:50:33.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-23T22:50:33.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-23T22:50:33.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-23T22:50:33.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:33.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-23T22:50:33.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-23T22:50:33.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-23T22:50:33.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-23T22:50:33.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-23T22:50:33.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-23T22:50:33.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-23T22:50:33.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-23T22:50:33.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-23T22:50:33.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:33.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-23T22:50:33.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-23T22:50:33.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-23T22:50:33.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-23T22:50:33.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:33.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-23T22:50:33.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-23T22:50:33.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-23T22:50:33.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-23T22:50:33.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-23T22:50:33.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-23T22:50:33.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "restful" 2024-03-23T22:50:33.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-23T22:50:33.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-23T22:50:33.051 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:33.051 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-23T22:50:33.051 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-23T22:50:33.051 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modified": "2024-03-23T22:50:14.907565+0000", 2024-03-23T22:50:33.051 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-23T22:50:33.051 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:33.051 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-23T22:50:33.051 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-03-23T22:50:33.056 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.188+0000 7ff68a600700 1 Processor -- start 2024-03-23T22:50:33.056 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.188+0000 7ff68a600700 1 -- start start 2024-03-23T22:50:33.056 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.189+0000 7ff68a600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff684108530 0x7ff68410a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:33.056 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.189+0000 7ff68a600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff68407adb0 con 0x7ff684108530 2024-03-23T22:50:33.056 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.189+0000 7ff683400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff684108530 0x7ff68410a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:33.056 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.189+0000 7ff683400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff684108530 0x7ff68410a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39912/0 (socket says 172.21.15.12:39912) 2024-03-23T22:50:33.056 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.189+0000 7ff683400700 1 -- 172.21.15.12:0/1359658651 learned_addr learned my addr 172.21.15.12:0/1359658651 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:33.056 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.189+0000 7ff683400700 1 -- 172.21.15.12:0/1359658651 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff68410ae60 con 0x7ff684108530 2024-03-23T22:50:33.056 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.190+0000 7ff683400700 1 --2- 172.21.15.12:0/1359658651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff684108530 0x7ff68410a920 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff6740098a0 tx=0x7ff674009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d7c5fdfab05c3d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:33.056 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.190+0000 7ff682000700 1 -- 172.21.15.12:0/1359658651 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff674004160 con 0x7ff684108530 2024-03-23T22:50:33.056 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.190+0000 7ff682000700 1 -- 172.21.15.12:0/1359658651 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff674004300 con 0x7ff684108530 2024-03-23T22:50:33.056 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.190+0000 7ff682000700 1 -- 172.21.15.12:0/1359658651 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff674011b50 con 0x7ff684108530 2024-03-23T22:50:33.057 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.191+0000 7ff68a600700 1 -- 172.21.15.12:0/1359658651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff684108530 msgr2=0x7ff68410a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:33.057 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.191+0000 7ff68a600700 1 --2- 172.21.15.12:0/1359658651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff684108530 0x7ff68410a920 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff6740098a0 tx=0x7ff674009bb0 comp rx=0 tx=0).stop 2024-03-23T22:50:33.057 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.191+0000 7ff68a600700 1 -- 172.21.15.12:0/1359658651 shutdown_connections 2024-03-23T22:50:33.057 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.191+0000 7ff68a600700 1 --2- 172.21.15.12:0/1359658651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff684108530 0x7ff68410a920 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:33.057 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.191+0000 7ff68a600700 1 -- 172.21.15.12:0/1359658651 >> 172.21.15.12:0/1359658651 conn(0x7ff684101f60 msgr2=0x7ff684104380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:33.057 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.191+0000 7ff68a600700 1 -- 172.21.15.12:0/1359658651 shutdown_connections 2024-03-23T22:50:33.057 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.191+0000 7ff68a600700 1 -- 172.21.15.12:0/1359658651 wait complete. 2024-03-23T22:50:33.057 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.192+0000 7ff68a600700 1 Processor -- start 2024-03-23T22:50:33.057 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.192+0000 7ff68a600700 1 -- start start 2024-03-23T22:50:33.057 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.193+0000 7ff68a600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff684108530 0x7ff68419dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:33.057 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.193+0000 7ff68a600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff68410bdc0 con 0x7ff684108530 2024-03-23T22:50:33.057 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.193+0000 7ff683400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff684108530 0x7ff68419dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:33.057 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.193+0000 7ff683400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff684108530 0x7ff68419dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39918/0 (socket says 172.21.15.12:39918) 2024-03-23T22:50:33.057 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.193+0000 7ff683400700 1 -- 172.21.15.12:0/3763380687 learned_addr learned my addr 172.21.15.12:0/3763380687 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:33.058 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.193+0000 7ff683400700 1 -- 172.21.15.12:0/3763380687 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff68419e150 con 0x7ff684108530 2024-03-23T22:50:33.058 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.193+0000 7ff683400700 1 --2- 172.21.15.12:0/3763380687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff684108530 0x7ff68419dc10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff674011b60 tx=0x7ff674011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:33.058 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.194+0000 7ff67be00700 1 -- 172.21.15.12:0/3763380687 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff674003970 con 0x7ff684108530 2024-03-23T22:50:33.058 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.194+0000 7ff67be00700 1 -- 172.21.15.12:0/3763380687 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff674003b10 con 0x7ff684108530 2024-03-23T22:50:33.058 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.194+0000 7ff68a600700 1 -- 172.21.15.12:0/3763380687 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff68419e3e0 con 0x7ff684108530 2024-03-23T22:50:33.058 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.194+0000 7ff67be00700 1 -- 172.21.15.12:0/3763380687 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff674024610 con 0x7ff684108530 2024-03-23T22:50:33.058 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.194+0000 7ff68a600700 1 -- 172.21.15.12:0/3763380687 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6841a10d0 con 0x7ff684108530 2024-03-23T22:50:33.058 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.194+0000 7ff67be00700 1 -- 172.21.15.12:0/3763380687 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ff674024c50 con 0x7ff684108530 2024-03-23T22:50:33.058 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.194+0000 7ff67be00700 1 -- 172.21.15.12:0/3763380687 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff674011e00 con 0x7ff684108530 2024-03-23T22:50:33.058 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.195+0000 7ff68a600700 1 -- 172.21.15.12:0/3763380687 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6840637c0 con 0x7ff684108530 2024-03-23T22:50:33.058 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.198+0000 7ff67be00700 1 -- 172.21.15.12:0/3763380687 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ff674043dd0 con 0x7ff684108530 2024-03-23T22:50:33.058 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.278+0000 7ff68a600700 1 -- 172.21.15.12:0/3763380687 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff6841a12a0 con 0x7ff684108530 2024-03-23T22:50:33.058 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.279+0000 7ff67be00700 1 -- 172.21.15.12:0/3763380687 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7ff674032030 con 0x7ff684108530 2024-03-23T22:50:33.059 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.280+0000 7ff68a600700 1 -- 172.21.15.12:0/3763380687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff684108530 msgr2=0x7ff68419dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:33.059 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.280+0000 7ff68a600700 1 --2- 172.21.15.12:0/3763380687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff684108530 0x7ff68419dc10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff674011b60 tx=0x7ff674011c40 comp rx=0 tx=0).stop 2024-03-23T22:50:33.059 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.280+0000 7ff68a600700 1 -- 172.21.15.12:0/3763380687 shutdown_connections 2024-03-23T22:50:33.059 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.280+0000 7ff68a600700 1 --2- 172.21.15.12:0/3763380687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff684108530 0x7ff68419dc10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:33.059 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.280+0000 7ff68a600700 1 -- 172.21.15.12:0/3763380687 >> 172.21.15.12:0/3763380687 conn(0x7ff684101f60 msgr2=0x7ff684102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:33.059 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.280+0000 7ff68a600700 1 -- 172.21.15.12:0/3763380687 shutdown_connections 2024-03-23T22:50:33.059 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:32.280+0000 7ff68a600700 1 -- 172.21.15.12:0/3763380687 wait complete. 2024-03-23T22:50:33.059 INFO:teuthology.orchestra.run.smithi012.stdout:mgr not available, waiting (3/15)... 2024-03-23T22:50:33.987 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:33 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:33.564+0000 7fbba2896200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-23T22:50:33.987 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:33 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:33.696+0000 7fbba2896200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-23T22:50:34.304 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:33 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:33.986+0000 7fbba2896200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-23T22:50:34.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:34 smithi012 ceph-mon[31695]: Activating manager daemon x 2024-03-23T22:50:34.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:34 smithi012 ceph-mon[31695]: mgrmap e2: x(active, starting, since 0.0123498s) 2024-03-23T22:50:34.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:34 smithi012 ceph-mon[31695]: from='mgr.14100 172.21.15.12:0/1023861769' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-23T22:50:34.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:34 smithi012 ceph-mon[31695]: from='mgr.14100 172.21.15.12:0/1023861769' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-23T22:50:34.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:34 smithi012 ceph-mon[31695]: from='mgr.14100 172.21.15.12:0/1023861769' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-23T22:50:34.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:34 smithi012 ceph-mon[31695]: from='mgr.14100 172.21.15.12:0/1023861769' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-23T22:50:34.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:34 smithi012 ceph-mon[31695]: from='mgr.14100 172.21.15.12:0/1023861769' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-23T22:50:34.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:34 smithi012 ceph-mon[31695]: Manager daemon x is now available 2024-03-23T22:50:34.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:34 smithi012 ceph-mon[31695]: from='mgr.14100 172.21.15.12:0/1023861769' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-23T22:50:34.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:34 smithi012 ceph-mon[31695]: from='mgr.14100 172.21.15.12:0/1023861769' entity='mgr.x' 2024-03-23T22:50:34.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:34 smithi012 ceph-mon[31695]: from='mgr.14100 172.21.15.12:0/1023861769' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-23T22:50:35.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:35 smithi012 ceph-mon[31695]: from='mgr.14100 172.21.15.12:0/1023861769' entity='mgr.x' 2024-03-23T22:50:35.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:35 smithi012 ceph-mon[31695]: from='mgr.14100 172.21.15.12:0/1023861769' entity='mgr.x' 2024-03-23T22:50:35.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:35 smithi012 ceph-mon[31695]: mgrmap e3: x(active, since 1.02039s) 2024-03-23T22:50:36.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:36 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/977311330' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-23T22:50:36.608 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-23T22:50:36.610 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-03-23T22:50:36.610 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsid": "90f679ae-e967-11ee-95cd-87774f69a715", 2024-03-23T22:50:36.610 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "health": { 2024-03-23T22:50:36.610 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-23T22:50:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-23T22:50:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-23T22:50:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-23T22:50:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-23T22:50:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 0 2024-03-23T22:50:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-23T22:50:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-23T22:50:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "a" 2024-03-23T22:50:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-23T22:50:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-23T22:50:36.611 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-23T22:50:36.612 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-23T22:50:36.612 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-23T22:50:36.613 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-23T22:50:36.613 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:36.613 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-23T22:50:36.613 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-23T22:50:36.613 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-23T22:50:36.613 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-23T22:50:36.613 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-23T22:50:36.613 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-23T22:50:36.617 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-23T22:50:36.617 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-23T22:50:36.617 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:36.617 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-23T22:50:36.617 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-23T22:50:36.617 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-23T22:50:36.617 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-23T22:50:36.617 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-23T22:50:36.617 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-23T22:50:36.617 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-23T22:50:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-23T22:50:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-23T22:50:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-23T22:50:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-23T22:50:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-23T22:50:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-23T22:50:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-23T22:50:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-23T22:50:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-23T22:50:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-23T22:50:36.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-23T22:50:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-23T22:50:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "restful" 2024-03-23T22:50:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-23T22:50:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-23T22:50:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-23T22:50:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-23T22:50:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modified": "2024-03-23T22:50:14.907565+0000", 2024-03-23T22:50:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-23T22:50:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-23T22:50:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-23T22:50:36.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-03-23T22:50:36.625 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.645+0000 7f114f200700 1 Processor -- start 2024-03-23T22:50:36.625 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.646+0000 7f114f200700 1 -- start start 2024-03-23T22:50:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.646+0000 7f114f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f114810b0f0 0x7f114810b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.646+0000 7f114f200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11481096d0 con 0x7f114810b0f0 2024-03-23T22:50:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.647+0000 7f114de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f114810b0f0 0x7f114810b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.647+0000 7f114de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f114810b0f0 0x7f114810b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34816/0 (socket says 172.21.15.12:34816) 2024-03-23T22:50:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.647+0000 7f114de00700 1 -- 172.21.15.12:0/2332242465 learned_addr learned my addr 172.21.15.12:0/2332242465 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.647+0000 7f114de00700 1 -- 172.21.15.12:0/2332242465 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1148109850 con 0x7f114810b0f0 2024-03-23T22:50:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.647+0000 7f114de00700 1 --2- 172.21.15.12:0/2332242465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f114810b0f0 0x7f114810b510 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f113c0098a0 tx=0x7f113c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16b0721cd9662c4f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.648+0000 7f114ca00700 1 -- 172.21.15.12:0/2332242465 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f113c004160 con 0x7f114810b0f0 2024-03-23T22:50:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.648+0000 7f114ca00700 1 -- 172.21.15.12:0/2332242465 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f113c004300 con 0x7f114810b0f0 2024-03-23T22:50:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.648+0000 7f114ca00700 1 -- 172.21.15.12:0/2332242465 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f113c011b50 con 0x7f114810b0f0 2024-03-23T22:50:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.649+0000 7f114f200700 1 -- 172.21.15.12:0/2332242465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f114810b0f0 msgr2=0x7f114810b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.649+0000 7f114f200700 1 --2- 172.21.15.12:0/2332242465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f114810b0f0 0x7f114810b510 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f113c0098a0 tx=0x7f113c009bb0 comp rx=0 tx=0).stop 2024-03-23T22:50:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.649+0000 7f114f200700 1 -- 172.21.15.12:0/2332242465 shutdown_connections 2024-03-23T22:50:36.626 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.649+0000 7f114f200700 1 --2- 172.21.15.12:0/2332242465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f114810b0f0 0x7f114810b510 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.649+0000 7f114f200700 1 -- 172.21.15.12:0/2332242465 >> 172.21.15.12:0/2332242465 conn(0x7f11481054a0 msgr2=0x7f1148107890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.649+0000 7f114f200700 1 -- 172.21.15.12:0/2332242465 shutdown_connections 2024-03-23T22:50:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.649+0000 7f114f200700 1 -- 172.21.15.12:0/2332242465 wait complete. 2024-03-23T22:50:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.650+0000 7f114f200700 1 Processor -- start 2024-03-23T22:50:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.650+0000 7f114f200700 1 -- start start 2024-03-23T22:50:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.651+0000 7f114f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f114810b0f0 0x7f114819db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.651+0000 7f114f200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114810dfc0 con 0x7f114810b0f0 2024-03-23T22:50:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.651+0000 7f114de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f114810b0f0 0x7f114819db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.651+0000 7f114de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f114810b0f0 0x7f114819db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34820/0 (socket says 172.21.15.12:34820) 2024-03-23T22:50:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.651+0000 7f114de00700 1 -- 172.21.15.12:0/977311330 learned_addr learned my addr 172.21.15.12:0/977311330 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.651+0000 7f114de00700 1 -- 172.21.15.12:0/977311330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f114819e0a0 con 0x7f114810b0f0 2024-03-23T22:50:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.652+0000 7f114de00700 1 --2- 172.21.15.12:0/977311330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f114810b0f0 0x7f114819db60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f113c011850 tx=0x7f113c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:36.627 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.652+0000 7f1142a00700 1 -- 172.21.15.12:0/977311330 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f113c003af0 con 0x7f114810b0f0 2024-03-23T22:50:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.652+0000 7f1142a00700 1 -- 172.21.15.12:0/977311330 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f113c020070 con 0x7f114810b0f0 2024-03-23T22:50:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.652+0000 7f114f200700 1 -- 172.21.15.12:0/977311330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f114819e330 con 0x7f114810b0f0 2024-03-23T22:50:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.652+0000 7f1142a00700 1 -- 172.21.15.12:0/977311330 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f113c0195a0 con 0x7f114810b0f0 2024-03-23T22:50:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.652+0000 7f114f200700 1 -- 172.21.15.12:0/977311330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11481a1020 con 0x7f114810b0f0 2024-03-23T22:50:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.653+0000 7f1142a00700 1 -- 172.21.15.12:0/977311330 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 3) v1 ==== 45989+0+0 (secure 0 0 0) 0x7f113c019850 con 0x7f114810b0f0 2024-03-23T22:50:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.653+0000 7f114f200700 1 -- 172.21.15.12:0/977311330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11480637c0 con 0x7f114810b0f0 2024-03-23T22:50:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.654+0000 7f1142a00700 1 --2- 172.21.15.12:0/977311330 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f1130038e00 0x7f113003b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.654+0000 7f1142a00700 1 -- 172.21.15.12:0/977311330 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f113c011e00 con 0x7f114810b0f0 2024-03-23T22:50:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.654+0000 7f114d400700 1 --2- 172.21.15.12:0/977311330 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f1130038e00 0x7f113003b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.655+0000 7f114d400700 1 --2- 172.21.15.12:0/977311330 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f1130038e00 0x7f113003b2c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1138006fd0 tx=0x7f1138006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.660+0000 7f1142a00700 1 -- 172.21.15.12:0/977311330 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f113c003c90 con 0x7f114810b0f0 2024-03-23T22:50:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.977+0000 7f114f200700 1 -- 172.21.15.12:0/977311330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f11481a11f0 con 0x7f114810b0f0 2024-03-23T22:50:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.978+0000 7f1142a00700 1 -- 172.21.15.12:0/977311330 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f113c022d90 con 0x7f114810b0f0 2024-03-23T22:50:36.628 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.983+0000 7f114f200700 1 -- 172.21.15.12:0/977311330 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f1130038e00 msgr2=0x7f113003b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:36.629 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.983+0000 7f114f200700 1 --2- 172.21.15.12:0/977311330 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f1130038e00 0x7f113003b2c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1138006fd0 tx=0x7f1138006e40 comp rx=0 tx=0).stop 2024-03-23T22:50:36.629 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.983+0000 7f114f200700 1 -- 172.21.15.12:0/977311330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f114810b0f0 msgr2=0x7f114819db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:36.629 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.983+0000 7f114f200700 1 --2- 172.21.15.12:0/977311330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f114810b0f0 0x7f114819db60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f113c011850 tx=0x7f113c011c10 comp rx=0 tx=0).stop 2024-03-23T22:50:36.629 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.983+0000 7f114f200700 1 -- 172.21.15.12:0/977311330 shutdown_connections 2024-03-23T22:50:36.629 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.983+0000 7f114f200700 1 --2- 172.21.15.12:0/977311330 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f1130038e00 0x7f113003b2c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:36.629 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.983+0000 7f114f200700 1 --2- 172.21.15.12:0/977311330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f114810b0f0 0x7f114819db60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:36.629 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.983+0000 7f114f200700 1 -- 172.21.15.12:0/977311330 >> 172.21.15.12:0/977311330 conn(0x7f11481054a0 msgr2=0x7f1148106ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:36.629 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.983+0000 7f114f200700 1 -- 172.21.15.12:0/977311330 shutdown_connections 2024-03-23T22:50:36.629 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:35.983+0000 7f114f200700 1 -- 172.21.15.12:0/977311330 wait complete. 2024-03-23T22:50:36.630 INFO:teuthology.orchestra.run.smithi012.stdout:mgr is available 2024-03-23T22:50:37.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:37 smithi012 ceph-mon[31695]: mgrmap e4: x(active, since 2s) 2024-03-23T22:50:38.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:38 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2672941521' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-23T22:50:38.364 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-23T22:50:38.364 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [global] 2024-03-23T22:50:38.365 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout fsid = 90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T22:50:38.365 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-23T22:50:38.365 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-23T22:50:38.365 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-23T22:50:38.365 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-23T22:50:38.365 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-23T22:50:38.365 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-23T22:50:38.365 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-23T22:50:38.365 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-23T22:50:38.365 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [osd] 2024-03-23T22:50:38.365 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-23T22:50:38.365 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-23T22:50:38.365 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-23T22:50:38.365 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-23T22:50:38.366 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-23T22:50:38.366 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.484+0000 7f3063800700 1 Processor -- start 2024-03-23T22:50:38.366 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.484+0000 7f3063800700 1 -- start start 2024-03-23T22:50:38.366 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.484+0000 7f3063800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f305c1062e0 0x7f305c106700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:38.366 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.485+0000 7f3063800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305c07adf0 con 0x7f305c1062e0 2024-03-23T22:50:38.366 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.485+0000 7f3060e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f305c1062e0 0x7f305c106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:38.366 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.485+0000 7f3060e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f305c1062e0 0x7f305c106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34832/0 (socket says 172.21.15.12:34832) 2024-03-23T22:50:38.366 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.485+0000 7f3060e00700 1 -- 172.21.15.12:0/615218479 learned_addr learned my addr 172.21.15.12:0/615218479 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:38.366 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.485+0000 7f3060e00700 1 -- 172.21.15.12:0/615218479 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305c106c40 con 0x7f305c1062e0 2024-03-23T22:50:38.366 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.486+0000 7f3060e00700 1 --2- 172.21.15.12:0/615218479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f305c1062e0 0x7f305c106700 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f30500098a0 tx=0x7f3050009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a63bc0eca1018191 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:38.366 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.486+0000 7f305b400700 1 -- 172.21.15.12:0/615218479 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3050004160 con 0x7f305c1062e0 2024-03-23T22:50:38.366 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.486+0000 7f305b400700 1 -- 172.21.15.12:0/615218479 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3050004300 con 0x7f305c1062e0 2024-03-23T22:50:38.366 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.486+0000 7f305b400700 1 -- 172.21.15.12:0/615218479 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3050011b50 con 0x7f305c1062e0 2024-03-23T22:50:38.366 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.487+0000 7f3063800700 1 -- 172.21.15.12:0/615218479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f305c1062e0 msgr2=0x7f305c106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:38.367 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.487+0000 7f3063800700 1 --2- 172.21.15.12:0/615218479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f305c1062e0 0x7f305c106700 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f30500098a0 tx=0x7f3050009bb0 comp rx=0 tx=0).stop 2024-03-23T22:50:38.367 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.487+0000 7f3063800700 1 -- 172.21.15.12:0/615218479 shutdown_connections 2024-03-23T22:50:38.367 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.487+0000 7f3063800700 1 --2- 172.21.15.12:0/615218479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f305c1062e0 0x7f305c106700 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:38.367 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.487+0000 7f3063800700 1 -- 172.21.15.12:0/615218479 >> 172.21.15.12:0/615218479 conn(0x7f305c101f60 msgr2=0x7f305c104380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:38.367 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.487+0000 7f3063800700 1 -- 172.21.15.12:0/615218479 shutdown_connections 2024-03-23T22:50:38.367 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.487+0000 7f3063800700 1 -- 172.21.15.12:0/615218479 wait complete. 2024-03-23T22:50:38.367 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.488+0000 7f3063800700 1 Processor -- start 2024-03-23T22:50:38.367 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.488+0000 7f3063800700 1 -- start start 2024-03-23T22:50:38.367 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.489+0000 7f3063800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f305c1062e0 0x7f305c19dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:38.367 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.489+0000 7f3063800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305c107cd0 con 0x7f305c1062e0 2024-03-23T22:50:38.367 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.489+0000 7f3060e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f305c1062e0 0x7f305c19dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:38.367 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.489+0000 7f3060e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f305c1062e0 0x7f305c19dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34836/0 (socket says 172.21.15.12:34836) 2024-03-23T22:50:38.367 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.489+0000 7f3060e00700 1 -- 172.21.15.12:0/2672941521 learned_addr learned my addr 172.21.15.12:0/2672941521 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:38.367 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.489+0000 7f3060e00700 1 -- 172.21.15.12:0/2672941521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305c19e0f0 con 0x7f305c1062e0 2024-03-23T22:50:38.368 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.490+0000 7f3060e00700 1 --2- 172.21.15.12:0/2672941521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f305c1062e0 0x7f305c19dbb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3050011b60 tx=0x7f3050011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:38.368 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.490+0000 7f3059600700 1 -- 172.21.15.12:0/2672941521 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3050003970 con 0x7f305c1062e0 2024-03-23T22:50:38.368 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.490+0000 7f3059600700 1 -- 172.21.15.12:0/2672941521 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3050003b10 con 0x7f305c1062e0 2024-03-23T22:50:38.368 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.490+0000 7f3059600700 1 -- 172.21.15.12:0/2672941521 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3050024610 con 0x7f305c1062e0 2024-03-23T22:50:38.368 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.490+0000 7f3063800700 1 -- 172.21.15.12:0/2672941521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f305c19e380 con 0x7f305c1062e0 2024-03-23T22:50:38.368 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.490+0000 7f3063800700 1 -- 172.21.15.12:0/2672941521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f305c1a1070 con 0x7f305c1062e0 2024-03-23T22:50:38.368 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.491+0000 7f3059600700 1 -- 172.21.15.12:0/2672941521 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f305001a9f0 con 0x7f305c1062e0 2024-03-23T22:50:38.368 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.491+0000 7f3063800700 1 -- 172.21.15.12:0/2672941521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f305c0637c0 con 0x7f305c1062e0 2024-03-23T22:50:38.368 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.492+0000 7f3059600700 1 --2- 172.21.15.12:0/2672941521 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f3044038f20 0x7f304403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:38.368 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.492+0000 7f3059600700 1 -- 172.21.15.12:0/2672941521 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3050011e00 con 0x7f305c1062e0 2024-03-23T22:50:38.368 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.492+0000 7f305be00700 1 --2- 172.21.15.12:0/2672941521 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f3044038f20 0x7f304403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:38.368 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.493+0000 7f305be00700 1 --2- 172.21.15.12:0/2672941521 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f3044038f20 0x7f304403b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3048006fd0 tx=0x7f3048006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:38.368 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.497+0000 7f3059600700 1 -- 172.21.15.12:0/2672941521 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3050019e10 con 0x7f305c1062e0 2024-03-23T22:50:38.368 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.722+0000 7f3063800700 1 -- 172.21.15.12:0/2672941521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f305c0008d0 con 0x7f305c1062e0 2024-03-23T22:50:38.368 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.724+0000 7f3059600700 1 -- 172.21.15.12:0/2672941521 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f305004c9f0 con 0x7f305c1062e0 2024-03-23T22:50:38.369 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.726+0000 7f3063800700 1 -- 172.21.15.12:0/2672941521 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f3044038f20 msgr2=0x7f304403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:38.369 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.726+0000 7f3063800700 1 --2- 172.21.15.12:0/2672941521 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f3044038f20 0x7f304403b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3048006fd0 tx=0x7f3048006e40 comp rx=0 tx=0).stop 2024-03-23T22:50:38.369 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.727+0000 7f3063800700 1 -- 172.21.15.12:0/2672941521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f305c1062e0 msgr2=0x7f305c19dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:38.369 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.727+0000 7f3063800700 1 --2- 172.21.15.12:0/2672941521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f305c1062e0 0x7f305c19dbb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3050011b60 tx=0x7f3050011c40 comp rx=0 tx=0).stop 2024-03-23T22:50:38.369 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.727+0000 7f3063800700 1 -- 172.21.15.12:0/2672941521 shutdown_connections 2024-03-23T22:50:38.369 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.727+0000 7f3063800700 1 --2- 172.21.15.12:0/2672941521 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f3044038f20 0x7f304403b3e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:38.369 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.727+0000 7f3063800700 1 --2- 172.21.15.12:0/2672941521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f305c1062e0 0x7f305c19dbb0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:38.369 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.727+0000 7f3063800700 1 -- 172.21.15.12:0/2672941521 >> 172.21.15.12:0/2672941521 conn(0x7f305c101f60 msgr2=0x7f305c1947f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:38.369 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.727+0000 7f3063800700 1 -- 172.21.15.12:0/2672941521 shutdown_connections 2024-03-23T22:50:38.369 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:37.727+0000 7f3063800700 1 -- 172.21.15.12:0/2672941521 wait complete. 2024-03-23T22:50:38.369 INFO:teuthology.orchestra.run.smithi012.stdout:Enabling cephadm module... 2024-03-23T22:50:39.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:39 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2767487488' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-23T22:50:39.554 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:39 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: ignoring --setuser ceph since I am not root 2024-03-23T22:50:39.554 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:39 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: ignoring --setgroup ceph since I am not root 2024-03-23T22:50:40.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.973+0000 7f8f4b400700 1 Processor -- start 2024-03-23T22:50:40.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.973+0000 7f8f4b400700 1 -- start start 2024-03-23T22:50:40.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.974+0000 7f8f4b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f441068f0 0x7f8f44106d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:40.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.974+0000 7f8f4b400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f441072e0 con 0x7f8f441068f0 2024-03-23T22:50:40.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.974+0000 7f8f48a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f441068f0 0x7f8f44106d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:40.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.975+0000 7f8f48a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f441068f0 0x7f8f44106d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34848/0 (socket says 172.21.15.12:34848) 2024-03-23T22:50:40.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.975+0000 7f8f48a00700 1 -- 172.21.15.12:0/1139698767 learned_addr learned my addr 172.21.15.12:0/1139698767 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:40.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.975+0000 7f8f48a00700 1 -- 172.21.15.12:0/1139698767 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f44107af0 con 0x7f8f441068f0 2024-03-23T22:50:40.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.975+0000 7f8f48a00700 1 --2- 172.21.15.12:0/1139698767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f441068f0 0x7f8f44106d10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8f380098a0 tx=0x7f8f38009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac92f47d15996157 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:40.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.976+0000 7f8f43400700 1 -- 172.21.15.12:0/1139698767 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f38004160 con 0x7f8f441068f0 2024-03-23T22:50:40.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.976+0000 7f8f43400700 1 -- 172.21.15.12:0/1139698767 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f38004300 con 0x7f8f441068f0 2024-03-23T22:50:40.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.976+0000 7f8f43400700 1 -- 172.21.15.12:0/1139698767 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f38011b50 con 0x7f8f441068f0 2024-03-23T22:50:40.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.977+0000 7f8f4b400700 1 -- 172.21.15.12:0/1139698767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f441068f0 msgr2=0x7f8f44106d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:40.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.977+0000 7f8f4b400700 1 --2- 172.21.15.12:0/1139698767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f441068f0 0x7f8f44106d10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8f380098a0 tx=0x7f8f38009bb0 comp rx=0 tx=0).stop 2024-03-23T22:50:40.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.977+0000 7f8f4b400700 1 -- 172.21.15.12:0/1139698767 shutdown_connections 2024-03-23T22:50:40.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.977+0000 7f8f4b400700 1 --2- 172.21.15.12:0/1139698767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f441068f0 0x7f8f44106d10 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:40.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.977+0000 7f8f4b400700 1 -- 172.21.15.12:0/1139698767 >> 172.21.15.12:0/1139698767 conn(0x7f8f44101f70 msgr2=0x7f8f44104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:40.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.977+0000 7f8f4b400700 1 -- 172.21.15.12:0/1139698767 shutdown_connections 2024-03-23T22:50:40.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.977+0000 7f8f4b400700 1 -- 172.21.15.12:0/1139698767 wait complete. 2024-03-23T22:50:40.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.978+0000 7f8f4b400700 1 Processor -- start 2024-03-23T22:50:40.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.978+0000 7f8f4b400700 1 -- start start 2024-03-23T22:50:40.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.979+0000 7f8f4b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f441068f0 0x7f8f4419b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:40.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.979+0000 7f8f4b400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4410aea0 con 0x7f8f441068f0 2024-03-23T22:50:40.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.979+0000 7f8f48a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f441068f0 0x7f8f4419b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:40.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.979+0000 7f8f48a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f441068f0 0x7f8f4419b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34852/0 (socket says 172.21.15.12:34852) 2024-03-23T22:50:40.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.979+0000 7f8f48a00700 1 -- 172.21.15.12:0/2767487488 learned_addr learned my addr 172.21.15.12:0/2767487488 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:40.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.979+0000 7f8f48a00700 1 -- 172.21.15.12:0/2767487488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f4419bed0 con 0x7f8f441068f0 2024-03-23T22:50:40.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.980+0000 7f8f48a00700 1 --2- 172.21.15.12:0/2767487488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f441068f0 0x7f8f4419b990 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8f38011670 tx=0x7f8f38011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:40.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.980+0000 7f8f41600700 1 -- 172.21.15.12:0/2767487488 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f380039c0 con 0x7f8f441068f0 2024-03-23T22:50:40.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.980+0000 7f8f41600700 1 -- 172.21.15.12:0/2767487488 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f38011e50 con 0x7f8f441068f0 2024-03-23T22:50:40.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.980+0000 7f8f41600700 1 -- 172.21.15.12:0/2767487488 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f3801a5c0 con 0x7f8f441068f0 2024-03-23T22:50:40.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.980+0000 7f8f4b400700 1 -- 172.21.15.12:0/2767487488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f4419c160 con 0x7f8f441068f0 2024-03-23T22:50:40.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.980+0000 7f8f4b400700 1 -- 172.21.15.12:0/2767487488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f4419ee50 con 0x7f8f441068f0 2024-03-23T22:50:40.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.982+0000 7f8f4b400700 1 -- 172.21.15.12:0/2767487488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f440637c0 con 0x7f8f441068f0 2024-03-23T22:50:40.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.982+0000 7f8f41600700 1 -- 172.21.15.12:0/2767487488 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f8f38024430 con 0x7f8f441068f0 2024-03-23T22:50:40.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.982+0000 7f8f41600700 1 --2- 172.21.15.12:0/2767487488 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f8f2c038e80 0x7f8f2c03b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:40.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.982+0000 7f8f41600700 1 -- 172.21.15.12:0/2767487488 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8f3804d140 con 0x7f8f441068f0 2024-03-23T22:50:40.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.982+0000 7f8f43e00700 1 --2- 172.21.15.12:0/2767487488 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f8f2c038e80 0x7f8f2c03b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:40.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.983+0000 7f8f43e00700 1 --2- 172.21.15.12:0/2767487488 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f8f2c038e80 0x7f8f2c03b340 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8f30006fd0 tx=0x7f8f30006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:40.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:38.987+0000 7f8f41600700 1 -- 172.21.15.12:0/2767487488 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8f3802d3f0 con 0x7f8f441068f0 2024-03-23T22:50:40.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:39.273+0000 7f8f4b400700 1 -- 172.21.15.12:0/2767487488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f8f4419f020 con 0x7f8f441068f0 2024-03-23T22:50:40.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:39.327+0000 7f8f41600700 1 -- 172.21.15.12:0/2767487488 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f8f4419f020 con 0x7f8f441068f0 2024-03-23T22:50:40.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:39.331+0000 7f8f4b400700 1 -- 172.21.15.12:0/2767487488 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f8f2c038e80 msgr2=0x7f8f2c03b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:40.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:39.331+0000 7f8f4b400700 1 --2- 172.21.15.12:0/2767487488 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f8f2c038e80 0x7f8f2c03b340 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8f30006fd0 tx=0x7f8f30006e40 comp rx=0 tx=0).stop 2024-03-23T22:50:40.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:39.331+0000 7f8f41600700 1 -- 172.21.15.12:0/2767487488 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f8f38024720 con 0x7f8f441068f0 2024-03-23T22:50:40.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:39.331+0000 7f8f4b400700 1 -- 172.21.15.12:0/2767487488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f441068f0 msgr2=0x7f8f4419b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:40.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:39.331+0000 7f8f4b400700 1 --2- 172.21.15.12:0/2767487488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f441068f0 0x7f8f4419b990 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8f38011670 tx=0x7f8f38011b60 comp rx=0 tx=0).stop 2024-03-23T22:50:40.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:39.331+0000 7f8f4b400700 1 -- 172.21.15.12:0/2767487488 shutdown_connections 2024-03-23T22:50:40.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:39.331+0000 7f8f4b400700 1 --2- 172.21.15.12:0/2767487488 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f8f2c038e80 0x7f8f2c03b340 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:40.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:39.331+0000 7f8f4b400700 1 --2- 172.21.15.12:0/2767487488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f441068f0 0x7f8f4419b990 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:40.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:39.331+0000 7f8f4b400700 1 -- 172.21.15.12:0/2767487488 >> 172.21.15.12:0/2767487488 conn(0x7f8f44101f70 msgr2=0x7f8f44102a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:40.006 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:39.332+0000 7f8f4b400700 1 -- 172.21.15.12:0/2767487488 shutdown_connections 2024-03-23T22:50:40.006 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:39.332+0000 7f8f4b400700 1 -- 172.21.15.12:0/2767487488 wait complete. 2024-03-23T22:50:40.270 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:40 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:40.143+0000 7fdacdd63200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-23T22:50:40.523 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:40 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2767487488' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-23T22:50:40.523 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:40 smithi012 ceph-mon[31695]: mgrmap e5: x(active, since 5s) 2024-03-23T22:50:40.523 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:40 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:40.269+0000 7fdacdd63200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-23T22:50:41.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:41 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3071510520' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-23T22:50:41.398 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:41 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:41.088+0000 7fdacdd63200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-23T22:50:41.398 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:41 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:41.214+0000 7fdacdd63200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-23T22:50:41.398 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:41 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:41.397+0000 7fdacdd63200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-23T22:50:41.559 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-03-23T22:50:41.559 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-23T22:50:41.559 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-23T22:50:41.559 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-23T22:50:41.559 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-23T22:50:41.560 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-03-23T22:50:41.560 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.589+0000 7f721a800700 1 Processor -- start 2024-03-23T22:50:41.560 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.590+0000 7f721a800700 1 -- start start 2024-03-23T22:50:41.560 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.590+0000 7f721a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721407b2a0 0x7f721407b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:41.560 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.590+0000 7f721a800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721407cfd0 con 0x7f721407b2a0 2024-03-23T22:50:41.560 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.591+0000 7f7213400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721407b2a0 0x7f721407b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:41.560 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.591+0000 7f7213400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721407b2a0 0x7f721407b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34870/0 (socket says 172.21.15.12:34870) 2024-03-23T22:50:41.560 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.591+0000 7f7213400700 1 -- 172.21.15.12:0/3528320616 learned_addr learned my addr 172.21.15.12:0/3528320616 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:41.560 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.591+0000 7f7213400700 1 -- 172.21.15.12:0/3528320616 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721407bc00 con 0x7f721407b2a0 2024-03-23T22:50:41.560 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.591+0000 7f7213400700 1 --2- 172.21.15.12:0/3528320616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721407b2a0 0x7f721407b6c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f72040098a0 tx=0x7f7204009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf1fd5b489a21cd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:41.560 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.592+0000 7f7212000700 1 -- 172.21.15.12:0/3528320616 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7204004160 con 0x7f721407b2a0 2024-03-23T22:50:41.560 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.592+0000 7f7212000700 1 -- 172.21.15.12:0/3528320616 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7204004300 con 0x7f721407b2a0 2024-03-23T22:50:41.560 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.592+0000 7f7212000700 1 -- 172.21.15.12:0/3528320616 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7204011b50 con 0x7f721407b2a0 2024-03-23T22:50:41.561 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.592+0000 7f721a800700 1 -- 172.21.15.12:0/3528320616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721407b2a0 msgr2=0x7f721407b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:41.561 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.592+0000 7f721a800700 1 --2- 172.21.15.12:0/3528320616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721407b2a0 0x7f721407b6c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f72040098a0 tx=0x7f7204009bb0 comp rx=0 tx=0).stop 2024-03-23T22:50:41.561 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.592+0000 7f721a800700 1 -- 172.21.15.12:0/3528320616 shutdown_connections 2024-03-23T22:50:41.561 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.592+0000 7f721a800700 1 --2- 172.21.15.12:0/3528320616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721407b2a0 0x7f721407b6c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:41.561 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.592+0000 7f721a800700 1 -- 172.21.15.12:0/3528320616 >> 172.21.15.12:0/3528320616 conn(0x7f7214103000 msgr2=0x7f7214105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:41.561 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.592+0000 7f721a800700 1 -- 172.21.15.12:0/3528320616 shutdown_connections 2024-03-23T22:50:41.561 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.593+0000 7f721a800700 1 -- 172.21.15.12:0/3528320616 wait complete. 2024-03-23T22:50:41.561 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.593+0000 7f721a800700 1 Processor -- start 2024-03-23T22:50:41.561 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.594+0000 7f721a800700 1 -- start start 2024-03-23T22:50:41.561 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.594+0000 7f721a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721407b2a0 0x7f721419dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:41.561 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.594+0000 7f721a800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721410e0f0 con 0x7f721407b2a0 2024-03-23T22:50:41.561 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.594+0000 7f7213400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721407b2a0 0x7f721419dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:41.561 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.594+0000 7f7213400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721407b2a0 0x7f721419dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34886/0 (socket says 172.21.15.12:34886) 2024-03-23T22:50:41.562 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.594+0000 7f7213400700 1 -- 172.21.15.12:0/3071510520 learned_addr learned my addr 172.21.15.12:0/3071510520 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:41.562 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.595+0000 7f7213400700 1 -- 172.21.15.12:0/3071510520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721419e230 con 0x7f721407b2a0 2024-03-23T22:50:41.562 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.595+0000 7f7213400700 1 --2- 172.21.15.12:0/3071510520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721407b2a0 0x7f721419dcf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7204009fb0 tx=0x7f7204011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:41.562 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.595+0000 7f720be00700 1 -- 172.21.15.12:0/3071510520 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f72040038b0 con 0x7f721407b2a0 2024-03-23T22:50:41.562 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.595+0000 7f720be00700 1 -- 172.21.15.12:0/3071510520 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7204011e50 con 0x7f721407b2a0 2024-03-23T22:50:41.562 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.595+0000 7f720be00700 1 -- 172.21.15.12:0/3071510520 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7204019420 con 0x7f721407b2a0 2024-03-23T22:50:41.562 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.595+0000 7f721a800700 1 -- 172.21.15.12:0/3071510520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f721419e4c0 con 0x7f721407b2a0 2024-03-23T22:50:41.562 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.595+0000 7f721a800700 1 -- 172.21.15.12:0/3071510520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72141a11a0 con 0x7f721407b2a0 2024-03-23T22:50:41.562 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.596+0000 7f720be00700 1 -- 172.21.15.12:0/3071510520 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f7204003a50 con 0x7f721407b2a0 2024-03-23T22:50:41.562 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.597+0000 7f721a800700 1 -- 172.21.15.12:0/3071510520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72140637c0 con 0x7f721407b2a0 2024-03-23T22:50:41.562 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.597+0000 7f720be00700 1 --2- 172.21.15.12:0/3071510520 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f71f8038f20 0x7f71f803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:41.562 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.597+0000 7f720be00700 1 -- 172.21.15.12:0/3071510520 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f720404cfe0 con 0x7f721407b2a0 2024-03-23T22:50:41.562 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.597+0000 7f7212a00700 1 -- 172.21.15.12:0/3071510520 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f71f8038f20 msgr2=0x7f71f803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/3270387158 2024-03-23T22:50:41.562 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.597+0000 7f7212a00700 1 --2- 172.21.15.12:0/3071510520 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f71f8038f20 0x7f71f803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T22:50:41.563 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.603+0000 7f720be00700 1 -- 172.21.15.12:0/3071510520 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7204016260 con 0x7f721407b2a0 2024-03-23T22:50:41.563 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.798+0000 7f7212a00700 1 -- 172.21.15.12:0/3071510520 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f71f8038f20 msgr2=0x7f71f803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/3270387158 2024-03-23T22:50:41.563 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.798+0000 7f7212a00700 1 --2- 172.21.15.12:0/3071510520 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f71f8038f20 0x7f71f803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T22:50:41.563 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.889+0000 7f721a800700 1 -- 172.21.15.12:0/3071510520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f72141a1370 con 0x7f721407b2a0 2024-03-23T22:50:41.563 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.890+0000 7f720be00700 1 -- 172.21.15.12:0/3071510520 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f720402ad70 con 0x7f721407b2a0 2024-03-23T22:50:41.563 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.893+0000 7f721a800700 1 -- 172.21.15.12:0/3071510520 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f71f8038f20 msgr2=0x7f71f803b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-23T22:50:41.563 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.893+0000 7f721a800700 1 --2- 172.21.15.12:0/3071510520 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f71f8038f20 0x7f71f803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:41.563 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.893+0000 7f721a800700 1 -- 172.21.15.12:0/3071510520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721407b2a0 msgr2=0x7f721419dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:41.563 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.893+0000 7f721a800700 1 --2- 172.21.15.12:0/3071510520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721407b2a0 0x7f721419dcf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7204009fb0 tx=0x7f7204011bf0 comp rx=0 tx=0).stop 2024-03-23T22:50:41.563 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.893+0000 7f721a800700 1 -- 172.21.15.12:0/3071510520 shutdown_connections 2024-03-23T22:50:41.563 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.893+0000 7f721a800700 1 --2- 172.21.15.12:0/3071510520 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7f71f8038f20 0x7f71f803b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:41.563 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.893+0000 7f721a800700 1 --2- 172.21.15.12:0/3071510520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721407b2a0 0x7f721419dcf0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:41.563 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.893+0000 7f721a800700 1 -- 172.21.15.12:0/3071510520 >> 172.21.15.12:0/3071510520 conn(0x7f7214103000 msgr2=0x7f7214103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:41.563 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.894+0000 7f721a800700 1 -- 172.21.15.12:0/3071510520 shutdown_connections 2024-03-23T22:50:41.564 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:40.894+0000 7f721a800700 1 -- 172.21.15.12:0/3071510520 wait complete. 2024-03-23T22:50:41.564 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for the mgr to restart... 2024-03-23T22:50:41.564 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mgr epoch 5... 2024-03-23T22:50:41.706 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:41 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:41.525+0000 7fdacdd63200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-23T22:50:41.707 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:41 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:41.640+0000 7fdacdd63200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-23T22:50:42.254 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:41 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:41.987+0000 7fdacdd63200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-23T22:50:42.254 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:42 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:42.137+0000 7fdacdd63200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-23T22:50:42.254 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:42 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:42.253+0000 7fdacdd63200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-23T22:50:42.553 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:42 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:42.380+0000 7fdacdd63200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-23T22:50:43.053 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:42 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:42.603+0000 7fdacdd63200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-23T22:50:44.304 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:43 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:43.922+0000 7fdacdd63200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-23T22:50:44.304 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:44 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:44.098+0000 7fdacdd63200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-23T22:50:44.804 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:44 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:44.525+0000 7fdacdd63200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-23T22:50:44.804 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:44 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:44.646+0000 7fdacdd63200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-23T22:50:45.168 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:44 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:44.912+0000 7fdacdd63200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-23T22:50:45.553 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:45 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:45.167+0000 7fdacdd63200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-23T22:50:45.882 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:45 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:45.556+0000 7fdacdd63200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-23T22:50:45.882 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:45 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:45.881+0000 7fdacdd63200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-23T22:50:46.553 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:46 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:46.196+0000 7fdacdd63200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-23T22:50:47.554 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:47 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:47.257+0000 7fdacdd63200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-23T22:50:48.303 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:47 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:47.973+0000 7fdacdd63200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-23T22:50:48.304 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:48 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:48.107+0000 7fdacdd63200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-23T22:50:48.804 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:48 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:50:48.396+0000 7fdacdd63200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-23T22:50:48.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:48 smithi012 ceph-mon[31695]: Active manager daemon x restarted 2024-03-23T22:50:48.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:48 smithi012 ceph-mon[31695]: Activating manager daemon x 2024-03-23T22:50:48.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:48 smithi012 ceph-mon[31695]: osdmap e2: 0 total, 0 up, 0 in 2024-03-23T22:50:48.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:48 smithi012 ceph-mon[31695]: mgrmap e6: x(active, starting, since 0.00790845s) 2024-03-23T22:50:48.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:48 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-23T22:50:48.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:48 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-23T22:50:48.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:48 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-23T22:50:48.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:48 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-23T22:50:48.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:48 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-23T22:50:48.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:48 smithi012 ceph-mon[31695]: Manager daemon x is now available 2024-03-23T22:50:48.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:48 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' 2024-03-23T22:50:48.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:48 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' 2024-03-23T22:50:49.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:49 smithi012 ceph-mon[31695]: Found migration_current of "None". Setting to last migration. 2024-03-23T22:50:49.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:49 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:50:49.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:49 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:50:49.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:49 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:50:49.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:49 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-23T22:50:49.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:49 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-23T22:50:49.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:49 smithi012 ceph-mon[31695]: mgrmap e7: x(active, since 1.01118s) 2024-03-23T22:50:50.187 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-03-23T22:50:50.187 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-23T22:50:50.187 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-23T22:50:50.187 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-03-23T22:50:50.187 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.127+0000 7fedbb200700 1 Processor -- start 2024-03-23T22:50:50.187 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.127+0000 7fedbb200700 1 -- start start 2024-03-23T22:50:50.188 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.128+0000 7fedbb200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedbc1082b0 0x7fedbc10a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:50.188 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.128+0000 7fedbb200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc07ae30 con 0x7fedbc1082b0 2024-03-23T22:50:50.188 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.128+0000 7fedb9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedbc1082b0 0x7fedbc10a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:50.188 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.128+0000 7fedb9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedbc1082b0 0x7fedbc10a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34900/0 (socket says 172.21.15.12:34900) 2024-03-23T22:50:50.188 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.128+0000 7fedb9e00700 1 -- 172.21.15.12:0/2978539459 learned_addr learned my addr 172.21.15.12:0/2978539459 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:50.188 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.129+0000 7fedb9e00700 1 -- 172.21.15.12:0/2978539459 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedbc10abe0 con 0x7fedbc1082b0 2024-03-23T22:50:50.188 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.129+0000 7fedb9e00700 1 --2- 172.21.15.12:0/2978539459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedbc1082b0 0x7fedbc10a6a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fedac0098a0 tx=0x7fedac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f2f4ef8b918adce server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:50.188 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.129+0000 7fedb8a00700 1 -- 172.21.15.12:0/2978539459 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fedac004160 con 0x7fedbc1082b0 2024-03-23T22:50:50.188 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.129+0000 7fedb8a00700 1 -- 172.21.15.12:0/2978539459 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fedac004300 con 0x7fedbc1082b0 2024-03-23T22:50:50.188 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.129+0000 7fedb8a00700 1 -- 172.21.15.12:0/2978539459 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fedac011b50 con 0x7fedbc1082b0 2024-03-23T22:50:50.188 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.130+0000 7fedbb200700 1 -- 172.21.15.12:0/2978539459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedbc1082b0 msgr2=0x7fedbc10a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:50.188 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.130+0000 7fedbb200700 1 --2- 172.21.15.12:0/2978539459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedbc1082b0 0x7fedbc10a6a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fedac0098a0 tx=0x7fedac009bb0 comp rx=0 tx=0).stop 2024-03-23T22:50:50.188 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.130+0000 7fedbb200700 1 -- 172.21.15.12:0/2978539459 shutdown_connections 2024-03-23T22:50:50.188 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.130+0000 7fedbb200700 1 --2- 172.21.15.12:0/2978539459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedbc1082b0 0x7fedbc10a6a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:50.189 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.130+0000 7fedbb200700 1 -- 172.21.15.12:0/2978539459 >> 172.21.15.12:0/2978539459 conn(0x7fedbc101d40 msgr2=0x7fedbc104160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:50.189 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.130+0000 7fedbb200700 1 -- 172.21.15.12:0/2978539459 shutdown_connections 2024-03-23T22:50:50.189 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.131+0000 7fedbb200700 1 -- 172.21.15.12:0/2978539459 wait complete. 2024-03-23T22:50:50.189 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.132+0000 7fedbb200700 1 Processor -- start 2024-03-23T22:50:50.189 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.132+0000 7fedbb200700 1 -- start start 2024-03-23T22:50:50.189 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.132+0000 7fedbb200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedbc1082b0 0x7fedbc19da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:50.189 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.132+0000 7fedbb200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc10bbb0 con 0x7fedbc1082b0 2024-03-23T22:50:50.189 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.133+0000 7fedb9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedbc1082b0 0x7fedbc19da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:50.189 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.133+0000 7fedb9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedbc1082b0 0x7fedbc19da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34906/0 (socket says 172.21.15.12:34906) 2024-03-23T22:50:50.189 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.133+0000 7fedb9e00700 1 -- 172.21.15.12:0/2264914566 learned_addr learned my addr 172.21.15.12:0/2264914566 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:50.189 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.133+0000 7fedb9e00700 1 -- 172.21.15.12:0/2264914566 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedbc19df50 con 0x7fedbc1082b0 2024-03-23T22:50:50.189 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.133+0000 7fedb9e00700 1 --2- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedbc1082b0 0x7fedbc19da10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fedac009fb0 tx=0x7fedac011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:50.189 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.134+0000 7fedb2a00700 1 -- 172.21.15.12:0/2264914566 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fedac0038b0 con 0x7fedbc1082b0 2024-03-23T22:50:50.189 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.134+0000 7fedb2a00700 1 -- 172.21.15.12:0/2264914566 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fedac011e50 con 0x7fedbc1082b0 2024-03-23T22:50:50.190 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.134+0000 7fedbb200700 1 -- 172.21.15.12:0/2264914566 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedbc19e1e0 con 0x7fedbc1082b0 2024-03-23T22:50:50.190 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.134+0000 7fedb2a00700 1 -- 172.21.15.12:0/2264914566 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fedac019420 con 0x7fedbc1082b0 2024-03-23T22:50:50.190 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.134+0000 7fedbb200700 1 -- 172.21.15.12:0/2264914566 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedbc1a0ed0 con 0x7fedbc1082b0 2024-03-23T22:50:50.190 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.135+0000 7fedb2a00700 1 -- 172.21.15.12:0/2264914566 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fedac01a430 con 0x7fedbc1082b0 2024-03-23T22:50:50.190 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.135+0000 7fedb2a00700 1 --2- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7feda0038ed0 0x7feda003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:50.190 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.135+0000 7fedb2a00700 1 -- 172.21.15.12:0/2264914566 --> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7feda003baa0 con 0x7feda0038ed0 2024-03-23T22:50:50.190 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.135+0000 7fedb2a00700 1 -- 172.21.15.12:0/2264914566 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fedac04d130 con 0x7fedbc1082b0 2024-03-23T22:50:50.190 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.136+0000 7fedb9400700 1 -- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7feda0038ed0 msgr2=0x7feda003b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/3270387158 2024-03-23T22:50:50.190 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.136+0000 7fedb9400700 1 --2- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7feda0038ed0 0x7feda003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T22:50:50.190 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.337+0000 7fedb9400700 1 -- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7feda0038ed0 msgr2=0x7feda003b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/3270387158 2024-03-23T22:50:50.190 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.337+0000 7fedb9400700 1 --2- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7feda0038ed0 0x7feda003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T22:50:50.190 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.738+0000 7fedb9400700 1 -- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7feda0038ed0 msgr2=0x7feda003b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/3270387158 2024-03-23T22:50:50.190 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:42.738+0000 7fedb9400700 1 --2- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7feda0038ed0 0x7feda003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-23T22:50:50.191 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:43.539+0000 7fedb9400700 1 -- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7feda0038ed0 msgr2=0x7feda003b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/3270387158 2024-03-23T22:50:50.191 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:43.539+0000 7fedb9400700 1 --2- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7feda0038ed0 0x7feda003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-23T22:50:50.191 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:45.141+0000 7fedb9400700 1 -- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7feda0038ed0 msgr2=0x7feda003b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/3270387158 2024-03-23T22:50:50.191 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:45.141+0000 7fedb9400700 1 --2- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7feda0038ed0 0x7feda003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-23T22:50:50.191 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:48.345+0000 7fedb9400700 1 -- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7feda0038ed0 msgr2=0x7feda003b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/3270387158 2024-03-23T22:50:50.191 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:48.345+0000 7fedb9400700 1 --2- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7feda0038ed0 0x7feda003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-23T22:50:50.191 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:48.407+0000 7fedb2a00700 1 -- 172.21.15.12:0/2264914566 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mgrmap(e 6) v1 ==== 45873+0+0 (secure 0 0 0) 0x7fedac019720 con 0x7fedbc1082b0 2024-03-23T22:50:50.191 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:48.407+0000 7fedb2a00700 1 -- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7feda0038ed0 msgr2=0x7feda003b390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-23T22:50:50.191 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:48.407+0000 7fedb2a00700 1 --2- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7feda0038ed0 0x7feda003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:50.191 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:49.409+0000 7fedb2a00700 1 -- 172.21.15.12:0/2264914566 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7fedac04ef50 con 0x7fedbc1082b0 2024-03-23T22:50:50.191 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:49.409+0000 7fedb2a00700 1 --2- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7feda003c7c0 0x7feda003ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:50.191 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:49.409+0000 7fedb2a00700 1 -- 172.21.15.12:0/2264914566 --> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7feda003baa0 con 0x7feda003c7c0 2024-03-23T22:50:50.191 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:49.410+0000 7fedb9400700 1 --2- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7feda003c7c0 0x7feda003ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:50.191 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:49.416+0000 7fedb9400700 1 --2- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7feda003c7c0 0x7feda003ebb0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7feda8003a10 tx=0x7feda8009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:50.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:49.418+0000 7fedb2a00700 1 -- 172.21.15.12:0/2264914566 <== mgr.14120 v2:172.21.15.12:6800/4127975083 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7feda003baa0 con 0x7feda003c7c0 2024-03-23T22:50:50.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:49.426+0000 7fedbb200700 1 -- 172.21.15.12:0/2264914566 --> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fedbc0637c0 con 0x7feda003c7c0 2024-03-23T22:50:50.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:49.426+0000 7fedb2a00700 1 -- 172.21.15.12:0/2264914566 <== mgr.14120 v2:172.21.15.12:6800/4127975083 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fedbc0637c0 con 0x7feda003c7c0 2024-03-23T22:50:50.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:49.427+0000 7fedbb200700 1 -- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7feda003c7c0 msgr2=0x7feda003ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:50.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:49.427+0000 7fedbb200700 1 --2- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7feda003c7c0 0x7feda003ebb0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7feda8003a10 tx=0x7feda8009290 comp rx=0 tx=0).stop 2024-03-23T22:50:50.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:49.427+0000 7fedbb200700 1 -- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedbc1082b0 msgr2=0x7fedbc19da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:50.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:49.427+0000 7fedbb200700 1 --2- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedbc1082b0 0x7fedbc19da10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fedac009fb0 tx=0x7fedac011bf0 comp rx=0 tx=0).stop 2024-03-23T22:50:50.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:49.427+0000 7fedbb200700 1 -- 172.21.15.12:0/2264914566 shutdown_connections 2024-03-23T22:50:50.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:49.427+0000 7fedbb200700 1 --2- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7feda003c7c0 0x7feda003ebb0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:50.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:49.427+0000 7fedbb200700 1 --2- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fedbc1082b0 0x7fedbc19da10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:50.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:49.427+0000 7fedbb200700 1 --2- 172.21.15.12:0/2264914566 >> [v2:172.21.15.12:6800/3270387158,v1:172.21.15.12:6801/3270387158] conn(0x7feda0038ed0 0x7feda003b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:50.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:49.427+0000 7fedbb200700 1 -- 172.21.15.12:0/2264914566 >> 172.21.15.12:0/2264914566 conn(0x7fedbc101d40 msgr2=0x7fedbc102890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:50.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:49.427+0000 7fedbb200700 1 -- 172.21.15.12:0/2264914566 shutdown_connections 2024-03-23T22:50:50.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:49.427+0000 7fedbb200700 1 -- 172.21.15.12:0/2264914566 wait complete. 2024-03-23T22:50:50.193 INFO:teuthology.orchestra.run.smithi012.stdout:mgr epoch 5 is available 2024-03-23T22:50:50.193 INFO:teuthology.orchestra.run.smithi012.stdout:Setting orchestrator backend to cephadm... 2024-03-23T22:50:50.715 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:50 smithi012 ceph-mon[31695]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-23T22:50:50.715 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:50 smithi012 ceph-mon[31695]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-23T22:50:50.715 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:50 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' 2024-03-23T22:50:50.715 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:50 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' 2024-03-23T22:50:50.715 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:50 smithi012 ceph-mon[31695]: [23/Mar/2024:22:50:50] ENGINE Bus STARTING 2024-03-23T22:50:51.606 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:51 smithi012 ceph-mon[31695]: [23/Mar/2024:22:50:50] ENGINE Serving on http://172.21.15.12:8765 2024-03-23T22:50:51.606 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:51 smithi012 ceph-mon[31695]: [23/Mar/2024:22:50:50] ENGINE Serving on https://172.21.15.12:7150 2024-03-23T22:50:51.606 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:51 smithi012 ceph-mon[31695]: [23/Mar/2024:22:50:50] ENGINE Bus STARTED 2024-03-23T22:50:51.606 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:51 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:50:51.606 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:51 smithi012 ceph-mon[31695]: mgrmap e8: x(active, since 2s) 2024-03-23T22:50:51.606 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:51 smithi012 ceph-mon[31695]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:50:51.606 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:51 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' 2024-03-23T22:50:51.607 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:51 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:50:51.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.715+0000 7f07ed600700 1 Processor -- start 2024-03-23T22:50:51.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.715+0000 7f07ed600700 1 -- start start 2024-03-23T22:50:51.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.716+0000 7f07ed600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07e8107800 0x7f07e8107c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:51.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.716+0000 7f07ed600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e81081f0 con 0x7f07e8107800 2024-03-23T22:50:51.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.716+0000 7f07e7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07e8107800 0x7f07e8107c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:51.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.716+0000 7f07e7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07e8107800 0x7f07e8107c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46602/0 (socket says 172.21.15.12:46602) 2024-03-23T22:50:51.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.717+0000 7f07e7e00700 1 -- 172.21.15.12:0/3610521890 learned_addr learned my addr 172.21.15.12:0/3610521890 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:51.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.717+0000 7f07e7e00700 1 -- 172.21.15.12:0/3610521890 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e81089b0 con 0x7f07e8107800 2024-03-23T22:50:51.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.717+0000 7f07e7e00700 1 --2- 172.21.15.12:0/3610521890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07e8107800 0x7f07e8107c20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f07d80098a0 tx=0x7f07d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c52dfecd88bc2af server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:51.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.718+0000 7f07e6a00700 1 -- 172.21.15.12:0/3610521890 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f07d8004160 con 0x7f07e8107800 2024-03-23T22:50:51.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.718+0000 7f07e6a00700 1 -- 172.21.15.12:0/3610521890 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f07d8004300 con 0x7f07e8107800 2024-03-23T22:50:51.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.718+0000 7f07e6a00700 1 -- 172.21.15.12:0/3610521890 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f07d8011b50 con 0x7f07e8107800 2024-03-23T22:50:51.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.718+0000 7f07ed600700 1 -- 172.21.15.12:0/3610521890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07e8107800 msgr2=0x7f07e8107c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:51.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.718+0000 7f07ed600700 1 --2- 172.21.15.12:0/3610521890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07e8107800 0x7f07e8107c20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f07d80098a0 tx=0x7f07d8009bb0 comp rx=0 tx=0).stop 2024-03-23T22:50:51.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.718+0000 7f07ed600700 1 -- 172.21.15.12:0/3610521890 shutdown_connections 2024-03-23T22:50:51.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.718+0000 7f07ed600700 1 --2- 172.21.15.12:0/3610521890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07e8107800 0x7f07e8107c20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:51.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.718+0000 7f07ed600700 1 -- 172.21.15.12:0/3610521890 >> 172.21.15.12:0/3610521890 conn(0x7f07e8102e20 msgr2=0x7f07e8105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:51.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.719+0000 7f07ed600700 1 -- 172.21.15.12:0/3610521890 shutdown_connections 2024-03-23T22:50:51.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.719+0000 7f07ed600700 1 -- 172.21.15.12:0/3610521890 wait complete. 2024-03-23T22:50:51.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.720+0000 7f07ed600700 1 Processor -- start 2024-03-23T22:50:51.620 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.720+0000 7f07ed600700 1 -- start start 2024-03-23T22:50:51.620 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.720+0000 7f07ed600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07e8107800 0x7f07e807cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:51.620 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.720+0000 7f07ed600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e810bd60 con 0x7f07e8107800 2024-03-23T22:50:51.620 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.721+0000 7f07e7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07e8107800 0x7f07e807cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:51.620 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.721+0000 7f07e7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07e8107800 0x7f07e807cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46608/0 (socket says 172.21.15.12:46608) 2024-03-23T22:50:51.621 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.721+0000 7f07e7e00700 1 -- 172.21.15.12:0/1902519018 learned_addr learned my addr 172.21.15.12:0/1902519018 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:51.621 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.721+0000 7f07e7e00700 1 -- 172.21.15.12:0/1902519018 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e807d090 con 0x7f07e8107800 2024-03-23T22:50:51.621 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.721+0000 7f07e7e00700 1 --2- 172.21.15.12:0/1902519018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07e8107800 0x7f07e807cb50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f07d8011670 tx=0x7f07d8011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:51.621 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.722+0000 7f07e4c00700 1 -- 172.21.15.12:0/1902519018 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f07d80039c0 con 0x7f07e8107800 2024-03-23T22:50:51.621 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.722+0000 7f07e4c00700 1 -- 172.21.15.12:0/1902519018 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f07d8011e50 con 0x7f07e8107800 2024-03-23T22:50:51.621 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.722+0000 7f07ed600700 1 -- 172.21.15.12:0/1902519018 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07e807b280 con 0x7f07e8107800 2024-03-23T22:50:51.621 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.722+0000 7f07e4c00700 1 -- 172.21.15.12:0/1902519018 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f07d801a5c0 con 0x7f07e8107800 2024-03-23T22:50:51.621 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.722+0000 7f07ed600700 1 -- 172.21.15.12:0/1902519018 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07e807b7c0 con 0x7f07e8107800 2024-03-23T22:50:51.621 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.723+0000 7f07e4c00700 1 -- 172.21.15.12:0/1902519018 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7f07d8024430 con 0x7f07e8107800 2024-03-23T22:50:51.621 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.723+0000 7f07ed600700 1 -- 172.21.15.12:0/1902519018 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07e80637c0 con 0x7f07e8107800 2024-03-23T22:50:51.622 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.724+0000 7f07e4c00700 1 --2- 172.21.15.12:0/1902519018 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7f07cc038e10 0x7f07cc03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:51.622 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.724+0000 7f07e4c00700 1 -- 172.21.15.12:0/1902519018 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f07d804d080 con 0x7f07e8107800 2024-03-23T22:50:51.622 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.724+0000 7f07e7400700 1 --2- 172.21.15.12:0/1902519018 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7f07cc038e10 0x7f07cc03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:51.622 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.725+0000 7f07e7400700 1 --2- 172.21.15.12:0/1902519018 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7f07cc038e10 0x7f07cc03b2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f07d4006fd0 tx=0x7f07d4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:51.622 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.730+0000 7f07e4c00700 1 -- 172.21.15.12:0/1902519018 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f07d8017ca0 con 0x7f07e8107800 2024-03-23T22:50:51.622 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.752+0000 7f07e4c00700 1 -- 172.21.15.12:0/1902519018 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f07d800bc90 con 0x7f07e8107800 2024-03-23T22:50:51.622 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.974+0000 7f07ed600700 1 -- 172.21.15.12:0/1902519018 --> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f07e81040a0 con 0x7f07cc038e10 2024-03-23T22:50:51.622 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.988+0000 7f07e4c00700 1 -- 172.21.15.12:0/1902519018 <== mgr.14120 v2:172.21.15.12:6800/4127975083 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f07e81040a0 con 0x7f07cc038e10 2024-03-23T22:50:51.622 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.991+0000 7f07ed600700 1 -- 172.21.15.12:0/1902519018 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7f07cc038e10 msgr2=0x7f07cc03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:51.622 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.991+0000 7f07ed600700 1 --2- 172.21.15.12:0/1902519018 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7f07cc038e10 0x7f07cc03b2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f07d4006fd0 tx=0x7f07d4006e40 comp rx=0 tx=0).stop 2024-03-23T22:50:51.622 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.991+0000 7f07ed600700 1 -- 172.21.15.12:0/1902519018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07e8107800 msgr2=0x7f07e807cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:51.622 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.991+0000 7f07ed600700 1 --2- 172.21.15.12:0/1902519018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07e8107800 0x7f07e807cb50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f07d8011670 tx=0x7f07d8011b60 comp rx=0 tx=0).stop 2024-03-23T22:50:51.622 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.991+0000 7f07ed600700 1 -- 172.21.15.12:0/1902519018 shutdown_connections 2024-03-23T22:50:51.622 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.991+0000 7f07ed600700 1 --2- 172.21.15.12:0/1902519018 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7f07cc038e10 0x7f07cc03b2d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:51.623 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.991+0000 7f07ed600700 1 --2- 172.21.15.12:0/1902519018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07e8107800 0x7f07e807cb50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:51.623 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.991+0000 7f07ed600700 1 -- 172.21.15.12:0/1902519018 >> 172.21.15.12:0/1902519018 conn(0x7f07e8102e20 msgr2=0x7f07e8103990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:51.623 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.991+0000 7f07ed600700 1 -- 172.21.15.12:0/1902519018 shutdown_connections 2024-03-23T22:50:51.623 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:50.991+0000 7f07ed600700 1 -- 172.21.15.12:0/1902519018 wait complete. 2024-03-23T22:50:53.142 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-23T22:50:53.142 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.235+0000 7fe9cae00700 1 Processor -- start 2024-03-23T22:50:53.142 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.235+0000 7fe9cae00700 1 -- start start 2024-03-23T22:50:53.143 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.236+0000 7fe9cae00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c4108510 0x7fe9c410a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:53.143 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.236+0000 7fe9cae00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c407add0 con 0x7fe9c4108510 2024-03-23T22:50:53.143 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.236+0000 7fe9c3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c4108510 0x7fe9c410a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:53.143 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.236+0000 7fe9c3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c4108510 0x7fe9c410a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46618/0 (socket says 172.21.15.12:46618) 2024-03-23T22:50:53.144 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.236+0000 7fe9c3e00700 1 -- 172.21.15.12:0/1302572703 learned_addr learned my addr 172.21.15.12:0/1302572703 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:53.144 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.237+0000 7fe9c3e00700 1 -- 172.21.15.12:0/1302572703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c410ae40 con 0x7fe9c4108510 2024-03-23T22:50:53.144 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.237+0000 7fe9c3e00700 1 --2- 172.21.15.12:0/1302572703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c4108510 0x7fe9c410a900 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe9b40098a0 tx=0x7fe9b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b7b1e97d264e16 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:53.144 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.237+0000 7fe9c2a00700 1 -- 172.21.15.12:0/1302572703 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe9b4004160 con 0x7fe9c4108510 2024-03-23T22:50:53.144 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.237+0000 7fe9c2a00700 1 -- 172.21.15.12:0/1302572703 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9b4004300 con 0x7fe9c4108510 2024-03-23T22:50:53.144 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.237+0000 7fe9c2a00700 1 -- 172.21.15.12:0/1302572703 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe9b4011b50 con 0x7fe9c4108510 2024-03-23T22:50:53.146 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.238+0000 7fe9cae00700 1 -- 172.21.15.12:0/1302572703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c4108510 msgr2=0x7fe9c410a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:53.146 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.238+0000 7fe9cae00700 1 --2- 172.21.15.12:0/1302572703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c4108510 0x7fe9c410a900 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe9b40098a0 tx=0x7fe9b4009bb0 comp rx=0 tx=0).stop 2024-03-23T22:50:53.146 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.238+0000 7fe9cae00700 1 -- 172.21.15.12:0/1302572703 shutdown_connections 2024-03-23T22:50:53.147 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.238+0000 7fe9cae00700 1 --2- 172.21.15.12:0/1302572703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c4108510 0x7fe9c410a900 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:53.147 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.238+0000 7fe9cae00700 1 -- 172.21.15.12:0/1302572703 >> 172.21.15.12:0/1302572703 conn(0x7fe9c4101f40 msgr2=0x7fe9c4104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:53.147 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.238+0000 7fe9cae00700 1 -- 172.21.15.12:0/1302572703 shutdown_connections 2024-03-23T22:50:53.147 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.238+0000 7fe9cae00700 1 -- 172.21.15.12:0/1302572703 wait complete. 2024-03-23T22:50:53.147 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.239+0000 7fe9cae00700 1 Processor -- start 2024-03-23T22:50:53.147 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.240+0000 7fe9cae00700 1 -- start start 2024-03-23T22:50:53.147 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.240+0000 7fe9cae00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c4108510 0x7fe9c4107d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:53.147 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.240+0000 7fe9cae00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c410bdb0 con 0x7fe9c4108510 2024-03-23T22:50:53.147 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.240+0000 7fe9c3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c4108510 0x7fe9c4107d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:53.147 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.240+0000 7fe9c3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c4108510 0x7fe9c4107d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46628/0 (socket says 172.21.15.12:46628) 2024-03-23T22:50:53.147 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.240+0000 7fe9c3e00700 1 -- 172.21.15.12:0/2956477465 learned_addr learned my addr 172.21.15.12:0/2956477465 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:53.147 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.240+0000 7fe9c3e00700 1 -- 172.21.15.12:0/2956477465 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c41082d0 con 0x7fe9c4108510 2024-03-23T22:50:53.147 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.241+0000 7fe9c3e00700 1 --2- 172.21.15.12:0/2956477465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c4108510 0x7fe9c4107d90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4011b60 tx=0x7fe9b4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:53.147 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.241+0000 7fe9c0c00700 1 -- 172.21.15.12:0/2956477465 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe9b4003b20 con 0x7fe9c4108510 2024-03-23T22:50:53.148 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.241+0000 7fe9c0c00700 1 -- 172.21.15.12:0/2956477465 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9b4003cc0 con 0x7fe9c4108510 2024-03-23T22:50:53.148 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.241+0000 7fe9c0c00700 1 -- 172.21.15.12:0/2956477465 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe9b4024840 con 0x7fe9c4108510 2024-03-23T22:50:53.148 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.241+0000 7fe9cae00700 1 -- 172.21.15.12:0/2956477465 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c41064c0 con 0x7fe9c4108510 2024-03-23T22:50:53.148 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.241+0000 7fe9cae00700 1 -- 172.21.15.12:0/2956477465 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c4106a00 con 0x7fe9c4108510 2024-03-23T22:50:53.148 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.242+0000 7fe9cae00700 1 -- 172.21.15.12:0/2956477465 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c40637c0 con 0x7fe9c4108510 2024-03-23T22:50:53.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.243+0000 7fe9c0c00700 1 -- 172.21.15.12:0/2956477465 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fe9b4022020 con 0x7fe9c4108510 2024-03-23T22:50:53.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.243+0000 7fe9c0c00700 1 --2- 172.21.15.12:0/2956477465 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fe9a8038f70 0x7fe9a803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:53.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.243+0000 7fe9c0c00700 1 -- 172.21.15.12:0/2956477465 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe9b4011e00 con 0x7fe9c4108510 2024-03-23T22:50:53.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.243+0000 7fe9c3400700 1 --2- 172.21.15.12:0/2956477465 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fe9a8038f70 0x7fe9a803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:53.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.244+0000 7fe9c3400700 1 --2- 172.21.15.12:0/2956477465 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fe9a8038f70 0x7fe9a803b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0006fd0 tx=0x7fe9b0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:53.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.251+0000 7fe9c0c00700 1 -- 172.21.15.12:0/2956477465 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe9b400b3e0 con 0x7fe9c4108510 2024-03-23T22:50:53.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.481+0000 7fe9cae00700 1 -- 172.21.15.12:0/2956477465 --> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9c41031a0 con 0x7fe9a8038f70 2024-03-23T22:50:53.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.483+0000 7fe9c0c00700 1 -- 172.21.15.12:0/2956477465 <== mgr.14120 v2:172.21.15.12:6800/4127975083 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fe9c41031a0 con 0x7fe9a8038f70 2024-03-23T22:50:53.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.486+0000 7fe9cae00700 1 -- 172.21.15.12:0/2956477465 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fe9a8038f70 msgr2=0x7fe9a803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:53.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.486+0000 7fe9cae00700 1 --2- 172.21.15.12:0/2956477465 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fe9a8038f70 0x7fe9a803b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0006fd0 tx=0x7fe9b0006e40 comp rx=0 tx=0).stop 2024-03-23T22:50:53.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.486+0000 7fe9cae00700 1 -- 172.21.15.12:0/2956477465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c4108510 msgr2=0x7fe9c4107d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:53.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.486+0000 7fe9cae00700 1 --2- 172.21.15.12:0/2956477465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c4108510 0x7fe9c4107d90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4011b60 tx=0x7fe9b4011c40 comp rx=0 tx=0).stop 2024-03-23T22:50:53.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.486+0000 7fe9cae00700 1 -- 172.21.15.12:0/2956477465 shutdown_connections 2024-03-23T22:50:53.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.486+0000 7fe9cae00700 1 --2- 172.21.15.12:0/2956477465 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fe9a8038f70 0x7fe9a803b430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:53.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.486+0000 7fe9cae00700 1 --2- 172.21.15.12:0/2956477465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c4108510 0x7fe9c4107d90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:53.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.486+0000 7fe9cae00700 1 -- 172.21.15.12:0/2956477465 >> 172.21.15.12:0/2956477465 conn(0x7fe9c4101f40 msgr2=0x7fe9c4102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:53.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.486+0000 7fe9cae00700 1 -- 172.21.15.12:0/2956477465 shutdown_connections 2024-03-23T22:50:53.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:52.487+0000 7fe9cae00700 1 -- 172.21.15.12:0/2956477465 wait complete. 2024-03-23T22:50:53.151 INFO:teuthology.orchestra.run.smithi012.stdout:Generating ssh key... 2024-03-23T22:50:53.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:53 smithi012 ceph-mon[31695]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:50:54.555 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:54 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: Generating public/private rsa key pair. 2024-03-23T22:50:54.555 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:54 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: Your identification has been saved in /tmp/tmpsquekxhl/key. 2024-03-23T22:50:54.555 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:54 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: Your public key has been saved in /tmp/tmpsquekxhl/key.pub. 2024-03-23T22:50:54.555 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:54 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: The key fingerprint is: 2024-03-23T22:50:54.555 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:54 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: SHA256:wss+FzKgkin9k00r80xYpf13zdwIWLPfNBpFFnYZajk ceph-90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T22:50:54.555 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:54 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: The key's randomart image is: 2024-03-23T22:50:54.555 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:54 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: +---[RSA 3072]----+ 2024-03-23T22:50:54.555 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:54 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: | *=| 2024-03-23T22:50:54.555 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:54 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: | *..| 2024-03-23T22:50:54.555 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:54 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: | . oE . | 2024-03-23T22:50:54.556 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:54 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: | .. + o.oo | 2024-03-23T22:50:54.556 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:54 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: | + . .= S . o. ..| 2024-03-23T22:50:54.556 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:54 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: |= o ++o.. ooBo| 2024-03-23T22:50:54.556 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:54 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: |.. ..+++ .. ..+ *| 2024-03-23T22:50:54.556 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:54 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: | *++ . . . | 2024-03-23T22:50:54.556 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:54 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: | =+o | 2024-03-23T22:50:54.556 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:50:54 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: +----[SHA256]-----+ 2024-03-23T22:50:55.000 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.760+0000 7f27aca00700 1 Processor -- start 2024-03-23T22:50:55.000 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.761+0000 7f27aca00700 1 -- start start 2024-03-23T22:50:55.000 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.761+0000 7f27aca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27a8109a30 0x7f27a8109e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:55.000 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.761+0000 7f27aca00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a810a420 con 0x7f27a8109a30 2024-03-23T22:50:55.000 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.761+0000 7f27a6800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27a8109a30 0x7f27a8109e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:55.000 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.762+0000 7f27a6800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27a8109a30 0x7f27a8109e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46634/0 (socket says 172.21.15.12:46634) 2024-03-23T22:50:55.000 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.762+0000 7f27a6800700 1 -- 172.21.15.12:0/4227374549 learned_addr learned my addr 172.21.15.12:0/4227374549 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:55.000 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.762+0000 7f27a6800700 1 -- 172.21.15.12:0/4227374549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a810ab90 con 0x7f27a8109a30 2024-03-23T22:50:55.000 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.762+0000 7f27a6800700 1 --2- 172.21.15.12:0/4227374549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27a8109a30 0x7f27a8109e50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f27980098a0 tx=0x7f2798009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8a6065795400a82 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:55.000 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.763+0000 7f27a5400700 1 -- 172.21.15.12:0/4227374549 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2798004160 con 0x7f27a8109a30 2024-03-23T22:50:55.001 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.763+0000 7f27a5400700 1 -- 172.21.15.12:0/4227374549 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2798004300 con 0x7f27a8109a30 2024-03-23T22:50:55.001 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.763+0000 7f27a5400700 1 -- 172.21.15.12:0/4227374549 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2798011b50 con 0x7f27a8109a30 2024-03-23T22:50:55.001 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.764+0000 7f27aca00700 1 -- 172.21.15.12:0/4227374549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27a8109a30 msgr2=0x7f27a8109e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:55.001 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.764+0000 7f27aca00700 1 --2- 172.21.15.12:0/4227374549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27a8109a30 0x7f27a8109e50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f27980098a0 tx=0x7f2798009bb0 comp rx=0 tx=0).stop 2024-03-23T22:50:55.001 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.764+0000 7f27aca00700 1 -- 172.21.15.12:0/4227374549 shutdown_connections 2024-03-23T22:50:55.001 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.764+0000 7f27aca00700 1 --2- 172.21.15.12:0/4227374549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27a8109a30 0x7f27a8109e50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:55.001 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.764+0000 7f27aca00700 1 -- 172.21.15.12:0/4227374549 >> 172.21.15.12:0/4227374549 conn(0x7f27a81054a0 msgr2=0x7f27a8107890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:55.001 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.764+0000 7f27aca00700 1 -- 172.21.15.12:0/4227374549 shutdown_connections 2024-03-23T22:50:55.001 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.764+0000 7f27aca00700 1 -- 172.21.15.12:0/4227374549 wait complete. 2024-03-23T22:50:55.001 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.765+0000 7f27aca00700 1 Processor -- start 2024-03-23T22:50:55.001 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.765+0000 7f27aca00700 1 -- start start 2024-03-23T22:50:55.001 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.766+0000 7f27aca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27a8109a30 0x7f27a819d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:55.001 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.766+0000 7f27aca00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a810df40 con 0x7f27a8109a30 2024-03-23T22:50:55.001 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.766+0000 7f27a6800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27a8109a30 0x7f27a819d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:55.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.766+0000 7f27a6800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27a8109a30 0x7f27a819d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46636/0 (socket says 172.21.15.12:46636) 2024-03-23T22:50:55.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.766+0000 7f27a6800700 1 -- 172.21.15.12:0/3918233672 learned_addr learned my addr 172.21.15.12:0/3918233672 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:55.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.766+0000 7f27a6800700 1 -- 172.21.15.12:0/3918233672 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a819dec0 con 0x7f27a8109a30 2024-03-23T22:50:55.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.767+0000 7f27a6800700 1 --2- 172.21.15.12:0/3918233672 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27a8109a30 0x7f27a819d980 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2798000c00 tx=0x7f2798011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:55.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.767+0000 7f279f400700 1 -- 172.21.15.12:0/3918233672 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2798003b20 con 0x7f27a8109a30 2024-03-23T22:50:55.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.767+0000 7f279f400700 1 -- 172.21.15.12:0/3918233672 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2798003cc0 con 0x7f27a8109a30 2024-03-23T22:50:55.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.767+0000 7f27aca00700 1 -- 172.21.15.12:0/3918233672 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27a819e150 con 0x7f27a8109a30 2024-03-23T22:50:55.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.767+0000 7f279f400700 1 -- 172.21.15.12:0/3918233672 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2798024910 con 0x7f27a8109a30 2024-03-23T22:50:55.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.767+0000 7f27aca00700 1 -- 172.21.15.12:0/3918233672 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a81a0e40 con 0x7f27a8109a30 2024-03-23T22:50:55.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.768+0000 7f27aca00700 1 -- 172.21.15.12:0/3918233672 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27a80637c0 con 0x7f27a8109a30 2024-03-23T22:50:55.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.768+0000 7f279f400700 1 -- 172.21.15.12:0/3918233672 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f2798022020 con 0x7f27a8109a30 2024-03-23T22:50:55.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.769+0000 7f279f400700 1 --2- 172.21.15.12:0/3918233672 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7f278c038f70 0x7f278c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:55.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.769+0000 7f279f400700 1 -- 172.21.15.12:0/3918233672 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2798011e00 con 0x7f27a8109a30 2024-03-23T22:50:55.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.769+0000 7f27a5e00700 1 --2- 172.21.15.12:0/3918233672 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7f278c038f70 0x7f278c03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:55.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.770+0000 7f27a5e00700 1 --2- 172.21.15.12:0/3918233672 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7f278c038f70 0x7f278c03b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2794006fd0 tx=0x7f2794006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:55.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:53.774+0000 7f279f400700 1 -- 172.21.15.12:0/3918233672 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2798016260 con 0x7f27a8109a30 2024-03-23T22:50:55.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:54.002+0000 7f27aca00700 1 -- 172.21.15.12:0/3918233672 --> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f27a8107740 con 0x7f278c038f70 2024-03-23T22:50:55.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:54.288+0000 7f279f400700 1 -- 172.21.15.12:0/3918233672 <== mgr.14120 v2:172.21.15.12:6800/4127975083 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f27a8107740 con 0x7f278c038f70 2024-03-23T22:50:55.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:54.291+0000 7f27aca00700 1 -- 172.21.15.12:0/3918233672 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7f278c038f70 msgr2=0x7f278c03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:55.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:54.291+0000 7f27aca00700 1 --2- 172.21.15.12:0/3918233672 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7f278c038f70 0x7f278c03b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2794006fd0 tx=0x7f2794006e40 comp rx=0 tx=0).stop 2024-03-23T22:50:55.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:54.291+0000 7f27aca00700 1 -- 172.21.15.12:0/3918233672 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27a8109a30 msgr2=0x7f27a819d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:55.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:54.291+0000 7f27aca00700 1 --2- 172.21.15.12:0/3918233672 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27a8109a30 0x7f27a819d980 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2798000c00 tx=0x7f2798011c10 comp rx=0 tx=0).stop 2024-03-23T22:50:55.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:54.291+0000 7f27aca00700 1 -- 172.21.15.12:0/3918233672 shutdown_connections 2024-03-23T22:50:55.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:54.291+0000 7f27aca00700 1 --2- 172.21.15.12:0/3918233672 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7f278c038f70 0x7f278c03b430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:55.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:54.291+0000 7f27aca00700 1 --2- 172.21.15.12:0/3918233672 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27a8109a30 0x7f27a819d980 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:55.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:54.292+0000 7f27aca00700 1 -- 172.21.15.12:0/3918233672 >> 172.21.15.12:0/3918233672 conn(0x7f27a81054a0 msgr2=0x7f27a8107030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:55.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:54.292+0000 7f27aca00700 1 -- 172.21.15.12:0/3918233672 shutdown_connections 2024-03-23T22:50:55.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:54.292+0000 7f27aca00700 1 -- 172.21.15.12:0/3918233672 wait complete. 2024-03-23T22:50:55.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:55 smithi012 ceph-mon[31695]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:50:55.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:55 smithi012 ceph-mon[31695]: Generating ssh key... 2024-03-23T22:50:55.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:55 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' 2024-03-23T22:50:55.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:55 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' 2024-03-23T22:50:56.526 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZoSciPO4yj+WOea6DWeuGsUiKT+SYUamLfbGi6V5fM+ONThvLcCn8zIjddVT8nvyygakcwGtZAvk2jNd/uyLSg8wQkh63BlvBbzw9WvvBaSl7ooRNRve509Crs4CkXIRPPPaDVqP5zrzB6A5el4RPSpOrOnZ5PZC8iXw0MGFNy4E6uHDxd3CniXqgOeJ0zVq4wEcy8LNAtHxanBgvISMDp0afBis1UIM3Li7fgEhZRoNhmIMtkG1MxwvDkyDfPimHbSgP0CN2wtbD6ogi5bPJGOzP/LG/5SA8eVWbPWHKuNOEiu0ndLNyDiMedACKQEdNNawx1UkNTpJVG0kdU3dlmMx0YBgkP1OmvSyOC6Dp4Smh8NAgT4WTryt0QSlKKe5ALKgvoGRvIacXxzdd0qmv357JYp4pXGLtz4HlJttWwhVQuPG8UNgugkMFYFVQCyyf8vLj/s6lC48/1mS+e+RQrUOc+8CXQr0QhIGefZmy+tUIePAA/OZBaH+HMxv0Xfk= ceph-90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T22:50:56.526 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.565+0000 7fd609a00700 1 Processor -- start 2024-03-23T22:50:56.526 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.565+0000 7fd609a00700 1 -- start start 2024-03-23T22:50:56.526 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.566+0000 7fd609a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6041060a0 0x7fd6041064c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:56.526 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.566+0000 7fd609a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60407ae30 con 0x7fd6041060a0 2024-03-23T22:50:56.526 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.566+0000 7fd603e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6041060a0 0x7fd6041064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:56.526 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.566+0000 7fd603e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6041060a0 0x7fd6041064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59128/0 (socket says 172.21.15.12:59128) 2024-03-23T22:50:56.526 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.566+0000 7fd603e00700 1 -- 172.21.15.12:0/4271809493 learned_addr learned my addr 172.21.15.12:0/4271809493 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:56.526 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.567+0000 7fd603e00700 1 -- 172.21.15.12:0/4271809493 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd604106a00 con 0x7fd6041060a0 2024-03-23T22:50:56.526 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.567+0000 7fd603e00700 1 --2- 172.21.15.12:0/4271809493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6041060a0 0x7fd6041064c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd5f40098a0 tx=0x7fd5f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d602f2915f3c50 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:56.526 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.567+0000 7fd602a00700 1 -- 172.21.15.12:0/4271809493 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5f4004160 con 0x7fd6041060a0 2024-03-23T22:50:56.526 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.567+0000 7fd602a00700 1 -- 172.21.15.12:0/4271809493 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd5f4004300 con 0x7fd6041060a0 2024-03-23T22:50:56.527 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.567+0000 7fd602a00700 1 -- 172.21.15.12:0/4271809493 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5f4011b50 con 0x7fd6041060a0 2024-03-23T22:50:56.527 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.568+0000 7fd609a00700 1 -- 172.21.15.12:0/4271809493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6041060a0 msgr2=0x7fd6041064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:56.527 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.568+0000 7fd609a00700 1 --2- 172.21.15.12:0/4271809493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6041060a0 0x7fd6041064c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd5f40098a0 tx=0x7fd5f4009bb0 comp rx=0 tx=0).stop 2024-03-23T22:50:56.527 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.568+0000 7fd609a00700 1 -- 172.21.15.12:0/4271809493 shutdown_connections 2024-03-23T22:50:56.527 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.568+0000 7fd609a00700 1 --2- 172.21.15.12:0/4271809493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6041060a0 0x7fd6041064c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:56.527 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.568+0000 7fd609a00700 1 -- 172.21.15.12:0/4271809493 >> 172.21.15.12:0/4271809493 conn(0x7fd604101d40 msgr2=0x7fd604104160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:56.527 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.568+0000 7fd609a00700 1 -- 172.21.15.12:0/4271809493 shutdown_connections 2024-03-23T22:50:56.527 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.568+0000 7fd609a00700 1 -- 172.21.15.12:0/4271809493 wait complete. 2024-03-23T22:50:56.527 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.569+0000 7fd609a00700 1 Processor -- start 2024-03-23T22:50:56.527 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.570+0000 7fd609a00700 1 -- start start 2024-03-23T22:50:56.527 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.570+0000 7fd609a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6041060a0 0x7fd60419d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:56.527 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.570+0000 7fd609a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd604107a90 con 0x7fd6041060a0 2024-03-23T22:50:56.527 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.570+0000 7fd603e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6041060a0 0x7fd60419d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:56.527 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.570+0000 7fd603e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6041060a0 0x7fd60419d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59132/0 (socket says 172.21.15.12:59132) 2024-03-23T22:50:56.527 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.570+0000 7fd603e00700 1 -- 172.21.15.12:0/53320488 learned_addr learned my addr 172.21.15.12:0/53320488 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:50:56.528 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.570+0000 7fd603e00700 1 -- 172.21.15.12:0/53320488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd60419df30 con 0x7fd6041060a0 2024-03-23T22:50:56.528 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.571+0000 7fd603e00700 1 --2- 172.21.15.12:0/53320488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6041060a0 0x7fd60419d9f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd5f4011b60 tx=0x7fd5f4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:56.528 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.571+0000 7fd600c00700 1 -- 172.21.15.12:0/53320488 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5f4003970 con 0x7fd6041060a0 2024-03-23T22:50:56.528 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.571+0000 7fd600c00700 1 -- 172.21.15.12:0/53320488 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd5f4003b10 con 0x7fd6041060a0 2024-03-23T22:50:56.528 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.571+0000 7fd600c00700 1 -- 172.21.15.12:0/53320488 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5f4024610 con 0x7fd6041060a0 2024-03-23T22:50:56.528 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.571+0000 7fd609a00700 1 -- 172.21.15.12:0/53320488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd60419e1c0 con 0x7fd6041060a0 2024-03-23T22:50:56.528 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.571+0000 7fd609a00700 1 -- 172.21.15.12:0/53320488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6041a0eb0 con 0x7fd6041060a0 2024-03-23T22:50:56.528 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.572+0000 7fd609a00700 1 -- 172.21.15.12:0/53320488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6040637c0 con 0x7fd6041060a0 2024-03-23T22:50:56.528 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.573+0000 7fd600c00700 1 -- 172.21.15.12:0/53320488 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fd5f401a9f0 con 0x7fd6041060a0 2024-03-23T22:50:56.528 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.573+0000 7fd600c00700 1 --2- 172.21.15.12:0/53320488 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fd5e8038f70 0x7fd5e803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:50:56.528 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.573+0000 7fd600c00700 1 -- 172.21.15.12:0/53320488 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd5f4011e00 con 0x7fd6041060a0 2024-03-23T22:50:56.529 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.574+0000 7fd603400700 1 --2- 172.21.15.12:0/53320488 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fd5e8038f70 0x7fd5e803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:50:56.529 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.574+0000 7fd603400700 1 --2- 172.21.15.12:0/53320488 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fd5e8038f70 0x7fd5e803b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0006fd0 tx=0x7fd5f0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:50:56.529 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.579+0000 7fd600c00700 1 -- 172.21.15.12:0/53320488 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd5f4019e10 con 0x7fd6041060a0 2024-03-23T22:50:56.529 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.804+0000 7fd609a00700 1 -- 172.21.15.12:0/53320488 --> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6040008d0 con 0x7fd5e8038f70 2024-03-23T22:50:56.529 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.805+0000 7fd600c00700 1 -- 172.21.15.12:0/53320488 <== mgr.14120 v2:172.21.15.12:6800/4127975083 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fd6040008d0 con 0x7fd5e8038f70 2024-03-23T22:50:56.529 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.808+0000 7fd609a00700 1 -- 172.21.15.12:0/53320488 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fd5e8038f70 msgr2=0x7fd5e803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:56.529 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.808+0000 7fd609a00700 1 --2- 172.21.15.12:0/53320488 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fd5e8038f70 0x7fd5e803b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0006fd0 tx=0x7fd5f0006e40 comp rx=0 tx=0).stop 2024-03-23T22:50:56.529 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.808+0000 7fd609a00700 1 -- 172.21.15.12:0/53320488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6041060a0 msgr2=0x7fd60419d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:50:56.529 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.808+0000 7fd609a00700 1 --2- 172.21.15.12:0/53320488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6041060a0 0x7fd60419d9f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd5f4011b60 tx=0x7fd5f4011c40 comp rx=0 tx=0).stop 2024-03-23T22:50:56.529 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.808+0000 7fd609a00700 1 -- 172.21.15.12:0/53320488 shutdown_connections 2024-03-23T22:50:56.529 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.808+0000 7fd609a00700 1 --2- 172.21.15.12:0/53320488 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fd5e8038f70 0x7fd5e803b430 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:56.529 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.808+0000 7fd609a00700 1 --2- 172.21.15.12:0/53320488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6041060a0 0x7fd60419d9f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:50:56.529 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.808+0000 7fd609a00700 1 -- 172.21.15.12:0/53320488 >> 172.21.15.12:0/53320488 conn(0x7fd604101d40 msgr2=0x7fd604194650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:50:56.530 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.808+0000 7fd609a00700 1 -- 172.21.15.12:0/53320488 shutdown_connections 2024-03-23T22:50:56.530 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:55.808+0000 7fd609a00700 1 -- 172.21.15.12:0/53320488 wait complete. 2024-03-23T22:50:56.530 INFO:teuthology.orchestra.run.smithi012.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-23T22:50:56.530 INFO:teuthology.orchestra.run.smithi012.stdout:Adding key to root@localhost authorized_keys... 2024-03-23T22:50:56.530 INFO:teuthology.orchestra.run.smithi012.stdout:Adding host smithi012... 2024-03-23T22:50:56.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:56 smithi012 ceph-mon[31695]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:50:57.609 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:57 smithi012 ceph-mon[31695]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi012", "addr": "172.21.15.12", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:50:59.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:50:59 smithi012 ceph-mon[31695]: Deploying cephadm binary to smithi012 2024-03-23T22:51:04.669 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout Added host 'smithi012' with addr '172.21.15.12' 2024-03-23T22:51:04.670 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.106+0000 7fc293200700 1 Processor -- start 2024-03-23T22:51:04.670 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.106+0000 7fc293200700 1 -- start start 2024-03-23T22:51:04.670 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.106+0000 7fc293200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc28c106270 0x7fc28c106690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:04.670 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.107+0000 7fc293200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c07ae50 con 0x7fc28c106270 2024-03-23T22:51:04.670 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.107+0000 7fc28be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc28c106270 0x7fc28c106690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:04.670 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.107+0000 7fc28be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc28c106270 0x7fc28c106690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59134/0 (socket says 172.21.15.12:59134) 2024-03-23T22:51:04.670 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.107+0000 7fc28be00700 1 -- 172.21.15.12:0/130330716 learned_addr learned my addr 172.21.15.12:0/130330716 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:04.670 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.108+0000 7fc28be00700 1 -- 172.21.15.12:0/130330716 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c106bd0 con 0x7fc28c106270 2024-03-23T22:51:04.670 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.108+0000 7fc28be00700 1 --2- 172.21.15.12:0/130330716 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc28c106270 0x7fc28c106690 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc27c0098a0 tx=0x7fc27c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de2f8bda0e52b10f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:04.670 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.108+0000 7fc28aa00700 1 -- 172.21.15.12:0/130330716 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc27c004160 con 0x7fc28c106270 2024-03-23T22:51:04.670 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.109+0000 7fc28aa00700 1 -- 172.21.15.12:0/130330716 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc27c004300 con 0x7fc28c106270 2024-03-23T22:51:04.671 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.109+0000 7fc28aa00700 1 -- 172.21.15.12:0/130330716 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc27c011b50 con 0x7fc28c106270 2024-03-23T22:51:04.671 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.109+0000 7fc293200700 1 -- 172.21.15.12:0/130330716 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc28c106270 msgr2=0x7fc28c106690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:04.671 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.109+0000 7fc293200700 1 --2- 172.21.15.12:0/130330716 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc28c106270 0x7fc28c106690 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc27c0098a0 tx=0x7fc27c009bb0 comp rx=0 tx=0).stop 2024-03-23T22:51:04.671 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.109+0000 7fc293200700 1 -- 172.21.15.12:0/130330716 shutdown_connections 2024-03-23T22:51:04.671 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.109+0000 7fc293200700 1 --2- 172.21.15.12:0/130330716 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc28c106270 0x7fc28c106690 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:04.671 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.109+0000 7fc293200700 1 -- 172.21.15.12:0/130330716 >> 172.21.15.12:0/130330716 conn(0x7fc28c101f90 msgr2=0x7fc28c1043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:04.671 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.109+0000 7fc293200700 1 -- 172.21.15.12:0/130330716 shutdown_connections 2024-03-23T22:51:04.671 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.109+0000 7fc293200700 1 -- 172.21.15.12:0/130330716 wait complete. 2024-03-23T22:51:04.671 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.111+0000 7fc293200700 1 Processor -- start 2024-03-23T22:51:04.671 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.111+0000 7fc293200700 1 -- start start 2024-03-23T22:51:04.671 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.111+0000 7fc293200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc28c106270 0x7fc28c19db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:04.671 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.111+0000 7fc293200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c107c60 con 0x7fc28c106270 2024-03-23T22:51:04.671 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.111+0000 7fc28be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc28c106270 0x7fc28c19db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:04.671 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.111+0000 7fc28be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc28c106270 0x7fc28c19db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59148/0 (socket says 172.21.15.12:59148) 2024-03-23T22:51:04.672 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.111+0000 7fc28be00700 1 -- 172.21.15.12:0/1931627295 learned_addr learned my addr 172.21.15.12:0/1931627295 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:04.672 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.112+0000 7fc28be00700 1 -- 172.21.15.12:0/1931627295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c19e0a0 con 0x7fc28c106270 2024-03-23T22:51:04.672 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.112+0000 7fc28be00700 1 --2- 172.21.15.12:0/1931627295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc28c106270 0x7fc28c19db60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc27c011670 tx=0x7fc27c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:04.672 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.112+0000 7fc288c00700 1 -- 172.21.15.12:0/1931627295 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc27c0039c0 con 0x7fc28c106270 2024-03-23T22:51:04.672 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.112+0000 7fc288c00700 1 -- 172.21.15.12:0/1931627295 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc27c011e50 con 0x7fc28c106270 2024-03-23T22:51:04.672 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.112+0000 7fc293200700 1 -- 172.21.15.12:0/1931627295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc28c19e330 con 0x7fc28c106270 2024-03-23T22:51:04.672 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.112+0000 7fc288c00700 1 -- 172.21.15.12:0/1931627295 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc27c01a5c0 con 0x7fc28c106270 2024-03-23T22:51:04.672 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.113+0000 7fc293200700 1 -- 172.21.15.12:0/1931627295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc28c1a1020 con 0x7fc28c106270 2024-03-23T22:51:04.672 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.114+0000 7fc288c00700 1 -- 172.21.15.12:0/1931627295 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fc27c024430 con 0x7fc28c106270 2024-03-23T22:51:04.672 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.114+0000 7fc293200700 1 -- 172.21.15.12:0/1931627295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc28c0637c0 con 0x7fc28c106270 2024-03-23T22:51:04.672 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.114+0000 7fc288c00700 1 --2- 172.21.15.12:0/1931627295 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fc270038f20 0x7fc27003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:04.672 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.114+0000 7fc288c00700 1 -- 172.21.15.12:0/1931627295 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc27c04d370 con 0x7fc28c106270 2024-03-23T22:51:04.672 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.115+0000 7fc28b400700 1 --2- 172.21.15.12:0/1931627295 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fc270038f20 0x7fc27003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:04.672 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.115+0000 7fc28b400700 1 --2- 172.21.15.12:0/1931627295 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fc270038f20 0x7fc27003b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc278006fd0 tx=0x7fc278006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:04.673 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.120+0000 7fc288c00700 1 -- 172.21.15.12:0/1931627295 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc27c04ccc0 con 0x7fc28c106270 2024-03-23T22:51:04.673 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:50:57.351+0000 7fc293200700 1 -- 172.21.15.12:0/1931627295 --> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi012", "addr": "172.21.15.12", "target": ["mon-mgr", ""]}) v1 -- 0x7fc28c0008d0 con 0x7fc270038f20 2024-03-23T22:51:04.673 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:03.956+0000 7fc288c00700 1 -- 172.21.15.12:0/1931627295 <== mgr.14120 v2:172.21.15.12:6800/4127975083 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fc28c0008d0 con 0x7fc270038f20 2024-03-23T22:51:04.673 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:03.961+0000 7fc293200700 1 -- 172.21.15.12:0/1931627295 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fc270038f20 msgr2=0x7fc27003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:04.673 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:03.961+0000 7fc293200700 1 --2- 172.21.15.12:0/1931627295 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fc270038f20 0x7fc27003b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc278006fd0 tx=0x7fc278006e40 comp rx=0 tx=0).stop 2024-03-23T22:51:04.673 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:03.961+0000 7fc293200700 1 -- 172.21.15.12:0/1931627295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc28c106270 msgr2=0x7fc28c19db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:04.673 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:03.961+0000 7fc293200700 1 --2- 172.21.15.12:0/1931627295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc28c106270 0x7fc28c19db60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc27c011670 tx=0x7fc27c011b60 comp rx=0 tx=0).stop 2024-03-23T22:51:04.673 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:03.961+0000 7fc293200700 1 -- 172.21.15.12:0/1931627295 shutdown_connections 2024-03-23T22:51:04.673 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:03.961+0000 7fc293200700 1 --2- 172.21.15.12:0/1931627295 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fc270038f20 0x7fc27003b3e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:04.673 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:03.961+0000 7fc293200700 1 --2- 172.21.15.12:0/1931627295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc28c106270 0x7fc28c19db60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:04.673 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:03.961+0000 7fc293200700 1 -- 172.21.15.12:0/1931627295 >> 172.21.15.12:0/1931627295 conn(0x7fc28c101f90 msgr2=0x7fc28c194820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:04.673 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:03.961+0000 7fc293200700 1 -- 172.21.15.12:0/1931627295 shutdown_connections 2024-03-23T22:51:04.673 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:03.961+0000 7fc293200700 1 -- 172.21.15.12:0/1931627295 wait complete. 2024-03-23T22:51:04.673 INFO:teuthology.orchestra.run.smithi012.stdout:Deploying unmanaged mon service... 2024-03-23T22:51:05.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:04 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' 2024-03-23T22:51:05.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:04 smithi012 ceph-mon[31695]: Added host smithi012 2024-03-23T22:51:05.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:04 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:51:06.699 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:06 smithi012 ceph-mon[31695]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:51:06.699 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:06 smithi012 ceph-mon[31695]: Saving service mon spec with placement count:5 2024-03-23T22:51:06.700 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:06 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' 2024-03-23T22:51:07.065 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-23T22:51:07.066 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.329+0000 7fd624c00700 1 Processor -- start 2024-03-23T22:51:07.067 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.330+0000 7fd624c00700 1 -- start start 2024-03-23T22:51:07.067 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.330+0000 7fd624c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd62010e080 0x7fd62010e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:07.067 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.330+0000 7fd624c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62010ea70 con 0x7fd62010e080 2024-03-23T22:51:07.067 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.331+0000 7fd61de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd62010e080 0x7fd62010e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:07.067 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.331+0000 7fd61de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd62010e080 0x7fd62010e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46364/0 (socket says 172.21.15.12:46364) 2024-03-23T22:51:07.067 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.331+0000 7fd61de00700 1 -- 172.21.15.12:0/537648381 learned_addr learned my addr 172.21.15.12:0/537648381 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:07.069 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.331+0000 7fd61de00700 1 -- 172.21.15.12:0/537648381 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd62010f200 con 0x7fd62010e080 2024-03-23T22:51:07.069 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.331+0000 7fd61de00700 1 --2- 172.21.15.12:0/537648381 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd62010e080 0x7fd62010e4a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd6100098a0 tx=0x7fd610009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4013296e53f4e9a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:07.069 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.332+0000 7fd61ca00700 1 -- 172.21.15.12:0/537648381 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd610004160 con 0x7fd62010e080 2024-03-23T22:51:07.069 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.332+0000 7fd61ca00700 1 -- 172.21.15.12:0/537648381 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd610004300 con 0x7fd62010e080 2024-03-23T22:51:07.069 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.332+0000 7fd61ca00700 1 -- 172.21.15.12:0/537648381 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd610011b50 con 0x7fd62010e080 2024-03-23T22:51:07.069 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.333+0000 7fd624c00700 1 -- 172.21.15.12:0/537648381 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd62010e080 msgr2=0x7fd62010e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:07.070 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.333+0000 7fd624c00700 1 --2- 172.21.15.12:0/537648381 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd62010e080 0x7fd62010e4a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd6100098a0 tx=0x7fd610009bb0 comp rx=0 tx=0).stop 2024-03-23T22:51:07.070 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.333+0000 7fd624c00700 1 -- 172.21.15.12:0/537648381 shutdown_connections 2024-03-23T22:51:07.070 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.333+0000 7fd624c00700 1 --2- 172.21.15.12:0/537648381 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd62010e080 0x7fd62010e4a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:07.070 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.333+0000 7fd624c00700 1 -- 172.21.15.12:0/537648381 >> 172.21.15.12:0/537648381 conn(0x7fd62007d230 msgr2=0x7fd62007f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:07.070 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.333+0000 7fd624c00700 1 -- 172.21.15.12:0/537648381 shutdown_connections 2024-03-23T22:51:07.070 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.333+0000 7fd624c00700 1 -- 172.21.15.12:0/537648381 wait complete. 2024-03-23T22:51:07.070 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.334+0000 7fd624c00700 1 Processor -- start 2024-03-23T22:51:07.070 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.334+0000 7fd624c00700 1 -- start start 2024-03-23T22:51:07.070 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.335+0000 7fd624c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd62010e080 0x7fd6201a1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:07.070 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.335+0000 7fd624c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6201125b0 con 0x7fd62010e080 2024-03-23T22:51:07.073 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.335+0000 7fd61de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd62010e080 0x7fd6201a1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:07.073 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.335+0000 7fd61de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd62010e080 0x7fd6201a1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46368/0 (socket says 172.21.15.12:46368) 2024-03-23T22:51:07.074 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.335+0000 7fd61de00700 1 -- 172.21.15.12:0/3109324372 learned_addr learned my addr 172.21.15.12:0/3109324372 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:07.074 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.335+0000 7fd61de00700 1 -- 172.21.15.12:0/3109324372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6201a2480 con 0x7fd62010e080 2024-03-23T22:51:07.074 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.336+0000 7fd61de00700 1 --2- 172.21.15.12:0/3109324372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd62010e080 0x7fd6201a1f40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd610011670 tx=0x7fd610011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:07.074 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.336+0000 7fd616a00700 1 -- 172.21.15.12:0/3109324372 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6100039c0 con 0x7fd62010e080 2024-03-23T22:51:07.074 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.336+0000 7fd616a00700 1 -- 172.21.15.12:0/3109324372 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd610011e50 con 0x7fd62010e080 2024-03-23T22:51:07.074 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.336+0000 7fd624c00700 1 -- 172.21.15.12:0/3109324372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6201a2710 con 0x7fd62010e080 2024-03-23T22:51:07.074 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.336+0000 7fd616a00700 1 -- 172.21.15.12:0/3109324372 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd61001a5c0 con 0x7fd62010e080 2024-03-23T22:51:07.074 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.336+0000 7fd624c00700 1 -- 172.21.15.12:0/3109324372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6201a5400 con 0x7fd62010e080 2024-03-23T22:51:07.074 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.338+0000 7fd616a00700 1 -- 172.21.15.12:0/3109324372 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fd610024430 con 0x7fd62010e080 2024-03-23T22:51:07.074 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.337+0000 7fd624c00700 1 -- 172.21.15.12:0/3109324372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6200637c0 con 0x7fd62010e080 2024-03-23T22:51:07.074 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.338+0000 7fd616a00700 1 --2- 172.21.15.12:0/3109324372 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fd604038f20 0x7fd60403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:07.074 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.338+0000 7fd616a00700 1 -- 172.21.15.12:0/3109324372 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd61004d320 con 0x7fd62010e080 2024-03-23T22:51:07.074 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.338+0000 7fd61d400700 1 --2- 172.21.15.12:0/3109324372 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fd604038f20 0x7fd60403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:07.075 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.339+0000 7fd61d400700 1 --2- 172.21.15.12:0/3109324372 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fd604038f20 0x7fd60403b3e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd60c006fd0 tx=0x7fd60c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:07.075 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.344+0000 7fd616a00700 1 -- 172.21.15.12:0/3109324372 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd61004cc70 con 0x7fd62010e080 2024-03-23T22:51:07.075 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.600+0000 7fd624c00700 1 -- 172.21.15.12:0/3109324372 --> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fd62007e4d0 con 0x7fd604038f20 2024-03-23T22:51:07.075 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.608+0000 7fd616a00700 1 -- 172.21.15.12:0/3109324372 <== mgr.14120 v2:172.21.15.12:6800/4127975083 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fd62007e4d0 con 0x7fd604038f20 2024-03-23T22:51:07.075 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.611+0000 7fd624c00700 1 -- 172.21.15.12:0/3109324372 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fd604038f20 msgr2=0x7fd60403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:07.075 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.611+0000 7fd624c00700 1 --2- 172.21.15.12:0/3109324372 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fd604038f20 0x7fd60403b3e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd60c006fd0 tx=0x7fd60c006e40 comp rx=0 tx=0).stop 2024-03-23T22:51:07.075 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.611+0000 7fd624c00700 1 -- 172.21.15.12:0/3109324372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd62010e080 msgr2=0x7fd6201a1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:07.075 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.611+0000 7fd624c00700 1 --2- 172.21.15.12:0/3109324372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd62010e080 0x7fd6201a1f40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd610011670 tx=0x7fd610011b60 comp rx=0 tx=0).stop 2024-03-23T22:51:07.075 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.611+0000 7fd624c00700 1 -- 172.21.15.12:0/3109324372 shutdown_connections 2024-03-23T22:51:07.075 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.611+0000 7fd624c00700 1 --2- 172.21.15.12:0/3109324372 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fd604038f20 0x7fd60403b3e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:07.075 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.611+0000 7fd624c00700 1 --2- 172.21.15.12:0/3109324372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd62010e080 0x7fd6201a1f40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:07.075 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.611+0000 7fd624c00700 1 -- 172.21.15.12:0/3109324372 >> 172.21.15.12:0/3109324372 conn(0x7fd62007d230 msgr2=0x7fd62007ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:07.075 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.612+0000 7fd624c00700 1 -- 172.21.15.12:0/3109324372 shutdown_connections 2024-03-23T22:51:07.075 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:05.612+0000 7fd624c00700 1 -- 172.21.15.12:0/3109324372 wait complete. 2024-03-23T22:51:07.076 INFO:teuthology.orchestra.run.smithi012.stdout:Deploying unmanaged mgr service... 2024-03-23T22:51:08.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:08 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' 2024-03-23T22:51:08.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:08 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' 2024-03-23T22:51:08.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:08 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' 2024-03-23T22:51:08.897 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-23T22:51:08.897 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.885+0000 7fcf1ec00700 1 Processor -- start 2024-03-23T22:51:08.897 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.885+0000 7fcf1ec00700 1 -- start start 2024-03-23T22:51:08.897 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.886+0000 7fcf1ec00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf1807cdf0 0x7fcf1807b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:08.897 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.886+0000 7fcf1ec00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1807b7e0 con 0x7fcf1807cdf0 2024-03-23T22:51:08.897 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.886+0000 7fcf17e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf1807cdf0 0x7fcf1807b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:08.897 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.886+0000 7fcf17e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf1807cdf0 0x7fcf1807b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46376/0 (socket says 172.21.15.12:46376) 2024-03-23T22:51:08.897 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.886+0000 7fcf17e00700 1 -- 172.21.15.12:0/1267261923 learned_addr learned my addr 172.21.15.12:0/1267261923 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:08.897 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.887+0000 7fcf17e00700 1 -- 172.21.15.12:0/1267261923 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1807b960 con 0x7fcf1807cdf0 2024-03-23T22:51:08.897 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.887+0000 7fcf17e00700 1 --2- 172.21.15.12:0/1267261923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf1807cdf0 0x7fcf1807b2a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcf080098a0 tx=0x7fcf08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25ec33e2d4d4912b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:08.897 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.888+0000 7fcf16a00700 1 -- 172.21.15.12:0/1267261923 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf08004160 con 0x7fcf1807cdf0 2024-03-23T22:51:08.898 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.888+0000 7fcf16a00700 1 -- 172.21.15.12:0/1267261923 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf08004300 con 0x7fcf1807cdf0 2024-03-23T22:51:08.898 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.888+0000 7fcf16a00700 1 -- 172.21.15.12:0/1267261923 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf08011b20 con 0x7fcf1807cdf0 2024-03-23T22:51:08.898 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.889+0000 7fcf1ec00700 1 -- 172.21.15.12:0/1267261923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf1807cdf0 msgr2=0x7fcf1807b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:08.898 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.889+0000 7fcf1ec00700 1 --2- 172.21.15.12:0/1267261923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf1807cdf0 0x7fcf1807b2a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcf080098a0 tx=0x7fcf08009bb0 comp rx=0 tx=0).stop 2024-03-23T22:51:08.898 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.889+0000 7fcf1ec00700 1 -- 172.21.15.12:0/1267261923 shutdown_connections 2024-03-23T22:51:08.898 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.889+0000 7fcf1ec00700 1 --2- 172.21.15.12:0/1267261923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf1807cdf0 0x7fcf1807b2a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:08.898 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.889+0000 7fcf1ec00700 1 -- 172.21.15.12:0/1267261923 >> 172.21.15.12:0/1267261923 conn(0x7fcf18103060 msgr2=0x7fcf18105480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:08.898 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.889+0000 7fcf1ec00700 1 -- 172.21.15.12:0/1267261923 shutdown_connections 2024-03-23T22:51:08.898 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.889+0000 7fcf1ec00700 1 -- 172.21.15.12:0/1267261923 wait complete. 2024-03-23T22:51:08.898 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.890+0000 7fcf1ec00700 1 Processor -- start 2024-03-23T22:51:08.898 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.891+0000 7fcf1ec00700 1 -- start start 2024-03-23T22:51:08.898 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.891+0000 7fcf1ec00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf1807cdf0 0x7fcf1819dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:08.898 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.891+0000 7fcf1ec00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1810e080 con 0x7fcf1807cdf0 2024-03-23T22:51:08.898 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.892+0000 7fcf17e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf1807cdf0 0x7fcf1819dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:08.898 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.892+0000 7fcf17e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf1807cdf0 0x7fcf1819dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46388/0 (socket says 172.21.15.12:46388) 2024-03-23T22:51:08.899 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.892+0000 7fcf17e00700 1 -- 172.21.15.12:0/167201293 learned_addr learned my addr 172.21.15.12:0/167201293 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:08.899 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.892+0000 7fcf17e00700 1 -- 172.21.15.12:0/167201293 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1819e110 con 0x7fcf1807cdf0 2024-03-23T22:51:08.899 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.892+0000 7fcf17e00700 1 --2- 172.21.15.12:0/167201293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf1807cdf0 0x7fcf1819dbd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcf080116a0 tx=0x7fcf08011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:08.899 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.893+0000 7fcf14c00700 1 -- 172.21.15.12:0/167201293 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf08003a40 con 0x7fcf1807cdf0 2024-03-23T22:51:08.899 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.893+0000 7fcf14c00700 1 -- 172.21.15.12:0/167201293 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf08003be0 con 0x7fcf1807cdf0 2024-03-23T22:51:08.899 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.893+0000 7fcf1ec00700 1 -- 172.21.15.12:0/167201293 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf1819e3a0 con 0x7fcf1807cdf0 2024-03-23T22:51:08.899 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.893+0000 7fcf14c00700 1 -- 172.21.15.12:0/167201293 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf0801a750 con 0x7fcf1807cdf0 2024-03-23T22:51:08.899 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.893+0000 7fcf1ec00700 1 -- 172.21.15.12:0/167201293 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf1819e800 con 0x7fcf1807cdf0 2024-03-23T22:51:08.899 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.894+0000 7fcf14c00700 1 -- 172.21.15.12:0/167201293 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fcf08024aa0 con 0x7fcf1807cdf0 2024-03-23T22:51:08.899 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.894+0000 7fcf1ec00700 1 -- 172.21.15.12:0/167201293 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf180637c0 con 0x7fcf1807cdf0 2024-03-23T22:51:08.899 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.895+0000 7fcf14c00700 1 --2- 172.21.15.12:0/167201293 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fcefc038ed0 0x7fcefc03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:08.899 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.895+0000 7fcf14c00700 1 -- 172.21.15.12:0/167201293 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fcf0804d550 con 0x7fcf1807cdf0 2024-03-23T22:51:08.899 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.895+0000 7fcf17400700 1 --2- 172.21.15.12:0/167201293 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fcefc038ed0 0x7fcefc03b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:08.899 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.896+0000 7fcf17400700 1 --2- 172.21.15.12:0/167201293 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fcefc038ed0 0x7fcefc03b390 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcf04006fd0 tx=0x7fcf04006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:08.900 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:07.900+0000 7fcf14c00700 1 -- 172.21.15.12:0/167201293 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcf08011e20 con 0x7fcf1807cdf0 2024-03-23T22:51:08.900 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:08.141+0000 7fcf1ec00700 1 -- 172.21.15.12:0/167201293 --> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fcf18104280 con 0x7fcefc038ed0 2024-03-23T22:51:08.900 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:08.148+0000 7fcf14c00700 1 -- 172.21.15.12:0/167201293 <== mgr.14120 v2:172.21.15.12:6800/4127975083 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fcf18104280 con 0x7fcefc038ed0 2024-03-23T22:51:08.900 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:08.151+0000 7fcf1ec00700 1 -- 172.21.15.12:0/167201293 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fcefc038ed0 msgr2=0x7fcefc03b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:08.900 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:08.151+0000 7fcf1ec00700 1 --2- 172.21.15.12:0/167201293 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fcefc038ed0 0x7fcefc03b390 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcf04006fd0 tx=0x7fcf04006e40 comp rx=0 tx=0).stop 2024-03-23T22:51:08.900 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:08.151+0000 7fcf1ec00700 1 -- 172.21.15.12:0/167201293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf1807cdf0 msgr2=0x7fcf1819dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:08.900 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:08.151+0000 7fcf1ec00700 1 --2- 172.21.15.12:0/167201293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf1807cdf0 0x7fcf1819dbd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcf080116a0 tx=0x7fcf08011ba0 comp rx=0 tx=0).stop 2024-03-23T22:51:08.901 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:08.152+0000 7fcf1ec00700 1 -- 172.21.15.12:0/167201293 shutdown_connections 2024-03-23T22:51:08.901 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:08.152+0000 7fcf1ec00700 1 --2- 172.21.15.12:0/167201293 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fcefc038ed0 0x7fcefc03b390 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:08.901 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:08.152+0000 7fcf1ec00700 1 --2- 172.21.15.12:0/167201293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf1807cdf0 0x7fcf1819dbd0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:08.901 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:08.152+0000 7fcf1ec00700 1 -- 172.21.15.12:0/167201293 >> 172.21.15.12:0/167201293 conn(0x7fcf18103060 msgr2=0x7fcf18103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:08.901 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:08.152+0000 7fcf1ec00700 1 -- 172.21.15.12:0/167201293 shutdown_connections 2024-03-23T22:51:08.901 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:08.152+0000 7fcf1ec00700 1 -- 172.21.15.12:0/167201293 wait complete. 2024-03-23T22:51:09.483 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:09 smithi012 ceph-mon[31695]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:51:09.483 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:09 smithi012 ceph-mon[31695]: Saving service mgr spec with placement count:2 2024-03-23T22:51:09.484 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:09 smithi012 ceph-mon[31695]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:51:11.005 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:10 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3765324209' entity='client.admin' 2024-03-23T22:51:11.006 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:10 smithi012 ceph-mon[31695]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:51:11.019 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.484+0000 7f1fb5e00700 1 Processor -- start 2024-03-23T22:51:11.019 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.484+0000 7f1fb5e00700 1 -- start start 2024-03-23T22:51:11.019 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.484+0000 7f1fb5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fb0106240 0x7f1fb0108670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:11.019 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.485+0000 7f1fb5e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb007ae90 con 0x7f1fb0106240 2024-03-23T22:51:11.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.485+0000 7f1faea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fb0106240 0x7f1fb0108670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:11.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.485+0000 7f1faea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fb0106240 0x7f1fb0108670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46392/0 (socket says 172.21.15.12:46392) 2024-03-23T22:51:11.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.485+0000 7f1faea00700 1 -- 172.21.15.12:0/1676171013 learned_addr learned my addr 172.21.15.12:0/1676171013 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:11.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.486+0000 7f1faea00700 1 -- 172.21.15.12:0/1676171013 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb0108bb0 con 0x7f1fb0106240 2024-03-23T22:51:11.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.486+0000 7f1faea00700 1 --2- 172.21.15.12:0/1676171013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fb0106240 0x7f1fb0108670 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1fa00098a0 tx=0x7f1fa0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b6c936cbcd52b5b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:11.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.486+0000 7f1fad600700 1 -- 172.21.15.12:0/1676171013 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1fa0004160 con 0x7f1fb0106240 2024-03-23T22:51:11.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.486+0000 7f1fad600700 1 -- 172.21.15.12:0/1676171013 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1fa0004300 con 0x7f1fb0106240 2024-03-23T22:51:11.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.486+0000 7f1fad600700 1 -- 172.21.15.12:0/1676171013 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1fa0011b20 con 0x7f1fb0106240 2024-03-23T22:51:11.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.487+0000 7f1fb5e00700 1 -- 172.21.15.12:0/1676171013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fb0106240 msgr2=0x7f1fb0108670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:11.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.487+0000 7f1fb5e00700 1 --2- 172.21.15.12:0/1676171013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fb0106240 0x7f1fb0108670 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1fa00098a0 tx=0x7f1fa0009bb0 comp rx=0 tx=0).stop 2024-03-23T22:51:11.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.487+0000 7f1fb5e00700 1 -- 172.21.15.12:0/1676171013 shutdown_connections 2024-03-23T22:51:11.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.487+0000 7f1fb5e00700 1 --2- 172.21.15.12:0/1676171013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fb0106240 0x7f1fb0108670 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:11.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.487+0000 7f1fb5e00700 1 -- 172.21.15.12:0/1676171013 >> 172.21.15.12:0/1676171013 conn(0x7f1fb0101f40 msgr2=0x7f1fb0104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:11.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.487+0000 7f1fb5e00700 1 -- 172.21.15.12:0/1676171013 shutdown_connections 2024-03-23T22:51:11.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.488+0000 7f1fb5e00700 1 -- 172.21.15.12:0/1676171013 wait complete. 2024-03-23T22:51:11.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.489+0000 7f1fb5e00700 1 Processor -- start 2024-03-23T22:51:11.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.489+0000 7f1fb5e00700 1 -- start start 2024-03-23T22:51:11.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.489+0000 7f1fb5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fb0106240 0x7f1fb01a1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:11.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.489+0000 7f1fb5e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb0109c80 con 0x7f1fb0106240 2024-03-23T22:51:11.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.490+0000 7f1faea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fb0106240 0x7f1fb01a1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:11.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.490+0000 7f1faea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fb0106240 0x7f1fb01a1f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46402/0 (socket says 172.21.15.12:46402) 2024-03-23T22:51:11.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.490+0000 7f1faea00700 1 -- 172.21.15.12:0/3765324209 learned_addr learned my addr 172.21.15.12:0/3765324209 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:11.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.490+0000 7f1faea00700 1 -- 172.21.15.12:0/3765324209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb01a24b0 con 0x7f1fb0106240 2024-03-23T22:51:11.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.490+0000 7f1faea00700 1 --2- 172.21.15.12:0/3765324209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fb0106240 0x7f1fb01a1f70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1fa00117e0 tx=0x7f1fa0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:11.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.491+0000 7f1fb4a00700 1 -- 172.21.15.12:0/3765324209 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1fa0003bf0 con 0x7f1fb0106240 2024-03-23T22:51:11.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.491+0000 7f1fb4a00700 1 -- 172.21.15.12:0/3765324209 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1fa001a910 con 0x7f1fb0106240 2024-03-23T22:51:11.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.491+0000 7f1fb5e00700 1 -- 172.21.15.12:0/3765324209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fb01a2740 con 0x7f1fb0106240 2024-03-23T22:51:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.491+0000 7f1fb4a00700 1 -- 172.21.15.12:0/3765324209 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1fa0024970 con 0x7f1fb0106240 2024-03-23T22:51:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.491+0000 7f1fb5e00700 1 -- 172.21.15.12:0/3765324209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fb01a2ba0 con 0x7f1fb0106240 2024-03-23T22:51:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.492+0000 7f1fb5e00700 1 -- 172.21.15.12:0/3765324209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fb00637c0 con 0x7f1fb0106240 2024-03-23T22:51:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.492+0000 7f1fb4a00700 1 -- 172.21.15.12:0/3765324209 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f1fa0022020 con 0x7f1fb0106240 2024-03-23T22:51:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.493+0000 7f1fb4a00700 1 --2- 172.21.15.12:0/3765324209 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7f1f94038ed0 0x7f1f9403b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.493+0000 7f1fb4a00700 1 -- 172.21.15.12:0/3765324209 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1fa004d290 con 0x7f1fb0106240 2024-03-23T22:51:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.493+0000 7f1fae000700 1 --2- 172.21.15.12:0/3765324209 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7f1f94038ed0 0x7f1f9403b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.494+0000 7f1fae000700 1 --2- 172.21.15.12:0/3765324209 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7f1f94038ed0 0x7f1f9403b390 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c006fd0 tx=0x7f1f9c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.498+0000 7f1fb4a00700 1 -- 172.21.15.12:0/3765324209 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1fa0011e20 con 0x7f1fb0106240 2024-03-23T22:51:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.732+0000 7f1fb5e00700 1 -- 172.21.15.12:0/3765324209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f1fb0062da0 con 0x7f1fb0106240 2024-03-23T22:51:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.736+0000 7f1fb4a00700 1 -- 172.21.15.12:0/3765324209 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f1fa0019c70 con 0x7f1fb0106240 2024-03-23T22:51:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.738+0000 7f1fb5e00700 1 -- 172.21.15.12:0/3765324209 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7f1f94038ed0 msgr2=0x7f1f9403b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.738+0000 7f1fb5e00700 1 --2- 172.21.15.12:0/3765324209 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7f1f94038ed0 0x7f1f9403b390 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c006fd0 tx=0x7f1f9c006e40 comp rx=0 tx=0).stop 2024-03-23T22:51:11.023 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.738+0000 7f1fb5e00700 1 -- 172.21.15.12:0/3765324209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fb0106240 msgr2=0x7f1fb01a1f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:11.023 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.738+0000 7f1fb5e00700 1 --2- 172.21.15.12:0/3765324209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fb0106240 0x7f1fb01a1f70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1fa00117e0 tx=0x7f1fa0011c10 comp rx=0 tx=0).stop 2024-03-23T22:51:11.023 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.738+0000 7f1fb5e00700 1 -- 172.21.15.12:0/3765324209 shutdown_connections 2024-03-23T22:51:11.023 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.738+0000 7f1fb5e00700 1 --2- 172.21.15.12:0/3765324209 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7f1f94038ed0 0x7f1f9403b390 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:11.023 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.738+0000 7f1fb5e00700 1 --2- 172.21.15.12:0/3765324209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fb0106240 0x7f1fb01a1f70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:11.023 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.738+0000 7f1fb5e00700 1 -- 172.21.15.12:0/3765324209 >> 172.21.15.12:0/3765324209 conn(0x7f1fb0101f40 msgr2=0x7f1fb0104330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:11.023 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.739+0000 7f1fb5e00700 1 -- 172.21.15.12:0/3765324209 shutdown_connections 2024-03-23T22:51:11.023 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:09.739+0000 7f1fb5e00700 1 -- 172.21.15.12:0/3765324209 wait complete. 2024-03-23T22:51:11.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:11 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' 2024-03-23T22:51:12.520 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.638+0000 7ff202c00700 1 Processor -- start 2024-03-23T22:51:12.520 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.638+0000 7ff202c00700 1 -- start start 2024-03-23T22:51:12.520 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.639+0000 7ff202c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1fc107a10 0x7ff1fc107e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:12.520 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.639+0000 7ff202c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc108400 con 0x7ff1fc107a10 2024-03-23T22:51:12.520 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.639+0000 7ff1fbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1fc107a10 0x7ff1fc107e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:12.520 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.639+0000 7ff1fbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1fc107a10 0x7ff1fc107e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46410/0 (socket says 172.21.15.12:46410) 2024-03-23T22:51:12.520 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.639+0000 7ff1fbe00700 1 -- 172.21.15.12:0/1600506613 learned_addr learned my addr 172.21.15.12:0/1600506613 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:12.520 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.640+0000 7ff1fbe00700 1 -- 172.21.15.12:0/1600506613 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1fc108c10 con 0x7ff1fc107a10 2024-03-23T22:51:12.520 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.640+0000 7ff1fbe00700 1 --2- 172.21.15.12:0/1600506613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1fc107a10 0x7ff1fc107e30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec0098a0 tx=0x7ff1ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f5686f6361543ae server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:12.521 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.641+0000 7ff1faa00700 1 -- 172.21.15.12:0/1600506613 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1ec004160 con 0x7ff1fc107a10 2024-03-23T22:51:12.521 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.641+0000 7ff1faa00700 1 -- 172.21.15.12:0/1600506613 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1ec004300 con 0x7ff1fc107a10 2024-03-23T22:51:12.521 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.641+0000 7ff1faa00700 1 -- 172.21.15.12:0/1600506613 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1ec011b20 con 0x7ff1fc107a10 2024-03-23T22:51:12.521 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.641+0000 7ff202c00700 1 -- 172.21.15.12:0/1600506613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1fc107a10 msgr2=0x7ff1fc107e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:12.521 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.641+0000 7ff202c00700 1 --2- 172.21.15.12:0/1600506613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1fc107a10 0x7ff1fc107e30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec0098a0 tx=0x7ff1ec009bb0 comp rx=0 tx=0).stop 2024-03-23T22:51:12.521 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.642+0000 7ff202c00700 1 -- 172.21.15.12:0/1600506613 shutdown_connections 2024-03-23T22:51:12.521 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.642+0000 7ff202c00700 1 --2- 172.21.15.12:0/1600506613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1fc107a10 0x7ff1fc107e30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:12.521 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.642+0000 7ff202c00700 1 -- 172.21.15.12:0/1600506613 >> 172.21.15.12:0/1600506613 conn(0x7ff1fc103090 msgr2=0x7ff1fc1054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:12.521 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.642+0000 7ff202c00700 1 -- 172.21.15.12:0/1600506613 shutdown_connections 2024-03-23T22:51:12.521 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.642+0000 7ff202c00700 1 -- 172.21.15.12:0/1600506613 wait complete. 2024-03-23T22:51:12.521 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.643+0000 7ff202c00700 1 Processor -- start 2024-03-23T22:51:12.521 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.644+0000 7ff202c00700 1 -- start start 2024-03-23T22:51:12.521 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.644+0000 7ff202c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1fc107a10 0x7ff1fc07cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:12.522 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.644+0000 7ff202c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc10bfc0 con 0x7ff1fc107a10 2024-03-23T22:51:12.522 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.644+0000 7ff1fbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1fc107a10 0x7ff1fc07cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:12.522 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.644+0000 7ff1fbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1fc107a10 0x7ff1fc07cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46412/0 (socket says 172.21.15.12:46412) 2024-03-23T22:51:12.522 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.644+0000 7ff1fbe00700 1 -- 172.21.15.12:0/2015514728 learned_addr learned my addr 172.21.15.12:0/2015514728 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:12.522 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.645+0000 7ff1fbe00700 1 -- 172.21.15.12:0/2015514728 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1fc07d090 con 0x7ff1fc107a10 2024-03-23T22:51:12.522 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.645+0000 7ff1fbe00700 1 --2- 172.21.15.12:0/2015514728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1fc107a10 0x7ff1fc07cb50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec0117e0 tx=0x7ff1ec011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:12.522 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.645+0000 7ff1f8c00700 1 -- 172.21.15.12:0/2015514728 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1ec003bf0 con 0x7ff1fc107a10 2024-03-23T22:51:12.522 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.646+0000 7ff1f8c00700 1 -- 172.21.15.12:0/2015514728 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1ec01a910 con 0x7ff1fc107a10 2024-03-23T22:51:12.522 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.646+0000 7ff1f8c00700 1 -- 172.21.15.12:0/2015514728 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1ec024970 con 0x7ff1fc107a10 2024-03-23T22:51:12.522 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.646+0000 7ff202c00700 1 -- 172.21.15.12:0/2015514728 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1fc07b280 con 0x7ff1fc107a10 2024-03-23T22:51:12.522 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.646+0000 7ff202c00700 1 -- 172.21.15.12:0/2015514728 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1fc07b740 con 0x7ff1fc107a10 2024-03-23T22:51:12.522 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.647+0000 7ff1f8c00700 1 -- 172.21.15.12:0/2015514728 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7ff1ec022020 con 0x7ff1fc107a10 2024-03-23T22:51:12.523 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.648+0000 7ff202c00700 1 -- 172.21.15.12:0/2015514728 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1fc0637c0 con 0x7ff1fc107a10 2024-03-23T22:51:12.523 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.648+0000 7ff1f8c00700 1 --2- 172.21.15.12:0/2015514728 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7ff1e0038f70 0x7ff1e003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:12.523 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.648+0000 7ff1f8c00700 1 -- 172.21.15.12:0/2015514728 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff1ec04d450 con 0x7ff1fc107a10 2024-03-23T22:51:12.523 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.648+0000 7ff1fb400700 1 --2- 172.21.15.12:0/2015514728 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7ff1e0038f70 0x7ff1e003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:12.523 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.649+0000 7ff1fb400700 1 --2- 172.21.15.12:0/2015514728 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7ff1e0038f70 0x7ff1e003b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8006fd0 tx=0x7ff1e8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:12.523 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.654+0000 7ff1f8c00700 1 -- 172.21.15.12:0/2015514728 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff1ec029060 con 0x7ff1fc107a10 2024-03-23T22:51:12.523 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.900+0000 7ff202c00700 1 -- 172.21.15.12:0/2015514728 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7ff1fc07b9c0 con 0x7ff1fc107a10 2024-03-23T22:51:12.523 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.909+0000 7ff1f8c00700 1 -- 172.21.15.12:0/2015514728 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7ff1ec00b3e0 con 0x7ff1fc107a10 2024-03-23T22:51:12.523 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.912+0000 7ff202c00700 1 -- 172.21.15.12:0/2015514728 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7ff1e0038f70 msgr2=0x7ff1e003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:12.523 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.912+0000 7ff202c00700 1 --2- 172.21.15.12:0/2015514728 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7ff1e0038f70 0x7ff1e003b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8006fd0 tx=0x7ff1e8006e40 comp rx=0 tx=0).stop 2024-03-23T22:51:12.523 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.912+0000 7ff202c00700 1 -- 172.21.15.12:0/2015514728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1fc107a10 msgr2=0x7ff1fc07cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:12.523 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.912+0000 7ff202c00700 1 --2- 172.21.15.12:0/2015514728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1fc107a10 0x7ff1fc07cb50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec0117e0 tx=0x7ff1ec011ba0 comp rx=0 tx=0).stop 2024-03-23T22:51:12.524 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.912+0000 7ff202c00700 1 -- 172.21.15.12:0/2015514728 shutdown_connections 2024-03-23T22:51:12.524 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.912+0000 7ff202c00700 1 --2- 172.21.15.12:0/2015514728 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7ff1e0038f70 0x7ff1e003b430 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:12.524 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.912+0000 7ff202c00700 1 --2- 172.21.15.12:0/2015514728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1fc107a10 0x7ff1fc07cb50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:12.524 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.912+0000 7ff202c00700 1 -- 172.21.15.12:0/2015514728 >> 172.21.15.12:0/2015514728 conn(0x7ff1fc103090 msgr2=0x7ff1fc103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:12.524 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.912+0000 7ff202c00700 1 -- 172.21.15.12:0/2015514728 shutdown_connections 2024-03-23T22:51:12.524 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:11.913+0000 7ff202c00700 1 -- 172.21.15.12:0/2015514728 wait complete. 2024-03-23T22:51:12.524 INFO:teuthology.orchestra.run.smithi012.stdout:Enabling the dashboard module... 2024-03-23T22:51:13.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:12 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2015514728' entity='client.admin' 2024-03-23T22:51:13.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:12 smithi012 ceph-mon[31695]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:51:13.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:12 smithi012 ceph-mon[31695]: from='mgr.14120 172.21.15.12:0/3651273136' entity='mgr.x' 2024-03-23T22:51:14.246 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:13 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/26107643' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-23T22:51:14.246 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:14 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: ignoring --setuser ceph since I am not root 2024-03-23T22:51:14.247 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:14 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: ignoring --setgroup ceph since I am not root 2024-03-23T22:51:14.924 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:14 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:51:14.775+0000 7f708cb93200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-23T22:51:14.924 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:14 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:51:14.899+0000 7f708cb93200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-23T22:51:15.181 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:14 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/26107643' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-23T22:51:15.181 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:14 smithi012 ceph-mon[31695]: mgrmap e9: x(active, since 25s) 2024-03-23T22:51:15.504 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.066+0000 7f60db600700 1 Processor -- start 2024-03-23T22:51:15.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.066+0000 7f60db600700 1 -- start start 2024-03-23T22:51:15.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.066+0000 7f60db600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60d4109c40 0x7f60d410a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:15.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.066+0000 7f60db600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d410a630 con 0x7f60d4109c40 2024-03-23T22:51:15.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.067+0000 7f60d8c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60d4109c40 0x7f60d410a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:15.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.067+0000 7f60d8c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60d4109c40 0x7f60d410a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46414/0 (socket says 172.21.15.12:46414) 2024-03-23T22:51:15.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.067+0000 7f60d8c00700 1 -- 172.21.15.12:0/838962984 learned_addr learned my addr 172.21.15.12:0/838962984 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:15.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.067+0000 7f60d8c00700 1 -- 172.21.15.12:0/838962984 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d410adf0 con 0x7f60d4109c40 2024-03-23T22:51:15.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.068+0000 7f60d8c00700 1 --2- 172.21.15.12:0/838962984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60d4109c40 0x7f60d410a060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f60c80098a0 tx=0x7f60c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4f2253fb7114b4c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:15.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.068+0000 7f60d3400700 1 -- 172.21.15.12:0/838962984 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f60c8004160 con 0x7f60d4109c40 2024-03-23T22:51:15.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.068+0000 7f60d3400700 1 -- 172.21.15.12:0/838962984 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60c8004300 con 0x7f60d4109c40 2024-03-23T22:51:15.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.068+0000 7f60d3400700 1 -- 172.21.15.12:0/838962984 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f60c8011b20 con 0x7f60d4109c40 2024-03-23T22:51:15.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.069+0000 7f60db600700 1 -- 172.21.15.12:0/838962984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60d4109c40 msgr2=0x7f60d410a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:15.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.069+0000 7f60db600700 1 --2- 172.21.15.12:0/838962984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60d4109c40 0x7f60d410a060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f60c80098a0 tx=0x7f60c8009bb0 comp rx=0 tx=0).stop 2024-03-23T22:51:15.506 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.069+0000 7f60db600700 1 -- 172.21.15.12:0/838962984 shutdown_connections 2024-03-23T22:51:15.506 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.069+0000 7f60db600700 1 --2- 172.21.15.12:0/838962984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60d4109c40 0x7f60d410a060 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:15.506 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.069+0000 7f60db600700 1 -- 172.21.15.12:0/838962984 >> 172.21.15.12:0/838962984 conn(0x7f60d41054a0 msgr2=0x7f60d4107890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:15.506 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.069+0000 7f60db600700 1 -- 172.21.15.12:0/838962984 shutdown_connections 2024-03-23T22:51:15.506 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.069+0000 7f60db600700 1 -- 172.21.15.12:0/838962984 wait complete. 2024-03-23T22:51:15.506 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.070+0000 7f60db600700 1 Processor -- start 2024-03-23T22:51:15.506 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.071+0000 7f60db600700 1 -- start start 2024-03-23T22:51:15.506 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.071+0000 7f60db600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60d4109c40 0x7f60d419db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:15.506 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.071+0000 7f60db600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d410e1a0 con 0x7f60d4109c40 2024-03-23T22:51:15.506 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.071+0000 7f60d8c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60d4109c40 0x7f60d419db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:15.506 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.071+0000 7f60d8c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60d4109c40 0x7f60d419db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46418/0 (socket says 172.21.15.12:46418) 2024-03-23T22:51:15.506 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.071+0000 7f60d8c00700 1 -- 172.21.15.12:0/26107643 learned_addr learned my addr 172.21.15.12:0/26107643 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:15.506 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.072+0000 7f60d8c00700 1 -- 172.21.15.12:0/26107643 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d419e040 con 0x7f60d4109c40 2024-03-23T22:51:15.506 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.072+0000 7f60d8c00700 1 --2- 172.21.15.12:0/26107643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60d4109c40 0x7f60d419db00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f60c80117e0 tx=0x7f60c8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:15.506 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.072+0000 7f60d1600700 1 -- 172.21.15.12:0/26107643 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f60c8003bf0 con 0x7f60d4109c40 2024-03-23T22:51:15.506 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.072+0000 7f60d1600700 1 -- 172.21.15.12:0/26107643 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60c801a910 con 0x7f60d4109c40 2024-03-23T22:51:15.509 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.072+0000 7f60db600700 1 -- 172.21.15.12:0/26107643 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60d419e2d0 con 0x7f60d4109c40 2024-03-23T22:51:15.509 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.073+0000 7f60d1600700 1 -- 172.21.15.12:0/26107643 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f60c8024970 con 0x7f60d4109c40 2024-03-23T22:51:15.509 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.073+0000 7f60db600700 1 -- 172.21.15.12:0/26107643 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60d419e730 con 0x7f60d4109c40 2024-03-23T22:51:15.509 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.074+0000 7f60d1600700 1 -- 172.21.15.12:0/26107643 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f60c8022020 con 0x7f60d4109c40 2024-03-23T22:51:15.509 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.074+0000 7f60db600700 1 -- 172.21.15.12:0/26107643 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60d40637c0 con 0x7f60d4109c40 2024-03-23T22:51:15.509 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.074+0000 7f60d1600700 1 --2- 172.21.15.12:0/26107643 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7f60bc038f20 0x7f60bc03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:15.509 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.074+0000 7f60d1600700 1 -- 172.21.15.12:0/26107643 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f60c804d5b0 con 0x7f60d4109c40 2024-03-23T22:51:15.509 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.075+0000 7f60d3e00700 1 --2- 172.21.15.12:0/26107643 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7f60bc038f20 0x7f60bc03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:15.509 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.075+0000 7f60d3e00700 1 --2- 172.21.15.12:0/26107643 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7f60bc038f20 0x7f60bc03b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f60c0006fd0 tx=0x7f60c0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:15.509 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.081+0000 7f60d1600700 1 -- 172.21.15.12:0/26107643 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f60c8017ca0 con 0x7f60d4109c40 2024-03-23T22:51:15.510 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.363+0000 7f60db600700 1 -- 172.21.15.12:0/26107643 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f60d41a1100 con 0x7f60d4109c40 2024-03-23T22:51:15.510 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.923+0000 7f60d1600700 1 -- 172.21.15.12:0/26107643 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f60c801ab70 con 0x7f60d4109c40 2024-03-23T22:51:15.510 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.927+0000 7f60d1600700 1 -- 172.21.15.12:0/26107643 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f60c802d3f0 con 0x7f60d4109c40 2024-03-23T22:51:15.510 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.929+0000 7f60db600700 1 -- 172.21.15.12:0/26107643 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7f60bc038f20 msgr2=0x7f60bc03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:15.510 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.929+0000 7f60db600700 1 --2- 172.21.15.12:0/26107643 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7f60bc038f20 0x7f60bc03b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f60c0006fd0 tx=0x7f60c0006e40 comp rx=0 tx=0).stop 2024-03-23T22:51:15.510 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.929+0000 7f60db600700 1 -- 172.21.15.12:0/26107643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60d4109c40 msgr2=0x7f60d419db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:15.510 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.929+0000 7f60db600700 1 --2- 172.21.15.12:0/26107643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60d4109c40 0x7f60d419db00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f60c80117e0 tx=0x7f60c8011ba0 comp rx=0 tx=0).stop 2024-03-23T22:51:15.510 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.929+0000 7f60db600700 1 -- 172.21.15.12:0/26107643 shutdown_connections 2024-03-23T22:51:15.510 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.929+0000 7f60db600700 1 --2- 172.21.15.12:0/26107643 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7f60bc038f20 0x7f60bc03b3e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:15.510 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.929+0000 7f60db600700 1 --2- 172.21.15.12:0/26107643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60d4109c40 0x7f60d419db00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:15.510 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.929+0000 7f60db600700 1 -- 172.21.15.12:0/26107643 >> 172.21.15.12:0/26107643 conn(0x7f60d41054a0 msgr2=0x7f60d4106f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:15.510 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.930+0000 7f60db600700 1 -- 172.21.15.12:0/26107643 shutdown_connections 2024-03-23T22:51:15.510 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:13.930+0000 7f60db600700 1 -- 172.21.15.12:0/26107643 wait complete. 2024-03-23T22:51:15.779 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:15 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:51:15.727+0000 7f708cb93200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-23T22:51:16.042 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:15 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:51:15.857+0000 7f708cb93200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-23T22:51:16.042 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:16 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:51:16.043+0000 7f708cb93200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-23T22:51:16.304 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:16 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:51:16.175+0000 7f708cb93200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-23T22:51:16.304 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:16 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:51:16.288+0000 7f708cb93200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-23T22:51:16.637 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:16 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1402046173' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-23T22:51:16.901 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:16 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:51:16.637+0000 7f708cb93200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-23T22:51:16.901 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:16 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:51:16.785+0000 7f708cb93200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-23T22:51:16.902 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:16 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:51:16.901+0000 7f708cb93200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-23T22:51:17.026 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-03-23T22:51:17.027 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-23T22:51:17.027 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-23T22:51:17.027 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-23T22:51:17.027 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-23T22:51:17.027 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-03-23T22:51:17.027 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.064+0000 7fd92d400700 1 Processor -- start 2024-03-23T22:51:17.027 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.064+0000 7fd92d400700 1 -- start start 2024-03-23T22:51:17.027 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.064+0000 7fd92d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd928109c80 0x7fd92810a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:17.027 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.064+0000 7fd92d400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92810a670 con 0x7fd928109c80 2024-03-23T22:51:17.027 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.065+0000 7fd926800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd928109c80 0x7fd92810a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:17.027 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.065+0000 7fd926800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd928109c80 0x7fd92810a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33754/0 (socket says 172.21.15.12:33754) 2024-03-23T22:51:17.027 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.065+0000 7fd926800700 1 -- 172.21.15.12:0/4255608226 learned_addr learned my addr 172.21.15.12:0/4255608226 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:17.027 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.065+0000 7fd926800700 1 -- 172.21.15.12:0/4255608226 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92810ade0 con 0x7fd928109c80 2024-03-23T22:51:17.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.065+0000 7fd926800700 1 --2- 172.21.15.12:0/4255608226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd928109c80 0x7fd92810a0a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd9180098a0 tx=0x7fd918009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42955a32ac5534fe server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:17.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.066+0000 7fd925e00700 1 -- 172.21.15.12:0/4255608226 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd918004160 con 0x7fd928109c80 2024-03-23T22:51:17.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.066+0000 7fd925e00700 1 -- 172.21.15.12:0/4255608226 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd918004300 con 0x7fd928109c80 2024-03-23T22:51:17.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.066+0000 7fd925e00700 1 -- 172.21.15.12:0/4255608226 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd918011b20 con 0x7fd928109c80 2024-03-23T22:51:17.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.066+0000 7fd92d400700 1 -- 172.21.15.12:0/4255608226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd928109c80 msgr2=0x7fd92810a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:17.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.066+0000 7fd92d400700 1 --2- 172.21.15.12:0/4255608226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd928109c80 0x7fd92810a0a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd9180098a0 tx=0x7fd918009bb0 comp rx=0 tx=0).stop 2024-03-23T22:51:17.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.067+0000 7fd92d400700 1 -- 172.21.15.12:0/4255608226 shutdown_connections 2024-03-23T22:51:17.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.067+0000 7fd92d400700 1 --2- 172.21.15.12:0/4255608226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd928109c80 0x7fd92810a0a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:17.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.067+0000 7fd92d400700 1 -- 172.21.15.12:0/4255608226 >> 172.21.15.12:0/4255608226 conn(0x7fd9281054a0 msgr2=0x7fd928107890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:17.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.067+0000 7fd92d400700 1 -- 172.21.15.12:0/4255608226 shutdown_connections 2024-03-23T22:51:17.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.067+0000 7fd92d400700 1 -- 172.21.15.12:0/4255608226 wait complete. 2024-03-23T22:51:17.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.068+0000 7fd92d400700 1 Processor -- start 2024-03-23T22:51:17.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.068+0000 7fd92d400700 1 -- start start 2024-03-23T22:51:17.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.068+0000 7fd92d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd928109c80 0x7fd92819dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:17.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.068+0000 7fd92d400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92810e190 con 0x7fd928109c80 2024-03-23T22:51:17.029 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.068+0000 7fd926800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd928109c80 0x7fd92819dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:17.029 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.068+0000 7fd926800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd928109c80 0x7fd92819dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33762/0 (socket says 172.21.15.12:33762) 2024-03-23T22:51:17.029 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.068+0000 7fd926800700 1 -- 172.21.15.12:0/1402046173 learned_addr learned my addr 172.21.15.12:0/1402046173 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:17.029 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.069+0000 7fd926800700 1 -- 172.21.15.12:0/1402046173 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92819e0f0 con 0x7fd928109c80 2024-03-23T22:51:17.029 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.069+0000 7fd926800700 1 --2- 172.21.15.12:0/1402046173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd928109c80 0x7fd92819dbb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd918000c00 tx=0x7fd918011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:17.029 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.069+0000 7fd91fe00700 1 -- 172.21.15.12:0/1402046173 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd918003b60 con 0x7fd928109c80 2024-03-23T22:51:17.032 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.069+0000 7fd91fe00700 1 -- 172.21.15.12:0/1402046173 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd91801a910 con 0x7fd928109c80 2024-03-23T22:51:17.032 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.069+0000 7fd92d400700 1 -- 172.21.15.12:0/1402046173 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd92819e380 con 0x7fd928109c80 2024-03-23T22:51:17.032 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.069+0000 7fd91fe00700 1 -- 172.21.15.12:0/1402046173 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd918024930 con 0x7fd928109c80 2024-03-23T22:51:17.032 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.070+0000 7fd92d400700 1 -- 172.21.15.12:0/1402046173 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd92819e7e0 con 0x7fd928109c80 2024-03-23T22:51:17.032 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.071+0000 7fd92d400700 1 -- 172.21.15.12:0/1402046173 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9280637c0 con 0x7fd928109c80 2024-03-23T22:51:17.032 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.071+0000 7fd91fe00700 1 -- 172.21.15.12:0/1402046173 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fd918022020 con 0x7fd928109c80 2024-03-23T22:51:17.032 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.071+0000 7fd91fe00700 1 --2- 172.21.15.12:0/1402046173 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fd90c038f70 0x7fd90c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:17.032 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.071+0000 7fd91fe00700 1 -- 172.21.15.12:0/1402046173 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd91804d1b0 con 0x7fd928109c80 2024-03-23T22:51:17.032 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.072+0000 7fd91de00700 1 -- 172.21.15.12:0/1402046173 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fd90c038f70 msgr2=0x7fd90c03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/4127975083 2024-03-23T22:51:17.033 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.072+0000 7fd91de00700 1 --2- 172.21.15.12:0/1402046173 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fd90c038f70 0x7fd90c03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T22:51:17.033 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.077+0000 7fd91fe00700 1 -- 172.21.15.12:0/1402046173 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd91801ab70 con 0x7fd928109c80 2024-03-23T22:51:17.033 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.273+0000 7fd91de00700 1 -- 172.21.15.12:0/1402046173 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fd90c038f70 msgr2=0x7fd90c03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/4127975083 2024-03-23T22:51:17.033 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.273+0000 7fd91de00700 1 --2- 172.21.15.12:0/1402046173 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fd90c038f70 0x7fd90c03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T22:51:17.033 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.359+0000 7fd92d400700 1 -- 172.21.15.12:0/1402046173 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fd9281a1250 con 0x7fd928109c80 2024-03-23T22:51:17.033 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.359+0000 7fd91fe00700 1 -- 172.21.15.12:0/1402046173 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fd91804cb00 con 0x7fd928109c80 2024-03-23T22:51:17.033 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.362+0000 7fd92d400700 1 -- 172.21.15.12:0/1402046173 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fd90c038f70 msgr2=0x7fd90c03b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-23T22:51:17.033 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.362+0000 7fd92d400700 1 --2- 172.21.15.12:0/1402046173 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fd90c038f70 0x7fd90c03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:17.033 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.362+0000 7fd92d400700 1 -- 172.21.15.12:0/1402046173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd928109c80 msgr2=0x7fd92819dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:17.033 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.362+0000 7fd92d400700 1 --2- 172.21.15.12:0/1402046173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd928109c80 0x7fd92819dbb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd918000c00 tx=0x7fd918011c10 comp rx=0 tx=0).stop 2024-03-23T22:51:17.033 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.362+0000 7fd92d400700 1 -- 172.21.15.12:0/1402046173 shutdown_connections 2024-03-23T22:51:17.033 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.362+0000 7fd92d400700 1 --2- 172.21.15.12:0/1402046173 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7fd90c038f70 0x7fd90c03b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:17.033 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.362+0000 7fd92d400700 1 --2- 172.21.15.12:0/1402046173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd928109c80 0x7fd92819dbb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:17.033 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.362+0000 7fd92d400700 1 -- 172.21.15.12:0/1402046173 >> 172.21.15.12:0/1402046173 conn(0x7fd9281054a0 msgr2=0x7fd928106f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:17.033 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.363+0000 7fd92d400700 1 -- 172.21.15.12:0/1402046173 shutdown_connections 2024-03-23T22:51:17.034 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:16.363+0000 7fd92d400700 1 -- 172.21.15.12:0/1402046173 wait complete. 2024-03-23T22:51:17.034 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for the mgr to restart... 2024-03-23T22:51:17.034 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mgr epoch 9... 2024-03-23T22:51:17.182 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:17 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:51:17.035+0000 7f708cb93200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-23T22:51:17.478 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:17 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:51:17.269+0000 7f708cb93200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-23T22:51:19.046 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:18 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:51:18.597+0000 7f708cb93200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-23T22:51:19.046 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:18 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:51:18.774+0000 7f708cb93200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-23T22:51:19.304 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:19 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:51:19.203+0000 7f708cb93200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-23T22:51:19.595 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:19 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:51:19.327+0000 7f708cb93200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-23T22:51:19.849 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:19 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:51:19.595+0000 7f708cb93200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-23T22:51:20.238 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:19 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:51:19.850+0000 7f708cb93200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-23T22:51:20.553 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:20 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:51:20.238+0000 7f708cb93200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-23T22:51:20.879 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:20 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:51:20.564+0000 7f708cb93200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-23T22:51:21.303 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:20 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:51:20.879+0000 7f708cb93200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-23T22:51:22.303 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:21 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:51:21.963+0000 7f708cb93200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-23T22:51:23.053 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:22 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:51:22.681+0000 7f708cb93200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-23T22:51:23.054 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:22 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:51:22.818+0000 7f708cb93200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-23T22:51:23.553 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:51:23 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:51:23.110+0000 7f708cb93200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-23T22:51:23.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:23 smithi012 ceph-mon[31695]: Active manager daemon x restarted 2024-03-23T22:51:23.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:23 smithi012 ceph-mon[31695]: Activating manager daemon x 2024-03-23T22:51:23.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:23 smithi012 ceph-mon[31695]: osdmap e3: 0 total, 0 up, 0 in 2024-03-23T22:51:23.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:23 smithi012 ceph-mon[31695]: mgrmap e10: x(active, starting, since 0.00800885s) 2024-03-23T22:51:23.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:23 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-23T22:51:23.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:23 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-23T22:51:23.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:23 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-23T22:51:23.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:23 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-23T22:51:23.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:23 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-23T22:51:23.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:23 smithi012 ceph-mon[31695]: Manager daemon x is now available 2024-03-23T22:51:24.450 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:51:24.450 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:51:24.450 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-23T22:51:24.450 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-23T22:51:24.450 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:24 smithi012 ceph-mon[31695]: [23/Mar/2024:22:51:23] ENGINE Bus STARTING 2024-03-23T22:51:24.450 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:24 smithi012 ceph-mon[31695]: [23/Mar/2024:22:51:23] ENGINE Serving on http://172.21.15.12:8765 2024-03-23T22:51:24.450 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:24 smithi012 ceph-mon[31695]: [23/Mar/2024:22:51:23] ENGINE Serving on https://172.21.15.12:7150 2024-03-23T22:51:24.450 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:24 smithi012 ceph-mon[31695]: [23/Mar/2024:22:51:23] ENGINE Bus STARTED 2024-03-23T22:51:24.450 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:24 smithi012 ceph-mon[31695]: mgrmap e11: x(active, since 1.01048s) 2024-03-23T22:51:24.817 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-03-23T22:51:24.817 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-23T22:51:24.817 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-23T22:51:24.817 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-03-23T22:51:24.817 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.635+0000 7ff20b200700 1 Processor -- start 2024-03-23T22:51:24.818 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.635+0000 7ff20b200700 1 -- start start 2024-03-23T22:51:24.818 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.635+0000 7ff20b200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2041090c0 0x7ff2041094e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:24.818 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.635+0000 7ff20b200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff204107600 con 0x7ff2041090c0 2024-03-23T22:51:24.818 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.635+0000 7ff203e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2041090c0 0x7ff2041094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:24.818 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.636+0000 7ff203e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2041090c0 0x7ff2041094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33774/0 (socket says 172.21.15.12:33774) 2024-03-23T22:51:24.818 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.636+0000 7ff203e00700 1 -- 172.21.15.12:0/1124474900 learned_addr learned my addr 172.21.15.12:0/1124474900 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:24.818 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.636+0000 7ff203e00700 1 -- 172.21.15.12:0/1124474900 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff204107780 con 0x7ff2041090c0 2024-03-23T22:51:24.818 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.636+0000 7ff203e00700 1 --2- 172.21.15.12:0/1124474900 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2041090c0 0x7ff2041094e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff1f40098a0 tx=0x7ff1f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddc024996c4894d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:24.818 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.637+0000 7ff202a00700 1 -- 172.21.15.12:0/1124474900 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1f4004160 con 0x7ff2041090c0 2024-03-23T22:51:24.818 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.637+0000 7ff202a00700 1 -- 172.21.15.12:0/1124474900 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1f4004300 con 0x7ff2041090c0 2024-03-23T22:51:24.818 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.637+0000 7ff202a00700 1 -- 172.21.15.12:0/1124474900 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1f4011b20 con 0x7ff2041090c0 2024-03-23T22:51:24.819 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.637+0000 7ff20b200700 1 -- 172.21.15.12:0/1124474900 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2041090c0 msgr2=0x7ff2041094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:24.819 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.637+0000 7ff20b200700 1 --2- 172.21.15.12:0/1124474900 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2041090c0 0x7ff2041094e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff1f40098a0 tx=0x7ff1f4009bb0 comp rx=0 tx=0).stop 2024-03-23T22:51:24.819 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.638+0000 7ff20b200700 1 -- 172.21.15.12:0/1124474900 shutdown_connections 2024-03-23T22:51:24.819 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.638+0000 7ff20b200700 1 --2- 172.21.15.12:0/1124474900 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2041090c0 0x7ff2041094e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:24.819 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.638+0000 7ff20b200700 1 -- 172.21.15.12:0/1124474900 >> 172.21.15.12:0/1124474900 conn(0x7ff204103000 msgr2=0x7ff204105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:24.819 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.638+0000 7ff20b200700 1 -- 172.21.15.12:0/1124474900 shutdown_connections 2024-03-23T22:51:24.819 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.638+0000 7ff20b200700 1 -- 172.21.15.12:0/1124474900 wait complete. 2024-03-23T22:51:24.819 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.639+0000 7ff20b200700 1 Processor -- start 2024-03-23T22:51:24.819 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.639+0000 7ff20b200700 1 -- start start 2024-03-23T22:51:24.819 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.639+0000 7ff20b200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2041090c0 0x7ff20419dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:24.819 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.639+0000 7ff20b200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20410bf90 con 0x7ff2041090c0 2024-03-23T22:51:24.819 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.640+0000 7ff203e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2041090c0 0x7ff20419dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:24.819 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.640+0000 7ff203e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2041090c0 0x7ff20419dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33790/0 (socket says 172.21.15.12:33790) 2024-03-23T22:51:24.820 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.640+0000 7ff203e00700 1 -- 172.21.15.12:0/626761533 learned_addr learned my addr 172.21.15.12:0/626761533 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:24.820 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.640+0000 7ff203e00700 1 -- 172.21.15.12:0/626761533 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20419e220 con 0x7ff2041090c0 2024-03-23T22:51:24.820 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.640+0000 7ff203e00700 1 --2- 172.21.15.12:0/626761533 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2041090c0 0x7ff20419dce0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4000c00 tx=0x7ff1f4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:24.820 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.641+0000 7ff200c00700 1 -- 172.21.15.12:0/626761533 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1f4003b60 con 0x7ff2041090c0 2024-03-23T22:51:24.820 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.641+0000 7ff200c00700 1 -- 172.21.15.12:0/626761533 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1f401a910 con 0x7ff2041090c0 2024-03-23T22:51:24.824 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.641+0000 7ff200c00700 1 -- 172.21.15.12:0/626761533 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1f4024930 con 0x7ff2041090c0 2024-03-23T22:51:24.824 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.641+0000 7ff20b200700 1 -- 172.21.15.12:0/626761533 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff20419e4b0 con 0x7ff2041090c0 2024-03-23T22:51:24.824 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.641+0000 7ff20b200700 1 -- 172.21.15.12:0/626761533 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff20419e910 con 0x7ff2041090c0 2024-03-23T22:51:24.824 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.642+0000 7ff200c00700 1 -- 172.21.15.12:0/626761533 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7ff1f4022020 con 0x7ff2041090c0 2024-03-23T22:51:24.824 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.643+0000 7ff200c00700 1 --2- 172.21.15.12:0/626761533 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7ff1e8038fd0 0x7ff1e803b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:24.825 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.643+0000 7ff200c00700 1 -- 172.21.15.12:0/626761533 --> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1e803bba0 con 0x7ff1e8038fd0 2024-03-23T22:51:24.825 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.643+0000 7ff200c00700 1 -- 172.21.15.12:0/626761533 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff1f404d100 con 0x7ff2041090c0 2024-03-23T22:51:24.825 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.643+0000 7ff203400700 1 -- 172.21.15.12:0/626761533 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7ff1e8038fd0 msgr2=0x7ff1e803b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/4127975083 2024-03-23T22:51:24.825 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.643+0000 7ff203400700 1 --2- 172.21.15.12:0/626761533 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7ff1e8038fd0 0x7ff1e803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T22:51:24.825 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.844+0000 7ff203400700 1 -- 172.21.15.12:0/626761533 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7ff1e8038fd0 msgr2=0x7ff1e803b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/4127975083 2024-03-23T22:51:24.825 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:17.844+0000 7ff203400700 1 --2- 172.21.15.12:0/626761533 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7ff1e8038fd0 0x7ff1e803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T22:51:24.825 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:18.245+0000 7ff203400700 1 -- 172.21.15.12:0/626761533 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7ff1e8038fd0 msgr2=0x7ff1e803b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/4127975083 2024-03-23T22:51:24.825 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:18.245+0000 7ff203400700 1 --2- 172.21.15.12:0/626761533 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7ff1e8038fd0 0x7ff1e803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-23T22:51:24.825 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:19.046+0000 7ff203400700 1 -- 172.21.15.12:0/626761533 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7ff1e8038fd0 msgr2=0x7ff1e803b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/4127975083 2024-03-23T22:51:24.825 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:19.046+0000 7ff203400700 1 --2- 172.21.15.12:0/626761533 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7ff1e8038fd0 0x7ff1e803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-23T22:51:24.825 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:20.648+0000 7ff203400700 1 -- 172.21.15.12:0/626761533 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7ff1e8038fd0 msgr2=0x7ff1e803b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/4127975083 2024-03-23T22:51:24.825 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:20.648+0000 7ff203400700 1 --2- 172.21.15.12:0/626761533 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7ff1e8038fd0 0x7ff1e803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-23T22:51:24.825 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:23.119+0000 7ff200c00700 1 -- 172.21.15.12:0/626761533 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mgrmap(e 10) v1 ==== 45886+0+0 (secure 0 0 0) 0x7ff1f402d3f0 con 0x7ff2041090c0 2024-03-23T22:51:24.825 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:23.119+0000 7ff200c00700 1 -- 172.21.15.12:0/626761533 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7ff1e8038fd0 msgr2=0x7ff1e803b490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-23T22:51:24.825 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:23.119+0000 7ff200c00700 1 --2- 172.21.15.12:0/626761533 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7ff1e8038fd0 0x7ff1e803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:24.826 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:24.120+0000 7ff200c00700 1 -- 172.21.15.12:0/626761533 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7ff1f404ecb0 con 0x7ff2041090c0 2024-03-23T22:51:24.826 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:24.120+0000 7ff200c00700 1 --2- 172.21.15.12:0/626761533 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff1e803ca80 0x7ff1e803ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:24.826 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:24.120+0000 7ff200c00700 1 -- 172.21.15.12:0/626761533 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1f4008ea0 con 0x7ff1e803ca80 2024-03-23T22:51:24.826 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:24.122+0000 7ff203400700 1 --2- 172.21.15.12:0/626761533 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff1e803ca80 0x7ff1e803ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:24.826 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:24.123+0000 7ff203400700 1 --2- 172.21.15.12:0/626761533 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff1e803ca80 0x7ff1e803ee70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0003a10 tx=0x7ff1f0009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:24.826 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:24.124+0000 7ff200c00700 1 -- 172.21.15.12:0/626761533 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7ff1f4008ea0 con 0x7ff1e803ca80 2024-03-23T22:51:24.826 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:24.133+0000 7ff20b200700 1 -- 172.21.15.12:0/626761533 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7ff20410bde0 con 0x7ff1e803ca80 2024-03-23T22:51:24.826 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:24.134+0000 7ff200c00700 1 -- 172.21.15.12:0/626761533 <== mgr.14152 v2:172.21.15.12:6800/1345467734 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7ff20410bde0 con 0x7ff1e803ca80 2024-03-23T22:51:24.826 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:24.134+0000 7ff20b200700 1 -- 172.21.15.12:0/626761533 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff1e803ca80 msgr2=0x7ff1e803ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:24.826 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:24.134+0000 7ff20b200700 1 --2- 172.21.15.12:0/626761533 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff1e803ca80 0x7ff1e803ee70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0003a10 tx=0x7ff1f0009270 comp rx=0 tx=0).stop 2024-03-23T22:51:24.826 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:24.134+0000 7ff20b200700 1 -- 172.21.15.12:0/626761533 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2041090c0 msgr2=0x7ff20419dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:24.826 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:24.134+0000 7ff20b200700 1 --2- 172.21.15.12:0/626761533 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2041090c0 0x7ff20419dce0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4000c00 tx=0x7ff1f4011c10 comp rx=0 tx=0).stop 2024-03-23T22:51:24.826 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:24.134+0000 7ff20b200700 1 -- 172.21.15.12:0/626761533 shutdown_connections 2024-03-23T22:51:24.827 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:24.134+0000 7ff20b200700 1 --2- 172.21.15.12:0/626761533 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2041090c0 0x7ff20419dce0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:24.827 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:24.134+0000 7ff20b200700 1 --2- 172.21.15.12:0/626761533 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff1e803ca80 0x7ff1e803ee70 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:24.827 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:24.134+0000 7ff20b200700 1 --2- 172.21.15.12:0/626761533 >> [v2:172.21.15.12:6800/4127975083,v1:172.21.15.12:6801/4127975083] conn(0x7ff1e8038fd0 0x7ff1e803b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:24.827 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:24.134+0000 7ff20b200700 1 -- 172.21.15.12:0/626761533 >> 172.21.15.12:0/626761533 conn(0x7ff204103000 msgr2=0x7ff204103b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:24.827 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:24.135+0000 7ff20b200700 1 -- 172.21.15.12:0/626761533 shutdown_connections 2024-03-23T22:51:24.827 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:24.135+0000 7ff20b200700 1 -- 172.21.15.12:0/626761533 wait complete. 2024-03-23T22:51:24.827 INFO:teuthology.orchestra.run.smithi012.stdout:mgr epoch 9 is available 2024-03-23T22:51:24.827 INFO:teuthology.orchestra.run.smithi012.stdout:Generating a dashboard self-signed certificate... 2024-03-23T22:51:25.375 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:25 smithi012 ceph-mon[31695]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-23T22:51:25.375 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:25 smithi012 ceph-mon[31695]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-23T22:51:26.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:26 smithi012 ceph-mon[31695]: mgrmap e12: x(active, since 2s) 2024-03-23T22:51:26.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:26 smithi012 ceph-mon[31695]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:51:26.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:26 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:51:26.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:26 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:51:26.632 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-23T22:51:26.638 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.628+0000 7f04ccc00700 1 Processor -- start 2024-03-23T22:51:26.638 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.628+0000 7f04ccc00700 1 -- start start 2024-03-23T22:51:26.638 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.628+0000 7f04ccc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04c81090c0 0x7f04c81094e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:26.638 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.628+0000 7f04ccc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c8107600 con 0x7f04c81090c0 2024-03-23T22:51:26.638 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.629+0000 7f04c5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04c81090c0 0x7f04c81094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:26.638 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.629+0000 7f04c5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04c81090c0 0x7f04c81094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42610/0 (socket says 172.21.15.12:42610) 2024-03-23T22:51:26.638 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.629+0000 7f04c5e00700 1 -- 172.21.15.12:0/65541177 learned_addr learned my addr 172.21.15.12:0/65541177 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:26.638 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.629+0000 7f04c5e00700 1 -- 172.21.15.12:0/65541177 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c8107780 con 0x7f04c81090c0 2024-03-23T22:51:26.638 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.630+0000 7f04c5e00700 1 --2- 172.21.15.12:0/65541177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04c81090c0 0x7f04c81094e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f04b80098a0 tx=0x7f04b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e43a06e0aa4e9aa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:26.638 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.630+0000 7f04c4a00700 1 -- 172.21.15.12:0/65541177 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f04b8004160 con 0x7f04c81090c0 2024-03-23T22:51:26.638 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.630+0000 7f04c4a00700 1 -- 172.21.15.12:0/65541177 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04b8004300 con 0x7f04c81090c0 2024-03-23T22:51:26.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.630+0000 7f04c4a00700 1 -- 172.21.15.12:0/65541177 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f04b8011b20 con 0x7f04c81090c0 2024-03-23T22:51:26.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.631+0000 7f04ccc00700 1 -- 172.21.15.12:0/65541177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04c81090c0 msgr2=0x7f04c81094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:26.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.631+0000 7f04ccc00700 1 --2- 172.21.15.12:0/65541177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04c81090c0 0x7f04c81094e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f04b80098a0 tx=0x7f04b8009bb0 comp rx=0 tx=0).stop 2024-03-23T22:51:26.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.631+0000 7f04ccc00700 1 -- 172.21.15.12:0/65541177 shutdown_connections 2024-03-23T22:51:26.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.631+0000 7f04ccc00700 1 --2- 172.21.15.12:0/65541177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04c81090c0 0x7f04c81094e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:26.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.631+0000 7f04ccc00700 1 -- 172.21.15.12:0/65541177 >> 172.21.15.12:0/65541177 conn(0x7f04c8103000 msgr2=0x7f04c8105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:26.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.631+0000 7f04ccc00700 1 -- 172.21.15.12:0/65541177 shutdown_connections 2024-03-23T22:51:26.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.631+0000 7f04ccc00700 1 -- 172.21.15.12:0/65541177 wait complete. 2024-03-23T22:51:26.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.632+0000 7f04ccc00700 1 Processor -- start 2024-03-23T22:51:26.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.632+0000 7f04ccc00700 1 -- start start 2024-03-23T22:51:26.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.633+0000 7f04ccc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04c81090c0 0x7f04c807cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:26.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.633+0000 7f04ccc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c810bf90 con 0x7f04c81090c0 2024-03-23T22:51:26.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.633+0000 7f04c5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04c81090c0 0x7f04c807cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:26.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.633+0000 7f04c5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04c81090c0 0x7f04c807cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42626/0 (socket says 172.21.15.12:42626) 2024-03-23T22:51:26.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.633+0000 7f04c5e00700 1 -- 172.21.15.12:0/732844941 learned_addr learned my addr 172.21.15.12:0/732844941 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:26.640 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.633+0000 7f04c5e00700 1 -- 172.21.15.12:0/732844941 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c807d090 con 0x7f04c81090c0 2024-03-23T22:51:26.640 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.634+0000 7f04c5e00700 1 --2- 172.21.15.12:0/732844941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04c81090c0 0x7f04c807cb50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f04b8000c00 tx=0x7f04b8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:26.640 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.634+0000 7f04bea00700 1 -- 172.21.15.12:0/732844941 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f04b8003b60 con 0x7f04c81090c0 2024-03-23T22:51:26.640 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.634+0000 7f04bea00700 1 -- 172.21.15.12:0/732844941 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04b801a910 con 0x7f04c81090c0 2024-03-23T22:51:26.640 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.634+0000 7f04ccc00700 1 -- 172.21.15.12:0/732844941 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c807b280 con 0x7f04c81090c0 2024-03-23T22:51:26.640 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.634+0000 7f04bea00700 1 -- 172.21.15.12:0/732844941 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f04b8024930 con 0x7f04c81090c0 2024-03-23T22:51:26.640 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.635+0000 7f04ccc00700 1 -- 172.21.15.12:0/732844941 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c807b740 con 0x7f04c81090c0 2024-03-23T22:51:26.640 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.635+0000 7f04bea00700 1 -- 172.21.15.12:0/732844941 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f04b8022020 con 0x7f04c81090c0 2024-03-23T22:51:26.640 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.636+0000 7f04bea00700 1 --2- 172.21.15.12:0/732844941 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f04ac038f70 0x7f04ac03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:26.640 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.636+0000 7f04bea00700 1 -- 172.21.15.12:0/732844941 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f04b8052660 con 0x7f04c81090c0 2024-03-23T22:51:26.640 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.636+0000 7f04ccc00700 1 -- 172.21.15.12:0/732844941 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c80637c0 con 0x7f04c81090c0 2024-03-23T22:51:26.640 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.636+0000 7f04c5400700 1 --2- 172.21.15.12:0/732844941 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f04ac038f70 0x7f04ac03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:26.640 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.637+0000 7f04c5400700 1 --2- 172.21.15.12:0/732844941 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f04ac038f70 0x7f04ac03b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f04b4006fd0 tx=0x7f04b4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:26.640 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.642+0000 7f04bea00700 1 -- 172.21.15.12:0/732844941 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f04b8028d50 con 0x7f04c81090c0 2024-03-23T22:51:26.641 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.875+0000 7f04ccc00700 1 -- 172.21.15.12:0/732844941 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f04c807be90 con 0x7f04ac038f70 2024-03-23T22:51:26.641 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.981+0000 7f04bea00700 1 -- 172.21.15.12:0/732844941 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f04c807be90 con 0x7f04ac038f70 2024-03-23T22:51:26.641 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.984+0000 7f04ccc00700 1 -- 172.21.15.12:0/732844941 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f04ac038f70 msgr2=0x7f04ac03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:26.641 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.984+0000 7f04ccc00700 1 --2- 172.21.15.12:0/732844941 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f04ac038f70 0x7f04ac03b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f04b4006fd0 tx=0x7f04b4006e40 comp rx=0 tx=0).stop 2024-03-23T22:51:26.641 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.984+0000 7f04ccc00700 1 -- 172.21.15.12:0/732844941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04c81090c0 msgr2=0x7f04c807cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:26.641 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.984+0000 7f04ccc00700 1 --2- 172.21.15.12:0/732844941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04c81090c0 0x7f04c807cb50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f04b8000c00 tx=0x7f04b8011c10 comp rx=0 tx=0).stop 2024-03-23T22:51:26.641 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.984+0000 7f04ccc00700 1 -- 172.21.15.12:0/732844941 shutdown_connections 2024-03-23T22:51:26.641 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.984+0000 7f04ccc00700 1 --2- 172.21.15.12:0/732844941 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f04ac038f70 0x7f04ac03b430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:26.641 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.984+0000 7f04ccc00700 1 --2- 172.21.15.12:0/732844941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04c81090c0 0x7f04c807cb50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:26.641 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.984+0000 7f04ccc00700 1 -- 172.21.15.12:0/732844941 >> 172.21.15.12:0/732844941 conn(0x7f04c8103000 msgr2=0x7f04c8103b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:26.641 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.985+0000 7f04ccc00700 1 -- 172.21.15.12:0/732844941 shutdown_connections 2024-03-23T22:51:26.641 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:25.985+0000 7f04ccc00700 1 -- 172.21.15.12:0/732844941 wait complete. 2024-03-23T22:51:26.641 INFO:teuthology.orchestra.run.smithi012.stdout:Creating initial admin user... 2024-03-23T22:51:27.491 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:27 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:51:28.566 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$ImHd2aMZJMta8LJlE.vhW.M/sOzMBxHTRJni93q7JHkadMJB8Mwl.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711234287, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-23T22:51:28.566 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.206+0000 7f0e7e200700 1 Processor -- start 2024-03-23T22:51:28.566 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.206+0000 7f0e7e200700 1 -- start start 2024-03-23T22:51:28.566 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.206+0000 7f0e7e200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e781090c0 0x7f0e781094e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:28.566 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.206+0000 7f0e7e200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e78107600 con 0x7f0e781090c0 2024-03-23T22:51:28.566 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.207+0000 7f0e77400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e781090c0 0x7f0e781094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:28.566 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.207+0000 7f0e77400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e781090c0 0x7f0e781094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42630/0 (socket says 172.21.15.12:42630) 2024-03-23T22:51:28.566 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.207+0000 7f0e77400700 1 -- 172.21.15.12:0/2955909982 learned_addr learned my addr 172.21.15.12:0/2955909982 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:28.566 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.207+0000 7f0e77400700 1 -- 172.21.15.12:0/2955909982 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e78107780 con 0x7f0e781090c0 2024-03-23T22:51:28.566 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.208+0000 7f0e77400700 1 --2- 172.21.15.12:0/2955909982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e781090c0 0x7f0e781094e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0e680098a0 tx=0x7f0e68009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e48e16fcc6f4428 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:28.567 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.208+0000 7f0e76000700 1 -- 172.21.15.12:0/2955909982 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0e68004160 con 0x7f0e781090c0 2024-03-23T22:51:28.567 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.208+0000 7f0e76000700 1 -- 172.21.15.12:0/2955909982 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e68004300 con 0x7f0e781090c0 2024-03-23T22:51:28.567 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.208+0000 7f0e76000700 1 -- 172.21.15.12:0/2955909982 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0e68011b20 con 0x7f0e781090c0 2024-03-23T22:51:28.567 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.209+0000 7f0e7e200700 1 -- 172.21.15.12:0/2955909982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e781090c0 msgr2=0x7f0e781094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:28.567 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.209+0000 7f0e7e200700 1 --2- 172.21.15.12:0/2955909982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e781090c0 0x7f0e781094e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0e680098a0 tx=0x7f0e68009bb0 comp rx=0 tx=0).stop 2024-03-23T22:51:28.567 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.209+0000 7f0e7e200700 1 -- 172.21.15.12:0/2955909982 shutdown_connections 2024-03-23T22:51:28.567 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.209+0000 7f0e7e200700 1 --2- 172.21.15.12:0/2955909982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e781090c0 0x7f0e781094e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:28.567 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.209+0000 7f0e7e200700 1 -- 172.21.15.12:0/2955909982 >> 172.21.15.12:0/2955909982 conn(0x7f0e78103000 msgr2=0x7f0e78105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:28.567 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.209+0000 7f0e7e200700 1 -- 172.21.15.12:0/2955909982 shutdown_connections 2024-03-23T22:51:28.567 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.209+0000 7f0e7e200700 1 -- 172.21.15.12:0/2955909982 wait complete. 2024-03-23T22:51:28.567 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.210+0000 7f0e7e200700 1 Processor -- start 2024-03-23T22:51:28.567 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.210+0000 7f0e7e200700 1 -- start start 2024-03-23T22:51:28.567 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.211+0000 7f0e7e200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e781090c0 0x7f0e7819bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:28.567 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.211+0000 7f0e7e200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7810bf90 con 0x7f0e781090c0 2024-03-23T22:51:28.567 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.211+0000 7f0e77400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e781090c0 0x7f0e7819bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:28.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.211+0000 7f0e77400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e781090c0 0x7f0e7819bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42634/0 (socket says 172.21.15.12:42634) 2024-03-23T22:51:28.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.211+0000 7f0e77400700 1 -- 172.21.15.12:0/2232067909 learned_addr learned my addr 172.21.15.12:0/2232067909 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:28.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.211+0000 7f0e77400700 1 -- 172.21.15.12:0/2232067909 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e7819c000 con 0x7f0e781090c0 2024-03-23T22:51:28.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.212+0000 7f0e77400700 1 --2- 172.21.15.12:0/2232067909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e781090c0 0x7f0e7819bac0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0e68000c00 tx=0x7f0e68011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:28.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.212+0000 7f0e6fe00700 1 -- 172.21.15.12:0/2232067909 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0e68003b60 con 0x7f0e781090c0 2024-03-23T22:51:28.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.212+0000 7f0e6fe00700 1 -- 172.21.15.12:0/2232067909 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e6801a910 con 0x7f0e781090c0 2024-03-23T22:51:28.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.212+0000 7f0e7e200700 1 -- 172.21.15.12:0/2232067909 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e7819c290 con 0x7f0e781090c0 2024-03-23T22:51:28.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.212+0000 7f0e6fe00700 1 -- 172.21.15.12:0/2232067909 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0e68024930 con 0x7f0e781090c0 2024-03-23T22:51:28.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.212+0000 7f0e7e200700 1 -- 172.21.15.12:0/2232067909 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e7819c6f0 con 0x7f0e781090c0 2024-03-23T22:51:28.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.214+0000 7f0e6fe00700 1 -- 172.21.15.12:0/2232067909 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f0e68022020 con 0x7f0e781090c0 2024-03-23T22:51:28.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.214+0000 7f0e7e200700 1 -- 172.21.15.12:0/2232067909 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e780637c0 con 0x7f0e781090c0 2024-03-23T22:51:28.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.214+0000 7f0e6fe00700 1 --2- 172.21.15.12:0/2232067909 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0e5c038fd0 0x7f0e5c03b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:28.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.214+0000 7f0e6fe00700 1 -- 172.21.15.12:0/2232067909 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0e6804d0e0 con 0x7f0e781090c0 2024-03-23T22:51:28.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.215+0000 7f0e76a00700 1 --2- 172.21.15.12:0/2232067909 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0e5c038fd0 0x7f0e5c03b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:28.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.215+0000 7f0e76a00700 1 --2- 172.21.15.12:0/2232067909 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0e5c038fd0 0x7f0e5c03b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0e64006fd0 tx=0x7f0e64006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:28.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.220+0000 7f0e6fe00700 1 -- 172.21.15.12:0/2232067909 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0e6802ddb0 con 0x7f0e781090c0 2024-03-23T22:51:28.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.492+0000 7f0e7e200700 1 -- 172.21.15.12:0/2232067909 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f0e7819f100 con 0x7f0e5c038fd0 2024-03-23T22:51:28.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.759+0000 7f0e6fe00700 1 -- 172.21.15.12:0/2232067909 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f0e7819f100 con 0x7f0e5c038fd0 2024-03-23T22:51:28.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.763+0000 7f0e7e200700 1 -- 172.21.15.12:0/2232067909 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0e5c038fd0 msgr2=0x7f0e5c03b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:28.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.763+0000 7f0e7e200700 1 --2- 172.21.15.12:0/2232067909 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0e5c038fd0 0x7f0e5c03b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0e64006fd0 tx=0x7f0e64006e40 comp rx=0 tx=0).stop 2024-03-23T22:51:28.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.763+0000 7f0e7e200700 1 -- 172.21.15.12:0/2232067909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e781090c0 msgr2=0x7f0e7819bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:28.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.763+0000 7f0e7e200700 1 --2- 172.21.15.12:0/2232067909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e781090c0 0x7f0e7819bac0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0e68000c00 tx=0x7f0e68011c10 comp rx=0 tx=0).stop 2024-03-23T22:51:28.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.763+0000 7f0e7e200700 1 -- 172.21.15.12:0/2232067909 shutdown_connections 2024-03-23T22:51:28.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.763+0000 7f0e7e200700 1 --2- 172.21.15.12:0/2232067909 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0e5c038fd0 0x7f0e5c03b490 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:28.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.763+0000 7f0e7e200700 1 --2- 172.21.15.12:0/2232067909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e781090c0 0x7f0e7819bac0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:28.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.763+0000 7f0e7e200700 1 -- 172.21.15.12:0/2232067909 >> 172.21.15.12:0/2232067909 conn(0x7f0e78103000 msgr2=0x7f0e78103b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:28.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.764+0000 7f0e7e200700 1 -- 172.21.15.12:0/2232067909 shutdown_connections 2024-03-23T22:51:28.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:27.764+0000 7f0e7e200700 1 -- 172.21.15.12:0/2232067909 wait complete. 2024-03-23T22:51:28.569 INFO:teuthology.orchestra.run.smithi012.stdout:Fetching dashboard port number... 2024-03-23T22:51:28.909 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:28 smithi012 ceph-mon[31695]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:51:28.909 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:28 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:51:28.909 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:28 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:51:30.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:29 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1121310414' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-23T22:51:30.707 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 8443 2024-03-23T22:51:30.708 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.198+0000 7fccd3c00700 1 Processor -- start 2024-03-23T22:51:30.708 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.199+0000 7fccd3c00700 1 -- start start 2024-03-23T22:51:30.709 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.199+0000 7fccd3c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcccc109c60 0x7fcccc10a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:30.709 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.199+0000 7fccd3c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc10a650 con 0x7fcccc109c60 2024-03-23T22:51:30.709 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.199+0000 7fccd1200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcccc109c60 0x7fcccc10a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:30.709 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.200+0000 7fccd1200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcccc109c60 0x7fcccc10a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42646/0 (socket says 172.21.15.12:42646) 2024-03-23T22:51:30.709 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.200+0000 7fccd1200700 1 -- 172.21.15.12:0/761724252 learned_addr learned my addr 172.21.15.12:0/761724252 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:30.709 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.200+0000 7fccd1200700 1 -- 172.21.15.12:0/761724252 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc10adc0 con 0x7fcccc109c60 2024-03-23T22:51:30.718 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.200+0000 7fccd1200700 1 --2- 172.21.15.12:0/761724252 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcccc109c60 0x7fcccc10a080 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fccc00098a0 tx=0x7fccc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c6d468322ad11eb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:30.718 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.201+0000 7fcccb400700 1 -- 172.21.15.12:0/761724252 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fccc0004160 con 0x7fcccc109c60 2024-03-23T22:51:30.718 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.201+0000 7fcccb400700 1 -- 172.21.15.12:0/761724252 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fccc0004300 con 0x7fcccc109c60 2024-03-23T22:51:30.718 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.201+0000 7fcccb400700 1 -- 172.21.15.12:0/761724252 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fccc0011b20 con 0x7fcccc109c60 2024-03-23T22:51:30.719 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.201+0000 7fccd3c00700 1 -- 172.21.15.12:0/761724252 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcccc109c60 msgr2=0x7fcccc10a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:30.719 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.201+0000 7fccd3c00700 1 --2- 172.21.15.12:0/761724252 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcccc109c60 0x7fcccc10a080 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fccc00098a0 tx=0x7fccc0009bb0 comp rx=0 tx=0).stop 2024-03-23T22:51:30.719 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.202+0000 7fccd3c00700 1 -- 172.21.15.12:0/761724252 shutdown_connections 2024-03-23T22:51:30.719 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.202+0000 7fccd3c00700 1 --2- 172.21.15.12:0/761724252 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcccc109c60 0x7fcccc10a080 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:30.719 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.202+0000 7fccd3c00700 1 -- 172.21.15.12:0/761724252 >> 172.21.15.12:0/761724252 conn(0x7fcccc1054a0 msgr2=0x7fcccc107890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:30.719 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.202+0000 7fccd3c00700 1 -- 172.21.15.12:0/761724252 shutdown_connections 2024-03-23T22:51:30.719 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.202+0000 7fccd3c00700 1 -- 172.21.15.12:0/761724252 wait complete. 2024-03-23T22:51:30.719 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.203+0000 7fccd3c00700 1 Processor -- start 2024-03-23T22:51:30.719 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.204+0000 7fccd3c00700 1 -- start start 2024-03-23T22:51:30.719 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.204+0000 7fccd3c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcccc109c60 0x7fcccc19daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:30.719 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.204+0000 7fccd3c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc10e170 con 0x7fcccc109c60 2024-03-23T22:51:30.719 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.205+0000 7fccd1200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcccc109c60 0x7fcccc19daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:30.719 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.205+0000 7fccd1200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcccc109c60 0x7fcccc19daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42656/0 (socket says 172.21.15.12:42656) 2024-03-23T22:51:30.719 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.205+0000 7fccd1200700 1 -- 172.21.15.12:0/1121310414 learned_addr learned my addr 172.21.15.12:0/1121310414 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:30.720 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.205+0000 7fccd1200700 1 -- 172.21.15.12:0/1121310414 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc19e030 con 0x7fcccc109c60 2024-03-23T22:51:30.720 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.206+0000 7fccd1200700 1 --2- 172.21.15.12:0/1121310414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcccc109c60 0x7fcccc19daf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fccc00117e0 tx=0x7fccc0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:30.720 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.206+0000 7fccc9600700 1 -- 172.21.15.12:0/1121310414 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fccc0003bf0 con 0x7fcccc109c60 2024-03-23T22:51:30.720 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.206+0000 7fccc9600700 1 -- 172.21.15.12:0/1121310414 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fccc001a910 con 0x7fcccc109c60 2024-03-23T22:51:30.720 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.206+0000 7fccc9600700 1 -- 172.21.15.12:0/1121310414 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fccc0024970 con 0x7fcccc109c60 2024-03-23T22:51:30.720 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.206+0000 7fccd3c00700 1 -- 172.21.15.12:0/1121310414 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcccc19e2c0 con 0x7fcccc109c60 2024-03-23T22:51:30.720 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.206+0000 7fccd3c00700 1 -- 172.21.15.12:0/1121310414 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcccc19e720 con 0x7fcccc109c60 2024-03-23T22:51:30.720 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.207+0000 7fccd3c00700 1 -- 172.21.15.12:0/1121310414 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcccc0637c0 con 0x7fcccc109c60 2024-03-23T22:51:30.720 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.207+0000 7fccc9600700 1 -- 172.21.15.12:0/1121310414 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fccc0022020 con 0x7fcccc109c60 2024-03-23T22:51:30.720 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.208+0000 7fccc9600700 1 --2- 172.21.15.12:0/1121310414 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fccb4038fd0 0x7fccb403b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:30.720 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.208+0000 7fccc9600700 1 -- 172.21.15.12:0/1121310414 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fccc004d330 con 0x7fcccc109c60 2024-03-23T22:51:30.721 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.208+0000 7fcccbe00700 1 --2- 172.21.15.12:0/1121310414 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fccb4038fd0 0x7fccb403b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:30.721 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.209+0000 7fcccbe00700 1 --2- 172.21.15.12:0/1121310414 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fccb4038fd0 0x7fccb403b490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fccb8006fd0 tx=0x7fccb8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:30.721 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.213+0000 7fccc9600700 1 -- 172.21.15.12:0/1121310414 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fccc001ab70 con 0x7fcccc109c60 2024-03-23T22:51:30.721 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.449+0000 7fccd3c00700 1 -- 172.21.15.12:0/1121310414 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fcccc1a11f0 con 0x7fcccc109c60 2024-03-23T22:51:30.721 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.450+0000 7fccc9600700 1 -- 172.21.15.12:0/1121310414 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fccc002d3f0 con 0x7fcccc109c60 2024-03-23T22:51:30.721 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.453+0000 7fccd3c00700 1 -- 172.21.15.12:0/1121310414 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fccb4038fd0 msgr2=0x7fccb403b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:30.721 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.453+0000 7fccd3c00700 1 --2- 172.21.15.12:0/1121310414 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fccb4038fd0 0x7fccb403b490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fccb8006fd0 tx=0x7fccb8006e40 comp rx=0 tx=0).stop 2024-03-23T22:51:30.721 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.453+0000 7fccd3c00700 1 -- 172.21.15.12:0/1121310414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcccc109c60 msgr2=0x7fcccc19daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:30.721 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.453+0000 7fccd3c00700 1 --2- 172.21.15.12:0/1121310414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcccc109c60 0x7fcccc19daf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fccc00117e0 tx=0x7fccc0011ba0 comp rx=0 tx=0).stop 2024-03-23T22:51:30.721 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.453+0000 7fccd3c00700 1 -- 172.21.15.12:0/1121310414 shutdown_connections 2024-03-23T22:51:30.722 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.453+0000 7fccd3c00700 1 --2- 172.21.15.12:0/1121310414 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fccb4038fd0 0x7fccb403b490 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:30.722 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.453+0000 7fccd3c00700 1 --2- 172.21.15.12:0/1121310414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcccc109c60 0x7fcccc19daf0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:30.722 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.453+0000 7fccd3c00700 1 -- 172.21.15.12:0/1121310414 >> 172.21.15.12:0/1121310414 conn(0x7fcccc1054a0 msgr2=0x7fcccc106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:30.722 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.454+0000 7fccd3c00700 1 -- 172.21.15.12:0/1121310414 shutdown_connections 2024-03-23T22:51:30.722 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:29.454+0000 7fccd3c00700 1 -- 172.21.15.12:0/1121310414 wait complete. 2024-03-23T22:51:30.748 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-23T22:51:30.748 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout disabled 2024-03-23T22:51:30.772 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-23T22:51:30.773 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-03-23T22:51:30.773 INFO:teuthology.orchestra.run.smithi012.stdout:firewalld.service is not enabled 2024-03-23T22:51:30.774 INFO:teuthology.orchestra.run.smithi012.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-23T22:51:30.781 INFO:teuthology.orchestra.run.smithi012.stdout:Ceph Dashboard is now available at: 2024-03-23T22:51:30.782 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:51:30.782 INFO:teuthology.orchestra.run.smithi012.stdout: URL: https://smithi012.front.sepia.ceph.com:8443/ 2024-03-23T22:51:30.782 INFO:teuthology.orchestra.run.smithi012.stdout: User: admin 2024-03-23T22:51:30.782 INFO:teuthology.orchestra.run.smithi012.stdout: Password: s1j9kpclq5 2024-03-23T22:51:30.782 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:51:30.782 INFO:teuthology.orchestra.run.smithi012.stdout:Saving cluster configuration to /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/config directory 2024-03-23T22:51:30.785 INFO:teuthology.orchestra.run.smithi012.stdout:Enabling autotune for osd_memory_target 2024-03-23T22:51:30.979 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:30 smithi012 ceph-mon[31695]: mgrmap e13: x(active, since 6s) 2024-03-23T22:51:32.473 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.543+0000 7f7846800700 1 Processor -- start 2024-03-23T22:51:32.473 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.544+0000 7f7846800700 1 -- start start 2024-03-23T22:51:32.473 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.544+0000 7f7846800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f784010e0b0 0x7f784010e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:32.473 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.544+0000 7f7846800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784010eaa0 con 0x7f784010e0b0 2024-03-23T22:51:32.473 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.544+0000 7f7845400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f784010e0b0 0x7f784010e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:32.475 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.544+0000 7f7845400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f784010e0b0 0x7f784010e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42668/0 (socket says 172.21.15.12:42668) 2024-03-23T22:51:32.475 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.545+0000 7f7845400700 1 -- 172.21.15.12:0/369186565 learned_addr learned my addr 172.21.15.12:0/369186565 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:32.475 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.545+0000 7f7845400700 1 -- 172.21.15.12:0/369186565 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f784010eff0 con 0x7f784010e0b0 2024-03-23T22:51:32.475 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.545+0000 7f7845400700 1 --2- 172.21.15.12:0/369186565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f784010e0b0 0x7f784010e4d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f78340098a0 tx=0x7f7834009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db870e7d117798cd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:32.475 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.546+0000 7f783fe00700 1 -- 172.21.15.12:0/369186565 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7834004160 con 0x7f784010e0b0 2024-03-23T22:51:32.475 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.546+0000 7f783fe00700 1 -- 172.21.15.12:0/369186565 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7834004300 con 0x7f784010e0b0 2024-03-23T22:51:32.475 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.546+0000 7f783fe00700 1 -- 172.21.15.12:0/369186565 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7834011b20 con 0x7f784010e0b0 2024-03-23T22:51:32.476 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.546+0000 7f7846800700 1 -- 172.21.15.12:0/369186565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f784010e0b0 msgr2=0x7f784010e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:32.476 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.546+0000 7f7846800700 1 --2- 172.21.15.12:0/369186565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f784010e0b0 0x7f784010e4d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f78340098a0 tx=0x7f7834009bb0 comp rx=0 tx=0).stop 2024-03-23T22:51:32.476 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.547+0000 7f7846800700 1 -- 172.21.15.12:0/369186565 shutdown_connections 2024-03-23T22:51:32.476 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.547+0000 7f7846800700 1 --2- 172.21.15.12:0/369186565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f784010e0b0 0x7f784010e4d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:32.476 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.547+0000 7f7846800700 1 -- 172.21.15.12:0/369186565 >> 172.21.15.12:0/369186565 conn(0x7f784007d010 msgr2=0x7f784007f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:32.476 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.547+0000 7f7846800700 1 -- 172.21.15.12:0/369186565 shutdown_connections 2024-03-23T22:51:32.476 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.547+0000 7f7846800700 1 -- 172.21.15.12:0/369186565 wait complete. 2024-03-23T22:51:32.476 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.548+0000 7f7846800700 1 Processor -- start 2024-03-23T22:51:32.476 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.548+0000 7f7846800700 1 -- start start 2024-03-23T22:51:32.479 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.548+0000 7f7846800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f784010e0b0 0x7f78401a1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:32.479 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.548+0000 7f7846800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7840112220 con 0x7f784010e0b0 2024-03-23T22:51:32.479 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.549+0000 7f7845400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f784010e0b0 0x7f78401a1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:32.479 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.549+0000 7f7845400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f784010e0b0 0x7f78401a1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42674/0 (socket says 172.21.15.12:42674) 2024-03-23T22:51:32.480 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.549+0000 7f7845400700 1 -- 172.21.15.12:0/4001343249 learned_addr learned my addr 172.21.15.12:0/4001343249 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:32.480 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.549+0000 7f7845400700 1 -- 172.21.15.12:0/4001343249 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78401a23f0 con 0x7f784010e0b0 2024-03-23T22:51:32.480 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.549+0000 7f7845400700 1 --2- 172.21.15.12:0/4001343249 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f784010e0b0 0x7f78401a1eb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f78340117e0 tx=0x7f7834011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:32.480 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.550+0000 7f783e000700 1 -- 172.21.15.12:0/4001343249 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7834003ba0 con 0x7f784010e0b0 2024-03-23T22:51:32.480 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.550+0000 7f783e000700 1 -- 172.21.15.12:0/4001343249 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f783401a910 con 0x7f784010e0b0 2024-03-23T22:51:32.480 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.550+0000 7f7846800700 1 -- 172.21.15.12:0/4001343249 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78401a2680 con 0x7f784010e0b0 2024-03-23T22:51:32.480 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.550+0000 7f783e000700 1 -- 172.21.15.12:0/4001343249 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f78340248f0 con 0x7f784010e0b0 2024-03-23T22:51:32.480 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.550+0000 7f7846800700 1 -- 172.21.15.12:0/4001343249 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78401a2ae0 con 0x7f784010e0b0 2024-03-23T22:51:32.480 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.551+0000 7f7846800700 1 -- 172.21.15.12:0/4001343249 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78401123a0 con 0x7f784010e0b0 2024-03-23T22:51:32.480 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.551+0000 7f783e000700 1 -- 172.21.15.12:0/4001343249 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f7834022020 con 0x7f784010e0b0 2024-03-23T22:51:32.480 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.551+0000 7f783e000700 1 --2- 172.21.15.12:0/4001343249 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7828039010 0x7f782803b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:32.480 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.551+0000 7f783e000700 1 -- 172.21.15.12:0/4001343249 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f783404d9c0 con 0x7f784010e0b0 2024-03-23T22:51:32.480 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.552+0000 7f7844a00700 1 --2- 172.21.15.12:0/4001343249 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7828039010 0x7f782803b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:32.481 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.552+0000 7f7844a00700 1 --2- 172.21.15.12:0/4001343249 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7828039010 0x7f782803b4d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f782c006fd0 tx=0x7f782c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:32.481 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.557+0000 7f783e000700 1 -- 172.21.15.12:0/4001343249 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f783401ab70 con 0x7f784010e0b0 2024-03-23T22:51:32.481 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.782+0000 7f7846800700 1 -- 172.21.15.12:0/4001343249 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f78401a5490 con 0x7f784010e0b0 2024-03-23T22:51:32.481 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.782+0000 7f783e000700 1 -- 172.21.15.12:0/4001343249 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f7834010380 con 0x7f784010e0b0 2024-03-23T22:51:32.481 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.785+0000 7f7846800700 1 -- 172.21.15.12:0/4001343249 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7828039010 msgr2=0x7f782803b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:32.481 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.785+0000 7f7846800700 1 --2- 172.21.15.12:0/4001343249 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7828039010 0x7f782803b4d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f782c006fd0 tx=0x7f782c006e40 comp rx=0 tx=0).stop 2024-03-23T22:51:32.481 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.785+0000 7f7846800700 1 -- 172.21.15.12:0/4001343249 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f784010e0b0 msgr2=0x7f78401a1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:32.481 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.785+0000 7f7846800700 1 --2- 172.21.15.12:0/4001343249 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f784010e0b0 0x7f78401a1eb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f78340117e0 tx=0x7f7834011ba0 comp rx=0 tx=0).stop 2024-03-23T22:51:32.481 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.786+0000 7f7846800700 1 -- 172.21.15.12:0/4001343249 shutdown_connections 2024-03-23T22:51:32.481 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.786+0000 7f7846800700 1 --2- 172.21.15.12:0/4001343249 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7828039010 0x7f782803b4d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:32.481 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.786+0000 7f7846800700 1 --2- 172.21.15.12:0/4001343249 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f784010e0b0 0x7f78401a1eb0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:32.482 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.786+0000 7f7846800700 1 -- 172.21.15.12:0/4001343249 >> 172.21.15.12:0/4001343249 conn(0x7f784007d010 msgr2=0x7f784007dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:32.482 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.786+0000 7f7846800700 1 -- 172.21.15.12:0/4001343249 shutdown_connections 2024-03-23T22:51:32.482 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:31.786+0000 7f7846800700 1 -- 172.21.15.12:0/4001343249 wait complete. 2024-03-23T22:51:34.116 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.125+0000 7f4231000700 1 Processor -- start 2024-03-23T22:51:34.116 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.126+0000 7f4231000700 1 -- start start 2024-03-23T22:51:34.116 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.126+0000 7f4231000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f422c1084e0 0x7f422c10a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:34.116 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.126+0000 7f4231000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c07ae10 con 0x7f422c1084e0 2024-03-23T22:51:34.116 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.126+0000 7f4229e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f422c1084e0 0x7f422c10a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:34.116 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.126+0000 7f4229e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f422c1084e0 0x7f422c10a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42684/0 (socket says 172.21.15.12:42684) 2024-03-23T22:51:34.118 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.127+0000 7f4229e00700 1 -- 172.21.15.12:0/811753740 learned_addr learned my addr 172.21.15.12:0/811753740 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:34.118 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.127+0000 7f4229e00700 1 -- 172.21.15.12:0/811753740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f422c10ae10 con 0x7f422c1084e0 2024-03-23T22:51:34.118 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.127+0000 7f4229e00700 1 --2- 172.21.15.12:0/811753740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f422c1084e0 0x7f422c10a8d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f421c0098a0 tx=0x7f421c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59dbd615784fdf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:34.118 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.128+0000 7f4228a00700 1 -- 172.21.15.12:0/811753740 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f421c004160 con 0x7f422c1084e0 2024-03-23T22:51:34.118 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.128+0000 7f4228a00700 1 -- 172.21.15.12:0/811753740 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f421c004300 con 0x7f422c1084e0 2024-03-23T22:51:34.118 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.128+0000 7f4228a00700 1 -- 172.21.15.12:0/811753740 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f421c011b20 con 0x7f422c1084e0 2024-03-23T22:51:34.118 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.128+0000 7f4231000700 1 -- 172.21.15.12:0/811753740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f422c1084e0 msgr2=0x7f422c10a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:34.118 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.128+0000 7f4231000700 1 --2- 172.21.15.12:0/811753740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f422c1084e0 0x7f422c10a8d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f421c0098a0 tx=0x7f421c009bb0 comp rx=0 tx=0).stop 2024-03-23T22:51:34.118 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.129+0000 7f4231000700 1 -- 172.21.15.12:0/811753740 shutdown_connections 2024-03-23T22:51:34.118 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.129+0000 7f4231000700 1 --2- 172.21.15.12:0/811753740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f422c1084e0 0x7f422c10a8d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:34.118 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.129+0000 7f4231000700 1 -- 172.21.15.12:0/811753740 >> 172.21.15.12:0/811753740 conn(0x7f422c101f70 msgr2=0x7f422c104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:34.119 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.129+0000 7f4231000700 1 -- 172.21.15.12:0/811753740 shutdown_connections 2024-03-23T22:51:34.119 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.129+0000 7f4231000700 1 -- 172.21.15.12:0/811753740 wait complete. 2024-03-23T22:51:34.119 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.130+0000 7f4231000700 1 Processor -- start 2024-03-23T22:51:34.120 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.130+0000 7f4231000700 1 -- start start 2024-03-23T22:51:34.120 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.130+0000 7f4231000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f422c1084e0 0x7f422c19dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:34.120 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.130+0000 7f4231000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c10be10 con 0x7f422c1084e0 2024-03-23T22:51:34.120 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.131+0000 7f4229e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f422c1084e0 0x7f422c19dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:34.120 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.131+0000 7f4229e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f422c1084e0 0x7f422c19dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42688/0 (socket says 172.21.15.12:42688) 2024-03-23T22:51:34.120 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.131+0000 7f4229e00700 1 -- 172.21.15.12:0/3173329343 learned_addr learned my addr 172.21.15.12:0/3173329343 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:34.120 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.131+0000 7f4229e00700 1 -- 172.21.15.12:0/3173329343 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f422c19e020 con 0x7f422c1084e0 2024-03-23T22:51:34.120 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.131+0000 7f4229e00700 1 --2- 172.21.15.12:0/3173329343 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f422c1084e0 0x7f422c19dae0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f421c0117e0 tx=0x7f421c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:34.120 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.132+0000 7f4222a00700 1 -- 172.21.15.12:0/3173329343 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f421c003ba0 con 0x7f422c1084e0 2024-03-23T22:51:34.121 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.132+0000 7f4222a00700 1 -- 172.21.15.12:0/3173329343 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f421c01a910 con 0x7f422c1084e0 2024-03-23T22:51:34.121 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.132+0000 7f4222a00700 1 -- 172.21.15.12:0/3173329343 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f421c0248f0 con 0x7f422c1084e0 2024-03-23T22:51:34.122 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.132+0000 7f4231000700 1 -- 172.21.15.12:0/3173329343 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f422c19e2b0 con 0x7f422c1084e0 2024-03-23T22:51:34.122 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.132+0000 7f4231000700 1 -- 172.21.15.12:0/3173329343 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f422c19e710 con 0x7f422c1084e0 2024-03-23T22:51:34.122 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.133+0000 7f4222a00700 1 -- 172.21.15.12:0/3173329343 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f421c022020 con 0x7f422c1084e0 2024-03-23T22:51:34.122 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.133+0000 7f4231000700 1 -- 172.21.15.12:0/3173329343 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f422c0637c0 con 0x7f422c1084e0 2024-03-23T22:51:34.122 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.134+0000 7f4222a00700 1 --2- 172.21.15.12:0/3173329343 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4210039070 0x7f421003b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:34.122 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.134+0000 7f4222a00700 1 -- 172.21.15.12:0/3173329343 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f421c04d4a0 con 0x7f422c1084e0 2024-03-23T22:51:34.122 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.134+0000 7f4229400700 1 --2- 172.21.15.12:0/3173329343 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4210039070 0x7f421003b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:34.123 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.135+0000 7f4229400700 1 --2- 172.21.15.12:0/3173329343 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4210039070 0x7f421003b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4218006fd0 tx=0x7f4218006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:34.123 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.139+0000 7f4222a00700 1 -- 172.21.15.12:0/3173329343 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f421c01ab70 con 0x7f422c1084e0 2024-03-23T22:51:34.123 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.486+0000 7f4231000700 1 -- 172.21.15.12:0/3173329343 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f422c1a11a0 con 0x7f422c1084e0 2024-03-23T22:51:34.123 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.492+0000 7f4222a00700 1 -- 172.21.15.12:0/3173329343 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f421c010380 con 0x7f422c1084e0 2024-03-23T22:51:34.123 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.495+0000 7f4231000700 1 -- 172.21.15.12:0/3173329343 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4210039070 msgr2=0x7f421003b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:34.123 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.495+0000 7f4231000700 1 --2- 172.21.15.12:0/3173329343 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4210039070 0x7f421003b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4218006fd0 tx=0x7f4218006e40 comp rx=0 tx=0).stop 2024-03-23T22:51:34.124 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.495+0000 7f4231000700 1 -- 172.21.15.12:0/3173329343 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f422c1084e0 msgr2=0x7f422c19dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:34.124 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.495+0000 7f4231000700 1 --2- 172.21.15.12:0/3173329343 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f422c1084e0 0x7f422c19dae0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f421c0117e0 tx=0x7f421c011ba0 comp rx=0 tx=0).stop 2024-03-23T22:51:34.124 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.495+0000 7f4231000700 1 -- 172.21.15.12:0/3173329343 shutdown_connections 2024-03-23T22:51:34.124 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.495+0000 7f4231000700 1 --2- 172.21.15.12:0/3173329343 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4210039070 0x7f421003b530 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:34.125 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.495+0000 7f4231000700 1 --2- 172.21.15.12:0/3173329343 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f422c1084e0 0x7f422c19dae0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:34.125 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.495+0000 7f4231000700 1 -- 172.21.15.12:0/3173329343 >> 172.21.15.12:0/3173329343 conn(0x7f422c101f70 msgr2=0x7f422c102950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:34.125 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.495+0000 7f4231000700 1 -- 172.21.15.12:0/3173329343 shutdown_connections 2024-03-23T22:51:34.125 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-23T22:51:33.495+0000 7f4231000700 1 -- 172.21.15.12:0/3173329343 wait complete. 2024-03-23T22:51:34.125 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-23T22:51:34.125 INFO:teuthology.orchestra.run.smithi012.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-23T22:51:34.125 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:51:34.125 INFO:teuthology.orchestra.run.smithi012.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-23T22:51:34.125 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:51:34.125 INFO:teuthology.orchestra.run.smithi012.stdout:Or, if you are only running a single cluster on this host: 2024-03-23T22:51:34.125 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:51:34.125 INFO:teuthology.orchestra.run.smithi012.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-23T22:51:34.125 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:51:34.125 INFO:teuthology.orchestra.run.smithi012.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-23T22:51:34.125 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:51:34.125 INFO:teuthology.orchestra.run.smithi012.stdout: ceph telemetry on 2024-03-23T22:51:34.126 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:51:34.126 INFO:teuthology.orchestra.run.smithi012.stdout:For more information see: 2024-03-23T22:51:34.126 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:51:34.126 INFO:teuthology.orchestra.run.smithi012.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-23T22:51:34.126 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:51:34.126 INFO:teuthology.orchestra.run.smithi012.stdout:Bootstrap complete. 2024-03-23T22:51:34.252 INFO:tasks.cephadm:Fetching config... 2024-03-23T22:51:34.252 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-23T22:51:34.253 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-23T22:51:34.309 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-23T22:51:34.310 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-23T22:51:34.310 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-23T22:51:34.350 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-23T22:51:34.350 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-23T22:51:34.350 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-23T22:51:34.484 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-23T22:51:34.484 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-23T22:51:34.484 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-23T22:51:34.535 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-23T22:51:34.535 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZoSciPO4yj+WOea6DWeuGsUiKT+SYUamLfbGi6V5fM+ONThvLcCn8zIjddVT8nvyygakcwGtZAvk2jNd/uyLSg8wQkh63BlvBbzw9WvvBaSl7ooRNRve509Crs4CkXIRPPPaDVqP5zrzB6A5el4RPSpOrOnZ5PZC8iXw0MGFNy4E6uHDxd3CniXqgOeJ0zVq4wEcy8LNAtHxanBgvISMDp0afBis1UIM3Li7fgEhZRoNhmIMtkG1MxwvDkyDfPimHbSgP0CN2wtbD6ogi5bPJGOzP/LG/5SA8eVWbPWHKuNOEiu0ndLNyDiMedACKQEdNNawx1UkNTpJVG0kdU3dlmMx0YBgkP1OmvSyOC6Dp4Smh8NAgT4WTryt0QSlKKe5ALKgvoGRvIacXxzdd0qmv357JYp4pXGLtz4HlJttWwhVQuPG8UNgugkMFYFVQCyyf8vLj/s6lC48/1mS+e+RQrUOc+8CXQr0QhIGefZmy+tUIePAA/OZBaH+HMxv0Xfk= ceph-90f679ae-e967-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-23T22:51:34.703 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:34 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3173329343' entity='client.admin' 2024-03-23T22:51:34.712 INFO:teuthology.orchestra.run.smithi012.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZoSciPO4yj+WOea6DWeuGsUiKT+SYUamLfbGi6V5fM+ONThvLcCn8zIjddVT8nvyygakcwGtZAvk2jNd/uyLSg8wQkh63BlvBbzw9WvvBaSl7ooRNRve509Crs4CkXIRPPPaDVqP5zrzB6A5el4RPSpOrOnZ5PZC8iXw0MGFNy4E6uHDxd3CniXqgOeJ0zVq4wEcy8LNAtHxanBgvISMDp0afBis1UIM3Li7fgEhZRoNhmIMtkG1MxwvDkyDfPimHbSgP0CN2wtbD6ogi5bPJGOzP/LG/5SA8eVWbPWHKuNOEiu0ndLNyDiMedACKQEdNNawx1UkNTpJVG0kdU3dlmMx0YBgkP1OmvSyOC6Dp4Smh8NAgT4WTryt0QSlKKe5ALKgvoGRvIacXxzdd0qmv357JYp4pXGLtz4HlJttWwhVQuPG8UNgugkMFYFVQCyyf8vLj/s6lC48/1mS+e+RQrUOc+8CXQr0QhIGefZmy+tUIePAA/OZBaH+HMxv0Xfk= ceph-90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T22:51:34.780 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZoSciPO4yj+WOea6DWeuGsUiKT+SYUamLfbGi6V5fM+ONThvLcCn8zIjddVT8nvyygakcwGtZAvk2jNd/uyLSg8wQkh63BlvBbzw9WvvBaSl7ooRNRve509Crs4CkXIRPPPaDVqP5zrzB6A5el4RPSpOrOnZ5PZC8iXw0MGFNy4E6uHDxd3CniXqgOeJ0zVq4wEcy8LNAtHxanBgvISMDp0afBis1UIM3Li7fgEhZRoNhmIMtkG1MxwvDkyDfPimHbSgP0CN2wtbD6ogi5bPJGOzP/LG/5SA8eVWbPWHKuNOEiu0ndLNyDiMedACKQEdNNawx1UkNTpJVG0kdU3dlmMx0YBgkP1OmvSyOC6Dp4Smh8NAgT4WTryt0QSlKKe5ALKgvoGRvIacXxzdd0qmv357JYp4pXGLtz4HlJttWwhVQuPG8UNgugkMFYFVQCyyf8vLj/s6lC48/1mS+e+RQrUOc+8CXQr0QhIGefZmy+tUIePAA/OZBaH+HMxv0Xfk= ceph-90f679ae-e967-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-23T22:51:34.828 INFO:teuthology.orchestra.run.smithi173.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZoSciPO4yj+WOea6DWeuGsUiKT+SYUamLfbGi6V5fM+ONThvLcCn8zIjddVT8nvyygakcwGtZAvk2jNd/uyLSg8wQkh63BlvBbzw9WvvBaSl7ooRNRve509Crs4CkXIRPPPaDVqP5zrzB6A5el4RPSpOrOnZ5PZC8iXw0MGFNy4E6uHDxd3CniXqgOeJ0zVq4wEcy8LNAtHxanBgvISMDp0afBis1UIM3Li7fgEhZRoNhmIMtkG1MxwvDkyDfPimHbSgP0CN2wtbD6ogi5bPJGOzP/LG/5SA8eVWbPWHKuNOEiu0ndLNyDiMedACKQEdNNawx1UkNTpJVG0kdU3dlmMx0YBgkP1OmvSyOC6Dp4Smh8NAgT4WTryt0QSlKKe5ALKgvoGRvIacXxzdd0qmv357JYp4pXGLtz4HlJttWwhVQuPG8UNgugkMFYFVQCyyf8vLj/s6lC48/1mS+e+RQrUOc+8CXQr0QhIGefZmy+tUIePAA/OZBaH+HMxv0Xfk= ceph-90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T22:51:34.843 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZoSciPO4yj+WOea6DWeuGsUiKT+SYUamLfbGi6V5fM+ONThvLcCn8zIjddVT8nvyygakcwGtZAvk2jNd/uyLSg8wQkh63BlvBbzw9WvvBaSl7ooRNRve509Crs4CkXIRPPPaDVqP5zrzB6A5el4RPSpOrOnZ5PZC8iXw0MGFNy4E6uHDxd3CniXqgOeJ0zVq4wEcy8LNAtHxanBgvISMDp0afBis1UIM3Li7fgEhZRoNhmIMtkG1MxwvDkyDfPimHbSgP0CN2wtbD6ogi5bPJGOzP/LG/5SA8eVWbPWHKuNOEiu0ndLNyDiMedACKQEdNNawx1UkNTpJVG0kdU3dlmMx0YBgkP1OmvSyOC6Dp4Smh8NAgT4WTryt0QSlKKe5ALKgvoGRvIacXxzdd0qmv357JYp4pXGLtz4HlJttWwhVQuPG8UNgugkMFYFVQCyyf8vLj/s6lC48/1mS+e+RQrUOc+8CXQr0QhIGefZmy+tUIePAA/OZBaH+HMxv0Xfk= ceph-90f679ae-e967-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-23T22:51:34.890 INFO:teuthology.orchestra.run.smithi203.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZoSciPO4yj+WOea6DWeuGsUiKT+SYUamLfbGi6V5fM+ONThvLcCn8zIjddVT8nvyygakcwGtZAvk2jNd/uyLSg8wQkh63BlvBbzw9WvvBaSl7ooRNRve509Crs4CkXIRPPPaDVqP5zrzB6A5el4RPSpOrOnZ5PZC8iXw0MGFNy4E6uHDxd3CniXqgOeJ0zVq4wEcy8LNAtHxanBgvISMDp0afBis1UIM3Li7fgEhZRoNhmIMtkG1MxwvDkyDfPimHbSgP0CN2wtbD6ogi5bPJGOzP/LG/5SA8eVWbPWHKuNOEiu0ndLNyDiMedACKQEdNNawx1UkNTpJVG0kdU3dlmMx0YBgkP1OmvSyOC6Dp4Smh8NAgT4WTryt0QSlKKe5ALKgvoGRvIacXxzdd0qmv357JYp4pXGLtz4HlJttWwhVQuPG8UNgugkMFYFVQCyyf8vLj/s6lC48/1mS+e+RQrUOc+8CXQr0QhIGefZmy+tUIePAA/OZBaH+HMxv0Xfk= ceph-90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T22:51:34.906 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-23T22:51:35.334 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:51:38.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.153+0000 7f2c98400700 1 -- 172.21.15.12:0/3941036038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c90101be0 msgr2=0x7f2c90102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:38.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.153+0000 7f2c98400700 1 --2- 172.21.15.12:0/3941036038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c90101be0 0x7f2c90102000 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2c8400b0a0 tx=0x7f2c8400b3b0 comp rx=0 tx=0).stop 2024-03-23T22:51:38.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.153+0000 7f2c98400700 1 -- 172.21.15.12:0/3941036038 shutdown_connections 2024-03-23T22:51:38.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.153+0000 7f2c98400700 1 --2- 172.21.15.12:0/3941036038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c90101be0 0x7f2c90102000 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:38.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.153+0000 7f2c98400700 1 -- 172.21.15.12:0/3941036038 >> 172.21.15.12:0/3941036038 conn(0x7f2c900fd180 msgr2=0x7f2c900ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:38.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.153+0000 7f2c98400700 1 -- 172.21.15.12:0/3941036038 shutdown_connections 2024-03-23T22:51:38.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.153+0000 7f2c98400700 1 -- 172.21.15.12:0/3941036038 wait complete. 2024-03-23T22:51:38.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.155+0000 7f2c98400700 1 Processor -- start 2024-03-23T22:51:38.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.155+0000 7f2c98400700 1 -- start start 2024-03-23T22:51:38.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.155+0000 7f2c98400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c90101be0 0x7f2c90199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:38.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.155+0000 7f2c98400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c90107c70 con 0x7f2c90101be0 2024-03-23T22:51:38.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.155+0000 7f2c95a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c90101be0 0x7f2c90199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:38.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.155+0000 7f2c95a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c90101be0 0x7f2c90199440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34692/0 (socket says 172.21.15.12:34692) 2024-03-23T22:51:38.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.155+0000 7f2c95a00700 1 -- 172.21.15.12:0/2069247966 learned_addr learned my addr 172.21.15.12:0/2069247966 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:38.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.156+0000 7f2c95a00700 1 -- 172.21.15.12:0/2069247966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c90199980 con 0x7f2c90101be0 2024-03-23T22:51:38.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.156+0000 7f2c95a00700 1 --2- 172.21.15.12:0/2069247966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c90101be0 0x7f2c90199440 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2c84007890 tx=0x7f2c84009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:38.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.157+0000 7f2c8e000700 1 -- 172.21.15.12:0/2069247966 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c84017040 con 0x7f2c90101be0 2024-03-23T22:51:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.157+0000 7f2c8e000700 1 -- 172.21.15.12:0/2069247966 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c8400bd50 con 0x7f2c90101be0 2024-03-23T22:51:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.157+0000 7f2c98400700 1 -- 172.21.15.12:0/2069247966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c90199c10 con 0x7f2c90101be0 2024-03-23T22:51:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.157+0000 7f2c8e000700 1 -- 172.21.15.12:0/2069247966 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c8401d7a0 con 0x7f2c90101be0 2024-03-23T22:51:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.157+0000 7f2c98400700 1 -- 172.21.15.12:0/2069247966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c9019c900 con 0x7f2c90101be0 2024-03-23T22:51:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.158+0000 7f2c98400700 1 -- 172.21.15.12:0/2069247966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c901083f0 con 0x7f2c90101be0 2024-03-23T22:51:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.158+0000 7f2c8e000700 1 -- 172.21.15.12:0/2069247966 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f2c84024020 con 0x7f2c90101be0 2024-03-23T22:51:38.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.159+0000 7f2c8e000700 1 --2- 172.21.15.12:0/2069247966 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2c78039010 0x7f2c7803b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:38.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.159+0000 7f2c8e000700 1 -- 172.21.15.12:0/2069247966 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2c84022020 con 0x7f2c90101be0 2024-03-23T22:51:38.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.159+0000 7f2c95000700 1 --2- 172.21.15.12:0/2069247966 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2c78039010 0x7f2c7803b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:38.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.160+0000 7f2c95000700 1 --2- 172.21.15.12:0/2069247966 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2c78039010 0x7f2c7803b4d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c006fd0 tx=0x7f2c7c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:38.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.169+0000 7f2c8e000700 1 -- 172.21.15.12:0/2069247966 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2c8402b020 con 0x7f2c90101be0 2024-03-23T22:51:38.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.402+0000 7f2c98400700 1 -- 172.21.15.12:0/2069247966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f2c9002db40 con 0x7f2c90101be0 2024-03-23T22:51:38.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.408+0000 7f2c8e000700 1 -- 172.21.15.12:0/2069247966 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f2c84007e20 con 0x7f2c90101be0 2024-03-23T22:51:38.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.412+0000 7f2c98400700 1 -- 172.21.15.12:0/2069247966 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2c78039010 msgr2=0x7f2c7803b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:38.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.412+0000 7f2c98400700 1 --2- 172.21.15.12:0/2069247966 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2c78039010 0x7f2c7803b4d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c006fd0 tx=0x7f2c7c006e40 comp rx=0 tx=0).stop 2024-03-23T22:51:38.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.412+0000 7f2c98400700 1 -- 172.21.15.12:0/2069247966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c90101be0 msgr2=0x7f2c90199440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:38.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.412+0000 7f2c98400700 1 --2- 172.21.15.12:0/2069247966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c90101be0 0x7f2c90199440 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2c84007890 tx=0x7f2c84009790 comp rx=0 tx=0).stop 2024-03-23T22:51:38.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.413+0000 7f2c98400700 1 -- 172.21.15.12:0/2069247966 shutdown_connections 2024-03-23T22:51:38.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.413+0000 7f2c98400700 1 --2- 172.21.15.12:0/2069247966 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2c78039010 0x7f2c7803b4d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:38.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.413+0000 7f2c98400700 1 --2- 172.21.15.12:0/2069247966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c90101be0 0x7f2c90199440 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:38.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.413+0000 7f2c98400700 1 -- 172.21.15.12:0/2069247966 >> 172.21.15.12:0/2069247966 conn(0x7f2c900fd180 msgr2=0x7f2c900ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:38.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.413+0000 7f2c98400700 1 -- 172.21.15.12:0/2069247966 shutdown_connections 2024-03-23T22:51:38.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:38.413+0000 7f2c98400700 1 -- 172.21.15.12:0/2069247966 wait complete. 2024-03-23T22:51:39.266 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-23T22:51:39.267 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-23T22:51:39.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:39 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2069247966' entity='client.admin' 2024-03-23T22:51:39.702 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:51:42.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.486+0000 7f26a1800700 1 -- 172.21.15.12:0/3518449889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f269c0761c0 msgr2=0x7f269c074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:42.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.486+0000 7f26a1800700 1 --2- 172.21.15.12:0/3518449889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f269c0761c0 0x7f269c074620 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f268c009910 tx=0x7f268c009c20 comp rx=0 tx=0).stop 2024-03-23T22:51:42.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.486+0000 7f26a1800700 1 -- 172.21.15.12:0/3518449889 shutdown_connections 2024-03-23T22:51:42.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.486+0000 7f26a1800700 1 --2- 172.21.15.12:0/3518449889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f269c0761c0 0x7f269c074620 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:42.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.486+0000 7f26a1800700 1 -- 172.21.15.12:0/3518449889 >> 172.21.15.12:0/3518449889 conn(0x7f269c0fd180 msgr2=0x7f269c0ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:42.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.486+0000 7f26a1800700 1 -- 172.21.15.12:0/3518449889 shutdown_connections 2024-03-23T22:51:42.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.486+0000 7f26a1800700 1 -- 172.21.15.12:0/3518449889 wait complete. 2024-03-23T22:51:42.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.487+0000 7f26a1800700 1 Processor -- start 2024-03-23T22:51:42.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.487+0000 7f26a1800700 1 -- start start 2024-03-23T22:51:42.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.488+0000 7f26a1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f269c0761c0 0x7f269c19d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:42.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.488+0000 7f26a1800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269c10e340 con 0x7f269c0761c0 2024-03-23T22:51:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.488+0000 7f269aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f269c0761c0 0x7f269c19d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.488+0000 7f269aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f269c0761c0 0x7f269c19d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34712/0 (socket says 172.21.15.12:34712) 2024-03-23T22:51:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.488+0000 7f269aa00700 1 -- 172.21.15.12:0/4209242583 learned_addr learned my addr 172.21.15.12:0/4209242583 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.488+0000 7f269aa00700 1 -- 172.21.15.12:0/4209242583 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f269c19de50 con 0x7f269c0761c0 2024-03-23T22:51:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.489+0000 7f269aa00700 1 --2- 172.21.15.12:0/4209242583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f269c0761c0 0x7f269c19d910 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f268c005850 tx=0x7f268c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.489+0000 7f2693400700 1 -- 172.21.15.12:0/4209242583 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f268c026070 con 0x7f269c0761c0 2024-03-23T22:51:42.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.489+0000 7f2693400700 1 -- 172.21.15.12:0/4209242583 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f268c0044a0 con 0x7f269c0761c0 2024-03-23T22:51:42.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.489+0000 7f2693400700 1 -- 172.21.15.12:0/4209242583 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f268c003ab0 con 0x7f269c0761c0 2024-03-23T22:51:42.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.489+0000 7f26a1800700 1 -- 172.21.15.12:0/4209242583 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f269c19e0e0 con 0x7f269c0761c0 2024-03-23T22:51:42.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.489+0000 7f26a1800700 1 -- 172.21.15.12:0/4209242583 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f269c1a0dd0 con 0x7f269c0761c0 2024-03-23T22:51:42.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.490+0000 7f2693400700 1 -- 172.21.15.12:0/4209242583 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f268c029070 con 0x7f269c0761c0 2024-03-23T22:51:42.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.490+0000 7f26a1800700 1 -- 172.21.15.12:0/4209242583 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f269c10e9f0 con 0x7f269c0761c0 2024-03-23T22:51:42.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.491+0000 7f2693400700 1 --2- 172.21.15.12:0/4209242583 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2680039020 0x7f268003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:42.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.491+0000 7f2693400700 1 -- 172.21.15.12:0/4209242583 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f268c01e070 con 0x7f269c0761c0 2024-03-23T22:51:42.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.491+0000 7f269a000700 1 --2- 172.21.15.12:0/4209242583 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2680039020 0x7f268003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:42.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.492+0000 7f269a000700 1 --2- 172.21.15.12:0/4209242583 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2680039020 0x7f268003b4e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2688006fd0 tx=0x7f2688006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:42.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.497+0000 7f2693400700 1 -- 172.21.15.12:0/4209242583 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f268c013070 con 0x7f269c0761c0 2024-03-23T22:51:42.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.730+0000 7f26a1800700 1 -- 172.21.15.12:0/4209242583 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f269c02db40 con 0x7f2680039020 2024-03-23T22:51:42.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.739+0000 7f2693400700 1 -- 172.21.15.12:0/4209242583 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f269c02db40 con 0x7f2680039020 2024-03-23T22:51:42.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.742+0000 7f26a1800700 1 -- 172.21.15.12:0/4209242583 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2680039020 msgr2=0x7f268003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:42.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.742+0000 7f26a1800700 1 --2- 172.21.15.12:0/4209242583 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2680039020 0x7f268003b4e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2688006fd0 tx=0x7f2688006e40 comp rx=0 tx=0).stop 2024-03-23T22:51:42.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.742+0000 7f26a1800700 1 -- 172.21.15.12:0/4209242583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f269c0761c0 msgr2=0x7f269c19d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:42.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.742+0000 7f26a1800700 1 --2- 172.21.15.12:0/4209242583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f269c0761c0 0x7f269c19d910 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f268c005850 tx=0x7f268c005030 comp rx=0 tx=0).stop 2024-03-23T22:51:42.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.742+0000 7f26a1800700 1 -- 172.21.15.12:0/4209242583 shutdown_connections 2024-03-23T22:51:42.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.742+0000 7f26a1800700 1 --2- 172.21.15.12:0/4209242583 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2680039020 0x7f268003b4e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:42.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.742+0000 7f26a1800700 1 --2- 172.21.15.12:0/4209242583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f269c0761c0 0x7f269c19d910 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:42.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.742+0000 7f26a1800700 1 -- 172.21.15.12:0/4209242583 >> 172.21.15.12:0/4209242583 conn(0x7f269c0fd180 msgr2=0x7f269c0fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:42.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.743+0000 7f26a1800700 1 -- 172.21.15.12:0/4209242583 shutdown_connections 2024-03-23T22:51:42.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:42.743+0000 7f26a1800700 1 -- 172.21.15.12:0/4209242583 wait complete. 2024-03-23T22:51:43.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:42 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:51:43.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:42 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:51:43.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:42 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi012", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:51:43.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:42 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:51:43.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:42 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:51:43.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:42 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:51:43.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:42 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:51:43.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:42 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:51:43.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:42 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:51:43.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:42 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:51:43.498 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi173 2024-03-23T22:51:43.498 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T22:51:43.498 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.conf 2024-03-23T22:51:43.517 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T22:51:43.517 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-23T22:51:43.573 INFO:tasks.cephadm:Adding host smithi173 to orchestrator... 2024-03-23T22:51:43.573 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph orch host add smithi173 2024-03-23T22:51:44.005 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:51:44.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:43 smithi012 ceph-mon[31695]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:51:44.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:43 smithi012 ceph-mon[31695]: Updating smithi012:/etc/ceph/ceph.conf 2024-03-23T22:51:45.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:44 smithi012 ceph-mon[31695]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:51:45.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:44 smithi012 ceph-mon[31695]: Updating smithi012:/var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T22:51:45.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:45 smithi012 ceph-mon[31695]: Updating smithi012:/etc/ceph/ceph.client.admin.keyring 2024-03-23T22:51:45.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:45 smithi012 ceph-mon[31695]: Updating smithi012:/var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-23T22:51:45.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:45 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:51:45.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:45 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:51:45.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:45 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:51:46.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.077+0000 7fd8bae00700 1 -- 172.21.15.12:0/2411567243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8b4103ff0 msgr2=0x7fd8b41043d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:46.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.077+0000 7fd8bae00700 1 --2- 172.21.15.12:0/2411567243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8b4103ff0 0x7fd8b41043d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4009910 tx=0x7fd8a4009c20 comp rx=0 tx=0).stop 2024-03-23T22:51:46.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.078+0000 7fd8bae00700 1 -- 172.21.15.12:0/2411567243 shutdown_connections 2024-03-23T22:51:46.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.078+0000 7fd8bae00700 1 --2- 172.21.15.12:0/2411567243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8b4103ff0 0x7fd8b41043d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:46.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.078+0000 7fd8bae00700 1 -- 172.21.15.12:0/2411567243 >> 172.21.15.12:0/2411567243 conn(0x7fd8b40ff880 msgr2=0x7fd8b4101ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:46.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.078+0000 7fd8bae00700 1 -- 172.21.15.12:0/2411567243 shutdown_connections 2024-03-23T22:51:46.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.078+0000 7fd8bae00700 1 -- 172.21.15.12:0/2411567243 wait complete. 2024-03-23T22:51:46.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.079+0000 7fd8bae00700 1 Processor -- start 2024-03-23T22:51:46.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.079+0000 7fd8bae00700 1 -- start start 2024-03-23T22:51:46.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.079+0000 7fd8bae00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8b4103ff0 0x7fd8b419f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:46.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.079+0000 7fd8bae00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b410e3c0 con 0x7fd8b4103ff0 2024-03-23T22:51:46.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.080+0000 7fd8b3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8b4103ff0 0x7fd8b419f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:46.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.080+0000 7fd8b3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8b4103ff0 0x7fd8b419f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35458/0 (socket says 172.21.15.12:35458) 2024-03-23T22:51:46.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.080+0000 7fd8b3e00700 1 -- 172.21.15.12:0/709409301 learned_addr learned my addr 172.21.15.12:0/709409301 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:46.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.080+0000 7fd8b3e00700 1 -- 172.21.15.12:0/709409301 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b419fb20 con 0x7fd8b4103ff0 2024-03-23T22:51:46.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.081+0000 7fd8b3e00700 1 --2- 172.21.15.12:0/709409301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8b4103ff0 0x7fd8b419f5e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4006010 tx=0x7fd8a4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:46.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.081+0000 7fd8b1600700 1 -- 172.21.15.12:0/709409301 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd8a4026070 con 0x7fd8b4103ff0 2024-03-23T22:51:46.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.081+0000 7fd8b1600700 1 -- 172.21.15.12:0/709409301 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8a400bbd0 con 0x7fd8b4103ff0 2024-03-23T22:51:46.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.081+0000 7fd8bae00700 1 -- 172.21.15.12:0/709409301 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b419fe10 con 0x7fd8b4103ff0 2024-03-23T22:51:46.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.081+0000 7fd8b1600700 1 -- 172.21.15.12:0/709409301 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd8a4003910 con 0x7fd8b4103ff0 2024-03-23T22:51:46.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.081+0000 7fd8bae00700 1 -- 172.21.15.12:0/709409301 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b4199880 con 0x7fd8b4103ff0 2024-03-23T22:51:46.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.082+0000 7fd8bae00700 1 -- 172.21.15.12:0/709409301 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b41054d0 con 0x7fd8b4103ff0 2024-03-23T22:51:46.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.082+0000 7fd8b1600700 1 -- 172.21.15.12:0/709409301 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fd8a4029070 con 0x7fd8b4103ff0 2024-03-23T22:51:46.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.083+0000 7fd8b1600700 1 --2- 172.21.15.12:0/709409301 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd898039020 0x7fd89803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:46.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.083+0000 7fd8b1600700 1 -- 172.21.15.12:0/709409301 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd8a401e070 con 0x7fd8b4103ff0 2024-03-23T22:51:46.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.083+0000 7fd8abe00700 1 --2- 172.21.15.12:0/709409301 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd898039020 0x7fd89803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:46.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.084+0000 7fd8abe00700 1 --2- 172.21.15.12:0/709409301 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd898039020 0x7fd89803b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd89c006fd0 tx=0x7fd89c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:46.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.088+0000 7fd8b1600700 1 -- 172.21.15.12:0/709409301 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd8a4013070 con 0x7fd8b4103ff0 2024-03-23T22:51:46.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:46.322+0000 7fd8bae00700 1 -- 172.21.15.12:0/709409301 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8b419abe0 con 0x7fd898039020 2024-03-23T22:51:47.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:46 smithi012 ceph-mon[31695]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:51:48.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:47 smithi012 ceph-mon[31695]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:51:49.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:48 smithi012 ceph-mon[31695]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:51:49.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:48 smithi012 ceph-mon[31695]: Deploying cephadm binary to smithi173 2024-03-23T22:51:51.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:51.278+0000 7fd8b1600700 1 -- 172.21.15.12:0/709409301 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fd8b419abe0 con 0x7fd898039020 2024-03-23T22:51:51.278 INFO:teuthology.orchestra.run.smithi012.stdout:Added host 'smithi173' with addr '172.21.15.173' 2024-03-23T22:51:51.279 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:50 smithi012 ceph-mon[31695]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:51:51.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:51.281+0000 7fd8bae00700 1 -- 172.21.15.12:0/709409301 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd898039020 msgr2=0x7fd89803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:51.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:51.281+0000 7fd8bae00700 1 --2- 172.21.15.12:0/709409301 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd898039020 0x7fd89803b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd89c006fd0 tx=0x7fd89c006e40 comp rx=0 tx=0).stop 2024-03-23T22:51:51.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:51.281+0000 7fd8bae00700 1 -- 172.21.15.12:0/709409301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8b4103ff0 msgr2=0x7fd8b419f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:51.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:51.281+0000 7fd8bae00700 1 --2- 172.21.15.12:0/709409301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8b4103ff0 0x7fd8b419f5e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4006010 tx=0x7fd8a4005030 comp rx=0 tx=0).stop 2024-03-23T22:51:51.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:51.281+0000 7fd8bae00700 1 -- 172.21.15.12:0/709409301 shutdown_connections 2024-03-23T22:51:51.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:51.281+0000 7fd8bae00700 1 --2- 172.21.15.12:0/709409301 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd898039020 0x7fd89803b4e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:51.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:51.281+0000 7fd8bae00700 1 --2- 172.21.15.12:0/709409301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8b4103ff0 0x7fd8b419f5e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:51.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:51.281+0000 7fd8bae00700 1 -- 172.21.15.12:0/709409301 >> 172.21.15.12:0/709409301 conn(0x7fd8b40ff880 msgr2=0x7fd8b41059c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:51.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:51.282+0000 7fd8bae00700 1 -- 172.21.15.12:0/709409301 shutdown_connections 2024-03-23T22:51:51.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:51.282+0000 7fd8bae00700 1 -- 172.21.15.12:0/709409301 wait complete. 2024-03-23T22:51:52.017 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-23T22:51:52.473 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:51:52.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:52 smithi012 ceph-mon[31695]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:51:52.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:52 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:51:52.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:52 smithi012 ceph-mon[31695]: Added host smithi173 2024-03-23T22:51:52.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:52 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:51:52.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:52 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:51:53.637 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:53 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:51:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.478+0000 7f3a54a00700 1 -- 172.21.15.12:0/1830330684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a50101de0 msgr2=0x7f3a501021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.478+0000 7f3a54a00700 1 --2- 172.21.15.12:0/1830330684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a50101de0 0x7f3a501021c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3a40009910 tx=0x7f3a40009c20 comp rx=0 tx=0).stop 2024-03-23T22:51:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.478+0000 7f3a54a00700 1 -- 172.21.15.12:0/1830330684 shutdown_connections 2024-03-23T22:51:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.478+0000 7f3a54a00700 1 --2- 172.21.15.12:0/1830330684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a50101de0 0x7f3a501021c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.478+0000 7f3a54a00700 1 -- 172.21.15.12:0/1830330684 >> 172.21.15.12:0/1830330684 conn(0x7f3a500fd690 msgr2=0x7f3a500ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:54.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.478+0000 7f3a54a00700 1 -- 172.21.15.12:0/1830330684 shutdown_connections 2024-03-23T22:51:54.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.478+0000 7f3a54a00700 1 -- 172.21.15.12:0/1830330684 wait complete. 2024-03-23T22:51:54.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.480+0000 7f3a54a00700 1 Processor -- start 2024-03-23T22:51:54.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.480+0000 7f3a54a00700 1 -- start start 2024-03-23T22:51:54.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.480+0000 7f3a54a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a50101de0 0x7f3a5019f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:54.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.480+0000 7f3a54a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5010c1b0 con 0x7f3a50101de0 2024-03-23T22:51:54.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.480+0000 7f3a4de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a50101de0 0x7f3a5019f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:54.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.481+0000 7f3a4de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a50101de0 0x7f3a5019f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40086/0 (socket says 172.21.15.12:40086) 2024-03-23T22:51:54.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.481+0000 7f3a4de00700 1 -- 172.21.15.12:0/1446638661 learned_addr learned my addr 172.21.15.12:0/1446638661 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:54.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.481+0000 7f3a4de00700 1 -- 172.21.15.12:0/1446638661 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a5019faf0 con 0x7f3a50101de0 2024-03-23T22:51:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.481+0000 7f3a4de00700 1 --2- 172.21.15.12:0/1446638661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a50101de0 0x7f3a5019f5b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3a40006010 tx=0x7f3a40005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.481+0000 7f3a46a00700 1 -- 172.21.15.12:0/1446638661 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3a40026070 con 0x7f3a50101de0 2024-03-23T22:51:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.481+0000 7f3a46a00700 1 -- 172.21.15.12:0/1446638661 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a4000bbd0 con 0x7f3a50101de0 2024-03-23T22:51:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.482+0000 7f3a54a00700 1 -- 172.21.15.12:0/1446638661 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a5019fde0 con 0x7f3a50101de0 2024-03-23T22:51:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.482+0000 7f3a46a00700 1 -- 172.21.15.12:0/1446638661 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3a40003910 con 0x7f3a50101de0 2024-03-23T22:51:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.482+0000 7f3a54a00700 1 -- 172.21.15.12:0/1446638661 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a50199850 con 0x7f3a50101de0 2024-03-23T22:51:54.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.483+0000 7f3a46a00700 1 -- 172.21.15.12:0/1446638661 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f3a40029070 con 0x7f3a50101de0 2024-03-23T22:51:54.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.483+0000 7f3a54a00700 1 -- 172.21.15.12:0/1446638661 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a501032c0 con 0x7f3a50101de0 2024-03-23T22:51:54.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.483+0000 7f3a46a00700 1 --2- 172.21.15.12:0/1446638661 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3a34039020 0x7f3a3403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:54.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.483+0000 7f3a46a00700 1 -- 172.21.15.12:0/1446638661 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3a4001e070 con 0x7f3a50101de0 2024-03-23T22:51:54.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.484+0000 7f3a4d400700 1 --2- 172.21.15.12:0/1446638661 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3a34039020 0x7f3a3403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:54.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.484+0000 7f3a4d400700 1 --2- 172.21.15.12:0/1446638661 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3a34039020 0x7f3a3403b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c006fd0 tx=0x7f3a3c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:54.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.489+0000 7f3a46a00700 1 -- 172.21.15.12:0/1446638661 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3a40013070 con 0x7f3a50101de0 2024-03-23T22:51:54.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.721+0000 7f3a54a00700 1 -- 172.21.15.12:0/1446638661 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3a5019abe0 con 0x7f3a34039020 2024-03-23T22:51:54.722 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:54 smithi012 ceph-mon[31695]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:51:54.722 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:54 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:51:54.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.724+0000 7f3a46a00700 1 -- 172.21.15.12:0/1446638661 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f3a5019abe0 con 0x7f3a34039020 2024-03-23T22:51:54.724 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:51:54.724 INFO:teuthology.orchestra.run.smithi012.stdout:[{"addr": "172.21.15.12", "hostname": "smithi012", "labels": [], "status": ""}, {"addr": "172.21.15.173", "hostname": "smithi173", "labels": [], "status": ""}] 2024-03-23T22:51:54.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.727+0000 7f3a54a00700 1 -- 172.21.15.12:0/1446638661 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3a34039020 msgr2=0x7f3a3403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:54.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.727+0000 7f3a54a00700 1 --2- 172.21.15.12:0/1446638661 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3a34039020 0x7f3a3403b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c006fd0 tx=0x7f3a3c006e40 comp rx=0 tx=0).stop 2024-03-23T22:51:54.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.727+0000 7f3a54a00700 1 -- 172.21.15.12:0/1446638661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a50101de0 msgr2=0x7f3a5019f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:54.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.727+0000 7f3a54a00700 1 --2- 172.21.15.12:0/1446638661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a50101de0 0x7f3a5019f5b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3a40006010 tx=0x7f3a40005030 comp rx=0 tx=0).stop 2024-03-23T22:51:54.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.727+0000 7f3a54a00700 1 -- 172.21.15.12:0/1446638661 shutdown_connections 2024-03-23T22:51:54.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.727+0000 7f3a54a00700 1 --2- 172.21.15.12:0/1446638661 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3a34039020 0x7f3a3403b4e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:54.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.727+0000 7f3a54a00700 1 --2- 172.21.15.12:0/1446638661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a50101de0 0x7f3a5019f5b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:54.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.727+0000 7f3a54a00700 1 -- 172.21.15.12:0/1446638661 >> 172.21.15.12:0/1446638661 conn(0x7f3a500fd690 msgr2=0x7f3a500743c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:54.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.727+0000 7f3a54a00700 1 -- 172.21.15.12:0/1446638661 shutdown_connections 2024-03-23T22:51:54.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:54.727+0000 7f3a54a00700 1 -- 172.21.15.12:0/1446638661 wait complete. 2024-03-23T22:51:55.442 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi203 2024-03-23T22:51:55.442 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-23T22:51:55.442 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/etc/ceph/ceph.conf 2024-03-23T22:51:55.461 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-23T22:51:55.461 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-23T22:51:55.518 INFO:tasks.cephadm:Adding host smithi203 to orchestrator... 2024-03-23T22:51:55.519 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph orch host add smithi203 2024-03-23T22:51:55.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:55 smithi012 ceph-mon[31695]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T22:51:55.965 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:51:56.671 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:56 smithi012 ceph-mon[31695]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:51:57.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.956+0000 7fa4ae600700 1 -- 172.21.15.12:0/1855862166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4a8101dc0 msgr2=0x7fa4a81021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:51:57.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.956+0000 7fa4ae600700 1 --2- 172.21.15.12:0/1855862166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4a8101dc0 0x7fa4a81021a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa498009910 tx=0x7fa498009c20 comp rx=0 tx=0).stop 2024-03-23T22:51:57.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.956+0000 7fa4ae600700 1 -- 172.21.15.12:0/1855862166 shutdown_connections 2024-03-23T22:51:57.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.956+0000 7fa4ae600700 1 --2- 172.21.15.12:0/1855862166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4a8101dc0 0x7fa4a81021a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:51:57.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.956+0000 7fa4ae600700 1 -- 172.21.15.12:0/1855862166 >> 172.21.15.12:0/1855862166 conn(0x7fa4a80fd670 msgr2=0x7fa4a80ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:51:57.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.956+0000 7fa4ae600700 1 -- 172.21.15.12:0/1855862166 shutdown_connections 2024-03-23T22:51:57.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.956+0000 7fa4ae600700 1 -- 172.21.15.12:0/1855862166 wait complete. 2024-03-23T22:51:57.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.957+0000 7fa4ae600700 1 Processor -- start 2024-03-23T22:51:57.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.957+0000 7fa4ae600700 1 -- start start 2024-03-23T22:51:57.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.958+0000 7fa4ae600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4a8101dc0 0x7fa4a819f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:57.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.958+0000 7fa4ae600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a810c190 con 0x7fa4a8101dc0 2024-03-23T22:51:57.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.958+0000 7fa4a7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4a8101dc0 0x7fa4a819f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:57.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.958+0000 7fa4a7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4a8101dc0 0x7fa4a819f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40108/0 (socket says 172.21.15.12:40108) 2024-03-23T22:51:57.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.958+0000 7fa4a7400700 1 -- 172.21.15.12:0/2864633711 learned_addr learned my addr 172.21.15.12:0/2864633711 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:51:57.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.959+0000 7fa4a7400700 1 -- 172.21.15.12:0/2864633711 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a819fb40 con 0x7fa4a8101dc0 2024-03-23T22:51:57.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.959+0000 7fa4a7400700 1 --2- 172.21.15.12:0/2864633711 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4a8101dc0 0x7fa4a819f600 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa498006010 tx=0x7fa498005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:57.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.959+0000 7fa49fe00700 1 -- 172.21.15.12:0/2864633711 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa498026070 con 0x7fa4a8101dc0 2024-03-23T22:51:57.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.959+0000 7fa49fe00700 1 -- 172.21.15.12:0/2864633711 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa49800bbd0 con 0x7fa4a8101dc0 2024-03-23T22:51:57.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.959+0000 7fa49fe00700 1 -- 172.21.15.12:0/2864633711 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa498003910 con 0x7fa4a8101dc0 2024-03-23T22:51:57.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.959+0000 7fa4ae600700 1 -- 172.21.15.12:0/2864633711 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a819fe30 con 0x7fa4a8101dc0 2024-03-23T22:51:57.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.960+0000 7fa4ae600700 1 -- 172.21.15.12:0/2864633711 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a81998a0 con 0x7fa4a8101dc0 2024-03-23T22:51:57.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.961+0000 7fa49fe00700 1 -- 172.21.15.12:0/2864633711 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fa498029070 con 0x7fa4a8101dc0 2024-03-23T22:51:57.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.961+0000 7fa4ae600700 1 -- 172.21.15.12:0/2864633711 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a81032a0 con 0x7fa4a8101dc0 2024-03-23T22:51:57.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.961+0000 7fa49fe00700 1 --2- 172.21.15.12:0/2864633711 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa48c039020 0x7fa48c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:51:57.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.961+0000 7fa49fe00700 1 -- 172.21.15.12:0/2864633711 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa49801e070 con 0x7fa4a8101dc0 2024-03-23T22:51:57.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.962+0000 7fa4a6a00700 1 --2- 172.21.15.12:0/2864633711 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa48c039020 0x7fa48c03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:51:57.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.962+0000 7fa4a6a00700 1 --2- 172.21.15.12:0/2864633711 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa48c039020 0x7fa48c03b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa494006fd0 tx=0x7fa494006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:51:57.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:57.967+0000 7fa49fe00700 1 -- 172.21.15.12:0/2864633711 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa498013070 con 0x7fa4a8101dc0 2024-03-23T22:51:58.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:51:58.196+0000 7fa4ae600700 1 -- 172.21.15.12:0/2864633711 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4a819ac00 con 0x7fa48c039020 2024-03-23T22:51:58.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:58 smithi012 ceph-mon[31695]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:51:58.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:58 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:51:58.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:58 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:51:58.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:58 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:51:58.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:58 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:51:58.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:58 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:51:58.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:58 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:51:58.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:58 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:51:58.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:58 smithi012 ceph-mon[31695]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-23T22:51:58.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:58 smithi012 ceph-mon[31695]: Updating smithi173:/var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T22:51:58.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:58 smithi012 ceph-mon[31695]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-03-23T22:52:00.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:59 smithi012 ceph-mon[31695]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:52:00.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:59 smithi012 ceph-mon[31695]: Updating smithi173:/var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-23T22:52:00.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:59 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:00.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:59 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:00.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:59 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:00.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:51:59 smithi012 ceph-mon[31695]: Deploying cephadm binary to smithi203 2024-03-23T22:52:01.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:00 smithi012 ceph-mon[31695]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:03.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:02 smithi012 ceph-mon[31695]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:03.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:03.060+0000 7fa49fe00700 1 -- 172.21.15.12:0/2864633711 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fa4a819ac00 con 0x7fa48c039020 2024-03-23T22:52:03.062 INFO:teuthology.orchestra.run.smithi012.stdout:Added host 'smithi203' with addr '172.21.15.203' 2024-03-23T22:52:03.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:03.066+0000 7fa4ae600700 1 -- 172.21.15.12:0/2864633711 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa48c039020 msgr2=0x7fa48c03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:03.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:03.066+0000 7fa4ae600700 1 --2- 172.21.15.12:0/2864633711 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa48c039020 0x7fa48c03b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa494006fd0 tx=0x7fa494006e40 comp rx=0 tx=0).stop 2024-03-23T22:52:03.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:03.066+0000 7fa4ae600700 1 -- 172.21.15.12:0/2864633711 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4a8101dc0 msgr2=0x7fa4a819f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:03.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:03.066+0000 7fa4ae600700 1 --2- 172.21.15.12:0/2864633711 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4a8101dc0 0x7fa4a819f600 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa498006010 tx=0x7fa498005030 comp rx=0 tx=0).stop 2024-03-23T22:52:03.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:03.066+0000 7fa4ae600700 1 -- 172.21.15.12:0/2864633711 shutdown_connections 2024-03-23T22:52:03.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:03.066+0000 7fa4ae600700 1 --2- 172.21.15.12:0/2864633711 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa48c039020 0x7fa48c03b4e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:03.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:03.066+0000 7fa4ae600700 1 --2- 172.21.15.12:0/2864633711 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4a8101dc0 0x7fa4a819f600 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:03.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:03.066+0000 7fa4ae600700 1 -- 172.21.15.12:0/2864633711 >> 172.21.15.12:0/2864633711 conn(0x7fa4a80fd670 msgr2=0x7fa4a80743c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:52:03.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:03.066+0000 7fa4ae600700 1 -- 172.21.15.12:0/2864633711 shutdown_connections 2024-03-23T22:52:03.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:03.067+0000 7fa4ae600700 1 -- 172.21.15.12:0/2864633711 wait complete. 2024-03-23T22:52:03.765 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-23T22:52:04.231 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:52:04.379 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:04 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:04.379 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:04 smithi012 ceph-mon[31695]: Added host smithi203 2024-03-23T22:52:04.379 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:04 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:52:04.379 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:04 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:05.429 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:05 smithi012 ceph-mon[31695]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:05.430 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:05 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:06.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.374+0000 7f8041e00700 1 -- 172.21.15.12:0/1371150377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f803c100b10 msgr2=0x7f803c100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:06.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.374+0000 7f8041e00700 1 --2- 172.21.15.12:0/1371150377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f803c100b10 0x7f803c100ef0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f802c009910 tx=0x7f802c009c20 comp rx=0 tx=0).stop 2024-03-23T22:52:06.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.374+0000 7f8041e00700 1 -- 172.21.15.12:0/1371150377 shutdown_connections 2024-03-23T22:52:06.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.374+0000 7f8041e00700 1 --2- 172.21.15.12:0/1371150377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f803c100b10 0x7f803c100ef0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:06.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.374+0000 7f8041e00700 1 -- 172.21.15.12:0/1371150377 >> 172.21.15.12:0/1371150377 conn(0x7f803c0776d0 msgr2=0x7f803c0fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:52:06.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.374+0000 7f8041e00700 1 -- 172.21.15.12:0/1371150377 shutdown_connections 2024-03-23T22:52:06.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.374+0000 7f8041e00700 1 -- 172.21.15.12:0/1371150377 wait complete. 2024-03-23T22:52:06.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.375+0000 7f8041e00700 1 Processor -- start 2024-03-23T22:52:06.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.375+0000 7f8041e00700 1 -- start start 2024-03-23T22:52:06.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.376+0000 7f8041e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f803c100b10 0x7f803c1a3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:06.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.376+0000 7f8041e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f803c112810 con 0x7f803c100b10 2024-03-23T22:52:06.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.376+0000 7f803aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f803c100b10 0x7f803c1a3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:06.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.377+0000 7f803aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f803c100b10 0x7f803c1a3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58074/0 (socket says 172.21.15.12:58074) 2024-03-23T22:52:06.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.377+0000 7f803aa00700 1 -- 172.21.15.12:0/3367376957 learned_addr learned my addr 172.21.15.12:0/3367376957 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:52:06.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.377+0000 7f803aa00700 1 -- 172.21.15.12:0/3367376957 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f803c1a3f80 con 0x7f803c100b10 2024-03-23T22:52:06.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.377+0000 7f803aa00700 1 --2- 172.21.15.12:0/3367376957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f803c100b10 0x7f803c1a3a40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f802c006010 tx=0x7f802c005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:52:06.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.378+0000 7f8040a00700 1 -- 172.21.15.12:0/3367376957 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f802c026070 con 0x7f803c100b10 2024-03-23T22:52:06.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.378+0000 7f8041e00700 1 -- 172.21.15.12:0/3367376957 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f803c1a4210 con 0x7f803c100b10 2024-03-23T22:52:06.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.378+0000 7f8040a00700 1 -- 172.21.15.12:0/3367376957 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f802c00bc30 con 0x7f803c100b10 2024-03-23T22:52:06.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.378+0000 7f8040a00700 1 -- 172.21.15.12:0/3367376957 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f802c003ab0 con 0x7f803c100b10 2024-03-23T22:52:06.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.378+0000 7f8041e00700 1 -- 172.21.15.12:0/3367376957 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f803c19db10 con 0x7f803c100b10 2024-03-23T22:52:06.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.379+0000 7f8041e00700 1 -- 172.21.15.12:0/3367376957 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f803c069e60 con 0x7f803c100b10 2024-03-23T22:52:06.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.379+0000 7f8040a00700 1 -- 172.21.15.12:0/3367376957 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f802c029070 con 0x7f803c100b10 2024-03-23T22:52:06.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.379+0000 7f8040a00700 1 --2- 172.21.15.12:0/3367376957 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8020038fc0 0x7f802003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:06.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.379+0000 7f8040a00700 1 -- 172.21.15.12:0/3367376957 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f802c01e070 con 0x7f803c100b10 2024-03-23T22:52:06.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.380+0000 7f803a000700 1 --2- 172.21.15.12:0/3367376957 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8020038fc0 0x7f802003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:06.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.381+0000 7f803a000700 1 --2- 172.21.15.12:0/3367376957 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8020038fc0 0x7f802003b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8028006fd0 tx=0x7f8028006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:52:06.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.385+0000 7f8040a00700 1 -- 172.21.15.12:0/3367376957 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f802c050030 con 0x7f803c100b10 2024-03-23T22:52:06.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.622+0000 7f8041e00700 1 -- 172.21.15.12:0/3367376957 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f803c0637c0 con 0x7f8020038fc0 2024-03-23T22:52:06.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.623+0000 7f8040a00700 1 -- 172.21.15.12:0/3367376957 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7f803c0637c0 con 0x7f8020038fc0 2024-03-23T22:52:06.623 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:52:06.623 INFO:teuthology.orchestra.run.smithi012.stdout:[{"addr": "172.21.15.12", "hostname": "smithi012", "labels": [], "status": ""}, {"addr": "172.21.15.173", "hostname": "smithi173", "labels": [], "status": ""}, {"addr": "172.21.15.203", "hostname": "smithi203", "labels": [], "status": ""}] 2024-03-23T22:52:06.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.626+0000 7f8041e00700 1 -- 172.21.15.12:0/3367376957 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8020038fc0 msgr2=0x7f802003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:06.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.626+0000 7f8041e00700 1 --2- 172.21.15.12:0/3367376957 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8020038fc0 0x7f802003b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8028006fd0 tx=0x7f8028006e40 comp rx=0 tx=0).stop 2024-03-23T22:52:06.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.626+0000 7f8041e00700 1 -- 172.21.15.12:0/3367376957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f803c100b10 msgr2=0x7f803c1a3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:06.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.626+0000 7f8041e00700 1 --2- 172.21.15.12:0/3367376957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f803c100b10 0x7f803c1a3a40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f802c006010 tx=0x7f802c005050 comp rx=0 tx=0).stop 2024-03-23T22:52:06.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.626+0000 7f8041e00700 1 -- 172.21.15.12:0/3367376957 shutdown_connections 2024-03-23T22:52:06.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.626+0000 7f8041e00700 1 --2- 172.21.15.12:0/3367376957 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8020038fc0 0x7f802003b480 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:06.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.626+0000 7f8041e00700 1 --2- 172.21.15.12:0/3367376957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f803c100b10 0x7f803c1a3a40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:06.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.626+0000 7f8041e00700 1 -- 172.21.15.12:0/3367376957 >> 172.21.15.12:0/3367376957 conn(0x7f803c0776d0 msgr2=0x7f803c0fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:52:06.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.627+0000 7f8041e00700 1 -- 172.21.15.12:0/3367376957 shutdown_connections 2024-03-23T22:52:06.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:06.627+0000 7f8041e00700 1 -- 172.21.15.12:0/3367376957 wait complete. 2024-03-23T22:52:06.959 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:06 smithi012 ceph-mon[31695]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:06.959 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:06 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:07.436 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-23T22:52:07.437 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-23T22:52:07.899 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:52:08.144 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:07 smithi012 ceph-mon[31695]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T22:52:09.178 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:08 smithi012 ceph-mon[31695]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:10.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.022+0000 7f61ed800700 1 -- 172.21.15.12:0/4193399634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61e8101dd0 msgr2=0x7f61e81021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:10.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.022+0000 7f61ed800700 1 --2- 172.21.15.12:0/4193399634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61e8101dd0 0x7f61e81021b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f61d8009910 tx=0x7f61d8009c20 comp rx=0 tx=0).stop 2024-03-23T22:52:10.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.022+0000 7f61ed800700 1 -- 172.21.15.12:0/4193399634 shutdown_connections 2024-03-23T22:52:10.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.022+0000 7f61ed800700 1 --2- 172.21.15.12:0/4193399634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61e8101dd0 0x7f61e81021b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:10.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.022+0000 7f61ed800700 1 -- 172.21.15.12:0/4193399634 >> 172.21.15.12:0/4193399634 conn(0x7f61e80fd6a0 msgr2=0x7f61e80ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:52:10.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.022+0000 7f61ed800700 1 -- 172.21.15.12:0/4193399634 shutdown_connections 2024-03-23T22:52:10.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.022+0000 7f61ed800700 1 -- 172.21.15.12:0/4193399634 wait complete. 2024-03-23T22:52:10.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.023+0000 7f61ed800700 1 Processor -- start 2024-03-23T22:52:10.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.023+0000 7f61ed800700 1 -- start start 2024-03-23T22:52:10.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.024+0000 7f61ed800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61e8101dd0 0x7f61e819f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:10.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.024+0000 7f61ed800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e810c1a0 con 0x7f61e8101dd0 2024-03-23T22:52:10.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.024+0000 7f61e6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61e8101dd0 0x7f61e819f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:10.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.024+0000 7f61e6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61e8101dd0 0x7f61e819f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58096/0 (socket says 172.21.15.12:58096) 2024-03-23T22:52:10.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.024+0000 7f61e6a00700 1 -- 172.21.15.12:0/2688364257 learned_addr learned my addr 172.21.15.12:0/2688364257 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:52:10.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.025+0000 7f61e6a00700 1 -- 172.21.15.12:0/2688364257 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e819fae0 con 0x7f61e8101dd0 2024-03-23T22:52:10.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.025+0000 7f61e6a00700 1 --2- 172.21.15.12:0/2688364257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61e8101dd0 0x7f61e819f5a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f61d8006010 tx=0x7f61d8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:52:10.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.025+0000 7f61df400700 1 -- 172.21.15.12:0/2688364257 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f61d8026070 con 0x7f61e8101dd0 2024-03-23T22:52:10.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.025+0000 7f61df400700 1 -- 172.21.15.12:0/2688364257 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61d800bbd0 con 0x7f61e8101dd0 2024-03-23T22:52:10.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.025+0000 7f61ed800700 1 -- 172.21.15.12:0/2688364257 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e819fdd0 con 0x7f61e8101dd0 2024-03-23T22:52:10.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.025+0000 7f61df400700 1 -- 172.21.15.12:0/2688364257 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f61d8003910 con 0x7f61e8101dd0 2024-03-23T22:52:10.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.025+0000 7f61ed800700 1 -- 172.21.15.12:0/2688364257 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e8199840 con 0x7f61e8101dd0 2024-03-23T22:52:10.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.026+0000 7f61df400700 1 -- 172.21.15.12:0/2688364257 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f61d8029070 con 0x7f61e8101dd0 2024-03-23T22:52:10.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.026+0000 7f61ed800700 1 -- 172.21.15.12:0/2688364257 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e81032b0 con 0x7f61e8101dd0 2024-03-23T22:52:10.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.027+0000 7f61df400700 1 --2- 172.21.15.12:0/2688364257 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f61cc038fc0 0x7f61cc03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:10.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.027+0000 7f61df400700 1 -- 172.21.15.12:0/2688364257 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f61d801e070 con 0x7f61e8101dd0 2024-03-23T22:52:10.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.027+0000 7f61e6000700 1 --2- 172.21.15.12:0/2688364257 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f61cc038fc0 0x7f61cc03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:10.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.028+0000 7f61e6000700 1 --2- 172.21.15.12:0/2688364257 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f61cc038fc0 0x7f61cc03b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f61d4006fd0 tx=0x7f61d4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:52:10.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.033+0000 7f61df400700 1 -- 172.21.15.12:0/2688364257 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f61d8013070 con 0x7f61e8101dd0 2024-03-23T22:52:10.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:10.269+0000 7f61ed800700 1 -- 172.21.15.12:0/2688364257 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f61e80008d0 con 0x7f61e8101dd0 2024-03-23T22:52:10.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:10 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:10.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:10 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:10.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:10 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:10.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:10 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:10.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:10 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi203", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:52:10.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:10 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:10.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:10 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:52:11.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:11.073+0000 7f61df400700 1 -- 172.21.15.12:0/2688364257 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f61d800fe20 con 0x7f61e8101dd0 2024-03-23T22:52:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:11.077+0000 7f61ed800700 1 -- 172.21.15.12:0/2688364257 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f61cc038fc0 msgr2=0x7f61cc03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:11.077+0000 7f61ed800700 1 --2- 172.21.15.12:0/2688364257 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f61cc038fc0 0x7f61cc03b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f61d4006fd0 tx=0x7f61d4006e40 comp rx=0 tx=0).stop 2024-03-23T22:52:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:11.077+0000 7f61ed800700 1 -- 172.21.15.12:0/2688364257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61e8101dd0 msgr2=0x7f61e819f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:11.077+0000 7f61ed800700 1 --2- 172.21.15.12:0/2688364257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61e8101dd0 0x7f61e819f5a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f61d8006010 tx=0x7f61d8005030 comp rx=0 tx=0).stop 2024-03-23T22:52:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:11.077+0000 7f61ed800700 1 -- 172.21.15.12:0/2688364257 shutdown_connections 2024-03-23T22:52:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:11.077+0000 7f61ed800700 1 --2- 172.21.15.12:0/2688364257 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f61cc038fc0 0x7f61cc03b480 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:11.077+0000 7f61ed800700 1 --2- 172.21.15.12:0/2688364257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61e8101dd0 0x7f61e819f5a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:11.078+0000 7f61ed800700 1 -- 172.21.15.12:0/2688364257 >> 172.21.15.12:0/2688364257 conn(0x7f61e80fd6a0 msgr2=0x7f61e8075ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:52:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:11.078+0000 7f61ed800700 1 -- 172.21.15.12:0/2688364257 shutdown_connections 2024-03-23T22:52:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:11.078+0000 7f61ed800700 1 -- 172.21.15.12:0/2688364257 wait complete. 2024-03-23T22:52:11.079 INFO:teuthology.orchestra.run.smithi012.stderr:adjusted tunables profile to default 2024-03-23T22:52:11.490 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:11 smithi012 ceph-mon[31695]: Updating smithi203:/etc/ceph/ceph.conf 2024-03-23T22:52:11.490 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:11 smithi012 ceph-mon[31695]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:11.490 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:11 smithi012 ceph-mon[31695]: Updating smithi203:/var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T22:52:11.490 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:11 smithi012 ceph-mon[31695]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-03-23T22:52:11.490 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:11 smithi012 ceph-mon[31695]: Updating smithi203:/var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-23T22:52:11.490 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:11 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2688364257' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-23T22:52:11.490 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:11 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:11.491 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:11 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:11.491 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:11 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:11.856 INFO:tasks.cephadm:Adding mon.a on smithi012 2024-03-23T22:52:11.856 INFO:tasks.cephadm:Adding mon.b on smithi173 2024-03-23T22:52:11.857 INFO:tasks.cephadm:Adding mon.c on smithi203 2024-03-23T22:52:11.857 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph orch apply mon '3;smithi012:172.21.15.12=a;smithi173:172.21.15.173=b;smithi203:172.21.15.203=c' 2024-03-23T22:52:12.099 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T22:52:12.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:12 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2688364257' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-23T22:52:12.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:12 smithi012 ceph-mon[31695]: osdmap e4: 0 total, 0 up, 0 in 2024-03-23T22:52:13.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:13 smithi012 ceph-mon[31695]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:13.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.645+0000 7f707e2f5700 1 -- 172.21.15.203:0/1385493083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7078101dd0 msgr2=0x7f70781021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:13.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.645+0000 7f707e2f5700 1 --2- 172.21.15.203:0/1385493083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7078101dd0 0x7f70781021b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7068009910 tx=0x7f7068009c20 comp rx=0 tx=0).stop 2024-03-23T22:52:13.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.645+0000 7f707e2f5700 1 -- 172.21.15.203:0/1385493083 shutdown_connections 2024-03-23T22:52:13.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.645+0000 7f707e2f5700 1 --2- 172.21.15.203:0/1385493083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7078101dd0 0x7f70781021b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:13.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.645+0000 7f707e2f5700 1 -- 172.21.15.203:0/1385493083 >> 172.21.15.203:0/1385493083 conn(0x7f70780fd6a0 msgr2=0x7f70780ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:52:13.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.645+0000 7f707e2f5700 1 -- 172.21.15.203:0/1385493083 shutdown_connections 2024-03-23T22:52:13.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.645+0000 7f707e2f5700 1 -- 172.21.15.203:0/1385493083 wait complete. 2024-03-23T22:52:13.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.646+0000 7f707e2f5700 1 Processor -- start 2024-03-23T22:52:13.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.646+0000 7f707e2f5700 1 -- start start 2024-03-23T22:52:13.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.646+0000 7f707e2f5700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7078101dd0 0x7f7078197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:13.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.646+0000 7f707e2f5700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707810c1a0 con 0x7f7078101dd0 2024-03-23T22:52:13.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.647+0000 7f7077fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7078101dd0 0x7f7078197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:13.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.647+0000 7f7077fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7078101dd0 0x7f7078197740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.203:58376/0 (socket says 172.21.15.203:58376) 2024-03-23T22:52:13.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.647+0000 7f7077fff700 1 -- 172.21.15.203:0/3918809663 learned_addr learned my addr 172.21.15.203:0/3918809663 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-23T22:52:13.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.648+0000 7f7077fff700 1 -- 172.21.15.203:0/3918809663 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7078197c80 con 0x7f7078101dd0 2024-03-23T22:52:13.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.648+0000 7f7077fff700 1 --2- 172.21.15.203:0/3918809663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7078101dd0 0x7f7078197740 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7068005850 tx=0x7f7068005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:52:13.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.648+0000 7f70757fa700 1 -- 172.21.15.203:0/3918809663 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7068026070 con 0x7f7078101dd0 2024-03-23T22:52:13.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.648+0000 7f707e2f5700 1 -- 172.21.15.203:0/3918809663 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7078197f70 con 0x7f7078101dd0 2024-03-23T22:52:13.649 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.648+0000 7f707e2f5700 1 -- 172.21.15.203:0/3918809663 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f707819c010 con 0x7f7078101dd0 2024-03-23T22:52:13.649 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.648+0000 7f70757fa700 1 -- 172.21.15.203:0/3918809663 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f706800b6f0 con 0x7f7078101dd0 2024-03-23T22:52:13.649 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.648+0000 7f70757fa700 1 -- 172.21.15.203:0/3918809663 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7068030060 con 0x7f7078101dd0 2024-03-23T22:52:13.649 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.649+0000 7f707e2f5700 1 -- 172.21.15.203:0/3918809663 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7064005330 con 0x7f7078101dd0 2024-03-23T22:52:13.649 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.650+0000 7f70757fa700 1 -- 172.21.15.203:0/3918809663 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f7068029070 con 0x7f7078101dd0 2024-03-23T22:52:13.650 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.650+0000 7f70757fa700 1 --2- 172.21.15.203:0/3918809663 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7060039020 0x7f706003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:13.650 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.650+0000 7f70757fa700 1 -- 172.21.15.203:0/3918809663 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f706801e070 con 0x7f7078101dd0 2024-03-23T22:52:13.650 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.651+0000 7f70777fe700 1 --2- 172.21.15.203:0/3918809663 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7060039020 0x7f706003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:13.651 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.652+0000 7f70777fe700 1 --2- 172.21.15.203:0/3918809663 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7060039020 0x7f706003b4e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f706c006fd0 tx=0x7f706c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:52:13.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.666+0000 7f70757fa700 1 -- 172.21.15.203:0/3918809663 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7068013070 con 0x7f7078101dd0 2024-03-23T22:52:13.869 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.870+0000 7f707e2f5700 1 -- 172.21.15.203:0/3918809663 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi012:172.21.15.12=a;smithi173:172.21.15.173=b;smithi203:172.21.15.203=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f7064000c80 con 0x7f7060039020 2024-03-23T22:52:13.885 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.885+0000 7f70757fa700 1 -- 172.21.15.203:0/3918809663 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f7064000c80 con 0x7f7060039020 2024-03-23T22:52:13.885 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mon update... 2024-03-23T22:52:13.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.887+0000 7f707e2f5700 1 -- 172.21.15.203:0/3918809663 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7060039020 msgr2=0x7f706003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:13.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.887+0000 7f707e2f5700 1 --2- 172.21.15.203:0/3918809663 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7060039020 0x7f706003b4e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f706c006fd0 tx=0x7f706c006e40 comp rx=0 tx=0).stop 2024-03-23T22:52:13.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.887+0000 7f707e2f5700 1 -- 172.21.15.203:0/3918809663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7078101dd0 msgr2=0x7f7078197740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:13.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.887+0000 7f707e2f5700 1 --2- 172.21.15.203:0/3918809663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7078101dd0 0x7f7078197740 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7068005850 tx=0x7f7068005030 comp rx=0 tx=0).stop 2024-03-23T22:52:13.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.887+0000 7f707e2f5700 1 -- 172.21.15.203:0/3918809663 shutdown_connections 2024-03-23T22:52:13.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.887+0000 7f707e2f5700 1 --2- 172.21.15.203:0/3918809663 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7060039020 0x7f706003b4e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:13.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.887+0000 7f707e2f5700 1 --2- 172.21.15.203:0/3918809663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7078101dd0 0x7f7078197740 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:13.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.887+0000 7f707e2f5700 1 -- 172.21.15.203:0/3918809663 >> 172.21.15.203:0/3918809663 conn(0x7f70780fd6a0 msgr2=0x7f7078104c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:52:13.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.887+0000 7f707e2f5700 1 -- 172.21.15.203:0/3918809663 shutdown_connections 2024-03-23T22:52:13.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:13.887+0000 7f707e2f5700 1 -- 172.21.15.203:0/3918809663 wait complete. 2024-03-23T22:52:14.459 DEBUG:teuthology.orchestra.run.smithi173:mon.b> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mon.b.service 2024-03-23T22:52:14.461 DEBUG:teuthology.orchestra.run.smithi203:mon.c> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mon.c.service 2024-03-23T22:52:14.464 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-23T22:52:14.464 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-23T22:52:15.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:14 smithi012 ceph-mon[31695]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:15.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:14 smithi012 ceph-mon[31695]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi012:172.21.15.12=a;smithi173:172.21.15.173=b;smithi203:172.21.15.203=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:52:15.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:14 smithi012 ceph-mon[31695]: Saving service mon spec with placement smithi012:172.21.15.12=a;smithi173:172.21.15.173=b;smithi203:172.21.15.203=c;count:3 2024-03-23T22:52:15.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:15.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:52:15.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:15.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:52:15.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:15.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T22:52:15.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:15.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:14 smithi012 ceph-mon[31695]: Deploying daemon mon.c on smithi203 2024-03-23T22:52:15.587 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T22:52:17.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:16 smithi012 ceph-mon[31695]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:17.576 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.575+0000 7f14eb1ac700 1 -- 172.21.15.203:0/1410998401 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e4106a60 msgr2=0x7f14e4106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:17.576 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.575+0000 7f14eb1ac700 1 --2- 172.21.15.203:0/1410998401 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e4106a60 0x7f14e4106e40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f14d8009910 tx=0x7f14d8009c20 comp rx=0 tx=0).stop 2024-03-23T22:52:17.576 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.575+0000 7f14eb1ac700 1 -- 172.21.15.203:0/1410998401 shutdown_connections 2024-03-23T22:52:17.576 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.575+0000 7f14eb1ac700 1 --2- 172.21.15.203:0/1410998401 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e4106a60 0x7f14e4106e40 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:17.576 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.575+0000 7f14eb1ac700 1 -- 172.21.15.203:0/1410998401 >> 172.21.15.203:0/1410998401 conn(0x7f14e40776d0 msgr2=0x7f14e40fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:52:17.576 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.575+0000 7f14eb1ac700 1 -- 172.21.15.203:0/1410998401 shutdown_connections 2024-03-23T22:52:17.576 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.575+0000 7f14eb1ac700 1 -- 172.21.15.203:0/1410998401 wait complete. 2024-03-23T22:52:17.576 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.576+0000 7f14eb1ac700 1 Processor -- start 2024-03-23T22:52:17.576 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.576+0000 7f14eb1ac700 1 -- start start 2024-03-23T22:52:17.577 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.576+0000 7f14eb1ac700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e4106a60 0x7f14e40fca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:17.577 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.576+0000 7f14eb1ac700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e410b0a0 con 0x7f14e4106a60 2024-03-23T22:52:17.577 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.577+0000 7f14e8f48700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e4106a60 0x7f14e40fca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:17.577 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.577+0000 7f14e8f48700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e4106a60 0x7f14e40fca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.203:58400/0 (socket says 172.21.15.203:58400) 2024-03-23T22:52:17.577 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.577+0000 7f14e8f48700 1 -- 172.21.15.203:0/1917855477 learned_addr learned my addr 172.21.15.203:0/1917855477 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-23T22:52:17.578 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.577+0000 7f14e8f48700 1 -- 172.21.15.203:0/1917855477 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e40fe950 con 0x7f14e4106a60 2024-03-23T22:52:17.578 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.578+0000 7f14e8f48700 1 --2- 172.21.15.203:0/1917855477 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e4106a60 0x7f14e40fca60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f14d8005850 tx=0x7f14d8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:52:17.578 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.578+0000 7f14e1ffb700 1 -- 172.21.15.203:0/1917855477 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14d8026070 con 0x7f14e4106a60 2024-03-23T22:52:17.578 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.578+0000 7f14eb1ac700 1 -- 172.21.15.203:0/1917855477 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e40fd000 con 0x7f14e4106a60 2024-03-23T22:52:17.578 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.578+0000 7f14eb1ac700 1 -- 172.21.15.203:0/1917855477 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e40fd4c0 con 0x7f14e4106a60 2024-03-23T22:52:17.579 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.578+0000 7f14e1ffb700 1 -- 172.21.15.203:0/1917855477 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14d800bbd0 con 0x7f14e4106a60 2024-03-23T22:52:17.579 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.579+0000 7f14e1ffb700 1 -- 172.21.15.203:0/1917855477 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14d80039b0 con 0x7f14e4106a60 2024-03-23T22:52:17.579 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.579+0000 7f14eb1ac700 1 -- 172.21.15.203:0/1917855477 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e41021a0 con 0x7f14e4106a60 2024-03-23T22:52:17.579 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.580+0000 7f14e1ffb700 1 -- 172.21.15.203:0/1917855477 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f14d8029070 con 0x7f14e4106a60 2024-03-23T22:52:17.580 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.580+0000 7f14e1ffb700 1 --2- 172.21.15.203:0/1917855477 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f14d4039020 0x7f14d403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:17.580 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.580+0000 7f14e1ffb700 1 -- 172.21.15.203:0/1917855477 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f14d801e070 con 0x7f14e4106a60 2024-03-23T22:52:17.580 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.581+0000 7f14e3fff700 1 --2- 172.21.15.203:0/1917855477 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f14d4039020 0x7f14d403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:17.581 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.582+0000 7f14e3fff700 1 --2- 172.21.15.203:0/1917855477 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f14d4039020 0x7f14d403b4e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f14d0006fd0 tx=0x7f14d0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:52:17.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.596+0000 7f14e1ffb700 1 -- 172.21.15.203:0/1917855477 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f14d8013070 con 0x7f14e4106a60 2024-03-23T22:52:17.877 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.877+0000 7f14eb1ac700 1 -- 172.21.15.203:0/1917855477 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f14e40fdf90 con 0x7f14e4106a60 2024-03-23T22:52:17.877 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.878+0000 7f14e1ffb700 1 -- 172.21.15.203:0/1917855477 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+745 (secure 0 0 0) 0x7f14d804b8c0 con 0x7f14e4106a60 2024-03-23T22:52:17.878 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-23T22:52:17.878 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"90f679ae-e967-11ee-95cd-87774f69a715","modified":"2024-03-23T22:50:10.340903Z","created":"2024-03-23T22:50:10.340903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-23T22:52:17.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.880+0000 7f14eb1ac700 1 -- 172.21.15.203:0/1917855477 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f14d4039020 msgr2=0x7f14d403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:17.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.880+0000 7f14eb1ac700 1 --2- 172.21.15.203:0/1917855477 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f14d4039020 0x7f14d403b4e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f14d0006fd0 tx=0x7f14d0006e40 comp rx=0 tx=0).stop 2024-03-23T22:52:17.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.880+0000 7f14eb1ac700 1 -- 172.21.15.203:0/1917855477 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e4106a60 msgr2=0x7f14e40fca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:17.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.880+0000 7f14eb1ac700 1 --2- 172.21.15.203:0/1917855477 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e4106a60 0x7f14e40fca60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f14d8005850 tx=0x7f14d8005030 comp rx=0 tx=0).stop 2024-03-23T22:52:17.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.880+0000 7f14eb1ac700 1 -- 172.21.15.203:0/1917855477 shutdown_connections 2024-03-23T22:52:17.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.880+0000 7f14eb1ac700 1 --2- 172.21.15.203:0/1917855477 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f14d4039020 0x7f14d403b4e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:17.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.880+0000 7f14eb1ac700 1 --2- 172.21.15.203:0/1917855477 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e4106a60 0x7f14e40fca60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:17.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.880+0000 7f14eb1ac700 1 -- 172.21.15.203:0/1917855477 >> 172.21.15.203:0/1917855477 conn(0x7f14e40776d0 msgr2=0x7f14e4104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:52:17.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.880+0000 7f14eb1ac700 1 -- 172.21.15.203:0/1917855477 shutdown_connections 2024-03-23T22:52:17.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:17.880+0000 7f14eb1ac700 1 -- 172.21.15.203:0/1917855477 wait complete. 2024-03-23T22:52:17.884 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-03-23T22:52:18.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:17 smithi012 ceph-mon[31695]: from='client.? 172.21.15.203:0/1917855477' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-23T22:52:19.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:18 smithi012 ceph-mon[31695]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:19.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 podman[36674]: 2024-03-23 22:52:19.208982988 +0000 UTC m=+0.018264459 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-23T22:52:19.602 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-23T22:52:19.602 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-23T22:52:19.836 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.c/config 2024-03-23T22:52:19.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 podman[36674]: 2024-03-23 22:52:19.502791241 +0000 UTC m=+0.312072705 container init f46e88dc1a11fc46375d0e892e4d6ebd408c6480a927ebc8cb0c8dfa47378640 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-c, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-23T22:52:19.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 podman[36674]: 2024-03-23 22:52:19.506164116 +0000 UTC m=+0.315445575 container start f46e88dc1a11fc46375d0e892e4d6ebd408c6480a927ebc8cb0c8dfa47378640 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-c, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-23T22:52:19.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: set uid:gid to 167:167 (ceph:ceph) 2024-03-23T22:52:19.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev), process ceph-mon, pid 2 2024-03-23T22:52:19.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: pidfile_write: ignore empty --pid-file 2024-03-23T22:52:19.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: load: jerasure load: lrc 2024-03-23T22:52:19.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: RocksDB version: 7.9.2 2024-03-23T22:52:19.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Git sha 0 2024-03-23T22:52:19.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Compile date 2024-03-22 04:58:55 2024-03-23T22:52:19.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: DB SUMMARY 2024-03-23T22:52:19.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: DB Session ID: 92ESYXM0VZ6NEW0OENYZ 2024-03-23T22:52:19.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: CURRENT file: CURRENT 2024-03-23T22:52:19.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: IDENTITY file: IDENTITY 2024-03-23T22:52:19.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-23T22:52:19.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-23T22:52:19.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-23T22:52:19.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.error_if_exists: 0 2024-03-23T22:52:19.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.create_if_missing: 0 2024-03-23T22:52:19.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.paranoid_checks: 1 2024-03-23T22:52:19.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-23T22:52:19.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-23T22:52:19.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-23T22:52:19.910 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.env: 0x55ac9eb62b20 2024-03-23T22:52:19.910 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.fs: PosixFileSystem 2024-03-23T22:52:19.910 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.info_log: 0x55aca0439920 2024-03-23T22:52:19.910 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_file_opening_threads: 16 2024-03-23T22:52:19.910 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.statistics: (nil) 2024-03-23T22:52:19.910 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.use_fsync: 0 2024-03-23T22:52:19.910 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_log_file_size: 0 2024-03-23T22:52:19.910 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-23T22:52:19.910 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-23T22:52:19.910 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.keep_log_file_num: 1000 2024-03-23T22:52:19.910 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.recycle_log_file_num: 0 2024-03-23T22:52:19.910 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.allow_fallocate: 1 2024-03-23T22:52:19.910 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.allow_mmap_reads: 0 2024-03-23T22:52:19.911 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.allow_mmap_writes: 0 2024-03-23T22:52:19.911 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.use_direct_reads: 0 2024-03-23T22:52:19.911 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-23T22:52:19.911 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.create_missing_column_families: 0 2024-03-23T22:52:19.911 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.db_log_dir: 2024-03-23T22:52:19.911 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.wal_dir: 2024-03-23T22:52:19.911 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-23T22:52:19.911 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-23T22:52:19.911 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-23T22:52:19.911 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-23T22:52:19.911 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-23T22:52:19.911 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-23T22:52:19.911 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.advise_random_on_open: 1 2024-03-23T22:52:19.911 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.db_write_buffer_size: 0 2024-03-23T22:52:19.911 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.write_buffer_manager: 0x55aca040c1e0 2024-03-23T22:52:19.912 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-23T22:52:19.912 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-23T22:52:19.912 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-23T22:52:19.912 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.rate_limiter: (nil) 2024-03-23T22:52:19.912 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-23T22:52:19.912 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.wal_recovery_mode: 2 2024-03-23T22:52:19.912 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.enable_thread_tracking: 0 2024-03-23T22:52:19.912 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.enable_pipelined_write: 0 2024-03-23T22:52:19.912 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.unordered_write: 0 2024-03-23T22:52:19.912 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-23T22:52:19.912 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-23T22:52:19.912 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-23T22:52:19.912 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-23T22:52:19.912 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.row_cache: None 2024-03-23T22:52:19.912 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.wal_filter: None 2024-03-23T22:52:19.914 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-23T22:52:19.914 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.allow_ingest_behind: 0 2024-03-23T22:52:19.914 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.two_write_queues: 0 2024-03-23T22:52:19.914 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.manual_wal_flush: 0 2024-03-23T22:52:19.914 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.wal_compression: 0 2024-03-23T22:52:19.914 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.atomic_flush: 0 2024-03-23T22:52:19.915 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-23T22:52:19.915 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-23T22:52:19.915 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-23T22:52:19.915 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.log_readahead_size: 0 2024-03-23T22:52:19.915 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-23T22:52:19.915 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.best_efforts_recovery: 0 2024-03-23T22:52:19.915 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-23T22:52:19.915 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-23T22:52:19.915 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.allow_data_in_errors: 0 2024-03-23T22:52:19.915 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.db_host_id: __hostname__ 2024-03-23T22:52:19.915 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-23T22:52:19.915 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_background_jobs: 2 2024-03-23T22:52:19.915 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_background_compactions: -1 2024-03-23T22:52:19.915 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_subcompactions: 1 2024-03-23T22:52:19.915 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-23T22:52:19.916 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-23T22:52:19.916 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-23T22:52:19.916 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_total_wal_size: 0 2024-03-23T22:52:19.916 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-23T22:52:19.916 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-23T22:52:19.916 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-23T22:52:19.916 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-23T22:52:19.916 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_open_files: -1 2024-03-23T22:52:19.916 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.bytes_per_sync: 0 2024-03-23T22:52:19.916 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-23T22:52:19.916 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-23T22:52:19.916 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.compaction_readahead_size: 0 2024-03-23T22:52:19.916 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_background_flushes: -1 2024-03-23T22:52:19.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Compression algorithms supported: 2024-03-23T22:52:19.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-23T22:52:19.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: kZSTD supported: 0 2024-03-23T22:52:19.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: kXpressCompression supported: 0 2024-03-23T22:52:19.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: kLZ4HCCompression supported: 1 2024-03-23T22:52:19.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: kZlibCompression supported: 1 2024-03-23T22:52:19.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: kSnappyCompression supported: 1 2024-03-23T22:52:19.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: kLZ4Compression supported: 1 2024-03-23T22:52:19.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: kBZip2Compression supported: 0 2024-03-23T22:52:19.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-23T22:52:19.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-23T22:52:19.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-23T22:52:19.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-23T22:52:19.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-23T22:52:19.918 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.merge_operator: 2024-03-23T22:52:19.918 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.compaction_filter: None 2024-03-23T22:52:19.918 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.compaction_filter_factory: None 2024-03-23T22:52:19.918 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.sst_partitioner_factory: None 2024-03-23T22:52:19.918 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-23T22:52:19.918 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-23T22:52:19.918 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55aca0439a20) 2024-03-23T22:52:19.918 INFO:journalctl@ceph.mon.c.smithi203.stdout: cache_index_and_filter_blocks: 1 2024-03-23T22:52:19.918 INFO:journalctl@ceph.mon.c.smithi203.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-23T22:52:19.918 INFO:journalctl@ceph.mon.c.smithi203.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-23T22:52:19.918 INFO:journalctl@ceph.mon.c.smithi203.stdout: pin_top_level_index_and_filter: 1 2024-03-23T22:52:19.918 INFO:journalctl@ceph.mon.c.smithi203.stdout: index_type: 0 2024-03-23T22:52:19.918 INFO:journalctl@ceph.mon.c.smithi203.stdout: data_block_index_type: 0 2024-03-23T22:52:19.921 INFO:journalctl@ceph.mon.c.smithi203.stdout: index_shortening: 1 2024-03-23T22:52:19.922 INFO:journalctl@ceph.mon.c.smithi203.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-23T22:52:19.922 INFO:journalctl@ceph.mon.c.smithi203.stdout: checksum: 4 2024-03-23T22:52:19.922 INFO:journalctl@ceph.mon.c.smithi203.stdout: no_block_cache: 0 2024-03-23T22:52:19.922 INFO:journalctl@ceph.mon.c.smithi203.stdout: block_cache: 0x55aca043f090 2024-03-23T22:52:19.922 INFO:journalctl@ceph.mon.c.smithi203.stdout: block_cache_name: BinnedLRUCache 2024-03-23T22:52:19.922 INFO:journalctl@ceph.mon.c.smithi203.stdout: block_cache_options: 2024-03-23T22:52:19.922 INFO:journalctl@ceph.mon.c.smithi203.stdout: capacity : 536870912 2024-03-23T22:52:19.922 INFO:journalctl@ceph.mon.c.smithi203.stdout: num_shard_bits : 4 2024-03-23T22:52:19.922 INFO:journalctl@ceph.mon.c.smithi203.stdout: strict_capacity_limit : 0 2024-03-23T22:52:19.923 INFO:journalctl@ceph.mon.c.smithi203.stdout: high_pri_pool_ratio: 0.000 2024-03-23T22:52:19.923 INFO:journalctl@ceph.mon.c.smithi203.stdout: block_cache_compressed: (nil) 2024-03-23T22:52:19.923 INFO:journalctl@ceph.mon.c.smithi203.stdout: persistent_cache: (nil) 2024-03-23T22:52:19.923 INFO:journalctl@ceph.mon.c.smithi203.stdout: block_size: 4096 2024-03-23T22:52:19.923 INFO:journalctl@ceph.mon.c.smithi203.stdout: block_size_deviation: 10 2024-03-23T22:52:19.923 INFO:journalctl@ceph.mon.c.smithi203.stdout: block_restart_interval: 16 2024-03-23T22:52:19.923 INFO:journalctl@ceph.mon.c.smithi203.stdout: index_block_restart_interval: 1 2024-03-23T22:52:19.923 INFO:journalctl@ceph.mon.c.smithi203.stdout: metadata_block_size: 4096 2024-03-23T22:52:19.923 INFO:journalctl@ceph.mon.c.smithi203.stdout: partition_filters: 0 2024-03-23T22:52:19.923 INFO:journalctl@ceph.mon.c.smithi203.stdout: use_delta_encoding: 1 2024-03-23T22:52:19.923 INFO:journalctl@ceph.mon.c.smithi203.stdout: filter_policy: bloomfilter 2024-03-23T22:52:19.923 INFO:journalctl@ceph.mon.c.smithi203.stdout: whole_key_filtering: 1 2024-03-23T22:52:19.923 INFO:journalctl@ceph.mon.c.smithi203.stdout: verify_compression: 0 2024-03-23T22:52:19.923 INFO:journalctl@ceph.mon.c.smithi203.stdout: read_amp_bytes_per_bit: 0 2024-03-23T22:52:19.924 INFO:journalctl@ceph.mon.c.smithi203.stdout: format_version: 5 2024-03-23T22:52:19.924 INFO:journalctl@ceph.mon.c.smithi203.stdout: enable_index_compression: 1 2024-03-23T22:52:19.924 INFO:journalctl@ceph.mon.c.smithi203.stdout: block_align: 0 2024-03-23T22:52:19.924 INFO:journalctl@ceph.mon.c.smithi203.stdout: max_auto_readahead_size: 262144 2024-03-23T22:52:19.924 INFO:journalctl@ceph.mon.c.smithi203.stdout: prepopulate_block_cache: 0 2024-03-23T22:52:19.924 INFO:journalctl@ceph.mon.c.smithi203.stdout: initial_auto_readahead_size: 8192 2024-03-23T22:52:19.924 INFO:journalctl@ceph.mon.c.smithi203.stdout: num_file_reads_for_auto_readahead: 2 2024-03-23T22:52:19.924 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.write_buffer_size: 33554432 2024-03-23T22:52:19.924 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_write_buffer_number: 2 2024-03-23T22:52:19.924 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.compression: NoCompression 2024-03-23T22:52:19.924 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.bottommost_compression: Disabled 2024-03-23T22:52:19.924 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.prefix_extractor: nullptr 2024-03-23T22:52:19.924 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-23T22:52:19.924 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.num_levels: 7 2024-03-23T22:52:19.925 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-23T22:52:19.925 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-23T22:52:19.925 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-23T22:52:19.925 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-23T22:52:19.925 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-23T22:52:19.925 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-23T22:52:19.925 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-23T22:52:19.925 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-23T22:52:19.925 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-23T22:52:19.925 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-23T22:52:19.925 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-23T22:52:19.925 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-23T22:52:19.925 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-23T22:52:19.925 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.compression_opts.level: 32767 2024-03-23T22:52:19.925 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.compression_opts.strategy: 0 2024-03-23T22:52:19.926 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-23T22:52:19.926 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-23T22:52:19.926 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-23T22:52:19.926 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-23T22:52:19.926 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.compression_opts.enabled: false 2024-03-23T22:52:19.926 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-23T22:52:19.926 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-23T22:52:19.926 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-23T22:52:19.926 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-23T22:52:19.926 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.target_file_size_base: 67108864 2024-03-23T22:52:19.926 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-23T22:52:19.926 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-23T22:52:19.926 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-23T22:52:19.927 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-23T22:52:19.927 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-23T22:52:19.927 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-23T22:52:19.927 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-23T22:52:19.927 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-23T22:52:19.927 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-23T22:52:19.927 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-23T22:52:19.927 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-23T22:52:19.927 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-23T22:52:19.927 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-23T22:52:19.927 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-23T22:52:19.927 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.arena_block_size: 1048576 2024-03-23T22:52:19.927 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-23T22:52:19.927 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-23T22:52:19.928 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.disable_auto_compactions: 0 2024-03-23T22:52:19.928 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-23T22:52:19.928 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-23T22:52:19.928 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-23T22:52:19.928 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-23T22:52:19.928 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-23T22:52:19.928 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-23T22:52:19.928 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-23T22:52:19.928 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-23T22:52:19.928 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-23T22:52:19.928 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-23T22:52:19.928 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.table_properties_collectors: 2024-03-23T22:52:19.928 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.inplace_update_support: 0 2024-03-23T22:52:19.928 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-23T22:52:19.929 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-23T22:52:19.929 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-23T22:52:19.929 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-23T22:52:19.929 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.bloom_locality: 0 2024-03-23T22:52:19.929 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.max_successive_merges: 0 2024-03-23T22:52:19.929 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-23T22:52:19.929 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.paranoid_file_checks: 0 2024-03-23T22:52:19.929 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.force_consistency_checks: 1 2024-03-23T22:52:19.929 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.report_bg_io_stats: 0 2024-03-23T22:52:19.929 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.ttl: 2592000 2024-03-23T22:52:19.929 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-23T22:52:19.929 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-23T22:52:19.929 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-23T22:52:19.929 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.enable_blob_files: false 2024-03-23T22:52:19.929 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.min_blob_size: 0 2024-03-23T22:52:19.930 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.blob_file_size: 268435456 2024-03-23T22:52:19.930 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-23T22:52:19.930 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-23T22:52:19.930 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-23T22:52:19.930 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-23T22:52:19.930 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-23T22:52:19.930 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.blob_file_starting_level: 0 2024-03-23T22:52:19.930 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-23T22:52:19.930 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-23T22:52:19.930 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-23T22:52:19.930 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 51582ea6-89ff-4e9f-8319-2cb94fa5c4f4 2024-03-23T22:52:19.930 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711234339535364, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-23T22:52:19.930 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-23T22:52:19.930 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711234339536011, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711234339, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "51582ea6-89ff-4e9f-8319-2cb94fa5c4f4", "db_session_id": "92ESYXM0VZ6NEW0OENYZ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-23T22:52:19.931 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711234339536093, "job": 1, "event": "recovery_finished"} 2024-03-23T22:52:19.931 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-23T22:52:19.931 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-23T22:52:19.931 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55aca0532000 2024-03-23T22:52:19.931 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: DB pointer 0x55aca051c000 2024-03-23T22:52:19.931 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-23T22:52:19.931 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-23T22:52:19.931 INFO:journalctl@ceph.mon.c.smithi203.stdout: ** DB Stats ** 2024-03-23T22:52:19.931 INFO:journalctl@ceph.mon.c.smithi203.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-23T22:52:19.931 INFO:journalctl@ceph.mon.c.smithi203.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-23T22:52:19.931 INFO:journalctl@ceph.mon.c.smithi203.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-23T22:52:19.931 INFO:journalctl@ceph.mon.c.smithi203.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-23T22:52:19.931 INFO:journalctl@ceph.mon.c.smithi203.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-23T22:52:19.932 INFO:journalctl@ceph.mon.c.smithi203.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-23T22:52:19.932 INFO:journalctl@ceph.mon.c.smithi203.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-23T22:52:19.932 INFO:journalctl@ceph.mon.c.smithi203.stdout: 2024-03-23T22:52:19.932 INFO:journalctl@ceph.mon.c.smithi203.stdout: ** Compaction Stats [default] ** 2024-03-23T22:52:19.932 INFO:journalctl@ceph.mon.c.smithi203.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-23T22:52:19.932 INFO:journalctl@ceph.mon.c.smithi203.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-23T22:52:19.932 INFO:journalctl@ceph.mon.c.smithi203.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-23T22:52:19.932 INFO:journalctl@ceph.mon.c.smithi203.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-23T22:52:19.932 INFO:journalctl@ceph.mon.c.smithi203.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-23T22:52:19.932 INFO:journalctl@ceph.mon.c.smithi203.stdout: 2024-03-23T22:52:19.932 INFO:journalctl@ceph.mon.c.smithi203.stdout: ** Compaction Stats [default] ** 2024-03-23T22:52:19.932 INFO:journalctl@ceph.mon.c.smithi203.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-23T22:52:19.932 INFO:journalctl@ceph.mon.c.smithi203.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-23T22:52:19.932 INFO:journalctl@ceph.mon.c.smithi203.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-23T22:52:19.932 INFO:journalctl@ceph.mon.c.smithi203.stdout: 2024-03-23T22:52:19.933 INFO:journalctl@ceph.mon.c.smithi203.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-23T22:52:19.933 INFO:journalctl@ceph.mon.c.smithi203.stdout: 2024-03-23T22:52:19.933 INFO:journalctl@ceph.mon.c.smithi203.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-23T22:52:19.933 INFO:journalctl@ceph.mon.c.smithi203.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-23T22:52:19.933 INFO:journalctl@ceph.mon.c.smithi203.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-23T22:52:19.933 INFO:journalctl@ceph.mon.c.smithi203.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-23T22:52:19.933 INFO:journalctl@ceph.mon.c.smithi203.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-23T22:52:19.933 INFO:journalctl@ceph.mon.c.smithi203.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-23T22:52:19.933 INFO:journalctl@ceph.mon.c.smithi203.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-23T22:52:19.933 INFO:journalctl@ceph.mon.c.smithi203.stdout: Interval compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-23T22:52:19.933 INFO:journalctl@ceph.mon.c.smithi203.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-23T22:52:19.933 INFO:journalctl@ceph.mon.c.smithi203.stdout: Block cache BinnedLRUCache@0x55aca043f090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-23T22:52:19.933 INFO:journalctl@ceph.mon.c.smithi203.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-23T22:52:19.934 INFO:journalctl@ceph.mon.c.smithi203.stdout: 2024-03-23T22:52:19.934 INFO:journalctl@ceph.mon.c.smithi203.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-23T22:52:19.934 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-23T22:52:19.934 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: using public_addr v2:172.21.15.203:0/0 -> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-03-23T22:52:19.934 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: starting mon.c rank -1 at public addrs [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] at bind addrs [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T22:52:19.934 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: mon.c@-1(???) e0 preinit fsid 90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T22:52:19.934 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: mon.c@-1(synchronizing).mds e1 new map 2024-03-23T22:52:19.934 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-23T22:52:19.934 INFO:journalctl@ceph.mon.c.smithi203.stdout: e1 2024-03-23T22:52:19.934 INFO:journalctl@ceph.mon.c.smithi203.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T22:52:19.934 INFO:journalctl@ceph.mon.c.smithi203.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-23T22:52:19.934 INFO:journalctl@ceph.mon.c.smithi203.stdout: legacy client fscid: -1 2024-03-23T22:52:19.934 INFO:journalctl@ceph.mon.c.smithi203.stdout: 2024-03-23T22:52:19.934 INFO:journalctl@ceph.mon.c.smithi203.stdout: No filesystems configured 2024-03-23T22:52:19.935 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-23T22:52:19.935 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-23T22:52:19.935 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-23T22:52:19.935 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-23T22:52:19.935 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-23T22:52:19.935 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-23T22:52:19.935 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-23T22:52:19.935 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-23T22:52:19.935 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-23T22:52:19.935 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-23T22:52:19.935 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:19.935 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:52:19.935 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:19.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: Deploying cephadm binary to smithi173 2024-03-23T22:52:19.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:19.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:19.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:19.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: Added host smithi173 2024-03-23T22:52:19.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:52:19.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:19.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:19.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:19.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:19.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T22:52:19.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:19.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:19.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:19.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:19.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:19.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:19.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:52:19.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:19.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:52:19.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-23T22:52:19.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: Updating smithi173:/var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T22:52:19.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-03-23T22:52:19.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:52:19.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: Updating smithi173:/var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-23T22:52:19.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:19.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:19.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:19.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: Deploying cephadm binary to smithi203 2024-03-23T22:52:19.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:19.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:19.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:19.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: Added host smithi203 2024-03-23T22:52:19.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:52:19.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:19.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:19.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:19.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:19.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:19.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T22:52:19.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:19.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:19.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:19.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:19.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:19.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi203", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:52:19.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:19.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:52:19.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: Updating smithi203:/etc/ceph/ceph.conf 2024-03-23T22:52:19.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:19.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: Updating smithi203:/var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T22:52:19.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-03-23T22:52:19.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: Updating smithi203:/var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-23T22:52:19.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2688364257' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-23T22:52:19.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:19.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:19.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:19.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2688364257' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-23T22:52:19.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: osdmap e4: 0 total, 0 up, 0 in 2024-03-23T22:52:19.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:19.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:19.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi012:172.21.15.12=a;smithi173:172.21.15.173=b;smithi203:172.21.15.203=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:52:19.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: Saving service mon spec with placement smithi012:172.21.15.12=a;smithi173:172.21.15.173=b;smithi203:172.21.15.203=c;count:3 2024-03-23T22:52:19.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:19.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:52:19.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:19.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:52:19.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:19.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T22:52:19.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:19.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: Deploying daemon mon.c on smithi203 2024-03-23T22:52:19.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:19.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: from='client.? 172.21.15.203:0/1917855477' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-23T22:52:19.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:19.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 ceph-mon[36693]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-23T22:52:19.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 bash[36674]: f46e88dc1a11fc46375d0e892e4d6ebd408c6480a927ebc8cb0c8dfa47378640 2024-03-23T22:52:19.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:19 smithi203 systemd[1]: Started Ceph mon.c for 90f679ae-e967-11ee-95cd-87774f69a715. 2024-03-23T22:52:24.958 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.957+0000 7feb6d3be700 1 -- 172.21.15.203:0/1031857650 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb540035c0 msgr2=0x7feb54005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:24.958 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.957+0000 7feb6d3be700 1 --2- 172.21.15.203:0/1031857650 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb540035c0 0x7feb54005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7feb68069bc0 tx=0x7feb58014ec0 comp rx=0 tx=0).stop 2024-03-23T22:52:24.958 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.957+0000 7feb6d3be700 1 -- 172.21.15.203:0/1031857650 shutdown_connections 2024-03-23T22:52:24.958 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.957+0000 7feb6d3be700 1 --2- 172.21.15.203:0/1031857650 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb540035c0 0x7feb54005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:24.958 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.957+0000 7feb6d3be700 1 -- 172.21.15.203:0/1031857650 >> 172.21.15.203:0/1031857650 conn(0x7feb680fc640 msgr2=0x7feb680fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:52:24.958 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.958+0000 7feb6d3be700 1 -- 172.21.15.203:0/1031857650 shutdown_connections 2024-03-23T22:52:24.958 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.958+0000 7feb6d3be700 1 -- 172.21.15.203:0/1031857650 wait complete. 2024-03-23T22:52:24.959 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:24 smithi203 ceph-mon[36693]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:24.959 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-23T22:52:24.959 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:24 smithi203 ceph-mon[36693]: mon.a calling monitor election 2024-03-23T22:52:24.959 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:52:24.959 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:52:24.959 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:24 smithi203 ceph-mon[36693]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:24.959 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:52:24.959 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:24 smithi203 ceph-mon[36693]: mon.c calling monitor election 2024-03-23T22:52:24.959 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:52:24.959 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:24 smithi203 ceph-mon[36693]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:24.959 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:52:24.959 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:52:24.959 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:24 smithi203 ceph-mon[36693]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-23T22:52:24.960 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:24 smithi203 ceph-mon[36693]: monmap e2: 2 mons at {a=[v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0],c=[v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-23T22:52:24.960 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:24 smithi203 ceph-mon[36693]: fsmap 2024-03-23T22:52:24.960 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:24 smithi203 ceph-mon[36693]: osdmap e4: 0 total, 0 up, 0 in 2024-03-23T22:52:24.960 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:24 smithi203 ceph-mon[36693]: mgrmap e13: x(active, since 61s) 2024-03-23T22:52:24.960 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:24 smithi203 ceph-mon[36693]: overall HEALTH_OK 2024-03-23T22:52:24.960 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:24.960 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:24.961 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.959+0000 7feb6d3be700 1 Processor -- start 2024-03-23T22:52:24.961 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.959+0000 7feb6d3be700 1 -- start start 2024-03-23T22:52:24.962 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.959+0000 7feb6d3be700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb540035c0 0x7feb68100a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:24.962 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.959+0000 7feb6d3be700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb68100fb0 0x7feb68101430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:24.962 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.959+0000 7feb6d3be700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb68109cf0 con 0x7feb540035c0 2024-03-23T22:52:24.962 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.959+0000 7feb6d3be700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb68109b70 con 0x7feb68100fb0 2024-03-23T22:52:24.962 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.960+0000 7feb5e7fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb68100fb0 0x7feb68101430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:24.962 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.960+0000 7feb5e7fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb68100fb0 0x7feb68101430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:48972/0 (socket says 172.21.15.203:48972) 2024-03-23T22:52:24.962 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.960+0000 7feb5e7fc700 1 -- 172.21.15.203:0/1762748771 learned_addr learned my addr 172.21.15.203:0/1762748771 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-23T22:52:24.962 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.960+0000 7feb5e7fc700 1 -- 172.21.15.203:0/1762748771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb68100fb0 msgr2=0x7feb68101430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-23T22:52:24.962 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.960+0000 7feb5e7fc700 1 -- 172.21.15.203:0/1762748771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb68100fb0 msgr2=0x7feb68101430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-23T22:52:24.962 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.960+0000 7feb5e7fc700 1 --2- 172.21.15.203:0/1762748771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb68100fb0 0x7feb68101430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T22:52:24.962 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.960+0000 7feb5e7fc700 1 --2- 172.21.15.203:0/1762748771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb68100fb0 0x7feb68101430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T22:52:24.962 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.960+0000 7feb66ffd700 1 --2- 172.21.15.203:0/1762748771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb540035c0 0x7feb68100a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:24.962 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.961+0000 7feb66ffd700 1 -- 172.21.15.203:0/1762748771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb68100fb0 msgr2=0x7feb68101430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-23T22:52:24.962 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.961+0000 7feb66ffd700 1 --2- 172.21.15.203:0/1762748771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb68100fb0 0x7feb68101430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:24.963 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.961+0000 7feb66ffd700 1 -- 172.21.15.203:0/1762748771 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb68102940 con 0x7feb540035c0 2024-03-23T22:52:24.963 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.961+0000 7feb66ffd700 1 --2- 172.21.15.203:0/1762748771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb540035c0 0x7feb68100a70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7feb5801d000 tx=0x7feb5800fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:52:24.963 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.961+0000 7feb64ff9700 1 -- 172.21.15.203:0/1762748771 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7feb58004aa0 con 0x7feb540035c0 2024-03-23T22:52:24.963 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.961+0000 7feb6d3be700 1 -- 172.21.15.203:0/1762748771 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb681a3aa0 con 0x7feb540035c0 2024-03-23T22:52:24.963 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.961+0000 7feb64ff9700 1 -- 172.21.15.203:0/1762748771 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feb58004c40 con 0x7feb540035c0 2024-03-23T22:52:24.963 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.962+0000 7feb64ff9700 1 -- 172.21.15.203:0/1762748771 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7feb5801e670 con 0x7feb540035c0 2024-03-23T22:52:24.963 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.962+0000 7feb6d3be700 1 -- 172.21.15.203:0/1762748771 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb681a4000 con 0x7feb540035c0 2024-03-23T22:52:24.964 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.963+0000 7feb64ff9700 1 -- 172.21.15.203:0/1762748771 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7feb5801e810 con 0x7feb540035c0 2024-03-23T22:52:24.964 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.963+0000 7feb6d3be700 1 -- 172.21.15.203:0/1762748771 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb6806a3c0 con 0x7feb540035c0 2024-03-23T22:52:24.964 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.963+0000 7feb64ff9700 1 --2- 172.21.15.203:0/1762748771 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7feb48038d80 0x7feb4803b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:24.964 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.963+0000 7feb64ff9700 1 -- 172.21.15.203:0/1762748771 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7feb58025070 con 0x7feb540035c0 2024-03-23T22:52:24.964 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.964+0000 7feb5e7fc700 1 --2- 172.21.15.203:0/1762748771 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7feb48038d80 0x7feb4803b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:24.965 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.965+0000 7feb5e7fc700 1 --2- 172.21.15.203:0/1762748771 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7feb48038d80 0x7feb4803b240 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7feb50009180 tx=0x7feb50008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:52:24.978 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:24.979+0000 7feb64ff9700 1 -- 172.21.15.203:0/1762748771 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7feb58050e20 con 0x7feb540035c0 2024-03-23T22:52:25.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:24 smithi012 ceph-mon[31695]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:25.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-23T22:52:25.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:24 smithi012 ceph-mon[31695]: mon.a calling monitor election 2024-03-23T22:52:25.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:52:25.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:52:25.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:24 smithi012 ceph-mon[31695]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:25.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:52:25.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:24 smithi012 ceph-mon[31695]: mon.c calling monitor election 2024-03-23T22:52:25.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:52:25.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:24 smithi012 ceph-mon[31695]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:25.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:52:25.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:52:25.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:24 smithi012 ceph-mon[31695]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-23T22:52:25.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:24 smithi012 ceph-mon[31695]: monmap e2: 2 mons at {a=[v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0],c=[v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-23T22:52:25.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:24 smithi012 ceph-mon[31695]: fsmap 2024-03-23T22:52:25.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:24 smithi012 ceph-mon[31695]: osdmap e4: 0 total, 0 up, 0 in 2024-03-23T22:52:25.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:24 smithi012 ceph-mon[31695]: mgrmap e13: x(active, since 61s) 2024-03-23T22:52:25.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:24 smithi012 ceph-mon[31695]: overall HEALTH_OK 2024-03-23T22:52:25.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:25.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:25.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:25.260+0000 7feb6d3be700 1 -- 172.21.15.203:0/1762748771 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb681a42d0 con 0x7feb540035c0 2024-03-23T22:52:25.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:25.260+0000 7feb64ff9700 1 -- 172.21.15.203:0/1762748771 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1015 (secure 0 0 0) 0x7feb58004e60 con 0x7feb540035c0 2024-03-23T22:52:25.260 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-23T22:52:25.260 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":2,"fsid":"90f679ae-e967-11ee-95cd-87774f69a715","modified":"2024-03-23T22:52:19.572330Z","created":"2024-03-23T22:50:10.340903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-23T22:52:25.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:25.262+0000 7feb6d3be700 1 -- 172.21.15.203:0/1762748771 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7feb48038d80 msgr2=0x7feb4803b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:25.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:25.262+0000 7feb6d3be700 1 --2- 172.21.15.203:0/1762748771 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7feb48038d80 0x7feb4803b240 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7feb50009180 tx=0x7feb50008040 comp rx=0 tx=0).stop 2024-03-23T22:52:25.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:25.263+0000 7feb6d3be700 1 -- 172.21.15.203:0/1762748771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb540035c0 msgr2=0x7feb68100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:25.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:25.263+0000 7feb6d3be700 1 --2- 172.21.15.203:0/1762748771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb540035c0 0x7feb68100a70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7feb5801d000 tx=0x7feb5800fc60 comp rx=0 tx=0).stop 2024-03-23T22:52:25.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:25.263+0000 7feb6d3be700 1 -- 172.21.15.203:0/1762748771 shutdown_connections 2024-03-23T22:52:25.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:25.263+0000 7feb6d3be700 1 --2- 172.21.15.203:0/1762748771 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7feb48038d80 0x7feb4803b240 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:25.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:25.263+0000 7feb6d3be700 1 --2- 172.21.15.203:0/1762748771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb540035c0 0x7feb68100a70 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:25.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:25.263+0000 7feb6d3be700 1 --2- 172.21.15.203:0/1762748771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb68100fb0 0x7feb68101430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:25.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:25.263+0000 7feb6d3be700 1 -- 172.21.15.203:0/1762748771 >> 172.21.15.203:0/1762748771 conn(0x7feb680fc640 msgr2=0x7feb68103bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:52:25.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:25.263+0000 7feb6d3be700 1 -- 172.21.15.203:0/1762748771 shutdown_connections 2024-03-23T22:52:25.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:25.263+0000 7feb6d3be700 1 -- 172.21.15.203:0/1762748771 wait complete. 2024-03-23T22:52:25.263 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 2 2024-03-23T22:52:25.856 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:25 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:25.856 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:25 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T22:52:25.856 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:25 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:25.856 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:25 smithi203 ceph-mon[36693]: Deploying daemon mon.b on smithi173 2024-03-23T22:52:25.856 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:25 smithi203 ceph-mon[36693]: from='client.? 172.21.15.203:0/1762748771' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-23T22:52:25.856 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:25 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:52:25.955 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:25 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:25.955 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:25 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T22:52:25.955 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:25 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:25.955 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:25 smithi012 ceph-mon[31695]: Deploying daemon mon.b on smithi173 2024-03-23T22:52:25.955 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:25 smithi012 ceph-mon[31695]: from='client.? 172.21.15.203:0/1762748771' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-23T22:52:25.955 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:25 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:52:26.883 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-23T22:52:26.883 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-23T22:52:26.980 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:26 smithi012 ceph-mon[31695]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:27.130 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.c/config 2024-03-23T22:52:27.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:26 smithi203 ceph-mon[36693]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:28.541 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:28 smithi173 ceph-mon[36116]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-23T22:52:28.541 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:28 smithi173 bash[36097]: 72286162409a4f9c012e1fe519afd4df1fd68da8a2593c2e35871bab01498d36 2024-03-23T22:52:28.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:28 smithi173 systemd[1]: Started Ceph mon.b for 90f679ae-e967-11ee-95cd-87774f69a715. 2024-03-23T22:52:33.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:33 smithi203 ceph-mon[36693]: mon.c calling monitor election 2024-03-23T22:52:33.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:33 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-23T22:52:33.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:33 smithi203 ceph-mon[36693]: mon.a calling monitor election 2024-03-23T22:52:33.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:33 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:52:33.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:33 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:52:33.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:33 smithi203 ceph-mon[36693]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:33.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:33 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:52:33.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:33 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:52:33.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:33 smithi203 ceph-mon[36693]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:33.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:33 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:52:33.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:33 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:52:33.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:33 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:52:33.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:33 smithi203 ceph-mon[36693]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-23T22:52:33.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:33 smithi203 ceph-mon[36693]: monmap e3: 3 mons at {a=[v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0],c=[v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-23T22:52:33.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:33 smithi203 ceph-mon[36693]: fsmap 2024-03-23T22:52:33.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:33 smithi203 ceph-mon[36693]: osdmap e4: 0 total, 0 up, 0 in 2024-03-23T22:52:33.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:33 smithi203 ceph-mon[36693]: mgrmap e13: x(active, since 70s) 2024-03-23T22:52:33.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:33 smithi203 ceph-mon[36693]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-23T22:52:33.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:33 smithi203 ceph-mon[36693]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-23T22:52:33.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:33 smithi203 ceph-mon[36693]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-23T22:52:33.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:33 smithi203 ceph-mon[36693]: mon.b (rank 2) addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] is down (out of quorum) 2024-03-23T22:52:33.910 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:33 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:33.910 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:33 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:33.910 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:33 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:33.910 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:33 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:34.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:33 smithi012 ceph-mon[31695]: mon.c calling monitor election 2024-03-23T22:52:34.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:33 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-23T22:52:34.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:33 smithi012 ceph-mon[31695]: mon.a calling monitor election 2024-03-23T22:52:34.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:33 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:52:34.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:33 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:52:34.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:33 smithi012 ceph-mon[31695]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:34.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:33 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:52:34.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:33 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:52:34.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:33 smithi012 ceph-mon[31695]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:34.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:33 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:52:34.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:33 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:52:34.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:33 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:52:34.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:33 smithi012 ceph-mon[31695]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-23T22:52:34.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:33 smithi012 ceph-mon[31695]: monmap e3: 3 mons at {a=[v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0],c=[v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-23T22:52:34.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:33 smithi012 ceph-mon[31695]: fsmap 2024-03-23T22:52:34.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:33 smithi012 ceph-mon[31695]: osdmap e4: 0 total, 0 up, 0 in 2024-03-23T22:52:34.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:33 smithi012 ceph-mon[31695]: mgrmap e13: x(active, since 70s) 2024-03-23T22:52:34.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:33 smithi012 ceph-mon[31695]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-23T22:52:34.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:33 smithi012 ceph-mon[31695]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-23T22:52:34.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:33 smithi012 ceph-mon[31695]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-23T22:52:34.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:33 smithi012 ceph-mon[31695]: mon.b (rank 2) addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] is down (out of quorum) 2024-03-23T22:52:34.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:33 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:34.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:33 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:34.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:33 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:34.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:33 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:34.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.590+0000 7fb55c6d6700 1 -- 172.21.15.203:0/719474071 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5400084b0 msgr2=0x7fb54000a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:34.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.590+0000 7fb55c6d6700 1 --2- 172.21.15.203:0/719474071 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5400084b0 0x7fb54000a920 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb554101c10 tx=0x7fb54c0095d0 comp rx=0 tx=0).stop 2024-03-23T22:52:34.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.590+0000 7fb55c6d6700 1 -- 172.21.15.203:0/719474071 shutdown_connections 2024-03-23T22:52:34.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.590+0000 7fb55c6d6700 1 --2- 172.21.15.203:0/719474071 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5400084b0 0x7fb54000a920 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:34.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.590+0000 7fb55c6d6700 1 -- 172.21.15.203:0/719474071 >> 172.21.15.203:0/719474071 conn(0x7fb5540fc640 msgr2=0x7fb5540fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:52:34.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.590+0000 7fb55c6d6700 1 -- 172.21.15.203:0/719474071 shutdown_connections 2024-03-23T22:52:34.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.590+0000 7fb55c6d6700 1 -- 172.21.15.203:0/719474071 wait complete. 2024-03-23T22:52:34.594 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.591+0000 7fb55c6d6700 1 Processor -- start 2024-03-23T22:52:34.594 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.592+0000 7fb55c6d6700 1 -- start start 2024-03-23T22:52:34.594 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.592+0000 7fb55c6d6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5400084b0 0x7fb55419b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:34.594 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.592+0000 7fb55c6d6700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb55419bac0 0x7fb554195650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:34.594 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.592+0000 7fb55c6d6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb554195b90 0x7fb554196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:34.594 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.592+0000 7fb55c6d6700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb554107a30 con 0x7fb55419bac0 2024-03-23T22:52:34.594 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.592+0000 7fb55c6d6700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55406aa70 con 0x7fb554195b90 2024-03-23T22:52:34.594 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.592+0000 7fb55c6d6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb554107bb0 con 0x7fb5400084b0 2024-03-23T22:52:34.594 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.592+0000 7fb55a472700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5400084b0 0x7fb55419b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:34.594 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.592+0000 7fb55a472700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5400084b0 0x7fb55419b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:45728/0 (socket says 172.21.15.203:45728) 2024-03-23T22:52:34.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.592+0000 7fb55a472700 1 -- 172.21.15.203:0/3640618903 learned_addr learned my addr 172.21.15.203:0/3640618903 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-23T22:52:34.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.592+0000 7fb55ac73700 1 --2- 172.21.15.203:0/3640618903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb554195b90 0x7fb554196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:34.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.592+0000 7fb559c71700 1 --2- 172.21.15.203:0/3640618903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb55419bac0 0x7fb554195650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:34.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.592+0000 7fb55a472700 1 -- 172.21.15.203:0/3640618903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb554195b90 msgr2=0x7fb554196040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:34.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.592+0000 7fb55a472700 1 --2- 172.21.15.203:0/3640618903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb554195b90 0x7fb554196040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:34.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.592+0000 7fb55a472700 1 -- 172.21.15.203:0/3640618903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb55419bac0 msgr2=0x7fb554195650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:34.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.592+0000 7fb55a472700 1 --2- 172.21.15.203:0/3640618903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb55419bac0 0x7fb554195650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:34.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.592+0000 7fb55a472700 1 -- 172.21.15.203:0/3640618903 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5541a2330 con 0x7fb5400084b0 2024-03-23T22:52:34.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.592+0000 7fb55a472700 1 --2- 172.21.15.203:0/3640618903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5400084b0 0x7fb55419b580 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb54c006080 tx=0x7fb54c003ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:52:34.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.593+0000 7fb54b7fe700 1 -- 172.21.15.203:0/3640618903 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c01bc30 con 0x7fb5400084b0 2024-03-23T22:52:34.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.593+0000 7fb54b7fe700 1 -- 172.21.15.203:0/3640618903 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb54c01bdd0 con 0x7fb5400084b0 2024-03-23T22:52:34.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.593+0000 7fb55c6d6700 1 -- 172.21.15.203:0/3640618903 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5541a2560 con 0x7fb5400084b0 2024-03-23T22:52:34.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.593+0000 7fb54b7fe700 1 -- 172.21.15.203:0/3640618903 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c019ab0 con 0x7fb5400084b0 2024-03-23T22:52:34.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.593+0000 7fb55c6d6700 1 -- 172.21.15.203:0/3640618903 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5541a2a40 con 0x7fb5400084b0 2024-03-23T22:52:34.596 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.593+0000 7fb54b7fe700 1 -- 172.21.15.203:0/3640618903 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fb54c015040 con 0x7fb5400084b0 2024-03-23T22:52:34.596 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.594+0000 7fb54b7fe700 1 --2- 172.21.15.203:0/3640618903 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb538039450 0x7fb53803b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:34.596 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.594+0000 7fb54b7fe700 1 -- 172.21.15.203:0/3640618903 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb54c04e950 con 0x7fb5400084b0 2024-03-23T22:52:34.596 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.594+0000 7fb55c6d6700 1 -- 172.21.15.203:0/3640618903 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb554102290 con 0x7fb5400084b0 2024-03-23T22:52:34.596 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.594+0000 7fb559c71700 1 --2- 172.21.15.203:0/3640618903 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb538039450 0x7fb53803b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:34.596 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.595+0000 7fb559c71700 1 --2- 172.21.15.203:0/3640618903 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb538039450 0x7fb53803b910 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb544009850 tx=0x7fb544008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:52:34.598 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.599+0000 7fb54b7fe700 1 -- 172.21.15.203:0/3640618903 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb54c04e140 con 0x7fb5400084b0 2024-03-23T22:52:34.868 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:34 smithi203 ceph-mon[36693]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:34.868 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:34 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:52:34.868 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:34 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:52:34.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.887+0000 7fb55c6d6700 1 -- 172.21.15.203:0/3640618903 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb55402da30 con 0x7fb5400084b0 2024-03-23T22:52:34.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.887+0000 7fb54b7fe700 1 -- 172.21.15.203:0/3640618903 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1283 (secure 0 0 0) 0x7fb54c03a090 con 0x7fb5400084b0 2024-03-23T22:52:34.887 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-23T22:52:34.887 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":3,"fsid":"90f679ae-e967-11ee-95cd-87774f69a715","modified":"2024-03-23T22:52:28.522515Z","created":"2024-03-23T22:50:10.340903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-23T22:52:34.889 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.889+0000 7fb55c6d6700 1 -- 172.21.15.203:0/3640618903 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb538039450 msgr2=0x7fb53803b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:34.889 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.889+0000 7fb55c6d6700 1 --2- 172.21.15.203:0/3640618903 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb538039450 0x7fb53803b910 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb544009850 tx=0x7fb544008040 comp rx=0 tx=0).stop 2024-03-23T22:52:34.889 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.889+0000 7fb55c6d6700 1 -- 172.21.15.203:0/3640618903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5400084b0 msgr2=0x7fb55419b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:34.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.890+0000 7fb55c6d6700 1 --2- 172.21.15.203:0/3640618903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5400084b0 0x7fb55419b580 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb54c006080 tx=0x7fb54c003ea0 comp rx=0 tx=0).stop 2024-03-23T22:52:34.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.890+0000 7fb55c6d6700 1 -- 172.21.15.203:0/3640618903 shutdown_connections 2024-03-23T22:52:34.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.890+0000 7fb55c6d6700 1 --2- 172.21.15.203:0/3640618903 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb538039450 0x7fb53803b910 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:34.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.890+0000 7fb55c6d6700 1 --2- 172.21.15.203:0/3640618903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5400084b0 0x7fb55419b580 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:34.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.890+0000 7fb55c6d6700 1 --2- 172.21.15.203:0/3640618903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb554195b90 0x7fb554196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:34.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.890+0000 7fb55c6d6700 1 --2- 172.21.15.203:0/3640618903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb55419bac0 0x7fb554195650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:34.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.890+0000 7fb55c6d6700 1 -- 172.21.15.203:0/3640618903 >> 172.21.15.203:0/3640618903 conn(0x7fb5540fc640 msgr2=0x7fb5540fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:52:34.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.890+0000 7fb55c6d6700 1 -- 172.21.15.203:0/3640618903 shutdown_connections 2024-03-23T22:52:34.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:34.890+0000 7fb55c6d6700 1 -- 172.21.15.203:0/3640618903 wait complete. 2024-03-23T22:52:34.891 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 3 2024-03-23T22:52:35.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:34 smithi012 ceph-mon[31695]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:35.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:34 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:52:35.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:34 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:52:35.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: mon.c calling monitor election 2024-03-23T22:52:35.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-23T22:52:35.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: mon.a calling monitor election 2024-03-23T22:52:35.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:52:35.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:52:35.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:35.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:52:35.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:52:35.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:35.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:52:35.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:52:35.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:52:35.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-23T22:52:35.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: monmap e3: 3 mons at {a=[v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0],c=[v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-23T22:52:35.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: fsmap 2024-03-23T22:52:35.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: osdmap e4: 0 total, 0 up, 0 in 2024-03-23T22:52:35.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: mgrmap e13: x(active, since 70s) 2024-03-23T22:52:35.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-23T22:52:35.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-23T22:52:35.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-23T22:52:35.870 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: mon.b (rank 2) addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] is down (out of quorum) 2024-03-23T22:52:35.870 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:35.870 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:35.870 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:35.870 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:35.870 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:35.870 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:52:35.870 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:35 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:52:35.931 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-23T22:52:35.932 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-23T22:52:36.357 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:52:36.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:36 smithi012 ceph-mon[31695]: mon.b calling monitor election 2024-03-23T22:52:36.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:36 smithi012 ceph-mon[31695]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:36.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:36 smithi012 ceph-mon[31695]: Updating smithi012:/etc/ceph/ceph.conf 2024-03-23T22:52:36.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:36 smithi012 ceph-mon[31695]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-23T22:52:36.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:36 smithi012 ceph-mon[31695]: Updating smithi203:/etc/ceph/ceph.conf 2024-03-23T22:52:36.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:36 smithi012 ceph-mon[31695]: mon.b calling monitor election 2024-03-23T22:52:36.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:36 smithi012 ceph-mon[31695]: mon.c calling monitor election 2024-03-23T22:52:36.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:36 smithi012 ceph-mon[31695]: mon.a calling monitor election 2024-03-23T22:52:36.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:36 smithi012 ceph-mon[31695]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-23T22:52:36.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:36 smithi012 ceph-mon[31695]: monmap e3: 3 mons at {a=[v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0],c=[v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-23T22:52:36.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:36 smithi012 ceph-mon[31695]: fsmap 2024-03-23T22:52:36.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:36 smithi012 ceph-mon[31695]: osdmap e4: 0 total, 0 up, 0 in 2024-03-23T22:52:36.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:36 smithi012 ceph-mon[31695]: mgrmap e13: x(active, since 72s) 2024-03-23T22:52:36.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:36 smithi012 ceph-mon[31695]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-23T22:52:36.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:36 smithi012 ceph-mon[31695]: Cluster is now healthy 2024-03-23T22:52:36.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:36 smithi012 ceph-mon[31695]: overall HEALTH_OK 2024-03-23T22:52:36.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:36 smithi012 ceph-mon[31695]: Updating smithi173:/var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T22:52:36.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:36 smithi012 ceph-mon[31695]: Updating smithi203:/var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T22:52:36.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:36 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:36.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:36 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:36.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:36 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:36.556 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:36 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:36.556 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:36 smithi012 ceph-mon[31695]: Updating smithi012:/var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T22:52:36.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:36 smithi173 ceph-mon[36116]: mon.b calling monitor election 2024-03-23T22:52:36.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:36 smithi173 ceph-mon[36116]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:36.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:36 smithi173 ceph-mon[36116]: Updating smithi012:/etc/ceph/ceph.conf 2024-03-23T22:52:36.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:36 smithi173 ceph-mon[36116]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-23T22:52:36.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:36 smithi173 ceph-mon[36116]: Updating smithi203:/etc/ceph/ceph.conf 2024-03-23T22:52:36.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:36 smithi173 ceph-mon[36116]: mon.b calling monitor election 2024-03-23T22:52:36.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:36 smithi173 ceph-mon[36116]: mon.c calling monitor election 2024-03-23T22:52:36.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:36 smithi173 ceph-mon[36116]: mon.a calling monitor election 2024-03-23T22:52:36.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:36 smithi173 ceph-mon[36116]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-23T22:52:36.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:36 smithi173 ceph-mon[36116]: monmap e3: 3 mons at {a=[v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0],c=[v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-23T22:52:36.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:36 smithi173 ceph-mon[36116]: fsmap 2024-03-23T22:52:36.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:36 smithi173 ceph-mon[36116]: osdmap e4: 0 total, 0 up, 0 in 2024-03-23T22:52:36.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:36 smithi173 ceph-mon[36116]: mgrmap e13: x(active, since 72s) 2024-03-23T22:52:36.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:36 smithi173 ceph-mon[36116]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-23T22:52:36.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:36 smithi173 ceph-mon[36116]: Cluster is now healthy 2024-03-23T22:52:36.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:36 smithi173 ceph-mon[36116]: overall HEALTH_OK 2024-03-23T22:52:36.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:36 smithi173 ceph-mon[36116]: Updating smithi173:/var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T22:52:36.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:36 smithi173 ceph-mon[36116]: Updating smithi203:/var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T22:52:36.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:36 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:36.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:36 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:36.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:36 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:36.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:36 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:36.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:36 smithi173 ceph-mon[36116]: Updating smithi012:/var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T22:52:36.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:36 smithi203 ceph-mon[36693]: mon.b calling monitor election 2024-03-23T22:52:36.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:36 smithi203 ceph-mon[36693]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:36.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:36 smithi203 ceph-mon[36693]: Updating smithi012:/etc/ceph/ceph.conf 2024-03-23T22:52:36.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:36 smithi203 ceph-mon[36693]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-23T22:52:36.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:36 smithi203 ceph-mon[36693]: Updating smithi203:/etc/ceph/ceph.conf 2024-03-23T22:52:36.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:36 smithi203 ceph-mon[36693]: mon.b calling monitor election 2024-03-23T22:52:36.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:36 smithi203 ceph-mon[36693]: mon.c calling monitor election 2024-03-23T22:52:36.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:36 smithi203 ceph-mon[36693]: mon.a calling monitor election 2024-03-23T22:52:36.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:36 smithi203 ceph-mon[36693]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-23T22:52:36.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:36 smithi203 ceph-mon[36693]: monmap e3: 3 mons at {a=[v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0],c=[v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-23T22:52:36.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:36 smithi203 ceph-mon[36693]: fsmap 2024-03-23T22:52:36.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:36 smithi203 ceph-mon[36693]: osdmap e4: 0 total, 0 up, 0 in 2024-03-23T22:52:36.659 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:36 smithi203 ceph-mon[36693]: mgrmap e13: x(active, since 72s) 2024-03-23T22:52:36.659 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:36 smithi203 ceph-mon[36693]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-23T22:52:36.659 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:36 smithi203 ceph-mon[36693]: Cluster is now healthy 2024-03-23T22:52:36.659 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:36 smithi203 ceph-mon[36693]: overall HEALTH_OK 2024-03-23T22:52:36.659 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:36 smithi203 ceph-mon[36693]: Updating smithi173:/var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T22:52:36.659 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:36 smithi203 ceph-mon[36693]: Updating smithi203:/var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T22:52:36.659 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:36 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:36.659 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:36 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:36.659 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:36 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:36.659 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:36 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:36.659 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:36 smithi203 ceph-mon[36693]: Updating smithi012:/var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T22:52:37.506 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:37 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:52:37.506 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:37 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:37.506 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:37 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:37.506 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:37 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:37.506 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:37 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:37.506 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:37 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:37.506 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:37 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:37.506 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:37 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:37.506 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:37 smithi012 ceph-mon[31695]: Reconfiguring mon.a (unknown last config time)... 2024-03-23T22:52:37.507 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:37 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T22:52:37.507 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:37 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-23T22:52:37.507 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:37 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:37.507 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:37 smithi012 ceph-mon[31695]: Reconfiguring daemon mon.a on smithi012 2024-03-23T22:52:37.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:37 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:52:37.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:37 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:37.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:37 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:37.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:37 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:37.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:37 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:37.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:37 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:37.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:37 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:37.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:37 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:37.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:37 smithi173 ceph-mon[36116]: Reconfiguring mon.a (unknown last config time)... 2024-03-23T22:52:37.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:37 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T22:52:37.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:37 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-23T22:52:37.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:37 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:37.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:37 smithi173 ceph-mon[36116]: Reconfiguring daemon mon.a on smithi012 2024-03-23T22:52:37.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:37 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:52:37.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:37 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:37.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:37 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:37.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:37 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:37.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:37 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:37.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:37 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:37.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:37 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:37.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:37 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:37.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:37 smithi203 ceph-mon[36693]: Reconfiguring mon.a (unknown last config time)... 2024-03-23T22:52:37.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:37 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T22:52:37.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:37 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-23T22:52:37.659 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:37 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:37.659 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:37 smithi203 ceph-mon[36693]: Reconfiguring daemon mon.a on smithi012 2024-03-23T22:52:37.804 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:52:37 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:52:37.522+0000 7f7041e00700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-23T22:52:38.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.439+0000 7f375fe00700 1 -- 172.21.15.12:0/1619803246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3758102d10 msgr2=0x7f37581030f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:39.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.439+0000 7f375fe00700 1 --2- 172.21.15.12:0/1619803246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3758102d10 0x7f37581030f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f374c009910 tx=0x7f374c009c20 comp rx=0 tx=0).stop 2024-03-23T22:52:39.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.439+0000 7f375fe00700 1 -- 172.21.15.12:0/1619803246 shutdown_connections 2024-03-23T22:52:39.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.439+0000 7f375fe00700 1 --2- 172.21.15.12:0/1619803246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3758102d10 0x7f37581030f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:39.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.439+0000 7f375fe00700 1 -- 172.21.15.12:0/1619803246 >> 172.21.15.12:0/1619803246 conn(0x7f37580fc540 msgr2=0x7f37580fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:52:39.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.439+0000 7f375fe00700 1 -- 172.21.15.12:0/1619803246 shutdown_connections 2024-03-23T22:52:39.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.440+0000 7f375fe00700 1 -- 172.21.15.12:0/1619803246 wait complete. 2024-03-23T22:52:39.421 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:38 smithi203 ceph-mon[36693]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:39.421 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:38 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:39.421 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:38 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/589982753' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:39.421 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:38 smithi173 ceph-mon[36116]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:39.421 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:38 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:39.421 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:38 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/589982753' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:39.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.441+0000 7f375fe00700 1 Processor -- start 2024-03-23T22:52:39.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.441+0000 7f375fe00700 1 -- start start 2024-03-23T22:52:39.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.441+0000 7f375fe00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3758102d10 0x7f375819f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:39.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.441+0000 7f375fe00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f375819fd40 0x7f37581999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:39.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.441+0000 7f375fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3758199f00 0x7f375819a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:39.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.441+0000 7f375fe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375810c200 con 0x7f3758199f00 2024-03-23T22:52:39.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.441+0000 7f375fe00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375810c080 con 0x7f375819fd40 2024-03-23T22:52:39.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.441+0000 7f375fe00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375810c380 con 0x7f3758102d10 2024-03-23T22:52:39.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.442+0000 7f375de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3758199f00 0x7f375819a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:39.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.442+0000 7f375d400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3758102d10 0x7f375819f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:39.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.442+0000 7f375ca00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f375819fd40 0x7f37581999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:39.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.442+0000 7f375de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3758199f00 0x7f375819a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48128/0 (socket says 172.21.15.12:48128) 2024-03-23T22:52:39.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.442+0000 7f375de00700 1 -- 172.21.15.12:0/589982753 learned_addr learned my addr 172.21.15.12:0/589982753 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:52:39.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.442+0000 7f375ca00700 1 -- 172.21.15.12:0/589982753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3758102d10 msgr2=0x7f375819f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:39.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.442+0000 7f375ca00700 1 --2- 172.21.15.12:0/589982753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3758102d10 0x7f375819f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:39.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.442+0000 7f375ca00700 1 -- 172.21.15.12:0/589982753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3758199f00 msgr2=0x7f375819a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:39.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.442+0000 7f375ca00700 1 --2- 172.21.15.12:0/589982753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3758199f00 0x7f375819a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:39.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.442+0000 7f375ca00700 1 -- 172.21.15.12:0/589982753 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37581a66d0 con 0x7f375819fd40 2024-03-23T22:52:39.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.442+0000 7f375de00700 1 --2- 172.21.15.12:0/589982753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3758199f00 0x7f375819a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T22:52:39.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.442+0000 7f375ca00700 1 --2- 172.21.15.12:0/589982753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f375819fd40 0x7f37581999c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f374400b550 tx=0x7f374400b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:52:39.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.443+0000 7f3756000700 1 -- 172.21.15.12:0/589982753 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3744004230 con 0x7f375819fd40 2024-03-23T22:52:39.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.443+0000 7f3756000700 1 -- 172.21.15.12:0/589982753 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f37440043d0 con 0x7f375819fd40 2024-03-23T22:52:39.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.443+0000 7f3756000700 1 -- 172.21.15.12:0/589982753 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3744011b20 con 0x7f375819fd40 2024-03-23T22:52:39.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.443+0000 7f375fe00700 1 -- 172.21.15.12:0/589982753 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37581a6900 con 0x7f375819fd40 2024-03-23T22:52:39.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.443+0000 7f375fe00700 1 -- 172.21.15.12:0/589982753 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37581a6df0 con 0x7f375819fd40 2024-03-23T22:52:39.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.444+0000 7f3756000700 1 -- 172.21.15.12:0/589982753 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f3744011d00 con 0x7f375819fd40 2024-03-23T22:52:39.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.444+0000 7f375fe00700 1 -- 172.21.15.12:0/589982753 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37580637c0 con 0x7f375819fd40 2024-03-23T22:52:39.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.445+0000 7f3756000700 1 --2- 172.21.15.12:0/589982753 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3740039450 0x7f374003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:39.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.445+0000 7f3756000700 1 -- 172.21.15.12:0/589982753 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f374404bb30 con 0x7f375819fd40 2024-03-23T22:52:39.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.445+0000 7f375d400700 1 --2- 172.21.15.12:0/589982753 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3740039450 0x7f374003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:39.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.446+0000 7f375d400700 1 --2- 172.21.15.12:0/589982753 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3740039450 0x7f374003b910 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f374c005e00 tx=0x7f374c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:52:39.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.450+0000 7f3756000700 1 -- 172.21.15.12:0/589982753 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3744020500 con 0x7f375819fd40 2024-03-23T22:52:39.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.676+0000 7f375fe00700 1 -- 172.21.15.12:0/589982753 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f37581a7110 con 0x7f375819fd40 2024-03-23T22:52:39.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.677+0000 7f3756000700 1 -- 172.21.15.12:0/589982753 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7f37440202a0 con 0x7f375819fd40 2024-03-23T22:52:39.427 INFO:teuthology.orchestra.run.smithi012.stdout:# minimal ceph.conf for 90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T22:52:39.427 INFO:teuthology.orchestra.run.smithi012.stdout:[global] 2024-03-23T22:52:39.427 INFO:teuthology.orchestra.run.smithi012.stdout: fsid = 90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T22:52:39.427 INFO:teuthology.orchestra.run.smithi012.stdout: mon_host = [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-03-23T22:52:39.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.681+0000 7f375fe00700 1 -- 172.21.15.12:0/589982753 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3740039450 msgr2=0x7f374003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:39.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.681+0000 7f375fe00700 1 --2- 172.21.15.12:0/589982753 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3740039450 0x7f374003b910 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f374c005e00 tx=0x7f374c0058e0 comp rx=0 tx=0).stop 2024-03-23T22:52:39.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.681+0000 7f375fe00700 1 -- 172.21.15.12:0/589982753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f375819fd40 msgr2=0x7f37581999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:39.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.681+0000 7f375fe00700 1 --2- 172.21.15.12:0/589982753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f375819fd40 0x7f37581999c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f374400b550 tx=0x7f374400b860 comp rx=0 tx=0).stop 2024-03-23T22:52:39.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.681+0000 7f375fe00700 1 -- 172.21.15.12:0/589982753 shutdown_connections 2024-03-23T22:52:39.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.681+0000 7f375fe00700 1 --2- 172.21.15.12:0/589982753 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3740039450 0x7f374003b910 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:39.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.681+0000 7f375fe00700 1 --2- 172.21.15.12:0/589982753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3758102d10 0x7f375819f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:39.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.681+0000 7f375fe00700 1 --2- 172.21.15.12:0/589982753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3758199f00 0x7f375819a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:39.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.681+0000 7f375fe00700 1 --2- 172.21.15.12:0/589982753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f375819fd40 0x7f37581999c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:39.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.681+0000 7f375fe00700 1 -- 172.21.15.12:0/589982753 >> 172.21.15.12:0/589982753 conn(0x7f37580fc540 msgr2=0x7f3758107fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:52:39.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.681+0000 7f375fe00700 1 -- 172.21.15.12:0/589982753 shutdown_connections 2024-03-23T22:52:39.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:38.681+0000 7f375fe00700 1 -- 172.21.15.12:0/589982753 wait complete. 2024-03-23T22:52:39.429 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:38 smithi012 ceph-mon[31695]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:39.429 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:38 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:39.429 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:38 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/589982753' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:39.430 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-23T22:52:39.430 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-23T22:52:39.430 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/ceph.conf 2024-03-23T22:52:39.523 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-23T22:52:39.523 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-23T22:52:39.615 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T22:52:39.615 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.conf 2024-03-23T22:52:39.651 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T22:52:39.651 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-23T22:52:39.721 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-23T22:52:39.721 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.conf 2024-03-23T22:52:39.756 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-23T22:52:39.756 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-23T22:52:39.828 INFO:tasks.cephadm:Adding mgr.x on smithi012 2024-03-23T22:52:39.828 INFO:tasks.cephadm:Adding mgr.y on smithi173 2024-03-23T22:52:39.828 INFO:tasks.cephadm:Adding mgr.z on smithi203 2024-03-23T22:52:39.828 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph orch apply mgr '3;smithi012=x;smithi173=y;smithi203=z' 2024-03-23T22:52:40.110 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.c/config 2024-03-23T22:52:41.250 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:40 smithi203 ceph-mon[36693]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:41.251 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:41.251 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:41.251 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:40 smithi203 ceph-mon[36693]: Reconfiguring mon.b (monmap changed)... 2024-03-23T22:52:41.251 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T22:52:41.251 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-23T22:52:41.251 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:41.251 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:40 smithi203 ceph-mon[36693]: Reconfiguring daemon mon.b on smithi173 2024-03-23T22:52:41.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:40 smithi012 ceph-mon[31695]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:41.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:41.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:41.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:40 smithi012 ceph-mon[31695]: Reconfiguring mon.b (monmap changed)... 2024-03-23T22:52:41.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T22:52:41.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-23T22:52:41.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:41.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:40 smithi012 ceph-mon[31695]: Reconfiguring daemon mon.b on smithi173 2024-03-23T22:52:41.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:40 smithi173 ceph-mon[36116]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:41.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:41.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:41.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:40 smithi173 ceph-mon[36116]: Reconfiguring mon.b (monmap changed)... 2024-03-23T22:52:41.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T22:52:41.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-23T22:52:41.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:41.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:40 smithi173 ceph-mon[36116]: Reconfiguring daemon mon.b on smithi173 2024-03-23T22:52:41.638 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.638+0000 7f89f204e700 1 -- 172.21.15.203:0/3829605009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89ec102740 msgr2=0x7f89ec104b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:41.638 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.638+0000 7f89f204e700 1 --2- 172.21.15.203:0/3829605009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89ec102740 0x7f89ec104b30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f89d4009910 tx=0x7f89d4009c20 comp rx=0 tx=0).stop 2024-03-23T22:52:41.638 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.638+0000 7f89f204e700 1 -- 172.21.15.203:0/3829605009 shutdown_connections 2024-03-23T22:52:41.638 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.638+0000 7f89f204e700 1 --2- 172.21.15.203:0/3829605009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89ec102740 0x7f89ec104b30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:41.638 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.638+0000 7f89f204e700 1 -- 172.21.15.203:0/3829605009 >> 172.21.15.203:0/3829605009 conn(0x7f89ec0fc170 msgr2=0x7f89ec0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:52:41.638 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.638+0000 7f89f204e700 1 -- 172.21.15.203:0/3829605009 shutdown_connections 2024-03-23T22:52:41.638 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.638+0000 7f89f204e700 1 -- 172.21.15.203:0/3829605009 wait complete. 2024-03-23T22:52:41.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.639+0000 7f89f204e700 1 Processor -- start 2024-03-23T22:52:41.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.639+0000 7f89f204e700 1 -- start start 2024-03-23T22:52:41.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.639+0000 7f89f204e700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89ec102740 0x7f89ec1020a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:41.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.639+0000 7f89f204e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89ec1006f0 0x7f89ec100b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:41.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.639+0000 7f89f204e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89ec1010b0 0x7f89ec1a14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:41.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.639+0000 7f89f204e700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ec107c90 con 0x7f89ec102740 2024-03-23T22:52:41.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.639+0000 7f89f204e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ec107b10 con 0x7f89ec1006f0 2024-03-23T22:52:41.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.639+0000 7f89f204e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ec107e10 con 0x7f89ec1010b0 2024-03-23T22:52:41.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.639+0000 7f89ebfff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89ec1010b0 0x7f89ec1a14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:41.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.640+0000 7f89ebfff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89ec1010b0 0x7f89ec1a14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:57796/0 (socket says 172.21.15.203:57796) 2024-03-23T22:52:41.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.640+0000 7f89ebfff700 1 -- 172.21.15.203:0/2944368219 learned_addr learned my addr 172.21.15.203:0/2944368219 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-23T22:52:41.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.640+0000 7f89eaffd700 1 --2- 172.21.15.203:0/2944368219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89ec1006f0 0x7f89ec100b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:41.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.640+0000 7f89eb7fe700 1 --2- 172.21.15.203:0/2944368219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89ec102740 0x7f89ec1020a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:41.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.640+0000 7f89ebfff700 1 -- 172.21.15.203:0/2944368219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89ec1006f0 msgr2=0x7f89ec100b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:41.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.640+0000 7f89ebfff700 1 --2- 172.21.15.203:0/2944368219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89ec1006f0 0x7f89ec100b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:41.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.640+0000 7f89ebfff700 1 -- 172.21.15.203:0/2944368219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89ec102740 msgr2=0x7f89ec1020a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:41.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.640+0000 7f89ebfff700 1 --2- 172.21.15.203:0/2944368219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89ec102740 0x7f89ec1020a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:41.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.640+0000 7f89ebfff700 1 -- 172.21.15.203:0/2944368219 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ec1a1a80 con 0x7f89ec1010b0 2024-03-23T22:52:41.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.640+0000 7f89eb7fe700 1 --2- 172.21.15.203:0/2944368219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89ec102740 0x7f89ec1020a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T22:52:41.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.640+0000 7f89ebfff700 1 --2- 172.21.15.203:0/2944368219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89ec1010b0 0x7f89ec1a14c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f89e400ba50 tx=0x7f89e400be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:52:41.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.640+0000 7f89e8ff9700 1 -- 172.21.15.203:0/2944368219 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e4019070 con 0x7f89ec1010b0 2024-03-23T22:52:41.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.640+0000 7f89e8ff9700 1 -- 172.21.15.203:0/2944368219 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f89e4014440 con 0x7f89ec1010b0 2024-03-23T22:52:41.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.640+0000 7f89f204e700 1 -- 172.21.15.203:0/2944368219 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89ec1a1d70 con 0x7f89ec1010b0 2024-03-23T22:52:41.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.640+0000 7f89e8ff9700 1 -- 172.21.15.203:0/2944368219 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e4007590 con 0x7f89ec1010b0 2024-03-23T22:52:41.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.640+0000 7f89f204e700 1 -- 172.21.15.203:0/2944368219 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89ec1a22d0 con 0x7f89ec1010b0 2024-03-23T22:52:41.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.641+0000 7f89e8ff9700 1 -- 172.21.15.203:0/2944368219 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f89e401c070 con 0x7f89ec1010b0 2024-03-23T22:52:41.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.641+0000 7f89e8ff9700 1 --2- 172.21.15.203:0/2944368219 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f89d8039450 0x7f89d803b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:41.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.642+0000 7f89e8ff9700 1 -- 172.21.15.203:0/2944368219 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f89e404b740 con 0x7f89ec1010b0 2024-03-23T22:52:41.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.642+0000 7f89f204e700 1 -- 172.21.15.203:0/2944368219 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89ec02da30 con 0x7f89ec1010b0 2024-03-23T22:52:41.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.642+0000 7f89eb7fe700 1 --2- 172.21.15.203:0/2944368219 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f89d8039450 0x7f89d803b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:41.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.643+0000 7f89eb7fe700 1 --2- 172.21.15.203:0/2944368219 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f89d8039450 0x7f89d803b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f89d4005330 tx=0x7f89d4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:52:41.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.647+0000 7f89e8ff9700 1 -- 172.21.15.203:0/2944368219 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f89e4014910 con 0x7f89ec1010b0 2024-03-23T22:52:41.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.862+0000 7f89f204e700 1 -- 172.21.15.203:0/2944368219 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi012=x;smithi173=y;smithi203=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f89ec0625d0 con 0x7f89d8039450 2024-03-23T22:52:41.874 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.875+0000 7f89e8ff9700 1 -- 172.21.15.203:0/2944368219 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f89ec0625d0 con 0x7f89d8039450 2024-03-23T22:52:41.875 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mgr update... 2024-03-23T22:52:41.877 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.877+0000 7f89f204e700 1 -- 172.21.15.203:0/2944368219 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f89d8039450 msgr2=0x7f89d803b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:41.877 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.877+0000 7f89f204e700 1 --2- 172.21.15.203:0/2944368219 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f89d8039450 0x7f89d803b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f89d4005330 tx=0x7f89d4005fd0 comp rx=0 tx=0).stop 2024-03-23T22:52:41.877 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.877+0000 7f89f204e700 1 -- 172.21.15.203:0/2944368219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89ec1010b0 msgr2=0x7f89ec1a14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:41.877 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.877+0000 7f89f204e700 1 --2- 172.21.15.203:0/2944368219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89ec1010b0 0x7f89ec1a14c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f89e400ba50 tx=0x7f89e400be10 comp rx=0 tx=0).stop 2024-03-23T22:52:41.877 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.877+0000 7f89f204e700 1 -- 172.21.15.203:0/2944368219 shutdown_connections 2024-03-23T22:52:41.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.877+0000 7f89f204e700 1 --2- 172.21.15.203:0/2944368219 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f89d8039450 0x7f89d803b910 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:41.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.877+0000 7f89f204e700 1 --2- 172.21.15.203:0/2944368219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89ec1010b0 0x7f89ec1a14c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:41.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.877+0000 7f89f204e700 1 --2- 172.21.15.203:0/2944368219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89ec102740 0x7f89ec1020a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:41.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.877+0000 7f89f204e700 1 --2- 172.21.15.203:0/2944368219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89ec1006f0 0x7f89ec100b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:41.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.877+0000 7f89f204e700 1 -- 172.21.15.203:0/2944368219 >> 172.21.15.203:0/2944368219 conn(0x7f89ec0fc170 msgr2=0x7f89ec0fcdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:52:41.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.877+0000 7f89f204e700 1 -- 172.21.15.203:0/2944368219 shutdown_connections 2024-03-23T22:52:41.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:52:41.877+0000 7f89f204e700 1 -- 172.21.15.203:0/2944368219 wait complete. 2024-03-23T22:52:42.460 DEBUG:teuthology.orchestra.run.smithi173:mgr.y> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mgr.y.service 2024-03-23T22:52:42.463 DEBUG:teuthology.orchestra.run.smithi203:mgr.z> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mgr.z.service 2024-03-23T22:52:42.465 INFO:tasks.cephadm:Deploying OSDs... 2024-03-23T22:52:42.465 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-23T22:52:42.465 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/scratch_devs of=/dev/stdout 2024-03-23T22:52:42.525 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-23T22:52:42.525 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_1 2024-03-23T22:52:42.574 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-23T22:52:42.574 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T22:52:42.574 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 634 Links: 1 2024-03-23T22:52:42.574 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T22:52:42.574 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T22:52:42.574 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-03-23 22:51:39.262981463 +0000 2024-03-23T22:52:42.575 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-03-23 22:46:14.580142336 +0000 2024-03-23T22:52:42.575 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-03-23 22:46:14.580142336 +0000 2024-03-23T22:52:42.575 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-03-23 22:46:14.580142336 +0000 2024-03-23T22:52:42.575 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-23T22:52:42.662 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-03-23T22:52:42.662 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-03-23T22:52:42.662 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000376671 s, 1.4 MB/s 2024-03-23T22:52:42.667 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-23T22:52:42.727 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_2 2024-03-23T22:52:42.771 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:42 smithi203 ceph-mon[36693]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:42.771 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:42 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:42.771 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:42 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:42.771 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:42 smithi203 ceph-mon[36693]: Reconfiguring mon.c (monmap changed)... 2024-03-23T22:52:42.771 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:42 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T22:52:42.771 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:42 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-23T22:52:42.771 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:42 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:42.771 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:42 smithi203 ceph-mon[36693]: Reconfiguring daemon mon.c on smithi203 2024-03-23T22:52:42.772 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:42 smithi203 ceph-mon[36693]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi012=x;smithi173=y;smithi203=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:52:42.772 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:42 smithi203 ceph-mon[36693]: Saving service mgr spec with placement smithi012=x;smithi173=y;smithi203=z;count:3 2024-03-23T22:52:42.772 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:42 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:42.780 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-23T22:52:42.780 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T22:52:42.781 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 651 Links: 1 2024-03-23T22:52:42.781 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T22:52:42.781 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T22:52:42.781 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-03-23 22:51:39.343980740 +0000 2024-03-23T22:52:42.781 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-03-23 22:46:15.182136937 +0000 2024-03-23T22:52:42.781 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-03-23 22:46:15.182136937 +0000 2024-03-23T22:52:42.781 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-03-23 22:46:15.182136937 +0000 2024-03-23T22:52:42.781 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-23T22:52:42.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:42 smithi012 ceph-mon[31695]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:42.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:42 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:42.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:42 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:42.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:42 smithi012 ceph-mon[31695]: Reconfiguring mon.c (monmap changed)... 2024-03-23T22:52:42.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:42 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T22:52:42.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:42 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-23T22:52:42.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:42 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:42.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:42 smithi012 ceph-mon[31695]: Reconfiguring daemon mon.c on smithi203 2024-03-23T22:52:42.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:42 smithi012 ceph-mon[31695]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi012=x;smithi173=y;smithi203=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:52:42.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:42 smithi012 ceph-mon[31695]: Saving service mgr spec with placement smithi012=x;smithi173=y;smithi203=z;count:3 2024-03-23T22:52:42.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:42 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:42.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:42 smithi173 ceph-mon[36116]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:42.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:42 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:42.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:42 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:42.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:42 smithi173 ceph-mon[36116]: Reconfiguring mon.c (monmap changed)... 2024-03-23T22:52:42.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:42 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T22:52:42.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:42 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-23T22:52:42.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:42 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:42.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:42 smithi173 ceph-mon[36116]: Reconfiguring daemon mon.c on smithi203 2024-03-23T22:52:42.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:42 smithi173 ceph-mon[36116]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi012=x;smithi173=y;smithi203=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:52:42.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:42 smithi173 ceph-mon[36116]: Saving service mgr spec with placement smithi012=x;smithi173=y;smithi203=z;count:3 2024-03-23T22:52:42.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:42 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:42.878 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-03-23T22:52:42.878 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-03-23T22:52:42.878 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.0004145 s, 1.2 MB/s 2024-03-23T22:52:42.884 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-23T22:52:42.985 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_3 2024-03-23T22:52:43.041 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-23T22:52:43.041 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T22:52:43.042 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 670 Links: 1 2024-03-23T22:52:43.042 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T22:52:43.042 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T22:52:43.042 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-03-23 22:51:39.409980152 +0000 2024-03-23T22:52:43.042 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-03-23 22:46:15.776131611 +0000 2024-03-23T22:52:43.042 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-03-23 22:46:15.776131611 +0000 2024-03-23T22:52:43.042 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-03-23 22:46:15.776131611 +0000 2024-03-23T22:52:43.042 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-23T22:52:43.143 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-03-23T22:52:43.144 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-03-23T22:52:43.144 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000398437 s, 1.3 MB/s 2024-03-23T22:52:43.147 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-23T22:52:43.208 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_4 2024-03-23T22:52:43.259 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-23T22:52:43.260 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T22:52:43.260 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 688 Links: 1 2024-03-23T22:52:43.260 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T22:52:43.260 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T22:52:43.260 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-03-23 22:51:39.470979609 +0000 2024-03-23T22:52:43.260 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-03-23 22:46:16.375126240 +0000 2024-03-23T22:52:43.260 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-03-23 22:46:16.375126240 +0000 2024-03-23T22:52:43.260 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-03-23 22:46:16.375126240 +0000 2024-03-23T22:52:43.260 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-23T22:52:43.354 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-03-23T22:52:43.354 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-03-23T22:52:43.354 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.00038201 s, 1.3 MB/s 2024-03-23T22:52:43.359 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-23T22:52:43.416 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T22:52:43.416 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2024-03-23T22:52:43.442 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-23T22:52:43.443 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_1 2024-03-23T22:52:43.508 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-23T22:52:43.508 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T22:52:43.509 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-03-23T22:52:43.509 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T22:52:43.509 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T22:52:43.509 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-23 22:52:28.010328200 +0000 2024-03-23T22:52:43.509 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-23 22:45:05.968340774 +0000 2024-03-23T22:52:43.509 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-23 22:45:05.968340774 +0000 2024-03-23T22:52:43.510 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-23 22:45:05.968340774 +0000 2024-03-23T22:52:43.510 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-23T22:52:43.582 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-23T22:52:43.582 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-23T22:52:43.582 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000317204 s, 1.6 MB/s 2024-03-23T22:52:43.583 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-23T22:52:43.646 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_2 2024-03-23T22:52:43.708 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-23T22:52:43.709 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T22:52:43.709 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 679 Links: 1 2024-03-23T22:52:43.709 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T22:52:43.709 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T22:52:43.709 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-23 22:52:28.011328177 +0000 2024-03-23T22:52:43.709 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-23 22:45:06.321332845 +0000 2024-03-23T22:52:43.709 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-23 22:45:06.321332845 +0000 2024-03-23T22:52:43.709 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-23 22:45:06.321332845 +0000 2024-03-23T22:52:43.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-23T22:52:43.776 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-23T22:52:43.777 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-23T22:52:43.777 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000205516 s, 2.5 MB/s 2024-03-23T22:52:43.778 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-23T22:52:43.842 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_3 2024-03-23T22:52:43.902 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-23T22:52:43.903 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T22:52:43.903 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 700 Links: 1 2024-03-23T22:52:43.903 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T22:52:43.903 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T22:52:43.903 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-23 22:52:28.011328177 +0000 2024-03-23T22:52:43.903 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-23 22:45:06.663325163 +0000 2024-03-23T22:52:43.903 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-23 22:45:06.663325163 +0000 2024-03-23T22:52:43.903 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-23 22:45:06.663325163 +0000 2024-03-23T22:52:43.903 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-23T22:52:43.970 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-23T22:52:43.970 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-23T22:52:43.970 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000227255 s, 2.3 MB/s 2024-03-23T22:52:43.971 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-23T22:52:44.028 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_4 2024-03-23T22:52:44.085 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-23T22:52:44.085 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T22:52:44.085 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-03-23T22:52:44.085 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T22:52:44.086 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T22:52:44.086 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-23 22:52:28.011328177 +0000 2024-03-23T22:52:44.086 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-23 22:45:06.995317706 +0000 2024-03-23T22:52:44.086 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-23 22:45:06.995317706 +0000 2024-03-23T22:52:44.086 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-23 22:45:06.995317706 +0000 2024-03-23T22:52:44.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-23T22:52:44.151 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-23T22:52:44.151 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-23T22:52:44.151 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000219236 s, 2.3 MB/s 2024-03-23T22:52:44.152 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-23T22:52:44.208 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-23T22:52:44.208 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/scratch_devs of=/dev/stdout 2024-03-23T22:52:44.225 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-23T22:52:44.226 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_1 2024-03-23T22:52:44.280 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-23T22:52:44.280 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T22:52:44.280 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 662 Links: 1 2024-03-23T22:52:44.280 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T22:52:44.280 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T22:52:44.280 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-23 22:52:19.013419028 +0000 2024-03-23T22:52:44.280 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-23 22:45:49.520468439 +0000 2024-03-23T22:52:44.280 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-23 22:45:49.520468439 +0000 2024-03-23T22:52:44.280 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-03-23 22:45:49.520468439 +0000 2024-03-23T22:52:44.281 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-23T22:52:44.347 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-23T22:52:44.347 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-23T22:52:44.347 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000248047 s, 2.1 MB/s 2024-03-23T22:52:44.349 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-23T22:52:44.405 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_2 2024-03-23T22:52:44.461 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-23T22:52:44.461 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T22:52:44.461 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 681 Links: 1 2024-03-23T22:52:44.461 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T22:52:44.461 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T22:52:44.461 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-23 22:52:19.013419028 +0000 2024-03-23T22:52:44.461 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-23 22:45:49.861459854 +0000 2024-03-23T22:52:44.461 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-23 22:45:49.861459854 +0000 2024-03-23T22:52:44.461 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-03-23 22:45:49.861459854 +0000 2024-03-23T22:52:44.462 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-23T22:52:44.527 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-23T22:52:44.527 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-23T22:52:44.527 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000213946 s, 2.4 MB/s 2024-03-23T22:52:44.528 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-23T22:52:44.584 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_3 2024-03-23T22:52:44.640 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-23T22:52:44.640 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T22:52:44.640 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-03-23T22:52:44.640 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T22:52:44.640 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T22:52:44.640 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-23 22:52:19.013419028 +0000 2024-03-23T22:52:44.640 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-23 22:45:50.195451444 +0000 2024-03-23T22:52:44.640 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-23 22:45:50.195451444 +0000 2024-03-23T22:52:44.640 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-03-23 22:45:50.195451444 +0000 2024-03-23T22:52:44.641 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-23T22:52:44.706 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-23T22:52:44.706 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-23T22:52:44.706 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000177423 s, 2.9 MB/s 2024-03-23T22:52:44.707 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-23T22:52:44.763 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_4 2024-03-23T22:52:44.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:44 smithi012 ceph-mon[31695]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:44.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:44 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:44.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:44 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:44.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:44 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:52:44.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:44 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:44.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:44 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:52:44.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:44 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:44.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:44 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T22:52:44.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:44 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-23T22:52:44.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:44 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-23T22:52:44.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:44 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:44.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:44 smithi012 ceph-mon[31695]: Deploying daemon mgr.z on smithi203 2024-03-23T22:52:44.819 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-23T22:52:44.820 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T22:52:44.820 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-03-23T22:52:44.820 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T22:52:44.820 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T22:52:44.820 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-23 22:52:19.014419003 +0000 2024-03-23T22:52:44.820 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-23 22:45:50.525443135 +0000 2024-03-23T22:52:44.820 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-23 22:45:50.525443135 +0000 2024-03-23T22:52:44.820 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-03-23 22:45:50.525443135 +0000 2024-03-23T22:52:44.820 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-23T22:52:44.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:44 smithi173 ceph-mon[36116]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:44.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:44 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:44.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:44 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:44.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:44 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:52:44.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:44 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:44.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:44 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:52:44.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:44 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:44.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:44 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T22:52:44.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:44 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-23T22:52:44.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:44 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-23T22:52:44.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:44 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:44.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:44 smithi173 ceph-mon[36116]: Deploying daemon mgr.z on smithi203 2024-03-23T22:52:44.883 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:44 smithi203 ceph-mon[36693]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:44.883 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:44 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:44.883 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:44 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:44.883 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:44 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:52:44.883 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:44 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:44.883 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:44 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:52:44.883 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:44 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:44.883 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:44 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T22:52:44.884 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:44 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-23T22:52:44.884 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:44 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-23T22:52:44.884 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:44 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:44.884 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:44 smithi203 ceph-mon[36693]: Deploying daemon mgr.z on smithi203 2024-03-23T22:52:44.885 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-23T22:52:44.885 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-23T22:52:44.885 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000211524 s, 2.4 MB/s 2024-03-23T22:52:44.886 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-23T22:52:44.941 INFO:tasks.cephadm:Deploying osd.0 on smithi012 with /dev/vg_nvme/lv_4... 2024-03-23T22:52:44.942 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-23T22:52:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:52:45.658 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:45 smithi203 systemd[1]: Starting Ceph mgr.z for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T22:52:45.945 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:45 smithi203 podman[38549]: 2024-03-23 22:52:45.580426211 +0000 UTC m=+0.017520258 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-23T22:52:45.945 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:45 smithi203 podman[38549]: 2024-03-23 22:52:45.700349226 +0000 UTC m=+0.137443259 container create b6f6dfb7120a8d84226d0f3fa7ff5580ebb692d74e1d9baffa969de3b8028be2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-23T22:52:45.945 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:45 smithi203 podman[38549]: 2024-03-23 22:52:45.919215253 +0000 UTC m=+0.356309299 container init b6f6dfb7120a8d84226d0f3fa7ff5580ebb692d74e1d9baffa969de3b8028be2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-23T22:52:45.945 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:45 smithi203 podman[38549]: 2024-03-23 22:52:45.921690747 +0000 UTC m=+0.358784787 container start b6f6dfb7120a8d84226d0f3fa7ff5580ebb692d74e1d9baffa969de3b8028be2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-23T22:52:45.945 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:45 smithi203 ceph-mgr[38568]: -- 172.21.15.203:0/2631989932 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56314d57e5a0 con 0x56314d552c00 2024-03-23T22:52:46.407 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:46 smithi203 bash[38549]: b6f6dfb7120a8d84226d0f3fa7ff5580ebb692d74e1d9baffa969de3b8028be2 2024-03-23T22:52:46.407 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:46 smithi203 systemd[1]: Started Ceph mgr.z for 90f679ae-e967-11ee-95cd-87774f69a715. 2024-03-23T22:52:46.824 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:46 smithi012 ceph-mon[31695]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:46.824 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:46 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:46.824 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:46 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:46.825 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:46 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:46.825 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:46 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T22:52:46.825 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:46 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-23T22:52:46.825 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:46 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-23T22:52:46.825 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:46 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:46.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:46 smithi173 ceph-mon[36116]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:46.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:46 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:46.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:46 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:46.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:46 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:46.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:46 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T22:52:46.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:46 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-23T22:52:46.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:46 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-23T22:52:46.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:46 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:46.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:46 smithi203 ceph-mon[36693]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:46.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:46 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:46.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:46 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:46.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:46 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:46.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:46 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T22:52:46.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:46 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-23T22:52:46.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:46 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-23T22:52:46.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:46 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:46.908 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:46 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z[38564]: 2024-03-23T22:52:46.480+0000 7fa228041200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-23T22:52:46.908 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:46 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z[38564]: 2024-03-23T22:52:46.573+0000 7fa228041200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-23T22:52:47.379 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:47 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z[38564]: 2024-03-23T22:52:47.219+0000 7fa228041200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-23T22:52:47.379 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:47 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z[38564]: 2024-03-23T22:52:47.314+0000 7fa228041200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-23T22:52:47.652 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:47 smithi203 ceph-mon[36693]: Deploying daemon mgr.y on smithi173 2024-03-23T22:52:47.652 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:47 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z[38564]: 2024-03-23T22:52:47.464+0000 7fa228041200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-23T22:52:47.652 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:47 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z[38564]: 2024-03-23T22:52:47.565+0000 7fa228041200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-23T22:52:47.652 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:47 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z[38564]: 2024-03-23T22:52:47.652+0000 7fa228041200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-23T22:52:47.735 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:47 smithi012 ceph-mon[31695]: Deploying daemon mgr.y on smithi173 2024-03-23T22:52:47.833 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:47 smithi173 ceph-mon[36116]: Deploying daemon mgr.y on smithi173 2024-03-23T22:52:48.118 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:48 smithi173 systemd[1]: Starting Ceph mgr.y for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T22:52:48.212 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:47 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z[38564]: 2024-03-23T22:52:47.922+0000 7fa228041200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-23T22:52:48.212 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:48 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z[38564]: 2024-03-23T22:52:48.031+0000 7fa228041200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-23T22:52:48.212 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:48 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z[38564]: 2024-03-23T22:52:48.118+0000 7fa228041200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-23T22:52:48.212 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:48 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z[38564]: 2024-03-23T22:52:48.212+0000 7fa228041200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-23T22:52:48.532 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-23T22:52:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-23T22:52:48.533 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10+0 records in 2024-03-23T22:52:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:10+0 records out 2024-03-23T22:52:48.533 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0379537 s, 276 MB/s 2024-03-23T22:52:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping successful for: 2024-03-23T22:52:48.562 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:48 smithi173 podman[37485]: 2024-03-23 22:52:48.277298139 +0000 UTC m=+0.107691343 container create 8fa22e2bce06b8119dbc6a585b9e58ac486d06df8b6333fe25e0b0eed935caab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux ) 2024-03-23T22:52:48.562 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:48 smithi173 podman[37485]: 2024-03-23 22:52:48.187317274 +0000 UTC m=+0.017710499 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-23T22:52:48.562 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:48 smithi173 podman[37485]: 2024-03-23 22:52:48.479748961 +0000 UTC m=+0.310142174 container init 8fa22e2bce06b8119dbc6a585b9e58ac486d06df8b6333fe25e0b0eed935caab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-23T22:52:48.562 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:48 smithi173 podman[37485]: 2024-03-23 22:52:48.482307636 +0000 UTC m=+0.312700847 container start 8fa22e2bce06b8119dbc6a585b9e58ac486d06df8b6333fe25e0b0eed935caab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-23T22:52:48.562 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:48 smithi173 ceph-mgr[37504]: -- 172.21.15.173:0/3740257047 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5576e1ce85a0 con 0x5576e1cbc800 2024-03-23T22:52:48.566 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:48 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z[38564]: 2024-03-23T22:52:48.392+0000 7fa228041200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-23T22:52:48.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:48 smithi012 ceph-mon[31695]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:48.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:48 smithi173 ceph-mon[36116]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:48.868 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:48 smithi173 bash[37485]: 8fa22e2bce06b8119dbc6a585b9e58ac486d06df8b6333fe25e0b0eed935caab 2024-03-23T22:52:48.868 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:48 smithi173 systemd[1]: Started Ceph mgr.y for 90f679ae-e967-11ee-95cd-87774f69a715. 2024-03-23T22:52:48.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:48 smithi203 ceph-mon[36693]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:49.368 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:49 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y[37500]: 2024-03-23T22:52:49.030+0000 7f22cec01200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-23T22:52:49.368 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:49 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y[37500]: 2024-03-23T22:52:49.128+0000 7f22cec01200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-23T22:52:49.375 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi012:vg_nvme/lv_4 2024-03-23T22:52:49.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:49 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:49.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:49 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:49.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:49 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:49.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:49 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:49.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:49 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:52:49.658 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:49 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z[38564]: 2024-03-23T22:52:49.396+0000 7fa228041200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-23T22:52:49.658 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:49 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z[38564]: 2024-03-23T22:52:49.547+0000 7fa228041200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-23T22:52:49.836 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:52:49.898 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:49 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y[37500]: 2024-03-23T22:52:49.808+0000 7f22cec01200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-23T22:52:49.898 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:49 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:49.898 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:49 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:49.898 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:49 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:49.898 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:49 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:49.898 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:49 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:52:49.974 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:49 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:49.974 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:49 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:49.974 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:49 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:49.974 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:49 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:49.974 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:49 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:52:50.157 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:49 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z[38564]: 2024-03-23T22:52:49.886+0000 7fa228041200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-23T22:52:50.157 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:49 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z[38564]: 2024-03-23T22:52:49.979+0000 7fa228041200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-23T22:52:50.232 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:49 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y[37500]: 2024-03-23T22:52:49.906+0000 7f22cec01200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-23T22:52:50.232 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:50 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y[37500]: 2024-03-23T22:52:50.048+0000 7f22cec01200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-23T22:52:50.232 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:50 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y[37500]: 2024-03-23T22:52:50.145+0000 7f22cec01200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-23T22:52:50.232 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:50 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y[37500]: 2024-03-23T22:52:50.232+0000 7f22cec01200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-23T22:52:50.651 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:50 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z[38564]: 2024-03-23T22:52:50.178+0000 7fa228041200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-23T22:52:50.651 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:50 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z[38564]: 2024-03-23T22:52:50.360+0000 7fa228041200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-23T22:52:50.791 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:50 smithi173 ceph-mon[36116]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:50.791 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:50 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:50.791 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:50 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:50.791 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:50 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:50.792 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:50 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:50.792 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:50 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:50.792 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:50 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:52:50.792 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:50 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:50.792 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:50 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y[37500]: 2024-03-23T22:52:50.498+0000 7f22cec01200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-23T22:52:50.792 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:50 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y[37500]: 2024-03-23T22:52:50.607+0000 7f22cec01200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-23T22:52:50.792 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:50 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y[37500]: 2024-03-23T22:52:50.695+0000 7f22cec01200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-23T22:52:50.792 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:50 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y[37500]: 2024-03-23T22:52:50.791+0000 7f22cec01200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-23T22:52:50.907 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:50 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z[38564]: 2024-03-23T22:52:50.651+0000 7fa228041200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-23T22:52:50.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:50 smithi203 ceph-mon[36693]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:50.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:50 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:50.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:50 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:50.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:50 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:50.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:50 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:50.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:50 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:50.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:50 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:52:50.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:50 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:50.985 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:50 smithi012 ceph-mon[31695]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:50.985 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:50 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:50.985 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:50 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:50.985 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:50 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:50.985 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:50 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:50.985 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:50 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:50.985 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:50 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:52:50.985 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:50 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:51.118 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:50 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y[37500]: 2024-03-23T22:52:50.967+0000 7f22cec01200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-23T22:52:51.407 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:50 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z[38564]: 2024-03-23T22:52:50.914+0000 7fa228041200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-23T22:52:51.407 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:51 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z[38564]: 2024-03-23T22:52:51.145+0000 7fa228041200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-23T22:52:51.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.808+0000 7fb3a2800700 1 -- 172.21.15.12:0/2657456529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb39c102b80 msgr2=0x7fb39c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:51.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.808+0000 7fb3a2800700 1 --2- 172.21.15.12:0/2657456529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb39c102b80 0x7fb39c109430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb39400b0a0 tx=0x7fb39400b3b0 comp rx=0 tx=0).stop 2024-03-23T22:52:51.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.809+0000 7fb3a2800700 1 -- 172.21.15.12:0/2657456529 shutdown_connections 2024-03-23T22:52:51.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.809+0000 7fb3a2800700 1 --2- 172.21.15.12:0/2657456529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb39c102b80 0x7fb39c109430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:51.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.809+0000 7fb3a2800700 1 --2- 172.21.15.12:0/2657456529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb39c100fe0 0x7fb39c101400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:51.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.809+0000 7fb3a2800700 1 --2- 172.21.15.12:0/2657456529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb39c1021e0 0x7fb39c102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:51.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.809+0000 7fb3a2800700 1 -- 172.21.15.12:0/2657456529 >> 172.21.15.12:0/2657456529 conn(0x7fb39c0fc580 msgr2=0x7fb39c0fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:52:51.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.809+0000 7fb3a2800700 1 -- 172.21.15.12:0/2657456529 shutdown_connections 2024-03-23T22:52:51.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.809+0000 7fb3a2800700 1 -- 172.21.15.12:0/2657456529 wait complete. 2024-03-23T22:52:51.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.810+0000 7fb3a2800700 1 Processor -- start 2024-03-23T22:52:51.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.810+0000 7fb3a2800700 1 -- start start 2024-03-23T22:52:51.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.810+0000 7fb3a2800700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb39c100fe0 0x7fb39c104f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:51.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.810+0000 7fb3a2800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb39c1021e0 0x7fb39c1035c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:51.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.810+0000 7fb3a2800700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb39c102b80 0x7fb39c103b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:51.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.810+0000 7fb3a2800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c10cc20 con 0x7fb39c1021e0 2024-03-23T22:52:51.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.810+0000 7fb3a2800700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c10caa0 con 0x7fb39c102b80 2024-03-23T22:52:51.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.810+0000 7fb3a2800700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c10cda0 con 0x7fb39c100fe0 2024-03-23T22:52:51.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.811+0000 7fb3a1400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb39c100fe0 0x7fb39c104f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:51.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.811+0000 7fb3a1400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb39c100fe0 0x7fb39c104f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45174/0 (socket says 172.21.15.12:45174) 2024-03-23T22:52:51.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.811+0000 7fb3a1400700 1 -- 172.21.15.12:0/929275631 learned_addr learned my addr 172.21.15.12:0/929275631 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:52:51.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.811+0000 7fb3a1e00700 1 --2- 172.21.15.12:0/929275631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb39c102b80 0x7fb39c103b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:51.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.811+0000 7fb3a0a00700 1 --2- 172.21.15.12:0/929275631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb39c1021e0 0x7fb39c1035c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:51.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.811+0000 7fb3a1400700 1 -- 172.21.15.12:0/929275631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb39c102b80 msgr2=0x7fb39c103b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:51.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.811+0000 7fb3a1400700 1 --2- 172.21.15.12:0/929275631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb39c102b80 0x7fb39c103b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:51.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.811+0000 7fb3a1400700 1 -- 172.21.15.12:0/929275631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb39c1021e0 msgr2=0x7fb39c1035c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:52:51.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.811+0000 7fb3a1400700 1 --2- 172.21.15.12:0/929275631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb39c1021e0 0x7fb39c1035c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:52:51.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.811+0000 7fb3a1400700 1 -- 172.21.15.12:0/929275631 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39c1043b0 con 0x7fb39c100fe0 2024-03-23T22:52:51.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.811+0000 7fb3a1e00700 1 --2- 172.21.15.12:0/929275631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb39c102b80 0x7fb39c103b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T22:52:51.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.811+0000 7fb3a0a00700 1 --2- 172.21.15.12:0/929275631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb39c1021e0 0x7fb39c1035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T22:52:51.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.812+0000 7fb3a1400700 1 --2- 172.21.15.12:0/929275631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb39c100fe0 0x7fb39c104f70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb39000ea50 tx=0x7fb39000ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:52:51.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.812+0000 7fb39a000700 1 -- 172.21.15.12:0/929275631 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39000cbd0 con 0x7fb39c100fe0 2024-03-23T22:52:51.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.812+0000 7fb39a000700 1 -- 172.21.15.12:0/929275631 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb39000cd70 con 0x7fb39c100fe0 2024-03-23T22:52:51.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.812+0000 7fb3a2800700 1 -- 172.21.15.12:0/929275631 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb39c1a2430 con 0x7fb39c100fe0 2024-03-23T22:52:51.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.812+0000 7fb3a2800700 1 -- 172.21.15.12:0/929275631 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb39c1a2970 con 0x7fb39c100fe0 2024-03-23T22:52:51.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.812+0000 7fb39a000700 1 -- 172.21.15.12:0/929275631 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3900187c0 con 0x7fb39c100fe0 2024-03-23T22:52:51.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.813+0000 7fb38f400700 1 -- 172.21.15.12:0/929275631 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb39c068e50 con 0x7fb39c100fe0 2024-03-23T22:52:51.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.813+0000 7fb39a000700 1 -- 172.21.15.12:0/929275631 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fb39001f020 con 0x7fb39c100fe0 2024-03-23T22:52:51.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.814+0000 7fb39a000700 1 --2- 172.21.15.12:0/929275631 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb384039200 0x7fb38403b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:52:51.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.814+0000 7fb39a000700 1 -- 172.21.15.12:0/929275631 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb39004c280 con 0x7fb39c100fe0 2024-03-23T22:52:51.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.814+0000 7fb3a0a00700 1 --2- 172.21.15.12:0/929275631 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb384039200 0x7fb38403b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:52:51.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.815+0000 7fb3a0a00700 1 --2- 172.21.15.12:0/929275631 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb384039200 0x7fb38403b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb388009be0 tx=0x7fb388009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:52:51.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:51.819+0000 7fb39a000700 1 -- 172.21.15.12:0/929275631 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb390014230 con 0x7fb39c100fe0 2024-03-23T22:52:51.997 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:51 smithi012 ceph-mon[31695]: Reconfiguring mgr.x (unknown last config time)... 2024-03-23T22:52:51.997 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:51 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T22:52:51.997 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:51 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-23T22:52:51.997 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:51 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:51.997 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:51 smithi012 ceph-mon[31695]: Reconfiguring daemon mgr.x on smithi012 2024-03-23T22:52:52.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:52.055+0000 7fb38f400700 1 -- 172.21.15.12:0/929275631 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fb39c0631a0 con 0x7fb384039200 2024-03-23T22:52:52.057 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:51 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y[37500]: 2024-03-23T22:52:51.929+0000 7f22cec01200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-23T22:52:52.057 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:51 smithi173 ceph-mon[36116]: Reconfiguring mgr.x (unknown last config time)... 2024-03-23T22:52:52.057 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:51 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T22:52:52.058 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:51 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-23T22:52:52.058 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:51 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:52.058 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:51 smithi173 ceph-mon[36116]: Reconfiguring daemon mgr.x on smithi012 2024-03-23T22:52:52.157 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:51 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z[38564]: 2024-03-23T22:52:51.935+0000 7fa228041200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-23T22:52:52.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:51 smithi203 ceph-mon[36693]: Reconfiguring mgr.x (unknown last config time)... 2024-03-23T22:52:52.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:51 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T22:52:52.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:51 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-23T22:52:52.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:51 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:52.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:51 smithi203 ceph-mon[36693]: Reconfiguring daemon mgr.x on smithi012 2024-03-23T22:52:52.368 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:52 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y[37500]: 2024-03-23T22:52:52.057+0000 7f22cec01200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-23T22:52:52.672 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:52 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y[37500]: 2024-03-23T22:52:52.391+0000 7f22cec01200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-23T22:52:52.672 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:52 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y[37500]: 2024-03-23T22:52:52.480+0000 7f22cec01200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-23T22:52:52.672 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:52 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y[37500]: 2024-03-23T22:52:52.672+0000 7f22cec01200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-23T22:52:52.807 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:52 smithi203 ceph-mon[36693]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:52.807 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:52 smithi203 ceph-mon[36693]: from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:52:52.807 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:52 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:52:52.807 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:52 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:52:52.807 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:52 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:52.807 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:52 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z[38564]: 2024-03-23T22:52:52.500+0000 7fa228041200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-23T22:52:52.807 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:52 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z[38564]: 2024-03-23T22:52:52.595+0000 7fa228041200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-23T22:52:52.807 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 22:52:52 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z[38564]: 2024-03-23T22:52:52.807+0000 7fa228041200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-23T22:52:52.980 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:52 smithi012 ceph-mon[31695]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:52.981 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:52 smithi012 ceph-mon[31695]: from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:52:52.981 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:52 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:52:52.981 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:52 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:52:52.981 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:52 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:53.118 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:52 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y[37500]: 2024-03-23T22:52:52.857+0000 7f22cec01200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-23T22:52:53.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:52 smithi173 ceph-mon[36116]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:53.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:52 smithi173 ceph-mon[36116]: from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:52:53.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:52 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:52:53.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:52 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:52:53.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:52 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:53.398 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:53 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y[37500]: 2024-03-23T22:52:53.136+0000 7f22cec01200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-23T22:52:53.398 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:53 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y[37500]: 2024-03-23T22:52:53.398+0000 7f22cec01200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-23T22:52:53.714 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:53 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y[37500]: 2024-03-23T22:52:53.630+0000 7f22cec01200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-23T22:52:53.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:53.733+0000 7fb39a000700 1 -- 172.21.15.12:0/929275631 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mgrmap(e 14) v1 ==== 91698+0+0 (secure 0 0 0) 0x7fb390018cf0 con 0x7fb39c100fe0 2024-03-23T22:52:53.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:53 smithi012 ceph-mon[31695]: Standby manager daemon z started 2024-03-23T22:52:53.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:53 smithi012 ceph-mon[31695]: from='mgr.? 172.21.15.203:0/794529038' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-23T22:52:53.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:53 smithi012 ceph-mon[31695]: from='mgr.? 172.21.15.203:0/794529038' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-23T22:52:53.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:53 smithi012 ceph-mon[31695]: from='mgr.? 172.21.15.203:0/794529038' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-23T22:52:53.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:53 smithi012 ceph-mon[31695]: from='mgr.? 172.21.15.203:0/794529038' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-23T22:52:53.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:53 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:53.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:53 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:53.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:53 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:52:53.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:53 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:53.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:53 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:52:53.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:53 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:53.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:53 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:54.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:53 smithi173 ceph-mon[36116]: Standby manager daemon z started 2024-03-23T22:52:54.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:53 smithi173 ceph-mon[36116]: from='mgr.? 172.21.15.203:0/794529038' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-23T22:52:54.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:53 smithi173 ceph-mon[36116]: from='mgr.? 172.21.15.203:0/794529038' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-23T22:52:54.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:53 smithi173 ceph-mon[36116]: from='mgr.? 172.21.15.203:0/794529038' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-23T22:52:54.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:53 smithi173 ceph-mon[36116]: from='mgr.? 172.21.15.203:0/794529038' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-23T22:52:54.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:53 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:54.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:53 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:54.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:53 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:52:54.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:53 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:54.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:53 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:52:54.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:53 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:54.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:53 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:54.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:53 smithi203 ceph-mon[36693]: Standby manager daemon z started 2024-03-23T22:52:54.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:53 smithi203 ceph-mon[36693]: from='mgr.? 172.21.15.203:0/794529038' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-23T22:52:54.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:53 smithi203 ceph-mon[36693]: from='mgr.? 172.21.15.203:0/794529038' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-23T22:52:54.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:53 smithi203 ceph-mon[36693]: from='mgr.? 172.21.15.203:0/794529038' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-23T22:52:54.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:53 smithi203 ceph-mon[36693]: from='mgr.? 172.21.15.203:0/794529038' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-23T22:52:54.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:53 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:54.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:53 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:54.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:53 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:52:54.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:53 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:52:54.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:53 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:52:54.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:53 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:54.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:53 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:52:54.730 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:54 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y[37500]: 2024-03-23T22:52:54.420+0000 7f22cec01200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-23T22:52:54.933 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:54 smithi012 ceph-mon[31695]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:54.933 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:54 smithi012 ceph-mon[31695]: mgrmap e14: x(active, since 90s), standbys: z 2024-03-23T22:52:54.933 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:54 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-23T22:52:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:54 smithi173 ceph-mon[36116]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:54 smithi173 ceph-mon[36116]: mgrmap e14: x(active, since 90s), standbys: z 2024-03-23T22:52:54.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:54 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-23T22:52:55.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:54 smithi203 ceph-mon[36693]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:55.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:54 smithi203 ceph-mon[36693]: mgrmap e14: x(active, since 90s), standbys: z 2024-03-23T22:52:55.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:54 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-23T22:52:55.293 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:54 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y[37500]: 2024-03-23T22:52:54.987+0000 7f22cec01200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-23T22:52:55.294 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:55 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y[37500]: 2024-03-23T22:52:55.084+0000 7f22cec01200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-23T22:52:55.555 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 22:52:55 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y[37500]: 2024-03-23T22:52:55.294+0000 7f22cec01200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-23T22:52:55.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:52:55.755+0000 7fb39a000700 1 -- 172.21.15.12:0/929275631 <== mon.1 v2:172.21.15.203:3300/0 8 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7fb390016330 con 0x7fb39c100fe0 2024-03-23T22:52:55.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:55 smithi173 ceph-mon[36116]: Standby manager daemon y started 2024-03-23T22:52:55.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:55 smithi173 ceph-mon[36116]: from='mgr.? 172.21.15.173:0/3772266993' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-23T22:52:55.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:55 smithi173 ceph-mon[36116]: from='mgr.? 172.21.15.173:0/3772266993' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-23T22:52:55.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:55 smithi173 ceph-mon[36116]: from='mgr.? 172.21.15.173:0/3772266993' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-23T22:52:55.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:55 smithi173 ceph-mon[36116]: from='mgr.? 172.21.15.173:0/3772266993' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-23T22:52:56.014 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:55 smithi012 ceph-mon[31695]: Standby manager daemon y started 2024-03-23T22:52:56.014 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:55 smithi012 ceph-mon[31695]: from='mgr.? 172.21.15.173:0/3772266993' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-23T22:52:56.014 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:55 smithi012 ceph-mon[31695]: from='mgr.? 172.21.15.173:0/3772266993' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-23T22:52:56.014 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:55 smithi012 ceph-mon[31695]: from='mgr.? 172.21.15.173:0/3772266993' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-23T22:52:56.014 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:55 smithi012 ceph-mon[31695]: from='mgr.? 172.21.15.173:0/3772266993' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-23T22:52:56.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:55 smithi203 ceph-mon[36693]: Standby manager daemon y started 2024-03-23T22:52:56.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:55 smithi203 ceph-mon[36693]: from='mgr.? 172.21.15.173:0/3772266993' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-23T22:52:56.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:55 smithi203 ceph-mon[36693]: from='mgr.? 172.21.15.173:0/3772266993' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-23T22:52:56.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:55 smithi203 ceph-mon[36693]: from='mgr.? 172.21.15.173:0/3772266993' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-23T22:52:56.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:55 smithi203 ceph-mon[36693]: from='mgr.? 172.21.15.173:0/3772266993' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-23T22:52:57.013 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:56 smithi012 ceph-mon[31695]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:57.014 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:56 smithi012 ceph-mon[31695]: mgrmap e15: x(active, since 92s), standbys: z, y 2024-03-23T22:52:57.014 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:56 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-23T22:52:57.014 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:56 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1419918158' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b70875d7-a23d-4251-9e8b-1a16d4edc2a4"}]: dispatch 2024-03-23T22:52:57.014 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:56 smithi012 ceph-mon[31695]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b70875d7-a23d-4251-9e8b-1a16d4edc2a4"}]: dispatch 2024-03-23T22:52:57.014 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:56 smithi012 ceph-mon[31695]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b70875d7-a23d-4251-9e8b-1a16d4edc2a4"}]': finished 2024-03-23T22:52:57.014 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:56 smithi012 ceph-mon[31695]: osdmap e5: 1 total, 0 up, 1 in 2024-03-23T22:52:57.014 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:56 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T22:52:57.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:56 smithi173 ceph-mon[36116]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:57.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:56 smithi173 ceph-mon[36116]: mgrmap e15: x(active, since 92s), standbys: z, y 2024-03-23T22:52:57.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:56 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-23T22:52:57.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:56 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1419918158' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b70875d7-a23d-4251-9e8b-1a16d4edc2a4"}]: dispatch 2024-03-23T22:52:57.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:56 smithi173 ceph-mon[36116]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b70875d7-a23d-4251-9e8b-1a16d4edc2a4"}]: dispatch 2024-03-23T22:52:57.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:56 smithi173 ceph-mon[36116]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b70875d7-a23d-4251-9e8b-1a16d4edc2a4"}]': finished 2024-03-23T22:52:57.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:56 smithi173 ceph-mon[36116]: osdmap e5: 1 total, 0 up, 1 in 2024-03-23T22:52:57.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:56 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T22:52:57.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:56 smithi203 ceph-mon[36693]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:57.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:56 smithi203 ceph-mon[36693]: mgrmap e15: x(active, since 92s), standbys: z, y 2024-03-23T22:52:57.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:56 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-23T22:52:57.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:56 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1419918158' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b70875d7-a23d-4251-9e8b-1a16d4edc2a4"}]: dispatch 2024-03-23T22:52:57.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:56 smithi203 ceph-mon[36693]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b70875d7-a23d-4251-9e8b-1a16d4edc2a4"}]: dispatch 2024-03-23T22:52:57.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:56 smithi203 ceph-mon[36693]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b70875d7-a23d-4251-9e8b-1a16d4edc2a4"}]': finished 2024-03-23T22:52:57.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:56 smithi203 ceph-mon[36693]: osdmap e5: 1 total, 0 up, 1 in 2024-03-23T22:52:57.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:56 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T22:52:58.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:57 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1716304134' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:52:58.058 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:57 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1716304134' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:52:58.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:57 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1716304134' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:52:59.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:52:58 smithi012 ceph-mon[31695]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:59.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:52:58 smithi173 ceph-mon[36116]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:52:59.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:52:58 smithi203 ceph-mon[36693]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:01.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:00 smithi012 ceph-mon[31695]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:01.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:00 smithi173 ceph-mon[36116]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:01.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:00 smithi203 ceph-mon[36693]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:02.980 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:02 smithi012 ceph-mon[31695]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:03.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:02 smithi173 ceph-mon[36116]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:03.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:02 smithi203 ceph-mon[36693]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:05.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:04 smithi012 ceph-mon[31695]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:05.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:04 smithi173 ceph-mon[36116]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:05.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:04 smithi203 ceph-mon[36693]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:06.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:05 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-23T22:53:06.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:05 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:06.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:05 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-23T22:53:06.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:05 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:06.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:05 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-23T22:53:06.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:05 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:06.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:06 smithi173 ceph-mon[36116]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:06.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:06 smithi173 ceph-mon[36116]: Deploying daemon osd.0 on smithi012 2024-03-23T22:53:07.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:06 smithi012 ceph-mon[31695]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:07.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:06 smithi012 ceph-mon[31695]: Deploying daemon osd.0 on smithi012 2024-03-23T22:53:07.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:06 smithi203 ceph-mon[36693]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:07.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:06 smithi203 ceph-mon[36693]: Deploying daemon osd.0 on smithi012 2024-03-23T22:53:09.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:08 smithi012 ceph-mon[31695]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:09.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:08 smithi203 ceph-mon[36693]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:09.282 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:08 smithi173 ceph-mon[36116]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:10.883 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:10 smithi012 ceph-mon[31695]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:11.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:10 smithi173 ceph-mon[36116]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:11.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:10 smithi203 ceph-mon[36693]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:12.971 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:12 smithi012 ceph-mon[31695]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:13.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:12 smithi173 ceph-mon[36116]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:13.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:12 smithi203 ceph-mon[36693]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:14.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:14 smithi012 ceph-mon[31695]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:14.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:14.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:14.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:53:14.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:14 smithi173 ceph-mon[36116]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:14.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:14 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:14.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:14 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:14.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:14 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:53:14.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:14 smithi203 ceph-mon[36693]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:14.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:14 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:14.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:14 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:14.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:14 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:53:16.666 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:16 smithi012 ceph-mon[31695]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:16.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:16 smithi173 ceph-mon[36116]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:16.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:16 smithi203 ceph-mon[36693]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:18.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:17 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:18.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:17 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:18.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:17 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:18.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:17 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:53:18.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:17 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:18.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:17 smithi012 ceph-mon[31695]: from='osd.0 [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-23T22:53:18.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:17 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:18.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:17 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:18.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:17 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:18.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:17 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:53:18.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:17 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:18.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:17 smithi173 ceph-mon[36116]: from='osd.0 [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-23T22:53:18.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:17 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:18.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:17 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:18.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:17 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:18.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:17 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:53:18.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:17 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:18.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:17 smithi203 ceph-mon[36693]: from='osd.0 [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-23T22:53:18.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:53:18.305+0000 7fb39a000700 1 -- 172.21.15.12:0/929275631 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb39c0631a0 con 0x7fb384039200 2024-03-23T22:53:18.305 INFO:teuthology.orchestra.run.smithi012.stdout:Created osd(s) 0 on host 'smithi012' 2024-03-23T22:53:18.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:53:18.308+0000 7fb38f400700 1 -- 172.21.15.12:0/929275631 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb384039200 msgr2=0x7fb38403b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:53:18.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:53:18.308+0000 7fb38f400700 1 --2- 172.21.15.12:0/929275631 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb384039200 0x7fb38403b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb388009be0 tx=0x7fb388009260 comp rx=0 tx=0).stop 2024-03-23T22:53:18.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:53:18.308+0000 7fb38f400700 1 -- 172.21.15.12:0/929275631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb39c100fe0 msgr2=0x7fb39c104f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:53:18.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:53:18.308+0000 7fb38f400700 1 --2- 172.21.15.12:0/929275631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb39c100fe0 0x7fb39c104f70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb39000ea50 tx=0x7fb39000ed60 comp rx=0 tx=0).stop 2024-03-23T22:53:18.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:53:18.309+0000 7fb38f400700 1 -- 172.21.15.12:0/929275631 shutdown_connections 2024-03-23T22:53:18.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:53:18.309+0000 7fb38f400700 1 --2- 172.21.15.12:0/929275631 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb384039200 0x7fb38403b6c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:53:18.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:53:18.309+0000 7fb38f400700 1 --2- 172.21.15.12:0/929275631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb39c100fe0 0x7fb39c104f70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:53:18.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:53:18.309+0000 7fb38f400700 1 --2- 172.21.15.12:0/929275631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb39c102b80 0x7fb39c103b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:53:18.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:53:18.309+0000 7fb38f400700 1 --2- 172.21.15.12:0/929275631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb39c1021e0 0x7fb39c1035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:53:18.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:53:18.309+0000 7fb38f400700 1 -- 172.21.15.12:0/929275631 >> 172.21.15.12:0/929275631 conn(0x7fb39c0fc580 msgr2=0x7fb39c0fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:53:18.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:53:18.309+0000 7fb38f400700 1 -- 172.21.15.12:0/929275631 shutdown_connections 2024-03-23T22:53:18.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:53:18.309+0000 7fb38f400700 1 -- 172.21.15.12:0/929275631 wait complete. 2024-03-23T22:53:19.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:18 smithi012 ceph-mon[31695]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:19.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:18 smithi012 ceph-mon[31695]: from='osd.0 [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-23T22:53:19.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:18 smithi012 ceph-mon[31695]: osdmap e6: 1 total, 0 up, 1 in 2024-03-23T22:53:19.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:18 smithi012 ceph-mon[31695]: from='osd.0 [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-23T22:53:19.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:18 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T22:53:19.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:18 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:19.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:18 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:19.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:18 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:53:19.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:18 smithi173 ceph-mon[36116]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:19.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:18 smithi173 ceph-mon[36116]: from='osd.0 [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-23T22:53:19.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:18 smithi173 ceph-mon[36116]: osdmap e6: 1 total, 0 up, 1 in 2024-03-23T22:53:19.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:18 smithi173 ceph-mon[36116]: from='osd.0 [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-23T22:53:19.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:18 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T22:53:19.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:18 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:19.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:18 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:19.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:18 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:53:19.122 DEBUG:teuthology.orchestra.run.smithi012:osd.0> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.0.service 2024-03-23T22:53:19.127 INFO:tasks.cephadm:Deploying osd.1 on smithi173 with /dev/vg_nvme/lv_4... 2024-03-23T22:53:19.127 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-23T22:53:19.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:18 smithi203 ceph-mon[36693]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:19.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:18 smithi203 ceph-mon[36693]: from='osd.0 [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-23T22:53:19.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:18 smithi203 ceph-mon[36693]: osdmap e6: 1 total, 0 up, 1 in 2024-03-23T22:53:19.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:18 smithi203 ceph-mon[36693]: from='osd.0 [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-23T22:53:19.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:18 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T22:53:19.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:18 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:19.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:18 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:19.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:18 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:53:19.368 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.b/config 2024-03-23T22:53:19.696 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 23 22:53:19 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-0[46564]: 2024-03-23T22:53:19.505+0000 7fc80d800700 -1 osd.0 0 waiting for initial osdmap 2024-03-23T22:53:19.696 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 23 22:53:19 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-0[46564]: 2024-03-23T22:53:19.523+0000 7fc803600700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T22:53:20.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:19 smithi012 ceph-mon[31695]: from='osd.0 [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-23T22:53:20.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:19 smithi012 ceph-mon[31695]: osdmap e7: 1 total, 0 up, 1 in 2024-03-23T22:53:20.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:19 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T22:53:20.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:19 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T22:53:20.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:19 smithi012 ceph-mon[31695]: from='osd.0 [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611]' entity='osd.0' 2024-03-23T22:53:20.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:19 smithi173 ceph-mon[36116]: from='osd.0 [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-23T22:53:20.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:19 smithi173 ceph-mon[36116]: osdmap e7: 1 total, 0 up, 1 in 2024-03-23T22:53:20.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:19 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T22:53:20.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:19 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T22:53:20.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:19 smithi173 ceph-mon[36116]: from='osd.0 [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611]' entity='osd.0' 2024-03-23T22:53:20.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:19 smithi203 ceph-mon[36693]: from='osd.0 [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-23T22:53:20.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:19 smithi203 ceph-mon[36693]: osdmap e7: 1 total, 0 up, 1 in 2024-03-23T22:53:20.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T22:53:20.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T22:53:20.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:19 smithi203 ceph-mon[36693]: from='osd.0 [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611]' entity='osd.0' 2024-03-23T22:53:21.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:20 smithi173 ceph-mon[36116]: purged_snaps scrub starts 2024-03-23T22:53:21.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:20 smithi173 ceph-mon[36116]: purged_snaps scrub ok 2024-03-23T22:53:21.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:20 smithi173 ceph-mon[36116]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:21.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:20 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T22:53:21.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:20 smithi173 ceph-mon[36116]: osd.0 [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611] boot 2024-03-23T22:53:21.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:20 smithi173 ceph-mon[36116]: osdmap e8: 1 total, 1 up, 1 in 2024-03-23T22:53:21.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:20 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T22:53:21.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:20 smithi203 ceph-mon[36693]: purged_snaps scrub starts 2024-03-23T22:53:21.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:20 smithi203 ceph-mon[36693]: purged_snaps scrub ok 2024-03-23T22:53:21.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:20 smithi203 ceph-mon[36693]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:21.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:20 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T22:53:21.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:20 smithi203 ceph-mon[36693]: osd.0 [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611] boot 2024-03-23T22:53:21.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:20 smithi203 ceph-mon[36693]: osdmap e8: 1 total, 1 up, 1 in 2024-03-23T22:53:21.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:20 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T22:53:21.181 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-23T22:53:21.181 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-23T22:53:21.181 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-03-23T22:53:21.181 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-03-23T22:53:21.181 INFO:teuthology.orchestra.run.smithi173.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111949 s, 937 MB/s 2024-03-23T22:53:21.181 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 2024-03-23T22:53:21.181 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-03-23T22:53:21.289 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:20 smithi012 ceph-mon[31695]: purged_snaps scrub starts 2024-03-23T22:53:21.289 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:20 smithi012 ceph-mon[31695]: purged_snaps scrub ok 2024-03-23T22:53:21.289 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:20 smithi012 ceph-mon[31695]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T22:53:21.289 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:20 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T22:53:21.289 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:20 smithi012 ceph-mon[31695]: osd.0 [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611] boot 2024-03-23T22:53:21.289 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:20 smithi012 ceph-mon[31695]: osdmap e8: 1 total, 1 up, 1 in 2024-03-23T22:53:21.289 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:20 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T22:53:21.813 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi173:vg_nvme/lv_4 2024-03-23T22:53:22.059 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.b/config 2024-03-23T22:53:23.069 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:22 smithi173 ceph-mon[36116]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:23.069 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:22 smithi173 ceph-mon[36116]: osdmap e9: 1 total, 1 up, 1 in 2024-03-23T22:53:23.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:22 smithi203 ceph-mon[36693]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:23.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:22 smithi203 ceph-mon[36693]: osdmap e9: 1 total, 1 up, 1 in 2024-03-23T22:53:23.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:22 smithi012 ceph-mon[31695]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:23.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:22 smithi012 ceph-mon[31695]: osdmap e9: 1 total, 1 up, 1 in 2024-03-23T22:53:23.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.364+0000 7f2cfcbd5700 1 -- 172.21.15.173:0/3934753524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cf81033c0 msgr2=0x7f2cf8103820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:53:23.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.364+0000 7f2cfcbd5700 1 --2- 172.21.15.173:0/3934753524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cf81033c0 0x7f2cf8103820 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2ce80099c0 tx=0x7f2ce8009cd0 comp rx=0 tx=0).stop 2024-03-23T22:53:23.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.365+0000 7f2cfcbd5700 1 -- 172.21.15.173:0/3934753524 shutdown_connections 2024-03-23T22:53:23.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.365+0000 7f2cfcbd5700 1 --2- 172.21.15.173:0/3934753524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf81021c0 0x7f2cf81025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:53:23.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.365+0000 7f2cfcbd5700 1 --2- 172.21.15.173:0/3934753524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cf8103d60 0x7f2cf810a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:53:23.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.365+0000 7f2cfcbd5700 1 --2- 172.21.15.173:0/3934753524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cf81033c0 0x7f2cf8103820 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:53:23.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.365+0000 7f2cfcbd5700 1 -- 172.21.15.173:0/3934753524 >> 172.21.15.173:0/3934753524 conn(0x7f2cf80fd7a0 msgr2=0x7f2cf80ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:53:23.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.365+0000 7f2cfcbd5700 1 -- 172.21.15.173:0/3934753524 shutdown_connections 2024-03-23T22:53:23.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.365+0000 7f2cfcbd5700 1 -- 172.21.15.173:0/3934753524 wait complete. 2024-03-23T22:53:23.367 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.365+0000 7f2cfcbd5700 1 Processor -- start 2024-03-23T22:53:23.367 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.366+0000 7f2cfcbd5700 1 -- start start 2024-03-23T22:53:23.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.366+0000 7f2cfcbd5700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cf81021c0 0x7f2cf819eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:53:23.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.366+0000 7f2cfcbd5700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cf81033c0 0x7f2cf819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:53:23.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.366+0000 7f2cfcbd5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf8103d60 0x7f2cf81a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:53:23.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.366+0000 7f2cfcbd5700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf810dea0 con 0x7f2cf81021c0 2024-03-23T22:53:23.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.366+0000 7f2cfcbd5700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf810dd20 con 0x7f2cf81033c0 2024-03-23T22:53:23.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.366+0000 7f2cfcbd5700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf810e020 con 0x7f2cf8103d60 2024-03-23T22:53:23.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.366+0000 7f2cf5d9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cf81033c0 0x7f2cf819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:53:23.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.366+0000 7f2cf5d9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cf81033c0 0x7f2cf819f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:35362/0 (socket says 172.21.15.173:35362) 2024-03-23T22:53:23.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.366+0000 7f2cf5d9b700 1 -- 172.21.15.173:0/2084831145 learned_addr learned my addr 172.21.15.173:0/2084831145 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-23T22:53:23.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.366+0000 7f2cf659c700 1 --2- 172.21.15.173:0/2084831145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cf81021c0 0x7f2cf819eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:53:23.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.366+0000 7f2cf6d9d700 1 --2- 172.21.15.173:0/2084831145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf8103d60 0x7f2cf81a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:53:23.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.366+0000 7f2cf5d9b700 1 -- 172.21.15.173:0/2084831145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf8103d60 msgr2=0x7f2cf81a6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:53:23.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.366+0000 7f2cf5d9b700 1 --2- 172.21.15.173:0/2084831145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf8103d60 0x7f2cf81a6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:53:23.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.366+0000 7f2cf5d9b700 1 -- 172.21.15.173:0/2084831145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cf81021c0 msgr2=0x7f2cf819eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:53:23.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.366+0000 7f2cf5d9b700 1 --2- 172.21.15.173:0/2084831145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cf81021c0 0x7f2cf819eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:53:23.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.366+0000 7f2cf5d9b700 1 -- 172.21.15.173:0/2084831145 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf81a67a0 con 0x7f2cf81033c0 2024-03-23T22:53:23.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.366+0000 7f2cf5d9b700 1 --2- 172.21.15.173:0/2084831145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cf81033c0 0x7f2cf819f080 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8003890 tx=0x7f2ce8003730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:53:23.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.367+0000 7f2ce77fe700 1 -- 172.21.15.173:0/2084831145 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce8027070 con 0x7f2cf81033c0 2024-03-23T22:53:23.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.367+0000 7f2ce77fe700 1 -- 172.21.15.173:0/2084831145 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ce8022470 con 0x7f2cf81033c0 2024-03-23T22:53:23.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.367+0000 7f2cfcbd5700 1 -- 172.21.15.173:0/2084831145 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cf81a6a30 con 0x7f2cf81033c0 2024-03-23T22:53:23.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.367+0000 7f2ce77fe700 1 -- 172.21.15.173:0/2084831145 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce800f6a0 con 0x7f2cf81033c0 2024-03-23T22:53:23.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.367+0000 7f2cfcbd5700 1 -- 172.21.15.173:0/2084831145 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cf81a6f60 con 0x7f2cf81033c0 2024-03-23T22:53:23.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.368+0000 7f2cfcbd5700 1 -- 172.21.15.173:0/2084831145 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cf8068e50 con 0x7f2cf81033c0 2024-03-23T22:53:23.370 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.369+0000 7f2ce77fe700 1 -- 172.21.15.173:0/2084831145 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f2ce802a070 con 0x7f2cf81033c0 2024-03-23T22:53:23.370 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.369+0000 7f2ce77fe700 1 --2- 172.21.15.173:0/2084831145 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ce00a2880 0x7f2ce00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:53:23.370 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.369+0000 7f2ce77fe700 1 -- 172.21.15.173:0/2084831145 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f2ce80cded0 con 0x7f2cf81033c0 2024-03-23T22:53:23.371 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.369+0000 7f2cf659c700 1 --2- 172.21.15.173:0/2084831145 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ce00a2880 0x7f2ce00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:53:23.371 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.370+0000 7f2cf659c700 1 --2- 172.21.15.173:0/2084831145 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ce00a2880 0x7f2ce00a4d40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2cec006fd0 tx=0x7f2cec008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:53:23.373 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.374+0000 7f2ce77fe700 1 -- 172.21.15.173:0/2084831145 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2ce8013070 con 0x7f2cf81033c0 2024-03-23T22:53:23.579 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:23.580+0000 7f2cfcbd5700 1 -- 172.21.15.173:0/2084831145 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f2cf80625d0 con 0x7f2ce00a2880 2024-03-23T22:53:24.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:23 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:53:24.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:23 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:53:24.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:23 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:24.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:23 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:53:24.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:23 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:53:24.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:23 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:24.268 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:23 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:53:24.268 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:23 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:53:24.268 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:23 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:25.070 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:24 smithi173 ceph-mon[36116]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:25.070 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:24 smithi173 ceph-mon[36116]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:53:25.070 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:24 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:25.070 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:24 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:25.070 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:24 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:53:25.070 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:24 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:25.070 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:24 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:25.071 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:24 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:53:25.071 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:24 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:25.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:24 smithi203 ceph-mon[36693]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:25.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:24 smithi203 ceph-mon[36693]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:53:25.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:25.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:25.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:53:25.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:25.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:25.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:53:25.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:25.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:24 smithi012 ceph-mon[31695]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:25.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:24 smithi012 ceph-mon[31695]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:53:25.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:25.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:25.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:53:25.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:25.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:25.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:53:25.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:26.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:25 smithi173 ceph-mon[36116]: Detected new or changed devices on smithi012 2024-03-23T22:53:26.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:25 smithi173 ceph-mon[36116]: Adjusting osd_memory_target on smithi012 to 17095M 2024-03-23T22:53:26.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:25 smithi203 ceph-mon[36693]: Detected new or changed devices on smithi012 2024-03-23T22:53:26.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:25 smithi203 ceph-mon[36693]: Adjusting osd_memory_target on smithi012 to 17095M 2024-03-23T22:53:26.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:25 smithi012 ceph-mon[31695]: Detected new or changed devices on smithi012 2024-03-23T22:53:26.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:25 smithi012 ceph-mon[31695]: Adjusting osd_memory_target on smithi012 to 17095M 2024-03-23T22:53:27.008 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:26 smithi012 ceph-mon[31695]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:27.008 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:26 smithi012 ceph-mon[31695]: from='client.? 172.21.15.173:0/957362397' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "416e87af-db5d-41ce-84d9-731e3e07e3c1"}]: dispatch 2024-03-23T22:53:27.008 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:26 smithi012 ceph-mon[31695]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "416e87af-db5d-41ce-84d9-731e3e07e3c1"}]: dispatch 2024-03-23T22:53:27.008 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:26 smithi012 ceph-mon[31695]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "416e87af-db5d-41ce-84d9-731e3e07e3c1"}]': finished 2024-03-23T22:53:27.008 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:26 smithi012 ceph-mon[31695]: osdmap e10: 2 total, 1 up, 2 in 2024-03-23T22:53:27.008 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:26 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T22:53:27.008 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:26 smithi012 ceph-mon[31695]: from='client.? 172.21.15.173:0/3588544451' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:53:27.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:26 smithi173 ceph-mon[36116]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:27.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:26 smithi173 ceph-mon[36116]: from='client.? 172.21.15.173:0/957362397' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "416e87af-db5d-41ce-84d9-731e3e07e3c1"}]: dispatch 2024-03-23T22:53:27.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:26 smithi173 ceph-mon[36116]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "416e87af-db5d-41ce-84d9-731e3e07e3c1"}]: dispatch 2024-03-23T22:53:27.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:26 smithi173 ceph-mon[36116]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "416e87af-db5d-41ce-84d9-731e3e07e3c1"}]': finished 2024-03-23T22:53:27.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:26 smithi173 ceph-mon[36116]: osdmap e10: 2 total, 1 up, 2 in 2024-03-23T22:53:27.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:26 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T22:53:27.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:26 smithi173 ceph-mon[36116]: from='client.? 172.21.15.173:0/3588544451' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:53:27.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:26 smithi203 ceph-mon[36693]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:27.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:26 smithi203 ceph-mon[36693]: from='client.? 172.21.15.173:0/957362397' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "416e87af-db5d-41ce-84d9-731e3e07e3c1"}]: dispatch 2024-03-23T22:53:27.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:26 smithi203 ceph-mon[36693]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "416e87af-db5d-41ce-84d9-731e3e07e3c1"}]: dispatch 2024-03-23T22:53:27.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:26 smithi203 ceph-mon[36693]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "416e87af-db5d-41ce-84d9-731e3e07e3c1"}]': finished 2024-03-23T22:53:27.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:26 smithi203 ceph-mon[36693]: osdmap e10: 2 total, 1 up, 2 in 2024-03-23T22:53:27.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:26 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T22:53:27.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:26 smithi203 ceph-mon[36693]: from='client.? 172.21.15.173:0/3588544451' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:53:29.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:28 smithi173 ceph-mon[36116]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:29.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:28 smithi203 ceph-mon[36693]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:29.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:28 smithi012 ceph-mon[31695]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:31.107 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:30 smithi173 ceph-mon[36116]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:31.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:30 smithi203 ceph-mon[36693]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:31.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:30 smithi012 ceph-mon[31695]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:33.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:32 smithi173 ceph-mon[36116]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:33.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:32 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-23T22:53:33.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:32 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:33.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:32 smithi203 ceph-mon[36693]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:33.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:32 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-23T22:53:33.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:32 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:33.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:32 smithi012 ceph-mon[31695]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:33.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:32 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-23T22:53:33.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:32 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:34.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:33 smithi173 ceph-mon[36116]: Deploying daemon osd.1 on smithi173 2024-03-23T22:53:34.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:33 smithi203 ceph-mon[36693]: Deploying daemon osd.1 on smithi173 2024-03-23T22:53:34.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:33 smithi012 ceph-mon[31695]: Deploying daemon osd.1 on smithi173 2024-03-23T22:53:35.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:34 smithi173 ceph-mon[36116]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:35.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:34 smithi203 ceph-mon[36693]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:35.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:34 smithi012 ceph-mon[31695]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:37.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:36 smithi173 ceph-mon[36116]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:37.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:36 smithi203 ceph-mon[36693]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:37.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:36 smithi012 ceph-mon[31695]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:39.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:38 smithi173 ceph-mon[36116]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:39.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:38 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:39.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:38 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:39.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:38 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:53:39.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:38 smithi203 ceph-mon[36693]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:39.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:38 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:39.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:38 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:39.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:38 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:53:39.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:38 smithi012 ceph-mon[31695]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:39.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:38 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:39.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:38 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:39.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:38 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:53:41.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:41 smithi173 ceph-mon[36116]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:41.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:41 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:41.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:41 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:41.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:41 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:41.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:41 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:53:41.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:41 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:41.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:41 smithi203 ceph-mon[36693]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:41.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:41 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:41.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:41 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:41.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:41 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:41.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:41 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:53:41.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:41 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:41.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:41.467+0000 7f2ce77fe700 1 -- 172.21.15.173:0/2084831145 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2cf80625d0 con 0x7f2ce00a2880 2024-03-23T22:53:41.467 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 1 on host 'smithi173' 2024-03-23T22:53:41.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:41.470+0000 7f2cfcbd5700 1 -- 172.21.15.173:0/2084831145 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ce00a2880 msgr2=0x7f2ce00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:53:41.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:41.470+0000 7f2cfcbd5700 1 --2- 172.21.15.173:0/2084831145 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ce00a2880 0x7f2ce00a4d40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2cec006fd0 tx=0x7f2cec008040 comp rx=0 tx=0).stop 2024-03-23T22:53:41.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:41.470+0000 7f2cfcbd5700 1 -- 172.21.15.173:0/2084831145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cf81033c0 msgr2=0x7f2cf819f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:53:41.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:41.470+0000 7f2cfcbd5700 1 --2- 172.21.15.173:0/2084831145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cf81033c0 0x7f2cf819f080 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8003890 tx=0x7f2ce8003730 comp rx=0 tx=0).stop 2024-03-23T22:53:41.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:41.470+0000 7f2cfcbd5700 1 -- 172.21.15.173:0/2084831145 shutdown_connections 2024-03-23T22:53:41.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:41.470+0000 7f2cfcbd5700 1 --2- 172.21.15.173:0/2084831145 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ce00a2880 0x7f2ce00a4d40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:53:41.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:41.470+0000 7f2cfcbd5700 1 --2- 172.21.15.173:0/2084831145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf8103d60 0x7f2cf81a6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:53:41.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:41.470+0000 7f2cfcbd5700 1 --2- 172.21.15.173:0/2084831145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cf81021c0 0x7f2cf819eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:53:41.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:41.470+0000 7f2cfcbd5700 1 --2- 172.21.15.173:0/2084831145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cf81033c0 0x7f2cf819f080 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:53:41.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:41.470+0000 7f2cfcbd5700 1 -- 172.21.15.173:0/2084831145 >> 172.21.15.173:0/2084831145 conn(0x7f2cf80fd7a0 msgr2=0x7f2cf81047a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:53:41.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:41.470+0000 7f2cfcbd5700 1 -- 172.21.15.173:0/2084831145 shutdown_connections 2024-03-23T22:53:41.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:53:41.470+0000 7f2cfcbd5700 1 -- 172.21.15.173:0/2084831145 wait complete. 2024-03-23T22:53:41.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:41 smithi012 ceph-mon[31695]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:41.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:41 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:41.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:41 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:41.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:41 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:41.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:41 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:53:41.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:41 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:42.016 DEBUG:teuthology.orchestra.run.smithi173:osd.1> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.1.service 2024-03-23T22:53:42.019 INFO:tasks.cephadm:Deploying osd.2 on smithi203 with /dev/vg_nvme/lv_4... 2024-03-23T22:53:42.019 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-23T22:53:42.264 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.c/config 2024-03-23T22:53:42.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:42 smithi173 ceph-mon[36116]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:42.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:42 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:42.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:42 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:42.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:42 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:53:42.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:42 smithi173 ceph-mon[36116]: from='osd.1 [v2:172.21.15.173:6800/1983639525,v1:172.21.15.173:6801/1983639525]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-23T22:53:42.619 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:42 smithi173 ceph-mon[36116]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-23T22:53:42.646 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:42 smithi203 ceph-mon[36693]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:42.646 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:42 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:42.646 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:42 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:42.646 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:42 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:53:42.646 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:42 smithi203 ceph-mon[36693]: from='osd.1 [v2:172.21.15.173:6800/1983639525,v1:172.21.15.173:6801/1983639525]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-23T22:53:42.646 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:42 smithi203 ceph-mon[36693]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-23T22:53:42.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:42 smithi012 ceph-mon[31695]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:42.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:42 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:42.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:42 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:42.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:42 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:53:42.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:42 smithi012 ceph-mon[31695]: from='osd.1 [v2:172.21.15.173:6800/1983639525,v1:172.21.15.173:6801/1983639525]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-23T22:53:42.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:42 smithi012 ceph-mon[31695]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-23T22:53:43.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:43 smithi203 ceph-mon[36693]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-23T22:53:43.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:43 smithi203 ceph-mon[36693]: from='osd.1 [v2:172.21.15.173:6800/1983639525,v1:172.21.15.173:6801/1983639525]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-23T22:53:43.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:43 smithi203 ceph-mon[36693]: osdmap e11: 2 total, 1 up, 2 in 2024-03-23T22:53:43.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:43 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T22:53:43.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:43 smithi203 ceph-mon[36693]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-23T22:53:43.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:43 smithi012 ceph-mon[31695]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-23T22:53:43.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:43 smithi012 ceph-mon[31695]: from='osd.1 [v2:172.21.15.173:6800/1983639525,v1:172.21.15.173:6801/1983639525]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-23T22:53:43.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:43 smithi012 ceph-mon[31695]: osdmap e11: 2 total, 1 up, 2 in 2024-03-23T22:53:43.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:43 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T22:53:43.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:43 smithi012 ceph-mon[31695]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-23T22:53:43.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:43 smithi173 ceph-mon[36116]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-23T22:53:43.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:43 smithi173 ceph-mon[36116]: from='osd.1 [v2:172.21.15.173:6800/1983639525,v1:172.21.15.173:6801/1983639525]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-23T22:53:43.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:43 smithi173 ceph-mon[36116]: osdmap e11: 2 total, 1 up, 2 in 2024-03-23T22:53:43.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:43 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T22:53:43.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:43 smithi173 ceph-mon[36116]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-23T22:53:43.868 INFO:journalctl@ceph.osd.1.smithi173.stdout:Mar 23 22:53:43 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-1[40706]: 2024-03-23T22:53:43.435+0000 7fcb4c4c9700 -1 osd.1 0 waiting for initial osdmap 2024-03-23T22:53:43.868 INFO:journalctl@ceph.osd.1.smithi173.stdout:Mar 23 22:53:43 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-1[40706]: 2024-03-23T22:53:43.439+0000 7fcb482bc700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T22:53:44.220 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-23T22:53:44.220 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-23T22:53:44.221 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-03-23T22:53:44.221 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-03-23T22:53:44.221 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113453 s, 924 MB/s 2024-03-23T22:53:44.221 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-03-23T22:53:44.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:44 smithi173 ceph-mon[36116]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:44.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:44 smithi173 ceph-mon[36116]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-23T22:53:44.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:44 smithi173 ceph-mon[36116]: osdmap e12: 2 total, 1 up, 2 in 2024-03-23T22:53:44.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:44 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T22:53:44.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:44 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T22:53:44.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:44 smithi012 ceph-mon[31695]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:44.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:44 smithi012 ceph-mon[31695]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-23T22:53:44.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:44 smithi012 ceph-mon[31695]: osdmap e12: 2 total, 1 up, 2 in 2024-03-23T22:53:44.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:44 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T22:53:44.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:44 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T22:53:44.813 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:44 smithi203 ceph-mon[36693]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:44.814 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:44 smithi203 ceph-mon[36693]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-23T22:53:44.814 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:44 smithi203 ceph-mon[36693]: osdmap e12: 2 total, 1 up, 2 in 2024-03-23T22:53:44.814 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:44 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T22:53:44.814 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:44 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T22:53:44.840 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi203:vg_nvme/lv_4 2024-03-23T22:53:45.079 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.c/config 2024-03-23T22:53:45.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:45 smithi012 ceph-mon[31695]: purged_snaps scrub starts 2024-03-23T22:53:45.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:45 smithi012 ceph-mon[31695]: purged_snaps scrub ok 2024-03-23T22:53:45.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:45 smithi012 ceph-mon[31695]: osd.1 [v2:172.21.15.173:6800/1983639525,v1:172.21.15.173:6801/1983639525] boot 2024-03-23T22:53:45.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:45 smithi012 ceph-mon[31695]: osdmap e13: 2 total, 2 up, 2 in 2024-03-23T22:53:45.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:45 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T22:53:45.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:45 smithi012 ceph-mon[31695]: Detected new or changed devices on smithi173 2024-03-23T22:53:45.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:45 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:45.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:45 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:45.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:45 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:53:45.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:45 smithi012 ceph-mon[31695]: Adjusting osd_memory_target on smithi173 to 17080M 2024-03-23T22:53:45.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:45 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:45.806 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:45 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:45.806 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:45 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:53:45.806 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:45 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:45.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:45 smithi173 ceph-mon[36116]: purged_snaps scrub starts 2024-03-23T22:53:45.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:45 smithi173 ceph-mon[36116]: purged_snaps scrub ok 2024-03-23T22:53:45.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:45 smithi173 ceph-mon[36116]: osd.1 [v2:172.21.15.173:6800/1983639525,v1:172.21.15.173:6801/1983639525] boot 2024-03-23T22:53:45.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:45 smithi173 ceph-mon[36116]: osdmap e13: 2 total, 2 up, 2 in 2024-03-23T22:53:45.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:45 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T22:53:45.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:45 smithi173 ceph-mon[36116]: Detected new or changed devices on smithi173 2024-03-23T22:53:45.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:45 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:45.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:45 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:45.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:45 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:53:45.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:45 smithi173 ceph-mon[36116]: Adjusting osd_memory_target on smithi173 to 17080M 2024-03-23T22:53:45.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:45 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:45.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:45 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:45.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:45 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:53:45.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:45 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:45.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:45 smithi203 ceph-mon[36693]: purged_snaps scrub starts 2024-03-23T22:53:45.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:45 smithi203 ceph-mon[36693]: purged_snaps scrub ok 2024-03-23T22:53:45.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:45 smithi203 ceph-mon[36693]: osd.1 [v2:172.21.15.173:6800/1983639525,v1:172.21.15.173:6801/1983639525] boot 2024-03-23T22:53:45.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:45 smithi203 ceph-mon[36693]: osdmap e13: 2 total, 2 up, 2 in 2024-03-23T22:53:45.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:45 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T22:53:45.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:45 smithi203 ceph-mon[36693]: Detected new or changed devices on smithi173 2024-03-23T22:53:45.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:45 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:45.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:45 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:45.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:45 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:53:45.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:45 smithi203 ceph-mon[36693]: Adjusting osd_memory_target on smithi173 to 17080M 2024-03-23T22:53:45.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:45 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:45.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:45 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:45.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:45 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:53:45.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:45 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:53:46.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.659+0000 7f3e5b7a2700 1 -- 172.21.15.203:0/2148016308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e54102cf0 msgr2=0x7f3e541050e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:53:46.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.659+0000 7f3e5b7a2700 1 --2- 172.21.15.203:0/2148016308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e54102cf0 0x7f3e541050e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3e50009990 tx=0x7f3e50009ca0 comp rx=0 tx=0).stop 2024-03-23T22:53:46.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.659+0000 7f3e5b7a2700 1 -- 172.21.15.203:0/2148016308 shutdown_connections 2024-03-23T22:53:46.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.659+0000 7f3e5b7a2700 1 --2- 172.21.15.203:0/2148016308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e54102cf0 0x7f3e541050e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:53:46.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.660+0000 7f3e5b7a2700 1 --2- 172.21.15.203:0/2148016308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e54107f50 0x7f3e5410a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:53:46.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.660+0000 7f3e5b7a2700 1 --2- 172.21.15.203:0/2148016308 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e54105620 0x7f3e54107a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:53:46.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.660+0000 7f3e5b7a2700 1 -- 172.21.15.203:0/2148016308 >> 172.21.15.203:0/2148016308 conn(0x7f3e540fc680 msgr2=0x7f3e540feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:53:46.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.660+0000 7f3e5b7a2700 1 -- 172.21.15.203:0/2148016308 shutdown_connections 2024-03-23T22:53:46.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.660+0000 7f3e5b7a2700 1 -- 172.21.15.203:0/2148016308 wait complete. 2024-03-23T22:53:46.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.660+0000 7f3e5b7a2700 1 Processor -- start 2024-03-23T22:53:46.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.660+0000 7f3e5b7a2700 1 -- start start 2024-03-23T22:53:46.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.660+0000 7f3e5b7a2700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e54102cf0 0x7f3e54102650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:53:46.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.660+0000 7f3e5b7a2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e54105620 0x7f3e54100ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:53:46.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.660+0000 7f3e5b7a2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e54107f50 0x7f3e541011e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:53:46.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.660+0000 7f3e5b7a2700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5410de10 con 0x7f3e54102cf0 2024-03-23T22:53:46.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.660+0000 7f3e5b7a2700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5410dc90 con 0x7f3e54107f50 2024-03-23T22:53:46.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.660+0000 7f3e5b7a2700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5410df90 con 0x7f3e54105620 2024-03-23T22:53:46.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.660+0000 7f3e58d3d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e54105620 0x7f3e54100ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:53:46.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.660+0000 7f3e58d3d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e54105620 0x7f3e54100ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:35136/0 (socket says 172.21.15.203:35136) 2024-03-23T22:53:46.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.661+0000 7f3e58d3d700 1 -- 172.21.15.203:0/471823123 learned_addr learned my addr 172.21.15.203:0/471823123 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-23T22:53:46.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.661+0000 7f3e59d3f700 1 --2- 172.21.15.203:0/471823123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e54107f50 0x7f3e541011e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:53:46.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.661+0000 7f3e58d3d700 1 -- 172.21.15.203:0/471823123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e54107f50 msgr2=0x7f3e541011e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:53:46.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.661+0000 7f3e58d3d700 1 --2- 172.21.15.203:0/471823123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e54107f50 0x7f3e541011e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:53:46.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.661+0000 7f3e58d3d700 1 -- 172.21.15.203:0/471823123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e54102cf0 msgr2=0x7f3e54102650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:53:46.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.661+0000 7f3e58d3d700 1 --2- 172.21.15.203:0/471823123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e54102cf0 0x7f3e54102650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:53:46.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.661+0000 7f3e58d3d700 1 -- 172.21.15.203:0/471823123 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e54101a90 con 0x7f3e54105620 2024-03-23T22:53:46.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.661+0000 7f3e58d3d700 1 --2- 172.21.15.203:0/471823123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e54105620 0x7f3e54100ca0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3e4400c920 tx=0x7f3e4400cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:53:46.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.661+0000 7f3e4a7fc700 1 -- 172.21.15.203:0/471823123 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e44007b00 con 0x7f3e54105620 2024-03-23T22:53:46.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.661+0000 7f3e5b7a2700 1 -- 172.21.15.203:0/471823123 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e541a2540 con 0x7f3e54105620 2024-03-23T22:53:46.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.661+0000 7f3e4a7fc700 1 -- 172.21.15.203:0/471823123 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e44004d10 con 0x7f3e54105620 2024-03-23T22:53:46.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.661+0000 7f3e4a7fc700 1 -- 172.21.15.203:0/471823123 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e44005680 con 0x7f3e54105620 2024-03-23T22:53:46.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.661+0000 7f3e5b7a2700 1 -- 172.21.15.203:0/471823123 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e541a2ad0 con 0x7f3e54105620 2024-03-23T22:53:46.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.662+0000 7f3e5b7a2700 1 -- 172.21.15.203:0/471823123 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e54068e50 con 0x7f3e54105620 2024-03-23T22:53:46.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.663+0000 7f3e4a7fc700 1 -- 172.21.15.203:0/471823123 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f3e44004750 con 0x7f3e54105620 2024-03-23T22:53:46.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.664+0000 7f3e4a7fc700 1 --2- 172.21.15.203:0/471823123 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3e400a2800 0x7f3e400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:53:46.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.664+0000 7f3e4a7fc700 1 -- 172.21.15.203:0/471823123 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f3e440ccda0 con 0x7f3e54105620 2024-03-23T22:53:46.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.664+0000 7f3e5953e700 1 --2- 172.21.15.203:0/471823123 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3e400a2800 0x7f3e400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:53:46.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.665+0000 7f3e5953e700 1 --2- 172.21.15.203:0/471823123 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3e400a2800 0x7f3e400a4cc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3e50005f50 tx=0x7f3e50005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:53:46.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.668+0000 7f3e4a7fc700 1 -- 172.21.15.203:0/471823123 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3e44014030 con 0x7f3e54105620 2024-03-23T22:53:46.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:46 smithi012 ceph-mon[31695]: pgmap v74: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:46.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:46 smithi012 ceph-mon[31695]: osdmap e14: 2 total, 2 up, 2 in 2024-03-23T22:53:46.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:46 smithi173 ceph-mon[36116]: pgmap v74: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:46.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:46 smithi173 ceph-mon[36116]: osdmap e14: 2 total, 2 up, 2 in 2024-03-23T22:53:46.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:53:46.868+0000 7f3e5b7a2700 1 -- 172.21.15.203:0/471823123 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e540625d0 con 0x7f3e400a2800 2024-03-23T22:53:46.869 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:46 smithi203 ceph-mon[36693]: pgmap v74: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-23T22:53:46.869 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:46 smithi203 ceph-mon[36693]: osdmap e14: 2 total, 2 up, 2 in 2024-03-23T22:53:47.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:47 smithi012 ceph-mon[31695]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:53:47.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:47 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:53:47.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:47 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:53:47.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:47 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:47.826 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:47 smithi203 ceph-mon[36693]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:53:47.826 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:47 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:53:47.826 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:47 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:53:47.826 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:47 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:47.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:47 smithi173 ceph-mon[36116]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:53:47.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:47 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:53:47.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:47 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:53:47.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:47 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:48.581 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:48 smithi203 ceph-mon[36693]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:53:48.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:48 smithi012 ceph-mon[31695]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:53:48.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:48 smithi173 ceph-mon[36116]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:53:49.750 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:49 smithi203 ceph-mon[36693]: from='client.? 172.21.15.203:0/2834313586' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a559c70-4c54-431b-8baf-4136b6e76493"}]: dispatch 2024-03-23T22:53:49.750 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:49 smithi203 ceph-mon[36693]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a559c70-4c54-431b-8baf-4136b6e76493"}]: dispatch 2024-03-23T22:53:49.750 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:49 smithi203 ceph-mon[36693]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a559c70-4c54-431b-8baf-4136b6e76493"}]': finished 2024-03-23T22:53:49.751 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:49 smithi203 ceph-mon[36693]: osdmap e15: 3 total, 2 up, 3 in 2024-03-23T22:53:49.751 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:49 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T22:53:49.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:49 smithi012 ceph-mon[31695]: from='client.? 172.21.15.203:0/2834313586' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a559c70-4c54-431b-8baf-4136b6e76493"}]: dispatch 2024-03-23T22:53:49.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:49 smithi012 ceph-mon[31695]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a559c70-4c54-431b-8baf-4136b6e76493"}]: dispatch 2024-03-23T22:53:49.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:49 smithi012 ceph-mon[31695]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a559c70-4c54-431b-8baf-4136b6e76493"}]': finished 2024-03-23T22:53:49.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:49 smithi012 ceph-mon[31695]: osdmap e15: 3 total, 2 up, 3 in 2024-03-23T22:53:49.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:49 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T22:53:49.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:49 smithi173 ceph-mon[36116]: from='client.? 172.21.15.203:0/2834313586' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a559c70-4c54-431b-8baf-4136b6e76493"}]: dispatch 2024-03-23T22:53:49.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:49 smithi173 ceph-mon[36116]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a559c70-4c54-431b-8baf-4136b6e76493"}]: dispatch 2024-03-23T22:53:49.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:49 smithi173 ceph-mon[36116]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a559c70-4c54-431b-8baf-4136b6e76493"}]': finished 2024-03-23T22:53:49.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:49 smithi173 ceph-mon[36116]: osdmap e15: 3 total, 2 up, 3 in 2024-03-23T22:53:49.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:49 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T22:53:50.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:50 smithi012 ceph-mon[31695]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:53:50.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:50 smithi012 ceph-mon[31695]: from='client.? 172.21.15.203:0/331240024' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:53:50.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:50 smithi173 ceph-mon[36116]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:53:50.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:50 smithi173 ceph-mon[36116]: from='client.? 172.21.15.203:0/331240024' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:53:50.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:50 smithi203 ceph-mon[36693]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:53:50.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:50 smithi203 ceph-mon[36693]: from='client.? 172.21.15.203:0/331240024' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:53:52.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:52 smithi012 ceph-mon[31695]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:53:52.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:52 smithi173 ceph-mon[36116]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:53:52.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:52 smithi203 ceph-mon[36693]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:53:54.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:54 smithi203 ceph-mon[36693]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:53:54.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:54 smithi012 ceph-mon[31695]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:53:54.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:54 smithi173 ceph-mon[36116]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:53:56.670 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:56 smithi203 ceph-mon[36693]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:53:56.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:56 smithi012 ceph-mon[31695]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:53:56.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:56 smithi173 ceph-mon[36116]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:53:57.640 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:57 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-23T22:53:57.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:57 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:57.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:57 smithi203 ceph-mon[36693]: Deploying daemon osd.2 on smithi203 2024-03-23T22:53:57.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:57 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-23T22:53:57.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:57 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:57.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:57 smithi012 ceph-mon[31695]: Deploying daemon osd.2 on smithi203 2024-03-23T22:53:57.860 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:57 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-23T22:53:57.860 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:57 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:53:57.860 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:57 smithi173 ceph-mon[36116]: Deploying daemon osd.2 on smithi203 2024-03-23T22:53:58.592 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:53:58 smithi203 ceph-mon[36693]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:53:58.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:53:58 smithi012 ceph-mon[31695]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:53:58.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:53:58 smithi173 ceph-mon[36116]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:54:00.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:00 smithi012 ceph-mon[31695]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:54:00.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:00 smithi173 ceph-mon[36116]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:54:00.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:00 smithi203 ceph-mon[36693]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:54:02.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:02 smithi012 ceph-mon[31695]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:54:02.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:02 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:02.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:02 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:02.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:02 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:54:02.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:02 smithi173 ceph-mon[36116]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:54:02.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:02 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:02.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:02 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:02.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:02 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:54:02.873 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:02 smithi203 ceph-mon[36693]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:54:02.873 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:02 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:02.873 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:02 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:02.873 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:02 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:54:04.736 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:04 smithi203 ceph-mon[36693]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:54:04.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:04 smithi012 ceph-mon[31695]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:54:04.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:04 smithi173 ceph-mon[36116]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:54:05.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:54:05.911+0000 7f3e4a7fc700 1 -- 172.21.15.203:0/471823123 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3e540625d0 con 0x7f3e400a2800 2024-03-23T22:54:05.911 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 2 on host 'smithi203' 2024-03-23T22:54:05.913 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:54:05.913+0000 7f3e5b7a2700 1 -- 172.21.15.203:0/471823123 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3e400a2800 msgr2=0x7f3e400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:54:05.913 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:54:05.913+0000 7f3e5b7a2700 1 --2- 172.21.15.203:0/471823123 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3e400a2800 0x7f3e400a4cc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3e50005f50 tx=0x7f3e50005e80 comp rx=0 tx=0).stop 2024-03-23T22:54:05.913 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:54:05.913+0000 7f3e5b7a2700 1 -- 172.21.15.203:0/471823123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e54105620 msgr2=0x7f3e54100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:54:05.913 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:54:05.913+0000 7f3e5b7a2700 1 --2- 172.21.15.203:0/471823123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e54105620 0x7f3e54100ca0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3e4400c920 tx=0x7f3e4400cc30 comp rx=0 tx=0).stop 2024-03-23T22:54:05.913 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:54:05.913+0000 7f3e5b7a2700 1 -- 172.21.15.203:0/471823123 shutdown_connections 2024-03-23T22:54:05.913 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:54:05.913+0000 7f3e5b7a2700 1 --2- 172.21.15.203:0/471823123 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3e400a2800 0x7f3e400a4cc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:54:05.913 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:54:05.913+0000 7f3e5b7a2700 1 --2- 172.21.15.203:0/471823123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e54107f50 0x7f3e541011e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:54:05.913 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:54:05.913+0000 7f3e5b7a2700 1 --2- 172.21.15.203:0/471823123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e54102cf0 0x7f3e54102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:54:05.913 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:54:05.913+0000 7f3e5b7a2700 1 --2- 172.21.15.203:0/471823123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e54105620 0x7f3e54100ca0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:54:05.913 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:54:05.913+0000 7f3e5b7a2700 1 -- 172.21.15.203:0/471823123 >> 172.21.15.203:0/471823123 conn(0x7f3e540fc680 msgr2=0x7f3e54103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:54:05.914 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:54:05.913+0000 7f3e5b7a2700 1 -- 172.21.15.203:0/471823123 shutdown_connections 2024-03-23T22:54:05.914 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:54:05.913+0000 7f3e5b7a2700 1 -- 172.21.15.203:0/471823123 wait complete. 2024-03-23T22:54:06.117 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:05 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:06.117 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:05 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:06.117 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:05 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:54:06.117 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:05 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:54:06.117 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:05 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:06.117 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:05 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:06.117 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:05 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:54:06.117 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:05 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:06.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:05 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:06.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:05 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:06.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:05 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:54:06.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:05 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:54:06.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:05 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:06.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:05 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:06.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:05 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:54:06.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:05 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:06.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:05 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:06.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:05 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:06.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:05 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:54:06.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:05 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:54:06.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:05 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:06.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:05 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:06.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:05 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:54:06.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:05 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:06.541 DEBUG:teuthology.orchestra.run.smithi203:osd.2> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.2.service 2024-03-23T22:54:06.543 INFO:tasks.cephadm:Deploying osd.3 on smithi012 with /dev/vg_nvme/lv_3... 2024-03-23T22:54:06.543 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-23T22:54:07.008 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:54:07.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:06 smithi203 ceph-mon[36693]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:54:07.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:06 smithi203 ceph-mon[36693]: from='osd.2 [v2:172.21.15.203:6800/2253997387,v1:172.21.15.203:6801/2253997387]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-23T22:54:07.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:06 smithi203 ceph-mon[36693]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-23T22:54:07.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:06 smithi012 ceph-mon[31695]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:54:07.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:06 smithi012 ceph-mon[31695]: from='osd.2 [v2:172.21.15.203:6800/2253997387,v1:172.21.15.203:6801/2253997387]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-23T22:54:07.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:06 smithi012 ceph-mon[31695]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-23T22:54:07.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:06 smithi173 ceph-mon[36116]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:54:07.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:06 smithi173 ceph-mon[36116]: from='osd.2 [v2:172.21.15.203:6800/2253997387,v1:172.21.15.203:6801/2253997387]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-23T22:54:07.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:06 smithi173 ceph-mon[36116]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-23T22:54:08.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:07 smithi203 ceph-mon[36693]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-23T22:54:08.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:07 smithi203 ceph-mon[36693]: osdmap e16: 3 total, 2 up, 3 in 2024-03-23T22:54:08.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:07 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T22:54:08.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:07 smithi203 ceph-mon[36693]: from='osd.2 [v2:172.21.15.203:6800/2253997387,v1:172.21.15.203:6801/2253997387]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-23T22:54:08.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:07 smithi203 ceph-mon[36693]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-23T22:54:08.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:07 smithi012 ceph-mon[31695]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-23T22:54:08.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:07 smithi012 ceph-mon[31695]: osdmap e16: 3 total, 2 up, 3 in 2024-03-23T22:54:08.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:07 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T22:54:08.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:07 smithi012 ceph-mon[31695]: from='osd.2 [v2:172.21.15.203:6800/2253997387,v1:172.21.15.203:6801/2253997387]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-23T22:54:08.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:07 smithi012 ceph-mon[31695]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-23T22:54:08.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:07 smithi173 ceph-mon[36116]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-23T22:54:08.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:07 smithi173 ceph-mon[36116]: osdmap e16: 3 total, 2 up, 3 in 2024-03-23T22:54:08.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:07 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T22:54:08.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:07 smithi173 ceph-mon[36116]: from='osd.2 [v2:172.21.15.203:6800/2253997387,v1:172.21.15.203:6801/2253997387]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-23T22:54:08.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:07 smithi173 ceph-mon[36116]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-23T22:54:08.740 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 23 22:54:08 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-2[41863]: 2024-03-23T22:54:08.431+0000 7fafa4fd3700 -1 osd.2 0 waiting for initial osdmap 2024-03-23T22:54:08.740 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 23 22:54:08 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-2[41863]: 2024-03-23T22:54:08.437+0000 7faf9c5bd700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T22:54:09.050 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:08 smithi203 ceph-mon[36693]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:54:09.050 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:08 smithi203 ceph-mon[36693]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-23T22:54:09.050 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:08 smithi203 ceph-mon[36693]: osdmap e17: 3 total, 2 up, 3 in 2024-03-23T22:54:09.050 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:08 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T22:54:09.050 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:08 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T22:54:09.051 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:08 smithi203 ceph-mon[36693]: from='osd.2 ' entity='osd.2' 2024-03-23T22:54:09.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:08 smithi012 ceph-mon[31695]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:54:09.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:08 smithi012 ceph-mon[31695]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-23T22:54:09.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:08 smithi012 ceph-mon[31695]: osdmap e17: 3 total, 2 up, 3 in 2024-03-23T22:54:09.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:08 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T22:54:09.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:08 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T22:54:09.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:08 smithi012 ceph-mon[31695]: from='osd.2 ' entity='osd.2' 2024-03-23T22:54:09.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:08 smithi173 ceph-mon[36116]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:54:09.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:08 smithi173 ceph-mon[36116]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-23T22:54:09.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:08 smithi173 ceph-mon[36116]: osdmap e17: 3 total, 2 up, 3 in 2024-03-23T22:54:09.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:08 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T22:54:09.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:08 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T22:54:09.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:08 smithi173 ceph-mon[36116]: from='osd.2 ' entity='osd.2' 2024-03-23T22:54:10.147 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-23T22:54:10.148 INFO:teuthology.orchestra.run.smithi012.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-23T22:54:10.148 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10+0 records in 2024-03-23T22:54:10.148 INFO:teuthology.orchestra.run.smithi012.stderr:10+0 records out 2024-03-23T22:54:10.148 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0355417 s, 295 MB/s 2024-03-23T22:54:10.148 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping successful for: 2024-03-23T22:54:10.295 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:09 smithi203 ceph-mon[36693]: purged_snaps scrub starts 2024-03-23T22:54:10.295 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:09 smithi203 ceph-mon[36693]: purged_snaps scrub ok 2024-03-23T22:54:10.295 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:09 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T22:54:10.295 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:09 smithi203 ceph-mon[36693]: osd.2 [v2:172.21.15.203:6800/2253997387,v1:172.21.15.203:6801/2253997387] boot 2024-03-23T22:54:10.295 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:09 smithi203 ceph-mon[36693]: osdmap e18: 3 total, 3 up, 3 in 2024-03-23T22:54:10.296 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:09 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T22:54:10.296 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:09 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:10.296 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:09 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:10.296 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:09 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:54:10.296 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:09 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:10.296 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:09 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:54:10.296 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:09 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:54:10.296 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:09 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:10.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:09 smithi012 ceph-mon[31695]: purged_snaps scrub starts 2024-03-23T22:54:10.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:09 smithi012 ceph-mon[31695]: purged_snaps scrub ok 2024-03-23T22:54:10.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:09 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T22:54:10.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:09 smithi012 ceph-mon[31695]: osd.2 [v2:172.21.15.203:6800/2253997387,v1:172.21.15.203:6801/2253997387] boot 2024-03-23T22:54:10.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:09 smithi012 ceph-mon[31695]: osdmap e18: 3 total, 3 up, 3 in 2024-03-23T22:54:10.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:09 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T22:54:10.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:09 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:10.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:09 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:10.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:09 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:54:10.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:09 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:10.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:09 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:54:10.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:09 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:54:10.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:09 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:10.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:09 smithi173 ceph-mon[36116]: purged_snaps scrub starts 2024-03-23T22:54:10.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:09 smithi173 ceph-mon[36116]: purged_snaps scrub ok 2024-03-23T22:54:10.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:09 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T22:54:10.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:09 smithi173 ceph-mon[36116]: osd.2 [v2:172.21.15.203:6800/2253997387,v1:172.21.15.203:6801/2253997387] boot 2024-03-23T22:54:10.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:09 smithi173 ceph-mon[36116]: osdmap e18: 3 total, 3 up, 3 in 2024-03-23T22:54:10.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:09 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T22:54:10.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:09 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:10.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:09 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:10.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:09 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:54:10.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:09 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:10.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:09 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:54:10.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:09 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:54:10.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:09 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:11.043 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi012:vg_nvme/lv_3 2024-03-23T22:54:11.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:10 smithi012 ceph-mon[31695]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:54:11.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:10 smithi012 ceph-mon[31695]: Detected new or changed devices on smithi203 2024-03-23T22:54:11.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:10 smithi012 ceph-mon[31695]: Adjusting osd_memory_target on smithi203 to 17080M 2024-03-23T22:54:11.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:10 smithi173 ceph-mon[36116]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:54:11.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:10 smithi173 ceph-mon[36116]: Detected new or changed devices on smithi203 2024-03-23T22:54:11.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:10 smithi173 ceph-mon[36116]: Adjusting osd_memory_target on smithi203 to 17080M 2024-03-23T22:54:11.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:10 smithi203 ceph-mon[36693]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-23T22:54:11.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:10 smithi203 ceph-mon[36693]: Detected new or changed devices on smithi203 2024-03-23T22:54:11.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:10 smithi203 ceph-mon[36693]: Adjusting osd_memory_target on smithi203 to 17080M 2024-03-23T22:54:11.504 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:54:12.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:12 smithi012 ceph-mon[31695]: osdmap e19: 3 total, 3 up, 3 in 2024-03-23T22:54:12.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:12 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-23T22:54:12.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:12 smithi173 ceph-mon[36116]: osdmap e19: 3 total, 3 up, 3 in 2024-03-23T22:54:12.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:12 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-23T22:54:12.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:12 smithi203 ceph-mon[36693]: osdmap e19: 3 total, 3 up, 3 in 2024-03-23T22:54:12.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:12 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-23T22:54:13.197 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:13 smithi012 ceph-mon[31695]: pgmap v93: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:13.197 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:13 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-23T22:54:13.197 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:13 smithi012 ceph-mon[31695]: osdmap e20: 3 total, 3 up, 3 in 2024-03-23T22:54:13.198 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:13 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-23T22:54:13.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:13 smithi173 ceph-mon[36116]: pgmap v93: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:13.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:13 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-23T22:54:13.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:13 smithi173 ceph-mon[36116]: osdmap e20: 3 total, 3 up, 3 in 2024-03-23T22:54:13.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:13 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-23T22:54:13.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:13 smithi203 ceph-mon[36693]: pgmap v93: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:13.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:13 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-23T22:54:13.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:13 smithi203 ceph-mon[36693]: osdmap e20: 3 total, 3 up, 3 in 2024-03-23T22:54:13.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:13 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-23T22:54:13.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.667+0000 7fe66f200700 1 -- 172.21.15.12:0/307990233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6681032e0 msgr2=0x7fe668103740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:54:13.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.667+0000 7fe66f200700 1 --2- 172.21.15.12:0/307990233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6681032e0 0x7fe668103740 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe6580098f0 tx=0x7fe658009c00 comp rx=0 tx=0).stop 2024-03-23T22:54:13.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.667+0000 7fe66f200700 1 -- 172.21.15.12:0/307990233 shutdown_connections 2024-03-23T22:54:13.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.667+0000 7fe66f200700 1 --2- 172.21.15.12:0/307990233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe668103c80 0x7fe66810a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:54:13.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.667+0000 7fe66f200700 1 --2- 172.21.15.12:0/307990233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6681020e0 0x7fe668102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:54:13.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.667+0000 7fe66f200700 1 --2- 172.21.15.12:0/307990233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6681032e0 0x7fe668103740 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:54:13.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.667+0000 7fe66f200700 1 -- 172.21.15.12:0/307990233 >> 172.21.15.12:0/307990233 conn(0x7fe6680fd6a0 msgr2=0x7fe6680ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:54:13.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.668+0000 7fe66f200700 1 -- 172.21.15.12:0/307990233 shutdown_connections 2024-03-23T22:54:13.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.668+0000 7fe66f200700 1 -- 172.21.15.12:0/307990233 wait complete. 2024-03-23T22:54:13.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.669+0000 7fe66f200700 1 Processor -- start 2024-03-23T22:54:13.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.669+0000 7fe66f200700 1 -- start start 2024-03-23T22:54:13.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.669+0000 7fe66f200700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6681020e0 0x7fe668075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:54:13.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.669+0000 7fe66f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6681032e0 0x7fe668074580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:54:13.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.669+0000 7fe66f200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe668103c80 0x7fe668074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:54:13.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.669+0000 7fe66f200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66810dd00 con 0x7fe6681032e0 2024-03-23T22:54:13.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.669+0000 7fe66f200700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66810db80 con 0x7fe6681020e0 2024-03-23T22:54:13.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.669+0000 7fe66f200700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66810de80 con 0x7fe668103c80 2024-03-23T22:54:13.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.670+0000 7fe66d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6681032e0 0x7fe668074580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:54:13.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.670+0000 7fe66d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6681032e0 0x7fe668074580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35962/0 (socket says 172.21.15.12:35962) 2024-03-23T22:54:13.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.670+0000 7fe66d400700 1 -- 172.21.15.12:0/893394362 learned_addr learned my addr 172.21.15.12:0/893394362 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:54:13.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.670+0000 7fe66de00700 1 --2- 172.21.15.12:0/893394362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6681020e0 0x7fe668075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:54:13.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.670+0000 7fe66e800700 1 --2- 172.21.15.12:0/893394362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe668103c80 0x7fe668074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:54:13.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.670+0000 7fe66d400700 1 -- 172.21.15.12:0/893394362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe668103c80 msgr2=0x7fe668074ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:54:13.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.670+0000 7fe66d400700 1 --2- 172.21.15.12:0/893394362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe668103c80 0x7fe668074ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:54:13.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.670+0000 7fe66d400700 1 -- 172.21.15.12:0/893394362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6681020e0 msgr2=0x7fe668075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:54:13.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.670+0000 7fe66d400700 1 --2- 172.21.15.12:0/893394362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6681020e0 0x7fe668075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:54:13.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.670+0000 7fe66d400700 1 -- 172.21.15.12:0/893394362 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe668075000 con 0x7fe6681032e0 2024-03-23T22:54:13.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.670+0000 7fe66e800700 1 --2- 172.21.15.12:0/893394362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe668103c80 0x7fe668074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T22:54:13.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.671+0000 7fe66d400700 1 --2- 172.21.15.12:0/893394362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6681032e0 0x7fe668074580 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe658009230 tx=0x7fe6580052f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:54:13.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.671+0000 7fe662a00700 1 -- 172.21.15.12:0/893394362 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe658025070 con 0x7fe6681032e0 2024-03-23T22:54:13.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.671+0000 7fe662a00700 1 -- 172.21.15.12:0/893394362 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe65800ba40 con 0x7fe6681032e0 2024-03-23T22:54:13.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.671+0000 7fe662a00700 1 -- 172.21.15.12:0/893394362 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe658004e50 con 0x7fe6681032e0 2024-03-23T22:54:13.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.671+0000 7fe66f200700 1 -- 172.21.15.12:0/893394362 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe668075290 con 0x7fe6681032e0 2024-03-23T22:54:13.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.671+0000 7fe66f200700 1 -- 172.21.15.12:0/893394362 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6681a6d80 con 0x7fe6681032e0 2024-03-23T22:54:13.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.672+0000 7fe66f200700 1 -- 172.21.15.12:0/893394362 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe668068e50 con 0x7fe6681032e0 2024-03-23T22:54:13.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.676+0000 7fe662a00700 1 -- 172.21.15.12:0/893394362 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7fe658028070 con 0x7fe6681032e0 2024-03-23T22:54:13.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.678+0000 7fe662a00700 1 --2- 172.21.15.12:0/893394362 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe6500a29a0 0x7fe6500a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:54:13.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.679+0000 7fe662a00700 1 -- 172.21.15.12:0/893394362 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3237+0+0 (secure 0 0 0) 0x7fe6580ce310 con 0x7fe6681032e0 2024-03-23T22:54:13.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.679+0000 7fe66de00700 1 --2- 172.21.15.12:0/893394362 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe6500a29a0 0x7fe6500a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:54:13.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.679+0000 7fe66de00700 1 --2- 172.21.15.12:0/893394362 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe6500a29a0 0x7fe6500a4e60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe65c009680 tx=0x7fe65c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:54:13.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.681+0000 7fe662a00700 1 -- 172.21.15.12:0/893394362 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe65802f020 con 0x7fe6681032e0 2024-03-23T22:54:13.803 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 23 22:54:13 smithi012 sudo[50838]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-23T22:54:13.804 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 23 22:54:13 smithi012 sudo[50838]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-23T22:54:13.804 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 23 22:54:13 smithi012 sudo[50838]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-23T22:54:13.804 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 23 22:54:13 smithi012 sudo[50838]: pam_unix(sudo:session): session closed for user root 2024-03-23T22:54:13.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:13.923+0000 7fe66f200700 1 -- 172.21.15.12:0/893394362 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fe66802dd90 con 0x7fe6500a29a0 2024-03-23T22:54:14.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:14.033+0000 7fe662a00700 1 -- 172.21.15.12:0/893394362 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe6580987c0 con 0x7fe6681032e0 2024-03-23T22:54:14.302 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-23T22:54:14.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:14 smithi012 ceph-mon[31695]: osdmap e21: 3 total, 3 up, 3 in 2024-03-23T22:54:14.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:54:14.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:54:14.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:54:14.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:14 smithi012 ceph-mon[31695]: osdmap e22: 3 total, 3 up, 3 in 2024-03-23T22:54:14.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:14 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-23T22:54:14.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:14 smithi173 ceph-mon[36116]: osdmap e21: 3 total, 3 up, 3 in 2024-03-23T22:54:14.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:14 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:54:14.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:14 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:54:14.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:14 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:54:14.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:14 smithi173 ceph-mon[36116]: osdmap e22: 3 total, 3 up, 3 in 2024-03-23T22:54:14.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:14 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-23T22:54:14.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:14 smithi203 ceph-mon[36693]: osdmap e21: 3 total, 3 up, 3 in 2024-03-23T22:54:14.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:14 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:54:14.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:14 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:54:14.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:14 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:54:14.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:14 smithi203 ceph-mon[36693]: osdmap e22: 3 total, 3 up, 3 in 2024-03-23T22:54:14.553 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 23 22:54:14 smithi012 sudo[50908]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-23T22:54:14.553 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 23 22:54:14 smithi012 sudo[50908]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-23T22:54:14.553 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 23 22:54:14 smithi012 sudo[50908]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-23T22:54:14.553 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 23 22:54:14 smithi012 sudo[50908]: pam_unix(sudo:session): session closed for user root 2024-03-23T22:54:14.843 INFO:journalctl@ceph.osd.1.smithi173.stdout:Mar 23 22:54:14 smithi173 sudo[44722]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-23T22:54:14.843 INFO:journalctl@ceph.osd.1.smithi173.stdout:Mar 23 22:54:14 smithi173 sudo[44722]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-23T22:54:14.843 INFO:journalctl@ceph.osd.1.smithi173.stdout:Mar 23 22:54:14 smithi173 sudo[44722]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-23T22:54:14.843 INFO:journalctl@ceph.osd.1.smithi173.stdout:Mar 23 22:54:14 smithi173 sudo[44722]: pam_unix(sudo:session): session closed for user root 2024-03-23T22:54:15.088 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:15 smithi012 ceph-mon[31695]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:15.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:15 smithi012 ceph-mon[31695]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:54:15.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:15 smithi012 ceph-mon[31695]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-23T22:54:15.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:15 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:15.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:15 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:15.118 INFO:journalctl@ceph.osd.1.smithi173.stdout:Mar 23 22:54:14 smithi173 sudo[44725]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-23T22:54:15.118 INFO:journalctl@ceph.osd.1.smithi173.stdout:Mar 23 22:54:14 smithi173 sudo[44725]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-23T22:54:15.118 INFO:journalctl@ceph.osd.1.smithi173.stdout:Mar 23 22:54:14 smithi173 sudo[44725]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-23T22:54:15.118 INFO:journalctl@ceph.osd.1.smithi173.stdout:Mar 23 22:54:14 smithi173 sudo[44725]: pam_unix(sudo:session): session closed for user root 2024-03-23T22:54:15.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:15 smithi173 ceph-mon[36116]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:15.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:15 smithi173 ceph-mon[36116]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:54:15.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:15 smithi173 ceph-mon[36116]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-23T22:54:15.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:15 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:15.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:15 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:15.384 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 23 22:54:15 smithi203 sudo[45950]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-23T22:54:15.384 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 23 22:54:15 smithi203 sudo[45950]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-23T22:54:15.384 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 23 22:54:15 smithi203 sudo[45950]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-23T22:54:15.384 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 23 22:54:15 smithi203 sudo[45950]: pam_unix(sudo:session): session closed for user root 2024-03-23T22:54:15.385 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:15 smithi203 ceph-mon[36693]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:15.385 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:15 smithi203 ceph-mon[36693]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:54:15.385 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:15 smithi203 ceph-mon[36693]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-23T22:54:15.385 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:15 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:15.385 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:15 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:15.657 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 23 22:54:15 smithi203 sudo[45953]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-23T22:54:15.657 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 23 22:54:15 smithi203 sudo[45953]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-23T22:54:15.657 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 23 22:54:15 smithi203 sudo[45953]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-23T22:54:15.657 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 23 22:54:15 smithi203 sudo[45953]: pam_unix(sudo:session): session closed for user root 2024-03-23T22:54:16.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:16 smithi012 sudo[50988]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-23T22:54:16.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:16 smithi012 sudo[50988]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-23T22:54:16.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:16 smithi012 sudo[50988]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-23T22:54:16.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:16 smithi012 sudo[50988]: pam_unix(sudo:session): session closed for user root 2024-03-23T22:54:17.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:16 smithi012 sudo[51017]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-23T22:54:17.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:16 smithi012 sudo[51017]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-23T22:54:17.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:16 smithi012 sudo[51017]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-23T22:54:17.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:16 smithi012 sudo[51017]: pam_unix(sudo:session): session closed for user root 2024-03-23T22:54:17.095 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:16 smithi203 sudo[45956]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-23T22:54:17.095 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:16 smithi203 sudo[45956]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-23T22:54:17.095 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:16 smithi203 sudo[45956]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-23T22:54:17.095 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:16 smithi203 sudo[45956]: pam_unix(sudo:session): session closed for user root 2024-03-23T22:54:17.356 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:17 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:17.357 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:17 smithi173 ceph-mon[36116]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-23T22:54:17.357 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:17 smithi173 sudo[44728]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-23T22:54:17.357 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:17 smithi173 sudo[44728]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-23T22:54:17.357 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:17 smithi173 sudo[44728]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-23T22:54:17.357 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:17 smithi173 sudo[44728]: pam_unix(sudo:session): session closed for user root 2024-03-23T22:54:17.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:17 smithi203 sudo[45959]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-23T22:54:17.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:17 smithi203 sudo[45959]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-23T22:54:17.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:17 smithi203 sudo[45959]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-23T22:54:17.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:17 smithi203 sudo[45959]: pam_unix(sudo:session): session closed for user root 2024-03-23T22:54:17.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:17 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:17.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:17 smithi203 ceph-mon[36693]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-23T22:54:17.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:17 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:17.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:17 smithi012 ceph-mon[31695]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-23T22:54:17.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:17 smithi173 sudo[44731]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-23T22:54:17.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:17 smithi173 sudo[44731]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-23T22:54:17.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:17 smithi173 sudo[44731]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-23T22:54:17.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:17 smithi173 sudo[44731]: pam_unix(sudo:session): session closed for user root 2024-03-23T22:54:18.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:18 smithi173 ceph-mon[36116]: pgmap v98: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:18.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:18 smithi173 ceph-mon[36116]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-23T22:54:18.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:18 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-23T22:54:18.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:18 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:54:18.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:18 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:54:18.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:18 smithi173 ceph-mon[36116]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-23T22:54:18.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:18 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-23T22:54:18.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:18 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:54:18.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:18 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:54:18.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:18 smithi173 ceph-mon[36116]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-23T22:54:18.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:18 smithi173 ceph-mon[36116]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-23T22:54:18.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:18 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-23T22:54:18.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:18 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:54:18.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:18 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:54:18.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:18 smithi173 ceph-mon[36116]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-23T22:54:18.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:18 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2214282628' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c41cd287-a674-486c-8025-328cd36e5ce5"}]: dispatch 2024-03-23T22:54:18.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:18 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2214282628' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c41cd287-a674-486c-8025-328cd36e5ce5"}]': finished 2024-03-23T22:54:18.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:18 smithi173 ceph-mon[36116]: osdmap e23: 4 total, 3 up, 4 in 2024-03-23T22:54:18.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:18 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T22:54:18.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:18 smithi203 ceph-mon[36693]: pgmap v98: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:18.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:18 smithi203 ceph-mon[36693]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-23T22:54:18.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:18 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-23T22:54:18.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:18 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:54:18.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:18 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:54:18.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:18 smithi203 ceph-mon[36693]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-23T22:54:18.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:18 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-23T22:54:18.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:18 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:54:18.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:18 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:54:18.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:18 smithi203 ceph-mon[36693]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-23T22:54:18.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:18 smithi203 ceph-mon[36693]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-23T22:54:18.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:18 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-23T22:54:18.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:18 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:54:18.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:18 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:54:18.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:18 smithi203 ceph-mon[36693]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-23T22:54:18.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:18 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2214282628' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c41cd287-a674-486c-8025-328cd36e5ce5"}]: dispatch 2024-03-23T22:54:18.409 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:18 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2214282628' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c41cd287-a674-486c-8025-328cd36e5ce5"}]': finished 2024-03-23T22:54:18.409 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:18 smithi203 ceph-mon[36693]: osdmap e23: 4 total, 3 up, 4 in 2024-03-23T22:54:18.409 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:18 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T22:54:18.422 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:18 smithi012 ceph-mon[31695]: pgmap v98: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:18.422 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:18 smithi012 ceph-mon[31695]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-23T22:54:18.422 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:18 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-23T22:54:18.423 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:18 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:54:18.423 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:18 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:54:18.423 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:18 smithi012 ceph-mon[31695]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-23T22:54:18.423 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:18 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-23T22:54:18.423 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:18 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:54:18.423 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:18 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:54:18.423 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:18 smithi012 ceph-mon[31695]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-23T22:54:18.423 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:18 smithi012 ceph-mon[31695]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-23T22:54:18.423 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:18 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-23T22:54:18.423 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:18 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-23T22:54:18.423 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:18 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-23T22:54:18.423 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:18 smithi012 ceph-mon[31695]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-23T22:54:18.423 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:18 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2214282628' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c41cd287-a674-486c-8025-328cd36e5ce5"}]: dispatch 2024-03-23T22:54:18.423 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:18 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2214282628' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c41cd287-a674-486c-8025-328cd36e5ce5"}]': finished 2024-03-23T22:54:18.424 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:18 smithi012 ceph-mon[31695]: osdmap e23: 4 total, 3 up, 4 in 2024-03-23T22:54:18.424 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:18 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T22:54:19.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:19 smithi173 ceph-mon[36116]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:19.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:19 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3378553018' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:54:19.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:19 smithi203 ceph-mon[36693]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:19.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:19 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3378553018' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:54:19.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:19 smithi012 ceph-mon[31695]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:19.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:19 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3378553018' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:54:20.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:20 smithi203 ceph-mon[36693]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:20.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:20 smithi012 ceph-mon[31695]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:20.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:20 smithi173 ceph-mon[36116]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:22.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:22 smithi012 ceph-mon[31695]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2024-03-23T22:54:22.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:22 smithi173 ceph-mon[36116]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2024-03-23T22:54:22.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:22 smithi203 ceph-mon[36693]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2024-03-23T22:54:24.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:24 smithi012 ceph-mon[31695]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-03-23T22:54:24.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:24 smithi173 ceph-mon[36116]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-03-23T22:54:24.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:24 smithi203 ceph-mon[36693]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-03-23T22:54:26.502 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:26 smithi012 ceph-mon[31695]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-03-23T22:54:26.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:26 smithi173 ceph-mon[36116]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-03-23T22:54:26.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:26 smithi203 ceph-mon[36693]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-03-23T22:54:28.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:28 smithi012 ceph-mon[31695]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-23T22:54:28.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:28 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-23T22:54:28.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:28 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:54:28.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:28 smithi012 ceph-mon[31695]: Deploying daemon osd.3 on smithi012 2024-03-23T22:54:28.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:28 smithi173 ceph-mon[36116]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-23T22:54:28.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:28 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-23T22:54:28.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:28 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:54:28.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:28 smithi173 ceph-mon[36116]: Deploying daemon osd.3 on smithi012 2024-03-23T22:54:28.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:28 smithi203 ceph-mon[36693]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-23T22:54:28.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:28 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-23T22:54:28.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:28 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:54:28.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:28 smithi203 ceph-mon[36693]: Deploying daemon osd.3 on smithi012 2024-03-23T22:54:30.535 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:30 smithi012 ceph-mon[31695]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 815 B/s rd, 23 KiB/s wr, 2 op/s 2024-03-23T22:54:30.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:30 smithi173 ceph-mon[36116]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 815 B/s rd, 23 KiB/s wr, 2 op/s 2024-03-23T22:54:30.653 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:30 smithi203 ceph-mon[36693]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 815 B/s rd, 23 KiB/s wr, 2 op/s 2024-03-23T22:54:32.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:32 smithi012 ceph-mon[31695]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-23T22:54:32.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:32 smithi173 ceph-mon[36116]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-23T22:54:32.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:32 smithi203 ceph-mon[36693]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-23T22:54:34.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:34 smithi012 ceph-mon[31695]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:34.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:34 smithi173 ceph-mon[36116]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:34.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:34 smithi203 ceph-mon[36693]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:36.318 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:36 smithi012 ceph-mon[31695]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:36.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:36 smithi173 ceph-mon[36116]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:36.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:36 smithi203 ceph-mon[36693]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:38.008 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:37 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:38.008 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:37 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:38.009 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:37 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:54:38.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:37 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:38.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:37 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:38.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:37 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:54:38.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:37 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:38.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:37 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:38.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:37 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:54:39.069 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:38 smithi012 ceph-mon[31695]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:39.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:38 smithi173 ceph-mon[36116]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:39.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:38 smithi203 ceph-mon[36693]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:41.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:41 smithi173 ceph-mon[36116]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:41.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:41 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:41.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:41 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:41.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:41 smithi173 ceph-mon[36116]: from='osd.3 [v2:172.21.15.12:6810/1761446905,v1:172.21.15.12:6811/1761446905]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-23T22:54:41.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:41 smithi203 ceph-mon[36693]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:41.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:41 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:41.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:41 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:41.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:41 smithi203 ceph-mon[36693]: from='osd.3 [v2:172.21.15.12:6810/1761446905,v1:172.21.15.12:6811/1761446905]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-23T22:54:41.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:41 smithi012 ceph-mon[31695]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:41.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:41 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:41.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:41 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:41.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:41 smithi012 ceph-mon[31695]: from='osd.3 [v2:172.21.15.12:6810/1761446905,v1:172.21.15.12:6811/1761446905]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-23T22:54:42.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:42.181+0000 7fe662a00700 1 -- 172.21.15.12:0/893394362 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe66802dd90 con 0x7fe6500a29a0 2024-03-23T22:54:42.182 INFO:teuthology.orchestra.run.smithi012.stdout:Created osd(s) 3 on host 'smithi012' 2024-03-23T22:54:42.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:42.186+0000 7fe66f200700 1 -- 172.21.15.12:0/893394362 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe6500a29a0 msgr2=0x7fe6500a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:54:42.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:42.186+0000 7fe66f200700 1 --2- 172.21.15.12:0/893394362 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe6500a29a0 0x7fe6500a4e60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe65c009680 tx=0x7fe65c009260 comp rx=0 tx=0).stop 2024-03-23T22:54:42.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:42.186+0000 7fe66f200700 1 -- 172.21.15.12:0/893394362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6681032e0 msgr2=0x7fe668074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:54:42.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:42.186+0000 7fe66f200700 1 --2- 172.21.15.12:0/893394362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6681032e0 0x7fe668074580 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe658009230 tx=0x7fe6580052f0 comp rx=0 tx=0).stop 2024-03-23T22:54:42.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:42.186+0000 7fe66f200700 1 -- 172.21.15.12:0/893394362 shutdown_connections 2024-03-23T22:54:42.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:42.186+0000 7fe66f200700 1 --2- 172.21.15.12:0/893394362 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe6500a29a0 0x7fe6500a4e60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:54:42.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:42.186+0000 7fe66f200700 1 --2- 172.21.15.12:0/893394362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe668103c80 0x7fe668074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:54:42.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:42.187+0000 7fe66f200700 1 --2- 172.21.15.12:0/893394362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6681020e0 0x7fe668075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:54:42.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:42.187+0000 7fe66f200700 1 --2- 172.21.15.12:0/893394362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6681032e0 0x7fe668074580 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:54:42.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:42.187+0000 7fe66f200700 1 -- 172.21.15.12:0/893394362 >> 172.21.15.12:0/893394362 conn(0x7fe6680fd6a0 msgr2=0x7fe668106660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:54:42.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:42.187+0000 7fe66f200700 1 -- 172.21.15.12:0/893394362 shutdown_connections 2024-03-23T22:54:42.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:54:42.187+0000 7fe66f200700 1 -- 172.21.15.12:0/893394362 wait complete. 2024-03-23T22:54:42.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:42 smithi203 ceph-mon[36693]: from='osd.3 [v2:172.21.15.12:6810/1761446905,v1:172.21.15.12:6811/1761446905]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-23T22:54:42.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:42 smithi203 ceph-mon[36693]: osdmap e24: 4 total, 3 up, 4 in 2024-03-23T22:54:42.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:42 smithi203 ceph-mon[36693]: from='osd.3 [v2:172.21.15.12:6810/1761446905,v1:172.21.15.12:6811/1761446905]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-23T22:54:42.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:42 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T22:54:42.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:42 smithi203 ceph-mon[36693]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:42.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:42 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:54:42.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:42 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:54:42.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:42 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:42.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:42 smithi012 ceph-mon[31695]: from='osd.3 [v2:172.21.15.12:6810/1761446905,v1:172.21.15.12:6811/1761446905]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-23T22:54:42.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:42 smithi012 ceph-mon[31695]: osdmap e24: 4 total, 3 up, 4 in 2024-03-23T22:54:42.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:42 smithi012 ceph-mon[31695]: from='osd.3 [v2:172.21.15.12:6810/1761446905,v1:172.21.15.12:6811/1761446905]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-23T22:54:42.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:42 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T22:54:42.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:42 smithi012 ceph-mon[31695]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:42.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:42 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:54:42.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:42 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:54:42.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:42 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:42.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:42 smithi173 ceph-mon[36116]: from='osd.3 [v2:172.21.15.12:6810/1761446905,v1:172.21.15.12:6811/1761446905]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-23T22:54:42.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:42 smithi173 ceph-mon[36116]: osdmap e24: 4 total, 3 up, 4 in 2024-03-23T22:54:42.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:42 smithi173 ceph-mon[36116]: from='osd.3 [v2:172.21.15.12:6810/1761446905,v1:172.21.15.12:6811/1761446905]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-23T22:54:42.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:42 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T22:54:42.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:42 smithi173 ceph-mon[36116]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-23T22:54:42.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:42 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:54:42.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:42 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:54:42.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:42 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:43.027 DEBUG:teuthology.orchestra.run.smithi012:osd.3> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.3.service 2024-03-23T22:54:43.033 INFO:tasks.cephadm:Deploying osd.4 on smithi173 with /dev/vg_nvme/lv_3... 2024-03-23T22:54:43.034 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-23T22:54:43.235 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:43 smithi012 ceph-mon[31695]: from='osd.3 [v2:172.21.15.12:6810/1761446905,v1:172.21.15.12:6811/1761446905]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-23T22:54:43.235 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:43 smithi012 ceph-mon[31695]: osdmap e25: 4 total, 3 up, 4 in 2024-03-23T22:54:43.235 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:43 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:43.235 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:43 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T22:54:43.235 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:43 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T22:54:43.235 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:43 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:43.235 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:43 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:54:43.282 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.b/config 2024-03-23T22:54:43.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:43 smithi173 ceph-mon[36116]: from='osd.3 [v2:172.21.15.12:6810/1761446905,v1:172.21.15.12:6811/1761446905]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-23T22:54:43.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:43 smithi173 ceph-mon[36116]: osdmap e25: 4 total, 3 up, 4 in 2024-03-23T22:54:43.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:43 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:43.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:43 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T22:54:43.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:43 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T22:54:43.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:43 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:43.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:43 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:54:43.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:43 smithi203 ceph-mon[36693]: from='osd.3 [v2:172.21.15.12:6810/1761446905,v1:172.21.15.12:6811/1761446905]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-23T22:54:43.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:43 smithi203 ceph-mon[36693]: osdmap e25: 4 total, 3 up, 4 in 2024-03-23T22:54:43.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:43 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:43.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:43 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T22:54:43.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:43 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T22:54:43.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:43 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:43.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:43 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:54:44.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:44 smithi012 ceph-mon[31695]: purged_snaps scrub starts 2024-03-23T22:54:44.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:44 smithi012 ceph-mon[31695]: purged_snaps scrub ok 2024-03-23T22:54:44.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:44 smithi012 ceph-mon[31695]: osd.3 [v2:172.21.15.12:6810/1761446905,v1:172.21.15.12:6811/1761446905] boot 2024-03-23T22:54:44.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:44 smithi012 ceph-mon[31695]: osdmap e26: 4 total, 4 up, 4 in 2024-03-23T22:54:44.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:44 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T22:54:44.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:44 smithi012 ceph-mon[31695]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:54:44.406 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:44 smithi173 ceph-mon[36116]: purged_snaps scrub starts 2024-03-23T22:54:44.406 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:44 smithi173 ceph-mon[36116]: purged_snaps scrub ok 2024-03-23T22:54:44.406 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:44 smithi173 ceph-mon[36116]: osd.3 [v2:172.21.15.12:6810/1761446905,v1:172.21.15.12:6811/1761446905] boot 2024-03-23T22:54:44.406 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:44 smithi173 ceph-mon[36116]: osdmap e26: 4 total, 4 up, 4 in 2024-03-23T22:54:44.406 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:44 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T22:54:44.406 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:44 smithi173 ceph-mon[36116]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:54:44.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:44 smithi203 ceph-mon[36693]: purged_snaps scrub starts 2024-03-23T22:54:44.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:44 smithi203 ceph-mon[36693]: purged_snaps scrub ok 2024-03-23T22:54:44.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:44 smithi203 ceph-mon[36693]: osd.3 [v2:172.21.15.12:6810/1761446905,v1:172.21.15.12:6811/1761446905] boot 2024-03-23T22:54:44.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:44 smithi203 ceph-mon[36693]: osdmap e26: 4 total, 4 up, 4 in 2024-03-23T22:54:44.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:44 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T22:54:44.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:44 smithi203 ceph-mon[36693]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:54:45.165 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-23T22:54:45.165 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-23T22:54:45.165 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-03-23T22:54:45.165 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-03-23T22:54:45.165 INFO:teuthology.orchestra.run.smithi173.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0126158 s, 831 MB/s 2024-03-23T22:54:45.165 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-03-23T22:54:45.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:45 smithi012 ceph-mon[31695]: osdmap e27: 4 total, 4 up, 4 in 2024-03-23T22:54:45.471 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:45 smithi173 ceph-mon[36116]: osdmap e27: 4 total, 4 up, 4 in 2024-03-23T22:54:45.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:45 smithi203 ceph-mon[36693]: osdmap e27: 4 total, 4 up, 4 in 2024-03-23T22:54:45.927 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi173:vg_nvme/lv_3 2024-03-23T22:54:46.171 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.b/config 2024-03-23T22:54:46.431 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:46 smithi173 ceph-mon[36116]: osdmap e28: 4 total, 4 up, 4 in 2024-03-23T22:54:46.431 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:46 smithi173 ceph-mon[36116]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:54:46.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:46 smithi012 ceph-mon[31695]: osdmap e28: 4 total, 4 up, 4 in 2024-03-23T22:54:46.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:46 smithi012 ceph-mon[31695]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:54:46.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:46 smithi203 ceph-mon[36693]: osdmap e28: 4 total, 4 up, 4 in 2024-03-23T22:54:46.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:46 smithi203 ceph-mon[36693]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:54:47.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:47 smithi012 ceph-mon[31695]: osdmap e29: 4 total, 4 up, 4 in 2024-03-23T22:54:47.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:47 smithi012 ceph-mon[31695]: osdmap e30: 4 total, 4 up, 4 in 2024-03-23T22:54:47.468 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:47 smithi173 ceph-mon[36116]: osdmap e29: 4 total, 4 up, 4 in 2024-03-23T22:54:47.468 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:47 smithi173 ceph-mon[36116]: osdmap e30: 4 total, 4 up, 4 in 2024-03-23T22:54:47.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:47 smithi203 ceph-mon[36693]: osdmap e29: 4 total, 4 up, 4 in 2024-03-23T22:54:47.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:47 smithi203 ceph-mon[36693]: osdmap e30: 4 total, 4 up, 4 in 2024-03-23T22:54:47.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.806+0000 7f68213d7700 1 -- 172.21.15.173:0/2461652902 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f681c102c90 msgr2=0x7f681c109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:54:47.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.806+0000 7f68213d7700 1 --2- 172.21.15.173:0/2461652902 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f681c102c90 0x7f681c109540 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f681400b0d0 tx=0x7f681400b3e0 comp rx=0 tx=0).stop 2024-03-23T22:54:47.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.806+0000 7f68213d7700 1 -- 172.21.15.173:0/2461652902 shutdown_connections 2024-03-23T22:54:47.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.806+0000 7f68213d7700 1 --2- 172.21.15.173:0/2461652902 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f681c1010f0 0x7f681c101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:54:47.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.806+0000 7f68213d7700 1 --2- 172.21.15.173:0/2461652902 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f681c102c90 0x7f681c109540 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:54:47.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.806+0000 7f68213d7700 1 --2- 172.21.15.173:0/2461652902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f681c1022f0 0x7f681c102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:54:47.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.807+0000 7f68213d7700 1 -- 172.21.15.173:0/2461652902 >> 172.21.15.173:0/2461652902 conn(0x7f681c0fc6b0 msgr2=0x7f681c0fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:54:47.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.807+0000 7f68213d7700 1 -- 172.21.15.173:0/2461652902 shutdown_connections 2024-03-23T22:54:47.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.807+0000 7f68213d7700 1 -- 172.21.15.173:0/2461652902 wait complete. 2024-03-23T22:54:47.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.807+0000 7f68213d7700 1 Processor -- start 2024-03-23T22:54:47.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.807+0000 7f68213d7700 1 -- start start 2024-03-23T22:54:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.808+0000 7f68213d7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f681c1010f0 0x7f681c198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:54:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.808+0000 7f68213d7700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f681c1022f0 0x7f681c198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:54:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.808+0000 7f68213d7700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f681c102c90 0x7f681c19fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:54:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.808+0000 7f68213d7700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c10cdd0 con 0x7f681c102c90 2024-03-23T22:54:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.808+0000 7f68213d7700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c10cc50 con 0x7f681c1010f0 2024-03-23T22:54:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.808+0000 7f68213d7700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c10cf50 con 0x7f681c1022f0 2024-03-23T22:54:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.808+0000 7f681affd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f681c1010f0 0x7f681c198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:54:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.808+0000 7f681affd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f681c1010f0 0x7f681c198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:35660/0 (socket says 172.21.15.173:35660) 2024-03-23T22:54:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.808+0000 7f681affd700 1 -- 172.21.15.173:0/4014905178 learned_addr learned my addr 172.21.15.173:0/4014905178 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-23T22:54:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.808+0000 7f681a7fc700 1 --2- 172.21.15.173:0/4014905178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f681c1022f0 0x7f681c198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:54:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.808+0000 7f681affd700 1 -- 172.21.15.173:0/4014905178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f681c1022f0 msgr2=0x7f681c198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:54:47.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.808+0000 7f681affd700 1 --2- 172.21.15.173:0/4014905178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f681c1022f0 0x7f681c198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:54:47.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.808+0000 7f681affd700 1 -- 172.21.15.173:0/4014905178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f681c102c90 msgr2=0x7f681c19fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:54:47.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.808+0000 7f681affd700 1 --2- 172.21.15.173:0/4014905178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f681c102c90 0x7f681c19fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:54:47.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.808+0000 7f681affd700 1 -- 172.21.15.173:0/4014905178 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c1a0170 con 0x7f681c1010f0 2024-03-23T22:54:47.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.808+0000 7f681affd700 1 --2- 172.21.15.173:0/4014905178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f681c1010f0 0x7f681c198510 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f680c00b500 tx=0x7f680c00b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:54:47.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.808+0000 7f6803fff700 1 -- 172.21.15.173:0/4014905178 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c015020 con 0x7f681c1010f0 2024-03-23T22:54:47.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.808+0000 7f6803fff700 1 -- 172.21.15.173:0/4014905178 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f680c011d10 con 0x7f681c1010f0 2024-03-23T22:54:47.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.808+0000 7f68213d7700 1 -- 172.21.15.173:0/4014905178 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f681c1a0460 con 0x7f681c1010f0 2024-03-23T22:54:47.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.808+0000 7f6803fff700 1 -- 172.21.15.173:0/4014905178 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c00f2e0 con 0x7f681c1010f0 2024-03-23T22:54:47.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.808+0000 7f68213d7700 1 -- 172.21.15.173:0/4014905178 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681c1a0a70 con 0x7f681c1010f0 2024-03-23T22:54:47.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.810+0000 7f68213d7700 1 -- 172.21.15.173:0/4014905178 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681c068e50 con 0x7f681c1010f0 2024-03-23T22:54:47.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.810+0000 7f6803fff700 1 -- 172.21.15.173:0/4014905178 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f680c018020 con 0x7f681c1010f0 2024-03-23T22:54:47.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.811+0000 7f6803fff700 1 --2- 172.21.15.173:0/4014905178 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f68040a2910 0x7f68040a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:54:47.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.811+0000 7f6803fff700 1 -- 172.21.15.173:0/4014905178 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3669+0+0 (secure 0 0 0) 0x7f680c0ccf80 con 0x7f681c1010f0 2024-03-23T22:54:47.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.811+0000 7f681a7fc700 1 --2- 172.21.15.173:0/4014905178 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f68040a2910 0x7f68040a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:54:47.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.813+0000 7f681a7fc700 1 --2- 172.21.15.173:0/4014905178 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f68040a2910 0x7f68040a4dd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6810006fd0 tx=0x7f6810008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:54:47.815 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:47.815+0000 7f6803fff700 1 -- 172.21.15.173:0/4014905178 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f680c01f020 con 0x7f681c1010f0 2024-03-23T22:54:48.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:54:48.024+0000 7f68213d7700 1 -- 172.21.15.173:0/4014905178 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f681c1a0d30 con 0x7f68040a2910 2024-03-23T22:54:48.958 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:48 smithi173 ceph-mon[36116]: pgmap v122: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-23T22:54:48.958 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:48 smithi173 ceph-mon[36116]: Detected new or changed devices on smithi012 2024-03-23T22:54:48.958 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:48 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:48.959 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:48 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:48.959 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:48 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:54:48.959 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:48 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:54:48.959 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:48 smithi173 ceph-mon[36116]: Adjusting osd_memory_target on smithi012 to 8547M 2024-03-23T22:54:48.959 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:48 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:48.959 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:48 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:54:48.959 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:48 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:54:48.959 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:48 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:48.959 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:48 smithi173 ceph-mon[36116]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:54:48.959 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:48 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:54:48.959 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:48 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:54:48.959 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:48 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:54:49.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:48 smithi203 ceph-mon[36693]: pgmap v122: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-23T22:54:49.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:48 smithi203 ceph-mon[36693]: Detected new or changed devices on smithi012 2024-03-23T22:54:49.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:48 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:49.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:48 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:49.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:48 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:54:49.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:48 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:54:49.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:48 smithi203 ceph-mon[36693]: Adjusting osd_memory_target on smithi012 to 8547M 2024-03-23T22:54:49.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:48 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:49.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:48 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:54:49.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:48 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:54:49.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:48 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:49.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:48 smithi203 ceph-mon[36693]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:54:49.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:48 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:54:49.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:48 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:54:49.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:48 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:54:49.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:48 smithi012 ceph-mon[31695]: pgmap v122: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-23T22:54:49.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:48 smithi012 ceph-mon[31695]: Detected new or changed devices on smithi012 2024-03-23T22:54:49.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:48 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:49.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:48 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:49.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:48 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:54:49.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:48 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:54:49.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:48 smithi012 ceph-mon[31695]: Adjusting osd_memory_target on smithi012 to 8547M 2024-03-23T22:54:49.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:48 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:49.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:48 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:54:49.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:48 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:54:49.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:48 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:54:49.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:48 smithi012 ceph-mon[31695]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:54:49.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:48 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:54:49.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:48 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:54:49.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:48 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:54:51.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:50 smithi173 ceph-mon[36116]: pgmap v123: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-03-23T22:54:51.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:50 smithi173 ceph-mon[36116]: from='client.? 172.21.15.173:0/508913421' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c48f886-0b42-4a09-b31e-abccc200bb1d"}]: dispatch 2024-03-23T22:54:51.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:50 smithi173 ceph-mon[36116]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c48f886-0b42-4a09-b31e-abccc200bb1d"}]: dispatch 2024-03-23T22:54:51.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:50 smithi173 ceph-mon[36116]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c48f886-0b42-4a09-b31e-abccc200bb1d"}]': finished 2024-03-23T22:54:51.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:50 smithi173 ceph-mon[36116]: osdmap e31: 5 total, 4 up, 5 in 2024-03-23T22:54:51.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:50 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T22:54:51.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:50 smithi203 ceph-mon[36693]: pgmap v123: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-03-23T22:54:51.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:50 smithi203 ceph-mon[36693]: from='client.? 172.21.15.173:0/508913421' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c48f886-0b42-4a09-b31e-abccc200bb1d"}]: dispatch 2024-03-23T22:54:51.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:50 smithi203 ceph-mon[36693]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c48f886-0b42-4a09-b31e-abccc200bb1d"}]: dispatch 2024-03-23T22:54:51.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:50 smithi203 ceph-mon[36693]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c48f886-0b42-4a09-b31e-abccc200bb1d"}]': finished 2024-03-23T22:54:51.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:50 smithi203 ceph-mon[36693]: osdmap e31: 5 total, 4 up, 5 in 2024-03-23T22:54:51.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:50 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T22:54:51.162 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:50 smithi012 ceph-mon[31695]: pgmap v123: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-03-23T22:54:51.162 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:50 smithi012 ceph-mon[31695]: from='client.? 172.21.15.173:0/508913421' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c48f886-0b42-4a09-b31e-abccc200bb1d"}]: dispatch 2024-03-23T22:54:51.162 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:50 smithi012 ceph-mon[31695]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c48f886-0b42-4a09-b31e-abccc200bb1d"}]: dispatch 2024-03-23T22:54:51.162 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:50 smithi012 ceph-mon[31695]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c48f886-0b42-4a09-b31e-abccc200bb1d"}]': finished 2024-03-23T22:54:51.162 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:50 smithi012 ceph-mon[31695]: osdmap e31: 5 total, 4 up, 5 in 2024-03-23T22:54:51.162 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:50 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T22:54:52.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:51 smithi203 ceph-mon[36693]: from='client.? 172.21.15.173:0/1791044367' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:54:52.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:51 smithi012 ceph-mon[31695]: from='client.? 172.21.15.173:0/1791044367' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:54:52.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:51 smithi173 ceph-mon[36116]: from='client.? 172.21.15.173:0/1791044367' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:54:53.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:52 smithi203 ceph-mon[36693]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-23T22:54:53.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:52 smithi012 ceph-mon[31695]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-23T22:54:53.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:52 smithi173 ceph-mon[36116]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-23T22:54:55.071 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:54 smithi173 ceph-mon[36116]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:54:55.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:54 smithi203 ceph-mon[36693]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:54:55.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:54 smithi012 ceph-mon[31695]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:54:56.930 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:56 smithi173 ceph-mon[36116]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:54:57.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:56 smithi203 ceph-mon[36693]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:54:57.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:56 smithi012 ceph-mon[31695]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:54:59.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:58 smithi173 ceph-mon[36116]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:54:59.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:58 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-23T22:54:59.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:58 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:54:59.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:54:58 smithi173 ceph-mon[36116]: Deploying daemon osd.4 on smithi173 2024-03-23T22:54:59.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:58 smithi203 ceph-mon[36693]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:54:59.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:58 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-23T22:54:59.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:58 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:54:59.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:54:58 smithi203 ceph-mon[36693]: Deploying daemon osd.4 on smithi173 2024-03-23T22:54:59.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:58 smithi012 ceph-mon[31695]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:54:59.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:58 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-23T22:54:59.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:58 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:54:59.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:54:58 smithi012 ceph-mon[31695]: Deploying daemon osd.4 on smithi173 2024-03-23T22:55:01.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:00 smithi173 ceph-mon[36116]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:55:01.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:00 smithi203 ceph-mon[36693]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:55:01.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:00 smithi012 ceph-mon[31695]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:55:03.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:02 smithi173 ceph-mon[36116]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:55:03.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:02 smithi203 ceph-mon[36693]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:55:03.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:02 smithi012 ceph-mon[31695]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:55:04.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:04 smithi012 ceph-mon[31695]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:55:04.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:04 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:04.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:04 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:04.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:04 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:55:04.805 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:04 smithi173 ceph-mon[36116]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:55:04.805 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:04 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:04.805 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:04 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:04.805 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:04 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:55:04.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:04 smithi203 ceph-mon[36693]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:55:04.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:04 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:04.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:04 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:04.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:04 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:55:06.532 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:06 smithi173 ceph-mon[36116]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:55:06.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:06 smithi012 ceph-mon[31695]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:55:06.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:55:06.854+0000 7f6803fff700 1 -- 172.21.15.173:0/4014905178 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f681c1a0d30 con 0x7f68040a2910 2024-03-23T22:55:06.854 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 4 on host 'smithi173' 2024-03-23T22:55:06.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:55:06.856+0000 7f68213d7700 1 -- 172.21.15.173:0/4014905178 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f68040a2910 msgr2=0x7f68040a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:55:06.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:55:06.856+0000 7f68213d7700 1 --2- 172.21.15.173:0/4014905178 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f68040a2910 0x7f68040a4dd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6810006fd0 tx=0x7f6810008040 comp rx=0 tx=0).stop 2024-03-23T22:55:06.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:55:06.856+0000 7f68213d7700 1 -- 172.21.15.173:0/4014905178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f681c1010f0 msgr2=0x7f681c198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:55:06.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:55:06.856+0000 7f68213d7700 1 --2- 172.21.15.173:0/4014905178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f681c1010f0 0x7f681c198510 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f680c00b500 tx=0x7f680c00b8c0 comp rx=0 tx=0).stop 2024-03-23T22:55:06.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:55:06.856+0000 7f68213d7700 1 -- 172.21.15.173:0/4014905178 shutdown_connections 2024-03-23T22:55:06.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:55:06.856+0000 7f68213d7700 1 --2- 172.21.15.173:0/4014905178 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f68040a2910 0x7f68040a4dd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:55:06.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:55:06.856+0000 7f68213d7700 1 --2- 172.21.15.173:0/4014905178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f681c102c90 0x7f681c19fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:55:06.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:55:06.856+0000 7f68213d7700 1 --2- 172.21.15.173:0/4014905178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f681c1010f0 0x7f681c198510 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:55:06.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:55:06.856+0000 7f68213d7700 1 --2- 172.21.15.173:0/4014905178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f681c1022f0 0x7f681c198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:55:06.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:55:06.856+0000 7f68213d7700 1 -- 172.21.15.173:0/4014905178 >> 172.21.15.173:0/4014905178 conn(0x7f681c0fc6b0 msgr2=0x7f681c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:55:06.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:55:06.856+0000 7f68213d7700 1 -- 172.21.15.173:0/4014905178 shutdown_connections 2024-03-23T22:55:06.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:55:06.857+0000 7f68213d7700 1 -- 172.21.15.173:0/4014905178 wait complete. 2024-03-23T22:55:06.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:06 smithi203 ceph-mon[36693]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:55:07.420 DEBUG:teuthology.orchestra.run.smithi173:osd.4> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.4.service 2024-03-23T22:55:07.423 INFO:tasks.cephadm:Deploying osd.5 on smithi203 with /dev/vg_nvme/lv_3... 2024-03-23T22:55:07.423 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-23T22:55:07.670 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.c/config 2024-03-23T22:55:08.118 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:07 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:08.118 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:07 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:08.118 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:07 smithi203 ceph-mon[36693]: from='osd.4 [v2:172.21.15.173:6808/242958272,v1:172.21.15.173:6809/242958272]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-23T22:55:08.118 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:07 smithi203 ceph-mon[36693]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-23T22:55:08.118 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:07 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:08.119 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:07 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:08.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:07 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:08.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:07 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:08.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:07 smithi173 ceph-mon[36116]: from='osd.4 [v2:172.21.15.173:6808/242958272,v1:172.21.15.173:6809/242958272]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-23T22:55:08.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:07 smithi173 ceph-mon[36116]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-23T22:55:08.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:07 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:08.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:07 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:08.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:07 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:08.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:07 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:08.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:07 smithi012 ceph-mon[31695]: from='osd.4 [v2:172.21.15.173:6808/242958272,v1:172.21.15.173:6809/242958272]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-23T22:55:08.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:07 smithi012 ceph-mon[31695]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-23T22:55:08.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:07 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:08.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:07 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:09.093 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:08 smithi173 ceph-mon[36116]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:55:09.093 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:08 smithi173 ceph-mon[36116]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-23T22:55:09.093 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:08 smithi173 ceph-mon[36116]: from='osd.4 [v2:172.21.15.173:6808/242958272,v1:172.21.15.173:6809/242958272]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-23T22:55:09.093 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:08 smithi173 ceph-mon[36116]: osdmap e32: 5 total, 4 up, 5 in 2024-03-23T22:55:09.093 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:08 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T22:55:09.093 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:08 smithi173 ceph-mon[36116]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-23T22:55:09.140 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:08 smithi203 ceph-mon[36693]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:55:09.140 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:08 smithi203 ceph-mon[36693]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-23T22:55:09.140 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:08 smithi203 ceph-mon[36693]: from='osd.4 [v2:172.21.15.173:6808/242958272,v1:172.21.15.173:6809/242958272]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-23T22:55:09.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:08 smithi203 ceph-mon[36693]: osdmap e32: 5 total, 4 up, 5 in 2024-03-23T22:55:09.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:08 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T22:55:09.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:08 smithi203 ceph-mon[36693]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-23T22:55:09.198 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:08 smithi012 ceph-mon[31695]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:55:09.198 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:08 smithi012 ceph-mon[31695]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-23T22:55:09.198 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:08 smithi012 ceph-mon[31695]: from='osd.4 [v2:172.21.15.173:6808/242958272,v1:172.21.15.173:6809/242958272]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-23T22:55:09.198 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:08 smithi012 ceph-mon[31695]: osdmap e32: 5 total, 4 up, 5 in 2024-03-23T22:55:09.198 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:08 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T22:55:09.198 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:08 smithi012 ceph-mon[31695]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-23T22:55:09.368 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 23 22:55:09 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-4[47541]: 2024-03-23T22:55:09.299+0000 7f3fd0fe9700 -1 osd.4 0 waiting for initial osdmap 2024-03-23T22:55:09.368 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 23 22:55:09 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-4[47541]: 2024-03-23T22:55:09.305+0000 7f3fc85d3700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T22:55:09.707 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-23T22:55:09.707 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-23T22:55:09.707 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-03-23T22:55:09.708 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-03-23T22:55:09.708 INFO:teuthology.orchestra.run.smithi203.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113801 s, 921 MB/s 2024-03-23T22:55:09.708 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 2024-03-23T22:55:09.708 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-03-23T22:55:10.025 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:09 smithi203 ceph-mon[36693]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-23T22:55:10.025 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:09 smithi203 ceph-mon[36693]: osdmap e33: 5 total, 4 up, 5 in 2024-03-23T22:55:10.026 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:09 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T22:55:10.026 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:09 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T22:55:10.026 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:09 smithi203 ceph-mon[36693]: from='osd.4 ' entity='osd.4' 2024-03-23T22:55:10.211 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:09 smithi173 ceph-mon[36116]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-23T22:55:10.211 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:09 smithi173 ceph-mon[36116]: osdmap e33: 5 total, 4 up, 5 in 2024-03-23T22:55:10.212 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:09 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T22:55:10.212 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:09 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T22:55:10.212 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:09 smithi173 ceph-mon[36116]: from='osd.4 ' entity='osd.4' 2024-03-23T22:55:10.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:09 smithi012 ceph-mon[31695]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-23T22:55:10.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:09 smithi012 ceph-mon[31695]: osdmap e33: 5 total, 4 up, 5 in 2024-03-23T22:55:10.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:09 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T22:55:10.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:09 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T22:55:10.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:09 smithi012 ceph-mon[31695]: from='osd.4 ' entity='osd.4' 2024-03-23T22:55:10.340 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi203:vg_nvme/lv_3 2024-03-23T22:55:10.586 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.c/config 2024-03-23T22:55:10.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:10 smithi203 ceph-mon[36693]: purged_snaps scrub starts 2024-03-23T22:55:10.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:10 smithi203 ceph-mon[36693]: purged_snaps scrub ok 2024-03-23T22:55:10.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:10 smithi203 ceph-mon[36693]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:55:10.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:10 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T22:55:10.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:10 smithi203 ceph-mon[36693]: osd.4 [v2:172.21.15.173:6808/242958272,v1:172.21.15.173:6809/242958272] boot 2024-03-23T22:55:10.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:10 smithi203 ceph-mon[36693]: osdmap e34: 5 total, 5 up, 5 in 2024-03-23T22:55:10.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:10 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T22:55:11.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:10 smithi012 ceph-mon[31695]: purged_snaps scrub starts 2024-03-23T22:55:11.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:10 smithi012 ceph-mon[31695]: purged_snaps scrub ok 2024-03-23T22:55:11.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:10 smithi012 ceph-mon[31695]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:55:11.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:10 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T22:55:11.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:10 smithi012 ceph-mon[31695]: osd.4 [v2:172.21.15.173:6808/242958272,v1:172.21.15.173:6809/242958272] boot 2024-03-23T22:55:11.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:10 smithi012 ceph-mon[31695]: osdmap e34: 5 total, 5 up, 5 in 2024-03-23T22:55:11.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:10 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T22:55:11.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:10 smithi173 ceph-mon[36116]: purged_snaps scrub starts 2024-03-23T22:55:11.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:10 smithi173 ceph-mon[36116]: purged_snaps scrub ok 2024-03-23T22:55:11.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:10 smithi173 ceph-mon[36116]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-23T22:55:11.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:10 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T22:55:11.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:10 smithi173 ceph-mon[36116]: osd.4 [v2:172.21.15.173:6808/242958272,v1:172.21.15.173:6809/242958272] boot 2024-03-23T22:55:11.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:10 smithi173 ceph-mon[36116]: osdmap e34: 5 total, 5 up, 5 in 2024-03-23T22:55:11.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:10 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T22:55:12.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.187+0000 7fcb5f3bc700 1 -- 172.21.15.203:0/2853185162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb58105580 msgr2=0x7fcb58107970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:55:12.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.187+0000 7fcb5f3bc700 1 --2- 172.21.15.203:0/2853185162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb58105580 0x7fcb58107970 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcb48009990 tx=0x7fcb48009ca0 comp rx=0 tx=0).stop 2024-03-23T22:55:12.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.188+0000 7fcb5f3bc700 1 -- 172.21.15.203:0/2853185162 shutdown_connections 2024-03-23T22:55:12.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.188+0000 7fcb5f3bc700 1 --2- 172.21.15.203:0/2853185162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb58107eb0 0x7fcb5810a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:55:12.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.188+0000 7fcb5f3bc700 1 --2- 172.21.15.203:0/2853185162 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb58102c50 0x7fcb58105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:55:12.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.188+0000 7fcb5f3bc700 1 --2- 172.21.15.203:0/2853185162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb58105580 0x7fcb58107970 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:55:12.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.188+0000 7fcb5f3bc700 1 -- 172.21.15.203:0/2853185162 >> 172.21.15.203:0/2853185162 conn(0x7fcb580fc680 msgr2=0x7fcb580feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:55:12.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.188+0000 7fcb5f3bc700 1 -- 172.21.15.203:0/2853185162 shutdown_connections 2024-03-23T22:55:12.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.188+0000 7fcb5f3bc700 1 -- 172.21.15.203:0/2853185162 wait complete. 2024-03-23T22:55:12.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.188+0000 7fcb5f3bc700 1 Processor -- start 2024-03-23T22:55:12.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.188+0000 7fcb5f3bc700 1 -- start start 2024-03-23T22:55:12.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.188+0000 7fcb5f3bc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb58102c50 0x7fcb58198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:55:12.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.188+0000 7fcb5f3bc700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb58105580 0x7fcb58198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:55:12.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.188+0000 7fcb5f3bc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb58107eb0 0x7fcb5819fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:55:12.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.188+0000 7fcb5f3bc700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5810de10 con 0x7fcb58105580 2024-03-23T22:55:12.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.188+0000 7fcb5f3bc700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5810dc90 con 0x7fcb58102c50 2024-03-23T22:55:12.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.188+0000 7fcb5f3bc700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5810df90 con 0x7fcb58107eb0 2024-03-23T22:55:12.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.188+0000 7fcb5d959700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb58107eb0 0x7fcb5819fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:55:12.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.188+0000 7fcb5d959700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb58107eb0 0x7fcb5819fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:48630/0 (socket says 172.21.15.203:48630) 2024-03-23T22:55:12.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.188+0000 7fcb5d959700 1 -- 172.21.15.203:0/126797863 learned_addr learned my addr 172.21.15.203:0/126797863 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-23T22:55:12.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.189+0000 7fcb5d158700 1 --2- 172.21.15.203:0/126797863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb58102c50 0x7fcb58198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:55:12.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.189+0000 7fcb5d959700 1 -- 172.21.15.203:0/126797863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb58102c50 msgr2=0x7fcb58198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:55:12.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.189+0000 7fcb5d959700 1 --2- 172.21.15.203:0/126797863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb58102c50 0x7fcb58198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:55:12.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.189+0000 7fcb5d959700 1 -- 172.21.15.203:0/126797863 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb58105580 msgr2=0x7fcb58198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:55:12.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.189+0000 7fcb5d959700 1 --2- 172.21.15.203:0/126797863 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb58105580 0x7fcb58198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:55:12.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.189+0000 7fcb5d959700 1 -- 172.21.15.203:0/126797863 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb581a00b0 con 0x7fcb58107eb0 2024-03-23T22:55:12.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.189+0000 7fcb5d158700 1 --2- 172.21.15.203:0/126797863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb58102c50 0x7fcb58198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T22:55:12.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.189+0000 7fcb5d959700 1 --2- 172.21.15.203:0/126797863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb58107eb0 0x7fcb5819fa10 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcb50007ee0 tx=0x7fcb5000d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:55:12.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.189+0000 7fcb4e7fc700 1 -- 172.21.15.203:0/126797863 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb50004db0 con 0x7fcb58107eb0 2024-03-23T22:55:12.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.189+0000 7fcb5f3bc700 1 -- 172.21.15.203:0/126797863 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb581a03a0 con 0x7fcb58107eb0 2024-03-23T22:55:12.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.189+0000 7fcb4e7fc700 1 -- 172.21.15.203:0/126797863 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcb5000dbd0 con 0x7fcb58107eb0 2024-03-23T22:55:12.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.189+0000 7fcb4e7fc700 1 -- 172.21.15.203:0/126797863 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb50015660 con 0x7fcb58107eb0 2024-03-23T22:55:12.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.189+0000 7fcb5f3bc700 1 -- 172.21.15.203:0/126797863 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb581a09b0 con 0x7fcb58107eb0 2024-03-23T22:55:12.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.190+0000 7fcb5f3bc700 1 -- 172.21.15.203:0/126797863 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb58068e50 con 0x7fcb58107eb0 2024-03-23T22:55:12.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.191+0000 7fcb4e7fc700 1 -- 172.21.15.203:0/126797863 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcb50005020 con 0x7fcb58107eb0 2024-03-23T22:55:12.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.192+0000 7fcb4e7fc700 1 --2- 172.21.15.203:0/126797863 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcb440a2890 0x7fcb440a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:55:12.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.192+0000 7fcb4e7fc700 1 -- 172.21.15.203:0/126797863 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4101+0+0 (secure 0 0 0) 0x7fcb500cd4f0 con 0x7fcb58107eb0 2024-03-23T22:55:12.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.192+0000 7fcb5d158700 1 --2- 172.21.15.203:0/126797863 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcb440a2890 0x7fcb440a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:55:12.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.193+0000 7fcb5d158700 1 --2- 172.21.15.203:0/126797863 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcb440a2890 0x7fcb440a4d50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcb540099d0 tx=0x7fcb54008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:55:12.195 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.196+0000 7fcb4e7fc700 1 -- 172.21.15.203:0/126797863 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcb50010070 con 0x7fcb58107eb0 2024-03-23T22:55:12.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:12 smithi173 ceph-mon[36116]: Detected new or changed devices on smithi173 2024-03-23T22:55:12.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:12 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:12.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:12 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:12.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:12 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:55:12.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:12 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:55:12.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:12 smithi173 ceph-mon[36116]: Adjusting osd_memory_target on smithi173 to 8540M 2024-03-23T22:55:12.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:12 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:12.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:12 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:55:12.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:12 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:55:12.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:12 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:12.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:12 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:55:12.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:12 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:55:12.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:12 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:55:12.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:12 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:12.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:12.400+0000 7fcb5f3bc700 1 -- 172.21.15.203:0/126797863 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb580625d0 con 0x7fcb440a2890 2024-03-23T22:55:12.400 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:12 smithi203 ceph-mon[36693]: Detected new or changed devices on smithi173 2024-03-23T22:55:12.400 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:12 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:12.400 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:12 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:12.400 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:12 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:55:12.400 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:12 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:55:12.400 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:12 smithi203 ceph-mon[36693]: Adjusting osd_memory_target on smithi173 to 8540M 2024-03-23T22:55:12.400 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:12 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:12.401 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:12 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:55:12.401 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:12 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:55:12.401 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:12 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:12.401 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:12 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:55:12.401 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:12 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:55:12.401 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:12 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:55:12.401 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:12 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:12.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:12 smithi012 ceph-mon[31695]: Detected new or changed devices on smithi173 2024-03-23T22:55:12.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:12 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:12.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:12 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:12.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:12 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:55:12.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:12 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:55:12.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:12 smithi012 ceph-mon[31695]: Adjusting osd_memory_target on smithi173 to 8540M 2024-03-23T22:55:12.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:12 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:12.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:12 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:55:12.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:12 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:55:12.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:12 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:12.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:12 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:55:12.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:12 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:55:12.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:12 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:55:12.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:12 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:13.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:13 smithi173 ceph-mon[36116]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:13.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:13 smithi173 ceph-mon[36116]: osdmap e35: 5 total, 5 up, 5 in 2024-03-23T22:55:13.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:13 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:55:13.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:13 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:55:13.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:13 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:55:13.369 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:13 smithi203 ceph-mon[36693]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:13.369 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:13 smithi203 ceph-mon[36693]: osdmap e35: 5 total, 5 up, 5 in 2024-03-23T22:55:13.369 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:13 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:55:13.369 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:13 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:55:13.369 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:13 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:55:13.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:13 smithi012 ceph-mon[31695]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:13.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:13 smithi012 ceph-mon[31695]: osdmap e35: 5 total, 5 up, 5 in 2024-03-23T22:55:13.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:13 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:55:13.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:13 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:55:13.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:13 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:55:14.303 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:14 smithi203 ceph-mon[36693]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:55:14.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:14 smithi012 ceph-mon[31695]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:55:14.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:14 smithi173 ceph-mon[36116]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:55:15.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:15 smithi203 ceph-mon[36693]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:15.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:15 smithi012 ceph-mon[31695]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:15.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:15 smithi173 ceph-mon[36116]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:16.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:16 smithi203 ceph-mon[36693]: from='client.? 172.21.15.203:0/362604378' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a34f06f-8805-40c0-b1ab-23407b0276d4"}]: dispatch 2024-03-23T22:55:16.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:16 smithi203 ceph-mon[36693]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a34f06f-8805-40c0-b1ab-23407b0276d4"}]: dispatch 2024-03-23T22:55:16.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:16 smithi203 ceph-mon[36693]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a34f06f-8805-40c0-b1ab-23407b0276d4"}]': finished 2024-03-23T22:55:16.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:16 smithi203 ceph-mon[36693]: osdmap e36: 6 total, 5 up, 6 in 2024-03-23T22:55:16.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:16 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T22:55:16.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:16 smithi203 ceph-mon[36693]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:16.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:16 smithi203 ceph-mon[36693]: from='client.? 172.21.15.203:0/2985416916' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:55:16.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:16 smithi012 ceph-mon[31695]: from='client.? 172.21.15.203:0/362604378' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a34f06f-8805-40c0-b1ab-23407b0276d4"}]: dispatch 2024-03-23T22:55:16.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:16 smithi012 ceph-mon[31695]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a34f06f-8805-40c0-b1ab-23407b0276d4"}]: dispatch 2024-03-23T22:55:16.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:16 smithi012 ceph-mon[31695]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a34f06f-8805-40c0-b1ab-23407b0276d4"}]': finished 2024-03-23T22:55:16.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:16 smithi012 ceph-mon[31695]: osdmap e36: 6 total, 5 up, 6 in 2024-03-23T22:55:16.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:16 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T22:55:16.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:16 smithi012 ceph-mon[31695]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:16.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:16 smithi012 ceph-mon[31695]: from='client.? 172.21.15.203:0/2985416916' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:55:16.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:16 smithi173 ceph-mon[36116]: from='client.? 172.21.15.203:0/362604378' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a34f06f-8805-40c0-b1ab-23407b0276d4"}]: dispatch 2024-03-23T22:55:16.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:16 smithi173 ceph-mon[36116]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a34f06f-8805-40c0-b1ab-23407b0276d4"}]: dispatch 2024-03-23T22:55:16.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:16 smithi173 ceph-mon[36116]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a34f06f-8805-40c0-b1ab-23407b0276d4"}]': finished 2024-03-23T22:55:16.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:16 smithi173 ceph-mon[36116]: osdmap e36: 6 total, 5 up, 6 in 2024-03-23T22:55:16.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:16 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T22:55:16.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:16 smithi173 ceph-mon[36116]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:16.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:16 smithi173 ceph-mon[36116]: from='client.? 172.21.15.203:0/2985416916' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:55:18.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:18 smithi012 ceph-mon[31695]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:18.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:18 smithi173 ceph-mon[36116]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:18.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:18 smithi203 ceph-mon[36693]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:20.329 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:20 smithi203 ceph-mon[36693]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:20.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:20 smithi012 ceph-mon[31695]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:20.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:20 smithi173 ceph-mon[36116]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:22.515 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:22 smithi203 ceph-mon[36693]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:22.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:22 smithi012 ceph-mon[31695]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:22.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:22 smithi173 ceph-mon[36116]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:23.433 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:23 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-23T22:55:23.433 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:23 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:55:23.433 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:23 smithi203 ceph-mon[36693]: Deploying daemon osd.5 on smithi203 2024-03-23T22:55:23.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:23 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-23T22:55:23.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:23 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:55:23.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:23 smithi012 ceph-mon[31695]: Deploying daemon osd.5 on smithi203 2024-03-23T22:55:23.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:23 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-23T22:55:23.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:23 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:55:23.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:23 smithi173 ceph-mon[36116]: Deploying daemon osd.5 on smithi203 2024-03-23T22:55:24.520 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:24 smithi203 ceph-mon[36693]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:24.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:24 smithi012 ceph-mon[31695]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:24.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:24 smithi173 ceph-mon[36116]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:26.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:26 smithi203 ceph-mon[36693]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:26.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:26 smithi012 ceph-mon[31695]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:26.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:26 smithi173 ceph-mon[36116]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:28.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:28 smithi203 ceph-mon[36693]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:28.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:28 smithi012 ceph-mon[31695]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:28.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:28 smithi173 ceph-mon[36116]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:29.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:29 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:29.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:29 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:29.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:29 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:55:29.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:29 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:29.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:29 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:29.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:29 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:55:29.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:29 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:29.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:29 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:29.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:29 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:55:30.648 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:30 smithi203 ceph-mon[36693]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:30.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:30 smithi012 ceph-mon[31695]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:30.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:30 smithi173 ceph-mon[36116]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:31.836 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:31.836+0000 7fcb4e7fc700 1 -- 172.21.15.203:0/126797863 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fcb580625d0 con 0x7fcb440a2890 2024-03-23T22:55:31.836 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 5 on host 'smithi203' 2024-03-23T22:55:31.838 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:31.839+0000 7fcb5f3bc700 1 -- 172.21.15.203:0/126797863 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcb440a2890 msgr2=0x7fcb440a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:55:31.838 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:31.839+0000 7fcb5f3bc700 1 --2- 172.21.15.203:0/126797863 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcb440a2890 0x7fcb440a4d50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcb540099d0 tx=0x7fcb54008040 comp rx=0 tx=0).stop 2024-03-23T22:55:31.839 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:31.839+0000 7fcb5f3bc700 1 -- 172.21.15.203:0/126797863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb58107eb0 msgr2=0x7fcb5819fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:55:31.839 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:31.839+0000 7fcb5f3bc700 1 --2- 172.21.15.203:0/126797863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb58107eb0 0x7fcb5819fa10 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcb50007ee0 tx=0x7fcb5000d3b0 comp rx=0 tx=0).stop 2024-03-23T22:55:31.839 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:31.839+0000 7fcb5f3bc700 1 -- 172.21.15.203:0/126797863 shutdown_connections 2024-03-23T22:55:31.839 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:31.839+0000 7fcb5f3bc700 1 --2- 172.21.15.203:0/126797863 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcb440a2890 0x7fcb440a4d50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:55:31.839 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:31.839+0000 7fcb5f3bc700 1 --2- 172.21.15.203:0/126797863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb58107eb0 0x7fcb5819fa10 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:55:31.839 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:31.839+0000 7fcb5f3bc700 1 --2- 172.21.15.203:0/126797863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb58102c50 0x7fcb58198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:55:31.839 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:31.839+0000 7fcb5f3bc700 1 --2- 172.21.15.203:0/126797863 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb58105580 0x7fcb58198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:55:31.839 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:31.839+0000 7fcb5f3bc700 1 -- 172.21.15.203:0/126797863 >> 172.21.15.203:0/126797863 conn(0x7fcb580fc680 msgr2=0x7fcb58103510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:55:31.839 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:31.839+0000 7fcb5f3bc700 1 -- 172.21.15.203:0/126797863 shutdown_connections 2024-03-23T22:55:31.839 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:55:31.839+0000 7fcb5f3bc700 1 -- 172.21.15.203:0/126797863 wait complete. 2024-03-23T22:55:32.517 DEBUG:teuthology.orchestra.run.smithi203:osd.5> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.5.service 2024-03-23T22:55:32.520 INFO:tasks.cephadm:Deploying osd.6 on smithi012 with /dev/vg_nvme/lv_2... 2024-03-23T22:55:32.520 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-23T22:55:32.993 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:55:33.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:32 smithi012 ceph-mon[31695]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:33.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:32 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:33.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:32 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:33.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:32 smithi012 ceph-mon[31695]: from='osd.5 [v2:172.21.15.203:6808/4270966092,v1:172.21.15.203:6809/4270966092]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-23T22:55:33.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:32 smithi012 ceph-mon[31695]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-23T22:55:33.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:32 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:33.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:32 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:33.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:32 smithi173 ceph-mon[36116]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:33.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:32 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:33.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:32 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:33.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:32 smithi173 ceph-mon[36116]: from='osd.5 [v2:172.21.15.203:6808/4270966092,v1:172.21.15.203:6809/4270966092]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-23T22:55:33.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:32 smithi173 ceph-mon[36116]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-23T22:55:33.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:32 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:33.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:32 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:33.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:32 smithi203 ceph-mon[36693]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:33.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:32 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:33.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:32 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:33.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:32 smithi203 ceph-mon[36693]: from='osd.5 [v2:172.21.15.203:6808/4270966092,v1:172.21.15.203:6809/4270966092]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-23T22:55:33.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:32 smithi203 ceph-mon[36693]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-23T22:55:33.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:32 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:33.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:32 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:34.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:33 smithi012 ceph-mon[31695]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-23T22:55:34.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:33 smithi012 ceph-mon[31695]: osdmap e37: 6 total, 5 up, 6 in 2024-03-23T22:55:34.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:33 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T22:55:34.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:33 smithi012 ceph-mon[31695]: from='osd.5 [v2:172.21.15.203:6808/4270966092,v1:172.21.15.203:6809/4270966092]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-23T22:55:34.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:33 smithi012 ceph-mon[31695]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-23T22:55:34.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:33 smithi173 ceph-mon[36116]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-23T22:55:34.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:33 smithi173 ceph-mon[36116]: osdmap e37: 6 total, 5 up, 6 in 2024-03-23T22:55:34.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:33 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T22:55:34.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:33 smithi173 ceph-mon[36116]: from='osd.5 [v2:172.21.15.203:6808/4270966092,v1:172.21.15.203:6809/4270966092]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-23T22:55:34.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:33 smithi173 ceph-mon[36116]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-23T22:55:34.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:33 smithi203 ceph-mon[36693]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-23T22:55:34.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:33 smithi203 ceph-mon[36693]: osdmap e37: 6 total, 5 up, 6 in 2024-03-23T22:55:34.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:33 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T22:55:34.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:33 smithi203 ceph-mon[36693]: from='osd.5 [v2:172.21.15.203:6808/4270966092,v1:172.21.15.203:6809/4270966092]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-23T22:55:34.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:33 smithi203 ceph-mon[36693]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-23T22:55:34.547 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 22:55:34 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T22:55:34.257+0000 7f09bdbee700 -1 osd.5 0 waiting for initial osdmap 2024-03-23T22:55:34.547 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 22:55:34 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T22:55:34.264+0000 7f09b59d9700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T22:55:35.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:34 smithi173 ceph-mon[36116]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:35.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:34 smithi173 ceph-mon[36116]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-23T22:55:35.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:34 smithi173 ceph-mon[36116]: osdmap e38: 6 total, 5 up, 6 in 2024-03-23T22:55:35.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:34 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T22:55:35.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:34 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T22:55:35.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:34 smithi173 ceph-mon[36116]: from='osd.5 ' entity='osd.5' 2024-03-23T22:55:35.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:34 smithi173 ceph-mon[36116]: osd.5 [v2:172.21.15.203:6808/4270966092,v1:172.21.15.203:6809/4270966092] boot 2024-03-23T22:55:35.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:34 smithi173 ceph-mon[36116]: osdmap e39: 6 total, 6 up, 6 in 2024-03-23T22:55:35.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:34 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T22:55:35.127 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:34 smithi012 ceph-mon[31695]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:35.127 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:34 smithi012 ceph-mon[31695]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-23T22:55:35.127 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:34 smithi012 ceph-mon[31695]: osdmap e38: 6 total, 5 up, 6 in 2024-03-23T22:55:35.127 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:34 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T22:55:35.127 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:34 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T22:55:35.127 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:34 smithi012 ceph-mon[31695]: from='osd.5 ' entity='osd.5' 2024-03-23T22:55:35.127 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:34 smithi012 ceph-mon[31695]: osd.5 [v2:172.21.15.203:6808/4270966092,v1:172.21.15.203:6809/4270966092] boot 2024-03-23T22:55:35.127 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:34 smithi012 ceph-mon[31695]: osdmap e39: 6 total, 6 up, 6 in 2024-03-23T22:55:35.127 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:34 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T22:55:35.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:34 smithi203 ceph-mon[36693]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-23T22:55:35.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:34 smithi203 ceph-mon[36693]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-23T22:55:35.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:34 smithi203 ceph-mon[36693]: osdmap e38: 6 total, 5 up, 6 in 2024-03-23T22:55:35.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:34 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T22:55:35.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:34 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T22:55:35.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:34 smithi203 ceph-mon[36693]: from='osd.5 ' entity='osd.5' 2024-03-23T22:55:35.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:34 smithi203 ceph-mon[36693]: osd.5 [v2:172.21.15.203:6808/4270966092,v1:172.21.15.203:6809/4270966092] boot 2024-03-23T22:55:35.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:34 smithi203 ceph-mon[36693]: osdmap e39: 6 total, 6 up, 6 in 2024-03-23T22:55:35.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:34 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T22:55:36.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:35 smithi173 ceph-mon[36116]: purged_snaps scrub starts 2024-03-23T22:55:36.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:35 smithi173 ceph-mon[36116]: purged_snaps scrub ok 2024-03-23T22:55:36.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:35 smithi173 ceph-mon[36116]: osdmap e40: 6 total, 6 up, 6 in 2024-03-23T22:55:36.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:35 smithi203 ceph-mon[36693]: purged_snaps scrub starts 2024-03-23T22:55:36.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:35 smithi203 ceph-mon[36693]: purged_snaps scrub ok 2024-03-23T22:55:36.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:35 smithi203 ceph-mon[36693]: osdmap e40: 6 total, 6 up, 6 in 2024-03-23T22:55:36.200 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:35 smithi012 ceph-mon[31695]: purged_snaps scrub starts 2024-03-23T22:55:36.201 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:35 smithi012 ceph-mon[31695]: purged_snaps scrub ok 2024-03-23T22:55:36.201 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:35 smithi012 ceph-mon[31695]: osdmap e40: 6 total, 6 up, 6 in 2024-03-23T22:55:36.201 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-23T22:55:36.201 INFO:teuthology.orchestra.run.smithi012.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-23T22:55:36.201 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10+0 records in 2024-03-23T22:55:36.201 INFO:teuthology.orchestra.run.smithi012.stderr:10+0 records out 2024-03-23T22:55:36.201 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0436771 s, 240 MB/s 2024-03-23T22:55:36.201 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping successful for: 2024-03-23T22:55:37.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:36 smithi012 ceph-mon[31695]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:55:37.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:36 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:37.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:36 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:37.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:36 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:55:37.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:36 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:55:37.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:36 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:37.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:36 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:55:37.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:36 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:55:37.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:36 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:37.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:36 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:55:37.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:36 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:55:37.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:36 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:55:37.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:36 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:37.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:36 smithi012 ceph-mon[31695]: osdmap e41: 6 total, 6 up, 6 in 2024-03-23T22:55:37.113 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:36 smithi173 ceph-mon[36116]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:55:37.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:36 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:37.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:36 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:37.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:36 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:55:37.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:36 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:55:37.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:36 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:37.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:36 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:55:37.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:36 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:55:37.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:36 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:37.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:36 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:55:37.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:36 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:55:37.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:36 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:55:37.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:36 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:37.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:36 smithi173 ceph-mon[36116]: osdmap e41: 6 total, 6 up, 6 in 2024-03-23T22:55:37.139 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi012:vg_nvme/lv_2 2024-03-23T22:55:37.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:36 smithi203 ceph-mon[36693]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:55:37.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:36 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:37.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:36 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:37.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:36 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:55:37.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:36 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:55:37.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:36 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:37.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:36 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:55:37.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:36 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:55:37.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:36 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:37.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:36 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:55:37.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:36 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:55:37.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:36 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:55:37.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:36 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:55:37.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:36 smithi203 ceph-mon[36693]: osdmap e41: 6 total, 6 up, 6 in 2024-03-23T22:55:37.608 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:55:38.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:37 smithi012 ceph-mon[31695]: Detected new or changed devices on smithi203 2024-03-23T22:55:38.077 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:37 smithi012 ceph-mon[31695]: Adjusting osd_memory_target on smithi203 to 8540M 2024-03-23T22:55:38.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:37 smithi173 ceph-mon[36116]: Detected new or changed devices on smithi203 2024-03-23T22:55:38.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:37 smithi173 ceph-mon[36116]: Adjusting osd_memory_target on smithi203 to 8540M 2024-03-23T22:55:38.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:37 smithi203 ceph-mon[36693]: Detected new or changed devices on smithi203 2024-03-23T22:55:38.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:37 smithi203 ceph-mon[36693]: Adjusting osd_memory_target on smithi203 to 8540M 2024-03-23T22:55:39.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:38 smithi203 ceph-mon[36693]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:55:39.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:38 smithi203 ceph-mon[36693]: osdmap e42: 6 total, 6 up, 6 in 2024-03-23T22:55:39.234 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:38 smithi012 ceph-mon[31695]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:55:39.234 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:38 smithi012 ceph-mon[31695]: osdmap e42: 6 total, 6 up, 6 in 2024-03-23T22:55:39.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:38 smithi173 ceph-mon[36116]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:55:39.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:38 smithi173 ceph-mon[36116]: osdmap e42: 6 total, 6 up, 6 in 2024-03-23T22:55:39.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.762+0000 7f10f1800700 1 -- 172.21.15.12:0/1199175657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10ec107eb0 msgr2=0x7f10ec10a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:55:39.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.762+0000 7f10f1800700 1 --2- 172.21.15.12:0/1199175657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10ec107eb0 0x7f10ec10a320 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f10e4009f90 tx=0x7f10e400b330 comp rx=0 tx=0).stop 2024-03-23T22:55:39.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.762+0000 7f10f1800700 1 -- 172.21.15.12:0/1199175657 shutdown_connections 2024-03-23T22:55:39.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.762+0000 7f10f1800700 1 --2- 172.21.15.12:0/1199175657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10ec107eb0 0x7f10ec10a320 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:55:39.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.762+0000 7f10f1800700 1 --2- 172.21.15.12:0/1199175657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10ec102c50 0x7f10ec105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:55:39.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.762+0000 7f10f1800700 1 --2- 172.21.15.12:0/1199175657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10ec105580 0x7f10ec107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:55:39.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.762+0000 7f10f1800700 1 -- 172.21.15.12:0/1199175657 >> 172.21.15.12:0/1199175657 conn(0x7f10ec0fc660 msgr2=0x7f10ec0feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:55:39.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.762+0000 7f10f1800700 1 -- 172.21.15.12:0/1199175657 shutdown_connections 2024-03-23T22:55:39.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.762+0000 7f10f1800700 1 -- 172.21.15.12:0/1199175657 wait complete. 2024-03-23T22:55:39.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.763+0000 7f10f1800700 1 Processor -- start 2024-03-23T22:55:39.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.763+0000 7f10f1800700 1 -- start start 2024-03-23T22:55:39.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.764+0000 7f10f1800700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10ec102c50 0x7f10ec19a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:55:39.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.764+0000 7f10f1800700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10ec105580 0x7f10ec19aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:55:39.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.764+0000 7f10f1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10ec107eb0 0x7f10ec1a1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:55:39.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.764+0000 7f10f1800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ec10dd70 con 0x7f10ec107eb0 2024-03-23T22:55:39.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.764+0000 7f10f1800700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ec10dbf0 con 0x7f10ec102c50 2024-03-23T22:55:39.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.764+0000 7f10f1800700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ec10def0 con 0x7f10ec105580 2024-03-23T22:55:39.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.764+0000 7f10eb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10ec107eb0 0x7f10ec1a1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:55:39.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.764+0000 7f10eaa00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10ec102c50 0x7f10ec19a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:55:39.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.764+0000 7f10eaa00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10ec102c50 0x7f10ec19a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:38742/0 (socket says 172.21.15.12:38742) 2024-03-23T22:55:39.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.764+0000 7f10eaa00700 1 -- 172.21.15.12:0/1928223515 learned_addr learned my addr 172.21.15.12:0/1928223515 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:55:39.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.764+0000 7f10ea000700 1 --2- 172.21.15.12:0/1928223515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10ec105580 0x7f10ec19aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:55:39.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.765+0000 7f10eaa00700 1 -- 172.21.15.12:0/1928223515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10ec105580 msgr2=0x7f10ec19aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:55:39.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.765+0000 7f10eaa00700 1 --2- 172.21.15.12:0/1928223515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10ec105580 0x7f10ec19aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:55:39.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.765+0000 7f10eaa00700 1 -- 172.21.15.12:0/1928223515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10ec107eb0 msgr2=0x7f10ec1a1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:55:39.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.765+0000 7f10eaa00700 1 --2- 172.21.15.12:0/1928223515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10ec107eb0 0x7f10ec1a1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:55:39.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.765+0000 7f10eaa00700 1 -- 172.21.15.12:0/1928223515 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ec1a2210 con 0x7f10ec102c50 2024-03-23T22:55:39.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.765+0000 7f10eb400700 1 --2- 172.21.15.12:0/1928223515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10ec107eb0 0x7f10ec1a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T22:55:39.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.765+0000 7f10eaa00700 1 --2- 172.21.15.12:0/1928223515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10ec102c50 0x7f10ec19a5b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f10dc00e9a0 tx=0x7f10dc00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:55:39.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.765+0000 7f10e3400700 1 -- 172.21.15.12:0/1928223515 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10dc00ca10 con 0x7f10ec102c50 2024-03-23T22:55:39.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.765+0000 7f10e3400700 1 -- 172.21.15.12:0/1928223515 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f10dc00cbb0 con 0x7f10ec102c50 2024-03-23T22:55:39.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.765+0000 7f10e3400700 1 -- 172.21.15.12:0/1928223515 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10dc018710 con 0x7f10ec102c50 2024-03-23T22:55:39.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.765+0000 7f10f1800700 1 -- 172.21.15.12:0/1928223515 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10ec1a2500 con 0x7f10ec102c50 2024-03-23T22:55:39.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.765+0000 7f10f1800700 1 -- 172.21.15.12:0/1928223515 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10ec1a2a90 con 0x7f10ec102c50 2024-03-23T22:55:39.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.767+0000 7f10f1800700 1 -- 172.21.15.12:0/1928223515 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10ec068e50 con 0x7f10ec102c50 2024-03-23T22:55:39.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.769+0000 7f10e3400700 1 -- 172.21.15.12:0/1928223515 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f10dc047a60 con 0x7f10ec102c50 2024-03-23T22:55:39.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.770+0000 7f10e3400700 1 --2- 172.21.15.12:0/1928223515 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f10d00a2910 0x7f10d00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:55:39.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.770+0000 7f10e3400700 1 -- 172.21.15.12:0/1928223515 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4533+0+0 (secure 0 0 0) 0x7f10dc0cdd10 con 0x7f10ec102c50 2024-03-23T22:55:39.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.770+0000 7f10ea000700 1 --2- 172.21.15.12:0/1928223515 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f10d00a2910 0x7f10d00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:55:39.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.771+0000 7f10ea000700 1 --2- 172.21.15.12:0/1928223515 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f10d00a2910 0x7f10d00a4dd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f10d8009750 tx=0x7f10d8009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:55:39.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:39.772+0000 7f10e3400700 1 -- 172.21.15.12:0/1928223515 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f10dc09bbd0 con 0x7f10ec102c50 2024-03-23T22:55:40.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:55:40.013+0000 7f10f1800700 1 -- 172.21.15.12:0/1928223515 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f10ec0625d0 con 0x7f10d00a2910 2024-03-23T22:55:40.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:39 smithi203 ceph-mon[36693]: osdmap e43: 6 total, 6 up, 6 in 2024-03-23T22:55:40.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:39 smithi012 ceph-mon[31695]: osdmap e43: 6 total, 6 up, 6 in 2024-03-23T22:55:40.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:39 smithi173 ceph-mon[36116]: osdmap e43: 6 total, 6 up, 6 in 2024-03-23T22:55:41.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:40 smithi203 ceph-mon[36693]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:55:41.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:40 smithi203 ceph-mon[36693]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:55:41.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:55:41.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:55:41.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:55:41.163 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:40 smithi012 ceph-mon[31695]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:55:41.164 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:40 smithi012 ceph-mon[31695]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:55:41.164 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:55:41.164 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:55:41.164 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:55:41.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:40 smithi173 ceph-mon[36116]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:55:41.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:40 smithi173 ceph-mon[36116]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:55:41.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:55:41.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:55:41.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:55:43.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:42 smithi203 ceph-mon[36693]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-23T22:55:43.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:42 smithi012 ceph-mon[31695]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-23T22:55:43.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:42 smithi173 ceph-mon[36116]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-23T22:55:45.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:44 smithi012 ceph-mon[31695]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-23T22:55:45.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:44 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1808511067' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26ddc663-5623-48c9-80d1-f8b968765d07"}]: dispatch 2024-03-23T22:55:45.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:44 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1808511067' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26ddc663-5623-48c9-80d1-f8b968765d07"}]': finished 2024-03-23T22:55:45.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:44 smithi012 ceph-mon[31695]: osdmap e44: 7 total, 6 up, 7 in 2024-03-23T22:55:45.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:44 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-23T22:55:45.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:44 smithi203 ceph-mon[36693]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-23T22:55:45.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:44 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1808511067' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26ddc663-5623-48c9-80d1-f8b968765d07"}]: dispatch 2024-03-23T22:55:45.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:44 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1808511067' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26ddc663-5623-48c9-80d1-f8b968765d07"}]': finished 2024-03-23T22:55:45.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:44 smithi203 ceph-mon[36693]: osdmap e44: 7 total, 6 up, 7 in 2024-03-23T22:55:45.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:44 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-23T22:55:45.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:44 smithi173 ceph-mon[36116]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-23T22:55:45.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:44 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1808511067' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26ddc663-5623-48c9-80d1-f8b968765d07"}]: dispatch 2024-03-23T22:55:45.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:44 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1808511067' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26ddc663-5623-48c9-80d1-f8b968765d07"}]': finished 2024-03-23T22:55:45.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:44 smithi173 ceph-mon[36116]: osdmap e44: 7 total, 6 up, 7 in 2024-03-23T22:55:45.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:44 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-23T22:55:46.220 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:45 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3200634404' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:55:46.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:45 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3200634404' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:55:46.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:45 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3200634404' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:55:47.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:46 smithi012 ceph-mon[31695]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-23T22:55:47.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:46 smithi173 ceph-mon[36116]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-23T22:55:47.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:46 smithi203 ceph-mon[36693]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-23T22:55:49.125 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:48 smithi012 ceph-mon[31695]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-23T22:55:49.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:48 smithi173 ceph-mon[36116]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-23T22:55:49.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:48 smithi203 ceph-mon[36693]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-23T22:55:51.228 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:50 smithi012 ceph-mon[31695]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-23T22:55:51.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:50 smithi173 ceph-mon[36116]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-23T22:55:51.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:50 smithi203 ceph-mon[36693]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-23T22:55:53.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:52 smithi012 ceph-mon[31695]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:55:53.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:52 smithi173 ceph-mon[36116]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:55:53.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:52 smithi203 ceph-mon[36693]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:55:54.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:53 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-23T22:55:54.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:53 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:55:54.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:53 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-23T22:55:54.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:53 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:55:54.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:53 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-23T22:55:54.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:53 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:55:55.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:54 smithi012 ceph-mon[31695]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:55:55.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:54 smithi012 ceph-mon[31695]: Deploying daemon osd.6 on smithi012 2024-03-23T22:55:55.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:54 smithi173 ceph-mon[36116]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:55:55.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:54 smithi173 ceph-mon[36116]: Deploying daemon osd.6 on smithi012 2024-03-23T22:55:55.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:54 smithi203 ceph-mon[36693]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:55:55.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:54 smithi203 ceph-mon[36693]: Deploying daemon osd.6 on smithi012 2024-03-23T22:55:57.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:56 smithi012 ceph-mon[31695]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:55:57.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:56 smithi173 ceph-mon[36116]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:55:57.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:56 smithi203 ceph-mon[36693]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:55:59.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:55:58 smithi012 ceph-mon[31695]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:55:59.209 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:55:58 smithi203 ceph-mon[36693]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:55:59.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:55:58 smithi173 ceph-mon[36116]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:56:01.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:00 smithi012 ceph-mon[31695]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:56:01.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:00 smithi173 ceph-mon[36116]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:56:01.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:00 smithi203 ceph-mon[36693]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:56:03.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:02 smithi012 ceph-mon[31695]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:56:03.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:02 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:03.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:02 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:03.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:02 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:56:03.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:02 smithi173 ceph-mon[36116]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:56:03.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:02 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:03.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:02 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:03.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:02 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:56:03.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:02 smithi203 ceph-mon[36693]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:56:03.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:02 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:03.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:02 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:03.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:02 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:56:05.225 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:04 smithi012 ceph-mon[31695]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:56:05.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:04 smithi173 ceph-mon[36116]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:56:05.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:04 smithi203 ceph-mon[36693]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:56:07.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:06 smithi012 ceph-mon[31695]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:56:07.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:06 smithi012 ceph-mon[31695]: from='osd.6 [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-23T22:56:07.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:06 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:07.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:06 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:07.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:06 smithi173 ceph-mon[36116]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:56:07.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:06 smithi173 ceph-mon[36116]: from='osd.6 [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-23T22:56:07.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:06 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:07.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:06 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:07.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:06 smithi203 ceph-mon[36693]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:56:07.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:06 smithi203 ceph-mon[36693]: from='osd.6 [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-23T22:56:07.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:06 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:07.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:06 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:56:07.472+0000 7f10e3400700 1 -- 172.21.15.12:0/1928223515 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f10ec0625d0 con 0x7f10d00a2910 2024-03-23T22:56:07.473 INFO:teuthology.orchestra.run.smithi012.stdout:Created osd(s) 6 on host 'smithi012' 2024-03-23T22:56:07.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:56:07.475+0000 7f10f1800700 1 -- 172.21.15.12:0/1928223515 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f10d00a2910 msgr2=0x7f10d00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:56:07.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:56:07.475+0000 7f10f1800700 1 --2- 172.21.15.12:0/1928223515 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f10d00a2910 0x7f10d00a4dd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f10d8009750 tx=0x7f10d8009310 comp rx=0 tx=0).stop 2024-03-23T22:56:07.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:56:07.475+0000 7f10f1800700 1 -- 172.21.15.12:0/1928223515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10ec102c50 msgr2=0x7f10ec19a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:56:07.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:56:07.475+0000 7f10f1800700 1 --2- 172.21.15.12:0/1928223515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10ec102c50 0x7f10ec19a5b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f10dc00e9a0 tx=0x7f10dc00ecb0 comp rx=0 tx=0).stop 2024-03-23T22:56:07.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:56:07.475+0000 7f10f1800700 1 -- 172.21.15.12:0/1928223515 shutdown_connections 2024-03-23T22:56:07.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:56:07.475+0000 7f10f1800700 1 --2- 172.21.15.12:0/1928223515 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f10d00a2910 0x7f10d00a4dd0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:56:07.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:56:07.475+0000 7f10f1800700 1 --2- 172.21.15.12:0/1928223515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10ec107eb0 0x7f10ec1a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:56:07.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:56:07.475+0000 7f10f1800700 1 --2- 172.21.15.12:0/1928223515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10ec102c50 0x7f10ec19a5b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:56:07.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:56:07.475+0000 7f10f1800700 1 --2- 172.21.15.12:0/1928223515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10ec105580 0x7f10ec19aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:56:07.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:56:07.475+0000 7f10f1800700 1 -- 172.21.15.12:0/1928223515 >> 172.21.15.12:0/1928223515 conn(0x7f10ec0fc660 msgr2=0x7f10ec102300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:56:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:56:07.476+0000 7f10f1800700 1 -- 172.21.15.12:0/1928223515 shutdown_connections 2024-03-23T22:56:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:56:07.476+0000 7f10f1800700 1 -- 172.21.15.12:0/1928223515 wait complete. 2024-03-23T22:56:08.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:07 smithi012 ceph-mon[31695]: from='osd.6 [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-23T22:56:08.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:07 smithi012 ceph-mon[31695]: osdmap e45: 7 total, 6 up, 7 in 2024-03-23T22:56:08.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:07 smithi012 ceph-mon[31695]: from='osd.6 [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-23T22:56:08.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:07 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-23T22:56:08.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:07 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:08.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:07 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:08.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:07 smithi012 ceph-mon[31695]: from='osd.6 [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-23T22:56:08.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:07 smithi012 ceph-mon[31695]: osdmap e46: 7 total, 6 up, 7 in 2024-03-23T22:56:08.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:07 smithi173 ceph-mon[36116]: from='osd.6 [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-23T22:56:08.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:07 smithi173 ceph-mon[36116]: osdmap e45: 7 total, 6 up, 7 in 2024-03-23T22:56:08.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:07 smithi173 ceph-mon[36116]: from='osd.6 [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-23T22:56:08.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:07 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-23T22:56:08.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:07 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:08.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:07 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:08.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:07 smithi173 ceph-mon[36116]: from='osd.6 [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-23T22:56:08.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:07 smithi173 ceph-mon[36116]: osdmap e46: 7 total, 6 up, 7 in 2024-03-23T22:56:08.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:07 smithi203 ceph-mon[36693]: from='osd.6 [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-23T22:56:08.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:07 smithi203 ceph-mon[36693]: osdmap e45: 7 total, 6 up, 7 in 2024-03-23T22:56:08.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:07 smithi203 ceph-mon[36693]: from='osd.6 [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-23T22:56:08.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:07 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-23T22:56:08.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:07 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:08.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:07 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:08.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:07 smithi203 ceph-mon[36693]: from='osd.6 [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-23T22:56:08.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:07 smithi203 ceph-mon[36693]: osdmap e46: 7 total, 6 up, 7 in 2024-03-23T22:56:08.466 DEBUG:teuthology.orchestra.run.smithi012:osd.6> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.6.service 2024-03-23T22:56:08.471 INFO:tasks.cephadm:Deploying osd.7 on smithi173 with /dev/vg_nvme/lv_2... 2024-03-23T22:56:08.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-23T22:56:08.718 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.b/config 2024-03-23T22:56:08.803 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 22:56:08 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T22:56:08.545+0000 7fe651200700 -1 osd.6 0 waiting for initial osdmap 2024-03-23T22:56:08.804 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 22:56:08 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T22:56:08.568+0000 7fe647000700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T22:56:09.089 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:08 smithi173 ceph-mon[36116]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:56:09.089 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:08 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-23T22:56:09.089 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:08 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-23T22:56:09.089 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:08 smithi173 ceph-mon[36116]: from='osd.6 [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295]' entity='osd.6' 2024-03-23T22:56:09.090 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:08 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-23T22:56:09.243 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:08 smithi012 ceph-mon[31695]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:56:09.243 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:08 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-23T22:56:09.243 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:08 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-23T22:56:09.243 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:08 smithi012 ceph-mon[31695]: from='osd.6 [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295]' entity='osd.6' 2024-03-23T22:56:09.243 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:08 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-23T22:56:09.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:08 smithi203 ceph-mon[36693]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:56:09.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:08 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-23T22:56:09.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:08 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-23T22:56:09.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:08 smithi203 ceph-mon[36693]: from='osd.6 [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295]' entity='osd.6' 2024-03-23T22:56:09.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:08 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-23T22:56:10.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:09 smithi012 ceph-mon[31695]: purged_snaps scrub starts 2024-03-23T22:56:10.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:09 smithi012 ceph-mon[31695]: purged_snaps scrub ok 2024-03-23T22:56:10.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:09 smithi012 ceph-mon[31695]: osd.6 [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295] boot 2024-03-23T22:56:10.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:09 smithi012 ceph-mon[31695]: osdmap e47: 7 total, 7 up, 7 in 2024-03-23T22:56:10.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:09 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-23T22:56:10.090 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:09 smithi173 ceph-mon[36116]: purged_snaps scrub starts 2024-03-23T22:56:10.090 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:09 smithi173 ceph-mon[36116]: purged_snaps scrub ok 2024-03-23T22:56:10.090 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:09 smithi173 ceph-mon[36116]: osd.6 [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295] boot 2024-03-23T22:56:10.090 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:09 smithi173 ceph-mon[36116]: osdmap e47: 7 total, 7 up, 7 in 2024-03-23T22:56:10.091 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:09 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-23T22:56:10.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:09 smithi203 ceph-mon[36693]: purged_snaps scrub starts 2024-03-23T22:56:10.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:09 smithi203 ceph-mon[36693]: purged_snaps scrub ok 2024-03-23T22:56:10.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:09 smithi203 ceph-mon[36693]: osd.6 [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295] boot 2024-03-23T22:56:10.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:09 smithi203 ceph-mon[36693]: osdmap e47: 7 total, 7 up, 7 in 2024-03-23T22:56:10.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:09 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-23T22:56:10.647 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-23T22:56:10.647 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-23T22:56:10.647 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-03-23T22:56:10.647 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-03-23T22:56:10.647 INFO:teuthology.orchestra.run.smithi173.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111902 s, 937 MB/s 2024-03-23T22:56:10.647 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 2024-03-23T22:56:10.648 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-03-23T22:56:11.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:11 smithi012 ceph-mon[31695]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:56:11.336 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:11 smithi173 ceph-mon[36116]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:56:11.365 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi173:vg_nvme/lv_2 2024-03-23T22:56:11.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:11 smithi203 ceph-mon[36693]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-23T22:56:11.608 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.b/config 2024-03-23T22:56:12.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:12 smithi012 ceph-mon[31695]: osdmap e48: 7 total, 7 up, 7 in 2024-03-23T22:56:12.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:12 smithi173 ceph-mon[36116]: osdmap e48: 7 total, 7 up, 7 in 2024-03-23T22:56:12.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:12 smithi203 ceph-mon[36693]: osdmap e48: 7 total, 7 up, 7 in 2024-03-23T22:56:13.155 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.132+0000 7f128be96700 1 -- 172.21.15.173:0/1472723732 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1284100c50 msgr2=0x7f128410a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:56:13.155 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.132+0000 7f128be96700 1 --2- 172.21.15.173:0/1472723732 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1284100c50 0x7f128410a800 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f127c00b0d0 tx=0x7f127c00b3e0 comp rx=0 tx=0).stop 2024-03-23T22:56:13.155 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.132+0000 7f128be96700 1 -- 172.21.15.173:0/1472723732 shutdown_connections 2024-03-23T22:56:13.155 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.132+0000 7f128be96700 1 --2- 172.21.15.173:0/1472723732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1284103810 0x7f1284103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:56:13.155 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.132+0000 7f128be96700 1 --2- 172.21.15.173:0/1472723732 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1284100c50 0x7f128410a800 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:56:13.155 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.132+0000 7f128be96700 1 --2- 172.21.15.173:0/1472723732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1284100290 0x7f1284100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:56:13.155 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.132+0000 7f128be96700 1 -- 172.21.15.173:0/1472723732 >> 172.21.15.173:0/1472723732 conn(0x7f12840fbe30 msgr2=0x7f12840fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:56:13.155 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.132+0000 7f128be96700 1 -- 172.21.15.173:0/1472723732 shutdown_connections 2024-03-23T22:56:13.155 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.132+0000 7f128be96700 1 -- 172.21.15.173:0/1472723732 wait complete. 2024-03-23T22:56:13.156 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.132+0000 7f128be96700 1 Processor -- start 2024-03-23T22:56:13.156 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.133+0000 7f128be96700 1 -- start start 2024-03-23T22:56:13.156 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.133+0000 7f128be96700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1284100290 0x7f12841984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:56:13.156 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.133+0000 7f128be96700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1284100c50 0x7f1284198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:56:13.156 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.133+0000 7f128be96700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1284103810 0x7f128419faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:56:13.156 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.133+0000 7f128be96700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f128410df30 con 0x7f1284103810 2024-03-23T22:56:13.156 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.133+0000 7f128be96700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f128410ddb0 con 0x7f1284100290 2024-03-23T22:56:13.156 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.133+0000 7f128be96700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f128410e0b0 con 0x7f1284100c50 2024-03-23T22:56:13.156 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.133+0000 7f1289c32700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1284100290 0x7f12841984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:56:13.156 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.133+0000 7f1289c32700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1284100290 0x7f12841984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:43770/0 (socket says 172.21.15.173:43770) 2024-03-23T22:56:13.156 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.133+0000 7f1289c32700 1 -- 172.21.15.173:0/1573878979 learned_addr learned my addr 172.21.15.173:0/1573878979 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-23T22:56:13.156 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.133+0000 7f1289c32700 1 -- 172.21.15.173:0/1573878979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1284100c50 msgr2=0x7f1284198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:56:13.156 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.133+0000 7f1289c32700 1 --2- 172.21.15.173:0/1573878979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1284100c50 0x7f1284198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:56:13.157 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.133+0000 7f1289c32700 1 -- 172.21.15.173:0/1573878979 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1284103810 msgr2=0x7f128419faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:56:13.157 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.133+0000 7f1289c32700 1 --2- 172.21.15.173:0/1573878979 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1284103810 0x7f128419faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:56:13.157 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.133+0000 7f1289c32700 1 -- 172.21.15.173:0/1573878979 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12841a0140 con 0x7f1284100290 2024-03-23T22:56:13.157 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.133+0000 7f1289c32700 1 --2- 172.21.15.173:0/1573878979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1284100290 0x7f12841984e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f128000b500 tx=0x7f128000b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:56:13.157 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.133+0000 7f127affd700 1 -- 172.21.15.173:0/1573878979 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1280015020 con 0x7f1284100290 2024-03-23T22:56:13.157 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.133+0000 7f127affd700 1 -- 172.21.15.173:0/1573878979 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1280011d10 con 0x7f1284100290 2024-03-23T22:56:13.157 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.133+0000 7f128be96700 1 -- 172.21.15.173:0/1573878979 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12841a0430 con 0x7f1284100290 2024-03-23T22:56:13.157 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.133+0000 7f127affd700 1 -- 172.21.15.173:0/1573878979 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128000f2e0 con 0x7f1284100290 2024-03-23T22:56:13.157 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.134+0000 7f128be96700 1 -- 172.21.15.173:0/1573878979 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12841a09a0 con 0x7f1284100290 2024-03-23T22:56:13.157 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.134+0000 7f128be96700 1 -- 172.21.15.173:0/1573878979 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1284068e50 con 0x7f1284100290 2024-03-23T22:56:13.157 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.136+0000 7f127affd700 1 -- 172.21.15.173:0/1573878979 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1280018020 con 0x7f1284100290 2024-03-23T22:56:13.157 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.136+0000 7f127affd700 1 --2- 172.21.15.173:0/1573878979 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f12700a2890 0x7f12700a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:56:13.157 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.136+0000 7f127affd700 1 -- 172.21.15.173:0/1573878979 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f12800cdb30 con 0x7f1284100290 2024-03-23T22:56:13.157 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.137+0000 7f1289431700 1 --2- 172.21.15.173:0/1573878979 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f12700a2890 0x7f12700a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:56:13.158 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.137+0000 7f1289431700 1 --2- 172.21.15.173:0/1573878979 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f12700a2890 0x7f12700a4d50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f128406a7e0 tx=0x7f1274006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:56:13.158 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.140+0000 7f127affd700 1 -- 172.21.15.173:0/1573878979 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f128001f020 con 0x7f1284100290 2024-03-23T22:56:13.287 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:13 smithi012 ceph-mon[31695]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:13.345 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:13.345+0000 7f128be96700 1 -- 172.21.15.173:0/1573878979 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f12840625d0 con 0x7f12700a2890 2024-03-23T22:56:13.345 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:13 smithi173 ceph-mon[36116]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:13.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:13 smithi203 ceph-mon[36693]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:14.232 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:14 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:56:14.232 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:14 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:56:14.232 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:14 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:56:14.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:56:14.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:56:14.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:56:14.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:14 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:56:14.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:14 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:56:14.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:14 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:56:15.278 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:15 smithi173 ceph-mon[36116]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:15.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:15 smithi173 ceph-mon[36116]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:56:15.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:15 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:15.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:15 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:15.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:15 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:56:15.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:15 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:56:15.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:15 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:56:15.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:15 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:15.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:15 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:56:15.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:15 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:56:15.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:15 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:15.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:15 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:56:15.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:15 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:56:15.280 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:15 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:56:15.280 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:15 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:15.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:15 smithi012 ceph-mon[31695]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:15.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:15 smithi012 ceph-mon[31695]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:56:15.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:15 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:15.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:15 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:15.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:15 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:56:15.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:15 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:56:15.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:15 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:56:15.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:15 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:15.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:15 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:56:15.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:15 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:56:15.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:15 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:15.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:15 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:56:15.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:15 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:56:15.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:15 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:56:15.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:15 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:15.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:15 smithi203 ceph-mon[36693]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:15.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:15 smithi203 ceph-mon[36693]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:56:15.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:15 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:15.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:15 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:15.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:15 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:56:15.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:15 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:56:15.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:15 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:56:15.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:15 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:15.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:15 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:56:15.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:15 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:56:15.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:15 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:15.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:15 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:56:15.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:15 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:56:15.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:15 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:56:15.409 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:15 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:16.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:16 smithi012 ceph-mon[31695]: Detected new or changed devices on smithi012 2024-03-23T22:56:16.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:16 smithi012 ceph-mon[31695]: Adjusting osd_memory_target on smithi012 to 5698M 2024-03-23T22:56:16.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:16 smithi173 ceph-mon[36116]: Detected new or changed devices on smithi012 2024-03-23T22:56:16.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:16 smithi173 ceph-mon[36116]: Adjusting osd_memory_target on smithi012 to 5698M 2024-03-23T22:56:16.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:16 smithi203 ceph-mon[36693]: Detected new or changed devices on smithi012 2024-03-23T22:56:16.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:16 smithi203 ceph-mon[36693]: Adjusting osd_memory_target on smithi012 to 5698M 2024-03-23T22:56:17.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:17 smithi012 ceph-mon[31695]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:17.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:17 smithi012 ceph-mon[31695]: from='client.? 172.21.15.173:0/3078652596' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17dc9417-71cd-4576-aa08-933212ec55e9"}]: dispatch 2024-03-23T22:56:17.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:17 smithi012 ceph-mon[31695]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17dc9417-71cd-4576-aa08-933212ec55e9"}]: dispatch 2024-03-23T22:56:17.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:17 smithi012 ceph-mon[31695]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "17dc9417-71cd-4576-aa08-933212ec55e9"}]': finished 2024-03-23T22:56:17.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:17 smithi012 ceph-mon[31695]: osdmap e49: 8 total, 7 up, 8 in 2024-03-23T22:56:17.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:17 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-23T22:56:17.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:17 smithi012 ceph-mon[31695]: from='client.? 172.21.15.173:0/3831643347' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:56:17.370 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:17 smithi173 ceph-mon[36116]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:17.370 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:17 smithi173 ceph-mon[36116]: from='client.? 172.21.15.173:0/3078652596' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17dc9417-71cd-4576-aa08-933212ec55e9"}]: dispatch 2024-03-23T22:56:17.370 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:17 smithi173 ceph-mon[36116]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17dc9417-71cd-4576-aa08-933212ec55e9"}]: dispatch 2024-03-23T22:56:17.370 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:17 smithi173 ceph-mon[36116]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "17dc9417-71cd-4576-aa08-933212ec55e9"}]': finished 2024-03-23T22:56:17.370 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:17 smithi173 ceph-mon[36116]: osdmap e49: 8 total, 7 up, 8 in 2024-03-23T22:56:17.370 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:17 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-23T22:56:17.370 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:17 smithi173 ceph-mon[36116]: from='client.? 172.21.15.173:0/3831643347' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:56:17.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:17 smithi203 ceph-mon[36693]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:17.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:17 smithi203 ceph-mon[36693]: from='client.? 172.21.15.173:0/3078652596' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17dc9417-71cd-4576-aa08-933212ec55e9"}]: dispatch 2024-03-23T22:56:17.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:17 smithi203 ceph-mon[36693]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17dc9417-71cd-4576-aa08-933212ec55e9"}]: dispatch 2024-03-23T22:56:17.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:17 smithi203 ceph-mon[36693]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "17dc9417-71cd-4576-aa08-933212ec55e9"}]': finished 2024-03-23T22:56:17.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:17 smithi203 ceph-mon[36693]: osdmap e49: 8 total, 7 up, 8 in 2024-03-23T22:56:17.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:17 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-23T22:56:17.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:17 smithi203 ceph-mon[36693]: from='client.? 172.21.15.173:0/3831643347' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:56:19.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:19 smithi012 ceph-mon[31695]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:19.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:19 smithi173 ceph-mon[36116]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:19.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:19 smithi203 ceph-mon[36693]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:21.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:21 smithi012 ceph-mon[31695]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:21.305 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:21 smithi173 ceph-mon[36116]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:21.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:21 smithi203 ceph-mon[36693]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:23.269 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:23 smithi173 ceph-mon[36116]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:23.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:23 smithi203 ceph-mon[36693]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:23.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:23 smithi012 ceph-mon[31695]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:24.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:24 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-23T22:56:24.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:24 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:56:24.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-23T22:56:24.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:56:24.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-23T22:56:24.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:56:25.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:25 smithi173 ceph-mon[36116]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:25.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:25 smithi173 ceph-mon[36116]: Deploying daemon osd.7 on smithi173 2024-03-23T22:56:25.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:25 smithi203 ceph-mon[36693]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:25.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:25 smithi203 ceph-mon[36693]: Deploying daemon osd.7 on smithi173 2024-03-23T22:56:25.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:25 smithi012 ceph-mon[31695]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:25.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:25 smithi012 ceph-mon[31695]: Deploying daemon osd.7 on smithi173 2024-03-23T22:56:27.329 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:27 smithi173 ceph-mon[36116]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:27.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:27 smithi203 ceph-mon[36693]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:27.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:27 smithi012 ceph-mon[31695]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:29.189 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:29 smithi173 ceph-mon[36116]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:29.356 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:29 smithi012 ceph-mon[31695]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:29.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:29 smithi203 ceph-mon[36693]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:30.485 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:30 smithi203 ceph-mon[36693]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:30.485 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:30 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:30.485 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:30 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:30.485 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:30 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:56:30.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:30 smithi173 ceph-mon[36116]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:30.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:30 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:30.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:30 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:30.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:30 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:56:30.739 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:30 smithi012 ceph-mon[31695]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:30.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:30 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:30.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:30 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:30.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:30 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:56:32.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:32 smithi173 ceph-mon[36116]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:32.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:32 smithi203 ceph-mon[36693]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:32.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:32 smithi012 ceph-mon[31695]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:32.843 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 7 on host 'smithi173' 2024-03-23T22:56:32.843 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:32.843+0000 7f127affd700 1 -- 172.21.15.173:0/1573878979 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f12840625d0 con 0x7f12700a2890 2024-03-23T22:56:32.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:32.845+0000 7f128be96700 1 -- 172.21.15.173:0/1573878979 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f12700a2890 msgr2=0x7f12700a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:56:32.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:32.845+0000 7f128be96700 1 --2- 172.21.15.173:0/1573878979 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f12700a2890 0x7f12700a4d50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f128406a7e0 tx=0x7f1274006d20 comp rx=0 tx=0).stop 2024-03-23T22:56:32.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:32.845+0000 7f128be96700 1 -- 172.21.15.173:0/1573878979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1284100290 msgr2=0x7f12841984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:56:32.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:32.845+0000 7f128be96700 1 --2- 172.21.15.173:0/1573878979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1284100290 0x7f12841984e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f128000b500 tx=0x7f128000b8c0 comp rx=0 tx=0).stop 2024-03-23T22:56:32.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:32.845+0000 7f128be96700 1 -- 172.21.15.173:0/1573878979 shutdown_connections 2024-03-23T22:56:32.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:32.845+0000 7f128be96700 1 --2- 172.21.15.173:0/1573878979 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f12700a2890 0x7f12700a4d50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:56:32.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:32.845+0000 7f128be96700 1 --2- 172.21.15.173:0/1573878979 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1284103810 0x7f128419faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:56:32.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:32.845+0000 7f128be96700 1 --2- 172.21.15.173:0/1573878979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1284100290 0x7f12841984e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:56:32.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:32.845+0000 7f128be96700 1 --2- 172.21.15.173:0/1573878979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1284100c50 0x7f1284198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:56:32.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:32.845+0000 7f128be96700 1 -- 172.21.15.173:0/1573878979 >> 172.21.15.173:0/1573878979 conn(0x7f12840fbe30 msgr2=0x7f1284106850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:56:32.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:32.845+0000 7f128be96700 1 -- 172.21.15.173:0/1573878979 shutdown_connections 2024-03-23T22:56:32.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:56:32.845+0000 7f128be96700 1 -- 172.21.15.173:0/1573878979 wait complete. 2024-03-23T22:56:33.513 DEBUG:teuthology.orchestra.run.smithi173:osd.7> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.7.service 2024-03-23T22:56:33.516 INFO:tasks.cephadm:Deploying osd.8 on smithi203 with /dev/vg_nvme/lv_2... 2024-03-23T22:56:33.517 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-23T22:56:33.765 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.c/config 2024-03-23T22:56:33.886 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:33 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:33.886 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:33 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:33.887 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:33 smithi203 ceph-mon[36693]: from='osd.7 [v2:172.21.15.173:6816/1241504346,v1:172.21.15.173:6817/1241504346]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-23T22:56:33.887 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:33 smithi203 ceph-mon[36693]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-23T22:56:33.887 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:33 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:33.887 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:33 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:34.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:33 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:34.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:33 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:34.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:33 smithi012 ceph-mon[31695]: from='osd.7 [v2:172.21.15.173:6816/1241504346,v1:172.21.15.173:6817/1241504346]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-23T22:56:34.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:33 smithi012 ceph-mon[31695]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-23T22:56:34.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:33 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:34.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:33 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:34.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:33 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:34.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:33 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:34.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:33 smithi173 ceph-mon[36116]: from='osd.7 [v2:172.21.15.173:6816/1241504346,v1:172.21.15.173:6817/1241504346]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-23T22:56:34.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:33 smithi173 ceph-mon[36116]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-23T22:56:34.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:33 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:34.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:33 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:35.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:34 smithi173 ceph-mon[36116]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:35.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:34 smithi173 ceph-mon[36116]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-23T22:56:35.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:34 smithi173 ceph-mon[36116]: from='osd.7 [v2:172.21.15.173:6816/1241504346,v1:172.21.15.173:6817/1241504346]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-23T22:56:35.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:34 smithi173 ceph-mon[36116]: osdmap e50: 8 total, 7 up, 8 in 2024-03-23T22:56:35.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:34 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-23T22:56:35.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:34 smithi173 ceph-mon[36116]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-23T22:56:35.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:34 smithi203 ceph-mon[36693]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:35.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:34 smithi203 ceph-mon[36693]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-23T22:56:35.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:34 smithi203 ceph-mon[36693]: from='osd.7 [v2:172.21.15.173:6816/1241504346,v1:172.21.15.173:6817/1241504346]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-23T22:56:35.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:34 smithi203 ceph-mon[36693]: osdmap e50: 8 total, 7 up, 8 in 2024-03-23T22:56:35.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:34 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-23T22:56:35.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:34 smithi203 ceph-mon[36693]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-23T22:56:35.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:34 smithi012 ceph-mon[31695]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:35.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:34 smithi012 ceph-mon[31695]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-23T22:56:35.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:34 smithi012 ceph-mon[31695]: from='osd.7 [v2:172.21.15.173:6816/1241504346,v1:172.21.15.173:6817/1241504346]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-23T22:56:35.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:34 smithi012 ceph-mon[31695]: osdmap e50: 8 total, 7 up, 8 in 2024-03-23T22:56:35.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:34 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-23T22:56:35.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:34 smithi012 ceph-mon[31695]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-23T22:56:35.523 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 22:56:35 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T22:56:35.229+0000 7efcdce2f700 -1 osd.7 0 waiting for initial osdmap 2024-03-23T22:56:35.524 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 22:56:35 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T22:56:35.240+0000 7efcd4c1a700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T22:56:35.892 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-23T22:56:35.892 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-23T22:56:35.893 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-03-23T22:56:35.893 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-03-23T22:56:35.893 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113961 s, 920 MB/s 2024-03-23T22:56:35.893 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-03-23T22:56:36.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:35 smithi173 ceph-mon[36116]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-23T22:56:36.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:35 smithi173 ceph-mon[36116]: osdmap e51: 8 total, 7 up, 8 in 2024-03-23T22:56:36.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:35 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-23T22:56:36.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:35 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-23T22:56:36.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:35 smithi173 ceph-mon[36116]: from='osd.7 ' entity='osd.7' 2024-03-23T22:56:36.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:35 smithi203 ceph-mon[36693]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-23T22:56:36.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:35 smithi203 ceph-mon[36693]: osdmap e51: 8 total, 7 up, 8 in 2024-03-23T22:56:36.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:35 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-23T22:56:36.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:35 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-23T22:56:36.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:35 smithi203 ceph-mon[36693]: from='osd.7 ' entity='osd.7' 2024-03-23T22:56:36.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:35 smithi012 ceph-mon[31695]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-23T22:56:36.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:35 smithi012 ceph-mon[31695]: osdmap e51: 8 total, 7 up, 8 in 2024-03-23T22:56:36.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:35 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-23T22:56:36.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:35 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-23T22:56:36.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:35 smithi012 ceph-mon[31695]: from='osd.7 ' entity='osd.7' 2024-03-23T22:56:36.591 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi203:vg_nvme/lv_2 2024-03-23T22:56:36.830 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.c/config 2024-03-23T22:56:37.036 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:36 smithi173 ceph-mon[36116]: purged_snaps scrub starts 2024-03-23T22:56:37.037 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:36 smithi173 ceph-mon[36116]: purged_snaps scrub ok 2024-03-23T22:56:37.037 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:36 smithi173 ceph-mon[36116]: pgmap v197: 1 pgs: 1 unknown; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:37.037 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:36 smithi173 ceph-mon[36116]: osd.7 [v2:172.21.15.173:6816/1241504346,v1:172.21.15.173:6817/1241504346] boot 2024-03-23T22:56:37.037 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:36 smithi173 ceph-mon[36116]: osdmap e52: 8 total, 8 up, 8 in 2024-03-23T22:56:37.037 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:36 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-23T22:56:37.037 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:36 smithi173 ceph-mon[36116]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-23T22:56:37.037 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:36 smithi173 ceph-mon[36116]: osdmap e53: 8 total, 8 up, 8 in 2024-03-23T22:56:37.076 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:36 smithi203 ceph-mon[36693]: purged_snaps scrub starts 2024-03-23T22:56:37.076 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:36 smithi203 ceph-mon[36693]: purged_snaps scrub ok 2024-03-23T22:56:37.076 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:36 smithi203 ceph-mon[36693]: pgmap v197: 1 pgs: 1 unknown; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:37.076 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:36 smithi203 ceph-mon[36693]: osd.7 [v2:172.21.15.173:6816/1241504346,v1:172.21.15.173:6817/1241504346] boot 2024-03-23T22:56:37.076 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:36 smithi203 ceph-mon[36693]: osdmap e52: 8 total, 8 up, 8 in 2024-03-23T22:56:37.076 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:36 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-23T22:56:37.076 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:36 smithi203 ceph-mon[36693]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-23T22:56:37.076 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:36 smithi203 ceph-mon[36693]: osdmap e53: 8 total, 8 up, 8 in 2024-03-23T22:56:37.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:36 smithi012 ceph-mon[31695]: purged_snaps scrub starts 2024-03-23T22:56:37.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:36 smithi012 ceph-mon[31695]: purged_snaps scrub ok 2024-03-23T22:56:37.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:36 smithi012 ceph-mon[31695]: pgmap v197: 1 pgs: 1 unknown; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-23T22:56:37.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:36 smithi012 ceph-mon[31695]: osd.7 [v2:172.21.15.173:6816/1241504346,v1:172.21.15.173:6817/1241504346] boot 2024-03-23T22:56:37.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:36 smithi012 ceph-mon[31695]: osdmap e52: 8 total, 8 up, 8 in 2024-03-23T22:56:37.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:36 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-23T22:56:37.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:36 smithi012 ceph-mon[31695]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-23T22:56:37.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:36 smithi012 ceph-mon[31695]: osdmap e53: 8 total, 8 up, 8 in 2024-03-23T22:56:38.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:38 smithi203 ceph-mon[36693]: Detected new or changed devices on smithi173 2024-03-23T22:56:38.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:38 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:38.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:38 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:38.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:38 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:56:38.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:38 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:56:38.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:38 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:56:38.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:38 smithi203 ceph-mon[36693]: Adjusting osd_memory_target on smithi173 to 5693M 2024-03-23T22:56:38.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:38 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:38.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:38 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:56:38.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:38 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:56:38.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:38 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:38.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:38 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:56:38.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:38 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:56:38.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:38 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:56:38.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:38 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:38.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:38 smithi203 ceph-mon[36693]: osdmap e54: 8 total, 8 up, 8 in 2024-03-23T22:56:38.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:38 smithi173 ceph-mon[36116]: Detected new or changed devices on smithi173 2024-03-23T22:56:38.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:38 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:38.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:38 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:38.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:38 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:56:38.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:38 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:56:38.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:38 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:56:38.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:38 smithi173 ceph-mon[36116]: Adjusting osd_memory_target on smithi173 to 5693M 2024-03-23T22:56:38.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:38 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:38.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:38 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:56:38.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:38 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:56:38.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:38 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:38.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:38 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:56:38.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:38 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:56:38.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:38 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:56:38.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:38 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:38.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:38 smithi173 ceph-mon[36116]: osdmap e54: 8 total, 8 up, 8 in 2024-03-23T22:56:38.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.470+0000 7fcd47adc700 1 -- 172.21.15.203:0/3648295035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd40102c60 msgr2=0x7fcd40109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:56:38.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.470+0000 7fcd47adc700 1 --2- 172.21.15.203:0/3648295035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd40102c60 0x7fcd40109510 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcd3800b0a0 tx=0x7fcd3800b3b0 comp rx=0 tx=0).stop 2024-03-23T22:56:38.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.470+0000 7fcd47adc700 1 -- 172.21.15.203:0/3648295035 shutdown_connections 2024-03-23T22:56:38.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.470+0000 7fcd47adc700 1 --2- 172.21.15.203:0/3648295035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd40102c60 0x7fcd40109510 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:56:38.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.470+0000 7fcd47adc700 1 --2- 172.21.15.203:0/3648295035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd401010c0 0x7fcd401014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:56:38.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.470+0000 7fcd47adc700 1 --2- 172.21.15.203:0/3648295035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd401022c0 0x7fcd40102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:56:38.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.470+0000 7fcd47adc700 1 -- 172.21.15.203:0/3648295035 >> 172.21.15.203:0/3648295035 conn(0x7fcd400fc680 msgr2=0x7fcd400feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:56:38.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.470+0000 7fcd47adc700 1 -- 172.21.15.203:0/3648295035 shutdown_connections 2024-03-23T22:56:38.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.470+0000 7fcd47adc700 1 -- 172.21.15.203:0/3648295035 wait complete. 2024-03-23T22:56:38.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.471+0000 7fcd47adc700 1 Processor -- start 2024-03-23T22:56:38.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.471+0000 7fcd47adc700 1 -- start start 2024-03-23T22:56:38.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.471+0000 7fcd47adc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd401010c0 0x7fcd4006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:56:38.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.471+0000 7fcd47adc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd401022c0 0x7fcd4019d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:56:38.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.471+0000 7fcd47adc700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd40102c60 0x7fcd4019f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:56:38.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.471+0000 7fcd47adc700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4010cdf0 con 0x7fcd40102c60 2024-03-23T22:56:38.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.471+0000 7fcd47adc700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4010cc70 con 0x7fcd401022c0 2024-03-23T22:56:38.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.471+0000 7fcd47adc700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4010cf70 con 0x7fcd401010c0 2024-03-23T22:56:38.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.471+0000 7fcd45878700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd401010c0 0x7fcd4006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:56:38.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.471+0000 7fcd45878700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd401010c0 0x7fcd4006b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:51410/0 (socket says 172.21.15.203:51410) 2024-03-23T22:56:38.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.471+0000 7fcd45878700 1 -- 172.21.15.203:0/2827057338 learned_addr learned my addr 172.21.15.203:0/2827057338 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-23T22:56:38.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.472+0000 7fcd45077700 1 --2- 172.21.15.203:0/2827057338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd401022c0 0x7fcd4019d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:56:38.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.472+0000 7fcd45878700 1 -- 172.21.15.203:0/2827057338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd401022c0 msgr2=0x7fcd4019d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:56:38.475 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.472+0000 7fcd45878700 1 --2- 172.21.15.203:0/2827057338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd401022c0 0x7fcd4019d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:56:38.475 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.472+0000 7fcd45878700 1 -- 172.21.15.203:0/2827057338 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd40102c60 msgr2=0x7fcd4019f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:56:38.475 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.472+0000 7fcd45878700 1 --2- 172.21.15.203:0/2827057338 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd40102c60 0x7fcd4019f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:56:38.475 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.472+0000 7fcd45878700 1 -- 172.21.15.203:0/2827057338 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd401a0170 con 0x7fcd401010c0 2024-03-23T22:56:38.475 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.472+0000 7fcd45077700 1 --2- 172.21.15.203:0/2827057338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd401022c0 0x7fcd4019d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T22:56:38.475 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.472+0000 7fcd45878700 1 --2- 172.21.15.203:0/2827057338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd401010c0 0x7fcd4006b410 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcd3000b500 tx=0x7fcd3000b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:56:38.475 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.472+0000 7fcd36ffd700 1 -- 172.21.15.203:0/2827057338 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd30015020 con 0x7fcd401010c0 2024-03-23T22:56:38.475 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.472+0000 7fcd36ffd700 1 -- 172.21.15.203:0/2827057338 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd30011d10 con 0x7fcd401010c0 2024-03-23T22:56:38.475 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.472+0000 7fcd47adc700 1 -- 172.21.15.203:0/2827057338 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd401a0460 con 0x7fcd401010c0 2024-03-23T22:56:38.475 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.472+0000 7fcd36ffd700 1 -- 172.21.15.203:0/2827057338 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3000f2e0 con 0x7fcd401010c0 2024-03-23T22:56:38.475 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.472+0000 7fcd47adc700 1 -- 172.21.15.203:0/2827057338 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd401a09d0 con 0x7fcd401010c0 2024-03-23T22:56:38.476 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.473+0000 7fcd47adc700 1 -- 172.21.15.203:0/2827057338 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd40068e50 con 0x7fcd401010c0 2024-03-23T22:56:38.476 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.474+0000 7fcd36ffd700 1 -- 172.21.15.203:0/2827057338 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcd30018020 con 0x7fcd401010c0 2024-03-23T22:56:38.476 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.475+0000 7fcd36ffd700 1 --2- 172.21.15.203:0/2827057338 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcd2c0a2890 0x7fcd2c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:56:38.476 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.475+0000 7fcd36ffd700 1 -- 172.21.15.203:0/2827057338 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5426+0+0 (secure 0 0 0) 0x7fcd300cd770 con 0x7fcd401010c0 2024-03-23T22:56:38.476 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.475+0000 7fcd45077700 1 --2- 172.21.15.203:0/2827057338 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcd2c0a2890 0x7fcd2c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:56:38.476 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.476+0000 7fcd45077700 1 --2- 172.21.15.203:0/2827057338 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcd2c0a2890 0x7fcd2c0a4d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcd4006b990 tx=0x7fcd3c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:56:38.479 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.479+0000 7fcd36ffd700 1 -- 172.21.15.203:0/2827057338 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcd3001f020 con 0x7fcd401010c0 2024-03-23T22:56:38.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:38 smithi012 ceph-mon[31695]: Detected new or changed devices on smithi173 2024-03-23T22:56:38.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:38 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:38.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:38 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:38.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:38 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:56:38.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:38 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:56:38.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:38 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:56:38.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:38 smithi012 ceph-mon[31695]: Adjusting osd_memory_target on smithi173 to 5693M 2024-03-23T22:56:38.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:38 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:38.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:38 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:56:38.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:38 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:56:38.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:38 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:38.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:38 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:56:38.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:38 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:56:38.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:38 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:56:38.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:38 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:38.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:38 smithi012 ceph-mon[31695]: osdmap e54: 8 total, 8 up, 8 in 2024-03-23T22:56:38.685 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:38.684+0000 7fcd47adc700 1 -- 172.21.15.203:0/2827057338 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd400625d0 con 0x7fcd2c0a2890 2024-03-23T22:56:39.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:39 smithi173 ceph-mon[36116]: pgmap v200: 1 pgs: 1 unknown; 577 KiB data, 275 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:39.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:39 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:56:39.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:39 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:56:39.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:39 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:56:39.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:39 smithi203 ceph-mon[36693]: pgmap v200: 1 pgs: 1 unknown; 577 KiB data, 275 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:39.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:39 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:56:39.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:39 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:56:39.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:39 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:56:39.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:39 smithi012 ceph-mon[31695]: pgmap v200: 1 pgs: 1 unknown; 577 KiB data, 275 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:39.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:39 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:56:39.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:39 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:56:39.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:39 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:56:40.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:40 smithi173 ceph-mon[36116]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:56:40.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:40 smithi173 ceph-mon[36116]: osdmap e55: 8 total, 8 up, 8 in 2024-03-23T22:56:40.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:40 smithi203 ceph-mon[36693]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:56:40.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:40 smithi203 ceph-mon[36693]: osdmap e55: 8 total, 8 up, 8 in 2024-03-23T22:56:40.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:40 smithi012 ceph-mon[31695]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:56:40.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:40 smithi012 ceph-mon[31695]: osdmap e55: 8 total, 8 up, 8 in 2024-03-23T22:56:41.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:41 smithi173 ceph-mon[36116]: pgmap v203: 1 pgs: 1 unknown; 577 KiB data, 275 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:41.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:41 smithi173 ceph-mon[36116]: osdmap e56: 8 total, 8 up, 8 in 2024-03-23T22:56:41.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:41 smithi203 ceph-mon[36693]: pgmap v203: 1 pgs: 1 unknown; 577 KiB data, 275 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:41.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:41 smithi203 ceph-mon[36693]: osdmap e56: 8 total, 8 up, 8 in 2024-03-23T22:56:41.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:41 smithi012 ceph-mon[31695]: pgmap v203: 1 pgs: 1 unknown; 577 KiB data, 275 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:41.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:41 smithi012 ceph-mon[31695]: osdmap e56: 8 total, 8 up, 8 in 2024-03-23T22:56:42.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:42 smithi203 ceph-mon[36693]: from='client.? 172.21.15.203:0/3646699571' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "90a2b63a-471d-43b8-ac18-bb1f16778d18"}]: dispatch 2024-03-23T22:56:42.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:42 smithi203 ceph-mon[36693]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "90a2b63a-471d-43b8-ac18-bb1f16778d18"}]: dispatch 2024-03-23T22:56:42.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:42 smithi203 ceph-mon[36693]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "90a2b63a-471d-43b8-ac18-bb1f16778d18"}]': finished 2024-03-23T22:56:42.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:42 smithi203 ceph-mon[36693]: osdmap e57: 9 total, 8 up, 9 in 2024-03-23T22:56:42.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:42 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-23T22:56:42.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:42 smithi012 ceph-mon[31695]: from='client.? 172.21.15.203:0/3646699571' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "90a2b63a-471d-43b8-ac18-bb1f16778d18"}]: dispatch 2024-03-23T22:56:42.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:42 smithi012 ceph-mon[31695]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "90a2b63a-471d-43b8-ac18-bb1f16778d18"}]: dispatch 2024-03-23T22:56:42.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:42 smithi012 ceph-mon[31695]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "90a2b63a-471d-43b8-ac18-bb1f16778d18"}]': finished 2024-03-23T22:56:42.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:42 smithi012 ceph-mon[31695]: osdmap e57: 9 total, 8 up, 9 in 2024-03-23T22:56:42.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:42 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-23T22:56:42.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:42 smithi173 ceph-mon[36116]: from='client.? 172.21.15.203:0/3646699571' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "90a2b63a-471d-43b8-ac18-bb1f16778d18"}]: dispatch 2024-03-23T22:56:42.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:42 smithi173 ceph-mon[36116]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "90a2b63a-471d-43b8-ac18-bb1f16778d18"}]: dispatch 2024-03-23T22:56:42.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:42 smithi173 ceph-mon[36116]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "90a2b63a-471d-43b8-ac18-bb1f16778d18"}]': finished 2024-03-23T22:56:42.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:42 smithi173 ceph-mon[36116]: osdmap e57: 9 total, 8 up, 9 in 2024-03-23T22:56:42.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:42 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-23T22:56:43.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:43 smithi203 ceph-mon[36693]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:43.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:43 smithi203 ceph-mon[36693]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-23T22:56:43.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:43 smithi203 ceph-mon[36693]: Cluster is now healthy 2024-03-23T22:56:43.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:43 smithi203 ceph-mon[36693]: from='client.? 172.21.15.203:0/2733657723' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:56:43.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:43 smithi012 ceph-mon[31695]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:43.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:43 smithi012 ceph-mon[31695]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-23T22:56:43.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:43 smithi012 ceph-mon[31695]: Cluster is now healthy 2024-03-23T22:56:43.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:43 smithi012 ceph-mon[31695]: from='client.? 172.21.15.203:0/2733657723' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:56:43.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:43 smithi173 ceph-mon[36116]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:43.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:43 smithi173 ceph-mon[36116]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-23T22:56:43.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:43 smithi173 ceph-mon[36116]: Cluster is now healthy 2024-03-23T22:56:43.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:43 smithi173 ceph-mon[36116]: from='client.? 172.21.15.203:0/2733657723' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:56:45.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:45 smithi203 ceph-mon[36693]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:45.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:45 smithi012 ceph-mon[31695]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:45.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:45 smithi173 ceph-mon[36116]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:47.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:47 smithi203 ceph-mon[36693]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:47.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:47 smithi012 ceph-mon[31695]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:47.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:47 smithi173 ceph-mon[36116]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:48.372 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:48 smithi203 ceph-mon[36693]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:48.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:48 smithi012 ceph-mon[31695]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:48.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:48 smithi173 ceph-mon[36116]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:49.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:49 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-23T22:56:49.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:49 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:56:49.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:49 smithi203 ceph-mon[36693]: Deploying daemon osd.8 on smithi203 2024-03-23T22:56:49.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:49 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-23T22:56:49.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:49 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:56:49.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:49 smithi012 ceph-mon[31695]: Deploying daemon osd.8 on smithi203 2024-03-23T22:56:49.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:49 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-23T22:56:49.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:49 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:56:49.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:49 smithi173 ceph-mon[36116]: Deploying daemon osd.8 on smithi203 2024-03-23T22:56:50.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:50 smithi203 ceph-mon[36693]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:50.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:50 smithi012 ceph-mon[31695]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:50.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:50 smithi173 ceph-mon[36116]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:52.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:52 smithi203 ceph-mon[36693]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:52.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:52 smithi012 ceph-mon[31695]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:52.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:52 smithi173 ceph-mon[36116]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:54.399 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:54 smithi203 ceph-mon[36693]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:54.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:54 smithi012 ceph-mon[31695]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:54.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:54 smithi173 ceph-mon[36116]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:55.503 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:55 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:56:55.503 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:55 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:55.503 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:55 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:55.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:55 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:56:55.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:55 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:55.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:55 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:55.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:55 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:56:55.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:55 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:55.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:55 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:57.082 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:56 smithi203 ceph-mon[36693]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:57.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:56 smithi173 ceph-mon[36116]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:57.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:56 smithi012 ceph-mon[31695]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:58.425 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 8 on host 'smithi203' 2024-03-23T22:56:58.425 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:58.424+0000 7fcd36ffd700 1 -- 172.21.15.203:0/2827057338 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fcd400625d0 con 0x7fcd2c0a2890 2024-03-23T22:56:58.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:58.427+0000 7fcd47adc700 1 -- 172.21.15.203:0/2827057338 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcd2c0a2890 msgr2=0x7fcd2c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:56:58.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:58.427+0000 7fcd47adc700 1 --2- 172.21.15.203:0/2827057338 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcd2c0a2890 0x7fcd2c0a4d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcd4006b990 tx=0x7fcd3c008040 comp rx=0 tx=0).stop 2024-03-23T22:56:58.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:58.427+0000 7fcd47adc700 1 -- 172.21.15.203:0/2827057338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd401010c0 msgr2=0x7fcd4006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:56:58.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:58.427+0000 7fcd47adc700 1 --2- 172.21.15.203:0/2827057338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd401010c0 0x7fcd4006b410 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcd3000b500 tx=0x7fcd3000b8c0 comp rx=0 tx=0).stop 2024-03-23T22:56:58.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:58.427+0000 7fcd47adc700 1 -- 172.21.15.203:0/2827057338 shutdown_connections 2024-03-23T22:56:58.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:58.427+0000 7fcd47adc700 1 --2- 172.21.15.203:0/2827057338 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcd2c0a2890 0x7fcd2c0a4d50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:56:58.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:58.427+0000 7fcd47adc700 1 --2- 172.21.15.203:0/2827057338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd401010c0 0x7fcd4006b410 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:56:58.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:58.428+0000 7fcd47adc700 1 --2- 172.21.15.203:0/2827057338 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd40102c60 0x7fcd4019f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:56:58.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:58.428+0000 7fcd47adc700 1 --2- 172.21.15.203:0/2827057338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd401022c0 0x7fcd4019d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:56:58.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:58.428+0000 7fcd47adc700 1 -- 172.21.15.203:0/2827057338 >> 172.21.15.203:0/2827057338 conn(0x7fcd400fc680 msgr2=0x7fcd401055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:56:58.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:58.428+0000 7fcd47adc700 1 -- 172.21.15.203:0/2827057338 shutdown_connections 2024-03-23T22:56:58.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:56:58.428+0000 7fcd47adc700 1 -- 172.21.15.203:0/2827057338 wait complete. 2024-03-23T22:56:59.107 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:58 smithi012 ceph-mon[31695]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:59.107 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:58 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:59.107 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:58 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:59.107 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:58 smithi012 ceph-mon[31695]: from='osd.8 [v2:172.21.15.203:6816/1516996192,v1:172.21.15.203:6817/1516996192]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-23T22:56:59.108 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:58 smithi012 ceph-mon[31695]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-23T22:56:59.108 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:58 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:59.108 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:56:58 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:59.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:58 smithi173 ceph-mon[36116]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:59.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:58 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:59.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:58 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:59.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:58 smithi173 ceph-mon[36116]: from='osd.8 [v2:172.21.15.203:6816/1516996192,v1:172.21.15.203:6817/1516996192]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-23T22:56:59.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:58 smithi173 ceph-mon[36116]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-23T22:56:59.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:58 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:59.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:56:58 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:59.137 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:58 smithi203 ceph-mon[36693]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:56:59.137 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:58 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:59.137 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:58 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:59.137 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:58 smithi203 ceph-mon[36693]: from='osd.8 [v2:172.21.15.203:6816/1516996192,v1:172.21.15.203:6817/1516996192]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-23T22:56:59.137 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:58 smithi203 ceph-mon[36693]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-23T22:56:59.137 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:58 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:59.138 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:56:58 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:56:59.164 DEBUG:teuthology.orchestra.run.smithi203:osd.8> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.8.service 2024-03-23T22:56:59.166 INFO:tasks.cephadm:Deploying osd.9 on smithi012 with /dev/vg_nvme/lv_1... 2024-03-23T22:56:59.167 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-23T22:56:59.638 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:57:00.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:00 smithi012 ceph-mon[31695]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:57:00.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:00 smithi012 ceph-mon[31695]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-23T22:57:00.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:00 smithi012 ceph-mon[31695]: from='osd.8 [v2:172.21.15.203:6816/1516996192,v1:172.21.15.203:6817/1516996192]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-23T22:57:00.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:00 smithi012 ceph-mon[31695]: osdmap e58: 9 total, 8 up, 9 in 2024-03-23T22:57:00.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:00 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-23T22:57:00.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:00 smithi012 ceph-mon[31695]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-23T22:57:00.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:00 smithi203 ceph-mon[36693]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:57:00.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:00 smithi203 ceph-mon[36693]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-23T22:57:00.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:00 smithi203 ceph-mon[36693]: from='osd.8 [v2:172.21.15.203:6816/1516996192,v1:172.21.15.203:6817/1516996192]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-23T22:57:00.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:00 smithi203 ceph-mon[36693]: osdmap e58: 9 total, 8 up, 9 in 2024-03-23T22:57:00.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:00 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-23T22:57:00.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:00 smithi203 ceph-mon[36693]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-23T22:57:00.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:00 smithi173 ceph-mon[36116]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:57:00.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:00 smithi173 ceph-mon[36116]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-23T22:57:00.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:00 smithi173 ceph-mon[36116]: from='osd.8 [v2:172.21.15.203:6816/1516996192,v1:172.21.15.203:6817/1516996192]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-23T22:57:00.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:00 smithi173 ceph-mon[36116]: osdmap e58: 9 total, 8 up, 9 in 2024-03-23T22:57:00.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:00 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-23T22:57:00.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:00 smithi173 ceph-mon[36116]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-23T22:57:00.943 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 22:57:00 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T22:57:00.828+0000 7f39ce718700 -1 osd.8 0 waiting for initial osdmap 2024-03-23T22:57:00.943 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 22:57:00 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T22:57:00.844+0000 7f39c6503700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T22:57:01.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:01 smithi012 ceph-mon[31695]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-23T22:57:01.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:01 smithi012 ceph-mon[31695]: osdmap e59: 9 total, 8 up, 9 in 2024-03-23T22:57:01.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:01 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-23T22:57:01.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:01 smithi012 ceph-mon[31695]: from='osd.8 ' entity='osd.8' 2024-03-23T22:57:01.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:01 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-23T22:57:01.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:01 smithi203 ceph-mon[36693]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-23T22:57:01.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:01 smithi203 ceph-mon[36693]: osdmap e59: 9 total, 8 up, 9 in 2024-03-23T22:57:01.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:01 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-23T22:57:01.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:01 smithi203 ceph-mon[36693]: from='osd.8 ' entity='osd.8' 2024-03-23T22:57:01.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:01 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-23T22:57:01.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:01 smithi173 ceph-mon[36116]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-23T22:57:01.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:01 smithi173 ceph-mon[36116]: osdmap e59: 9 total, 8 up, 9 in 2024-03-23T22:57:01.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:01 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-23T22:57:01.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:01 smithi173 ceph-mon[36116]: from='osd.8 ' entity='osd.8' 2024-03-23T22:57:01.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:01 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-23T22:57:02.539 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:02 smithi203 ceph-mon[36693]: purged_snaps scrub starts 2024-03-23T22:57:02.539 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:02 smithi203 ceph-mon[36693]: purged_snaps scrub ok 2024-03-23T22:57:02.539 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:02 smithi203 ceph-mon[36693]: pgmap v218: 1 pgs: 1 peering; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:57:02.539 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:02 smithi203 ceph-mon[36693]: osd.8 [v2:172.21.15.203:6816/1516996192,v1:172.21.15.203:6817/1516996192] boot 2024-03-23T22:57:02.539 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:02 smithi203 ceph-mon[36693]: osdmap e60: 9 total, 9 up, 9 in 2024-03-23T22:57:02.540 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:02 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-23T22:57:02.540 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:02 smithi203 ceph-mon[36693]: osdmap e61: 9 total, 9 up, 9 in 2024-03-23T22:57:02.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:02 smithi012 ceph-mon[31695]: purged_snaps scrub starts 2024-03-23T22:57:02.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:02 smithi012 ceph-mon[31695]: purged_snaps scrub ok 2024-03-23T22:57:02.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:02 smithi012 ceph-mon[31695]: pgmap v218: 1 pgs: 1 peering; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:57:02.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:02 smithi012 ceph-mon[31695]: osd.8 [v2:172.21.15.203:6816/1516996192,v1:172.21.15.203:6817/1516996192] boot 2024-03-23T22:57:02.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:02 smithi012 ceph-mon[31695]: osdmap e60: 9 total, 9 up, 9 in 2024-03-23T22:57:02.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:02 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-23T22:57:02.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:02 smithi012 ceph-mon[31695]: osdmap e61: 9 total, 9 up, 9 in 2024-03-23T22:57:02.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:02 smithi173 ceph-mon[36116]: purged_snaps scrub starts 2024-03-23T22:57:02.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:02 smithi173 ceph-mon[36116]: purged_snaps scrub ok 2024-03-23T22:57:02.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:02 smithi173 ceph-mon[36116]: pgmap v218: 1 pgs: 1 peering; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-23T22:57:02.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:02 smithi173 ceph-mon[36116]: osd.8 [v2:172.21.15.203:6816/1516996192,v1:172.21.15.203:6817/1516996192] boot 2024-03-23T22:57:02.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:02 smithi173 ceph-mon[36116]: osdmap e60: 9 total, 9 up, 9 in 2024-03-23T22:57:02.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:02 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-23T22:57:02.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:02 smithi173 ceph-mon[36116]: osdmap e61: 9 total, 9 up, 9 in 2024-03-23T22:57:02.915 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-23T22:57:02.915 INFO:teuthology.orchestra.run.smithi012.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-23T22:57:02.915 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10+0 records in 2024-03-23T22:57:02.915 INFO:teuthology.orchestra.run.smithi012.stderr:10+0 records out 2024-03-23T22:57:02.915 INFO:teuthology.orchestra.run.smithi012.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0382953 s, 274 MB/s 2024-03-23T22:57:02.916 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping successful for: 2024-03-23T22:57:03.803 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi012:vg_nvme/lv_1 2024-03-23T22:57:03.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:03 smithi173 ceph-mon[36116]: Detected new or changed devices on smithi203 2024-03-23T22:57:03.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:03.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:03.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:57:03.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:57:03.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:57:03.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:03 smithi173 ceph-mon[36116]: Adjusting osd_memory_target on smithi203 to 5693M 2024-03-23T22:57:03.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:03.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:57:03.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:57:03.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:03.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:57:03.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:57:03.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:57:03.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:03.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:03 smithi173 ceph-mon[36116]: osdmap e62: 9 total, 9 up, 9 in 2024-03-23T22:57:03.894 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:03 smithi012 ceph-mon[31695]: Detected new or changed devices on smithi203 2024-03-23T22:57:03.894 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:03.894 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:03.894 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:57:03.894 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:57:03.894 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:57:03.894 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:03 smithi012 ceph-mon[31695]: Adjusting osd_memory_target on smithi203 to 5693M 2024-03-23T22:57:03.894 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:03.894 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:57:03.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:57:03.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:03.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:57:03.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:57:03.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:57:03.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:03.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:03 smithi012 ceph-mon[31695]: osdmap e62: 9 total, 9 up, 9 in 2024-03-23T22:57:03.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:03 smithi203 ceph-mon[36693]: Detected new or changed devices on smithi203 2024-03-23T22:57:03.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:03.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:03.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:57:03.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:57:03.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:57:03.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:03 smithi203 ceph-mon[36693]: Adjusting osd_memory_target on smithi203 to 5693M 2024-03-23T22:57:03.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:03.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:57:03.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:57:03.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:03.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:57:03.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:57:03.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:57:03.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:03.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:03 smithi203 ceph-mon[36693]: osdmap e62: 9 total, 9 up, 9 in 2024-03-23T22:57:04.266 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:57:05.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:05 smithi173 ceph-mon[36116]: pgmap v222: 1 pgs: 1 peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:05.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:05 smithi173 ceph-mon[36116]: osdmap e63: 9 total, 9 up, 9 in 2024-03-23T22:57:05.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:05 smithi203 ceph-mon[36693]: pgmap v222: 1 pgs: 1 peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:05.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:05 smithi203 ceph-mon[36693]: osdmap e63: 9 total, 9 up, 9 in 2024-03-23T22:57:05.465 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:05 smithi012 ceph-mon[31695]: pgmap v222: 1 pgs: 1 peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:05.465 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:05 smithi012 ceph-mon[31695]: osdmap e63: 9 total, 9 up, 9 in 2024-03-23T22:57:06.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:06 smithi012 ceph-mon[31695]: osdmap e64: 9 total, 9 up, 9 in 2024-03-23T22:57:06.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:06 smithi203 ceph-mon[36693]: osdmap e64: 9 total, 9 up, 9 in 2024-03-23T22:57:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.594+0000 7f6747e00700 1 -- 172.21.15.12:0/503025952 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6748103ca0 msgr2=0x7f674810a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:57:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.594+0000 7f6747e00700 1 --2- 172.21.15.12:0/503025952 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6748103ca0 0x7f674810a550 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f67400099c0 tx=0x7f6740009cd0 comp rx=0 tx=0).stop 2024-03-23T22:57:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.595+0000 7f6747e00700 1 -- 172.21.15.12:0/503025952 shutdown_connections 2024-03-23T22:57:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.595+0000 7f6747e00700 1 --2- 172.21.15.12:0/503025952 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6748103ca0 0x7f674810a550 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:57:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.595+0000 7f6747e00700 1 --2- 172.21.15.12:0/503025952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6748102100 0x7f6748102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:57:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.595+0000 7f6747e00700 1 --2- 172.21.15.12:0/503025952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6748103300 0x7f6748103760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:57:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.595+0000 7f6747e00700 1 -- 172.21.15.12:0/503025952 >> 172.21.15.12:0/503025952 conn(0x7f67480fd6a0 msgr2=0x7f67480ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:57:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.595+0000 7f6747e00700 1 -- 172.21.15.12:0/503025952 shutdown_connections 2024-03-23T22:57:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.595+0000 7f6747e00700 1 -- 172.21.15.12:0/503025952 wait complete. 2024-03-23T22:57:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.596+0000 7f6747e00700 1 Processor -- start 2024-03-23T22:57:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.596+0000 7f6747e00700 1 -- start start 2024-03-23T22:57:06.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.597+0000 7f6747e00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6748102100 0x7f6748075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:57:06.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.597+0000 7f6747e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6748103300 0x7f6748074580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:57:06.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.597+0000 7f6747e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6748103ca0 0x7f6748074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:57:06.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.597+0000 7f6747e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674810dcd0 con 0x7f6748103300 2024-03-23T22:57:06.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.597+0000 7f6747e00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674810db50 con 0x7f6748103ca0 2024-03-23T22:57:06.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.597+0000 7f6747e00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674810de50 con 0x7f6748102100 2024-03-23T22:57:06.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.597+0000 7f6746a00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6748102100 0x7f6748075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:57:06.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.597+0000 7f673e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6748103300 0x7f6748074580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:57:06.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.597+0000 7f6746a00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6748102100 0x7f6748075f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54558/0 (socket says 172.21.15.12:54558) 2024-03-23T22:57:06.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.597+0000 7f6746a00700 1 -- 172.21.15.12:0/47701357 learned_addr learned my addr 172.21.15.12:0/47701357 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:57:06.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.597+0000 7f6747400700 1 --2- 172.21.15.12:0/47701357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6748103ca0 0x7f6748074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:57:06.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.598+0000 7f6746a00700 1 -- 172.21.15.12:0/47701357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6748103ca0 msgr2=0x7f6748074ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:57:06.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.598+0000 7f6746a00700 1 --2- 172.21.15.12:0/47701357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6748103ca0 0x7f6748074ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:57:06.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.598+0000 7f6746a00700 1 -- 172.21.15.12:0/47701357 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6748103300 msgr2=0x7f6748074580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:57:06.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.598+0000 7f6746a00700 1 --2- 172.21.15.12:0/47701357 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6748103300 0x7f6748074580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:57:06.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.598+0000 7f6746a00700 1 -- 172.21.15.12:0/47701357 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6748075370 con 0x7f6748102100 2024-03-23T22:57:06.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.598+0000 7f6746a00700 1 --2- 172.21.15.12:0/47701357 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6748102100 0x7f6748075f30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f673800efc0 tx=0x7f673800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:57:06.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.598+0000 7f673fe00700 1 -- 172.21.15.12:0/47701357 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67380092e0 con 0x7f6748102100 2024-03-23T22:57:06.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.598+0000 7f6747e00700 1 -- 172.21.15.12:0/47701357 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67481a2360 con 0x7f6748102100 2024-03-23T22:57:06.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.598+0000 7f673fe00700 1 -- 172.21.15.12:0/47701357 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f673800c9c0 con 0x7f6748102100 2024-03-23T22:57:06.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.598+0000 7f673fe00700 1 -- 172.21.15.12:0/47701357 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738004610 con 0x7f6748102100 2024-03-23T22:57:06.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.599+0000 7f6747e00700 1 -- 172.21.15.12:0/47701357 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67481a28a0 con 0x7f6748102100 2024-03-23T22:57:06.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.600+0000 7f6747e00700 1 -- 172.21.15.12:0/47701357 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6748068e50 con 0x7f6748102100 2024-03-23T22:57:06.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.602+0000 7f673fe00700 1 -- 172.21.15.12:0/47701357 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f673801b030 con 0x7f6748102100 2024-03-23T22:57:06.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.603+0000 7f673fe00700 1 --2- 172.21.15.12:0/47701357 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f672c0a2890 0x7f672c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:57:06.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.603+0000 7f673fe00700 1 -- 172.21.15.12:0/47701357 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5829+0+0 (secure 0 0 0) 0x7f67380cdfe0 con 0x7f6748102100 2024-03-23T22:57:06.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.603+0000 7f673e000700 1 --2- 172.21.15.12:0/47701357 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f672c0a2890 0x7f672c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:57:06.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.604+0000 7f673e000700 1 --2- 172.21.15.12:0/47701357 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f672c0a2890 0x7f672c0a4d50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6730009c60 tx=0x7f6730009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:57:06.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.608+0000 7f673fe00700 1 -- 172.21.15.12:0/47701357 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6738022020 con 0x7f6748102100 2024-03-23T22:57:06.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:06 smithi173 ceph-mon[36116]: osdmap e64: 9 total, 9 up, 9 in 2024-03-23T22:57:06.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:06.847+0000 7f6747e00700 1 -- 172.21.15.12:0/47701357 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f67480625d0 con 0x7f672c0a2890 2024-03-23T22:57:07.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:07 smithi203 ceph-mon[36693]: pgmap v225: 1 pgs: 1 peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:07.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:07 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:57:07.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:07 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:57:07.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:07 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:57:07.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:07 smithi012 ceph-mon[31695]: pgmap v225: 1 pgs: 1 peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:07.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:07 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:57:07.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:07 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:57:07.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:07 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:57:07.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:07 smithi173 ceph-mon[36116]: pgmap v225: 1 pgs: 1 peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:07.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:07 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:57:07.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:07 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:57:07.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:07 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:57:08.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:08 smithi203 ceph-mon[36693]: from='client.24340 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:57:08.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:08 smithi012 ceph-mon[31695]: from='client.24340 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:57:08.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:08 smithi173 ceph-mon[36116]: from='client.24340 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:57:09.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:09 smithi203 ceph-mon[36693]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-23T22:57:09.449 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:09 smithi012 ceph-mon[31695]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-23T22:57:09.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:09 smithi173 ceph-mon[36116]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-23T22:57:11.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:11 smithi203 ceph-mon[36693]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-23T22:57:11.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:11 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3681741605' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e1ca504-40b4-49d0-b245-7946181abfd1"}]: dispatch 2024-03-23T22:57:11.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:11 smithi203 ceph-mon[36693]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e1ca504-40b4-49d0-b245-7946181abfd1"}]: dispatch 2024-03-23T22:57:11.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:11 smithi203 ceph-mon[36693]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e1ca504-40b4-49d0-b245-7946181abfd1"}]': finished 2024-03-23T22:57:11.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:11 smithi203 ceph-mon[36693]: osdmap e65: 10 total, 9 up, 10 in 2024-03-23T22:57:11.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:11 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-23T22:57:11.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:11 smithi012 ceph-mon[31695]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-23T22:57:11.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:11 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3681741605' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e1ca504-40b4-49d0-b245-7946181abfd1"}]: dispatch 2024-03-23T22:57:11.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:11 smithi012 ceph-mon[31695]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e1ca504-40b4-49d0-b245-7946181abfd1"}]: dispatch 2024-03-23T22:57:11.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:11 smithi012 ceph-mon[31695]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e1ca504-40b4-49d0-b245-7946181abfd1"}]': finished 2024-03-23T22:57:11.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:11 smithi012 ceph-mon[31695]: osdmap e65: 10 total, 9 up, 10 in 2024-03-23T22:57:11.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:11 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-23T22:57:11.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:11 smithi173 ceph-mon[36116]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-23T22:57:11.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:11 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3681741605' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e1ca504-40b4-49d0-b245-7946181abfd1"}]: dispatch 2024-03-23T22:57:11.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:11 smithi173 ceph-mon[36116]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e1ca504-40b4-49d0-b245-7946181abfd1"}]: dispatch 2024-03-23T22:57:11.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:11 smithi173 ceph-mon[36116]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e1ca504-40b4-49d0-b245-7946181abfd1"}]': finished 2024-03-23T22:57:11.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:11 smithi173 ceph-mon[36116]: osdmap e65: 10 total, 9 up, 10 in 2024-03-23T22:57:11.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:11 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-23T22:57:12.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:12 smithi203 ceph-mon[36693]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-23T22:57:12.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:12 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2925861711' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:57:12.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:12 smithi012 ceph-mon[31695]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-23T22:57:12.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:12 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2925861711' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:57:12.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:12 smithi173 ceph-mon[36116]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-23T22:57:12.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:12 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2925861711' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:57:14.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:14 smithi012 ceph-mon[31695]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-23T22:57:14.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:14 smithi173 ceph-mon[36116]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-23T22:57:14.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:14 smithi203 ceph-mon[36693]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-23T22:57:16.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:16 smithi012 ceph-mon[31695]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-23T22:57:16.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:16 smithi173 ceph-mon[36116]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-23T22:57:16.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:16 smithi203 ceph-mon[36693]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-23T22:57:18.458 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:18 smithi012 ceph-mon[31695]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:18.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:18 smithi173 ceph-mon[36116]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:18.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:18 smithi203 ceph-mon[36693]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:20.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:20 smithi012 ceph-mon[31695]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:20.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:20 smithi173 ceph-mon[36116]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:20.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:20 smithi203 ceph-mon[36693]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:21.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:21 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-23T22:57:21.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:21 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:57:21.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:21 smithi012 ceph-mon[31695]: Deploying daemon osd.9 on smithi012 2024-03-23T22:57:21.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:21 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-23T22:57:21.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:21 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:57:21.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:21 smithi173 ceph-mon[36116]: Deploying daemon osd.9 on smithi012 2024-03-23T22:57:21.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:21 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-23T22:57:21.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:21 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:57:21.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:21 smithi203 ceph-mon[36693]: Deploying daemon osd.9 on smithi012 2024-03-23T22:57:22.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:22 smithi012 ceph-mon[31695]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:22.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:22 smithi173 ceph-mon[36116]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:22.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:22 smithi203 ceph-mon[36693]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:24.499 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:24 smithi012 ceph-mon[31695]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:24.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:24 smithi173 ceph-mon[36116]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:24.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:24 smithi203 ceph-mon[36693]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:26.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:26 smithi012 ceph-mon[31695]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:26.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:26 smithi173 ceph-mon[36116]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:26.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:26 smithi203 ceph-mon[36693]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:28.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:28 smithi173 ceph-mon[36116]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:28.646 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:28 smithi012 ceph-mon[31695]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:28.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:28 smithi203 ceph-mon[36693]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:30.116 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:29 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:30.117 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:29 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:30.117 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:29 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:57:30.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:29 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:30.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:29 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:30.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:29 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:57:30.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:29 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:30.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:29 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:30.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:29 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:57:30.957 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:30 smithi012 ceph-mon[31695]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:31.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:30 smithi203 ceph-mon[36693]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:31.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:30 smithi173 ceph-mon[36116]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:33.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:32 smithi203 ceph-mon[36693]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:33.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:32 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:33.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:32 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:33.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:32 smithi203 ceph-mon[36693]: from='osd.9 [v2:172.21.15.12:6826/1175260774,v1:172.21.15.12:6827/1175260774]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-23T22:57:33.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:32 smithi203 ceph-mon[36693]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-23T22:57:33.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:32 smithi012 ceph-mon[31695]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:33.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:32 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:33.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:32 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:33.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:32 smithi012 ceph-mon[31695]: from='osd.9 [v2:172.21.15.12:6826/1175260774,v1:172.21.15.12:6827/1175260774]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-23T22:57:33.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:32 smithi012 ceph-mon[31695]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-23T22:57:33.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:32 smithi173 ceph-mon[36116]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:33.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:32 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:33.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:32 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:33.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:32 smithi173 ceph-mon[36116]: from='osd.9 [v2:172.21.15.12:6826/1175260774,v1:172.21.15.12:6827/1175260774]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-23T22:57:33.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:32 smithi173 ceph-mon[36116]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-23T22:57:33.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:33.866+0000 7f673fe00700 1 -- 172.21.15.12:0/47701357 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f67480625d0 con 0x7f672c0a2890 2024-03-23T22:57:33.867 INFO:teuthology.orchestra.run.smithi012.stdout:Created osd(s) 9 on host 'smithi012' 2024-03-23T22:57:33.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:33.870+0000 7f6747e00700 1 -- 172.21.15.12:0/47701357 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f672c0a2890 msgr2=0x7f672c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:57:33.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:33.870+0000 7f6747e00700 1 --2- 172.21.15.12:0/47701357 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f672c0a2890 0x7f672c0a4d50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6730009c60 tx=0x7f6730009260 comp rx=0 tx=0).stop 2024-03-23T22:57:33.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:33.870+0000 7f6747e00700 1 -- 172.21.15.12:0/47701357 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6748102100 msgr2=0x7f6748075f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:57:33.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:33.870+0000 7f6747e00700 1 --2- 172.21.15.12:0/47701357 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6748102100 0x7f6748075f30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f673800efc0 tx=0x7f673800c3a0 comp rx=0 tx=0).stop 2024-03-23T22:57:33.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:33.870+0000 7f6747e00700 1 -- 172.21.15.12:0/47701357 shutdown_connections 2024-03-23T22:57:33.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:33.870+0000 7f6747e00700 1 --2- 172.21.15.12:0/47701357 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f672c0a2890 0x7f672c0a4d50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:57:33.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:33.870+0000 7f6747e00700 1 --2- 172.21.15.12:0/47701357 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6748102100 0x7f6748075f30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:57:33.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:33.870+0000 7f6747e00700 1 --2- 172.21.15.12:0/47701357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6748103ca0 0x7f6748074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:57:33.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:33.870+0000 7f6747e00700 1 --2- 172.21.15.12:0/47701357 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6748103300 0x7f6748074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:57:33.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:33.870+0000 7f6747e00700 1 -- 172.21.15.12:0/47701357 >> 172.21.15.12:0/47701357 conn(0x7f67480fd6a0 msgr2=0x7f6748106530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:57:33.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:33.871+0000 7f6747e00700 1 -- 172.21.15.12:0/47701357 shutdown_connections 2024-03-23T22:57:33.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:57:33.871+0000 7f6747e00700 1 -- 172.21.15.12:0/47701357 wait complete. 2024-03-23T22:57:34.731 DEBUG:teuthology.orchestra.run.smithi012:osd.9> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.9.service 2024-03-23T22:57:34.736 INFO:tasks.cephadm:Deploying osd.10 on smithi173 with /dev/vg_nvme/lv_1... 2024-03-23T22:57:34.736 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-23T22:57:34.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:34 smithi203 ceph-mon[36693]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:34.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:34 smithi203 ceph-mon[36693]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-23T22:57:34.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:34 smithi203 ceph-mon[36693]: from='osd.9 [v2:172.21.15.12:6826/1175260774,v1:172.21.15.12:6827/1175260774]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-23T22:57:34.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:34 smithi203 ceph-mon[36693]: osdmap e66: 10 total, 9 up, 10 in 2024-03-23T22:57:34.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:34 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-23T22:57:34.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:34 smithi203 ceph-mon[36693]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-23T22:57:34.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:34 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:34.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:34 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:34.982 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:34 smithi012 ceph-mon[31695]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:34.982 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:34 smithi012 ceph-mon[31695]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-23T22:57:34.982 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:34 smithi012 ceph-mon[31695]: from='osd.9 [v2:172.21.15.12:6826/1175260774,v1:172.21.15.12:6827/1175260774]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-23T22:57:34.982 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:34 smithi012 ceph-mon[31695]: osdmap e66: 10 total, 9 up, 10 in 2024-03-23T22:57:34.982 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:34 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-23T22:57:34.982 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:34 smithi012 ceph-mon[31695]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-23T22:57:34.982 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:34 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:34.983 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:34 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:34.983 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.b/config 2024-03-23T22:57:35.105 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:34 smithi173 ceph-mon[36116]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:35.105 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:34 smithi173 ceph-mon[36116]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-23T22:57:35.105 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:34 smithi173 ceph-mon[36116]: from='osd.9 [v2:172.21.15.12:6826/1175260774,v1:172.21.15.12:6827/1175260774]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-23T22:57:35.105 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:34 smithi173 ceph-mon[36116]: osdmap e66: 10 total, 9 up, 10 in 2024-03-23T22:57:35.105 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:34 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-23T22:57:35.106 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:34 smithi173 ceph-mon[36116]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-23T22:57:35.106 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:34 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:35.106 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:34 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:36.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:35 smithi012 ceph-mon[31695]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-23T22:57:36.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:35 smithi012 ceph-mon[31695]: osdmap e67: 10 total, 9 up, 10 in 2024-03-23T22:57:36.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:35 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-23T22:57:36.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:35 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-23T22:57:36.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:35 smithi173 ceph-mon[36116]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-23T22:57:36.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:35 smithi173 ceph-mon[36116]: osdmap e67: 10 total, 9 up, 10 in 2024-03-23T22:57:36.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:35 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-23T22:57:36.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:35 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-23T22:57:36.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:35 smithi203 ceph-mon[36693]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-23T22:57:36.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:35 smithi203 ceph-mon[36693]: osdmap e67: 10 total, 9 up, 10 in 2024-03-23T22:57:36.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:35 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-23T22:57:36.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:35 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-23T22:57:36.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:36 smithi012 ceph-mon[31695]: purged_snaps scrub starts 2024-03-23T22:57:36.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:36 smithi012 ceph-mon[31695]: purged_snaps scrub ok 2024-03-23T22:57:36.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:36 smithi012 ceph-mon[31695]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:36.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:36 smithi012 ceph-mon[31695]: osd.9 [v2:172.21.15.12:6826/1175260774,v1:172.21.15.12:6827/1175260774] boot 2024-03-23T22:57:36.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:36 smithi012 ceph-mon[31695]: osdmap e68: 10 total, 10 up, 10 in 2024-03-23T22:57:36.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:36 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-23T22:57:36.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:36 smithi173 ceph-mon[36116]: purged_snaps scrub starts 2024-03-23T22:57:36.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:36 smithi173 ceph-mon[36116]: purged_snaps scrub ok 2024-03-23T22:57:36.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:36 smithi173 ceph-mon[36116]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:36.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:36 smithi173 ceph-mon[36116]: osd.9 [v2:172.21.15.12:6826/1175260774,v1:172.21.15.12:6827/1175260774] boot 2024-03-23T22:57:36.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:36 smithi173 ceph-mon[36116]: osdmap e68: 10 total, 10 up, 10 in 2024-03-23T22:57:36.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:36 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-23T22:57:37.047 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-23T22:57:37.047 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-23T22:57:37.047 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-03-23T22:57:37.047 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-03-23T22:57:37.047 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112161 s, 935 MB/s 2024-03-23T22:57:37.047 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-03-23T22:57:37.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:36 smithi203 ceph-mon[36693]: purged_snaps scrub starts 2024-03-23T22:57:37.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:36 smithi203 ceph-mon[36693]: purged_snaps scrub ok 2024-03-23T22:57:37.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:36 smithi203 ceph-mon[36693]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-23T22:57:37.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:36 smithi203 ceph-mon[36693]: osd.9 [v2:172.21.15.12:6826/1175260774,v1:172.21.15.12:6827/1175260774] boot 2024-03-23T22:57:37.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:36 smithi203 ceph-mon[36693]: osdmap e68: 10 total, 10 up, 10 in 2024-03-23T22:57:37.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:36 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-23T22:57:37.659 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi173:vg_nvme/lv_1 2024-03-23T22:57:37.898 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.b/config 2024-03-23T22:57:38.016 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:37 smithi173 ceph-mon[36116]: osdmap e69: 10 total, 10 up, 10 in 2024-03-23T22:57:38.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:37 smithi012 ceph-mon[31695]: osdmap e69: 10 total, 10 up, 10 in 2024-03-23T22:57:38.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:37 smithi203 ceph-mon[36693]: osdmap e69: 10 total, 10 up, 10 in 2024-03-23T22:57:39.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:38 smithi173 ceph-mon[36116]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:39.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:38 smithi012 ceph-mon[31695]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:39.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:38 smithi203 ceph-mon[36693]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:39.556 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.555+0000 7fc361b73700 1 -- 172.21.15.173:0/3791130105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc35c102c60 msgr2=0x7fc35c109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:57:39.557 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.555+0000 7fc361b73700 1 --2- 172.21.15.173:0/3791130105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc35c102c60 0x7fc35c109510 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc35400b0d0 tx=0x7fc35400b3e0 comp rx=0 tx=0).stop 2024-03-23T22:57:39.557 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.555+0000 7fc361b73700 1 -- 172.21.15.173:0/3791130105 shutdown_connections 2024-03-23T22:57:39.557 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.555+0000 7fc361b73700 1 --2- 172.21.15.173:0/3791130105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc35c102c60 0x7fc35c109510 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:57:39.557 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.555+0000 7fc361b73700 1 --2- 172.21.15.173:0/3791130105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc35c1010c0 0x7fc35c1014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:57:39.557 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.555+0000 7fc361b73700 1 --2- 172.21.15.173:0/3791130105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc35c1022c0 0x7fc35c102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:57:39.557 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.555+0000 7fc361b73700 1 -- 172.21.15.173:0/3791130105 >> 172.21.15.173:0/3791130105 conn(0x7fc35c0fc680 msgr2=0x7fc35c0feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:57:39.557 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.555+0000 7fc361b73700 1 -- 172.21.15.173:0/3791130105 shutdown_connections 2024-03-23T22:57:39.557 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.555+0000 7fc361b73700 1 -- 172.21.15.173:0/3791130105 wait complete. 2024-03-23T22:57:39.557 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.555+0000 7fc361b73700 1 Processor -- start 2024-03-23T22:57:39.557 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.556+0000 7fc361b73700 1 -- start start 2024-03-23T22:57:39.557 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.556+0000 7fc361b73700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc35c1010c0 0x7fc35c197250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:57:39.557 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.556+0000 7fc361b73700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc35c1022c0 0x7fc35c197790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:57:39.557 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.556+0000 7fc361b73700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc35c102c60 0x7fc35c19e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:57:39.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.556+0000 7fc361b73700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c10cd30 con 0x7fc35c1010c0 2024-03-23T22:57:39.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.556+0000 7fc361b73700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c10cbb0 con 0x7fc35c102c60 2024-03-23T22:57:39.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.556+0000 7fc361b73700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c10ceb0 con 0x7fc35c1022c0 2024-03-23T22:57:39.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.556+0000 7fc35bfff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc35c102c60 0x7fc35c19e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:57:39.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.556+0000 7fc35bfff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc35c102c60 0x7fc35c19e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:55052/0 (socket says 172.21.15.173:55052) 2024-03-23T22:57:39.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.556+0000 7fc35bfff700 1 -- 172.21.15.173:0/2491417347 learned_addr learned my addr 172.21.15.173:0/2491417347 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-23T22:57:39.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.556+0000 7fc35affd700 1 --2- 172.21.15.173:0/2491417347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc35c1022c0 0x7fc35c197790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:57:39.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.556+0000 7fc35bfff700 1 -- 172.21.15.173:0/2491417347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc35c1022c0 msgr2=0x7fc35c197790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:57:39.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.556+0000 7fc35bfff700 1 --2- 172.21.15.173:0/2491417347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc35c1022c0 0x7fc35c197790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:57:39.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.556+0000 7fc35bfff700 1 -- 172.21.15.173:0/2491417347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc35c1010c0 msgr2=0x7fc35c197250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:57:39.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.556+0000 7fc35bfff700 1 --2- 172.21.15.173:0/2491417347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc35c1010c0 0x7fc35c197250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:57:39.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.556+0000 7fc35bfff700 1 -- 172.21.15.173:0/2491417347 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35c19eeb0 con 0x7fc35c102c60 2024-03-23T22:57:39.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.556+0000 7fc35bfff700 1 --2- 172.21.15.173:0/2491417347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc35c102c60 0x7fc35c19e810 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc3540062a0 tx=0x7fc354007e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:57:39.559 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.556+0000 7fc358ff9700 1 -- 172.21.15.173:0/2491417347 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc354003c00 con 0x7fc35c102c60 2024-03-23T22:57:39.559 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.557+0000 7fc361b73700 1 -- 172.21.15.173:0/2491417347 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc35c06a8b0 con 0x7fc35c102c60 2024-03-23T22:57:39.559 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.557+0000 7fc358ff9700 1 -- 172.21.15.173:0/2491417347 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc354003da0 con 0x7fc35c102c60 2024-03-23T22:57:39.559 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.557+0000 7fc358ff9700 1 -- 172.21.15.173:0/2491417347 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3540055d0 con 0x7fc35c102c60 2024-03-23T22:57:39.559 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.557+0000 7fc361b73700 1 -- 172.21.15.173:0/2491417347 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc35c06ae10 con 0x7fc35c102c60 2024-03-23T22:57:39.560 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.557+0000 7fc361b73700 1 -- 172.21.15.173:0/2491417347 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc35c068e50 con 0x7fc35c102c60 2024-03-23T22:57:39.560 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.559+0000 7fc358ff9700 1 -- 172.21.15.173:0/2491417347 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc354005770 con 0x7fc35c102c60 2024-03-23T22:57:39.560 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.559+0000 7fc358ff9700 1 --2- 172.21.15.173:0/2491417347 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc3480a2890 0x7fc3480a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:57:39.560 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.560+0000 7fc358ff9700 1 -- 172.21.15.173:0/2491417347 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6261+0+0 (secure 0 0 0) 0x7fc35400f020 con 0x7fc35c102c60 2024-03-23T22:57:39.560 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.560+0000 7fc35b7fe700 1 --2- 172.21.15.173:0/2491417347 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc3480a2890 0x7fc3480a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:57:39.561 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.561+0000 7fc35b7fe700 1 --2- 172.21.15.173:0/2491417347 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc3480a2890 0x7fc3480a4d50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc35c102120 tx=0x7fc34c006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:57:39.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.563+0000 7fc358ff9700 1 -- 172.21.15.173:0/2491417347 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc35402a030 con 0x7fc35c102c60 2024-03-23T22:57:39.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:39.774+0000 7fc361b73700 1 -- 172.21.15.173:0/2491417347 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc35c06b070 con 0x7fc3480a2890 2024-03-23T22:57:40.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:40 smithi173 ceph-mon[36116]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:40.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:40 smithi173 ceph-mon[36116]: Detected new or changed devices on smithi012 2024-03-23T22:57:40.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:40.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:40.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:57:40.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:57:40.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:57:40.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:57:40.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:40 smithi173 ceph-mon[36116]: Adjusting osd_memory_target on smithi012 to 4273M 2024-03-23T22:57:40.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:40.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:57:40.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:57:40.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:40.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:40 smithi173 ceph-mon[36116]: from='client.24374 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:57:40.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:57:40.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:57:40.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:57:40.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:57:40.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:57:40.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:57:40.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:41.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:40 smithi012 ceph-mon[31695]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:41.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:40 smithi012 ceph-mon[31695]: Detected new or changed devices on smithi012 2024-03-23T22:57:41.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:41.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:41.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:57:41.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:57:41.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:57:41.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:57:41.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:40 smithi012 ceph-mon[31695]: Adjusting osd_memory_target on smithi012 to 4273M 2024-03-23T22:57:41.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:41.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:57:41.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:57:41.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:41.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:40 smithi012 ceph-mon[31695]: from='client.24374 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:57:41.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:57:41.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:57:41.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:57:41.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:57:41.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:57:41.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:57:41.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:41.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:40 smithi203 ceph-mon[36693]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:41.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:40 smithi203 ceph-mon[36693]: Detected new or changed devices on smithi012 2024-03-23T22:57:41.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:41.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:41.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:57:41.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:57:41.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:57:41.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:57:41.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:40 smithi203 ceph-mon[36693]: Adjusting osd_memory_target on smithi012 to 4273M 2024-03-23T22:57:41.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:41.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:57:41.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:57:41.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:41.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:40 smithi203 ceph-mon[36693]: from='client.24374 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:57:41.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:57:41.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:57:41.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:57:41.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:57:41.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:57:41.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:57:41.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:42.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:42 smithi203 ceph-mon[36693]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:43.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:42 smithi012 ceph-mon[31695]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:43.069 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:42 smithi173 ceph-mon[36116]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:44.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:43 smithi012 ceph-mon[31695]: from='client.? 172.21.15.173:0/2600075568' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09a595ed-5614-4386-8e61-0170af66e01c"}]: dispatch 2024-03-23T22:57:44.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:43 smithi012 ceph-mon[31695]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09a595ed-5614-4386-8e61-0170af66e01c"}]: dispatch 2024-03-23T22:57:44.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:43 smithi012 ceph-mon[31695]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09a595ed-5614-4386-8e61-0170af66e01c"}]': finished 2024-03-23T22:57:44.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:43 smithi012 ceph-mon[31695]: osdmap e70: 11 total, 10 up, 11 in 2024-03-23T22:57:44.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:43 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-23T22:57:44.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:43 smithi012 ceph-mon[31695]: from='client.? 172.21.15.173:0/3152355964' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:57:44.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:43 smithi173 ceph-mon[36116]: from='client.? 172.21.15.173:0/2600075568' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09a595ed-5614-4386-8e61-0170af66e01c"}]: dispatch 2024-03-23T22:57:44.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:43 smithi173 ceph-mon[36116]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09a595ed-5614-4386-8e61-0170af66e01c"}]: dispatch 2024-03-23T22:57:44.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:43 smithi173 ceph-mon[36116]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09a595ed-5614-4386-8e61-0170af66e01c"}]': finished 2024-03-23T22:57:44.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:43 smithi173 ceph-mon[36116]: osdmap e70: 11 total, 10 up, 11 in 2024-03-23T22:57:44.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:43 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-23T22:57:44.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:43 smithi173 ceph-mon[36116]: from='client.? 172.21.15.173:0/3152355964' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:57:44.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:43 smithi203 ceph-mon[36693]: from='client.? 172.21.15.173:0/2600075568' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09a595ed-5614-4386-8e61-0170af66e01c"}]: dispatch 2024-03-23T22:57:44.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:43 smithi203 ceph-mon[36693]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09a595ed-5614-4386-8e61-0170af66e01c"}]: dispatch 2024-03-23T22:57:44.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:43 smithi203 ceph-mon[36693]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09a595ed-5614-4386-8e61-0170af66e01c"}]': finished 2024-03-23T22:57:44.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:43 smithi203 ceph-mon[36693]: osdmap e70: 11 total, 10 up, 11 in 2024-03-23T22:57:44.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:43 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-23T22:57:44.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:43 smithi203 ceph-mon[36693]: from='client.? 172.21.15.173:0/3152355964' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:57:45.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:44 smithi012 ceph-mon[31695]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:45.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:44 smithi173 ceph-mon[36116]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:45.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:44 smithi203 ceph-mon[36693]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:47.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:46 smithi012 ceph-mon[31695]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:47.103 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:46 smithi173 ceph-mon[36116]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:47.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:46 smithi203 ceph-mon[36693]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:48.940 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:48 smithi173 ceph-mon[36116]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:49.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:48 smithi012 ceph-mon[31695]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:49.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:48 smithi203 ceph-mon[36693]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:50.981 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:50 smithi173 ceph-mon[36116]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:50.982 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:50 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-23T22:57:50.982 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:50 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:57:51.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:50 smithi012 ceph-mon[31695]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:51.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:50 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-23T22:57:51.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:50 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:57:51.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:50 smithi203 ceph-mon[36693]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:51.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:50 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-23T22:57:51.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:50 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:57:51.961 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:51 smithi173 ceph-mon[36116]: Deploying daemon osd.10 on smithi173 2024-03-23T22:57:52.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:51 smithi012 ceph-mon[31695]: Deploying daemon osd.10 on smithi173 2024-03-23T22:57:52.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:51 smithi203 ceph-mon[36693]: Deploying daemon osd.10 on smithi173 2024-03-23T22:57:53.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:52 smithi012 ceph-mon[31695]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:53.093 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:52 smithi173 ceph-mon[36116]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:53.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:52 smithi203 ceph-mon[36693]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:54.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:54 smithi173 ceph-mon[36116]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:55.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:54 smithi012 ceph-mon[31695]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:55.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:54 smithi203 ceph-mon[36693]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:57.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:56 smithi012 ceph-mon[31695]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:57.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:56 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:57:57.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:56 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:57.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:56 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:57.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:56 smithi173 ceph-mon[36116]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:57.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:56 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:57:57.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:56 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:57.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:56 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:57.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:56 smithi203 ceph-mon[36693]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:57.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:56 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:57:57.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:56 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:57.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:56 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:57:59.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:58 smithi012 ceph-mon[31695]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:59.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:58 smithi173 ceph-mon[36116]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:59.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:58 smithi203 ceph-mon[36693]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:57:59.611 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 10 on host 'smithi173' 2024-03-23T22:57:59.611 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:59.612+0000 7fc358ff9700 1 -- 172.21.15.173:0/2491417347 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fc35c06b070 con 0x7fc3480a2890 2024-03-23T22:57:59.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:59.614+0000 7fc361b73700 1 -- 172.21.15.173:0/2491417347 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc3480a2890 msgr2=0x7fc3480a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:57:59.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:59.614+0000 7fc361b73700 1 --2- 172.21.15.173:0/2491417347 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc3480a2890 0x7fc3480a4d50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc35c102120 tx=0x7fc34c006d20 comp rx=0 tx=0).stop 2024-03-23T22:57:59.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:59.614+0000 7fc361b73700 1 -- 172.21.15.173:0/2491417347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc35c102c60 msgr2=0x7fc35c19e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:57:59.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:59.614+0000 7fc361b73700 1 --2- 172.21.15.173:0/2491417347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc35c102c60 0x7fc35c19e810 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc3540062a0 tx=0x7fc354007e40 comp rx=0 tx=0).stop 2024-03-23T22:57:59.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:59.614+0000 7fc361b73700 1 -- 172.21.15.173:0/2491417347 shutdown_connections 2024-03-23T22:57:59.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:59.614+0000 7fc361b73700 1 --2- 172.21.15.173:0/2491417347 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc3480a2890 0x7fc3480a4d50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:57:59.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:59.614+0000 7fc361b73700 1 --2- 172.21.15.173:0/2491417347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc35c102c60 0x7fc35c19e810 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:57:59.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:59.614+0000 7fc361b73700 1 --2- 172.21.15.173:0/2491417347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc35c1010c0 0x7fc35c197250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:57:59.614 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:59.614+0000 7fc361b73700 1 --2- 172.21.15.173:0/2491417347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc35c1022c0 0x7fc35c197790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:57:59.614 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:59.614+0000 7fc361b73700 1 -- 172.21.15.173:0/2491417347 >> 172.21.15.173:0/2491417347 conn(0x7fc35c0fc680 msgr2=0x7fc35c105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:57:59.614 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:59.615+0000 7fc361b73700 1 -- 172.21.15.173:0/2491417347 shutdown_connections 2024-03-23T22:57:59.614 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T22:57:59.615+0000 7fc361b73700 1 -- 172.21.15.173:0/2491417347 wait complete. 2024-03-23T22:58:00.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:59 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:00.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:59 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:00.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:59 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:00.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:59 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:58:00.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:59 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:00.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:59 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:00.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:59 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:00.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:59 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:58:00.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:59 smithi173 ceph-mon[36116]: from='osd.10 [v2:172.21.15.173:6824/3303506808,v1:172.21.15.173:6825/3303506808]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-23T22:58:00.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:57:59 smithi173 ceph-mon[36116]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-23T22:58:00.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:59 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:00.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:59 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:00.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:59 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:00.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:59 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:58:00.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:59 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:00.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:59 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:00.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:59 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:00.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:59 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:58:00.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:59 smithi203 ceph-mon[36693]: from='osd.10 [v2:172.21.15.173:6824/3303506808,v1:172.21.15.173:6825/3303506808]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-23T22:58:00.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:57:59 smithi203 ceph-mon[36693]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-23T22:58:00.302 DEBUG:teuthology.orchestra.run.smithi173:osd.10> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.10.service 2024-03-23T22:58:00.304 INFO:tasks.cephadm:Deploying osd.11 on smithi203 with /dev/vg_nvme/lv_1... 2024-03-23T22:58:00.304 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-23T22:58:00.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:59 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:00.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:59 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:00.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:59 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:00.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:59 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:58:00.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:59 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:00.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:59 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:00.306 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:59 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:00.306 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:59 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:58:00.306 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:59 smithi012 ceph-mon[31695]: from='osd.10 [v2:172.21.15.173:6824/3303506808,v1:172.21.15.173:6825/3303506808]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-23T22:58:00.306 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:57:59 smithi012 ceph-mon[31695]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-23T22:58:00.561 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.c/config 2024-03-23T22:58:01.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:00 smithi012 ceph-mon[31695]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:58:01.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:00 smithi012 ceph-mon[31695]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-23T22:58:01.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:00 smithi012 ceph-mon[31695]: from='osd.10 [v2:172.21.15.173:6824/3303506808,v1:172.21.15.173:6825/3303506808]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-23T22:58:01.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:00 smithi012 ceph-mon[31695]: osdmap e71: 11 total, 10 up, 11 in 2024-03-23T22:58:01.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:00 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-23T22:58:01.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:00 smithi012 ceph-mon[31695]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-23T22:58:01.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:00 smithi173 ceph-mon[36116]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:58:01.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:00 smithi173 ceph-mon[36116]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-23T22:58:01.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:00 smithi173 ceph-mon[36116]: from='osd.10 [v2:172.21.15.173:6824/3303506808,v1:172.21.15.173:6825/3303506808]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-23T22:58:01.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:00 smithi173 ceph-mon[36116]: osdmap e71: 11 total, 10 up, 11 in 2024-03-23T22:58:01.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:00 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-23T22:58:01.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:00 smithi173 ceph-mon[36116]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-23T22:58:01.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:00 smithi203 ceph-mon[36693]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:58:01.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:00 smithi203 ceph-mon[36693]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-23T22:58:01.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:00 smithi203 ceph-mon[36693]: from='osd.10 [v2:172.21.15.173:6824/3303506808,v1:172.21.15.173:6825/3303506808]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-23T22:58:01.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:00 smithi203 ceph-mon[36693]: osdmap e71: 11 total, 10 up, 11 in 2024-03-23T22:58:01.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:00 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-23T22:58:01.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:00 smithi203 ceph-mon[36693]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-23T22:58:01.718 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 22:58:01 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T22:58:01.587+0000 7f740fff2700 -1 osd.10 0 waiting for initial osdmap 2024-03-23T22:58:01.718 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 22:58:01 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T22:58:01.603+0000 7f740bde5700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T22:58:02.684 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-23T22:58:02.684 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-23T22:58:02.684 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-03-23T22:58:02.684 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-03-23T22:58:02.684 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0138666 s, 756 MB/s 2024-03-23T22:58:02.684 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-03-23T22:58:02.685 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:02 smithi203 ceph-mon[36693]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:58:02.685 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:02 smithi203 ceph-mon[36693]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-23T22:58:02.685 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:02 smithi203 ceph-mon[36693]: osdmap e72: 11 total, 10 up, 11 in 2024-03-23T22:58:02.685 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:02 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-23T22:58:02.685 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:02 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-23T22:58:02.685 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:02 smithi203 ceph-mon[36693]: osd.10 [v2:172.21.15.173:6824/3303506808,v1:172.21.15.173:6825/3303506808] boot 2024-03-23T22:58:02.685 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:02 smithi203 ceph-mon[36693]: osdmap e73: 11 total, 11 up, 11 in 2024-03-23T22:58:02.685 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:02 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-23T22:58:02.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:02 smithi173 ceph-mon[36116]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:58:02.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:02 smithi173 ceph-mon[36116]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-23T22:58:02.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:02 smithi173 ceph-mon[36116]: osdmap e72: 11 total, 10 up, 11 in 2024-03-23T22:58:02.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:02 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-23T22:58:02.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:02 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-23T22:58:02.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:02 smithi173 ceph-mon[36116]: osd.10 [v2:172.21.15.173:6824/3303506808,v1:172.21.15.173:6825/3303506808] boot 2024-03-23T22:58:02.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:02 smithi173 ceph-mon[36116]: osdmap e73: 11 total, 11 up, 11 in 2024-03-23T22:58:02.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:02 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-23T22:58:03.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:02 smithi012 ceph-mon[31695]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-23T22:58:03.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:02 smithi012 ceph-mon[31695]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-23T22:58:03.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:02 smithi012 ceph-mon[31695]: osdmap e72: 11 total, 10 up, 11 in 2024-03-23T22:58:03.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:02 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-23T22:58:03.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:02 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-23T22:58:03.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:02 smithi012 ceph-mon[31695]: osd.10 [v2:172.21.15.173:6824/3303506808,v1:172.21.15.173:6825/3303506808] boot 2024-03-23T22:58:03.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:02 smithi012 ceph-mon[31695]: osdmap e73: 11 total, 11 up, 11 in 2024-03-23T22:58:03.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:02 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-23T22:58:03.342 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi203:vg_nvme/lv_1 2024-03-23T22:58:03.580 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.c/config 2024-03-23T22:58:03.845 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:03 smithi203 ceph-mon[36693]: purged_snaps scrub starts 2024-03-23T22:58:03.845 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:03 smithi203 ceph-mon[36693]: purged_snaps scrub ok 2024-03-23T22:58:03.846 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:03 smithi203 ceph-mon[36693]: osdmap e74: 11 total, 11 up, 11 in 2024-03-23T22:58:03.846 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:03.846 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:03.846 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:58:03.846 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:58:03.846 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:58:03.846 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:58:03.846 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:03.846 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:03.846 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:58:03.846 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:03.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:03 smithi173 ceph-mon[36116]: purged_snaps scrub starts 2024-03-23T22:58:03.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:03 smithi173 ceph-mon[36116]: purged_snaps scrub ok 2024-03-23T22:58:03.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:03 smithi173 ceph-mon[36116]: osdmap e74: 11 total, 11 up, 11 in 2024-03-23T22:58:03.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:03.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:03.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:58:03.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:58:03.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:58:03.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:58:03.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:03.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:03.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:58:03.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:04.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:03 smithi012 ceph-mon[31695]: purged_snaps scrub starts 2024-03-23T22:58:04.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:03 smithi012 ceph-mon[31695]: purged_snaps scrub ok 2024-03-23T22:58:04.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:03 smithi012 ceph-mon[31695]: osdmap e74: 11 total, 11 up, 11 in 2024-03-23T22:58:04.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:04.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:04.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:58:04.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:58:04.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:58:04.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:58:04.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:04.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:04.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:58:04.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:05.274 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.272+0000 7f9429445700 1 -- 172.21.15.203:0/4166802744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9424102c50 msgr2=0x7f9424109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:05.274 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.272+0000 7f9429445700 1 --2- 172.21.15.203:0/4166802744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9424102c50 0x7f9424109500 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f941c00b0a0 tx=0x7f941c00b3b0 comp rx=0 tx=0).stop 2024-03-23T22:58:05.274 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.272+0000 7f9429445700 1 -- 172.21.15.203:0/4166802744 shutdown_connections 2024-03-23T22:58:05.274 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.272+0000 7f9429445700 1 --2- 172.21.15.203:0/4166802744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9424102c50 0x7f9424109500 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:05.274 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.272+0000 7f9429445700 1 --2- 172.21.15.203:0/4166802744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94241010b0 0x7f94241014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:05.274 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.272+0000 7f9429445700 1 --2- 172.21.15.203:0/4166802744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94241022b0 0x7f9424102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:05.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.272+0000 7f9429445700 1 -- 172.21.15.203:0/4166802744 >> 172.21.15.203:0/4166802744 conn(0x7f94240fc650 msgr2=0x7f94240fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:05.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.273+0000 7f9429445700 1 -- 172.21.15.203:0/4166802744 shutdown_connections 2024-03-23T22:58:05.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.273+0000 7f9429445700 1 -- 172.21.15.203:0/4166802744 wait complete. 2024-03-23T22:58:05.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.273+0000 7f9429445700 1 Processor -- start 2024-03-23T22:58:05.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.273+0000 7f9429445700 1 -- start start 2024-03-23T22:58:05.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.273+0000 7f9429445700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94241010b0 0x7f942406b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:05.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.273+0000 7f9429445700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94241022b0 0x7f942419f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:05.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.273+0000 7f9429445700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9424102c50 0x7f94241a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:05.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.273+0000 7f9429445700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942410cd10 con 0x7f94241010b0 2024-03-23T22:58:05.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.273+0000 7f9429445700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942410cb90 con 0x7f9424102c50 2024-03-23T22:58:05.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.273+0000 7f9429445700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942410ce90 con 0x7f94241022b0 2024-03-23T22:58:05.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.274+0000 7f94227fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94241022b0 0x7f942419f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:05.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.274+0000 7f94227fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94241022b0 0x7f942419f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:51238/0 (socket says 172.21.15.203:51238) 2024-03-23T22:58:05.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.274+0000 7f94227fc700 1 -- 172.21.15.203:0/1747157047 learned_addr learned my addr 172.21.15.203:0/1747157047 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-23T22:58:05.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.274+0000 7f94237fe700 1 --2- 172.21.15.203:0/1747157047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9424102c50 0x7f94241a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:05.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.274+0000 7f94227fc700 1 -- 172.21.15.203:0/1747157047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9424102c50 msgr2=0x7f94241a19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:05.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.274+0000 7f94227fc700 1 --2- 172.21.15.203:0/1747157047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9424102c50 0x7f94241a19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:05.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.274+0000 7f94227fc700 1 -- 172.21.15.203:0/1747157047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94241010b0 msgr2=0x7f942406b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:05.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.274+0000 7f94227fc700 1 --2- 172.21.15.203:0/1747157047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94241010b0 0x7f942406b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:05.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.274+0000 7f94227fc700 1 -- 172.21.15.203:0/1747157047 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94241a21c0 con 0x7f94241022b0 2024-03-23T22:58:05.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.274+0000 7f94237fe700 1 --2- 172.21.15.203:0/1747157047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9424102c50 0x7f94241a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T22:58:05.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.274+0000 7f94227fc700 1 --2- 172.21.15.203:0/1747157047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94241022b0 0x7f942419f490 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f940c00c9b0 tx=0x7f940c00cd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:05.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.274+0000 7f941bfff700 1 -- 172.21.15.203:0/1747157047 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940c007b00 con 0x7f94241022b0 2024-03-23T22:58:05.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.274+0000 7f9429445700 1 -- 172.21.15.203:0/1747157047 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94241a24b0 con 0x7f94241022b0 2024-03-23T22:58:05.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.274+0000 7f941bfff700 1 -- 172.21.15.203:0/1747157047 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f940c007ca0 con 0x7f94241022b0 2024-03-23T22:58:05.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.274+0000 7f9429445700 1 -- 172.21.15.203:0/1747157047 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94241a2ac0 con 0x7f94241022b0 2024-03-23T22:58:05.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.274+0000 7f941bfff700 1 -- 172.21.15.203:0/1747157047 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940c018780 con 0x7f94241022b0 2024-03-23T22:58:05.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.275+0000 7f9429445700 1 -- 172.21.15.203:0/1747157047 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9424068e50 con 0x7f94241022b0 2024-03-23T22:58:05.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.276+0000 7f941bfff700 1 -- 172.21.15.203:0/1747157047 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f940c00f450 con 0x7f94241022b0 2024-03-23T22:58:05.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.277+0000 7f941bfff700 1 --2- 172.21.15.203:0/1747157047 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f94100a2890 0x7f94100a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:05.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.277+0000 7f941bfff700 1 -- 172.21.15.203:0/1747157047 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6694+0+0 (secure 0 0 0) 0x7f940c0ce4a0 con 0x7f94241022b0 2024-03-23T22:58:05.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.277+0000 7f9422ffd700 1 --2- 172.21.15.203:0/1747157047 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f94100a2890 0x7f94100a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:05.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.278+0000 7f9422ffd700 1 --2- 172.21.15.203:0/1747157047 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f94100a2890 0x7f94100a4d50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9424102110 tx=0x7f9414006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:05.281 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.281+0000 7f941bfff700 1 -- 172.21.15.203:0/1747157047 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f940c014030 con 0x7f94241022b0 2024-03-23T22:58:05.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:05 smithi203 ceph-mon[36693]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:05.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:05 smithi203 ceph-mon[36693]: Detected new or changed devices on smithi173 2024-03-23T22:58:05.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:05 smithi203 ceph-mon[36693]: Adjusting osd_memory_target on smithi173 to 4270M 2024-03-23T22:58:05.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:05 smithi203 ceph-mon[36693]: osdmap e75: 11 total, 11 up, 11 in 2024-03-23T22:58:05.489 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:05.489+0000 7f9429445700 1 -- 172.21.15.203:0/1747157047 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f94240625d0 con 0x7f94100a2890 2024-03-23T22:58:05.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:05 smithi012 ceph-mon[31695]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:05.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:05 smithi012 ceph-mon[31695]: Detected new or changed devices on smithi173 2024-03-23T22:58:05.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:05 smithi012 ceph-mon[31695]: Adjusting osd_memory_target on smithi173 to 4270M 2024-03-23T22:58:05.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:05 smithi012 ceph-mon[31695]: osdmap e75: 11 total, 11 up, 11 in 2024-03-23T22:58:05.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:05 smithi173 ceph-mon[36116]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:05.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:05 smithi173 ceph-mon[36116]: Detected new or changed devices on smithi173 2024-03-23T22:58:05.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:05 smithi173 ceph-mon[36116]: Adjusting osd_memory_target on smithi173 to 4270M 2024-03-23T22:58:05.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:05 smithi173 ceph-mon[36116]: osdmap e75: 11 total, 11 up, 11 in 2024-03-23T22:58:06.388 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:06 smithi203 ceph-mon[36693]: osdmap e76: 11 total, 11 up, 11 in 2024-03-23T22:58:06.388 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:06 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:58:06.389 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:06 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:58:06.389 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:06 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:06.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:06 smithi012 ceph-mon[31695]: osdmap e76: 11 total, 11 up, 11 in 2024-03-23T22:58:06.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:06 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:58:06.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:06 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:58:06.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:06 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:06.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:06 smithi173 ceph-mon[36116]: osdmap e76: 11 total, 11 up, 11 in 2024-03-23T22:58:06.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:06 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T22:58:06.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:06 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T22:58:06.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:06 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:07.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:07 smithi203 ceph-mon[36693]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:07.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:07 smithi203 ceph-mon[36693]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:58:07.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:07 smithi203 ceph-mon[36693]: osdmap e77: 11 total, 11 up, 11 in 2024-03-23T22:58:07.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:07 smithi012 ceph-mon[31695]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:07.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:07 smithi012 ceph-mon[31695]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:58:07.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:07 smithi012 ceph-mon[31695]: osdmap e77: 11 total, 11 up, 11 in 2024-03-23T22:58:07.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:07 smithi173 ceph-mon[36116]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:07.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:07 smithi173 ceph-mon[36116]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:58:07.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:07 smithi173 ceph-mon[36116]: osdmap e77: 11 total, 11 up, 11 in 2024-03-23T22:58:08.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:08 smithi012 ceph-mon[31695]: pgmap v269: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-03-23T22:58:08.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:08 smithi173 ceph-mon[36116]: pgmap v269: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-03-23T22:58:08.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:08 smithi203 ceph-mon[36693]: pgmap v269: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-03-23T22:58:09.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:09 smithi012 ceph-mon[31695]: from='client.? 172.21.15.203:0/1531083506' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ef1d595-a977-4ba7-9c6b-d2b3295f20b0"}]: dispatch 2024-03-23T22:58:09.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:09 smithi012 ceph-mon[31695]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ef1d595-a977-4ba7-9c6b-d2b3295f20b0"}]: dispatch 2024-03-23T22:58:09.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:09 smithi012 ceph-mon[31695]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ef1d595-a977-4ba7-9c6b-d2b3295f20b0"}]': finished 2024-03-23T22:58:09.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:09 smithi012 ceph-mon[31695]: osdmap e78: 12 total, 11 up, 12 in 2024-03-23T22:58:09.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:09 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-23T22:58:09.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:09 smithi012 ceph-mon[31695]: from='client.? 172.21.15.203:0/2447544094' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:58:09.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:09 smithi173 ceph-mon[36116]: from='client.? 172.21.15.203:0/1531083506' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ef1d595-a977-4ba7-9c6b-d2b3295f20b0"}]: dispatch 2024-03-23T22:58:09.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:09 smithi173 ceph-mon[36116]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ef1d595-a977-4ba7-9c6b-d2b3295f20b0"}]: dispatch 2024-03-23T22:58:09.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:09 smithi173 ceph-mon[36116]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ef1d595-a977-4ba7-9c6b-d2b3295f20b0"}]': finished 2024-03-23T22:58:09.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:09 smithi173 ceph-mon[36116]: osdmap e78: 12 total, 11 up, 12 in 2024-03-23T22:58:09.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:09 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-23T22:58:09.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:09 smithi173 ceph-mon[36116]: from='client.? 172.21.15.203:0/2447544094' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:58:09.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:09 smithi203 ceph-mon[36693]: from='client.? 172.21.15.203:0/1531083506' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ef1d595-a977-4ba7-9c6b-d2b3295f20b0"}]: dispatch 2024-03-23T22:58:09.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:09 smithi203 ceph-mon[36693]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ef1d595-a977-4ba7-9c6b-d2b3295f20b0"}]: dispatch 2024-03-23T22:58:09.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:09 smithi203 ceph-mon[36693]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ef1d595-a977-4ba7-9c6b-d2b3295f20b0"}]': finished 2024-03-23T22:58:09.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:09 smithi203 ceph-mon[36693]: osdmap e78: 12 total, 11 up, 12 in 2024-03-23T22:58:09.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:09 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-23T22:58:09.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:09 smithi203 ceph-mon[36693]: from='client.? 172.21.15.203:0/2447544094' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T22:58:10.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:10 smithi012 ceph-mon[31695]: pgmap v271: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-23T22:58:10.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:10 smithi173 ceph-mon[36116]: pgmap v271: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-23T22:58:10.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:10 smithi203 ceph-mon[36693]: pgmap v271: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-23T22:58:12.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:12 smithi012 ceph-mon[31695]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-03-23T22:58:12.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:12 smithi173 ceph-mon[36116]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-03-23T22:58:12.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:12 smithi203 ceph-mon[36693]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-03-23T22:58:14.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:14 smithi012 ceph-mon[31695]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-23T22:58:14.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:14 smithi173 ceph-mon[36116]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-23T22:58:14.629 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:14 smithi203 ceph-mon[36693]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-23T22:58:16.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:16 smithi012 ceph-mon[31695]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:16.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:16 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-23T22:58:16.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:16 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:16.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:16 smithi012 ceph-mon[31695]: Deploying daemon osd.11 on smithi203 2024-03-23T22:58:16.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:16 smithi173 ceph-mon[36116]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:16.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:16 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-23T22:58:16.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:16 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:16.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:16 smithi173 ceph-mon[36116]: Deploying daemon osd.11 on smithi203 2024-03-23T22:58:16.623 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:16 smithi203 ceph-mon[36693]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:16.623 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:16 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-23T22:58:16.623 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:16 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:16.623 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:16 smithi203 ceph-mon[36693]: Deploying daemon osd.11 on smithi203 2024-03-23T22:58:19.293 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:19 smithi203 ceph-mon[36693]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:19.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:19 smithi012 ceph-mon[31695]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:19.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:19 smithi173 ceph-mon[36116]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:21.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:21 smithi203 ceph-mon[36693]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:21.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:21 smithi012 ceph-mon[31695]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:21.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:21 smithi173 ceph-mon[36116]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:23.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:22 smithi012 ceph-mon[31695]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:23.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:22 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:23.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:22 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:58:23.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:22 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:23.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:22 smithi173 ceph-mon[36116]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:23.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:22 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:23.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:22 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:58:23.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:22 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:23.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:22 smithi203 ceph-mon[36693]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:23.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:22 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:23.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:22 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:58:23.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:22 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:25.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:24 smithi012 ceph-mon[31695]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:25.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:24 smithi173 ceph-mon[36116]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:25.129 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:24 smithi203 ceph-mon[36693]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:25.531 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 11 on host 'smithi203' 2024-03-23T22:58:25.531 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:25.530+0000 7f941bfff700 1 -- 172.21.15.203:0/1747157047 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f94240625d0 con 0x7f94100a2890 2024-03-23T22:58:25.533 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:25.534+0000 7f9429445700 1 -- 172.21.15.203:0/1747157047 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f94100a2890 msgr2=0x7f94100a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:25.533 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:25.534+0000 7f9429445700 1 --2- 172.21.15.203:0/1747157047 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f94100a2890 0x7f94100a4d50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9424102110 tx=0x7f9414006d20 comp rx=0 tx=0).stop 2024-03-23T22:58:25.533 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:25.534+0000 7f9429445700 1 -- 172.21.15.203:0/1747157047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94241022b0 msgr2=0x7f942419f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:25.534 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:25.534+0000 7f9429445700 1 --2- 172.21.15.203:0/1747157047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94241022b0 0x7f942419f490 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f940c00c9b0 tx=0x7f940c00cd70 comp rx=0 tx=0).stop 2024-03-23T22:58:25.534 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:25.534+0000 7f9429445700 1 -- 172.21.15.203:0/1747157047 shutdown_connections 2024-03-23T22:58:25.534 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:25.534+0000 7f9429445700 1 --2- 172.21.15.203:0/1747157047 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f94100a2890 0x7f94100a4d50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:25.534 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:25.534+0000 7f9429445700 1 --2- 172.21.15.203:0/1747157047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9424102c50 0x7f94241a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:25.534 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:25.534+0000 7f9429445700 1 --2- 172.21.15.203:0/1747157047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94241010b0 0x7f942406b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:25.534 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:25.534+0000 7f9429445700 1 --2- 172.21.15.203:0/1747157047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94241022b0 0x7f942419f490 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:25.534 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:25.534+0000 7f9429445700 1 -- 172.21.15.203:0/1747157047 >> 172.21.15.203:0/1747157047 conn(0x7f94240fc650 msgr2=0x7f9424105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:25.534 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:25.534+0000 7f9429445700 1 -- 172.21.15.203:0/1747157047 shutdown_connections 2024-03-23T22:58:25.534 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:25.534+0000 7f9429445700 1 -- 172.21.15.203:0/1747157047 wait complete. 2024-03-23T22:58:26.263 DEBUG:teuthology.orchestra.run.smithi203:osd.11> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.11.service 2024-03-23T22:58:26.265 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-23T22:58:26.265 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-23T22:58:26.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:26 smithi203 ceph-mon[36693]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:26.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:26 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:26.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:26 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:26.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:26 smithi203 ceph-mon[36693]: from='osd.11 [v2:172.21.15.203:6824/2323604506,v1:172.21.15.203:6825/2323604506]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-23T22:58:26.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:26 smithi203 ceph-mon[36693]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-23T22:58:26.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:26 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:26.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:26 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:26.748 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:58:26.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:26 smithi012 ceph-mon[31695]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:26.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:26 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:26.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:26 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:26.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:26 smithi012 ceph-mon[31695]: from='osd.11 [v2:172.21.15.203:6824/2323604506,v1:172.21.15.203:6825/2323604506]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-23T22:58:26.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:26 smithi012 ceph-mon[31695]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-23T22:58:26.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:26 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:26.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:26 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:26.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:26 smithi173 ceph-mon[36116]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:26.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:26 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:26.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:26 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:26.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:26 smithi173 ceph-mon[36116]: from='osd.11 [v2:172.21.15.203:6824/2323604506,v1:172.21.15.203:6825/2323604506]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-23T22:58:26.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:26 smithi173 ceph-mon[36116]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-23T22:58:26.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:26 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:26.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:26 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:27.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:27 smithi203 ceph-mon[36693]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-23T22:58:27.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:27 smithi203 ceph-mon[36693]: from='osd.11 [v2:172.21.15.203:6824/2323604506,v1:172.21.15.203:6825/2323604506]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-23T22:58:27.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:27 smithi203 ceph-mon[36693]: osdmap e79: 12 total, 11 up, 12 in 2024-03-23T22:58:27.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:27 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-23T22:58:27.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:27 smithi203 ceph-mon[36693]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-23T22:58:27.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:27 smithi012 ceph-mon[31695]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-23T22:58:27.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:27 smithi012 ceph-mon[31695]: from='osd.11 [v2:172.21.15.203:6824/2323604506,v1:172.21.15.203:6825/2323604506]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-23T22:58:27.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:27 smithi012 ceph-mon[31695]: osdmap e79: 12 total, 11 up, 12 in 2024-03-23T22:58:27.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:27 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-23T22:58:27.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:27 smithi012 ceph-mon[31695]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-23T22:58:27.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:27 smithi173 ceph-mon[36116]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-23T22:58:27.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:27 smithi173 ceph-mon[36116]: from='osd.11 [v2:172.21.15.203:6824/2323604506,v1:172.21.15.203:6825/2323604506]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-23T22:58:27.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:27 smithi173 ceph-mon[36116]: osdmap e79: 12 total, 11 up, 12 in 2024-03-23T22:58:27.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:27 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-23T22:58:27.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:27 smithi173 ceph-mon[36116]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-23T22:58:28.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 22:58:27 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T22:58:27.899+0000 7f331f835700 -1 osd.11 0 waiting for initial osdmap 2024-03-23T22:58:28.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 22:58:27 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T22:58:27.912+0000 7f3316e1f700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T22:58:28.705 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:28 smithi012 ceph-mon[31695]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:28.706 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:28 smithi012 ceph-mon[31695]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-23T22:58:28.706 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:28 smithi012 ceph-mon[31695]: osdmap e80: 12 total, 11 up, 12 in 2024-03-23T22:58:28.706 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:28 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-23T22:58:28.706 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:28 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-23T22:58:28.706 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:28 smithi012 ceph-mon[31695]: from='osd.11 ' entity='osd.11' 2024-03-23T22:58:28.706 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:28 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-23T22:58:28.769 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:28 smithi173 ceph-mon[36116]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:28.769 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:28 smithi173 ceph-mon[36116]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-23T22:58:28.769 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:28 smithi173 ceph-mon[36116]: osdmap e80: 12 total, 11 up, 12 in 2024-03-23T22:58:28.770 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:28 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-23T22:58:28.770 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:28 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-23T22:58:28.770 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:28 smithi173 ceph-mon[36116]: from='osd.11 ' entity='osd.11' 2024-03-23T22:58:28.770 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:28 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-23T22:58:28.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:28 smithi203 ceph-mon[36693]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:28.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:28 smithi203 ceph-mon[36693]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-23T22:58:28.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:28 smithi203 ceph-mon[36693]: osdmap e80: 12 total, 11 up, 12 in 2024-03-23T22:58:28.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:28 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-23T22:58:28.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:28 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-23T22:58:28.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:28 smithi203 ceph-mon[36693]: from='osd.11 ' entity='osd.11' 2024-03-23T22:58:28.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:28 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-23T22:58:29.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.239+0000 7f76f9c00700 1 -- 172.21.15.12:0/2973515022 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f41033c0 msgr2=0x7f76f4103820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:29.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.239+0000 7f76f9c00700 1 --2- 172.21.15.12:0/2973515022 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f41033c0 0x7f76f4103820 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f76e00099c0 tx=0x7f76e0009cd0 comp rx=0 tx=0).stop 2024-03-23T22:58:29.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.240+0000 7f76f9c00700 1 -- 172.21.15.12:0/2973515022 shutdown_connections 2024-03-23T22:58:29.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.240+0000 7f76f9c00700 1 --2- 172.21.15.12:0/2973515022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76f41021c0 0x7f76f41025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:29.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.240+0000 7f76f9c00700 1 --2- 172.21.15.12:0/2973515022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76f4103d60 0x7f76f410a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:29.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.240+0000 7f76f9c00700 1 --2- 172.21.15.12:0/2973515022 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f41033c0 0x7f76f4103820 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:29.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.240+0000 7f76f9c00700 1 -- 172.21.15.12:0/2973515022 >> 172.21.15.12:0/2973515022 conn(0x7f76f40fd740 msgr2=0x7f76f40ffba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:29.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.240+0000 7f76f9c00700 1 -- 172.21.15.12:0/2973515022 shutdown_connections 2024-03-23T22:58:29.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.240+0000 7f76f9c00700 1 -- 172.21.15.12:0/2973515022 wait complete. 2024-03-23T22:58:29.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.241+0000 7f76f9c00700 1 Processor -- start 2024-03-23T22:58:29.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.241+0000 7f76f9c00700 1 -- start start 2024-03-23T22:58:29.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.241+0000 7f76f9c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76f41021c0 0x7f76f419a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:29.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.241+0000 7f76f9c00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76f41033c0 0x7f76f419ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:29.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.241+0000 7f76f9c00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f4103d60 0x7f76f41a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:29.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.242+0000 7f76f9c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f410ddd0 con 0x7f76f41021c0 2024-03-23T22:58:29.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.242+0000 7f76f9c00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f410dc50 con 0x7f76f4103d60 2024-03-23T22:58:29.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.242+0000 7f76f9c00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f410df50 con 0x7f76f41033c0 2024-03-23T22:58:29.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.242+0000 7f76f2000700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76f41033c0 0x7f76f419ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:29.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.242+0000 7f76f2000700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76f41033c0 0x7f76f419ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44954/0 (socket says 172.21.15.12:44954) 2024-03-23T22:58:29.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.242+0000 7f76f2000700 1 -- 172.21.15.12:0/3626427631 learned_addr learned my addr 172.21.15.12:0/3626427631 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:58:29.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.242+0000 7f76f2a00700 1 --2- 172.21.15.12:0/3626427631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76f41021c0 0x7f76f419a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:29.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.242+0000 7f76f3400700 1 --2- 172.21.15.12:0/3626427631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f4103d60 0x7f76f41a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:29.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.242+0000 7f76f2000700 1 -- 172.21.15.12:0/3626427631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f4103d60 msgr2=0x7f76f41a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:29.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.242+0000 7f76f2000700 1 --2- 172.21.15.12:0/3626427631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f4103d60 0x7f76f41a1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:29.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.242+0000 7f76f2000700 1 -- 172.21.15.12:0/3626427631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76f41021c0 msgr2=0x7f76f419a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:29.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.242+0000 7f76f2000700 1 --2- 172.21.15.12:0/3626427631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76f41021c0 0x7f76f419a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:29.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.242+0000 7f76f2000700 1 -- 172.21.15.12:0/3626427631 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f41a2330 con 0x7f76f41033c0 2024-03-23T22:58:29.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.243+0000 7f76f2000700 1 --2- 172.21.15.12:0/3626427631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76f41033c0 0x7f76f419ac60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f76e0005910 tx=0x7f76e000fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:29.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.243+0000 7f76eb400700 1 -- 172.21.15.12:0/3626427631 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e0027070 con 0x7f76f41033c0 2024-03-23T22:58:29.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.243+0000 7f76f9c00700 1 -- 172.21.15.12:0/3626427631 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f41a25c0 con 0x7f76f41033c0 2024-03-23T22:58:29.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.243+0000 7f76eb400700 1 -- 172.21.15.12:0/3626427631 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76e000be50 con 0x7f76f41033c0 2024-03-23T22:58:29.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.243+0000 7f76eb400700 1 -- 172.21.15.12:0/3626427631 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e00173f0 con 0x7f76f41033c0 2024-03-23T22:58:29.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.243+0000 7f76f9c00700 1 -- 172.21.15.12:0/3626427631 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f41a2a70 con 0x7f76f41033c0 2024-03-23T22:58:29.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.244+0000 7f76f9c00700 1 -- 172.21.15.12:0/3626427631 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f4068e50 con 0x7f76f41033c0 2024-03-23T22:58:29.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.246+0000 7f76eb400700 1 -- 172.21.15.12:0/3626427631 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f76e002a070 con 0x7f76f41033c0 2024-03-23T22:58:29.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.248+0000 7f76eb400700 1 --2- 172.21.15.12:0/3626427631 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f76d80a2890 0x7f76d80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:29.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.248+0000 7f76eb400700 1 -- 172.21.15.12:0/3626427631 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f76e00cf930 con 0x7f76f41033c0 2024-03-23T22:58:29.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.248+0000 7f76f2a00700 1 --2- 172.21.15.12:0/3626427631 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f76d80a2890 0x7f76d80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:29.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.249+0000 7f76f2a00700 1 --2- 172.21.15.12:0/3626427631 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f76d80a2890 0x7f76d80a4d50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f76f4103220 tx=0x7f76e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:29.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.254+0000 7f76eb400700 1 -- 172.21.15.12:0/3626427631 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f76e0031070 con 0x7f76f41033c0 2024-03-23T22:58:29.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.482+0000 7f76f9c00700 1 -- 172.21.15.12:0/3626427631 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f76f41a2e40 con 0x7f76f41033c0 2024-03-23T22:58:29.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.483+0000 7f76eb400700 1 -- 172.21.15.12:0/3626427631 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) v1 ==== 74+0+133 (secure 0 0 0) 0x7f76e0013070 con 0x7f76f41033c0 2024-03-23T22:58:29.482 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:58:29.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.486+0000 7f76f9c00700 1 -- 172.21.15.12:0/3626427631 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f76d80a2890 msgr2=0x7f76d80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:29.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.486+0000 7f76f9c00700 1 --2- 172.21.15.12:0/3626427631 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f76d80a2890 0x7f76d80a4d50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f76f4103220 tx=0x7f76e4009260 comp rx=0 tx=0).stop 2024-03-23T22:58:29.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.486+0000 7f76f9c00700 1 -- 172.21.15.12:0/3626427631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76f41033c0 msgr2=0x7f76f419ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:29.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.486+0000 7f76f9c00700 1 --2- 172.21.15.12:0/3626427631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76f41033c0 0x7f76f419ac60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f76e0005910 tx=0x7f76e000fa00 comp rx=0 tx=0).stop 2024-03-23T22:58:29.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.486+0000 7f76f9c00700 1 -- 172.21.15.12:0/3626427631 shutdown_connections 2024-03-23T22:58:29.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.486+0000 7f76f9c00700 1 --2- 172.21.15.12:0/3626427631 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f76d80a2890 0x7f76d80a4d50 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:29.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.486+0000 7f76f9c00700 1 --2- 172.21.15.12:0/3626427631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f4103d60 0x7f76f41a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:29.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.486+0000 7f76f9c00700 1 --2- 172.21.15.12:0/3626427631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76f41021c0 0x7f76f419a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:29.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.486+0000 7f76f9c00700 1 --2- 172.21.15.12:0/3626427631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76f41033c0 0x7f76f419ac60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:29.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.486+0000 7f76f9c00700 1 -- 172.21.15.12:0/3626427631 >> 172.21.15.12:0/3626427631 conn(0x7f76f40fd740 msgr2=0x7f76f40743c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:29.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.486+0000 7f76f9c00700 1 -- 172.21.15.12:0/3626427631 shutdown_connections 2024-03-23T22:58:29.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:29.486+0000 7f76f9c00700 1 -- 172.21.15.12:0/3626427631 wait complete. 2024-03-23T22:58:29.782 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:29 smithi173 ceph-mon[36116]: purged_snaps scrub starts 2024-03-23T22:58:29.782 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:29 smithi173 ceph-mon[36116]: purged_snaps scrub ok 2024-03-23T22:58:29.782 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:29 smithi173 ceph-mon[36116]: osd.11 [v2:172.21.15.203:6824/2323604506,v1:172.21.15.203:6825/2323604506] boot 2024-03-23T22:58:29.783 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:29 smithi173 ceph-mon[36116]: osdmap e81: 12 total, 12 up, 12 in 2024-03-23T22:58:29.783 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:29 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-23T22:58:29.783 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:29 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3626427631' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-23T22:58:29.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:29 smithi012 ceph-mon[31695]: purged_snaps scrub starts 2024-03-23T22:58:29.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:29 smithi012 ceph-mon[31695]: purged_snaps scrub ok 2024-03-23T22:58:29.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:29 smithi012 ceph-mon[31695]: osd.11 [v2:172.21.15.203:6824/2323604506,v1:172.21.15.203:6825/2323604506] boot 2024-03-23T22:58:29.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:29 smithi012 ceph-mon[31695]: osdmap e81: 12 total, 12 up, 12 in 2024-03-23T22:58:29.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:29 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-23T22:58:29.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:29 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3626427631' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-23T22:58:29.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:29 smithi203 ceph-mon[36693]: purged_snaps scrub starts 2024-03-23T22:58:29.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:29 smithi203 ceph-mon[36693]: purged_snaps scrub ok 2024-03-23T22:58:29.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:29 smithi203 ceph-mon[36693]: osd.11 [v2:172.21.15.203:6824/2323604506,v1:172.21.15.203:6825/2323604506] boot 2024-03-23T22:58:29.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:29 smithi203 ceph-mon[36693]: osdmap e81: 12 total, 12 up, 12 in 2024-03-23T22:58:29.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:29 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-23T22:58:29.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:29 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3626427631' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-23T22:58:30.283 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1711234708,"num_in_osds":12,"osd_in_since":1711234688,"num_remapped_pgs":0} 2024-03-23T22:58:30.283 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-23T22:58:30.725 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:58:31.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:31 smithi012 ceph-mon[31695]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:31.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:31 smithi012 ceph-mon[31695]: Detected new or changed devices on smithi203 2024-03-23T22:58:31.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:31 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:31.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:31 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:31.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:31 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:58:31.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:31 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:58:31.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:31 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:58:31.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:31 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:58:31.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:31 smithi012 ceph-mon[31695]: Adjusting osd_memory_target on smithi203 to 4270M 2024-03-23T22:58:31.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:31 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:31.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:31 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:31.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:31 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:58:31.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:31 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:31.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:31 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:58:31.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:31 smithi012 ceph-mon[31695]: osdmap e82: 12 total, 12 up, 12 in 2024-03-23T22:58:31.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:31 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:31.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:31 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:58:31.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:31 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:31.276 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:31 smithi203 ceph-mon[36693]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:31.276 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:31 smithi203 ceph-mon[36693]: Detected new or changed devices on smithi203 2024-03-23T22:58:31.277 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:31 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:31.277 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:31 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:31.277 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:31 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:58:31.277 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:31 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:58:31.277 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:31 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:58:31.277 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:31 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:58:31.277 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:31 smithi203 ceph-mon[36693]: Adjusting osd_memory_target on smithi203 to 4270M 2024-03-23T22:58:31.277 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:31 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:31.277 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:31 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:31.277 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:31 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:58:31.277 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:31 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:31.277 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:31 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:58:31.277 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:31 smithi203 ceph-mon[36693]: osdmap e82: 12 total, 12 up, 12 in 2024-03-23T22:58:31.277 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:31 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:31.278 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:31 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:58:31.278 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:31 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:31.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:31 smithi173 ceph-mon[36116]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-23T22:58:31.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:31 smithi173 ceph-mon[36116]: Detected new or changed devices on smithi203 2024-03-23T22:58:31.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:31 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:31.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:31 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:31.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:31 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:58:31.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:31 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:58:31.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:31 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:58:31.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:31 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-23T22:58:31.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:31 smithi173 ceph-mon[36116]: Adjusting osd_memory_target on smithi203 to 4270M 2024-03-23T22:58:31.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:31 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:31.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:31 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:31.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:31 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:58:31.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:31 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:31.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:31 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:58:31.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:31 smithi173 ceph-mon[36116]: osdmap e82: 12 total, 12 up, 12 in 2024-03-23T22:58:31.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:31 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:31.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:31 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:58:31.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:31 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:32.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.879+0000 7f3686800700 1 -- 172.21.15.12:0/319368316 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f368006a7e0 msgr2=0x7f368010f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:32.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.879+0000 7f3686800700 1 --2- 172.21.15.12:0/319368316 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f368006a7e0 0x7f368010f020 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f367800b0d0 tx=0x7f367800b3e0 comp rx=0 tx=0).stop 2024-03-23T22:58:32.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.879+0000 7f3686800700 1 -- 172.21.15.12:0/319368316 shutdown_connections 2024-03-23T22:58:32.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.879+0000 7f3686800700 1 --2- 172.21.15.12:0/319368316 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f368006a7e0 0x7f368010f020 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:32.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.879+0000 7f3686800700 1 --2- 172.21.15.12:0/319368316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36801081e0 0x7f36801085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:32.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.879+0000 7f3686800700 1 --2- 172.21.15.12:0/319368316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3680069e40 0x7f368006a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:32.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.879+0000 7f3686800700 1 -- 172.21.15.12:0/319368316 >> 172.21.15.12:0/319368316 conn(0x7f3680077fc0 msgr2=0x7f36800783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:32.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.879+0000 7f3686800700 1 -- 172.21.15.12:0/319368316 shutdown_connections 2024-03-23T22:58:32.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.879+0000 7f3686800700 1 -- 172.21.15.12:0/319368316 wait complete. 2024-03-23T22:58:32.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.880+0000 7f3686800700 1 Processor -- start 2024-03-23T22:58:32.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.881+0000 7f3686800700 1 -- start start 2024-03-23T22:58:32.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.881+0000 7f3686800700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3680069e40 0x7f36801a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:32.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.881+0000 7f3686800700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f368006a7e0 0x7f36801a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:32.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.881+0000 7f3686800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36801081e0 0x7f368019a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:32.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.881+0000 7f3686800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3680112460 con 0x7f36801081e0 2024-03-23T22:58:32.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.881+0000 7f3686800700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36801122e0 con 0x7f3680069e40 2024-03-23T22:58:32.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.881+0000 7f3686800700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36801125e0 con 0x7f368006a7e0 2024-03-23T22:58:32.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.881+0000 7f367f400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3680069e40 0x7f36801a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:32.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.881+0000 7f367ea00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f368006a7e0 0x7f36801a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:32.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.881+0000 7f367fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36801081e0 0x7f368019a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:32.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.881+0000 7f367ea00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f368006a7e0 0x7f36801a0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44982/0 (socket says 172.21.15.12:44982) 2024-03-23T22:58:32.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.881+0000 7f367f400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3680069e40 0x7f36801a06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:40128/0 (socket says 172.21.15.12:40128) 2024-03-23T22:58:32.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.881+0000 7f367ea00700 1 -- 172.21.15.12:0/3338177140 learned_addr learned my addr 172.21.15.12:0/3338177140 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:58:32.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.882+0000 7f367ea00700 1 -- 172.21.15.12:0/3338177140 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3680069e40 msgr2=0x7f36801a06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:32.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.882+0000 7f367ea00700 1 --2- 172.21.15.12:0/3338177140 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3680069e40 0x7f36801a06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:32.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.882+0000 7f367ea00700 1 -- 172.21.15.12:0/3338177140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36801081e0 msgr2=0x7f368019a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:32.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.882+0000 7f367ea00700 1 --2- 172.21.15.12:0/3338177140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36801081e0 0x7f368019a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:32.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.882+0000 7f367ea00700 1 -- 172.21.15.12:0/3338177140 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f368019b010 con 0x7f368006a7e0 2024-03-23T22:58:32.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.882+0000 7f367fe00700 1 --2- 172.21.15.12:0/3338177140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36801081e0 0x7f368019a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T22:58:32.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.882+0000 7f367ea00700 1 --2- 172.21.15.12:0/3338177140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f368006a7e0 0x7f36801a0c30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f366c00e9f0 tx=0x7f366c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:32.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.882+0000 7f3677e00700 1 -- 172.21.15.12:0/3338177140 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c004a30 con 0x7f368006a7e0 2024-03-23T22:58:32.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.882+0000 7f3686800700 1 -- 172.21.15.12:0/3338177140 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f368019b300 con 0x7f368006a7e0 2024-03-23T22:58:32.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.882+0000 7f3677e00700 1 -- 172.21.15.12:0/3338177140 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f366c005500 con 0x7f368006a7e0 2024-03-23T22:58:32.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.883+0000 7f3677e00700 1 -- 172.21.15.12:0/3338177140 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c00cda0 con 0x7f368006a7e0 2024-03-23T22:58:32.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.883+0000 7f3686800700 1 -- 172.21.15.12:0/3338177140 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36801a75c0 con 0x7f368006a7e0 2024-03-23T22:58:32.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.884+0000 7f3686800700 1 -- 172.21.15.12:0/3338177140 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3680068e50 con 0x7f368006a7e0 2024-03-23T22:58:32.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.886+0000 7f3677e00700 1 -- 172.21.15.12:0/3338177140 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f366c005020 con 0x7f368006a7e0 2024-03-23T22:58:32.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.887+0000 7f3677e00700 1 --2- 172.21.15.12:0/3338177140 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f36640a2910 0x7f36640a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:32.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.887+0000 7f3677e00700 1 -- 172.21.15.12:0/3338177140 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f366c0ce8b0 con 0x7f368006a7e0 2024-03-23T22:58:32.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.888+0000 7f367f400700 1 --2- 172.21.15.12:0/3338177140 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f36640a2910 0x7f36640a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:32.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.888+0000 7f367f400700 1 --2- 172.21.15.12:0/3338177140 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f36640a2910 0x7f36640a4dd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f368019b9f0 tx=0x7f3670009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:32.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:32.890+0000 7f3677e00700 1 -- 172.21.15.12:0/3338177140 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f366c09bd50 con 0x7f368006a7e0 2024-03-23T22:58:33.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:33.114+0000 7f3686800700 1 -- 172.21.15.12:0/3338177140 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f368019bde0 con 0x7f368006a7e0 2024-03-23T22:58:33.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:33.116+0000 7f3677e00700 1 -- 172.21.15.12:0/3338177140 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18481 (secure 0 0 0) 0x7f368019bde0 con 0x7f368006a7e0 2024-03-23T22:58:33.116 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:58:33.116 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":82,"fsid":"90f679ae-e967-11ee-95cd-87774f69a715","created":"2024-03-23T22:50:14.906952+0000","modified":"2024-03-23T22:58:30.496820+0000","last_up_change":"2024-03-23T22:58:28.907824+0000","last_in_change":"2024-03-23T22:58:08.342487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T22:54:11.831010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b70875d7-a23d-4251-9e8b-1a16d4edc2a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6803","nonce":1190079611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6805","nonce":1190079611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6809","nonce":1190079611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6807","nonce":1190079611}]},"public_addr":"172.21.15.12:6803/1190079611","cluster_addr":"172.21.15.12:6805/1190079611","heartbeat_back_addr":"172.21.15.12:6809/1190079611","heartbeat_front_addr":"172.21.15.12:6807/1190079611","state":["exists","up"]},{"osd":1,"uuid":"416e87af-db5d-41ce-84d9-731e3e07e3c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6801","nonce":1983639525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6803","nonce":1983639525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6807","nonce":1983639525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6805","nonce":1983639525}]},"public_addr":"172.21.15.173:6801/1983639525","cluster_addr":"172.21.15.173:6803/1983639525","heartbeat_back_addr":"172.21.15.173:6807/1983639525","heartbeat_front_addr":"172.21.15.173:6805/1983639525","state":["exists","up"]},{"osd":2,"uuid":"7a559c70-4c54-431b-8baf-4136b6e76493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6801","nonce":2253997387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6803","nonce":2253997387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6807","nonce":2253997387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6805","nonce":2253997387}]},"public_addr":"172.21.15.203:6801/2253997387","cluster_addr":"172.21.15.203:6803/2253997387","heartbeat_back_addr":"172.21.15.203:6807/2253997387","heartbeat_front_addr":"172.21.15.203:6805/2253997387","state":["exists","up"]},{"osd":3,"uuid":"c41cd287-a674-486c-8025-328cd36e5ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6811","nonce":1761446905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6813","nonce":1761446905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6817","nonce":1761446905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6815","nonce":1761446905}]},"public_addr":"172.21.15.12:6811/1761446905","cluster_addr":"172.21.15.12:6813/1761446905","heartbeat_back_addr":"172.21.15.12:6817/1761446905","heartbeat_front_addr":"172.21.15.12:6815/1761446905","state":["exists","up"]},{"osd":4,"uuid":"3c48f886-0b42-4a09-b31e-abccc200bb1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6809","nonce":242958272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6811","nonce":242958272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6815","nonce":242958272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6813","nonce":242958272}]},"public_addr":"172.21.15.173:6809/242958272","cluster_addr":"172.21.15.173:6811/242958272","heartbeat_back_addr":"172.21.15.173:6815/242958272","heartbeat_front_addr":"172.21.15.173:6813/242958272","state":["exists","up"]},{"osd":5,"uuid":"4a34f06f-8805-40c0-b1ab-23407b0276d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6809","nonce":4270966092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6811","nonce":4270966092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6815","nonce":4270966092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6813","nonce":4270966092}]},"public_addr":"172.21.15.203:6809/4270966092","cluster_addr":"172.21.15.203:6811/4270966092","heartbeat_back_addr":"172.21.15.203:6815/4270966092","heartbeat_front_addr":"172.21.15.203:6813/4270966092","state":["exists","up"]},{"osd":6,"uuid":"26ddc663-5623-48c9-80d1-f8b968765d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6819","nonce":3606839295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6821","nonce":3606839295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6825","nonce":3606839295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6823","nonce":3606839295}]},"public_addr":"172.21.15.12:6819/3606839295","cluster_addr":"172.21.15.12:6821/3606839295","heartbeat_back_addr":"172.21.15.12:6825/3606839295","heartbeat_front_addr":"172.21.15.12:6823/3606839295","state":["exists","up"]},{"osd":7,"uuid":"17dc9417-71cd-4576-aa08-933212ec55e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6817","nonce":1241504346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6819","nonce":1241504346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6823","nonce":1241504346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6821","nonce":1241504346}]},"public_addr":"172.21.15.173:6817/1241504346","cluster_addr":"172.21.15.173:6819/1241504346","heartbeat_back_addr":"172.21.15.173:6823/1241504346","heartbeat_front_addr":"172.21.15.173:6821/1241504346","state":["exists","up"]},{"osd":8,"uuid":"90a2b63a-471d-43b8-ac18-bb1f16778d18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6817","nonce":1516996192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6819","nonce":1516996192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6823","nonce":1516996192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6821","nonce":1516996192}]},"public_addr":"172.21.15.203:6817/1516996192","cluster_addr":"172.21.15.203:6819/1516996192","heartbeat_back_addr":"172.21.15.203:6823/1516996192","heartbeat_front_addr":"172.21.15.203:6821/1516996192","state":["exists","up"]},{"osd":9,"uuid":"6e1ca504-40b4-49d0-b245-7946181abfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6827","nonce":1175260774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6829","nonce":1175260774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6833","nonce":1175260774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6831","nonce":1175260774}]},"public_addr":"172.21.15.12:6827/1175260774","cluster_addr":"172.21.15.12:6829/1175260774","heartbeat_back_addr":"172.21.15.12:6833/1175260774","heartbeat_front_addr":"172.21.15.12:6831/1175260774","state":["exists","up"]},{"osd":10,"uuid":"09a595ed-5614-4386-8e61-0170af66e01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6825","nonce":3303506808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6827","nonce":3303506808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6831","nonce":3303506808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6829","nonce":3303506808}]},"public_addr":"172.21.15.173:6825/3303506808","cluster_addr":"172.21.15.173:6827/3303506808","heartbeat_back_addr":"172.21.15.173:6831/3303506808","heartbeat_front_addr":"172.21.15.173:6829/3303506808","state":["exists","up"]},{"osd":11,"uuid":"9ef1d595-a977-4ba7-9c6b-d2b3295f20b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6825","nonce":2323604506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6827","nonce":2323604506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6831","nonce":2323604506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6829","nonce":2323604506}]},"public_addr":"172.21.15.203:6825/2323604506","cluster_addr":"172.21.15.203:6827/2323604506","heartbeat_back_addr":"172.21.15.203:6831/2323604506","heartbeat_front_addr":"172.21.15.203:6829/2323604506","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:18.469038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:42.752628+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:07.069493+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:41.962478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:08.363104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:33.341336+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:07.286541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:34.243302+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:59.749699+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:57:33.822913+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:00.737269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:26.599854+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/4182783120":"2024-03-24T22:51:23.113201+0000","172.21.15.12:6800/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1900938637":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/3414861652":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6801/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/2580541477":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6800/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/501058839":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1192554716":"2024-03-24T22:50:48.401223+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T22:58:33.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:33.119+0000 7f3686800700 1 -- 172.21.15.12:0/3338177140 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f36640a2910 msgr2=0x7f36640a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:33.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:33.119+0000 7f3686800700 1 --2- 172.21.15.12:0/3338177140 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f36640a2910 0x7f36640a4dd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f368019b9f0 tx=0x7f3670009260 comp rx=0 tx=0).stop 2024-03-23T22:58:33.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:33.119+0000 7f3686800700 1 -- 172.21.15.12:0/3338177140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f368006a7e0 msgr2=0x7f36801a0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:33.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:33.119+0000 7f3686800700 1 --2- 172.21.15.12:0/3338177140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f368006a7e0 0x7f36801a0c30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f366c00e9f0 tx=0x7f366c00ed00 comp rx=0 tx=0).stop 2024-03-23T22:58:33.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:33.119+0000 7f3686800700 1 -- 172.21.15.12:0/3338177140 shutdown_connections 2024-03-23T22:58:33.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:33.119+0000 7f3686800700 1 --2- 172.21.15.12:0/3338177140 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f36640a2910 0x7f36640a4dd0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:33.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:33.119+0000 7f3686800700 1 --2- 172.21.15.12:0/3338177140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36801081e0 0x7f368019a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:33.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:33.119+0000 7f3686800700 1 --2- 172.21.15.12:0/3338177140 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3680069e40 0x7f36801a06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:33.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:33.119+0000 7f3686800700 1 --2- 172.21.15.12:0/3338177140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f368006a7e0 0x7f36801a0c30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:33.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:33.119+0000 7f3686800700 1 -- 172.21.15.12:0/3338177140 >> 172.21.15.12:0/3338177140 conn(0x7f3680077fc0 msgr2=0x7f368010d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:33.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:33.119+0000 7f3686800700 1 -- 172.21.15.12:0/3338177140 shutdown_connections 2024-03-23T22:58:33.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:33.120+0000 7f3686800700 1 -- 172.21.15.12:0/3338177140 wait complete. 2024-03-23T22:58:33.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:33 smithi173 ceph-mon[36116]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:33.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:33 smithi203 ceph-mon[36693]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:33.551 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:33 smithi012 ceph-mon[31695]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:33.968 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T22:54:11.831010+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T22:58:33.969 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-23T22:58:34.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:34 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3338177140' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T22:58:34.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:34 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3338177140' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T22:58:34.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:34 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3338177140' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T22:58:34.430 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:58:35.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:35 smithi012 ceph-mon[31695]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:35.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:35 smithi173 ceph-mon[36116]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:35.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:35 smithi203 ceph-mon[36693]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:36.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.695+0000 7fd996000700 1 -- 172.21.15.12:0/441389389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd99006a7e0 msgr2=0x7fd99010f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:36.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.695+0000 7fd996000700 1 --2- 172.21.15.12:0/441389389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd99006a7e0 0x7fd99010f020 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd98800b0d0 tx=0x7fd98800b3e0 comp rx=0 tx=0).stop 2024-03-23T22:58:36.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.695+0000 7fd996000700 1 -- 172.21.15.12:0/441389389 shutdown_connections 2024-03-23T22:58:36.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.695+0000 7fd996000700 1 --2- 172.21.15.12:0/441389389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd99006a7e0 0x7fd99010f020 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:36.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.695+0000 7fd996000700 1 --2- 172.21.15.12:0/441389389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9901081e0 0x7fd9901085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:36.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.695+0000 7fd996000700 1 --2- 172.21.15.12:0/441389389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd990069e40 0x7fd99006a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:36.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.695+0000 7fd996000700 1 -- 172.21.15.12:0/441389389 >> 172.21.15.12:0/441389389 conn(0x7fd990077fc0 msgr2=0x7fd9900783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:36.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.695+0000 7fd996000700 1 -- 172.21.15.12:0/441389389 shutdown_connections 2024-03-23T22:58:36.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.695+0000 7fd996000700 1 -- 172.21.15.12:0/441389389 wait complete. 2024-03-23T22:58:36.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.697+0000 7fd996000700 1 Processor -- start 2024-03-23T22:58:36.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.697+0000 7fd996000700 1 -- start start 2024-03-23T22:58:36.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.697+0000 7fd996000700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd990069e40 0x7fd9901a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:36.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.697+0000 7fd996000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd99006a7e0 0x7fd9901a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:36.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.697+0000 7fd996000700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9901081e0 0x7fd99019a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:36.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.697+0000 7fd996000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd990112480 con 0x7fd99006a7e0 2024-03-23T22:58:36.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.697+0000 7fd996000700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd990112300 con 0x7fd990069e40 2024-03-23T22:58:36.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.697+0000 7fd996000700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd990112600 con 0x7fd9901081e0 2024-03-23T22:58:36.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.698+0000 7fd98f400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd990069e40 0x7fd9901a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:36.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.698+0000 7fd986a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd99006a7e0 0x7fd9901a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:36.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.698+0000 7fd98f400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd990069e40 0x7fd9901a0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:47716/0 (socket says 172.21.15.12:47716) 2024-03-23T22:58:36.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.698+0000 7fd98f400700 1 -- 172.21.15.12:0/1430105290 learned_addr learned my addr 172.21.15.12:0/1430105290 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:58:36.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.698+0000 7fd98fe00700 1 --2- 172.21.15.12:0/1430105290 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9901081e0 0x7fd99019a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:36.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.698+0000 7fd98f400700 1 -- 172.21.15.12:0/1430105290 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9901081e0 msgr2=0x7fd99019a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:36.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.698+0000 7fd98f400700 1 --2- 172.21.15.12:0/1430105290 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9901081e0 0x7fd99019a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:36.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.698+0000 7fd98f400700 1 -- 172.21.15.12:0/1430105290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd99006a7e0 msgr2=0x7fd9901a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:36.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.698+0000 7fd98f400700 1 --2- 172.21.15.12:0/1430105290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd99006a7e0 0x7fd9901a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:36.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.698+0000 7fd98f400700 1 -- 172.21.15.12:0/1430105290 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd99019b030 con 0x7fd990069e40 2024-03-23T22:58:36.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.698+0000 7fd986a00700 1 --2- 172.21.15.12:0/1430105290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd99006a7e0 0x7fd9901a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T22:58:36.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.698+0000 7fd98f400700 1 --2- 172.21.15.12:0/1430105290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd990069e40 0x7fd9901a0710 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd98000eb30 tx=0x7fd98000ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:36.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.699+0000 7fd98cc00700 1 -- 172.21.15.12:0/1430105290 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98000cbd0 con 0x7fd990069e40 2024-03-23T22:58:36.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.699+0000 7fd98cc00700 1 -- 172.21.15.12:0/1430105290 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd980004500 con 0x7fd990069e40 2024-03-23T22:58:36.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.699+0000 7fd98cc00700 1 -- 172.21.15.12:0/1430105290 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd980010430 con 0x7fd990069e40 2024-03-23T22:58:36.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.699+0000 7fd996000700 1 -- 172.21.15.12:0/1430105290 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd99019b320 con 0x7fd990069e40 2024-03-23T22:58:36.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.699+0000 7fd996000700 1 -- 172.21.15.12:0/1430105290 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9901a75e0 con 0x7fd990069e40 2024-03-23T22:58:36.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.700+0000 7fd996000700 1 -- 172.21.15.12:0/1430105290 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd990068e50 con 0x7fd990069e40 2024-03-23T22:58:36.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.702+0000 7fd98cc00700 1 -- 172.21.15.12:0/1430105290 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd980003f00 con 0x7fd990069e40 2024-03-23T22:58:36.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.703+0000 7fd98cc00700 1 --2- 172.21.15.12:0/1430105290 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd9740a2910 0x7fd9740a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:36.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.703+0000 7fd98cc00700 1 -- 172.21.15.12:0/1430105290 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd9800ce8a0 con 0x7fd990069e40 2024-03-23T22:58:36.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.703+0000 7fd986a00700 1 --2- 172.21.15.12:0/1430105290 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd9740a2910 0x7fd9740a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:36.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.704+0000 7fd986a00700 1 --2- 172.21.15.12:0/1430105290 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd9740a2910 0x7fd9740a4dd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd99010f3d0 tx=0x7fd978009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:36.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.708+0000 7fd98cc00700 1 -- 172.21.15.12:0/1430105290 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd98009bd40 con 0x7fd990069e40 2024-03-23T22:58:36.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.932+0000 7fd996000700 1 -- 172.21.15.12:0/1430105290 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd9901a7870 con 0x7fd990069e40 2024-03-23T22:58:36.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.933+0000 7fd98cc00700 1 -- 172.21.15.12:0/1430105290 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd980097030 con 0x7fd990069e40 2024-03-23T22:58:36.932 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-23T22:58:36.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.935+0000 7fd996000700 1 -- 172.21.15.12:0/1430105290 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd9740a2910 msgr2=0x7fd9740a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:36.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.935+0000 7fd996000700 1 --2- 172.21.15.12:0/1430105290 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd9740a2910 0x7fd9740a4dd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd99010f3d0 tx=0x7fd978009260 comp rx=0 tx=0).stop 2024-03-23T22:58:36.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.935+0000 7fd996000700 1 -- 172.21.15.12:0/1430105290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd990069e40 msgr2=0x7fd9901a0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:36.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.935+0000 7fd996000700 1 --2- 172.21.15.12:0/1430105290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd990069e40 0x7fd9901a0710 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd98000eb30 tx=0x7fd98000ee40 comp rx=0 tx=0).stop 2024-03-23T22:58:36.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.936+0000 7fd996000700 1 -- 172.21.15.12:0/1430105290 shutdown_connections 2024-03-23T22:58:36.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.936+0000 7fd996000700 1 --2- 172.21.15.12:0/1430105290 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd9740a2910 0x7fd9740a4dd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:36.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.936+0000 7fd996000700 1 --2- 172.21.15.12:0/1430105290 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9901081e0 0x7fd99019a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:36.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.936+0000 7fd996000700 1 --2- 172.21.15.12:0/1430105290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd990069e40 0x7fd9901a0710 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:36.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.936+0000 7fd996000700 1 --2- 172.21.15.12:0/1430105290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd99006a7e0 0x7fd9901a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:36.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.936+0000 7fd996000700 1 -- 172.21.15.12:0/1430105290 >> 172.21.15.12:0/1430105290 conn(0x7fd990077fc0 msgr2=0x7fd9900ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:36.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.936+0000 7fd996000700 1 -- 172.21.15.12:0/1430105290 shutdown_connections 2024-03-23T22:58:36.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:36.936+0000 7fd996000700 1 -- 172.21.15.12:0/1430105290 wait complete. 2024-03-23T22:58:37.306 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:37 smithi012 ceph-mon[31695]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:37.306 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:37 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1430105290' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T22:58:37.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:37 smithi173 ceph-mon[36116]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:37.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:37 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1430105290' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T22:58:37.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:37 smithi203 ceph-mon[36693]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:37.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:37 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1430105290' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T22:58:37.761 INFO:tasks.cephadm:Adding mds.a on smithi012 2024-03-23T22:58:37.761 INFO:tasks.cephadm:Adding mds.d on smithi012 2024-03-23T22:58:37.761 INFO:tasks.cephadm:Adding mds.g on smithi012 2024-03-23T22:58:37.762 INFO:tasks.cephadm:Adding mds.j on smithi012 2024-03-23T22:58:37.762 INFO:tasks.cephadm:Adding mds.b on smithi173 2024-03-23T22:58:37.762 INFO:tasks.cephadm:Adding mds.e on smithi173 2024-03-23T22:58:37.762 INFO:tasks.cephadm:Adding mds.h on smithi173 2024-03-23T22:58:37.762 INFO:tasks.cephadm:Adding mds.k on smithi173 2024-03-23T22:58:37.762 INFO:tasks.cephadm:Adding mds.c on smithi203 2024-03-23T22:58:37.762 INFO:tasks.cephadm:Adding mds.f on smithi203 2024-03-23T22:58:37.762 INFO:tasks.cephadm:Adding mds.i on smithi203 2024-03-23T22:58:37.762 INFO:tasks.cephadm:Adding mds.l on smithi203 2024-03-23T22:58:37.762 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph orch apply mds all '12;smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l' 2024-03-23T22:58:38.013 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.c/config 2024-03-23T22:58:39.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:39 smithi173 ceph-mon[36116]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:39.384 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:39 smithi203 ceph-mon[36693]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:39.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:39 smithi012 ceph-mon[31695]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:39.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.743+0000 7f2eb4c2f700 1 -- 172.21.15.203:0/4109763615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2eb01022f0 msgr2=0x7f2eb0102750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:39.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.743+0000 7f2eb4c2f700 1 --2- 172.21.15.203:0/4109763615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2eb01022f0 0x7f2eb0102750 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4009990 tx=0x7f2ea4009ca0 comp rx=0 tx=0).stop 2024-03-23T22:58:39.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.743+0000 7f2eb4c2f700 1 -- 172.21.15.203:0/4109763615 shutdown_connections 2024-03-23T22:58:39.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.743+0000 7f2eb4c2f700 1 --2- 172.21.15.203:0/4109763615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2eb0102c90 0x7f2eb0109540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:39.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.743+0000 7f2eb4c2f700 1 --2- 172.21.15.203:0/4109763615 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2eb01010f0 0x7f2eb0101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:39.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.743+0000 7f2eb4c2f700 1 --2- 172.21.15.203:0/4109763615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2eb01022f0 0x7f2eb0102750 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:39.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.743+0000 7f2eb4c2f700 1 -- 172.21.15.203:0/4109763615 >> 172.21.15.203:0/4109763615 conn(0x7f2eb00fc6b0 msgr2=0x7f2eb00fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:39.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.743+0000 7f2eb4c2f700 1 -- 172.21.15.203:0/4109763615 shutdown_connections 2024-03-23T22:58:39.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.743+0000 7f2eb4c2f700 1 -- 172.21.15.203:0/4109763615 wait complete. 2024-03-23T22:58:39.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.743+0000 7f2eb4c2f700 1 Processor -- start 2024-03-23T22:58:39.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.744+0000 7f2eb4c2f700 1 -- start start 2024-03-23T22:58:39.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.744+0000 7f2eb4c2f700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2eb01010f0 0x7f2eb006b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:39.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.744+0000 7f2eb4c2f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2eb01022f0 0x7f2eb019d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:39.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.744+0000 7f2eb4c2f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2eb0102c90 0x7f2eb019f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:39.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.744+0000 7f2eb4c2f700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb010cd80 con 0x7f2eb01010f0 2024-03-23T22:58:39.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.744+0000 7f2eb4c2f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb010cc00 con 0x7f2eb01022f0 2024-03-23T22:58:39.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.744+0000 7f2eb4c2f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb010cf00 con 0x7f2eb0102c90 2024-03-23T22:58:39.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.744+0000 7f2eaed9d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2eb0102c90 0x7f2eb019f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:39.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.744+0000 7f2eadd9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2eb01022f0 0x7f2eb019d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:39.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.744+0000 7f2eaed9d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2eb0102c90 0x7f2eb019f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:42770/0 (socket says 172.21.15.203:42770) 2024-03-23T22:58:39.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.744+0000 7f2eaed9d700 1 -- 172.21.15.203:0/1935841131 learned_addr learned my addr 172.21.15.203:0/1935841131 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-23T22:58:39.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.744+0000 7f2eaed9d700 1 -- 172.21.15.203:0/1935841131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2eb01022f0 msgr2=0x7f2eb019d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:39.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.744+0000 7f2eaed9d700 1 --2- 172.21.15.203:0/1935841131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2eb01022f0 0x7f2eb019d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:39.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.744+0000 7f2eaed9d700 1 -- 172.21.15.203:0/1935841131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2eb01010f0 msgr2=0x7f2eb006b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:39.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.744+0000 7f2eaed9d700 1 --2- 172.21.15.203:0/1935841131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2eb01010f0 0x7f2eb006b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:39.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.744+0000 7f2eaed9d700 1 -- 172.21.15.203:0/1935841131 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb01a00b0 con 0x7f2eb0102c90 2024-03-23T22:58:39.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.744+0000 7f2eaed9d700 1 --2- 172.21.15.203:0/1935841131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2eb0102c90 0x7f2eb019f7e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8007ee0 tx=0x7f2ea800d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:39.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.744+0000 7f2e9f7fe700 1 -- 172.21.15.203:0/1935841131 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea800dbd0 con 0x7f2eb0102c90 2024-03-23T22:58:39.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.745+0000 7f2eb4c2f700 1 -- 172.21.15.203:0/1935841131 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb01a03a0 con 0x7f2eb0102c90 2024-03-23T22:58:39.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.745+0000 7f2e9f7fe700 1 -- 172.21.15.203:0/1935841131 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ea800dd70 con 0x7f2eb0102c90 2024-03-23T22:58:39.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.745+0000 7f2e9f7fe700 1 -- 172.21.15.203:0/1935841131 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8005a60 con 0x7f2eb0102c90 2024-03-23T22:58:39.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.745+0000 7f2eb4c2f700 1 -- 172.21.15.203:0/1935841131 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb01a0980 con 0x7f2eb0102c90 2024-03-23T22:58:39.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.746+0000 7f2eb4c2f700 1 -- 172.21.15.203:0/1935841131 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb0068e50 con 0x7f2eb0102c90 2024-03-23T22:58:39.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.747+0000 7f2e9f7fe700 1 -- 172.21.15.203:0/1935841131 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2ea801a020 con 0x7f2eb0102c90 2024-03-23T22:58:39.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.747+0000 7f2e9f7fe700 1 --2- 172.21.15.203:0/1935841131 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2e980a2890 0x7f2e980a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:39.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.747+0000 7f2e9f7fe700 1 -- 172.21.15.203:0/1935841131 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2ea80ce390 con 0x7f2eb0102c90 2024-03-23T22:58:39.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.748+0000 7f2eae59c700 1 --2- 172.21.15.203:0/1935841131 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2e980a2890 0x7f2e980a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:39.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.749+0000 7f2eae59c700 1 --2- 172.21.15.203:0/1935841131 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2e980a2890 0x7f2e980a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0102150 tx=0x7f2ea0006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:39.751 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.752+0000 7f2e9f7fe700 1 -- 172.21.15.203:0/1935841131 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2ea8010070 con 0x7f2eb0102c90 2024-03-23T22:58:39.954 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.954+0000 7f2eb4c2f700 1 -- 172.21.15.203:0/1935841131 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f2eb00627d0 con 0x7f2e980a2890 2024-03-23T22:58:39.971 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.972+0000 7f2e9f7fe700 1 -- 172.21.15.203:0/1935841131 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f2eb00627d0 con 0x7f2e980a2890 2024-03-23T22:58:39.971 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mds.all update... 2024-03-23T22:58:39.974 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.973+0000 7f2eb4c2f700 1 -- 172.21.15.203:0/1935841131 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2e980a2890 msgr2=0x7f2e980a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:39.974 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.973+0000 7f2eb4c2f700 1 --2- 172.21.15.203:0/1935841131 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2e980a2890 0x7f2e980a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0102150 tx=0x7f2ea0006d20 comp rx=0 tx=0).stop 2024-03-23T22:58:39.974 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.973+0000 7f2eb4c2f700 1 -- 172.21.15.203:0/1935841131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2eb0102c90 msgr2=0x7f2eb019f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:39.974 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.973+0000 7f2eb4c2f700 1 --2- 172.21.15.203:0/1935841131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2eb0102c90 0x7f2eb019f7e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8007ee0 tx=0x7f2ea800d3b0 comp rx=0 tx=0).stop 2024-03-23T22:58:39.974 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.973+0000 7f2eb4c2f700 1 -- 172.21.15.203:0/1935841131 shutdown_connections 2024-03-23T22:58:39.974 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.973+0000 7f2eb4c2f700 1 --2- 172.21.15.203:0/1935841131 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2e980a2890 0x7f2e980a4d50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:39.974 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.973+0000 7f2eb4c2f700 1 --2- 172.21.15.203:0/1935841131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2eb0102c90 0x7f2eb019f7e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:39.974 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.973+0000 7f2eb4c2f700 1 --2- 172.21.15.203:0/1935841131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2eb01010f0 0x7f2eb006b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:39.974 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.973+0000 7f2eb4c2f700 1 --2- 172.21.15.203:0/1935841131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2eb01022f0 0x7f2eb019d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:39.974 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.973+0000 7f2eb4c2f700 1 -- 172.21.15.203:0/1935841131 >> 172.21.15.203:0/1935841131 conn(0x7f2eb00fc6b0 msgr2=0x7f2eb01055f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:39.974 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.973+0000 7f2eb4c2f700 1 -- 172.21.15.203:0/1935841131 shutdown_connections 2024-03-23T22:58:39.974 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-23T22:58:39.973+0000 7f2eb4c2f700 1 -- 172.21.15.203:0/1935841131 wait complete. 2024-03-23T22:58:40.670 DEBUG:teuthology.orchestra.run.smithi012:mds.a> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.a.service 2024-03-23T22:58:40.676 DEBUG:teuthology.orchestra.run.smithi012:mds.d> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.d.service 2024-03-23T22:58:40.720 DEBUG:teuthology.orchestra.run.smithi012:mds.g> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.g.service 2024-03-23T22:58:40.765 DEBUG:teuthology.orchestra.run.smithi012:mds.j> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.j.service 2024-03-23T22:58:40.812 DEBUG:teuthology.orchestra.run.smithi173:mds.b> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.b.service 2024-03-23T22:58:40.815 DEBUG:teuthology.orchestra.run.smithi173:mds.e> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.e.service 2024-03-23T22:58:40.860 DEBUG:teuthology.orchestra.run.smithi173:mds.h> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.h.service 2024-03-23T22:58:40.902 DEBUG:teuthology.orchestra.run.smithi173:mds.k> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.k.service 2024-03-23T22:58:40.946 DEBUG:teuthology.orchestra.run.smithi203:mds.c> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.c.service 2024-03-23T22:58:40.949 DEBUG:teuthology.orchestra.run.smithi203:mds.f> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.f.service 2024-03-23T22:58:40.992 DEBUG:teuthology.orchestra.run.smithi203:mds.i> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.i.service 2024-03-23T22:58:41.035 DEBUG:teuthology.orchestra.run.smithi203:mds.l> sudo journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.l.service 2024-03-23T22:58:41.078 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-23T22:58:41.079 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T22:58:41.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:40 smithi173 ceph-mon[36116]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:41.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:40 smithi173 ceph-mon[36116]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:58:41.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:40 smithi173 ceph-mon[36116]: Saving service mds.all spec with placement smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-03-23T22:58:41.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:41.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:40 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:58:41.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:40 smithi012 ceph-mon[31695]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:41.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:40 smithi012 ceph-mon[31695]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:58:41.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:40 smithi012 ceph-mon[31695]: Saving service mds.all spec with placement smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-03-23T22:58:41.172 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:41.172 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:40 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:58:41.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.353+0000 7f431b200640 1 -- 172.21.15.12:0/334657725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f431c077c40 msgr2=0x7f431c0784e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:41.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.353+0000 7f431b200640 1 --2- 172.21.15.12:0/334657725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f431c077c40 0x7f431c0784e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4314009be0 tx=0x7f431402bef0 comp rx=0 tx=0).stop 2024-03-23T22:58:41.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.353+0000 7f431b200640 1 -- 172.21.15.12:0/334657725 shutdown_connections 2024-03-23T22:58:41.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.353+0000 7f431b200640 1 --2- 172.21.15.12:0/334657725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f431c077c40 0x7f431c0784e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:41.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.353+0000 7f431b200640 1 --2- 172.21.15.12:0/334657725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f431c0772a0 0x7f431c077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:41.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.353+0000 7f431b200640 1 --2- 172.21.15.12:0/334657725 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f431c076060 0x7f431c076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:41.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.353+0000 7f431b200640 1 -- 172.21.15.12:0/334657725 >> 172.21.15.12:0/334657725 conn(0x7f431c0fec80 msgr2=0x7f431c1010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:41.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.353+0000 7f431b200640 1 -- 172.21.15.12:0/334657725 shutdown_connections 2024-03-23T22:58:41.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.353+0000 7f431b200640 1 -- 172.21.15.12:0/334657725 wait complete. 2024-03-23T22:58:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.354+0000 7f431b200640 1 Processor -- start 2024-03-23T22:58:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.354+0000 7f431b200640 1 -- start start 2024-03-23T22:58:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.354+0000 7f431b200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f431c076060 0x7f431c19dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.354+0000 7f431b200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f431c0772a0 0x7f431c19e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.355+0000 7f431b200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f431c077c40 0x7f431c1a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.355+0000 7f431b200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431c1137c0 con 0x7f431c0772a0 2024-03-23T22:58:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.355+0000 7f431b200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431c113640 con 0x7f431c076060 2024-03-23T22:58:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.355+0000 7f431b200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431c113940 con 0x7f431c077c40 2024-03-23T22:58:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.355+0000 7f4319400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f431c0772a0 0x7f431c19e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.355+0000 7f431a800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f431c077c40 0x7f431c1a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.355+0000 7f431a800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f431c077c40 0x7f431c1a52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45900/0 (socket says 172.21.15.12:45900) 2024-03-23T22:58:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.355+0000 7f4319400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f431c0772a0 0x7f431c19e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57154/0 (socket says 172.21.15.12:57154) 2024-03-23T22:58:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.355+0000 7f4319e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f431c076060 0x7f431c19dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.355+0000 7f431a800640 1 -- 172.21.15.12:0/2349613837 learned_addr learned my addr 172.21.15.12:0/2349613837 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:58:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.355+0000 7f431a800640 1 -- 172.21.15.12:0/2349613837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f431c076060 msgr2=0x7f431c19dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.355+0000 7f431a800640 1 --2- 172.21.15.12:0/2349613837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f431c076060 0x7f431c19dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.355+0000 7f431a800640 1 -- 172.21.15.12:0/2349613837 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f431c0772a0 msgr2=0x7f431c19e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.355+0000 7f431a800640 1 --2- 172.21.15.12:0/2349613837 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f431c0772a0 0x7f431c19e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.355+0000 7f431a800640 1 -- 172.21.15.12:0/2349613837 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431c1a5960 con 0x7f431c077c40 2024-03-23T22:58:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.356+0000 7f431a800640 1 --2- 172.21.15.12:0/2349613837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f431c077c40 0x7f431c1a52c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4314036040 tx=0x7f43140337b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.356+0000 7f4312a00640 1 -- 172.21.15.12:0/2349613837 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4314043020 con 0x7f431c077c40 2024-03-23T22:58:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.356+0000 7f4312a00640 1 -- 172.21.15.12:0/2349613837 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f431402ca60 con 0x7f431c077c40 2024-03-23T22:58:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.356+0000 7f431b200640 1 -- 172.21.15.12:0/2349613837 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f431c1a5bf0 con 0x7f431c077c40 2024-03-23T22:58:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.356+0000 7f4312a00640 1 -- 172.21.15.12:0/2349613837 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4314034690 con 0x7f431c077c40 2024-03-23T22:58:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.356+0000 7f431b200640 1 -- 172.21.15.12:0/2349613837 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f431c1a6140 con 0x7f431c077c40 2024-03-23T22:58:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.357+0000 7f431b200640 1 -- 172.21.15.12:0/2349613837 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f431c110170 con 0x7f431c077c40 2024-03-23T22:58:41.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.359+0000 7f4312a00640 1 -- 172.21.15.12:0/2349613837 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4314046020 con 0x7f431c077c40 2024-03-23T22:58:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.360+0000 7f4312a00640 1 --2- 172.21.15.12:0/2349613837 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f43000a2800 0x7f43000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.361+0000 7f4312a00640 1 -- 172.21.15.12:0/2349613837 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f43140f0cd0 con 0x7f431c077c40 2024-03-23T22:58:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.361+0000 7f4319e00640 1 --2- 172.21.15.12:0/2349613837 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f43000a2800 0x7f43000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:41.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.361+0000 7f4319e00640 1 --2- 172.21.15.12:0/2349613837 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f43000a2800 0x7f43000a4cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f431c19f220 tx=0x7f430c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:41.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.364+0000 7f4312a00640 1 -- 172.21.15.12:0/2349613837 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f431404d020 con 0x7f431c077c40 2024-03-23T22:58:41.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:40 smithi203 ceph-mon[36693]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:41.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:40 smithi203 ceph-mon[36693]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:58:41.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:40 smithi203 ceph-mon[36693]: Saving service mds.all spec with placement smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-03-23T22:58:41.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:41.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:40 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:58:41.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.570+0000 7f431b200640 1 -- 172.21.15.12:0/2349613837 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f431c0622a0 con 0x7f431c077c40 2024-03-23T22:58:41.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.571+0000 7f4312a00640 1 -- 172.21.15.12:0/2349613837 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18481 (secure 0 0 0) 0x7f43140be170 con 0x7f431c077c40 2024-03-23T22:58:41.571 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:58:41.571 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":82,"fsid":"90f679ae-e967-11ee-95cd-87774f69a715","created":"2024-03-23T22:50:14.906952+0000","modified":"2024-03-23T22:58:30.496820+0000","last_up_change":"2024-03-23T22:58:28.907824+0000","last_in_change":"2024-03-23T22:58:08.342487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T22:54:11.831010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b70875d7-a23d-4251-9e8b-1a16d4edc2a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6803","nonce":1190079611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6805","nonce":1190079611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6809","nonce":1190079611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6807","nonce":1190079611}]},"public_addr":"172.21.15.12:6803/1190079611","cluster_addr":"172.21.15.12:6805/1190079611","heartbeat_back_addr":"172.21.15.12:6809/1190079611","heartbeat_front_addr":"172.21.15.12:6807/1190079611","state":["exists","up"]},{"osd":1,"uuid":"416e87af-db5d-41ce-84d9-731e3e07e3c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6801","nonce":1983639525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6803","nonce":1983639525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6807","nonce":1983639525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6805","nonce":1983639525}]},"public_addr":"172.21.15.173:6801/1983639525","cluster_addr":"172.21.15.173:6803/1983639525","heartbeat_back_addr":"172.21.15.173:6807/1983639525","heartbeat_front_addr":"172.21.15.173:6805/1983639525","state":["exists","up"]},{"osd":2,"uuid":"7a559c70-4c54-431b-8baf-4136b6e76493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6801","nonce":2253997387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6803","nonce":2253997387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6807","nonce":2253997387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6805","nonce":2253997387}]},"public_addr":"172.21.15.203:6801/2253997387","cluster_addr":"172.21.15.203:6803/2253997387","heartbeat_back_addr":"172.21.15.203:6807/2253997387","heartbeat_front_addr":"172.21.15.203:6805/2253997387","state":["exists","up"]},{"osd":3,"uuid":"c41cd287-a674-486c-8025-328cd36e5ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6811","nonce":1761446905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6813","nonce":1761446905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6817","nonce":1761446905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6815","nonce":1761446905}]},"public_addr":"172.21.15.12:6811/1761446905","cluster_addr":"172.21.15.12:6813/1761446905","heartbeat_back_addr":"172.21.15.12:6817/1761446905","heartbeat_front_addr":"172.21.15.12:6815/1761446905","state":["exists","up"]},{"osd":4,"uuid":"3c48f886-0b42-4a09-b31e-abccc200bb1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6809","nonce":242958272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6811","nonce":242958272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6815","nonce":242958272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6813","nonce":242958272}]},"public_addr":"172.21.15.173:6809/242958272","cluster_addr":"172.21.15.173:6811/242958272","heartbeat_back_addr":"172.21.15.173:6815/242958272","heartbeat_front_addr":"172.21.15.173:6813/242958272","state":["exists","up"]},{"osd":5,"uuid":"4a34f06f-8805-40c0-b1ab-23407b0276d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6809","nonce":4270966092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6811","nonce":4270966092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6815","nonce":4270966092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6813","nonce":4270966092}]},"public_addr":"172.21.15.203:6809/4270966092","cluster_addr":"172.21.15.203:6811/4270966092","heartbeat_back_addr":"172.21.15.203:6815/4270966092","heartbeat_front_addr":"172.21.15.203:6813/4270966092","state":["exists","up"]},{"osd":6,"uuid":"26ddc663-5623-48c9-80d1-f8b968765d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6819","nonce":3606839295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6821","nonce":3606839295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6825","nonce":3606839295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6823","nonce":3606839295}]},"public_addr":"172.21.15.12:6819/3606839295","cluster_addr":"172.21.15.12:6821/3606839295","heartbeat_back_addr":"172.21.15.12:6825/3606839295","heartbeat_front_addr":"172.21.15.12:6823/3606839295","state":["exists","up"]},{"osd":7,"uuid":"17dc9417-71cd-4576-aa08-933212ec55e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6817","nonce":1241504346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6819","nonce":1241504346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6823","nonce":1241504346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6821","nonce":1241504346}]},"public_addr":"172.21.15.173:6817/1241504346","cluster_addr":"172.21.15.173:6819/1241504346","heartbeat_back_addr":"172.21.15.173:6823/1241504346","heartbeat_front_addr":"172.21.15.173:6821/1241504346","state":["exists","up"]},{"osd":8,"uuid":"90a2b63a-471d-43b8-ac18-bb1f16778d18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6817","nonce":1516996192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6819","nonce":1516996192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6823","nonce":1516996192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6821","nonce":1516996192}]},"public_addr":"172.21.15.203:6817/1516996192","cluster_addr":"172.21.15.203:6819/1516996192","heartbeat_back_addr":"172.21.15.203:6823/1516996192","heartbeat_front_addr":"172.21.15.203:6821/1516996192","state":["exists","up"]},{"osd":9,"uuid":"6e1ca504-40b4-49d0-b245-7946181abfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6827","nonce":1175260774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6829","nonce":1175260774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6833","nonce":1175260774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6831","nonce":1175260774}]},"public_addr":"172.21.15.12:6827/1175260774","cluster_addr":"172.21.15.12:6829/1175260774","heartbeat_back_addr":"172.21.15.12:6833/1175260774","heartbeat_front_addr":"172.21.15.12:6831/1175260774","state":["exists","up"]},{"osd":10,"uuid":"09a595ed-5614-4386-8e61-0170af66e01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6825","nonce":3303506808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6827","nonce":3303506808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6831","nonce":3303506808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6829","nonce":3303506808}]},"public_addr":"172.21.15.173:6825/3303506808","cluster_addr":"172.21.15.173:6827/3303506808","heartbeat_back_addr":"172.21.15.173:6831/3303506808","heartbeat_front_addr":"172.21.15.173:6829/3303506808","state":["exists","up"]},{"osd":11,"uuid":"9ef1d595-a977-4ba7-9c6b-d2b3295f20b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6825","nonce":2323604506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6827","nonce":2323604506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6831","nonce":2323604506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6829","nonce":2323604506}]},"public_addr":"172.21.15.203:6825/2323604506","cluster_addr":"172.21.15.203:6827/2323604506","heartbeat_back_addr":"172.21.15.203:6831/2323604506","heartbeat_front_addr":"172.21.15.203:6829/2323604506","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:18.469038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:42.752628+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:07.069493+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:41.962478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:08.363104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:33.341336+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:07.286541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:34.243302+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:59.749699+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:57:33.822913+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:00.737269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:26.599854+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/4182783120":"2024-03-24T22:51:23.113201+0000","172.21.15.12:6800/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1900938637":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/3414861652":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6801/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/2580541477":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6800/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/501058839":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1192554716":"2024-03-24T22:50:48.401223+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T22:58:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.574+0000 7f431b200640 1 -- 172.21.15.12:0/2349613837 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f43000a2800 msgr2=0x7f43000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:41.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.574+0000 7f431b200640 1 --2- 172.21.15.12:0/2349613837 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f43000a2800 0x7f43000a4cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f431c19f220 tx=0x7f430c005f90 comp rx=0 tx=0).stop 2024-03-23T22:58:41.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.574+0000 7f431b200640 1 -- 172.21.15.12:0/2349613837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f431c077c40 msgr2=0x7f431c1a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:41.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.574+0000 7f431b200640 1 --2- 172.21.15.12:0/2349613837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f431c077c40 0x7f431c1a52c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4314036040 tx=0x7f43140337b0 comp rx=0 tx=0).stop 2024-03-23T22:58:41.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.574+0000 7f431b200640 1 -- 172.21.15.12:0/2349613837 shutdown_connections 2024-03-23T22:58:41.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.574+0000 7f431b200640 1 --2- 172.21.15.12:0/2349613837 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f43000a2800 0x7f43000a4cc0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:41.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.574+0000 7f431b200640 1 --2- 172.21.15.12:0/2349613837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f431c077c40 0x7f431c1a52c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:41.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.574+0000 7f431b200640 1 --2- 172.21.15.12:0/2349613837 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f431c0772a0 0x7f431c19e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:41.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.574+0000 7f431b200640 1 --2- 172.21.15.12:0/2349613837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f431c076060 0x7f431c19dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:41.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.574+0000 7f431b200640 1 -- 172.21.15.12:0/2349613837 >> 172.21.15.12:0/2349613837 conn(0x7f431c0fec80 msgr2=0x7f431c1001d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:41.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.575+0000 7f431b200640 1 -- 172.21.15.12:0/2349613837 shutdown_connections 2024-03-23T22:58:41.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.575+0000 7f431b200640 1 -- 172.21.15.12:0/2349613837 wait complete. 2024-03-23T22:58:41.596 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T22:54:11.831010+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T22:58:41.596 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T22:58:41.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.827+0000 7f52f7a00640 1 -- 172.21.15.12:0/3459611268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52f01038e0 msgr2=0x7f52f0103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:41.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.827+0000 7f52f7a00640 1 --2- 172.21.15.12:0/3459611268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52f01038e0 0x7f52f0103d00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f52e0009ca0 tx=0x7f52e002c100 comp rx=0 tx=0).stop 2024-03-23T22:58:41.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.828+0000 7f52f7a00640 1 -- 172.21.15.12:0/3459611268 shutdown_connections 2024-03-23T22:58:41.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.828+0000 7f52f7a00640 1 --2- 172.21.15.12:0/3459611268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52f0105490 0x7f52f010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:41.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.828+0000 7f52f7a00640 1 --2- 172.21.15.12:0/3459611268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52f0104ad0 0x7f52f0104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:41.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.828+0000 7f52f7a00640 1 --2- 172.21.15.12:0/3459611268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52f01038e0 0x7f52f0103d00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:41.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.828+0000 7f52f7a00640 1 -- 172.21.15.12:0/3459611268 >> 172.21.15.12:0/3459611268 conn(0x7f52f00fee50 msgr2=0x7f52f0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:41.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.828+0000 7f52f7a00640 1 -- 172.21.15.12:0/3459611268 shutdown_connections 2024-03-23T22:58:41.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.828+0000 7f52f7a00640 1 -- 172.21.15.12:0/3459611268 wait complete. 2024-03-23T22:58:41.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.829+0000 7f52f7a00640 1 Processor -- start 2024-03-23T22:58:41.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.829+0000 7f52f7a00640 1 -- start start 2024-03-23T22:58:41.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.830+0000 7f52f7a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52f01038e0 0x7f52f0199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:41.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.830+0000 7f52f7a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52f0104ad0 0x7f52f019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:41.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.830+0000 7f52f7a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52f0105490 0x7f52f01a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:41.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.830+0000 7f52f7a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f010f550 con 0x7f52f01038e0 2024-03-23T22:58:41.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.830+0000 7f52f7a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f010f3d0 con 0x7f52f0104ad0 2024-03-23T22:58:41.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.830+0000 7f52f7a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f010f6d0 con 0x7f52f0105490 2024-03-23T22:58:41.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.830+0000 7f52f5000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52f01038e0 0x7f52f0199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:41.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.830+0000 7f52f5000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52f01038e0 0x7f52f0199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57168/0 (socket says 172.21.15.12:57168) 2024-03-23T22:58:41.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.830+0000 7f52f5000640 1 -- 172.21.15.12:0/1202357808 learned_addr learned my addr 172.21.15.12:0/1202357808 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:58:41.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.830+0000 7f52f5a00640 1 --2- 172.21.15.12:0/1202357808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52f0105490 0x7f52f01a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:41.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.830+0000 7f52efe00640 1 --2- 172.21.15.12:0/1202357808 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52f0104ad0 0x7f52f019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:41.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.830+0000 7f52f5000640 1 -- 172.21.15.12:0/1202357808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52f0105490 msgr2=0x7f52f01a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:41.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.830+0000 7f52f5000640 1 --2- 172.21.15.12:0/1202357808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52f0105490 0x7f52f01a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:41.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.830+0000 7f52f5000640 1 -- 172.21.15.12:0/1202357808 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52f0104ad0 msgr2=0x7f52f019a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:41.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.830+0000 7f52f5000640 1 --2- 172.21.15.12:0/1202357808 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52f0104ad0 0x7f52f019a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:41.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.830+0000 7f52f5000640 1 -- 172.21.15.12:0/1202357808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f01a1810 con 0x7f52f01038e0 2024-03-23T22:58:41.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.831+0000 7f52f5000640 1 --2- 172.21.15.12:0/1202357808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52f01038e0 0x7f52f0199bf0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f52e0000f80 tx=0x7f52e0035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:41.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.831+0000 7f52ed600640 1 -- 172.21.15.12:0/1202357808 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e0039040 con 0x7f52f01038e0 2024-03-23T22:58:41.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.831+0000 7f52ed600640 1 -- 172.21.15.12:0/1202357808 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52e0044070 con 0x7f52f01038e0 2024-03-23T22:58:41.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.831+0000 7f52ed600640 1 -- 172.21.15.12:0/1202357808 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e0004430 con 0x7f52f01038e0 2024-03-23T22:58:41.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.831+0000 7f52f7a00640 1 -- 172.21.15.12:0/1202357808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f01a1aa0 con 0x7f52f01038e0 2024-03-23T22:58:41.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.831+0000 7f52f7a00640 1 -- 172.21.15.12:0/1202357808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f01a1f60 con 0x7f52f01038e0 2024-03-23T22:58:41.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.833+0000 7f52f7a00640 1 -- 172.21.15.12:0/1202357808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f010ac00 con 0x7f52f01038e0 2024-03-23T22:58:41.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.835+0000 7f52ed600640 1 -- 172.21.15.12:0/1202357808 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f52e00045d0 con 0x7f52f01038e0 2024-03-23T22:58:41.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.837+0000 7f52ed600640 1 --2- 172.21.15.12:0/1202357808 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f52d40a2800 0x7f52d40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:41.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.837+0000 7f52ed600640 1 -- 172.21.15.12:0/1202357808 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f52e00f16f0 con 0x7f52f01038e0 2024-03-23T22:58:41.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.838+0000 7f52efe00640 1 --2- 172.21.15.12:0/1202357808 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f52d40a2800 0x7f52d40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:41.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.839+0000 7f52efe00640 1 --2- 172.21.15.12:0/1202357808 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f52d40a2800 0x7f52d40a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f52f019b100 tx=0x7f52d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:41.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:41.841+0000 7f52ed600640 1 -- 172.21.15.12:0/1202357808 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f52e00beb90 con 0x7f52f01038e0 2024-03-23T22:58:42.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.046+0000 7f52f7a00640 1 -- 172.21.15.12:0/1202357808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f52f00622a0 con 0x7f52f01038e0 2024-03-23T22:58:42.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.047+0000 7f52ed600640 1 -- 172.21.15.12:0/1202357808 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f52e00be490 con 0x7f52f01038e0 2024-03-23T22:58:42.046 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-23T22:58:42.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.049+0000 7f52f7a00640 1 -- 172.21.15.12:0/1202357808 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f52d40a2800 msgr2=0x7f52d40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:42.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.049+0000 7f52f7a00640 1 --2- 172.21.15.12:0/1202357808 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f52d40a2800 0x7f52d40a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f52f019b100 tx=0x7f52d8005f90 comp rx=0 tx=0).stop 2024-03-23T22:58:42.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.049+0000 7f52f7a00640 1 -- 172.21.15.12:0/1202357808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52f01038e0 msgr2=0x7f52f0199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:42.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.049+0000 7f52f7a00640 1 --2- 172.21.15.12:0/1202357808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52f01038e0 0x7f52f0199bf0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f52e0000f80 tx=0x7f52e0035d10 comp rx=0 tx=0).stop 2024-03-23T22:58:42.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.049+0000 7f52f7a00640 1 -- 172.21.15.12:0/1202357808 shutdown_connections 2024-03-23T22:58:42.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.049+0000 7f52f7a00640 1 --2- 172.21.15.12:0/1202357808 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f52d40a2800 0x7f52d40a4cc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:42.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.049+0000 7f52f7a00640 1 --2- 172.21.15.12:0/1202357808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52f0105490 0x7f52f01a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:42.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.049+0000 7f52f7a00640 1 --2- 172.21.15.12:0/1202357808 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52f0104ad0 0x7f52f019a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:42.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.049+0000 7f52f7a00640 1 --2- 172.21.15.12:0/1202357808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52f01038e0 0x7f52f0199bf0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:42.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.049+0000 7f52f7a00640 1 -- 172.21.15.12:0/1202357808 >> 172.21.15.12:0/1202357808 conn(0x7f52f00fee50 msgr2=0x7f52f0107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:42.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.050+0000 7f52f7a00640 1 -- 172.21.15.12:0/1202357808 shutdown_connections 2024-03-23T22:58:42.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.050+0000 7f52f7a00640 1 -- 172.21.15.12:0/1202357808 wait complete. 2024-03-23T22:58:42.069 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T22:58:42.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:41 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:42.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:41 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:58:42.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:41 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:42.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:41 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:58:42.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:41 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:58:42.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:41 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:42.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:41 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2349613837' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T22:58:42.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.298+0000 7f5ce9800640 1 -- 172.21.15.12:0/3303635923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ce4101500 msgr2=0x7f5ce4101920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:42.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.298+0000 7f5ce9800640 1 --2- 172.21.15.12:0/3303635923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ce4101500 0x7f5ce4101920 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4009ca0 tx=0x7f5cd402c100 comp rx=0 tx=0).stop 2024-03-23T22:58:42.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.298+0000 7f5ce9800640 1 -- 172.21.15.12:0/3303635923 shutdown_connections 2024-03-23T22:58:42.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.298+0000 7f5ce9800640 1 --2- 172.21.15.12:0/3303635923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ce41030b0 0x7f5ce4109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:42.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.298+0000 7f5ce9800640 1 --2- 172.21.15.12:0/3303635923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ce41026f0 0x7f5ce4102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:42.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.298+0000 7f5ce9800640 1 --2- 172.21.15.12:0/3303635923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ce4101500 0x7f5ce4101920 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:42.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.298+0000 7f5ce9800640 1 -- 172.21.15.12:0/3303635923 >> 172.21.15.12:0/3303635923 conn(0x7f5ce40fca70 msgr2=0x7f5ce40feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:42.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.298+0000 7f5ce9800640 1 -- 172.21.15.12:0/3303635923 shutdown_connections 2024-03-23T22:58:42.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.298+0000 7f5ce9800640 1 -- 172.21.15.12:0/3303635923 wait complete. 2024-03-23T22:58:42.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.299+0000 7f5ce9800640 1 Processor -- start 2024-03-23T22:58:42.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.299+0000 7f5ce9800640 1 -- start start 2024-03-23T22:58:42.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.300+0000 7f5ce9800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ce4101500 0x7f5ce41977b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:42.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.300+0000 7f5ce9800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ce41026f0 0x7f5ce4197cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:42.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.300+0000 7f5ce9800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ce41030b0 0x7f5ce419ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:42.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.300+0000 7f5ce9800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce410d190 con 0x7f5ce41030b0 2024-03-23T22:58:42.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.300+0000 7f5ce9800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce410d010 con 0x7f5ce4101500 2024-03-23T22:58:42.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.300+0000 7f5ce9800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce410d310 con 0x7f5ce41026f0 2024-03-23T22:58:42.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.300+0000 7f5ce8e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ce41030b0 0x7f5ce419ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:42.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.300+0000 7f5cdb400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ce41026f0 0x7f5ce4197cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:42.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.300+0000 7f5ce8e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ce41030b0 0x7f5ce419ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57188/0 (socket says 172.21.15.12:57188) 2024-03-23T22:58:42.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.300+0000 7f5cdb400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ce41026f0 0x7f5ce4197cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45946/0 (socket says 172.21.15.12:45946) 2024-03-23T22:58:42.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.300+0000 7f5ce8e00640 1 -- 172.21.15.12:0/2406906622 learned_addr learned my addr 172.21.15.12:0/2406906622 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:58:42.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.300+0000 7f5ce3e00640 1 --2- 172.21.15.12:0/2406906622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ce4101500 0x7f5ce41977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:42.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.300+0000 7f5ce8e00640 1 -- 172.21.15.12:0/2406906622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ce41026f0 msgr2=0x7f5ce4197cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:42.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.300+0000 7f5ce8e00640 1 --2- 172.21.15.12:0/2406906622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ce41026f0 0x7f5ce4197cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:42.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.300+0000 7f5ce8e00640 1 -- 172.21.15.12:0/2406906622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ce4101500 msgr2=0x7f5ce41977b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:42.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.300+0000 7f5ce8e00640 1 --2- 172.21.15.12:0/2406906622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ce4101500 0x7f5ce41977b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.300+0000 7f5ce8e00640 1 -- 172.21.15.12:0/2406906622 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce419f3d0 con 0x7f5ce41030b0 2024-03-23T22:58:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.300+0000 7f5ce3e00640 1 --2- 172.21.15.12:0/2406906622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ce4101500 0x7f5ce41977b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T22:58:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.300+0000 7f5cdb400640 1 --2- 172.21.15.12:0/2406906622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ce41026f0 0x7f5ce4197cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T22:58:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.301+0000 7f5ce8e00640 1 --2- 172.21.15.12:0/2406906622 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ce41030b0 0x7f5ce419ed60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc00e970 tx=0x7f5cdc00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.301+0000 7f5ce1600640 1 -- 172.21.15.12:0/2406906622 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc00ccc0 con 0x7f5ce41030b0 2024-03-23T22:58:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.301+0000 7f5ce1600640 1 -- 172.21.15.12:0/2406906622 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5cdc00ce60 con 0x7f5ce41030b0 2024-03-23T22:58:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.301+0000 7f5ce1600640 1 -- 172.21.15.12:0/2406906622 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc010490 con 0x7f5ce41030b0 2024-03-23T22:58:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.301+0000 7f5ce9800640 1 -- 172.21.15.12:0/2406906622 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce419f6c0 con 0x7f5ce41030b0 2024-03-23T22:58:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.301+0000 7f5ce9800640 1 -- 172.21.15.12:0/2406906622 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce419fc10 con 0x7f5ce41030b0 2024-03-23T22:58:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.302+0000 7f5ce9800640 1 -- 172.21.15.12:0/2406906622 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce4108820 con 0x7f5ce41030b0 2024-03-23T22:58:42.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:41 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:42.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:41 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:58:42.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:41 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:42.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:41 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:58:42.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:41 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:58:42.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:41 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:42.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:41 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2349613837' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T22:58:42.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.305+0000 7f5ce1600640 1 -- 172.21.15.12:0/2406906622 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5cdc005000 con 0x7f5ce41030b0 2024-03-23T22:58:42.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.306+0000 7f5ce1600640 1 --2- 172.21.15.12:0/2406906622 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5cc80a2800 0x7f5cc80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:42.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.306+0000 7f5ce1600640 1 -- 172.21.15.12:0/2406906622 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5cdc0cecb0 con 0x7f5ce41030b0 2024-03-23T22:58:42.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.306+0000 7f5ce3e00640 1 --2- 172.21.15.12:0/2406906622 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5cc80a2800 0x7f5cc80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:42.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.307+0000 7f5ce3e00640 1 --2- 172.21.15.12:0/2406906622 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5cc80a2800 0x7f5cc80a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5cd402c640 tx=0x7f5cd4032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:42.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.309+0000 7f5ce1600640 1 -- 172.21.15.12:0/2406906622 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5cdc053740 con 0x7f5ce41030b0 2024-03-23T22:58:42.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:41 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:42.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:41 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:58:42.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:41 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:42.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:41 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:58:42.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:41 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:58:42.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:41 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:42.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:41 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2349613837' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T22:58:42.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.554+0000 7f5ce9800640 1 -- 172.21.15.12:0/2406906622 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ce402d070 con 0x7f5ce41030b0 2024-03-23T22:58:42.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.555+0000 7f5ce1600640 1 -- 172.21.15.12:0/2406906622 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7f5cdc09c150 con 0x7f5ce41030b0 2024-03-23T22:58:42.554 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:58:42.554 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-23T22:58:42.555 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 1 2024-03-23T22:58:42.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.558+0000 7f5ce9800640 1 -- 172.21.15.12:0/2406906622 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5cc80a2800 msgr2=0x7f5cc80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:42.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.558+0000 7f5ce9800640 1 --2- 172.21.15.12:0/2406906622 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5cc80a2800 0x7f5cc80a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5cd402c640 tx=0x7f5cd4032590 comp rx=0 tx=0).stop 2024-03-23T22:58:42.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.558+0000 7f5ce9800640 1 -- 172.21.15.12:0/2406906622 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ce41030b0 msgr2=0x7f5ce419ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:42.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.558+0000 7f5ce9800640 1 --2- 172.21.15.12:0/2406906622 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ce41030b0 0x7f5ce419ed60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc00e970 tx=0x7f5cdc00ed70 comp rx=0 tx=0).stop 2024-03-23T22:58:42.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.558+0000 7f5ce9800640 1 -- 172.21.15.12:0/2406906622 shutdown_connections 2024-03-23T22:58:42.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.558+0000 7f5ce9800640 1 --2- 172.21.15.12:0/2406906622 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5cc80a2800 0x7f5cc80a4cc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:42.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.558+0000 7f5ce9800640 1 --2- 172.21.15.12:0/2406906622 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ce41030b0 0x7f5ce419ed60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:42.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.558+0000 7f5ce9800640 1 --2- 172.21.15.12:0/2406906622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ce41026f0 0x7f5ce4197cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:42.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.558+0000 7f5ce9800640 1 --2- 172.21.15.12:0/2406906622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ce4101500 0x7f5ce41977b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:42.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.558+0000 7f5ce9800640 1 -- 172.21.15.12:0/2406906622 >> 172.21.15.12:0/2406906622 conn(0x7f5ce40fca70 msgr2=0x7f5ce41059f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:42.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.558+0000 7f5ce9800640 1 -- 172.21.15.12:0/2406906622 shutdown_connections 2024-03-23T22:58:42.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:42.559+0000 7f5ce9800640 1 -- 172.21.15.12:0/2406906622 wait complete. 2024-03-23T22:58:43.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:42 smithi203 ceph-mon[36693]: Deploying daemon mds.l on smithi203 2024-03-23T22:58:43.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:42 smithi203 ceph-mon[36693]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:43.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:42 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1202357808' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T22:58:43.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:42 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2406906622' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:58:43.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:42 smithi012 ceph-mon[31695]: Deploying daemon mds.l on smithi203 2024-03-23T22:58:43.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:42 smithi012 ceph-mon[31695]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:43.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:42 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1202357808' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T22:58:43.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:42 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2406906622' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:58:43.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:42 smithi173 ceph-mon[36116]: Deploying daemon mds.l on smithi203 2024-03-23T22:58:43.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:42 smithi173 ceph-mon[36116]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:43.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:42 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1202357808' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T22:58:43.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:42 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2406906622' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:58:43.657 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 23 22:58:43 smithi203 systemd[1]: Starting Ceph mds.l for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T22:58:43.974 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 23 22:58:43 smithi203 podman[67761]: 2024-03-23 22:58:43.596137156 +0000 UTC m=+0.017415622 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-23T22:58:43.975 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 23 22:58:43 smithi203 podman[67761]: 2024-03-23 22:58:43.71448113 +0000 UTC m=+0.135759603 container create 15b8198a9551323425dec04f47fa99f35d5b7d17c1450aa3c2d90f61c4e25b5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-l, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-23T22:58:43.975 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 23 22:58:43 smithi203 podman[67761]: 2024-03-23 22:58:43.975899004 +0000 UTC m=+0.397177461 container init 15b8198a9551323425dec04f47fa99f35d5b7d17c1450aa3c2d90f61c4e25b5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-l, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-23T22:58:44.407 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 23 22:58:43 smithi203 podman[67761]: 2024-03-23 22:58:43.978830088 +0000 UTC m=+0.400108547 container start 15b8198a9551323425dec04f47fa99f35d5b7d17c1450aa3c2d90f61c4e25b5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-l, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-23T22:58:44.408 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 23 22:58:44 smithi203 ceph-mds[67780]: -- 172.21.15.203:0/2522108205 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5628fd9a9a40 con 0x5628fd97c800 2024-03-23T22:58:44.408 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 23 22:58:44 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-l[67776]: starting mds.l at 2024-03-23T22:58:44.408 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 23 22:58:44 smithi203 bash[67761]: 15b8198a9551323425dec04f47fa99f35d5b7d17c1450aa3c2d90f61c4e25b5b 2024-03-23T22:58:44.408 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 23 22:58:44 smithi203 systemd[1]: Started Ceph mds.l for 90f679ae-e967-11ee-95cd-87774f69a715. 2024-03-23T22:58:44.578 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T22:58:44.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.811+0000 7fb602000640 1 -- 172.21.15.12:0/966020581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5fc0740a0 msgr2=0x7fb5fc074500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:44.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.811+0000 7fb602000640 1 --2- 172.21.15.12:0/966020581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5fc0740a0 0x7fb5fc074500 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8009900 tx=0x7fb5e802bcd0 comp rx=0 tx=0).stop 2024-03-23T22:58:44.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.811+0000 7fb602000640 1 -- 172.21.15.12:0/966020581 shutdown_connections 2024-03-23T22:58:44.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.811+0000 7fb602000640 1 --2- 172.21.15.12:0/966020581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5fc109390 0x7fb5fc10b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:44.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.811+0000 7fb602000640 1 --2- 172.21.15.12:0/966020581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5fc0740a0 0x7fb5fc074500 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:44.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.811+0000 7fb602000640 1 --2- 172.21.15.12:0/966020581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5fc075680 0x7fb5fc073ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:44.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.811+0000 7fb602000640 1 -- 172.21.15.12:0/966020581 >> 172.21.15.12:0/966020581 conn(0x7fb5fc0fca70 msgr2=0x7fb5fc0feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:44.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.812+0000 7fb602000640 1 -- 172.21.15.12:0/966020581 shutdown_connections 2024-03-23T22:58:44.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.812+0000 7fb602000640 1 -- 172.21.15.12:0/966020581 wait complete. 2024-03-23T22:58:44.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.813+0000 7fb602000640 1 Processor -- start 2024-03-23T22:58:44.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.813+0000 7fb602000640 1 -- start start 2024-03-23T22:58:44.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.813+0000 7fb602000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5fc0740a0 0x7fb5fc1999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:44.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.813+0000 7fb602000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5fc075680 0x7fb5fc199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:44.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.813+0000 7fb602000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5fc109390 0x7fb5fc1a0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:44.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.813+0000 7fb602000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc10f360 con 0x7fb5fc075680 2024-03-23T22:58:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.813+0000 7fb602000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc10f1e0 con 0x7fb5fc0740a0 2024-03-23T22:58:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.813+0000 7fb602000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc10f4e0 con 0x7fb5fc109390 2024-03-23T22:58:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.814+0000 7fb600c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5fc0740a0 0x7fb5fc1999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.814+0000 7fb600c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5fc0740a0 0x7fb5fc1999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44084/0 (socket says 172.21.15.12:44084) 2024-03-23T22:58:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.814+0000 7fb600c00640 1 -- 172.21.15.12:0/1712585937 learned_addr learned my addr 172.21.15.12:0/1712585937 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:58:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.814+0000 7fb5fbe00640 1 --2- 172.21.15.12:0/1712585937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5fc075680 0x7fb5fc199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.814+0000 7fb601600640 1 --2- 172.21.15.12:0/1712585937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5fc109390 0x7fb5fc1a0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.814+0000 7fb600c00640 1 -- 172.21.15.12:0/1712585937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5fc109390 msgr2=0x7fb5fc1a0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.814+0000 7fb600c00640 1 --2- 172.21.15.12:0/1712585937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5fc109390 0x7fb5fc1a0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.814+0000 7fb600c00640 1 -- 172.21.15.12:0/1712585937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5fc075680 msgr2=0x7fb5fc199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.814+0000 7fb600c00640 1 --2- 172.21.15.12:0/1712585937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5fc075680 0x7fb5fc199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.814+0000 7fb600c00640 1 -- 172.21.15.12:0/1712585937 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5fc1a15c0 con 0x7fb5fc0740a0 2024-03-23T22:58:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.814+0000 7fb601600640 1 --2- 172.21.15.12:0/1712585937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5fc109390 0x7fb5fc1a0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T22:58:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.814+0000 7fb600c00640 1 --2- 172.21.15.12:0/1712585937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5fc0740a0 0x7fb5fc1999f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec00e9f0 tx=0x7fb5ec00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.815+0000 7fb5f9600640 1 -- 172.21.15.12:0/1712585937 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec00cd80 con 0x7fb5fc0740a0 2024-03-23T22:58:44.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.815+0000 7fb5f9600640 1 -- 172.21.15.12:0/1712585937 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5ec010910 con 0x7fb5fc0740a0 2024-03-23T22:58:44.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.815+0000 7fb602000640 1 -- 172.21.15.12:0/1712585937 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5fc1a18b0 con 0x7fb5fc0740a0 2024-03-23T22:58:44.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.815+0000 7fb5f9600640 1 -- 172.21.15.12:0/1712585937 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec018990 con 0x7fb5fc0740a0 2024-03-23T22:58:44.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.815+0000 7fb602000640 1 -- 172.21.15.12:0/1712585937 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5fc1a1dd0 con 0x7fb5fc0740a0 2024-03-23T22:58:44.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.816+0000 7fb602000640 1 -- 172.21.15.12:0/1712585937 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5fc108150 con 0x7fb5fc0740a0 2024-03-23T22:58:44.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.818+0000 7fb5f9600640 1 -- 172.21.15.12:0/1712585937 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb5ec010ac0 con 0x7fb5fc0740a0 2024-03-23T22:58:44.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.819+0000 7fb5f9600640 1 --2- 172.21.15.12:0/1712585937 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb5e00a2800 0x7fb5e00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:44.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.819+0000 7fb5f9600640 1 -- 172.21.15.12:0/1712585937 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb5ec0cea60 con 0x7fb5fc0740a0 2024-03-23T22:58:44.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.819+0000 7fb5fbe00640 1 --2- 172.21.15.12:0/1712585937 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb5e00a2800 0x7fb5e00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:44.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.820+0000 7fb5fbe00640 1 --2- 172.21.15.12:0/1712585937 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb5e00a2800 0x7fb5e00a4cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb5e802c850 tx=0x7fb5e8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:44.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:44.823+0000 7fb5f9600640 1 -- 172.21.15.12:0/1712585937 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb5ec09bf00 con 0x7fb5fc0740a0 2024-03-23T22:58:45.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:45.070+0000 7fb602000640 1 -- 172.21.15.12:0/1712585937 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5fc02d070 con 0x7fb5fc0740a0 2024-03-23T22:58:45.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:45.071+0000 7fb5f9600640 1 -- 172.21.15.12:0/1712585937 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1328 (secure 0 0 0) 0x7fb5fc02d070 con 0x7fb5fc0740a0 2024-03-23T22:58:45.070 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:58:45.070 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-23T22:58:45.070 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 2 2024-03-23T22:58:45.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:45.073+0000 7fb602000640 1 -- 172.21.15.12:0/1712585937 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb5e00a2800 msgr2=0x7fb5e00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:45.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:45.073+0000 7fb602000640 1 --2- 172.21.15.12:0/1712585937 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb5e00a2800 0x7fb5e00a4cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb5e802c850 tx=0x7fb5e8000bc0 comp rx=0 tx=0).stop 2024-03-23T22:58:45.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:45.073+0000 7fb602000640 1 -- 172.21.15.12:0/1712585937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5fc0740a0 msgr2=0x7fb5fc1999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:45.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:45.073+0000 7fb602000640 1 --2- 172.21.15.12:0/1712585937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5fc0740a0 0x7fb5fc1999f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec00e9f0 tx=0x7fb5ec00eeb0 comp rx=0 tx=0).stop 2024-03-23T22:58:45.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:45.074+0000 7fb602000640 1 -- 172.21.15.12:0/1712585937 shutdown_connections 2024-03-23T22:58:45.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:45.074+0000 7fb602000640 1 --2- 172.21.15.12:0/1712585937 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb5e00a2800 0x7fb5e00a4cc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:45.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:45.074+0000 7fb602000640 1 --2- 172.21.15.12:0/1712585937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5fc109390 0x7fb5fc1a0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:45.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:45.074+0000 7fb602000640 1 --2- 172.21.15.12:0/1712585937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5fc075680 0x7fb5fc199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:45.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:45.074+0000 7fb602000640 1 --2- 172.21.15.12:0/1712585937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5fc0740a0 0x7fb5fc1999f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:45.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:45.074+0000 7fb602000640 1 -- 172.21.15.12:0/1712585937 >> 172.21.15.12:0/1712585937 conn(0x7fb5fc0fca70 msgr2=0x7fb5fc105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:45.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:45.074+0000 7fb602000640 1 -- 172.21.15.12:0/1712585937 shutdown_connections 2024-03-23T22:58:45.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:45.074+0000 7fb602000640 1 -- 172.21.15.12:0/1712585937 wait complete. 2024-03-23T22:58:45.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:45 smithi173 ceph-mon[36116]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:45.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:45 smithi173 ceph-mon[36116]: mds.? [v2:172.21.15.203:6832/3254902484,v1:172.21.15.203:6833/3254902484] up:boot 2024-03-23T22:58:45.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:45 smithi173 ceph-mon[36116]: fsmap 1 up:standby 2024-03-23T22:58:45.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:45 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-23T22:58:45.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:45 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:45.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:45 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:45.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:45 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:45.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:45 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:58:45.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:45 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:58:45.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:45 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:45.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:45 smithi012 ceph-mon[31695]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:45.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:45 smithi012 ceph-mon[31695]: mds.? [v2:172.21.15.203:6832/3254902484,v1:172.21.15.203:6833/3254902484] up:boot 2024-03-23T22:58:45.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:45 smithi012 ceph-mon[31695]: fsmap 1 up:standby 2024-03-23T22:58:45.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:45 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-23T22:58:45.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:45 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:45.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:45 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:45.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:45 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:45.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:45 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:58:45.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:45 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:58:45.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:45 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:45.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:45 smithi203 ceph-mon[36693]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:45.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:45 smithi203 ceph-mon[36693]: mds.? [v2:172.21.15.203:6832/3254902484,v1:172.21.15.203:6833/3254902484] up:boot 2024-03-23T22:58:45.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:45 smithi203 ceph-mon[36693]: fsmap 1 up:standby 2024-03-23T22:58:45.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:45 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-23T22:58:45.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:45 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:45.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:45 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:45.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:45 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:45.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:45 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:58:45.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:45 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:58:45.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:45 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:46.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:46 smithi173 ceph-mon[36116]: Deploying daemon mds.k on smithi173 2024-03-23T22:58:46.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:46 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1712585937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:58:46.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:46 smithi012 ceph-mon[31695]: Deploying daemon mds.k on smithi173 2024-03-23T22:58:46.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:46 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1712585937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:58:46.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:46 smithi203 ceph-mon[36693]: Deploying daemon mds.k on smithi173 2024-03-23T22:58:46.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:46 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1712585937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:58:46.602 INFO:journalctl@ceph.mds.k.smithi173.stdout:Mar 23 22:58:46 smithi173 systemd[1]: Starting Ceph mds.k for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T22:58:46.868 INFO:journalctl@ceph.mds.k.smithi173.stdout:Mar 23 22:58:46 smithi173 podman[65574]: 2024-03-23 22:58:46.505976179 +0000 UTC m=+0.017644636 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-23T22:58:46.868 INFO:journalctl@ceph.mds.k.smithi173.stdout:Mar 23 22:58:46 smithi173 podman[65574]: 2024-03-23 22:58:46.616338451 +0000 UTC m=+0.128006940 container create 05355527fbafc9e98c2f88a6d1e3cd53f1370c212966536547f8f65f1aeaab69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-k, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-23T22:58:47.093 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T22:58:47.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.299+0000 7f4e44000640 1 -- 172.21.15.12:0/2192502892 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e3c076060 msgr2=0x7f4e3c076480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:47.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.299+0000 7f4e44000640 1 --2- 172.21.15.12:0/2192502892 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e3c076060 0x7f4e3c076480 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4e30009900 tx=0x7f4e3002bc30 comp rx=0 tx=0).stop 2024-03-23T22:58:47.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.300+0000 7f4e44000640 1 -- 172.21.15.12:0/2192502892 shutdown_connections 2024-03-23T22:58:47.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.300+0000 7f4e44000640 1 --2- 172.21.15.12:0/2192502892 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e3c077c40 0x7f4e3c0784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:47.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.300+0000 7f4e44000640 1 --2- 172.21.15.12:0/2192502892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e3c0772a0 0x7f4e3c077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:47.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.300+0000 7f4e44000640 1 --2- 172.21.15.12:0/2192502892 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e3c076060 0x7f4e3c076480 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:47.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.300+0000 7f4e44000640 1 -- 172.21.15.12:0/2192502892 >> 172.21.15.12:0/2192502892 conn(0x7f4e3c0fee50 msgr2=0x7f4e3c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:47.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.300+0000 7f4e44000640 1 -- 172.21.15.12:0/2192502892 shutdown_connections 2024-03-23T22:58:47.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.300+0000 7f4e44000640 1 -- 172.21.15.12:0/2192502892 wait complete. 2024-03-23T22:58:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.301+0000 7f4e44000640 1 Processor -- start 2024-03-23T22:58:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.301+0000 7f4e44000640 1 -- start start 2024-03-23T22:58:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.301+0000 7f4e44000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e3c076060 0x7f4e3c19dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.301+0000 7f4e44000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e3c0772a0 0x7f4e3c19e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.301+0000 7f4e44000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e3c077c40 0x7f4e3c1a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.301+0000 7f4e44000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3c1139b0 con 0x7f4e3c076060 2024-03-23T22:58:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.301+0000 7f4e44000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3c113830 con 0x7f4e3c0772a0 2024-03-23T22:58:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.301+0000 7f4e44000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3c113b30 con 0x7f4e3c077c40 2024-03-23T22:58:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.301+0000 7f4e41600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e3c076060 0x7f4e3c19dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.301+0000 7f4e41600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e3c076060 0x7f4e3c19dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35498/0 (socket says 172.21.15.12:35498) 2024-03-23T22:58:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.301+0000 7f4e41600640 1 -- 172.21.15.12:0/1790925960 learned_addr learned my addr 172.21.15.12:0/1790925960 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:58:47.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.302+0000 7f4e40c00640 1 --2- 172.21.15.12:0/1790925960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e3c0772a0 0x7f4e3c19e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:47.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.302+0000 7f4e42000640 1 --2- 172.21.15.12:0/1790925960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e3c077c40 0x7f4e3c1a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:47.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.302+0000 7f4e41600640 1 -- 172.21.15.12:0/1790925960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e3c077c40 msgr2=0x7f4e3c1a5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:47.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.302+0000 7f4e41600640 1 --2- 172.21.15.12:0/1790925960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e3c077c40 0x7f4e3c1a5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:47.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.302+0000 7f4e41600640 1 -- 172.21.15.12:0/1790925960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e3c0772a0 msgr2=0x7f4e3c19e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:47.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.302+0000 7f4e41600640 1 --2- 172.21.15.12:0/1790925960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e3c0772a0 0x7f4e3c19e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:47.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.302+0000 7f4e41600640 1 -- 172.21.15.12:0/1790925960 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3c1a5c00 con 0x7f4e3c076060 2024-03-23T22:58:47.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.302+0000 7f4e41600640 1 --2- 172.21.15.12:0/1790925960 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e3c076060 0x7f4e3c19dfe0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4e30005ab0 tx=0x7f4e3002c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:47.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.302+0000 7f4e3a000640 1 -- 172.21.15.12:0/1790925960 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e30043020 con 0x7f4e3c076060 2024-03-23T22:58:47.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.303+0000 7f4e3a000640 1 -- 172.21.15.12:0/1790925960 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e300054e0 con 0x7f4e3c076060 2024-03-23T22:58:47.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.303+0000 7f4e44000640 1 -- 172.21.15.12:0/1790925960 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e3c1a5e90 con 0x7f4e3c076060 2024-03-23T22:58:47.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.303+0000 7f4e3a000640 1 -- 172.21.15.12:0/1790925960 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e30003c70 con 0x7f4e3c076060 2024-03-23T22:58:47.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.303+0000 7f4e44000640 1 -- 172.21.15.12:0/1790925960 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e3c1a62f0 con 0x7f4e3c076060 2024-03-23T22:58:47.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.304+0000 7f4e44000640 1 -- 172.21.15.12:0/1790925960 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e3c0688e0 con 0x7f4e3c076060 2024-03-23T22:58:47.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:47 smithi012 ceph-mon[31695]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:47.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:47 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:47.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.306+0000 7f4e3a000640 1 -- 172.21.15.12:0/1790925960 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4e30046020 con 0x7f4e3c076060 2024-03-23T22:58:47.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.308+0000 7f4e3a000640 1 --2- 172.21.15.12:0/1790925960 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4e240a2800 0x7f4e240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:47.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.308+0000 7f4e3a000640 1 -- 172.21.15.12:0/1790925960 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4e300f0aa0 con 0x7f4e3c076060 2024-03-23T22:58:47.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.308+0000 7f4e40c00640 1 --2- 172.21.15.12:0/1790925960 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4e240a2800 0x7f4e240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:47.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.309+0000 7f4e40c00640 1 --2- 172.21.15.12:0/1790925960 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4e240a2800 0x7f4e240a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c19f4f0 tx=0x7f4e2c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:47.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.311+0000 7f4e3a000640 1 -- 172.21.15.12:0/1790925960 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4e3004d020 con 0x7f4e3c076060 2024-03-23T22:58:47.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:47 smithi173 ceph-mon[36116]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:47.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:47 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:47.368 INFO:journalctl@ceph.mds.k.smithi173.stdout:Mar 23 22:58:46 smithi173 podman[65574]: 2024-03-23 22:58:46.885692509 +0000 UTC m=+0.397360972 container init 05355527fbafc9e98c2f88a6d1e3cd53f1370c212966536547f8f65f1aeaab69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-k, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-23T22:58:47.368 INFO:journalctl@ceph.mds.k.smithi173.stdout:Mar 23 22:58:46 smithi173 podman[65574]: 2024-03-23 22:58:46.888340485 +0000 UTC m=+0.400008930 container start 05355527fbafc9e98c2f88a6d1e3cd53f1370c212966536547f8f65f1aeaab69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-k, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-23T22:58:47.368 INFO:journalctl@ceph.mds.k.smithi173.stdout:Mar 23 22:58:46 smithi173 ceph-mds[65593]: -- 172.21.15.173:0/3982605762 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5603c025ba40 con 0x5603c022e800 2024-03-23T22:58:47.368 INFO:journalctl@ceph.mds.k.smithi173.stdout:Mar 23 22:58:46 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-k[65589]: starting mds.k at 2024-03-23T22:58:47.368 INFO:journalctl@ceph.mds.k.smithi173.stdout:Mar 23 22:58:46 smithi173 bash[65574]: 05355527fbafc9e98c2f88a6d1e3cd53f1370c212966536547f8f65f1aeaab69 2024-03-23T22:58:47.368 INFO:journalctl@ceph.mds.k.smithi173.stdout:Mar 23 22:58:46 smithi173 systemd[1]: Started Ceph mds.k for 90f679ae-e967-11ee-95cd-87774f69a715. 2024-03-23T22:58:47.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:47 smithi203 ceph-mon[36693]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:47.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:47 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:47.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.555+0000 7f4e44000640 1 -- 172.21.15.12:0/1790925960 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e3c02d140 con 0x7f4e3c076060 2024-03-23T22:58:47.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.556+0000 7f4e3a000640 1 -- 172.21.15.12:0/1790925960 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2122 (secure 0 0 0) 0x7f4e3c02d140 con 0x7f4e3c076060 2024-03-23T22:58:47.556 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:58:47.556 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-23T22:58:47.556 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 3 2024-03-23T22:58:47.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.558+0000 7f4e44000640 1 -- 172.21.15.12:0/1790925960 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4e240a2800 msgr2=0x7f4e240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:47.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.558+0000 7f4e44000640 1 --2- 172.21.15.12:0/1790925960 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4e240a2800 0x7f4e240a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c19f4f0 tx=0x7f4e2c00b410 comp rx=0 tx=0).stop 2024-03-23T22:58:47.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.558+0000 7f4e44000640 1 -- 172.21.15.12:0/1790925960 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e3c076060 msgr2=0x7f4e3c19dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:47.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.558+0000 7f4e44000640 1 --2- 172.21.15.12:0/1790925960 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e3c076060 0x7f4e3c19dfe0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4e30005ab0 tx=0x7f4e3002c6c0 comp rx=0 tx=0).stop 2024-03-23T22:58:47.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.559+0000 7f4e44000640 1 -- 172.21.15.12:0/1790925960 shutdown_connections 2024-03-23T22:58:47.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.559+0000 7f4e44000640 1 --2- 172.21.15.12:0/1790925960 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4e240a2800 0x7f4e240a4cc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:47.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.559+0000 7f4e44000640 1 --2- 172.21.15.12:0/1790925960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e3c077c40 0x7f4e3c1a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:47.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.559+0000 7f4e44000640 1 --2- 172.21.15.12:0/1790925960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e3c0772a0 0x7f4e3c19e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:47.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.559+0000 7f4e44000640 1 --2- 172.21.15.12:0/1790925960 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e3c076060 0x7f4e3c19dfe0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:47.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.559+0000 7f4e44000640 1 -- 172.21.15.12:0/1790925960 >> 172.21.15.12:0/1790925960 conn(0x7f4e3c0fee50 msgr2=0x7f4e3c100560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:47.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.559+0000 7f4e44000640 1 -- 172.21.15.12:0/1790925960 shutdown_connections 2024-03-23T22:58:47.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:47.559+0000 7f4e44000640 1 -- 172.21.15.12:0/1790925960 wait complete. 2024-03-23T22:58:48.286 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:48 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:48.286 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:48 smithi173 ceph-mon[36116]: mds.? [v2:172.21.15.173:6832/4291212010,v1:172.21.15.173:6833/4291212010] up:boot 2024-03-23T22:58:48.286 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:48 smithi173 ceph-mon[36116]: fsmap 2 up:standby 2024-03-23T22:58:48.286 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:48 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-23T22:58:48.286 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:48 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:48.286 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:48 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:58:48.287 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:48 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:58:48.287 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:48 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:48.287 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:48 smithi173 ceph-mon[36116]: Deploying daemon mds.j on smithi012 2024-03-23T22:58:48.287 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:48 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1790925960' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:58:48.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:48 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:48.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:48 smithi012 ceph-mon[31695]: mds.? [v2:172.21.15.173:6832/4291212010,v1:172.21.15.173:6833/4291212010] up:boot 2024-03-23T22:58:48.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:48 smithi012 ceph-mon[31695]: fsmap 2 up:standby 2024-03-23T22:58:48.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:48 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-23T22:58:48.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:48 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:48.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:48 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:58:48.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:48 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:58:48.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:48 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:48.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:48 smithi012 ceph-mon[31695]: Deploying daemon mds.j on smithi012 2024-03-23T22:58:48.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:48 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1790925960' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:58:48.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:48 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:48.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:48 smithi203 ceph-mon[36693]: mds.? [v2:172.21.15.173:6832/4291212010,v1:172.21.15.173:6833/4291212010] up:boot 2024-03-23T22:58:48.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:48 smithi203 ceph-mon[36693]: fsmap 2 up:standby 2024-03-23T22:58:48.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:48 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-23T22:58:48.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:48 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:48.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:48 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:58:48.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:48 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:58:48.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:48 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:48.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:48 smithi203 ceph-mon[36693]: Deploying daemon mds.j on smithi012 2024-03-23T22:58:48.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:48 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1790925960' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:58:49.247 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:49 smithi012 ceph-mon[31695]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:49.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:49 smithi173 ceph-mon[36116]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:49.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:49 smithi203 ceph-mon[36693]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:49.577 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T22:58:49.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.855+0000 7f7e90200640 1 -- 172.21.15.12:0/455224992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e88109000 msgr2=0x7f7e8810f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:49.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.855+0000 7f7e90200640 1 --2- 172.21.15.12:0/455224992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e88109000 0x7f7e8810f9e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7e74009900 tx=0x7f7e7402bc30 comp rx=0 tx=0).stop 2024-03-23T22:58:49.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.855+0000 7f7e90200640 1 -- 172.21.15.12:0/455224992 shutdown_connections 2024-03-23T22:58:49.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.855+0000 7f7e90200640 1 --2- 172.21.15.12:0/455224992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e8810ff20 0x7f7e88112310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:49.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.855+0000 7f7e90200640 1 --2- 172.21.15.12:0/455224992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e88109000 0x7f7e8810f9e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:49.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.855+0000 7f7e90200640 1 --2- 172.21.15.12:0/455224992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e881086a0 0x7f7e88108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:49.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.855+0000 7f7e90200640 1 -- 172.21.15.12:0/455224992 >> 172.21.15.12:0/455224992 conn(0x7f7e8806e8b0 msgr2=0x7f7e88070cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:49.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.855+0000 7f7e90200640 1 -- 172.21.15.12:0/455224992 shutdown_connections 2024-03-23T22:58:49.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.855+0000 7f7e90200640 1 -- 172.21.15.12:0/455224992 wait complete. 2024-03-23T22:58:49.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.856+0000 7f7e90200640 1 Processor -- start 2024-03-23T22:58:49.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.856+0000 7f7e90200640 1 -- start start 2024-03-23T22:58:49.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.856+0000 7f7e90200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e881086a0 0x7f7e8810b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:49.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.856+0000 7f7e90200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e88109000 0x7f7e88109970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:49.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.857+0000 7f7e90200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e8810ff20 0x7f7e88109eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:49.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.857+0000 7f7e90200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e88115b50 con 0x7f7e881086a0 2024-03-23T22:58:49.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.857+0000 7f7e90200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e881159d0 con 0x7f7e88109000 2024-03-23T22:58:49.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.857+0000 7f7e90200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e88115cd0 con 0x7f7e8810ff20 2024-03-23T22:58:49.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.857+0000 7f7e8d800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e881086a0 0x7f7e8810b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:49.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.857+0000 7f7e8d800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e881086a0 0x7f7e8810b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35510/0 (socket says 172.21.15.12:35510) 2024-03-23T22:58:49.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.857+0000 7f7e8d800640 1 -- 172.21.15.12:0/1416149881 learned_addr learned my addr 172.21.15.12:0/1416149881 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:58:49.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.857+0000 7f7e8ce00640 1 --2- 172.21.15.12:0/1416149881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e88109000 0x7f7e88109970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:49.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.857+0000 7f7e8e200640 1 --2- 172.21.15.12:0/1416149881 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e8810ff20 0x7f7e88109eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:49.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.857+0000 7f7e8d800640 1 -- 172.21.15.12:0/1416149881 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e8810ff20 msgr2=0x7f7e88109eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:49.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.857+0000 7f7e8d800640 1 --2- 172.21.15.12:0/1416149881 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e8810ff20 0x7f7e88109eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:49.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.857+0000 7f7e8d800640 1 -- 172.21.15.12:0/1416149881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e88109000 msgr2=0x7f7e88109970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:49.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.857+0000 7f7e8d800640 1 --2- 172.21.15.12:0/1416149881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e88109000 0x7f7e88109970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:49.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.857+0000 7f7e8d800640 1 -- 172.21.15.12:0/1416149881 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8810a740 con 0x7f7e881086a0 2024-03-23T22:58:49.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.857+0000 7f7e8ce00640 1 --2- 172.21.15.12:0/1416149881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e88109000 0x7f7e88109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T22:58:49.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.858+0000 7f7e8d800640 1 --2- 172.21.15.12:0/1416149881 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e881086a0 0x7f7e8810b330 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c00e9f0 tx=0x7f7e7c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:49.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.858+0000 7f7e86000640 1 -- 172.21.15.12:0/1416149881 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c00cd80 con 0x7f7e881086a0 2024-03-23T22:58:49.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.858+0000 7f7e86000640 1 -- 172.21.15.12:0/1416149881 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7e7c010910 con 0x7f7e881086a0 2024-03-23T22:58:49.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.858+0000 7f7e86000640 1 -- 172.21.15.12:0/1416149881 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c018990 con 0x7f7e881086a0 2024-03-23T22:58:49.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.858+0000 7f7e90200640 1 -- 172.21.15.12:0/1416149881 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e881aa2b0 con 0x7f7e881086a0 2024-03-23T22:58:49.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.858+0000 7f7e90200640 1 -- 172.21.15.12:0/1416149881 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e881aa780 con 0x7f7e881086a0 2024-03-23T22:58:49.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.860+0000 7f7e90200640 1 -- 172.21.15.12:0/1416149881 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e8810e7f0 con 0x7f7e881086a0 2024-03-23T22:58:49.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.862+0000 7f7e86000640 1 -- 172.21.15.12:0/1416149881 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7e7c010ac0 con 0x7f7e881086a0 2024-03-23T22:58:49.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.864+0000 7f7e86000640 1 --2- 172.21.15.12:0/1416149881 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7e700a2880 0x7f7e700a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:49.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.864+0000 7f7e86000640 1 -- 172.21.15.12:0/1416149881 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7e7c0ce920 con 0x7f7e881086a0 2024-03-23T22:58:49.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.864+0000 7f7e8ce00640 1 --2- 172.21.15.12:0/1416149881 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7e700a2880 0x7f7e700a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:49.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.865+0000 7f7e8ce00640 1 --2- 172.21.15.12:0/1416149881 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7e700a2880 0x7f7e700a4d40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7e7402c240 tx=0x7f7e7402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:49.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:49.867+0000 7f7e86000640 1 -- 172.21.15.12:0/1416149881 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7e7c09b770 con 0x7f7e881086a0 2024-03-23T22:58:50.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:50.127+0000 7f7e90200640 1 -- 172.21.15.12:0/1416149881 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e8802d070 con 0x7f7e881086a0 2024-03-23T22:58:50.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:50.127+0000 7f7e86000640 1 -- 172.21.15.12:0/1416149881 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2122 (secure 0 0 0) 0x7f7e7c09f810 con 0x7f7e881086a0 2024-03-23T22:58:50.126 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:58:50.126 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-23T22:58:50.127 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 3 2024-03-23T22:58:50.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:50.130+0000 7f7e90200640 1 -- 172.21.15.12:0/1416149881 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7e700a2880 msgr2=0x7f7e700a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:50.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:50.130+0000 7f7e90200640 1 --2- 172.21.15.12:0/1416149881 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7e700a2880 0x7f7e700a4d40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7e7402c240 tx=0x7f7e7402df90 comp rx=0 tx=0).stop 2024-03-23T22:58:50.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:50.130+0000 7f7e90200640 1 -- 172.21.15.12:0/1416149881 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e881086a0 msgr2=0x7f7e8810b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:50.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:50.130+0000 7f7e90200640 1 --2- 172.21.15.12:0/1416149881 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e881086a0 0x7f7e8810b330 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c00e9f0 tx=0x7f7e7c00eeb0 comp rx=0 tx=0).stop 2024-03-23T22:58:50.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:50.130+0000 7f7e90200640 1 -- 172.21.15.12:0/1416149881 shutdown_connections 2024-03-23T22:58:50.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:50.130+0000 7f7e90200640 1 --2- 172.21.15.12:0/1416149881 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7e700a2880 0x7f7e700a4d40 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:50.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:50.130+0000 7f7e90200640 1 --2- 172.21.15.12:0/1416149881 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e8810ff20 0x7f7e88109eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:50.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:50.130+0000 7f7e90200640 1 --2- 172.21.15.12:0/1416149881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e88109000 0x7f7e88109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:50.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:50.130+0000 7f7e90200640 1 --2- 172.21.15.12:0/1416149881 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e881086a0 0x7f7e8810b330 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:50.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:50.130+0000 7f7e90200640 1 -- 172.21.15.12:0/1416149881 >> 172.21.15.12:0/1416149881 conn(0x7f7e8806e8b0 msgr2=0x7f7e8810b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:50.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:50.130+0000 7f7e90200640 1 -- 172.21.15.12:0/1416149881 shutdown_connections 2024-03-23T22:58:50.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:50.130+0000 7f7e90200640 1 -- 172.21.15.12:0/1416149881 wait complete. 2024-03-23T22:58:50.781 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 23 22:58:50 smithi012 podman[71995]: 2024-03-23 22:58:50.477768069 +0000 UTC m=+0.165994351 container create ff60b87f6b4183a1b8d352dc4aa688ea51122a4ed0dc3d8e026d401ef3f1b19e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-23T22:58:50.781 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 23 22:58:50 smithi012 podman[71995]: 2024-03-23 22:58:50.781863059 +0000 UTC m=+0.470089348 container init ff60b87f6b4183a1b8d352dc4aa688ea51122a4ed0dc3d8e026d401ef3f1b19e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-j, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-23T22:58:51.054 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 23 22:58:50 smithi012 podman[71995]: 2024-03-23 22:58:50.792720601 +0000 UTC m=+0.480946890 container start ff60b87f6b4183a1b8d352dc4aa688ea51122a4ed0dc3d8e026d401ef3f1b19e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-23T22:58:51.054 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 23 22:58:50 smithi012 ceph-mds[72014]: -- 172.21.15.12:0/3142644793 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55eecf877a40 con 0x55eecf84a800 2024-03-23T22:58:51.054 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 23 22:58:50 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-j[72010]: starting mds.j at 2024-03-23T22:58:51.054 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 23 22:58:50 smithi012 bash[71995]: ff60b87f6b4183a1b8d352dc4aa688ea51122a4ed0dc3d8e026d401ef3f1b19e 2024-03-23T22:58:51.054 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 23 22:58:50 smithi012 systemd[1]: Started Ceph mds.j for 90f679ae-e967-11ee-95cd-87774f69a715. 2024-03-23T22:58:51.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:51 smithi173 ceph-mon[36116]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:51.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:51 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1416149881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:58:51.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:51 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:51.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:51 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:51.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:51 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:51.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:51 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:58:51.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:51 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:58:51.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:51 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:51.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:51 smithi203 ceph-mon[36693]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:51.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:51 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1416149881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:58:51.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:51 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:51.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:51 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:51.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:51 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:51.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:51 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:58:51.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:51 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:58:51.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:51 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:51.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:51 smithi012 ceph-mon[31695]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:51.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:51 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1416149881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:58:51.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:51 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:51.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:51 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:51.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:51 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:51.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:51 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:58:51.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:51 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:58:51.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:51 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:52.150 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T22:58:52.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:52 smithi173 ceph-mon[36116]: Deploying daemon mds.d on smithi012 2024-03-23T22:58:52.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:52 smithi173 ceph-mon[36116]: mds.? [v2:172.21.15.12:6834/2201661812,v1:172.21.15.12:6835/2201661812] up:boot 2024-03-23T22:58:52.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:52 smithi173 ceph-mon[36116]: fsmap 3 up:standby 2024-03-23T22:58:52.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:52 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-23T22:58:52.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.383+0000 7f9bf2200640 1 -- 172.21.15.12:0/1071049628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bec074100 msgr2=0x7f9bec074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:52.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.383+0000 7f9bf2200640 1 --2- 172.21.15.12:0/1071049628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bec074100 0x7f9bec074580 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4009900 tx=0x7f9bd402bc50 comp rx=0 tx=0).stop 2024-03-23T22:58:52.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.384+0000 7f9bf2200640 1 -- 172.21.15.12:0/1071049628 shutdown_connections 2024-03-23T22:58:52.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.384+0000 7f9bf2200640 1 --2- 172.21.15.12:0/1071049628 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bec1093d0 0x7f9bec10b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:52.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.384+0000 7f9bf2200640 1 --2- 172.21.15.12:0/1071049628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bec074100 0x7f9bec074580 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:52.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.384+0000 7f9bf2200640 1 --2- 172.21.15.12:0/1071049628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bec0756e0 0x7f9bec073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:52.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.384+0000 7f9bf2200640 1 -- 172.21.15.12:0/1071049628 >> 172.21.15.12:0/1071049628 conn(0x7f9bec0fca80 msgr2=0x7f9bec0feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:52.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.384+0000 7f9bf2200640 1 -- 172.21.15.12:0/1071049628 shutdown_connections 2024-03-23T22:58:52.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.384+0000 7f9bf2200640 1 -- 172.21.15.12:0/1071049628 wait complete. 2024-03-23T22:58:52.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.385+0000 7f9bf2200640 1 Processor -- start 2024-03-23T22:58:52.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.385+0000 7f9bf2200640 1 -- start start 2024-03-23T22:58:52.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.385+0000 7f9bf2200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bec074100 0x7f9bec1999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:52.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.385+0000 7f9bf2200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bec0756e0 0x7f9bec199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:52.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.385+0000 7f9bf2200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bec1093d0 0x7f9bec1a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:52.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.385+0000 7f9bf2200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec10f3a0 con 0x7f9bec0756e0 2024-03-23T22:58:52.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.385+0000 7f9bf2200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec10f220 con 0x7f9bec1093d0 2024-03-23T22:58:52.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.385+0000 7f9bf2200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec10f520 con 0x7f9bec074100 2024-03-23T22:58:52.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.385+0000 7f9bf0e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bec074100 0x7f9bec1999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:52.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.385+0000 7f9bebe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bec0756e0 0x7f9bec199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:52.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.385+0000 7f9bf0e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bec074100 0x7f9bec1999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:38042/0 (socket says 172.21.15.12:38042) 2024-03-23T22:58:52.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.385+0000 7f9bf0e00640 1 -- 172.21.15.12:0/3593461636 learned_addr learned my addr 172.21.15.12:0/3593461636 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:58:52.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.386+0000 7f9bf1800640 1 --2- 172.21.15.12:0/3593461636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bec1093d0 0x7f9bec1a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:52.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.386+0000 7f9bf0e00640 1 -- 172.21.15.12:0/3593461636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bec1093d0 msgr2=0x7f9bec1a0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:52.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.386+0000 7f9bf0e00640 1 --2- 172.21.15.12:0/3593461636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bec1093d0 0x7f9bec1a0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:52.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.386+0000 7f9bf0e00640 1 -- 172.21.15.12:0/3593461636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bec0756e0 msgr2=0x7f9bec199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:52.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.386+0000 7f9bf0e00640 1 --2- 172.21.15.12:0/3593461636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bec0756e0 0x7f9bec199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:52.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.386+0000 7f9bf0e00640 1 -- 172.21.15.12:0/3593461636 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bec1a1610 con 0x7f9bec074100 2024-03-23T22:58:52.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.386+0000 7f9bf0e00640 1 --2- 172.21.15.12:0/3593461636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bec074100 0x7f9bec1999f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9be400db80 tx=0x7f9be400b630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:52.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.386+0000 7f9be9600640 1 -- 172.21.15.12:0/3593461636 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be4017070 con 0x7f9bec074100 2024-03-23T22:58:52.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.386+0000 7f9be9600640 1 -- 172.21.15.12:0/3593461636 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9be4003b70 con 0x7f9bec074100 2024-03-23T22:58:52.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.386+0000 7f9be9600640 1 -- 172.21.15.12:0/3593461636 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be40106c0 con 0x7f9bec074100 2024-03-23T22:58:52.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.386+0000 7f9bf2200640 1 -- 172.21.15.12:0/3593461636 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bec1a1900 con 0x7f9bec074100 2024-03-23T22:58:52.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.387+0000 7f9bf2200640 1 -- 172.21.15.12:0/3593461636 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bec1a1e20 con 0x7f9bec074100 2024-03-23T22:58:52.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.388+0000 7f9bf2200640 1 -- 172.21.15.12:0/3593461636 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bec108100 con 0x7f9bec074100 2024-03-23T22:58:52.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.390+0000 7f9be9600640 1 -- 172.21.15.12:0/3593461636 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9be4043b90 con 0x7f9bec074100 2024-03-23T22:58:52.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.391+0000 7f9be9600640 1 --2- 172.21.15.12:0/3593461636 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9bd00a2800 0x7f9bd00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:52.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.391+0000 7f9be9600640 1 -- 172.21.15.12:0/3593461636 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9be40ce4c0 con 0x7f9bec074100 2024-03-23T22:58:52.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.391+0000 7f9bebe00640 1 --2- 172.21.15.12:0/3593461636 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9bd00a2800 0x7f9bd00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:52.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.392+0000 7f9bebe00640 1 --2- 172.21.15.12:0/3593461636 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9bd00a2800 0x7f9bd00a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9bd402c260 tx=0x7f9bd4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:52.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.394+0000 7f9be9600640 1 -- 172.21.15.12:0/3593461636 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9be4020070 con 0x7f9bec074100 2024-03-23T22:58:52.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:52 smithi203 ceph-mon[36693]: Deploying daemon mds.d on smithi012 2024-03-23T22:58:52.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:52 smithi203 ceph-mon[36693]: mds.? [v2:172.21.15.12:6834/2201661812,v1:172.21.15.12:6835/2201661812] up:boot 2024-03-23T22:58:52.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:52 smithi203 ceph-mon[36693]: fsmap 3 up:standby 2024-03-23T22:58:52.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:52 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-23T22:58:52.451 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:52 smithi012 ceph-mon[31695]: Deploying daemon mds.d on smithi012 2024-03-23T22:58:52.451 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:52 smithi012 ceph-mon[31695]: mds.? [v2:172.21.15.12:6834/2201661812,v1:172.21.15.12:6835/2201661812] up:boot 2024-03-23T22:58:52.451 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:52 smithi012 ceph-mon[31695]: fsmap 3 up:standby 2024-03-23T22:58:52.451 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:52 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-23T22:58:52.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.642+0000 7f9bf2200640 1 -- 172.21.15.12:0/3593461636 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bec02d070 con 0x7f9bec074100 2024-03-23T22:58:52.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.643+0000 7f9be9600640 1 -- 172.21.15.12:0/3593461636 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2913 (secure 0 0 0) 0x7f9be409b960 con 0x7f9bec074100 2024-03-23T22:58:52.642 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:58:52.643 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-23T22:58:52.643 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 4 2024-03-23T22:58:52.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.647+0000 7f9bf2200640 1 -- 172.21.15.12:0/3593461636 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9bd00a2800 msgr2=0x7f9bd00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:52.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.647+0000 7f9bf2200640 1 --2- 172.21.15.12:0/3593461636 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9bd00a2800 0x7f9bd00a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9bd402c260 tx=0x7f9bd4000bc0 comp rx=0 tx=0).stop 2024-03-23T22:58:52.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.647+0000 7f9bf2200640 1 -- 172.21.15.12:0/3593461636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bec074100 msgr2=0x7f9bec1999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:52.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.647+0000 7f9bf2200640 1 --2- 172.21.15.12:0/3593461636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bec074100 0x7f9bec1999f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9be400db80 tx=0x7f9be400b630 comp rx=0 tx=0).stop 2024-03-23T22:58:52.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.647+0000 7f9bf2200640 1 -- 172.21.15.12:0/3593461636 shutdown_connections 2024-03-23T22:58:52.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.647+0000 7f9bf2200640 1 --2- 172.21.15.12:0/3593461636 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9bd00a2800 0x7f9bd00a4cc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:52.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.647+0000 7f9bf2200640 1 --2- 172.21.15.12:0/3593461636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bec1093d0 0x7f9bec1a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:52.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.647+0000 7f9bf2200640 1 --2- 172.21.15.12:0/3593461636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bec0756e0 0x7f9bec199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:52.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.647+0000 7f9bf2200640 1 --2- 172.21.15.12:0/3593461636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bec074100 0x7f9bec1999f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:52.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.647+0000 7f9bf2200640 1 -- 172.21.15.12:0/3593461636 >> 172.21.15.12:0/3593461636 conn(0x7f9bec0fca80 msgr2=0x7f9bec1052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:52.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.647+0000 7f9bf2200640 1 -- 172.21.15.12:0/3593461636 shutdown_connections 2024-03-23T22:58:52.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:52.647+0000 7f9bf2200640 1 -- 172.21.15.12:0/3593461636 wait complete. 2024-03-23T22:58:53.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:53 smithi173 ceph-mon[36116]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:53.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:53 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3593461636' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:58:53.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:53 smithi203 ceph-mon[36693]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:53.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:53 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3593461636' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:58:53.463 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:53 smithi012 ceph-mon[31695]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:53.463 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:53 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3593461636' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:58:54.667 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T22:58:54.702 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 23 22:58:54 smithi012 systemd[1]: Starting Ceph mds.d for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T22:58:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.898+0000 7f9619000640 1 -- 172.21.15.12:0/2804448397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96141038e0 msgr2=0x7f9614103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.898+0000 7f9619000640 1 --2- 172.21.15.12:0/2804448397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96141038e0 0x7f9614103d00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9604009be0 tx=0x7f960402bef0 comp rx=0 tx=0).stop 2024-03-23T22:58:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.898+0000 7f9619000640 1 -- 172.21.15.12:0/2804448397 shutdown_connections 2024-03-23T22:58:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.898+0000 7f9619000640 1 --2- 172.21.15.12:0/2804448397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9614105490 0x7f961410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.898+0000 7f9619000640 1 --2- 172.21.15.12:0/2804448397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9614104ad0 0x7f9614104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.898+0000 7f9619000640 1 --2- 172.21.15.12:0/2804448397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96141038e0 0x7f9614103d00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.898+0000 7f9619000640 1 -- 172.21.15.12:0/2804448397 >> 172.21.15.12:0/2804448397 conn(0x7f96140fee90 msgr2=0x7f96141012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.899+0000 7f9619000640 1 -- 172.21.15.12:0/2804448397 shutdown_connections 2024-03-23T22:58:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.899+0000 7f9619000640 1 -- 172.21.15.12:0/2804448397 wait complete. 2024-03-23T22:58:54.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.900+0000 7f9619000640 1 Processor -- start 2024-03-23T22:58:54.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.900+0000 7f9619000640 1 -- start start 2024-03-23T22:58:54.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.901+0000 7f9619000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96141038e0 0x7f9614199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:54.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.901+0000 7f9619000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9614104ad0 0x7f961419a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:54.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.901+0000 7f9619000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9614105490 0x7f96141a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:54.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.901+0000 7f9619000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961410f5a0 con 0x7f9614105490 2024-03-23T22:58:54.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.901+0000 7f9619000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961410f420 con 0x7f96141038e0 2024-03-23T22:58:54.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.901+0000 7f9619000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961410f720 con 0x7f9614104ad0 2024-03-23T22:58:54.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.901+0000 7f9612800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9614105490 0x7f96141a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:54.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.901+0000 7f9612800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9614105490 0x7f96141a11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60778/0 (socket says 172.21.15.12:60778) 2024-03-23T22:58:54.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.901+0000 7f9612800640 1 -- 172.21.15.12:0/178925797 learned_addr learned my addr 172.21.15.12:0/178925797 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:58:54.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.901+0000 7f9611e00640 1 --2- 172.21.15.12:0/178925797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96141038e0 0x7f9614199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:54.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.901+0000 7f9609400640 1 --2- 172.21.15.12:0/178925797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9614104ad0 0x7f961419a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:54.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.901+0000 7f9612800640 1 -- 172.21.15.12:0/178925797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9614104ad0 msgr2=0x7f961419a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:54.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.901+0000 7f9612800640 1 --2- 172.21.15.12:0/178925797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9614104ad0 0x7f961419a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:54.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.901+0000 7f9612800640 1 -- 172.21.15.12:0/178925797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96141038e0 msgr2=0x7f9614199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:54.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.901+0000 7f9612800640 1 --2- 172.21.15.12:0/178925797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96141038e0 0x7f9614199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:54.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.901+0000 7f9612800640 1 -- 172.21.15.12:0/178925797 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96141a1840 con 0x7f9614105490 2024-03-23T22:58:54.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.901+0000 7f9609400640 1 --2- 172.21.15.12:0/178925797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9614104ad0 0x7f961419a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T22:58:54.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.902+0000 7f9612800640 1 --2- 172.21.15.12:0/178925797 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9614105490 0x7f96141a11a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f960c00e970 tx=0x7f960c00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:54.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.902+0000 7f960b400640 1 -- 172.21.15.12:0/178925797 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c00ccc0 con 0x7f9614105490 2024-03-23T22:58:54.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.902+0000 7f960b400640 1 -- 172.21.15.12:0/178925797 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f960c005500 con 0x7f9614105490 2024-03-23T22:58:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.902+0000 7f9619000640 1 -- 172.21.15.12:0/178925797 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96141a1b30 con 0x7f9614105490 2024-03-23T22:58:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.902+0000 7f960b400640 1 -- 172.21.15.12:0/178925797 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c010690 con 0x7f9614105490 2024-03-23T22:58:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.902+0000 7f9619000640 1 -- 172.21.15.12:0/178925797 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96141a2000 con 0x7f9614105490 2024-03-23T22:58:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.904+0000 7f9619000640 1 -- 172.21.15.12:0/178925797 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f961410ac00 con 0x7f9614105490 2024-03-23T22:58:54.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.906+0000 7f960b400640 1 -- 172.21.15.12:0/178925797 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f960c004e50 con 0x7f9614105490 2024-03-23T22:58:54.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.908+0000 7f960b400640 1 --2- 172.21.15.12:0/178925797 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f95f80a2800 0x7f95f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:54.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.909+0000 7f960b400640 1 -- 172.21.15.12:0/178925797 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f960c0cec40 con 0x7f9614105490 2024-03-23T22:58:54.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.909+0000 7f9611e00640 1 --2- 172.21.15.12:0/178925797 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f95f80a2800 0x7f95f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:54.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.909+0000 7f9611e00640 1 --2- 172.21.15.12:0/178925797 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f95f80a2800 0x7f95f80a4cc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9604034fd0 tx=0x7f9604000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:54.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:54.911+0000 7f960b400640 1 -- 172.21.15.12:0/178925797 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f960c053740 con 0x7f9614105490 2024-03-23T22:58:55.012 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 23 22:58:54 smithi012 podman[72339]: 2024-03-23 22:58:54.611400743 +0000 UTC m=+0.036394262 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-23T22:58:55.012 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 23 22:58:54 smithi012 podman[72339]: 2024-03-23 22:58:54.758738015 +0000 UTC m=+0.183731536 container create c3543e91c6973124fb066feeea8aae11a08085976248c01e0d90b586133cd836 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-23T22:58:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:55.172+0000 7f9619000640 1 -- 172.21.15.12:0/178925797 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f961402d2c0 con 0x7f9614105490 2024-03-23T22:58:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:55.173+0000 7f960b400640 1 -- 172.21.15.12:0/178925797 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3704 (secure 0 0 0) 0x7f960c09c0e0 con 0x7f9614105490 2024-03-23T22:58:55.172 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:58:55.172 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-23T22:58:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 5 2024-03-23T22:58:55.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:55.175+0000 7f9619000640 1 -- 172.21.15.12:0/178925797 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f95f80a2800 msgr2=0x7f95f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:55.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:55.175+0000 7f9619000640 1 --2- 172.21.15.12:0/178925797 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f95f80a2800 0x7f95f80a4cc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9604034fd0 tx=0x7f9604000f40 comp rx=0 tx=0).stop 2024-03-23T22:58:55.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:55.175+0000 7f9619000640 1 -- 172.21.15.12:0/178925797 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9614105490 msgr2=0x7f96141a11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:55.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:55.175+0000 7f9619000640 1 --2- 172.21.15.12:0/178925797 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9614105490 0x7f96141a11a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f960c00e970 tx=0x7f960c00ee30 comp rx=0 tx=0).stop 2024-03-23T22:58:55.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:55.175+0000 7f9619000640 1 -- 172.21.15.12:0/178925797 shutdown_connections 2024-03-23T22:58:55.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:55.175+0000 7f9619000640 1 --2- 172.21.15.12:0/178925797 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f95f80a2800 0x7f95f80a4cc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:55.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:55.175+0000 7f9619000640 1 --2- 172.21.15.12:0/178925797 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9614105490 0x7f96141a11a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:55.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:55.175+0000 7f9619000640 1 --2- 172.21.15.12:0/178925797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9614104ad0 0x7f961419a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:55.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:55.175+0000 7f9619000640 1 --2- 172.21.15.12:0/178925797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96141038e0 0x7f9614199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:55.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:55.175+0000 7f9619000640 1 -- 172.21.15.12:0/178925797 >> 172.21.15.12:0/178925797 conn(0x7f96140fee90 msgr2=0x7f9614107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:55.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:55.176+0000 7f9619000640 1 -- 172.21.15.12:0/178925797 shutdown_connections 2024-03-23T22:58:55.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:55.176+0000 7f9619000640 1 -- 172.21.15.12:0/178925797 wait complete. 2024-03-23T22:58:55.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:55 smithi012 ceph-mon[31695]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:55.304 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 23 22:58:55 smithi012 podman[72339]: 2024-03-23 22:58:55.012714082 +0000 UTC m=+0.437707630 container init c3543e91c6973124fb066feeea8aae11a08085976248c01e0d90b586133cd836 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-d, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-23T22:58:55.304 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 23 22:58:55 smithi012 podman[72339]: 2024-03-23 22:58:55.02172579 +0000 UTC m=+0.446719341 container start c3543e91c6973124fb066feeea8aae11a08085976248c01e0d90b586133cd836 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-23T22:58:55.304 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 23 22:58:55 smithi012 ceph-mds[72397]: -- 172.21.15.12:0/4113036633 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5633f2cbfa40 con 0x5633f2c92c00 2024-03-23T22:58:55.304 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 23 22:58:55 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-d[72375]: starting mds.d at 2024-03-23T22:58:55.304 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 23 22:58:55 smithi012 bash[72339]: c3543e91c6973124fb066feeea8aae11a08085976248c01e0d90b586133cd836 2024-03-23T22:58:55.305 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 23 22:58:55 smithi012 systemd[1]: Started Ceph mds.d for 90f679ae-e967-11ee-95cd-87774f69a715. 2024-03-23T22:58:55.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:55 smithi203 ceph-mon[36693]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:55.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:55 smithi173 ceph-mon[36116]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:56.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:56 smithi173 ceph-mon[36116]: mds.? [v2:172.21.15.12:6836/2696205127,v1:172.21.15.12:6837/2696205127] up:boot 2024-03-23T22:58:56.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:56 smithi173 ceph-mon[36116]: fsmap 4 up:standby 2024-03-23T22:58:56.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:56 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-23T22:58:56.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:56 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/178925797' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:58:56.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:56 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:56.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:56 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:56.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:56 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:56.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:56 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:58:56.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:56 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:58:56.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:56 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:56.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:56 smithi203 ceph-mon[36693]: mds.? [v2:172.21.15.12:6836/2696205127,v1:172.21.15.12:6837/2696205127] up:boot 2024-03-23T22:58:56.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:56 smithi203 ceph-mon[36693]: fsmap 4 up:standby 2024-03-23T22:58:56.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:56 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-23T22:58:56.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:56 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/178925797' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:58:56.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:56 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:56.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:56 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:56.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:56 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:56.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:56 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:58:56.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:56 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:58:56.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:56 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:56.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:56 smithi012 ceph-mon[31695]: mds.? [v2:172.21.15.12:6836/2696205127,v1:172.21.15.12:6837/2696205127] up:boot 2024-03-23T22:58:56.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:56 smithi012 ceph-mon[31695]: fsmap 4 up:standby 2024-03-23T22:58:56.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:56 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-23T22:58:56.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:56 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/178925797' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:58:56.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:56 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:56.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:56 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:56.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:56 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:56.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:56 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:58:56.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:56 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:58:56.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:56 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:57.193 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T22:58:57.328 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:57 smithi173 ceph-mon[36116]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:57.328 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:57 smithi173 ceph-mon[36116]: Deploying daemon mds.e on smithi173 2024-03-23T22:58:57.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:57 smithi203 ceph-mon[36693]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:57.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:57 smithi203 ceph-mon[36693]: Deploying daemon mds.e on smithi173 2024-03-23T22:58:57.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.422+0000 7f22e7200640 1 -- 172.21.15.12:0/4084195087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22e01014e0 msgr2=0x7f22e0101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:57.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.422+0000 7f22e7200640 1 --2- 172.21.15.12:0/4084195087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22e01014e0 0x7f22e0101900 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f22d4009900 tx=0x7f22d402bc30 comp rx=0 tx=0).stop 2024-03-23T22:58:57.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.423+0000 7f22e7200640 1 -- 172.21.15.12:0/4084195087 shutdown_connections 2024-03-23T22:58:57.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.423+0000 7f22e7200640 1 --2- 172.21.15.12:0/4084195087 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22e0103090 0x7f22e0109920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:57.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.423+0000 7f22e7200640 1 --2- 172.21.15.12:0/4084195087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22e01026d0 0x7f22e0102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:57.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.423+0000 7f22e7200640 1 --2- 172.21.15.12:0/4084195087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22e01014e0 0x7f22e0101900 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:57.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.423+0000 7f22e7200640 1 -- 172.21.15.12:0/4084195087 >> 172.21.15.12:0/4084195087 conn(0x7f22e00fca30 msgr2=0x7f22e00fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:57.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.423+0000 7f22e7200640 1 -- 172.21.15.12:0/4084195087 shutdown_connections 2024-03-23T22:58:57.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.423+0000 7f22e7200640 1 -- 172.21.15.12:0/4084195087 wait complete. 2024-03-23T22:58:57.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.424+0000 7f22e7200640 1 Processor -- start 2024-03-23T22:58:57.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.424+0000 7f22e7200640 1 -- start start 2024-03-23T22:58:57.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.424+0000 7f22e7200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22e01014e0 0x7f22e0199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:57.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.424+0000 7f22e7200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22e01026d0 0x7f22e0199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:57.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.425+0000 7f22e7200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22e0103090 0x7f22e01a1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:57.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.425+0000 7f22e7200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e010d1c0 con 0x7f22e01014e0 2024-03-23T22:58:57.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.425+0000 7f22e7200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e010d040 con 0x7f22e01026d0 2024-03-23T22:58:57.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.425+0000 7f22e7200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e010d340 con 0x7f22e0103090 2024-03-23T22:58:57.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.425+0000 7f22e5400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22e01026d0 0x7f22e0199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:57.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.425+0000 7f22e5400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22e01026d0 0x7f22e0199fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:45714/0 (socket says 172.21.15.12:45714) 2024-03-23T22:58:57.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.425+0000 7f22e5400640 1 -- 172.21.15.12:0/1138856980 learned_addr learned my addr 172.21.15.12:0/1138856980 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:58:57.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.425+0000 7f22e6800640 1 --2- 172.21.15.12:0/1138856980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22e0103090 0x7f22e01a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:57.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.425+0000 7f22e5e00640 1 --2- 172.21.15.12:0/1138856980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22e01014e0 0x7f22e0199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:57.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.425+0000 7f22e6800640 1 -- 172.21.15.12:0/1138856980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22e01026d0 msgr2=0x7f22e0199fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:57.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.425+0000 7f22e6800640 1 --2- 172.21.15.12:0/1138856980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22e01026d0 0x7f22e0199fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:57.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.425+0000 7f22e6800640 1 -- 172.21.15.12:0/1138856980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22e01014e0 msgr2=0x7f22e0199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:57.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.425+0000 7f22e6800640 1 --2- 172.21.15.12:0/1138856980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22e01014e0 0x7f22e0199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:57.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.425+0000 7f22e6800640 1 -- 172.21.15.12:0/1138856980 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e01a16b0 con 0x7f22e0103090 2024-03-23T22:58:57.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.425+0000 7f22e5400640 1 --2- 172.21.15.12:0/1138856980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22e01026d0 0x7f22e0199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T22:58:57.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.426+0000 7f22e6800640 1 --2- 172.21.15.12:0/1138856980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22e0103090 0x7f22e01a1040 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f22dc009fd0 tx=0x7f22dc00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:57.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.426+0000 7f22daa00640 1 -- 172.21.15.12:0/1138856980 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22dc00ce20 con 0x7f22e0103090 2024-03-23T22:58:57.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.426+0000 7f22daa00640 1 -- 172.21.15.12:0/1138856980 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22dc005020 con 0x7f22e0103090 2024-03-23T22:58:57.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.426+0000 7f22e7200640 1 -- 172.21.15.12:0/1138856980 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e01a19a0 con 0x7f22e0103090 2024-03-23T22:58:57.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.426+0000 7f22daa00640 1 -- 172.21.15.12:0/1138856980 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22dc010720 con 0x7f22e0103090 2024-03-23T22:58:57.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.426+0000 7f22e7200640 1 -- 172.21.15.12:0/1138856980 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e01a1ea0 con 0x7f22e0103090 2024-03-23T22:58:57.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.427+0000 7f22e7200640 1 -- 172.21.15.12:0/1138856980 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22e01a2190 con 0x7f22e0103090 2024-03-23T22:58:57.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.429+0000 7f22daa00640 1 -- 172.21.15.12:0/1138856980 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f22dc005540 con 0x7f22e0103090 2024-03-23T22:58:57.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.430+0000 7f22daa00640 1 --2- 172.21.15.12:0/1138856980 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f22c80a2800 0x7f22c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:57.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.431+0000 7f22daa00640 1 -- 172.21.15.12:0/1138856980 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f22dc0ced40 con 0x7f22e0103090 2024-03-23T22:58:57.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.431+0000 7f22e5e00640 1 --2- 172.21.15.12:0/1138856980 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f22c80a2800 0x7f22c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:57.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.431+0000 7f22e5e00640 1 --2- 172.21.15.12:0/1138856980 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f22c80a2800 0x7f22c80a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f22d402c240 tx=0x7f22d402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:57.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.435+0000 7f22daa00640 1 -- 172.21.15.12:0/1138856980 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f22dc09c1e0 con 0x7f22e0103090 2024-03-23T22:58:57.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:57 smithi012 ceph-mon[31695]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:57.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:57 smithi012 ceph-mon[31695]: Deploying daemon mds.e on smithi173 2024-03-23T22:58:57.618 INFO:journalctl@ceph.mds.e.smithi173.stdout:Mar 23 22:58:57 smithi173 podman[65897]: 2024-03-23 22:58:57.553788487 +0000 UTC m=+0.125158918 container create 8270dbda57bfa0eb45fe09f46a9731e8ee776337790f296917aad80cc7bba999 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-23T22:58:57.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.687+0000 7f22e7200640 1 -- 172.21.15.12:0/1138856980 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22e01a2360 con 0x7f22e0103090 2024-03-23T22:58:57.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.688+0000 7f22daa00640 1 -- 172.21.15.12:0/1138856980 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3704 (secure 0 0 0) 0x7f22dc09bae0 con 0x7f22e0103090 2024-03-23T22:58:57.687 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:58:57.687 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-23T22:58:57.687 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 5 2024-03-23T22:58:57.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.690+0000 7f22e7200640 1 -- 172.21.15.12:0/1138856980 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f22c80a2800 msgr2=0x7f22c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:57.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.690+0000 7f22e7200640 1 --2- 172.21.15.12:0/1138856980 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f22c80a2800 0x7f22c80a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f22d402c240 tx=0x7f22d402df90 comp rx=0 tx=0).stop 2024-03-23T22:58:57.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.690+0000 7f22e7200640 1 -- 172.21.15.12:0/1138856980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22e0103090 msgr2=0x7f22e01a1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:57.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.690+0000 7f22e7200640 1 --2- 172.21.15.12:0/1138856980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22e0103090 0x7f22e01a1040 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f22dc009fd0 tx=0x7f22dc00ef10 comp rx=0 tx=0).stop 2024-03-23T22:58:57.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.691+0000 7f22e7200640 1 -- 172.21.15.12:0/1138856980 shutdown_connections 2024-03-23T22:58:57.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.691+0000 7f22e7200640 1 --2- 172.21.15.12:0/1138856980 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f22c80a2800 0x7f22c80a4cc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:57.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.691+0000 7f22e7200640 1 --2- 172.21.15.12:0/1138856980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22e0103090 0x7f22e01a1040 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:57.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.691+0000 7f22e7200640 1 --2- 172.21.15.12:0/1138856980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22e01026d0 0x7f22e0199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:57.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.691+0000 7f22e7200640 1 --2- 172.21.15.12:0/1138856980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22e01014e0 0x7f22e0199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:57.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.691+0000 7f22e7200640 1 -- 172.21.15.12:0/1138856980 >> 172.21.15.12:0/1138856980 conn(0x7f22e00fca30 msgr2=0x7f22e00738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:57.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.691+0000 7f22e7200640 1 -- 172.21.15.12:0/1138856980 shutdown_connections 2024-03-23T22:58:57.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:57.691+0000 7f22e7200640 1 -- 172.21.15.12:0/1138856980 wait complete. 2024-03-23T22:58:58.118 INFO:journalctl@ceph.mds.e.smithi173.stdout:Mar 23 22:58:57 smithi173 podman[65897]: 2024-03-23 22:58:57.798607101 +0000 UTC m=+0.369977529 container init 8270dbda57bfa0eb45fe09f46a9731e8ee776337790f296917aad80cc7bba999 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-23T22:58:58.118 INFO:journalctl@ceph.mds.e.smithi173.stdout:Mar 23 22:58:57 smithi173 podman[65897]: 2024-03-23 22:58:57.801106427 +0000 UTC m=+0.372476852 container start 8270dbda57bfa0eb45fe09f46a9731e8ee776337790f296917aad80cc7bba999 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-e, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-23T22:58:58.118 INFO:journalctl@ceph.mds.e.smithi173.stdout:Mar 23 22:58:57 smithi173 ceph-mds[65916]: -- 172.21.15.173:0/3456406225 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d7a81b1a40 con 0x55d7a8184c00 2024-03-23T22:58:58.118 INFO:journalctl@ceph.mds.e.smithi173.stdout:Mar 23 22:58:57 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-e[65912]: starting mds.e at 2024-03-23T22:58:58.119 INFO:journalctl@ceph.mds.e.smithi173.stdout:Mar 23 22:58:57 smithi173 bash[65897]: 8270dbda57bfa0eb45fe09f46a9731e8ee776337790f296917aad80cc7bba999 2024-03-23T22:58:58.119 INFO:journalctl@ceph.mds.e.smithi173.stdout:Mar 23 22:58:57 smithi173 systemd[1]: Started Ceph mds.e for 90f679ae-e967-11ee-95cd-87774f69a715. 2024-03-23T22:58:58.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:58 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1138856980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:58:58.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:58 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:58.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:58 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:58.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:58 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:58.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:58 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:58:58.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:58 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:58:58.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:58 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:58.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:58 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1138856980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:58:58.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:58 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:58.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:58 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:58.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:58 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:58.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:58 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:58:58.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:58 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:58:58.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:58 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:58.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:58 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1138856980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:58:58.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:58 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:58.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:58 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:58.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:58 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:58:58.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:58 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:58:58.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:58 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:58:58.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:58 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:58:59.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:59 smithi203 ceph-mon[36693]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:59.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:59 smithi203 ceph-mon[36693]: Deploying daemon mds.f on smithi203 2024-03-23T22:58:59.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:59 smithi203 ceph-mon[36693]: mds.? [v2:172.21.15.173:6834/2648278843,v1:172.21.15.173:6835/2648278843] up:boot 2024-03-23T22:58:59.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:59 smithi203 ceph-mon[36693]: fsmap 5 up:standby 2024-03-23T22:58:59.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:58:59 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-23T22:58:59.457 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:59 smithi012 ceph-mon[31695]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:59.457 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:59 smithi012 ceph-mon[31695]: Deploying daemon mds.f on smithi203 2024-03-23T22:58:59.458 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:59 smithi012 ceph-mon[31695]: mds.? [v2:172.21.15.173:6834/2648278843,v1:172.21.15.173:6835/2648278843] up:boot 2024-03-23T22:58:59.458 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:59 smithi012 ceph-mon[31695]: fsmap 5 up:standby 2024-03-23T22:58:59.458 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:58:59 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-23T22:58:59.545 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:59 smithi173 ceph-mon[36116]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:58:59.546 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:59 smithi173 ceph-mon[36116]: Deploying daemon mds.f on smithi203 2024-03-23T22:58:59.546 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:59 smithi173 ceph-mon[36116]: mds.? [v2:172.21.15.173:6834/2648278843,v1:172.21.15.173:6835/2648278843] up:boot 2024-03-23T22:58:59.546 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:59 smithi173 ceph-mon[36116]: fsmap 5 up:standby 2024-03-23T22:58:59.546 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:58:59 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-23T22:58:59.708 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T22:58:59.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.933+0000 7fab49e00640 1 -- 172.21.15.12:0/1735280135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab44101510 msgr2=0x7fab44101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.933+0000 7fab49e00640 1 --2- 172.21.15.12:0/1735280135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab44101510 0x7fab44101930 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fab38009900 tx=0x7fab3802bc30 comp rx=0 tx=0).stop 2024-03-23T22:58:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.933+0000 7fab49e00640 1 -- 172.21.15.12:0/1735280135 shutdown_connections 2024-03-23T22:58:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.933+0000 7fab49e00640 1 --2- 172.21.15.12:0/1735280135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab441030c0 0x7fab44109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.933+0000 7fab49e00640 1 --2- 172.21.15.12:0/1735280135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab44102700 0x7fab44102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.933+0000 7fab49e00640 1 --2- 172.21.15.12:0/1735280135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab44101510 0x7fab44101930 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.933+0000 7fab49e00640 1 -- 172.21.15.12:0/1735280135 >> 172.21.15.12:0/1735280135 conn(0x7fab440fca80 msgr2=0x7fab440feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:58:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.933+0000 7fab49e00640 1 -- 172.21.15.12:0/1735280135 shutdown_connections 2024-03-23T22:58:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.933+0000 7fab49e00640 1 -- 172.21.15.12:0/1735280135 wait complete. 2024-03-23T22:58:59.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.934+0000 7fab49e00640 1 Processor -- start 2024-03-23T22:58:59.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.934+0000 7fab49e00640 1 -- start start 2024-03-23T22:58:59.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.935+0000 7fab49e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab44101510 0x7fab44197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:59.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.935+0000 7fab49e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab44102700 0x7fab44197d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:59.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.935+0000 7fab49e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab441030c0 0x7fab4419edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:59.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.935+0000 7fab49e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4410d1f0 con 0x7fab44101510 2024-03-23T22:58:59.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.935+0000 7fab49e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4410d070 con 0x7fab44102700 2024-03-23T22:58:59.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.935+0000 7fab49e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4410d370 con 0x7fab441030c0 2024-03-23T22:58:59.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.935+0000 7fab48a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab44101510 0x7fab44197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:59.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.935+0000 7fab43e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab44102700 0x7fab44197d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:59.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.935+0000 7fab43e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab44102700 0x7fab44197d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:45730/0 (socket says 172.21.15.12:45730) 2024-03-23T22:58:59.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.935+0000 7fab48a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab44101510 0x7fab44197840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60842/0 (socket says 172.21.15.12:60842) 2024-03-23T22:58:59.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.935+0000 7fab43e00640 1 -- 172.21.15.12:0/2611508693 learned_addr learned my addr 172.21.15.12:0/2611508693 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:58:59.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.935+0000 7fab49400640 1 --2- 172.21.15.12:0/2611508693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab441030c0 0x7fab4419edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:59.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.935+0000 7fab43e00640 1 -- 172.21.15.12:0/2611508693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab441030c0 msgr2=0x7fab4419edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:59.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.935+0000 7fab43e00640 1 --2- 172.21.15.12:0/2611508693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab441030c0 0x7fab4419edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:59.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.935+0000 7fab43e00640 1 -- 172.21.15.12:0/2611508693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab44101510 msgr2=0x7fab44197840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:58:59.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.935+0000 7fab43e00640 1 --2- 172.21.15.12:0/2611508693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab44101510 0x7fab44197840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:58:59.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.935+0000 7fab43e00640 1 -- 172.21.15.12:0/2611508693 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4419f460 con 0x7fab44102700 2024-03-23T22:58:59.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.936+0000 7fab43e00640 1 --2- 172.21.15.12:0/2611508693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab44102700 0x7fab44197d80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fab3000e9f0 tx=0x7fab3000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:59.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.936+0000 7fab41600640 1 -- 172.21.15.12:0/2611508693 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3000cd80 con 0x7fab44102700 2024-03-23T22:58:59.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.936+0000 7fab49e00640 1 -- 172.21.15.12:0/2611508693 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab4419f750 con 0x7fab44102700 2024-03-23T22:58:59.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.936+0000 7fab41600640 1 -- 172.21.15.12:0/2611508693 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab30010910 con 0x7fab44102700 2024-03-23T22:58:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.936+0000 7fab41600640 1 -- 172.21.15.12:0/2611508693 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab300188f0 con 0x7fab44102700 2024-03-23T22:58:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.936+0000 7fab49e00640 1 -- 172.21.15.12:0/2611508693 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab4419fcf0 con 0x7fab44102700 2024-03-23T22:58:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.937+0000 7fab49e00640 1 -- 172.21.15.12:0/2611508693 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab44108830 con 0x7fab44102700 2024-03-23T22:58:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.940+0000 7fab41600640 1 -- 172.21.15.12:0/2611508693 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fab30010ac0 con 0x7fab44102700 2024-03-23T22:58:59.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.941+0000 7fab41600640 1 --2- 172.21.15.12:0/2611508693 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fab2c0a2800 0x7fab2c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:58:59.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.941+0000 7fab41600640 1 -- 172.21.15.12:0/2611508693 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fab300ceac0 con 0x7fab44102700 2024-03-23T22:58:59.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.941+0000 7fab48a00640 1 --2- 172.21.15.12:0/2611508693 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fab2c0a2800 0x7fab2c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:58:59.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.942+0000 7fab48a00640 1 --2- 172.21.15.12:0/2611508693 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fab2c0a2800 0x7fab2c0a4cc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fab3802c170 tx=0x7fab3802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:58:59.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:58:59.945+0000 7fab41600640 1 -- 172.21.15.12:0/2611508693 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fab3009bf60 con 0x7fab44102700 2024-03-23T22:59:00.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:00.194+0000 7fab49e00640 1 -- 172.21.15.12:0/2611508693 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab440622a0 con 0x7fab44102700 2024-03-23T22:59:00.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:00.195+0000 7fab41600640 1 -- 172.21.15.12:0/2611508693 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4498 (secure 0 0 0) 0x7fab3009b860 con 0x7fab44102700 2024-03-23T22:59:00.194 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:59:00.194 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24521,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-23T22:59:00.194 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 6 2024-03-23T22:59:00.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:00.197+0000 7fab49e00640 1 -- 172.21.15.12:0/2611508693 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fab2c0a2800 msgr2=0x7fab2c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:00.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:00.197+0000 7fab49e00640 1 --2- 172.21.15.12:0/2611508693 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fab2c0a2800 0x7fab2c0a4cc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fab3802c170 tx=0x7fab3802df90 comp rx=0 tx=0).stop 2024-03-23T22:59:00.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:00.197+0000 7fab49e00640 1 -- 172.21.15.12:0/2611508693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab44102700 msgr2=0x7fab44197d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:00.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:00.197+0000 7fab49e00640 1 --2- 172.21.15.12:0/2611508693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab44102700 0x7fab44197d80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fab3000e9f0 tx=0x7fab3000eeb0 comp rx=0 tx=0).stop 2024-03-23T22:59:00.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:00.197+0000 7fab49e00640 1 -- 172.21.15.12:0/2611508693 shutdown_connections 2024-03-23T22:59:00.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:00.197+0000 7fab49e00640 1 --2- 172.21.15.12:0/2611508693 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fab2c0a2800 0x7fab2c0a4cc0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:00.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:00.197+0000 7fab49e00640 1 --2- 172.21.15.12:0/2611508693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab441030c0 0x7fab4419edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:00.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:00.197+0000 7fab49e00640 1 --2- 172.21.15.12:0/2611508693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab44102700 0x7fab44197d80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:00.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:00.197+0000 7fab49e00640 1 --2- 172.21.15.12:0/2611508693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab44101510 0x7fab44197840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:00.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:00.197+0000 7fab49e00640 1 -- 172.21.15.12:0/2611508693 >> 172.21.15.12:0/2611508693 conn(0x7fab440fca80 msgr2=0x7fab44105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:00.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:00.198+0000 7fab49e00640 1 -- 172.21.15.12:0/2611508693 shutdown_connections 2024-03-23T22:59:00.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:00.198+0000 7fab49e00640 1 -- 172.21.15.12:0/2611508693 wait complete. 2024-03-23T22:59:00.407 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 23 22:59:00 smithi203 systemd[1]: Starting Ceph mds.f for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T22:59:00.765 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 23 22:59:00 smithi203 podman[68068]: 2024-03-23 22:59:00.406824433 +0000 UTC m=+0.017924802 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-23T22:59:00.765 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 23 22:59:00 smithi203 podman[68068]: 2024-03-23 22:59:00.539100464 +0000 UTC m=+0.150200819 container create 58f37a969a279d3b72419cc564b453853153f6b47b750d1c1c637e767826c389 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-f, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-03-23T22:59:00.766 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 23 22:59:00 smithi203 podman[68068]: 2024-03-23 22:59:00.766695076 +0000 UTC m=+0.377795439 container init 58f37a969a279d3b72419cc564b453853153f6b47b750d1c1c637e767826c389 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-f, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-23T22:59:01.154 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 23 22:59:00 smithi203 podman[68068]: 2024-03-23 22:59:00.769362703 +0000 UTC m=+0.380463066 container start 58f37a969a279d3b72419cc564b453853153f6b47b750d1c1c637e767826c389 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-f, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-23T22:59:01.154 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 23 22:59:00 smithi203 ceph-mds[68087]: -- 172.21.15.203:0/164286846 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562950f15a40 con 0x562950ee9000 2024-03-23T22:59:01.154 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 23 22:59:00 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-f[68083]: starting mds.f at 2024-03-23T22:59:01.154 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 23 22:59:00 smithi203 bash[68068]: 58f37a969a279d3b72419cc564b453853153f6b47b750d1c1c637e767826c389 2024-03-23T22:59:01.154 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 23 22:59:00 smithi203 systemd[1]: Started Ceph mds.f for 90f679ae-e967-11ee-95cd-87774f69a715. 2024-03-23T22:59:01.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:01 smithi203 ceph-mon[36693]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:01.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:01 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2611508693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:01.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:01 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:01.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:01 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:01.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:01 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:01.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:01 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:59:01.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:01 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:59:01.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:01 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:59:01.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:01 smithi012 ceph-mon[31695]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:01.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:01 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2611508693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:01.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:01 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:01.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:01 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:01.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:01 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:01.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:01 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:59:01.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:01 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:59:01.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:01 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:59:01.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:01 smithi173 ceph-mon[36116]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:01.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:01 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2611508693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:01.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:01 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:01.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:01 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:01.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:01 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:01.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:01 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:59:01.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:01 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:59:01.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:01 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:59:02.215 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T22:59:02.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.450+0000 7f3d85a00640 1 -- 172.21.15.12:0/3369933065 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d801096b0 msgr2=0x7f3d8010ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:02.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.450+0000 7f3d85a00640 1 --2- 172.21.15.12:0/3369933065 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d801096b0 0x7f3d8010ff40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009ca0 tx=0x7f3d7802c100 comp rx=0 tx=0).stop 2024-03-23T22:59:02.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.450+0000 7f3d85a00640 1 -- 172.21.15.12:0/3369933065 shutdown_connections 2024-03-23T22:59:02.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.450+0000 7f3d85a00640 1 --2- 172.21.15.12:0/3369933065 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d801096b0 0x7f3d8010ff40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:02.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.450+0000 7f3d85a00640 1 --2- 172.21.15.12:0/3369933065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d80108cf0 0x7f3d80109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:02.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.450+0000 7f3d85a00640 1 --2- 172.21.15.12:0/3369933065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d80107b00 0x7f3d80107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:02.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.450+0000 7f3d85a00640 1 -- 172.21.15.12:0/3369933065 >> 172.21.15.12:0/3369933065 conn(0x7f3d80076ab0 msgr2=0x7f3d80078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:02.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.450+0000 7f3d85a00640 1 -- 172.21.15.12:0/3369933065 shutdown_connections 2024-03-23T22:59:02.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.450+0000 7f3d85a00640 1 -- 172.21.15.12:0/3369933065 wait complete. 2024-03-23T22:59:02.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.451+0000 7f3d85a00640 1 Processor -- start 2024-03-23T22:59:02.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.451+0000 7f3d85a00640 1 -- start start 2024-03-23T22:59:02.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.451+0000 7f3d85a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d80107b00 0x7f3d8019dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:02.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.451+0000 7f3d85a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d80108cf0 0x7f3d8019e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:02.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.451+0000 7f3d85a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d801096b0 0x7f3d801a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:02.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.451+0000 7f3d85a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d80113780 con 0x7f3d801096b0 2024-03-23T22:59:02.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.451+0000 7f3d85a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d80113600 con 0x7f3d80108cf0 2024-03-23T22:59:02.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.451+0000 7f3d85a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d80113900 con 0x7f3d80107b00 2024-03-23T22:59:02.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.452+0000 7f3d85000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d801096b0 0x7f3d801a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:02.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.452+0000 7f3d7fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d80107b00 0x7f3d8019dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:02.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.452+0000 7f3d7fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d80107b00 0x7f3d8019dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:33718/0 (socket says 172.21.15.12:33718) 2024-03-23T22:59:02.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.452+0000 7f3d7fe00640 1 -- 172.21.15.12:0/2192454147 learned_addr learned my addr 172.21.15.12:0/2192454147 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:02.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.452+0000 7f3d77e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d80108cf0 0x7f3d8019e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:02.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.452+0000 7f3d85000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d801096b0 0x7f3d801a5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60868/0 (socket says 172.21.15.12:60868) 2024-03-23T22:59:02.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.452+0000 7f3d7fe00640 1 -- 172.21.15.12:0/2192454147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d80108cf0 msgr2=0x7f3d8019e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:02.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.452+0000 7f3d7fe00640 1 --2- 172.21.15.12:0/2192454147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d80108cf0 0x7f3d8019e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:02.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.452+0000 7f3d7fe00640 1 -- 172.21.15.12:0/2192454147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d801096b0 msgr2=0x7f3d801a5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:02.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.452+0000 7f3d7fe00640 1 --2- 172.21.15.12:0/2192454147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d801096b0 0x7f3d801a5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:02.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.452+0000 7f3d7fe00640 1 -- 172.21.15.12:0/2192454147 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d801a5980 con 0x7f3d80107b00 2024-03-23T22:59:02.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.452+0000 7f3d7fe00640 1 --2- 172.21.15.12:0/2192454147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d80107b00 0x7f3d8019dd60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3d7000e9c0 tx=0x7f3d7000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:02.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.453+0000 7f3d7d600640 1 -- 172.21.15.12:0/2192454147 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7000cd80 con 0x7f3d80107b00 2024-03-23T22:59:02.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.453+0000 7f3d7d600640 1 -- 172.21.15.12:0/2192454147 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d700035f0 con 0x7f3d80107b00 2024-03-23T22:59:02.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.453+0000 7f3d7d600640 1 -- 172.21.15.12:0/2192454147 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d70010690 con 0x7f3d80107b00 2024-03-23T22:59:02.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.453+0000 7f3d85a00640 1 -- 172.21.15.12:0/2192454147 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d801a5c70 con 0x7f3d80107b00 2024-03-23T22:59:02.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.453+0000 7f3d85a00640 1 -- 172.21.15.12:0/2192454147 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d801a6170 con 0x7f3d80107b00 2024-03-23T22:59:02.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.454+0000 7f3d85a00640 1 -- 172.21.15.12:0/2192454147 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d8010eea0 con 0x7f3d80107b00 2024-03-23T22:59:02.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.456+0000 7f3d7d600640 1 -- 172.21.15.12:0/2192454147 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3d700037a0 con 0x7f3d80107b00 2024-03-23T22:59:02.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.457+0000 7f3d7d600640 1 --2- 172.21.15.12:0/2192454147 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3d640a2880 0x7f3d640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:02.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.457+0000 7f3d7d600640 1 -- 172.21.15.12:0/2192454147 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3d700ce990 con 0x7f3d80107b00 2024-03-23T22:59:02.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.457+0000 7f3d77e00640 1 --2- 172.21.15.12:0/2192454147 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3d640a2880 0x7f3d640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:02.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.458+0000 7f3d77e00640 1 --2- 172.21.15.12:0/2192454147 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3d640a2880 0x7f3d640a4d40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3d8019f270 tx=0x7f3d68005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:02.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.461+0000 7f3d7d600640 1 -- 172.21.15.12:0/2192454147 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3d7009be30 con 0x7f3d80107b00 2024-03-23T22:59:02.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:02 smithi012 ceph-mon[31695]: Deploying daemon mds.c on smithi203 2024-03-23T22:59:02.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:02 smithi012 ceph-mon[31695]: mds.? [v2:172.21.15.203:6834/915277278,v1:172.21.15.203:6835/915277278] up:boot 2024-03-23T22:59:02.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:02 smithi012 ceph-mon[31695]: fsmap 6 up:standby 2024-03-23T22:59:02.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:02 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-23T22:59:02.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:02 smithi173 ceph-mon[36116]: Deploying daemon mds.c on smithi203 2024-03-23T22:59:02.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:02 smithi173 ceph-mon[36116]: mds.? [v2:172.21.15.203:6834/915277278,v1:172.21.15.203:6835/915277278] up:boot 2024-03-23T22:59:02.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:02 smithi173 ceph-mon[36116]: fsmap 6 up:standby 2024-03-23T22:59:02.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:02 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-23T22:59:02.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:02 smithi203 ceph-mon[36693]: Deploying daemon mds.c on smithi203 2024-03-23T22:59:02.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:02 smithi203 ceph-mon[36693]: mds.? [v2:172.21.15.203:6834/915277278,v1:172.21.15.203:6835/915277278] up:boot 2024-03-23T22:59:02.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:02 smithi203 ceph-mon[36693]: fsmap 6 up:standby 2024-03-23T22:59:02.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:02 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-23T22:59:02.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.704+0000 7f3d85a00640 1 -- 172.21.15.12:0/2192454147 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d8002d070 con 0x7f3d80107b00 2024-03-23T22:59:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.705+0000 7f3d7d600640 1 -- 172.21.15.12:0/2192454147 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5289 (secure 0 0 0) 0x7f3d7009b730 con 0x7f3d80107b00 2024-03-23T22:59:02.705 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:59:02.705 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24521,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-23T22:59:02.705 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 7 2024-03-23T22:59:02.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.708+0000 7f3d85a00640 1 -- 172.21.15.12:0/2192454147 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3d640a2880 msgr2=0x7f3d640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:02.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.708+0000 7f3d85a00640 1 --2- 172.21.15.12:0/2192454147 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3d640a2880 0x7f3d640a4d40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3d8019f270 tx=0x7f3d68005f90 comp rx=0 tx=0).stop 2024-03-23T22:59:02.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.708+0000 7f3d85a00640 1 -- 172.21.15.12:0/2192454147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d80107b00 msgr2=0x7f3d8019dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:02.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.708+0000 7f3d85a00640 1 --2- 172.21.15.12:0/2192454147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d80107b00 0x7f3d8019dd60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3d7000e9c0 tx=0x7f3d7000ee80 comp rx=0 tx=0).stop 2024-03-23T22:59:02.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.708+0000 7f3d85a00640 1 -- 172.21.15.12:0/2192454147 shutdown_connections 2024-03-23T22:59:02.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.708+0000 7f3d85a00640 1 --2- 172.21.15.12:0/2192454147 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3d640a2880 0x7f3d640a4d40 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:02.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.708+0000 7f3d85a00640 1 --2- 172.21.15.12:0/2192454147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d801096b0 0x7f3d801a5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:02.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.708+0000 7f3d85a00640 1 --2- 172.21.15.12:0/2192454147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d80108cf0 0x7f3d8019e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:02.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.708+0000 7f3d85a00640 1 --2- 172.21.15.12:0/2192454147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d80107b00 0x7f3d8019dd60 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:02.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.708+0000 7f3d85a00640 1 -- 172.21.15.12:0/2192454147 >> 172.21.15.12:0/2192454147 conn(0x7f3d80076ab0 msgr2=0x7f3d8010c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:02.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.708+0000 7f3d85a00640 1 -- 172.21.15.12:0/2192454147 shutdown_connections 2024-03-23T22:59:02.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:02.708+0000 7f3d85a00640 1 -- 172.21.15.12:0/2192454147 wait complete. 2024-03-23T22:59:03.223 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:03 smithi203 ceph-mon[36693]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:03.223 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:03 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2192454147' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:03.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:03 smithi012 ceph-mon[31695]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:03.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:03 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2192454147' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:03.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:03 smithi173 ceph-mon[36116]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:03.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:03 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2192454147' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:03.907 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 23 22:59:03 smithi203 systemd[1]: Started Ceph mds.c for 90f679ae-e967-11ee-95cd-87774f69a715. 2024-03-23T22:59:04.727 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T22:59:04.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.962+0000 7fdf55800640 1 -- 172.21.15.12:0/2039769040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf501048a0 msgr2=0x7fdf50106c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:04.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.962+0000 7fdf55800640 1 --2- 172.21.15.12:0/2039769040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf501048a0 0x7fdf50106c90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fdf38009900 tx=0x7fdf3802bc30 comp rx=0 tx=0).stop 2024-03-23T22:59:04.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.962+0000 7fdf55800640 1 -- 172.21.15.12:0/2039769040 shutdown_connections 2024-03-23T22:59:04.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.962+0000 7fdf55800640 1 --2- 172.21.15.12:0/2039769040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf501071d0 0x7fdf50109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:04.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.962+0000 7fdf55800640 1 --2- 172.21.15.12:0/2039769040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf501048a0 0x7fdf50106c90 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:04.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.962+0000 7fdf55800640 1 --2- 172.21.15.12:0/2039769040 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf50101f70 0x7fdf50104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:04.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.962+0000 7fdf55800640 1 -- 172.21.15.12:0/2039769040 >> 172.21.15.12:0/2039769040 conn(0x7fdf500fb980 msgr2=0x7fdf500fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:04.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.962+0000 7fdf55800640 1 -- 172.21.15.12:0/2039769040 shutdown_connections 2024-03-23T22:59:04.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.962+0000 7fdf55800640 1 -- 172.21.15.12:0/2039769040 wait complete. 2024-03-23T22:59:04.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.963+0000 7fdf55800640 1 Processor -- start 2024-03-23T22:59:04.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.963+0000 7fdf55800640 1 -- start start 2024-03-23T22:59:04.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.964+0000 7fdf55800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf50101f70 0x7fdf50199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:04.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.964+0000 7fdf55800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf501048a0 0x7fdf50199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:04.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.964+0000 7fdf55800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf501071d0 0x7fdf501a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:04.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.964+0000 7fdf55800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5010d1d0 con 0x7fdf501048a0 2024-03-23T22:59:04.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.964+0000 7fdf55800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5010d050 con 0x7fdf50101f70 2024-03-23T22:59:04.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.964+0000 7fdf55800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5010d350 con 0x7fdf501071d0 2024-03-23T22:59:04.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.964+0000 7fdf47400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf501048a0 0x7fdf50199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:04.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.964+0000 7fdf47400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf501048a0 0x7fdf50199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59452/0 (socket says 172.21.15.12:59452) 2024-03-23T22:59:04.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.964+0000 7fdf4fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf50101f70 0x7fdf50199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:04.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.964+0000 7fdf47400640 1 -- 172.21.15.12:0/3019013379 learned_addr learned my addr 172.21.15.12:0/3019013379 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:04.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.964+0000 7fdf54e00640 1 --2- 172.21.15.12:0/3019013379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf501071d0 0x7fdf501a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:04.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.964+0000 7fdf4fe00640 1 -- 172.21.15.12:0/3019013379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf501071d0 msgr2=0x7fdf501a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:04.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.964+0000 7fdf4fe00640 1 --2- 172.21.15.12:0/3019013379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf501071d0 0x7fdf501a0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:04.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.964+0000 7fdf4fe00640 1 -- 172.21.15.12:0/3019013379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf501048a0 msgr2=0x7fdf50199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:04.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.964+0000 7fdf4fe00640 1 --2- 172.21.15.12:0/3019013379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf501048a0 0x7fdf50199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:04.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.964+0000 7fdf4fe00640 1 -- 172.21.15.12:0/3019013379 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf501a1640 con 0x7fdf50101f70 2024-03-23T22:59:04.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.964+0000 7fdf47400640 1 --2- 172.21.15.12:0/3019013379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf501048a0 0x7fdf50199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T22:59:04.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.965+0000 7fdf4fe00640 1 --2- 172.21.15.12:0/3019013379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf50101f70 0x7fdf50199a20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdf4000e9f0 tx=0x7fdf4000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:04.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.965+0000 7fdf4d600640 1 -- 172.21.15.12:0/3019013379 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4000ccc0 con 0x7fdf50101f70 2024-03-23T22:59:04.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.965+0000 7fdf4d600640 1 -- 172.21.15.12:0/3019013379 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdf40010910 con 0x7fdf50101f70 2024-03-23T22:59:04.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.965+0000 7fdf55800640 1 -- 172.21.15.12:0/3019013379 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf501a1930 con 0x7fdf50101f70 2024-03-23T22:59:04.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.965+0000 7fdf4d600640 1 -- 172.21.15.12:0/3019013379 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf40018960 con 0x7fdf50101f70 2024-03-23T22:59:04.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.965+0000 7fdf55800640 1 -- 172.21.15.12:0/3019013379 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf501a1ea0 con 0x7fdf50101f70 2024-03-23T22:59:04.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.966+0000 7fdf55800640 1 -- 172.21.15.12:0/3019013379 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf50108d20 con 0x7fdf50101f70 2024-03-23T22:59:04.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.968+0000 7fdf4d600640 1 -- 172.21.15.12:0/3019013379 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fdf40010ac0 con 0x7fdf50101f70 2024-03-23T22:59:04.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.970+0000 7fdf4d600640 1 --2- 172.21.15.12:0/3019013379 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdf340a2880 0x7fdf340a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:04.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.970+0000 7fdf4d600640 1 -- 172.21.15.12:0/3019013379 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdf400ce950 con 0x7fdf50101f70 2024-03-23T22:59:04.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.970+0000 7fdf47400640 1 --2- 172.21.15.12:0/3019013379 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdf340a2880 0x7fdf340a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:04.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.971+0000 7fdf47400640 1 --2- 172.21.15.12:0/3019013379 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdf340a2880 0x7fdf340a4d40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fdf38006010 tx=0x7fdf3802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:04.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:04.971+0000 7fdf4d600640 1 -- 172.21.15.12:0/3019013379 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdf4009b7a0 con 0x7fdf50101f70 2024-03-23T22:59:05.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:04 smithi173 ceph-mon[36116]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:05.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:04 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:05.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:04 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:05.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:04 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:05.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:04 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:59:05.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:04 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:59:05.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:04 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:59:05.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:04 smithi173 ceph-mon[36116]: Deploying daemon mds.h on smithi173 2024-03-23T22:59:05.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:04 smithi173 ceph-mon[36116]: mds.? [v2:172.21.15.203:6836/438676939,v1:172.21.15.203:6837/438676939] up:boot 2024-03-23T22:59:05.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:04 smithi173 ceph-mon[36116]: fsmap 7 up:standby 2024-03-23T22:59:05.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:04 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-23T22:59:05.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:04 smithi203 ceph-mon[36693]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:05.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:04 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:05.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:04 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:05.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:04 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:05.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:04 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:59:05.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:04 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:59:05.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:04 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:59:05.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:04 smithi203 ceph-mon[36693]: Deploying daemon mds.h on smithi173 2024-03-23T22:59:05.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:04 smithi203 ceph-mon[36693]: mds.? [v2:172.21.15.203:6836/438676939,v1:172.21.15.203:6837/438676939] up:boot 2024-03-23T22:59:05.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:04 smithi203 ceph-mon[36693]: fsmap 7 up:standby 2024-03-23T22:59:05.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:04 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-23T22:59:05.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:05.213+0000 7fdf55800640 1 -- 172.21.15.12:0/3019013379 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf5002d070 con 0x7fdf50101f70 2024-03-23T22:59:05.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:05.214+0000 7fdf4d600640 1 -- 172.21.15.12:0/3019013379 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6080 (secure 0 0 0) 0x7fdf4009f840 con 0x7fdf50101f70 2024-03-23T22:59:05.214 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:59:05.214 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24521,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-23T22:59:05.214 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 8 2024-03-23T22:59:05.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:05.217+0000 7fdf55800640 1 -- 172.21.15.12:0/3019013379 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdf340a2880 msgr2=0x7fdf340a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:05.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:05.217+0000 7fdf55800640 1 --2- 172.21.15.12:0/3019013379 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdf340a2880 0x7fdf340a4d40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fdf38006010 tx=0x7fdf3802df90 comp rx=0 tx=0).stop 2024-03-23T22:59:05.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:05.217+0000 7fdf55800640 1 -- 172.21.15.12:0/3019013379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf50101f70 msgr2=0x7fdf50199a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:05.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:05.217+0000 7fdf55800640 1 --2- 172.21.15.12:0/3019013379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf50101f70 0x7fdf50199a20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdf4000e9f0 tx=0x7fdf4000eeb0 comp rx=0 tx=0).stop 2024-03-23T22:59:05.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:05.217+0000 7fdf55800640 1 -- 172.21.15.12:0/3019013379 shutdown_connections 2024-03-23T22:59:05.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:05.218+0000 7fdf55800640 1 --2- 172.21.15.12:0/3019013379 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdf340a2880 0x7fdf340a4d40 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:05.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:05.218+0000 7fdf55800640 1 --2- 172.21.15.12:0/3019013379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf501071d0 0x7fdf501a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:05.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:05.218+0000 7fdf55800640 1 --2- 172.21.15.12:0/3019013379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf501048a0 0x7fdf50199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:05.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:05.218+0000 7fdf55800640 1 --2- 172.21.15.12:0/3019013379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf50101f70 0x7fdf50199a20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:05.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:05.218+0000 7fdf55800640 1 -- 172.21.15.12:0/3019013379 >> 172.21.15.12:0/3019013379 conn(0x7fdf500fb980 msgr2=0x7fdf50101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:05.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:05.218+0000 7fdf55800640 1 -- 172.21.15.12:0/3019013379 shutdown_connections 2024-03-23T22:59:05.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:05.218+0000 7fdf55800640 1 -- 172.21.15.12:0/3019013379 wait complete. 2024-03-23T22:59:05.232 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:04 smithi012 ceph-mon[31695]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:05.233 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:04 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:05.233 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:04 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:05.233 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:04 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:05.233 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:04 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:59:05.233 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:04 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:59:05.233 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:04 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:59:05.233 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:04 smithi012 ceph-mon[31695]: Deploying daemon mds.h on smithi173 2024-03-23T22:59:05.233 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:04 smithi012 ceph-mon[31695]: mds.? [v2:172.21.15.203:6836/438676939,v1:172.21.15.203:6837/438676939] up:boot 2024-03-23T22:59:05.233 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:04 smithi012 ceph-mon[31695]: fsmap 7 up:standby 2024-03-23T22:59:05.233 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:04 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-23T22:59:06.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:05 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3019013379' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:06.118 INFO:journalctl@ceph.mds.h.smithi173.stdout:Mar 23 22:59:06 smithi173 systemd[1]: Starting Ceph mds.h for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T22:59:06.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:05 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3019013379' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:06.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:05 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3019013379' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:06.501 INFO:journalctl@ceph.mds.h.smithi173.stdout:Mar 23 22:59:06 smithi173 podman[66215]: 2024-03-23 22:59:06.225236871 +0000 UTC m=+0.095059150 container create e4b852201291c8fff7f085c94186afbee0393f16c3a32f50be20b3f60a2b8f8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-23T22:59:06.501 INFO:journalctl@ceph.mds.h.smithi173.stdout:Mar 23 22:59:06 smithi173 podman[66215]: 2024-03-23 22:59:06.147416086 +0000 UTC m=+0.017238386 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-23T22:59:06.502 INFO:journalctl@ceph.mds.h.smithi173.stdout:Mar 23 22:59:06 smithi173 podman[66215]: 2024-03-23 22:59:06.503029664 +0000 UTC m=+0.372851954 container init e4b852201291c8fff7f085c94186afbee0393f16c3a32f50be20b3f60a2b8f8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-h, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-23T22:59:06.849 INFO:journalctl@ceph.mds.h.smithi173.stdout:Mar 23 22:59:06 smithi173 podman[66215]: 2024-03-23 22:59:06.505651226 +0000 UTC m=+0.375473511 container start e4b852201291c8fff7f085c94186afbee0393f16c3a32f50be20b3f60a2b8f8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-h, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-23T22:59:06.849 INFO:journalctl@ceph.mds.h.smithi173.stdout:Mar 23 22:59:06 smithi173 ceph-mds[66234]: -- 172.21.15.173:0/155090288 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561c55479a40 con 0x561c5544c800 2024-03-23T22:59:06.849 INFO:journalctl@ceph.mds.h.smithi173.stdout:Mar 23 22:59:06 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-h[66230]: starting mds.h at 2024-03-23T22:59:06.850 INFO:journalctl@ceph.mds.h.smithi173.stdout:Mar 23 22:59:06 smithi173 bash[66215]: e4b852201291c8fff7f085c94186afbee0393f16c3a32f50be20b3f60a2b8f8a 2024-03-23T22:59:06.850 INFO:journalctl@ceph.mds.h.smithi173.stdout:Mar 23 22:59:06 smithi173 systemd[1]: Started Ceph mds.h for 90f679ae-e967-11ee-95cd-87774f69a715. 2024-03-23T22:59:07.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:06 smithi173 ceph-mon[36116]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:07.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:06 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:07.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:06 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:07.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:06 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:07.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:06 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:59:07.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:06 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:59:07.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:06 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:59:07.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:06 smithi203 ceph-mon[36693]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:07.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:06 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:07.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:06 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:07.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:06 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:07.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:06 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:59:07.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:06 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:59:07.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:06 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:59:07.235 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T22:59:07.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:06 smithi012 ceph-mon[31695]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:07.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:06 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:07.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:06 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:07.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:06 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:07.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:06 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:59:07.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:06 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:59:07.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:06 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:59:07.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.465+0000 7f7f46400640 1 -- 172.21.15.12:0/1261699113 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f401016b0 msgr2=0x7f7f40101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:07.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.465+0000 7f7f46400640 1 --2- 172.21.15.12:0/1261699113 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f401016b0 0x7f7f40101ad0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7f30009860 tx=0x7f7f3002bb70 comp rx=0 tx=0).stop 2024-03-23T22:59:07.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.465+0000 7f7f46400640 1 -- 172.21.15.12:0/1261699113 shutdown_connections 2024-03-23T22:59:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.465+0000 7f7f46400640 1 --2- 172.21.15.12:0/1261699113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f40103260 0x7f7f40109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.465+0000 7f7f46400640 1 --2- 172.21.15.12:0/1261699113 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f401028a0 0x7f7f40102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.465+0000 7f7f46400640 1 --2- 172.21.15.12:0/1261699113 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f401016b0 0x7f7f40101ad0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.465+0000 7f7f46400640 1 -- 172.21.15.12:0/1261699113 >> 172.21.15.12:0/1261699113 conn(0x7f7f400fcc80 msgr2=0x7f7f400ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.465+0000 7f7f46400640 1 -- 172.21.15.12:0/1261699113 shutdown_connections 2024-03-23T22:59:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.465+0000 7f7f46400640 1 -- 172.21.15.12:0/1261699113 wait complete. 2024-03-23T22:59:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.467+0000 7f7f46400640 1 Processor -- start 2024-03-23T22:59:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.467+0000 7f7f46400640 1 -- start start 2024-03-23T22:59:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.467+0000 7f7f46400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f401028a0 0x7f7f40197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.467+0000 7f7f46400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f40103260 0x7f7f40198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.467+0000 7f7f46400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f4019d180 0x7f7f4019f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.467+0000 7f7f46400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4010d330 con 0x7f7f4019d180 2024-03-23T22:59:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.467+0000 7f7f46400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4010d1b0 con 0x7f7f401028a0 2024-03-23T22:59:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.467+0000 7f7f46400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4010d4b0 con 0x7f7f40103260 2024-03-23T22:59:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.468+0000 7f7f3f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f401028a0 0x7f7f40197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.468+0000 7f7f3ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f40103260 0x7f7f40198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.468+0000 7f7f3f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f401028a0 0x7f7f40197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:34984/0 (socket says 172.21.15.12:34984) 2024-03-23T22:59:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.468+0000 7f7f3fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f4019d180 0x7f7f4019f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.468+0000 7f7f3f400640 1 -- 172.21.15.12:0/2467509528 learned_addr learned my addr 172.21.15.12:0/2467509528 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.468+0000 7f7f3fe00640 1 -- 172.21.15.12:0/2467509528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f40103260 msgr2=0x7f7f40198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.468+0000 7f7f3fe00640 1 --2- 172.21.15.12:0/2467509528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f40103260 0x7f7f40198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.468+0000 7f7f3fe00640 1 -- 172.21.15.12:0/2467509528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f401028a0 msgr2=0x7f7f40197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.468+0000 7f7f3fe00640 1 --2- 172.21.15.12:0/2467509528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f401028a0 0x7f7f40197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.468+0000 7f7f3fe00640 1 -- 172.21.15.12:0/2467509528 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4019fbe0 con 0x7f7f4019d180 2024-03-23T22:59:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.468+0000 7f7f3ea00640 1 --2- 172.21.15.12:0/2467509528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f40103260 0x7f7f40198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T22:59:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.468+0000 7f7f3fe00640 1 --2- 172.21.15.12:0/2467509528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f4019d180 0x7f7f4019f570 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7f38009fd0 tx=0x7f7f3800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.469+0000 7f7f37e00640 1 -- 172.21.15.12:0/2467509528 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3800ce20 con 0x7f7f4019d180 2024-03-23T22:59:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.469+0000 7f7f37e00640 1 -- 172.21.15.12:0/2467509528 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7f38005020 con 0x7f7f4019d180 2024-03-23T22:59:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.469+0000 7f7f46400640 1 -- 172.21.15.12:0/2467509528 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f4019fed0 con 0x7f7f4019d180 2024-03-23T22:59:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.469+0000 7f7f37e00640 1 -- 172.21.15.12:0/2467509528 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f38010720 con 0x7f7f4019d180 2024-03-23T22:59:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.469+0000 7f7f46400640 1 -- 172.21.15.12:0/2467509528 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f401a03e0 con 0x7f7f4019d180 2024-03-23T22:59:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.470+0000 7f7f46400640 1 -- 172.21.15.12:0/2467509528 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f40108aa0 con 0x7f7f4019d180 2024-03-23T22:59:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.472+0000 7f7f37e00640 1 -- 172.21.15.12:0/2467509528 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7f38005540 con 0x7f7f4019d180 2024-03-23T22:59:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.474+0000 7f7f37e00640 1 --2- 172.21.15.12:0/2467509528 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7f240a2880 0x7f7f240a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.474+0000 7f7f37e00640 1 -- 172.21.15.12:0/2467509528 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7f380cebe0 con 0x7f7f4019d180 2024-03-23T22:59:07.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.475+0000 7f7f3f400640 1 --2- 172.21.15.12:0/2467509528 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7f240a2880 0x7f7f240a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:07.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.475+0000 7f7f3f400640 1 --2- 172.21.15.12:0/2467509528 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7f240a2880 0x7f7f240a4d40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7f30009830 tx=0x7f7f30034ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.479+0000 7f7f37e00640 1 -- 172.21.15.12:0/2467509528 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7f3809c080 con 0x7f7f4019d180 2024-03-23T22:59:07.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.721+0000 7f7f46400640 1 -- 172.21.15.12:0/2467509528 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f4002d070 con 0x7f7f4019d180 2024-03-23T22:59:07.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.722+0000 7f7f37e00640 1 -- 172.21.15.12:0/2467509528 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6874 (secure 0 0 0) 0x7f7f3809b980 con 0x7f7f4019d180 2024-03-23T22:59:07.721 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:59:07.721 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24521,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24554,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-23T22:59:07.722 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 9 2024-03-23T22:59:07.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.725+0000 7f7f46400640 1 -- 172.21.15.12:0/2467509528 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7f240a2880 msgr2=0x7f7f240a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:07.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.725+0000 7f7f46400640 1 --2- 172.21.15.12:0/2467509528 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7f240a2880 0x7f7f240a4d40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7f30009830 tx=0x7f7f30034ad0 comp rx=0 tx=0).stop 2024-03-23T22:59:07.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.725+0000 7f7f46400640 1 -- 172.21.15.12:0/2467509528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f4019d180 msgr2=0x7f7f4019f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:07.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.725+0000 7f7f46400640 1 --2- 172.21.15.12:0/2467509528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f4019d180 0x7f7f4019f570 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7f38009fd0 tx=0x7f7f3800ef10 comp rx=0 tx=0).stop 2024-03-23T22:59:07.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.725+0000 7f7f46400640 1 -- 172.21.15.12:0/2467509528 shutdown_connections 2024-03-23T22:59:07.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.725+0000 7f7f46400640 1 --2- 172.21.15.12:0/2467509528 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7f240a2880 0x7f7f240a4d40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:07.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.725+0000 7f7f46400640 1 --2- 172.21.15.12:0/2467509528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f4019d180 0x7f7f4019f570 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:07.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.725+0000 7f7f46400640 1 --2- 172.21.15.12:0/2467509528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f40103260 0x7f7f40198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:07.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.725+0000 7f7f46400640 1 --2- 172.21.15.12:0/2467509528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f401028a0 0x7f7f40197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:07.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.725+0000 7f7f46400640 1 -- 172.21.15.12:0/2467509528 >> 172.21.15.12:0/2467509528 conn(0x7f7f400fcc80 msgr2=0x7f7f40105c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:07.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.725+0000 7f7f46400640 1 -- 172.21.15.12:0/2467509528 shutdown_connections 2024-03-23T22:59:07.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:07.725+0000 7f7f46400640 1 -- 172.21.15.12:0/2467509528 wait complete. 2024-03-23T22:59:08.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:07 smithi012 ceph-mon[31695]: Deploying daemon mds.i on smithi203 2024-03-23T22:59:08.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:07 smithi012 ceph-mon[31695]: mds.? [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] up:boot 2024-03-23T22:59:08.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:07 smithi012 ceph-mon[31695]: fsmap 8 up:standby 2024-03-23T22:59:08.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:07 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-23T22:59:08.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:07 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2467509528' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:08.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:07 smithi173 ceph-mon[36116]: Deploying daemon mds.i on smithi203 2024-03-23T22:59:08.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:07 smithi173 ceph-mon[36116]: mds.? [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] up:boot 2024-03-23T22:59:08.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:07 smithi173 ceph-mon[36116]: fsmap 8 up:standby 2024-03-23T22:59:08.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:07 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-23T22:59:08.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:07 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2467509528' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:08.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:07 smithi203 ceph-mon[36693]: Deploying daemon mds.i on smithi203 2024-03-23T22:59:08.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:07 smithi203 ceph-mon[36693]: mds.? [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] up:boot 2024-03-23T22:59:08.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:07 smithi203 ceph-mon[36693]: fsmap 8 up:standby 2024-03-23T22:59:08.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:07 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-23T22:59:08.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:07 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2467509528' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:09.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:08 smithi173 ceph-mon[36116]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:09.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:08 smithi203 ceph-mon[36693]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:09.158 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 23 22:59:08 smithi203 systemd[1]: Starting Ceph mds.i for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T22:59:09.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:08 smithi012 ceph-mon[31695]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:09.469 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 23 22:59:09 smithi203 podman[68695]: 2024-03-23 22:59:09.105810881 +0000 UTC m=+0.018049629 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-23T22:59:09.470 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 23 22:59:09 smithi203 podman[68695]: 2024-03-23 22:59:09.251104244 +0000 UTC m=+0.163342979 container create 92d4d344701bdee28505373753be3365f944490888263d351c5f4f4858d2b310 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-i, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-23T22:59:09.470 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 23 22:59:09 smithi203 podman[68695]: 2024-03-23 22:59:09.470468717 +0000 UTC m=+0.382707464 container init 92d4d344701bdee28505373753be3365f944490888263d351c5f4f4858d2b310 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-i, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-23T22:59:09.745 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T22:59:09.907 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 23 22:59:09 smithi203 podman[68695]: 2024-03-23 22:59:09.473379103 +0000 UTC m=+0.385617839 container start 92d4d344701bdee28505373753be3365f944490888263d351c5f4f4858d2b310 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-i, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True) 2024-03-23T22:59:09.908 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 23 22:59:09 smithi203 ceph-mds[68714]: -- 172.21.15.203:0/263800659 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56019e469a40 con 0x56019e43d000 2024-03-23T22:59:09.908 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 23 22:59:09 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-i[68710]: starting mds.i at 2024-03-23T22:59:09.908 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 23 22:59:09 smithi203 bash[68695]: 92d4d344701bdee28505373753be3365f944490888263d351c5f4f4858d2b310 2024-03-23T22:59:09.908 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 23 22:59:09 smithi203 systemd[1]: Started Ceph mds.i for 90f679ae-e967-11ee-95cd-87774f69a715. 2024-03-23T22:59:09.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.950+0000 7fa023200640 1 -- 172.21.15.12:0/2883455419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0240756e0 msgr2=0x7fa024073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:09.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.950+0000 7fa023200640 1 --2- 172.21.15.12:0/2883455419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0240756e0 0x7fa024073b30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa014009900 tx=0x7fa01402bc30 comp rx=0 tx=0).stop 2024-03-23T22:59:09.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.951+0000 7fa023200640 1 -- 172.21.15.12:0/2883455419 shutdown_connections 2024-03-23T22:59:09.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.951+0000 7fa023200640 1 --2- 172.21.15.12:0/2883455419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa0241093a0 0x7fa02410b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:09.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.951+0000 7fa023200640 1 --2- 172.21.15.12:0/2883455419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa024074100 0x7fa024074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:09.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.951+0000 7fa023200640 1 --2- 172.21.15.12:0/2883455419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0240756e0 0x7fa024073b30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:09.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.951+0000 7fa023200640 1 -- 172.21.15.12:0/2883455419 >> 172.21.15.12:0/2883455419 conn(0x7fa0240fca30 msgr2=0x7fa0240fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:09.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.951+0000 7fa023200640 1 -- 172.21.15.12:0/2883455419 shutdown_connections 2024-03-23T22:59:09.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.951+0000 7fa023200640 1 -- 172.21.15.12:0/2883455419 wait complete. 2024-03-23T22:59:09.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.952+0000 7fa023200640 1 Processor -- start 2024-03-23T22:59:09.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.952+0000 7fa023200640 1 -- start start 2024-03-23T22:59:09.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.952+0000 7fa023200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa024074100 0x7fa024199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:09.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.952+0000 7fa023200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0240756e0 0x7fa024199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:09.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.952+0000 7fa023200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa0241093a0 0x7fa0241a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:09.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.952+0000 7fa023200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02410f370 con 0x7fa0240756e0 2024-03-23T22:59:09.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.952+0000 7fa023200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02410f1f0 con 0x7fa0241093a0 2024-03-23T22:59:09.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.952+0000 7fa023200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02410f4f0 con 0x7fa024074100 2024-03-23T22:59:09.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.953+0000 7fa021e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa024074100 0x7fa024199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:09.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.953+0000 7fa021e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa024074100 0x7fa024199a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55744/0 (socket says 172.21.15.12:55744) 2024-03-23T22:59:09.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.953+0000 7fa01be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0240756e0 0x7fa024199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:09.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.953+0000 7fa021e00640 1 -- 172.21.15.12:0/1697092587 learned_addr learned my addr 172.21.15.12:0/1697092587 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:09.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.953+0000 7fa022800640 1 --2- 172.21.15.12:0/1697092587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa0241093a0 0x7fa0241a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:09.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.953+0000 7fa01be00640 1 -- 172.21.15.12:0/1697092587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa024074100 msgr2=0x7fa024199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:09.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.953+0000 7fa01be00640 1 --2- 172.21.15.12:0/1697092587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa024074100 0x7fa024199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:09.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.953+0000 7fa01be00640 1 -- 172.21.15.12:0/1697092587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa0241093a0 msgr2=0x7fa0241a0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:09.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.953+0000 7fa01be00640 1 --2- 172.21.15.12:0/1697092587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa0241093a0 0x7fa0241a0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:09.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.953+0000 7fa01be00640 1 -- 172.21.15.12:0/1697092587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0241a1620 con 0x7fa0240756e0 2024-03-23T22:59:09.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.953+0000 7fa021e00640 1 --2- 172.21.15.12:0/1697092587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa024074100 0x7fa024199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T22:59:09.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.954+0000 7fa01be00640 1 --2- 172.21.15.12:0/1697092587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0240756e0 0x7fa024199f40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa00c00e9f0 tx=0x7fa00c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:09.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.954+0000 7fa01aa00640 1 -- 172.21.15.12:0/1697092587 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c00cd80 con 0x7fa0240756e0 2024-03-23T22:59:09.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.954+0000 7fa01aa00640 1 -- 172.21.15.12:0/1697092587 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa00c0035f0 con 0x7fa0240756e0 2024-03-23T22:59:09.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.954+0000 7fa023200640 1 -- 172.21.15.12:0/1697092587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0241a1910 con 0x7fa0240756e0 2024-03-23T22:59:09.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.954+0000 7fa01aa00640 1 -- 172.21.15.12:0/1697092587 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c010690 con 0x7fa0240756e0 2024-03-23T22:59:09.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.954+0000 7fa023200640 1 -- 172.21.15.12:0/1697092587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0241a1e60 con 0x7fa0240756e0 2024-03-23T22:59:09.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.955+0000 7fa023200640 1 -- 172.21.15.12:0/1697092587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0241080d0 con 0x7fa0240756e0 2024-03-23T22:59:09.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.957+0000 7fa01aa00640 1 -- 172.21.15.12:0/1697092587 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa00c003b30 con 0x7fa0240756e0 2024-03-23T22:59:09.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.958+0000 7fa01aa00640 1 --2- 172.21.15.12:0/1697092587 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa0080a2800 0x7fa0080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:09.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.959+0000 7fa01aa00640 1 -- 172.21.15.12:0/1697092587 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa00c0ce910 con 0x7fa0240756e0 2024-03-23T22:59:09.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.959+0000 7fa021e00640 1 --2- 172.21.15.12:0/1697092587 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa0080a2800 0x7fa0080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:09.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.959+0000 7fa021e00640 1 --2- 172.21.15.12:0/1697092587 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa0080a2800 0x7fa0080a4cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa01402c8e0 tx=0x7fa01402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:09.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:09.968+0000 7fa01aa00640 1 -- 172.21.15.12:0/1697092587 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa00c09bdb0 con 0x7fa0240756e0 2024-03-23T22:59:10.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:10.213+0000 7fa023200640 1 -- 172.21.15.12:0/1697092587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa02402d070 con 0x7fa0240756e0 2024-03-23T22:59:10.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:10.214+0000 7fa01aa00640 1 -- 172.21.15.12:0/1697092587 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7667 (secure 0 0 0) 0x7fa02402d070 con 0x7fa0240756e0 2024-03-23T22:59:10.213 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:59:10.213 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24521,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24554,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-23T22:59:10.213 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 10 2024-03-23T22:59:10.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:10.216+0000 7fa023200640 1 -- 172.21.15.12:0/1697092587 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa0080a2800 msgr2=0x7fa0080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:10.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:10.216+0000 7fa023200640 1 --2- 172.21.15.12:0/1697092587 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa0080a2800 0x7fa0080a4cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa01402c8e0 tx=0x7fa01402df90 comp rx=0 tx=0).stop 2024-03-23T22:59:10.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:10.216+0000 7fa023200640 1 -- 172.21.15.12:0/1697092587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0240756e0 msgr2=0x7fa024199f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:10.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:10.216+0000 7fa023200640 1 --2- 172.21.15.12:0/1697092587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0240756e0 0x7fa024199f40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa00c00e9f0 tx=0x7fa00c00eeb0 comp rx=0 tx=0).stop 2024-03-23T22:59:10.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:10.217+0000 7fa023200640 1 -- 172.21.15.12:0/1697092587 shutdown_connections 2024-03-23T22:59:10.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:10.217+0000 7fa023200640 1 --2- 172.21.15.12:0/1697092587 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa0080a2800 0x7fa0080a4cc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:10.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:10.217+0000 7fa023200640 1 --2- 172.21.15.12:0/1697092587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa0241093a0 0x7fa0241a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:10.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:10.217+0000 7fa023200640 1 --2- 172.21.15.12:0/1697092587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0240756e0 0x7fa024199f40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:10.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:10.217+0000 7fa023200640 1 --2- 172.21.15.12:0/1697092587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa024074100 0x7fa024199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:10.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:10.217+0000 7fa023200640 1 -- 172.21.15.12:0/1697092587 >> 172.21.15.12:0/1697092587 conn(0x7fa0240fca30 msgr2=0x7fa0241052a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:10.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:10.217+0000 7fa023200640 1 -- 172.21.15.12:0/1697092587 shutdown_connections 2024-03-23T22:59:10.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:10.217+0000 7fa023200640 1 -- 172.21.15.12:0/1697092587 wait complete. 2024-03-23T22:59:10.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:10 smithi173 ceph-mon[36116]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:10.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:10 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:10.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:10 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:10.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:10 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:10.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:10 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:59:10.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:10 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:59:10.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:10 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:59:10.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:10 smithi173 ceph-mon[36116]: Deploying daemon mds.g on smithi012 2024-03-23T22:59:10.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:10 smithi173 ceph-mon[36116]: mds.? [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] up:boot 2024-03-23T22:59:10.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:10 smithi173 ceph-mon[36116]: fsmap 9 up:standby 2024-03-23T22:59:10.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:10 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-23T22:59:10.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:10 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1697092587' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:10.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:10 smithi012 ceph-mon[31695]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:10.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:10 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:10.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:10 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:10.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:10 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:10.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:10 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:59:10.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:10 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:59:10.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:10 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:59:10.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:10 smithi012 ceph-mon[31695]: Deploying daemon mds.g on smithi012 2024-03-23T22:59:10.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:10 smithi012 ceph-mon[31695]: mds.? [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] up:boot 2024-03-23T22:59:10.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:10 smithi012 ceph-mon[31695]: fsmap 9 up:standby 2024-03-23T22:59:10.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:10 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-23T22:59:10.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:10 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1697092587' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:10.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:10 smithi203 ceph-mon[36693]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:10.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:10 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:10.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:10 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:10.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:10 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:10.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:10 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:59:10.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:10 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:59:10.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:10 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:59:10.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:10 smithi203 ceph-mon[36693]: Deploying daemon mds.g on smithi012 2024-03-23T22:59:10.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:10 smithi203 ceph-mon[36693]: mds.? [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] up:boot 2024-03-23T22:59:10.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:10 smithi203 ceph-mon[36693]: fsmap 9 up:standby 2024-03-23T22:59:10.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:10 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-23T22:59:10.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:10 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1697092587' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:12.238 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T22:59:12.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.590+0000 7fb9a1c00640 1 -- 172.21.15.12:0/712806267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb99c10d9e0 msgr2=0x7fb99c10de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:12.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.590+0000 7fb9a1c00640 1 --2- 172.21.15.12:0/712806267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb99c10d9e0 0x7fb99c10de90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb994009ca0 tx=0x7fb99402c120 comp rx=0 tx=0).stop 2024-03-23T22:59:12.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.591+0000 7fb9a1c00640 1 -- 172.21.15.12:0/712806267 shutdown_connections 2024-03-23T22:59:12.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.591+0000 7fb9a1c00640 1 --2- 172.21.15.12:0/712806267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb99c10d9e0 0x7fb99c10de90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:12.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.591+0000 7fb9a1c00640 1 --2- 172.21.15.12:0/712806267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb99c10b010 0x7fb99c10d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:12.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.591+0000 7fb9a1c00640 1 --2- 172.21.15.12:0/712806267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb99c1086a0 0x7fb99c10aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:12.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.591+0000 7fb9a1c00640 1 -- 172.21.15.12:0/712806267 >> 172.21.15.12:0/712806267 conn(0x7fb99c06e510 msgr2=0x7fb99c070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:12.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.591+0000 7fb9a1c00640 1 -- 172.21.15.12:0/712806267 shutdown_connections 2024-03-23T22:59:12.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.592+0000 7fb9a1c00640 1 -- 172.21.15.12:0/712806267 wait complete. 2024-03-23T22:59:12.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.593+0000 7fb9a1c00640 1 Processor -- start 2024-03-23T22:59:12.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.593+0000 7fb9a1c00640 1 -- start start 2024-03-23T22:59:12.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.594+0000 7fb9a1c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb99c1086a0 0x7fb99c1a6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:12.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.594+0000 7fb9a1c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb99c10b010 0x7fb99c1a6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:12.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.594+0000 7fb9a1c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb99c10d9e0 0x7fb99c1ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:12.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.594+0000 7fb9a1c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c11c1f0 con 0x7fb99c1086a0 2024-03-23T22:59:12.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.594+0000 7fb9a1c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c11c070 con 0x7fb99c10b010 2024-03-23T22:59:12.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.594+0000 7fb9a1c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c11c370 con 0x7fb99c10d9e0 2024-03-23T22:59:12.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.594+0000 7fb99aa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb99c1086a0 0x7fb99c1a6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:12.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.594+0000 7fb99aa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb99c1086a0 0x7fb99c1a6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59498/0 (socket says 172.21.15.12:59498) 2024-03-23T22:59:12.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.594+0000 7fb99aa00640 1 -- 172.21.15.12:0/3281276074 learned_addr learned my addr 172.21.15.12:0/3281276074 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:12.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.594+0000 7fb99a000640 1 --2- 172.21.15.12:0/3281276074 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb99c10b010 0x7fb99c1a6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:12.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.594+0000 7fb99b400640 1 --2- 172.21.15.12:0/3281276074 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb99c10d9e0 0x7fb99c1ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:12.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.595+0000 7fb99b400640 1 -- 172.21.15.12:0/3281276074 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb99c10b010 msgr2=0x7fb99c1a6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:12.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.595+0000 7fb99b400640 1 --2- 172.21.15.12:0/3281276074 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb99c10b010 0x7fb99c1a6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:12.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.595+0000 7fb99b400640 1 -- 172.21.15.12:0/3281276074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb99c1086a0 msgr2=0x7fb99c1a6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:12.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.595+0000 7fb99b400640 1 --2- 172.21.15.12:0/3281276074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb99c1086a0 0x7fb99c1a6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:12.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.595+0000 7fb99b400640 1 -- 172.21.15.12:0/3281276074 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb99c1ae500 con 0x7fb99c10d9e0 2024-03-23T22:59:12.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.595+0000 7fb99b400640 1 --2- 172.21.15.12:0/3281276074 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb99c10d9e0 0x7fb99c1ade90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb994032530 tx=0x7fb994035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:12.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.595+0000 7fb993400640 1 -- 172.21.15.12:0/3281276074 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb994039040 con 0x7fb99c10d9e0 2024-03-23T22:59:12.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.595+0000 7fb993400640 1 -- 172.21.15.12:0/3281276074 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb994035e00 con 0x7fb99c10d9e0 2024-03-23T22:59:12.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.595+0000 7fb993400640 1 -- 172.21.15.12:0/3281276074 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb994004400 con 0x7fb99c10d9e0 2024-03-23T22:59:12.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.596+0000 7fb9a1c00640 1 -- 172.21.15.12:0/3281276074 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb99c1ae790 con 0x7fb99c10d9e0 2024-03-23T22:59:12.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.596+0000 7fb9a1c00640 1 -- 172.21.15.12:0/3281276074 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb99c1aed20 con 0x7fb99c10d9e0 2024-03-23T22:59:12.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.597+0000 7fb9a1c00640 1 -- 172.21.15.12:0/3281276074 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb99c071010 con 0x7fb99c10d9e0 2024-03-23T22:59:12.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.601+0000 7fb993400640 1 -- 172.21.15.12:0/3281276074 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb994046030 con 0x7fb99c10d9e0 2024-03-23T22:59:12.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.602+0000 7fb993400640 1 --2- 172.21.15.12:0/3281276074 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb9800a28d0 0x7fb9800a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:12.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.602+0000 7fb993400640 1 -- 172.21.15.12:0/3281276074 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb9940f02a0 con 0x7fb99c10d9e0 2024-03-23T22:59:12.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.602+0000 7fb99aa00640 1 --2- 172.21.15.12:0/3281276074 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb9800a28d0 0x7fb9800a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:12.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.603+0000 7fb99aa00640 1 --2- 172.21.15.12:0/3281276074 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb9800a28d0 0x7fb9800a4d90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb98c006fd0 tx=0x7fb98c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:12.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.606+0000 7fb993400640 1 -- 172.21.15.12:0/3281276074 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb99404d060 con 0x7fb99c10d9e0 2024-03-23T22:59:12.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.856+0000 7fb9a1c00640 1 -- 172.21.15.12:0/3281276074 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb99c0622a0 con 0x7fb99c10d9e0 2024-03-23T22:59:12.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.857+0000 7fb993400640 1 -- 172.21.15.12:0/3281276074 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7667 (secure 0 0 0) 0x7fb9940bd7d0 con 0x7fb99c10d9e0 2024-03-23T22:59:12.857 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:59:12.858 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24521,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24554,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-23T22:59:12.858 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 10 2024-03-23T22:59:12.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.860+0000 7fb9a1c00640 1 -- 172.21.15.12:0/3281276074 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb9800a28d0 msgr2=0x7fb9800a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:12.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.860+0000 7fb9a1c00640 1 --2- 172.21.15.12:0/3281276074 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb9800a28d0 0x7fb9800a4d90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb98c006fd0 tx=0x7fb98c009190 comp rx=0 tx=0).stop 2024-03-23T22:59:12.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.860+0000 7fb9a1c00640 1 -- 172.21.15.12:0/3281276074 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb99c10d9e0 msgr2=0x7fb99c1ade90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:12.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.860+0000 7fb9a1c00640 1 --2- 172.21.15.12:0/3281276074 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb99c10d9e0 0x7fb99c1ade90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb994032530 tx=0x7fb994035d10 comp rx=0 tx=0).stop 2024-03-23T22:59:12.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.860+0000 7fb9a1c00640 1 -- 172.21.15.12:0/3281276074 shutdown_connections 2024-03-23T22:59:12.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.860+0000 7fb9a1c00640 1 --2- 172.21.15.12:0/3281276074 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb9800a28d0 0x7fb9800a4d90 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:12.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.860+0000 7fb9a1c00640 1 --2- 172.21.15.12:0/3281276074 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb99c10d9e0 0x7fb99c1ade90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:12.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.860+0000 7fb9a1c00640 1 --2- 172.21.15.12:0/3281276074 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb99c10b010 0x7fb99c1a6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:12.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.860+0000 7fb9a1c00640 1 --2- 172.21.15.12:0/3281276074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb99c1086a0 0x7fb99c1a6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:12.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.860+0000 7fb9a1c00640 1 -- 172.21.15.12:0/3281276074 >> 172.21.15.12:0/3281276074 conn(0x7fb99c06e510 msgr2=0x7fb99c109c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:12.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.860+0000 7fb9a1c00640 1 -- 172.21.15.12:0/3281276074 shutdown_connections 2024-03-23T22:59:12.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:12.860+0000 7fb9a1c00640 1 -- 172.21.15.12:0/3281276074 wait complete. 2024-03-23T22:59:12.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:12 smithi203 ceph-mon[36693]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:13.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:12 smithi012 ceph-mon[31695]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:13.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:12 smithi173 ceph-mon[36116]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:13.494 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 23 22:59:13 smithi012 podman[72984]: 2024-03-23 22:59:13.213736506 +0000 UTC m=+0.172107242 container create b3ca6f1c01d5cbeefe4069601c8a0970d6063de86a7e8337aa65eb5259845e2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-23T22:59:13.495 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 23 22:59:13 smithi012 podman[72984]: 2024-03-23 22:59:13.49272893 +0000 UTC m=+0.451099673 container init b3ca6f1c01d5cbeefe4069601c8a0970d6063de86a7e8337aa65eb5259845e2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-g, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-23T22:59:13.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:13 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3281276074' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:13.804 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 23 22:59:13 smithi012 podman[72984]: 2024-03-23 22:59:13.503254094 +0000 UTC m=+0.461624860 container start b3ca6f1c01d5cbeefe4069601c8a0970d6063de86a7e8337aa65eb5259845e2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-23T22:59:13.804 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 23 22:59:13 smithi012 ceph-mds[73004]: -- 172.21.15.12:0/1593887529 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564bf45d3a40 con 0x564bf45a6c00 2024-03-23T22:59:13.804 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 23 22:59:13 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-g[73000]: starting mds.g at 2024-03-23T22:59:13.804 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 23 22:59:13 smithi012 bash[72984]: b3ca6f1c01d5cbeefe4069601c8a0970d6063de86a7e8337aa65eb5259845e2e 2024-03-23T22:59:13.804 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 23 22:59:13 smithi012 systemd[1]: Started Ceph mds.g for 90f679ae-e967-11ee-95cd-87774f69a715. 2024-03-23T22:59:13.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:13 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3281276074' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:14.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:13 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3281276074' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:14.881 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T22:59:14.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:14 smithi203 ceph-mon[36693]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:14.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:14 smithi203 ceph-mon[36693]: mds.? [v2:172.21.15.12:6838/771072116,v1:172.21.15.12:6839/771072116] up:boot 2024-03-23T22:59:14.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:14 smithi203 ceph-mon[36693]: fsmap 10 up:standby 2024-03-23T22:59:14.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:14 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-23T22:59:14.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:14 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:14.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:14 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:14.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:14 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:14.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:14 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:59:14.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:14 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:59:14.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:14 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:59:14.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:14 smithi203 ceph-mon[36693]: Deploying daemon mds.a on smithi012 2024-03-23T22:59:14.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:14 smithi012 ceph-mon[31695]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:14.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:14 smithi012 ceph-mon[31695]: mds.? [v2:172.21.15.12:6838/771072116,v1:172.21.15.12:6839/771072116] up:boot 2024-03-23T22:59:14.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:14 smithi012 ceph-mon[31695]: fsmap 10 up:standby 2024-03-23T22:59:14.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-23T22:59:14.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:14.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:14.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:14.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:59:14.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:59:14.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:59:14.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:14 smithi012 ceph-mon[31695]: Deploying daemon mds.a on smithi012 2024-03-23T22:59:15.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.115+0000 7f9930400640 1 -- 172.21.15.12:0/2790062616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99281028a0 msgr2=0x7f9928102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:15.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.115+0000 7f9930400640 1 --2- 172.21.15.12:0/2790062616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99281028a0 0x7f9928102d20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9914009900 tx=0x7f991402bc30 comp rx=0 tx=0).stop 2024-03-23T22:59:15.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.116+0000 7f9930400640 1 -- 172.21.15.12:0/2790062616 shutdown_connections 2024-03-23T22:59:15.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.116+0000 7f9930400640 1 --2- 172.21.15.12:0/2790062616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9928103260 0x7f9928109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:15.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.116+0000 7f9930400640 1 --2- 172.21.15.12:0/2790062616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99281028a0 0x7f9928102d20 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:15.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.116+0000 7f9930400640 1 --2- 172.21.15.12:0/2790062616 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99281016b0 0x7f9928101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:15.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.116+0000 7f9930400640 1 -- 172.21.15.12:0/2790062616 >> 172.21.15.12:0/2790062616 conn(0x7f99280fcc80 msgr2=0x7f99280ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:15.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.116+0000 7f9930400640 1 -- 172.21.15.12:0/2790062616 shutdown_connections 2024-03-23T22:59:15.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.116+0000 7f9930400640 1 -- 172.21.15.12:0/2790062616 wait complete. 2024-03-23T22:59:15.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.117+0000 7f9930400640 1 Processor -- start 2024-03-23T22:59:15.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.117+0000 7f9930400640 1 -- start start 2024-03-23T22:59:15.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.117+0000 7f9930400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99281016b0 0x7f9928199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:15.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.117+0000 7f9930400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99281028a0 0x7f992819a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:15.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.117+0000 7f9930400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9928103260 0x7f99281a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:15.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.117+0000 7f9930400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992810d360 con 0x7f99281028a0 2024-03-23T22:59:15.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.117+0000 7f9930400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992810d1e0 con 0x7f99281016b0 2024-03-23T22:59:15.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.117+0000 7f9930400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992810d4e0 con 0x7f9928103260 2024-03-23T22:59:15.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.117+0000 7f992d000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99281028a0 0x7f992819a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:15.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.117+0000 7f992d000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99281028a0 0x7f992819a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53336/0 (socket says 172.21.15.12:53336) 2024-03-23T22:59:15.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.117+0000 7f992d000640 1 -- 172.21.15.12:0/705497256 learned_addr learned my addr 172.21.15.12:0/705497256 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:15.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.117+0000 7f992da00640 1 --2- 172.21.15.12:0/705497256 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99281016b0 0x7f9928199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:15.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.117+0000 7f992e400640 1 --2- 172.21.15.12:0/705497256 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9928103260 0x7f99281a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:15.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.118+0000 7f992d000640 1 -- 172.21.15.12:0/705497256 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9928103260 msgr2=0x7f99281a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:15.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.118+0000 7f992d000640 1 --2- 172.21.15.12:0/705497256 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9928103260 0x7f99281a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:15.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.118+0000 7f992d000640 1 -- 172.21.15.12:0/705497256 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99281016b0 msgr2=0x7f9928199c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:15.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.118+0000 7f992d000640 1 --2- 172.21.15.12:0/705497256 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99281016b0 0x7f9928199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:15.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.118+0000 7f992d000640 1 -- 172.21.15.12:0/705497256 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99281a1870 con 0x7f99281028a0 2024-03-23T22:59:15.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.118+0000 7f992d000640 1 --2- 172.21.15.12:0/705497256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99281028a0 0x7f992819a1a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f991402c8e0 tx=0x7f991402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:15.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.118+0000 7f9926000640 1 -- 172.21.15.12:0/705497256 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9914043020 con 0x7f99281028a0 2024-03-23T22:59:15.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.118+0000 7f9930400640 1 -- 172.21.15.12:0/705497256 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99281a1b00 con 0x7f99281028a0 2024-03-23T22:59:15.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.118+0000 7f9926000640 1 -- 172.21.15.12:0/705497256 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f99140054e0 con 0x7f99281028a0 2024-03-23T22:59:15.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.118+0000 7f9926000640 1 -- 172.21.15.12:0/705497256 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9914003c70 con 0x7f99281028a0 2024-03-23T22:59:15.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.119+0000 7f9930400640 1 -- 172.21.15.12:0/705497256 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99281a2050 con 0x7f99281028a0 2024-03-23T22:59:15.121 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:14 smithi173 ceph-mon[36116]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:15.121 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:14 smithi173 ceph-mon[36116]: mds.? [v2:172.21.15.12:6838/771072116,v1:172.21.15.12:6839/771072116] up:boot 2024-03-23T22:59:15.121 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:14 smithi173 ceph-mon[36116]: fsmap 10 up:standby 2024-03-23T22:59:15.121 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:14 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-23T22:59:15.121 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:14 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:15.121 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:14 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:15.121 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:14 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:15.121 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:14 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:59:15.121 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:14 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:59:15.121 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:14 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:59:15.121 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:14 smithi173 ceph-mon[36116]: Deploying daemon mds.a on smithi012 2024-03-23T22:59:15.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.120+0000 7f9930400640 1 -- 172.21.15.12:0/705497256 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99281a2220 con 0x7f99281028a0 2024-03-23T22:59:15.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.122+0000 7f9926000640 1 -- 172.21.15.12:0/705497256 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9914046020 con 0x7f99281028a0 2024-03-23T22:59:15.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.123+0000 7f9926000640 1 --2- 172.21.15.12:0/705497256 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f99100a2800 0x7f99100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:15.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.123+0000 7f9926000640 1 -- 172.21.15.12:0/705497256 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f99140f0770 con 0x7f99281028a0 2024-03-23T22:59:15.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.123+0000 7f992da00640 1 --2- 172.21.15.12:0/705497256 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f99100a2800 0x7f99100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:15.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.124+0000 7f992da00640 1 --2- 172.21.15.12:0/705497256 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f99100a2800 0x7f99100a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f992819b130 tx=0x7f991c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:15.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.128+0000 7f9926000640 1 -- 172.21.15.12:0/705497256 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f991404d020 con 0x7f99281028a0 2024-03-23T22:59:15.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.386+0000 7f9930400640 1 -- 172.21.15.12:0/705497256 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99281a23f0 con 0x7f99281028a0 2024-03-23T22:59:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.388+0000 7f9926000640 1 -- 172.21.15.12:0/705497256 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8456 (secure 0 0 0) 0x7f99140bdc10 con 0x7f99281028a0 2024-03-23T22:59:15.388 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:59:15.388 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24521,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24554,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-23T22:59:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 11 2024-03-23T22:59:15.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.391+0000 7f9930400640 1 -- 172.21.15.12:0/705497256 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f99100a2800 msgr2=0x7f99100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:15.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.391+0000 7f9930400640 1 --2- 172.21.15.12:0/705497256 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f99100a2800 0x7f99100a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f992819b130 tx=0x7f991c005f90 comp rx=0 tx=0).stop 2024-03-23T22:59:15.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.391+0000 7f9930400640 1 -- 172.21.15.12:0/705497256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99281028a0 msgr2=0x7f992819a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:15.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.391+0000 7f9930400640 1 --2- 172.21.15.12:0/705497256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99281028a0 0x7f992819a1a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f991402c8e0 tx=0x7f991402c650 comp rx=0 tx=0).stop 2024-03-23T22:59:15.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.391+0000 7f9930400640 1 -- 172.21.15.12:0/705497256 shutdown_connections 2024-03-23T22:59:15.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.391+0000 7f9930400640 1 --2- 172.21.15.12:0/705497256 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f99100a2800 0x7f99100a4cc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:15.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.391+0000 7f9930400640 1 --2- 172.21.15.12:0/705497256 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9928103260 0x7f99281a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:15.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.391+0000 7f9930400640 1 --2- 172.21.15.12:0/705497256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99281028a0 0x7f992819a1a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.391+0000 7f9930400640 1 --2- 172.21.15.12:0/705497256 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99281016b0 0x7f9928199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.391+0000 7f9930400640 1 -- 172.21.15.12:0/705497256 >> 172.21.15.12:0/705497256 conn(0x7f99280fcc80 msgr2=0x7f99280738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.391+0000 7f9930400640 1 -- 172.21.15.12:0/705497256 shutdown_connections 2024-03-23T22:59:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:15.392+0000 7f9930400640 1 -- 172.21.15.12:0/705497256 wait complete. 2024-03-23T22:59:15.952 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:15 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/705497256' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:16.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:15 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/705497256' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:16.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:15 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/705497256' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:17.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:16 smithi012 ceph-mon[31695]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:17.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:16 smithi173 ceph-mon[36116]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:17.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:16 smithi203 ceph-mon[36693]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:17.410 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T22:59:17.475 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 23 22:59:17 smithi012 systemd[1]: Starting Ceph mds.a for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T22:59:17.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.603+0000 7f505a200640 1 -- 172.21.15.12:0/3775843444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f505410dab0 msgr2=0x7f505410fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:17.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.603+0000 7f505a200640 1 --2- 172.21.15.12:0/3775843444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f505410dab0 0x7f505410fea0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f504c009cd0 tx=0x7f504c02c0f0 comp rx=0 tx=0).stop 2024-03-23T22:59:17.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.603+0000 7f505a200640 1 -- 172.21.15.12:0/3775843444 shutdown_connections 2024-03-23T22:59:17.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.603+0000 7f505a200640 1 --2- 172.21.15.12:0/3775843444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f505410dab0 0x7f505410fea0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:17.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.603+0000 7f505a200640 1 --2- 172.21.15.12:0/3775843444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5054074100 0x7f5054074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:17.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.603+0000 7f505a200640 1 --2- 172.21.15.12:0/3775843444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50540756e0 0x7f5054073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:17.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.603+0000 7f505a200640 1 -- 172.21.15.12:0/3775843444 >> 172.21.15.12:0/3775843444 conn(0x7f50540fca50 msgr2=0x7f50540fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:17.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.603+0000 7f505a200640 1 -- 172.21.15.12:0/3775843444 shutdown_connections 2024-03-23T22:59:17.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.603+0000 7f505a200640 1 -- 172.21.15.12:0/3775843444 wait complete. 2024-03-23T22:59:17.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.604+0000 7f505a200640 1 Processor -- start 2024-03-23T22:59:17.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.604+0000 7f505a200640 1 -- start start 2024-03-23T22:59:17.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.604+0000 7f505a200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5054074100 0x7f505419ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:17.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.604+0000 7f505a200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50540756e0 0x7f505419e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:17.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.604+0000 7f505a200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f505410dab0 0x7f50541a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:17.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.604+0000 7f505a200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5054113750 con 0x7f50540756e0 2024-03-23T22:59:17.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.604+0000 7f505a200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50541135d0 con 0x7f5054074100 2024-03-23T22:59:17.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.604+0000 7f505a200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50541138d0 con 0x7f505410dab0 2024-03-23T22:59:17.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.605+0000 7f5053e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50540756e0 0x7f505419e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:17.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.605+0000 7f5058e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5054074100 0x7f505419ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:17.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.605+0000 7f5058e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5054074100 0x7f505419ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:56584/0 (socket says 172.21.15.12:56584) 2024-03-23T22:59:17.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.605+0000 7f5058e00640 1 -- 172.21.15.12:0/3329487117 learned_addr learned my addr 172.21.15.12:0/3329487117 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:17.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.605+0000 7f5059800640 1 --2- 172.21.15.12:0/3329487117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f505410dab0 0x7f50541a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:17.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.605+0000 7f5059800640 1 -- 172.21.15.12:0/3329487117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5054074100 msgr2=0x7f505419ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:17.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.605+0000 7f5059800640 1 --2- 172.21.15.12:0/3329487117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5054074100 0x7f505419ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:17.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.605+0000 7f5059800640 1 -- 172.21.15.12:0/3329487117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50540756e0 msgr2=0x7f505419e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:17.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.605+0000 7f5059800640 1 --2- 172.21.15.12:0/3329487117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50540756e0 0x7f505419e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:17.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.605+0000 7f5059800640 1 -- 172.21.15.12:0/3329487117 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50541a59e0 con 0x7f505410dab0 2024-03-23T22:59:17.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.605+0000 7f5059800640 1 --2- 172.21.15.12:0/3329487117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f505410dab0 0x7f50541a5370 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f504c009bf0 tx=0x7f504c032ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:17.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.606+0000 7f5051600640 1 -- 172.21.15.12:0/3329487117 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504c041020 con 0x7f505410dab0 2024-03-23T22:59:17.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.606+0000 7f5051600640 1 -- 172.21.15.12:0/3329487117 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f504c034cd0 con 0x7f505410dab0 2024-03-23T22:59:17.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.606+0000 7f505a200640 1 -- 172.21.15.12:0/3329487117 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50541a5c70 con 0x7f505410dab0 2024-03-23T22:59:17.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.606+0000 7f5051600640 1 -- 172.21.15.12:0/3329487117 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504c039040 con 0x7f505410dab0 2024-03-23T22:59:17.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.606+0000 7f505a200640 1 -- 172.21.15.12:0/3329487117 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50541a6130 con 0x7f505410dab0 2024-03-23T22:59:17.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.607+0000 7f505a200640 1 -- 172.21.15.12:0/3329487117 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5054103b70 con 0x7f505410dab0 2024-03-23T22:59:17.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.609+0000 7f5051600640 1 -- 172.21.15.12:0/3329487117 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f504c044020 con 0x7f505410dab0 2024-03-23T22:59:17.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.610+0000 7f5051600640 1 --2- 172.21.15.12:0/3329487117 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f503c0a2800 0x7f503c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:17.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.611+0000 7f5051600640 1 -- 172.21.15.12:0/3329487117 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f504c0f0770 con 0x7f505410dab0 2024-03-23T22:59:17.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.611+0000 7f5058e00640 1 --2- 172.21.15.12:0/3329487117 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f503c0a2800 0x7f503c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:17.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.611+0000 7f5058e00640 1 --2- 172.21.15.12:0/3329487117 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f503c0a2800 0x7f503c0a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5048009cd0 tx=0x7f5048005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:17.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.613+0000 7f5051600640 1 -- 172.21.15.12:0/3329487117 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f504c04b020 con 0x7f505410dab0 2024-03-23T22:59:17.741 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 23 22:59:17 smithi012 podman[73329]: 2024-03-23 22:59:17.446041313 +0000 UTC m=+0.034149241 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-23T22:59:17.742 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 23 22:59:17 smithi012 podman[73329]: 2024-03-23 22:59:17.569835523 +0000 UTC m=+0.157943483 container create 48a7fe6e2e688de9b2609ed0dcb9f229aa5a3b1aaca668b80000aaf350281e7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-23T22:59:17.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.852+0000 7f505a200640 1 -- 172.21.15.12:0/3329487117 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f505402d230 con 0x7f505410dab0 2024-03-23T22:59:17.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.853+0000 7f5051600640 1 -- 172.21.15.12:0/3329487117 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8456 (secure 0 0 0) 0x7f504c0bdc10 con 0x7f505410dab0 2024-03-23T22:59:17.853 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:59:17.854 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24521,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24554,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-23T22:59:17.854 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 11 2024-03-23T22:59:17.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.856+0000 7f505a200640 1 -- 172.21.15.12:0/3329487117 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f503c0a2800 msgr2=0x7f503c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:17.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.856+0000 7f505a200640 1 --2- 172.21.15.12:0/3329487117 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f503c0a2800 0x7f503c0a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5048009cd0 tx=0x7f5048005f90 comp rx=0 tx=0).stop 2024-03-23T22:59:17.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.856+0000 7f505a200640 1 -- 172.21.15.12:0/3329487117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f505410dab0 msgr2=0x7f50541a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:17.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.856+0000 7f505a200640 1 --2- 172.21.15.12:0/3329487117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f505410dab0 0x7f50541a5370 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f504c009bf0 tx=0x7f504c032ea0 comp rx=0 tx=0).stop 2024-03-23T22:59:17.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.856+0000 7f505a200640 1 -- 172.21.15.12:0/3329487117 shutdown_connections 2024-03-23T22:59:17.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.856+0000 7f505a200640 1 --2- 172.21.15.12:0/3329487117 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f503c0a2800 0x7f503c0a4cc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:17.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.856+0000 7f505a200640 1 --2- 172.21.15.12:0/3329487117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f505410dab0 0x7f50541a5370 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:17.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.856+0000 7f505a200640 1 --2- 172.21.15.12:0/3329487117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50540756e0 0x7f505419e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:17.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.856+0000 7f505a200640 1 --2- 172.21.15.12:0/3329487117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5054074100 0x7f505419ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:17.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.856+0000 7f505a200640 1 -- 172.21.15.12:0/3329487117 >> 172.21.15.12:0/3329487117 conn(0x7f50540fca50 msgr2=0x7f5054100d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:17.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.857+0000 7f505a200640 1 -- 172.21.15.12:0/3329487117 shutdown_connections 2024-03-23T22:59:17.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:17.857+0000 7f505a200640 1 -- 172.21.15.12:0/3329487117 wait complete. 2024-03-23T22:59:18.304 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 23 22:59:17 smithi012 podman[73329]: 2024-03-23 22:59:17.84937145 +0000 UTC m=+0.437479417 container init 48a7fe6e2e688de9b2609ed0dcb9f229aa5a3b1aaca668b80000aaf350281e7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-a, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-23T22:59:18.304 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 23 22:59:17 smithi012 podman[73329]: 2024-03-23 22:59:17.859939892 +0000 UTC m=+0.448047825 container start 48a7fe6e2e688de9b2609ed0dcb9f229aa5a3b1aaca668b80000aaf350281e7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-a, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-23T22:59:18.304 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 23 22:59:17 smithi012 ceph-mds[73389]: -- 172.21.15.12:0/33769704 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557978f1da40 con 0x557978ef0c00 2024-03-23T22:59:18.304 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 23 22:59:17 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-a[73383]: starting mds.a at 2024-03-23T22:59:18.304 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 23 22:59:17 smithi012 bash[73329]: 48a7fe6e2e688de9b2609ed0dcb9f229aa5a3b1aaca668b80000aaf350281e7f 2024-03-23T22:59:18.305 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 23 22:59:17 smithi012 systemd[1]: Started Ceph mds.a for 90f679ae-e967-11ee-95cd-87774f69a715. 2024-03-23T22:59:18.939 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:18 smithi173 ceph-mon[36116]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:18.939 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:18 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3329487117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:18.939 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:18 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:18.940 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:18 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:18.940 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:18 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:18.940 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:18 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:59:18.940 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:18 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:59:18.940 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:18 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:59:18.940 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:18 smithi173 ceph-mon[36116]: Deploying daemon mds.b on smithi173 2024-03-23T22:59:19.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:18 smithi012 ceph-mon[31695]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:19.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:18 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3329487117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:19.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:18 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:19.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:18 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:19.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:18 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:19.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:18 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:59:19.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:18 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:59:19.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:18 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:59:19.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:18 smithi012 ceph-mon[31695]: Deploying daemon mds.b on smithi173 2024-03-23T22:59:19.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:18 smithi203 ceph-mon[36693]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:19.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:18 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3329487117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:19.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:18 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:19.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:18 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:19.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:18 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:19.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:18 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T22:59:19.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:18 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T22:59:19.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:18 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:59:19.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:18 smithi203 ceph-mon[36693]: Deploying daemon mds.b on smithi173 2024-03-23T22:59:19.876 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T22:59:19.967 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:19 smithi012 ceph-mon[31695]: mds.? [v2:172.21.15.12:6840/732195840,v1:172.21.15.12:6841/732195840] up:boot 2024-03-23T22:59:19.967 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:19 smithi012 ceph-mon[31695]: fsmap 11 up:standby 2024-03-23T22:59:19.967 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:19 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-23T22:59:20.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.112+0000 7f0787e00640 1 -- 172.21.15.12:0/303545926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f078810dae0 msgr2=0x7f078810fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:20.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.112+0000 7f0787e00640 1 --2- 172.21.15.12:0/303545926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f078810dae0 0x7f078810fed0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0780009cd0 tx=0x7f078002c0e0 comp rx=0 tx=0).stop 2024-03-23T22:59:20.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.112+0000 7f0787e00640 1 -- 172.21.15.12:0/303545926 shutdown_connections 2024-03-23T22:59:20.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.112+0000 7f0787e00640 1 --2- 172.21.15.12:0/303545926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f078810dae0 0x7f078810fed0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:20.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.112+0000 7f0787e00640 1 --2- 172.21.15.12:0/303545926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0788074100 0x7f0788074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:20.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.112+0000 7f0787e00640 1 --2- 172.21.15.12:0/303545926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07880756e0 0x7f0788073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:20.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.112+0000 7f0787e00640 1 -- 172.21.15.12:0/303545926 >> 172.21.15.12:0/303545926 conn(0x7f07880fca80 msgr2=0x7f07880feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:20.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.112+0000 7f0787e00640 1 -- 172.21.15.12:0/303545926 shutdown_connections 2024-03-23T22:59:20.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.112+0000 7f0787e00640 1 -- 172.21.15.12:0/303545926 wait complete. 2024-03-23T22:59:20.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.113+0000 7f0787e00640 1 Processor -- start 2024-03-23T22:59:20.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.113+0000 7f0787e00640 1 -- start start 2024-03-23T22:59:20.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.113+0000 7f0787e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0788074100 0x7f078819ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:20.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.113+0000 7f0787e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07880756e0 0x7f078819e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:20.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.113+0000 7f0787e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f078810dae0 0x7f07881a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:20.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.113+0000 7f0787e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0788113760 con 0x7f078810dae0 2024-03-23T22:59:20.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.113+0000 7f0787e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07881135e0 con 0x7f07880756e0 2024-03-23T22:59:20.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.113+0000 7f0787e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07881138e0 con 0x7f0788074100 2024-03-23T22:59:20.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.114+0000 7f077e000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07880756e0 0x7f078819e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:20.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.114+0000 7f077e000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07880756e0 0x7f078819e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:56634/0 (socket says 172.21.15.12:56634) 2024-03-23T22:59:20.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.114+0000 7f077e000640 1 -- 172.21.15.12:0/844713834 learned_addr learned my addr 172.21.15.12:0/844713834 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:20.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.114+0000 7f0787400640 1 --2- 172.21.15.12:0/844713834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f078810dae0 0x7f07881a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:20.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.114+0000 7f0786a00640 1 --2- 172.21.15.12:0/844713834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0788074100 0x7f078819ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:20.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.114+0000 7f077e000640 1 -- 172.21.15.12:0/844713834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0788074100 msgr2=0x7f078819ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:20.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.114+0000 7f077e000640 1 --2- 172.21.15.12:0/844713834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0788074100 0x7f078819ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:20.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.114+0000 7f077e000640 1 -- 172.21.15.12:0/844713834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f078810dae0 msgr2=0x7f07881a5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:20.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.114+0000 7f077e000640 1 --2- 172.21.15.12:0/844713834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f078810dae0 0x7f07881a5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:20.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.114+0000 7f077e000640 1 -- 172.21.15.12:0/844713834 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07881a59d0 con 0x7f07880756e0 2024-03-23T22:59:20.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.114+0000 7f077e000640 1 --2- 172.21.15.12:0/844713834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07880756e0 0x7f078819e300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f077000e9f0 tx=0x7f077000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:20.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.115+0000 7f077fe00640 1 -- 172.21.15.12:0/844713834 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077000cd80 con 0x7f07880756e0 2024-03-23T22:59:20.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.115+0000 7f0787e00640 1 -- 172.21.15.12:0/844713834 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07881a5cc0 con 0x7f07880756e0 2024-03-23T22:59:20.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.115+0000 7f077fe00640 1 -- 172.21.15.12:0/844713834 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0770010910 con 0x7f07880756e0 2024-03-23T22:59:20.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.115+0000 7f077fe00640 1 -- 172.21.15.12:0/844713834 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0770018990 con 0x7f07880756e0 2024-03-23T22:59:20.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.115+0000 7f0787e00640 1 -- 172.21.15.12:0/844713834 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07881a61c0 con 0x7f07880756e0 2024-03-23T22:59:20.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.116+0000 7f0787e00640 1 -- 172.21.15.12:0/844713834 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0788103cb0 con 0x7f07880756e0 2024-03-23T22:59:20.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:19 smithi173 ceph-mon[36116]: mds.? [v2:172.21.15.12:6840/732195840,v1:172.21.15.12:6841/732195840] up:boot 2024-03-23T22:59:20.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:19 smithi173 ceph-mon[36116]: fsmap 11 up:standby 2024-03-23T22:59:20.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:19 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-23T22:59:20.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.118+0000 7f077fe00640 1 -- 172.21.15.12:0/844713834 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0770010ac0 con 0x7f07880756e0 2024-03-23T22:59:20.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.120+0000 7f077fe00640 1 --2- 172.21.15.12:0/844713834 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f076c0a2800 0x7f076c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:20.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.120+0000 7f077fe00640 1 -- 172.21.15.12:0/844713834 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f07700cea40 con 0x7f07880756e0 2024-03-23T22:59:20.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.120+0000 7f0786a00640 1 --2- 172.21.15.12:0/844713834 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f076c0a2800 0x7f076c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:20.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.121+0000 7f0786a00640 1 --2- 172.21.15.12:0/844713834 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f076c0a2800 0x7f076c0a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f078819f290 tx=0x7f0778005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:20.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.124+0000 7f077fe00640 1 -- 172.21.15.12:0/844713834 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f077009b890 con 0x7f07880756e0 2024-03-23T22:59:20.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:19 smithi203 ceph-mon[36693]: mds.? [v2:172.21.15.12:6840/732195840,v1:172.21.15.12:6841/732195840] up:boot 2024-03-23T22:59:20.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:19 smithi203 ceph-mon[36693]: fsmap 11 up:standby 2024-03-23T22:59:20.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-23T22:59:20.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.369+0000 7f0787e00640 1 -- 172.21.15.12:0/844713834 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0788062340 con 0x7f07880756e0 2024-03-23T22:59:20.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.370+0000 7f077fe00640 1 -- 172.21.15.12:0/844713834 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9245 (secure 0 0 0) 0x7f077009f930 con 0x7f07880756e0 2024-03-23T22:59:20.371 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:59:20.371 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24521,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24554,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-23T22:59:20.371 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 12 2024-03-23T22:59:20.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.373+0000 7f0787e00640 1 -- 172.21.15.12:0/844713834 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f076c0a2800 msgr2=0x7f076c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:20.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.373+0000 7f0787e00640 1 --2- 172.21.15.12:0/844713834 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f076c0a2800 0x7f076c0a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f078819f290 tx=0x7f0778005f90 comp rx=0 tx=0).stop 2024-03-23T22:59:20.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.373+0000 7f0787e00640 1 -- 172.21.15.12:0/844713834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07880756e0 msgr2=0x7f078819e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:20.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.374+0000 7f0787e00640 1 --2- 172.21.15.12:0/844713834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07880756e0 0x7f078819e300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f077000e9f0 tx=0x7f077000eeb0 comp rx=0 tx=0).stop 2024-03-23T22:59:20.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.374+0000 7f0787e00640 1 -- 172.21.15.12:0/844713834 shutdown_connections 2024-03-23T22:59:20.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.374+0000 7f0787e00640 1 --2- 172.21.15.12:0/844713834 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f076c0a2800 0x7f076c0a4cc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:20.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.374+0000 7f0787e00640 1 --2- 172.21.15.12:0/844713834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f078810dae0 0x7f07881a5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:20.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.374+0000 7f0787e00640 1 --2- 172.21.15.12:0/844713834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07880756e0 0x7f078819e300 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:20.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.374+0000 7f0787e00640 1 --2- 172.21.15.12:0/844713834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0788074100 0x7f078819ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:20.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.374+0000 7f0787e00640 1 -- 172.21.15.12:0/844713834 >> 172.21.15.12:0/844713834 conn(0x7f07880fca80 msgr2=0x7f0788100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:20.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.374+0000 7f0787e00640 1 -- 172.21.15.12:0/844713834 shutdown_connections 2024-03-23T22:59:20.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:20.374+0000 7f0787e00640 1 -- 172.21.15.12:0/844713834 wait complete. 2024-03-23T22:59:20.540 INFO:journalctl@ceph.mds.b.smithi173.stdout:Mar 23 22:59:20 smithi173 systemd[1]: Starting Ceph mds.b for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T22:59:20.823 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:20 smithi173 ceph-mon[36116]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:20.823 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:20 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/844713834' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:20.823 INFO:journalctl@ceph.mds.b.smithi173.stdout:Mar 23 22:59:20 smithi173 podman[66525]: 2024-03-23 22:59:20.444814363 +0000 UTC m=+0.017238707 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-23T22:59:20.823 INFO:journalctl@ceph.mds.b.smithi173.stdout:Mar 23 22:59:20 smithi173 podman[66525]: 2024-03-23 22:59:20.565103425 +0000 UTC m=+0.137527777 container create 7d92fbafbf17eb5156105fe9749d3a5ec7d0ca0d869c4d4ef0fc43496d50ccf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-23T22:59:20.823 INFO:journalctl@ceph.mds.b.smithi173.stdout:Mar 23 22:59:20 smithi173 podman[66525]: 2024-03-23 22:59:20.824253651 +0000 UTC m=+0.396677996 container init 7d92fbafbf17eb5156105fe9749d3a5ec7d0ca0d869c4d4ef0fc43496d50ccf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-23T22:59:21.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:20 smithi012 ceph-mon[31695]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:21.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:20 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/844713834' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:21.118 INFO:journalctl@ceph.mds.b.smithi173.stdout:Mar 23 22:59:20 smithi173 podman[66525]: 2024-03-23 22:59:20.827069606 +0000 UTC m=+0.399493962 container start 7d92fbafbf17eb5156105fe9749d3a5ec7d0ca0d869c4d4ef0fc43496d50ccf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-23T22:59:21.118 INFO:journalctl@ceph.mds.b.smithi173.stdout:Mar 23 22:59:20 smithi173 ceph-mds[66544]: -- 172.21.15.173:0/1258706876 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555addab5a40 con 0x555adda88800 2024-03-23T22:59:21.118 INFO:journalctl@ceph.mds.b.smithi173.stdout:Mar 23 22:59:20 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-b[66540]: starting mds.b at 2024-03-23T22:59:21.118 INFO:journalctl@ceph.mds.b.smithi173.stdout:Mar 23 22:59:20 smithi173 bash[66525]: 7d92fbafbf17eb5156105fe9749d3a5ec7d0ca0d869c4d4ef0fc43496d50ccf5 2024-03-23T22:59:21.118 INFO:journalctl@ceph.mds.b.smithi173.stdout:Mar 23 22:59:20 smithi173 systemd[1]: Started Ceph mds.b for 90f679ae-e967-11ee-95cd-87774f69a715. 2024-03-23T22:59:21.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:20 smithi203 ceph-mon[36693]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:21.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:20 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/844713834' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:22.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:21 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:22.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:21 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:22.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:21 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:22.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:21 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:22.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:21 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:22.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:21 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:59:22.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:21 smithi173 ceph-mon[36116]: mds.? [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] up:boot 2024-03-23T22:59:22.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:21 smithi173 ceph-mon[36116]: fsmap 12 up:standby 2024-03-23T22:59:22.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:21 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-23T22:59:22.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:21 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-23T22:59:22.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:21 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:22.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:21 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:22.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:21 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:22.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:21 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:22.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:21 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:22.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:21 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:59:22.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:21 smithi012 ceph-mon[31695]: mds.? [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] up:boot 2024-03-23T22:59:22.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:21 smithi012 ceph-mon[31695]: fsmap 12 up:standby 2024-03-23T22:59:22.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:21 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-23T22:59:22.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:21 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-23T22:59:22.341 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:21 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:22.341 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:21 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:22.341 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:21 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:22.341 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:21 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:22.341 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:21 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:22.341 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:21 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:59:22.342 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:21 smithi203 ceph-mon[36693]: mds.? [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] up:boot 2024-03-23T22:59:22.342 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:21 smithi203 ceph-mon[36693]: fsmap 12 up:standby 2024-03-23T22:59:22.342 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:21 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-23T22:59:22.342 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:21 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-23T22:59:22.394 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T22:59:22.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.593+0000 7f24f3e00640 1 -- 172.21.15.12:0/3205222003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24f4107b00 msgr2=0x7f24f4107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:22.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.593+0000 7f24f3e00640 1 --2- 172.21.15.12:0/3205222003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24f4107b00 0x7f24f4107f20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f24e4009900 tx=0x7f24e402bc30 comp rx=0 tx=0).stop 2024-03-23T22:59:22.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.593+0000 7f24f3e00640 1 -- 172.21.15.12:0/3205222003 shutdown_connections 2024-03-23T22:59:22.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.593+0000 7f24f3e00640 1 --2- 172.21.15.12:0/3205222003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24f41096b0 0x7f24f410ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:22.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.593+0000 7f24f3e00640 1 --2- 172.21.15.12:0/3205222003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24f4108cf0 0x7f24f4109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:22.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.593+0000 7f24f3e00640 1 --2- 172.21.15.12:0/3205222003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24f4107b00 0x7f24f4107f20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:22.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.593+0000 7f24f3e00640 1 -- 172.21.15.12:0/3205222003 >> 172.21.15.12:0/3205222003 conn(0x7f24f4076ab0 msgr2=0x7f24f4078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:22.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.593+0000 7f24f3e00640 1 -- 172.21.15.12:0/3205222003 shutdown_connections 2024-03-23T22:59:22.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.593+0000 7f24f3e00640 1 -- 172.21.15.12:0/3205222003 wait complete. 2024-03-23T22:59:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.594+0000 7f24f3e00640 1 Processor -- start 2024-03-23T22:59:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.595+0000 7f24f3e00640 1 -- start start 2024-03-23T22:59:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.595+0000 7f24f3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24f4107b00 0x7f24f419deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.595+0000 7f24f3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24f4108cf0 0x7f24f419e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.595+0000 7f24f3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24f41096b0 0x7f24f41a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.595+0000 7f24f3e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f41137c0 con 0x7f24f4107b00 2024-03-23T22:59:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.595+0000 7f24f3e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f4113640 con 0x7f24f41096b0 2024-03-23T22:59:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.595+0000 7f24f3e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f4113940 con 0x7f24f4108cf0 2024-03-23T22:59:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.595+0000 7f24f2a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24f4107b00 0x7f24f419deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.595+0000 7f24f2a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24f4107b00 0x7f24f419deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53394/0 (socket says 172.21.15.12:53394) 2024-03-23T22:59:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.595+0000 7f24f2a00640 1 -- 172.21.15.12:0/4234335701 learned_addr learned my addr 172.21.15.12:0/4234335701 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.595+0000 7f24ebe00640 1 --2- 172.21.15.12:0/4234335701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24f4108cf0 0x7f24f419e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.595+0000 7f24f3400640 1 --2- 172.21.15.12:0/4234335701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24f41096b0 0x7f24f41a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.595+0000 7f24f2a00640 1 -- 172.21.15.12:0/4234335701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24f4108cf0 msgr2=0x7f24f419e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.596+0000 7f24f2a00640 1 --2- 172.21.15.12:0/4234335701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24f4108cf0 0x7f24f419e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:22.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.596+0000 7f24f2a00640 1 -- 172.21.15.12:0/4234335701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24f41096b0 msgr2=0x7f24f41a5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:22.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.596+0000 7f24f2a00640 1 --2- 172.21.15.12:0/4234335701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24f41096b0 0x7f24f41a5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:22.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.596+0000 7f24f2a00640 1 -- 172.21.15.12:0/4234335701 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f41a5a80 con 0x7f24f4107b00 2024-03-23T22:59:22.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.596+0000 7f24ebe00640 1 --2- 172.21.15.12:0/4234335701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24f4108cf0 0x7f24f419e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T22:59:22.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.596+0000 7f24f2a00640 1 --2- 172.21.15.12:0/4234335701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24f4107b00 0x7f24f419deb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f24e4000c00 tx=0x7f24e402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:22.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.596+0000 7f24eb400640 1 -- 172.21.15.12:0/4234335701 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e4043020 con 0x7f24f4107b00 2024-03-23T22:59:22.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.596+0000 7f24eb400640 1 -- 172.21.15.12:0/4234335701 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f24e40054e0 con 0x7f24f4107b00 2024-03-23T22:59:22.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.596+0000 7f24eb400640 1 -- 172.21.15.12:0/4234335701 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e4003bd0 con 0x7f24f4107b00 2024-03-23T22:59:22.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.596+0000 7f24f3e00640 1 -- 172.21.15.12:0/4234335701 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24f41a5d10 con 0x7f24f4107b00 2024-03-23T22:59:22.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.596+0000 7f24f3e00640 1 -- 172.21.15.12:0/4234335701 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24f41a61b0 con 0x7f24f4107b00 2024-03-23T22:59:22.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.598+0000 7f24f3e00640 1 -- 172.21.15.12:0/4234335701 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24f410ee20 con 0x7f24f4107b00 2024-03-23T22:59:22.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.599+0000 7f24eb400640 1 -- 172.21.15.12:0/4234335701 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f24e4046020 con 0x7f24f4107b00 2024-03-23T22:59:22.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.602+0000 7f24eb400640 1 --2- 172.21.15.12:0/4234335701 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f24d80a2800 0x7f24d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:22.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.602+0000 7f24eb400640 1 -- 172.21.15.12:0/4234335701 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f24e40f06a0 con 0x7f24f4107b00 2024-03-23T22:59:22.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.602+0000 7f24ebe00640 1 --2- 172.21.15.12:0/4234335701 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f24d80a2800 0x7f24d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:22.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.603+0000 7f24ebe00640 1 --2- 172.21.15.12:0/4234335701 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f24d80a2800 0x7f24d80a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f24f419f370 tx=0x7f24dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:22.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.611+0000 7f24eb400640 1 -- 172.21.15.12:0/4234335701 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f24e404d020 con 0x7f24f4107b00 2024-03-23T22:59:22.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.860+0000 7f24f3e00640 1 -- 172.21.15.12:0/4234335701 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24f402d070 con 0x7f24f4107b00 2024-03-23T22:59:22.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.861+0000 7f24eb400640 1 -- 172.21.15.12:0/4234335701 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10040 (secure 0 0 0) 0x7f24e40bdb40 con 0x7f24f4107b00 2024-03-23T22:59:22.862 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:59:22.862 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24521,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24554,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24599,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[]} 2024-03-23T22:59:22.862 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 13 2024-03-23T22:59:22.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.864+0000 7f24f3e00640 1 -- 172.21.15.12:0/4234335701 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f24d80a2800 msgr2=0x7f24d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:22.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.864+0000 7f24f3e00640 1 --2- 172.21.15.12:0/4234335701 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f24d80a2800 0x7f24d80a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f24f419f370 tx=0x7f24dc005f90 comp rx=0 tx=0).stop 2024-03-23T22:59:22.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.864+0000 7f24f3e00640 1 -- 172.21.15.12:0/4234335701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24f4107b00 msgr2=0x7f24f419deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:22.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.864+0000 7f24f3e00640 1 --2- 172.21.15.12:0/4234335701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24f4107b00 0x7f24f419deb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f24e4000c00 tx=0x7f24e402c650 comp rx=0 tx=0).stop 2024-03-23T22:59:22.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.864+0000 7f24f3e00640 1 -- 172.21.15.12:0/4234335701 shutdown_connections 2024-03-23T22:59:22.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.864+0000 7f24f3e00640 1 --2- 172.21.15.12:0/4234335701 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f24d80a2800 0x7f24d80a4cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:22.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.864+0000 7f24f3e00640 1 --2- 172.21.15.12:0/4234335701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24f41096b0 0x7f24f41a5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:22.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.864+0000 7f24f3e00640 1 --2- 172.21.15.12:0/4234335701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24f4108cf0 0x7f24f419e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:22.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.864+0000 7f24f3e00640 1 --2- 172.21.15.12:0/4234335701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24f4107b00 0x7f24f419deb0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:22.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.864+0000 7f24f3e00640 1 -- 172.21.15.12:0/4234335701 >> 172.21.15.12:0/4234335701 conn(0x7f24f4076ab0 msgr2=0x7f24f410bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:22.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.864+0000 7f24f3e00640 1 -- 172.21.15.12:0/4234335701 shutdown_connections 2024-03-23T22:59:22.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:22.864+0000 7f24f3e00640 1 -- 172.21.15.12:0/4234335701 wait complete. 2024-03-23T22:59:22.885 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T22:59:23.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.094+0000 7f22baa00640 1 -- 172.21.15.12:0/2897297643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22b4108cf0 msgr2=0x7f22b4109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:23.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.094+0000 7f22baa00640 1 --2- 172.21.15.12:0/2897297643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22b4108cf0 0x7f22b4109170 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f22a0009900 tx=0x7f22a002bc30 comp rx=0 tx=0).stop 2024-03-23T22:59:23.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.094+0000 7f22baa00640 1 -- 172.21.15.12:0/2897297643 shutdown_connections 2024-03-23T22:59:23.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.094+0000 7f22baa00640 1 --2- 172.21.15.12:0/2897297643 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22b41096b0 0x7f22b410ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:23.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.094+0000 7f22baa00640 1 --2- 172.21.15.12:0/2897297643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22b4108cf0 0x7f22b4109170 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:23.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.094+0000 7f22baa00640 1 --2- 172.21.15.12:0/2897297643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22b4107b00 0x7f22b4107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:23.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.094+0000 7f22baa00640 1 -- 172.21.15.12:0/2897297643 >> 172.21.15.12:0/2897297643 conn(0x7f22b4076ab0 msgr2=0x7f22b4078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:23.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.094+0000 7f22baa00640 1 -- 172.21.15.12:0/2897297643 shutdown_connections 2024-03-23T22:59:23.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.094+0000 7f22baa00640 1 -- 172.21.15.12:0/2897297643 wait complete. 2024-03-23T22:59:23.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.095+0000 7f22baa00640 1 Processor -- start 2024-03-23T22:59:23.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.095+0000 7f22baa00640 1 -- start start 2024-03-23T22:59:23.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.096+0000 7f22baa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22b4107b00 0x7f22b419de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:23.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.096+0000 7f22baa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22b4108cf0 0x7f22b419e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.096+0000 7f22baa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22b41096b0 0x7f22b41a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.096+0000 7f22baa00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b41137d0 con 0x7f22b4107b00 2024-03-23T22:59:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.096+0000 7f22baa00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b4113650 con 0x7f22b41096b0 2024-03-23T22:59:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.096+0000 7f22baa00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b4113950 con 0x7f22b4108cf0 2024-03-23T22:59:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.096+0000 7f22b9600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22b4107b00 0x7f22b419de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.096+0000 7f22b9600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22b4107b00 0x7f22b419de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53420/0 (socket says 172.21.15.12:53420) 2024-03-23T22:59:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.096+0000 7f22b9600640 1 -- 172.21.15.12:0/3221855605 learned_addr learned my addr 172.21.15.12:0/3221855605 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.096+0000 7f22ba000640 1 --2- 172.21.15.12:0/3221855605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22b41096b0 0x7f22b41a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.096+0000 7f22b8c00640 1 --2- 172.21.15.12:0/3221855605 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22b4108cf0 0x7f22b419e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.096+0000 7f22b9600640 1 -- 172.21.15.12:0/3221855605 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22b4108cf0 msgr2=0x7f22b419e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.096+0000 7f22b9600640 1 --2- 172.21.15.12:0/3221855605 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22b4108cf0 0x7f22b419e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.096+0000 7f22b9600640 1 -- 172.21.15.12:0/3221855605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22b41096b0 msgr2=0x7f22b41a53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.096+0000 7f22b9600640 1 --2- 172.21.15.12:0/3221855605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22b41096b0 0x7f22b41a53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.096+0000 7f22b9600640 1 -- 172.21.15.12:0/3221855605 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b41a5a60 con 0x7f22b4107b00 2024-03-23T22:59:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.097+0000 7f22b9600640 1 --2- 172.21.15.12:0/3221855605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22b4107b00 0x7f22b419de40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f22a800e9c0 tx=0x7f22a800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:23.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.097+0000 7f22b2000640 1 -- 172.21.15.12:0/3221855605 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a800ccc0 con 0x7f22b4107b00 2024-03-23T22:59:23.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.097+0000 7f22baa00640 1 -- 172.21.15.12:0/3221855605 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22b41a5d50 con 0x7f22b4107b00 2024-03-23T22:59:23.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.097+0000 7f22b2000640 1 -- 172.21.15.12:0/3221855605 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22a80035f0 con 0x7f22b4107b00 2024-03-23T22:59:23.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.097+0000 7f22b2000640 1 -- 172.21.15.12:0/3221855605 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a800ce60 con 0x7f22b4107b00 2024-03-23T22:59:23.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.097+0000 7f22baa00640 1 -- 172.21.15.12:0/3221855605 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22b41a61c0 con 0x7f22b4107b00 2024-03-23T22:59:23.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.098+0000 7f22baa00640 1 -- 172.21.15.12:0/3221855605 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22b410ee20 con 0x7f22b4107b00 2024-03-23T22:59:23.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.101+0000 7f22b2000640 1 -- 172.21.15.12:0/3221855605 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f22a80037a0 con 0x7f22b4107b00 2024-03-23T22:59:23.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.103+0000 7f22b2000640 1 --2- 172.21.15.12:0/3221855605 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f229c0a2800 0x7f229c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:23.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.103+0000 7f22b2000640 1 -- 172.21.15.12:0/3221855605 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f22a80cea80 con 0x7f22b4107b00 2024-03-23T22:59:23.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.103+0000 7f22b8c00640 1 --2- 172.21.15.12:0/3221855605 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f229c0a2800 0x7f229c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:23.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.104+0000 7f22b8c00640 1 --2- 172.21.15.12:0/3221855605 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f229c0a2800 0x7f229c0a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f22a002c240 tx=0x7f22a002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:23.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.107+0000 7f22b2000640 1 -- 172.21.15.12:0/3221855605 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f22a80d0e20 con 0x7f22b4107b00 2024-03-23T22:59:23.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.332+0000 7f22baa00640 1 -- 172.21.15.12:0/3221855605 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22b402d2b0 con 0x7f22b4107b00 2024-03-23T22:59:23.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.333+0000 7f22b2000640 1 -- 172.21.15.12:0/3221855605 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18481 (secure 0 0 0) 0x7f22a809bf20 con 0x7f22b4107b00 2024-03-23T22:59:23.333 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:59:23.334 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":82,"fsid":"90f679ae-e967-11ee-95cd-87774f69a715","created":"2024-03-23T22:50:14.906952+0000","modified":"2024-03-23T22:58:30.496820+0000","last_up_change":"2024-03-23T22:58:28.907824+0000","last_in_change":"2024-03-23T22:58:08.342487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T22:54:11.831010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b70875d7-a23d-4251-9e8b-1a16d4edc2a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6803","nonce":1190079611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6805","nonce":1190079611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6809","nonce":1190079611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6807","nonce":1190079611}]},"public_addr":"172.21.15.12:6803/1190079611","cluster_addr":"172.21.15.12:6805/1190079611","heartbeat_back_addr":"172.21.15.12:6809/1190079611","heartbeat_front_addr":"172.21.15.12:6807/1190079611","state":["exists","up"]},{"osd":1,"uuid":"416e87af-db5d-41ce-84d9-731e3e07e3c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6801","nonce":1983639525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6803","nonce":1983639525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6807","nonce":1983639525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6805","nonce":1983639525}]},"public_addr":"172.21.15.173:6801/1983639525","cluster_addr":"172.21.15.173:6803/1983639525","heartbeat_back_addr":"172.21.15.173:6807/1983639525","heartbeat_front_addr":"172.21.15.173:6805/1983639525","state":["exists","up"]},{"osd":2,"uuid":"7a559c70-4c54-431b-8baf-4136b6e76493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6801","nonce":2253997387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6803","nonce":2253997387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6807","nonce":2253997387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6805","nonce":2253997387}]},"public_addr":"172.21.15.203:6801/2253997387","cluster_addr":"172.21.15.203:6803/2253997387","heartbeat_back_addr":"172.21.15.203:6807/2253997387","heartbeat_front_addr":"172.21.15.203:6805/2253997387","state":["exists","up"]},{"osd":3,"uuid":"c41cd287-a674-486c-8025-328cd36e5ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6811","nonce":1761446905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6813","nonce":1761446905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6817","nonce":1761446905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6815","nonce":1761446905}]},"public_addr":"172.21.15.12:6811/1761446905","cluster_addr":"172.21.15.12:6813/1761446905","heartbeat_back_addr":"172.21.15.12:6817/1761446905","heartbeat_front_addr":"172.21.15.12:6815/1761446905","state":["exists","up"]},{"osd":4,"uuid":"3c48f886-0b42-4a09-b31e-abccc200bb1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6809","nonce":242958272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6811","nonce":242958272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6815","nonce":242958272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6813","nonce":242958272}]},"public_addr":"172.21.15.173:6809/242958272","cluster_addr":"172.21.15.173:6811/242958272","heartbeat_back_addr":"172.21.15.173:6815/242958272","heartbeat_front_addr":"172.21.15.173:6813/242958272","state":["exists","up"]},{"osd":5,"uuid":"4a34f06f-8805-40c0-b1ab-23407b0276d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6809","nonce":4270966092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6811","nonce":4270966092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6815","nonce":4270966092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6813","nonce":4270966092}]},"public_addr":"172.21.15.203:6809/4270966092","cluster_addr":"172.21.15.203:6811/4270966092","heartbeat_back_addr":"172.21.15.203:6815/4270966092","heartbeat_front_addr":"172.21.15.203:6813/4270966092","state":["exists","up"]},{"osd":6,"uuid":"26ddc663-5623-48c9-80d1-f8b968765d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6819","nonce":3606839295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6821","nonce":3606839295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6825","nonce":3606839295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6823","nonce":3606839295}]},"public_addr":"172.21.15.12:6819/3606839295","cluster_addr":"172.21.15.12:6821/3606839295","heartbeat_back_addr":"172.21.15.12:6825/3606839295","heartbeat_front_addr":"172.21.15.12:6823/3606839295","state":["exists","up"]},{"osd":7,"uuid":"17dc9417-71cd-4576-aa08-933212ec55e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6817","nonce":1241504346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6819","nonce":1241504346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6823","nonce":1241504346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6821","nonce":1241504346}]},"public_addr":"172.21.15.173:6817/1241504346","cluster_addr":"172.21.15.173:6819/1241504346","heartbeat_back_addr":"172.21.15.173:6823/1241504346","heartbeat_front_addr":"172.21.15.173:6821/1241504346","state":["exists","up"]},{"osd":8,"uuid":"90a2b63a-471d-43b8-ac18-bb1f16778d18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6817","nonce":1516996192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6819","nonce":1516996192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6823","nonce":1516996192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6821","nonce":1516996192}]},"public_addr":"172.21.15.203:6817/1516996192","cluster_addr":"172.21.15.203:6819/1516996192","heartbeat_back_addr":"172.21.15.203:6823/1516996192","heartbeat_front_addr":"172.21.15.203:6821/1516996192","state":["exists","up"]},{"osd":9,"uuid":"6e1ca504-40b4-49d0-b245-7946181abfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6827","nonce":1175260774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6829","nonce":1175260774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6833","nonce":1175260774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6831","nonce":1175260774}]},"public_addr":"172.21.15.12:6827/1175260774","cluster_addr":"172.21.15.12:6829/1175260774","heartbeat_back_addr":"172.21.15.12:6833/1175260774","heartbeat_front_addr":"172.21.15.12:6831/1175260774","state":["exists","up"]},{"osd":10,"uuid":"09a595ed-5614-4386-8e61-0170af66e01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6825","nonce":3303506808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6827","nonce":3303506808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6831","nonce":3303506808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6829","nonce":3303506808}]},"public_addr":"172.21.15.173:6825/3303506808","cluster_addr":"172.21.15.173:6827/3303506808","heartbeat_back_addr":"172.21.15.173:6831/3303506808","heartbeat_front_addr":"172.21.15.173:6829/3303506808","state":["exists","up"]},{"osd":11,"uuid":"9ef1d595-a977-4ba7-9c6b-d2b3295f20b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6825","nonce":2323604506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6827","nonce":2323604506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6831","nonce":2323604506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6829","nonce":2323604506}]},"public_addr":"172.21.15.203:6825/2323604506","cluster_addr":"172.21.15.203:6827/2323604506","heartbeat_back_addr":"172.21.15.203:6831/2323604506","heartbeat_front_addr":"172.21.15.203:6829/2323604506","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:18.469038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:42.752628+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:07.069493+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:41.962478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:08.363104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:33.341336+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:07.286541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:34.243302+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:59.749699+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:57:33.822913+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:00.737269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:26.599854+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/4182783120":"2024-03-24T22:51:23.113201+0000","172.21.15.12:6800/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1900938637":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/3414861652":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6801/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/2580541477":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6800/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/501058839":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1192554716":"2024-03-24T22:50:48.401223+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T22:59:23.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.336+0000 7f22baa00640 1 -- 172.21.15.12:0/3221855605 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f229c0a2800 msgr2=0x7f229c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:23.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.336+0000 7f22baa00640 1 --2- 172.21.15.12:0/3221855605 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f229c0a2800 0x7f229c0a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f22a002c240 tx=0x7f22a002df90 comp rx=0 tx=0).stop 2024-03-23T22:59:23.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.336+0000 7f22baa00640 1 -- 172.21.15.12:0/3221855605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22b4107b00 msgr2=0x7f22b419de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:23.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.336+0000 7f22baa00640 1 --2- 172.21.15.12:0/3221855605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22b4107b00 0x7f22b419de40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f22a800e9c0 tx=0x7f22a800edc0 comp rx=0 tx=0).stop 2024-03-23T22:59:23.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.336+0000 7f22baa00640 1 -- 172.21.15.12:0/3221855605 shutdown_connections 2024-03-23T22:59:23.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.336+0000 7f22baa00640 1 --2- 172.21.15.12:0/3221855605 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f229c0a2800 0x7f229c0a4cc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:23.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.336+0000 7f22baa00640 1 --2- 172.21.15.12:0/3221855605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22b41096b0 0x7f22b41a53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:23.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.336+0000 7f22baa00640 1 --2- 172.21.15.12:0/3221855605 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22b4108cf0 0x7f22b419e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:23.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.336+0000 7f22baa00640 1 --2- 172.21.15.12:0/3221855605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22b4107b00 0x7f22b419de40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:23.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.336+0000 7f22baa00640 1 -- 172.21.15.12:0/3221855605 >> 172.21.15.12:0/3221855605 conn(0x7f22b4076ab0 msgr2=0x7f22b410bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:23.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.337+0000 7f22baa00640 1 -- 172.21.15.12:0/3221855605 shutdown_connections 2024-03-23T22:59:23.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.337+0000 7f22baa00640 1 -- 172.21.15.12:0/3221855605 wait complete. 2024-03-23T22:59:23.354 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:23 smithi012 ceph-mon[31695]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:23.355 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:23 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/4234335701' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:23.355 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:23 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:23.355 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:23 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:23.357 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T22:54:11.831010+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T22:59:23.357 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T22:59:23.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:23 smithi173 ceph-mon[36116]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:23.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:23 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/4234335701' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:23.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:23 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:23.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:23 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:23.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:23 smithi203 ceph-mon[36693]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:23.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:23 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/4234335701' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:23.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:23 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:23.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:23 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:23.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.569+0000 7f9753200640 1 -- 172.21.15.12:0/3859346585 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f974c1008f0 msgr2=0x7f974c100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:23.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.569+0000 7f9753200640 1 --2- 172.21.15.12:0/3859346585 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f974c1008f0 0x7f974c100d50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9738009900 tx=0x7f973802bcc0 comp rx=0 tx=0).stop 2024-03-23T22:59:23.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.570+0000 7f9753200640 1 -- 172.21.15.12:0/3859346585 shutdown_connections 2024-03-23T22:59:23.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.570+0000 7f9753200640 1 --2- 172.21.15.12:0/3859346585 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f974c101290 0x7f974c101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:23.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.570+0000 7f9753200640 1 --2- 172.21.15.12:0/3859346585 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f974c1008f0 0x7f974c100d50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:23.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.570+0000 7f9753200640 1 --2- 172.21.15.12:0/3859346585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f974c0fff90 0x7f974c1003b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:23.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.570+0000 7f9753200640 1 -- 172.21.15.12:0/3859346585 >> 172.21.15.12:0/3859346585 conn(0x7f974c0fbb30 msgr2=0x7f974c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:23.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.570+0000 7f9753200640 1 -- 172.21.15.12:0/3859346585 shutdown_connections 2024-03-23T22:59:23.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.570+0000 7f9753200640 1 -- 172.21.15.12:0/3859346585 wait complete. 2024-03-23T22:59:23.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.571+0000 7f9753200640 1 Processor -- start 2024-03-23T22:59:23.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.571+0000 7f9753200640 1 -- start start 2024-03-23T22:59:23.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.571+0000 7f9753200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f974c0fff90 0x7f974c199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:23.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.571+0000 7f9753200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f974c1008f0 0x7f974c19a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:23.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.571+0000 7f9753200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f974c101290 0x7f974c1a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:23.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.571+0000 7f9753200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c10f570 con 0x7f974c101290 2024-03-23T22:59:23.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.571+0000 7f9753200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c10f3f0 con 0x7f974c1008f0 2024-03-23T22:59:23.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.571+0000 7f9753200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c10f6f0 con 0x7f974c0fff90 2024-03-23T22:59:23.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.571+0000 7f974b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f974c1008f0 0x7f974c19a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:23.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.571+0000 7f9751200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f974c101290 0x7f974c1a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:23.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.571+0000 7f974b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f974c1008f0 0x7f974c19a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:56676/0 (socket says 172.21.15.12:56676) 2024-03-23T22:59:23.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.571+0000 7f974b400640 1 -- 172.21.15.12:0/1142437538 learned_addr learned my addr 172.21.15.12:0/1142437538 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:23.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.571+0000 7f974be00640 1 --2- 172.21.15.12:0/1142437538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f974c0fff90 0x7f974c199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:23.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.572+0000 7f974b400640 1 -- 172.21.15.12:0/1142437538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f974c0fff90 msgr2=0x7f974c199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:23.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.572+0000 7f974b400640 1 --2- 172.21.15.12:0/1142437538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f974c0fff90 0x7f974c199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:23.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.572+0000 7f974b400640 1 -- 172.21.15.12:0/1142437538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f974c101290 msgr2=0x7f974c1a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:23.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.572+0000 7f974b400640 1 --2- 172.21.15.12:0/1142437538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f974c101290 0x7f974c1a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:23.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.572+0000 7f974b400640 1 -- 172.21.15.12:0/1142437538 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974c1a17a0 con 0x7f974c1008f0 2024-03-23T22:59:23.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.572+0000 7f974b400640 1 --2- 172.21.15.12:0/1142437538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f974c1008f0 0x7f974c19a0c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9738000c00 tx=0x7f9738005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:23.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.573+0000 7f9748c00640 1 -- 172.21.15.12:0/1142437538 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9738043020 con 0x7f974c1008f0 2024-03-23T22:59:23.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.573+0000 7f9748c00640 1 -- 172.21.15.12:0/1142437538 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f97380054e0 con 0x7f974c1008f0 2024-03-23T22:59:23.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.573+0000 7f9753200640 1 -- 172.21.15.12:0/1142437538 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f974c1a1a30 con 0x7f974c1008f0 2024-03-23T22:59:23.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.573+0000 7f9748c00640 1 -- 172.21.15.12:0/1142437538 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9738003d20 con 0x7f974c1008f0 2024-03-23T22:59:23.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.573+0000 7f9753200640 1 -- 172.21.15.12:0/1142437538 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f974c1a1fd0 con 0x7f974c1008f0 2024-03-23T22:59:23.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.574+0000 7f9753200640 1 -- 172.21.15.12:0/1142437538 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f974c10ae20 con 0x7f974c1008f0 2024-03-23T22:59:23.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.579+0000 7f9748c00640 1 -- 172.21.15.12:0/1142437538 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9738046020 con 0x7f974c1008f0 2024-03-23T22:59:23.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.580+0000 7f9748c00640 1 --2- 172.21.15.12:0/1142437538 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f97300a2880 0x7f97300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:23.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.580+0000 7f9748c00640 1 -- 172.21.15.12:0/1142437538 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f97380f07b0 con 0x7f974c1008f0 2024-03-23T22:59:23.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.580+0000 7f974be00640 1 --2- 172.21.15.12:0/1142437538 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f97300a2880 0x7f97300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:23.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.581+0000 7f974be00640 1 --2- 172.21.15.12:0/1142437538 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f97300a2880 0x7f97300a4d40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f973c005fd0 tx=0x7f973c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:23.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.584+0000 7f9748c00640 1 -- 172.21.15.12:0/1142437538 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f973804d020 con 0x7f974c1008f0 2024-03-23T22:59:23.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.829+0000 7f9753200640 1 -- 172.21.15.12:0/1142437538 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f974c0622a0 con 0x7f974c1008f0 2024-03-23T22:59:23.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.830+0000 7f9748c00640 1 -- 172.21.15.12:0/1142437538 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f97380bdc50 con 0x7f974c1008f0 2024-03-23T22:59:23.830 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-23T22:59:23.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.833+0000 7f9753200640 1 -- 172.21.15.12:0/1142437538 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f97300a2880 msgr2=0x7f97300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:23.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.833+0000 7f9753200640 1 --2- 172.21.15.12:0/1142437538 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f97300a2880 0x7f97300a4d40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f973c005fd0 tx=0x7f973c00b390 comp rx=0 tx=0).stop 2024-03-23T22:59:23.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.833+0000 7f9753200640 1 -- 172.21.15.12:0/1142437538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f974c1008f0 msgr2=0x7f974c19a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:23.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.833+0000 7f9753200640 1 --2- 172.21.15.12:0/1142437538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f974c1008f0 0x7f974c19a0c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9738000c00 tx=0x7f9738005dd0 comp rx=0 tx=0).stop 2024-03-23T22:59:23.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.833+0000 7f9753200640 1 -- 172.21.15.12:0/1142437538 shutdown_connections 2024-03-23T22:59:23.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.833+0000 7f9753200640 1 --2- 172.21.15.12:0/1142437538 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f97300a2880 0x7f97300a4d40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:23.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.833+0000 7f9753200640 1 --2- 172.21.15.12:0/1142437538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f974c101290 0x7f974c1a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:23.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.833+0000 7f9753200640 1 --2- 172.21.15.12:0/1142437538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f974c1008f0 0x7f974c19a0c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:23.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.833+0000 7f9753200640 1 --2- 172.21.15.12:0/1142437538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f974c0fff90 0x7f974c199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:23.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.833+0000 7f9753200640 1 -- 172.21.15.12:0/1142437538 >> 172.21.15.12:0/1142437538 conn(0x7f974c0fbb30 msgr2=0x7f974c0fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:23.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.833+0000 7f9753200640 1 -- 172.21.15.12:0/1142437538 shutdown_connections 2024-03-23T22:59:23.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:23.833+0000 7f9753200640 1 -- 172.21.15.12:0/1142437538 wait complete. 2024-03-23T22:59:23.853 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-23T22:59:24.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.073+0000 7efd95800640 1 -- 172.21.15.12:0/579758149 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd901038e0 msgr2=0x7efd90103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:24.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.073+0000 7efd95800640 1 --2- 172.21.15.12:0/579758149 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd901038e0 0x7efd90103d00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efd80009900 tx=0x7efd8002bc30 comp rx=0 tx=0).stop 2024-03-23T22:59:24.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.073+0000 7efd95800640 1 -- 172.21.15.12:0/579758149 shutdown_connections 2024-03-23T22:59:24.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.073+0000 7efd95800640 1 --2- 172.21.15.12:0/579758149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd90105490 0x7efd9010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:24.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.073+0000 7efd95800640 1 --2- 172.21.15.12:0/579758149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd90104ad0 0x7efd90104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:24.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.073+0000 7efd95800640 1 --2- 172.21.15.12:0/579758149 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd901038e0 0x7efd90103d00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:24.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.073+0000 7efd95800640 1 -- 172.21.15.12:0/579758149 >> 172.21.15.12:0/579758149 conn(0x7efd900fee90 msgr2=0x7efd901012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:24.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.073+0000 7efd95800640 1 -- 172.21.15.12:0/579758149 shutdown_connections 2024-03-23T22:59:24.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.073+0000 7efd95800640 1 -- 172.21.15.12:0/579758149 wait complete. 2024-03-23T22:59:24.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.074+0000 7efd95800640 1 Processor -- start 2024-03-23T22:59:24.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.074+0000 7efd95800640 1 -- start start 2024-03-23T22:59:24.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.074+0000 7efd95800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd901038e0 0x7efd90199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:24.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.074+0000 7efd95800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd90104ad0 0x7efd9019a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:24.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.074+0000 7efd95800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd90105490 0x7efd901a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:24.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.074+0000 7efd95800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9010f5b0 con 0x7efd90105490 2024-03-23T22:59:24.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.074+0000 7efd95800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9010f430 con 0x7efd90104ad0 2024-03-23T22:59:24.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.074+0000 7efd95800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9010f730 con 0x7efd901038e0 2024-03-23T22:59:24.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.075+0000 7efd8f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd90105490 0x7efd901a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:24.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.075+0000 7efd8f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd90105490 0x7efd901a1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37046/0 (socket says 172.21.15.12:37046) 2024-03-23T22:59:24.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.075+0000 7efd8f400640 1 -- 172.21.15.12:0/461141532 learned_addr learned my addr 172.21.15.12:0/461141532 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:24.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.075+0000 7efd8ea00640 1 --2- 172.21.15.12:0/461141532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd901038e0 0x7efd90199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:24.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.075+0000 7efd8e000640 1 --2- 172.21.15.12:0/461141532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd90104ad0 0x7efd9019a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:24.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.075+0000 7efd8f400640 1 -- 172.21.15.12:0/461141532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd901038e0 msgr2=0x7efd90199bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:24.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.075+0000 7efd8f400640 1 --2- 172.21.15.12:0/461141532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd901038e0 0x7efd90199bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:24.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.075+0000 7efd8f400640 1 -- 172.21.15.12:0/461141532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd90104ad0 msgr2=0x7efd9019a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:24.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.075+0000 7efd8f400640 1 --2- 172.21.15.12:0/461141532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd90104ad0 0x7efd9019a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:24.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.075+0000 7efd8f400640 1 -- 172.21.15.12:0/461141532 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd901a17f0 con 0x7efd90105490 2024-03-23T22:59:24.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.076+0000 7efd8f400640 1 --2- 172.21.15.12:0/461141532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd90105490 0x7efd901a1180 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efd8800eb10 tx=0x7efd8800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:24.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.076+0000 7efd87400640 1 -- 172.21.15.12:0/461141532 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd88004b20 con 0x7efd90105490 2024-03-23T22:59:24.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.076+0000 7efd95800640 1 -- 172.21.15.12:0/461141532 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd901a1ae0 con 0x7efd90105490 2024-03-23T22:59:24.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.076+0000 7efd87400640 1 -- 172.21.15.12:0/461141532 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd88004cc0 con 0x7efd90105490 2024-03-23T22:59:24.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.076+0000 7efd87400640 1 -- 172.21.15.12:0/461141532 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd88010760 con 0x7efd90105490 2024-03-23T22:59:24.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.076+0000 7efd95800640 1 -- 172.21.15.12:0/461141532 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd901a2080 con 0x7efd90105490 2024-03-23T22:59:24.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.077+0000 7efd95800640 1 -- 172.21.15.12:0/461141532 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd9010aba0 con 0x7efd90105490 2024-03-23T22:59:24.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.079+0000 7efd87400640 1 -- 172.21.15.12:0/461141532 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7efd88005020 con 0x7efd90105490 2024-03-23T22:59:24.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.081+0000 7efd87400640 1 --2- 172.21.15.12:0/461141532 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7efd740a2800 0x7efd740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:24.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.081+0000 7efd87400640 1 -- 172.21.15.12:0/461141532 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7efd880ced20 con 0x7efd90105490 2024-03-23T22:59:24.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.081+0000 7efd8ea00640 1 --2- 172.21.15.12:0/461141532 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7efd740a2800 0x7efd740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:24.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.082+0000 7efd8ea00640 1 --2- 172.21.15.12:0/461141532 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7efd740a2800 0x7efd740a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efd8002c170 tx=0x7efd8002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:24.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.085+0000 7efd87400640 1 -- 172.21.15.12:0/461141532 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efd8809c1c0 con 0x7efd90105490 2024-03-23T22:59:24.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.301+0000 7efd95800640 1 -- 172.21.15.12:0/461141532 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7efd900622a0 con 0x7efd90105490 2024-03-23T22:59:24.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.302+0000 7efd87400640 1 -- 172.21.15.12:0/461141532 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) v1 ==== 84+0+65 (secure 0 0 0) 0x7efd8809bac0 con 0x7efd90105490 2024-03-23T22:59:24.302 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:59:24.302 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-23T22:59:24.302 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T22:59:24.302 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-03-23T22:59:24.302 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-03-23T22:59:24.302 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T22:59:24.302 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-23T22:59:24.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:24.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:24.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:24.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:24 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3221855605' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T22:59:24.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:24 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1142437538' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T22:59:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.305+0000 7efd95800640 1 -- 172.21.15.12:0/461141532 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7efd740a2800 msgr2=0x7efd740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.305+0000 7efd95800640 1 --2- 172.21.15.12:0/461141532 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7efd740a2800 0x7efd740a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efd8002c170 tx=0x7efd8002df90 comp rx=0 tx=0).stop 2024-03-23T22:59:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.305+0000 7efd95800640 1 -- 172.21.15.12:0/461141532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd90105490 msgr2=0x7efd901a1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.305+0000 7efd95800640 1 --2- 172.21.15.12:0/461141532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd90105490 0x7efd901a1180 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efd8800eb10 tx=0x7efd8800c2d0 comp rx=0 tx=0).stop 2024-03-23T22:59:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.305+0000 7efd95800640 1 -- 172.21.15.12:0/461141532 shutdown_connections 2024-03-23T22:59:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.305+0000 7efd95800640 1 --2- 172.21.15.12:0/461141532 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7efd740a2800 0x7efd740a4cc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.305+0000 7efd95800640 1 --2- 172.21.15.12:0/461141532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd90105490 0x7efd901a1180 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.305+0000 7efd95800640 1 --2- 172.21.15.12:0/461141532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd90104ad0 0x7efd9019a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.305+0000 7efd95800640 1 --2- 172.21.15.12:0/461141532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd901038e0 0x7efd90199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.305+0000 7efd95800640 1 -- 172.21.15.12:0/461141532 >> 172.21.15.12:0/461141532 conn(0x7efd900fee90 msgr2=0x7efd90107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.305+0000 7efd95800640 1 -- 172.21.15.12:0/461141532 shutdown_connections 2024-03-23T22:59:24.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.305+0000 7efd95800640 1 -- 172.21.15.12:0/461141532 wait complete. 2024-03-23T22:59:24.328 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-23T22:59:24.328 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-23T22:59:24.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:24 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:24.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:24 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:24.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:24 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:24.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:24 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3221855605' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T22:59:24.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:24 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1142437538' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T22:59:24.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:24.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:24.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:24.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:24 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3221855605' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T22:59:24.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:24 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1142437538' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T22:59:24.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.552+0000 7f2213200640 1 -- 172.21.15.12:0/2183054154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22141015a0 msgr2=0x7f22141019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.552+0000 7f2213200640 1 --2- 172.21.15.12:0/2183054154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22141015a0 0x7f22141019c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2204009900 tx=0x7f220402bc40 comp rx=0 tx=0).stop 2024-03-23T22:59:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.552+0000 7f2213200640 1 -- 172.21.15.12:0/2183054154 shutdown_connections 2024-03-23T22:59:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.552+0000 7f2213200640 1 --2- 172.21.15.12:0/2183054154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2214103150 0x7f22141099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.552+0000 7f2213200640 1 --2- 172.21.15.12:0/2183054154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2214102790 0x7f2214102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.552+0000 7f2213200640 1 --2- 172.21.15.12:0/2183054154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22141015a0 0x7f22141019c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.552+0000 7f2213200640 1 -- 172.21.15.12:0/2183054154 >> 172.21.15.12:0/2183054154 conn(0x7f22140fcb10 msgr2=0x7f22140fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.552+0000 7f2213200640 1 -- 172.21.15.12:0/2183054154 shutdown_connections 2024-03-23T22:59:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.552+0000 7f2213200640 1 -- 172.21.15.12:0/2183054154 wait complete. 2024-03-23T22:59:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.554+0000 7f2213200640 1 Processor -- start 2024-03-23T22:59:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.554+0000 7f2213200640 1 -- start start 2024-03-23T22:59:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.554+0000 7f2213200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22141015a0 0x7f2214075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.554+0000 7f2213200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2214102790 0x7f2214073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.554+0000 7f2213200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2214103150 0x7f2214073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.554+0000 7f2213200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221410d260 con 0x7f2214102790 2024-03-23T22:59:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.554+0000 7f2213200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221410d0e0 con 0x7f22141015a0 2024-03-23T22:59:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.554+0000 7f2213200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221410d3e0 con 0x7f2214103150 2024-03-23T22:59:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.555+0000 7f2211e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22141015a0 0x7f2214075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.555+0000 7f2209400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2214102790 0x7f2214073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.555+0000 7f2211e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22141015a0 0x7f2214075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:32844/0 (socket says 172.21.15.12:32844) 2024-03-23T22:59:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.555+0000 7f2211e00640 1 -- 172.21.15.12:0/1817463655 learned_addr learned my addr 172.21.15.12:0/1817463655 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.555+0000 7f2212800640 1 --2- 172.21.15.12:0/1817463655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2214103150 0x7f2214073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.555+0000 7f2212800640 1 -- 172.21.15.12:0/1817463655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22141015a0 msgr2=0x7f2214075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.555+0000 7f2212800640 1 --2- 172.21.15.12:0/1817463655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22141015a0 0x7f2214075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.555+0000 7f2212800640 1 -- 172.21.15.12:0/1817463655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2214102790 msgr2=0x7f2214073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.555+0000 7f2212800640 1 --2- 172.21.15.12:0/1817463655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2214102790 0x7f2214073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.555+0000 7f2212800640 1 -- 172.21.15.12:0/1817463655 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2214074860 con 0x7f2214103150 2024-03-23T22:59:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.555+0000 7f2211e00640 1 --2- 172.21.15.12:0/1817463655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22141015a0 0x7f2214075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T22:59:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.556+0000 7f2212800640 1 --2- 172.21.15.12:0/1817463655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2214103150 0x7f2214073fd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f220c009fd0 tx=0x7f220c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.556+0000 7f220b400640 1 -- 172.21.15.12:0/1817463655 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220c00ce20 con 0x7f2214103150 2024-03-23T22:59:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.556+0000 7f220b400640 1 -- 172.21.15.12:0/1817463655 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f220c005020 con 0x7f2214103150 2024-03-23T22:59:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.556+0000 7f220b400640 1 -- 172.21.15.12:0/1817463655 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220c010720 con 0x7f2214103150 2024-03-23T22:59:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.556+0000 7f2213200640 1 -- 172.21.15.12:0/1817463655 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22141a19c0 con 0x7f2214103150 2024-03-23T22:59:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.556+0000 7f2213200640 1 -- 172.21.15.12:0/1817463655 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22141a1f60 con 0x7f2214103150 2024-03-23T22:59:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.557+0000 7f2213200640 1 -- 172.21.15.12:0/1817463655 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22141088c0 con 0x7f2214103150 2024-03-23T22:59:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.559+0000 7f220b400640 1 -- 172.21.15.12:0/1817463655 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f220c005540 con 0x7f2214103150 2024-03-23T22:59:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.560+0000 7f220b400640 1 --2- 172.21.15.12:0/1817463655 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f21f40a2880 0x7f21f40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:24.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.560+0000 7f220b400640 1 -- 172.21.15.12:0/1817463655 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f220c0cec10 con 0x7f2214103150 2024-03-23T22:59:24.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.561+0000 7f2211e00640 1 --2- 172.21.15.12:0/1817463655 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f21f40a2880 0x7f21f40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:24.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.561+0000 7f2211e00640 1 --2- 172.21.15.12:0/1817463655 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f21f40a2880 0x7f21f40a4d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2204005ab0 tx=0x7f220402cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:24.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.564+0000 7f220b400640 1 -- 172.21.15.12:0/1817463655 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f220c09c0b0 con 0x7f2214103150 2024-03-23T22:59:24.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:24.773+0000 7f2213200640 1 -- 172.21.15.12:0/1817463655 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f221402d070 con 0x7f2214103150 2024-03-23T22:59:25.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.407+0000 7f220b400640 1 -- 172.21.15.12:0/1817463655 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) v1 ==== 138+0+0 (secure 0 0 0) 0x7f220c09b9b0 con 0x7f2214103150 2024-03-23T22:59:25.410 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' created 2024-03-23T22:59:25.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.413+0000 7f2213200640 1 -- 172.21.15.12:0/1817463655 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f21f40a2880 msgr2=0x7f21f40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:25.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.413+0000 7f2213200640 1 --2- 172.21.15.12:0/1817463655 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f21f40a2880 0x7f21f40a4d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2204005ab0 tx=0x7f220402cf90 comp rx=0 tx=0).stop 2024-03-23T22:59:25.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.413+0000 7f2213200640 1 -- 172.21.15.12:0/1817463655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2214103150 msgr2=0x7f2214073fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:25.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.413+0000 7f2213200640 1 --2- 172.21.15.12:0/1817463655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2214103150 0x7f2214073fd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f220c009fd0 tx=0x7f220c00ef10 comp rx=0 tx=0).stop 2024-03-23T22:59:25.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.414+0000 7f2213200640 1 -- 172.21.15.12:0/1817463655 shutdown_connections 2024-03-23T22:59:25.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.414+0000 7f2213200640 1 --2- 172.21.15.12:0/1817463655 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f21f40a2880 0x7f21f40a4d40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:25.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.414+0000 7f2213200640 1 --2- 172.21.15.12:0/1817463655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2214103150 0x7f2214073fd0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:25.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.414+0000 7f2213200640 1 --2- 172.21.15.12:0/1817463655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2214102790 0x7f2214073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:25.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.414+0000 7f2213200640 1 --2- 172.21.15.12:0/1817463655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22141015a0 0x7f2214075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:25.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.414+0000 7f2213200640 1 -- 172.21.15.12:0/1817463655 >> 172.21.15.12:0/1817463655 conn(0x7f22140fcb10 msgr2=0x7f2214105a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:25.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.414+0000 7f2213200640 1 -- 172.21.15.12:0/1817463655 shutdown_connections 2024-03-23T22:59:25.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.414+0000 7f2213200640 1 -- 172.21.15.12:0/1817463655 wait complete. 2024-03-23T22:59:25.430 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-23T22:59:25.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:25 smithi173 ceph-mon[36116]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:25.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:25 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:25.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:25 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:25.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:25 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:59:25.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:25 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:59:25.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:25 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:25.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:25 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/461141532' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-23T22:59:25.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:25 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:59:25.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.621+0000 7f2b8b200640 1 -- 172.21.15.12:0/775822144 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b8c104990 msgr2=0x7f2b8c104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:25.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.621+0000 7f2b8b200640 1 --2- 172.21.15.12:0/775822144 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b8c104990 0x7f2b8c104e10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009900 tx=0x7f2b7802bc30 comp rx=0 tx=0).stop 2024-03-23T22:59:25.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.621+0000 7f2b8b200640 1 -- 172.21.15.12:0/775822144 shutdown_connections 2024-03-23T22:59:25.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.621+0000 7f2b8b200640 1 --2- 172.21.15.12:0/775822144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b8c105350 0x7f2b8c10bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:25.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.621+0000 7f2b8b200640 1 --2- 172.21.15.12:0/775822144 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b8c104990 0x7f2b8c104e10 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:25.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.621+0000 7f2b8b200640 1 --2- 172.21.15.12:0/775822144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b8c1037a0 0x7f2b8c103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:25.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.621+0000 7f2b8b200640 1 -- 172.21.15.12:0/775822144 >> 172.21.15.12:0/775822144 conn(0x7f2b8c0fed10 msgr2=0x7f2b8c101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:25.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.621+0000 7f2b8b200640 1 -- 172.21.15.12:0/775822144 shutdown_connections 2024-03-23T22:59:25.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.621+0000 7f2b8b200640 1 -- 172.21.15.12:0/775822144 wait complete. 2024-03-23T22:59:25.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.622+0000 7f2b8b200640 1 Processor -- start 2024-03-23T22:59:25.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.622+0000 7f2b8b200640 1 -- start start 2024-03-23T22:59:25.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.622+0000 7f2b8b200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b8c1037a0 0x7f2b8c199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:25.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.622+0000 7f2b8b200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b8c104990 0x7f2b8c19a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.622+0000 7f2b8b200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b8c105350 0x7f2b8c1a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.622+0000 7f2b8b200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c10f470 con 0x7f2b8c1037a0 2024-03-23T22:59:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.622+0000 7f2b8b200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c10f2f0 con 0x7f2b8c104990 2024-03-23T22:59:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.622+0000 7f2b8b200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c10f5f0 con 0x7f2b8c105350 2024-03-23T22:59:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.622+0000 7f2b89e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b8c1037a0 0x7f2b8c199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.622+0000 7f2b89e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b8c1037a0 0x7f2b8c199ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37090/0 (socket says 172.21.15.12:37090) 2024-03-23T22:59:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.622+0000 7f2b89e00640 1 -- 172.21.15.12:0/2754606415 learned_addr learned my addr 172.21.15.12:0/2754606415 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.622+0000 7f2b89400640 1 --2- 172.21.15.12:0/2754606415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b8c104990 0x7f2b8c19a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.622+0000 7f2b8a800640 1 --2- 172.21.15.12:0/2754606415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b8c105350 0x7f2b8c1a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.623+0000 7f2b89e00640 1 -- 172.21.15.12:0/2754606415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b8c105350 msgr2=0x7f2b8c1a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.623+0000 7f2b89e00640 1 --2- 172.21.15.12:0/2754606415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b8c105350 0x7f2b8c1a1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.623+0000 7f2b89e00640 1 -- 172.21.15.12:0/2754606415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b8c104990 msgr2=0x7f2b8c19a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.623+0000 7f2b89e00640 1 --2- 172.21.15.12:0/2754606415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b8c104990 0x7f2b8c19a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.623+0000 7f2b89e00640 1 -- 172.21.15.12:0/2754606415 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8c1a16e0 con 0x7f2b8c1037a0 2024-03-23T22:59:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.623+0000 7f2b89e00640 1 --2- 172.21.15.12:0/2754606415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b8c1037a0 0x7f2b8c199ac0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c00e9c0 tx=0x7f2b7c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.623+0000 7f2b82a00640 1 -- 172.21.15.12:0/2754606415 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c00cd80 con 0x7f2b8c1037a0 2024-03-23T22:59:25.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.623+0000 7f2b82a00640 1 -- 172.21.15.12:0/2754606415 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2b7c0035f0 con 0x7f2b8c1037a0 2024-03-23T22:59:25.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.623+0000 7f2b8b200640 1 -- 172.21.15.12:0/2754606415 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b8c1a19d0 con 0x7f2b8c1037a0 2024-03-23T22:59:25.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.623+0000 7f2b82a00640 1 -- 172.21.15.12:0/2754606415 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c010690 con 0x7f2b8c1037a0 2024-03-23T22:59:25.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.623+0000 7f2b8b200640 1 -- 172.21.15.12:0/2754606415 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b8c1a1ea0 con 0x7f2b8c1037a0 2024-03-23T22:59:25.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.625+0000 7f2b8b200640 1 -- 172.21.15.12:0/2754606415 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b8c10aac0 con 0x7f2b8c1037a0 2024-03-23T22:59:25.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.627+0000 7f2b82a00640 1 -- 172.21.15.12:0/2754606415 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2b7c0037a0 con 0x7f2b8c1037a0 2024-03-23T22:59:25.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.628+0000 7f2b82a00640 1 --2- 172.21.15.12:0/2754606415 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2b700a2800 0x7f2b700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:25.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.628+0000 7f2b82a00640 1 -- 172.21.15.12:0/2754606415 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f2b7c0ceb30 con 0x7f2b8c1037a0 2024-03-23T22:59:25.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.628+0000 7f2b89400640 1 --2- 172.21.15.12:0/2754606415 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2b700a2800 0x7f2b700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:25.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.629+0000 7f2b89400640 1 --2- 172.21.15.12:0/2754606415 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2b700a2800 0x7f2b700a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2b7802c240 tx=0x7f2b7802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:25.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.632+0000 7f2b82a00640 1 -- 172.21.15.12:0/2754606415 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2b7c09be60 con 0x7f2b8c1037a0 2024-03-23T22:59:25.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:25 smithi203 ceph-mon[36693]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:25.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:25 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:25.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:25 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:25.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:25 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:59:25.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:25 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:59:25.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:25 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:25.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:25 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/461141532' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-23T22:59:25.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:25 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:59:25.661 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:25 smithi012 ceph-mon[31695]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:25.662 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:25 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:25.662 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:25 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:25.662 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:25 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:59:25.662 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:25 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:59:25.662 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:25 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:25.662 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:25 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/461141532' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-23T22:59:25.662 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:25 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T22:59:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:25.842+0000 7f2b8b200640 1 -- 172.21.15.12:0/2754606415 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f2b8c02d070 con 0x7f2b8c1037a0 2024-03-23T22:59:26.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.411+0000 7f2b82a00640 1 -- 172.21.15.12:0/2754606415 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v84) v1 ==== 170+0+0 (secure 0 0 0) 0x7f2b7c09b760 con 0x7f2b8c1037a0 2024-03-23T22:59:26.412 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' created 2024-03-23T22:59:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.415+0000 7f2b8b200640 1 -- 172.21.15.12:0/2754606415 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2b700a2800 msgr2=0x7f2b700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:26.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.415+0000 7f2b8b200640 1 --2- 172.21.15.12:0/2754606415 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2b700a2800 0x7f2b700a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2b7802c240 tx=0x7f2b7802df90 comp rx=0 tx=0).stop 2024-03-23T22:59:26.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.415+0000 7f2b8b200640 1 -- 172.21.15.12:0/2754606415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b8c1037a0 msgr2=0x7f2b8c199ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:26.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.415+0000 7f2b8b200640 1 --2- 172.21.15.12:0/2754606415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b8c1037a0 0x7f2b8c199ac0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c00e9c0 tx=0x7f2b7c00ee80 comp rx=0 tx=0).stop 2024-03-23T22:59:26.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.416+0000 7f2b8b200640 1 -- 172.21.15.12:0/2754606415 shutdown_connections 2024-03-23T22:59:26.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.416+0000 7f2b8b200640 1 --2- 172.21.15.12:0/2754606415 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2b700a2800 0x7f2b700a4cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:26.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.417+0000 7f2b8b200640 1 --2- 172.21.15.12:0/2754606415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b8c105350 0x7f2b8c1a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:26.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.417+0000 7f2b8b200640 1 --2- 172.21.15.12:0/2754606415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b8c104990 0x7f2b8c19a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:26.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.417+0000 7f2b8b200640 1 --2- 172.21.15.12:0/2754606415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b8c1037a0 0x7f2b8c199ac0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:26.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.417+0000 7f2b8b200640 1 -- 172.21.15.12:0/2754606415 >> 172.21.15.12:0/2754606415 conn(0x7f2b8c0fed10 msgr2=0x7f2b8c107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:26.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.417+0000 7f2b8b200640 1 -- 172.21.15.12:0/2754606415 shutdown_connections 2024-03-23T22:59:26.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.418+0000 7f2b8b200640 1 -- 172.21.15.12:0/2754606415 wait complete. 2024-03-23T22:59:26.457 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-23T22:59:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.647+0000 7f5630200640 1 -- 172.21.15.12:0/2090442513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56281038e0 msgr2=0x7f5628103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.647+0000 7f5630200640 1 --2- 172.21.15.12:0/2090442513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56281038e0 0x7f5628103d00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f561c009900 tx=0x7f561c02bc50 comp rx=0 tx=0).stop 2024-03-23T22:59:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.647+0000 7f5630200640 1 -- 172.21.15.12:0/2090442513 shutdown_connections 2024-03-23T22:59:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.647+0000 7f5630200640 1 --2- 172.21.15.12:0/2090442513 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5628105490 0x7f562810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.647+0000 7f5630200640 1 --2- 172.21.15.12:0/2090442513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5628104ad0 0x7f5628104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.647+0000 7f5630200640 1 --2- 172.21.15.12:0/2090442513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56281038e0 0x7f5628103d00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.647+0000 7f5630200640 1 -- 172.21.15.12:0/2090442513 >> 172.21.15.12:0/2090442513 conn(0x7f56280fee50 msgr2=0x7f5628101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.648+0000 7f5630200640 1 -- 172.21.15.12:0/2090442513 shutdown_connections 2024-03-23T22:59:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.648+0000 7f5630200640 1 -- 172.21.15.12:0/2090442513 wait complete. 2024-03-23T22:59:26.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.649+0000 7f5630200640 1 Processor -- start 2024-03-23T22:59:26.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.649+0000 7f5630200640 1 -- start start 2024-03-23T22:59:26.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.649+0000 7f5630200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56281038e0 0x7f562819e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:26.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.649+0000 7f5630200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5628104ad0 0x7f562819e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:26.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.649+0000 7f5630200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5628105490 0x7f56281a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:26.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.649+0000 7f5630200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562810f580 con 0x7f5628104ad0 2024-03-23T22:59:26.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.649+0000 7f5630200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562810f400 con 0x7f56281038e0 2024-03-23T22:59:26.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.649+0000 7f5630200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562810f700 con 0x7f5628105490 2024-03-23T22:59:26.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.649+0000 7f562ce00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5628104ad0 0x7f562819e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:26.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.649+0000 7f562ce00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5628104ad0 0x7f562819e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37114/0 (socket says 172.21.15.12:37114) 2024-03-23T22:59:26.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.649+0000 7f562ce00640 1 -- 172.21.15.12:0/641547794 learned_addr learned my addr 172.21.15.12:0/641547794 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:26.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.649+0000 7f562e200640 1 --2- 172.21.15.12:0/641547794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5628105490 0x7f56281a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:26.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.649+0000 7f562d800640 1 --2- 172.21.15.12:0/641547794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56281038e0 0x7f562819e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:26.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.650+0000 7f562ce00640 1 -- 172.21.15.12:0/641547794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5628105490 msgr2=0x7f56281a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:26.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.650+0000 7f562ce00640 1 --2- 172.21.15.12:0/641547794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5628105490 0x7f56281a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:26.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.650+0000 7f562ce00640 1 -- 172.21.15.12:0/641547794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56281038e0 msgr2=0x7f562819e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:26.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.650+0000 7f562ce00640 1 --2- 172.21.15.12:0/641547794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56281038e0 0x7f562819e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:26.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.650+0000 7f562ce00640 1 -- 172.21.15.12:0/641547794 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56281a5c70 con 0x7f5628104ad0 2024-03-23T22:59:26.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.650+0000 7f562e200640 1 --2- 172.21.15.12:0/641547794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5628105490 0x7f56281a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T22:59:26.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.650+0000 7f562d800640 1 --2- 172.21.15.12:0/641547794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56281038e0 0x7f562819e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T22:59:26.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.650+0000 7f562ce00640 1 --2- 172.21.15.12:0/641547794 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5628104ad0 0x7f562819e590 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f561400ea60 tx=0x7f561400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:26.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.650+0000 7f5626000640 1 -- 172.21.15.12:0/641547794 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561400cde0 con 0x7f5628104ad0 2024-03-23T22:59:26.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.650+0000 7f5626000640 1 -- 172.21.15.12:0/641547794 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56140035f0 con 0x7f5628104ad0 2024-03-23T22:59:26.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.650+0000 7f5626000640 1 -- 172.21.15.12:0/641547794 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5614010670 con 0x7f5628104ad0 2024-03-23T22:59:26.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.650+0000 7f5630200640 1 -- 172.21.15.12:0/641547794 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56281a5f60 con 0x7f5628104ad0 2024-03-23T22:59:26.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.651+0000 7f5630200640 1 -- 172.21.15.12:0/641547794 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56281a6420 con 0x7f5628104ad0 2024-03-23T22:59:26.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.652+0000 7f5630200640 1 -- 172.21.15.12:0/641547794 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56281a66b0 con 0x7f5628104ad0 2024-03-23T22:59:26.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.657+0000 7f5626000640 1 -- 172.21.15.12:0/641547794 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f56140037a0 con 0x7f5628104ad0 2024-03-23T22:59:26.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:26 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1817463655' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-23T22:59:26.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:26 smithi203 ceph-mon[36693]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-23T22:59:26.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:26 smithi203 ceph-mon[36693]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-23T22:59:26.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:26 smithi203 ceph-mon[36693]: osdmap e83: 12 total, 12 up, 12 in 2024-03-23T22:59:26.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.658+0000 7f5626000640 1 --2- 172.21.15.12:0/641547794 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f56100a2800 0x7f56100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:26.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.658+0000 7f5626000640 1 -- 172.21.15.12:0/641547794 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f56140cec50 con 0x7f5628104ad0 2024-03-23T22:59:26.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.658+0000 7f562d800640 1 --2- 172.21.15.12:0/641547794 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f56100a2800 0x7f56100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:26.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.659+0000 7f5626000640 1 -- 172.21.15.12:0/641547794 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f561409bf10 con 0x7f5628104ad0 2024-03-23T22:59:26.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.659+0000 7f562d800640 1 --2- 172.21.15.12:0/641547794 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f56100a2800 0x7f56100a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f561c02c190 tx=0x7f561c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:26.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:26 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1817463655' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-23T22:59:26.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:26 smithi012 ceph-mon[31695]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-23T22:59:26.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:26 smithi012 ceph-mon[31695]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-23T22:59:26.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:26 smithi012 ceph-mon[31695]: osdmap e83: 12 total, 12 up, 12 in 2024-03-23T22:59:26.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:26 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1817463655' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-23T22:59:26.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:26 smithi173 ceph-mon[36116]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-23T22:59:26.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:26 smithi173 ceph-mon[36116]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-23T22:59:26.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:26 smithi173 ceph-mon[36116]: osdmap e83: 12 total, 12 up, 12 in 2024-03-23T22:59:26.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:26.903+0000 7f5630200640 1 -- 172.21.15.12:0/641547794 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f56281a6880 con 0x7f5628104ad0 2024-03-23T22:59:27.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.435+0000 7f5626000640 1 -- 172.21.15.12:0/641547794 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f561409b810 con 0x7f5628104ad0 2024-03-23T22:59:27.436 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 2 and data pool 3 2024-03-23T22:59:27.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.439+0000 7f5630200640 1 -- 172.21.15.12:0/641547794 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f56100a2800 msgr2=0x7f56100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:27.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.439+0000 7f5630200640 1 --2- 172.21.15.12:0/641547794 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f56100a2800 0x7f56100a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f561c02c190 tx=0x7f561c000bc0 comp rx=0 tx=0).stop 2024-03-23T22:59:27.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.439+0000 7f5630200640 1 -- 172.21.15.12:0/641547794 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5628104ad0 msgr2=0x7f562819e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:27.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.439+0000 7f5630200640 1 --2- 172.21.15.12:0/641547794 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5628104ad0 0x7f562819e590 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f561400ea60 tx=0x7f561400ef20 comp rx=0 tx=0).stop 2024-03-23T22:59:27.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.439+0000 7f5630200640 1 -- 172.21.15.12:0/641547794 shutdown_connections 2024-03-23T22:59:27.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.439+0000 7f5630200640 1 --2- 172.21.15.12:0/641547794 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f56100a2800 0x7f56100a4cc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:27.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.439+0000 7f5630200640 1 --2- 172.21.15.12:0/641547794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5628105490 0x7f56281a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:27.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.439+0000 7f5630200640 1 --2- 172.21.15.12:0/641547794 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5628104ad0 0x7f562819e590 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:27.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.439+0000 7f5630200640 1 --2- 172.21.15.12:0/641547794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56281038e0 0x7f562819e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:27.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.439+0000 7f5630200640 1 -- 172.21.15.12:0/641547794 >> 172.21.15.12:0/641547794 conn(0x7f56280fee50 msgr2=0x7f5628077af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:27.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.439+0000 7f5630200640 1 -- 172.21.15.12:0/641547794 shutdown_connections 2024-03-23T22:59:27.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.439+0000 7f5630200640 1 -- 172.21.15.12:0/641547794 wait complete. 2024-03-23T22:59:27.465 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T22:59:27.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:27 smithi203 ceph-mon[36693]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:27.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:27 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2754606415' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-23T22:59:27.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:27 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:27.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:27 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:27.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:27 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2754606415' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-23T22:59:27.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:27 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:27.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:27 smithi203 ceph-mon[36693]: osdmap e84: 12 total, 12 up, 12 in 2024-03-23T22:59:27.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:27 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:27.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:27 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/641547794' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-23T22:59:27.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:27 smithi203 ceph-mon[36693]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-23T22:59:27.659 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:27 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:27.659 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:27 smithi203 ceph-mon[36693]: osdmap e85: 12 total, 12 up, 12 in 2024-03-23T22:59:27.659 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:27 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:27.659 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:27 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:59:27.659 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:27 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:59:27.659 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:27 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:27.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.668+0000 7f42a6800640 1 -- 172.21.15.12:0/2888777949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42a00772a0 msgr2=0x7f42a0077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:27.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.668+0000 7f42a6800640 1 --2- 172.21.15.12:0/2888777949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42a00772a0 0x7f42a0077700 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4288009900 tx=0x7f428802bc30 comp rx=0 tx=0).stop 2024-03-23T22:59:27.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.668+0000 7f42a6800640 1 -- 172.21.15.12:0/2888777949 shutdown_connections 2024-03-23T22:59:27.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.668+0000 7f42a6800640 1 --2- 172.21.15.12:0/2888777949 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42a0077c40 0x7f42a00784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:27.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.668+0000 7f42a6800640 1 --2- 172.21.15.12:0/2888777949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42a00772a0 0x7f42a0077700 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:27.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.669+0000 7f42a6800640 1 --2- 172.21.15.12:0/2888777949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42a0076060 0x7f42a0076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:27.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.669+0000 7f42a6800640 1 -- 172.21.15.12:0/2888777949 >> 172.21.15.12:0/2888777949 conn(0x7f42a00fee50 msgr2=0x7f42a0101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:27.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.669+0000 7f42a6800640 1 -- 172.21.15.12:0/2888777949 shutdown_connections 2024-03-23T22:59:27.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.669+0000 7f42a6800640 1 -- 172.21.15.12:0/2888777949 wait complete. 2024-03-23T22:59:27.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.670+0000 7f42a6800640 1 Processor -- start 2024-03-23T22:59:27.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.670+0000 7f42a6800640 1 -- start start 2024-03-23T22:59:27.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.670+0000 7f42a6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42a0076060 0x7f42a019dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:27.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.670+0000 7f42a6800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42a00772a0 0x7f42a019e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:27.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.670+0000 7f42a6800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42a0077c40 0x7f42a01a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:27.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.670+0000 7f42a6800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a0113990 con 0x7f42a00772a0 2024-03-23T22:59:27.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.670+0000 7f42a6800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a0113810 con 0x7f42a0076060 2024-03-23T22:59:27.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.670+0000 7f42a6800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a0113b10 con 0x7f42a0077c40 2024-03-23T22:59:27.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.670+0000 7f429ea00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42a00772a0 0x7f42a019e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:27.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.670+0000 7f429f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42a0076060 0x7f42a019dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:27.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.670+0000 7f429f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42a0076060 0x7f42a019dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:32894/0 (socket says 172.21.15.12:32894) 2024-03-23T22:59:27.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.670+0000 7f429ea00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42a00772a0 0x7f42a019e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37142/0 (socket says 172.21.15.12:37142) 2024-03-23T22:59:27.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.670+0000 7f429f400640 1 -- 172.21.15.12:0/3419639484 learned_addr learned my addr 172.21.15.12:0/3419639484 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:27.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.671+0000 7f429fe00640 1 --2- 172.21.15.12:0/3419639484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42a0077c40 0x7f42a01a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:27.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.671+0000 7f429f400640 1 -- 172.21.15.12:0/3419639484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42a0077c40 msgr2=0x7f42a01a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:27.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.671+0000 7f429f400640 1 --2- 172.21.15.12:0/3419639484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42a0077c40 0x7f42a01a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:27.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.671+0000 7f429f400640 1 -- 172.21.15.12:0/3419639484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42a00772a0 msgr2=0x7f42a019e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:27.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.671+0000 7f429f400640 1 --2- 172.21.15.12:0/3419639484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42a00772a0 0x7f42a019e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:27.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.671+0000 7f429f400640 1 -- 172.21.15.12:0/3419639484 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a01a5bd0 con 0x7f42a0076060 2024-03-23T22:59:27.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.671+0000 7f429ea00640 1 --2- 172.21.15.12:0/3419639484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42a00772a0 0x7f42a019e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T22:59:27.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.671+0000 7f429f400640 1 --2- 172.21.15.12:0/3419639484 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42a0076060 0x7f42a019dfb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f429000e9f0 tx=0x7f429000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:27.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.672+0000 7f4297e00640 1 -- 172.21.15.12:0/3419639484 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429000cd80 con 0x7f42a0076060 2024-03-23T22:59:27.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.672+0000 7f42a6800640 1 -- 172.21.15.12:0/3419639484 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a01a5ec0 con 0x7f42a0076060 2024-03-23T22:59:27.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.672+0000 7f4297e00640 1 -- 172.21.15.12:0/3419639484 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4290010910 con 0x7f42a0076060 2024-03-23T22:59:27.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.672+0000 7f4297e00640 1 -- 172.21.15.12:0/3419639484 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42900188f0 con 0x7f42a0076060 2024-03-23T22:59:27.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.672+0000 7f42a6800640 1 -- 172.21.15.12:0/3419639484 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a01a6330 con 0x7f42a0076060 2024-03-23T22:59:27.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.673+0000 7f42a6800640 1 -- 172.21.15.12:0/3419639484 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a0110230 con 0x7f42a0076060 2024-03-23T22:59:27.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.675+0000 7f4297e00640 1 -- 172.21.15.12:0/3419639484 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4290010ac0 con 0x7f42a0076060 2024-03-23T22:59:27.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.676+0000 7f4297e00640 1 --2- 172.21.15.12:0/3419639484 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f42840a2800 0x7f42840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:27.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.677+0000 7f4297e00640 1 -- 172.21.15.12:0/3419639484 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f42900cef00 con 0x7f42a0076060 2024-03-23T22:59:27.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.677+0000 7f429ea00640 1 --2- 172.21.15.12:0/3419639484 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f42840a2800 0x7f42840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:27.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.677+0000 7f429ea00640 1 --2- 172.21.15.12:0/3419639484 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f42840a2800 0x7f42840a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f428802c240 tx=0x7f428802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:27.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.679+0000 7f4297e00640 1 -- 172.21.15.12:0/3419639484 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f429009ba00 con 0x7f42a0076060 2024-03-23T22:59:27.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:27 smithi012 ceph-mon[31695]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:27.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:27 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2754606415' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-23T22:59:27.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:27 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:27.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:27 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:27.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:27 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2754606415' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-23T22:59:27.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:27 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:27.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:27 smithi012 ceph-mon[31695]: osdmap e84: 12 total, 12 up, 12 in 2024-03-23T22:59:27.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:27 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:27.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:27 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/641547794' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-23T22:59:27.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:27 smithi012 ceph-mon[31695]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-23T22:59:27.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:27 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:27.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:27 smithi012 ceph-mon[31695]: osdmap e85: 12 total, 12 up, 12 in 2024-03-23T22:59:27.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:27 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:27.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:27 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:59:27.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:27 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:59:27.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:27 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:27.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:27 smithi173 ceph-mon[36116]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:27.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:27 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2754606415' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-23T22:59:27.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:27 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:27.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:27 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:27.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:27 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2754606415' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-23T22:59:27.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:27 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:27.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:27 smithi173 ceph-mon[36116]: osdmap e84: 12 total, 12 up, 12 in 2024-03-23T22:59:27.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:27 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:27.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:27 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/641547794' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-23T22:59:27.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:27 smithi173 ceph-mon[36116]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-23T22:59:27.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:27 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:27.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:27 smithi173 ceph-mon[36116]: osdmap e85: 12 total, 12 up, 12 in 2024-03-23T22:59:27.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:27 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:27.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:27 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T22:59:27.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:27 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T22:59:27.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:27 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T22:59:27.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.883+0000 7f42a6800640 1 -- 172.21.15.12:0/3419639484 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42a00622a0 con 0x7f42a0076060 2024-03-23T22:59:27.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.885+0000 7f4297e00640 1 -- 172.21.15.12:0/3419639484 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22257 (secure 0 0 0) 0x7f429009faa0 con 0x7f42a0076060 2024-03-23T22:59:27.885 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:59:27.885 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":85,"fsid":"90f679ae-e967-11ee-95cd-87774f69a715","created":"2024-03-23T22:50:14.906952+0000","modified":"2024-03-23T22:59:26.905742+0000","last_up_change":"2024-03-23T22:58:28.907824+0000","last_in_change":"2024-03-23T22:58:08.342487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T22:54:11.831010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T22:59:25.350974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T22:59:26.390323+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b70875d7-a23d-4251-9e8b-1a16d4edc2a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6803","nonce":1190079611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6805","nonce":1190079611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6809","nonce":1190079611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6807","nonce":1190079611}]},"public_addr":"172.21.15.12:6803/1190079611","cluster_addr":"172.21.15.12:6805/1190079611","heartbeat_back_addr":"172.21.15.12:6809/1190079611","heartbeat_front_addr":"172.21.15.12:6807/1190079611","state":["exists","up"]},{"osd":1,"uuid":"416e87af-db5d-41ce-84d9-731e3e07e3c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6801","nonce":1983639525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6803","nonce":1983639525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6807","nonce":1983639525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6805","nonce":1983639525}]},"public_addr":"172.21.15.173:6801/1983639525","cluster_addr":"172.21.15.173:6803/1983639525","heartbeat_back_addr":"172.21.15.173:6807/1983639525","heartbeat_front_addr":"172.21.15.173:6805/1983639525","state":["exists","up"]},{"osd":2,"uuid":"7a559c70-4c54-431b-8baf-4136b6e76493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6801","nonce":2253997387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6803","nonce":2253997387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6807","nonce":2253997387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6805","nonce":2253997387}]},"public_addr":"172.21.15.203:6801/2253997387","cluster_addr":"172.21.15.203:6803/2253997387","heartbeat_back_addr":"172.21.15.203:6807/2253997387","heartbeat_front_addr":"172.21.15.203:6805/2253997387","state":["exists","up"]},{"osd":3,"uuid":"c41cd287-a674-486c-8025-328cd36e5ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6811","nonce":1761446905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6813","nonce":1761446905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6817","nonce":1761446905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6815","nonce":1761446905}]},"public_addr":"172.21.15.12:6811/1761446905","cluster_addr":"172.21.15.12:6813/1761446905","heartbeat_back_addr":"172.21.15.12:6817/1761446905","heartbeat_front_addr":"172.21.15.12:6815/1761446905","state":["exists","up"]},{"osd":4,"uuid":"3c48f886-0b42-4a09-b31e-abccc200bb1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6809","nonce":242958272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6811","nonce":242958272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6815","nonce":242958272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6813","nonce":242958272}]},"public_addr":"172.21.15.173:6809/242958272","cluster_addr":"172.21.15.173:6811/242958272","heartbeat_back_addr":"172.21.15.173:6815/242958272","heartbeat_front_addr":"172.21.15.173:6813/242958272","state":["exists","up"]},{"osd":5,"uuid":"4a34f06f-8805-40c0-b1ab-23407b0276d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6809","nonce":4270966092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6811","nonce":4270966092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6815","nonce":4270966092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6813","nonce":4270966092}]},"public_addr":"172.21.15.203:6809/4270966092","cluster_addr":"172.21.15.203:6811/4270966092","heartbeat_back_addr":"172.21.15.203:6815/4270966092","heartbeat_front_addr":"172.21.15.203:6813/4270966092","state":["exists","up"]},{"osd":6,"uuid":"26ddc663-5623-48c9-80d1-f8b968765d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6819","nonce":3606839295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6821","nonce":3606839295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6825","nonce":3606839295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6823","nonce":3606839295}]},"public_addr":"172.21.15.12:6819/3606839295","cluster_addr":"172.21.15.12:6821/3606839295","heartbeat_back_addr":"172.21.15.12:6825/3606839295","heartbeat_front_addr":"172.21.15.12:6823/3606839295","state":["exists","up"]},{"osd":7,"uuid":"17dc9417-71cd-4576-aa08-933212ec55e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6817","nonce":1241504346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6819","nonce":1241504346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6823","nonce":1241504346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6821","nonce":1241504346}]},"public_addr":"172.21.15.173:6817/1241504346","cluster_addr":"172.21.15.173:6819/1241504346","heartbeat_back_addr":"172.21.15.173:6823/1241504346","heartbeat_front_addr":"172.21.15.173:6821/1241504346","state":["exists","up"]},{"osd":8,"uuid":"90a2b63a-471d-43b8-ac18-bb1f16778d18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6817","nonce":1516996192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6819","nonce":1516996192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6823","nonce":1516996192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6821","nonce":1516996192}]},"public_addr":"172.21.15.203:6817/1516996192","cluster_addr":"172.21.15.203:6819/1516996192","heartbeat_back_addr":"172.21.15.203:6823/1516996192","heartbeat_front_addr":"172.21.15.203:6821/1516996192","state":["exists","up"]},{"osd":9,"uuid":"6e1ca504-40b4-49d0-b245-7946181abfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6827","nonce":1175260774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6829","nonce":1175260774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6833","nonce":1175260774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6831","nonce":1175260774}]},"public_addr":"172.21.15.12:6827/1175260774","cluster_addr":"172.21.15.12:6829/1175260774","heartbeat_back_addr":"172.21.15.12:6833/1175260774","heartbeat_front_addr":"172.21.15.12:6831/1175260774","state":["exists","up"]},{"osd":10,"uuid":"09a595ed-5614-4386-8e61-0170af66e01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6825","nonce":3303506808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6827","nonce":3303506808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6831","nonce":3303506808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6829","nonce":3303506808}]},"public_addr":"172.21.15.173:6825/3303506808","cluster_addr":"172.21.15.173:6827/3303506808","heartbeat_back_addr":"172.21.15.173:6831/3303506808","heartbeat_front_addr":"172.21.15.173:6829/3303506808","state":["exists","up"]},{"osd":11,"uuid":"9ef1d595-a977-4ba7-9c6b-d2b3295f20b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6825","nonce":2323604506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6827","nonce":2323604506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6831","nonce":2323604506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6829","nonce":2323604506}]},"public_addr":"172.21.15.203:6825/2323604506","cluster_addr":"172.21.15.203:6827/2323604506","heartbeat_back_addr":"172.21.15.203:6831/2323604506","heartbeat_front_addr":"172.21.15.203:6829/2323604506","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:18.469038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:42.752628+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:07.069493+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:41.962478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:08.363104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:33.341336+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:07.286541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:34.243302+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:59.749699+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:57:33.822913+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:00.737269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:26.599854+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/4182783120":"2024-03-24T22:51:23.113201+0000","172.21.15.12:6800/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1900938637":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/3414861652":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6801/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/2580541477":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6800/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/501058839":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1192554716":"2024-03-24T22:50:48.401223+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T22:59:27.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.888+0000 7f42a6800640 1 -- 172.21.15.12:0/3419639484 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f42840a2800 msgr2=0x7f42840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:27.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.888+0000 7f42a6800640 1 --2- 172.21.15.12:0/3419639484 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f42840a2800 0x7f42840a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f428802c240 tx=0x7f428802df90 comp rx=0 tx=0).stop 2024-03-23T22:59:27.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.888+0000 7f42a6800640 1 -- 172.21.15.12:0/3419639484 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42a0076060 msgr2=0x7f42a019dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:27.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.888+0000 7f42a6800640 1 --2- 172.21.15.12:0/3419639484 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42a0076060 0x7f42a019dfb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f429000e9f0 tx=0x7f429000eeb0 comp rx=0 tx=0).stop 2024-03-23T22:59:27.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.888+0000 7f42a6800640 1 -- 172.21.15.12:0/3419639484 shutdown_connections 2024-03-23T22:59:27.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.888+0000 7f42a6800640 1 --2- 172.21.15.12:0/3419639484 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f42840a2800 0x7f42840a4cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:27.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.888+0000 7f42a6800640 1 --2- 172.21.15.12:0/3419639484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42a0077c40 0x7f42a01a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:27.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.888+0000 7f42a6800640 1 --2- 172.21.15.12:0/3419639484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42a00772a0 0x7f42a019e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:27.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.888+0000 7f42a6800640 1 --2- 172.21.15.12:0/3419639484 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42a0076060 0x7f42a019dfb0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:27.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.888+0000 7f42a6800640 1 -- 172.21.15.12:0/3419639484 >> 172.21.15.12:0/3419639484 conn(0x7f42a00fee50 msgr2=0x7f42a0100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:27.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.888+0000 7f42a6800640 1 -- 172.21.15.12:0/3419639484 shutdown_connections 2024-03-23T22:59:27.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:27.889+0000 7f42a6800640 1 -- 172.21.15.12:0/3419639484 wait complete. 2024-03-23T22:59:27.906 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T22:59:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.120+0000 7f3070c00640 1 -- 172.21.15.12:0/2525702495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f306c109890 msgr2=0x7f306c110120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.120+0000 7f3070c00640 1 --2- 172.21.15.12:0/2525702495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f306c109890 0x7f306c110120 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3064009ca0 tx=0x7f306402c0f0 comp rx=0 tx=0).stop 2024-03-23T22:59:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.121+0000 7f3070c00640 1 -- 172.21.15.12:0/2525702495 shutdown_connections 2024-03-23T22:59:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.121+0000 7f3070c00640 1 --2- 172.21.15.12:0/2525702495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f306c109890 0x7f306c110120 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:28.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.121+0000 7f3070c00640 1 --2- 172.21.15.12:0/2525702495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f306c108ed0 0x7f306c109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:28.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.121+0000 7f3070c00640 1 --2- 172.21.15.12:0/2525702495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f306c107ce0 0x7f306c108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:28.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.121+0000 7f3070c00640 1 -- 172.21.15.12:0/2525702495 >> 172.21.15.12:0/2525702495 conn(0x7f306c076cb0 msgr2=0x7f306c0790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:28.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.121+0000 7f3070c00640 1 -- 172.21.15.12:0/2525702495 shutdown_connections 2024-03-23T22:59:28.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.121+0000 7f3070c00640 1 -- 172.21.15.12:0/2525702495 wait complete. 2024-03-23T22:59:28.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.122+0000 7f3070c00640 1 Processor -- start 2024-03-23T22:59:28.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.122+0000 7f3070c00640 1 -- start start 2024-03-23T22:59:28.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.122+0000 7f3070c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f306c107ce0 0x7f306c19df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:28.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.122+0000 7f3070c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f306c108ed0 0x7f306c19e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:28.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.122+0000 7f3070c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f306c109890 0x7f306c1a5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:28.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.122+0000 7f3070c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306c113960 con 0x7f306c109890 2024-03-23T22:59:28.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.122+0000 7f3070c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306c1137e0 con 0x7f306c107ce0 2024-03-23T22:59:28.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.122+0000 7f3070c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306c113ae0 con 0x7f306c108ed0 2024-03-23T22:59:28.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.122+0000 7f3069e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f306c107ce0 0x7f306c19df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:28.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.122+0000 7f306a800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f306c109890 0x7f306c1a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:28.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.122+0000 7f3069e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f306c107ce0 0x7f306c19df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:32904/0 (socket says 172.21.15.12:32904) 2024-03-23T22:59:28.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.122+0000 7f3069e00640 1 -- 172.21.15.12:0/2607124704 learned_addr learned my addr 172.21.15.12:0/2607124704 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:28.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.123+0000 7f3061400640 1 --2- 172.21.15.12:0/2607124704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f306c108ed0 0x7f306c19e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:28.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.123+0000 7f3069e00640 1 -- 172.21.15.12:0/2607124704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f306c108ed0 msgr2=0x7f306c19e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:28.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.123+0000 7f3069e00640 1 --2- 172.21.15.12:0/2607124704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f306c108ed0 0x7f306c19e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:28.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.123+0000 7f3069e00640 1 -- 172.21.15.12:0/2607124704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f306c109890 msgr2=0x7f306c1a5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:28.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.123+0000 7f3069e00640 1 --2- 172.21.15.12:0/2607124704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f306c109890 0x7f306c1a5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:28.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.123+0000 7f3069e00640 1 -- 172.21.15.12:0/2607124704 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306c1a5ba0 con 0x7f306c107ce0 2024-03-23T22:59:28.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.123+0000 7f306a800640 1 --2- 172.21.15.12:0/2607124704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f306c109890 0x7f306c1a5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T22:59:28.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.123+0000 7f3069e00640 1 --2- 172.21.15.12:0/2607124704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f306c107ce0 0x7f306c19df80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f305c00e9c0 tx=0x7f305c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:28.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.124+0000 7f3063400640 1 -- 172.21.15.12:0/2607124704 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305c00cd80 con 0x7f306c107ce0 2024-03-23T22:59:28.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.124+0000 7f3070c00640 1 -- 172.21.15.12:0/2607124704 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f306c1a5e90 con 0x7f306c107ce0 2024-03-23T22:59:28.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.124+0000 7f3063400640 1 -- 172.21.15.12:0/2607124704 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f305c0035f0 con 0x7f306c107ce0 2024-03-23T22:59:28.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.124+0000 7f3063400640 1 -- 172.21.15.12:0/2607124704 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305c010690 con 0x7f306c107ce0 2024-03-23T22:59:28.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.124+0000 7f3070c00640 1 -- 172.21.15.12:0/2607124704 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f306c1a62f0 con 0x7f306c107ce0 2024-03-23T22:59:28.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.125+0000 7f3070c00640 1 -- 172.21.15.12:0/2607124704 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f306c10f000 con 0x7f306c107ce0 2024-03-23T22:59:28.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.127+0000 7f3063400640 1 -- 172.21.15.12:0/2607124704 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f305c0037a0 con 0x7f306c107ce0 2024-03-23T22:59:28.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.128+0000 7f3063400640 1 --2- 172.21.15.12:0/2607124704 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f30500a2800 0x7f30500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:28.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.129+0000 7f3063400640 1 -- 172.21.15.12:0/2607124704 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f305c0ced10 con 0x7f306c107ce0 2024-03-23T22:59:28.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.129+0000 7f3061400640 1 --2- 172.21.15.12:0/2607124704 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f30500a2800 0x7f30500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:28.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.129+0000 7f3061400640 1 --2- 172.21.15.12:0/2607124704 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f30500a2800 0x7f30500a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f30540095f0 tx=0x7f3054005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:28.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.130+0000 7f3063400640 1 -- 172.21.15.12:0/2607124704 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f305c09be60 con 0x7f306c107ce0 2024-03-23T22:59:28.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.337+0000 7f3070c00640 1 -- 172.21.15.12:0/2607124704 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f306c02d1f0 con 0x7f306c107ce0 2024-03-23T22:59:28.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.339+0000 7f3063400640 1 -- 172.21.15.12:0/2607124704 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+22244 (secure 0 0 0) 0x7f305c09b760 con 0x7f306c107ce0 2024-03-23T22:59:28.340 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:59:28.340 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":86,"fsid":"90f679ae-e967-11ee-95cd-87774f69a715","created":"2024-03-23T22:50:14.906952+0000","modified":"2024-03-23T22:59:28.166927+0000","last_up_change":"2024-03-23T22:58:28.907824+0000","last_in_change":"2024-03-23T22:58:08.342487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T22:54:11.831010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T22:59:25.350974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T22:59:26.390323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b70875d7-a23d-4251-9e8b-1a16d4edc2a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6803","nonce":1190079611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6805","nonce":1190079611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6809","nonce":1190079611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6807","nonce":1190079611}]},"public_addr":"172.21.15.12:6803/1190079611","cluster_addr":"172.21.15.12:6805/1190079611","heartbeat_back_addr":"172.21.15.12:6809/1190079611","heartbeat_front_addr":"172.21.15.12:6807/1190079611","state":["exists","up"]},{"osd":1,"uuid":"416e87af-db5d-41ce-84d9-731e3e07e3c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6801","nonce":1983639525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6803","nonce":1983639525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6807","nonce":1983639525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6805","nonce":1983639525}]},"public_addr":"172.21.15.173:6801/1983639525","cluster_addr":"172.21.15.173:6803/1983639525","heartbeat_back_addr":"172.21.15.173:6807/1983639525","heartbeat_front_addr":"172.21.15.173:6805/1983639525","state":["exists","up"]},{"osd":2,"uuid":"7a559c70-4c54-431b-8baf-4136b6e76493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6801","nonce":2253997387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6803","nonce":2253997387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6807","nonce":2253997387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6805","nonce":2253997387}]},"public_addr":"172.21.15.203:6801/2253997387","cluster_addr":"172.21.15.203:6803/2253997387","heartbeat_back_addr":"172.21.15.203:6807/2253997387","heartbeat_front_addr":"172.21.15.203:6805/2253997387","state":["exists","up"]},{"osd":3,"uuid":"c41cd287-a674-486c-8025-328cd36e5ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6811","nonce":1761446905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6813","nonce":1761446905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6817","nonce":1761446905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6815","nonce":1761446905}]},"public_addr":"172.21.15.12:6811/1761446905","cluster_addr":"172.21.15.12:6813/1761446905","heartbeat_back_addr":"172.21.15.12:6817/1761446905","heartbeat_front_addr":"172.21.15.12:6815/1761446905","state":["exists","up"]},{"osd":4,"uuid":"3c48f886-0b42-4a09-b31e-abccc200bb1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6809","nonce":242958272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6811","nonce":242958272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6815","nonce":242958272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6813","nonce":242958272}]},"public_addr":"172.21.15.173:6809/242958272","cluster_addr":"172.21.15.173:6811/242958272","heartbeat_back_addr":"172.21.15.173:6815/242958272","heartbeat_front_addr":"172.21.15.173:6813/242958272","state":["exists","up"]},{"osd":5,"uuid":"4a34f06f-8805-40c0-b1ab-23407b0276d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6809","nonce":4270966092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6811","nonce":4270966092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6815","nonce":4270966092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6813","nonce":4270966092}]},"public_addr":"172.21.15.203:6809/4270966092","cluster_addr":"172.21.15.203:6811/4270966092","heartbeat_back_addr":"172.21.15.203:6815/4270966092","heartbeat_front_addr":"172.21.15.203:6813/4270966092","state":["exists","up"]},{"osd":6,"uuid":"26ddc663-5623-48c9-80d1-f8b968765d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6819","nonce":3606839295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6821","nonce":3606839295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6825","nonce":3606839295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6823","nonce":3606839295}]},"public_addr":"172.21.15.12:6819/3606839295","cluster_addr":"172.21.15.12:6821/3606839295","heartbeat_back_addr":"172.21.15.12:6825/3606839295","heartbeat_front_addr":"172.21.15.12:6823/3606839295","state":["exists","up"]},{"osd":7,"uuid":"17dc9417-71cd-4576-aa08-933212ec55e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6817","nonce":1241504346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6819","nonce":1241504346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6823","nonce":1241504346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6821","nonce":1241504346}]},"public_addr":"172.21.15.173:6817/1241504346","cluster_addr":"172.21.15.173:6819/1241504346","heartbeat_back_addr":"172.21.15.173:6823/1241504346","heartbeat_front_addr":"172.21.15.173:6821/1241504346","state":["exists","up"]},{"osd":8,"uuid":"90a2b63a-471d-43b8-ac18-bb1f16778d18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6817","nonce":1516996192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6819","nonce":1516996192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6823","nonce":1516996192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6821","nonce":1516996192}]},"public_addr":"172.21.15.203:6817/1516996192","cluster_addr":"172.21.15.203:6819/1516996192","heartbeat_back_addr":"172.21.15.203:6823/1516996192","heartbeat_front_addr":"172.21.15.203:6821/1516996192","state":["exists","up"]},{"osd":9,"uuid":"6e1ca504-40b4-49d0-b245-7946181abfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6827","nonce":1175260774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6829","nonce":1175260774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6833","nonce":1175260774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6831","nonce":1175260774}]},"public_addr":"172.21.15.12:6827/1175260774","cluster_addr":"172.21.15.12:6829/1175260774","heartbeat_back_addr":"172.21.15.12:6833/1175260774","heartbeat_front_addr":"172.21.15.12:6831/1175260774","state":["exists","up"]},{"osd":10,"uuid":"09a595ed-5614-4386-8e61-0170af66e01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6825","nonce":3303506808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6827","nonce":3303506808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6831","nonce":3303506808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6829","nonce":3303506808}]},"public_addr":"172.21.15.173:6825/3303506808","cluster_addr":"172.21.15.173:6827/3303506808","heartbeat_back_addr":"172.21.15.173:6831/3303506808","heartbeat_front_addr":"172.21.15.173:6829/3303506808","state":["exists","up"]},{"osd":11,"uuid":"9ef1d595-a977-4ba7-9c6b-d2b3295f20b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6825","nonce":2323604506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6827","nonce":2323604506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6831","nonce":2323604506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6829","nonce":2323604506}]},"public_addr":"172.21.15.203:6825/2323604506","cluster_addr":"172.21.15.203:6827/2323604506","heartbeat_back_addr":"172.21.15.203:6831/2323604506","heartbeat_front_addr":"172.21.15.203:6829/2323604506","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:18.469038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:42.752628+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:07.069493+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:41.962478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:08.363104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:33.341336+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:07.286541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:34.243302+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:59.749699+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:57:33.822913+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:00.737269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:26.599854+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/4182783120":"2024-03-24T22:51:23.113201+0000","172.21.15.12:6800/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1900938637":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/3414861652":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6801/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/2580541477":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6800/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/501058839":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1192554716":"2024-03-24T22:50:48.401223+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T22:59:28.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.342+0000 7f3070c00640 1 -- 172.21.15.12:0/2607124704 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f30500a2800 msgr2=0x7f30500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:28.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.342+0000 7f3070c00640 1 --2- 172.21.15.12:0/2607124704 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f30500a2800 0x7f30500a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f30540095f0 tx=0x7f3054005f90 comp rx=0 tx=0).stop 2024-03-23T22:59:28.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.342+0000 7f3070c00640 1 -- 172.21.15.12:0/2607124704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f306c107ce0 msgr2=0x7f306c19df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:28.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.342+0000 7f3070c00640 1 --2- 172.21.15.12:0/2607124704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f306c107ce0 0x7f306c19df80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f305c00e9c0 tx=0x7f305c00ee80 comp rx=0 tx=0).stop 2024-03-23T22:59:28.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.342+0000 7f3070c00640 1 -- 172.21.15.12:0/2607124704 shutdown_connections 2024-03-23T22:59:28.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.342+0000 7f3070c00640 1 --2- 172.21.15.12:0/2607124704 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f30500a2800 0x7f30500a4cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:28.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.342+0000 7f3070c00640 1 --2- 172.21.15.12:0/2607124704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f306c109890 0x7f306c1a5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:28.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.342+0000 7f3070c00640 1 --2- 172.21.15.12:0/2607124704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f306c108ed0 0x7f306c19e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:28.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.342+0000 7f3070c00640 1 --2- 172.21.15.12:0/2607124704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f306c107ce0 0x7f306c19df80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:28.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.342+0000 7f3070c00640 1 -- 172.21.15.12:0/2607124704 >> 172.21.15.12:0/2607124704 conn(0x7f306c076cb0 msgr2=0x7f306c10c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:28.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.342+0000 7f3070c00640 1 -- 172.21.15.12:0/2607124704 shutdown_connections 2024-03-23T22:59:28.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.342+0000 7f3070c00640 1 -- 172.21.15.12:0/2607124704 wait complete. 2024-03-23T22:59:28.361 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-23T22:59:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.586+0000 7f6beee00640 1 -- 172.21.15.12:0/1518240810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6be81015a0 msgr2=0x7f6be81019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.586+0000 7f6beee00640 1 --2- 172.21.15.12:0/1518240810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6be81015a0 0x7f6be81019c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc009900 tx=0x7f6bdc02bc30 comp rx=0 tx=0).stop 2024-03-23T22:59:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.586+0000 7f6beee00640 1 -- 172.21.15.12:0/1518240810 shutdown_connections 2024-03-23T22:59:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.586+0000 7f6beee00640 1 --2- 172.21.15.12:0/1518240810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6be8103150 0x7f6be81099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.586+0000 7f6beee00640 1 --2- 172.21.15.12:0/1518240810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6be8102790 0x7f6be8102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.586+0000 7f6beee00640 1 --2- 172.21.15.12:0/1518240810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6be81015a0 0x7f6be81019c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.586+0000 7f6beee00640 1 -- 172.21.15.12:0/1518240810 >> 172.21.15.12:0/1518240810 conn(0x7f6be80fcb10 msgr2=0x7f6be80fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.586+0000 7f6beee00640 1 -- 172.21.15.12:0/1518240810 shutdown_connections 2024-03-23T22:59:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.586+0000 7f6beee00640 1 -- 172.21.15.12:0/1518240810 wait complete. 2024-03-23T22:59:28.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.587+0000 7f6beee00640 1 Processor -- start 2024-03-23T22:59:28.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.587+0000 7f6beee00640 1 -- start start 2024-03-23T22:59:28.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.588+0000 7f6beee00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6be81015a0 0x7f6be8197870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.588+0000 7f6beee00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6be8102790 0x7f6be8197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.588+0000 7f6beee00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6be8103150 0x7f6be819ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.588+0000 7f6beee00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be810d260 con 0x7f6be81015a0 2024-03-23T22:59:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.588+0000 7f6beee00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be810d0e0 con 0x7f6be8103150 2024-03-23T22:59:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.588+0000 7f6beee00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be810d3e0 con 0x7f6be8102790 2024-03-23T22:59:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.588+0000 7f6beda00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6be81015a0 0x7f6be8197870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.588+0000 7f6beda00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6be81015a0 0x7f6be8197870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37198/0 (socket says 172.21.15.12:37198) 2024-03-23T22:59:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.588+0000 7f6beda00640 1 -- 172.21.15.12:0/1875849901 learned_addr learned my addr 172.21.15.12:0/1875849901 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.588+0000 7f6bee400640 1 --2- 172.21.15.12:0/1875849901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6be8103150 0x7f6be819ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.588+0000 7f6bed000640 1 --2- 172.21.15.12:0/1875849901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6be8102790 0x7f6be8197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.588+0000 7f6beda00640 1 -- 172.21.15.12:0/1875849901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6be8102790 msgr2=0x7f6be8197db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.588+0000 7f6beda00640 1 --2- 172.21.15.12:0/1875849901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6be8102790 0x7f6be8197db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.588+0000 7f6beda00640 1 -- 172.21.15.12:0/1875849901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6be8103150 msgr2=0x7f6be819ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.588+0000 7f6beda00640 1 --2- 172.21.15.12:0/1875849901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6be8103150 0x7f6be819ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:28.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.588+0000 7f6beda00640 1 -- 172.21.15.12:0/1875849901 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be819f490 con 0x7f6be81015a0 2024-03-23T22:59:28.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.588+0000 7f6bed000640 1 --2- 172.21.15.12:0/1875849901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6be8102790 0x7f6be8197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T22:59:28.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.588+0000 7f6bee400640 1 --2- 172.21.15.12:0/1875849901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6be8103150 0x7f6be819ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T22:59:28.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.589+0000 7f6beda00640 1 --2- 172.21.15.12:0/1875849901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6be81015a0 0x7f6be8197870 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc000c00 tx=0x7f6bdc02c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:28.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.589+0000 7f6be6000640 1 -- 172.21.15.12:0/1875849901 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc043020 con 0x7f6be81015a0 2024-03-23T22:59:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.589+0000 7f6beee00640 1 -- 172.21.15.12:0/1875849901 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be819f720 con 0x7f6be81015a0 2024-03-23T22:59:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.589+0000 7f6be6000640 1 -- 172.21.15.12:0/1875849901 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6bdc0054e0 con 0x7f6be81015a0 2024-03-23T22:59:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.589+0000 7f6be6000640 1 -- 172.21.15.12:0/1875849901 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc003ce0 con 0x7f6be81015a0 2024-03-23T22:59:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.589+0000 7f6beee00640 1 -- 172.21.15.12:0/1875849901 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be819fc90 con 0x7f6be81015a0 2024-03-23T22:59:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.590+0000 7f6beee00640 1 -- 172.21.15.12:0/1875849901 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be81088c0 con 0x7f6be81015a0 2024-03-23T22:59:28.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.592+0000 7f6be6000640 1 -- 172.21.15.12:0/1875849901 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6bdc046020 con 0x7f6be81015a0 2024-03-23T22:59:28.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.594+0000 7f6be6000640 1 --2- 172.21.15.12:0/1875849901 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6bd00a2880 0x7f6bd00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:28.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.594+0000 7f6be6000640 1 -- 172.21.15.12:0/1875849901 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f6bdc0f0e60 con 0x7f6be81015a0 2024-03-23T22:59:28.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.594+0000 7f6bed000640 1 --2- 172.21.15.12:0/1875849901 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6bd00a2880 0x7f6bd00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:28.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.595+0000 7f6bed000640 1 --2- 172.21.15.12:0/1875849901 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6bd00a2880 0x7f6bd00a4d40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6be8198d80 tx=0x7f6bd4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:28.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.596+0000 7f6be6000640 1 -- 172.21.15.12:0/1875849901 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6bdc04d020 con 0x7f6be81015a0 2024-03-23T22:59:28.769 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:28 smithi173 ceph-mon[36116]: pgmap v317: 97 pgs: 9 creating+peering, 84 unknown, 4 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:28.769 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:28 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/641547794' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-23T22:59:28.769 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:28 smithi173 ceph-mon[36116]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-23T22:59:28.770 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:28 smithi173 ceph-mon[36116]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-23T22:59:28.770 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:28 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3419639484' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T22:59:28.770 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:28 smithi173 ceph-mon[36116]: osdmap e86: 12 total, 12 up, 12 in 2024-03-23T22:59:28.770 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:28 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2607124704' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T22:59:28.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:28 smithi012 ceph-mon[31695]: pgmap v317: 97 pgs: 9 creating+peering, 84 unknown, 4 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:28.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:28 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/641547794' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-23T22:59:28.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:28 smithi012 ceph-mon[31695]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-23T22:59:28.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:28 smithi012 ceph-mon[31695]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-23T22:59:28.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:28 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3419639484' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T22:59:28.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:28 smithi012 ceph-mon[31695]: osdmap e86: 12 total, 12 up, 12 in 2024-03-23T22:59:28.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:28 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2607124704' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T22:59:28.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:28.840+0000 7f6beee00640 1 -- 172.21.15.12:0/1875849901 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f6be802d070 con 0x7f6be81015a0 2024-03-23T22:59:28.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:28 smithi203 ceph-mon[36693]: pgmap v317: 97 pgs: 9 creating+peering, 84 unknown, 4 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-23T22:59:28.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:28 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/641547794' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-23T22:59:28.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:28 smithi203 ceph-mon[36693]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-23T22:59:28.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:28 smithi203 ceph-mon[36693]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-23T22:59:28.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:28 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3419639484' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T22:59:28.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:28 smithi203 ceph-mon[36693]: osdmap e86: 12 total, 12 up, 12 in 2024-03-23T22:59:28.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:28 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2607124704' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T22:59:29.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.461+0000 7f6be6000640 1 -- 172.21.15.12:0/1875849901 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6bdc0bdfb0 con 0x7f6be81015a0 2024-03-23T22:59:29.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.465+0000 7f6beee00640 1 -- 172.21.15.12:0/1875849901 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6bd00a2880 msgr2=0x7f6bd00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:29.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.465+0000 7f6beee00640 1 --2- 172.21.15.12:0/1875849901 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6bd00a2880 0x7f6bd00a4d40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6be8198d80 tx=0x7f6bd4005f90 comp rx=0 tx=0).stop 2024-03-23T22:59:29.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.465+0000 7f6beee00640 1 -- 172.21.15.12:0/1875849901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6be81015a0 msgr2=0x7f6be8197870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:29.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.465+0000 7f6beee00640 1 --2- 172.21.15.12:0/1875849901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6be81015a0 0x7f6be8197870 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc000c00 tx=0x7f6bdc02c6c0 comp rx=0 tx=0).stop 2024-03-23T22:59:29.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.465+0000 7f6beee00640 1 -- 172.21.15.12:0/1875849901 shutdown_connections 2024-03-23T22:59:29.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.465+0000 7f6beee00640 1 --2- 172.21.15.12:0/1875849901 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6bd00a2880 0x7f6bd00a4d40 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:29.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.465+0000 7f6beee00640 1 --2- 172.21.15.12:0/1875849901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6be8103150 0x7f6be819ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:29.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.465+0000 7f6beee00640 1 --2- 172.21.15.12:0/1875849901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6be8102790 0x7f6be8197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:29.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.465+0000 7f6beee00640 1 --2- 172.21.15.12:0/1875849901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6be81015a0 0x7f6be8197870 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:29.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.465+0000 7f6beee00640 1 -- 172.21.15.12:0/1875849901 >> 172.21.15.12:0/1875849901 conn(0x7f6be80fcb10 msgr2=0x7f6be8105a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:29.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.465+0000 7f6beee00640 1 -- 172.21.15.12:0/1875849901 shutdown_connections 2024-03-23T22:59:29.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.466+0000 7f6beee00640 1 -- 172.21.15.12:0/1875849901 wait complete. 2024-03-23T22:59:29.488 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-23T22:59:29.489 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-23T22:59:29.707 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:29 smithi012 ceph-mon[31695]: mds.? [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] up:active 2024-03-23T22:59:29.707 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:29 smithi012 ceph-mon[31695]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-23T22:59:29.707 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:29 smithi012 ceph-mon[31695]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-23T22:59:29.708 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:29 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1875849901' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-23T22:59:29.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.721+0000 7f247fe00640 1 -- 172.21.15.12:0/671334707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2480076060 msgr2=0x7f2480076480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:29.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.721+0000 7f247fe00640 1 --2- 172.21.15.12:0/671334707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2480076060 0x7f2480076480 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2470009900 tx=0x7f247002bc30 comp rx=0 tx=0).stop 2024-03-23T22:59:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.721+0000 7f247fe00640 1 -- 172.21.15.12:0/671334707 shutdown_connections 2024-03-23T22:59:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.721+0000 7f247fe00640 1 --2- 172.21.15.12:0/671334707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2480077c40 0x7f24800784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.721+0000 7f247fe00640 1 --2- 172.21.15.12:0/671334707 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24800772a0 0x7f2480077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.721+0000 7f247fe00640 1 --2- 172.21.15.12:0/671334707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2480076060 0x7f2480076480 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.721+0000 7f247fe00640 1 -- 172.21.15.12:0/671334707 >> 172.21.15.12:0/671334707 conn(0x7f24800fed10 msgr2=0x7f2480101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.721+0000 7f247fe00640 1 -- 172.21.15.12:0/671334707 shutdown_connections 2024-03-23T22:59:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.721+0000 7f247fe00640 1 -- 172.21.15.12:0/671334707 wait complete. 2024-03-23T22:59:29.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.722+0000 7f247fe00640 1 Processor -- start 2024-03-23T22:59:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.723+0000 7f247fe00640 1 -- start start 2024-03-23T22:59:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.723+0000 7f247fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2480076060 0x7f248019dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.723+0000 7f247fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24800772a0 0x7f248019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.723+0000 7f247fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2480077c40 0x7f24801a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.723+0000 7f247fe00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2480113870 con 0x7f2480077c40 2024-03-23T22:59:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.723+0000 7f247fe00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24801136f0 con 0x7f2480076060 2024-03-23T22:59:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.723+0000 7f247fe00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24801139f0 con 0x7f24800772a0 2024-03-23T22:59:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.723+0000 7f247ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2480076060 0x7f248019dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.723+0000 7f247ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2480076060 0x7f248019dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:32964/0 (socket says 172.21.15.12:32964) 2024-03-23T22:59:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.724+0000 7f247f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2480077c40 0x7f24801a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.724+0000 7f247ea00640 1 -- 172.21.15.12:0/3925252028 learned_addr learned my addr 172.21.15.12:0/3925252028 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.724+0000 7f247e000640 1 --2- 172.21.15.12:0/3925252028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24800772a0 0x7f248019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.724+0000 7f247ea00640 1 -- 172.21.15.12:0/3925252028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24800772a0 msgr2=0x7f248019e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.724+0000 7f247ea00640 1 --2- 172.21.15.12:0/3925252028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24800772a0 0x7f248019e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.724+0000 7f247ea00640 1 -- 172.21.15.12:0/3925252028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2480077c40 msgr2=0x7f24801a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.724+0000 7f247ea00640 1 --2- 172.21.15.12:0/3925252028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2480077c40 0x7f24801a5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.724+0000 7f247ea00640 1 -- 172.21.15.12:0/3925252028 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24801a5ac0 con 0x7f2480076060 2024-03-23T22:59:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.724+0000 7f247e000640 1 --2- 172.21.15.12:0/3925252028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24800772a0 0x7f248019e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T22:59:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.724+0000 7f247ea00640 1 --2- 172.21.15.12:0/3925252028 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2480076060 0x7f248019dea0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2470005ab0 tx=0x7f247002c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.724+0000 7f2477400640 1 -- 172.21.15.12:0/3925252028 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2470043020 con 0x7f2480076060 2024-03-23T22:59:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.724+0000 7f2477400640 1 -- 172.21.15.12:0/3925252028 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f24700054e0 con 0x7f2480076060 2024-03-23T22:59:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.724+0000 7f247fe00640 1 -- 172.21.15.12:0/3925252028 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24801a5d50 con 0x7f2480076060 2024-03-23T22:59:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.724+0000 7f2477400640 1 -- 172.21.15.12:0/3925252028 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2470003d70 con 0x7f2480076060 2024-03-23T22:59:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.725+0000 7f247fe00640 1 -- 172.21.15.12:0/3925252028 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24801a6210 con 0x7f2480076060 2024-03-23T22:59:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.726+0000 7f247fe00640 1 -- 172.21.15.12:0/3925252028 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2480110110 con 0x7f2480076060 2024-03-23T22:59:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.727+0000 7f2477400640 1 -- 172.21.15.12:0/3925252028 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2470046020 con 0x7f2480076060 2024-03-23T22:59:29.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.729+0000 7f2477400640 1 --2- 172.21.15.12:0/3925252028 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f24640a2880 0x7f24640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:29.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.729+0000 7f2477400640 1 -- 172.21.15.12:0/3925252028 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f24700f0b10 con 0x7f2480076060 2024-03-23T22:59:29.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.729+0000 7f247e000640 1 --2- 172.21.15.12:0/3925252028 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f24640a2880 0x7f24640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:29.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.730+0000 7f247e000640 1 --2- 172.21.15.12:0/3925252028 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f24640a2880 0x7f24640a4d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f248019f3b0 tx=0x7f246c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:29.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.733+0000 7f2477400640 1 -- 172.21.15.12:0/3925252028 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f247004d020 con 0x7f2480076060 2024-03-23T22:59:29.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:29 smithi173 ceph-mon[36116]: mds.? [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] up:active 2024-03-23T22:59:29.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:29 smithi173 ceph-mon[36116]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-23T22:59:29.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:29 smithi173 ceph-mon[36116]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-23T22:59:29.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:29 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1875849901' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-23T22:59:29.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:29 smithi203 ceph-mon[36693]: mds.? [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] up:active 2024-03-23T22:59:29.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:29 smithi203 ceph-mon[36693]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-23T22:59:29.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:29 smithi203 ceph-mon[36693]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-23T22:59:29.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:29 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1875849901' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-23T22:59:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:29.977+0000 7f247fe00640 1 -- 172.21.15.12:0/3925252028 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f248002d070 con 0x7f2480076060 2024-03-23T22:59:30.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.466+0000 7f2477400640 1 -- 172.21.15.12:0/3925252028 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) v1 ==== 105+0+0 (secure 0 0 0) 0x7f24700bdc60 con 0x7f2480076060 2024-03-23T22:59:30.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.470+0000 7f247fe00640 1 -- 172.21.15.12:0/3925252028 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f24640a2880 msgr2=0x7f24640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:30.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.470+0000 7f247fe00640 1 --2- 172.21.15.12:0/3925252028 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f24640a2880 0x7f24640a4d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f248019f3b0 tx=0x7f246c005f90 comp rx=0 tx=0).stop 2024-03-23T22:59:30.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.470+0000 7f247fe00640 1 -- 172.21.15.12:0/3925252028 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2480076060 msgr2=0x7f248019dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:30.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.470+0000 7f247fe00640 1 --2- 172.21.15.12:0/3925252028 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2480076060 0x7f248019dea0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2470005ab0 tx=0x7f247002c6c0 comp rx=0 tx=0).stop 2024-03-23T22:59:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.470+0000 7f247fe00640 1 -- 172.21.15.12:0/3925252028 shutdown_connections 2024-03-23T22:59:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.470+0000 7f247fe00640 1 --2- 172.21.15.12:0/3925252028 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f24640a2880 0x7f24640a4d40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.470+0000 7f247fe00640 1 --2- 172.21.15.12:0/3925252028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2480077c40 0x7f24801a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.470+0000 7f247fe00640 1 --2- 172.21.15.12:0/3925252028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24800772a0 0x7f248019e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.470+0000 7f247fe00640 1 --2- 172.21.15.12:0/3925252028 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2480076060 0x7f248019dea0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.470+0000 7f247fe00640 1 -- 172.21.15.12:0/3925252028 >> 172.21.15.12:0/3925252028 conn(0x7f24800fed10 msgr2=0x7f2480100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.470+0000 7f247fe00640 1 -- 172.21.15.12:0/3925252028 shutdown_connections 2024-03-23T22:59:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.471+0000 7f247fe00640 1 -- 172.21.15.12:0/3925252028 wait complete. 2024-03-23T22:59:30.498 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-23T22:59:30.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.710+0000 7f64ebe00640 1 -- 172.21.15.12:0/2423084339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64ec102770 msgr2=0x7f64ec104be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:30.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.710+0000 7f64ebe00640 1 --2- 172.21.15.12:0/2423084339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64ec102770 0x7f64ec104be0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f64d8009900 tx=0x7f64d802bc30 comp rx=0 tx=0).stop 2024-03-23T22:59:30.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.710+0000 7f64ebe00640 1 -- 172.21.15.12:0/2423084339 shutdown_connections 2024-03-23T22:59:30.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.710+0000 7f64ebe00640 1 --2- 172.21.15.12:0/2423084339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64ec105120 0x7f64ec1055a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:30.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.710+0000 7f64ebe00640 1 --2- 172.21.15.12:0/2423084339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64ec102770 0x7f64ec104be0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:30.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.710+0000 7f64ebe00640 1 --2- 172.21.15.12:0/2423084339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64ec0ffe00 0x7f64ec102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:30.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.710+0000 7f64ebe00640 1 -- 172.21.15.12:0/2423084339 >> 172.21.15.12:0/2423084339 conn(0x7f64ec0fba20 msgr2=0x7f64ec0fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:30.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.710+0000 7f64ebe00640 1 -- 172.21.15.12:0/2423084339 shutdown_connections 2024-03-23T22:59:30.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.710+0000 7f64ebe00640 1 -- 172.21.15.12:0/2423084339 wait complete. 2024-03-23T22:59:30.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.711+0000 7f64ebe00640 1 Processor -- start 2024-03-23T22:59:30.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.711+0000 7f64ebe00640 1 -- start start 2024-03-23T22:59:30.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.712+0000 7f64ebe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64ec0ffe00 0x7f64ec19de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:30.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.712+0000 7f64ebe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64ec102770 0x7f64ec19e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:30.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.712+0000 7f64ebe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64ec105120 0x7f64ec1a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:30.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.712+0000 7f64ebe00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ec113830 con 0x7f64ec105120 2024-03-23T22:59:30.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.712+0000 7f64ebe00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ec1136b0 con 0x7f64ec0ffe00 2024-03-23T22:59:30.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.712+0000 7f64ebe00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ec1139b0 con 0x7f64ec102770 2024-03-23T22:59:30.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.712+0000 7f64eaa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64ec0ffe00 0x7f64ec19de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:30.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.712+0000 7f64eaa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64ec0ffe00 0x7f64ec19de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:32982/0 (socket says 172.21.15.12:32982) 2024-03-23T22:59:30.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.712+0000 7f64eaa00640 1 -- 172.21.15.12:0/2881395393 learned_addr learned my addr 172.21.15.12:0/2881395393 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:30.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.712+0000 7f64eb400640 1 --2- 172.21.15.12:0/2881395393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64ec105120 0x7f64ec1a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:30.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.712+0000 7f64ea000640 1 --2- 172.21.15.12:0/2881395393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64ec102770 0x7f64ec19e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:30.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.712+0000 7f64eaa00640 1 -- 172.21.15.12:0/2881395393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64ec102770 msgr2=0x7f64ec19e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:30.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.712+0000 7f64eaa00640 1 --2- 172.21.15.12:0/2881395393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64ec102770 0x7f64ec19e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:30.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.712+0000 7f64eaa00640 1 -- 172.21.15.12:0/2881395393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64ec105120 msgr2=0x7f64ec1a5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:30.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.712+0000 7f64eaa00640 1 --2- 172.21.15.12:0/2881395393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64ec105120 0x7f64ec1a5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:30.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.712+0000 7f64eaa00640 1 -- 172.21.15.12:0/2881395393 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64ec1a5a90 con 0x7f64ec0ffe00 2024-03-23T22:59:30.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.713+0000 7f64eaa00640 1 --2- 172.21.15.12:0/2881395393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64ec0ffe00 0x7f64ec19de70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f64dc00e9f0 tx=0x7f64dc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:30.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.713+0000 7f64e3400640 1 -- 172.21.15.12:0/2881395393 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64dc00cd80 con 0x7f64ec0ffe00 2024-03-23T22:59:30.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.713+0000 7f64e3400640 1 -- 172.21.15.12:0/2881395393 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64dc010910 con 0x7f64ec0ffe00 2024-03-23T22:59:30.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.713+0000 7f64e3400640 1 -- 172.21.15.12:0/2881395393 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64dc018990 con 0x7f64ec0ffe00 2024-03-23T22:59:30.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.713+0000 7f64ebe00640 1 -- 172.21.15.12:0/2881395393 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64ec1a5d80 con 0x7f64ec0ffe00 2024-03-23T22:59:30.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.713+0000 7f64ebe00640 1 -- 172.21.15.12:0/2881395393 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64ec1a6240 con 0x7f64ec0ffe00 2024-03-23T22:59:30.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.714+0000 7f64ebe00640 1 -- 172.21.15.12:0/2881395393 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64ec0fd510 con 0x7f64ec0ffe00 2024-03-23T22:59:30.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.716+0000 7f64e3400640 1 -- 172.21.15.12:0/2881395393 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f64dc010ac0 con 0x7f64ec0ffe00 2024-03-23T22:59:30.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.717+0000 7f64e3400640 1 --2- 172.21.15.12:0/2881395393 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f64d00a2880 0x7f64d00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:30.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.718+0000 7f64e3400640 1 -- 172.21.15.12:0/2881395393 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f64dc0cec60 con 0x7f64ec0ffe00 2024-03-23T22:59:30.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.718+0000 7f64ea000640 1 --2- 172.21.15.12:0/2881395393 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f64d00a2880 0x7f64d00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:30.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.719+0000 7f64ea000640 1 --2- 172.21.15.12:0/2881395393 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f64d00a2880 0x7f64d00a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f64d802c240 tx=0x7f64d802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:30.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.721+0000 7f64e3400640 1 -- 172.21.15.12:0/2881395393 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f64dc09b760 con 0x7f64ec0ffe00 2024-03-23T22:59:30.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:30 smithi012 ceph-mon[31695]: pgmap v319: 97 pgs: 9 creating+peering, 57 unknown, 31 active+clean; 578 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 1 op/s 2024-03-23T22:59:30.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:30 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1875849901' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-23T22:59:30.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:30 smithi012 ceph-mon[31695]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-23T22:59:30.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:30 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3925252028' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-23T22:59:30.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:30 smithi012 ceph-mon[31695]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-23T22:59:30.808 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:30 smithi203 ceph-mon[36693]: pgmap v319: 97 pgs: 9 creating+peering, 57 unknown, 31 active+clean; 578 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 1 op/s 2024-03-23T22:59:30.808 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:30 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1875849901' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-23T22:59:30.809 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:30 smithi203 ceph-mon[36693]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-23T22:59:30.809 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:30 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3925252028' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-23T22:59:30.809 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:30 smithi203 ceph-mon[36693]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-23T22:59:30.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:30 smithi173 ceph-mon[36116]: pgmap v319: 97 pgs: 9 creating+peering, 57 unknown, 31 active+clean; 578 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 1 op/s 2024-03-23T22:59:30.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:30 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1875849901' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-23T22:59:30.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:30 smithi173 ceph-mon[36116]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-23T22:59:30.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:30 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3925252028' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-23T22:59:30.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:30 smithi173 ceph-mon[36116]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-23T22:59:30.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:30.968+0000 7f64ebe00640 1 -- 172.21.15.12:0/2881395393 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f64ec0622d0 con 0x7f64ec0ffe00 2024-03-23T22:59:31.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.479+0000 7f64e3400640 1 -- 172.21.15.12:0/2881395393 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) v1 ==== 121+0+0 (secure 0 0 0) 0x7f64dc0bb020 con 0x7f64ec0ffe00 2024-03-23T22:59:31.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.482+0000 7f64ebe00640 1 -- 172.21.15.12:0/2881395393 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f64d00a2880 msgr2=0x7f64d00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:31.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.482+0000 7f64ebe00640 1 --2- 172.21.15.12:0/2881395393 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f64d00a2880 0x7f64d00a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f64d802c240 tx=0x7f64d802df90 comp rx=0 tx=0).stop 2024-03-23T22:59:31.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.482+0000 7f64ebe00640 1 -- 172.21.15.12:0/2881395393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64ec0ffe00 msgr2=0x7f64ec19de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:31.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.482+0000 7f64ebe00640 1 --2- 172.21.15.12:0/2881395393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64ec0ffe00 0x7f64ec19de70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f64dc00e9f0 tx=0x7f64dc00eeb0 comp rx=0 tx=0).stop 2024-03-23T22:59:31.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.482+0000 7f64ebe00640 1 -- 172.21.15.12:0/2881395393 shutdown_connections 2024-03-23T22:59:31.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.482+0000 7f64ebe00640 1 --2- 172.21.15.12:0/2881395393 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f64d00a2880 0x7f64d00a4d40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:31.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.482+0000 7f64ebe00640 1 --2- 172.21.15.12:0/2881395393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64ec105120 0x7f64ec1a5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:31.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.482+0000 7f64ebe00640 1 --2- 172.21.15.12:0/2881395393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64ec102770 0x7f64ec19e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:31.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.482+0000 7f64ebe00640 1 --2- 172.21.15.12:0/2881395393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64ec0ffe00 0x7f64ec19de70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:31.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.482+0000 7f64ebe00640 1 -- 172.21.15.12:0/2881395393 >> 172.21.15.12:0/2881395393 conn(0x7f64ec0fba20 msgr2=0x7f64ec100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:31.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.483+0000 7f64ebe00640 1 -- 172.21.15.12:0/2881395393 shutdown_connections 2024-03-23T22:59:31.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.483+0000 7f64ebe00640 1 -- 172.21.15.12:0/2881395393 wait complete. 2024-03-23T22:59:31.505 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-23T22:59:31.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.690+0000 7fd79ce00640 1 -- 172.21.15.12:0/1320994951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd798105470 msgr2=0x7fd79810bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:31.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.690+0000 7fd79ce00640 1 --2- 172.21.15.12:0/1320994951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd798105470 0x7fd79810bd00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd790009ca0 tx=0x7fd79002c0f0 comp rx=0 tx=0).stop 2024-03-23T22:59:31.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.690+0000 7fd79ce00640 1 -- 172.21.15.12:0/1320994951 shutdown_connections 2024-03-23T22:59:31.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.690+0000 7fd79ce00640 1 --2- 172.21.15.12:0/1320994951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd798105470 0x7fd79810bd00 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:31.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.690+0000 7fd79ce00640 1 --2- 172.21.15.12:0/1320994951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd798104ab0 0x7fd798104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:31.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.690+0000 7fd79ce00640 1 --2- 172.21.15.12:0/1320994951 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7981038c0 0x7fd798103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:31.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.690+0000 7fd79ce00640 1 -- 172.21.15.12:0/1320994951 >> 172.21.15.12:0/1320994951 conn(0x7fd7980fee50 msgr2=0x7fd798101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:31.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.690+0000 7fd79ce00640 1 -- 172.21.15.12:0/1320994951 shutdown_connections 2024-03-23T22:59:31.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.690+0000 7fd79ce00640 1 -- 172.21.15.12:0/1320994951 wait complete. 2024-03-23T22:59:31.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.691+0000 7fd79ce00640 1 Processor -- start 2024-03-23T22:59:31.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.692+0000 7fd79ce00640 1 -- start start 2024-03-23T22:59:31.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.692+0000 7fd79ce00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7981038c0 0x7fd798199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:31.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.692+0000 7fd79ce00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd798104ab0 0x7fd79819a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:31.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.692+0000 7fd79ce00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd798105470 0x7fd7981a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:31.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.692+0000 7fd79ce00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79810f520 con 0x7fd798105470 2024-03-23T22:59:31.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.692+0000 7fd79ce00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79810f3a0 con 0x7fd798104ab0 2024-03-23T22:59:31.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.692+0000 7fd79ce00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79810f6a0 con 0x7fd7981038c0 2024-03-23T22:59:31.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.692+0000 7fd795400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd798104ab0 0x7fd79819a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:31.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.692+0000 7fd795400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd798104ab0 0x7fd79819a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:33002/0 (socket says 172.21.15.12:33002) 2024-03-23T22:59:31.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.692+0000 7fd795400640 1 -- 172.21.15.12:0/503128153 learned_addr learned my addr 172.21.15.12:0/503128153 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:31.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.692+0000 7fd796800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd798105470 0x7fd7981a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:31.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.692+0000 7fd795e00640 1 --2- 172.21.15.12:0/503128153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7981038c0 0x7fd798199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:31.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.693+0000 7fd795400640 1 -- 172.21.15.12:0/503128153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7981038c0 msgr2=0x7fd798199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:31.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.693+0000 7fd795400640 1 --2- 172.21.15.12:0/503128153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7981038c0 0x7fd798199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:31.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.693+0000 7fd795400640 1 -- 172.21.15.12:0/503128153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd798105470 msgr2=0x7fd7981a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:31.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.693+0000 7fd795400640 1 --2- 172.21.15.12:0/503128153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd798105470 0x7fd7981a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:31.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.693+0000 7fd795400640 1 -- 172.21.15.12:0/503128153 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7981a17e0 con 0x7fd798104ab0 2024-03-23T22:59:31.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.693+0000 7fd795400640 1 --2- 172.21.15.12:0/503128153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd798104ab0 0x7fd79819a100 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd78400e9f0 tx=0x7fd78400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:31.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.693+0000 7fd78ea00640 1 -- 172.21.15.12:0/503128153 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78400cd80 con 0x7fd798104ab0 2024-03-23T22:59:31.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.693+0000 7fd79ce00640 1 -- 172.21.15.12:0/503128153 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7981a1ad0 con 0x7fd798104ab0 2024-03-23T22:59:31.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.693+0000 7fd78ea00640 1 -- 172.21.15.12:0/503128153 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd784010910 con 0x7fd798104ab0 2024-03-23T22:59:31.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.693+0000 7fd78ea00640 1 -- 172.21.15.12:0/503128153 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7840188f0 con 0x7fd798104ab0 2024-03-23T22:59:31.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.693+0000 7fd79ce00640 1 -- 172.21.15.12:0/503128153 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7981a2070 con 0x7fd798104ab0 2024-03-23T22:59:31.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.695+0000 7fd79ce00640 1 -- 172.21.15.12:0/503128153 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd79810ab80 con 0x7fd798104ab0 2024-03-23T22:59:31.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.697+0000 7fd78ea00640 1 -- 172.21.15.12:0/503128153 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd784010ac0 con 0x7fd798104ab0 2024-03-23T22:59:31.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.698+0000 7fd78ea00640 1 --2- 172.21.15.12:0/503128153 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd77c0a2880 0x7fd77c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:31.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.698+0000 7fd78ea00640 1 -- 172.21.15.12:0/503128153 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd7840ceca0 con 0x7fd798104ab0 2024-03-23T22:59:31.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.699+0000 7fd795e00640 1 --2- 172.21.15.12:0/503128153 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd77c0a2880 0x7fd77c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:31.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.699+0000 7fd795e00640 1 --2- 172.21.15.12:0/503128153 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd77c0a2880 0x7fd77c0a4d40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd798104910 tx=0x7fd788005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:31.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.703+0000 7fd78ea00640 1 -- 172.21.15.12:0/503128153 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd78409b7a0 con 0x7fd798104ab0 2024-03-23T22:59:31.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:31 smithi012 ceph-mon[31695]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-23T22:59:31.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:31 smithi012 ceph-mon[31695]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-23T22:59:31.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:31 smithi012 ceph-mon[31695]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-23T22:59:31.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:31 smithi012 ceph-mon[31695]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-23T22:59:31.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:31 smithi012 ceph-mon[31695]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-03-23T22:59:31.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:31 smithi012 ceph-mon[31695]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-23T22:59:31.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:31 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2881395393' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-23T22:59:31.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:31 smithi012 ceph-mon[31695]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-23T22:59:31.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:31 smithi173 ceph-mon[36116]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-23T22:59:31.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:31 smithi173 ceph-mon[36116]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-23T22:59:31.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:31 smithi173 ceph-mon[36116]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-23T22:59:31.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:31 smithi173 ceph-mon[36116]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-23T22:59:31.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:31 smithi173 ceph-mon[36116]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-03-23T22:59:31.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:31 smithi173 ceph-mon[36116]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-23T22:59:31.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:31 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2881395393' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-23T22:59:31.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:31 smithi173 ceph-mon[36116]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-23T22:59:31.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:31 smithi203 ceph-mon[36693]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-23T22:59:31.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:31 smithi203 ceph-mon[36693]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-23T22:59:31.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:31 smithi203 ceph-mon[36693]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-23T22:59:31.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:31 smithi203 ceph-mon[36693]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-23T22:59:31.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:31 smithi203 ceph-mon[36693]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-03-23T22:59:31.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:31 smithi203 ceph-mon[36693]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-23T22:59:31.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:31 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2881395393' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-23T22:59:31.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:31 smithi203 ceph-mon[36693]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-23T22:59:31.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:31.939+0000 7fd79ce00640 1 -- 172.21.15.12:0/503128153 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fd79802d070 con 0x7fd798104ab0 2024-03-23T22:59:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.198+0000 7fd78ea00640 1 -- 172.21.15.12:0/503128153 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) v1 ==== 115+0+0 (secure 0 0 0) 0x7fd78409f840 con 0x7fd798104ab0 2024-03-23T22:59:32.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.201+0000 7fd79ce00640 1 -- 172.21.15.12:0/503128153 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd77c0a2880 msgr2=0x7fd77c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:32.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.201+0000 7fd79ce00640 1 --2- 172.21.15.12:0/503128153 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd77c0a2880 0x7fd77c0a4d40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd798104910 tx=0x7fd788005f90 comp rx=0 tx=0).stop 2024-03-23T22:59:32.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.201+0000 7fd79ce00640 1 -- 172.21.15.12:0/503128153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd798104ab0 msgr2=0x7fd79819a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:32.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.201+0000 7fd79ce00640 1 --2- 172.21.15.12:0/503128153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd798104ab0 0x7fd79819a100 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd78400e9f0 tx=0x7fd78400eeb0 comp rx=0 tx=0).stop 2024-03-23T22:59:32.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.201+0000 7fd79ce00640 1 -- 172.21.15.12:0/503128153 shutdown_connections 2024-03-23T22:59:32.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.201+0000 7fd79ce00640 1 --2- 172.21.15.12:0/503128153 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd77c0a2880 0x7fd77c0a4d40 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:32.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.201+0000 7fd79ce00640 1 --2- 172.21.15.12:0/503128153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd798105470 0x7fd7981a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:32.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.201+0000 7fd79ce00640 1 --2- 172.21.15.12:0/503128153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd798104ab0 0x7fd79819a100 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:32.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.201+0000 7fd79ce00640 1 --2- 172.21.15.12:0/503128153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7981038c0 0x7fd798199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:32.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.201+0000 7fd79ce00640 1 -- 172.21.15.12:0/503128153 >> 172.21.15.12:0/503128153 conn(0x7fd7980fee50 msgr2=0x7fd798107d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:32.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.201+0000 7fd79ce00640 1 -- 172.21.15.12:0/503128153 shutdown_connections 2024-03-23T22:59:32.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.201+0000 7fd79ce00640 1 -- 172.21.15.12:0/503128153 wait complete. 2024-03-23T22:59:32.221 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-03-23T22:59:32.221 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-23T22:59:32.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.443+0000 7f09f9a00640 1 -- 172.21.15.12:0/3651594031 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09f40740a0 msgr2=0x7f09f4074500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:32.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.443+0000 7f09f9a00640 1 --2- 172.21.15.12:0/3651594031 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09f40740a0 0x7f09f4074500 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f09dc009900 tx=0x7f09dc02bcd0 comp rx=0 tx=0).stop 2024-03-23T22:59:32.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.443+0000 7f09f9a00640 1 -- 172.21.15.12:0/3651594031 shutdown_connections 2024-03-23T22:59:32.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.443+0000 7f09f9a00640 1 --2- 172.21.15.12:0/3651594031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09f4109370 0x7f09f410b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:32.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.443+0000 7f09f9a00640 1 --2- 172.21.15.12:0/3651594031 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09f40740a0 0x7f09f4074500 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:32.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.443+0000 7f09f9a00640 1 --2- 172.21.15.12:0/3651594031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09f4075680 0x7f09f4073ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:32.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.443+0000 7f09f9a00640 1 -- 172.21.15.12:0/3651594031 >> 172.21.15.12:0/3651594031 conn(0x7f09f40fca50 msgr2=0x7f09f40fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:32.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.443+0000 7f09f9a00640 1 -- 172.21.15.12:0/3651594031 shutdown_connections 2024-03-23T22:59:32.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.443+0000 7f09f9a00640 1 -- 172.21.15.12:0/3651594031 wait complete. 2024-03-23T22:59:32.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.444+0000 7f09f9a00640 1 Processor -- start 2024-03-23T22:59:32.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.444+0000 7f09f9a00640 1 -- start start 2024-03-23T22:59:32.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.445+0000 7f09f9a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09f40740a0 0x7f09f41999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:32.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.445+0000 7f09f9a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09f4075680 0x7f09f4199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:32.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.445+0000 7f09f9a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09f4109370 0x7f09f41a0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:32.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.445+0000 7f09f9a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f410f390 con 0x7f09f4075680 2024-03-23T22:59:32.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.445+0000 7f09f9a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f410f210 con 0x7f09f40740a0 2024-03-23T22:59:32.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.445+0000 7f09f9a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f410f510 con 0x7f09f4109370 2024-03-23T22:59:32.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.445+0000 7f09f9000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09f4109370 0x7f09f41a0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:32.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.445+0000 7f09f3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09f40740a0 0x7f09f41999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:32.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.445+0000 7f09f9000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09f4109370 0x7f09f41a0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45426/0 (socket says 172.21.15.12:45426) 2024-03-23T22:59:32.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.445+0000 7f09f9000640 1 -- 172.21.15.12:0/1121762176 learned_addr learned my addr 172.21.15.12:0/1121762176 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:32.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.445+0000 7f09f3400640 1 --2- 172.21.15.12:0/1121762176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09f4075680 0x7f09f4199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:32.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.445+0000 7f09f3e00640 1 -- 172.21.15.12:0/1121762176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09f4109370 msgr2=0x7f09f41a0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:32.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.445+0000 7f09f3e00640 1 --2- 172.21.15.12:0/1121762176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09f4109370 0x7f09f41a0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:32.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.445+0000 7f09f3e00640 1 -- 172.21.15.12:0/1121762176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09f4075680 msgr2=0x7f09f4199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:32.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.445+0000 7f09f3e00640 1 --2- 172.21.15.12:0/1121762176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09f4075680 0x7f09f4199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:32.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.445+0000 7f09f3e00640 1 -- 172.21.15.12:0/1121762176 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f41a15c0 con 0x7f09f40740a0 2024-03-23T22:59:32.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.445+0000 7f09f3400640 1 --2- 172.21.15.12:0/1121762176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09f4075680 0x7f09f4199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T22:59:32.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.446+0000 7f09f3e00640 1 --2- 172.21.15.12:0/1121762176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09f40740a0 0x7f09f41999a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f09e400eb00 tx=0x7f09e400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:32.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.446+0000 7f09f0c00640 1 -- 172.21.15.12:0/1121762176 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e4010430 con 0x7f09f40740a0 2024-03-23T22:59:32.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.446+0000 7f09f0c00640 1 -- 172.21.15.12:0/1121762176 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09e4010ab0 con 0x7f09f40740a0 2024-03-23T22:59:32.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.446+0000 7f09f9a00640 1 -- 172.21.15.12:0/1121762176 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f41a18b0 con 0x7f09f40740a0 2024-03-23T22:59:32.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.446+0000 7f09f0c00640 1 -- 172.21.15.12:0/1121762176 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e4017bb0 con 0x7f09f40740a0 2024-03-23T22:59:32.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.446+0000 7f09f9a00640 1 -- 172.21.15.12:0/1121762176 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f41a1dd0 con 0x7f09f40740a0 2024-03-23T22:59:32.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.447+0000 7f09f9a00640 1 -- 172.21.15.12:0/1121762176 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09f4108130 con 0x7f09f40740a0 2024-03-23T22:59:32.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.449+0000 7f09f0c00640 1 -- 172.21.15.12:0/1121762176 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f09e4010c60 con 0x7f09f40740a0 2024-03-23T22:59:32.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.450+0000 7f09f0c00640 1 --2- 172.21.15.12:0/1121762176 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f09d80a2800 0x7f09d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:32.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.450+0000 7f09f0c00640 1 -- 172.21.15.12:0/1121762176 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f09e40cf400 con 0x7f09f40740a0 2024-03-23T22:59:32.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.450+0000 7f09f3400640 1 --2- 172.21.15.12:0/1121762176 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f09d80a2800 0x7f09d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:32.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.451+0000 7f09f3400640 1 --2- 172.21.15.12:0/1121762176 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f09d80a2800 0x7f09d80a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f09dc02c2e0 tx=0x7f09dc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:32.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.454+0000 7f09f0c00640 1 -- 172.21.15.12:0/1121762176 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f09e409c550 con 0x7f09f40740a0 2024-03-23T22:59:32.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.684+0000 7f09f9a00640 1 -- 172.21.15.12:0/1121762176 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f09f4061ae0 con 0x7f09d80a2800 2024-03-23T22:59:32.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.708+0000 7f09f0c00640 1 -- 172.21.15.12:0/1121762176 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f09f4061ae0 con 0x7f09d80a2800 2024-03-23T22:59:32.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.711+0000 7f09f9a00640 1 -- 172.21.15.12:0/1121762176 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f09d80a2800 msgr2=0x7f09d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:32.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.711+0000 7f09f9a00640 1 --2- 172.21.15.12:0/1121762176 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f09d80a2800 0x7f09d80a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f09dc02c2e0 tx=0x7f09dc005fd0 comp rx=0 tx=0).stop 2024-03-23T22:59:32.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.711+0000 7f09f9a00640 1 -- 172.21.15.12:0/1121762176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09f40740a0 msgr2=0x7f09f41999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:32.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.711+0000 7f09f9a00640 1 --2- 172.21.15.12:0/1121762176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09f40740a0 0x7f09f41999a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f09e400eb00 tx=0x7f09e400c2d0 comp rx=0 tx=0).stop 2024-03-23T22:59:32.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.711+0000 7f09f9a00640 1 -- 172.21.15.12:0/1121762176 shutdown_connections 2024-03-23T22:59:32.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.711+0000 7f09f9a00640 1 --2- 172.21.15.12:0/1121762176 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f09d80a2800 0x7f09d80a4cc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:32.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.711+0000 7f09f9a00640 1 --2- 172.21.15.12:0/1121762176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09f4109370 0x7f09f41a0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:32.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.711+0000 7f09f9a00640 1 --2- 172.21.15.12:0/1121762176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09f4075680 0x7f09f4199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:32.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.711+0000 7f09f9a00640 1 --2- 172.21.15.12:0/1121762176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09f40740a0 0x7f09f41999a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:32.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.711+0000 7f09f9a00640 1 -- 172.21.15.12:0/1121762176 >> 172.21.15.12:0/1121762176 conn(0x7f09f40fca50 msgr2=0x7f09f4105300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:32.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.711+0000 7f09f9a00640 1 -- 172.21.15.12:0/1121762176 shutdown_connections 2024-03-23T22:59:32.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.711+0000 7f09f9a00640 1 -- 172.21.15.12:0/1121762176 wait complete. 2024-03-23T22:59:32.727 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-23T22:59:32.796 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:59:32 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:59:32.692+0000 7f6fe2600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-23T22:59:32.796 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:59:32 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:59:32.692+0000 7f6fe2600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-23T22:59:32.796 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:59:32 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:59:32.692+0000 7f6fe2600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-23T22:59:32.796 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:59:32 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:59:32.692+0000 7f6fe2600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-23T22:59:32.796 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 22:59:32 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T22:59:32.692+0000 7f6fe2600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-23T22:59:32.796 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:32 smithi012 ceph-mon[31695]: pgmap v320: 97 pgs: 2 creating+peering, 95 active+clean; 578 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s wr, 4 op/s 2024-03-23T22:59:32.797 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:32 smithi012 ceph-mon[31695]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-23T22:59:32.797 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:32 smithi012 ceph-mon[31695]: mds.? [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] up:active 2024-03-23T22:59:32.797 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:32 smithi012 ceph-mon[31695]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-23T22:59:32.797 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:32 smithi012 ceph-mon[31695]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-23T22:59:32.797 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:32 smithi012 ceph-mon[31695]: Cluster is now healthy 2024-03-23T22:59:32.797 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:32 smithi012 ceph-mon[31695]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-03-23T22:59:32.797 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:32 smithi012 ceph-mon[31695]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-03-23T22:59:32.797 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:32 smithi012 ceph-mon[31695]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-23T22:59:32.797 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:32 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/503128153' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-23T22:59:32.797 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:32 smithi012 ceph-mon[31695]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-23T22:59:32.797 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:32 smithi012 ceph-mon[31695]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-23T22:59:32.797 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:32 smithi012 ceph-mon[31695]: mds.? [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] up:active 2024-03-23T22:59:32.797 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:32 smithi012 ceph-mon[31695]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-23T22:59:32.868 INFO:journalctl@ceph.mds.b.smithi173.stdout:Mar 23 22:59:32 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-b[66540]: 2024-03-23T22:59:32.704+0000 7f9e7b0ff700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T22:59:32.868 INFO:journalctl@ceph.mds.b.smithi173.stdout:Mar 23 22:59:32 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-b[66540]: 2024-03-23T22:59:32.704+0000 7f9e7b0ff700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-23T22:59:32.868 INFO:journalctl@ceph.mds.b.smithi173.stdout:Mar 23 22:59:32 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-b[66540]: 2024-03-23T22:59:32.704+0000 7f9e7b0ff700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-23T22:59:32.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:32 smithi173 ceph-mon[36116]: pgmap v320: 97 pgs: 2 creating+peering, 95 active+clean; 578 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s wr, 4 op/s 2024-03-23T22:59:32.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:32 smithi173 ceph-mon[36116]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-23T22:59:32.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:32 smithi173 ceph-mon[36116]: mds.? [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] up:active 2024-03-23T22:59:32.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:32 smithi173 ceph-mon[36116]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-23T22:59:32.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:32 smithi173 ceph-mon[36116]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-23T22:59:32.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:32 smithi173 ceph-mon[36116]: Cluster is now healthy 2024-03-23T22:59:32.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:32 smithi173 ceph-mon[36116]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-03-23T22:59:32.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:32 smithi173 ceph-mon[36116]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-03-23T22:59:32.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:32 smithi173 ceph-mon[36116]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-23T22:59:32.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:32 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/503128153' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-23T22:59:32.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:32 smithi173 ceph-mon[36116]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-23T22:59:32.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:32 smithi173 ceph-mon[36116]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-23T22:59:32.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:32 smithi173 ceph-mon[36116]: mds.? [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] up:active 2024-03-23T22:59:32.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:32 smithi173 ceph-mon[36116]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-23T22:59:32.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:32 smithi203 ceph-mon[36693]: pgmap v320: 97 pgs: 2 creating+peering, 95 active+clean; 578 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s wr, 4 op/s 2024-03-23T22:59:32.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:32 smithi203 ceph-mon[36693]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-23T22:59:32.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:32 smithi203 ceph-mon[36693]: mds.? [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] up:active 2024-03-23T22:59:32.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:32 smithi203 ceph-mon[36693]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-23T22:59:32.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:32 smithi203 ceph-mon[36693]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-23T22:59:32.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:32 smithi203 ceph-mon[36693]: Cluster is now healthy 2024-03-23T22:59:32.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:32 smithi203 ceph-mon[36693]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-03-23T22:59:32.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:32 smithi203 ceph-mon[36693]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-03-23T22:59:32.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:32 smithi203 ceph-mon[36693]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-23T22:59:32.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:32 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/503128153' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-23T22:59:32.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:32 smithi203 ceph-mon[36693]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-23T22:59:32.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:32 smithi203 ceph-mon[36693]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-23T22:59:32.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:32 smithi203 ceph-mon[36693]: mds.? [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] up:active 2024-03-23T22:59:32.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:32 smithi203 ceph-mon[36693]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-23T22:59:32.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.935+0000 7fbb8f600640 1 -- 172.21.15.12:0/3789518093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb88076060 msgr2=0x7fbb88076480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:32.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.935+0000 7fbb8f600640 1 --2- 172.21.15.12:0/3789518093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb88076060 0x7fbb88076480 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c009900 tx=0x7fbb7c02bc30 comp rx=0 tx=0).stop 2024-03-23T22:59:32.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.935+0000 7fbb8f600640 1 -- 172.21.15.12:0/3789518093 shutdown_connections 2024-03-23T22:59:32.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.935+0000 7fbb8f600640 1 --2- 172.21.15.12:0/3789518093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb88077c40 0x7fbb880784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:32.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.935+0000 7fbb8f600640 1 --2- 172.21.15.12:0/3789518093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb880772a0 0x7fbb88077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:32.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.935+0000 7fbb8f600640 1 --2- 172.21.15.12:0/3789518093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb88076060 0x7fbb88076480 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:32.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.935+0000 7fbb8f600640 1 -- 172.21.15.12:0/3789518093 >> 172.21.15.12:0/3789518093 conn(0x7fbb880fee50 msgr2=0x7fbb88101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:32.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.935+0000 7fbb8f600640 1 -- 172.21.15.12:0/3789518093 shutdown_connections 2024-03-23T22:59:32.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.935+0000 7fbb8f600640 1 -- 172.21.15.12:0/3789518093 wait complete. 2024-03-23T22:59:32.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.936+0000 7fbb8f600640 1 Processor -- start 2024-03-23T22:59:32.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.936+0000 7fbb8f600640 1 -- start start 2024-03-23T22:59:32.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.937+0000 7fbb8f600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb88076060 0x7fbb8819dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:32.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.937+0000 7fbb8f600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb880772a0 0x7fbb8819e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:32.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.937+0000 7fbb8f600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb88077c40 0x7fbb881a55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:32.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.937+0000 7fbb8f600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb88113990 con 0x7fbb88076060 2024-03-23T22:59:32.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.937+0000 7fbb8f600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb88113810 con 0x7fbb880772a0 2024-03-23T22:59:32.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.937+0000 7fbb8f600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb88113b10 con 0x7fbb88077c40 2024-03-23T22:59:32.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.937+0000 7fbb8cc00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb88076060 0x7fbb8819dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:32.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.937+0000 7fbb8d600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb88077c40 0x7fbb881a55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:32.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.937+0000 7fbb8cc00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb88076060 0x7fbb8819dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37306/0 (socket says 172.21.15.12:37306) 2024-03-23T22:59:32.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.937+0000 7fbb8cc00640 1 -- 172.21.15.12:0/2917049281 learned_addr learned my addr 172.21.15.12:0/2917049281 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:32.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.937+0000 7fbb87e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb880772a0 0x7fbb8819e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:32.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.937+0000 7fbb87e00640 1 -- 172.21.15.12:0/2917049281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb88077c40 msgr2=0x7fbb881a55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:32.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.937+0000 7fbb87e00640 1 --2- 172.21.15.12:0/2917049281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb88077c40 0x7fbb881a55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:32.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.937+0000 7fbb87e00640 1 -- 172.21.15.12:0/2917049281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb88076060 msgr2=0x7fbb8819dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:32.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.937+0000 7fbb87e00640 1 --2- 172.21.15.12:0/2917049281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb88076060 0x7fbb8819dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:32.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.937+0000 7fbb87e00640 1 -- 172.21.15.12:0/2917049281 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb881a5c10 con 0x7fbb880772a0 2024-03-23T22:59:32.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.937+0000 7fbb8cc00640 1 --2- 172.21.15.12:0/2917049281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb88076060 0x7fbb8819dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T22:59:32.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.938+0000 7fbb87e00640 1 --2- 172.21.15.12:0/2917049281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb880772a0 0x7fbb8819e530 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbb7400e9f0 tx=0x7fbb7400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:32.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.938+0000 7fbb85600640 1 -- 172.21.15.12:0/2917049281 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7400ccc0 con 0x7fbb880772a0 2024-03-23T22:59:32.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.938+0000 7fbb85600640 1 -- 172.21.15.12:0/2917049281 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb74010910 con 0x7fbb880772a0 2024-03-23T22:59:32.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.938+0000 7fbb85600640 1 -- 172.21.15.12:0/2917049281 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb74018800 con 0x7fbb880772a0 2024-03-23T22:59:32.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.938+0000 7fbb8f600640 1 -- 172.21.15.12:0/2917049281 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb881a5f00 con 0x7fbb880772a0 2024-03-23T22:59:32.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.938+0000 7fbb8f600640 1 -- 172.21.15.12:0/2917049281 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb881a63c0 con 0x7fbb880772a0 2024-03-23T22:59:32.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.939+0000 7fbb8f600640 1 -- 172.21.15.12:0/2917049281 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb88110230 con 0x7fbb880772a0 2024-03-23T22:59:32.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.942+0000 7fbb85600640 1 -- 172.21.15.12:0/2917049281 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbb74010ac0 con 0x7fbb880772a0 2024-03-23T22:59:32.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.943+0000 7fbb85600640 1 --2- 172.21.15.12:0/2917049281 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbb700a2800 0x7fbb700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:32.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.943+0000 7fbb85600640 1 -- 172.21.15.12:0/2917049281 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fbb740cec70 con 0x7fbb880772a0 2024-03-23T22:59:32.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.943+0000 7fbb8cc00640 1 --2- 172.21.15.12:0/2917049281 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbb700a2800 0x7fbb700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:32.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.944+0000 7fbb8cc00640 1 --2- 172.21.15.12:0/2917049281 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbb700a2800 0x7fbb700a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c02c8e0 tx=0x7fbb7c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:32.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:32.947+0000 7fbb85600640 1 -- 172.21.15.12:0/2917049281 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbb7409bdc0 con 0x7fbb880772a0 2024-03-23T22:59:33.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.172+0000 7fbb8f600640 1 -- 172.21.15.12:0/2917049281 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb88061b70 con 0x7fbb700a2800 2024-03-23T22:59:33.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.203+0000 7fbb85600640 1 -- 172.21.15.12:0/2917049281 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbb88061b70 con 0x7fbb700a2800 2024-03-23T22:59:33.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.206+0000 7fbb8f600640 1 -- 172.21.15.12:0/2917049281 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbb700a2800 msgr2=0x7fbb700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:33.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.206+0000 7fbb8f600640 1 --2- 172.21.15.12:0/2917049281 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbb700a2800 0x7fbb700a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c02c8e0 tx=0x7fbb7c02df90 comp rx=0 tx=0).stop 2024-03-23T22:59:33.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.206+0000 7fbb8f600640 1 -- 172.21.15.12:0/2917049281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb880772a0 msgr2=0x7fbb8819e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:33.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.206+0000 7fbb8f600640 1 --2- 172.21.15.12:0/2917049281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb880772a0 0x7fbb8819e530 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbb7400e9f0 tx=0x7fbb7400edf0 comp rx=0 tx=0).stop 2024-03-23T22:59:33.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.206+0000 7fbb8f600640 1 -- 172.21.15.12:0/2917049281 shutdown_connections 2024-03-23T22:59:33.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.206+0000 7fbb8f600640 1 --2- 172.21.15.12:0/2917049281 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbb700a2800 0x7fbb700a4cc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:33.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.206+0000 7fbb8f600640 1 --2- 172.21.15.12:0/2917049281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb88077c40 0x7fbb881a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:33.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.206+0000 7fbb8f600640 1 --2- 172.21.15.12:0/2917049281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb880772a0 0x7fbb8819e530 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:33.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.206+0000 7fbb8f600640 1 --2- 172.21.15.12:0/2917049281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb88076060 0x7fbb8819dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:33.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.206+0000 7fbb8f600640 1 -- 172.21.15.12:0/2917049281 >> 172.21.15.12:0/2917049281 conn(0x7fbb880fee50 msgr2=0x7fbb88100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:33.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.206+0000 7fbb8f600640 1 -- 172.21.15.12:0/2917049281 shutdown_connections 2024-03-23T22:59:33.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.206+0000 7fbb8f600640 1 -- 172.21.15.12:0/2917049281 wait complete. 2024-03-23T22:59:33.224 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-23T22:59:33.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.447+0000 7fd871800640 1 -- 172.21.15.12:0/3712041592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd86c102060 msgr2=0x7fd86c104450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:33.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.447+0000 7fd871800640 1 --2- 172.21.15.12:0/3712041592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd86c102060 0x7fd86c104450 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd85c009ca0 tx=0x7fd85c02c100 comp rx=0 tx=0).stop 2024-03-23T22:59:33.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.447+0000 7fd871800640 1 -- 172.21.15.12:0/3712041592 shutdown_connections 2024-03-23T22:59:33.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.447+0000 7fd871800640 1 --2- 172.21.15.12:0/3712041592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd86c1072c0 0x7fd86c1096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:33.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.447+0000 7fd871800640 1 --2- 172.21.15.12:0/3712041592 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd86c104990 0x7fd86c106d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:33.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.448+0000 7fd871800640 1 --2- 172.21.15.12:0/3712041592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd86c102060 0x7fd86c104450 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:33.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.448+0000 7fd871800640 1 -- 172.21.15.12:0/3712041592 >> 172.21.15.12:0/3712041592 conn(0x7fd86c0fba10 msgr2=0x7fd86c0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:33.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.448+0000 7fd871800640 1 -- 172.21.15.12:0/3712041592 shutdown_connections 2024-03-23T22:59:33.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.448+0000 7fd871800640 1 -- 172.21.15.12:0/3712041592 wait complete. 2024-03-23T22:59:33.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.449+0000 7fd871800640 1 Processor -- start 2024-03-23T22:59:33.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.449+0000 7fd871800640 1 -- start start 2024-03-23T22:59:33.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.449+0000 7fd871800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd86c102060 0x7fd86c1019d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:33.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.449+0000 7fd871800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd86c104990 0x7fd86c100010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:33.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.449+0000 7fd871800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd86c1072c0 0x7fd86c100550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:33.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.449+0000 7fd871800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c10d210 con 0x7fd86c1072c0 2024-03-23T22:59:33.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.449+0000 7fd871800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c10d090 con 0x7fd86c102060 2024-03-23T22:59:33.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.449+0000 7fd871800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c10d390 con 0x7fd86c104990 2024-03-23T22:59:33.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.449+0000 7fd86be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd86c102060 0x7fd86c1019d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:33.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.449+0000 7fd86be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd86c102060 0x7fd86c1019d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:33068/0 (socket says 172.21.15.12:33068) 2024-03-23T22:59:33.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.450+0000 7fd86be00640 1 -- 172.21.15.12:0/1841914144 learned_addr learned my addr 172.21.15.12:0/1841914144 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:33.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.450+0000 7fd863400640 1 --2- 172.21.15.12:0/1841914144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd86c104990 0x7fd86c100010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:33.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.450+0000 7fd870e00640 1 --2- 172.21.15.12:0/1841914144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd86c1072c0 0x7fd86c100550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:33.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.450+0000 7fd86be00640 1 -- 172.21.15.12:0/1841914144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd86c104990 msgr2=0x7fd86c100010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:33.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.450+0000 7fd86be00640 1 --2- 172.21.15.12:0/1841914144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd86c104990 0x7fd86c100010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:33.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.450+0000 7fd86be00640 1 -- 172.21.15.12:0/1841914144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd86c1072c0 msgr2=0x7fd86c100550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:33.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.450+0000 7fd86be00640 1 --2- 172.21.15.12:0/1841914144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd86c1072c0 0x7fd86c100550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:33.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.450+0000 7fd86be00640 1 -- 172.21.15.12:0/1841914144 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86c100de0 con 0x7fd86c102060 2024-03-23T22:59:33.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.450+0000 7fd86be00640 1 --2- 172.21.15.12:0/1841914144 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd86c102060 0x7fd86c1019d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd85c0325f0 tx=0x7fd85c035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:33.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.450+0000 7fd869600640 1 -- 172.21.15.12:0/1841914144 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c039040 con 0x7fd86c102060 2024-03-23T22:59:33.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.450+0000 7fd869600640 1 -- 172.21.15.12:0/1841914144 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd85c044070 con 0x7fd86c102060 2024-03-23T22:59:33.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.450+0000 7fd869600640 1 -- 172.21.15.12:0/1841914144 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0044d0 con 0x7fd86c102060 2024-03-23T22:59:33.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.450+0000 7fd871800640 1 -- 172.21.15.12:0/1841914144 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86c1a19a0 con 0x7fd86c102060 2024-03-23T22:59:33.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.451+0000 7fd871800640 1 -- 172.21.15.12:0/1841914144 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd86c1a1e70 con 0x7fd86c102060 2024-03-23T22:59:33.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.452+0000 7fd871800640 1 -- 172.21.15.12:0/1841914144 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd86c0fd500 con 0x7fd86c102060 2024-03-23T22:59:33.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.453+0000 7fd869600640 1 -- 172.21.15.12:0/1841914144 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd85c004670 con 0x7fd86c102060 2024-03-23T22:59:33.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.455+0000 7fd869600640 1 --2- 172.21.15.12:0/1841914144 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd8500a2880 0x7fd8500a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:33.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.455+0000 7fd869600640 1 -- 172.21.15.12:0/1841914144 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd85c0f0e80 con 0x7fd86c102060 2024-03-23T22:59:33.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.455+0000 7fd863400640 1 --2- 172.21.15.12:0/1841914144 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd8500a2880 0x7fd8500a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:33.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.456+0000 7fd863400640 1 --2- 172.21.15.12:0/1841914144 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd8500a2880 0x7fd8500a4d40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd86c101740 tx=0x7fd854005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:33.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.458+0000 7fd869600640 1 -- 172.21.15.12:0/1841914144 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd85c0bdfd0 con 0x7fd86c102060 2024-03-23T22:59:33.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.513+0000 7fd869600640 1 -- 172.21.15.12:0/1841914144 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fd85c0b9f40 con 0x7fd86c102060 2024-03-23T22:59:33.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.691+0000 7fd871800640 1 -- 172.21.15.12:0/1841914144 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7fd86c061b70 con 0x7fd8500a2880 2024-03-23T22:59:33.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.701+0000 7fd869600640 1 -- 172.21.15.12:0/1841914144 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7fd86c061b70 con 0x7fd8500a2880 2024-03-23T22:59:33.701 INFO:teuthology.orchestra.run.smithi012.stdout:/volumes/qa/sv_0/b2b88360-276a-45e6-84f5-370fa7d36b36 2024-03-23T22:59:33.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.703+0000 7fd871800640 1 -- 172.21.15.12:0/1841914144 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd8500a2880 msgr2=0x7fd8500a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:33.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.703+0000 7fd871800640 1 --2- 172.21.15.12:0/1841914144 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd8500a2880 0x7fd8500a4d40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd86c101740 tx=0x7fd854005f90 comp rx=0 tx=0).stop 2024-03-23T22:59:33.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.704+0000 7fd871800640 1 -- 172.21.15.12:0/1841914144 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd86c102060 msgr2=0x7fd86c1019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:33.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.704+0000 7fd871800640 1 --2- 172.21.15.12:0/1841914144 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd86c102060 0x7fd86c1019d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd85c0325f0 tx=0x7fd85c035d10 comp rx=0 tx=0).stop 2024-03-23T22:59:33.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.704+0000 7fd871800640 1 -- 172.21.15.12:0/1841914144 shutdown_connections 2024-03-23T22:59:33.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.704+0000 7fd871800640 1 --2- 172.21.15.12:0/1841914144 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd8500a2880 0x7fd8500a4d40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:33.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.704+0000 7fd871800640 1 --2- 172.21.15.12:0/1841914144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd86c1072c0 0x7fd86c100550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:33.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.704+0000 7fd871800640 1 --2- 172.21.15.12:0/1841914144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd86c104990 0x7fd86c100010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:33.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.704+0000 7fd871800640 1 --2- 172.21.15.12:0/1841914144 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd86c102060 0x7fd86c1019d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:33.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.704+0000 7fd871800640 1 -- 172.21.15.12:0/1841914144 >> 172.21.15.12:0/1841914144 conn(0x7fd86c0fba10 msgr2=0x7fd86c1029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:33.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.704+0000 7fd871800640 1 -- 172.21.15.12:0/1841914144 shutdown_connections 2024-03-23T22:59:33.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.704+0000 7fd871800640 1 -- 172.21.15.12:0/1841914144 wait complete. 2024-03-23T22:59:33.723 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-23T22:59:33.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:33 smithi012 ceph-mon[31695]: from='client.24683 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:59:33.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:33 smithi173 ceph-mon[36116]: from='client.24683 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:59:33.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:33 smithi203 ceph-mon[36693]: from='client.24683 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:59:33.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.948+0000 7fa9b0a00640 1 -- 172.21.15.12:0/3533469937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9ac109460 msgr2=0x7fa9ac10b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:33.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.948+0000 7fa9b0a00640 1 --2- 172.21.15.12:0/3533469937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9ac109460 0x7fa9ac10b8f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4009be0 tx=0x7fa9a402bef0 comp rx=0 tx=0).stop 2024-03-23T22:59:33.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.948+0000 7fa9b0a00640 1 -- 172.21.15.12:0/3533469937 shutdown_connections 2024-03-23T22:59:33.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.948+0000 7fa9b0a00640 1 --2- 172.21.15.12:0/3533469937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9ac109460 0x7fa9ac10b8f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:33.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.948+0000 7fa9b0a00640 1 --2- 172.21.15.12:0/3533469937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9ac074100 0x7fa9ac074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:33.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.948+0000 7fa9b0a00640 1 --2- 172.21.15.12:0/3533469937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ac0756e0 0x7fa9ac073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:33.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.948+0000 7fa9b0a00640 1 -- 172.21.15.12:0/3533469937 >> 172.21.15.12:0/3533469937 conn(0x7fa9ac0fcb10 msgr2=0x7fa9ac0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:33.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.948+0000 7fa9b0a00640 1 -- 172.21.15.12:0/3533469937 shutdown_connections 2024-03-23T22:59:33.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.948+0000 7fa9b0a00640 1 -- 172.21.15.12:0/3533469937 wait complete. 2024-03-23T22:59:33.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.949+0000 7fa9b0a00640 1 Processor -- start 2024-03-23T22:59:33.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.949+0000 7fa9b0a00640 1 -- start start 2024-03-23T22:59:33.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.949+0000 7fa9b0a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9ac074100 0x7fa9ac199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:33.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.949+0000 7fa9b0a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9ac0756e0 0x7fa9ac199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:33.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.949+0000 7fa9b0a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ac109460 0x7fa9ac1a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:33.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.949+0000 7fa9b0a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac10f450 con 0x7fa9ac109460 2024-03-23T22:59:33.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.949+0000 7fa9b0a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac10f2d0 con 0x7fa9ac0756e0 2024-03-23T22:59:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.949+0000 7fa9b0a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac10f5d0 con 0x7fa9ac074100 2024-03-23T22:59:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.950+0000 7fa9aa800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9ac074100 0x7fa9ac199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.950+0000 7fa9aa800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9ac074100 0x7fa9ac199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:46600/0 (socket says 172.21.15.12:46600) 2024-03-23T22:59:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.950+0000 7fa9aa800640 1 -- 172.21.15.12:0/1232867555 learned_addr learned my addr 172.21.15.12:0/1232867555 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.950+0000 7fa9ab200640 1 --2- 172.21.15.12:0/1232867555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ac109460 0x7fa9ac1a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.950+0000 7fa9a1e00640 1 --2- 172.21.15.12:0/1232867555 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9ac0756e0 0x7fa9ac199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.950+0000 7fa9aa800640 1 -- 172.21.15.12:0/1232867555 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9ac0756e0 msgr2=0x7fa9ac199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.950+0000 7fa9aa800640 1 --2- 172.21.15.12:0/1232867555 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9ac0756e0 0x7fa9ac199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.950+0000 7fa9aa800640 1 -- 172.21.15.12:0/1232867555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ac109460 msgr2=0x7fa9ac1a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.950+0000 7fa9aa800640 1 --2- 172.21.15.12:0/1232867555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ac109460 0x7fa9ac1a0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.950+0000 7fa9aa800640 1 -- 172.21.15.12:0/1232867555 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ac1a1640 con 0x7fa9ac074100 2024-03-23T22:59:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.950+0000 7fa9a1e00640 1 --2- 172.21.15.12:0/1232867555 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9ac0756e0 0x7fa9ac199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T22:59:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.950+0000 7fa9ab200640 1 --2- 172.21.15.12:0/1232867555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ac109460 0x7fa9ac1a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T22:59:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.950+0000 7fa9aa800640 1 --2- 172.21.15.12:0/1232867555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9ac074100 0x7fa9ac199a70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa99c00d950 tx=0x7fa99c00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.951+0000 7fa9a3e00640 1 -- 172.21.15.12:0/1232867555 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99c017030 con 0x7fa9ac074100 2024-03-23T22:59:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.951+0000 7fa9a3e00640 1 -- 172.21.15.12:0/1232867555 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa99c004020 con 0x7fa9ac074100 2024-03-23T22:59:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.951+0000 7fa9a3e00640 1 -- 172.21.15.12:0/1232867555 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99c0047a0 con 0x7fa9ac074100 2024-03-23T22:59:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.951+0000 7fa9b0a00640 1 -- 172.21.15.12:0/1232867555 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9ac1a1930 con 0x7fa9ac074100 2024-03-23T22:59:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.951+0000 7fa9b0a00640 1 -- 172.21.15.12:0/1232867555 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9ac1a1e30 con 0x7fa9ac074100 2024-03-23T22:59:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.952+0000 7fa9b0a00640 1 -- 172.21.15.12:0/1232867555 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9ac108190 con 0x7fa9ac074100 2024-03-23T22:59:33.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.954+0000 7fa9a3e00640 1 -- 172.21.15.12:0/1232867555 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fa99c0041d0 con 0x7fa9ac074100 2024-03-23T22:59:33.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.955+0000 7fa9a3e00640 1 --2- 172.21.15.12:0/1232867555 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa9900a2910 0x7fa9900a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:33.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.955+0000 7fa9a3e00640 1 -- 172.21.15.12:0/1232867555 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa99c0ce550 con 0x7fa9ac074100 2024-03-23T22:59:33.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.955+0000 7fa9a1e00640 1 --2- 172.21.15.12:0/1232867555 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa9900a2910 0x7fa9900a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:33.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.956+0000 7fa9a1e00640 1 --2- 172.21.15.12:0/1232867555 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa9900a2910 0x7fa9900a4dd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac19af30 tx=0x7fa994009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:33.960+0000 7fa9a3e00640 1 -- 172.21.15.12:0/1232867555 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa99c09b540 con 0x7fa9ac074100 2024-03-23T22:59:34.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.183+0000 7fa9b0a00640 1 -- 172.21.15.12:0/1232867555 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9ac061ae0 con 0x7fa9900a2910 2024-03-23T22:59:34.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.218+0000 7fa9a3e00640 1 -- 172.21.15.12:0/1232867555 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa9ac061ae0 con 0x7fa9900a2910 2024-03-23T22:59:34.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.221+0000 7fa9b0a00640 1 -- 172.21.15.12:0/1232867555 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa9900a2910 msgr2=0x7fa9900a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:34.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.221+0000 7fa9b0a00640 1 --2- 172.21.15.12:0/1232867555 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa9900a2910 0x7fa9900a4dd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac19af30 tx=0x7fa994009260 comp rx=0 tx=0).stop 2024-03-23T22:59:34.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.221+0000 7fa9b0a00640 1 -- 172.21.15.12:0/1232867555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9ac074100 msgr2=0x7fa9ac199a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:34.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.221+0000 7fa9b0a00640 1 --2- 172.21.15.12:0/1232867555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9ac074100 0x7fa9ac199a70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa99c00d950 tx=0x7fa99c00de10 comp rx=0 tx=0).stop 2024-03-23T22:59:34.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.221+0000 7fa9b0a00640 1 -- 172.21.15.12:0/1232867555 shutdown_connections 2024-03-23T22:59:34.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.221+0000 7fa9b0a00640 1 --2- 172.21.15.12:0/1232867555 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa9900a2910 0x7fa9900a4dd0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:34.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.221+0000 7fa9b0a00640 1 --2- 172.21.15.12:0/1232867555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ac109460 0x7fa9ac1a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:34.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.221+0000 7fa9b0a00640 1 --2- 172.21.15.12:0/1232867555 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9ac0756e0 0x7fa9ac199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:34.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.221+0000 7fa9b0a00640 1 --2- 172.21.15.12:0/1232867555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9ac074100 0x7fa9ac199a70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:34.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.221+0000 7fa9b0a00640 1 -- 172.21.15.12:0/1232867555 >> 172.21.15.12:0/1232867555 conn(0x7fa9ac0fcb10 msgr2=0x7fa9ac105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:34.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.221+0000 7fa9b0a00640 1 -- 172.21.15.12:0/1232867555 shutdown_connections 2024-03-23T22:59:34.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.221+0000 7fa9b0a00640 1 -- 172.21.15.12:0/1232867555 wait complete. 2024-03-23T22:59:34.238 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-23T22:59:34.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.464+0000 7fdd4d800640 1 -- 172.21.15.12:0/669284962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd48104ad0 msgr2=0x7fdd48104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:34.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.464+0000 7fdd4d800640 1 --2- 172.21.15.12:0/669284962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd48104ad0 0x7fdd48104f50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdd34009900 tx=0x7fdd3402bc30 comp rx=0 tx=0).stop 2024-03-23T22:59:34.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.464+0000 7fdd4d800640 1 -- 172.21.15.12:0/669284962 shutdown_connections 2024-03-23T22:59:34.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.464+0000 7fdd4d800640 1 --2- 172.21.15.12:0/669284962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd48105490 0x7fdd4810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:34.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.464+0000 7fdd4d800640 1 --2- 172.21.15.12:0/669284962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd48104ad0 0x7fdd48104f50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:34.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.464+0000 7fdd4d800640 1 --2- 172.21.15.12:0/669284962 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd481038e0 0x7fdd48103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:34.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.464+0000 7fdd4d800640 1 -- 172.21.15.12:0/669284962 >> 172.21.15.12:0/669284962 conn(0x7fdd480fee50 msgr2=0x7fdd48101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:34.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.465+0000 7fdd4d800640 1 -- 172.21.15.12:0/669284962 shutdown_connections 2024-03-23T22:59:34.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.465+0000 7fdd4d800640 1 -- 172.21.15.12:0/669284962 wait complete. 2024-03-23T22:59:34.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.466+0000 7fdd4d800640 1 Processor -- start 2024-03-23T22:59:34.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.466+0000 7fdd4d800640 1 -- start start 2024-03-23T22:59:34.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.466+0000 7fdd4d800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd481038e0 0x7fdd48199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:34.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.466+0000 7fdd4d800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd48104ad0 0x7fdd4819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:34.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.466+0000 7fdd4d800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd48105490 0x7fdd481a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:34.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.466+0000 7fdd4d800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4810f5b0 con 0x7fdd48104ad0 2024-03-23T22:59:34.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.466+0000 7fdd4d800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4810f430 con 0x7fdd481038e0 2024-03-23T22:59:34.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.466+0000 7fdd4d800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4810f730 con 0x7fdd48105490 2024-03-23T22:59:34.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.467+0000 7fdd46a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd481038e0 0x7fdd48199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:34.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.467+0000 7fdd46a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd481038e0 0x7fdd48199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:46362/0 (socket says 172.21.15.12:46362) 2024-03-23T22:59:34.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.467+0000 7fdd46a00640 1 -- 172.21.15.12:0/3809262470 learned_addr learned my addr 172.21.15.12:0/3809262470 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:34.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.467+0000 7fdd46000640 1 --2- 172.21.15.12:0/3809262470 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd48104ad0 0x7fdd4819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:34.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.467+0000 7fdd47400640 1 --2- 172.21.15.12:0/3809262470 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd48105490 0x7fdd481a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.467+0000 7fdd46a00640 1 -- 172.21.15.12:0/3809262470 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd48105490 msgr2=0x7fdd481a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.467+0000 7fdd46a00640 1 --2- 172.21.15.12:0/3809262470 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd48105490 0x7fdd481a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.467+0000 7fdd46a00640 1 -- 172.21.15.12:0/3809262470 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd48104ad0 msgr2=0x7fdd4819a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.467+0000 7fdd46a00640 1 --2- 172.21.15.12:0/3809262470 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd48104ad0 0x7fdd4819a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.467+0000 7fdd46a00640 1 -- 172.21.15.12:0/3809262470 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd481a1860 con 0x7fdd481038e0 2024-03-23T22:59:34.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.467+0000 7fdd46a00640 1 --2- 172.21.15.12:0/3809262470 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd481038e0 0x7fdd48199c50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdd3800e9c0 tx=0x7fdd3800edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:34.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.468+0000 7fdd3f400640 1 -- 172.21.15.12:0/3809262470 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3800ccc0 con 0x7fdd481038e0 2024-03-23T22:59:34.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.468+0000 7fdd3f400640 1 -- 172.21.15.12:0/3809262470 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd380035f0 con 0x7fdd481038e0 2024-03-23T22:59:34.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.468+0000 7fdd4d800640 1 -- 172.21.15.12:0/3809262470 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd481a1b50 con 0x7fdd481038e0 2024-03-23T22:59:34.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.468+0000 7fdd3f400640 1 -- 172.21.15.12:0/3809262470 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3800ce60 con 0x7fdd481038e0 2024-03-23T22:59:34.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.468+0000 7fdd4d800640 1 -- 172.21.15.12:0/3809262470 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd481a2050 con 0x7fdd481038e0 2024-03-23T22:59:34.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.469+0000 7fdd4d800640 1 -- 172.21.15.12:0/3809262470 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd4810acd0 con 0x7fdd481038e0 2024-03-23T22:59:34.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.471+0000 7fdd3f400640 1 -- 172.21.15.12:0/3809262470 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fdd380037a0 con 0x7fdd481038e0 2024-03-23T22:59:34.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.472+0000 7fdd3f400640 1 --2- 172.21.15.12:0/3809262470 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdd2c0a2890 0x7fdd2c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:34.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.472+0000 7fdd3f400640 1 -- 172.21.15.12:0/3809262470 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fdd380cef10 con 0x7fdd481038e0 2024-03-23T22:59:34.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.473+0000 7fdd46000640 1 --2- 172.21.15.12:0/3809262470 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdd2c0a2890 0x7fdd2c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:34.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.473+0000 7fdd46000640 1 --2- 172.21.15.12:0/3809262470 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdd2c0a2890 0x7fdd2c0a4d50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdd3402c240 tx=0x7fdd3402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:34.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.479+0000 7fdd3f400640 1 -- 172.21.15.12:0/3809262470 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdd3809bf00 con 0x7fdd481038e0 2024-03-23T22:59:34.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.715+0000 7fdd4d800640 1 -- 172.21.15.12:0/3809262470 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd48061ae0 con 0x7fdd2c0a2890 2024-03-23T22:59:34.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.728+0000 7fdd3f400640 1 -- 172.21.15.12:0/3809262470 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7fdd48061ae0 con 0x7fdd2c0a2890 2024-03-23T22:59:34.728 INFO:teuthology.orchestra.run.smithi012.stdout:/volumes/qa/sv_1/d55fdac7-2855-47ef-9e03-4894504668e6 2024-03-23T22:59:34.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.731+0000 7fdd4d800640 1 -- 172.21.15.12:0/3809262470 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdd2c0a2890 msgr2=0x7fdd2c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:34.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.731+0000 7fdd4d800640 1 --2- 172.21.15.12:0/3809262470 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdd2c0a2890 0x7fdd2c0a4d50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdd3402c240 tx=0x7fdd3402df90 comp rx=0 tx=0).stop 2024-03-23T22:59:34.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.731+0000 7fdd4d800640 1 -- 172.21.15.12:0/3809262470 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd481038e0 msgr2=0x7fdd48199c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:34.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.731+0000 7fdd4d800640 1 --2- 172.21.15.12:0/3809262470 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd481038e0 0x7fdd48199c50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdd3800e9c0 tx=0x7fdd3800edc0 comp rx=0 tx=0).stop 2024-03-23T22:59:34.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.731+0000 7fdd4d800640 1 -- 172.21.15.12:0/3809262470 shutdown_connections 2024-03-23T22:59:34.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.731+0000 7fdd4d800640 1 --2- 172.21.15.12:0/3809262470 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdd2c0a2890 0x7fdd2c0a4d50 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:34.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.731+0000 7fdd4d800640 1 --2- 172.21.15.12:0/3809262470 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd48105490 0x7fdd481a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:34.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.731+0000 7fdd4d800640 1 --2- 172.21.15.12:0/3809262470 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd48104ad0 0x7fdd4819a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:34.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.731+0000 7fdd4d800640 1 --2- 172.21.15.12:0/3809262470 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd481038e0 0x7fdd48199c50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:34.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.731+0000 7fdd4d800640 1 -- 172.21.15.12:0/3809262470 >> 172.21.15.12:0/3809262470 conn(0x7fdd480fee50 msgr2=0x7fdd48107ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:34.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.731+0000 7fdd4d800640 1 -- 172.21.15.12:0/3809262470 shutdown_connections 2024-03-23T22:59:34.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.731+0000 7fdd4d800640 1 -- 172.21.15.12:0/3809262470 wait complete. 2024-03-23T22:59:34.753 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T22:59:34.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:34 smithi012 ceph-mon[31695]: from='client.24695 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:59:34.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:34 smithi012 ceph-mon[31695]: pgmap v321: 97 pgs: 97 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 4.5 KiB/s wr, 14 op/s 2024-03-23T22:59:34.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:34 smithi012 ceph-mon[31695]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-23T22:59:34.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:34 smithi012 ceph-mon[31695]: from='client.24701 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:59:34.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:34 smithi173 ceph-mon[36116]: from='client.24695 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:59:34.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:34 smithi173 ceph-mon[36116]: pgmap v321: 97 pgs: 97 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 4.5 KiB/s wr, 14 op/s 2024-03-23T22:59:34.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:34 smithi173 ceph-mon[36116]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-23T22:59:34.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:34 smithi173 ceph-mon[36116]: from='client.24701 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:59:34.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:34 smithi203 ceph-mon[36693]: from='client.24695 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:59:34.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:34 smithi203 ceph-mon[36693]: pgmap v321: 97 pgs: 97 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 4.5 KiB/s wr, 14 op/s 2024-03-23T22:59:34.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:34 smithi203 ceph-mon[36693]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-23T22:59:34.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:34 smithi203 ceph-mon[36693]: from='client.24701 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:59:34.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.981+0000 7f22cf400640 1 -- 172.21.15.12:0/2017706431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22c8103270 msgr2=0x7f22c8109b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:34.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.981+0000 7f22cf400640 1 --2- 172.21.15.12:0/2017706431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22c8103270 0x7f22c8109b00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f22b0009900 tx=0x7f22b002bc30 comp rx=0 tx=0).stop 2024-03-23T22:59:34.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.981+0000 7f22cf400640 1 -- 172.21.15.12:0/2017706431 shutdown_connections 2024-03-23T22:59:34.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.981+0000 7f22cf400640 1 --2- 172.21.15.12:0/2017706431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22c8103270 0x7f22c8109b00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:34.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.981+0000 7f22cf400640 1 --2- 172.21.15.12:0/2017706431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22c81028b0 0x7f22c8102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:34.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.981+0000 7f22cf400640 1 --2- 172.21.15.12:0/2017706431 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22c81016c0 0x7f22c8101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:34.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.981+0000 7f22cf400640 1 -- 172.21.15.12:0/2017706431 >> 172.21.15.12:0/2017706431 conn(0x7f22c80fcc50 msgr2=0x7f22c80ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:34.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.981+0000 7f22cf400640 1 -- 172.21.15.12:0/2017706431 shutdown_connections 2024-03-23T22:59:34.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.981+0000 7f22cf400640 1 -- 172.21.15.12:0/2017706431 wait complete. 2024-03-23T22:59:34.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.982+0000 7f22cf400640 1 Processor -- start 2024-03-23T22:59:34.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.982+0000 7f22cf400640 1 -- start start 2024-03-23T22:59:34.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.982+0000 7f22cf400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22c81016c0 0x7f22c8199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:34.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.982+0000 7f22cf400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22c81028b0 0x7f22c819a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:34.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.982+0000 7f22cf400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22c8103270 0x7f22c81a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:34.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.982+0000 7f22cf400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c810d370 con 0x7f22c8103270 2024-03-23T22:59:34.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.982+0000 7f22cf400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c810d1f0 con 0x7f22c81028b0 2024-03-23T22:59:34.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.982+0000 7f22cf400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c810d4f0 con 0x7f22c81016c0 2024-03-23T22:59:34.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.982+0000 7f22cca00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22c81016c0 0x7f22c8199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:34.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.983+0000 7f22c7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22c81028b0 0x7f22c819a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:34.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.983+0000 7f22c7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22c81028b0 0x7f22c819a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:46384/0 (socket says 172.21.15.12:46384) 2024-03-23T22:59:34.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.983+0000 7f22cca00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22c81016c0 0x7f22c8199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:46628/0 (socket says 172.21.15.12:46628) 2024-03-23T22:59:34.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.983+0000 7f22c7e00640 1 -- 172.21.15.12:0/1243579943 learned_addr learned my addr 172.21.15.12:0/1243579943 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:34.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.983+0000 7f22cd400640 1 --2- 172.21.15.12:0/1243579943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22c8103270 0x7f22c81a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:34.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.983+0000 7f22c7e00640 1 -- 172.21.15.12:0/1243579943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22c81016c0 msgr2=0x7f22c8199b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:34.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.983+0000 7f22c7e00640 1 --2- 172.21.15.12:0/1243579943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22c81016c0 0x7f22c8199b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:34.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.983+0000 7f22c7e00640 1 -- 172.21.15.12:0/1243579943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22c8103270 msgr2=0x7f22c81a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:34.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.983+0000 7f22c7e00640 1 --2- 172.21.15.12:0/1243579943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22c8103270 0x7f22c81a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:34.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.983+0000 7f22c7e00640 1 -- 172.21.15.12:0/1243579943 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c81a1770 con 0x7f22c81028b0 2024-03-23T22:59:34.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.983+0000 7f22cca00640 1 --2- 172.21.15.12:0/1243579943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22c81016c0 0x7f22c8199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T22:59:34.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.983+0000 7f22c7e00640 1 --2- 172.21.15.12:0/1243579943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22c81028b0 0x7f22c819a090 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f22b800e9f0 tx=0x7f22b800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:34.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.984+0000 7f22c5600640 1 -- 172.21.15.12:0/1243579943 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b800cd80 con 0x7f22c81028b0 2024-03-23T22:59:34.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.984+0000 7f22c5600640 1 -- 172.21.15.12:0/1243579943 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22b8010910 con 0x7f22c81028b0 2024-03-23T22:59:34.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.984+0000 7f22cf400640 1 -- 172.21.15.12:0/1243579943 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22c81a1a60 con 0x7f22c81028b0 2024-03-23T22:59:34.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.984+0000 7f22c5600640 1 -- 172.21.15.12:0/1243579943 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b8018990 con 0x7f22c81028b0 2024-03-23T22:59:34.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.984+0000 7f22cf400640 1 -- 172.21.15.12:0/1243579943 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22c81a2000 con 0x7f22c81028b0 2024-03-23T22:59:34.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.985+0000 7f22cf400640 1 -- 172.21.15.12:0/1243579943 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22c81a21d0 con 0x7f22c81028b0 2024-03-23T22:59:34.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.987+0000 7f22c5600640 1 -- 172.21.15.12:0/1243579943 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f22b8010ac0 con 0x7f22c81028b0 2024-03-23T22:59:34.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.988+0000 7f22c5600640 1 --2- 172.21.15.12:0/1243579943 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f22ac0a2910 0x7f22ac0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:34.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.988+0000 7f22c5600640 1 -- 172.21.15.12:0/1243579943 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f22b80cedd0 con 0x7f22c81028b0 2024-03-23T22:59:34.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.988+0000 7f22cca00640 1 --2- 172.21.15.12:0/1243579943 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f22ac0a2910 0x7f22ac0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:34.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.989+0000 7f22cca00640 1 --2- 172.21.15.12:0/1243579943 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f22ac0a2910 0x7f22ac0a4dd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f22c00099a0 tx=0x7f22c0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:34.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:34.992+0000 7f22c5600640 1 -- 172.21.15.12:0/1243579943 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f22b809b770 con 0x7f22c81028b0 2024-03-23T22:59:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.244+0000 7f22cf400640 1 -- 172.21.15.12:0/1243579943 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22c81a2560 con 0x7f22c81028b0 2024-03-23T22:59:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.245+0000 7f22c5600640 1 -- 172.21.15.12:0/1243579943 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f22b809f810 con 0x7f22c81028b0 2024-03-23T22:59:35.245 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:59:35.245 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T22:59:32.190756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24599,24550]},"id":1}]} 2024-03-23T22:59:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 22 2024-03-23T22:59:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.247+0000 7f22cf400640 1 -- 172.21.15.12:0/1243579943 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f22ac0a2910 msgr2=0x7f22ac0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.247+0000 7f22cf400640 1 --2- 172.21.15.12:0/1243579943 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f22ac0a2910 0x7f22ac0a4dd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f22c00099a0 tx=0x7f22c0009500 comp rx=0 tx=0).stop 2024-03-23T22:59:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.247+0000 7f22cf400640 1 -- 172.21.15.12:0/1243579943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22c81028b0 msgr2=0x7f22c819a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.247+0000 7f22cf400640 1 --2- 172.21.15.12:0/1243579943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22c81028b0 0x7f22c819a090 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f22b800e9f0 tx=0x7f22b800eeb0 comp rx=0 tx=0).stop 2024-03-23T22:59:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.248+0000 7f22cf400640 1 -- 172.21.15.12:0/1243579943 shutdown_connections 2024-03-23T22:59:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.248+0000 7f22cf400640 1 --2- 172.21.15.12:0/1243579943 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f22ac0a2910 0x7f22ac0a4dd0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.248+0000 7f22cf400640 1 --2- 172.21.15.12:0/1243579943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22c8103270 0x7f22c81a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.248+0000 7f22cf400640 1 --2- 172.21.15.12:0/1243579943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22c81028b0 0x7f22c819a090 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.248+0000 7f22cf400640 1 --2- 172.21.15.12:0/1243579943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22c81016c0 0x7f22c8199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.248+0000 7f22cf400640 1 -- 172.21.15.12:0/1243579943 >> 172.21.15.12:0/1243579943 conn(0x7f22c80fcc50 msgr2=0x7f22c80738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.248+0000 7f22cf400640 1 -- 172.21.15.12:0/1243579943 shutdown_connections 2024-03-23T22:59:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.248+0000 7f22cf400640 1 -- 172.21.15.12:0/1243579943 wait complete. 2024-03-23T22:59:35.269 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T22:59:35.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.492+0000 7f175b200640 1 -- 172.21.15.12:0/3978067078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1754108cf0 msgr2=0x7f1754109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:35.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.492+0000 7f175b200640 1 --2- 172.21.15.12:0/3978067078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1754108cf0 0x7f1754109170 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1744009860 tx=0x7f174402bb70 comp rx=0 tx=0).stop 2024-03-23T22:59:35.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.493+0000 7f175b200640 1 -- 172.21.15.12:0/3978067078 shutdown_connections 2024-03-23T22:59:35.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.493+0000 7f175b200640 1 --2- 172.21.15.12:0/3978067078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17541096b0 0x7f175410ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:35.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.493+0000 7f175b200640 1 --2- 172.21.15.12:0/3978067078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1754108cf0 0x7f1754109170 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:35.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.493+0000 7f175b200640 1 --2- 172.21.15.12:0/3978067078 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1754107b00 0x7f1754107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:35.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.493+0000 7f175b200640 1 -- 172.21.15.12:0/3978067078 >> 172.21.15.12:0/3978067078 conn(0x7f1754076ab0 msgr2=0x7f1754078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:35.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.493+0000 7f175b200640 1 -- 172.21.15.12:0/3978067078 shutdown_connections 2024-03-23T22:59:35.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.493+0000 7f175b200640 1 -- 172.21.15.12:0/3978067078 wait complete. 2024-03-23T22:59:35.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.494+0000 7f175b200640 1 Processor -- start 2024-03-23T22:59:35.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.494+0000 7f175b200640 1 -- start start 2024-03-23T22:59:35.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.494+0000 7f175b200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1754107b00 0x7f175419ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:35.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.494+0000 7f175b200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1754108cf0 0x7f175419e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:35.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.494+0000 7f175b200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17541096b0 0x7f17541a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:35.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.494+0000 7f175b200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1754113760 con 0x7f1754108cf0 2024-03-23T22:59:35.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.494+0000 7f175b200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17541135e0 con 0x7f17541096b0 2024-03-23T22:59:35.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.494+0000 7f175b200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17541138e0 con 0x7f1754107b00 2024-03-23T22:59:35.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.494+0000 7f1759400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1754108cf0 0x7f175419e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:35.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.494+0000 7f1759e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1754107b00 0x7f175419ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:35.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.494+0000 7f1759e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1754107b00 0x7f175419ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:46654/0 (socket says 172.21.15.12:46654) 2024-03-23T22:59:35.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.494+0000 7f1759e00640 1 -- 172.21.15.12:0/1828720937 learned_addr learned my addr 172.21.15.12:0/1828720937 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:35.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.494+0000 7f175a800640 1 --2- 172.21.15.12:0/1828720937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17541096b0 0x7f17541a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:35.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.495+0000 7f1759e00640 1 -- 172.21.15.12:0/1828720937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17541096b0 msgr2=0x7f17541a5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:35.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.495+0000 7f1759e00640 1 --2- 172.21.15.12:0/1828720937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17541096b0 0x7f17541a5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:35.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.495+0000 7f1759e00640 1 -- 172.21.15.12:0/1828720937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1754108cf0 msgr2=0x7f175419e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:35.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.495+0000 7f1759e00640 1 --2- 172.21.15.12:0/1828720937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1754108cf0 0x7f175419e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:35.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.495+0000 7f1759e00640 1 -- 172.21.15.12:0/1828720937 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17541a59d0 con 0x7f1754107b00 2024-03-23T22:59:35.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.495+0000 7f1759400640 1 --2- 172.21.15.12:0/1828720937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1754108cf0 0x7f175419e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T22:59:35.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.495+0000 7f1759e00640 1 --2- 172.21.15.12:0/1828720937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1754107b00 0x7f175419ddc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f174800e9c0 tx=0x7f174800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:35.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.495+0000 7f174ea00640 1 -- 172.21.15.12:0/1828720937 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174800cd80 con 0x7f1754107b00 2024-03-23T22:59:35.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.495+0000 7f174ea00640 1 -- 172.21.15.12:0/1828720937 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17480035f0 con 0x7f1754107b00 2024-03-23T22:59:35.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.495+0000 7f174ea00640 1 -- 172.21.15.12:0/1828720937 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1748010690 con 0x7f1754107b00 2024-03-23T22:59:35.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.495+0000 7f175b200640 1 -- 172.21.15.12:0/1828720937 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17541a5cc0 con 0x7f1754107b00 2024-03-23T22:59:35.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.495+0000 7f175b200640 1 -- 172.21.15.12:0/1828720937 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17541a61c0 con 0x7f1754107b00 2024-03-23T22:59:35.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.496+0000 7f175b200640 1 -- 172.21.15.12:0/1828720937 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f175410ee20 con 0x7f1754107b00 2024-03-23T22:59:35.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.498+0000 7f174ea00640 1 -- 172.21.15.12:0/1828720937 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f17480037a0 con 0x7f1754107b00 2024-03-23T22:59:35.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.499+0000 7f174ea00640 1 --2- 172.21.15.12:0/1828720937 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f173c0a2890 0x7f173c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:35.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.500+0000 7f174ea00640 1 -- 172.21.15.12:0/1828720937 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f17480cef60 con 0x7f1754107b00 2024-03-23T22:59:35.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.500+0000 7f1759400640 1 --2- 172.21.15.12:0/1828720937 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f173c0a2890 0x7f173c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:35.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.500+0000 7f1759400640 1 --2- 172.21.15.12:0/1828720937 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f173c0a2890 0x7f173c0a4d50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1744009830 tx=0x7f174402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:35.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.504+0000 7f174ea00640 1 -- 172.21.15.12:0/1828720937 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f174809bf50 con 0x7f1754107b00 2024-03-23T22:59:35.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:35 smithi012 ceph-mon[31695]: from='client.24700 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:59:35.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:35 smithi012 ceph-mon[31695]: from='client.24713 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:59:35.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:35 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1243579943' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:35.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.709+0000 7f175b200640 1 -- 172.21.15.12:0/1828720937 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f175402d2e0 con 0x7f1754107b00 2024-03-23T22:59:35.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.711+0000 7f174ea00640 1 -- 172.21.15.12:0/1828720937 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+22244 (secure 0 0 0) 0x7f174809b850 con 0x7f1754107b00 2024-03-23T22:59:35.711 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:59:35.712 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":86,"fsid":"90f679ae-e967-11ee-95cd-87774f69a715","created":"2024-03-23T22:50:14.906952+0000","modified":"2024-03-23T22:59:28.166927+0000","last_up_change":"2024-03-23T22:58:28.907824+0000","last_in_change":"2024-03-23T22:58:08.342487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T22:54:11.831010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T22:59:25.350974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T22:59:26.390323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b70875d7-a23d-4251-9e8b-1a16d4edc2a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6803","nonce":1190079611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6805","nonce":1190079611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6809","nonce":1190079611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6807","nonce":1190079611}]},"public_addr":"172.21.15.12:6803/1190079611","cluster_addr":"172.21.15.12:6805/1190079611","heartbeat_back_addr":"172.21.15.12:6809/1190079611","heartbeat_front_addr":"172.21.15.12:6807/1190079611","state":["exists","up"]},{"osd":1,"uuid":"416e87af-db5d-41ce-84d9-731e3e07e3c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6801","nonce":1983639525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6803","nonce":1983639525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6807","nonce":1983639525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6805","nonce":1983639525}]},"public_addr":"172.21.15.173:6801/1983639525","cluster_addr":"172.21.15.173:6803/1983639525","heartbeat_back_addr":"172.21.15.173:6807/1983639525","heartbeat_front_addr":"172.21.15.173:6805/1983639525","state":["exists","up"]},{"osd":2,"uuid":"7a559c70-4c54-431b-8baf-4136b6e76493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6801","nonce":2253997387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6803","nonce":2253997387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6807","nonce":2253997387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6805","nonce":2253997387}]},"public_addr":"172.21.15.203:6801/2253997387","cluster_addr":"172.21.15.203:6803/2253997387","heartbeat_back_addr":"172.21.15.203:6807/2253997387","heartbeat_front_addr":"172.21.15.203:6805/2253997387","state":["exists","up"]},{"osd":3,"uuid":"c41cd287-a674-486c-8025-328cd36e5ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6811","nonce":1761446905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6813","nonce":1761446905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6817","nonce":1761446905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6815","nonce":1761446905}]},"public_addr":"172.21.15.12:6811/1761446905","cluster_addr":"172.21.15.12:6813/1761446905","heartbeat_back_addr":"172.21.15.12:6817/1761446905","heartbeat_front_addr":"172.21.15.12:6815/1761446905","state":["exists","up"]},{"osd":4,"uuid":"3c48f886-0b42-4a09-b31e-abccc200bb1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6809","nonce":242958272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6811","nonce":242958272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6815","nonce":242958272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6813","nonce":242958272}]},"public_addr":"172.21.15.173:6809/242958272","cluster_addr":"172.21.15.173:6811/242958272","heartbeat_back_addr":"172.21.15.173:6815/242958272","heartbeat_front_addr":"172.21.15.173:6813/242958272","state":["exists","up"]},{"osd":5,"uuid":"4a34f06f-8805-40c0-b1ab-23407b0276d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6809","nonce":4270966092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6811","nonce":4270966092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6815","nonce":4270966092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6813","nonce":4270966092}]},"public_addr":"172.21.15.203:6809/4270966092","cluster_addr":"172.21.15.203:6811/4270966092","heartbeat_back_addr":"172.21.15.203:6815/4270966092","heartbeat_front_addr":"172.21.15.203:6813/4270966092","state":["exists","up"]},{"osd":6,"uuid":"26ddc663-5623-48c9-80d1-f8b968765d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6819","nonce":3606839295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6821","nonce":3606839295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6825","nonce":3606839295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6823","nonce":3606839295}]},"public_addr":"172.21.15.12:6819/3606839295","cluster_addr":"172.21.15.12:6821/3606839295","heartbeat_back_addr":"172.21.15.12:6825/3606839295","heartbeat_front_addr":"172.21.15.12:6823/3606839295","state":["exists","up"]},{"osd":7,"uuid":"17dc9417-71cd-4576-aa08-933212ec55e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6817","nonce":1241504346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6819","nonce":1241504346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6823","nonce":1241504346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6821","nonce":1241504346}]},"public_addr":"172.21.15.173:6817/1241504346","cluster_addr":"172.21.15.173:6819/1241504346","heartbeat_back_addr":"172.21.15.173:6823/1241504346","heartbeat_front_addr":"172.21.15.173:6821/1241504346","state":["exists","up"]},{"osd":8,"uuid":"90a2b63a-471d-43b8-ac18-bb1f16778d18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6817","nonce":1516996192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6819","nonce":1516996192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6823","nonce":1516996192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6821","nonce":1516996192}]},"public_addr":"172.21.15.203:6817/1516996192","cluster_addr":"172.21.15.203:6819/1516996192","heartbeat_back_addr":"172.21.15.203:6823/1516996192","heartbeat_front_addr":"172.21.15.203:6821/1516996192","state":["exists","up"]},{"osd":9,"uuid":"6e1ca504-40b4-49d0-b245-7946181abfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6827","nonce":1175260774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6829","nonce":1175260774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6833","nonce":1175260774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6831","nonce":1175260774}]},"public_addr":"172.21.15.12:6827/1175260774","cluster_addr":"172.21.15.12:6829/1175260774","heartbeat_back_addr":"172.21.15.12:6833/1175260774","heartbeat_front_addr":"172.21.15.12:6831/1175260774","state":["exists","up"]},{"osd":10,"uuid":"09a595ed-5614-4386-8e61-0170af66e01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6825","nonce":3303506808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6827","nonce":3303506808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6831","nonce":3303506808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6829","nonce":3303506808}]},"public_addr":"172.21.15.173:6825/3303506808","cluster_addr":"172.21.15.173:6827/3303506808","heartbeat_back_addr":"172.21.15.173:6831/3303506808","heartbeat_front_addr":"172.21.15.173:6829/3303506808","state":["exists","up"]},{"osd":11,"uuid":"9ef1d595-a977-4ba7-9c6b-d2b3295f20b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6825","nonce":2323604506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6827","nonce":2323604506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6831","nonce":2323604506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6829","nonce":2323604506}]},"public_addr":"172.21.15.203:6825/2323604506","cluster_addr":"172.21.15.203:6827/2323604506","heartbeat_back_addr":"172.21.15.203:6831/2323604506","heartbeat_front_addr":"172.21.15.203:6829/2323604506","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:18.469038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:42.752628+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:07.069493+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:41.962478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:08.363104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:33.341336+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:07.286541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:34.243302+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:59.749699+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:57:33.822913+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:00.737269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:26.599854+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/4182783120":"2024-03-24T22:51:23.113201+0000","172.21.15.12:6800/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1900938637":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/3414861652":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6801/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/2580541477":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6800/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/501058839":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1192554716":"2024-03-24T22:50:48.401223+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T22:59:35.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.714+0000 7f175b200640 1 -- 172.21.15.12:0/1828720937 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f173c0a2890 msgr2=0x7f173c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:35.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.714+0000 7f175b200640 1 --2- 172.21.15.12:0/1828720937 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f173c0a2890 0x7f173c0a4d50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1744009830 tx=0x7f174402e040 comp rx=0 tx=0).stop 2024-03-23T22:59:35.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.714+0000 7f175b200640 1 -- 172.21.15.12:0/1828720937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1754107b00 msgr2=0x7f175419ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:35.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.714+0000 7f175b200640 1 --2- 172.21.15.12:0/1828720937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1754107b00 0x7f175419ddc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f174800e9c0 tx=0x7f174800ee80 comp rx=0 tx=0).stop 2024-03-23T22:59:35.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.714+0000 7f175b200640 1 -- 172.21.15.12:0/1828720937 shutdown_connections 2024-03-23T22:59:35.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.714+0000 7f175b200640 1 --2- 172.21.15.12:0/1828720937 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f173c0a2890 0x7f173c0a4d50 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:35.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.714+0000 7f175b200640 1 --2- 172.21.15.12:0/1828720937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17541096b0 0x7f17541a5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:35.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.714+0000 7f175b200640 1 --2- 172.21.15.12:0/1828720937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1754108cf0 0x7f175419e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:35.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.714+0000 7f175b200640 1 --2- 172.21.15.12:0/1828720937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1754107b00 0x7f175419ddc0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:35.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.714+0000 7f175b200640 1 -- 172.21.15.12:0/1828720937 >> 172.21.15.12:0/1828720937 conn(0x7f1754076ab0 msgr2=0x7f175410bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:35.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.714+0000 7f175b200640 1 -- 172.21.15.12:0/1828720937 shutdown_connections 2024-03-23T22:59:35.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.714+0000 7f175b200640 1 -- 172.21.15.12:0/1828720937 wait complete. 2024-03-23T22:59:35.734 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-23T22:59:35.735 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T22:59:35.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:35 smithi173 ceph-mon[36116]: from='client.24700 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:59:35.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:35 smithi173 ceph-mon[36116]: from='client.24713 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:59:35.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:35 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1243579943' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:35.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:35 smithi203 ceph-mon[36693]: from='client.24700 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:59:35.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:35 smithi203 ceph-mon[36693]: from='client.24713 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T22:59:35.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:35 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1243579943' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T22:59:35.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.956+0000 7f2b67000640 1 -- 172.21.15.12:0/2519738333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b601028a0 msgr2=0x7f2b60102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:35.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.956+0000 7f2b67000640 1 --2- 172.21.15.12:0/2519738333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b601028a0 0x7f2b60102d20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c009900 tx=0x7f2b4c02bc30 comp rx=0 tx=0).stop 2024-03-23T22:59:35.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.956+0000 7f2b67000640 1 -- 172.21.15.12:0/2519738333 shutdown_connections 2024-03-23T22:59:35.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.956+0000 7f2b67000640 1 --2- 172.21.15.12:0/2519738333 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b60103260 0x7f2b60109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:35.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.956+0000 7f2b67000640 1 --2- 172.21.15.12:0/2519738333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b601028a0 0x7f2b60102d20 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:35.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.956+0000 7f2b67000640 1 --2- 172.21.15.12:0/2519738333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b601016b0 0x7f2b60101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:35.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.956+0000 7f2b67000640 1 -- 172.21.15.12:0/2519738333 >> 172.21.15.12:0/2519738333 conn(0x7f2b600fcc80 msgr2=0x7f2b600ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:35.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.956+0000 7f2b67000640 1 -- 172.21.15.12:0/2519738333 shutdown_connections 2024-03-23T22:59:35.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.956+0000 7f2b67000640 1 -- 172.21.15.12:0/2519738333 wait complete. 2024-03-23T22:59:35.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.957+0000 7f2b67000640 1 Processor -- start 2024-03-23T22:59:35.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.957+0000 7f2b67000640 1 -- start start 2024-03-23T22:59:35.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.958+0000 7f2b67000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b601016b0 0x7f2b60197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:35.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.958+0000 7f2b67000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b601028a0 0x7f2b60197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:35.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.958+0000 7f2b67000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b60103260 0x7f2b6019eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:35.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.958+0000 7f2b67000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6010d360 con 0x7f2b601028a0 2024-03-23T22:59:35.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.958+0000 7f2b67000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6010d1e0 con 0x7f2b60103260 2024-03-23T22:59:35.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.958+0000 7f2b67000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6010d4e0 con 0x7f2b601016b0 2024-03-23T22:59:35.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.958+0000 7f2b5fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b601016b0 0x7f2b60197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:35.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.958+0000 7f2b5f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b601028a0 0x7f2b60197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:35.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.958+0000 7f2b5fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b601016b0 0x7f2b60197910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:46676/0 (socket says 172.21.15.12:46676) 2024-03-23T22:59:35.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.958+0000 7f2b5fe00640 1 -- 172.21.15.12:0/3097595441 learned_addr learned my addr 172.21.15.12:0/3097595441 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:35.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.958+0000 7f2b65000640 1 --2- 172.21.15.12:0/3097595441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b60103260 0x7f2b6019eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:35.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.959+0000 7f2b5fe00640 1 -- 172.21.15.12:0/3097595441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b60103260 msgr2=0x7f2b6019eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:35.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.959+0000 7f2b5fe00640 1 --2- 172.21.15.12:0/3097595441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b60103260 0x7f2b6019eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:35.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.959+0000 7f2b5fe00640 1 -- 172.21.15.12:0/3097595441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b601028a0 msgr2=0x7f2b60197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:35.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.959+0000 7f2b5fe00640 1 --2- 172.21.15.12:0/3097595441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b601028a0 0x7f2b60197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:35.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.959+0000 7f2b5fe00640 1 -- 172.21.15.12:0/3097595441 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b6019f530 con 0x7f2b601016b0 2024-03-23T22:59:35.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.959+0000 7f2b65000640 1 --2- 172.21.15.12:0/3097595441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b60103260 0x7f2b6019eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T22:59:35.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.959+0000 7f2b5fe00640 1 --2- 172.21.15.12:0/3097595441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b601016b0 0x7f2b60197910 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2b5000c970 tx=0x7f2b5000ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:35.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.959+0000 7f2b5cc00640 1 -- 172.21.15.12:0/3097595441 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b50007c10 con 0x7f2b601016b0 2024-03-23T22:59:35.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.959+0000 7f2b5cc00640 1 -- 172.21.15.12:0/3097595441 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2b50007db0 con 0x7f2b601016b0 2024-03-23T22:59:35.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.959+0000 7f2b5cc00640 1 -- 172.21.15.12:0/3097595441 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5000f940 con 0x7f2b601016b0 2024-03-23T22:59:35.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.959+0000 7f2b67000640 1 -- 172.21.15.12:0/3097595441 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b6019f820 con 0x7f2b601016b0 2024-03-23T22:59:35.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.960+0000 7f2b67000640 1 -- 172.21.15.12:0/3097595441 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b6019fd40 con 0x7f2b601016b0 2024-03-23T22:59:35.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.961+0000 7f2b67000640 1 -- 172.21.15.12:0/3097595441 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b60108900 con 0x7f2b601016b0 2024-03-23T22:59:35.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.962+0000 7f2b5cc00640 1 -- 172.21.15.12:0/3097595441 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f2b5000fae0 con 0x7f2b601016b0 2024-03-23T22:59:35.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.964+0000 7f2b5cc00640 1 --2- 172.21.15.12:0/3097595441 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2b440a2890 0x7f2b440a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:35.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.964+0000 7f2b5cc00640 1 -- 172.21.15.12:0/3097595441 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f2b500cee10 con 0x7f2b601016b0 2024-03-23T22:59:35.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.964+0000 7f2b5f400640 1 --2- 172.21.15.12:0/3097595441 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2b440a2890 0x7f2b440a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:35.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.965+0000 7f2b5f400640 1 --2- 172.21.15.12:0/3097595441 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2b440a2890 0x7f2b440a4d50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c02c240 tx=0x7f2b4c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:35.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:35.968+0000 7f2b5cc00640 1 -- 172.21.15.12:0/3097595441 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2b500d2030 con 0x7f2b601016b0 2024-03-23T22:59:36.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.166+0000 7f2b67000640 1 -- 172.21.15.12:0/3097595441 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2b60061ae0 con 0x7f2b440a2890 2024-03-23T22:59:36.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.180+0000 7f2b5cc00640 1 -- 172.21.15.12:0/3097595441 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258937 (secure 0 0 0) 0x7f2b60061ae0 con 0x7f2b440a2890 2024-03-23T22:59:36.184 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:59:36.185 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-23T22:59:36.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.186+0000 7f2b67000640 1 -- 172.21.15.12:0/3097595441 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2b440a2890 msgr2=0x7f2b440a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:36.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.186+0000 7f2b67000640 1 --2- 172.21.15.12:0/3097595441 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2b440a2890 0x7f2b440a4d50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c02c240 tx=0x7f2b4c02df90 comp rx=0 tx=0).stop 2024-03-23T22:59:36.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.186+0000 7f2b67000640 1 -- 172.21.15.12:0/3097595441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b601016b0 msgr2=0x7f2b60197910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:36.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.186+0000 7f2b67000640 1 --2- 172.21.15.12:0/3097595441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b601016b0 0x7f2b60197910 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2b5000c970 tx=0x7f2b5000ce30 comp rx=0 tx=0).stop 2024-03-23T22:59:36.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.186+0000 7f2b67000640 1 -- 172.21.15.12:0/3097595441 shutdown_connections 2024-03-23T22:59:36.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.186+0000 7f2b67000640 1 --2- 172.21.15.12:0/3097595441 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2b440a2890 0x7f2b440a4d50 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:36.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.186+0000 7f2b67000640 1 --2- 172.21.15.12:0/3097595441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b60103260 0x7f2b6019eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:36.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.186+0000 7f2b67000640 1 --2- 172.21.15.12:0/3097595441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b601028a0 0x7f2b60197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:36.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.186+0000 7f2b67000640 1 --2- 172.21.15.12:0/3097595441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b601016b0 0x7f2b60197910 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:36.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.186+0000 7f2b67000640 1 -- 172.21.15.12:0/3097595441 >> 172.21.15.12:0/3097595441 conn(0x7f2b600fcc80 msgr2=0x7f2b60105ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:36.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.186+0000 7f2b67000640 1 -- 172.21.15.12:0/3097595441 shutdown_connections 2024-03-23T22:59:36.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.187+0000 7f2b67000640 1 -- 172.21.15.12:0/3097595441 wait complete. 2024-03-23T22:59:36.207 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-03-23T22:59:35.259511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594968,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":148,"num_read_kb":127,"num_write":296,"num_write_kb":2974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":236,"ondisk_log_size":236,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":256,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":368668,"kb_used_data":11184,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124573156,"statfs":{"total":1151940427776,"available":1151562911744,"internally_reserved":0,"allocated":11452416,"data_stored":7119817,"data_compressed":109252,"data_compressed_allocated":139264,"data_compressed_original":1777664,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4600,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":22,"num_read_kb":18,"num_write":115,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.091193"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.761766+0000","last_change":"2024-03-23T22:59:27.200234+0000","last_active":"2024-03-23T22:59:29.761766+0000","last_peered":"2024-03-23T22:59:29.761766+0000","last_clean":"2024-03-23T22:59:29.761766+0000","last_became_active":"2024-03-23T22:59:27.199864+0000","last_became_peered":"2024-03-23T22:59:27.199864+0000","last_unstale":"2024-03-23T22:59:29.761766+0000","last_undegraded":"2024-03-23T22:59:29.761766+0000","last_fullsized":"2024-03-23T22:59:29.761766+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:56:26.811964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.608911+0000","last_change":"2024-03-23T22:59:27.195101+0000","last_active":"2024-03-23T22:59:29.608911+0000","last_peered":"2024-03-23T22:59:29.608911+0000","last_clean":"2024-03-23T22:59:29.608911+0000","last_became_active":"2024-03-23T22:59:27.194650+0000","last_became_peered":"2024-03-23T22:59:27.194650+0000","last_unstale":"2024-03-23T22:59:29.608911+0000","last_undegraded":"2024-03-23T22:59:29.608911+0000","last_fullsized":"2024-03-23T22:59:29.608911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:20.396811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.607568+0000","last_change":"2024-03-23T22:59:27.185575+0000","last_active":"2024-03-23T22:59:29.607568+0000","last_peered":"2024-03-23T22:59:29.607568+0000","last_clean":"2024-03-23T22:59:29.607568+0000","last_became_active":"2024-03-23T22:59:27.184328+0000","last_became_peered":"2024-03-23T22:59:27.184328+0000","last_unstale":"2024-03-23T22:59:29.607568+0000","last_undegraded":"2024-03-23T22:59:29.607568+0000","last_fullsized":"2024-03-23T22:59:29.607568+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:31:46.221574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.174918+0000","last_change":"2024-03-23T22:59:27.184434+0000","last_active":"2024-03-23T22:59:28.174918+0000","last_peered":"2024-03-23T22:59:28.174918+0000","last_clean":"2024-03-23T22:59:28.174918+0000","last_became_active":"2024-03-23T22:59:27.183970+0000","last_became_peered":"2024-03-23T22:59:27.183970+0000","last_unstale":"2024-03-23T22:59:28.174918+0000","last_undegraded":"2024-03-23T22:59:28.174918+0000","last_fullsized":"2024-03-23T22:59:28.174918+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:39:05.517976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.342463+0000","last_change":"2024-03-23T22:59:27.214741+0000","last_active":"2024-03-23T22:59:28.342463+0000","last_peered":"2024-03-23T22:59:28.342463+0000","last_clean":"2024-03-23T22:59:28.342463+0000","last_became_active":"2024-03-23T22:59:27.214311+0000","last_became_peered":"2024-03-23T22:59:27.214311+0000","last_unstale":"2024-03-23T22:59:28.342463+0000","last_undegraded":"2024-03-23T22:59:28.342463+0000","last_fullsized":"2024-03-23T22:59:28.342463+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:35:38.464955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.428692+0000","last_change":"2024-03-23T22:59:27.200695+0000","last_active":"2024-03-23T22:59:28.428692+0000","last_peered":"2024-03-23T22:59:28.428692+0000","last_clean":"2024-03-23T22:59:28.428692+0000","last_became_active":"2024-03-23T22:59:27.200159+0000","last_became_peered":"2024-03-23T22:59:27.200159+0000","last_unstale":"2024-03-23T22:59:28.428692+0000","last_undegraded":"2024-03-23T22:59:28.428692+0000","last_fullsized":"2024-03-23T22:59:28.428692+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:01:15.147856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.863135+0000","last_change":"2024-03-23T22:59:27.199928+0000","last_active":"2024-03-23T22:59:28.863135+0000","last_peered":"2024-03-23T22:59:28.863135+0000","last_clean":"2024-03-23T22:59:28.863135+0000","last_became_active":"2024-03-23T22:59:27.199507+0000","last_became_peered":"2024-03-23T22:59:27.199507+0000","last_unstale":"2024-03-23T22:59:28.863135+0000","last_undegraded":"2024-03-23T22:59:28.863135+0000","last_fullsized":"2024-03-23T22:59:28.863135+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:51:19.145785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.173193+0000","last_change":"2024-03-23T22:59:27.217832+0000","last_active":"2024-03-23T22:59:28.173193+0000","last_peered":"2024-03-23T22:59:28.173193+0000","last_clean":"2024-03-23T22:59:28.173193+0000","last_became_active":"2024-03-23T22:59:27.217496+0000","last_became_peered":"2024-03-23T22:59:27.217496+0000","last_unstale":"2024-03-23T22:59:28.173193+0000","last_undegraded":"2024-03-23T22:59:28.173193+0000","last_fullsized":"2024-03-23T22:59:28.173193+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:01.778627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.606781+0000","last_change":"2024-03-23T22:59:27.190982+0000","last_active":"2024-03-23T22:59:29.606781+0000","last_peered":"2024-03-23T22:59:29.606781+0000","last_clean":"2024-03-23T22:59:29.606781+0000","last_became_active":"2024-03-23T22:59:27.190661+0000","last_became_peered":"2024-03-23T22:59:27.190661+0000","last_unstale":"2024-03-23T22:59:29.606781+0000","last_undegraded":"2024-03-23T22:59:29.606781+0000","last_fullsized":"2024-03-23T22:59:29.606781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:29.822253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.428928+0000","last_change":"2024-03-23T22:59:27.187395+0000","last_active":"2024-03-23T22:59:28.428928+0000","last_peered":"2024-03-23T22:59:28.428928+0000","last_clean":"2024-03-23T22:59:28.428928+0000","last_became_active":"2024-03-23T22:59:27.186849+0000","last_became_peered":"2024-03-23T22:59:27.186849+0000","last_unstale":"2024-03-23T22:59:28.428928+0000","last_undegraded":"2024-03-23T22:59:28.428928+0000","last_fullsized":"2024-03-23T22:59:28.428928+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:05:48.047102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.863460+0000","last_change":"2024-03-23T22:59:27.190023+0000","last_active":"2024-03-23T22:59:28.863460+0000","last_peered":"2024-03-23T22:59:28.863460+0000","last_clean":"2024-03-23T22:59:28.863460+0000","last_became_active":"2024-03-23T22:59:27.189697+0000","last_became_peered":"2024-03-23T22:59:27.189697+0000","last_unstale":"2024-03-23T22:59:28.863460+0000","last_undegraded":"2024-03-23T22:59:28.863460+0000","last_fullsized":"2024-03-23T22:59:28.863460+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:59:51.121055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.174930+0000","last_change":"2024-03-23T22:59:27.187382+0000","last_active":"2024-03-23T22:59:28.174930+0000","last_peered":"2024-03-23T22:59:28.174930+0000","last_clean":"2024-03-23T22:59:28.174930+0000","last_became_active":"2024-03-23T22:59:27.186931+0000","last_became_peered":"2024-03-23T22:59:27.186931+0000","last_unstale":"2024-03-23T22:59:28.174930+0000","last_undegraded":"2024-03-23T22:59:28.174930+0000","last_fullsized":"2024-03-23T22:59:28.174930+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:33.583123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.609279+0000","last_change":"2024-03-23T22:59:27.202577+0000","last_active":"2024-03-23T22:59:29.609279+0000","last_peered":"2024-03-23T22:59:29.609279+0000","last_clean":"2024-03-23T22:59:29.609279+0000","last_became_active":"2024-03-23T22:59:27.202122+0000","last_became_peered":"2024-03-23T22:59:27.202122+0000","last_unstale":"2024-03-23T22:59:29.609279+0000","last_undegraded":"2024-03-23T22:59:29.609279+0000","last_fullsized":"2024-03-23T22:59:29.609279+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:01:10.809635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.608450+0000","last_change":"2024-03-23T22:59:27.192915+0000","last_active":"2024-03-23T22:59:29.608450+0000","last_peered":"2024-03-23T22:59:29.608450+0000","last_clean":"2024-03-23T22:59:29.608450+0000","last_became_active":"2024-03-23T22:59:27.192107+0000","last_became_peered":"2024-03-23T22:59:27.192107+0000","last_unstale":"2024-03-23T22:59:29.608450+0000","last_undegraded":"2024-03-23T22:59:29.608450+0000","last_fullsized":"2024-03-23T22:59:29.608450+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:21:19.169211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.607622+0000","last_change":"2024-03-23T22:59:27.192156+0000","last_active":"2024-03-23T22:59:29.607622+0000","last_peered":"2024-03-23T22:59:29.607622+0000","last_clean":"2024-03-23T22:59:29.607622+0000","last_became_active":"2024-03-23T22:59:27.191594+0000","last_became_peered":"2024-03-23T22:59:27.191594+0000","last_unstale":"2024-03-23T22:59:29.607622+0000","last_undegraded":"2024-03-23T22:59:29.607622+0000","last_fullsized":"2024-03-23T22:59:29.607622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:00:45.480281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.606350+0000","last_change":"2024-03-23T22:59:27.186048+0000","last_active":"2024-03-23T22:59:29.606350+0000","last_peered":"2024-03-23T22:59:29.606350+0000","last_clean":"2024-03-23T22:59:29.606350+0000","last_became_active":"2024-03-23T22:59:27.185710+0000","last_became_peered":"2024-03-23T22:59:27.185710+0000","last_unstale":"2024-03-23T22:59:29.606350+0000","last_undegraded":"2024-03-23T22:59:29.606350+0000","last_fullsized":"2024-03-23T22:59:29.606350+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:06.792665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.761858+0000","last_change":"2024-03-23T22:59:27.206290+0000","last_active":"2024-03-23T22:59:29.761858+0000","last_peered":"2024-03-23T22:59:29.761858+0000","last_clean":"2024-03-23T22:59:29.761858+0000","last_became_active":"2024-03-23T22:59:27.205997+0000","last_became_peered":"2024-03-23T22:59:27.205997+0000","last_unstale":"2024-03-23T22:59:29.761858+0000","last_undegraded":"2024-03-23T22:59:29.761858+0000","last_fullsized":"2024-03-23T22:59:29.761858+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:38:42.006166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.447184+0000","last_change":"2024-03-23T22:59:27.203671+0000","last_active":"2024-03-23T22:59:28.447184+0000","last_peered":"2024-03-23T22:59:28.447184+0000","last_clean":"2024-03-23T22:59:28.447184+0000","last_became_active":"2024-03-23T22:59:27.203273+0000","last_became_peered":"2024-03-23T22:59:27.203273+0000","last_unstale":"2024-03-23T22:59:28.447184+0000","last_undegraded":"2024-03-23T22:59:28.447184+0000","last_fullsized":"2024-03-23T22:59:28.447184+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:48:16.018671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.606794+0000","last_change":"2024-03-23T22:59:27.210063+0000","last_active":"2024-03-23T22:59:29.606794+0000","last_peered":"2024-03-23T22:59:29.606794+0000","last_clean":"2024-03-23T22:59:29.606794+0000","last_became_active":"2024-03-23T22:59:27.209698+0000","last_became_peered":"2024-03-23T22:59:27.209698+0000","last_unstale":"2024-03-23T22:59:29.606794+0000","last_undegraded":"2024-03-23T22:59:29.606794+0000","last_fullsized":"2024-03-23T22:59:29.606794+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:19:01.408901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:31.508224+0000","last_change":"2024-03-23T22:59:26.445053+0000","last_active":"2024-03-23T22:59:31.508224+0000","last_peered":"2024-03-23T22:59:31.508224+0000","last_clean":"2024-03-23T22:59:31.508224+0000","last_became_active":"2024-03-23T22:59:26.444660+0000","last_became_peered":"2024-03-23T22:59:26.444660+0000","last_unstale":"2024-03-23T22:59:31.508224+0000","last_undegraded":"2024-03-23T22:59:31.508224+0000","last_fullsized":"2024-03-23T22:59:31.508224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:19:16.958228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.174647+0000","last_change":"2024-03-23T22:59:27.208652+0000","last_active":"2024-03-23T22:59:28.174647+0000","last_peered":"2024-03-23T22:59:28.174647+0000","last_clean":"2024-03-23T22:59:28.174647+0000","last_became_active":"2024-03-23T22:59:27.207724+0000","last_became_peered":"2024-03-23T22:59:27.207724+0000","last_unstale":"2024-03-23T22:59:28.174647+0000","last_undegraded":"2024-03-23T22:59:28.174647+0000","last_fullsized":"2024-03-23T22:59:28.174647+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:08.986439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.183896+0000","last_change":"2024-03-23T22:59:26.445371+0000","last_active":"2024-03-23T22:59:28.183896+0000","last_peered":"2024-03-23T22:59:28.183896+0000","last_clean":"2024-03-23T22:59:28.183896+0000","last_became_active":"2024-03-23T22:59:26.445016+0000","last_became_peered":"2024-03-23T22:59:26.445016+0000","last_unstale":"2024-03-23T22:59:28.183896+0000","last_undegraded":"2024-03-23T22:59:28.183896+0000","last_fullsized":"2024-03-23T22:59:28.183896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:31:44.003218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.609355+0000","last_change":"2024-03-23T22:59:27.200624+0000","last_active":"2024-03-23T22:59:29.609355+0000","last_peered":"2024-03-23T22:59:29.609355+0000","last_clean":"2024-03-23T22:59:29.609355+0000","last_became_active":"2024-03-23T22:59:27.200207+0000","last_became_peered":"2024-03-23T22:59:27.200207+0000","last_unstale":"2024-03-23T22:59:29.609355+0000","last_undegraded":"2024-03-23T22:59:29.609355+0000","last_fullsized":"2024-03-23T22:59:29.609355+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:05:31.015869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:31.499004+0000","last_change":"2024-03-23T22:59:26.431458+0000","last_active":"2024-03-23T22:59:31.499004+0000","last_peered":"2024-03-23T22:59:31.499004+0000","last_clean":"2024-03-23T22:59:31.499004+0000","last_became_active":"2024-03-23T22:59:26.430948+0000","last_became_peered":"2024-03-23T22:59:26.430948+0000","last_unstale":"2024-03-23T22:59:31.499004+0000","last_undegraded":"2024-03-23T22:59:31.499004+0000","last_fullsized":"2024-03-23T22:59:31.499004+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:08:29.777912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.606634+0000","last_change":"2024-03-23T22:59:27.204157+0000","last_active":"2024-03-23T22:59:29.606634+0000","last_peered":"2024-03-23T22:59:29.606634+0000","last_clean":"2024-03-23T22:59:29.606634+0000","last_became_active":"2024-03-23T22:59:27.203559+0000","last_became_peered":"2024-03-23T22:59:27.203559+0000","last_unstale":"2024-03-23T22:59:29.606634+0000","last_undegraded":"2024-03-23T22:59:29.606634+0000","last_fullsized":"2024-03-23T22:59:29.606634+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:43.834369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.342668+0000","last_change":"2024-03-23T22:59:26.425898+0000","last_active":"2024-03-23T22:59:28.342668+0000","last_peered":"2024-03-23T22:59:28.342668+0000","last_clean":"2024-03-23T22:59:28.342668+0000","last_became_active":"2024-03-23T22:59:26.425567+0000","last_became_peered":"2024-03-23T22:59:26.425567+0000","last_unstale":"2024-03-23T22:59:28.342668+0000","last_undegraded":"2024-03-23T22:59:28.342668+0000","last_fullsized":"2024-03-23T22:59:28.342668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:26:31.888225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.172739+0000","last_change":"2024-03-23T22:59:27.186599+0000","last_active":"2024-03-23T22:59:28.172739+0000","last_peered":"2024-03-23T22:59:28.172739+0000","last_clean":"2024-03-23T22:59:28.172739+0000","last_became_active":"2024-03-23T22:59:27.186262+0000","last_became_peered":"2024-03-23T22:59:27.186262+0000","last_unstale":"2024-03-23T22:59:28.172739+0000","last_undegraded":"2024-03-23T22:59:28.172739+0000","last_fullsized":"2024-03-23T22:59:28.172739+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:11:39.545558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:33.205152+0000","last_change":"2024-03-23T22:59:26.430096+0000","last_active":"2024-03-23T22:59:33.205152+0000","last_peered":"2024-03-23T22:59:33.205152+0000","last_clean":"2024-03-23T22:59:33.205152+0000","last_became_active":"2024-03-23T22:59:26.429740+0000","last_became_peered":"2024-03-23T22:59:26.429740+0000","last_unstale":"2024-03-23T22:59:33.205152+0000","last_undegraded":"2024-03-23T22:59:33.205152+0000","last_fullsized":"2024-03-23T22:59:33.205152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:54.831689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.342577+0000","last_change":"2024-03-23T22:59:27.202750+0000","last_active":"2024-03-23T22:59:28.342577+0000","last_peered":"2024-03-23T22:59:28.342577+0000","last_clean":"2024-03-23T22:59:28.342577+0000","last_became_active":"2024-03-23T22:59:27.202391+0000","last_became_peered":"2024-03-23T22:59:27.202391+0000","last_unstale":"2024-03-23T22:59:28.342577+0000","last_undegraded":"2024-03-23T22:59:28.342577+0000","last_fullsized":"2024-03-23T22:59:28.342577+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:27:18.370792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:31.501847+0000","last_change":"2024-03-23T22:59:26.425607+0000","last_active":"2024-03-23T22:59:31.501847+0000","last_peered":"2024-03-23T22:59:31.501847+0000","last_clean":"2024-03-23T22:59:31.501847+0000","last_became_active":"2024-03-23T22:59:26.425208+0000","last_became_peered":"2024-03-23T22:59:26.425208+0000","last_unstale":"2024-03-23T22:59:31.501847+0000","last_undegraded":"2024-03-23T22:59:31.501847+0000","last_fullsized":"2024-03-23T22:59:31.501847+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:51.071662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.172751+0000","last_change":"2024-03-23T22:59:27.215637+0000","last_active":"2024-03-23T22:59:28.172751+0000","last_peered":"2024-03-23T22:59:28.172751+0000","last_clean":"2024-03-23T22:59:28.172751+0000","last_became_active":"2024-03-23T22:59:27.215339+0000","last_became_peered":"2024-03-23T22:59:27.215339+0000","last_unstale":"2024-03-23T22:59:28.172751+0000","last_undegraded":"2024-03-23T22:59:28.172751+0000","last_fullsized":"2024-03-23T22:59:28.172751+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:25:42.097326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:31.504047+0000","last_change":"2024-03-23T22:59:26.442023+0000","last_active":"2024-03-23T22:59:31.504047+0000","last_peered":"2024-03-23T22:59:31.504047+0000","last_clean":"2024-03-23T22:59:31.504047+0000","last_became_active":"2024-03-23T22:59:26.441661+0000","last_became_peered":"2024-03-23T22:59:26.441661+0000","last_unstale":"2024-03-23T22:59:31.504047+0000","last_undegraded":"2024-03-23T22:59:31.504047+0000","last_fullsized":"2024-03-23T22:59:31.504047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:59:14.914860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.173662+0000","last_change":"2024-03-23T22:59:27.221660+0000","last_active":"2024-03-23T22:59:28.173662+0000","last_peered":"2024-03-23T22:59:28.173662+0000","last_clean":"2024-03-23T22:59:28.173662+0000","last_became_active":"2024-03-23T22:59:27.221298+0000","last_became_peered":"2024-03-23T22:59:27.221298+0000","last_unstale":"2024-03-23T22:59:28.173662+0000","last_undegraded":"2024-03-23T22:59:28.173662+0000","last_fullsized":"2024-03-23T22:59:28.173662+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:15.189932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.873475+0000","last_change":"2024-03-23T22:59:26.430147+0000","last_active":"2024-03-23T22:59:29.873475+0000","last_peered":"2024-03-23T22:59:29.873475+0000","last_clean":"2024-03-23T22:59:29.873475+0000","last_became_active":"2024-03-23T22:59:26.429843+0000","last_became_peered":"2024-03-23T22:59:26.429843+0000","last_unstale":"2024-03-23T22:59:29.873475+0000","last_undegraded":"2024-03-23T22:59:29.873475+0000","last_fullsized":"2024-03-23T22:59:29.873475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:06:45.691894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.446099+0000","last_change":"2024-03-23T22:59:27.212235+0000","last_active":"2024-03-23T22:59:28.446099+0000","last_peered":"2024-03-23T22:59:28.446099+0000","last_clean":"2024-03-23T22:59:28.446099+0000","last_became_active":"2024-03-23T22:59:27.211919+0000","last_became_peered":"2024-03-23T22:59:27.211919+0000","last_unstale":"2024-03-23T22:59:28.446099+0000","last_undegraded":"2024-03-23T22:59:28.446099+0000","last_fullsized":"2024-03-23T22:59:28.446099+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:24.752109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:30.486244+0000","last_change":"2024-03-23T22:59:26.440623+0000","last_active":"2024-03-23T22:59:30.486244+0000","last_peered":"2024-03-23T22:59:30.486244+0000","last_clean":"2024-03-23T22:59:30.486244+0000","last_became_active":"2024-03-23T22:59:26.440204+0000","last_became_peered":"2024-03-23T22:59:26.440204+0000","last_unstale":"2024-03-23T22:59:30.486244+0000","last_undegraded":"2024-03-23T22:59:30.486244+0000","last_fullsized":"2024-03-23T22:59:30.486244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:58:44.088552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.606704+0000","last_change":"2024-03-23T22:59:27.203916+0000","last_active":"2024-03-23T22:59:29.606704+0000","last_peered":"2024-03-23T22:59:29.606704+0000","last_clean":"2024-03-23T22:59:29.606704+0000","last_became_active":"2024-03-23T22:59:27.203565+0000","last_became_peered":"2024-03-23T22:59:27.203565+0000","last_unstale":"2024-03-23T22:59:29.606704+0000","last_undegraded":"2024-03-23T22:59:29.606704+0000","last_fullsized":"2024-03-23T22:59:29.606704+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:06:39.635025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:30.484141+0000","last_change":"2024-03-23T22:59:26.430080+0000","last_active":"2024-03-23T22:59:30.484141+0000","last_peered":"2024-03-23T22:59:30.484141+0000","last_clean":"2024-03-23T22:59:30.484141+0000","last_became_active":"2024-03-23T22:59:26.429618+0000","last_became_peered":"2024-03-23T22:59:26.429618+0000","last_unstale":"2024-03-23T22:59:30.484141+0000","last_undegraded":"2024-03-23T22:59:30.484141+0000","last_fullsized":"2024-03-23T22:59:30.484141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:01.963573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.609080+0000","last_change":"2024-03-23T22:59:27.190505+0000","last_active":"2024-03-23T22:59:29.609080+0000","last_peered":"2024-03-23T22:59:29.609080+0000","last_clean":"2024-03-23T22:59:29.609080+0000","last_became_active":"2024-03-23T22:59:27.190088+0000","last_became_peered":"2024-03-23T22:59:27.190088+0000","last_unstale":"2024-03-23T22:59:29.609080+0000","last_undegraded":"2024-03-23T22:59:29.609080+0000","last_fullsized":"2024-03-23T22:59:29.609080+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:29:04.255832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:30.488604+0000","last_change":"2024-03-23T22:59:26.438329+0000","last_active":"2024-03-23T22:59:30.488604+0000","last_peered":"2024-03-23T22:59:30.488604+0000","last_clean":"2024-03-23T22:59:30.488604+0000","last_became_active":"2024-03-23T22:59:26.438033+0000","last_became_peered":"2024-03-23T22:59:26.438033+0000","last_unstale":"2024-03-23T22:59:30.488604+0000","last_undegraded":"2024-03-23T22:59:30.488604+0000","last_fullsized":"2024-03-23T22:59:30.488604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:02:38.789574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.606601+0000","last_change":"2024-03-23T22:59:27.188417+0000","last_active":"2024-03-23T22:59:29.606601+0000","last_peered":"2024-03-23T22:59:29.606601+0000","last_clean":"2024-03-23T22:59:29.606601+0000","last_became_active":"2024-03-23T22:59:27.188023+0000","last_became_peered":"2024-03-23T22:59:27.188023+0000","last_unstale":"2024-03-23T22:59:29.606601+0000","last_undegraded":"2024-03-23T22:59:29.606601+0000","last_fullsized":"2024-03-23T22:59:29.606601+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:19:11.014604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:32.205798+0000","last_change":"2024-03-23T22:59:26.425963+0000","last_active":"2024-03-23T22:59:32.205798+0000","last_peered":"2024-03-23T22:59:32.205798+0000","last_clean":"2024-03-23T22:59:32.205798+0000","last_became_active":"2024-03-23T22:59:26.425640+0000","last_became_peered":"2024-03-23T22:59:26.425640+0000","last_unstale":"2024-03-23T22:59:32.205798+0000","last_undegraded":"2024-03-23T22:59:32.205798+0000","last_fullsized":"2024-03-23T22:59:32.205798+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:11:26.529034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.342661+0000","last_change":"2024-03-23T22:59:27.187254+0000","last_active":"2024-03-23T22:59:28.342661+0000","last_peered":"2024-03-23T22:59:28.342661+0000","last_clean":"2024-03-23T22:59:28.342661+0000","last_became_active":"2024-03-23T22:59:27.186849+0000","last_became_peered":"2024-03-23T22:59:27.186849+0000","last_unstale":"2024-03-23T22:59:28.342661+0000","last_undegraded":"2024-03-23T22:59:28.342661+0000","last_fullsized":"2024-03-23T22:59:28.342661+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:35:30.958565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:31.505287+0000","last_change":"2024-03-23T22:59:26.429767+0000","last_active":"2024-03-23T22:59:31.505287+0000","last_peered":"2024-03-23T22:59:31.505287+0000","last_clean":"2024-03-23T22:59:31.505287+0000","last_became_active":"2024-03-23T22:59:26.429449+0000","last_became_peered":"2024-03-23T22:59:26.429449+0000","last_unstale":"2024-03-23T22:59:31.505287+0000","last_undegraded":"2024-03-23T22:59:31.505287+0000","last_fullsized":"2024-03-23T22:59:31.505287+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:46:13.752427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.608887+0000","last_change":"2024-03-23T22:59:27.223544+0000","last_active":"2024-03-23T22:59:29.608887+0000","last_peered":"2024-03-23T22:59:29.608887+0000","last_clean":"2024-03-23T22:59:29.608887+0000","last_became_active":"2024-03-23T22:59:27.223177+0000","last_became_peered":"2024-03-23T22:59:27.223177+0000","last_unstale":"2024-03-23T22:59:29.608887+0000","last_undegraded":"2024-03-23T22:59:29.608887+0000","last_fullsized":"2024-03-23T22:59:29.608887+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:33:18.521579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:30.488651+0000","last_change":"2024-03-23T22:59:26.427781+0000","last_active":"2024-03-23T22:59:30.488651+0000","last_peered":"2024-03-23T22:59:30.488651+0000","last_clean":"2024-03-23T22:59:30.488651+0000","last_became_active":"2024-03-23T22:59:26.427458+0000","last_became_peered":"2024-03-23T22:59:26.427458+0000","last_unstale":"2024-03-23T22:59:30.488651+0000","last_undegraded":"2024-03-23T22:59:30.488651+0000","last_fullsized":"2024-03-23T22:59:30.488651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:50:33.956611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.863313+0000","last_change":"2024-03-23T22:59:27.203681+0000","last_active":"2024-03-23T22:59:28.863313+0000","last_peered":"2024-03-23T22:59:28.863313+0000","last_clean":"2024-03-23T22:59:28.863313+0000","last_became_active":"2024-03-23T22:59:27.202457+0000","last_became_peered":"2024-03-23T22:59:27.202457+0000","last_unstale":"2024-03-23T22:59:28.863313+0000","last_undegraded":"2024-03-23T22:59:28.863313+0000","last_fullsized":"2024-03-23T22:59:28.863313+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:55:46.725350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.174494+0000","last_change":"2024-03-23T22:59:26.428517+0000","last_active":"2024-03-23T22:59:28.174494+0000","last_peered":"2024-03-23T22:59:28.174494+0000","last_clean":"2024-03-23T22:59:28.174494+0000","last_became_active":"2024-03-23T22:59:26.428130+0000","last_became_peered":"2024-03-23T22:59:26.428130+0000","last_unstale":"2024-03-23T22:59:28.174494+0000","last_undegraded":"2024-03-23T22:59:28.174494+0000","last_fullsized":"2024-03-23T22:59:28.174494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:01.185776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.863148+0000","last_change":"2024-03-23T22:59:27.191428+0000","last_active":"2024-03-23T22:59:28.863148+0000","last_peered":"2024-03-23T22:59:28.863148+0000","last_clean":"2024-03-23T22:59:28.863148+0000","last_became_active":"2024-03-23T22:59:27.190958+0000","last_became_peered":"2024-03-23T22:59:27.190958+0000","last_unstale":"2024-03-23T22:59:28.863148+0000","last_undegraded":"2024-03-23T22:59:28.863148+0000","last_fullsized":"2024-03-23T22:59:28.863148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:36:43.748554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:31.504838+0000","last_change":"2024-03-23T22:59:26.442637+0000","last_active":"2024-03-23T22:59:31.504838+0000","last_peered":"2024-03-23T22:59:31.504838+0000","last_clean":"2024-03-23T22:59:31.504838+0000","last_became_active":"2024-03-23T22:59:26.442267+0000","last_became_peered":"2024-03-23T22:59:26.442267+0000","last_unstale":"2024-03-23T22:59:31.504838+0000","last_undegraded":"2024-03-23T22:59:31.504838+0000","last_fullsized":"2024-03-23T22:59:31.504838+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:49.975048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.172240+0000","last_change":"2024-03-23T22:59:27.187812+0000","last_active":"2024-03-23T22:59:28.172240+0000","last_peered":"2024-03-23T22:59:28.172240+0000","last_clean":"2024-03-23T22:59:28.172240+0000","last_became_active":"2024-03-23T22:59:27.187423+0000","last_became_peered":"2024-03-23T22:59:27.187423+0000","last_unstale":"2024-03-23T22:59:28.172240+0000","last_undegraded":"2024-03-23T22:59:28.172240+0000","last_fullsized":"2024-03-23T22:59:28.172240+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:12:16.427695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.342924+0000","last_change":"2024-03-23T22:59:26.423875+0000","last_active":"2024-03-23T22:59:28.342924+0000","last_peered":"2024-03-23T22:59:28.342924+0000","last_clean":"2024-03-23T22:59:28.342924+0000","last_became_active":"2024-03-23T22:59:26.423513+0000","last_became_peered":"2024-03-23T22:59:26.423513+0000","last_unstale":"2024-03-23T22:59:28.342924+0000","last_undegraded":"2024-03-23T22:59:28.342924+0000","last_fullsized":"2024-03-23T22:59:28.342924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:43.173365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.761656+0000","last_change":"2024-03-23T22:59:27.191454+0000","last_active":"2024-03-23T22:59:29.761656+0000","last_peered":"2024-03-23T22:59:29.761656+0000","last_clean":"2024-03-23T22:59:29.761656+0000","last_became_active":"2024-03-23T22:59:27.190927+0000","last_became_peered":"2024-03-23T22:59:27.190927+0000","last_unstale":"2024-03-23T22:59:29.761656+0000","last_undegraded":"2024-03-23T22:59:29.761656+0000","last_fullsized":"2024-03-23T22:59:29.761656+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:00.333702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.446810+0000","last_change":"2024-03-23T22:59:26.443238+0000","last_active":"2024-03-23T22:59:28.446810+0000","last_peered":"2024-03-23T22:59:28.446810+0000","last_clean":"2024-03-23T22:59:28.446810+0000","last_became_active":"2024-03-23T22:59:26.442834+0000","last_became_peered":"2024-03-23T22:59:26.442834+0000","last_unstale":"2024-03-23T22:59:28.446810+0000","last_undegraded":"2024-03-23T22:59:28.446810+0000","last_fullsized":"2024-03-23T22:59:28.446810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:11:21.072836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.428955+0000","last_change":"2024-03-23T22:59:27.212864+0000","last_active":"2024-03-23T22:59:28.428955+0000","last_peered":"2024-03-23T22:59:28.428955+0000","last_clean":"2024-03-23T22:59:28.428955+0000","last_became_active":"2024-03-23T22:59:27.212455+0000","last_became_peered":"2024-03-23T22:59:27.212455+0000","last_unstale":"2024-03-23T22:59:28.428955+0000","last_undegraded":"2024-03-23T22:59:28.428955+0000","last_fullsized":"2024-03-23T22:59:28.428955+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:25:59.122531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.863137+0000","last_change":"2024-03-23T22:59:26.432838+0000","last_active":"2024-03-23T22:59:28.863137+0000","last_peered":"2024-03-23T22:59:28.863137+0000","last_clean":"2024-03-23T22:59:28.863137+0000","last_became_active":"2024-03-23T22:59:26.432544+0000","last_became_peered":"2024-03-23T22:59:26.432544+0000","last_unstale":"2024-03-23T22:59:28.863137+0000","last_undegraded":"2024-03-23T22:59:28.863137+0000","last_fullsized":"2024-03-23T22:59:28.863137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:32:52.121569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.428691+0000","last_change":"2024-03-23T22:59:27.192870+0000","last_active":"2024-03-23T22:59:28.428691+0000","last_peered":"2024-03-23T22:59:28.428691+0000","last_clean":"2024-03-23T22:59:28.428691+0000","last_became_active":"2024-03-23T22:59:27.192423+0000","last_became_peered":"2024-03-23T22:59:27.192423+0000","last_unstale":"2024-03-23T22:59:28.428691+0000","last_undegraded":"2024-03-23T22:59:28.428691+0000","last_fullsized":"2024-03-23T22:59:28.428691+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:27:52.709256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.172451+0000","last_change":"2024-03-23T22:59:27.185585+0000","last_active":"2024-03-23T22:59:28.172451+0000","last_peered":"2024-03-23T22:59:28.172451+0000","last_clean":"2024-03-23T22:59:28.172451+0000","last_became_active":"2024-03-23T22:59:27.185278+0000","last_became_peered":"2024-03-23T22:59:27.185278+0000","last_unstale":"2024-03-23T22:59:28.172451+0000","last_undegraded":"2024-03-23T22:59:28.172451+0000","last_fullsized":"2024-03-23T22:59:28.172451+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:45:09.759771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:33.209053+0000","last_change":"2024-03-23T22:59:26.450679+0000","last_active":"2024-03-23T22:59:33.209053+0000","last_peered":"2024-03-23T22:59:33.209053+0000","last_clean":"2024-03-23T22:59:33.209053+0000","last_became_active":"2024-03-23T22:59:26.450234+0000","last_became_peered":"2024-03-23T22:59:26.450234+0000","last_unstale":"2024-03-23T22:59:33.209053+0000","last_undegraded":"2024-03-23T22:59:33.209053+0000","last_fullsized":"2024-03-23T22:59:33.209053+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:52:25.869194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.173444+0000","last_change":"2024-03-23T22:59:27.220171+0000","last_active":"2024-03-23T22:59:28.173444+0000","last_peered":"2024-03-23T22:59:28.173444+0000","last_clean":"2024-03-23T22:59:28.173444+0000","last_became_active":"2024-03-23T22:59:27.219787+0000","last_became_peered":"2024-03-23T22:59:27.219787+0000","last_unstale":"2024-03-23T22:59:28.173444+0000","last_undegraded":"2024-03-23T22:59:28.173444+0000","last_fullsized":"2024-03-23T22:59:28.173444+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:51:52.634264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:30.487301+0000","last_change":"2024-03-23T22:59:26.440205+0000","last_active":"2024-03-23T22:59:30.487301+0000","last_peered":"2024-03-23T22:59:30.487301+0000","last_clean":"2024-03-23T22:59:30.487301+0000","last_became_active":"2024-03-23T22:59:26.439752+0000","last_became_peered":"2024-03-23T22:59:26.439752+0000","last_unstale":"2024-03-23T22:59:30.487301+0000","last_undegraded":"2024-03-23T22:59:30.487301+0000","last_fullsized":"2024-03-23T22:59:30.487301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:55:25.661887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":340,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.863034+0000","last_change":"2024-03-23T22:58:06.171317+0000","last_active":"2024-03-23T22:59:28.863034+0000","last_peered":"2024-03-23T22:59:28.863034+0000","last_clean":"2024-03-23T22:59:28.863034+0000","last_became_active":"2024-03-23T22:58:06.170816+0000","last_became_peered":"2024-03-23T22:58:06.170816+0000","last_unstale":"2024-03-23T22:59:28.863034+0000","last_undegraded":"2024-03-23T22:59:28.863034+0000","last_fullsized":"2024-03-23T22:59:28.863034+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:54:11.991284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:54:11.991284+0000","last_clean_scrub_stamp":"2024-03-23T22:54:11.991284+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:20:55.980613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.607348+0000","last_change":"2024-03-23T22:59:27.200348+0000","last_active":"2024-03-23T22:59:29.607348+0000","last_peered":"2024-03-23T22:59:29.607348+0000","last_clean":"2024-03-23T22:59:29.607348+0000","last_became_active":"2024-03-23T22:59:27.199979+0000","last_became_peered":"2024-03-23T22:59:27.199979+0000","last_unstale":"2024-03-23T22:59:29.607348+0000","last_undegraded":"2024-03-23T22:59:29.607348+0000","last_fullsized":"2024-03-23T22:59:29.607348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:31:19.258890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.863267+0000","last_change":"2024-03-23T22:59:26.431837+0000","last_active":"2024-03-23T22:59:28.863267+0000","last_peered":"2024-03-23T22:59:28.863267+0000","last_clean":"2024-03-23T22:59:28.863267+0000","last_became_active":"2024-03-23T22:59:26.431580+0000","last_became_peered":"2024-03-23T22:59:26.431580+0000","last_unstale":"2024-03-23T22:59:28.863267+0000","last_undegraded":"2024-03-23T22:59:28.863267+0000","last_fullsized":"2024-03-23T22:59:28.863267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:49:53.316952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:30.487730+0000","last_change":"2024-03-23T22:59:26.442878+0000","last_active":"2024-03-23T22:59:30.487730+0000","last_peered":"2024-03-23T22:59:30.487730+0000","last_clean":"2024-03-23T22:59:30.487730+0000","last_became_active":"2024-03-23T22:59:26.442303+0000","last_became_peered":"2024-03-23T22:59:26.442303+0000","last_unstale":"2024-03-23T22:59:30.487730+0000","last_undegraded":"2024-03-23T22:59:30.487730+0000","last_fullsized":"2024-03-23T22:59:30.487730+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:51.717553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.608814+0000","last_change":"2024-03-23T22:59:27.212550+0000","last_active":"2024-03-23T22:59:29.608814+0000","last_peered":"2024-03-23T22:59:29.608814+0000","last_clean":"2024-03-23T22:59:29.608814+0000","last_became_active":"2024-03-23T22:59:27.212105+0000","last_became_peered":"2024-03-23T22:59:27.212105+0000","last_unstale":"2024-03-23T22:59:29.608814+0000","last_undegraded":"2024-03-23T22:59:29.608814+0000","last_fullsized":"2024-03-23T22:59:29.608814+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:23:37.374081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:31.503691+0000","last_change":"2024-03-23T22:59:26.452146+0000","last_active":"2024-03-23T22:59:31.503691+0000","last_peered":"2024-03-23T22:59:31.503691+0000","last_clean":"2024-03-23T22:59:31.503691+0000","last_became_active":"2024-03-23T22:59:26.450398+0000","last_became_peered":"2024-03-23T22:59:26.450398+0000","last_unstale":"2024-03-23T22:59:31.503691+0000","last_undegraded":"2024-03-23T22:59:31.503691+0000","last_fullsized":"2024-03-23T22:59:31.503691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:55:47.085457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.174976+0000","last_change":"2024-03-23T22:59:27.209690+0000","last_active":"2024-03-23T22:59:28.174976+0000","last_peered":"2024-03-23T22:59:28.174976+0000","last_clean":"2024-03-23T22:59:28.174976+0000","last_became_active":"2024-03-23T22:59:27.209187+0000","last_became_peered":"2024-03-23T22:59:27.209187+0000","last_unstale":"2024-03-23T22:59:28.174976+0000","last_undegraded":"2024-03-23T22:59:28.174976+0000","last_fullsized":"2024-03-23T22:59:28.174976+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:19:45.315459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:30.485058+0000","last_change":"2024-03-23T22:59:26.434958+0000","last_active":"2024-03-23T22:59:30.485058+0000","last_peered":"2024-03-23T22:59:30.485058+0000","last_clean":"2024-03-23T22:59:30.485058+0000","last_became_active":"2024-03-23T22:59:26.434611+0000","last_became_peered":"2024-03-23T22:59:26.434611+0000","last_unstale":"2024-03-23T22:59:30.485058+0000","last_undegraded":"2024-03-23T22:59:30.485058+0000","last_fullsized":"2024-03-23T22:59:30.485058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:42:18.201885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.609415+0000","last_change":"2024-03-23T22:59:27.211700+0000","last_active":"2024-03-23T22:59:29.609415+0000","last_peered":"2024-03-23T22:59:29.609415+0000","last_clean":"2024-03-23T22:59:29.609415+0000","last_became_active":"2024-03-23T22:59:27.211248+0000","last_became_peered":"2024-03-23T22:59:27.211248+0000","last_unstale":"2024-03-23T22:59:29.609415+0000","last_undegraded":"2024-03-23T22:59:29.609415+0000","last_fullsized":"2024-03-23T22:59:29.609415+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:49.868397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.606220+0000","last_change":"2024-03-23T22:59:26.442625+0000","last_active":"2024-03-23T22:59:29.606220+0000","last_peered":"2024-03-23T22:59:29.606220+0000","last_clean":"2024-03-23T22:59:29.606220+0000","last_became_active":"2024-03-23T22:59:26.442312+0000","last_became_peered":"2024-03-23T22:59:26.442312+0000","last_unstale":"2024-03-23T22:59:29.606220+0000","last_undegraded":"2024-03-23T22:59:29.606220+0000","last_fullsized":"2024-03-23T22:59:29.606220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:27:38.808784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.342409+0000","last_change":"2024-03-23T22:59:27.187363+0000","last_active":"2024-03-23T22:59:28.342409+0000","last_peered":"2024-03-23T22:59:28.342409+0000","last_clean":"2024-03-23T22:59:28.342409+0000","last_became_active":"2024-03-23T22:59:27.187007+0000","last_became_peered":"2024-03-23T22:59:27.187007+0000","last_unstale":"2024-03-23T22:59:28.342409+0000","last_undegraded":"2024-03-23T22:59:28.342409+0000","last_fullsized":"2024-03-23T22:59:28.342409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:41:37.041254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:30.492845+0000","last_change":"2024-03-23T22:59:26.445753+0000","last_active":"2024-03-23T22:59:30.492845+0000","last_peered":"2024-03-23T22:59:30.492845+0000","last_clean":"2024-03-23T22:59:30.492845+0000","last_became_active":"2024-03-23T22:59:26.445167+0000","last_became_peered":"2024-03-23T22:59:26.445167+0000","last_unstale":"2024-03-23T22:59:30.492845+0000","last_undegraded":"2024-03-23T22:59:30.492845+0000","last_fullsized":"2024-03-23T22:59:30.492845+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:46:46.058523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.761371+0000","last_change":"2024-03-23T22:59:27.203855+0000","last_active":"2024-03-23T22:59:29.761371+0000","last_peered":"2024-03-23T22:59:29.761371+0000","last_clean":"2024-03-23T22:59:29.761371+0000","last_became_active":"2024-03-23T22:59:27.203438+0000","last_became_peered":"2024-03-23T22:59:27.203438+0000","last_unstale":"2024-03-23T22:59:29.761371+0000","last_undegraded":"2024-03-23T22:59:29.761371+0000","last_fullsized":"2024-03-23T22:59:29.761371+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:28.725743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.994795+0000","last_change":"2024-03-23T22:59:26.439002+0000","last_active":"2024-03-23T22:59:29.994795+0000","last_peered":"2024-03-23T22:59:29.994795+0000","last_clean":"2024-03-23T22:59:29.994795+0000","last_became_active":"2024-03-23T22:59:26.438634+0000","last_became_peered":"2024-03-23T22:59:26.438634+0000","last_unstale":"2024-03-23T22:59:29.994795+0000","last_undegraded":"2024-03-23T22:59:29.994795+0000","last_fullsized":"2024-03-23T22:59:29.994795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:25:02.988163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.173987+0000","last_change":"2024-03-23T22:59:27.217314+0000","last_active":"2024-03-23T22:59:28.173987+0000","last_peered":"2024-03-23T22:59:28.173987+0000","last_clean":"2024-03-23T22:59:28.173987+0000","last_became_active":"2024-03-23T22:59:27.216790+0000","last_became_peered":"2024-03-23T22:59:27.216790+0000","last_unstale":"2024-03-23T22:59:28.173987+0000","last_undegraded":"2024-03-23T22:59:28.173987+0000","last_fullsized":"2024-03-23T22:59:28.173987+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:47:13.902236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:30.485663+0000","last_change":"2024-03-23T22:59:26.431453+0000","last_active":"2024-03-23T22:59:30.485663+0000","last_peered":"2024-03-23T22:59:30.485663+0000","last_clean":"2024-03-23T22:59:30.485663+0000","last_became_active":"2024-03-23T22:59:26.429731+0000","last_became_peered":"2024-03-23T22:59:26.429731+0000","last_unstale":"2024-03-23T22:59:30.485663+0000","last_undegraded":"2024-03-23T22:59:30.485663+0000","last_fullsized":"2024-03-23T22:59:30.485663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:33:47.717064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.863310+0000","last_change":"2024-03-23T22:59:27.203097+0000","last_active":"2024-03-23T22:59:28.863310+0000","last_peered":"2024-03-23T22:59:28.863310+0000","last_clean":"2024-03-23T22:59:28.863310+0000","last_became_active":"2024-03-23T22:59:27.202484+0000","last_became_peered":"2024-03-23T22:59:27.202484+0000","last_unstale":"2024-03-23T22:59:28.863310+0000","last_undegraded":"2024-03-23T22:59:28.863310+0000","last_fullsized":"2024-03-23T22:59:28.863310+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:07.829866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:31.504672+0000","last_change":"2024-03-23T22:59:26.442815+0000","last_active":"2024-03-23T22:59:31.504672+0000","last_peered":"2024-03-23T22:59:31.504672+0000","last_clean":"2024-03-23T22:59:31.504672+0000","last_became_active":"2024-03-23T22:59:26.442401+0000","last_became_peered":"2024-03-23T22:59:26.442401+0000","last_unstale":"2024-03-23T22:59:31.504672+0000","last_undegraded":"2024-03-23T22:59:31.504672+0000","last_fullsized":"2024-03-23T22:59:31.504672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:44:06.208357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.863535+0000","last_change":"2024-03-23T22:59:27.189244+0000","last_active":"2024-03-23T22:59:28.863535+0000","last_peered":"2024-03-23T22:59:28.863535+0000","last_clean":"2024-03-23T22:59:28.863535+0000","last_became_active":"2024-03-23T22:59:27.188629+0000","last_became_peered":"2024-03-23T22:59:27.188629+0000","last_unstale":"2024-03-23T22:59:28.863535+0000","last_undegraded":"2024-03-23T22:59:28.863535+0000","last_fullsized":"2024-03-23T22:59:28.863535+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:47:26.742204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.873569+0000","last_change":"2024-03-23T22:59:26.434326+0000","last_active":"2024-03-23T22:59:29.873569+0000","last_peered":"2024-03-23T22:59:29.873569+0000","last_clean":"2024-03-23T22:59:29.873569+0000","last_became_active":"2024-03-23T22:59:26.433878+0000","last_became_peered":"2024-03-23T22:59:26.433878+0000","last_unstale":"2024-03-23T22:59:29.873569+0000","last_undegraded":"2024-03-23T22:59:29.873569+0000","last_fullsized":"2024-03-23T22:59:29.873569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:08.580572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.863451+0000","last_change":"2024-03-23T22:59:27.212909+0000","last_active":"2024-03-23T22:59:28.863451+0000","last_peered":"2024-03-23T22:59:28.863451+0000","last_clean":"2024-03-23T22:59:28.863451+0000","last_became_active":"2024-03-23T22:59:27.212504+0000","last_became_peered":"2024-03-23T22:59:27.212504+0000","last_unstale":"2024-03-23T22:59:28.863451+0000","last_undegraded":"2024-03-23T22:59:28.863451+0000","last_fullsized":"2024-03-23T22:59:28.863451+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:09.238272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.608248+0000","last_change":"2024-03-23T22:59:26.452577+0000","last_active":"2024-03-23T22:59:29.608248+0000","last_peered":"2024-03-23T22:59:29.608248+0000","last_clean":"2024-03-23T22:59:29.608248+0000","last_became_active":"2024-03-23T22:59:26.452107+0000","last_became_peered":"2024-03-23T22:59:26.452107+0000","last_unstale":"2024-03-23T22:59:29.608248+0000","last_undegraded":"2024-03-23T22:59:29.608248+0000","last_fullsized":"2024-03-23T22:59:29.608248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:32:18.472334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.863589+0000","last_change":"2024-03-23T22:59:27.188539+0000","last_active":"2024-03-23T22:59:28.863589+0000","last_peered":"2024-03-23T22:59:28.863589+0000","last_clean":"2024-03-23T22:59:28.863589+0000","last_became_active":"2024-03-23T22:59:27.188184+0000","last_became_peered":"2024-03-23T22:59:27.188184+0000","last_unstale":"2024-03-23T22:59:28.863589+0000","last_undegraded":"2024-03-23T22:59:28.863589+0000","last_fullsized":"2024-03-23T22:59:28.863589+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:37:59.756492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.761432+0000","last_change":"2024-03-23T22:59:27.192083+0000","last_active":"2024-03-23T22:59:29.761432+0000","last_peered":"2024-03-23T22:59:29.761432+0000","last_clean":"2024-03-23T22:59:29.761432+0000","last_became_active":"2024-03-23T22:59:27.191709+0000","last_became_peered":"2024-03-23T22:59:27.191709+0000","last_unstale":"2024-03-23T22:59:29.761432+0000","last_undegraded":"2024-03-23T22:59:29.761432+0000","last_fullsized":"2024-03-23T22:59:29.761432+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:18:42.905177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.606521+0000","last_change":"2024-03-23T22:59:27.186315+0000","last_active":"2024-03-23T22:59:29.606521+0000","last_peered":"2024-03-23T22:59:29.606521+0000","last_clean":"2024-03-23T22:59:29.606521+0000","last_became_active":"2024-03-23T22:59:27.185960+0000","last_became_peered":"2024-03-23T22:59:27.185960+0000","last_unstale":"2024-03-23T22:59:29.606521+0000","last_undegraded":"2024-03-23T22:59:29.606521+0000","last_fullsized":"2024-03-23T22:59:29.606521+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:57:07.739725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.606619+0000","last_change":"2024-03-23T22:59:27.221696+0000","last_active":"2024-03-23T22:59:29.606619+0000","last_peered":"2024-03-23T22:59:29.606619+0000","last_clean":"2024-03-23T22:59:29.606619+0000","last_became_active":"2024-03-23T22:59:27.221400+0000","last_became_peered":"2024-03-23T22:59:27.221400+0000","last_unstale":"2024-03-23T22:59:29.606619+0000","last_undegraded":"2024-03-23T22:59:29.606619+0000","last_fullsized":"2024-03-23T22:59:29.606619+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:17.905737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.608462+0000","last_change":"2024-03-23T22:59:27.206286+0000","last_active":"2024-03-23T22:59:29.608462+0000","last_peered":"2024-03-23T22:59:29.608462+0000","last_clean":"2024-03-23T22:59:29.608462+0000","last_became_active":"2024-03-23T22:59:27.205854+0000","last_became_peered":"2024-03-23T22:59:27.205854+0000","last_unstale":"2024-03-23T22:59:29.608462+0000","last_undegraded":"2024-03-23T22:59:29.608462+0000","last_fullsized":"2024-03-23T22:59:29.608462+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:00:45.216211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.609037+0000","last_change":"2024-03-23T22:59:27.211376+0000","last_active":"2024-03-23T22:59:29.609037+0000","last_peered":"2024-03-23T22:59:29.609037+0000","last_clean":"2024-03-23T22:59:29.609037+0000","last_became_active":"2024-03-23T22:59:27.211061+0000","last_became_peered":"2024-03-23T22:59:27.211061+0000","last_unstale":"2024-03-23T22:59:29.609037+0000","last_undegraded":"2024-03-23T22:59:29.609037+0000","last_fullsized":"2024-03-23T22:59:29.609037+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:53:25.776927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.761562+0000","last_change":"2024-03-23T22:59:27.191028+0000","last_active":"2024-03-23T22:59:29.761562+0000","last_peered":"2024-03-23T22:59:29.761562+0000","last_clean":"2024-03-23T22:59:29.761562+0000","last_became_active":"2024-03-23T22:59:27.190634+0000","last_became_peered":"2024-03-23T22:59:27.190634+0000","last_unstale":"2024-03-23T22:59:29.761562+0000","last_undegraded":"2024-03-23T22:59:29.761562+0000","last_fullsized":"2024-03-23T22:59:29.761562+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:50:52.110636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.606910+0000","last_change":"2024-03-23T22:59:27.183235+0000","last_active":"2024-03-23T22:59:29.606910+0000","last_peered":"2024-03-23T22:59:29.606910+0000","last_clean":"2024-03-23T22:59:29.606910+0000","last_became_active":"2024-03-23T22:59:27.181682+0000","last_became_peered":"2024-03-23T22:59:27.181682+0000","last_unstale":"2024-03-23T22:59:29.606910+0000","last_undegraded":"2024-03-23T22:59:29.606910+0000","last_fullsized":"2024-03-23T22:59:29.606910+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:33:26.002324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.174878+0000","last_change":"2024-03-23T22:59:27.212788+0000","last_active":"2024-03-23T22:59:28.174878+0000","last_peered":"2024-03-23T22:59:28.174878+0000","last_clean":"2024-03-23T22:59:28.174878+0000","last_became_active":"2024-03-23T22:59:27.211101+0000","last_became_peered":"2024-03-23T22:59:27.211101+0000","last_unstale":"2024-03-23T22:59:28.174878+0000","last_undegraded":"2024-03-23T22:59:28.174878+0000","last_fullsized":"2024-03-23T22:59:28.174878+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:54:59.599903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.172349+0000","last_change":"2024-03-23T22:59:27.215230+0000","last_active":"2024-03-23T22:59:28.172349+0000","last_peered":"2024-03-23T22:59:28.172349+0000","last_clean":"2024-03-23T22:59:28.172349+0000","last_became_active":"2024-03-23T22:59:27.214881+0000","last_became_peered":"2024-03-23T22:59:27.214881+0000","last_unstale":"2024-03-23T22:59:28.172349+0000","last_undegraded":"2024-03-23T22:59:28.172349+0000","last_fullsized":"2024-03-23T22:59:28.172349+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:10:40.428481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.173247+0000","last_change":"2024-03-23T22:59:27.213454+0000","last_active":"2024-03-23T22:59:28.173247+0000","last_peered":"2024-03-23T22:59:28.173247+0000","last_clean":"2024-03-23T22:59:28.173247+0000","last_became_active":"2024-03-23T22:59:27.212684+0000","last_became_peered":"2024-03-23T22:59:27.212684+0000","last_unstale":"2024-03-23T22:59:28.173247+0000","last_undegraded":"2024-03-23T22:59:28.173247+0000","last_fullsized":"2024-03-23T22:59:28.173247+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:48:51.584892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.606558+0000","last_change":"2024-03-23T22:59:27.220414+0000","last_active":"2024-03-23T22:59:29.606558+0000","last_peered":"2024-03-23T22:59:29.606558+0000","last_clean":"2024-03-23T22:59:29.606558+0000","last_became_active":"2024-03-23T22:59:27.220020+0000","last_became_peered":"2024-03-23T22:59:27.220020+0000","last_unstale":"2024-03-23T22:59:29.606558+0000","last_undegraded":"2024-03-23T22:59:29.606558+0000","last_fullsized":"2024-03-23T22:59:29.606558+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:35:48.890004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.609276+0000","last_change":"2024-03-23T22:59:27.215648+0000","last_active":"2024-03-23T22:59:29.609276+0000","last_peered":"2024-03-23T22:59:29.609276+0000","last_clean":"2024-03-23T22:59:29.609276+0000","last_became_active":"2024-03-23T22:59:27.212252+0000","last_became_peered":"2024-03-23T22:59:27.212252+0000","last_unstale":"2024-03-23T22:59:29.609276+0000","last_undegraded":"2024-03-23T22:59:29.609276+0000","last_fullsized":"2024-03-23T22:59:29.609276+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:08:38.956508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.761439+0000","last_change":"2024-03-23T22:59:27.190527+0000","last_active":"2024-03-23T22:59:29.761439+0000","last_peered":"2024-03-23T22:59:29.761439+0000","last_clean":"2024-03-23T22:59:29.761439+0000","last_became_active":"2024-03-23T22:59:27.190026+0000","last_became_peered":"2024-03-23T22:59:27.190026+0000","last_unstale":"2024-03-23T22:59:29.761439+0000","last_undegraded":"2024-03-23T22:59:29.761439+0000","last_fullsized":"2024-03-23T22:59:29.761439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:31:24.663584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247269,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4600,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":22,"num_read_kb":18,"num_write":115,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":249856,"data_stored":50253,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350991,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":442154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612627,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":1035130,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776153,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":442356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037792,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":442310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299428,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":1033144,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462955,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":929792,"data_stored":441779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":933888,"data_stored":443184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738444,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":925696,"data_stored":441662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411395,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":991232,"data_stored":481115,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149756,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":441856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888117,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":925696,"data_stored":441594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724593,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":1033533,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34152,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34152,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":39543,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T22:59:36.211 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T22:59:36.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.480+0000 7ffa69800640 1 -- 172.21.15.12:0/2688405593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa641048e0 msgr2=0x7ffa64104d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:36.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.480+0000 7ffa69800640 1 --2- 172.21.15.12:0/2688405593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa641048e0 0x7ffa64104d60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c009900 tx=0x7ffa4c02bc30 comp rx=0 tx=0).stop 2024-03-23T22:59:36.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.480+0000 7ffa69800640 1 -- 172.21.15.12:0/2688405593 shutdown_connections 2024-03-23T22:59:36.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.480+0000 7ffa69800640 1 --2- 172.21.15.12:0/2688405593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa641052a0 0x7ffa6410bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:36.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.480+0000 7ffa69800640 1 --2- 172.21.15.12:0/2688405593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa641048e0 0x7ffa64104d60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:36.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.480+0000 7ffa69800640 1 --2- 172.21.15.12:0/2688405593 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa641036f0 0x7ffa64103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:36.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.480+0000 7ffa69800640 1 -- 172.21.15.12:0/2688405593 >> 172.21.15.12:0/2688405593 conn(0x7ffa640fec80 msgr2=0x7ffa641010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:36.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.480+0000 7ffa69800640 1 -- 172.21.15.12:0/2688405593 shutdown_connections 2024-03-23T22:59:36.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.480+0000 7ffa69800640 1 -- 172.21.15.12:0/2688405593 wait complete. 2024-03-23T22:59:36.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.481+0000 7ffa69800640 1 Processor -- start 2024-03-23T22:59:36.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.481+0000 7ffa69800640 1 -- start start 2024-03-23T22:59:36.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.482+0000 7ffa69800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa641036f0 0x7ffa641999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:36.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.482+0000 7ffa69800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa641048e0 0x7ffa64199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:36.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.482+0000 7ffa69800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa641052a0 0x7ffa641a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:36.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.482+0000 7ffa69800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6410f3a0 con 0x7ffa641052a0 2024-03-23T22:59:36.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.482+0000 7ffa69800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6410f220 con 0x7ffa641036f0 2024-03-23T22:59:36.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.482+0000 7ffa69800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6410f520 con 0x7ffa641048e0 2024-03-23T22:59:36.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.482+0000 7ffa63e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa641036f0 0x7ffa641999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:36.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.482+0000 7ffa63e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa641036f0 0x7ffa641999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:46438/0 (socket says 172.21.15.12:46438) 2024-03-23T22:59:36.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.482+0000 7ffa5be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa641048e0 0x7ffa64199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:36.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.482+0000 7ffa63e00640 1 -- 172.21.15.12:0/1527333008 learned_addr learned my addr 172.21.15.12:0/1527333008 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:36.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.482+0000 7ffa68e00640 1 --2- 172.21.15.12:0/1527333008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa641052a0 0x7ffa641a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:36.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.482+0000 7ffa63e00640 1 -- 172.21.15.12:0/1527333008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa641048e0 msgr2=0x7ffa64199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:36.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.482+0000 7ffa63e00640 1 --2- 172.21.15.12:0/1527333008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa641048e0 0x7ffa64199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:36.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.482+0000 7ffa63e00640 1 -- 172.21.15.12:0/1527333008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa641052a0 msgr2=0x7ffa641a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:36.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.482+0000 7ffa63e00640 1 --2- 172.21.15.12:0/1527333008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa641052a0 0x7ffa641a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:36.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.482+0000 7ffa63e00640 1 -- 172.21.15.12:0/1527333008 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa641a1600 con 0x7ffa641036f0 2024-03-23T22:59:36.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.482+0000 7ffa5be00640 1 --2- 172.21.15.12:0/1527333008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa641048e0 0x7ffa64199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T22:59:36.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.483+0000 7ffa63e00640 1 --2- 172.21.15.12:0/1527333008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa641036f0 0x7ffa641999e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ffa5400e9c0 tx=0x7ffa5400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:36.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.483+0000 7ffa61600640 1 -- 172.21.15.12:0/1527333008 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5400cd80 con 0x7ffa641036f0 2024-03-23T22:59:36.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.483+0000 7ffa61600640 1 -- 172.21.15.12:0/1527333008 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa540035f0 con 0x7ffa641036f0 2024-03-23T22:59:36.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.483+0000 7ffa61600640 1 -- 172.21.15.12:0/1527333008 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa54010690 con 0x7ffa641036f0 2024-03-23T22:59:36.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.483+0000 7ffa69800640 1 -- 172.21.15.12:0/1527333008 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa641a18f0 con 0x7ffa641036f0 2024-03-23T22:59:36.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.483+0000 7ffa69800640 1 -- 172.21.15.12:0/1527333008 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa641a1dc0 con 0x7ffa641036f0 2024-03-23T22:59:36.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.484+0000 7ffa69800640 1 -- 172.21.15.12:0/1527333008 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa6410aa10 con 0x7ffa641036f0 2024-03-23T22:59:36.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.486+0000 7ffa61600640 1 -- 172.21.15.12:0/1527333008 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7ffa540037a0 con 0x7ffa641036f0 2024-03-23T22:59:36.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.487+0000 7ffa61600640 1 --2- 172.21.15.12:0/1527333008 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ffa480a2910 0x7ffa480a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:36.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.488+0000 7ffa61600640 1 -- 172.21.15.12:0/1527333008 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ffa540cef20 con 0x7ffa641036f0 2024-03-23T22:59:36.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.488+0000 7ffa5be00640 1 --2- 172.21.15.12:0/1527333008 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ffa480a2910 0x7ffa480a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:36.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.489+0000 7ffa5be00640 1 --2- 172.21.15.12:0/1527333008 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ffa480a2910 0x7ffa480a4dd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c02c240 tx=0x7ffa4c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:36.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.494+0000 7ffa61600640 1 -- 172.21.15.12:0/1527333008 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffa5409bf10 con 0x7ffa641036f0 2024-03-23T22:59:36.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.697+0000 7ffa69800640 1 -- 172.21.15.12:0/1527333008 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffa64061ae0 con 0x7ffa480a2910 2024-03-23T22:59:36.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.720+0000 7ffa61600640 1 -- 172.21.15.12:0/1527333008 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258937 (secure 0 0 0) 0x7ffa64061ae0 con 0x7ffa480a2910 2024-03-23T22:59:36.723 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:59:36.724 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-23T22:59:36.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.726+0000 7ffa69800640 1 -- 172.21.15.12:0/1527333008 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ffa480a2910 msgr2=0x7ffa480a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:36.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.726+0000 7ffa69800640 1 --2- 172.21.15.12:0/1527333008 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ffa480a2910 0x7ffa480a4dd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c02c240 tx=0x7ffa4c02df90 comp rx=0 tx=0).stop 2024-03-23T22:59:36.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.726+0000 7ffa69800640 1 -- 172.21.15.12:0/1527333008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa641036f0 msgr2=0x7ffa641999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:36.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.726+0000 7ffa69800640 1 --2- 172.21.15.12:0/1527333008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa641036f0 0x7ffa641999e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ffa5400e9c0 tx=0x7ffa5400ee80 comp rx=0 tx=0).stop 2024-03-23T22:59:36.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.726+0000 7ffa69800640 1 -- 172.21.15.12:0/1527333008 shutdown_connections 2024-03-23T22:59:36.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.726+0000 7ffa69800640 1 --2- 172.21.15.12:0/1527333008 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ffa480a2910 0x7ffa480a4dd0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:36.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.726+0000 7ffa69800640 1 --2- 172.21.15.12:0/1527333008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa641052a0 0x7ffa641a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:36.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.726+0000 7ffa69800640 1 --2- 172.21.15.12:0/1527333008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa641048e0 0x7ffa64199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:36.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.726+0000 7ffa69800640 1 --2- 172.21.15.12:0/1527333008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa641036f0 0x7ffa641999e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:36.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.726+0000 7ffa69800640 1 -- 172.21.15.12:0/1527333008 >> 172.21.15.12:0/1527333008 conn(0x7ffa640fec80 msgr2=0x7ffa64107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:36.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.726+0000 7ffa69800640 1 -- 172.21.15.12:0/1527333008 shutdown_connections 2024-03-23T22:59:36.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:36.726+0000 7ffa69800640 1 -- 172.21.15.12:0/1527333008 wait complete. 2024-03-23T22:59:36.747 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-03-23T22:59:35.259511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594968,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":148,"num_read_kb":127,"num_write":296,"num_write_kb":2974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":236,"ondisk_log_size":236,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":256,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":368668,"kb_used_data":11184,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124573156,"statfs":{"total":1151940427776,"available":1151562911744,"internally_reserved":0,"allocated":11452416,"data_stored":7119817,"data_compressed":109252,"data_compressed_allocated":139264,"data_compressed_original":1777664,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4600,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":22,"num_read_kb":18,"num_write":115,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.091193"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.761766+0000","last_change":"2024-03-23T22:59:27.200234+0000","last_active":"2024-03-23T22:59:29.761766+0000","last_peered":"2024-03-23T22:59:29.761766+0000","last_clean":"2024-03-23T22:59:29.761766+0000","last_became_active":"2024-03-23T22:59:27.199864+0000","last_became_peered":"2024-03-23T22:59:27.199864+0000","last_unstale":"2024-03-23T22:59:29.761766+0000","last_undegraded":"2024-03-23T22:59:29.761766+0000","last_fullsized":"2024-03-23T22:59:29.761766+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:56:26.811964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.608911+0000","last_change":"2024-03-23T22:59:27.195101+0000","last_active":"2024-03-23T22:59:29.608911+0000","last_peered":"2024-03-23T22:59:29.608911+0000","last_clean":"2024-03-23T22:59:29.608911+0000","last_became_active":"2024-03-23T22:59:27.194650+0000","last_became_peered":"2024-03-23T22:59:27.194650+0000","last_unstale":"2024-03-23T22:59:29.608911+0000","last_undegraded":"2024-03-23T22:59:29.608911+0000","last_fullsized":"2024-03-23T22:59:29.608911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:20.396811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.607568+0000","last_change":"2024-03-23T22:59:27.185575+0000","last_active":"2024-03-23T22:59:29.607568+0000","last_peered":"2024-03-23T22:59:29.607568+0000","last_clean":"2024-03-23T22:59:29.607568+0000","last_became_active":"2024-03-23T22:59:27.184328+0000","last_became_peered":"2024-03-23T22:59:27.184328+0000","last_unstale":"2024-03-23T22:59:29.607568+0000","last_undegraded":"2024-03-23T22:59:29.607568+0000","last_fullsized":"2024-03-23T22:59:29.607568+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:31:46.221574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.174918+0000","last_change":"2024-03-23T22:59:27.184434+0000","last_active":"2024-03-23T22:59:28.174918+0000","last_peered":"2024-03-23T22:59:28.174918+0000","last_clean":"2024-03-23T22:59:28.174918+0000","last_became_active":"2024-03-23T22:59:27.183970+0000","last_became_peered":"2024-03-23T22:59:27.183970+0000","last_unstale":"2024-03-23T22:59:28.174918+0000","last_undegraded":"2024-03-23T22:59:28.174918+0000","last_fullsized":"2024-03-23T22:59:28.174918+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:39:05.517976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.342463+0000","last_change":"2024-03-23T22:59:27.214741+0000","last_active":"2024-03-23T22:59:28.342463+0000","last_peered":"2024-03-23T22:59:28.342463+0000","last_clean":"2024-03-23T22:59:28.342463+0000","last_became_active":"2024-03-23T22:59:27.214311+0000","last_became_peered":"2024-03-23T22:59:27.214311+0000","last_unstale":"2024-03-23T22:59:28.342463+0000","last_undegraded":"2024-03-23T22:59:28.342463+0000","last_fullsized":"2024-03-23T22:59:28.342463+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:35:38.464955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.428692+0000","last_change":"2024-03-23T22:59:27.200695+0000","last_active":"2024-03-23T22:59:28.428692+0000","last_peered":"2024-03-23T22:59:28.428692+0000","last_clean":"2024-03-23T22:59:28.428692+0000","last_became_active":"2024-03-23T22:59:27.200159+0000","last_became_peered":"2024-03-23T22:59:27.200159+0000","last_unstale":"2024-03-23T22:59:28.428692+0000","last_undegraded":"2024-03-23T22:59:28.428692+0000","last_fullsized":"2024-03-23T22:59:28.428692+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:01:15.147856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.863135+0000","last_change":"2024-03-23T22:59:27.199928+0000","last_active":"2024-03-23T22:59:28.863135+0000","last_peered":"2024-03-23T22:59:28.863135+0000","last_clean":"2024-03-23T22:59:28.863135+0000","last_became_active":"2024-03-23T22:59:27.199507+0000","last_became_peered":"2024-03-23T22:59:27.199507+0000","last_unstale":"2024-03-23T22:59:28.863135+0000","last_undegraded":"2024-03-23T22:59:28.863135+0000","last_fullsized":"2024-03-23T22:59:28.863135+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:51:19.145785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.173193+0000","last_change":"2024-03-23T22:59:27.217832+0000","last_active":"2024-03-23T22:59:28.173193+0000","last_peered":"2024-03-23T22:59:28.173193+0000","last_clean":"2024-03-23T22:59:28.173193+0000","last_became_active":"2024-03-23T22:59:27.217496+0000","last_became_peered":"2024-03-23T22:59:27.217496+0000","last_unstale":"2024-03-23T22:59:28.173193+0000","last_undegraded":"2024-03-23T22:59:28.173193+0000","last_fullsized":"2024-03-23T22:59:28.173193+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:01.778627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.606781+0000","last_change":"2024-03-23T22:59:27.190982+0000","last_active":"2024-03-23T22:59:29.606781+0000","last_peered":"2024-03-23T22:59:29.606781+0000","last_clean":"2024-03-23T22:59:29.606781+0000","last_became_active":"2024-03-23T22:59:27.190661+0000","last_became_peered":"2024-03-23T22:59:27.190661+0000","last_unstale":"2024-03-23T22:59:29.606781+0000","last_undegraded":"2024-03-23T22:59:29.606781+0000","last_fullsized":"2024-03-23T22:59:29.606781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:29.822253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.428928+0000","last_change":"2024-03-23T22:59:27.187395+0000","last_active":"2024-03-23T22:59:28.428928+0000","last_peered":"2024-03-23T22:59:28.428928+0000","last_clean":"2024-03-23T22:59:28.428928+0000","last_became_active":"2024-03-23T22:59:27.186849+0000","last_became_peered":"2024-03-23T22:59:27.186849+0000","last_unstale":"2024-03-23T22:59:28.428928+0000","last_undegraded":"2024-03-23T22:59:28.428928+0000","last_fullsized":"2024-03-23T22:59:28.428928+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:05:48.047102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.863460+0000","last_change":"2024-03-23T22:59:27.190023+0000","last_active":"2024-03-23T22:59:28.863460+0000","last_peered":"2024-03-23T22:59:28.863460+0000","last_clean":"2024-03-23T22:59:28.863460+0000","last_became_active":"2024-03-23T22:59:27.189697+0000","last_became_peered":"2024-03-23T22:59:27.189697+0000","last_unstale":"2024-03-23T22:59:28.863460+0000","last_undegraded":"2024-03-23T22:59:28.863460+0000","last_fullsized":"2024-03-23T22:59:28.863460+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:59:51.121055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.174930+0000","last_change":"2024-03-23T22:59:27.187382+0000","last_active":"2024-03-23T22:59:28.174930+0000","last_peered":"2024-03-23T22:59:28.174930+0000","last_clean":"2024-03-23T22:59:28.174930+0000","last_became_active":"2024-03-23T22:59:27.186931+0000","last_became_peered":"2024-03-23T22:59:27.186931+0000","last_unstale":"2024-03-23T22:59:28.174930+0000","last_undegraded":"2024-03-23T22:59:28.174930+0000","last_fullsized":"2024-03-23T22:59:28.174930+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:33.583123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.609279+0000","last_change":"2024-03-23T22:59:27.202577+0000","last_active":"2024-03-23T22:59:29.609279+0000","last_peered":"2024-03-23T22:59:29.609279+0000","last_clean":"2024-03-23T22:59:29.609279+0000","last_became_active":"2024-03-23T22:59:27.202122+0000","last_became_peered":"2024-03-23T22:59:27.202122+0000","last_unstale":"2024-03-23T22:59:29.609279+0000","last_undegraded":"2024-03-23T22:59:29.609279+0000","last_fullsized":"2024-03-23T22:59:29.609279+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:01:10.809635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.608450+0000","last_change":"2024-03-23T22:59:27.192915+0000","last_active":"2024-03-23T22:59:29.608450+0000","last_peered":"2024-03-23T22:59:29.608450+0000","last_clean":"2024-03-23T22:59:29.608450+0000","last_became_active":"2024-03-23T22:59:27.192107+0000","last_became_peered":"2024-03-23T22:59:27.192107+0000","last_unstale":"2024-03-23T22:59:29.608450+0000","last_undegraded":"2024-03-23T22:59:29.608450+0000","last_fullsized":"2024-03-23T22:59:29.608450+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:21:19.169211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.607622+0000","last_change":"2024-03-23T22:59:27.192156+0000","last_active":"2024-03-23T22:59:29.607622+0000","last_peered":"2024-03-23T22:59:29.607622+0000","last_clean":"2024-03-23T22:59:29.607622+0000","last_became_active":"2024-03-23T22:59:27.191594+0000","last_became_peered":"2024-03-23T22:59:27.191594+0000","last_unstale":"2024-03-23T22:59:29.607622+0000","last_undegraded":"2024-03-23T22:59:29.607622+0000","last_fullsized":"2024-03-23T22:59:29.607622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:00:45.480281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.606350+0000","last_change":"2024-03-23T22:59:27.186048+0000","last_active":"2024-03-23T22:59:29.606350+0000","last_peered":"2024-03-23T22:59:29.606350+0000","last_clean":"2024-03-23T22:59:29.606350+0000","last_became_active":"2024-03-23T22:59:27.185710+0000","last_became_peered":"2024-03-23T22:59:27.185710+0000","last_unstale":"2024-03-23T22:59:29.606350+0000","last_undegraded":"2024-03-23T22:59:29.606350+0000","last_fullsized":"2024-03-23T22:59:29.606350+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:06.792665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.761858+0000","last_change":"2024-03-23T22:59:27.206290+0000","last_active":"2024-03-23T22:59:29.761858+0000","last_peered":"2024-03-23T22:59:29.761858+0000","last_clean":"2024-03-23T22:59:29.761858+0000","last_became_active":"2024-03-23T22:59:27.205997+0000","last_became_peered":"2024-03-23T22:59:27.205997+0000","last_unstale":"2024-03-23T22:59:29.761858+0000","last_undegraded":"2024-03-23T22:59:29.761858+0000","last_fullsized":"2024-03-23T22:59:29.761858+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:38:42.006166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.447184+0000","last_change":"2024-03-23T22:59:27.203671+0000","last_active":"2024-03-23T22:59:28.447184+0000","last_peered":"2024-03-23T22:59:28.447184+0000","last_clean":"2024-03-23T22:59:28.447184+0000","last_became_active":"2024-03-23T22:59:27.203273+0000","last_became_peered":"2024-03-23T22:59:27.203273+0000","last_unstale":"2024-03-23T22:59:28.447184+0000","last_undegraded":"2024-03-23T22:59:28.447184+0000","last_fullsized":"2024-03-23T22:59:28.447184+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:48:16.018671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.606794+0000","last_change":"2024-03-23T22:59:27.210063+0000","last_active":"2024-03-23T22:59:29.606794+0000","last_peered":"2024-03-23T22:59:29.606794+0000","last_clean":"2024-03-23T22:59:29.606794+0000","last_became_active":"2024-03-23T22:59:27.209698+0000","last_became_peered":"2024-03-23T22:59:27.209698+0000","last_unstale":"2024-03-23T22:59:29.606794+0000","last_undegraded":"2024-03-23T22:59:29.606794+0000","last_fullsized":"2024-03-23T22:59:29.606794+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:19:01.408901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:31.508224+0000","last_change":"2024-03-23T22:59:26.445053+0000","last_active":"2024-03-23T22:59:31.508224+0000","last_peered":"2024-03-23T22:59:31.508224+0000","last_clean":"2024-03-23T22:59:31.508224+0000","last_became_active":"2024-03-23T22:59:26.444660+0000","last_became_peered":"2024-03-23T22:59:26.444660+0000","last_unstale":"2024-03-23T22:59:31.508224+0000","last_undegraded":"2024-03-23T22:59:31.508224+0000","last_fullsized":"2024-03-23T22:59:31.508224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:19:16.958228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.174647+0000","last_change":"2024-03-23T22:59:27.208652+0000","last_active":"2024-03-23T22:59:28.174647+0000","last_peered":"2024-03-23T22:59:28.174647+0000","last_clean":"2024-03-23T22:59:28.174647+0000","last_became_active":"2024-03-23T22:59:27.207724+0000","last_became_peered":"2024-03-23T22:59:27.207724+0000","last_unstale":"2024-03-23T22:59:28.174647+0000","last_undegraded":"2024-03-23T22:59:28.174647+0000","last_fullsized":"2024-03-23T22:59:28.174647+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:08.986439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.183896+0000","last_change":"2024-03-23T22:59:26.445371+0000","last_active":"2024-03-23T22:59:28.183896+0000","last_peered":"2024-03-23T22:59:28.183896+0000","last_clean":"2024-03-23T22:59:28.183896+0000","last_became_active":"2024-03-23T22:59:26.445016+0000","last_became_peered":"2024-03-23T22:59:26.445016+0000","last_unstale":"2024-03-23T22:59:28.183896+0000","last_undegraded":"2024-03-23T22:59:28.183896+0000","last_fullsized":"2024-03-23T22:59:28.183896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:31:44.003218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.609355+0000","last_change":"2024-03-23T22:59:27.200624+0000","last_active":"2024-03-23T22:59:29.609355+0000","last_peered":"2024-03-23T22:59:29.609355+0000","last_clean":"2024-03-23T22:59:29.609355+0000","last_became_active":"2024-03-23T22:59:27.200207+0000","last_became_peered":"2024-03-23T22:59:27.200207+0000","last_unstale":"2024-03-23T22:59:29.609355+0000","last_undegraded":"2024-03-23T22:59:29.609355+0000","last_fullsized":"2024-03-23T22:59:29.609355+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:05:31.015869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:31.499004+0000","last_change":"2024-03-23T22:59:26.431458+0000","last_active":"2024-03-23T22:59:31.499004+0000","last_peered":"2024-03-23T22:59:31.499004+0000","last_clean":"2024-03-23T22:59:31.499004+0000","last_became_active":"2024-03-23T22:59:26.430948+0000","last_became_peered":"2024-03-23T22:59:26.430948+0000","last_unstale":"2024-03-23T22:59:31.499004+0000","last_undegraded":"2024-03-23T22:59:31.499004+0000","last_fullsized":"2024-03-23T22:59:31.499004+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:08:29.777912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.606634+0000","last_change":"2024-03-23T22:59:27.204157+0000","last_active":"2024-03-23T22:59:29.606634+0000","last_peered":"2024-03-23T22:59:29.606634+0000","last_clean":"2024-03-23T22:59:29.606634+0000","last_became_active":"2024-03-23T22:59:27.203559+0000","last_became_peered":"2024-03-23T22:59:27.203559+0000","last_unstale":"2024-03-23T22:59:29.606634+0000","last_undegraded":"2024-03-23T22:59:29.606634+0000","last_fullsized":"2024-03-23T22:59:29.606634+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:43.834369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.342668+0000","last_change":"2024-03-23T22:59:26.425898+0000","last_active":"2024-03-23T22:59:28.342668+0000","last_peered":"2024-03-23T22:59:28.342668+0000","last_clean":"2024-03-23T22:59:28.342668+0000","last_became_active":"2024-03-23T22:59:26.425567+0000","last_became_peered":"2024-03-23T22:59:26.425567+0000","last_unstale":"2024-03-23T22:59:28.342668+0000","last_undegraded":"2024-03-23T22:59:28.342668+0000","last_fullsized":"2024-03-23T22:59:28.342668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:26:31.888225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.172739+0000","last_change":"2024-03-23T22:59:27.186599+0000","last_active":"2024-03-23T22:59:28.172739+0000","last_peered":"2024-03-23T22:59:28.172739+0000","last_clean":"2024-03-23T22:59:28.172739+0000","last_became_active":"2024-03-23T22:59:27.186262+0000","last_became_peered":"2024-03-23T22:59:27.186262+0000","last_unstale":"2024-03-23T22:59:28.172739+0000","last_undegraded":"2024-03-23T22:59:28.172739+0000","last_fullsized":"2024-03-23T22:59:28.172739+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:11:39.545558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:33.205152+0000","last_change":"2024-03-23T22:59:26.430096+0000","last_active":"2024-03-23T22:59:33.205152+0000","last_peered":"2024-03-23T22:59:33.205152+0000","last_clean":"2024-03-23T22:59:33.205152+0000","last_became_active":"2024-03-23T22:59:26.429740+0000","last_became_peered":"2024-03-23T22:59:26.429740+0000","last_unstale":"2024-03-23T22:59:33.205152+0000","last_undegraded":"2024-03-23T22:59:33.205152+0000","last_fullsized":"2024-03-23T22:59:33.205152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:54.831689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.342577+0000","last_change":"2024-03-23T22:59:27.202750+0000","last_active":"2024-03-23T22:59:28.342577+0000","last_peered":"2024-03-23T22:59:28.342577+0000","last_clean":"2024-03-23T22:59:28.342577+0000","last_became_active":"2024-03-23T22:59:27.202391+0000","last_became_peered":"2024-03-23T22:59:27.202391+0000","last_unstale":"2024-03-23T22:59:28.342577+0000","last_undegraded":"2024-03-23T22:59:28.342577+0000","last_fullsized":"2024-03-23T22:59:28.342577+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:27:18.370792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:31.501847+0000","last_change":"2024-03-23T22:59:26.425607+0000","last_active":"2024-03-23T22:59:31.501847+0000","last_peered":"2024-03-23T22:59:31.501847+0000","last_clean":"2024-03-23T22:59:31.501847+0000","last_became_active":"2024-03-23T22:59:26.425208+0000","last_became_peered":"2024-03-23T22:59:26.425208+0000","last_unstale":"2024-03-23T22:59:31.501847+0000","last_undegraded":"2024-03-23T22:59:31.501847+0000","last_fullsized":"2024-03-23T22:59:31.501847+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:51.071662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.172751+0000","last_change":"2024-03-23T22:59:27.215637+0000","last_active":"2024-03-23T22:59:28.172751+0000","last_peered":"2024-03-23T22:59:28.172751+0000","last_clean":"2024-03-23T22:59:28.172751+0000","last_became_active":"2024-03-23T22:59:27.215339+0000","last_became_peered":"2024-03-23T22:59:27.215339+0000","last_unstale":"2024-03-23T22:59:28.172751+0000","last_undegraded":"2024-03-23T22:59:28.172751+0000","last_fullsized":"2024-03-23T22:59:28.172751+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:25:42.097326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:31.504047+0000","last_change":"2024-03-23T22:59:26.442023+0000","last_active":"2024-03-23T22:59:31.504047+0000","last_peered":"2024-03-23T22:59:31.504047+0000","last_clean":"2024-03-23T22:59:31.504047+0000","last_became_active":"2024-03-23T22:59:26.441661+0000","last_became_peered":"2024-03-23T22:59:26.441661+0000","last_unstale":"2024-03-23T22:59:31.504047+0000","last_undegraded":"2024-03-23T22:59:31.504047+0000","last_fullsized":"2024-03-23T22:59:31.504047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:59:14.914860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.173662+0000","last_change":"2024-03-23T22:59:27.221660+0000","last_active":"2024-03-23T22:59:28.173662+0000","last_peered":"2024-03-23T22:59:28.173662+0000","last_clean":"2024-03-23T22:59:28.173662+0000","last_became_active":"2024-03-23T22:59:27.221298+0000","last_became_peered":"2024-03-23T22:59:27.221298+0000","last_unstale":"2024-03-23T22:59:28.173662+0000","last_undegraded":"2024-03-23T22:59:28.173662+0000","last_fullsized":"2024-03-23T22:59:28.173662+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:15.189932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.873475+0000","last_change":"2024-03-23T22:59:26.430147+0000","last_active":"2024-03-23T22:59:29.873475+0000","last_peered":"2024-03-23T22:59:29.873475+0000","last_clean":"2024-03-23T22:59:29.873475+0000","last_became_active":"2024-03-23T22:59:26.429843+0000","last_became_peered":"2024-03-23T22:59:26.429843+0000","last_unstale":"2024-03-23T22:59:29.873475+0000","last_undegraded":"2024-03-23T22:59:29.873475+0000","last_fullsized":"2024-03-23T22:59:29.873475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:06:45.691894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.446099+0000","last_change":"2024-03-23T22:59:27.212235+0000","last_active":"2024-03-23T22:59:28.446099+0000","last_peered":"2024-03-23T22:59:28.446099+0000","last_clean":"2024-03-23T22:59:28.446099+0000","last_became_active":"2024-03-23T22:59:27.211919+0000","last_became_peered":"2024-03-23T22:59:27.211919+0000","last_unstale":"2024-03-23T22:59:28.446099+0000","last_undegraded":"2024-03-23T22:59:28.446099+0000","last_fullsized":"2024-03-23T22:59:28.446099+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:24.752109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:30.486244+0000","last_change":"2024-03-23T22:59:26.440623+0000","last_active":"2024-03-23T22:59:30.486244+0000","last_peered":"2024-03-23T22:59:30.486244+0000","last_clean":"2024-03-23T22:59:30.486244+0000","last_became_active":"2024-03-23T22:59:26.440204+0000","last_became_peered":"2024-03-23T22:59:26.440204+0000","last_unstale":"2024-03-23T22:59:30.486244+0000","last_undegraded":"2024-03-23T22:59:30.486244+0000","last_fullsized":"2024-03-23T22:59:30.486244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:58:44.088552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.606704+0000","last_change":"2024-03-23T22:59:27.203916+0000","last_active":"2024-03-23T22:59:29.606704+0000","last_peered":"2024-03-23T22:59:29.606704+0000","last_clean":"2024-03-23T22:59:29.606704+0000","last_became_active":"2024-03-23T22:59:27.203565+0000","last_became_peered":"2024-03-23T22:59:27.203565+0000","last_unstale":"2024-03-23T22:59:29.606704+0000","last_undegraded":"2024-03-23T22:59:29.606704+0000","last_fullsized":"2024-03-23T22:59:29.606704+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:06:39.635025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:30.484141+0000","last_change":"2024-03-23T22:59:26.430080+0000","last_active":"2024-03-23T22:59:30.484141+0000","last_peered":"2024-03-23T22:59:30.484141+0000","last_clean":"2024-03-23T22:59:30.484141+0000","last_became_active":"2024-03-23T22:59:26.429618+0000","last_became_peered":"2024-03-23T22:59:26.429618+0000","last_unstale":"2024-03-23T22:59:30.484141+0000","last_undegraded":"2024-03-23T22:59:30.484141+0000","last_fullsized":"2024-03-23T22:59:30.484141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:01.963573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.609080+0000","last_change":"2024-03-23T22:59:27.190505+0000","last_active":"2024-03-23T22:59:29.609080+0000","last_peered":"2024-03-23T22:59:29.609080+0000","last_clean":"2024-03-23T22:59:29.609080+0000","last_became_active":"2024-03-23T22:59:27.190088+0000","last_became_peered":"2024-03-23T22:59:27.190088+0000","last_unstale":"2024-03-23T22:59:29.609080+0000","last_undegraded":"2024-03-23T22:59:29.609080+0000","last_fullsized":"2024-03-23T22:59:29.609080+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:29:04.255832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:30.488604+0000","last_change":"2024-03-23T22:59:26.438329+0000","last_active":"2024-03-23T22:59:30.488604+0000","last_peered":"2024-03-23T22:59:30.488604+0000","last_clean":"2024-03-23T22:59:30.488604+0000","last_became_active":"2024-03-23T22:59:26.438033+0000","last_became_peered":"2024-03-23T22:59:26.438033+0000","last_unstale":"2024-03-23T22:59:30.488604+0000","last_undegraded":"2024-03-23T22:59:30.488604+0000","last_fullsized":"2024-03-23T22:59:30.488604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:02:38.789574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.606601+0000","last_change":"2024-03-23T22:59:27.188417+0000","last_active":"2024-03-23T22:59:29.606601+0000","last_peered":"2024-03-23T22:59:29.606601+0000","last_clean":"2024-03-23T22:59:29.606601+0000","last_became_active":"2024-03-23T22:59:27.188023+0000","last_became_peered":"2024-03-23T22:59:27.188023+0000","last_unstale":"2024-03-23T22:59:29.606601+0000","last_undegraded":"2024-03-23T22:59:29.606601+0000","last_fullsized":"2024-03-23T22:59:29.606601+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:19:11.014604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:32.205798+0000","last_change":"2024-03-23T22:59:26.425963+0000","last_active":"2024-03-23T22:59:32.205798+0000","last_peered":"2024-03-23T22:59:32.205798+0000","last_clean":"2024-03-23T22:59:32.205798+0000","last_became_active":"2024-03-23T22:59:26.425640+0000","last_became_peered":"2024-03-23T22:59:26.425640+0000","last_unstale":"2024-03-23T22:59:32.205798+0000","last_undegraded":"2024-03-23T22:59:32.205798+0000","last_fullsized":"2024-03-23T22:59:32.205798+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:11:26.529034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.342661+0000","last_change":"2024-03-23T22:59:27.187254+0000","last_active":"2024-03-23T22:59:28.342661+0000","last_peered":"2024-03-23T22:59:28.342661+0000","last_clean":"2024-03-23T22:59:28.342661+0000","last_became_active":"2024-03-23T22:59:27.186849+0000","last_became_peered":"2024-03-23T22:59:27.186849+0000","last_unstale":"2024-03-23T22:59:28.342661+0000","last_undegraded":"2024-03-23T22:59:28.342661+0000","last_fullsized":"2024-03-23T22:59:28.342661+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:35:30.958565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:31.505287+0000","last_change":"2024-03-23T22:59:26.429767+0000","last_active":"2024-03-23T22:59:31.505287+0000","last_peered":"2024-03-23T22:59:31.505287+0000","last_clean":"2024-03-23T22:59:31.505287+0000","last_became_active":"2024-03-23T22:59:26.429449+0000","last_became_peered":"2024-03-23T22:59:26.429449+0000","last_unstale":"2024-03-23T22:59:31.505287+0000","last_undegraded":"2024-03-23T22:59:31.505287+0000","last_fullsized":"2024-03-23T22:59:31.505287+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:46:13.752427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.608887+0000","last_change":"2024-03-23T22:59:27.223544+0000","last_active":"2024-03-23T22:59:29.608887+0000","last_peered":"2024-03-23T22:59:29.608887+0000","last_clean":"2024-03-23T22:59:29.608887+0000","last_became_active":"2024-03-23T22:59:27.223177+0000","last_became_peered":"2024-03-23T22:59:27.223177+0000","last_unstale":"2024-03-23T22:59:29.608887+0000","last_undegraded":"2024-03-23T22:59:29.608887+0000","last_fullsized":"2024-03-23T22:59:29.608887+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:33:18.521579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:30.488651+0000","last_change":"2024-03-23T22:59:26.427781+0000","last_active":"2024-03-23T22:59:30.488651+0000","last_peered":"2024-03-23T22:59:30.488651+0000","last_clean":"2024-03-23T22:59:30.488651+0000","last_became_active":"2024-03-23T22:59:26.427458+0000","last_became_peered":"2024-03-23T22:59:26.427458+0000","last_unstale":"2024-03-23T22:59:30.488651+0000","last_undegraded":"2024-03-23T22:59:30.488651+0000","last_fullsized":"2024-03-23T22:59:30.488651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:50:33.956611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.863313+0000","last_change":"2024-03-23T22:59:27.203681+0000","last_active":"2024-03-23T22:59:28.863313+0000","last_peered":"2024-03-23T22:59:28.863313+0000","last_clean":"2024-03-23T22:59:28.863313+0000","last_became_active":"2024-03-23T22:59:27.202457+0000","last_became_peered":"2024-03-23T22:59:27.202457+0000","last_unstale":"2024-03-23T22:59:28.863313+0000","last_undegraded":"2024-03-23T22:59:28.863313+0000","last_fullsized":"2024-03-23T22:59:28.863313+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:55:46.725350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.174494+0000","last_change":"2024-03-23T22:59:26.428517+0000","last_active":"2024-03-23T22:59:28.174494+0000","last_peered":"2024-03-23T22:59:28.174494+0000","last_clean":"2024-03-23T22:59:28.174494+0000","last_became_active":"2024-03-23T22:59:26.428130+0000","last_became_peered":"2024-03-23T22:59:26.428130+0000","last_unstale":"2024-03-23T22:59:28.174494+0000","last_undegraded":"2024-03-23T22:59:28.174494+0000","last_fullsized":"2024-03-23T22:59:28.174494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:01.185776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.863148+0000","last_change":"2024-03-23T22:59:27.191428+0000","last_active":"2024-03-23T22:59:28.863148+0000","last_peered":"2024-03-23T22:59:28.863148+0000","last_clean":"2024-03-23T22:59:28.863148+0000","last_became_active":"2024-03-23T22:59:27.190958+0000","last_became_peered":"2024-03-23T22:59:27.190958+0000","last_unstale":"2024-03-23T22:59:28.863148+0000","last_undegraded":"2024-03-23T22:59:28.863148+0000","last_fullsized":"2024-03-23T22:59:28.863148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:36:43.748554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:31.504838+0000","last_change":"2024-03-23T22:59:26.442637+0000","last_active":"2024-03-23T22:59:31.504838+0000","last_peered":"2024-03-23T22:59:31.504838+0000","last_clean":"2024-03-23T22:59:31.504838+0000","last_became_active":"2024-03-23T22:59:26.442267+0000","last_became_peered":"2024-03-23T22:59:26.442267+0000","last_unstale":"2024-03-23T22:59:31.504838+0000","last_undegraded":"2024-03-23T22:59:31.504838+0000","last_fullsized":"2024-03-23T22:59:31.504838+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:49.975048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.172240+0000","last_change":"2024-03-23T22:59:27.187812+0000","last_active":"2024-03-23T22:59:28.172240+0000","last_peered":"2024-03-23T22:59:28.172240+0000","last_clean":"2024-03-23T22:59:28.172240+0000","last_became_active":"2024-03-23T22:59:27.187423+0000","last_became_peered":"2024-03-23T22:59:27.187423+0000","last_unstale":"2024-03-23T22:59:28.172240+0000","last_undegraded":"2024-03-23T22:59:28.172240+0000","last_fullsized":"2024-03-23T22:59:28.172240+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:12:16.427695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.342924+0000","last_change":"2024-03-23T22:59:26.423875+0000","last_active":"2024-03-23T22:59:28.342924+0000","last_peered":"2024-03-23T22:59:28.342924+0000","last_clean":"2024-03-23T22:59:28.342924+0000","last_became_active":"2024-03-23T22:59:26.423513+0000","last_became_peered":"2024-03-23T22:59:26.423513+0000","last_unstale":"2024-03-23T22:59:28.342924+0000","last_undegraded":"2024-03-23T22:59:28.342924+0000","last_fullsized":"2024-03-23T22:59:28.342924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:43.173365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.761656+0000","last_change":"2024-03-23T22:59:27.191454+0000","last_active":"2024-03-23T22:59:29.761656+0000","last_peered":"2024-03-23T22:59:29.761656+0000","last_clean":"2024-03-23T22:59:29.761656+0000","last_became_active":"2024-03-23T22:59:27.190927+0000","last_became_peered":"2024-03-23T22:59:27.190927+0000","last_unstale":"2024-03-23T22:59:29.761656+0000","last_undegraded":"2024-03-23T22:59:29.761656+0000","last_fullsized":"2024-03-23T22:59:29.761656+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:00.333702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.446810+0000","last_change":"2024-03-23T22:59:26.443238+0000","last_active":"2024-03-23T22:59:28.446810+0000","last_peered":"2024-03-23T22:59:28.446810+0000","last_clean":"2024-03-23T22:59:28.446810+0000","last_became_active":"2024-03-23T22:59:26.442834+0000","last_became_peered":"2024-03-23T22:59:26.442834+0000","last_unstale":"2024-03-23T22:59:28.446810+0000","last_undegraded":"2024-03-23T22:59:28.446810+0000","last_fullsized":"2024-03-23T22:59:28.446810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:11:21.072836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.428955+0000","last_change":"2024-03-23T22:59:27.212864+0000","last_active":"2024-03-23T22:59:28.428955+0000","last_peered":"2024-03-23T22:59:28.428955+0000","last_clean":"2024-03-23T22:59:28.428955+0000","last_became_active":"2024-03-23T22:59:27.212455+0000","last_became_peered":"2024-03-23T22:59:27.212455+0000","last_unstale":"2024-03-23T22:59:28.428955+0000","last_undegraded":"2024-03-23T22:59:28.428955+0000","last_fullsized":"2024-03-23T22:59:28.428955+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:25:59.122531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.863137+0000","last_change":"2024-03-23T22:59:26.432838+0000","last_active":"2024-03-23T22:59:28.863137+0000","last_peered":"2024-03-23T22:59:28.863137+0000","last_clean":"2024-03-23T22:59:28.863137+0000","last_became_active":"2024-03-23T22:59:26.432544+0000","last_became_peered":"2024-03-23T22:59:26.432544+0000","last_unstale":"2024-03-23T22:59:28.863137+0000","last_undegraded":"2024-03-23T22:59:28.863137+0000","last_fullsized":"2024-03-23T22:59:28.863137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:32:52.121569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.428691+0000","last_change":"2024-03-23T22:59:27.192870+0000","last_active":"2024-03-23T22:59:28.428691+0000","last_peered":"2024-03-23T22:59:28.428691+0000","last_clean":"2024-03-23T22:59:28.428691+0000","last_became_active":"2024-03-23T22:59:27.192423+0000","last_became_peered":"2024-03-23T22:59:27.192423+0000","last_unstale":"2024-03-23T22:59:28.428691+0000","last_undegraded":"2024-03-23T22:59:28.428691+0000","last_fullsized":"2024-03-23T22:59:28.428691+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:27:52.709256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.172451+0000","last_change":"2024-03-23T22:59:27.185585+0000","last_active":"2024-03-23T22:59:28.172451+0000","last_peered":"2024-03-23T22:59:28.172451+0000","last_clean":"2024-03-23T22:59:28.172451+0000","last_became_active":"2024-03-23T22:59:27.185278+0000","last_became_peered":"2024-03-23T22:59:27.185278+0000","last_unstale":"2024-03-23T22:59:28.172451+0000","last_undegraded":"2024-03-23T22:59:28.172451+0000","last_fullsized":"2024-03-23T22:59:28.172451+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:45:09.759771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:33.209053+0000","last_change":"2024-03-23T22:59:26.450679+0000","last_active":"2024-03-23T22:59:33.209053+0000","last_peered":"2024-03-23T22:59:33.209053+0000","last_clean":"2024-03-23T22:59:33.209053+0000","last_became_active":"2024-03-23T22:59:26.450234+0000","last_became_peered":"2024-03-23T22:59:26.450234+0000","last_unstale":"2024-03-23T22:59:33.209053+0000","last_undegraded":"2024-03-23T22:59:33.209053+0000","last_fullsized":"2024-03-23T22:59:33.209053+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:52:25.869194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.173444+0000","last_change":"2024-03-23T22:59:27.220171+0000","last_active":"2024-03-23T22:59:28.173444+0000","last_peered":"2024-03-23T22:59:28.173444+0000","last_clean":"2024-03-23T22:59:28.173444+0000","last_became_active":"2024-03-23T22:59:27.219787+0000","last_became_peered":"2024-03-23T22:59:27.219787+0000","last_unstale":"2024-03-23T22:59:28.173444+0000","last_undegraded":"2024-03-23T22:59:28.173444+0000","last_fullsized":"2024-03-23T22:59:28.173444+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:51:52.634264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:30.487301+0000","last_change":"2024-03-23T22:59:26.440205+0000","last_active":"2024-03-23T22:59:30.487301+0000","last_peered":"2024-03-23T22:59:30.487301+0000","last_clean":"2024-03-23T22:59:30.487301+0000","last_became_active":"2024-03-23T22:59:26.439752+0000","last_became_peered":"2024-03-23T22:59:26.439752+0000","last_unstale":"2024-03-23T22:59:30.487301+0000","last_undegraded":"2024-03-23T22:59:30.487301+0000","last_fullsized":"2024-03-23T22:59:30.487301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:55:25.661887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":340,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.863034+0000","last_change":"2024-03-23T22:58:06.171317+0000","last_active":"2024-03-23T22:59:28.863034+0000","last_peered":"2024-03-23T22:59:28.863034+0000","last_clean":"2024-03-23T22:59:28.863034+0000","last_became_active":"2024-03-23T22:58:06.170816+0000","last_became_peered":"2024-03-23T22:58:06.170816+0000","last_unstale":"2024-03-23T22:59:28.863034+0000","last_undegraded":"2024-03-23T22:59:28.863034+0000","last_fullsized":"2024-03-23T22:59:28.863034+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:54:11.991284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:54:11.991284+0000","last_clean_scrub_stamp":"2024-03-23T22:54:11.991284+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:20:55.980613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.607348+0000","last_change":"2024-03-23T22:59:27.200348+0000","last_active":"2024-03-23T22:59:29.607348+0000","last_peered":"2024-03-23T22:59:29.607348+0000","last_clean":"2024-03-23T22:59:29.607348+0000","last_became_active":"2024-03-23T22:59:27.199979+0000","last_became_peered":"2024-03-23T22:59:27.199979+0000","last_unstale":"2024-03-23T22:59:29.607348+0000","last_undegraded":"2024-03-23T22:59:29.607348+0000","last_fullsized":"2024-03-23T22:59:29.607348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:31:19.258890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.863267+0000","last_change":"2024-03-23T22:59:26.431837+0000","last_active":"2024-03-23T22:59:28.863267+0000","last_peered":"2024-03-23T22:59:28.863267+0000","last_clean":"2024-03-23T22:59:28.863267+0000","last_became_active":"2024-03-23T22:59:26.431580+0000","last_became_peered":"2024-03-23T22:59:26.431580+0000","last_unstale":"2024-03-23T22:59:28.863267+0000","last_undegraded":"2024-03-23T22:59:28.863267+0000","last_fullsized":"2024-03-23T22:59:28.863267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:49:53.316952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:30.487730+0000","last_change":"2024-03-23T22:59:26.442878+0000","last_active":"2024-03-23T22:59:30.487730+0000","last_peered":"2024-03-23T22:59:30.487730+0000","last_clean":"2024-03-23T22:59:30.487730+0000","last_became_active":"2024-03-23T22:59:26.442303+0000","last_became_peered":"2024-03-23T22:59:26.442303+0000","last_unstale":"2024-03-23T22:59:30.487730+0000","last_undegraded":"2024-03-23T22:59:30.487730+0000","last_fullsized":"2024-03-23T22:59:30.487730+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:51.717553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.608814+0000","last_change":"2024-03-23T22:59:27.212550+0000","last_active":"2024-03-23T22:59:29.608814+0000","last_peered":"2024-03-23T22:59:29.608814+0000","last_clean":"2024-03-23T22:59:29.608814+0000","last_became_active":"2024-03-23T22:59:27.212105+0000","last_became_peered":"2024-03-23T22:59:27.212105+0000","last_unstale":"2024-03-23T22:59:29.608814+0000","last_undegraded":"2024-03-23T22:59:29.608814+0000","last_fullsized":"2024-03-23T22:59:29.608814+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:23:37.374081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:31.503691+0000","last_change":"2024-03-23T22:59:26.452146+0000","last_active":"2024-03-23T22:59:31.503691+0000","last_peered":"2024-03-23T22:59:31.503691+0000","last_clean":"2024-03-23T22:59:31.503691+0000","last_became_active":"2024-03-23T22:59:26.450398+0000","last_became_peered":"2024-03-23T22:59:26.450398+0000","last_unstale":"2024-03-23T22:59:31.503691+0000","last_undegraded":"2024-03-23T22:59:31.503691+0000","last_fullsized":"2024-03-23T22:59:31.503691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:55:47.085457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.174976+0000","last_change":"2024-03-23T22:59:27.209690+0000","last_active":"2024-03-23T22:59:28.174976+0000","last_peered":"2024-03-23T22:59:28.174976+0000","last_clean":"2024-03-23T22:59:28.174976+0000","last_became_active":"2024-03-23T22:59:27.209187+0000","last_became_peered":"2024-03-23T22:59:27.209187+0000","last_unstale":"2024-03-23T22:59:28.174976+0000","last_undegraded":"2024-03-23T22:59:28.174976+0000","last_fullsized":"2024-03-23T22:59:28.174976+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:19:45.315459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:30.485058+0000","last_change":"2024-03-23T22:59:26.434958+0000","last_active":"2024-03-23T22:59:30.485058+0000","last_peered":"2024-03-23T22:59:30.485058+0000","last_clean":"2024-03-23T22:59:30.485058+0000","last_became_active":"2024-03-23T22:59:26.434611+0000","last_became_peered":"2024-03-23T22:59:26.434611+0000","last_unstale":"2024-03-23T22:59:30.485058+0000","last_undegraded":"2024-03-23T22:59:30.485058+0000","last_fullsized":"2024-03-23T22:59:30.485058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:42:18.201885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.609415+0000","last_change":"2024-03-23T22:59:27.211700+0000","last_active":"2024-03-23T22:59:29.609415+0000","last_peered":"2024-03-23T22:59:29.609415+0000","last_clean":"2024-03-23T22:59:29.609415+0000","last_became_active":"2024-03-23T22:59:27.211248+0000","last_became_peered":"2024-03-23T22:59:27.211248+0000","last_unstale":"2024-03-23T22:59:29.609415+0000","last_undegraded":"2024-03-23T22:59:29.609415+0000","last_fullsized":"2024-03-23T22:59:29.609415+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:49.868397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.606220+0000","last_change":"2024-03-23T22:59:26.442625+0000","last_active":"2024-03-23T22:59:29.606220+0000","last_peered":"2024-03-23T22:59:29.606220+0000","last_clean":"2024-03-23T22:59:29.606220+0000","last_became_active":"2024-03-23T22:59:26.442312+0000","last_became_peered":"2024-03-23T22:59:26.442312+0000","last_unstale":"2024-03-23T22:59:29.606220+0000","last_undegraded":"2024-03-23T22:59:29.606220+0000","last_fullsized":"2024-03-23T22:59:29.606220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:27:38.808784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.342409+0000","last_change":"2024-03-23T22:59:27.187363+0000","last_active":"2024-03-23T22:59:28.342409+0000","last_peered":"2024-03-23T22:59:28.342409+0000","last_clean":"2024-03-23T22:59:28.342409+0000","last_became_active":"2024-03-23T22:59:27.187007+0000","last_became_peered":"2024-03-23T22:59:27.187007+0000","last_unstale":"2024-03-23T22:59:28.342409+0000","last_undegraded":"2024-03-23T22:59:28.342409+0000","last_fullsized":"2024-03-23T22:59:28.342409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:41:37.041254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:30.492845+0000","last_change":"2024-03-23T22:59:26.445753+0000","last_active":"2024-03-23T22:59:30.492845+0000","last_peered":"2024-03-23T22:59:30.492845+0000","last_clean":"2024-03-23T22:59:30.492845+0000","last_became_active":"2024-03-23T22:59:26.445167+0000","last_became_peered":"2024-03-23T22:59:26.445167+0000","last_unstale":"2024-03-23T22:59:30.492845+0000","last_undegraded":"2024-03-23T22:59:30.492845+0000","last_fullsized":"2024-03-23T22:59:30.492845+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:46:46.058523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.761371+0000","last_change":"2024-03-23T22:59:27.203855+0000","last_active":"2024-03-23T22:59:29.761371+0000","last_peered":"2024-03-23T22:59:29.761371+0000","last_clean":"2024-03-23T22:59:29.761371+0000","last_became_active":"2024-03-23T22:59:27.203438+0000","last_became_peered":"2024-03-23T22:59:27.203438+0000","last_unstale":"2024-03-23T22:59:29.761371+0000","last_undegraded":"2024-03-23T22:59:29.761371+0000","last_fullsized":"2024-03-23T22:59:29.761371+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:28.725743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.994795+0000","last_change":"2024-03-23T22:59:26.439002+0000","last_active":"2024-03-23T22:59:29.994795+0000","last_peered":"2024-03-23T22:59:29.994795+0000","last_clean":"2024-03-23T22:59:29.994795+0000","last_became_active":"2024-03-23T22:59:26.438634+0000","last_became_peered":"2024-03-23T22:59:26.438634+0000","last_unstale":"2024-03-23T22:59:29.994795+0000","last_undegraded":"2024-03-23T22:59:29.994795+0000","last_fullsized":"2024-03-23T22:59:29.994795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:25:02.988163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.173987+0000","last_change":"2024-03-23T22:59:27.217314+0000","last_active":"2024-03-23T22:59:28.173987+0000","last_peered":"2024-03-23T22:59:28.173987+0000","last_clean":"2024-03-23T22:59:28.173987+0000","last_became_active":"2024-03-23T22:59:27.216790+0000","last_became_peered":"2024-03-23T22:59:27.216790+0000","last_unstale":"2024-03-23T22:59:28.173987+0000","last_undegraded":"2024-03-23T22:59:28.173987+0000","last_fullsized":"2024-03-23T22:59:28.173987+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:47:13.902236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:30.485663+0000","last_change":"2024-03-23T22:59:26.431453+0000","last_active":"2024-03-23T22:59:30.485663+0000","last_peered":"2024-03-23T22:59:30.485663+0000","last_clean":"2024-03-23T22:59:30.485663+0000","last_became_active":"2024-03-23T22:59:26.429731+0000","last_became_peered":"2024-03-23T22:59:26.429731+0000","last_unstale":"2024-03-23T22:59:30.485663+0000","last_undegraded":"2024-03-23T22:59:30.485663+0000","last_fullsized":"2024-03-23T22:59:30.485663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:33:47.717064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.863310+0000","last_change":"2024-03-23T22:59:27.203097+0000","last_active":"2024-03-23T22:59:28.863310+0000","last_peered":"2024-03-23T22:59:28.863310+0000","last_clean":"2024-03-23T22:59:28.863310+0000","last_became_active":"2024-03-23T22:59:27.202484+0000","last_became_peered":"2024-03-23T22:59:27.202484+0000","last_unstale":"2024-03-23T22:59:28.863310+0000","last_undegraded":"2024-03-23T22:59:28.863310+0000","last_fullsized":"2024-03-23T22:59:28.863310+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:07.829866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:31.504672+0000","last_change":"2024-03-23T22:59:26.442815+0000","last_active":"2024-03-23T22:59:31.504672+0000","last_peered":"2024-03-23T22:59:31.504672+0000","last_clean":"2024-03-23T22:59:31.504672+0000","last_became_active":"2024-03-23T22:59:26.442401+0000","last_became_peered":"2024-03-23T22:59:26.442401+0000","last_unstale":"2024-03-23T22:59:31.504672+0000","last_undegraded":"2024-03-23T22:59:31.504672+0000","last_fullsized":"2024-03-23T22:59:31.504672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:44:06.208357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.863535+0000","last_change":"2024-03-23T22:59:27.189244+0000","last_active":"2024-03-23T22:59:28.863535+0000","last_peered":"2024-03-23T22:59:28.863535+0000","last_clean":"2024-03-23T22:59:28.863535+0000","last_became_active":"2024-03-23T22:59:27.188629+0000","last_became_peered":"2024-03-23T22:59:27.188629+0000","last_unstale":"2024-03-23T22:59:28.863535+0000","last_undegraded":"2024-03-23T22:59:28.863535+0000","last_fullsized":"2024-03-23T22:59:28.863535+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:47:26.742204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.873569+0000","last_change":"2024-03-23T22:59:26.434326+0000","last_active":"2024-03-23T22:59:29.873569+0000","last_peered":"2024-03-23T22:59:29.873569+0000","last_clean":"2024-03-23T22:59:29.873569+0000","last_became_active":"2024-03-23T22:59:26.433878+0000","last_became_peered":"2024-03-23T22:59:26.433878+0000","last_unstale":"2024-03-23T22:59:29.873569+0000","last_undegraded":"2024-03-23T22:59:29.873569+0000","last_fullsized":"2024-03-23T22:59:29.873569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:08.580572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.863451+0000","last_change":"2024-03-23T22:59:27.212909+0000","last_active":"2024-03-23T22:59:28.863451+0000","last_peered":"2024-03-23T22:59:28.863451+0000","last_clean":"2024-03-23T22:59:28.863451+0000","last_became_active":"2024-03-23T22:59:27.212504+0000","last_became_peered":"2024-03-23T22:59:27.212504+0000","last_unstale":"2024-03-23T22:59:28.863451+0000","last_undegraded":"2024-03-23T22:59:28.863451+0000","last_fullsized":"2024-03-23T22:59:28.863451+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:09.238272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.608248+0000","last_change":"2024-03-23T22:59:26.452577+0000","last_active":"2024-03-23T22:59:29.608248+0000","last_peered":"2024-03-23T22:59:29.608248+0000","last_clean":"2024-03-23T22:59:29.608248+0000","last_became_active":"2024-03-23T22:59:26.452107+0000","last_became_peered":"2024-03-23T22:59:26.452107+0000","last_unstale":"2024-03-23T22:59:29.608248+0000","last_undegraded":"2024-03-23T22:59:29.608248+0000","last_fullsized":"2024-03-23T22:59:29.608248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:32:18.472334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.863589+0000","last_change":"2024-03-23T22:59:27.188539+0000","last_active":"2024-03-23T22:59:28.863589+0000","last_peered":"2024-03-23T22:59:28.863589+0000","last_clean":"2024-03-23T22:59:28.863589+0000","last_became_active":"2024-03-23T22:59:27.188184+0000","last_became_peered":"2024-03-23T22:59:27.188184+0000","last_unstale":"2024-03-23T22:59:28.863589+0000","last_undegraded":"2024-03-23T22:59:28.863589+0000","last_fullsized":"2024-03-23T22:59:28.863589+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:37:59.756492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.761432+0000","last_change":"2024-03-23T22:59:27.192083+0000","last_active":"2024-03-23T22:59:29.761432+0000","last_peered":"2024-03-23T22:59:29.761432+0000","last_clean":"2024-03-23T22:59:29.761432+0000","last_became_active":"2024-03-23T22:59:27.191709+0000","last_became_peered":"2024-03-23T22:59:27.191709+0000","last_unstale":"2024-03-23T22:59:29.761432+0000","last_undegraded":"2024-03-23T22:59:29.761432+0000","last_fullsized":"2024-03-23T22:59:29.761432+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:18:42.905177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.606521+0000","last_change":"2024-03-23T22:59:27.186315+0000","last_active":"2024-03-23T22:59:29.606521+0000","last_peered":"2024-03-23T22:59:29.606521+0000","last_clean":"2024-03-23T22:59:29.606521+0000","last_became_active":"2024-03-23T22:59:27.185960+0000","last_became_peered":"2024-03-23T22:59:27.185960+0000","last_unstale":"2024-03-23T22:59:29.606521+0000","last_undegraded":"2024-03-23T22:59:29.606521+0000","last_fullsized":"2024-03-23T22:59:29.606521+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:57:07.739725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.606619+0000","last_change":"2024-03-23T22:59:27.221696+0000","last_active":"2024-03-23T22:59:29.606619+0000","last_peered":"2024-03-23T22:59:29.606619+0000","last_clean":"2024-03-23T22:59:29.606619+0000","last_became_active":"2024-03-23T22:59:27.221400+0000","last_became_peered":"2024-03-23T22:59:27.221400+0000","last_unstale":"2024-03-23T22:59:29.606619+0000","last_undegraded":"2024-03-23T22:59:29.606619+0000","last_fullsized":"2024-03-23T22:59:29.606619+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:17.905737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.608462+0000","last_change":"2024-03-23T22:59:27.206286+0000","last_active":"2024-03-23T22:59:29.608462+0000","last_peered":"2024-03-23T22:59:29.608462+0000","last_clean":"2024-03-23T22:59:29.608462+0000","last_became_active":"2024-03-23T22:59:27.205854+0000","last_became_peered":"2024-03-23T22:59:27.205854+0000","last_unstale":"2024-03-23T22:59:29.608462+0000","last_undegraded":"2024-03-23T22:59:29.608462+0000","last_fullsized":"2024-03-23T22:59:29.608462+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:00:45.216211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.609037+0000","last_change":"2024-03-23T22:59:27.211376+0000","last_active":"2024-03-23T22:59:29.609037+0000","last_peered":"2024-03-23T22:59:29.609037+0000","last_clean":"2024-03-23T22:59:29.609037+0000","last_became_active":"2024-03-23T22:59:27.211061+0000","last_became_peered":"2024-03-23T22:59:27.211061+0000","last_unstale":"2024-03-23T22:59:29.609037+0000","last_undegraded":"2024-03-23T22:59:29.609037+0000","last_fullsized":"2024-03-23T22:59:29.609037+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:53:25.776927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.761562+0000","last_change":"2024-03-23T22:59:27.191028+0000","last_active":"2024-03-23T22:59:29.761562+0000","last_peered":"2024-03-23T22:59:29.761562+0000","last_clean":"2024-03-23T22:59:29.761562+0000","last_became_active":"2024-03-23T22:59:27.190634+0000","last_became_peered":"2024-03-23T22:59:27.190634+0000","last_unstale":"2024-03-23T22:59:29.761562+0000","last_undegraded":"2024-03-23T22:59:29.761562+0000","last_fullsized":"2024-03-23T22:59:29.761562+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:50:52.110636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.606910+0000","last_change":"2024-03-23T22:59:27.183235+0000","last_active":"2024-03-23T22:59:29.606910+0000","last_peered":"2024-03-23T22:59:29.606910+0000","last_clean":"2024-03-23T22:59:29.606910+0000","last_became_active":"2024-03-23T22:59:27.181682+0000","last_became_peered":"2024-03-23T22:59:27.181682+0000","last_unstale":"2024-03-23T22:59:29.606910+0000","last_undegraded":"2024-03-23T22:59:29.606910+0000","last_fullsized":"2024-03-23T22:59:29.606910+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:33:26.002324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.174878+0000","last_change":"2024-03-23T22:59:27.212788+0000","last_active":"2024-03-23T22:59:28.174878+0000","last_peered":"2024-03-23T22:59:28.174878+0000","last_clean":"2024-03-23T22:59:28.174878+0000","last_became_active":"2024-03-23T22:59:27.211101+0000","last_became_peered":"2024-03-23T22:59:27.211101+0000","last_unstale":"2024-03-23T22:59:28.174878+0000","last_undegraded":"2024-03-23T22:59:28.174878+0000","last_fullsized":"2024-03-23T22:59:28.174878+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:54:59.599903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.172349+0000","last_change":"2024-03-23T22:59:27.215230+0000","last_active":"2024-03-23T22:59:28.172349+0000","last_peered":"2024-03-23T22:59:28.172349+0000","last_clean":"2024-03-23T22:59:28.172349+0000","last_became_active":"2024-03-23T22:59:27.214881+0000","last_became_peered":"2024-03-23T22:59:27.214881+0000","last_unstale":"2024-03-23T22:59:28.172349+0000","last_undegraded":"2024-03-23T22:59:28.172349+0000","last_fullsized":"2024-03-23T22:59:28.172349+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:10:40.428481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:28.173247+0000","last_change":"2024-03-23T22:59:27.213454+0000","last_active":"2024-03-23T22:59:28.173247+0000","last_peered":"2024-03-23T22:59:28.173247+0000","last_clean":"2024-03-23T22:59:28.173247+0000","last_became_active":"2024-03-23T22:59:27.212684+0000","last_became_peered":"2024-03-23T22:59:27.212684+0000","last_unstale":"2024-03-23T22:59:28.173247+0000","last_undegraded":"2024-03-23T22:59:28.173247+0000","last_fullsized":"2024-03-23T22:59:28.173247+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:48:51.584892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.606558+0000","last_change":"2024-03-23T22:59:27.220414+0000","last_active":"2024-03-23T22:59:29.606558+0000","last_peered":"2024-03-23T22:59:29.606558+0000","last_clean":"2024-03-23T22:59:29.606558+0000","last_became_active":"2024-03-23T22:59:27.220020+0000","last_became_peered":"2024-03-23T22:59:27.220020+0000","last_unstale":"2024-03-23T22:59:29.606558+0000","last_undegraded":"2024-03-23T22:59:29.606558+0000","last_fullsized":"2024-03-23T22:59:29.606558+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:35:48.890004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.609276+0000","last_change":"2024-03-23T22:59:27.215648+0000","last_active":"2024-03-23T22:59:29.609276+0000","last_peered":"2024-03-23T22:59:29.609276+0000","last_clean":"2024-03-23T22:59:29.609276+0000","last_became_active":"2024-03-23T22:59:27.212252+0000","last_became_peered":"2024-03-23T22:59:27.212252+0000","last_unstale":"2024-03-23T22:59:29.609276+0000","last_undegraded":"2024-03-23T22:59:29.609276+0000","last_fullsized":"2024-03-23T22:59:29.609276+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:08:38.956508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-23T22:59:29.761439+0000","last_change":"2024-03-23T22:59:27.190527+0000","last_active":"2024-03-23T22:59:29.761439+0000","last_peered":"2024-03-23T22:59:29.761439+0000","last_clean":"2024-03-23T22:59:29.761439+0000","last_became_active":"2024-03-23T22:59:27.190026+0000","last_became_peered":"2024-03-23T22:59:27.190026+0000","last_unstale":"2024-03-23T22:59:29.761439+0000","last_undegraded":"2024-03-23T22:59:29.761439+0000","last_fullsized":"2024-03-23T22:59:29.761439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:31:24.663584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247269,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4600,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":22,"num_read_kb":18,"num_write":115,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":249856,"data_stored":50253,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350991,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":442154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612627,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":1035130,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776153,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":442356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037792,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":442310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299428,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":1033144,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462955,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":929792,"data_stored":441779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":933888,"data_stored":443184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738444,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":925696,"data_stored":441662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411395,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":991232,"data_stored":481115,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149756,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":441856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888117,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":925696,"data_stored":441594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724593,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":1033533,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34152,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34152,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":39543,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T22:59:36.752 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-23T22:59:36.753 INFO:tasks.cephadm:Setting up client nodes... 2024-03-23T22:59:36.753 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-23T22:59:36.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:36 smithi012 ceph-mon[31695]: pgmap v322: 97 pgs: 97 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4.4 KiB/s wr, 16 op/s 2024-03-23T22:59:36.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:36 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1828720937' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T22:59:36.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:36 smithi012 ceph-mon[31695]: from='client.24724 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T22:59:36.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:36 smithi173 ceph-mon[36116]: pgmap v322: 97 pgs: 97 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4.4 KiB/s wr, 16 op/s 2024-03-23T22:59:36.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:36 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1828720937' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T22:59:36.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:36 smithi173 ceph-mon[36116]: from='client.24724 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T22:59:36.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:36 smithi203 ceph-mon[36693]: pgmap v322: 97 pgs: 97 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4.4 KiB/s wr, 16 op/s 2024-03-23T22:59:36.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:36 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1828720937' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T22:59:36.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:36 smithi203 ceph-mon[36693]: from='client.24724 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T22:59:37.259 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:59:37.744 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:37 smithi012 ceph-mon[31695]: from='client.24737 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T22:59:37.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:37 smithi173 ceph-mon[36116]: from='client.24737 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T22:59:37.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:37 smithi203 ceph-mon[36693]: from='client.24737 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T22:59:38.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:38 smithi012 ceph-mon[31695]: pgmap v323: 97 pgs: 97 active+clean; 582 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 16 op/s 2024-03-23T22:59:38.819 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:38 smithi173 ceph-mon[36116]: pgmap v323: 97 pgs: 97 active+clean; 582 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 16 op/s 2024-03-23T22:59:38.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:38 smithi203 ceph-mon[36693]: pgmap v323: 97 pgs: 97 active+clean; 582 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 16 op/s 2024-03-23T22:59:39.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.583+0000 7fb5c7600700 1 -- 172.21.15.12:0/4104660642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c0103db0 msgr2=0x7fb5c010a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:39.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.583+0000 7fb5c7600700 1 --2- 172.21.15.12:0/4104660642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c0103db0 0x7fb5c010a660 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800b0d0 tx=0x7fb5b800b3e0 comp rx=0 tx=0).stop 2024-03-23T22:59:39.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.583+0000 7fb5c7600700 1 -- 172.21.15.12:0/4104660642 shutdown_connections 2024-03-23T22:59:39.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.583+0000 7fb5c7600700 1 --2- 172.21.15.12:0/4104660642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c0103db0 0x7fb5c010a660 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:39.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.583+0000 7fb5c7600700 1 --2- 172.21.15.12:0/4104660642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5c0102210 0x7fb5c0102630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:39.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.583+0000 7fb5c7600700 1 --2- 172.21.15.12:0/4104660642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5c0103410 0x7fb5c0103870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:39.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.583+0000 7fb5c7600700 1 -- 172.21.15.12:0/4104660642 >> 172.21.15.12:0/4104660642 conn(0x7fb5c00fd7b0 msgr2=0x7fb5c00ffbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:39.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.583+0000 7fb5c7600700 1 -- 172.21.15.12:0/4104660642 shutdown_connections 2024-03-23T22:59:39.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.583+0000 7fb5c7600700 1 -- 172.21.15.12:0/4104660642 wait complete. 2024-03-23T22:59:39.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.584+0000 7fb5c7600700 1 Processor -- start 2024-03-23T22:59:39.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.584+0000 7fb5c7600700 1 -- start start 2024-03-23T22:59:39.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.585+0000 7fb5c7600700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5c0102210 0x7fb5c019a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:39.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.585+0000 7fb5c7600700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c0103410 0x7fb5c019a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:39.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.585+0000 7fb5c7600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5c0103db0 0x7fb5c01a1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:39.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.585+0000 7fb5c7600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c010de10 con 0x7fb5c0103db0 2024-03-23T22:59:39.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.585+0000 7fb5c7600700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c010dc90 con 0x7fb5c0103410 2024-03-23T22:59:39.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.585+0000 7fb5c7600700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c010df90 con 0x7fb5c0102210 2024-03-23T22:59:39.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.585+0000 7fb5bfe00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c0103410 0x7fb5c019a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:39.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.585+0000 7fb5c4c00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5c0102210 0x7fb5c019a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:39.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.585+0000 7fb5bfe00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c0103410 0x7fb5c019a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:46462/0 (socket says 172.21.15.12:46462) 2024-03-23T22:59:39.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.585+0000 7fb5bfe00700 1 -- 172.21.15.12:0/1214621280 learned_addr learned my addr 172.21.15.12:0/1214621280 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:39.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.585+0000 7fb5c5600700 1 --2- 172.21.15.12:0/1214621280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5c0103db0 0x7fb5c01a1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:39.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.585+0000 7fb5c4c00700 1 -- 172.21.15.12:0/1214621280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c0103410 msgr2=0x7fb5c019a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:39.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.585+0000 7fb5c4c00700 1 --2- 172.21.15.12:0/1214621280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c0103410 0x7fb5c019a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:39.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.585+0000 7fb5c4c00700 1 -- 172.21.15.12:0/1214621280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5c0103db0 msgr2=0x7fb5c01a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:39.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.585+0000 7fb5c4c00700 1 --2- 172.21.15.12:0/1214621280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5c0103db0 0x7fb5c01a1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:39.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.585+0000 7fb5c4c00700 1 -- 172.21.15.12:0/1214621280 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c01a2110 con 0x7fb5c0102210 2024-03-23T22:59:39.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.585+0000 7fb5bfe00700 1 --2- 172.21.15.12:0/1214621280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c0103410 0x7fb5c019a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T22:59:39.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.585+0000 7fb5c5600700 1 --2- 172.21.15.12:0/1214621280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5c0103db0 0x7fb5c01a1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T22:59:39.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.586+0000 7fb5c4c00700 1 --2- 172.21.15.12:0/1214621280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5c0102210 0x7fb5c019a4b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb5b400e9f0 tx=0x7fb5b400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:39.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.586+0000 7fb5bd600700 1 -- 172.21.15.12:0/1214621280 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b400cbd0 con 0x7fb5c0102210 2024-03-23T22:59:39.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.586+0000 7fb5bd600700 1 -- 172.21.15.12:0/1214621280 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5b4004500 con 0x7fb5c0102210 2024-03-23T22:59:39.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.586+0000 7fb5bd600700 1 -- 172.21.15.12:0/1214621280 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b4010430 con 0x7fb5c0102210 2024-03-23T22:59:39.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.586+0000 7fb5c7600700 1 -- 172.21.15.12:0/1214621280 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5c01a2400 con 0x7fb5c0102210 2024-03-23T22:59:39.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.586+0000 7fb5c7600700 1 -- 172.21.15.12:0/1214621280 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c01a28f0 con 0x7fb5c0102210 2024-03-23T22:59:39.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.587+0000 7fb5c7600700 1 -- 172.21.15.12:0/1214621280 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c00506a0 con 0x7fb5c0102210 2024-03-23T22:59:39.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.589+0000 7fb5bd600700 1 -- 172.21.15.12:0/1214621280 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fb5b4003e90 con 0x7fb5c0102210 2024-03-23T22:59:39.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.591+0000 7fb5bd600700 1 --2- 172.21.15.12:0/1214621280 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb5a80a2a50 0x7fb5a80a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:39.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.591+0000 7fb5bd600700 1 -- 172.21.15.12:0/1214621280 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb5b40ced80 con 0x7fb5c0102210 2024-03-23T22:59:39.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.591+0000 7fb5bfe00700 1 --2- 172.21.15.12:0/1214621280 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb5a80a2a50 0x7fb5a80a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:39.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.592+0000 7fb5bfe00700 1 --2- 172.21.15.12:0/1214621280 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb5a80a2a50 0x7fb5a80a4f10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac011590 tx=0x7fb5ac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:39.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.594+0000 7fb5bd600700 1 -- 172.21.15.12:0/1214621280 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb5b409bd70 con 0x7fb5c0102210 2024-03-23T22:59:39.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.907+0000 7fb5c7600700 1 -- 172.21.15.12:0/1214621280 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fb5c01a2bb0 con 0x7fb5c0102210 2024-03-23T22:59:39.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.915+0000 7fb5bd600700 1 -- 172.21.15.12:0/1214621280 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7fb5b409b670 con 0x7fb5c0102210 2024-03-23T22:59:39.915 INFO:teuthology.orchestra.run.smithi012.stdout:[client.0] 2024-03-23T22:59:39.915 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQDbXv9lfPE+NhAAScZ+QCXSRCrT45LYOXr3qA== 2024-03-23T22:59:39.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.918+0000 7fb5c7600700 1 -- 172.21.15.12:0/1214621280 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb5a80a2a50 msgr2=0x7fb5a80a4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:39.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.918+0000 7fb5c7600700 1 --2- 172.21.15.12:0/1214621280 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb5a80a2a50 0x7fb5a80a4f10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac011590 tx=0x7fb5ac009260 comp rx=0 tx=0).stop 2024-03-23T22:59:39.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.918+0000 7fb5c7600700 1 -- 172.21.15.12:0/1214621280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5c0102210 msgr2=0x7fb5c019a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:39.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.918+0000 7fb5c7600700 1 --2- 172.21.15.12:0/1214621280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5c0102210 0x7fb5c019a4b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb5b400e9f0 tx=0x7fb5b400ed00 comp rx=0 tx=0).stop 2024-03-23T22:59:39.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.918+0000 7fb5c7600700 1 -- 172.21.15.12:0/1214621280 shutdown_connections 2024-03-23T22:59:39.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.918+0000 7fb5c7600700 1 --2- 172.21.15.12:0/1214621280 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb5a80a2a50 0x7fb5a80a4f10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:39.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.918+0000 7fb5c7600700 1 --2- 172.21.15.12:0/1214621280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5c0102210 0x7fb5c019a4b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:39.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.918+0000 7fb5c7600700 1 --2- 172.21.15.12:0/1214621280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5c0103db0 0x7fb5c01a1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:39.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.918+0000 7fb5c7600700 1 --2- 172.21.15.12:0/1214621280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c0103410 0x7fb5c019a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:39.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.918+0000 7fb5c7600700 1 -- 172.21.15.12:0/1214621280 >> 172.21.15.12:0/1214621280 conn(0x7fb5c00fd7b0 msgr2=0x7fb5c00743c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:39.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.918+0000 7fb5c7600700 1 -- 172.21.15.12:0/1214621280 shutdown_connections 2024-03-23T22:59:39.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:39.918+0000 7fb5c7600700 1 -- 172.21.15.12:0/1214621280 wait complete. 2024-03-23T22:59:40.639 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:40 smithi012 ceph-mon[31695]: pgmap v324: 97 pgs: 97 active+clean; 623 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 7.6 KiB/s wr, 17 op/s 2024-03-23T22:59:40.640 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:40 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1214621280' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-23T22:59:40.640 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:40 smithi012 ceph-mon[31695]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-23T22:59:40.640 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:40 smithi012 ceph-mon[31695]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-23T22:59:40.717 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-23T22:59:40.717 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-23T22:59:40.718 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-23T22:59:40.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:40 smithi173 ceph-mon[36116]: pgmap v324: 97 pgs: 97 active+clean; 623 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 7.6 KiB/s wr, 17 op/s 2024-03-23T22:59:40.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:40 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1214621280' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-23T22:59:40.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:40 smithi173 ceph-mon[36116]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-23T22:59:40.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:40 smithi173 ceph-mon[36116]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-23T22:59:40.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:40 smithi203 ceph-mon[36693]: pgmap v324: 97 pgs: 97 active+clean; 623 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 7.6 KiB/s wr, 17 op/s 2024-03-23T22:59:40.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:40 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1214621280' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-23T22:59:40.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:40 smithi203 ceph-mon[36693]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-23T22:59:40.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:40 smithi203 ceph-mon[36693]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-23T22:59:40.915 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-23T22:59:40.916 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-23T22:59:40.916 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-23T22:59:41.454 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:59:42.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:42 smithi012 ceph-mon[31695]: pgmap v325: 97 pgs: 97 active+clean; 623 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 7.0 KiB/s wr, 17 op/s 2024-03-23T22:59:42.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:42 smithi173 ceph-mon[36116]: pgmap v325: 97 pgs: 97 active+clean; 623 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 7.0 KiB/s wr, 17 op/s 2024-03-23T22:59:42.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:42 smithi203 ceph-mon[36693]: pgmap v325: 97 pgs: 97 active+clean; 623 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 7.0 KiB/s wr, 17 op/s 2024-03-23T22:59:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.918+0000 7fc31b200700 1 -- 172.21.15.12:0/27110089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3141081c0 msgr2=0x7fc3141085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.918+0000 7fc31b200700 1 --2- 172.21.15.12:0/27110089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3141081c0 0x7fc3141085a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc304009990 tx=0x7fc304009ca0 comp rx=0 tx=0).stop 2024-03-23T22:59:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.918+0000 7fc31b200700 1 -- 172.21.15.12:0/27110089 shutdown_connections 2024-03-23T22:59:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.918+0000 7fc31b200700 1 --2- 172.21.15.12:0/27110089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3141081c0 0x7fc3141085a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.918+0000 7fc31b200700 1 --2- 172.21.15.12:0/27110089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc314102ba0 0x7fc31410efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.918+0000 7fc31b200700 1 --2- 172.21.15.12:0/27110089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3141021e0 0x7fc314102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.918+0000 7fc31b200700 1 -- 172.21.15.12:0/27110089 >> 172.21.15.12:0/27110089 conn(0x7fc3140792c0 msgr2=0x7fc3140796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.918+0000 7fc31b200700 1 -- 172.21.15.12:0/27110089 shutdown_connections 2024-03-23T22:59:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.918+0000 7fc31b200700 1 -- 172.21.15.12:0/27110089 wait complete. 2024-03-23T22:59:43.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.920+0000 7fc31b200700 1 Processor -- start 2024-03-23T22:59:43.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.920+0000 7fc31b200700 1 -- start start 2024-03-23T22:59:43.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.921+0000 7fc31b200700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3141021e0 0x7fc31419e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:43.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.921+0000 7fc31b200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc314102ba0 0x7fc31419eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.921+0000 7fc31b200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3141081c0 0x7fc3141987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.921+0000 7fc31b200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc314112490 con 0x7fc3141081c0 2024-03-23T22:59:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.921+0000 7fc31b200700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc314112310 con 0x7fc3141021e0 2024-03-23T22:59:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.921+0000 7fc31b200700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc314112610 con 0x7fc314102ba0 2024-03-23T22:59:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.921+0000 7fc313e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3141021e0 0x7fc31419e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.921+0000 7fc313e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3141021e0 0x7fc31419e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:52004/0 (socket says 172.21.15.12:52004) 2024-03-23T22:59:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.921+0000 7fc313e00700 1 -- 172.21.15.12:0/1670690116 learned_addr learned my addr 172.21.15.12:0/1670690116 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.921+0000 7fc313400700 1 --2- 172.21.15.12:0/1670690116 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc314102ba0 0x7fc31419eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.921+0000 7fc319200700 1 --2- 172.21.15.12:0/1670690116 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3141081c0 0x7fc3141987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.921+0000 7fc313e00700 1 -- 172.21.15.12:0/1670690116 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc314102ba0 msgr2=0x7fc31419eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.921+0000 7fc313e00700 1 --2- 172.21.15.12:0/1670690116 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc314102ba0 0x7fc31419eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.921+0000 7fc313e00700 1 -- 172.21.15.12:0/1670690116 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3141081c0 msgr2=0x7fc3141987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:43.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.921+0000 7fc313e00700 1 --2- 172.21.15.12:0/1670690116 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3141081c0 0x7fc3141987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:43.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.921+0000 7fc313e00700 1 -- 172.21.15.12:0/1670690116 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc314198fb0 con 0x7fc3141021e0 2024-03-23T22:59:43.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.922+0000 7fc313e00700 1 --2- 172.21.15.12:0/1670690116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3141021e0 0x7fc31419e600 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc304000c00 tx=0x7fc304005590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:43.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.922+0000 7fc310c00700 1 -- 172.21.15.12:0/1670690116 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc304027070 con 0x7fc3141021e0 2024-03-23T22:59:43.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.922+0000 7fc310c00700 1 -- 172.21.15.12:0/1670690116 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc30400fda0 con 0x7fc3141021e0 2024-03-23T22:59:43.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.922+0000 7fc310c00700 1 -- 172.21.15.12:0/1670690116 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3040173f0 con 0x7fc3141021e0 2024-03-23T22:59:43.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.922+0000 7fc31b200700 1 -- 172.21.15.12:0/1670690116 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc314199240 con 0x7fc3141021e0 2024-03-23T22:59:43.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.922+0000 7fc31b200700 1 -- 172.21.15.12:0/1670690116 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3141a5460 con 0x7fc3141021e0 2024-03-23T22:59:43.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.923+0000 7fc31b200700 1 -- 172.21.15.12:0/1670690116 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc314103870 con 0x7fc3141021e0 2024-03-23T22:59:43.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.925+0000 7fc310c00700 1 -- 172.21.15.12:0/1670690116 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fc30402a070 con 0x7fc3141021e0 2024-03-23T22:59:43.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.926+0000 7fc310c00700 1 --2- 172.21.15.12:0/1670690116 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc2f80a29a0 0x7fc2f80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:43.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.927+0000 7fc310c00700 1 -- 172.21.15.12:0/1670690116 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fc3040cf860 con 0x7fc3141021e0 2024-03-23T22:59:43.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.927+0000 7fc313400700 1 --2- 172.21.15.12:0/1670690116 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc2f80a29a0 0x7fc2f80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:43.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.927+0000 7fc313400700 1 --2- 172.21.15.12:0/1670690116 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc2f80a29a0 0x7fc2f80a4e60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc3141070d0 tx=0x7fc300009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:43.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:43.930+0000 7fc310c00700 1 -- 172.21.15.12:0/1670690116 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc30405e740 con 0x7fc3141021e0 2024-03-23T22:59:44.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:44.225+0000 7fc31b200700 1 -- 172.21.15.12:0/1670690116 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fc31402da30 con 0x7fc3141021e0 2024-03-23T22:59:44.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:44.231+0000 7fc310c00700 1 -- 172.21.15.12:0/1670690116 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+262661 (secure 0 0 0) 0x7fc304013070 con 0x7fc3141021e0 2024-03-23T22:59:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:59:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:44.238+0000 7fc31b200700 1 -- 172.21.15.12:0/1670690116 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc2f80a29a0 msgr2=0x7fc2f80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:44.238+0000 7fc31b200700 1 --2- 172.21.15.12:0/1670690116 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc2f80a29a0 0x7fc2f80a4e60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc3141070d0 tx=0x7fc300009260 comp rx=0 tx=0).stop 2024-03-23T22:59:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:44.238+0000 7fc31b200700 1 -- 172.21.15.12:0/1670690116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3141021e0 msgr2=0x7fc31419e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:44.238+0000 7fc31b200700 1 --2- 172.21.15.12:0/1670690116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3141021e0 0x7fc31419e600 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc304000c00 tx=0x7fc304005590 comp rx=0 tx=0).stop 2024-03-23T22:59:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:44.238+0000 7fc31b200700 1 -- 172.21.15.12:0/1670690116 shutdown_connections 2024-03-23T22:59:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:44.238+0000 7fc31b200700 1 --2- 172.21.15.12:0/1670690116 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc2f80a29a0 0x7fc2f80a4e60 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:44.238+0000 7fc31b200700 1 --2- 172.21.15.12:0/1670690116 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3141081c0 0x7fc3141987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:44.238+0000 7fc31b200700 1 --2- 172.21.15.12:0/1670690116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3141021e0 0x7fc31419e600 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:44.238+0000 7fc31b200700 1 --2- 172.21.15.12:0/1670690116 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc314102ba0 0x7fc31419eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:44.238+0000 7fc31b200700 1 -- 172.21.15.12:0/1670690116 >> 172.21.15.12:0/1670690116 conn(0x7fc3140792c0 msgr2=0x7fc314100eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:44.238+0000 7fc31b200700 1 -- 172.21.15.12:0/1670690116 shutdown_connections 2024-03-23T22:59:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:44.238+0000 7fc31b200700 1 -- 172.21.15.12:0/1670690116 wait complete. 2024-03-23T22:59:44.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:44 smithi012 ceph-mon[31695]: pgmap v326: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.2 KiB/s wr, 17 op/s 2024-03-23T22:59:44.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:44 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1670690116' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-23T22:59:44.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:44 smithi173 ceph-mon[36116]: pgmap v326: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.2 KiB/s wr, 17 op/s 2024-03-23T22:59:44.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:44 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1670690116' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-23T22:59:44.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:44 smithi203 ceph-mon[36693]: pgmap v326: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.2 KiB/s wr, 17 op/s 2024-03-23T22:59:44.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:44 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1670690116' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-23T22:59:45.169 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":1345467734},{"type":"v1","addr":"172.21.15.12:6801","nonce":1345467734}]},"active_addr":"172.21.15.12:6801/1345467734","active_change":"2024-03-23T22:51:23.113484+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.12:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":3160893115}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":2110872719}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":3634400133}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":1818821458}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":1730711914}]}]} 2024-03-23T22:59:45.175 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-23T22:59:45.175 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-23T22:59:45.175 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-23T22:59:45.621 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:59:46.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:46 smithi012 ceph-mon[31695]: pgmap v327: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 6.2 KiB/s wr, 12 op/s 2024-03-23T22:59:46.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:46 smithi173 ceph-mon[36116]: pgmap v327: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 6.2 KiB/s wr, 12 op/s 2024-03-23T22:59:46.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:46 smithi203 ceph-mon[36693]: pgmap v327: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 6.2 KiB/s wr, 12 op/s 2024-03-23T22:59:47.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.806+0000 7f0280a00700 1 -- 172.21.15.12:0/3236823723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f027c10c660 msgr2=0x7f027c10eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:47.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.806+0000 7f0280a00700 1 --2- 172.21.15.12:0/3236823723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f027c10c660 0x7f027c10eb20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f027400b0a0 tx=0x7f027400b3b0 comp rx=0 tx=0).stop 2024-03-23T22:59:47.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.806+0000 7f0280a00700 1 -- 172.21.15.12:0/3236823723 shutdown_connections 2024-03-23T22:59:47.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.806+0000 7f0280a00700 1 --2- 172.21.15.12:0/3236823723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f027c10c660 0x7f027c10eb20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:47.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.806+0000 7f0280a00700 1 --2- 172.21.15.12:0/3236823723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f027c103230 0x7f027c103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:47.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.806+0000 7f0280a00700 1 --2- 172.21.15.12:0/3236823723 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f027c103b50 0x7f027c10c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:47.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.806+0000 7f0280a00700 1 -- 172.21.15.12:0/3236823723 >> 172.21.15.12:0/3236823723 conn(0x7f027c0fca60 msgr2=0x7f027c0fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:47.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.807+0000 7f0280a00700 1 -- 172.21.15.12:0/3236823723 shutdown_connections 2024-03-23T22:59:47.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.807+0000 7f0280a00700 1 -- 172.21.15.12:0/3236823723 wait complete. 2024-03-23T22:59:47.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.809+0000 7f0280a00700 1 Processor -- start 2024-03-23T22:59:47.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.809+0000 7f0280a00700 1 -- start start 2024-03-23T22:59:47.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.810+0000 7f0280a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f027c103230 0x7f027c19e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:47.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.810+0000 7f0280a00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f027c103b50 0x7f027c19eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:47.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.810+0000 7f0280a00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f027c10c660 0x7f027c198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:47.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.810+0000 7f0280a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027c1123b0 con 0x7f027c103230 2024-03-23T22:59:47.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.810+0000 7f0280a00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027c112230 con 0x7f027c103b50 2024-03-23T22:59:47.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.810+0000 7f0280a00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027c112530 con 0x7f027c10c660 2024-03-23T22:59:47.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.810+0000 7f027b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f027c103230 0x7f027c19e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:47.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.810+0000 7f027aa00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f027c103b50 0x7f027c19eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:47.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.810+0000 7f027aa00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f027c103b50 0x7f027c19eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:52028/0 (socket says 172.21.15.12:52028) 2024-03-23T22:59:47.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.810+0000 7f027aa00700 1 -- 172.21.15.12:0/2326842974 learned_addr learned my addr 172.21.15.12:0/2326842974 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:47.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.810+0000 7f027be00700 1 --2- 172.21.15.12:0/2326842974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f027c10c660 0x7f027c198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:47.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.810+0000 7f027aa00700 1 -- 172.21.15.12:0/2326842974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f027c10c660 msgr2=0x7f027c198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:47.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.810+0000 7f027aa00700 1 --2- 172.21.15.12:0/2326842974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f027c10c660 0x7f027c198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:47.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.810+0000 7f027aa00700 1 -- 172.21.15.12:0/2326842974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f027c103230 msgr2=0x7f027c19e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:47.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.810+0000 7f027aa00700 1 --2- 172.21.15.12:0/2326842974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f027c103230 0x7f027c19e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:47.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.810+0000 7f027aa00700 1 -- 172.21.15.12:0/2326842974 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027c198f10 con 0x7f027c103b50 2024-03-23T22:59:47.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.811+0000 7f027aa00700 1 --2- 172.21.15.12:0/2326842974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f027c103b50 0x7f027c19eaa0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f026800e9c0 tx=0x7f026800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:47.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.811+0000 7f0273e00700 1 -- 172.21.15.12:0/2326842974 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0268004a30 con 0x7f027c103b50 2024-03-23T22:59:47.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.811+0000 7f0273e00700 1 -- 172.21.15.12:0/2326842974 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f026800cca0 con 0x7f027c103b50 2024-03-23T22:59:47.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.811+0000 7f0280a00700 1 -- 172.21.15.12:0/2326842974 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f027c199200 con 0x7f027c103b50 2024-03-23T22:59:47.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.811+0000 7f0273e00700 1 -- 172.21.15.12:0/2326842974 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0268005610 con 0x7f027c103b50 2024-03-23T22:59:47.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.811+0000 7f0280a00700 1 -- 172.21.15.12:0/2326842974 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f027c1a54e0 con 0x7f027c103b50 2024-03-23T22:59:47.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.812+0000 7f0280a00700 1 -- 172.21.15.12:0/2326842974 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f027c06a720 con 0x7f027c103b50 2024-03-23T22:59:47.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.814+0000 7f0273e00700 1 -- 172.21.15.12:0/2326842974 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f026800c7c0 con 0x7f027c103b50 2024-03-23T22:59:47.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.815+0000 7f0273e00700 1 --2- 172.21.15.12:0/2326842974 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f02600a29a0 0x7f02600a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:47.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.816+0000 7f0273e00700 1 -- 172.21.15.12:0/2326842974 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f02680ceb60 con 0x7f027c103b50 2024-03-23T22:59:47.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.816+0000 7f027b400700 1 --2- 172.21.15.12:0/2326842974 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f02600a29a0 0x7f02600a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:47.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.816+0000 7f027b400700 1 --2- 172.21.15.12:0/2326842974 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f02600a29a0 0x7f02600a4e60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f026c009c60 tx=0x7f026c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:47.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:47.819+0000 7f0273e00700 1 -- 172.21.15.12:0/2326842974 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f026809bb50 con 0x7f027c103b50 2024-03-23T22:59:48.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:48.047+0000 7f0280a00700 1 -- 172.21.15.12:0/2326842974 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f027c199b60 con 0x7f027c103b50 2024-03-23T22:59:48.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:48.049+0000 7f0273e00700 1 -- 172.21.15.12:0/2326842974 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+22244 (secure 0 0 0) 0x7f026809b450 con 0x7f027c103b50 2024-03-23T22:59:48.049 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:59:48.049 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":86,"fsid":"90f679ae-e967-11ee-95cd-87774f69a715","created":"2024-03-23T22:50:14.906952+0000","modified":"2024-03-23T22:59:28.166927+0000","last_up_change":"2024-03-23T22:58:28.907824+0000","last_in_change":"2024-03-23T22:58:08.342487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T22:54:11.831010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T22:59:25.350974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T22:59:26.390323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b70875d7-a23d-4251-9e8b-1a16d4edc2a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6803","nonce":1190079611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6805","nonce":1190079611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6809","nonce":1190079611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6807","nonce":1190079611}]},"public_addr":"172.21.15.12:6803/1190079611","cluster_addr":"172.21.15.12:6805/1190079611","heartbeat_back_addr":"172.21.15.12:6809/1190079611","heartbeat_front_addr":"172.21.15.12:6807/1190079611","state":["exists","up"]},{"osd":1,"uuid":"416e87af-db5d-41ce-84d9-731e3e07e3c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6801","nonce":1983639525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6803","nonce":1983639525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6807","nonce":1983639525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6805","nonce":1983639525}]},"public_addr":"172.21.15.173:6801/1983639525","cluster_addr":"172.21.15.173:6803/1983639525","heartbeat_back_addr":"172.21.15.173:6807/1983639525","heartbeat_front_addr":"172.21.15.173:6805/1983639525","state":["exists","up"]},{"osd":2,"uuid":"7a559c70-4c54-431b-8baf-4136b6e76493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6801","nonce":2253997387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6803","nonce":2253997387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6807","nonce":2253997387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6805","nonce":2253997387}]},"public_addr":"172.21.15.203:6801/2253997387","cluster_addr":"172.21.15.203:6803/2253997387","heartbeat_back_addr":"172.21.15.203:6807/2253997387","heartbeat_front_addr":"172.21.15.203:6805/2253997387","state":["exists","up"]},{"osd":3,"uuid":"c41cd287-a674-486c-8025-328cd36e5ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6811","nonce":1761446905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6813","nonce":1761446905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6817","nonce":1761446905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6815","nonce":1761446905}]},"public_addr":"172.21.15.12:6811/1761446905","cluster_addr":"172.21.15.12:6813/1761446905","heartbeat_back_addr":"172.21.15.12:6817/1761446905","heartbeat_front_addr":"172.21.15.12:6815/1761446905","state":["exists","up"]},{"osd":4,"uuid":"3c48f886-0b42-4a09-b31e-abccc200bb1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6809","nonce":242958272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6811","nonce":242958272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6815","nonce":242958272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6813","nonce":242958272}]},"public_addr":"172.21.15.173:6809/242958272","cluster_addr":"172.21.15.173:6811/242958272","heartbeat_back_addr":"172.21.15.173:6815/242958272","heartbeat_front_addr":"172.21.15.173:6813/242958272","state":["exists","up"]},{"osd":5,"uuid":"4a34f06f-8805-40c0-b1ab-23407b0276d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6809","nonce":4270966092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6811","nonce":4270966092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6815","nonce":4270966092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6813","nonce":4270966092}]},"public_addr":"172.21.15.203:6809/4270966092","cluster_addr":"172.21.15.203:6811/4270966092","heartbeat_back_addr":"172.21.15.203:6815/4270966092","heartbeat_front_addr":"172.21.15.203:6813/4270966092","state":["exists","up"]},{"osd":6,"uuid":"26ddc663-5623-48c9-80d1-f8b968765d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6819","nonce":3606839295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6821","nonce":3606839295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6825","nonce":3606839295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6823","nonce":3606839295}]},"public_addr":"172.21.15.12:6819/3606839295","cluster_addr":"172.21.15.12:6821/3606839295","heartbeat_back_addr":"172.21.15.12:6825/3606839295","heartbeat_front_addr":"172.21.15.12:6823/3606839295","state":["exists","up"]},{"osd":7,"uuid":"17dc9417-71cd-4576-aa08-933212ec55e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6817","nonce":1241504346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6819","nonce":1241504346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6823","nonce":1241504346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6821","nonce":1241504346}]},"public_addr":"172.21.15.173:6817/1241504346","cluster_addr":"172.21.15.173:6819/1241504346","heartbeat_back_addr":"172.21.15.173:6823/1241504346","heartbeat_front_addr":"172.21.15.173:6821/1241504346","state":["exists","up"]},{"osd":8,"uuid":"90a2b63a-471d-43b8-ac18-bb1f16778d18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6817","nonce":1516996192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6819","nonce":1516996192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6823","nonce":1516996192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6821","nonce":1516996192}]},"public_addr":"172.21.15.203:6817/1516996192","cluster_addr":"172.21.15.203:6819/1516996192","heartbeat_back_addr":"172.21.15.203:6823/1516996192","heartbeat_front_addr":"172.21.15.203:6821/1516996192","state":["exists","up"]},{"osd":9,"uuid":"6e1ca504-40b4-49d0-b245-7946181abfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6827","nonce":1175260774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6829","nonce":1175260774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6833","nonce":1175260774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6831","nonce":1175260774}]},"public_addr":"172.21.15.12:6827/1175260774","cluster_addr":"172.21.15.12:6829/1175260774","heartbeat_back_addr":"172.21.15.12:6833/1175260774","heartbeat_front_addr":"172.21.15.12:6831/1175260774","state":["exists","up"]},{"osd":10,"uuid":"09a595ed-5614-4386-8e61-0170af66e01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6825","nonce":3303506808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6827","nonce":3303506808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6831","nonce":3303506808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6829","nonce":3303506808}]},"public_addr":"172.21.15.173:6825/3303506808","cluster_addr":"172.21.15.173:6827/3303506808","heartbeat_back_addr":"172.21.15.173:6831/3303506808","heartbeat_front_addr":"172.21.15.173:6829/3303506808","state":["exists","up"]},{"osd":11,"uuid":"9ef1d595-a977-4ba7-9c6b-d2b3295f20b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6825","nonce":2323604506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6827","nonce":2323604506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6831","nonce":2323604506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6829","nonce":2323604506}]},"public_addr":"172.21.15.203:6825/2323604506","cluster_addr":"172.21.15.203:6827/2323604506","heartbeat_back_addr":"172.21.15.203:6831/2323604506","heartbeat_front_addr":"172.21.15.203:6829/2323604506","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:18.469038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:42.752628+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:07.069493+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:41.962478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:08.363104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:33.341336+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:07.286541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:34.243302+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:59.749699+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:57:33.822913+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:00.737269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:26.599854+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/4182783120":"2024-03-24T22:51:23.113201+0000","172.21.15.12:6800/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1900938637":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/3414861652":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6801/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/2580541477":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6800/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/501058839":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1192554716":"2024-03-24T22:50:48.401223+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T22:59:48.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:48.052+0000 7f0280a00700 1 -- 172.21.15.12:0/2326842974 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f02600a29a0 msgr2=0x7f02600a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:48.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:48.052+0000 7f0280a00700 1 --2- 172.21.15.12:0/2326842974 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f02600a29a0 0x7f02600a4e60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f026c009c60 tx=0x7f026c009260 comp rx=0 tx=0).stop 2024-03-23T22:59:48.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:48.052+0000 7f0280a00700 1 -- 172.21.15.12:0/2326842974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f027c103b50 msgr2=0x7f027c19eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:48.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:48.052+0000 7f0280a00700 1 --2- 172.21.15.12:0/2326842974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f027c103b50 0x7f027c19eaa0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f026800e9c0 tx=0x7f026800ecd0 comp rx=0 tx=0).stop 2024-03-23T22:59:48.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:48.052+0000 7f0280a00700 1 -- 172.21.15.12:0/2326842974 shutdown_connections 2024-03-23T22:59:48.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:48.052+0000 7f0280a00700 1 --2- 172.21.15.12:0/2326842974 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f02600a29a0 0x7f02600a4e60 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:48.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:48.052+0000 7f0280a00700 1 --2- 172.21.15.12:0/2326842974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f027c10c660 0x7f027c198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:48.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:48.052+0000 7f0280a00700 1 --2- 172.21.15.12:0/2326842974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f027c103230 0x7f027c19e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:48.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:48.052+0000 7f0280a00700 1 --2- 172.21.15.12:0/2326842974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f027c103b50 0x7f027c19eaa0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:48.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:48.052+0000 7f0280a00700 1 -- 172.21.15.12:0/2326842974 >> 172.21.15.12:0/2326842974 conn(0x7f027c0fca60 msgr2=0x7f027c0fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:48.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:48.053+0000 7f0280a00700 1 -- 172.21.15.12:0/2326842974 shutdown_connections 2024-03-23T22:59:48.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:48.053+0000 7f0280a00700 1 -- 172.21.15.12:0/2326842974 wait complete. 2024-03-23T22:59:48.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:48 smithi012 ceph-mon[31695]: pgmap v328: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 5.7 KiB/s wr, 9 op/s 2024-03-23T22:59:48.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:48 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2326842974' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T22:59:48.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:48 smithi173 ceph-mon[36116]: pgmap v328: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 5.7 KiB/s wr, 9 op/s 2024-03-23T22:59:48.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:48 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2326842974' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T22:59:48.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:48 smithi203 ceph-mon[36693]: pgmap v328: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 5.7 KiB/s wr, 9 op/s 2024-03-23T22:59:48.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:48 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2326842974' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T22:59:48.989 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-23T22:59:48.990 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-23T22:59:49.473 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:59:50.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:50 smithi012 ceph-mon[31695]: pgmap v329: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-23T22:59:50.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:50 smithi173 ceph-mon[36116]: pgmap v329: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-23T22:59:50.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:50 smithi203 ceph-mon[36693]: pgmap v329: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-23T22:59:51.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.654+0000 7ff6ce400700 1 -- 172.21.15.12:0/4055459681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6c81021e0 msgr2=0x7ff6c8102660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:51.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.654+0000 7ff6ce400700 1 --2- 172.21.15.12:0/4055459681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6c81021e0 0x7ff6c8102660 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff6b40098f0 tx=0x7ff6b4009c00 comp rx=0 tx=0).stop 2024-03-23T22:59:51.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.654+0000 7ff6ce400700 1 -- 172.21.15.12:0/4055459681 shutdown_connections 2024-03-23T22:59:51.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.654+0000 7ff6ce400700 1 --2- 172.21.15.12:0/4055459681 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6c8102ba0 0x7ff6c810efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:51.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.654+0000 7ff6ce400700 1 --2- 172.21.15.12:0/4055459681 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6c81081c0 0x7ff6c81085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:51.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.654+0000 7ff6ce400700 1 --2- 172.21.15.12:0/4055459681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6c81021e0 0x7ff6c8102660 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:51.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.654+0000 7ff6ce400700 1 -- 172.21.15.12:0/4055459681 >> 172.21.15.12:0/4055459681 conn(0x7ff6c80792c0 msgr2=0x7ff6c80796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:51.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.654+0000 7ff6ce400700 1 -- 172.21.15.12:0/4055459681 shutdown_connections 2024-03-23T22:59:51.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.654+0000 7ff6ce400700 1 -- 172.21.15.12:0/4055459681 wait complete. 2024-03-23T22:59:51.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.655+0000 7ff6ce400700 1 Processor -- start 2024-03-23T22:59:51.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.656+0000 7ff6ce400700 1 -- start start 2024-03-23T22:59:51.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.656+0000 7ff6ce400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6c81021e0 0x7ff6c8074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:51.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.656+0000 7ff6ce400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6c8102ba0 0x7ff6c80749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:51.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.656+0000 7ff6ce400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6c81081c0 0x7ff6c8074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:51.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.656+0000 7ff6ce400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c8112490 con 0x7ff6c8102ba0 2024-03-23T22:59:51.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.656+0000 7ff6ce400700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c8112310 con 0x7ff6c81021e0 2024-03-23T22:59:51.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.656+0000 7ff6ce400700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c8112610 con 0x7ff6c81081c0 2024-03-23T22:59:51.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.656+0000 7ff6c7400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6c81021e0 0x7ff6c8074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:51.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.656+0000 7ff6c7400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6c81021e0 0x7ff6c8074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:52046/0 (socket says 172.21.15.12:52046) 2024-03-23T22:59:51.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.656+0000 7ff6c7400700 1 -- 172.21.15.12:0/3434498794 learned_addr learned my addr 172.21.15.12:0/3434498794 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T22:59:51.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.656+0000 7ff6c7e00700 1 --2- 172.21.15.12:0/3434498794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6c81081c0 0x7ff6c8074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:51.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.656+0000 7ff6c6a00700 1 --2- 172.21.15.12:0/3434498794 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6c8102ba0 0x7ff6c80749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:51.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.657+0000 7ff6c7400700 1 -- 172.21.15.12:0/3434498794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6c81081c0 msgr2=0x7ff6c8074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:51.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.657+0000 7ff6c7400700 1 --2- 172.21.15.12:0/3434498794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6c81081c0 0x7ff6c8074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:51.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.657+0000 7ff6c7400700 1 -- 172.21.15.12:0/3434498794 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6c8102ba0 msgr2=0x7ff6c80749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:51.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.657+0000 7ff6c7400700 1 --2- 172.21.15.12:0/3434498794 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6c8102ba0 0x7ff6c80749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:51.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.657+0000 7ff6c7400700 1 -- 172.21.15.12:0/3434498794 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c81a6e40 con 0x7ff6c81021e0 2024-03-23T22:59:51.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.657+0000 7ff6c7400700 1 --2- 172.21.15.12:0/3434498794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6c81021e0 0x7ff6c8074490 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff6b800e9c0 tx=0x7ff6b800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:51.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.657+0000 7ff6bfe00700 1 -- 172.21.15.12:0/3434498794 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b8004a30 con 0x7ff6c81021e0 2024-03-23T22:59:51.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.657+0000 7ff6bfe00700 1 -- 172.21.15.12:0/3434498794 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff6b800cca0 con 0x7ff6c81021e0 2024-03-23T22:59:51.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.657+0000 7ff6ce400700 1 -- 172.21.15.12:0/3434498794 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6c81a7130 con 0x7ff6c81021e0 2024-03-23T22:59:51.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.657+0000 7ff6bfe00700 1 -- 172.21.15.12:0/3434498794 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b8005610 con 0x7ff6c81021e0 2024-03-23T22:59:51.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.658+0000 7ff6ce400700 1 -- 172.21.15.12:0/3434498794 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6c81a7640 con 0x7ff6c81021e0 2024-03-23T22:59:51.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.659+0000 7ff6ce400700 1 -- 172.21.15.12:0/3434498794 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6c8068e50 con 0x7ff6c81021e0 2024-03-23T22:59:51.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.660+0000 7ff6bfe00700 1 -- 172.21.15.12:0/3434498794 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7ff6b8004cf0 con 0x7ff6c81021e0 2024-03-23T22:59:51.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.662+0000 7ff6bfe00700 1 --2- 172.21.15.12:0/3434498794 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff6ac0a2920 0x7ff6ac0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T22:59:51.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.662+0000 7ff6bfe00700 1 -- 172.21.15.12:0/3434498794 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff6b80ced10 con 0x7ff6c81021e0 2024-03-23T22:59:51.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.662+0000 7ff6c6a00700 1 --2- 172.21.15.12:0/3434498794 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff6ac0a2920 0x7ff6ac0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T22:59:51.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.663+0000 7ff6c6a00700 1 --2- 172.21.15.12:0/3434498794 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff6ac0a2920 0x7ff6ac0a4de0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff6b40098c0 tx=0x7ff6b401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T22:59:51.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.666+0000 7ff6bfe00700 1 -- 172.21.15.12:0/3434498794 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff6b80d4020 con 0x7ff6c81021e0 2024-03-23T22:59:51.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.895+0000 7ff6ce400700 1 -- 172.21.15.12:0/3434498794 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6c802da30 con 0x7ff6c81021e0 2024-03-23T22:59:51.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.897+0000 7ff6bfe00700 1 -- 172.21.15.12:0/3434498794 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+22244 (secure 0 0 0) 0x7ff6b809bd00 con 0x7ff6c81021e0 2024-03-23T22:59:51.898 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T22:59:51.898 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":86,"fsid":"90f679ae-e967-11ee-95cd-87774f69a715","created":"2024-03-23T22:50:14.906952+0000","modified":"2024-03-23T22:59:28.166927+0000","last_up_change":"2024-03-23T22:58:28.907824+0000","last_in_change":"2024-03-23T22:58:08.342487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T22:54:11.831010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T22:59:25.350974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T22:59:26.390323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b70875d7-a23d-4251-9e8b-1a16d4edc2a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6803","nonce":1190079611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6805","nonce":1190079611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6809","nonce":1190079611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6807","nonce":1190079611}]},"public_addr":"172.21.15.12:6803/1190079611","cluster_addr":"172.21.15.12:6805/1190079611","heartbeat_back_addr":"172.21.15.12:6809/1190079611","heartbeat_front_addr":"172.21.15.12:6807/1190079611","state":["exists","up"]},{"osd":1,"uuid":"416e87af-db5d-41ce-84d9-731e3e07e3c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6801","nonce":1983639525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6803","nonce":1983639525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6807","nonce":1983639525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6805","nonce":1983639525}]},"public_addr":"172.21.15.173:6801/1983639525","cluster_addr":"172.21.15.173:6803/1983639525","heartbeat_back_addr":"172.21.15.173:6807/1983639525","heartbeat_front_addr":"172.21.15.173:6805/1983639525","state":["exists","up"]},{"osd":2,"uuid":"7a559c70-4c54-431b-8baf-4136b6e76493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6801","nonce":2253997387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6803","nonce":2253997387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6807","nonce":2253997387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6805","nonce":2253997387}]},"public_addr":"172.21.15.203:6801/2253997387","cluster_addr":"172.21.15.203:6803/2253997387","heartbeat_back_addr":"172.21.15.203:6807/2253997387","heartbeat_front_addr":"172.21.15.203:6805/2253997387","state":["exists","up"]},{"osd":3,"uuid":"c41cd287-a674-486c-8025-328cd36e5ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6811","nonce":1761446905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6813","nonce":1761446905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6817","nonce":1761446905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6815","nonce":1761446905}]},"public_addr":"172.21.15.12:6811/1761446905","cluster_addr":"172.21.15.12:6813/1761446905","heartbeat_back_addr":"172.21.15.12:6817/1761446905","heartbeat_front_addr":"172.21.15.12:6815/1761446905","state":["exists","up"]},{"osd":4,"uuid":"3c48f886-0b42-4a09-b31e-abccc200bb1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6809","nonce":242958272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6811","nonce":242958272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6815","nonce":242958272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6813","nonce":242958272}]},"public_addr":"172.21.15.173:6809/242958272","cluster_addr":"172.21.15.173:6811/242958272","heartbeat_back_addr":"172.21.15.173:6815/242958272","heartbeat_front_addr":"172.21.15.173:6813/242958272","state":["exists","up"]},{"osd":5,"uuid":"4a34f06f-8805-40c0-b1ab-23407b0276d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6809","nonce":4270966092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6811","nonce":4270966092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6815","nonce":4270966092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6813","nonce":4270966092}]},"public_addr":"172.21.15.203:6809/4270966092","cluster_addr":"172.21.15.203:6811/4270966092","heartbeat_back_addr":"172.21.15.203:6815/4270966092","heartbeat_front_addr":"172.21.15.203:6813/4270966092","state":["exists","up"]},{"osd":6,"uuid":"26ddc663-5623-48c9-80d1-f8b968765d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6819","nonce":3606839295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6821","nonce":3606839295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6825","nonce":3606839295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6823","nonce":3606839295}]},"public_addr":"172.21.15.12:6819/3606839295","cluster_addr":"172.21.15.12:6821/3606839295","heartbeat_back_addr":"172.21.15.12:6825/3606839295","heartbeat_front_addr":"172.21.15.12:6823/3606839295","state":["exists","up"]},{"osd":7,"uuid":"17dc9417-71cd-4576-aa08-933212ec55e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6817","nonce":1241504346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6819","nonce":1241504346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6823","nonce":1241504346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6821","nonce":1241504346}]},"public_addr":"172.21.15.173:6817/1241504346","cluster_addr":"172.21.15.173:6819/1241504346","heartbeat_back_addr":"172.21.15.173:6823/1241504346","heartbeat_front_addr":"172.21.15.173:6821/1241504346","state":["exists","up"]},{"osd":8,"uuid":"90a2b63a-471d-43b8-ac18-bb1f16778d18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6817","nonce":1516996192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6819","nonce":1516996192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6823","nonce":1516996192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6821","nonce":1516996192}]},"public_addr":"172.21.15.203:6817/1516996192","cluster_addr":"172.21.15.203:6819/1516996192","heartbeat_back_addr":"172.21.15.203:6823/1516996192","heartbeat_front_addr":"172.21.15.203:6821/1516996192","state":["exists","up"]},{"osd":9,"uuid":"6e1ca504-40b4-49d0-b245-7946181abfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6827","nonce":1175260774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6829","nonce":1175260774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6833","nonce":1175260774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6831","nonce":1175260774}]},"public_addr":"172.21.15.12:6827/1175260774","cluster_addr":"172.21.15.12:6829/1175260774","heartbeat_back_addr":"172.21.15.12:6833/1175260774","heartbeat_front_addr":"172.21.15.12:6831/1175260774","state":["exists","up"]},{"osd":10,"uuid":"09a595ed-5614-4386-8e61-0170af66e01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6825","nonce":3303506808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6827","nonce":3303506808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6831","nonce":3303506808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6829","nonce":3303506808}]},"public_addr":"172.21.15.173:6825/3303506808","cluster_addr":"172.21.15.173:6827/3303506808","heartbeat_back_addr":"172.21.15.173:6831/3303506808","heartbeat_front_addr":"172.21.15.173:6829/3303506808","state":["exists","up"]},{"osd":11,"uuid":"9ef1d595-a977-4ba7-9c6b-d2b3295f20b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6825","nonce":2323604506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6827","nonce":2323604506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6831","nonce":2323604506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6829","nonce":2323604506}]},"public_addr":"172.21.15.203:6825/2323604506","cluster_addr":"172.21.15.203:6827/2323604506","heartbeat_back_addr":"172.21.15.203:6831/2323604506","heartbeat_front_addr":"172.21.15.203:6829/2323604506","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:18.469038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:42.752628+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:07.069493+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:41.962478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:08.363104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:33.341336+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:07.286541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:34.243302+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:59.749699+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:57:33.822913+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:00.737269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:26.599854+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/4182783120":"2024-03-24T22:51:23.113201+0000","172.21.15.12:6800/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1900938637":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/3414861652":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6801/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/2580541477":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6800/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/501058839":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1192554716":"2024-03-24T22:50:48.401223+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T22:59:51.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.901+0000 7ff6ce400700 1 -- 172.21.15.12:0/3434498794 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff6ac0a2920 msgr2=0x7ff6ac0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:51.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.901+0000 7ff6ce400700 1 --2- 172.21.15.12:0/3434498794 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff6ac0a2920 0x7ff6ac0a4de0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff6b40098c0 tx=0x7ff6b401a040 comp rx=0 tx=0).stop 2024-03-23T22:59:51.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.901+0000 7ff6ce400700 1 -- 172.21.15.12:0/3434498794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6c81021e0 msgr2=0x7ff6c8074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T22:59:51.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.901+0000 7ff6ce400700 1 --2- 172.21.15.12:0/3434498794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6c81021e0 0x7ff6c8074490 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff6b800e9c0 tx=0x7ff6b800ecd0 comp rx=0 tx=0).stop 2024-03-23T22:59:51.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.901+0000 7ff6ce400700 1 -- 172.21.15.12:0/3434498794 shutdown_connections 2024-03-23T22:59:51.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.901+0000 7ff6ce400700 1 --2- 172.21.15.12:0/3434498794 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff6ac0a2920 0x7ff6ac0a4de0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:51.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.901+0000 7ff6ce400700 1 --2- 172.21.15.12:0/3434498794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6c81081c0 0x7ff6c8074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:51.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.901+0000 7ff6ce400700 1 --2- 172.21.15.12:0/3434498794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6c81021e0 0x7ff6c8074490 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:51.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.901+0000 7ff6ce400700 1 --2- 172.21.15.12:0/3434498794 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6c8102ba0 0x7ff6c80749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T22:59:51.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.901+0000 7ff6ce400700 1 -- 172.21.15.12:0/3434498794 >> 172.21.15.12:0/3434498794 conn(0x7ff6c80792c0 msgr2=0x7ff6c8100ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T22:59:51.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.901+0000 7ff6ce400700 1 -- 172.21.15.12:0/3434498794 shutdown_connections 2024-03-23T22:59:51.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T22:59:51.901+0000 7ff6ce400700 1 -- 172.21.15.12:0/3434498794 wait complete. 2024-03-23T22:59:52.838 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:52 smithi012 ceph-mon[31695]: pgmap v330: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-23T22:59:52.838 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:52 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3434498794' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T22:59:52.843 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-23T22:59:52.844 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-23T22:59:52.844 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-23T22:59:52.845 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-23T22:59:52.845 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-23T22:59:52.845 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-23T22:59:52.846 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-23T22:59:52.846 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-23T22:59:52.847 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-23T22:59:52.847 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-23T22:59:52.847 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-23T22:59:52.848 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-23T22:59:52.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:52 smithi173 ceph-mon[36116]: pgmap v330: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-23T22:59:52.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:52 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3434498794' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T22:59:52.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:52 smithi203 ceph-mon[36693]: pgmap v330: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-23T22:59:52.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:52 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3434498794' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T22:59:53.856 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:59:53.880 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:59:54.000 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:59:54.005 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:59:54.008 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:59:54.009 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:59:54.011 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:59:54.609 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:54 smithi012 ceph-mon[31695]: pgmap v331: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-23T22:59:54.805 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:59:54.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:54 smithi173 ceph-mon[36116]: pgmap v331: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-23T22:59:54.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:54 smithi203 ceph-mon[36693]: pgmap v331: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-23T22:59:55.542 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:59:55.545 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T22:59:56.833 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:56 smithi012 ceph-mon[31695]: pgmap v332: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T22:59:56.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:56 smithi173 ceph-mon[36116]: pgmap v332: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T22:59:56.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:56 smithi203 ceph-mon[36693]: pgmap v332: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T22:59:58.821 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 22:59:58 smithi012 ceph-mon[31695]: pgmap v333: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-23T22:59:58.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 22:59:58 smithi173 ceph-mon[36116]: pgmap v333: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-23T22:59:58.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 22:59:58 smithi203 ceph-mon[36693]: pgmap v333: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-23T23:00:00.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.676+0000 7f4303a00700 1 -- 172.21.15.12:0/310129806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42fc104f90 msgr2=0x7f42fc108fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:00.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.676+0000 7f4303a00700 1 --2- 172.21.15.12:0/310129806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42fc104f90 0x7f42fc108fe0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f42e80099c0 tx=0x7f42e8009cd0 comp rx=0 tx=0).stop 2024-03-23T23:00:00.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.676+0000 7f4303a00700 1 -- 172.21.15.12:0/310129806 shutdown_connections 2024-03-23T23:00:00.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.676+0000 7f4303a00700 1 --2- 172.21.15.12:0/310129806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42fc1096c0 0x7f42fc111290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:00.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.676+0000 7f4303a00700 1 --2- 172.21.15.12:0/310129806 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42fc1045e0 0x7f42fc1049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:00.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.676+0000 7f4303a00700 1 --2- 172.21.15.12:0/310129806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42fc104f90 0x7f42fc108fe0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:00.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.676+0000 7f4303a00700 1 -- 172.21.15.12:0/310129806 >> 172.21.15.12:0/310129806 conn(0x7f42fc0ffe70 msgr2=0x7f42fc102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:00.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.676+0000 7f4303a00700 1 -- 172.21.15.12:0/310129806 shutdown_connections 2024-03-23T23:00:00.678 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:00 smithi012 ceph-mon[31695]: pgmap v334: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:00:00.678 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:00 smithi012 ceph-mon[31695]: overall HEALTH_OK 2024-03-23T23:00:00.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.676+0000 7f4303a00700 1 -- 172.21.15.12:0/310129806 wait complete. 2024-03-23T23:00:00.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.678+0000 7f4303a00700 1 Processor -- start 2024-03-23T23:00:00.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.678+0000 7f4303a00700 1 -- start start 2024-03-23T23:00:00.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.678+0000 7f4303a00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42fc1045e0 0x7f42fc1a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:00.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.678+0000 7f4303a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42fc104f90 0x7f42fc1a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:00.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.678+0000 7f4303a00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42fc1096c0 0x7f42fc19a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:00.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.678+0000 7f4303a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42fc1146f0 con 0x7f42fc104f90 2024-03-23T23:00:00.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.678+0000 7f4303a00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42fc114570 con 0x7f42fc1096c0 2024-03-23T23:00:00.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.678+0000 7f4303a00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42fc114870 con 0x7f42fc1045e0 2024-03-23T23:00:00.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.678+0000 7f4301000700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42fc1045e0 0x7f42fc1a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:00.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.678+0000 7f4301000700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42fc1045e0 0x7f42fc1a07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:34248/0 (socket says 172.21.15.12:34248) 2024-03-23T23:00:00.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.678+0000 7f4301000700 1 -- 172.21.15.12:0/920969690 learned_addr learned my addr 172.21.15.12:0/920969690 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:00.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.679+0000 7f4301a00700 1 --2- 172.21.15.12:0/920969690 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42fc1096c0 0x7f42fc19a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:00.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.679+0000 7f42fbe00700 1 --2- 172.21.15.12:0/920969690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42fc104f90 0x7f42fc1a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:00.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.679+0000 7f4301000700 1 -- 172.21.15.12:0/920969690 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42fc1096c0 msgr2=0x7f42fc19a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:00.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.679+0000 7f4301000700 1 --2- 172.21.15.12:0/920969690 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42fc1096c0 0x7f42fc19a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:00.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.679+0000 7f4301000700 1 -- 172.21.15.12:0/920969690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42fc104f90 msgr2=0x7f42fc1a0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:00.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.679+0000 7f4301000700 1 --2- 172.21.15.12:0/920969690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42fc104f90 0x7f42fc1a0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:00.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.679+0000 7f4301000700 1 -- 172.21.15.12:0/920969690 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42fc19b0f0 con 0x7f42fc1045e0 2024-03-23T23:00:00.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.679+0000 7f42fbe00700 1 --2- 172.21.15.12:0/920969690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42fc104f90 0x7f42fc1a0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:00:00.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.679+0000 7f4301000700 1 --2- 172.21.15.12:0/920969690 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42fc1045e0 0x7f42fc1a07d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f42f000e9c0 tx=0x7f42f000ed80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:00.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.679+0000 7f42f9600700 1 -- 172.21.15.12:0/920969690 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f000cbd0 con 0x7f42fc1045e0 2024-03-23T23:00:00.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.679+0000 7f42f9600700 1 -- 172.21.15.12:0/920969690 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42f000cd70 con 0x7f42fc1045e0 2024-03-23T23:00:00.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.679+0000 7f4303a00700 1 -- 172.21.15.12:0/920969690 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42fc19b3e0 con 0x7f42fc1045e0 2024-03-23T23:00:00.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.679+0000 7f42f9600700 1 -- 172.21.15.12:0/920969690 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f0010430 con 0x7f42fc1045e0 2024-03-23T23:00:00.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.680+0000 7f4303a00700 1 -- 172.21.15.12:0/920969690 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42fc1a76a0 con 0x7f42fc1045e0 2024-03-23T23:00:00.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.680+0000 7f4303a00700 1 -- 172.21.15.12:0/920969690 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f42fc062920 con 0x7f42fc1045e0 2024-03-23T23:00:00.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.682+0000 7f42f9600700 1 -- 172.21.15.12:0/920969690 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f42f0004750 con 0x7f42fc1045e0 2024-03-23T23:00:00.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.683+0000 7f42f9600700 1 --2- 172.21.15.12:0/920969690 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f42e40a29a0 0x7f42e40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:00.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.684+0000 7f42f9600700 1 -- 172.21.15.12:0/920969690 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f42f00ced70 con 0x7f42fc1045e0 2024-03-23T23:00:00.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.684+0000 7f42fbe00700 1 --2- 172.21.15.12:0/920969690 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f42e40a29a0 0x7f42e40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:00.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.684+0000 7f42f9600700 1 --2- 172.21.15.12:0/920969690 >> [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295] conn(0x7f42e40ac180 0x7f42e40ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:00.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.684+0000 7f42f9600700 1 -- 172.21.15.12:0/920969690 --> [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f42f0017e50 con 0x7f42e40ac180 2024-03-23T23:00:00.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.684+0000 7f42f9600700 1 -- 172.21.15.12:0/920969690 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f42f00d1390 con 0x7f42fc1045e0 2024-03-23T23:00:00.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.684+0000 7f4301a00700 1 --2- 172.21.15.12:0/920969690 >> [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295] conn(0x7f42e40ac180 0x7f42e40ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:00.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.685+0000 7f42fbe00700 1 --2- 172.21.15.12:0/920969690 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f42e40a29a0 0x7f42e40a4e60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f42e8000c00 tx=0x7f42e8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:00.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.685+0000 7f4301a00700 1 --2- 172.21.15.12:0/920969690 >> [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295] conn(0x7f42e40ac180 0x7f42e40ae5a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:00.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.686+0000 7f42f9600700 1 -- 172.21.15.12:0/920969690 <== osd.6 v2:172.21.15.12:6818/3606839295 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f42f0017e50 con 0x7f42e40ac180 2024-03-23T23:00:00.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.704+0000 7f4303a00700 1 -- 172.21.15.12:0/920969690 --> [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f42fc062da0 con 0x7f42e40ac180 2024-03-23T23:00:00.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.705+0000 7f42f9600700 1 -- 172.21.15.12:0/920969690 <== osd.6 v2:172.21.15.12:6818/3606839295 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f42fc062da0 con 0x7f42e40ac180 2024-03-23T23:00:00.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.705+0000 7f4303a00700 1 -- 172.21.15.12:0/920969690 >> [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295] conn(0x7f42e40ac180 msgr2=0x7f42e40ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:00.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.705+0000 7f4303a00700 1 --2- 172.21.15.12:0/920969690 >> [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295] conn(0x7f42e40ac180 0x7f42e40ae5a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:00.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.706+0000 7f4303a00700 1 -- 172.21.15.12:0/920969690 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f42e40a29a0 msgr2=0x7f42e40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:00.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.706+0000 7f4303a00700 1 --2- 172.21.15.12:0/920969690 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f42e40a29a0 0x7f42e40a4e60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f42e8000c00 tx=0x7f42e8005fd0 comp rx=0 tx=0).stop 2024-03-23T23:00:00.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.706+0000 7f4303a00700 1 -- 172.21.15.12:0/920969690 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42fc1045e0 msgr2=0x7f42fc1a07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:00.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.706+0000 7f4303a00700 1 --2- 172.21.15.12:0/920969690 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42fc1045e0 0x7f42fc1a07d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f42f000e9c0 tx=0x7f42f000ed80 comp rx=0 tx=0).stop 2024-03-23T23:00:00.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.706+0000 7f4301a00700 1 -- 172.21.15.12:0/920969690 reap_dead start 2024-03-23T23:00:00.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.706+0000 7f4303a00700 1 -- 172.21.15.12:0/920969690 shutdown_connections 2024-03-23T23:00:00.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.706+0000 7f4303a00700 1 -- 172.21.15.12:0/920969690 >> 172.21.15.12:0/920969690 conn(0x7f42fc0ffe70 msgr2=0x7f42fc102140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:00.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.706+0000 7f4303a00700 1 -- 172.21.15.12:0/920969690 shutdown_connections 2024-03-23T23:00:00.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:00.706+0000 7f4303a00700 1 -- 172.21.15.12:0/920969690 wait complete. 2024-03-23T23:00:00.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:00 smithi173 ceph-mon[36116]: pgmap v334: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:00:00.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:00 smithi173 ceph-mon[36116]: overall HEALTH_OK 2024-03-23T23:00:00.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:00 smithi203 ceph-mon[36693]: pgmap v334: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:00:00.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:00 smithi203 ceph-mon[36693]: overall HEALTH_OK 2024-03-23T23:00:02.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.170+0000 7f85fd200700 1 -- 172.21.15.12:0/3865469035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85f8069e40 msgr2=0x7f85f806a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:02.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.170+0000 7f85fd200700 1 --2- 172.21.15.12:0/3865469035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85f8069e40 0x7f85f806a2a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f85e4009990 tx=0x7f85e4009ca0 comp rx=0 tx=0).stop 2024-03-23T23:00:02.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.170+0000 7f85fd200700 1 -- 172.21.15.12:0/3865469035 shutdown_connections 2024-03-23T23:00:02.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.170+0000 7f85fd200700 1 --2- 172.21.15.12:0/3865469035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85f806a7e0 0x7f85f810f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:02.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.170+0000 7f85fd200700 1 --2- 172.21.15.12:0/3865469035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85f81081e0 0x7f85f81085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:02.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.170+0000 7f85fd200700 1 --2- 172.21.15.12:0/3865469035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85f8069e40 0x7f85f806a2a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:02.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.170+0000 7f85fd200700 1 -- 172.21.15.12:0/3865469035 >> 172.21.15.12:0/3865469035 conn(0x7f85f8077fc0 msgr2=0x7f85f80783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:02.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.171+0000 7f85fd200700 1 -- 172.21.15.12:0/3865469035 shutdown_connections 2024-03-23T23:00:02.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.171+0000 7f85fd200700 1 -- 172.21.15.12:0/3865469035 wait complete. 2024-03-23T23:00:02.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.172+0000 7f85fd200700 1 Processor -- start 2024-03-23T23:00:02.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.172+0000 7f85fd200700 1 -- start start 2024-03-23T23:00:02.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.173+0000 7f85fd200700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85f8069e40 0x7f85f819e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:02.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.173+0000 7f85fd200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85f806a7e0 0x7f85f819eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:02.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.173+0000 7f85fd200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85f81081e0 0x7f85f81986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:02.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.173+0000 7f85fd200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f8112490 con 0x7f85f806a7e0 2024-03-23T23:00:02.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.173+0000 7f85fd200700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f8112310 con 0x7f85f8069e40 2024-03-23T23:00:02.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.173+0000 7f85fd200700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f8112610 con 0x7f85f81081e0 2024-03-23T23:00:02.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.173+0000 7f85f5e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85f8069e40 0x7f85f819e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:02.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.173+0000 7f85f5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85f806a7e0 0x7f85f819eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:02.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.173+0000 7f85f5e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85f8069e40 0x7f85f819e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48900/0 (socket says 172.21.15.12:48900) 2024-03-23T23:00:02.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.173+0000 7f85f5e00700 1 -- 172.21.15.12:0/1855851317 learned_addr learned my addr 172.21.15.12:0/1855851317 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:02.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.173+0000 7f85f6800700 1 --2- 172.21.15.12:0/1855851317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85f81081e0 0x7f85f81986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:02.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.174+0000 7f85f5e00700 1 -- 172.21.15.12:0/1855851317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85f81081e0 msgr2=0x7f85f81986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:02.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.174+0000 7f85f5e00700 1 --2- 172.21.15.12:0/1855851317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85f81081e0 0x7f85f81986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:02.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.174+0000 7f85f5e00700 1 -- 172.21.15.12:0/1855851317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85f806a7e0 msgr2=0x7f85f819eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:02.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.174+0000 7f85f5e00700 1 --2- 172.21.15.12:0/1855851317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85f806a7e0 0x7f85f819eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:02.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.174+0000 7f85f5e00700 1 -- 172.21.15.12:0/1855851317 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f8198fd0 con 0x7f85f8069e40 2024-03-23T23:00:02.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.174+0000 7f85f5e00700 1 --2- 172.21.15.12:0/1855851317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85f8069e40 0x7f85f819e620 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f85e800e9a0 tx=0x7f85e800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:02.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.174+0000 7f85eea00700 1 -- 172.21.15.12:0/1855851317 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e800ca10 con 0x7f85f8069e40 2024-03-23T23:00:02.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.175+0000 7f85eea00700 1 -- 172.21.15.12:0/1855851317 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f85e800cbb0 con 0x7f85f8069e40 2024-03-23T23:00:02.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.175+0000 7f85eea00700 1 -- 172.21.15.12:0/1855851317 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e8018680 con 0x7f85f8069e40 2024-03-23T23:00:02.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:02 smithi012 ceph-mon[31695]: pgmap v335: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:00:02.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.175+0000 7f85fd200700 1 -- 172.21.15.12:0/1855851317 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f8199260 con 0x7f85f8069e40 2024-03-23T23:00:02.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.175+0000 7f85fd200700 1 -- 172.21.15.12:0/1855851317 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f8101090 con 0x7f85f8069e40 2024-03-23T23:00:02.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.176+0000 7f85fd200700 1 -- 172.21.15.12:0/1855851317 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f85f8062920 con 0x7f85f8069e40 2024-03-23T23:00:02.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.180+0000 7f85eea00700 1 -- 172.21.15.12:0/1855851317 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f85e8010430 con 0x7f85f8069e40 2024-03-23T23:00:02.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.181+0000 7f85eea00700 1 --2- 172.21.15.12:0/1855851317 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f85dc0a6db0 0x7f85dc0a9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:02.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.181+0000 7f85eea00700 1 -- 172.21.15.12:0/1855851317 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f85e80cebe0 con 0x7f85f8069e40 2024-03-23T23:00:02.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.181+0000 7f85f5400700 1 --2- 172.21.15.12:0/1855851317 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f85dc0a6db0 0x7f85dc0a9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:02.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.181+0000 7f85eea00700 1 --2- 172.21.15.12:0/1855851317 >> [v2:172.21.15.173:6800/1983639525,v1:172.21.15.173:6801/1983639525] conn(0x7f85dc0b0590 0x7f85dc0b29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:02.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.181+0000 7f85eea00700 1 -- 172.21.15.12:0/1855851317 --> [v2:172.21.15.173:6800/1983639525,v1:172.21.15.173:6801/1983639525] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f85e8012ec0 con 0x7f85dc0b0590 2024-03-23T23:00:02.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.181+0000 7f85eea00700 1 -- 172.21.15.12:0/1855851317 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f85e80d12b0 con 0x7f85f8069e40 2024-03-23T23:00:02.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.182+0000 7f85f6800700 1 --2- 172.21.15.12:0/1855851317 >> [v2:172.21.15.173:6800/1983639525,v1:172.21.15.173:6801/1983639525] conn(0x7f85dc0b0590 0x7f85dc0b29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:02.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.182+0000 7f85f5400700 1 --2- 172.21.15.12:0/1855851317 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f85dc0a6db0 0x7f85dc0a9270 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f85f8199eb0 tx=0x7f85e4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:02.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.182+0000 7f85f6800700 1 --2- 172.21.15.12:0/1855851317 >> [v2:172.21.15.173:6800/1983639525,v1:172.21.15.173:6801/1983639525] conn(0x7f85dc0b0590 0x7f85dc0b29b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:02.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.183+0000 7f85eea00700 1 -- 172.21.15.12:0/1855851317 <== osd.1 v2:172.21.15.173:6800/1983639525 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f85e8012ec0 con 0x7f85dc0b0590 2024-03-23T23:00:02.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.204+0000 7f85fd200700 1 -- 172.21.15.12:0/1855851317 --> [v2:172.21.15.173:6800/1983639525,v1:172.21.15.173:6801/1983639525] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f85f8062da0 con 0x7f85dc0b0590 2024-03-23T23:00:02.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.204+0000 7f85eea00700 1 -- 172.21.15.12:0/1855851317 <== osd.1 v2:172.21.15.173:6800/1983639525 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f85f8062da0 con 0x7f85dc0b0590 2024-03-23T23:00:02.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.205+0000 7f85fd200700 1 -- 172.21.15.12:0/1855851317 >> [v2:172.21.15.173:6800/1983639525,v1:172.21.15.173:6801/1983639525] conn(0x7f85dc0b0590 msgr2=0x7f85dc0b29b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:02.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.205+0000 7f85fd200700 1 --2- 172.21.15.12:0/1855851317 >> [v2:172.21.15.173:6800/1983639525,v1:172.21.15.173:6801/1983639525] conn(0x7f85dc0b0590 0x7f85dc0b29b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:02.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.205+0000 7f85fd200700 1 -- 172.21.15.12:0/1855851317 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f85dc0a6db0 msgr2=0x7f85dc0a9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:02.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.205+0000 7f85fd200700 1 --2- 172.21.15.12:0/1855851317 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f85dc0a6db0 0x7f85dc0a9270 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f85f8199eb0 tx=0x7f85e4005fd0 comp rx=0 tx=0).stop 2024-03-23T23:00:02.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.205+0000 7f85fd200700 1 -- 172.21.15.12:0/1855851317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85f8069e40 msgr2=0x7f85f819e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:02.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.205+0000 7f85fd200700 1 --2- 172.21.15.12:0/1855851317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85f8069e40 0x7f85f819e620 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f85e800e9a0 tx=0x7f85e800ecb0 comp rx=0 tx=0).stop 2024-03-23T23:00:02.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.205+0000 7f85f6800700 1 -- 172.21.15.12:0/1855851317 reap_dead start 2024-03-23T23:00:02.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.206+0000 7f85fd200700 1 -- 172.21.15.12:0/1855851317 shutdown_connections 2024-03-23T23:00:02.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.206+0000 7f85fd200700 1 -- 172.21.15.12:0/1855851317 >> 172.21.15.12:0/1855851317 conn(0x7f85f8077fc0 msgr2=0x7f85f80ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:02.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.206+0000 7f85fd200700 1 -- 172.21.15.12:0/1855851317 shutdown_connections 2024-03-23T23:00:02.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:02.206+0000 7f85fd200700 1 -- 172.21.15.12:0/1855851317 wait complete. 2024-03-23T23:00:02.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:02 smithi173 ceph-mon[36116]: pgmap v335: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:00:02.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:02 smithi203 ceph-mon[36693]: pgmap v335: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:00:04.173 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:00:04.173 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:00:04.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:04 smithi012 ceph-mon[31695]: pgmap v336: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:00:04.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:04 smithi173 ceph-mon[36116]: pgmap v336: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:00:04.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:04 smithi203 ceph-mon[36693]: pgmap v336: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:00:06.123 INFO:teuthology.orchestra.run.smithi012.stdout:201863462961 2024-03-23T23:00:06.123 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-23T23:00:06.662 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:06 smithi012 ceph-mon[31695]: pgmap v337: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-23T23:00:06.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.958+0000 7fc748200700 1 -- 172.21.15.12:0/4093846724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc740069e40 msgr2=0x7fc74006a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:06.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.958+0000 7fc748200700 1 --2- 172.21.15.12:0/4093846724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc740069e40 0x7fc74006a2a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc72c0099c0 tx=0x7fc72c009cd0 comp rx=0 tx=0).stop 2024-03-23T23:00:06.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.958+0000 7fc748200700 1 -- 172.21.15.12:0/4093846724 shutdown_connections 2024-03-23T23:00:06.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.958+0000 7fc748200700 1 --2- 172.21.15.12:0/4093846724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc74006a7e0 0x7fc74010f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:06.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.958+0000 7fc748200700 1 --2- 172.21.15.12:0/4093846724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7401081e0 0x7fc7401085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:06.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.958+0000 7fc748200700 1 --2- 172.21.15.12:0/4093846724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc740069e40 0x7fc74006a2a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:06.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.958+0000 7fc748200700 1 -- 172.21.15.12:0/4093846724 >> 172.21.15.12:0/4093846724 conn(0x7fc740077fc0 msgr2=0x7fc7400783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:06.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.958+0000 7fc748200700 1 -- 172.21.15.12:0/4093846724 shutdown_connections 2024-03-23T23:00:06.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.958+0000 7fc748200700 1 -- 172.21.15.12:0/4093846724 wait complete. 2024-03-23T23:00:06.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.959+0000 7fc748200700 1 Processor -- start 2024-03-23T23:00:06.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.960+0000 7fc748200700 1 -- start start 2024-03-23T23:00:06.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.960+0000 7fc748200700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc740069e40 0x7fc7401a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:06.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.960+0000 7fc748200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc74006a7e0 0x7fc7401a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:06.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.960+0000 7fc748200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7401081e0 0x7fc74019aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:06.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.960+0000 7fc748200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7401124b0 con 0x7fc74006a7e0 2024-03-23T23:00:06.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.960+0000 7fc748200700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc740112330 con 0x7fc740069e40 2024-03-23T23:00:06.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.960+0000 7fc748200700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc740112630 con 0x7fc7401081e0 2024-03-23T23:00:06.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.960+0000 7fc745800700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc740069e40 0x7fc7401a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:06.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.960+0000 7fc746200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7401081e0 0x7fc74019aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:06.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.960+0000 7fc745800700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc740069e40 0x7fc7401a08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:49686/0 (socket says 172.21.15.12:49686) 2024-03-23T23:00:06.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.960+0000 7fc745800700 1 -- 172.21.15.12:0/1370707249 learned_addr learned my addr 172.21.15.12:0/1370707249 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:06.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.960+0000 7fc744e00700 1 --2- 172.21.15.12:0/1370707249 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc74006a7e0 0x7fc7401a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:06.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.961+0000 7fc746200700 1 -- 172.21.15.12:0/1370707249 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc740069e40 msgr2=0x7fc7401a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:06.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.961+0000 7fc746200700 1 --2- 172.21.15.12:0/1370707249 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc740069e40 0x7fc7401a08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:06.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.961+0000 7fc746200700 1 -- 172.21.15.12:0/1370707249 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc74006a7e0 msgr2=0x7fc7401a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:06.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.961+0000 7fc746200700 1 --2- 172.21.15.12:0/1370707249 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc74006a7e0 0x7fc7401a0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:06.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.961+0000 7fc746200700 1 -- 172.21.15.12:0/1370707249 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc74019b1f0 con 0x7fc7401081e0 2024-03-23T23:00:06.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.961+0000 7fc746200700 1 --2- 172.21.15.12:0/1370707249 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7401081e0 0x7fc74019aa90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc73800efc0 tx=0x7fc73800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:06.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.961+0000 7fc73e000700 1 -- 172.21.15.12:0/1370707249 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7380092e0 con 0x7fc7401081e0 2024-03-23T23:00:06.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.961+0000 7fc73e000700 1 -- 172.21.15.12:0/1370707249 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc738004bb0 con 0x7fc7401081e0 2024-03-23T23:00:06.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.961+0000 7fc748200700 1 -- 172.21.15.12:0/1370707249 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc74019b480 con 0x7fc7401081e0 2024-03-23T23:00:06.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.961+0000 7fc73e000700 1 -- 172.21.15.12:0/1370707249 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc738004280 con 0x7fc7401081e0 2024-03-23T23:00:06.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.962+0000 7fc748200700 1 -- 172.21.15.12:0/1370707249 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7401a77a0 con 0x7fc7401081e0 2024-03-23T23:00:06.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.962+0000 7fc748200700 1 -- 172.21.15.12:0/1370707249 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc74010dd50 con 0x7fc7401081e0 2024-03-23T23:00:06.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.965+0000 7fc73e000700 1 -- 172.21.15.12:0/1370707249 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fc73801b030 con 0x7fc7401081e0 2024-03-23T23:00:06.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.966+0000 7fc73e000700 1 --2- 172.21.15.12:0/1370707249 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc7280a29a0 0x7fc7280a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:06.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.966+0000 7fc73e000700 1 -- 172.21.15.12:0/1370707249 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fc7380ce6a0 con 0x7fc7401081e0 2024-03-23T23:00:06.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.966+0000 7fc745800700 1 --2- 172.21.15.12:0/1370707249 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc7280a29a0 0x7fc7280a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:06.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.966+0000 7fc73e000700 1 --2- 172.21.15.12:0/1370707249 >> [v2:172.21.15.173:6808/242958272,v1:172.21.15.173:6809/242958272] conn(0x7fc7280ac180 0x7fc7280ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:06.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.966+0000 7fc73e000700 1 -- 172.21.15.12:0/1370707249 --> [v2:172.21.15.173:6808/242958272,v1:172.21.15.173:6809/242958272] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc7380048a0 con 0x7fc7280ac180 2024-03-23T23:00:06.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.966+0000 7fc73e000700 1 -- 172.21.15.12:0/1370707249 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc7380cea50 con 0x7fc7401081e0 2024-03-23T23:00:06.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.967+0000 7fc744e00700 1 --2- 172.21.15.12:0/1370707249 >> [v2:172.21.15.173:6808/242958272,v1:172.21.15.173:6809/242958272] conn(0x7fc7280ac180 0x7fc7280ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:06.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.967+0000 7fc745800700 1 --2- 172.21.15.12:0/1370707249 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc7280a29a0 0x7fc7280a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc74019bbd0 tx=0x7fc73400b500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:06.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.967+0000 7fc744e00700 1 --2- 172.21.15.12:0/1370707249 >> [v2:172.21.15.173:6808/242958272,v1:172.21.15.173:6809/242958272] conn(0x7fc7280ac180 0x7fc7280ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:06.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.969+0000 7fc73e000700 1 -- 172.21.15.12:0/1370707249 <== osd.4 v2:172.21.15.173:6808/242958272 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc7380048a0 con 0x7fc7280ac180 2024-03-23T23:00:06.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.987+0000 7fc748200700 1 -- 172.21.15.12:0/1370707249 --> [v2:172.21.15.173:6808/242958272,v1:172.21.15.173:6809/242958272] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc740103890 con 0x7fc7280ac180 2024-03-23T23:00:06.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.987+0000 7fc73e000700 1 -- 172.21.15.12:0/1370707249 <== osd.4 v2:172.21.15.173:6808/242958272 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc740103890 con 0x7fc7280ac180 2024-03-23T23:00:06.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.988+0000 7fc748200700 1 -- 172.21.15.12:0/1370707249 >> [v2:172.21.15.173:6808/242958272,v1:172.21.15.173:6809/242958272] conn(0x7fc7280ac180 msgr2=0x7fc7280ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:06.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.988+0000 7fc748200700 1 --2- 172.21.15.12:0/1370707249 >> [v2:172.21.15.173:6808/242958272,v1:172.21.15.173:6809/242958272] conn(0x7fc7280ac180 0x7fc7280ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:06.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.988+0000 7fc748200700 1 -- 172.21.15.12:0/1370707249 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc7280a29a0 msgr2=0x7fc7280a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:06.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.988+0000 7fc748200700 1 --2- 172.21.15.12:0/1370707249 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc7280a29a0 0x7fc7280a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc74019bbd0 tx=0x7fc73400b500 comp rx=0 tx=0).stop 2024-03-23T23:00:06.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.988+0000 7fc748200700 1 -- 172.21.15.12:0/1370707249 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7401081e0 msgr2=0x7fc74019aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:06.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.988+0000 7fc748200700 1 --2- 172.21.15.12:0/1370707249 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7401081e0 0x7fc74019aa90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc73800efc0 tx=0x7fc73800c3a0 comp rx=0 tx=0).stop 2024-03-23T23:00:06.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.988+0000 7fc746200700 1 -- 172.21.15.12:0/1370707249 reap_dead start 2024-03-23T23:00:06.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.988+0000 7fc748200700 1 -- 172.21.15.12:0/1370707249 shutdown_connections 2024-03-23T23:00:06.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.988+0000 7fc748200700 1 -- 172.21.15.12:0/1370707249 >> 172.21.15.12:0/1370707249 conn(0x7fc740077fc0 msgr2=0x7fc74010d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:06.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.988+0000 7fc748200700 1 -- 172.21.15.12:0/1370707249 shutdown_connections 2024-03-23T23:00:06.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:05.988+0000 7fc748200700 1 -- 172.21.15.12:0/1370707249 wait complete. 2024-03-23T23:00:06.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:06 smithi173 ceph-mon[36116]: pgmap v337: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-23T23:00:06.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:06 smithi203 ceph-mon[36693]: pgmap v337: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-23T23:00:07.549 INFO:teuthology.orchestra.run.smithi012.stdout:55834574926 2024-03-23T23:00:07.550 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-23T23:00:07.802 INFO:teuthology.orchestra.run.smithi012.stdout:146028888125 2024-03-23T23:00:07.803 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-23T23:00:08.747 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:08 smithi012 ceph-mon[31695]: pgmap v338: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-23T23:00:08.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:08 smithi173 ceph-mon[36116]: pgmap v338: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-23T23:00:08.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:08 smithi203 ceph-mon[36693]: pgmap v338: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-23T23:00:09.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.747+0000 7f8ec2e00700 1 -- 172.21.15.12:0/552758813 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ebc104fb0 msgr2=0x7f8ebc109000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:09.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.747+0000 7f8ec2e00700 1 --2- 172.21.15.12:0/552758813 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ebc104fb0 0x7f8ebc109000 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8ea80099c0 tx=0x7f8ea8009cd0 comp rx=0 tx=0).stop 2024-03-23T23:00:09.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.747+0000 7f8ec2e00700 1 -- 172.21.15.12:0/552758813 shutdown_connections 2024-03-23T23:00:09.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.747+0000 7f8ec2e00700 1 --2- 172.21.15.12:0/552758813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ebc104600 0x7f8ebc1049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.747+0000 7f8ec2e00700 1 --2- 172.21.15.12:0/552758813 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ebc1096e0 0x7f8ebc1112b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.747+0000 7f8ec2e00700 1 --2- 172.21.15.12:0/552758813 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ebc104fb0 0x7f8ebc109000 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.747+0000 7f8ec2e00700 1 -- 172.21.15.12:0/552758813 >> 172.21.15.12:0/552758813 conn(0x7f8ebc0ffe90 msgr2=0x7f8ebc1022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.747+0000 7f8ec2e00700 1 -- 172.21.15.12:0/552758813 shutdown_connections 2024-03-23T23:00:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.747+0000 7f8ec2e00700 1 -- 172.21.15.12:0/552758813 wait complete. 2024-03-23T23:00:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.749+0000 7f8ec2e00700 1 Processor -- start 2024-03-23T23:00:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.749+0000 7f8ec2e00700 1 -- start start 2024-03-23T23:00:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.750+0000 7f8ec2e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ebc104600 0x7f8ebc1a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.750+0000 7f8ec2e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ebc104fb0 0x7f8ebc1a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.750+0000 7f8ec2e00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ebc1096e0 0x7f8ebc19aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.750+0000 7f8ec2e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc114710 con 0x7f8ebc104600 2024-03-23T23:00:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.750+0000 7f8ec2e00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc114590 con 0x7f8ebc104fb0 2024-03-23T23:00:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.750+0000 7f8ec2e00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc114890 con 0x7f8ebc1096e0 2024-03-23T23:00:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.750+0000 7f8ebb400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ebc104fb0 0x7f8ebc1a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.750+0000 7f8ebbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ebc104600 0x7f8ebc1a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.750+0000 7f8ebb400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ebc104fb0 0x7f8ebc1a0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:49708/0 (socket says 172.21.15.12:49708) 2024-03-23T23:00:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.750+0000 7f8ebb400700 1 -- 172.21.15.12:0/2066598075 learned_addr learned my addr 172.21.15.12:0/2066598075 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.750+0000 7f8ebbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ebc104600 0x7f8ebc1a0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35534/0 (socket says 172.21.15.12:35534) 2024-03-23T23:00:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.750+0000 7f8ec0e00700 1 --2- 172.21.15.12:0/2066598075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ebc1096e0 0x7f8ebc19aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.750+0000 7f8ebb400700 1 -- 172.21.15.12:0/2066598075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ebc1096e0 msgr2=0x7f8ebc19aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.750+0000 7f8ebb400700 1 --2- 172.21.15.12:0/2066598075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ebc1096e0 0x7f8ebc19aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.750+0000 7f8ebb400700 1 -- 172.21.15.12:0/2066598075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ebc104600 msgr2=0x7f8ebc1a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.750+0000 7f8ebb400700 1 --2- 172.21.15.12:0/2066598075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ebc104600 0x7f8ebc1a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.750+0000 7f8ebb400700 1 -- 172.21.15.12:0/2066598075 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ebc19b170 con 0x7f8ebc104fb0 2024-03-23T23:00:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.750+0000 7f8ec0e00700 1 --2- 172.21.15.12:0/2066598075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ebc1096e0 0x7f8ebc19aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:00:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.751+0000 7f8ebb400700 1 --2- 172.21.15.12:0/2066598075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ebc104fb0 0x7f8ebc1a0d40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8005f50 tx=0x7f8ea8003a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.751+0000 7f8eb8c00700 1 -- 172.21.15.12:0/2066598075 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea8003680 con 0x7f8ebc104fb0 2024-03-23T23:00:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.751+0000 7f8eb8c00700 1 -- 172.21.15.12:0/2066598075 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ea80055a0 con 0x7f8ebc104fb0 2024-03-23T23:00:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.751+0000 7f8eb8c00700 1 -- 172.21.15.12:0/2066598075 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea800fd70 con 0x7f8ebc104fb0 2024-03-23T23:00:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.751+0000 7f8ec2e00700 1 -- 172.21.15.12:0/2066598075 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ebc19b400 con 0x7f8ebc104fb0 2024-03-23T23:00:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.751+0000 7f8ec2e00700 1 -- 172.21.15.12:0/2066598075 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ebc1a76d0 con 0x7f8ebc104fb0 2024-03-23T23:00:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.752+0000 7f8ec2e00700 1 -- 172.21.15.12:0/2066598075 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8ebc062920 con 0x7f8ebc104fb0 2024-03-23T23:00:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.754+0000 7f8eb8c00700 1 -- 172.21.15.12:0/2066598075 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f8ea800f460 con 0x7f8ebc104fb0 2024-03-23T23:00:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.756+0000 7f8eb8c00700 1 --2- 172.21.15.12:0/2066598075 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8ea00a2b40 0x7f8ea00a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.757+0000 7f8ebbe00700 1 --2- 172.21.15.12:0/2066598075 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8ea00a2b40 0x7f8ea00a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.757+0000 7f8eb8c00700 1 -- 172.21.15.12:0/2066598075 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f8ea80cf8f0 con 0x7f8ebc104fb0 2024-03-23T23:00:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.758+0000 7f8eb8c00700 1 --2- 172.21.15.12:0/2066598075 >> [v2:172.21.15.173:6824/3303506808,v1:172.21.15.173:6825/3303506808] conn(0x7f8ea00ac320 0x7f8ea00ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.758+0000 7f8eb8c00700 1 -- 172.21.15.12:0/2066598075 --> [v2:172.21.15.173:6824/3303506808,v1:172.21.15.173:6825/3303506808] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8ea800b790 con 0x7f8ea00ac320 2024-03-23T23:00:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.758+0000 7f8eb8c00700 1 -- 172.21.15.12:0/2066598075 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ea80d42a0 con 0x7f8ebc104fb0 2024-03-23T23:00:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.758+0000 7f8ec0e00700 1 --2- 172.21.15.12:0/2066598075 >> [v2:172.21.15.173:6824/3303506808,v1:172.21.15.173:6825/3303506808] conn(0x7f8ea00ac320 0x7f8ea00ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.758+0000 7f8ebbe00700 1 --2- 172.21.15.12:0/2066598075 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8ea00a2b40 0x7f8ea00a5000 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc1002e0 tx=0x7f8eac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.759+0000 7f8ec0e00700 1 --2- 172.21.15.12:0/2066598075 >> [v2:172.21.15.173:6824/3303506808,v1:172.21.15.173:6825/3303506808] conn(0x7f8ea00ac320 0x7f8ea00ae740 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.760+0000 7f8eb8c00700 1 -- 172.21.15.12:0/2066598075 <== osd.10 v2:172.21.15.173:6824/3303506808 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8ea800b790 con 0x7f8ea00ac320 2024-03-23T23:00:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.779+0000 7f8ec2e00700 1 -- 172.21.15.12:0/2066598075 --> [v2:172.21.15.173:6824/3303506808,v1:172.21.15.173:6825/3303506808] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8ebc062df0 con 0x7f8ea00ac320 2024-03-23T23:00:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.779+0000 7f8eb8c00700 1 -- 172.21.15.12:0/2066598075 <== osd.10 v2:172.21.15.173:6824/3303506808 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8ebc062df0 con 0x7f8ea00ac320 2024-03-23T23:00:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.780+0000 7f8ec2e00700 1 -- 172.21.15.12:0/2066598075 >> [v2:172.21.15.173:6824/3303506808,v1:172.21.15.173:6825/3303506808] conn(0x7f8ea00ac320 msgr2=0x7f8ea00ae740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.780+0000 7f8ec2e00700 1 --2- 172.21.15.12:0/2066598075 >> [v2:172.21.15.173:6824/3303506808,v1:172.21.15.173:6825/3303506808] conn(0x7f8ea00ac320 0x7f8ea00ae740 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.780+0000 7f8ec2e00700 1 -- 172.21.15.12:0/2066598075 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8ea00a2b40 msgr2=0x7f8ea00a5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.780+0000 7f8ec2e00700 1 --2- 172.21.15.12:0/2066598075 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8ea00a2b40 0x7f8ea00a5000 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc1002e0 tx=0x7f8eac009260 comp rx=0 tx=0).stop 2024-03-23T23:00:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.780+0000 7f8ec2e00700 1 -- 172.21.15.12:0/2066598075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ebc104fb0 msgr2=0x7f8ebc1a0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.780+0000 7f8ec2e00700 1 --2- 172.21.15.12:0/2066598075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ebc104fb0 0x7f8ebc1a0d40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8005f50 tx=0x7f8ea8003a90 comp rx=0 tx=0).stop 2024-03-23T23:00:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.780+0000 7f8ec0e00700 1 -- 172.21.15.12:0/2066598075 reap_dead start 2024-03-23T23:00:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.780+0000 7f8ec2e00700 1 -- 172.21.15.12:0/2066598075 shutdown_connections 2024-03-23T23:00:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.780+0000 7f8ec2e00700 1 -- 172.21.15.12:0/2066598075 >> 172.21.15.12:0/2066598075 conn(0x7f8ebc0ffe90 msgr2=0x7f8ebc100a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.780+0000 7f8ec2e00700 1 -- 172.21.15.12:0/2066598075 shutdown_connections 2024-03-23T23:00:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:08.780+0000 7f8ec2e00700 1 -- 172.21.15.12:0/2066598075 wait complete. 2024-03-23T23:00:10.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:10 smithi173 ceph-mon[36116]: pgmap v339: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-23T23:00:10.878 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:10 smithi012 ceph-mon[31695]: pgmap v339: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-23T23:00:10.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:10 smithi203 ceph-mon[36693]: pgmap v339: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-23T23:00:11.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.571+0000 7f0459000700 1 -- 172.21.15.12:0/3394638521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04540743c0 msgr2=0x7f04540747a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:11.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.571+0000 7f0459000700 1 --2- 172.21.15.12:0/3394638521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04540743c0 0x7f04540747a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0444009990 tx=0x7f0444009ca0 comp rx=0 tx=0).stop 2024-03-23T23:00:11.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.571+0000 7f0459000700 1 -- 172.21.15.12:0/3394638521 shutdown_connections 2024-03-23T23:00:11.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.571+0000 7f0459000700 1 --2- 172.21.15.12:0/3394638521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f045410eb90 0x7f0454110f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:11.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.571+0000 7f0459000700 1 --2- 172.21.15.12:0/3394638521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04540743c0 0x7f04540747a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:11.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.571+0000 7f0459000700 1 --2- 172.21.15.12:0/3394638521 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0454074ce0 0x7f045410e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:11.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.571+0000 7f0459000700 1 -- 172.21.15.12:0/3394638521 >> 172.21.15.12:0/3394638521 conn(0x7f04540792f0 msgr2=0x7f0454079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:11.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.572+0000 7f0459000700 1 -- 172.21.15.12:0/3394638521 shutdown_connections 2024-03-23T23:00:11.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.572+0000 7f0459000700 1 -- 172.21.15.12:0/3394638521 wait complete. 2024-03-23T23:00:11.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.573+0000 7f0459000700 1 Processor -- start 2024-03-23T23:00:11.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.573+0000 7f0459000700 1 -- start start 2024-03-23T23:00:11.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.573+0000 7f0459000700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04540743c0 0x7f04541a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:11.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.573+0000 7f0459000700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0454074ce0 0x7f04541a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:11.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.573+0000 7f0459000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f045410eb90 0x7f045419aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:11.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.573+0000 7f0459000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0454114780 con 0x7f045410eb90 2024-03-23T23:00:11.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.573+0000 7f0459000700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0454114600 con 0x7f0454074ce0 2024-03-23T23:00:11.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.573+0000 7f0459000700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0454114900 con 0x7f04540743c0 2024-03-23T23:00:11.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.574+0000 7f0451e00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04540743c0 0x7f04541a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:11.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.574+0000 7f0451e00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04540743c0 0x7f04541a0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47752/0 (socket says 172.21.15.12:47752) 2024-03-23T23:00:11.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.574+0000 7f0451e00700 1 -- 172.21.15.12:0/3401536985 learned_addr learned my addr 172.21.15.12:0/3401536985 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:11.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.574+0000 7f0451400700 1 --2- 172.21.15.12:0/3401536985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0454074ce0 0x7f04541a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:11.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.574+0000 7f0452800700 1 --2- 172.21.15.12:0/3401536985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f045410eb90 0x7f045419aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:11.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.574+0000 7f0451e00700 1 -- 172.21.15.12:0/3401536985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0454074ce0 msgr2=0x7f04541a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:11.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.574+0000 7f0451e00700 1 --2- 172.21.15.12:0/3401536985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0454074ce0 0x7f04541a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:11.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.574+0000 7f0451e00700 1 -- 172.21.15.12:0/3401536985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f045410eb90 msgr2=0x7f045419aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:11.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.574+0000 7f0451e00700 1 --2- 172.21.15.12:0/3401536985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f045410eb90 0x7f045419aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:11.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.574+0000 7f0451e00700 1 -- 172.21.15.12:0/3401536985 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f045419b180 con 0x7f04540743c0 2024-03-23T23:00:11.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.574+0000 7f0451400700 1 --2- 172.21.15.12:0/3401536985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0454074ce0 0x7f04541a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:00:11.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.574+0000 7f0452800700 1 --2- 172.21.15.12:0/3401536985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f045410eb90 0x7f045419aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:00:11.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.574+0000 7f0451e00700 1 --2- 172.21.15.12:0/3401536985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04540743c0 0x7f04541a0860 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0444000c00 tx=0x7f044400bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:11.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.575+0000 7f044aa00700 1 -- 172.21.15.12:0/3401536985 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0444027070 con 0x7f04540743c0 2024-03-23T23:00:11.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.575+0000 7f044aa00700 1 -- 172.21.15.12:0/3401536985 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0444003d70 con 0x7f04540743c0 2024-03-23T23:00:11.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.575+0000 7f0459000700 1 -- 172.21.15.12:0/3401536985 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f045419b410 con 0x7f04540743c0 2024-03-23T23:00:11.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.575+0000 7f044aa00700 1 -- 172.21.15.12:0/3401536985 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044400f460 con 0x7f04540743c0 2024-03-23T23:00:11.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.575+0000 7f0459000700 1 -- 172.21.15.12:0/3401536985 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04541a7730 con 0x7f04540743c0 2024-03-23T23:00:11.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.576+0000 7f0459000700 1 -- 172.21.15.12:0/3401536985 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f04540628a0 con 0x7f04540743c0 2024-03-23T23:00:11.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.578+0000 7f044aa00700 1 -- 172.21.15.12:0/3401536985 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f044402a070 con 0x7f04540743c0 2024-03-23T23:00:11.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.579+0000 7f044aa00700 1 --2- 172.21.15.12:0/3401536985 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f04380a29a0 0x7f04380a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:11.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.579+0000 7f044aa00700 1 -- 172.21.15.12:0/3401536985 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f04440cfba0 con 0x7f04540743c0 2024-03-23T23:00:11.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.579+0000 7f0451400700 1 --2- 172.21.15.12:0/3401536985 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f04380a29a0 0x7f04380a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:11.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.580+0000 7f044aa00700 1 --2- 172.21.15.12:0/3401536985 >> [v2:172.21.15.203:6816/1516996192,v1:172.21.15.203:6817/1516996192] conn(0x7f04380ac050 0x7f04380ae470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:11.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.580+0000 7f044aa00700 1 -- 172.21.15.12:0/3401536985 --> [v2:172.21.15.203:6816/1516996192,v1:172.21.15.203:6817/1516996192] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0444010ea0 con 0x7f04380ac050 2024-03-23T23:00:11.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.580+0000 7f044aa00700 1 -- 172.21.15.12:0/3401536985 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04440cff50 con 0x7f04540743c0 2024-03-23T23:00:11.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.580+0000 7f0452800700 1 --2- 172.21.15.12:0/3401536985 >> [v2:172.21.15.203:6816/1516996192,v1:172.21.15.203:6817/1516996192] conn(0x7f04380ac050 0x7f04380ae470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:11.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.580+0000 7f0451400700 1 --2- 172.21.15.12:0/3401536985 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f04380a29a0 0x7f04380a4e60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0440009680 tx=0x7f0440009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:11.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.580+0000 7f0452800700 1 --2- 172.21.15.12:0/3401536985 >> [v2:172.21.15.203:6816/1516996192,v1:172.21.15.203:6817/1516996192] conn(0x7f04380ac050 0x7f04380ae470 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:11.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.581+0000 7f044aa00700 1 -- 172.21.15.12:0/3401536985 <== osd.8 v2:172.21.15.203:6816/1516996192 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0444010ea0 con 0x7f04380ac050 2024-03-23T23:00:11.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.600+0000 7f0459000700 1 -- 172.21.15.12:0/3401536985 --> [v2:172.21.15.203:6816/1516996192,v1:172.21.15.203:6817/1516996192] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0454062ca0 con 0x7f04380ac050 2024-03-23T23:00:11.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.601+0000 7f044aa00700 1 -- 172.21.15.12:0/3401536985 <== osd.8 v2:172.21.15.203:6816/1516996192 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0454062ca0 con 0x7f04380ac050 2024-03-23T23:00:11.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.601+0000 7f0459000700 1 -- 172.21.15.12:0/3401536985 >> [v2:172.21.15.203:6816/1516996192,v1:172.21.15.203:6817/1516996192] conn(0x7f04380ac050 msgr2=0x7f04380ae470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:11.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.601+0000 7f0459000700 1 --2- 172.21.15.12:0/3401536985 >> [v2:172.21.15.203:6816/1516996192,v1:172.21.15.203:6817/1516996192] conn(0x7f04380ac050 0x7f04380ae470 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:11.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.602+0000 7f0459000700 1 -- 172.21.15.12:0/3401536985 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f04380a29a0 msgr2=0x7f04380a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:11.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.602+0000 7f0459000700 1 --2- 172.21.15.12:0/3401536985 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f04380a29a0 0x7f04380a4e60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0440009680 tx=0x7f0440009260 comp rx=0 tx=0).stop 2024-03-23T23:00:11.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.602+0000 7f0459000700 1 -- 172.21.15.12:0/3401536985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04540743c0 msgr2=0x7f04541a0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:11.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.602+0000 7f0459000700 1 --2- 172.21.15.12:0/3401536985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04540743c0 0x7f04541a0860 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0444000c00 tx=0x7f044400bab0 comp rx=0 tx=0).stop 2024-03-23T23:00:11.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.602+0000 7f0452800700 1 -- 172.21.15.12:0/3401536985 reap_dead start 2024-03-23T23:00:11.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.602+0000 7f0459000700 1 -- 172.21.15.12:0/3401536985 shutdown_connections 2024-03-23T23:00:11.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.602+0000 7f0459000700 1 -- 172.21.15.12:0/3401536985 >> 172.21.15.12:0/3401536985 conn(0x7f04540792f0 msgr2=0x7f04540ff400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:11.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.602+0000 7f0459000700 1 -- 172.21.15.12:0/3401536985 shutdown_connections 2024-03-23T23:00:11.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.602+0000 7f0459000700 1 -- 172.21.15.12:0/3401536985 wait complete. 2024-03-23T23:00:12.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.447+0000 7fd80d800700 1 -- 172.21.15.12:0/1652859110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd808104600 msgr2=0x7fd8081049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:12.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.447+0000 7fd80d800700 1 --2- 172.21.15.12:0/1652859110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd808104600 0x7fd8081049e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8009920 tx=0x7fd7f8009c30 comp rx=0 tx=0).stop 2024-03-23T23:00:12.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.447+0000 7fd80d800700 1 -- 172.21.15.12:0/1652859110 shutdown_connections 2024-03-23T23:00:12.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.447+0000 7fd80d800700 1 --2- 172.21.15.12:0/1652859110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8081096e0 0x7fd8081112b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:12.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.447+0000 7fd80d800700 1 --2- 172.21.15.12:0/1652859110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd808104600 0x7fd8081049e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:12.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.447+0000 7fd80d800700 1 --2- 172.21.15.12:0/1652859110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd808104fb0 0x7fd808109000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:12.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.447+0000 7fd80d800700 1 -- 172.21.15.12:0/1652859110 >> 172.21.15.12:0/1652859110 conn(0x7fd8080ffe70 msgr2=0x7fd808102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:12.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.447+0000 7fd80d800700 1 -- 172.21.15.12:0/1652859110 shutdown_connections 2024-03-23T23:00:12.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.447+0000 7fd80d800700 1 -- 172.21.15.12:0/1652859110 wait complete. 2024-03-23T23:00:12.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.448+0000 7fd80d800700 1 Processor -- start 2024-03-23T23:00:12.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.448+0000 7fd80d800700 1 -- start start 2024-03-23T23:00:12.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.449+0000 7fd80d800700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd808104600 0x7fd8081a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:12.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.449+0000 7fd80d800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd808104fb0 0x7fd8081a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:12.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.449+0000 7fd80d800700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8081096e0 0x7fd80819aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:12.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.449+0000 7fd80d800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd808114730 con 0x7fd808104fb0 2024-03-23T23:00:12.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.449+0000 7fd80d800700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8081145b0 con 0x7fd808104600 2024-03-23T23:00:12.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.449+0000 7fd80d800700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8081148b0 con 0x7fd8081096e0 2024-03-23T23:00:12.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.449+0000 7fd807400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8081096e0 0x7fd80819aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:12.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.449+0000 7fd806a00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd808104600 0x7fd8081a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:12.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.449+0000 7fd7ffe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd808104fb0 0x7fd8081a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:12.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.449+0000 7fd807400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8081096e0 0x7fd80819aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47732/0 (socket says 172.21.15.12:47732) 2024-03-23T23:00:12.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.449+0000 7fd806a00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd808104600 0x7fd8081a08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:49720/0 (socket says 172.21.15.12:49720) 2024-03-23T23:00:12.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.449+0000 7fd807400700 1 -- 172.21.15.12:0/3385549332 learned_addr learned my addr 172.21.15.12:0/3385549332 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:12.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.449+0000 7fd807400700 1 -- 172.21.15.12:0/3385549332 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd808104600 msgr2=0x7fd8081a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:12.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.449+0000 7fd807400700 1 --2- 172.21.15.12:0/3385549332 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd808104600 0x7fd8081a08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:12.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.449+0000 7fd807400700 1 -- 172.21.15.12:0/3385549332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd808104fb0 msgr2=0x7fd8081a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:12.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.449+0000 7fd807400700 1 --2- 172.21.15.12:0/3385549332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd808104fb0 0x7fd8081a0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:12.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.449+0000 7fd807400700 1 -- 172.21.15.12:0/3385549332 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd80819b1f0 con 0x7fd8081096e0 2024-03-23T23:00:12.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.450+0000 7fd806a00700 1 --2- 172.21.15.12:0/3385549332 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd808104600 0x7fd8081a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:00:12.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.450+0000 7fd807400700 1 --2- 172.21.15.12:0/3385549332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8081096e0 0x7fd80819aa90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd80000ef30 tx=0x7fd80000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:12.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.450+0000 7fd7ff400700 1 -- 172.21.15.12:0/3385549332 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8000092e0 con 0x7fd8081096e0 2024-03-23T23:00:12.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.450+0000 7fd7ff400700 1 -- 172.21.15.12:0/3385549332 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd800004bb0 con 0x7fd8081096e0 2024-03-23T23:00:12.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.450+0000 7fd7ff400700 1 -- 172.21.15.12:0/3385549332 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd800004280 con 0x7fd8081096e0 2024-03-23T23:00:12.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.450+0000 7fd80d800700 1 -- 172.21.15.12:0/3385549332 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd80819b480 con 0x7fd8081096e0 2024-03-23T23:00:12.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.450+0000 7fd80d800700 1 -- 172.21.15.12:0/3385549332 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8081a77a0 con 0x7fd8081096e0 2024-03-23T23:00:12.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.451+0000 7fd80d800700 1 -- 172.21.15.12:0/3385549332 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd808062920 con 0x7fd8081096e0 2024-03-23T23:00:12.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.455+0000 7fd7ff400700 1 -- 172.21.15.12:0/3385549332 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fd80001b030 con 0x7fd8081096e0 2024-03-23T23:00:12.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.456+0000 7fd7ff400700 1 --2- 172.21.15.12:0/3385549332 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd7ec0a2920 0x7fd7ec0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:12.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.456+0000 7fd7ff400700 1 -- 172.21.15.12:0/3385549332 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd8000ceef0 con 0x7fd8081096e0 2024-03-23T23:00:12.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.457+0000 7fd806a00700 1 --2- 172.21.15.12:0/3385549332 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd7ec0a2920 0x7fd7ec0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:12.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.457+0000 7fd7ff400700 1 --2- 172.21.15.12:0/3385549332 >> [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611] conn(0x7fd7ec0ac100 0x7fd7ec0ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:12.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.457+0000 7fd7ff400700 1 -- 172.21.15.12:0/3385549332 --> [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd80001fe50 con 0x7fd7ec0ac100 2024-03-23T23:00:12.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.457+0000 7fd7ff400700 1 -- 172.21.15.12:0/3385549332 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8000cf2a0 con 0x7fd8081096e0 2024-03-23T23:00:12.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.457+0000 7fd7ffe00700 1 --2- 172.21.15.12:0/3385549332 >> [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611] conn(0x7fd7ec0ac100 0x7fd7ec0ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:12.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.458+0000 7fd806a00700 1 --2- 172.21.15.12:0/3385549332 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd7ec0a2920 0x7fd7ec0a4de0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd80819bbd0 tx=0x7fd7f8019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:12.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.458+0000 7fd7ffe00700 1 --2- 172.21.15.12:0/3385549332 >> [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611] conn(0x7fd7ec0ac100 0x7fd7ec0ae520 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:12.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.459+0000 7fd7ff400700 1 -- 172.21.15.12:0/3385549332 <== osd.0 v2:172.21.15.12:6802/1190079611 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd80001fe50 con 0x7fd7ec0ac100 2024-03-23T23:00:12.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.477+0000 7fd80d800700 1 -- 172.21.15.12:0/3385549332 --> [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd808062da0 con 0x7fd7ec0ac100 2024-03-23T23:00:12.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.478+0000 7fd7ff400700 1 -- 172.21.15.12:0/3385549332 <== osd.0 v2:172.21.15.12:6802/1190079611 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd808062da0 con 0x7fd7ec0ac100 2024-03-23T23:00:12.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.478+0000 7fd80d800700 1 -- 172.21.15.12:0/3385549332 >> [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611] conn(0x7fd7ec0ac100 msgr2=0x7fd7ec0ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:12.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.478+0000 7fd80d800700 1 --2- 172.21.15.12:0/3385549332 >> [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611] conn(0x7fd7ec0ac100 0x7fd7ec0ae520 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:12.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.479+0000 7fd80d800700 1 -- 172.21.15.12:0/3385549332 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd7ec0a2920 msgr2=0x7fd7ec0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:12.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.479+0000 7fd80d800700 1 --2- 172.21.15.12:0/3385549332 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd7ec0a2920 0x7fd7ec0a4de0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd80819bbd0 tx=0x7fd7f8019040 comp rx=0 tx=0).stop 2024-03-23T23:00:12.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.479+0000 7fd80d800700 1 -- 172.21.15.12:0/3385549332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8081096e0 msgr2=0x7fd80819aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:12.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.479+0000 7fd80d800700 1 --2- 172.21.15.12:0/3385549332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8081096e0 0x7fd80819aa90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd80000ef30 tx=0x7fd80000c3a0 comp rx=0 tx=0).stop 2024-03-23T23:00:12.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.479+0000 7fd807400700 1 -- 172.21.15.12:0/3385549332 reap_dead start 2024-03-23T23:00:12.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.479+0000 7fd80d800700 1 -- 172.21.15.12:0/3385549332 shutdown_connections 2024-03-23T23:00:12.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.479+0000 7fd80d800700 1 -- 172.21.15.12:0/3385549332 >> 172.21.15.12:0/3385549332 conn(0x7fd8080ffe70 msgr2=0x7fd808101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:12.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.479+0000 7fd80d800700 1 -- 172.21.15.12:0/3385549332 shutdown_connections 2024-03-23T23:00:12.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:11.479+0000 7fd80d800700 1 -- 172.21.15.12:0/3385549332 wait complete. 2024-03-23T23:00:12.790 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:12 smithi012 ceph-mon[31695]: pgmap v340: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-23T23:00:12.846 INFO:teuthology.orchestra.run.smithi012.stdout:313532612635 2024-03-23T23:00:12.846 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-23T23:00:12.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:12 smithi173 ceph-mon[36116]: pgmap v340: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-23T23:00:12.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:12 smithi203 ceph-mon[36693]: pgmap v340: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-23T23:00:13.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.388+0000 7f17c5c00700 1 -- 172.21.15.12:0/928708043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17c01081e0 msgr2=0x7f17c01085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:13.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.388+0000 7f17c5c00700 1 --2- 172.21.15.12:0/928708043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17c01081e0 0x7f17c01085c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f17b000b0a0 tx=0x7f17b000b3b0 comp rx=0 tx=0).stop 2024-03-23T23:00:13.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.388+0000 7f17c5c00700 1 -- 172.21.15.12:0/928708043 shutdown_connections 2024-03-23T23:00:13.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.388+0000 7f17c5c00700 1 --2- 172.21.15.12:0/928708043 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17c006a7e0 0x7f17c010f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:13.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.388+0000 7f17c5c00700 1 --2- 172.21.15.12:0/928708043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17c01081e0 0x7f17c01085c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:13.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.388+0000 7f17c5c00700 1 --2- 172.21.15.12:0/928708043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17c0069e40 0x7f17c006a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:13.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.388+0000 7f17c5c00700 1 -- 172.21.15.12:0/928708043 >> 172.21.15.12:0/928708043 conn(0x7f17c0077fc0 msgr2=0x7f17c00783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:13.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.388+0000 7f17c5c00700 1 -- 172.21.15.12:0/928708043 shutdown_connections 2024-03-23T23:00:13.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.388+0000 7f17c5c00700 1 -- 172.21.15.12:0/928708043 wait complete. 2024-03-23T23:00:13.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.390+0000 7f17c5c00700 1 Processor -- start 2024-03-23T23:00:13.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.390+0000 7f17c5c00700 1 -- start start 2024-03-23T23:00:13.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.390+0000 7f17c5c00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17c0069e40 0x7f17c01a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:13.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.390+0000 7f17c5c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17c006a7e0 0x7f17c01a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:13.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.390+0000 7f17c5c00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17c01081e0 0x7f17c019a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:13.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.390+0000 7f17c5c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c0112470 con 0x7f17c006a7e0 2024-03-23T23:00:13.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.390+0000 7f17c5c00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c01122f0 con 0x7f17c0069e40 2024-03-23T23:00:13.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.390+0000 7f17c5c00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c01125f0 con 0x7f17c01081e0 2024-03-23T23:00:13.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.390+0000 7f17bea00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17c0069e40 0x7f17c01a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:13.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.390+0000 7f17bea00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17c0069e40 0x7f17c01a0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:49760/0 (socket says 172.21.15.12:49760) 2024-03-23T23:00:13.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.390+0000 7f17bea00700 1 -- 172.21.15.12:0/2933543384 learned_addr learned my addr 172.21.15.12:0/2933543384 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:13.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.390+0000 7f17bf400700 1 --2- 172.21.15.12:0/2933543384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17c01081e0 0x7f17c019a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:13.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.391+0000 7f17b6000700 1 --2- 172.21.15.12:0/2933543384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17c006a7e0 0x7f17c01a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:13.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.391+0000 7f17bea00700 1 -- 172.21.15.12:0/2933543384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17c01081e0 msgr2=0x7f17c019a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:13.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.391+0000 7f17bea00700 1 --2- 172.21.15.12:0/2933543384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17c01081e0 0x7f17c019a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:13.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.391+0000 7f17bea00700 1 -- 172.21.15.12:0/2933543384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17c006a7e0 msgr2=0x7f17c01a0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:13.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.391+0000 7f17bea00700 1 --2- 172.21.15.12:0/2933543384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17c006a7e0 0x7f17c01a0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:13.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.391+0000 7f17bea00700 1 -- 172.21.15.12:0/2933543384 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c019b120 con 0x7f17c0069e40 2024-03-23T23:00:13.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.391+0000 7f17b6000700 1 --2- 172.21.15.12:0/2933543384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17c006a7e0 0x7f17c01a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:00:13.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.391+0000 7f17bea00700 1 --2- 172.21.15.12:0/2933543384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17c0069e40 0x7f17c01a0800 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f17b00062a0 tx=0x7f17b0004ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:13.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.391+0000 7f17b7e00700 1 -- 172.21.15.12:0/2933543384 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b0021070 con 0x7f17c0069e40 2024-03-23T23:00:13.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.391+0000 7f17b7e00700 1 -- 172.21.15.12:0/2933543384 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17b0007b80 con 0x7f17c0069e40 2024-03-23T23:00:13.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.391+0000 7f17b7e00700 1 -- 172.21.15.12:0/2933543384 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b00052a0 con 0x7f17c0069e40 2024-03-23T23:00:13.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.391+0000 7f17c5c00700 1 -- 172.21.15.12:0/2933543384 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c019b3b0 con 0x7f17c0069e40 2024-03-23T23:00:13.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.392+0000 7f17c5c00700 1 -- 172.21.15.12:0/2933543384 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c01a76d0 con 0x7f17c0069e40 2024-03-23T23:00:13.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.392+0000 7f17c5c00700 1 -- 172.21.15.12:0/2933543384 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f17c010dd50 con 0x7f17c0069e40 2024-03-23T23:00:13.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.395+0000 7f17b7e00700 1 -- 172.21.15.12:0/2933543384 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f17b0024070 con 0x7f17c0069e40 2024-03-23T23:00:13.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.396+0000 7f17b7e00700 1 --2- 172.21.15.12:0/2933543384 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f17a40a29a0 0x7f17a40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:13.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.396+0000 7f17b7e00700 1 -- 172.21.15.12:0/2933543384 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f17b000f020 con 0x7f17c0069e40 2024-03-23T23:00:13.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.396+0000 7f17b6000700 1 --2- 172.21.15.12:0/2933543384 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f17a40a29a0 0x7f17a40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:13.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.397+0000 7f17b7e00700 1 --2- 172.21.15.12:0/2933543384 >> [v2:172.21.15.173:6816/1241504346,v1:172.21.15.173:6817/1241504346] conn(0x7f17a40ac180 0x7f17a40ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:13.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.397+0000 7f17b7e00700 1 -- 172.21.15.12:0/2933543384 --> [v2:172.21.15.173:6816/1241504346,v1:172.21.15.173:6817/1241504346] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f17a40aec50 con 0x7f17a40ac180 2024-03-23T23:00:13.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.397+0000 7f17b7e00700 1 -- 172.21.15.12:0/2933543384 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f17b00d0570 con 0x7f17c0069e40 2024-03-23T23:00:13.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.397+0000 7f17bf400700 1 --2- 172.21.15.12:0/2933543384 >> [v2:172.21.15.173:6816/1241504346,v1:172.21.15.173:6817/1241504346] conn(0x7f17a40ac180 0x7f17a40ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:13.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.397+0000 7f17b6000700 1 --2- 172.21.15.12:0/2933543384 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f17a40a29a0 0x7f17a40a4e60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f17c0103090 tx=0x7f17a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:13.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.397+0000 7f17bf400700 1 --2- 172.21.15.12:0/2933543384 >> [v2:172.21.15.173:6816/1241504346,v1:172.21.15.173:6817/1241504346] conn(0x7f17a40ac180 0x7f17a40ae5a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:13.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.399+0000 7f17b7e00700 1 -- 172.21.15.12:0/2933543384 <== osd.7 v2:172.21.15.173:6816/1241504346 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f17a40aec50 con 0x7f17a40ac180 2024-03-23T23:00:13.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.417+0000 7f17c5c00700 1 -- 172.21.15.12:0/2933543384 --> [v2:172.21.15.173:6816/1241504346,v1:172.21.15.173:6817/1241504346] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f17c0103890 con 0x7f17a40ac180 2024-03-23T23:00:13.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.417+0000 7f17b7e00700 1 -- 172.21.15.12:0/2933543384 <== osd.7 v2:172.21.15.173:6816/1241504346 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f17c0103890 con 0x7f17a40ac180 2024-03-23T23:00:13.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.418+0000 7f17c5c00700 1 -- 172.21.15.12:0/2933543384 >> [v2:172.21.15.173:6816/1241504346,v1:172.21.15.173:6817/1241504346] conn(0x7f17a40ac180 msgr2=0x7f17a40ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:13.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.418+0000 7f17c5c00700 1 --2- 172.21.15.12:0/2933543384 >> [v2:172.21.15.173:6816/1241504346,v1:172.21.15.173:6817/1241504346] conn(0x7f17a40ac180 0x7f17a40ae5a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:13.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.418+0000 7f17c5c00700 1 -- 172.21.15.12:0/2933543384 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f17a40a29a0 msgr2=0x7f17a40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:13.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.418+0000 7f17c5c00700 1 --2- 172.21.15.12:0/2933543384 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f17a40a29a0 0x7f17a40a4e60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f17c0103090 tx=0x7f17a8009260 comp rx=0 tx=0).stop 2024-03-23T23:00:13.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.418+0000 7f17c5c00700 1 -- 172.21.15.12:0/2933543384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17c0069e40 msgr2=0x7f17c01a0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:13.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.418+0000 7f17c5c00700 1 --2- 172.21.15.12:0/2933543384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17c0069e40 0x7f17c01a0800 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f17b00062a0 tx=0x7f17b0004ce0 comp rx=0 tx=0).stop 2024-03-23T23:00:13.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.418+0000 7f17bf400700 1 -- 172.21.15.12:0/2933543384 reap_dead start 2024-03-23T23:00:13.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.418+0000 7f17c5c00700 1 -- 172.21.15.12:0/2933543384 shutdown_connections 2024-03-23T23:00:13.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.418+0000 7f17c5c00700 1 -- 172.21.15.12:0/2933543384 >> 172.21.15.12:0/2933543384 conn(0x7f17c0077fc0 msgr2=0x7f17c010d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:13.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.419+0000 7f17c5c00700 1 -- 172.21.15.12:0/2933543384 shutdown_connections 2024-03-23T23:00:13.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:13.419+0000 7f17c5c00700 1 -- 172.21.15.12:0/2933543384 wait complete. 2024-03-23T23:00:13.750 INFO:teuthology.orchestra.run.smithi012.stdout:34359738453 2024-03-23T23:00:13.751 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-23T23:00:14.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.080+0000 7efc0c000700 1 -- 172.21.15.12:0/1571154702 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc04069e40 msgr2=0x7efc0406a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:14.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.080+0000 7efc0c000700 1 --2- 172.21.15.12:0/1571154702 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc04069e40 0x7efc0406a220 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7efbf8009990 tx=0x7efbf8009ca0 comp rx=0 tx=0).stop 2024-03-23T23:00:14.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.081+0000 7efc0c000700 1 -- 172.21.15.12:0/1571154702 shutdown_connections 2024-03-23T23:00:14.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.081+0000 7efc0c000700 1 --2- 172.21.15.12:0/1571154702 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc04069e40 0x7efc0406a220 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:14.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.081+0000 7efc0c000700 1 --2- 172.21.15.12:0/1571154702 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc0410ec40 0x7efc04111030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:14.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.081+0000 7efc0c000700 1 --2- 172.21.15.12:0/1571154702 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc0406a760 0x7efc0410e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:14.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.081+0000 7efc0c000700 1 -- 172.21.15.12:0/1571154702 >> 172.21.15.12:0/1571154702 conn(0x7efc04077fc0 msgr2=0x7efc040783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:14.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.081+0000 7efc0c000700 1 -- 172.21.15.12:0/1571154702 shutdown_connections 2024-03-23T23:00:14.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.081+0000 7efc0c000700 1 -- 172.21.15.12:0/1571154702 wait complete. 2024-03-23T23:00:14.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.082+0000 7efc0c000700 1 Processor -- start 2024-03-23T23:00:14.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.082+0000 7efc0c000700 1 -- start start 2024-03-23T23:00:14.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.082+0000 7efc0c000700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc04069e40 0x7efc041a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:14.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.082+0000 7efc0c000700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc0406a760 0x7efc041a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:14.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.082+0000 7efc0c000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc0410ec40 0x7efc0419a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:14.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.082+0000 7efc0c000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc041146a0 con 0x7efc0410ec40 2024-03-23T23:00:14.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.082+0000 7efc0c000700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc04114520 con 0x7efc04069e40 2024-03-23T23:00:14.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.082+0000 7efc0c000700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc04114820 con 0x7efc0406a760 2024-03-23T23:00:14.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.083+0000 7efc09600700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc04069e40 0x7efc041a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:14.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.083+0000 7efc08c00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc0406a760 0x7efc041a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:14.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.083+0000 7efc08c00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc0406a760 0x7efc041a0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47062/0 (socket says 172.21.15.12:47062) 2024-03-23T23:00:14.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.083+0000 7efc08c00700 1 -- 172.21.15.12:0/525010158 learned_addr learned my addr 172.21.15.12:0/525010158 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:14.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.083+0000 7efc0a000700 1 --2- 172.21.15.12:0/525010158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc0410ec40 0x7efc0419a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:14.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.083+0000 7efc08c00700 1 -- 172.21.15.12:0/525010158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc04069e40 msgr2=0x7efc041a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:14.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.083+0000 7efc08c00700 1 --2- 172.21.15.12:0/525010158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc04069e40 0x7efc041a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:14.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.083+0000 7efc08c00700 1 -- 172.21.15.12:0/525010158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc0410ec40 msgr2=0x7efc0419a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:14.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.083+0000 7efc08c00700 1 --2- 172.21.15.12:0/525010158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc0410ec40 0x7efc0419a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:14.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.083+0000 7efc08c00700 1 -- 172.21.15.12:0/525010158 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc0419b120 con 0x7efc0406a760 2024-03-23T23:00:14.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.084+0000 7efc08c00700 1 --2- 172.21.15.12:0/525010158 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc0406a760 0x7efc041a0d40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7efbf000e9a0 tx=0x7efbf000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:14.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.084+0000 7efc02000700 1 -- 172.21.15.12:0/525010158 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf000cbd0 con 0x7efc0406a760 2024-03-23T23:00:14.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.084+0000 7efc02000700 1 -- 172.21.15.12:0/525010158 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efbf000cd70 con 0x7efc0406a760 2024-03-23T23:00:14.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.084+0000 7efc02000700 1 -- 172.21.15.12:0/525010158 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf0018750 con 0x7efc0406a760 2024-03-23T23:00:14.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.084+0000 7efc0c000700 1 -- 172.21.15.12:0/525010158 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc0419b410 con 0x7efc0406a760 2024-03-23T23:00:14.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.084+0000 7efc0c000700 1 -- 172.21.15.12:0/525010158 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc041a76d0 con 0x7efc0406a760 2024-03-23T23:00:14.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.085+0000 7efc0c000700 1 -- 172.21.15.12:0/525010158 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efc04062860 con 0x7efc0406a760 2024-03-23T23:00:14.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.088+0000 7efc02000700 1 -- 172.21.15.12:0/525010158 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7efbf0010430 con 0x7efc0406a760 2024-03-23T23:00:14.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.089+0000 7efc02000700 1 --2- 172.21.15.12:0/525010158 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7efbec0a29a0 0x7efbec0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:14.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.089+0000 7efc02000700 1 -- 172.21.15.12:0/525010158 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7efbf00cf5b0 con 0x7efc0406a760 2024-03-23T23:00:14.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.089+0000 7efc09600700 1 --2- 172.21.15.12:0/525010158 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7efbec0a29a0 0x7efbec0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:14.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.090+0000 7efc02000700 1 --2- 172.21.15.12:0/525010158 >> [v2:172.21.15.203:6800/2253997387,v1:172.21.15.203:6801/2253997387] conn(0x7efbec0ac180 0x7efbec0ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:14.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.090+0000 7efc02000700 1 -- 172.21.15.12:0/525010158 --> [v2:172.21.15.203:6800/2253997387,v1:172.21.15.203:6801/2253997387] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efbf001bec0 con 0x7efbec0ac180 2024-03-23T23:00:14.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.090+0000 7efc02000700 1 -- 172.21.15.12:0/525010158 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7efbf00cf960 con 0x7efc0406a760 2024-03-23T23:00:14.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.090+0000 7efc0a000700 1 --2- 172.21.15.12:0/525010158 >> [v2:172.21.15.203:6800/2253997387,v1:172.21.15.203:6801/2253997387] conn(0x7efbec0ac180 0x7efbec0ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:14.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.090+0000 7efc09600700 1 --2- 172.21.15.12:0/525010158 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7efbec0a29a0 0x7efbec0a4e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efc0419bb00 tx=0x7efbf800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:14.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.090+0000 7efc0a000700 1 --2- 172.21.15.12:0/525010158 >> [v2:172.21.15.203:6800/2253997387,v1:172.21.15.203:6801/2253997387] conn(0x7efbec0ac180 0x7efbec0ae5a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:14.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.091+0000 7efc02000700 1 -- 172.21.15.12:0/525010158 <== osd.2 v2:172.21.15.203:6800/2253997387 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7efbf001bec0 con 0x7efbec0ac180 2024-03-23T23:00:14.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.110+0000 7efc0c000700 1 -- 172.21.15.12:0/525010158 --> [v2:172.21.15.203:6800/2253997387,v1:172.21.15.203:6801/2253997387] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efc04062ce0 con 0x7efbec0ac180 2024-03-23T23:00:14.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.110+0000 7efc02000700 1 -- 172.21.15.12:0/525010158 <== osd.2 v2:172.21.15.203:6800/2253997387 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7efc04062ce0 con 0x7efbec0ac180 2024-03-23T23:00:14.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.111+0000 7efc0c000700 1 -- 172.21.15.12:0/525010158 >> [v2:172.21.15.203:6800/2253997387,v1:172.21.15.203:6801/2253997387] conn(0x7efbec0ac180 msgr2=0x7efbec0ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:14.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.111+0000 7efc0c000700 1 --2- 172.21.15.12:0/525010158 >> [v2:172.21.15.203:6800/2253997387,v1:172.21.15.203:6801/2253997387] conn(0x7efbec0ac180 0x7efbec0ae5a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:14.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.111+0000 7efc0c000700 1 -- 172.21.15.12:0/525010158 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7efbec0a29a0 msgr2=0x7efbec0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:14.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.111+0000 7efc0c000700 1 --2- 172.21.15.12:0/525010158 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7efbec0a29a0 0x7efbec0a4e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efc0419bb00 tx=0x7efbf800b260 comp rx=0 tx=0).stop 2024-03-23T23:00:14.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.111+0000 7efc0c000700 1 -- 172.21.15.12:0/525010158 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc0406a760 msgr2=0x7efc041a0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:14.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.111+0000 7efc0c000700 1 --2- 172.21.15.12:0/525010158 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc0406a760 0x7efc041a0d40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7efbf000e9a0 tx=0x7efbf000ecb0 comp rx=0 tx=0).stop 2024-03-23T23:00:14.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.111+0000 7efc0a000700 1 -- 172.21.15.12:0/525010158 reap_dead start 2024-03-23T23:00:14.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.111+0000 7efc0c000700 1 -- 172.21.15.12:0/525010158 shutdown_connections 2024-03-23T23:00:14.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.111+0000 7efc0c000700 1 -- 172.21.15.12:0/525010158 >> 172.21.15.12:0/525010158 conn(0x7efc04077fc0 msgr2=0x7efc04110170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:14.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.111+0000 7efc0c000700 1 -- 172.21.15.12:0/525010158 shutdown_connections 2024-03-23T23:00:14.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:14.111+0000 7efc0c000700 1 -- 172.21.15.12:0/525010158 wait complete. 2024-03-23T23:00:14.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:14 smithi012 ceph-mon[31695]: pgmap v341: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-23T23:00:14.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:14 smithi173 ceph-mon[36116]: pgmap v341: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-23T23:00:14.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:14 smithi203 ceph-mon[36693]: pgmap v341: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-23T23:00:14.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.180+0000 7f7efca00700 1 -- 172.21.15.12:0/1831701984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ef8107510 msgr2=0x7f7ef810f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:14.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.180+0000 7f7efca00700 1 --2- 172.21.15.12:0/1831701984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ef8107510 0x7f7ef810f0e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7ef000b0a0 tx=0x7f7ef000b3b0 comp rx=0 tx=0).stop 2024-03-23T23:00:14.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.180+0000 7f7efca00700 1 -- 172.21.15.12:0/1831701984 shutdown_connections 2024-03-23T23:00:14.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.180+0000 7f7efca00700 1 --2- 172.21.15.12:0/1831701984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef8102440 0x7f7ef8102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:14.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.180+0000 7f7efca00700 1 --2- 172.21.15.12:0/1831701984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ef8107510 0x7f7ef810f0e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:14.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.180+0000 7f7efca00700 1 --2- 172.21.15.12:0/1831701984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ef8102df0 0x7f7ef8106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:14.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.180+0000 7f7efca00700 1 -- 172.21.15.12:0/1831701984 >> 172.21.15.12:0/1831701984 conn(0x7f7ef80792f0 msgr2=0x7f7ef8079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:14.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.180+0000 7f7efca00700 1 -- 172.21.15.12:0/1831701984 shutdown_connections 2024-03-23T23:00:14.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.180+0000 7f7efca00700 1 -- 172.21.15.12:0/1831701984 wait complete. 2024-03-23T23:00:14.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.181+0000 7f7efca00700 1 Processor -- start 2024-03-23T23:00:14.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.182+0000 7f7efca00700 1 -- start start 2024-03-23T23:00:14.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.182+0000 7f7efca00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ef8102440 0x7f7ef81a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:14.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.182+0000 7f7efca00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef8102df0 0x7f7ef81a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:14.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.182+0000 7f7efca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ef8107510 0x7f7ef819aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:14.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.182+0000 7f7efca00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef8112580 con 0x7f7ef8107510 2024-03-23T23:00:14.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.182+0000 7f7efca00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef8112400 con 0x7f7ef8102440 2024-03-23T23:00:14.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.182+0000 7f7efca00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef8112700 con 0x7f7ef8102df0 2024-03-23T23:00:14.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.182+0000 7f7ef5400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef8102df0 0x7f7ef81a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:14.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.182+0000 7f7ef5e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ef8102440 0x7f7ef81a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:14.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.182+0000 7f7ef5e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ef8102440 0x7f7ef81a0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:49740/0 (socket says 172.21.15.12:49740) 2024-03-23T23:00:14.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.182+0000 7f7ef5400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef8102df0 0x7f7ef81a0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47768/0 (socket says 172.21.15.12:47768) 2024-03-23T23:00:14.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.182+0000 7f7ef5e00700 1 -- 172.21.15.12:0/2533531792 learned_addr learned my addr 172.21.15.12:0/2533531792 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:14.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.183+0000 7f7ef6800700 1 --2- 172.21.15.12:0/2533531792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ef8107510 0x7f7ef819aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:14.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.183+0000 7f7ef5e00700 1 -- 172.21.15.12:0/2533531792 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef8102df0 msgr2=0x7f7ef81a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:14.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.183+0000 7f7ef5e00700 1 --2- 172.21.15.12:0/2533531792 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef8102df0 0x7f7ef81a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:14.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.183+0000 7f7ef5e00700 1 -- 172.21.15.12:0/2533531792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ef8107510 msgr2=0x7f7ef819aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:14.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.183+0000 7f7ef5e00700 1 --2- 172.21.15.12:0/2533531792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ef8107510 0x7f7ef819aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:14.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.183+0000 7f7ef5e00700 1 -- 172.21.15.12:0/2533531792 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef819b180 con 0x7f7ef8102440 2024-03-23T23:00:14.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.183+0000 7f7ef5e00700 1 --2- 172.21.15.12:0/2533531792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ef8102440 0x7f7ef81a0860 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7ee800e9c0 tx=0x7f7ee800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:14.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.183+0000 7f7eeea00700 1 -- 172.21.15.12:0/2533531792 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee800cbd0 con 0x7f7ef8102440 2024-03-23T23:00:14.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.183+0000 7f7eeea00700 1 -- 172.21.15.12:0/2533531792 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ee800cd70 con 0x7f7ef8102440 2024-03-23T23:00:14.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.183+0000 7f7efca00700 1 -- 172.21.15.12:0/2533531792 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef819b470 con 0x7f7ef8102440 2024-03-23T23:00:14.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.183+0000 7f7eeea00700 1 -- 172.21.15.12:0/2533531792 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee8010430 con 0x7f7ef8102440 2024-03-23T23:00:14.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.184+0000 7f7efca00700 1 -- 172.21.15.12:0/2533531792 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef81a7730 con 0x7f7ef8102440 2024-03-23T23:00:14.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.185+0000 7f7efca00700 1 -- 172.21.15.12:0/2533531792 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7ef81012f0 con 0x7f7ef8102440 2024-03-23T23:00:14.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.186+0000 7f7eeea00700 1 -- 172.21.15.12:0/2533531792 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f7ee8004750 con 0x7f7ef8102440 2024-03-23T23:00:14.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.188+0000 7f7eeea00700 1 --2- 172.21.15.12:0/2533531792 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7edc0a29a0 0x7f7edc0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:14.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.188+0000 7f7eeea00700 1 -- 172.21.15.12:0/2533531792 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f7ee80ced80 con 0x7f7ef8102440 2024-03-23T23:00:14.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.188+0000 7f7ef5400700 1 --2- 172.21.15.12:0/2533531792 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7edc0a29a0 0x7f7edc0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:14.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.188+0000 7f7eeea00700 1 --2- 172.21.15.12:0/2533531792 >> [v2:172.21.15.12:6810/1761446905,v1:172.21.15.12:6811/1761446905] conn(0x7f7edc0ac180 0x7f7edc0ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:14.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.188+0000 7f7eeea00700 1 -- 172.21.15.12:0/2533531792 --> [v2:172.21.15.12:6810/1761446905,v1:172.21.15.12:6811/1761446905] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7edc0aec50 con 0x7f7edc0ac180 2024-03-23T23:00:14.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.188+0000 7f7eeea00700 1 -- 172.21.15.12:0/2533531792 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ee80d3030 con 0x7f7ef8102440 2024-03-23T23:00:14.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.189+0000 7f7ef6800700 1 --2- 172.21.15.12:0/2533531792 >> [v2:172.21.15.12:6810/1761446905,v1:172.21.15.12:6811/1761446905] conn(0x7f7edc0ac180 0x7f7edc0ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:14.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.189+0000 7f7ef5400700 1 --2- 172.21.15.12:0/2533531792 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7edc0a29a0 0x7f7edc0a4e60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4009680 tx=0x7f7ee4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:14.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.189+0000 7f7ef6800700 1 --2- 172.21.15.12:0/2533531792 >> [v2:172.21.15.12:6810/1761446905,v1:172.21.15.12:6811/1761446905] conn(0x7f7edc0ac180 0x7f7edc0ae5a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:14.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.191+0000 7f7eeea00700 1 -- 172.21.15.12:0/2533531792 <== osd.3 v2:172.21.15.12:6810/1761446905 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7edc0aec50 con 0x7f7edc0ac180 2024-03-23T23:00:14.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.210+0000 7f7efca00700 1 -- 172.21.15.12:0/2533531792 --> [v2:172.21.15.12:6810/1761446905,v1:172.21.15.12:6811/1761446905] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7ef8103910 con 0x7f7edc0ac180 2024-03-23T23:00:14.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.210+0000 7f7eeea00700 1 -- 172.21.15.12:0/2533531792 <== osd.3 v2:172.21.15.12:6810/1761446905 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7ef8103910 con 0x7f7edc0ac180 2024-03-23T23:00:14.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.211+0000 7f7efca00700 1 -- 172.21.15.12:0/2533531792 >> [v2:172.21.15.12:6810/1761446905,v1:172.21.15.12:6811/1761446905] conn(0x7f7edc0ac180 msgr2=0x7f7edc0ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:14.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.211+0000 7f7efca00700 1 --2- 172.21.15.12:0/2533531792 >> [v2:172.21.15.12:6810/1761446905,v1:172.21.15.12:6811/1761446905] conn(0x7f7edc0ac180 0x7f7edc0ae5a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:14.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.211+0000 7f7efca00700 1 -- 172.21.15.12:0/2533531792 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7edc0a29a0 msgr2=0x7f7edc0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:14.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.211+0000 7f7efca00700 1 --2- 172.21.15.12:0/2533531792 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7edc0a29a0 0x7f7edc0a4e60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4009680 tx=0x7f7ee4009260 comp rx=0 tx=0).stop 2024-03-23T23:00:14.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.211+0000 7f7efca00700 1 -- 172.21.15.12:0/2533531792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ef8102440 msgr2=0x7f7ef81a0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:14.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.211+0000 7f7efca00700 1 --2- 172.21.15.12:0/2533531792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ef8102440 0x7f7ef81a0860 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7ee800e9c0 tx=0x7f7ee800ecd0 comp rx=0 tx=0).stop 2024-03-23T23:00:14.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.211+0000 7f7ef6800700 1 -- 172.21.15.12:0/2533531792 reap_dead start 2024-03-23T23:00:14.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.211+0000 7f7efca00700 1 -- 172.21.15.12:0/2533531792 shutdown_connections 2024-03-23T23:00:14.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.211+0000 7f7efca00700 1 -- 172.21.15.12:0/2533531792 >> 172.21.15.12:0/2533531792 conn(0x7f7ef80792f0 msgr2=0x7f7ef810d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:14.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.212+0000 7f7efca00700 1 -- 172.21.15.12:0/2533531792 shutdown_connections 2024-03-23T23:00:14.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:12.212+0000 7f7efca00700 1 -- 172.21.15.12:0/2533531792 wait complete. 2024-03-23T23:00:16.268 INFO:teuthology.orchestra.run.smithi012.stdout:77309411403 2024-03-23T23:00:16.268 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-23T23:00:16.650 INFO:teuthology.orchestra.run.smithi012.stdout:111669149764 2024-03-23T23:00:16.651 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-23T23:00:16.838 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:16 smithi012 ceph-mon[31695]: pgmap v342: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-23T23:00:16.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:16 smithi173 ceph-mon[36116]: pgmap v342: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-23T23:00:16.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:16 smithi203 ceph-mon[36693]: pgmap v342: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-23T23:00:17.171 INFO:teuthology.orchestra.run.smithi012.stdout:223338299437 2024-03-23T23:00:17.172 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-23T23:00:17.805 INFO:teuthology.orchestra.run.smithi012.stdout:257698037800 2024-03-23T23:00:17.805 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-23T23:00:17.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.989+0000 7f45cfe00700 1 -- 172.21.15.12:0/303605459 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45c81081c0 msgr2=0x7f45c81085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.989+0000 7f45cfe00700 1 --2- 172.21.15.12:0/303605459 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45c81081c0 0x7f45c81085a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f45bc009990 tx=0x7f45bc009ca0 comp rx=0 tx=0).stop 2024-03-23T23:00:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.989+0000 7f45cfe00700 1 -- 172.21.15.12:0/303605459 shutdown_connections 2024-03-23T23:00:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.989+0000 7f45cfe00700 1 --2- 172.21.15.12:0/303605459 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45c81081c0 0x7f45c81085a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.989+0000 7f45cfe00700 1 --2- 172.21.15.12:0/303605459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45c8102ba0 0x7f45c810efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.989+0000 7f45cfe00700 1 --2- 172.21.15.12:0/303605459 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45c81021e0 0x7f45c8102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.989+0000 7f45cfe00700 1 -- 172.21.15.12:0/303605459 >> 172.21.15.12:0/303605459 conn(0x7f45c80792c0 msgr2=0x7f45c80796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.990+0000 7f45cfe00700 1 -- 172.21.15.12:0/303605459 shutdown_connections 2024-03-23T23:00:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.990+0000 7f45cfe00700 1 -- 172.21.15.12:0/303605459 wait complete. 2024-03-23T23:00:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.991+0000 7f45cfe00700 1 Processor -- start 2024-03-23T23:00:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.991+0000 7f45cfe00700 1 -- start start 2024-03-23T23:00:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.992+0000 7f45cfe00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45c81021e0 0x7f45c819e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.992+0000 7f45cfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45c8102ba0 0x7f45c819eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:17.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.992+0000 7f45cfe00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45c81081c0 0x7f45c8198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:17.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.992+0000 7f45cfe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c81124c0 con 0x7f45c8102ba0 2024-03-23T23:00:17.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.992+0000 7f45cfe00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c8112340 con 0x7f45c81021e0 2024-03-23T23:00:17.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.992+0000 7f45cfe00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c8112640 con 0x7f45c81081c0 2024-03-23T23:00:17.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.992+0000 7f45cd400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45c81021e0 0x7f45c819e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:17.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.992+0000 7f45cd400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45c81021e0 0x7f45c819e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:42530/0 (socket says 172.21.15.12:42530) 2024-03-23T23:00:17.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.992+0000 7f45cd400700 1 -- 172.21.15.12:0/1862766089 learned_addr learned my addr 172.21.15.12:0/1862766089 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:17.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.992+0000 7f45cca00700 1 --2- 172.21.15.12:0/1862766089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45c8102ba0 0x7f45c819eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:17.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.993+0000 7f45cd400700 1 -- 172.21.15.12:0/1862766089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45c81081c0 msgr2=0x7f45c8198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:17.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.993+0000 7f45cd400700 1 --2- 172.21.15.12:0/1862766089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45c81081c0 0x7f45c8198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:17.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.993+0000 7f45cd400700 1 -- 172.21.15.12:0/1862766089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45c8102ba0 msgr2=0x7f45c819eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:17.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.993+0000 7f45cd400700 1 --2- 172.21.15.12:0/1862766089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45c8102ba0 0x7f45c819eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:17.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.993+0000 7f45cd400700 1 -- 172.21.15.12:0/1862766089 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c8199020 con 0x7f45c81021e0 2024-03-23T23:00:17.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.993+0000 7f45cca00700 1 --2- 172.21.15.12:0/1862766089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45c8102ba0 0x7f45c819eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:00:17.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.993+0000 7f45cd400700 1 --2- 172.21.15.12:0/1862766089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45c81021e0 0x7f45c819e620 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f45bc000c00 tx=0x7f45bc005640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:17.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.993+0000 7f45c6000700 1 -- 172.21.15.12:0/1862766089 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45bc027070 con 0x7f45c81021e0 2024-03-23T23:00:17.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.993+0000 7f45c6000700 1 -- 172.21.15.12:0/1862766089 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45bc022470 con 0x7f45c81021e0 2024-03-23T23:00:17.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.994+0000 7f45cfe00700 1 -- 172.21.15.12:0/1862766089 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45c81992b0 con 0x7f45c81021e0 2024-03-23T23:00:17.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.993+0000 7f45c6000700 1 -- 172.21.15.12:0/1862766089 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45bc017650 con 0x7f45c81021e0 2024-03-23T23:00:17.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.994+0000 7f45cfe00700 1 -- 172.21.15.12:0/1862766089 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45c81a55a0 con 0x7f45c81021e0 2024-03-23T23:00:17.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.995+0000 7f45cfe00700 1 -- 172.21.15.12:0/1862766089 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f45c80628a0 con 0x7f45c81021e0 2024-03-23T23:00:17.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.997+0000 7f45c6000700 1 -- 172.21.15.12:0/1862766089 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f45bc009d10 con 0x7f45c81021e0 2024-03-23T23:00:17.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.999+0000 7f45c6000700 1 --2- 172.21.15.12:0/1862766089 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f45b00a29a0 0x7f45b00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:17.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.999+0000 7f45c6000700 1 -- 172.21.15.12:0/1862766089 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f45bc0cfc90 con 0x7f45c81021e0 2024-03-23T23:00:17.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.999+0000 7f45cca00700 1 --2- 172.21.15.12:0/1862766089 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f45b00a29a0 0x7f45b00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:17.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.999+0000 7f45c6000700 1 --2- 172.21.15.12:0/1862766089 >> [v2:172.21.15.203:6808/4270966092,v1:172.21.15.203:6809/4270966092] conn(0x7f45b00ac050 0x7f45b00ae470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:17.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.999+0000 7f45c6000700 1 -- 172.21.15.12:0/1862766089 --> [v2:172.21.15.203:6808/4270966092,v1:172.21.15.203:6809/4270966092] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f45bc001d30 con 0x7f45b00ac050 2024-03-23T23:00:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:16.999+0000 7f45c6000700 1 -- 172.21.15.12:0/1862766089 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45bc0d0040 con 0x7f45c81021e0 2024-03-23T23:00:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:17.000+0000 7f45cde00700 1 --2- 172.21.15.12:0/1862766089 >> [v2:172.21.15.203:6808/4270966092,v1:172.21.15.203:6809/4270966092] conn(0x7f45b00ac050 0x7f45b00ae470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:17.000+0000 7f45cca00700 1 --2- 172.21.15.12:0/1862766089 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f45b00a29a0 0x7f45b00a4e60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f45c8199eb0 tx=0x7f45b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:17.000+0000 7f45cde00700 1 --2- 172.21.15.12:0/1862766089 >> [v2:172.21.15.203:6808/4270966092,v1:172.21.15.203:6809/4270966092] conn(0x7f45b00ac050 0x7f45b00ae470 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:17.001+0000 7f45c6000700 1 -- 172.21.15.12:0/1862766089 <== osd.5 v2:172.21.15.203:6808/4270966092 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f45bc001d30 con 0x7f45b00ac050 2024-03-23T23:00:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:17.020+0000 7f45cfe00700 1 -- 172.21.15.12:0/1862766089 --> [v2:172.21.15.203:6808/4270966092,v1:172.21.15.203:6809/4270966092] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f45c8062cd0 con 0x7f45b00ac050 2024-03-23T23:00:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:17.021+0000 7f45c6000700 1 -- 172.21.15.12:0/1862766089 <== osd.5 v2:172.21.15.203:6808/4270966092 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f45c8062cd0 con 0x7f45b00ac050 2024-03-23T23:00:17.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:17.021+0000 7f45cfe00700 1 -- 172.21.15.12:0/1862766089 >> [v2:172.21.15.203:6808/4270966092,v1:172.21.15.203:6809/4270966092] conn(0x7f45b00ac050 msgr2=0x7f45b00ae470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:17.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:17.021+0000 7f45cfe00700 1 --2- 172.21.15.12:0/1862766089 >> [v2:172.21.15.203:6808/4270966092,v1:172.21.15.203:6809/4270966092] conn(0x7f45b00ac050 0x7f45b00ae470 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:17.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:17.021+0000 7f45cfe00700 1 -- 172.21.15.12:0/1862766089 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f45b00a29a0 msgr2=0x7f45b00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:17.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:17.021+0000 7f45cfe00700 1 --2- 172.21.15.12:0/1862766089 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f45b00a29a0 0x7f45b00a4e60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f45c8199eb0 tx=0x7f45b4009260 comp rx=0 tx=0).stop 2024-03-23T23:00:17.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:17.021+0000 7f45cfe00700 1 -- 172.21.15.12:0/1862766089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45c81021e0 msgr2=0x7f45c819e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:17.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:17.021+0000 7f45cfe00700 1 --2- 172.21.15.12:0/1862766089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45c81021e0 0x7f45c819e620 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f45bc000c00 tx=0x7f45bc005640 comp rx=0 tx=0).stop 2024-03-23T23:00:17.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:17.022+0000 7f45cde00700 1 -- 172.21.15.12:0/1862766089 reap_dead start 2024-03-23T23:00:17.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:17.022+0000 7f45cfe00700 1 -- 172.21.15.12:0/1862766089 shutdown_connections 2024-03-23T23:00:17.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:17.022+0000 7f45cfe00700 1 -- 172.21.15.12:0/1862766089 >> 172.21.15.12:0/1862766089 conn(0x7f45c80792c0 msgr2=0x7f45c8100de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:17.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:17.022+0000 7f45cfe00700 1 -- 172.21.15.12:0/1862766089 shutdown_connections 2024-03-23T23:00:17.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:17.022+0000 7f45cfe00700 1 -- 172.21.15.12:0/1862766089 wait complete. 2024-03-23T23:00:18.473 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:18 smithi012 ceph-mon[31695]: pgmap v343: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-23T23:00:18.533 INFO:teuthology.orchestra.run.smithi012.stdout:167503724602 2024-03-23T23:00:18.533 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-23T23:00:18.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:18 smithi173 ceph-mon[36116]: pgmap v343: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-23T23:00:18.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:18 smithi203 ceph-mon[36693]: pgmap v343: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-23T23:00:19.487 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:00:19.491 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:00:19.493 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:00:19.515 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:00:20.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:20 smithi012 ceph-mon[31695]: pgmap v344: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:00:20.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.256+0000 7ff448c00700 1 -- 172.21.15.12:0/1711564239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff444104f90 msgr2=0x7ff444108fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:20.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.256+0000 7ff448c00700 1 --2- 172.21.15.12:0/1711564239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff444104f90 0x7ff444108fe0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff4300099c0 tx=0x7ff430009cd0 comp rx=0 tx=0).stop 2024-03-23T23:00:20.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.256+0000 7ff448c00700 1 -- 172.21.15.12:0/1711564239 shutdown_connections 2024-03-23T23:00:20.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.256+0000 7ff448c00700 1 --2- 172.21.15.12:0/1711564239 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4441096c0 0x7ff444111290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:20.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.256+0000 7ff448c00700 1 --2- 172.21.15.12:0/1711564239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4441045e0 0x7ff4441049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:20.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.256+0000 7ff448c00700 1 --2- 172.21.15.12:0/1711564239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff444104f90 0x7ff444108fe0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:20.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.256+0000 7ff448c00700 1 -- 172.21.15.12:0/1711564239 >> 172.21.15.12:0/1711564239 conn(0x7ff4440ffe70 msgr2=0x7ff444102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:20.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.257+0000 7ff448c00700 1 -- 172.21.15.12:0/1711564239 shutdown_connections 2024-03-23T23:00:20.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.257+0000 7ff448c00700 1 -- 172.21.15.12:0/1711564239 wait complete. 2024-03-23T23:00:20.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.258+0000 7ff448c00700 1 Processor -- start 2024-03-23T23:00:20.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.258+0000 7ff448c00700 1 -- start start 2024-03-23T23:00:20.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.259+0000 7ff448c00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4441045e0 0x7ff4441a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:20.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.259+0000 7ff448c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff444104f90 0x7ff4441a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:20.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.259+0000 7ff448c00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4441096c0 0x7ff44419a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:20.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.259+0000 7ff448c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4441146d0 con 0x7ff444104f90 2024-03-23T23:00:20.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.259+0000 7ff448c00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff444114550 con 0x7ff4441045e0 2024-03-23T23:00:20.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.259+0000 7ff448c00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff444114850 con 0x7ff4441096c0 2024-03-23T23:00:20.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.259+0000 7ff441e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4441045e0 0x7ff4441a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:20.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.259+0000 7ff441e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4441045e0 0x7ff4441a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:42542/0 (socket says 172.21.15.12:42542) 2024-03-23T23:00:20.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.259+0000 7ff441e00700 1 -- 172.21.15.12:0/2099926765 learned_addr learned my addr 172.21.15.12:0/2099926765 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:20.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.259+0000 7ff441400700 1 --2- 172.21.15.12:0/2099926765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff444104f90 0x7ff4441a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:20.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.259+0000 7ff442800700 1 --2- 172.21.15.12:0/2099926765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4441096c0 0x7ff44419a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:20.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.259+0000 7ff441e00700 1 -- 172.21.15.12:0/2099926765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4441096c0 msgr2=0x7ff44419a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:20.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.259+0000 7ff441e00700 1 --2- 172.21.15.12:0/2099926765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4441096c0 0x7ff44419a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:20.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.259+0000 7ff441e00700 1 -- 172.21.15.12:0/2099926765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff444104f90 msgr2=0x7ff4441a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:20.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.259+0000 7ff441e00700 1 --2- 172.21.15.12:0/2099926765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff444104f90 0x7ff4441a0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.259+0000 7ff441e00700 1 -- 172.21.15.12:0/2099926765 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44419b0f0 con 0x7ff4441045e0 2024-03-23T23:00:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.260+0000 7ff441e00700 1 --2- 172.21.15.12:0/2099926765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4441045e0 0x7ff4441a0780 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff43400e9a0 tx=0x7ff43400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.260+0000 7ff43aa00700 1 -- 172.21.15.12:0/2099926765 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43400ca10 con 0x7ff4441045e0 2024-03-23T23:00:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.260+0000 7ff448c00700 1 -- 172.21.15.12:0/2099926765 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff44419b3e0 con 0x7ff4441045e0 2024-03-23T23:00:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.260+0000 7ff43aa00700 1 -- 172.21.15.12:0/2099926765 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff43400cbb0 con 0x7ff4441045e0 2024-03-23T23:00:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.260+0000 7ff43aa00700 1 -- 172.21.15.12:0/2099926765 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff434018680 con 0x7ff4441045e0 2024-03-23T23:00:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.260+0000 7ff448c00700 1 -- 172.21.15.12:0/2099926765 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4441a7650 con 0x7ff4441045e0 2024-03-23T23:00:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.261+0000 7ff448c00700 1 -- 172.21.15.12:0/2099926765 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff444062920 con 0x7ff4441045e0 2024-03-23T23:00:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.263+0000 7ff43aa00700 1 -- 172.21.15.12:0/2099926765 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7ff434010430 con 0x7ff4441045e0 2024-03-23T23:00:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.264+0000 7ff43aa00700 1 --2- 172.21.15.12:0/2099926765 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff4280a2920 0x7ff4280a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.265+0000 7ff43aa00700 1 -- 172.21.15.12:0/2099926765 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff4340ced80 con 0x7ff4441045e0 2024-03-23T23:00:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.265+0000 7ff441400700 1 --2- 172.21.15.12:0/2099926765 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff4280a2920 0x7ff4280a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.265+0000 7ff43aa00700 1 --2- 172.21.15.12:0/2099926765 >> [v2:172.21.15.12:6826/1175260774,v1:172.21.15.12:6827/1175260774] conn(0x7ff4280ac100 0x7ff4280ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.265+0000 7ff43aa00700 1 -- 172.21.15.12:0/2099926765 --> [v2:172.21.15.12:6826/1175260774,v1:172.21.15.12:6827/1175260774] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff434012ec0 con 0x7ff4280ac100 2024-03-23T23:00:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.265+0000 7ff43aa00700 1 -- 172.21.15.12:0/2099926765 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4340d3030 con 0x7ff4441045e0 2024-03-23T23:00:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.265+0000 7ff442800700 1 --2- 172.21.15.12:0/2099926765 >> [v2:172.21.15.12:6826/1175260774,v1:172.21.15.12:6827/1175260774] conn(0x7ff4280ac100 0x7ff4280ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.265+0000 7ff441400700 1 --2- 172.21.15.12:0/2099926765 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff4280a2920 0x7ff4280a4de0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff430000c00 tx=0x7ff430005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.266+0000 7ff442800700 1 --2- 172.21.15.12:0/2099926765 >> [v2:172.21.15.12:6826/1175260774,v1:172.21.15.12:6827/1175260774] conn(0x7ff4280ac100 0x7ff4280ae520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.267+0000 7ff43aa00700 1 -- 172.21.15.12:0/2099926765 <== osd.9 v2:172.21.15.12:6826/1175260774 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff434012ec0 con 0x7ff4280ac100 2024-03-23T23:00:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.285+0000 7ff448c00700 1 -- 172.21.15.12:0/2099926765 --> [v2:172.21.15.12:6826/1175260774,v1:172.21.15.12:6827/1175260774] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff444062da0 con 0x7ff4280ac100 2024-03-23T23:00:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.285+0000 7ff43aa00700 1 -- 172.21.15.12:0/2099926765 <== osd.9 v2:172.21.15.12:6826/1175260774 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff444062da0 con 0x7ff4280ac100 2024-03-23T23:00:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.286+0000 7ff448c00700 1 -- 172.21.15.12:0/2099926765 >> [v2:172.21.15.12:6826/1175260774,v1:172.21.15.12:6827/1175260774] conn(0x7ff4280ac100 msgr2=0x7ff4280ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.286+0000 7ff448c00700 1 --2- 172.21.15.12:0/2099926765 >> [v2:172.21.15.12:6826/1175260774,v1:172.21.15.12:6827/1175260774] conn(0x7ff4280ac100 0x7ff4280ae520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.286+0000 7ff448c00700 1 -- 172.21.15.12:0/2099926765 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff4280a2920 msgr2=0x7ff4280a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.286+0000 7ff448c00700 1 --2- 172.21.15.12:0/2099926765 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff4280a2920 0x7ff4280a4de0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff430000c00 tx=0x7ff430005fb0 comp rx=0 tx=0).stop 2024-03-23T23:00:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.286+0000 7ff448c00700 1 -- 172.21.15.12:0/2099926765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4441045e0 msgr2=0x7ff4441a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.286+0000 7ff448c00700 1 --2- 172.21.15.12:0/2099926765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4441045e0 0x7ff4441a0780 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff43400e9a0 tx=0x7ff43400ecb0 comp rx=0 tx=0).stop 2024-03-23T23:00:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.286+0000 7ff442800700 1 -- 172.21.15.12:0/2099926765 reap_dead start 2024-03-23T23:00:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.286+0000 7ff448c00700 1 -- 172.21.15.12:0/2099926765 shutdown_connections 2024-03-23T23:00:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.286+0000 7ff448c00700 1 -- 172.21.15.12:0/2099926765 >> 172.21.15.12:0/2099926765 conn(0x7ff4440ffe70 msgr2=0x7ff4441009e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.286+0000 7ff448c00700 1 -- 172.21.15.12:0/2099926765 shutdown_connections 2024-03-23T23:00:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:20.286+0000 7ff448c00700 1 -- 172.21.15.12:0/2099926765 wait complete. 2024-03-23T23:00:20.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:20 smithi173 ceph-mon[36116]: pgmap v344: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:00:20.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:20 smithi203 ceph-mon[36693]: pgmap v344: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:00:21.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.408+0000 7ff5be400700 1 -- 172.21.15.12:0/3225186104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5b8101090 msgr2=0x7ff5b81014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:21.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.408+0000 7ff5be400700 1 --2- 172.21.15.12:0/3225186104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5b8101090 0x7ff5b81014f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff5a4009990 tx=0x7ff5a4009ca0 comp rx=0 tx=0).stop 2024-03-23T23:00:21.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.409+0000 7ff5be400700 1 -- 172.21.15.12:0/3225186104 shutdown_connections 2024-03-23T23:00:21.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.409+0000 7ff5be400700 1 --2- 172.21.15.12:0/3225186104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5b8101a30 0x7ff5b810df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:21.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.409+0000 7ff5be400700 1 --2- 172.21.15.12:0/3225186104 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5b81070d0 0x7ff5b81074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:21.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.409+0000 7ff5be400700 1 --2- 172.21.15.12:0/3225186104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5b8101090 0x7ff5b81014f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:21.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.409+0000 7ff5be400700 1 -- 172.21.15.12:0/3225186104 >> 172.21.15.12:0/3225186104 conn(0x7ff5b8077fc0 msgr2=0x7ff5b80783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:21.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.409+0000 7ff5be400700 1 -- 172.21.15.12:0/3225186104 shutdown_connections 2024-03-23T23:00:21.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.409+0000 7ff5be400700 1 -- 172.21.15.12:0/3225186104 wait complete. 2024-03-23T23:00:21.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.410+0000 7ff5be400700 1 Processor -- start 2024-03-23T23:00:21.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.411+0000 7ff5be400700 1 -- start start 2024-03-23T23:00:21.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.411+0000 7ff5be400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5b8101090 0x7ff5b819e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:21.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.411+0000 7ff5be400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5b8101a30 0x7ff5b819ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:21.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.411+0000 7ff5be400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5b81070d0 0x7ff5b8198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:21.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.411+0000 7ff5be400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b81113a0 con 0x7ff5b8101090 2024-03-23T23:00:21.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.411+0000 7ff5be400700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b8111220 con 0x7ff5b8101a30 2024-03-23T23:00:21.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.411+0000 7ff5be400700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b8111520 con 0x7ff5b81070d0 2024-03-23T23:00:21.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.412+0000 7ff5b6a00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5b8101a30 0x7ff5b819ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:21.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.412+0000 7ff5b7e00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5b81070d0 0x7ff5b8198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:21.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.412+0000 7ff5b6a00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5b8101a30 0x7ff5b819ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:42564/0 (socket says 172.21.15.12:42564) 2024-03-23T23:00:21.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.412+0000 7ff5b6a00700 1 -- 172.21.15.12:0/2019075719 learned_addr learned my addr 172.21.15.12:0/2019075719 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:21.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.412+0000 7ff5b7e00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5b81070d0 0x7ff5b8198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47118/0 (socket says 172.21.15.12:47118) 2024-03-23T23:00:21.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.412+0000 7ff5b7400700 1 --2- 172.21.15.12:0/2019075719 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5b8101090 0x7ff5b819e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:21.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.412+0000 7ff5b6a00700 1 -- 172.21.15.12:0/2019075719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5b81070d0 msgr2=0x7ff5b8198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:21.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.412+0000 7ff5b6a00700 1 --2- 172.21.15.12:0/2019075719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5b81070d0 0x7ff5b8198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:21.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.412+0000 7ff5b6a00700 1 -- 172.21.15.12:0/2019075719 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5b8101090 msgr2=0x7ff5b819e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:21.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.412+0000 7ff5b6a00700 1 --2- 172.21.15.12:0/2019075719 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5b8101090 0x7ff5b819e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:21.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.412+0000 7ff5b6a00700 1 -- 172.21.15.12:0/2019075719 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b8199040 con 0x7ff5b8101a30 2024-03-23T23:00:21.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.412+0000 7ff5b7e00700 1 --2- 172.21.15.12:0/2019075719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5b81070d0 0x7ff5b8198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:00:21.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.412+0000 7ff5b6a00700 1 --2- 172.21.15.12:0/2019075719 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5b8101a30 0x7ff5b819ebd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff5a4000c00 tx=0x7ff5a400bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:21.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.413+0000 7ff5afe00700 1 -- 172.21.15.12:0/2019075719 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a4027070 con 0x7ff5b8101a30 2024-03-23T23:00:21.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.413+0000 7ff5afe00700 1 -- 172.21.15.12:0/2019075719 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff5a40056d0 con 0x7ff5b8101a30 2024-03-23T23:00:21.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.413+0000 7ff5be400700 1 -- 172.21.15.12:0/2019075719 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5b81992d0 con 0x7ff5b8101a30 2024-03-23T23:00:21.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.413+0000 7ff5afe00700 1 -- 172.21.15.12:0/2019075719 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a4022e10 con 0x7ff5b8101a30 2024-03-23T23:00:21.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.413+0000 7ff5be400700 1 -- 172.21.15.12:0/2019075719 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b81a5610 con 0x7ff5b8101a30 2024-03-23T23:00:21.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.414+0000 7ff5be400700 1 -- 172.21.15.12:0/2019075719 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff5b8105190 con 0x7ff5b8101a30 2024-03-23T23:00:21.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.416+0000 7ff5afe00700 1 -- 172.21.15.12:0/2019075719 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7ff5a402a070 con 0x7ff5b8101a30 2024-03-23T23:00:21.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.417+0000 7ff5afe00700 1 --2- 172.21.15.12:0/2019075719 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff59c0a29a0 0x7ff59c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:21.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.417+0000 7ff5afe00700 1 -- 172.21.15.12:0/2019075719 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff5a40cf760 con 0x7ff5b8101a30 2024-03-23T23:00:21.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.417+0000 7ff5b7400700 1 --2- 172.21.15.12:0/2019075719 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff59c0a29a0 0x7ff59c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:21.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.418+0000 7ff5afe00700 1 --2- 172.21.15.12:0/2019075719 >> [v2:172.21.15.203:6824/2323604506,v1:172.21.15.203:6825/2323604506] conn(0x7ff59c0ac180 0x7ff59c0ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:21.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.418+0000 7ff5afe00700 1 -- 172.21.15.12:0/2019075719 --> [v2:172.21.15.203:6824/2323604506,v1:172.21.15.203:6825/2323604506] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff5a4001d30 con 0x7ff59c0ac180 2024-03-23T23:00:21.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.418+0000 7ff5afe00700 1 -- 172.21.15.12:0/2019075719 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5a40d6050 con 0x7ff5b8101a30 2024-03-23T23:00:21.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.418+0000 7ff5b7e00700 1 --2- 172.21.15.12:0/2019075719 >> [v2:172.21.15.203:6824/2323604506,v1:172.21.15.203:6825/2323604506] conn(0x7ff59c0ac180 0x7ff59c0ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:21.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.418+0000 7ff5b7400700 1 --2- 172.21.15.12:0/2019075719 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff59c0a29a0 0x7ff59c0a4e60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8009be0 tx=0x7ff5a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:21.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.418+0000 7ff5b7e00700 1 --2- 172.21.15.12:0/2019075719 >> [v2:172.21.15.203:6824/2323604506,v1:172.21.15.203:6825/2323604506] conn(0x7ff59c0ac180 0x7ff59c0ae5a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:21.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.420+0000 7ff5afe00700 1 -- 172.21.15.12:0/2019075719 <== osd.11 v2:172.21.15.203:6824/2323604506 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff5a4001d30 con 0x7ff59c0ac180 2024-03-23T23:00:21.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.438+0000 7ff5be400700 1 -- 172.21.15.12:0/2019075719 --> [v2:172.21.15.203:6824/2323604506,v1:172.21.15.203:6825/2323604506] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff5b8102780 con 0x7ff59c0ac180 2024-03-23T23:00:21.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.439+0000 7ff5afe00700 1 -- 172.21.15.12:0/2019075719 <== osd.11 v2:172.21.15.203:6824/2323604506 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff5b8102780 con 0x7ff59c0ac180 2024-03-23T23:00:21.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.439+0000 7ff5be400700 1 -- 172.21.15.12:0/2019075719 >> [v2:172.21.15.203:6824/2323604506,v1:172.21.15.203:6825/2323604506] conn(0x7ff59c0ac180 msgr2=0x7ff59c0ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:21.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.439+0000 7ff5be400700 1 --2- 172.21.15.12:0/2019075719 >> [v2:172.21.15.203:6824/2323604506,v1:172.21.15.203:6825/2323604506] conn(0x7ff59c0ac180 0x7ff59c0ae5a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:21.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.439+0000 7ff5be400700 1 -- 172.21.15.12:0/2019075719 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff59c0a29a0 msgr2=0x7ff59c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:21.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.439+0000 7ff5be400700 1 --2- 172.21.15.12:0/2019075719 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff59c0a29a0 0x7ff59c0a4e60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8009be0 tx=0x7ff5a8009260 comp rx=0 tx=0).stop 2024-03-23T23:00:21.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.439+0000 7ff5be400700 1 -- 172.21.15.12:0/2019075719 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5b8101a30 msgr2=0x7ff5b819ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:21.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.439+0000 7ff5be400700 1 --2- 172.21.15.12:0/2019075719 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5b8101a30 0x7ff5b819ebd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff5a4000c00 tx=0x7ff5a400bab0 comp rx=0 tx=0).stop 2024-03-23T23:00:21.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.439+0000 7ff5b7e00700 1 -- 172.21.15.12:0/2019075719 reap_dead start 2024-03-23T23:00:21.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.440+0000 7ff5be400700 1 -- 172.21.15.12:0/2019075719 shutdown_connections 2024-03-23T23:00:21.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.440+0000 7ff5be400700 1 -- 172.21.15.12:0/2019075719 >> 172.21.15.12:0/2019075719 conn(0x7ff5b8077fc0 msgr2=0x7ff5b8100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:21.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.440+0000 7ff5be400700 1 -- 172.21.15.12:0/2019075719 shutdown_connections 2024-03-23T23:00:21.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:21.440+0000 7ff5be400700 1 -- 172.21.15.12:0/2019075719 wait complete. 2024-03-23T23:00:22.517 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:22 smithi012 ceph-mon[31695]: pgmap v345: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-23T23:00:22.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:22 smithi173 ceph-mon[36116]: pgmap v345: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-23T23:00:22.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:22 smithi203 ceph-mon[36693]: pgmap v345: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-23T23:00:23.395 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:00:23.395 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:00:23.495 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:23 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2024-03-23T23:00:23.495 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:23 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-03-23T23:00:23.495 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:23 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]: dispatch 2024-03-23T23:00:23.495 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:23 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2024-03-23T23:00:23.496 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:23 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-23T23:00:23.496 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:23 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-03-23T23:00:23.496 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:23 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2024-03-23T23:00:23.496 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:23 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]: dispatch 2024-03-23T23:00:23.528 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:00:23.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:23 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2024-03-23T23:00:23.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:23 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-03-23T23:00:23.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:23 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]: dispatch 2024-03-23T23:00:23.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:23 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2024-03-23T23:00:23.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:23 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-23T23:00:23.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:23 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-03-23T23:00:23.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:23 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2024-03-23T23:00:23.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:23 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]: dispatch 2024-03-23T23:00:23.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:23 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2024-03-23T23:00:23.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:23 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-03-23T23:00:23.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:23 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]: dispatch 2024-03-23T23:00:23.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:23 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2024-03-23T23:00:23.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:23 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-23T23:00:23.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:23 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-03-23T23:00:23.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:23 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2024-03-23T23:00:23.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:23 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]: dispatch 2024-03-23T23:00:24.636 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:24 smithi012 ceph-mon[31695]: pgmap v346: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-23T23:00:24.636 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2024-03-23T23:00:24.636 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-03-23T23:00:24.636 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]': finished 2024-03-23T23:00:24.636 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2024-03-23T23:00:24.636 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-23T23:00:24.636 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-03-23T23:00:24.636 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2024-03-23T23:00:24.637 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:24 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]': finished 2024-03-23T23:00:24.637 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:24 smithi012 ceph-mon[31695]: osdmap e87: 12 total, 12 up, 12 in 2024-03-23T23:00:24.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:24 smithi203 ceph-mon[36693]: pgmap v346: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-23T23:00:24.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2024-03-23T23:00:24.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-03-23T23:00:24.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]': finished 2024-03-23T23:00:24.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2024-03-23T23:00:24.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-23T23:00:24.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-03-23T23:00:24.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2024-03-23T23:00:24.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:24 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]': finished 2024-03-23T23:00:24.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:24 smithi203 ceph-mon[36693]: osdmap e87: 12 total, 12 up, 12 in 2024-03-23T23:00:24.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:24 smithi173 ceph-mon[36116]: pgmap v346: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-23T23:00:24.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:24 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2024-03-23T23:00:24.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:24 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-03-23T23:00:24.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:24 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]': finished 2024-03-23T23:00:24.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:24 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2024-03-23T23:00:24.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:24 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-23T23:00:24.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:24 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-03-23T23:00:24.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:24 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2024-03-23T23:00:24.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:24 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]': finished 2024-03-23T23:00:24.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:24 smithi173 ceph-mon[36116]: osdmap e87: 12 total, 12 up, 12 in 2024-03-23T23:00:25.101 INFO:teuthology.orchestra.run.smithi012.stdout:347892351001 2024-03-23T23:00:25.102 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-23T23:00:25.463 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:25 smithi012 ceph-mon[31695]: osdmap e88: 12 total, 12 up, 12 in 2024-03-23T23:00:25.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:25 smithi203 ceph-mon[36693]: osdmap e88: 12 total, 12 up, 12 in 2024-03-23T23:00:25.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:25 smithi173 ceph-mon[36116]: osdmap e88: 12 total, 12 up, 12 in 2024-03-23T23:00:26.343 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:00:26.355 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:00:26.358 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:00:26.577 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:26 smithi012 ceph-mon[31695]: pgmap v349: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-23T23:00:26.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:26 smithi203 ceph-mon[36693]: pgmap v349: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-23T23:00:26.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:26 smithi173 ceph-mon[36116]: pgmap v349: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-23T23:00:27.728 INFO:teuthology.orchestra.run.smithi012.stdout:292057776163 2024-03-23T23:00:27.728 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-23T23:00:28.640 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:28 smithi012 ceph-mon[31695]: pgmap v350: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 72 B/s, 1 keys/s, 0 objects/s recovering 2024-03-23T23:00:28.640 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:28 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:00:28.644 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:28 smithi203 ceph-mon[36693]: pgmap v350: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 72 B/s, 1 keys/s, 0 objects/s recovering 2024-03-23T23:00:28.644 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:28 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:00:28.670 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:28 smithi173 ceph-mon[36116]: pgmap v350: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 72 B/s, 1 keys/s, 0 objects/s recovering 2024-03-23T23:00:28.670 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:28 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:00:30.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:30 smithi012 ceph-mon[31695]: pgmap v351: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 72 B/s, 1 keys/s, 0 objects/s recovering 2024-03-23T23:00:30.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:30 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:00:30.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:30 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:00:30.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:30 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:00:30.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:30 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:00:30.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:30 smithi173 ceph-mon[36116]: pgmap v351: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 72 B/s, 1 keys/s, 0 objects/s recovering 2024-03-23T23:00:30.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:30 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:00:30.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:30 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:00:30.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:30 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:00:30.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:30 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:00:30.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:30 smithi203 ceph-mon[36693]: pgmap v351: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 72 B/s, 1 keys/s, 0 objects/s recovering 2024-03-23T23:00:30.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:30 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:00:30.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:30 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:00:30.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:30 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:00:30.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:30 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:00:31.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.159+0000 7f7e15c00700 1 -- 172.21.15.12:0/680441117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e10104da0 msgr2=0x7f7e101112b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:31.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.159+0000 7f7e15c00700 1 --2- 172.21.15.12:0/680441117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e10104da0 0x7f7e101112b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7e0800b0a0 tx=0x7f7e0800b3b0 comp rx=0 tx=0).stop 2024-03-23T23:00:31.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.159+0000 7f7e15c00700 1 -- 172.21.15.12:0/680441117 shutdown_connections 2024-03-23T23:00:31.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.159+0000 7f7e15c00700 1 --2- 172.21.15.12:0/680441117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e10104da0 0x7f7e101112b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:31.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.159+0000 7f7e15c00700 1 --2- 172.21.15.12:0/680441117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e1010a470 0x7f7e1010a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:31.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.159+0000 7f7e15c00700 1 --2- 172.21.15.12:0/680441117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e101043e0 0x7f7e10104860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:31.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.159+0000 7f7e15c00700 1 -- 172.21.15.12:0/680441117 >> 172.21.15.12:0/680441117 conn(0x7f7e100fff30 msgr2=0x7f7e10102350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:31.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.159+0000 7f7e15c00700 1 -- 172.21.15.12:0/680441117 shutdown_connections 2024-03-23T23:00:31.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.159+0000 7f7e15c00700 1 -- 172.21.15.12:0/680441117 wait complete. 2024-03-23T23:00:31.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.161+0000 7f7e15c00700 1 Processor -- start 2024-03-23T23:00:31.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.161+0000 7f7e15c00700 1 -- start start 2024-03-23T23:00:31.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.161+0000 7f7e15c00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e101043e0 0x7f7e101a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:31.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.161+0000 7f7e15c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e10104da0 0x7f7e101a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:31.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.161+0000 7f7e15c00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e1010a470 0x7f7e1019a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:31.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.161+0000 7f7e15c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e10114750 con 0x7f7e10104da0 2024-03-23T23:00:31.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.161+0000 7f7e15c00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e101145d0 con 0x7f7e101043e0 2024-03-23T23:00:31.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.161+0000 7f7e15c00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e101148d0 con 0x7f7e1010a470 2024-03-23T23:00:31.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.162+0000 7f7e0e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e10104da0 0x7f7e101a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:31.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.162+0000 7f7e0ea00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e101043e0 0x7f7e101a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:31.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.162+0000 7f7e0ea00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e101043e0 0x7f7e101a07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:56736/0 (socket says 172.21.15.12:56736) 2024-03-23T23:00:31.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.162+0000 7f7e0e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e10104da0 0x7f7e101a0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54088/0 (socket says 172.21.15.12:54088) 2024-03-23T23:00:31.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.162+0000 7f7e0ea00700 1 -- 172.21.15.12:0/1958293281 learned_addr learned my addr 172.21.15.12:0/1958293281 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:31.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.162+0000 7f7e0f400700 1 --2- 172.21.15.12:0/1958293281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e1010a470 0x7f7e1019a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:31.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.162+0000 7f7e0ea00700 1 -- 172.21.15.12:0/1958293281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e1010a470 msgr2=0x7f7e1019a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:31.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.162+0000 7f7e0ea00700 1 --2- 172.21.15.12:0/1958293281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e1010a470 0x7f7e1019a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:31.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.162+0000 7f7e0ea00700 1 -- 172.21.15.12:0/1958293281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e10104da0 msgr2=0x7f7e101a0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:31.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.162+0000 7f7e0ea00700 1 --2- 172.21.15.12:0/1958293281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e10104da0 0x7f7e101a0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:31.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.162+0000 7f7e0ea00700 1 -- 172.21.15.12:0/1958293281 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1019b110 con 0x7f7e101043e0 2024-03-23T23:00:31.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.162+0000 7f7e0e000700 1 --2- 172.21.15.12:0/1958293281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e10104da0 0x7f7e101a0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:00:31.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.162+0000 7f7e0f400700 1 --2- 172.21.15.12:0/1958293281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e1010a470 0x7f7e1019a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:00:31.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.162+0000 7f7e0ea00700 1 --2- 172.21.15.12:0/1958293281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e101043e0 0x7f7e101a07f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7e0000e9a0 tx=0x7f7e0000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:31.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.163+0000 7f7e07400700 1 -- 172.21.15.12:0/1958293281 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0000ca10 con 0x7f7e101043e0 2024-03-23T23:00:31.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.163+0000 7f7e07400700 1 -- 172.21.15.12:0/1958293281 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7e0000cbb0 con 0x7f7e101043e0 2024-03-23T23:00:31.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.163+0000 7f7e15c00700 1 -- 172.21.15.12:0/1958293281 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e1019b400 con 0x7f7e101043e0 2024-03-23T23:00:31.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.163+0000 7f7e07400700 1 -- 172.21.15.12:0/1958293281 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e00018680 con 0x7f7e101043e0 2024-03-23T23:00:31.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.163+0000 7f7e15c00700 1 -- 172.21.15.12:0/1958293281 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e101a76c0 con 0x7f7e101043e0 2024-03-23T23:00:31.643 INFO:teuthology.orchestra.run.smithi012.stdout:146028888130 2024-03-23T23:00:31.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.164+0000 7f7e15c00700 1 -- 172.21.15.12:0/1958293281 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e10068e50 con 0x7f7e101043e0 2024-03-23T23:00:31.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.166+0000 7f7e07400700 1 -- 172.21.15.12:0/1958293281 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f7e00010430 con 0x7f7e101043e0 2024-03-23T23:00:31.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.167+0000 7f7e07400700 1 --2- 172.21.15.12:0/1958293281 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7df40a29a0 0x7f7df40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:31.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.167+0000 7f7e07400700 1 -- 172.21.15.12:0/1958293281 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7e000ced90 con 0x7f7e101043e0 2024-03-23T23:00:31.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.167+0000 7f7e0e000700 1 --2- 172.21.15.12:0/1958293281 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7df40a29a0 0x7f7df40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:31.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.168+0000 7f7e0e000700 1 --2- 172.21.15.12:0/1958293281 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7df40a29a0 0x7f7df40a4e60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7e101099a0 tx=0x7f7dfc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:31.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.175+0000 7f7e07400700 1 -- 172.21.15.12:0/1958293281 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7e0009bc80 con 0x7f7e101043e0 2024-03-23T23:00:31.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.425+0000 7f7e15c00700 1 -- 172.21.15.12:0/1958293281 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f7e1002da30 con 0x7f7e101043e0 2024-03-23T23:00:31.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.426+0000 7f7e07400700 1 -- 172.21.15.12:0/1958293281 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f7e0009b580 con 0x7f7e101043e0 2024-03-23T23:00:31.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.429+0000 7f7e15c00700 1 -- 172.21.15.12:0/1958293281 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7df40a29a0 msgr2=0x7f7df40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:31.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.429+0000 7f7e15c00700 1 --2- 172.21.15.12:0/1958293281 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7df40a29a0 0x7f7df40a4e60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7e101099a0 tx=0x7f7dfc009260 comp rx=0 tx=0).stop 2024-03-23T23:00:31.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.429+0000 7f7e15c00700 1 -- 172.21.15.12:0/1958293281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e101043e0 msgr2=0x7f7e101a07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:31.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.429+0000 7f7e15c00700 1 --2- 172.21.15.12:0/1958293281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e101043e0 0x7f7e101a07f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7e0000e9a0 tx=0x7f7e0000ecb0 comp rx=0 tx=0).stop 2024-03-23T23:00:31.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.429+0000 7f7e15c00700 1 -- 172.21.15.12:0/1958293281 shutdown_connections 2024-03-23T23:00:31.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.429+0000 7f7e15c00700 1 --2- 172.21.15.12:0/1958293281 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7df40a29a0 0x7f7df40a4e60 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:31.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.429+0000 7f7e15c00700 1 --2- 172.21.15.12:0/1958293281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e1010a470 0x7f7e1019a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:31.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.429+0000 7f7e15c00700 1 --2- 172.21.15.12:0/1958293281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e101043e0 0x7f7e101a07f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:31.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.429+0000 7f7e15c00700 1 --2- 172.21.15.12:0/1958293281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e10104da0 0x7f7e101a0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:31.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.429+0000 7f7e15c00700 1 -- 172.21.15.12:0/1958293281 >> 172.21.15.12:0/1958293281 conn(0x7f7e100fff30 msgr2=0x7f7e10100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:31.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.429+0000 7f7e15c00700 1 -- 172.21.15.12:0/1958293281 shutdown_connections 2024-03-23T23:00:31.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.430+0000 7f7e15c00700 1 -- 172.21.15.12:0/1958293281 wait complete. 2024-03-23T23:00:32.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:32 smithi012 ceph-mon[31695]: pgmap v352: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 72 B/s, 1 keys/s, 0 objects/s recovering 2024-03-23T23:00:32.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:32 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1958293281' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-23T23:00:32.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:32 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2653197486' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-23T23:00:32.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:32 smithi173 ceph-mon[36116]: pgmap v352: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 72 B/s, 1 keys/s, 0 objects/s recovering 2024-03-23T23:00:32.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:32 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1958293281' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-23T23:00:32.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:32 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2653197486' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-23T23:00:32.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:32 smithi203 ceph-mon[36693]: pgmap v352: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 72 B/s, 1 keys/s, 0 objects/s recovering 2024-03-23T23:00:32.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:32 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1958293281' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-23T23:00:32.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:32 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2653197486' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-23T23:00:33.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.042+0000 7f777e200700 1 -- 172.21.15.12:0/2475952387 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777810e7a0 msgr2=0x7f777810eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.042+0000 7f777e200700 1 --2- 172.21.15.12:0/2475952387 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777810e7a0 0x7f777810eb80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7768009990 tx=0x7f7768009ca0 comp rx=0 tx=0).stop 2024-03-23T23:00:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.042+0000 7f777e200700 1 -- 172.21.15.12:0/2475952387 shutdown_connections 2024-03-23T23:00:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.042+0000 7f777e200700 1 --2- 172.21.15.12:0/2475952387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7778109160 0x7f7778115670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.042+0000 7f777e200700 1 --2- 172.21.15.12:0/2475952387 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777810e7a0 0x7f777810eb80 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.042+0000 7f777e200700 1 --2- 172.21.15.12:0/2475952387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77781087a0 0x7f7778108c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.042+0000 7f777e200700 1 -- 172.21.15.12:0/2475952387 >> 172.21.15.12:0/2475952387 conn(0x7f7778075960 msgr2=0x7f7778077d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.042+0000 7f777e200700 1 -- 172.21.15.12:0/2475952387 shutdown_connections 2024-03-23T23:00:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.042+0000 7f777e200700 1 -- 172.21.15.12:0/2475952387 wait complete. 2024-03-23T23:00:33.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.043+0000 7f777e200700 1 Processor -- start 2024-03-23T23:00:33.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.043+0000 7f777e200700 1 -- start start 2024-03-23T23:00:33.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.044+0000 7f777e200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77781087a0 0x7f777819ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:33.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.044+0000 7f777e200700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7778109160 0x7f777819f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:33.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.044+0000 7f777e200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777810e7a0 0x7f77781a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:33.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.044+0000 7f777e200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7778118b40 con 0x7f777810e7a0 2024-03-23T23:00:33.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.044+0000 7f777e200700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77781189c0 con 0x7f7778109160 2024-03-23T23:00:33.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.044+0000 7f777e200700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7778118cc0 con 0x7f77781087a0 2024-03-23T23:00:33.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.044+0000 7f7777400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77781087a0 0x7f777819ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:33.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.044+0000 7f7777400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77781087a0 0x7f777819ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:56140/0 (socket says 172.21.15.12:56140) 2024-03-23T23:00:33.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.044+0000 7f7777400700 1 -- 172.21.15.12:0/1208267452 learned_addr learned my addr 172.21.15.12:0/1208267452 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:33.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.044+0000 7f7776a00700 1 --2- 172.21.15.12:0/1208267452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7778109160 0x7f777819f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:33.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.045+0000 7f7777e00700 1 --2- 172.21.15.12:0/1208267452 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777810e7a0 0x7f77781a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:33.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.045+0000 7f7777400700 1 -- 172.21.15.12:0/1208267452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7778109160 msgr2=0x7f777819f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:33.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.045+0000 7f7777400700 1 --2- 172.21.15.12:0/1208267452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7778109160 0x7f777819f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:33.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.045+0000 7f7777400700 1 -- 172.21.15.12:0/1208267452 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777810e7a0 msgr2=0x7f77781a3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:33.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.045+0000 7f7777400700 1 --2- 172.21.15.12:0/1208267452 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777810e7a0 0x7f77781a3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:33.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.045+0000 7f7777400700 1 -- 172.21.15.12:0/1208267452 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77781a3f40 con 0x7f77781087a0 2024-03-23T23:00:33.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.045+0000 7f7776a00700 1 --2- 172.21.15.12:0/1208267452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7778109160 0x7f777819f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:00:33.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.045+0000 7f7777400700 1 --2- 172.21.15.12:0/1208267452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77781087a0 0x7f777819ef30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7768000c00 tx=0x7f776800bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:33.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.046+0000 7f776fe00700 1 -- 172.21.15.12:0/1208267452 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7768027070 con 0x7f77781087a0 2024-03-23T23:00:33.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.046+0000 7f776fe00700 1 -- 172.21.15.12:0/1208267452 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77680056d0 con 0x7f77781087a0 2024-03-23T23:00:33.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.046+0000 7f776fe00700 1 -- 172.21.15.12:0/1208267452 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7768022e10 con 0x7f77781087a0 2024-03-23T23:00:33.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.046+0000 7f777e200700 1 -- 172.21.15.12:0/1208267452 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77781a41d0 con 0x7f77781087a0 2024-03-23T23:00:33.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.046+0000 7f777e200700 1 -- 172.21.15.12:0/1208267452 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77781abab0 con 0x7f77781087a0 2024-03-23T23:00:33.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.047+0000 7f777e200700 1 -- 172.21.15.12:0/1208267452 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7778068e50 con 0x7f77781087a0 2024-03-23T23:00:33.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.049+0000 7f776fe00700 1 -- 172.21.15.12:0/1208267452 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f776802a070 con 0x7f77781087a0 2024-03-23T23:00:33.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.050+0000 7f776fe00700 1 --2- 172.21.15.12:0/1208267452 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f775c0a29a0 0x7f775c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:33.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.050+0000 7f776fe00700 1 -- 172.21.15.12:0/1208267452 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f77680cfbc0 con 0x7f77781087a0 2024-03-23T23:00:33.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.050+0000 7f7776a00700 1 --2- 172.21.15.12:0/1208267452 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f775c0a29a0 0x7f775c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:33.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.051+0000 7f7776a00700 1 --2- 172.21.15.12:0/1208267452 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f775c0a29a0 0x7f775c0a4e60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7764009680 tx=0x7f7764009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:33.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.053+0000 7f776fe00700 1 -- 172.21.15.12:0/1208267452 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7768013070 con 0x7f77781087a0 2024-03-23T23:00:33.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.707+0000 7fc718a00700 1 -- 172.21.15.12:0/2480283384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc714074ce0 msgr2=0x7fc71410e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:33.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.707+0000 7fc718a00700 1 --2- 172.21.15.12:0/2480283384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc714074ce0 0x7fc71410e680 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc700009990 tx=0x7fc700009ca0 comp rx=0 tx=0).stop 2024-03-23T23:00:33.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.707+0000 7fc718a00700 1 -- 172.21.15.12:0/2480283384 shutdown_connections 2024-03-23T23:00:33.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.707+0000 7fc718a00700 1 --2- 172.21.15.12:0/2480283384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc71410ebc0 0x7fc714110fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:33.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.707+0000 7fc718a00700 1 --2- 172.21.15.12:0/2480283384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7140743c0 0x7fc7140747a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:33.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.707+0000 7fc718a00700 1 --2- 172.21.15.12:0/2480283384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc714074ce0 0x7fc71410e680 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:33.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.707+0000 7fc718a00700 1 -- 172.21.15.12:0/2480283384 >> 172.21.15.12:0/2480283384 conn(0x7fc7140792e0 msgr2=0x7fc7140796f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:33.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.707+0000 7fc718a00700 1 -- 172.21.15.12:0/2480283384 shutdown_connections 2024-03-23T23:00:33.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.707+0000 7fc718a00700 1 -- 172.21.15.12:0/2480283384 wait complete. 2024-03-23T23:00:33.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.709+0000 7fc718a00700 1 Processor -- start 2024-03-23T23:00:33.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.709+0000 7fc718a00700 1 -- start start 2024-03-23T23:00:33.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.709+0000 7fc718a00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7140743c0 0x7fc7141a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:33.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.709+0000 7fc718a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc714074ce0 0x7fc7141a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:33.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.709+0000 7fc718a00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc71410ebc0 0x7fc71419ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:33.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.710+0000 7fc718a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc714114780 con 0x7fc714074ce0 2024-03-23T23:00:33.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.710+0000 7fc718a00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc714114600 con 0x7fc7140743c0 2024-03-23T23:00:33.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.710+0000 7fc718a00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc714114900 con 0x7fc71410ebc0 2024-03-23T23:00:33.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.710+0000 7fc711e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7140743c0 0x7fc7141a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:33.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.710+0000 7fc712800700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc71410ebc0 0x7fc71419ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:33.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.710+0000 7fc712800700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc71410ebc0 0x7fc71419ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:56134/0 (socket says 172.21.15.12:56134) 2024-03-23T23:00:33.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.710+0000 7fc712800700 1 -- 172.21.15.12:0/2653197486 learned_addr learned my addr 172.21.15.12:0/2653197486 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:33.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.710+0000 7fc711400700 1 --2- 172.21.15.12:0/2653197486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc714074ce0 0x7fc7141a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:33.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.710+0000 7fc711e00700 1 -- 172.21.15.12:0/2653197486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc71410ebc0 msgr2=0x7fc71419ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:33.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.710+0000 7fc711e00700 1 --2- 172.21.15.12:0/2653197486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc71410ebc0 0x7fc71419ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:33.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.710+0000 7fc711e00700 1 -- 172.21.15.12:0/2653197486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc714074ce0 msgr2=0x7fc7141a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:33.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.710+0000 7fc711e00700 1 --2- 172.21.15.12:0/2653197486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc714074ce0 0x7fc7141a0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:33.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.710+0000 7fc711e00700 1 -- 172.21.15.12:0/2653197486 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71419b280 con 0x7fc7140743c0 2024-03-23T23:00:33.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.710+0000 7fc712800700 1 --2- 172.21.15.12:0/2653197486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc71410ebc0 0x7fc71419ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:00:33.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.711+0000 7fc711e00700 1 --2- 172.21.15.12:0/2653197486 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7140743c0 0x7fc7141a0910 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc70400e9f0 tx=0x7fc70400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:33.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.711+0000 7fc70aa00700 1 -- 172.21.15.12:0/2653197486 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70400cb70 con 0x7fc7140743c0 2024-03-23T23:00:33.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.711+0000 7fc70aa00700 1 -- 172.21.15.12:0/2653197486 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc704004500 con 0x7fc7140743c0 2024-03-23T23:00:33.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.711+0000 7fc70aa00700 1 -- 172.21.15.12:0/2653197486 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc704010430 con 0x7fc7140743c0 2024-03-23T23:00:33.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.711+0000 7fc718a00700 1 -- 172.21.15.12:0/2653197486 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc71419b570 con 0x7fc7140743c0 2024-03-23T23:00:33.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.711+0000 7fc718a00700 1 -- 172.21.15.12:0/2653197486 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7141a77e0 con 0x7fc7140743c0 2024-03-23T23:00:33.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.712+0000 7fc718a00700 1 -- 172.21.15.12:0/2653197486 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc714068e50 con 0x7fc7140743c0 2024-03-23T23:00:33.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.714+0000 7fc70aa00700 1 -- 172.21.15.12:0/2653197486 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fc704003eb0 con 0x7fc7140743c0 2024-03-23T23:00:33.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.715+0000 7fc70aa00700 1 --2- 172.21.15.12:0/2653197486 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc6f80a29a0 0x7fc6f80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:33.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.715+0000 7fc70aa00700 1 -- 172.21.15.12:0/2653197486 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fc7040cedb0 con 0x7fc7140743c0 2024-03-23T23:00:33.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.715+0000 7fc711400700 1 --2- 172.21.15.12:0/2653197486 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc6f80a29a0 0x7fc6f80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:33.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.716+0000 7fc711400700 1 --2- 172.21.15.12:0/2653197486 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc6f80a29a0 0x7fc6f80a4e60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc70000b2e0 tx=0x7fc70001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:33.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.719+0000 7fc70aa00700 1 -- 172.21.15.12:0/2653197486 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc7040d3070 con 0x7fc7140743c0 2024-03-23T23:00:33.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.952+0000 7fc718a00700 1 -- 172.21.15.12:0/2653197486 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fc71419bff0 con 0x7fc7140743c0 2024-03-23T23:00:33.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.953+0000 7fc70aa00700 1 -- 172.21.15.12:0/2653197486 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fc71419bff0 con 0x7fc7140743c0 2024-03-23T23:00:33.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.956+0000 7fc718a00700 1 -- 172.21.15.12:0/2653197486 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc6f80a29a0 msgr2=0x7fc6f80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:33.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.956+0000 7fc718a00700 1 --2- 172.21.15.12:0/2653197486 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc6f80a29a0 0x7fc6f80a4e60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc70000b2e0 tx=0x7fc70001a040 comp rx=0 tx=0).stop 2024-03-23T23:00:33.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.956+0000 7fc718a00700 1 -- 172.21.15.12:0/2653197486 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7140743c0 msgr2=0x7fc7141a0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:33.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.956+0000 7fc718a00700 1 --2- 172.21.15.12:0/2653197486 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7140743c0 0x7fc7141a0910 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc70400e9f0 tx=0x7fc70400ed00 comp rx=0 tx=0).stop 2024-03-23T23:00:33.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.957+0000 7fc718a00700 1 -- 172.21.15.12:0/2653197486 shutdown_connections 2024-03-23T23:00:33.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.957+0000 7fc718a00700 1 --2- 172.21.15.12:0/2653197486 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc6f80a29a0 0x7fc6f80a4e60 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:33.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.957+0000 7fc718a00700 1 --2- 172.21.15.12:0/2653197486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc71410ebc0 0x7fc71419ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:33.081 INFO:teuthology.orchestra.run.smithi012.stdout:201863462967 2024-03-23T23:00:33.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.957+0000 7fc718a00700 1 --2- 172.21.15.12:0/2653197486 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7140743c0 0x7fc7141a0910 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:33.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.957+0000 7fc718a00700 1 --2- 172.21.15.12:0/2653197486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc714074ce0 0x7fc7141a0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:33.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.957+0000 7fc718a00700 1 -- 172.21.15.12:0/2653197486 >> 172.21.15.12:0/2653197486 conn(0x7fc7140792e0 msgr2=0x7fc714100640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:33.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.957+0000 7fc718a00700 1 -- 172.21.15.12:0/2653197486 shutdown_connections 2024-03-23T23:00:33.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:31.957+0000 7fc718a00700 1 -- 172.21.15.12:0/2653197486 wait complete. 2024-03-23T23:00:33.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.285+0000 7f777e200700 1 -- 172.21.15.12:0/1208267452 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f777802da30 con 0x7f77781087a0 2024-03-23T23:00:33.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.289+0000 7f776fe00700 1 -- 172.21.15.12:0/1208267452 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f776809cb40 con 0x7f77781087a0 2024-03-23T23:00:33.288 INFO:teuthology.orchestra.run.smithi012.stdout:313532612639 2024-03-23T23:00:33.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.293+0000 7f777e200700 1 -- 172.21.15.12:0/1208267452 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f775c0a29a0 msgr2=0x7f775c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:33.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.293+0000 7f777e200700 1 --2- 172.21.15.12:0/1208267452 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f775c0a29a0 0x7f775c0a4e60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7764009680 tx=0x7f7764009260 comp rx=0 tx=0).stop 2024-03-23T23:00:33.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.293+0000 7f777e200700 1 -- 172.21.15.12:0/1208267452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77781087a0 msgr2=0x7f777819ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:33.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.293+0000 7f777e200700 1 --2- 172.21.15.12:0/1208267452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77781087a0 0x7f777819ef30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7768000c00 tx=0x7f776800bab0 comp rx=0 tx=0).stop 2024-03-23T23:00:33.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.294+0000 7f777e200700 1 -- 172.21.15.12:0/1208267452 shutdown_connections 2024-03-23T23:00:33.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.294+0000 7f777e200700 1 --2- 172.21.15.12:0/1208267452 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f775c0a29a0 0x7f775c0a4e60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:33.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.294+0000 7f777e200700 1 --2- 172.21.15.12:0/1208267452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77781087a0 0x7f777819ef30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:33.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.294+0000 7f777e200700 1 --2- 172.21.15.12:0/1208267452 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777810e7a0 0x7f77781a3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:33.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.294+0000 7f777e200700 1 --2- 172.21.15.12:0/1208267452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7778109160 0x7f777819f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:33.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.294+0000 7f777e200700 1 -- 172.21.15.12:0/1208267452 >> 172.21.15.12:0/1208267452 conn(0x7f7778075960 msgr2=0x7f7778075e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:33.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.294+0000 7f777e200700 1 -- 172.21.15.12:0/1208267452 shutdown_connections 2024-03-23T23:00:33.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:33.294+0000 7f777e200700 1 -- 172.21.15.12:0/1208267452 wait complete. 2024-03-23T23:00:33.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:33 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1208267452' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-23T23:00:33.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:33 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1208267452' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-23T23:00:33.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:33 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1208267452' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-23T23:00:34.529 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462961 got 201863462967 for osd.6 2024-03-23T23:00:34.529 DEBUG:teuthology.parallel:result is None 2024-03-23T23:00:34.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:34 smithi173 ceph-mon[36116]: pgmap v353: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 58 B/s, 1 keys/s, 0 objects/s recovering 2024-03-23T23:00:34.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:34 smithi203 ceph-mon[36693]: pgmap v353: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 58 B/s, 1 keys/s, 0 objects/s recovering 2024-03-23T23:00:34.728 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:34 smithi012 ceph-mon[31695]: pgmap v353: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 58 B/s, 1 keys/s, 0 objects/s recovering 2024-03-23T23:00:35.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.097+0000 7f3015200700 1 -- 172.21.15.12:0/2616036903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f301006aa90 msgr2=0x7f301006ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:35.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.097+0000 7f3015200700 1 --2- 172.21.15.12:0/2616036903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f301006aa90 0x7f301006ae70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f30000099c0 tx=0x7f3000009cd0 comp rx=0 tx=0).stop 2024-03-23T23:00:35.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.097+0000 7f3015200700 1 -- 172.21.15.12:0/2616036903 shutdown_connections 2024-03-23T23:00:35.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.097+0000 7f3015200700 1 --2- 172.21.15.12:0/2616036903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f301006aa90 0x7f301006ae70 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:35.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.097+0000 7f3015200700 1 --2- 172.21.15.12:0/2616036903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3010107500 0x7f301010f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:35.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.097+0000 7f3015200700 1 --2- 172.21.15.12:0/2616036903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f301006b440 0x7f3010106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.097+0000 7f3015200700 1 -- 172.21.15.12:0/2616036903 >> 172.21.15.12:0/2616036903 conn(0x7f3010078000 msgr2=0x7f3010078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.097+0000 7f3015200700 1 -- 172.21.15.12:0/2616036903 shutdown_connections 2024-03-23T23:00:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.097+0000 7f3015200700 1 -- 172.21.15.12:0/2616036903 wait complete. 2024-03-23T23:00:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.099+0000 7f3015200700 1 Processor -- start 2024-03-23T23:00:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.099+0000 7f3015200700 1 -- start start 2024-03-23T23:00:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.099+0000 7f3015200700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f301006b440 0x7f301019ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.099+0000 7f3015200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3010107500 0x7f301019b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.099+0000 7f3015200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301019f6d0 0x7f301019fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.099+0000 7f3015200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30101124e0 con 0x7f301019f6d0 2024-03-23T23:00:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.099+0000 7f3015200700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3010112360 con 0x7f301006b440 2024-03-23T23:00:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.099+0000 7f3015200700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3010112660 con 0x7f3010107500 2024-03-23T23:00:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.100+0000 7f300de00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f301006b440 0x7f301019ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.100+0000 7f300de00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f301006b440 0x7f301019ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:40838/0 (socket says 172.21.15.12:40838) 2024-03-23T23:00:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.100+0000 7f300de00700 1 -- 172.21.15.12:0/830941699 learned_addr learned my addr 172.21.15.12:0/830941699 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.100+0000 7f300d400700 1 --2- 172.21.15.12:0/830941699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3010107500 0x7f301019b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.100+0000 7f300e800700 1 --2- 172.21.15.12:0/830941699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301019f6d0 0x7f301019fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:35.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.100+0000 7f300de00700 1 -- 172.21.15.12:0/830941699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3010107500 msgr2=0x7f301019b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:35.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.100+0000 7f300de00700 1 --2- 172.21.15.12:0/830941699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3010107500 0x7f301019b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:35.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.100+0000 7f300de00700 1 -- 172.21.15.12:0/830941699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301019f6d0 msgr2=0x7f301019fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:35.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.100+0000 7f300de00700 1 --2- 172.21.15.12:0/830941699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301019f6d0 0x7f301019fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:35.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.100+0000 7f300de00700 1 -- 172.21.15.12:0/830941699 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30101a01f0 con 0x7f301006b440 2024-03-23T23:00:35.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.100+0000 7f300de00700 1 --2- 172.21.15.12:0/830941699 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f301006b440 0x7f301019ae50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f30000098e0 tx=0x7f3000003a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:35.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.101+0000 7f3006a00700 1 -- 172.21.15.12:0/830941699 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3000003680 con 0x7f301006b440 2024-03-23T23:00:35.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.101+0000 7f3006a00700 1 -- 172.21.15.12:0/830941699 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30000055a0 con 0x7f301006b440 2024-03-23T23:00:35.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.101+0000 7f3015200700 1 -- 172.21.15.12:0/830941699 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30101a0480 con 0x7f301006b440 2024-03-23T23:00:35.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.101+0000 7f3006a00700 1 -- 172.21.15.12:0/830941699 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300000fce0 con 0x7f301006b440 2024-03-23T23:00:35.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.101+0000 7f3015200700 1 -- 172.21.15.12:0/830941699 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30101a7ba0 con 0x7f301006b440 2024-03-23T23:00:35.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.102+0000 7f3015200700 1 -- 172.21.15.12:0/830941699 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30101038b0 con 0x7f301006b440 2024-03-23T23:00:35.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.104+0000 7f3006a00700 1 -- 172.21.15.12:0/830941699 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f300000f460 con 0x7f301006b440 2024-03-23T23:00:35.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.105+0000 7f3006a00700 1 --2- 172.21.15.12:0/830941699 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ff40a2b40 0x7f2ff40a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:35.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.105+0000 7f3006a00700 1 -- 172.21.15.12:0/830941699 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f30000cf9e0 con 0x7f301006b440 2024-03-23T23:00:35.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.106+0000 7f300d400700 1 --2- 172.21.15.12:0/830941699 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ff40a2b40 0x7f2ff40a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:35.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.106+0000 7f300d400700 1 --2- 172.21.15.12:0/830941699 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ff40a2b40 0x7f2ff40a5000 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f301010ef60 tx=0x7f2ffc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:35.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.114+0000 7f3006a00700 1 -- 172.21.15.12:0/830941699 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3000013070 con 0x7f301006b440 2024-03-23T23:00:35.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.343+0000 7f3015200700 1 -- 172.21.15.12:0/830941699 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f30101a7f30 con 0x7f301006b440 2024-03-23T23:00:35.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.343+0000 7f3006a00700 1 -- 172.21.15.12:0/830941699 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f300009c960 con 0x7f301006b440 2024-03-23T23:00:35.343 INFO:teuthology.orchestra.run.smithi012.stdout:111669149769 2024-03-23T23:00:35.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.346+0000 7f3015200700 1 -- 172.21.15.12:0/830941699 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ff40a2b40 msgr2=0x7f2ff40a5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:35.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.346+0000 7f3015200700 1 --2- 172.21.15.12:0/830941699 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ff40a2b40 0x7f2ff40a5000 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f301010ef60 tx=0x7f2ffc009260 comp rx=0 tx=0).stop 2024-03-23T23:00:35.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.346+0000 7f3015200700 1 -- 172.21.15.12:0/830941699 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f301006b440 msgr2=0x7f301019ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:35.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.346+0000 7f3015200700 1 --2- 172.21.15.12:0/830941699 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f301006b440 0x7f301019ae50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f30000098e0 tx=0x7f3000003a90 comp rx=0 tx=0).stop 2024-03-23T23:00:35.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.346+0000 7f3015200700 1 -- 172.21.15.12:0/830941699 shutdown_connections 2024-03-23T23:00:35.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.346+0000 7f3015200700 1 --2- 172.21.15.12:0/830941699 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ff40a2b40 0x7f2ff40a5000 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:35.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.346+0000 7f3015200700 1 --2- 172.21.15.12:0/830941699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301019f6d0 0x7f301019fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:35.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.346+0000 7f3015200700 1 --2- 172.21.15.12:0/830941699 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f301006b440 0x7f301019ae50 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:35.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.346+0000 7f3015200700 1 --2- 172.21.15.12:0/830941699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3010107500 0x7f301019b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:35.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.346+0000 7f3015200700 1 -- 172.21.15.12:0/830941699 >> 172.21.15.12:0/830941699 conn(0x7f3010078000 msgr2=0x7f301010d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:35.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.347+0000 7f3015200700 1 -- 172.21.15.12:0/830941699 shutdown_connections 2024-03-23T23:00:35.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:35.347+0000 7f3015200700 1 -- 172.21.15.12:0/830941699 wait complete. 2024-03-23T23:00:36.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:36 smithi012 ceph-mon[31695]: pgmap v354: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 53 B/s, 0 keys/s, 0 objects/s recovering 2024-03-23T23:00:36.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:36 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/830941699' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-23T23:00:36.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:36 smithi173 ceph-mon[36116]: pgmap v354: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 53 B/s, 0 keys/s, 0 objects/s recovering 2024-03-23T23:00:36.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:36 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/830941699' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-23T23:00:36.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:36 smithi203 ceph-mon[36693]: pgmap v354: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 53 B/s, 0 keys/s, 0 objects/s recovering 2024-03-23T23:00:36.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:36 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/830941699' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-23T23:00:36.787 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612635 got 313532612639 for osd.10 2024-03-23T23:00:36.787 DEBUG:teuthology.parallel:result is None 2024-03-23T23:00:37.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.007+0000 7fb0ee800700 1 -- 172.21.15.12:0/3548175173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0e810a470 msgr2=0x7fb0e810a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:37.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.007+0000 7fb0ee800700 1 --2- 172.21.15.12:0/3548175173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0e810a470 0x7fb0e810a850 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009990 tx=0x7fb0d8009ca0 comp rx=0 tx=0).stop 2024-03-23T23:00:37.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.007+0000 7fb0ee800700 1 -- 172.21.15.12:0/3548175173 shutdown_connections 2024-03-23T23:00:37.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.007+0000 7fb0ee800700 1 --2- 172.21.15.12:0/3548175173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0e8104da0 0x7fb0e81112b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:37.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.007+0000 7fb0ee800700 1 --2- 172.21.15.12:0/3548175173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0e810a470 0x7fb0e810a850 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:37.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.007+0000 7fb0ee800700 1 --2- 172.21.15.12:0/3548175173 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0e81043e0 0x7fb0e8104860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:37.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.007+0000 7fb0ee800700 1 -- 172.21.15.12:0/3548175173 >> 172.21.15.12:0/3548175173 conn(0x7fb0e80fff30 msgr2=0x7fb0e8102350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:37.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.007+0000 7fb0ee800700 1 -- 172.21.15.12:0/3548175173 shutdown_connections 2024-03-23T23:00:37.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.007+0000 7fb0ee800700 1 -- 172.21.15.12:0/3548175173 wait complete. 2024-03-23T23:00:37.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.008+0000 7fb0ee800700 1 Processor -- start 2024-03-23T23:00:37.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.009+0000 7fb0ee800700 1 -- start start 2024-03-23T23:00:37.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.009+0000 7fb0ee800700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0e81043e0 0x7fb0e81a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:37.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.009+0000 7fb0ee800700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0e8104da0 0x7fb0e81a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:37.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.009+0000 7fb0ee800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0e810a470 0x7fb0e819aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:37.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.009+0000 7fb0ee800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e8114720 con 0x7fb0e810a470 2024-03-23T23:00:37.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.009+0000 7fb0ee800700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e81145a0 con 0x7fb0e81043e0 2024-03-23T23:00:37.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.009+0000 7fb0ee800700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e81148a0 con 0x7fb0e8104da0 2024-03-23T23:00:37.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.009+0000 7fb0e7400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0e81043e0 0x7fb0e81a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:37.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.009+0000 7fb0e6a00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0e8104da0 0x7fb0e81a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:37.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.010+0000 7fb0e6a00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0e8104da0 0x7fb0e81a0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47546/0 (socket says 172.21.15.12:47546) 2024-03-23T23:00:37.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.010+0000 7fb0e7400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0e81043e0 0x7fb0e81a08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:40860/0 (socket says 172.21.15.12:40860) 2024-03-23T23:00:37.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.010+0000 7fb0e6a00700 1 -- 172.21.15.12:0/3242368535 learned_addr learned my addr 172.21.15.12:0/3242368535 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:37.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.010+0000 7fb0e7e00700 1 --2- 172.21.15.12:0/3242368535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0e810a470 0x7fb0e819aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:37.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.010+0000 7fb0e6a00700 1 -- 172.21.15.12:0/3242368535 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0e81043e0 msgr2=0x7fb0e81a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:37.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.010+0000 7fb0e6a00700 1 --2- 172.21.15.12:0/3242368535 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0e81043e0 0x7fb0e81a08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:37.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.010+0000 7fb0e6a00700 1 -- 172.21.15.12:0/3242368535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0e810a470 msgr2=0x7fb0e819aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:37.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.010+0000 7fb0e6a00700 1 --2- 172.21.15.12:0/3242368535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0e810a470 0x7fb0e819aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:37.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.010+0000 7fb0e6a00700 1 -- 172.21.15.12:0/3242368535 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e819b1e0 con 0x7fb0e8104da0 2024-03-23T23:00:37.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.010+0000 7fb0e6a00700 1 --2- 172.21.15.12:0/3242368535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0e8104da0 0x7fb0e81a0e00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb0d400e9a0 tx=0x7fb0d400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:37.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.011+0000 7fb0dfe00700 1 -- 172.21.15.12:0/3242368535 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d400ca10 con 0x7fb0e8104da0 2024-03-23T23:00:37.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.011+0000 7fb0dfe00700 1 -- 172.21.15.12:0/3242368535 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb0d400cbb0 con 0x7fb0e8104da0 2024-03-23T23:00:37.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.011+0000 7fb0dfe00700 1 -- 172.21.15.12:0/3242368535 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d4018680 con 0x7fb0e8104da0 2024-03-23T23:00:37.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.011+0000 7fb0ee800700 1 -- 172.21.15.12:0/3242368535 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e819b4d0 con 0x7fb0e8104da0 2024-03-23T23:00:37.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.011+0000 7fb0ee800700 1 -- 172.21.15.12:0/3242368535 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e81a7790 con 0x7fb0e8104da0 2024-03-23T23:00:37.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.012+0000 7fb0ee800700 1 -- 172.21.15.12:0/3242368535 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e8068e50 con 0x7fb0e8104da0 2024-03-23T23:00:37.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.014+0000 7fb0dfe00700 1 -- 172.21.15.12:0/3242368535 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fb0d4010430 con 0x7fb0e8104da0 2024-03-23T23:00:37.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.015+0000 7fb0dfe00700 1 --2- 172.21.15.12:0/3242368535 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb0cc0a29a0 0x7fb0cc0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:37.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.015+0000 7fb0dfe00700 1 -- 172.21.15.12:0/3242368535 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb0d40cedf0 con 0x7fb0e8104da0 2024-03-23T23:00:37.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.015+0000 7fb0e7400700 1 --2- 172.21.15.12:0/3242368535 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb0cc0a29a0 0x7fb0cc0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:37.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.016+0000 7fb0e7400700 1 --2- 172.21.15.12:0/3242368535 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb0cc0a29a0 0x7fb0cc0a4e60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb0e819bbc0 tx=0x7fb0d801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:37.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.019+0000 7fb0dfe00700 1 -- 172.21.15.12:0/3242368535 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb0d409bce0 con 0x7fb0e8104da0 2024-03-23T23:00:37.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.277+0000 7fb0ee800700 1 -- 172.21.15.12:0/3242368535 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fb0e802da30 con 0x7fb0e8104da0 2024-03-23T23:00:37.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.277+0000 7fb0dfe00700 1 -- 172.21.15.12:0/3242368535 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb0d409b5e0 con 0x7fb0e8104da0 2024-03-23T23:00:37.277 INFO:teuthology.orchestra.run.smithi012.stdout:34359738457 2024-03-23T23:00:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.280+0000 7fb0ee800700 1 -- 172.21.15.12:0/3242368535 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb0cc0a29a0 msgr2=0x7fb0cc0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.280+0000 7fb0ee800700 1 --2- 172.21.15.12:0/3242368535 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb0cc0a29a0 0x7fb0cc0a4e60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb0e819bbc0 tx=0x7fb0d801a040 comp rx=0 tx=0).stop 2024-03-23T23:00:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.280+0000 7fb0ee800700 1 -- 172.21.15.12:0/3242368535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0e8104da0 msgr2=0x7fb0e81a0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.280+0000 7fb0ee800700 1 --2- 172.21.15.12:0/3242368535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0e8104da0 0x7fb0e81a0e00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb0d400e9a0 tx=0x7fb0d400ecb0 comp rx=0 tx=0).stop 2024-03-23T23:00:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.280+0000 7fb0ee800700 1 -- 172.21.15.12:0/3242368535 shutdown_connections 2024-03-23T23:00:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.280+0000 7fb0ee800700 1 --2- 172.21.15.12:0/3242368535 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb0cc0a29a0 0x7fb0cc0a4e60 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:37.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.280+0000 7fb0ee800700 1 --2- 172.21.15.12:0/3242368535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0e810a470 0x7fb0e819aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:37.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.280+0000 7fb0ee800700 1 --2- 172.21.15.12:0/3242368535 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0e81043e0 0x7fb0e81a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:37.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.280+0000 7fb0ee800700 1 --2- 172.21.15.12:0/3242368535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0e8104da0 0x7fb0e81a0e00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:37.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.280+0000 7fb0ee800700 1 -- 172.21.15.12:0/3242368535 >> 172.21.15.12:0/3242368535 conn(0x7fb0e80fff30 msgr2=0x7fb0e81005d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:37.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.281+0000 7fb0ee800700 1 -- 172.21.15.12:0/3242368535 shutdown_connections 2024-03-23T23:00:37.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.281+0000 7fb0ee800700 1 -- 172.21.15.12:0/3242368535 wait complete. 2024-03-23T23:00:37.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:37 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3242368535' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-23T23:00:37.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:37 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3242368535' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-23T23:00:37.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:37 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3242368535' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-23T23:00:37.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.806+0000 7f45d6400700 1 -- 172.21.15.12:0/2057397605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d0102240 msgr2=0x7f45d01026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:37.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.806+0000 7f45d6400700 1 --2- 172.21.15.12:0/2057397605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d0102240 0x7f45d01026c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f45b8009990 tx=0x7f45b8009ca0 comp rx=0 tx=0).stop 2024-03-23T23:00:37.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.806+0000 7f45d6400700 1 -- 172.21.15.12:0/2057397605 shutdown_connections 2024-03-23T23:00:37.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.806+0000 7f45d6400700 1 --2- 172.21.15.12:0/2057397605 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45d0102c00 0x7f45d010eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:37.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.806+0000 7f45d6400700 1 --2- 172.21.15.12:0/2057397605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d01082b0 0x7f45d0108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:37.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.806+0000 7f45d6400700 1 --2- 172.21.15.12:0/2057397605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d0102240 0x7f45d01026c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:37.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.806+0000 7f45d6400700 1 -- 172.21.15.12:0/2057397605 >> 172.21.15.12:0/2057397605 conn(0x7f45d0079260 msgr2=0x7f45d0079670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:37.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.806+0000 7f45d6400700 1 -- 172.21.15.12:0/2057397605 shutdown_connections 2024-03-23T23:00:37.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.806+0000 7f45d6400700 1 -- 172.21.15.12:0/2057397605 wait complete. 2024-03-23T23:00:37.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.807+0000 7f45d6400700 1 Processor -- start 2024-03-23T23:00:37.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.808+0000 7f45d6400700 1 -- start start 2024-03-23T23:00:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.808+0000 7f45d6400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d0102240 0x7f45d0074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.808+0000 7f45d6400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d0102c00 0x7f45d00749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.808+0000 7f45d6400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45d01082b0 0x7f45d0074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.808+0000 7f45d6400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d0112520 con 0x7f45d0102240 2024-03-23T23:00:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.808+0000 7f45d6400700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d01123a0 con 0x7f45d0102c00 2024-03-23T23:00:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.808+0000 7f45d6400700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d01126a0 con 0x7f45d01082b0 2024-03-23T23:00:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.808+0000 7f45cf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d0102240 0x7f45d0074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.809+0000 7f45cf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d0102240 0x7f45d0074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52618/0 (socket says 172.21.15.12:52618) 2024-03-23T23:00:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.809+0000 7f45cf400700 1 -- 172.21.15.12:0/4123334939 learned_addr learned my addr 172.21.15.12:0/4123334939 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.809+0000 7f45c6a00700 1 --2- 172.21.15.12:0/4123334939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d0102c00 0x7f45d00749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.809+0000 7f45cfe00700 1 --2- 172.21.15.12:0/4123334939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45d01082b0 0x7f45d0074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.809+0000 7f45cf400700 1 -- 172.21.15.12:0/4123334939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45d01082b0 msgr2=0x7f45d0074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.809+0000 7f45cf400700 1 --2- 172.21.15.12:0/4123334939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45d01082b0 0x7f45d0074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.809+0000 7f45cf400700 1 -- 172.21.15.12:0/4123334939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d0102c00 msgr2=0x7f45d00749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:37.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.809+0000 7f45cf400700 1 --2- 172.21.15.12:0/4123334939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d0102c00 0x7f45d00749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:37.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.809+0000 7f45cf400700 1 -- 172.21.15.12:0/4123334939 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d01a6ea0 con 0x7f45d0102240 2024-03-23T23:00:37.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.809+0000 7f45cf400700 1 --2- 172.21.15.12:0/4123334939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d0102240 0x7f45d0074490 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f45c000ef10 tx=0x7f45c000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:37.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.810+0000 7f45ccc00700 1 -- 172.21.15.12:0/4123334939 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c000c9c0 con 0x7f45d0102240 2024-03-23T23:00:37.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.810+0000 7f45ccc00700 1 -- 172.21.15.12:0/4123334939 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45c0019070 con 0x7f45d0102240 2024-03-23T23:00:37.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.810+0000 7f45d6400700 1 -- 172.21.15.12:0/4123334939 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45d01a7130 con 0x7f45d0102240 2024-03-23T23:00:37.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.810+0000 7f45ccc00700 1 -- 172.21.15.12:0/4123334939 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c0007e60 con 0x7f45d0102240 2024-03-23T23:00:37.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.810+0000 7f45d6400700 1 -- 172.21.15.12:0/4123334939 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d01a7670 con 0x7f45d0102240 2024-03-23T23:00:37.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.811+0000 7f45d6400700 1 -- 172.21.15.12:0/4123334939 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45d0068e50 con 0x7f45d0102240 2024-03-23T23:00:37.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.813+0000 7f45ccc00700 1 -- 172.21.15.12:0/4123334939 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f45c0012050 con 0x7f45d0102240 2024-03-23T23:00:37.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.814+0000 7f45ccc00700 1 --2- 172.21.15.12:0/4123334939 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f45b40a2920 0x7f45b40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:37.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.815+0000 7f45ccc00700 1 -- 172.21.15.12:0/4123334939 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f45c00ceb50 con 0x7f45d0102240 2024-03-23T23:00:37.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.815+0000 7f45c6a00700 1 --2- 172.21.15.12:0/4123334939 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f45b40a2920 0x7f45b40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:37.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.815+0000 7f45c6a00700 1 --2- 172.21.15.12:0/4123334939 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f45b40a2920 0x7f45b40a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f45d00797f0 tx=0x7f45b800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:37.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:37.819+0000 7f45ccc00700 1 -- 172.21.15.12:0/4123334939 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f45c0010040 con 0x7f45d0102240 2024-03-23T23:00:38.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:38.060+0000 7f45d6400700 1 -- 172.21.15.12:0/4123334939 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f45d002da30 con 0x7f45d0102240 2024-03-23T23:00:38.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:38.061+0000 7f45ccc00700 1 -- 172.21.15.12:0/4123334939 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f45c009ba40 con 0x7f45d0102240 2024-03-23T23:00:38.060 INFO:teuthology.orchestra.run.smithi012.stdout:55834574933 2024-03-23T23:00:38.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:38.064+0000 7f45d6400700 1 -- 172.21.15.12:0/4123334939 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f45b40a2920 msgr2=0x7f45b40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:38.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:38.064+0000 7f45d6400700 1 --2- 172.21.15.12:0/4123334939 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f45b40a2920 0x7f45b40a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f45d00797f0 tx=0x7f45b800b260 comp rx=0 tx=0).stop 2024-03-23T23:00:38.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:38.064+0000 7f45d6400700 1 -- 172.21.15.12:0/4123334939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d0102240 msgr2=0x7f45d0074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:38.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:38.064+0000 7f45d6400700 1 --2- 172.21.15.12:0/4123334939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d0102240 0x7f45d0074490 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f45c000ef10 tx=0x7f45c000c3a0 comp rx=0 tx=0).stop 2024-03-23T23:00:38.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:38.064+0000 7f45d6400700 1 -- 172.21.15.12:0/4123334939 shutdown_connections 2024-03-23T23:00:38.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:38.064+0000 7f45d6400700 1 --2- 172.21.15.12:0/4123334939 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f45b40a2920 0x7f45b40a4de0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:38.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:38.064+0000 7f45d6400700 1 --2- 172.21.15.12:0/4123334939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45d01082b0 0x7f45d0074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:38.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:38.064+0000 7f45d6400700 1 --2- 172.21.15.12:0/4123334939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d0102240 0x7f45d0074490 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:38.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:38.064+0000 7f45d6400700 1 --2- 172.21.15.12:0/4123334939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d0102c00 0x7f45d00749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:38.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:38.064+0000 7f45d6400700 1 -- 172.21.15.12:0/4123334939 >> 172.21.15.12:0/4123334939 conn(0x7f45d0079260 msgr2=0x7f45d0100df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:38.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:38.064+0000 7f45d6400700 1 -- 172.21.15.12:0/4123334939 shutdown_connections 2024-03-23T23:00:38.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:38.064+0000 7f45d6400700 1 -- 172.21.15.12:0/4123334939 wait complete. 2024-03-23T23:00:38.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:38 smithi012 ceph-mon[31695]: pgmap v355: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 48 B/s, 0 objects/s recovering 2024-03-23T23:00:38.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:38 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/4123334939' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-23T23:00:38.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:38 smithi173 ceph-mon[36116]: pgmap v355: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 48 B/s, 0 objects/s recovering 2024-03-23T23:00:38.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:38 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/4123334939' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-23T23:00:38.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:38 smithi203 ceph-mon[36693]: pgmap v355: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 48 B/s, 0 objects/s recovering 2024-03-23T23:00:38.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:38 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/4123334939' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-23T23:00:39.128 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888125 got 146028888130 for osd.4 2024-03-23T23:00:39.128 DEBUG:teuthology.parallel:result is None 2024-03-23T23:00:39.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.356+0000 7f9802200700 1 -- 172.21.15.12:0/252908191 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97fc10e7a0 msgr2=0x7f97fc10eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:39.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.356+0000 7f9802200700 1 --2- 172.21.15.12:0/252908191 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97fc10e7a0 0x7f97fc10eb80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f97ec009990 tx=0x7f97ec009ca0 comp rx=0 tx=0).stop 2024-03-23T23:00:39.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.357+0000 7f9802200700 1 -- 172.21.15.12:0/252908191 shutdown_connections 2024-03-23T23:00:39.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.357+0000 7f9802200700 1 --2- 172.21.15.12:0/252908191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97fc109160 0x7f97fc115670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:39.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.357+0000 7f9802200700 1 --2- 172.21.15.12:0/252908191 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97fc10e7a0 0x7f97fc10eb80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:39.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.357+0000 7f9802200700 1 --2- 172.21.15.12:0/252908191 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97fc1087a0 0x7f97fc108c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:39.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.357+0000 7f9802200700 1 -- 172.21.15.12:0/252908191 >> 172.21.15.12:0/252908191 conn(0x7f97fc075960 msgr2=0x7f97fc077d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:39.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.357+0000 7f9802200700 1 -- 172.21.15.12:0/252908191 shutdown_connections 2024-03-23T23:00:39.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.357+0000 7f9802200700 1 -- 172.21.15.12:0/252908191 wait complete. 2024-03-23T23:00:39.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.358+0000 7f9802200700 1 Processor -- start 2024-03-23T23:00:39.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.358+0000 7f9802200700 1 -- start start 2024-03-23T23:00:39.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.359+0000 7f9802200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97fc1087a0 0x7f97fc19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:39.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.359+0000 7f9802200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97fc109160 0x7f97fc19f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:39.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.359+0000 7f9802200700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97fc10e7a0 0x7f97fc1a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:39.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.359+0000 7f9802200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc118b20 con 0x7f97fc1087a0 2024-03-23T23:00:39.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.359+0000 7f9802200700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc1189a0 con 0x7f97fc10e7a0 2024-03-23T23:00:39.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.359+0000 7f9802200700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc118ca0 con 0x7f97fc109160 2024-03-23T23:00:39.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.359+0000 7f97faa00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97fc109160 0x7f97fc19f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:39.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.359+0000 7f97fb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97fc1087a0 0x7f97fc19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:39.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.359+0000 7f97faa00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97fc109160 0x7f97fc19f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47584/0 (socket says 172.21.15.12:47584) 2024-03-23T23:00:39.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.359+0000 7f97faa00700 1 -- 172.21.15.12:0/3415067626 learned_addr learned my addr 172.21.15.12:0/3415067626 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:39.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.359+0000 7f97fbe00700 1 --2- 172.21.15.12:0/3415067626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97fc10e7a0 0x7f97fc1a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:39.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.359+0000 7f97faa00700 1 -- 172.21.15.12:0/3415067626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97fc10e7a0 msgr2=0x7f97fc1a3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:39.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.360+0000 7f97faa00700 1 --2- 172.21.15.12:0/3415067626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97fc10e7a0 0x7f97fc1a3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:39.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.360+0000 7f97faa00700 1 -- 172.21.15.12:0/3415067626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97fc1087a0 msgr2=0x7f97fc19ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:39.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.360+0000 7f97faa00700 1 --2- 172.21.15.12:0/3415067626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97fc1087a0 0x7f97fc19ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:39.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.360+0000 7f97faa00700 1 -- 172.21.15.12:0/3415067626 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97fc1a3f40 con 0x7f97fc109160 2024-03-23T23:00:39.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.360+0000 7f97fbe00700 1 --2- 172.21.15.12:0/3415067626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97fc10e7a0 0x7f97fc1a3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:00:39.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.360+0000 7f97faa00700 1 --2- 172.21.15.12:0/3415067626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97fc109160 0x7f97fc19f470 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f97e800eb30 tx=0x7f97e800ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:39.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.360+0000 7f97f3e00700 1 -- 172.21.15.12:0/3415067626 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e800cbd0 con 0x7f97fc109160 2024-03-23T23:00:39.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.360+0000 7f97f3e00700 1 -- 172.21.15.12:0/3415067626 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f97e8004500 con 0x7f97fc109160 2024-03-23T23:00:39.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.360+0000 7f97f3e00700 1 -- 172.21.15.12:0/3415067626 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e8010430 con 0x7f97fc109160 2024-03-23T23:00:39.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.360+0000 7f9802200700 1 -- 172.21.15.12:0/3415067626 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97fc1a4230 con 0x7f97fc109160 2024-03-23T23:00:39.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.360+0000 7f9802200700 1 -- 172.21.15.12:0/3415067626 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97fc1abab0 con 0x7f97fc109160 2024-03-23T23:00:39.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.362+0000 7f9802200700 1 -- 172.21.15.12:0/3415067626 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97fc068e50 con 0x7f97fc109160 2024-03-23T23:00:39.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.363+0000 7f97f3e00700 1 -- 172.21.15.12:0/3415067626 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f97e8003f00 con 0x7f97fc109160 2024-03-23T23:00:39.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.364+0000 7f97f3e00700 1 --2- 172.21.15.12:0/3415067626 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f97e00a29a0 0x7f97e00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:39.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.364+0000 7f97f3e00700 1 -- 172.21.15.12:0/3415067626 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f97e80ceef0 con 0x7f97fc109160 2024-03-23T23:00:39.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.365+0000 7f97fb400700 1 --2- 172.21.15.12:0/3415067626 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f97e00a29a0 0x7f97e00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:39.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.365+0000 7f97fb400700 1 --2- 172.21.15.12:0/3415067626 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f97e00a29a0 0x7f97e00a4e60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f97ec006010 tx=0x7f97ec00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.368+0000 7f97f3e00700 1 -- 172.21.15.12:0/3415067626 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f97e809bde0 con 0x7f97fc109160 2024-03-23T23:00:39.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.540+0000 7f91a8200700 1 -- 172.21.15.12:0/569053840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91a01081e0 msgr2=0x7f91a01085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:39.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.540+0000 7f91a8200700 1 --2- 172.21.15.12:0/569053840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91a01081e0 0x7f91a01085c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f91940099c0 tx=0x7f9194009cd0 comp rx=0 tx=0).stop 2024-03-23T23:00:39.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.540+0000 7f91a8200700 1 -- 172.21.15.12:0/569053840 shutdown_connections 2024-03-23T23:00:39.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.540+0000 7f91a8200700 1 --2- 172.21.15.12:0/569053840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91a01081e0 0x7f91a01085c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:39.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.540+0000 7f91a8200700 1 --2- 172.21.15.12:0/569053840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91a006a7e0 0x7f91a010f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:39.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.540+0000 7f91a8200700 1 --2- 172.21.15.12:0/569053840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91a0069e40 0x7f91a006a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:39.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.540+0000 7f91a8200700 1 -- 172.21.15.12:0/569053840 >> 172.21.15.12:0/569053840 conn(0x7f91a0077fc0 msgr2=0x7f91a00783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:39.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.540+0000 7f91a8200700 1 -- 172.21.15.12:0/569053840 shutdown_connections 2024-03-23T23:00:39.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.541+0000 7f91a8200700 1 -- 172.21.15.12:0/569053840 wait complete. 2024-03-23T23:00:39.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.542+0000 7f91a8200700 1 Processor -- start 2024-03-23T23:00:39.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.542+0000 7f91a8200700 1 -- start start 2024-03-23T23:00:39.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.543+0000 7f91a8200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91a0069e40 0x7f91a019e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:39.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.543+0000 7f91a8200700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91a006a7e0 0x7f91a019ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:39.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.543+0000 7f91a8200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91a01081e0 0x7f91a0198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:39.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.543+0000 7f91a8200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a0112420 con 0x7f91a0069e40 2024-03-23T23:00:39.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.543+0000 7f91a8200700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a01122a0 con 0x7f91a006a7e0 2024-03-23T23:00:39.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.543+0000 7f91a8200700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a01125a0 con 0x7f91a01081e0 2024-03-23T23:00:39.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.543+0000 7f91a5800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91a0069e40 0x7f91a019e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:39.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.543+0000 7f91a6200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91a01081e0 0x7f91a0198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:39.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.543+0000 7f91a5800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91a0069e40 0x7f91a019e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52656/0 (socket says 172.21.15.12:52656) 2024-03-23T23:00:39.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.543+0000 7f91a5800700 1 -- 172.21.15.12:0/781502733 learned_addr learned my addr 172.21.15.12:0/781502733 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:39.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.543+0000 7f91a4e00700 1 --2- 172.21.15.12:0/781502733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91a006a7e0 0x7f91a019ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:39.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.544+0000 7f91a6200700 1 -- 172.21.15.12:0/781502733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91a006a7e0 msgr2=0x7f91a019ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:39.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.544+0000 7f91a6200700 1 --2- 172.21.15.12:0/781502733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91a006a7e0 0x7f91a019ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:39.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.544+0000 7f91a6200700 1 -- 172.21.15.12:0/781502733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91a0069e40 msgr2=0x7f91a019e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:39.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.544+0000 7f91a6200700 1 --2- 172.21.15.12:0/781502733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91a0069e40 0x7f91a019e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:39.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.544+0000 7f91a6200700 1 -- 172.21.15.12:0/781502733 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a0199020 con 0x7f91a01081e0 2024-03-23T23:00:39.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.544+0000 7f91a6200700 1 --2- 172.21.15.12:0/781502733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91a01081e0 0x7f91a0198740 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f919800efc0 tx=0x7f919800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:39.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.544+0000 7f919e000700 1 -- 172.21.15.12:0/781502733 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91980092e0 con 0x7f91a01081e0 2024-03-23T23:00:39.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.544+0000 7f919e000700 1 -- 172.21.15.12:0/781502733 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9198004bb0 con 0x7f91a01081e0 2024-03-23T23:00:39.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.544+0000 7f91a8200700 1 -- 172.21.15.12:0/781502733 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a01992b0 con 0x7f91a01081e0 2024-03-23T23:00:39.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.545+0000 7f919e000700 1 -- 172.21.15.12:0/781502733 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9198004280 con 0x7f91a01081e0 2024-03-23T23:00:39.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.545+0000 7f91a8200700 1 -- 172.21.15.12:0/781502733 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a01a55f0 con 0x7f91a01081e0 2024-03-23T23:00:39.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.546+0000 7f91a8200700 1 -- 172.21.15.12:0/781502733 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9180005330 con 0x7f91a01081e0 2024-03-23T23:00:39.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.547+0000 7f919e000700 1 -- 172.21.15.12:0/781502733 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f919801b030 con 0x7f91a01081e0 2024-03-23T23:00:39.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.549+0000 7f919e000700 1 --2- 172.21.15.12:0/781502733 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f91880a29a0 0x7f91880a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:39.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.549+0000 7f919e000700 1 -- 172.21.15.12:0/781502733 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f91980ce570 con 0x7f91a01081e0 2024-03-23T23:00:39.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.549+0000 7f91a5800700 1 --2- 172.21.15.12:0/781502733 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f91880a29a0 0x7f91880a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:39.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.550+0000 7f91a5800700 1 --2- 172.21.15.12:0/781502733 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f91880a29a0 0x7f91880a4e60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9194009750 tx=0x7f9194005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:39.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.557+0000 7f919e000700 1 -- 172.21.15.12:0/781502733 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9198022020 con 0x7f91a01081e0 2024-03-23T23:00:39.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.607+0000 7f9802200700 1 -- 172.21.15.12:0/3415067626 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f97fc02da30 con 0x7f97fc109160 2024-03-23T23:00:39.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.607+0000 7f97f3e00700 1 -- 172.21.15.12:0/3415067626 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f97e809b6e0 con 0x7f97fc109160 2024-03-23T23:00:39.607 INFO:teuthology.orchestra.run.smithi012.stdout:223338299442 2024-03-23T23:00:39.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.610+0000 7f9802200700 1 -- 172.21.15.12:0/3415067626 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f97e00a29a0 msgr2=0x7f97e00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:39.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.610+0000 7f9802200700 1 --2- 172.21.15.12:0/3415067626 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f97e00a29a0 0x7f97e00a4e60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f97ec006010 tx=0x7f97ec00b260 comp rx=0 tx=0).stop 2024-03-23T23:00:39.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.610+0000 7f9802200700 1 -- 172.21.15.12:0/3415067626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97fc109160 msgr2=0x7f97fc19f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:39.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.610+0000 7f9802200700 1 --2- 172.21.15.12:0/3415067626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97fc109160 0x7f97fc19f470 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f97e800eb30 tx=0x7f97e800ee40 comp rx=0 tx=0).stop 2024-03-23T23:00:39.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.610+0000 7f9802200700 1 -- 172.21.15.12:0/3415067626 shutdown_connections 2024-03-23T23:00:39.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.610+0000 7f9802200700 1 --2- 172.21.15.12:0/3415067626 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f97e00a29a0 0x7f97e00a4e60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:39.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.610+0000 7f9802200700 1 --2- 172.21.15.12:0/3415067626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97fc10e7a0 0x7f97fc1a3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:39.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.610+0000 7f9802200700 1 --2- 172.21.15.12:0/3415067626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97fc1087a0 0x7f97fc19ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:39.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.611+0000 7f9802200700 1 --2- 172.21.15.12:0/3415067626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97fc109160 0x7f97fc19f470 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:39.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.611+0000 7f9802200700 1 -- 172.21.15.12:0/3415067626 >> 172.21.15.12:0/3415067626 conn(0x7f97fc075960 msgr2=0x7f97fc075ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:39.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.611+0000 7f9802200700 1 -- 172.21.15.12:0/3415067626 shutdown_connections 2024-03-23T23:00:39.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.611+0000 7f9802200700 1 -- 172.21.15.12:0/3415067626 wait complete. 2024-03-23T23:00:39.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.792+0000 7f91a8200700 1 -- 172.21.15.12:0/781502733 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f91800058f0 con 0x7f91a01081e0 2024-03-23T23:00:39.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.793+0000 7f919e000700 1 -- 172.21.15.12:0/781502733 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9198010040 con 0x7f91a01081e0 2024-03-23T23:00:39.792 INFO:teuthology.orchestra.run.smithi012.stdout:77309411408 2024-03-23T23:00:39.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.796+0000 7f91a8200700 1 -- 172.21.15.12:0/781502733 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f91880a29a0 msgr2=0x7f91880a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:39.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.796+0000 7f91a8200700 1 --2- 172.21.15.12:0/781502733 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f91880a29a0 0x7f91880a4e60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9194009750 tx=0x7f9194005960 comp rx=0 tx=0).stop 2024-03-23T23:00:39.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.796+0000 7f91a8200700 1 -- 172.21.15.12:0/781502733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91a01081e0 msgr2=0x7f91a0198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:39.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.796+0000 7f91a8200700 1 --2- 172.21.15.12:0/781502733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91a01081e0 0x7f91a0198740 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f919800efc0 tx=0x7f919800c3a0 comp rx=0 tx=0).stop 2024-03-23T23:00:39.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.796+0000 7f91a8200700 1 -- 172.21.15.12:0/781502733 shutdown_connections 2024-03-23T23:00:39.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.796+0000 7f91a8200700 1 --2- 172.21.15.12:0/781502733 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f91880a29a0 0x7f91880a4e60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:39.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.796+0000 7f91a8200700 1 --2- 172.21.15.12:0/781502733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91a01081e0 0x7f91a0198740 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:39.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.796+0000 7f91a8200700 1 --2- 172.21.15.12:0/781502733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91a0069e40 0x7f91a019e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:39.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.796+0000 7f91a8200700 1 --2- 172.21.15.12:0/781502733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91a006a7e0 0x7f91a019ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:39.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.796+0000 7f91a8200700 1 -- 172.21.15.12:0/781502733 >> 172.21.15.12:0/781502733 conn(0x7f91a0077fc0 msgr2=0x7f91a00ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:39.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.796+0000 7f91a8200700 1 -- 172.21.15.12:0/781502733 shutdown_connections 2024-03-23T23:00:39.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:39.796+0000 7f91a8200700 1 -- 172.21.15.12:0/781502733 wait complete. 2024-03-23T23:00:40.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:40 smithi012 ceph-mon[31695]: pgmap v356: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:00:40.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:40 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3415067626' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-23T23:00:40.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:40 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/781502733' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-23T23:00:40.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:40 smithi173 ceph-mon[36116]: pgmap v356: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:00:40.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:40 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3415067626' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-23T23:00:40.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:40 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/781502733' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-23T23:00:40.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:40 smithi203 ceph-mon[36693]: pgmap v356: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:00:40.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:40 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3415067626' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-23T23:00:40.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:40 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/781502733' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-23T23:00:40.945 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149764 got 111669149769 for osd.3 2024-03-23T23:00:40.945 DEBUG:teuthology.parallel:result is None 2024-03-23T23:00:41.916 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738453 got 34359738457 for osd.0 2024-03-23T23:00:41.916 DEBUG:teuthology.parallel:result is None 2024-03-23T23:00:42.127 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574926 got 55834574933 for osd.1 2024-03-23T23:00:42.128 DEBUG:teuthology.parallel:result is None 2024-03-23T23:00:42.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.323+0000 7f6243200700 1 -- 172.21.15.12:0/2791006441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f623c10ebc0 msgr2=0x7f623c110fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:42.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.323+0000 7f6243200700 1 --2- 172.21.15.12:0/2791006441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f623c10ebc0 0x7f623c110fb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f623400b0a0 tx=0x7f623400b3b0 comp rx=0 tx=0).stop 2024-03-23T23:00:42.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.323+0000 7f6243200700 1 -- 172.21.15.12:0/2791006441 shutdown_connections 2024-03-23T23:00:42.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.323+0000 7f6243200700 1 --2- 172.21.15.12:0/2791006441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f623c10ebc0 0x7f623c110fb0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:42.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.323+0000 7f6243200700 1 --2- 172.21.15.12:0/2791006441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f623c0743c0 0x7f623c0747a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:42.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.323+0000 7f6243200700 1 --2- 172.21.15.12:0/2791006441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f623c074ce0 0x7f623c10e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:42.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.323+0000 7f6243200700 1 -- 172.21.15.12:0/2791006441 >> 172.21.15.12:0/2791006441 conn(0x7f623c0792f0 msgr2=0x7f623c079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:42.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.323+0000 7f6243200700 1 -- 172.21.15.12:0/2791006441 shutdown_connections 2024-03-23T23:00:42.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.323+0000 7f6243200700 1 -- 172.21.15.12:0/2791006441 wait complete. 2024-03-23T23:00:42.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.325+0000 7f6243200700 1 Processor -- start 2024-03-23T23:00:42.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.325+0000 7f6243200700 1 -- start start 2024-03-23T23:00:42.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.325+0000 7f6243200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f623c0743c0 0x7f623c1a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:42.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.325+0000 7f6243200700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f623c074ce0 0x7f623c1a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:42.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.325+0000 7f6243200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f623c10ebc0 0x7f623c19a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:42.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.325+0000 7f6243200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623c1146d0 con 0x7f623c0743c0 2024-03-23T23:00:42.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.325+0000 7f6243200700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623c114550 con 0x7f623c074ce0 2024-03-23T23:00:42.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.325+0000 7f6243200700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623c114850 con 0x7f623c10ebc0 2024-03-23T23:00:42.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.325+0000 7f623b400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f623c074ce0 0x7f623c1a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:42.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.326+0000 7f623b400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f623c074ce0 0x7f623c1a0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:40930/0 (socket says 172.21.15.12:40930) 2024-03-23T23:00:42.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.326+0000 7f623b400700 1 -- 172.21.15.12:0/2170102760 learned_addr learned my addr 172.21.15.12:0/2170102760 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:42.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.326+0000 7f6241200700 1 --2- 172.21.15.12:0/2170102760 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f623c10ebc0 0x7f623c19a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:42.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.326+0000 7f623be00700 1 --2- 172.21.15.12:0/2170102760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f623c0743c0 0x7f623c1a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.326+0000 7f623b400700 1 -- 172.21.15.12:0/2170102760 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f623c10ebc0 msgr2=0x7f623c19a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.326+0000 7f623b400700 1 --2- 172.21.15.12:0/2170102760 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f623c10ebc0 0x7f623c19a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.326+0000 7f623b400700 1 -- 172.21.15.12:0/2170102760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f623c0743c0 msgr2=0x7f623c1a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.326+0000 7f623b400700 1 --2- 172.21.15.12:0/2170102760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f623c0743c0 0x7f623c1a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.326+0000 7f623b400700 1 -- 172.21.15.12:0/2170102760 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623c19b0f0 con 0x7f623c074ce0 2024-03-23T23:00:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.326+0000 7f6241200700 1 --2- 172.21.15.12:0/2170102760 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f623c10ebc0 0x7f623c19a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:00:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.326+0000 7f623b400700 1 --2- 172.21.15.12:0/2170102760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f623c074ce0 0x7f623c1a0d10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f622800eb30 tx=0x7f622800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.327+0000 7f6238c00700 1 -- 172.21.15.12:0/2170102760 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622800cc90 con 0x7f623c074ce0 2024-03-23T23:00:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.327+0000 7f6238c00700 1 -- 172.21.15.12:0/2170102760 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6228004500 con 0x7f623c074ce0 2024-03-23T23:00:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.327+0000 7f6243200700 1 -- 172.21.15.12:0/2170102760 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f623c19b3e0 con 0x7f623c074ce0 2024-03-23T23:00:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.327+0000 7f6238c00700 1 -- 172.21.15.12:0/2170102760 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6228010430 con 0x7f623c074ce0 2024-03-23T23:00:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.327+0000 7f6243200700 1 -- 172.21.15.12:0/2170102760 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f623c1a76a0 con 0x7f623c074ce0 2024-03-23T23:00:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.328+0000 7f6243200700 1 -- 172.21.15.12:0/2170102760 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f623c068e50 con 0x7f623c074ce0 2024-03-23T23:00:42.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.330+0000 7f6238c00700 1 -- 172.21.15.12:0/2170102760 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f6228003f70 con 0x7f623c074ce0 2024-03-23T23:00:42.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.331+0000 7f6238c00700 1 --2- 172.21.15.12:0/2170102760 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f62200a29a0 0x7f62200a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:42.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.332+0000 7f6238c00700 1 -- 172.21.15.12:0/2170102760 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f62280ceeb0 con 0x7f623c074ce0 2024-03-23T23:00:42.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.332+0000 7f623be00700 1 --2- 172.21.15.12:0/2170102760 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f62200a29a0 0x7f62200a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:42.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.333+0000 7f623be00700 1 --2- 172.21.15.12:0/2170102760 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f62200a29a0 0x7f62200a4e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f623c19bad0 tx=0x7f622c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:42.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.335+0000 7f6238c00700 1 -- 172.21.15.12:0/2170102760 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f622809bda0 con 0x7f623c074ce0 2024-03-23T23:00:42.518 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:42 smithi012 ceph-mon[31695]: pgmap v357: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-23T23:00:42.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.590+0000 7f6243200700 1 -- 172.21.15.12:0/2170102760 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f623c19bec0 con 0x7f623c074ce0 2024-03-23T23:00:42.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.590+0000 7f6238c00700 1 -- 172.21.15.12:0/2170102760 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f622809b6a0 con 0x7f623c074ce0 2024-03-23T23:00:42.589 INFO:teuthology.orchestra.run.smithi012.stdout:167503724607 2024-03-23T23:00:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.593+0000 7f6243200700 1 -- 172.21.15.12:0/2170102760 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f62200a29a0 msgr2=0x7f62200a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.593+0000 7f6243200700 1 --2- 172.21.15.12:0/2170102760 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f62200a29a0 0x7f62200a4e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f623c19bad0 tx=0x7f622c009260 comp rx=0 tx=0).stop 2024-03-23T23:00:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.593+0000 7f6243200700 1 -- 172.21.15.12:0/2170102760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f623c074ce0 msgr2=0x7f623c1a0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.593+0000 7f6243200700 1 --2- 172.21.15.12:0/2170102760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f623c074ce0 0x7f623c1a0d10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f622800eb30 tx=0x7f622800ee40 comp rx=0 tx=0).stop 2024-03-23T23:00:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.593+0000 7f6243200700 1 -- 172.21.15.12:0/2170102760 shutdown_connections 2024-03-23T23:00:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.593+0000 7f6243200700 1 --2- 172.21.15.12:0/2170102760 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f62200a29a0 0x7f62200a4e60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.593+0000 7f6243200700 1 --2- 172.21.15.12:0/2170102760 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f623c10ebc0 0x7f623c19a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.593+0000 7f6243200700 1 --2- 172.21.15.12:0/2170102760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f623c0743c0 0x7f623c1a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.593+0000 7f6243200700 1 --2- 172.21.15.12:0/2170102760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f623c074ce0 0x7f623c1a0d10 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.593+0000 7f6243200700 1 -- 172.21.15.12:0/2170102760 >> 172.21.15.12:0/2170102760 conn(0x7f623c0792f0 msgr2=0x7f623c0ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.594+0000 7f6243200700 1 -- 172.21.15.12:0/2170102760 shutdown_connections 2024-03-23T23:00:42.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:42.594+0000 7f6243200700 1 -- 172.21.15.12:0/2170102760 wait complete. 2024-03-23T23:00:42.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:42 smithi173 ceph-mon[36116]: pgmap v357: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-23T23:00:42.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:42 smithi203 ceph-mon[36693]: pgmap v357: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-23T23:00:43.297 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411403 got 77309411408 for osd.2 2024-03-23T23:00:43.298 DEBUG:teuthology.parallel:result is None 2024-03-23T23:00:43.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:43 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2170102760' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-23T23:00:43.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:43 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2170102760' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-23T23:00:43.655 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:00:43.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:43 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2170102760' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-23T23:00:43.690 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299437 got 223338299442 for osd.7 2024-03-23T23:00:43.690 DEBUG:teuthology.parallel:result is None 2024-03-23T23:00:43.914 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:00:43.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.965+0000 7fe1e4a00700 1 -- 172.21.15.12:0/4218783178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1e0069e40 msgr2=0x7fe1e006a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:43.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.965+0000 7fe1e4a00700 1 --2- 172.21.15.12:0/4218783178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1e0069e40 0x7fe1e006a2a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc0099c0 tx=0x7fe1cc009cd0 comp rx=0 tx=0).stop 2024-03-23T23:00:43.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.966+0000 7fe1e4a00700 1 -- 172.21.15.12:0/4218783178 shutdown_connections 2024-03-23T23:00:43.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.966+0000 7fe1e4a00700 1 --2- 172.21.15.12:0/4218783178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1e006a7e0 0x7fe1e010f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:43.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.966+0000 7fe1e4a00700 1 --2- 172.21.15.12:0/4218783178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e01081e0 0x7fe1e01085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:43.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.966+0000 7fe1e4a00700 1 --2- 172.21.15.12:0/4218783178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1e0069e40 0x7fe1e006a2a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:43.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.966+0000 7fe1e4a00700 1 -- 172.21.15.12:0/4218783178 >> 172.21.15.12:0/4218783178 conn(0x7fe1e0077fc0 msgr2=0x7fe1e00783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:43.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.966+0000 7fe1e4a00700 1 -- 172.21.15.12:0/4218783178 shutdown_connections 2024-03-23T23:00:43.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.966+0000 7fe1e4a00700 1 -- 172.21.15.12:0/4218783178 wait complete. 2024-03-23T23:00:43.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.967+0000 7fe1e4a00700 1 Processor -- start 2024-03-23T23:00:43.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.967+0000 7fe1e4a00700 1 -- start start 2024-03-23T23:00:43.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.968+0000 7fe1e4a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e0069e40 0x7fe1e019e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:43.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.968+0000 7fe1e4a00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1e006a7e0 0x7fe1e019eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:43.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.968+0000 7fe1e4a00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1e01081e0 0x7fe1e0198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:43.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.968+0000 7fe1e4a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e0112490 con 0x7fe1e0069e40 2024-03-23T23:00:43.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.968+0000 7fe1e4a00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e0112310 con 0x7fe1e01081e0 2024-03-23T23:00:43.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.968+0000 7fe1e4a00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e0112610 con 0x7fe1e006a7e0 2024-03-23T23:00:43.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.968+0000 7fe1dd400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1e006a7e0 0x7fe1e019eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:43.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.968+0000 7fe1dd400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1e006a7e0 0x7fe1e019eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45430/0 (socket says 172.21.15.12:45430) 2024-03-23T23:00:43.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.968+0000 7fe1dd400700 1 -- 172.21.15.12:0/3041722362 learned_addr learned my addr 172.21.15.12:0/3041722362 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:43.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.968+0000 7fe1dde00700 1 --2- 172.21.15.12:0/3041722362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e0069e40 0x7fe1e019e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:43.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.968+0000 7fe1dd400700 1 -- 172.21.15.12:0/3041722362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1e01081e0 msgr2=0x7fe1e0198730 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-23T23:00:43.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.968+0000 7fe1dd400700 1 --2- 172.21.15.12:0/3041722362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1e01081e0 0x7fe1e0198730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:43.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.968+0000 7fe1dd400700 1 -- 172.21.15.12:0/3041722362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e0069e40 msgr2=0x7fe1e019e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:43.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.968+0000 7fe1dd400700 1 --2- 172.21.15.12:0/3041722362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e0069e40 0x7fe1e019e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:43.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.968+0000 7fe1dd400700 1 -- 172.21.15.12:0/3041722362 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e0199010 con 0x7fe1e006a7e0 2024-03-23T23:00:43.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.969+0000 7fe1dd400700 1 --2- 172.21.15.12:0/3041722362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1e006a7e0 0x7fe1e019eba0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc006010 tx=0x7fe1cc003eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:43.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.969+0000 7fe1d6a00700 1 -- 172.21.15.12:0/3041722362 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc003680 con 0x7fe1e006a7e0 2024-03-23T23:00:43.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.969+0000 7fe1d6a00700 1 -- 172.21.15.12:0/3041722362 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe1cc00f460 con 0x7fe1e006a7e0 2024-03-23T23:00:43.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.969+0000 7fe1d6a00700 1 -- 172.21.15.12:0/3041722362 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc017650 con 0x7fe1e006a7e0 2024-03-23T23:00:43.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.969+0000 7fe1e4a00700 1 -- 172.21.15.12:0/3041722362 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e01992a0 con 0x7fe1e006a7e0 2024-03-23T23:00:43.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.969+0000 7fe1e4a00700 1 -- 172.21.15.12:0/3041722362 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1e01a5410 con 0x7fe1e006a7e0 2024-03-23T23:00:43.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.971+0000 7fe1e4a00700 1 -- 172.21.15.12:0/3041722362 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1e0103890 con 0x7fe1e006a7e0 2024-03-23T23:00:43.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.972+0000 7fe1d6a00700 1 -- 172.21.15.12:0/3041722362 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fe1cc00f610 con 0x7fe1e006a7e0 2024-03-23T23:00:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.974+0000 7fe1d6a00700 1 --2- 172.21.15.12:0/3041722362 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe1c40a2b40 0x7fe1c40a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.974+0000 7fe1d6a00700 1 -- 172.21.15.12:0/3041722362 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe1cc0cfa30 con 0x7fe1e006a7e0 2024-03-23T23:00:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.974+0000 7fe1dde00700 1 --2- 172.21.15.12:0/3041722362 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe1c40a2b40 0x7fe1c40a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.975+0000 7fe1dde00700 1 --2- 172.21.15.12:0/3041722362 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe1c40a2b40 0x7fe1c40a5000 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0005950 tx=0x7fe1d00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:43.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:43.978+0000 7fe1d6a00700 1 -- 172.21.15.12:0/3041722362 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe1cc013070 con 0x7fe1e006a7e0 2024-03-23T23:00:44.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:44.218+0000 7fe1e4a00700 1 -- 172.21.15.12:0/3041722362 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7fe1e002da30 con 0x7fe1e006a7e0 2024-03-23T23:00:44.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:44.219+0000 7fe1d6a00700 1 -- 172.21.15.12:0/3041722362 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe1cc096e20 con 0x7fe1e006a7e0 2024-03-23T23:00:44.218 INFO:teuthology.orchestra.run.smithi012.stdout:257698037807 2024-03-23T23:00:44.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:44.221+0000 7fe1e4a00700 1 -- 172.21.15.12:0/3041722362 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe1c40a2b40 msgr2=0x7fe1c40a5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:44.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:44.221+0000 7fe1e4a00700 1 --2- 172.21.15.12:0/3041722362 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe1c40a2b40 0x7fe1c40a5000 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0005950 tx=0x7fe1d00058e0 comp rx=0 tx=0).stop 2024-03-23T23:00:44.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:44.221+0000 7fe1e4a00700 1 -- 172.21.15.12:0/3041722362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1e006a7e0 msgr2=0x7fe1e019eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:44.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:44.221+0000 7fe1e4a00700 1 --2- 172.21.15.12:0/3041722362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1e006a7e0 0x7fe1e019eba0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc006010 tx=0x7fe1cc003eb0 comp rx=0 tx=0).stop 2024-03-23T23:00:44.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:44.222+0000 7fe1e4a00700 1 -- 172.21.15.12:0/3041722362 shutdown_connections 2024-03-23T23:00:44.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:44.222+0000 7fe1e4a00700 1 --2- 172.21.15.12:0/3041722362 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe1c40a2b40 0x7fe1c40a5000 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:44.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:44.222+0000 7fe1e4a00700 1 --2- 172.21.15.12:0/3041722362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1e01081e0 0x7fe1e0198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:44.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:44.222+0000 7fe1e4a00700 1 --2- 172.21.15.12:0/3041722362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e0069e40 0x7fe1e019e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:44.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:44.222+0000 7fe1e4a00700 1 --2- 172.21.15.12:0/3041722362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1e006a7e0 0x7fe1e019eba0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:44.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:44.222+0000 7fe1e4a00700 1 -- 172.21.15.12:0/3041722362 >> 172.21.15.12:0/3041722362 conn(0x7fe1e0077fc0 msgr2=0x7fe1e00fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:44.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:44.222+0000 7fe1e4a00700 1 -- 172.21.15.12:0/3041722362 shutdown_connections 2024-03-23T23:00:44.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:44.222+0000 7fe1e4a00700 1 -- 172.21.15.12:0/3041722362 wait complete. 2024-03-23T23:00:44.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:44 smithi012 ceph-mon[31695]: pgmap v358: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:00:44.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:44 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3041722362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-23T23:00:44.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:44 smithi173 ceph-mon[36116]: pgmap v358: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:00:44.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:44 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3041722362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-23T23:00:44.653 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724602 got 167503724607 for osd.5 2024-03-23T23:00:44.653 DEBUG:teuthology.parallel:result is None 2024-03-23T23:00:44.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:44 smithi203 ceph-mon[36693]: pgmap v358: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:00:44.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:44 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3041722362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-23T23:00:46.529 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:46 smithi012 ceph-mon[31695]: pgmap v359: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-23T23:00:46.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:46 smithi173 ceph-mon[36116]: pgmap v359: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-23T23:00:46.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:46 smithi203 ceph-mon[36693]: pgmap v359: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-23T23:00:46.910 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037800 got 257698037807 for osd.8 2024-03-23T23:00:46.911 DEBUG:teuthology.parallel:result is None 2024-03-23T23:00:48.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.096+0000 7faa46200700 1 -- 172.21.15.12:0/689684241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa40102df0 msgr2=0x7faa40106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:48.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.096+0000 7faa46200700 1 --2- 172.21.15.12:0/689684241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa40102df0 0x7faa40106e30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7faa28009990 tx=0x7faa28009ca0 comp rx=0 tx=0).stop 2024-03-23T23:00:48.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.096+0000 7faa46200700 1 -- 172.21.15.12:0/689684241 shutdown_connections 2024-03-23T23:00:48.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.096+0000 7faa46200700 1 --2- 172.21.15.12:0/689684241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa40107510 0x7faa4010f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:48.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.096+0000 7faa46200700 1 --2- 172.21.15.12:0/689684241 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa40102440 0x7faa40102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:48.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.096+0000 7faa46200700 1 --2- 172.21.15.12:0/689684241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa40102df0 0x7faa40106e30 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:48.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.096+0000 7faa46200700 1 -- 172.21.15.12:0/689684241 >> 172.21.15.12:0/689684241 conn(0x7faa400792f0 msgr2=0x7faa40079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:48.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.096+0000 7faa46200700 1 -- 172.21.15.12:0/689684241 shutdown_connections 2024-03-23T23:00:48.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.096+0000 7faa46200700 1 -- 172.21.15.12:0/689684241 wait complete. 2024-03-23T23:00:48.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.098+0000 7faa46200700 1 Processor -- start 2024-03-23T23:00:48.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.098+0000 7faa46200700 1 -- start start 2024-03-23T23:00:48.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.098+0000 7faa46200700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa40102440 0x7faa40074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:48.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.098+0000 7faa46200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa40102df0 0x7faa400749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:48.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.098+0000 7faa46200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa40107510 0x7faa40074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:48.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.098+0000 7faa46200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa40112520 con 0x7faa40102df0 2024-03-23T23:00:48.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.098+0000 7faa46200700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa401123a0 con 0x7faa40102440 2024-03-23T23:00:48.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.098+0000 7faa46200700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa401126a0 con 0x7faa40107510 2024-03-23T23:00:48.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.099+0000 7faa3f400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa40102440 0x7faa40074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:48.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.099+0000 7faa3f400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa40102440 0x7faa40074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:40936/0 (socket says 172.21.15.12:40936) 2024-03-23T23:00:48.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.099+0000 7faa3f400700 1 -- 172.21.15.12:0/3366557105 learned_addr learned my addr 172.21.15.12:0/3366557105 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:48.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.099+0000 7faa37e00700 1 --2- 172.21.15.12:0/3366557105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa40102df0 0x7faa400749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:48.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.099+0000 7faa3fe00700 1 --2- 172.21.15.12:0/3366557105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa40107510 0x7faa40074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:48.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.099+0000 7faa3f400700 1 -- 172.21.15.12:0/3366557105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa40107510 msgr2=0x7faa40074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:48.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.099+0000 7faa3f400700 1 --2- 172.21.15.12:0/3366557105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa40107510 0x7faa40074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:48.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.099+0000 7faa3f400700 1 -- 172.21.15.12:0/3366557105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa40102df0 msgr2=0x7faa400749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:48.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.099+0000 7faa3f400700 1 --2- 172.21.15.12:0/3366557105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa40102df0 0x7faa400749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:48.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.099+0000 7faa3f400700 1 -- 172.21.15.12:0/3366557105 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa401a6f10 con 0x7faa40102440 2024-03-23T23:00:48.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.099+0000 7faa3f400700 1 --2- 172.21.15.12:0/3366557105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa40102440 0x7faa40074490 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7faa3000e9a0 tx=0x7faa3000ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:48.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.100+0000 7faa3cc00700 1 -- 172.21.15.12:0/3366557105 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3000cbd0 con 0x7faa40102440 2024-03-23T23:00:48.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.100+0000 7faa3cc00700 1 -- 172.21.15.12:0/3366557105 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faa3000cd70 con 0x7faa40102440 2024-03-23T23:00:48.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.100+0000 7faa46200700 1 -- 172.21.15.12:0/3366557105 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa401a7200 con 0x7faa40102440 2024-03-23T23:00:48.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.100+0000 7faa3cc00700 1 -- 172.21.15.12:0/3366557105 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa30018780 con 0x7faa40102440 2024-03-23T23:00:48.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.100+0000 7faa46200700 1 -- 172.21.15.12:0/3366557105 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa401a7790 con 0x7faa40102440 2024-03-23T23:00:48.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.101+0000 7faa46200700 1 -- 172.21.15.12:0/3366557105 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa40068e50 con 0x7faa40102440 2024-03-23T23:00:48.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.104+0000 7faa3cc00700 1 -- 172.21.15.12:0/3366557105 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7faa30010430 con 0x7faa40102440 2024-03-23T23:00:48.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.106+0000 7faa3cc00700 1 --2- 172.21.15.12:0/3366557105 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7faa240a29a0 0x7faa240a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:48.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.106+0000 7faa3cc00700 1 -- 172.21.15.12:0/3366557105 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7faa300ceed0 con 0x7faa40102440 2024-03-23T23:00:48.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.106+0000 7faa37e00700 1 --2- 172.21.15.12:0/3366557105 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7faa240a29a0 0x7faa240a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:48.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.107+0000 7faa37e00700 1 --2- 172.21.15.12:0/3366557105 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7faa240a29a0 0x7faa240a4e60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7faa40101d50 tx=0x7faa2801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:48.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.110+0000 7faa3cc00700 1 -- 172.21.15.12:0/3366557105 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faa3009bdc0 con 0x7faa40102440 2024-03-23T23:00:48.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.347+0000 7faa46200700 1 -- 172.21.15.12:0/3366557105 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7faa4002da30 con 0x7faa40102440 2024-03-23T23:00:48.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.349+0000 7faa3cc00700 1 -- 172.21.15.12:0/3366557105 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7faa3009b6c0 con 0x7faa40102440 2024-03-23T23:00:48.348 INFO:teuthology.orchestra.run.smithi012.stdout:347892351006 2024-03-23T23:00:48.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.353+0000 7faa46200700 1 -- 172.21.15.12:0/3366557105 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7faa240a29a0 msgr2=0x7faa240a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:48.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.353+0000 7faa46200700 1 --2- 172.21.15.12:0/3366557105 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7faa240a29a0 0x7faa240a4e60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7faa40101d50 tx=0x7faa2801a040 comp rx=0 tx=0).stop 2024-03-23T23:00:48.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.353+0000 7faa46200700 1 -- 172.21.15.12:0/3366557105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa40102440 msgr2=0x7faa40074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:48.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.353+0000 7faa46200700 1 --2- 172.21.15.12:0/3366557105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa40102440 0x7faa40074490 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7faa3000e9a0 tx=0x7faa3000ed60 comp rx=0 tx=0).stop 2024-03-23T23:00:48.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.353+0000 7faa46200700 1 -- 172.21.15.12:0/3366557105 shutdown_connections 2024-03-23T23:00:48.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.353+0000 7faa46200700 1 --2- 172.21.15.12:0/3366557105 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7faa240a29a0 0x7faa240a4e60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:48.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.353+0000 7faa46200700 1 --2- 172.21.15.12:0/3366557105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa40107510 0x7faa40074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:48.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.353+0000 7faa46200700 1 --2- 172.21.15.12:0/3366557105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa40102440 0x7faa40074490 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:48.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.353+0000 7faa46200700 1 --2- 172.21.15.12:0/3366557105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa40102df0 0x7faa400749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:48.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.353+0000 7faa46200700 1 -- 172.21.15.12:0/3366557105 >> 172.21.15.12:0/3366557105 conn(0x7faa400792f0 msgr2=0x7faa401003d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:48.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.353+0000 7faa46200700 1 -- 172.21.15.12:0/3366557105 shutdown_connections 2024-03-23T23:00:48.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:48.353+0000 7faa46200700 1 -- 172.21.15.12:0/3366557105 wait complete. 2024-03-23T23:00:48.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:48 smithi012 ceph-mon[31695]: pgmap v360: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-23T23:00:48.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:48 smithi173 ceph-mon[36116]: pgmap v360: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-23T23:00:48.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:48 smithi203 ceph-mon[36693]: pgmap v360: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-23T23:00:49.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.605+0000 7fd21e600700 1 -- 172.21.15.12:0/1078673699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd21806a7e0 msgr2=0x7fd21810f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:49.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.605+0000 7fd21e600700 1 --2- 172.21.15.12:0/1078673699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd21806a7e0 0x7fd21810f020 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd210009fc0 tx=0x7fd21000b330 comp rx=0 tx=0).stop 2024-03-23T23:00:49.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.605+0000 7fd21e600700 1 -- 172.21.15.12:0/1078673699 shutdown_connections 2024-03-23T23:00:49.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.605+0000 7fd21e600700 1 --2- 172.21.15.12:0/1078673699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd21806a7e0 0x7fd21810f020 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:49.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.605+0000 7fd21e600700 1 --2- 172.21.15.12:0/1078673699 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd2181081e0 0x7fd2181085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:49.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.605+0000 7fd21e600700 1 --2- 172.21.15.12:0/1078673699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd218069e40 0x7fd21806a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:49.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.605+0000 7fd21e600700 1 -- 172.21.15.12:0/1078673699 >> 172.21.15.12:0/1078673699 conn(0x7fd218077fc0 msgr2=0x7fd2180783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:49.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.605+0000 7fd21e600700 1 -- 172.21.15.12:0/1078673699 shutdown_connections 2024-03-23T23:00:49.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.605+0000 7fd21e600700 1 -- 172.21.15.12:0/1078673699 wait complete. 2024-03-23T23:00:49.606 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:49 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3366557105' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-23T23:00:49.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.607+0000 7fd21e600700 1 Processor -- start 2024-03-23T23:00:49.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.607+0000 7fd21e600700 1 -- start start 2024-03-23T23:00:49.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.607+0000 7fd21e600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd218069e40 0x7fd21819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:49.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.607+0000 7fd21e600700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd21806a7e0 0x7fd21819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:49.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.608+0000 7fd21e600700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2181081e0 0x7fd218198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:49.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.608+0000 7fd21e600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd218112530 con 0x7fd218069e40 2024-03-23T23:00:49.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.608+0000 7fd21e600700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2181123b0 con 0x7fd21806a7e0 2024-03-23T23:00:49.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.608+0000 7fd21e600700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2181126b0 con 0x7fd2181081e0 2024-03-23T23:00:49.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.608+0000 7fd217400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd218069e40 0x7fd21819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:49.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.608+0000 7fd217400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd218069e40 0x7fd21819e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47610/0 (socket says 172.21.15.12:47610) 2024-03-23T23:00:49.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.608+0000 7fd217400700 1 -- 172.21.15.12:0/783815740 learned_addr learned my addr 172.21.15.12:0/783815740 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:49.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.608+0000 7fd216a00700 1 --2- 172.21.15.12:0/783815740 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd21806a7e0 0x7fd21819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:49.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.608+0000 7fd217400700 1 -- 172.21.15.12:0/783815740 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2181081e0 msgr2=0x7fd218198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:49.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.608+0000 7fd217400700 1 --2- 172.21.15.12:0/783815740 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2181081e0 0x7fd218198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:49.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.608+0000 7fd217400700 1 -- 172.21.15.12:0/783815740 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd21806a7e0 msgr2=0x7fd21819eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:49.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.608+0000 7fd217400700 1 --2- 172.21.15.12:0/783815740 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd21806a7e0 0x7fd21819eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:49.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.608+0000 7fd217400700 1 -- 172.21.15.12:0/783815740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd218198fd0 con 0x7fd218069e40 2024-03-23T23:00:49.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.609+0000 7fd216a00700 1 --2- 172.21.15.12:0/783815740 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd21806a7e0 0x7fd21819eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:00:49.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.609+0000 7fd217400700 1 --2- 172.21.15.12:0/783815740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd218069e40 0x7fd21819e650 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd20800ec10 tx=0x7fd20800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:49.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.609+0000 7fd20fe00700 1 -- 172.21.15.12:0/783815740 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20800cde0 con 0x7fd218069e40 2024-03-23T23:00:49.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.609+0000 7fd21e600700 1 -- 172.21.15.12:0/783815740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2181992c0 con 0x7fd218069e40 2024-03-23T23:00:49.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.609+0000 7fd21e600700 1 -- 172.21.15.12:0/783815740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd218101260 con 0x7fd218069e40 2024-03-23T23:00:49.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.609+0000 7fd20fe00700 1 -- 172.21.15.12:0/783815740 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd208004500 con 0x7fd218069e40 2024-03-23T23:00:49.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.609+0000 7fd20fe00700 1 -- 172.21.15.12:0/783815740 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd208010690 con 0x7fd218069e40 2024-03-23T23:00:49.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.611+0000 7fd21e600700 1 -- 172.21.15.12:0/783815740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd218103890 con 0x7fd218069e40 2024-03-23T23:00:49.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.613+0000 7fd20fe00700 1 -- 172.21.15.12:0/783815740 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fd208003680 con 0x7fd218069e40 2024-03-23T23:00:49.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.614+0000 7fd20fe00700 1 --2- 172.21.15.12:0/783815740 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd1fc0a29a0 0x7fd1fc0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:49.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.614+0000 7fd20fe00700 1 -- 172.21.15.12:0/783815740 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd2080cf110 con 0x7fd218069e40 2024-03-23T23:00:49.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.614+0000 7fd216a00700 1 --2- 172.21.15.12:0/783815740 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd1fc0a29a0 0x7fd1fc0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:49.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.615+0000 7fd216a00700 1 --2- 172.21.15.12:0/783815740 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd1fc0a29a0 0x7fd1fc0a4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd218199ee0 tx=0x7fd204008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:49.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:49 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3366557105' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-23T23:00:49.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.620+0000 7fd20fe00700 1 -- 172.21.15.12:0/783815740 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd208016330 con 0x7fd218069e40 2024-03-23T23:00:49.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:49 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3366557105' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-23T23:00:49.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.857+0000 7fd21e600700 1 -- 172.21.15.12:0/783815740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7fd218199c50 con 0x7fd218069e40 2024-03-23T23:00:49.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.858+0000 7fd20fe00700 1 -- 172.21.15.12:0/783815740 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd20809c000 con 0x7fd218069e40 2024-03-23T23:00:49.857 INFO:teuthology.orchestra.run.smithi012.stdout:292057776169 2024-03-23T23:00:49.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.861+0000 7fd21e600700 1 -- 172.21.15.12:0/783815740 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd1fc0a29a0 msgr2=0x7fd1fc0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:49.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.861+0000 7fd21e600700 1 --2- 172.21.15.12:0/783815740 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd1fc0a29a0 0x7fd1fc0a4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd218199ee0 tx=0x7fd204008040 comp rx=0 tx=0).stop 2024-03-23T23:00:49.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.861+0000 7fd21e600700 1 -- 172.21.15.12:0/783815740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd218069e40 msgr2=0x7fd21819e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:49.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.861+0000 7fd21e600700 1 --2- 172.21.15.12:0/783815740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd218069e40 0x7fd21819e650 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd20800ec10 tx=0x7fd20800c2d0 comp rx=0 tx=0).stop 2024-03-23T23:00:49.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.862+0000 7fd21e600700 1 -- 172.21.15.12:0/783815740 shutdown_connections 2024-03-23T23:00:49.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.862+0000 7fd21e600700 1 --2- 172.21.15.12:0/783815740 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd1fc0a29a0 0x7fd1fc0a4e60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:49.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.862+0000 7fd21e600700 1 --2- 172.21.15.12:0/783815740 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2181081e0 0x7fd218198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:49.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.862+0000 7fd21e600700 1 --2- 172.21.15.12:0/783815740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd218069e40 0x7fd21819e650 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:49.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.862+0000 7fd21e600700 1 --2- 172.21.15.12:0/783815740 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd21806a7e0 0x7fd21819eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:49.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.862+0000 7fd21e600700 1 -- 172.21.15.12:0/783815740 >> 172.21.15.12:0/783815740 conn(0x7fd218077fc0 msgr2=0x7fd2180ffc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:49.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.862+0000 7fd21e600700 1 -- 172.21.15.12:0/783815740 shutdown_connections 2024-03-23T23:00:49.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:49.862+0000 7fd21e600700 1 -- 172.21.15.12:0/783815740 wait complete. 2024-03-23T23:00:50.024 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892351001 got 347892351006 for osd.11 2024-03-23T23:00:50.024 DEBUG:teuthology.parallel:result is None 2024-03-23T23:00:50.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:50 smithi012 ceph-mon[31695]: pgmap v361: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:00:50.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:50 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/783815740' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-23T23:00:50.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:50 smithi173 ceph-mon[36116]: pgmap v361: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:00:50.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:50 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/783815740' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-23T23:00:50.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:50 smithi203 ceph-mon[36693]: pgmap v361: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:00:50.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:50 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/783815740' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-23T23:00:50.760 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776163 got 292057776169 for osd.9 2024-03-23T23:00:50.761 DEBUG:teuthology.parallel:result is None 2024-03-23T23:00:50.761 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-23T23:00:50.761 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-23T23:00:51.226 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:00:52.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:52 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:00:52.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:52 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:00:52.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:52 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:00:52.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:52 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:00:52.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:52 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:00:52.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:52 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:00:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.364+0000 7fcbada00700 1 -- 172.21.15.12:0/1985366516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcba8104f60 msgr2=0x7fcba8108fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.364+0000 7fcbada00700 1 --2- 172.21.15.12:0/1985366516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcba8104f60 0x7fcba8108fb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcb94009990 tx=0x7fcb94009ca0 comp rx=0 tx=0).stop 2024-03-23T23:00:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.364+0000 7fcbada00700 1 -- 172.21.15.12:0/1985366516 shutdown_connections 2024-03-23T23:00:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.364+0000 7fcbada00700 1 --2- 172.21.15.12:0/1985366516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcba8109690 0x7fcba8111260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.364+0000 7fcbada00700 1 --2- 172.21.15.12:0/1985366516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcba81045b0 0x7fcba8104990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.364+0000 7fcbada00700 1 --2- 172.21.15.12:0/1985366516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcba8104f60 0x7fcba8108fb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.364+0000 7fcbada00700 1 -- 172.21.15.12:0/1985366516 >> 172.21.15.12:0/1985366516 conn(0x7fcba80ffe60 msgr2=0x7fcba8102280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.364+0000 7fcbada00700 1 -- 172.21.15.12:0/1985366516 shutdown_connections 2024-03-23T23:00:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.364+0000 7fcbada00700 1 -- 172.21.15.12:0/1985366516 wait complete. 2024-03-23T23:00:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.365+0000 7fcbada00700 1 Processor -- start 2024-03-23T23:00:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.366+0000 7fcbada00700 1 -- start start 2024-03-23T23:00:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.366+0000 7fcbada00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcba81045b0 0x7fcba81a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.366+0000 7fcbada00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcba8104f60 0x7fcba81a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.366+0000 7fcbada00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcba8109690 0x7fcba819aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.366+0000 7fcbada00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba81146a0 con 0x7fcba8104f60 2024-03-23T23:00:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.366+0000 7fcbada00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba8114520 con 0x7fcba81045b0 2024-03-23T23:00:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.366+0000 7fcbada00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba8114820 con 0x7fcba8109690 2024-03-23T23:00:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.366+0000 7fcba6a00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcba81045b0 0x7fcba81a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.366+0000 7fcba7400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcba8109690 0x7fcba819aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.366+0000 7fcba6a00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcba81045b0 0x7fcba81a0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:40980/0 (socket says 172.21.15.12:40980) 2024-03-23T23:00:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.366+0000 7fcba6a00700 1 -- 172.21.15.12:0/3095291386 learned_addr learned my addr 172.21.15.12:0/3095291386 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.366+0000 7fcba6000700 1 --2- 172.21.15.12:0/3095291386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcba8104f60 0x7fcba81a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.367+0000 7fcba6a00700 1 -- 172.21.15.12:0/3095291386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcba8109690 msgr2=0x7fcba819aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.367+0000 7fcba6a00700 1 --2- 172.21.15.12:0/3095291386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcba8109690 0x7fcba819aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.367+0000 7fcba6a00700 1 -- 172.21.15.12:0/3095291386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcba8104f60 msgr2=0x7fcba81a0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.367+0000 7fcba6a00700 1 --2- 172.21.15.12:0/3095291386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcba8104f60 0x7fcba81a0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.367+0000 7fcba6a00700 1 -- 172.21.15.12:0/3095291386 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba819b1a0 con 0x7fcba81045b0 2024-03-23T23:00:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.367+0000 7fcba7400700 1 --2- 172.21.15.12:0/3095291386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcba8109690 0x7fcba819aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:00:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.367+0000 7fcba6a00700 1 --2- 172.21.15.12:0/3095291386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcba81045b0 0x7fcba81a0830 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcb9800e9a0 tx=0x7fcb9800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.367+0000 7fcb9f400700 1 -- 172.21.15.12:0/3095291386 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9800ca10 con 0x7fcba81045b0 2024-03-23T23:00:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.367+0000 7fcb9f400700 1 -- 172.21.15.12:0/3095291386 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcb9800cbb0 con 0x7fcba81045b0 2024-03-23T23:00:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.367+0000 7fcbada00700 1 -- 172.21.15.12:0/3095291386 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba819b490 con 0x7fcba81045b0 2024-03-23T23:00:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.367+0000 7fcb9f400700 1 -- 172.21.15.12:0/3095291386 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb98018680 con 0x7fcba81045b0 2024-03-23T23:00:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.367+0000 7fcbada00700 1 -- 172.21.15.12:0/3095291386 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba81a7700 con 0x7fcba81045b0 2024-03-23T23:00:53.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.368+0000 7fcbada00700 1 -- 172.21.15.12:0/3095291386 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcba8068e50 con 0x7fcba81045b0 2024-03-23T23:00:53.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.371+0000 7fcb9f400700 1 -- 172.21.15.12:0/3095291386 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fcb98010430 con 0x7fcba81045b0 2024-03-23T23:00:53.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.372+0000 7fcb9f400700 1 --2- 172.21.15.12:0/3095291386 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcb8c0a29a0 0x7fcb8c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:53.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.372+0000 7fcb9f400700 1 -- 172.21.15.12:0/3095291386 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fcb980cedf0 con 0x7fcba81045b0 2024-03-23T23:00:53.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.372+0000 7fcba6000700 1 --2- 172.21.15.12:0/3095291386 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcb8c0a29a0 0x7fcb8c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:53.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.373+0000 7fcba6000700 1 --2- 172.21.15.12:0/3095291386 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcb8c0a29a0 0x7fcb8c0a4e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fcba810abe0 tx=0x7fcb9400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:53.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.381+0000 7fcb9f400700 1 -- 172.21.15.12:0/3095291386 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcb9809bce0 con 0x7fcba81045b0 2024-03-23T23:00:53.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:53 smithi012 ceph-mon[31695]: pgmap v362: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-23T23:00:53.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:53 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:00:53.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:53 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:00:53.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:53 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:00:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.606+0000 7fcbada00700 1 -- 172.21.15.12:0/3095291386 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcba80625d0 con 0x7fcb8c0a29a0 2024-03-23T23:00:53.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:53 smithi173 ceph-mon[36116]: pgmap v362: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-23T23:00:53.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:53 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:00:53.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:53 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:00:53.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:53 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:00:53.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.631+0000 7fcb9f400700 1 -- 172.21.15.12:0/3095291386 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258917 (secure 0 0 0) 0x7fcb94065380 con 0x7fcb8c0a29a0 2024-03-23T23:00:53.635 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:00:53.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.638+0000 7fcbada00700 1 -- 172.21.15.12:0/3095291386 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcb8c0a29a0 msgr2=0x7fcb8c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:53.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.638+0000 7fcbada00700 1 --2- 172.21.15.12:0/3095291386 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcb8c0a29a0 0x7fcb8c0a4e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fcba810abe0 tx=0x7fcb9400b260 comp rx=0 tx=0).stop 2024-03-23T23:00:53.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.638+0000 7fcbada00700 1 -- 172.21.15.12:0/3095291386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcba81045b0 msgr2=0x7fcba81a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:53.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.638+0000 7fcbada00700 1 --2- 172.21.15.12:0/3095291386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcba81045b0 0x7fcba81a0830 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcb9800e9a0 tx=0x7fcb9800ecb0 comp rx=0 tx=0).stop 2024-03-23T23:00:53.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.638+0000 7fcbada00700 1 -- 172.21.15.12:0/3095291386 shutdown_connections 2024-03-23T23:00:53.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.638+0000 7fcbada00700 1 --2- 172.21.15.12:0/3095291386 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcb8c0a29a0 0x7fcb8c0a4e60 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:53.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.638+0000 7fcbada00700 1 --2- 172.21.15.12:0/3095291386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcba8109690 0x7fcba819aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:53.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.638+0000 7fcbada00700 1 --2- 172.21.15.12:0/3095291386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcba81045b0 0x7fcba81a0830 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:53.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.638+0000 7fcbada00700 1 --2- 172.21.15.12:0/3095291386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcba8104f60 0x7fcba81a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:53.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.638+0000 7fcbada00700 1 -- 172.21.15.12:0/3095291386 >> 172.21.15.12:0/3095291386 conn(0x7fcba80ffe60 msgr2=0x7fcba81021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:53.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.639+0000 7fcbada00700 1 -- 172.21.15.12:0/3095291386 shutdown_connections 2024-03-23T23:00:53.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:53.639+0000 7fcbada00700 1 -- 172.21.15.12:0/3095291386 wait complete. 2024-03-23T23:00:53.639 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-23T23:00:53.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:53 smithi203 ceph-mon[36693]: pgmap v362: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-23T23:00:53.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:53 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:00:53.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:53 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:00:53.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:53 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:00:54.407 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2024-03-23T23:00:53.293126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653677,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":629,"num_read_kb":427,"num_write":339,"num_write_kb":3046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1181318,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369204,"kb_used_data":11652,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572620,"statfs":{"total":1151940427776,"available":1151562362880,"internally_reserved":0,"allocated":11931648,"data_stored":7465503,"data_compressed":112998,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":40,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004925"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.354641+0000","last_change":"2024-03-23T22:59:27.200234+0000","last_active":"2024-03-23T23:00:24.354641+0000","last_peered":"2024-03-23T23:00:24.354641+0000","last_clean":"2024-03-23T23:00:24.354641+0000","last_became_active":"2024-03-23T22:59:27.199864+0000","last_became_peered":"2024-03-23T22:59:27.199864+0000","last_unstale":"2024-03-23T23:00:24.354641+0000","last_undegraded":"2024-03-23T23:00:24.354641+0000","last_fullsized":"2024-03-23T23:00:24.354641+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:56:26.811964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.365125+0000","last_change":"2024-03-23T22:59:27.195101+0000","last_active":"2024-03-23T23:00:24.365125+0000","last_peered":"2024-03-23T23:00:24.365125+0000","last_clean":"2024-03-23T23:00:24.365125+0000","last_became_active":"2024-03-23T22:59:27.194650+0000","last_became_peered":"2024-03-23T22:59:27.194650+0000","last_unstale":"2024-03-23T23:00:24.365125+0000","last_undegraded":"2024-03-23T23:00:24.365125+0000","last_fullsized":"2024-03-23T23:00:24.365125+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:20.396811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.362565+0000","last_change":"2024-03-23T22:59:27.185575+0000","last_active":"2024-03-23T23:00:24.362565+0000","last_peered":"2024-03-23T23:00:24.362565+0000","last_clean":"2024-03-23T23:00:24.362565+0000","last_became_active":"2024-03-23T22:59:27.184328+0000","last_became_peered":"2024-03-23T22:59:27.184328+0000","last_unstale":"2024-03-23T23:00:24.362565+0000","last_undegraded":"2024-03-23T23:00:24.362565+0000","last_fullsized":"2024-03-23T23:00:24.362565+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:31:46.221574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.373336+0000","last_change":"2024-03-23T22:59:27.184434+0000","last_active":"2024-03-23T23:00:24.373336+0000","last_peered":"2024-03-23T23:00:24.373336+0000","last_clean":"2024-03-23T23:00:24.373336+0000","last_became_active":"2024-03-23T22:59:27.183970+0000","last_became_peered":"2024-03-23T22:59:27.183970+0000","last_unstale":"2024-03-23T23:00:24.373336+0000","last_undegraded":"2024-03-23T23:00:24.373336+0000","last_fullsized":"2024-03-23T23:00:24.373336+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:39:05.517976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357053+0000","last_change":"2024-03-23T22:59:27.214741+0000","last_active":"2024-03-23T23:00:24.357053+0000","last_peered":"2024-03-23T23:00:24.357053+0000","last_clean":"2024-03-23T23:00:24.357053+0000","last_became_active":"2024-03-23T22:59:27.214311+0000","last_became_peered":"2024-03-23T22:59:27.214311+0000","last_unstale":"2024-03-23T23:00:24.357053+0000","last_undegraded":"2024-03-23T23:00:24.357053+0000","last_fullsized":"2024-03-23T23:00:24.357053+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:35:38.464955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.365884+0000","last_change":"2024-03-23T22:59:27.200695+0000","last_active":"2024-03-23T23:00:24.365884+0000","last_peered":"2024-03-23T23:00:24.365884+0000","last_clean":"2024-03-23T23:00:24.365884+0000","last_became_active":"2024-03-23T22:59:27.200159+0000","last_became_peered":"2024-03-23T22:59:27.200159+0000","last_unstale":"2024-03-23T23:00:24.365884+0000","last_undegraded":"2024-03-23T23:00:24.365884+0000","last_fullsized":"2024-03-23T23:00:24.365884+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:01:15.147856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356767+0000","last_change":"2024-03-23T22:59:27.199928+0000","last_active":"2024-03-23T23:00:24.356767+0000","last_peered":"2024-03-23T23:00:24.356767+0000","last_clean":"2024-03-23T23:00:24.356767+0000","last_became_active":"2024-03-23T22:59:27.199507+0000","last_became_peered":"2024-03-23T22:59:27.199507+0000","last_unstale":"2024-03-23T23:00:24.356767+0000","last_undegraded":"2024-03-23T23:00:24.356767+0000","last_fullsized":"2024-03-23T23:00:24.356767+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:51:19.145785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.364118+0000","last_change":"2024-03-23T22:59:27.217832+0000","last_active":"2024-03-23T23:00:24.364118+0000","last_peered":"2024-03-23T23:00:24.364118+0000","last_clean":"2024-03-23T23:00:24.364118+0000","last_became_active":"2024-03-23T22:59:27.217496+0000","last_became_peered":"2024-03-23T22:59:27.217496+0000","last_unstale":"2024-03-23T23:00:24.364118+0000","last_undegraded":"2024-03-23T23:00:24.364118+0000","last_fullsized":"2024-03-23T23:00:24.364118+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:01.778627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356932+0000","last_change":"2024-03-23T22:59:27.190982+0000","last_active":"2024-03-23T23:00:24.356932+0000","last_peered":"2024-03-23T23:00:24.356932+0000","last_clean":"2024-03-23T23:00:24.356932+0000","last_became_active":"2024-03-23T22:59:27.190661+0000","last_became_peered":"2024-03-23T22:59:27.190661+0000","last_unstale":"2024-03-23T23:00:24.356932+0000","last_undegraded":"2024-03-23T23:00:24.356932+0000","last_fullsized":"2024-03-23T23:00:24.356932+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:29.822253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.366155+0000","last_change":"2024-03-23T22:59:27.187395+0000","last_active":"2024-03-23T23:00:24.366155+0000","last_peered":"2024-03-23T23:00:24.366155+0000","last_clean":"2024-03-23T23:00:24.366155+0000","last_became_active":"2024-03-23T22:59:27.186849+0000","last_became_peered":"2024-03-23T22:59:27.186849+0000","last_unstale":"2024-03-23T23:00:24.366155+0000","last_undegraded":"2024-03-23T23:00:24.366155+0000","last_fullsized":"2024-03-23T23:00:24.366155+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:05:48.047102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357481+0000","last_change":"2024-03-23T22:59:27.190023+0000","last_active":"2024-03-23T23:00:24.357481+0000","last_peered":"2024-03-23T23:00:24.357481+0000","last_clean":"2024-03-23T23:00:24.357481+0000","last_became_active":"2024-03-23T22:59:27.189697+0000","last_became_peered":"2024-03-23T22:59:27.189697+0000","last_unstale":"2024-03-23T23:00:24.357481+0000","last_undegraded":"2024-03-23T23:00:24.357481+0000","last_fullsized":"2024-03-23T23:00:24.357481+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:59:51.121055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.373246+0000","last_change":"2024-03-23T22:59:27.187382+0000","last_active":"2024-03-23T23:00:24.373246+0000","last_peered":"2024-03-23T23:00:24.373246+0000","last_clean":"2024-03-23T23:00:24.373246+0000","last_became_active":"2024-03-23T22:59:27.186931+0000","last_became_peered":"2024-03-23T22:59:27.186931+0000","last_unstale":"2024-03-23T23:00:24.373246+0000","last_undegraded":"2024-03-23T23:00:24.373246+0000","last_fullsized":"2024-03-23T23:00:24.373246+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:33.583123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.379365+0000","last_change":"2024-03-23T22:59:27.202577+0000","last_active":"2024-03-23T23:00:24.379365+0000","last_peered":"2024-03-23T23:00:24.379365+0000","last_clean":"2024-03-23T23:00:24.379365+0000","last_became_active":"2024-03-23T22:59:27.202122+0000","last_became_peered":"2024-03-23T22:59:27.202122+0000","last_unstale":"2024-03-23T23:00:24.379365+0000","last_undegraded":"2024-03-23T23:00:24.379365+0000","last_fullsized":"2024-03-23T23:00:24.379365+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:01:10.809635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.362279+0000","last_change":"2024-03-23T22:59:27.192915+0000","last_active":"2024-03-23T23:00:24.362279+0000","last_peered":"2024-03-23T23:00:24.362279+0000","last_clean":"2024-03-23T23:00:24.362279+0000","last_became_active":"2024-03-23T22:59:27.192107+0000","last_became_peered":"2024-03-23T22:59:27.192107+0000","last_unstale":"2024-03-23T23:00:24.362279+0000","last_undegraded":"2024-03-23T23:00:24.362279+0000","last_fullsized":"2024-03-23T23:00:24.362279+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:21:19.169211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.362261+0000","last_change":"2024-03-23T22:59:27.192156+0000","last_active":"2024-03-23T23:00:24.362261+0000","last_peered":"2024-03-23T23:00:24.362261+0000","last_clean":"2024-03-23T23:00:24.362261+0000","last_became_active":"2024-03-23T22:59:27.191594+0000","last_became_peered":"2024-03-23T22:59:27.191594+0000","last_unstale":"2024-03-23T23:00:24.362261+0000","last_undegraded":"2024-03-23T23:00:24.362261+0000","last_fullsized":"2024-03-23T23:00:24.362261+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:00:45.480281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.355980+0000","last_change":"2024-03-23T22:59:27.186048+0000","last_active":"2024-03-23T23:00:24.355980+0000","last_peered":"2024-03-23T23:00:24.355980+0000","last_clean":"2024-03-23T23:00:24.355980+0000","last_became_active":"2024-03-23T22:59:27.185710+0000","last_became_peered":"2024-03-23T22:59:27.185710+0000","last_unstale":"2024-03-23T23:00:24.355980+0000","last_undegraded":"2024-03-23T23:00:24.355980+0000","last_fullsized":"2024-03-23T23:00:24.355980+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:06.792665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.354779+0000","last_change":"2024-03-23T22:59:27.206290+0000","last_active":"2024-03-23T23:00:24.354779+0000","last_peered":"2024-03-23T23:00:24.354779+0000","last_clean":"2024-03-23T23:00:24.354779+0000","last_became_active":"2024-03-23T22:59:27.205997+0000","last_became_peered":"2024-03-23T22:59:27.205997+0000","last_unstale":"2024-03-23T23:00:24.354779+0000","last_undegraded":"2024-03-23T23:00:24.354779+0000","last_fullsized":"2024-03-23T23:00:24.354779+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:38:42.006166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.359103+0000","last_change":"2024-03-23T22:59:27.203671+0000","last_active":"2024-03-23T23:00:24.359103+0000","last_peered":"2024-03-23T23:00:24.359103+0000","last_clean":"2024-03-23T23:00:24.359103+0000","last_became_active":"2024-03-23T22:59:27.203273+0000","last_became_peered":"2024-03-23T22:59:27.203273+0000","last_unstale":"2024-03-23T23:00:24.359103+0000","last_undegraded":"2024-03-23T23:00:24.359103+0000","last_fullsized":"2024-03-23T23:00:24.359103+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:48:16.018671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.358691+0000","last_change":"2024-03-23T22:59:27.210063+0000","last_active":"2024-03-23T23:00:24.358691+0000","last_peered":"2024-03-23T23:00:24.358691+0000","last_clean":"2024-03-23T23:00:24.358691+0000","last_became_active":"2024-03-23T22:59:27.209698+0000","last_became_peered":"2024-03-23T22:59:27.209698+0000","last_unstale":"2024-03-23T23:00:24.358691+0000","last_undegraded":"2024-03-23T23:00:24.358691+0000","last_fullsized":"2024-03-23T23:00:24.358691+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:19:01.408901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.358003+0000","last_change":"2024-03-23T22:59:26.445053+0000","last_active":"2024-03-23T23:00:24.358003+0000","last_peered":"2024-03-23T23:00:24.358003+0000","last_clean":"2024-03-23T23:00:24.358003+0000","last_became_active":"2024-03-23T22:59:26.444660+0000","last_became_peered":"2024-03-23T22:59:26.444660+0000","last_unstale":"2024-03-23T23:00:24.358003+0000","last_undegraded":"2024-03-23T23:00:24.358003+0000","last_fullsized":"2024-03-23T23:00:24.358003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:19:16.958228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.367185+0000","last_change":"2024-03-23T22:59:27.208652+0000","last_active":"2024-03-23T23:00:24.367185+0000","last_peered":"2024-03-23T23:00:24.367185+0000","last_clean":"2024-03-23T23:00:24.367185+0000","last_became_active":"2024-03-23T22:59:27.207724+0000","last_became_peered":"2024-03-23T22:59:27.207724+0000","last_unstale":"2024-03-23T23:00:24.367185+0000","last_undegraded":"2024-03-23T23:00:24.367185+0000","last_fullsized":"2024-03-23T23:00:24.367185+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:08.986439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.368619+0000","last_change":"2024-03-23T22:59:26.445371+0000","last_active":"2024-03-23T23:00:24.368619+0000","last_peered":"2024-03-23T23:00:24.368619+0000","last_clean":"2024-03-23T23:00:24.368619+0000","last_became_active":"2024-03-23T22:59:26.445016+0000","last_became_peered":"2024-03-23T22:59:26.445016+0000","last_unstale":"2024-03-23T23:00:24.368619+0000","last_undegraded":"2024-03-23T23:00:24.368619+0000","last_fullsized":"2024-03-23T23:00:24.368619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:31:44.003218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.368618+0000","last_change":"2024-03-23T22:59:27.200624+0000","last_active":"2024-03-23T23:00:24.368618+0000","last_peered":"2024-03-23T23:00:24.368618+0000","last_clean":"2024-03-23T23:00:24.368618+0000","last_became_active":"2024-03-23T22:59:27.200207+0000","last_became_peered":"2024-03-23T22:59:27.200207+0000","last_unstale":"2024-03-23T23:00:24.368618+0000","last_undegraded":"2024-03-23T23:00:24.368618+0000","last_fullsized":"2024-03-23T23:00:24.368618+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:05:31.015869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.380616+0000","last_change":"2024-03-23T23:00:24.380296+0000","last_active":"2024-03-23T23:00:24.380616+0000","last_peered":"2024-03-23T23:00:24.380616+0000","last_clean":"2024-03-23T23:00:24.380616+0000","last_became_active":"2024-03-23T23:00:24.374344+0000","last_became_peered":"2024-03-23T23:00:24.374344+0000","last_unstale":"2024-03-23T23:00:24.380616+0000","last_undegraded":"2024-03-23T23:00:24.380616+0000","last_fullsized":"2024-03-23T23:00:24.380616+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:30:20.174783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356687+0000","last_change":"2024-03-23T22:59:27.204157+0000","last_active":"2024-03-23T23:00:24.356687+0000","last_peered":"2024-03-23T23:00:24.356687+0000","last_clean":"2024-03-23T23:00:24.356687+0000","last_became_active":"2024-03-23T22:59:27.203559+0000","last_became_peered":"2024-03-23T22:59:27.203559+0000","last_unstale":"2024-03-23T23:00:24.356687+0000","last_undegraded":"2024-03-23T23:00:24.356687+0000","last_fullsized":"2024-03-23T23:00:24.356687+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:43.834369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.361574+0000","last_change":"2024-03-23T23:00:24.361233+0000","last_active":"2024-03-23T23:00:24.361574+0000","last_peered":"2024-03-23T23:00:24.361574+0000","last_clean":"2024-03-23T23:00:24.361574+0000","last_became_active":"2024-03-23T23:00:24.358048+0000","last_became_peered":"2024-03-23T23:00:24.358048+0000","last_unstale":"2024-03-23T23:00:24.361574+0000","last_undegraded":"2024-03-23T23:00:24.361574+0000","last_fullsized":"2024-03-23T23:00:24.361574+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:38:37.792273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357220+0000","last_change":"2024-03-23T22:59:27.186599+0000","last_active":"2024-03-23T23:00:24.357220+0000","last_peered":"2024-03-23T23:00:24.357220+0000","last_clean":"2024-03-23T23:00:24.357220+0000","last_became_active":"2024-03-23T22:59:27.186262+0000","last_became_peered":"2024-03-23T22:59:27.186262+0000","last_unstale":"2024-03-23T23:00:24.357220+0000","last_undegraded":"2024-03-23T23:00:24.357220+0000","last_fullsized":"2024-03-23T23:00:24.357220+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:11:39.545558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":104,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:48.280978+0000","last_change":"2024-03-23T22:59:26.430096+0000","last_active":"2024-03-23T23:00:48.280978+0000","last_peered":"2024-03-23T23:00:48.280978+0000","last_clean":"2024-03-23T23:00:48.280978+0000","last_became_active":"2024-03-23T22:59:26.429740+0000","last_became_peered":"2024-03-23T22:59:26.429740+0000","last_unstale":"2024-03-23T23:00:48.280978+0000","last_undegraded":"2024-03-23T23:00:48.280978+0000","last_fullsized":"2024-03-23T23:00:48.280978+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:54.831689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357358+0000","last_change":"2024-03-23T22:59:27.202750+0000","last_active":"2024-03-23T23:00:24.357358+0000","last_peered":"2024-03-23T23:00:24.357358+0000","last_clean":"2024-03-23T23:00:24.357358+0000","last_became_active":"2024-03-23T22:59:27.202391+0000","last_became_peered":"2024-03-23T22:59:27.202391+0000","last_unstale":"2024-03-23T23:00:24.357358+0000","last_undegraded":"2024-03-23T23:00:24.357358+0000","last_fullsized":"2024-03-23T23:00:24.357358+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:27:18.370792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.365876+0000","last_change":"2024-03-23T22:59:26.425607+0000","last_active":"2024-03-23T23:00:24.365876+0000","last_peered":"2024-03-23T23:00:24.365876+0000","last_clean":"2024-03-23T23:00:24.365876+0000","last_became_active":"2024-03-23T22:59:26.425208+0000","last_became_peered":"2024-03-23T22:59:26.425208+0000","last_unstale":"2024-03-23T23:00:24.365876+0000","last_undegraded":"2024-03-23T23:00:24.365876+0000","last_fullsized":"2024-03-23T23:00:24.365876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:51.071662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356486+0000","last_change":"2024-03-23T22:59:27.215637+0000","last_active":"2024-03-23T23:00:24.356486+0000","last_peered":"2024-03-23T23:00:24.356486+0000","last_clean":"2024-03-23T23:00:24.356486+0000","last_became_active":"2024-03-23T22:59:27.215339+0000","last_became_peered":"2024-03-23T22:59:27.215339+0000","last_unstale":"2024-03-23T23:00:24.356486+0000","last_undegraded":"2024-03-23T23:00:24.356486+0000","last_fullsized":"2024-03-23T23:00:24.356486+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:25:42.097326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.377892+0000","last_change":"2024-03-23T22:59:26.442023+0000","last_active":"2024-03-23T23:00:24.377892+0000","last_peered":"2024-03-23T23:00:24.377892+0000","last_clean":"2024-03-23T23:00:24.377892+0000","last_became_active":"2024-03-23T22:59:26.441661+0000","last_became_peered":"2024-03-23T22:59:26.441661+0000","last_unstale":"2024-03-23T23:00:24.377892+0000","last_undegraded":"2024-03-23T23:00:24.377892+0000","last_fullsized":"2024-03-23T23:00:24.377892+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:59:14.914860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.385168+0000","last_change":"2024-03-23T23:00:24.384190+0000","last_active":"2024-03-23T23:00:24.385168+0000","last_peered":"2024-03-23T23:00:24.385168+0000","last_clean":"2024-03-23T23:00:24.385168+0000","last_became_active":"2024-03-23T23:00:24.382141+0000","last_became_peered":"2024-03-23T23:00:24.382141+0000","last_unstale":"2024-03-23T23:00:24.385168+0000","last_undegraded":"2024-03-23T23:00:24.385168+0000","last_fullsized":"2024-03-23T23:00:24.385168+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:48:52.311407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356353+0000","last_change":"2024-03-23T22:59:26.430147+0000","last_active":"2024-03-23T23:00:24.356353+0000","last_peered":"2024-03-23T23:00:24.356353+0000","last_clean":"2024-03-23T23:00:24.356353+0000","last_became_active":"2024-03-23T22:59:26.429843+0000","last_became_peered":"2024-03-23T22:59:26.429843+0000","last_unstale":"2024-03-23T23:00:24.356353+0000","last_undegraded":"2024-03-23T23:00:24.356353+0000","last_fullsized":"2024-03-23T23:00:24.356353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:06:45.691894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357332+0000","last_change":"2024-03-23T22:59:27.212235+0000","last_active":"2024-03-23T23:00:24.357332+0000","last_peered":"2024-03-23T23:00:24.357332+0000","last_clean":"2024-03-23T23:00:24.357332+0000","last_became_active":"2024-03-23T22:59:27.211919+0000","last_became_peered":"2024-03-23T22:59:27.211919+0000","last_unstale":"2024-03-23T23:00:24.357332+0000","last_undegraded":"2024-03-23T23:00:24.357332+0000","last_fullsized":"2024-03-23T23:00:24.357332+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:24.752109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":103,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:51.283623+0000","last_change":"2024-03-23T22:59:26.440623+0000","last_active":"2024-03-23T23:00:51.283623+0000","last_peered":"2024-03-23T23:00:51.283623+0000","last_clean":"2024-03-23T23:00:51.283623+0000","last_became_active":"2024-03-23T22:59:26.440204+0000","last_became_peered":"2024-03-23T22:59:26.440204+0000","last_unstale":"2024-03-23T23:00:51.283623+0000","last_undegraded":"2024-03-23T23:00:51.283623+0000","last_fullsized":"2024-03-23T23:00:51.283623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:58:44.088552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":81,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357177+0000","last_change":"2024-03-23T22:59:27.203916+0000","last_active":"2024-03-23T23:00:24.357177+0000","last_peered":"2024-03-23T23:00:24.357177+0000","last_clean":"2024-03-23T23:00:24.357177+0000","last_became_active":"2024-03-23T22:59:27.203565+0000","last_became_peered":"2024-03-23T22:59:27.203565+0000","last_unstale":"2024-03-23T23:00:24.357177+0000","last_undegraded":"2024-03-23T23:00:24.357177+0000","last_fullsized":"2024-03-23T23:00:24.357177+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:06:39.635025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357536+0000","last_change":"2024-03-23T22:59:26.430080+0000","last_active":"2024-03-23T23:00:24.357536+0000","last_peered":"2024-03-23T23:00:24.357536+0000","last_clean":"2024-03-23T23:00:24.357536+0000","last_became_active":"2024-03-23T22:59:26.429618+0000","last_became_peered":"2024-03-23T22:59:26.429618+0000","last_unstale":"2024-03-23T23:00:24.357536+0000","last_undegraded":"2024-03-23T23:00:24.357536+0000","last_fullsized":"2024-03-23T23:00:24.357536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:01.963573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.364941+0000","last_change":"2024-03-23T22:59:27.190505+0000","last_active":"2024-03-23T23:00:24.364941+0000","last_peered":"2024-03-23T23:00:24.364941+0000","last_clean":"2024-03-23T23:00:24.364941+0000","last_became_active":"2024-03-23T22:59:27.190088+0000","last_became_peered":"2024-03-23T22:59:27.190088+0000","last_unstale":"2024-03-23T23:00:24.364941+0000","last_undegraded":"2024-03-23T23:00:24.364941+0000","last_fullsized":"2024-03-23T23:00:24.364941+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:29:04.255832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357315+0000","last_change":"2024-03-23T22:59:26.438329+0000","last_active":"2024-03-23T23:00:24.357315+0000","last_peered":"2024-03-23T23:00:24.357315+0000","last_clean":"2024-03-23T23:00:24.357315+0000","last_became_active":"2024-03-23T22:59:26.438033+0000","last_became_peered":"2024-03-23T22:59:26.438033+0000","last_unstale":"2024-03-23T23:00:24.357315+0000","last_undegraded":"2024-03-23T23:00:24.357315+0000","last_fullsized":"2024-03-23T23:00:24.357315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:02:38.789574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356751+0000","last_change":"2024-03-23T22:59:27.188417+0000","last_active":"2024-03-23T23:00:24.356751+0000","last_peered":"2024-03-23T23:00:24.356751+0000","last_clean":"2024-03-23T23:00:24.356751+0000","last_became_active":"2024-03-23T22:59:27.188023+0000","last_became_peered":"2024-03-23T22:59:27.188023+0000","last_unstale":"2024-03-23T23:00:24.356751+0000","last_undegraded":"2024-03-23T23:00:24.356751+0000","last_fullsized":"2024-03-23T23:00:24.356751+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:19:11.014604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'27","reported_seq":216,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:52.287344+0000","last_change":"2024-03-23T22:59:26.425963+0000","last_active":"2024-03-23T23:00:52.287344+0000","last_peered":"2024-03-23T23:00:52.287344+0000","last_clean":"2024-03-23T23:00:52.287344+0000","last_became_active":"2024-03-23T22:59:26.425640+0000","last_became_peered":"2024-03-23T22:59:26.425640+0000","last_unstale":"2024-03-23T23:00:52.287344+0000","last_undegraded":"2024-03-23T23:00:52.287344+0000","last_fullsized":"2024-03-23T23:00:52.287344+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:11:26.529034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59465,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":170,"num_read_kb":142,"num_write":28,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356781+0000","last_change":"2024-03-23T22:59:27.187254+0000","last_active":"2024-03-23T23:00:24.356781+0000","last_peered":"2024-03-23T23:00:24.356781+0000","last_clean":"2024-03-23T23:00:24.356781+0000","last_became_active":"2024-03-23T22:59:27.186849+0000","last_became_peered":"2024-03-23T22:59:27.186849+0000","last_unstale":"2024-03-23T23:00:24.356781+0000","last_undegraded":"2024-03-23T23:00:24.356781+0000","last_fullsized":"2024-03-23T23:00:24.356781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:35:30.958565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.358432+0000","last_change":"2024-03-23T22:59:26.429767+0000","last_active":"2024-03-23T23:00:24.358432+0000","last_peered":"2024-03-23T23:00:24.358432+0000","last_clean":"2024-03-23T23:00:24.358432+0000","last_became_active":"2024-03-23T22:59:26.429449+0000","last_became_peered":"2024-03-23T22:59:26.429449+0000","last_unstale":"2024-03-23T23:00:24.358432+0000","last_undegraded":"2024-03-23T23:00:24.358432+0000","last_fullsized":"2024-03-23T23:00:24.358432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:46:13.752427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.378297+0000","last_change":"2024-03-23T22:59:27.223544+0000","last_active":"2024-03-23T23:00:24.378297+0000","last_peered":"2024-03-23T23:00:24.378297+0000","last_clean":"2024-03-23T23:00:24.378297+0000","last_became_active":"2024-03-23T22:59:27.223177+0000","last_became_peered":"2024-03-23T22:59:27.223177+0000","last_unstale":"2024-03-23T23:00:24.378297+0000","last_undegraded":"2024-03-23T23:00:24.378297+0000","last_fullsized":"2024-03-23T23:00:24.378297+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:33:18.521579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356513+0000","last_change":"2024-03-23T22:59:26.427781+0000","last_active":"2024-03-23T23:00:24.356513+0000","last_peered":"2024-03-23T23:00:24.356513+0000","last_clean":"2024-03-23T23:00:24.356513+0000","last_became_active":"2024-03-23T22:59:26.427458+0000","last_became_peered":"2024-03-23T22:59:26.427458+0000","last_unstale":"2024-03-23T23:00:24.356513+0000","last_undegraded":"2024-03-23T23:00:24.356513+0000","last_fullsized":"2024-03-23T23:00:24.356513+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:50:33.956611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357393+0000","last_change":"2024-03-23T22:59:27.203681+0000","last_active":"2024-03-23T23:00:24.357393+0000","last_peered":"2024-03-23T23:00:24.357393+0000","last_clean":"2024-03-23T23:00:24.357393+0000","last_became_active":"2024-03-23T22:59:27.202457+0000","last_became_peered":"2024-03-23T22:59:27.202457+0000","last_unstale":"2024-03-23T23:00:24.357393+0000","last_undegraded":"2024-03-23T23:00:24.357393+0000","last_fullsized":"2024-03-23T23:00:24.357393+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:55:46.725350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'3","reported_seq":103,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:49.337102+0000","last_change":"2024-03-23T22:59:26.428517+0000","last_active":"2024-03-23T23:00:49.337102+0000","last_peered":"2024-03-23T23:00:49.337102+0000","last_clean":"2024-03-23T23:00:49.337102+0000","last_became_active":"2024-03-23T22:59:26.428130+0000","last_became_peered":"2024-03-23T22:59:26.428130+0000","last_unstale":"2024-03-23T23:00:49.337102+0000","last_undegraded":"2024-03-23T23:00:49.337102+0000","last_fullsized":"2024-03-23T23:00:49.337102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:01.185776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.367049+0000","last_change":"2024-03-23T23:00:24.367049+0000","last_active":"2024-03-23T23:00:24.367049+0000","last_peered":"2024-03-23T23:00:24.367049+0000","last_clean":"2024-03-23T23:00:24.367049+0000","last_became_active":"2024-03-23T23:00:24.366636+0000","last_became_peered":"2024-03-23T23:00:24.366636+0000","last_unstale":"2024-03-23T23:00:24.367049+0000","last_undegraded":"2024-03-23T23:00:24.367049+0000","last_fullsized":"2024-03-23T23:00:24.367049+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:07.068500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356895+0000","last_change":"2024-03-23T22:59:26.442637+0000","last_active":"2024-03-23T23:00:24.356895+0000","last_peered":"2024-03-23T23:00:24.356895+0000","last_clean":"2024-03-23T23:00:24.356895+0000","last_became_active":"2024-03-23T22:59:26.442267+0000","last_became_peered":"2024-03-23T22:59:26.442267+0000","last_unstale":"2024-03-23T23:00:24.356895+0000","last_undegraded":"2024-03-23T23:00:24.356895+0000","last_fullsized":"2024-03-23T23:00:24.356895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:49.975048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356983+0000","last_change":"2024-03-23T22:59:27.187812+0000","last_active":"2024-03-23T23:00:24.356983+0000","last_peered":"2024-03-23T23:00:24.356983+0000","last_clean":"2024-03-23T23:00:24.356983+0000","last_became_active":"2024-03-23T22:59:27.187423+0000","last_became_peered":"2024-03-23T22:59:27.187423+0000","last_unstale":"2024-03-23T23:00:24.356983+0000","last_undegraded":"2024-03-23T23:00:24.356983+0000","last_fullsized":"2024-03-23T23:00:24.356983+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:12:16.427695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357307+0000","last_change":"2024-03-23T22:59:26.423875+0000","last_active":"2024-03-23T23:00:24.357307+0000","last_peered":"2024-03-23T23:00:24.357307+0000","last_clean":"2024-03-23T23:00:24.357307+0000","last_became_active":"2024-03-23T22:59:26.423513+0000","last_became_peered":"2024-03-23T22:59:26.423513+0000","last_unstale":"2024-03-23T23:00:24.357307+0000","last_undegraded":"2024-03-23T23:00:24.357307+0000","last_fullsized":"2024-03-23T23:00:24.357307+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:43.173365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.354486+0000","last_change":"2024-03-23T22:59:27.191454+0000","last_active":"2024-03-23T23:00:24.354486+0000","last_peered":"2024-03-23T23:00:24.354486+0000","last_clean":"2024-03-23T23:00:24.354486+0000","last_became_active":"2024-03-23T22:59:27.190927+0000","last_became_peered":"2024-03-23T22:59:27.190927+0000","last_unstale":"2024-03-23T23:00:24.354486+0000","last_undegraded":"2024-03-23T23:00:24.354486+0000","last_fullsized":"2024-03-23T23:00:24.354486+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:00.333702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.358466+0000","last_change":"2024-03-23T22:59:26.443238+0000","last_active":"2024-03-23T23:00:24.358466+0000","last_peered":"2024-03-23T23:00:24.358466+0000","last_clean":"2024-03-23T23:00:24.358466+0000","last_became_active":"2024-03-23T22:59:26.442834+0000","last_became_peered":"2024-03-23T22:59:26.442834+0000","last_unstale":"2024-03-23T23:00:24.358466+0000","last_undegraded":"2024-03-23T23:00:24.358466+0000","last_fullsized":"2024-03-23T23:00:24.358466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:11:21.072836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.365898+0000","last_change":"2024-03-23T22:59:27.212864+0000","last_active":"2024-03-23T23:00:24.365898+0000","last_peered":"2024-03-23T23:00:24.365898+0000","last_clean":"2024-03-23T23:00:24.365898+0000","last_became_active":"2024-03-23T22:59:27.212455+0000","last_became_peered":"2024-03-23T22:59:27.212455+0000","last_unstale":"2024-03-23T23:00:24.365898+0000","last_undegraded":"2024-03-23T23:00:24.365898+0000","last_fullsized":"2024-03-23T23:00:24.365898+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:25:59.122531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357057+0000","last_change":"2024-03-23T22:59:26.432838+0000","last_active":"2024-03-23T23:00:24.357057+0000","last_peered":"2024-03-23T23:00:24.357057+0000","last_clean":"2024-03-23T23:00:24.357057+0000","last_became_active":"2024-03-23T22:59:26.432544+0000","last_became_peered":"2024-03-23T22:59:26.432544+0000","last_unstale":"2024-03-23T23:00:24.357057+0000","last_undegraded":"2024-03-23T23:00:24.357057+0000","last_fullsized":"2024-03-23T23:00:24.357057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:32:52.121569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.376926+0000","last_change":"2024-03-23T23:00:24.376926+0000","last_active":"2024-03-23T23:00:24.376926+0000","last_peered":"2024-03-23T23:00:24.376926+0000","last_clean":"2024-03-23T23:00:24.376926+0000","last_became_active":"2024-03-23T23:00:24.376515+0000","last_became_peered":"2024-03-23T23:00:24.376515+0000","last_unstale":"2024-03-23T23:00:24.376926+0000","last_undegraded":"2024-03-23T23:00:24.376926+0000","last_fullsized":"2024-03-23T23:00:24.376926+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:55:57.730564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.381861+0000","last_change":"2024-03-23T23:00:24.381861+0000","last_active":"2024-03-23T23:00:24.381861+0000","last_peered":"2024-03-23T23:00:24.381861+0000","last_clean":"2024-03-23T23:00:24.381861+0000","last_became_active":"2024-03-23T23:00:24.381472+0000","last_became_peered":"2024-03-23T23:00:24.381472+0000","last_unstale":"2024-03-23T23:00:24.381861+0000","last_undegraded":"2024-03-23T23:00:24.381861+0000","last_fullsized":"2024-03-23T23:00:24.381861+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:44:07.518911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":101,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:48.330883+0000","last_change":"2024-03-23T22:59:26.450679+0000","last_active":"2024-03-23T23:00:48.330883+0000","last_peered":"2024-03-23T23:00:48.330883+0000","last_clean":"2024-03-23T23:00:48.330883+0000","last_became_active":"2024-03-23T22:59:26.450234+0000","last_became_peered":"2024-03-23T22:59:26.450234+0000","last_unstale":"2024-03-23T23:00:48.330883+0000","last_undegraded":"2024-03-23T23:00:48.330883+0000","last_fullsized":"2024-03-23T23:00:48.330883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:52:25.869194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":78,"num_read_kb":78,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.364904+0000","last_change":"2024-03-23T22:59:27.220171+0000","last_active":"2024-03-23T23:00:24.364904+0000","last_peered":"2024-03-23T23:00:24.364904+0000","last_clean":"2024-03-23T23:00:24.364904+0000","last_became_active":"2024-03-23T22:59:27.219787+0000","last_became_peered":"2024-03-23T22:59:27.219787+0000","last_unstale":"2024-03-23T23:00:24.364904+0000","last_undegraded":"2024-03-23T23:00:24.364904+0000","last_fullsized":"2024-03-23T23:00:24.364904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:51:52.634264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.373077+0000","last_change":"2024-03-23T22:59:26.440205+0000","last_active":"2024-03-23T23:00:24.373077+0000","last_peered":"2024-03-23T23:00:24.373077+0000","last_clean":"2024-03-23T23:00:24.373077+0000","last_became_active":"2024-03-23T22:59:26.439752+0000","last_became_peered":"2024-03-23T22:59:26.439752+0000","last_unstale":"2024-03-23T23:00:24.373077+0000","last_undegraded":"2024-03-23T23:00:24.373077+0000","last_fullsized":"2024-03-23T23:00:24.373077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:55:25.661887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":345,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356800+0000","last_change":"2024-03-23T22:58:06.171317+0000","last_active":"2024-03-23T23:00:24.356800+0000","last_peered":"2024-03-23T23:00:24.356800+0000","last_clean":"2024-03-23T23:00:24.356800+0000","last_became_active":"2024-03-23T22:58:06.170816+0000","last_became_peered":"2024-03-23T22:58:06.170816+0000","last_unstale":"2024-03-23T23:00:24.356800+0000","last_undegraded":"2024-03-23T23:00:24.356800+0000","last_fullsized":"2024-03-23T23:00:24.356800+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:54:11.991284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:54:11.991284+0000","last_clean_scrub_stamp":"2024-03-23T22:54:11.991284+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:20:55.980613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.377587+0000","last_change":"2024-03-23T22:59:27.200348+0000","last_active":"2024-03-23T23:00:24.377587+0000","last_peered":"2024-03-23T23:00:24.377587+0000","last_clean":"2024-03-23T23:00:24.377587+0000","last_became_active":"2024-03-23T22:59:27.199979+0000","last_became_peered":"2024-03-23T22:59:27.199979+0000","last_unstale":"2024-03-23T23:00:24.377587+0000","last_undegraded":"2024-03-23T23:00:24.377587+0000","last_fullsized":"2024-03-23T23:00:24.377587+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:31:19.258890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356890+0000","last_change":"2024-03-23T22:59:26.431837+0000","last_active":"2024-03-23T23:00:24.356890+0000","last_peered":"2024-03-23T23:00:24.356890+0000","last_clean":"2024-03-23T23:00:24.356890+0000","last_became_active":"2024-03-23T22:59:26.431580+0000","last_became_peered":"2024-03-23T22:59:26.431580+0000","last_unstale":"2024-03-23T23:00:24.356890+0000","last_undegraded":"2024-03-23T23:00:24.356890+0000","last_fullsized":"2024-03-23T23:00:24.356890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:49:53.316952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356883+0000","last_change":"2024-03-23T22:59:26.442878+0000","last_active":"2024-03-23T23:00:24.356883+0000","last_peered":"2024-03-23T23:00:24.356883+0000","last_clean":"2024-03-23T23:00:24.356883+0000","last_became_active":"2024-03-23T22:59:26.442303+0000","last_became_peered":"2024-03-23T22:59:26.442303+0000","last_unstale":"2024-03-23T23:00:24.356883+0000","last_undegraded":"2024-03-23T23:00:24.356883+0000","last_fullsized":"2024-03-23T23:00:24.356883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:51.717553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.380036+0000","last_change":"2024-03-23T22:59:27.212550+0000","last_active":"2024-03-23T23:00:24.380036+0000","last_peered":"2024-03-23T23:00:24.380036+0000","last_clean":"2024-03-23T23:00:24.380036+0000","last_became_active":"2024-03-23T22:59:27.212105+0000","last_became_peered":"2024-03-23T22:59:27.212105+0000","last_unstale":"2024-03-23T23:00:24.380036+0000","last_undegraded":"2024-03-23T23:00:24.380036+0000","last_fullsized":"2024-03-23T23:00:24.380036+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:23:37.374081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.358910+0000","last_change":"2024-03-23T22:59:26.452146+0000","last_active":"2024-03-23T23:00:24.358910+0000","last_peered":"2024-03-23T23:00:24.358910+0000","last_clean":"2024-03-23T23:00:24.358910+0000","last_became_active":"2024-03-23T22:59:26.450398+0000","last_became_peered":"2024-03-23T22:59:26.450398+0000","last_unstale":"2024-03-23T23:00:24.358910+0000","last_undegraded":"2024-03-23T23:00:24.358910+0000","last_fullsized":"2024-03-23T23:00:24.358910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:55:47.085457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.366547+0000","last_change":"2024-03-23T22:59:27.209690+0000","last_active":"2024-03-23T23:00:24.366547+0000","last_peered":"2024-03-23T23:00:24.366547+0000","last_clean":"2024-03-23T23:00:24.366547+0000","last_became_active":"2024-03-23T22:59:27.209187+0000","last_became_peered":"2024-03-23T22:59:27.209187+0000","last_unstale":"2024-03-23T23:00:24.366547+0000","last_undegraded":"2024-03-23T23:00:24.366547+0000","last_fullsized":"2024-03-23T23:00:24.366547+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:19:45.315459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.373690+0000","last_change":"2024-03-23T22:59:26.434958+0000","last_active":"2024-03-23T23:00:24.373690+0000","last_peered":"2024-03-23T23:00:24.373690+0000","last_clean":"2024-03-23T23:00:24.373690+0000","last_became_active":"2024-03-23T22:59:26.434611+0000","last_became_peered":"2024-03-23T22:59:26.434611+0000","last_unstale":"2024-03-23T23:00:24.373690+0000","last_undegraded":"2024-03-23T23:00:24.373690+0000","last_fullsized":"2024-03-23T23:00:24.373690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:42:18.201885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.368130+0000","last_change":"2024-03-23T22:59:27.211700+0000","last_active":"2024-03-23T23:00:24.368130+0000","last_peered":"2024-03-23T23:00:24.368130+0000","last_clean":"2024-03-23T23:00:24.368130+0000","last_became_active":"2024-03-23T22:59:27.211248+0000","last_became_peered":"2024-03-23T22:59:27.211248+0000","last_unstale":"2024-03-23T23:00:24.368130+0000","last_undegraded":"2024-03-23T23:00:24.368130+0000","last_fullsized":"2024-03-23T23:00:24.368130+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:49.868397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356020+0000","last_change":"2024-03-23T22:59:26.442625+0000","last_active":"2024-03-23T23:00:24.356020+0000","last_peered":"2024-03-23T23:00:24.356020+0000","last_clean":"2024-03-23T23:00:24.356020+0000","last_became_active":"2024-03-23T22:59:26.442312+0000","last_became_peered":"2024-03-23T22:59:26.442312+0000","last_unstale":"2024-03-23T23:00:24.356020+0000","last_undegraded":"2024-03-23T23:00:24.356020+0000","last_fullsized":"2024-03-23T23:00:24.356020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:27:38.808784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356155+0000","last_change":"2024-03-23T22:59:27.187363+0000","last_active":"2024-03-23T23:00:24.356155+0000","last_peered":"2024-03-23T23:00:24.356155+0000","last_clean":"2024-03-23T23:00:24.356155+0000","last_became_active":"2024-03-23T22:59:27.187007+0000","last_became_peered":"2024-03-23T22:59:27.187007+0000","last_unstale":"2024-03-23T23:00:24.356155+0000","last_undegraded":"2024-03-23T23:00:24.356155+0000","last_fullsized":"2024-03-23T23:00:24.356155+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:41:37.041254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.369396+0000","last_change":"2024-03-23T22:59:26.445753+0000","last_active":"2024-03-23T23:00:24.369396+0000","last_peered":"2024-03-23T23:00:24.369396+0000","last_clean":"2024-03-23T23:00:24.369396+0000","last_became_active":"2024-03-23T22:59:26.445167+0000","last_became_peered":"2024-03-23T22:59:26.445167+0000","last_unstale":"2024-03-23T23:00:24.369396+0000","last_undegraded":"2024-03-23T23:00:24.369396+0000","last_fullsized":"2024-03-23T23:00:24.369396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:46:46.058523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.354286+0000","last_change":"2024-03-23T22:59:27.203855+0000","last_active":"2024-03-23T23:00:24.354286+0000","last_peered":"2024-03-23T23:00:24.354286+0000","last_clean":"2024-03-23T23:00:24.354286+0000","last_became_active":"2024-03-23T22:59:27.203438+0000","last_became_peered":"2024-03-23T22:59:27.203438+0000","last_unstale":"2024-03-23T23:00:24.354286+0000","last_undegraded":"2024-03-23T23:00:24.354286+0000","last_fullsized":"2024-03-23T23:00:24.354286+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:28.725743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.358167+0000","last_change":"2024-03-23T22:59:26.439002+0000","last_active":"2024-03-23T23:00:24.358167+0000","last_peered":"2024-03-23T23:00:24.358167+0000","last_clean":"2024-03-23T23:00:24.358167+0000","last_became_active":"2024-03-23T22:59:26.438634+0000","last_became_peered":"2024-03-23T22:59:26.438634+0000","last_unstale":"2024-03-23T23:00:24.358167+0000","last_undegraded":"2024-03-23T23:00:24.358167+0000","last_fullsized":"2024-03-23T23:00:24.358167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:25:02.988163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.365706+0000","last_change":"2024-03-23T22:59:27.217314+0000","last_active":"2024-03-23T23:00:24.365706+0000","last_peered":"2024-03-23T23:00:24.365706+0000","last_clean":"2024-03-23T23:00:24.365706+0000","last_became_active":"2024-03-23T22:59:27.216790+0000","last_became_peered":"2024-03-23T22:59:27.216790+0000","last_unstale":"2024-03-23T23:00:24.365706+0000","last_undegraded":"2024-03-23T23:00:24.365706+0000","last_fullsized":"2024-03-23T23:00:24.365706+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:47:13.902236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357102+0000","last_change":"2024-03-23T22:59:26.431453+0000","last_active":"2024-03-23T23:00:24.357102+0000","last_peered":"2024-03-23T23:00:24.357102+0000","last_clean":"2024-03-23T23:00:24.357102+0000","last_became_active":"2024-03-23T22:59:26.429731+0000","last_became_peered":"2024-03-23T22:59:26.429731+0000","last_unstale":"2024-03-23T23:00:24.357102+0000","last_undegraded":"2024-03-23T23:00:24.357102+0000","last_fullsized":"2024-03-23T23:00:24.357102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:33:47.717064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357172+0000","last_change":"2024-03-23T22:59:27.203097+0000","last_active":"2024-03-23T23:00:24.357172+0000","last_peered":"2024-03-23T23:00:24.357172+0000","last_clean":"2024-03-23T23:00:24.357172+0000","last_became_active":"2024-03-23T22:59:27.202484+0000","last_became_peered":"2024-03-23T22:59:27.202484+0000","last_unstale":"2024-03-23T23:00:24.357172+0000","last_undegraded":"2024-03-23T23:00:24.357172+0000","last_fullsized":"2024-03-23T23:00:24.357172+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:07.829866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.377539+0000","last_change":"2024-03-23T23:00:24.377180+0000","last_active":"2024-03-23T23:00:24.377539+0000","last_peered":"2024-03-23T23:00:24.377539+0000","last_clean":"2024-03-23T23:00:24.377539+0000","last_became_active":"2024-03-23T23:00:24.364530+0000","last_became_peered":"2024-03-23T23:00:24.364530+0000","last_unstale":"2024-03-23T23:00:24.377539+0000","last_undegraded":"2024-03-23T23:00:24.377539+0000","last_fullsized":"2024-03-23T23:00:24.377539+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:57.507891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357526+0000","last_change":"2024-03-23T22:59:27.189244+0000","last_active":"2024-03-23T23:00:24.357526+0000","last_peered":"2024-03-23T23:00:24.357526+0000","last_clean":"2024-03-23T23:00:24.357526+0000","last_became_active":"2024-03-23T22:59:27.188629+0000","last_became_peered":"2024-03-23T22:59:27.188629+0000","last_unstale":"2024-03-23T23:00:24.357526+0000","last_undegraded":"2024-03-23T23:00:24.357526+0000","last_fullsized":"2024-03-23T23:00:24.357526+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:47:26.742204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356440+0000","last_change":"2024-03-23T22:59:26.434326+0000","last_active":"2024-03-23T23:00:24.356440+0000","last_peered":"2024-03-23T23:00:24.356440+0000","last_clean":"2024-03-23T23:00:24.356440+0000","last_became_active":"2024-03-23T22:59:26.433878+0000","last_became_peered":"2024-03-23T22:59:26.433878+0000","last_unstale":"2024-03-23T23:00:24.356440+0000","last_undegraded":"2024-03-23T23:00:24.356440+0000","last_fullsized":"2024-03-23T23:00:24.356440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:08.580572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357402+0000","last_change":"2024-03-23T22:59:27.212909+0000","last_active":"2024-03-23T23:00:24.357402+0000","last_peered":"2024-03-23T23:00:24.357402+0000","last_clean":"2024-03-23T23:00:24.357402+0000","last_became_active":"2024-03-23T22:59:27.212504+0000","last_became_peered":"2024-03-23T22:59:27.212504+0000","last_unstale":"2024-03-23T23:00:24.357402+0000","last_undegraded":"2024-03-23T23:00:24.357402+0000","last_fullsized":"2024-03-23T23:00:24.357402+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:09.238272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.380222+0000","last_change":"2024-03-23T22:59:26.452577+0000","last_active":"2024-03-23T23:00:24.380222+0000","last_peered":"2024-03-23T23:00:24.380222+0000","last_clean":"2024-03-23T23:00:24.380222+0000","last_became_active":"2024-03-23T22:59:26.452107+0000","last_became_peered":"2024-03-23T22:59:26.452107+0000","last_unstale":"2024-03-23T23:00:24.380222+0000","last_undegraded":"2024-03-23T23:00:24.380222+0000","last_fullsized":"2024-03-23T23:00:24.380222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:32:18.472334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357677+0000","last_change":"2024-03-23T22:59:27.188539+0000","last_active":"2024-03-23T23:00:24.357677+0000","last_peered":"2024-03-23T23:00:24.357677+0000","last_clean":"2024-03-23T23:00:24.357677+0000","last_became_active":"2024-03-23T22:59:27.188184+0000","last_became_peered":"2024-03-23T22:59:27.188184+0000","last_unstale":"2024-03-23T23:00:24.357677+0000","last_undegraded":"2024-03-23T23:00:24.357677+0000","last_fullsized":"2024-03-23T23:00:24.357677+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:37:59.756492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.354571+0000","last_change":"2024-03-23T22:59:27.192083+0000","last_active":"2024-03-23T23:00:24.354571+0000","last_peered":"2024-03-23T23:00:24.354571+0000","last_clean":"2024-03-23T23:00:24.354571+0000","last_became_active":"2024-03-23T22:59:27.191709+0000","last_became_peered":"2024-03-23T22:59:27.191709+0000","last_unstale":"2024-03-23T23:00:24.354571+0000","last_undegraded":"2024-03-23T23:00:24.354571+0000","last_fullsized":"2024-03-23T23:00:24.354571+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:18:42.905177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.358496+0000","last_change":"2024-03-23T22:59:27.186315+0000","last_active":"2024-03-23T23:00:24.358496+0000","last_peered":"2024-03-23T23:00:24.358496+0000","last_clean":"2024-03-23T23:00:24.358496+0000","last_became_active":"2024-03-23T22:59:27.185960+0000","last_became_peered":"2024-03-23T22:59:27.185960+0000","last_unstale":"2024-03-23T23:00:24.358496+0000","last_undegraded":"2024-03-23T23:00:24.358496+0000","last_fullsized":"2024-03-23T23:00:24.358496+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:57:07.739725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.358060+0000","last_change":"2024-03-23T22:59:27.221696+0000","last_active":"2024-03-23T23:00:24.358060+0000","last_peered":"2024-03-23T23:00:24.358060+0000","last_clean":"2024-03-23T23:00:24.358060+0000","last_became_active":"2024-03-23T22:59:27.221400+0000","last_became_peered":"2024-03-23T22:59:27.221400+0000","last_unstale":"2024-03-23T23:00:24.358060+0000","last_undegraded":"2024-03-23T23:00:24.358060+0000","last_fullsized":"2024-03-23T23:00:24.358060+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:17.905737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.363724+0000","last_change":"2024-03-23T22:59:27.206286+0000","last_active":"2024-03-23T23:00:24.363724+0000","last_peered":"2024-03-23T23:00:24.363724+0000","last_clean":"2024-03-23T23:00:24.363724+0000","last_became_active":"2024-03-23T22:59:27.205854+0000","last_became_peered":"2024-03-23T22:59:27.205854+0000","last_unstale":"2024-03-23T23:00:24.363724+0000","last_undegraded":"2024-03-23T23:00:24.363724+0000","last_fullsized":"2024-03-23T23:00:24.363724+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:00:45.216211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.378891+0000","last_change":"2024-03-23T22:59:27.211376+0000","last_active":"2024-03-23T23:00:24.378891+0000","last_peered":"2024-03-23T23:00:24.378891+0000","last_clean":"2024-03-23T23:00:24.378891+0000","last_became_active":"2024-03-23T22:59:27.211061+0000","last_became_peered":"2024-03-23T22:59:27.211061+0000","last_unstale":"2024-03-23T23:00:24.378891+0000","last_undegraded":"2024-03-23T23:00:24.378891+0000","last_fullsized":"2024-03-23T23:00:24.378891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:53:25.776927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.354556+0000","last_change":"2024-03-23T22:59:27.191028+0000","last_active":"2024-03-23T23:00:24.354556+0000","last_peered":"2024-03-23T23:00:24.354556+0000","last_clean":"2024-03-23T23:00:24.354556+0000","last_became_active":"2024-03-23T22:59:27.190634+0000","last_became_peered":"2024-03-23T22:59:27.190634+0000","last_unstale":"2024-03-23T23:00:24.354556+0000","last_undegraded":"2024-03-23T23:00:24.354556+0000","last_fullsized":"2024-03-23T23:00:24.354556+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:50:52.110636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.358975+0000","last_change":"2024-03-23T22:59:27.183235+0000","last_active":"2024-03-23T23:00:24.358975+0000","last_peered":"2024-03-23T23:00:24.358975+0000","last_clean":"2024-03-23T23:00:24.358975+0000","last_became_active":"2024-03-23T22:59:27.181682+0000","last_became_peered":"2024-03-23T22:59:27.181682+0000","last_unstale":"2024-03-23T23:00:24.358975+0000","last_undegraded":"2024-03-23T23:00:24.358975+0000","last_fullsized":"2024-03-23T23:00:24.358975+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:33:26.002324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.378143+0000","last_change":"2024-03-23T23:00:24.378143+0000","last_active":"2024-03-23T23:00:24.378143+0000","last_peered":"2024-03-23T23:00:24.378143+0000","last_clean":"2024-03-23T23:00:24.378143+0000","last_became_active":"2024-03-23T23:00:24.377660+0000","last_became_peered":"2024-03-23T23:00:24.377660+0000","last_unstale":"2024-03-23T23:00:24.378143+0000","last_undegraded":"2024-03-23T23:00:24.378143+0000","last_fullsized":"2024-03-23T23:00:24.378143+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:24:29.250679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356464+0000","last_change":"2024-03-23T22:59:27.215230+0000","last_active":"2024-03-23T23:00:24.356464+0000","last_peered":"2024-03-23T23:00:24.356464+0000","last_clean":"2024-03-23T23:00:24.356464+0000","last_became_active":"2024-03-23T22:59:27.214881+0000","last_became_peered":"2024-03-23T22:59:27.214881+0000","last_unstale":"2024-03-23T23:00:24.356464+0000","last_undegraded":"2024-03-23T23:00:24.356464+0000","last_fullsized":"2024-03-23T23:00:24.356464+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:10:40.428481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.365535+0000","last_change":"2024-03-23T22:59:27.213454+0000","last_active":"2024-03-23T23:00:24.365535+0000","last_peered":"2024-03-23T23:00:24.365535+0000","last_clean":"2024-03-23T23:00:24.365535+0000","last_became_active":"2024-03-23T22:59:27.212684+0000","last_became_peered":"2024-03-23T22:59:27.212684+0000","last_unstale":"2024-03-23T23:00:24.365535+0000","last_undegraded":"2024-03-23T23:00:24.365535+0000","last_fullsized":"2024-03-23T23:00:24.365535+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:48:51.584892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356287+0000","last_change":"2024-03-23T22:59:27.220414+0000","last_active":"2024-03-23T23:00:24.356287+0000","last_peered":"2024-03-23T23:00:24.356287+0000","last_clean":"2024-03-23T23:00:24.356287+0000","last_became_active":"2024-03-23T22:59:27.220020+0000","last_became_peered":"2024-03-23T22:59:27.220020+0000","last_unstale":"2024-03-23T23:00:24.356287+0000","last_undegraded":"2024-03-23T23:00:24.356287+0000","last_fullsized":"2024-03-23T23:00:24.356287+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:35:48.890004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.379596+0000","last_change":"2024-03-23T22:59:27.215648+0000","last_active":"2024-03-23T23:00:24.379596+0000","last_peered":"2024-03-23T23:00:24.379596+0000","last_clean":"2024-03-23T23:00:24.379596+0000","last_became_active":"2024-03-23T22:59:27.212252+0000","last_became_peered":"2024-03-23T22:59:27.212252+0000","last_unstale":"2024-03-23T23:00:24.379596+0000","last_undegraded":"2024-03-23T23:00:24.379596+0000","last_fullsized":"2024-03-23T23:00:24.379596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:08:38.956508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.354885+0000","last_change":"2024-03-23T22:59:27.190527+0000","last_active":"2024-03-23T23:00:24.354885+0000","last_peered":"2024-03-23T23:00:24.354885+0000","last_clean":"2024-03-23T23:00:24.354885+0000","last_became_active":"2024-03-23T22:59:27.190026+0000","last_became_peered":"2024-03-23T22:59:27.190026+0000","last_unstale":"2024-03-23T23:00:24.354885+0000","last_undegraded":"2024-03-23T23:00:24.354885+0000","last_fullsized":"2024-03-23T23:00:24.354885+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:31:24.663584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247269,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63075,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":503,"num_read_kb":318,"num_write":156,"num_write_kb":106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":189225,"data_compressed":5619,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351007,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":950272,"data_stored":459288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612644,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":1110234,"data_compressed":37666,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776170,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":459524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037809,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":459444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299445,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1015808,"data_stored":1050983,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462971,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":954368,"data_stored":458913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":518753,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":974848,"data_stored":460184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":518939,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149773,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":459224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888134,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":459350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724609,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":1050667,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34152,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34152,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":60047,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":60233,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":61043,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T23:00:54.412 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-23T23:00:54.947 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:00:55.434 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:55 smithi012 ceph-mon[31695]: pgmap v363: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:00:55.434 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:55 smithi012 ceph-mon[31695]: from='client.24908 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T23:00:55.434 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:55 smithi012 ceph-mon[31695]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-23T23:00:55.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:55 smithi173 ceph-mon[36116]: pgmap v363: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:00:55.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:55 smithi173 ceph-mon[36116]: from='client.24908 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T23:00:55.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:55 smithi173 ceph-mon[36116]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-23T23:00:55.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:55 smithi203 ceph-mon[36693]: pgmap v363: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:00:55.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:55 smithi203 ceph-mon[36693]: from='client.24908 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T23:00:55.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:55 smithi203 ceph-mon[36693]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-23T23:00:57.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.269+0000 7f8e9e600700 1 -- 172.21.15.12:0/537436294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e98104e90 msgr2=0x7f8e98108ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:57.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.269+0000 7f8e9e600700 1 --2- 172.21.15.12:0/537436294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e98104e90 0x7f8e98108ee0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8e84009990 tx=0x7f8e84009ca0 comp rx=0 tx=0).stop 2024-03-23T23:00:57.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.269+0000 7f8e9e600700 1 -- 172.21.15.12:0/537436294 shutdown_connections 2024-03-23T23:00:57.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.269+0000 7f8e9e600700 1 --2- 172.21.15.12:0/537436294 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8e981095c0 0x7f8e98111190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:57.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.269+0000 7f8e9e600700 1 --2- 172.21.15.12:0/537436294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e981044e0 0x7f8e981048c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:57.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.269+0000 7f8e9e600700 1 --2- 172.21.15.12:0/537436294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e98104e90 0x7f8e98108ee0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:57.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.269+0000 7f8e9e600700 1 -- 172.21.15.12:0/537436294 >> 172.21.15.12:0/537436294 conn(0x7f8e980ffdb0 msgr2=0x7f8e981021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:57.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.269+0000 7f8e9e600700 1 -- 172.21.15.12:0/537436294 shutdown_connections 2024-03-23T23:00:57.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.269+0000 7f8e9e600700 1 -- 172.21.15.12:0/537436294 wait complete. 2024-03-23T23:00:57.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.270+0000 7f8e9e600700 1 Processor -- start 2024-03-23T23:00:57.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.271+0000 7f8e9e600700 1 -- start start 2024-03-23T23:00:57.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.271+0000 7f8e9e600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e981044e0 0x7f8e981a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:57.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.271+0000 7f8e9e600700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e98104e90 0x7f8e981a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:57.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.271+0000 7f8e9e600700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8e981095c0 0x7f8e9819a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:57.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.271+0000 7f8e9e600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e981145a0 con 0x7f8e981044e0 2024-03-23T23:00:57.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.271+0000 7f8e9e600700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e98114420 con 0x7f8e98104e90 2024-03-23T23:00:57.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.271+0000 7f8e9e600700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e98114720 con 0x7f8e981095c0 2024-03-23T23:00:57.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.271+0000 7f8e97e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e98104e90 0x7f8e981a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:57.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.271+0000 7f8e9d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e981044e0 0x7f8e981a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:57.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.271+0000 7f8e9dc00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8e981095c0 0x7f8e9819a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:57.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.271+0000 7f8e97e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e98104e90 0x7f8e981a0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:41442/0 (socket says 172.21.15.12:41442) 2024-03-23T23:00:57.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.271+0000 7f8e97e00700 1 -- 172.21.15.12:0/3679585006 learned_addr learned my addr 172.21.15.12:0/3679585006 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:00:57.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.272+0000 7f8e97e00700 1 -- 172.21.15.12:0/3679585006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8e981095c0 msgr2=0x7f8e9819a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:57.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.272+0000 7f8e97e00700 1 --2- 172.21.15.12:0/3679585006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8e981095c0 0x7f8e9819a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:57.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.272+0000 7f8e97e00700 1 -- 172.21.15.12:0/3679585006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e981044e0 msgr2=0x7f8e981a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:57.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.272+0000 7f8e97e00700 1 --2- 172.21.15.12:0/3679585006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e981044e0 0x7f8e981a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:57.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.272+0000 7f8e97e00700 1 -- 172.21.15.12:0/3679585006 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9819b020 con 0x7f8e98104e90 2024-03-23T23:00:57.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.272+0000 7f8e97e00700 1 --2- 172.21.15.12:0/3679585006 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e98104e90 0x7f8e981a0c00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8e84000c00 tx=0x7f8e8400b610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:57.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.272+0000 7f8e95600700 1 -- 172.21.15.12:0/3679585006 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e84027070 con 0x7f8e98104e90 2024-03-23T23:00:57.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.273+0000 7f8e95600700 1 -- 172.21.15.12:0/3679585006 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e84022470 con 0x7f8e98104e90 2024-03-23T23:00:57.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.273+0000 7f8e95600700 1 -- 172.21.15.12:0/3679585006 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e84017650 con 0x7f8e98104e90 2024-03-23T23:00:57.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.273+0000 7f8e9e600700 1 -- 172.21.15.12:0/3679585006 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e9819b2b0 con 0x7f8e98104e90 2024-03-23T23:00:57.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.273+0000 7f8e9e600700 1 -- 172.21.15.12:0/3679585006 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e981a7520 con 0x7f8e98104e90 2024-03-23T23:00:57.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.274+0000 7f8e9e600700 1 -- 172.21.15.12:0/3679585006 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e981059c0 con 0x7f8e98104e90 2024-03-23T23:00:57.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.276+0000 7f8e95600700 1 -- 172.21.15.12:0/3679585006 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8e84009d10 con 0x7f8e98104e90 2024-03-23T23:00:57.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.277+0000 7f8e95600700 1 --2- 172.21.15.12:0/3679585006 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8e800a2910 0x7f8e800a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:00:57.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.277+0000 7f8e95600700 1 -- 172.21.15.12:0/3679585006 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8e840cfa80 con 0x7f8e98104e90 2024-03-23T23:00:57.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.277+0000 7f8e9d200700 1 --2- 172.21.15.12:0/3679585006 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8e800a2910 0x7f8e800a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:00:57.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.278+0000 7f8e9d200700 1 --2- 172.21.15.12:0/3679585006 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8e800a2910 0x7f8e800a4dd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c009c60 tx=0x7f8e8c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:00:57.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.280+0000 7f8e95600700 1 -- 172.21.15.12:0/3679585006 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8e84013070 con 0x7f8e98104e90 2024-03-23T23:00:57.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.504+0000 7f8e9e600700 1 -- 172.21.15.12:0/3679585006 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8e9819bd70 con 0x7f8e800a2910 2024-03-23T23:00:57.504 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:57 smithi012 ceph-mon[31695]: pgmap v364: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-23T23:00:57.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.514+0000 7f8e95600700 1 -- 172.21.15.12:0/3679585006 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259148 (secure 0 0 0) 0x7f8e9819bd70 con 0x7f8e800a2910 2024-03-23T23:00:57.517 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:00:57.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.519+0000 7f8e9e600700 1 -- 172.21.15.12:0/3679585006 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8e800a2910 msgr2=0x7f8e800a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:57.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.519+0000 7f8e9e600700 1 --2- 172.21.15.12:0/3679585006 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8e800a2910 0x7f8e800a4dd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c009c60 tx=0x7f8e8c009260 comp rx=0 tx=0).stop 2024-03-23T23:00:57.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.519+0000 7f8e9e600700 1 -- 172.21.15.12:0/3679585006 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e98104e90 msgr2=0x7f8e981a0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:00:57.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.519+0000 7f8e9e600700 1 --2- 172.21.15.12:0/3679585006 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e98104e90 0x7f8e981a0c00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8e84000c00 tx=0x7f8e8400b610 comp rx=0 tx=0).stop 2024-03-23T23:00:57.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.519+0000 7f8e9e600700 1 -- 172.21.15.12:0/3679585006 shutdown_connections 2024-03-23T23:00:57.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.519+0000 7f8e9e600700 1 --2- 172.21.15.12:0/3679585006 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8e800a2910 0x7f8e800a4dd0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:57.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.519+0000 7f8e9e600700 1 --2- 172.21.15.12:0/3679585006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8e981095c0 0x7f8e9819a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:57.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.519+0000 7f8e9e600700 1 --2- 172.21.15.12:0/3679585006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e981044e0 0x7f8e981a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:57.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.519+0000 7f8e9e600700 1 --2- 172.21.15.12:0/3679585006 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e98104e90 0x7f8e981a0c00 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:00:57.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.520+0000 7f8e9e600700 1 -- 172.21.15.12:0/3679585006 >> 172.21.15.12:0/3679585006 conn(0x7f8e980ffdb0 msgr2=0x7f8e981008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:00:57.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.520+0000 7f8e9e600700 1 -- 172.21.15.12:0/3679585006 shutdown_connections 2024-03-23T23:00:57.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:00:57.520+0000 7f8e9e600700 1 -- 172.21.15.12:0/3679585006 wait complete. 2024-03-23T23:00:57.521 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-23T23:00:57.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:57 smithi173 ceph-mon[36116]: pgmap v364: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-23T23:00:57.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:57 smithi203 ceph-mon[36693]: pgmap v364: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-23T23:00:58.467 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:58 smithi012 ceph-mon[31695]: pgmap v365: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-23T23:00:58.467 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:00:58 smithi012 ceph-mon[31695]: from='client.24914 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T23:00:58.518 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:58 smithi173 ceph-mon[36116]: pgmap v365: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-23T23:00:58.518 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:00:58 smithi173 ceph-mon[36116]: from='client.24914 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T23:00:58.550 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2024-03-23T23:00:57.294626+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653677,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":644,"num_read_kb":432,"num_write":341,"num_write_kb":3046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1181318,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":677,"ondisk_log_size":677,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369140,"kb_used_data":11652,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572684,"statfs":{"total":1151940427776,"available":1151562428416,"internally_reserved":0,"allocated":11931648,"data_stored":7466001,"data_compressed":112998,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,2,11,12],"upper_bound":32},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":30,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004482"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.285590+0000","last_change":"2024-03-23T22:59:27.200234+0000","last_active":"2024-03-23T23:00:53.285590+0000","last_peered":"2024-03-23T23:00:53.285590+0000","last_clean":"2024-03-23T23:00:53.285590+0000","last_became_active":"2024-03-23T22:59:27.199864+0000","last_became_peered":"2024-03-23T22:59:27.199864+0000","last_unstale":"2024-03-23T23:00:53.285590+0000","last_undegraded":"2024-03-23T23:00:53.285590+0000","last_fullsized":"2024-03-23T23:00:53.285590+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:56:26.811964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.299711+0000","last_change":"2024-03-23T22:59:27.195101+0000","last_active":"2024-03-23T23:00:53.299711+0000","last_peered":"2024-03-23T23:00:53.299711+0000","last_clean":"2024-03-23T23:00:53.299711+0000","last_became_active":"2024-03-23T22:59:27.194650+0000","last_became_peered":"2024-03-23T22:59:27.194650+0000","last_unstale":"2024-03-23T23:00:53.299711+0000","last_undegraded":"2024-03-23T23:00:53.299711+0000","last_fullsized":"2024-03-23T23:00:53.299711+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:20.396811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.285994+0000","last_change":"2024-03-23T22:59:27.185575+0000","last_active":"2024-03-23T23:00:53.285994+0000","last_peered":"2024-03-23T23:00:53.285994+0000","last_clean":"2024-03-23T23:00:53.285994+0000","last_became_active":"2024-03-23T22:59:27.184328+0000","last_became_peered":"2024-03-23T22:59:27.184328+0000","last_unstale":"2024-03-23T23:00:53.285994+0000","last_undegraded":"2024-03-23T23:00:53.285994+0000","last_fullsized":"2024-03-23T23:00:53.285994+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:31:46.221574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.268258+0000","last_change":"2024-03-23T22:59:27.184434+0000","last_active":"2024-03-23T23:00:53.268258+0000","last_peered":"2024-03-23T23:00:53.268258+0000","last_clean":"2024-03-23T23:00:53.268258+0000","last_became_active":"2024-03-23T22:59:27.183970+0000","last_became_peered":"2024-03-23T22:59:27.183970+0000","last_unstale":"2024-03-23T23:00:53.268258+0000","last_undegraded":"2024-03-23T23:00:53.268258+0000","last_fullsized":"2024-03-23T23:00:53.268258+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:39:05.517976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357053+0000","last_change":"2024-03-23T22:59:27.214741+0000","last_active":"2024-03-23T23:00:24.357053+0000","last_peered":"2024-03-23T23:00:24.357053+0000","last_clean":"2024-03-23T23:00:24.357053+0000","last_became_active":"2024-03-23T22:59:27.214311+0000","last_became_peered":"2024-03-23T22:59:27.214311+0000","last_unstale":"2024-03-23T23:00:24.357053+0000","last_undegraded":"2024-03-23T23:00:24.357053+0000","last_fullsized":"2024-03-23T23:00:24.357053+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:35:38.464955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.365884+0000","last_change":"2024-03-23T22:59:27.200695+0000","last_active":"2024-03-23T23:00:24.365884+0000","last_peered":"2024-03-23T23:00:24.365884+0000","last_clean":"2024-03-23T23:00:24.365884+0000","last_became_active":"2024-03-23T22:59:27.200159+0000","last_became_peered":"2024-03-23T22:59:27.200159+0000","last_unstale":"2024-03-23T23:00:24.365884+0000","last_undegraded":"2024-03-23T23:00:24.365884+0000","last_fullsized":"2024-03-23T23:00:24.365884+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:01:15.147856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.280147+0000","last_change":"2024-03-23T22:59:27.199928+0000","last_active":"2024-03-23T23:00:53.280147+0000","last_peered":"2024-03-23T23:00:53.280147+0000","last_clean":"2024-03-23T23:00:53.280147+0000","last_became_active":"2024-03-23T22:59:27.199507+0000","last_became_peered":"2024-03-23T22:59:27.199507+0000","last_unstale":"2024-03-23T23:00:53.280147+0000","last_undegraded":"2024-03-23T23:00:53.280147+0000","last_fullsized":"2024-03-23T23:00:53.280147+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:51:19.145785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.364118+0000","last_change":"2024-03-23T22:59:27.217832+0000","last_active":"2024-03-23T23:00:24.364118+0000","last_peered":"2024-03-23T23:00:24.364118+0000","last_clean":"2024-03-23T23:00:24.364118+0000","last_became_active":"2024-03-23T22:59:27.217496+0000","last_became_peered":"2024-03-23T22:59:27.217496+0000","last_unstale":"2024-03-23T23:00:24.364118+0000","last_undegraded":"2024-03-23T23:00:24.364118+0000","last_fullsized":"2024-03-23T23:00:24.364118+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:01.778627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.288651+0000","last_change":"2024-03-23T22:59:27.190982+0000","last_active":"2024-03-23T23:00:53.288651+0000","last_peered":"2024-03-23T23:00:53.288651+0000","last_clean":"2024-03-23T23:00:53.288651+0000","last_became_active":"2024-03-23T22:59:27.190661+0000","last_became_peered":"2024-03-23T22:59:27.190661+0000","last_unstale":"2024-03-23T23:00:53.288651+0000","last_undegraded":"2024-03-23T23:00:53.288651+0000","last_fullsized":"2024-03-23T23:00:53.288651+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:29.822253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.366155+0000","last_change":"2024-03-23T22:59:27.187395+0000","last_active":"2024-03-23T23:00:24.366155+0000","last_peered":"2024-03-23T23:00:24.366155+0000","last_clean":"2024-03-23T23:00:24.366155+0000","last_became_active":"2024-03-23T22:59:27.186849+0000","last_became_peered":"2024-03-23T22:59:27.186849+0000","last_unstale":"2024-03-23T23:00:24.366155+0000","last_undegraded":"2024-03-23T23:00:24.366155+0000","last_fullsized":"2024-03-23T23:00:24.366155+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:05:48.047102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.286208+0000","last_change":"2024-03-23T22:59:27.190023+0000","last_active":"2024-03-23T23:00:53.286208+0000","last_peered":"2024-03-23T23:00:53.286208+0000","last_clean":"2024-03-23T23:00:53.286208+0000","last_became_active":"2024-03-23T22:59:27.189697+0000","last_became_peered":"2024-03-23T22:59:27.189697+0000","last_unstale":"2024-03-23T23:00:53.286208+0000","last_undegraded":"2024-03-23T23:00:53.286208+0000","last_fullsized":"2024-03-23T23:00:53.286208+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:59:51.121055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.264109+0000","last_change":"2024-03-23T22:59:27.187382+0000","last_active":"2024-03-23T23:00:53.264109+0000","last_peered":"2024-03-23T23:00:53.264109+0000","last_clean":"2024-03-23T23:00:53.264109+0000","last_became_active":"2024-03-23T22:59:27.186931+0000","last_became_peered":"2024-03-23T22:59:27.186931+0000","last_unstale":"2024-03-23T23:00:53.264109+0000","last_undegraded":"2024-03-23T23:00:53.264109+0000","last_fullsized":"2024-03-23T23:00:53.264109+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:33.583123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.293942+0000","last_change":"2024-03-23T22:59:27.202577+0000","last_active":"2024-03-23T23:00:53.293942+0000","last_peered":"2024-03-23T23:00:53.293942+0000","last_clean":"2024-03-23T23:00:53.293942+0000","last_became_active":"2024-03-23T22:59:27.202122+0000","last_became_peered":"2024-03-23T22:59:27.202122+0000","last_unstale":"2024-03-23T23:00:53.293942+0000","last_undegraded":"2024-03-23T23:00:53.293942+0000","last_fullsized":"2024-03-23T23:00:53.293942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:01:10.809635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.298358+0000","last_change":"2024-03-23T22:59:27.192915+0000","last_active":"2024-03-23T23:00:53.298358+0000","last_peered":"2024-03-23T23:00:53.298358+0000","last_clean":"2024-03-23T23:00:53.298358+0000","last_became_active":"2024-03-23T22:59:27.192107+0000","last_became_peered":"2024-03-23T22:59:27.192107+0000","last_unstale":"2024-03-23T23:00:53.298358+0000","last_undegraded":"2024-03-23T23:00:53.298358+0000","last_fullsized":"2024-03-23T23:00:53.298358+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:21:19.169211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.299609+0000","last_change":"2024-03-23T22:59:27.192156+0000","last_active":"2024-03-23T23:00:53.299609+0000","last_peered":"2024-03-23T23:00:53.299609+0000","last_clean":"2024-03-23T23:00:53.299609+0000","last_became_active":"2024-03-23T22:59:27.191594+0000","last_became_peered":"2024-03-23T22:59:27.191594+0000","last_unstale":"2024-03-23T23:00:53.299609+0000","last_undegraded":"2024-03-23T23:00:53.299609+0000","last_fullsized":"2024-03-23T23:00:53.299609+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:00:45.480281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.299719+0000","last_change":"2024-03-23T22:59:27.186048+0000","last_active":"2024-03-23T23:00:53.299719+0000","last_peered":"2024-03-23T23:00:53.299719+0000","last_clean":"2024-03-23T23:00:53.299719+0000","last_became_active":"2024-03-23T22:59:27.185710+0000","last_became_peered":"2024-03-23T22:59:27.185710+0000","last_unstale":"2024-03-23T23:00:53.299719+0000","last_undegraded":"2024-03-23T23:00:53.299719+0000","last_fullsized":"2024-03-23T23:00:53.299719+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:06.792665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.298373+0000","last_change":"2024-03-23T22:59:27.206290+0000","last_active":"2024-03-23T23:00:53.298373+0000","last_peered":"2024-03-23T23:00:53.298373+0000","last_clean":"2024-03-23T23:00:53.298373+0000","last_became_active":"2024-03-23T22:59:27.205997+0000","last_became_peered":"2024-03-23T22:59:27.205997+0000","last_unstale":"2024-03-23T23:00:53.298373+0000","last_undegraded":"2024-03-23T23:00:53.298373+0000","last_fullsized":"2024-03-23T23:00:53.298373+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:38:42.006166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.288923+0000","last_change":"2024-03-23T22:59:27.203671+0000","last_active":"2024-03-23T23:00:53.288923+0000","last_peered":"2024-03-23T23:00:53.288923+0000","last_clean":"2024-03-23T23:00:53.288923+0000","last_became_active":"2024-03-23T22:59:27.203273+0000","last_became_peered":"2024-03-23T22:59:27.203273+0000","last_unstale":"2024-03-23T23:00:53.288923+0000","last_undegraded":"2024-03-23T23:00:53.288923+0000","last_fullsized":"2024-03-23T23:00:53.288923+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:48:16.018671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.292273+0000","last_change":"2024-03-23T22:59:27.210063+0000","last_active":"2024-03-23T23:00:53.292273+0000","last_peered":"2024-03-23T23:00:53.292273+0000","last_clean":"2024-03-23T23:00:53.292273+0000","last_became_active":"2024-03-23T22:59:27.209698+0000","last_became_peered":"2024-03-23T22:59:27.209698+0000","last_unstale":"2024-03-23T23:00:53.292273+0000","last_undegraded":"2024-03-23T23:00:53.292273+0000","last_fullsized":"2024-03-23T23:00:53.292273+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:19:01.408901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.358003+0000","last_change":"2024-03-23T22:59:26.445053+0000","last_active":"2024-03-23T23:00:24.358003+0000","last_peered":"2024-03-23T23:00:24.358003+0000","last_clean":"2024-03-23T23:00:24.358003+0000","last_became_active":"2024-03-23T22:59:26.444660+0000","last_became_peered":"2024-03-23T22:59:26.444660+0000","last_unstale":"2024-03-23T23:00:24.358003+0000","last_undegraded":"2024-03-23T23:00:24.358003+0000","last_fullsized":"2024-03-23T23:00:24.358003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:19:16.958228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.367185+0000","last_change":"2024-03-23T22:59:27.208652+0000","last_active":"2024-03-23T23:00:24.367185+0000","last_peered":"2024-03-23T23:00:24.367185+0000","last_clean":"2024-03-23T23:00:24.367185+0000","last_became_active":"2024-03-23T22:59:27.207724+0000","last_became_peered":"2024-03-23T22:59:27.207724+0000","last_unstale":"2024-03-23T23:00:24.367185+0000","last_undegraded":"2024-03-23T23:00:24.367185+0000","last_fullsized":"2024-03-23T23:00:24.367185+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:08.986439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.368619+0000","last_change":"2024-03-23T22:59:26.445371+0000","last_active":"2024-03-23T23:00:24.368619+0000","last_peered":"2024-03-23T23:00:24.368619+0000","last_clean":"2024-03-23T23:00:24.368619+0000","last_became_active":"2024-03-23T22:59:26.445016+0000","last_became_peered":"2024-03-23T22:59:26.445016+0000","last_unstale":"2024-03-23T23:00:24.368619+0000","last_undegraded":"2024-03-23T23:00:24.368619+0000","last_fullsized":"2024-03-23T23:00:24.368619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:31:44.003218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.300409+0000","last_change":"2024-03-23T22:59:27.200624+0000","last_active":"2024-03-23T23:00:53.300409+0000","last_peered":"2024-03-23T23:00:53.300409+0000","last_clean":"2024-03-23T23:00:53.300409+0000","last_became_active":"2024-03-23T22:59:27.200207+0000","last_became_peered":"2024-03-23T22:59:27.200207+0000","last_unstale":"2024-03-23T23:00:53.300409+0000","last_undegraded":"2024-03-23T23:00:53.300409+0000","last_fullsized":"2024-03-23T23:00:53.300409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:05:31.015869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.380616+0000","last_change":"2024-03-23T23:00:24.380296+0000","last_active":"2024-03-23T23:00:24.380616+0000","last_peered":"2024-03-23T23:00:24.380616+0000","last_clean":"2024-03-23T23:00:24.380616+0000","last_became_active":"2024-03-23T23:00:24.374344+0000","last_became_peered":"2024-03-23T23:00:24.374344+0000","last_unstale":"2024-03-23T23:00:24.380616+0000","last_undegraded":"2024-03-23T23:00:24.380616+0000","last_fullsized":"2024-03-23T23:00:24.380616+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:30:20.174783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.288535+0000","last_change":"2024-03-23T22:59:27.204157+0000","last_active":"2024-03-23T23:00:53.288535+0000","last_peered":"2024-03-23T23:00:53.288535+0000","last_clean":"2024-03-23T23:00:53.288535+0000","last_became_active":"2024-03-23T22:59:27.203559+0000","last_became_peered":"2024-03-23T22:59:27.203559+0000","last_unstale":"2024-03-23T23:00:53.288535+0000","last_undegraded":"2024-03-23T23:00:53.288535+0000","last_fullsized":"2024-03-23T23:00:53.288535+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:43.834369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.361574+0000","last_change":"2024-03-23T23:00:24.361233+0000","last_active":"2024-03-23T23:00:24.361574+0000","last_peered":"2024-03-23T23:00:24.361574+0000","last_clean":"2024-03-23T23:00:24.361574+0000","last_became_active":"2024-03-23T23:00:24.358048+0000","last_became_peered":"2024-03-23T23:00:24.358048+0000","last_unstale":"2024-03-23T23:00:24.361574+0000","last_undegraded":"2024-03-23T23:00:24.361574+0000","last_fullsized":"2024-03-23T23:00:24.361574+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:38:37.792273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.286722+0000","last_change":"2024-03-23T22:59:27.186599+0000","last_active":"2024-03-23T23:00:53.286722+0000","last_peered":"2024-03-23T23:00:53.286722+0000","last_clean":"2024-03-23T23:00:53.286722+0000","last_became_active":"2024-03-23T22:59:27.186262+0000","last_became_peered":"2024-03-23T22:59:27.186262+0000","last_unstale":"2024-03-23T23:00:53.286722+0000","last_undegraded":"2024-03-23T23:00:53.286722+0000","last_fullsized":"2024-03-23T23:00:53.286722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:11:39.545558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":109,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.286626+0000","last_change":"2024-03-23T22:59:26.430096+0000","last_active":"2024-03-23T23:00:53.286626+0000","last_peered":"2024-03-23T23:00:53.286626+0000","last_clean":"2024-03-23T23:00:53.286626+0000","last_became_active":"2024-03-23T22:59:26.429740+0000","last_became_peered":"2024-03-23T22:59:26.429740+0000","last_unstale":"2024-03-23T23:00:53.286626+0000","last_undegraded":"2024-03-23T23:00:53.286626+0000","last_fullsized":"2024-03-23T23:00:53.286626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:54.831689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357358+0000","last_change":"2024-03-23T22:59:27.202750+0000","last_active":"2024-03-23T23:00:24.357358+0000","last_peered":"2024-03-23T23:00:24.357358+0000","last_clean":"2024-03-23T23:00:24.357358+0000","last_became_active":"2024-03-23T22:59:27.202391+0000","last_became_peered":"2024-03-23T22:59:27.202391+0000","last_unstale":"2024-03-23T23:00:24.357358+0000","last_undegraded":"2024-03-23T23:00:24.357358+0000","last_fullsized":"2024-03-23T23:00:24.357358+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:27:18.370792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.365876+0000","last_change":"2024-03-23T22:59:26.425607+0000","last_active":"2024-03-23T23:00:24.365876+0000","last_peered":"2024-03-23T23:00:24.365876+0000","last_clean":"2024-03-23T23:00:24.365876+0000","last_became_active":"2024-03-23T22:59:26.425208+0000","last_became_peered":"2024-03-23T22:59:26.425208+0000","last_unstale":"2024-03-23T23:00:24.365876+0000","last_undegraded":"2024-03-23T23:00:24.365876+0000","last_fullsized":"2024-03-23T23:00:24.365876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:51.071662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.282112+0000","last_change":"2024-03-23T22:59:27.215637+0000","last_active":"2024-03-23T23:00:53.282112+0000","last_peered":"2024-03-23T23:00:53.282112+0000","last_clean":"2024-03-23T23:00:53.282112+0000","last_became_active":"2024-03-23T22:59:27.215339+0000","last_became_peered":"2024-03-23T22:59:27.215339+0000","last_unstale":"2024-03-23T23:00:53.282112+0000","last_undegraded":"2024-03-23T23:00:53.282112+0000","last_fullsized":"2024-03-23T23:00:53.282112+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:25:42.097326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.377892+0000","last_change":"2024-03-23T22:59:26.442023+0000","last_active":"2024-03-23T23:00:24.377892+0000","last_peered":"2024-03-23T23:00:24.377892+0000","last_clean":"2024-03-23T23:00:24.377892+0000","last_became_active":"2024-03-23T22:59:26.441661+0000","last_became_peered":"2024-03-23T22:59:26.441661+0000","last_unstale":"2024-03-23T23:00:24.377892+0000","last_undegraded":"2024-03-23T23:00:24.377892+0000","last_fullsized":"2024-03-23T23:00:24.377892+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:59:14.914860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.279352+0000","last_change":"2024-03-23T23:00:24.384190+0000","last_active":"2024-03-23T23:00:53.279352+0000","last_peered":"2024-03-23T23:00:53.279352+0000","last_clean":"2024-03-23T23:00:53.279352+0000","last_became_active":"2024-03-23T23:00:24.382141+0000","last_became_peered":"2024-03-23T23:00:24.382141+0000","last_unstale":"2024-03-23T23:00:53.279352+0000","last_undegraded":"2024-03-23T23:00:53.279352+0000","last_fullsized":"2024-03-23T23:00:53.279352+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:48:52.311407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356353+0000","last_change":"2024-03-23T22:59:26.430147+0000","last_active":"2024-03-23T23:00:24.356353+0000","last_peered":"2024-03-23T23:00:24.356353+0000","last_clean":"2024-03-23T23:00:24.356353+0000","last_became_active":"2024-03-23T22:59:26.429843+0000","last_became_peered":"2024-03-23T22:59:26.429843+0000","last_unstale":"2024-03-23T23:00:24.356353+0000","last_undegraded":"2024-03-23T23:00:24.356353+0000","last_fullsized":"2024-03-23T23:00:24.356353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:06:45.691894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.288269+0000","last_change":"2024-03-23T22:59:27.212235+0000","last_active":"2024-03-23T23:00:53.288269+0000","last_peered":"2024-03-23T23:00:53.288269+0000","last_clean":"2024-03-23T23:00:53.288269+0000","last_became_active":"2024-03-23T22:59:27.211919+0000","last_became_peered":"2024-03-23T22:59:27.211919+0000","last_unstale":"2024-03-23T23:00:53.288269+0000","last_undegraded":"2024-03-23T23:00:53.288269+0000","last_fullsized":"2024-03-23T23:00:53.288269+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:24.752109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":103,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:51.283623+0000","last_change":"2024-03-23T22:59:26.440623+0000","last_active":"2024-03-23T23:00:51.283623+0000","last_peered":"2024-03-23T23:00:51.283623+0000","last_clean":"2024-03-23T23:00:51.283623+0000","last_became_active":"2024-03-23T22:59:26.440204+0000","last_became_peered":"2024-03-23T22:59:26.440204+0000","last_unstale":"2024-03-23T23:00:51.283623+0000","last_undegraded":"2024-03-23T23:00:51.283623+0000","last_fullsized":"2024-03-23T23:00:51.283623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:58:44.088552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":81,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.287377+0000","last_change":"2024-03-23T22:59:27.203916+0000","last_active":"2024-03-23T23:00:53.287377+0000","last_peered":"2024-03-23T23:00:53.287377+0000","last_clean":"2024-03-23T23:00:53.287377+0000","last_became_active":"2024-03-23T22:59:27.203565+0000","last_became_peered":"2024-03-23T22:59:27.203565+0000","last_unstale":"2024-03-23T23:00:53.287377+0000","last_undegraded":"2024-03-23T23:00:53.287377+0000","last_fullsized":"2024-03-23T23:00:53.287377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:06:39.635025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357536+0000","last_change":"2024-03-23T22:59:26.430080+0000","last_active":"2024-03-23T23:00:24.357536+0000","last_peered":"2024-03-23T23:00:24.357536+0000","last_clean":"2024-03-23T23:00:24.357536+0000","last_became_active":"2024-03-23T22:59:26.429618+0000","last_became_peered":"2024-03-23T22:59:26.429618+0000","last_unstale":"2024-03-23T23:00:24.357536+0000","last_undegraded":"2024-03-23T23:00:24.357536+0000","last_fullsized":"2024-03-23T23:00:24.357536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:01.963573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.300287+0000","last_change":"2024-03-23T22:59:27.190505+0000","last_active":"2024-03-23T23:00:53.300287+0000","last_peered":"2024-03-23T23:00:53.300287+0000","last_clean":"2024-03-23T23:00:53.300287+0000","last_became_active":"2024-03-23T22:59:27.190088+0000","last_became_peered":"2024-03-23T22:59:27.190088+0000","last_unstale":"2024-03-23T23:00:53.300287+0000","last_undegraded":"2024-03-23T23:00:53.300287+0000","last_fullsized":"2024-03-23T23:00:53.300287+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:29:04.255832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357315+0000","last_change":"2024-03-23T22:59:26.438329+0000","last_active":"2024-03-23T23:00:24.357315+0000","last_peered":"2024-03-23T23:00:24.357315+0000","last_clean":"2024-03-23T23:00:24.357315+0000","last_became_active":"2024-03-23T22:59:26.438033+0000","last_became_peered":"2024-03-23T22:59:26.438033+0000","last_unstale":"2024-03-23T23:00:24.357315+0000","last_undegraded":"2024-03-23T23:00:24.357315+0000","last_fullsized":"2024-03-23T23:00:24.357315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:02:38.789574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.287024+0000","last_change":"2024-03-23T22:59:27.188417+0000","last_active":"2024-03-23T23:00:53.287024+0000","last_peered":"2024-03-23T23:00:53.287024+0000","last_clean":"2024-03-23T23:00:53.287024+0000","last_became_active":"2024-03-23T22:59:27.188023+0000","last_became_peered":"2024-03-23T22:59:27.188023+0000","last_unstale":"2024-03-23T23:00:53.287024+0000","last_undegraded":"2024-03-23T23:00:53.287024+0000","last_fullsized":"2024-03-23T23:00:53.287024+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:19:11.014604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'27","reported_seq":216,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:52.287344+0000","last_change":"2024-03-23T22:59:26.425963+0000","last_active":"2024-03-23T23:00:52.287344+0000","last_peered":"2024-03-23T23:00:52.287344+0000","last_clean":"2024-03-23T23:00:52.287344+0000","last_became_active":"2024-03-23T22:59:26.425640+0000","last_became_peered":"2024-03-23T22:59:26.425640+0000","last_unstale":"2024-03-23T23:00:52.287344+0000","last_undegraded":"2024-03-23T23:00:52.287344+0000","last_fullsized":"2024-03-23T23:00:52.287344+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:11:26.529034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59465,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":170,"num_read_kb":142,"num_write":28,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356781+0000","last_change":"2024-03-23T22:59:27.187254+0000","last_active":"2024-03-23T23:00:24.356781+0000","last_peered":"2024-03-23T23:00:24.356781+0000","last_clean":"2024-03-23T23:00:24.356781+0000","last_became_active":"2024-03-23T22:59:27.186849+0000","last_became_peered":"2024-03-23T22:59:27.186849+0000","last_unstale":"2024-03-23T23:00:24.356781+0000","last_undegraded":"2024-03-23T23:00:24.356781+0000","last_fullsized":"2024-03-23T23:00:24.356781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:35:30.958565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.358432+0000","last_change":"2024-03-23T22:59:26.429767+0000","last_active":"2024-03-23T23:00:24.358432+0000","last_peered":"2024-03-23T23:00:24.358432+0000","last_clean":"2024-03-23T23:00:24.358432+0000","last_became_active":"2024-03-23T22:59:26.429449+0000","last_became_peered":"2024-03-23T22:59:26.429449+0000","last_unstale":"2024-03-23T23:00:24.358432+0000","last_undegraded":"2024-03-23T23:00:24.358432+0000","last_fullsized":"2024-03-23T23:00:24.358432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:46:13.752427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.298248+0000","last_change":"2024-03-23T22:59:27.223544+0000","last_active":"2024-03-23T23:00:53.298248+0000","last_peered":"2024-03-23T23:00:53.298248+0000","last_clean":"2024-03-23T23:00:53.298248+0000","last_became_active":"2024-03-23T22:59:27.223177+0000","last_became_peered":"2024-03-23T22:59:27.223177+0000","last_unstale":"2024-03-23T23:00:53.298248+0000","last_undegraded":"2024-03-23T23:00:53.298248+0000","last_fullsized":"2024-03-23T23:00:53.298248+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:33:18.521579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356513+0000","last_change":"2024-03-23T22:59:26.427781+0000","last_active":"2024-03-23T23:00:24.356513+0000","last_peered":"2024-03-23T23:00:24.356513+0000","last_clean":"2024-03-23T23:00:24.356513+0000","last_became_active":"2024-03-23T22:59:26.427458+0000","last_became_peered":"2024-03-23T22:59:26.427458+0000","last_unstale":"2024-03-23T23:00:24.356513+0000","last_undegraded":"2024-03-23T23:00:24.356513+0000","last_fullsized":"2024-03-23T23:00:24.356513+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:50:33.956611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.293397+0000","last_change":"2024-03-23T22:59:27.203681+0000","last_active":"2024-03-23T23:00:53.293397+0000","last_peered":"2024-03-23T23:00:53.293397+0000","last_clean":"2024-03-23T23:00:53.293397+0000","last_became_active":"2024-03-23T22:59:27.202457+0000","last_became_peered":"2024-03-23T22:59:27.202457+0000","last_unstale":"2024-03-23T23:00:53.293397+0000","last_undegraded":"2024-03-23T23:00:53.293397+0000","last_fullsized":"2024-03-23T23:00:53.293397+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:55:46.725350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'3","reported_seq":108,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:54.344476+0000","last_change":"2024-03-23T22:59:26.428517+0000","last_active":"2024-03-23T23:00:54.344476+0000","last_peered":"2024-03-23T23:00:54.344476+0000","last_clean":"2024-03-23T23:00:54.344476+0000","last_became_active":"2024-03-23T22:59:26.428130+0000","last_became_peered":"2024-03-23T22:59:26.428130+0000","last_unstale":"2024-03-23T23:00:54.344476+0000","last_undegraded":"2024-03-23T23:00:54.344476+0000","last_fullsized":"2024-03-23T23:00:54.344476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:01.185776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.282947+0000","last_change":"2024-03-23T23:00:24.367049+0000","last_active":"2024-03-23T23:00:53.282947+0000","last_peered":"2024-03-23T23:00:53.282947+0000","last_clean":"2024-03-23T23:00:53.282947+0000","last_became_active":"2024-03-23T23:00:24.366636+0000","last_became_peered":"2024-03-23T23:00:24.366636+0000","last_unstale":"2024-03-23T23:00:53.282947+0000","last_undegraded":"2024-03-23T23:00:53.282947+0000","last_fullsized":"2024-03-23T23:00:53.282947+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:07.068500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356895+0000","last_change":"2024-03-23T22:59:26.442637+0000","last_active":"2024-03-23T23:00:24.356895+0000","last_peered":"2024-03-23T23:00:24.356895+0000","last_clean":"2024-03-23T23:00:24.356895+0000","last_became_active":"2024-03-23T22:59:26.442267+0000","last_became_peered":"2024-03-23T22:59:26.442267+0000","last_unstale":"2024-03-23T23:00:24.356895+0000","last_undegraded":"2024-03-23T23:00:24.356895+0000","last_fullsized":"2024-03-23T23:00:24.356895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:49.975048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.286232+0000","last_change":"2024-03-23T22:59:27.187812+0000","last_active":"2024-03-23T23:00:53.286232+0000","last_peered":"2024-03-23T23:00:53.286232+0000","last_clean":"2024-03-23T23:00:53.286232+0000","last_became_active":"2024-03-23T22:59:27.187423+0000","last_became_peered":"2024-03-23T22:59:27.187423+0000","last_unstale":"2024-03-23T23:00:53.286232+0000","last_undegraded":"2024-03-23T23:00:53.286232+0000","last_fullsized":"2024-03-23T23:00:53.286232+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:12:16.427695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357307+0000","last_change":"2024-03-23T22:59:26.423875+0000","last_active":"2024-03-23T23:00:24.357307+0000","last_peered":"2024-03-23T23:00:24.357307+0000","last_clean":"2024-03-23T23:00:24.357307+0000","last_became_active":"2024-03-23T22:59:26.423513+0000","last_became_peered":"2024-03-23T22:59:26.423513+0000","last_unstale":"2024-03-23T23:00:24.357307+0000","last_undegraded":"2024-03-23T23:00:24.357307+0000","last_fullsized":"2024-03-23T23:00:24.357307+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:43.173365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.267669+0000","last_change":"2024-03-23T22:59:27.191454+0000","last_active":"2024-03-23T23:00:53.267669+0000","last_peered":"2024-03-23T23:00:53.267669+0000","last_clean":"2024-03-23T23:00:53.267669+0000","last_became_active":"2024-03-23T22:59:27.190927+0000","last_became_peered":"2024-03-23T22:59:27.190927+0000","last_unstale":"2024-03-23T23:00:53.267669+0000","last_undegraded":"2024-03-23T23:00:53.267669+0000","last_fullsized":"2024-03-23T23:00:53.267669+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:00.333702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.358466+0000","last_change":"2024-03-23T22:59:26.443238+0000","last_active":"2024-03-23T23:00:24.358466+0000","last_peered":"2024-03-23T23:00:24.358466+0000","last_clean":"2024-03-23T23:00:24.358466+0000","last_became_active":"2024-03-23T22:59:26.442834+0000","last_became_peered":"2024-03-23T22:59:26.442834+0000","last_unstale":"2024-03-23T23:00:24.358466+0000","last_undegraded":"2024-03-23T23:00:24.358466+0000","last_fullsized":"2024-03-23T23:00:24.358466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:11:21.072836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.365898+0000","last_change":"2024-03-23T22:59:27.212864+0000","last_active":"2024-03-23T23:00:24.365898+0000","last_peered":"2024-03-23T23:00:24.365898+0000","last_clean":"2024-03-23T23:00:24.365898+0000","last_became_active":"2024-03-23T22:59:27.212455+0000","last_became_peered":"2024-03-23T22:59:27.212455+0000","last_unstale":"2024-03-23T23:00:24.365898+0000","last_undegraded":"2024-03-23T23:00:24.365898+0000","last_fullsized":"2024-03-23T23:00:24.365898+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:25:59.122531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357057+0000","last_change":"2024-03-23T22:59:26.432838+0000","last_active":"2024-03-23T23:00:24.357057+0000","last_peered":"2024-03-23T23:00:24.357057+0000","last_clean":"2024-03-23T23:00:24.357057+0000","last_became_active":"2024-03-23T22:59:26.432544+0000","last_became_peered":"2024-03-23T22:59:26.432544+0000","last_unstale":"2024-03-23T23:00:24.357057+0000","last_undegraded":"2024-03-23T23:00:24.357057+0000","last_fullsized":"2024-03-23T23:00:24.357057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:32:52.121569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.376926+0000","last_change":"2024-03-23T23:00:24.376926+0000","last_active":"2024-03-23T23:00:24.376926+0000","last_peered":"2024-03-23T23:00:24.376926+0000","last_clean":"2024-03-23T23:00:24.376926+0000","last_became_active":"2024-03-23T23:00:24.376515+0000","last_became_peered":"2024-03-23T23:00:24.376515+0000","last_unstale":"2024-03-23T23:00:24.376926+0000","last_undegraded":"2024-03-23T23:00:24.376926+0000","last_fullsized":"2024-03-23T23:00:24.376926+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:55:57.730564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.278207+0000","last_change":"2024-03-23T23:00:24.381861+0000","last_active":"2024-03-23T23:00:53.278207+0000","last_peered":"2024-03-23T23:00:53.278207+0000","last_clean":"2024-03-23T23:00:53.278207+0000","last_became_active":"2024-03-23T23:00:24.381472+0000","last_became_peered":"2024-03-23T23:00:24.381472+0000","last_unstale":"2024-03-23T23:00:53.278207+0000","last_undegraded":"2024-03-23T23:00:53.278207+0000","last_fullsized":"2024-03-23T23:00:53.278207+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:44:07.518911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":106,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.342484+0000","last_change":"2024-03-23T22:59:26.450679+0000","last_active":"2024-03-23T23:00:53.342484+0000","last_peered":"2024-03-23T23:00:53.342484+0000","last_clean":"2024-03-23T23:00:53.342484+0000","last_became_active":"2024-03-23T22:59:26.450234+0000","last_became_peered":"2024-03-23T22:59:26.450234+0000","last_unstale":"2024-03-23T23:00:53.342484+0000","last_undegraded":"2024-03-23T23:00:53.342484+0000","last_fullsized":"2024-03-23T23:00:53.342484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:52:25.869194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":83,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.364904+0000","last_change":"2024-03-23T22:59:27.220171+0000","last_active":"2024-03-23T23:00:24.364904+0000","last_peered":"2024-03-23T23:00:24.364904+0000","last_clean":"2024-03-23T23:00:24.364904+0000","last_became_active":"2024-03-23T22:59:27.219787+0000","last_became_peered":"2024-03-23T22:59:27.219787+0000","last_unstale":"2024-03-23T23:00:24.364904+0000","last_undegraded":"2024-03-23T23:00:24.364904+0000","last_fullsized":"2024-03-23T23:00:24.364904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:51:52.634264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.373077+0000","last_change":"2024-03-23T22:59:26.440205+0000","last_active":"2024-03-23T23:00:24.373077+0000","last_peered":"2024-03-23T23:00:24.373077+0000","last_clean":"2024-03-23T23:00:24.373077+0000","last_became_active":"2024-03-23T22:59:26.439752+0000","last_became_peered":"2024-03-23T22:59:26.439752+0000","last_unstale":"2024-03-23T23:00:24.373077+0000","last_undegraded":"2024-03-23T23:00:24.373077+0000","last_fullsized":"2024-03-23T23:00:24.373077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:55:25.661887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":345,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356800+0000","last_change":"2024-03-23T22:58:06.171317+0000","last_active":"2024-03-23T23:00:24.356800+0000","last_peered":"2024-03-23T23:00:24.356800+0000","last_clean":"2024-03-23T23:00:24.356800+0000","last_became_active":"2024-03-23T22:58:06.170816+0000","last_became_peered":"2024-03-23T22:58:06.170816+0000","last_unstale":"2024-03-23T23:00:24.356800+0000","last_undegraded":"2024-03-23T23:00:24.356800+0000","last_fullsized":"2024-03-23T23:00:24.356800+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:54:11.991284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:54:11.991284+0000","last_clean_scrub_stamp":"2024-03-23T22:54:11.991284+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:20:55.980613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.283499+0000","last_change":"2024-03-23T22:59:27.200348+0000","last_active":"2024-03-23T23:00:53.283499+0000","last_peered":"2024-03-23T23:00:53.283499+0000","last_clean":"2024-03-23T23:00:53.283499+0000","last_became_active":"2024-03-23T22:59:27.199979+0000","last_became_peered":"2024-03-23T22:59:27.199979+0000","last_unstale":"2024-03-23T23:00:53.283499+0000","last_undegraded":"2024-03-23T23:00:53.283499+0000","last_fullsized":"2024-03-23T23:00:53.283499+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:31:19.258890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356890+0000","last_change":"2024-03-23T22:59:26.431837+0000","last_active":"2024-03-23T23:00:24.356890+0000","last_peered":"2024-03-23T23:00:24.356890+0000","last_clean":"2024-03-23T23:00:24.356890+0000","last_became_active":"2024-03-23T22:59:26.431580+0000","last_became_peered":"2024-03-23T22:59:26.431580+0000","last_unstale":"2024-03-23T23:00:24.356890+0000","last_undegraded":"2024-03-23T23:00:24.356890+0000","last_fullsized":"2024-03-23T23:00:24.356890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:49:53.316952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356883+0000","last_change":"2024-03-23T22:59:26.442878+0000","last_active":"2024-03-23T23:00:24.356883+0000","last_peered":"2024-03-23T23:00:24.356883+0000","last_clean":"2024-03-23T23:00:24.356883+0000","last_became_active":"2024-03-23T22:59:26.442303+0000","last_became_peered":"2024-03-23T22:59:26.442303+0000","last_unstale":"2024-03-23T23:00:24.356883+0000","last_undegraded":"2024-03-23T23:00:24.356883+0000","last_fullsized":"2024-03-23T23:00:24.356883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:51.717553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.279507+0000","last_change":"2024-03-23T22:59:27.212550+0000","last_active":"2024-03-23T23:00:53.279507+0000","last_peered":"2024-03-23T23:00:53.279507+0000","last_clean":"2024-03-23T23:00:53.279507+0000","last_became_active":"2024-03-23T22:59:27.212105+0000","last_became_peered":"2024-03-23T22:59:27.212105+0000","last_unstale":"2024-03-23T23:00:53.279507+0000","last_undegraded":"2024-03-23T23:00:53.279507+0000","last_fullsized":"2024-03-23T23:00:53.279507+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:23:37.374081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.359065+0000","last_change":"2024-03-23T22:59:26.452146+0000","last_active":"2024-03-23T23:00:24.359065+0000","last_peered":"2024-03-23T23:00:24.359065+0000","last_clean":"2024-03-23T23:00:24.359065+0000","last_became_active":"2024-03-23T22:59:26.450398+0000","last_became_peered":"2024-03-23T22:59:26.450398+0000","last_unstale":"2024-03-23T23:00:24.359065+0000","last_undegraded":"2024-03-23T23:00:24.359065+0000","last_fullsized":"2024-03-23T23:00:24.359065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:55:47.085457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.366547+0000","last_change":"2024-03-23T22:59:27.209690+0000","last_active":"2024-03-23T23:00:24.366547+0000","last_peered":"2024-03-23T23:00:24.366547+0000","last_clean":"2024-03-23T23:00:24.366547+0000","last_became_active":"2024-03-23T22:59:27.209187+0000","last_became_peered":"2024-03-23T22:59:27.209187+0000","last_unstale":"2024-03-23T23:00:24.366547+0000","last_undegraded":"2024-03-23T23:00:24.366547+0000","last_fullsized":"2024-03-23T23:00:24.366547+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:19:45.315459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.373690+0000","last_change":"2024-03-23T22:59:26.434958+0000","last_active":"2024-03-23T23:00:24.373690+0000","last_peered":"2024-03-23T23:00:24.373690+0000","last_clean":"2024-03-23T23:00:24.373690+0000","last_became_active":"2024-03-23T22:59:26.434611+0000","last_became_peered":"2024-03-23T22:59:26.434611+0000","last_unstale":"2024-03-23T23:00:24.373690+0000","last_undegraded":"2024-03-23T23:00:24.373690+0000","last_fullsized":"2024-03-23T23:00:24.373690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:42:18.201885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.300005+0000","last_change":"2024-03-23T22:59:27.211700+0000","last_active":"2024-03-23T23:00:53.300005+0000","last_peered":"2024-03-23T23:00:53.300005+0000","last_clean":"2024-03-23T23:00:53.300005+0000","last_became_active":"2024-03-23T22:59:27.211248+0000","last_became_peered":"2024-03-23T22:59:27.211248+0000","last_unstale":"2024-03-23T23:00:53.300005+0000","last_undegraded":"2024-03-23T23:00:53.300005+0000","last_fullsized":"2024-03-23T23:00:53.300005+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:49.868397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356020+0000","last_change":"2024-03-23T22:59:26.442625+0000","last_active":"2024-03-23T23:00:24.356020+0000","last_peered":"2024-03-23T23:00:24.356020+0000","last_clean":"2024-03-23T23:00:24.356020+0000","last_became_active":"2024-03-23T22:59:26.442312+0000","last_became_peered":"2024-03-23T22:59:26.442312+0000","last_unstale":"2024-03-23T23:00:24.356020+0000","last_undegraded":"2024-03-23T23:00:24.356020+0000","last_fullsized":"2024-03-23T23:00:24.356020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:27:38.808784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356155+0000","last_change":"2024-03-23T22:59:27.187363+0000","last_active":"2024-03-23T23:00:24.356155+0000","last_peered":"2024-03-23T23:00:24.356155+0000","last_clean":"2024-03-23T23:00:24.356155+0000","last_became_active":"2024-03-23T22:59:27.187007+0000","last_became_peered":"2024-03-23T22:59:27.187007+0000","last_unstale":"2024-03-23T23:00:24.356155+0000","last_undegraded":"2024-03-23T23:00:24.356155+0000","last_fullsized":"2024-03-23T23:00:24.356155+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:41:37.041254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.369396+0000","last_change":"2024-03-23T22:59:26.445753+0000","last_active":"2024-03-23T23:00:24.369396+0000","last_peered":"2024-03-23T23:00:24.369396+0000","last_clean":"2024-03-23T23:00:24.369396+0000","last_became_active":"2024-03-23T22:59:26.445167+0000","last_became_peered":"2024-03-23T22:59:26.445167+0000","last_unstale":"2024-03-23T23:00:24.369396+0000","last_undegraded":"2024-03-23T23:00:24.369396+0000","last_fullsized":"2024-03-23T23:00:24.369396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:46:46.058523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.281814+0000","last_change":"2024-03-23T22:59:27.203855+0000","last_active":"2024-03-23T23:00:53.281814+0000","last_peered":"2024-03-23T23:00:53.281814+0000","last_clean":"2024-03-23T23:00:53.281814+0000","last_became_active":"2024-03-23T22:59:27.203438+0000","last_became_peered":"2024-03-23T22:59:27.203438+0000","last_unstale":"2024-03-23T23:00:53.281814+0000","last_undegraded":"2024-03-23T23:00:53.281814+0000","last_fullsized":"2024-03-23T23:00:53.281814+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:28.725743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.358167+0000","last_change":"2024-03-23T22:59:26.439002+0000","last_active":"2024-03-23T23:00:24.358167+0000","last_peered":"2024-03-23T23:00:24.358167+0000","last_clean":"2024-03-23T23:00:24.358167+0000","last_became_active":"2024-03-23T22:59:26.438634+0000","last_became_peered":"2024-03-23T22:59:26.438634+0000","last_unstale":"2024-03-23T23:00:24.358167+0000","last_undegraded":"2024-03-23T23:00:24.358167+0000","last_fullsized":"2024-03-23T23:00:24.358167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:25:02.988163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.365706+0000","last_change":"2024-03-23T22:59:27.217314+0000","last_active":"2024-03-23T23:00:24.365706+0000","last_peered":"2024-03-23T23:00:24.365706+0000","last_clean":"2024-03-23T23:00:24.365706+0000","last_became_active":"2024-03-23T22:59:27.216790+0000","last_became_peered":"2024-03-23T22:59:27.216790+0000","last_unstale":"2024-03-23T23:00:24.365706+0000","last_undegraded":"2024-03-23T23:00:24.365706+0000","last_fullsized":"2024-03-23T23:00:24.365706+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:47:13.902236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357102+0000","last_change":"2024-03-23T22:59:26.431453+0000","last_active":"2024-03-23T23:00:24.357102+0000","last_peered":"2024-03-23T23:00:24.357102+0000","last_clean":"2024-03-23T23:00:24.357102+0000","last_became_active":"2024-03-23T22:59:26.429731+0000","last_became_peered":"2024-03-23T22:59:26.429731+0000","last_unstale":"2024-03-23T23:00:24.357102+0000","last_undegraded":"2024-03-23T23:00:24.357102+0000","last_fullsized":"2024-03-23T23:00:24.357102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:33:47.717064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.295328+0000","last_change":"2024-03-23T22:59:27.203097+0000","last_active":"2024-03-23T23:00:53.295328+0000","last_peered":"2024-03-23T23:00:53.295328+0000","last_clean":"2024-03-23T23:00:53.295328+0000","last_became_active":"2024-03-23T22:59:27.202484+0000","last_became_peered":"2024-03-23T22:59:27.202484+0000","last_unstale":"2024-03-23T23:00:53.295328+0000","last_undegraded":"2024-03-23T23:00:53.295328+0000","last_fullsized":"2024-03-23T23:00:53.295328+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:07.829866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.377539+0000","last_change":"2024-03-23T23:00:24.377180+0000","last_active":"2024-03-23T23:00:24.377539+0000","last_peered":"2024-03-23T23:00:24.377539+0000","last_clean":"2024-03-23T23:00:24.377539+0000","last_became_active":"2024-03-23T23:00:24.364530+0000","last_became_peered":"2024-03-23T23:00:24.364530+0000","last_unstale":"2024-03-23T23:00:24.377539+0000","last_undegraded":"2024-03-23T23:00:24.377539+0000","last_fullsized":"2024-03-23T23:00:24.377539+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:57.507891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.286396+0000","last_change":"2024-03-23T22:59:27.189244+0000","last_active":"2024-03-23T23:00:53.286396+0000","last_peered":"2024-03-23T23:00:53.286396+0000","last_clean":"2024-03-23T23:00:53.286396+0000","last_became_active":"2024-03-23T22:59:27.188629+0000","last_became_peered":"2024-03-23T22:59:27.188629+0000","last_unstale":"2024-03-23T23:00:53.286396+0000","last_undegraded":"2024-03-23T23:00:53.286396+0000","last_fullsized":"2024-03-23T23:00:53.286396+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:47:26.742204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356440+0000","last_change":"2024-03-23T22:59:26.434326+0000","last_active":"2024-03-23T23:00:24.356440+0000","last_peered":"2024-03-23T23:00:24.356440+0000","last_clean":"2024-03-23T23:00:24.356440+0000","last_became_active":"2024-03-23T22:59:26.433878+0000","last_became_peered":"2024-03-23T22:59:26.433878+0000","last_unstale":"2024-03-23T23:00:24.356440+0000","last_undegraded":"2024-03-23T23:00:24.356440+0000","last_fullsized":"2024-03-23T23:00:24.356440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:08.580572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.297012+0000","last_change":"2024-03-23T22:59:27.212909+0000","last_active":"2024-03-23T23:00:53.297012+0000","last_peered":"2024-03-23T23:00:53.297012+0000","last_clean":"2024-03-23T23:00:53.297012+0000","last_became_active":"2024-03-23T22:59:27.212504+0000","last_became_peered":"2024-03-23T22:59:27.212504+0000","last_unstale":"2024-03-23T23:00:53.297012+0000","last_undegraded":"2024-03-23T23:00:53.297012+0000","last_fullsized":"2024-03-23T23:00:53.297012+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:09.238272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.380222+0000","last_change":"2024-03-23T22:59:26.452577+0000","last_active":"2024-03-23T23:00:24.380222+0000","last_peered":"2024-03-23T23:00:24.380222+0000","last_clean":"2024-03-23T23:00:24.380222+0000","last_became_active":"2024-03-23T22:59:26.452107+0000","last_became_peered":"2024-03-23T22:59:26.452107+0000","last_unstale":"2024-03-23T23:00:24.380222+0000","last_undegraded":"2024-03-23T23:00:24.380222+0000","last_fullsized":"2024-03-23T23:00:24.380222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:32:18.472334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.283757+0000","last_change":"2024-03-23T22:59:27.188539+0000","last_active":"2024-03-23T23:00:53.283757+0000","last_peered":"2024-03-23T23:00:53.283757+0000","last_clean":"2024-03-23T23:00:53.283757+0000","last_became_active":"2024-03-23T22:59:27.188184+0000","last_became_peered":"2024-03-23T22:59:27.188184+0000","last_unstale":"2024-03-23T23:00:53.283757+0000","last_undegraded":"2024-03-23T23:00:53.283757+0000","last_fullsized":"2024-03-23T23:00:53.283757+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:37:59.756492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.268720+0000","last_change":"2024-03-23T22:59:27.192083+0000","last_active":"2024-03-23T23:00:53.268720+0000","last_peered":"2024-03-23T23:00:53.268720+0000","last_clean":"2024-03-23T23:00:53.268720+0000","last_became_active":"2024-03-23T22:59:27.191709+0000","last_became_peered":"2024-03-23T22:59:27.191709+0000","last_unstale":"2024-03-23T23:00:53.268720+0000","last_undegraded":"2024-03-23T23:00:53.268720+0000","last_fullsized":"2024-03-23T23:00:53.268720+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:18:42.905177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.274891+0000","last_change":"2024-03-23T22:59:27.186315+0000","last_active":"2024-03-23T23:00:53.274891+0000","last_peered":"2024-03-23T23:00:53.274891+0000","last_clean":"2024-03-23T23:00:53.274891+0000","last_became_active":"2024-03-23T22:59:27.185960+0000","last_became_peered":"2024-03-23T22:59:27.185960+0000","last_unstale":"2024-03-23T23:00:53.274891+0000","last_undegraded":"2024-03-23T23:00:53.274891+0000","last_fullsized":"2024-03-23T23:00:53.274891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:57:07.739725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.291772+0000","last_change":"2024-03-23T22:59:27.221696+0000","last_active":"2024-03-23T23:00:53.291772+0000","last_peered":"2024-03-23T23:00:53.291772+0000","last_clean":"2024-03-23T23:00:53.291772+0000","last_became_active":"2024-03-23T22:59:27.221400+0000","last_became_peered":"2024-03-23T22:59:27.221400+0000","last_unstale":"2024-03-23T23:00:53.291772+0000","last_undegraded":"2024-03-23T23:00:53.291772+0000","last_fullsized":"2024-03-23T23:00:53.291772+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:17.905737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.283639+0000","last_change":"2024-03-23T22:59:27.206286+0000","last_active":"2024-03-23T23:00:53.283639+0000","last_peered":"2024-03-23T23:00:53.283639+0000","last_clean":"2024-03-23T23:00:53.283639+0000","last_became_active":"2024-03-23T22:59:27.205854+0000","last_became_peered":"2024-03-23T22:59:27.205854+0000","last_unstale":"2024-03-23T23:00:53.283639+0000","last_undegraded":"2024-03-23T23:00:53.283639+0000","last_fullsized":"2024-03-23T23:00:53.283639+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:00:45.216211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.299572+0000","last_change":"2024-03-23T22:59:27.211376+0000","last_active":"2024-03-23T23:00:53.299572+0000","last_peered":"2024-03-23T23:00:53.299572+0000","last_clean":"2024-03-23T23:00:53.299572+0000","last_became_active":"2024-03-23T22:59:27.211061+0000","last_became_peered":"2024-03-23T22:59:27.211061+0000","last_unstale":"2024-03-23T23:00:53.299572+0000","last_undegraded":"2024-03-23T23:00:53.299572+0000","last_fullsized":"2024-03-23T23:00:53.299572+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:53:25.776927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.267891+0000","last_change":"2024-03-23T22:59:27.191028+0000","last_active":"2024-03-23T23:00:53.267891+0000","last_peered":"2024-03-23T23:00:53.267891+0000","last_clean":"2024-03-23T23:00:53.267891+0000","last_became_active":"2024-03-23T22:59:27.190634+0000","last_became_peered":"2024-03-23T22:59:27.190634+0000","last_unstale":"2024-03-23T23:00:53.267891+0000","last_undegraded":"2024-03-23T23:00:53.267891+0000","last_fullsized":"2024-03-23T23:00:53.267891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:50:52.110636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.289393+0000","last_change":"2024-03-23T22:59:27.183235+0000","last_active":"2024-03-23T23:00:53.289393+0000","last_peered":"2024-03-23T23:00:53.289393+0000","last_clean":"2024-03-23T23:00:53.289393+0000","last_became_active":"2024-03-23T22:59:27.181682+0000","last_became_peered":"2024-03-23T22:59:27.181682+0000","last_unstale":"2024-03-23T23:00:53.289393+0000","last_undegraded":"2024-03-23T23:00:53.289393+0000","last_fullsized":"2024-03-23T23:00:53.289393+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:33:26.002324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.378143+0000","last_change":"2024-03-23T23:00:24.378143+0000","last_active":"2024-03-23T23:00:24.378143+0000","last_peered":"2024-03-23T23:00:24.378143+0000","last_clean":"2024-03-23T23:00:24.378143+0000","last_became_active":"2024-03-23T23:00:24.377660+0000","last_became_peered":"2024-03-23T23:00:24.377660+0000","last_unstale":"2024-03-23T23:00:24.378143+0000","last_undegraded":"2024-03-23T23:00:24.378143+0000","last_fullsized":"2024-03-23T23:00:24.378143+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:24:29.250679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.284352+0000","last_change":"2024-03-23T22:59:27.215230+0000","last_active":"2024-03-23T23:00:53.284352+0000","last_peered":"2024-03-23T23:00:53.284352+0000","last_clean":"2024-03-23T23:00:53.284352+0000","last_became_active":"2024-03-23T22:59:27.214881+0000","last_became_peered":"2024-03-23T22:59:27.214881+0000","last_unstale":"2024-03-23T23:00:53.284352+0000","last_undegraded":"2024-03-23T23:00:53.284352+0000","last_fullsized":"2024-03-23T23:00:53.284352+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:10:40.428481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.365535+0000","last_change":"2024-03-23T22:59:27.213454+0000","last_active":"2024-03-23T23:00:24.365535+0000","last_peered":"2024-03-23T23:00:24.365535+0000","last_clean":"2024-03-23T23:00:24.365535+0000","last_became_active":"2024-03-23T22:59:27.212684+0000","last_became_peered":"2024-03-23T22:59:27.212684+0000","last_unstale":"2024-03-23T23:00:24.365535+0000","last_undegraded":"2024-03-23T23:00:24.365535+0000","last_fullsized":"2024-03-23T23:00:24.365535+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:48:51.584892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.287040+0000","last_change":"2024-03-23T22:59:27.220414+0000","last_active":"2024-03-23T23:00:53.287040+0000","last_peered":"2024-03-23T23:00:53.287040+0000","last_clean":"2024-03-23T23:00:53.287040+0000","last_became_active":"2024-03-23T22:59:27.220020+0000","last_became_peered":"2024-03-23T22:59:27.220020+0000","last_unstale":"2024-03-23T23:00:53.287040+0000","last_undegraded":"2024-03-23T23:00:53.287040+0000","last_fullsized":"2024-03-23T23:00:53.287040+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:35:48.890004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.287090+0000","last_change":"2024-03-23T22:59:27.215648+0000","last_active":"2024-03-23T23:00:53.287090+0000","last_peered":"2024-03-23T23:00:53.287090+0000","last_clean":"2024-03-23T23:00:53.287090+0000","last_became_active":"2024-03-23T22:59:27.212252+0000","last_became_peered":"2024-03-23T22:59:27.212252+0000","last_unstale":"2024-03-23T23:00:53.287090+0000","last_undegraded":"2024-03-23T23:00:53.287090+0000","last_fullsized":"2024-03-23T23:00:53.287090+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:08:38.956508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.274545+0000","last_change":"2024-03-23T22:59:27.190527+0000","last_active":"2024-03-23T23:00:53.274545+0000","last_peered":"2024-03-23T23:00:53.274545+0000","last_clean":"2024-03-23T23:00:53.274545+0000","last_became_active":"2024-03-23T22:59:27.190026+0000","last_became_peered":"2024-03-23T22:59:27.190026+0000","last_unstale":"2024-03-23T23:00:53.274545+0000","last_undegraded":"2024-03-23T23:00:53.274545+0000","last_fullsized":"2024-03-23T23:00:53.274545+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:31:24.663584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":405,"ondisk_log_size":405,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247269,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63075,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":518,"num_read_kb":323,"num_write":158,"num_write_kb":106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":189723,"data_compressed":5619,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351008,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":459288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,2,11,12],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612644,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":1110234,"data_compressed":37666,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776171,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":459524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037809,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":459444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299446,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":1050983,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462972,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":954368,"data_stored":458913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":519002,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":460184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":519188,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149773,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":459224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888135,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":459350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724610,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":1050667,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34152,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34152,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":60296,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":60482,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":61043,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T23:00:58.554 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-23T23:00:58.554 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-23T23:00:58.554 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-23T23:00:58.554 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-23T23:00:58.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:58 smithi203 ceph-mon[36693]: pgmap v365: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-23T23:00:58.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:00:58 smithi203 ceph-mon[36693]: from='client.24914 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T23:00:59.072 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:01:00.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:00 smithi012 ceph-mon[31695]: pgmap v366: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:01:00.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:00 smithi173 ceph-mon[36116]: pgmap v366: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:01:00.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:00 smithi203 ceph-mon[36693]: pgmap v366: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:01:01.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.316+0000 7f90ae600700 1 -- 172.21.15.12:0/7117659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90a81096b0 msgr2=0x7f90a8111280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:01.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.316+0000 7f90ae600700 1 --2- 172.21.15.12:0/7117659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90a81096b0 0x7f90a8111280 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f90a000b0d0 tx=0x7f90a000b3e0 comp rx=0 tx=0).stop 2024-03-23T23:01:01.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.316+0000 7f90ae600700 1 -- 172.21.15.12:0/7117659 shutdown_connections 2024-03-23T23:01:01.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.316+0000 7f90ae600700 1 --2- 172.21.15.12:0/7117659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90a81096b0 0x7f90a8111280 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:01.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.316+0000 7f90ae600700 1 --2- 172.21.15.12:0/7117659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f90a81045d0 0x7f90a81049b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:01.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.316+0000 7f90ae600700 1 --2- 172.21.15.12:0/7117659 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90a8104f80 0x7f90a8108fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:01.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.316+0000 7f90ae600700 1 -- 172.21.15.12:0/7117659 >> 172.21.15.12:0/7117659 conn(0x7f90a80ffea0 msgr2=0x7f90a81022c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:01.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.316+0000 7f90ae600700 1 -- 172.21.15.12:0/7117659 shutdown_connections 2024-03-23T23:01:01.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.316+0000 7f90ae600700 1 -- 172.21.15.12:0/7117659 wait complete. 2024-03-23T23:01:01.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.317+0000 7f90ae600700 1 Processor -- start 2024-03-23T23:01:01.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.317+0000 7f90ae600700 1 -- start start 2024-03-23T23:01:01.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.318+0000 7f90ae600700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f90a81045d0 0x7f90a81a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:01.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.318+0000 7f90ae600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90a8104f80 0x7f90a81a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:01.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.318+0000 7f90ae600700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90a81096b0 0x7f90a819a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:01.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.318+0000 7f90ae600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a81146f0 con 0x7f90a8104f80 2024-03-23T23:01:01.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.318+0000 7f90ae600700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a8114570 con 0x7f90a81045d0 2024-03-23T23:01:01.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.318+0000 7f90ae600700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a8114870 con 0x7f90a81096b0 2024-03-23T23:01:01.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.318+0000 7f90a7400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f90a81045d0 0x7f90a81a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:01.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.318+0000 7f90a7400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f90a81045d0 0x7f90a81a0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:41464/0 (socket says 172.21.15.12:41464) 2024-03-23T23:01:01.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.318+0000 7f909fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90a8104f80 0x7f90a81a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:01.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.318+0000 7f90a7400700 1 -- 172.21.15.12:0/952397977 learned_addr learned my addr 172.21.15.12:0/952397977 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:01.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.318+0000 7f90a7e00700 1 --2- 172.21.15.12:0/952397977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90a81096b0 0x7f90a819a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:01.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.318+0000 7f90a7400700 1 -- 172.21.15.12:0/952397977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90a81096b0 msgr2=0x7f90a819a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:01.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.318+0000 7f90a7400700 1 --2- 172.21.15.12:0/952397977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90a81096b0 0x7f90a819a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:01.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.318+0000 7f90a7400700 1 -- 172.21.15.12:0/952397977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90a8104f80 msgr2=0x7f90a81a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:01.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.318+0000 7f90a7400700 1 --2- 172.21.15.12:0/952397977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90a8104f80 0x7f90a81a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:01.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.318+0000 7f90a7400700 1 -- 172.21.15.12:0/952397977 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a819b030 con 0x7f90a81045d0 2024-03-23T23:01:01.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.318+0000 7f90a7e00700 1 --2- 172.21.15.12:0/952397977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90a81096b0 0x7f90a819a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:01.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.319+0000 7f90a7400700 1 --2- 172.21.15.12:0/952397977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f90a81045d0 0x7f90a81a0710 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f909800eb30 tx=0x7f909800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:01.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.319+0000 7f90a4c00700 1 -- 172.21.15.12:0/952397977 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909800cbd0 con 0x7f90a81045d0 2024-03-23T23:01:01.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.319+0000 7f90a4c00700 1 -- 172.21.15.12:0/952397977 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9098004500 con 0x7f90a81045d0 2024-03-23T23:01:01.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.319+0000 7f90ae600700 1 -- 172.21.15.12:0/952397977 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a819b320 con 0x7f90a81045d0 2024-03-23T23:01:01.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.319+0000 7f90a4c00700 1 -- 172.21.15.12:0/952397977 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9098010430 con 0x7f90a81045d0 2024-03-23T23:01:01.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.319+0000 7f90ae600700 1 -- 172.21.15.12:0/952397977 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a81a75e0 con 0x7f90a81045d0 2024-03-23T23:01:01.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.320+0000 7f90ae600700 1 -- 172.21.15.12:0/952397977 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a8068e50 con 0x7f90a81045d0 2024-03-23T23:01:01.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.322+0000 7f90a4c00700 1 -- 172.21.15.12:0/952397977 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9098003f00 con 0x7f90a81045d0 2024-03-23T23:01:01.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.323+0000 7f90a4c00700 1 --2- 172.21.15.12:0/952397977 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f908c0a2890 0x7f908c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:01.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.323+0000 7f90a4c00700 1 -- 172.21.15.12:0/952397977 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f90980cede0 con 0x7f90a81045d0 2024-03-23T23:01:01.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.324+0000 7f909fe00700 1 --2- 172.21.15.12:0/952397977 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f908c0a2890 0x7f908c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:01.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.324+0000 7f909fe00700 1 --2- 172.21.15.12:0/952397977 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f908c0a2890 0x7f908c0a4d50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f90a8103580 tx=0x7f9090009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:01.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.331+0000 7f90a4c00700 1 -- 172.21.15.12:0/952397977 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f909809be30 con 0x7f90a81045d0 2024-03-23T23:01:01.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.605+0000 7f90ae600700 1 -- 172.21.15.12:0/952397977 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f90a81a7870 con 0x7f90a81045d0 2024-03-23T23:01:01.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.605+0000 7f90a4c00700 1 -- 172.21.15.12:0/952397977 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f909809b730 con 0x7f90a81045d0 2024-03-23T23:01:01.605 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:01:01.605 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-23T23:01:01.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.608+0000 7f90ae600700 1 -- 172.21.15.12:0/952397977 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f908c0a2890 msgr2=0x7f908c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:01.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.608+0000 7f90ae600700 1 --2- 172.21.15.12:0/952397977 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f908c0a2890 0x7f908c0a4d50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f90a8103580 tx=0x7f9090009260 comp rx=0 tx=0).stop 2024-03-23T23:01:01.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.608+0000 7f90ae600700 1 -- 172.21.15.12:0/952397977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f90a81045d0 msgr2=0x7f90a81a0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:01.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.608+0000 7f90ae600700 1 --2- 172.21.15.12:0/952397977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f90a81045d0 0x7f90a81a0710 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f909800eb30 tx=0x7f909800ee40 comp rx=0 tx=0).stop 2024-03-23T23:01:01.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.608+0000 7f90ae600700 1 -- 172.21.15.12:0/952397977 shutdown_connections 2024-03-23T23:01:01.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.608+0000 7f90ae600700 1 --2- 172.21.15.12:0/952397977 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f908c0a2890 0x7f908c0a4d50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:01.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.608+0000 7f90ae600700 1 --2- 172.21.15.12:0/952397977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90a81096b0 0x7f90a819a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:01.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.608+0000 7f90ae600700 1 --2- 172.21.15.12:0/952397977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f90a81045d0 0x7f90a81a0710 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:01.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.608+0000 7f90ae600700 1 --2- 172.21.15.12:0/952397977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90a8104f80 0x7f90a81a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:01.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.608+0000 7f90ae600700 1 -- 172.21.15.12:0/952397977 >> 172.21.15.12:0/952397977 conn(0x7f90a80ffea0 msgr2=0x7f90a8100870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:01.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.608+0000 7f90ae600700 1 -- 172.21.15.12:0/952397977 shutdown_connections 2024-03-23T23:01:01.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:01.609+0000 7f90ae600700 1 -- 172.21.15.12:0/952397977 wait complete. 2024-03-23T23:01:02.464 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-23T23:01:02.465 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T23:01:02.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:02 smithi173 ceph-mon[36116]: pgmap v367: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:01:02.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:02 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/952397977' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-23T23:01:02.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:02 smithi203 ceph-mon[36693]: pgmap v367: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:01:02.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:02 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/952397977' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-23T23:01:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.703+0000 7fd673200640 1 -- 172.21.15.12:0/3486726075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd674101f70 msgr2=0x7fd674104360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.703+0000 7fd673200640 1 --2- 172.21.15.12:0/3486726075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd674101f70 0x7fd674104360 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd664009900 tx=0x7fd66402bc50 comp rx=0 tx=0).stop 2024-03-23T23:01:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.703+0000 7fd673200640 1 -- 172.21.15.12:0/3486726075 shutdown_connections 2024-03-23T23:01:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.703+0000 7fd673200640 1 --2- 172.21.15.12:0/3486726075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6741071d0 0x7fd674109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.703+0000 7fd673200640 1 --2- 172.21.15.12:0/3486726075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6741048a0 0x7fd674106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.703+0000 7fd673200640 1 --2- 172.21.15.12:0/3486726075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd674101f70 0x7fd674104360 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.703+0000 7fd673200640 1 -- 172.21.15.12:0/3486726075 >> 172.21.15.12:0/3486726075 conn(0x7fd6740fb980 msgr2=0x7fd6740fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.704+0000 7fd673200640 1 -- 172.21.15.12:0/3486726075 shutdown_connections 2024-03-23T23:01:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.704+0000 7fd673200640 1 -- 172.21.15.12:0/3486726075 wait complete. 2024-03-23T23:01:02.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.704+0000 7fd673200640 1 Processor -- start 2024-03-23T23:01:02.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.705+0000 7fd673200640 1 -- start start 2024-03-23T23:01:02.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.705+0000 7fd673200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd674101f70 0x7fd674199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:02.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.705+0000 7fd673200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6741048a0 0x7fd674199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:02.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.705+0000 7fd673200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6741071d0 0x7fd6741a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:02.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.705+0000 7fd673200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67410d1d0 con 0x7fd6741048a0 2024-03-23T23:01:02.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.705+0000 7fd673200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67410d050 con 0x7fd6741071d0 2024-03-23T23:01:02.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.705+0000 7fd673200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67410d350 con 0x7fd674101f70 2024-03-23T23:01:02.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.705+0000 7fd671e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd674101f70 0x7fd674199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:02.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.705+0000 7fd671e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd674101f70 0x7fd674199a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:37586/0 (socket says 172.21.15.12:37586) 2024-03-23T23:01:02.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.705+0000 7fd671e00640 1 -- 172.21.15.12:0/1610924914 learned_addr learned my addr 172.21.15.12:0/1610924914 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:02.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.705+0000 7fd671400640 1 --2- 172.21.15.12:0/1610924914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6741048a0 0x7fd674199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:02.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.705+0000 7fd672800640 1 --2- 172.21.15.12:0/1610924914 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6741071d0 0x7fd6741a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:02.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.706+0000 7fd671e00640 1 -- 172.21.15.12:0/1610924914 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6741071d0 msgr2=0x7fd6741a0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:02.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.706+0000 7fd671e00640 1 --2- 172.21.15.12:0/1610924914 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6741071d0 0x7fd6741a0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:02.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.706+0000 7fd671e00640 1 -- 172.21.15.12:0/1610924914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6741048a0 msgr2=0x7fd674199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:02.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.706+0000 7fd671e00640 1 --2- 172.21.15.12:0/1610924914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6741048a0 0x7fd674199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:02.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.706+0000 7fd671e00640 1 -- 172.21.15.12:0/1610924914 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6741a1620 con 0x7fd674101f70 2024-03-23T23:01:02.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.706+0000 7fd671e00640 1 --2- 172.21.15.12:0/1610924914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd674101f70 0x7fd674199a00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd66402c260 tx=0x7fd66402c670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:02.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.706+0000 7fd66aa00640 1 -- 172.21.15.12:0/1610924914 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd664041070 con 0x7fd674101f70 2024-03-23T23:01:02.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.706+0000 7fd66aa00640 1 -- 172.21.15.12:0/1610924914 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd664004450 con 0x7fd674101f70 2024-03-23T23:01:02.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.706+0000 7fd673200640 1 -- 172.21.15.12:0/1610924914 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6741a18b0 con 0x7fd674101f70 2024-03-23T23:01:02.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.706+0000 7fd66aa00640 1 -- 172.21.15.12:0/1610924914 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd664034440 con 0x7fd674101f70 2024-03-23T23:01:02.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.706+0000 7fd673200640 1 -- 172.21.15.12:0/1610924914 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6741a1e50 con 0x7fd674101f70 2024-03-23T23:01:02.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.708+0000 7fd673200640 1 -- 172.21.15.12:0/1610924914 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd674108d20 con 0x7fd674101f70 2024-03-23T23:01:02.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.709+0000 7fd66aa00640 1 -- 172.21.15.12:0/1610924914 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd664044070 con 0x7fd674101f70 2024-03-23T23:01:02.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.711+0000 7fd66aa00640 1 --2- 172.21.15.12:0/1610924914 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd6580a2880 0x7fd6580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:02.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.711+0000 7fd66aa00640 1 -- 172.21.15.12:0/1610924914 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd6640f04d0 con 0x7fd674101f70 2024-03-23T23:01:02.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.711+0000 7fd671400640 1 --2- 172.21.15.12:0/1610924914 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd6580a2880 0x7fd6580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:02.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.711+0000 7fd671400640 1 --2- 172.21.15.12:0/1610924914 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd6580a2880 0x7fd6580a4d40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd6600095f0 tx=0x7fd660005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:02.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.715+0000 7fd66aa00640 1 -- 172.21.15.12:0/1610924914 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd6640bced0 con 0x7fd674101f70 2024-03-23T23:01:02.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:02 smithi012 ceph-mon[31695]: pgmap v367: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:01:02.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:02 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/952397977' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-23T23:01:02.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.920+0000 7fd673200640 1 -- 172.21.15.12:0/1610924914 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6740622a0 con 0x7fd674101f70 2024-03-23T23:01:02.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.922+0000 7fd66aa00640 1 -- 172.21.15.12:0/1610924914 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22695 (secure 0 0 0) 0x7fd6640c0f70 con 0x7fd674101f70 2024-03-23T23:01:02.922 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:01:02.922 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":88,"fsid":"90f679ae-e967-11ee-95cd-87774f69a715","created":"2024-03-23T22:50:14.906952+0000","modified":"2024-03-23T23:00:24.349052+0000","last_up_change":"2024-03-23T22:58:28.907824+0000","last_in_change":"2024-03-23T22:58:08.342487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T22:54:11.831010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T22:59:25.350974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T22:59:26.390323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b70875d7-a23d-4251-9e8b-1a16d4edc2a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6803","nonce":1190079611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6805","nonce":1190079611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6809","nonce":1190079611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6807","nonce":1190079611}]},"public_addr":"172.21.15.12:6803/1190079611","cluster_addr":"172.21.15.12:6805/1190079611","heartbeat_back_addr":"172.21.15.12:6809/1190079611","heartbeat_front_addr":"172.21.15.12:6807/1190079611","state":["exists","up"]},{"osd":1,"uuid":"416e87af-db5d-41ce-84d9-731e3e07e3c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6801","nonce":1983639525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6803","nonce":1983639525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6807","nonce":1983639525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6805","nonce":1983639525}]},"public_addr":"172.21.15.173:6801/1983639525","cluster_addr":"172.21.15.173:6803/1983639525","heartbeat_back_addr":"172.21.15.173:6807/1983639525","heartbeat_front_addr":"172.21.15.173:6805/1983639525","state":["exists","up"]},{"osd":2,"uuid":"7a559c70-4c54-431b-8baf-4136b6e76493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6801","nonce":2253997387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6803","nonce":2253997387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6807","nonce":2253997387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6805","nonce":2253997387}]},"public_addr":"172.21.15.203:6801/2253997387","cluster_addr":"172.21.15.203:6803/2253997387","heartbeat_back_addr":"172.21.15.203:6807/2253997387","heartbeat_front_addr":"172.21.15.203:6805/2253997387","state":["exists","up"]},{"osd":3,"uuid":"c41cd287-a674-486c-8025-328cd36e5ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6811","nonce":1761446905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6813","nonce":1761446905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6817","nonce":1761446905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6815","nonce":1761446905}]},"public_addr":"172.21.15.12:6811/1761446905","cluster_addr":"172.21.15.12:6813/1761446905","heartbeat_back_addr":"172.21.15.12:6817/1761446905","heartbeat_front_addr":"172.21.15.12:6815/1761446905","state":["exists","up"]},{"osd":4,"uuid":"3c48f886-0b42-4a09-b31e-abccc200bb1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6809","nonce":242958272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6811","nonce":242958272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6815","nonce":242958272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6813","nonce":242958272}]},"public_addr":"172.21.15.173:6809/242958272","cluster_addr":"172.21.15.173:6811/242958272","heartbeat_back_addr":"172.21.15.173:6815/242958272","heartbeat_front_addr":"172.21.15.173:6813/242958272","state":["exists","up"]},{"osd":5,"uuid":"4a34f06f-8805-40c0-b1ab-23407b0276d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6809","nonce":4270966092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6811","nonce":4270966092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6815","nonce":4270966092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6813","nonce":4270966092}]},"public_addr":"172.21.15.203:6809/4270966092","cluster_addr":"172.21.15.203:6811/4270966092","heartbeat_back_addr":"172.21.15.203:6815/4270966092","heartbeat_front_addr":"172.21.15.203:6813/4270966092","state":["exists","up"]},{"osd":6,"uuid":"26ddc663-5623-48c9-80d1-f8b968765d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6819","nonce":3606839295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6821","nonce":3606839295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6825","nonce":3606839295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6823","nonce":3606839295}]},"public_addr":"172.21.15.12:6819/3606839295","cluster_addr":"172.21.15.12:6821/3606839295","heartbeat_back_addr":"172.21.15.12:6825/3606839295","heartbeat_front_addr":"172.21.15.12:6823/3606839295","state":["exists","up"]},{"osd":7,"uuid":"17dc9417-71cd-4576-aa08-933212ec55e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6817","nonce":1241504346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6819","nonce":1241504346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6823","nonce":1241504346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6821","nonce":1241504346}]},"public_addr":"172.21.15.173:6817/1241504346","cluster_addr":"172.21.15.173:6819/1241504346","heartbeat_back_addr":"172.21.15.173:6823/1241504346","heartbeat_front_addr":"172.21.15.173:6821/1241504346","state":["exists","up"]},{"osd":8,"uuid":"90a2b63a-471d-43b8-ac18-bb1f16778d18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6817","nonce":1516996192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6819","nonce":1516996192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6823","nonce":1516996192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6821","nonce":1516996192}]},"public_addr":"172.21.15.203:6817/1516996192","cluster_addr":"172.21.15.203:6819/1516996192","heartbeat_back_addr":"172.21.15.203:6823/1516996192","heartbeat_front_addr":"172.21.15.203:6821/1516996192","state":["exists","up"]},{"osd":9,"uuid":"6e1ca504-40b4-49d0-b245-7946181abfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6827","nonce":1175260774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6829","nonce":1175260774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6833","nonce":1175260774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6831","nonce":1175260774}]},"public_addr":"172.21.15.12:6827/1175260774","cluster_addr":"172.21.15.12:6829/1175260774","heartbeat_back_addr":"172.21.15.12:6833/1175260774","heartbeat_front_addr":"172.21.15.12:6831/1175260774","state":["exists","up"]},{"osd":10,"uuid":"09a595ed-5614-4386-8e61-0170af66e01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6825","nonce":3303506808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6827","nonce":3303506808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6831","nonce":3303506808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6829","nonce":3303506808}]},"public_addr":"172.21.15.173:6825/3303506808","cluster_addr":"172.21.15.173:6827/3303506808","heartbeat_back_addr":"172.21.15.173:6831/3303506808","heartbeat_front_addr":"172.21.15.173:6829/3303506808","state":["exists","up"]},{"osd":11,"uuid":"9ef1d595-a977-4ba7-9c6b-d2b3295f20b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6825","nonce":2323604506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6827","nonce":2323604506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6831","nonce":2323604506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6829","nonce":2323604506}]},"public_addr":"172.21.15.203:6825/2323604506","cluster_addr":"172.21.15.203:6827/2323604506","heartbeat_back_addr":"172.21.15.203:6831/2323604506","heartbeat_front_addr":"172.21.15.203:6829/2323604506","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:18.469038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:42.752628+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:07.069493+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:41.962478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:08.363104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:33.341336+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:07.286541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:34.243302+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:59.749699+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:57:33.822913+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:00.737269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:26.599854+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/4182783120":"2024-03-24T22:51:23.113201+0000","172.21.15.12:6800/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1900938637":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/3414861652":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6801/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/2580541477":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6800/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/501058839":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1192554716":"2024-03-24T22:50:48.401223+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T23:01:02.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.924+0000 7fd673200640 1 -- 172.21.15.12:0/1610924914 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd6580a2880 msgr2=0x7fd6580a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:02.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.924+0000 7fd673200640 1 --2- 172.21.15.12:0/1610924914 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd6580a2880 0x7fd6580a4d40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd6600095f0 tx=0x7fd660005f90 comp rx=0 tx=0).stop 2024-03-23T23:01:02.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.924+0000 7fd673200640 1 -- 172.21.15.12:0/1610924914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd674101f70 msgr2=0x7fd674199a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:02.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.924+0000 7fd673200640 1 --2- 172.21.15.12:0/1610924914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd674101f70 0x7fd674199a00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd66402c260 tx=0x7fd66402c670 comp rx=0 tx=0).stop 2024-03-23T23:01:02.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.924+0000 7fd673200640 1 -- 172.21.15.12:0/1610924914 shutdown_connections 2024-03-23T23:01:02.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.924+0000 7fd673200640 1 --2- 172.21.15.12:0/1610924914 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd6580a2880 0x7fd6580a4d40 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:02.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.924+0000 7fd673200640 1 --2- 172.21.15.12:0/1610924914 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6741071d0 0x7fd6741a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:02.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.924+0000 7fd673200640 1 --2- 172.21.15.12:0/1610924914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6741048a0 0x7fd674199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:02.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.924+0000 7fd673200640 1 --2- 172.21.15.12:0/1610924914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd674101f70 0x7fd674199a00 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:02.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.924+0000 7fd673200640 1 -- 172.21.15.12:0/1610924914 >> 172.21.15.12:0/1610924914 conn(0x7fd6740fb980 msgr2=0x7fd6741016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:02.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.925+0000 7fd673200640 1 -- 172.21.15.12:0/1610924914 shutdown_connections 2024-03-23T23:01:02.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:02.925+0000 7fd673200640 1 -- 172.21.15.12:0/1610924914 wait complete. 2024-03-23T23:01:02.947 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T22:54:11.831010+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T22:59:25.350974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T22:59:26.390323+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T23:01:02.947 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T23:01:03.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.174+0000 7f492b000640 1 -- 172.21.15.12:0/3111398848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49241049b0 msgr2=0x7f4924104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:03.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.174+0000 7f492b000640 1 --2- 172.21.15.12:0/3111398848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49241049b0 0x7f4924104e30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4910009900 tx=0x7f491002bc40 comp rx=0 tx=0).stop 2024-03-23T23:01:03.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.174+0000 7f492b000640 1 -- 172.21.15.12:0/3111398848 shutdown_connections 2024-03-23T23:01:03.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.174+0000 7f492b000640 1 --2- 172.21.15.12:0/3111398848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4924105370 0x7f492410bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:03.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.174+0000 7f492b000640 1 --2- 172.21.15.12:0/3111398848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49241049b0 0x7f4924104e30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:03.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.174+0000 7f492b000640 1 --2- 172.21.15.12:0/3111398848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49241037c0 0x7f4924103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:03.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.174+0000 7f492b000640 1 -- 172.21.15.12:0/3111398848 >> 172.21.15.12:0/3111398848 conn(0x7f49240fed50 msgr2=0x7f4924101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:03.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.175+0000 7f492b000640 1 -- 172.21.15.12:0/3111398848 shutdown_connections 2024-03-23T23:01:03.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.175+0000 7f492b000640 1 -- 172.21.15.12:0/3111398848 wait complete. 2024-03-23T23:01:03.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.176+0000 7f492b000640 1 Processor -- start 2024-03-23T23:01:03.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.176+0000 7f492b000640 1 -- start start 2024-03-23T23:01:03.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.176+0000 7f492b000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49241037c0 0x7f4924199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:03.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.176+0000 7f492b000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49241049b0 0x7f492419a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:03.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.176+0000 7f492b000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4924105370 0x7f49241a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:03.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.176+0000 7f492b000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492410f490 con 0x7f49241049b0 2024-03-23T23:01:03.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.176+0000 7f492b000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492410f310 con 0x7f49241037c0 2024-03-23T23:01:03.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.176+0000 7f492b000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492410f610 con 0x7f4924105370 2024-03-23T23:01:03.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.177+0000 7f492a600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4924105370 0x7f49241a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:03.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.177+0000 7f492a600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4924105370 0x7f49241a10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:37606/0 (socket says 172.21.15.12:37606) 2024-03-23T23:01:03.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.177+0000 7f492a600640 1 -- 172.21.15.12:0/2046619245 learned_addr learned my addr 172.21.15.12:0/2046619245 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:03.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.177+0000 7f4929200640 1 --2- 172.21.15.12:0/2046619245 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49241049b0 0x7f492419a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:03.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.177+0000 7f4929c00640 1 --2- 172.21.15.12:0/2046619245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49241037c0 0x7f4924199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:03.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.177+0000 7f492a600640 1 -- 172.21.15.12:0/2046619245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49241037c0 msgr2=0x7f4924199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:03.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.177+0000 7f492a600640 1 --2- 172.21.15.12:0/2046619245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49241037c0 0x7f4924199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:03.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.177+0000 7f492a600640 1 -- 172.21.15.12:0/2046619245 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49241049b0 msgr2=0x7f492419a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:03.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.177+0000 7f492a600640 1 --2- 172.21.15.12:0/2046619245 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49241049b0 0x7f492419a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:03.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.177+0000 7f492a600640 1 -- 172.21.15.12:0/2046619245 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49241a1710 con 0x7f4924105370 2024-03-23T23:01:03.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.177+0000 7f492a600640 1 --2- 172.21.15.12:0/2046619245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4924105370 0x7f49241a10a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f491c00e970 tx=0x7f491c00ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:03.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.178+0000 7f4922000640 1 -- 172.21.15.12:0/2046619245 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491c00ccc0 con 0x7f4924105370 2024-03-23T23:01:03.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.178+0000 7f492b000640 1 -- 172.21.15.12:0/2046619245 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49241a1a00 con 0x7f4924105370 2024-03-23T23:01:03.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.178+0000 7f4922000640 1 -- 172.21.15.12:0/2046619245 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f491c00ce60 con 0x7f4924105370 2024-03-23T23:01:03.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.178+0000 7f4922000640 1 -- 172.21.15.12:0/2046619245 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491c010450 con 0x7f4924105370 2024-03-23T23:01:03.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.178+0000 7f492b000640 1 -- 172.21.15.12:0/2046619245 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49241a1f20 con 0x7f4924105370 2024-03-23T23:01:03.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.179+0000 7f492b000640 1 -- 172.21.15.12:0/2046619245 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4924068800 con 0x7f4924105370 2024-03-23T23:01:03.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.181+0000 7f4922000640 1 -- 172.21.15.12:0/2046619245 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f491c005020 con 0x7f4924105370 2024-03-23T23:01:03.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.182+0000 7f4922000640 1 --2- 172.21.15.12:0/2046619245 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f490c0a2800 0x7f490c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:03.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.182+0000 7f4922000640 1 -- 172.21.15.12:0/2046619245 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f491c0ceea0 con 0x7f4924105370 2024-03-23T23:01:03.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.182+0000 7f4929c00640 1 --2- 172.21.15.12:0/2046619245 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f490c0a2800 0x7f490c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:03.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.183+0000 7f4929c00640 1 --2- 172.21.15.12:0/2046619245 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f490c0a2800 0x7f490c0a4cc0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4924104810 tx=0x7f4918009670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.189+0000 7f4922000640 1 -- 172.21.15.12:0/2046619245 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f491c09bef0 con 0x7f4924105370 2024-03-23T23:01:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.400+0000 7f492b000640 1 -- 172.21.15.12:0/2046619245 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f49240622a0 con 0x7f4924105370 2024-03-23T23:01:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.401+0000 7f4922000640 1 -- 172.21.15.12:0/2046619245 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f491c09b7f0 con 0x7f4924105370 2024-03-23T23:01:03.401 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-23T23:01:03.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.403+0000 7f492b000640 1 -- 172.21.15.12:0/2046619245 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f490c0a2800 msgr2=0x7f490c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:03.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.403+0000 7f492b000640 1 --2- 172.21.15.12:0/2046619245 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f490c0a2800 0x7f490c0a4cc0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4924104810 tx=0x7f4918009670 comp rx=0 tx=0).stop 2024-03-23T23:01:03.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.403+0000 7f492b000640 1 -- 172.21.15.12:0/2046619245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4924105370 msgr2=0x7f49241a10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:03.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.403+0000 7f492b000640 1 --2- 172.21.15.12:0/2046619245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4924105370 0x7f49241a10a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f491c00e970 tx=0x7f491c00ed70 comp rx=0 tx=0).stop 2024-03-23T23:01:03.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.403+0000 7f492b000640 1 -- 172.21.15.12:0/2046619245 shutdown_connections 2024-03-23T23:01:03.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.403+0000 7f492b000640 1 --2- 172.21.15.12:0/2046619245 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f490c0a2800 0x7f490c0a4cc0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:03.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.403+0000 7f492b000640 1 --2- 172.21.15.12:0/2046619245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4924105370 0x7f49241a10a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:03.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.403+0000 7f492b000640 1 --2- 172.21.15.12:0/2046619245 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49241049b0 0x7f492419a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:03.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.403+0000 7f492b000640 1 --2- 172.21.15.12:0/2046619245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49241037c0 0x7f4924199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:03.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.403+0000 7f492b000640 1 -- 172.21.15.12:0/2046619245 >> 172.21.15.12:0/2046619245 conn(0x7f49240fed50 msgr2=0x7f4924107c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:03.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.403+0000 7f492b000640 1 -- 172.21.15.12:0/2046619245 shutdown_connections 2024-03-23T23:01:03.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.404+0000 7f492b000640 1 -- 172.21.15.12:0/2046619245 wait complete. 2024-03-23T23:01:03.422 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T23:01:03.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:03 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1610924914' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:03.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.647+0000 7fc0e3200640 1 -- 172.21.15.12:0/1549275322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0dc104930 msgr2=0x7fc0dc106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:03.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.647+0000 7fc0e3200640 1 --2- 172.21.15.12:0/1549275322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0dc104930 0x7fc0dc106d20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc009860 tx=0x7fc0cc02bb70 comp rx=0 tx=0).stop 2024-03-23T23:01:03.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.647+0000 7fc0e3200640 1 -- 172.21.15.12:0/1549275322 shutdown_connections 2024-03-23T23:01:03.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.647+0000 7fc0e3200640 1 --2- 172.21.15.12:0/1549275322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0dc107260 0x7fc0dc1096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:03.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.647+0000 7fc0e3200640 1 --2- 172.21.15.12:0/1549275322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0dc104930 0x7fc0dc106d20 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:03.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.647+0000 7fc0e3200640 1 --2- 172.21.15.12:0/1549275322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0dc102000 0x7fc0dc1043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:03.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.647+0000 7fc0e3200640 1 -- 172.21.15.12:0/1549275322 >> 172.21.15.12:0/1549275322 conn(0x7fc0dc0fba10 msgr2=0x7fc0dc0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:03.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.648+0000 7fc0e3200640 1 -- 172.21.15.12:0/1549275322 shutdown_connections 2024-03-23T23:01:03.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.648+0000 7fc0e3200640 1 -- 172.21.15.12:0/1549275322 wait complete. 2024-03-23T23:01:03.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.649+0000 7fc0e3200640 1 Processor -- start 2024-03-23T23:01:03.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.649+0000 7fc0e3200640 1 -- start start 2024-03-23T23:01:03.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.650+0000 7fc0e3200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0dc102000 0x7fc0dc1978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:03.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.650+0000 7fc0e3200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0dc104930 0x7fc0dc197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:03.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.650+0000 7fc0e3200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0dc107260 0x7fc0dc19ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:03.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.650+0000 7fc0e3200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0dc10d210 con 0x7fc0dc107260 2024-03-23T23:01:03.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.650+0000 7fc0e3200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0dc10d090 con 0x7fc0dc104930 2024-03-23T23:01:03.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.650+0000 7fc0e3200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0dc10d390 con 0x7fc0dc102000 2024-03-23T23:01:03.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.650+0000 7fc0e1e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0dc102000 0x7fc0dc1978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:03.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.650+0000 7fc0e1e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0dc102000 0x7fc0dc1978c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:37612/0 (socket says 172.21.15.12:37612) 2024-03-23T23:01:03.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.650+0000 7fc0e1e00640 1 -- 172.21.15.12:0/628290652 learned_addr learned my addr 172.21.15.12:0/628290652 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:03.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.650+0000 7fc0e1400640 1 --2- 172.21.15.12:0/628290652 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0dc104930 0x7fc0dc197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:03.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.650+0000 7fc0e2800640 1 --2- 172.21.15.12:0/628290652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0dc107260 0x7fc0dc19ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:03.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.650+0000 7fc0e1e00640 1 -- 172.21.15.12:0/628290652 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0dc104930 msgr2=0x7fc0dc197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:03.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.651+0000 7fc0e1e00640 1 --2- 172.21.15.12:0/628290652 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0dc104930 0x7fc0dc197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:03.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.651+0000 7fc0e1e00640 1 -- 172.21.15.12:0/628290652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0dc107260 msgr2=0x7fc0dc19ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:03.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.651+0000 7fc0e1e00640 1 --2- 172.21.15.12:0/628290652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0dc107260 0x7fc0dc19ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:03.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.651+0000 7fc0e1e00640 1 -- 172.21.15.12:0/628290652 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0dc19f4e0 con 0x7fc0dc102000 2024-03-23T23:01:03.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.651+0000 7fc0e1e00640 1 --2- 172.21.15.12:0/628290652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0dc102000 0x7fc0dc1978c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc0d000e9c0 tx=0x7fc0d000edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:03.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.651+0000 7fc0d6a00640 1 -- 172.21.15.12:0/628290652 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d000cd80 con 0x7fc0dc102000 2024-03-23T23:01:03.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.651+0000 7fc0e3200640 1 -- 172.21.15.12:0/628290652 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0dc19f7d0 con 0x7fc0dc102000 2024-03-23T23:01:03.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.651+0000 7fc0d6a00640 1 -- 172.21.15.12:0/628290652 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0d00035f0 con 0x7fc0dc102000 2024-03-23T23:01:03.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.651+0000 7fc0d6a00640 1 -- 172.21.15.12:0/628290652 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0010670 con 0x7fc0dc102000 2024-03-23T23:01:03.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.651+0000 7fc0e3200640 1 -- 172.21.15.12:0/628290652 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0dc19fcf0 con 0x7fc0dc102000 2024-03-23T23:01:03.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.652+0000 7fc0e3200640 1 -- 172.21.15.12:0/628290652 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0dc0fd500 con 0x7fc0dc102000 2024-03-23T23:01:03.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.654+0000 7fc0d6a00640 1 -- 172.21.15.12:0/628290652 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc0d00037a0 con 0x7fc0dc102000 2024-03-23T23:01:03.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.655+0000 7fc0d6a00640 1 --2- 172.21.15.12:0/628290652 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc0c40a2800 0x7fc0c40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:03.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.656+0000 7fc0d6a00640 1 -- 172.21.15.12:0/628290652 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fc0d00cecc0 con 0x7fc0dc102000 2024-03-23T23:01:03.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.656+0000 7fc0e1400640 1 --2- 172.21.15.12:0/628290652 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc0c40a2800 0x7fc0c40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:03.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.657+0000 7fc0e1400640 1 --2- 172.21.15.12:0/628290652 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc0c40a2800 0x7fc0c40a4cc0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc0349e0 tx=0x7fc0cc034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:03.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:03 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1610924914' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:03.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.659+0000 7fc0d6a00640 1 -- 172.21.15.12:0/628290652 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc0d00d5080 con 0x7fc0dc102000 2024-03-23T23:01:03.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:03 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1610924914' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:03.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.862+0000 7fc0e3200640 1 -- 172.21.15.12:0/628290652 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc0dc0622a0 con 0x7fc0dc102000 2024-03-23T23:01:03.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.862+0000 7fc0d6a00640 1 -- 172.21.15.12:0/628290652 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc0d009bd10 con 0x7fc0dc102000 2024-03-23T23:01:03.862 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-23T23:01:03.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.865+0000 7fc0e3200640 1 -- 172.21.15.12:0/628290652 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc0c40a2800 msgr2=0x7fc0c40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:03.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.865+0000 7fc0e3200640 1 --2- 172.21.15.12:0/628290652 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc0c40a2800 0x7fc0c40a4cc0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc0349e0 tx=0x7fc0cc034ab0 comp rx=0 tx=0).stop 2024-03-23T23:01:03.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.865+0000 7fc0e3200640 1 -- 172.21.15.12:0/628290652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0dc102000 msgr2=0x7fc0dc1978c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:03.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.865+0000 7fc0e3200640 1 --2- 172.21.15.12:0/628290652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0dc102000 0x7fc0dc1978c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc0d000e9c0 tx=0x7fc0d000edc0 comp rx=0 tx=0).stop 2024-03-23T23:01:03.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.865+0000 7fc0e3200640 1 -- 172.21.15.12:0/628290652 shutdown_connections 2024-03-23T23:01:03.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.865+0000 7fc0e3200640 1 --2- 172.21.15.12:0/628290652 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc0c40a2800 0x7fc0c40a4cc0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:03.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.865+0000 7fc0e3200640 1 --2- 172.21.15.12:0/628290652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0dc107260 0x7fc0dc19ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:03.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.865+0000 7fc0e3200640 1 --2- 172.21.15.12:0/628290652 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0dc104930 0x7fc0dc197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:03.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.865+0000 7fc0e3200640 1 --2- 172.21.15.12:0/628290652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0dc102000 0x7fc0dc1978c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:03.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.865+0000 7fc0e3200640 1 -- 172.21.15.12:0/628290652 >> 172.21.15.12:0/628290652 conn(0x7fc0dc0fba10 msgr2=0x7fc0dc102960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:03.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.866+0000 7fc0e3200640 1 -- 172.21.15.12:0/628290652 shutdown_connections 2024-03-23T23:01:03.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:03.866+0000 7fc0e3200640 1 -- 172.21.15.12:0/628290652 wait complete. 2024-03-23T23:01:03.884 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T23:01:04.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.117+0000 7f76f5200640 1 -- 172.21.15.12:0/1073359063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76f0104930 msgr2=0x7f76f0106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:04.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.117+0000 7f76f5200640 1 --2- 172.21.15.12:0/1073359063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76f0104930 0x7f76f0106d20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f76d8009900 tx=0x7f76d802bc40 comp rx=0 tx=0).stop 2024-03-23T23:01:04.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.117+0000 7f76f5200640 1 -- 172.21.15.12:0/1073359063 shutdown_connections 2024-03-23T23:01:04.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.117+0000 7f76f5200640 1 --2- 172.21.15.12:0/1073359063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f0107260 0x7f76f01096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:04.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.117+0000 7f76f5200640 1 --2- 172.21.15.12:0/1073359063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76f0104930 0x7f76f0106d20 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:04.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.117+0000 7f76f5200640 1 --2- 172.21.15.12:0/1073359063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76f0102000 0x7f76f01043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:04.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.117+0000 7f76f5200640 1 -- 172.21.15.12:0/1073359063 >> 172.21.15.12:0/1073359063 conn(0x7f76f00fba10 msgr2=0x7f76f00fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:04.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.117+0000 7f76f5200640 1 -- 172.21.15.12:0/1073359063 shutdown_connections 2024-03-23T23:01:04.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.117+0000 7f76f5200640 1 -- 172.21.15.12:0/1073359063 wait complete. 2024-03-23T23:01:04.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.118+0000 7f76f5200640 1 Processor -- start 2024-03-23T23:01:04.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.119+0000 7f76f5200640 1 -- start start 2024-03-23T23:01:04.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.119+0000 7f76f5200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76f0102000 0x7f76f01978a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:04.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.119+0000 7f76f5200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76f0104930 0x7f76f0197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:04.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.119+0000 7f76f5200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f0107260 0x7f76f019ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:04.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.119+0000 7f76f5200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f010d220 con 0x7f76f0104930 2024-03-23T23:01:04.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.119+0000 7f76f5200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f010d0a0 con 0x7f76f0107260 2024-03-23T23:01:04.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.119+0000 7f76f5200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f010d3a0 con 0x7f76f0102000 2024-03-23T23:01:04.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.119+0000 7f76e6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76f0104930 0x7f76f0197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:04.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.119+0000 7f76ef400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76f0102000 0x7f76f01978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:04.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.119+0000 7f76ef400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76f0102000 0x7f76f01978a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49016/0 (socket says 172.21.15.12:49016) 2024-03-23T23:01:04.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.119+0000 7f76e6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76f0104930 0x7f76f0197de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54312/0 (socket says 172.21.15.12:54312) 2024-03-23T23:01:04.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.119+0000 7f76ef400640 1 -- 172.21.15.12:0/2304294520 learned_addr learned my addr 172.21.15.12:0/2304294520 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:04.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.119+0000 7f76efe00640 1 --2- 172.21.15.12:0/2304294520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f0107260 0x7f76f019ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:04.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.120+0000 7f76ef400640 1 -- 172.21.15.12:0/2304294520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f0107260 msgr2=0x7f76f019ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:04.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.120+0000 7f76ef400640 1 --2- 172.21.15.12:0/2304294520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f0107260 0x7f76f019ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:04.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.120+0000 7f76ef400640 1 -- 172.21.15.12:0/2304294520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76f0104930 msgr2=0x7f76f0197de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:04.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.120+0000 7f76ef400640 1 --2- 172.21.15.12:0/2304294520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76f0104930 0x7f76f0197de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:04.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.120+0000 7f76ef400640 1 -- 172.21.15.12:0/2304294520 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f019f4c0 con 0x7f76f0102000 2024-03-23T23:01:04.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.120+0000 7f76efe00640 1 --2- 172.21.15.12:0/2304294520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f0107260 0x7f76f019ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:04.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.120+0000 7f76ef400640 1 --2- 172.21.15.12:0/2304294520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76f0102000 0x7f76f01978a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f76e000d950 tx=0x7f76e000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:04.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.120+0000 7f76ecc00640 1 -- 172.21.15.12:0/2304294520 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e0017070 con 0x7f76f0102000 2024-03-23T23:01:04.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.120+0000 7f76ecc00640 1 -- 172.21.15.12:0/2304294520 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76e0009c30 con 0x7f76f0102000 2024-03-23T23:01:04.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.120+0000 7f76f5200640 1 -- 172.21.15.12:0/2304294520 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f019f7b0 con 0x7f76f0102000 2024-03-23T23:01:04.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.121+0000 7f76ecc00640 1 -- 172.21.15.12:0/2304294520 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e000f6b0 con 0x7f76f0102000 2024-03-23T23:01:04.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.121+0000 7f76f5200640 1 -- 172.21.15.12:0/2304294520 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f019fd20 con 0x7f76f0102000 2024-03-23T23:01:04.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.122+0000 7f76f5200640 1 -- 172.21.15.12:0/2304294520 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f00fd500 con 0x7f76f0102000 2024-03-23T23:01:04.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.123+0000 7f76ecc00640 1 -- 172.21.15.12:0/2304294520 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f76e00109d0 con 0x7f76f0102000 2024-03-23T23:01:04.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.125+0000 7f76ecc00640 1 --2- 172.21.15.12:0/2304294520 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f76d40a2880 0x7f76d40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:04.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.125+0000 7f76ecc00640 1 -- 172.21.15.12:0/2304294520 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f76e00ce5f0 con 0x7f76f0102000 2024-03-23T23:01:04.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.125+0000 7f76e6a00640 1 --2- 172.21.15.12:0/2304294520 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f76d40a2880 0x7f76d40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:04.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.126+0000 7f76e6a00640 1 --2- 172.21.15.12:0/2304294520 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f76d40a2880 0x7f76d40a4d40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f76d802c180 tx=0x7f76d802cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:04.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.129+0000 7f76ecc00640 1 -- 172.21.15.12:0/2304294520 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f76e0009de0 con 0x7f76f0102000 2024-03-23T23:01:04.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.334+0000 7f76f5200640 1 -- 172.21.15.12:0/2304294520 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f76f00622a0 con 0x7f76f0102000 2024-03-23T23:01:04.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.335+0000 7f76ecc00640 1 -- 172.21.15.12:0/2304294520 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f76e009aff0 con 0x7f76f0102000 2024-03-23T23:01:04.334 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-23T23:01:04.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.337+0000 7f76f5200640 1 -- 172.21.15.12:0/2304294520 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f76d40a2880 msgr2=0x7f76d40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:04.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.337+0000 7f76f5200640 1 --2- 172.21.15.12:0/2304294520 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f76d40a2880 0x7f76d40a4d40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f76d802c180 tx=0x7f76d802cf90 comp rx=0 tx=0).stop 2024-03-23T23:01:04.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.337+0000 7f76f5200640 1 -- 172.21.15.12:0/2304294520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76f0102000 msgr2=0x7f76f01978a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:04.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.337+0000 7f76f5200640 1 --2- 172.21.15.12:0/2304294520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76f0102000 0x7f76f01978a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f76e000d950 tx=0x7f76e000de10 comp rx=0 tx=0).stop 2024-03-23T23:01:04.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.338+0000 7f76f5200640 1 -- 172.21.15.12:0/2304294520 shutdown_connections 2024-03-23T23:01:04.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.338+0000 7f76f5200640 1 --2- 172.21.15.12:0/2304294520 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f76d40a2880 0x7f76d40a4d40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:04.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.338+0000 7f76f5200640 1 --2- 172.21.15.12:0/2304294520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f0107260 0x7f76f019ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:04.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.338+0000 7f76f5200640 1 --2- 172.21.15.12:0/2304294520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76f0104930 0x7f76f0197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:04.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.338+0000 7f76f5200640 1 --2- 172.21.15.12:0/2304294520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76f0102000 0x7f76f01978a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:04.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.338+0000 7f76f5200640 1 -- 172.21.15.12:0/2304294520 >> 172.21.15.12:0/2304294520 conn(0x7f76f00fba10 msgr2=0x7f76f0102a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:04.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.338+0000 7f76f5200640 1 -- 172.21.15.12:0/2304294520 shutdown_connections 2024-03-23T23:01:04.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.338+0000 7f76f5200640 1 -- 172.21.15.12:0/2304294520 wait complete. 2024-03-23T23:01:04.359 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:01:04.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.573+0000 7f797de00640 1 -- 172.21.15.12:0/11640647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79781052a0 msgr2=0x7f797810bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:04.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.573+0000 7f797de00640 1 --2- 172.21.15.12:0/11640647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79781052a0 0x7f797810bb30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7970009860 tx=0x7f797002bb70 comp rx=0 tx=0).stop 2024-03-23T23:01:04.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.573+0000 7f797de00640 1 -- 172.21.15.12:0/11640647 shutdown_connections 2024-03-23T23:01:04.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.573+0000 7f797de00640 1 --2- 172.21.15.12:0/11640647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79781052a0 0x7f797810bb30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:04.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.573+0000 7f797de00640 1 --2- 172.21.15.12:0/11640647 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79781048e0 0x7f7978104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:04.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.573+0000 7f797de00640 1 --2- 172.21.15.12:0/11640647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79781036f0 0x7f7978103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:04.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.573+0000 7f797de00640 1 -- 172.21.15.12:0/11640647 >> 172.21.15.12:0/11640647 conn(0x7f79780fec80 msgr2=0x7f79781010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:04.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.573+0000 7f797de00640 1 -- 172.21.15.12:0/11640647 shutdown_connections 2024-03-23T23:01:04.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.573+0000 7f797de00640 1 -- 172.21.15.12:0/11640647 wait complete. 2024-03-23T23:01:04.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.574+0000 7f797de00640 1 Processor -- start 2024-03-23T23:01:04.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.574+0000 7f797de00640 1 -- start start 2024-03-23T23:01:04.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.574+0000 7f797de00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79781036f0 0x7f7978199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:04.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.574+0000 7f797de00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79781048e0 0x7f7978199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:04.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.574+0000 7f797de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79781052a0 0x7f79781a0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:04.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.574+0000 7f797de00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797810f350 con 0x7f79781048e0 2024-03-23T23:01:04.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.574+0000 7f797de00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797810f1d0 con 0x7f79781052a0 2024-03-23T23:01:04.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.574+0000 7f797de00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797810f4d0 con 0x7f79781036f0 2024-03-23T23:01:04.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.575+0000 7f796fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79781048e0 0x7f7978199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:04.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.575+0000 7f797ca00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79781036f0 0x7f7978199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:04.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.575+0000 7f797d400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79781052a0 0x7f79781a0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:04.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.575+0000 7f797d400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79781052a0 0x7f79781a0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:56792/0 (socket says 172.21.15.12:56792) 2024-03-23T23:01:04.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.575+0000 7f797ca00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79781036f0 0x7f7978199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49032/0 (socket says 172.21.15.12:49032) 2024-03-23T23:01:04.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.575+0000 7f797d400640 1 -- 172.21.15.12:0/2825622483 learned_addr learned my addr 172.21.15.12:0/2825622483 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:04.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.575+0000 7f797d400640 1 -- 172.21.15.12:0/2825622483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79781036f0 msgr2=0x7f7978199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:04.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.575+0000 7f797d400640 1 --2- 172.21.15.12:0/2825622483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79781036f0 0x7f7978199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:04.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.575+0000 7f797d400640 1 -- 172.21.15.12:0/2825622483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79781048e0 msgr2=0x7f7978199e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:04.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.575+0000 7f797d400640 1 --2- 172.21.15.12:0/2825622483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79781048e0 0x7f7978199e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:04.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.575+0000 7f797d400640 1 -- 172.21.15.12:0/2825622483 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79781a1550 con 0x7f79781052a0 2024-03-23T23:01:04.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.575+0000 7f797d400640 1 --2- 172.21.15.12:0/2825622483 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79781052a0 0x7f79781a0ee0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f797002c180 tx=0x7f7970004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:04.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.575+0000 7f7976000640 1 -- 172.21.15.12:0/2825622483 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7970041020 con 0x7f79781052a0 2024-03-23T23:01:04.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.576+0000 7f7976000640 1 -- 172.21.15.12:0/2825622483 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79700347b0 con 0x7f79781052a0 2024-03-23T23:01:04.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.576+0000 7f797de00640 1 -- 172.21.15.12:0/2825622483 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79781a17e0 con 0x7f79781052a0 2024-03-23T23:01:04.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.576+0000 7f7976000640 1 -- 172.21.15.12:0/2825622483 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7970039040 con 0x7f79781052a0 2024-03-23T23:01:04.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.576+0000 7f797de00640 1 -- 172.21.15.12:0/2825622483 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79781a1d20 con 0x7f79781052a0 2024-03-23T23:01:04.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.577+0000 7f797de00640 1 -- 172.21.15.12:0/2825622483 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f797810aa10 con 0x7f79781052a0 2024-03-23T23:01:04.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.579+0000 7f7976000640 1 -- 172.21.15.12:0/2825622483 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7970044020 con 0x7f79781052a0 2024-03-23T23:01:04.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.580+0000 7f7976000640 1 --2- 172.21.15.12:0/2825622483 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f795c0a2880 0x7f795c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:04.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.580+0000 7f7976000640 1 -- 172.21.15.12:0/2825622483 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f79700f0450 con 0x7f79781052a0 2024-03-23T23:01:04.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.580+0000 7f797ca00640 1 --2- 172.21.15.12:0/2825622483 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f795c0a2880 0x7f795c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:04.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.581+0000 7f797ca00640 1 --2- 172.21.15.12:0/2825622483 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f795c0a2880 0x7f795c0a4d40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7978104740 tx=0x7f7968009670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:04.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.584+0000 7f7976000640 1 -- 172.21.15.12:0/2825622483 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f797004b020 con 0x7f79781052a0 2024-03-23T23:01:04.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:04 smithi173 ceph-mon[36116]: pgmap v368: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:01:04.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:04 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2046619245' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:01:04.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:04 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/628290652' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:01:04.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:04 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2304294520' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:01:04.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:04 smithi203 ceph-mon[36693]: pgmap v368: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:01:04.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:04 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2046619245' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:01:04.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:04 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/628290652' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:01:04.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:04 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2304294520' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:01:04.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:04 smithi012 ceph-mon[31695]: pgmap v368: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:01:04.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:04 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2046619245' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:01:04.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:04 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/628290652' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:01:04.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:04 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2304294520' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:01:04.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.827+0000 7f797de00640 1 -- 172.21.15.12:0/2825622483 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79780622a0 con 0x7f79781052a0 2024-03-23T23:01:04.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.828+0000 7f7976000640 1 -- 172.21.15.12:0/2825622483 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f79700bd4a0 con 0x7f79781052a0 2024-03-23T23:01:04.829 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:01:04.829 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T22:59:32.190756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24599,24550]},"id":1}]} 2024-03-23T23:01:04.829 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 22 2024-03-23T23:01:04.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.831+0000 7f797de00640 1 -- 172.21.15.12:0/2825622483 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f795c0a2880 msgr2=0x7f795c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:04.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.831+0000 7f797de00640 1 --2- 172.21.15.12:0/2825622483 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f795c0a2880 0x7f795c0a4d40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7978104740 tx=0x7f7968009670 comp rx=0 tx=0).stop 2024-03-23T23:01:04.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.832+0000 7f797de00640 1 -- 172.21.15.12:0/2825622483 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79781052a0 msgr2=0x7f79781a0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:04.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.832+0000 7f797de00640 1 --2- 172.21.15.12:0/2825622483 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79781052a0 0x7f79781a0ee0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f797002c180 tx=0x7f7970004270 comp rx=0 tx=0).stop 2024-03-23T23:01:04.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.832+0000 7f797de00640 1 -- 172.21.15.12:0/2825622483 shutdown_connections 2024-03-23T23:01:04.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.832+0000 7f797de00640 1 --2- 172.21.15.12:0/2825622483 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f795c0a2880 0x7f795c0a4d40 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:04.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.832+0000 7f797de00640 1 --2- 172.21.15.12:0/2825622483 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79781052a0 0x7f79781a0ee0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:04.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.832+0000 7f797de00640 1 --2- 172.21.15.12:0/2825622483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79781048e0 0x7f7978199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:04.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.832+0000 7f797de00640 1 --2- 172.21.15.12:0/2825622483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79781036f0 0x7f7978199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:04.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.832+0000 7f797de00640 1 -- 172.21.15.12:0/2825622483 >> 172.21.15.12:0/2825622483 conn(0x7f79780fec80 msgr2=0x7f7978107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:04.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.832+0000 7f797de00640 1 -- 172.21.15.12:0/2825622483 shutdown_connections 2024-03-23T23:01:04.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:04.832+0000 7f797de00640 1 -- 172.21.15.12:0/2825622483 wait complete. 2024-03-23T23:01:04.852 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T23:01:05.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.075+0000 7f2fc1200640 1 -- 172.21.15.12:0/445451770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fbc104ad0 msgr2=0x7f2fbc106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:05.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.075+0000 7f2fc1200640 1 --2- 172.21.15.12:0/445451770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fbc104ad0 0x7f2fbc106ec0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4009900 tx=0x7f2fa402bc30 comp rx=0 tx=0).stop 2024-03-23T23:01:05.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.075+0000 7f2fc1200640 1 -- 172.21.15.12:0/445451770 shutdown_connections 2024-03-23T23:01:05.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.075+0000 7f2fc1200640 1 --2- 172.21.15.12:0/445451770 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fbc107400 0x7f2fbc1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:05.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.075+0000 7f2fc1200640 1 --2- 172.21.15.12:0/445451770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fbc104ad0 0x7f2fbc106ec0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:05.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.075+0000 7f2fc1200640 1 --2- 172.21.15.12:0/445451770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fbc1021a0 0x7f2fbc104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:05.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.075+0000 7f2fc1200640 1 -- 172.21.15.12:0/445451770 >> 172.21.15.12:0/445451770 conn(0x7f2fbc0fbb30 msgr2=0x7f2fbc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:05.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.075+0000 7f2fc1200640 1 -- 172.21.15.12:0/445451770 shutdown_connections 2024-03-23T23:01:05.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.075+0000 7f2fc1200640 1 -- 172.21.15.12:0/445451770 wait complete. 2024-03-23T23:01:05.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.076+0000 7f2fc1200640 1 Processor -- start 2024-03-23T23:01:05.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.076+0000 7f2fc1200640 1 -- start start 2024-03-23T23:01:05.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.077+0000 7f2fc1200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fbc1021a0 0x7f2fbc197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:05.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.077+0000 7f2fc1200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fbc104ad0 0x7f2fbc197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:05.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.077+0000 7f2fc1200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fbc107400 0x7f2fbc19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:05.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.077+0000 7f2fc1200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fbc10d330 con 0x7f2fbc1021a0 2024-03-23T23:01:05.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.077+0000 7f2fc1200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fbc10d1b0 con 0x7f2fbc107400 2024-03-23T23:01:05.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.077+0000 7f2fc1200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fbc10d4b0 con 0x7f2fbc104ad0 2024-03-23T23:01:05.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.077+0000 7f2fb9400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fbc104ad0 0x7f2fbc197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:05.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.077+0000 7f2fb9e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fbc1021a0 0x7f2fbc197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.077+0000 7f2fb9400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fbc104ad0 0x7f2fbc197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49050/0 (socket says 172.21.15.12:49050) 2024-03-23T23:01:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.077+0000 7f2fb9400640 1 -- 172.21.15.12:0/195995212 learned_addr learned my addr 172.21.15.12:0/195995212 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.077+0000 7f2fba800640 1 --2- 172.21.15.12:0/195995212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fbc107400 0x7f2fbc19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.077+0000 7f2fb9400640 1 -- 172.21.15.12:0/195995212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fbc107400 msgr2=0x7f2fbc19f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.077+0000 7f2fb9400640 1 --2- 172.21.15.12:0/195995212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fbc107400 0x7f2fbc19f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.077+0000 7f2fb9400640 1 -- 172.21.15.12:0/195995212 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fbc1021a0 msgr2=0x7f2fbc197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.077+0000 7f2fb9400640 1 --2- 172.21.15.12:0/195995212 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fbc1021a0 0x7f2fbc197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.077+0000 7f2fb9400640 1 -- 172.21.15.12:0/195995212 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fbc19f670 con 0x7f2fbc104ad0 2024-03-23T23:01:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.077+0000 7f2fba800640 1 --2- 172.21.15.12:0/195995212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fbc107400 0x7f2fbc19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.078+0000 7f2fb9400640 1 --2- 172.21.15.12:0/195995212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fbc104ad0 0x7f2fbc197fe0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4000c00 tx=0x7f2fa402c640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.078+0000 7f2fb2a00640 1 -- 172.21.15.12:0/195995212 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa4041070 con 0x7f2fbc104ad0 2024-03-23T23:01:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.078+0000 7f2fb2a00640 1 -- 172.21.15.12:0/195995212 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2fa4004450 con 0x7f2fbc104ad0 2024-03-23T23:01:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.078+0000 7f2fb2a00640 1 -- 172.21.15.12:0/195995212 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa4003df0 con 0x7f2fbc104ad0 2024-03-23T23:01:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.078+0000 7f2fc1200640 1 -- 172.21.15.12:0/195995212 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fbc19f900 con 0x7f2fbc104ad0 2024-03-23T23:01:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.078+0000 7f2fc1200640 1 -- 172.21.15.12:0/195995212 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fbc19fda0 con 0x7f2fbc104ad0 2024-03-23T23:01:05.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.079+0000 7f2fc1200640 1 -- 172.21.15.12:0/195995212 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fbc0fd620 con 0x7f2fbc104ad0 2024-03-23T23:01:05.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.081+0000 7f2fb2a00640 1 -- 172.21.15.12:0/195995212 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2fa4044070 con 0x7f2fbc104ad0 2024-03-23T23:01:05.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.083+0000 7f2fb2a00640 1 --2- 172.21.15.12:0/195995212 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2fa00a2800 0x7f2fa00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:05.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.083+0000 7f2fb2a00640 1 -- 172.21.15.12:0/195995212 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2fa40f0770 con 0x7f2fbc104ad0 2024-03-23T23:01:05.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.083+0000 7f2fb9e00640 1 --2- 172.21.15.12:0/195995212 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2fa00a2800 0x7f2fa00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:05.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.084+0000 7f2fb9e00640 1 --2- 172.21.15.12:0/195995212 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2fa00a2800 0x7f2fa00a4cc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2fac005fd0 tx=0x7f2fac009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:05.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.085+0000 7f2fb2a00640 1 -- 172.21.15.12:0/195995212 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2fa40bd850 con 0x7f2fbc104ad0 2024-03-23T23:01:05.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.289+0000 7f2fc1200640 1 -- 172.21.15.12:0/195995212 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fbc02d240 con 0x7f2fbc104ad0 2024-03-23T23:01:05.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.291+0000 7f2fb2a00640 1 -- 172.21.15.12:0/195995212 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22695 (secure 0 0 0) 0x7f2fa40bd150 con 0x7f2fbc104ad0 2024-03-23T23:01:05.291 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:01:05.292 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":88,"fsid":"90f679ae-e967-11ee-95cd-87774f69a715","created":"2024-03-23T22:50:14.906952+0000","modified":"2024-03-23T23:00:24.349052+0000","last_up_change":"2024-03-23T22:58:28.907824+0000","last_in_change":"2024-03-23T22:58:08.342487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T22:54:11.831010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T22:59:25.350974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T22:59:26.390323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b70875d7-a23d-4251-9e8b-1a16d4edc2a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6803","nonce":1190079611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6805","nonce":1190079611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6809","nonce":1190079611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6807","nonce":1190079611}]},"public_addr":"172.21.15.12:6803/1190079611","cluster_addr":"172.21.15.12:6805/1190079611","heartbeat_back_addr":"172.21.15.12:6809/1190079611","heartbeat_front_addr":"172.21.15.12:6807/1190079611","state":["exists","up"]},{"osd":1,"uuid":"416e87af-db5d-41ce-84d9-731e3e07e3c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6801","nonce":1983639525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6803","nonce":1983639525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6807","nonce":1983639525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6805","nonce":1983639525}]},"public_addr":"172.21.15.173:6801/1983639525","cluster_addr":"172.21.15.173:6803/1983639525","heartbeat_back_addr":"172.21.15.173:6807/1983639525","heartbeat_front_addr":"172.21.15.173:6805/1983639525","state":["exists","up"]},{"osd":2,"uuid":"7a559c70-4c54-431b-8baf-4136b6e76493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6801","nonce":2253997387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6803","nonce":2253997387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6807","nonce":2253997387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6805","nonce":2253997387}]},"public_addr":"172.21.15.203:6801/2253997387","cluster_addr":"172.21.15.203:6803/2253997387","heartbeat_back_addr":"172.21.15.203:6807/2253997387","heartbeat_front_addr":"172.21.15.203:6805/2253997387","state":["exists","up"]},{"osd":3,"uuid":"c41cd287-a674-486c-8025-328cd36e5ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6811","nonce":1761446905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6813","nonce":1761446905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6817","nonce":1761446905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6815","nonce":1761446905}]},"public_addr":"172.21.15.12:6811/1761446905","cluster_addr":"172.21.15.12:6813/1761446905","heartbeat_back_addr":"172.21.15.12:6817/1761446905","heartbeat_front_addr":"172.21.15.12:6815/1761446905","state":["exists","up"]},{"osd":4,"uuid":"3c48f886-0b42-4a09-b31e-abccc200bb1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6809","nonce":242958272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6811","nonce":242958272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6815","nonce":242958272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6813","nonce":242958272}]},"public_addr":"172.21.15.173:6809/242958272","cluster_addr":"172.21.15.173:6811/242958272","heartbeat_back_addr":"172.21.15.173:6815/242958272","heartbeat_front_addr":"172.21.15.173:6813/242958272","state":["exists","up"]},{"osd":5,"uuid":"4a34f06f-8805-40c0-b1ab-23407b0276d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6809","nonce":4270966092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6811","nonce":4270966092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6815","nonce":4270966092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6813","nonce":4270966092}]},"public_addr":"172.21.15.203:6809/4270966092","cluster_addr":"172.21.15.203:6811/4270966092","heartbeat_back_addr":"172.21.15.203:6815/4270966092","heartbeat_front_addr":"172.21.15.203:6813/4270966092","state":["exists","up"]},{"osd":6,"uuid":"26ddc663-5623-48c9-80d1-f8b968765d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6819","nonce":3606839295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6821","nonce":3606839295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6825","nonce":3606839295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6823","nonce":3606839295}]},"public_addr":"172.21.15.12:6819/3606839295","cluster_addr":"172.21.15.12:6821/3606839295","heartbeat_back_addr":"172.21.15.12:6825/3606839295","heartbeat_front_addr":"172.21.15.12:6823/3606839295","state":["exists","up"]},{"osd":7,"uuid":"17dc9417-71cd-4576-aa08-933212ec55e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6817","nonce":1241504346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6819","nonce":1241504346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6823","nonce":1241504346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6821","nonce":1241504346}]},"public_addr":"172.21.15.173:6817/1241504346","cluster_addr":"172.21.15.173:6819/1241504346","heartbeat_back_addr":"172.21.15.173:6823/1241504346","heartbeat_front_addr":"172.21.15.173:6821/1241504346","state":["exists","up"]},{"osd":8,"uuid":"90a2b63a-471d-43b8-ac18-bb1f16778d18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6817","nonce":1516996192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6819","nonce":1516996192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6823","nonce":1516996192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6821","nonce":1516996192}]},"public_addr":"172.21.15.203:6817/1516996192","cluster_addr":"172.21.15.203:6819/1516996192","heartbeat_back_addr":"172.21.15.203:6823/1516996192","heartbeat_front_addr":"172.21.15.203:6821/1516996192","state":["exists","up"]},{"osd":9,"uuid":"6e1ca504-40b4-49d0-b245-7946181abfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6827","nonce":1175260774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6829","nonce":1175260774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6833","nonce":1175260774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6831","nonce":1175260774}]},"public_addr":"172.21.15.12:6827/1175260774","cluster_addr":"172.21.15.12:6829/1175260774","heartbeat_back_addr":"172.21.15.12:6833/1175260774","heartbeat_front_addr":"172.21.15.12:6831/1175260774","state":["exists","up"]},{"osd":10,"uuid":"09a595ed-5614-4386-8e61-0170af66e01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6825","nonce":3303506808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6827","nonce":3303506808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6831","nonce":3303506808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6829","nonce":3303506808}]},"public_addr":"172.21.15.173:6825/3303506808","cluster_addr":"172.21.15.173:6827/3303506808","heartbeat_back_addr":"172.21.15.173:6831/3303506808","heartbeat_front_addr":"172.21.15.173:6829/3303506808","state":["exists","up"]},{"osd":11,"uuid":"9ef1d595-a977-4ba7-9c6b-d2b3295f20b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6825","nonce":2323604506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6827","nonce":2323604506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6831","nonce":2323604506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6829","nonce":2323604506}]},"public_addr":"172.21.15.203:6825/2323604506","cluster_addr":"172.21.15.203:6827/2323604506","heartbeat_back_addr":"172.21.15.203:6831/2323604506","heartbeat_front_addr":"172.21.15.203:6829/2323604506","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:18.469038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:42.752628+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:07.069493+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:41.962478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:08.363104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:33.341336+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:07.286541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:34.243302+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:59.749699+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:57:33.822913+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:00.737269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:26.599854+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/4182783120":"2024-03-24T22:51:23.113201+0000","172.21.15.12:6800/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1900938637":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/3414861652":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6801/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/2580541477":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6800/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/501058839":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1192554716":"2024-03-24T22:50:48.401223+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T23:01:05.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.294+0000 7f2fc1200640 1 -- 172.21.15.12:0/195995212 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2fa00a2800 msgr2=0x7f2fa00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:05.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.294+0000 7f2fc1200640 1 --2- 172.21.15.12:0/195995212 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2fa00a2800 0x7f2fa00a4cc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2fac005fd0 tx=0x7f2fac009500 comp rx=0 tx=0).stop 2024-03-23T23:01:05.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.294+0000 7f2fc1200640 1 -- 172.21.15.12:0/195995212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fbc104ad0 msgr2=0x7f2fbc197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:05.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.294+0000 7f2fc1200640 1 --2- 172.21.15.12:0/195995212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fbc104ad0 0x7f2fbc197fe0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4000c00 tx=0x7f2fa402c640 comp rx=0 tx=0).stop 2024-03-23T23:01:05.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.294+0000 7f2fc1200640 1 -- 172.21.15.12:0/195995212 shutdown_connections 2024-03-23T23:01:05.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.294+0000 7f2fc1200640 1 --2- 172.21.15.12:0/195995212 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2fa00a2800 0x7f2fa00a4cc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:05.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.294+0000 7f2fc1200640 1 --2- 172.21.15.12:0/195995212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fbc107400 0x7f2fbc19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:05.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.294+0000 7f2fc1200640 1 --2- 172.21.15.12:0/195995212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fbc104ad0 0x7f2fbc197fe0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:05.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.294+0000 7f2fc1200640 1 --2- 172.21.15.12:0/195995212 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fbc1021a0 0x7f2fbc197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:05.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.295+0000 7f2fc1200640 1 -- 172.21.15.12:0/195995212 >> 172.21.15.12:0/195995212 conn(0x7f2fbc0fbb30 msgr2=0x7f2fbc102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:05.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.295+0000 7f2fc1200640 1 -- 172.21.15.12:0/195995212 shutdown_connections 2024-03-23T23:01:05.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.295+0000 7f2fc1200640 1 -- 172.21.15.12:0/195995212 wait complete. 2024-03-23T23:01:05.316 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T22:54:11.831010+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T22:59:25.350974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T22:59:26.390323+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T23:01:05.316 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T23:01:05.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.544+0000 7f9016000640 1 -- 172.21.15.12:0/2201388033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9010104990 msgr2=0x7f9010104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:05.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.544+0000 7f9016000640 1 --2- 172.21.15.12:0/2201388033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9010104990 0x7f9010104e10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9000009900 tx=0x7f900002bc30 comp rx=0 tx=0).stop 2024-03-23T23:01:05.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.544+0000 7f9016000640 1 -- 172.21.15.12:0/2201388033 shutdown_connections 2024-03-23T23:01:05.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.544+0000 7f9016000640 1 --2- 172.21.15.12:0/2201388033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9010105350 0x7f901010bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:05.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.544+0000 7f9016000640 1 --2- 172.21.15.12:0/2201388033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9010104990 0x7f9010104e10 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:05.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.544+0000 7f9016000640 1 --2- 172.21.15.12:0/2201388033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90101037a0 0x7f9010103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:05.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.544+0000 7f9016000640 1 -- 172.21.15.12:0/2201388033 >> 172.21.15.12:0/2201388033 conn(0x7f90100fed50 msgr2=0x7f9010101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:05.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.544+0000 7f9016000640 1 -- 172.21.15.12:0/2201388033 shutdown_connections 2024-03-23T23:01:05.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.544+0000 7f9016000640 1 -- 172.21.15.12:0/2201388033 wait complete. 2024-03-23T23:01:05.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.545+0000 7f9016000640 1 Processor -- start 2024-03-23T23:01:05.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.545+0000 7f9016000640 1 -- start start 2024-03-23T23:01:05.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.546+0000 7f9016000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90101037a0 0x7f9010199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:05.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.546+0000 7f9016000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9010104990 0x7f901019a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:05.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.546+0000 7f9016000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9010105350 0x7f90101a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:05.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.546+0000 7f9016000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901010f420 con 0x7f90101037a0 2024-03-23T23:01:05.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.546+0000 7f9016000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901010f2a0 con 0x7f9010105350 2024-03-23T23:01:05.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.546+0000 7f9016000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901010f5a0 con 0x7f9010104990 2024-03-23T23:01:05.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.546+0000 7f9014c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90101037a0 0x7f9010199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:05.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.546+0000 7f9015600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9010105350 0x7f90101a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:05.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.546+0000 7f900fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9010104990 0x7f901019a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:05.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.546+0000 7f9015600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9010105350 0x7f90101a10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:56828/0 (socket says 172.21.15.12:56828) 2024-03-23T23:01:05.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.546+0000 7f9015600640 1 -- 172.21.15.12:0/2948024599 learned_addr learned my addr 172.21.15.12:0/2948024599 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:05.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.546+0000 7f9015600640 1 -- 172.21.15.12:0/2948024599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9010104990 msgr2=0x7f901019a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:05.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.546+0000 7f9015600640 1 --2- 172.21.15.12:0/2948024599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9010104990 0x7f901019a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:05.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.546+0000 7f9015600640 1 -- 172.21.15.12:0/2948024599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90101037a0 msgr2=0x7f9010199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:05.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.546+0000 7f9015600640 1 --2- 172.21.15.12:0/2948024599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90101037a0 0x7f9010199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:05.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.547+0000 7f9015600640 1 -- 172.21.15.12:0/2948024599 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90101a1720 con 0x7f9010105350 2024-03-23T23:01:05.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.547+0000 7f9015600640 1 --2- 172.21.15.12:0/2948024599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9010105350 0x7f90101a10b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9008009fd0 tx=0x7f900800b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:05.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.547+0000 7f900d600640 1 -- 172.21.15.12:0/2948024599 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9008017070 con 0x7f9010105350 2024-03-23T23:01:05.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.547+0000 7f900d600640 1 -- 172.21.15.12:0/2948024599 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9008003e50 con 0x7f9010105350 2024-03-23T23:01:05.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.547+0000 7f9016000640 1 -- 172.21.15.12:0/2948024599 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90101a1a10 con 0x7f9010105350 2024-03-23T23:01:05.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.547+0000 7f900d600640 1 -- 172.21.15.12:0/2948024599 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90080106c0 con 0x7f9010105350 2024-03-23T23:01:05.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.547+0000 7f9016000640 1 -- 172.21.15.12:0/2948024599 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90101a1ee0 con 0x7f9010105350 2024-03-23T23:01:05.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.549+0000 7f9016000640 1 -- 172.21.15.12:0/2948024599 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f901010ab10 con 0x7f9010105350 2024-03-23T23:01:05.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.550+0000 7f900d600640 1 -- 172.21.15.12:0/2948024599 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9008003970 con 0x7f9010105350 2024-03-23T23:01:05.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.551+0000 7f900d600640 1 --2- 172.21.15.12:0/2948024599 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8ff80a2800 0x7f8ff80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:05.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.552+0000 7f900d600640 1 -- 172.21.15.12:0/2948024599 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f90080ce7b0 con 0x7f9010105350 2024-03-23T23:01:05.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.552+0000 7f9014c00640 1 --2- 172.21.15.12:0/2948024599 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8ff80a2800 0x7f8ff80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:05.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.552+0000 7f9014c00640 1 --2- 172.21.15.12:0/2948024599 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8ff80a2800 0x7f8ff80a4cc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f90101047f0 tx=0x7f9004005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:05.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.556+0000 7f900d600640 1 -- 172.21.15.12:0/2948024599 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f900809b1b0 con 0x7f9010105350 2024-03-23T23:01:05.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:05 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2825622483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:01:05.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:05 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/195995212' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:05.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.760+0000 7f9016000640 1 -- 172.21.15.12:0/2948024599 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f90100622a0 con 0x7f9010105350 2024-03-23T23:01:05.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.761+0000 7f900d600640 1 -- 172.21.15.12:0/2948024599 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f900809f250 con 0x7f9010105350 2024-03-23T23:01:05.761 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-23T23:01:05.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.763+0000 7f9016000640 1 -- 172.21.15.12:0/2948024599 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8ff80a2800 msgr2=0x7f8ff80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:05.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.763+0000 7f9016000640 1 --2- 172.21.15.12:0/2948024599 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8ff80a2800 0x7f8ff80a4cc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f90101047f0 tx=0x7f9004005f90 comp rx=0 tx=0).stop 2024-03-23T23:01:05.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.763+0000 7f9016000640 1 -- 172.21.15.12:0/2948024599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9010105350 msgr2=0x7f90101a10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:05.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.763+0000 7f9016000640 1 --2- 172.21.15.12:0/2948024599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9010105350 0x7f90101a10b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9008009fd0 tx=0x7f900800b260 comp rx=0 tx=0).stop 2024-03-23T23:01:05.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.764+0000 7f9016000640 1 -- 172.21.15.12:0/2948024599 shutdown_connections 2024-03-23T23:01:05.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.764+0000 7f9016000640 1 --2- 172.21.15.12:0/2948024599 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8ff80a2800 0x7f8ff80a4cc0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:05.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.764+0000 7f9016000640 1 --2- 172.21.15.12:0/2948024599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9010105350 0x7f90101a10b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:05.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.764+0000 7f9016000640 1 --2- 172.21.15.12:0/2948024599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9010104990 0x7f901019a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:05.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.764+0000 7f9016000640 1 --2- 172.21.15.12:0/2948024599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90101037a0 0x7f9010199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:05.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.764+0000 7f9016000640 1 -- 172.21.15.12:0/2948024599 >> 172.21.15.12:0/2948024599 conn(0x7f90100fed50 msgr2=0x7f9010107ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:05.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.764+0000 7f9016000640 1 -- 172.21.15.12:0/2948024599 shutdown_connections 2024-03-23T23:01:05.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.764+0000 7f9016000640 1 -- 172.21.15.12:0/2948024599 wait complete. 2024-03-23T23:01:05.779 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:05 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2825622483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:01:05.780 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:05 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/195995212' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:05.783 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T23:01:05.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:05 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2825622483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:01:05.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:05 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/195995212' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:06.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.997+0000 7f3881e00640 1 -- 172.21.15.12:0/3523217454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f387c105490 msgr2=0x7f387c10bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:06.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.997+0000 7f3881e00640 1 --2- 172.21.15.12:0/3523217454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f387c105490 0x7f387c10bd20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3874009be0 tx=0x7f387402bef0 comp rx=0 tx=0).stop 2024-03-23T23:01:06.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.997+0000 7f3881e00640 1 -- 172.21.15.12:0/3523217454 shutdown_connections 2024-03-23T23:01:06.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.997+0000 7f3881e00640 1 --2- 172.21.15.12:0/3523217454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f387c105490 0x7f387c10bd20 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:06.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.997+0000 7f3881e00640 1 --2- 172.21.15.12:0/3523217454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f387c104ad0 0x7f387c104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:06.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.997+0000 7f3881e00640 1 --2- 172.21.15.12:0/3523217454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f387c1038e0 0x7f387c103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:06.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.997+0000 7f3881e00640 1 -- 172.21.15.12:0/3523217454 >> 172.21.15.12:0/3523217454 conn(0x7f387c0fee50 msgr2=0x7f387c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:06.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.997+0000 7f3881e00640 1 -- 172.21.15.12:0/3523217454 shutdown_connections 2024-03-23T23:01:06.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.998+0000 7f3881e00640 1 -- 172.21.15.12:0/3523217454 wait complete. 2024-03-23T23:01:06.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.998+0000 7f3881e00640 1 Processor -- start 2024-03-23T23:01:06.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.999+0000 7f3881e00640 1 -- start start 2024-03-23T23:01:06.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.999+0000 7f3881e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f387c1038e0 0x7f387c199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:06.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.999+0000 7f3881e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f387c104ad0 0x7f387c19a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:06.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.999+0000 7f3881e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f387c105490 0x7f387c1a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:06.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.999+0000 7f3881e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387c10f5b0 con 0x7f387c1038e0 2024-03-23T23:01:06.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.999+0000 7f3881e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387c10f430 con 0x7f387c104ad0 2024-03-23T23:01:06.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.999+0000 7f3881e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387c10f730 con 0x7f387c105490 2024-03-23T23:01:06.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.999+0000 7f387b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f387c105490 0x7f387c1a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:06.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.999+0000 7f387b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f387c105490 0x7f387c1a1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49086/0 (socket says 172.21.15.12:49086) 2024-03-23T23:01:06.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.999+0000 7f387b400640 1 -- 172.21.15.12:0/865756966 learned_addr learned my addr 172.21.15.12:0/865756966 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:06.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.999+0000 7f387aa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f387c1038e0 0x7f387c199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:06.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:05.999+0000 7f3873e00640 1 --2- 172.21.15.12:0/865756966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f387c104ad0 0x7f387c19a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:06.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.000+0000 7f387b400640 1 -- 172.21.15.12:0/865756966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f387c104ad0 msgr2=0x7f387c19a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:06.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.000+0000 7f387b400640 1 --2- 172.21.15.12:0/865756966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f387c104ad0 0x7f387c19a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:06.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.000+0000 7f387b400640 1 -- 172.21.15.12:0/865756966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f387c1038e0 msgr2=0x7f387c199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:06.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.000+0000 7f387b400640 1 --2- 172.21.15.12:0/865756966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f387c1038e0 0x7f387c199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:06.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.000+0000 7f387b400640 1 -- 172.21.15.12:0/865756966 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387c1a17e0 con 0x7f387c105490 2024-03-23T23:01:06.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.000+0000 7f3873e00640 1 --2- 172.21.15.12:0/865756966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f387c104ad0 0x7f387c19a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:06.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.000+0000 7f387b400640 1 --2- 172.21.15.12:0/865756966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f387c105490 0x7f387c1a1170 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3874036040 tx=0x7f38740337b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:06.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.000+0000 7f3880a00640 1 -- 172.21.15.12:0/865756966 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3874043020 con 0x7f387c105490 2024-03-23T23:01:06.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.000+0000 7f3880a00640 1 -- 172.21.15.12:0/865756966 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f387402ca60 con 0x7f387c105490 2024-03-23T23:01:06.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.000+0000 7f3881e00640 1 -- 172.21.15.12:0/865756966 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f387c1a1a70 con 0x7f387c105490 2024-03-23T23:01:06.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.000+0000 7f3880a00640 1 -- 172.21.15.12:0/865756966 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3874034690 con 0x7f387c105490 2024-03-23T23:01:06.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.001+0000 7f3881e00640 1 -- 172.21.15.12:0/865756966 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f387c1a1fc0 con 0x7f387c105490 2024-03-23T23:01:06.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.002+0000 7f3881e00640 1 -- 172.21.15.12:0/865756966 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f387c10ac00 con 0x7f387c105490 2024-03-23T23:01:06.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.004+0000 7f3880a00640 1 -- 172.21.15.12:0/865756966 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3874046020 con 0x7f387c105490 2024-03-23T23:01:06.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.005+0000 7f3880a00640 1 --2- 172.21.15.12:0/865756966 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f38600a2800 0x7f38600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:06.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.005+0000 7f3880a00640 1 -- 172.21.15.12:0/865756966 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f38740f1830 con 0x7f387c105490 2024-03-23T23:01:06.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.005+0000 7f387aa00640 1 --2- 172.21.15.12:0/865756966 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f38600a2800 0x7f38600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:06.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.006+0000 7f387aa00640 1 --2- 172.21.15.12:0/865756966 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f38600a2800 0x7f38600a4cc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f387c104930 tx=0x7f386c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:06.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.007+0000 7f3880a00640 1 -- 172.21.15.12:0/865756966 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f387404d020 con 0x7f387c105490 2024-03-23T23:01:06.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.210+0000 7f3881e00640 1 -- 172.21.15.12:0/865756966 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f387c02d200 con 0x7f387c105490 2024-03-23T23:01:06.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.211+0000 7f3880a00640 1 -- 172.21.15.12:0/865756966 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f38740be9a0 con 0x7f387c105490 2024-03-23T23:01:06.211 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-23T23:01:06.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.213+0000 7f3881e00640 1 -- 172.21.15.12:0/865756966 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f38600a2800 msgr2=0x7f38600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:06.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.213+0000 7f3881e00640 1 --2- 172.21.15.12:0/865756966 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f38600a2800 0x7f38600a4cc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f387c104930 tx=0x7f386c009260 comp rx=0 tx=0).stop 2024-03-23T23:01:06.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.213+0000 7f3881e00640 1 -- 172.21.15.12:0/865756966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f387c105490 msgr2=0x7f387c1a1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:06.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.213+0000 7f3881e00640 1 --2- 172.21.15.12:0/865756966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f387c105490 0x7f387c1a1170 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3874036040 tx=0x7f38740337b0 comp rx=0 tx=0).stop 2024-03-23T23:01:06.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.213+0000 7f3881e00640 1 -- 172.21.15.12:0/865756966 shutdown_connections 2024-03-23T23:01:06.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.213+0000 7f3881e00640 1 --2- 172.21.15.12:0/865756966 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f38600a2800 0x7f38600a4cc0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:06.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.213+0000 7f3881e00640 1 --2- 172.21.15.12:0/865756966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f387c105490 0x7f387c1a1170 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:06.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.213+0000 7f3881e00640 1 --2- 172.21.15.12:0/865756966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f387c104ad0 0x7f387c19a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:06.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.213+0000 7f3881e00640 1 --2- 172.21.15.12:0/865756966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f387c1038e0 0x7f387c199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:06.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.213+0000 7f3881e00640 1 -- 172.21.15.12:0/865756966 >> 172.21.15.12:0/865756966 conn(0x7f387c0fee50 msgr2=0x7f387c107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:06.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.213+0000 7f3881e00640 1 -- 172.21.15.12:0/865756966 shutdown_connections 2024-03-23T23:01:06.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.214+0000 7f3881e00640 1 -- 172.21.15.12:0/865756966 wait complete. 2024-03-23T23:01:06.231 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T23:01:06.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.457+0000 7f25d8200640 1 -- 172.21.15.12:0/3118338398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25d0109870 msgr2=0x7f25d0110100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:06.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.457+0000 7f25d8200640 1 --2- 172.21.15.12:0/3118338398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25d0109870 0x7f25d0110100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f25c8009ca0 tx=0x7f25c802c100 comp rx=0 tx=0).stop 2024-03-23T23:01:06.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.457+0000 7f25d8200640 1 -- 172.21.15.12:0/3118338398 shutdown_connections 2024-03-23T23:01:06.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.457+0000 7f25d8200640 1 --2- 172.21.15.12:0/3118338398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25d0109870 0x7f25d0110100 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:06.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.457+0000 7f25d8200640 1 --2- 172.21.15.12:0/3118338398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25d0108eb0 0x7f25d0109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:06.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.457+0000 7f25d8200640 1 --2- 172.21.15.12:0/3118338398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25d0107cc0 0x7f25d01080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:06.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.457+0000 7f25d8200640 1 -- 172.21.15.12:0/3118338398 >> 172.21.15.12:0/3118338398 conn(0x7f25d0076c60 msgr2=0x7f25d00790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:06.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.457+0000 7f25d8200640 1 -- 172.21.15.12:0/3118338398 shutdown_connections 2024-03-23T23:01:06.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.457+0000 7f25d8200640 1 -- 172.21.15.12:0/3118338398 wait complete. 2024-03-23T23:01:06.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.458+0000 7f25d8200640 1 Processor -- start 2024-03-23T23:01:06.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.458+0000 7f25d8200640 1 -- start start 2024-03-23T23:01:06.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.458+0000 7f25d8200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25d0107cc0 0x7f25d019df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:06.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.458+0000 7f25d8200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25d0108eb0 0x7f25d019e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:06.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.458+0000 7f25d8200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25d0109870 0x7f25d01a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:06.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.458+0000 7f25d8200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d0113940 con 0x7f25d0108eb0 2024-03-23T23:01:06.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.458+0000 7f25d8200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d01137c0 con 0x7f25d0107cc0 2024-03-23T23:01:06.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.458+0000 7f25d8200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d0113ac0 con 0x7f25d0109870 2024-03-23T23:01:06.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.459+0000 7f25d5800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25d0107cc0 0x7f25d019df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:06.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.459+0000 7f25d5800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25d0107cc0 0x7f25d019df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:56876/0 (socket says 172.21.15.12:56876) 2024-03-23T23:01:06.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.459+0000 7f25d5800640 1 -- 172.21.15.12:0/4056792430 learned_addr learned my addr 172.21.15.12:0/4056792430 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:06.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.459+0000 7f25d6200640 1 --2- 172.21.15.12:0/4056792430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25d0109870 0x7f25d01a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.459+0000 7f25d4e00640 1 --2- 172.21.15.12:0/4056792430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25d0108eb0 0x7f25d019e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.459+0000 7f25d5800640 1 -- 172.21.15.12:0/4056792430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25d0109870 msgr2=0x7f25d01a5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.459+0000 7f25d5800640 1 --2- 172.21.15.12:0/4056792430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25d0109870 0x7f25d01a5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.459+0000 7f25d5800640 1 -- 172.21.15.12:0/4056792430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25d0108eb0 msgr2=0x7f25d019e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.459+0000 7f25d5800640 1 --2- 172.21.15.12:0/4056792430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25d0108eb0 0x7f25d019e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.459+0000 7f25d5800640 1 -- 172.21.15.12:0/4056792430 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d01a5b80 con 0x7f25d0107cc0 2024-03-23T23:01:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.459+0000 7f25d4e00640 1 --2- 172.21.15.12:0/4056792430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25d0108eb0 0x7f25d019e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:01:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.459+0000 7f25d5800640 1 --2- 172.21.15.12:0/4056792430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25d0107cc0 0x7f25d019df60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f25c400e9c0 tx=0x7f25c400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.460+0000 7f25ce000640 1 -- 172.21.15.12:0/4056792430 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c400cd80 con 0x7f25d0107cc0 2024-03-23T23:01:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.460+0000 7f25ce000640 1 -- 172.21.15.12:0/4056792430 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f25c40035f0 con 0x7f25d0107cc0 2024-03-23T23:01:06.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.460+0000 7f25ce000640 1 -- 172.21.15.12:0/4056792430 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c4010690 con 0x7f25d0107cc0 2024-03-23T23:01:06.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.460+0000 7f25d8200640 1 -- 172.21.15.12:0/4056792430 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d01a5e70 con 0x7f25d0107cc0 2024-03-23T23:01:06.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.460+0000 7f25d8200640 1 -- 172.21.15.12:0/4056792430 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d01a6280 con 0x7f25d0107cc0 2024-03-23T23:01:06.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.461+0000 7f25d8200640 1 -- 172.21.15.12:0/4056792430 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d010efe0 con 0x7f25d0107cc0 2024-03-23T23:01:06.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.463+0000 7f25ce000640 1 -- 172.21.15.12:0/4056792430 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f25c40037a0 con 0x7f25d0107cc0 2024-03-23T23:01:06.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.464+0000 7f25ce000640 1 --2- 172.21.15.12:0/4056792430 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f25b80a2800 0x7f25b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:06.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.464+0000 7f25ce000640 1 -- 172.21.15.12:0/4056792430 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f25c40ceec0 con 0x7f25d0107cc0 2024-03-23T23:01:06.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.464+0000 7f25d4e00640 1 --2- 172.21.15.12:0/4056792430 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f25b80a2800 0x7f25b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:06.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.465+0000 7f25d4e00640 1 --2- 172.21.15.12:0/4056792430 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f25b80a2800 0x7f25b80a4cc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f25d019f470 tx=0x7f25bc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:06.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.468+0000 7f25ce000640 1 -- 172.21.15.12:0/4056792430 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f25c409bf10 con 0x7f25d0107cc0 2024-03-23T23:01:06.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:06 smithi012 ceph-mon[31695]: pgmap v369: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:01:06.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:06 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2948024599' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:01:06.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:06 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/865756966' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:01:06.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:06 smithi203 ceph-mon[36693]: pgmap v369: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:01:06.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:06 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2948024599' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:01:06.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:06 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/865756966' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:01:06.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.672+0000 7f25d8200640 1 -- 172.21.15.12:0/4056792430 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f25d00623a0 con 0x7f25d0107cc0 2024-03-23T23:01:06.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.673+0000 7f25ce000640 1 -- 172.21.15.12:0/4056792430 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f25c409b810 con 0x7f25d0107cc0 2024-03-23T23:01:06.673 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-23T23:01:06.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.676+0000 7f25d8200640 1 -- 172.21.15.12:0/4056792430 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f25b80a2800 msgr2=0x7f25b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:06.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.676+0000 7f25d8200640 1 --2- 172.21.15.12:0/4056792430 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f25b80a2800 0x7f25b80a4cc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f25d019f470 tx=0x7f25bc005f90 comp rx=0 tx=0).stop 2024-03-23T23:01:06.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.676+0000 7f25d8200640 1 -- 172.21.15.12:0/4056792430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25d0107cc0 msgr2=0x7f25d019df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:06.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.676+0000 7f25d8200640 1 --2- 172.21.15.12:0/4056792430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25d0107cc0 0x7f25d019df60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f25c400e9c0 tx=0x7f25c400ee80 comp rx=0 tx=0).stop 2024-03-23T23:01:06.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.676+0000 7f25d8200640 1 -- 172.21.15.12:0/4056792430 shutdown_connections 2024-03-23T23:01:06.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.676+0000 7f25d8200640 1 --2- 172.21.15.12:0/4056792430 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f25b80a2800 0x7f25b80a4cc0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:06.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.676+0000 7f25d8200640 1 --2- 172.21.15.12:0/4056792430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25d0109870 0x7f25d01a5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:06.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.676+0000 7f25d8200640 1 --2- 172.21.15.12:0/4056792430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25d0108eb0 0x7f25d019e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:06.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.676+0000 7f25d8200640 1 --2- 172.21.15.12:0/4056792430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25d0107cc0 0x7f25d019df60 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:06.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.676+0000 7f25d8200640 1 -- 172.21.15.12:0/4056792430 >> 172.21.15.12:0/4056792430 conn(0x7f25d0076c60 msgr2=0x7f25d010c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:06.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.676+0000 7f25d8200640 1 -- 172.21.15.12:0/4056792430 shutdown_connections 2024-03-23T23:01:06.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.676+0000 7f25d8200640 1 -- 172.21.15.12:0/4056792430 wait complete. 2024-03-23T23:01:06.695 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:01:06.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:06 smithi173 ceph-mon[36116]: pgmap v369: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:01:06.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:06 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2948024599' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:01:06.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:06 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/865756966' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:01:06.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.917+0000 7fae0e000640 1 -- 172.21.15.12:0/861435744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae08104ad0 msgr2=0x7fae08104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:06.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.917+0000 7fae0e000640 1 --2- 172.21.15.12:0/861435744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae08104ad0 0x7fae08104f50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fadf0009900 tx=0x7fadf002bc30 comp rx=0 tx=0).stop 2024-03-23T23:01:06.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.917+0000 7fae0e000640 1 -- 172.21.15.12:0/861435744 shutdown_connections 2024-03-23T23:01:06.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.917+0000 7fae0e000640 1 --2- 172.21.15.12:0/861435744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae08105490 0x7fae0810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:06.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.917+0000 7fae0e000640 1 --2- 172.21.15.12:0/861435744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae08104ad0 0x7fae08104f50 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:06.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.917+0000 7fae0e000640 1 --2- 172.21.15.12:0/861435744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae081038e0 0x7fae08103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:06.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.917+0000 7fae0e000640 1 -- 172.21.15.12:0/861435744 >> 172.21.15.12:0/861435744 conn(0x7fae080fee50 msgr2=0x7fae08101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:06.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.917+0000 7fae0e000640 1 -- 172.21.15.12:0/861435744 shutdown_connections 2024-03-23T23:01:06.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.917+0000 7fae0e000640 1 -- 172.21.15.12:0/861435744 wait complete. 2024-03-23T23:01:06.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.918+0000 7fae0e000640 1 Processor -- start 2024-03-23T23:01:06.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.918+0000 7fae0e000640 1 -- start start 2024-03-23T23:01:06.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.918+0000 7fae0e000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae081038e0 0x7fae08199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:06.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.918+0000 7fae0e000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae08104ad0 0x7fae0819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:06.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.918+0000 7fae0e000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae08105490 0x7fae081a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:06.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.918+0000 7fae0e000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0810f5b0 con 0x7fae08105490 2024-03-23T23:01:06.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.918+0000 7fae0e000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0810f430 con 0x7fae081038e0 2024-03-23T23:01:06.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.918+0000 7fae0e000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0810f730 con 0x7fae08104ad0 2024-03-23T23:01:06.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.918+0000 7fae07400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae081038e0 0x7fae08199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:06.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.919+0000 7fae07400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae081038e0 0x7fae08199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:56894/0 (socket says 172.21.15.12:56894) 2024-03-23T23:01:06.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.919+0000 7fae07400640 1 -- 172.21.15.12:0/3164192934 learned_addr learned my addr 172.21.15.12:0/3164192934 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:06.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.919+0000 7fadfea00640 1 --2- 172.21.15.12:0/3164192934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae08104ad0 0x7fae0819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:06.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.919+0000 7fae07e00640 1 --2- 172.21.15.12:0/3164192934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae08105490 0x7fae081a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:06.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.919+0000 7fae07400640 1 -- 172.21.15.12:0/3164192934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae08104ad0 msgr2=0x7fae0819a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:06.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.919+0000 7fae07400640 1 --2- 172.21.15.12:0/3164192934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae08104ad0 0x7fae0819a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:06.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.919+0000 7fae07400640 1 -- 172.21.15.12:0/3164192934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae08105490 msgr2=0x7fae081a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:06.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.919+0000 7fae07400640 1 --2- 172.21.15.12:0/3164192934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae08105490 0x7fae081a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:06.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.919+0000 7fae07400640 1 -- 172.21.15.12:0/3164192934 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae081a1830 con 0x7fae081038e0 2024-03-23T23:01:06.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.919+0000 7fae07400640 1 --2- 172.21.15.12:0/3164192934 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae081038e0 0x7fae08199c20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fadf800d950 tx=0x7fadf800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:06.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.919+0000 7fae04c00640 1 -- 172.21.15.12:0/3164192934 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf8017070 con 0x7fae081038e0 2024-03-23T23:01:06.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.920+0000 7fae0e000640 1 -- 172.21.15.12:0/3164192934 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae081a1b20 con 0x7fae081038e0 2024-03-23T23:01:06.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.920+0000 7fae04c00640 1 -- 172.21.15.12:0/3164192934 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fadf8009c30 con 0x7fae081038e0 2024-03-23T23:01:06.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.920+0000 7fae04c00640 1 -- 172.21.15.12:0/3164192934 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf800f600 con 0x7fae081038e0 2024-03-23T23:01:06.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.920+0000 7fae0e000640 1 -- 172.21.15.12:0/3164192934 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae081a2020 con 0x7fae081038e0 2024-03-23T23:01:06.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.921+0000 7fae0e000640 1 -- 172.21.15.12:0/3164192934 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae0810ac00 con 0x7fae081038e0 2024-03-23T23:01:06.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.923+0000 7fae04c00640 1 -- 172.21.15.12:0/3164192934 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fadf801a070 con 0x7fae081038e0 2024-03-23T23:01:06.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.924+0000 7fae04c00640 1 --2- 172.21.15.12:0/3164192934 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fadec0a2800 0x7fadec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:06.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.924+0000 7fae04c00640 1 -- 172.21.15.12:0/3164192934 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fadf80ce4e0 con 0x7fae081038e0 2024-03-23T23:01:06.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.924+0000 7fadfea00640 1 --2- 172.21.15.12:0/3164192934 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fadec0a2800 0x7fadec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:06.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.925+0000 7fadfea00640 1 --2- 172.21.15.12:0/3164192934 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fadec0a2800 0x7fadec0a4cc0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fadf002c240 tx=0x7fadf002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:06.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:06.931+0000 7fae04c00640 1 -- 172.21.15.12:0/3164192934 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fadf8009de0 con 0x7fae081038e0 2024-03-23T23:01:07.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.176+0000 7fae0e000640 1 -- 172.21.15.12:0/3164192934 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae0802d070 con 0x7fae081038e0 2024-03-23T23:01:07.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.177+0000 7fae04c00640 1 -- 172.21.15.12:0/3164192934 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11393 (secure 0 0 0) 0x7fadf809aee0 con 0x7fae081038e0 2024-03-23T23:01:07.178 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:01:07.178 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T22:59:32.190756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24599,24550]},"id":1}]} 2024-03-23T23:01:07.178 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 22 2024-03-23T23:01:07.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.180+0000 7fae0e000640 1 -- 172.21.15.12:0/3164192934 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fadec0a2800 msgr2=0x7fadec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:07.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.180+0000 7fae0e000640 1 --2- 172.21.15.12:0/3164192934 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fadec0a2800 0x7fadec0a4cc0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fadf002c240 tx=0x7fadf002df90 comp rx=0 tx=0).stop 2024-03-23T23:01:07.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.180+0000 7fae0e000640 1 -- 172.21.15.12:0/3164192934 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae081038e0 msgr2=0x7fae08199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:07.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.180+0000 7fae0e000640 1 --2- 172.21.15.12:0/3164192934 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae081038e0 0x7fae08199c20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fadf800d950 tx=0x7fadf800de10 comp rx=0 tx=0).stop 2024-03-23T23:01:07.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.180+0000 7fae0e000640 1 -- 172.21.15.12:0/3164192934 shutdown_connections 2024-03-23T23:01:07.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.180+0000 7fae0e000640 1 --2- 172.21.15.12:0/3164192934 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fadec0a2800 0x7fadec0a4cc0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:07.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.180+0000 7fae0e000640 1 --2- 172.21.15.12:0/3164192934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae08105490 0x7fae081a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:07.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.180+0000 7fae0e000640 1 --2- 172.21.15.12:0/3164192934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae08104ad0 0x7fae0819a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:07.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.180+0000 7fae0e000640 1 --2- 172.21.15.12:0/3164192934 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae081038e0 0x7fae08199c20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:07.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.180+0000 7fae0e000640 1 -- 172.21.15.12:0/3164192934 >> 172.21.15.12:0/3164192934 conn(0x7fae080fee50 msgr2=0x7fae08107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:07.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.180+0000 7fae0e000640 1 -- 172.21.15.12:0/3164192934 shutdown_connections 2024-03-23T23:01:07.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.180+0000 7fae0e000640 1 -- 172.21.15.12:0/3164192934 wait complete. 2024-03-23T23:01:07.203 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T23:01:07.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.434+0000 7f41aea00640 1 -- 172.21.15.12:0/2282738504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41a81016c0 msgr2=0x7f41a8101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:07.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.434+0000 7f41aea00640 1 --2- 172.21.15.12:0/2282738504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41a81016c0 0x7f41a8101ae0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4198009900 tx=0x7f419802bc30 comp rx=0 tx=0).stop 2024-03-23T23:01:07.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.434+0000 7f41aea00640 1 -- 172.21.15.12:0/2282738504 shutdown_connections 2024-03-23T23:01:07.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.434+0000 7f41aea00640 1 --2- 172.21.15.12:0/2282738504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41a8103270 0x7f41a8109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:07.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.434+0000 7f41aea00640 1 --2- 172.21.15.12:0/2282738504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41a81028b0 0x7f41a8102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:07.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.434+0000 7f41aea00640 1 --2- 172.21.15.12:0/2282738504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41a81016c0 0x7f41a8101ae0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:07.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.434+0000 7f41aea00640 1 -- 172.21.15.12:0/2282738504 >> 172.21.15.12:0/2282738504 conn(0x7f41a80fcc50 msgr2=0x7f41a80ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:07.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.434+0000 7f41aea00640 1 -- 172.21.15.12:0/2282738504 shutdown_connections 2024-03-23T23:01:07.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.434+0000 7f41aea00640 1 -- 172.21.15.12:0/2282738504 wait complete. 2024-03-23T23:01:07.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.435+0000 7f41aea00640 1 Processor -- start 2024-03-23T23:01:07.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.435+0000 7f41aea00640 1 -- start start 2024-03-23T23:01:07.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.435+0000 7f41aea00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41a81016c0 0x7f41a8197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:07.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.435+0000 7f41aea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41a81028b0 0x7f41a8197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:07.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.435+0000 7f41aea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41a8103270 0x7f41a819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:07.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.435+0000 7f41aea00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a810d390 con 0x7f41a81016c0 2024-03-23T23:01:07.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.435+0000 7f41aea00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a810d210 con 0x7f41a81028b0 2024-03-23T23:01:07.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.435+0000 7f41aea00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a810d510 con 0x7f41a8103270 2024-03-23T23:01:07.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.436+0000 7f41a7e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41a81016c0 0x7f41a8197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:07.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.436+0000 7f41a7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41a81028b0 0x7f41a8197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:07.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.436+0000 7f41a7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41a81028b0 0x7f41a8197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:56904/0 (socket says 172.21.15.12:56904) 2024-03-23T23:01:07.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.436+0000 7f41a7400640 1 -- 172.21.15.12:0/955487466 learned_addr learned my addr 172.21.15.12:0/955487466 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:07.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.436+0000 7f41aca00640 1 --2- 172.21.15.12:0/955487466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41a8103270 0x7f41a819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:07.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.436+0000 7f41a7400640 1 -- 172.21.15.12:0/955487466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41a8103270 msgr2=0x7f41a819efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:07.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.436+0000 7f41a7400640 1 --2- 172.21.15.12:0/955487466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41a8103270 0x7f41a819efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:07.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.436+0000 7f41a7400640 1 -- 172.21.15.12:0/955487466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41a81016c0 msgr2=0x7f41a8197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:07.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.436+0000 7f41a7400640 1 --2- 172.21.15.12:0/955487466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41a81016c0 0x7f41a8197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:07.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.436+0000 7f41a7400640 1 -- 172.21.15.12:0/955487466 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a819f620 con 0x7f41a81028b0 2024-03-23T23:01:07.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.436+0000 7f41a7400640 1 --2- 172.21.15.12:0/955487466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41a81028b0 0x7f41a8197f40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f419400ea60 tx=0x7f419400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:07.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.436+0000 7f41a4c00640 1 -- 172.21.15.12:0/955487466 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419400cde0 con 0x7f41a81028b0 2024-03-23T23:01:07.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.437+0000 7f41a4c00640 1 -- 172.21.15.12:0/955487466 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f41940035f0 con 0x7f41a81028b0 2024-03-23T23:01:07.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.437+0000 7f41aea00640 1 -- 172.21.15.12:0/955487466 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a819f910 con 0x7f41a81028b0 2024-03-23T23:01:07.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.437+0000 7f41a4c00640 1 -- 172.21.15.12:0/955487466 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4194010670 con 0x7f41a81028b0 2024-03-23T23:01:07.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.437+0000 7f41aea00640 1 -- 172.21.15.12:0/955487466 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a819fe60 con 0x7f41a81028b0 2024-03-23T23:01:07.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.438+0000 7f41aea00640 1 -- 172.21.15.12:0/955487466 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41a81089e0 con 0x7f41a81028b0 2024-03-23T23:01:07.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.440+0000 7f41a4c00640 1 -- 172.21.15.12:0/955487466 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f41940037a0 con 0x7f41a81028b0 2024-03-23T23:01:07.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.441+0000 7f41a4c00640 1 --2- 172.21.15.12:0/955487466 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f418c0a2800 0x7f418c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:07.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.441+0000 7f41a4c00640 1 -- 172.21.15.12:0/955487466 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f41940ceda0 con 0x7f41a81028b0 2024-03-23T23:01:07.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.442+0000 7f41a7e00640 1 --2- 172.21.15.12:0/955487466 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f418c0a2800 0x7f418c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:07.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.442+0000 7f41a7e00640 1 --2- 172.21.15.12:0/955487466 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f418c0a2800 0x7f418c0a4cc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f419802c170 tx=0x7f419802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:07.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.445+0000 7f41a4c00640 1 -- 172.21.15.12:0/955487466 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f419409bdf0 con 0x7f41a81028b0 2024-03-23T23:01:07.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:07 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/4056792430' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:01:07.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:07 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3164192934' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:01:07.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.651+0000 7f41aea00640 1 -- 172.21.15.12:0/955487466 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41a80622a0 con 0x7f41a81028b0 2024-03-23T23:01:07.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.653+0000 7f41a4c00640 1 -- 172.21.15.12:0/955487466 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22695 (secure 0 0 0) 0x7f419409b6f0 con 0x7f41a81028b0 2024-03-23T23:01:07.653 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:01:07.653 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":88,"fsid":"90f679ae-e967-11ee-95cd-87774f69a715","created":"2024-03-23T22:50:14.906952+0000","modified":"2024-03-23T23:00:24.349052+0000","last_up_change":"2024-03-23T22:58:28.907824+0000","last_in_change":"2024-03-23T22:58:08.342487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T22:54:11.831010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T22:59:25.350974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T22:59:26.390323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b70875d7-a23d-4251-9e8b-1a16d4edc2a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6803","nonce":1190079611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6805","nonce":1190079611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6809","nonce":1190079611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6807","nonce":1190079611}]},"public_addr":"172.21.15.12:6803/1190079611","cluster_addr":"172.21.15.12:6805/1190079611","heartbeat_back_addr":"172.21.15.12:6809/1190079611","heartbeat_front_addr":"172.21.15.12:6807/1190079611","state":["exists","up"]},{"osd":1,"uuid":"416e87af-db5d-41ce-84d9-731e3e07e3c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6801","nonce":1983639525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6803","nonce":1983639525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6807","nonce":1983639525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6805","nonce":1983639525}]},"public_addr":"172.21.15.173:6801/1983639525","cluster_addr":"172.21.15.173:6803/1983639525","heartbeat_back_addr":"172.21.15.173:6807/1983639525","heartbeat_front_addr":"172.21.15.173:6805/1983639525","state":["exists","up"]},{"osd":2,"uuid":"7a559c70-4c54-431b-8baf-4136b6e76493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6801","nonce":2253997387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6803","nonce":2253997387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6807","nonce":2253997387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6805","nonce":2253997387}]},"public_addr":"172.21.15.203:6801/2253997387","cluster_addr":"172.21.15.203:6803/2253997387","heartbeat_back_addr":"172.21.15.203:6807/2253997387","heartbeat_front_addr":"172.21.15.203:6805/2253997387","state":["exists","up"]},{"osd":3,"uuid":"c41cd287-a674-486c-8025-328cd36e5ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6811","nonce":1761446905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6813","nonce":1761446905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6817","nonce":1761446905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6815","nonce":1761446905}]},"public_addr":"172.21.15.12:6811/1761446905","cluster_addr":"172.21.15.12:6813/1761446905","heartbeat_back_addr":"172.21.15.12:6817/1761446905","heartbeat_front_addr":"172.21.15.12:6815/1761446905","state":["exists","up"]},{"osd":4,"uuid":"3c48f886-0b42-4a09-b31e-abccc200bb1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6809","nonce":242958272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6811","nonce":242958272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6815","nonce":242958272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6813","nonce":242958272}]},"public_addr":"172.21.15.173:6809/242958272","cluster_addr":"172.21.15.173:6811/242958272","heartbeat_back_addr":"172.21.15.173:6815/242958272","heartbeat_front_addr":"172.21.15.173:6813/242958272","state":["exists","up"]},{"osd":5,"uuid":"4a34f06f-8805-40c0-b1ab-23407b0276d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6809","nonce":4270966092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6811","nonce":4270966092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6815","nonce":4270966092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6813","nonce":4270966092}]},"public_addr":"172.21.15.203:6809/4270966092","cluster_addr":"172.21.15.203:6811/4270966092","heartbeat_back_addr":"172.21.15.203:6815/4270966092","heartbeat_front_addr":"172.21.15.203:6813/4270966092","state":["exists","up"]},{"osd":6,"uuid":"26ddc663-5623-48c9-80d1-f8b968765d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6819","nonce":3606839295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6821","nonce":3606839295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6825","nonce":3606839295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6823","nonce":3606839295}]},"public_addr":"172.21.15.12:6819/3606839295","cluster_addr":"172.21.15.12:6821/3606839295","heartbeat_back_addr":"172.21.15.12:6825/3606839295","heartbeat_front_addr":"172.21.15.12:6823/3606839295","state":["exists","up"]},{"osd":7,"uuid":"17dc9417-71cd-4576-aa08-933212ec55e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6817","nonce":1241504346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6819","nonce":1241504346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6823","nonce":1241504346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6821","nonce":1241504346}]},"public_addr":"172.21.15.173:6817/1241504346","cluster_addr":"172.21.15.173:6819/1241504346","heartbeat_back_addr":"172.21.15.173:6823/1241504346","heartbeat_front_addr":"172.21.15.173:6821/1241504346","state":["exists","up"]},{"osd":8,"uuid":"90a2b63a-471d-43b8-ac18-bb1f16778d18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6817","nonce":1516996192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6819","nonce":1516996192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6823","nonce":1516996192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6821","nonce":1516996192}]},"public_addr":"172.21.15.203:6817/1516996192","cluster_addr":"172.21.15.203:6819/1516996192","heartbeat_back_addr":"172.21.15.203:6823/1516996192","heartbeat_front_addr":"172.21.15.203:6821/1516996192","state":["exists","up"]},{"osd":9,"uuid":"6e1ca504-40b4-49d0-b245-7946181abfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6827","nonce":1175260774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6829","nonce":1175260774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6833","nonce":1175260774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6831","nonce":1175260774}]},"public_addr":"172.21.15.12:6827/1175260774","cluster_addr":"172.21.15.12:6829/1175260774","heartbeat_back_addr":"172.21.15.12:6833/1175260774","heartbeat_front_addr":"172.21.15.12:6831/1175260774","state":["exists","up"]},{"osd":10,"uuid":"09a595ed-5614-4386-8e61-0170af66e01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6825","nonce":3303506808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6827","nonce":3303506808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6831","nonce":3303506808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6829","nonce":3303506808}]},"public_addr":"172.21.15.173:6825/3303506808","cluster_addr":"172.21.15.173:6827/3303506808","heartbeat_back_addr":"172.21.15.173:6831/3303506808","heartbeat_front_addr":"172.21.15.173:6829/3303506808","state":["exists","up"]},{"osd":11,"uuid":"9ef1d595-a977-4ba7-9c6b-d2b3295f20b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6825","nonce":2323604506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6827","nonce":2323604506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6831","nonce":2323604506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6829","nonce":2323604506}]},"public_addr":"172.21.15.203:6825/2323604506","cluster_addr":"172.21.15.203:6827/2323604506","heartbeat_back_addr":"172.21.15.203:6831/2323604506","heartbeat_front_addr":"172.21.15.203:6829/2323604506","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:18.469038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:42.752628+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:07.069493+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:41.962478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:08.363104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:33.341336+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:07.286541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:34.243302+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:59.749699+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:57:33.822913+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:00.737269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:26.599854+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/4182783120":"2024-03-24T22:51:23.113201+0000","172.21.15.12:6800/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1900938637":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/3414861652":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6801/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/2580541477":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6800/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/501058839":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1192554716":"2024-03-24T22:50:48.401223+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T23:01:07.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.655+0000 7f41aea00640 1 -- 172.21.15.12:0/955487466 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f418c0a2800 msgr2=0x7f418c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:07.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.655+0000 7f41aea00640 1 --2- 172.21.15.12:0/955487466 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f418c0a2800 0x7f418c0a4cc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f419802c170 tx=0x7f419802df90 comp rx=0 tx=0).stop 2024-03-23T23:01:07.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.655+0000 7f41aea00640 1 -- 172.21.15.12:0/955487466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41a81028b0 msgr2=0x7f41a8197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:07.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.655+0000 7f41aea00640 1 --2- 172.21.15.12:0/955487466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41a81028b0 0x7f41a8197f40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f419400ea60 tx=0x7f419400ef20 comp rx=0 tx=0).stop 2024-03-23T23:01:07.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.656+0000 7f41aea00640 1 -- 172.21.15.12:0/955487466 shutdown_connections 2024-03-23T23:01:07.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.656+0000 7f41aea00640 1 --2- 172.21.15.12:0/955487466 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f418c0a2800 0x7f418c0a4cc0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:07.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.656+0000 7f41aea00640 1 --2- 172.21.15.12:0/955487466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41a8103270 0x7f41a819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:07.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.656+0000 7f41aea00640 1 --2- 172.21.15.12:0/955487466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41a81028b0 0x7f41a8197f40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:07.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.656+0000 7f41aea00640 1 --2- 172.21.15.12:0/955487466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41a81016c0 0x7f41a8197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:07.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.656+0000 7f41aea00640 1 -- 172.21.15.12:0/955487466 >> 172.21.15.12:0/955487466 conn(0x7f41a80fcc50 msgr2=0x7f41a8105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:07.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.656+0000 7f41aea00640 1 -- 172.21.15.12:0/955487466 shutdown_connections 2024-03-23T23:01:07.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.656+0000 7f41aea00640 1 -- 172.21.15.12:0/955487466 wait complete. 2024-03-23T23:01:07.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:07 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/4056792430' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:01:07.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:07 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3164192934' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:01:07.678 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:01:07.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:07 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/4056792430' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:01:07.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:07 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3164192934' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:01:07.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.904+0000 7fb75f200640 1 -- 172.21.15.12:0/473354358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7601096b0 msgr2=0x7fb76010ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:07.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.904+0000 7fb75f200640 1 --2- 172.21.15.12:0/473354358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7601096b0 0x7fb76010ff40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb758009ca0 tx=0x7fb75802c100 comp rx=0 tx=0).stop 2024-03-23T23:01:07.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.904+0000 7fb75f200640 1 -- 172.21.15.12:0/473354358 shutdown_connections 2024-03-23T23:01:07.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.904+0000 7fb75f200640 1 --2- 172.21.15.12:0/473354358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7601096b0 0x7fb76010ff40 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:07.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.904+0000 7fb75f200640 1 --2- 172.21.15.12:0/473354358 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb760108cf0 0x7fb760109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:07.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.904+0000 7fb75f200640 1 --2- 172.21.15.12:0/473354358 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb760107b00 0x7fb760107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:07.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.904+0000 7fb75f200640 1 -- 172.21.15.12:0/473354358 >> 172.21.15.12:0/473354358 conn(0x7fb760076ab0 msgr2=0x7fb760078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:07.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.904+0000 7fb75f200640 1 -- 172.21.15.12:0/473354358 shutdown_connections 2024-03-23T23:01:07.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.904+0000 7fb75f200640 1 -- 172.21.15.12:0/473354358 wait complete. 2024-03-23T23:01:07.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.905+0000 7fb75f200640 1 Processor -- start 2024-03-23T23:01:07.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.905+0000 7fb75f200640 1 -- start start 2024-03-23T23:01:07.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.905+0000 7fb75f200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb760107b00 0x7fb76019dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:07.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.905+0000 7fb75f200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb760108cf0 0x7fb76019e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:07.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.905+0000 7fb75f200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7601096b0 0x7fb7601a5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:07.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.905+0000 7fb75f200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb760113760 con 0x7fb760107b00 2024-03-23T23:01:07.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.905+0000 7fb75f200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7601135e0 con 0x7fb760108cf0 2024-03-23T23:01:07.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.905+0000 7fb75f200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7601138e0 con 0x7fb7601096b0 2024-03-23T23:01:07.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.906+0000 7fb75e800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7601096b0 0x7fb7601a5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:07.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.906+0000 7fb75d400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb760108cf0 0x7fb76019e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:07.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.906+0000 7fb75de00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb760107b00 0x7fb76019dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:07.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.906+0000 7fb75e800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7601096b0 0x7fb7601a5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49166/0 (socket says 172.21.15.12:49166) 2024-03-23T23:01:07.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.906+0000 7fb75e800640 1 -- 172.21.15.12:0/1455569712 learned_addr learned my addr 172.21.15.12:0/1455569712 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:07.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.906+0000 7fb75e800640 1 -- 172.21.15.12:0/1455569712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb760108cf0 msgr2=0x7fb76019e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:07.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.906+0000 7fb75e800640 1 --2- 172.21.15.12:0/1455569712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb760108cf0 0x7fb76019e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:07.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.906+0000 7fb75e800640 1 -- 172.21.15.12:0/1455569712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb760107b00 msgr2=0x7fb76019dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:07.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.906+0000 7fb75e800640 1 --2- 172.21.15.12:0/1455569712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb760107b00 0x7fb76019dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:07.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.906+0000 7fb75e800640 1 -- 172.21.15.12:0/1455569712 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7601a5990 con 0x7fb7601096b0 2024-03-23T23:01:07.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.906+0000 7fb75d400640 1 --2- 172.21.15.12:0/1455569712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb760108cf0 0x7fb76019e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:07.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.907+0000 7fb75e800640 1 --2- 172.21.15.12:0/1455569712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7601096b0 0x7fb7601a5320 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb75802c640 tx=0x7fb758035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:07.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.907+0000 7fb756a00640 1 -- 172.21.15.12:0/1455569712 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb758039040 con 0x7fb7601096b0 2024-03-23T23:01:07.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.907+0000 7fb756a00640 1 -- 172.21.15.12:0/1455569712 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb758044070 con 0x7fb7601096b0 2024-03-23T23:01:07.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.907+0000 7fb756a00640 1 -- 172.21.15.12:0/1455569712 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb758004460 con 0x7fb7601096b0 2024-03-23T23:01:07.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.907+0000 7fb75f200640 1 -- 172.21.15.12:0/1455569712 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7601a5c20 con 0x7fb7601096b0 2024-03-23T23:01:07.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.907+0000 7fb75f200640 1 -- 172.21.15.12:0/1455569712 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7601a60c0 con 0x7fb7601096b0 2024-03-23T23:01:07.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.908+0000 7fb75f200640 1 -- 172.21.15.12:0/1455569712 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb76010ee20 con 0x7fb7601096b0 2024-03-23T23:01:07.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.910+0000 7fb756a00640 1 -- 172.21.15.12:0/1455569712 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb758004600 con 0x7fb7601096b0 2024-03-23T23:01:07.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.911+0000 7fb756a00640 1 --2- 172.21.15.12:0/1455569712 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb7440a2880 0x7fb7440a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:07.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.911+0000 7fb756a00640 1 -- 172.21.15.12:0/1455569712 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb7580f0fe0 con 0x7fb7601096b0 2024-03-23T23:01:07.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.912+0000 7fb75de00640 1 --2- 172.21.15.12:0/1455569712 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb7440a2880 0x7fb7440a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:07.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.912+0000 7fb75de00640 1 --2- 172.21.15.12:0/1455569712 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb7440a2880 0x7fb7440a4d40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb760108b50 tx=0x7fb75000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:07.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:07.915+0000 7fb756a00640 1 -- 172.21.15.12:0/1455569712 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb7580be090 con 0x7fb7601096b0 2024-03-23T23:01:08.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.157+0000 7fb75f200640 1 -- 172.21.15.12:0/1455569712 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb76002d290 con 0x7fb7601096b0 2024-03-23T23:01:08.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.158+0000 7fb756a00640 1 -- 172.21.15.12:0/1455569712 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11393 (secure 0 0 0) 0x7fb7580098b0 con 0x7fb7601096b0 2024-03-23T23:01:08.158 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:01:08.158 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T22:59:32.190756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24599,24550]},"id":1}]} 2024-03-23T23:01:08.159 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 22 2024-03-23T23:01:08.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.161+0000 7fb75f200640 1 -- 172.21.15.12:0/1455569712 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb7440a2880 msgr2=0x7fb7440a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:08.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.161+0000 7fb75f200640 1 --2- 172.21.15.12:0/1455569712 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb7440a2880 0x7fb7440a4d40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb760108b50 tx=0x7fb75000b390 comp rx=0 tx=0).stop 2024-03-23T23:01:08.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.161+0000 7fb75f200640 1 -- 172.21.15.12:0/1455569712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7601096b0 msgr2=0x7fb7601a5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:08.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.161+0000 7fb75f200640 1 --2- 172.21.15.12:0/1455569712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7601096b0 0x7fb7601a5320 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb75802c640 tx=0x7fb758035d10 comp rx=0 tx=0).stop 2024-03-23T23:01:08.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.161+0000 7fb75f200640 1 -- 172.21.15.12:0/1455569712 shutdown_connections 2024-03-23T23:01:08.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.161+0000 7fb75f200640 1 --2- 172.21.15.12:0/1455569712 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb7440a2880 0x7fb7440a4d40 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.161+0000 7fb75f200640 1 --2- 172.21.15.12:0/1455569712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7601096b0 0x7fb7601a5320 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.161+0000 7fb75f200640 1 --2- 172.21.15.12:0/1455569712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb760108cf0 0x7fb76019e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.161+0000 7fb75f200640 1 --2- 172.21.15.12:0/1455569712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb760107b00 0x7fb76019dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.161+0000 7fb75f200640 1 -- 172.21.15.12:0/1455569712 >> 172.21.15.12:0/1455569712 conn(0x7fb760076ab0 msgr2=0x7fb76010bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:08.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.161+0000 7fb75f200640 1 -- 172.21.15.12:0/1455569712 shutdown_connections 2024-03-23T23:01:08.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.161+0000 7fb75f200640 1 -- 172.21.15.12:0/1455569712 wait complete. 2024-03-23T23:01:08.180 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-23T22:59:26.906237+0000', 'modified': '2024-03-23T22:59:32.190756+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24599, 'mds_1': 24554, 'mds_2': 24550}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24517': {'gid': 24517, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6835/915277278', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6834', 'nonce': 915277278}, {'type': 'v1', 'addr': '172.21.15.203:6835', 'nonce': 915277278}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24521': {'gid': 24521, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6835/2648278843', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6834', 'nonce': 2648278843}, {'type': 'v1', 'addr': '172.21.15.173:6835', 'nonce': 2648278843}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24529': {'gid': 24529, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6837/438676939', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6836', 'nonce': 438676939}, {'type': 'v1', 'addr': '172.21.15.203:6837', 'nonce': 438676939}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24550': {'gid': 24550, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.203:6839/852477167', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6838', 'nonce': 852477167}, {'type': 'v1', 'addr': '172.21.15.203:6839', 'nonce': 852477167}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24554': {'gid': 24554, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.173:6837/2648089141', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6836', 'nonce': 2648089141}, {'type': 'v1', 'addr': '172.21.15.173:6837', 'nonce': 2648089141}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24599': {'gid': 24599, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.173:6839/1645433214', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6838', 'nonce': 1645433214}, {'type': 'v1', 'addr': '172.21.15.173:6839', 'nonce': 1645433214}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24599, 'qdb_cluster': [24554, 24599, 24550]} 2024-03-23T23:01:08.180 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-23T23:01:08.180 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-23T23:01:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.412+0000 7f14e9600640 1 -- 172.21.15.12:0/3253090276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14e4107390 msgr2=0x7f14e41097e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.412+0000 7f14e9600640 1 --2- 172.21.15.12:0/3253090276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14e4107390 0x7f14e41097e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f14dc009ca0 tx=0x7f14dc02c0f0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.412+0000 7f14e9600640 1 -- 172.21.15.12:0/3253090276 shutdown_connections 2024-03-23T23:01:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.412+0000 7f14e9600640 1 --2- 172.21.15.12:0/3253090276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14e4107390 0x7f14e41097e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.412+0000 7f14e9600640 1 --2- 172.21.15.12:0/3253090276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14e4104a60 0x7f14e4106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.412+0000 7f14e9600640 1 --2- 172.21.15.12:0/3253090276 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e4102130 0x7f14e4104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.412+0000 7f14e9600640 1 -- 172.21.15.12:0/3253090276 >> 172.21.15.12:0/3253090276 conn(0x7f14e40fbb40 msgr2=0x7f14e40fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.412+0000 7f14e9600640 1 -- 172.21.15.12:0/3253090276 shutdown_connections 2024-03-23T23:01:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.412+0000 7f14e9600640 1 -- 172.21.15.12:0/3253090276 wait complete. 2024-03-23T23:01:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.413+0000 7f14e9600640 1 Processor -- start 2024-03-23T23:01:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.413+0000 7f14e9600640 1 -- start start 2024-03-23T23:01:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.413+0000 7f14e9600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e4102130 0x7f14e4101aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.413+0000 7f14e9600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14e4104a60 0x7f14e41000e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.414+0000 7f14e9600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14e4107390 0x7f14e4100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.414+0000 7f14e9600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e410d350 con 0x7f14e4102130 2024-03-23T23:01:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.414+0000 7f14e9600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e410d1d0 con 0x7f14e4107390 2024-03-23T23:01:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.414+0000 7f14e9600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e410d4d0 con 0x7f14e4104a60 2024-03-23T23:01:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.414+0000 7f14e2a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e4102130 0x7f14e4101aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.414+0000 7f14e3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14e4107390 0x7f14e4100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.414+0000 7f14da000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14e4104a60 0x7f14e41000e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.414+0000 7f14e3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14e4107390 0x7f14e4100620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:56952/0 (socket says 172.21.15.12:56952) 2024-03-23T23:01:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.414+0000 7f14e3400640 1 -- 172.21.15.12:0/7954814 learned_addr learned my addr 172.21.15.12:0/7954814 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.414+0000 7f14e3400640 1 -- 172.21.15.12:0/7954814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14e4104a60 msgr2=0x7f14e41000e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.414+0000 7f14e3400640 1 --2- 172.21.15.12:0/7954814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14e4104a60 0x7f14e41000e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.414+0000 7f14e3400640 1 -- 172.21.15.12:0/7954814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e4102130 msgr2=0x7f14e4101aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.414+0000 7f14e3400640 1 --2- 172.21.15.12:0/7954814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e4102130 0x7f14e4101aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.414+0000 7f14e3400640 1 -- 172.21.15.12:0/7954814 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e4100eb0 con 0x7f14e4107390 2024-03-23T23:01:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.414+0000 7f14e3400640 1 --2- 172.21.15.12:0/7954814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14e4107390 0x7f14e4100620 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f14dc02c630 tx=0x7f14dc032ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.415+0000 7f14dbe00640 1 -- 172.21.15.12:0/7954814 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14dc041020 con 0x7f14e4107390 2024-03-23T23:01:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.415+0000 7f14dbe00640 1 -- 172.21.15.12:0/7954814 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14dc034cd0 con 0x7f14e4107390 2024-03-23T23:01:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.415+0000 7f14dbe00640 1 -- 172.21.15.12:0/7954814 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14dc039040 con 0x7f14e4107390 2024-03-23T23:01:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.415+0000 7f14e9600640 1 -- 172.21.15.12:0/7954814 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e41a1a60 con 0x7f14e4107390 2024-03-23T23:01:08.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.415+0000 7f14e9600640 1 -- 172.21.15.12:0/7954814 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e41a1fb0 con 0x7f14e4107390 2024-03-23T23:01:08.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.418+0000 7f14dbe00640 1 -- 172.21.15.12:0/7954814 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f14dc044020 con 0x7f14e4107390 2024-03-23T23:01:08.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.418+0000 7f14ec265740 1 Processor -- start 2024-03-23T23:01:08.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.418+0000 7f14ec265740 1 -- start start 2024-03-23T23:01:08.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.419+0000 7f14ec265740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a623484ba0 0x55a623484fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.419+0000 7f14ec265740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a623485590 0x55a62348a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.419+0000 7f14ec265740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a62348a700 0x55a62348cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.419+0000 7f14ec265740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6233e9c90 con 0x55a623484ba0 2024-03-23T23:01:08.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.419+0000 7f14ec265740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a623392980 con 0x55a623485590 2024-03-23T23:01:08.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.419+0000 7f14ec265740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6233e5d30 con 0x55a62348a700 2024-03-23T23:01:08.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.419+0000 7f14da000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a623484ba0 0x55a623484fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.419+0000 7f14e2a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a623485590 0x55a62348a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.419+0000 7f14da000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a623484ba0 0x55a623484fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54474/0 (socket says 172.21.15.12:54474) 2024-03-23T23:01:08.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.419+0000 7f14da000640 1 -- 172.21.15.12:0/1706678685 learned_addr learned my addr 172.21.15.12:0/1706678685 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:08.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.419+0000 7f14da000640 1 --2- 172.21.15.12:0/1706678685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a62348a700 0x55a62348cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.419+0000 7f14da000640 1 -- 172.21.15.12:0/1706678685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a62348a700 msgr2=0x55a62348cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:08.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.419+0000 7f14da000640 1 --2- 172.21.15.12:0/1706678685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a62348a700 0x55a62348cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.419+0000 7f14da000640 1 -- 172.21.15.12:0/1706678685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a623485590 msgr2=0x55a62348a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:08.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.419+0000 7f14da000640 1 --2- 172.21.15.12:0/1706678685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a623485590 0x55a62348a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.419+0000 7f14da000640 1 -- 172.21.15.12:0/1706678685 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6233aa510 con 0x55a623484ba0 2024-03-23T23:01:08.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.419+0000 7f14e2a00640 1 --2- 172.21.15.12:0/1706678685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a623485590 0x55a62348a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:08.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.419+0000 7f14dbe00640 1 --2- 172.21.15.12:0/7954814 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f14c80a2800 0x7f14c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.420+0000 7f14da000640 1 --2- 172.21.15.12:0/1706678685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a623484ba0 0x55a623484fc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f14cc009f70 tx=0x7f14cc005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d20b53fa8369138 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:08.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.420+0000 7f14dbe00640 1 -- 172.21.15.12:0/7954814 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f14dc0f0930 con 0x7f14e4107390 2024-03-23T23:01:08.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.420+0000 7f14e2a00640 1 --2- 172.21.15.12:0/7954814 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f14c80a2800 0x7f14c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.420+0000 7f14d3400640 1 -- 172.21.15.12:0/1706678685 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc004050 con 0x55a623484ba0 2024-03-23T23:01:08.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.420+0000 7f14d3400640 1 -- 172.21.15.12:0/1706678685 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14cc005500 con 0x55a623484ba0 2024-03-23T23:01:08.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.420+0000 7f14d3400640 1 -- 172.21.15.12:0/1706678685 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc004c00 con 0x55a623484ba0 2024-03-23T23:01:08.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.421+0000 7f14ec265740 1 -- 172.21.15.12:0/1706678685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a623484ba0 msgr2=0x55a623484fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:08.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.421+0000 7f14e2a00640 1 --2- 172.21.15.12:0/7954814 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f14c80a2800 0x7f14c80a4cc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x55a623485990 tx=0x7f14d400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:08.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.421+0000 7f14ec265740 1 --2- 172.21.15.12:0/1706678685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a623484ba0 0x55a623484fc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f14cc009f70 tx=0x7f14cc005850 comp rx=0 tx=0).stop 2024-03-23T23:01:08.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.421+0000 7f14ec265740 1 -- 172.21.15.12:0/1706678685 shutdown_connections 2024-03-23T23:01:08.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.421+0000 7f14ec265740 1 --2- 172.21.15.12:0/1706678685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a62348a700 0x55a62348cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.421+0000 7f14ec265740 1 --2- 172.21.15.12:0/1706678685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a623485590 0x55a62348a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.421+0000 7f14ec265740 1 --2- 172.21.15.12:0/1706678685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a623484ba0 0x55a623484fc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.421+0000 7f14ec265740 1 -- 172.21.15.12:0/1706678685 >> 172.21.15.12:0/1706678685 conn(0x55a623480110 msgr2=0x55a623480510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:08.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.421+0000 7f14ec265740 1 -- 172.21.15.12:0/1706678685 shutdown_connections 2024-03-23T23:01:08.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.421+0000 7f14ec265740 1 -- 172.21.15.12:0/1706678685 wait complete. 2024-03-23T23:01:08.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.421+0000 7f14ec265740 1 -- start start 2024-03-23T23:01:08.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.422+0000 7f14ec265740 1 Processor -- start 2024-03-23T23:01:08.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.422+0000 7f14ec265740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a623484ba0 0x55a62348f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.422+0000 7f14ec265740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a623485590 0x55a62348f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.422+0000 7f14ec265740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a62348a700 0x55a623496690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.422+0000 7f14ec265740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a62348d6b0 con 0x55a623485590 2024-03-23T23:01:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.422+0000 7f14ec265740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6233a5dd0 con 0x55a62348a700 2024-03-23T23:01:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.422+0000 7f14ec265740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a62348d830 con 0x55a623484ba0 2024-03-23T23:01:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.422+0000 7f14e2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a623484ba0 0x55a62348f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.422+0000 7f14e2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a623484ba0 0x55a62348f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49204/0 (socket says 172.21.15.12:49204) 2024-03-23T23:01:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.422+0000 7f14e2a00640 1 -- 172.21.15.12:0/277430606 learned_addr learned my addr 172.21.15.12:0/277430606 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.422+0000 7f14da000640 1 --2- 172.21.15.12:0/277430606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a623485590 0x55a62348f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.422+0000 7f14e3400640 1 --2- 172.21.15.12:0/277430606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a62348a700 0x55a623496690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.422+0000 7f14ec265740 10 client.0 _ll_register_callbacks cb 0x55a6230fdac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:01:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.423+0000 7f14e2a00640 1 -- 172.21.15.12:0/277430606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a62348a700 msgr2=0x55a623496690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.423+0000 7f14e2a00640 1 --2- 172.21.15.12:0/277430606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a62348a700 0x55a623496690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.423+0000 7f14e2a00640 1 -- 172.21.15.12:0/277430606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a623485590 msgr2=0x55a62348f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.423+0000 7f14e2a00640 1 --2- 172.21.15.12:0/277430606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a623485590 0x55a62348f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.423+0000 7f14e2a00640 1 -- 172.21.15.12:0/277430606 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a6230fe2d0 con 0x55a623484ba0 2024-03-23T23:01:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.423+0000 7f14e3400640 1 --2- 172.21.15.12:0/277430606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a62348a700 0x55a623496690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.423+0000 7f14e2a00640 1 --2- 172.21.15.12:0/277430606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a623484ba0 0x55a62348f110 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f14d400bfd0 tx=0x7f14d4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d6c9c34d55508a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.423+0000 7f14c7e00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:01:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.423+0000 7f14c7e00640 1 -- 172.21.15.12:0/277430606 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d4005020 con 0x55a623484ba0 2024-03-23T23:01:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.423+0000 7f14c7e00640 1 -- 172.21.15.12:0/277430606 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f14d4017030 con 0x55a623484ba0 2024-03-23T23:01:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.423+0000 7f14ec265740 1 -- 172.21.15.12:0/277430606 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f14cc005500 con 0x55a623484ba0 2024-03-23T23:01:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.424+0000 7f14c7e00640 1 -- 172.21.15.12:0/277430606 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14d40051c0 con 0x55a623484ba0 2024-03-23T23:01:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.424+0000 7f14ec265740 10 client.24991 fetch_fsmap learned FSMap version 22 2024-03-23T23:01:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.424+0000 7f14ec265740 1 -- 172.21.15.12:0/277430606 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55a6231c86b0 con 0x55a623484ba0 2024-03-23T23:01:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.424+0000 7f14c7e00640 1 -- 172.21.15.12:0/277430606 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14cc005500 con 0x55a623484ba0 2024-03-23T23:01:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.424+0000 7f14c7e00640 1 -- 172.21.15.12:0/277430606 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f14d4004830 con 0x55a623484ba0 2024-03-23T23:01:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.424+0000 7f14ec265740 10 client.24991 fetch_fsmap finished waiting for FSMap version 22 2024-03-23T23:01:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.424+0000 7f14ec265740 10 client.24991 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-23T23:01:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.424+0000 7f14ec265740 20 client.24991 populate_metadata read hostname 'smithi012' 2024-03-23T23:01:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.424+0000 7f14ec265740 1 --2- 172.21.15.12:0/277430606 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55a62349ba10 0x55a62349aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.424+0000 7f14ec265740 4 client.24991 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:01:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.424+0000 7f14ec265740 1 -- 172.21.15.12:0/277430606 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a62338b520 con 0x55a62349ba10 2024-03-23T23:01:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.424+0000 7f14da000640 1 --2- 172.21.15.12:0/277430606 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55a62349ba10 0x55a62349aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.425+0000 7f14da000640 1 --2- 172.21.15.12:0/277430606 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55a62349ba10 0x55a62349aa00 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9fe5a977be1dd43f server_cookie=61d8126459e02983 in_seq=0 out_seq=0 2024-03-23T23:01:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.425+0000 7f14c7e00640 10 client.24991 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:01:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.426+0000 7f14c7e00640 1 -- 172.21.15.12:0/277430606 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a62338b520 con 0x55a62349ba10 2024-03-23T23:01:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.426+0000 7f14c7e00640 10 client.24991 handle_command_reply: tid=0 2024-03-23T23:01:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.426+0000 7f14ec265740 1 client.24991 shutdown 2024-03-23T23:01:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.427+0000 7f14ec265740 1 -- 172.21.15.12:0/277430606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a623484ba0 msgr2=0x55a62348f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.427+0000 7f14ec265740 1 --2- 172.21.15.12:0/277430606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a623484ba0 0x55a62348f110 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f14d400bfd0 tx=0x7f14d4004730 comp rx=0 tx=0).stop 2024-03-23T23:01:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.427+0000 7f14ec265740 1 -- 172.21.15.12:0/277430606 shutdown_connections 2024-03-23T23:01:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.427+0000 7f14ec265740 1 --2- 172.21.15.12:0/277430606 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55a62349ba10 0x55a62349aa00 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.427+0000 7f14ec265740 1 --2- 172.21.15.12:0/277430606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a62348a700 0x55a623496690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.427+0000 7f14ec265740 1 --2- 172.21.15.12:0/277430606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a623485590 0x55a62348f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.427+0000 7f14ec265740 1 --2- 172.21.15.12:0/277430606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a623484ba0 0x55a62348f110 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.427+0000 7f14c7e00640 0 client.24991 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:01:08.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.427+0000 7f14ec265740 1 -- 172.21.15.12:0/277430606 >> 172.21.15.12:0/277430606 conn(0x55a623480110 msgr2=0x55a62347f310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:08.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.427+0000 7f14ec265740 1 -- 172.21.15.12:0/277430606 shutdown_connections 2024-03-23T23:01:08.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.427+0000 7f14ec265740 1 -- 172.21.15.12:0/277430606 wait complete. 2024-03-23T23:01:08.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.427+0000 7f14ec265740 20 client.24991 trim_cache size 0 max 16384 2024-03-23T23:01:08.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.445+0000 7f14ec265740 1 Processor -- start 2024-03-23T23:01:08.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.446+0000 7f14ec265740 1 -- start start 2024-03-23T23:01:08.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.446+0000 7f14ec265740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a623484ba0 0x55a62348caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.446+0000 7f14ec265740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a623485590 0x55a623496620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.446+0000 7f14ec265740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a62348a700 0x55a62349aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.446+0000 7f14ec265740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a62349b230 con 0x55a62348a700 2024-03-23T23:01:08.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.446+0000 7f14ec265740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a62349b890 con 0x55a623485590 2024-03-23T23:01:08.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.446+0000 7f14ec265740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a62349bef0 con 0x55a623484ba0 2024-03-23T23:01:08.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.446+0000 7f14da000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a623485590 0x55a623496620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.446+0000 7f14da000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a623485590 0x55a623496620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:56996/0 (socket says 172.21.15.12:56996) 2024-03-23T23:01:08.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.446+0000 7f14da000640 1 -- 172.21.15.12:0/1884073524 learned_addr learned my addr 172.21.15.12:0/1884073524 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:08.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.446+0000 7f14e2a00640 1 --2- 172.21.15.12:0/1884073524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a623484ba0 0x55a62348caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.447+0000 7f14e3400640 1 --2- 172.21.15.12:0/1884073524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a62348a700 0x55a62349aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.447+0000 7f14da000640 1 -- 172.21.15.12:0/1884073524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a623484ba0 msgr2=0x55a62348caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:08.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.447+0000 7f14da000640 1 --2- 172.21.15.12:0/1884073524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a623484ba0 0x55a62348caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.447+0000 7f14da000640 1 -- 172.21.15.12:0/1884073524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a62348a700 msgr2=0x55a62349aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:08.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.447+0000 7f14da000640 1 --2- 172.21.15.12:0/1884073524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a62348a700 0x55a62349aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.447+0000 7f14da000640 1 -- 172.21.15.12:0/1884073524 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a623489d30 con 0x55a623485590 2024-03-23T23:01:08.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.447+0000 7f14e2a00640 1 --2- 172.21.15.12:0/1884073524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a623484ba0 0x55a62348caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:01:08.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.447+0000 7f14e3400640 1 --2- 172.21.15.12:0/1884073524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a62348a700 0x55a62349aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:01:08.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.447+0000 7f14da000640 1 --2- 172.21.15.12:0/1884073524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a623485590 0x55a623496620 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f14cc009e20 tx=0x7f14cc025040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dba7609924d3eaac server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:08.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.447+0000 7f14d2a00640 1 -- 172.21.15.12:0/1884073524 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc005c50 con 0x55a623485590 2024-03-23T23:01:08.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.447+0000 7f14d2a00640 1 -- 172.21.15.12:0/1884073524 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14cc005030 con 0x55a623485590 2024-03-23T23:01:08.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.447+0000 7f14d2a00640 1 -- 172.21.15.12:0/1884073524 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc004ae0 con 0x55a623485590 2024-03-23T23:01:08.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.448+0000 7f14ec265740 1 -- 172.21.15.12:0/1884073524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a623485590 msgr2=0x55a623496620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:08.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.448+0000 7f14ec265740 1 --2- 172.21.15.12:0/1884073524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a623485590 0x55a623496620 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f14cc009e20 tx=0x7f14cc025040 comp rx=0 tx=0).stop 2024-03-23T23:01:08.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.448+0000 7f14ec265740 1 -- 172.21.15.12:0/1884073524 shutdown_connections 2024-03-23T23:01:08.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.448+0000 7f14ec265740 1 --2- 172.21.15.12:0/1884073524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a62348a700 0x55a62349aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.448+0000 7f14ec265740 1 --2- 172.21.15.12:0/1884073524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a623485590 0x55a623496620 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.448+0000 7f14ec265740 1 --2- 172.21.15.12:0/1884073524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a623484ba0 0x55a62348caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.448+0000 7f14ec265740 1 -- 172.21.15.12:0/1884073524 >> 172.21.15.12:0/1884073524 conn(0x55a623480110 msgr2=0x55a6234804f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:08.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.448+0000 7f14ec265740 1 -- 172.21.15.12:0/1884073524 shutdown_connections 2024-03-23T23:01:08.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.448+0000 7f14ec265740 1 -- 172.21.15.12:0/1884073524 wait complete. 2024-03-23T23:01:08.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.448+0000 7f14ec265740 1 -- start start 2024-03-23T23:01:08.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.448+0000 7f14ec265740 1 Processor -- start 2024-03-23T23:01:08.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.449+0000 7f14ec265740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a623484ba0 0x55a6234922e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.449+0000 7f14ec265740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a623485590 0x55a623492820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.449+0000 7f14ec265740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a62348a700 0x55a623495840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.449+0000 7f14ec265740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a62349cbb0 con 0x55a623484ba0 2024-03-23T23:01:08.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.449+0000 7f14ec265740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a62349c550 con 0x55a623485590 2024-03-23T23:01:08.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.449+0000 7f14ec265740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a62349d210 con 0x55a62348a700 2024-03-23T23:01:08.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.449+0000 7f14ec265740 10 client.0 _ll_register_callbacks cb 0x7f14d400b900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:01:08.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.449+0000 7f14e2a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a623484ba0 0x55a6234922e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.449+0000 7f14e3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a62348a700 0x55a623495840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.449+0000 7f14da000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a623485590 0x55a623492820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.449+0000 7f14da000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a623485590 0x55a623492820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:56998/0 (socket says 172.21.15.12:56998) 2024-03-23T23:01:08.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.449+0000 7f14da000640 1 -- 172.21.15.12:0/1208339995 learned_addr learned my addr 172.21.15.12:0/1208339995 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:08.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.450+0000 7f14da000640 1 -- 172.21.15.12:0/1208339995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a62348a700 msgr2=0x55a623495840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:08.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.450+0000 7f14da000640 1 --2- 172.21.15.12:0/1208339995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a62348a700 0x55a623495840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.450+0000 7f14da000640 1 -- 172.21.15.12:0/1208339995 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a623484ba0 msgr2=0x55a6234922e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:08.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.450+0000 7f14da000640 1 --2- 172.21.15.12:0/1208339995 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a623484ba0 0x55a6234922e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.450+0000 7f14da000640 1 -- 172.21.15.12:0/1208339995 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a623495e30 con 0x55a623485590 2024-03-23T23:01:08.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.450+0000 7f14e2a00640 1 --2- 172.21.15.12:0/1208339995 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a623484ba0 0x55a6234922e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:08.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.450+0000 7f14da000640 1 --2- 172.21.15.12:0/1208339995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a623485590 0x55a623492820 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f14cc013ea0 tx=0x7f14cc009b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f339b55cd683a1b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:08.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.450+0000 7f14c7e00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:01:08.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.450+0000 7f14c7e00640 1 -- 172.21.15.12:0/1208339995 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc034070 con 0x55a623485590 2024-03-23T23:01:08.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.450+0000 7f14ec265740 1 -- 172.21.15.12:0/1208339995 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f14cc005030 con 0x55a623485590 2024-03-23T23:01:08.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.450+0000 7f14c7e00640 1 -- 172.21.15.12:0/1208339995 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f14cc02d220 con 0x55a623485590 2024-03-23T23:01:08.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.450+0000 7f14c7e00640 1 -- 172.21.15.12:0/1208339995 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14cc03f050 con 0x55a623485590 2024-03-23T23:01:08.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.450+0000 7f14ec265740 10 client.25010 fetch_fsmap learned FSMap version 22 2024-03-23T23:01:08.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.450+0000 7f14ec265740 1 -- 172.21.15.12:0/1208339995 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55a6234ff7c0 con 0x55a623485590 2024-03-23T23:01:08.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.451+0000 7f14c7e00640 1 -- 172.21.15.12:0/1208339995 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14cc005030 con 0x55a623485590 2024-03-23T23:01:08.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.451+0000 7f14c7e00640 1 -- 172.21.15.12:0/1208339995 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f14cc01d020 con 0x55a623485590 2024-03-23T23:01:08.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.451+0000 7f14ec265740 10 client.25010 fetch_fsmap finished waiting for FSMap version 22 2024-03-23T23:01:08.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.451+0000 7f14ec265740 10 client.25010 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-23T23:01:08.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.451+0000 7f14ec265740 20 client.25010 populate_metadata read hostname 'smithi012' 2024-03-23T23:01:08.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.451+0000 7f14ec265740 1 --2- 172.21.15.12:0/1208339995 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55a623503730 0x55a6234ff990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.451+0000 7f14ec265740 4 client.25010 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-23T23:01:08.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.451+0000 7f14ec265740 1 -- 172.21.15.12:0/1208339995 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a6233f71d0 con 0x55a623503730 2024-03-23T23:01:08.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.451+0000 7f14e2a00640 1 --2- 172.21.15.12:0/1208339995 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55a623503730 0x55a6234ff990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.452+0000 7f14e2a00640 1 --2- 172.21.15.12:0/1208339995 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55a623503730 0x55a6234ff990 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8d41c59efd25d966 server_cookie=fdb86bb2b66f9122 in_seq=0 out_seq=0 2024-03-23T23:01:08.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.452+0000 7f14c7e00640 10 client.25010 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:01:08.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.452+0000 7f14c7e00640 1 -- 172.21.15.12:0/1208339995 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55a6233f71d0 con 0x55a623503730 2024-03-23T23:01:08.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.452+0000 7f14c7e00640 10 client.25010 handle_command_reply: tid=0 2024-03-23T23:01:08.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.452+0000 7f14ec265740 1 client.25010 shutdown 2024-03-23T23:01:08.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.452+0000 7f14ec265740 1 -- 172.21.15.12:0/1208339995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a623485590 msgr2=0x55a623492820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:08.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.452+0000 7f14ec265740 1 --2- 172.21.15.12:0/1208339995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a623485590 0x55a623492820 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f14cc013ea0 tx=0x7f14cc009b10 comp rx=0 tx=0).stop 2024-03-23T23:01:08.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.452+0000 7f14ec265740 1 -- 172.21.15.12:0/1208339995 shutdown_connections 2024-03-23T23:01:08.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.452+0000 7f14ec265740 1 --2- 172.21.15.12:0/1208339995 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55a623503730 0x55a6234ff990 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.452+0000 7f14ec265740 1 --2- 172.21.15.12:0/1208339995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a62348a700 0x55a623495840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.452+0000 7f14ec265740 1 --2- 172.21.15.12:0/1208339995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a623485590 0x55a623492820 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.452+0000 7f14ec265740 1 --2- 172.21.15.12:0/1208339995 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a623484ba0 0x55a6234922e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.452+0000 7f14c7e00640 0 client.25010 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:01:08.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.452+0000 7f14ec265740 1 -- 172.21.15.12:0/1208339995 >> 172.21.15.12:0/1208339995 conn(0x55a623480110 msgr2=0x55a62347ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:08.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.452+0000 7f14ec265740 1 -- 172.21.15.12:0/1208339995 shutdown_connections 2024-03-23T23:01:08.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.452+0000 7f14ec265740 1 -- 172.21.15.12:0/1208339995 wait complete. 2024-03-23T23:01:08.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.453+0000 7f14ec265740 20 client.25010 trim_cache size 0 max 16384 2024-03-23T23:01:08.464 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:01:08.464 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "90f679ae-e967-11ee-95cd-87774f69a715", 2024-03-23T23:01:08.465 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 2, 2024-03-23T23:01:08.465 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 24550, 2024-03-23T23:01:08.465 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-23T23:01:08.465 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-23T23:01:08.465 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-23T23:01:08.465 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 96.964163639000006, 2024-03-23T23:01:08.465 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 22, 2024-03-23T23:01:08.465 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 88, 2024-03-23T23:01:08.465 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 86, 2024-03-23T23:01:08.465 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 118.95502164200001 2024-03-23T23:01:08.465 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:01:08.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.453+0000 7f14e9600640 1 -- 172.21.15.12:0/7954814 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f14c80a2800 msgr2=0x7f14c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:08.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.453+0000 7f14e9600640 1 --2- 172.21.15.12:0/7954814 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f14c80a2800 0x7f14c80a4cc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x55a623485990 tx=0x7f14d400b410 comp rx=0 tx=0).stop 2024-03-23T23:01:08.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.453+0000 7f14e9600640 1 -- 172.21.15.12:0/7954814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14e4107390 msgr2=0x7f14e4100620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:08.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.453+0000 7f14e9600640 1 --2- 172.21.15.12:0/7954814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14e4107390 0x7f14e4100620 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f14dc02c630 tx=0x7f14dc032ea0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.453+0000 7f14e9600640 1 -- 172.21.15.12:0/7954814 shutdown_connections 2024-03-23T23:01:08.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.453+0000 7f14e9600640 1 --2- 172.21.15.12:0/7954814 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f14c80a2800 0x7f14c80a4cc0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.453+0000 7f14e9600640 1 --2- 172.21.15.12:0/7954814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14e4107390 0x7f14e4100620 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.453+0000 7f14e9600640 1 --2- 172.21.15.12:0/7954814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14e4104a60 0x7f14e41000e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.453+0000 7f14e9600640 1 --2- 172.21.15.12:0/7954814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e4102130 0x7f14e4101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.453+0000 7f14e9600640 1 -- 172.21.15.12:0/7954814 >> 172.21.15.12:0/7954814 conn(0x7f14e40fbb40 msgr2=0x7f14e4102a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:08.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.453+0000 7f14e9600640 1 -- 172.21.15.12:0/7954814 shutdown_connections 2024-03-23T23:01:08.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.453+0000 7f14e9600640 1 -- 172.21.15.12:0/7954814 wait complete. 2024-03-23T23:01:08.469 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:08 smithi012 ceph-mon[31695]: pgmap v370: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-23T23:01:08.469 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:08 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/955487466' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:08.469 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:08 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1455569712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:01:08.471 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-23T23:01:08.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:08 smithi203 ceph-mon[36693]: pgmap v370: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-23T23:01:08.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:08 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/955487466' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:08.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:08 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1455569712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:01:08.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.689+0000 7ffbd4a00640 1 -- 172.21.15.12:0/1106655256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbd010db70 msgr2=0x7ffbd010ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:08.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.689+0000 7ffbd4a00640 1 --2- 172.21.15.12:0/1106655256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbd010db70 0x7ffbd010ff60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8009930 tx=0x7ffbc802bc30 comp rx=0 tx=0).stop 2024-03-23T23:01:08.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.689+0000 7ffbd4a00640 1 -- 172.21.15.12:0/1106655256 shutdown_connections 2024-03-23T23:01:08.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.689+0000 7ffbd4a00640 1 --2- 172.21.15.12:0/1106655256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbd010db70 0x7ffbd010ff60 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.689+0000 7ffbd4a00640 1 --2- 172.21.15.12:0/1106655256 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbd0074100 0x7ffbd0074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.689+0000 7ffbd4a00640 1 --2- 172.21.15.12:0/1106655256 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbd00756e0 0x7ffbd0073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.689+0000 7ffbd4a00640 1 -- 172.21.15.12:0/1106655256 >> 172.21.15.12:0/1106655256 conn(0x7ffbd00fcb10 msgr2=0x7ffbd00fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:08.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.689+0000 7ffbd4a00640 1 -- 172.21.15.12:0/1106655256 shutdown_connections 2024-03-23T23:01:08.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.690+0000 7ffbd4a00640 1 -- 172.21.15.12:0/1106655256 wait complete. 2024-03-23T23:01:08.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.690+0000 7ffbd4a00640 1 Processor -- start 2024-03-23T23:01:08.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.691+0000 7ffbd4a00640 1 -- start start 2024-03-23T23:01:08.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.691+0000 7ffbd4a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbd0074100 0x7ffbd019df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.691+0000 7ffbd4a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbd00756e0 0x7ffbd019e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.691+0000 7ffbd4a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbd010db70 0x7ffbd01a5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.691+0000 7ffbd4a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd0113940 con 0x7ffbd0074100 2024-03-23T23:01:08.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.691+0000 7ffbd4a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd01137c0 con 0x7ffbd00756e0 2024-03-23T23:01:08.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.691+0000 7ffbd4a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd0113ac0 con 0x7ffbd010db70 2024-03-23T23:01:08.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.691+0000 7ffbcf200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbd010db70 0x7ffbd01a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.691+0000 7ffbcde00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbd00756e0 0x7ffbd019e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.691+0000 7ffbcf200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbd010db70 0x7ffbd01a5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49236/0 (socket says 172.21.15.12:49236) 2024-03-23T23:01:08.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.691+0000 7ffbce800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbd0074100 0x7ffbd019df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.691+0000 7ffbcf200640 1 -- 172.21.15.12:0/3726045297 learned_addr learned my addr 172.21.15.12:0/3726045297 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:08.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.692+0000 7ffbcf200640 1 -- 172.21.15.12:0/3726045297 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbd00756e0 msgr2=0x7ffbd019e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:08.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.692+0000 7ffbcf200640 1 --2- 172.21.15.12:0/3726045297 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbd00756e0 0x7ffbd019e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.692+0000 7ffbcf200640 1 -- 172.21.15.12:0/3726045297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbd0074100 msgr2=0x7ffbd019df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:08.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.692+0000 7ffbcf200640 1 --2- 172.21.15.12:0/3726045297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbd0074100 0x7ffbd019df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.692+0000 7ffbcf200640 1 -- 172.21.15.12:0/3726045297 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd01a5ba0 con 0x7ffbd010db70 2024-03-23T23:01:08.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.692+0000 7ffbce800640 1 --2- 172.21.15.12:0/3726045297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbd0074100 0x7ffbd019df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:08.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.692+0000 7ffbcf200640 1 --2- 172.21.15.12:0/3726045297 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbd010db70 0x7ffbd01a5530 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8004f50 tx=0x7ffbc8005e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:08.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.692+0000 7ffbc7400640 1 -- 172.21.15.12:0/3726045297 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8043020 con 0x7ffbd010db70 2024-03-23T23:01:08.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.692+0000 7ffbc7400640 1 -- 172.21.15.12:0/3726045297 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffbc80054e0 con 0x7ffbd010db70 2024-03-23T23:01:08.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.692+0000 7ffbc7400640 1 -- 172.21.15.12:0/3726045297 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8003e50 con 0x7ffbd010db70 2024-03-23T23:01:08.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.692+0000 7ffbd4a00640 1 -- 172.21.15.12:0/3726045297 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd01a5e30 con 0x7ffbd010db70 2024-03-23T23:01:08.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.693+0000 7ffbd4a00640 1 -- 172.21.15.12:0/3726045297 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd01a6340 con 0x7ffbd010db70 2024-03-23T23:01:08.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.697+0000 7ffbd8000740 1 Processor -- start 2024-03-23T23:01:08.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.697+0000 7ffbc7400640 1 -- 172.21.15.12:0/3726045297 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ffbc8046020 con 0x7ffbd010db70 2024-03-23T23:01:08.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.697+0000 7ffbd8000740 1 -- start start 2024-03-23T23:01:08.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.698+0000 7ffbd8000740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560414d83d80 0x560414d841a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.698+0000 7ffbd8000740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560414d84770 0x560414d89270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.698+0000 7ffbd8000740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560414d898e0 0x560414d8bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.698+0000 7ffbd8000740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x560414ce8c90 con 0x560414d83d80 2024-03-23T23:01:08.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.698+0000 7ffbd8000740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x560414c91980 con 0x560414d898e0 2024-03-23T23:01:08.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.698+0000 7ffbd8000740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x560414ce4d30 con 0x560414d84770 2024-03-23T23:01:08.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.698+0000 7ffbcde00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560414d83d80 0x560414d841a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.698+0000 7ffbcde00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560414d83d80 0x560414d841a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54538/0 (socket says 172.21.15.12:54538) 2024-03-23T23:01:08.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.698+0000 7ffbcde00640 1 -- 172.21.15.12:0/157509584 learned_addr learned my addr 172.21.15.12:0/157509584 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:08.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.698+0000 7ffbce800640 1 --2- 172.21.15.12:0/157509584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560414d84770 0x560414d89270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.698+0000 7ffbcde00640 1 --2- 172.21.15.12:0/157509584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560414d898e0 0x560414d8bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.698+0000 7ffbce800640 1 -- 172.21.15.12:0/157509584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560414d898e0 msgr2=0x560414d8bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:08.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.698+0000 7ffbce800640 1 --2- 172.21.15.12:0/157509584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560414d898e0 0x560414d8bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.698+0000 7ffbce800640 1 -- 172.21.15.12:0/157509584 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560414d83d80 msgr2=0x560414d841a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:08.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.698+0000 7ffbce800640 1 --2- 172.21.15.12:0/157509584 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560414d83d80 0x560414d841a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.698+0000 7ffbce800640 1 -- 172.21.15.12:0/157509584 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560414ca9510 con 0x560414d84770 2024-03-23T23:01:08.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.698+0000 7ffbcde00640 1 --2- 172.21.15.12:0/157509584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560414d898e0 0x560414d8bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:01:08.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.699+0000 7ffbce800640 1 --2- 172.21.15.12:0/157509584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560414d84770 0x560414d89270 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ffbc000a7a0 tx=0x7ffbc00096d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=972b665e67f0596c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:08.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.699+0000 7ffbc7400640 1 --2- 172.21.15.12:0/3726045297 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ffbb40a2a20 0x7ffbb40a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.699+0000 7ffbbb400640 1 -- 172.21.15.12:0/157509584 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0018020 con 0x560414d84770 2024-03-23T23:01:08.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.699+0000 7ffbbb400640 1 -- 172.21.15.12:0/157509584 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffbc000ae40 con 0x560414d84770 2024-03-23T23:01:08.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.699+0000 7ffbbb400640 1 -- 172.21.15.12:0/157509584 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0013360 con 0x560414d84770 2024-03-23T23:01:08.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.699+0000 7ffbc7400640 1 -- 172.21.15.12:0/3726045297 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ffbc80f0d30 con 0x7ffbd010db70 2024-03-23T23:01:08.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.699+0000 7ffbcde00640 1 --2- 172.21.15.12:0/3726045297 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ffbb40a2a20 0x7ffbb40a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.700+0000 7ffbd8000740 1 -- 172.21.15.12:0/157509584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560414d84770 msgr2=0x560414d89270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:08.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.700+0000 7ffbd8000740 1 --2- 172.21.15.12:0/157509584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560414d84770 0x560414d89270 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ffbc000a7a0 tx=0x7ffbc00096d0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.700+0000 7ffbd8000740 1 -- 172.21.15.12:0/157509584 shutdown_connections 2024-03-23T23:01:08.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.700+0000 7ffbd8000740 1 --2- 172.21.15.12:0/157509584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560414d898e0 0x560414d8bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.700+0000 7ffbd8000740 1 --2- 172.21.15.12:0/157509584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560414d84770 0x560414d89270 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.700+0000 7ffbd8000740 1 --2- 172.21.15.12:0/157509584 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560414d83d80 0x560414d841a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.700+0000 7ffbcde00640 1 --2- 172.21.15.12:0/3726045297 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ffbb40a2a20 0x7ffbb40a4ee0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x560414d84fd0 tx=0x7ffbbc0094c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:08.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.700+0000 7ffbd8000740 1 -- 172.21.15.12:0/157509584 >> 172.21.15.12:0/157509584 conn(0x560414d7f2f0 msgr2=0x560414d7f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:08.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.700+0000 7ffbd8000740 1 -- 172.21.15.12:0/157509584 shutdown_connections 2024-03-23T23:01:08.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.700+0000 7ffbd8000740 1 -- 172.21.15.12:0/157509584 wait complete. 2024-03-23T23:01:08.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.701+0000 7ffbd8000740 1 -- start start 2024-03-23T23:01:08.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.701+0000 7ffbd8000740 1 Processor -- start 2024-03-23T23:01:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.701+0000 7ffbd8000740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560414d83d80 0x560414d8e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.701+0000 7ffbd8000740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560414d84770 0x560414d8e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.701+0000 7ffbd8000740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560414d898e0 0x560414d95810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.701+0000 7ffbd8000740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x560414d8c890 con 0x560414d898e0 2024-03-23T23:01:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.701+0000 7ffbd8000740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x560414ca4dd0 con 0x560414d84770 2024-03-23T23:01:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.701+0000 7ffbd8000740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x560414d8ca10 con 0x560414d83d80 2024-03-23T23:01:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.702+0000 7ffbd8000740 10 client.0 _ll_register_callbacks cb 0x5604149fcac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:01:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.702+0000 7ffbcf200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560414d898e0 0x560414d95810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.702+0000 7ffbcf200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560414d898e0 0x560414d95810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54540/0 (socket says 172.21.15.12:54540) 2024-03-23T23:01:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.702+0000 7ffbcf200640 1 -- 172.21.15.12:0/406087097 learned_addr learned my addr 172.21.15.12:0/406087097 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.702+0000 7ffbcf200640 1 -- 172.21.15.12:0/406087097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560414d83d80 msgr2=0x560414d8e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.702+0000 7ffbcf200640 1 --2- 172.21.15.12:0/406087097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560414d83d80 0x560414d8e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.702+0000 7ffbcf200640 1 -- 172.21.15.12:0/406087097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560414d84770 msgr2=0x560414d8e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.702+0000 7ffbcf200640 1 --2- 172.21.15.12:0/406087097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560414d84770 0x560414d8e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.702+0000 7ffbcf200640 1 -- 172.21.15.12:0/406087097 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5604149fd2d0 con 0x560414d898e0 2024-03-23T23:01:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.702+0000 7ffbcf200640 1 --2- 172.21.15.12:0/406087097 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560414d898e0 0x560414d95810 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ffbc80ba170 tx=0x7ffbc80bd630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b4a0e4b28badd19 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.702+0000 7ffbafe00640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:01:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.703+0000 7ffbafe00640 1 -- 172.21.15.12:0/406087097 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc80d5020 con 0x560414d898e0 2024-03-23T23:01:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.703+0000 7ffbafe00640 1 -- 172.21.15.12:0/406087097 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ffbc80c5cb0 con 0x560414d898e0 2024-03-23T23:01:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.703+0000 7ffbd8000740 1 -- 172.21.15.12:0/406087097 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560414c8a520 con 0x560414d898e0 2024-03-23T23:01:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.703+0000 7ffbafe00640 1 -- 172.21.15.12:0/406087097 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffbc8004c60 con 0x560414d898e0 2024-03-23T23:01:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.703+0000 7ffbd8000740 10 client.14892 fetch_fsmap learned FSMap version 22 2024-03-23T23:01:08.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.703+0000 7ffbd8000740 1 -- 172.21.15.12:0/406087097 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x560414ac76b0 con 0x560414d898e0 2024-03-23T23:01:08.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.703+0000 7ffbafe00640 1 -- 172.21.15.12:0/406087097 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x560414c8a520 con 0x560414d898e0 2024-03-23T23:01:08.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.703+0000 7ffbafe00640 1 -- 172.21.15.12:0/406087097 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ffbc80d1b80 con 0x560414d898e0 2024-03-23T23:01:08.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.703+0000 7ffbd8000740 10 client.14892 fetch_fsmap finished waiting for FSMap version 22 2024-03-23T23:01:08.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.703+0000 7ffbd8000740 10 client.14892 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-23T23:01:08.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.703+0000 7ffbd8000740 20 client.14892 populate_metadata read hostname 'smithi012' 2024-03-23T23:01:08.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.703+0000 7ffbd8000740 1 --2- 172.21.15.12:0/406087097 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x560414d9ab80 0x560414d99970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.704+0000 7ffbd8000740 4 client.14892 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:01:08.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.704+0000 7ffbd8000740 1 -- 172.21.15.12:0/406087097 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560414cf61d0 con 0x560414d9ab80 2024-03-23T23:01:08.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.704+0000 7ffbce800640 1 --2- 172.21.15.12:0/406087097 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x560414d9ab80 0x560414d99970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.705+0000 7ffbce800640 1 --2- 172.21.15.12:0/406087097 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x560414d9ab80 0x560414d99970 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=eedb1a0429a9584c server_cookie=84d33d0bfd4f1d42 in_seq=0 out_seq=0 2024-03-23T23:01:08.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.705+0000 7ffbafe00640 10 client.14892 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:01:08.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.705+0000 7ffbafe00640 1 -- 172.21.15.12:0/406087097 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560414cf61d0 con 0x560414d9ab80 2024-03-23T23:01:08.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.705+0000 7ffbafe00640 10 client.14892 handle_command_reply: tid=0 2024-03-23T23:01:08.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.706+0000 7ffbd8000740 1 client.14892 shutdown 2024-03-23T23:01:08.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.706+0000 7ffbd8000740 1 -- 172.21.15.12:0/406087097 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560414d898e0 msgr2=0x560414d95810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:08.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.706+0000 7ffbd8000740 1 --2- 172.21.15.12:0/406087097 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560414d898e0 0x560414d95810 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ffbc80ba170 tx=0x7ffbc80bd630 comp rx=0 tx=0).stop 2024-03-23T23:01:08.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.706+0000 7ffbd8000740 1 -- 172.21.15.12:0/406087097 shutdown_connections 2024-03-23T23:01:08.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.706+0000 7ffbd8000740 1 --2- 172.21.15.12:0/406087097 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560414d898e0 0x560414d95810 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.706+0000 7ffbd8000740 1 --2- 172.21.15.12:0/406087097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560414d84770 0x560414d8e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.706+0000 7ffbd8000740 1 --2- 172.21.15.12:0/406087097 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x560414d9ab80 0x560414d99970 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.706+0000 7ffbd8000740 1 --2- 172.21.15.12:0/406087097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560414d83d80 0x560414d8e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.706+0000 7ffbd8000740 1 -- 172.21.15.12:0/406087097 >> 172.21.15.12:0/406087097 conn(0x560414d7f2f0 msgr2=0x560414d7e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:08.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.706+0000 7ffbafe00640 0 client.14892 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:01:08.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.706+0000 7ffbd8000740 1 -- 172.21.15.12:0/406087097 shutdown_connections 2024-03-23T23:01:08.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.706+0000 7ffbd8000740 1 -- 172.21.15.12:0/406087097 wait complete. 2024-03-23T23:01:08.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.706+0000 7ffbd8000740 20 client.14892 trim_cache size 0 max 16384 2024-03-23T23:01:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.725+0000 7ffbd8000740 1 Processor -- start 2024-03-23T23:01:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.725+0000 7ffbd8000740 1 -- start start 2024-03-23T23:01:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.725+0000 7ffbd8000740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560414d83d80 0x560414d85c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.725+0000 7ffbd8000740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560414d84770 0x560414d861b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.725+0000 7ffbd8000740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560414d898e0 0x560414d957d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.725+0000 7ffbd8000740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x560414d9a3a0 con 0x560414d83d80 2024-03-23T23:01:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.725+0000 7ffbd8000740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x560414d9aa00 con 0x560414d898e0 2024-03-23T23:01:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.725+0000 7ffbd8000740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x560414d9b060 con 0x560414d84770 2024-03-23T23:01:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.726+0000 7ffbcde00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560414d84770 0x560414d861b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.726+0000 7ffbcde00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560414d84770 0x560414d861b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49264/0 (socket says 172.21.15.12:49264) 2024-03-23T23:01:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.726+0000 7ffbcde00640 1 -- 172.21.15.12:0/2698785799 learned_addr learned my addr 172.21.15.12:0/2698785799 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.726+0000 7ffbcf200640 1 --2- 172.21.15.12:0/2698785799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560414d898e0 0x560414d957d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.726+0000 7ffbcde00640 1 -- 172.21.15.12:0/2698785799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560414d898e0 msgr2=0x560414d957d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.726+0000 7ffbcde00640 1 --2- 172.21.15.12:0/2698785799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560414d898e0 0x560414d957d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.726+0000 7ffbcde00640 1 -- 172.21.15.12:0/2698785799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560414d83d80 msgr2=0x560414d85c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.726+0000 7ffbce800640 1 --2- 172.21.15.12:0/2698785799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560414d83d80 0x560414d85c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.726+0000 7ffbcde00640 1 --2- 172.21.15.12:0/2698785799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560414d83d80 0x560414d85c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.726+0000 7ffbcde00640 1 -- 172.21.15.12:0/2698785799 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560414d85000 con 0x560414d84770 2024-03-23T23:01:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.726+0000 7ffbce800640 1 --2- 172.21.15.12:0/2698785799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560414d83d80 0x560414d85c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:01:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.726+0000 7ffbcde00640 1 --2- 172.21.15.12:0/2698785799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560414d84770 0x560414d861b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc018700 tx=0x7ffbbc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddd8de9e1a9ee95e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.727+0000 7ffbbaa00640 1 -- 172.21.15.12:0/2698785799 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc027070 con 0x560414d84770 2024-03-23T23:01:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.727+0000 7ffbbaa00640 1 -- 172.21.15.12:0/2698785799 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffbbc01d070 con 0x560414d84770 2024-03-23T23:01:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.727+0000 7ffbbaa00640 1 -- 172.21.15.12:0/2698785799 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc023c40 con 0x560414d84770 2024-03-23T23:01:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.727+0000 7ffbd8000740 1 -- 172.21.15.12:0/2698785799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560414d84770 msgr2=0x560414d861b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.727+0000 7ffbd8000740 1 --2- 172.21.15.12:0/2698785799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560414d84770 0x560414d861b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc018700 tx=0x7ffbbc0046c0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.727+0000 7ffbd8000740 1 -- 172.21.15.12:0/2698785799 shutdown_connections 2024-03-23T23:01:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.727+0000 7ffbd8000740 1 --2- 172.21.15.12:0/2698785799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560414d898e0 0x560414d957d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.727+0000 7ffbd8000740 1 --2- 172.21.15.12:0/2698785799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560414d84770 0x560414d861b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.727+0000 7ffbd8000740 1 --2- 172.21.15.12:0/2698785799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560414d83d80 0x560414d85c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.727+0000 7ffbd8000740 1 -- 172.21.15.12:0/2698785799 >> 172.21.15.12:0/2698785799 conn(0x560414d7f2f0 msgr2=0x560414d8e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.727+0000 7ffbd8000740 1 -- 172.21.15.12:0/2698785799 shutdown_connections 2024-03-23T23:01:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.727+0000 7ffbd8000740 1 -- 172.21.15.12:0/2698785799 wait complete. 2024-03-23T23:01:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.728+0000 7ffbd8000740 1 -- start start 2024-03-23T23:01:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.728+0000 7ffbd8000740 1 Processor -- start 2024-03-23T23:01:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.728+0000 7ffbd8000740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560414d83d80 0x560414d951f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.728+0000 7ffbd8000740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560414d84770 0x560414d95730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.728+0000 7ffbd8000740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560414d898e0 0x560414d91c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.728+0000 7ffbd8000740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x560414d9bd20 con 0x560414d898e0 2024-03-23T23:01:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.728+0000 7ffbd8000740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x560414d9b6c0 con 0x560414d84770 2024-03-23T23:01:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.728+0000 7ffbd8000740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x560414d9c380 con 0x560414d83d80 2024-03-23T23:01:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.728+0000 7ffbd8000740 10 client.0 _ll_register_callbacks cb 0x7ffbc80045b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:01:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.729+0000 7ffbcde00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560414d84770 0x560414d95730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.729+0000 7ffbce800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560414d83d80 0x560414d951f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.729+0000 7ffbcde00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560414d84770 0x560414d95730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57044/0 (socket says 172.21.15.12:57044) 2024-03-23T23:01:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.729+0000 7ffbcde00640 1 -- 172.21.15.12:0/4079226646 learned_addr learned my addr 172.21.15.12:0/4079226646 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.729+0000 7ffbcf200640 1 --2- 172.21.15.12:0/4079226646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560414d898e0 0x560414d91c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.729+0000 7ffbcde00640 1 -- 172.21.15.12:0/4079226646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560414d83d80 msgr2=0x560414d951f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.729+0000 7ffbcde00640 1 --2- 172.21.15.12:0/4079226646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560414d83d80 0x560414d951f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.729+0000 7ffbcde00640 1 -- 172.21.15.12:0/4079226646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560414d898e0 msgr2=0x560414d91c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.729+0000 7ffbcde00640 1 --2- 172.21.15.12:0/4079226646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560414d898e0 0x560414d91c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.729+0000 7ffbcde00640 1 -- 172.21.15.12:0/4079226646 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560414d92470 con 0x560414d84770 2024-03-23T23:01:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.729+0000 7ffbce800640 1 --2- 172.21.15.12:0/4079226646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560414d83d80 0x560414d951f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.729+0000 7ffbcde00640 1 --2- 172.21.15.12:0/4079226646 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560414d84770 0x560414d95730 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc0198c0 tx=0x7ffbbc0198f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c96d7cc44081bbc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.729+0000 7ffbafe00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:01:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.729+0000 7ffbafe00640 1 -- 172.21.15.12:0/4079226646 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc032070 con 0x560414d84770 2024-03-23T23:01:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.730+0000 7ffbd8000740 1 -- 172.21.15.12:0/4079226646 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffbbc01d070 con 0x560414d84770 2024-03-23T23:01:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.730+0000 7ffbafe00640 1 -- 172.21.15.12:0/4079226646 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ffbbc02c450 con 0x560414d84770 2024-03-23T23:01:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.730+0000 7ffbafe00640 1 -- 172.21.15.12:0/4079226646 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffbbc027040 con 0x560414d84770 2024-03-23T23:01:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.730+0000 7ffbd8000740 10 client.25022 fetch_fsmap learned FSMap version 22 2024-03-23T23:01:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.730+0000 7ffbd8000740 1 -- 172.21.15.12:0/4079226646 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x560414df68d0 con 0x560414d84770 2024-03-23T23:01:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.730+0000 7ffbafe00640 1 -- 172.21.15.12:0/4079226646 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffbbc01d070 con 0x560414d84770 2024-03-23T23:01:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.730+0000 7ffbafe00640 1 -- 172.21.15.12:0/4079226646 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ffbbc022070 con 0x560414d84770 2024-03-23T23:01:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.730+0000 7ffbd8000740 10 client.25022 fetch_fsmap finished waiting for FSMap version 22 2024-03-23T23:01:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.731+0000 7ffbd8000740 10 client.25022 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-23T23:01:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.731+0000 7ffbd8000740 20 client.25022 populate_metadata read hostname 'smithi012' 2024-03-23T23:01:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.731+0000 7ffbd8000740 1 --2- 172.21.15.12:0/4079226646 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x560414dfac50 0x560414df6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.731+0000 7ffbd8000740 4 client.25022 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-23T23:01:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.731+0000 7ffbd8000740 1 -- 172.21.15.12:0/4079226646 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560414cf0520 con 0x560414dfac50 2024-03-23T23:01:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.731+0000 7ffbce800640 1 --2- 172.21.15.12:0/4079226646 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x560414dfac50 0x560414df6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.732+0000 7ffbce800640 1 --2- 172.21.15.12:0/4079226646 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x560414dfac50 0x560414df6bc0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4ef4470bd2f9050e server_cookie=88a66615a658da6d in_seq=0 out_seq=0 2024-03-23T23:01:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.732+0000 7ffbafe00640 10 client.25022 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:01:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.732+0000 7ffbafe00640 1 -- 172.21.15.12:0/4079226646 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x560414cf0520 con 0x560414dfac50 2024-03-23T23:01:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.732+0000 7ffbafe00640 10 client.25022 handle_command_reply: tid=0 2024-03-23T23:01:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.732+0000 7ffbd8000740 1 client.25022 shutdown 2024-03-23T23:01:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.732+0000 7ffbd8000740 1 -- 172.21.15.12:0/4079226646 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560414d84770 msgr2=0x560414d95730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.732+0000 7ffbd8000740 1 --2- 172.21.15.12:0/4079226646 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560414d84770 0x560414d95730 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc0198c0 tx=0x7ffbbc0198f0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.732+0000 7ffbd8000740 1 -- 172.21.15.12:0/4079226646 shutdown_connections 2024-03-23T23:01:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.732+0000 7ffbd8000740 1 --2- 172.21.15.12:0/4079226646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560414d898e0 0x560414d91c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.732+0000 7ffbd8000740 1 --2- 172.21.15.12:0/4079226646 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560414d84770 0x560414d95730 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.732+0000 7ffbd8000740 1 --2- 172.21.15.12:0/4079226646 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x560414dfac50 0x560414df6bc0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.732+0000 7ffbd8000740 1 --2- 172.21.15.12:0/4079226646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560414d83d80 0x560414d951f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.732+0000 7ffbd8000740 1 -- 172.21.15.12:0/4079226646 >> 172.21.15.12:0/4079226646 conn(0x560414d7f2f0 msgr2=0x560414d8e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.732+0000 7ffbafe00640 0 client.25022 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:01:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.732+0000 7ffbd8000740 1 -- 172.21.15.12:0/4079226646 shutdown_connections 2024-03-23T23:01:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.732+0000 7ffbd8000740 1 -- 172.21.15.12:0/4079226646 wait complete. 2024-03-23T23:01:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.732+0000 7ffbd8000740 20 client.25022 trim_cache size 0 max 16384 2024-03-23T23:01:08.737 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:01:08.738 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "90f679ae-e967-11ee-95cd-87774f69a715", 2024-03-23T23:01:08.738 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 1, 2024-03-23T23:01:08.738 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 24554, 2024-03-23T23:01:08.738 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-23T23:01:08.738 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-23T23:01:08.738 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-23T23:01:08.738 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 98.257386134000001, 2024-03-23T23:01:08.738 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 22, 2024-03-23T23:01:08.738 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 88, 2024-03-23T23:01:08.738 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 86, 2024-03-23T23:01:08.738 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 122.20248870899999 2024-03-23T23:01:08.738 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:01:08.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.733+0000 7ffbd4a00640 1 -- 172.21.15.12:0/3726045297 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ffbb40a2a20 msgr2=0x7ffbb40a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:08.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.733+0000 7ffbd4a00640 1 --2- 172.21.15.12:0/3726045297 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ffbb40a2a20 0x7ffbb40a4ee0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x560414d84fd0 tx=0x7ffbbc0094c0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.733+0000 7ffbd4a00640 1 -- 172.21.15.12:0/3726045297 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbd010db70 msgr2=0x7ffbd01a5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:08.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.733+0000 7ffbd4a00640 1 --2- 172.21.15.12:0/3726045297 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbd010db70 0x7ffbd01a5530 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8004f50 tx=0x7ffbc8005e20 comp rx=0 tx=0).stop 2024-03-23T23:01:08.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.733+0000 7ffbd4a00640 1 -- 172.21.15.12:0/3726045297 shutdown_connections 2024-03-23T23:01:08.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.733+0000 7ffbd4a00640 1 --2- 172.21.15.12:0/3726045297 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ffbb40a2a20 0x7ffbb40a4ee0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.733+0000 7ffbd4a00640 1 --2- 172.21.15.12:0/3726045297 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbd010db70 0x7ffbd01a5530 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.733+0000 7ffbd4a00640 1 --2- 172.21.15.12:0/3726045297 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbd00756e0 0x7ffbd019e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.733+0000 7ffbd4a00640 1 --2- 172.21.15.12:0/3726045297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbd0074100 0x7ffbd019df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.733+0000 7ffbd4a00640 1 -- 172.21.15.12:0/3726045297 >> 172.21.15.12:0/3726045297 conn(0x7ffbd00fcb10 msgr2=0x7ffbd0100f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:08.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.733+0000 7ffbd4a00640 1 -- 172.21.15.12:0/3726045297 shutdown_connections 2024-03-23T23:01:08.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.733+0000 7ffbd4a00640 1 -- 172.21.15.12:0/3726045297 wait complete. 2024-03-23T23:01:08.750 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-23T23:01:08.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:08 smithi173 ceph-mon[36116]: pgmap v370: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-23T23:01:08.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:08 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/955487466' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:08.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:08 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1455569712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:01:08.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.956+0000 7f3633800640 1 -- 172.21.15.12:0/1842556582 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f362c1016b0 msgr2=0x7f362c101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:08.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.956+0000 7f3633800640 1 --2- 172.21.15.12:0/1842556582 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f362c1016b0 0x7f362c101ad0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3620009900 tx=0x7f362002bc40 comp rx=0 tx=0).stop 2024-03-23T23:01:08.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.956+0000 7f3633800640 1 -- 172.21.15.12:0/1842556582 shutdown_connections 2024-03-23T23:01:08.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.956+0000 7f3633800640 1 --2- 172.21.15.12:0/1842556582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f362c103260 0x7f362c109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.956+0000 7f3633800640 1 --2- 172.21.15.12:0/1842556582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f362c1028a0 0x7f362c102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.956+0000 7f3633800640 1 --2- 172.21.15.12:0/1842556582 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f362c1016b0 0x7f362c101ad0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.956+0000 7f3633800640 1 -- 172.21.15.12:0/1842556582 >> 172.21.15.12:0/1842556582 conn(0x7f362c0fcc40 msgr2=0x7f362c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:08.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.956+0000 7f3633800640 1 -- 172.21.15.12:0/1842556582 shutdown_connections 2024-03-23T23:01:08.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.956+0000 7f3633800640 1 -- 172.21.15.12:0/1842556582 wait complete. 2024-03-23T23:01:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.957+0000 7f3633800640 1 Processor -- start 2024-03-23T23:01:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.957+0000 7f3633800640 1 -- start start 2024-03-23T23:01:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.958+0000 7f3633800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f362c1016b0 0x7f362c199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.958+0000 7f3633800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f362c1028a0 0x7f362c19a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.958+0000 7f3633800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f362c103260 0x7f362c1a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.958+0000 7f3633800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362c10d380 con 0x7f362c1016b0 2024-03-23T23:01:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.958+0000 7f3633800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362c10d200 con 0x7f362c1028a0 2024-03-23T23:01:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.958+0000 7f3633800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362c10d500 con 0x7f362c103260 2024-03-23T23:01:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.958+0000 7f3630e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f362c1016b0 0x7f362c199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.958+0000 7f3630e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f362c1016b0 0x7f362c199c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54588/0 (socket says 172.21.15.12:54588) 2024-03-23T23:01:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.958+0000 7f3630e00640 1 -- 172.21.15.12:0/414557331 learned_addr learned my addr 172.21.15.12:0/414557331 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.958+0000 7f362be00640 1 --2- 172.21.15.12:0/414557331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f362c1028a0 0x7f362c19a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.958+0000 7f3630e00640 1 -- 172.21.15.12:0/414557331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f362c103260 msgr2=0x7f362c1a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.958+0000 7f3630e00640 1 --2- 172.21.15.12:0/414557331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f362c103260 0x7f362c1a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.958+0000 7f3630e00640 1 -- 172.21.15.12:0/414557331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f362c1028a0 msgr2=0x7f362c19a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.958+0000 7f3630e00640 1 --2- 172.21.15.12:0/414557331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f362c1028a0 0x7f362c19a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.958+0000 7f3630e00640 1 -- 172.21.15.12:0/414557331 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f362c1a1840 con 0x7f362c1016b0 2024-03-23T23:01:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.958+0000 7f362be00640 1 --2- 172.21.15.12:0/414557331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f362c1028a0 0x7f362c19a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:01:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.959+0000 7f3630e00640 1 --2- 172.21.15.12:0/414557331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f362c1016b0 0x7f362c199c30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f362002efd0 tx=0x7f362002c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.959+0000 7f3629600640 1 -- 172.21.15.12:0/414557331 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3620041070 con 0x7f362c1016b0 2024-03-23T23:01:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.959+0000 7f3629600640 1 -- 172.21.15.12:0/414557331 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3620004450 con 0x7f362c1016b0 2024-03-23T23:01:08.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.959+0000 7f3633800640 1 -- 172.21.15.12:0/414557331 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f362c1a1ad0 con 0x7f362c1016b0 2024-03-23T23:01:08.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.959+0000 7f3629600640 1 -- 172.21.15.12:0/414557331 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3620003e50 con 0x7f362c1016b0 2024-03-23T23:01:08.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.959+0000 7f3633800640 1 -- 172.21.15.12:0/414557331 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f362c1a1f90 con 0x7f362c1016b0 2024-03-23T23:01:08.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.962+0000 7f3629600640 1 -- 172.21.15.12:0/414557331 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3620044070 con 0x7f362c1016b0 2024-03-23T23:01:08.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.962+0000 7f36365a5740 1 Processor -- start 2024-03-23T23:01:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.963+0000 7f36365a5740 1 -- start start 2024-03-23T23:01:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.963+0000 7f36365a5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561c0890bba0 0x561c0890bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.963+0000 7f36365a5740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561c0890c590 0x561c08911090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.963+0000 7f36365a5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561c08911700 0x561c08913b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.963+0000 7f36365a5740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c08870c90 con 0x561c0890c590 2024-03-23T23:01:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.963+0000 7f36365a5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c08819980 con 0x561c0890bba0 2024-03-23T23:01:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.963+0000 7f36365a5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c0886cd30 con 0x561c08911700 2024-03-23T23:01:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.963+0000 7f3630e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561c0890c590 0x561c08911090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.963+0000 7f3630e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561c0890c590 0x561c08911090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54600/0 (socket says 172.21.15.12:54600) 2024-03-23T23:01:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.963+0000 7f3630e00640 1 -- 172.21.15.12:0/1401069365 learned_addr learned my addr 172.21.15.12:0/1401069365 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.964+0000 7f3631800640 1 --2- 172.21.15.12:0/1401069365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561c0890bba0 0x561c0890bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.964+0000 7f362be00640 1 --2- 172.21.15.12:0/1401069365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561c08911700 0x561c08913b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.964+0000 7f3630e00640 1 -- 172.21.15.12:0/1401069365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561c08911700 msgr2=0x561c08913b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:08.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.964+0000 7f3630e00640 1 --2- 172.21.15.12:0/1401069365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561c08911700 0x561c08913b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.964+0000 7f3630e00640 1 -- 172.21.15.12:0/1401069365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561c0890bba0 msgr2=0x561c0890bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:08.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.964+0000 7f3630e00640 1 --2- 172.21.15.12:0/1401069365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561c0890bba0 0x561c0890bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.964+0000 7f3630e00640 1 -- 172.21.15.12:0/1401069365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c08831510 con 0x561c0890c590 2024-03-23T23:01:08.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.964+0000 7f3629600640 1 --2- 172.21.15.12:0/414557331 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f36140a29a0 0x7f36140a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.964+0000 7f3630e00640 1 --2- 172.21.15.12:0/1401069365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561c0890c590 0x561c08911090 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f36200f6a90 tx=0x7f36200f6b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=349890f14646338f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:08.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.964+0000 7f3629600640 1 -- 172.21.15.12:0/414557331 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f36200f0810 con 0x7f362c1016b0 2024-03-23T23:01:08.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.964+0000 7f3631800640 1 --2- 172.21.15.12:0/414557331 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f36140a29a0 0x7f36140a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.964+0000 7f361e000640 1 -- 172.21.15.12:0/1401069365 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36200b9370 con 0x561c0890c590 2024-03-23T23:01:08.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.964+0000 7f361e000640 1 -- 172.21.15.12:0/1401069365 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3620004c60 con 0x561c0890c590 2024-03-23T23:01:08.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.964+0000 7f361e000640 1 -- 172.21.15.12:0/1401069365 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36200bb2a0 con 0x561c0890c590 2024-03-23T23:01:08.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.965+0000 7f36365a5740 1 -- 172.21.15.12:0/1401069365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561c0890c590 msgr2=0x561c08911090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:08.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.965+0000 7f36365a5740 1 --2- 172.21.15.12:0/1401069365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561c0890c590 0x561c08911090 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f36200f6a90 tx=0x7f36200f6b80 comp rx=0 tx=0).stop 2024-03-23T23:01:08.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.965+0000 7f3631800640 1 --2- 172.21.15.12:0/414557331 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f36140a29a0 0x7f36140a4e60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x561c0890cdf0 tx=0x7f3624009b60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:08.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.965+0000 7f36365a5740 1 -- 172.21.15.12:0/1401069365 shutdown_connections 2024-03-23T23:01:08.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.965+0000 7f36365a5740 1 --2- 172.21.15.12:0/1401069365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561c08911700 0x561c08913b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.965+0000 7f36365a5740 1 --2- 172.21.15.12:0/1401069365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561c0890c590 0x561c08911090 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.965+0000 7f36365a5740 1 --2- 172.21.15.12:0/1401069365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561c0890bba0 0x561c0890bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.965+0000 7f36365a5740 1 -- 172.21.15.12:0/1401069365 >> 172.21.15.12:0/1401069365 conn(0x561c08907110 msgr2=0x561c08907510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:08.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.965+0000 7f36365a5740 1 -- 172.21.15.12:0/1401069365 shutdown_connections 2024-03-23T23:01:08.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.966+0000 7f36365a5740 1 -- 172.21.15.12:0/1401069365 wait complete. 2024-03-23T23:01:08.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.966+0000 7f36365a5740 1 -- start start 2024-03-23T23:01:08.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.966+0000 7f36365a5740 1 Processor -- start 2024-03-23T23:01:08.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.966+0000 7f36365a5740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561c0890bba0 0x561c08915d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.967+0000 7f36365a5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561c0890c590 0x561c08916290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.967+0000 7f36365a5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561c08911700 0x561c0891d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.967+0000 7f36365a5740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c08870c90 con 0x561c0890bba0 2024-03-23T23:01:08.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.967+0000 7f36365a5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c0882cdd0 con 0x561c08911700 2024-03-23T23:01:08.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.967+0000 7f36365a5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c08914470 con 0x561c0890c590 2024-03-23T23:01:08.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.967+0000 7f36365a5740 10 client.0 _ll_register_callbacks cb 0x561c08584ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:01:08.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.967+0000 7f3630e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561c0890bba0 0x561c08915d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.967+0000 7f3630e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561c0890bba0 0x561c08915d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54610/0 (socket says 172.21.15.12:54610) 2024-03-23T23:01:08.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.967+0000 7f3630e00640 1 -- 172.21.15.12:0/912216009 learned_addr learned my addr 172.21.15.12:0/912216009 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:08.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.967+0000 7f3630e00640 1 -- 172.21.15.12:0/912216009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561c0890c590 msgr2=0x561c08916290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:08.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.967+0000 7f3630e00640 1 --2- 172.21.15.12:0/912216009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561c0890c590 0x561c08916290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.967+0000 7f3630e00640 1 -- 172.21.15.12:0/912216009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561c08911700 msgr2=0x561c0891d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:08.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.967+0000 7f3630e00640 1 --2- 172.21.15.12:0/912216009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561c08911700 0x561c0891d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.967+0000 7f3630e00640 1 -- 172.21.15.12:0/912216009 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c085852d0 con 0x561c0890bba0 2024-03-23T23:01:08.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.967+0000 7f3630e00640 1 --2- 172.21.15.12:0/912216009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561c0890bba0 0x561c08915d50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f36200f8670 tx=0x7f362002bc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39d2ef342761c14a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:08.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.967+0000 7f3612a00640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:01:08.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.968+0000 7f3612a00640 1 -- 172.21.15.12:0/912216009 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362004b070 con 0x561c0890bba0 2024-03-23T23:01:08.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.968+0000 7f3612a00640 1 -- 172.21.15.12:0/912216009 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f36200035f0 con 0x561c0890bba0 2024-03-23T23:01:08.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.968+0000 7f36365a5740 1 -- 172.21.15.12:0/912216009 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3620004c60 con 0x561c0890bba0 2024-03-23T23:01:08.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.968+0000 7f3612a00640 1 -- 172.21.15.12:0/912216009 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36200bcac0 con 0x561c0890bba0 2024-03-23T23:01:08.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.968+0000 7f36365a5740 10 client.14907 fetch_fsmap learned FSMap version 22 2024-03-23T23:01:08.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.968+0000 7f36365a5740 1 -- 172.21.15.12:0/912216009 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x561c0864f6b0 con 0x561c0890bba0 2024-03-23T23:01:08.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.968+0000 7f3612a00640 1 -- 172.21.15.12:0/912216009 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3620004c60 con 0x561c0890bba0 2024-03-23T23:01:08.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.968+0000 7f3612a00640 1 -- 172.21.15.12:0/912216009 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f36200bd3c0 con 0x561c0890bba0 2024-03-23T23:01:08.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.968+0000 7f36365a5740 10 client.14907 fetch_fsmap finished waiting for FSMap version 22 2024-03-23T23:01:08.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.969+0000 7f36365a5740 10 client.14907 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-23T23:01:08.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.969+0000 7f36365a5740 20 client.14907 populate_metadata read hostname 'smithi012' 2024-03-23T23:01:08.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.969+0000 7f36365a5740 1 --2- 172.21.15.12:0/912216009 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561c089222c0 0x561c08921430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.969+0000 7f36365a5740 4 client.14907 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:01:08.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.969+0000 7f36365a5740 1 -- 172.21.15.12:0/912216009 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561c08812520 con 0x561c089222c0 2024-03-23T23:01:08.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.969+0000 7f362be00640 1 --2- 172.21.15.12:0/912216009 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561c089222c0 0x561c08921430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:08.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.969+0000 7f362be00640 1 --2- 172.21.15.12:0/912216009 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561c089222c0 0x561c08921430 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9a35466bd6aecfa server_cookie=97de4791a4e2cd87 in_seq=0 out_seq=0 2024-03-23T23:01:08.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.969+0000 7f3612a00640 10 client.14907 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:01:08.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.970+0000 7f3612a00640 1 -- 172.21.15.12:0/912216009 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561c08812520 con 0x561c089222c0 2024-03-23T23:01:08.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.970+0000 7f3612a00640 10 client.14907 handle_command_reply: tid=0 2024-03-23T23:01:08.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.970+0000 7f36365a5740 1 client.14907 shutdown 2024-03-23T23:01:08.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.970+0000 7f36365a5740 1 -- 172.21.15.12:0/912216009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561c0890bba0 msgr2=0x561c08915d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:08.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.970+0000 7f36365a5740 1 --2- 172.21.15.12:0/912216009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561c0890bba0 0x561c08915d50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f36200f8670 tx=0x7f362002bc40 comp rx=0 tx=0).stop 2024-03-23T23:01:08.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.970+0000 7f36365a5740 1 -- 172.21.15.12:0/912216009 shutdown_connections 2024-03-23T23:01:08.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.970+0000 7f36365a5740 1 --2- 172.21.15.12:0/912216009 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561c089222c0 0x561c08921430 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.970+0000 7f36365a5740 1 --2- 172.21.15.12:0/912216009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561c08911700 0x561c0891d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.970+0000 7f36365a5740 1 --2- 172.21.15.12:0/912216009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561c0890c590 0x561c08916290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.970+0000 7f36365a5740 1 --2- 172.21.15.12:0/912216009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561c0890bba0 0x561c08915d50 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:08.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.970+0000 7f36365a5740 1 -- 172.21.15.12:0/912216009 >> 172.21.15.12:0/912216009 conn(0x561c08907110 msgr2=0x561c08906310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:08.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.970+0000 7f3612a00640 0 client.14907 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:01:08.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.971+0000 7f36365a5740 1 -- 172.21.15.12:0/912216009 shutdown_connections 2024-03-23T23:01:08.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.971+0000 7f36365a5740 1 -- 172.21.15.12:0/912216009 wait complete. 2024-03-23T23:01:08.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.971+0000 7f36365a5740 20 client.14907 trim_cache size 0 max 16384 2024-03-23T23:01:08.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.989+0000 7f36365a5740 1 Processor -- start 2024-03-23T23:01:08.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.989+0000 7f36365a5740 1 -- start start 2024-03-23T23:01:08.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.989+0000 7f36365a5740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561c0890bba0 0x561c089074f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.989+0000 7f36365a5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561c0890c590 0x561c0891d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.989+0000 7f36365a5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561c08911700 0x561c08915960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:08.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.989+0000 7f36365a5740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c08921ae0 con 0x561c0890bba0 2024-03-23T23:01:08.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.989+0000 7f36365a5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c08922140 con 0x561c0890c590 2024-03-23T23:01:08.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.989+0000 7f36365a5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c089227a0 con 0x561c08911700 2024-03-23T23:01:08.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.990+0000 7f3631800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561c08911700 0x561c08915960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:09.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.990+0000 7f3631800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561c08911700 0x561c08915960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49316/0 (socket says 172.21.15.12:49316) 2024-03-23T23:01:09.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.990+0000 7f3631800640 1 -- 172.21.15.12:0/3942477425 learned_addr learned my addr 172.21.15.12:0/3942477425 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:09.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.990+0000 7f3630e00640 1 --2- 172.21.15.12:0/3942477425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561c0890bba0 0x561c089074f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:09.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.990+0000 7f362be00640 1 --2- 172.21.15.12:0/3942477425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561c0890c590 0x561c0891d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:09.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.990+0000 7f3631800640 1 -- 172.21.15.12:0/3942477425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561c0890c590 msgr2=0x561c0891d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:09.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.990+0000 7f3631800640 1 --2- 172.21.15.12:0/3942477425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561c0890c590 0x561c0891d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:09.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.990+0000 7f3631800640 1 -- 172.21.15.12:0/3942477425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561c0890bba0 msgr2=0x561c089074f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:09.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.990+0000 7f3631800640 1 --2- 172.21.15.12:0/3942477425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561c0890bba0 0x561c089074f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:09.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.990+0000 7f3631800640 1 -- 172.21.15.12:0/3942477425 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c08905700 con 0x561c08911700 2024-03-23T23:01:09.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.990+0000 7f362be00640 1 --2- 172.21.15.12:0/3942477425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561c0890c590 0x561c0891d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:09.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.991+0000 7f3631800640 1 --2- 172.21.15.12:0/3942477425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561c08911700 0x561c08915960 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f36240115d0 tx=0x7f3624013070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68b9150efb487700 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:09.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.991+0000 7f361d600640 1 -- 172.21.15.12:0/3942477425 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362401e620 con 0x561c08911700 2024-03-23T23:01:09.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.991+0000 7f361d600640 1 -- 172.21.15.12:0/3942477425 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f362401eca0 con 0x561c08911700 2024-03-23T23:01:09.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.991+0000 7f361d600640 1 -- 172.21.15.12:0/3942477425 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3624026d50 con 0x561c08911700 2024-03-23T23:01:09.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.992+0000 7f36365a5740 1 -- 172.21.15.12:0/3942477425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561c08911700 msgr2=0x561c08915960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:09.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.992+0000 7f36365a5740 1 --2- 172.21.15.12:0/3942477425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561c08911700 0x561c08915960 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f36240115d0 tx=0x7f3624013070 comp rx=0 tx=0).stop 2024-03-23T23:01:09.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.992+0000 7f36365a5740 1 -- 172.21.15.12:0/3942477425 shutdown_connections 2024-03-23T23:01:09.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.992+0000 7f36365a5740 1 --2- 172.21.15.12:0/3942477425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561c08911700 0x561c08915960 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:09.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.992+0000 7f36365a5740 1 --2- 172.21.15.12:0/3942477425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561c0890c590 0x561c0891d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:09.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.992+0000 7f36365a5740 1 --2- 172.21.15.12:0/3942477425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561c0890bba0 0x561c089074f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:09.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.992+0000 7f36365a5740 1 -- 172.21.15.12:0/3942477425 >> 172.21.15.12:0/3942477425 conn(0x561c08907110 msgr2=0x561c089148a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:09.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.992+0000 7f36365a5740 1 -- 172.21.15.12:0/3942477425 shutdown_connections 2024-03-23T23:01:09.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.992+0000 7f36365a5740 1 -- 172.21.15.12:0/3942477425 wait complete. 2024-03-23T23:01:09.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.992+0000 7f36365a5740 1 -- start start 2024-03-23T23:01:09.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.992+0000 7f36365a5740 1 Processor -- start 2024-03-23T23:01:09.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.993+0000 7f36365a5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561c0890bba0 0x561c0891cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:09.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.993+0000 7f36365a5740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561c0890c590 0x561c08919b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:09.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.993+0000 7f36365a5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561c08911700 0x561c0891a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:09.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.993+0000 7f36365a5740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c08923460 con 0x561c0890c590 2024-03-23T23:01:09.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.993+0000 7f36365a5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c08922e00 con 0x561c0890bba0 2024-03-23T23:01:09.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.993+0000 7f36365a5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c08923ac0 con 0x561c08911700 2024-03-23T23:01:09.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.993+0000 7f36365a5740 10 client.0 _ll_register_callbacks cb 0x7f36200f68e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:01:09.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.993+0000 7f3630e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561c0890bba0 0x561c0891cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:09.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.993+0000 7f362be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561c0890c590 0x561c08919b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:09.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.993+0000 7f3630e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561c0890bba0 0x561c0891cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57096/0 (socket says 172.21.15.12:57096) 2024-03-23T23:01:09.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.993+0000 7f3630e00640 1 -- 172.21.15.12:0/1925232150 learned_addr learned my addr 172.21.15.12:0/1925232150 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:09.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.993+0000 7f3631800640 1 --2- 172.21.15.12:0/1925232150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561c08911700 0x561c0891a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:09.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.993+0000 7f3630e00640 1 -- 172.21.15.12:0/1925232150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561c08911700 msgr2=0x561c0891a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:09.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.993+0000 7f3630e00640 1 --2- 172.21.15.12:0/1925232150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561c08911700 0x561c0891a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:09.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.993+0000 7f3630e00640 1 -- 172.21.15.12:0/1925232150 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561c0890c590 msgr2=0x561c08919b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:09.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.993+0000 7f3630e00640 1 --2- 172.21.15.12:0/1925232150 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561c0890c590 0x561c08919b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:09.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.994+0000 7f3630e00640 1 -- 172.21.15.12:0/1925232150 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c0891a690 con 0x561c0890bba0 2024-03-23T23:01:09.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.994+0000 7f362be00640 1 --2- 172.21.15.12:0/1925232150 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561c0890c590 0x561c08919b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:09.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.994+0000 7f3630e00640 1 --2- 172.21.15.12:0/1925232150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561c0890bba0 0x561c0891cfd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f36200f69b0 tx=0x7f36200f6b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c2f81ffd543ef57 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:09.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.994+0000 7f3612a00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:01:09.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.994+0000 7f3612a00640 1 -- 172.21.15.12:0/1925232150 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362004b070 con 0x561c0890bba0 2024-03-23T23:01:09.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.994+0000 7f3612a00640 1 -- 172.21.15.12:0/1925232150 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f36200f00c0 con 0x561c0890bba0 2024-03-23T23:01:09.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.994+0000 7f36365a5740 1 -- 172.21.15.12:0/1925232150 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f362401eca0 con 0x561c0890bba0 2024-03-23T23:01:09.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.994+0000 7f3612a00640 1 -- 172.21.15.12:0/1925232150 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36200bcca0 con 0x561c0890bba0 2024-03-23T23:01:09.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.994+0000 7f36365a5740 10 client.25034 fetch_fsmap learned FSMap version 22 2024-03-23T23:01:09.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.994+0000 7f36365a5740 1 -- 172.21.15.12:0/1925232150 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x561c0891a690 con 0x561c0890bba0 2024-03-23T23:01:09.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.994+0000 7f3612a00640 1 -- 172.21.15.12:0/1925232150 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f362401eca0 con 0x561c0890bba0 2024-03-23T23:01:09.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.995+0000 7f3612a00640 1 -- 172.21.15.12:0/1925232150 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f36200bc5b0 con 0x561c0890bba0 2024-03-23T23:01:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.995+0000 7f36365a5740 10 client.25034 fetch_fsmap finished waiting for FSMap version 22 2024-03-23T23:01:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.995+0000 7f36365a5740 10 client.25034 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-23T23:01:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.995+0000 7f36365a5740 20 client.25034 populate_metadata read hostname 'smithi012' 2024-03-23T23:01:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.995+0000 7f36365a5740 1 --2- 172.21.15.12:0/1925232150 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561c08982630 0x561c0897e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.995+0000 7f36365a5740 4 client.25034 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-23T23:01:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.995+0000 7f36365a5740 1 -- 172.21.15.12:0/1925232150 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561c0887e1d0 con 0x561c08982630 2024-03-23T23:01:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.995+0000 7f362be00640 1 --2- 172.21.15.12:0/1925232150 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561c08982630 0x561c0897e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.997+0000 7f362be00640 1 --2- 172.21.15.12:0/1925232150 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561c08982630 0x561c0897e490 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=291d1317c32d38c3 server_cookie=42224246a5d28650 in_seq=0 out_seq=0 2024-03-23T23:01:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.997+0000 7f3612a00640 10 client.25034 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:01:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.997+0000 7f3612a00640 1 -- 172.21.15.12:0/1925232150 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x561c0887e1d0 con 0x561c08982630 2024-03-23T23:01:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.997+0000 7f3612a00640 10 client.25034 handle_command_reply: tid=0 2024-03-23T23:01:09.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.997+0000 7f36365a5740 1 client.25034 shutdown 2024-03-23T23:01:09.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.997+0000 7f36365a5740 1 -- 172.21.15.12:0/1925232150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561c0890bba0 msgr2=0x561c0891cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:09.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.997+0000 7f36365a5740 1 --2- 172.21.15.12:0/1925232150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561c0890bba0 0x561c0891cfd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f36200f69b0 tx=0x7f36200f6b80 comp rx=0 tx=0).stop 2024-03-23T23:01:09.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.997+0000 7f36365a5740 1 -- 172.21.15.12:0/1925232150 shutdown_connections 2024-03-23T23:01:09.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.997+0000 7f36365a5740 1 --2- 172.21.15.12:0/1925232150 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561c08982630 0x561c0897e490 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:09.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.997+0000 7f36365a5740 1 --2- 172.21.15.12:0/1925232150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561c08911700 0x561c0891a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:09.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.997+0000 7f36365a5740 1 --2- 172.21.15.12:0/1925232150 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561c0890c590 0x561c08919b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:09.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.997+0000 7f36365a5740 1 --2- 172.21.15.12:0/1925232150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561c0890bba0 0x561c0891cfd0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:09.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.997+0000 7f36365a5740 1 -- 172.21.15.12:0/1925232150 >> 172.21.15.12:0/1925232150 conn(0x561c08907110 msgr2=0x561c089074f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:09.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.997+0000 7f3612a00640 0 client.25034 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:01:09.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.997+0000 7f36365a5740 1 -- 172.21.15.12:0/1925232150 shutdown_connections 2024-03-23T23:01:09.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.997+0000 7f36365a5740 1 -- 172.21.15.12:0/1925232150 wait complete. 2024-03-23T23:01:09.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.997+0000 7f36365a5740 20 client.25034 trim_cache size 0 max 16384 2024-03-23T23:01:09.007 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:01:09.007 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "90f679ae-e967-11ee-95cd-87774f69a715", 2024-03-23T23:01:09.007 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-03-23T23:01:09.007 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 24599, 2024-03-23T23:01:09.007 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-23T23:01:09.007 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-23T23:01:09.007 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-23T23:01:09.007 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 101.559887836, 2024-03-23T23:01:09.007 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 22, 2024-03-23T23:01:09.007 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 88, 2024-03-23T23:01:09.007 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 86, 2024-03-23T23:01:09.007 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 108.145868218 2024-03-23T23:01:09.008 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:01:09.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.998+0000 7f3633800640 1 -- 172.21.15.12:0/414557331 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f36140a29a0 msgr2=0x7f36140a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:09.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.998+0000 7f3633800640 1 --2- 172.21.15.12:0/414557331 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f36140a29a0 0x7f36140a4e60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x561c0890cdf0 tx=0x7f3624009b60 comp rx=0 tx=0).stop 2024-03-23T23:01:09.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.998+0000 7f3633800640 1 -- 172.21.15.12:0/414557331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f362c1016b0 msgr2=0x7f362c199c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:09.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.998+0000 7f3633800640 1 --2- 172.21.15.12:0/414557331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f362c1016b0 0x7f362c199c30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f362002efd0 tx=0x7f362002c660 comp rx=0 tx=0).stop 2024-03-23T23:01:09.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.998+0000 7f3633800640 1 -- 172.21.15.12:0/414557331 shutdown_connections 2024-03-23T23:01:09.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.998+0000 7f3633800640 1 --2- 172.21.15.12:0/414557331 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f36140a29a0 0x7f36140a4e60 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:09.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.998+0000 7f3633800640 1 --2- 172.21.15.12:0/414557331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f362c103260 0x7f362c1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:09.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.998+0000 7f3633800640 1 --2- 172.21.15.12:0/414557331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f362c1028a0 0x7f362c19a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:09.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.998+0000 7f3633800640 1 --2- 172.21.15.12:0/414557331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f362c1016b0 0x7f362c199c30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:09.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.998+0000 7f3633800640 1 -- 172.21.15.12:0/414557331 >> 172.21.15.12:0/414557331 conn(0x7f362c0fcc40 msgr2=0x7f362c0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:09.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.998+0000 7f3633800640 1 -- 172.21.15.12:0/414557331 shutdown_connections 2024-03-23T23:01:09.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:08.998+0000 7f3633800640 1 -- 172.21.15.12:0/414557331 wait complete. 2024-03-23T23:01:09.018 INFO:tasks.cephadm:Setup complete, yielding 2024-03-23T23:01:09.018 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-23T23:01:09.029 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi012.front.sepia.ceph.com 2024-03-23T23:01:09.029 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- bash -c 'ceph orch status' 2024-03-23T23:01:09.481 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:01:10.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:10 smithi203 ceph-mon[36693]: pgmap v371: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:01:10.665 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:10 smithi012 ceph-mon[31695]: pgmap v371: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:01:10.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:10 smithi173 ceph-mon[36116]: pgmap v371: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:01:11.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.723+0000 7fae21e00700 1 -- 172.21.15.12:0/1584476008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae1c109a10 msgr2=0x7fae1c109e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:11.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.723+0000 7fae21e00700 1 --2- 172.21.15.12:0/1584476008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae1c109a10 0x7fae1c109e70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fae08009990 tx=0x7fae08009ca0 comp rx=0 tx=0).stop 2024-03-23T23:01:11.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.724+0000 7fae21e00700 1 -- 172.21.15.12:0/1584476008 shutdown_connections 2024-03-23T23:01:11.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.724+0000 7fae21e00700 1 --2- 172.21.15.12:0/1584476008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae1c10a3b0 0x7fae1c110c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:11.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.724+0000 7fae21e00700 1 --2- 172.21.15.12:0/1584476008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae1c108810 0x7fae1c108c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:11.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.724+0000 7fae21e00700 1 --2- 172.21.15.12:0/1584476008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae1c109a10 0x7fae1c109e70 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:11.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.724+0000 7fae21e00700 1 -- 172.21.15.12:0/1584476008 >> 172.21.15.12:0/1584476008 conn(0x7fae1c0777e0 msgr2=0x7fae1c079c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:11.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.724+0000 7fae21e00700 1 -- 172.21.15.12:0/1584476008 shutdown_connections 2024-03-23T23:01:11.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.724+0000 7fae21e00700 1 -- 172.21.15.12:0/1584476008 wait complete. 2024-03-23T23:01:11.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.725+0000 7fae21e00700 1 Processor -- start 2024-03-23T23:01:11.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.725+0000 7fae21e00700 1 -- start start 2024-03-23T23:01:11.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.726+0000 7fae21e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae1c108810 0x7fae1c19e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:11.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.726+0000 7fae21e00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae1c109a10 0x7fae1c19ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:11.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.726+0000 7fae21e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae1c10a3b0 0x7fae1c1a5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:11.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.726+0000 7fae21e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1c114420 con 0x7fae1c10a3b0 2024-03-23T23:01:11.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.726+0000 7fae21e00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1c1142a0 con 0x7fae1c108810 2024-03-23T23:01:11.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.726+0000 7fae21e00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1c1145a0 con 0x7fae1c109a10 2024-03-23T23:01:11.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.726+0000 7fae1aa00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae1c108810 0x7fae1c19e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:11.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.726+0000 7fae1aa00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae1c108810 0x7fae1c19e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57110/0 (socket says 172.21.15.12:57110) 2024-03-23T23:01:11.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.726+0000 7fae1aa00700 1 -- 172.21.15.12:0/576221577 learned_addr learned my addr 172.21.15.12:0/576221577 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:11.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.726+0000 7fae1a000700 1 --2- 172.21.15.12:0/576221577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae1c109a10 0x7fae1c19ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:11.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.726+0000 7fae1b400700 1 --2- 172.21.15.12:0/576221577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae1c10a3b0 0x7fae1c1a5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:11.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.727+0000 7fae1aa00700 1 -- 172.21.15.12:0/576221577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae1c109a10 msgr2=0x7fae1c19ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:11.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.727+0000 7fae1aa00700 1 --2- 172.21.15.12:0/576221577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae1c109a10 0x7fae1c19ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:11.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.727+0000 7fae1aa00700 1 -- 172.21.15.12:0/576221577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae1c10a3b0 msgr2=0x7fae1c1a5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:11.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.727+0000 7fae1aa00700 1 --2- 172.21.15.12:0/576221577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae1c10a3b0 0x7fae1c1a5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:11.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.727+0000 7fae1aa00700 1 -- 172.21.15.12:0/576221577 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1c1a6630 con 0x7fae1c108810 2024-03-23T23:01:11.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.727+0000 7fae1a000700 1 --2- 172.21.15.12:0/576221577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae1c109a10 0x7fae1c19ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:11.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.727+0000 7fae1aa00700 1 --2- 172.21.15.12:0/576221577 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae1c108810 0x7fae1c19e9d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fae0c00ea50 tx=0x7fae0c00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:11.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.727+0000 7fae20a00700 1 -- 172.21.15.12:0/576221577 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0c00cbd0 con 0x7fae1c108810 2024-03-23T23:01:11.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.727+0000 7fae20a00700 1 -- 172.21.15.12:0/576221577 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae0c00cd70 con 0x7fae1c108810 2024-03-23T23:01:11.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.727+0000 7fae20a00700 1 -- 172.21.15.12:0/576221577 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0c0187a0 con 0x7fae1c108810 2024-03-23T23:01:11.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.727+0000 7fae21e00700 1 -- 172.21.15.12:0/576221577 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae1c1a6920 con 0x7fae1c108810 2024-03-23T23:01:11.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.728+0000 7fae21e00700 1 -- 172.21.15.12:0/576221577 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae1c1a6e10 con 0x7fae1c108810 2024-03-23T23:01:11.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.729+0000 7fae21e00700 1 -- 172.21.15.12:0/576221577 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae1c068e50 con 0x7fae1c108810 2024-03-23T23:01:11.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.731+0000 7fae20a00700 1 -- 172.21.15.12:0/576221577 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fae0c01f020 con 0x7fae1c108810 2024-03-23T23:01:11.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.732+0000 7fae20a00700 1 --2- 172.21.15.12:0/576221577 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fae000a2910 0x7fae000a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:11.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.732+0000 7fae20a00700 1 -- 172.21.15.12:0/576221577 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fae0c0ceae0 con 0x7fae1c108810 2024-03-23T23:01:11.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.732+0000 7fae1a000700 1 --2- 172.21.15.12:0/576221577 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fae000a2910 0x7fae000a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:11.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.733+0000 7fae1a000700 1 --2- 172.21.15.12:0/576221577 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fae000a2910 0x7fae000a4dd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fae0800b2e0 tx=0x7fae08005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:11.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.736+0000 7fae20a00700 1 -- 172.21.15.12:0/576221577 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fae0c053740 con 0x7fae1c108810 2024-03-23T23:01:11.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.985+0000 7fae21e00700 1 -- 172.21.15.12:0/576221577 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fae1c0625d0 con 0x7fae000a2910 2024-03-23T23:01:11.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.988+0000 7fae20a00700 1 -- 172.21.15.12:0/576221577 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fae1c0625d0 con 0x7fae000a2910 2024-03-23T23:01:11.988 INFO:teuthology.orchestra.run.smithi012.stdout:Backend: cephadm 2024-03-23T23:01:11.989 INFO:teuthology.orchestra.run.smithi012.stdout:Available: Yes 2024-03-23T23:01:11.989 INFO:teuthology.orchestra.run.smithi012.stdout:Paused: No 2024-03-23T23:01:11.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.991+0000 7fae21e00700 1 -- 172.21.15.12:0/576221577 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fae000a2910 msgr2=0x7fae000a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:11.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.991+0000 7fae21e00700 1 --2- 172.21.15.12:0/576221577 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fae000a2910 0x7fae000a4dd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fae0800b2e0 tx=0x7fae08005fd0 comp rx=0 tx=0).stop 2024-03-23T23:01:11.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.991+0000 7fae21e00700 1 -- 172.21.15.12:0/576221577 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae1c108810 msgr2=0x7fae1c19e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:11.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.991+0000 7fae21e00700 1 --2- 172.21.15.12:0/576221577 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae1c108810 0x7fae1c19e9d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fae0c00ea50 tx=0x7fae0c00ed60 comp rx=0 tx=0).stop 2024-03-23T23:01:11.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.992+0000 7fae21e00700 1 -- 172.21.15.12:0/576221577 shutdown_connections 2024-03-23T23:01:11.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.992+0000 7fae21e00700 1 --2- 172.21.15.12:0/576221577 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fae000a2910 0x7fae000a4dd0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:11.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.992+0000 7fae21e00700 1 --2- 172.21.15.12:0/576221577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae1c10a3b0 0x7fae1c1a5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:11.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.992+0000 7fae21e00700 1 --2- 172.21.15.12:0/576221577 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae1c108810 0x7fae1c19e9d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:11.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.992+0000 7fae21e00700 1 --2- 172.21.15.12:0/576221577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae1c109a10 0x7fae1c19ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:11.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.992+0000 7fae21e00700 1 -- 172.21.15.12:0/576221577 >> 172.21.15.12:0/576221577 conn(0x7fae1c0777e0 msgr2=0x7fae1c10cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:11.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.992+0000 7fae21e00700 1 -- 172.21.15.12:0/576221577 shutdown_connections 2024-03-23T23:01:11.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:11.992+0000 7fae21e00700 1 -- 172.21.15.12:0/576221577 wait complete. 2024-03-23T23:01:12.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:12 smithi203 ceph-mon[36693]: pgmap v372: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-23T23:01:12.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:12 smithi203 ceph-mon[36693]: from='client.25040 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T23:01:12.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:12 smithi012 ceph-mon[31695]: pgmap v372: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-23T23:01:12.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:12 smithi012 ceph-mon[31695]: from='client.25040 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T23:01:12.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:12 smithi173 ceph-mon[36116]: pgmap v372: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-23T23:01:12.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:12 smithi173 ceph-mon[36116]: from='client.25040 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T23:01:12.899 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-23T23:01:13.385 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:01:14.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:14 smithi203 ceph-mon[36693]: pgmap v373: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:01:14.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:14 smithi012 ceph-mon[31695]: pgmap v373: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:01:14.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:14 smithi173 ceph-mon[36116]: pgmap v373: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:01:15.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.622+0000 7f599c200700 1 -- 172.21.15.12:0/1253955744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5994102c50 msgr2=0x7f5994105040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:15.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.622+0000 7f599c200700 1 --2- 172.21.15.12:0/1253955744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5994102c50 0x7f5994105040 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f598800b0a0 tx=0x7f598800b3b0 comp rx=0 tx=0).stop 2024-03-23T23:01:15.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.622+0000 7f599c200700 1 -- 172.21.15.12:0/1253955744 shutdown_connections 2024-03-23T23:01:15.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.622+0000 7f599c200700 1 --2- 172.21.15.12:0/1253955744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5994102c50 0x7f5994105040 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:15.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.622+0000 7f599c200700 1 --2- 172.21.15.12:0/1253955744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5994107eb0 0x7f599410a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:15.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.622+0000 7f599c200700 1 --2- 172.21.15.12:0/1253955744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5994105580 0x7f5994107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:15.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.622+0000 7f599c200700 1 -- 172.21.15.12:0/1253955744 >> 172.21.15.12:0/1253955744 conn(0x7f59940fc660 msgr2=0x7f59940feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:15.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.622+0000 7f599c200700 1 -- 172.21.15.12:0/1253955744 shutdown_connections 2024-03-23T23:01:15.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.622+0000 7f599c200700 1 -- 172.21.15.12:0/1253955744 wait complete. 2024-03-23T23:01:15.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.623+0000 7f599c200700 1 Processor -- start 2024-03-23T23:01:15.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.624+0000 7f599c200700 1 -- start start 2024-03-23T23:01:15.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.624+0000 7f599c200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5994102c50 0x7f599419a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:15.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.624+0000 7f599c200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5994105580 0x7f599419aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:15.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.624+0000 7f599c200700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5994107eb0 0x7f59941a1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:15.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.624+0000 7f599c200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599410de20 con 0x7f5994102c50 2024-03-23T23:01:15.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.624+0000 7f599c200700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599410dca0 con 0x7f5994107eb0 2024-03-23T23:01:15.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.624+0000 7f599c200700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599410dfa0 con 0x7f5994105580 2024-03-23T23:01:15.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.624+0000 7f5998e00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5994105580 0x7f599419aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:15.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.624+0000 7f599a200700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5994107eb0 0x7f59941a1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:15.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.624+0000 7f5998e00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5994105580 0x7f599419aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47242/0 (socket says 172.21.15.12:47242) 2024-03-23T23:01:15.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.624+0000 7f5998e00700 1 -- 172.21.15.12:0/1378742923 learned_addr learned my addr 172.21.15.12:0/1378742923 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:15.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.624+0000 7f5999800700 1 --2- 172.21.15.12:0/1378742923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5994102c50 0x7f599419a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:15.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.625+0000 7f599a200700 1 -- 172.21.15.12:0/1378742923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5994105580 msgr2=0x7f599419aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:15.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.625+0000 7f599a200700 1 --2- 172.21.15.12:0/1378742923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5994105580 0x7f599419aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:15.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.625+0000 7f599a200700 1 -- 172.21.15.12:0/1378742923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5994102c50 msgr2=0x7f599419a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:15.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.625+0000 7f599a200700 1 --2- 172.21.15.12:0/1378742923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5994102c50 0x7f599419a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:15.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.625+0000 7f599a200700 1 -- 172.21.15.12:0/1378742923 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59941a22c0 con 0x7f5994107eb0 2024-03-23T23:01:15.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.625+0000 7f5998e00700 1 --2- 172.21.15.12:0/1378742923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5994105580 0x7f599419aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:15.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.625+0000 7f599a200700 1 --2- 172.21.15.12:0/1378742923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5994107eb0 0x7f59941a1c20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f598c00e940 tx=0x7f598c00ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:15.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.626+0000 7f5992000700 1 -- 172.21.15.12:0/1378742923 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598c004220 con 0x7f5994107eb0 2024-03-23T23:01:15.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.626+0000 7f5992000700 1 -- 172.21.15.12:0/1378742923 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f598c004d10 con 0x7f5994107eb0 2024-03-23T23:01:15.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.626+0000 7f599c200700 1 -- 172.21.15.12:0/1378742923 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59941a25b0 con 0x7f5994107eb0 2024-03-23T23:01:15.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.626+0000 7f5992000700 1 -- 172.21.15.12:0/1378742923 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598c010700 con 0x7f5994107eb0 2024-03-23T23:01:15.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.626+0000 7f599c200700 1 -- 172.21.15.12:0/1378742923 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59941a2b40 con 0x7f5994107eb0 2024-03-23T23:01:15.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.627+0000 7f599c200700 1 -- 172.21.15.12:0/1378742923 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5994068e50 con 0x7f5994107eb0 2024-03-23T23:01:15.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.630+0000 7f5992000700 1 -- 172.21.15.12:0/1378742923 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f598c004830 con 0x7f5994107eb0 2024-03-23T23:01:15.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.631+0000 7f5992000700 1 --2- 172.21.15.12:0/1378742923 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f597c0a2910 0x7f597c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:15.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.631+0000 7f5992000700 1 -- 172.21.15.12:0/1378742923 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f598c0ce740 con 0x7f5994107eb0 2024-03-23T23:01:15.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.632+0000 7f5999800700 1 --2- 172.21.15.12:0/1378742923 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f597c0a2910 0x7f597c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:15.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.632+0000 7f5999800700 1 --2- 172.21.15.12:0/1378742923 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f597c0a2910 0x7f597c0a4dd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5988009bc0 tx=0x7f598800b700 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:15.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.635+0000 7f5992000700 1 -- 172.21.15.12:0/1378742923 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f598c09b790 con 0x7f5994107eb0 2024-03-23T23:01:15.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.877+0000 7f599c200700 1 -- 172.21.15.12:0/1378742923 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f59940625d0 con 0x7f597c0a2910 2024-03-23T23:01:15.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.894+0000 7f5992000700 1 -- 172.21.15.12:0/1378742923 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7f59940625d0 con 0x7f597c0a2910 2024-03-23T23:01:15.894 INFO:teuthology.orchestra.run.smithi012.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T23:01:15.894 INFO:teuthology.orchestra.run.smithi012.stdout:mds.a smithi012 running (118s) 24s ago 117s 12.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 48a7fe6e2e68 2024-03-23T23:01:15.894 INFO:teuthology.orchestra.run.smithi012.stdout:mds.b smithi173 running (115s) 46s ago 114s 25.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 7d92fbafbf17 2024-03-23T23:01:15.894 INFO:teuthology.orchestra.run.smithi012.stdout:mds.c smithi203 running (2m) 46s ago 2m 20.5M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 a9182ac7da09 2024-03-23T23:01:15.894 INFO:teuthology.orchestra.run.smithi012.stdout:mds.d smithi012 running (2m) 24s ago 2m 12.4M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 c3543e91c697 2024-03-23T23:01:15.894 INFO:teuthology.orchestra.run.smithi012.stdout:mds.e smithi173 running (2m) 46s ago 2m 19.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 8270dbda57bf 2024-03-23T23:01:15.894 INFO:teuthology.orchestra.run.smithi012.stdout:mds.f smithi203 running (2m) 46s ago 2m 15.9M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 58f37a969a27 2024-03-23T23:01:15.894 INFO:teuthology.orchestra.run.smithi012.stdout:mds.g smithi012 running (2m) 24s ago 2m 12.4M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 b3ca6f1c01d5 2024-03-23T23:01:15.894 INFO:teuthology.orchestra.run.smithi012.stdout:mds.h smithi173 running (2m) 46s ago 2m 17.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 e4b852201291 2024-03-23T23:01:15.894 INFO:teuthology.orchestra.run.smithi012.stdout:mds.i smithi203 running (2m) 46s ago 2m 17.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 92d4d344701b 2024-03-23T23:01:15.895 INFO:teuthology.orchestra.run.smithi012.stdout:mds.j smithi012 running (2m) 24s ago 2m 12.6M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 ff60b87f6b41 2024-03-23T23:01:15.895 INFO:teuthology.orchestra.run.smithi012.stdout:mds.k smithi173 running (2m) 46s ago 2m 13.7M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 05355527fbaf 2024-03-23T23:01:15.895 INFO:teuthology.orchestra.run.smithi012.stdout:mds.l smithi203 running (2m) 46s ago 2m 14.0M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 15b8198a9551 2024-03-23T23:01:15.895 INFO:teuthology.orchestra.run.smithi012.stdout:mgr.x smithi012 *:9283,8765 running (10m) 24s ago 10m 513M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 1b29f43ea2ee 2024-03-23T23:01:15.895 INFO:teuthology.orchestra.run.smithi012.stdout:mgr.y smithi173 *:8443,8765 running (8m) 46s ago 8m 451M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 8fa22e2bce06 2024-03-23T23:01:15.895 INFO:teuthology.orchestra.run.smithi012.stdout:mgr.z smithi203 *:8443,8765 running (8m) 46s ago 8m 452M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 b6f6dfb7120a 2024-03-23T23:01:15.895 INFO:teuthology.orchestra.run.smithi012.stdout:mon.a smithi012 running (10m) 24s ago 11m 48.9M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 bdb2aed276e7 2024-03-23T23:01:15.895 INFO:teuthology.orchestra.run.smithi012.stdout:mon.b smithi173 running (8m) 46s ago 8m 39.9M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 72286162409a 2024-03-23T23:01:15.895 INFO:teuthology.orchestra.run.smithi012.stdout:mon.c smithi203 running (8m) 46s ago 8m 42.4M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 f46e88dc1a11 2024-03-23T23:01:15.895 INFO:teuthology.orchestra.run.smithi012.stdout:osd.0 smithi012 running (8m) 24s ago 8m 54.0M 4273M 19.0.0-2145-g3be9a5c9 c664afe56bb0 72a7dbc0a9b3 2024-03-23T23:01:15.895 INFO:teuthology.orchestra.run.smithi012.stdout:osd.1 smithi173 running (7m) 46s ago 7m 45.1M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 8ab51716323e 2024-03-23T23:01:15.895 INFO:teuthology.orchestra.run.smithi012.stdout:osd.2 smithi203 running (7m) 46s ago 7m 55.6M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 a4ee56e761e9 2024-03-23T23:01:15.895 INFO:teuthology.orchestra.run.smithi012.stdout:osd.3 smithi012 running (6m) 24s ago 6m 42.0M 4273M 19.0.0-2145-g3be9a5c9 c664afe56bb0 2badf254fb62 2024-03-23T23:01:15.895 INFO:teuthology.orchestra.run.smithi012.stdout:osd.4 smithi173 running (6m) 46s ago 6m 54.6M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 a1232df2ca07 2024-03-23T23:01:15.895 INFO:teuthology.orchestra.run.smithi012.stdout:osd.5 smithi203 running (5m) 46s ago 5m 58.0M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 24fbc3d1ce1e 2024-03-23T23:01:15.896 INFO:teuthology.orchestra.run.smithi012.stdout:osd.6 smithi012 running (5m) 24s ago 5m 50.0M 4273M 19.0.0-2145-g3be9a5c9 c664afe56bb0 a2b25b14b545 2024-03-23T23:01:15.896 INFO:teuthology.orchestra.run.smithi012.stdout:osd.7 smithi173 running (4m) 46s ago 4m 55.0M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 81d6f1234806 2024-03-23T23:01:15.896 INFO:teuthology.orchestra.run.smithi012.stdout:osd.8 smithi203 running (4m) 46s ago 4m 53.1M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 6df573fb668b 2024-03-23T23:01:15.896 INFO:teuthology.orchestra.run.smithi012.stdout:osd.9 smithi012 running (3m) 24s ago 3m 40.7M 4273M 19.0.0-2145-g3be9a5c9 c664afe56bb0 3e97407675e6 2024-03-23T23:01:15.896 INFO:teuthology.orchestra.run.smithi012.stdout:osd.10 smithi173 running (3m) 46s ago 3m 43.2M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 8c751c7e3151 2024-03-23T23:01:15.896 INFO:teuthology.orchestra.run.smithi012.stdout:osd.11 smithi203 running (2m) 46s ago 2m 53.9M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 7f3bc901a452 2024-03-23T23:01:15.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.897+0000 7f599c200700 1 -- 172.21.15.12:0/1378742923 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f597c0a2910 msgr2=0x7f597c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:15.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.897+0000 7f599c200700 1 --2- 172.21.15.12:0/1378742923 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f597c0a2910 0x7f597c0a4dd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5988009bc0 tx=0x7f598800b700 comp rx=0 tx=0).stop 2024-03-23T23:01:15.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.897+0000 7f599c200700 1 -- 172.21.15.12:0/1378742923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5994107eb0 msgr2=0x7f59941a1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:15.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.897+0000 7f599c200700 1 --2- 172.21.15.12:0/1378742923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5994107eb0 0x7f59941a1c20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f598c00e940 tx=0x7f598c00ec50 comp rx=0 tx=0).stop 2024-03-23T23:01:15.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.897+0000 7f599c200700 1 -- 172.21.15.12:0/1378742923 shutdown_connections 2024-03-23T23:01:15.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.897+0000 7f599c200700 1 --2- 172.21.15.12:0/1378742923 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f597c0a2910 0x7f597c0a4dd0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:15.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.897+0000 7f599c200700 1 --2- 172.21.15.12:0/1378742923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5994107eb0 0x7f59941a1c20 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:15.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.897+0000 7f599c200700 1 --2- 172.21.15.12:0/1378742923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5994102c50 0x7f599419a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:15.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.897+0000 7f599c200700 1 --2- 172.21.15.12:0/1378742923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5994105580 0x7f599419aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:15.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.897+0000 7f599c200700 1 -- 172.21.15.12:0/1378742923 >> 172.21.15.12:0/1378742923 conn(0x7f59940fc660 msgr2=0x7f59941023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:15.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.897+0000 7f599c200700 1 -- 172.21.15.12:0/1378742923 shutdown_connections 2024-03-23T23:01:15.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:15.897+0000 7f599c200700 1 -- 172.21.15.12:0/1378742923 wait complete. 2024-03-23T23:01:16.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:16 smithi012 ceph-mon[31695]: pgmap v374: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-23T23:01:16.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:16 smithi012 ceph-mon[31695]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T23:01:16.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:16 smithi203 ceph-mon[36693]: pgmap v374: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-23T23:01:16.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:16 smithi203 ceph-mon[36693]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T23:01:16.708 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-23T23:01:16.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:16 smithi173 ceph-mon[36116]: pgmap v374: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-23T23:01:16.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:16 smithi173 ceph-mon[36116]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T23:01:17.151 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:01:18.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:18 smithi203 ceph-mon[36693]: pgmap v375: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-23T23:01:18.694 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:18 smithi012 ceph-mon[31695]: pgmap v375: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-23T23:01:18.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:18 smithi173 ceph-mon[36116]: pgmap v375: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-23T23:01:19.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.263+0000 7f9fa7200700 1 -- 172.21.15.12:0/3814766156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9fa0104410 msgr2=0x7f9fa0104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:19.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.263+0000 7f9fa7200700 1 --2- 172.21.15.12:0/3814766156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9fa0104410 0x7f9fa0104830 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9f9000b0d0 tx=0x7f9f9000b3e0 comp rx=0 tx=0).stop 2024-03-23T23:01:19.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.264+0000 7f9fa7200700 1 -- 172.21.15.12:0/3814766156 shutdown_connections 2024-03-23T23:01:19.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.264+0000 7f9fa7200700 1 --2- 172.21.15.12:0/3814766156 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fa0105fb0 0x7f9fa010c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:19.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.264+0000 7f9fa7200700 1 --2- 172.21.15.12:0/3814766156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9fa0104410 0x7f9fa0104830 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.264+0000 7f9fa7200700 1 --2- 172.21.15.12:0/3814766156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fa0105610 0x7f9fa0105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.264+0000 7f9fa7200700 1 -- 172.21.15.12:0/3814766156 >> 172.21.15.12:0/3814766156 conn(0x7f9fa00ff9b0 msgr2=0x7f9fa0101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.264+0000 7f9fa7200700 1 -- 172.21.15.12:0/3814766156 shutdown_connections 2024-03-23T23:01:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.264+0000 7f9fa7200700 1 -- 172.21.15.12:0/3814766156 wait complete. 2024-03-23T23:01:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.265+0000 7f9fa7200700 1 Processor -- start 2024-03-23T23:01:19.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.265+0000 7f9fa7200700 1 -- start start 2024-03-23T23:01:19.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.265+0000 7f9fa7200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fa0104410 0x7f9fa019a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:19.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.265+0000 7f9fa7200700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9fa0105610 0x7f9fa019ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:19.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.265+0000 7f9fa7200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fa0105fb0 0x7f9fa01a1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:19.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.265+0000 7f9fa7200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa010ffd0 con 0x7f9fa0105fb0 2024-03-23T23:01:19.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.265+0000 7f9fa7200700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa010fe50 con 0x7f9fa0105610 2024-03-23T23:01:19.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.266+0000 7f9fa7200700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa0110150 con 0x7f9fa0104410 2024-03-23T23:01:19.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.266+0000 7f9f9fe00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fa0104410 0x7f9fa019a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:19.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.266+0000 7f9f9fe00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fa0104410 0x7f9fa019a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47254/0 (socket says 172.21.15.12:47254) 2024-03-23T23:01:19.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.266+0000 7f9f9fe00700 1 -- 172.21.15.12:0/244493444 learned_addr learned my addr 172.21.15.12:0/244493444 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:19.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.266+0000 7f9fa5200700 1 --2- 172.21.15.12:0/244493444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fa0105fb0 0x7f9fa01a1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:19.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.266+0000 7f9f97400700 1 --2- 172.21.15.12:0/244493444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9fa0105610 0x7f9fa019ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:19.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.266+0000 7f9f9fe00700 1 -- 172.21.15.12:0/244493444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9fa0105610 msgr2=0x7f9fa019ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:19.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.266+0000 7f9f9fe00700 1 --2- 172.21.15.12:0/244493444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9fa0105610 0x7f9fa019ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:19.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.266+0000 7f9f9fe00700 1 -- 172.21.15.12:0/244493444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fa0105fb0 msgr2=0x7f9fa01a1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:19.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.266+0000 7f9f9fe00700 1 --2- 172.21.15.12:0/244493444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fa0105fb0 0x7f9fa01a1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:19.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.266+0000 7f9f9fe00700 1 -- 172.21.15.12:0/244493444 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa01a2250 con 0x7f9fa0104410 2024-03-23T23:01:19.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.266+0000 7f9fa5200700 1 --2- 172.21.15.12:0/244493444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fa0105fb0 0x7f9fa01a1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:01:19.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.266+0000 7f9f9fe00700 1 --2- 172.21.15.12:0/244493444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fa0104410 0x7f9fa019a5f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9f90007a20 tx=0x7f9f90009810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:19.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.267+0000 7f9f9d600700 1 -- 172.21.15.12:0/244493444 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f90003e10 con 0x7f9fa0104410 2024-03-23T23:01:19.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.267+0000 7f9f9d600700 1 -- 172.21.15.12:0/244493444 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f90022070 con 0x7f9fa0104410 2024-03-23T23:01:19.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.267+0000 7f9fa7200700 1 -- 172.21.15.12:0/244493444 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fa01a24e0 con 0x7f9fa0104410 2024-03-23T23:01:19.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.267+0000 7f9f9d600700 1 -- 172.21.15.12:0/244493444 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9001b7c0 con 0x7f9fa0104410 2024-03-23T23:01:19.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.267+0000 7f9fa7200700 1 -- 172.21.15.12:0/244493444 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa01a2930 con 0x7f9fa0104410 2024-03-23T23:01:19.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.268+0000 7f9fa7200700 1 -- 172.21.15.12:0/244493444 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fa0068e50 con 0x7f9fa0104410 2024-03-23T23:01:19.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.270+0000 7f9f9d600700 1 -- 172.21.15.12:0/244493444 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9f9001dae0 con 0x7f9fa0104410 2024-03-23T23:01:19.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.271+0000 7f9f9d600700 1 --2- 172.21.15.12:0/244493444 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9f840a2890 0x7f9f840a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:19.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.272+0000 7f9f9d600700 1 -- 172.21.15.12:0/244493444 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9f9000f020 con 0x7f9fa0104410 2024-03-23T23:01:19.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.272+0000 7f9f97400700 1 --2- 172.21.15.12:0/244493444 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9f840a2890 0x7f9f840a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:19.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.272+0000 7f9f97400700 1 --2- 172.21.15.12:0/244493444 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9f840a2890 0x7f9f840a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9f88009c60 tx=0x7f9f88009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:19.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.280+0000 7f9f9d600700 1 -- 172.21.15.12:0/244493444 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9f9009ca80 con 0x7f9fa0104410 2024-03-23T23:01:19.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.519+0000 7f9fa7200700 1 -- 172.21.15.12:0/244493444 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f9fa00625d0 con 0x7f9f840a2890 2024-03-23T23:01:19.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.528+0000 7f9f9d600700 1 -- 172.21.15.12:0/244493444 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7f9fa00625d0 con 0x7f9f840a2890 2024-03-23T23:01:19.528 INFO:teuthology.orchestra.run.smithi012.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-23T23:01:19.528 INFO:teuthology.orchestra.run.smithi012.stdout:mds.all 12/12 50s ago 2m smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-03-23T23:01:19.528 INFO:teuthology.orchestra.run.smithi012.stdout:mgr 3/3 50s ago 8m smithi012=x;smithi173=y;smithi203=z;count:3 2024-03-23T23:01:19.528 INFO:teuthology.orchestra.run.smithi012.stdout:mon 3/3 50s ago 9m smithi012:172.21.15.12=a;smithi173:172.21.15.173=b;smithi203:172.21.15.203=c;count:3 2024-03-23T23:01:19.528 INFO:teuthology.orchestra.run.smithi012.stdout:osd 12 50s ago - 2024-03-23T23:01:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.531+0000 7f9fa7200700 1 -- 172.21.15.12:0/244493444 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9f840a2890 msgr2=0x7f9f840a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.531+0000 7f9fa7200700 1 --2- 172.21.15.12:0/244493444 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9f840a2890 0x7f9f840a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9f88009c60 tx=0x7f9f88009260 comp rx=0 tx=0).stop 2024-03-23T23:01:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.531+0000 7f9fa7200700 1 -- 172.21.15.12:0/244493444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fa0104410 msgr2=0x7f9fa019a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.531+0000 7f9fa7200700 1 --2- 172.21.15.12:0/244493444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fa0104410 0x7f9fa019a5f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9f90007a20 tx=0x7f9f90009810 comp rx=0 tx=0).stop 2024-03-23T23:01:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.531+0000 7f9fa7200700 1 -- 172.21.15.12:0/244493444 shutdown_connections 2024-03-23T23:01:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.531+0000 7f9fa7200700 1 --2- 172.21.15.12:0/244493444 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9f840a2890 0x7f9f840a4d50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.531+0000 7f9fa7200700 1 --2- 172.21.15.12:0/244493444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fa0104410 0x7f9fa019a5f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.531+0000 7f9fa7200700 1 --2- 172.21.15.12:0/244493444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fa0105fb0 0x7f9fa01a1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.531+0000 7f9fa7200700 1 --2- 172.21.15.12:0/244493444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9fa0105610 0x7f9fa019ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.531+0000 7f9fa7200700 1 -- 172.21.15.12:0/244493444 >> 172.21.15.12:0/244493444 conn(0x7f9fa00ff9b0 msgr2=0x7f9fa0108990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.531+0000 7f9fa7200700 1 -- 172.21.15.12:0/244493444 shutdown_connections 2024-03-23T23:01:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:19.531+0000 7f9fa7200700 1 -- 172.21.15.12:0/244493444 wait complete. 2024-03-23T23:01:20.357 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-23T23:01:20.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:20 smithi012 ceph-mon[31695]: pgmap v376: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:01:20.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:20 smithi012 ceph-mon[31695]: from='client.25039 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T23:01:20.840 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:01:20.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:20 smithi173 ceph-mon[36116]: pgmap v376: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:01:20.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:20 smithi173 ceph-mon[36116]: from='client.25039 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T23:01:20.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:20 smithi203 ceph-mon[36693]: pgmap v376: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:01:20.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:20 smithi203 ceph-mon[36693]: from='client.25039 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T23:01:22.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:22 smithi012 ceph-mon[31695]: pgmap v377: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-23T23:01:22.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:22 smithi173 ceph-mon[36116]: pgmap v377: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-23T23:01:22.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:22 smithi203 ceph-mon[36693]: pgmap v377: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-23T23:01:22.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.994+0000 7fc89ce00700 1 -- 172.21.15.12:0/2226830240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc898105fb0 msgr2=0x7fc89810c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:22.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.994+0000 7fc89ce00700 1 --2- 172.21.15.12:0/2226830240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc898105fb0 0x7fc89810c860 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc890009fc0 tx=0x7fc89000b330 comp rx=0 tx=0).stop 2024-03-23T23:01:22.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.994+0000 7fc89ce00700 1 -- 172.21.15.12:0/2226830240 shutdown_connections 2024-03-23T23:01:22.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.994+0000 7fc89ce00700 1 --2- 172.21.15.12:0/2226830240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc898104410 0x7fc898104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:22.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.994+0000 7fc89ce00700 1 --2- 172.21.15.12:0/2226830240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc898105fb0 0x7fc89810c860 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:22.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.994+0000 7fc89ce00700 1 --2- 172.21.15.12:0/2226830240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc898105610 0x7fc898105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:22.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.994+0000 7fc89ce00700 1 -- 172.21.15.12:0/2226830240 >> 172.21.15.12:0/2226830240 conn(0x7fc8980ff9b0 msgr2=0x7fc898101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:22.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.995+0000 7fc89ce00700 1 -- 172.21.15.12:0/2226830240 shutdown_connections 2024-03-23T23:01:22.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.995+0000 7fc89ce00700 1 -- 172.21.15.12:0/2226830240 wait complete. 2024-03-23T23:01:22.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.996+0000 7fc89ce00700 1 Processor -- start 2024-03-23T23:01:22.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.996+0000 7fc89ce00700 1 -- start start 2024-03-23T23:01:22.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.996+0000 7fc89ce00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc898104410 0x7fc89819a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:22.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.996+0000 7fc89ce00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc898105610 0x7fc89819aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:22.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.996+0000 7fc89ce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc898105fb0 0x7fc8981a1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:22.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.996+0000 7fc89ce00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc898110050 con 0x7fc898105fb0 2024-03-23T23:01:22.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.996+0000 7fc89ce00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc89810fed0 con 0x7fc898104410 2024-03-23T23:01:22.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.996+0000 7fc89ce00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8981101d0 con 0x7fc898105610 2024-03-23T23:01:22.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.997+0000 7fc895e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc898104410 0x7fc89819a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:22.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.997+0000 7fc88fe00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc898105610 0x7fc89819aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:22.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.997+0000 7fc895e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc898104410 0x7fc89819a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:40082/0 (socket says 172.21.15.12:40082) 2024-03-23T23:01:22.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.997+0000 7fc895e00700 1 -- 172.21.15.12:0/3558776895 learned_addr learned my addr 172.21.15.12:0/3558776895 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:22.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.997+0000 7fc896800700 1 --2- 172.21.15.12:0/3558776895 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc898105fb0 0x7fc8981a1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:22.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.997+0000 7fc88fe00700 1 -- 172.21.15.12:0/3558776895 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc898104410 msgr2=0x7fc89819a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:22.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.997+0000 7fc88fe00700 1 --2- 172.21.15.12:0/3558776895 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc898104410 0x7fc89819a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:22.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.997+0000 7fc88fe00700 1 -- 172.21.15.12:0/3558776895 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc898105fb0 msgr2=0x7fc8981a1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:22.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.997+0000 7fc88fe00700 1 --2- 172.21.15.12:0/3558776895 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc898105fb0 0x7fc8981a1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:22.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.997+0000 7fc88fe00700 1 -- 172.21.15.12:0/3558776895 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8981a2210 con 0x7fc898105610 2024-03-23T23:01:22.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.997+0000 7fc88fe00700 1 --2- 172.21.15.12:0/3558776895 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc898105610 0x7fc89819aaf0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc88000ea20 tx=0x7fc88000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:22.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.997+0000 7fc88ea00700 1 -- 172.21.15.12:0/3558776895 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88000cbd0 con 0x7fc898105610 2024-03-23T23:01:22.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.998+0000 7fc89ce00700 1 -- 172.21.15.12:0/3558776895 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8981a2500 con 0x7fc898105610 2024-03-23T23:01:22.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.998+0000 7fc88ea00700 1 -- 172.21.15.12:0/3558776895 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc88000ee50 con 0x7fc898105610 2024-03-23T23:01:22.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.998+0000 7fc88ea00700 1 -- 172.21.15.12:0/3558776895 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc880018660 con 0x7fc898105610 2024-03-23T23:01:22.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.998+0000 7fc89ce00700 1 -- 172.21.15.12:0/3558776895 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8981a2a40 con 0x7fc898105610 2024-03-23T23:01:22.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:22.999+0000 7fc89ce00700 1 -- 172.21.15.12:0/3558776895 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc898068e50 con 0x7fc898105610 2024-03-23T23:01:23.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:23.000+0000 7fc88ea00700 1 -- 172.21.15.12:0/3558776895 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc880010430 con 0x7fc898105610 2024-03-23T23:01:23.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:23.002+0000 7fc88ea00700 1 --2- 172.21.15.12:0/3558776895 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc87c0a2890 0x7fc87c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:23.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:23.002+0000 7fc88ea00700 1 -- 172.21.15.12:0/3558776895 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fc8800cebe0 con 0x7fc898105610 2024-03-23T23:01:23.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:23.002+0000 7fc895e00700 1 --2- 172.21.15.12:0/3558776895 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc87c0a2890 0x7fc87c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:23.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:23.003+0000 7fc895e00700 1 --2- 172.21.15.12:0/3558776895 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc87c0a2890 0x7fc87c0a4d50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc888009d00 tx=0x7fc888009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:23.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:23.006+0000 7fc88ea00700 1 -- 172.21.15.12:0/3558776895 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc88009bc30 con 0x7fc898105610 2024-03-23T23:01:23.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:23.241+0000 7fc89ce00700 1 -- 172.21.15.12:0/3558776895 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8980625d0 con 0x7fc87c0a2890 2024-03-23T23:01:23.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:23.245+0000 7fc88ea00700 1 -- 172.21.15.12:0/3558776895 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7fc8980625d0 con 0x7fc87c0a2890 2024-03-23T23:01:23.245 INFO:teuthology.orchestra.run.smithi012.stdout:HOST ADDR LABELS STATUS 2024-03-23T23:01:23.245 INFO:teuthology.orchestra.run.smithi012.stdout:smithi012 172.21.15.12 2024-03-23T23:01:23.245 INFO:teuthology.orchestra.run.smithi012.stdout:smithi173 172.21.15.173 2024-03-23T23:01:23.245 INFO:teuthology.orchestra.run.smithi012.stdout:smithi203 172.21.15.203 2024-03-23T23:01:23.245 INFO:teuthology.orchestra.run.smithi012.stdout:3 hosts in cluster 2024-03-23T23:01:23.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:23.247+0000 7fc89ce00700 1 -- 172.21.15.12:0/3558776895 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc87c0a2890 msgr2=0x7fc87c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:23.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:23.248+0000 7fc89ce00700 1 --2- 172.21.15.12:0/3558776895 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc87c0a2890 0x7fc87c0a4d50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc888009d00 tx=0x7fc888009310 comp rx=0 tx=0).stop 2024-03-23T23:01:23.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:23.248+0000 7fc89ce00700 1 -- 172.21.15.12:0/3558776895 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc898105610 msgr2=0x7fc89819aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:23.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:23.248+0000 7fc89ce00700 1 --2- 172.21.15.12:0/3558776895 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc898105610 0x7fc89819aaf0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc88000ea20 tx=0x7fc88000ed30 comp rx=0 tx=0).stop 2024-03-23T23:01:23.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:23.248+0000 7fc89ce00700 1 -- 172.21.15.12:0/3558776895 shutdown_connections 2024-03-23T23:01:23.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:23.248+0000 7fc89ce00700 1 --2- 172.21.15.12:0/3558776895 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc87c0a2890 0x7fc87c0a4d50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:23.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:23.248+0000 7fc89ce00700 1 --2- 172.21.15.12:0/3558776895 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc898105fb0 0x7fc8981a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:23.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:23.248+0000 7fc89ce00700 1 --2- 172.21.15.12:0/3558776895 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc898104410 0x7fc89819a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:23.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:23.248+0000 7fc89ce00700 1 --2- 172.21.15.12:0/3558776895 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc898105610 0x7fc89819aaf0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:23.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:23.248+0000 7fc89ce00700 1 -- 172.21.15.12:0/3558776895 >> 172.21.15.12:0/3558776895 conn(0x7fc8980ff9b0 msgr2=0x7fc898108840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:23.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:23.248+0000 7fc89ce00700 1 -- 172.21.15.12:0/3558776895 shutdown_connections 2024-03-23T23:01:23.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:23.248+0000 7fc89ce00700 1 -- 172.21.15.12:0/3558776895 wait complete. 2024-03-23T23:01:24.040 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-23T23:01:24.514 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:01:24.667 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:24 smithi012 ceph-mon[31695]: from='client.25045 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T23:01:24.667 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:24 smithi012 ceph-mon[31695]: pgmap v378: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:01:24.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:24 smithi173 ceph-mon[36116]: from='client.25045 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T23:01:24.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:24 smithi173 ceph-mon[36116]: pgmap v378: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:01:24.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:24 smithi203 ceph-mon[36693]: from='client.25045 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T23:01:24.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:24 smithi203 ceph-mon[36693]: pgmap v378: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:01:26.447 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:26 smithi012 ceph-mon[31695]: pgmap v379: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-23T23:01:26.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.791+0000 7f368d200700 1 -- 172.21.15.12:0/4049375375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3688105f90 msgr2=0x7f368810c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:26.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.791+0000 7f368d200700 1 --2- 172.21.15.12:0/4049375375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3688105f90 0x7f368810c840 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f368000b0a0 tx=0x7f368000b3b0 comp rx=0 tx=0).stop 2024-03-23T23:01:26.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.791+0000 7f368d200700 1 -- 172.21.15.12:0/4049375375 shutdown_connections 2024-03-23T23:01:26.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.791+0000 7f368d200700 1 --2- 172.21.15.12:0/4049375375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3688105f90 0x7f368810c840 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:26.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.791+0000 7f368d200700 1 --2- 172.21.15.12:0/4049375375 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36881043f0 0x7f3688104810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:26.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.791+0000 7f368d200700 1 --2- 172.21.15.12:0/4049375375 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36881055f0 0x7f3688105a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:26.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.791+0000 7f368d200700 1 -- 172.21.15.12:0/4049375375 >> 172.21.15.12:0/4049375375 conn(0x7f36880ff9b0 msgr2=0x7f3688101dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:26.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.791+0000 7f368d200700 1 -- 172.21.15.12:0/4049375375 shutdown_connections 2024-03-23T23:01:26.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.792+0000 7f368d200700 1 -- 172.21.15.12:0/4049375375 wait complete. 2024-03-23T23:01:26.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.793+0000 7f368d200700 1 Processor -- start 2024-03-23T23:01:26.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.793+0000 7f368d200700 1 -- start start 2024-03-23T23:01:26.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.793+0000 7f368d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36881043f0 0x7f368819a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:26.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.793+0000 7f368d200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36881055f0 0x7f368819aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:26.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.793+0000 7f368d200700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3688105f90 0x7f36881a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:26.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.793+0000 7f368d200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3688110030 con 0x7f36881043f0 2024-03-23T23:01:26.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.793+0000 7f368d200700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368810feb0 con 0x7f3688105f90 2024-03-23T23:01:26.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.793+0000 7f368d200700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36881101b0 con 0x7f36881055f0 2024-03-23T23:01:26.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.793+0000 7f3685e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36881043f0 0x7f368819a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:26.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.794+0000 7f3686800700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3688105f90 0x7f36881a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:26.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.794+0000 7f3685e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36881043f0 0x7f368819a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51326/0 (socket says 172.21.15.12:51326) 2024-03-23T23:01:26.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.794+0000 7f3685e00700 1 -- 172.21.15.12:0/3548819429 learned_addr learned my addr 172.21.15.12:0/3548819429 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:26.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.794+0000 7f367d400700 1 --2- 172.21.15.12:0/3548819429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36881055f0 0x7f368819aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:26.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.794+0000 7f3686800700 1 -- 172.21.15.12:0/3548819429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36881055f0 msgr2=0x7f368819aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:26.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.794+0000 7f3686800700 1 --2- 172.21.15.12:0/3548819429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36881055f0 0x7f368819aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:26.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.794+0000 7f3686800700 1 -- 172.21.15.12:0/3548819429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36881043f0 msgr2=0x7f368819a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:26.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.794+0000 7f3686800700 1 --2- 172.21.15.12:0/3548819429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36881043f0 0x7f368819a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:26.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.794+0000 7f3686800700 1 -- 172.21.15.12:0/3548819429 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36881a21e0 con 0x7f3688105f90 2024-03-23T23:01:26.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.794+0000 7f367d400700 1 --2- 172.21.15.12:0/3548819429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36881055f0 0x7f368819aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:01:26.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.794+0000 7f3686800700 1 --2- 172.21.15.12:0/3548819429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3688105f90 0x7f36881a1b40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f368000b070 tx=0x7f3680009800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:26.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.795+0000 7f367f400700 1 -- 172.21.15.12:0/3548819429 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3680021070 con 0x7f3688105f90 2024-03-23T23:01:26.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.795+0000 7f367f400700 1 -- 172.21.15.12:0/3548819429 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3680007c40 con 0x7f3688105f90 2024-03-23T23:01:26.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.795+0000 7f368d200700 1 -- 172.21.15.12:0/3548819429 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36881a2470 con 0x7f3688105f90 2024-03-23T23:01:26.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.795+0000 7f367f400700 1 -- 172.21.15.12:0/3548819429 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3680005260 con 0x7f3688105f90 2024-03-23T23:01:26.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.795+0000 7f368d200700 1 -- 172.21.15.12:0/3548819429 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36881a2920 con 0x7f3688105f90 2024-03-23T23:01:26.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.796+0000 7f368d200700 1 -- 172.21.15.12:0/3548819429 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3688068e50 con 0x7f3688105f90 2024-03-23T23:01:26.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.798+0000 7f367f400700 1 -- 172.21.15.12:0/3548819429 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3680024070 con 0x7f3688105f90 2024-03-23T23:01:26.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.799+0000 7f367f400700 1 --2- 172.21.15.12:0/3548819429 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f366c0a2890 0x7f366c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:26.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.799+0000 7f367f400700 1 -- 172.21.15.12:0/3548819429 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f368000f020 con 0x7f3688105f90 2024-03-23T23:01:26.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.800+0000 7f3685e00700 1 --2- 172.21.15.12:0/3548819429 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f366c0a2890 0x7f366c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:26.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.800+0000 7f3685e00700 1 --2- 172.21.15.12:0/3548819429 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f366c0a2890 0x7f366c0a4d50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f367800bd70 tx=0x7f367800b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:26.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:26.808+0000 7f367f400700 1 -- 172.21.15.12:0/3548819429 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f368002b020 con 0x7f3688105f90 2024-03-23T23:01:26.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:26 smithi173 ceph-mon[36116]: pgmap v379: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-23T23:01:26.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:26 smithi203 ceph-mon[36693]: pgmap v379: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-23T23:01:27.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:27.071+0000 7f368d200700 1 -- 172.21.15.12:0/3548819429 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f36880625d0 con 0x7f366c0a2890 2024-03-23T23:01:27.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:27.074+0000 7f367f400700 1 -- 172.21.15.12:0/3548819429 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7f36880625d0 con 0x7f366c0a2890 2024-03-23T23:01:27.074 INFO:teuthology.orchestra.run.smithi012.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-23T23:01:27.074 INFO:teuthology.orchestra.run.smithi012.stdout:smithi012 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5331005J400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-23T23:01:27.075 INFO:teuthology.orchestra.run.smithi012.stdout:smithi173 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204019S400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-23T23:01:27.075 INFO:teuthology.orchestra.run.smithi012.stdout:smithi203 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GK400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-23T23:01:27.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:27.077+0000 7f368d200700 1 -- 172.21.15.12:0/3548819429 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f366c0a2890 msgr2=0x7f366c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:27.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:27.077+0000 7f368d200700 1 --2- 172.21.15.12:0/3548819429 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f366c0a2890 0x7f366c0a4d50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f367800bd70 tx=0x7f367800b4a0 comp rx=0 tx=0).stop 2024-03-23T23:01:27.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:27.077+0000 7f368d200700 1 -- 172.21.15.12:0/3548819429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3688105f90 msgr2=0x7f36881a1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:27.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:27.077+0000 7f368d200700 1 --2- 172.21.15.12:0/3548819429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3688105f90 0x7f36881a1b40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f368000b070 tx=0x7f3680009800 comp rx=0 tx=0).stop 2024-03-23T23:01:27.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:27.078+0000 7f368d200700 1 -- 172.21.15.12:0/3548819429 shutdown_connections 2024-03-23T23:01:27.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:27.078+0000 7f368d200700 1 --2- 172.21.15.12:0/3548819429 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f366c0a2890 0x7f366c0a4d50 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:27.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:27.078+0000 7f368d200700 1 --2- 172.21.15.12:0/3548819429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3688105f90 0x7f36881a1b40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:27.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:27.078+0000 7f368d200700 1 --2- 172.21.15.12:0/3548819429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36881043f0 0x7f368819a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:27.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:27.078+0000 7f368d200700 1 --2- 172.21.15.12:0/3548819429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36881055f0 0x7f368819aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:27.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:27.078+0000 7f368d200700 1 -- 172.21.15.12:0/3548819429 >> 172.21.15.12:0/3548819429 conn(0x7f36880ff9b0 msgr2=0x7f3688108820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:27.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:27.078+0000 7f368d200700 1 -- 172.21.15.12:0/3548819429 shutdown_connections 2024-03-23T23:01:27.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:27.078+0000 7f368d200700 1 -- 172.21.15.12:0/3548819429 wait complete. 2024-03-23T23:01:27.800 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:27 smithi012 ceph-mon[31695]: from='client.25064 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T23:01:27.863 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-23T23:01:27.873 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi012.front.sepia.ceph.com 2024-03-23T23:01:27.873 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-23T23:01:27.874 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:27 smithi173 ceph-mon[36116]: from='client.25064 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T23:01:27.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:27 smithi203 ceph-mon[36693]: from='client.25064 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T23:01:28.357 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:01:28.697 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:28 smithi012 ceph-mon[31695]: pgmap v380: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-23T23:01:28.769 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:28 smithi173 ceph-mon[36116]: pgmap v380: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-23T23:01:28.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:28 smithi203 ceph-mon[36693]: pgmap v380: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-23T23:01:30.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.651+0000 7f21b1e00700 1 -- 172.21.15.12:0/3168490633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21ac105580 msgr2=0x7f21ac107970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:30.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.651+0000 7f21b1e00700 1 --2- 172.21.15.12:0/3168490633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21ac105580 0x7f21ac107970 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2194009990 tx=0x7f2194009ca0 comp rx=0 tx=0).stop 2024-03-23T23:01:30.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.651+0000 7f21b1e00700 1 -- 172.21.15.12:0/3168490633 shutdown_connections 2024-03-23T23:01:30.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.651+0000 7f21b1e00700 1 --2- 172.21.15.12:0/3168490633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21ac107eb0 0x7f21ac10a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:30.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.651+0000 7f21b1e00700 1 --2- 172.21.15.12:0/3168490633 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21ac102c50 0x7f21ac105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:30.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.651+0000 7f21b1e00700 1 --2- 172.21.15.12:0/3168490633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21ac105580 0x7f21ac107970 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:30.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.651+0000 7f21b1e00700 1 -- 172.21.15.12:0/3168490633 >> 172.21.15.12:0/3168490633 conn(0x7f21ac0fc660 msgr2=0x7f21ac0feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:30.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.651+0000 7f21b1e00700 1 -- 172.21.15.12:0/3168490633 shutdown_connections 2024-03-23T23:01:30.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:30 smithi012 ceph-mon[31695]: pgmap v381: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:01:30.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.652+0000 7f21b1e00700 1 -- 172.21.15.12:0/3168490633 wait complete. 2024-03-23T23:01:30.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.653+0000 7f21b1e00700 1 Processor -- start 2024-03-23T23:01:30.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.653+0000 7f21b1e00700 1 -- start start 2024-03-23T23:01:30.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.653+0000 7f21b1e00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21ac102c50 0x7f21ac19a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:30.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.653+0000 7f21b1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21ac105580 0x7f21ac19abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:30.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.653+0000 7f21b1e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21ac107eb0 0x7f21ac1a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:30.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.653+0000 7f21b1e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac10de20 con 0x7f21ac105580 2024-03-23T23:01:30.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.653+0000 7f21b1e00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac10dca0 con 0x7f21ac107eb0 2024-03-23T23:01:30.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.653+0000 7f21b1e00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac10dfa0 con 0x7f21ac102c50 2024-03-23T23:01:30.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.654+0000 7f21ab400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21ac107eb0 0x7f21ac1a1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:30.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.654+0000 7f21ab400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21ac107eb0 0x7f21ac1a1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:34582/0 (socket says 172.21.15.12:34582) 2024-03-23T23:01:30.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.654+0000 7f21ab400700 1 -- 172.21.15.12:0/3334281718 learned_addr learned my addr 172.21.15.12:0/3334281718 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:30.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.654+0000 7f21a2000700 1 --2- 172.21.15.12:0/3334281718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21ac105580 0x7f21ac19abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:30.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.654+0000 7f21aaa00700 1 --2- 172.21.15.12:0/3334281718 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21ac102c50 0x7f21ac19a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:30.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.654+0000 7f21ab400700 1 -- 172.21.15.12:0/3334281718 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21ac102c50 msgr2=0x7f21ac19a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:30.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.654+0000 7f21ab400700 1 --2- 172.21.15.12:0/3334281718 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21ac102c50 0x7f21ac19a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:30.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.654+0000 7f21ab400700 1 -- 172.21.15.12:0/3334281718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21ac105580 msgr2=0x7f21ac19abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:30.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.654+0000 7f21ab400700 1 --2- 172.21.15.12:0/3334281718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21ac105580 0x7f21ac19abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:30.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.654+0000 7f21ab400700 1 -- 172.21.15.12:0/3334281718 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ac1a2300 con 0x7f21ac107eb0 2024-03-23T23:01:30.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.655+0000 7f21ab400700 1 --2- 172.21.15.12:0/3334281718 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21ac107eb0 0x7f21ac1a1c60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f21a400ef90 tx=0x7f21a400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:30.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.655+0000 7f21b0a00700 1 -- 172.21.15.12:0/3334281718 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a40092e0 con 0x7f21ac107eb0 2024-03-23T23:01:30.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.655+0000 7f21b0a00700 1 -- 172.21.15.12:0/3334281718 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f21a4004bb0 con 0x7f21ac107eb0 2024-03-23T23:01:30.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.655+0000 7f21b0a00700 1 -- 172.21.15.12:0/3334281718 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a4004280 con 0x7f21ac107eb0 2024-03-23T23:01:30.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.655+0000 7f21b1e00700 1 -- 172.21.15.12:0/3334281718 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21ac1a2590 con 0x7f21ac107eb0 2024-03-23T23:01:30.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.655+0000 7f21b1e00700 1 -- 172.21.15.12:0/3334281718 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21ac1a2a80 con 0x7f21ac107eb0 2024-03-23T23:01:30.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.656+0000 7f21b1e00700 1 -- 172.21.15.12:0/3334281718 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21ac068e50 con 0x7f21ac107eb0 2024-03-23T23:01:30.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.658+0000 7f21b0a00700 1 -- 172.21.15.12:0/3334281718 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f21a401b030 con 0x7f21ac107eb0 2024-03-23T23:01:30.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.659+0000 7f21b0a00700 1 --2- 172.21.15.12:0/3334281718 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f21900a2910 0x7f21900a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:30.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.659+0000 7f21b0a00700 1 -- 172.21.15.12:0/3334281718 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f21a40ce5d0 con 0x7f21ac107eb0 2024-03-23T23:01:30.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.660+0000 7f21aaa00700 1 --2- 172.21.15.12:0/3334281718 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f21900a2910 0x7f21900a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:30.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.660+0000 7f21aaa00700 1 --2- 172.21.15.12:0/3334281718 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f21900a2910 0x7f21900a4dd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f219c009bf0 tx=0x7f219c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:30.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.664+0000 7f21b0a00700 1 -- 172.21.15.12:0/3334281718 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f21a4022020 con 0x7f21ac107eb0 2024-03-23T23:01:30.708 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:30 smithi203 ceph-mon[36693]: pgmap v381: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:01:30.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:30 smithi173 ceph-mon[36116]: pgmap v381: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:01:30.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.941+0000 7f21b1e00700 1 -- 172.21.15.12:0/3334281718 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f21ac062da0 con 0x7f21ac107eb0 2024-03-23T23:01:30.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.941+0000 7f21b0a00700 1 -- 172.21.15.12:0/3334281718 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+2992 (secure 0 0 0) 0x7f21a4010040 con 0x7f21ac107eb0 2024-03-23T23:01:30.942 INFO:teuthology.orchestra.run.smithi012.stdout:e22 2024-03-23T23:01:30.942 INFO:teuthology.orchestra.run.smithi012.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T23:01:30.942 INFO:teuthology.orchestra.run.smithi012.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-23T23:01:30.942 INFO:teuthology.orchestra.run.smithi012.stdout:legacy client fscid: 1 2024-03-23T23:01:30.942 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:01:30.943 INFO:teuthology.orchestra.run.smithi012.stdout:Filesystem 'cephfs' (1) 2024-03-23T23:01:30.943 INFO:teuthology.orchestra.run.smithi012.stdout:fs_name cephfs 2024-03-23T23:01:30.943 INFO:teuthology.orchestra.run.smithi012.stdout:epoch 22 2024-03-23T23:01:30.943 INFO:teuthology.orchestra.run.smithi012.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-23T23:01:30.943 INFO:teuthology.orchestra.run.smithi012.stdout:created 2024-03-23T22:59:26.906237+0000 2024-03-23T23:01:30.943 INFO:teuthology.orchestra.run.smithi012.stdout:modified 2024-03-23T22:59:32.190756+0000 2024-03-23T23:01:30.943 INFO:teuthology.orchestra.run.smithi012.stdout:tableserver 0 2024-03-23T23:01:30.943 INFO:teuthology.orchestra.run.smithi012.stdout:root 0 2024-03-23T23:01:30.943 INFO:teuthology.orchestra.run.smithi012.stdout:session_timeout 300 2024-03-23T23:01:30.943 INFO:teuthology.orchestra.run.smithi012.stdout:session_autoclose 300 2024-03-23T23:01:30.943 INFO:teuthology.orchestra.run.smithi012.stdout:max_file_size 1099511627776 2024-03-23T23:01:30.943 INFO:teuthology.orchestra.run.smithi012.stdout:max_xattr_size 65536 2024-03-23T23:01:30.943 INFO:teuthology.orchestra.run.smithi012.stdout:required_client_features {} 2024-03-23T23:01:30.943 INFO:teuthology.orchestra.run.smithi012.stdout:last_failure 0 2024-03-23T23:01:30.943 INFO:teuthology.orchestra.run.smithi012.stdout:last_failure_osd_epoch 0 2024-03-23T23:01:30.943 INFO:teuthology.orchestra.run.smithi012.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-23T23:01:30.944 INFO:teuthology.orchestra.run.smithi012.stdout:max_mds 3 2024-03-23T23:01:30.944 INFO:teuthology.orchestra.run.smithi012.stdout:in 0,1,2 2024-03-23T23:01:30.944 INFO:teuthology.orchestra.run.smithi012.stdout:up {0=24599,1=24554,2=24550} 2024-03-23T23:01:30.944 INFO:teuthology.orchestra.run.smithi012.stdout:failed 2024-03-23T23:01:30.944 INFO:teuthology.orchestra.run.smithi012.stdout:damaged 2024-03-23T23:01:30.944 INFO:teuthology.orchestra.run.smithi012.stdout:stopped 2024-03-23T23:01:30.944 INFO:teuthology.orchestra.run.smithi012.stdout:data_pools [3] 2024-03-23T23:01:30.944 INFO:teuthology.orchestra.run.smithi012.stdout:metadata_pool 2 2024-03-23T23:01:30.944 INFO:teuthology.orchestra.run.smithi012.stdout:inline_data disabled 2024-03-23T23:01:30.944 INFO:teuthology.orchestra.run.smithi012.stdout:balancer 2024-03-23T23:01:30.944 INFO:teuthology.orchestra.run.smithi012.stdout:bal_rank_mask -1 2024-03-23T23:01:30.944 INFO:teuthology.orchestra.run.smithi012.stdout:standby_count_wanted 0 2024-03-23T23:01:30.944 INFO:teuthology.orchestra.run.smithi012.stdout:qdb_cluster leader: 24599 members: 24554,24599,24550 2024-03-23T23:01:30.944 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.b{0:24599} state up:active seq 3 addr [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] compat {c=[1],r=[1],i=[fff]}] 2024-03-23T23:01:30.944 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.c{0:24529} state up:standby-replay seq 1 addr [v2:172.21.15.203:6836/438676939,v1:172.21.15.203:6837/438676939] compat {c=[1],r=[1],i=[fff]}] 2024-03-23T23:01:30.945 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.h{1:24554} state up:active seq 7 addr [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] compat {c=[1],r=[1],i=[fff]}] 2024-03-23T23:01:30.945 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.e{1:24521} state up:standby-replay seq 1 addr [v2:172.21.15.173:6834/2648278843,v1:172.21.15.173:6835/2648278843] compat {c=[1],r=[1],i=[fff]}] 2024-03-23T23:01:30.945 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.i{2:24550} state up:active seq 7 addr [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] compat {c=[1],r=[1],i=[fff]}] 2024-03-23T23:01:30.945 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.f{2:24517} state up:standby-replay seq 1 addr [v2:172.21.15.203:6834/915277278,v1:172.21.15.203:6835/915277278] compat {c=[1],r=[1],i=[fff]}] 2024-03-23T23:01:30.945 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:01:30.945 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:01:30.945 INFO:teuthology.orchestra.run.smithi012.stdout:Standby daemons: 2024-03-23T23:01:30.945 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:01:30.945 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.j{-1:14385} state up:standby seq 1 addr [v2:172.21.15.12:6834/2201661812,v1:172.21.15.12:6835/2201661812] compat {c=[1],r=[1],i=[fff]}] 2024-03-23T23:01:30.945 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.d{-1:14403} state up:standby seq 1 addr [v2:172.21.15.12:6836/2696205127,v1:172.21.15.12:6837/2696205127] compat {c=[1],r=[1],i=[fff]}] 2024-03-23T23:01:30.945 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.g{-1:14451} state up:standby seq 1 addr [v2:172.21.15.12:6838/771072116,v1:172.21.15.12:6839/771072116] compat {c=[1],r=[1],i=[fff]}] 2024-03-23T23:01:30.945 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.a{-1:14469} state up:standby seq 1 addr [v2:172.21.15.12:6840/732195840,v1:172.21.15.12:6841/732195840] compat {c=[1],r=[1],i=[fff]}] 2024-03-23T23:01:30.945 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.l{-1:24451} state up:standby seq 1 addr [v2:172.21.15.203:6832/3254902484,v1:172.21.15.203:6833/3254902484] compat {c=[1],r=[1],i=[fff]}] 2024-03-23T23:01:30.945 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.k{-1:24473} state up:standby seq 1 addr [v2:172.21.15.173:6832/4291212010,v1:172.21.15.173:6833/4291212010] compat {c=[1],r=[1],i=[fff]}] 2024-03-23T23:01:30.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.944+0000 7f21b1e00700 1 -- 172.21.15.12:0/3334281718 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f21900a2910 msgr2=0x7f21900a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:30.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.944+0000 7f21b1e00700 1 --2- 172.21.15.12:0/3334281718 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f21900a2910 0x7f21900a4dd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f219c009bf0 tx=0x7f219c009260 comp rx=0 tx=0).stop 2024-03-23T23:01:30.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.944+0000 7f21b1e00700 1 -- 172.21.15.12:0/3334281718 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21ac107eb0 msgr2=0x7f21ac1a1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:30.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.944+0000 7f21b1e00700 1 --2- 172.21.15.12:0/3334281718 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21ac107eb0 0x7f21ac1a1c60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f21a400ef90 tx=0x7f21a400c3a0 comp rx=0 tx=0).stop 2024-03-23T23:01:30.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.944+0000 7f21b1e00700 1 -- 172.21.15.12:0/3334281718 shutdown_connections 2024-03-23T23:01:30.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.944+0000 7f21b1e00700 1 --2- 172.21.15.12:0/3334281718 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f21900a2910 0x7f21900a4dd0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:30.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.944+0000 7f21b1e00700 1 --2- 172.21.15.12:0/3334281718 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21ac102c50 0x7f21ac19a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:30.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.944+0000 7f21b1e00700 1 --2- 172.21.15.12:0/3334281718 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21ac107eb0 0x7f21ac1a1c60 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:30.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.945+0000 7f21b1e00700 1 --2- 172.21.15.12:0/3334281718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21ac105580 0x7f21ac19abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:30.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.945+0000 7f21b1e00700 1 -- 172.21.15.12:0/3334281718 >> 172.21.15.12:0/3334281718 conn(0x7f21ac0fc660 msgr2=0x7f21ac102300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:30.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.945+0000 7f21b1e00700 1 -- 172.21.15.12:0/3334281718 shutdown_connections 2024-03-23T23:01:30.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:30.945+0000 7f21b1e00700 1 -- 172.21.15.12:0/3334281718 wait complete. 2024-03-23T23:01:30.947 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 22 2024-03-23T23:01:31.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:31 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3334281718' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T23:01:31.687 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-23T23:01:31.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:31 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3334281718' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T23:01:31.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:31 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3334281718' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T23:01:32.169 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:01:32.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:32 smithi012 ceph-mon[31695]: pgmap v382: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-23T23:01:32.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:32 smithi173 ceph-mon[36116]: pgmap v382: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-23T23:01:32.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:32 smithi203 ceph-mon[36693]: pgmap v382: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-23T23:01:34.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.228+0000 7fe4c2e00700 1 -- 172.21.15.12:0/2241936482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4bc102c50 msgr2=0x7fe4bc105040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:34.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.228+0000 7fe4c2e00700 1 --2- 172.21.15.12:0/2241936482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4bc102c50 0x7fe4bc105040 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac009990 tx=0x7fe4ac009ca0 comp rx=0 tx=0).stop 2024-03-23T23:01:34.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.229+0000 7fe4c2e00700 1 -- 172.21.15.12:0/2241936482 shutdown_connections 2024-03-23T23:01:34.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.229+0000 7fe4c2e00700 1 --2- 172.21.15.12:0/2241936482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4bc102c50 0x7fe4bc105040 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:34.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.229+0000 7fe4c2e00700 1 --2- 172.21.15.12:0/2241936482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4bc107eb0 0x7fe4bc10a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:34.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.229+0000 7fe4c2e00700 1 --2- 172.21.15.12:0/2241936482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe4bc105580 0x7fe4bc107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:34.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.229+0000 7fe4c2e00700 1 -- 172.21.15.12:0/2241936482 >> 172.21.15.12:0/2241936482 conn(0x7fe4bc0fc660 msgr2=0x7fe4bc0feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:34.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.229+0000 7fe4c2e00700 1 -- 172.21.15.12:0/2241936482 shutdown_connections 2024-03-23T23:01:34.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.229+0000 7fe4c2e00700 1 -- 172.21.15.12:0/2241936482 wait complete. 2024-03-23T23:01:34.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.230+0000 7fe4c2e00700 1 Processor -- start 2024-03-23T23:01:34.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.230+0000 7fe4c2e00700 1 -- start start 2024-03-23T23:01:34.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.231+0000 7fe4c2e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4bc102c50 0x7fe4bc19a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:34.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.231+0000 7fe4c2e00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4bc105580 0x7fe4bc19ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:34.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.231+0000 7fe4c2e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe4bc107eb0 0x7fe4bc1a1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:34.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.231+0000 7fe4c2e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4bc10de20 con 0x7fe4bc102c50 2024-03-23T23:01:34.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.231+0000 7fe4c2e00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4bc10dca0 con 0x7fe4bc107eb0 2024-03-23T23:01:34.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.231+0000 7fe4c2e00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4bc10dfa0 con 0x7fe4bc105580 2024-03-23T23:01:34.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.231+0000 7fe4b3400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4bc105580 0x7fe4bc19ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:34.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.231+0000 7fe4b3400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4bc105580 0x7fe4bc19ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35274/0 (socket says 172.21.15.12:35274) 2024-03-23T23:01:34.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.231+0000 7fe4c0e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe4bc107eb0 0x7fe4bc1a1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:34.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.231+0000 7fe4b3400700 1 -- 172.21.15.12:0/3524074518 learned_addr learned my addr 172.21.15.12:0/3524074518 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:34.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.231+0000 7fe4bbe00700 1 --2- 172.21.15.12:0/3524074518 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4bc102c50 0x7fe4bc19a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:34.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.231+0000 7fe4b3400700 1 -- 172.21.15.12:0/3524074518 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe4bc107eb0 msgr2=0x7fe4bc1a1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:34.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.231+0000 7fe4b3400700 1 --2- 172.21.15.12:0/3524074518 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe4bc107eb0 0x7fe4bc1a1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:34.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.231+0000 7fe4b3400700 1 -- 172.21.15.12:0/3524074518 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4bc102c50 msgr2=0x7fe4bc19a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:34.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.231+0000 7fe4b3400700 1 --2- 172.21.15.12:0/3524074518 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4bc102c50 0x7fe4bc19a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:34.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.231+0000 7fe4b3400700 1 -- 172.21.15.12:0/3524074518 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4bc1a22a0 con 0x7fe4bc105580 2024-03-23T23:01:34.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.231+0000 7fe4bbe00700 1 --2- 172.21.15.12:0/3524074518 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4bc102c50 0x7fe4bc19a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:01:34.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.232+0000 7fe4b3400700 1 --2- 172.21.15.12:0/3524074518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4bc105580 0x7fe4bc19ab80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe4a400e9f0 tx=0x7fe4a400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:34.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.232+0000 7fe4b9600700 1 -- 172.21.15.12:0/3524074518 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a400cbd0 con 0x7fe4bc105580 2024-03-23T23:01:34.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.232+0000 7fe4b9600700 1 -- 172.21.15.12:0/3524074518 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4a400cd70 con 0x7fe4bc105580 2024-03-23T23:01:34.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.232+0000 7fe4c2e00700 1 -- 172.21.15.12:0/3524074518 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4bc1a2590 con 0x7fe4bc105580 2024-03-23T23:01:34.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.232+0000 7fe4b9600700 1 -- 172.21.15.12:0/3524074518 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a4010430 con 0x7fe4bc105580 2024-03-23T23:01:34.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.232+0000 7fe4c2e00700 1 -- 172.21.15.12:0/3524074518 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4bc1a2ad0 con 0x7fe4bc105580 2024-03-23T23:01:34.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.233+0000 7fe4c2e00700 1 -- 172.21.15.12:0/3524074518 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4bc068e50 con 0x7fe4bc105580 2024-03-23T23:01:34.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.235+0000 7fe4b9600700 1 -- 172.21.15.12:0/3524074518 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe4a4004750 con 0x7fe4bc105580 2024-03-23T23:01:34.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.236+0000 7fe4b9600700 1 --2- 172.21.15.12:0/3524074518 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe4a00a2910 0x7fe4a00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:34.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.237+0000 7fe4b9600700 1 -- 172.21.15.12:0/3524074518 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe4a40cecc0 con 0x7fe4bc105580 2024-03-23T23:01:34.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.237+0000 7fe4bbe00700 1 --2- 172.21.15.12:0/3524074518 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe4a00a2910 0x7fe4a00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:34.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.237+0000 7fe4bbe00700 1 --2- 172.21.15.12:0/3524074518 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe4a00a2910 0x7fe4a00a4dd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac00b2e0 tx=0x7fe4ac01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:34.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.240+0000 7fe4b9600700 1 -- 172.21.15.12:0/3524074518 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe4a40d5080 con 0x7fe4bc105580 2024-03-23T23:01:34.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.474+0000 7fe4c2e00700 1 -- 172.21.15.12:0/3524074518 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7fe4bc062da0 con 0x7fe4bc105580 2024-03-23T23:01:34.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.475+0000 7fe4b9600700 1 -- 172.21.15.12:0/3524074518 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v88) v1 ==== 56+0+5296 (secure 0 0 0) 0x7fe4a409bd10 con 0x7fe4bc105580 2024-03-23T23:01:34.476 INFO:teuthology.orchestra.run.smithi012.stdout:epoch 88 2024-03-23T23:01:34.477 INFO:teuthology.orchestra.run.smithi012.stdout:fsid 90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T23:01:34.477 INFO:teuthology.orchestra.run.smithi012.stdout:created 2024-03-23T22:50:14.906952+0000 2024-03-23T23:01:34.477 INFO:teuthology.orchestra.run.smithi012.stdout:modified 2024-03-23T23:00:24.349052+0000 2024-03-23T23:01:34.477 INFO:teuthology.orchestra.run.smithi012.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-23T23:01:34.477 INFO:teuthology.orchestra.run.smithi012.stdout:crush_version 26 2024-03-23T23:01:34.477 INFO:teuthology.orchestra.run.smithi012.stdout:full_ratio 0.95 2024-03-23T23:01:34.477 INFO:teuthology.orchestra.run.smithi012.stdout:backfillfull_ratio 0.9 2024-03-23T23:01:34.477 INFO:teuthology.orchestra.run.smithi012.stdout:nearfull_ratio 0.85 2024-03-23T23:01:34.477 INFO:teuthology.orchestra.run.smithi012.stdout:require_min_compat_client luminous 2024-03-23T23:01:34.477 INFO:teuthology.orchestra.run.smithi012.stdout:min_compat_client luminous 2024-03-23T23:01:34.477 INFO:teuthology.orchestra.run.smithi012.stdout:require_osd_release squid 2024-03-23T23:01:34.477 INFO:teuthology.orchestra.run.smithi012.stdout:stretch_mode_enabled false 2024-03-23T23:01:34.477 INFO:teuthology.orchestra.run.smithi012.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-23T23:01:34.478 INFO:teuthology.orchestra.run.smithi012.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-23T23:01:34.478 INFO:teuthology.orchestra.run.smithi012.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-03-23T23:01:34.478 INFO:teuthology.orchestra.run.smithi012.stdout:max_osd 12 2024-03-23T23:01:34.478 INFO:teuthology.orchestra.run.smithi012.stdout:osd.0 up in weight 1 up_from 8 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.12:6802/1190079611,v1:172.21.15.12:6803/1190079611] [v2:172.21.15.12:6804/1190079611,v1:172.21.15.12:6805/1190079611] exists,up b70875d7-a23d-4251-9e8b-1a16d4edc2a4 2024-03-23T23:01:34.478 INFO:teuthology.orchestra.run.smithi012.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6800/1983639525,v1:172.21.15.173:6801/1983639525] [v2:172.21.15.173:6802/1983639525,v1:172.21.15.173:6803/1983639525] exists,up 416e87af-db5d-41ce-84d9-731e3e07e3c1 2024-03-23T23:01:34.478 INFO:teuthology.orchestra.run.smithi012.stdout:osd.2 up in weight 1 up_from 18 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6800/2253997387,v1:172.21.15.203:6801/2253997387] [v2:172.21.15.203:6802/2253997387,v1:172.21.15.203:6803/2253997387] exists,up 7a559c70-4c54-431b-8baf-4136b6e76493 2024-03-23T23:01:34.478 INFO:teuthology.orchestra.run.smithi012.stdout:osd.3 up in weight 1 up_from 26 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.12:6810/1761446905,v1:172.21.15.12:6811/1761446905] [v2:172.21.15.12:6812/1761446905,v1:172.21.15.12:6813/1761446905] exists,up c41cd287-a674-486c-8025-328cd36e5ce5 2024-03-23T23:01:34.478 INFO:teuthology.orchestra.run.smithi012.stdout:osd.4 up in weight 1 up_from 34 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6808/242958272,v1:172.21.15.173:6809/242958272] [v2:172.21.15.173:6810/242958272,v1:172.21.15.173:6811/242958272] exists,up 3c48f886-0b42-4a09-b31e-abccc200bb1d 2024-03-23T23:01:34.478 INFO:teuthology.orchestra.run.smithi012.stdout:osd.5 up in weight 1 up_from 39 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6808/4270966092,v1:172.21.15.203:6809/4270966092] [v2:172.21.15.203:6810/4270966092,v1:172.21.15.203:6811/4270966092] exists,up 4a34f06f-8805-40c0-b1ab-23407b0276d4 2024-03-23T23:01:34.478 INFO:teuthology.orchestra.run.smithi012.stdout:osd.6 up in weight 1 up_from 47 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.12:6818/3606839295,v1:172.21.15.12:6819/3606839295] [v2:172.21.15.12:6820/3606839295,v1:172.21.15.12:6821/3606839295] exists,up 26ddc663-5623-48c9-80d1-f8b968765d07 2024-03-23T23:01:34.478 INFO:teuthology.orchestra.run.smithi012.stdout:osd.7 up in weight 1 up_from 52 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6816/1241504346,v1:172.21.15.173:6817/1241504346] [v2:172.21.15.173:6818/1241504346,v1:172.21.15.173:6819/1241504346] exists,up 17dc9417-71cd-4576-aa08-933212ec55e9 2024-03-23T23:01:34.478 INFO:teuthology.orchestra.run.smithi012.stdout:osd.8 up in weight 1 up_from 60 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6816/1516996192,v1:172.21.15.203:6817/1516996192] [v2:172.21.15.203:6818/1516996192,v1:172.21.15.203:6819/1516996192] exists,up 90a2b63a-471d-43b8-ac18-bb1f16778d18 2024-03-23T23:01:34.478 INFO:teuthology.orchestra.run.smithi012.stdout:osd.9 up in weight 1 up_from 68 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.12:6826/1175260774,v1:172.21.15.12:6827/1175260774] [v2:172.21.15.12:6828/1175260774,v1:172.21.15.12:6829/1175260774] exists,up 6e1ca504-40b4-49d0-b245-7946181abfd1 2024-03-23T23:01:34.479 INFO:teuthology.orchestra.run.smithi012.stdout:osd.10 up in weight 1 up_from 73 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6824/3303506808,v1:172.21.15.173:6825/3303506808] [v2:172.21.15.173:6826/3303506808,v1:172.21.15.173:6827/3303506808] exists,up 09a595ed-5614-4386-8e61-0170af66e01c 2024-03-23T23:01:34.479 INFO:teuthology.orchestra.run.smithi012.stdout:osd.11 up in weight 1 up_from 81 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6824/2323604506,v1:172.21.15.203:6825/2323604506] [v2:172.21.15.203:6826/2323604506,v1:172.21.15.203:6827/2323604506] exists,up 9ef1d595-a977-4ba7-9c6b-d2b3295f20b0 2024-03-23T23:01:34.479 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 2.c [2,1] 2024-03-23T23:01:34.479 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 2.1c [8,11] 2024-03-23T23:01:34.479 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 2.1d [2,4,8,11] 2024-03-23T23:01:34.479 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 3.0 [4,6] 2024-03-23T23:01:34.479 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 3.3 [4,9] 2024-03-23T23:01:34.479 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 3.10 [5,9] 2024-03-23T23:01:34.479 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 3.18 [3,9] 2024-03-23T23:01:34.479 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 3.27 [4,8] 2024-03-23T23:01:34.479 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:6801/4127975083 expires 2024-03-24T22:51:23.113201+0000 2024-03-23T23:01:34.479 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:0/4182783120 expires 2024-03-24T22:51:23.113201+0000 2024-03-23T23:01:34.479 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:6800/4127975083 expires 2024-03-24T22:51:23.113201+0000 2024-03-23T23:01:34.479 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:0/1900938637 expires 2024-03-24T22:51:23.113201+0000 2024-03-23T23:01:34.479 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:0/3414861652 expires 2024-03-24T22:50:48.401223+0000 2024-03-23T23:01:34.479 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:6801/3270387158 expires 2024-03-24T22:50:48.401223+0000 2024-03-23T23:01:34.480 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:0/2580541477 expires 2024-03-24T22:50:48.401223+0000 2024-03-23T23:01:34.480 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:6800/3270387158 expires 2024-03-24T22:50:48.401223+0000 2024-03-23T23:01:34.480 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:0/501058839 expires 2024-03-24T22:51:23.113201+0000 2024-03-23T23:01:34.480 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:0/1192554716 expires 2024-03-24T22:50:48.401223+0000 2024-03-23T23:01:34.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.478+0000 7fe4c2e00700 1 -- 172.21.15.12:0/3524074518 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe4a00a2910 msgr2=0x7fe4a00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:34.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.478+0000 7fe4c2e00700 1 --2- 172.21.15.12:0/3524074518 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe4a00a2910 0x7fe4a00a4dd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac00b2e0 tx=0x7fe4ac01a040 comp rx=0 tx=0).stop 2024-03-23T23:01:34.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.479+0000 7fe4c2e00700 1 -- 172.21.15.12:0/3524074518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4bc105580 msgr2=0x7fe4bc19ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:34.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.479+0000 7fe4c2e00700 1 --2- 172.21.15.12:0/3524074518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4bc105580 0x7fe4bc19ab80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe4a400e9f0 tx=0x7fe4a400ed00 comp rx=0 tx=0).stop 2024-03-23T23:01:34.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.479+0000 7fe4c2e00700 1 -- 172.21.15.12:0/3524074518 shutdown_connections 2024-03-23T23:01:34.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.479+0000 7fe4c2e00700 1 --2- 172.21.15.12:0/3524074518 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe4a00a2910 0x7fe4a00a4dd0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:34.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.479+0000 7fe4c2e00700 1 --2- 172.21.15.12:0/3524074518 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe4bc107eb0 0x7fe4bc1a1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:34.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.479+0000 7fe4c2e00700 1 --2- 172.21.15.12:0/3524074518 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4bc102c50 0x7fe4bc19a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:34.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.479+0000 7fe4c2e00700 1 --2- 172.21.15.12:0/3524074518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4bc105580 0x7fe4bc19ab80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:34.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.479+0000 7fe4c2e00700 1 -- 172.21.15.12:0/3524074518 >> 172.21.15.12:0/3524074518 conn(0x7fe4bc0fc660 msgr2=0x7fe4bc1023a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:34.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.479+0000 7fe4c2e00700 1 -- 172.21.15.12:0/3524074518 shutdown_connections 2024-03-23T23:01:34.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:34.479+0000 7fe4c2e00700 1 -- 172.21.15.12:0/3524074518 wait complete. 2024-03-23T23:01:34.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:34 smithi012 ceph-mon[31695]: pgmap v383: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:01:34.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:34 smithi173 ceph-mon[36116]: pgmap v383: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:01:34.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:34 smithi203 ceph-mon[36693]: pgmap v383: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:01:35.314 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-23T23:01:35.325 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T23:01:35.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:35 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3524074518' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-23T23:01:35.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.556+0000 7f7cb3200640 1 -- 172.21.15.12:0/3165826626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cac107ce0 msgr2=0x7f7cac108100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:35.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.556+0000 7f7cb3200640 1 --2- 172.21.15.12:0/3165826626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cac107ce0 0x7f7cac108100 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c009900 tx=0x7f7c9c02bc40 comp rx=0 tx=0).stop 2024-03-23T23:01:35.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.556+0000 7f7cb3200640 1 -- 172.21.15.12:0/3165826626 shutdown_connections 2024-03-23T23:01:35.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.556+0000 7f7cb3200640 1 --2- 172.21.15.12:0/3165826626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cac109890 0x7f7cac110120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:35.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.556+0000 7f7cb3200640 1 --2- 172.21.15.12:0/3165826626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cac108ed0 0x7f7cac109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:35.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.556+0000 7f7cb3200640 1 --2- 172.21.15.12:0/3165826626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cac107ce0 0x7f7cac108100 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:35.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.556+0000 7f7cb3200640 1 -- 172.21.15.12:0/3165826626 >> 172.21.15.12:0/3165826626 conn(0x7f7cac076cb0 msgr2=0x7f7cac0790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:35.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.556+0000 7f7cb3200640 1 -- 172.21.15.12:0/3165826626 shutdown_connections 2024-03-23T23:01:35.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.556+0000 7f7cb3200640 1 -- 172.21.15.12:0/3165826626 wait complete. 2024-03-23T23:01:35.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.557+0000 7f7cb3200640 1 Processor -- start 2024-03-23T23:01:35.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.558+0000 7f7cb3200640 1 -- start start 2024-03-23T23:01:35.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.558+0000 7f7cb3200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cac107ce0 0x7f7cac19e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:35.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.558+0000 7f7cb3200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cac108ed0 0x7f7cac19e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:35.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.558+0000 7f7cb3200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cac109890 0x7f7cac1a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:35.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.558+0000 7f7cb3200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac1139a0 con 0x7f7cac109890 2024-03-23T23:01:35.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.558+0000 7f7cb3200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac113820 con 0x7f7cac108ed0 2024-03-23T23:01:35.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.558+0000 7f7cb3200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac113b20 con 0x7f7cac107ce0 2024-03-23T23:01:35.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.558+0000 7f7cabe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cac107ce0 0x7f7cac19e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:35.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.558+0000 7f7cab400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cac108ed0 0x7f7cac19e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:35.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.558+0000 7f7cabe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cac107ce0 0x7f7cac19e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35288/0 (socket says 172.21.15.12:35288) 2024-03-23T23:01:35.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.558+0000 7f7cab400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cac108ed0 0x7f7cac19e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36064/0 (socket says 172.21.15.12:36064) 2024-03-23T23:01:35.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.558+0000 7f7cabe00640 1 -- 172.21.15.12:0/2775534988 learned_addr learned my addr 172.21.15.12:0/2775534988 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:35.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.558+0000 7f7cb1200640 1 --2- 172.21.15.12:0/2775534988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cac109890 0x7f7cac1a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:35.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.559+0000 7f7cab400640 1 -- 172.21.15.12:0/2775534988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cac107ce0 msgr2=0x7f7cac19e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:35.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.559+0000 7f7cab400640 1 --2- 172.21.15.12:0/2775534988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cac107ce0 0x7f7cac19e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:35.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.559+0000 7f7cab400640 1 -- 172.21.15.12:0/2775534988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cac109890 msgr2=0x7f7cac1a55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:35.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.559+0000 7f7cab400640 1 --2- 172.21.15.12:0/2775534988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cac109890 0x7f7cac1a55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:35.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.559+0000 7f7cab400640 1 -- 172.21.15.12:0/2775534988 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cac1a5c30 con 0x7f7cac108ed0 2024-03-23T23:01:35.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.559+0000 7f7cb1200640 1 --2- 172.21.15.12:0/2775534988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cac109890 0x7f7cac1a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:01:35.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.559+0000 7f7cab400640 1 --2- 172.21.15.12:0/2775534988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cac108ed0 0x7f7cac19e550 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7c9800d720 tx=0x7f7c9800dbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:35.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.559+0000 7f7ca8c00640 1 -- 172.21.15.12:0/2775534988 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c98017070 con 0x7f7cac108ed0 2024-03-23T23:01:35.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.559+0000 7f7ca8c00640 1 -- 172.21.15.12:0/2775534988 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c9800de30 con 0x7f7cac108ed0 2024-03-23T23:01:35.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.559+0000 7f7cb3200640 1 -- 172.21.15.12:0/2775534988 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cac1a5f20 con 0x7f7cac108ed0 2024-03-23T23:01:35.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.559+0000 7f7ca8c00640 1 -- 172.21.15.12:0/2775534988 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c980047a0 con 0x7f7cac108ed0 2024-03-23T23:01:35.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.560+0000 7f7cb3200640 1 -- 172.21.15.12:0/2775534988 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cac1a63e0 con 0x7f7cac108ed0 2024-03-23T23:01:35.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.561+0000 7f7cb3200640 1 -- 172.21.15.12:0/2775534988 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cac10f000 con 0x7f7cac108ed0 2024-03-23T23:01:35.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.563+0000 7f7ca8c00640 1 -- 172.21.15.12:0/2775534988 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7c980099d0 con 0x7f7cac108ed0 2024-03-23T23:01:35.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.564+0000 7f7ca8c00640 1 --2- 172.21.15.12:0/2775534988 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7c900a2880 0x7f7c900a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:35.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.564+0000 7f7ca8c00640 1 -- 172.21.15.12:0/2775534988 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7c980ce680 con 0x7f7cac108ed0 2024-03-23T23:01:35.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.565+0000 7f7cabe00640 1 --2- 172.21.15.12:0/2775534988 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7c900a2880 0x7f7c900a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:35.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.565+0000 7f7cabe00640 1 --2- 172.21.15.12:0/2775534988 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7c900a2880 0x7f7c900a4d40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c005ab0 tx=0x7f7c9c02cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:35.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.573+0000 7f7ca8c00640 1 -- 172.21.15.12:0/2775534988 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7c98020020 con 0x7f7cac108ed0 2024-03-23T23:01:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.781+0000 7f7cb3200640 1 -- 172.21.15.12:0/2775534988 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7cac0622a0 con 0x7f7cac108ed0 2024-03-23T23:01:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.783+0000 7f7ca8c00640 1 -- 172.21.15.12:0/2775534988 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22695 (secure 0 0 0) 0x7f7c9809b6d0 con 0x7f7cac108ed0 2024-03-23T23:01:35.784 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:01:35.784 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":88,"fsid":"90f679ae-e967-11ee-95cd-87774f69a715","created":"2024-03-23T22:50:14.906952+0000","modified":"2024-03-23T23:00:24.349052+0000","last_up_change":"2024-03-23T22:58:28.907824+0000","last_in_change":"2024-03-23T22:58:08.342487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T22:54:11.831010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T22:59:25.350974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T22:59:26.390323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b70875d7-a23d-4251-9e8b-1a16d4edc2a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6803","nonce":1190079611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6805","nonce":1190079611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6809","nonce":1190079611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6807","nonce":1190079611}]},"public_addr":"172.21.15.12:6803/1190079611","cluster_addr":"172.21.15.12:6805/1190079611","heartbeat_back_addr":"172.21.15.12:6809/1190079611","heartbeat_front_addr":"172.21.15.12:6807/1190079611","state":["exists","up"]},{"osd":1,"uuid":"416e87af-db5d-41ce-84d9-731e3e07e3c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6801","nonce":1983639525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6803","nonce":1983639525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6807","nonce":1983639525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6805","nonce":1983639525}]},"public_addr":"172.21.15.173:6801/1983639525","cluster_addr":"172.21.15.173:6803/1983639525","heartbeat_back_addr":"172.21.15.173:6807/1983639525","heartbeat_front_addr":"172.21.15.173:6805/1983639525","state":["exists","up"]},{"osd":2,"uuid":"7a559c70-4c54-431b-8baf-4136b6e76493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6801","nonce":2253997387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6803","nonce":2253997387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6807","nonce":2253997387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6805","nonce":2253997387}]},"public_addr":"172.21.15.203:6801/2253997387","cluster_addr":"172.21.15.203:6803/2253997387","heartbeat_back_addr":"172.21.15.203:6807/2253997387","heartbeat_front_addr":"172.21.15.203:6805/2253997387","state":["exists","up"]},{"osd":3,"uuid":"c41cd287-a674-486c-8025-328cd36e5ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6811","nonce":1761446905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6813","nonce":1761446905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6817","nonce":1761446905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6815","nonce":1761446905}]},"public_addr":"172.21.15.12:6811/1761446905","cluster_addr":"172.21.15.12:6813/1761446905","heartbeat_back_addr":"172.21.15.12:6817/1761446905","heartbeat_front_addr":"172.21.15.12:6815/1761446905","state":["exists","up"]},{"osd":4,"uuid":"3c48f886-0b42-4a09-b31e-abccc200bb1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6809","nonce":242958272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6811","nonce":242958272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6815","nonce":242958272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6813","nonce":242958272}]},"public_addr":"172.21.15.173:6809/242958272","cluster_addr":"172.21.15.173:6811/242958272","heartbeat_back_addr":"172.21.15.173:6815/242958272","heartbeat_front_addr":"172.21.15.173:6813/242958272","state":["exists","up"]},{"osd":5,"uuid":"4a34f06f-8805-40c0-b1ab-23407b0276d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6809","nonce":4270966092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6811","nonce":4270966092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6815","nonce":4270966092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6813","nonce":4270966092}]},"public_addr":"172.21.15.203:6809/4270966092","cluster_addr":"172.21.15.203:6811/4270966092","heartbeat_back_addr":"172.21.15.203:6815/4270966092","heartbeat_front_addr":"172.21.15.203:6813/4270966092","state":["exists","up"]},{"osd":6,"uuid":"26ddc663-5623-48c9-80d1-f8b968765d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6819","nonce":3606839295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6821","nonce":3606839295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6825","nonce":3606839295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6823","nonce":3606839295}]},"public_addr":"172.21.15.12:6819/3606839295","cluster_addr":"172.21.15.12:6821/3606839295","heartbeat_back_addr":"172.21.15.12:6825/3606839295","heartbeat_front_addr":"172.21.15.12:6823/3606839295","state":["exists","up"]},{"osd":7,"uuid":"17dc9417-71cd-4576-aa08-933212ec55e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6817","nonce":1241504346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6819","nonce":1241504346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6823","nonce":1241504346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6821","nonce":1241504346}]},"public_addr":"172.21.15.173:6817/1241504346","cluster_addr":"172.21.15.173:6819/1241504346","heartbeat_back_addr":"172.21.15.173:6823/1241504346","heartbeat_front_addr":"172.21.15.173:6821/1241504346","state":["exists","up"]},{"osd":8,"uuid":"90a2b63a-471d-43b8-ac18-bb1f16778d18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6817","nonce":1516996192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6819","nonce":1516996192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6823","nonce":1516996192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6821","nonce":1516996192}]},"public_addr":"172.21.15.203:6817/1516996192","cluster_addr":"172.21.15.203:6819/1516996192","heartbeat_back_addr":"172.21.15.203:6823/1516996192","heartbeat_front_addr":"172.21.15.203:6821/1516996192","state":["exists","up"]},{"osd":9,"uuid":"6e1ca504-40b4-49d0-b245-7946181abfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6827","nonce":1175260774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6829","nonce":1175260774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6833","nonce":1175260774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6831","nonce":1175260774}]},"public_addr":"172.21.15.12:6827/1175260774","cluster_addr":"172.21.15.12:6829/1175260774","heartbeat_back_addr":"172.21.15.12:6833/1175260774","heartbeat_front_addr":"172.21.15.12:6831/1175260774","state":["exists","up"]},{"osd":10,"uuid":"09a595ed-5614-4386-8e61-0170af66e01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6825","nonce":3303506808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6827","nonce":3303506808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6831","nonce":3303506808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6829","nonce":3303506808}]},"public_addr":"172.21.15.173:6825/3303506808","cluster_addr":"172.21.15.173:6827/3303506808","heartbeat_back_addr":"172.21.15.173:6831/3303506808","heartbeat_front_addr":"172.21.15.173:6829/3303506808","state":["exists","up"]},{"osd":11,"uuid":"9ef1d595-a977-4ba7-9c6b-d2b3295f20b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6825","nonce":2323604506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6827","nonce":2323604506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6831","nonce":2323604506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6829","nonce":2323604506}]},"public_addr":"172.21.15.203:6825/2323604506","cluster_addr":"172.21.15.203:6827/2323604506","heartbeat_back_addr":"172.21.15.203:6831/2323604506","heartbeat_front_addr":"172.21.15.203:6829/2323604506","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:18.469038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:42.752628+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:07.069493+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:41.962478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:08.363104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:33.341336+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:07.286541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:34.243302+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:59.749699+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:57:33.822913+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:00.737269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:26.599854+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/4182783120":"2024-03-24T22:51:23.113201+0000","172.21.15.12:6800/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1900938637":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/3414861652":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6801/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/2580541477":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6800/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/501058839":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1192554716":"2024-03-24T22:50:48.401223+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T23:01:35.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.786+0000 7f7cb3200640 1 -- 172.21.15.12:0/2775534988 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7c900a2880 msgr2=0x7f7c900a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:35.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.786+0000 7f7cb3200640 1 --2- 172.21.15.12:0/2775534988 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7c900a2880 0x7f7c900a4d40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c005ab0 tx=0x7f7c9c02cf90 comp rx=0 tx=0).stop 2024-03-23T23:01:35.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.786+0000 7f7cb3200640 1 -- 172.21.15.12:0/2775534988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cac108ed0 msgr2=0x7f7cac19e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:35.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.786+0000 7f7cb3200640 1 --2- 172.21.15.12:0/2775534988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cac108ed0 0x7f7cac19e550 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7c9800d720 tx=0x7f7c9800dbe0 comp rx=0 tx=0).stop 2024-03-23T23:01:35.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.786+0000 7f7cb3200640 1 -- 172.21.15.12:0/2775534988 shutdown_connections 2024-03-23T23:01:35.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.786+0000 7f7cb3200640 1 --2- 172.21.15.12:0/2775534988 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7c900a2880 0x7f7c900a4d40 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:35.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.786+0000 7f7cb3200640 1 --2- 172.21.15.12:0/2775534988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cac109890 0x7f7cac1a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:35.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.786+0000 7f7cb3200640 1 --2- 172.21.15.12:0/2775534988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cac108ed0 0x7f7cac19e550 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:35.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.786+0000 7f7cb3200640 1 --2- 172.21.15.12:0/2775534988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cac107ce0 0x7f7cac19e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:35.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.786+0000 7f7cb3200640 1 -- 172.21.15.12:0/2775534988 >> 172.21.15.12:0/2775534988 conn(0x7f7cac076cb0 msgr2=0x7f7cac10c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:35.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.786+0000 7f7cb3200640 1 -- 172.21.15.12:0/2775534988 shutdown_connections 2024-03-23T23:01:35.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:35.786+0000 7f7cb3200640 1 -- 172.21.15.12:0/2775534988 wait complete. 2024-03-23T23:01:35.808 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T22:54:11.831010+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T22:59:25.350974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T22:59:26.390323+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T23:01:35.809 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T23:01:35.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:35 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3524074518' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-23T23:01:35.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:35 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3524074518' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-23T23:01:36.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.036+0000 7f770f200640 1 -- 172.21.15.12:0/258043466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77101015a0 msgr2=0x7f77101019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:36.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.036+0000 7f770f200640 1 --2- 172.21.15.12:0/258043466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77101015a0 0x7f77101019c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7700009900 tx=0x7f770002bc30 comp rx=0 tx=0).stop 2024-03-23T23:01:36.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.036+0000 7f770f200640 1 -- 172.21.15.12:0/258043466 shutdown_connections 2024-03-23T23:01:36.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.036+0000 7f770f200640 1 --2- 172.21.15.12:0/258043466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7710103150 0x7f77101099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.036+0000 7f770f200640 1 --2- 172.21.15.12:0/258043466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7710102790 0x7f7710102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.036+0000 7f770f200640 1 --2- 172.21.15.12:0/258043466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77101015a0 0x7f77101019c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.036+0000 7f770f200640 1 -- 172.21.15.12:0/258043466 >> 172.21.15.12:0/258043466 conn(0x7f77100fcb10 msgr2=0x7f77100fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:36.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.036+0000 7f770f200640 1 -- 172.21.15.12:0/258043466 shutdown_connections 2024-03-23T23:01:36.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.036+0000 7f770f200640 1 -- 172.21.15.12:0/258043466 wait complete. 2024-03-23T23:01:36.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.037+0000 7f770f200640 1 Processor -- start 2024-03-23T23:01:36.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.038+0000 7f770f200640 1 -- start start 2024-03-23T23:01:36.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.038+0000 7f770f200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77101015a0 0x7f7710075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:36.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.038+0000 7f770f200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7710102790 0x7f7710073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:36.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.038+0000 7f770f200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7710103150 0x7f7710073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:36.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.038+0000 7f770f200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771010d240 con 0x7f7710103150 2024-03-23T23:01:36.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.038+0000 7f770f200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771010d0c0 con 0x7f77101015a0 2024-03-23T23:01:36.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.038+0000 7f770f200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771010d3c0 con 0x7f7710102790 2024-03-23T23:01:36.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.039+0000 7f770d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7710102790 0x7f7710073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:36.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.039+0000 7f770de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77101015a0 0x7f7710075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:36.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.039+0000 7f770d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7710102790 0x7f7710073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35294/0 (socket says 172.21.15.12:35294) 2024-03-23T23:01:36.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.039+0000 7f770d400640 1 -- 172.21.15.12:0/2604996005 learned_addr learned my addr 172.21.15.12:0/2604996005 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:36.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.039+0000 7f770e800640 1 --2- 172.21.15.12:0/2604996005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7710103150 0x7f7710073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:36.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.039+0000 7f770d400640 1 -- 172.21.15.12:0/2604996005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77101015a0 msgr2=0x7f7710075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:36.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.039+0000 7f770d400640 1 --2- 172.21.15.12:0/2604996005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77101015a0 0x7f7710075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.039+0000 7f770d400640 1 -- 172.21.15.12:0/2604996005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7710103150 msgr2=0x7f7710073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:36.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.039+0000 7f770d400640 1 --2- 172.21.15.12:0/2604996005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7710103150 0x7f7710073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.039+0000 7f770d400640 1 -- 172.21.15.12:0/2604996005 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7710074860 con 0x7f7710102790 2024-03-23T23:01:36.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.039+0000 7f770e800640 1 --2- 172.21.15.12:0/2604996005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7710103150 0x7f7710073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:01:36.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.039+0000 7f770d400640 1 --2- 172.21.15.12:0/2604996005 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7710102790 0x7f7710073a90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f76fc00e9f0 tx=0x7f76fc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:36.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.040+0000 7f7706a00640 1 -- 172.21.15.12:0/2604996005 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76fc00cd80 con 0x7f7710102790 2024-03-23T23:01:36.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.040+0000 7f7706a00640 1 -- 172.21.15.12:0/2604996005 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76fc010910 con 0x7f7710102790 2024-03-23T23:01:36.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.040+0000 7f7706a00640 1 -- 172.21.15.12:0/2604996005 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76fc018990 con 0x7f7710102790 2024-03-23T23:01:36.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.040+0000 7f770f200640 1 -- 172.21.15.12:0/2604996005 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77101a1a00 con 0x7f7710102790 2024-03-23T23:01:36.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.040+0000 7f770f200640 1 -- 172.21.15.12:0/2604996005 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77101a1f00 con 0x7f7710102790 2024-03-23T23:01:36.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.041+0000 7f770f200640 1 -- 172.21.15.12:0/2604996005 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77101088e0 con 0x7f7710102790 2024-03-23T23:01:36.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.043+0000 7f7706a00640 1 -- 172.21.15.12:0/2604996005 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f76fc010ac0 con 0x7f7710102790 2024-03-23T23:01:36.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.045+0000 7f7706a00640 1 --2- 172.21.15.12:0/2604996005 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f76f40a2880 0x7f76f40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:36.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.045+0000 7f7706a00640 1 -- 172.21.15.12:0/2604996005 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f76fc0cee20 con 0x7f7710102790 2024-03-23T23:01:36.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.045+0000 7f770de00640 1 --2- 172.21.15.12:0/2604996005 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f76f40a2880 0x7f76f40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:36.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.046+0000 7f770de00640 1 --2- 172.21.15.12:0/2604996005 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f76f40a2880 0x7f76f40a4d40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f770002c170 tx=0x7f770002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:36.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.049+0000 7f7706a00640 1 -- 172.21.15.12:0/2604996005 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f76fc09be70 con 0x7f7710102790 2024-03-23T23:01:36.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.251+0000 7f770f200640 1 -- 172.21.15.12:0/2604996005 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f77100622a0 con 0x7f7710102790 2024-03-23T23:01:36.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.252+0000 7f7706a00640 1 -- 172.21.15.12:0/2604996005 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f76fc09b770 con 0x7f7710102790 2024-03-23T23:01:36.252 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-23T23:01:36.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.254+0000 7f770f200640 1 -- 172.21.15.12:0/2604996005 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f76f40a2880 msgr2=0x7f76f40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:36.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.254+0000 7f770f200640 1 --2- 172.21.15.12:0/2604996005 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f76f40a2880 0x7f76f40a4d40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f770002c170 tx=0x7f770002df90 comp rx=0 tx=0).stop 2024-03-23T23:01:36.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.254+0000 7f770f200640 1 -- 172.21.15.12:0/2604996005 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7710102790 msgr2=0x7f7710073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:36.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.254+0000 7f770f200640 1 --2- 172.21.15.12:0/2604996005 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7710102790 0x7f7710073a90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f76fc00e9f0 tx=0x7f76fc00eeb0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.255+0000 7f770f200640 1 -- 172.21.15.12:0/2604996005 shutdown_connections 2024-03-23T23:01:36.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.255+0000 7f770f200640 1 --2- 172.21.15.12:0/2604996005 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f76f40a2880 0x7f76f40a4d40 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.255+0000 7f770f200640 1 --2- 172.21.15.12:0/2604996005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7710103150 0x7f7710073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.255+0000 7f770f200640 1 --2- 172.21.15.12:0/2604996005 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7710102790 0x7f7710073a90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.255+0000 7f770f200640 1 --2- 172.21.15.12:0/2604996005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77101015a0 0x7f7710075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.255+0000 7f770f200640 1 -- 172.21.15.12:0/2604996005 >> 172.21.15.12:0/2604996005 conn(0x7f77100fcb10 msgr2=0x7f7710105ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:36.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.255+0000 7f770f200640 1 -- 172.21.15.12:0/2604996005 shutdown_connections 2024-03-23T23:01:36.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.255+0000 7f770f200640 1 -- 172.21.15.12:0/2604996005 wait complete. 2024-03-23T23:01:36.275 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T23:01:36.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.507+0000 7fd7d3400640 1 -- 172.21.15.12:0/2110619398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7cc109870 msgr2=0x7fd7cc110100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:36.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.507+0000 7fd7d3400640 1 --2- 172.21.15.12:0/2110619398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7cc109870 0x7fd7cc110100 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd7c4009be0 tx=0x7fd7c402bef0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.507+0000 7fd7d3400640 1 -- 172.21.15.12:0/2110619398 shutdown_connections 2024-03-23T23:01:36.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.507+0000 7fd7d3400640 1 --2- 172.21.15.12:0/2110619398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7cc109870 0x7fd7cc110100 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.507+0000 7fd7d3400640 1 --2- 172.21.15.12:0/2110619398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7cc108eb0 0x7fd7cc109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.507+0000 7fd7d3400640 1 --2- 172.21.15.12:0/2110619398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7cc107cc0 0x7fd7cc1080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.507+0000 7fd7d3400640 1 -- 172.21.15.12:0/2110619398 >> 172.21.15.12:0/2110619398 conn(0x7fd7cc076c60 msgr2=0x7fd7cc0790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:36.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.507+0000 7fd7d3400640 1 -- 172.21.15.12:0/2110619398 shutdown_connections 2024-03-23T23:01:36.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.507+0000 7fd7d3400640 1 -- 172.21.15.12:0/2110619398 wait complete. 2024-03-23T23:01:36.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.508+0000 7fd7d3400640 1 Processor -- start 2024-03-23T23:01:36.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.508+0000 7fd7d3400640 1 -- start start 2024-03-23T23:01:36.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.509+0000 7fd7d3400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7cc107cc0 0x7fd7cc19df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:36.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.509+0000 7fd7d3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7cc108eb0 0x7fd7cc19e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:36.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.509+0000 7fd7d3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7cc109870 0x7fd7cc1a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:36.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.509+0000 7fd7d3400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc113970 con 0x7fd7cc107cc0 2024-03-23T23:01:36.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.509+0000 7fd7d3400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc1137f0 con 0x7fd7cc109870 2024-03-23T23:01:36.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.509+0000 7fd7d3400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc113af0 con 0x7fd7cc108eb0 2024-03-23T23:01:36.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.509+0000 7fd7cbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7cc108eb0 0x7fd7cc19e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:36.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.509+0000 7fd7d1400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7cc109870 0x7fd7cc1a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:36.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.509+0000 7fd7cbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7cc108eb0 0x7fd7cc19e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35310/0 (socket says 172.21.15.12:35310) 2024-03-23T23:01:36.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.509+0000 7fd7cbe00640 1 -- 172.21.15.12:0/1794139274 learned_addr learned my addr 172.21.15.12:0/1794139274 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:36.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.509+0000 7fd7d0a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7cc107cc0 0x7fd7cc19df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:36.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.509+0000 7fd7cbe00640 1 -- 172.21.15.12:0/1794139274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7cc109870 msgr2=0x7fd7cc1a5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:36.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.509+0000 7fd7cbe00640 1 --2- 172.21.15.12:0/1794139274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7cc109870 0x7fd7cc1a5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.509+0000 7fd7cbe00640 1 -- 172.21.15.12:0/1794139274 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7cc107cc0 msgr2=0x7fd7cc19df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:36.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.510+0000 7fd7cbe00640 1 --2- 172.21.15.12:0/1794139274 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7cc107cc0 0x7fd7cc19df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.510+0000 7fd7cbe00640 1 -- 172.21.15.12:0/1794139274 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7cc1a5b70 con 0x7fd7cc108eb0 2024-03-23T23:01:36.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.510+0000 7fd7cbe00640 1 --2- 172.21.15.12:0/1794139274 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7cc108eb0 0x7fd7cc19e490 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd7b800e9f0 tx=0x7fd7b800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:36.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.510+0000 7fd7c9600640 1 -- 172.21.15.12:0/1794139274 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b800cd80 con 0x7fd7cc108eb0 2024-03-23T23:01:36.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.510+0000 7fd7c9600640 1 -- 172.21.15.12:0/1794139274 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd7b8010910 con 0x7fd7cc108eb0 2024-03-23T23:01:36.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.510+0000 7fd7c9600640 1 -- 172.21.15.12:0/1794139274 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8018990 con 0x7fd7cc108eb0 2024-03-23T23:01:36.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.510+0000 7fd7d3400640 1 -- 172.21.15.12:0/1794139274 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7cc1a5e60 con 0x7fd7cc108eb0 2024-03-23T23:01:36.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.510+0000 7fd7d3400640 1 -- 172.21.15.12:0/1794139274 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7cc1a62d0 con 0x7fd7cc108eb0 2024-03-23T23:01:36.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.512+0000 7fd7d3400640 1 -- 172.21.15.12:0/1794139274 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7cc10ef80 con 0x7fd7cc108eb0 2024-03-23T23:01:36.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.514+0000 7fd7c9600640 1 -- 172.21.15.12:0/1794139274 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd7b8010ac0 con 0x7fd7cc108eb0 2024-03-23T23:01:36.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.515+0000 7fd7c9600640 1 --2- 172.21.15.12:0/1794139274 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd7b40a2880 0x7fd7b40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:36.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.515+0000 7fd7c9600640 1 -- 172.21.15.12:0/1794139274 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd7b80cecd0 con 0x7fd7cc108eb0 2024-03-23T23:01:36.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.515+0000 7fd7d0a00640 1 --2- 172.21.15.12:0/1794139274 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd7b40a2880 0x7fd7b40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:36.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.516+0000 7fd7d0a00640 1 --2- 172.21.15.12:0/1794139274 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd7b40a2880 0x7fd7b40a4d40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0009530 tx=0x7fd7c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:36.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.519+0000 7fd7c9600640 1 -- 172.21.15.12:0/1794139274 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd7b809b730 con 0x7fd7cc108eb0 2024-03-23T23:01:36.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:36 smithi012 ceph-mon[31695]: pgmap v384: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-23T23:01:36.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:36 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2775534988' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:36.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:36 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2604996005' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:01:36.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.721+0000 7fd7d3400640 1 -- 172.21.15.12:0/1794139274 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd7cc02d070 con 0x7fd7cc108eb0 2024-03-23T23:01:36.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.722+0000 7fd7c9600640 1 -- 172.21.15.12:0/1794139274 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd7b809f7d0 con 0x7fd7cc108eb0 2024-03-23T23:01:36.722 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-23T23:01:36.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.725+0000 7fd7d3400640 1 -- 172.21.15.12:0/1794139274 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd7b40a2880 msgr2=0x7fd7b40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:36.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.725+0000 7fd7d3400640 1 --2- 172.21.15.12:0/1794139274 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd7b40a2880 0x7fd7b40a4d40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0009530 tx=0x7fd7c0009260 comp rx=0 tx=0).stop 2024-03-23T23:01:36.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.725+0000 7fd7d3400640 1 -- 172.21.15.12:0/1794139274 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7cc108eb0 msgr2=0x7fd7cc19e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:36.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.725+0000 7fd7d3400640 1 --2- 172.21.15.12:0/1794139274 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7cc108eb0 0x7fd7cc19e490 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd7b800e9f0 tx=0x7fd7b800eeb0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.725+0000 7fd7d3400640 1 -- 172.21.15.12:0/1794139274 shutdown_connections 2024-03-23T23:01:36.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.725+0000 7fd7d3400640 1 --2- 172.21.15.12:0/1794139274 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd7b40a2880 0x7fd7b40a4d40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.725+0000 7fd7d3400640 1 --2- 172.21.15.12:0/1794139274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7cc109870 0x7fd7cc1a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.725+0000 7fd7d3400640 1 --2- 172.21.15.12:0/1794139274 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7cc108eb0 0x7fd7cc19e490 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.725+0000 7fd7d3400640 1 --2- 172.21.15.12:0/1794139274 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7cc107cc0 0x7fd7cc19df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.725+0000 7fd7d3400640 1 -- 172.21.15.12:0/1794139274 >> 172.21.15.12:0/1794139274 conn(0x7fd7cc076c60 msgr2=0x7fd7cc10c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:36.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.725+0000 7fd7d3400640 1 -- 172.21.15.12:0/1794139274 shutdown_connections 2024-03-23T23:01:36.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.725+0000 7fd7d3400640 1 -- 172.21.15.12:0/1794139274 wait complete. 2024-03-23T23:01:36.744 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T23:01:36.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:36 smithi173 ceph-mon[36116]: pgmap v384: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-23T23:01:36.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:36 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2775534988' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:36.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:36 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2604996005' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:01:36.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:36 smithi203 ceph-mon[36693]: pgmap v384: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-23T23:01:36.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:36 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2775534988' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:36.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:36 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2604996005' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:01:36.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.972+0000 7f630d400640 1 -- 172.21.15.12:0/806409868 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63081037c0 msgr2=0x7f6308103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:36.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.972+0000 7f630d400640 1 --2- 172.21.15.12:0/806409868 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63081037c0 0x7f6308103be0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f62f8009900 tx=0x7f62f802bc30 comp rx=0 tx=0).stop 2024-03-23T23:01:36.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.972+0000 7f630d400640 1 -- 172.21.15.12:0/806409868 shutdown_connections 2024-03-23T23:01:36.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.972+0000 7f630d400640 1 --2- 172.21.15.12:0/806409868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6308105370 0x7f630810bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.973+0000 7f630d400640 1 --2- 172.21.15.12:0/806409868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63081049b0 0x7f6308104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.973+0000 7f630d400640 1 --2- 172.21.15.12:0/806409868 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63081037c0 0x7f6308103be0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.973+0000 7f630d400640 1 -- 172.21.15.12:0/806409868 >> 172.21.15.12:0/806409868 conn(0x7f63080fed50 msgr2=0x7f6308101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:36.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.973+0000 7f630d400640 1 -- 172.21.15.12:0/806409868 shutdown_connections 2024-03-23T23:01:36.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.973+0000 7f630d400640 1 -- 172.21.15.12:0/806409868 wait complete. 2024-03-23T23:01:36.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.974+0000 7f630d400640 1 Processor -- start 2024-03-23T23:01:36.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.974+0000 7f630d400640 1 -- start start 2024-03-23T23:01:36.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.974+0000 7f630d400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63081037c0 0x7f6308199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:36.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.974+0000 7f630d400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63081049b0 0x7f6308199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:36.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.974+0000 7f630d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6308105370 0x7f63081a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:36.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.974+0000 7f630d400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630810f490 con 0x7f63081049b0 2024-03-23T23:01:36.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.974+0000 7f630d400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630810f310 con 0x7f63081037c0 2024-03-23T23:01:36.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.974+0000 7f630d400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630810f610 con 0x7f6308105370 2024-03-23T23:01:36.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.974+0000 7f6307e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63081037c0 0x7f6308199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:36.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.974+0000 7f6307e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63081037c0 0x7f6308199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36142/0 (socket says 172.21.15.12:36142) 2024-03-23T23:01:36.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.974+0000 7f630ca00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6308105370 0x7f63081a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:36.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.974+0000 7f6307e00640 1 -- 172.21.15.12:0/1712219604 learned_addr learned my addr 172.21.15.12:0/1712219604 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:36.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.974+0000 7f62ffe00640 1 --2- 172.21.15.12:0/1712219604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63081049b0 0x7f6308199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:36.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.975+0000 7f6307e00640 1 -- 172.21.15.12:0/1712219604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6308105370 msgr2=0x7f63081a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:36.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.975+0000 7f6307e00640 1 --2- 172.21.15.12:0/1712219604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6308105370 0x7f63081a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.975+0000 7f6307e00640 1 -- 172.21.15.12:0/1712219604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63081049b0 msgr2=0x7f6308199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:36.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.975+0000 7f6307e00640 1 --2- 172.21.15.12:0/1712219604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63081049b0 0x7f6308199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:36.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.975+0000 7f6307e00640 1 -- 172.21.15.12:0/1712219604 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63081a16d0 con 0x7f63081037c0 2024-03-23T23:01:36.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.975+0000 7f630ca00640 1 --2- 172.21.15.12:0/1712219604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6308105370 0x7f63081a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:36.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.975+0000 7f6307e00640 1 --2- 172.21.15.12:0/1712219604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63081037c0 0x7f6308199ab0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f62f8000c00 tx=0x7f62f802c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:36.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.975+0000 7f6305600640 1 -- 172.21.15.12:0/1712219604 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f8043020 con 0x7f63081037c0 2024-03-23T23:01:36.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.975+0000 7f6305600640 1 -- 172.21.15.12:0/1712219604 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62f80054e0 con 0x7f63081037c0 2024-03-23T23:01:36.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.975+0000 7f6305600640 1 -- 172.21.15.12:0/1712219604 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f8003ce0 con 0x7f63081037c0 2024-03-23T23:01:36.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.975+0000 7f630d400640 1 -- 172.21.15.12:0/1712219604 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63081a1960 con 0x7f63081037c0 2024-03-23T23:01:36.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.976+0000 7f630d400640 1 -- 172.21.15.12:0/1712219604 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63081a1e30 con 0x7f63081037c0 2024-03-23T23:01:36.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.977+0000 7f630d400640 1 -- 172.21.15.12:0/1712219604 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f630810aa80 con 0x7f63081037c0 2024-03-23T23:01:36.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.979+0000 7f6305600640 1 -- 172.21.15.12:0/1712219604 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f62f8046020 con 0x7f63081037c0 2024-03-23T23:01:36.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.980+0000 7f6305600640 1 --2- 172.21.15.12:0/1712219604 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f62ec0a2880 0x7f62ec0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:36.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.980+0000 7f6305600640 1 -- 172.21.15.12:0/1712219604 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f62f80f0b00 con 0x7f63081037c0 2024-03-23T23:01:36.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.980+0000 7f62ffe00640 1 --2- 172.21.15.12:0/1712219604 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f62ec0a2880 0x7f62ec0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:36.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.981+0000 7f62ffe00640 1 --2- 172.21.15.12:0/1712219604 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f62ec0a2880 0x7f62ec0a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f630819afc0 tx=0x7f62f000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:36.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:36.984+0000 7f6305600640 1 -- 172.21.15.12:0/1712219604 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f62f804d020 con 0x7f63081037c0 2024-03-23T23:01:37.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.190+0000 7f630d400640 1 -- 172.21.15.12:0/1712219604 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f63080622a0 con 0x7f63081037c0 2024-03-23T23:01:37.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.191+0000 7f6305600640 1 -- 172.21.15.12:0/1712219604 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f62f80bdb50 con 0x7f63081037c0 2024-03-23T23:01:37.191 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-23T23:01:37.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.194+0000 7f630d400640 1 -- 172.21.15.12:0/1712219604 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f62ec0a2880 msgr2=0x7f62ec0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:37.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.194+0000 7f630d400640 1 --2- 172.21.15.12:0/1712219604 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f62ec0a2880 0x7f62ec0a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f630819afc0 tx=0x7f62f000b390 comp rx=0 tx=0).stop 2024-03-23T23:01:37.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.194+0000 7f630d400640 1 -- 172.21.15.12:0/1712219604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63081037c0 msgr2=0x7f6308199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:37.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.194+0000 7f630d400640 1 --2- 172.21.15.12:0/1712219604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63081037c0 0x7f6308199ab0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f62f8000c00 tx=0x7f62f802c6c0 comp rx=0 tx=0).stop 2024-03-23T23:01:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.194+0000 7f630d400640 1 -- 172.21.15.12:0/1712219604 shutdown_connections 2024-03-23T23:01:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.194+0000 7f630d400640 1 --2- 172.21.15.12:0/1712219604 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f62ec0a2880 0x7f62ec0a4d40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.194+0000 7f630d400640 1 --2- 172.21.15.12:0/1712219604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6308105370 0x7f63081a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.194+0000 7f630d400640 1 --2- 172.21.15.12:0/1712219604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63081049b0 0x7f6308199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.194+0000 7f630d400640 1 --2- 172.21.15.12:0/1712219604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63081037c0 0x7f6308199ab0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.194+0000 7f630d400640 1 -- 172.21.15.12:0/1712219604 >> 172.21.15.12:0/1712219604 conn(0x7f63080fed50 msgr2=0x7f6308107c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.194+0000 7f630d400640 1 -- 172.21.15.12:0/1712219604 shutdown_connections 2024-03-23T23:01:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.194+0000 7f630d400640 1 -- 172.21.15.12:0/1712219604 wait complete. 2024-03-23T23:01:37.217 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:01:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.444+0000 7f8a33200640 1 -- 172.21.15.12:0/4219569306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a341048e0 msgr2=0x7f8a34104d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.444+0000 7f8a33200640 1 --2- 172.21.15.12:0/4219569306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a341048e0 0x7f8a34104d60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c009900 tx=0x7f8a1c02bc40 comp rx=0 tx=0).stop 2024-03-23T23:01:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.444+0000 7f8a33200640 1 -- 172.21.15.12:0/4219569306 shutdown_connections 2024-03-23T23:01:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.444+0000 7f8a33200640 1 --2- 172.21.15.12:0/4219569306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a341052a0 0x7f8a3410bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.444+0000 7f8a33200640 1 --2- 172.21.15.12:0/4219569306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a341048e0 0x7f8a34104d60 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.444+0000 7f8a33200640 1 --2- 172.21.15.12:0/4219569306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a341036f0 0x7f8a34103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.444+0000 7f8a33200640 1 -- 172.21.15.12:0/4219569306 >> 172.21.15.12:0/4219569306 conn(0x7f8a340fec80 msgr2=0x7f8a341010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.444+0000 7f8a33200640 1 -- 172.21.15.12:0/4219569306 shutdown_connections 2024-03-23T23:01:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.444+0000 7f8a33200640 1 -- 172.21.15.12:0/4219569306 wait complete. 2024-03-23T23:01:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.445+0000 7f8a33200640 1 Processor -- start 2024-03-23T23:01:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.446+0000 7f8a33200640 1 -- start start 2024-03-23T23:01:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.446+0000 7f8a33200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a341036f0 0x7f8a34199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.446+0000 7f8a33200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a341048e0 0x7f8a34199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.446+0000 7f8a33200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a341052a0 0x7f8a341a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.446+0000 7f8a33200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3410f3c0 con 0x7f8a341048e0 2024-03-23T23:01:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.446+0000 7f8a33200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3410f240 con 0x7f8a341052a0 2024-03-23T23:01:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.446+0000 7f8a33200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3410f540 con 0x7f8a341036f0 2024-03-23T23:01:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.446+0000 7f8a2be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a341048e0 0x7f8a34199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.446+0000 7f8a31e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a341036f0 0x7f8a34199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.446+0000 7f8a32800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a341052a0 0x7f8a341a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.446+0000 7f8a2be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a341048e0 0x7f8a34199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38832/0 (socket says 172.21.15.12:38832) 2024-03-23T23:01:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.446+0000 7f8a2be00640 1 -- 172.21.15.12:0/4165173766 learned_addr learned my addr 172.21.15.12:0/4165173766 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.447+0000 7f8a2be00640 1 -- 172.21.15.12:0/4165173766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a341036f0 msgr2=0x7f8a34199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.447+0000 7f8a2be00640 1 --2- 172.21.15.12:0/4165173766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a341036f0 0x7f8a34199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.447+0000 7f8a2be00640 1 -- 172.21.15.12:0/4165173766 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a341052a0 msgr2=0x7f8a341a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.447+0000 7f8a2be00640 1 --2- 172.21.15.12:0/4165173766 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a341052a0 0x7f8a341a0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.447+0000 7f8a2be00640 1 -- 172.21.15.12:0/4165173766 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a341a1640 con 0x7f8a341048e0 2024-03-23T23:01:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.447+0000 7f8a32800640 1 --2- 172.21.15.12:0/4165173766 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a341052a0 0x7f8a341a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.447+0000 7f8a2be00640 1 --2- 172.21.15.12:0/4165173766 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a341048e0 0x7f8a34199f60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c005ab0 tx=0x7f8a1c005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.447+0000 7f8a2aa00640 1 -- 172.21.15.12:0/4165173766 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c041070 con 0x7f8a341048e0 2024-03-23T23:01:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.447+0000 7f8a2aa00640 1 -- 172.21.15.12:0/4165173766 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8a1c0054e0 con 0x7f8a341048e0 2024-03-23T23:01:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.447+0000 7f8a33200640 1 -- 172.21.15.12:0/4165173766 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a341a18d0 con 0x7f8a341048e0 2024-03-23T23:01:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.447+0000 7f8a2aa00640 1 -- 172.21.15.12:0/4165173766 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c034440 con 0x7f8a341048e0 2024-03-23T23:01:37.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.447+0000 7f8a33200640 1 -- 172.21.15.12:0/4165173766 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a341a1dc0 con 0x7f8a341048e0 2024-03-23T23:01:37.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.449+0000 7f8a33200640 1 -- 172.21.15.12:0/4165173766 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a3410a9b0 con 0x7f8a341048e0 2024-03-23T23:01:37.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.450+0000 7f8a2aa00640 1 -- 172.21.15.12:0/4165173766 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8a1c044070 con 0x7f8a341048e0 2024-03-23T23:01:37.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.453+0000 7f8a2aa00640 1 --2- 172.21.15.12:0/4165173766 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8a180a2800 0x7f8a180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:37.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.453+0000 7f8a2aa00640 1 -- 172.21.15.12:0/4165173766 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8a1c0f0610 con 0x7f8a341048e0 2024-03-23T23:01:37.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.453+0000 7f8a31e00640 1 --2- 172.21.15.12:0/4165173766 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8a180a2800 0x7f8a180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:37.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.454+0000 7f8a31e00640 1 --2- 172.21.15.12:0/4165173766 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8a180a2800 0x7f8a180a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8a34104740 tx=0x7f8a24005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:37.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.458+0000 7f8a2aa00640 1 -- 172.21.15.12:0/4165173766 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8a1c0bd010 con 0x7f8a341048e0 2024-03-23T23:01:37.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.706+0000 7f8a33200640 1 -- 172.21.15.12:0/4165173766 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a340622a0 con 0x7f8a341048e0 2024-03-23T23:01:37.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.708+0000 7f8a2aa00640 1 -- 172.21.15.12:0/4165173766 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f8a1c0c10b0 con 0x7f8a341048e0 2024-03-23T23:01:37.708 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:01:37.708 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T22:59:32.190756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24599,24550]},"id":1}]} 2024-03-23T23:01:37.709 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 22 2024-03-23T23:01:37.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.710+0000 7f8a33200640 1 -- 172.21.15.12:0/4165173766 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8a180a2800 msgr2=0x7f8a180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:37.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.710+0000 7f8a33200640 1 --2- 172.21.15.12:0/4165173766 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8a180a2800 0x7f8a180a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8a34104740 tx=0x7f8a24005f90 comp rx=0 tx=0).stop 2024-03-23T23:01:37.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.711+0000 7f8a33200640 1 -- 172.21.15.12:0/4165173766 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a341048e0 msgr2=0x7f8a34199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:37.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.711+0000 7f8a33200640 1 --2- 172.21.15.12:0/4165173766 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a341048e0 0x7f8a34199f60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c005ab0 tx=0x7f8a1c005dd0 comp rx=0 tx=0).stop 2024-03-23T23:01:37.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.711+0000 7f8a33200640 1 -- 172.21.15.12:0/4165173766 shutdown_connections 2024-03-23T23:01:37.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.711+0000 7f8a33200640 1 --2- 172.21.15.12:0/4165173766 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8a180a2800 0x7f8a180a4cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:37.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.711+0000 7f8a33200640 1 --2- 172.21.15.12:0/4165173766 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a341052a0 0x7f8a341a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:37.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.711+0000 7f8a33200640 1 --2- 172.21.15.12:0/4165173766 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a341048e0 0x7f8a34199f60 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:37.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.711+0000 7f8a33200640 1 --2- 172.21.15.12:0/4165173766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a341036f0 0x7f8a34199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:37.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.711+0000 7f8a33200640 1 -- 172.21.15.12:0/4165173766 >> 172.21.15.12:0/4165173766 conn(0x7f8a340fec80 msgr2=0x7f8a34107b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:37.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.711+0000 7f8a33200640 1 -- 172.21.15.12:0/4165173766 shutdown_connections 2024-03-23T23:01:37.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.711+0000 7f8a33200640 1 -- 172.21.15.12:0/4165173766 wait complete. 2024-03-23T23:01:37.728 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:37 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1794139274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:01:37.728 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:37 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1712219604' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:01:37.732 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T23:01:37.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:37 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1794139274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:01:37.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:37 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1712219604' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:01:37.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:37 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1794139274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:01:37.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:37 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1712219604' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:01:37.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.958+0000 7f187a600640 1 -- 172.21.15.12:0/1623876226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f18740756e0 msgr2=0x7f1874073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:37.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.958+0000 7f187a600640 1 --2- 172.21.15.12:0/1623876226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f18740756e0 0x7f1874073b30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1864009ca0 tx=0x7f186402c100 comp rx=0 tx=0).stop 2024-03-23T23:01:37.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.958+0000 7f187a600640 1 -- 172.21.15.12:0/1623876226 shutdown_connections 2024-03-23T23:01:37.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.958+0000 7f187a600640 1 --2- 172.21.15.12:0/1623876226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18741093d0 0x7f187410b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:37.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.958+0000 7f187a600640 1 --2- 172.21.15.12:0/1623876226 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1874074100 0x7f1874074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:37.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.958+0000 7f187a600640 1 --2- 172.21.15.12:0/1623876226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f18740756e0 0x7f1874073b30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:37.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.958+0000 7f187a600640 1 -- 172.21.15.12:0/1623876226 >> 172.21.15.12:0/1623876226 conn(0x7f18740fca80 msgr2=0x7f18740feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:37.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.958+0000 7f187a600640 1 -- 172.21.15.12:0/1623876226 shutdown_connections 2024-03-23T23:01:37.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.958+0000 7f187a600640 1 -- 172.21.15.12:0/1623876226 wait complete. 2024-03-23T23:01:37.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.959+0000 7f187a600640 1 Processor -- start 2024-03-23T23:01:37.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.959+0000 7f187a600640 1 -- start start 2024-03-23T23:01:37.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.959+0000 7f187a600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1874074100 0x7f18741999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:37.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.959+0000 7f187a600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18740756e0 0x7f1874199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:37.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.959+0000 7f187a600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18741093d0 0x7f18741a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:37.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.959+0000 7f187a600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187410f350 con 0x7f18740756e0 2024-03-23T23:01:37.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.960+0000 7f187a600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187410f1d0 con 0x7f18741093d0 2024-03-23T23:01:37.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.960+0000 7f187a600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187410f4d0 con 0x7f1874074100 2024-03-23T23:01:37.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.960+0000 7f1879200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1874074100 0x7f18741999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:37.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.960+0000 7f1873e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18740756e0 0x7f1874199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:37.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.960+0000 7f1879200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1874074100 0x7f18741999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35362/0 (socket says 172.21.15.12:35362) 2024-03-23T23:01:37.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.960+0000 7f1879200640 1 -- 172.21.15.12:0/3093179507 learned_addr learned my addr 172.21.15.12:0/3093179507 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:37.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.960+0000 7f1879c00640 1 --2- 172.21.15.12:0/3093179507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18741093d0 0x7f18741a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:37.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.960+0000 7f1879200640 1 -- 172.21.15.12:0/3093179507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18741093d0 msgr2=0x7f18741a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:37.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.960+0000 7f1879200640 1 --2- 172.21.15.12:0/3093179507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18741093d0 0x7f18741a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:37.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.960+0000 7f1879200640 1 -- 172.21.15.12:0/3093179507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18740756e0 msgr2=0x7f1874199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:37.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.960+0000 7f1879200640 1 --2- 172.21.15.12:0/3093179507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18740756e0 0x7f1874199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:37.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.960+0000 7f1879200640 1 -- 172.21.15.12:0/3093179507 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18741a15f0 con 0x7f1874074100 2024-03-23T23:01:37.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.960+0000 7f1873e00640 1 --2- 172.21.15.12:0/3093179507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18740756e0 0x7f1874199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:37.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.961+0000 7f1879200640 1 --2- 172.21.15.12:0/3093179507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1874074100 0x7f18741999d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f18640325f0 tx=0x7f1864035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:37.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.961+0000 7f1871600640 1 -- 172.21.15.12:0/3093179507 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1864039040 con 0x7f1874074100 2024-03-23T23:01:37.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.961+0000 7f1871600640 1 -- 172.21.15.12:0/3093179507 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1864044070 con 0x7f1874074100 2024-03-23T23:01:37.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.961+0000 7f1871600640 1 -- 172.21.15.12:0/3093179507 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18640044d0 con 0x7f1874074100 2024-03-23T23:01:37.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.961+0000 7f187a600640 1 -- 172.21.15.12:0/3093179507 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18741a1880 con 0x7f1874074100 2024-03-23T23:01:37.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.961+0000 7f187a600640 1 -- 172.21.15.12:0/3093179507 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18741a1d90 con 0x7f1874074100 2024-03-23T23:01:37.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.962+0000 7f187a600640 1 -- 172.21.15.12:0/3093179507 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1874108100 con 0x7f1874074100 2024-03-23T23:01:37.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.964+0000 7f1871600640 1 -- 172.21.15.12:0/3093179507 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1864004670 con 0x7f1874074100 2024-03-23T23:01:37.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.965+0000 7f1871600640 1 --2- 172.21.15.12:0/3093179507 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f18580a2880 0x7f18580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:37.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.966+0000 7f1871600640 1 -- 172.21.15.12:0/3093179507 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f18640f0f80 con 0x7f1874074100 2024-03-23T23:01:37.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.966+0000 7f1873e00640 1 --2- 172.21.15.12:0/3093179507 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f18580a2880 0x7f18580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:37.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.966+0000 7f1873e00640 1 --2- 172.21.15.12:0/3093179507 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f18580a2880 0x7f18580a4d40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f185c0095f0 tx=0x7f185c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:37.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:37.969+0000 7f1871600640 1 -- 172.21.15.12:0/3093179507 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f18640bdfd0 con 0x7f1874074100 2024-03-23T23:01:38.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.173+0000 7f187a600640 1 -- 172.21.15.12:0/3093179507 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18740622a0 con 0x7f1874074100 2024-03-23T23:01:38.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.175+0000 7f1871600640 1 -- 172.21.15.12:0/3093179507 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22695 (secure 0 0 0) 0x7f18640bd8d0 con 0x7f1874074100 2024-03-23T23:01:38.175 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:01:38.175 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":88,"fsid":"90f679ae-e967-11ee-95cd-87774f69a715","created":"2024-03-23T22:50:14.906952+0000","modified":"2024-03-23T23:00:24.349052+0000","last_up_change":"2024-03-23T22:58:28.907824+0000","last_in_change":"2024-03-23T22:58:08.342487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T22:54:11.831010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T22:59:25.350974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T22:59:26.390323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b70875d7-a23d-4251-9e8b-1a16d4edc2a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6803","nonce":1190079611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6805","nonce":1190079611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6809","nonce":1190079611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6807","nonce":1190079611}]},"public_addr":"172.21.15.12:6803/1190079611","cluster_addr":"172.21.15.12:6805/1190079611","heartbeat_back_addr":"172.21.15.12:6809/1190079611","heartbeat_front_addr":"172.21.15.12:6807/1190079611","state":["exists","up"]},{"osd":1,"uuid":"416e87af-db5d-41ce-84d9-731e3e07e3c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6801","nonce":1983639525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6803","nonce":1983639525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6807","nonce":1983639525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6805","nonce":1983639525}]},"public_addr":"172.21.15.173:6801/1983639525","cluster_addr":"172.21.15.173:6803/1983639525","heartbeat_back_addr":"172.21.15.173:6807/1983639525","heartbeat_front_addr":"172.21.15.173:6805/1983639525","state":["exists","up"]},{"osd":2,"uuid":"7a559c70-4c54-431b-8baf-4136b6e76493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6801","nonce":2253997387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6803","nonce":2253997387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6807","nonce":2253997387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6805","nonce":2253997387}]},"public_addr":"172.21.15.203:6801/2253997387","cluster_addr":"172.21.15.203:6803/2253997387","heartbeat_back_addr":"172.21.15.203:6807/2253997387","heartbeat_front_addr":"172.21.15.203:6805/2253997387","state":["exists","up"]},{"osd":3,"uuid":"c41cd287-a674-486c-8025-328cd36e5ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6811","nonce":1761446905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6813","nonce":1761446905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6817","nonce":1761446905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6815","nonce":1761446905}]},"public_addr":"172.21.15.12:6811/1761446905","cluster_addr":"172.21.15.12:6813/1761446905","heartbeat_back_addr":"172.21.15.12:6817/1761446905","heartbeat_front_addr":"172.21.15.12:6815/1761446905","state":["exists","up"]},{"osd":4,"uuid":"3c48f886-0b42-4a09-b31e-abccc200bb1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6809","nonce":242958272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6811","nonce":242958272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6815","nonce":242958272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6813","nonce":242958272}]},"public_addr":"172.21.15.173:6809/242958272","cluster_addr":"172.21.15.173:6811/242958272","heartbeat_back_addr":"172.21.15.173:6815/242958272","heartbeat_front_addr":"172.21.15.173:6813/242958272","state":["exists","up"]},{"osd":5,"uuid":"4a34f06f-8805-40c0-b1ab-23407b0276d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6809","nonce":4270966092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6811","nonce":4270966092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6815","nonce":4270966092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6813","nonce":4270966092}]},"public_addr":"172.21.15.203:6809/4270966092","cluster_addr":"172.21.15.203:6811/4270966092","heartbeat_back_addr":"172.21.15.203:6815/4270966092","heartbeat_front_addr":"172.21.15.203:6813/4270966092","state":["exists","up"]},{"osd":6,"uuid":"26ddc663-5623-48c9-80d1-f8b968765d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6819","nonce":3606839295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6821","nonce":3606839295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6825","nonce":3606839295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6823","nonce":3606839295}]},"public_addr":"172.21.15.12:6819/3606839295","cluster_addr":"172.21.15.12:6821/3606839295","heartbeat_back_addr":"172.21.15.12:6825/3606839295","heartbeat_front_addr":"172.21.15.12:6823/3606839295","state":["exists","up"]},{"osd":7,"uuid":"17dc9417-71cd-4576-aa08-933212ec55e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6817","nonce":1241504346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6819","nonce":1241504346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6823","nonce":1241504346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6821","nonce":1241504346}]},"public_addr":"172.21.15.173:6817/1241504346","cluster_addr":"172.21.15.173:6819/1241504346","heartbeat_back_addr":"172.21.15.173:6823/1241504346","heartbeat_front_addr":"172.21.15.173:6821/1241504346","state":["exists","up"]},{"osd":8,"uuid":"90a2b63a-471d-43b8-ac18-bb1f16778d18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6817","nonce":1516996192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6819","nonce":1516996192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6823","nonce":1516996192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6821","nonce":1516996192}]},"public_addr":"172.21.15.203:6817/1516996192","cluster_addr":"172.21.15.203:6819/1516996192","heartbeat_back_addr":"172.21.15.203:6823/1516996192","heartbeat_front_addr":"172.21.15.203:6821/1516996192","state":["exists","up"]},{"osd":9,"uuid":"6e1ca504-40b4-49d0-b245-7946181abfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6827","nonce":1175260774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6829","nonce":1175260774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6833","nonce":1175260774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6831","nonce":1175260774}]},"public_addr":"172.21.15.12:6827/1175260774","cluster_addr":"172.21.15.12:6829/1175260774","heartbeat_back_addr":"172.21.15.12:6833/1175260774","heartbeat_front_addr":"172.21.15.12:6831/1175260774","state":["exists","up"]},{"osd":10,"uuid":"09a595ed-5614-4386-8e61-0170af66e01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6825","nonce":3303506808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6827","nonce":3303506808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6831","nonce":3303506808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6829","nonce":3303506808}]},"public_addr":"172.21.15.173:6825/3303506808","cluster_addr":"172.21.15.173:6827/3303506808","heartbeat_back_addr":"172.21.15.173:6831/3303506808","heartbeat_front_addr":"172.21.15.173:6829/3303506808","state":["exists","up"]},{"osd":11,"uuid":"9ef1d595-a977-4ba7-9c6b-d2b3295f20b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6825","nonce":2323604506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6827","nonce":2323604506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6831","nonce":2323604506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6829","nonce":2323604506}]},"public_addr":"172.21.15.203:6825/2323604506","cluster_addr":"172.21.15.203:6827/2323604506","heartbeat_back_addr":"172.21.15.203:6831/2323604506","heartbeat_front_addr":"172.21.15.203:6829/2323604506","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:18.469038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:42.752628+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:07.069493+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:41.962478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:08.363104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:33.341336+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:07.286541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:34.243302+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:59.749699+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:57:33.822913+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:00.737269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:26.599854+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/4182783120":"2024-03-24T22:51:23.113201+0000","172.21.15.12:6800/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1900938637":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/3414861652":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6801/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/2580541477":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6800/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/501058839":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1192554716":"2024-03-24T22:50:48.401223+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T23:01:38.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.178+0000 7f187a600640 1 -- 172.21.15.12:0/3093179507 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f18580a2880 msgr2=0x7f18580a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:38.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.178+0000 7f187a600640 1 --2- 172.21.15.12:0/3093179507 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f18580a2880 0x7f18580a4d40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f185c0095f0 tx=0x7f185c005f90 comp rx=0 tx=0).stop 2024-03-23T23:01:38.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.178+0000 7f187a600640 1 -- 172.21.15.12:0/3093179507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1874074100 msgr2=0x7f18741999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:38.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.178+0000 7f187a600640 1 --2- 172.21.15.12:0/3093179507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1874074100 0x7f18741999d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f18640325f0 tx=0x7f1864035d10 comp rx=0 tx=0).stop 2024-03-23T23:01:38.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.178+0000 7f187a600640 1 -- 172.21.15.12:0/3093179507 shutdown_connections 2024-03-23T23:01:38.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.178+0000 7f187a600640 1 --2- 172.21.15.12:0/3093179507 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f18580a2880 0x7f18580a4d40 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:38.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.178+0000 7f187a600640 1 --2- 172.21.15.12:0/3093179507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18741093d0 0x7f18741a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:38.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.178+0000 7f187a600640 1 --2- 172.21.15.12:0/3093179507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18740756e0 0x7f1874199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:38.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.178+0000 7f187a600640 1 --2- 172.21.15.12:0/3093179507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1874074100 0x7f18741999d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:38.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.178+0000 7f187a600640 1 -- 172.21.15.12:0/3093179507 >> 172.21.15.12:0/3093179507 conn(0x7f18740fca80 msgr2=0x7f18741052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:38.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.179+0000 7f187a600640 1 -- 172.21.15.12:0/3093179507 shutdown_connections 2024-03-23T23:01:38.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.179+0000 7f187a600640 1 -- 172.21.15.12:0/3093179507 wait complete. 2024-03-23T23:01:38.198 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T22:54:11.831010+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T22:59:25.350974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T22:59:26.390323+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T23:01:38.199 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T23:01:38.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.431+0000 7f537b200640 1 -- 172.21.15.12:0/3319099192 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f537c0740a0 msgr2=0x7f537c074500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:38.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.431+0000 7f537b200640 1 --2- 172.21.15.12:0/3319099192 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f537c0740a0 0x7f537c074500 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5364009ca0 tx=0x7f536402c190 comp rx=0 tx=0).stop 2024-03-23T23:01:38.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.431+0000 7f537b200640 1 -- 172.21.15.12:0/3319099192 shutdown_connections 2024-03-23T23:01:38.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.431+0000 7f537b200640 1 --2- 172.21.15.12:0/3319099192 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f537c109430 0x7f537c10b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:38.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.431+0000 7f537b200640 1 --2- 172.21.15.12:0/3319099192 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f537c0740a0 0x7f537c074500 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:38.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.431+0000 7f537b200640 1 --2- 172.21.15.12:0/3319099192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f537c075680 0x7f537c073ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:38.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.431+0000 7f537b200640 1 -- 172.21.15.12:0/3319099192 >> 172.21.15.12:0/3319099192 conn(0x7f537c0fcb10 msgr2=0x7f537c0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:38.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.431+0000 7f537b200640 1 -- 172.21.15.12:0/3319099192 shutdown_connections 2024-03-23T23:01:38.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.431+0000 7f537b200640 1 -- 172.21.15.12:0/3319099192 wait complete. 2024-03-23T23:01:38.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.432+0000 7f537b200640 1 Processor -- start 2024-03-23T23:01:38.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.432+0000 7f537b200640 1 -- start start 2024-03-23T23:01:38.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.432+0000 7f537b200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f537c0740a0 0x7f537c199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:38.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.432+0000 7f537b200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f537c075680 0x7f537c199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:38.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.433+0000 7f537b200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f537c109430 0x7f537c1a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:38.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.433+0000 7f537b200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537c10f400 con 0x7f537c109430 2024-03-23T23:01:38.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.433+0000 7f537b200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537c10f280 con 0x7f537c075680 2024-03-23T23:01:38.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.433+0000 7f537b200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537c10f580 con 0x7f537c0740a0 2024-03-23T23:01:38.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.433+0000 7f537a800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f537c109430 0x7f537c1a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:38.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.433+0000 7f537a800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f537c109430 0x7f537c1a1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38850/0 (socket says 172.21.15.12:38850) 2024-03-23T23:01:38.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.433+0000 7f537a800640 1 -- 172.21.15.12:0/743057613 learned_addr learned my addr 172.21.15.12:0/743057613 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:38.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.433+0000 7f537a800640 1 -- 172.21.15.12:0/743057613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f537c0740a0 msgr2=0x7f537c199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:38.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.433+0000 7f537a800640 1 --2- 172.21.15.12:0/743057613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f537c0740a0 0x7f537c199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:38.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.433+0000 7f537a800640 1 -- 172.21.15.12:0/743057613 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f537c075680 msgr2=0x7f537c199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:38.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.433+0000 7f5379400640 1 --2- 172.21.15.12:0/743057613 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f537c075680 0x7f537c199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:38.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.433+0000 7f537a800640 1 --2- 172.21.15.12:0/743057613 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f537c075680 0x7f537c199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:38.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.433+0000 7f537a800640 1 -- 172.21.15.12:0/743057613 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f537c1a1670 con 0x7f537c109430 2024-03-23T23:01:38.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.433+0000 7f5379400640 1 --2- 172.21.15.12:0/743057613 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f537c075680 0x7f537c199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:01:38.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.434+0000 7f537a800640 1 --2- 172.21.15.12:0/743057613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f537c109430 0x7f537c1a1000 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f537400e970 tx=0x7f537400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:38.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.434+0000 7f5372a00640 1 -- 172.21.15.12:0/743057613 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537400ccc0 con 0x7f537c109430 2024-03-23T23:01:38.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.434+0000 7f537b200640 1 -- 172.21.15.12:0/743057613 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f537c1a1960 con 0x7f537c109430 2024-03-23T23:01:38.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.434+0000 7f5372a00640 1 -- 172.21.15.12:0/743057613 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f537400ce60 con 0x7f537c109430 2024-03-23T23:01:38.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.434+0000 7f5372a00640 1 -- 172.21.15.12:0/743057613 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5374010490 con 0x7f537c109430 2024-03-23T23:01:38.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.434+0000 7f537b200640 1 -- 172.21.15.12:0/743057613 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f537c1a1e60 con 0x7f537c109430 2024-03-23T23:01:38.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.435+0000 7f537b200640 1 -- 172.21.15.12:0/743057613 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f537c1081f0 con 0x7f537c109430 2024-03-23T23:01:38.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.437+0000 7f5372a00640 1 -- 172.21.15.12:0/743057613 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5374005020 con 0x7f537c109430 2024-03-23T23:01:38.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.438+0000 7f5372a00640 1 --2- 172.21.15.12:0/743057613 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f535c0a2800 0x7f535c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:38.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.438+0000 7f5372a00640 1 -- 172.21.15.12:0/743057613 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f53740cf030 con 0x7f537c109430 2024-03-23T23:01:38.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.439+0000 7f5379e00640 1 --2- 172.21.15.12:0/743057613 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f535c0a2800 0x7f535c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:38.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.439+0000 7f5379e00640 1 --2- 172.21.15.12:0/743057613 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f535c0a2800 0x7f535c0a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f536c005d00 tx=0x7f536c005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:38.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.442+0000 7f5372a00640 1 -- 172.21.15.12:0/743057613 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f537409c0e0 con 0x7f537c109430 2024-03-23T23:01:38.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.648+0000 7f537b200640 1 -- 172.21.15.12:0/743057613 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f537c0622a0 con 0x7f537c109430 2024-03-23T23:01:38.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.648+0000 7f5372a00640 1 -- 172.21.15.12:0/743057613 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f537409b9e0 con 0x7f537c109430 2024-03-23T23:01:38.648 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-23T23:01:38.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.651+0000 7f537b200640 1 -- 172.21.15.12:0/743057613 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f535c0a2800 msgr2=0x7f535c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:38.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.651+0000 7f537b200640 1 --2- 172.21.15.12:0/743057613 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f535c0a2800 0x7f535c0a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f536c005d00 tx=0x7f536c005c50 comp rx=0 tx=0).stop 2024-03-23T23:01:38.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.651+0000 7f537b200640 1 -- 172.21.15.12:0/743057613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f537c109430 msgr2=0x7f537c1a1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:38.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.651+0000 7f537b200640 1 --2- 172.21.15.12:0/743057613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f537c109430 0x7f537c1a1000 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f537400e970 tx=0x7f537400ed70 comp rx=0 tx=0).stop 2024-03-23T23:01:38.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.651+0000 7f537b200640 1 -- 172.21.15.12:0/743057613 shutdown_connections 2024-03-23T23:01:38.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.651+0000 7f537b200640 1 --2- 172.21.15.12:0/743057613 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f535c0a2800 0x7f535c0a4cc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:38.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.651+0000 7f537b200640 1 --2- 172.21.15.12:0/743057613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f537c109430 0x7f537c1a1000 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:38.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.651+0000 7f537b200640 1 --2- 172.21.15.12:0/743057613 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f537c075680 0x7f537c199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:38.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.651+0000 7f537b200640 1 --2- 172.21.15.12:0/743057613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f537c0740a0 0x7f537c199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:38.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.651+0000 7f537b200640 1 -- 172.21.15.12:0/743057613 >> 172.21.15.12:0/743057613 conn(0x7f537c0fcb10 msgr2=0x7f537c1053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:38.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.651+0000 7f537b200640 1 -- 172.21.15.12:0/743057613 shutdown_connections 2024-03-23T23:01:38.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.651+0000 7f537b200640 1 -- 172.21.15.12:0/743057613 wait complete. 2024-03-23T23:01:38.670 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T23:01:38.753 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:38 smithi012 ceph-mon[31695]: pgmap v385: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-23T23:01:38.753 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:38 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/4165173766' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:01:38.753 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:38 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3093179507' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:38.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:38 smithi173 ceph-mon[36116]: pgmap v385: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-23T23:01:38.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:38 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/4165173766' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:01:38.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:38 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3093179507' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:38.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.896+0000 7f2f1a600640 1 -- 172.21.15.12:0/1878835430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f14107260 msgr2=0x7f2f141096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:38.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.896+0000 7f2f1a600640 1 --2- 172.21.15.12:0/1878835430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f14107260 0x7f2f141096b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c009be0 tx=0x7f2f0c02bef0 comp rx=0 tx=0).stop 2024-03-23T23:01:38.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.896+0000 7f2f1a600640 1 -- 172.21.15.12:0/1878835430 shutdown_connections 2024-03-23T23:01:38.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.896+0000 7f2f1a600640 1 --2- 172.21.15.12:0/1878835430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f14107260 0x7f2f141096b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:38.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.896+0000 7f2f1a600640 1 --2- 172.21.15.12:0/1878835430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f14104930 0x7f2f14106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:38.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.896+0000 7f2f1a600640 1 --2- 172.21.15.12:0/1878835430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f14102000 0x7f2f141043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:38.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.896+0000 7f2f1a600640 1 -- 172.21.15.12:0/1878835430 >> 172.21.15.12:0/1878835430 conn(0x7f2f140fba10 msgr2=0x7f2f140fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:38.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.896+0000 7f2f1a600640 1 -- 172.21.15.12:0/1878835430 shutdown_connections 2024-03-23T23:01:38.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.896+0000 7f2f1a600640 1 -- 172.21.15.12:0/1878835430 wait complete. 2024-03-23T23:01:38.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.897+0000 7f2f1a600640 1 Processor -- start 2024-03-23T23:01:38.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.897+0000 7f2f1a600640 1 -- start start 2024-03-23T23:01:38.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.897+0000 7f2f1a600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f14102000 0x7f2f14199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:38.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.897+0000 7f2f1a600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f14104930 0x7f2f14199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:38.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.898+0000 7f2f1a600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f14107260 0x7f2f141a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:38.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.898+0000 7f2f1a600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1410d210 con 0x7f2f14102000 2024-03-23T23:01:38.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.898+0000 7f2f1a600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1410d090 con 0x7f2f14107260 2024-03-23T23:01:38.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.898+0000 7f2f1a600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1410d390 con 0x7f2f14104930 2024-03-23T23:01:38.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.898+0000 7f2f19200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f14102000 0x7f2f14199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:38.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.898+0000 7f2f0be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f14104930 0x7f2f14199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:38.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.898+0000 7f2f0be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f14104930 0x7f2f14199fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35394/0 (socket says 172.21.15.12:35394) 2024-03-23T23:01:38.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.898+0000 7f2f0be00640 1 -- 172.21.15.12:0/1890946496 learned_addr learned my addr 172.21.15.12:0/1890946496 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:38.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.898+0000 7f2f19c00640 1 --2- 172.21.15.12:0/1890946496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f14107260 0x7f2f141a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:38.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.898+0000 7f2f0be00640 1 -- 172.21.15.12:0/1890946496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f14107260 msgr2=0x7f2f141a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:38.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.898+0000 7f2f0be00640 1 --2- 172.21.15.12:0/1890946496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f14107260 0x7f2f141a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:38.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.898+0000 7f2f0be00640 1 -- 172.21.15.12:0/1890946496 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f14102000 msgr2=0x7f2f14199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:38.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.898+0000 7f2f0be00640 1 --2- 172.21.15.12:0/1890946496 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f14102000 0x7f2f14199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:38.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.898+0000 7f2f0be00640 1 -- 172.21.15.12:0/1890946496 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f141a1630 con 0x7f2f14104930 2024-03-23T23:01:38.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.898+0000 7f2f19200640 1 --2- 172.21.15.12:0/1890946496 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f14102000 0x7f2f14199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:38.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.899+0000 7f2f0be00640 1 --2- 172.21.15.12:0/1890946496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f14104930 0x7f2f14199fc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2efc00e9c0 tx=0x7f2efc00edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:38.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.899+0000 7f2f12000640 1 -- 172.21.15.12:0/1890946496 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc00ccc0 con 0x7f2f14104930 2024-03-23T23:01:38.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.899+0000 7f2f12000640 1 -- 172.21.15.12:0/1890946496 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2efc005500 con 0x7f2f14104930 2024-03-23T23:01:38.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.899+0000 7f2f1a600640 1 -- 172.21.15.12:0/1890946496 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f141a1920 con 0x7f2f14104930 2024-03-23T23:01:38.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.899+0000 7f2f12000640 1 -- 172.21.15.12:0/1890946496 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc010430 con 0x7f2f14104930 2024-03-23T23:01:38.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.899+0000 7f2f1a600640 1 -- 172.21.15.12:0/1890946496 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f141a1e70 con 0x7f2f14104930 2024-03-23T23:01:38.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.900+0000 7f2f1a600640 1 -- 172.21.15.12:0/1890946496 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f14108d50 con 0x7f2f14104930 2024-03-23T23:01:38.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.902+0000 7f2f12000640 1 -- 172.21.15.12:0/1890946496 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2efc005000 con 0x7f2f14104930 2024-03-23T23:01:38.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.903+0000 7f2f12000640 1 --2- 172.21.15.12:0/1890946496 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ef80a2800 0x7f2ef80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:38.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.903+0000 7f2f12000640 1 -- 172.21.15.12:0/1890946496 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2efc0cee90 con 0x7f2f14104930 2024-03-23T23:01:38.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.903+0000 7f2f19200640 1 --2- 172.21.15.12:0/1890946496 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ef80a2800 0x7f2ef80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:38.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.904+0000 7f2f19200640 1 --2- 172.21.15.12:0/1890946496 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ef80a2800 0x7f2ef80a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2f04005fd0 tx=0x7f2f04009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:38.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:38.907+0000 7f2f12000640 1 -- 172.21.15.12:0/1890946496 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2efc09bee0 con 0x7f2f14104930 2024-03-23T23:01:38.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:38 smithi203 ceph-mon[36693]: pgmap v385: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-23T23:01:38.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:38 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/4165173766' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:01:38.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:38 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3093179507' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:39.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.107+0000 7f2f1a600640 1 -- 172.21.15.12:0/1890946496 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2f140622a0 con 0x7f2f14104930 2024-03-23T23:01:39.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.108+0000 7f2f12000640 1 -- 172.21.15.12:0/1890946496 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2efc09b7e0 con 0x7f2f14104930 2024-03-23T23:01:39.108 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-23T23:01:39.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.110+0000 7f2f1a600640 1 -- 172.21.15.12:0/1890946496 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ef80a2800 msgr2=0x7f2ef80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:39.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.110+0000 7f2f1a600640 1 --2- 172.21.15.12:0/1890946496 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ef80a2800 0x7f2ef80a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2f04005fd0 tx=0x7f2f04009260 comp rx=0 tx=0).stop 2024-03-23T23:01:39.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.110+0000 7f2f1a600640 1 -- 172.21.15.12:0/1890946496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f14104930 msgr2=0x7f2f14199fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:39.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.110+0000 7f2f1a600640 1 --2- 172.21.15.12:0/1890946496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f14104930 0x7f2f14199fc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2efc00e9c0 tx=0x7f2efc00edc0 comp rx=0 tx=0).stop 2024-03-23T23:01:39.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.110+0000 7f2f1a600640 1 -- 172.21.15.12:0/1890946496 shutdown_connections 2024-03-23T23:01:39.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.110+0000 7f2f1a600640 1 --2- 172.21.15.12:0/1890946496 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ef80a2800 0x7f2ef80a4cc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:39.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.110+0000 7f2f1a600640 1 --2- 172.21.15.12:0/1890946496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f14107260 0x7f2f141a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:39.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.110+0000 7f2f1a600640 1 --2- 172.21.15.12:0/1890946496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f14104930 0x7f2f14199fc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:39.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.110+0000 7f2f1a600640 1 --2- 172.21.15.12:0/1890946496 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f14102000 0x7f2f14199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:39.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.110+0000 7f2f1a600640 1 -- 172.21.15.12:0/1890946496 >> 172.21.15.12:0/1890946496 conn(0x7f2f140fba10 msgr2=0x7f2f141016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:39.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.111+0000 7f2f1a600640 1 -- 172.21.15.12:0/1890946496 shutdown_connections 2024-03-23T23:01:39.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.111+0000 7f2f1a600640 1 -- 172.21.15.12:0/1890946496 wait complete. 2024-03-23T23:01:39.131 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T23:01:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.364+0000 7f5c63600640 1 -- 172.21.15.12:0/1493362172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c5c074100 msgr2=0x7f5c5c074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.364+0000 7f5c63600640 1 --2- 172.21.15.12:0/1493362172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c5c074100 0x7f5c5c074580 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5c48009930 tx=0x7f5c4802bc30 comp rx=0 tx=0).stop 2024-03-23T23:01:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.364+0000 7f5c63600640 1 -- 172.21.15.12:0/1493362172 shutdown_connections 2024-03-23T23:01:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.364+0000 7f5c63600640 1 --2- 172.21.15.12:0/1493362172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c5c10dc80 0x7f5c5c110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.364+0000 7f5c63600640 1 --2- 172.21.15.12:0/1493362172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c5c074100 0x7f5c5c074580 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.364+0000 7f5c63600640 1 --2- 172.21.15.12:0/1493362172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c5c0756e0 0x7f5c5c073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.364+0000 7f5c63600640 1 -- 172.21.15.12:0/1493362172 >> 172.21.15.12:0/1493362172 conn(0x7f5c5c0fcc40 msgr2=0x7f5c5c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.365+0000 7f5c63600640 1 -- 172.21.15.12:0/1493362172 shutdown_connections 2024-03-23T23:01:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.365+0000 7f5c63600640 1 -- 172.21.15.12:0/1493362172 wait complete. 2024-03-23T23:01:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.365+0000 7f5c63600640 1 Processor -- start 2024-03-23T23:01:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.366+0000 7f5c63600640 1 -- start start 2024-03-23T23:01:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.366+0000 7f5c63600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c5c074100 0x7f5c5c19e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.366+0000 7f5c63600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c5c0756e0 0x7f5c5c19e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.366+0000 7f5c63600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c5c10dc80 0x7f5c5c1a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.366+0000 7f5c63600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c113950 con 0x7f5c5c10dc80 2024-03-23T23:01:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.366+0000 7f5c63600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c1137d0 con 0x7f5c5c0756e0 2024-03-23T23:01:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.366+0000 7f5c63600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c113ad0 con 0x7f5c5c074100 2024-03-23T23:01:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.366+0000 7f5c60c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c5c074100 0x7f5c5c19e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.366+0000 7f5c60c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c5c074100 0x7f5c5c19e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35406/0 (socket says 172.21.15.12:35406) 2024-03-23T23:01:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.366+0000 7f5c60c00640 1 -- 172.21.15.12:0/235874400 learned_addr learned my addr 172.21.15.12:0/235874400 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.366+0000 7f5c61600640 1 --2- 172.21.15.12:0/235874400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c5c10dc80 0x7f5c5c1a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.366+0000 7f5c5be00640 1 --2- 172.21.15.12:0/235874400 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c5c0756e0 0x7f5c5c19e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.367+0000 7f5c60c00640 1 -- 172.21.15.12:0/235874400 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c5c0756e0 msgr2=0x7f5c5c19e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.367+0000 7f5c60c00640 1 --2- 172.21.15.12:0/235874400 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c5c0756e0 0x7f5c5c19e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.367+0000 7f5c60c00640 1 -- 172.21.15.12:0/235874400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c5c10dc80 msgr2=0x7f5c5c1a55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.367+0000 7f5c60c00640 1 --2- 172.21.15.12:0/235874400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c5c10dc80 0x7f5c5c1a55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.367+0000 7f5c60c00640 1 -- 172.21.15.12:0/235874400 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5c1a5c40 con 0x7f5c5c074100 2024-03-23T23:01:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.367+0000 7f5c61600640 1 --2- 172.21.15.12:0/235874400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c5c10dc80 0x7f5c5c1a55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:01:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.367+0000 7f5c60c00640 1 --2- 172.21.15.12:0/235874400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c5c074100 0x7f5c5c19e020 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5c5000e9f0 tx=0x7f5c5000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.367+0000 7f5c59600640 1 -- 172.21.15.12:0/235874400 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5000cd80 con 0x7f5c5c074100 2024-03-23T23:01:39.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.367+0000 7f5c59600640 1 -- 172.21.15.12:0/235874400 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5c50010910 con 0x7f5c5c074100 2024-03-23T23:01:39.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.367+0000 7f5c63600640 1 -- 172.21.15.12:0/235874400 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5c1a5f30 con 0x7f5c5c074100 2024-03-23T23:01:39.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.367+0000 7f5c59600640 1 -- 172.21.15.12:0/235874400 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c50018990 con 0x7f5c5c074100 2024-03-23T23:01:39.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.367+0000 7f5c63600640 1 -- 172.21.15.12:0/235874400 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5c1a6390 con 0x7f5c5c074100 2024-03-23T23:01:39.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.368+0000 7f5c63600640 1 -- 172.21.15.12:0/235874400 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c5c103e50 con 0x7f5c5c074100 2024-03-23T23:01:39.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.370+0000 7f5c59600640 1 -- 172.21.15.12:0/235874400 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5c50010ac0 con 0x7f5c5c074100 2024-03-23T23:01:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.372+0000 7f5c59600640 1 --2- 172.21.15.12:0/235874400 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5c440a2800 0x7f5c440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.372+0000 7f5c59600640 1 -- 172.21.15.12:0/235874400 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5c500cee90 con 0x7f5c5c074100 2024-03-23T23:01:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.372+0000 7f5c5be00640 1 --2- 172.21.15.12:0/235874400 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5c440a2800 0x7f5c440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.373+0000 7f5c5be00640 1 --2- 172.21.15.12:0/235874400 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5c440a2800 0x7f5c440a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5c48036040 tx=0x7f5c4802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:39.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.376+0000 7f5c59600640 1 -- 172.21.15.12:0/235874400 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5c5009bee0 con 0x7f5c5c074100 2024-03-23T23:01:39.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.581+0000 7f5c63600640 1 -- 172.21.15.12:0/235874400 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5c5c062370 con 0x7f5c5c074100 2024-03-23T23:01:39.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.582+0000 7f5c59600640 1 -- 172.21.15.12:0/235874400 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5c5009b7e0 con 0x7f5c5c074100 2024-03-23T23:01:39.582 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-23T23:01:39.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.585+0000 7f5c63600640 1 -- 172.21.15.12:0/235874400 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5c440a2800 msgr2=0x7f5c440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:39.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.585+0000 7f5c63600640 1 --2- 172.21.15.12:0/235874400 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5c440a2800 0x7f5c440a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5c48036040 tx=0x7f5c4802df90 comp rx=0 tx=0).stop 2024-03-23T23:01:39.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.585+0000 7f5c63600640 1 -- 172.21.15.12:0/235874400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c5c074100 msgr2=0x7f5c5c19e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:39.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.585+0000 7f5c63600640 1 --2- 172.21.15.12:0/235874400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c5c074100 0x7f5c5c19e020 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5c5000e9f0 tx=0x7f5c5000eeb0 comp rx=0 tx=0).stop 2024-03-23T23:01:39.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.585+0000 7f5c63600640 1 -- 172.21.15.12:0/235874400 shutdown_connections 2024-03-23T23:01:39.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.585+0000 7f5c63600640 1 --2- 172.21.15.12:0/235874400 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5c440a2800 0x7f5c440a4cc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:39.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.585+0000 7f5c63600640 1 --2- 172.21.15.12:0/235874400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c5c10dc80 0x7f5c5c1a55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:39.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.585+0000 7f5c63600640 1 --2- 172.21.15.12:0/235874400 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c5c0756e0 0x7f5c5c19e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:39.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.585+0000 7f5c63600640 1 --2- 172.21.15.12:0/235874400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c5c074100 0x7f5c5c19e020 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:39.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.585+0000 7f5c63600640 1 -- 172.21.15.12:0/235874400 >> 172.21.15.12:0/235874400 conn(0x7f5c5c0fcc40 msgr2=0x7f5c5c101020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:39.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.585+0000 7f5c63600640 1 -- 172.21.15.12:0/235874400 shutdown_connections 2024-03-23T23:01:39.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.585+0000 7f5c63600640 1 -- 172.21.15.12:0/235874400 wait complete. 2024-03-23T23:01:39.607 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:01:39.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:39 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/743057613' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:01:39.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:39 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1890946496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:01:39.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.829+0000 7f6048000640 1 -- 172.21.15.12:0/809060650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6040104ad0 msgr2=0x7f6040104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:39.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.829+0000 7f6048000640 1 --2- 172.21.15.12:0/809060650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6040104ad0 0x7f6040104f50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6030009900 tx=0x7f603002bc30 comp rx=0 tx=0).stop 2024-03-23T23:01:39.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.830+0000 7f6048000640 1 -- 172.21.15.12:0/809060650 shutdown_connections 2024-03-23T23:01:39.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.830+0000 7f6048000640 1 --2- 172.21.15.12:0/809060650 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6040105490 0x7f604010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:39.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.830+0000 7f6048000640 1 --2- 172.21.15.12:0/809060650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6040104ad0 0x7f6040104f50 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:39.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.830+0000 7f6048000640 1 --2- 172.21.15.12:0/809060650 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60401038e0 0x7f6040103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:39.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.830+0000 7f6048000640 1 -- 172.21.15.12:0/809060650 >> 172.21.15.12:0/809060650 conn(0x7f60400fee50 msgr2=0x7f6040101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:39.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.830+0000 7f6048000640 1 -- 172.21.15.12:0/809060650 shutdown_connections 2024-03-23T23:01:39.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.830+0000 7f6048000640 1 -- 172.21.15.12:0/809060650 wait complete. 2024-03-23T23:01:39.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.831+0000 7f6048000640 1 Processor -- start 2024-03-23T23:01:39.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.831+0000 7f6048000640 1 -- start start 2024-03-23T23:01:39.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.831+0000 7f6048000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60401038e0 0x7f6040199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:39.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.832+0000 7f6048000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6040104ad0 0x7f604019a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:39.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.832+0000 7f6048000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6040105490 0x7f60401a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:39.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.832+0000 7f6048000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604010f5b0 con 0x7f60401038e0 2024-03-23T23:01:39.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.832+0000 7f6048000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604010f430 con 0x7f6040105490 2024-03-23T23:01:39.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.832+0000 7f6048000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604010f730 con 0x7f6040104ad0 2024-03-23T23:01:39.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.832+0000 7f6044c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6040104ad0 0x7f604019a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:39.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.832+0000 7f6044c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6040104ad0 0x7f604019a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35430/0 (socket says 172.21.15.12:35430) 2024-03-23T23:01:39.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.832+0000 7f6044c00640 1 -- 172.21.15.12:0/1829134322 learned_addr learned my addr 172.21.15.12:0/1829134322 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:39.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.832+0000 7f6046000640 1 --2- 172.21.15.12:0/1829134322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6040105490 0x7f60401a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:39.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.832+0000 7f6045600640 1 --2- 172.21.15.12:0/1829134322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60401038e0 0x7f6040199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:39.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.832+0000 7f6044c00640 1 -- 172.21.15.12:0/1829134322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6040105490 msgr2=0x7f60401a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:39.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.832+0000 7f6044c00640 1 --2- 172.21.15.12:0/1829134322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6040105490 0x7f60401a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:39.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.832+0000 7f6044c00640 1 -- 172.21.15.12:0/1829134322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60401038e0 msgr2=0x7f6040199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:39.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.832+0000 7f6044c00640 1 --2- 172.21.15.12:0/1829134322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60401038e0 0x7f6040199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:39.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.832+0000 7f6044c00640 1 -- 172.21.15.12:0/1829134322 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60401a17e0 con 0x7f6040104ad0 2024-03-23T23:01:39.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.832+0000 7f6045600640 1 --2- 172.21.15.12:0/1829134322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60401038e0 0x7f6040199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:01:39.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.833+0000 7f6044c00640 1 --2- 172.21.15.12:0/1829134322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6040104ad0 0x7f604019a100 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6030005ab0 tx=0x7f603002c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:39.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.833+0000 7f603e000640 1 -- 172.21.15.12:0/1829134322 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6030043020 con 0x7f6040104ad0 2024-03-23T23:01:39.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.833+0000 7f603e000640 1 -- 172.21.15.12:0/1829134322 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60300054e0 con 0x7f6040104ad0 2024-03-23T23:01:39.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.833+0000 7f603e000640 1 -- 172.21.15.12:0/1829134322 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6030003ce0 con 0x7f6040104ad0 2024-03-23T23:01:39.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.833+0000 7f6048000640 1 -- 172.21.15.12:0/1829134322 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60401a1a70 con 0x7f6040104ad0 2024-03-23T23:01:39.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.833+0000 7f6048000640 1 -- 172.21.15.12:0/1829134322 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60401a2010 con 0x7f6040104ad0 2024-03-23T23:01:39.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.834+0000 7f6048000640 1 -- 172.21.15.12:0/1829134322 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f604010ac00 con 0x7f6040104ad0 2024-03-23T23:01:39.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.836+0000 7f603e000640 1 -- 172.21.15.12:0/1829134322 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6030046020 con 0x7f6040104ad0 2024-03-23T23:01:39.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.837+0000 7f603e000640 1 --2- 172.21.15.12:0/1829134322 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f60280a2880 0x7f60280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:39.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.837+0000 7f603e000640 1 -- 172.21.15.12:0/1829134322 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f60300f0bb0 con 0x7f6040104ad0 2024-03-23T23:01:39.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.837+0000 7f6045600640 1 --2- 172.21.15.12:0/1829134322 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f60280a2880 0x7f60280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:39.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.838+0000 7f6045600640 1 --2- 172.21.15.12:0/1829134322 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f60280a2880 0x7f60280a4d40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f60340058c0 tx=0x7f6034005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:39.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:39.841+0000 7f603e000640 1 -- 172.21.15.12:0/1829134322 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f603004d020 con 0x7f6040104ad0 2024-03-23T23:01:39.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:39 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/743057613' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:01:39.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:39 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1890946496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:01:39.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:39 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/743057613' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:01:39.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:39 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1890946496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:01:40.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.086+0000 7f6048000640 1 -- 172.21.15.12:0/1829134322 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60400622a0 con 0x7f6040104ad0 2024-03-23T23:01:40.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.087+0000 7f603e000640 1 -- 172.21.15.12:0/1829134322 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f60300bdc00 con 0x7f6040104ad0 2024-03-23T23:01:40.088 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:01:40.088 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T22:59:32.190756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24599,24550]},"id":1}]} 2024-03-23T23:01:40.088 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 22 2024-03-23T23:01:40.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.091+0000 7f6048000640 1 -- 172.21.15.12:0/1829134322 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f60280a2880 msgr2=0x7f60280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:40.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.091+0000 7f6048000640 1 --2- 172.21.15.12:0/1829134322 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f60280a2880 0x7f60280a4d40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f60340058c0 tx=0x7f6034005850 comp rx=0 tx=0).stop 2024-03-23T23:01:40.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.091+0000 7f6048000640 1 -- 172.21.15.12:0/1829134322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6040104ad0 msgr2=0x7f604019a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:40.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.091+0000 7f6048000640 1 --2- 172.21.15.12:0/1829134322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6040104ad0 0x7f604019a100 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6030005ab0 tx=0x7f603002c6c0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.091+0000 7f6048000640 1 -- 172.21.15.12:0/1829134322 shutdown_connections 2024-03-23T23:01:40.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.091+0000 7f6048000640 1 --2- 172.21.15.12:0/1829134322 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f60280a2880 0x7f60280a4d40 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.091+0000 7f6048000640 1 --2- 172.21.15.12:0/1829134322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6040105490 0x7f60401a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.091+0000 7f6048000640 1 --2- 172.21.15.12:0/1829134322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6040104ad0 0x7f604019a100 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.091+0000 7f6048000640 1 --2- 172.21.15.12:0/1829134322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60401038e0 0x7f6040199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.091+0000 7f6048000640 1 -- 172.21.15.12:0/1829134322 >> 172.21.15.12:0/1829134322 conn(0x7f60400fee50 msgr2=0x7f6040107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:40.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.091+0000 7f6048000640 1 -- 172.21.15.12:0/1829134322 shutdown_connections 2024-03-23T23:01:40.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.091+0000 7f6048000640 1 -- 172.21.15.12:0/1829134322 wait complete. 2024-03-23T23:01:40.110 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T23:01:40.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.336+0000 7f53b1000640 1 -- 172.21.15.12:0/832398655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53ac0ffd60 msgr2=0x7f53ac102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:40.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.336+0000 7f53b1000640 1 --2- 172.21.15.12:0/832398655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53ac0ffd60 0x7f53ac102190 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f539c009900 tx=0x7f539c02bc40 comp rx=0 tx=0).stop 2024-03-23T23:01:40.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.336+0000 7f53b1000640 1 -- 172.21.15.12:0/832398655 shutdown_connections 2024-03-23T23:01:40.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.336+0000 7f53b1000640 1 --2- 172.21.15.12:0/832398655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53ac105080 0x7f53ac105500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.336+0000 7f53b1000640 1 --2- 172.21.15.12:0/832398655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53ac1026d0 0x7f53ac104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.336+0000 7f53b1000640 1 --2- 172.21.15.12:0/832398655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53ac0ffd60 0x7f53ac102190 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.336+0000 7f53b1000640 1 -- 172.21.15.12:0/832398655 >> 172.21.15.12:0/832398655 conn(0x7f53ac0fb980 msgr2=0x7f53ac0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:40.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.336+0000 7f53b1000640 1 -- 172.21.15.12:0/832398655 shutdown_connections 2024-03-23T23:01:40.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.336+0000 7f53b1000640 1 -- 172.21.15.12:0/832398655 wait complete. 2024-03-23T23:01:40.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.337+0000 7f53b1000640 1 Processor -- start 2024-03-23T23:01:40.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.337+0000 7f53b1000640 1 -- start start 2024-03-23T23:01:40.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.338+0000 7f53b1000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53ac0ffd60 0x7f53ac19de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:40.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.338+0000 7f53b1000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53ac1026d0 0x7f53ac19e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:40.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.338+0000 7f53b1000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53ac105080 0x7f53ac1a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:40.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.338+0000 7f53b1000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac113790 con 0x7f53ac105080 2024-03-23T23:01:40.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.338+0000 7f53b1000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac113610 con 0x7f53ac0ffd60 2024-03-23T23:01:40.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.338+0000 7f53b1000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac113910 con 0x7f53ac1026d0 2024-03-23T23:01:40.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.338+0000 7f53a3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53ac1026d0 0x7f53ac19e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:40.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.338+0000 7f53abe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53ac105080 0x7f53ac1a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:40.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.338+0000 7f53a3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53ac1026d0 0x7f53ac19e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35446/0 (socket says 172.21.15.12:35446) 2024-03-23T23:01:40.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.338+0000 7f53a3e00640 1 -- 172.21.15.12:0/2947110733 learned_addr learned my addr 172.21.15.12:0/2947110733 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:40.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.338+0000 7f53ab400640 1 --2- 172.21.15.12:0/2947110733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53ac0ffd60 0x7f53ac19de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:40.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.338+0000 7f53a3e00640 1 -- 172.21.15.12:0/2947110733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53ac0ffd60 msgr2=0x7f53ac19de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:40.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.338+0000 7f53a3e00640 1 --2- 172.21.15.12:0/2947110733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53ac0ffd60 0x7f53ac19de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.338+0000 7f53a3e00640 1 -- 172.21.15.12:0/2947110733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53ac105080 msgr2=0x7f53ac1a5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:40.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.338+0000 7f53a3e00640 1 --2- 172.21.15.12:0/2947110733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53ac105080 0x7f53ac1a5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.338+0000 7f53a3e00640 1 -- 172.21.15.12:0/2947110733 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ac1a5a00 con 0x7f53ac1026d0 2024-03-23T23:01:40.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.338+0000 7f53abe00640 1 --2- 172.21.15.12:0/2947110733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53ac105080 0x7f53ac1a5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:40.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.339+0000 7f53a3e00640 1 --2- 172.21.15.12:0/2947110733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53ac1026d0 0x7f53ac19e370 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f539400d950 tx=0x7f539400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:40.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.339+0000 7f53a8c00640 1 -- 172.21.15.12:0/2947110733 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5394017070 con 0x7f53ac1026d0 2024-03-23T23:01:40.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.339+0000 7f53a8c00640 1 -- 172.21.15.12:0/2947110733 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5394009c30 con 0x7f53ac1026d0 2024-03-23T23:01:40.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.339+0000 7f53a8c00640 1 -- 172.21.15.12:0/2947110733 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539400f6b0 con 0x7f53ac1026d0 2024-03-23T23:01:40.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.339+0000 7f53b1000640 1 -- 172.21.15.12:0/2947110733 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53ac1a5cf0 con 0x7f53ac1026d0 2024-03-23T23:01:40.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.339+0000 7f53b1000640 1 -- 172.21.15.12:0/2947110733 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53ac1a6190 con 0x7f53ac1026d0 2024-03-23T23:01:40.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.340+0000 7f53b1000640 1 -- 172.21.15.12:0/2947110733 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53ac0fd470 con 0x7f53ac1026d0 2024-03-23T23:01:40.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.342+0000 7f53a8c00640 1 -- 172.21.15.12:0/2947110733 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f53940109d0 con 0x7f53ac1026d0 2024-03-23T23:01:40.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.344+0000 7f53a8c00640 1 --2- 172.21.15.12:0/2947110733 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f53900a2880 0x7f53900a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:40.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.344+0000 7f53a8c00640 1 -- 172.21.15.12:0/2947110733 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f53940ce680 con 0x7f53ac1026d0 2024-03-23T23:01:40.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.344+0000 7f53ab400640 1 --2- 172.21.15.12:0/2947110733 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f53900a2880 0x7f53900a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:40.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.345+0000 7f53ab400640 1 --2- 172.21.15.12:0/2947110733 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f53900a2880 0x7f53900a4d40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f539c02c250 tx=0x7f539c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:40.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.348+0000 7f53a8c00640 1 -- 172.21.15.12:0/2947110733 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5394009de0 con 0x7f53ac1026d0 2024-03-23T23:01:40.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.553+0000 7f53b1000640 1 -- 172.21.15.12:0/2947110733 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53ac0623a0 con 0x7f53ac1026d0 2024-03-23T23:01:40.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.555+0000 7f53a8c00640 1 -- 172.21.15.12:0/2947110733 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22695 (secure 0 0 0) 0x7f539409b6d0 con 0x7f53ac1026d0 2024-03-23T23:01:40.555 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:01:40.555 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":88,"fsid":"90f679ae-e967-11ee-95cd-87774f69a715","created":"2024-03-23T22:50:14.906952+0000","modified":"2024-03-23T23:00:24.349052+0000","last_up_change":"2024-03-23T22:58:28.907824+0000","last_in_change":"2024-03-23T22:58:08.342487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T22:54:11.831010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T22:59:25.350974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T22:59:26.390323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b70875d7-a23d-4251-9e8b-1a16d4edc2a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6803","nonce":1190079611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6805","nonce":1190079611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6809","nonce":1190079611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6807","nonce":1190079611}]},"public_addr":"172.21.15.12:6803/1190079611","cluster_addr":"172.21.15.12:6805/1190079611","heartbeat_back_addr":"172.21.15.12:6809/1190079611","heartbeat_front_addr":"172.21.15.12:6807/1190079611","state":["exists","up"]},{"osd":1,"uuid":"416e87af-db5d-41ce-84d9-731e3e07e3c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6801","nonce":1983639525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6803","nonce":1983639525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6807","nonce":1983639525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6805","nonce":1983639525}]},"public_addr":"172.21.15.173:6801/1983639525","cluster_addr":"172.21.15.173:6803/1983639525","heartbeat_back_addr":"172.21.15.173:6807/1983639525","heartbeat_front_addr":"172.21.15.173:6805/1983639525","state":["exists","up"]},{"osd":2,"uuid":"7a559c70-4c54-431b-8baf-4136b6e76493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6801","nonce":2253997387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6803","nonce":2253997387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6807","nonce":2253997387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6805","nonce":2253997387}]},"public_addr":"172.21.15.203:6801/2253997387","cluster_addr":"172.21.15.203:6803/2253997387","heartbeat_back_addr":"172.21.15.203:6807/2253997387","heartbeat_front_addr":"172.21.15.203:6805/2253997387","state":["exists","up"]},{"osd":3,"uuid":"c41cd287-a674-486c-8025-328cd36e5ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6811","nonce":1761446905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6813","nonce":1761446905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6817","nonce":1761446905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6815","nonce":1761446905}]},"public_addr":"172.21.15.12:6811/1761446905","cluster_addr":"172.21.15.12:6813/1761446905","heartbeat_back_addr":"172.21.15.12:6817/1761446905","heartbeat_front_addr":"172.21.15.12:6815/1761446905","state":["exists","up"]},{"osd":4,"uuid":"3c48f886-0b42-4a09-b31e-abccc200bb1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6809","nonce":242958272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6811","nonce":242958272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6815","nonce":242958272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6813","nonce":242958272}]},"public_addr":"172.21.15.173:6809/242958272","cluster_addr":"172.21.15.173:6811/242958272","heartbeat_back_addr":"172.21.15.173:6815/242958272","heartbeat_front_addr":"172.21.15.173:6813/242958272","state":["exists","up"]},{"osd":5,"uuid":"4a34f06f-8805-40c0-b1ab-23407b0276d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6809","nonce":4270966092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6811","nonce":4270966092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6815","nonce":4270966092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6813","nonce":4270966092}]},"public_addr":"172.21.15.203:6809/4270966092","cluster_addr":"172.21.15.203:6811/4270966092","heartbeat_back_addr":"172.21.15.203:6815/4270966092","heartbeat_front_addr":"172.21.15.203:6813/4270966092","state":["exists","up"]},{"osd":6,"uuid":"26ddc663-5623-48c9-80d1-f8b968765d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6819","nonce":3606839295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6821","nonce":3606839295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6825","nonce":3606839295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6823","nonce":3606839295}]},"public_addr":"172.21.15.12:6819/3606839295","cluster_addr":"172.21.15.12:6821/3606839295","heartbeat_back_addr":"172.21.15.12:6825/3606839295","heartbeat_front_addr":"172.21.15.12:6823/3606839295","state":["exists","up"]},{"osd":7,"uuid":"17dc9417-71cd-4576-aa08-933212ec55e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6817","nonce":1241504346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6819","nonce":1241504346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6823","nonce":1241504346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6821","nonce":1241504346}]},"public_addr":"172.21.15.173:6817/1241504346","cluster_addr":"172.21.15.173:6819/1241504346","heartbeat_back_addr":"172.21.15.173:6823/1241504346","heartbeat_front_addr":"172.21.15.173:6821/1241504346","state":["exists","up"]},{"osd":8,"uuid":"90a2b63a-471d-43b8-ac18-bb1f16778d18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6817","nonce":1516996192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6819","nonce":1516996192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6823","nonce":1516996192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6821","nonce":1516996192}]},"public_addr":"172.21.15.203:6817/1516996192","cluster_addr":"172.21.15.203:6819/1516996192","heartbeat_back_addr":"172.21.15.203:6823/1516996192","heartbeat_front_addr":"172.21.15.203:6821/1516996192","state":["exists","up"]},{"osd":9,"uuid":"6e1ca504-40b4-49d0-b245-7946181abfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6827","nonce":1175260774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6829","nonce":1175260774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6833","nonce":1175260774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6831","nonce":1175260774}]},"public_addr":"172.21.15.12:6827/1175260774","cluster_addr":"172.21.15.12:6829/1175260774","heartbeat_back_addr":"172.21.15.12:6833/1175260774","heartbeat_front_addr":"172.21.15.12:6831/1175260774","state":["exists","up"]},{"osd":10,"uuid":"09a595ed-5614-4386-8e61-0170af66e01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6825","nonce":3303506808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6827","nonce":3303506808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6831","nonce":3303506808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6829","nonce":3303506808}]},"public_addr":"172.21.15.173:6825/3303506808","cluster_addr":"172.21.15.173:6827/3303506808","heartbeat_back_addr":"172.21.15.173:6831/3303506808","heartbeat_front_addr":"172.21.15.173:6829/3303506808","state":["exists","up"]},{"osd":11,"uuid":"9ef1d595-a977-4ba7-9c6b-d2b3295f20b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6825","nonce":2323604506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6827","nonce":2323604506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6831","nonce":2323604506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6829","nonce":2323604506}]},"public_addr":"172.21.15.203:6825/2323604506","cluster_addr":"172.21.15.203:6827/2323604506","heartbeat_back_addr":"172.21.15.203:6831/2323604506","heartbeat_front_addr":"172.21.15.203:6829/2323604506","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:18.469038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:42.752628+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:07.069493+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:41.962478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:08.363104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:33.341336+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:07.286541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:34.243302+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:59.749699+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:57:33.822913+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:00.737269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:26.599854+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/4182783120":"2024-03-24T22:51:23.113201+0000","172.21.15.12:6800/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1900938637":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/3414861652":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6801/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/2580541477":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6800/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/501058839":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1192554716":"2024-03-24T22:50:48.401223+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T23:01:40.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.557+0000 7f53b1000640 1 -- 172.21.15.12:0/2947110733 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f53900a2880 msgr2=0x7f53900a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:40.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.557+0000 7f53b1000640 1 --2- 172.21.15.12:0/2947110733 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f53900a2880 0x7f53900a4d40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f539c02c250 tx=0x7f539c005fd0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.557+0000 7f53b1000640 1 -- 172.21.15.12:0/2947110733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53ac1026d0 msgr2=0x7f53ac19e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:40.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.557+0000 7f53b1000640 1 --2- 172.21.15.12:0/2947110733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53ac1026d0 0x7f53ac19e370 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f539400d950 tx=0x7f539400de10 comp rx=0 tx=0).stop 2024-03-23T23:01:40.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.558+0000 7f53b1000640 1 -- 172.21.15.12:0/2947110733 shutdown_connections 2024-03-23T23:01:40.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.558+0000 7f53b1000640 1 --2- 172.21.15.12:0/2947110733 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f53900a2880 0x7f53900a4d40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.558+0000 7f53b1000640 1 --2- 172.21.15.12:0/2947110733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53ac105080 0x7f53ac1a5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.558+0000 7f53b1000640 1 --2- 172.21.15.12:0/2947110733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53ac1026d0 0x7f53ac19e370 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.558+0000 7f53b1000640 1 --2- 172.21.15.12:0/2947110733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53ac0ffd60 0x7f53ac19de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.558+0000 7f53b1000640 1 -- 172.21.15.12:0/2947110733 >> 172.21.15.12:0/2947110733 conn(0x7f53ac0fb980 msgr2=0x7f53ac103390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:40.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.558+0000 7f53b1000640 1 -- 172.21.15.12:0/2947110733 shutdown_connections 2024-03-23T23:01:40.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.558+0000 7f53b1000640 1 -- 172.21.15.12:0/2947110733 wait complete. 2024-03-23T23:01:40.576 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-23T22:59:26.906237+0000', 'modified': '2024-03-23T22:59:32.190756+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24599, 'mds_1': 24554, 'mds_2': 24550}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24517': {'gid': 24517, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6835/915277278', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6834', 'nonce': 915277278}, {'type': 'v1', 'addr': '172.21.15.203:6835', 'nonce': 915277278}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24521': {'gid': 24521, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6835/2648278843', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6834', 'nonce': 2648278843}, {'type': 'v1', 'addr': '172.21.15.173:6835', 'nonce': 2648278843}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24529': {'gid': 24529, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6837/438676939', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6836', 'nonce': 438676939}, {'type': 'v1', 'addr': '172.21.15.203:6837', 'nonce': 438676939}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24550': {'gid': 24550, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.203:6839/852477167', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6838', 'nonce': 852477167}, {'type': 'v1', 'addr': '172.21.15.203:6839', 'nonce': 852477167}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24554': {'gid': 24554, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.173:6837/2648089141', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6836', 'nonce': 2648089141}, {'type': 'v1', 'addr': '172.21.15.173:6837', 'nonce': 2648089141}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24599': {'gid': 24599, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.173:6839/1645433214', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6838', 'nonce': 1645433214}, {'type': 'v1', 'addr': '172.21.15.173:6839', 'nonce': 1645433214}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24599, 'qdb_cluster': [24554, 24599, 24550]} 2024-03-23T23:01:40.576 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-23T23:01:40.576 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-23T23:01:40.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.796+0000 7fba2fa00640 1 -- 172.21.15.12:0/3092063750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba28103260 msgr2=0x7fba28109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:40.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.796+0000 7fba2fa00640 1 --2- 172.21.15.12:0/3092063750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba28103260 0x7fba28109af0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fba20009ca0 tx=0x7fba2002c0f0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.796+0000 7fba2fa00640 1 -- 172.21.15.12:0/3092063750 shutdown_connections 2024-03-23T23:01:40.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.796+0000 7fba2fa00640 1 --2- 172.21.15.12:0/3092063750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba28103260 0x7fba28109af0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.796+0000 7fba2fa00640 1 --2- 172.21.15.12:0/3092063750 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba281028a0 0x7fba28102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.796+0000 7fba2fa00640 1 --2- 172.21.15.12:0/3092063750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba281016b0 0x7fba28101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.796+0000 7fba2fa00640 1 -- 172.21.15.12:0/3092063750 >> 172.21.15.12:0/3092063750 conn(0x7fba280fcc40 msgr2=0x7fba280ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:40.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.796+0000 7fba2fa00640 1 -- 172.21.15.12:0/3092063750 shutdown_connections 2024-03-23T23:01:40.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.796+0000 7fba2fa00640 1 -- 172.21.15.12:0/3092063750 wait complete. 2024-03-23T23:01:40.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.797+0000 7fba2fa00640 1 Processor -- start 2024-03-23T23:01:40.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.798+0000 7fba2fa00640 1 -- start start 2024-03-23T23:01:40.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.798+0000 7fba2fa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba281016b0 0x7fba28199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:40.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.798+0000 7fba2fa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba281028a0 0x7fba2819a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:40.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.798+0000 7fba2fa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba28103260 0x7fba281a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:40.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.798+0000 7fba2fa00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba2810d310 con 0x7fba28103260 2024-03-23T23:01:40.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.798+0000 7fba2fa00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba2810d190 con 0x7fba281016b0 2024-03-23T23:01:40.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.798+0000 7fba2fa00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba2810d490 con 0x7fba281028a0 2024-03-23T23:01:40.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.798+0000 7fba2d000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba281016b0 0x7fba28199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:40.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.798+0000 7fba2d000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba281016b0 0x7fba28199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36272/0 (socket says 172.21.15.12:36272) 2024-03-23T23:01:40.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.798+0000 7fba2d000640 1 -- 172.21.15.12:0/3243300424 learned_addr learned my addr 172.21.15.12:0/3243300424 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:40.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.798+0000 7fba1fe00640 1 --2- 172.21.15.12:0/3243300424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba281028a0 0x7fba2819a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:40.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.798+0000 7fba2da00640 1 --2- 172.21.15.12:0/3243300424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba28103260 0x7fba281a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:40.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.798+0000 7fba2d000640 1 -- 172.21.15.12:0/3243300424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba281028a0 msgr2=0x7fba2819a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:40.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.798+0000 7fba2d000640 1 --2- 172.21.15.12:0/3243300424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba281028a0 0x7fba2819a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.799+0000 7fba2d000640 1 -- 172.21.15.12:0/3243300424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba28103260 msgr2=0x7fba281a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:40.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.799+0000 7fba2d000640 1 --2- 172.21.15.12:0/3243300424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba28103260 0x7fba281a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.799+0000 7fba2d000640 1 -- 172.21.15.12:0/3243300424 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba281a1850 con 0x7fba281016b0 2024-03-23T23:01:40.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.799+0000 7fba2da00640 1 --2- 172.21.15.12:0/3243300424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba28103260 0x7fba281a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:01:40.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.799+0000 7fba2d000640 1 --2- 172.21.15.12:0/3243300424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba281016b0 0x7fba28199c40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fba1800e9c0 tx=0x7fba1800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:40.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.799+0000 7fba26000640 1 -- 172.21.15.12:0/3243300424 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1800cde0 con 0x7fba281016b0 2024-03-23T23:01:40.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.799+0000 7fba26000640 1 -- 172.21.15.12:0/3243300424 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba18010910 con 0x7fba281016b0 2024-03-23T23:01:40.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.799+0000 7fba2fa00640 1 -- 172.21.15.12:0/3243300424 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba281a1b40 con 0x7fba281016b0 2024-03-23T23:01:40.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.799+0000 7fba26000640 1 -- 172.21.15.12:0/3243300424 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba18018a00 con 0x7fba281016b0 2024-03-23T23:01:40.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.799+0000 7fba2fa00640 1 -- 172.21.15.12:0/3243300424 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba281a20e0 con 0x7fba281016b0 2024-03-23T23:01:40.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.802+0000 7fba327c4740 1 Processor -- start 2024-03-23T23:01:40.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.802+0000 7fba327c4740 1 -- start start 2024-03-23T23:01:40.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.802+0000 7fba26000640 1 -- 172.21.15.12:0/3243300424 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fba1801f020 con 0x7fba281016b0 2024-03-23T23:01:40.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.803+0000 7fba327c4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555af42a7ba0 0x555af42a7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:40.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.803+0000 7fba327c4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555af42a8590 0x555af42ad090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:40.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.803+0000 7fba327c4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555af42ad700 0x555af42afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:40.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.803+0000 7fba327c4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x555af420cc90 con 0x555af42a7ba0 2024-03-23T23:01:40.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.803+0000 7fba327c4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x555af41b5980 con 0x555af42a8590 2024-03-23T23:01:40.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.803+0000 7fba327c4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x555af4208d30 con 0x555af42ad700 2024-03-23T23:01:40.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.803+0000 7fba2da00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555af42a7ba0 0x555af42a7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:40.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.803+0000 7fba2d000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555af42a8590 0x555af42ad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:40.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.803+0000 7fba1fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555af42ad700 0x555af42afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:40.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.803+0000 7fba2da00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555af42a7ba0 0x555af42a7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38922/0 (socket says 172.21.15.12:38922) 2024-03-23T23:01:40.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.803+0000 7fba2da00640 1 -- 172.21.15.12:0/1603090263 learned_addr learned my addr 172.21.15.12:0/1603090263 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:40.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:40 smithi012 ceph-mon[31695]: pgmap v386: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:01:40.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:40 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/235874400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:01:40.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:40 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1829134322' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:01:40.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.803+0000 7fba1fe00640 1 -- 172.21.15.12:0/1603090263 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555af42a8590 msgr2=0x555af42ad090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:40.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.803+0000 7fba1fe00640 1 --2- 172.21.15.12:0/1603090263 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555af42a8590 0x555af42ad090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.803+0000 7fba1fe00640 1 -- 172.21.15.12:0/1603090263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555af42a7ba0 msgr2=0x555af42a7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.804+0000 7fba1fe00640 1 --2- 172.21.15.12:0/1603090263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555af42a7ba0 0x555af42a7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.804+0000 7fba1fe00640 1 -- 172.21.15.12:0/1603090263 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555af41cd510 con 0x555af42ad700 2024-03-23T23:01:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.804+0000 7fba2da00640 1 --2- 172.21.15.12:0/1603090263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555af42a7ba0 0x555af42a7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.804+0000 7fba1fe00640 1 --2- 172.21.15.12:0/1603090263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555af42ad700 0x555af42afb90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fba1000bbf0 tx=0x7fba100107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a9c7798226fb3a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.804+0000 7fba1e000640 1 -- 172.21.15.12:0/1603090263 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba10004030 con 0x555af42ad700 2024-03-23T23:01:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.804+0000 7fba1e000640 1 -- 172.21.15.12:0/1603090263 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba100041d0 con 0x555af42ad700 2024-03-23T23:01:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.804+0000 7fba1e000640 1 -- 172.21.15.12:0/1603090263 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba100116c0 con 0x555af42ad700 2024-03-23T23:01:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.804+0000 7fba26000640 1 --2- 172.21.15.12:0/3243300424 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fba0c0a2880 0x7fba0c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.805+0000 7fba26000640 1 -- 172.21.15.12:0/3243300424 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fba180ced50 con 0x7fba281016b0 2024-03-23T23:01:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.805+0000 7fba2da00640 1 --2- 172.21.15.12:0/3243300424 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fba0c0a2880 0x7fba0c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.805+0000 7fba327c4740 1 -- 172.21.15.12:0/1603090263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555af42ad700 msgr2=0x555af42afb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.805+0000 7fba327c4740 1 --2- 172.21.15.12:0/1603090263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555af42ad700 0x555af42afb90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fba1000bbf0 tx=0x7fba100107a0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.805+0000 7fba327c4740 1 -- 172.21.15.12:0/1603090263 shutdown_connections 2024-03-23T23:01:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.805+0000 7fba327c4740 1 --2- 172.21.15.12:0/1603090263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555af42ad700 0x555af42afb90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.805+0000 7fba327c4740 1 --2- 172.21.15.12:0/1603090263 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555af42a8590 0x555af42ad090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.805+0000 7fba327c4740 1 --2- 172.21.15.12:0/1603090263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555af42a7ba0 0x555af42a7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.805+0000 7fba327c4740 1 -- 172.21.15.12:0/1603090263 >> 172.21.15.12:0/1603090263 conn(0x555af42a3110 msgr2=0x555af42a3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:40.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.805+0000 7fba327c4740 1 -- 172.21.15.12:0/1603090263 shutdown_connections 2024-03-23T23:01:40.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.805+0000 7fba2da00640 1 --2- 172.21.15.12:0/3243300424 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fba0c0a2880 0x7fba0c0a4d40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fba20004af0 tx=0x7fba20005d80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:40.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.805+0000 7fba327c4740 1 -- 172.21.15.12:0/1603090263 wait complete. 2024-03-23T23:01:40.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.806+0000 7fba327c4740 1 -- start start 2024-03-23T23:01:40.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.806+0000 7fba327c4740 1 Processor -- start 2024-03-23T23:01:40.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.806+0000 7fba327c4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555af42a7ba0 0x555af42b2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:40.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.806+0000 7fba327c4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555af42a8590 0x555af42b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:40.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.806+0000 7fba327c4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555af42ad700 0x555af42b9680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:40.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.806+0000 7fba327c4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x555af42b06b0 con 0x555af42a7ba0 2024-03-23T23:01:40.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.806+0000 7fba327c4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x555af41c8dd0 con 0x555af42ad700 2024-03-23T23:01:40.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.806+0000 7fba327c4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x555af42b0830 con 0x555af42a8590 2024-03-23T23:01:40.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.807+0000 7fba327c4740 10 client.0 _ll_register_callbacks cb 0x555af3f20ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:01:40.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.807+0000 7fba2da00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555af42ad700 0x555af42b9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:40.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.807+0000 7fba2da00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555af42ad700 0x555af42b9680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36294/0 (socket says 172.21.15.12:36294) 2024-03-23T23:01:40.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.807+0000 7fba2da00640 1 -- 172.21.15.12:0/1844616228 learned_addr learned my addr 172.21.15.12:0/1844616228 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.807+0000 7fba2d000640 1 --2- 172.21.15.12:0/1844616228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555af42a7ba0 0x555af42b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.807+0000 7fba1fe00640 1 --2- 172.21.15.12:0/1844616228 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555af42a8590 0x555af42b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.807+0000 7fba2da00640 1 -- 172.21.15.12:0/1844616228 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555af42a8590 msgr2=0x555af42b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.807+0000 7fba2da00640 1 --2- 172.21.15.12:0/1844616228 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555af42a8590 0x555af42b26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.807+0000 7fba2da00640 1 -- 172.21.15.12:0/1844616228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555af42a7ba0 msgr2=0x555af42b2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.807+0000 7fba2da00640 1 --2- 172.21.15.12:0/1844616228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555af42a7ba0 0x555af42b2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.807+0000 7fba2da00640 1 -- 172.21.15.12:0/1844616228 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555af3f212d0 con 0x555af42ad700 2024-03-23T23:01:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.807+0000 7fba2da00640 1 --2- 172.21.15.12:0/1844616228 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555af42ad700 0x555af42b9680 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fba20037040 tx=0x7fba200095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c9bf2f63269d4c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.807+0000 7fba16a00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:01:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.808+0000 7fba16a00640 1 -- 172.21.15.12:0/1844616228 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba20052060 con 0x555af42ad700 2024-03-23T23:01:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.808+0000 7fba16a00640 1 -- 172.21.15.12:0/1844616228 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fba20003b70 con 0x555af42ad700 2024-03-23T23:01:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.808+0000 7fba327c4740 1 -- 172.21.15.12:0/1844616228 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fba100041d0 con 0x555af42ad700 2024-03-23T23:01:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.808+0000 7fba16a00640 1 -- 172.21.15.12:0/1844616228 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba20039040 con 0x555af42ad700 2024-03-23T23:01:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.808+0000 7fba327c4740 10 client.25151 fetch_fsmap learned FSMap version 22 2024-03-23T23:01:40.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.808+0000 7fba327c4740 1 -- 172.21.15.12:0/1844616228 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x555af3feb6b0 con 0x555af42ad700 2024-03-23T23:01:40.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.808+0000 7fba16a00640 1 -- 172.21.15.12:0/1844616228 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba100041d0 con 0x555af42ad700 2024-03-23T23:01:40.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.808+0000 7fba16a00640 1 -- 172.21.15.12:0/1844616228 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fba2004d4e0 con 0x555af42ad700 2024-03-23T23:01:40.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.808+0000 7fba327c4740 10 client.25151 fetch_fsmap finished waiting for FSMap version 22 2024-03-23T23:01:40.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.808+0000 7fba327c4740 10 client.25151 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-23T23:01:40.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.808+0000 7fba327c4740 20 client.25151 populate_metadata read hostname 'smithi012' 2024-03-23T23:01:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.808+0000 7fba327c4740 1 --2- 172.21.15.12:0/1844616228 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x555af42be930 0x555af42bd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.809+0000 7fba327c4740 4 client.25151 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:01:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.809+0000 7fba327c4740 1 -- 172.21.15.12:0/1844616228 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555af41ae520 con 0x555af42be930 2024-03-23T23:01:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.809+0000 7fba2d000640 1 --2- 172.21.15.12:0/1844616228 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x555af42be930 0x555af42bd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.809+0000 7fba2d000640 1 --2- 172.21.15.12:0/1844616228 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x555af42be930 0x555af42bd960 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6ae085b34952f49d server_cookie=6a4d2a753709da1a in_seq=0 out_seq=0 2024-03-23T23:01:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.809+0000 7fba16a00640 10 client.25151 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:01:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.810+0000 7fba16a00640 1 -- 172.21.15.12:0/1844616228 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555af41ae520 con 0x555af42be930 2024-03-23T23:01:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.810+0000 7fba16a00640 10 client.25151 handle_command_reply: tid=0 2024-03-23T23:01:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.810+0000 7fba327c4740 1 client.25151 shutdown 2024-03-23T23:01:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.810+0000 7fba327c4740 1 -- 172.21.15.12:0/1844616228 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555af42ad700 msgr2=0x555af42b9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.810+0000 7fba327c4740 1 --2- 172.21.15.12:0/1844616228 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555af42ad700 0x555af42b9680 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fba20037040 tx=0x7fba200095c0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.810+0000 7fba327c4740 1 -- 172.21.15.12:0/1844616228 shutdown_connections 2024-03-23T23:01:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.810+0000 7fba327c4740 1 --2- 172.21.15.12:0/1844616228 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x555af42be930 0x555af42bd960 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.810+0000 7fba327c4740 1 --2- 172.21.15.12:0/1844616228 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555af42ad700 0x555af42b9680 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.810+0000 7fba16a00640 0 client.25151 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:01:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.810+0000 7fba327c4740 1 --2- 172.21.15.12:0/1844616228 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555af42a8590 0x555af42b26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.810+0000 7fba327c4740 1 --2- 172.21.15.12:0/1844616228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555af42a7ba0 0x555af42b2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.810+0000 7fba327c4740 1 -- 172.21.15.12:0/1844616228 >> 172.21.15.12:0/1844616228 conn(0x555af42a3110 msgr2=0x555af42a2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:40.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.810+0000 7fba327c4740 1 -- 172.21.15.12:0/1844616228 shutdown_connections 2024-03-23T23:01:40.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.810+0000 7fba327c4740 1 -- 172.21.15.12:0/1844616228 wait complete. 2024-03-23T23:01:40.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.810+0000 7fba327c4740 20 client.25151 trim_cache size 0 max 16384 2024-03-23T23:01:40.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.828+0000 7fba327c4740 1 Processor -- start 2024-03-23T23:01:40.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.828+0000 7fba327c4740 1 -- start start 2024-03-23T23:01:40.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.828+0000 7fba327c4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555af42a7ba0 0x555af42a1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:40.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.828+0000 7fba327c4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555af42a8590 0x555af42b9640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:40.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.828+0000 7fba327c4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555af42ad700 0x555af42a1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:40.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.829+0000 7fba327c4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x555af42be150 con 0x555af42ad700 2024-03-23T23:01:40.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.829+0000 7fba327c4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x555af42be7b0 con 0x555af42a8590 2024-03-23T23:01:40.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.829+0000 7fba327c4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x555af42bee10 con 0x555af42a7ba0 2024-03-23T23:01:40.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.829+0000 7fba2d000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555af42a7ba0 0x555af42a1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:40.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.829+0000 7fba2d000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555af42a7ba0 0x555af42a1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35482/0 (socket says 172.21.15.12:35482) 2024-03-23T23:01:40.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.829+0000 7fba1fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555af42a8590 0x555af42b9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:40.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.829+0000 7fba2d000640 1 -- 172.21.15.12:0/1611675611 learned_addr learned my addr 172.21.15.12:0/1611675611 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:40.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.829+0000 7fba2da00640 1 --2- 172.21.15.12:0/1611675611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555af42ad700 0x555af42a1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:40.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.829+0000 7fba1fe00640 1 -- 172.21.15.12:0/1611675611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555af42a7ba0 msgr2=0x555af42a1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:40.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.829+0000 7fba1fe00640 1 --2- 172.21.15.12:0/1611675611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555af42a7ba0 0x555af42a1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.829+0000 7fba1fe00640 1 -- 172.21.15.12:0/1611675611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555af42ad700 msgr2=0x555af42a1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:40.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.829+0000 7fba1fe00640 1 --2- 172.21.15.12:0/1611675611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555af42ad700 0x555af42a1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.829+0000 7fba1fe00640 1 -- 172.21.15.12:0/1611675611 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555af42b09f0 con 0x555af42a8590 2024-03-23T23:01:40.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.829+0000 7fba1fe00640 1 --2- 172.21.15.12:0/1611675611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555af42a8590 0x555af42b9640 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fba10000c00 tx=0x7fba1000b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17a634db6317f3dc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:40.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.830+0000 7fba1d600640 1 -- 172.21.15.12:0/1611675611 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1002a070 con 0x555af42a8590 2024-03-23T23:01:40.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.830+0000 7fba1d600640 1 -- 172.21.15.12:0/1611675611 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba10005520 con 0x555af42a8590 2024-03-23T23:01:40.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.830+0000 7fba1d600640 1 -- 172.21.15.12:0/1611675611 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba10025470 con 0x555af42a8590 2024-03-23T23:01:40.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.830+0000 7fba327c4740 1 -- 172.21.15.12:0/1611675611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555af42a8590 msgr2=0x555af42b9640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:40.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.830+0000 7fba327c4740 1 --2- 172.21.15.12:0/1611675611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555af42a8590 0x555af42b9640 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fba10000c00 tx=0x7fba1000b820 comp rx=0 tx=0).stop 2024-03-23T23:01:40.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.831+0000 7fba327c4740 1 -- 172.21.15.12:0/1611675611 shutdown_connections 2024-03-23T23:01:40.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.831+0000 7fba327c4740 1 --2- 172.21.15.12:0/1611675611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555af42ad700 0x555af42a1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.831+0000 7fba327c4740 1 --2- 172.21.15.12:0/1611675611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555af42a8590 0x555af42b9640 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.831+0000 7fba327c4740 1 --2- 172.21.15.12:0/1611675611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555af42a7ba0 0x555af42a1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.831+0000 7fba327c4740 1 -- 172.21.15.12:0/1611675611 >> 172.21.15.12:0/1611675611 conn(0x555af42a3110 msgr2=0x555af42a34f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:40.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.831+0000 7fba327c4740 1 -- 172.21.15.12:0/1611675611 shutdown_connections 2024-03-23T23:01:40.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.831+0000 7fba327c4740 1 -- 172.21.15.12:0/1611675611 wait complete. 2024-03-23T23:01:40.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.831+0000 7fba327c4740 1 -- start start 2024-03-23T23:01:40.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.831+0000 7fba327c4740 1 Processor -- start 2024-03-23T23:01:40.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.832+0000 7fba327c4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555af42a7ba0 0x555af42b9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:40.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.832+0000 7fba327c4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555af42a8590 0x555af42b5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:40.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.832+0000 7fba327c4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555af42ad700 0x555af42b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:40.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.832+0000 7fba327c4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x555af42bfad0 con 0x555af42a7ba0 2024-03-23T23:01:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.832+0000 7fba327c4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x555af42bf470 con 0x555af42a8590 2024-03-23T23:01:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.832+0000 7fba327c4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x555af42c0130 con 0x555af42ad700 2024-03-23T23:01:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.832+0000 7fba327c4740 10 client.0 _ll_register_callbacks cb 0x7fba18005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:01:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.832+0000 7fba1fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555af42a8590 0x555af42b5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.832+0000 7fba2d000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555af42a7ba0 0x555af42b9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.832+0000 7fba1fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555af42a8590 0x555af42b5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36304/0 (socket says 172.21.15.12:36304) 2024-03-23T23:01:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.832+0000 7fba1fe00640 1 -- 172.21.15.12:0/2269280493 learned_addr learned my addr 172.21.15.12:0/2269280493 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.832+0000 7fba2da00640 1 --2- 172.21.15.12:0/2269280493 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555af42ad700 0x555af42b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.832+0000 7fba1fe00640 1 -- 172.21.15.12:0/2269280493 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555af42ad700 msgr2=0x555af42b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.832+0000 7fba1fe00640 1 --2- 172.21.15.12:0/2269280493 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555af42ad700 0x555af42b6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.832+0000 7fba1fe00640 1 -- 172.21.15.12:0/2269280493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555af42a7ba0 msgr2=0x555af42b9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.832+0000 7fba1fe00640 1 --2- 172.21.15.12:0/2269280493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555af42a7ba0 0x555af42b9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.832+0000 7fba1fe00640 1 -- 172.21.15.12:0/2269280493 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555af42b9790 con 0x555af42a8590 2024-03-23T23:01:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.833+0000 7fba1fe00640 1 --2- 172.21.15.12:0/2269280493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555af42a8590 0x555af42b5c00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fba10011010 tx=0x7fba10003e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d532439fd925d76d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.833+0000 7fba16a00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:01:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.833+0000 7fba16a00640 1 -- 172.21.15.12:0/2269280493 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1002a070 con 0x555af42a8590 2024-03-23T23:01:40.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.833+0000 7fba327c4740 1 -- 172.21.15.12:0/2269280493 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fba10005520 con 0x555af42a8590 2024-03-23T23:01:40.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.833+0000 7fba16a00640 1 -- 172.21.15.12:0/2269280493 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fba100050b0 con 0x555af42a8590 2024-03-23T23:01:40.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.833+0000 7fba16a00640 1 -- 172.21.15.12:0/2269280493 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba100052b0 con 0x555af42a8590 2024-03-23T23:01:40.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.833+0000 7fba327c4740 10 client.25157 fetch_fsmap learned FSMap version 22 2024-03-23T23:01:40.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.833+0000 7fba327c4740 1 -- 172.21.15.12:0/2269280493 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x555af431a850 con 0x555af42a8590 2024-03-23T23:01:40.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.833+0000 7fba16a00640 1 -- 172.21.15.12:0/2269280493 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba10005520 con 0x555af42a8590 2024-03-23T23:01:40.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.834+0000 7fba16a00640 1 -- 172.21.15.12:0/2269280493 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fba10025a50 con 0x555af42a8590 2024-03-23T23:01:40.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.834+0000 7fba327c4740 10 client.25157 fetch_fsmap finished waiting for FSMap version 22 2024-03-23T23:01:40.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.834+0000 7fba327c4740 10 client.25157 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-23T23:01:40.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.834+0000 7fba327c4740 20 client.25157 populate_metadata read hostname 'smithi012' 2024-03-23T23:01:40.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.834+0000 7fba327c4740 1 --2- 172.21.15.12:0/2269280493 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x555af431e6a0 0x555af431aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:40.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.834+0000 7fba327c4740 4 client.25157 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-23T23:01:40.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.834+0000 7fba327c4740 1 -- 172.21.15.12:0/2269280493 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555af421a1d0 con 0x555af431e6a0 2024-03-23T23:01:40.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.834+0000 7fba2d000640 1 --2- 172.21.15.12:0/2269280493 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x555af431e6a0 0x555af431aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:40.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.834+0000 7fba2d000640 1 --2- 172.21.15.12:0/2269280493 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x555af431e6a0 0x555af431aa20 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1e31b88a2bce9829 server_cookie=6fb03300f309fc19 in_seq=0 out_seq=0 2024-03-23T23:01:40.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.834+0000 7fba16a00640 10 client.25157 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:01:40.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.835+0000 7fba16a00640 1 -- 172.21.15.12:0/2269280493 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x555af421a1d0 con 0x555af431e6a0 2024-03-23T23:01:40.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.835+0000 7fba16a00640 10 client.25157 handle_command_reply: tid=0 2024-03-23T23:01:40.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.835+0000 7fba327c4740 1 client.25157 shutdown 2024-03-23T23:01:40.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.835+0000 7fba327c4740 1 -- 172.21.15.12:0/2269280493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555af42a8590 msgr2=0x555af42b5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:40.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.835+0000 7fba327c4740 1 --2- 172.21.15.12:0/2269280493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555af42a8590 0x555af42b5c00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fba10011010 tx=0x7fba10003e70 comp rx=0 tx=0).stop 2024-03-23T23:01:40.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.835+0000 7fba327c4740 1 -- 172.21.15.12:0/2269280493 shutdown_connections 2024-03-23T23:01:40.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.835+0000 7fba327c4740 1 --2- 172.21.15.12:0/2269280493 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x555af431e6a0 0x555af431aa20 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.835+0000 7fba327c4740 1 --2- 172.21.15.12:0/2269280493 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555af42ad700 0x555af42b6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.835+0000 7fba327c4740 1 --2- 172.21.15.12:0/2269280493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555af42a8590 0x555af42b5c00 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.835+0000 7fba16a00640 0 client.25157 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:01:40.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.835+0000 7fba327c4740 1 --2- 172.21.15.12:0/2269280493 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555af42a7ba0 0x555af42b9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.835+0000 7fba327c4740 1 -- 172.21.15.12:0/2269280493 >> 172.21.15.12:0/2269280493 conn(0x555af42a3110 msgr2=0x555af42a6370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:40.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.835+0000 7fba327c4740 1 -- 172.21.15.12:0/2269280493 shutdown_connections 2024-03-23T23:01:40.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.835+0000 7fba327c4740 1 -- 172.21.15.12:0/2269280493 wait complete. 2024-03-23T23:01:40.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.835+0000 7fba327c4740 20 client.25157 trim_cache size 0 max 16384 2024-03-23T23:01:40.839 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:01:40.839 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "90f679ae-e967-11ee-95cd-87774f69a715", 2024-03-23T23:01:40.839 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 2, 2024-03-23T23:01:40.839 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 24550, 2024-03-23T23:01:40.839 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-23T23:01:40.839 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-23T23:01:40.839 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-23T23:01:40.839 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 129.34679679300001, 2024-03-23T23:01:40.839 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 22, 2024-03-23T23:01:40.840 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 88, 2024-03-23T23:01:40.840 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 86, 2024-03-23T23:01:40.840 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 151.33765012699999 2024-03-23T23:01:40.840 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:01:40.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.836+0000 7fba2fa00640 1 -- 172.21.15.12:0/3243300424 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fba0c0a2880 msgr2=0x7fba0c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:40.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.836+0000 7fba2fa00640 1 --2- 172.21.15.12:0/3243300424 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fba0c0a2880 0x7fba0c0a4d40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fba20004af0 tx=0x7fba20005d80 comp rx=0 tx=0).stop 2024-03-23T23:01:40.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.836+0000 7fba2fa00640 1 -- 172.21.15.12:0/3243300424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba281016b0 msgr2=0x7fba28199c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:40.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.836+0000 7fba2fa00640 1 --2- 172.21.15.12:0/3243300424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba281016b0 0x7fba28199c40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fba1800e9c0 tx=0x7fba1800ee80 comp rx=0 tx=0).stop 2024-03-23T23:01:40.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.836+0000 7fba2fa00640 1 -- 172.21.15.12:0/3243300424 shutdown_connections 2024-03-23T23:01:40.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.836+0000 7fba2fa00640 1 --2- 172.21.15.12:0/3243300424 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fba0c0a2880 0x7fba0c0a4d40 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.836+0000 7fba2fa00640 1 --2- 172.21.15.12:0/3243300424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba28103260 0x7fba281a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.836+0000 7fba2fa00640 1 --2- 172.21.15.12:0/3243300424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba281028a0 0x7fba2819a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.836+0000 7fba2fa00640 1 --2- 172.21.15.12:0/3243300424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba281016b0 0x7fba28199c40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:40.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.836+0000 7fba2fa00640 1 -- 172.21.15.12:0/3243300424 >> 172.21.15.12:0/3243300424 conn(0x7fba280fcc40 msgr2=0x7fba280738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:40.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.836+0000 7fba2fa00640 1 -- 172.21.15.12:0/3243300424 shutdown_connections 2024-03-23T23:01:40.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:40.836+0000 7fba2fa00640 1 -- 172.21.15.12:0/3243300424 wait complete. 2024-03-23T23:01:40.855 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-23T23:01:40.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:40 smithi173 ceph-mon[36116]: pgmap v386: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:01:40.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:40 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/235874400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:01:40.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:40 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1829134322' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:01:40.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:40 smithi203 ceph-mon[36693]: pgmap v386: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:01:40.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:40 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/235874400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:01:40.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:40 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1829134322' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:01:41.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.068+0000 7fc2b7200640 1 -- 172.21.15.12:0/1830251538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2b0105490 msgr2=0x7fc2b010bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:41.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.068+0000 7fc2b7200640 1 --2- 172.21.15.12:0/1830251538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2b0105490 0x7fc2b010bd20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc2a8009ca0 tx=0x7fc2a802c100 comp rx=0 tx=0).stop 2024-03-23T23:01:41.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.069+0000 7fc2b7200640 1 -- 172.21.15.12:0/1830251538 shutdown_connections 2024-03-23T23:01:41.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.069+0000 7fc2b7200640 1 --2- 172.21.15.12:0/1830251538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2b0105490 0x7fc2b010bd20 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.069+0000 7fc2b7200640 1 --2- 172.21.15.12:0/1830251538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b0104ad0 0x7fc2b0104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.069+0000 7fc2b7200640 1 --2- 172.21.15.12:0/1830251538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2b01038e0 0x7fc2b0103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.069+0000 7fc2b7200640 1 -- 172.21.15.12:0/1830251538 >> 172.21.15.12:0/1830251538 conn(0x7fc2b00fee50 msgr2=0x7fc2b0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:41.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.069+0000 7fc2b7200640 1 -- 172.21.15.12:0/1830251538 shutdown_connections 2024-03-23T23:01:41.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.069+0000 7fc2b7200640 1 -- 172.21.15.12:0/1830251538 wait complete. 2024-03-23T23:01:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.070+0000 7fc2b7200640 1 Processor -- start 2024-03-23T23:01:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.070+0000 7fc2b7200640 1 -- start start 2024-03-23T23:01:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.070+0000 7fc2b7200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b01038e0 0x7fc2b0199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.070+0000 7fc2b7200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2b0104ad0 0x7fc2b019a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.070+0000 7fc2b7200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2b0105490 0x7fc2b01a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.070+0000 7fc2b7200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b010f540 con 0x7fc2b01038e0 2024-03-23T23:01:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.070+0000 7fc2b7200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b010f3c0 con 0x7fc2b0105490 2024-03-23T23:01:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.070+0000 7fc2b7200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b010f6c0 con 0x7fc2b0104ad0 2024-03-23T23:01:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.071+0000 7fc2afe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b01038e0 0x7fc2b0199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.071+0000 7fc2af400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2b0104ad0 0x7fc2b019a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.071+0000 7fc2afe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b01038e0 0x7fc2b0199ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38982/0 (socket says 172.21.15.12:38982) 2024-03-23T23:01:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.071+0000 7fc2afe00640 1 -- 172.21.15.12:0/671818310 learned_addr learned my addr 172.21.15.12:0/671818310 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.071+0000 7fc2b5200640 1 --2- 172.21.15.12:0/671818310 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2b0105490 0x7fc2b01a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.071+0000 7fc2b5200640 1 -- 172.21.15.12:0/671818310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2b0104ad0 msgr2=0x7fc2b019a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:41.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.071+0000 7fc2b5200640 1 --2- 172.21.15.12:0/671818310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2b0104ad0 0x7fc2b019a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.071+0000 7fc2b5200640 1 -- 172.21.15.12:0/671818310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b01038e0 msgr2=0x7fc2b0199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:41.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.071+0000 7fc2b5200640 1 --2- 172.21.15.12:0/671818310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b01038e0 0x7fc2b0199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.071+0000 7fc2b5200640 1 -- 172.21.15.12:0/671818310 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b01a16e0 con 0x7fc2b0105490 2024-03-23T23:01:41.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.071+0000 7fc2af400640 1 --2- 172.21.15.12:0/671818310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2b0104ad0 0x7fc2b019a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:41.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.071+0000 7fc2b5200640 1 --2- 172.21.15.12:0/671818310 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2b0105490 0x7fc2b01a1070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc2a8037040 tx=0x7fc2a8035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:41.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.072+0000 7fc2acc00640 1 -- 172.21.15.12:0/671818310 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8039040 con 0x7fc2b0105490 2024-03-23T23:01:41.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.072+0000 7fc2acc00640 1 -- 172.21.15.12:0/671818310 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2a8044070 con 0x7fc2b0105490 2024-03-23T23:01:41.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.072+0000 7fc2acc00640 1 -- 172.21.15.12:0/671818310 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8004370 con 0x7fc2b0105490 2024-03-23T23:01:41.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.072+0000 7fc2b7200640 1 -- 172.21.15.12:0/671818310 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2b01a1970 con 0x7fc2b0105490 2024-03-23T23:01:41.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.072+0000 7fc2b7200640 1 -- 172.21.15.12:0/671818310 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2b01a1de0 con 0x7fc2b0105490 2024-03-23T23:01:41.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.076+0000 7fc2b9e0f740 1 Processor -- start 2024-03-23T23:01:41.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.077+0000 7fc2acc00640 1 -- 172.21.15.12:0/671818310 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc2a8004510 con 0x7fc2b0105490 2024-03-23T23:01:41.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.077+0000 7fc2b9e0f740 1 -- start start 2024-03-23T23:01:41.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.077+0000 7fc2b9e0f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef013efba0 0x55ef013effc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.077+0000 7fc2b9e0f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ef013f0590 0x55ef013f5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.077+0000 7fc2b9e0f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef013f5700 0x55ef013f7b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.077+0000 7fc2b9e0f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef01354c90 con 0x55ef013f0590 2024-03-23T23:01:41.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.077+0000 7fc2b9e0f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef012fd980 con 0x55ef013f5700 2024-03-23T23:01:41.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.077+0000 7fc2b9e0f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef01350d30 con 0x55ef013efba0 2024-03-23T23:01:41.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.077+0000 7fc2afe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ef013f0590 0x55ef013f5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.077+0000 7fc2af400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef013efba0 0x55ef013effc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.077+0000 7fc2afe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ef013f0590 0x55ef013f5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38984/0 (socket says 172.21.15.12:38984) 2024-03-23T23:01:41.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.078+0000 7fc2afe00640 1 -- 172.21.15.12:0/2567905240 learned_addr learned my addr 172.21.15.12:0/2567905240 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:41.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.078+0000 7fc2af400640 1 --2- 172.21.15.12:0/2567905240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef013f5700 0x55ef013f7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.078+0000 7fc2afe00640 1 -- 172.21.15.12:0/2567905240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef013efba0 msgr2=0x55ef013effc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:41.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.078+0000 7fc2afe00640 1 --2- 172.21.15.12:0/2567905240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef013efba0 0x55ef013effc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.078+0000 7fc2afe00640 1 -- 172.21.15.12:0/2567905240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef013f5700 msgr2=0x55ef013f7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:41.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.078+0000 7fc2afe00640 1 --2- 172.21.15.12:0/2567905240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef013f5700 0x55ef013f7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.078+0000 7fc2afe00640 1 -- 172.21.15.12:0/2567905240 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef01315510 con 0x55ef013f0590 2024-03-23T23:01:41.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.078+0000 7fc2af400640 1 --2- 172.21.15.12:0/2567905240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef013efba0 0x55ef013effc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.078+0000 7fc2afe00640 1 --2- 172.21.15.12:0/2567905240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ef013f0590 0x55ef013f5090 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0009d60 tx=0x7fc2a0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cd68ea6a1051082 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.078+0000 7fc2acc00640 1 --2- 172.21.15.12:0/671818310 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc2940a2800 0x7fc2940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.079+0000 7fc2a5600640 1 -- 172.21.15.12:0/2567905240 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0004230 con 0x55ef013f0590 2024-03-23T23:01:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.079+0000 7fc2acc00640 1 -- 172.21.15.12:0/671818310 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fc2a80f1060 con 0x7fc2b0105490 2024-03-23T23:01:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.079+0000 7fc2a5600640 1 -- 172.21.15.12:0/2567905240 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2a00043d0 con 0x55ef013f0590 2024-03-23T23:01:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.079+0000 7fc2a5600640 1 -- 172.21.15.12:0/2567905240 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0019820 con 0x55ef013f0590 2024-03-23T23:01:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.079+0000 7fc2af400640 1 --2- 172.21.15.12:0/671818310 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc2940a2800 0x7fc2940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.079+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/2567905240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ef013f0590 msgr2=0x55ef013f5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.079+0000 7fc2b9e0f740 1 --2- 172.21.15.12:0/2567905240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ef013f0590 0x55ef013f5090 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0009d60 tx=0x7fc2a0011810 comp rx=0 tx=0).stop 2024-03-23T23:01:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.079+0000 7fc2af400640 1 --2- 172.21.15.12:0/671818310 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc2940a2800 0x7fc2940a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x55ef013f0df0 tx=0x7fc29c00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.080+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/2567905240 shutdown_connections 2024-03-23T23:01:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.080+0000 7fc2b9e0f740 1 --2- 172.21.15.12:0/2567905240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef013f5700 0x55ef013f7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.080+0000 7fc2b9e0f740 1 --2- 172.21.15.12:0/2567905240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ef013f0590 0x55ef013f5090 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.080+0000 7fc2b9e0f740 1 --2- 172.21.15.12:0/2567905240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef013efba0 0x55ef013effc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.080+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/2567905240 >> 172.21.15.12:0/2567905240 conn(0x55ef013eb110 msgr2=0x55ef013eb510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:41.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.080+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/2567905240 shutdown_connections 2024-03-23T23:01:41.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.080+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/2567905240 wait complete. 2024-03-23T23:01:41.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.080+0000 7fc2b9e0f740 1 -- start start 2024-03-23T23:01:41.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.080+0000 7fc2b9e0f740 1 Processor -- start 2024-03-23T23:01:41.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.081+0000 7fc2b9e0f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ef013efba0 0x55ef013fa150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.081+0000 7fc2b9e0f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef013f0590 0x55ef013fa690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.081+0000 7fc2b9e0f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef013f5700 0x55ef01401660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.081+0000 7fc2b9e0f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef013f86b0 con 0x55ef013efba0 2024-03-23T23:01:41.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.081+0000 7fc2b9e0f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef01310dd0 con 0x55ef013f0590 2024-03-23T23:01:41.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.081+0000 7fc2b9e0f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef013f8830 con 0x55ef013f5700 2024-03-23T23:01:41.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.081+0000 7fc2b9e0f740 10 client.0 _ll_register_callbacks cb 0x55ef01068ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:01:41.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.081+0000 7fc2afe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ef013efba0 0x55ef013fa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.081+0000 7fc2af400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef013f0590 0x55ef013fa690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.081+0000 7fc2af400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef013f0590 0x55ef013fa690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36336/0 (socket says 172.21.15.12:36336) 2024-03-23T23:01:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.081+0000 7fc2af400640 1 -- 172.21.15.12:0/3608762319 learned_addr learned my addr 172.21.15.12:0/3608762319 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.081+0000 7fc2b5200640 1 --2- 172.21.15.12:0/3608762319 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef013f5700 0x55ef01401660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.081+0000 7fc2af400640 1 -- 172.21.15.12:0/3608762319 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef013f5700 msgr2=0x55ef01401660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.081+0000 7fc2af400640 1 --2- 172.21.15.12:0/3608762319 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef013f5700 0x55ef01401660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.081+0000 7fc2af400640 1 -- 172.21.15.12:0/3608762319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ef013efba0 msgr2=0x55ef013fa150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.081+0000 7fc2af400640 1 --2- 172.21.15.12:0/3608762319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ef013efba0 0x55ef013fa150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.081+0000 7fc2af400640 1 -- 172.21.15.12:0/3608762319 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ef010692d0 con 0x55ef013f0590 2024-03-23T23:01:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.082+0000 7fc2af400640 1 --2- 172.21.15.12:0/3608762319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef013f0590 0x55ef013fa690 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc29c020760 tx=0x7fc29c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b50a461a96f0e98 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.082+0000 7fc29a000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:01:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.082+0000 7fc29a000640 1 -- 172.21.15.12:0/3608762319 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c02d060 con 0x55ef013f0590 2024-03-23T23:01:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.082+0000 7fc29a000640 1 -- 172.21.15.12:0/3608762319 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fc29c01a030 con 0x55ef013f0590 2024-03-23T23:01:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.082+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/3608762319 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc2a00043d0 con 0x55ef013f0590 2024-03-23T23:01:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.082+0000 7fc29a000640 1 -- 172.21.15.12:0/3608762319 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc29c01e7a0 con 0x55ef013f0590 2024-03-23T23:01:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.082+0000 7fc2b9e0f740 10 client.25169 fetch_fsmap learned FSMap version 22 2024-03-23T23:01:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.082+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/3608762319 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55ef011336b0 con 0x55ef013f0590 2024-03-23T23:01:41.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.082+0000 7fc29a000640 1 -- 172.21.15.12:0/3608762319 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2a00043d0 con 0x55ef013f0590 2024-03-23T23:01:41.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.083+0000 7fc29a000640 1 -- 172.21.15.12:0/3608762319 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fc29c033bc0 con 0x55ef013f0590 2024-03-23T23:01:41.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.083+0000 7fc2b9e0f740 10 client.25169 fetch_fsmap finished waiting for FSMap version 22 2024-03-23T23:01:41.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.083+0000 7fc2b9e0f740 10 client.25169 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-23T23:01:41.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.083+0000 7fc2b9e0f740 20 client.25169 populate_metadata read hostname 'smithi012' 2024-03-23T23:01:41.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.083+0000 7fc2b9e0f740 1 --2- 172.21.15.12:0/3608762319 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55ef014068a0 0x55ef01405940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.083+0000 7fc2b9e0f740 4 client.25169 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:01:41.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.083+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/3608762319 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ef012f6520 con 0x55ef014068a0 2024-03-23T23:01:41.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.083+0000 7fc2afe00640 1 --2- 172.21.15.12:0/3608762319 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55ef014068a0 0x55ef01405940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.084+0000 7fc2afe00640 1 --2- 172.21.15.12:0/3608762319 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55ef014068a0 0x55ef01405940 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5c4eafa272a051c1 server_cookie=64c53a2e8dca4c8d in_seq=0 out_seq=0 2024-03-23T23:01:41.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.084+0000 7fc29a000640 10 client.25169 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:01:41.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.084+0000 7fc29a000640 1 -- 172.21.15.12:0/3608762319 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ef012f6520 con 0x55ef014068a0 2024-03-23T23:01:41.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.084+0000 7fc29a000640 10 client.25169 handle_command_reply: tid=0 2024-03-23T23:01:41.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.084+0000 7fc2b9e0f740 1 client.25169 shutdown 2024-03-23T23:01:41.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.084+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/3608762319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef013f0590 msgr2=0x55ef013fa690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:41.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.084+0000 7fc2b9e0f740 1 --2- 172.21.15.12:0/3608762319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef013f0590 0x55ef013fa690 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc29c020760 tx=0x7fc29c0046c0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.085+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/3608762319 shutdown_connections 2024-03-23T23:01:41.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.085+0000 7fc2b9e0f740 1 --2- 172.21.15.12:0/3608762319 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55ef014068a0 0x55ef01405940 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.085+0000 7fc2b9e0f740 1 --2- 172.21.15.12:0/3608762319 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef013f5700 0x55ef01401660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.085+0000 7fc2b9e0f740 1 --2- 172.21.15.12:0/3608762319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef013f0590 0x55ef013fa690 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.085+0000 7fc2b9e0f740 1 --2- 172.21.15.12:0/3608762319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ef013efba0 0x55ef013fa150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.085+0000 7fc29a000640 0 client.25169 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:01:41.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.085+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/3608762319 >> 172.21.15.12:0/3608762319 conn(0x55ef013eb110 msgr2=0x55ef013ea310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:41.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.085+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/3608762319 shutdown_connections 2024-03-23T23:01:41.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.085+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/3608762319 wait complete. 2024-03-23T23:01:41.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.085+0000 7fc2b9e0f740 20 client.25169 trim_cache size 0 max 16384 2024-03-23T23:01:41.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.107+0000 7fc2b9e0f740 1 Processor -- start 2024-03-23T23:01:41.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.108+0000 7fc2b9e0f740 1 -- start start 2024-03-23T23:01:41.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.108+0000 7fc2b9e0f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef013efba0 0x55ef013eb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.108+0000 7fc2b9e0f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ef013f0590 0x55ef01401620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.108+0000 7fc2b9e0f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef013f5700 0x55ef013f1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.108+0000 7fc2b9e0f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef014060c0 con 0x55ef013f0590 2024-03-23T23:01:41.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.108+0000 7fc2b9e0f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef01406720 con 0x55ef013f5700 2024-03-23T23:01:41.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.108+0000 7fc2b9e0f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef01406d80 con 0x55ef013efba0 2024-03-23T23:01:41.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.108+0000 7fc2af400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ef013f0590 0x55ef01401620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.108+0000 7fc2b5200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef013f5700 0x55ef013f1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.108+0000 7fc2afe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef013efba0 0x55ef013eb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.108+0000 7fc2afe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef013efba0 0x55ef013eb4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35534/0 (socket says 172.21.15.12:35534) 2024-03-23T23:01:41.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.108+0000 7fc2af400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ef013f0590 0x55ef01401620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38990/0 (socket says 172.21.15.12:38990) 2024-03-23T23:01:41.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.108+0000 7fc2b5200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef013f5700 0x55ef013f1030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36350/0 (socket says 172.21.15.12:36350) 2024-03-23T23:01:41.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.108+0000 7fc2afe00640 1 -- 172.21.15.12:0/3711813190 learned_addr learned my addr 172.21.15.12:0/3711813190 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:41.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.109+0000 7fc2afe00640 1 -- 172.21.15.12:0/3711813190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef013f5700 msgr2=0x55ef013f1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:41.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.109+0000 7fc2afe00640 1 --2- 172.21.15.12:0/3711813190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef013f5700 0x55ef013f1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.109+0000 7fc2afe00640 1 -- 172.21.15.12:0/3711813190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ef013f0590 msgr2=0x55ef01401620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:41.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.109+0000 7fc2afe00640 1 --2- 172.21.15.12:0/3711813190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ef013f0590 0x55ef01401620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.109+0000 7fc2afe00640 1 -- 172.21.15.12:0/3711813190 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef013f0e20 con 0x55ef013efba0 2024-03-23T23:01:41.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.109+0000 7fc2afe00640 1 --2- 172.21.15.12:0/3711813190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef013efba0 0x55ef013eb4f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0000c00 tx=0x7fc2a0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f1c7561ee4d72ab server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:41.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.109+0000 7fc2a4c00640 1 -- 172.21.15.12:0/3711813190 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a00040c0 con 0x55ef013efba0 2024-03-23T23:01:41.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.109+0000 7fc2a4c00640 1 -- 172.21.15.12:0/3711813190 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2a002f020 con 0x55ef013efba0 2024-03-23T23:01:41.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.109+0000 7fc2a4c00640 1 -- 172.21.15.12:0/3711813190 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a00035f0 con 0x55ef013efba0 2024-03-23T23:01:41.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.110+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/3711813190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef013efba0 msgr2=0x55ef013eb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:41.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.110+0000 7fc2b9e0f740 1 --2- 172.21.15.12:0/3711813190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef013efba0 0x55ef013eb4f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0000c00 tx=0x7fc2a0011810 comp rx=0 tx=0).stop 2024-03-23T23:01:41.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.110+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/3711813190 shutdown_connections 2024-03-23T23:01:41.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.110+0000 7fc2b9e0f740 1 --2- 172.21.15.12:0/3711813190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef013f5700 0x55ef013f1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.110+0000 7fc2b9e0f740 1 --2- 172.21.15.12:0/3711813190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ef013f0590 0x55ef01401620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.110+0000 7fc2b9e0f740 1 --2- 172.21.15.12:0/3711813190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef013efba0 0x55ef013eb4f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.110+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/3711813190 >> 172.21.15.12:0/3711813190 conn(0x55ef013eb110 msgr2=0x55ef013eec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:41.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.110+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/3711813190 shutdown_connections 2024-03-23T23:01:41.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.110+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/3711813190 wait complete. 2024-03-23T23:01:41.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.111+0000 7fc2b9e0f740 1 -- start start 2024-03-23T23:01:41.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.111+0000 7fc2b9e0f740 1 Processor -- start 2024-03-23T23:01:41.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.111+0000 7fc2b9e0f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ef013efba0 0x55ef013fd850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.111+0000 7fc2b9e0f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef013f0590 0x55ef013fdd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.111+0000 7fc2b9e0f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef013f5700 0x55ef01400c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.111+0000 7fc2b9e0f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef01407a40 con 0x55ef013efba0 2024-03-23T23:01:41.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.111+0000 7fc2b9e0f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef014073e0 con 0x55ef013f5700 2024-03-23T23:01:41.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.111+0000 7fc2b9e0f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef014080a0 con 0x55ef013f0590 2024-03-23T23:01:41.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.111+0000 7fc2b9e0f740 10 client.0 _ll_register_callbacks cb 0x7fc29c00ace0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:01:41.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.112+0000 7fc2afe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ef013efba0 0x55ef013fd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.112+0000 7fc2b5200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef013f5700 0x55ef01400c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.112+0000 7fc2af400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef013f0590 0x55ef013fdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.112+0000 7fc2b5200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef013f5700 0x55ef01400c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36356/0 (socket says 172.21.15.12:36356) 2024-03-23T23:01:41.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.112+0000 7fc2afe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ef013efba0 0x55ef013fd850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39000/0 (socket says 172.21.15.12:39000) 2024-03-23T23:01:41.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.112+0000 7fc2af400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef013f0590 0x55ef013fdd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35540/0 (socket says 172.21.15.12:35540) 2024-03-23T23:01:41.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.112+0000 7fc2b5200640 1 -- 172.21.15.12:0/575609220 learned_addr learned my addr 172.21.15.12:0/575609220 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:41.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.112+0000 7fc2b5200640 1 -- 172.21.15.12:0/575609220 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef013f0590 msgr2=0x55ef013fdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:41.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.112+0000 7fc2b5200640 1 --2- 172.21.15.12:0/575609220 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef013f0590 0x55ef013fdd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.112+0000 7fc2b5200640 1 -- 172.21.15.12:0/575609220 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ef013efba0 msgr2=0x55ef013fd850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:41.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.112+0000 7fc2b5200640 1 --2- 172.21.15.12:0/575609220 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ef013efba0 0x55ef013fd850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.112+0000 7fc2b5200640 1 -- 172.21.15.12:0/575609220 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ef01401230 con 0x55ef013f5700 2024-03-23T23:01:41.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.112+0000 7fc2af400640 1 --2- 172.21.15.12:0/575609220 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef013f0590 0x55ef013fdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:41.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.112+0000 7fc2b5200640 1 --2- 172.21.15.12:0/575609220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef013f5700 0x55ef01400c40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80eefd0 tx=0x7fc2a80bd850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c19796dc880d2af server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.112+0000 7fc29a000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:01:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.112+0000 7fc29a000640 1 -- 172.21.15.12:0/575609220 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a80be0b0 con 0x55ef013f5700 2024-03-23T23:01:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.112+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/575609220 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc2a002f020 con 0x55ef013f5700 2024-03-23T23:01:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.112+0000 7fc29a000640 1 -- 172.21.15.12:0/575609220 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fc2a80bdc10 con 0x55ef013f5700 2024-03-23T23:01:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.113+0000 7fc29a000640 1 -- 172.21.15.12:0/575609220 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2a80098b0 con 0x55ef013f5700 2024-03-23T23:01:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.113+0000 7fc2b9e0f740 10 client.25175 fetch_fsmap learned FSMap version 22 2024-03-23T23:01:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.113+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/575609220 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55ef014622e0 con 0x55ef013f5700 2024-03-23T23:01:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.113+0000 7fc29a000640 1 -- 172.21.15.12:0/575609220 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2a002f020 con 0x55ef013f5700 2024-03-23T23:01:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.113+0000 7fc29a000640 1 -- 172.21.15.12:0/575609220 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fc2a80bece0 con 0x55ef013f5700 2024-03-23T23:01:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.113+0000 7fc2b9e0f740 10 client.25175 fetch_fsmap finished waiting for FSMap version 22 2024-03-23T23:01:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.113+0000 7fc2b9e0f740 10 client.25175 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-23T23:01:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.113+0000 7fc2b9e0f740 20 client.25175 populate_metadata read hostname 'smithi012' 2024-03-23T23:01:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.113+0000 7fc2b9e0f740 1 --2- 172.21.15.12:0/575609220 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55ef01466a10 0x55ef014625a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.113+0000 7fc2b9e0f740 4 client.25175 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-23T23:01:41.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.113+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/575609220 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ef013621d0 con 0x55ef01466a10 2024-03-23T23:01:41.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.113+0000 7fc2afe00640 1 --2- 172.21.15.12:0/575609220 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55ef01466a10 0x55ef014625a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.114+0000 7fc2afe00640 1 --2- 172.21.15.12:0/575609220 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55ef01466a10 0x55ef014625a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=66aebbef18af51a8 server_cookie=7372c5381a1a68da in_seq=0 out_seq=0 2024-03-23T23:01:41.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.114+0000 7fc29a000640 10 client.25175 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:01:41.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.114+0000 7fc29a000640 1 -- 172.21.15.12:0/575609220 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x55ef013621d0 con 0x55ef01466a10 2024-03-23T23:01:41.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.114+0000 7fc29a000640 10 client.25175 handle_command_reply: tid=0 2024-03-23T23:01:41.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.114+0000 7fc2b9e0f740 1 client.25175 shutdown 2024-03-23T23:01:41.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.114+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/575609220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef013f5700 msgr2=0x55ef01400c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:41.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.114+0000 7fc2b9e0f740 1 --2- 172.21.15.12:0/575609220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef013f5700 0x55ef01400c40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80eefd0 tx=0x7fc2a80bd850 comp rx=0 tx=0).stop 2024-03-23T23:01:41.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.114+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/575609220 shutdown_connections 2024-03-23T23:01:41.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.114+0000 7fc2b9e0f740 1 --2- 172.21.15.12:0/575609220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef013f5700 0x55ef01400c40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.114+0000 7fc2b9e0f740 1 --2- 172.21.15.12:0/575609220 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55ef01466a10 0x55ef014625a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.114+0000 7fc2b9e0f740 1 --2- 172.21.15.12:0/575609220 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef013f0590 0x55ef013fdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.114+0000 7fc2b9e0f740 1 --2- 172.21.15.12:0/575609220 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ef013efba0 0x55ef013fd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.114+0000 7fc29a000640 0 client.25175 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:01:41.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.114+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/575609220 >> 172.21.15.12:0/575609220 conn(0x55ef013eb110 msgr2=0x55ef013eec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:41.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.115+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/575609220 shutdown_connections 2024-03-23T23:01:41.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.115+0000 7fc2b9e0f740 1 -- 172.21.15.12:0/575609220 wait complete. 2024-03-23T23:01:41.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.115+0000 7fc2b9e0f740 20 client.25175 trim_cache size 0 max 16384 2024-03-23T23:01:41.119 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:01:41.119 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "90f679ae-e967-11ee-95cd-87774f69a715", 2024-03-23T23:01:41.119 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 1, 2024-03-23T23:01:41.119 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 24554, 2024-03-23T23:01:41.119 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-23T23:01:41.119 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-23T23:01:41.120 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-23T23:01:41.120 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 130.639675635, 2024-03-23T23:01:41.120 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 22, 2024-03-23T23:01:41.120 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 88, 2024-03-23T23:01:41.120 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 86, 2024-03-23T23:01:41.120 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 154.584779228 2024-03-23T23:01:41.120 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:01:41.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.115+0000 7fc2b7200640 1 -- 172.21.15.12:0/671818310 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc2940a2800 msgr2=0x7fc2940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:41.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.115+0000 7fc2b7200640 1 --2- 172.21.15.12:0/671818310 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc2940a2800 0x7fc2940a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x55ef013f0df0 tx=0x7fc29c00a2d0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.115+0000 7fc2b7200640 1 -- 172.21.15.12:0/671818310 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2b0105490 msgr2=0x7fc2b01a1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:41.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.115+0000 7fc2b7200640 1 --2- 172.21.15.12:0/671818310 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2b0105490 0x7fc2b01a1070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc2a8037040 tx=0x7fc2a8035d10 comp rx=0 tx=0).stop 2024-03-23T23:01:41.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.116+0000 7fc2b7200640 1 -- 172.21.15.12:0/671818310 shutdown_connections 2024-03-23T23:01:41.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.116+0000 7fc2b7200640 1 --2- 172.21.15.12:0/671818310 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc2940a2800 0x7fc2940a4cc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.116+0000 7fc2b7200640 1 --2- 172.21.15.12:0/671818310 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2b0105490 0x7fc2b01a1070 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.116+0000 7fc2b7200640 1 --2- 172.21.15.12:0/671818310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2b0104ad0 0x7fc2b019a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.116+0000 7fc2b7200640 1 --2- 172.21.15.12:0/671818310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b01038e0 0x7fc2b0199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.116+0000 7fc2b7200640 1 -- 172.21.15.12:0/671818310 >> 172.21.15.12:0/671818310 conn(0x7fc2b00fee50 msgr2=0x7fc2b0107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:41.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.116+0000 7fc2b7200640 1 -- 172.21.15.12:0/671818310 shutdown_connections 2024-03-23T23:01:41.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.116+0000 7fc2b7200640 1 -- 172.21.15.12:0/671818310 wait complete. 2024-03-23T23:01:41.136 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-23T23:01:41.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.364+0000 7f9011600640 1 -- 172.21.15.12:0/1053963354 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c1016b0 msgr2=0x7f900c101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:41.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.364+0000 7f9011600640 1 --2- 172.21.15.12:0/1053963354 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c1016b0 0x7f900c101ad0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc009900 tx=0x7f8ffc02bc50 comp rx=0 tx=0).stop 2024-03-23T23:01:41.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.364+0000 7f9011600640 1 -- 172.21.15.12:0/1053963354 shutdown_connections 2024-03-23T23:01:41.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.364+0000 7f9011600640 1 --2- 172.21.15.12:0/1053963354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f900c103260 0x7f900c109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.364+0000 7f9011600640 1 --2- 172.21.15.12:0/1053963354 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f900c1028a0 0x7f900c102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.364+0000 7f9011600640 1 --2- 172.21.15.12:0/1053963354 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c1016b0 0x7f900c101ad0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.364+0000 7f9011600640 1 -- 172.21.15.12:0/1053963354 >> 172.21.15.12:0/1053963354 conn(0x7f900c0fcc40 msgr2=0x7f900c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:41.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.364+0000 7f9011600640 1 -- 172.21.15.12:0/1053963354 shutdown_connections 2024-03-23T23:01:41.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.364+0000 7f9011600640 1 -- 172.21.15.12:0/1053963354 wait complete. 2024-03-23T23:01:41.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.365+0000 7f9011600640 1 Processor -- start 2024-03-23T23:01:41.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.365+0000 7f9011600640 1 -- start start 2024-03-23T23:01:41.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.366+0000 7f9011600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c1016b0 0x7f900c199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.366+0000 7f9011600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f900c1028a0 0x7f900c19a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.366+0000 7f9011600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f900c103260 0x7f900c1a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.366+0000 7f9011600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900c10d370 con 0x7f900c103260 2024-03-23T23:01:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.366+0000 7f9011600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900c10d1f0 con 0x7f900c1028a0 2024-03-23T23:01:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.366+0000 7f9011600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900c10d4f0 con 0x7f900c1016b0 2024-03-23T23:01:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.366+0000 7f900aa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c1016b0 0x7f900c199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.366+0000 7f9003e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f900c1028a0 0x7f900c19a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.366+0000 7f900aa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c1016b0 0x7f900c199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35558/0 (socket says 172.21.15.12:35558) 2024-03-23T23:01:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.366+0000 7f900aa00640 1 -- 172.21.15.12:0/813272854 learned_addr learned my addr 172.21.15.12:0/813272854 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.366+0000 7f900b400640 1 --2- 172.21.15.12:0/813272854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f900c103260 0x7f900c1a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.366+0000 7f900aa00640 1 -- 172.21.15.12:0/813272854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f900c1028a0 msgr2=0x7f900c19a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.366+0000 7f900aa00640 1 --2- 172.21.15.12:0/813272854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f900c1028a0 0x7f900c19a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.366+0000 7f900aa00640 1 -- 172.21.15.12:0/813272854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f900c103260 msgr2=0x7f900c1a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.366+0000 7f900aa00640 1 --2- 172.21.15.12:0/813272854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f900c103260 0x7f900c1a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.366+0000 7f900aa00640 1 -- 172.21.15.12:0/813272854 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900c1a18d0 con 0x7f900c1016b0 2024-03-23T23:01:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.367+0000 7f900aa00640 1 --2- 172.21.15.12:0/813272854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c1016b0 0x7f900c199c90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc005ab0 tx=0x7f8ffc02c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.367+0000 7f9003400640 1 -- 172.21.15.12:0/813272854 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc041070 con 0x7f900c1016b0 2024-03-23T23:01:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.367+0000 7f9003400640 1 -- 172.21.15.12:0/813272854 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ffc004450 con 0x7f900c1016b0 2024-03-23T23:01:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.367+0000 7f9003400640 1 -- 172.21.15.12:0/813272854 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc034440 con 0x7f900c1016b0 2024-03-23T23:01:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.367+0000 7f9011600640 1 -- 172.21.15.12:0/813272854 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f900c1a1b60 con 0x7f900c1016b0 2024-03-23T23:01:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.367+0000 7f9011600640 1 -- 172.21.15.12:0/813272854 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f900c1a2060 con 0x7f900c1016b0 2024-03-23T23:01:41.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.371+0000 7f9014217740 1 Processor -- start 2024-03-23T23:01:41.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.371+0000 7f9014217740 1 -- start start 2024-03-23T23:01:41.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.371+0000 7f9014217740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5572f27eeba0 0x5572f27eefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.371+0000 7f9014217740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572f27ef590 0x5572f27f4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.371+0000 7f9014217740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27f4700 0x5572f27f6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.371+0000 7f9014217740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572f2753c90 con 0x5572f27eeba0 2024-03-23T23:01:41.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.371+0000 7f9014217740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572f26fc980 con 0x5572f27f4700 2024-03-23T23:01:41.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.371+0000 7f9014217740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572f274fd30 con 0x5572f27ef590 2024-03-23T23:01:41.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.372+0000 7f9003400640 1 -- 172.21.15.12:0/813272854 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8ffc044070 con 0x7f900c1016b0 2024-03-23T23:01:41.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.372+0000 7f9003e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27f4700 0x5572f27f6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.372+0000 7f900b400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5572f27eeba0 0x5572f27eefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.372+0000 7f9003e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27f4700 0x5572f27f6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36392/0 (socket says 172.21.15.12:36392) 2024-03-23T23:01:41.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.372+0000 7f9003e00640 1 -- 172.21.15.12:0/3391723126 learned_addr learned my addr 172.21.15.12:0/3391723126 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:41.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.372+0000 7f900aa00640 1 --2- 172.21.15.12:0/3391723126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572f27ef590 0x5572f27f4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.372+0000 7f9003e00640 1 -- 172.21.15.12:0/3391723126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572f27ef590 msgr2=0x5572f27f4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:41.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.372+0000 7f9003e00640 1 --2- 172.21.15.12:0/3391723126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572f27ef590 0x5572f27f4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.372+0000 7f9003e00640 1 -- 172.21.15.12:0/3391723126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5572f27eeba0 msgr2=0x5572f27eefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:41.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.372+0000 7f9003e00640 1 --2- 172.21.15.12:0/3391723126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5572f27eeba0 0x5572f27eefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.372+0000 7f9003e00640 1 -- 172.21.15.12:0/3391723126 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572f2714510 con 0x5572f27f4700 2024-03-23T23:01:41.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.372+0000 7f900aa00640 1 --2- 172.21.15.12:0/3391723126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572f27ef590 0x5572f27f4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:01:41.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.373+0000 7f9003e00640 1 --2- 172.21.15.12:0/3391723126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27f4700 0x5572f27f6b90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4009ca0 tx=0x7f8ff4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a06942383b527deb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:41.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.373+0000 7f8ffb400640 1 -- 172.21.15.12:0/3391723126 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff401e070 con 0x5572f27f4700 2024-03-23T23:01:41.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.373+0000 7f8ffb400640 1 -- 172.21.15.12:0/3391723126 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ff4019b10 con 0x5572f27f4700 2024-03-23T23:01:41.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.373+0000 7f8ffb400640 1 -- 172.21.15.12:0/3391723126 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff400eae0 con 0x5572f27f4700 2024-03-23T23:01:41.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.373+0000 7f9003400640 1 --2- 172.21.15.12:0/813272854 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8ff00a2880 0x7f8ff00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.374+0000 7f9003400640 1 -- 172.21.15.12:0/813272854 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8ffc0f0580 con 0x7f900c1016b0 2024-03-23T23:01:41.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.374+0000 7f900b400640 1 --2- 172.21.15.12:0/813272854 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8ff00a2880 0x7f8ff00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.374+0000 7f9014217740 1 -- 172.21.15.12:0/3391723126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27f4700 msgr2=0x5572f27f6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:41.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.374+0000 7f9014217740 1 --2- 172.21.15.12:0/3391723126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27f4700 0x5572f27f6b90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4009ca0 tx=0x7f8ff4011810 comp rx=0 tx=0).stop 2024-03-23T23:01:41.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.374+0000 7f9014217740 1 -- 172.21.15.12:0/3391723126 shutdown_connections 2024-03-23T23:01:41.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.374+0000 7f9014217740 1 --2- 172.21.15.12:0/3391723126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27f4700 0x5572f27f6b90 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.374+0000 7f9014217740 1 --2- 172.21.15.12:0/3391723126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572f27ef590 0x5572f27f4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.374+0000 7f9014217740 1 --2- 172.21.15.12:0/3391723126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5572f27eeba0 0x5572f27eefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.374+0000 7f9014217740 1 -- 172.21.15.12:0/3391723126 >> 172.21.15.12:0/3391723126 conn(0x5572f27ea110 msgr2=0x5572f27ea510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:41.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.374+0000 7f9014217740 1 -- 172.21.15.12:0/3391723126 shutdown_connections 2024-03-23T23:01:41.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.374+0000 7f9014217740 1 -- 172.21.15.12:0/3391723126 wait complete. 2024-03-23T23:01:41.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.374+0000 7f900b400640 1 --2- 172.21.15.12:0/813272854 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8ff00a2880 0x7f8ff00a4d40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x5572f27efdf0 tx=0x7f9004009600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:41.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.375+0000 7f9014217740 1 -- start start 2024-03-23T23:01:41.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.375+0000 7f9014217740 1 Processor -- start 2024-03-23T23:01:41.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.375+0000 7f9014217740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27eeba0 0x5572f27f9070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.375+0000 7f9014217740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572f27ef590 0x5572f27f95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.375+0000 7f9014217740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5572f27f4700 0x5572f28005f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.375+0000 7f9014217740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572f27f76b0 con 0x5572f27f4700 2024-03-23T23:01:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.376+0000 7f9014217740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572f270fdd0 con 0x5572f27eeba0 2024-03-23T23:01:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.376+0000 7f9014217740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572f27f7830 con 0x5572f27ef590 2024-03-23T23:01:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.376+0000 7f9014217740 10 client.0 _ll_register_callbacks cb 0x5572f2467ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:01:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.376+0000 7f900aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27eeba0 0x5572f27f9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.376+0000 7f900aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27eeba0 0x5572f27f9070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36406/0 (socket says 172.21.15.12:36406) 2024-03-23T23:01:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.376+0000 7f900aa00640 1 -- 172.21.15.12:0/2423087469 learned_addr learned my addr 172.21.15.12:0/2423087469 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.376+0000 7f9003e00640 1 --2- 172.21.15.12:0/2423087469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572f27ef590 0x5572f27f95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.376+0000 7f900b400640 1 --2- 172.21.15.12:0/2423087469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5572f27f4700 0x5572f28005f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.376+0000 7f900aa00640 1 -- 172.21.15.12:0/2423087469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572f27ef590 msgr2=0x5572f27f95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.376+0000 7f900aa00640 1 --2- 172.21.15.12:0/2423087469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572f27ef590 0x5572f27f95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.376+0000 7f900aa00640 1 -- 172.21.15.12:0/2423087469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5572f27f4700 msgr2=0x5572f28005f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.376+0000 7f900aa00640 1 --2- 172.21.15.12:0/2423087469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5572f27f4700 0x5572f28005f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.376+0000 7f900aa00640 1 -- 172.21.15.12:0/2423087469 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5572f24682d0 con 0x5572f27eeba0 2024-03-23T23:01:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.376+0000 7f9003e00640 1 --2- 172.21.15.12:0/2423087469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572f27ef590 0x5572f27f95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.376+0000 7f900aa00640 1 --2- 172.21.15.12:0/2423087469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27eeba0 0x5572f27f9070 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc0096c0 tx=0x7f8ffc02bc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e96a5a4ee50b851 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:41.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.377+0000 7f8fefe00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:01:41.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.377+0000 7f8fefe00640 1 -- 172.21.15.12:0/2423087469 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc04b070 con 0x5572f27eeba0 2024-03-23T23:01:41.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.377+0000 7f8fefe00640 1 -- 172.21.15.12:0/2423087469 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8ffc0c5240 con 0x5572f27eeba0 2024-03-23T23:01:41.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.377+0000 7f9014217740 1 -- 172.21.15.12:0/2423087469 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ff4019b10 con 0x5572f27eeba0 2024-03-23T23:01:41.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.377+0000 7f8fefe00640 1 -- 172.21.15.12:0/2423087469 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ffc0bcfe0 con 0x5572f27eeba0 2024-03-23T23:01:41.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.377+0000 7f9014217740 10 client.25187 fetch_fsmap learned FSMap version 22 2024-03-23T23:01:41.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.377+0000 7f9014217740 1 -- 172.21.15.12:0/2423087469 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5572f25326b0 con 0x5572f27eeba0 2024-03-23T23:01:41.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.377+0000 7f8fefe00640 1 -- 172.21.15.12:0/2423087469 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ff4019b10 con 0x5572f27eeba0 2024-03-23T23:01:41.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.377+0000 7f8fefe00640 1 -- 172.21.15.12:0/2423087469 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8ffc0f0110 con 0x5572f27eeba0 2024-03-23T23:01:41.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.377+0000 7f9014217740 10 client.25187 fetch_fsmap finished waiting for FSMap version 22 2024-03-23T23:01:41.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.378+0000 7f9014217740 10 client.25187 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-23T23:01:41.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.378+0000 7f9014217740 20 client.25187 populate_metadata read hostname 'smithi012' 2024-03-23T23:01:41.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.378+0000 7f9014217740 1 --2- 172.21.15.12:0/2423087469 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5572f2805870 0x5572f2804750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.378+0000 7f9014217740 4 client.25187 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:01:41.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.378+0000 7f9014217740 1 -- 172.21.15.12:0/2423087469 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5572f26f5520 con 0x5572f2805870 2024-03-23T23:01:41.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.378+0000 7f9003e00640 1 --2- 172.21.15.12:0/2423087469 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5572f2805870 0x5572f2804750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.378+0000 7f9003e00640 1 --2- 172.21.15.12:0/2423087469 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5572f2805870 0x5572f2804750 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f52a747317e04fdb server_cookie=a7e3df7023379fc7 in_seq=0 out_seq=0 2024-03-23T23:01:41.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.378+0000 7f8fefe00640 10 client.25187 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:01:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.379+0000 7f8fefe00640 1 -- 172.21.15.12:0/2423087469 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5572f26f5520 con 0x5572f2805870 2024-03-23T23:01:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.379+0000 7f8fefe00640 10 client.25187 handle_command_reply: tid=0 2024-03-23T23:01:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.379+0000 7f9014217740 1 client.25187 shutdown 2024-03-23T23:01:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.379+0000 7f9014217740 1 -- 172.21.15.12:0/2423087469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27eeba0 msgr2=0x5572f27f9070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.379+0000 7f9014217740 1 --2- 172.21.15.12:0/2423087469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27eeba0 0x5572f27f9070 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc0096c0 tx=0x7f8ffc02bc50 comp rx=0 tx=0).stop 2024-03-23T23:01:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.379+0000 7f9014217740 1 -- 172.21.15.12:0/2423087469 shutdown_connections 2024-03-23T23:01:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.379+0000 7f9014217740 1 --2- 172.21.15.12:0/2423087469 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5572f2805870 0x5572f2804750 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.379+0000 7f9014217740 1 --2- 172.21.15.12:0/2423087469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5572f27f4700 0x5572f28005f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.379+0000 7f9014217740 1 --2- 172.21.15.12:0/2423087469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572f27ef590 0x5572f27f95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.379+0000 7f9014217740 1 --2- 172.21.15.12:0/2423087469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27eeba0 0x5572f27f9070 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.379+0000 7f8fefe00640 0 client.25187 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:01:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.379+0000 7f9014217740 1 -- 172.21.15.12:0/2423087469 >> 172.21.15.12:0/2423087469 conn(0x5572f27ea110 msgr2=0x5572f27e9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:41.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.380+0000 7f9014217740 1 -- 172.21.15.12:0/2423087469 shutdown_connections 2024-03-23T23:01:41.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.380+0000 7f9014217740 1 -- 172.21.15.12:0/2423087469 wait complete. 2024-03-23T23:01:41.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.380+0000 7f9014217740 20 client.25187 trim_cache size 0 max 16384 2024-03-23T23:01:41.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.399+0000 7f9014217740 1 Processor -- start 2024-03-23T23:01:41.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.399+0000 7f9014217740 1 -- start start 2024-03-23T23:01:41.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.399+0000 7f9014217740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5572f27eeba0 0x5572f27ea4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.399+0000 7f9014217740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572f27ef590 0x5572f2800580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.399+0000 7f9014217740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27f4700 0x5572f27f0a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.399+0000 7f9014217740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572f2805090 con 0x5572f27eeba0 2024-03-23T23:01:41.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.399+0000 7f9014217740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572f28056f0 con 0x5572f27f4700 2024-03-23T23:01:41.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.399+0000 7f9014217740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572f2805d50 con 0x5572f27ef590 2024-03-23T23:01:41.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.399+0000 7f900aa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5572f27eeba0 0x5572f27ea4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.399+0000 7f900b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27f4700 0x5572f27f0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.399+0000 7f9003e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572f27ef590 0x5572f2800580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.399+0000 7f900b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27f4700 0x5572f27f0a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36416/0 (socket says 172.21.15.12:36416) 2024-03-23T23:01:41.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.399+0000 7f900b400640 1 -- 172.21.15.12:0/4027963499 learned_addr learned my addr 172.21.15.12:0/4027963499 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:41.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.400+0000 7f900b400640 1 -- 172.21.15.12:0/4027963499 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572f27ef590 msgr2=0x5572f2800580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.400+0000 7f900b400640 1 --2- 172.21.15.12:0/4027963499 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572f27ef590 0x5572f2800580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.400+0000 7f900b400640 1 -- 172.21.15.12:0/4027963499 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5572f27eeba0 msgr2=0x5572f27ea4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.400+0000 7f900b400640 1 --2- 172.21.15.12:0/4027963499 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5572f27eeba0 0x5572f27ea4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.400+0000 7f900b400640 1 -- 172.21.15.12:0/4027963499 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572f27ef970 con 0x5572f27f4700 2024-03-23T23:01:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.400+0000 7f900b400640 1 --2- 172.21.15.12:0/4027963499 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27f4700 0x5572f27f0a90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9004003e30 tx=0x7f900400b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8227bf5df6a87c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.400+0000 7f8ffaa00640 1 -- 172.21.15.12:0/4027963499 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90040118a0 con 0x5572f27f4700 2024-03-23T23:01:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.400+0000 7f8ffaa00640 1 -- 172.21.15.12:0/4027963499 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f900401d470 con 0x5572f27f4700 2024-03-23T23:01:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.400+0000 7f8ffaa00640 1 -- 172.21.15.12:0/4027963499 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900401e620 con 0x5572f27f4700 2024-03-23T23:01:41.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.401+0000 7f9014217740 1 -- 172.21.15.12:0/4027963499 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27f4700 msgr2=0x5572f27f0a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:41.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.401+0000 7f9014217740 1 --2- 172.21.15.12:0/4027963499 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27f4700 0x5572f27f0a90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9004003e30 tx=0x7f900400b000 comp rx=0 tx=0).stop 2024-03-23T23:01:41.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.401+0000 7f9014217740 1 -- 172.21.15.12:0/4027963499 shutdown_connections 2024-03-23T23:01:41.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.401+0000 7f9014217740 1 --2- 172.21.15.12:0/4027963499 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27f4700 0x5572f27f0a90 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.401+0000 7f9014217740 1 --2- 172.21.15.12:0/4027963499 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572f27ef590 0x5572f2800580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.401+0000 7f9014217740 1 --2- 172.21.15.12:0/4027963499 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5572f27eeba0 0x5572f27ea4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.401+0000 7f9014217740 1 -- 172.21.15.12:0/4027963499 >> 172.21.15.12:0/4027963499 conn(0x5572f27ea110 msgr2=0x5572f27eff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.401+0000 7f9014217740 1 -- 172.21.15.12:0/4027963499 shutdown_connections 2024-03-23T23:01:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.401+0000 7f9014217740 1 -- 172.21.15.12:0/4027963499 wait complete. 2024-03-23T23:01:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.402+0000 7f9014217740 1 -- start start 2024-03-23T23:01:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.402+0000 7f9014217740 1 Processor -- start 2024-03-23T23:01:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.402+0000 7f9014217740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27eeba0 0x5572f27fc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.402+0000 7f9014217740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572f27ef590 0x5572f27fccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.402+0000 7f9014217740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5572f27f4700 0x5572f27ffbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.402+0000 7f9014217740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572f2806a10 con 0x5572f27f4700 2024-03-23T23:01:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.402+0000 7f9014217740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572f28063b0 con 0x5572f27eeba0 2024-03-23T23:01:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.402+0000 7f9014217740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572f2807070 con 0x5572f27ef590 2024-03-23T23:01:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.402+0000 7f9014217740 10 client.0 _ll_register_callbacks cb 0x7f8ffc009260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:01:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.402+0000 7f900aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27eeba0 0x5572f27fc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.403+0000 7f9003e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572f27ef590 0x5572f27fccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.403+0000 7f900b400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5572f27f4700 0x5572f27ffbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.403+0000 7f900aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27eeba0 0x5572f27fc790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36424/0 (socket says 172.21.15.12:36424) 2024-03-23T23:01:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.403+0000 7f900aa00640 1 -- 172.21.15.12:0/1202712697 learned_addr learned my addr 172.21.15.12:0/1202712697 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.403+0000 7f9003e00640 1 -- 172.21.15.12:0/1202712697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27eeba0 msgr2=0x5572f27fc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.403+0000 7f9003e00640 1 --2- 172.21.15.12:0/1202712697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27eeba0 0x5572f27fc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.403+0000 7f9003e00640 1 -- 172.21.15.12:0/1202712697 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5572f27f4700 msgr2=0x5572f27ffbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.403+0000 7f9003e00640 1 --2- 172.21.15.12:0/1202712697 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5572f27f4700 0x5572f27ffbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.403+0000 7f9003e00640 1 -- 172.21.15.12:0/1202712697 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5572f28001e0 con 0x5572f27ef590 2024-03-23T23:01:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.403+0000 7f900aa00640 1 --2- 172.21.15.12:0/1202712697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27eeba0 0x5572f27fc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.403+0000 7f9003e00640 1 --2- 172.21.15.12:0/1202712697 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572f27ef590 0x5572f27fccd0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4011d10 tx=0x7f8ff4009e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=656a6a67c53768b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.403+0000 7f8fefe00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:01:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.403+0000 7f8fefe00640 1 -- 172.21.15.12:0/1202712697 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff401e040 con 0x5572f27ef590 2024-03-23T23:01:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.404+0000 7f9014217740 1 -- 172.21.15.12:0/1202712697 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f900401d470 con 0x5572f27ef590 2024-03-23T23:01:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.404+0000 7f8fefe00640 1 -- 172.21.15.12:0/1202712697 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8ff402b070 con 0x5572f27ef590 2024-03-23T23:01:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.404+0000 7f8fefe00640 1 -- 172.21.15.12:0/1202712697 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ff402c310 con 0x5572f27ef590 2024-03-23T23:01:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.404+0000 7f9014217740 10 client.25177 fetch_fsmap learned FSMap version 22 2024-03-23T23:01:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.404+0000 7f9014217740 1 -- 172.21.15.12:0/1202712697 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5572f2861360 con 0x5572f27ef590 2024-03-23T23:01:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.404+0000 7f8fefe00640 1 -- 172.21.15.12:0/1202712697 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f900401d470 con 0x5572f27ef590 2024-03-23T23:01:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.404+0000 7f8fefe00640 1 -- 172.21.15.12:0/1202712697 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8ff4035070 con 0x5572f27ef590 2024-03-23T23:01:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.404+0000 7f9014217740 10 client.25177 fetch_fsmap finished waiting for FSMap version 22 2024-03-23T23:01:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.404+0000 7f9014217740 10 client.25177 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-23T23:01:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.404+0000 7f9014217740 20 client.25177 populate_metadata read hostname 'smithi012' 2024-03-23T23:01:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.404+0000 7f9014217740 1 --2- 172.21.15.12:0/1202712697 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5572f28657c0 0x5572f2861590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.404+0000 7f9014217740 4 client.25177 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-23T23:01:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.404+0000 7f9014217740 1 -- 172.21.15.12:0/1202712697 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5572f27611d0 con 0x5572f28657c0 2024-03-23T23:01:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.405+0000 7f900aa00640 1 --2- 172.21.15.12:0/1202712697 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5572f28657c0 0x5572f2861590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.405+0000 7f900aa00640 1 --2- 172.21.15.12:0/1202712697 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5572f28657c0 0x5572f2861590 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b36de1349803bbea server_cookie=65122b690378f745 in_seq=0 out_seq=0 2024-03-23T23:01:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.405+0000 7f8fefe00640 10 client.25177 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:01:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.405+0000 7f8fefe00640 1 -- 172.21.15.12:0/1202712697 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+321 (crc 0 0 0) 0x5572f27611d0 con 0x5572f28657c0 2024-03-23T23:01:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.405+0000 7f8fefe00640 10 client.25177 handle_command_reply: tid=0 2024-03-23T23:01:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.405+0000 7f9014217740 1 client.25177 shutdown 2024-03-23T23:01:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.405+0000 7f9014217740 1 -- 172.21.15.12:0/1202712697 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572f27ef590 msgr2=0x5572f27fccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.405+0000 7f9014217740 1 --2- 172.21.15.12:0/1202712697 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572f27ef590 0x5572f27fccd0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4011d10 tx=0x7f8ff4009e50 comp rx=0 tx=0).stop 2024-03-23T23:01:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.406+0000 7f9014217740 1 -- 172.21.15.12:0/1202712697 shutdown_connections 2024-03-23T23:01:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.406+0000 7f9014217740 1 --2- 172.21.15.12:0/1202712697 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5572f28657c0 0x5572f2861590 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.406+0000 7f9014217740 1 --2- 172.21.15.12:0/1202712697 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5572f27f4700 0x5572f27ffbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.406+0000 7f9014217740 1 --2- 172.21.15.12:0/1202712697 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572f27ef590 0x5572f27fccd0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.406+0000 7f8fefe00640 0 client.25177 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:01:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.406+0000 7f9014217740 1 --2- 172.21.15.12:0/1202712697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572f27eeba0 0x5572f27fc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.406+0000 7f9014217740 1 -- 172.21.15.12:0/1202712697 >> 172.21.15.12:0/1202712697 conn(0x5572f27ea110 msgr2=0x5572f27ea4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.406+0000 7f9014217740 1 -- 172.21.15.12:0/1202712697 shutdown_connections 2024-03-23T23:01:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.406+0000 7f9014217740 1 -- 172.21.15.12:0/1202712697 wait complete. 2024-03-23T23:01:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.406+0000 7f9014217740 20 client.25177 trim_cache size 0 max 16384 2024-03-23T23:01:41.409 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:01:41.409 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "90f679ae-e967-11ee-95cd-87774f69a715", 2024-03-23T23:01:41.409 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-03-23T23:01:41.409 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 24599, 2024-03-23T23:01:41.410 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-23T23:01:41.410 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-23T23:01:41.410 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-23T23:01:41.410 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 133.968292091, 2024-03-23T23:01:41.410 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 22, 2024-03-23T23:01:41.410 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 88, 2024-03-23T23:01:41.410 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 86, 2024-03-23T23:01:41.410 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 140.55427226 2024-03-23T23:01:41.410 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:01:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.406+0000 7f9011600640 1 -- 172.21.15.12:0/813272854 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8ff00a2880 msgr2=0x7f8ff00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.406+0000 7f9011600640 1 --2- 172.21.15.12:0/813272854 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8ff00a2880 0x7f8ff00a4d40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x5572f27efdf0 tx=0x7f9004009600 comp rx=0 tx=0).stop 2024-03-23T23:01:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.407+0000 7f9011600640 1 -- 172.21.15.12:0/813272854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c1016b0 msgr2=0x7f900c199c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.407+0000 7f9011600640 1 --2- 172.21.15.12:0/813272854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c1016b0 0x7f900c199c90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc005ab0 tx=0x7f8ffc02c6e0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.407+0000 7f9011600640 1 -- 172.21.15.12:0/813272854 shutdown_connections 2024-03-23T23:01:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.407+0000 7f9011600640 1 --2- 172.21.15.12:0/813272854 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8ff00a2880 0x7f8ff00a4d40 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.407+0000 7f9011600640 1 --2- 172.21.15.12:0/813272854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f900c103260 0x7f900c1a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.407+0000 7f9011600640 1 --2- 172.21.15.12:0/813272854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f900c1028a0 0x7f900c19a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.407+0000 7f9011600640 1 --2- 172.21.15.12:0/813272854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c1016b0 0x7f900c199c90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.407+0000 7f9011600640 1 -- 172.21.15.12:0/813272854 >> 172.21.15.12:0/813272854 conn(0x7f900c0fcc40 msgr2=0x7f900c0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.407+0000 7f9011600640 1 -- 172.21.15.12:0/813272854 shutdown_connections 2024-03-23T23:01:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:41.407+0000 7f9011600640 1 -- 172.21.15.12:0/813272854 wait complete. 2024-03-23T23:01:41.424 INFO:teuthology.run_tasks:Running task kclient... 2024-03-23T23:01:41.448 INFO:tasks.kclient:Mounting kernel clients... 2024-03-23T23:01:41.448 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-23T23:01:41.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:01:41.449 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-23T23:01:41.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:01:41.508 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link delete ceph-brx 2024-03-23T23:01:41.613 INFO:teuthology.orchestra.run.smithi012.stderr:Cannot find device "ceph-brx" 2024-03-23T23:01:41.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T23:01:41.619 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-03-23T23:01:41.619 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-23T23:01:41.619 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T23:01:41.619 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-23T23:01:41.705 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:41 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2947110733' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:41.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:41 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2947110733' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:41.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:41 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2947110733' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:42.103 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:01:42.622 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:42 smithi012 ceph-mon[31695]: pgmap v387: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-23T23:01:42.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:42 smithi173 ceph-mon[36116]: pgmap v387: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-23T23:01:42.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:42 smithi203 ceph-mon[36693]: pgmap v387: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-23T23:01:44.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.232+0000 7f23ec000700 1 -- 172.21.15.12:0/207410041 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23e41096c0 msgr2=0x7f23e4111290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:44.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.232+0000 7f23ec000700 1 --2- 172.21.15.12:0/207410041 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23e41096c0 0x7f23e4111290 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f23dc00b0a0 tx=0x7f23dc00b3b0 comp rx=0 tx=0).stop 2024-03-23T23:01:44.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.232+0000 7f23ec000700 1 -- 172.21.15.12:0/207410041 shutdown_connections 2024-03-23T23:01:44.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.232+0000 7f23ec000700 1 --2- 172.21.15.12:0/207410041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23e41045e0 0x7f23e41049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:44.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.232+0000 7f23ec000700 1 --2- 172.21.15.12:0/207410041 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23e41096c0 0x7f23e4111290 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:44.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.232+0000 7f23ec000700 1 --2- 172.21.15.12:0/207410041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23e4104f90 0x7f23e4108fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:44.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.232+0000 7f23ec000700 1 -- 172.21.15.12:0/207410041 >> 172.21.15.12:0/207410041 conn(0x7f23e40ffe70 msgr2=0x7f23e4102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:44.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.232+0000 7f23ec000700 1 -- 172.21.15.12:0/207410041 shutdown_connections 2024-03-23T23:01:44.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.232+0000 7f23ec000700 1 -- 172.21.15.12:0/207410041 wait complete. 2024-03-23T23:01:44.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.234+0000 7f23ec000700 1 Processor -- start 2024-03-23T23:01:44.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.234+0000 7f23ec000700 1 -- start start 2024-03-23T23:01:44.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.234+0000 7f23ec000700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23e41045e0 0x7f23e41a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:44.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.234+0000 7f23ec000700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23e4104f90 0x7f23e41a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:44.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.234+0000 7f23ec000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23e41096c0 0x7f23e419a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:44.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.234+0000 7f23ec000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e4114700 con 0x7f23e41096c0 2024-03-23T23:01:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.234+0000 7f23ec000700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e4114580 con 0x7f23e4104f90 2024-03-23T23:01:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.234+0000 7f23ec000700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e4114880 con 0x7f23e41045e0 2024-03-23T23:01:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.235+0000 7f23e9600700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23e41045e0 0x7f23e41a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.235+0000 7f23e9600700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23e41045e0 0x7f23e41a0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55676/0 (socket says 172.21.15.12:55676) 2024-03-23T23:01:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.235+0000 7f23e9600700 1 -- 172.21.15.12:0/769897226 learned_addr learned my addr 172.21.15.12:0/769897226 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.235+0000 7f23e8c00700 1 --2- 172.21.15.12:0/769897226 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23e4104f90 0x7f23e41a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.235+0000 7f23ea000700 1 --2- 172.21.15.12:0/769897226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23e41096c0 0x7f23e419a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.235+0000 7f23e9600700 1 -- 172.21.15.12:0/769897226 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23e4104f90 msgr2=0x7f23e41a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.235+0000 7f23e9600700 1 --2- 172.21.15.12:0/769897226 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23e4104f90 0x7f23e41a0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.235+0000 7f23e9600700 1 -- 172.21.15.12:0/769897226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23e41096c0 msgr2=0x7f23e419a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.235+0000 7f23e9600700 1 --2- 172.21.15.12:0/769897226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23e41096c0 0x7f23e419a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.235+0000 7f23e9600700 1 -- 172.21.15.12:0/769897226 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e419b080 con 0x7f23e41045e0 2024-03-23T23:01:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.235+0000 7f23ea000700 1 --2- 172.21.15.12:0/769897226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23e41096c0 0x7f23e419a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:01:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.236+0000 7f23e9600700 1 --2- 172.21.15.12:0/769897226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23e41045e0 0x7f23e41a0760 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f23d800eb30 tx=0x7f23d800ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:44.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.236+0000 7f23e2000700 1 -- 172.21.15.12:0/769897226 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d800cc90 con 0x7f23e41045e0 2024-03-23T23:01:44.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.236+0000 7f23e2000700 1 -- 172.21.15.12:0/769897226 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23d8004500 con 0x7f23e41045e0 2024-03-23T23:01:44.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.236+0000 7f23ec000700 1 -- 172.21.15.12:0/769897226 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e419b370 con 0x7f23e41045e0 2024-03-23T23:01:44.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.236+0000 7f23e2000700 1 -- 172.21.15.12:0/769897226 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d8010430 con 0x7f23e41045e0 2024-03-23T23:01:44.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.236+0000 7f23ec000700 1 -- 172.21.15.12:0/769897226 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e41a7630 con 0x7f23e41045e0 2024-03-23T23:01:44.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.237+0000 7f23ec000700 1 -- 172.21.15.12:0/769897226 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23e4068e50 con 0x7f23e41045e0 2024-03-23T23:01:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.239+0000 7f23e2000700 1 -- 172.21.15.12:0/769897226 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f23d8003f70 con 0x7f23e41045e0 2024-03-23T23:01:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.240+0000 7f23e2000700 1 --2- 172.21.15.12:0/769897226 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f23cc0a2910 0x7f23cc0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.240+0000 7f23e2000700 1 -- 172.21.15.12:0/769897226 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f23d80ced70 con 0x7f23e41045e0 2024-03-23T23:01:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.241+0000 7f23e8c00700 1 --2- 172.21.15.12:0/769897226 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f23cc0a2910 0x7f23cc0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:44.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.241+0000 7f23e8c00700 1 --2- 172.21.15.12:0/769897226 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f23cc0a2910 0x7f23cc0a4dd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f23e4109020 tx=0x7f23d0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:44.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.244+0000 7f23e2000700 1 -- 172.21.15.12:0/769897226 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f23d809bdc0 con 0x7f23e41045e0 2024-03-23T23:01:44.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.528+0000 7f23ec000700 1 -- 172.21.15.12:0/769897226 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f23e41a78c0 con 0x7f23e41045e0 2024-03-23T23:01:44.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.529+0000 7f23e2000700 1 -- 172.21.15.12:0/769897226 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v22) v1 ==== 53+0+73 (secure 0 0 0) 0x7f23d809b6c0 con 0x7f23e41045e0 2024-03-23T23:01:44.529 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T23:01:44.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.534+0000 7f23ec000700 1 -- 172.21.15.12:0/769897226 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f23cc0a2910 msgr2=0x7f23cc0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:44.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.534+0000 7f23ec000700 1 --2- 172.21.15.12:0/769897226 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f23cc0a2910 0x7f23cc0a4dd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f23e4109020 tx=0x7f23d0009260 comp rx=0 tx=0).stop 2024-03-23T23:01:44.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.534+0000 7f23ec000700 1 -- 172.21.15.12:0/769897226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23e41045e0 msgr2=0x7f23e41a0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:44.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.534+0000 7f23ec000700 1 --2- 172.21.15.12:0/769897226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23e41045e0 0x7f23e41a0760 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f23d800eb30 tx=0x7f23d800ee40 comp rx=0 tx=0).stop 2024-03-23T23:01:44.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.534+0000 7f23ec000700 1 -- 172.21.15.12:0/769897226 shutdown_connections 2024-03-23T23:01:44.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.534+0000 7f23ec000700 1 --2- 172.21.15.12:0/769897226 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f23cc0a2910 0x7f23cc0a4dd0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:44.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.534+0000 7f23ec000700 1 --2- 172.21.15.12:0/769897226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23e41045e0 0x7f23e41a0760 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:44.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.534+0000 7f23ec000700 1 --2- 172.21.15.12:0/769897226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23e41096c0 0x7f23e419a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:44.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.534+0000 7f23ec000700 1 --2- 172.21.15.12:0/769897226 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23e4104f90 0x7f23e41a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:44.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.534+0000 7f23ec000700 1 -- 172.21.15.12:0/769897226 >> 172.21.15.12:0/769897226 conn(0x7f23e40ffe70 msgr2=0x7f23e41007d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:44.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.534+0000 7f23ec000700 1 -- 172.21.15.12:0/769897226 shutdown_connections 2024-03-23T23:01:44.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:44.534+0000 7f23ec000700 1 -- 172.21.15.12:0/769897226 wait complete. 2024-03-23T23:01:44.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:44 smithi012 ceph-mon[31695]: pgmap v388: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:01:44.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:44 smithi173 ceph-mon[36116]: pgmap v388: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:01:44.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:44 smithi203 ceph-mon[36693]: pgmap v388: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:01:45.352 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T23:01:45.353 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T23:01:45.353 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-03-23T23:01:45.353 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T23:01:45.353 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T23:01:45.353 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T23:01:45.353 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T23:01:45.353 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T23:01:45.353 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T23:01:45.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:01:45.353 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-03-23T23:01:45.412 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T23:01:45.412 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T23:01:45.412 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T23:01:45.412 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-23T23:01:45.412 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-23T23:01:45.412 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-23T23:01:45.412 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T23:01:45.413 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-03-23T23:01:45.413 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-03-23T23:01:45.413 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T23:01:45.413 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-03-23T23:01:45.413 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-03-23T23:01:45.413 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T23:01:45.413 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-03-23T23:01:45.413 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-03-23T23:01:45.413 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-23T23:01:45.413 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 41918sec preferred_lft 41918sec 2024-03-23T23:01:45.413 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link noprefixroute 2024-03-23T23:01:45.413 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-23T23:01:45.413 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T23:01:45.413 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-03-23T23:01:45.413 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-03-23T23:01:45.415 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-23T23:01:45.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:01:45.415 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-23T23:01:45.415 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-23T23:01:45.415 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link add name ceph-brx type bridge 2024-03-23T23:01:45.415 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip addr flush dev ceph-brx 2024-03-23T23:01:45.415 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set ceph-brx up 2024-03-23T23:01:45.415 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-23T23:01:45.415 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-23T23:01:45.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:45 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/769897226' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-23T23:01:45.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:45 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/769897226' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-23T23:01:45.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:01:45.876 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T23:01:45.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:45 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/769897226' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-23T23:01:45.967 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-03-23T23:01:45.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:01:45.971 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-03-23T23:01:46.031 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-03-23T23:01:46.031 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T23:01:46.031 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-23T23:01:46.031 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-23T23:01:46.032 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T23:01:46.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:01:46.032 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-23T23:01:46.032 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-23T23:01:46.032 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-23T23:01:46.032 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-23T23:01:46.032 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-23T23:01:46.032 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-23T23:01:46.744 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:46 smithi012 ceph-mon[31695]: pgmap v389: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-23T23:01:46.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:01:46.763 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-23T23:01:46.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:01:46.810 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list-id 2024-03-23T23:01:46.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:01:46.855 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-23T23:01:46.855 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-23T23:01:46.855 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T23:01:46.855 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-23T23:01:46.855 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-23T23:01:46.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:46 smithi173 ceph-mon[36116]: pgmap v389: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-23T23:01:46.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:46 smithi203 ceph-mon[36693]: pgmap v389: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-23T23:01:47.036 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-23T23:01:47.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:01:47.036 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-23T23:01:47.036 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-23T23:01:47.036 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-23T23:01:47.036 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-23T23:01:47.036 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-23T23:01:47.036 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-23T23:01:47.036 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-23T23:01:47.036 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-23T23:01:47.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:01:47.322 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec bash -c ' 2024-03-23T23:01:47.322 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-23T23:01:47.322 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-03-23T23:01:47.322 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set dev brx.0 master ceph-brx 2024-03-23T23:01:47.322 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-23T23:01:47.488 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T23:01:47.488 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T23:01:47.534 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T23:01:47.534 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T23:01:47.534 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T23:01:47.577 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-23T23:01:47.577 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-23T23:01:47.577 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/d55fdac7-2855-47ef-9e03-4894504668e6 2024-03-23T23:01:47.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:01:47.577 DEBUG:teuthology.orchestra.run.smithi012:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/d55fdac7-2855-47ef-9e03-4894504668e6 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-03-23T23:01:47.752 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-03-23T23:01:47.752 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2024-03-23T23:01:48.472 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-03-23T23:01:48.472 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2024-03-23T23:01:48.472 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: resolved to: "172.21.15.12:3300,172.21.15.173:3300,172.21.15.203:3300" 2024-03-23T23:01:48.472 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: trying mount with new device syntax: 0@90f679ae-e967-11ee-95cd-87774f69a715.cephfs=/volumes/qa/sv_1/d55fdac7-2855-47ef-9e03-4894504668e6 2024-03-23T23:01:48.473 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync,name=0,key=0,mon_addr=172.21.15.12:3300/172.21.15.173:3300/172.21.15.203:3300" will pass to kernel 2024-03-23T23:01:48.478 INFO:teuthology.orchestra.run.smithi012.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-23T23:01:48.478 INFO:teuthology.orchestra.run.smithi012.stdout: You just mounted a file system that supports labels which does not 2024-03-23T23:01:48.478 INFO:teuthology.orchestra.run.smithi012.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-23T23:01:48.478 INFO:teuthology.orchestra.run.smithi012.stdout: applications will generate AVC messages and not be allowed access to 2024-03-23T23:01:48.478 INFO:teuthology.orchestra.run.smithi012.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-23T23:01:48.483 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-23T23:01:48.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:01:48.483 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T23:01:48.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:01:48.589 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T23:01:48.652 INFO:teuthology.orchestra.run.smithi012.stdout:client25202 2024-03-23T23:01:48.652 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-23T23:01:48.652 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T23:01:48.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:01:48.707 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T23:01:48.763 INFO:teuthology.orchestra.run.smithi012.stdout:client25202 2024-03-23T23:01:48.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:01:48.764 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T23:01:48.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:48 smithi012 ceph-mon[31695]: pgmap v390: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-23T23:01:48.823 INFO:teuthology.orchestra.run.smithi012.stdout:client25202 2024-03-23T23:01:48.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:01:48.823 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/90f679ae-e967-11ee-95cd-87774f69a715.client25202/status') 2024-03-23T23:01:48.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:48 smithi173 ceph-mon[36116]: pgmap v390: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-23T23:01:48.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:48 smithi203 ceph-mon[36693]: pgmap v390: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-23T23:01:48.966 INFO:teuthology.orchestra.run.smithi012.stdout:instance: client.25202 (3)172.21.15.12:0/1096231714 2024-03-23T23:01:48.966 INFO:teuthology.orchestra.run.smithi012.stdout:blocklisted: false 2024-03-23T23:01:48.966 INFO:teuthology.orchestra.run.smithi012.stderr:0+1 records in 2024-03-23T23:01:48.967 INFO:teuthology.orchestra.run.smithi012.stderr:0+1 records out 2024-03-23T23:01:48.967 INFO:teuthology.orchestra.run.smithi012.stderr:71 bytes copied, 7.1291e-05 s, 996 kB/s 2024-03-23T23:01:48.971 INFO:teuthology.run_tasks:Running task exec... 2024-03-23T23:01:48.981 INFO:teuthology.task.exec:Executing custom commands... 2024-03-23T23:01:48.981 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi012.front.sepia.ceph.com 2024-03-23T23:01:48.981 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-03-23T23:01:49.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.203+0000 7fcf57000640 1 -- 172.21.15.12:0/3322973898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf50104ae0 msgr2=0x7fcf50104f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:49.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.203+0000 7fcf57000640 1 --2- 172.21.15.12:0/3322973898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf50104ae0 0x7fcf50104f60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c009900 tx=0x7fcf3c02bc40 comp rx=0 tx=0).stop 2024-03-23T23:01:49.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.203+0000 7fcf57000640 1 -- 172.21.15.12:0/3322973898 shutdown_connections 2024-03-23T23:01:49.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.203+0000 7fcf57000640 1 --2- 172.21.15.12:0/3322973898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf501054a0 0x7fcf5010bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:49.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.203+0000 7fcf57000640 1 --2- 172.21.15.12:0/3322973898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf50104ae0 0x7fcf50104f60 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:49.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.203+0000 7fcf57000640 1 --2- 172.21.15.12:0/3322973898 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf501038f0 0x7fcf50103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:49.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.203+0000 7fcf57000640 1 -- 172.21.15.12:0/3322973898 >> 172.21.15.12:0/3322973898 conn(0x7fcf500fee60 msgr2=0x7fcf501012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:49.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.203+0000 7fcf57000640 1 -- 172.21.15.12:0/3322973898 shutdown_connections 2024-03-23T23:01:49.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.203+0000 7fcf57000640 1 -- 172.21.15.12:0/3322973898 wait complete. 2024-03-23T23:01:49.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.204+0000 7fcf57000640 1 Processor -- start 2024-03-23T23:01:49.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.204+0000 7fcf57000640 1 -- start start 2024-03-23T23:01:49.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.205+0000 7fcf57000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf501038f0 0x7fcf50199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:49.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.205+0000 7fcf57000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf50104ae0 0x7fcf5019a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:49.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.205+0000 7fcf57000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf501054a0 0x7fcf501a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:49.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.205+0000 7fcf57000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5010f5c0 con 0x7fcf501038f0 2024-03-23T23:01:49.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.205+0000 7fcf57000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5010f440 con 0x7fcf501054a0 2024-03-23T23:01:49.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.205+0000 7fcf57000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5010f740 con 0x7fcf50104ae0 2024-03-23T23:01:49.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.205+0000 7fcf4fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf501038f0 0x7fcf50199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:49.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.205+0000 7fcf4f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf50104ae0 0x7fcf5019a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:49.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.205+0000 7fcf4f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf50104ae0 0x7fcf5019a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55704/0 (socket says 172.21.15.12:55704) 2024-03-23T23:01:49.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.205+0000 7fcf4f400640 1 -- 172.21.15.12:0/1143141919 learned_addr learned my addr 172.21.15.12:0/1143141919 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:49.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.205+0000 7fcf55000640 1 --2- 172.21.15.12:0/1143141919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf501054a0 0x7fcf501a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:49.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.206+0000 7fcf4f400640 1 -- 172.21.15.12:0/1143141919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf501054a0 msgr2=0x7fcf501a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:49.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.206+0000 7fcf4f400640 1 --2- 172.21.15.12:0/1143141919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf501054a0 0x7fcf501a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:49.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.206+0000 7fcf4f400640 1 -- 172.21.15.12:0/1143141919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf501038f0 msgr2=0x7fcf50199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:49.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.206+0000 7fcf4f400640 1 --2- 172.21.15.12:0/1143141919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf501038f0 0x7fcf50199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:49.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.206+0000 7fcf4f400640 1 -- 172.21.15.12:0/1143141919 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf501a1760 con 0x7fcf50104ae0 2024-03-23T23:01:49.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.206+0000 7fcf4f400640 1 --2- 172.21.15.12:0/1143141919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf50104ae0 0x7fcf5019a080 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c02c8f0 tx=0x7fcf3c02c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:49.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.206+0000 7fcf4cc00640 1 -- 172.21.15.12:0/1143141919 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c041070 con 0x7fcf50104ae0 2024-03-23T23:01:49.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.206+0000 7fcf4cc00640 1 -- 172.21.15.12:0/1143141919 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf3c004450 con 0x7fcf50104ae0 2024-03-23T23:01:49.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.206+0000 7fcf4cc00640 1 -- 172.21.15.12:0/1143141919 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c034440 con 0x7fcf50104ae0 2024-03-23T23:01:49.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.206+0000 7fcf57000640 1 -- 172.21.15.12:0/1143141919 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf501a19f0 con 0x7fcf50104ae0 2024-03-23T23:01:49.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.206+0000 7fcf57000640 1 -- 172.21.15.12:0/1143141919 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf501a1f30 con 0x7fcf50104ae0 2024-03-23T23:01:49.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.208+0000 7fcf57000640 1 -- 172.21.15.12:0/1143141919 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf5010ab40 con 0x7fcf50104ae0 2024-03-23T23:01:49.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.210+0000 7fcf4cc00640 1 -- 172.21.15.12:0/1143141919 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcf3c044070 con 0x7fcf50104ae0 2024-03-23T23:01:49.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.212+0000 7fcf4cc00640 1 --2- 172.21.15.12:0/1143141919 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcf340a2920 0x7fcf340a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:49.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.212+0000 7fcf4cc00640 1 -- 172.21.15.12:0/1143141919 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fcf3c0f0930 con 0x7fcf50104ae0 2024-03-23T23:01:49.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.212+0000 7fcf4fe00640 1 --2- 172.21.15.12:0/1143141919 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcf340a2920 0x7fcf340a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:49.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.213+0000 7fcf4fe00640 1 --2- 172.21.15.12:0/1143141919 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcf340a2920 0x7fcf340a4de0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcf50104940 tx=0x7fcf40009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:49.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.213+0000 7fcf4cc00640 1 -- 172.21.15.12:0/1143141919 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcf3c0bd980 con 0x7fcf50104ae0 2024-03-23T23:01:49.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.461+0000 7fcf57000640 1 -- 172.21.15.12:0/1143141919 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) v1 -- 0x7fcf5002d070 con 0x7fcf50104ae0 2024-03-23T23:01:49.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.527+0000 7fcf4cc00640 1 -- 172.21.15.12:0/1143141919 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v23) v1 ==== 117+0+0 (secure 0 0 0) 0x7fcf3c0bd280 con 0x7fcf50104ae0 2024-03-23T23:01:49.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.530+0000 7fcf57000640 1 -- 172.21.15.12:0/1143141919 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcf340a2920 msgr2=0x7fcf340a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:49.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.530+0000 7fcf57000640 1 --2- 172.21.15.12:0/1143141919 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcf340a2920 0x7fcf340a4de0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcf50104940 tx=0x7fcf40009500 comp rx=0 tx=0).stop 2024-03-23T23:01:49.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.530+0000 7fcf57000640 1 -- 172.21.15.12:0/1143141919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf50104ae0 msgr2=0x7fcf5019a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:49.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.530+0000 7fcf57000640 1 --2- 172.21.15.12:0/1143141919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf50104ae0 0x7fcf5019a080 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c02c8f0 tx=0x7fcf3c02c660 comp rx=0 tx=0).stop 2024-03-23T23:01:49.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.530+0000 7fcf57000640 1 -- 172.21.15.12:0/1143141919 shutdown_connections 2024-03-23T23:01:49.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.530+0000 7fcf57000640 1 --2- 172.21.15.12:0/1143141919 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcf340a2920 0x7fcf340a4de0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:49.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.530+0000 7fcf57000640 1 --2- 172.21.15.12:0/1143141919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf501054a0 0x7fcf501a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:49.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.530+0000 7fcf57000640 1 --2- 172.21.15.12:0/1143141919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf50104ae0 0x7fcf5019a080 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:49.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.530+0000 7fcf57000640 1 --2- 172.21.15.12:0/1143141919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf501038f0 0x7fcf50199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:49.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.530+0000 7fcf57000640 1 -- 172.21.15.12:0/1143141919 >> 172.21.15.12:0/1143141919 conn(0x7fcf500fee60 msgr2=0x7fcf50107d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:49.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.530+0000 7fcf57000640 1 -- 172.21.15.12:0/1143141919 shutdown_connections 2024-03-23T23:01:49.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.531+0000 7fcf57000640 1 -- 172.21.15.12:0/1143141919 wait complete. 2024-03-23T23:01:49.550 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-23T23:01:49.561 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-23T23:01:49.573 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T23:01:49.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.800+0000 7f0739800640 1 -- 172.21.15.12:0/809630163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0734104ad0 msgr2=0x7f0734104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:49.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.800+0000 7f0739800640 1 --2- 172.21.15.12:0/809630163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0734104ad0 0x7f0734104f50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0724009ca0 tx=0x7f072402c100 comp rx=0 tx=0).stop 2024-03-23T23:01:49.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.800+0000 7f0739800640 1 -- 172.21.15.12:0/809630163 shutdown_connections 2024-03-23T23:01:49.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.800+0000 7f0739800640 1 --2- 172.21.15.12:0/809630163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0734105490 0x7f073410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:49.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.800+0000 7f0739800640 1 --2- 172.21.15.12:0/809630163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0734104ad0 0x7f0734104f50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:49.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.800+0000 7f0739800640 1 --2- 172.21.15.12:0/809630163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07341038e0 0x7f0734103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:49.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.800+0000 7f0739800640 1 -- 172.21.15.12:0/809630163 >> 172.21.15.12:0/809630163 conn(0x7f07340fee50 msgr2=0x7f0734101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:49.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.801+0000 7f0739800640 1 -- 172.21.15.12:0/809630163 shutdown_connections 2024-03-23T23:01:49.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.801+0000 7f0739800640 1 -- 172.21.15.12:0/809630163 wait complete. 2024-03-23T23:01:49.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.802+0000 7f0739800640 1 Processor -- start 2024-03-23T23:01:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.802+0000 7f0739800640 1 -- start start 2024-03-23T23:01:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.802+0000 7f0739800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07341038e0 0x7f0734199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.802+0000 7f0739800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0734104ad0 0x7f073419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.802+0000 7f0739800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0734105490 0x7f07341a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.802+0000 7f0739800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073410f560 con 0x7f07341038e0 2024-03-23T23:01:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.802+0000 7f0739800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073410f3e0 con 0x7f0734105490 2024-03-23T23:01:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.802+0000 7f0739800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073410f6e0 con 0x7f0734104ad0 2024-03-23T23:01:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.802+0000 7f0732a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07341038e0 0x7f0734199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.802+0000 7f0732000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0734104ad0 0x7f073419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.802+0000 7f0732000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0734104ad0 0x7f073419a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55726/0 (socket says 172.21.15.12:55726) 2024-03-23T23:01:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.802+0000 7f0733400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0734105490 0x7f07341a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.802+0000 7f0732000640 1 -- 172.21.15.12:0/151334985 learned_addr learned my addr 172.21.15.12:0/151334985 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:49.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.802+0000 7f0732a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07341038e0 0x7f0734199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36280/0 (socket says 172.21.15.12:36280) 2024-03-23T23:01:49.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.803+0000 7f0732000640 1 -- 172.21.15.12:0/151334985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0734105490 msgr2=0x7f07341a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:49.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.803+0000 7f0732000640 1 --2- 172.21.15.12:0/151334985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0734105490 0x7f07341a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:49.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.803+0000 7f0732000640 1 -- 172.21.15.12:0/151334985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07341038e0 msgr2=0x7f0734199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:49.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.803+0000 7f0732000640 1 --2- 172.21.15.12:0/151334985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07341038e0 0x7f0734199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:49.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.803+0000 7f0732000640 1 -- 172.21.15.12:0/151334985 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07341a17c0 con 0x7f0734104ad0 2024-03-23T23:01:49.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:49 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1143141919' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-23T23:01:49.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:49 smithi012 ceph-mon[31695]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-23T23:01:49.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.803+0000 7f0732000640 1 --2- 172.21.15.12:0/151334985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0734104ad0 0x7f073419a0e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f07240325f0 tx=0x7f0724035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:49.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.804+0000 7f072b400640 1 -- 172.21.15.12:0/151334985 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0724039040 con 0x7f0734104ad0 2024-03-23T23:01:49.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.804+0000 7f072b400640 1 -- 172.21.15.12:0/151334985 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0724044070 con 0x7f0734104ad0 2024-03-23T23:01:49.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.804+0000 7f072b400640 1 -- 172.21.15.12:0/151334985 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07240044d0 con 0x7f0734104ad0 2024-03-23T23:01:49.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.804+0000 7f0739800640 1 -- 172.21.15.12:0/151334985 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07341a1a50 con 0x7f0734104ad0 2024-03-23T23:01:49.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.804+0000 7f0739800640 1 -- 172.21.15.12:0/151334985 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07341a1f60 con 0x7f0734104ad0 2024-03-23T23:01:49.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.805+0000 7f0739800640 1 -- 172.21.15.12:0/151334985 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f073410ac50 con 0x7f0734104ad0 2024-03-23T23:01:49.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.807+0000 7f072b400640 1 -- 172.21.15.12:0/151334985 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0724004670 con 0x7f0734104ad0 2024-03-23T23:01:49.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.808+0000 7f072b400640 1 --2- 172.21.15.12:0/151334985 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f07180a2880 0x7f07180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:49.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.808+0000 7f072b400640 1 -- 172.21.15.12:0/151334985 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f07240f0f80 con 0x7f0734104ad0 2024-03-23T23:01:49.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.809+0000 7f0732a00640 1 --2- 172.21.15.12:0/151334985 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f07180a2880 0x7f07180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:49.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.809+0000 7f0732a00640 1 --2- 172.21.15.12:0/151334985 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f07180a2880 0x7f07180a4d40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0734104930 tx=0x7f071c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:49.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:49.812+0000 7f072b400640 1 -- 172.21.15.12:0/151334985 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f07240bdfd0 con 0x7f0734104ad0 2024-03-23T23:01:49.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:49 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1143141919' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-23T23:01:49.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:49 smithi173 ceph-mon[36116]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-23T23:01:49.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:49 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1143141919' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-23T23:01:49.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:49 smithi203 ceph-mon[36693]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-23T23:01:50.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.019+0000 7f0739800640 1 -- 172.21.15.12:0/151334985 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07340622a0 con 0x7f0734104ad0 2024-03-23T23:01:50.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.021+0000 7f072b400640 1 -- 172.21.15.12:0/151334985 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22695 (secure 0 0 0) 0x7f07240bd8d0 con 0x7f0734104ad0 2024-03-23T23:01:50.022 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:01:50.023 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":88,"fsid":"90f679ae-e967-11ee-95cd-87774f69a715","created":"2024-03-23T22:50:14.906952+0000","modified":"2024-03-23T23:00:24.349052+0000","last_up_change":"2024-03-23T22:58:28.907824+0000","last_in_change":"2024-03-23T22:58:08.342487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T22:54:11.831010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T22:59:25.350974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T22:59:26.390323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b70875d7-a23d-4251-9e8b-1a16d4edc2a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6803","nonce":1190079611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6805","nonce":1190079611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6809","nonce":1190079611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6807","nonce":1190079611}]},"public_addr":"172.21.15.12:6803/1190079611","cluster_addr":"172.21.15.12:6805/1190079611","heartbeat_back_addr":"172.21.15.12:6809/1190079611","heartbeat_front_addr":"172.21.15.12:6807/1190079611","state":["exists","up"]},{"osd":1,"uuid":"416e87af-db5d-41ce-84d9-731e3e07e3c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6801","nonce":1983639525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6803","nonce":1983639525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6807","nonce":1983639525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6805","nonce":1983639525}]},"public_addr":"172.21.15.173:6801/1983639525","cluster_addr":"172.21.15.173:6803/1983639525","heartbeat_back_addr":"172.21.15.173:6807/1983639525","heartbeat_front_addr":"172.21.15.173:6805/1983639525","state":["exists","up"]},{"osd":2,"uuid":"7a559c70-4c54-431b-8baf-4136b6e76493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6801","nonce":2253997387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6803","nonce":2253997387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6807","nonce":2253997387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6805","nonce":2253997387}]},"public_addr":"172.21.15.203:6801/2253997387","cluster_addr":"172.21.15.203:6803/2253997387","heartbeat_back_addr":"172.21.15.203:6807/2253997387","heartbeat_front_addr":"172.21.15.203:6805/2253997387","state":["exists","up"]},{"osd":3,"uuid":"c41cd287-a674-486c-8025-328cd36e5ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6811","nonce":1761446905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6813","nonce":1761446905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6817","nonce":1761446905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6815","nonce":1761446905}]},"public_addr":"172.21.15.12:6811/1761446905","cluster_addr":"172.21.15.12:6813/1761446905","heartbeat_back_addr":"172.21.15.12:6817/1761446905","heartbeat_front_addr":"172.21.15.12:6815/1761446905","state":["exists","up"]},{"osd":4,"uuid":"3c48f886-0b42-4a09-b31e-abccc200bb1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6809","nonce":242958272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6811","nonce":242958272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6815","nonce":242958272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6813","nonce":242958272}]},"public_addr":"172.21.15.173:6809/242958272","cluster_addr":"172.21.15.173:6811/242958272","heartbeat_back_addr":"172.21.15.173:6815/242958272","heartbeat_front_addr":"172.21.15.173:6813/242958272","state":["exists","up"]},{"osd":5,"uuid":"4a34f06f-8805-40c0-b1ab-23407b0276d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6809","nonce":4270966092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6811","nonce":4270966092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6815","nonce":4270966092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6813","nonce":4270966092}]},"public_addr":"172.21.15.203:6809/4270966092","cluster_addr":"172.21.15.203:6811/4270966092","heartbeat_back_addr":"172.21.15.203:6815/4270966092","heartbeat_front_addr":"172.21.15.203:6813/4270966092","state":["exists","up"]},{"osd":6,"uuid":"26ddc663-5623-48c9-80d1-f8b968765d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6819","nonce":3606839295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6821","nonce":3606839295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6825","nonce":3606839295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6823","nonce":3606839295}]},"public_addr":"172.21.15.12:6819/3606839295","cluster_addr":"172.21.15.12:6821/3606839295","heartbeat_back_addr":"172.21.15.12:6825/3606839295","heartbeat_front_addr":"172.21.15.12:6823/3606839295","state":["exists","up"]},{"osd":7,"uuid":"17dc9417-71cd-4576-aa08-933212ec55e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6817","nonce":1241504346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6819","nonce":1241504346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6823","nonce":1241504346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6821","nonce":1241504346}]},"public_addr":"172.21.15.173:6817/1241504346","cluster_addr":"172.21.15.173:6819/1241504346","heartbeat_back_addr":"172.21.15.173:6823/1241504346","heartbeat_front_addr":"172.21.15.173:6821/1241504346","state":["exists","up"]},{"osd":8,"uuid":"90a2b63a-471d-43b8-ac18-bb1f16778d18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6817","nonce":1516996192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6819","nonce":1516996192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6823","nonce":1516996192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6821","nonce":1516996192}]},"public_addr":"172.21.15.203:6817/1516996192","cluster_addr":"172.21.15.203:6819/1516996192","heartbeat_back_addr":"172.21.15.203:6823/1516996192","heartbeat_front_addr":"172.21.15.203:6821/1516996192","state":["exists","up"]},{"osd":9,"uuid":"6e1ca504-40b4-49d0-b245-7946181abfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6827","nonce":1175260774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6829","nonce":1175260774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6833","nonce":1175260774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6831","nonce":1175260774}]},"public_addr":"172.21.15.12:6827/1175260774","cluster_addr":"172.21.15.12:6829/1175260774","heartbeat_back_addr":"172.21.15.12:6833/1175260774","heartbeat_front_addr":"172.21.15.12:6831/1175260774","state":["exists","up"]},{"osd":10,"uuid":"09a595ed-5614-4386-8e61-0170af66e01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6825","nonce":3303506808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6827","nonce":3303506808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6831","nonce":3303506808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6829","nonce":3303506808}]},"public_addr":"172.21.15.173:6825/3303506808","cluster_addr":"172.21.15.173:6827/3303506808","heartbeat_back_addr":"172.21.15.173:6831/3303506808","heartbeat_front_addr":"172.21.15.173:6829/3303506808","state":["exists","up"]},{"osd":11,"uuid":"9ef1d595-a977-4ba7-9c6b-d2b3295f20b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6825","nonce":2323604506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6827","nonce":2323604506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6831","nonce":2323604506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6829","nonce":2323604506}]},"public_addr":"172.21.15.203:6825/2323604506","cluster_addr":"172.21.15.203:6827/2323604506","heartbeat_back_addr":"172.21.15.203:6831/2323604506","heartbeat_front_addr":"172.21.15.203:6829/2323604506","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:18.469038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:42.752628+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:07.069493+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:41.962478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:08.363104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:33.341336+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:07.286541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:34.243302+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:59.749699+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:57:33.822913+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:00.737269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:26.599854+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/4182783120":"2024-03-24T22:51:23.113201+0000","172.21.15.12:6800/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1900938637":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/3414861652":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6801/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/2580541477":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6800/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/501058839":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1192554716":"2024-03-24T22:50:48.401223+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T23:01:50.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.024+0000 7f0739800640 1 -- 172.21.15.12:0/151334985 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f07180a2880 msgr2=0x7f07180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:50.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.024+0000 7f0739800640 1 --2- 172.21.15.12:0/151334985 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f07180a2880 0x7f07180a4d40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0734104930 tx=0x7f071c005f90 comp rx=0 tx=0).stop 2024-03-23T23:01:50.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.024+0000 7f0739800640 1 -- 172.21.15.12:0/151334985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0734104ad0 msgr2=0x7f073419a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:50.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.024+0000 7f0739800640 1 --2- 172.21.15.12:0/151334985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0734104ad0 0x7f073419a0e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f07240325f0 tx=0x7f0724035d10 comp rx=0 tx=0).stop 2024-03-23T23:01:50.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.024+0000 7f0739800640 1 -- 172.21.15.12:0/151334985 shutdown_connections 2024-03-23T23:01:50.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.024+0000 7f0739800640 1 --2- 172.21.15.12:0/151334985 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f07180a2880 0x7f07180a4d40 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:50.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.024+0000 7f0739800640 1 --2- 172.21.15.12:0/151334985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0734105490 0x7f07341a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:50.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.024+0000 7f0739800640 1 --2- 172.21.15.12:0/151334985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0734104ad0 0x7f073419a0e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:50.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.024+0000 7f0739800640 1 --2- 172.21.15.12:0/151334985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07341038e0 0x7f0734199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:50.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.024+0000 7f0739800640 1 -- 172.21.15.12:0/151334985 >> 172.21.15.12:0/151334985 conn(0x7f07340fee50 msgr2=0x7f0734107e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:50.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.024+0000 7f0739800640 1 -- 172.21.15.12:0/151334985 shutdown_connections 2024-03-23T23:01:50.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.025+0000 7f0739800640 1 -- 172.21.15.12:0/151334985 wait complete. 2024-03-23T23:01:50.045 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T22:54:11.831010+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T22:59:25.350974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T22:59:26.390323+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T23:01:50.046 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T23:01:50.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.273+0000 7fcbd4c00640 1 -- 172.21.15.12:0/3135306985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbd00ffe70 msgr2=0x7fcbd0100290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:50.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.273+0000 7fcbd4c00640 1 --2- 172.21.15.12:0/3135306985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbd00ffe70 0x7fcbd0100290 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0009ca0 tx=0x7fcbc002c180 comp rx=0 tx=0).stop 2024-03-23T23:01:50.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.273+0000 7fcbd4c00640 1 -- 172.21.15.12:0/3135306985 shutdown_connections 2024-03-23T23:01:50.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.273+0000 7fcbd4c00640 1 --2- 172.21.15.12:0/3135306985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbd0101170 0x7fcbd01015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:50.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.273+0000 7fcbd4c00640 1 --2- 172.21.15.12:0/3135306985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbd01007d0 0x7fcbd0100c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:50.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.273+0000 7fcbd4c00640 1 --2- 172.21.15.12:0/3135306985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbd00ffe70 0x7fcbd0100290 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:50.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.273+0000 7fcbd4c00640 1 -- 172.21.15.12:0/3135306985 >> 172.21.15.12:0/3135306985 conn(0x7fcbd00fba10 msgr2=0x7fcbd00fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:50.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.273+0000 7fcbd4c00640 1 -- 172.21.15.12:0/3135306985 shutdown_connections 2024-03-23T23:01:50.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.274+0000 7fcbd4c00640 1 -- 172.21.15.12:0/3135306985 wait complete. 2024-03-23T23:01:50.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.274+0000 7fcbd4c00640 1 Processor -- start 2024-03-23T23:01:50.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.275+0000 7fcbd4c00640 1 -- start start 2024-03-23T23:01:50.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.275+0000 7fcbd4c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbd00ffe70 0x7fcbd0199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:50.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.275+0000 7fcbd4c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbd01007d0 0x7fcbd019a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:50.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.275+0000 7fcbd4c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbd0101170 0x7fcbd01a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:50.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.275+0000 7fcbd4c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd010f3d0 con 0x7fcbd01007d0 2024-03-23T23:01:50.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.275+0000 7fcbd4c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd010f250 con 0x7fcbd0101170 2024-03-23T23:01:50.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.275+0000 7fcbd4c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd010f550 con 0x7fcbd00ffe70 2024-03-23T23:01:50.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.275+0000 7fcbcf400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbd00ffe70 0x7fcbd0199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:50.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.275+0000 7fcbcf400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbd00ffe70 0x7fcbd0199b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55742/0 (socket says 172.21.15.12:55742) 2024-03-23T23:01:50.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.275+0000 7fcbcf400640 1 -- 172.21.15.12:0/2726455268 learned_addr learned my addr 172.21.15.12:0/2726455268 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:50.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.275+0000 7fcbc6a00640 1 --2- 172.21.15.12:0/2726455268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbd01007d0 0x7fcbd019a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:50.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.275+0000 7fcbcfe00640 1 --2- 172.21.15.12:0/2726455268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbd0101170 0x7fcbd01a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:50.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.276+0000 7fcbcf400640 1 -- 172.21.15.12:0/2726455268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbd0101170 msgr2=0x7fcbd01a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:50.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.276+0000 7fcbcf400640 1 --2- 172.21.15.12:0/2726455268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbd0101170 0x7fcbd01a10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:50.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.276+0000 7fcbcf400640 1 -- 172.21.15.12:0/2726455268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbd01007d0 msgr2=0x7fcbd019a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:50.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.276+0000 7fcbcf400640 1 --2- 172.21.15.12:0/2726455268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbd01007d0 0x7fcbd019a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:50.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.276+0000 7fcbcf400640 1 -- 172.21.15.12:0/2726455268 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd01a1740 con 0x7fcbd00ffe70 2024-03-23T23:01:50.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.276+0000 7fcbcf400640 1 --2- 172.21.15.12:0/2726455268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbd00ffe70 0x7fcbd0199b70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcbc002c790 tx=0x7fcbc0009360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:50.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.276+0000 7fcbccc00640 1 -- 172.21.15.12:0/2726455268 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0039040 con 0x7fcbd00ffe70 2024-03-23T23:01:50.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.276+0000 7fcbccc00640 1 -- 172.21.15.12:0/2726455268 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbc0044070 con 0x7fcbd00ffe70 2024-03-23T23:01:50.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.276+0000 7fcbccc00640 1 -- 172.21.15.12:0/2726455268 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0004330 con 0x7fcbd00ffe70 2024-03-23T23:01:50.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.276+0000 7fcbd4c00640 1 -- 172.21.15.12:0/2726455268 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbd01a19d0 con 0x7fcbd00ffe70 2024-03-23T23:01:50.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.276+0000 7fcbd4c00640 1 -- 172.21.15.12:0/2726455268 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbd01a1f10 con 0x7fcbd00ffe70 2024-03-23T23:01:50.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.278+0000 7fcbd4c00640 1 -- 172.21.15.12:0/2726455268 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbd010abc0 con 0x7fcbd00ffe70 2024-03-23T23:01:50.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.279+0000 7fcbccc00640 1 -- 172.21.15.12:0/2726455268 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcbc00044d0 con 0x7fcbd00ffe70 2024-03-23T23:01:50.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.280+0000 7fcbccc00640 1 --2- 172.21.15.12:0/2726455268 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcbb40a2880 0x7fcbb40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:50.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.281+0000 7fcbccc00640 1 -- 172.21.15.12:0/2726455268 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fcbc00f0fd0 con 0x7fcbd00ffe70 2024-03-23T23:01:50.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.281+0000 7fcbc6a00640 1 --2- 172.21.15.12:0/2726455268 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcbb40a2880 0x7fcbb40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:50.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.281+0000 7fcbc6a00640 1 --2- 172.21.15.12:0/2726455268 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcbb40a2880 0x7fcbb40a4d40 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fcbd019b030 tx=0x7fcbb8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:50.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.285+0000 7fcbccc00640 1 -- 172.21.15.12:0/2726455268 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcbc00be0b0 con 0x7fcbd00ffe70 2024-03-23T23:01:50.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.491+0000 7fcbd4c00640 1 -- 172.21.15.12:0/2726455268 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcbd00622a0 con 0x7fcbd00ffe70 2024-03-23T23:01:50.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.492+0000 7fcbccc00640 1 -- 172.21.15.12:0/2726455268 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcbc00bd9b0 con 0x7fcbd00ffe70 2024-03-23T23:01:50.492 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-23T23:01:50.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.494+0000 7fcbd4c00640 1 -- 172.21.15.12:0/2726455268 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcbb40a2880 msgr2=0x7fcbb40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:50.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.494+0000 7fcbd4c00640 1 --2- 172.21.15.12:0/2726455268 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcbb40a2880 0x7fcbb40a4d40 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fcbd019b030 tx=0x7fcbb8005f90 comp rx=0 tx=0).stop 2024-03-23T23:01:50.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.494+0000 7fcbd4c00640 1 -- 172.21.15.12:0/2726455268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbd00ffe70 msgr2=0x7fcbd0199b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:50.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.494+0000 7fcbd4c00640 1 --2- 172.21.15.12:0/2726455268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbd00ffe70 0x7fcbd0199b70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcbc002c790 tx=0x7fcbc0009360 comp rx=0 tx=0).stop 2024-03-23T23:01:50.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.494+0000 7fcbd4c00640 1 -- 172.21.15.12:0/2726455268 shutdown_connections 2024-03-23T23:01:50.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.494+0000 7fcbd4c00640 1 --2- 172.21.15.12:0/2726455268 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcbb40a2880 0x7fcbb40a4d40 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:50.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.494+0000 7fcbd4c00640 1 --2- 172.21.15.12:0/2726455268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbd0101170 0x7fcbd01a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:50.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.494+0000 7fcbd4c00640 1 --2- 172.21.15.12:0/2726455268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbd01007d0 0x7fcbd019a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:50.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.494+0000 7fcbd4c00640 1 --2- 172.21.15.12:0/2726455268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbd00ffe70 0x7fcbd0199b70 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:50.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.494+0000 7fcbd4c00640 1 -- 172.21.15.12:0/2726455268 >> 172.21.15.12:0/2726455268 conn(0x7fcbd00fba10 msgr2=0x7fcbd00fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:50.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.495+0000 7fcbd4c00640 1 -- 172.21.15.12:0/2726455268 shutdown_connections 2024-03-23T23:01:50.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.495+0000 7fcbd4c00640 1 -- 172.21.15.12:0/2726455268 wait complete. 2024-03-23T23:01:50.515 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T23:01:50.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.745+0000 7f7510600640 1 -- 172.21.15.12:0/1603943183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7508103260 msgr2=0x7f7508109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:50.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.745+0000 7f7510600640 1 --2- 172.21.15.12:0/1603943183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7508103260 0x7f7508109af0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7500009ca0 tx=0x7f750002c100 comp rx=0 tx=0).stop 2024-03-23T23:01:50.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.745+0000 7f7510600640 1 -- 172.21.15.12:0/1603943183 shutdown_connections 2024-03-23T23:01:50.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.745+0000 7f7510600640 1 --2- 172.21.15.12:0/1603943183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7508103260 0x7f7508109af0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:50.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.745+0000 7f7510600640 1 --2- 172.21.15.12:0/1603943183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75081028a0 0x7f7508102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:50.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.745+0000 7f7510600640 1 --2- 172.21.15.12:0/1603943183 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75081016b0 0x7f7508101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:50.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.745+0000 7f7510600640 1 -- 172.21.15.12:0/1603943183 >> 172.21.15.12:0/1603943183 conn(0x7f75080fcc40 msgr2=0x7f75080ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:50.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.745+0000 7f7510600640 1 -- 172.21.15.12:0/1603943183 shutdown_connections 2024-03-23T23:01:50.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.745+0000 7f7510600640 1 -- 172.21.15.12:0/1603943183 wait complete. 2024-03-23T23:01:50.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.746+0000 7f7510600640 1 Processor -- start 2024-03-23T23:01:50.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.746+0000 7f7510600640 1 -- start start 2024-03-23T23:01:50.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.747+0000 7f7510600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75081016b0 0x7f7508197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:50.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.747+0000 7f7510600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75081028a0 0x7f7508197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:50.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.747+0000 7f7510600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7508103260 0x7f750819ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:50.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.747+0000 7f7510600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750810d310 con 0x7f7508103260 2024-03-23T23:01:50.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.747+0000 7f7510600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750810d190 con 0x7f75081028a0 2024-03-23T23:01:50.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.747+0000 7f7510600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750810d490 con 0x7f75081016b0 2024-03-23T23:01:50.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.747+0000 7f750dc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75081016b0 0x7f7508197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:50.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.747+0000 7f750d200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75081028a0 0x7f7508197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:50.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.747+0000 7f750d200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75081028a0 0x7f7508197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:46834/0 (socket says 172.21.15.12:46834) 2024-03-23T23:01:50.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.747+0000 7f750dc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75081016b0 0x7f7508197950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55762/0 (socket says 172.21.15.12:55762) 2024-03-23T23:01:50.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.747+0000 7f750d200640 1 -- 172.21.15.12:0/561746498 learned_addr learned my addr 172.21.15.12:0/561746498 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:50.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.747+0000 7f750e600640 1 --2- 172.21.15.12:0/561746498 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7508103260 0x7f750819ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:50.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.747+0000 7f750d200640 1 -- 172.21.15.12:0/561746498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75081016b0 msgr2=0x7f7508197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:50.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.747+0000 7f750d200640 1 --2- 172.21.15.12:0/561746498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75081016b0 0x7f7508197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:50.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.747+0000 7f750d200640 1 -- 172.21.15.12:0/561746498 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7508103260 msgr2=0x7f750819ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:50.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.747+0000 7f750d200640 1 --2- 172.21.15.12:0/561746498 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7508103260 0x7f750819ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:50.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.747+0000 7f750d200640 1 -- 172.21.15.12:0/561746498 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750819f570 con 0x7f75081028a0 2024-03-23T23:01:50.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.747+0000 7f750e600640 1 --2- 172.21.15.12:0/561746498 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7508103260 0x7f750819ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:01:50.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.747+0000 7f750dc00640 1 --2- 172.21.15.12:0/561746498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75081016b0 0x7f7508197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:50.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.748+0000 7f750d200640 1 --2- 172.21.15.12:0/561746498 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75081028a0 0x7f7508197e90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f74f800e9f0 tx=0x7f74f800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:50.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.748+0000 7f7506000640 1 -- 172.21.15.12:0/561746498 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f800cd80 con 0x7f75081028a0 2024-03-23T23:01:50.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.748+0000 7f7506000640 1 -- 172.21.15.12:0/561746498 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f74f8010910 con 0x7f75081028a0 2024-03-23T23:01:50.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.748+0000 7f7510600640 1 -- 172.21.15.12:0/561746498 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f750819f860 con 0x7f75081028a0 2024-03-23T23:01:50.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.748+0000 7f7506000640 1 -- 172.21.15.12:0/561746498 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f8018990 con 0x7f75081028a0 2024-03-23T23:01:50.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.748+0000 7f7510600640 1 -- 172.21.15.12:0/561746498 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f750819fe00 con 0x7f75081028a0 2024-03-23T23:01:50.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.749+0000 7f7510600640 1 -- 172.21.15.12:0/561746498 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7508108970 con 0x7f75081028a0 2024-03-23T23:01:50.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.751+0000 7f7506000640 1 -- 172.21.15.12:0/561746498 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f74f8010ac0 con 0x7f75081028a0 2024-03-23T23:01:50.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.752+0000 7f7506000640 1 --2- 172.21.15.12:0/561746498 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f74f00a2880 0x7f74f00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:50.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.753+0000 7f7506000640 1 -- 172.21.15.12:0/561746498 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f74f80cee10 con 0x7f75081028a0 2024-03-23T23:01:50.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.753+0000 7f750dc00640 1 --2- 172.21.15.12:0/561746498 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f74f00a2880 0x7f74f00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:50.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.753+0000 7f750dc00640 1 --2- 172.21.15.12:0/561746498 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f74f00a2880 0x7f74f00a4d40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7508102700 tx=0x7f74fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:50.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.757+0000 7f7506000640 1 -- 172.21.15.12:0/561746498 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f74f80d2030 con 0x7f75081028a0 2024-03-23T23:01:50.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:50 smithi012 ceph-mon[31695]: pgmap v391: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:01:50.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:50 smithi012 ceph-mon[31695]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-23T23:01:50.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:50 smithi012 ceph-mon[31695]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-23T23:01:50.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:50 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/151334985' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:50.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:50 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2726455268' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:01:50.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:50 smithi173 ceph-mon[36116]: pgmap v391: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:01:50.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:50 smithi173 ceph-mon[36116]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-23T23:01:50.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:50 smithi173 ceph-mon[36116]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-23T23:01:50.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:50 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/151334985' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:50.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:50 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2726455268' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:01:50.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:50 smithi203 ceph-mon[36693]: pgmap v391: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:01:50.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:50 smithi203 ceph-mon[36693]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-23T23:01:50.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:50 smithi203 ceph-mon[36693]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-23T23:01:50.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:50 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/151334985' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:50.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:50 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2726455268' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:01:50.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.965+0000 7f7510600640 1 -- 172.21.15.12:0/561746498 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f75080622a0 con 0x7f75081028a0 2024-03-23T23:01:50.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.966+0000 7f7506000640 1 -- 172.21.15.12:0/561746498 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f74f809be60 con 0x7f75081028a0 2024-03-23T23:01:50.966 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-23T23:01:50.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.968+0000 7f7510600640 1 -- 172.21.15.12:0/561746498 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f74f00a2880 msgr2=0x7f74f00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:50.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.968+0000 7f7510600640 1 --2- 172.21.15.12:0/561746498 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f74f00a2880 0x7f74f00a4d40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7508102700 tx=0x7f74fc005f90 comp rx=0 tx=0).stop 2024-03-23T23:01:50.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.968+0000 7f7510600640 1 -- 172.21.15.12:0/561746498 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75081028a0 msgr2=0x7f7508197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:50.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.968+0000 7f7510600640 1 --2- 172.21.15.12:0/561746498 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75081028a0 0x7f7508197e90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f74f800e9f0 tx=0x7f74f800eeb0 comp rx=0 tx=0).stop 2024-03-23T23:01:50.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.968+0000 7f7510600640 1 -- 172.21.15.12:0/561746498 shutdown_connections 2024-03-23T23:01:50.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.968+0000 7f7510600640 1 --2- 172.21.15.12:0/561746498 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f74f00a2880 0x7f74f00a4d40 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:50.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.968+0000 7f7510600640 1 --2- 172.21.15.12:0/561746498 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7508103260 0x7f750819ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:50.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.968+0000 7f7510600640 1 --2- 172.21.15.12:0/561746498 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75081028a0 0x7f7508197e90 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:50.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.968+0000 7f7510600640 1 --2- 172.21.15.12:0/561746498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75081016b0 0x7f7508197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:50.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.968+0000 7f7510600640 1 -- 172.21.15.12:0/561746498 >> 172.21.15.12:0/561746498 conn(0x7f75080fcc40 msgr2=0x7f7508105b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:50.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.969+0000 7f7510600640 1 -- 172.21.15.12:0/561746498 shutdown_connections 2024-03-23T23:01:50.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:50.969+0000 7f7510600640 1 -- 172.21.15.12:0/561746498 wait complete. 2024-03-23T23:01:50.989 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T23:01:51.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.215+0000 7fcd60c00640 1 -- 172.21.15.12:0/4061749197 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd5c102790 msgr2=0x7fcd5c102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:51.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.215+0000 7fcd60c00640 1 --2- 172.21.15.12:0/4061749197 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd5c102790 0x7fcd5c102c10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c009900 tx=0x7fcd4c02bc30 comp rx=0 tx=0).stop 2024-03-23T23:01:51.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.215+0000 7fcd60c00640 1 -- 172.21.15.12:0/4061749197 shutdown_connections 2024-03-23T23:01:51.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.215+0000 7fcd60c00640 1 --2- 172.21.15.12:0/4061749197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd5c103150 0x7fcd5c1099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:51.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.215+0000 7fcd60c00640 1 --2- 172.21.15.12:0/4061749197 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd5c102790 0x7fcd5c102c10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:51.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.215+0000 7fcd60c00640 1 --2- 172.21.15.12:0/4061749197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd5c1015a0 0x7fcd5c1019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:51.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.215+0000 7fcd60c00640 1 -- 172.21.15.12:0/4061749197 >> 172.21.15.12:0/4061749197 conn(0x7fcd5c0fcb10 msgr2=0x7fcd5c0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:51.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.216+0000 7fcd60c00640 1 -- 172.21.15.12:0/4061749197 shutdown_connections 2024-03-23T23:01:51.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.216+0000 7fcd60c00640 1 -- 172.21.15.12:0/4061749197 wait complete. 2024-03-23T23:01:51.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.217+0000 7fcd60c00640 1 Processor -- start 2024-03-23T23:01:51.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.217+0000 7fcd60c00640 1 -- start start 2024-03-23T23:01:51.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.217+0000 7fcd60c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd5c1015a0 0x7fcd5c199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:51.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.217+0000 7fcd60c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd5c102790 0x7fcd5c199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:51.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.217+0000 7fcd60c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd5c103150 0x7fcd5c1a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:51.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.217+0000 7fcd60c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c10d250 con 0x7fcd5c102790 2024-03-23T23:01:51.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.218+0000 7fcd60c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c10d0d0 con 0x7fcd5c1015a0 2024-03-23T23:01:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.218+0000 7fcd60c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c10d3d0 con 0x7fcd5c103150 2024-03-23T23:01:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.218+0000 7fcd5b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd5c1015a0 0x7fcd5c199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.218+0000 7fcd5b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd5c1015a0 0x7fcd5c199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:46858/0 (socket says 172.21.15.12:46858) 2024-03-23T23:01:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.218+0000 7fcd5b400640 1 -- 172.21.15.12:0/3451733416 learned_addr learned my addr 172.21.15.12:0/3451733416 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.218+0000 7fcd5aa00640 1 --2- 172.21.15.12:0/3451733416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd5c102790 0x7fcd5c199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.218+0000 7fcd5be00640 1 --2- 172.21.15.12:0/3451733416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd5c103150 0x7fcd5c1a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.218+0000 7fcd5b400640 1 -- 172.21.15.12:0/3451733416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd5c103150 msgr2=0x7fcd5c1a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.218+0000 7fcd5b400640 1 --2- 172.21.15.12:0/3451733416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd5c103150 0x7fcd5c1a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.218+0000 7fcd5b400640 1 -- 172.21.15.12:0/3451733416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd5c102790 msgr2=0x7fcd5c199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.218+0000 7fcd5b400640 1 --2- 172.21.15.12:0/3451733416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd5c102790 0x7fcd5c199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.218+0000 7fcd5b400640 1 -- 172.21.15.12:0/3451733416 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd5c1a16c0 con 0x7fcd5c1015a0 2024-03-23T23:01:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.218+0000 7fcd5aa00640 1 --2- 172.21.15.12:0/3451733416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd5c102790 0x7fcd5c199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:01:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.219+0000 7fcd5b400640 1 --2- 172.21.15.12:0/3451733416 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd5c1015a0 0x7fcd5c199aa0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcd4400d770 tx=0x7fcd4400dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:51.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.219+0000 7fcd53e00640 1 -- 172.21.15.12:0/3451733416 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd44017070 con 0x7fcd5c1015a0 2024-03-23T23:01:51.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.219+0000 7fcd53e00640 1 -- 172.21.15.12:0/3451733416 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd44009c30 con 0x7fcd5c1015a0 2024-03-23T23:01:51.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.219+0000 7fcd60c00640 1 -- 172.21.15.12:0/3451733416 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd5c1a19b0 con 0x7fcd5c1015a0 2024-03-23T23:01:51.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.219+0000 7fcd53e00640 1 -- 172.21.15.12:0/3451733416 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd44003dd0 con 0x7fcd5c1015a0 2024-03-23T23:01:51.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.219+0000 7fcd60c00640 1 -- 172.21.15.12:0/3451733416 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd5c1a1f20 con 0x7fcd5c1015a0 2024-03-23T23:01:51.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.221+0000 7fcd60c00640 1 -- 172.21.15.12:0/3451733416 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd5c1a2150 con 0x7fcd5c1015a0 2024-03-23T23:01:51.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.222+0000 7fcd53e00640 1 -- 172.21.15.12:0/3451733416 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcd44003fb0 con 0x7fcd5c1015a0 2024-03-23T23:01:51.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.224+0000 7fcd53e00640 1 --2- 172.21.15.12:0/3451733416 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcd400a2880 0x7fcd400a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:51.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.224+0000 7fcd53e00640 1 -- 172.21.15.12:0/3451733416 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fcd440ce3c0 con 0x7fcd5c1015a0 2024-03-23T23:01:51.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.224+0000 7fcd5aa00640 1 --2- 172.21.15.12:0/3451733416 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcd400a2880 0x7fcd400a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:51.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.225+0000 7fcd5aa00640 1 --2- 172.21.15.12:0/3451733416 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcd400a2880 0x7fcd400a4d40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c02c240 tx=0x7fcd4c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:51.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.228+0000 7fcd53e00640 1 -- 172.21.15.12:0/3451733416 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcd4409b410 con 0x7fcd5c1015a0 2024-03-23T23:01:51.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.432+0000 7fcd60c00640 1 -- 172.21.15.12:0/3451733416 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcd5c1a2440 con 0x7fcd5c1015a0 2024-03-23T23:01:51.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.433+0000 7fcd53e00640 1 -- 172.21.15.12:0/3451733416 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcd44009de0 con 0x7fcd5c1015a0 2024-03-23T23:01:51.433 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-23T23:01:51.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.435+0000 7fcd60c00640 1 -- 172.21.15.12:0/3451733416 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcd400a2880 msgr2=0x7fcd400a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:51.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.435+0000 7fcd60c00640 1 --2- 172.21.15.12:0/3451733416 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcd400a2880 0x7fcd400a4d40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c02c240 tx=0x7fcd4c02df90 comp rx=0 tx=0).stop 2024-03-23T23:01:51.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.435+0000 7fcd60c00640 1 -- 172.21.15.12:0/3451733416 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd5c1015a0 msgr2=0x7fcd5c199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:51.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.435+0000 7fcd60c00640 1 --2- 172.21.15.12:0/3451733416 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd5c1015a0 0x7fcd5c199aa0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcd4400d770 tx=0x7fcd4400dc30 comp rx=0 tx=0).stop 2024-03-23T23:01:51.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.435+0000 7fcd60c00640 1 -- 172.21.15.12:0/3451733416 shutdown_connections 2024-03-23T23:01:51.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.435+0000 7fcd60c00640 1 --2- 172.21.15.12:0/3451733416 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcd400a2880 0x7fcd400a4d40 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:51.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.435+0000 7fcd60c00640 1 --2- 172.21.15.12:0/3451733416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd5c103150 0x7fcd5c1a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:51.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.435+0000 7fcd60c00640 1 --2- 172.21.15.12:0/3451733416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd5c102790 0x7fcd5c199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:51.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.435+0000 7fcd60c00640 1 --2- 172.21.15.12:0/3451733416 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd5c1015a0 0x7fcd5c199aa0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:51.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.435+0000 7fcd60c00640 1 -- 172.21.15.12:0/3451733416 >> 172.21.15.12:0/3451733416 conn(0x7fcd5c0fcb10 msgr2=0x7fcd5c0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:51.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.436+0000 7fcd60c00640 1 -- 172.21.15.12:0/3451733416 shutdown_connections 2024-03-23T23:01:51.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.436+0000 7fcd60c00640 1 -- 172.21.15.12:0/3451733416 wait complete. 2024-03-23T23:01:51.454 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T23:01:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.670+0000 7fcd2b200640 1 -- 172.21.15.12:0/3937801249 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd24074080 msgr2=0x7fcd240744e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.670+0000 7fcd2b200640 1 --2- 172.21.15.12:0/3937801249 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd24074080 0x7fcd240744e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcd14009860 tx=0x7fcd1402bc00 comp rx=0 tx=0).stop 2024-03-23T23:01:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.670+0000 7fcd2b200640 1 -- 172.21.15.12:0/3937801249 shutdown_connections 2024-03-23T23:01:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.670+0000 7fcd2b200640 1 --2- 172.21.15.12:0/3937801249 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd241093b0 0x7fcd2410b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.670+0000 7fcd2b200640 1 --2- 172.21.15.12:0/3937801249 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd24074080 0x7fcd240744e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.670+0000 7fcd2b200640 1 --2- 172.21.15.12:0/3937801249 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd24075660 0x7fcd24073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.670+0000 7fcd2b200640 1 -- 172.21.15.12:0/3937801249 >> 172.21.15.12:0/3937801249 conn(0x7fcd240fca70 msgr2=0x7fcd240feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.671+0000 7fcd2b200640 1 -- 172.21.15.12:0/3937801249 shutdown_connections 2024-03-23T23:01:51.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.671+0000 7fcd2b200640 1 -- 172.21.15.12:0/3937801249 wait complete. 2024-03-23T23:01:51.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.672+0000 7fcd2b200640 1 Processor -- start 2024-03-23T23:01:51.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.672+0000 7fcd2b200640 1 -- start start 2024-03-23T23:01:51.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.672+0000 7fcd2b200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd24074080 0x7fcd24199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:51.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.672+0000 7fcd2b200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd24075660 0x7fcd24199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:51.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.672+0000 7fcd2b200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd241093b0 0x7fcd241a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:51.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.672+0000 7fcd2b200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2410f360 con 0x7fcd24074080 2024-03-23T23:01:51.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.672+0000 7fcd2b200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2410f1e0 con 0x7fcd241093b0 2024-03-23T23:01:51.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.672+0000 7fcd2b200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2410f4e0 con 0x7fcd24075660 2024-03-23T23:01:51.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.672+0000 7fcd29e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd24074080 0x7fcd24199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:51.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.672+0000 7fcd29400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd24075660 0x7fcd24199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:51.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.672+0000 7fcd29400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd24075660 0x7fcd24199f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55800/0 (socket says 172.21.15.12:55800) 2024-03-23T23:01:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.672+0000 7fcd29400640 1 -- 172.21.15.12:0/2103328428 learned_addr learned my addr 172.21.15.12:0/2103328428 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.673+0000 7fcd2a800640 1 --2- 172.21.15.12:0/2103328428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd241093b0 0x7fcd241a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.673+0000 7fcd29400640 1 -- 172.21.15.12:0/2103328428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd241093b0 msgr2=0x7fcd241a0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.673+0000 7fcd29400640 1 --2- 172.21.15.12:0/2103328428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd241093b0 0x7fcd241a0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.673+0000 7fcd29400640 1 -- 172.21.15.12:0/2103328428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd24074080 msgr2=0x7fcd24199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.673+0000 7fcd29400640 1 --2- 172.21.15.12:0/2103328428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd24074080 0x7fcd24199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.673+0000 7fcd29400640 1 -- 172.21.15.12:0/2103328428 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd241a1620 con 0x7fcd24075660 2024-03-23T23:01:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.673+0000 7fcd29400640 1 --2- 172.21.15.12:0/2103328428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd24075660 0x7fcd24199f40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcd14034a80 tx=0x7fcd140042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.673+0000 7fcd1aa00640 1 -- 172.21.15.12:0/2103328428 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd14041030 con 0x7fcd24075660 2024-03-23T23:01:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.674+0000 7fcd1aa00640 1 -- 172.21.15.12:0/2103328428 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd1402ca80 con 0x7fcd24075660 2024-03-23T23:01:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.674+0000 7fcd2b200640 1 -- 172.21.15.12:0/2103328428 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd241a18b0 con 0x7fcd24075660 2024-03-23T23:01:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.674+0000 7fcd1aa00640 1 -- 172.21.15.12:0/2103328428 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd14039040 con 0x7fcd24075660 2024-03-23T23:01:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.674+0000 7fcd2b200640 1 -- 172.21.15.12:0/2103328428 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd241a1df0 con 0x7fcd24075660 2024-03-23T23:01:51.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.675+0000 7fcd2b200640 1 -- 172.21.15.12:0/2103328428 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd24108170 con 0x7fcd24075660 2024-03-23T23:01:51.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.676+0000 7fcd1aa00640 1 -- 172.21.15.12:0/2103328428 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcd14044070 con 0x7fcd24075660 2024-03-23T23:01:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.678+0000 7fcd1aa00640 1 --2- 172.21.15.12:0/2103328428 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcd100a2880 0x7fcd100a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:51.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.678+0000 7fcd1aa00640 1 -- 172.21.15.12:0/2103328428 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fcd140efea0 con 0x7fcd24075660 2024-03-23T23:01:51.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.678+0000 7fcd29e00640 1 --2- 172.21.15.12:0/2103328428 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcd100a2880 0x7fcd100a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:51.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.679+0000 7fcd29e00640 1 --2- 172.21.15.12:0/2103328428 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcd100a2880 0x7fcd100a4d40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009a60 tx=0x7fcd1c010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:51.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.682+0000 7fcd1aa00640 1 -- 172.21.15.12:0/2103328428 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcd140bcef0 con 0x7fcd24075660 2024-03-23T23:01:51.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:51 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/561746498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:01:51.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:51 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3451733416' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:01:51.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:51 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/561746498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:01:51.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:51 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3451733416' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:01:51.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.891+0000 7fcd2b200640 1 -- 172.21.15.12:0/2103328428 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd240622a0 con 0x7fcd24075660 2024-03-23T23:01:51.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.893+0000 7fcd1aa00640 1 -- 172.21.15.12:0/2103328428 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22695 (secure 0 0 0) 0x7fcd140bc7f0 con 0x7fcd24075660 2024-03-23T23:01:51.893 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:01:51.893 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":88,"fsid":"90f679ae-e967-11ee-95cd-87774f69a715","created":"2024-03-23T22:50:14.906952+0000","modified":"2024-03-23T23:00:24.349052+0000","last_up_change":"2024-03-23T22:58:28.907824+0000","last_in_change":"2024-03-23T22:58:08.342487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T22:54:11.831010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T22:59:25.350974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T22:59:26.390323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b70875d7-a23d-4251-9e8b-1a16d4edc2a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6803","nonce":1190079611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6805","nonce":1190079611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6809","nonce":1190079611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6807","nonce":1190079611}]},"public_addr":"172.21.15.12:6803/1190079611","cluster_addr":"172.21.15.12:6805/1190079611","heartbeat_back_addr":"172.21.15.12:6809/1190079611","heartbeat_front_addr":"172.21.15.12:6807/1190079611","state":["exists","up"]},{"osd":1,"uuid":"416e87af-db5d-41ce-84d9-731e3e07e3c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6801","nonce":1983639525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6803","nonce":1983639525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6807","nonce":1983639525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6805","nonce":1983639525}]},"public_addr":"172.21.15.173:6801/1983639525","cluster_addr":"172.21.15.173:6803/1983639525","heartbeat_back_addr":"172.21.15.173:6807/1983639525","heartbeat_front_addr":"172.21.15.173:6805/1983639525","state":["exists","up"]},{"osd":2,"uuid":"7a559c70-4c54-431b-8baf-4136b6e76493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6801","nonce":2253997387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6803","nonce":2253997387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6807","nonce":2253997387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6805","nonce":2253997387}]},"public_addr":"172.21.15.203:6801/2253997387","cluster_addr":"172.21.15.203:6803/2253997387","heartbeat_back_addr":"172.21.15.203:6807/2253997387","heartbeat_front_addr":"172.21.15.203:6805/2253997387","state":["exists","up"]},{"osd":3,"uuid":"c41cd287-a674-486c-8025-328cd36e5ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6811","nonce":1761446905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6813","nonce":1761446905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6817","nonce":1761446905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6815","nonce":1761446905}]},"public_addr":"172.21.15.12:6811/1761446905","cluster_addr":"172.21.15.12:6813/1761446905","heartbeat_back_addr":"172.21.15.12:6817/1761446905","heartbeat_front_addr":"172.21.15.12:6815/1761446905","state":["exists","up"]},{"osd":4,"uuid":"3c48f886-0b42-4a09-b31e-abccc200bb1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6809","nonce":242958272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6811","nonce":242958272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6815","nonce":242958272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6813","nonce":242958272}]},"public_addr":"172.21.15.173:6809/242958272","cluster_addr":"172.21.15.173:6811/242958272","heartbeat_back_addr":"172.21.15.173:6815/242958272","heartbeat_front_addr":"172.21.15.173:6813/242958272","state":["exists","up"]},{"osd":5,"uuid":"4a34f06f-8805-40c0-b1ab-23407b0276d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6809","nonce":4270966092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6811","nonce":4270966092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6815","nonce":4270966092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6813","nonce":4270966092}]},"public_addr":"172.21.15.203:6809/4270966092","cluster_addr":"172.21.15.203:6811/4270966092","heartbeat_back_addr":"172.21.15.203:6815/4270966092","heartbeat_front_addr":"172.21.15.203:6813/4270966092","state":["exists","up"]},{"osd":6,"uuid":"26ddc663-5623-48c9-80d1-f8b968765d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6819","nonce":3606839295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6821","nonce":3606839295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6825","nonce":3606839295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6823","nonce":3606839295}]},"public_addr":"172.21.15.12:6819/3606839295","cluster_addr":"172.21.15.12:6821/3606839295","heartbeat_back_addr":"172.21.15.12:6825/3606839295","heartbeat_front_addr":"172.21.15.12:6823/3606839295","state":["exists","up"]},{"osd":7,"uuid":"17dc9417-71cd-4576-aa08-933212ec55e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6817","nonce":1241504346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6819","nonce":1241504346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6823","nonce":1241504346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6821","nonce":1241504346}]},"public_addr":"172.21.15.173:6817/1241504346","cluster_addr":"172.21.15.173:6819/1241504346","heartbeat_back_addr":"172.21.15.173:6823/1241504346","heartbeat_front_addr":"172.21.15.173:6821/1241504346","state":["exists","up"]},{"osd":8,"uuid":"90a2b63a-471d-43b8-ac18-bb1f16778d18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6817","nonce":1516996192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6819","nonce":1516996192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6823","nonce":1516996192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6821","nonce":1516996192}]},"public_addr":"172.21.15.203:6817/1516996192","cluster_addr":"172.21.15.203:6819/1516996192","heartbeat_back_addr":"172.21.15.203:6823/1516996192","heartbeat_front_addr":"172.21.15.203:6821/1516996192","state":["exists","up"]},{"osd":9,"uuid":"6e1ca504-40b4-49d0-b245-7946181abfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6827","nonce":1175260774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6829","nonce":1175260774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6833","nonce":1175260774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6831","nonce":1175260774}]},"public_addr":"172.21.15.12:6827/1175260774","cluster_addr":"172.21.15.12:6829/1175260774","heartbeat_back_addr":"172.21.15.12:6833/1175260774","heartbeat_front_addr":"172.21.15.12:6831/1175260774","state":["exists","up"]},{"osd":10,"uuid":"09a595ed-5614-4386-8e61-0170af66e01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6825","nonce":3303506808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6827","nonce":3303506808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6831","nonce":3303506808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6829","nonce":3303506808}]},"public_addr":"172.21.15.173:6825/3303506808","cluster_addr":"172.21.15.173:6827/3303506808","heartbeat_back_addr":"172.21.15.173:6831/3303506808","heartbeat_front_addr":"172.21.15.173:6829/3303506808","state":["exists","up"]},{"osd":11,"uuid":"9ef1d595-a977-4ba7-9c6b-d2b3295f20b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6825","nonce":2323604506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6827","nonce":2323604506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6831","nonce":2323604506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6829","nonce":2323604506}]},"public_addr":"172.21.15.203:6825/2323604506","cluster_addr":"172.21.15.203:6827/2323604506","heartbeat_back_addr":"172.21.15.203:6831/2323604506","heartbeat_front_addr":"172.21.15.203:6829/2323604506","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:18.469038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:42.752628+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:07.069493+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:41.962478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:08.363104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:33.341336+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:07.286541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:34.243302+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:59.749699+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:57:33.822913+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:00.737269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:26.599854+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/4182783120":"2024-03-24T22:51:23.113201+0000","172.21.15.12:6800/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1900938637":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/3414861652":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6801/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/2580541477":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6800/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/501058839":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1192554716":"2024-03-24T22:50:48.401223+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T23:01:51.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.895+0000 7fcd2b200640 1 -- 172.21.15.12:0/2103328428 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcd100a2880 msgr2=0x7fcd100a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:51.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.895+0000 7fcd2b200640 1 --2- 172.21.15.12:0/2103328428 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcd100a2880 0x7fcd100a4d40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009a60 tx=0x7fcd1c010040 comp rx=0 tx=0).stop 2024-03-23T23:01:51.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.896+0000 7fcd2b200640 1 -- 172.21.15.12:0/2103328428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd24075660 msgr2=0x7fcd24199f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:51.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.896+0000 7fcd2b200640 1 --2- 172.21.15.12:0/2103328428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd24075660 0x7fcd24199f40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcd14034a80 tx=0x7fcd140042f0 comp rx=0 tx=0).stop 2024-03-23T23:01:51.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.896+0000 7fcd2b200640 1 -- 172.21.15.12:0/2103328428 shutdown_connections 2024-03-23T23:01:51.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.896+0000 7fcd2b200640 1 --2- 172.21.15.12:0/2103328428 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcd100a2880 0x7fcd100a4d40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:51.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.896+0000 7fcd2b200640 1 --2- 172.21.15.12:0/2103328428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd241093b0 0x7fcd241a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:51.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.896+0000 7fcd2b200640 1 --2- 172.21.15.12:0/2103328428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd24075660 0x7fcd24199f40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:51.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.896+0000 7fcd2b200640 1 --2- 172.21.15.12:0/2103328428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd24074080 0x7fcd24199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:51.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.896+0000 7fcd2b200640 1 -- 172.21.15.12:0/2103328428 >> 172.21.15.12:0/2103328428 conn(0x7fcd240fca70 msgr2=0x7fcd24105340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:51.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.896+0000 7fcd2b200640 1 -- 172.21.15.12:0/2103328428 shutdown_connections 2024-03-23T23:01:51.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:51.896+0000 7fcd2b200640 1 -- 172.21.15.12:0/2103328428 wait complete. 2024-03-23T23:01:51.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:51 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/561746498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:01:51.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:51 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3451733416' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:01:51.915 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T22:54:11.831010+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T22:59:25.350974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T22:59:26.390323+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T23:01:51.915 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T23:01:52.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.137+0000 7fadb3200640 1 -- 172.21.15.12:0/3771693078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadb4076060 msgr2=0x7fadb4076480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:52.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.137+0000 7fadb3200640 1 --2- 172.21.15.12:0/3771693078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadb4076060 0x7fadb4076480 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fada4009ca0 tx=0x7fada402c100 comp rx=0 tx=0).stop 2024-03-23T23:01:52.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.137+0000 7fadb3200640 1 -- 172.21.15.12:0/3771693078 shutdown_connections 2024-03-23T23:01:52.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.137+0000 7fadb3200640 1 --2- 172.21.15.12:0/3771693078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fadb4077c40 0x7fadb40784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:52.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.137+0000 7fadb3200640 1 --2- 172.21.15.12:0/3771693078 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadb40772a0 0x7fadb4077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:52.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.137+0000 7fadb3200640 1 --2- 172.21.15.12:0/3771693078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadb4076060 0x7fadb4076480 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:52.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.137+0000 7fadb3200640 1 -- 172.21.15.12:0/3771693078 >> 172.21.15.12:0/3771693078 conn(0x7fadb40fed50 msgr2=0x7fadb4101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:52.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.138+0000 7fadb3200640 1 -- 172.21.15.12:0/3771693078 shutdown_connections 2024-03-23T23:01:52.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.138+0000 7fadb3200640 1 -- 172.21.15.12:0/3771693078 wait complete. 2024-03-23T23:01:52.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.139+0000 7fadb3200640 1 Processor -- start 2024-03-23T23:01:52.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.139+0000 7fadb3200640 1 -- start start 2024-03-23T23:01:52.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.139+0000 7fadb3200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadb4076060 0x7fadb419de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:52.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.139+0000 7fadb3200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadb40772a0 0x7fadb419e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:52.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.139+0000 7fadb3200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fadb4077c40 0x7fadb41a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:52.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.140+0000 7fadb3200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb4113830 con 0x7fadb4076060 2024-03-23T23:01:52.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.140+0000 7fadb3200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb41136b0 con 0x7fadb40772a0 2024-03-23T23:01:52.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.140+0000 7fadb3200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb41139b0 con 0x7fadb4077c40 2024-03-23T23:01:52.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.140+0000 7fada9400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadb40772a0 0x7fadb419e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:52.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.140+0000 7fada9400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadb40772a0 0x7fadb419e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:46882/0 (socket says 172.21.15.12:46882) 2024-03-23T23:01:52.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.140+0000 7fada9400640 1 -- 172.21.15.12:0/287181571 learned_addr learned my addr 172.21.15.12:0/287181571 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:52.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.140+0000 7fadb1e00640 1 --2- 172.21.15.12:0/287181571 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadb4076060 0x7fadb419de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:52.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.140+0000 7fadb2800640 1 --2- 172.21.15.12:0/287181571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fadb4077c40 0x7fadb41a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:52.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.140+0000 7fada9400640 1 -- 172.21.15.12:0/287181571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fadb4077c40 msgr2=0x7fadb41a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:52.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.140+0000 7fada9400640 1 --2- 172.21.15.12:0/287181571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fadb4077c40 0x7fadb41a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:52.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.140+0000 7fada9400640 1 -- 172.21.15.12:0/287181571 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadb4076060 msgr2=0x7fadb419de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:52.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.140+0000 7fada9400640 1 --2- 172.21.15.12:0/287181571 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadb4076060 0x7fadb419de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:52.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.140+0000 7fada9400640 1 -- 172.21.15.12:0/287181571 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb41a5aa0 con 0x7fadb40772a0 2024-03-23T23:01:52.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.141+0000 7fada9400640 1 --2- 172.21.15.12:0/287181571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadb40772a0 0x7fadb419e3c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fad9c00d950 tx=0x7fad9c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:52.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.141+0000 7fadab400640 1 -- 172.21.15.12:0/287181571 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c017070 con 0x7fadb40772a0 2024-03-23T23:01:52.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.141+0000 7fadab400640 1 -- 172.21.15.12:0/287181571 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad9c009c30 con 0x7fadb40772a0 2024-03-23T23:01:52.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.141+0000 7fadab400640 1 -- 172.21.15.12:0/287181571 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c00f6b0 con 0x7fadb40772a0 2024-03-23T23:01:52.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.141+0000 7fadb3200640 1 -- 172.21.15.12:0/287181571 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb41a5d90 con 0x7fadb40772a0 2024-03-23T23:01:52.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.141+0000 7fadb3200640 1 -- 172.21.15.12:0/287181571 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb41a6230 con 0x7fadb40772a0 2024-03-23T23:01:52.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.142+0000 7fadb3200640 1 -- 172.21.15.12:0/287181571 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb4110110 con 0x7fadb40772a0 2024-03-23T23:01:52.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.144+0000 7fadab400640 1 -- 172.21.15.12:0/287181571 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fad9c0109d0 con 0x7fadb40772a0 2024-03-23T23:01:52.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.145+0000 7fadab400640 1 --2- 172.21.15.12:0/287181571 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fad980a2880 0x7fad980a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:52.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.145+0000 7fadab400640 1 -- 172.21.15.12:0/287181571 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fad9c0ce5d0 con 0x7fadb40772a0 2024-03-23T23:01:52.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.146+0000 7fadb1e00640 1 --2- 172.21.15.12:0/287181571 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fad980a2880 0x7fad980a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:52.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.146+0000 7fadb1e00640 1 --2- 172.21.15.12:0/287181571 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fad980a2880 0x7fad980a4d40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fada402c710 tx=0x7fada40323c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:52.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.149+0000 7fadab400640 1 -- 172.21.15.12:0/287181571 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fad9c009de0 con 0x7fadb40772a0 2024-03-23T23:01:52.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.355+0000 7fadb3200640 1 -- 172.21.15.12:0/287181571 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fadb40622a0 con 0x7fadb40772a0 2024-03-23T23:01:52.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.356+0000 7fadab400640 1 -- 172.21.15.12:0/287181571 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7fad9c09afd0 con 0x7fadb40772a0 2024-03-23T23:01:52.356 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-23T23:01:52.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.358+0000 7fadb3200640 1 -- 172.21.15.12:0/287181571 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fad980a2880 msgr2=0x7fad980a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:52.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.358+0000 7fadb3200640 1 --2- 172.21.15.12:0/287181571 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fad980a2880 0x7fad980a4d40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fada402c710 tx=0x7fada40323c0 comp rx=0 tx=0).stop 2024-03-23T23:01:52.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.358+0000 7fadb3200640 1 -- 172.21.15.12:0/287181571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadb40772a0 msgr2=0x7fadb419e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:52.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.358+0000 7fadb3200640 1 --2- 172.21.15.12:0/287181571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadb40772a0 0x7fadb419e3c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fad9c00d950 tx=0x7fad9c00de10 comp rx=0 tx=0).stop 2024-03-23T23:01:52.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.358+0000 7fadb3200640 1 -- 172.21.15.12:0/287181571 shutdown_connections 2024-03-23T23:01:52.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.358+0000 7fadb3200640 1 --2- 172.21.15.12:0/287181571 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fad980a2880 0x7fad980a4d40 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:52.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.358+0000 7fadb3200640 1 --2- 172.21.15.12:0/287181571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fadb4077c40 0x7fadb41a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:52.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.358+0000 7fadb3200640 1 --2- 172.21.15.12:0/287181571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadb40772a0 0x7fadb419e3c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:52.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.358+0000 7fadb3200640 1 --2- 172.21.15.12:0/287181571 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadb4076060 0x7fadb419de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:52.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.358+0000 7fadb3200640 1 -- 172.21.15.12:0/287181571 >> 172.21.15.12:0/287181571 conn(0x7fadb40fed50 msgr2=0x7fadb4100850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:52.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.359+0000 7fadb3200640 1 -- 172.21.15.12:0/287181571 shutdown_connections 2024-03-23T23:01:52.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.359+0000 7fadb3200640 1 -- 172.21.15.12:0/287181571 wait complete. 2024-03-23T23:01:52.379 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T23:01:52.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.612+0000 7fe27ba00640 1 -- 172.21.15.12:0/4107574308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe274104ab0 msgr2=0x7fe274104f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:52.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.612+0000 7fe27ba00640 1 --2- 172.21.15.12:0/4107574308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe274104ab0 0x7fe274104f30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe25c009900 tx=0x7fe25c02bc50 comp rx=0 tx=0).stop 2024-03-23T23:01:52.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.612+0000 7fe27ba00640 1 -- 172.21.15.12:0/4107574308 shutdown_connections 2024-03-23T23:01:52.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.612+0000 7fe27ba00640 1 --2- 172.21.15.12:0/4107574308 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe274105470 0x7fe27410bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:52.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.612+0000 7fe27ba00640 1 --2- 172.21.15.12:0/4107574308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe274104ab0 0x7fe274104f30 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:52.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.612+0000 7fe27ba00640 1 --2- 172.21.15.12:0/4107574308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2741038c0 0x7fe274103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:52.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.612+0000 7fe27ba00640 1 -- 172.21.15.12:0/4107574308 >> 172.21.15.12:0/4107574308 conn(0x7fe2740fee50 msgr2=0x7fe274101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:52.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.612+0000 7fe27ba00640 1 -- 172.21.15.12:0/4107574308 shutdown_connections 2024-03-23T23:01:52.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.613+0000 7fe27ba00640 1 -- 172.21.15.12:0/4107574308 wait complete. 2024-03-23T23:01:52.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.614+0000 7fe27ba00640 1 Processor -- start 2024-03-23T23:01:52.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.614+0000 7fe27ba00640 1 -- start start 2024-03-23T23:01:52.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.614+0000 7fe27ba00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2741038c0 0x7fe274199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:52.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.614+0000 7fe27ba00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe274104ab0 0x7fe27419a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:52.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.614+0000 7fe27ba00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe274105470 0x7fe2741a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:52.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.614+0000 7fe27ba00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27410f590 con 0x7fe274105470 2024-03-23T23:01:52.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.614+0000 7fe27ba00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27410f410 con 0x7fe2741038c0 2024-03-23T23:01:52.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.614+0000 7fe27ba00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27410f710 con 0x7fe274104ab0 2024-03-23T23:01:52.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.615+0000 7fe279000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2741038c0 0x7fe274199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:52.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.615+0000 7fe279000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2741038c0 0x7fe274199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:46896/0 (socket says 172.21.15.12:46896) 2024-03-23T23:01:52.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.615+0000 7fe279000640 1 -- 172.21.15.12:0/3036454608 learned_addr learned my addr 172.21.15.12:0/3036454608 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:52.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.615+0000 7fe26be00640 1 --2- 172.21.15.12:0/3036454608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe274104ab0 0x7fe27419a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:52.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.615+0000 7fe279a00640 1 --2- 172.21.15.12:0/3036454608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe274105470 0x7fe2741a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:52.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.615+0000 7fe279000640 1 -- 172.21.15.12:0/3036454608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe274104ab0 msgr2=0x7fe27419a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:52.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.615+0000 7fe279000640 1 --2- 172.21.15.12:0/3036454608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe274104ab0 0x7fe27419a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:52.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.615+0000 7fe279000640 1 -- 172.21.15.12:0/3036454608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe274105470 msgr2=0x7fe2741a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:52.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.615+0000 7fe279000640 1 --2- 172.21.15.12:0/3036454608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe274105470 0x7fe2741a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:52.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.615+0000 7fe279000640 1 -- 172.21.15.12:0/3036454608 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2741a1800 con 0x7fe2741038c0 2024-03-23T23:01:52.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.615+0000 7fe279000640 1 --2- 172.21.15.12:0/3036454608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2741038c0 0x7fe274199be0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe26400e9c0 tx=0x7fe26400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:52.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.616+0000 7fe272000640 1 -- 172.21.15.12:0/3036454608 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe26400cd80 con 0x7fe2741038c0 2024-03-23T23:01:52.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.616+0000 7fe272000640 1 -- 172.21.15.12:0/3036454608 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe2640035f0 con 0x7fe2741038c0 2024-03-23T23:01:52.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.616+0000 7fe272000640 1 -- 172.21.15.12:0/3036454608 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe264010690 con 0x7fe2741038c0 2024-03-23T23:01:52.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.616+0000 7fe27ba00640 1 -- 172.21.15.12:0/3036454608 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2741a1af0 con 0x7fe2741038c0 2024-03-23T23:01:52.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.616+0000 7fe27ba00640 1 -- 172.21.15.12:0/3036454608 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2741a2060 con 0x7fe2741038c0 2024-03-23T23:01:52.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.617+0000 7fe27ba00640 1 -- 172.21.15.12:0/3036454608 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe27410abe0 con 0x7fe2741038c0 2024-03-23T23:01:52.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.619+0000 7fe272000640 1 -- 172.21.15.12:0/3036454608 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe2640037a0 con 0x7fe2741038c0 2024-03-23T23:01:52.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.620+0000 7fe272000640 1 --2- 172.21.15.12:0/3036454608 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe2580a2880 0x7fe2580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:52.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.620+0000 7fe272000640 1 -- 172.21.15.12:0/3036454608 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe2640cedc0 con 0x7fe2741038c0 2024-03-23T23:01:52.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.620+0000 7fe26be00640 1 --2- 172.21.15.12:0/3036454608 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe2580a2880 0x7fe2580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:52.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.621+0000 7fe26be00640 1 --2- 172.21.15.12:0/3036454608 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe2580a2880 0x7fe2580a4d40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe25c02c260 tx=0x7fe25c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:52.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.624+0000 7fe272000640 1 -- 172.21.15.12:0/3036454608 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe26409be10 con 0x7fe2741038c0 2024-03-23T23:01:52.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:52 smithi012 ceph-mon[31695]: pgmap v392: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-23T23:01:52.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:52 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2103328428' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:52.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:52 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/287181571' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:01:52.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.826+0000 7fe27ba00640 1 -- 172.21.15.12:0/3036454608 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe274062340 con 0x7fe2741038c0 2024-03-23T23:01:52.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.827+0000 7fe272000640 1 -- 172.21.15.12:0/3036454608 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe26409b710 con 0x7fe2741038c0 2024-03-23T23:01:52.827 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-23T23:01:52.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.829+0000 7fe27ba00640 1 -- 172.21.15.12:0/3036454608 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe2580a2880 msgr2=0x7fe2580a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:52.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.829+0000 7fe27ba00640 1 --2- 172.21.15.12:0/3036454608 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe2580a2880 0x7fe2580a4d40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe25c02c260 tx=0x7fe25c000bc0 comp rx=0 tx=0).stop 2024-03-23T23:01:52.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.829+0000 7fe27ba00640 1 -- 172.21.15.12:0/3036454608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2741038c0 msgr2=0x7fe274199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:52.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.829+0000 7fe27ba00640 1 --2- 172.21.15.12:0/3036454608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2741038c0 0x7fe274199be0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe26400e9c0 tx=0x7fe26400ee80 comp rx=0 tx=0).stop 2024-03-23T23:01:52.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.830+0000 7fe27ba00640 1 -- 172.21.15.12:0/3036454608 shutdown_connections 2024-03-23T23:01:52.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.830+0000 7fe27ba00640 1 --2- 172.21.15.12:0/3036454608 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe2580a2880 0x7fe2580a4d40 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:52.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.830+0000 7fe27ba00640 1 --2- 172.21.15.12:0/3036454608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe274105470 0x7fe2741a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:52.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.830+0000 7fe27ba00640 1 --2- 172.21.15.12:0/3036454608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe274104ab0 0x7fe27419a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:52.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.830+0000 7fe27ba00640 1 --2- 172.21.15.12:0/3036454608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2741038c0 0x7fe274199be0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:52.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.830+0000 7fe27ba00640 1 -- 172.21.15.12:0/3036454608 >> 172.21.15.12:0/3036454608 conn(0x7fe2740fee50 msgr2=0x7fe274107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:52.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.830+0000 7fe27ba00640 1 -- 172.21.15.12:0/3036454608 shutdown_connections 2024-03-23T23:01:52.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:52.830+0000 7fe27ba00640 1 -- 172.21.15.12:0/3036454608 wait complete. 2024-03-23T23:01:52.848 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T23:01:52.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:52 smithi173 ceph-mon[36116]: pgmap v392: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-23T23:01:52.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:52 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2103328428' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:52.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:52 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/287181571' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:01:52.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:52 smithi203 ceph-mon[36693]: pgmap v392: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-23T23:01:52.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:52 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2103328428' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:52.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:52 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/287181571' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:01:53.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.067+0000 7f9e9d800640 1 -- 172.21.15.12:0/1956079517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e98105490 msgr2=0x7f9e9810bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:53.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.067+0000 7f9e9d800640 1 --2- 172.21.15.12:0/1956079517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e98105490 0x7f9e9810bd20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9e90009be0 tx=0x7f9e9002bef0 comp rx=0 tx=0).stop 2024-03-23T23:01:53.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.067+0000 7f9e9d800640 1 -- 172.21.15.12:0/1956079517 shutdown_connections 2024-03-23T23:01:53.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.067+0000 7f9e9d800640 1 --2- 172.21.15.12:0/1956079517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e98105490 0x7f9e9810bd20 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:53.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.067+0000 7f9e9d800640 1 --2- 172.21.15.12:0/1956079517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e98104ad0 0x7f9e98104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:53.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.067+0000 7f9e9d800640 1 --2- 172.21.15.12:0/1956079517 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e981038e0 0x7f9e98103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:53.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.067+0000 7f9e9d800640 1 -- 172.21.15.12:0/1956079517 >> 172.21.15.12:0/1956079517 conn(0x7f9e980fee50 msgr2=0x7f9e98101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:53.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.067+0000 7f9e9d800640 1 -- 172.21.15.12:0/1956079517 shutdown_connections 2024-03-23T23:01:53.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.067+0000 7f9e9d800640 1 -- 172.21.15.12:0/1956079517 wait complete. 2024-03-23T23:01:53.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.068+0000 7f9e9d800640 1 Processor -- start 2024-03-23T23:01:53.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.068+0000 7f9e9d800640 1 -- start start 2024-03-23T23:01:53.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.069+0000 7f9e9d800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e981038e0 0x7f9e98199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:53.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.069+0000 7f9e9d800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e98104ad0 0x7f9e9819a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:53.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.069+0000 7f9e9d800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e98105490 0x7f9e981a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:53.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.069+0000 7f9e9d800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9810f5b0 con 0x7f9e98104ad0 2024-03-23T23:01:53.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.069+0000 7f9e9d800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9810f430 con 0x7f9e98105490 2024-03-23T23:01:53.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.069+0000 7f9e9d800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9810f730 con 0x7f9e981038e0 2024-03-23T23:01:53.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.069+0000 7f9e97400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e98105490 0x7f9e981a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:53.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.069+0000 7f9e96000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e98104ad0 0x7f9e9819a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:53.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.069+0000 7f9e96a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e981038e0 0x7f9e98199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:53.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.069+0000 7f9e97400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e98105490 0x7f9e981a1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:46906/0 (socket says 172.21.15.12:46906) 2024-03-23T23:01:53.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.069+0000 7f9e97400640 1 -- 172.21.15.12:0/843215303 learned_addr learned my addr 172.21.15.12:0/843215303 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:53.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.070+0000 7f9e97400640 1 -- 172.21.15.12:0/843215303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e981038e0 msgr2=0x7f9e98199bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:53.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.070+0000 7f9e97400640 1 --2- 172.21.15.12:0/843215303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e981038e0 0x7f9e98199bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:53.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.070+0000 7f9e97400640 1 -- 172.21.15.12:0/843215303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e98104ad0 msgr2=0x7f9e9819a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:53.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.070+0000 7f9e97400640 1 --2- 172.21.15.12:0/843215303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e98104ad0 0x7f9e9819a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:53.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.070+0000 7f9e97400640 1 -- 172.21.15.12:0/843215303 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e981a17f0 con 0x7f9e98105490 2024-03-23T23:01:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.070+0000 7f9e97400640 1 --2- 172.21.15.12:0/843215303 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e98105490 0x7f9e981a1180 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9e90036040 tx=0x7f9e900337b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.070+0000 7f9e8f400640 1 -- 172.21.15.12:0/843215303 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e90043020 con 0x7f9e98105490 2024-03-23T23:01:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.070+0000 7f9e9d800640 1 -- 172.21.15.12:0/843215303 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e981a1a80 con 0x7f9e98105490 2024-03-23T23:01:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.070+0000 7f9e8f400640 1 -- 172.21.15.12:0/843215303 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e9002ca60 con 0x7f9e98105490 2024-03-23T23:01:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.070+0000 7f9e8f400640 1 -- 172.21.15.12:0/843215303 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e90034690 con 0x7f9e98105490 2024-03-23T23:01:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.070+0000 7f9e9d800640 1 -- 172.21.15.12:0/843215303 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e981a2020 con 0x7f9e98105490 2024-03-23T23:01:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.071+0000 7f9e9d800640 1 -- 172.21.15.12:0/843215303 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e9810ac00 con 0x7f9e98105490 2024-03-23T23:01:53.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.074+0000 7f9e8f400640 1 -- 172.21.15.12:0/843215303 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9e90046020 con 0x7f9e98105490 2024-03-23T23:01:53.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.075+0000 7f9e8f400640 1 --2- 172.21.15.12:0/843215303 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9e7c0a2800 0x7f9e7c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:53.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.075+0000 7f9e8f400640 1 -- 172.21.15.12:0/843215303 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9e900f1180 con 0x7f9e98105490 2024-03-23T23:01:53.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.075+0000 7f9e96a00640 1 --2- 172.21.15.12:0/843215303 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9e7c0a2800 0x7f9e7c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:53.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.076+0000 7f9e96a00640 1 --2- 172.21.15.12:0/843215303 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9e7c0a2800 0x7f9e7c0a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9e98104930 tx=0x7f9e88005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:53.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.083+0000 7f9e8f400640 1 -- 172.21.15.12:0/843215303 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9e9004d030 con 0x7f9e98105490 2024-03-23T23:01:53.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.287+0000 7f9e9d800640 1 -- 172.21.15.12:0/843215303 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9e9802d070 con 0x7f9e98105490 2024-03-23T23:01:53.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.288+0000 7f9e8f400640 1 -- 172.21.15.12:0/843215303 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9e900be1d0 con 0x7f9e98105490 2024-03-23T23:01:53.288 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-23T23:01:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.290+0000 7f9e9d800640 1 -- 172.21.15.12:0/843215303 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9e7c0a2800 msgr2=0x7f9e7c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.290+0000 7f9e9d800640 1 --2- 172.21.15.12:0/843215303 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9e7c0a2800 0x7f9e7c0a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9e98104930 tx=0x7f9e88005f90 comp rx=0 tx=0).stop 2024-03-23T23:01:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.290+0000 7f9e9d800640 1 -- 172.21.15.12:0/843215303 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e98105490 msgr2=0x7f9e981a1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.290+0000 7f9e9d800640 1 --2- 172.21.15.12:0/843215303 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e98105490 0x7f9e981a1180 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9e90036040 tx=0x7f9e900337b0 comp rx=0 tx=0).stop 2024-03-23T23:01:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.290+0000 7f9e9d800640 1 -- 172.21.15.12:0/843215303 shutdown_connections 2024-03-23T23:01:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.290+0000 7f9e9d800640 1 --2- 172.21.15.12:0/843215303 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9e7c0a2800 0x7f9e7c0a4cc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.291+0000 7f9e9d800640 1 --2- 172.21.15.12:0/843215303 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e98105490 0x7f9e981a1180 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.291+0000 7f9e9d800640 1 --2- 172.21.15.12:0/843215303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e98104ad0 0x7f9e9819a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.291+0000 7f9e9d800640 1 --2- 172.21.15.12:0/843215303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e981038e0 0x7f9e98199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.291+0000 7f9e9d800640 1 -- 172.21.15.12:0/843215303 >> 172.21.15.12:0/843215303 conn(0x7f9e980fee50 msgr2=0x7f9e98107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.291+0000 7f9e9d800640 1 -- 172.21.15.12:0/843215303 shutdown_connections 2024-03-23T23:01:53.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.291+0000 7f9e9d800640 1 -- 172.21.15.12:0/843215303 wait complete. 2024-03-23T23:01:53.310 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-23T23:01:53.310 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:01:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.531+0000 7f169d000640 1 -- 172.21.15.12:0/2791653070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1698107ce0 msgr2=0x7f1698108100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.531+0000 7f169d000640 1 --2- 172.21.15.12:0/2791653070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1698107ce0 0x7f1698108100 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1688009ca0 tx=0x7f168802c100 comp rx=0 tx=0).stop 2024-03-23T23:01:53.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.531+0000 7f169d000640 1 -- 172.21.15.12:0/2791653070 shutdown_connections 2024-03-23T23:01:53.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.531+0000 7f169d000640 1 --2- 172.21.15.12:0/2791653070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1698109890 0x7f1698110120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:53.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.531+0000 7f169d000640 1 --2- 172.21.15.12:0/2791653070 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1698108ed0 0x7f1698109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:53.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.531+0000 7f169d000640 1 --2- 172.21.15.12:0/2791653070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1698107ce0 0x7f1698108100 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:53.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.531+0000 7f169d000640 1 -- 172.21.15.12:0/2791653070 >> 172.21.15.12:0/2791653070 conn(0x7f1698076cb0 msgr2=0x7f16980790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:53.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.531+0000 7f169d000640 1 -- 172.21.15.12:0/2791653070 shutdown_connections 2024-03-23T23:01:53.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.531+0000 7f169d000640 1 -- 172.21.15.12:0/2791653070 wait complete. 2024-03-23T23:01:53.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.532+0000 7f169d000640 1 Processor -- start 2024-03-23T23:01:53.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.532+0000 7f169d000640 1 -- start start 2024-03-23T23:01:53.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.533+0000 7f169d000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1698107ce0 0x7f169819e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:53.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.533+0000 7f169d000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1698108ed0 0x7f169819e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:53.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.533+0000 7f169d000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1698109890 0x7f16981a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:53.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.533+0000 7f169d000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1698113930 con 0x7f1698109890 2024-03-23T23:01:53.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.533+0000 7f169d000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16981137b0 con 0x7f1698107ce0 2024-03-23T23:01:53.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.533+0000 7f169d000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1698113ab0 con 0x7f1698108ed0 2024-03-23T23:01:53.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.533+0000 7f1695e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1698107ce0 0x7f169819e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:53.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.533+0000 7f1695e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1698107ce0 0x7f169819e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:46924/0 (socket says 172.21.15.12:46924) 2024-03-23T23:01:53.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.533+0000 7f1695e00640 1 -- 172.21.15.12:0/1890798614 learned_addr learned my addr 172.21.15.12:0/1890798614 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:53.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.533+0000 7f1696800640 1 --2- 172.21.15.12:0/1890798614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1698109890 0x7f16981a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:53.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.533+0000 7f168d400640 1 --2- 172.21.15.12:0/1890798614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1698108ed0 0x7f169819e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:53.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.533+0000 7f1695e00640 1 -- 172.21.15.12:0/1890798614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1698108ed0 msgr2=0x7f169819e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:53.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.533+0000 7f1695e00640 1 --2- 172.21.15.12:0/1890798614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1698108ed0 0x7f169819e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:53.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.533+0000 7f1695e00640 1 -- 172.21.15.12:0/1890798614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1698109890 msgr2=0x7f16981a55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:53.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.533+0000 7f1695e00640 1 --2- 172.21.15.12:0/1890798614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1698109890 0x7f16981a55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:53.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.533+0000 7f1695e00640 1 -- 172.21.15.12:0/1890798614 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16981a5c30 con 0x7f1698107ce0 2024-03-23T23:01:53.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.534+0000 7f168d400640 1 --2- 172.21.15.12:0/1890798614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1698108ed0 0x7f169819e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:01:53.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.534+0000 7f1695e00640 1 --2- 172.21.15.12:0/1890798614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1698107ce0 0x7f169819e060 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1688000f80 tx=0x7f1688035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:53.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.534+0000 7f168f400640 1 -- 172.21.15.12:0/1890798614 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1688039040 con 0x7f1698107ce0 2024-03-23T23:01:53.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.534+0000 7f168f400640 1 -- 172.21.15.12:0/1890798614 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1688044070 con 0x7f1698107ce0 2024-03-23T23:01:53.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.534+0000 7f168f400640 1 -- 172.21.15.12:0/1890798614 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1688004330 con 0x7f1698107ce0 2024-03-23T23:01:53.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.534+0000 7f169d000640 1 -- 172.21.15.12:0/1890798614 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16981a5ec0 con 0x7f1698107ce0 2024-03-23T23:01:53.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.534+0000 7f169d000640 1 -- 172.21.15.12:0/1890798614 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16981a6320 con 0x7f1698107ce0 2024-03-23T23:01:53.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.536+0000 7f169d000640 1 -- 172.21.15.12:0/1890798614 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f169810f050 con 0x7f1698107ce0 2024-03-23T23:01:53.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.537+0000 7f168f400640 1 -- 172.21.15.12:0/1890798614 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f16880044d0 con 0x7f1698107ce0 2024-03-23T23:01:53.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.539+0000 7f168f400640 1 --2- 172.21.15.12:0/1890798614 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f167c0a2880 0x7f167c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:53.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.539+0000 7f168f400640 1 -- 172.21.15.12:0/1890798614 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f16880f1050 con 0x7f1698107ce0 2024-03-23T23:01:53.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.539+0000 7f168d400640 1 --2- 172.21.15.12:0/1890798614 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f167c0a2880 0x7f167c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:53.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.540+0000 7f168d400640 1 --2- 172.21.15.12:0/1890798614 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f167c0a2880 0x7f167c0a4d40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f169819f520 tx=0x7f1680005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:53.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.547+0000 7f168f400640 1 -- 172.21.15.12:0/1890798614 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f16880be100 con 0x7f1698107ce0 2024-03-23T23:01:53.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.794+0000 7f169d000640 1 -- 172.21.15.12:0/1890798614 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f169802d070 con 0x7f1698107ce0 2024-03-23T23:01:53.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.795+0000 7f168f400640 1 -- 172.21.15.12:0/1890798614 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f16880bda00 con 0x7f1698107ce0 2024-03-23T23:01:53.795 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:01:53.795 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:01:53.795 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:01:53.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.798+0000 7f169d000640 1 -- 172.21.15.12:0/1890798614 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f167c0a2880 msgr2=0x7f167c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:53.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.798+0000 7f169d000640 1 --2- 172.21.15.12:0/1890798614 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f167c0a2880 0x7f167c0a4d40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f169819f520 tx=0x7f1680005f90 comp rx=0 tx=0).stop 2024-03-23T23:01:53.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.798+0000 7f169d000640 1 -- 172.21.15.12:0/1890798614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1698107ce0 msgr2=0x7f169819e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:53.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.798+0000 7f169d000640 1 --2- 172.21.15.12:0/1890798614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1698107ce0 0x7f169819e060 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1688000f80 tx=0x7f1688035d10 comp rx=0 tx=0).stop 2024-03-23T23:01:53.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.798+0000 7f169d000640 1 -- 172.21.15.12:0/1890798614 shutdown_connections 2024-03-23T23:01:53.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.798+0000 7f169d000640 1 --2- 172.21.15.12:0/1890798614 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f167c0a2880 0x7f167c0a4d40 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:53.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.798+0000 7f169d000640 1 --2- 172.21.15.12:0/1890798614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1698109890 0x7f16981a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:53.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.798+0000 7f169d000640 1 --2- 172.21.15.12:0/1890798614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1698108ed0 0x7f169819e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:53.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.798+0000 7f169d000640 1 --2- 172.21.15.12:0/1890798614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1698107ce0 0x7f169819e060 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:53.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.798+0000 7f169d000640 1 -- 172.21.15.12:0/1890798614 >> 172.21.15.12:0/1890798614 conn(0x7f1698076cb0 msgr2=0x7f169810c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:53.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.798+0000 7f169d000640 1 -- 172.21.15.12:0/1890798614 shutdown_connections 2024-03-23T23:01:53.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:53.798+0000 7f169d000640 1 -- 172.21.15.12:0/1890798614 wait complete. 2024-03-23T23:01:53.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:53 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3036454608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:01:53.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:53 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/843215303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:01:53.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:53 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:01:53.816 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-23T23:01:53.816 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-23T23:01:53.817 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T23:01:53.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:53 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3036454608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:01:53.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:53 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/843215303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:01:53.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:53 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:01:53.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:53 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3036454608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:01:53.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:53 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/843215303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:01:53.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:53 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:01:54.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.032+0000 7f2add000640 1 -- 172.21.15.12:0/1943425571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ad8105490 msgr2=0x7f2ad810bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:54.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.032+0000 7f2add000640 1 --2- 172.21.15.12:0/1943425571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ad8105490 0x7f2ad810bd20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2ad0009ca0 tx=0x7f2ad002c100 comp rx=0 tx=0).stop 2024-03-23T23:01:54.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.032+0000 7f2add000640 1 -- 172.21.15.12:0/1943425571 shutdown_connections 2024-03-23T23:01:54.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.032+0000 7f2add000640 1 --2- 172.21.15.12:0/1943425571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ad8105490 0x7f2ad810bd20 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:54.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.032+0000 7f2add000640 1 --2- 172.21.15.12:0/1943425571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ad8104ad0 0x7f2ad8104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:54.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.032+0000 7f2add000640 1 --2- 172.21.15.12:0/1943425571 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ad81038e0 0x7f2ad8103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:54.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.032+0000 7f2add000640 1 -- 172.21.15.12:0/1943425571 >> 172.21.15.12:0/1943425571 conn(0x7f2ad80fee90 msgr2=0x7f2ad81012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:54.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.032+0000 7f2add000640 1 -- 172.21.15.12:0/1943425571 shutdown_connections 2024-03-23T23:01:54.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.032+0000 7f2add000640 1 -- 172.21.15.12:0/1943425571 wait complete. 2024-03-23T23:01:54.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.033+0000 7f2add000640 1 Processor -- start 2024-03-23T23:01:54.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.034+0000 7f2add000640 1 -- start start 2024-03-23T23:01:54.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.034+0000 7f2add000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ad81038e0 0x7f2ad8199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:54.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.034+0000 7f2add000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ad8104ad0 0x7f2ad819a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:54.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.034+0000 7f2add000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ad8105490 0x7f2ad81a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:54.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.034+0000 7f2add000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad810f560 con 0x7f2ad8105490 2024-03-23T23:01:54.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.034+0000 7f2add000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad810f3e0 con 0x7f2ad8104ad0 2024-03-23T23:01:54.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.034+0000 7f2add000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad810f6e0 con 0x7f2ad81038e0 2024-03-23T23:01:54.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.034+0000 7f2acfe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ad8104ad0 0x7f2ad819a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:54.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.034+0000 7f2ad5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ad81038e0 0x7f2ad8199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:54.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.034+0000 7f2acfe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ad8104ad0 0x7f2ad819a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36442/0 (socket says 172.21.15.12:36442) 2024-03-23T23:01:54.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.034+0000 7f2acfe00640 1 -- 172.21.15.12:0/2537183645 learned_addr learned my addr 172.21.15.12:0/2537183645 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:54.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.034+0000 7f2ad6800640 1 --2- 172.21.15.12:0/2537183645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ad8105490 0x7f2ad81a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:54.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.035+0000 7f2acfe00640 1 -- 172.21.15.12:0/2537183645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ad81038e0 msgr2=0x7f2ad8199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:54.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.035+0000 7f2acfe00640 1 --2- 172.21.15.12:0/2537183645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ad81038e0 0x7f2ad8199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:54.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.035+0000 7f2acfe00640 1 -- 172.21.15.12:0/2537183645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ad8105490 msgr2=0x7f2ad81a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:54.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.035+0000 7f2acfe00640 1 --2- 172.21.15.12:0/2537183645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ad8105490 0x7f2ad81a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:54.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.035+0000 7f2acfe00640 1 -- 172.21.15.12:0/2537183645 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad81a17c0 con 0x7f2ad8104ad0 2024-03-23T23:01:54.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.035+0000 7f2ad5e00640 1 --2- 172.21.15.12:0/2537183645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ad81038e0 0x7f2ad8199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:54.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.035+0000 7f2acfe00640 1 --2- 172.21.15.12:0/2537183645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ad8104ad0 0x7f2ad819a0e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000e9f0 tx=0x7f2ac000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:54.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.035+0000 7f2acea00640 1 -- 172.21.15.12:0/2537183645 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac000cd80 con 0x7f2ad8104ad0 2024-03-23T23:01:54.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.035+0000 7f2acea00640 1 -- 172.21.15.12:0/2537183645 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ac0005500 con 0x7f2ad8104ad0 2024-03-23T23:01:54.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.035+0000 7f2add000640 1 -- 172.21.15.12:0/2537183645 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ad81a1ab0 con 0x7f2ad8104ad0 2024-03-23T23:01:54.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.035+0000 7f2acea00640 1 -- 172.21.15.12:0/2537183645 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0010690 con 0x7f2ad8104ad0 2024-03-23T23:01:54.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.035+0000 7f2add000640 1 -- 172.21.15.12:0/2537183645 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ad81a1fb0 con 0x7f2ad8104ad0 2024-03-23T23:01:54.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.037+0000 7f2add000640 1 -- 172.21.15.12:0/2537183645 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ad810ac00 con 0x7f2ad8104ad0 2024-03-23T23:01:54.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.040+0000 7f2acea00640 1 -- 172.21.15.12:0/2537183645 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2ac00046c0 con 0x7f2ad8104ad0 2024-03-23T23:01:54.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.041+0000 7f2acea00640 1 --2- 172.21.15.12:0/2537183645 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2abc0a2800 0x7f2abc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:54.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.041+0000 7f2acea00640 1 -- 172.21.15.12:0/2537183645 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2ac00cee20 con 0x7f2ad8104ad0 2024-03-23T23:01:54.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.042+0000 7f2ad5e00640 1 --2- 172.21.15.12:0/2537183645 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2abc0a2800 0x7f2abc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:54.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.042+0000 7f2ad5e00640 1 --2- 172.21.15.12:0/2537183645 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2abc0a2800 0x7f2abc0a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8104930 tx=0x7f2ac8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:54.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.049+0000 7f2acea00640 1 -- 172.21.15.12:0/2537183645 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2ac009be70 con 0x7f2ad8104ad0 2024-03-23T23:01:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.257+0000 7f2add000640 1 -- 172.21.15.12:0/2537183645 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2ad8061ae0 con 0x7f2abc0a2800 2024-03-23T23:01:54.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.267+0000 7f2acea00640 1 -- 172.21.15.12:0/2537183645 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259036 (secure 0 0 0) 0x7f2ad8061ae0 con 0x7f2abc0a2800 2024-03-23T23:01:54.271 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:01:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-23T23:01:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.271+0000 7f2add000640 1 -- 172.21.15.12:0/2537183645 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2abc0a2800 msgr2=0x7f2abc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.271+0000 7f2add000640 1 --2- 172.21.15.12:0/2537183645 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2abc0a2800 0x7f2abc0a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8104930 tx=0x7f2ac8005f90 comp rx=0 tx=0).stop 2024-03-23T23:01:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.272+0000 7f2add000640 1 -- 172.21.15.12:0/2537183645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ad8104ad0 msgr2=0x7f2ad819a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.272+0000 7f2add000640 1 --2- 172.21.15.12:0/2537183645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ad8104ad0 0x7f2ad819a0e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000e9f0 tx=0x7f2ac000edf0 comp rx=0 tx=0).stop 2024-03-23T23:01:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.272+0000 7f2add000640 1 -- 172.21.15.12:0/2537183645 shutdown_connections 2024-03-23T23:01:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.272+0000 7f2add000640 1 --2- 172.21.15.12:0/2537183645 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2abc0a2800 0x7f2abc0a4cc0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.272+0000 7f2add000640 1 --2- 172.21.15.12:0/2537183645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ad8105490 0x7f2ad81a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.272+0000 7f2add000640 1 --2- 172.21.15.12:0/2537183645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ad8104ad0 0x7f2ad819a0e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.272+0000 7f2add000640 1 --2- 172.21.15.12:0/2537183645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ad81038e0 0x7f2ad8199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.272+0000 7f2add000640 1 -- 172.21.15.12:0/2537183645 >> 172.21.15.12:0/2537183645 conn(0x7f2ad80fee90 msgr2=0x7f2ad8107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.272+0000 7f2add000640 1 -- 172.21.15.12:0/2537183645 shutdown_connections 2024-03-23T23:01:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.272+0000 7f2add000640 1 -- 172.21.15.12:0/2537183645 wait complete. 2024-03-23T23:01:54.288 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":393,"stamp":"2024-03-23T23:01:53.318484+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658852,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":994,"num_read_kb":616,"num_write":353,"num_write_kb":3060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1181318,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":784,"ondisk_log_size":784,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369128,"kb_used_data":11656,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572696,"statfs":{"total":1151940427776,"available":1151562440704,"internally_reserved":0,"allocated":11935744,"data_stored":7472400,"data_compressed":112998,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4926,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":78,"num_read_kb":47,"num_write":9,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005487"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.285759+0000","last_change":"2024-03-23T22:59:27.200234+0000","last_active":"2024-03-23T23:00:53.285759+0000","last_peered":"2024-03-23T23:00:53.285759+0000","last_clean":"2024-03-23T23:00:53.285759+0000","last_became_active":"2024-03-23T22:59:27.199864+0000","last_became_peered":"2024-03-23T22:59:27.199864+0000","last_unstale":"2024-03-23T23:00:53.285759+0000","last_undegraded":"2024-03-23T23:00:53.285759+0000","last_fullsized":"2024-03-23T23:00:53.285759+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:56:26.811964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.300077+0000","last_change":"2024-03-23T22:59:27.195101+0000","last_active":"2024-03-23T23:00:53.300077+0000","last_peered":"2024-03-23T23:00:53.300077+0000","last_clean":"2024-03-23T23:00:53.300077+0000","last_became_active":"2024-03-23T22:59:27.194650+0000","last_became_peered":"2024-03-23T22:59:27.194650+0000","last_unstale":"2024-03-23T23:00:53.300077+0000","last_undegraded":"2024-03-23T23:00:53.300077+0000","last_fullsized":"2024-03-23T23:00:53.300077+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:20.396811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.286097+0000","last_change":"2024-03-23T22:59:27.185575+0000","last_active":"2024-03-23T23:00:53.286097+0000","last_peered":"2024-03-23T23:00:53.286097+0000","last_clean":"2024-03-23T23:00:53.286097+0000","last_became_active":"2024-03-23T22:59:27.184328+0000","last_became_peered":"2024-03-23T22:59:27.184328+0000","last_unstale":"2024-03-23T23:00:53.286097+0000","last_undegraded":"2024-03-23T23:00:53.286097+0000","last_fullsized":"2024-03-23T23:00:53.286097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:31:46.221574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.270875+0000","last_change":"2024-03-23T22:59:27.184434+0000","last_active":"2024-03-23T23:00:53.270875+0000","last_peered":"2024-03-23T23:00:53.270875+0000","last_clean":"2024-03-23T23:00:53.270875+0000","last_became_active":"2024-03-23T22:59:27.183970+0000","last_became_peered":"2024-03-23T22:59:27.183970+0000","last_unstale":"2024-03-23T23:00:53.270875+0000","last_undegraded":"2024-03-23T23:00:53.270875+0000","last_fullsized":"2024-03-23T23:00:53.270875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:39:05.517976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.266775+0000","last_change":"2024-03-23T22:59:27.214741+0000","last_active":"2024-03-23T23:00:53.266775+0000","last_peered":"2024-03-23T23:00:53.266775+0000","last_clean":"2024-03-23T23:00:53.266775+0000","last_became_active":"2024-03-23T22:59:27.214311+0000","last_became_peered":"2024-03-23T22:59:27.214311+0000","last_unstale":"2024-03-23T23:00:53.266775+0000","last_undegraded":"2024-03-23T23:00:53.266775+0000","last_fullsized":"2024-03-23T23:00:53.266775+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:35:38.464955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.291377+0000","last_change":"2024-03-23T22:59:27.200695+0000","last_active":"2024-03-23T23:00:53.291377+0000","last_peered":"2024-03-23T23:00:53.291377+0000","last_clean":"2024-03-23T23:00:53.291377+0000","last_became_active":"2024-03-23T22:59:27.200159+0000","last_became_peered":"2024-03-23T22:59:27.200159+0000","last_unstale":"2024-03-23T23:00:53.291377+0000","last_undegraded":"2024-03-23T23:00:53.291377+0000","last_fullsized":"2024-03-23T23:00:53.291377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:01:15.147856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.280147+0000","last_change":"2024-03-23T22:59:27.199928+0000","last_active":"2024-03-23T23:00:53.280147+0000","last_peered":"2024-03-23T23:00:53.280147+0000","last_clean":"2024-03-23T23:00:53.280147+0000","last_became_active":"2024-03-23T22:59:27.199507+0000","last_became_peered":"2024-03-23T22:59:27.199507+0000","last_unstale":"2024-03-23T23:00:53.280147+0000","last_undegraded":"2024-03-23T23:00:53.280147+0000","last_fullsized":"2024-03-23T23:00:53.280147+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:51:19.145785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.295760+0000","last_change":"2024-03-23T22:59:27.217832+0000","last_active":"2024-03-23T23:00:53.295760+0000","last_peered":"2024-03-23T23:00:53.295760+0000","last_clean":"2024-03-23T23:00:53.295760+0000","last_became_active":"2024-03-23T22:59:27.217496+0000","last_became_peered":"2024-03-23T22:59:27.217496+0000","last_unstale":"2024-03-23T23:00:53.295760+0000","last_undegraded":"2024-03-23T23:00:53.295760+0000","last_fullsized":"2024-03-23T23:00:53.295760+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:01.778627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.288651+0000","last_change":"2024-03-23T22:59:27.190982+0000","last_active":"2024-03-23T23:00:53.288651+0000","last_peered":"2024-03-23T23:00:53.288651+0000","last_clean":"2024-03-23T23:00:53.288651+0000","last_became_active":"2024-03-23T22:59:27.190661+0000","last_became_peered":"2024-03-23T22:59:27.190661+0000","last_unstale":"2024-03-23T23:00:53.288651+0000","last_undegraded":"2024-03-23T23:00:53.288651+0000","last_fullsized":"2024-03-23T23:00:53.288651+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:29.822253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.281328+0000","last_change":"2024-03-23T22:59:27.187395+0000","last_active":"2024-03-23T23:00:53.281328+0000","last_peered":"2024-03-23T23:00:53.281328+0000","last_clean":"2024-03-23T23:00:53.281328+0000","last_became_active":"2024-03-23T22:59:27.186849+0000","last_became_peered":"2024-03-23T22:59:27.186849+0000","last_unstale":"2024-03-23T23:00:53.281328+0000","last_undegraded":"2024-03-23T23:00:53.281328+0000","last_fullsized":"2024-03-23T23:00:53.281328+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:05:48.047102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.286208+0000","last_change":"2024-03-23T22:59:27.190023+0000","last_active":"2024-03-23T23:00:53.286208+0000","last_peered":"2024-03-23T23:00:53.286208+0000","last_clean":"2024-03-23T23:00:53.286208+0000","last_became_active":"2024-03-23T22:59:27.189697+0000","last_became_peered":"2024-03-23T22:59:27.189697+0000","last_unstale":"2024-03-23T23:00:53.286208+0000","last_undegraded":"2024-03-23T23:00:53.286208+0000","last_fullsized":"2024-03-23T23:00:53.286208+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:59:51.121055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.264109+0000","last_change":"2024-03-23T22:59:27.187382+0000","last_active":"2024-03-23T23:00:53.264109+0000","last_peered":"2024-03-23T23:00:53.264109+0000","last_clean":"2024-03-23T23:00:53.264109+0000","last_became_active":"2024-03-23T22:59:27.186931+0000","last_became_peered":"2024-03-23T22:59:27.186931+0000","last_unstale":"2024-03-23T23:00:53.264109+0000","last_undegraded":"2024-03-23T23:00:53.264109+0000","last_fullsized":"2024-03-23T23:00:53.264109+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:33.583123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.293942+0000","last_change":"2024-03-23T22:59:27.202577+0000","last_active":"2024-03-23T23:00:53.293942+0000","last_peered":"2024-03-23T23:00:53.293942+0000","last_clean":"2024-03-23T23:00:53.293942+0000","last_became_active":"2024-03-23T22:59:27.202122+0000","last_became_peered":"2024-03-23T22:59:27.202122+0000","last_unstale":"2024-03-23T23:00:53.293942+0000","last_undegraded":"2024-03-23T23:00:53.293942+0000","last_fullsized":"2024-03-23T23:00:53.293942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:01:10.809635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.298452+0000","last_change":"2024-03-23T22:59:27.192915+0000","last_active":"2024-03-23T23:00:53.298452+0000","last_peered":"2024-03-23T23:00:53.298452+0000","last_clean":"2024-03-23T23:00:53.298452+0000","last_became_active":"2024-03-23T22:59:27.192107+0000","last_became_peered":"2024-03-23T22:59:27.192107+0000","last_unstale":"2024-03-23T23:00:53.298452+0000","last_undegraded":"2024-03-23T23:00:53.298452+0000","last_fullsized":"2024-03-23T23:00:53.298452+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:21:19.169211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.300473+0000","last_change":"2024-03-23T22:59:27.192156+0000","last_active":"2024-03-23T23:00:53.300473+0000","last_peered":"2024-03-23T23:00:53.300473+0000","last_clean":"2024-03-23T23:00:53.300473+0000","last_became_active":"2024-03-23T22:59:27.191594+0000","last_became_peered":"2024-03-23T22:59:27.191594+0000","last_unstale":"2024-03-23T23:00:53.300473+0000","last_undegraded":"2024-03-23T23:00:53.300473+0000","last_fullsized":"2024-03-23T23:00:53.300473+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:00:45.480281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.299835+0000","last_change":"2024-03-23T22:59:27.186048+0000","last_active":"2024-03-23T23:00:53.299835+0000","last_peered":"2024-03-23T23:00:53.299835+0000","last_clean":"2024-03-23T23:00:53.299835+0000","last_became_active":"2024-03-23T22:59:27.185710+0000","last_became_peered":"2024-03-23T22:59:27.185710+0000","last_unstale":"2024-03-23T23:00:53.299835+0000","last_undegraded":"2024-03-23T23:00:53.299835+0000","last_fullsized":"2024-03-23T23:00:53.299835+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:06.792665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.298495+0000","last_change":"2024-03-23T22:59:27.206290+0000","last_active":"2024-03-23T23:00:53.298495+0000","last_peered":"2024-03-23T23:00:53.298495+0000","last_clean":"2024-03-23T23:00:53.298495+0000","last_became_active":"2024-03-23T22:59:27.205997+0000","last_became_peered":"2024-03-23T22:59:27.205997+0000","last_unstale":"2024-03-23T23:00:53.298495+0000","last_undegraded":"2024-03-23T23:00:53.298495+0000","last_fullsized":"2024-03-23T23:00:53.298495+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:38:42.006166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.289029+0000","last_change":"2024-03-23T22:59:27.203671+0000","last_active":"2024-03-23T23:00:53.289029+0000","last_peered":"2024-03-23T23:00:53.289029+0000","last_clean":"2024-03-23T23:00:53.289029+0000","last_became_active":"2024-03-23T22:59:27.203273+0000","last_became_peered":"2024-03-23T22:59:27.203273+0000","last_unstale":"2024-03-23T23:00:53.289029+0000","last_undegraded":"2024-03-23T23:00:53.289029+0000","last_fullsized":"2024-03-23T23:00:53.289029+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:48:16.018671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.292388+0000","last_change":"2024-03-23T22:59:27.210063+0000","last_active":"2024-03-23T23:00:53.292388+0000","last_peered":"2024-03-23T23:00:53.292388+0000","last_clean":"2024-03-23T23:00:53.292388+0000","last_became_active":"2024-03-23T22:59:27.209698+0000","last_became_peered":"2024-03-23T22:59:27.209698+0000","last_unstale":"2024-03-23T23:00:53.292388+0000","last_undegraded":"2024-03-23T23:00:53.292388+0000","last_fullsized":"2024-03-23T23:00:53.292388+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:19:01.408901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.358191+0000","last_change":"2024-03-23T22:59:26.445053+0000","last_active":"2024-03-23T23:00:24.358191+0000","last_peered":"2024-03-23T23:00:24.358191+0000","last_clean":"2024-03-23T23:00:24.358191+0000","last_became_active":"2024-03-23T22:59:26.444660+0000","last_became_peered":"2024-03-23T22:59:26.444660+0000","last_unstale":"2024-03-23T23:00:24.358191+0000","last_undegraded":"2024-03-23T23:00:24.358191+0000","last_fullsized":"2024-03-23T23:00:24.358191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:19:16.958228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.284909+0000","last_change":"2024-03-23T22:59:27.208652+0000","last_active":"2024-03-23T23:00:53.284909+0000","last_peered":"2024-03-23T23:00:53.284909+0000","last_clean":"2024-03-23T23:00:53.284909+0000","last_became_active":"2024-03-23T22:59:27.207724+0000","last_became_peered":"2024-03-23T22:59:27.207724+0000","last_unstale":"2024-03-23T23:00:53.284909+0000","last_undegraded":"2024-03-23T23:00:53.284909+0000","last_fullsized":"2024-03-23T23:00:53.284909+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:08.986439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.368748+0000","last_change":"2024-03-23T22:59:26.445371+0000","last_active":"2024-03-23T23:00:24.368748+0000","last_peered":"2024-03-23T23:00:24.368748+0000","last_clean":"2024-03-23T23:00:24.368748+0000","last_became_active":"2024-03-23T22:59:26.445016+0000","last_became_peered":"2024-03-23T22:59:26.445016+0000","last_unstale":"2024-03-23T23:00:24.368748+0000","last_undegraded":"2024-03-23T23:00:24.368748+0000","last_fullsized":"2024-03-23T23:00:24.368748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:31:44.003218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.300501+0000","last_change":"2024-03-23T22:59:27.200624+0000","last_active":"2024-03-23T23:00:53.300501+0000","last_peered":"2024-03-23T23:00:53.300501+0000","last_clean":"2024-03-23T23:00:53.300501+0000","last_became_active":"2024-03-23T22:59:27.200207+0000","last_became_peered":"2024-03-23T22:59:27.200207+0000","last_unstale":"2024-03-23T23:00:53.300501+0000","last_undegraded":"2024-03-23T23:00:53.300501+0000","last_fullsized":"2024-03-23T23:00:53.300501+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:05:31.015869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.380616+0000","last_change":"2024-03-23T23:00:24.380296+0000","last_active":"2024-03-23T23:00:24.380616+0000","last_peered":"2024-03-23T23:00:24.380616+0000","last_clean":"2024-03-23T23:00:24.380616+0000","last_became_active":"2024-03-23T23:00:24.374344+0000","last_became_peered":"2024-03-23T23:00:24.374344+0000","last_unstale":"2024-03-23T23:00:24.380616+0000","last_undegraded":"2024-03-23T23:00:24.380616+0000","last_fullsized":"2024-03-23T23:00:24.380616+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:30:20.174783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.288535+0000","last_change":"2024-03-23T22:59:27.204157+0000","last_active":"2024-03-23T23:00:53.288535+0000","last_peered":"2024-03-23T23:00:53.288535+0000","last_clean":"2024-03-23T23:00:53.288535+0000","last_became_active":"2024-03-23T22:59:27.203559+0000","last_became_peered":"2024-03-23T22:59:27.203559+0000","last_unstale":"2024-03-23T23:00:53.288535+0000","last_undegraded":"2024-03-23T23:00:53.288535+0000","last_fullsized":"2024-03-23T23:00:53.288535+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:43.834369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.361574+0000","last_change":"2024-03-23T23:00:24.361233+0000","last_active":"2024-03-23T23:00:24.361574+0000","last_peered":"2024-03-23T23:00:24.361574+0000","last_clean":"2024-03-23T23:00:24.361574+0000","last_became_active":"2024-03-23T23:00:24.358048+0000","last_became_peered":"2024-03-23T23:00:24.358048+0000","last_unstale":"2024-03-23T23:00:24.361574+0000","last_undegraded":"2024-03-23T23:00:24.361574+0000","last_fullsized":"2024-03-23T23:00:24.361574+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:38:37.792273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.286722+0000","last_change":"2024-03-23T22:59:27.186599+0000","last_active":"2024-03-23T23:00:53.286722+0000","last_peered":"2024-03-23T23:00:53.286722+0000","last_clean":"2024-03-23T23:00:53.286722+0000","last_became_active":"2024-03-23T22:59:27.186262+0000","last_became_peered":"2024-03-23T22:59:27.186262+0000","last_unstale":"2024-03-23T23:00:53.286722+0000","last_undegraded":"2024-03-23T23:00:53.286722+0000","last_fullsized":"2024-03-23T23:00:53.286722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:11:39.545558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":166,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:01:49.342977+0000","last_change":"2024-03-23T22:59:26.430096+0000","last_active":"2024-03-23T23:01:49.342977+0000","last_peered":"2024-03-23T23:01:49.342977+0000","last_clean":"2024-03-23T23:01:49.342977+0000","last_became_active":"2024-03-23T22:59:26.429740+0000","last_became_peered":"2024-03-23T22:59:26.429740+0000","last_unstale":"2024-03-23T23:01:49.342977+0000","last_undegraded":"2024-03-23T23:01:49.342977+0000","last_fullsized":"2024-03-23T23:01:49.342977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:54.831689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":928,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":141,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.262380+0000","last_change":"2024-03-23T22:59:27.202750+0000","last_active":"2024-03-23T23:00:53.262380+0000","last_peered":"2024-03-23T23:00:53.262380+0000","last_clean":"2024-03-23T23:00:53.262380+0000","last_became_active":"2024-03-23T22:59:27.202391+0000","last_became_peered":"2024-03-23T22:59:27.202391+0000","last_unstale":"2024-03-23T23:00:53.262380+0000","last_undegraded":"2024-03-23T23:00:53.262380+0000","last_fullsized":"2024-03-23T23:00:53.262380+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:27:18.370792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.365876+0000","last_change":"2024-03-23T22:59:26.425607+0000","last_active":"2024-03-23T23:00:24.365876+0000","last_peered":"2024-03-23T23:00:24.365876+0000","last_clean":"2024-03-23T23:00:24.365876+0000","last_became_active":"2024-03-23T22:59:26.425208+0000","last_became_peered":"2024-03-23T22:59:26.425208+0000","last_unstale":"2024-03-23T23:00:24.365876+0000","last_undegraded":"2024-03-23T23:00:24.365876+0000","last_fullsized":"2024-03-23T23:00:24.365876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:51.071662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.282112+0000","last_change":"2024-03-23T22:59:27.215637+0000","last_active":"2024-03-23T23:00:53.282112+0000","last_peered":"2024-03-23T23:00:53.282112+0000","last_clean":"2024-03-23T23:00:53.282112+0000","last_became_active":"2024-03-23T22:59:27.215339+0000","last_became_peered":"2024-03-23T22:59:27.215339+0000","last_unstale":"2024-03-23T23:00:53.282112+0000","last_undegraded":"2024-03-23T23:00:53.282112+0000","last_fullsized":"2024-03-23T23:00:53.282112+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:25:42.097326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.377892+0000","last_change":"2024-03-23T22:59:26.442023+0000","last_active":"2024-03-23T23:00:24.377892+0000","last_peered":"2024-03-23T23:00:24.377892+0000","last_clean":"2024-03-23T23:00:24.377892+0000","last_became_active":"2024-03-23T22:59:26.441661+0000","last_became_peered":"2024-03-23T22:59:26.441661+0000","last_unstale":"2024-03-23T23:00:24.377892+0000","last_undegraded":"2024-03-23T23:00:24.377892+0000","last_fullsized":"2024-03-23T23:00:24.377892+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:59:14.914860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.279352+0000","last_change":"2024-03-23T23:00:24.384190+0000","last_active":"2024-03-23T23:00:53.279352+0000","last_peered":"2024-03-23T23:00:53.279352+0000","last_clean":"2024-03-23T23:00:53.279352+0000","last_became_active":"2024-03-23T23:00:24.382141+0000","last_became_peered":"2024-03-23T23:00:24.382141+0000","last_unstale":"2024-03-23T23:00:53.279352+0000","last_undegraded":"2024-03-23T23:00:53.279352+0000","last_fullsized":"2024-03-23T23:00:53.279352+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:48:52.311407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356353+0000","last_change":"2024-03-23T22:59:26.430147+0000","last_active":"2024-03-23T23:00:24.356353+0000","last_peered":"2024-03-23T23:00:24.356353+0000","last_clean":"2024-03-23T23:00:24.356353+0000","last_became_active":"2024-03-23T22:59:26.429843+0000","last_became_peered":"2024-03-23T22:59:26.429843+0000","last_unstale":"2024-03-23T23:00:24.356353+0000","last_undegraded":"2024-03-23T23:00:24.356353+0000","last_fullsized":"2024-03-23T23:00:24.356353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:06:45.691894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.288385+0000","last_change":"2024-03-23T22:59:27.212235+0000","last_active":"2024-03-23T23:00:53.288385+0000","last_peered":"2024-03-23T23:00:53.288385+0000","last_clean":"2024-03-23T23:00:53.288385+0000","last_became_active":"2024-03-23T22:59:27.211919+0000","last_became_peered":"2024-03-23T22:59:27.211919+0000","last_unstale":"2024-03-23T23:00:53.288385+0000","last_undegraded":"2024-03-23T23:00:53.288385+0000","last_fullsized":"2024-03-23T23:00:53.288385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:24.752109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":165,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:01:52.346610+0000","last_change":"2024-03-23T22:59:26.440623+0000","last_active":"2024-03-23T23:01:52.346610+0000","last_peered":"2024-03-23T23:01:52.346610+0000","last_clean":"2024-03-23T23:01:52.346610+0000","last_became_active":"2024-03-23T22:59:26.440204+0000","last_became_peered":"2024-03-23T22:59:26.440204+0000","last_unstale":"2024-03-23T23:01:52.346610+0000","last_undegraded":"2024-03-23T23:01:52.346610+0000","last_fullsized":"2024-03-23T23:01:52.346610+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:58:44.088552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":141,"num_read_kb":141,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.287377+0000","last_change":"2024-03-23T22:59:27.203916+0000","last_active":"2024-03-23T23:00:53.287377+0000","last_peered":"2024-03-23T23:00:53.287377+0000","last_clean":"2024-03-23T23:00:53.287377+0000","last_became_active":"2024-03-23T22:59:27.203565+0000","last_became_peered":"2024-03-23T22:59:27.203565+0000","last_unstale":"2024-03-23T23:00:53.287377+0000","last_undegraded":"2024-03-23T23:00:53.287377+0000","last_fullsized":"2024-03-23T23:00:53.287377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:06:39.635025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357614+0000","last_change":"2024-03-23T22:59:26.430080+0000","last_active":"2024-03-23T23:00:24.357614+0000","last_peered":"2024-03-23T23:00:24.357614+0000","last_clean":"2024-03-23T23:00:24.357614+0000","last_became_active":"2024-03-23T22:59:26.429618+0000","last_became_peered":"2024-03-23T22:59:26.429618+0000","last_unstale":"2024-03-23T23:00:24.357614+0000","last_undegraded":"2024-03-23T23:00:24.357614+0000","last_fullsized":"2024-03-23T23:00:24.357614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:01.963573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.300287+0000","last_change":"2024-03-23T22:59:27.190505+0000","last_active":"2024-03-23T23:00:53.300287+0000","last_peered":"2024-03-23T23:00:53.300287+0000","last_clean":"2024-03-23T23:00:53.300287+0000","last_became_active":"2024-03-23T22:59:27.190088+0000","last_became_peered":"2024-03-23T22:59:27.190088+0000","last_unstale":"2024-03-23T23:00:53.300287+0000","last_undegraded":"2024-03-23T23:00:53.300287+0000","last_fullsized":"2024-03-23T23:00:53.300287+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:29:04.255832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357315+0000","last_change":"2024-03-23T22:59:26.438329+0000","last_active":"2024-03-23T23:00:24.357315+0000","last_peered":"2024-03-23T23:00:24.357315+0000","last_clean":"2024-03-23T23:00:24.357315+0000","last_became_active":"2024-03-23T22:59:26.438033+0000","last_became_peered":"2024-03-23T22:59:26.438033+0000","last_unstale":"2024-03-23T23:00:24.357315+0000","last_undegraded":"2024-03-23T23:00:24.357315+0000","last_fullsized":"2024-03-23T23:00:24.357315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:02:38.789574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.287183+0000","last_change":"2024-03-23T22:59:27.188417+0000","last_active":"2024-03-23T23:00:53.287183+0000","last_peered":"2024-03-23T23:00:53.287183+0000","last_clean":"2024-03-23T23:00:53.287183+0000","last_became_active":"2024-03-23T22:59:27.188023+0000","last_became_peered":"2024-03-23T22:59:27.188023+0000","last_unstale":"2024-03-23T23:00:53.287183+0000","last_undegraded":"2024-03-23T23:00:53.287183+0000","last_fullsized":"2024-03-23T23:00:53.287183+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:19:11.014604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'33","reported_seq":347,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:01:52.347319+0000","last_change":"2024-03-23T22:59:26.425963+0000","last_active":"2024-03-23T23:01:52.347319+0000","last_peered":"2024-03-23T23:01:52.347319+0000","last_clean":"2024-03-23T23:01:52.347319+0000","last_became_active":"2024-03-23T22:59:26.425640+0000","last_became_peered":"2024-03-23T22:59:26.425640+0000","last_unstale":"2024-03-23T23:01:52.347319+0000","last_undegraded":"2024-03-23T23:01:52.347319+0000","last_fullsized":"2024-03-23T23:01:52.347319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:11:26.529034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64334,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":294,"num_read_kb":210,"num_write":34,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.256447+0000","last_change":"2024-03-23T22:59:27.187254+0000","last_active":"2024-03-23T23:00:53.256447+0000","last_peered":"2024-03-23T23:00:53.256447+0000","last_clean":"2024-03-23T23:00:53.256447+0000","last_became_active":"2024-03-23T22:59:27.186849+0000","last_became_peered":"2024-03-23T22:59:27.186849+0000","last_unstale":"2024-03-23T23:00:53.256447+0000","last_undegraded":"2024-03-23T23:00:53.256447+0000","last_fullsized":"2024-03-23T23:00:53.256447+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:35:30.958565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.358477+0000","last_change":"2024-03-23T22:59:26.429767+0000","last_active":"2024-03-23T23:00:24.358477+0000","last_peered":"2024-03-23T23:00:24.358477+0000","last_clean":"2024-03-23T23:00:24.358477+0000","last_became_active":"2024-03-23T22:59:26.429449+0000","last_became_peered":"2024-03-23T22:59:26.429449+0000","last_unstale":"2024-03-23T23:00:24.358477+0000","last_undegraded":"2024-03-23T23:00:24.358477+0000","last_fullsized":"2024-03-23T23:00:24.358477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:46:13.752427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.298386+0000","last_change":"2024-03-23T22:59:27.223544+0000","last_active":"2024-03-23T23:00:53.298386+0000","last_peered":"2024-03-23T23:00:53.298386+0000","last_clean":"2024-03-23T23:00:53.298386+0000","last_became_active":"2024-03-23T22:59:27.223177+0000","last_became_peered":"2024-03-23T22:59:27.223177+0000","last_unstale":"2024-03-23T23:00:53.298386+0000","last_undegraded":"2024-03-23T23:00:53.298386+0000","last_fullsized":"2024-03-23T23:00:53.298386+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:33:18.521579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356567+0000","last_change":"2024-03-23T22:59:26.427781+0000","last_active":"2024-03-23T23:00:24.356567+0000","last_peered":"2024-03-23T23:00:24.356567+0000","last_clean":"2024-03-23T23:00:24.356567+0000","last_became_active":"2024-03-23T22:59:26.427458+0000","last_became_peered":"2024-03-23T22:59:26.427458+0000","last_unstale":"2024-03-23T23:00:24.356567+0000","last_undegraded":"2024-03-23T23:00:24.356567+0000","last_fullsized":"2024-03-23T23:00:24.356567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:50:33.956611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.295652+0000","last_change":"2024-03-23T22:59:27.203681+0000","last_active":"2024-03-23T23:00:53.295652+0000","last_peered":"2024-03-23T23:00:53.295652+0000","last_clean":"2024-03-23T23:00:53.295652+0000","last_became_active":"2024-03-23T22:59:27.202457+0000","last_became_peered":"2024-03-23T22:59:27.202457+0000","last_unstale":"2024-03-23T23:00:53.295652+0000","last_undegraded":"2024-03-23T23:00:53.295652+0000","last_fullsized":"2024-03-23T23:00:53.295652+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:55:46.725350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'3","reported_seq":164,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:01:49.431259+0000","last_change":"2024-03-23T22:59:26.428517+0000","last_active":"2024-03-23T23:01:49.431259+0000","last_peered":"2024-03-23T23:01:49.431259+0000","last_clean":"2024-03-23T23:01:49.431259+0000","last_became_active":"2024-03-23T22:59:26.428130+0000","last_became_peered":"2024-03-23T22:59:26.428130+0000","last_unstale":"2024-03-23T23:01:49.431259+0000","last_undegraded":"2024-03-23T23:01:49.431259+0000","last_fullsized":"2024-03-23T23:01:49.431259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:01.185776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":142,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.283053+0000","last_change":"2024-03-23T23:00:24.367049+0000","last_active":"2024-03-23T23:00:53.283053+0000","last_peered":"2024-03-23T23:00:53.283053+0000","last_clean":"2024-03-23T23:00:53.283053+0000","last_became_active":"2024-03-23T23:00:24.366636+0000","last_became_peered":"2024-03-23T23:00:24.366636+0000","last_unstale":"2024-03-23T23:00:53.283053+0000","last_undegraded":"2024-03-23T23:00:53.283053+0000","last_fullsized":"2024-03-23T23:00:53.283053+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:07.068500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356958+0000","last_change":"2024-03-23T22:59:26.442637+0000","last_active":"2024-03-23T23:00:24.356958+0000","last_peered":"2024-03-23T23:00:24.356958+0000","last_clean":"2024-03-23T23:00:24.356958+0000","last_became_active":"2024-03-23T22:59:26.442267+0000","last_became_peered":"2024-03-23T22:59:26.442267+0000","last_unstale":"2024-03-23T23:00:24.356958+0000","last_undegraded":"2024-03-23T23:00:24.356958+0000","last_fullsized":"2024-03-23T23:00:24.356958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:49.975048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.286407+0000","last_change":"2024-03-23T22:59:27.187812+0000","last_active":"2024-03-23T23:00:53.286407+0000","last_peered":"2024-03-23T23:00:53.286407+0000","last_clean":"2024-03-23T23:00:53.286407+0000","last_became_active":"2024-03-23T22:59:27.187423+0000","last_became_peered":"2024-03-23T22:59:27.187423+0000","last_unstale":"2024-03-23T23:00:53.286407+0000","last_undegraded":"2024-03-23T23:00:53.286407+0000","last_fullsized":"2024-03-23T23:00:53.286407+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:12:16.427695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357353+0000","last_change":"2024-03-23T22:59:26.423875+0000","last_active":"2024-03-23T23:00:24.357353+0000","last_peered":"2024-03-23T23:00:24.357353+0000","last_clean":"2024-03-23T23:00:24.357353+0000","last_became_active":"2024-03-23T22:59:26.423513+0000","last_became_peered":"2024-03-23T22:59:26.423513+0000","last_unstale":"2024-03-23T23:00:24.357353+0000","last_undegraded":"2024-03-23T23:00:24.357353+0000","last_fullsized":"2024-03-23T23:00:24.357353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:43.173365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.268696+0000","last_change":"2024-03-23T22:59:27.191454+0000","last_active":"2024-03-23T23:00:53.268696+0000","last_peered":"2024-03-23T23:00:53.268696+0000","last_clean":"2024-03-23T23:00:53.268696+0000","last_became_active":"2024-03-23T22:59:27.190927+0000","last_became_peered":"2024-03-23T22:59:27.190927+0000","last_unstale":"2024-03-23T23:00:53.268696+0000","last_undegraded":"2024-03-23T23:00:53.268696+0000","last_fullsized":"2024-03-23T23:00:53.268696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:00.333702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.358997+0000","last_change":"2024-03-23T22:59:26.443238+0000","last_active":"2024-03-23T23:00:24.358997+0000","last_peered":"2024-03-23T23:00:24.358997+0000","last_clean":"2024-03-23T23:00:24.358997+0000","last_became_active":"2024-03-23T22:59:26.442834+0000","last_became_peered":"2024-03-23T22:59:26.442834+0000","last_unstale":"2024-03-23T23:00:24.358997+0000","last_undegraded":"2024-03-23T23:00:24.358997+0000","last_fullsized":"2024-03-23T23:00:24.358997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:11:21.072836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.295551+0000","last_change":"2024-03-23T22:59:27.212864+0000","last_active":"2024-03-23T23:00:53.295551+0000","last_peered":"2024-03-23T23:00:53.295551+0000","last_clean":"2024-03-23T23:00:53.295551+0000","last_became_active":"2024-03-23T22:59:27.212455+0000","last_became_peered":"2024-03-23T22:59:27.212455+0000","last_unstale":"2024-03-23T23:00:53.295551+0000","last_undegraded":"2024-03-23T23:00:53.295551+0000","last_fullsized":"2024-03-23T23:00:53.295551+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:25:59.122531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357120+0000","last_change":"2024-03-23T22:59:26.432838+0000","last_active":"2024-03-23T23:00:24.357120+0000","last_peered":"2024-03-23T23:00:24.357120+0000","last_clean":"2024-03-23T23:00:24.357120+0000","last_became_active":"2024-03-23T22:59:26.432544+0000","last_became_peered":"2024-03-23T22:59:26.432544+0000","last_unstale":"2024-03-23T23:00:24.357120+0000","last_undegraded":"2024-03-23T23:00:24.357120+0000","last_fullsized":"2024-03-23T23:00:24.357120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:32:52.121569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.290395+0000","last_change":"2024-03-23T23:00:24.376926+0000","last_active":"2024-03-23T23:00:53.290395+0000","last_peered":"2024-03-23T23:00:53.290395+0000","last_clean":"2024-03-23T23:00:53.290395+0000","last_became_active":"2024-03-23T23:00:24.376515+0000","last_became_peered":"2024-03-23T23:00:24.376515+0000","last_unstale":"2024-03-23T23:00:53.290395+0000","last_undegraded":"2024-03-23T23:00:53.290395+0000","last_fullsized":"2024-03-23T23:00:53.290395+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:55:57.730564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.279322+0000","last_change":"2024-03-23T23:00:24.381861+0000","last_active":"2024-03-23T23:00:53.279322+0000","last_peered":"2024-03-23T23:00:53.279322+0000","last_clean":"2024-03-23T23:00:53.279322+0000","last_became_active":"2024-03-23T23:00:24.381472+0000","last_became_peered":"2024-03-23T23:00:24.381472+0000","last_unstale":"2024-03-23T23:00:53.279322+0000","last_undegraded":"2024-03-23T23:00:53.279322+0000","last_fullsized":"2024-03-23T23:00:53.279322+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:44:07.518911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":162,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:01:48.428986+0000","last_change":"2024-03-23T22:59:26.450679+0000","last_active":"2024-03-23T23:01:48.428986+0000","last_peered":"2024-03-23T23:01:48.428986+0000","last_clean":"2024-03-23T23:01:48.428986+0000","last_became_active":"2024-03-23T22:59:26.450234+0000","last_became_peered":"2024-03-23T22:59:26.450234+0000","last_unstale":"2024-03-23T23:01:48.428986+0000","last_undegraded":"2024-03-23T23:01:48.428986+0000","last_fullsized":"2024-03-23T23:01:48.428986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:52:25.869194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.289045+0000","last_change":"2024-03-23T22:59:27.220171+0000","last_active":"2024-03-23T23:00:53.289045+0000","last_peered":"2024-03-23T23:00:53.289045+0000","last_clean":"2024-03-23T23:00:53.289045+0000","last_became_active":"2024-03-23T22:59:27.219787+0000","last_became_peered":"2024-03-23T22:59:27.219787+0000","last_unstale":"2024-03-23T23:00:53.289045+0000","last_undegraded":"2024-03-23T23:00:53.289045+0000","last_fullsized":"2024-03-23T23:00:53.289045+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:51:52.634264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.373128+0000","last_change":"2024-03-23T22:59:26.440205+0000","last_active":"2024-03-23T23:00:24.373128+0000","last_peered":"2024-03-23T23:00:24.373128+0000","last_clean":"2024-03-23T23:00:24.373128+0000","last_became_active":"2024-03-23T22:59:26.439752+0000","last_became_peered":"2024-03-23T22:59:26.439752+0000","last_unstale":"2024-03-23T23:00:24.373128+0000","last_undegraded":"2024-03-23T23:00:24.373128+0000","last_fullsized":"2024-03-23T23:00:24.373128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:55:25.661887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":345,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356800+0000","last_change":"2024-03-23T22:58:06.171317+0000","last_active":"2024-03-23T23:00:24.356800+0000","last_peered":"2024-03-23T23:00:24.356800+0000","last_clean":"2024-03-23T23:00:24.356800+0000","last_became_active":"2024-03-23T22:58:06.170816+0000","last_became_peered":"2024-03-23T22:58:06.170816+0000","last_unstale":"2024-03-23T23:00:24.356800+0000","last_undegraded":"2024-03-23T23:00:24.356800+0000","last_fullsized":"2024-03-23T23:00:24.356800+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:54:11.991284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:54:11.991284+0000","last_clean_scrub_stamp":"2024-03-23T22:54:11.991284+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:20:55.980613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.283579+0000","last_change":"2024-03-23T22:59:27.200348+0000","last_active":"2024-03-23T23:00:53.283579+0000","last_peered":"2024-03-23T23:00:53.283579+0000","last_clean":"2024-03-23T23:00:53.283579+0000","last_became_active":"2024-03-23T22:59:27.199979+0000","last_became_peered":"2024-03-23T22:59:27.199979+0000","last_unstale":"2024-03-23T23:00:53.283579+0000","last_undegraded":"2024-03-23T23:00:53.283579+0000","last_fullsized":"2024-03-23T23:00:53.283579+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:31:19.258890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356946+0000","last_change":"2024-03-23T22:59:26.431837+0000","last_active":"2024-03-23T23:00:24.356946+0000","last_peered":"2024-03-23T23:00:24.356946+0000","last_clean":"2024-03-23T23:00:24.356946+0000","last_became_active":"2024-03-23T22:59:26.431580+0000","last_became_peered":"2024-03-23T22:59:26.431580+0000","last_unstale":"2024-03-23T23:00:24.356946+0000","last_undegraded":"2024-03-23T23:00:24.356946+0000","last_fullsized":"2024-03-23T23:00:24.356946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:49:53.316952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356929+0000","last_change":"2024-03-23T22:59:26.442878+0000","last_active":"2024-03-23T23:00:24.356929+0000","last_peered":"2024-03-23T23:00:24.356929+0000","last_clean":"2024-03-23T23:00:24.356929+0000","last_became_active":"2024-03-23T22:59:26.442303+0000","last_became_peered":"2024-03-23T22:59:26.442303+0000","last_unstale":"2024-03-23T23:00:24.356929+0000","last_undegraded":"2024-03-23T23:00:24.356929+0000","last_fullsized":"2024-03-23T23:00:24.356929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:51.717553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.279622+0000","last_change":"2024-03-23T22:59:27.212550+0000","last_active":"2024-03-23T23:00:53.279622+0000","last_peered":"2024-03-23T23:00:53.279622+0000","last_clean":"2024-03-23T23:00:53.279622+0000","last_became_active":"2024-03-23T22:59:27.212105+0000","last_became_peered":"2024-03-23T22:59:27.212105+0000","last_unstale":"2024-03-23T23:00:53.279622+0000","last_undegraded":"2024-03-23T23:00:53.279622+0000","last_fullsized":"2024-03-23T23:00:53.279622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:23:37.374081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'11","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:01:36.692498+0000","last_change":"2024-03-23T22:59:26.452146+0000","last_active":"2024-03-23T23:01:36.692498+0000","last_peered":"2024-03-23T23:01:36.692498+0000","last_clean":"2024-03-23T23:01:36.692498+0000","last_became_active":"2024-03-23T22:59:26.450398+0000","last_became_peered":"2024-03-23T22:59:26.450398+0000","last_unstale":"2024-03-23T23:01:36.692498+0000","last_undegraded":"2024-03-23T23:01:36.692498+0000","last_fullsized":"2024-03-23T23:01:36.692498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:55:47.085457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.295334+0000","last_change":"2024-03-23T22:59:27.209690+0000","last_active":"2024-03-23T23:00:53.295334+0000","last_peered":"2024-03-23T23:00:53.295334+0000","last_clean":"2024-03-23T23:00:53.295334+0000","last_became_active":"2024-03-23T22:59:27.209187+0000","last_became_peered":"2024-03-23T22:59:27.209187+0000","last_unstale":"2024-03-23T23:00:53.295334+0000","last_undegraded":"2024-03-23T23:00:53.295334+0000","last_fullsized":"2024-03-23T23:00:53.295334+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:19:45.315459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.373744+0000","last_change":"2024-03-23T22:59:26.434958+0000","last_active":"2024-03-23T23:00:24.373744+0000","last_peered":"2024-03-23T23:00:24.373744+0000","last_clean":"2024-03-23T23:00:24.373744+0000","last_became_active":"2024-03-23T22:59:26.434611+0000","last_became_peered":"2024-03-23T22:59:26.434611+0000","last_unstale":"2024-03-23T23:00:24.373744+0000","last_undegraded":"2024-03-23T23:00:24.373744+0000","last_fullsized":"2024-03-23T23:00:24.373744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:42:18.201885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.300163+0000","last_change":"2024-03-23T22:59:27.211700+0000","last_active":"2024-03-23T23:00:53.300163+0000","last_peered":"2024-03-23T23:00:53.300163+0000","last_clean":"2024-03-23T23:00:53.300163+0000","last_became_active":"2024-03-23T22:59:27.211248+0000","last_became_peered":"2024-03-23T22:59:27.211248+0000","last_unstale":"2024-03-23T23:00:53.300163+0000","last_undegraded":"2024-03-23T23:00:53.300163+0000","last_fullsized":"2024-03-23T23:00:53.300163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:49.868397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356086+0000","last_change":"2024-03-23T22:59:26.442625+0000","last_active":"2024-03-23T23:00:24.356086+0000","last_peered":"2024-03-23T23:00:24.356086+0000","last_clean":"2024-03-23T23:00:24.356086+0000","last_became_active":"2024-03-23T22:59:26.442312+0000","last_became_peered":"2024-03-23T22:59:26.442312+0000","last_unstale":"2024-03-23T23:00:24.356086+0000","last_undegraded":"2024-03-23T23:00:24.356086+0000","last_fullsized":"2024-03-23T23:00:24.356086+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:27:38.808784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.282372+0000","last_change":"2024-03-23T22:59:27.187363+0000","last_active":"2024-03-23T23:00:53.282372+0000","last_peered":"2024-03-23T23:00:53.282372+0000","last_clean":"2024-03-23T23:00:53.282372+0000","last_became_active":"2024-03-23T22:59:27.187007+0000","last_became_peered":"2024-03-23T22:59:27.187007+0000","last_unstale":"2024-03-23T23:00:53.282372+0000","last_undegraded":"2024-03-23T23:00:53.282372+0000","last_fullsized":"2024-03-23T23:00:53.282372+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:41:37.041254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.369860+0000","last_change":"2024-03-23T22:59:26.445753+0000","last_active":"2024-03-23T23:00:24.369860+0000","last_peered":"2024-03-23T23:00:24.369860+0000","last_clean":"2024-03-23T23:00:24.369860+0000","last_became_active":"2024-03-23T22:59:26.445167+0000","last_became_peered":"2024-03-23T22:59:26.445167+0000","last_unstale":"2024-03-23T23:00:24.369860+0000","last_undegraded":"2024-03-23T23:00:24.369860+0000","last_fullsized":"2024-03-23T23:00:24.369860+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:46:46.058523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.281974+0000","last_change":"2024-03-23T22:59:27.203855+0000","last_active":"2024-03-23T23:00:53.281974+0000","last_peered":"2024-03-23T23:00:53.281974+0000","last_clean":"2024-03-23T23:00:53.281974+0000","last_became_active":"2024-03-23T22:59:27.203438+0000","last_became_peered":"2024-03-23T22:59:27.203438+0000","last_unstale":"2024-03-23T23:00:53.281974+0000","last_undegraded":"2024-03-23T23:00:53.281974+0000","last_fullsized":"2024-03-23T23:00:53.281974+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:28.725743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:01:40.004294+0000","last_change":"2024-03-23T22:59:26.439002+0000","last_active":"2024-03-23T23:01:40.004294+0000","last_peered":"2024-03-23T23:01:40.004294+0000","last_clean":"2024-03-23T23:01:40.004294+0000","last_became_active":"2024-03-23T22:59:26.438634+0000","last_became_peered":"2024-03-23T22:59:26.438634+0000","last_unstale":"2024-03-23T23:01:40.004294+0000","last_undegraded":"2024-03-23T23:01:40.004294+0000","last_fullsized":"2024-03-23T23:01:40.004294+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:25:02.988163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.295264+0000","last_change":"2024-03-23T22:59:27.217314+0000","last_active":"2024-03-23T23:00:53.295264+0000","last_peered":"2024-03-23T23:00:53.295264+0000","last_clean":"2024-03-23T23:00:53.295264+0000","last_became_active":"2024-03-23T22:59:27.216790+0000","last_became_peered":"2024-03-23T22:59:27.216790+0000","last_unstale":"2024-03-23T23:00:53.295264+0000","last_undegraded":"2024-03-23T23:00:53.295264+0000","last_fullsized":"2024-03-23T23:00:53.295264+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:47:13.902236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357159+0000","last_change":"2024-03-23T22:59:26.431453+0000","last_active":"2024-03-23T23:00:24.357159+0000","last_peered":"2024-03-23T23:00:24.357159+0000","last_clean":"2024-03-23T23:00:24.357159+0000","last_became_active":"2024-03-23T22:59:26.429731+0000","last_became_peered":"2024-03-23T22:59:26.429731+0000","last_unstale":"2024-03-23T23:00:24.357159+0000","last_undegraded":"2024-03-23T23:00:24.357159+0000","last_fullsized":"2024-03-23T23:00:24.357159+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:33:47.717064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.295443+0000","last_change":"2024-03-23T22:59:27.203097+0000","last_active":"2024-03-23T23:00:53.295443+0000","last_peered":"2024-03-23T23:00:53.295443+0000","last_clean":"2024-03-23T23:00:53.295443+0000","last_became_active":"2024-03-23T22:59:27.202484+0000","last_became_peered":"2024-03-23T22:59:27.202484+0000","last_unstale":"2024-03-23T23:00:53.295443+0000","last_undegraded":"2024-03-23T23:00:53.295443+0000","last_fullsized":"2024-03-23T23:00:53.295443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:07.829866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.377539+0000","last_change":"2024-03-23T23:00:24.377180+0000","last_active":"2024-03-23T23:00:24.377539+0000","last_peered":"2024-03-23T23:00:24.377539+0000","last_clean":"2024-03-23T23:00:24.377539+0000","last_became_active":"2024-03-23T23:00:24.364530+0000","last_became_peered":"2024-03-23T23:00:24.364530+0000","last_unstale":"2024-03-23T23:00:24.377539+0000","last_undegraded":"2024-03-23T23:00:24.377539+0000","last_fullsized":"2024-03-23T23:00:24.377539+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:57.507891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.286545+0000","last_change":"2024-03-23T22:59:27.189244+0000","last_active":"2024-03-23T23:00:53.286545+0000","last_peered":"2024-03-23T23:00:53.286545+0000","last_clean":"2024-03-23T23:00:53.286545+0000","last_became_active":"2024-03-23T22:59:27.188629+0000","last_became_peered":"2024-03-23T22:59:27.188629+0000","last_unstale":"2024-03-23T23:00:53.286545+0000","last_undegraded":"2024-03-23T23:00:53.286545+0000","last_fullsized":"2024-03-23T23:00:53.286545+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:47:26.742204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356493+0000","last_change":"2024-03-23T22:59:26.434326+0000","last_active":"2024-03-23T23:00:24.356493+0000","last_peered":"2024-03-23T23:00:24.356493+0000","last_clean":"2024-03-23T23:00:24.356493+0000","last_became_active":"2024-03-23T22:59:26.433878+0000","last_became_peered":"2024-03-23T22:59:26.433878+0000","last_unstale":"2024-03-23T23:00:24.356493+0000","last_undegraded":"2024-03-23T23:00:24.356493+0000","last_fullsized":"2024-03-23T23:00:24.356493+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:08.580572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.297139+0000","last_change":"2024-03-23T22:59:27.212909+0000","last_active":"2024-03-23T23:00:53.297139+0000","last_peered":"2024-03-23T23:00:53.297139+0000","last_clean":"2024-03-23T23:00:53.297139+0000","last_became_active":"2024-03-23T22:59:27.212504+0000","last_became_peered":"2024-03-23T22:59:27.212504+0000","last_unstale":"2024-03-23T23:00:53.297139+0000","last_undegraded":"2024-03-23T23:00:53.297139+0000","last_fullsized":"2024-03-23T23:00:53.297139+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:09.238272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.380290+0000","last_change":"2024-03-23T22:59:26.452577+0000","last_active":"2024-03-23T23:00:24.380290+0000","last_peered":"2024-03-23T23:00:24.380290+0000","last_clean":"2024-03-23T23:00:24.380290+0000","last_became_active":"2024-03-23T22:59:26.452107+0000","last_became_peered":"2024-03-23T22:59:26.452107+0000","last_unstale":"2024-03-23T23:00:24.380290+0000","last_undegraded":"2024-03-23T23:00:24.380290+0000","last_fullsized":"2024-03-23T23:00:24.380290+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:32:18.472334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.283849+0000","last_change":"2024-03-23T22:59:27.188539+0000","last_active":"2024-03-23T23:00:53.283849+0000","last_peered":"2024-03-23T23:00:53.283849+0000","last_clean":"2024-03-23T23:00:53.283849+0000","last_became_active":"2024-03-23T22:59:27.188184+0000","last_became_peered":"2024-03-23T22:59:27.188184+0000","last_unstale":"2024-03-23T23:00:53.283849+0000","last_undegraded":"2024-03-23T23:00:53.283849+0000","last_fullsized":"2024-03-23T23:00:53.283849+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:37:59.756492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.269022+0000","last_change":"2024-03-23T22:59:27.192083+0000","last_active":"2024-03-23T23:00:53.269022+0000","last_peered":"2024-03-23T23:00:53.269022+0000","last_clean":"2024-03-23T23:00:53.269022+0000","last_became_active":"2024-03-23T22:59:27.191709+0000","last_became_peered":"2024-03-23T22:59:27.191709+0000","last_unstale":"2024-03-23T23:00:53.269022+0000","last_undegraded":"2024-03-23T23:00:53.269022+0000","last_fullsized":"2024-03-23T23:00:53.269022+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:18:42.905177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.275138+0000","last_change":"2024-03-23T22:59:27.186315+0000","last_active":"2024-03-23T23:00:53.275138+0000","last_peered":"2024-03-23T23:00:53.275138+0000","last_clean":"2024-03-23T23:00:53.275138+0000","last_became_active":"2024-03-23T22:59:27.185960+0000","last_became_peered":"2024-03-23T22:59:27.185960+0000","last_unstale":"2024-03-23T23:00:53.275138+0000","last_undegraded":"2024-03-23T23:00:53.275138+0000","last_fullsized":"2024-03-23T23:00:53.275138+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:57:07.739725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.291985+0000","last_change":"2024-03-23T22:59:27.221696+0000","last_active":"2024-03-23T23:00:53.291985+0000","last_peered":"2024-03-23T23:00:53.291985+0000","last_clean":"2024-03-23T23:00:53.291985+0000","last_became_active":"2024-03-23T22:59:27.221400+0000","last_became_peered":"2024-03-23T22:59:27.221400+0000","last_unstale":"2024-03-23T23:00:53.291985+0000","last_undegraded":"2024-03-23T23:00:53.291985+0000","last_fullsized":"2024-03-23T23:00:53.291985+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:17.905737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.287320+0000","last_change":"2024-03-23T22:59:27.206286+0000","last_active":"2024-03-23T23:00:53.287320+0000","last_peered":"2024-03-23T23:00:53.287320+0000","last_clean":"2024-03-23T23:00:53.287320+0000","last_became_active":"2024-03-23T22:59:27.205854+0000","last_became_peered":"2024-03-23T22:59:27.205854+0000","last_unstale":"2024-03-23T23:00:53.287320+0000","last_undegraded":"2024-03-23T23:00:53.287320+0000","last_fullsized":"2024-03-23T23:00:53.287320+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:00:45.216211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.299730+0000","last_change":"2024-03-23T22:59:27.211376+0000","last_active":"2024-03-23T23:00:53.299730+0000","last_peered":"2024-03-23T23:00:53.299730+0000","last_clean":"2024-03-23T23:00:53.299730+0000","last_became_active":"2024-03-23T22:59:27.211061+0000","last_became_peered":"2024-03-23T22:59:27.211061+0000","last_unstale":"2024-03-23T23:00:53.299730+0000","last_undegraded":"2024-03-23T23:00:53.299730+0000","last_fullsized":"2024-03-23T23:00:53.299730+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:53:25.776927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.268390+0000","last_change":"2024-03-23T22:59:27.191028+0000","last_active":"2024-03-23T23:00:53.268390+0000","last_peered":"2024-03-23T23:00:53.268390+0000","last_clean":"2024-03-23T23:00:53.268390+0000","last_became_active":"2024-03-23T22:59:27.190634+0000","last_became_peered":"2024-03-23T22:59:27.190634+0000","last_unstale":"2024-03-23T23:00:53.268390+0000","last_undegraded":"2024-03-23T23:00:53.268390+0000","last_fullsized":"2024-03-23T23:00:53.268390+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:50:52.110636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.289464+0000","last_change":"2024-03-23T22:59:27.183235+0000","last_active":"2024-03-23T23:00:53.289464+0000","last_peered":"2024-03-23T23:00:53.289464+0000","last_clean":"2024-03-23T23:00:53.289464+0000","last_became_active":"2024-03-23T22:59:27.181682+0000","last_became_peered":"2024-03-23T22:59:27.181682+0000","last_unstale":"2024-03-23T23:00:53.289464+0000","last_undegraded":"2024-03-23T23:00:53.289464+0000","last_fullsized":"2024-03-23T23:00:53.289464+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:33:26.002324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.284323+0000","last_change":"2024-03-23T23:00:24.378143+0000","last_active":"2024-03-23T23:00:53.284323+0000","last_peered":"2024-03-23T23:00:53.284323+0000","last_clean":"2024-03-23T23:00:53.284323+0000","last_became_active":"2024-03-23T23:00:24.377660+0000","last_became_peered":"2024-03-23T23:00:24.377660+0000","last_unstale":"2024-03-23T23:00:53.284323+0000","last_undegraded":"2024-03-23T23:00:53.284323+0000","last_fullsized":"2024-03-23T23:00:53.284323+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:24:29.250679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.284520+0000","last_change":"2024-03-23T22:59:27.215230+0000","last_active":"2024-03-23T23:00:53.284520+0000","last_peered":"2024-03-23T23:00:53.284520+0000","last_clean":"2024-03-23T23:00:53.284520+0000","last_became_active":"2024-03-23T22:59:27.214881+0000","last_became_peered":"2024-03-23T22:59:27.214881+0000","last_unstale":"2024-03-23T23:00:53.284520+0000","last_undegraded":"2024-03-23T23:00:53.284520+0000","last_fullsized":"2024-03-23T23:00:53.284520+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:10:40.428481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.296915+0000","last_change":"2024-03-23T22:59:27.213454+0000","last_active":"2024-03-23T23:00:53.296915+0000","last_peered":"2024-03-23T23:00:53.296915+0000","last_clean":"2024-03-23T23:00:53.296915+0000","last_became_active":"2024-03-23T22:59:27.212684+0000","last_became_peered":"2024-03-23T22:59:27.212684+0000","last_unstale":"2024-03-23T23:00:53.296915+0000","last_undegraded":"2024-03-23T23:00:53.296915+0000","last_fullsized":"2024-03-23T23:00:53.296915+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:48:51.584892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.287158+0000","last_change":"2024-03-23T22:59:27.220414+0000","last_active":"2024-03-23T23:00:53.287158+0000","last_peered":"2024-03-23T23:00:53.287158+0000","last_clean":"2024-03-23T23:00:53.287158+0000","last_became_active":"2024-03-23T22:59:27.220020+0000","last_became_peered":"2024-03-23T22:59:27.220020+0000","last_unstale":"2024-03-23T23:00:53.287158+0000","last_undegraded":"2024-03-23T23:00:53.287158+0000","last_fullsized":"2024-03-23T23:00:53.287158+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:35:48.890004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.287204+0000","last_change":"2024-03-23T22:59:27.215648+0000","last_active":"2024-03-23T23:00:53.287204+0000","last_peered":"2024-03-23T23:00:53.287204+0000","last_clean":"2024-03-23T23:00:53.287204+0000","last_became_active":"2024-03-23T22:59:27.212252+0000","last_became_peered":"2024-03-23T22:59:27.212252+0000","last_unstale":"2024-03-23T23:00:53.287204+0000","last_undegraded":"2024-03-23T23:00:53.287204+0000","last_fullsized":"2024-03-23T23:00:53.287204+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:08:38.956508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.274679+0000","last_change":"2024-03-23T22:59:27.190527+0000","last_active":"2024-03-23T23:00:53.274679+0000","last_peered":"2024-03-23T23:00:53.274679+0000","last_clean":"2024-03-23T23:00:53.274679+0000","last_became_active":"2024-03-23T22:59:27.190026+0000","last_became_peered":"2024-03-23T22:59:27.190026+0000","last_unstale":"2024-03-23T23:00:53.274679+0000","last_undegraded":"2024-03-23T23:00:53.274679+0000","last_fullsized":"2024-03-23T23:00:53.274679+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:31:24.663584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247269,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68250,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":868,"num_read_kb":507,"num_write":170,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":196122,"data_compressed":5619,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351019,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":459288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612656,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":1115103,"data_compressed":37666,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776182,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":459524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037821,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":459444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299457,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":1051289,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462983,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":954368,"data_stored":458913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":519308,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":460184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411423,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":519800,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149785,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":459224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888146,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":459350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724621,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":1050973,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34152,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34152,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":60602,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":61094,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":65912,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T23:01:54.293 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T23:01:54.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.526+0000 7f4ef1800640 1 -- 172.21.15.12:0/317521857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4eec1008b0 msgr2=0x7f4eec100d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:54.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.526+0000 7f4ef1800640 1 --2- 172.21.15.12:0/317521857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4eec1008b0 0x7f4eec100d10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4ed4009860 tx=0x7f4ed402bb70 comp rx=0 tx=0).stop 2024-03-23T23:01:54.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.526+0000 7f4ef1800640 1 -- 172.21.15.12:0/317521857 shutdown_connections 2024-03-23T23:01:54.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.526+0000 7f4ef1800640 1 --2- 172.21.15.12:0/317521857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4eec101250 0x7f4eec1016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:54.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.526+0000 7f4ef1800640 1 --2- 172.21.15.12:0/317521857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4eec1008b0 0x7f4eec100d10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:54.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.526+0000 7f4ef1800640 1 --2- 172.21.15.12:0/317521857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4eec0fff50 0x7f4eec100370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:54.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.526+0000 7f4ef1800640 1 -- 172.21.15.12:0/317521857 >> 172.21.15.12:0/317521857 conn(0x7f4eec0fbb50 msgr2=0x7f4eec0fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:54.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.526+0000 7f4ef1800640 1 -- 172.21.15.12:0/317521857 shutdown_connections 2024-03-23T23:01:54.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.526+0000 7f4ef1800640 1 -- 172.21.15.12:0/317521857 wait complete. 2024-03-23T23:01:54.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.527+0000 7f4ef1800640 1 Processor -- start 2024-03-23T23:01:54.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.528+0000 7f4ef1800640 1 -- start start 2024-03-23T23:01:54.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.528+0000 7f4ef1800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4eec0fff50 0x7f4eec199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:54.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.528+0000 7f4ef1800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4eec1008b0 0x7f4eec19a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:54.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.528+0000 7f4ef1800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4eec101250 0x7f4eec1a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:54.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.528+0000 7f4ef1800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eec10f520 con 0x7f4eec1008b0 2024-03-23T23:01:54.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.528+0000 7f4ef1800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eec10f3a0 con 0x7f4eec0fff50 2024-03-23T23:01:54.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.528+0000 7f4ef1800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eec10f6a0 con 0x7f4eec101250 2024-03-23T23:01:54.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.528+0000 7f4eea000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4eec1008b0 0x7f4eec19a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:54.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.528+0000 7f4eeb400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4eec101250 0x7f4eec1a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:54.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.528+0000 7f4eeb400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4eec101250 0x7f4eec1a1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:37118/0 (socket says 172.21.15.12:37118) 2024-03-23T23:01:54.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.528+0000 7f4eea000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4eec1008b0 0x7f4eec19a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45252/0 (socket says 172.21.15.12:45252) 2024-03-23T23:01:54.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.528+0000 7f4eeb400640 1 -- 172.21.15.12:0/1039330027 learned_addr learned my addr 172.21.15.12:0/1039330027 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:54.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.528+0000 7f4eeaa00640 1 --2- 172.21.15.12:0/1039330027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4eec0fff50 0x7f4eec199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:54.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.529+0000 7f4eeb400640 1 -- 172.21.15.12:0/1039330027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4eec0fff50 msgr2=0x7f4eec199bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:54.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.529+0000 7f4eeb400640 1 --2- 172.21.15.12:0/1039330027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4eec0fff50 0x7f4eec199bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:54.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.529+0000 7f4eeb400640 1 -- 172.21.15.12:0/1039330027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4eec1008b0 msgr2=0x7f4eec19a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:54.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.529+0000 7f4eeb400640 1 --2- 172.21.15.12:0/1039330027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4eec1008b0 0x7f4eec19a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:54.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.529+0000 7f4eeb400640 1 -- 172.21.15.12:0/1039330027 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eec1a17f0 con 0x7f4eec101250 2024-03-23T23:01:54.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.529+0000 7f4eea000640 1 --2- 172.21.15.12:0/1039330027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4eec1008b0 0x7f4eec19a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:54.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.529+0000 7f4eeb400640 1 --2- 172.21.15.12:0/1039330027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4eec101250 0x7f4eec1a1180 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4ee400e940 tx=0x7f4ee400ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:54.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.529+0000 7f4ee3400640 1 -- 172.21.15.12:0/1039330027 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee400cd80 con 0x7f4eec101250 2024-03-23T23:01:54.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.529+0000 7f4ee3400640 1 -- 172.21.15.12:0/1039330027 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ee40035f0 con 0x7f4eec101250 2024-03-23T23:01:54.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.530+0000 7f4ef1800640 1 -- 172.21.15.12:0/1039330027 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eec1a1ae0 con 0x7f4eec101250 2024-03-23T23:01:54.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.530+0000 7f4ee3400640 1 -- 172.21.15.12:0/1039330027 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4010690 con 0x7f4eec101250 2024-03-23T23:01:54.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.530+0000 7f4ef1800640 1 -- 172.21.15.12:0/1039330027 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eec1a2000 con 0x7f4eec101250 2024-03-23T23:01:54.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.531+0000 7f4ef1800640 1 -- 172.21.15.12:0/1039330027 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eec10ae20 con 0x7f4eec101250 2024-03-23T23:01:54.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.533+0000 7f4ee3400640 1 -- 172.21.15.12:0/1039330027 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4ee40037a0 con 0x7f4eec101250 2024-03-23T23:01:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.534+0000 7f4ee3400640 1 --2- 172.21.15.12:0/1039330027 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4ed00a2880 0x7f4ed00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.534+0000 7f4ee3400640 1 -- 172.21.15.12:0/1039330027 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4ee40ced50 con 0x7f4eec101250 2024-03-23T23:01:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.535+0000 7f4eeaa00640 1 --2- 172.21.15.12:0/1039330027 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4ed00a2880 0x7f4ed00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:54.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.535+0000 7f4eeaa00640 1 --2- 172.21.15.12:0/1039330027 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4ed00a2880 0x7f4ed00a4d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4edc005fd0 tx=0x7f4edc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.538+0000 7f4ee3400640 1 -- 172.21.15.12:0/1039330027 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4ee409bda0 con 0x7f4eec101250 2024-03-23T23:01:54.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.742+0000 7f4ef1800640 1 -- 172.21.15.12:0/1039330027 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4eec061ae0 con 0x7f4ed00a2880 2024-03-23T23:01:54.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.764+0000 7f4ee3400640 1 -- 172.21.15.12:0/1039330027 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259036 (secure 0 0 0) 0x7f4eec061ae0 con 0x7f4ed00a2880 2024-03-23T23:01:54.768 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:01:54.769 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-23T23:01:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.769+0000 7f4ef1800640 1 -- 172.21.15.12:0/1039330027 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4ed00a2880 msgr2=0x7f4ed00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.769+0000 7f4ef1800640 1 --2- 172.21.15.12:0/1039330027 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4ed00a2880 0x7f4ed00a4d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4edc005fd0 tx=0x7f4edc009500 comp rx=0 tx=0).stop 2024-03-23T23:01:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.769+0000 7f4ef1800640 1 -- 172.21.15.12:0/1039330027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4eec101250 msgr2=0x7f4eec1a1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.769+0000 7f4ef1800640 1 --2- 172.21.15.12:0/1039330027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4eec101250 0x7f4eec1a1180 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4ee400e940 tx=0x7f4ee400ee00 comp rx=0 tx=0).stop 2024-03-23T23:01:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.770+0000 7f4ef1800640 1 -- 172.21.15.12:0/1039330027 shutdown_connections 2024-03-23T23:01:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.770+0000 7f4ef1800640 1 --2- 172.21.15.12:0/1039330027 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4ed00a2880 0x7f4ed00a4d40 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.770+0000 7f4ef1800640 1 --2- 172.21.15.12:0/1039330027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4eec101250 0x7f4eec1a1180 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.770+0000 7f4ef1800640 1 --2- 172.21.15.12:0/1039330027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4eec1008b0 0x7f4eec19a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.770+0000 7f4ef1800640 1 --2- 172.21.15.12:0/1039330027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4eec0fff50 0x7f4eec199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.770+0000 7f4ef1800640 1 -- 172.21.15.12:0/1039330027 >> 172.21.15.12:0/1039330027 conn(0x7f4eec0fbb50 msgr2=0x7f4eec0fd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.770+0000 7f4ef1800640 1 -- 172.21.15.12:0/1039330027 shutdown_connections 2024-03-23T23:01:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:54.770+0000 7f4ef1800640 1 -- 172.21.15.12:0/1039330027 wait complete. 2024-03-23T23:01:54.790 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":393,"stamp":"2024-03-23T23:01:53.318484+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658852,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":994,"num_read_kb":616,"num_write":353,"num_write_kb":3060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1181318,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":784,"ondisk_log_size":784,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369128,"kb_used_data":11656,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572696,"statfs":{"total":1151940427776,"available":1151562440704,"internally_reserved":0,"allocated":11935744,"data_stored":7472400,"data_compressed":112998,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4926,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":78,"num_read_kb":47,"num_write":9,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005487"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.285759+0000","last_change":"2024-03-23T22:59:27.200234+0000","last_active":"2024-03-23T23:00:53.285759+0000","last_peered":"2024-03-23T23:00:53.285759+0000","last_clean":"2024-03-23T23:00:53.285759+0000","last_became_active":"2024-03-23T22:59:27.199864+0000","last_became_peered":"2024-03-23T22:59:27.199864+0000","last_unstale":"2024-03-23T23:00:53.285759+0000","last_undegraded":"2024-03-23T23:00:53.285759+0000","last_fullsized":"2024-03-23T23:00:53.285759+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:56:26.811964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.300077+0000","last_change":"2024-03-23T22:59:27.195101+0000","last_active":"2024-03-23T23:00:53.300077+0000","last_peered":"2024-03-23T23:00:53.300077+0000","last_clean":"2024-03-23T23:00:53.300077+0000","last_became_active":"2024-03-23T22:59:27.194650+0000","last_became_peered":"2024-03-23T22:59:27.194650+0000","last_unstale":"2024-03-23T23:00:53.300077+0000","last_undegraded":"2024-03-23T23:00:53.300077+0000","last_fullsized":"2024-03-23T23:00:53.300077+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:20.396811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.286097+0000","last_change":"2024-03-23T22:59:27.185575+0000","last_active":"2024-03-23T23:00:53.286097+0000","last_peered":"2024-03-23T23:00:53.286097+0000","last_clean":"2024-03-23T23:00:53.286097+0000","last_became_active":"2024-03-23T22:59:27.184328+0000","last_became_peered":"2024-03-23T22:59:27.184328+0000","last_unstale":"2024-03-23T23:00:53.286097+0000","last_undegraded":"2024-03-23T23:00:53.286097+0000","last_fullsized":"2024-03-23T23:00:53.286097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:31:46.221574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.270875+0000","last_change":"2024-03-23T22:59:27.184434+0000","last_active":"2024-03-23T23:00:53.270875+0000","last_peered":"2024-03-23T23:00:53.270875+0000","last_clean":"2024-03-23T23:00:53.270875+0000","last_became_active":"2024-03-23T22:59:27.183970+0000","last_became_peered":"2024-03-23T22:59:27.183970+0000","last_unstale":"2024-03-23T23:00:53.270875+0000","last_undegraded":"2024-03-23T23:00:53.270875+0000","last_fullsized":"2024-03-23T23:00:53.270875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:39:05.517976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.266775+0000","last_change":"2024-03-23T22:59:27.214741+0000","last_active":"2024-03-23T23:00:53.266775+0000","last_peered":"2024-03-23T23:00:53.266775+0000","last_clean":"2024-03-23T23:00:53.266775+0000","last_became_active":"2024-03-23T22:59:27.214311+0000","last_became_peered":"2024-03-23T22:59:27.214311+0000","last_unstale":"2024-03-23T23:00:53.266775+0000","last_undegraded":"2024-03-23T23:00:53.266775+0000","last_fullsized":"2024-03-23T23:00:53.266775+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:35:38.464955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.291377+0000","last_change":"2024-03-23T22:59:27.200695+0000","last_active":"2024-03-23T23:00:53.291377+0000","last_peered":"2024-03-23T23:00:53.291377+0000","last_clean":"2024-03-23T23:00:53.291377+0000","last_became_active":"2024-03-23T22:59:27.200159+0000","last_became_peered":"2024-03-23T22:59:27.200159+0000","last_unstale":"2024-03-23T23:00:53.291377+0000","last_undegraded":"2024-03-23T23:00:53.291377+0000","last_fullsized":"2024-03-23T23:00:53.291377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:01:15.147856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.280147+0000","last_change":"2024-03-23T22:59:27.199928+0000","last_active":"2024-03-23T23:00:53.280147+0000","last_peered":"2024-03-23T23:00:53.280147+0000","last_clean":"2024-03-23T23:00:53.280147+0000","last_became_active":"2024-03-23T22:59:27.199507+0000","last_became_peered":"2024-03-23T22:59:27.199507+0000","last_unstale":"2024-03-23T23:00:53.280147+0000","last_undegraded":"2024-03-23T23:00:53.280147+0000","last_fullsized":"2024-03-23T23:00:53.280147+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:51:19.145785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.295760+0000","last_change":"2024-03-23T22:59:27.217832+0000","last_active":"2024-03-23T23:00:53.295760+0000","last_peered":"2024-03-23T23:00:53.295760+0000","last_clean":"2024-03-23T23:00:53.295760+0000","last_became_active":"2024-03-23T22:59:27.217496+0000","last_became_peered":"2024-03-23T22:59:27.217496+0000","last_unstale":"2024-03-23T23:00:53.295760+0000","last_undegraded":"2024-03-23T23:00:53.295760+0000","last_fullsized":"2024-03-23T23:00:53.295760+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:01.778627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.288651+0000","last_change":"2024-03-23T22:59:27.190982+0000","last_active":"2024-03-23T23:00:53.288651+0000","last_peered":"2024-03-23T23:00:53.288651+0000","last_clean":"2024-03-23T23:00:53.288651+0000","last_became_active":"2024-03-23T22:59:27.190661+0000","last_became_peered":"2024-03-23T22:59:27.190661+0000","last_unstale":"2024-03-23T23:00:53.288651+0000","last_undegraded":"2024-03-23T23:00:53.288651+0000","last_fullsized":"2024-03-23T23:00:53.288651+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:29.822253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.281328+0000","last_change":"2024-03-23T22:59:27.187395+0000","last_active":"2024-03-23T23:00:53.281328+0000","last_peered":"2024-03-23T23:00:53.281328+0000","last_clean":"2024-03-23T23:00:53.281328+0000","last_became_active":"2024-03-23T22:59:27.186849+0000","last_became_peered":"2024-03-23T22:59:27.186849+0000","last_unstale":"2024-03-23T23:00:53.281328+0000","last_undegraded":"2024-03-23T23:00:53.281328+0000","last_fullsized":"2024-03-23T23:00:53.281328+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:05:48.047102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.286208+0000","last_change":"2024-03-23T22:59:27.190023+0000","last_active":"2024-03-23T23:00:53.286208+0000","last_peered":"2024-03-23T23:00:53.286208+0000","last_clean":"2024-03-23T23:00:53.286208+0000","last_became_active":"2024-03-23T22:59:27.189697+0000","last_became_peered":"2024-03-23T22:59:27.189697+0000","last_unstale":"2024-03-23T23:00:53.286208+0000","last_undegraded":"2024-03-23T23:00:53.286208+0000","last_fullsized":"2024-03-23T23:00:53.286208+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:59:51.121055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.264109+0000","last_change":"2024-03-23T22:59:27.187382+0000","last_active":"2024-03-23T23:00:53.264109+0000","last_peered":"2024-03-23T23:00:53.264109+0000","last_clean":"2024-03-23T23:00:53.264109+0000","last_became_active":"2024-03-23T22:59:27.186931+0000","last_became_peered":"2024-03-23T22:59:27.186931+0000","last_unstale":"2024-03-23T23:00:53.264109+0000","last_undegraded":"2024-03-23T23:00:53.264109+0000","last_fullsized":"2024-03-23T23:00:53.264109+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:33.583123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.293942+0000","last_change":"2024-03-23T22:59:27.202577+0000","last_active":"2024-03-23T23:00:53.293942+0000","last_peered":"2024-03-23T23:00:53.293942+0000","last_clean":"2024-03-23T23:00:53.293942+0000","last_became_active":"2024-03-23T22:59:27.202122+0000","last_became_peered":"2024-03-23T22:59:27.202122+0000","last_unstale":"2024-03-23T23:00:53.293942+0000","last_undegraded":"2024-03-23T23:00:53.293942+0000","last_fullsized":"2024-03-23T23:00:53.293942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:01:10.809635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.298452+0000","last_change":"2024-03-23T22:59:27.192915+0000","last_active":"2024-03-23T23:00:53.298452+0000","last_peered":"2024-03-23T23:00:53.298452+0000","last_clean":"2024-03-23T23:00:53.298452+0000","last_became_active":"2024-03-23T22:59:27.192107+0000","last_became_peered":"2024-03-23T22:59:27.192107+0000","last_unstale":"2024-03-23T23:00:53.298452+0000","last_undegraded":"2024-03-23T23:00:53.298452+0000","last_fullsized":"2024-03-23T23:00:53.298452+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:21:19.169211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.300473+0000","last_change":"2024-03-23T22:59:27.192156+0000","last_active":"2024-03-23T23:00:53.300473+0000","last_peered":"2024-03-23T23:00:53.300473+0000","last_clean":"2024-03-23T23:00:53.300473+0000","last_became_active":"2024-03-23T22:59:27.191594+0000","last_became_peered":"2024-03-23T22:59:27.191594+0000","last_unstale":"2024-03-23T23:00:53.300473+0000","last_undegraded":"2024-03-23T23:00:53.300473+0000","last_fullsized":"2024-03-23T23:00:53.300473+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:00:45.480281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.299835+0000","last_change":"2024-03-23T22:59:27.186048+0000","last_active":"2024-03-23T23:00:53.299835+0000","last_peered":"2024-03-23T23:00:53.299835+0000","last_clean":"2024-03-23T23:00:53.299835+0000","last_became_active":"2024-03-23T22:59:27.185710+0000","last_became_peered":"2024-03-23T22:59:27.185710+0000","last_unstale":"2024-03-23T23:00:53.299835+0000","last_undegraded":"2024-03-23T23:00:53.299835+0000","last_fullsized":"2024-03-23T23:00:53.299835+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:06.792665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.298495+0000","last_change":"2024-03-23T22:59:27.206290+0000","last_active":"2024-03-23T23:00:53.298495+0000","last_peered":"2024-03-23T23:00:53.298495+0000","last_clean":"2024-03-23T23:00:53.298495+0000","last_became_active":"2024-03-23T22:59:27.205997+0000","last_became_peered":"2024-03-23T22:59:27.205997+0000","last_unstale":"2024-03-23T23:00:53.298495+0000","last_undegraded":"2024-03-23T23:00:53.298495+0000","last_fullsized":"2024-03-23T23:00:53.298495+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:38:42.006166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.289029+0000","last_change":"2024-03-23T22:59:27.203671+0000","last_active":"2024-03-23T23:00:53.289029+0000","last_peered":"2024-03-23T23:00:53.289029+0000","last_clean":"2024-03-23T23:00:53.289029+0000","last_became_active":"2024-03-23T22:59:27.203273+0000","last_became_peered":"2024-03-23T22:59:27.203273+0000","last_unstale":"2024-03-23T23:00:53.289029+0000","last_undegraded":"2024-03-23T23:00:53.289029+0000","last_fullsized":"2024-03-23T23:00:53.289029+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:48:16.018671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.292388+0000","last_change":"2024-03-23T22:59:27.210063+0000","last_active":"2024-03-23T23:00:53.292388+0000","last_peered":"2024-03-23T23:00:53.292388+0000","last_clean":"2024-03-23T23:00:53.292388+0000","last_became_active":"2024-03-23T22:59:27.209698+0000","last_became_peered":"2024-03-23T22:59:27.209698+0000","last_unstale":"2024-03-23T23:00:53.292388+0000","last_undegraded":"2024-03-23T23:00:53.292388+0000","last_fullsized":"2024-03-23T23:00:53.292388+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:19:01.408901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.358191+0000","last_change":"2024-03-23T22:59:26.445053+0000","last_active":"2024-03-23T23:00:24.358191+0000","last_peered":"2024-03-23T23:00:24.358191+0000","last_clean":"2024-03-23T23:00:24.358191+0000","last_became_active":"2024-03-23T22:59:26.444660+0000","last_became_peered":"2024-03-23T22:59:26.444660+0000","last_unstale":"2024-03-23T23:00:24.358191+0000","last_undegraded":"2024-03-23T23:00:24.358191+0000","last_fullsized":"2024-03-23T23:00:24.358191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:19:16.958228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.284909+0000","last_change":"2024-03-23T22:59:27.208652+0000","last_active":"2024-03-23T23:00:53.284909+0000","last_peered":"2024-03-23T23:00:53.284909+0000","last_clean":"2024-03-23T23:00:53.284909+0000","last_became_active":"2024-03-23T22:59:27.207724+0000","last_became_peered":"2024-03-23T22:59:27.207724+0000","last_unstale":"2024-03-23T23:00:53.284909+0000","last_undegraded":"2024-03-23T23:00:53.284909+0000","last_fullsized":"2024-03-23T23:00:53.284909+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:08.986439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.368748+0000","last_change":"2024-03-23T22:59:26.445371+0000","last_active":"2024-03-23T23:00:24.368748+0000","last_peered":"2024-03-23T23:00:24.368748+0000","last_clean":"2024-03-23T23:00:24.368748+0000","last_became_active":"2024-03-23T22:59:26.445016+0000","last_became_peered":"2024-03-23T22:59:26.445016+0000","last_unstale":"2024-03-23T23:00:24.368748+0000","last_undegraded":"2024-03-23T23:00:24.368748+0000","last_fullsized":"2024-03-23T23:00:24.368748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:31:44.003218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.300501+0000","last_change":"2024-03-23T22:59:27.200624+0000","last_active":"2024-03-23T23:00:53.300501+0000","last_peered":"2024-03-23T23:00:53.300501+0000","last_clean":"2024-03-23T23:00:53.300501+0000","last_became_active":"2024-03-23T22:59:27.200207+0000","last_became_peered":"2024-03-23T22:59:27.200207+0000","last_unstale":"2024-03-23T23:00:53.300501+0000","last_undegraded":"2024-03-23T23:00:53.300501+0000","last_fullsized":"2024-03-23T23:00:53.300501+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:05:31.015869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.380616+0000","last_change":"2024-03-23T23:00:24.380296+0000","last_active":"2024-03-23T23:00:24.380616+0000","last_peered":"2024-03-23T23:00:24.380616+0000","last_clean":"2024-03-23T23:00:24.380616+0000","last_became_active":"2024-03-23T23:00:24.374344+0000","last_became_peered":"2024-03-23T23:00:24.374344+0000","last_unstale":"2024-03-23T23:00:24.380616+0000","last_undegraded":"2024-03-23T23:00:24.380616+0000","last_fullsized":"2024-03-23T23:00:24.380616+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:30:20.174783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.288535+0000","last_change":"2024-03-23T22:59:27.204157+0000","last_active":"2024-03-23T23:00:53.288535+0000","last_peered":"2024-03-23T23:00:53.288535+0000","last_clean":"2024-03-23T23:00:53.288535+0000","last_became_active":"2024-03-23T22:59:27.203559+0000","last_became_peered":"2024-03-23T22:59:27.203559+0000","last_unstale":"2024-03-23T23:00:53.288535+0000","last_undegraded":"2024-03-23T23:00:53.288535+0000","last_fullsized":"2024-03-23T23:00:53.288535+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:43.834369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.361574+0000","last_change":"2024-03-23T23:00:24.361233+0000","last_active":"2024-03-23T23:00:24.361574+0000","last_peered":"2024-03-23T23:00:24.361574+0000","last_clean":"2024-03-23T23:00:24.361574+0000","last_became_active":"2024-03-23T23:00:24.358048+0000","last_became_peered":"2024-03-23T23:00:24.358048+0000","last_unstale":"2024-03-23T23:00:24.361574+0000","last_undegraded":"2024-03-23T23:00:24.361574+0000","last_fullsized":"2024-03-23T23:00:24.361574+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:38:37.792273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.286722+0000","last_change":"2024-03-23T22:59:27.186599+0000","last_active":"2024-03-23T23:00:53.286722+0000","last_peered":"2024-03-23T23:00:53.286722+0000","last_clean":"2024-03-23T23:00:53.286722+0000","last_became_active":"2024-03-23T22:59:27.186262+0000","last_became_peered":"2024-03-23T22:59:27.186262+0000","last_unstale":"2024-03-23T23:00:53.286722+0000","last_undegraded":"2024-03-23T23:00:53.286722+0000","last_fullsized":"2024-03-23T23:00:53.286722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:11:39.545558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":166,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:01:49.342977+0000","last_change":"2024-03-23T22:59:26.430096+0000","last_active":"2024-03-23T23:01:49.342977+0000","last_peered":"2024-03-23T23:01:49.342977+0000","last_clean":"2024-03-23T23:01:49.342977+0000","last_became_active":"2024-03-23T22:59:26.429740+0000","last_became_peered":"2024-03-23T22:59:26.429740+0000","last_unstale":"2024-03-23T23:01:49.342977+0000","last_undegraded":"2024-03-23T23:01:49.342977+0000","last_fullsized":"2024-03-23T23:01:49.342977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:54.831689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":928,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":141,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.262380+0000","last_change":"2024-03-23T22:59:27.202750+0000","last_active":"2024-03-23T23:00:53.262380+0000","last_peered":"2024-03-23T23:00:53.262380+0000","last_clean":"2024-03-23T23:00:53.262380+0000","last_became_active":"2024-03-23T22:59:27.202391+0000","last_became_peered":"2024-03-23T22:59:27.202391+0000","last_unstale":"2024-03-23T23:00:53.262380+0000","last_undegraded":"2024-03-23T23:00:53.262380+0000","last_fullsized":"2024-03-23T23:00:53.262380+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:27:18.370792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.365876+0000","last_change":"2024-03-23T22:59:26.425607+0000","last_active":"2024-03-23T23:00:24.365876+0000","last_peered":"2024-03-23T23:00:24.365876+0000","last_clean":"2024-03-23T23:00:24.365876+0000","last_became_active":"2024-03-23T22:59:26.425208+0000","last_became_peered":"2024-03-23T22:59:26.425208+0000","last_unstale":"2024-03-23T23:00:24.365876+0000","last_undegraded":"2024-03-23T23:00:24.365876+0000","last_fullsized":"2024-03-23T23:00:24.365876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:51.071662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.282112+0000","last_change":"2024-03-23T22:59:27.215637+0000","last_active":"2024-03-23T23:00:53.282112+0000","last_peered":"2024-03-23T23:00:53.282112+0000","last_clean":"2024-03-23T23:00:53.282112+0000","last_became_active":"2024-03-23T22:59:27.215339+0000","last_became_peered":"2024-03-23T22:59:27.215339+0000","last_unstale":"2024-03-23T23:00:53.282112+0000","last_undegraded":"2024-03-23T23:00:53.282112+0000","last_fullsized":"2024-03-23T23:00:53.282112+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:25:42.097326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.377892+0000","last_change":"2024-03-23T22:59:26.442023+0000","last_active":"2024-03-23T23:00:24.377892+0000","last_peered":"2024-03-23T23:00:24.377892+0000","last_clean":"2024-03-23T23:00:24.377892+0000","last_became_active":"2024-03-23T22:59:26.441661+0000","last_became_peered":"2024-03-23T22:59:26.441661+0000","last_unstale":"2024-03-23T23:00:24.377892+0000","last_undegraded":"2024-03-23T23:00:24.377892+0000","last_fullsized":"2024-03-23T23:00:24.377892+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:59:14.914860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.279352+0000","last_change":"2024-03-23T23:00:24.384190+0000","last_active":"2024-03-23T23:00:53.279352+0000","last_peered":"2024-03-23T23:00:53.279352+0000","last_clean":"2024-03-23T23:00:53.279352+0000","last_became_active":"2024-03-23T23:00:24.382141+0000","last_became_peered":"2024-03-23T23:00:24.382141+0000","last_unstale":"2024-03-23T23:00:53.279352+0000","last_undegraded":"2024-03-23T23:00:53.279352+0000","last_fullsized":"2024-03-23T23:00:53.279352+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:48:52.311407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356353+0000","last_change":"2024-03-23T22:59:26.430147+0000","last_active":"2024-03-23T23:00:24.356353+0000","last_peered":"2024-03-23T23:00:24.356353+0000","last_clean":"2024-03-23T23:00:24.356353+0000","last_became_active":"2024-03-23T22:59:26.429843+0000","last_became_peered":"2024-03-23T22:59:26.429843+0000","last_unstale":"2024-03-23T23:00:24.356353+0000","last_undegraded":"2024-03-23T23:00:24.356353+0000","last_fullsized":"2024-03-23T23:00:24.356353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:06:45.691894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.288385+0000","last_change":"2024-03-23T22:59:27.212235+0000","last_active":"2024-03-23T23:00:53.288385+0000","last_peered":"2024-03-23T23:00:53.288385+0000","last_clean":"2024-03-23T23:00:53.288385+0000","last_became_active":"2024-03-23T22:59:27.211919+0000","last_became_peered":"2024-03-23T22:59:27.211919+0000","last_unstale":"2024-03-23T23:00:53.288385+0000","last_undegraded":"2024-03-23T23:00:53.288385+0000","last_fullsized":"2024-03-23T23:00:53.288385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:24.752109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":165,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:01:52.346610+0000","last_change":"2024-03-23T22:59:26.440623+0000","last_active":"2024-03-23T23:01:52.346610+0000","last_peered":"2024-03-23T23:01:52.346610+0000","last_clean":"2024-03-23T23:01:52.346610+0000","last_became_active":"2024-03-23T22:59:26.440204+0000","last_became_peered":"2024-03-23T22:59:26.440204+0000","last_unstale":"2024-03-23T23:01:52.346610+0000","last_undegraded":"2024-03-23T23:01:52.346610+0000","last_fullsized":"2024-03-23T23:01:52.346610+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:58:44.088552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":141,"num_read_kb":141,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.287377+0000","last_change":"2024-03-23T22:59:27.203916+0000","last_active":"2024-03-23T23:00:53.287377+0000","last_peered":"2024-03-23T23:00:53.287377+0000","last_clean":"2024-03-23T23:00:53.287377+0000","last_became_active":"2024-03-23T22:59:27.203565+0000","last_became_peered":"2024-03-23T22:59:27.203565+0000","last_unstale":"2024-03-23T23:00:53.287377+0000","last_undegraded":"2024-03-23T23:00:53.287377+0000","last_fullsized":"2024-03-23T23:00:53.287377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:06:39.635025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357614+0000","last_change":"2024-03-23T22:59:26.430080+0000","last_active":"2024-03-23T23:00:24.357614+0000","last_peered":"2024-03-23T23:00:24.357614+0000","last_clean":"2024-03-23T23:00:24.357614+0000","last_became_active":"2024-03-23T22:59:26.429618+0000","last_became_peered":"2024-03-23T22:59:26.429618+0000","last_unstale":"2024-03-23T23:00:24.357614+0000","last_undegraded":"2024-03-23T23:00:24.357614+0000","last_fullsized":"2024-03-23T23:00:24.357614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:01.963573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.300287+0000","last_change":"2024-03-23T22:59:27.190505+0000","last_active":"2024-03-23T23:00:53.300287+0000","last_peered":"2024-03-23T23:00:53.300287+0000","last_clean":"2024-03-23T23:00:53.300287+0000","last_became_active":"2024-03-23T22:59:27.190088+0000","last_became_peered":"2024-03-23T22:59:27.190088+0000","last_unstale":"2024-03-23T23:00:53.300287+0000","last_undegraded":"2024-03-23T23:00:53.300287+0000","last_fullsized":"2024-03-23T23:00:53.300287+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:29:04.255832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357315+0000","last_change":"2024-03-23T22:59:26.438329+0000","last_active":"2024-03-23T23:00:24.357315+0000","last_peered":"2024-03-23T23:00:24.357315+0000","last_clean":"2024-03-23T23:00:24.357315+0000","last_became_active":"2024-03-23T22:59:26.438033+0000","last_became_peered":"2024-03-23T22:59:26.438033+0000","last_unstale":"2024-03-23T23:00:24.357315+0000","last_undegraded":"2024-03-23T23:00:24.357315+0000","last_fullsized":"2024-03-23T23:00:24.357315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:02:38.789574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.287183+0000","last_change":"2024-03-23T22:59:27.188417+0000","last_active":"2024-03-23T23:00:53.287183+0000","last_peered":"2024-03-23T23:00:53.287183+0000","last_clean":"2024-03-23T23:00:53.287183+0000","last_became_active":"2024-03-23T22:59:27.188023+0000","last_became_peered":"2024-03-23T22:59:27.188023+0000","last_unstale":"2024-03-23T23:00:53.287183+0000","last_undegraded":"2024-03-23T23:00:53.287183+0000","last_fullsized":"2024-03-23T23:00:53.287183+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:19:11.014604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'33","reported_seq":347,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:01:52.347319+0000","last_change":"2024-03-23T22:59:26.425963+0000","last_active":"2024-03-23T23:01:52.347319+0000","last_peered":"2024-03-23T23:01:52.347319+0000","last_clean":"2024-03-23T23:01:52.347319+0000","last_became_active":"2024-03-23T22:59:26.425640+0000","last_became_peered":"2024-03-23T22:59:26.425640+0000","last_unstale":"2024-03-23T23:01:52.347319+0000","last_undegraded":"2024-03-23T23:01:52.347319+0000","last_fullsized":"2024-03-23T23:01:52.347319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:11:26.529034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64334,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":294,"num_read_kb":210,"num_write":34,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.256447+0000","last_change":"2024-03-23T22:59:27.187254+0000","last_active":"2024-03-23T23:00:53.256447+0000","last_peered":"2024-03-23T23:00:53.256447+0000","last_clean":"2024-03-23T23:00:53.256447+0000","last_became_active":"2024-03-23T22:59:27.186849+0000","last_became_peered":"2024-03-23T22:59:27.186849+0000","last_unstale":"2024-03-23T23:00:53.256447+0000","last_undegraded":"2024-03-23T23:00:53.256447+0000","last_fullsized":"2024-03-23T23:00:53.256447+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:35:30.958565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.358477+0000","last_change":"2024-03-23T22:59:26.429767+0000","last_active":"2024-03-23T23:00:24.358477+0000","last_peered":"2024-03-23T23:00:24.358477+0000","last_clean":"2024-03-23T23:00:24.358477+0000","last_became_active":"2024-03-23T22:59:26.429449+0000","last_became_peered":"2024-03-23T22:59:26.429449+0000","last_unstale":"2024-03-23T23:00:24.358477+0000","last_undegraded":"2024-03-23T23:00:24.358477+0000","last_fullsized":"2024-03-23T23:00:24.358477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:46:13.752427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.298386+0000","last_change":"2024-03-23T22:59:27.223544+0000","last_active":"2024-03-23T23:00:53.298386+0000","last_peered":"2024-03-23T23:00:53.298386+0000","last_clean":"2024-03-23T23:00:53.298386+0000","last_became_active":"2024-03-23T22:59:27.223177+0000","last_became_peered":"2024-03-23T22:59:27.223177+0000","last_unstale":"2024-03-23T23:00:53.298386+0000","last_undegraded":"2024-03-23T23:00:53.298386+0000","last_fullsized":"2024-03-23T23:00:53.298386+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:33:18.521579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356567+0000","last_change":"2024-03-23T22:59:26.427781+0000","last_active":"2024-03-23T23:00:24.356567+0000","last_peered":"2024-03-23T23:00:24.356567+0000","last_clean":"2024-03-23T23:00:24.356567+0000","last_became_active":"2024-03-23T22:59:26.427458+0000","last_became_peered":"2024-03-23T22:59:26.427458+0000","last_unstale":"2024-03-23T23:00:24.356567+0000","last_undegraded":"2024-03-23T23:00:24.356567+0000","last_fullsized":"2024-03-23T23:00:24.356567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:50:33.956611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.295652+0000","last_change":"2024-03-23T22:59:27.203681+0000","last_active":"2024-03-23T23:00:53.295652+0000","last_peered":"2024-03-23T23:00:53.295652+0000","last_clean":"2024-03-23T23:00:53.295652+0000","last_became_active":"2024-03-23T22:59:27.202457+0000","last_became_peered":"2024-03-23T22:59:27.202457+0000","last_unstale":"2024-03-23T23:00:53.295652+0000","last_undegraded":"2024-03-23T23:00:53.295652+0000","last_fullsized":"2024-03-23T23:00:53.295652+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:55:46.725350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'3","reported_seq":164,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:01:49.431259+0000","last_change":"2024-03-23T22:59:26.428517+0000","last_active":"2024-03-23T23:01:49.431259+0000","last_peered":"2024-03-23T23:01:49.431259+0000","last_clean":"2024-03-23T23:01:49.431259+0000","last_became_active":"2024-03-23T22:59:26.428130+0000","last_became_peered":"2024-03-23T22:59:26.428130+0000","last_unstale":"2024-03-23T23:01:49.431259+0000","last_undegraded":"2024-03-23T23:01:49.431259+0000","last_fullsized":"2024-03-23T23:01:49.431259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:01.185776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":142,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.283053+0000","last_change":"2024-03-23T23:00:24.367049+0000","last_active":"2024-03-23T23:00:53.283053+0000","last_peered":"2024-03-23T23:00:53.283053+0000","last_clean":"2024-03-23T23:00:53.283053+0000","last_became_active":"2024-03-23T23:00:24.366636+0000","last_became_peered":"2024-03-23T23:00:24.366636+0000","last_unstale":"2024-03-23T23:00:53.283053+0000","last_undegraded":"2024-03-23T23:00:53.283053+0000","last_fullsized":"2024-03-23T23:00:53.283053+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:07.068500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356958+0000","last_change":"2024-03-23T22:59:26.442637+0000","last_active":"2024-03-23T23:00:24.356958+0000","last_peered":"2024-03-23T23:00:24.356958+0000","last_clean":"2024-03-23T23:00:24.356958+0000","last_became_active":"2024-03-23T22:59:26.442267+0000","last_became_peered":"2024-03-23T22:59:26.442267+0000","last_unstale":"2024-03-23T23:00:24.356958+0000","last_undegraded":"2024-03-23T23:00:24.356958+0000","last_fullsized":"2024-03-23T23:00:24.356958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:49.975048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.286407+0000","last_change":"2024-03-23T22:59:27.187812+0000","last_active":"2024-03-23T23:00:53.286407+0000","last_peered":"2024-03-23T23:00:53.286407+0000","last_clean":"2024-03-23T23:00:53.286407+0000","last_became_active":"2024-03-23T22:59:27.187423+0000","last_became_peered":"2024-03-23T22:59:27.187423+0000","last_unstale":"2024-03-23T23:00:53.286407+0000","last_undegraded":"2024-03-23T23:00:53.286407+0000","last_fullsized":"2024-03-23T23:00:53.286407+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:12:16.427695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357353+0000","last_change":"2024-03-23T22:59:26.423875+0000","last_active":"2024-03-23T23:00:24.357353+0000","last_peered":"2024-03-23T23:00:24.357353+0000","last_clean":"2024-03-23T23:00:24.357353+0000","last_became_active":"2024-03-23T22:59:26.423513+0000","last_became_peered":"2024-03-23T22:59:26.423513+0000","last_unstale":"2024-03-23T23:00:24.357353+0000","last_undegraded":"2024-03-23T23:00:24.357353+0000","last_fullsized":"2024-03-23T23:00:24.357353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:43.173365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.268696+0000","last_change":"2024-03-23T22:59:27.191454+0000","last_active":"2024-03-23T23:00:53.268696+0000","last_peered":"2024-03-23T23:00:53.268696+0000","last_clean":"2024-03-23T23:00:53.268696+0000","last_became_active":"2024-03-23T22:59:27.190927+0000","last_became_peered":"2024-03-23T22:59:27.190927+0000","last_unstale":"2024-03-23T23:00:53.268696+0000","last_undegraded":"2024-03-23T23:00:53.268696+0000","last_fullsized":"2024-03-23T23:00:53.268696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:00.333702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.358997+0000","last_change":"2024-03-23T22:59:26.443238+0000","last_active":"2024-03-23T23:00:24.358997+0000","last_peered":"2024-03-23T23:00:24.358997+0000","last_clean":"2024-03-23T23:00:24.358997+0000","last_became_active":"2024-03-23T22:59:26.442834+0000","last_became_peered":"2024-03-23T22:59:26.442834+0000","last_unstale":"2024-03-23T23:00:24.358997+0000","last_undegraded":"2024-03-23T23:00:24.358997+0000","last_fullsized":"2024-03-23T23:00:24.358997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:11:21.072836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.295551+0000","last_change":"2024-03-23T22:59:27.212864+0000","last_active":"2024-03-23T23:00:53.295551+0000","last_peered":"2024-03-23T23:00:53.295551+0000","last_clean":"2024-03-23T23:00:53.295551+0000","last_became_active":"2024-03-23T22:59:27.212455+0000","last_became_peered":"2024-03-23T22:59:27.212455+0000","last_unstale":"2024-03-23T23:00:53.295551+0000","last_undegraded":"2024-03-23T23:00:53.295551+0000","last_fullsized":"2024-03-23T23:00:53.295551+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:25:59.122531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357120+0000","last_change":"2024-03-23T22:59:26.432838+0000","last_active":"2024-03-23T23:00:24.357120+0000","last_peered":"2024-03-23T23:00:24.357120+0000","last_clean":"2024-03-23T23:00:24.357120+0000","last_became_active":"2024-03-23T22:59:26.432544+0000","last_became_peered":"2024-03-23T22:59:26.432544+0000","last_unstale":"2024-03-23T23:00:24.357120+0000","last_undegraded":"2024-03-23T23:00:24.357120+0000","last_fullsized":"2024-03-23T23:00:24.357120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:32:52.121569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.290395+0000","last_change":"2024-03-23T23:00:24.376926+0000","last_active":"2024-03-23T23:00:53.290395+0000","last_peered":"2024-03-23T23:00:53.290395+0000","last_clean":"2024-03-23T23:00:53.290395+0000","last_became_active":"2024-03-23T23:00:24.376515+0000","last_became_peered":"2024-03-23T23:00:24.376515+0000","last_unstale":"2024-03-23T23:00:53.290395+0000","last_undegraded":"2024-03-23T23:00:53.290395+0000","last_fullsized":"2024-03-23T23:00:53.290395+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:55:57.730564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.279322+0000","last_change":"2024-03-23T23:00:24.381861+0000","last_active":"2024-03-23T23:00:53.279322+0000","last_peered":"2024-03-23T23:00:53.279322+0000","last_clean":"2024-03-23T23:00:53.279322+0000","last_became_active":"2024-03-23T23:00:24.381472+0000","last_became_peered":"2024-03-23T23:00:24.381472+0000","last_unstale":"2024-03-23T23:00:53.279322+0000","last_undegraded":"2024-03-23T23:00:53.279322+0000","last_fullsized":"2024-03-23T23:00:53.279322+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:44:07.518911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":162,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:01:48.428986+0000","last_change":"2024-03-23T22:59:26.450679+0000","last_active":"2024-03-23T23:01:48.428986+0000","last_peered":"2024-03-23T23:01:48.428986+0000","last_clean":"2024-03-23T23:01:48.428986+0000","last_became_active":"2024-03-23T22:59:26.450234+0000","last_became_peered":"2024-03-23T22:59:26.450234+0000","last_unstale":"2024-03-23T23:01:48.428986+0000","last_undegraded":"2024-03-23T23:01:48.428986+0000","last_fullsized":"2024-03-23T23:01:48.428986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:52:25.869194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.289045+0000","last_change":"2024-03-23T22:59:27.220171+0000","last_active":"2024-03-23T23:00:53.289045+0000","last_peered":"2024-03-23T23:00:53.289045+0000","last_clean":"2024-03-23T23:00:53.289045+0000","last_became_active":"2024-03-23T22:59:27.219787+0000","last_became_peered":"2024-03-23T22:59:27.219787+0000","last_unstale":"2024-03-23T23:00:53.289045+0000","last_undegraded":"2024-03-23T23:00:53.289045+0000","last_fullsized":"2024-03-23T23:00:53.289045+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:51:52.634264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.373128+0000","last_change":"2024-03-23T22:59:26.440205+0000","last_active":"2024-03-23T23:00:24.373128+0000","last_peered":"2024-03-23T23:00:24.373128+0000","last_clean":"2024-03-23T23:00:24.373128+0000","last_became_active":"2024-03-23T22:59:26.439752+0000","last_became_peered":"2024-03-23T22:59:26.439752+0000","last_unstale":"2024-03-23T23:00:24.373128+0000","last_undegraded":"2024-03-23T23:00:24.373128+0000","last_fullsized":"2024-03-23T23:00:24.373128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:55:25.661887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":345,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356800+0000","last_change":"2024-03-23T22:58:06.171317+0000","last_active":"2024-03-23T23:00:24.356800+0000","last_peered":"2024-03-23T23:00:24.356800+0000","last_clean":"2024-03-23T23:00:24.356800+0000","last_became_active":"2024-03-23T22:58:06.170816+0000","last_became_peered":"2024-03-23T22:58:06.170816+0000","last_unstale":"2024-03-23T23:00:24.356800+0000","last_undegraded":"2024-03-23T23:00:24.356800+0000","last_fullsized":"2024-03-23T23:00:24.356800+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:54:11.991284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:54:11.991284+0000","last_clean_scrub_stamp":"2024-03-23T22:54:11.991284+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:20:55.980613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.283579+0000","last_change":"2024-03-23T22:59:27.200348+0000","last_active":"2024-03-23T23:00:53.283579+0000","last_peered":"2024-03-23T23:00:53.283579+0000","last_clean":"2024-03-23T23:00:53.283579+0000","last_became_active":"2024-03-23T22:59:27.199979+0000","last_became_peered":"2024-03-23T22:59:27.199979+0000","last_unstale":"2024-03-23T23:00:53.283579+0000","last_undegraded":"2024-03-23T23:00:53.283579+0000","last_fullsized":"2024-03-23T23:00:53.283579+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:31:19.258890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356946+0000","last_change":"2024-03-23T22:59:26.431837+0000","last_active":"2024-03-23T23:00:24.356946+0000","last_peered":"2024-03-23T23:00:24.356946+0000","last_clean":"2024-03-23T23:00:24.356946+0000","last_became_active":"2024-03-23T22:59:26.431580+0000","last_became_peered":"2024-03-23T22:59:26.431580+0000","last_unstale":"2024-03-23T23:00:24.356946+0000","last_undegraded":"2024-03-23T23:00:24.356946+0000","last_fullsized":"2024-03-23T23:00:24.356946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:49:53.316952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356929+0000","last_change":"2024-03-23T22:59:26.442878+0000","last_active":"2024-03-23T23:00:24.356929+0000","last_peered":"2024-03-23T23:00:24.356929+0000","last_clean":"2024-03-23T23:00:24.356929+0000","last_became_active":"2024-03-23T22:59:26.442303+0000","last_became_peered":"2024-03-23T22:59:26.442303+0000","last_unstale":"2024-03-23T23:00:24.356929+0000","last_undegraded":"2024-03-23T23:00:24.356929+0000","last_fullsized":"2024-03-23T23:00:24.356929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:51.717553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.279622+0000","last_change":"2024-03-23T22:59:27.212550+0000","last_active":"2024-03-23T23:00:53.279622+0000","last_peered":"2024-03-23T23:00:53.279622+0000","last_clean":"2024-03-23T23:00:53.279622+0000","last_became_active":"2024-03-23T22:59:27.212105+0000","last_became_peered":"2024-03-23T22:59:27.212105+0000","last_unstale":"2024-03-23T23:00:53.279622+0000","last_undegraded":"2024-03-23T23:00:53.279622+0000","last_fullsized":"2024-03-23T23:00:53.279622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:23:37.374081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'11","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:01:36.692498+0000","last_change":"2024-03-23T22:59:26.452146+0000","last_active":"2024-03-23T23:01:36.692498+0000","last_peered":"2024-03-23T23:01:36.692498+0000","last_clean":"2024-03-23T23:01:36.692498+0000","last_became_active":"2024-03-23T22:59:26.450398+0000","last_became_peered":"2024-03-23T22:59:26.450398+0000","last_unstale":"2024-03-23T23:01:36.692498+0000","last_undegraded":"2024-03-23T23:01:36.692498+0000","last_fullsized":"2024-03-23T23:01:36.692498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:55:47.085457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.295334+0000","last_change":"2024-03-23T22:59:27.209690+0000","last_active":"2024-03-23T23:00:53.295334+0000","last_peered":"2024-03-23T23:00:53.295334+0000","last_clean":"2024-03-23T23:00:53.295334+0000","last_became_active":"2024-03-23T22:59:27.209187+0000","last_became_peered":"2024-03-23T22:59:27.209187+0000","last_unstale":"2024-03-23T23:00:53.295334+0000","last_undegraded":"2024-03-23T23:00:53.295334+0000","last_fullsized":"2024-03-23T23:00:53.295334+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:19:45.315459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.373744+0000","last_change":"2024-03-23T22:59:26.434958+0000","last_active":"2024-03-23T23:00:24.373744+0000","last_peered":"2024-03-23T23:00:24.373744+0000","last_clean":"2024-03-23T23:00:24.373744+0000","last_became_active":"2024-03-23T22:59:26.434611+0000","last_became_peered":"2024-03-23T22:59:26.434611+0000","last_unstale":"2024-03-23T23:00:24.373744+0000","last_undegraded":"2024-03-23T23:00:24.373744+0000","last_fullsized":"2024-03-23T23:00:24.373744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:42:18.201885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.300163+0000","last_change":"2024-03-23T22:59:27.211700+0000","last_active":"2024-03-23T23:00:53.300163+0000","last_peered":"2024-03-23T23:00:53.300163+0000","last_clean":"2024-03-23T23:00:53.300163+0000","last_became_active":"2024-03-23T22:59:27.211248+0000","last_became_peered":"2024-03-23T22:59:27.211248+0000","last_unstale":"2024-03-23T23:00:53.300163+0000","last_undegraded":"2024-03-23T23:00:53.300163+0000","last_fullsized":"2024-03-23T23:00:53.300163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:49.868397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356086+0000","last_change":"2024-03-23T22:59:26.442625+0000","last_active":"2024-03-23T23:00:24.356086+0000","last_peered":"2024-03-23T23:00:24.356086+0000","last_clean":"2024-03-23T23:00:24.356086+0000","last_became_active":"2024-03-23T22:59:26.442312+0000","last_became_peered":"2024-03-23T22:59:26.442312+0000","last_unstale":"2024-03-23T23:00:24.356086+0000","last_undegraded":"2024-03-23T23:00:24.356086+0000","last_fullsized":"2024-03-23T23:00:24.356086+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:27:38.808784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.282372+0000","last_change":"2024-03-23T22:59:27.187363+0000","last_active":"2024-03-23T23:00:53.282372+0000","last_peered":"2024-03-23T23:00:53.282372+0000","last_clean":"2024-03-23T23:00:53.282372+0000","last_became_active":"2024-03-23T22:59:27.187007+0000","last_became_peered":"2024-03-23T22:59:27.187007+0000","last_unstale":"2024-03-23T23:00:53.282372+0000","last_undegraded":"2024-03-23T23:00:53.282372+0000","last_fullsized":"2024-03-23T23:00:53.282372+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:41:37.041254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.369860+0000","last_change":"2024-03-23T22:59:26.445753+0000","last_active":"2024-03-23T23:00:24.369860+0000","last_peered":"2024-03-23T23:00:24.369860+0000","last_clean":"2024-03-23T23:00:24.369860+0000","last_became_active":"2024-03-23T22:59:26.445167+0000","last_became_peered":"2024-03-23T22:59:26.445167+0000","last_unstale":"2024-03-23T23:00:24.369860+0000","last_undegraded":"2024-03-23T23:00:24.369860+0000","last_fullsized":"2024-03-23T23:00:24.369860+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:46:46.058523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.281974+0000","last_change":"2024-03-23T22:59:27.203855+0000","last_active":"2024-03-23T23:00:53.281974+0000","last_peered":"2024-03-23T23:00:53.281974+0000","last_clean":"2024-03-23T23:00:53.281974+0000","last_became_active":"2024-03-23T22:59:27.203438+0000","last_became_peered":"2024-03-23T22:59:27.203438+0000","last_unstale":"2024-03-23T23:00:53.281974+0000","last_undegraded":"2024-03-23T23:00:53.281974+0000","last_fullsized":"2024-03-23T23:00:53.281974+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:28.725743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:01:40.004294+0000","last_change":"2024-03-23T22:59:26.439002+0000","last_active":"2024-03-23T23:01:40.004294+0000","last_peered":"2024-03-23T23:01:40.004294+0000","last_clean":"2024-03-23T23:01:40.004294+0000","last_became_active":"2024-03-23T22:59:26.438634+0000","last_became_peered":"2024-03-23T22:59:26.438634+0000","last_unstale":"2024-03-23T23:01:40.004294+0000","last_undegraded":"2024-03-23T23:01:40.004294+0000","last_fullsized":"2024-03-23T23:01:40.004294+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:25:02.988163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.295264+0000","last_change":"2024-03-23T22:59:27.217314+0000","last_active":"2024-03-23T23:00:53.295264+0000","last_peered":"2024-03-23T23:00:53.295264+0000","last_clean":"2024-03-23T23:00:53.295264+0000","last_became_active":"2024-03-23T22:59:27.216790+0000","last_became_peered":"2024-03-23T22:59:27.216790+0000","last_unstale":"2024-03-23T23:00:53.295264+0000","last_undegraded":"2024-03-23T23:00:53.295264+0000","last_fullsized":"2024-03-23T23:00:53.295264+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:47:13.902236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357159+0000","last_change":"2024-03-23T22:59:26.431453+0000","last_active":"2024-03-23T23:00:24.357159+0000","last_peered":"2024-03-23T23:00:24.357159+0000","last_clean":"2024-03-23T23:00:24.357159+0000","last_became_active":"2024-03-23T22:59:26.429731+0000","last_became_peered":"2024-03-23T22:59:26.429731+0000","last_unstale":"2024-03-23T23:00:24.357159+0000","last_undegraded":"2024-03-23T23:00:24.357159+0000","last_fullsized":"2024-03-23T23:00:24.357159+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:33:47.717064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.295443+0000","last_change":"2024-03-23T22:59:27.203097+0000","last_active":"2024-03-23T23:00:53.295443+0000","last_peered":"2024-03-23T23:00:53.295443+0000","last_clean":"2024-03-23T23:00:53.295443+0000","last_became_active":"2024-03-23T22:59:27.202484+0000","last_became_peered":"2024-03-23T22:59:27.202484+0000","last_unstale":"2024-03-23T23:00:53.295443+0000","last_undegraded":"2024-03-23T23:00:53.295443+0000","last_fullsized":"2024-03-23T23:00:53.295443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:07.829866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.377539+0000","last_change":"2024-03-23T23:00:24.377180+0000","last_active":"2024-03-23T23:00:24.377539+0000","last_peered":"2024-03-23T23:00:24.377539+0000","last_clean":"2024-03-23T23:00:24.377539+0000","last_became_active":"2024-03-23T23:00:24.364530+0000","last_became_peered":"2024-03-23T23:00:24.364530+0000","last_unstale":"2024-03-23T23:00:24.377539+0000","last_undegraded":"2024-03-23T23:00:24.377539+0000","last_fullsized":"2024-03-23T23:00:24.377539+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:57.507891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.286545+0000","last_change":"2024-03-23T22:59:27.189244+0000","last_active":"2024-03-23T23:00:53.286545+0000","last_peered":"2024-03-23T23:00:53.286545+0000","last_clean":"2024-03-23T23:00:53.286545+0000","last_became_active":"2024-03-23T22:59:27.188629+0000","last_became_peered":"2024-03-23T22:59:27.188629+0000","last_unstale":"2024-03-23T23:00:53.286545+0000","last_undegraded":"2024-03-23T23:00:53.286545+0000","last_fullsized":"2024-03-23T23:00:53.286545+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:47:26.742204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356493+0000","last_change":"2024-03-23T22:59:26.434326+0000","last_active":"2024-03-23T23:00:24.356493+0000","last_peered":"2024-03-23T23:00:24.356493+0000","last_clean":"2024-03-23T23:00:24.356493+0000","last_became_active":"2024-03-23T22:59:26.433878+0000","last_became_peered":"2024-03-23T22:59:26.433878+0000","last_unstale":"2024-03-23T23:00:24.356493+0000","last_undegraded":"2024-03-23T23:00:24.356493+0000","last_fullsized":"2024-03-23T23:00:24.356493+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:08.580572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.297139+0000","last_change":"2024-03-23T22:59:27.212909+0000","last_active":"2024-03-23T23:00:53.297139+0000","last_peered":"2024-03-23T23:00:53.297139+0000","last_clean":"2024-03-23T23:00:53.297139+0000","last_became_active":"2024-03-23T22:59:27.212504+0000","last_became_peered":"2024-03-23T22:59:27.212504+0000","last_unstale":"2024-03-23T23:00:53.297139+0000","last_undegraded":"2024-03-23T23:00:53.297139+0000","last_fullsized":"2024-03-23T23:00:53.297139+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:09.238272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.380290+0000","last_change":"2024-03-23T22:59:26.452577+0000","last_active":"2024-03-23T23:00:24.380290+0000","last_peered":"2024-03-23T23:00:24.380290+0000","last_clean":"2024-03-23T23:00:24.380290+0000","last_became_active":"2024-03-23T22:59:26.452107+0000","last_became_peered":"2024-03-23T22:59:26.452107+0000","last_unstale":"2024-03-23T23:00:24.380290+0000","last_undegraded":"2024-03-23T23:00:24.380290+0000","last_fullsized":"2024-03-23T23:00:24.380290+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:32:18.472334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.283849+0000","last_change":"2024-03-23T22:59:27.188539+0000","last_active":"2024-03-23T23:00:53.283849+0000","last_peered":"2024-03-23T23:00:53.283849+0000","last_clean":"2024-03-23T23:00:53.283849+0000","last_became_active":"2024-03-23T22:59:27.188184+0000","last_became_peered":"2024-03-23T22:59:27.188184+0000","last_unstale":"2024-03-23T23:00:53.283849+0000","last_undegraded":"2024-03-23T23:00:53.283849+0000","last_fullsized":"2024-03-23T23:00:53.283849+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:37:59.756492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.269022+0000","last_change":"2024-03-23T22:59:27.192083+0000","last_active":"2024-03-23T23:00:53.269022+0000","last_peered":"2024-03-23T23:00:53.269022+0000","last_clean":"2024-03-23T23:00:53.269022+0000","last_became_active":"2024-03-23T22:59:27.191709+0000","last_became_peered":"2024-03-23T22:59:27.191709+0000","last_unstale":"2024-03-23T23:00:53.269022+0000","last_undegraded":"2024-03-23T23:00:53.269022+0000","last_fullsized":"2024-03-23T23:00:53.269022+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:18:42.905177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.275138+0000","last_change":"2024-03-23T22:59:27.186315+0000","last_active":"2024-03-23T23:00:53.275138+0000","last_peered":"2024-03-23T23:00:53.275138+0000","last_clean":"2024-03-23T23:00:53.275138+0000","last_became_active":"2024-03-23T22:59:27.185960+0000","last_became_peered":"2024-03-23T22:59:27.185960+0000","last_unstale":"2024-03-23T23:00:53.275138+0000","last_undegraded":"2024-03-23T23:00:53.275138+0000","last_fullsized":"2024-03-23T23:00:53.275138+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:57:07.739725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.291985+0000","last_change":"2024-03-23T22:59:27.221696+0000","last_active":"2024-03-23T23:00:53.291985+0000","last_peered":"2024-03-23T23:00:53.291985+0000","last_clean":"2024-03-23T23:00:53.291985+0000","last_became_active":"2024-03-23T22:59:27.221400+0000","last_became_peered":"2024-03-23T22:59:27.221400+0000","last_unstale":"2024-03-23T23:00:53.291985+0000","last_undegraded":"2024-03-23T23:00:53.291985+0000","last_fullsized":"2024-03-23T23:00:53.291985+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:17.905737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.287320+0000","last_change":"2024-03-23T22:59:27.206286+0000","last_active":"2024-03-23T23:00:53.287320+0000","last_peered":"2024-03-23T23:00:53.287320+0000","last_clean":"2024-03-23T23:00:53.287320+0000","last_became_active":"2024-03-23T22:59:27.205854+0000","last_became_peered":"2024-03-23T22:59:27.205854+0000","last_unstale":"2024-03-23T23:00:53.287320+0000","last_undegraded":"2024-03-23T23:00:53.287320+0000","last_fullsized":"2024-03-23T23:00:53.287320+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:00:45.216211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.299730+0000","last_change":"2024-03-23T22:59:27.211376+0000","last_active":"2024-03-23T23:00:53.299730+0000","last_peered":"2024-03-23T23:00:53.299730+0000","last_clean":"2024-03-23T23:00:53.299730+0000","last_became_active":"2024-03-23T22:59:27.211061+0000","last_became_peered":"2024-03-23T22:59:27.211061+0000","last_unstale":"2024-03-23T23:00:53.299730+0000","last_undegraded":"2024-03-23T23:00:53.299730+0000","last_fullsized":"2024-03-23T23:00:53.299730+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:53:25.776927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.268390+0000","last_change":"2024-03-23T22:59:27.191028+0000","last_active":"2024-03-23T23:00:53.268390+0000","last_peered":"2024-03-23T23:00:53.268390+0000","last_clean":"2024-03-23T23:00:53.268390+0000","last_became_active":"2024-03-23T22:59:27.190634+0000","last_became_peered":"2024-03-23T22:59:27.190634+0000","last_unstale":"2024-03-23T23:00:53.268390+0000","last_undegraded":"2024-03-23T23:00:53.268390+0000","last_fullsized":"2024-03-23T23:00:53.268390+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:50:52.110636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.289464+0000","last_change":"2024-03-23T22:59:27.183235+0000","last_active":"2024-03-23T23:00:53.289464+0000","last_peered":"2024-03-23T23:00:53.289464+0000","last_clean":"2024-03-23T23:00:53.289464+0000","last_became_active":"2024-03-23T22:59:27.181682+0000","last_became_peered":"2024-03-23T22:59:27.181682+0000","last_unstale":"2024-03-23T23:00:53.289464+0000","last_undegraded":"2024-03-23T23:00:53.289464+0000","last_fullsized":"2024-03-23T23:00:53.289464+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:33:26.002324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.284323+0000","last_change":"2024-03-23T23:00:24.378143+0000","last_active":"2024-03-23T23:00:53.284323+0000","last_peered":"2024-03-23T23:00:53.284323+0000","last_clean":"2024-03-23T23:00:53.284323+0000","last_became_active":"2024-03-23T23:00:24.377660+0000","last_became_peered":"2024-03-23T23:00:24.377660+0000","last_unstale":"2024-03-23T23:00:53.284323+0000","last_undegraded":"2024-03-23T23:00:53.284323+0000","last_fullsized":"2024-03-23T23:00:53.284323+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:24:29.250679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.284520+0000","last_change":"2024-03-23T22:59:27.215230+0000","last_active":"2024-03-23T23:00:53.284520+0000","last_peered":"2024-03-23T23:00:53.284520+0000","last_clean":"2024-03-23T23:00:53.284520+0000","last_became_active":"2024-03-23T22:59:27.214881+0000","last_became_peered":"2024-03-23T22:59:27.214881+0000","last_unstale":"2024-03-23T23:00:53.284520+0000","last_undegraded":"2024-03-23T23:00:53.284520+0000","last_fullsized":"2024-03-23T23:00:53.284520+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:10:40.428481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.296915+0000","last_change":"2024-03-23T22:59:27.213454+0000","last_active":"2024-03-23T23:00:53.296915+0000","last_peered":"2024-03-23T23:00:53.296915+0000","last_clean":"2024-03-23T23:00:53.296915+0000","last_became_active":"2024-03-23T22:59:27.212684+0000","last_became_peered":"2024-03-23T22:59:27.212684+0000","last_unstale":"2024-03-23T23:00:53.296915+0000","last_undegraded":"2024-03-23T23:00:53.296915+0000","last_fullsized":"2024-03-23T23:00:53.296915+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:48:51.584892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.287158+0000","last_change":"2024-03-23T22:59:27.220414+0000","last_active":"2024-03-23T23:00:53.287158+0000","last_peered":"2024-03-23T23:00:53.287158+0000","last_clean":"2024-03-23T23:00:53.287158+0000","last_became_active":"2024-03-23T22:59:27.220020+0000","last_became_peered":"2024-03-23T22:59:27.220020+0000","last_unstale":"2024-03-23T23:00:53.287158+0000","last_undegraded":"2024-03-23T23:00:53.287158+0000","last_fullsized":"2024-03-23T23:00:53.287158+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:35:48.890004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.287204+0000","last_change":"2024-03-23T22:59:27.215648+0000","last_active":"2024-03-23T23:00:53.287204+0000","last_peered":"2024-03-23T23:00:53.287204+0000","last_clean":"2024-03-23T23:00:53.287204+0000","last_became_active":"2024-03-23T22:59:27.212252+0000","last_became_peered":"2024-03-23T22:59:27.212252+0000","last_unstale":"2024-03-23T23:00:53.287204+0000","last_undegraded":"2024-03-23T23:00:53.287204+0000","last_fullsized":"2024-03-23T23:00:53.287204+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:08:38.956508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.274679+0000","last_change":"2024-03-23T22:59:27.190527+0000","last_active":"2024-03-23T23:00:53.274679+0000","last_peered":"2024-03-23T23:00:53.274679+0000","last_clean":"2024-03-23T23:00:53.274679+0000","last_became_active":"2024-03-23T22:59:27.190026+0000","last_became_peered":"2024-03-23T22:59:27.190026+0000","last_unstale":"2024-03-23T23:00:53.274679+0000","last_undegraded":"2024-03-23T23:00:53.274679+0000","last_fullsized":"2024-03-23T23:00:53.274679+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:31:24.663584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247269,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68250,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":868,"num_read_kb":507,"num_write":170,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":196122,"data_compressed":5619,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351019,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":459288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612656,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":1115103,"data_compressed":37666,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776182,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":459524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037821,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":459444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299457,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":1051289,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462983,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":954368,"data_stored":458913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":519308,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":460184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411423,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":519800,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149785,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":459224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888146,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":459350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724621,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":1050973,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34152,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34152,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":60602,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":61094,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":65912,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T23:01:54.793 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-23T23:01:54.794 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T23:01:54.827 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:54 smithi173 ceph-mon[36116]: pgmap v393: 97 pgs: 97 active+clean; 643 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-23T23:01:54.827 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:54 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1890798614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:01:54.827 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:54 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:54.827 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:54 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:54.827 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:54 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:54.827 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:54 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:54.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:54 smithi012 ceph-mon[31695]: pgmap v393: 97 pgs: 97 active+clean; 643 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-23T23:01:54.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:54 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1890798614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:01:54.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:54 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:54.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:54 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:54.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:54 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:54.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:54 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:54.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:54 smithi203 ceph-mon[36693]: pgmap v393: 97 pgs: 97 active+clean; 643 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-23T23:01:54.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:54 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1890798614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:01:54.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:54 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:54.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:54 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:54.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:54 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:54.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:54 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:55.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.051+0000 7fdb9e400640 1 -- 172.21.15.12:0/2780081454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb98104a90 msgr2=0x7fdb98106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:55.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.051+0000 7fdb9e400640 1 --2- 172.21.15.12:0/2780081454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb98104a90 0x7fdb98106e80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdb84009900 tx=0x7fdb8402bc30 comp rx=0 tx=0).stop 2024-03-23T23:01:55.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.051+0000 7fdb9e400640 1 -- 172.21.15.12:0/2780081454 shutdown_connections 2024-03-23T23:01:55.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.051+0000 7fdb9e400640 1 --2- 172.21.15.12:0/2780081454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb981073c0 0x7fdb98109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:55.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.051+0000 7fdb9e400640 1 --2- 172.21.15.12:0/2780081454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb98104a90 0x7fdb98106e80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:55.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.051+0000 7fdb9e400640 1 --2- 172.21.15.12:0/2780081454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb98102160 0x7fdb98104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:55.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.051+0000 7fdb9e400640 1 -- 172.21.15.12:0/2780081454 >> 172.21.15.12:0/2780081454 conn(0x7fdb980fbb50 msgr2=0x7fdb980fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:55.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.051+0000 7fdb9e400640 1 -- 172.21.15.12:0/2780081454 shutdown_connections 2024-03-23T23:01:55.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.051+0000 7fdb9e400640 1 -- 172.21.15.12:0/2780081454 wait complete. 2024-03-23T23:01:55.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.052+0000 7fdb9e400640 1 Processor -- start 2024-03-23T23:01:55.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.052+0000 7fdb9e400640 1 -- start start 2024-03-23T23:01:55.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.052+0000 7fdb9e400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb98102160 0x7fdb98197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:55.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.052+0000 7fdb9e400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb98104a90 0x7fdb98197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:55.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.053+0000 7fdb9e400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb981073c0 0x7fdb9819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:55.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.053+0000 7fdb9e400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9810d3c0 con 0x7fdb981073c0 2024-03-23T23:01:55.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.053+0000 7fdb9e400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9810d240 con 0x7fdb98102160 2024-03-23T23:01:55.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.053+0000 7fdb9e400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9810d540 con 0x7fdb98104a90 2024-03-23T23:01:55.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.053+0000 7fdb97400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb98102160 0x7fdb98197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.053+0000 7fdb97400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb98102160 0x7fdb98197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36464/0 (socket says 172.21.15.12:36464) 2024-03-23T23:01:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.053+0000 7fdb97400640 1 -- 172.21.15.12:0/4078107313 learned_addr learned my addr 172.21.15.12:0/4078107313 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.053+0000 7fdb97e00640 1 --2- 172.21.15.12:0/4078107313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb981073c0 0x7fdb9819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.053+0000 7fdb96a00640 1 --2- 172.21.15.12:0/4078107313 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb98104a90 0x7fdb98197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.053+0000 7fdb97400640 1 -- 172.21.15.12:0/4078107313 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb98104a90 msgr2=0x7fdb98197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.053+0000 7fdb97400640 1 --2- 172.21.15.12:0/4078107313 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb98104a90 0x7fdb98197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.053+0000 7fdb97400640 1 -- 172.21.15.12:0/4078107313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb981073c0 msgr2=0x7fdb9819efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.053+0000 7fdb97400640 1 --2- 172.21.15.12:0/4078107313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb981073c0 0x7fdb9819efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.053+0000 7fdb97400640 1 -- 172.21.15.12:0/4078107313 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb9819f630 con 0x7fdb98102160 2024-03-23T23:01:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.053+0000 7fdb97e00640 1 --2- 172.21.15.12:0/4078107313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb981073c0 0x7fdb9819efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:01:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.054+0000 7fdb97400640 1 --2- 172.21.15.12:0/4078107313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb98102160 0x7fdb98197a10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800e9f0 tx=0x7fdb8800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.054+0000 7fdb8fe00640 1 -- 172.21.15.12:0/4078107313 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8800cd80 con 0x7fdb98102160 2024-03-23T23:01:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.054+0000 7fdb8fe00640 1 -- 172.21.15.12:0/4078107313 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb88005500 con 0x7fdb98102160 2024-03-23T23:01:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.054+0000 7fdb9e400640 1 -- 172.21.15.12:0/4078107313 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb9819f920 con 0x7fdb98102160 2024-03-23T23:01:55.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.054+0000 7fdb8fe00640 1 -- 172.21.15.12:0/4078107313 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb88010690 con 0x7fdb98102160 2024-03-23T23:01:55.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.054+0000 7fdb9e400640 1 -- 172.21.15.12:0/4078107313 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb9819fdf0 con 0x7fdb98102160 2024-03-23T23:01:55.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.055+0000 7fdb9e400640 1 -- 172.21.15.12:0/4078107313 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb980fd640 con 0x7fdb98102160 2024-03-23T23:01:55.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.057+0000 7fdb8fe00640 1 -- 172.21.15.12:0/4078107313 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fdb880046c0 con 0x7fdb98102160 2024-03-23T23:01:55.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.058+0000 7fdb8fe00640 1 --2- 172.21.15.12:0/4078107313 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdb7c0a2800 0x7fdb7c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:55.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.059+0000 7fdb8fe00640 1 -- 172.21.15.12:0/4078107313 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fdb880cf070 con 0x7fdb98102160 2024-03-23T23:01:55.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.059+0000 7fdb96a00640 1 --2- 172.21.15.12:0/4078107313 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdb7c0a2800 0x7fdb7c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:55.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.060+0000 7fdb96a00640 1 --2- 172.21.15.12:0/4078107313 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdb7c0a2800 0x7fdb7c0a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdb8402c170 tx=0x7fdb8402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:55.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.062+0000 7fdb8fe00640 1 -- 172.21.15.12:0/4078107313 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdb8809c0c0 con 0x7fdb98102160 2024-03-23T23:01:55.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.280+0000 7fdb9e400640 1 -- 172.21.15.12:0/4078107313 --> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdb98061ae0 con 0x7fdb7c0a2800 2024-03-23T23:01:55.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.291+0000 7fdb8fe00640 1 -- 172.21.15.12:0/4078107313 <== mgr.14152 v2:172.21.15.12:6800/1345467734 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259036 (secure 0 0 0) 0x7fdb98061ae0 con 0x7fdb7c0a2800 2024-03-23T23:01:55.294 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:01:55.296 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-23T23:01:55.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.296+0000 7fdb9e400640 1 -- 172.21.15.12:0/4078107313 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdb7c0a2800 msgr2=0x7fdb7c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:55.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.296+0000 7fdb9e400640 1 --2- 172.21.15.12:0/4078107313 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdb7c0a2800 0x7fdb7c0a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdb8402c170 tx=0x7fdb8402df90 comp rx=0 tx=0).stop 2024-03-23T23:01:55.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.296+0000 7fdb9e400640 1 -- 172.21.15.12:0/4078107313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb98102160 msgr2=0x7fdb98197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:55.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.296+0000 7fdb9e400640 1 --2- 172.21.15.12:0/4078107313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb98102160 0x7fdb98197a10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800e9f0 tx=0x7fdb8800edf0 comp rx=0 tx=0).stop 2024-03-23T23:01:55.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.296+0000 7fdb9e400640 1 -- 172.21.15.12:0/4078107313 shutdown_connections 2024-03-23T23:01:55.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.296+0000 7fdb9e400640 1 --2- 172.21.15.12:0/4078107313 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdb7c0a2800 0x7fdb7c0a4cc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:55.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.296+0000 7fdb9e400640 1 --2- 172.21.15.12:0/4078107313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb981073c0 0x7fdb9819efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:55.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.296+0000 7fdb9e400640 1 --2- 172.21.15.12:0/4078107313 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb98104a90 0x7fdb98197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:55.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.296+0000 7fdb9e400640 1 --2- 172.21.15.12:0/4078107313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb98102160 0x7fdb98197a10 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:55.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.297+0000 7fdb9e400640 1 -- 172.21.15.12:0/4078107313 >> 172.21.15.12:0/4078107313 conn(0x7fdb980fbb50 msgr2=0x7fdb98102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:55.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.297+0000 7fdb9e400640 1 -- 172.21.15.12:0/4078107313 shutdown_connections 2024-03-23T23:01:55.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.297+0000 7fdb9e400640 1 -- 172.21.15.12:0/4078107313 wait complete. 2024-03-23T23:01:55.315 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":393,"stamp":"2024-03-23T23:01:53.318484+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658852,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":994,"num_read_kb":616,"num_write":353,"num_write_kb":3060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1181318,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":784,"ondisk_log_size":784,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369128,"kb_used_data":11656,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572696,"statfs":{"total":1151940427776,"available":1151562440704,"internally_reserved":0,"allocated":11935744,"data_stored":7472400,"data_compressed":112998,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4926,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":78,"num_read_kb":47,"num_write":9,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005487"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.285759+0000","last_change":"2024-03-23T22:59:27.200234+0000","last_active":"2024-03-23T23:00:53.285759+0000","last_peered":"2024-03-23T23:00:53.285759+0000","last_clean":"2024-03-23T23:00:53.285759+0000","last_became_active":"2024-03-23T22:59:27.199864+0000","last_became_peered":"2024-03-23T22:59:27.199864+0000","last_unstale":"2024-03-23T23:00:53.285759+0000","last_undegraded":"2024-03-23T23:00:53.285759+0000","last_fullsized":"2024-03-23T23:00:53.285759+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:56:26.811964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.300077+0000","last_change":"2024-03-23T22:59:27.195101+0000","last_active":"2024-03-23T23:00:53.300077+0000","last_peered":"2024-03-23T23:00:53.300077+0000","last_clean":"2024-03-23T23:00:53.300077+0000","last_became_active":"2024-03-23T22:59:27.194650+0000","last_became_peered":"2024-03-23T22:59:27.194650+0000","last_unstale":"2024-03-23T23:00:53.300077+0000","last_undegraded":"2024-03-23T23:00:53.300077+0000","last_fullsized":"2024-03-23T23:00:53.300077+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:20.396811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.286097+0000","last_change":"2024-03-23T22:59:27.185575+0000","last_active":"2024-03-23T23:00:53.286097+0000","last_peered":"2024-03-23T23:00:53.286097+0000","last_clean":"2024-03-23T23:00:53.286097+0000","last_became_active":"2024-03-23T22:59:27.184328+0000","last_became_peered":"2024-03-23T22:59:27.184328+0000","last_unstale":"2024-03-23T23:00:53.286097+0000","last_undegraded":"2024-03-23T23:00:53.286097+0000","last_fullsized":"2024-03-23T23:00:53.286097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:31:46.221574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.270875+0000","last_change":"2024-03-23T22:59:27.184434+0000","last_active":"2024-03-23T23:00:53.270875+0000","last_peered":"2024-03-23T23:00:53.270875+0000","last_clean":"2024-03-23T23:00:53.270875+0000","last_became_active":"2024-03-23T22:59:27.183970+0000","last_became_peered":"2024-03-23T22:59:27.183970+0000","last_unstale":"2024-03-23T23:00:53.270875+0000","last_undegraded":"2024-03-23T23:00:53.270875+0000","last_fullsized":"2024-03-23T23:00:53.270875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:39:05.517976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.266775+0000","last_change":"2024-03-23T22:59:27.214741+0000","last_active":"2024-03-23T23:00:53.266775+0000","last_peered":"2024-03-23T23:00:53.266775+0000","last_clean":"2024-03-23T23:00:53.266775+0000","last_became_active":"2024-03-23T22:59:27.214311+0000","last_became_peered":"2024-03-23T22:59:27.214311+0000","last_unstale":"2024-03-23T23:00:53.266775+0000","last_undegraded":"2024-03-23T23:00:53.266775+0000","last_fullsized":"2024-03-23T23:00:53.266775+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:35:38.464955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.291377+0000","last_change":"2024-03-23T22:59:27.200695+0000","last_active":"2024-03-23T23:00:53.291377+0000","last_peered":"2024-03-23T23:00:53.291377+0000","last_clean":"2024-03-23T23:00:53.291377+0000","last_became_active":"2024-03-23T22:59:27.200159+0000","last_became_peered":"2024-03-23T22:59:27.200159+0000","last_unstale":"2024-03-23T23:00:53.291377+0000","last_undegraded":"2024-03-23T23:00:53.291377+0000","last_fullsized":"2024-03-23T23:00:53.291377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:01:15.147856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.280147+0000","last_change":"2024-03-23T22:59:27.199928+0000","last_active":"2024-03-23T23:00:53.280147+0000","last_peered":"2024-03-23T23:00:53.280147+0000","last_clean":"2024-03-23T23:00:53.280147+0000","last_became_active":"2024-03-23T22:59:27.199507+0000","last_became_peered":"2024-03-23T22:59:27.199507+0000","last_unstale":"2024-03-23T23:00:53.280147+0000","last_undegraded":"2024-03-23T23:00:53.280147+0000","last_fullsized":"2024-03-23T23:00:53.280147+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:51:19.145785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.295760+0000","last_change":"2024-03-23T22:59:27.217832+0000","last_active":"2024-03-23T23:00:53.295760+0000","last_peered":"2024-03-23T23:00:53.295760+0000","last_clean":"2024-03-23T23:00:53.295760+0000","last_became_active":"2024-03-23T22:59:27.217496+0000","last_became_peered":"2024-03-23T22:59:27.217496+0000","last_unstale":"2024-03-23T23:00:53.295760+0000","last_undegraded":"2024-03-23T23:00:53.295760+0000","last_fullsized":"2024-03-23T23:00:53.295760+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:01.778627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.288651+0000","last_change":"2024-03-23T22:59:27.190982+0000","last_active":"2024-03-23T23:00:53.288651+0000","last_peered":"2024-03-23T23:00:53.288651+0000","last_clean":"2024-03-23T23:00:53.288651+0000","last_became_active":"2024-03-23T22:59:27.190661+0000","last_became_peered":"2024-03-23T22:59:27.190661+0000","last_unstale":"2024-03-23T23:00:53.288651+0000","last_undegraded":"2024-03-23T23:00:53.288651+0000","last_fullsized":"2024-03-23T23:00:53.288651+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:29.822253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.281328+0000","last_change":"2024-03-23T22:59:27.187395+0000","last_active":"2024-03-23T23:00:53.281328+0000","last_peered":"2024-03-23T23:00:53.281328+0000","last_clean":"2024-03-23T23:00:53.281328+0000","last_became_active":"2024-03-23T22:59:27.186849+0000","last_became_peered":"2024-03-23T22:59:27.186849+0000","last_unstale":"2024-03-23T23:00:53.281328+0000","last_undegraded":"2024-03-23T23:00:53.281328+0000","last_fullsized":"2024-03-23T23:00:53.281328+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:05:48.047102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.286208+0000","last_change":"2024-03-23T22:59:27.190023+0000","last_active":"2024-03-23T23:00:53.286208+0000","last_peered":"2024-03-23T23:00:53.286208+0000","last_clean":"2024-03-23T23:00:53.286208+0000","last_became_active":"2024-03-23T22:59:27.189697+0000","last_became_peered":"2024-03-23T22:59:27.189697+0000","last_unstale":"2024-03-23T23:00:53.286208+0000","last_undegraded":"2024-03-23T23:00:53.286208+0000","last_fullsized":"2024-03-23T23:00:53.286208+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:59:51.121055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.264109+0000","last_change":"2024-03-23T22:59:27.187382+0000","last_active":"2024-03-23T23:00:53.264109+0000","last_peered":"2024-03-23T23:00:53.264109+0000","last_clean":"2024-03-23T23:00:53.264109+0000","last_became_active":"2024-03-23T22:59:27.186931+0000","last_became_peered":"2024-03-23T22:59:27.186931+0000","last_unstale":"2024-03-23T23:00:53.264109+0000","last_undegraded":"2024-03-23T23:00:53.264109+0000","last_fullsized":"2024-03-23T23:00:53.264109+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:33.583123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.293942+0000","last_change":"2024-03-23T22:59:27.202577+0000","last_active":"2024-03-23T23:00:53.293942+0000","last_peered":"2024-03-23T23:00:53.293942+0000","last_clean":"2024-03-23T23:00:53.293942+0000","last_became_active":"2024-03-23T22:59:27.202122+0000","last_became_peered":"2024-03-23T22:59:27.202122+0000","last_unstale":"2024-03-23T23:00:53.293942+0000","last_undegraded":"2024-03-23T23:00:53.293942+0000","last_fullsized":"2024-03-23T23:00:53.293942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:01:10.809635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.298452+0000","last_change":"2024-03-23T22:59:27.192915+0000","last_active":"2024-03-23T23:00:53.298452+0000","last_peered":"2024-03-23T23:00:53.298452+0000","last_clean":"2024-03-23T23:00:53.298452+0000","last_became_active":"2024-03-23T22:59:27.192107+0000","last_became_peered":"2024-03-23T22:59:27.192107+0000","last_unstale":"2024-03-23T23:00:53.298452+0000","last_undegraded":"2024-03-23T23:00:53.298452+0000","last_fullsized":"2024-03-23T23:00:53.298452+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:21:19.169211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.300473+0000","last_change":"2024-03-23T22:59:27.192156+0000","last_active":"2024-03-23T23:00:53.300473+0000","last_peered":"2024-03-23T23:00:53.300473+0000","last_clean":"2024-03-23T23:00:53.300473+0000","last_became_active":"2024-03-23T22:59:27.191594+0000","last_became_peered":"2024-03-23T22:59:27.191594+0000","last_unstale":"2024-03-23T23:00:53.300473+0000","last_undegraded":"2024-03-23T23:00:53.300473+0000","last_fullsized":"2024-03-23T23:00:53.300473+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:00:45.480281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.299835+0000","last_change":"2024-03-23T22:59:27.186048+0000","last_active":"2024-03-23T23:00:53.299835+0000","last_peered":"2024-03-23T23:00:53.299835+0000","last_clean":"2024-03-23T23:00:53.299835+0000","last_became_active":"2024-03-23T22:59:27.185710+0000","last_became_peered":"2024-03-23T22:59:27.185710+0000","last_unstale":"2024-03-23T23:00:53.299835+0000","last_undegraded":"2024-03-23T23:00:53.299835+0000","last_fullsized":"2024-03-23T23:00:53.299835+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:06.792665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.298495+0000","last_change":"2024-03-23T22:59:27.206290+0000","last_active":"2024-03-23T23:00:53.298495+0000","last_peered":"2024-03-23T23:00:53.298495+0000","last_clean":"2024-03-23T23:00:53.298495+0000","last_became_active":"2024-03-23T22:59:27.205997+0000","last_became_peered":"2024-03-23T22:59:27.205997+0000","last_unstale":"2024-03-23T23:00:53.298495+0000","last_undegraded":"2024-03-23T23:00:53.298495+0000","last_fullsized":"2024-03-23T23:00:53.298495+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:38:42.006166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.289029+0000","last_change":"2024-03-23T22:59:27.203671+0000","last_active":"2024-03-23T23:00:53.289029+0000","last_peered":"2024-03-23T23:00:53.289029+0000","last_clean":"2024-03-23T23:00:53.289029+0000","last_became_active":"2024-03-23T22:59:27.203273+0000","last_became_peered":"2024-03-23T22:59:27.203273+0000","last_unstale":"2024-03-23T23:00:53.289029+0000","last_undegraded":"2024-03-23T23:00:53.289029+0000","last_fullsized":"2024-03-23T23:00:53.289029+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:48:16.018671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.292388+0000","last_change":"2024-03-23T22:59:27.210063+0000","last_active":"2024-03-23T23:00:53.292388+0000","last_peered":"2024-03-23T23:00:53.292388+0000","last_clean":"2024-03-23T23:00:53.292388+0000","last_became_active":"2024-03-23T22:59:27.209698+0000","last_became_peered":"2024-03-23T22:59:27.209698+0000","last_unstale":"2024-03-23T23:00:53.292388+0000","last_undegraded":"2024-03-23T23:00:53.292388+0000","last_fullsized":"2024-03-23T23:00:53.292388+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:19:01.408901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.358191+0000","last_change":"2024-03-23T22:59:26.445053+0000","last_active":"2024-03-23T23:00:24.358191+0000","last_peered":"2024-03-23T23:00:24.358191+0000","last_clean":"2024-03-23T23:00:24.358191+0000","last_became_active":"2024-03-23T22:59:26.444660+0000","last_became_peered":"2024-03-23T22:59:26.444660+0000","last_unstale":"2024-03-23T23:00:24.358191+0000","last_undegraded":"2024-03-23T23:00:24.358191+0000","last_fullsized":"2024-03-23T23:00:24.358191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:19:16.958228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.284909+0000","last_change":"2024-03-23T22:59:27.208652+0000","last_active":"2024-03-23T23:00:53.284909+0000","last_peered":"2024-03-23T23:00:53.284909+0000","last_clean":"2024-03-23T23:00:53.284909+0000","last_became_active":"2024-03-23T22:59:27.207724+0000","last_became_peered":"2024-03-23T22:59:27.207724+0000","last_unstale":"2024-03-23T23:00:53.284909+0000","last_undegraded":"2024-03-23T23:00:53.284909+0000","last_fullsized":"2024-03-23T23:00:53.284909+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:08.986439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.368748+0000","last_change":"2024-03-23T22:59:26.445371+0000","last_active":"2024-03-23T23:00:24.368748+0000","last_peered":"2024-03-23T23:00:24.368748+0000","last_clean":"2024-03-23T23:00:24.368748+0000","last_became_active":"2024-03-23T22:59:26.445016+0000","last_became_peered":"2024-03-23T22:59:26.445016+0000","last_unstale":"2024-03-23T23:00:24.368748+0000","last_undegraded":"2024-03-23T23:00:24.368748+0000","last_fullsized":"2024-03-23T23:00:24.368748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:31:44.003218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.300501+0000","last_change":"2024-03-23T22:59:27.200624+0000","last_active":"2024-03-23T23:00:53.300501+0000","last_peered":"2024-03-23T23:00:53.300501+0000","last_clean":"2024-03-23T23:00:53.300501+0000","last_became_active":"2024-03-23T22:59:27.200207+0000","last_became_peered":"2024-03-23T22:59:27.200207+0000","last_unstale":"2024-03-23T23:00:53.300501+0000","last_undegraded":"2024-03-23T23:00:53.300501+0000","last_fullsized":"2024-03-23T23:00:53.300501+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:05:31.015869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.380616+0000","last_change":"2024-03-23T23:00:24.380296+0000","last_active":"2024-03-23T23:00:24.380616+0000","last_peered":"2024-03-23T23:00:24.380616+0000","last_clean":"2024-03-23T23:00:24.380616+0000","last_became_active":"2024-03-23T23:00:24.374344+0000","last_became_peered":"2024-03-23T23:00:24.374344+0000","last_unstale":"2024-03-23T23:00:24.380616+0000","last_undegraded":"2024-03-23T23:00:24.380616+0000","last_fullsized":"2024-03-23T23:00:24.380616+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:30:20.174783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.288535+0000","last_change":"2024-03-23T22:59:27.204157+0000","last_active":"2024-03-23T23:00:53.288535+0000","last_peered":"2024-03-23T23:00:53.288535+0000","last_clean":"2024-03-23T23:00:53.288535+0000","last_became_active":"2024-03-23T22:59:27.203559+0000","last_became_peered":"2024-03-23T22:59:27.203559+0000","last_unstale":"2024-03-23T23:00:53.288535+0000","last_undegraded":"2024-03-23T23:00:53.288535+0000","last_fullsized":"2024-03-23T23:00:53.288535+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:43.834369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.361574+0000","last_change":"2024-03-23T23:00:24.361233+0000","last_active":"2024-03-23T23:00:24.361574+0000","last_peered":"2024-03-23T23:00:24.361574+0000","last_clean":"2024-03-23T23:00:24.361574+0000","last_became_active":"2024-03-23T23:00:24.358048+0000","last_became_peered":"2024-03-23T23:00:24.358048+0000","last_unstale":"2024-03-23T23:00:24.361574+0000","last_undegraded":"2024-03-23T23:00:24.361574+0000","last_fullsized":"2024-03-23T23:00:24.361574+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:38:37.792273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.286722+0000","last_change":"2024-03-23T22:59:27.186599+0000","last_active":"2024-03-23T23:00:53.286722+0000","last_peered":"2024-03-23T23:00:53.286722+0000","last_clean":"2024-03-23T23:00:53.286722+0000","last_became_active":"2024-03-23T22:59:27.186262+0000","last_became_peered":"2024-03-23T22:59:27.186262+0000","last_unstale":"2024-03-23T23:00:53.286722+0000","last_undegraded":"2024-03-23T23:00:53.286722+0000","last_fullsized":"2024-03-23T23:00:53.286722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:11:39.545558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":166,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:01:49.342977+0000","last_change":"2024-03-23T22:59:26.430096+0000","last_active":"2024-03-23T23:01:49.342977+0000","last_peered":"2024-03-23T23:01:49.342977+0000","last_clean":"2024-03-23T23:01:49.342977+0000","last_became_active":"2024-03-23T22:59:26.429740+0000","last_became_peered":"2024-03-23T22:59:26.429740+0000","last_unstale":"2024-03-23T23:01:49.342977+0000","last_undegraded":"2024-03-23T23:01:49.342977+0000","last_fullsized":"2024-03-23T23:01:49.342977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:54.831689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":928,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":141,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.262380+0000","last_change":"2024-03-23T22:59:27.202750+0000","last_active":"2024-03-23T23:00:53.262380+0000","last_peered":"2024-03-23T23:00:53.262380+0000","last_clean":"2024-03-23T23:00:53.262380+0000","last_became_active":"2024-03-23T22:59:27.202391+0000","last_became_peered":"2024-03-23T22:59:27.202391+0000","last_unstale":"2024-03-23T23:00:53.262380+0000","last_undegraded":"2024-03-23T23:00:53.262380+0000","last_fullsized":"2024-03-23T23:00:53.262380+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:27:18.370792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.365876+0000","last_change":"2024-03-23T22:59:26.425607+0000","last_active":"2024-03-23T23:00:24.365876+0000","last_peered":"2024-03-23T23:00:24.365876+0000","last_clean":"2024-03-23T23:00:24.365876+0000","last_became_active":"2024-03-23T22:59:26.425208+0000","last_became_peered":"2024-03-23T22:59:26.425208+0000","last_unstale":"2024-03-23T23:00:24.365876+0000","last_undegraded":"2024-03-23T23:00:24.365876+0000","last_fullsized":"2024-03-23T23:00:24.365876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:51.071662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.282112+0000","last_change":"2024-03-23T22:59:27.215637+0000","last_active":"2024-03-23T23:00:53.282112+0000","last_peered":"2024-03-23T23:00:53.282112+0000","last_clean":"2024-03-23T23:00:53.282112+0000","last_became_active":"2024-03-23T22:59:27.215339+0000","last_became_peered":"2024-03-23T22:59:27.215339+0000","last_unstale":"2024-03-23T23:00:53.282112+0000","last_undegraded":"2024-03-23T23:00:53.282112+0000","last_fullsized":"2024-03-23T23:00:53.282112+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:25:42.097326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.377892+0000","last_change":"2024-03-23T22:59:26.442023+0000","last_active":"2024-03-23T23:00:24.377892+0000","last_peered":"2024-03-23T23:00:24.377892+0000","last_clean":"2024-03-23T23:00:24.377892+0000","last_became_active":"2024-03-23T22:59:26.441661+0000","last_became_peered":"2024-03-23T22:59:26.441661+0000","last_unstale":"2024-03-23T23:00:24.377892+0000","last_undegraded":"2024-03-23T23:00:24.377892+0000","last_fullsized":"2024-03-23T23:00:24.377892+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:59:14.914860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.279352+0000","last_change":"2024-03-23T23:00:24.384190+0000","last_active":"2024-03-23T23:00:53.279352+0000","last_peered":"2024-03-23T23:00:53.279352+0000","last_clean":"2024-03-23T23:00:53.279352+0000","last_became_active":"2024-03-23T23:00:24.382141+0000","last_became_peered":"2024-03-23T23:00:24.382141+0000","last_unstale":"2024-03-23T23:00:53.279352+0000","last_undegraded":"2024-03-23T23:00:53.279352+0000","last_fullsized":"2024-03-23T23:00:53.279352+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:48:52.311407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356353+0000","last_change":"2024-03-23T22:59:26.430147+0000","last_active":"2024-03-23T23:00:24.356353+0000","last_peered":"2024-03-23T23:00:24.356353+0000","last_clean":"2024-03-23T23:00:24.356353+0000","last_became_active":"2024-03-23T22:59:26.429843+0000","last_became_peered":"2024-03-23T22:59:26.429843+0000","last_unstale":"2024-03-23T23:00:24.356353+0000","last_undegraded":"2024-03-23T23:00:24.356353+0000","last_fullsized":"2024-03-23T23:00:24.356353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:06:45.691894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.288385+0000","last_change":"2024-03-23T22:59:27.212235+0000","last_active":"2024-03-23T23:00:53.288385+0000","last_peered":"2024-03-23T23:00:53.288385+0000","last_clean":"2024-03-23T23:00:53.288385+0000","last_became_active":"2024-03-23T22:59:27.211919+0000","last_became_peered":"2024-03-23T22:59:27.211919+0000","last_unstale":"2024-03-23T23:00:53.288385+0000","last_undegraded":"2024-03-23T23:00:53.288385+0000","last_fullsized":"2024-03-23T23:00:53.288385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:24.752109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":165,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:01:52.346610+0000","last_change":"2024-03-23T22:59:26.440623+0000","last_active":"2024-03-23T23:01:52.346610+0000","last_peered":"2024-03-23T23:01:52.346610+0000","last_clean":"2024-03-23T23:01:52.346610+0000","last_became_active":"2024-03-23T22:59:26.440204+0000","last_became_peered":"2024-03-23T22:59:26.440204+0000","last_unstale":"2024-03-23T23:01:52.346610+0000","last_undegraded":"2024-03-23T23:01:52.346610+0000","last_fullsized":"2024-03-23T23:01:52.346610+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:58:44.088552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":141,"num_read_kb":141,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.287377+0000","last_change":"2024-03-23T22:59:27.203916+0000","last_active":"2024-03-23T23:00:53.287377+0000","last_peered":"2024-03-23T23:00:53.287377+0000","last_clean":"2024-03-23T23:00:53.287377+0000","last_became_active":"2024-03-23T22:59:27.203565+0000","last_became_peered":"2024-03-23T22:59:27.203565+0000","last_unstale":"2024-03-23T23:00:53.287377+0000","last_undegraded":"2024-03-23T23:00:53.287377+0000","last_fullsized":"2024-03-23T23:00:53.287377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:06:39.635025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357614+0000","last_change":"2024-03-23T22:59:26.430080+0000","last_active":"2024-03-23T23:00:24.357614+0000","last_peered":"2024-03-23T23:00:24.357614+0000","last_clean":"2024-03-23T23:00:24.357614+0000","last_became_active":"2024-03-23T22:59:26.429618+0000","last_became_peered":"2024-03-23T22:59:26.429618+0000","last_unstale":"2024-03-23T23:00:24.357614+0000","last_undegraded":"2024-03-23T23:00:24.357614+0000","last_fullsized":"2024-03-23T23:00:24.357614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:01.963573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.300287+0000","last_change":"2024-03-23T22:59:27.190505+0000","last_active":"2024-03-23T23:00:53.300287+0000","last_peered":"2024-03-23T23:00:53.300287+0000","last_clean":"2024-03-23T23:00:53.300287+0000","last_became_active":"2024-03-23T22:59:27.190088+0000","last_became_peered":"2024-03-23T22:59:27.190088+0000","last_unstale":"2024-03-23T23:00:53.300287+0000","last_undegraded":"2024-03-23T23:00:53.300287+0000","last_fullsized":"2024-03-23T23:00:53.300287+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:29:04.255832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357315+0000","last_change":"2024-03-23T22:59:26.438329+0000","last_active":"2024-03-23T23:00:24.357315+0000","last_peered":"2024-03-23T23:00:24.357315+0000","last_clean":"2024-03-23T23:00:24.357315+0000","last_became_active":"2024-03-23T22:59:26.438033+0000","last_became_peered":"2024-03-23T22:59:26.438033+0000","last_unstale":"2024-03-23T23:00:24.357315+0000","last_undegraded":"2024-03-23T23:00:24.357315+0000","last_fullsized":"2024-03-23T23:00:24.357315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:02:38.789574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.287183+0000","last_change":"2024-03-23T22:59:27.188417+0000","last_active":"2024-03-23T23:00:53.287183+0000","last_peered":"2024-03-23T23:00:53.287183+0000","last_clean":"2024-03-23T23:00:53.287183+0000","last_became_active":"2024-03-23T22:59:27.188023+0000","last_became_peered":"2024-03-23T22:59:27.188023+0000","last_unstale":"2024-03-23T23:00:53.287183+0000","last_undegraded":"2024-03-23T23:00:53.287183+0000","last_fullsized":"2024-03-23T23:00:53.287183+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:19:11.014604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'33","reported_seq":347,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:01:52.347319+0000","last_change":"2024-03-23T22:59:26.425963+0000","last_active":"2024-03-23T23:01:52.347319+0000","last_peered":"2024-03-23T23:01:52.347319+0000","last_clean":"2024-03-23T23:01:52.347319+0000","last_became_active":"2024-03-23T22:59:26.425640+0000","last_became_peered":"2024-03-23T22:59:26.425640+0000","last_unstale":"2024-03-23T23:01:52.347319+0000","last_undegraded":"2024-03-23T23:01:52.347319+0000","last_fullsized":"2024-03-23T23:01:52.347319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:11:26.529034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64334,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":294,"num_read_kb":210,"num_write":34,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.256447+0000","last_change":"2024-03-23T22:59:27.187254+0000","last_active":"2024-03-23T23:00:53.256447+0000","last_peered":"2024-03-23T23:00:53.256447+0000","last_clean":"2024-03-23T23:00:53.256447+0000","last_became_active":"2024-03-23T22:59:27.186849+0000","last_became_peered":"2024-03-23T22:59:27.186849+0000","last_unstale":"2024-03-23T23:00:53.256447+0000","last_undegraded":"2024-03-23T23:00:53.256447+0000","last_fullsized":"2024-03-23T23:00:53.256447+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:35:30.958565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.358477+0000","last_change":"2024-03-23T22:59:26.429767+0000","last_active":"2024-03-23T23:00:24.358477+0000","last_peered":"2024-03-23T23:00:24.358477+0000","last_clean":"2024-03-23T23:00:24.358477+0000","last_became_active":"2024-03-23T22:59:26.429449+0000","last_became_peered":"2024-03-23T22:59:26.429449+0000","last_unstale":"2024-03-23T23:00:24.358477+0000","last_undegraded":"2024-03-23T23:00:24.358477+0000","last_fullsized":"2024-03-23T23:00:24.358477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:46:13.752427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.298386+0000","last_change":"2024-03-23T22:59:27.223544+0000","last_active":"2024-03-23T23:00:53.298386+0000","last_peered":"2024-03-23T23:00:53.298386+0000","last_clean":"2024-03-23T23:00:53.298386+0000","last_became_active":"2024-03-23T22:59:27.223177+0000","last_became_peered":"2024-03-23T22:59:27.223177+0000","last_unstale":"2024-03-23T23:00:53.298386+0000","last_undegraded":"2024-03-23T23:00:53.298386+0000","last_fullsized":"2024-03-23T23:00:53.298386+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:33:18.521579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356567+0000","last_change":"2024-03-23T22:59:26.427781+0000","last_active":"2024-03-23T23:00:24.356567+0000","last_peered":"2024-03-23T23:00:24.356567+0000","last_clean":"2024-03-23T23:00:24.356567+0000","last_became_active":"2024-03-23T22:59:26.427458+0000","last_became_peered":"2024-03-23T22:59:26.427458+0000","last_unstale":"2024-03-23T23:00:24.356567+0000","last_undegraded":"2024-03-23T23:00:24.356567+0000","last_fullsized":"2024-03-23T23:00:24.356567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:50:33.956611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.295652+0000","last_change":"2024-03-23T22:59:27.203681+0000","last_active":"2024-03-23T23:00:53.295652+0000","last_peered":"2024-03-23T23:00:53.295652+0000","last_clean":"2024-03-23T23:00:53.295652+0000","last_became_active":"2024-03-23T22:59:27.202457+0000","last_became_peered":"2024-03-23T22:59:27.202457+0000","last_unstale":"2024-03-23T23:00:53.295652+0000","last_undegraded":"2024-03-23T23:00:53.295652+0000","last_fullsized":"2024-03-23T23:00:53.295652+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:55:46.725350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'3","reported_seq":164,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:01:49.431259+0000","last_change":"2024-03-23T22:59:26.428517+0000","last_active":"2024-03-23T23:01:49.431259+0000","last_peered":"2024-03-23T23:01:49.431259+0000","last_clean":"2024-03-23T23:01:49.431259+0000","last_became_active":"2024-03-23T22:59:26.428130+0000","last_became_peered":"2024-03-23T22:59:26.428130+0000","last_unstale":"2024-03-23T23:01:49.431259+0000","last_undegraded":"2024-03-23T23:01:49.431259+0000","last_fullsized":"2024-03-23T23:01:49.431259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:01.185776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":142,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.283053+0000","last_change":"2024-03-23T23:00:24.367049+0000","last_active":"2024-03-23T23:00:53.283053+0000","last_peered":"2024-03-23T23:00:53.283053+0000","last_clean":"2024-03-23T23:00:53.283053+0000","last_became_active":"2024-03-23T23:00:24.366636+0000","last_became_peered":"2024-03-23T23:00:24.366636+0000","last_unstale":"2024-03-23T23:00:53.283053+0000","last_undegraded":"2024-03-23T23:00:53.283053+0000","last_fullsized":"2024-03-23T23:00:53.283053+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:07.068500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356958+0000","last_change":"2024-03-23T22:59:26.442637+0000","last_active":"2024-03-23T23:00:24.356958+0000","last_peered":"2024-03-23T23:00:24.356958+0000","last_clean":"2024-03-23T23:00:24.356958+0000","last_became_active":"2024-03-23T22:59:26.442267+0000","last_became_peered":"2024-03-23T22:59:26.442267+0000","last_unstale":"2024-03-23T23:00:24.356958+0000","last_undegraded":"2024-03-23T23:00:24.356958+0000","last_fullsized":"2024-03-23T23:00:24.356958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:49.975048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.286407+0000","last_change":"2024-03-23T22:59:27.187812+0000","last_active":"2024-03-23T23:00:53.286407+0000","last_peered":"2024-03-23T23:00:53.286407+0000","last_clean":"2024-03-23T23:00:53.286407+0000","last_became_active":"2024-03-23T22:59:27.187423+0000","last_became_peered":"2024-03-23T22:59:27.187423+0000","last_unstale":"2024-03-23T23:00:53.286407+0000","last_undegraded":"2024-03-23T23:00:53.286407+0000","last_fullsized":"2024-03-23T23:00:53.286407+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:12:16.427695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357353+0000","last_change":"2024-03-23T22:59:26.423875+0000","last_active":"2024-03-23T23:00:24.357353+0000","last_peered":"2024-03-23T23:00:24.357353+0000","last_clean":"2024-03-23T23:00:24.357353+0000","last_became_active":"2024-03-23T22:59:26.423513+0000","last_became_peered":"2024-03-23T22:59:26.423513+0000","last_unstale":"2024-03-23T23:00:24.357353+0000","last_undegraded":"2024-03-23T23:00:24.357353+0000","last_fullsized":"2024-03-23T23:00:24.357353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:43.173365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.268696+0000","last_change":"2024-03-23T22:59:27.191454+0000","last_active":"2024-03-23T23:00:53.268696+0000","last_peered":"2024-03-23T23:00:53.268696+0000","last_clean":"2024-03-23T23:00:53.268696+0000","last_became_active":"2024-03-23T22:59:27.190927+0000","last_became_peered":"2024-03-23T22:59:27.190927+0000","last_unstale":"2024-03-23T23:00:53.268696+0000","last_undegraded":"2024-03-23T23:00:53.268696+0000","last_fullsized":"2024-03-23T23:00:53.268696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:00.333702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.358997+0000","last_change":"2024-03-23T22:59:26.443238+0000","last_active":"2024-03-23T23:00:24.358997+0000","last_peered":"2024-03-23T23:00:24.358997+0000","last_clean":"2024-03-23T23:00:24.358997+0000","last_became_active":"2024-03-23T22:59:26.442834+0000","last_became_peered":"2024-03-23T22:59:26.442834+0000","last_unstale":"2024-03-23T23:00:24.358997+0000","last_undegraded":"2024-03-23T23:00:24.358997+0000","last_fullsized":"2024-03-23T23:00:24.358997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:11:21.072836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.295551+0000","last_change":"2024-03-23T22:59:27.212864+0000","last_active":"2024-03-23T23:00:53.295551+0000","last_peered":"2024-03-23T23:00:53.295551+0000","last_clean":"2024-03-23T23:00:53.295551+0000","last_became_active":"2024-03-23T22:59:27.212455+0000","last_became_peered":"2024-03-23T22:59:27.212455+0000","last_unstale":"2024-03-23T23:00:53.295551+0000","last_undegraded":"2024-03-23T23:00:53.295551+0000","last_fullsized":"2024-03-23T23:00:53.295551+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:25:59.122531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357120+0000","last_change":"2024-03-23T22:59:26.432838+0000","last_active":"2024-03-23T23:00:24.357120+0000","last_peered":"2024-03-23T23:00:24.357120+0000","last_clean":"2024-03-23T23:00:24.357120+0000","last_became_active":"2024-03-23T22:59:26.432544+0000","last_became_peered":"2024-03-23T22:59:26.432544+0000","last_unstale":"2024-03-23T23:00:24.357120+0000","last_undegraded":"2024-03-23T23:00:24.357120+0000","last_fullsized":"2024-03-23T23:00:24.357120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:32:52.121569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.290395+0000","last_change":"2024-03-23T23:00:24.376926+0000","last_active":"2024-03-23T23:00:53.290395+0000","last_peered":"2024-03-23T23:00:53.290395+0000","last_clean":"2024-03-23T23:00:53.290395+0000","last_became_active":"2024-03-23T23:00:24.376515+0000","last_became_peered":"2024-03-23T23:00:24.376515+0000","last_unstale":"2024-03-23T23:00:53.290395+0000","last_undegraded":"2024-03-23T23:00:53.290395+0000","last_fullsized":"2024-03-23T23:00:53.290395+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:55:57.730564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.279322+0000","last_change":"2024-03-23T23:00:24.381861+0000","last_active":"2024-03-23T23:00:53.279322+0000","last_peered":"2024-03-23T23:00:53.279322+0000","last_clean":"2024-03-23T23:00:53.279322+0000","last_became_active":"2024-03-23T23:00:24.381472+0000","last_became_peered":"2024-03-23T23:00:24.381472+0000","last_unstale":"2024-03-23T23:00:53.279322+0000","last_undegraded":"2024-03-23T23:00:53.279322+0000","last_fullsized":"2024-03-23T23:00:53.279322+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:44:07.518911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":162,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:01:48.428986+0000","last_change":"2024-03-23T22:59:26.450679+0000","last_active":"2024-03-23T23:01:48.428986+0000","last_peered":"2024-03-23T23:01:48.428986+0000","last_clean":"2024-03-23T23:01:48.428986+0000","last_became_active":"2024-03-23T22:59:26.450234+0000","last_became_peered":"2024-03-23T22:59:26.450234+0000","last_unstale":"2024-03-23T23:01:48.428986+0000","last_undegraded":"2024-03-23T23:01:48.428986+0000","last_fullsized":"2024-03-23T23:01:48.428986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:52:25.869194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.289045+0000","last_change":"2024-03-23T22:59:27.220171+0000","last_active":"2024-03-23T23:00:53.289045+0000","last_peered":"2024-03-23T23:00:53.289045+0000","last_clean":"2024-03-23T23:00:53.289045+0000","last_became_active":"2024-03-23T22:59:27.219787+0000","last_became_peered":"2024-03-23T22:59:27.219787+0000","last_unstale":"2024-03-23T23:00:53.289045+0000","last_undegraded":"2024-03-23T23:00:53.289045+0000","last_fullsized":"2024-03-23T23:00:53.289045+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:51:52.634264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.373128+0000","last_change":"2024-03-23T22:59:26.440205+0000","last_active":"2024-03-23T23:00:24.373128+0000","last_peered":"2024-03-23T23:00:24.373128+0000","last_clean":"2024-03-23T23:00:24.373128+0000","last_became_active":"2024-03-23T22:59:26.439752+0000","last_became_peered":"2024-03-23T22:59:26.439752+0000","last_unstale":"2024-03-23T23:00:24.373128+0000","last_undegraded":"2024-03-23T23:00:24.373128+0000","last_fullsized":"2024-03-23T23:00:24.373128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:55:25.661887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":345,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356800+0000","last_change":"2024-03-23T22:58:06.171317+0000","last_active":"2024-03-23T23:00:24.356800+0000","last_peered":"2024-03-23T23:00:24.356800+0000","last_clean":"2024-03-23T23:00:24.356800+0000","last_became_active":"2024-03-23T22:58:06.170816+0000","last_became_peered":"2024-03-23T22:58:06.170816+0000","last_unstale":"2024-03-23T23:00:24.356800+0000","last_undegraded":"2024-03-23T23:00:24.356800+0000","last_fullsized":"2024-03-23T23:00:24.356800+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:54:11.991284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:54:11.991284+0000","last_clean_scrub_stamp":"2024-03-23T22:54:11.991284+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:20:55.980613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.283579+0000","last_change":"2024-03-23T22:59:27.200348+0000","last_active":"2024-03-23T23:00:53.283579+0000","last_peered":"2024-03-23T23:00:53.283579+0000","last_clean":"2024-03-23T23:00:53.283579+0000","last_became_active":"2024-03-23T22:59:27.199979+0000","last_became_peered":"2024-03-23T22:59:27.199979+0000","last_unstale":"2024-03-23T23:00:53.283579+0000","last_undegraded":"2024-03-23T23:00:53.283579+0000","last_fullsized":"2024-03-23T23:00:53.283579+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:31:19.258890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356946+0000","last_change":"2024-03-23T22:59:26.431837+0000","last_active":"2024-03-23T23:00:24.356946+0000","last_peered":"2024-03-23T23:00:24.356946+0000","last_clean":"2024-03-23T23:00:24.356946+0000","last_became_active":"2024-03-23T22:59:26.431580+0000","last_became_peered":"2024-03-23T22:59:26.431580+0000","last_unstale":"2024-03-23T23:00:24.356946+0000","last_undegraded":"2024-03-23T23:00:24.356946+0000","last_fullsized":"2024-03-23T23:00:24.356946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:49:53.316952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356929+0000","last_change":"2024-03-23T22:59:26.442878+0000","last_active":"2024-03-23T23:00:24.356929+0000","last_peered":"2024-03-23T23:00:24.356929+0000","last_clean":"2024-03-23T23:00:24.356929+0000","last_became_active":"2024-03-23T22:59:26.442303+0000","last_became_peered":"2024-03-23T22:59:26.442303+0000","last_unstale":"2024-03-23T23:00:24.356929+0000","last_undegraded":"2024-03-23T23:00:24.356929+0000","last_fullsized":"2024-03-23T23:00:24.356929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:51.717553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.279622+0000","last_change":"2024-03-23T22:59:27.212550+0000","last_active":"2024-03-23T23:00:53.279622+0000","last_peered":"2024-03-23T23:00:53.279622+0000","last_clean":"2024-03-23T23:00:53.279622+0000","last_became_active":"2024-03-23T22:59:27.212105+0000","last_became_peered":"2024-03-23T22:59:27.212105+0000","last_unstale":"2024-03-23T23:00:53.279622+0000","last_undegraded":"2024-03-23T23:00:53.279622+0000","last_fullsized":"2024-03-23T23:00:53.279622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:23:37.374081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'11","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:01:36.692498+0000","last_change":"2024-03-23T22:59:26.452146+0000","last_active":"2024-03-23T23:01:36.692498+0000","last_peered":"2024-03-23T23:01:36.692498+0000","last_clean":"2024-03-23T23:01:36.692498+0000","last_became_active":"2024-03-23T22:59:26.450398+0000","last_became_peered":"2024-03-23T22:59:26.450398+0000","last_unstale":"2024-03-23T23:01:36.692498+0000","last_undegraded":"2024-03-23T23:01:36.692498+0000","last_fullsized":"2024-03-23T23:01:36.692498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:55:47.085457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.295334+0000","last_change":"2024-03-23T22:59:27.209690+0000","last_active":"2024-03-23T23:00:53.295334+0000","last_peered":"2024-03-23T23:00:53.295334+0000","last_clean":"2024-03-23T23:00:53.295334+0000","last_became_active":"2024-03-23T22:59:27.209187+0000","last_became_peered":"2024-03-23T22:59:27.209187+0000","last_unstale":"2024-03-23T23:00:53.295334+0000","last_undegraded":"2024-03-23T23:00:53.295334+0000","last_fullsized":"2024-03-23T23:00:53.295334+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:19:45.315459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.373744+0000","last_change":"2024-03-23T22:59:26.434958+0000","last_active":"2024-03-23T23:00:24.373744+0000","last_peered":"2024-03-23T23:00:24.373744+0000","last_clean":"2024-03-23T23:00:24.373744+0000","last_became_active":"2024-03-23T22:59:26.434611+0000","last_became_peered":"2024-03-23T22:59:26.434611+0000","last_unstale":"2024-03-23T23:00:24.373744+0000","last_undegraded":"2024-03-23T23:00:24.373744+0000","last_fullsized":"2024-03-23T23:00:24.373744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:42:18.201885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.300163+0000","last_change":"2024-03-23T22:59:27.211700+0000","last_active":"2024-03-23T23:00:53.300163+0000","last_peered":"2024-03-23T23:00:53.300163+0000","last_clean":"2024-03-23T23:00:53.300163+0000","last_became_active":"2024-03-23T22:59:27.211248+0000","last_became_peered":"2024-03-23T22:59:27.211248+0000","last_unstale":"2024-03-23T23:00:53.300163+0000","last_undegraded":"2024-03-23T23:00:53.300163+0000","last_fullsized":"2024-03-23T23:00:53.300163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:49.868397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356086+0000","last_change":"2024-03-23T22:59:26.442625+0000","last_active":"2024-03-23T23:00:24.356086+0000","last_peered":"2024-03-23T23:00:24.356086+0000","last_clean":"2024-03-23T23:00:24.356086+0000","last_became_active":"2024-03-23T22:59:26.442312+0000","last_became_peered":"2024-03-23T22:59:26.442312+0000","last_unstale":"2024-03-23T23:00:24.356086+0000","last_undegraded":"2024-03-23T23:00:24.356086+0000","last_fullsized":"2024-03-23T23:00:24.356086+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:27:38.808784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.282372+0000","last_change":"2024-03-23T22:59:27.187363+0000","last_active":"2024-03-23T23:00:53.282372+0000","last_peered":"2024-03-23T23:00:53.282372+0000","last_clean":"2024-03-23T23:00:53.282372+0000","last_became_active":"2024-03-23T22:59:27.187007+0000","last_became_peered":"2024-03-23T22:59:27.187007+0000","last_unstale":"2024-03-23T23:00:53.282372+0000","last_undegraded":"2024-03-23T23:00:53.282372+0000","last_fullsized":"2024-03-23T23:00:53.282372+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:41:37.041254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.369860+0000","last_change":"2024-03-23T22:59:26.445753+0000","last_active":"2024-03-23T23:00:24.369860+0000","last_peered":"2024-03-23T23:00:24.369860+0000","last_clean":"2024-03-23T23:00:24.369860+0000","last_became_active":"2024-03-23T22:59:26.445167+0000","last_became_peered":"2024-03-23T22:59:26.445167+0000","last_unstale":"2024-03-23T23:00:24.369860+0000","last_undegraded":"2024-03-23T23:00:24.369860+0000","last_fullsized":"2024-03-23T23:00:24.369860+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:46:46.058523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.281974+0000","last_change":"2024-03-23T22:59:27.203855+0000","last_active":"2024-03-23T23:00:53.281974+0000","last_peered":"2024-03-23T23:00:53.281974+0000","last_clean":"2024-03-23T23:00:53.281974+0000","last_became_active":"2024-03-23T22:59:27.203438+0000","last_became_peered":"2024-03-23T22:59:27.203438+0000","last_unstale":"2024-03-23T23:00:53.281974+0000","last_undegraded":"2024-03-23T23:00:53.281974+0000","last_fullsized":"2024-03-23T23:00:53.281974+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:28.725743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:01:40.004294+0000","last_change":"2024-03-23T22:59:26.439002+0000","last_active":"2024-03-23T23:01:40.004294+0000","last_peered":"2024-03-23T23:01:40.004294+0000","last_clean":"2024-03-23T23:01:40.004294+0000","last_became_active":"2024-03-23T22:59:26.438634+0000","last_became_peered":"2024-03-23T22:59:26.438634+0000","last_unstale":"2024-03-23T23:01:40.004294+0000","last_undegraded":"2024-03-23T23:01:40.004294+0000","last_fullsized":"2024-03-23T23:01:40.004294+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:25:02.988163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.295264+0000","last_change":"2024-03-23T22:59:27.217314+0000","last_active":"2024-03-23T23:00:53.295264+0000","last_peered":"2024-03-23T23:00:53.295264+0000","last_clean":"2024-03-23T23:00:53.295264+0000","last_became_active":"2024-03-23T22:59:27.216790+0000","last_became_peered":"2024-03-23T22:59:27.216790+0000","last_unstale":"2024-03-23T23:00:53.295264+0000","last_undegraded":"2024-03-23T23:00:53.295264+0000","last_fullsized":"2024-03-23T23:00:53.295264+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:47:13.902236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.357159+0000","last_change":"2024-03-23T22:59:26.431453+0000","last_active":"2024-03-23T23:00:24.357159+0000","last_peered":"2024-03-23T23:00:24.357159+0000","last_clean":"2024-03-23T23:00:24.357159+0000","last_became_active":"2024-03-23T22:59:26.429731+0000","last_became_peered":"2024-03-23T22:59:26.429731+0000","last_unstale":"2024-03-23T23:00:24.357159+0000","last_undegraded":"2024-03-23T23:00:24.357159+0000","last_fullsized":"2024-03-23T23:00:24.357159+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:33:47.717064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.295443+0000","last_change":"2024-03-23T22:59:27.203097+0000","last_active":"2024-03-23T23:00:53.295443+0000","last_peered":"2024-03-23T23:00:53.295443+0000","last_clean":"2024-03-23T23:00:53.295443+0000","last_became_active":"2024-03-23T22:59:27.202484+0000","last_became_peered":"2024-03-23T22:59:27.202484+0000","last_unstale":"2024-03-23T23:00:53.295443+0000","last_undegraded":"2024-03-23T23:00:53.295443+0000","last_fullsized":"2024-03-23T23:00:53.295443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:07.829866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.377539+0000","last_change":"2024-03-23T23:00:24.377180+0000","last_active":"2024-03-23T23:00:24.377539+0000","last_peered":"2024-03-23T23:00:24.377539+0000","last_clean":"2024-03-23T23:00:24.377539+0000","last_became_active":"2024-03-23T23:00:24.364530+0000","last_became_peered":"2024-03-23T23:00:24.364530+0000","last_unstale":"2024-03-23T23:00:24.377539+0000","last_undegraded":"2024-03-23T23:00:24.377539+0000","last_fullsized":"2024-03-23T23:00:24.377539+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:57.507891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.286545+0000","last_change":"2024-03-23T22:59:27.189244+0000","last_active":"2024-03-23T23:00:53.286545+0000","last_peered":"2024-03-23T23:00:53.286545+0000","last_clean":"2024-03-23T23:00:53.286545+0000","last_became_active":"2024-03-23T22:59:27.188629+0000","last_became_peered":"2024-03-23T22:59:27.188629+0000","last_unstale":"2024-03-23T23:00:53.286545+0000","last_undegraded":"2024-03-23T23:00:53.286545+0000","last_fullsized":"2024-03-23T23:00:53.286545+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:47:26.742204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.356493+0000","last_change":"2024-03-23T22:59:26.434326+0000","last_active":"2024-03-23T23:00:24.356493+0000","last_peered":"2024-03-23T23:00:24.356493+0000","last_clean":"2024-03-23T23:00:24.356493+0000","last_became_active":"2024-03-23T22:59:26.433878+0000","last_became_peered":"2024-03-23T22:59:26.433878+0000","last_unstale":"2024-03-23T23:00:24.356493+0000","last_undegraded":"2024-03-23T23:00:24.356493+0000","last_fullsized":"2024-03-23T23:00:24.356493+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:08.580572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.297139+0000","last_change":"2024-03-23T22:59:27.212909+0000","last_active":"2024-03-23T23:00:53.297139+0000","last_peered":"2024-03-23T23:00:53.297139+0000","last_clean":"2024-03-23T23:00:53.297139+0000","last_became_active":"2024-03-23T22:59:27.212504+0000","last_became_peered":"2024-03-23T22:59:27.212504+0000","last_unstale":"2024-03-23T23:00:53.297139+0000","last_undegraded":"2024-03-23T23:00:53.297139+0000","last_fullsized":"2024-03-23T23:00:53.297139+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:09.238272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:24.380290+0000","last_change":"2024-03-23T22:59:26.452577+0000","last_active":"2024-03-23T23:00:24.380290+0000","last_peered":"2024-03-23T23:00:24.380290+0000","last_clean":"2024-03-23T23:00:24.380290+0000","last_became_active":"2024-03-23T22:59:26.452107+0000","last_became_peered":"2024-03-23T22:59:26.452107+0000","last_unstale":"2024-03-23T23:00:24.380290+0000","last_undegraded":"2024-03-23T23:00:24.380290+0000","last_fullsized":"2024-03-23T23:00:24.380290+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:25.401134+0000","last_clean_scrub_stamp":"2024-03-23T22:59:25.401134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:32:18.472334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.283849+0000","last_change":"2024-03-23T22:59:27.188539+0000","last_active":"2024-03-23T23:00:53.283849+0000","last_peered":"2024-03-23T23:00:53.283849+0000","last_clean":"2024-03-23T23:00:53.283849+0000","last_became_active":"2024-03-23T22:59:27.188184+0000","last_became_peered":"2024-03-23T22:59:27.188184+0000","last_unstale":"2024-03-23T23:00:53.283849+0000","last_undegraded":"2024-03-23T23:00:53.283849+0000","last_fullsized":"2024-03-23T23:00:53.283849+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:37:59.756492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.269022+0000","last_change":"2024-03-23T22:59:27.192083+0000","last_active":"2024-03-23T23:00:53.269022+0000","last_peered":"2024-03-23T23:00:53.269022+0000","last_clean":"2024-03-23T23:00:53.269022+0000","last_became_active":"2024-03-23T22:59:27.191709+0000","last_became_peered":"2024-03-23T22:59:27.191709+0000","last_unstale":"2024-03-23T23:00:53.269022+0000","last_undegraded":"2024-03-23T23:00:53.269022+0000","last_fullsized":"2024-03-23T23:00:53.269022+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:18:42.905177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.275138+0000","last_change":"2024-03-23T22:59:27.186315+0000","last_active":"2024-03-23T23:00:53.275138+0000","last_peered":"2024-03-23T23:00:53.275138+0000","last_clean":"2024-03-23T23:00:53.275138+0000","last_became_active":"2024-03-23T22:59:27.185960+0000","last_became_peered":"2024-03-23T22:59:27.185960+0000","last_unstale":"2024-03-23T23:00:53.275138+0000","last_undegraded":"2024-03-23T23:00:53.275138+0000","last_fullsized":"2024-03-23T23:00:53.275138+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:57:07.739725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.291985+0000","last_change":"2024-03-23T22:59:27.221696+0000","last_active":"2024-03-23T23:00:53.291985+0000","last_peered":"2024-03-23T23:00:53.291985+0000","last_clean":"2024-03-23T23:00:53.291985+0000","last_became_active":"2024-03-23T22:59:27.221400+0000","last_became_peered":"2024-03-23T22:59:27.221400+0000","last_unstale":"2024-03-23T23:00:53.291985+0000","last_undegraded":"2024-03-23T23:00:53.291985+0000","last_fullsized":"2024-03-23T23:00:53.291985+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:17.905737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.287320+0000","last_change":"2024-03-23T22:59:27.206286+0000","last_active":"2024-03-23T23:00:53.287320+0000","last_peered":"2024-03-23T23:00:53.287320+0000","last_clean":"2024-03-23T23:00:53.287320+0000","last_became_active":"2024-03-23T22:59:27.205854+0000","last_became_peered":"2024-03-23T22:59:27.205854+0000","last_unstale":"2024-03-23T23:00:53.287320+0000","last_undegraded":"2024-03-23T23:00:53.287320+0000","last_fullsized":"2024-03-23T23:00:53.287320+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:00:45.216211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.299730+0000","last_change":"2024-03-23T22:59:27.211376+0000","last_active":"2024-03-23T23:00:53.299730+0000","last_peered":"2024-03-23T23:00:53.299730+0000","last_clean":"2024-03-23T23:00:53.299730+0000","last_became_active":"2024-03-23T22:59:27.211061+0000","last_became_peered":"2024-03-23T22:59:27.211061+0000","last_unstale":"2024-03-23T23:00:53.299730+0000","last_undegraded":"2024-03-23T23:00:53.299730+0000","last_fullsized":"2024-03-23T23:00:53.299730+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:53:25.776927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.268390+0000","last_change":"2024-03-23T22:59:27.191028+0000","last_active":"2024-03-23T23:00:53.268390+0000","last_peered":"2024-03-23T23:00:53.268390+0000","last_clean":"2024-03-23T23:00:53.268390+0000","last_became_active":"2024-03-23T22:59:27.190634+0000","last_became_peered":"2024-03-23T22:59:27.190634+0000","last_unstale":"2024-03-23T23:00:53.268390+0000","last_undegraded":"2024-03-23T23:00:53.268390+0000","last_fullsized":"2024-03-23T23:00:53.268390+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:50:52.110636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.289464+0000","last_change":"2024-03-23T22:59:27.183235+0000","last_active":"2024-03-23T23:00:53.289464+0000","last_peered":"2024-03-23T23:00:53.289464+0000","last_clean":"2024-03-23T23:00:53.289464+0000","last_became_active":"2024-03-23T22:59:27.181682+0000","last_became_peered":"2024-03-23T22:59:27.181682+0000","last_unstale":"2024-03-23T23:00:53.289464+0000","last_undegraded":"2024-03-23T23:00:53.289464+0000","last_fullsized":"2024-03-23T23:00:53.289464+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:33:26.002324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.284323+0000","last_change":"2024-03-23T23:00:24.378143+0000","last_active":"2024-03-23T23:00:53.284323+0000","last_peered":"2024-03-23T23:00:53.284323+0000","last_clean":"2024-03-23T23:00:53.284323+0000","last_became_active":"2024-03-23T23:00:24.377660+0000","last_became_peered":"2024-03-23T23:00:24.377660+0000","last_unstale":"2024-03-23T23:00:53.284323+0000","last_undegraded":"2024-03-23T23:00:53.284323+0000","last_fullsized":"2024-03-23T23:00:53.284323+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:24:29.250679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.284520+0000","last_change":"2024-03-23T22:59:27.215230+0000","last_active":"2024-03-23T23:00:53.284520+0000","last_peered":"2024-03-23T23:00:53.284520+0000","last_clean":"2024-03-23T23:00:53.284520+0000","last_became_active":"2024-03-23T22:59:27.214881+0000","last_became_peered":"2024-03-23T22:59:27.214881+0000","last_unstale":"2024-03-23T23:00:53.284520+0000","last_undegraded":"2024-03-23T23:00:53.284520+0000","last_fullsized":"2024-03-23T23:00:53.284520+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:10:40.428481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.296915+0000","last_change":"2024-03-23T22:59:27.213454+0000","last_active":"2024-03-23T23:00:53.296915+0000","last_peered":"2024-03-23T23:00:53.296915+0000","last_clean":"2024-03-23T23:00:53.296915+0000","last_became_active":"2024-03-23T22:59:27.212684+0000","last_became_peered":"2024-03-23T22:59:27.212684+0000","last_unstale":"2024-03-23T23:00:53.296915+0000","last_undegraded":"2024-03-23T23:00:53.296915+0000","last_fullsized":"2024-03-23T23:00:53.296915+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:48:51.584892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.287158+0000","last_change":"2024-03-23T22:59:27.220414+0000","last_active":"2024-03-23T23:00:53.287158+0000","last_peered":"2024-03-23T23:00:53.287158+0000","last_clean":"2024-03-23T23:00:53.287158+0000","last_became_active":"2024-03-23T22:59:27.220020+0000","last_became_peered":"2024-03-23T22:59:27.220020+0000","last_unstale":"2024-03-23T23:00:53.287158+0000","last_undegraded":"2024-03-23T23:00:53.287158+0000","last_fullsized":"2024-03-23T23:00:53.287158+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:35:48.890004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.287204+0000","last_change":"2024-03-23T22:59:27.215648+0000","last_active":"2024-03-23T23:00:53.287204+0000","last_peered":"2024-03-23T23:00:53.287204+0000","last_clean":"2024-03-23T23:00:53.287204+0000","last_became_active":"2024-03-23T22:59:27.212252+0000","last_became_peered":"2024-03-23T22:59:27.212252+0000","last_unstale":"2024-03-23T23:00:53.287204+0000","last_undegraded":"2024-03-23T23:00:53.287204+0000","last_fullsized":"2024-03-23T23:00:53.287204+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:08:38.956508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T23:00:53.274679+0000","last_change":"2024-03-23T22:59:27.190527+0000","last_active":"2024-03-23T23:00:53.274679+0000","last_peered":"2024-03-23T23:00:53.274679+0000","last_clean":"2024-03-23T23:00:53.274679+0000","last_became_active":"2024-03-23T22:59:27.190026+0000","last_became_peered":"2024-03-23T22:59:27.190026+0000","last_unstale":"2024-03-23T23:00:53.274679+0000","last_undegraded":"2024-03-23T23:00:53.274679+0000","last_fullsized":"2024-03-23T23:00:53.274679+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T22:59:26.403507+0000","last_clean_scrub_stamp":"2024-03-23T22:59:26.403507+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:31:24.663584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247269,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68250,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":868,"num_read_kb":507,"num_write":170,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":196122,"data_compressed":5619,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351019,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":459288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612656,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":1115103,"data_compressed":37666,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776182,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":459524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037821,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":459444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299457,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":1051289,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462983,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":954368,"data_stored":458913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":519308,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":460184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411423,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":519800,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149785,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":459224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888146,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":459350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724621,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":1050973,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34152,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34152,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35793,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":60602,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":61094,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":65912,"data_compressed":1873,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T23:01:55.320 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T23:01:55.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.589+0000 7f19c3000640 1 -- 172.21.15.12:0/1906652334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19bc076060 msgr2=0x7f19bc076480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:55.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.589+0000 7f19c3000640 1 --2- 172.21.15.12:0/1906652334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19bc076060 0x7f19bc076480 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f19ac009860 tx=0x7f19ac02bb70 comp rx=0 tx=0).stop 2024-03-23T23:01:55.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.589+0000 7f19c3000640 1 -- 172.21.15.12:0/1906652334 shutdown_connections 2024-03-23T23:01:55.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.589+0000 7f19c3000640 1 --2- 172.21.15.12:0/1906652334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19bc077c40 0x7f19bc0784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:55.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.589+0000 7f19c3000640 1 --2- 172.21.15.12:0/1906652334 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19bc0772a0 0x7f19bc077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:55.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.589+0000 7f19c3000640 1 --2- 172.21.15.12:0/1906652334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19bc076060 0x7f19bc076480 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:55.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.589+0000 7f19c3000640 1 -- 172.21.15.12:0/1906652334 >> 172.21.15.12:0/1906652334 conn(0x7f19bc0fee50 msgr2=0x7f19bc101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:55.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.589+0000 7f19c3000640 1 -- 172.21.15.12:0/1906652334 shutdown_connections 2024-03-23T23:01:55.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.589+0000 7f19c3000640 1 -- 172.21.15.12:0/1906652334 wait complete. 2024-03-23T23:01:55.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.590+0000 7f19c3000640 1 Processor -- start 2024-03-23T23:01:55.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.590+0000 7f19c3000640 1 -- start start 2024-03-23T23:01:55.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.590+0000 7f19c3000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19bc076060 0x7f19bc19dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:55.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.590+0000 7f19c3000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19bc0772a0 0x7f19bc19e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:55.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.591+0000 7f19c3000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19bc077c40 0x7f19bc1a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:55.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.591+0000 7f19c3000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc113960 con 0x7f19bc0772a0 2024-03-23T23:01:55.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.591+0000 7f19c3000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc1137e0 con 0x7f19bc076060 2024-03-23T23:01:55.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.591+0000 7f19c3000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc113ae0 con 0x7f19bc077c40 2024-03-23T23:01:55.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.591+0000 7f19bb400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19bc0772a0 0x7f19bc19e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:55.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.591+0000 7f19bbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19bc076060 0x7f19bc19dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:55.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.591+0000 7f19bbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19bc076060 0x7f19bc19dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36492/0 (socket says 172.21.15.12:36492) 2024-03-23T23:01:55.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.591+0000 7f19c1000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19bc077c40 0x7f19bc1a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:55.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.591+0000 7f19bbe00640 1 -- 172.21.15.12:0/1342413705 learned_addr learned my addr 172.21.15.12:0/1342413705 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:55.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.591+0000 7f19bbe00640 1 -- 172.21.15.12:0/1342413705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19bc077c40 msgr2=0x7f19bc1a5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:55.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.591+0000 7f19bbe00640 1 --2- 172.21.15.12:0/1342413705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19bc077c40 0x7f19bc1a5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:55.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.591+0000 7f19bbe00640 1 -- 172.21.15.12:0/1342413705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19bc0772a0 msgr2=0x7f19bc19e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:55.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.591+0000 7f19bbe00640 1 --2- 172.21.15.12:0/1342413705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19bc0772a0 0x7f19bc19e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:55.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.591+0000 7f19bbe00640 1 -- 172.21.15.12:0/1342413705 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19bc1a5bc0 con 0x7f19bc076060 2024-03-23T23:01:55.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.592+0000 7f19bbe00640 1 --2- 172.21.15.12:0/1342413705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19bc076060 0x7f19bc19dfa0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f19ac0349e0 tx=0x7f19ac0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:55.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.592+0000 7f19b8c00640 1 -- 172.21.15.12:0/1342413705 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac041030 con 0x7f19bc076060 2024-03-23T23:01:55.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.592+0000 7f19b8c00640 1 -- 172.21.15.12:0/1342413705 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19ac0347b0 con 0x7f19bc076060 2024-03-23T23:01:55.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.592+0000 7f19c3000640 1 -- 172.21.15.12:0/1342413705 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19bc1a5e50 con 0x7f19bc076060 2024-03-23T23:01:55.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.592+0000 7f19b8c00640 1 -- 172.21.15.12:0/1342413705 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac039040 con 0x7f19bc076060 2024-03-23T23:01:55.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.592+0000 7f19c3000640 1 -- 172.21.15.12:0/1342413705 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19bc1a6260 con 0x7f19bc076060 2024-03-23T23:01:55.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.593+0000 7f19c3000640 1 -- 172.21.15.12:0/1342413705 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19bc110400 con 0x7f19bc076060 2024-03-23T23:01:55.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.595+0000 7f19b8c00640 1 -- 172.21.15.12:0/1342413705 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f19ac044070 con 0x7f19bc076060 2024-03-23T23:01:55.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.597+0000 7f19b8c00640 1 --2- 172.21.15.12:0/1342413705 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f19a00a2800 0x7f19a00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:55.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.597+0000 7f19b8c00640 1 -- 172.21.15.12:0/1342413705 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f19ac0f0090 con 0x7f19bc076060 2024-03-23T23:01:55.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.597+0000 7f19bb400640 1 --2- 172.21.15.12:0/1342413705 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f19a00a2800 0x7f19a00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:55.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.598+0000 7f19bb400640 1 --2- 172.21.15.12:0/1342413705 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f19a00a2800 0x7f19a00a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f19bc19f4b0 tx=0x7f19a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:55.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.601+0000 7f19b8c00640 1 -- 172.21.15.12:0/1342413705 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f19ac0bd0e0 con 0x7f19bc076060 2024-03-23T23:01:55.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.812+0000 7f19c3000640 1 -- 172.21.15.12:0/1342413705 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19bc0623e0 con 0x7f19bc076060 2024-03-23T23:01:55.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.814+0000 7f19b8c00640 1 -- 172.21.15.12:0/1342413705 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22695 (secure 0 0 0) 0x7f19ac0bc9e0 con 0x7f19bc076060 2024-03-23T23:01:55.815 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:01:55.815 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":88,"fsid":"90f679ae-e967-11ee-95cd-87774f69a715","created":"2024-03-23T22:50:14.906952+0000","modified":"2024-03-23T23:00:24.349052+0000","last_up_change":"2024-03-23T22:58:28.907824+0000","last_in_change":"2024-03-23T22:58:08.342487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T22:54:11.831010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T22:59:25.350974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T22:59:26.390323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b70875d7-a23d-4251-9e8b-1a16d4edc2a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6803","nonce":1190079611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6805","nonce":1190079611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6809","nonce":1190079611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6807","nonce":1190079611}]},"public_addr":"172.21.15.12:6803/1190079611","cluster_addr":"172.21.15.12:6805/1190079611","heartbeat_back_addr":"172.21.15.12:6809/1190079611","heartbeat_front_addr":"172.21.15.12:6807/1190079611","state":["exists","up"]},{"osd":1,"uuid":"416e87af-db5d-41ce-84d9-731e3e07e3c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6801","nonce":1983639525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6803","nonce":1983639525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6807","nonce":1983639525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6805","nonce":1983639525}]},"public_addr":"172.21.15.173:6801/1983639525","cluster_addr":"172.21.15.173:6803/1983639525","heartbeat_back_addr":"172.21.15.173:6807/1983639525","heartbeat_front_addr":"172.21.15.173:6805/1983639525","state":["exists","up"]},{"osd":2,"uuid":"7a559c70-4c54-431b-8baf-4136b6e76493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6801","nonce":2253997387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6803","nonce":2253997387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6807","nonce":2253997387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6805","nonce":2253997387}]},"public_addr":"172.21.15.203:6801/2253997387","cluster_addr":"172.21.15.203:6803/2253997387","heartbeat_back_addr":"172.21.15.203:6807/2253997387","heartbeat_front_addr":"172.21.15.203:6805/2253997387","state":["exists","up"]},{"osd":3,"uuid":"c41cd287-a674-486c-8025-328cd36e5ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6811","nonce":1761446905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6813","nonce":1761446905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6817","nonce":1761446905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6815","nonce":1761446905}]},"public_addr":"172.21.15.12:6811/1761446905","cluster_addr":"172.21.15.12:6813/1761446905","heartbeat_back_addr":"172.21.15.12:6817/1761446905","heartbeat_front_addr":"172.21.15.12:6815/1761446905","state":["exists","up"]},{"osd":4,"uuid":"3c48f886-0b42-4a09-b31e-abccc200bb1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6809","nonce":242958272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6811","nonce":242958272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6815","nonce":242958272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6813","nonce":242958272}]},"public_addr":"172.21.15.173:6809/242958272","cluster_addr":"172.21.15.173:6811/242958272","heartbeat_back_addr":"172.21.15.173:6815/242958272","heartbeat_front_addr":"172.21.15.173:6813/242958272","state":["exists","up"]},{"osd":5,"uuid":"4a34f06f-8805-40c0-b1ab-23407b0276d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6809","nonce":4270966092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6811","nonce":4270966092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6815","nonce":4270966092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6813","nonce":4270966092}]},"public_addr":"172.21.15.203:6809/4270966092","cluster_addr":"172.21.15.203:6811/4270966092","heartbeat_back_addr":"172.21.15.203:6815/4270966092","heartbeat_front_addr":"172.21.15.203:6813/4270966092","state":["exists","up"]},{"osd":6,"uuid":"26ddc663-5623-48c9-80d1-f8b968765d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6819","nonce":3606839295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6821","nonce":3606839295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6825","nonce":3606839295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6823","nonce":3606839295}]},"public_addr":"172.21.15.12:6819/3606839295","cluster_addr":"172.21.15.12:6821/3606839295","heartbeat_back_addr":"172.21.15.12:6825/3606839295","heartbeat_front_addr":"172.21.15.12:6823/3606839295","state":["exists","up"]},{"osd":7,"uuid":"17dc9417-71cd-4576-aa08-933212ec55e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6817","nonce":1241504346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6819","nonce":1241504346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6823","nonce":1241504346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6821","nonce":1241504346}]},"public_addr":"172.21.15.173:6817/1241504346","cluster_addr":"172.21.15.173:6819/1241504346","heartbeat_back_addr":"172.21.15.173:6823/1241504346","heartbeat_front_addr":"172.21.15.173:6821/1241504346","state":["exists","up"]},{"osd":8,"uuid":"90a2b63a-471d-43b8-ac18-bb1f16778d18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6817","nonce":1516996192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6819","nonce":1516996192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6823","nonce":1516996192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6821","nonce":1516996192}]},"public_addr":"172.21.15.203:6817/1516996192","cluster_addr":"172.21.15.203:6819/1516996192","heartbeat_back_addr":"172.21.15.203:6823/1516996192","heartbeat_front_addr":"172.21.15.203:6821/1516996192","state":["exists","up"]},{"osd":9,"uuid":"6e1ca504-40b4-49d0-b245-7946181abfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6827","nonce":1175260774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6829","nonce":1175260774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6833","nonce":1175260774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6831","nonce":1175260774}]},"public_addr":"172.21.15.12:6827/1175260774","cluster_addr":"172.21.15.12:6829/1175260774","heartbeat_back_addr":"172.21.15.12:6833/1175260774","heartbeat_front_addr":"172.21.15.12:6831/1175260774","state":["exists","up"]},{"osd":10,"uuid":"09a595ed-5614-4386-8e61-0170af66e01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6825","nonce":3303506808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6827","nonce":3303506808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6831","nonce":3303506808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6829","nonce":3303506808}]},"public_addr":"172.21.15.173:6825/3303506808","cluster_addr":"172.21.15.173:6827/3303506808","heartbeat_back_addr":"172.21.15.173:6831/3303506808","heartbeat_front_addr":"172.21.15.173:6829/3303506808","state":["exists","up"]},{"osd":11,"uuid":"9ef1d595-a977-4ba7-9c6b-d2b3295f20b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6825","nonce":2323604506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6827","nonce":2323604506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6831","nonce":2323604506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6829","nonce":2323604506}]},"public_addr":"172.21.15.203:6825/2323604506","cluster_addr":"172.21.15.203:6827/2323604506","heartbeat_back_addr":"172.21.15.203:6831/2323604506","heartbeat_front_addr":"172.21.15.203:6829/2323604506","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:18.469038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:42.752628+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:07.069493+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:41.962478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:08.363104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:33.341336+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:07.286541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:34.243302+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:59.749699+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:57:33.822913+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:00.737269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:26.599854+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/4182783120":"2024-03-24T22:51:23.113201+0000","172.21.15.12:6800/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1900938637":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/3414861652":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6801/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/2580541477":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6800/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/501058839":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1192554716":"2024-03-24T22:50:48.401223+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T23:01:55.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.817+0000 7f19c3000640 1 -- 172.21.15.12:0/1342413705 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f19a00a2800 msgr2=0x7f19a00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:55.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.817+0000 7f19c3000640 1 --2- 172.21.15.12:0/1342413705 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f19a00a2800 0x7f19a00a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f19bc19f4b0 tx=0x7f19a8009260 comp rx=0 tx=0).stop 2024-03-23T23:01:55.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.817+0000 7f19c3000640 1 -- 172.21.15.12:0/1342413705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19bc076060 msgr2=0x7f19bc19dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:55.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.817+0000 7f19c3000640 1 --2- 172.21.15.12:0/1342413705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19bc076060 0x7f19bc19dfa0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f19ac0349e0 tx=0x7f19ac0042f0 comp rx=0 tx=0).stop 2024-03-23T23:01:55.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.817+0000 7f19c3000640 1 -- 172.21.15.12:0/1342413705 shutdown_connections 2024-03-23T23:01:55.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.817+0000 7f19c3000640 1 --2- 172.21.15.12:0/1342413705 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f19a00a2800 0x7f19a00a4cc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:55.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.817+0000 7f19c3000640 1 --2- 172.21.15.12:0/1342413705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19bc077c40 0x7f19bc1a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:55.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.817+0000 7f19c3000640 1 --2- 172.21.15.12:0/1342413705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19bc0772a0 0x7f19bc19e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:55.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.817+0000 7f19c3000640 1 --2- 172.21.15.12:0/1342413705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19bc076060 0x7f19bc19dfa0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:55.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.817+0000 7f19c3000640 1 -- 172.21.15.12:0/1342413705 >> 172.21.15.12:0/1342413705 conn(0x7f19bc0fee50 msgr2=0x7f19bc100580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:55.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.817+0000 7f19c3000640 1 -- 172.21.15.12:0/1342413705 shutdown_connections 2024-03-23T23:01:55.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:55.818+0000 7f19c3000640 1 -- 172.21.15.12:0/1342413705 wait complete. 2024-03-23T23:01:55.839 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T22:54:11.831010+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T22:59:25.350974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T22:59:26.390323+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T23:01:55.840 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T23:01:56.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.065+0000 7f394b200640 1 -- 172.21.15.12:0/1034022924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39441028a0 msgr2=0x7f3944102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:56.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.065+0000 7f394b200640 1 --2- 172.21.15.12:0/1034022924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39441028a0 0x7f3944102d20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3930009900 tx=0x7f393002bc40 comp rx=0 tx=0).stop 2024-03-23T23:01:56.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.065+0000 7f394b200640 1 -- 172.21.15.12:0/1034022924 shutdown_connections 2024-03-23T23:01:56.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.065+0000 7f394b200640 1 --2- 172.21.15.12:0/1034022924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3944103260 0x7f3944109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:56.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.065+0000 7f394b200640 1 --2- 172.21.15.12:0/1034022924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39441028a0 0x7f3944102d20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:56.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.065+0000 7f394b200640 1 --2- 172.21.15.12:0/1034022924 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39441016b0 0x7f3944101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:56.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.065+0000 7f394b200640 1 -- 172.21.15.12:0/1034022924 >> 172.21.15.12:0/1034022924 conn(0x7f39440fcc40 msgr2=0x7f39440ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:56.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.065+0000 7f394b200640 1 -- 172.21.15.12:0/1034022924 shutdown_connections 2024-03-23T23:01:56.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.065+0000 7f394b200640 1 -- 172.21.15.12:0/1034022924 wait complete. 2024-03-23T23:01:56.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.066+0000 7f394b200640 1 Processor -- start 2024-03-23T23:01:56.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.066+0000 7f394b200640 1 -- start start 2024-03-23T23:01:56.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.066+0000 7f394b200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39441016b0 0x7f3944199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:56.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.067+0000 7f394b200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39441028a0 0x7f394419a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:56.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.067+0000 7f394b200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3944103260 0x7f39441a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:56.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.067+0000 7f394b200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394410d360 con 0x7f39441016b0 2024-03-23T23:01:56.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.067+0000 7f394b200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394410d1e0 con 0x7f3944103260 2024-03-23T23:01:56.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.067+0000 7f394b200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394410d4e0 con 0x7f39441028a0 2024-03-23T23:01:56.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.067+0000 7f3943400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39441028a0 0x7f394419a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:56.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.067+0000 7f3943e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39441016b0 0x7f3944199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:56.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.067+0000 7f3949200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3944103260 0x7f39441a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:56.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.067+0000 7f3943400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39441028a0 0x7f394419a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:37172/0 (socket says 172.21.15.12:37172) 2024-03-23T23:01:56.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.067+0000 7f3943400640 1 -- 172.21.15.12:0/145160981 learned_addr learned my addr 172.21.15.12:0/145160981 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:56.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.067+0000 7f3949200640 1 -- 172.21.15.12:0/145160981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39441028a0 msgr2=0x7f394419a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:56.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.067+0000 7f3949200640 1 --2- 172.21.15.12:0/145160981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39441028a0 0x7f394419a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:56.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.067+0000 7f3949200640 1 -- 172.21.15.12:0/145160981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39441016b0 msgr2=0x7f3944199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:56.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.067+0000 7f3949200640 1 --2- 172.21.15.12:0/145160981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39441016b0 0x7f3944199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:56.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.067+0000 7f3949200640 1 -- 172.21.15.12:0/145160981 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39441a1810 con 0x7f3944103260 2024-03-23T23:01:56.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.067+0000 7f3943e00640 1 --2- 172.21.15.12:0/145160981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39441016b0 0x7f3944199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:56.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.067+0000 7f3943400640 1 --2- 172.21.15.12:0/145160981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39441028a0 0x7f394419a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:56.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.068+0000 7f3949200640 1 --2- 172.21.15.12:0/145160981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3944103260 0x7f39441a11a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f393c00e970 tx=0x7f393c00ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:56.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.068+0000 7f3940c00640 1 -- 172.21.15.12:0/145160981 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c00ccc0 con 0x7f3944103260 2024-03-23T23:01:56.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.068+0000 7f3940c00640 1 -- 172.21.15.12:0/145160981 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f393c005020 con 0x7f3944103260 2024-03-23T23:01:56.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.068+0000 7f394b200640 1 -- 172.21.15.12:0/145160981 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39441a1b00 con 0x7f3944103260 2024-03-23T23:01:56.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.068+0000 7f3940c00640 1 -- 172.21.15.12:0/145160981 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c004a00 con 0x7f3944103260 2024-03-23T23:01:56.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.068+0000 7f394b200640 1 -- 172.21.15.12:0/145160981 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39441a1fd0 con 0x7f3944103260 2024-03-23T23:01:56.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.070+0000 7f394b200640 1 -- 172.21.15.12:0/145160981 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39441a2260 con 0x7f3944103260 2024-03-23T23:01:56.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.071+0000 7f3940c00640 1 -- 172.21.15.12:0/145160981 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f393c0051d0 con 0x7f3944103260 2024-03-23T23:01:56.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.072+0000 7f3940c00640 1 --2- 172.21.15.12:0/145160981 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f39280a2800 0x7f39280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:56.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.073+0000 7f3940c00640 1 -- 172.21.15.12:0/145160981 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f393c0cea20 con 0x7f3944103260 2024-03-23T23:01:56.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.073+0000 7f3943e00640 1 --2- 172.21.15.12:0/145160981 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f39280a2800 0x7f39280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:56.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.073+0000 7f3943e00640 1 --2- 172.21.15.12:0/145160981 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f39280a2800 0x7f39280a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f39340099a0 tx=0x7f3934009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:56.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.076+0000 7f3940c00640 1 -- 172.21.15.12:0/145160981 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f393c09ba70 con 0x7f3944103260 2024-03-23T23:01:56.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:56 smithi012 ceph-mon[31695]: from='client.25268 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T23:01:56.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:56 smithi012 ceph-mon[31695]: from='client.25255 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T23:01:56.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:56 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:56.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:56 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:56.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:56 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:56.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:56 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:56.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:56 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1342413705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:56.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.332+0000 7f394b200640 1 -- 172.21.15.12:0/145160981 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f39441a2430 con 0x7f3944103260 2024-03-23T23:01:56.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.333+0000 7f3940c00640 1 -- 172.21.15.12:0/145160981 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f393c09b370 con 0x7f3944103260 2024-03-23T23:01:56.333 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-23T23:01:56.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.335+0000 7f394b200640 1 -- 172.21.15.12:0/145160981 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f39280a2800 msgr2=0x7f39280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:56.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.335+0000 7f394b200640 1 --2- 172.21.15.12:0/145160981 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f39280a2800 0x7f39280a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f39340099a0 tx=0x7f3934009500 comp rx=0 tx=0).stop 2024-03-23T23:01:56.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.335+0000 7f394b200640 1 -- 172.21.15.12:0/145160981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3944103260 msgr2=0x7f39441a11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:56.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.335+0000 7f394b200640 1 --2- 172.21.15.12:0/145160981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3944103260 0x7f39441a11a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f393c00e970 tx=0x7f393c00ee30 comp rx=0 tx=0).stop 2024-03-23T23:01:56.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.335+0000 7f394b200640 1 -- 172.21.15.12:0/145160981 shutdown_connections 2024-03-23T23:01:56.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.335+0000 7f394b200640 1 --2- 172.21.15.12:0/145160981 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f39280a2800 0x7f39280a4cc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:56.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.335+0000 7f394b200640 1 --2- 172.21.15.12:0/145160981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3944103260 0x7f39441a11a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:56.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.335+0000 7f394b200640 1 --2- 172.21.15.12:0/145160981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39441028a0 0x7f394419a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:56.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.335+0000 7f394b200640 1 --2- 172.21.15.12:0/145160981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39441016b0 0x7f3944199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:56.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.335+0000 7f394b200640 1 -- 172.21.15.12:0/145160981 >> 172.21.15.12:0/145160981 conn(0x7f39440fcc40 msgr2=0x7f39440738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:56.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.336+0000 7f394b200640 1 -- 172.21.15.12:0/145160981 shutdown_connections 2024-03-23T23:01:56.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.336+0000 7f394b200640 1 -- 172.21.15.12:0/145160981 wait complete. 2024-03-23T23:01:56.358 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T23:01:56.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:56 smithi173 ceph-mon[36116]: from='client.25268 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T23:01:56.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:56 smithi173 ceph-mon[36116]: from='client.25255 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T23:01:56.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:56 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:56.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:56 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:56.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:56 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:56.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:56 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:56.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:56 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1342413705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:56.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:56 smithi203 ceph-mon[36693]: from='client.25268 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T23:01:56.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:56 smithi203 ceph-mon[36693]: from='client.25255 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T23:01:56.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:56 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:56.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:56 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:56.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:56 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:56.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:56 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:56.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:56 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1342413705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:56.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.574+0000 7f522ca00640 1 -- 172.21.15.12:0/4097437163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5228105110 msgr2=0x7f5228105590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:56.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.574+0000 7f522ca00640 1 --2- 172.21.15.12:0/4097437163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5228105110 0x7f5228105590 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5220009ca0 tx=0x7f522002c0f0 comp rx=0 tx=0).stop 2024-03-23T23:01:56.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.574+0000 7f522ca00640 1 -- 172.21.15.12:0/4097437163 shutdown_connections 2024-03-23T23:01:56.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.574+0000 7f522ca00640 1 --2- 172.21.15.12:0/4097437163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5228105110 0x7f5228105590 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:56.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.574+0000 7f522ca00640 1 --2- 172.21.15.12:0/4097437163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5228102760 0x7f5228104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:56.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.574+0000 7f522ca00640 1 --2- 172.21.15.12:0/4097437163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52280ffdf0 0x7f5228102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:56.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.574+0000 7f522ca00640 1 -- 172.21.15.12:0/4097437163 >> 172.21.15.12:0/4097437163 conn(0x7f52280fba10 msgr2=0x7f52280fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:56.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.574+0000 7f522ca00640 1 -- 172.21.15.12:0/4097437163 shutdown_connections 2024-03-23T23:01:56.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.574+0000 7f522ca00640 1 -- 172.21.15.12:0/4097437163 wait complete. 2024-03-23T23:01:56.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.575+0000 7f522ca00640 1 Processor -- start 2024-03-23T23:01:56.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.575+0000 7f522ca00640 1 -- start start 2024-03-23T23:01:56.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.575+0000 7f522ca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52280ffdf0 0x7f522819ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:56.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.575+0000 7f522ca00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5228102760 0x7f522819e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:56.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.575+0000 7f522ca00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5228105110 0x7f52281a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:56.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.575+0000 7f522ca00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52281137d0 con 0x7f5228102760 2024-03-23T23:01:56.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.575+0000 7f522ca00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5228113650 con 0x7f52280ffdf0 2024-03-23T23:01:56.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.575+0000 7f522ca00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5228113950 con 0x7f5228105110 2024-03-23T23:01:56.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.576+0000 7f5226800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52280ffdf0 0x7f522819ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:56.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.576+0000 7f5226800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52280ffdf0 0x7f522819ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36512/0 (socket says 172.21.15.12:36512) 2024-03-23T23:01:56.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.576+0000 7f5227200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5228105110 0x7f52281a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:56.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.576+0000 7f5226800640 1 -- 172.21.15.12:0/834164362 learned_addr learned my addr 172.21.15.12:0/834164362 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:56.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.576+0000 7f5225e00640 1 --2- 172.21.15.12:0/834164362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5228102760 0x7f522819e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:56.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.576+0000 7f5226800640 1 -- 172.21.15.12:0/834164362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5228105110 msgr2=0x7f52281a53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:56.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.576+0000 7f5226800640 1 --2- 172.21.15.12:0/834164362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5228105110 0x7f52281a53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:56.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.576+0000 7f5226800640 1 -- 172.21.15.12:0/834164362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5228102760 msgr2=0x7f522819e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:56.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.576+0000 7f5226800640 1 --2- 172.21.15.12:0/834164362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5228102760 0x7f522819e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:56.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.576+0000 7f5226800640 1 -- 172.21.15.12:0/834164362 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52281a5a10 con 0x7f52280ffdf0 2024-03-23T23:01:56.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.576+0000 7f5226800640 1 --2- 172.21.15.12:0/834164362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52280ffdf0 0x7f522819ddf0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f521800e9f0 tx=0x7f521800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:56.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.577+0000 7f521f400640 1 -- 172.21.15.12:0/834164362 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521800cd80 con 0x7f52280ffdf0 2024-03-23T23:01:56.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.577+0000 7f521f400640 1 -- 172.21.15.12:0/834164362 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5218010910 con 0x7f52280ffdf0 2024-03-23T23:01:56.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.577+0000 7f521f400640 1 -- 172.21.15.12:0/834164362 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5218018990 con 0x7f52280ffdf0 2024-03-23T23:01:56.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.577+0000 7f522ca00640 1 -- 172.21.15.12:0/834164362 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52281a5d00 con 0x7f52280ffdf0 2024-03-23T23:01:56.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.577+0000 7f522ca00640 1 -- 172.21.15.12:0/834164362 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52281a61a0 con 0x7f52280ffdf0 2024-03-23T23:01:56.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.578+0000 7f522ca00640 1 -- 172.21.15.12:0/834164362 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52280fd500 con 0x7f52280ffdf0 2024-03-23T23:01:56.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.580+0000 7f521f400640 1 -- 172.21.15.12:0/834164362 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5218010ac0 con 0x7f52280ffdf0 2024-03-23T23:01:56.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.581+0000 7f521f400640 1 --2- 172.21.15.12:0/834164362 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f520c0a2880 0x7f520c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:56.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.581+0000 7f521f400640 1 -- 172.21.15.12:0/834164362 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f52180ced60 con 0x7f52280ffdf0 2024-03-23T23:01:56.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.581+0000 7f5225e00640 1 --2- 172.21.15.12:0/834164362 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f520c0a2880 0x7f520c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:56.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.582+0000 7f5225e00640 1 --2- 172.21.15.12:0/834164362 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f520c0a2880 0x7f520c0a4d40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f52140095f0 tx=0x7f5214005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:56.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.585+0000 7f521f400640 1 -- 172.21.15.12:0/834164362 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f521809b760 con 0x7f52280ffdf0 2024-03-23T23:01:56.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.801+0000 7f522ca00640 1 -- 172.21.15.12:0/834164362 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f522802d070 con 0x7f52280ffdf0 2024-03-23T23:01:56.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.801+0000 7f521f400640 1 -- 172.21.15.12:0/834164362 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f52180bb0a0 con 0x7f52280ffdf0 2024-03-23T23:01:56.802 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-23T23:01:56.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.804+0000 7f522ca00640 1 -- 172.21.15.12:0/834164362 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f520c0a2880 msgr2=0x7f520c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:56.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.804+0000 7f522ca00640 1 --2- 172.21.15.12:0/834164362 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f520c0a2880 0x7f520c0a4d40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f52140095f0 tx=0x7f5214005f90 comp rx=0 tx=0).stop 2024-03-23T23:01:56.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.804+0000 7f522ca00640 1 -- 172.21.15.12:0/834164362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52280ffdf0 msgr2=0x7f522819ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:56.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.804+0000 7f522ca00640 1 --2- 172.21.15.12:0/834164362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52280ffdf0 0x7f522819ddf0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f521800e9f0 tx=0x7f521800eeb0 comp rx=0 tx=0).stop 2024-03-23T23:01:56.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.804+0000 7f522ca00640 1 -- 172.21.15.12:0/834164362 shutdown_connections 2024-03-23T23:01:56.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.804+0000 7f522ca00640 1 --2- 172.21.15.12:0/834164362 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f520c0a2880 0x7f520c0a4d40 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:56.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.804+0000 7f522ca00640 1 --2- 172.21.15.12:0/834164362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5228105110 0x7f52281a53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:56.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.804+0000 7f522ca00640 1 --2- 172.21.15.12:0/834164362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5228102760 0x7f522819e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:56.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.804+0000 7f522ca00640 1 --2- 172.21.15.12:0/834164362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52280ffdf0 0x7f522819ddf0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:56.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.804+0000 7f522ca00640 1 -- 172.21.15.12:0/834164362 >> 172.21.15.12:0/834164362 conn(0x7f52280fba10 msgr2=0x7f5228100770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:56.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.804+0000 7f522ca00640 1 -- 172.21.15.12:0/834164362 shutdown_connections 2024-03-23T23:01:56.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:56.804+0000 7f522ca00640 1 -- 172.21.15.12:0/834164362 wait complete. 2024-03-23T23:01:56.822 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T23:01:57.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.030+0000 7f539e400640 1 -- 172.21.15.12:0/2579920894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53980772a0 msgr2=0x7f5398077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:57.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.030+0000 7f539e400640 1 --2- 172.21.15.12:0/2579920894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53980772a0 0x7f5398077700 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5384009900 tx=0x7f538402bc30 comp rx=0 tx=0).stop 2024-03-23T23:01:57.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.030+0000 7f539e400640 1 -- 172.21.15.12:0/2579920894 shutdown_connections 2024-03-23T23:01:57.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.030+0000 7f539e400640 1 --2- 172.21.15.12:0/2579920894 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5398077c40 0x7f53980784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:57.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.030+0000 7f539e400640 1 --2- 172.21.15.12:0/2579920894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53980772a0 0x7f5398077700 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:57.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.030+0000 7f539e400640 1 --2- 172.21.15.12:0/2579920894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5398076060 0x7f5398076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:57.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.030+0000 7f539e400640 1 -- 172.21.15.12:0/2579920894 >> 172.21.15.12:0/2579920894 conn(0x7f53980fee50 msgr2=0x7f5398101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:57.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.030+0000 7f539e400640 1 -- 172.21.15.12:0/2579920894 shutdown_connections 2024-03-23T23:01:57.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.030+0000 7f539e400640 1 -- 172.21.15.12:0/2579920894 wait complete. 2024-03-23T23:01:57.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.031+0000 7f539e400640 1 Processor -- start 2024-03-23T23:01:57.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.031+0000 7f539e400640 1 -- start start 2024-03-23T23:01:57.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.031+0000 7f539e400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5398076060 0x7f539819dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:57.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.031+0000 7f539e400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53980772a0 0x7f539819e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:57.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.032+0000 7f539e400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5398077c40 0x7f53981a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:57.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.032+0000 7f539e400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5398113970 con 0x7f53980772a0 2024-03-23T23:01:57.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.032+0000 7f539e400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53981137f0 con 0x7f5398077c40 2024-03-23T23:01:57.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.032+0000 7f539e400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5398113af0 con 0x7f5398076060 2024-03-23T23:01:57.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.032+0000 7f5397400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5398076060 0x7f539819dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:57.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.032+0000 7f5397400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5398076060 0x7f539819dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:37212/0 (socket says 172.21.15.12:37212) 2024-03-23T23:01:57.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.032+0000 7f5397400640 1 -- 172.21.15.12:0/3945695243 learned_addr learned my addr 172.21.15.12:0/3945695243 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:57.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.032+0000 7f5396a00640 1 --2- 172.21.15.12:0/3945695243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53980772a0 0x7f539819e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:57.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.032+0000 7f5397e00640 1 --2- 172.21.15.12:0/3945695243 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5398077c40 0x7f53981a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:57.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.032+0000 7f5397400640 1 -- 172.21.15.12:0/3945695243 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5398077c40 msgr2=0x7f53981a5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:57.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.032+0000 7f5397400640 1 --2- 172.21.15.12:0/3945695243 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5398077c40 0x7f53981a5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:57.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.032+0000 7f5397400640 1 -- 172.21.15.12:0/3945695243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53980772a0 msgr2=0x7f539819e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:57.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.032+0000 7f5397400640 1 --2- 172.21.15.12:0/3945695243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53980772a0 0x7f539819e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:57.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.032+0000 7f5397400640 1 -- 172.21.15.12:0/3945695243 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53981a5c00 con 0x7f5398076060 2024-03-23T23:01:57.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.032+0000 7f5397e00640 1 --2- 172.21.15.12:0/3945695243 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5398077c40 0x7f53981a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:01:57.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.033+0000 7f5397400640 1 --2- 172.21.15.12:0/3945695243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5398076060 0x7f539819dfe0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f538800e9f0 tx=0x7f538800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:57.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.033+0000 7f538fe00640 1 -- 172.21.15.12:0/3945695243 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538800cd80 con 0x7f5398076060 2024-03-23T23:01:57.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.033+0000 7f538fe00640 1 -- 172.21.15.12:0/3945695243 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5388010910 con 0x7f5398076060 2024-03-23T23:01:57.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.033+0000 7f539e400640 1 -- 172.21.15.12:0/3945695243 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53981a5ef0 con 0x7f5398076060 2024-03-23T23:01:57.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.033+0000 7f538fe00640 1 -- 172.21.15.12:0/3945695243 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5388018990 con 0x7f5398076060 2024-03-23T23:01:57.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.033+0000 7f539e400640 1 -- 172.21.15.12:0/3945695243 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53981a6360 con 0x7f5398076060 2024-03-23T23:01:57.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.035+0000 7f539e400640 1 -- 172.21.15.12:0/3945695243 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5398110250 con 0x7f5398076060 2024-03-23T23:01:57.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.036+0000 7f538fe00640 1 -- 172.21.15.12:0/3945695243 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5388018b30 con 0x7f5398076060 2024-03-23T23:01:57.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.038+0000 7f538fe00640 1 --2- 172.21.15.12:0/3945695243 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f537c0a2880 0x7f537c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:57.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.038+0000 7f538fe00640 1 -- 172.21.15.12:0/3945695243 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f53880cedf0 con 0x7f5398076060 2024-03-23T23:01:57.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.038+0000 7f5396a00640 1 --2- 172.21.15.12:0/3945695243 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f537c0a2880 0x7f537c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:57.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.039+0000 7f5396a00640 1 --2- 172.21.15.12:0/3945695243 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f537c0a2880 0x7f537c0a4d40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f538402c240 tx=0x7f538402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:57.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.041+0000 7f538fe00640 1 -- 172.21.15.12:0/3945695243 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f538809be40 con 0x7f5398076060 2024-03-23T23:01:57.121 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:57 smithi012 ceph-mon[31695]: from='client.25280 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T23:01:57.121 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:57 smithi012 ceph-mon[31695]: pgmap v394: 97 pgs: 97 active+clean; 643 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 938 B/s wr, 6 op/s 2024-03-23T23:01:57.121 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:57 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/145160981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:01:57.121 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:57 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/834164362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:01:57.121 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:57 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:57.121 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:57 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:57.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.261+0000 7f539e400640 1 -- 172.21.15.12:0/3945695243 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5398062340 con 0x7f5398076060 2024-03-23T23:01:57.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.261+0000 7f538fe00640 1 -- 172.21.15.12:0/3945695243 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f538809b740 con 0x7f5398076060 2024-03-23T23:01:57.261 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-23T23:01:57.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.264+0000 7f539e400640 1 -- 172.21.15.12:0/3945695243 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f537c0a2880 msgr2=0x7f537c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:57.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.264+0000 7f539e400640 1 --2- 172.21.15.12:0/3945695243 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f537c0a2880 0x7f537c0a4d40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f538402c240 tx=0x7f538402df90 comp rx=0 tx=0).stop 2024-03-23T23:01:57.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.264+0000 7f539e400640 1 -- 172.21.15.12:0/3945695243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5398076060 msgr2=0x7f539819dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:57.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.264+0000 7f539e400640 1 --2- 172.21.15.12:0/3945695243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5398076060 0x7f539819dfe0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f538800e9f0 tx=0x7f538800eeb0 comp rx=0 tx=0).stop 2024-03-23T23:01:57.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.264+0000 7f539e400640 1 -- 172.21.15.12:0/3945695243 shutdown_connections 2024-03-23T23:01:57.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.264+0000 7f539e400640 1 --2- 172.21.15.12:0/3945695243 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f537c0a2880 0x7f537c0a4d40 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:57.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.264+0000 7f539e400640 1 --2- 172.21.15.12:0/3945695243 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5398077c40 0x7f53981a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:57.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.264+0000 7f539e400640 1 --2- 172.21.15.12:0/3945695243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53980772a0 0x7f539819e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:57.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.264+0000 7f539e400640 1 --2- 172.21.15.12:0/3945695243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5398076060 0x7f539819dfe0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:57.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.264+0000 7f539e400640 1 -- 172.21.15.12:0/3945695243 >> 172.21.15.12:0/3945695243 conn(0x7f53980fee50 msgr2=0x7f5398100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:57.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.264+0000 7f539e400640 1 -- 172.21.15.12:0/3945695243 shutdown_connections 2024-03-23T23:01:57.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.264+0000 7f539e400640 1 -- 172.21.15.12:0/3945695243 wait complete. 2024-03-23T23:01:57.283 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:01:57.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:57 smithi173 ceph-mon[36116]: from='client.25280 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T23:01:57.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:57 smithi173 ceph-mon[36116]: pgmap v394: 97 pgs: 97 active+clean; 643 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 938 B/s wr, 6 op/s 2024-03-23T23:01:57.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:57 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/145160981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:01:57.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:57 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/834164362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:01:57.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:57 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:57.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:57 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:57.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:57 smithi203 ceph-mon[36693]: from='client.25280 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T23:01:57.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:57 smithi203 ceph-mon[36693]: pgmap v394: 97 pgs: 97 active+clean; 643 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 938 B/s wr, 6 op/s 2024-03-23T23:01:57.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:57 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/145160981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:01:57.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:57 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/834164362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:01:57.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:57 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:57.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:57 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:57.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.503+0000 7f6675c00640 1 -- 172.21.15.12:0/558005874 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66701038e0 msgr2=0x7f6670103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:57.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.503+0000 7f6675c00640 1 --2- 172.21.15.12:0/558005874 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66701038e0 0x7f6670103d00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6660009900 tx=0x7f666002bc50 comp rx=0 tx=0).stop 2024-03-23T23:01:57.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.503+0000 7f6675c00640 1 -- 172.21.15.12:0/558005874 shutdown_connections 2024-03-23T23:01:57.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.503+0000 7f6675c00640 1 --2- 172.21.15.12:0/558005874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6670105490 0x7f667010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:57.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.503+0000 7f6675c00640 1 --2- 172.21.15.12:0/558005874 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6670104ad0 0x7f6670104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:57.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.503+0000 7f6675c00640 1 --2- 172.21.15.12:0/558005874 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66701038e0 0x7f6670103d00 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:57.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.503+0000 7f6675c00640 1 -- 172.21.15.12:0/558005874 >> 172.21.15.12:0/558005874 conn(0x7f66700fee50 msgr2=0x7f6670101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:57.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.503+0000 7f6675c00640 1 -- 172.21.15.12:0/558005874 shutdown_connections 2024-03-23T23:01:57.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.503+0000 7f6675c00640 1 -- 172.21.15.12:0/558005874 wait complete. 2024-03-23T23:01:57.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.504+0000 7f6675c00640 1 Processor -- start 2024-03-23T23:01:57.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.504+0000 7f6675c00640 1 -- start start 2024-03-23T23:01:57.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.505+0000 7f6675c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66701038e0 0x7f6670199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:57.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.505+0000 7f6675c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6670104ad0 0x7f667019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:57.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.505+0000 7f6675c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6670105490 0x7f66701a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:57.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.505+0000 7f6675c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667010f580 con 0x7f6670105490 2024-03-23T23:01:57.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.505+0000 7f6675c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667010f400 con 0x7f66701038e0 2024-03-23T23:01:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.505+0000 7f6675c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667010f700 con 0x7f6670104ad0 2024-03-23T23:01:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.505+0000 7f666ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66701038e0 0x7f6670199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.505+0000 7f666e000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6670104ad0 0x7f667019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.505+0000 7f666ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66701038e0 0x7f6670199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36558/0 (socket says 172.21.15.12:36558) 2024-03-23T23:01:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.505+0000 7f666ea00640 1 -- 172.21.15.12:0/1357245323 learned_addr learned my addr 172.21.15.12:0/1357245323 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.505+0000 7f666e000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6670104ad0 0x7f667019a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:37230/0 (socket says 172.21.15.12:37230) 2024-03-23T23:01:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.505+0000 7f666f400640 1 --2- 172.21.15.12:0/1357245323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6670105490 0x7f66701a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.505+0000 7f666e000640 1 -- 172.21.15.12:0/1357245323 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66701038e0 msgr2=0x7f6670199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.505+0000 7f666e000640 1 --2- 172.21.15.12:0/1357245323 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66701038e0 0x7f6670199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.505+0000 7f666e000640 1 -- 172.21.15.12:0/1357245323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6670105490 msgr2=0x7f66701a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.505+0000 7f666e000640 1 --2- 172.21.15.12:0/1357245323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6670105490 0x7f66701a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.505+0000 7f666e000640 1 -- 172.21.15.12:0/1357245323 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66701a1810 con 0x7f6670104ad0 2024-03-23T23:01:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.505+0000 7f666f400640 1 --2- 172.21.15.12:0/1357245323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6670105490 0x7f66701a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:01:57.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.506+0000 7f666e000640 1 --2- 172.21.15.12:0/1357245323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6670104ad0 0x7f667019a130 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f665c00e9f0 tx=0x7f665c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:57.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.506+0000 7f6667400640 1 -- 172.21.15.12:0/1357245323 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665c00ccc0 con 0x7f6670104ad0 2024-03-23T23:01:57.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.506+0000 7f6667400640 1 -- 172.21.15.12:0/1357245323 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f665c010910 con 0x7f6670104ad0 2024-03-23T23:01:57.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.506+0000 7f6675c00640 1 -- 172.21.15.12:0/1357245323 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66701a1b00 con 0x7f6670104ad0 2024-03-23T23:01:57.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.506+0000 7f6667400640 1 -- 172.21.15.12:0/1357245323 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665c018960 con 0x7f6670104ad0 2024-03-23T23:01:57.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.506+0000 7f6675c00640 1 -- 172.21.15.12:0/1357245323 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66701a20a0 con 0x7f6670104ad0 2024-03-23T23:01:57.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.507+0000 7f6675c00640 1 -- 172.21.15.12:0/1357245323 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f667010ac00 con 0x7f6670104ad0 2024-03-23T23:01:57.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.510+0000 7f6667400640 1 -- 172.21.15.12:0/1357245323 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f665c010ac0 con 0x7f6670104ad0 2024-03-23T23:01:57.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.511+0000 7f6667400640 1 --2- 172.21.15.12:0/1357245323 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f66540a2880 0x7f66540a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:57.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.511+0000 7f6667400640 1 -- 172.21.15.12:0/1357245323 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f665c0cee50 con 0x7f6670104ad0 2024-03-23T23:01:57.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.511+0000 7f666ea00640 1 --2- 172.21.15.12:0/1357245323 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f66540a2880 0x7f66540a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:57.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.512+0000 7f666ea00640 1 --2- 172.21.15.12:0/1357245323 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f66540a2880 0x7f66540a4d40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f666002c900 tx=0x7f6660000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:57.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.515+0000 7f6667400640 1 -- 172.21.15.12:0/1357245323 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f665c09bea0 con 0x7f6670104ad0 2024-03-23T23:01:57.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.773+0000 7f6675c00640 1 -- 172.21.15.12:0/1357245323 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66700622d0 con 0x7f6670104ad0 2024-03-23T23:01:57.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.774+0000 7f6667400640 1 -- 172.21.15.12:0/1357245323 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f665c09b7a0 con 0x7f6670104ad0 2024-03-23T23:01:57.774 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:01:57.774 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:01:57.774 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:01:57.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.776+0000 7f6675c00640 1 -- 172.21.15.12:0/1357245323 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f66540a2880 msgr2=0x7f66540a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:57.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.776+0000 7f6675c00640 1 --2- 172.21.15.12:0/1357245323 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f66540a2880 0x7f66540a4d40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f666002c900 tx=0x7f6660000bc0 comp rx=0 tx=0).stop 2024-03-23T23:01:57.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.776+0000 7f6675c00640 1 -- 172.21.15.12:0/1357245323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6670104ad0 msgr2=0x7f667019a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:57.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.776+0000 7f6675c00640 1 --2- 172.21.15.12:0/1357245323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6670104ad0 0x7f667019a130 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f665c00e9f0 tx=0x7f665c00eeb0 comp rx=0 tx=0).stop 2024-03-23T23:01:57.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.777+0000 7f6675c00640 1 -- 172.21.15.12:0/1357245323 shutdown_connections 2024-03-23T23:01:57.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.777+0000 7f6675c00640 1 --2- 172.21.15.12:0/1357245323 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f66540a2880 0x7f66540a4d40 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:57.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.777+0000 7f6675c00640 1 --2- 172.21.15.12:0/1357245323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6670105490 0x7f66701a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:57.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.777+0000 7f6675c00640 1 --2- 172.21.15.12:0/1357245323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6670104ad0 0x7f667019a130 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:57.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.777+0000 7f6675c00640 1 --2- 172.21.15.12:0/1357245323 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66701038e0 0x7f6670199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:57.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.777+0000 7f6675c00640 1 -- 172.21.15.12:0/1357245323 >> 172.21.15.12:0/1357245323 conn(0x7f66700fee50 msgr2=0x7f6670107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:57.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.777+0000 7f6675c00640 1 -- 172.21.15.12:0/1357245323 shutdown_connections 2024-03-23T23:01:57.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:57.777+0000 7f6675c00640 1 -- 172.21.15.12:0/1357245323 wait complete. 2024-03-23T23:01:57.797 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T23:01:58.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.005+0000 7f0cb1600640 1 -- 172.21.15.12:0/3900041390 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cac103700 msgr2=0x7f0cac103b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:58.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.005+0000 7f0cb1600640 1 --2- 172.21.15.12:0/3900041390 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cac103700 0x7f0cac103b20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c009900 tx=0x7f0c9c02bc30 comp rx=0 tx=0).stop 2024-03-23T23:01:58.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.005+0000 7f0cb1600640 1 -- 172.21.15.12:0/3900041390 shutdown_connections 2024-03-23T23:01:58.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.005+0000 7f0cb1600640 1 --2- 172.21.15.12:0/3900041390 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0cac1052b0 0x7f0cac10bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.005+0000 7f0cb1600640 1 --2- 172.21.15.12:0/3900041390 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cac1048f0 0x7f0cac104d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.005+0000 7f0cb1600640 1 --2- 172.21.15.12:0/3900041390 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cac103700 0x7f0cac103b20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.005+0000 7f0cb1600640 1 -- 172.21.15.12:0/3900041390 >> 172.21.15.12:0/3900041390 conn(0x7f0cac0fec70 msgr2=0x7f0cac1010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:58.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.005+0000 7f0cb1600640 1 -- 172.21.15.12:0/3900041390 shutdown_connections 2024-03-23T23:01:58.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.005+0000 7f0cb1600640 1 -- 172.21.15.12:0/3900041390 wait complete. 2024-03-23T23:01:58.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.006+0000 7f0cb1600640 1 Processor -- start 2024-03-23T23:01:58.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.006+0000 7f0cb1600640 1 -- start start 2024-03-23T23:01:58.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.007+0000 7f0cb1600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0cac103700 0x7f0cac199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.007+0000 7f0cb1600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cac1048f0 0x7f0cac199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.007+0000 7f0cb1600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cac1052b0 0x7f0cac1a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.007+0000 7f0cb1600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cac10f3c0 con 0x7f0cac103700 2024-03-23T23:01:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.007+0000 7f0cb1600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cac10f240 con 0x7f0cac1052b0 2024-03-23T23:01:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.007+0000 7f0cb1600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cac10f540 con 0x7f0cac1048f0 2024-03-23T23:01:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.007+0000 7f0cab400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cac1048f0 0x7f0cac199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.007+0000 7f0cabe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0cac103700 0x7f0cac199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.007+0000 7f0cb0c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cac1052b0 0x7f0cac1a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.007+0000 7f0cab400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cac1048f0 0x7f0cac199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:37238/0 (socket says 172.21.15.12:37238) 2024-03-23T23:01:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.007+0000 7f0cab400640 1 -- 172.21.15.12:0/3765090095 learned_addr learned my addr 172.21.15.12:0/3765090095 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.007+0000 7f0cb0c00640 1 -- 172.21.15.12:0/3765090095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cac1048f0 msgr2=0x7f0cac199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.008+0000 7f0cb0c00640 1 --2- 172.21.15.12:0/3765090095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cac1048f0 0x7f0cac199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.008+0000 7f0cb0c00640 1 -- 172.21.15.12:0/3765090095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0cac103700 msgr2=0x7f0cac199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.008+0000 7f0cb0c00640 1 --2- 172.21.15.12:0/3765090095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0cac103700 0x7f0cac199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.008+0000 7f0cb0c00640 1 -- 172.21.15.12:0/3765090095 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cac1a1630 con 0x7f0cac1052b0 2024-03-23T23:01:58.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.008+0000 7f0cb0c00640 1 --2- 172.21.15.12:0/3765090095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cac1052b0 0x7f0cac1a0fc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0ca400cab0 tx=0x7f0ca400cf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:58.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.008+0000 7f0ca8c00640 1 -- 172.21.15.12:0/3765090095 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca400ae20 con 0x7f0cac1052b0 2024-03-23T23:01:58.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.008+0000 7f0ca8c00640 1 -- 172.21.15.12:0/3765090095 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ca400f930 con 0x7f0cac1052b0 2024-03-23T23:01:58.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.008+0000 7f0ca8c00640 1 -- 172.21.15.12:0/3765090095 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca4018b30 con 0x7f0cac1052b0 2024-03-23T23:01:58.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.008+0000 7f0cb1600640 1 -- 172.21.15.12:0/3765090095 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cac1a1920 con 0x7f0cac1052b0 2024-03-23T23:01:58.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.008+0000 7f0cb1600640 1 -- 172.21.15.12:0/3765090095 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cac1a1e20 con 0x7f0cac1052b0 2024-03-23T23:01:58.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.010+0000 7f0cb1600640 1 -- 172.21.15.12:0/3765090095 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cac10aa20 con 0x7f0cac1052b0 2024-03-23T23:01:58.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.012+0000 7f0ca8c00640 1 -- 172.21.15.12:0/3765090095 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0ca400fae0 con 0x7f0cac1052b0 2024-03-23T23:01:58.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.013+0000 7f0ca8c00640 1 --2- 172.21.15.12:0/3765090095 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0c900a2880 0x7f0c900a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.013+0000 7f0ca8c00640 1 -- 172.21.15.12:0/3765090095 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0ca40cf100 con 0x7f0cac1052b0 2024-03-23T23:01:58.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.013+0000 7f0cabe00640 1 --2- 172.21.15.12:0/3765090095 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0c900a2880 0x7f0c900a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.014+0000 7f0cabe00640 1 --2- 172.21.15.12:0/3765090095 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0c900a2880 0x7f0c900a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c02c170 tx=0x7f0c9c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:58.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.017+0000 7f0ca8c00640 1 -- 172.21.15.12:0/3765090095 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0ca4014030 con 0x7f0cac1052b0 2024-03-23T23:01:58.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.258+0000 7f0cb1600640 1 -- 172.21.15.12:0/3765090095 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0cac0622d0 con 0x7f0cac1052b0 2024-03-23T23:01:58.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.260+0000 7f0ca8c00640 1 -- 172.21.15.12:0/3765090095 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22695 (secure 0 0 0) 0x7f0ca409c150 con 0x7f0cac1052b0 2024-03-23T23:01:58.260 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:01:58.260 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":88,"fsid":"90f679ae-e967-11ee-95cd-87774f69a715","created":"2024-03-23T22:50:14.906952+0000","modified":"2024-03-23T23:00:24.349052+0000","last_up_change":"2024-03-23T22:58:28.907824+0000","last_in_change":"2024-03-23T22:58:08.342487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T22:54:11.831010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T22:59:25.350974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T22:59:26.390323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b70875d7-a23d-4251-9e8b-1a16d4edc2a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6803","nonce":1190079611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6805","nonce":1190079611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6809","nonce":1190079611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6807","nonce":1190079611}]},"public_addr":"172.21.15.12:6803/1190079611","cluster_addr":"172.21.15.12:6805/1190079611","heartbeat_back_addr":"172.21.15.12:6809/1190079611","heartbeat_front_addr":"172.21.15.12:6807/1190079611","state":["exists","up"]},{"osd":1,"uuid":"416e87af-db5d-41ce-84d9-731e3e07e3c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6801","nonce":1983639525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6803","nonce":1983639525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6807","nonce":1983639525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6805","nonce":1983639525}]},"public_addr":"172.21.15.173:6801/1983639525","cluster_addr":"172.21.15.173:6803/1983639525","heartbeat_back_addr":"172.21.15.173:6807/1983639525","heartbeat_front_addr":"172.21.15.173:6805/1983639525","state":["exists","up"]},{"osd":2,"uuid":"7a559c70-4c54-431b-8baf-4136b6e76493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6801","nonce":2253997387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6803","nonce":2253997387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6807","nonce":2253997387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6805","nonce":2253997387}]},"public_addr":"172.21.15.203:6801/2253997387","cluster_addr":"172.21.15.203:6803/2253997387","heartbeat_back_addr":"172.21.15.203:6807/2253997387","heartbeat_front_addr":"172.21.15.203:6805/2253997387","state":["exists","up"]},{"osd":3,"uuid":"c41cd287-a674-486c-8025-328cd36e5ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6811","nonce":1761446905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6813","nonce":1761446905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6817","nonce":1761446905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6815","nonce":1761446905}]},"public_addr":"172.21.15.12:6811/1761446905","cluster_addr":"172.21.15.12:6813/1761446905","heartbeat_back_addr":"172.21.15.12:6817/1761446905","heartbeat_front_addr":"172.21.15.12:6815/1761446905","state":["exists","up"]},{"osd":4,"uuid":"3c48f886-0b42-4a09-b31e-abccc200bb1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6809","nonce":242958272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6811","nonce":242958272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6815","nonce":242958272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6813","nonce":242958272}]},"public_addr":"172.21.15.173:6809/242958272","cluster_addr":"172.21.15.173:6811/242958272","heartbeat_back_addr":"172.21.15.173:6815/242958272","heartbeat_front_addr":"172.21.15.173:6813/242958272","state":["exists","up"]},{"osd":5,"uuid":"4a34f06f-8805-40c0-b1ab-23407b0276d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6809","nonce":4270966092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6811","nonce":4270966092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6815","nonce":4270966092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6813","nonce":4270966092}]},"public_addr":"172.21.15.203:6809/4270966092","cluster_addr":"172.21.15.203:6811/4270966092","heartbeat_back_addr":"172.21.15.203:6815/4270966092","heartbeat_front_addr":"172.21.15.203:6813/4270966092","state":["exists","up"]},{"osd":6,"uuid":"26ddc663-5623-48c9-80d1-f8b968765d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6819","nonce":3606839295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6821","nonce":3606839295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6825","nonce":3606839295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6823","nonce":3606839295}]},"public_addr":"172.21.15.12:6819/3606839295","cluster_addr":"172.21.15.12:6821/3606839295","heartbeat_back_addr":"172.21.15.12:6825/3606839295","heartbeat_front_addr":"172.21.15.12:6823/3606839295","state":["exists","up"]},{"osd":7,"uuid":"17dc9417-71cd-4576-aa08-933212ec55e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6817","nonce":1241504346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6819","nonce":1241504346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6823","nonce":1241504346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6821","nonce":1241504346}]},"public_addr":"172.21.15.173:6817/1241504346","cluster_addr":"172.21.15.173:6819/1241504346","heartbeat_back_addr":"172.21.15.173:6823/1241504346","heartbeat_front_addr":"172.21.15.173:6821/1241504346","state":["exists","up"]},{"osd":8,"uuid":"90a2b63a-471d-43b8-ac18-bb1f16778d18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6817","nonce":1516996192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6819","nonce":1516996192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6823","nonce":1516996192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6821","nonce":1516996192}]},"public_addr":"172.21.15.203:6817/1516996192","cluster_addr":"172.21.15.203:6819/1516996192","heartbeat_back_addr":"172.21.15.203:6823/1516996192","heartbeat_front_addr":"172.21.15.203:6821/1516996192","state":["exists","up"]},{"osd":9,"uuid":"6e1ca504-40b4-49d0-b245-7946181abfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6827","nonce":1175260774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6829","nonce":1175260774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6833","nonce":1175260774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6831","nonce":1175260774}]},"public_addr":"172.21.15.12:6827/1175260774","cluster_addr":"172.21.15.12:6829/1175260774","heartbeat_back_addr":"172.21.15.12:6833/1175260774","heartbeat_front_addr":"172.21.15.12:6831/1175260774","state":["exists","up"]},{"osd":10,"uuid":"09a595ed-5614-4386-8e61-0170af66e01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6825","nonce":3303506808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6827","nonce":3303506808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6831","nonce":3303506808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6829","nonce":3303506808}]},"public_addr":"172.21.15.173:6825/3303506808","cluster_addr":"172.21.15.173:6827/3303506808","heartbeat_back_addr":"172.21.15.173:6831/3303506808","heartbeat_front_addr":"172.21.15.173:6829/3303506808","state":["exists","up"]},{"osd":11,"uuid":"9ef1d595-a977-4ba7-9c6b-d2b3295f20b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6825","nonce":2323604506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6827","nonce":2323604506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6831","nonce":2323604506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6829","nonce":2323604506}]},"public_addr":"172.21.15.203:6825/2323604506","cluster_addr":"172.21.15.203:6827/2323604506","heartbeat_back_addr":"172.21.15.203:6831/2323604506","heartbeat_front_addr":"172.21.15.203:6829/2323604506","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:18.469038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:42.752628+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:07.069493+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:41.962478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:08.363104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:33.341336+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:07.286541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:34.243302+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:59.749699+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:57:33.822913+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:00.737269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:26.599854+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/4182783120":"2024-03-24T22:51:23.113201+0000","172.21.15.12:6800/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1900938637":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/3414861652":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6801/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/2580541477":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6800/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/501058839":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1192554716":"2024-03-24T22:50:48.401223+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T23:01:58.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.262+0000 7f0cb1600640 1 -- 172.21.15.12:0/3765090095 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0c900a2880 msgr2=0x7f0c900a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:58.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.262+0000 7f0cb1600640 1 --2- 172.21.15.12:0/3765090095 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0c900a2880 0x7f0c900a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c02c170 tx=0x7f0c9c02df90 comp rx=0 tx=0).stop 2024-03-23T23:01:58.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.262+0000 7f0cb1600640 1 -- 172.21.15.12:0/3765090095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cac1052b0 msgr2=0x7f0cac1a0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:58.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.262+0000 7f0cb1600640 1 --2- 172.21.15.12:0/3765090095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cac1052b0 0x7f0cac1a0fc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0ca400cab0 tx=0x7f0ca400cf70 comp rx=0 tx=0).stop 2024-03-23T23:01:58.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.263+0000 7f0cb1600640 1 -- 172.21.15.12:0/3765090095 shutdown_connections 2024-03-23T23:01:58.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.263+0000 7f0cb1600640 1 --2- 172.21.15.12:0/3765090095 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0c900a2880 0x7f0c900a4d40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.263+0000 7f0cb1600640 1 --2- 172.21.15.12:0/3765090095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cac1052b0 0x7f0cac1a0fc0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.263+0000 7f0cb1600640 1 --2- 172.21.15.12:0/3765090095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cac1048f0 0x7f0cac199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.263+0000 7f0cb1600640 1 --2- 172.21.15.12:0/3765090095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0cac103700 0x7f0cac199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.263+0000 7f0cb1600640 1 -- 172.21.15.12:0/3765090095 >> 172.21.15.12:0/3765090095 conn(0x7f0cac0fec70 msgr2=0x7f0cac107bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:58.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.263+0000 7f0cb1600640 1 -- 172.21.15.12:0/3765090095 shutdown_connections 2024-03-23T23:01:58.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.263+0000 7f0cb1600640 1 -- 172.21.15.12:0/3765090095 wait complete. 2024-03-23T23:01:58.282 DEBUG:tasks.fwd_scrub:Yielding 2024-03-23T23:01:58.282 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-23T23:01:58.294 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi012.front.sepia.ceph.com 2024-03-23T23:01:58.294 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-23T23:01:58.294 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-23T23:01:58.294 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-23T23:01:58.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:01:58.295 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-23T23:01:58.298 INFO:teuthology.run_tasks:Running task workunit... 2024-03-23T23:01:58.313 INFO:tasks.workunit:Pulling workunits from ref 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-23T23:01:58.314 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-23T23:01:58.314 INFO:tasks.workunit:timeout=3h 2024-03-23T23:01:58.314 INFO:tasks.workunit:cleanup=True 2024-03-23T23:01:58.317 DEBUG:teuthology.orchestra.run.smithi012:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-23T23:01:58.319 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:58 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3945695243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:01:58.319 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:58 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1357245323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:01:58.361 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:58 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3945695243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:01:58.361 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:58 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1357245323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:01:58.400 INFO:teuthology.orchestra.run.smithi012.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-23T23:01:58.400 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-23T23:01:58.400 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 76h/118d Inode: 1099511628283 Links: 2 2024-03-23T23:01:58.400 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T23:01:58.400 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-23T23:01:58.400 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-03-23 22:59:34.194706702 +0000 2024-03-23T23:01:58.401 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-03-23 22:59:34.194706702 +0000 2024-03-23T23:01:58.401 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-03-23 23:01:48.578584257 +0000 2024-03-23T23:01:58.401 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-03-23 22:59:34.194706702 +0000 2024-03-23T23:01:58.401 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-23T23:01:58.401 DEBUG:teuthology.orchestra.run.smithi012:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-23T23:01:58.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:58 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3945695243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:01:58.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:58 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1357245323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:01:58.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.493+0000 7effdb200640 1 -- 172.21.15.12:0/4055210994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effdc104930 msgr2=0x7effdc106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:58.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.493+0000 7effdb200640 1 --2- 172.21.15.12:0/4055210994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effdc104930 0x7effdc106d20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7effc8009900 tx=0x7effc802bc30 comp rx=0 tx=0).stop 2024-03-23T23:01:58.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.493+0000 7effdb200640 1 -- 172.21.15.12:0/4055210994 shutdown_connections 2024-03-23T23:01:58.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.493+0000 7effdb200640 1 --2- 172.21.15.12:0/4055210994 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7effdc107260 0x7effdc1096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.493+0000 7effdb200640 1 --2- 172.21.15.12:0/4055210994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effdc104930 0x7effdc106d20 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.493+0000 7effdb200640 1 --2- 172.21.15.12:0/4055210994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effdc102000 0x7effdc1043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.493+0000 7effdb200640 1 -- 172.21.15.12:0/4055210994 >> 172.21.15.12:0/4055210994 conn(0x7effdc0fba10 msgr2=0x7effdc0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:58.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.493+0000 7effdb200640 1 -- 172.21.15.12:0/4055210994 shutdown_connections 2024-03-23T23:01:58.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.493+0000 7effdb200640 1 -- 172.21.15.12:0/4055210994 wait complete. 2024-03-23T23:01:58.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.494+0000 7effdb200640 1 Processor -- start 2024-03-23T23:01:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.494+0000 7effdb200640 1 -- start start 2024-03-23T23:01:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.494+0000 7effdb200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7effdc102000 0x7effdc199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.494+0000 7effdb200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effdc104930 0x7effdc19a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.494+0000 7effdb200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effdc107260 0x7effdc1a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.494+0000 7effdb200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc10d260 con 0x7effdc107260 2024-03-23T23:01:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.494+0000 7effdb200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc10d0e0 con 0x7effdc102000 2024-03-23T23:01:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.494+0000 7effdb200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc10d3e0 con 0x7effdc104930 2024-03-23T23:01:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.495+0000 7effd9e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7effdc102000 0x7effdc199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.495+0000 7effd9e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7effdc102000 0x7effdc199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36598/0 (socket says 172.21.15.12:36598) 2024-03-23T23:01:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.495+0000 7effd9e00640 1 -- 172.21.15.12:0/3056717471 learned_addr learned my addr 172.21.15.12:0/3056717471 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.495+0000 7effd9400640 1 --2- 172.21.15.12:0/3056717471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effdc104930 0x7effdc19a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.495+0000 7effda800640 1 --2- 172.21.15.12:0/3056717471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effdc107260 0x7effdc1a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.495+0000 7effd9e00640 1 -- 172.21.15.12:0/3056717471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effdc104930 msgr2=0x7effdc19a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.495+0000 7effd9e00640 1 --2- 172.21.15.12:0/3056717471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effdc104930 0x7effdc19a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.495+0000 7effd9e00640 1 -- 172.21.15.12:0/3056717471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effdc107260 msgr2=0x7effdc1a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.495+0000 7effd9e00640 1 --2- 172.21.15.12:0/3056717471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effdc107260 0x7effdc1a1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.495+0000 7effd9e00640 1 -- 172.21.15.12:0/3056717471 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effdc1a16f0 con 0x7effdc102000 2024-03-23T23:01:58.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.495+0000 7effd9e00640 1 --2- 172.21.15.12:0/3056717471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7effdc102000 0x7effdc199ad0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7effcc00e9f0 tx=0x7effcc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:58.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.496+0000 7effd2a00640 1 -- 172.21.15.12:0/3056717471 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc00cd80 con 0x7effdc102000 2024-03-23T23:01:58.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.496+0000 7effd2a00640 1 -- 172.21.15.12:0/3056717471 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effcc005500 con 0x7effdc102000 2024-03-23T23:01:58.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.496+0000 7effdb200640 1 -- 172.21.15.12:0/3056717471 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effdc1a19e0 con 0x7effdc102000 2024-03-23T23:01:58.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.496+0000 7effd2a00640 1 -- 172.21.15.12:0/3056717471 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc010690 con 0x7effdc102000 2024-03-23T23:01:58.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.496+0000 7effdb200640 1 -- 172.21.15.12:0/3056717471 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effdc1a1eb0 con 0x7effdc102000 2024-03-23T23:01:58.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.500+0000 7effe3a40740 1 Processor -- start 2024-03-23T23:01:58.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.500+0000 7effe3a40740 1 -- start start 2024-03-23T23:01:58.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.501+0000 7effe3a40740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559b757e2b0 0x5559b757e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.501+0000 7effe3a40740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559b757eca0 0x5559b75837a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.501+0000 7effe3a40740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559b7583e10 0x5559b75862a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.501+0000 7effe3a40740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559b74e3eb0 con 0x5559b757eca0 2024-03-23T23:01:58.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.501+0000 7effe3a40740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559b748ca60 con 0x5559b7583e10 2024-03-23T23:01:58.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.501+0000 7effe3a40740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559b74dff50 con 0x5559b757e2b0 2024-03-23T23:01:58.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.501+0000 7effd2a00640 1 -- 172.21.15.12:0/3056717471 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7effcc0046c0 con 0x7effdc102000 2024-03-23T23:01:58.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.501+0000 7effda800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559b757e2b0 0x5559b757e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.501+0000 7effda800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559b757e2b0 0x5559b757e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:37276/0 (socket says 172.21.15.12:37276) 2024-03-23T23:01:58.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.501+0000 7effda800640 1 -- 172.21.15.12:0/995934154 learned_addr learned my addr 172.21.15.12:0/995934154 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:58.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.501+0000 7effd9400640 1 --2- 172.21.15.12:0/995934154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559b7583e10 0x5559b75862a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.501+0000 7effda800640 1 -- 172.21.15.12:0/995934154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559b7583e10 msgr2=0x5559b75862a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:58.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.501+0000 7effda800640 1 --2- 172.21.15.12:0/995934154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559b7583e10 0x5559b75862a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.501+0000 7effda800640 1 -- 172.21.15.12:0/995934154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559b757eca0 msgr2=0x5559b75837a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:58.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.501+0000 7effd9e00640 1 --2- 172.21.15.12:0/995934154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559b757eca0 0x5559b75837a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.502+0000 7effda800640 1 --2- 172.21.15.12:0/995934154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559b757eca0 0x5559b75837a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.502+0000 7effda800640 1 -- 172.21.15.12:0/995934154 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559b72c07e0 con 0x5559b757e2b0 2024-03-23T23:01:58.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.502+0000 7effd9e00640 1 --2- 172.21.15.12:0/995934154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559b757eca0 0x5559b75837a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:01:58.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.502+0000 7effda800640 1 --2- 172.21.15.12:0/995934154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559b757e2b0 0x5559b757e6d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7effd4009fc0 tx=0x7effd4010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad223afc1016bfa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:58.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.502+0000 7effc7400640 1 -- 172.21.15.12:0/995934154 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4003b70 con 0x5559b757e2b0 2024-03-23T23:01:58.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.502+0000 7effc7400640 1 -- 172.21.15.12:0/995934154 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effd4003d10 con 0x5559b757e2b0 2024-03-23T23:01:58.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.502+0000 7effc7400640 1 -- 172.21.15.12:0/995934154 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4011ab0 con 0x5559b757e2b0 2024-03-23T23:01:58.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.503+0000 7effd2a00640 1 --2- 172.21.15.12:0/3056717471 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7effc00a2800 0x7effc00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.503+0000 7effd2a00640 1 -- 172.21.15.12:0/3056717471 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7effcc0ceee0 con 0x7effdc102000 2024-03-23T23:01:58.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.503+0000 7effd9e00640 1 --2- 172.21.15.12:0/3056717471 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7effc00a2800 0x7effc00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.503+0000 7effe3a40740 1 -- 172.21.15.12:0/995934154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559b757e2b0 msgr2=0x5559b757e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:58.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.503+0000 7effe3a40740 1 --2- 172.21.15.12:0/995934154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559b757e2b0 0x5559b757e6d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7effd4009fc0 tx=0x7effd4010860 comp rx=0 tx=0).stop 2024-03-23T23:01:58.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.503+0000 7effe3a40740 1 -- 172.21.15.12:0/995934154 shutdown_connections 2024-03-23T23:01:58.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.503+0000 7effe3a40740 1 --2- 172.21.15.12:0/995934154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559b7583e10 0x5559b75862a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.503+0000 7effe3a40740 1 --2- 172.21.15.12:0/995934154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559b757eca0 0x5559b75837a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.503+0000 7effe3a40740 1 --2- 172.21.15.12:0/995934154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559b757e2b0 0x5559b757e6d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.503+0000 7effe3a40740 1 -- 172.21.15.12:0/995934154 >> 172.21.15.12:0/995934154 conn(0x5559b7579820 msgr2=0x5559b7579c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:58.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.504+0000 7effe3a40740 1 -- 172.21.15.12:0/995934154 shutdown_connections 2024-03-23T23:01:58.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.504+0000 7effe3a40740 1 -- 172.21.15.12:0/995934154 wait complete. 2024-03-23T23:01:58.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.504+0000 7effd9e00640 1 --2- 172.21.15.12:0/3056717471 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7effc00a2800 0x7effc00a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7effcc0ccd60 tx=0x7effcc004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:58.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.504+0000 7effe3a40740 1 -- start start 2024-03-23T23:01:58.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.504+0000 7effe3a40740 1 Processor -- start 2024-03-23T23:01:58.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.505+0000 7effe3a40740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559b757e2b0 0x5559b7588490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.505+0000 7effe3a40740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559b757eca0 0x5559b75889d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.505+0000 7effe3a40740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559b7583e10 0x5559b758fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.505+0000 7effe3a40740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559b7586dc0 con 0x5559b7583e10 2024-03-23T23:01:58.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.505+0000 7effe3a40740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559b749feb0 con 0x5559b757e2b0 2024-03-23T23:01:58.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.505+0000 7effe3a40740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559b7586f40 con 0x5559b757eca0 2024-03-23T23:01:58.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.505+0000 7effe3a40740 10 client.0 _ll_register_callbacks cb 0x5559b7494150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:01:58.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.505+0000 7effd9400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559b757eca0 0x5559b75889d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.505+0000 7effda800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559b757e2b0 0x5559b7588490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.505+0000 7effd9400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559b757eca0 0x5559b75889d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:37290/0 (socket says 172.21.15.12:37290) 2024-03-23T23:01:58.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.505+0000 7effda800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559b757e2b0 0x5559b7588490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36608/0 (socket says 172.21.15.12:36608) 2024-03-23T23:01:58.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.505+0000 7effd9400640 1 -- 172.21.15.12:0/1733013287 learned_addr learned my addr 172.21.15.12:0/1733013287 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:58.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.505+0000 7effda800640 1 --2- 172.21.15.12:0/1733013287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559b7583e10 0x5559b758fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.506+0000 7effd9400640 1 -- 172.21.15.12:0/1733013287 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559b757e2b0 msgr2=0x5559b7588490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:58.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.506+0000 7effd9400640 1 --2- 172.21.15.12:0/1733013287 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559b757e2b0 0x5559b7588490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.506+0000 7effd9400640 1 -- 172.21.15.12:0/1733013287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559b7583e10 msgr2=0x5559b758fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:58.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.506+0000 7effd9400640 1 --2- 172.21.15.12:0/1733013287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559b7583e10 0x5559b758fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.506+0000 7effd9400640 1 -- 172.21.15.12:0/1733013287 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5559b7320f70 con 0x5559b757eca0 2024-03-23T23:01:58.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.506+0000 7effd9400640 1 --2- 172.21.15.12:0/1733013287 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559b757eca0 0x5559b75889d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7effc802c240 tx=0x7effc8004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd558d2081f642e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:58.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.506+0000 7effbbe00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:01:58.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.506+0000 7effbbe00640 1 -- 172.21.15.12:0/1733013287 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8005d50 con 0x5559b757eca0 2024-03-23T23:01:58.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.506+0000 7effbbe00640 1 -- 172.21.15.12:0/1733013287 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7effc8005040 con 0x5559b757eca0 2024-03-23T23:01:58.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.506+0000 7effe3a40740 1 -- 172.21.15.12:0/1733013287 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7effd4003d10 con 0x5559b757eca0 2024-03-23T23:01:58.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.507+0000 7effbbe00640 1 -- 172.21.15.12:0/1733013287 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7effc8005240 con 0x5559b757eca0 2024-03-23T23:01:58.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.507+0000 7effe3a40740 10 client.25309 fetch_fsmap learned FSMap version 23 2024-03-23T23:01:58.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.507+0000 7effe3a40740 1 -- 172.21.15.12:0/1733013287 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5559b71b8170 con 0x5559b757eca0 2024-03-23T23:01:58.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.507+0000 7effbbe00640 1 -- 172.21.15.12:0/1733013287 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effd4003d10 con 0x5559b757eca0 2024-03-23T23:01:58.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.507+0000 7effbbe00640 1 -- 172.21.15.12:0/1733013287 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7effc80035f0 con 0x5559b757eca0 2024-03-23T23:01:58.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.507+0000 7effe3a40740 10 client.25309 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:01:58.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.507+0000 7effe3a40740 10 client.25309 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:01:58.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.507+0000 7effe3a40740 20 client.25309 populate_metadata read hostname 'smithi012' 2024-03-23T23:01:58.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.508+0000 7effe3a40740 1 --2- 172.21.15.12:0/1733013287 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5559b7594290 0x5559b75966b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.508+0000 7effe3a40740 4 client.25309 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:01:58.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.508+0000 7effe3a40740 1 -- 172.21.15.12:0/1733013287 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5559b74f13f0 con 0x5559b7594290 2024-03-23T23:01:58.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.508+0000 7effda800640 1 --2- 172.21.15.12:0/1733013287 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5559b7594290 0x5559b75966b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.508+0000 7effda800640 1 --2- 172.21.15.12:0/1733013287 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5559b7594290 0x5559b75966b0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33c0b2dd913bc4e9 server_cookie=acc22c7dcce7d32c in_seq=0 out_seq=0 2024-03-23T23:01:58.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.508+0000 7effbbe00640 10 client.25309 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:01:58.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.509+0000 7effbbe00640 1 -- 172.21.15.12:0/1733013287 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5559b74f13f0 con 0x5559b7594290 2024-03-23T23:01:58.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.509+0000 7effbbe00640 10 client.25309 handle_command_reply: tid=0 2024-03-23T23:01:58.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.509+0000 7effe3a40740 1 client.25309 shutdown 2024-03-23T23:01:58.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.509+0000 7effe3a40740 1 -- 172.21.15.12:0/1733013287 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559b757eca0 msgr2=0x5559b75889d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:58.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.509+0000 7effe3a40740 1 --2- 172.21.15.12:0/1733013287 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559b757eca0 0x5559b75889d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7effc802c240 tx=0x7effc8004270 comp rx=0 tx=0).stop 2024-03-23T23:01:58.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.509+0000 7effe3a40740 1 -- 172.21.15.12:0/1733013287 shutdown_connections 2024-03-23T23:01:58.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.509+0000 7effe3a40740 1 --2- 172.21.15.12:0/1733013287 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5559b7594290 0x5559b75966b0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.509+0000 7effe3a40740 1 --2- 172.21.15.12:0/1733013287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559b7583e10 0x5559b758fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.509+0000 7effe3a40740 1 --2- 172.21.15.12:0/1733013287 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559b757eca0 0x5559b75889d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.509+0000 7effe3a40740 1 --2- 172.21.15.12:0/1733013287 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559b757e2b0 0x5559b7588490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.509+0000 7effe3a40740 1 -- 172.21.15.12:0/1733013287 >> 172.21.15.12:0/1733013287 conn(0x5559b7579820 msgr2=0x5559b7578a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:58.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.509+0000 7effbbe00640 0 client.25309 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:01:58.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.510+0000 7effe3a40740 1 -- 172.21.15.12:0/1733013287 shutdown_connections 2024-03-23T23:01:58.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.510+0000 7effe3a40740 1 -- 172.21.15.12:0/1733013287 wait complete. 2024-03-23T23:01:58.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.510+0000 7effe3a40740 20 client.25309 trim_cache size 0 max 16384 2024-03-23T23:01:58.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.528+0000 7effe3a40740 1 Processor -- start 2024-03-23T23:01:58.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.528+0000 7effe3a40740 1 -- start start 2024-03-23T23:01:58.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.528+0000 7effe3a40740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559b757e2b0 0x5559b757f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.528+0000 7effe3a40740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559b757eca0 0x5559b758f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.528+0000 7effe3a40740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559b7583e10 0x5559b757fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.528+0000 7effe3a40740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc1aced0 con 0x5559b7583e10 2024-03-23T23:01:58.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.528+0000 7effe3a40740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc1aa9d0 con 0x5559b757eca0 2024-03-23T23:01:58.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.528+0000 7effe3a40740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc1a8a70 con 0x5559b757e2b0 2024-03-23T23:01:58.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.529+0000 7effda800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559b757e2b0 0x5559b757f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.529+0000 7effd9400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559b757eca0 0x5559b758f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.529+0000 7effda800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559b757e2b0 0x5559b757f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:37292/0 (socket says 172.21.15.12:37292) 2024-03-23T23:01:58.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.529+0000 7effd9400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559b757eca0 0x5559b758f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36612/0 (socket says 172.21.15.12:36612) 2024-03-23T23:01:58.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.529+0000 7effda800640 1 -- 172.21.15.12:0/3346373641 learned_addr learned my addr 172.21.15.12:0/3346373641 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:58.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.529+0000 7effda800640 1 --2- 172.21.15.12:0/3346373641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559b7583e10 0x5559b757fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.529+0000 7effd9400640 1 -- 172.21.15.12:0/3346373641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559b757e2b0 msgr2=0x5559b757f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:58.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.529+0000 7effd9400640 1 --2- 172.21.15.12:0/3346373641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559b757e2b0 0x5559b757f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.529+0000 7effd9400640 1 -- 172.21.15.12:0/3346373641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559b7583e10 msgr2=0x5559b757fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:58.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.529+0000 7effd9400640 1 --2- 172.21.15.12:0/3346373641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559b7583e10 0x5559b757fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.529+0000 7effd9400640 1 -- 172.21.15.12:0/3346373641 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559b7583130 con 0x5559b757eca0 2024-03-23T23:01:58.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.529+0000 7effd9400640 1 --2- 172.21.15.12:0/3346373641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559b757eca0 0x5559b758f9d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7effc80039e0 tx=0x7effc8000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f364e98d7eb2b733 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:58.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.530+0000 7effc6a00640 1 -- 172.21.15.12:0/3346373641 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8005d50 con 0x5559b757eca0 2024-03-23T23:01:58.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.530+0000 7effc6a00640 1 -- 172.21.15.12:0/3346373641 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effc8003bd0 con 0x5559b757eca0 2024-03-23T23:01:58.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.530+0000 7effc6a00640 1 -- 172.21.15.12:0/3346373641 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc804ba20 con 0x5559b757eca0 2024-03-23T23:01:58.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.531+0000 7effe3a40740 1 -- 172.21.15.12:0/3346373641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559b757eca0 msgr2=0x5559b758f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:58.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.531+0000 7effe3a40740 1 --2- 172.21.15.12:0/3346373641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559b757eca0 0x5559b758f9d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7effc80039e0 tx=0x7effc8000bc0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.531+0000 7effe3a40740 1 -- 172.21.15.12:0/3346373641 shutdown_connections 2024-03-23T23:01:58.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.531+0000 7effe3a40740 1 --2- 172.21.15.12:0/3346373641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559b7583e10 0x5559b757fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.531+0000 7effe3a40740 1 --2- 172.21.15.12:0/3346373641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559b757eca0 0x5559b758f9d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.531+0000 7effe3a40740 1 --2- 172.21.15.12:0/3346373641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559b757e2b0 0x5559b757f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.534 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-23T23:01:58.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.531+0000 7effe3a40740 1 -- 172.21.15.12:0/3346373641 >> 172.21.15.12:0/3346373641 conn(0x5559b7579820 msgr2=0x5559b757d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:58.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.531+0000 7effe3a40740 1 -- 172.21.15.12:0/3346373641 shutdown_connections 2024-03-23T23:01:58.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.531+0000 7effe3a40740 1 -- 172.21.15.12:0/3346373641 wait complete. 2024-03-23T23:01:58.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.531+0000 7effe3a40740 1 -- start start 2024-03-23T23:01:58.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.532+0000 7effe3a40740 1 Processor -- start 2024-03-23T23:01:58.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.532+0000 7effe3a40740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559b757e2b0 0x5559b758e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.532+0000 7effe3a40740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559b757eca0 0x5559b758e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.532+0000 7effe3a40740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559b7583e10 0x5559b758ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.532+0000 7effe3a40740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc1a9070 con 0x5559b757eca0 2024-03-23T23:01:58.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.532+0000 7effe3a40740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc1aa370 con 0x5559b7583e10 2024-03-23T23:01:58.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.532+0000 7effe3a40740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc1a96b0 con 0x5559b757e2b0 2024-03-23T23:01:58.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.532+0000 7effe3a40740 10 client.0 _ll_register_callbacks cb 0x7effc8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:01:58.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.532+0000 7effd9400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559b757eca0 0x5559b758e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.532+0000 7effda800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559b757e2b0 0x5559b758e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.532+0000 7effd9400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559b757eca0 0x5559b758e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45418/0 (socket says 172.21.15.12:45418) 2024-03-23T23:01:58.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.532+0000 7effd9400640 1 -- 172.21.15.12:0/1940324915 learned_addr learned my addr 172.21.15.12:0/1940324915 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:58.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.533+0000 7effda800640 1 --2- 172.21.15.12:0/1940324915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559b7583e10 0x5559b758ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.533+0000 7effd9400640 1 -- 172.21.15.12:0/1940324915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559b757e2b0 msgr2=0x5559b758e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:58.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.533+0000 7effd9400640 1 --2- 172.21.15.12:0/1940324915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559b757e2b0 0x5559b758e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.533+0000 7effd9400640 1 -- 172.21.15.12:0/1940324915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559b7583e10 msgr2=0x5559b758ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:58.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.533+0000 7effd9400640 1 --2- 172.21.15.12:0/1940324915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559b7583e10 0x5559b758ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.533+0000 7effd9400640 1 -- 172.21.15.12:0/1940324915 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5559b75efb40 con 0x5559b757eca0 2024-03-23T23:01:58.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.533+0000 7effda800640 1 --2- 172.21.15.12:0/1940324915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559b7583e10 0x5559b758ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:01:58.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.533+0000 7effd9400640 1 --2- 172.21.15.12:0/1940324915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559b757eca0 0x5559b758e6b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7effc802cda0 tx=0x7effc802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82103d4fe813b86f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:58.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.533+0000 7effbbe00640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:01:58.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.533+0000 7effbbe00640 1 -- 172.21.15.12:0/1940324915 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8005d50 con 0x5559b757eca0 2024-03-23T23:01:58.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.533+0000 7effe3a40740 1 -- 172.21.15.12:0/1940324915 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7effc8003bd0 con 0x5559b757eca0 2024-03-23T23:01:58.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.533+0000 7effbbe00640 1 -- 172.21.15.12:0/1940324915 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7effc804b920 con 0x5559b757eca0 2024-03-23T23:01:58.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.534+0000 7effbbe00640 1 -- 172.21.15.12:0/1940324915 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7effc804bb60 con 0x5559b757eca0 2024-03-23T23:01:58.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.534+0000 7effe3a40740 10 client.15201 fetch_fsmap learned FSMap version 23 2024-03-23T23:01:58.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.534+0000 7effe3a40740 1 -- 172.21.15.12:0/1940324915 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5559b75f0db0 con 0x5559b757eca0 2024-03-23T23:01:58.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.534+0000 7effbbe00640 1 -- 172.21.15.12:0/1940324915 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effc8003bd0 con 0x5559b757eca0 2024-03-23T23:01:58.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.534+0000 7effbbe00640 1 -- 172.21.15.12:0/1940324915 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7effc8004d80 con 0x5559b757eca0 2024-03-23T23:01:58.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.534+0000 7effe3a40740 10 client.15201 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:01:58.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.534+0000 7effe3a40740 10 client.15201 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:01:58.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.534+0000 7effe3a40740 20 client.15201 populate_metadata read hostname 'smithi012' 2024-03-23T23:01:58.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.534+0000 7effe3a40740 1 --2- 172.21.15.12:0/1940324915 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5559b75f1660 0x5559b75f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.534+0000 7effe3a40740 4 client.15201 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-23T23:01:58.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.534+0000 7effe3a40740 1 -- 172.21.15.12:0/1940324915 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5559b74eb740 con 0x5559b75f1660 2024-03-23T23:01:58.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.535+0000 7effda800640 1 --2- 172.21.15.12:0/1940324915 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5559b75f1660 0x5559b75f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.535+0000 7effda800640 1 --2- 172.21.15.12:0/1940324915 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5559b75f1660 0x5559b75f3a80 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=baca48fc81d51b47 server_cookie=99ac40c790f5c9e in_seq=0 out_seq=0 2024-03-23T23:01:58.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.535+0000 7effbbe00640 10 client.15201 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:01:58.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.537+0000 7effbbe00640 1 -- 172.21.15.12:0/1940324915 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5559b74eb740 con 0x5559b75f1660 2024-03-23T23:01:58.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.537+0000 7effbbe00640 10 client.15201 handle_command_reply: tid=0 2024-03-23T23:01:58.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.537+0000 7effe3a40740 1 client.15201 shutdown 2024-03-23T23:01:58.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.537+0000 7effe3a40740 1 -- 172.21.15.12:0/1940324915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559b757eca0 msgr2=0x5559b758e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:58.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.537+0000 7effe3a40740 1 --2- 172.21.15.12:0/1940324915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559b757eca0 0x5559b758e6b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7effc802cda0 tx=0x7effc802bc30 comp rx=0 tx=0).stop 2024-03-23T23:01:58.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.537+0000 7effe3a40740 1 -- 172.21.15.12:0/1940324915 shutdown_connections 2024-03-23T23:01:58.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.537+0000 7effe3a40740 1 --2- 172.21.15.12:0/1940324915 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5559b75f1660 0x5559b75f3a80 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.537+0000 7effe3a40740 1 --2- 172.21.15.12:0/1940324915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559b7583e10 0x5559b758ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.537+0000 7effe3a40740 1 --2- 172.21.15.12:0/1940324915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5559b757eca0 0x5559b758e6b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.537+0000 7effe3a40740 1 --2- 172.21.15.12:0/1940324915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559b757e2b0 0x5559b758e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.537+0000 7effe3a40740 1 -- 172.21.15.12:0/1940324915 >> 172.21.15.12:0/1940324915 conn(0x5559b7579820 msgr2=0x5559b75781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:58.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.537+0000 7effbbe00640 0 client.15201 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:01:58.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.537+0000 7effe3a40740 1 -- 172.21.15.12:0/1940324915 shutdown_connections 2024-03-23T23:01:58.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.537+0000 7effe3a40740 1 -- 172.21.15.12:0/1940324915 wait complete. 2024-03-23T23:01:58.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.537+0000 7effe3a40740 20 client.15201 trim_cache size 0 max 16384 2024-03-23T23:01:58.544 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:01:58.544 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-03-23T23:01:58.544 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "babab875-4a87-49c6-bb53-ffdd4f6dd1d8", 2024-03-23T23:01:58.544 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-03-23T23:01:58.544 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:01:58.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.538+0000 7effdb200640 1 -- 172.21.15.12:0/3056717471 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7effc00a2800 msgr2=0x7effc00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:58.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.538+0000 7effdb200640 1 --2- 172.21.15.12:0/3056717471 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7effc00a2800 0x7effc00a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7effcc0ccd60 tx=0x7effcc004a40 comp rx=0 tx=0).stop 2024-03-23T23:01:58.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.538+0000 7effdb200640 1 -- 172.21.15.12:0/3056717471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7effdc102000 msgr2=0x7effdc199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:58.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.538+0000 7effdb200640 1 --2- 172.21.15.12:0/3056717471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7effdc102000 0x7effdc199ad0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7effcc00e9f0 tx=0x7effcc00edf0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.538+0000 7effdb200640 1 -- 172.21.15.12:0/3056717471 shutdown_connections 2024-03-23T23:01:58.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.538+0000 7effdb200640 1 --2- 172.21.15.12:0/3056717471 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7effc00a2800 0x7effc00a4cc0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.538+0000 7effdb200640 1 --2- 172.21.15.12:0/3056717471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effdc107260 0x7effdc1a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.538+0000 7effdb200640 1 --2- 172.21.15.12:0/3056717471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effdc104930 0x7effdc19a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.538+0000 7effdb200640 1 --2- 172.21.15.12:0/3056717471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7effdc102000 0x7effdc199ad0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.538+0000 7effdb200640 1 -- 172.21.15.12:0/3056717471 >> 172.21.15.12:0/3056717471 conn(0x7effdc0fba10 msgr2=0x7effdc101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:58.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.538+0000 7effdb200640 1 -- 172.21.15.12:0/3056717471 shutdown_connections 2024-03-23T23:01:58.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.538+0000 7effdb200640 1 -- 172.21.15.12:0/3056717471 wait complete. 2024-03-23T23:01:58.556 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:01:58.556 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:01:58.579 INFO:tasks.workunit.client.0.smithi012.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-23T23:01:58.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.796+0000 7fc367e00640 1 -- 172.21.15.12:0/4042970415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc368108db0 msgr2=0x7fc368109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:58.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.796+0000 7fc367e00640 1 --2- 172.21.15.12:0/4042970415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc368108db0 0x7fc368109230 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc350009900 tx=0x7fc35002bc40 comp rx=0 tx=0).stop 2024-03-23T23:01:58.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.796+0000 7fc367e00640 1 -- 172.21.15.12:0/4042970415 shutdown_connections 2024-03-23T23:01:58.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.796+0000 7fc367e00640 1 --2- 172.21.15.12:0/4042970415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc368109770 0x7fc368110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.796+0000 7fc367e00640 1 --2- 172.21.15.12:0/4042970415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc368108db0 0x7fc368109230 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.796+0000 7fc367e00640 1 --2- 172.21.15.12:0/4042970415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc368107bc0 0x7fc368107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.796+0000 7fc367e00640 1 -- 172.21.15.12:0/4042970415 >> 172.21.15.12:0/4042970415 conn(0x7fc368076b70 msgr2=0x7fc368078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:58.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.797+0000 7fc367e00640 1 -- 172.21.15.12:0/4042970415 shutdown_connections 2024-03-23T23:01:58.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.797+0000 7fc367e00640 1 -- 172.21.15.12:0/4042970415 wait complete. 2024-03-23T23:01:58.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.798+0000 7fc367e00640 1 Processor -- start 2024-03-23T23:01:58.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.798+0000 7fc367e00640 1 -- start start 2024-03-23T23:01:58.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.798+0000 7fc367e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc368107bc0 0x7fc36819deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.798+0000 7fc367e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc368108db0 0x7fc36819e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.798+0000 7fc367e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc368109770 0x7fc3681a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.798+0000 7fc367e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc368113890 con 0x7fc368109770 2024-03-23T23:01:58.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.798+0000 7fc367e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc368113710 con 0x7fc368108db0 2024-03-23T23:01:58.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.798+0000 7fc367e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc368113a10 con 0x7fc368107bc0 2024-03-23T23:01:58.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.798+0000 7fc366000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc368108db0 0x7fc36819e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.798+0000 7fc366000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc368108db0 0x7fc36819e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36648/0 (socket says 172.21.15.12:36648) 2024-03-23T23:01:58.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.798+0000 7fc366000640 1 -- 172.21.15.12:0/3600349587 learned_addr learned my addr 172.21.15.12:0/3600349587 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:58.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.798+0000 7fc366a00640 1 --2- 172.21.15.12:0/3600349587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc368107bc0 0x7fc36819deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.799+0000 7fc367400640 1 --2- 172.21.15.12:0/3600349587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc368109770 0x7fc3681a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.799+0000 7fc366000640 1 -- 172.21.15.12:0/3600349587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc368107bc0 msgr2=0x7fc36819deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:58.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.799+0000 7fc366000640 1 --2- 172.21.15.12:0/3600349587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc368107bc0 0x7fc36819deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.799+0000 7fc366000640 1 -- 172.21.15.12:0/3600349587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc368109770 msgr2=0x7fc3681a5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:58.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.799+0000 7fc366000640 1 --2- 172.21.15.12:0/3600349587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc368109770 0x7fc3681a5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.799+0000 7fc366000640 1 -- 172.21.15.12:0/3600349587 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3681a5ad0 con 0x7fc368108db0 2024-03-23T23:01:58.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.799+0000 7fc366000640 1 --2- 172.21.15.12:0/3600349587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc368108db0 0x7fc36819e3f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc35002c8f0 tx=0x7fc35002c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:58.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.800+0000 7fc35f400640 1 -- 172.21.15.12:0/3600349587 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc350041070 con 0x7fc368108db0 2024-03-23T23:01:58.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.800+0000 7fc35f400640 1 -- 172.21.15.12:0/3600349587 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc350004450 con 0x7fc368108db0 2024-03-23T23:01:58.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.800+0000 7fc367e00640 1 -- 172.21.15.12:0/3600349587 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3681a5d60 con 0x7fc368108db0 2024-03-23T23:01:58.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.800+0000 7fc35f400640 1 -- 172.21.15.12:0/3600349587 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc350034440 con 0x7fc368108db0 2024-03-23T23:01:58.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.800+0000 7fc367e00640 1 -- 172.21.15.12:0/3600349587 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3681a6240 con 0x7fc368108db0 2024-03-23T23:01:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.803+0000 7fc370786740 1 Processor -- start 2024-03-23T23:01:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.803+0000 7fc370786740 1 -- start start 2024-03-23T23:01:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.803+0000 7fc370786740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f1f328e050 0x55f1f328e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.803+0000 7fc370786740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f328ea40 0x55f1f3293540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.804+0000 7fc370786740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f1f3293bb0 0x55f1f3296040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.804+0000 7fc370786740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1f31f3250 con 0x55f1f3293bb0 2024-03-23T23:01:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.804+0000 7fc370786740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1f319ba30 con 0x55f1f328ea40 2024-03-23T23:01:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.804+0000 7fc35f400640 1 -- 172.21.15.12:0/3600349587 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc350044070 con 0x7fc368108db0 2024-03-23T23:01:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.804+0000 7fc370786740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1f31ef2f0 con 0x55f1f328e050 2024-03-23T23:01:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.804+0000 7fc367400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f1f328e050 0x55f1f328e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.804+0000 7fc366a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f328ea40 0x55f1f3293540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.804+0000 7fc367400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f1f328e050 0x55f1f328e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:37328/0 (socket says 172.21.15.12:37328) 2024-03-23T23:01:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.804+0000 7fc367400640 1 -- 172.21.15.12:0/2578382447 learned_addr learned my addr 172.21.15.12:0/2578382447 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.804+0000 7fc366000640 1 --2- 172.21.15.12:0/2578382447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f1f3293bb0 0x55f1f3296040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.804+0000 7fc367400640 1 -- 172.21.15.12:0/2578382447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f328ea40 msgr2=0x55f1f3293540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.804+0000 7fc367400640 1 --2- 172.21.15.12:0/2578382447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f328ea40 0x55f1f3293540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.804+0000 7fc367400640 1 -- 172.21.15.12:0/2578382447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f1f3293bb0 msgr2=0x55f1f3296040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.804+0000 7fc367400640 1 --2- 172.21.15.12:0/2578382447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f1f3293bb0 0x55f1f3296040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.804+0000 7fc367400640 1 -- 172.21.15.12:0/2578382447 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1f3186690 con 0x55f1f328e050 2024-03-23T23:01:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.805+0000 7fc367400640 1 --2- 172.21.15.12:0/2578382447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f1f328e050 0x55f1f328e470 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc360009ca0 tx=0x7fc360011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30acf7f294e178d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:58.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.805+0000 7fc357e00640 1 -- 172.21.15.12:0/2578382447 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36001e070 con 0x55f1f328e050 2024-03-23T23:01:58.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.805+0000 7fc357e00640 1 -- 172.21.15.12:0/2578382447 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc360009e50 con 0x55f1f328e050 2024-03-23T23:01:58.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.805+0000 7fc357e00640 1 -- 172.21.15.12:0/2578382447 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36000eac0 con 0x55f1f328e050 2024-03-23T23:01:58.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.805+0000 7fc35f400640 1 --2- 172.21.15.12:0/3600349587 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc34c0a2880 0x7fc34c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.806+0000 7fc370786740 1 -- 172.21.15.12:0/2578382447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f1f328e050 msgr2=0x55f1f328e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:58.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.806+0000 7fc370786740 1 --2- 172.21.15.12:0/2578382447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f1f328e050 0x55f1f328e470 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc360009ca0 tx=0x7fc360011810 comp rx=0 tx=0).stop 2024-03-23T23:01:58.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.806+0000 7fc35f400640 1 -- 172.21.15.12:0/3600349587 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fc3500f0520 con 0x7fc368108db0 2024-03-23T23:01:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.806+0000 7fc370786740 1 -- 172.21.15.12:0/2578382447 shutdown_connections 2024-03-23T23:01:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.806+0000 7fc370786740 1 --2- 172.21.15.12:0/2578382447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f1f3293bb0 0x55f1f3296040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.806+0000 7fc366a00640 1 --2- 172.21.15.12:0/3600349587 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc34c0a2880 0x7fc34c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.806+0000 7fc370786740 1 --2- 172.21.15.12:0/2578382447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f328ea40 0x55f1f3293540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.806+0000 7fc370786740 1 --2- 172.21.15.12:0/2578382447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f1f328e050 0x55f1f328e470 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.806+0000 7fc370786740 1 -- 172.21.15.12:0/2578382447 >> 172.21.15.12:0/2578382447 conn(0x55f1f3256200 msgr2=0x55f1f32899c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.806+0000 7fc370786740 1 -- 172.21.15.12:0/2578382447 shutdown_connections 2024-03-23T23:01:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.806+0000 7fc370786740 1 -- 172.21.15.12:0/2578382447 wait complete. 2024-03-23T23:01:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.806+0000 7fc370786740 1 -- start start 2024-03-23T23:01:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.807+0000 7fc366a00640 1 --2- 172.21.15.12:0/3600349587 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc34c0a2880 0x7fc34c0a4d40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x55f1f328a910 tx=0x7fc358009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.807+0000 7fc370786740 1 Processor -- start 2024-03-23T23:01:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.807+0000 7fc370786740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f1f328e050 0x55f1f3298060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.807+0000 7fc370786740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f1f328ea40 0x55f1f32985a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.807+0000 7fc370786740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f3293bb0 0x55f1f329f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.807+0000 7fc370786740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1f3296b60 con 0x55f1f328ea40 2024-03-23T23:01:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.807+0000 7fc370786740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1f31aee80 con 0x55f1f3293bb0 2024-03-23T23:01:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.807+0000 7fc370786740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1f3296ce0 con 0x55f1f328e050 2024-03-23T23:01:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.807+0000 7fc370786740 10 client.0 _ll_register_callbacks cb 0x55f1f31c1130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:01:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.808+0000 7fc366a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f1f328e050 0x55f1f3298060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.808+0000 7fc367400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f3293bb0 0x55f1f329f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.808+0000 7fc366a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f1f328e050 0x55f1f3298060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:37330/0 (socket says 172.21.15.12:37330) 2024-03-23T23:01:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.808+0000 7fc366a00640 1 -- 172.21.15.12:0/3259382031 learned_addr learned my addr 172.21.15.12:0/3259382031 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.808+0000 7fc367400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f3293bb0 0x55f1f329f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36654/0 (socket says 172.21.15.12:36654) 2024-03-23T23:01:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.808+0000 7fc366000640 1 --2- 172.21.15.12:0/3259382031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f1f328ea40 0x55f1f32985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.808+0000 7fc366a00640 1 -- 172.21.15.12:0/3259382031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f3293bb0 msgr2=0x55f1f329f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.808+0000 7fc366a00640 1 --2- 172.21.15.12:0/3259382031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f3293bb0 0x55f1f329f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.808+0000 7fc366a00640 1 -- 172.21.15.12:0/3259382031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f1f328ea40 msgr2=0x55f1f32985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.808+0000 7fc366a00640 1 --2- 172.21.15.12:0/3259382031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f1f328ea40 0x55f1f32985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.808+0000 7fc366a00640 1 -- 172.21.15.12:0/3259382031 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f1f2fd17d0 con 0x55f1f328e050 2024-03-23T23:01:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.809+0000 7fc366a00640 1 --2- 172.21.15.12:0/3259382031 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f1f328e050 0x55f1f3298060 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc358011fd0 tx=0x7fc358004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62891348207f1821 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.809+0000 7fc354c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:01:58.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.809+0000 7fc354c00640 1 -- 172.21.15.12:0/3259382031 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc358004b20 con 0x55f1f328e050 2024-03-23T23:01:58.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.809+0000 7fc354c00640 1 -- 172.21.15.12:0/3259382031 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fc358004cc0 con 0x55f1f328e050 2024-03-23T23:01:58.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.809+0000 7fc370786740 1 -- 172.21.15.12:0/3259382031 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc360009e50 con 0x55f1f328e050 2024-03-23T23:01:58.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.809+0000 7fc354c00640 1 -- 172.21.15.12:0/3259382031 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc35800f810 con 0x55f1f328e050 2024-03-23T23:01:58.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.809+0000 7fc370786740 10 client.25327 fetch_fsmap learned FSMap version 23 2024-03-23T23:01:58.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.809+0000 7fc370786740 1 -- 172.21.15.12:0/3259382031 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f1f2f073f0 con 0x55f1f328e050 2024-03-23T23:01:58.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.809+0000 7fc354c00640 1 -- 172.21.15.12:0/3259382031 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc360009e50 con 0x55f1f328e050 2024-03-23T23:01:58.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.810+0000 7fc354c00640 1 -- 172.21.15.12:0/3259382031 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fc358026070 con 0x55f1f328e050 2024-03-23T23:01:58.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.810+0000 7fc370786740 10 client.25327 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:01:58.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.810+0000 7fc370786740 10 client.25327 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:01:58.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.810+0000 7fc370786740 20 client.25327 populate_metadata read hostname 'smithi012' 2024-03-23T23:01:58.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.810+0000 7fc370786740 1 --2- 172.21.15.12:0/3259382031 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f1f32a3f10 0x55f1f32a6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.810+0000 7fc370786740 4 client.25327 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:01:58.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.810+0000 7fc370786740 1 -- 172.21.15.12:0/3259382031 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f1f3200790 con 0x55f1f32a3f10 2024-03-23T23:01:58.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.810+0000 7fc366000640 1 --2- 172.21.15.12:0/3259382031 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f1f32a3f10 0x55f1f32a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.811+0000 7fc366000640 1 --2- 172.21.15.12:0/3259382031 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f1f32a3f10 0x55f1f32a6360 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6210119e91aae026 server_cookie=6c75c6481e83e04f in_seq=0 out_seq=0 2024-03-23T23:01:58.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.811+0000 7fc354c00640 10 client.25327 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:01:58.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.811+0000 7fc354c00640 1 -- 172.21.15.12:0/3259382031 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f1f3200790 con 0x55f1f32a3f10 2024-03-23T23:01:58.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.811+0000 7fc354c00640 10 client.25327 handle_command_reply: tid=0 2024-03-23T23:01:58.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.811+0000 7fc370786740 1 client.25327 shutdown 2024-03-23T23:01:58.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.812+0000 7fc370786740 1 -- 172.21.15.12:0/3259382031 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f1f328e050 msgr2=0x55f1f3298060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:58.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.812+0000 7fc370786740 1 --2- 172.21.15.12:0/3259382031 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f1f328e050 0x55f1f3298060 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc358011fd0 tx=0x7fc358004ab0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.812+0000 7fc370786740 1 -- 172.21.15.12:0/3259382031 shutdown_connections 2024-03-23T23:01:58.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.812+0000 7fc370786740 1 --2- 172.21.15.12:0/3259382031 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f1f32a3f10 0x55f1f32a6360 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.812+0000 7fc370786740 1 --2- 172.21.15.12:0/3259382031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f3293bb0 0x55f1f329f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.812+0000 7fc370786740 1 --2- 172.21.15.12:0/3259382031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f1f328ea40 0x55f1f32985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.812+0000 7fc354c00640 0 client.25327 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:01:58.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.812+0000 7fc370786740 1 --2- 172.21.15.12:0/3259382031 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f1f328e050 0x55f1f3298060 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.812+0000 7fc370786740 1 -- 172.21.15.12:0/3259382031 >> 172.21.15.12:0/3259382031 conn(0x55f1f3256200 msgr2=0x55f1f3288bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:58.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.812+0000 7fc370786740 1 -- 172.21.15.12:0/3259382031 shutdown_connections 2024-03-23T23:01:58.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.812+0000 7fc370786740 1 -- 172.21.15.12:0/3259382031 wait complete. 2024-03-23T23:01:58.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.812+0000 7fc370786740 20 client.25327 trim_cache size 0 max 16384 2024-03-23T23:01:58.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.833+0000 7fc370786740 1 Processor -- start 2024-03-23T23:01:58.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.833+0000 7fc370786740 1 -- start start 2024-03-23T23:01:58.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.833+0000 7fc370786740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f328e050 0x55f1f328fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.833+0000 7fc370786740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f1f328ea40 0x55f1f329f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.833+0000 7fc370786740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f1f3293bb0 0x55f1f3288a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.833+0000 7fc370786740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3681b12f0 con 0x55f1f328ea40 2024-03-23T23:01:58.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.833+0000 7fc370786740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3681aedf0 con 0x55f1f328e050 2024-03-23T23:01:58.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.833+0000 7fc370786740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3681ace90 con 0x55f1f3293bb0 2024-03-23T23:01:58.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.834+0000 7fc366a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f328e050 0x55f1f328fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.834+0000 7fc367400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f1f3293bb0 0x55f1f3288a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.834+0000 7fc366a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f328e050 0x55f1f328fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36662/0 (socket says 172.21.15.12:36662) 2024-03-23T23:01:58.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.834+0000 7fc366a00640 1 -- 172.21.15.12:0/2436113100 learned_addr learned my addr 172.21.15.12:0/2436113100 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:58.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.834+0000 7fc366000640 1 --2- 172.21.15.12:0/2436113100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f1f328ea40 0x55f1f329f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.834+0000 7fc366a00640 1 -- 172.21.15.12:0/2436113100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f1f3293bb0 msgr2=0x55f1f3288a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:58.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.834+0000 7fc366a00640 1 --2- 172.21.15.12:0/2436113100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f1f3293bb0 0x55f1f3288a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.834+0000 7fc366a00640 1 -- 172.21.15.12:0/2436113100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f1f328ea40 msgr2=0x55f1f329f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:58.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.834+0000 7fc366a00640 1 --2- 172.21.15.12:0/2436113100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f1f328ea40 0x55f1f329f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.834+0000 7fc366a00640 1 -- 172.21.15.12:0/2436113100 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1f2ec70f0 con 0x55f1f328e050 2024-03-23T23:01:58.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.834+0000 7fc366000640 1 --2- 172.21.15.12:0/2436113100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f1f328ea40 0x55f1f329f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:01:58.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.834+0000 7fc366a00640 1 --2- 172.21.15.12:0/2436113100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f328e050 0x55f1f328fcd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc358004010 tx=0x7fc35801c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6137ac4c17445def server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:58.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.835+0000 7fc357400640 1 -- 172.21.15.12:0/2436113100 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc358036030 con 0x55f1f328e050 2024-03-23T23:01:58.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.835+0000 7fc357400640 1 -- 172.21.15.12:0/2436113100 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc35802d030 con 0x55f1f328e050 2024-03-23T23:01:58.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.835+0000 7fc357400640 1 -- 172.21.15.12:0/2436113100 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35801e8e0 con 0x55f1f328e050 2024-03-23T23:01:58.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.835+0000 7fc370786740 1 -- 172.21.15.12:0/2436113100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f328e050 msgr2=0x55f1f328fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:58.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.835+0000 7fc370786740 1 --2- 172.21.15.12:0/2436113100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f328e050 0x55f1f328fcd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc358004010 tx=0x7fc35801c480 comp rx=0 tx=0).stop 2024-03-23T23:01:58.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.835+0000 7fc370786740 1 -- 172.21.15.12:0/2436113100 shutdown_connections 2024-03-23T23:01:58.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.835+0000 7fc370786740 1 --2- 172.21.15.12:0/2436113100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f1f3293bb0 0x55f1f3288a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.835+0000 7fc370786740 1 --2- 172.21.15.12:0/2436113100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f1f328ea40 0x55f1f329f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.835+0000 7fc370786740 1 --2- 172.21.15.12:0/2436113100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f328e050 0x55f1f328fcd0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.835+0000 7fc370786740 1 -- 172.21.15.12:0/2436113100 >> 172.21.15.12:0/2436113100 conn(0x55f1f3256200 msgr2=0x55f1f3288330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:58.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.835+0000 7fc370786740 1 -- 172.21.15.12:0/2436113100 shutdown_connections 2024-03-23T23:01:58.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.835+0000 7fc370786740 1 -- 172.21.15.12:0/2436113100 wait complete. 2024-03-23T23:01:58.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.836+0000 7fc370786740 1 -- start start 2024-03-23T23:01:58.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.836+0000 7fc370786740 1 Processor -- start 2024-03-23T23:01:58.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.836+0000 7fc370786740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f1f328e050 0x55f1f329d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.836+0000 7fc370786740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f328ea40 0x55f1f329da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.836+0000 7fc370786740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f1f3293bb0 0x55f1f329ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.837+0000 7fc370786740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3681ad490 con 0x55f1f328e050 2024-03-23T23:01:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.837+0000 7fc370786740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3681ae790 con 0x55f1f328ea40 2024-03-23T23:01:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.837+0000 7fc370786740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3681adad0 con 0x55f1f3293bb0 2024-03-23T23:01:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.837+0000 7fc370786740 10 client.0 _ll_register_callbacks cb 0x7fc358003970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:01:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.837+0000 7fc366a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f1f328e050 0x55f1f329d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.837+0000 7fc366a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f1f328e050 0x55f1f329d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45466/0 (socket says 172.21.15.12:45466) 2024-03-23T23:01:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.837+0000 7fc366a00640 1 -- 172.21.15.12:0/2685571521 learned_addr learned my addr 172.21.15.12:0/2685571521 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:01:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.837+0000 7fc366000640 1 --2- 172.21.15.12:0/2685571521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f328ea40 0x55f1f329da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.837+0000 7fc367400640 1 --2- 172.21.15.12:0/2685571521 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f1f3293bb0 0x55f1f329ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.837+0000 7fc366a00640 1 -- 172.21.15.12:0/2685571521 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f1f3293bb0 msgr2=0x55f1f329ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.837+0000 7fc366a00640 1 --2- 172.21.15.12:0/2685571521 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f1f3293bb0 0x55f1f329ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.837+0000 7fc366a00640 1 -- 172.21.15.12:0/2685571521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f328ea40 msgr2=0x55f1f329da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:01:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.837+0000 7fc366a00640 1 --2- 172.21.15.12:0/2685571521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f328ea40 0x55f1f329da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.837+0000 7fc366a00640 1 -- 172.21.15.12:0/2685571521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f1f329f060 con 0x55f1f328e050 2024-03-23T23:01:58.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.837+0000 7fc366000640 1 --2- 172.21.15.12:0/2685571521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f328ea40 0x55f1f329da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:01:58.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.838+0000 7fc366a00640 1 --2- 172.21.15.12:0/2685571521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f1f328e050 0x55f1f329d520 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc35801e470 tx=0x7fc3580114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad839553fd75e574 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:01:58.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.838+0000 7fc354c00640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:01:58.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.838+0000 7fc354c00640 1 -- 172.21.15.12:0/2685571521 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35803f060 con 0x55f1f328e050 2024-03-23T23:01:58.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.838+0000 7fc354c00640 1 -- 172.21.15.12:0/2685571521 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fc35801e620 con 0x55f1f328e050 2024-03-23T23:01:58.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.838+0000 7fc370786740 1 -- 172.21.15.12:0/2685571521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc35802d030 con 0x55f1f328e050 2024-03-23T23:01:58.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.838+0000 7fc354c00640 1 -- 172.21.15.12:0/2685571521 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc358036070 con 0x55f1f328e050 2024-03-23T23:01:58.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.838+0000 7fc370786740 10 client.15219 fetch_fsmap learned FSMap version 23 2024-03-23T23:01:58.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.838+0000 7fc370786740 1 -- 172.21.15.12:0/2685571521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f1f32fffc0 con 0x55f1f328e050 2024-03-23T23:01:58.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.839+0000 7fc354c00640 1 -- 172.21.15.12:0/2685571521 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc35802d030 con 0x55f1f328e050 2024-03-23T23:01:58.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.839+0000 7fc354c00640 1 -- 172.21.15.12:0/2685571521 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fc358026040 con 0x55f1f328e050 2024-03-23T23:01:58.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.839+0000 7fc370786740 10 client.15219 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:01:58.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.839+0000 7fc370786740 10 client.15219 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:01:58.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.839+0000 7fc370786740 20 client.15219 populate_metadata read hostname 'smithi012' 2024-03-23T23:01:58.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.839+0000 7fc370786740 1 --2- 172.21.15.12:0/2685571521 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f1f3300860 0x55f1f3302c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:01:58.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.839+0000 7fc370786740 4 client.15219 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:01:58.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.839+0000 7fc370786740 1 -- 172.21.15.12:0/2685571521 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f1f31faae0 con 0x55f1f3300860 2024-03-23T23:01:58.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.839+0000 7fc366000640 1 --2- 172.21.15.12:0/2685571521 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f1f3300860 0x55f1f3302c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:01:58.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.840+0000 7fc366000640 1 --2- 172.21.15.12:0/2685571521 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f1f3300860 0x55f1f3302c80 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6cfa3fb78f1d17f server_cookie=12449ab5e3002f00 in_seq=0 out_seq=0 2024-03-23T23:01:58.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.840+0000 7fc354c00640 10 client.15219 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:01:58.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.840+0000 7fc354c00640 1 -- 172.21.15.12:0/2685571521 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55f1f31faae0 con 0x55f1f3300860 2024-03-23T23:01:58.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.840+0000 7fc354c00640 10 client.15219 handle_command_reply: tid=0 2024-03-23T23:01:58.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.840+0000 7fc370786740 1 client.15219 shutdown 2024-03-23T23:01:58.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.840+0000 7fc370786740 1 -- 172.21.15.12:0/2685571521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f1f328e050 msgr2=0x55f1f329d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:58.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.840+0000 7fc370786740 1 --2- 172.21.15.12:0/2685571521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f1f328e050 0x55f1f329d520 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc35801e470 tx=0x7fc3580114b0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.840+0000 7fc370786740 1 -- 172.21.15.12:0/2685571521 shutdown_connections 2024-03-23T23:01:58.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.840+0000 7fc370786740 1 --2- 172.21.15.12:0/2685571521 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f1f3300860 0x55f1f3302c80 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.840+0000 7fc370786740 1 --2- 172.21.15.12:0/2685571521 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f1f3293bb0 0x55f1f329ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.840+0000 7fc370786740 1 --2- 172.21.15.12:0/2685571521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f328ea40 0x55f1f329da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.840+0000 7fc354c00640 0 client.15219 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:01:58.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.840+0000 7fc370786740 1 --2- 172.21.15.12:0/2685571521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f1f328e050 0x55f1f329d520 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.840+0000 7fc370786740 1 -- 172.21.15.12:0/2685571521 >> 172.21.15.12:0/2685571521 conn(0x55f1f3256200 msgr2=0x55f1f328fb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:58.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.840+0000 7fc370786740 1 -- 172.21.15.12:0/2685571521 shutdown_connections 2024-03-23T23:01:58.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.841+0000 7fc370786740 1 -- 172.21.15.12:0/2685571521 wait complete. 2024-03-23T23:01:58.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.841+0000 7fc370786740 20 client.15219 trim_cache size 0 max 16384 2024-03-23T23:01:58.845 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:01:58.845 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T23:01:58.845 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-03-23T23:01:58.845 INFO:teuthology.orchestra.run.smithi012.stdout: "babab875-4a87-49c6-bb53-ffdd4f6dd1d8": { 2024-03-23T23:01:58.845 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-03-23T23:01:58.845 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "babab875-4a87-49c6-bb53-ffdd4f6dd1d8", 2024-03-23T23:01:58.845 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-03-23T23:01:58.845 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:01:58.846 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:01:58.846 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:01:58.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.841+0000 7fc367e00640 1 -- 172.21.15.12:0/3600349587 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc34c0a2880 msgr2=0x7fc34c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:58.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.841+0000 7fc367e00640 1 --2- 172.21.15.12:0/3600349587 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc34c0a2880 0x7fc34c0a4d40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x55f1f328a910 tx=0x7fc358009500 comp rx=0 tx=0).stop 2024-03-23T23:01:58.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.841+0000 7fc367e00640 1 -- 172.21.15.12:0/3600349587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc368108db0 msgr2=0x7fc36819e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:01:58.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.841+0000 7fc367e00640 1 --2- 172.21.15.12:0/3600349587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc368108db0 0x7fc36819e3f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc35002c8f0 tx=0x7fc35002c660 comp rx=0 tx=0).stop 2024-03-23T23:01:58.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.841+0000 7fc367e00640 1 -- 172.21.15.12:0/3600349587 shutdown_connections 2024-03-23T23:01:58.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.841+0000 7fc367e00640 1 --2- 172.21.15.12:0/3600349587 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc34c0a2880 0x7fc34c0a4d40 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.841+0000 7fc367e00640 1 --2- 172.21.15.12:0/3600349587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc368109770 0x7fc3681a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.841+0000 7fc367e00640 1 --2- 172.21.15.12:0/3600349587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc368108db0 0x7fc36819e3f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.841+0000 7fc367e00640 1 --2- 172.21.15.12:0/3600349587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc368107bc0 0x7fc36819deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:01:58.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.842+0000 7fc367e00640 1 -- 172.21.15.12:0/3600349587 >> 172.21.15.12:0/3600349587 conn(0x7fc368076b70 msgr2=0x7fc36810c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:01:58.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.842+0000 7fc367e00640 1 -- 172.21.15.12:0/3600349587 shutdown_connections 2024-03-23T23:01:58.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:01:58.842+0000 7fc367e00640 1 -- 172.21.15.12:0/3600349587 wait complete. 2024-03-23T23:01:58.859 INFO:tasks.cephfs.filesystem:scrub status for tag:babab875-4a87-49c6-bb53-ffdd4f6dd1d8 - {'path': '/', 'tag': 'babab875-4a87-49c6-bb53-ffdd4f6dd1d8', 'options': 'recursive,force'} 2024-03-23T23:01:59.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:59 smithi012 ceph-mon[31695]: pgmap v395: 97 pgs: 97 active+clean; 643 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 6 op/s 2024-03-23T23:01:59.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:59 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:01:59.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:59 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:01:59.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:59 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:59.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:59 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3765090095' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:59.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:59 smithi012 ceph-mon[31695]: scrub queued for path: / 2024-03-23T23:01:59.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:59 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:01:59.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:59 smithi012 ceph-mon[31695]: scrub summary: active paths [/] 2024-03-23T23:01:59.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:01:59 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:01:59.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:59 smithi173 ceph-mon[36116]: pgmap v395: 97 pgs: 97 active+clean; 643 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 6 op/s 2024-03-23T23:01:59.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:59 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:01:59.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:59 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:01:59.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:59 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:59.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:59 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3765090095' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:59.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:59 smithi173 ceph-mon[36116]: scrub queued for path: / 2024-03-23T23:01:59.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:59 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:01:59.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:59 smithi173 ceph-mon[36116]: scrub summary: active paths [/] 2024-03-23T23:01:59.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:01:59 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:01:59.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:59 smithi203 ceph-mon[36693]: pgmap v395: 97 pgs: 97 active+clean; 643 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 6 op/s 2024-03-23T23:01:59.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:59 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:01:59.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:59 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:01:59.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:59 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:01:59.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:59 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3765090095' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:01:59.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:59 smithi203 ceph-mon[36693]: scrub queued for path: / 2024-03-23T23:01:59.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:59 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:01:59.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:59 smithi203 ceph-mon[36693]: scrub summary: active paths [/] 2024-03-23T23:01:59.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:01:59 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:02:01.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:01 smithi173 ceph-mon[36116]: pgmap v396: 97 pgs: 97 active+clean; 643 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-23T23:02:01.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:01 smithi173 ceph-mon[36116]: scrub summary: idle 2024-03-23T23:02:01.409 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:01 smithi203 ceph-mon[36693]: pgmap v396: 97 pgs: 97 active+clean; 643 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-23T23:02:01.409 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:01 smithi203 ceph-mon[36693]: scrub summary: idle 2024-03-23T23:02:01.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:01 smithi012 ceph-mon[31695]: pgmap v396: 97 pgs: 97 active+clean; 643 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-23T23:02:01.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:01 smithi012 ceph-mon[31695]: scrub summary: idle 2024-03-23T23:02:03.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:03 smithi173 ceph-mon[36116]: pgmap v397: 97 pgs: 97 active+clean; 643 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1023 B/s wr, 7 op/s 2024-03-23T23:02:03.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:03 smithi203 ceph-mon[36693]: pgmap v397: 97 pgs: 97 active+clean; 643 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1023 B/s wr, 7 op/s 2024-03-23T23:02:03.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:03 smithi012 ceph-mon[31695]: pgmap v397: 97 pgs: 97 active+clean; 643 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1023 B/s wr, 7 op/s 2024-03-23T23:02:05.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:05 smithi173 ceph-mon[36116]: pgmap v398: 97 pgs: 97 active+clean; 649 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-23T23:02:05.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:05 smithi203 ceph-mon[36693]: pgmap v398: 97 pgs: 97 active+clean; 649 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-23T23:02:05.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:05 smithi012 ceph-mon[31695]: pgmap v398: 97 pgs: 97 active+clean; 649 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-23T23:02:07.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:07 smithi173 ceph-mon[36116]: pgmap v399: 97 pgs: 97 active+clean; 649 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-23T23:02:07.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:07 smithi203 ceph-mon[36693]: pgmap v399: 97 pgs: 97 active+clean; 649 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-23T23:02:07.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:07 smithi012 ceph-mon[31695]: pgmap v399: 97 pgs: 97 active+clean; 649 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-23T23:02:09.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:09 smithi173 ceph-mon[36116]: pgmap v400: 97 pgs: 97 active+clean; 649 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-23T23:02:09.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:09 smithi203 ceph-mon[36693]: pgmap v400: 97 pgs: 97 active+clean; 649 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-23T23:02:09.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:09 smithi012 ceph-mon[31695]: pgmap v400: 97 pgs: 97 active+clean; 649 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-23T23:02:11.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:11 smithi173 ceph-mon[36116]: pgmap v401: 97 pgs: 97 active+clean; 649 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-23T23:02:11.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:11 smithi203 ceph-mon[36693]: pgmap v401: 97 pgs: 97 active+clean; 649 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-23T23:02:11.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.546+0000 7f0cbe600640 1 -- 172.21.15.12:0/147799699 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cb8103720 msgr2=0x7f0cb8103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:11.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.546+0000 7f0cbe600640 1 --2- 172.21.15.12:0/147799699 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cb8103720 0x7f0cb8103b40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8009900 tx=0x7f0ca802bc50 comp rx=0 tx=0).stop 2024-03-23T23:02:11.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.546+0000 7f0cbe600640 1 -- 172.21.15.12:0/147799699 shutdown_connections 2024-03-23T23:02:11.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.546+0000 7f0cbe600640 1 --2- 172.21.15.12:0/147799699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cb81052d0 0x7f0cb810bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.546+0000 7f0cbe600640 1 --2- 172.21.15.12:0/147799699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0cb8104910 0x7f0cb8104d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.546+0000 7f0cbe600640 1 --2- 172.21.15.12:0/147799699 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cb8103720 0x7f0cb8103b40 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.546+0000 7f0cbe600640 1 -- 172.21.15.12:0/147799699 >> 172.21.15.12:0/147799699 conn(0x7f0cb80fecb0 msgr2=0x7f0cb81010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:11.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.546+0000 7f0cbe600640 1 -- 172.21.15.12:0/147799699 shutdown_connections 2024-03-23T23:02:11.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.546+0000 7f0cbe600640 1 -- 172.21.15.12:0/147799699 wait complete. 2024-03-23T23:02:11.548 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.548+0000 7f0cbe600640 1 Processor -- start 2024-03-23T23:02:11.548 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.548+0000 7f0cbe600640 1 -- start start 2024-03-23T23:02:11.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.548+0000 7f0cbe600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0cb8103720 0x7f0cb81999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:11.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.548+0000 7f0cbe600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cb8104910 0x7f0cb8199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:11.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.548+0000 7f0cbe600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cb81052d0 0x7f0cb81a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:11.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.548+0000 7f0cbe600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb810f3c0 con 0x7f0cb8103720 2024-03-23T23:02:11.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.548+0000 7f0cbe600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb810f240 con 0x7f0cb81052d0 2024-03-23T23:02:11.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.548+0000 7f0cbe600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb810f540 con 0x7f0cb8104910 2024-03-23T23:02:11.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.548+0000 7f0cb7e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cb8104910 0x7f0cb8199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:11.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.549+0000 7f0cbdc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cb81052d0 0x7f0cb81a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:11.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.549+0000 7f0cbd200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0cb8103720 0x7f0cb81999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:11.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.549+0000 7f0cb7e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cb8104910 0x7f0cb8199f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:43368/0 (socket says 172.21.15.12:43368) 2024-03-23T23:02:11.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.549+0000 7f0cb7e00640 1 -- 172.21.15.12:0/822005271 learned_addr learned my addr 172.21.15.12:0/822005271 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:11.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.549+0000 7f0cb7e00640 1 -- 172.21.15.12:0/822005271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cb81052d0 msgr2=0x7f0cb81a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:11.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.549+0000 7f0cb7e00640 1 --2- 172.21.15.12:0/822005271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cb81052d0 0x7f0cb81a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.550 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.549+0000 7f0cb7e00640 1 -- 172.21.15.12:0/822005271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0cb8103720 msgr2=0x7f0cb81999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:11.550 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.549+0000 7f0cb7e00640 1 --2- 172.21.15.12:0/822005271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0cb8103720 0x7f0cb81999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.550 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.549+0000 7f0cb7e00640 1 -- 172.21.15.12:0/822005271 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb81a1600 con 0x7f0cb8104910 2024-03-23T23:02:11.550 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.549+0000 7f0cb7e00640 1 --2- 172.21.15.12:0/822005271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cb8104910 0x7f0cb8199f20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0ca400e9f0 tx=0x7f0ca400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:11.550 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.549+0000 7f0cb5600640 1 -- 172.21.15.12:0/822005271 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca400cd80 con 0x7f0cb8104910 2024-03-23T23:02:11.550 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.550+0000 7f0cbe600640 1 -- 172.21.15.12:0/822005271 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb81a18f0 con 0x7f0cb8104910 2024-03-23T23:02:11.550 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.550+0000 7f0cb5600640 1 -- 172.21.15.12:0/822005271 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ca4010910 con 0x7f0cb8104910 2024-03-23T23:02:11.550 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.550+0000 7f0cb5600640 1 -- 172.21.15.12:0/822005271 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca40188f0 con 0x7f0cb8104910 2024-03-23T23:02:11.550 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.550+0000 7f0cbe600640 1 -- 172.21.15.12:0/822005271 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb81a1e40 con 0x7f0cb8104910 2024-03-23T23:02:11.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.553+0000 7f0cc2985740 1 Processor -- start 2024-03-23T23:02:11.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.553+0000 7f0cc2985740 1 -- start start 2024-03-23T23:02:11.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.553+0000 7f0cc2985740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649ca1293f0 0x5649ca129810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:11.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.553+0000 7f0cc2985740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca129de0 0x5649ca12e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:11.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.553+0000 7f0cc2985740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5649ca12ef50 0x5649ca1313e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:11.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.553+0000 7f0cc2985740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649ca0ec3b0 con 0x5649ca12ef50 2024-03-23T23:02:11.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.553+0000 7f0cc2985740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649ca08e830 con 0x5649ca129de0 2024-03-23T23:02:11.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.553+0000 7f0cb5600640 1 -- 172.21.15.12:0/822005271 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0ca4010ac0 con 0x7f0cb8104910 2024-03-23T23:02:11.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.553+0000 7f0cc2985740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649ca0375d0 con 0x5649ca1293f0 2024-03-23T23:02:11.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.554+0000 7f0cbdc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649ca1293f0 0x5649ca129810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:11.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.554+0000 7f0cbd200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca129de0 0x5649ca12e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:11.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.554+0000 7f0cbd200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca129de0 0x5649ca12e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:49020/0 (socket says 172.21.15.12:49020) 2024-03-23T23:02:11.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.554+0000 7f0cbdc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649ca1293f0 0x5649ca129810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:43378/0 (socket says 172.21.15.12:43378) 2024-03-23T23:02:11.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.554+0000 7f0cbd200640 1 -- 172.21.15.12:0/618642253 learned_addr learned my addr 172.21.15.12:0/618642253 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:11.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.554+0000 7f0cb7e00640 1 --2- 172.21.15.12:0/618642253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5649ca12ef50 0x5649ca1313e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:11.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.554+0000 7f0cbd200640 1 -- 172.21.15.12:0/618642253 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649ca1293f0 msgr2=0x5649ca129810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:11.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.554+0000 7f0cbd200640 1 --2- 172.21.15.12:0/618642253 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649ca1293f0 0x5649ca129810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.554+0000 7f0cbd200640 1 -- 172.21.15.12:0/618642253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5649ca12ef50 msgr2=0x5649ca1313e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:11.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:11 smithi012 ceph-mon[31695]: pgmap v401: 97 pgs: 97 active+clean; 649 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-23T23:02:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.554+0000 7f0cbd200640 1 --2- 172.21.15.12:0/618642253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5649ca12ef50 0x5649ca1313e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.554+0000 7f0cbd200640 1 -- 172.21.15.12:0/618642253 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649ca04f2c0 con 0x5649ca129de0 2024-03-23T23:02:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.554+0000 7f0cb7e00640 1 --2- 172.21.15.12:0/618642253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5649ca12ef50 0x5649ca1313e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:02:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.555+0000 7f0cbd200640 1 --2- 172.21.15.12:0/618642253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca129de0 0x5649ca12e8e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0ca802c260 tx=0x7f0ca8009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afe132b7140f0f7f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.555+0000 7f0cae000640 1 -- 172.21.15.12:0/618642253 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca8042070 con 0x5649ca129de0 2024-03-23T23:02:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.555+0000 7f0cae000640 1 -- 172.21.15.12:0/618642253 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ca802cba0 con 0x5649ca129de0 2024-03-23T23:02:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.555+0000 7f0cae000640 1 -- 172.21.15.12:0/618642253 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca8005110 con 0x5649ca129de0 2024-03-23T23:02:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.555+0000 7f0cb5600640 1 --2- 172.21.15.12:0/822005271 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0c9c0a27f0 0x7f0c9c0a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.555+0000 7f0cb5600640 1 -- 172.21.15.12:0/822005271 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0ca40cee50 con 0x7f0cb8104910 2024-03-23T23:02:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.555+0000 7f0cbdc00640 1 --2- 172.21.15.12:0/822005271 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0c9c0a27f0 0x7f0c9c0a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.556+0000 7f0cc2985740 1 -- 172.21.15.12:0/618642253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca129de0 msgr2=0x5649ca12e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.556+0000 7f0cc2985740 1 --2- 172.21.15.12:0/618642253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca129de0 0x5649ca12e8e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0ca802c260 tx=0x7f0ca8009560 comp rx=0 tx=0).stop 2024-03-23T23:02:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.556+0000 7f0cc2985740 1 -- 172.21.15.12:0/618642253 shutdown_connections 2024-03-23T23:02:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.556+0000 7f0cc2985740 1 --2- 172.21.15.12:0/618642253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5649ca12ef50 0x5649ca1313e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.556+0000 7f0cc2985740 1 --2- 172.21.15.12:0/618642253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca129de0 0x5649ca12e8e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.556+0000 7f0cc2985740 1 --2- 172.21.15.12:0/618642253 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649ca1293f0 0x5649ca129810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.556+0000 7f0cc2985740 1 -- 172.21.15.12:0/618642253 >> 172.21.15.12:0/618642253 conn(0x5649ca124960 msgr2=0x5649ca124d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.556+0000 7f0cc2985740 1 -- 172.21.15.12:0/618642253 shutdown_connections 2024-03-23T23:02:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.556+0000 7f0cbdc00640 1 --2- 172.21.15.12:0/822005271 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0c9c0a27f0 0x7f0c9c0a4cb0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x5649ca12a640 tx=0x7f0cb0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.556+0000 7f0cc2985740 1 -- 172.21.15.12:0/618642253 wait complete. 2024-03-23T23:02:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.557+0000 7f0cc2985740 1 -- start start 2024-03-23T23:02:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.557+0000 7f0cc2985740 1 Processor -- start 2024-03-23T23:02:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.557+0000 7f0cc2985740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649ca1293f0 0x5649ca133570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.557+0000 7f0cc2985740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca129de0 0x5649ca133ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.557+0000 7f0cc2985740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5649ca12ef50 0x5649ca13aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.557+0000 7f0cc2985740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649ca04aa20 con 0x5649ca12ef50 2024-03-23T23:02:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.557+0000 7f0cc2985740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649ca08a8d0 con 0x5649ca129de0 2024-03-23T23:02:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.557+0000 7f0cc2985740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649ca131ea0 con 0x5649ca1293f0 2024-03-23T23:02:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.557+0000 7f0cc2985740 10 client.0 _ll_register_callbacks cb 0x5649ca0871c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:02:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.558+0000 7f0cb7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca129de0 0x5649ca133ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.558+0000 7f0cbd200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649ca1293f0 0x5649ca133570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.558+0000 7f0cb7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca129de0 0x5649ca133ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:49028/0 (socket says 172.21.15.12:49028) 2024-03-23T23:02:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.558+0000 7f0cb7e00640 1 -- 172.21.15.12:0/2412697133 learned_addr learned my addr 172.21.15.12:0/2412697133 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.558+0000 7f0cbdc00640 1 --2- 172.21.15.12:0/2412697133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5649ca12ef50 0x5649ca13aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.558+0000 7f0cb7e00640 1 -- 172.21.15.12:0/2412697133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649ca1293f0 msgr2=0x5649ca133570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.558+0000 7f0cb7e00640 1 --2- 172.21.15.12:0/2412697133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649ca1293f0 0x5649ca133570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.558+0000 7f0cb7e00640 1 -- 172.21.15.12:0/2412697133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5649ca12ef50 msgr2=0x5649ca13aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.558+0000 7f0cb7e00640 1 --2- 172.21.15.12:0/2412697133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5649ca12ef50 0x5649ca13aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.558+0000 7f0cb7e00640 1 -- 172.21.15.12:0/2412697133 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5649c9da3210 con 0x5649ca129de0 2024-03-23T23:02:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.558+0000 7f0cbd200640 1 --2- 172.21.15.12:0/2412697133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649ca1293f0 0x5649ca133570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:02:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.558+0000 7f0cbdc00640 1 --2- 172.21.15.12:0/2412697133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5649ca12ef50 0x5649ca13aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:02:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.558+0000 7f0cb7e00640 1 --2- 172.21.15.12:0/2412697133 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca129de0 0x5649ca133ab0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0ca40d2470 tx=0x7f0ca409b1b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a3f208a03e6b486 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.558+0000 7f0ca2a00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:02:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.558+0000 7f0ca2a00640 1 -- 172.21.15.12:0/2412697133 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca409f8f0 con 0x5649ca129de0 2024-03-23T23:02:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.558+0000 7f0ca2a00640 1 -- 172.21.15.12:0/2412697133 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0ca409aca0 con 0x5649ca129de0 2024-03-23T23:02:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.558+0000 7f0cc2985740 1 -- 172.21.15.12:0/2412697133 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ca802cba0 con 0x5649ca129de0 2024-03-23T23:02:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.559+0000 7f0ca2a00640 1 -- 172.21.15.12:0/2412697133 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ca40a3f40 con 0x5649ca129de0 2024-03-23T23:02:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.559+0000 7f0cc2985740 10 client.25364 fetch_fsmap learned FSMap version 23 2024-03-23T23:02:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.559+0000 7f0cc2985740 1 -- 172.21.15.12:0/2412697133 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5649c9e6f620 con 0x5649ca129de0 2024-03-23T23:02:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.559+0000 7f0ca2a00640 1 -- 172.21.15.12:0/2412697133 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ca802cba0 con 0x5649ca129de0 2024-03-23T23:02:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.559+0000 7f0ca2a00640 1 -- 172.21.15.12:0/2412697133 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f0ca409cad0 con 0x5649ca129de0 2024-03-23T23:02:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.559+0000 7f0cc2985740 10 client.25364 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:02:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.559+0000 7f0cc2985740 10 client.25364 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:02:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.559+0000 7f0cc2985740 20 client.25364 populate_metadata read hostname 'smithi012' 2024-03-23T23:02:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.559+0000 7f0cc2985740 1 --2- 172.21.15.12:0/2412697133 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5649ca13f3e0 0x5649ca1417d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.559+0000 7f0cc2985740 4 client.25364 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:02:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.559+0000 7f0cc2985740 1 -- 172.21.15.12:0/2412697133 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5649ca09bd70 con 0x5649ca13f3e0 2024-03-23T23:02:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.559+0000 7f0cbd200640 1 --2- 172.21.15.12:0/2412697133 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5649ca13f3e0 0x5649ca1417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.560+0000 7f0cbd200640 1 --2- 172.21.15.12:0/2412697133 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5649ca13f3e0 0x5649ca1417d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38078e5c90093902 server_cookie=520474cffd14a2b in_seq=0 out_seq=0 2024-03-23T23:02:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.560+0000 7f0ca2a00640 10 client.25364 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.561+0000 7f0ca2a00640 1 -- 172.21.15.12:0/2412697133 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5649ca09bd70 con 0x5649ca13f3e0 2024-03-23T23:02:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.561+0000 7f0ca2a00640 10 client.25364 handle_command_reply: tid=0 2024-03-23T23:02:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.561+0000 7f0cc2985740 1 client.25364 shutdown 2024-03-23T23:02:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.561+0000 7f0cc2985740 1 -- 172.21.15.12:0/2412697133 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca129de0 msgr2=0x5649ca133ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.561+0000 7f0cc2985740 1 --2- 172.21.15.12:0/2412697133 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca129de0 0x5649ca133ab0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0ca40d2470 tx=0x7f0ca409b1b0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.561+0000 7f0cc2985740 1 -- 172.21.15.12:0/2412697133 shutdown_connections 2024-03-23T23:02:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.561+0000 7f0cc2985740 1 --2- 172.21.15.12:0/2412697133 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5649ca13f3e0 0x5649ca1417d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.561+0000 7f0cc2985740 1 --2- 172.21.15.12:0/2412697133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5649ca12ef50 0x5649ca13aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.561+0000 7f0cc2985740 1 --2- 172.21.15.12:0/2412697133 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca129de0 0x5649ca133ab0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.561+0000 7f0cc2985740 1 --2- 172.21.15.12:0/2412697133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649ca1293f0 0x5649ca133570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.561+0000 7f0ca2a00640 0 client.25364 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.561+0000 7f0cc2985740 1 -- 172.21.15.12:0/2412697133 >> 172.21.15.12:0/2412697133 conn(0x5649ca124960 msgr2=0x5649ca123b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.561+0000 7f0cc2985740 1 -- 172.21.15.12:0/2412697133 shutdown_connections 2024-03-23T23:02:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.561+0000 7f0cc2985740 1 -- 172.21.15.12:0/2412697133 wait complete. 2024-03-23T23:02:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.561+0000 7f0cc2985740 20 client.25364 trim_cache size 0 max 16384 2024-03-23T23:02:11.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.576+0000 7f0cc2985740 1 Processor -- start 2024-03-23T23:02:11.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.576+0000 7f0cc2985740 1 -- start start 2024-03-23T23:02:11.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.576+0000 7f0cc2985740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649ca1293f0 0x5649ca12b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:11.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.577+0000 7f0cc2985740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5649ca129de0 0x5649ca13aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:11.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.577+0000 7f0cc2985740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca12ef50 0x5649ca12b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:11.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.577+0000 7f0cc2985740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb81acf00 con 0x5649ca129de0 2024-03-23T23:02:11.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.577+0000 7f0cc2985740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb81aaa00 con 0x5649ca12ef50 2024-03-23T23:02:11.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.577+0000 7f0cc2985740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb81a8aa0 con 0x5649ca1293f0 2024-03-23T23:02:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.577+0000 7f0cbd200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649ca1293f0 0x5649ca12b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.577+0000 7f0cbdc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca12ef50 0x5649ca12b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.577+0000 7f0cbd200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649ca1293f0 0x5649ca12b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:43404/0 (socket says 172.21.15.12:43404) 2024-03-23T23:02:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.577+0000 7f0cbd200640 1 -- 172.21.15.12:0/2684914378 learned_addr learned my addr 172.21.15.12:0/2684914378 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.577+0000 7f0cb7e00640 1 --2- 172.21.15.12:0/2684914378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5649ca129de0 0x5649ca13aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.577+0000 7f0cbdc00640 1 -- 172.21.15.12:0/2684914378 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649ca1293f0 msgr2=0x5649ca12b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.577+0000 7f0cbdc00640 1 --2- 172.21.15.12:0/2684914378 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649ca1293f0 0x5649ca12b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.577+0000 7f0cbdc00640 1 -- 172.21.15.12:0/2684914378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5649ca129de0 msgr2=0x5649ca13aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.577+0000 7f0cbdc00640 1 --2- 172.21.15.12:0/2684914378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5649ca129de0 0x5649ca13aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.577+0000 7f0cbdc00640 1 -- 172.21.15.12:0/2684914378 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649ca131fe0 con 0x5649ca12ef50 2024-03-23T23:02:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.577+0000 7f0cbd200640 1 --2- 172.21.15.12:0/2684914378 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649ca1293f0 0x5649ca12b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:02:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.578+0000 7f0cbdc00640 1 --2- 172.21.15.12:0/2684914378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca12ef50 0x5649ca12b640 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0003c90 tx=0x7f0cb0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94cfbac450a0ee7d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.578+0000 7f0cad600640 1 -- 172.21.15.12:0/2684914378 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0003e40 con 0x5649ca12ef50 2024-03-23T23:02:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.578+0000 7f0cad600640 1 -- 172.21.15.12:0/2684914378 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0cb00045c0 con 0x5649ca12ef50 2024-03-23T23:02:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.578+0000 7f0cad600640 1 -- 172.21.15.12:0/2684914378 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb001e630 con 0x5649ca12ef50 2024-03-23T23:02:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.579+0000 7f0cc2985740 1 -- 172.21.15.12:0/2684914378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca12ef50 msgr2=0x5649ca12b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.579+0000 7f0cc2985740 1 --2- 172.21.15.12:0/2684914378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca12ef50 0x5649ca12b640 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0003c90 tx=0x7f0cb0016000 comp rx=0 tx=0).stop 2024-03-23T23:02:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.579+0000 7f0cc2985740 1 -- 172.21.15.12:0/2684914378 shutdown_connections 2024-03-23T23:02:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.579+0000 7f0cc2985740 1 --2- 172.21.15.12:0/2684914378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca12ef50 0x5649ca12b640 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.579+0000 7f0cc2985740 1 --2- 172.21.15.12:0/2684914378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5649ca129de0 0x5649ca13aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.579+0000 7f0cc2985740 1 --2- 172.21.15.12:0/2684914378 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649ca1293f0 0x5649ca12b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.579+0000 7f0cc2985740 1 -- 172.21.15.12:0/2684914378 >> 172.21.15.12:0/2684914378 conn(0x5649ca124960 msgr2=0x5649ca133570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.579+0000 7f0cc2985740 1 -- 172.21.15.12:0/2684914378 shutdown_connections 2024-03-23T23:02:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.579+0000 7f0cc2985740 1 -- 172.21.15.12:0/2684914378 wait complete. 2024-03-23T23:02:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.579+0000 7f0cc2985740 1 -- start start 2024-03-23T23:02:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.579+0000 7f0cc2985740 1 Processor -- start 2024-03-23T23:02:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.580+0000 7f0cc2985740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca1293f0 0x5649ca137d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.580+0000 7f0cc2985740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5649ca129de0 0x5649ca138240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.580+0000 7f0cc2985740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649ca12ef50 0x5649ca139250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.580+0000 7f0cc2985740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb81a90a0 con 0x5649ca129de0 2024-03-23T23:02:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.580+0000 7f0cc2985740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb81aa3a0 con 0x5649ca1293f0 2024-03-23T23:02:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.580+0000 7f0cc2985740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb81a96e0 con 0x5649ca12ef50 2024-03-23T23:02:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.580+0000 7f0cc2985740 10 client.0 _ll_register_callbacks cb 0x7f0ca409b620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:02:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.580+0000 7f0cb7e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5649ca129de0 0x5649ca138240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.580+0000 7f0cb7e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5649ca129de0 0x5649ca138240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49292/0 (socket says 172.21.15.12:49292) 2024-03-23T23:02:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.580+0000 7f0cbdc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649ca12ef50 0x5649ca139250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.580+0000 7f0cb7e00640 1 -- 172.21.15.12:0/3360988177 learned_addr learned my addr 172.21.15.12:0/3360988177 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.580+0000 7f0cbd200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca1293f0 0x5649ca137d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.580+0000 7f0cb7e00640 1 -- 172.21.15.12:0/3360988177 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649ca12ef50 msgr2=0x5649ca139250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.581+0000 7f0cb7e00640 1 --2- 172.21.15.12:0/3360988177 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649ca12ef50 0x5649ca139250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.581+0000 7f0cb7e00640 1 -- 172.21.15.12:0/3360988177 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca1293f0 msgr2=0x5649ca137d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.581+0000 7f0cb7e00640 1 --2- 172.21.15.12:0/3360988177 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca1293f0 0x5649ca137d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.581+0000 7f0cb7e00640 1 -- 172.21.15.12:0/3360988177 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5649ca139840 con 0x5649ca129de0 2024-03-23T23:02:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.581+0000 7f0cbdc00640 1 --2- 172.21.15.12:0/3360988177 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649ca12ef50 0x5649ca139250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:02:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.581+0000 7f0cbd200640 1 --2- 172.21.15.12:0/3360988177 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca1293f0 0x5649ca137d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:02:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.581+0000 7f0cb7e00640 1 --2- 172.21.15.12:0/3360988177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5649ca129de0 0x5649ca138240 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0ca400e9c0 tx=0x7f0ca40cf0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6990f3afb0d152 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.581+0000 7f0ca2a00640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:02:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.581+0000 7f0ca2a00640 1 -- 172.21.15.12:0/3360988177 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca409aa20 con 0x5649ca129de0 2024-03-23T23:02:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.581+0000 7f0ca2a00640 1 -- 172.21.15.12:0/3360988177 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0ca40975a0 con 0x5649ca129de0 2024-03-23T23:02:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.581+0000 7f0cc2985740 1 -- 172.21.15.12:0/3360988177 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0cb00045c0 con 0x5649ca129de0 2024-03-23T23:02:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.581+0000 7f0ca2a00640 1 -- 172.21.15.12:0/3360988177 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ca409f8f0 con 0x5649ca129de0 2024-03-23T23:02:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.581+0000 7f0cc2985740 10 client.15237 fetch_fsmap learned FSMap version 23 2024-03-23T23:02:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.581+0000 7f0cc2985740 1 -- 172.21.15.12:0/3360988177 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5649ca13abc0 con 0x5649ca129de0 2024-03-23T23:02:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.582+0000 7f0ca2a00640 1 -- 172.21.15.12:0/3360988177 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0cb00045c0 con 0x5649ca129de0 2024-03-23T23:02:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.582+0000 7f0ca2a00640 1 -- 172.21.15.12:0/3360988177 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f0ca409aca0 con 0x5649ca129de0 2024-03-23T23:02:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.582+0000 7f0cc2985740 10 client.15237 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:02:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.582+0000 7f0cc2985740 10 client.15237 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:02:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.582+0000 7f0cc2985740 20 client.15237 populate_metadata read hostname 'smithi012' 2024-03-23T23:02:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.582+0000 7f0cc2985740 1 --2- 172.21.15.12:0/3360988177 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5649ca19afd0 0x5649ca19d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.582+0000 7f0cc2985740 4 client.15237 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:02:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.582+0000 7f0cc2985740 1 -- 172.21.15.12:0/3360988177 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5649ca0960c0 con 0x5649ca19afd0 2024-03-23T23:02:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.582+0000 7f0cbd200640 1 --2- 172.21.15.12:0/3360988177 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5649ca19afd0 0x5649ca19d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.583+0000 7f0cbd200640 1 --2- 172.21.15.12:0/3360988177 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5649ca19afd0 0x5649ca19d3c0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3eed73a632d2657a server_cookie=548a933ad9671d5f in_seq=0 out_seq=0 2024-03-23T23:02:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.583+0000 7f0ca2a00640 10 client.15237 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.594+0000 7f0ca2a00640 1 -- 172.21.15.12:0/3360988177 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5649ca0960c0 con 0x5649ca19afd0 2024-03-23T23:02:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.594+0000 7f0ca2a00640 10 client.15237 handle_command_reply: tid=0 2024-03-23T23:02:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.594+0000 7f0cc2985740 1 client.15237 shutdown 2024-03-23T23:02:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.594+0000 7f0cc2985740 1 -- 172.21.15.12:0/3360988177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5649ca129de0 msgr2=0x5649ca138240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.594+0000 7f0cc2985740 1 --2- 172.21.15.12:0/3360988177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5649ca129de0 0x5649ca138240 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0ca400e9c0 tx=0x7f0ca40cf0d0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.594+0000 7f0cc2985740 1 -- 172.21.15.12:0/3360988177 shutdown_connections 2024-03-23T23:02:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.594+0000 7f0cc2985740 1 --2- 172.21.15.12:0/3360988177 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5649ca19afd0 0x5649ca19d3c0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.594+0000 7f0cc2985740 1 --2- 172.21.15.12:0/3360988177 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649ca12ef50 0x5649ca139250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.594+0000 7f0cc2985740 1 --2- 172.21.15.12:0/3360988177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5649ca129de0 0x5649ca138240 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.594+0000 7f0ca2a00640 0 client.15237 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.594+0000 7f0cc2985740 1 --2- 172.21.15.12:0/3360988177 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649ca1293f0 0x5649ca137d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.594+0000 7f0cc2985740 1 -- 172.21.15.12:0/3360988177 >> 172.21.15.12:0/3360988177 conn(0x5649ca124960 msgr2=0x5649ca133570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.594+0000 7f0cc2985740 1 -- 172.21.15.12:0/3360988177 shutdown_connections 2024-03-23T23:02:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.594+0000 7f0cc2985740 1 -- 172.21.15.12:0/3360988177 wait complete. 2024-03-23T23:02:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.594+0000 7f0cc2985740 20 client.15237 trim_cache size 0 max 16384 2024-03-23T23:02:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:02:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:02:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:02:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:02:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.595+0000 7f0cbe600640 1 -- 172.21.15.12:0/822005271 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0c9c0a27f0 msgr2=0x7f0c9c0a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.595+0000 7f0cbe600640 1 --2- 172.21.15.12:0/822005271 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0c9c0a27f0 0x7f0c9c0a4cb0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x5649ca12a640 tx=0x7f0cb0009500 comp rx=0 tx=0).stop 2024-03-23T23:02:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.595+0000 7f0cbe600640 1 -- 172.21.15.12:0/822005271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cb8104910 msgr2=0x7f0cb8199f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.595+0000 7f0cbe600640 1 --2- 172.21.15.12:0/822005271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cb8104910 0x7f0cb8199f20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0ca400e9f0 tx=0x7f0ca400eeb0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.595+0000 7f0cbe600640 1 -- 172.21.15.12:0/822005271 shutdown_connections 2024-03-23T23:02:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.595+0000 7f0cbe600640 1 --2- 172.21.15.12:0/822005271 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0c9c0a27f0 0x7f0c9c0a4cb0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.595+0000 7f0cbe600640 1 --2- 172.21.15.12:0/822005271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cb81052d0 0x7f0cb81a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.595+0000 7f0cbe600640 1 --2- 172.21.15.12:0/822005271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cb8104910 0x7f0cb8199f20 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.595+0000 7f0cbe600640 1 --2- 172.21.15.12:0/822005271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0cb8103720 0x7f0cb81999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.595+0000 7f0cbe600640 1 -- 172.21.15.12:0/822005271 >> 172.21.15.12:0/822005271 conn(0x7f0cb80fecb0 msgr2=0x7f0cb8107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.595+0000 7f0cbe600640 1 -- 172.21.15.12:0/822005271 shutdown_connections 2024-03-23T23:02:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:11.596+0000 7f0cbe600640 1 -- 172.21.15.12:0/822005271 wait complete. 2024-03-23T23:02:13.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:13 smithi173 ceph-mon[36116]: pgmap v402: 97 pgs: 97 active+clean; 649 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-23T23:02:13.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:13 smithi203 ceph-mon[36693]: pgmap v402: 97 pgs: 97 active+clean; 649 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-23T23:02:13.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:13 smithi012 ceph-mon[31695]: pgmap v402: 97 pgs: 97 active+clean; 649 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-23T23:02:15.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:15 smithi173 ceph-mon[36116]: pgmap v403: 97 pgs: 97 active+clean; 652 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-23T23:02:15.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:15 smithi203 ceph-mon[36693]: pgmap v403: 97 pgs: 97 active+clean; 652 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-23T23:02:15.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:15 smithi012 ceph-mon[31695]: pgmap v403: 97 pgs: 97 active+clean; 652 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-23T23:02:17.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:17 smithi173 ceph-mon[36116]: pgmap v404: 97 pgs: 97 active+clean; 652 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-03-23T23:02:17.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:17 smithi203 ceph-mon[36693]: pgmap v404: 97 pgs: 97 active+clean; 652 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-03-23T23:02:17.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:17 smithi012 ceph-mon[31695]: pgmap v404: 97 pgs: 97 active+clean; 652 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-03-23T23:02:18.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:18 smithi012 ceph-mon[31695]: pgmap v405: 97 pgs: 97 active+clean; 652 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-23T23:02:18.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:18 smithi173 ceph-mon[36116]: pgmap v405: 97 pgs: 97 active+clean; 652 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-23T23:02:18.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:18 smithi203 ceph-mon[36693]: pgmap v405: 97 pgs: 97 active+clean; 652 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-23T23:02:20.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:20 smithi203 ceph-mon[36693]: pgmap v406: 97 pgs: 97 active+clean; 652 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-23T23:02:20.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:20 smithi012 ceph-mon[31695]: pgmap v406: 97 pgs: 97 active+clean; 652 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-23T23:02:20.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:20 smithi173 ceph-mon[36116]: pgmap v406: 97 pgs: 97 active+clean; 652 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-23T23:02:22.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:22 smithi203 ceph-mon[36693]: pgmap v407: 97 pgs: 97 active+clean; 652 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-23T23:02:22.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:22 smithi012 ceph-mon[31695]: pgmap v407: 97 pgs: 97 active+clean; 652 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-23T23:02:22.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:22 smithi173 ceph-mon[36116]: pgmap v407: 97 pgs: 97 active+clean; 652 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-23T23:02:24.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:24 smithi203 ceph-mon[36693]: pgmap v408: 97 pgs: 97 active+clean; 652 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-23T23:02:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.743+0000 7f5e37e00640 1 -- 172.21.15.12:0/3475226642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e38108d10 msgr2=0x7f5e38109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.743+0000 7f5e37e00640 1 --2- 172.21.15.12:0/3475226642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e38108d10 0x7f5e38109190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5e20009900 tx=0x7f5e2002bc50 comp rx=0 tx=0).stop 2024-03-23T23:02:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.743+0000 7f5e37e00640 1 -- 172.21.15.12:0/3475226642 shutdown_connections 2024-03-23T23:02:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.743+0000 7f5e37e00640 1 --2- 172.21.15.12:0/3475226642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e381096d0 0x7f5e3810ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.743+0000 7f5e37e00640 1 --2- 172.21.15.12:0/3475226642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e38108d10 0x7f5e38109190 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.743+0000 7f5e37e00640 1 --2- 172.21.15.12:0/3475226642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e38107b20 0x7f5e38107f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.743+0000 7f5e37e00640 1 -- 172.21.15.12:0/3475226642 >> 172.21.15.12:0/3475226642 conn(0x7f5e38076ae0 msgr2=0x7f5e38078f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.744+0000 7f5e37e00640 1 -- 172.21.15.12:0/3475226642 shutdown_connections 2024-03-23T23:02:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.744+0000 7f5e37e00640 1 -- 172.21.15.12:0/3475226642 wait complete. 2024-03-23T23:02:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.745+0000 7f5e37e00640 1 Processor -- start 2024-03-23T23:02:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.745+0000 7f5e37e00640 1 -- start start 2024-03-23T23:02:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.745+0000 7f5e37e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e38107b20 0x7f5e3819de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.745+0000 7f5e37e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e38108d10 0x7f5e3819e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.745+0000 7f5e37e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e381096d0 0x7f5e381a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.746+0000 7f5e37e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e381137f0 con 0x7f5e38107b20 2024-03-23T23:02:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.746+0000 7f5e37e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e38113670 con 0x7f5e38108d10 2024-03-23T23:02:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.746+0000 7f5e37e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e38113970 con 0x7f5e381096d0 2024-03-23T23:02:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.746+0000 7f5e2fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e38108d10 0x7f5e3819e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.746+0000 7f5e36a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e38107b20 0x7f5e3819de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.746+0000 7f5e2fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e38108d10 0x7f5e3819e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57144/0 (socket says 172.21.15.12:57144) 2024-03-23T23:02:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.746+0000 7f5e2fe00640 1 -- 172.21.15.12:0/4189093961 learned_addr learned my addr 172.21.15.12:0/4189093961 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.746+0000 7f5e37400640 1 --2- 172.21.15.12:0/4189093961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e381096d0 0x7f5e381a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.746+0000 7f5e2fe00640 1 -- 172.21.15.12:0/4189093961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e381096d0 msgr2=0x7f5e381a53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.746+0000 7f5e2fe00640 1 --2- 172.21.15.12:0/4189093961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e381096d0 0x7f5e381a53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.746+0000 7f5e2fe00640 1 -- 172.21.15.12:0/4189093961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e38107b20 msgr2=0x7f5e3819de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.746+0000 7f5e2fe00640 1 --2- 172.21.15.12:0/4189093961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e38107b20 0x7f5e3819de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.746+0000 7f5e2fe00640 1 -- 172.21.15.12:0/4189093961 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e381a5a20 con 0x7f5e38108d10 2024-03-23T23:02:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.746+0000 7f5e37400640 1 --2- 172.21.15.12:0/4189093961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e381096d0 0x7f5e381a53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:02:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.747+0000 7f5e2fe00640 1 --2- 172.21.15.12:0/4189093961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e38108d10 0x7f5e3819e340 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5e20005ab0 tx=0x7f5e2002c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.747+0000 7f5e2f400640 1 -- 172.21.15.12:0/4189093961 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e20041070 con 0x7f5e38108d10 2024-03-23T23:02:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.747+0000 7f5e2f400640 1 -- 172.21.15.12:0/4189093961 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e20004450 con 0x7f5e38108d10 2024-03-23T23:02:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.747+0000 7f5e2f400640 1 -- 172.21.15.12:0/4189093961 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e20034440 con 0x7f5e38108d10 2024-03-23T23:02:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.747+0000 7f5e37e00640 1 -- 172.21.15.12:0/4189093961 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e381a5cb0 con 0x7f5e38108d10 2024-03-23T23:02:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.747+0000 7f5e37e00640 1 -- 172.21.15.12:0/4189093961 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e381a61f0 con 0x7f5e38108d10 2024-03-23T23:02:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.750+0000 7f5e2f400640 1 -- 172.21.15.12:0/4189093961 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5e20044070 con 0x7f5e38108d10 2024-03-23T23:02:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.751+0000 7f5e4037e740 1 Processor -- start 2024-03-23T23:02:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.751+0000 7f5e4037e740 1 -- start start 2024-03-23T23:02:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.751+0000 7f5e4037e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cde19ea3f0 0x55cde19ea810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.751+0000 7f5e4037e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cde19eade0 0x55cde19ef8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.751+0000 7f5e4037e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cde19eff50 0x55cde19f23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.751+0000 7f5e4037e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cde19ad3b0 con 0x55cde19ea3f0 2024-03-23T23:02:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.751+0000 7f5e4037e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cde194f830 con 0x55cde19eade0 2024-03-23T23:02:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.751+0000 7f5e4037e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cde18f85d0 con 0x55cde19eff50 2024-03-23T23:02:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.751+0000 7f5e36a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cde19eade0 0x55cde19ef8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.751+0000 7f5e37400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cde19ea3f0 0x55cde19ea810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.751+0000 7f5e2fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cde19eff50 0x55cde19f23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.751+0000 7f5e36a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cde19eade0 0x55cde19ef8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57148/0 (socket says 172.21.15.12:57148) 2024-03-23T23:02:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.751+0000 7f5e36a00640 1 -- 172.21.15.12:0/4135471203 learned_addr learned my addr 172.21.15.12:0/4135471203 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.752+0000 7f5e36a00640 1 -- 172.21.15.12:0/4135471203 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cde19eff50 msgr2=0x55cde19f23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.752+0000 7f5e36a00640 1 --2- 172.21.15.12:0/4135471203 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cde19eff50 0x55cde19f23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.752+0000 7f5e36a00640 1 -- 172.21.15.12:0/4135471203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cde19ea3f0 msgr2=0x55cde19ea810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.752+0000 7f5e36a00640 1 --2- 172.21.15.12:0/4135471203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cde19ea3f0 0x55cde19ea810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.752+0000 7f5e36a00640 1 -- 172.21.15.12:0/4135471203 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cde19102c0 con 0x55cde19eade0 2024-03-23T23:02:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.752+0000 7f5e36a00640 1 --2- 172.21.15.12:0/4135471203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cde19eade0 0x55cde19ef8e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5e28009ca0 tx=0x7f5e28011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=226270f850443641 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.752+0000 7f5e27400640 1 -- 172.21.15.12:0/4135471203 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28004220 con 0x55cde19eade0 2024-03-23T23:02:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.752+0000 7f5e27400640 1 -- 172.21.15.12:0/4135471203 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e280043c0 con 0x55cde19eade0 2024-03-23T23:02:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.752+0000 7f5e27400640 1 -- 172.21.15.12:0/4135471203 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28019930 con 0x55cde19eade0 2024-03-23T23:02:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.753+0000 7f5e4037e740 1 -- 172.21.15.12:0/4135471203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cde19eade0 msgr2=0x55cde19ef8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.753+0000 7f5e4037e740 1 --2- 172.21.15.12:0/4135471203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cde19eade0 0x55cde19ef8e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5e28009ca0 tx=0x7f5e28011810 comp rx=0 tx=0).stop 2024-03-23T23:02:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.753+0000 7f5e2f400640 1 --2- 172.21.15.12:0/4189093961 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5e1c0a2870 0x7f5e1c0a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.753+0000 7f5e4037e740 1 -- 172.21.15.12:0/4135471203 shutdown_connections 2024-03-23T23:02:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.753+0000 7f5e4037e740 1 --2- 172.21.15.12:0/4135471203 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cde19eff50 0x55cde19f23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.753+0000 7f5e4037e740 1 --2- 172.21.15.12:0/4135471203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cde19eade0 0x55cde19ef8e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.753+0000 7f5e4037e740 1 --2- 172.21.15.12:0/4135471203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cde19ea3f0 0x55cde19ea810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.753+0000 7f5e4037e740 1 -- 172.21.15.12:0/4135471203 >> 172.21.15.12:0/4135471203 conn(0x55cde19e5960 msgr2=0x55cde19e5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.753+0000 7f5e2f400640 1 -- 172.21.15.12:0/4189093961 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5e200f05c0 con 0x7f5e38108d10 2024-03-23T23:02:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.753+0000 7f5e37400640 1 --2- 172.21.15.12:0/4189093961 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5e1c0a2870 0x7f5e1c0a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.753+0000 7f5e4037e740 1 -- 172.21.15.12:0/4135471203 shutdown_connections 2024-03-23T23:02:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.754+0000 7f5e4037e740 1 -- 172.21.15.12:0/4135471203 wait complete. 2024-03-23T23:02:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.754+0000 7f5e4037e740 1 -- start start 2024-03-23T23:02:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.754+0000 7f5e37400640 1 --2- 172.21.15.12:0/4189093961 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5e1c0a2870 0x7f5e1c0a4d30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x55cde19eb640 tx=0x7f5e30009600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.754+0000 7f5e4037e740 1 Processor -- start 2024-03-23T23:02:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.755+0000 7f5e4037e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cde19ea3f0 0x55cde19f45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.755+0000 7f5e4037e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cde19eade0 0x55cde19f4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.755+0000 7f5e4037e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cde19eff50 0x55cde19fbb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.755+0000 7f5e4037e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cde190ba20 con 0x55cde19eff50 2024-03-23T23:02:24.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.755+0000 7f5e4037e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cde194b8d0 con 0x55cde19ea3f0 2024-03-23T23:02:24.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.755+0000 7f5e4037e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cde19f2f00 con 0x55cde19eade0 2024-03-23T23:02:24.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.755+0000 7f5e4037e740 10 client.0 _ll_register_callbacks cb 0x55cde19481c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:02:24.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.755+0000 7f5e36a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cde19ea3f0 0x55cde19f45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:24.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.755+0000 7f5e36a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cde19ea3f0 0x55cde19f45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57164/0 (socket says 172.21.15.12:57164) 2024-03-23T23:02:24.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.755+0000 7f5e36a00640 1 -- 172.21.15.12:0/3200866022 learned_addr learned my addr 172.21.15.12:0/3200866022 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:24.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.755+0000 7f5e2fe00640 1 --2- 172.21.15.12:0/3200866022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cde19eade0 0x55cde19f4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:24.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.755+0000 7f5e37400640 1 --2- 172.21.15.12:0/3200866022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cde19eff50 0x55cde19fbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:24.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.755+0000 7f5e2fe00640 1 -- 172.21.15.12:0/3200866022 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cde19ea3f0 msgr2=0x55cde19f45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:24.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.755+0000 7f5e2fe00640 1 --2- 172.21.15.12:0/3200866022 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cde19ea3f0 0x55cde19f45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.755+0000 7f5e2fe00640 1 -- 172.21.15.12:0/3200866022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cde19eff50 msgr2=0x55cde19fbb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:24.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.755+0000 7f5e2fe00640 1 --2- 172.21.15.12:0/3200866022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cde19eff50 0x55cde19fbb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.755+0000 7f5e2fe00640 1 -- 172.21.15.12:0/3200866022 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cde1664210 con 0x55cde19eade0 2024-03-23T23:02:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.756+0000 7f5e2fe00640 1 --2- 172.21.15.12:0/3200866022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cde19eade0 0x55cde19f4b00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5e200ee070 tx=0x7f5e20009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42138051944e166f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.756+0000 7f5e1be00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:02:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.756+0000 7f5e1be00640 1 -- 172.21.15.12:0/3200866022 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2004b070 con 0x55cde19eade0 2024-03-23T23:02:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.756+0000 7f5e1be00640 1 -- 172.21.15.12:0/3200866022 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5e200c5220 con 0x55cde19eade0 2024-03-23T23:02:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.756+0000 7f5e4037e740 1 -- 172.21.15.12:0/3200866022 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e280043c0 con 0x55cde19eade0 2024-03-23T23:02:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.756+0000 7f5e1be00640 1 -- 172.21.15.12:0/3200866022 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e200bcb00 con 0x55cde19eade0 2024-03-23T23:02:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.756+0000 7f5e4037e740 10 client.25363 fetch_fsmap learned FSMap version 23 2024-03-23T23:02:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.756+0000 7f5e4037e740 1 -- 172.21.15.12:0/3200866022 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55cde1730620 con 0x55cde19eade0 2024-03-23T23:02:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.756+0000 7f5e1be00640 1 -- 172.21.15.12:0/3200866022 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e280043c0 con 0x55cde19eade0 2024-03-23T23:02:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.757+0000 7f5e1be00640 1 -- 172.21.15.12:0/3200866022 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f5e200d2030 con 0x55cde19eade0 2024-03-23T23:02:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.757+0000 7f5e4037e740 10 client.25363 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:02:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.757+0000 7f5e4037e740 10 client.25363 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:02:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.757+0000 7f5e4037e740 20 client.25363 populate_metadata read hostname 'smithi012' 2024-03-23T23:02:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.757+0000 7f5e4037e740 1 --2- 172.21.15.12:0/3200866022 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cde1a003c0 0x55cde1a02810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.757+0000 7f5e4037e740 4 client.25363 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:02:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.757+0000 7f5e4037e740 1 -- 172.21.15.12:0/3200866022 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cde195cd70 con 0x55cde1a003c0 2024-03-23T23:02:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.757+0000 7f5e36a00640 1 --2- 172.21.15.12:0/3200866022 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cde1a003c0 0x55cde1a02810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.758+0000 7f5e36a00640 1 --2- 172.21.15.12:0/3200866022 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cde1a003c0 0x55cde1a02810 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da79ac9a6a8264c0 server_cookie=e4e009644d949fcf in_seq=0 out_seq=0 2024-03-23T23:02:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.758+0000 7f5e1be00640 10 client.25363 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.759+0000 7f5e1be00640 1 -- 172.21.15.12:0/3200866022 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cde195cd70 con 0x55cde1a003c0 2024-03-23T23:02:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.759+0000 7f5e1be00640 10 client.25363 handle_command_reply: tid=0 2024-03-23T23:02:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.759+0000 7f5e4037e740 1 client.25363 shutdown 2024-03-23T23:02:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.759+0000 7f5e4037e740 1 -- 172.21.15.12:0/3200866022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cde19eade0 msgr2=0x55cde19f4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.759+0000 7f5e4037e740 1 --2- 172.21.15.12:0/3200866022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cde19eade0 0x55cde19f4b00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5e200ee070 tx=0x7f5e20009680 comp rx=0 tx=0).stop 2024-03-23T23:02:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.759+0000 7f5e4037e740 1 -- 172.21.15.12:0/3200866022 shutdown_connections 2024-03-23T23:02:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.759+0000 7f5e4037e740 1 --2- 172.21.15.12:0/3200866022 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cde1a003c0 0x55cde1a02810 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.759+0000 7f5e4037e740 1 --2- 172.21.15.12:0/3200866022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cde19eff50 0x55cde19fbb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.759+0000 7f5e4037e740 1 --2- 172.21.15.12:0/3200866022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cde19eade0 0x55cde19f4b00 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.759+0000 7f5e4037e740 1 --2- 172.21.15.12:0/3200866022 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cde19ea3f0 0x55cde19f45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.759+0000 7f5e4037e740 1 -- 172.21.15.12:0/3200866022 >> 172.21.15.12:0/3200866022 conn(0x55cde19e5960 msgr2=0x55cde19e4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.759+0000 7f5e1be00640 0 client.25363 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.759+0000 7f5e4037e740 1 -- 172.21.15.12:0/3200866022 shutdown_connections 2024-03-23T23:02:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.759+0000 7f5e4037e740 1 -- 172.21.15.12:0/3200866022 wait complete. 2024-03-23T23:02:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.759+0000 7f5e4037e740 20 client.25363 trim_cache size 0 max 16384 2024-03-23T23:02:24.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.774+0000 7f5e4037e740 1 Processor -- start 2024-03-23T23:02:24.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.774+0000 7f5e4037e740 1 -- start start 2024-03-23T23:02:24.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.775+0000 7f5e4037e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cde19ea3f0 0x55cde19e42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:24.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.775+0000 7f5e4037e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cde19eade0 0x55cde19fbb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:24.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.775+0000 7f5e4037e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cde19eff50 0x55cde19e4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:24.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.775+0000 7f5e4037e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e381b1380 con 0x55cde19eade0 2024-03-23T23:02:24.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.775+0000 7f5e4037e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e381aee80 con 0x55cde19ea3f0 2024-03-23T23:02:24.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.775+0000 7f5e4037e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e381acf20 con 0x55cde19eff50 2024-03-23T23:02:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.775+0000 7f5e36a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cde19ea3f0 0x55cde19e42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.775+0000 7f5e37400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cde19eff50 0x55cde19e4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.775+0000 7f5e2fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cde19eade0 0x55cde19fbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.775+0000 7f5e36a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cde19ea3f0 0x55cde19e42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57174/0 (socket says 172.21.15.12:57174) 2024-03-23T23:02:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.775+0000 7f5e36a00640 1 -- 172.21.15.12:0/577926418 learned_addr learned my addr 172.21.15.12:0/577926418 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.775+0000 7f5e37400640 1 -- 172.21.15.12:0/577926418 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cde19ea3f0 msgr2=0x55cde19e42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.775+0000 7f5e37400640 1 --2- 172.21.15.12:0/577926418 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cde19ea3f0 0x55cde19e42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.775+0000 7f5e37400640 1 -- 172.21.15.12:0/577926418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cde19eade0 msgr2=0x55cde19fbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.775+0000 7f5e37400640 1 --2- 172.21.15.12:0/577926418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cde19eade0 0x55cde19fbb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.775+0000 7f5e37400640 1 -- 172.21.15.12:0/577926418 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cde19e0d10 con 0x55cde19eff50 2024-03-23T23:02:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.775+0000 7f5e2fe00640 1 --2- 172.21.15.12:0/577926418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cde19eade0 0x55cde19fbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:02:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.776+0000 7f5e37400640 1 --2- 172.21.15.12:0/577926418 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cde19eff50 0x55cde19e4820 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5e30009fb0 tx=0x7f5e3000b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9656016ef79a9cd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.776+0000 7f5e26a00640 1 -- 172.21.15.12:0/577926418 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3002c070 con 0x55cde19eff50 2024-03-23T23:02:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.776+0000 7f5e26a00640 1 -- 172.21.15.12:0/577926418 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e3001c940 con 0x55cde19eff50 2024-03-23T23:02:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.776+0000 7f5e26a00640 1 -- 172.21.15.12:0/577926418 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e300277c0 con 0x55cde19eff50 2024-03-23T23:02:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.776+0000 7f5e4037e740 1 -- 172.21.15.12:0/577926418 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cde19eff50 msgr2=0x55cde19e4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.776+0000 7f5e4037e740 1 --2- 172.21.15.12:0/577926418 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cde19eff50 0x55cde19e4820 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5e30009fb0 tx=0x7f5e3000b000 comp rx=0 tx=0).stop 2024-03-23T23:02:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.777+0000 7f5e4037e740 1 -- 172.21.15.12:0/577926418 shutdown_connections 2024-03-23T23:02:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.777+0000 7f5e4037e740 1 --2- 172.21.15.12:0/577926418 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cde19eff50 0x55cde19e4820 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.777+0000 7f5e4037e740 1 --2- 172.21.15.12:0/577926418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cde19eade0 0x55cde19fbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.777+0000 7f5e4037e740 1 --2- 172.21.15.12:0/577926418 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cde19ea3f0 0x55cde19e42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.777+0000 7f5e4037e740 1 -- 172.21.15.12:0/577926418 >> 172.21.15.12:0/577926418 conn(0x55cde19e5960 msgr2=0x55cde19e5d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.777+0000 7f5e4037e740 1 -- 172.21.15.12:0/577926418 shutdown_connections 2024-03-23T23:02:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.777+0000 7f5e4037e740 1 -- 172.21.15.12:0/577926418 wait complete. 2024-03-23T23:02:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.777+0000 7f5e4037e740 1 -- start start 2024-03-23T23:02:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.777+0000 7f5e4037e740 1 Processor -- start 2024-03-23T23:02:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.778+0000 7f5e4037e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cde19ea3f0 0x55cde19f9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.778+0000 7f5e4037e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cde19eade0 0x55cde19f9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.778+0000 7f5e4037e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cde19eff50 0x55cde19faf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.778+0000 7f5e4037e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e381ad520 con 0x55cde19eade0 2024-03-23T23:02:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.778+0000 7f5e4037e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e381ae820 con 0x55cde19eff50 2024-03-23T23:02:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.778+0000 7f5e4037e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e381adb60 con 0x55cde19ea3f0 2024-03-23T23:02:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.778+0000 7f5e4037e740 10 client.0 _ll_register_callbacks cb 0x7f5e20005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:02:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.778+0000 7f5e36a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cde19ea3f0 0x55cde19f9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.778+0000 7f5e36a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cde19ea3f0 0x55cde19f9a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:40134/0 (socket says 172.21.15.12:40134) 2024-03-23T23:02:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.778+0000 7f5e36a00640 1 -- 172.21.15.12:0/3697662391 learned_addr learned my addr 172.21.15.12:0/3697662391 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.778+0000 7f5e2fe00640 1 --2- 172.21.15.12:0/3697662391 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cde19eade0 0x55cde19f9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.778+0000 7f5e37400640 1 --2- 172.21.15.12:0/3697662391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cde19eff50 0x55cde19faf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.778+0000 7f5e36a00640 1 -- 172.21.15.12:0/3697662391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cde19eff50 msgr2=0x55cde19faf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.778+0000 7f5e36a00640 1 --2- 172.21.15.12:0/3697662391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cde19eff50 0x55cde19faf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.778+0000 7f5e36a00640 1 -- 172.21.15.12:0/3697662391 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cde19eade0 msgr2=0x55cde19f9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.778+0000 7f5e36a00640 1 --2- 172.21.15.12:0/3697662391 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cde19eade0 0x55cde19f9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.778+0000 7f5e36a00640 1 -- 172.21.15.12:0/3697662391 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cde19fb570 con 0x55cde19ea3f0 2024-03-23T23:02:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.779+0000 7f5e36a00640 1 --2- 172.21.15.12:0/3697662391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cde19ea3f0 0x55cde19f9a30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5e28000c00 tx=0x7f5e28005f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8958c9df22f9cfe3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.779+0000 7f5e1be00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:02:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.779+0000 7f5e1be00640 1 -- 172.21.15.12:0/3697662391 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28005290 con 0x55cde19ea3f0 2024-03-23T23:02:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.779+0000 7f5e1be00640 1 -- 172.21.15.12:0/3697662391 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5e28005430 con 0x55cde19ea3f0 2024-03-23T23:02:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.779+0000 7f5e4037e740 1 -- 172.21.15.12:0/3697662391 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e3001c940 con 0x55cde19ea3f0 2024-03-23T23:02:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.779+0000 7f5e1be00640 1 -- 172.21.15.12:0/3697662391 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e2803b050 con 0x55cde19ea3f0 2024-03-23T23:02:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.779+0000 7f5e4037e740 10 client.25369 fetch_fsmap learned FSMap version 23 2024-03-23T23:02:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.779+0000 7f5e4037e740 1 -- 172.21.15.12:0/3697662391 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55cde1a5c5a0 con 0x55cde19ea3f0 2024-03-23T23:02:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.779+0000 7f5e1be00640 1 -- 172.21.15.12:0/3697662391 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e3001c940 con 0x55cde19ea3f0 2024-03-23T23:02:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.779+0000 7f5e1be00640 1 -- 172.21.15.12:0/3697662391 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f5e2802b5b0 con 0x55cde19ea3f0 2024-03-23T23:02:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.780+0000 7f5e4037e740 10 client.25369 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:02:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.780+0000 7f5e4037e740 10 client.25369 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:02:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.780+0000 7f5e4037e740 20 client.25369 populate_metadata read hostname 'smithi012' 2024-03-23T23:02:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.780+0000 7f5e4037e740 1 --2- 172.21.15.12:0/3697662391 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cde1a5ce10 0x55cde1a5f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.780+0000 7f5e4037e740 4 client.25369 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:02:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.780+0000 7f5e4037e740 1 -- 172.21.15.12:0/3697662391 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55cde19570c0 con 0x55cde1a5ce10 2024-03-23T23:02:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.780+0000 7f5e2fe00640 1 --2- 172.21.15.12:0/3697662391 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cde1a5ce10 0x55cde1a5f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.780+0000 7f5e2fe00640 1 --2- 172.21.15.12:0/3697662391 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cde1a5ce10 0x55cde1a5f230 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28cf0dfd40032f4a server_cookie=7f95d0d88b2f34d8 in_seq=0 out_seq=0 2024-03-23T23:02:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.780+0000 7f5e1be00640 10 client.25369 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.786+0000 7f5e1be00640 1 -- 172.21.15.12:0/3697662391 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55cde19570c0 con 0x55cde1a5ce10 2024-03-23T23:02:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.786+0000 7f5e1be00640 10 client.25369 handle_command_reply: tid=0 2024-03-23T23:02:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.786+0000 7f5e4037e740 1 client.25369 shutdown 2024-03-23T23:02:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.786+0000 7f5e4037e740 1 -- 172.21.15.12:0/3697662391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cde19ea3f0 msgr2=0x55cde19f9a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.786+0000 7f5e4037e740 1 --2- 172.21.15.12:0/3697662391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cde19ea3f0 0x55cde19f9a30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5e28000c00 tx=0x7f5e28005f10 comp rx=0 tx=0).stop 2024-03-23T23:02:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.786+0000 7f5e4037e740 1 -- 172.21.15.12:0/3697662391 shutdown_connections 2024-03-23T23:02:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.786+0000 7f5e4037e740 1 --2- 172.21.15.12:0/3697662391 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cde1a5ce10 0x55cde1a5f230 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.786+0000 7f5e4037e740 1 --2- 172.21.15.12:0/3697662391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cde19eff50 0x55cde19faf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.786+0000 7f5e4037e740 1 --2- 172.21.15.12:0/3697662391 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cde19eade0 0x55cde19f9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.786+0000 7f5e4037e740 1 --2- 172.21.15.12:0/3697662391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cde19ea3f0 0x55cde19f9a30 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.786+0000 7f5e1be00640 0 client.25369 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.786+0000 7f5e4037e740 1 -- 172.21.15.12:0/3697662391 >> 172.21.15.12:0/3697662391 conn(0x55cde19e5960 msgr2=0x55cde19ef300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.786+0000 7f5e4037e740 1 -- 172.21.15.12:0/3697662391 shutdown_connections 2024-03-23T23:02:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.786+0000 7f5e4037e740 1 -- 172.21.15.12:0/3697662391 wait complete. 2024-03-23T23:02:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.786+0000 7f5e4037e740 20 client.25369 trim_cache size 0 max 16384 2024-03-23T23:02:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:02:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:02:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:02:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:02:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.787+0000 7f5e37e00640 1 -- 172.21.15.12:0/4189093961 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5e1c0a2870 msgr2=0x7f5e1c0a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.787+0000 7f5e37e00640 1 --2- 172.21.15.12:0/4189093961 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5e1c0a2870 0x7f5e1c0a4d30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x55cde19eb640 tx=0x7f5e30009600 comp rx=0 tx=0).stop 2024-03-23T23:02:24.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.787+0000 7f5e37e00640 1 -- 172.21.15.12:0/4189093961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e38108d10 msgr2=0x7f5e3819e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:24.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.787+0000 7f5e37e00640 1 --2- 172.21.15.12:0/4189093961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e38108d10 0x7f5e3819e340 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5e20005ab0 tx=0x7f5e2002c6e0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.787+0000 7f5e37e00640 1 -- 172.21.15.12:0/4189093961 shutdown_connections 2024-03-23T23:02:24.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.787+0000 7f5e37e00640 1 --2- 172.21.15.12:0/4189093961 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5e1c0a2870 0x7f5e1c0a4d30 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.787+0000 7f5e37e00640 1 --2- 172.21.15.12:0/4189093961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e381096d0 0x7f5e381a53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.787+0000 7f5e37e00640 1 --2- 172.21.15.12:0/4189093961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e38108d10 0x7f5e3819e340 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.787+0000 7f5e37e00640 1 --2- 172.21.15.12:0/4189093961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e38107b20 0x7f5e3819de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:24.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.787+0000 7f5e37e00640 1 -- 172.21.15.12:0/4189093961 >> 172.21.15.12:0/4189093961 conn(0x7f5e38076ae0 msgr2=0x7f5e3810c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:24.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.787+0000 7f5e37e00640 1 -- 172.21.15.12:0/4189093961 shutdown_connections 2024-03-23T23:02:24.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:24.787+0000 7f5e37e00640 1 -- 172.21.15.12:0/4189093961 wait complete. 2024-03-23T23:02:24.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:24 smithi012 ceph-mon[31695]: pgmap v408: 97 pgs: 97 active+clean; 652 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-23T23:02:24.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:24 smithi173 ceph-mon[36116]: pgmap v408: 97 pgs: 97 active+clean; 652 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-23T23:02:26.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:26 smithi203 ceph-mon[36693]: pgmap v409: 97 pgs: 97 active+clean; 652 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-23T23:02:26.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:26 smithi012 ceph-mon[31695]: pgmap v409: 97 pgs: 97 active+clean; 652 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-23T23:02:26.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:26 smithi173 ceph-mon[36116]: pgmap v409: 97 pgs: 97 active+clean; 652 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-23T23:02:28.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:28 smithi203 ceph-mon[36693]: pgmap v410: 97 pgs: 97 active+clean; 652 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:02:28.675 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:28 smithi173 ceph-mon[36116]: pgmap v410: 97 pgs: 97 active+clean; 652 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:02:28.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:28 smithi012 ceph-mon[31695]: pgmap v410: 97 pgs: 97 active+clean; 652 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:02:28.860 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:02:28.860 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:02:29.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.102+0000 7f4de6200640 1 -- 172.21.15.12:0/3060099628 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de01038c0 msgr2=0x7f4de0103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:29.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.102+0000 7f4de6200640 1 --2- 172.21.15.12:0/3060099628 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de01038c0 0x7f4de0103ce0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0009900 tx=0x7f4dd002bc30 comp rx=0 tx=0).stop 2024-03-23T23:02:29.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.102+0000 7f4de6200640 1 -- 172.21.15.12:0/3060099628 shutdown_connections 2024-03-23T23:02:29.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.102+0000 7f4de6200640 1 --2- 172.21.15.12:0/3060099628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4de0105470 0x7f4de010bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.102+0000 7f4de6200640 1 --2- 172.21.15.12:0/3060099628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4de0104ab0 0x7f4de0104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.102+0000 7f4de6200640 1 --2- 172.21.15.12:0/3060099628 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de01038c0 0x7f4de0103ce0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.102+0000 7f4de6200640 1 -- 172.21.15.12:0/3060099628 >> 172.21.15.12:0/3060099628 conn(0x7f4de00fee50 msgr2=0x7f4de0101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:29.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.102+0000 7f4de6200640 1 -- 172.21.15.12:0/3060099628 shutdown_connections 2024-03-23T23:02:29.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.102+0000 7f4de6200640 1 -- 172.21.15.12:0/3060099628 wait complete. 2024-03-23T23:02:29.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.103+0000 7f4de6200640 1 Processor -- start 2024-03-23T23:02:29.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.103+0000 7f4de6200640 1 -- start start 2024-03-23T23:02:29.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.103+0000 7f4de6200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4de01038c0 0x7f4de0199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.103+0000 7f4de6200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de0104ab0 0x7f4de019a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.103+0000 7f4de6200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4de0105470 0x7f4de01a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.103+0000 7f4de6200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de010f560 con 0x7f4de0105470 2024-03-23T23:02:29.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.103+0000 7f4de6200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de010f3e0 con 0x7f4de01038c0 2024-03-23T23:02:29.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.103+0000 7f4de6200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de010f6e0 con 0x7f4de0104ab0 2024-03-23T23:02:29.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.104+0000 7f4ddf400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4de01038c0 0x7f4de0199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.104+0000 7f4ddf400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4de01038c0 0x7f4de0199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57202/0 (socket says 172.21.15.12:57202) 2024-03-23T23:02:29.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.104+0000 7f4ddf400640 1 -- 172.21.15.12:0/51664492 learned_addr learned my addr 172.21.15.12:0/51664492 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:29.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.104+0000 7f4ddfe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4de0105470 0x7f4de01a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.104+0000 7f4ddea00640 1 --2- 172.21.15.12:0/51664492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de0104ab0 0x7f4de019a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.104+0000 7f4ddf400640 1 -- 172.21.15.12:0/51664492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de0104ab0 msgr2=0x7f4de019a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:29.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.104+0000 7f4ddf400640 1 --2- 172.21.15.12:0/51664492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de0104ab0 0x7f4de019a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.104+0000 7f4ddf400640 1 -- 172.21.15.12:0/51664492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4de0105470 msgr2=0x7f4de01a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:29.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.104+0000 7f4ddf400640 1 --2- 172.21.15.12:0/51664492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4de0105470 0x7f4de01a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.104+0000 7f4ddf400640 1 -- 172.21.15.12:0/51664492 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de01a1800 con 0x7f4de01038c0 2024-03-23T23:02:29.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.104+0000 7f4ddea00640 1 --2- 172.21.15.12:0/51664492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de0104ab0 0x7f4de019a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:02:29.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.105+0000 7f4ddf400640 1 --2- 172.21.15.12:0/51664492 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4de01038c0 0x7f4de0199be0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0000c00 tx=0x7f4dd002c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:29.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.105+0000 7f4dd7e00640 1 -- 172.21.15.12:0/51664492 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd0043020 con 0x7f4de01038c0 2024-03-23T23:02:29.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.105+0000 7f4dd7e00640 1 -- 172.21.15.12:0/51664492 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4dd00054e0 con 0x7f4de01038c0 2024-03-23T23:02:29.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.105+0000 7f4de6200640 1 -- 172.21.15.12:0/51664492 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de01a1a90 con 0x7f4de01038c0 2024-03-23T23:02:29.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.105+0000 7f4dd7e00640 1 -- 172.21.15.12:0/51664492 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd0003ce0 con 0x7f4de01038c0 2024-03-23T23:02:29.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.105+0000 7f4de6200640 1 -- 172.21.15.12:0/51664492 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de01a2000 con 0x7f4de01038c0 2024-03-23T23:02:29.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.108+0000 7f4dd7e00640 1 -- 172.21.15.12:0/51664492 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4dd0046020 con 0x7f4de01038c0 2024-03-23T23:02:29.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.108+0000 7f4de8fee740 1 Processor -- start 2024-03-23T23:02:29.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.109+0000 7f4de8fee740 1 -- start start 2024-03-23T23:02:29.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.109+0000 7f4de8fee740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557ca61e7e80 0x557ca61e82a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.109+0000 7f4de8fee740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557ca61e8870 0x557ca61ed370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.109+0000 7f4de8fee740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ca61ed9e0 0x557ca61efe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.109+0000 7f4de8fee740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca614d250 con 0x557ca61e7e80 2024-03-23T23:02:29.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.109+0000 7f4de8fee740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca60f5a30 con 0x557ca61e8870 2024-03-23T23:02:29.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.109+0000 7f4de8fee740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca61492f0 con 0x557ca61ed9e0 2024-03-23T23:02:29.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.109+0000 7f4ddf400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557ca61e8870 0x557ca61ed370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.109+0000 7f4ddfe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557ca61e7e80 0x557ca61e82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.109+0000 7f4ddfe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557ca61e7e80 0x557ca61e82a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54288/0 (socket says 172.21.15.12:54288) 2024-03-23T23:02:29.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.109+0000 7f4ddea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ca61ed9e0 0x557ca61efe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.109+0000 7f4ddfe00640 1 -- 172.21.15.12:0/247182244 learned_addr learned my addr 172.21.15.12:0/247182244 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:29.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.110+0000 7f4ddfe00640 1 -- 172.21.15.12:0/247182244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ca61ed9e0 msgr2=0x557ca61efe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:29.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.110+0000 7f4ddfe00640 1 --2- 172.21.15.12:0/247182244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ca61ed9e0 0x557ca61efe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.110+0000 7f4ddfe00640 1 -- 172.21.15.12:0/247182244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557ca61e8870 msgr2=0x557ca61ed370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:29.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.110+0000 7f4ddfe00640 1 --2- 172.21.15.12:0/247182244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557ca61e8870 0x557ca61ed370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.110+0000 7f4ddfe00640 1 -- 172.21.15.12:0/247182244 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ca60e0690 con 0x557ca61e7e80 2024-03-23T23:02:29.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.110+0000 7f4ddea00640 1 --2- 172.21.15.12:0/247182244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ca61ed9e0 0x557ca61efe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:02:29.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.110+0000 7f4dd7e00640 1 --2- 172.21.15.12:0/51664492 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4dc40a2880 0x7f4dc40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.110+0000 7f4dd7e00640 1 -- 172.21.15.12:0/51664492 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4dd00f0b00 con 0x7f4de01038c0 2024-03-23T23:02:29.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.110+0000 7f4ddfe00640 1 --2- 172.21.15.12:0/247182244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557ca61e7e80 0x557ca61e82a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4dd80039f0 tx=0x7f4dd8003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2fa8c46723764a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:29.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.110+0000 7f4ddf400640 1 --2- 172.21.15.12:0/51664492 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4dc40a2880 0x7f4dc40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.110+0000 7f4dcfe00640 1 -- 172.21.15.12:0/247182244 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd801a470 con 0x557ca61e7e80 2024-03-23T23:02:29.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.111+0000 7f4dcfe00640 1 -- 172.21.15.12:0/247182244 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4dd8015070 con 0x557ca61e7e80 2024-03-23T23:02:29.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.111+0000 7f4dcfe00640 1 -- 172.21.15.12:0/247182244 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd8010900 con 0x557ca61e7e80 2024-03-23T23:02:29.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.111+0000 7f4de8fee740 1 -- 172.21.15.12:0/247182244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557ca61e7e80 msgr2=0x557ca61e82a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:29.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.111+0000 7f4de8fee740 1 --2- 172.21.15.12:0/247182244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557ca61e7e80 0x557ca61e82a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4dd80039f0 tx=0x7f4dd8003eb0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.111+0000 7f4ddf400640 1 --2- 172.21.15.12:0/51664492 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4dc40a2880 0x7f4dc40a4d40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4dd00b98b0 tx=0x7f4dd00b9040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:29.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.111+0000 7f4de8fee740 1 -- 172.21.15.12:0/247182244 shutdown_connections 2024-03-23T23:02:29.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.111+0000 7f4de8fee740 1 --2- 172.21.15.12:0/247182244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ca61ed9e0 0x557ca61efe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.111+0000 7f4de8fee740 1 --2- 172.21.15.12:0/247182244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557ca61e8870 0x557ca61ed370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.111+0000 7f4de8fee740 1 --2- 172.21.15.12:0/247182244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557ca61e7e80 0x557ca61e82a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.111+0000 7f4de8fee740 1 -- 172.21.15.12:0/247182244 >> 172.21.15.12:0/247182244 conn(0x557ca61b0200 msgr2=0x557ca61e37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:29.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.111+0000 7f4de8fee740 1 -- 172.21.15.12:0/247182244 shutdown_connections 2024-03-23T23:02:29.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.111+0000 7f4de8fee740 1 -- 172.21.15.12:0/247182244 wait complete. 2024-03-23T23:02:29.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.112+0000 7f4de8fee740 1 -- start start 2024-03-23T23:02:29.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.112+0000 7f4de8fee740 1 Processor -- start 2024-03-23T23:02:29.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.112+0000 7f4de8fee740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557ca61e7e80 0x557ca61f1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.112+0000 7f4de8fee740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557ca61e8870 0x557ca61f2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.112+0000 7f4de8fee740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ca61ed9e0 0x557ca61f93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.112+0000 7f4de8fee740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca61f0990 con 0x557ca61e7e80 2024-03-23T23:02:29.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.113+0000 7f4de8fee740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca6108e80 con 0x557ca61e8870 2024-03-23T23:02:29.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.113+0000 7f4de8fee740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca61f0b10 con 0x557ca61ed9e0 2024-03-23T23:02:29.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.113+0000 7f4de8fee740 10 client.0 _ll_register_callbacks cb 0x557ca611b130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:02:29.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.113+0000 7f4ddea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557ca61e8870 0x557ca61f2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.113+0000 7f4ddfe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557ca61e7e80 0x557ca61f1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.113+0000 7f4ddea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557ca61e8870 0x557ca61f2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57226/0 (socket says 172.21.15.12:57226) 2024-03-23T23:02:29.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.113+0000 7f4ddea00640 1 -- 172.21.15.12:0/731185355 learned_addr learned my addr 172.21.15.12:0/731185355 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:29.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.113+0000 7f4ddea00640 1 -- 172.21.15.12:0/731185355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ca61ed9e0 msgr2=0x557ca61f93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:29.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.113+0000 7f4ddea00640 1 --2- 172.21.15.12:0/731185355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ca61ed9e0 0x557ca61f93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.113+0000 7f4ddea00640 1 -- 172.21.15.12:0/731185355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557ca61e7e80 msgr2=0x557ca61f1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:29.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.113+0000 7f4ddea00640 1 --2- 172.21.15.12:0/731185355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557ca61e7e80 0x557ca61f1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.113+0000 7f4ddea00640 1 -- 172.21.15.12:0/731185355 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557ca5f2b7d0 con 0x557ca61e8870 2024-03-23T23:02:29.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.113+0000 7f4ddea00640 1 --2- 172.21.15.12:0/731185355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557ca61e8870 0x557ca61f2400 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4dc8012d00 tx=0x7f4dc800f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86abec73e7b40b37 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:29.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.113+0000 7f4dccc00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:02:29.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.113+0000 7f4dccc00640 1 -- 172.21.15.12:0/731185355 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc8009d90 con 0x557ca61e8870 2024-03-23T23:02:29.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.114+0000 7f4de8fee740 1 -- 172.21.15.12:0/731185355 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4dd8015070 con 0x557ca61e8870 2024-03-23T23:02:29.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.114+0000 7f4dccc00640 1 -- 172.21.15.12:0/731185355 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4dc80120d0 con 0x557ca61e8870 2024-03-23T23:02:29.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.114+0000 7f4dccc00640 1 -- 172.21.15.12:0/731185355 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4dc8005020 con 0x557ca61e8870 2024-03-23T23:02:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.114+0000 7f4de8fee740 10 client.25400 fetch_fsmap learned FSMap version 23 2024-03-23T23:02:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.114+0000 7f4de8fee740 1 -- 172.21.15.12:0/731185355 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x557ca5e613f0 con 0x557ca61e8870 2024-03-23T23:02:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.114+0000 7f4dccc00640 1 -- 172.21.15.12:0/731185355 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4dd8015070 con 0x557ca61e8870 2024-03-23T23:02:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.114+0000 7f4dccc00640 1 -- 172.21.15.12:0/731185355 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f4dc801c470 con 0x557ca61e8870 2024-03-23T23:02:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.114+0000 7f4de8fee740 10 client.25400 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:02:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.114+0000 7f4de8fee740 10 client.25400 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:02:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.114+0000 7f4de8fee740 20 client.25400 populate_metadata read hostname 'smithi012' 2024-03-23T23:02:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.115+0000 7f4de8fee740 1 --2- 172.21.15.12:0/731185355 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557ca61fdc40 0x557ca6200080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.115+0000 7f4de8fee740 4 client.25400 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:02:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.115+0000 7f4de8fee740 1 -- 172.21.15.12:0/731185355 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557ca615a790 con 0x557ca61fdc40 2024-03-23T23:02:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.115+0000 7f4ddfe00640 1 --2- 172.21.15.12:0/731185355 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557ca61fdc40 0x557ca6200080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.115+0000 7f4ddfe00640 1 --2- 172.21.15.12:0/731185355 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557ca61fdc40 0x557ca6200080 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6246f8ceb9dadb6 server_cookie=cba175287ba2f892 in_seq=0 out_seq=0 2024-03-23T23:02:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.115+0000 7f4dccc00640 10 client.25400 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.116+0000 7f4dccc00640 1 -- 172.21.15.12:0/731185355 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557ca615a790 con 0x557ca61fdc40 2024-03-23T23:02:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.116+0000 7f4dccc00640 10 client.25400 handle_command_reply: tid=0 2024-03-23T23:02:29.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.116+0000 7f4de8fee740 1 client.25400 shutdown 2024-03-23T23:02:29.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.116+0000 7f4de8fee740 1 -- 172.21.15.12:0/731185355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557ca61e8870 msgr2=0x557ca61f2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:29.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.116+0000 7f4de8fee740 1 --2- 172.21.15.12:0/731185355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557ca61e8870 0x557ca61f2400 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4dc8012d00 tx=0x7f4dc800f790 comp rx=0 tx=0).stop 2024-03-23T23:02:29.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.116+0000 7f4de8fee740 1 -- 172.21.15.12:0/731185355 shutdown_connections 2024-03-23T23:02:29.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.116+0000 7f4de8fee740 1 --2- 172.21.15.12:0/731185355 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557ca61fdc40 0x557ca6200080 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.116+0000 7f4de8fee740 1 --2- 172.21.15.12:0/731185355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ca61ed9e0 0x557ca61f93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.116+0000 7f4de8fee740 1 --2- 172.21.15.12:0/731185355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557ca61e8870 0x557ca61f2400 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.116+0000 7f4de8fee740 1 --2- 172.21.15.12:0/731185355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557ca61e7e80 0x557ca61f1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.116+0000 7f4dccc00640 0 client.25400 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:29.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.116+0000 7f4de8fee740 1 -- 172.21.15.12:0/731185355 >> 172.21.15.12:0/731185355 conn(0x557ca61b0200 msgr2=0x557ca61e2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:29.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.117+0000 7f4de8fee740 1 -- 172.21.15.12:0/731185355 shutdown_connections 2024-03-23T23:02:29.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.117+0000 7f4de8fee740 1 -- 172.21.15.12:0/731185355 wait complete. 2024-03-23T23:02:29.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.117+0000 7f4de8fee740 20 client.25400 trim_cache size 0 max 16384 2024-03-23T23:02:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.135+0000 7f4de8fee740 1 Processor -- start 2024-03-23T23:02:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.135+0000 7f4de8fee740 1 -- start start 2024-03-23T23:02:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.135+0000 7f4de8fee740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557ca61e7e80 0x557ca61e6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.135+0000 7f4de8fee740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557ca61e8870 0x557ca61f9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.135+0000 7f4de8fee740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ca61ed9e0 0x557ca61e6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.135+0000 7f4de8fee740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de01ad130 con 0x557ca61e7e80 2024-03-23T23:02:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.135+0000 7f4de8fee740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de01aac30 con 0x557ca61e8870 2024-03-23T23:02:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.135+0000 7f4de8fee740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de01a8cd0 con 0x557ca61ed9e0 2024-03-23T23:02:29.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.135+0000 7f4ddfe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557ca61e7e80 0x557ca61e6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.136+0000 7f4ddfe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ca61ed9e0 0x557ca61e6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.136+0000 7f4ddfe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ca61ed9e0 0x557ca61e6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:40168/0 (socket says 172.21.15.12:40168) 2024-03-23T23:02:29.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.136+0000 7f4ddfe00640 1 -- 172.21.15.12:0/1988989832 learned_addr learned my addr 172.21.15.12:0/1988989832 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:29.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.136+0000 7f4ddea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557ca61e8870 0x557ca61f9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.136+0000 7f4ddfe00640 1 -- 172.21.15.12:0/1988989832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557ca61e8870 msgr2=0x557ca61f9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:29.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.136+0000 7f4ddfe00640 1 --2- 172.21.15.12:0/1988989832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557ca61e8870 0x557ca61f9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.136+0000 7f4ddfe00640 1 -- 172.21.15.12:0/1988989832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557ca61e7e80 msgr2=0x557ca61e6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:29.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.136+0000 7f4ddfe00640 1 --2- 172.21.15.12:0/1988989832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557ca61e7e80 0x557ca61e6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.136+0000 7f4ddfe00640 1 -- 172.21.15.12:0/1988989832 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ca5e210f0 con 0x557ca61ed9e0 2024-03-23T23:02:29.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.136+0000 7f4ddfe00640 1 --2- 172.21.15.12:0/1988989832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ca61ed9e0 0x557ca61e6f70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4dd80039f0 tx=0x7f4dd8009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5e248465e68c63b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:29.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.136+0000 7f4dcf400640 1 -- 172.21.15.12:0/1988989832 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd80048e0 con 0x557ca61ed9e0 2024-03-23T23:02:29.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.136+0000 7f4dcf400640 1 -- 172.21.15.12:0/1988989832 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4dd8015040 con 0x557ca61ed9e0 2024-03-23T23:02:29.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.136+0000 7f4dcf400640 1 -- 172.21.15.12:0/1988989832 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd80116a0 con 0x557ca61ed9e0 2024-03-23T23:02:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.137+0000 7f4de8fee740 1 -- 172.21.15.12:0/1988989832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ca61ed9e0 msgr2=0x557ca61e6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.137+0000 7f4de8fee740 1 --2- 172.21.15.12:0/1988989832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ca61ed9e0 0x557ca61e6f70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4dd80039f0 tx=0x7f4dd8009920 comp rx=0 tx=0).stop 2024-03-23T23:02:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.137+0000 7f4de8fee740 1 -- 172.21.15.12:0/1988989832 shutdown_connections 2024-03-23T23:02:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.137+0000 7f4de8fee740 1 --2- 172.21.15.12:0/1988989832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ca61ed9e0 0x557ca61e6f70 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.137+0000 7f4de8fee740 1 --2- 172.21.15.12:0/1988989832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557ca61e8870 0x557ca61f9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.137+0000 7f4de8fee740 1 --2- 172.21.15.12:0/1988989832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557ca61e7e80 0x557ca61e6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.137+0000 7f4de8fee740 1 -- 172.21.15.12:0/1988989832 >> 172.21.15.12:0/1988989832 conn(0x557ca61b0200 msgr2=0x557ca61e9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.137+0000 7f4de8fee740 1 -- 172.21.15.12:0/1988989832 shutdown_connections 2024-03-23T23:02:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.137+0000 7f4de8fee740 1 -- 172.21.15.12:0/1988989832 wait complete. 2024-03-23T23:02:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.137+0000 7f4de8fee740 1 -- start start 2024-03-23T23:02:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.138+0000 7f4de8fee740 1 Processor -- start 2024-03-23T23:02:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.138+0000 7f4de8fee740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557ca61e7e80 0x557ca61f7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.138+0000 7f4de8fee740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ca61e8870 0x557ca61f75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.138+0000 7f4de8fee740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557ca61ed9e0 0x557ca61f85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.138+0000 7f4de8fee740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de01a92d0 con 0x557ca61e7e80 2024-03-23T23:02:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.138+0000 7f4de8fee740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de01aa5d0 con 0x557ca61ed9e0 2024-03-23T23:02:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.138+0000 7f4de8fee740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de01a9910 con 0x557ca61e8870 2024-03-23T23:02:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.138+0000 7f4de8fee740 10 client.0 _ll_register_callbacks cb 0x7f4dc8009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:02:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.138+0000 7f4ddea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ca61e8870 0x557ca61f75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.138+0000 7f4ddfe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557ca61e7e80 0x557ca61f7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.138+0000 7f4ddea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ca61e8870 0x557ca61f75a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:40174/0 (socket says 172.21.15.12:40174) 2024-03-23T23:02:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.138+0000 7f4ddea00640 1 -- 172.21.15.12:0/3305721376 learned_addr learned my addr 172.21.15.12:0/3305721376 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.138+0000 7f4ddfe00640 1 --2- 172.21.15.12:0/3305721376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557ca61ed9e0 0x557ca61f85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.139+0000 7f4ddea00640 1 -- 172.21.15.12:0/3305721376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557ca61ed9e0 msgr2=0x557ca61f85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.139+0000 7f4ddea00640 1 --2- 172.21.15.12:0/3305721376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557ca61ed9e0 0x557ca61f85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.139+0000 7f4ddea00640 1 -- 172.21.15.12:0/3305721376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557ca61e7e80 msgr2=0x557ca61f7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.139+0000 7f4ddea00640 1 --2- 172.21.15.12:0/3305721376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557ca61e7e80 0x557ca61f7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.139+0000 7f4ddea00640 1 -- 172.21.15.12:0/3305721376 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557ca61f8ba0 con 0x557ca61e8870 2024-03-23T23:02:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.139+0000 7f4ddea00640 1 --2- 172.21.15.12:0/3305721376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ca61e8870 0x557ca61f75a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4dc8009cb0 tx=0x7f4dc8009fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c552b2bddb61fe2b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.139+0000 7f4dccc00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:02:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.139+0000 7f4dccc00640 1 -- 172.21.15.12:0/3305721376 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc80120d0 con 0x557ca61e8870 2024-03-23T23:02:29.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.139+0000 7f4dccc00640 1 -- 172.21.15.12:0/3305721376 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4dc8018040 con 0x557ca61e8870 2024-03-23T23:02:29.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.139+0000 7f4de8fee740 1 -- 172.21.15.12:0/3305721376 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4dd8015040 con 0x557ca61e8870 2024-03-23T23:02:29.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.140+0000 7f4dccc00640 1 -- 172.21.15.12:0/3305721376 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4dc8012270 con 0x557ca61e8870 2024-03-23T23:02:29.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.140+0000 7f4de8fee740 10 client.25384 fetch_fsmap learned FSMap version 23 2024-03-23T23:02:29.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.140+0000 7f4de8fee740 1 -- 172.21.15.12:0/3305721376 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x557ca62623a0 con 0x557ca61e8870 2024-03-23T23:02:29.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.140+0000 7f4dccc00640 1 -- 172.21.15.12:0/3305721376 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4dd8015040 con 0x557ca61e8870 2024-03-23T23:02:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.140+0000 7f4dccc00640 1 -- 172.21.15.12:0/3305721376 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f4dc8010430 con 0x557ca61e8870 2024-03-23T23:02:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.140+0000 7f4de8fee740 10 client.25384 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:02:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.140+0000 7f4de8fee740 10 client.25384 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:02:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.140+0000 7f4de8fee740 20 client.25384 populate_metadata read hostname 'smithi012' 2024-03-23T23:02:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.140+0000 7f4de8fee740 1 --2- 172.21.15.12:0/3305721376 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557ca6262cc0 0x557ca62650e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.140+0000 7f4de8fee740 4 client.25384 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:02:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.140+0000 7f4de8fee740 1 -- 172.21.15.12:0/3305721376 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557ca6154ae0 con 0x557ca6262cc0 2024-03-23T23:02:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.141+0000 7f4ddfe00640 1 --2- 172.21.15.12:0/3305721376 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557ca6262cc0 0x557ca62650e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.141+0000 7f4ddfe00640 1 --2- 172.21.15.12:0/3305721376 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557ca6262cc0 0x557ca62650e0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1f7de0be8496946 server_cookie=c3fab13e6f2d4f6f in_seq=0 out_seq=0 2024-03-23T23:02:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.141+0000 7f4dccc00640 10 client.25384 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.141+0000 7f4dccc00640 1 -- 172.21.15.12:0/3305721376 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x557ca6154ae0 con 0x557ca6262cc0 2024-03-23T23:02:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.141+0000 7f4dccc00640 10 client.25384 handle_command_reply: tid=0 2024-03-23T23:02:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.141+0000 7f4de8fee740 1 client.25384 shutdown 2024-03-23T23:02:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.142+0000 7f4de8fee740 1 -- 172.21.15.12:0/3305721376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ca61e8870 msgr2=0x557ca61f75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:29.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.142+0000 7f4de8fee740 1 --2- 172.21.15.12:0/3305721376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ca61e8870 0x557ca61f75a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4dc8009cb0 tx=0x7f4dc8009fc0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.142+0000 7f4de8fee740 1 -- 172.21.15.12:0/3305721376 shutdown_connections 2024-03-23T23:02:29.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.142+0000 7f4de8fee740 1 --2- 172.21.15.12:0/3305721376 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557ca6262cc0 0x557ca62650e0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.142+0000 7f4de8fee740 1 --2- 172.21.15.12:0/3305721376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557ca61ed9e0 0x557ca61f85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.142+0000 7f4de8fee740 1 --2- 172.21.15.12:0/3305721376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ca61e8870 0x557ca61f75a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.142+0000 7f4de8fee740 1 --2- 172.21.15.12:0/3305721376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557ca61e7e80 0x557ca61f7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.142+0000 7f4de8fee740 1 -- 172.21.15.12:0/3305721376 >> 172.21.15.12:0/3305721376 conn(0x557ca61b0200 msgr2=0x557ca61e9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:29.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.142+0000 7f4dccc00640 0 client.25384 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:29.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.142+0000 7f4de8fee740 1 -- 172.21.15.12:0/3305721376 shutdown_connections 2024-03-23T23:02:29.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.142+0000 7f4de8fee740 1 -- 172.21.15.12:0/3305721376 wait complete. 2024-03-23T23:02:29.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.142+0000 7f4de8fee740 20 client.25384 trim_cache size 0 max 16384 2024-03-23T23:02:29.144 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:02:29.145 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-03-23T23:02:29.145 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-03-23T23:02:29.145 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:02:29.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.142+0000 7f4de6200640 1 -- 172.21.15.12:0/51664492 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4dc40a2880 msgr2=0x7f4dc40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:29.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.142+0000 7f4de6200640 1 --2- 172.21.15.12:0/51664492 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4dc40a2880 0x7f4dc40a4d40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4dd00b98b0 tx=0x7f4dd00b9040 comp rx=0 tx=0).stop 2024-03-23T23:02:29.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.143+0000 7f4de6200640 1 -- 172.21.15.12:0/51664492 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4de01038c0 msgr2=0x7f4de0199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:29.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.143+0000 7f4de6200640 1 --2- 172.21.15.12:0/51664492 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4de01038c0 0x7f4de0199be0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0000c00 tx=0x7f4dd002c6c0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.143+0000 7f4de6200640 1 -- 172.21.15.12:0/51664492 shutdown_connections 2024-03-23T23:02:29.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.143+0000 7f4de6200640 1 --2- 172.21.15.12:0/51664492 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4dc40a2880 0x7f4dc40a4d40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.143+0000 7f4de6200640 1 --2- 172.21.15.12:0/51664492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4de0105470 0x7f4de01a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.143+0000 7f4de6200640 1 --2- 172.21.15.12:0/51664492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de0104ab0 0x7f4de019a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.143+0000 7f4de6200640 1 --2- 172.21.15.12:0/51664492 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4de01038c0 0x7f4de0199be0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.143+0000 7f4de6200640 1 -- 172.21.15.12:0/51664492 >> 172.21.15.12:0/51664492 conn(0x7f4de00fee50 msgr2=0x7f4de0107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:29.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.143+0000 7f4de6200640 1 -- 172.21.15.12:0/51664492 shutdown_connections 2024-03-23T23:02:29.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.143+0000 7f4de6200640 1 -- 172.21.15.12:0/51664492 wait complete. 2024-03-23T23:02:29.159 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T23:02:29.160 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:02:29.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.384+0000 7f008ee00640 1 -- 172.21.15.12:0/2007146634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00881016c0 msgr2=0x7f0088101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:29.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.384+0000 7f008ee00640 1 --2- 172.21.15.12:0/2007146634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00881016c0 0x7f0088101ae0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0078009900 tx=0x7f007802bc50 comp rx=0 tx=0).stop 2024-03-23T23:02:29.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.384+0000 7f008ee00640 1 -- 172.21.15.12:0/2007146634 shutdown_connections 2024-03-23T23:02:29.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.384+0000 7f008ee00640 1 --2- 172.21.15.12:0/2007146634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0088103270 0x7f0088109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.384+0000 7f008ee00640 1 --2- 172.21.15.12:0/2007146634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00881028b0 0x7f0088102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.384+0000 7f008ee00640 1 --2- 172.21.15.12:0/2007146634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00881016c0 0x7f0088101ae0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.384+0000 7f008ee00640 1 -- 172.21.15.12:0/2007146634 >> 172.21.15.12:0/2007146634 conn(0x7f00880fcc50 msgr2=0x7f00880ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:29.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.385+0000 7f008ee00640 1 -- 172.21.15.12:0/2007146634 shutdown_connections 2024-03-23T23:02:29.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.385+0000 7f008ee00640 1 -- 172.21.15.12:0/2007146634 wait complete. 2024-03-23T23:02:29.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.386+0000 7f008ee00640 1 Processor -- start 2024-03-23T23:02:29.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.386+0000 7f008ee00640 1 -- start start 2024-03-23T23:02:29.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.386+0000 7f008ee00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00881016c0 0x7f0088199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.386+0000 7f008ee00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00881028b0 0x7f008819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.386+0000 7f008ee00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0088103270 0x7f00881a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.386+0000 7f008ee00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008810d360 con 0x7f0088103270 2024-03-23T23:02:29.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.386+0000 7f008ee00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008810d1e0 con 0x7f00881016c0 2024-03-23T23:02:29.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.386+0000 7f008ee00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008810d4e0 con 0x7f00881028b0 2024-03-23T23:02:29.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.386+0000 7f0087e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00881016c0 0x7f0088199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.386+0000 7f0087e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00881016c0 0x7f0088199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57280/0 (socket says 172.21.15.12:57280) 2024-03-23T23:02:29.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.386+0000 7f0087e00640 1 -- 172.21.15.12:0/1213836952 learned_addr learned my addr 172.21.15.12:0/1213836952 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:29.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.386+0000 7f008ce00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0088103270 0x7f00881a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.387+0000 7f0087400640 1 --2- 172.21.15.12:0/1213836952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00881028b0 0x7f008819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.387+0000 7f0087e00640 1 -- 172.21.15.12:0/1213836952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00881028b0 msgr2=0x7f008819a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:29.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.387+0000 7f0087e00640 1 --2- 172.21.15.12:0/1213836952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00881028b0 0x7f008819a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.387+0000 7f0087e00640 1 -- 172.21.15.12:0/1213836952 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0088103270 msgr2=0x7f00881a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:29.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.387+0000 7f0087e00640 1 --2- 172.21.15.12:0/1213836952 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0088103270 0x7f00881a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.387+0000 7f0087e00640 1 -- 172.21.15.12:0/1213836952 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00881a1810 con 0x7f00881016c0 2024-03-23T23:02:29.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.387+0000 7f0087e00640 1 --2- 172.21.15.12:0/1213836952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00881016c0 0x7f0088199bf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f007802c260 tx=0x7f007802c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:29.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.387+0000 7f0084c00640 1 -- 172.21.15.12:0/1213836952 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0078041070 con 0x7f00881016c0 2024-03-23T23:02:29.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.387+0000 7f0084c00640 1 -- 172.21.15.12:0/1213836952 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0078004450 con 0x7f00881016c0 2024-03-23T23:02:29.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.387+0000 7f0084c00640 1 -- 172.21.15.12:0/1213836952 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0078034440 con 0x7f00881016c0 2024-03-23T23:02:29.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.388+0000 7f008ee00640 1 -- 172.21.15.12:0/1213836952 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00881a1aa0 con 0x7f00881016c0 2024-03-23T23:02:29.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.388+0000 7f008ee00640 1 -- 172.21.15.12:0/1213836952 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00881a1fa0 con 0x7f00881016c0 2024-03-23T23:02:29.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.389+0000 7f008ee00640 1 -- 172.21.15.12:0/1213836952 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00881a21d0 con 0x7f00881016c0 2024-03-23T23:02:29.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.391+0000 7f0084c00640 1 -- 172.21.15.12:0/1213836952 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0078044070 con 0x7f00881016c0 2024-03-23T23:02:29.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.392+0000 7f0084c00640 1 --2- 172.21.15.12:0/1213836952 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f006c0a2880 0x7f006c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.392+0000 7f0084c00640 1 -- 172.21.15.12:0/1213836952 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f00780f0560 con 0x7f00881016c0 2024-03-23T23:02:29.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.392+0000 7f0087400640 1 --2- 172.21.15.12:0/1213836952 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f006c0a2880 0x7f006c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.393+0000 7f0087400640 1 --2- 172.21.15.12:0/1213836952 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f006c0a2880 0x7f006c0a4d40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f008819b100 tx=0x7f0074009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:29.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.396+0000 7f0084c00640 1 -- 172.21.15.12:0/1213836952 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f00780bd5b0 con 0x7f00881016c0 2024-03-23T23:02:29.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.644+0000 7f008ee00640 1 -- 172.21.15.12:0/1213836952 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00881a23a0 con 0x7f00881016c0 2024-03-23T23:02:29.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.645+0000 7f0084c00640 1 -- 172.21.15.12:0/1213836952 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f00780bceb0 con 0x7f00881016c0 2024-03-23T23:02:29.645 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:02:29.645 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:02:29.645 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:02:29.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.648+0000 7f008ee00640 1 -- 172.21.15.12:0/1213836952 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f006c0a2880 msgr2=0x7f006c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:29.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.648+0000 7f008ee00640 1 --2- 172.21.15.12:0/1213836952 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f006c0a2880 0x7f006c0a4d40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f008819b100 tx=0x7f0074009500 comp rx=0 tx=0).stop 2024-03-23T23:02:29.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.649+0000 7f008ee00640 1 -- 172.21.15.12:0/1213836952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00881016c0 msgr2=0x7f0088199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:29.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.649+0000 7f008ee00640 1 --2- 172.21.15.12:0/1213836952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00881016c0 0x7f0088199bf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f007802c260 tx=0x7f007802c670 comp rx=0 tx=0).stop 2024-03-23T23:02:29.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.649+0000 7f008ee00640 1 -- 172.21.15.12:0/1213836952 shutdown_connections 2024-03-23T23:02:29.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.649+0000 7f008ee00640 1 --2- 172.21.15.12:0/1213836952 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f006c0a2880 0x7f006c0a4d40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.649+0000 7f008ee00640 1 --2- 172.21.15.12:0/1213836952 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0088103270 0x7f00881a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.649+0000 7f008ee00640 1 --2- 172.21.15.12:0/1213836952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00881028b0 0x7f008819a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.649+0000 7f008ee00640 1 --2- 172.21.15.12:0/1213836952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00881016c0 0x7f0088199bf0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.649+0000 7f008ee00640 1 -- 172.21.15.12:0/1213836952 >> 172.21.15.12:0/1213836952 conn(0x7f00880fcc50 msgr2=0x7f00880738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:29.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.649+0000 7f008ee00640 1 -- 172.21.15.12:0/1213836952 shutdown_connections 2024-03-23T23:02:29.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.649+0000 7f008ee00640 1 -- 172.21.15.12:0/1213836952 wait complete. 2024-03-23T23:02:29.668 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:02:29.669 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-23T23:02:29.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.898+0000 7f9478c00640 1 -- 172.21.15.12:0/2563631957 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9474104ad0 msgr2=0x7f9474104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:29.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.898+0000 7f9478c00640 1 --2- 172.21.15.12:0/2563631957 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9474104ad0 0x7f9474104f50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f945c009900 tx=0x7f945c02bc50 comp rx=0 tx=0).stop 2024-03-23T23:02:29.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.898+0000 7f9478c00640 1 -- 172.21.15.12:0/2563631957 shutdown_connections 2024-03-23T23:02:29.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.898+0000 7f9478c00640 1 --2- 172.21.15.12:0/2563631957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9474105490 0x7f947410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.898+0000 7f9478c00640 1 --2- 172.21.15.12:0/2563631957 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9474104ad0 0x7f9474104f50 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.898+0000 7f9478c00640 1 --2- 172.21.15.12:0/2563631957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94741038e0 0x7f9474103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.898+0000 7f9478c00640 1 -- 172.21.15.12:0/2563631957 >> 172.21.15.12:0/2563631957 conn(0x7f94740fee50 msgr2=0x7f9474101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:29.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.899+0000 7f9478c00640 1 -- 172.21.15.12:0/2563631957 shutdown_connections 2024-03-23T23:02:29.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.899+0000 7f9478c00640 1 -- 172.21.15.12:0/2563631957 wait complete. 2024-03-23T23:02:29.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.900+0000 7f9478c00640 1 Processor -- start 2024-03-23T23:02:29.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.900+0000 7f9478c00640 1 -- start start 2024-03-23T23:02:29.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.901+0000 7f9478c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94741038e0 0x7f9474199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.901+0000 7f9478c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9474104ad0 0x7f947419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.901+0000 7f9478c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9474105490 0x7f94741a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.901+0000 7f9478c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947410f590 con 0x7f9474104ad0 2024-03-23T23:02:29.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.901+0000 7f9478c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947410f410 con 0x7f9474105490 2024-03-23T23:02:29.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.901+0000 7f9478c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947410f710 con 0x7f94741038e0 2024-03-23T23:02:29.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.901+0000 7f9471e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94741038e0 0x7f9474199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.901+0000 7f9471e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94741038e0 0x7f9474199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:40214/0 (socket says 172.21.15.12:40214) 2024-03-23T23:02:29.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.901+0000 7f9471e00640 1 -- 172.21.15.12:0/601282665 learned_addr learned my addr 172.21.15.12:0/601282665 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:29.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.901+0000 7f9471400640 1 --2- 172.21.15.12:0/601282665 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9474104ad0 0x7f947419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.901+0000 7f9472800640 1 --2- 172.21.15.12:0/601282665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9474105490 0x7f94741a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.901+0000 7f9471e00640 1 -- 172.21.15.12:0/601282665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9474105490 msgr2=0x7f94741a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:29.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.901+0000 7f9471e00640 1 --2- 172.21.15.12:0/601282665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9474105490 0x7f94741a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.901+0000 7f9471e00640 1 -- 172.21.15.12:0/601282665 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9474104ad0 msgr2=0x7f947419a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:29.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.901+0000 7f9471e00640 1 --2- 172.21.15.12:0/601282665 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9474104ad0 0x7f947419a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.901+0000 7f9471e00640 1 -- 172.21.15.12:0/601282665 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94741a1830 con 0x7f94741038e0 2024-03-23T23:02:29.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.901+0000 7f9472800640 1 --2- 172.21.15.12:0/601282665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9474105490 0x7f94741a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:02:29.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.902+0000 7f9471e00640 1 --2- 172.21.15.12:0/601282665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94741038e0 0x7f9474199c20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f946400ebc0 tx=0x7f946400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:29.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.902+0000 7f946aa00640 1 -- 172.21.15.12:0/601282665 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9464003b70 con 0x7f94741038e0 2024-03-23T23:02:29.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.902+0000 7f946aa00640 1 -- 172.21.15.12:0/601282665 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9464003d10 con 0x7f94741038e0 2024-03-23T23:02:29.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.902+0000 7f946aa00640 1 -- 172.21.15.12:0/601282665 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9464019980 con 0x7f94741038e0 2024-03-23T23:02:29.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.902+0000 7f9478c00640 1 -- 172.21.15.12:0/601282665 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94741a1b20 con 0x7f94741038e0 2024-03-23T23:02:29.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.902+0000 7f9478c00640 1 -- 172.21.15.12:0/601282665 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94741a20c0 con 0x7f94741038e0 2024-03-23T23:02:29.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.905+0000 7f946aa00640 1 -- 172.21.15.12:0/601282665 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9464019b20 con 0x7f94741038e0 2024-03-23T23:02:29.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.905+0000 7f947b857740 1 Processor -- start 2024-03-23T23:02:29.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.906+0000 7f947b857740 1 -- start start 2024-03-23T23:02:29.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.906+0000 7f947b857740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 0x55abec5b7c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.906+0000 7f947b857740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55abec5b89e0 0x55abec5b8e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.906+0000 7f947b857740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abec5b93a0 0x55abec5bfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.906+0000 7f947b857740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abec51c2c0 con 0x55abec5b89e0 2024-03-23T23:02:29.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.906+0000 7f947b857740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abec4c4c90 con 0x55abec5b7810 2024-03-23T23:02:29.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.906+0000 7f947b857740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abec518360 con 0x55abec5b93a0 2024-03-23T23:02:29.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.906+0000 7f9472800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 0x55abec5b7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.906+0000 7f9471e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55abec5b89e0 0x55abec5b8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.906+0000 7f9472800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 0x55abec5b7c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57308/0 (socket says 172.21.15.12:57308) 2024-03-23T23:02:29.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.906+0000 7f9472800640 1 -- 172.21.15.12:0/66250914 learned_addr learned my addr 172.21.15.12:0/66250914 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:29.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.906+0000 7f9471400640 1 --2- 172.21.15.12:0/66250914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abec5b93a0 0x55abec5bfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.907+0000 7f9472800640 1 -- 172.21.15.12:0/66250914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abec5b93a0 msgr2=0x55abec5bfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:29.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.907+0000 7f9472800640 1 --2- 172.21.15.12:0/66250914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abec5b93a0 0x55abec5bfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.907+0000 7f9472800640 1 -- 172.21.15.12:0/66250914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55abec5b89e0 msgr2=0x55abec5b8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:29.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.907+0000 7f9472800640 1 --2- 172.21.15.12:0/66250914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55abec5b89e0 0x55abec5b8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.907+0000 7f9472800640 1 -- 172.21.15.12:0/66250914 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55abec4e9080 con 0x55abec5b7810 2024-03-23T23:02:29.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.907+0000 7f946aa00640 1 --2- 172.21.15.12:0/601282665 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f94580a2880 0x7f94580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.907+0000 7f9472800640 1 --2- 172.21.15.12:0/66250914 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 0x55abec5b7c30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f946c009cc0 tx=0x7f946c0117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6f86a94e25b84e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:29.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.907+0000 7f946aa00640 1 -- 172.21.15.12:0/601282665 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f94640cf420 con 0x7f94741038e0 2024-03-23T23:02:29.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.907+0000 7f9463000640 1 -- 172.21.15.12:0/66250914 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946c012450 con 0x55abec5b7810 2024-03-23T23:02:29.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.907+0000 7f9471e00640 1 --2- 172.21.15.12:0/601282665 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f94580a2880 0x7f94580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.907+0000 7f9463000640 1 -- 172.21.15.12:0/66250914 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f946c012ad0 con 0x55abec5b7810 2024-03-23T23:02:29.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.907+0000 7f9463000640 1 -- 172.21.15.12:0/66250914 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946c00fc70 con 0x55abec5b7810 2024-03-23T23:02:29.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.908+0000 7f947b857740 1 -- 172.21.15.12:0/66250914 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 msgr2=0x55abec5b7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:29.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.908+0000 7f9471e00640 1 --2- 172.21.15.12:0/601282665 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f94580a2880 0x7f94580a4d40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f94640a3fe0 tx=0x7f94640d2e60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:29.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.908+0000 7f947b857740 1 --2- 172.21.15.12:0/66250914 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 0x55abec5b7c30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f946c009cc0 tx=0x7f946c0117b0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.908+0000 7f947b857740 1 -- 172.21.15.12:0/66250914 shutdown_connections 2024-03-23T23:02:29.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.908+0000 7f947b857740 1 --2- 172.21.15.12:0/66250914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abec5b93a0 0x55abec5bfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.908+0000 7f947b857740 1 --2- 172.21.15.12:0/66250914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55abec5b89e0 0x55abec5b8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.908+0000 7f947b857740 1 --2- 172.21.15.12:0/66250914 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 0x55abec5b7c30 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.908+0000 7f947b857740 1 -- 172.21.15.12:0/66250914 >> 172.21.15.12:0/66250914 conn(0x55abec5b2c00 msgr2=0x55abec5b3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:29.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.908+0000 7f947b857740 1 -- 172.21.15.12:0/66250914 shutdown_connections 2024-03-23T23:02:29.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.908+0000 7f947b857740 1 -- 172.21.15.12:0/66250914 wait complete. 2024-03-23T23:02:29.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.909+0000 7f947b857740 1 -- start start 2024-03-23T23:02:29.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.909+0000 7f947b857740 1 Processor -- start 2024-03-23T23:02:29.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.909+0000 7f947b857740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 0x55abec5c1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.909+0000 7f947b857740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55abec5b89e0 0x55abec5c1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.910+0000 7f947b857740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abec5b93a0 0x55abec5c8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.910+0000 7f947b857740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abec5c03b0 con 0x55abec5b89e0 2024-03-23T23:02:29.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.910+0000 7f947b857740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abec4d80d0 con 0x55abec5b7810 2024-03-23T23:02:29.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.910+0000 7f947b857740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abec5c0530 con 0x55abec5b93a0 2024-03-23T23:02:29.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.910+0000 7f947b857740 10 client.0 _ll_register_callbacks cb 0x55abec4dce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:02:29.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.910+0000 7f9472800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 0x55abec5c1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.910+0000 7f9471400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55abec5b89e0 0x55abec5c1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.910+0000 7f9472800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 0x55abec5c1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57318/0 (socket says 172.21.15.12:57318) 2024-03-23T23:02:29.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.910+0000 7f9472800640 1 -- 172.21.15.12:0/1519290564 learned_addr learned my addr 172.21.15.12:0/1519290564 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:29.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.910+0000 7f9472800640 1 --2- 172.21.15.12:0/1519290564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abec5b93a0 0x55abec5c8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.910+0000 7f9471400640 1 -- 172.21.15.12:0/1519290564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abec5b93a0 msgr2=0x55abec5c8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:29.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.910+0000 7f9471400640 1 --2- 172.21.15.12:0/1519290564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abec5b93a0 0x55abec5c8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.910+0000 7f9471400640 1 -- 172.21.15.12:0/1519290564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 msgr2=0x55abec5c1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:29.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.910+0000 7f9471400640 1 --2- 172.21.15.12:0/1519290564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 0x55abec5c1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.910+0000 7f9471400640 1 -- 172.21.15.12:0/1519290564 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55abec2303f0 con 0x55abec5b89e0 2024-03-23T23:02:29.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.911+0000 7f9471400640 1 --2- 172.21.15.12:0/1519290564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55abec5b89e0 0x55abec5c1f40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f945c036040 tx=0x7f945c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6061c586e458977c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:29.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.911+0000 7f9457400640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:02:29.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.911+0000 7f9457400640 1 -- 172.21.15.12:0/1519290564 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c005d50 con 0x55abec5b89e0 2024-03-23T23:02:29.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.911+0000 7f9457400640 1 -- 172.21.15.12:0/1519290564 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f945c005020 con 0x55abec5b89e0 2024-03-23T23:02:29.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.911+0000 7f947b857740 1 -- 172.21.15.12:0/1519290564 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f946c012ad0 con 0x55abec5b89e0 2024-03-23T23:02:29.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.911+0000 7f9457400640 1 -- 172.21.15.12:0/1519290564 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f945c005220 con 0x55abec5b89e0 2024-03-23T23:02:29.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.911+0000 7f947b857740 10 client.15291 fetch_fsmap learned FSMap version 23 2024-03-23T23:02:29.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.911+0000 7f947b857740 1 -- 172.21.15.12:0/1519290564 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55abec2fa7d0 con 0x55abec5b89e0 2024-03-23T23:02:29.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.912+0000 7f9457400640 1 -- 172.21.15.12:0/1519290564 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f946c012ad0 con 0x55abec5b89e0 2024-03-23T23:02:29.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.912+0000 7f9457400640 1 -- 172.21.15.12:0/1519290564 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f945c0035f0 con 0x55abec5b89e0 2024-03-23T23:02:29.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.912+0000 7f947b857740 10 client.15291 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:02:29.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.912+0000 7f947b857740 10 client.15291 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:02:29.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.912+0000 7f947b857740 20 client.15291 populate_metadata read hostname 'smithi012' 2024-03-23T23:02:29.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.912+0000 7f947b857740 1 --2- 172.21.15.12:0/1519290564 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55abec5cd440 0x55abec5cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.912+0000 7f947b857740 4 client.15291 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:02:29.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.912+0000 7f947b857740 1 -- 172.21.15.12:0/1519290564 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55abec529800 con 0x55abec5cd440 2024-03-23T23:02:29.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.912+0000 7f9472800640 1 --2- 172.21.15.12:0/1519290564 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55abec5cd440 0x55abec5cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.913+0000 7f9472800640 1 --2- 172.21.15.12:0/1519290564 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55abec5cd440 0x55abec5cf850 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e9c38642a03c98a4 server_cookie=5d9179307d124018 in_seq=0 out_seq=0 2024-03-23T23:02:29.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.913+0000 7f9457400640 10 client.15291 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:02:29.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.914+0000 7f9457400640 1 -- 172.21.15.12:0/1519290564 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55abec529800 con 0x55abec5cd440 2024-03-23T23:02:29.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.914+0000 7f9457400640 10 client.15291 handle_command_reply: tid=0 2024-03-23T23:02:29.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.914+0000 7f947b857740 1 client.15291 shutdown 2024-03-23T23:02:29.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.914+0000 7f947b857740 1 -- 172.21.15.12:0/1519290564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55abec5b89e0 msgr2=0x55abec5c1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:29.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.914+0000 7f947b857740 1 --2- 172.21.15.12:0/1519290564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55abec5b89e0 0x55abec5c1f40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f945c036040 tx=0x7f945c004270 comp rx=0 tx=0).stop 2024-03-23T23:02:29.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.914+0000 7f947b857740 1 -- 172.21.15.12:0/1519290564 shutdown_connections 2024-03-23T23:02:29.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.914+0000 7f947b857740 1 --2- 172.21.15.12:0/1519290564 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55abec5cd440 0x55abec5cf850 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.914+0000 7f947b857740 1 --2- 172.21.15.12:0/1519290564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abec5b93a0 0x55abec5c8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.914+0000 7f947b857740 1 --2- 172.21.15.12:0/1519290564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55abec5b89e0 0x55abec5c1f40 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.914+0000 7f947b857740 1 --2- 172.21.15.12:0/1519290564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 0x55abec5c1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.914+0000 7f9457400640 0 client.15291 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:02:29.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.914+0000 7f947b857740 1 -- 172.21.15.12:0/1519290564 >> 172.21.15.12:0/1519290564 conn(0x55abec5b2c00 msgr2=0x55abec5b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:29.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.914+0000 7f947b857740 1 -- 172.21.15.12:0/1519290564 shutdown_connections 2024-03-23T23:02:29.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.914+0000 7f947b857740 1 -- 172.21.15.12:0/1519290564 wait complete. 2024-03-23T23:02:29.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.914+0000 7f947b857740 20 client.15291 trim_cache size 0 max 16384 2024-03-23T23:02:29.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.928+0000 7f947b857740 1 Processor -- start 2024-03-23T23:02:29.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.929+0000 7f947b857740 1 -- start start 2024-03-23T23:02:29.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.929+0000 7f947b857740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 0x55abec5b1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.929+0000 7f947b857740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55abec5b89e0 0x55abec5c8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.929+0000 7f947b857740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abec5b93a0 0x55abec5d4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.929+0000 7f947b857740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94741ad1c0 con 0x55abec5b89e0 2024-03-23T23:02:29.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.929+0000 7f947b857740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94741aacc0 con 0x55abec5b7810 2024-03-23T23:02:29.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.929+0000 7f947b857740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94741a8d60 con 0x55abec5b93a0 2024-03-23T23:02:29.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.929+0000 7f9471400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55abec5b89e0 0x55abec5c8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.929+0000 7f9472800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 0x55abec5b1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.929+0000 7f9472800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 0x55abec5b1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57322/0 (socket says 172.21.15.12:57322) 2024-03-23T23:02:29.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.929+0000 7f9472800640 1 -- 172.21.15.12:0/3856882630 learned_addr learned my addr 172.21.15.12:0/3856882630 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:29.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.929+0000 7f9472800640 1 --2- 172.21.15.12:0/3856882630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abec5b93a0 0x55abec5d4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.930+0000 7f9472800640 1 -- 172.21.15.12:0/3856882630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abec5b93a0 msgr2=0x55abec5d4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:29.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.930+0000 7f9472800640 1 --2- 172.21.15.12:0/3856882630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abec5b93a0 0x55abec5d4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.930+0000 7f9472800640 1 -- 172.21.15.12:0/3856882630 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55abec5b89e0 msgr2=0x55abec5c8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:29.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.930+0000 7f9472800640 1 --2- 172.21.15.12:0/3856882630 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55abec5b89e0 0x55abec5c8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.930+0000 7f9472800640 1 -- 172.21.15.12:0/3856882630 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55abec5b2970 con 0x55abec5b7810 2024-03-23T23:02:29.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.930+0000 7f9472800640 1 --2- 172.21.15.12:0/3856882630 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 0x55abec5b1d30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f946c009fd0 tx=0x7f946c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30c664829ec88a8d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:29.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.930+0000 7f9462600640 1 -- 172.21.15.12:0/3856882630 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946c02e2c0 con 0x55abec5b7810 2024-03-23T23:02:29.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.930+0000 7f9462600640 1 -- 172.21.15.12:0/3856882630 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f946c02e8c0 con 0x55abec5b7810 2024-03-23T23:02:29.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.930+0000 7f9462600640 1 -- 172.21.15.12:0/3856882630 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946c00fb40 con 0x55abec5b7810 2024-03-23T23:02:29.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.931+0000 7f947b857740 1 -- 172.21.15.12:0/3856882630 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 msgr2=0x55abec5b1d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:29.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.931+0000 7f947b857740 1 --2- 172.21.15.12:0/3856882630 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 0x55abec5b1d30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f946c009fd0 tx=0x7f946c005f90 comp rx=0 tx=0).stop 2024-03-23T23:02:29.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.931+0000 7f947b857740 1 -- 172.21.15.12:0/3856882630 shutdown_connections 2024-03-23T23:02:29.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.931+0000 7f947b857740 1 --2- 172.21.15.12:0/3856882630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abec5b93a0 0x55abec5d4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.931+0000 7f947b857740 1 --2- 172.21.15.12:0/3856882630 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55abec5b89e0 0x55abec5c8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.931+0000 7f947b857740 1 --2- 172.21.15.12:0/3856882630 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 0x55abec5b1d30 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.931+0000 7f947b857740 1 -- 172.21.15.12:0/3856882630 >> 172.21.15.12:0/3856882630 conn(0x55abec5b2c00 msgr2=0x55abec5b1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:29.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.931+0000 7f947b857740 1 -- 172.21.15.12:0/3856882630 shutdown_connections 2024-03-23T23:02:29.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.931+0000 7f947b857740 1 -- 172.21.15.12:0/3856882630 wait complete. 2024-03-23T23:02:29.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.931+0000 7f947b857740 1 -- start start 2024-03-23T23:02:29.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.932+0000 7f947b857740 1 Processor -- start 2024-03-23T23:02:29.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.932+0000 7f947b857740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 0x55abec5c8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.932+0000 7f947b857740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55abec5b89e0 0x55abec5c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.932+0000 7f947b857740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abec5b93a0 0x55abec5c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.932+0000 7f947b857740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94741a9360 con 0x55abec5b89e0 2024-03-23T23:02:29.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.932+0000 7f947b857740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94741aa660 con 0x55abec5b7810 2024-03-23T23:02:29.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.932+0000 7f947b857740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94741a99a0 con 0x55abec5b93a0 2024-03-23T23:02:29.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.932+0000 7f947b857740 10 client.0 _ll_register_callbacks cb 0x7f946c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:02:29.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.932+0000 7f9471400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55abec5b89e0 0x55abec5c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.932+0000 7f9472800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 0x55abec5c8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.932+0000 7f9471400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55abec5b89e0 0x55abec5c73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54416/0 (socket says 172.21.15.12:54416) 2024-03-23T23:02:29.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.932+0000 7f9471400640 1 -- 172.21.15.12:0/43679549 learned_addr learned my addr 172.21.15.12:0/43679549 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:29.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.932+0000 7f9472800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 0x55abec5c8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57332/0 (socket says 172.21.15.12:57332) 2024-03-23T23:02:29.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.932+0000 7f9472800640 1 --2- 172.21.15.12:0/43679549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abec5b93a0 0x55abec5c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.933+0000 7f9471400640 1 -- 172.21.15.12:0/43679549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abec5b93a0 msgr2=0x55abec5c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:29.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.933+0000 7f9471400640 1 --2- 172.21.15.12:0/43679549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abec5b93a0 0x55abec5c78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.933+0000 7f9471400640 1 -- 172.21.15.12:0/43679549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 msgr2=0x55abec5c8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:29.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.933+0000 7f9471400640 1 --2- 172.21.15.12:0/43679549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 0x55abec5c8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.933+0000 7f9471400640 1 -- 172.21.15.12:0/43679549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55abec5c7fd0 con 0x55abec5b89e0 2024-03-23T23:02:29.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.933+0000 7f9471400640 1 --2- 172.21.15.12:0/43679549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55abec5b89e0 0x55abec5c73a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f945c001eb0 tx=0x7f945c02c2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f1312a83e110654 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:29.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.933+0000 7f9457400640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:02:29.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.933+0000 7f9457400640 1 -- 172.21.15.12:0/43679549 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c003dd0 con 0x55abec5b89e0 2024-03-23T23:02:29.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.933+0000 7f947b857740 1 -- 172.21.15.12:0/43679549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f946c02e8c0 con 0x55abec5b89e0 2024-03-23T23:02:29.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.933+0000 7f9457400640 1 -- 172.21.15.12:0/43679549 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f945c0053c0 con 0x55abec5b89e0 2024-03-23T23:02:29.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.934+0000 7f9457400640 1 -- 172.21.15.12:0/43679549 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f945c02cd40 con 0x55abec5b89e0 2024-03-23T23:02:29.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.934+0000 7f947b857740 10 client.15297 fetch_fsmap learned FSMap version 23 2024-03-23T23:02:29.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.934+0000 7f947b857740 1 -- 172.21.15.12:0/43679549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55abec629c50 con 0x55abec5b89e0 2024-03-23T23:02:29.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.934+0000 7f9457400640 1 -- 172.21.15.12:0/43679549 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f946c02e8c0 con 0x55abec5b89e0 2024-03-23T23:02:29.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.934+0000 7f9457400640 1 -- 172.21.15.12:0/43679549 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f945c004f00 con 0x55abec5b89e0 2024-03-23T23:02:29.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.934+0000 7f947b857740 10 client.15297 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:02:29.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.934+0000 7f947b857740 10 client.15297 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:02:29.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.934+0000 7f947b857740 20 client.15297 populate_metadata read hostname 'smithi012' 2024-03-23T23:02:29.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.934+0000 7f947b857740 1 --2- 172.21.15.12:0/43679549 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55abec62a5a0 0x55abec62c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:29.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.934+0000 7f947b857740 4 client.15297 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:02:29.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.934+0000 7f947b857740 1 -- 172.21.15.12:0/43679549 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55abec523b50 con 0x55abec62a5a0 2024-03-23T23:02:29.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.934+0000 7f9472800640 1 --2- 172.21.15.12:0/43679549 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55abec62a5a0 0x55abec62c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:29.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.935+0000 7f9472800640 1 --2- 172.21.15.12:0/43679549 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55abec62a5a0 0x55abec62c9c0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6754e91551d2d14e server_cookie=678a8907522be379 in_seq=0 out_seq=0 2024-03-23T23:02:29.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.935+0000 7f9457400640 10 client.15297 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:02:29.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.935+0000 7f9457400640 1 -- 172.21.15.12:0/43679549 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55abec523b50 con 0x55abec62a5a0 2024-03-23T23:02:29.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.935+0000 7f9457400640 10 client.15297 handle_command_reply: tid=0 2024-03-23T23:02:29.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.935+0000 7f947b857740 1 client.15297 shutdown 2024-03-23T23:02:29.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.935+0000 7f947b857740 1 -- 172.21.15.12:0/43679549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55abec5b89e0 msgr2=0x55abec5c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:29.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.935+0000 7f947b857740 1 --2- 172.21.15.12:0/43679549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55abec5b89e0 0x55abec5c73a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f945c001eb0 tx=0x7f945c02c2f0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.935+0000 7f947b857740 1 -- 172.21.15.12:0/43679549 shutdown_connections 2024-03-23T23:02:29.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.935+0000 7f947b857740 1 --2- 172.21.15.12:0/43679549 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55abec62a5a0 0x55abec62c9c0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.935+0000 7f947b857740 1 --2- 172.21.15.12:0/43679549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abec5b93a0 0x55abec5c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.935+0000 7f947b857740 1 --2- 172.21.15.12:0/43679549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55abec5b89e0 0x55abec5c73a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.935+0000 7f947b857740 1 --2- 172.21.15.12:0/43679549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abec5b7810 0x55abec5c8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.935+0000 7f9457400640 0 client.15297 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:02:29.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.935+0000 7f947b857740 1 -- 172.21.15.12:0/43679549 >> 172.21.15.12:0/43679549 conn(0x55abec5b2c00 msgr2=0x55abec5bc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:29.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.936+0000 7f947b857740 1 -- 172.21.15.12:0/43679549 shutdown_connections 2024-03-23T23:02:29.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.936+0000 7f947b857740 1 -- 172.21.15.12:0/43679549 wait complete. 2024-03-23T23:02:29.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.936+0000 7f947b857740 20 client.15297 trim_cache size 0 max 16384 2024-03-23T23:02:29.941 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:02:29.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.936+0000 7f9478c00640 1 -- 172.21.15.12:0/601282665 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f94580a2880 msgr2=0x7f94580a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:29.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.936+0000 7f9478c00640 1 --2- 172.21.15.12:0/601282665 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f94580a2880 0x7f94580a4d40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f94640a3fe0 tx=0x7f94640d2e60 comp rx=0 tx=0).stop 2024-03-23T23:02:29.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.936+0000 7f9478c00640 1 -- 172.21.15.12:0/601282665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94741038e0 msgr2=0x7f9474199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:29.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.936+0000 7f9478c00640 1 --2- 172.21.15.12:0/601282665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94741038e0 0x7f9474199c20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f946400ebc0 tx=0x7f946400c6a0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.936+0000 7f9478c00640 1 -- 172.21.15.12:0/601282665 shutdown_connections 2024-03-23T23:02:29.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.936+0000 7f9478c00640 1 --2- 172.21.15.12:0/601282665 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f94580a2880 0x7f94580a4d40 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.936+0000 7f9478c00640 1 --2- 172.21.15.12:0/601282665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9474105490 0x7f94741a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.936+0000 7f9478c00640 1 --2- 172.21.15.12:0/601282665 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9474104ad0 0x7f947419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.936+0000 7f9478c00640 1 --2- 172.21.15.12:0/601282665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94741038e0 0x7f9474199c20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:29.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.936+0000 7f9478c00640 1 -- 172.21.15.12:0/601282665 >> 172.21.15.12:0/601282665 conn(0x7f94740fee50 msgr2=0x7f9474107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:29.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.937+0000 7f9478c00640 1 -- 172.21.15.12:0/601282665 shutdown_connections 2024-03-23T23:02:29.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:29.937+0000 7f9478c00640 1 -- 172.21.15.12:0/601282665 wait complete. 2024-03-23T23:02:29.956 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:02:29.957 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-23T23:02:30.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.175+0000 7f7c42a00640 1 -- 172.21.15.12:0/1953647190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c3c108eb0 msgr2=0x7f7c3c109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:30.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.175+0000 7f7c42a00640 1 --2- 172.21.15.12:0/1953647190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c3c108eb0 0x7f7c3c109330 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c009900 tx=0x7f7c2c02bc30 comp rx=0 tx=0).stop 2024-03-23T23:02:30.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.175+0000 7f7c42a00640 1 -- 172.21.15.12:0/1953647190 shutdown_connections 2024-03-23T23:02:30.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.175+0000 7f7c42a00640 1 --2- 172.21.15.12:0/1953647190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c3c109870 0x7f7c3c110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.175+0000 7f7c42a00640 1 --2- 172.21.15.12:0/1953647190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c3c108eb0 0x7f7c3c109330 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.175+0000 7f7c42a00640 1 --2- 172.21.15.12:0/1953647190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c3c107cc0 0x7f7c3c1080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.175+0000 7f7c42a00640 1 -- 172.21.15.12:0/1953647190 >> 172.21.15.12:0/1953647190 conn(0x7f7c3c076c60 msgr2=0x7f7c3c0790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:30.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.175+0000 7f7c42a00640 1 -- 172.21.15.12:0/1953647190 shutdown_connections 2024-03-23T23:02:30.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.175+0000 7f7c42a00640 1 -- 172.21.15.12:0/1953647190 wait complete. 2024-03-23T23:02:30.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.176+0000 7f7c42a00640 1 Processor -- start 2024-03-23T23:02:30.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.177+0000 7f7c42a00640 1 -- start start 2024-03-23T23:02:30.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.177+0000 7f7c42a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c3c107cc0 0x7f7c3c19dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.177+0000 7f7c42a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c3c108eb0 0x7f7c3c19e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.177+0000 7f7c42a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c3c109870 0x7f7c3c1a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.177+0000 7f7c42a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c113990 con 0x7f7c3c109870 2024-03-23T23:02:30.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.177+0000 7f7c42a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c113810 con 0x7f7c3c107cc0 2024-03-23T23:02:30.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.177+0000 7f7c42a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c113b10 con 0x7f7c3c108eb0 2024-03-23T23:02:30.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.177+0000 7f7c3be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c3c107cc0 0x7f7c3c19dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.177+0000 7f7c3b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c3c108eb0 0x7f7c3c19e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.177+0000 7f7c3b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c3c108eb0 0x7f7c3c19e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:40262/0 (socket says 172.21.15.12:40262) 2024-03-23T23:02:30.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.177+0000 7f7c3b400640 1 -- 172.21.15.12:0/3079293258 learned_addr learned my addr 172.21.15.12:0/3079293258 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:30.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.177+0000 7f7c3be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c3c107cc0 0x7f7c3c19dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57352/0 (socket says 172.21.15.12:57352) 2024-03-23T23:02:30.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.177+0000 7f7c40a00640 1 --2- 172.21.15.12:0/3079293258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c3c109870 0x7f7c3c1a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.178+0000 7f7c3b400640 1 -- 172.21.15.12:0/3079293258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c3c107cc0 msgr2=0x7f7c3c19dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:30.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.178+0000 7f7c3b400640 1 --2- 172.21.15.12:0/3079293258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c3c107cc0 0x7f7c3c19dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.178+0000 7f7c3b400640 1 -- 172.21.15.12:0/3079293258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c3c109870 msgr2=0x7f7c3c1a5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:30.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.178+0000 7f7c3b400640 1 --2- 172.21.15.12:0/3079293258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c3c109870 0x7f7c3c1a5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.178+0000 7f7c3b400640 1 -- 172.21.15.12:0/3079293258 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c3c1a5bc0 con 0x7f7c3c108eb0 2024-03-23T23:02:30.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.178+0000 7f7c3b400640 1 --2- 172.21.15.12:0/3079293258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c3c108eb0 0x7f7c3c19e4e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c000c00 tx=0x7f7c2c02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:30.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.179+0000 7f7c38c00640 1 -- 172.21.15.12:0/3079293258 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c2c043020 con 0x7f7c3c108eb0 2024-03-23T23:02:30.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.179+0000 7f7c38c00640 1 -- 172.21.15.12:0/3079293258 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c2c0054e0 con 0x7f7c3c108eb0 2024-03-23T23:02:30.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.179+0000 7f7c38c00640 1 -- 172.21.15.12:0/3079293258 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c2c003c40 con 0x7f7c3c108eb0 2024-03-23T23:02:30.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.179+0000 7f7c42a00640 1 -- 172.21.15.12:0/3079293258 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c3c1a5e50 con 0x7f7c3c108eb0 2024-03-23T23:02:30.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.179+0000 7f7c42a00640 1 -- 172.21.15.12:0/3079293258 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c3c1a6360 con 0x7f7c3c108eb0 2024-03-23T23:02:30.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.182+0000 7f7c38c00640 1 -- 172.21.15.12:0/3079293258 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7c2c046020 con 0x7f7c3c108eb0 2024-03-23T23:02:30.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.183+0000 7f7c45771740 1 Processor -- start 2024-03-23T23:02:30.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.183+0000 7f7c45771740 1 -- start start 2024-03-23T23:02:30.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.183+0000 7f7c45771740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccc810 0x563764cccc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.183+0000 7f7c45771740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563764ccd9e0 0x563764ccde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.183+0000 7f7c45771740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563764cce3a0 0x563764cd4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.183+0000 7f7c45771740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563764c312c0 con 0x563764ccd9e0 2024-03-23T23:02:30.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.183+0000 7f7c45771740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x563764bd9c90 con 0x563764ccc810 2024-03-23T23:02:30.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.183+0000 7f7c45771740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x563764c2d360 con 0x563764cce3a0 2024-03-23T23:02:30.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.184+0000 7f7c40a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccc810 0x563764cccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.184+0000 7f7c40a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccc810 0x563764cccc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57358/0 (socket says 172.21.15.12:57358) 2024-03-23T23:02:30.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.184+0000 7f7c3be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563764ccd9e0 0x563764ccde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.184+0000 7f7c40a00640 1 -- 172.21.15.12:0/3883648305 learned_addr learned my addr 172.21.15.12:0/3883648305 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:30.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.184+0000 7f7c3b400640 1 --2- 172.21.15.12:0/3883648305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563764cce3a0 0x563764cd4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.184+0000 7f7c40a00640 1 -- 172.21.15.12:0/3883648305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563764cce3a0 msgr2=0x563764cd4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:30.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.184+0000 7f7c40a00640 1 --2- 172.21.15.12:0/3883648305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563764cce3a0 0x563764cd4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.184+0000 7f7c40a00640 1 -- 172.21.15.12:0/3883648305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563764ccd9e0 msgr2=0x563764ccde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:30.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.184+0000 7f7c40a00640 1 --2- 172.21.15.12:0/3883648305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563764ccd9e0 0x563764ccde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.184+0000 7f7c40a00640 1 -- 172.21.15.12:0/3883648305 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563764bfe080 con 0x563764ccc810 2024-03-23T23:02:30.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.184+0000 7f7c38c00640 1 --2- 172.21.15.12:0/3079293258 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7c200a2880 0x7f7c200a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.184+0000 7f7c40a00640 1 --2- 172.21.15.12:0/3883648305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccc810 0x563764cccc30 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7c34009fd0 tx=0x7f7c340118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90c76ad16550751c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:30.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.185+0000 7f7c38c00640 1 -- 172.21.15.12:0/3079293258 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7c2c0f0a70 con 0x7f7c3c108eb0 2024-03-23T23:02:30.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.185+0000 7f7c31200640 1 -- 172.21.15.12:0/3883648305 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c34004d20 con 0x563764ccc810 2024-03-23T23:02:30.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.185+0000 7f7c31200640 1 -- 172.21.15.12:0/3883648305 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c34004050 con 0x563764ccc810 2024-03-23T23:02:30.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.185+0000 7f7c31200640 1 -- 172.21.15.12:0/3883648305 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c34005260 con 0x563764ccc810 2024-03-23T23:02:30.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.185+0000 7f7c3be00640 1 --2- 172.21.15.12:0/3079293258 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7c200a2880 0x7f7c200a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.185+0000 7f7c3be00640 1 --2- 172.21.15.12:0/3079293258 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7c200a2880 0x7f7c200a4d40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c108d10 tx=0x7f7c24005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:30.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.185+0000 7f7c45771740 1 -- 172.21.15.12:0/3883648305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccc810 msgr2=0x563764cccc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:30.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.185+0000 7f7c45771740 1 --2- 172.21.15.12:0/3883648305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccc810 0x563764cccc30 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7c34009fd0 tx=0x7f7c340118d0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.186+0000 7f7c45771740 1 -- 172.21.15.12:0/3883648305 shutdown_connections 2024-03-23T23:02:30.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.186+0000 7f7c45771740 1 --2- 172.21.15.12:0/3883648305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563764cce3a0 0x563764cd4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.186+0000 7f7c45771740 1 --2- 172.21.15.12:0/3883648305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563764ccd9e0 0x563764ccde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.186+0000 7f7c45771740 1 --2- 172.21.15.12:0/3883648305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccc810 0x563764cccc30 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.186+0000 7f7c45771740 1 -- 172.21.15.12:0/3883648305 >> 172.21.15.12:0/3883648305 conn(0x563764cc7c00 msgr2=0x563764cc8000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:30.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.186+0000 7f7c45771740 1 -- 172.21.15.12:0/3883648305 shutdown_connections 2024-03-23T23:02:30.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.186+0000 7f7c45771740 1 -- 172.21.15.12:0/3883648305 wait complete. 2024-03-23T23:02:30.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.186+0000 7f7c45771740 1 -- start start 2024-03-23T23:02:30.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.186+0000 7f7c45771740 1 Processor -- start 2024-03-23T23:02:30.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.187+0000 7f7c45771740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563764ccc810 0x563764cd6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.187+0000 7f7c45771740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccd9e0 0x563764cd7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.187+0000 7f7c45771740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563764cce3a0 0x563764cde040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.187+0000 7f7c45771740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563764cd53b0 con 0x563764cce3a0 2024-03-23T23:02:30.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.187+0000 7f7c45771740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x563764bed0d0 con 0x563764ccd9e0 2024-03-23T23:02:30.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.187+0000 7f7c45771740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x563764cd5530 con 0x563764ccc810 2024-03-23T23:02:30.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.187+0000 7f7c45771740 10 client.0 _ll_register_callbacks cb 0x563764bf1e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:02:30.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.187+0000 7f7c3be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563764ccc810 0x563764cd6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.187+0000 7f7c3be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563764ccc810 0x563764cd6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:40272/0 (socket says 172.21.15.12:40272) 2024-03-23T23:02:30.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.187+0000 7f7c40a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563764cce3a0 0x563764cde040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.187+0000 7f7c3be00640 1 -- 172.21.15.12:0/2749872208 learned_addr learned my addr 172.21.15.12:0/2749872208 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:30.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.187+0000 7f7c3b400640 1 --2- 172.21.15.12:0/2749872208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccd9e0 0x563764cd7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.188+0000 7f7c3be00640 1 -- 172.21.15.12:0/2749872208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccd9e0 msgr2=0x563764cd7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:30.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.188+0000 7f7c3be00640 1 --2- 172.21.15.12:0/2749872208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccd9e0 0x563764cd7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.188+0000 7f7c3be00640 1 -- 172.21.15.12:0/2749872208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563764cce3a0 msgr2=0x563764cde040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:30.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.188+0000 7f7c3be00640 1 --2- 172.21.15.12:0/2749872208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563764cce3a0 0x563764cde040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.188+0000 7f7c3be00640 1 -- 172.21.15.12:0/2749872208 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5637649453f0 con 0x563764ccc810 2024-03-23T23:02:30.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.188+0000 7f7c3b400640 1 --2- 172.21.15.12:0/2749872208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccd9e0 0x563764cd7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:02:30.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.188+0000 7f7c3be00640 1 --2- 172.21.15.12:0/2749872208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563764ccc810 0x563764cd6ac0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7c24004630 tx=0x7f7c24004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e719b2177393407b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:30.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.188+0000 7f7c29600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:02:30.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.188+0000 7f7c29600640 1 -- 172.21.15.12:0/2749872208 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c24009e60 con 0x563764ccc810 2024-03-23T23:02:30.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.188+0000 7f7c29600640 1 -- 172.21.15.12:0/2749872208 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7c2401f630 con 0x563764ccc810 2024-03-23T23:02:30.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.188+0000 7f7c45771740 1 -- 172.21.15.12:0/2749872208 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7c34004050 con 0x563764ccc810 2024-03-23T23:02:30.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.189+0000 7f7c29600640 1 -- 172.21.15.12:0/2749872208 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c2401f830 con 0x563764ccc810 2024-03-23T23:02:30.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.189+0000 7f7c45771740 10 client.25420 fetch_fsmap learned FSMap version 23 2024-03-23T23:02:30.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.189+0000 7f7c45771740 1 -- 172.21.15.12:0/2749872208 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563764a0f7d0 con 0x563764ccc810 2024-03-23T23:02:30.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.189+0000 7f7c29600640 1 -- 172.21.15.12:0/2749872208 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c34004050 con 0x563764ccc810 2024-03-23T23:02:30.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.189+0000 7f7c29600640 1 -- 172.21.15.12:0/2749872208 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f7c24027070 con 0x563764ccc810 2024-03-23T23:02:30.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.189+0000 7f7c45771740 10 client.25420 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:02:30.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.189+0000 7f7c45771740 10 client.25420 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:02:30.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.189+0000 7f7c45771740 20 client.25420 populate_metadata read hostname 'smithi012' 2024-03-23T23:02:30.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.189+0000 7f7c45771740 1 --2- 172.21.15.12:0/2749872208 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x563764ce2540 0x563764ce49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.189+0000 7f7c45771740 4 client.25420 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:02:30.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.189+0000 7f7c45771740 1 -- 172.21.15.12:0/2749872208 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563764c3e800 con 0x563764ce2540 2024-03-23T23:02:30.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.190+0000 7f7c3b400640 1 --2- 172.21.15.12:0/2749872208 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x563764ce2540 0x563764ce49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.190+0000 7f7c3b400640 1 --2- 172.21.15.12:0/2749872208 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x563764ce2540 0x563764ce49c0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=251318f7fd4cc904 server_cookie=b2a9310839024a69 in_seq=0 out_seq=0 2024-03-23T23:02:30.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.190+0000 7f7c29600640 10 client.25420 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:02:30.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.191+0000 7f7c29600640 1 -- 172.21.15.12:0/2749872208 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563764c3e800 con 0x563764ce2540 2024-03-23T23:02:30.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.191+0000 7f7c29600640 10 client.25420 handle_command_reply: tid=0 2024-03-23T23:02:30.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.191+0000 7f7c45771740 1 client.25420 shutdown 2024-03-23T23:02:30.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.191+0000 7f7c45771740 1 -- 172.21.15.12:0/2749872208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563764ccc810 msgr2=0x563764cd6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:30.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.191+0000 7f7c45771740 1 --2- 172.21.15.12:0/2749872208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563764ccc810 0x563764cd6ac0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7c24004630 tx=0x7f7c24004730 comp rx=0 tx=0).stop 2024-03-23T23:02:30.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.191+0000 7f7c45771740 1 -- 172.21.15.12:0/2749872208 shutdown_connections 2024-03-23T23:02:30.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.191+0000 7f7c45771740 1 --2- 172.21.15.12:0/2749872208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563764cce3a0 0x563764cde040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.191+0000 7f7c45771740 1 --2- 172.21.15.12:0/2749872208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccd9e0 0x563764cd7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.191+0000 7f7c45771740 1 --2- 172.21.15.12:0/2749872208 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x563764ce2540 0x563764ce49c0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.191+0000 7f7c45771740 1 --2- 172.21.15.12:0/2749872208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563764ccc810 0x563764cd6ac0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.191+0000 7f7c29600640 0 client.25420 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:02:30.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.191+0000 7f7c45771740 1 -- 172.21.15.12:0/2749872208 >> 172.21.15.12:0/2749872208 conn(0x563764cc7c00 msgr2=0x563764cc6eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:30.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.191+0000 7f7c45771740 1 -- 172.21.15.12:0/2749872208 shutdown_connections 2024-03-23T23:02:30.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.191+0000 7f7c45771740 1 -- 172.21.15.12:0/2749872208 wait complete. 2024-03-23T23:02:30.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.191+0000 7f7c45771740 20 client.25420 trim_cache size 0 max 16384 2024-03-23T23:02:30.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.205+0000 7f7c45771740 1 Processor -- start 2024-03-23T23:02:30.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.205+0000 7f7c45771740 1 -- start start 2024-03-23T23:02:30.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.206+0000 7f7c45771740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccc810 0x563764cd0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.206+0000 7f7c45771740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563764ccd9e0 0x563764cde000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.206+0000 7f7c45771740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563764cce3a0 0x563764cd0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.206+0000 7f7c45771740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c1b14c0 con 0x563764cce3a0 2024-03-23T23:02:30.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.206+0000 7f7c45771740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c1aefc0 con 0x563764ccc810 2024-03-23T23:02:30.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.206+0000 7f7c45771740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c1ad060 con 0x563764ccd9e0 2024-03-23T23:02:30.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.206+0000 7f7c3be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccc810 0x563764cd0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.206+0000 7f7c3be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccc810 0x563764cd0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57374/0 (socket says 172.21.15.12:57374) 2024-03-23T23:02:30.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.206+0000 7f7c3b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563764ccd9e0 0x563764cde000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.206+0000 7f7c3be00640 1 -- 172.21.15.12:0/879070042 learned_addr learned my addr 172.21.15.12:0/879070042 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:30.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.206+0000 7f7c40a00640 1 --2- 172.21.15.12:0/879070042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563764cce3a0 0x563764cd0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.206+0000 7f7c3be00640 1 -- 172.21.15.12:0/879070042 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563764ccd9e0 msgr2=0x563764cde000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:30.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.206+0000 7f7c3be00640 1 --2- 172.21.15.12:0/879070042 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563764ccd9e0 0x563764cde000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.206+0000 7f7c3be00640 1 -- 172.21.15.12:0/879070042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563764cce3a0 msgr2=0x563764cd0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:30.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.206+0000 7f7c3be00640 1 --2- 172.21.15.12:0/879070042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563764cce3a0 0x563764cd0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.206+0000 7f7c3be00640 1 -- 172.21.15.12:0/879070042 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563764cc8650 con 0x563764ccc810 2024-03-23T23:02:30.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.206+0000 7f7c3b400640 1 --2- 172.21.15.12:0/879070042 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563764ccd9e0 0x563764cde000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:02:30.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.207+0000 7f7c3be00640 1 --2- 172.21.15.12:0/879070042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccc810 0x563764cd0a80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7c24000c00 tx=0x7f7c24009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e195876eb98a6af8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:30.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.207+0000 7f7c2be00640 1 -- 172.21.15.12:0/879070042 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c24038030 con 0x563764ccc810 2024-03-23T23:02:30.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.207+0000 7f7c2be00640 1 -- 172.21.15.12:0/879070042 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c2402f030 con 0x563764ccc810 2024-03-23T23:02:30.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.207+0000 7f7c2be00640 1 -- 172.21.15.12:0/879070042 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c240333f0 con 0x563764ccc810 2024-03-23T23:02:30.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.207+0000 7f7c45771740 1 -- 172.21.15.12:0/879070042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccc810 msgr2=0x563764cd0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:30.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.207+0000 7f7c45771740 1 --2- 172.21.15.12:0/879070042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccc810 0x563764cd0a80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7c24000c00 tx=0x7f7c24009b90 comp rx=0 tx=0).stop 2024-03-23T23:02:30.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.208+0000 7f7c45771740 1 -- 172.21.15.12:0/879070042 shutdown_connections 2024-03-23T23:02:30.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.208+0000 7f7c45771740 1 --2- 172.21.15.12:0/879070042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563764cce3a0 0x563764cd0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.208+0000 7f7c45771740 1 --2- 172.21.15.12:0/879070042 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563764ccd9e0 0x563764cde000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.208+0000 7f7c45771740 1 --2- 172.21.15.12:0/879070042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccc810 0x563764cd0a80 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.208+0000 7f7c45771740 1 -- 172.21.15.12:0/879070042 >> 172.21.15.12:0/879070042 conn(0x563764cc7c00 msgr2=0x563764ccb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:30.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.208+0000 7f7c45771740 1 -- 172.21.15.12:0/879070042 shutdown_connections 2024-03-23T23:02:30.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.208+0000 7f7c45771740 1 -- 172.21.15.12:0/879070042 wait complete. 2024-03-23T23:02:30.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.208+0000 7f7c45771740 1 -- start start 2024-03-23T23:02:30.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.208+0000 7f7c45771740 1 Processor -- start 2024-03-23T23:02:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.209+0000 7f7c45771740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccc810 0x563764cda0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.209+0000 7f7c45771740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563764ccd9e0 0x563764cda620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.209+0000 7f7c45771740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563764cce3a0 0x563764cdd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.209+0000 7f7c45771740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c1ad660 con 0x563764cce3a0 2024-03-23T23:02:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.209+0000 7f7c45771740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c1ae960 con 0x563764ccc810 2024-03-23T23:02:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.209+0000 7f7c45771740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c1adca0 con 0x563764ccd9e0 2024-03-23T23:02:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.209+0000 7f7c45771740 10 client.0 _ll_register_callbacks cb 0x7f7c240044c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:02:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.209+0000 7f7c3be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccc810 0x563764cda0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.209+0000 7f7c3b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563764ccd9e0 0x563764cda620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.209+0000 7f7c3b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563764ccd9e0 0x563764cda620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:40290/0 (socket says 172.21.15.12:40290) 2024-03-23T23:02:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.209+0000 7f7c3be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccc810 0x563764cda0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57390/0 (socket says 172.21.15.12:57390) 2024-03-23T23:02:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.209+0000 7f7c3b400640 1 -- 172.21.15.12:0/1635004224 learned_addr learned my addr 172.21.15.12:0/1635004224 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.209+0000 7f7c40a00640 1 --2- 172.21.15.12:0/1635004224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563764cce3a0 0x563764cdd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.209+0000 7f7c3b400640 1 -- 172.21.15.12:0/1635004224 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccc810 msgr2=0x563764cda0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.209+0000 7f7c3b400640 1 --2- 172.21.15.12:0/1635004224 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccc810 0x563764cda0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.209+0000 7f7c3b400640 1 -- 172.21.15.12:0/1635004224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563764cce3a0 msgr2=0x563764cdd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.209+0000 7f7c3b400640 1 --2- 172.21.15.12:0/1635004224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563764cce3a0 0x563764cdd5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.209+0000 7f7c3b400640 1 -- 172.21.15.12:0/1635004224 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563764cddbe0 con 0x563764ccd9e0 2024-03-23T23:02:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.209+0000 7f7c3be00640 1 --2- 172.21.15.12:0/1635004224 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccc810 0x563764cda0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:02:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.210+0000 7f7c3b400640 1 --2- 172.21.15.12:0/1635004224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563764ccd9e0 0x563764cda620 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c0096c0 tx=0x7f7c2c0b9040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e976f012d93e9e75 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.210+0000 7f7c29600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:02:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.210+0000 7f7c29600640 1 -- 172.21.15.12:0/1635004224 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c2c0f0e60 con 0x563764ccd9e0 2024-03-23T23:02:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.210+0000 7f7c29600640 1 -- 172.21.15.12:0/1635004224 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7c2c0bb6c0 con 0x563764ccd9e0 2024-03-23T23:02:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.210+0000 7f7c45771740 1 -- 172.21.15.12:0/1635004224 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7c2402f030 con 0x563764ccd9e0 2024-03-23T23:02:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.210+0000 7f7c29600640 1 -- 172.21.15.12:0/1635004224 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c2c004250 con 0x563764ccd9e0 2024-03-23T23:02:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.210+0000 7f7c45771740 10 client.25426 fetch_fsmap learned FSMap version 23 2024-03-23T23:02:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.210+0000 7f7c45771740 1 -- 172.21.15.12:0/1635004224 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563764d3e8c0 con 0x563764ccd9e0 2024-03-23T23:02:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.210+0000 7f7c29600640 1 -- 172.21.15.12:0/1635004224 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c2402f030 con 0x563764ccd9e0 2024-03-23T23:02:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.210+0000 7f7c29600640 1 -- 172.21.15.12:0/1635004224 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f7c2c0c1150 con 0x563764ccd9e0 2024-03-23T23:02:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.211+0000 7f7c45771740 10 client.25426 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:02:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.211+0000 7f7c45771740 10 client.25426 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:02:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.211+0000 7f7c45771740 20 client.25426 populate_metadata read hostname 'smithi012' 2024-03-23T23:02:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.211+0000 7f7c45771740 1 --2- 172.21.15.12:0/1635004224 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x563764d3f220 0x563764d41640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.211+0000 7f7c45771740 4 client.25426 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:02:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.211+0000 7f7c45771740 1 -- 172.21.15.12:0/1635004224 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563764c38b50 con 0x563764d3f220 2024-03-23T23:02:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.211+0000 7f7c3be00640 1 --2- 172.21.15.12:0/1635004224 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x563764d3f220 0x563764d41640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.211+0000 7f7c3be00640 1 --2- 172.21.15.12:0/1635004224 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x563764d3f220 0x563764d41640 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e04764d05d3d4c50 server_cookie=a54dbd8410760ae8 in_seq=0 out_seq=0 2024-03-23T23:02:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.211+0000 7f7c29600640 10 client.25426 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:02:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.211+0000 7f7c29600640 1 -- 172.21.15.12:0/1635004224 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563764c38b50 con 0x563764d3f220 2024-03-23T23:02:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.211+0000 7f7c29600640 10 client.25426 handle_command_reply: tid=0 2024-03-23T23:02:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.212+0000 7f7c45771740 1 client.25426 shutdown 2024-03-23T23:02:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.212+0000 7f7c45771740 1 -- 172.21.15.12:0/1635004224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563764ccd9e0 msgr2=0x563764cda620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.212+0000 7f7c45771740 1 --2- 172.21.15.12:0/1635004224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563764ccd9e0 0x563764cda620 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c0096c0 tx=0x7f7c2c0b9040 comp rx=0 tx=0).stop 2024-03-23T23:02:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.212+0000 7f7c45771740 1 -- 172.21.15.12:0/1635004224 shutdown_connections 2024-03-23T23:02:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.212+0000 7f7c45771740 1 --2- 172.21.15.12:0/1635004224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563764cce3a0 0x563764cdd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.212+0000 7f7c45771740 1 --2- 172.21.15.12:0/1635004224 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x563764d3f220 0x563764d41640 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.212+0000 7f7c45771740 1 --2- 172.21.15.12:0/1635004224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563764ccd9e0 0x563764cda620 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.212+0000 7f7c45771740 1 --2- 172.21.15.12:0/1635004224 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563764ccc810 0x563764cda0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.212+0000 7f7c45771740 1 -- 172.21.15.12:0/1635004224 >> 172.21.15.12:0/1635004224 conn(0x563764cc7c00 msgr2=0x563764cc6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.212+0000 7f7c29600640 0 client.25426 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:02:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.212+0000 7f7c45771740 1 -- 172.21.15.12:0/1635004224 shutdown_connections 2024-03-23T23:02:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.212+0000 7f7c45771740 1 -- 172.21.15.12:0/1635004224 wait complete. 2024-03-23T23:02:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.212+0000 7f7c45771740 20 client.25426 trim_cache size 0 max 16384 2024-03-23T23:02:30.216 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:02:30.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.213+0000 7f7c42a00640 1 -- 172.21.15.12:0/3079293258 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7c200a2880 msgr2=0x7f7c200a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.213+0000 7f7c42a00640 1 --2- 172.21.15.12:0/3079293258 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7c200a2880 0x7f7c200a4d40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c108d10 tx=0x7f7c24005f90 comp rx=0 tx=0).stop 2024-03-23T23:02:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.213+0000 7f7c42a00640 1 -- 172.21.15.12:0/3079293258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c3c108eb0 msgr2=0x7f7c3c19e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.213+0000 7f7c42a00640 1 --2- 172.21.15.12:0/3079293258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c3c108eb0 0x7f7c3c19e4e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c000c00 tx=0x7f7c2c02c650 comp rx=0 tx=0).stop 2024-03-23T23:02:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.213+0000 7f7c42a00640 1 -- 172.21.15.12:0/3079293258 shutdown_connections 2024-03-23T23:02:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.213+0000 7f7c42a00640 1 --2- 172.21.15.12:0/3079293258 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7c200a2880 0x7f7c200a4d40 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.213+0000 7f7c42a00640 1 --2- 172.21.15.12:0/3079293258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c3c109870 0x7f7c3c1a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.213+0000 7f7c42a00640 1 --2- 172.21.15.12:0/3079293258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c3c108eb0 0x7f7c3c19e4e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.213+0000 7f7c42a00640 1 --2- 172.21.15.12:0/3079293258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c3c107cc0 0x7f7c3c19dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.213+0000 7f7c42a00640 1 -- 172.21.15.12:0/3079293258 >> 172.21.15.12:0/3079293258 conn(0x7f7c3c076c60 msgr2=0x7f7c3c10c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.213+0000 7f7c42a00640 1 -- 172.21.15.12:0/3079293258 shutdown_connections 2024-03-23T23:02:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.213+0000 7f7c42a00640 1 -- 172.21.15.12:0/3079293258 wait complete. 2024-03-23T23:02:30.229 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:02:30.229 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-23T23:02:30.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.432+0000 7f01c4a00640 1 -- 172.21.15.12:0/2145563991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01c0107bc0 msgr2=0x7f01c0107fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:30.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.432+0000 7f01c4a00640 1 --2- 172.21.15.12:0/2145563991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01c0107bc0 0x7f01c0107fe0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f01b0009900 tx=0x7f01b002bc30 comp rx=0 tx=0).stop 2024-03-23T23:02:30.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.432+0000 7f01c4a00640 1 -- 172.21.15.12:0/2145563991 shutdown_connections 2024-03-23T23:02:30.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.432+0000 7f01c4a00640 1 --2- 172.21.15.12:0/2145563991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01c0109770 0x7f01c0110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.432+0000 7f01c4a00640 1 --2- 172.21.15.12:0/2145563991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01c0108db0 0x7f01c0109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.432+0000 7f01c4a00640 1 --2- 172.21.15.12:0/2145563991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01c0107bc0 0x7f01c0107fe0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.432+0000 7f01c4a00640 1 -- 172.21.15.12:0/2145563991 >> 172.21.15.12:0/2145563991 conn(0x7f01c0076b70 msgr2=0x7f01c0078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:30.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.432+0000 7f01c4a00640 1 -- 172.21.15.12:0/2145563991 shutdown_connections 2024-03-23T23:02:30.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.432+0000 7f01c4a00640 1 -- 172.21.15.12:0/2145563991 wait complete. 2024-03-23T23:02:30.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.433+0000 7f01c4a00640 1 Processor -- start 2024-03-23T23:02:30.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.433+0000 7f01c4a00640 1 -- start start 2024-03-23T23:02:30.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.433+0000 7f01c4a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01c0107bc0 0x7f01c019ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.434+0000 7f01c4a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01c0108db0 0x7f01c019e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.434+0000 7f01c4a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01c0109770 0x7f01c01a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.434+0000 7f01c4a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c0113860 con 0x7f01c0109770 2024-03-23T23:02:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.434+0000 7f01c4a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c01136e0 con 0x7f01c0108db0 2024-03-23T23:02:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.434+0000 7f01c4a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c01139e0 con 0x7f01c0107bc0 2024-03-23T23:02:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.434+0000 7f01bf400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01c0107bc0 0x7f01c019ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.434+0000 7f01bf400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01c0107bc0 0x7f01c019ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:40314/0 (socket says 172.21.15.12:40314) 2024-03-23T23:02:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.434+0000 7f01bf400640 1 -- 172.21.15.12:0/3002091588 learned_addr learned my addr 172.21.15.12:0/3002091588 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.434+0000 7f01bfe00640 1 --2- 172.21.15.12:0/3002091588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01c0109770 0x7f01c01a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.434+0000 7f01b7e00640 1 --2- 172.21.15.12:0/3002091588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01c0108db0 0x7f01c019e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.434+0000 7f01bf400640 1 -- 172.21.15.12:0/3002091588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01c0108db0 msgr2=0x7f01c019e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.434+0000 7f01bf400640 1 --2- 172.21.15.12:0/3002091588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01c0108db0 0x7f01c019e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.434+0000 7f01bf400640 1 -- 172.21.15.12:0/3002091588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01c0109770 msgr2=0x7f01c01a5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.434+0000 7f01bf400640 1 --2- 172.21.15.12:0/3002091588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01c0109770 0x7f01c01a5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.434+0000 7f01bf400640 1 -- 172.21.15.12:0/3002091588 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c01a5af0 con 0x7f01c0107bc0 2024-03-23T23:02:30.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.434+0000 7f01b7e00640 1 --2- 172.21.15.12:0/3002091588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01c0108db0 0x7f01c019e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:02:30.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.435+0000 7f01bf400640 1 --2- 172.21.15.12:0/3002091588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01c0107bc0 0x7f01c019ded0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f01b0000c00 tx=0x7f01b002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:30.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.435+0000 7f01bcc00640 1 -- 172.21.15.12:0/3002091588 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b0043020 con 0x7f01c0107bc0 2024-03-23T23:02:30.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.435+0000 7f01bcc00640 1 -- 172.21.15.12:0/3002091588 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f01b00054e0 con 0x7f01c0107bc0 2024-03-23T23:02:30.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.435+0000 7f01c4a00640 1 -- 172.21.15.12:0/3002091588 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01c01a5d80 con 0x7f01c0107bc0 2024-03-23T23:02:30.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.435+0000 7f01bcc00640 1 -- 172.21.15.12:0/3002091588 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b0003bd0 con 0x7f01c0107bc0 2024-03-23T23:02:30.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.435+0000 7f01c4a00640 1 -- 172.21.15.12:0/3002091588 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c01a6210 con 0x7f01c0107bc0 2024-03-23T23:02:30.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.438+0000 7f01bcc00640 1 -- 172.21.15.12:0/3002091588 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f01b0046020 con 0x7f01c0107bc0 2024-03-23T23:02:30.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.439+0000 7f01c8d94740 1 Processor -- start 2024-03-23T23:02:30.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.439+0000 7f01c8d94740 1 -- start start 2024-03-23T23:02:30.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.439+0000 7f01c8d94740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55578704f9f0 0x55578704fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.439+0000 7f01c8d94740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555787050bc0 0x555787051040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.439+0000 7f01c8d94740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555787051580 0x555787057e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.439+0000 7f01c8d94740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x555786fb42c0 con 0x555787050bc0 2024-03-23T23:02:30.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.439+0000 7f01c8d94740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x555786f5cc90 con 0x55578704f9f0 2024-03-23T23:02:30.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.439+0000 7f01c8d94740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x555786fb0360 con 0x555787051580 2024-03-23T23:02:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.439+0000 7f01b7e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555787051580 0x555787057e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.439+0000 7f01bfe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55578704f9f0 0x55578704fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.439+0000 7f01b7e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555787051580 0x555787057e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:40330/0 (socket says 172.21.15.12:40330) 2024-03-23T23:02:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.439+0000 7f01b7e00640 1 -- 172.21.15.12:0/4153933940 learned_addr learned my addr 172.21.15.12:0/4153933940 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.440+0000 7f01bf400640 1 --2- 172.21.15.12:0/4153933940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555787050bc0 0x555787051040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.440+0000 7f01b7e00640 1 -- 172.21.15.12:0/4153933940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55578704f9f0 msgr2=0x55578704fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.440+0000 7f01b7e00640 1 --2- 172.21.15.12:0/4153933940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55578704f9f0 0x55578704fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.440+0000 7f01b7e00640 1 -- 172.21.15.12:0/4153933940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555787050bc0 msgr2=0x555787051040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.440+0000 7f01b7e00640 1 --2- 172.21.15.12:0/4153933940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555787050bc0 0x555787051040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.440+0000 7f01b7e00640 1 -- 172.21.15.12:0/4153933940 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555786f81080 con 0x555787051580 2024-03-23T23:02:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.440+0000 7f01bcc00640 1 --2- 172.21.15.12:0/3002091588 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f01a40a2800 0x7f01a40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.440+0000 7f01b7e00640 1 --2- 172.21.15.12:0/4153933940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555787051580 0x555787057e10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f01a80114f0 tx=0x7f01a80119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5eef8f2607ad568 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.440+0000 7f01bcc00640 1 -- 172.21.15.12:0/3002091588 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f01b00f0aa0 con 0x7f01c0107bc0 2024-03-23T23:02:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.440+0000 7f01afe00640 1 -- 172.21.15.12:0/4153933940 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a801e070 con 0x555787051580 2024-03-23T23:02:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.440+0000 7f01afe00640 1 -- 172.21.15.12:0/4153933940 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f01a80035f0 con 0x555787051580 2024-03-23T23:02:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.440+0000 7f01afe00640 1 -- 172.21.15.12:0/4153933940 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a800f820 con 0x555787051580 2024-03-23T23:02:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.440+0000 7f01bfe00640 1 --2- 172.21.15.12:0/3002091588 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f01a40a2800 0x7f01a40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.441+0000 7f01c8d94740 1 -- 172.21.15.12:0/4153933940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555787051580 msgr2=0x555787057e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.441+0000 7f01c8d94740 1 --2- 172.21.15.12:0/4153933940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555787051580 0x555787057e10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f01a80114f0 tx=0x7f01a80119b0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.441+0000 7f01bfe00640 1 --2- 172.21.15.12:0/3002091588 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f01a40a2800 0x7f01a40a4cc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x555787050ac0 tx=0x7f01b8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.441+0000 7f01c8d94740 1 -- 172.21.15.12:0/4153933940 shutdown_connections 2024-03-23T23:02:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.441+0000 7f01c8d94740 1 --2- 172.21.15.12:0/4153933940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555787051580 0x555787057e10 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.441+0000 7f01c8d94740 1 --2- 172.21.15.12:0/4153933940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555787050bc0 0x555787051040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.441+0000 7f01c8d94740 1 --2- 172.21.15.12:0/4153933940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55578704f9f0 0x55578704fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.441+0000 7f01c8d94740 1 -- 172.21.15.12:0/4153933940 >> 172.21.15.12:0/4153933940 conn(0x55578704ade0 msgr2=0x55578704b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.441+0000 7f01c8d94740 1 -- 172.21.15.12:0/4153933940 shutdown_connections 2024-03-23T23:02:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.441+0000 7f01c8d94740 1 -- 172.21.15.12:0/4153933940 wait complete. 2024-03-23T23:02:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.442+0000 7f01c8d94740 1 -- start start 2024-03-23T23:02:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.442+0000 7f01c8d94740 1 Processor -- start 2024-03-23T23:02:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.443+0000 7f01c8d94740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55578704f9f0 0x555787059ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.443+0000 7f01c8d94740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555787050bc0 0x55578705a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.443+0000 7f01c8d94740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555787051580 0x555787061220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.443+0000 7f01c8d94740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x555787058590 con 0x555787051580 2024-03-23T23:02:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.443+0000 7f01c8d94740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x555786f700d0 con 0x555787050bc0 2024-03-23T23:02:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.443+0000 7f01c8d94740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x555787058710 con 0x55578704f9f0 2024-03-23T23:02:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.443+0000 7f01c8d94740 10 client.0 _ll_register_callbacks cb 0x555786f74e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:02:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.443+0000 7f01bf400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55578704f9f0 0x555787059ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.443+0000 7f01bf400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55578704f9f0 0x555787059ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:40338/0 (socket says 172.21.15.12:40338) 2024-03-23T23:02:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.443+0000 7f01b7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555787050bc0 0x55578705a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.443+0000 7f01bf400640 1 -- 172.21.15.12:0/44879550 learned_addr learned my addr 172.21.15.12:0/44879550 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.443+0000 7f01bfe00640 1 --2- 172.21.15.12:0/44879550 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555787051580 0x555787061220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.443+0000 7f01bf400640 1 -- 172.21.15.12:0/44879550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555787050bc0 msgr2=0x55578705a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.443+0000 7f01bf400640 1 --2- 172.21.15.12:0/44879550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555787050bc0 0x55578705a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.443+0000 7f01bf400640 1 -- 172.21.15.12:0/44879550 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555787051580 msgr2=0x555787061220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.443+0000 7f01bf400640 1 --2- 172.21.15.12:0/44879550 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555787051580 0x555787061220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.443+0000 7f01bf400640 1 -- 172.21.15.12:0/44879550 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555786cc83f0 con 0x55578704f9f0 2024-03-23T23:02:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.444+0000 7f01bf400640 1 --2- 172.21.15.12:0/44879550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55578704f9f0 0x555787059ca0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f01b0006010 tx=0x7f01b0009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=142f8004a43da528 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.444+0000 7f01acc00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:02:30.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.444+0000 7f01acc00640 1 -- 172.21.15.12:0/44879550 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b00bcf30 con 0x55578704f9f0 2024-03-23T23:02:30.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.444+0000 7f01acc00640 1 -- 172.21.15.12:0/44879550 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f01b00ee660 con 0x55578704f9f0 2024-03-23T23:02:30.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.444+0000 7f01c8d94740 1 -- 172.21.15.12:0/44879550 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f01a80035f0 con 0x55578704f9f0 2024-03-23T23:02:30.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.444+0000 7f01acc00640 1 -- 172.21.15.12:0/44879550 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f01b00bcd50 con 0x55578704f9f0 2024-03-23T23:02:30.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.444+0000 7f01c8d94740 10 client.25438 fetch_fsmap learned FSMap version 23 2024-03-23T23:02:30.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.444+0000 7f01c8d94740 1 -- 172.21.15.12:0/44879550 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x555786d927d0 con 0x55578704f9f0 2024-03-23T23:02:30.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.444+0000 7f01acc00640 1 -- 172.21.15.12:0/44879550 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f01a80035f0 con 0x55578704f9f0 2024-03-23T23:02:30.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.444+0000 7f01acc00640 1 -- 172.21.15.12:0/44879550 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f01b00c1110 con 0x55578704f9f0 2024-03-23T23:02:30.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.444+0000 7f01c8d94740 10 client.25438 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:02:30.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.444+0000 7f01c8d94740 10 client.25438 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:02:30.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.444+0000 7f01c8d94740 20 client.25438 populate_metadata read hostname 'smithi012' 2024-03-23T23:02:30.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.444+0000 7f01c8d94740 1 --2- 172.21.15.12:0/44879550 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5557870657b0 0x555787067c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.444+0000 7f01c8d94740 4 client.25438 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:02:30.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.444+0000 7f01c8d94740 1 -- 172.21.15.12:0/44879550 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555786fc1800 con 0x5557870657b0 2024-03-23T23:02:30.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.445+0000 7f01b7e00640 1 --2- 172.21.15.12:0/44879550 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5557870657b0 0x555787067c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.445+0000 7f01b7e00640 1 --2- 172.21.15.12:0/44879550 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5557870657b0 0x555787067c00 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b23c07e13d656fb3 server_cookie=399383cb2115f9af in_seq=0 out_seq=0 2024-03-23T23:02:30.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.445+0000 7f01acc00640 10 client.25438 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:30.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.446+0000 7f01acc00640 1 -- 172.21.15.12:0/44879550 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555786fc1800 con 0x5557870657b0 2024-03-23T23:02:30.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.446+0000 7f01acc00640 10 client.25438 handle_command_reply: tid=0 2024-03-23T23:02:30.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.446+0000 7f01c8d94740 1 client.25438 shutdown 2024-03-23T23:02:30.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.446+0000 7f01c8d94740 1 -- 172.21.15.12:0/44879550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55578704f9f0 msgr2=0x555787059ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:30.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.446+0000 7f01c8d94740 1 --2- 172.21.15.12:0/44879550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55578704f9f0 0x555787059ca0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f01b0006010 tx=0x7f01b0009560 comp rx=0 tx=0).stop 2024-03-23T23:02:30.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.446+0000 7f01c8d94740 1 -- 172.21.15.12:0/44879550 shutdown_connections 2024-03-23T23:02:30.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.446+0000 7f01c8d94740 1 --2- 172.21.15.12:0/44879550 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5557870657b0 0x555787067c00 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.446+0000 7f01c8d94740 1 --2- 172.21.15.12:0/44879550 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555787051580 0x555787061220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.446+0000 7f01c8d94740 1 --2- 172.21.15.12:0/44879550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555787050bc0 0x55578705a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.446+0000 7f01c8d94740 1 --2- 172.21.15.12:0/44879550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55578704f9f0 0x555787059ca0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.446+0000 7f01c8d94740 1 -- 172.21.15.12:0/44879550 >> 172.21.15.12:0/44879550 conn(0x55578704ade0 msgr2=0x55578704a160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:30.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.446+0000 7f01acc00640 0 client.25438 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:30.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.447+0000 7f01c8d94740 1 -- 172.21.15.12:0/44879550 shutdown_connections 2024-03-23T23:02:30.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.447+0000 7f01c8d94740 1 -- 172.21.15.12:0/44879550 wait complete. 2024-03-23T23:02:30.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.447+0000 7f01c8d94740 20 client.25438 trim_cache size 0 max 16384 2024-03-23T23:02:30.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.462+0000 7f01c8d94740 1 Processor -- start 2024-03-23T23:02:30.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.462+0000 7f01c8d94740 1 -- start start 2024-03-23T23:02:30.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.462+0000 7f01c8d94740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55578704f9f0 0x55578704e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.462+0000 7f01c8d94740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555787050bc0 0x5557870611e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.462+0000 7f01c8d94740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555787051580 0x555787053c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.462+0000 7f01c8d94740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c01b1240 con 0x555787051580 2024-03-23T23:02:30.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.462+0000 7f01c8d94740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c01aed40 con 0x55578704f9f0 2024-03-23T23:02:30.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.462+0000 7f01c8d94740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c01acde0 con 0x555787050bc0 2024-03-23T23:02:30.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.462+0000 7f01bf400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55578704f9f0 0x55578704e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.462+0000 7f01bf400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55578704f9f0 0x55578704e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57428/0 (socket says 172.21.15.12:57428) 2024-03-23T23:02:30.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.462+0000 7f01bf400640 1 -- 172.21.15.12:0/2275670158 learned_addr learned my addr 172.21.15.12:0/2275670158 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:30.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.462+0000 7f01b7e00640 1 --2- 172.21.15.12:0/2275670158 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555787050bc0 0x5557870611e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.463+0000 7f01bfe00640 1 --2- 172.21.15.12:0/2275670158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555787051580 0x555787053c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.463+0000 7f01bf400640 1 -- 172.21.15.12:0/2275670158 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555787050bc0 msgr2=0x5557870611e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:30.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.463+0000 7f01bf400640 1 --2- 172.21.15.12:0/2275670158 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555787050bc0 0x5557870611e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.463+0000 7f01bf400640 1 -- 172.21.15.12:0/2275670158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555787051580 msgr2=0x555787053c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:30.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.463+0000 7f01bf400640 1 --2- 172.21.15.12:0/2275670158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555787051580 0x555787053c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.463+0000 7f01bf400640 1 -- 172.21.15.12:0/2275670158 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555787050640 con 0x55578704f9f0 2024-03-23T23:02:30.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.463+0000 7f01bf400640 1 --2- 172.21.15.12:0/2275670158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55578704f9f0 0x55578704e750 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f01b00b9dc0 tx=0x7f01b002bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48a9bea0dd9eceac server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:30.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.463+0000 7f01af400640 1 -- 172.21.15.12:0/2275670158 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b00bcf30 con 0x55578704f9f0 2024-03-23T23:02:30.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.463+0000 7f01af400640 1 -- 172.21.15.12:0/2275670158 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f01b0005690 con 0x55578704f9f0 2024-03-23T23:02:30.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.463+0000 7f01af400640 1 -- 172.21.15.12:0/2275670158 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b00b9690 con 0x55578704f9f0 2024-03-23T23:02:30.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.464+0000 7f01c8d94740 1 -- 172.21.15.12:0/2275670158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55578704f9f0 msgr2=0x55578704e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:30.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.464+0000 7f01c8d94740 1 --2- 172.21.15.12:0/2275670158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55578704f9f0 0x55578704e750 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f01b00b9dc0 tx=0x7f01b002bc30 comp rx=0 tx=0).stop 2024-03-23T23:02:30.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.464+0000 7f01c8d94740 1 -- 172.21.15.12:0/2275670158 shutdown_connections 2024-03-23T23:02:30.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.464+0000 7f01c8d94740 1 --2- 172.21.15.12:0/2275670158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555787051580 0x555787053c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.464+0000 7f01c8d94740 1 --2- 172.21.15.12:0/2275670158 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555787050bc0 0x5557870611e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.464+0000 7f01c8d94740 1 --2- 172.21.15.12:0/2275670158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55578704f9f0 0x55578704e750 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.464+0000 7f01c8d94740 1 -- 172.21.15.12:0/2275670158 >> 172.21.15.12:0/2275670158 conn(0x55578704ade0 msgr2=0x55578704e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:30.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.464+0000 7f01c8d94740 1 -- 172.21.15.12:0/2275670158 shutdown_connections 2024-03-23T23:02:30.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.464+0000 7f01c8d94740 1 -- 172.21.15.12:0/2275670158 wait complete. 2024-03-23T23:02:30.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.465+0000 7f01c8d94740 1 -- start start 2024-03-23T23:02:30.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.465+0000 7f01c8d94740 1 Processor -- start 2024-03-23T23:02:30.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.465+0000 7f01c8d94740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55578704f9f0 0x55578705d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.465+0000 7f01c8d94740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555787050bc0 0x55578705d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.465+0000 7f01c8d94740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555787051580 0x555787060780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.465+0000 7f01c8d94740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c01ad3e0 con 0x555787050bc0 2024-03-23T23:02:30.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.465+0000 7f01c8d94740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c01ae6e0 con 0x555787051580 2024-03-23T23:02:30.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.465+0000 7f01c8d94740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c01ada20 con 0x55578704f9f0 2024-03-23T23:02:30.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.465+0000 7f01c8d94740 10 client.0 _ll_register_callbacks cb 0x7f01b0004250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:02:30.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.465+0000 7f01bf400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55578704f9f0 0x55578705d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.465+0000 7f01bf400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55578704f9f0 0x55578705d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:40358/0 (socket says 172.21.15.12:40358) 2024-03-23T23:02:30.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.465+0000 7f01bf400640 1 -- 172.21.15.12:0/3456598234 learned_addr learned my addr 172.21.15.12:0/3456598234 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:30.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.466+0000 7f01bfe00640 1 --2- 172.21.15.12:0/3456598234 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555787051580 0x555787060780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.466+0000 7f01b7e00640 1 --2- 172.21.15.12:0/3456598234 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555787050bc0 0x55578705d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.466+0000 7f01bf400640 1 -- 172.21.15.12:0/3456598234 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555787051580 msgr2=0x555787060780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:30.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.466+0000 7f01bf400640 1 --2- 172.21.15.12:0/3456598234 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555787051580 0x555787060780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.466+0000 7f01bf400640 1 -- 172.21.15.12:0/3456598234 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555787050bc0 msgr2=0x55578705d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:30.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.466+0000 7f01bf400640 1 --2- 172.21.15.12:0/3456598234 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555787050bc0 0x55578705d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.466+0000 7f01bf400640 1 -- 172.21.15.12:0/3456598234 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555787060d70 con 0x55578704f9f0 2024-03-23T23:02:30.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.466+0000 7f01bfe00640 1 --2- 172.21.15.12:0/3456598234 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555787051580 0x555787060780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:02:30.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.466+0000 7f01bf400640 1 --2- 172.21.15.12:0/3456598234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55578704f9f0 0x55578705d290 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f01b00bb6a0 tx=0x7f01b00bb900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecaec32bd867b3a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:30.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.466+0000 7f01acc00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:02:30.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.466+0000 7f01acc00640 1 -- 172.21.15.12:0/3456598234 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b00b9690 con 0x55578704f9f0 2024-03-23T23:02:30.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.466+0000 7f01acc00640 1 -- 172.21.15.12:0/3456598234 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f01b00c5900 con 0x55578704f9f0 2024-03-23T23:02:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.466+0000 7f01c8d94740 1 -- 172.21.15.12:0/3456598234 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555786fbbb50 con 0x55578704f9f0 2024-03-23T23:02:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.466+0000 7f01acc00640 1 -- 172.21.15.12:0/3456598234 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f01b00bcf30 con 0x55578704f9f0 2024-03-23T23:02:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.466+0000 7f01c8d94740 10 client.25444 fetch_fsmap learned FSMap version 23 2024-03-23T23:02:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.466+0000 7f01c8d94740 1 -- 172.21.15.12:0/3456598234 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5557870c1930 con 0x55578704f9f0 2024-03-23T23:02:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.467+0000 7f01acc00640 1 -- 172.21.15.12:0/3456598234 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x555786fbbb50 con 0x55578704f9f0 2024-03-23T23:02:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.467+0000 7f01acc00640 1 -- 172.21.15.12:0/3456598234 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f01b00bc8d0 con 0x55578704f9f0 2024-03-23T23:02:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.467+0000 7f01c8d94740 10 client.25444 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:02:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.467+0000 7f01c8d94740 10 client.25444 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:02:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.467+0000 7f01c8d94740 20 client.25444 populate_metadata read hostname 'smithi012' 2024-03-23T23:02:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.467+0000 7f01c8d94740 1 --2- 172.21.15.12:0/3456598234 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5557870c1fd0 0x5557870c43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.467+0000 7f01c8d94740 4 client.25444 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:02:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.467+0000 7f01c8d94740 1 -- 172.21.15.12:0/3456598234 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555786f39500 con 0x5557870c1fd0 2024-03-23T23:02:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.467+0000 7f01b7e00640 1 --2- 172.21.15.12:0/3456598234 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5557870c1fd0 0x5557870c43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.468+0000 7f01b7e00640 1 --2- 172.21.15.12:0/3456598234 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5557870c1fd0 0x5557870c43c0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e07477b47aeea98 server_cookie=bd48dfd3019c7b57 in_seq=0 out_seq=0 2024-03-23T23:02:30.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.468+0000 7f01acc00640 10 client.25444 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:30.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.468+0000 7f01acc00640 1 -- 172.21.15.12:0/3456598234 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555786f39500 con 0x5557870c1fd0 2024-03-23T23:02:30.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.468+0000 7f01acc00640 10 client.25444 handle_command_reply: tid=0 2024-03-23T23:02:30.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.468+0000 7f01c8d94740 1 client.25444 shutdown 2024-03-23T23:02:30.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.468+0000 7f01c8d94740 1 -- 172.21.15.12:0/3456598234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55578704f9f0 msgr2=0x55578705d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:30.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.468+0000 7f01c8d94740 1 --2- 172.21.15.12:0/3456598234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55578704f9f0 0x55578705d290 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f01b00bb6a0 tx=0x7f01b00bb900 comp rx=0 tx=0).stop 2024-03-23T23:02:30.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.468+0000 7f01c8d94740 1 -- 172.21.15.12:0/3456598234 shutdown_connections 2024-03-23T23:02:30.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.468+0000 7f01c8d94740 1 --2- 172.21.15.12:0/3456598234 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5557870c1fd0 0x5557870c43c0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.468+0000 7f01c8d94740 1 --2- 172.21.15.12:0/3456598234 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555787051580 0x555787060780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.468+0000 7f01c8d94740 1 --2- 172.21.15.12:0/3456598234 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555787050bc0 0x55578705d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.468+0000 7f01c8d94740 1 --2- 172.21.15.12:0/3456598234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55578704f9f0 0x55578705d290 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.468+0000 7f01acc00640 0 client.25444 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:30.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.468+0000 7f01c8d94740 1 -- 172.21.15.12:0/3456598234 >> 172.21.15.12:0/3456598234 conn(0x55578704ade0 msgr2=0x55578704e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:30.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.468+0000 7f01c8d94740 1 -- 172.21.15.12:0/3456598234 shutdown_connections 2024-03-23T23:02:30.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.468+0000 7f01c8d94740 1 -- 172.21.15.12:0/3456598234 wait complete. 2024-03-23T23:02:30.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.468+0000 7f01c8d94740 20 client.25444 trim_cache size 0 max 16384 2024-03-23T23:02:30.473 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:02:30.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.469+0000 7f01c4a00640 1 -- 172.21.15.12:0/3002091588 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f01a40a2800 msgr2=0x7f01a40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:30.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.469+0000 7f01c4a00640 1 --2- 172.21.15.12:0/3002091588 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f01a40a2800 0x7f01a40a4cc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x555787050ac0 tx=0x7f01b8009500 comp rx=0 tx=0).stop 2024-03-23T23:02:30.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.469+0000 7f01c4a00640 1 -- 172.21.15.12:0/3002091588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01c0107bc0 msgr2=0x7f01c019ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:30.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.469+0000 7f01c4a00640 1 --2- 172.21.15.12:0/3002091588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01c0107bc0 0x7f01c019ded0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f01b0000c00 tx=0x7f01b002c650 comp rx=0 tx=0).stop 2024-03-23T23:02:30.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.469+0000 7f01c4a00640 1 -- 172.21.15.12:0/3002091588 shutdown_connections 2024-03-23T23:02:30.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.469+0000 7f01c4a00640 1 --2- 172.21.15.12:0/3002091588 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f01a40a2800 0x7f01a40a4cc0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.469+0000 7f01c4a00640 1 --2- 172.21.15.12:0/3002091588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01c0109770 0x7f01c01a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.469+0000 7f01c4a00640 1 --2- 172.21.15.12:0/3002091588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01c0108db0 0x7f01c019e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.469+0000 7f01c4a00640 1 --2- 172.21.15.12:0/3002091588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01c0107bc0 0x7f01c019ded0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:30.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.469+0000 7f01c4a00640 1 -- 172.21.15.12:0/3002091588 >> 172.21.15.12:0/3002091588 conn(0x7f01c0076b70 msgr2=0x7f01c010c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:30.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.469+0000 7f01c4a00640 1 -- 172.21.15.12:0/3002091588 shutdown_connections 2024-03-23T23:02:30.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:30.469+0000 7f01c4a00640 1 -- 172.21.15.12:0/3002091588 wait complete. 2024-03-23T23:02:30.483 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:30 smithi012 ceph-mon[31695]: pgmap v411: 97 pgs: 97 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-23T23:02:30.483 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:30 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1213836952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:02:30.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:30 smithi173 ceph-mon[36116]: pgmap v411: 97 pgs: 97 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-23T23:02:30.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:30 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1213836952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:02:30.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:30 smithi203 ceph-mon[36693]: pgmap v411: 97 pgs: 97 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-23T23:02:30.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:30 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1213836952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:02:31.487 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-23T23:02:31.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:02:31.487 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-23T23:02:31.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.719+0000 7f43f5e00640 1 -- 172.21.15.12:0/1339346218 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43f0109770 msgr2=0x7f43f0110000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:31.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.719+0000 7f43f5e00640 1 --2- 172.21.15.12:0/1339346218 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43f0109770 0x7f43f0110000 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f43e8009860 tx=0x7f43e802bb70 comp rx=0 tx=0).stop 2024-03-23T23:02:31.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.719+0000 7f43f5e00640 1 -- 172.21.15.12:0/1339346218 shutdown_connections 2024-03-23T23:02:31.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.719+0000 7f43f5e00640 1 --2- 172.21.15.12:0/1339346218 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43f0109770 0x7f43f0110000 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.719+0000 7f43f5e00640 1 --2- 172.21.15.12:0/1339346218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43f0108db0 0x7f43f0109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.719+0000 7f43f5e00640 1 --2- 172.21.15.12:0/1339346218 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43f0107bc0 0x7f43f0107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.719+0000 7f43f5e00640 1 -- 172.21.15.12:0/1339346218 >> 172.21.15.12:0/1339346218 conn(0x7f43f0076b70 msgr2=0x7f43f0078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:31.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.719+0000 7f43f5e00640 1 -- 172.21.15.12:0/1339346218 shutdown_connections 2024-03-23T23:02:31.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.719+0000 7f43f5e00640 1 -- 172.21.15.12:0/1339346218 wait complete. 2024-03-23T23:02:31.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.720+0000 7f43f5e00640 1 Processor -- start 2024-03-23T23:02:31.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.720+0000 7f43f5e00640 1 -- start start 2024-03-23T23:02:31.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.720+0000 7f43f5e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43f0107bc0 0x7f43f019de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:31.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.721+0000 7f43f5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43f0108db0 0x7f43f019e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:31.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.721+0000 7f43f5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43f0109770 0x7f43f01a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:31.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.721+0000 7f43f5e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f0113840 con 0x7f43f0107bc0 2024-03-23T23:02:31.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.721+0000 7f43f5e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f01136c0 con 0x7f43f0108db0 2024-03-23T23:02:31.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.721+0000 7f43f5e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f01139c0 con 0x7f43f0109770 2024-03-23T23:02:31.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.721+0000 7f43f5400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43f0109770 0x7f43f01a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:31.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.721+0000 7f43f4a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43f0107bc0 0x7f43f019de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:31.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.721+0000 7f43f5400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43f0109770 0x7f43f01a5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:40382/0 (socket says 172.21.15.12:40382) 2024-03-23T23:02:31.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.721+0000 7f43efe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43f0108db0 0x7f43f019e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:31.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.721+0000 7f43f5400640 1 -- 172.21.15.12:0/375003438 learned_addr learned my addr 172.21.15.12:0/375003438 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:31.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.721+0000 7f43f5400640 1 -- 172.21.15.12:0/375003438 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43f0108db0 msgr2=0x7f43f019e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:31.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.721+0000 7f43f5400640 1 --2- 172.21.15.12:0/375003438 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43f0108db0 0x7f43f019e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.721+0000 7f43f5400640 1 -- 172.21.15.12:0/375003438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43f0107bc0 msgr2=0x7f43f019de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:31.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.721+0000 7f43f5400640 1 --2- 172.21.15.12:0/375003438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43f0107bc0 0x7f43f019de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.721+0000 7f43f5400640 1 -- 172.21.15.12:0/375003438 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f01a5a70 con 0x7f43f0109770 2024-03-23T23:02:31.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.722+0000 7f43f5400640 1 --2- 172.21.15.12:0/375003438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43f0109770 0x7f43f01a5400 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f43e802c180 tx=0x7f43e8032460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:31.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.722+0000 7f43ed600640 1 -- 172.21.15.12:0/375003438 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e8041020 con 0x7f43f0109770 2024-03-23T23:02:31.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.722+0000 7f43ed600640 1 -- 172.21.15.12:0/375003438 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43e802c9f0 con 0x7f43f0109770 2024-03-23T23:02:31.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.722+0000 7f43f5e00640 1 -- 172.21.15.12:0/375003438 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f01a5d00 con 0x7f43f0109770 2024-03-23T23:02:31.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.722+0000 7f43ed600640 1 -- 172.21.15.12:0/375003438 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e8039040 con 0x7f43f0109770 2024-03-23T23:02:31.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.722+0000 7f43f5e00640 1 -- 172.21.15.12:0/375003438 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f01a61e0 con 0x7f43f0109770 2024-03-23T23:02:31.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.726+0000 7f43ed600640 1 -- 172.21.15.12:0/375003438 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f43e8044020 con 0x7f43f0109770 2024-03-23T23:02:31.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.726+0000 7f43fa19f740 1 Processor -- start 2024-03-23T23:02:31.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.727+0000 7f43fa19f740 1 -- start start 2024-03-23T23:02:31.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.727+0000 7f43fa19f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9d81422b0 0x55f9d81426d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:31.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.727+0000 7f43fa19f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8142ca0 0x55f9d81477a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:31.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.727+0000 7f43fa19f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9d8147e10 0x55f9d814a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:31.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.727+0000 7f43fa19f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9d80a7eb0 con 0x55f9d8147e10 2024-03-23T23:02:31.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.727+0000 7f43fa19f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9d8050a60 con 0x55f9d8142ca0 2024-03-23T23:02:31.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.727+0000 7f43fa19f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9d80a3f50 con 0x55f9d81422b0 2024-03-23T23:02:31.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.727+0000 7f43f4a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8142ca0 0x55f9d81477a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:31.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.727+0000 7f43efe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9d81422b0 0x55f9d81426d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:31.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.727+0000 7f43f4a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8142ca0 0x55f9d81477a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57466/0 (socket says 172.21.15.12:57466) 2024-03-23T23:02:31.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.727+0000 7f43f4a00640 1 -- 172.21.15.12:0/2178822910 learned_addr learned my addr 172.21.15.12:0/2178822910 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:31.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.727+0000 7f43efe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9d81422b0 0x55f9d81426d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:40396/0 (socket says 172.21.15.12:40396) 2024-03-23T23:02:31.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.727+0000 7f43efe00640 1 --2- 172.21.15.12:0/2178822910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9d8147e10 0x55f9d814a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:31.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.728+0000 7f43f4a00640 1 -- 172.21.15.12:0/2178822910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9d81422b0 msgr2=0x55f9d81426d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:31.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.728+0000 7f43f4a00640 1 --2- 172.21.15.12:0/2178822910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9d81422b0 0x55f9d81426d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.728+0000 7f43f4a00640 1 -- 172.21.15.12:0/2178822910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9d8147e10 msgr2=0x55f9d814a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:31.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.728+0000 7f43f4a00640 1 --2- 172.21.15.12:0/2178822910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9d8147e10 0x55f9d814a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.728+0000 7f43f4a00640 1 -- 172.21.15.12:0/2178822910 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9d7e847e0 con 0x55f9d8142ca0 2024-03-23T23:02:31.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.728+0000 7f43efe00640 1 --2- 172.21.15.12:0/2178822910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9d8147e10 0x55f9d814a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:02:31.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.728+0000 7f43f4a00640 1 --2- 172.21.15.12:0/2178822910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8142ca0 0x55f9d81477a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f43e0011470 tx=0x7f43e0011930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=934ad04a5c38b7aa server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:31.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.728+0000 7f43ed600640 1 --2- 172.21.15.12:0/375003438 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f43d40a2880 0x7f43d40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:31.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.728+0000 7f43e5c00640 1 -- 172.21.15.12:0/2178822910 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e0003970 con 0x55f9d8142ca0 2024-03-23T23:02:31.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.728+0000 7f43ed600640 1 -- 172.21.15.12:0/375003438 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f43e80f0350 con 0x7f43f0109770 2024-03-23T23:02:31.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.728+0000 7f43e5c00640 1 -- 172.21.15.12:0/2178822910 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43e0004020 con 0x55f9d8142ca0 2024-03-23T23:02:31.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.728+0000 7f43e5c00640 1 -- 172.21.15.12:0/2178822910 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e000fa00 con 0x55f9d8142ca0 2024-03-23T23:02:31.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.728+0000 7f43efe00640 1 --2- 172.21.15.12:0/375003438 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f43d40a2880 0x7f43d40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:31.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.729+0000 7f43fa19f740 1 -- 172.21.15.12:0/2178822910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8142ca0 msgr2=0x55f9d81477a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:31.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.729+0000 7f43fa19f740 1 --2- 172.21.15.12:0/2178822910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8142ca0 0x55f9d81477a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f43e0011470 tx=0x7f43e0011930 comp rx=0 tx=0).stop 2024-03-23T23:02:31.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.729+0000 7f43efe00640 1 --2- 172.21.15.12:0/375003438 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f43d40a2880 0x7f43d40a4d40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x55f9d8143500 tx=0x7f43dc013040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:31.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.729+0000 7f43fa19f740 1 -- 172.21.15.12:0/2178822910 shutdown_connections 2024-03-23T23:02:31.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.729+0000 7f43fa19f740 1 --2- 172.21.15.12:0/2178822910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9d8147e10 0x55f9d814a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.729+0000 7f43fa19f740 1 --2- 172.21.15.12:0/2178822910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8142ca0 0x55f9d81477a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.729+0000 7f43fa19f740 1 --2- 172.21.15.12:0/2178822910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9d81422b0 0x55f9d81426d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.729+0000 7f43fa19f740 1 -- 172.21.15.12:0/2178822910 >> 172.21.15.12:0/2178822910 conn(0x55f9d813d820 msgr2=0x55f9d813dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:31.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.729+0000 7f43fa19f740 1 -- 172.21.15.12:0/2178822910 shutdown_connections 2024-03-23T23:02:31.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.729+0000 7f43fa19f740 1 -- 172.21.15.12:0/2178822910 wait complete. 2024-03-23T23:02:31.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.730+0000 7f43fa19f740 1 -- start start 2024-03-23T23:02:31.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.730+0000 7f43fa19f740 1 Processor -- start 2024-03-23T23:02:31.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.730+0000 7f43fa19f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9d81422b0 0x55f9d814c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:31.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.730+0000 7f43fa19f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9d8142ca0 0x55f9d814ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:31.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.730+0000 7f43fa19f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8147e10 0x55f9d8153a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:31.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.730+0000 7f43fa19f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9d814adc0 con 0x55f9d81422b0 2024-03-23T23:02:31.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.730+0000 7f43fa19f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9d8063eb0 con 0x55f9d8147e10 2024-03-23T23:02:31.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.730+0000 7f43fa19f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9d814af40 con 0x55f9d8142ca0 2024-03-23T23:02:31.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.731+0000 7f43fa19f740 10 client.0 _ll_register_callbacks cb 0x55f9d8058150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:02:31.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.731+0000 7f43efe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9d8142ca0 0x55f9d814ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:31.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.731+0000 7f43efe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9d8142ca0 0x55f9d814ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:40402/0 (socket says 172.21.15.12:40402) 2024-03-23T23:02:31.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.731+0000 7f43f5400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8147e10 0x55f9d8153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:31.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.731+0000 7f43efe00640 1 -- 172.21.15.12:0/3392025270 learned_addr learned my addr 172.21.15.12:0/3392025270 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:31.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.731+0000 7f43f4a00640 1 --2- 172.21.15.12:0/3392025270 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9d81422b0 0x55f9d814c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:31.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.731+0000 7f43f5400640 1 -- 172.21.15.12:0/3392025270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9d8142ca0 msgr2=0x55f9d814ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:31.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.731+0000 7f43f5400640 1 --2- 172.21.15.12:0/3392025270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9d8142ca0 0x55f9d814ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.731+0000 7f43f5400640 1 -- 172.21.15.12:0/3392025270 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9d81422b0 msgr2=0x55f9d814c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:31.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.731+0000 7f43f5400640 1 --2- 172.21.15.12:0/3392025270 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9d81422b0 0x55f9d814c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.731+0000 7f43f5400640 1 -- 172.21.15.12:0/3392025270 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f9d7ee4f70 con 0x55f9d8147e10 2024-03-23T23:02:31.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.731+0000 7f43efe00640 1 --2- 172.21.15.12:0/3392025270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9d8142ca0 0x55f9d814ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:02:31.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.731+0000 7f43f5400640 1 --2- 172.21.15.12:0/3392025270 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8147e10 0x55f9d8153a40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f43e80edeb0 tx=0x7f43e80bc900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa03da099db29ed4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:31.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.731+0000 7f43da000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:02:31.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.732+0000 7f43da000640 1 -- 172.21.15.12:0/3392025270 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e80bcd00 con 0x55f9d8147e10 2024-03-23T23:02:31.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.732+0000 7f43da000640 1 -- 172.21.15.12:0/3392025270 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f43e8032e30 con 0x55f9d8147e10 2024-03-23T23:02:31.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.732+0000 7f43fa19f740 1 -- 172.21.15.12:0/3392025270 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f43e0004020 con 0x55f9d8147e10 2024-03-23T23:02:31.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.732+0000 7f43da000640 1 -- 172.21.15.12:0/3392025270 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43e80c0e50 con 0x55f9d8147e10 2024-03-23T23:02:31.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.732+0000 7f43fa19f740 10 client.25478 fetch_fsmap learned FSMap version 23 2024-03-23T23:02:31.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.732+0000 7f43fa19f740 1 -- 172.21.15.12:0/3392025270 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f9d7d7c170 con 0x55f9d8147e10 2024-03-23T23:02:31.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.732+0000 7f43da000640 1 -- 172.21.15.12:0/3392025270 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43e0004020 con 0x55f9d8147e10 2024-03-23T23:02:31.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.732+0000 7f43da000640 1 -- 172.21.15.12:0/3392025270 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f43e803c030 con 0x55f9d8147e10 2024-03-23T23:02:31.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.732+0000 7f43fa19f740 10 client.25478 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:02:31.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.732+0000 7f43fa19f740 10 client.25478 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:02:31.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.732+0000 7f43fa19f740 20 client.25478 populate_metadata read hostname 'smithi012' 2024-03-23T23:02:31.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.732+0000 7f43fa19f740 1 --2- 172.21.15.12:0/3392025270 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f9d8158310 0x55f9d815a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:31.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.732+0000 7f43fa19f740 4 client.25478 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:02:31.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.732+0000 7f43fa19f740 1 -- 172.21.15.12:0/3392025270 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f9d80b53f0 con 0x55f9d8158310 2024-03-23T23:02:31.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.733+0000 7f43f4a00640 1 --2- 172.21.15.12:0/3392025270 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f9d8158310 0x55f9d815a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:31.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.733+0000 7f43f4a00640 1 --2- 172.21.15.12:0/3392025270 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f9d8158310 0x55f9d815a770 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1e969be1e9832ba server_cookie=9ae176e725073fd5 in_seq=0 out_seq=0 2024-03-23T23:02:31.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.733+0000 7f43da000640 10 client.25478 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:31.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.734+0000 7f43da000640 1 -- 172.21.15.12:0/3392025270 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f9d80b53f0 con 0x55f9d8158310 2024-03-23T23:02:31.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.734+0000 7f43da000640 10 client.25478 handle_command_reply: tid=0 2024-03-23T23:02:31.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.734+0000 7f43fa19f740 1 client.25478 shutdown 2024-03-23T23:02:31.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.734+0000 7f43fa19f740 1 -- 172.21.15.12:0/3392025270 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8147e10 msgr2=0x55f9d8153a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:31.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.734+0000 7f43fa19f740 1 --2- 172.21.15.12:0/3392025270 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8147e10 0x55f9d8153a40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f43e80edeb0 tx=0x7f43e80bc900 comp rx=0 tx=0).stop 2024-03-23T23:02:31.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.734+0000 7f43fa19f740 1 -- 172.21.15.12:0/3392025270 shutdown_connections 2024-03-23T23:02:31.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.734+0000 7f43fa19f740 1 --2- 172.21.15.12:0/3392025270 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f9d8158310 0x55f9d815a770 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.734+0000 7f43fa19f740 1 --2- 172.21.15.12:0/3392025270 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8147e10 0x55f9d8153a40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.734+0000 7f43fa19f740 1 --2- 172.21.15.12:0/3392025270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9d8142ca0 0x55f9d814ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.734+0000 7f43fa19f740 1 --2- 172.21.15.12:0/3392025270 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9d81422b0 0x55f9d814c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.734+0000 7f43fa19f740 1 -- 172.21.15.12:0/3392025270 >> 172.21.15.12:0/3392025270 conn(0x55f9d813d820 msgr2=0x55f9d813ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:31.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.734+0000 7f43da000640 0 client.25478 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:31.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.734+0000 7f43fa19f740 1 -- 172.21.15.12:0/3392025270 shutdown_connections 2024-03-23T23:02:31.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.734+0000 7f43fa19f740 1 -- 172.21.15.12:0/3392025270 wait complete. 2024-03-23T23:02:31.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.734+0000 7f43fa19f740 20 client.25478 trim_cache size 0 max 16384 2024-03-23T23:02:31.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.752+0000 7f43fa19f740 1 Processor -- start 2024-03-23T23:02:31.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.752+0000 7f43fa19f740 1 -- start start 2024-03-23T23:02:31.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.752+0000 7f43fa19f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9d81422b0 0x55f9d814c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:31.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.753+0000 7f43fa19f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8142ca0 0x55f9d8153a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:31.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.753+0000 7f43fa19f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9d8147e10 0x55f9d813c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:31.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.753+0000 7f43fa19f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f01b1210 con 0x55f9d81422b0 2024-03-23T23:02:31.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.753+0000 7f43fa19f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f01aed10 con 0x55f9d8142ca0 2024-03-23T23:02:31.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.753+0000 7f43fa19f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f01acdb0 con 0x55f9d8147e10 2024-03-23T23:02:31.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.753+0000 7f43efe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8142ca0 0x55f9d8153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:31.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.753+0000 7f43efe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8142ca0 0x55f9d8153a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57488/0 (socket says 172.21.15.12:57488) 2024-03-23T23:02:31.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.753+0000 7f43efe00640 1 -- 172.21.15.12:0/3156216409 learned_addr learned my addr 172.21.15.12:0/3156216409 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:31.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.753+0000 7f43f5400640 1 --2- 172.21.15.12:0/3156216409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9d8147e10 0x55f9d813c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:31.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.753+0000 7f43f4a00640 1 --2- 172.21.15.12:0/3156216409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9d81422b0 0x55f9d814c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:31.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.753+0000 7f43efe00640 1 -- 172.21.15.12:0/3156216409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9d8147e10 msgr2=0x55f9d813c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:31.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.753+0000 7f43efe00640 1 --2- 172.21.15.12:0/3156216409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9d8147e10 0x55f9d813c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.753+0000 7f43efe00640 1 -- 172.21.15.12:0/3156216409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9d81422b0 msgr2=0x55f9d814c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:31.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.753+0000 7f43efe00640 1 --2- 172.21.15.12:0/3156216409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9d81422b0 0x55f9d814c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.753+0000 7f43efe00640 1 -- 172.21.15.12:0/3156216409 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9d813e290 con 0x55f9d8142ca0 2024-03-23T23:02:31.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.753+0000 7f43f4a00640 1 --2- 172.21.15.12:0/3156216409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9d81422b0 0x55f9d814c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:02:31.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.754+0000 7f43efe00640 1 --2- 172.21.15.12:0/3156216409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8142ca0 0x55f9d8153a00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f43dc0035f0 tx=0x7f43dc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=feb0b18109ec6b52 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:31.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.754+0000 7f43e5200640 1 -- 172.21.15.12:0/3156216409 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc005020 con 0x55f9d8142ca0 2024-03-23T23:02:31.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.754+0000 7f43e5200640 1 -- 172.21.15.12:0/3156216409 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43dc0047a0 con 0x55f9d8142ca0 2024-03-23T23:02:31.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.754+0000 7f43e5200640 1 -- 172.21.15.12:0/3156216409 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc00f9b0 con 0x55f9d8142ca0 2024-03-23T23:02:31.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.754+0000 7f43fa19f740 1 -- 172.21.15.12:0/3156216409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8142ca0 msgr2=0x55f9d8153a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:31.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.754+0000 7f43fa19f740 1 --2- 172.21.15.12:0/3156216409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8142ca0 0x55f9d8153a00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f43dc0035f0 tx=0x7f43dc0046c0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.754+0000 7f43fa19f740 1 -- 172.21.15.12:0/3156216409 shutdown_connections 2024-03-23T23:02:31.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.754+0000 7f43fa19f740 1 --2- 172.21.15.12:0/3156216409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9d8147e10 0x55f9d813c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.754+0000 7f43fa19f740 1 --2- 172.21.15.12:0/3156216409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8142ca0 0x55f9d8153a00 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.754+0000 7f43fa19f740 1 --2- 172.21.15.12:0/3156216409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9d81422b0 0x55f9d814c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.754+0000 7f43fa19f740 1 -- 172.21.15.12:0/3156216409 >> 172.21.15.12:0/3156216409 conn(0x55f9d813d820 msgr2=0x55f9d813c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:31.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.755+0000 7f43fa19f740 1 -- 172.21.15.12:0/3156216409 shutdown_connections 2024-03-23T23:02:31.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.755+0000 7f43fa19f740 1 -- 172.21.15.12:0/3156216409 wait complete. 2024-03-23T23:02:31.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.755+0000 7f43fa19f740 1 -- start start 2024-03-23T23:02:31.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.755+0000 7f43fa19f740 1 Processor -- start 2024-03-23T23:02:31.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.755+0000 7f43fa19f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9d81422b0 0x55f9d81519a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:31.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.755+0000 7f43fa19f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8142ca0 0x55f9d8151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:31.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.756+0000 7f43fa19f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9d8147e10 0x55f9d8152ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:31.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.756+0000 7f43fa19f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f01ad3b0 con 0x55f9d8147e10 2024-03-23T23:02:31.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.756+0000 7f43fa19f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f01ae6b0 con 0x55f9d8142ca0 2024-03-23T23:02:31.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.756+0000 7f43fa19f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f01ad9f0 con 0x55f9d81422b0 2024-03-23T23:02:31.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.756+0000 7f43fa19f740 10 client.0 _ll_register_callbacks cb 0x7f43e80bcad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:02:31.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.756+0000 7f43efe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8142ca0 0x55f9d8151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:31.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.756+0000 7f43f4a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9d81422b0 0x55f9d81519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:31.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.756+0000 7f43efe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8142ca0 0x55f9d8151ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57494/0 (socket says 172.21.15.12:57494) 2024-03-23T23:02:31.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.756+0000 7f43efe00640 1 -- 172.21.15.12:0/3090542226 learned_addr learned my addr 172.21.15.12:0/3090542226 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:31.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.756+0000 7f43f5400640 1 --2- 172.21.15.12:0/3090542226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9d8147e10 0x55f9d8152ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:31.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.756+0000 7f43efe00640 1 -- 172.21.15.12:0/3090542226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9d81422b0 msgr2=0x55f9d81519a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:31.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.756+0000 7f43efe00640 1 --2- 172.21.15.12:0/3090542226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9d81422b0 0x55f9d81519a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.756+0000 7f43efe00640 1 -- 172.21.15.12:0/3090542226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9d8147e10 msgr2=0x55f9d8152ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:31.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.756+0000 7f43efe00640 1 --2- 172.21.15.12:0/3090542226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9d8147e10 0x55f9d8152ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.756+0000 7f43efe00640 1 -- 172.21.15.12:0/3090542226 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f9d81534e0 con 0x55f9d8142ca0 2024-03-23T23:02:31.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.756+0000 7f43efe00640 1 --2- 172.21.15.12:0/3090542226 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8142ca0 0x55f9d8151ee0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f43dc006010 tx=0x7f43dc004c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1625f1e851c23f14 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:31.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.756+0000 7f43da000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:02:31.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.757+0000 7f43da000640 1 -- 172.21.15.12:0/3090542226 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc038070 con 0x55f9d8142ca0 2024-03-23T23:02:31.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.757+0000 7f43fa19f740 1 -- 172.21.15.12:0/3090542226 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f43dc0047a0 con 0x55f9d8142ca0 2024-03-23T23:02:31.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.757+0000 7f43da000640 1 -- 172.21.15.12:0/3090542226 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f43dc0051b0 con 0x55f9d8142ca0 2024-03-23T23:02:31.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.757+0000 7f43da000640 1 -- 172.21.15.12:0/3090542226 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43dc0053b0 con 0x55f9d8142ca0 2024-03-23T23:02:31.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.757+0000 7f43fa19f740 10 client.25484 fetch_fsmap learned FSMap version 23 2024-03-23T23:02:31.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.757+0000 7f43fa19f740 1 -- 172.21.15.12:0/3090542226 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f9d81b4510 con 0x55f9d8142ca0 2024-03-23T23:02:31.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.757+0000 7f43da000640 1 -- 172.21.15.12:0/3090542226 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43dc0047a0 con 0x55f9d8142ca0 2024-03-23T23:02:31.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.757+0000 7f43da000640 1 -- 172.21.15.12:0/3090542226 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f43dc0204d0 con 0x55f9d8142ca0 2024-03-23T23:02:31.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.757+0000 7f43fa19f740 10 client.25484 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:02:31.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.757+0000 7f43fa19f740 10 client.25484 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:02:31.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.757+0000 7f43fa19f740 20 client.25484 populate_metadata read hostname 'smithi012' 2024-03-23T23:02:31.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.757+0000 7f43fa19f740 1 --2- 172.21.15.12:0/3090542226 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f9d81b4e90 0x55f9d81b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:31.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.758+0000 7f43fa19f740 4 client.25484 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-23T23:02:31.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.758+0000 7f43fa19f740 1 -- 172.21.15.12:0/3090542226 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f9d80af740 con 0x55f9d81b4e90 2024-03-23T23:02:31.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.758+0000 7f43f4a00640 1 --2- 172.21.15.12:0/3090542226 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f9d81b4e90 0x55f9d81b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:31.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.758+0000 7f43f4a00640 1 --2- 172.21.15.12:0/3090542226 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f9d81b4e90 0x55f9d81b72b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f0efff9c754a66e server_cookie=5c0809deb7615c76 in_seq=0 out_seq=0 2024-03-23T23:02:31.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.758+0000 7f43da000640 10 client.25484 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:31.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.759+0000 7f43da000640 1 -- 172.21.15.12:0/3090542226 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f9d80af740 con 0x55f9d81b4e90 2024-03-23T23:02:31.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.759+0000 7f43da000640 10 client.25484 handle_command_reply: tid=0 2024-03-23T23:02:31.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.759+0000 7f43fa19f740 1 client.25484 shutdown 2024-03-23T23:02:31.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.759+0000 7f43fa19f740 1 -- 172.21.15.12:0/3090542226 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8142ca0 msgr2=0x55f9d8151ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:31.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.759+0000 7f43fa19f740 1 --2- 172.21.15.12:0/3090542226 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8142ca0 0x55f9d8151ee0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f43dc006010 tx=0x7f43dc004c00 comp rx=0 tx=0).stop 2024-03-23T23:02:31.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.760+0000 7f43fa19f740 1 -- 172.21.15.12:0/3090542226 shutdown_connections 2024-03-23T23:02:31.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.760+0000 7f43fa19f740 1 --2- 172.21.15.12:0/3090542226 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f9d81b4e90 0x55f9d81b72b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.760+0000 7f43fa19f740 1 --2- 172.21.15.12:0/3090542226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9d8147e10 0x55f9d8152ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.760+0000 7f43fa19f740 1 --2- 172.21.15.12:0/3090542226 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9d8142ca0 0x55f9d8151ee0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.760+0000 7f43fa19f740 1 --2- 172.21.15.12:0/3090542226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9d81422b0 0x55f9d81519a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.760+0000 7f43fa19f740 1 -- 172.21.15.12:0/3090542226 >> 172.21.15.12:0/3090542226 conn(0x55f9d813d820 msgr2=0x55f9d813ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:31.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.760+0000 7f43da000640 0 client.25484 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:31.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.760+0000 7f43fa19f740 1 -- 172.21.15.12:0/3090542226 shutdown_connections 2024-03-23T23:02:31.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.760+0000 7f43fa19f740 1 -- 172.21.15.12:0/3090542226 wait complete. 2024-03-23T23:02:31.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.760+0000 7f43fa19f740 20 client.25484 trim_cache size 0 max 16384 2024-03-23T23:02:31.764 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:02:31.764 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-03-23T23:02:31.764 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "1f69d79e-356d-488e-a311-fe221b87084a", 2024-03-23T23:02:31.764 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-03-23T23:02:31.764 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:02:31.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.760+0000 7f43f5e00640 1 -- 172.21.15.12:0/375003438 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f43d40a2880 msgr2=0x7f43d40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:31.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.760+0000 7f43f5e00640 1 --2- 172.21.15.12:0/375003438 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f43d40a2880 0x7f43d40a4d40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x55f9d8143500 tx=0x7f43dc013040 comp rx=0 tx=0).stop 2024-03-23T23:02:31.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.760+0000 7f43f5e00640 1 -- 172.21.15.12:0/375003438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43f0109770 msgr2=0x7f43f01a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:31.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.760+0000 7f43f5e00640 1 --2- 172.21.15.12:0/375003438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43f0109770 0x7f43f01a5400 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f43e802c180 tx=0x7f43e8032460 comp rx=0 tx=0).stop 2024-03-23T23:02:31.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.761+0000 7f43f5e00640 1 -- 172.21.15.12:0/375003438 shutdown_connections 2024-03-23T23:02:31.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.761+0000 7f43f5e00640 1 --2- 172.21.15.12:0/375003438 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f43d40a2880 0x7f43d40a4d40 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.761+0000 7f43f5e00640 1 --2- 172.21.15.12:0/375003438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43f0109770 0x7f43f01a5400 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.761+0000 7f43f5e00640 1 --2- 172.21.15.12:0/375003438 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43f0108db0 0x7f43f019e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.761+0000 7f43f5e00640 1 --2- 172.21.15.12:0/375003438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43f0107bc0 0x7f43f019de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.761+0000 7f43f5e00640 1 -- 172.21.15.12:0/375003438 >> 172.21.15.12:0/375003438 conn(0x7f43f0076b70 msgr2=0x7f43f010bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:31.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.761+0000 7f43f5e00640 1 -- 172.21.15.12:0/375003438 shutdown_connections 2024-03-23T23:02:31.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.761+0000 7f43f5e00640 1 -- 172.21.15.12:0/375003438 wait complete. 2024-03-23T23:02:31.777 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:02:31.777 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:02:31.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.999+0000 7f9b4d600640 1 -- 172.21.15.12:0/3938322866 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b48102760 msgr2=0x7f9b48104bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:31.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:31.999+0000 7f9b4d600640 1 --2- 172.21.15.12:0/3938322866 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b48102760 0x7f9b48104bd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9b30009900 tx=0x7f9b3002bc40 comp rx=0 tx=0).stop 2024-03-23T23:02:31.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.000+0000 7f9b4d600640 1 -- 172.21.15.12:0/3938322866 shutdown_connections 2024-03-23T23:02:31.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.000+0000 7f9b4d600640 1 --2- 172.21.15.12:0/3938322866 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b48105110 0x7f9b48105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.000+0000 7f9b4d600640 1 --2- 172.21.15.12:0/3938322866 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b48102760 0x7f9b48104bd0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.000+0000 7f9b4d600640 1 --2- 172.21.15.12:0/3938322866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b480ffdf0 0x7f9b48102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:31.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.000+0000 7f9b4d600640 1 -- 172.21.15.12:0/3938322866 >> 172.21.15.12:0/3938322866 conn(0x7f9b480fba10 msgr2=0x7f9b480fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:31.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.000+0000 7f9b4d600640 1 -- 172.21.15.12:0/3938322866 shutdown_connections 2024-03-23T23:02:31.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.000+0000 7f9b4d600640 1 -- 172.21.15.12:0/3938322866 wait complete. 2024-03-23T23:02:32.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.001+0000 7f9b4d600640 1 Processor -- start 2024-03-23T23:02:32.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.001+0000 7f9b4d600640 1 -- start start 2024-03-23T23:02:32.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.001+0000 7f9b4d600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b480ffdf0 0x7f9b4819de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:32.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.001+0000 7f9b4d600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b48102760 0x7f9b4819e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:32.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.001+0000 7f9b4d600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b48105110 0x7f9b481a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.001+0000 7f9b4d600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b48113820 con 0x7f9b48102760 2024-03-23T23:02:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.001+0000 7f9b4d600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b481136a0 con 0x7f9b480ffdf0 2024-03-23T23:02:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.001+0000 7f9b4d600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b481139a0 con 0x7f9b48105110 2024-03-23T23:02:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.001+0000 7f9b47e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b480ffdf0 0x7f9b4819de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.001+0000 7f9b47e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b480ffdf0 0x7f9b4819de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57518/0 (socket says 172.21.15.12:57518) 2024-03-23T23:02:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.001+0000 7f9b47e00640 1 -- 172.21.15.12:0/1064200775 learned_addr learned my addr 172.21.15.12:0/1064200775 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.002+0000 7f9b4cc00640 1 --2- 172.21.15.12:0/1064200775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b48105110 0x7f9b481a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.002+0000 7f9b47400640 1 --2- 172.21.15.12:0/1064200775 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b48102760 0x7f9b4819e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.002+0000 7f9b47e00640 1 -- 172.21.15.12:0/1064200775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b48105110 msgr2=0x7f9b481a5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.002+0000 7f9b47e00640 1 --2- 172.21.15.12:0/1064200775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b48105110 0x7f9b481a5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.002+0000 7f9b47e00640 1 -- 172.21.15.12:0/1064200775 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b48102760 msgr2=0x7f9b4819e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.002+0000 7f9b47e00640 1 --2- 172.21.15.12:0/1064200775 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b48102760 0x7f9b4819e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.002+0000 7f9b47e00640 1 -- 172.21.15.12:0/1064200775 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b481a5a80 con 0x7f9b480ffdf0 2024-03-23T23:02:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.002+0000 7f9b47400640 1 --2- 172.21.15.12:0/1064200775 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b48102760 0x7f9b4819e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:02:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.002+0000 7f9b47e00640 1 --2- 172.21.15.12:0/1064200775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b480ffdf0 0x7f9b4819de60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9b3800d950 tx=0x7f9b3800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:32.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.002+0000 7f9b44c00640 1 -- 172.21.15.12:0/1064200775 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b38017070 con 0x7f9b480ffdf0 2024-03-23T23:02:32.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.002+0000 7f9b44c00640 1 -- 172.21.15.12:0/1064200775 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b38009c30 con 0x7f9b480ffdf0 2024-03-23T23:02:32.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.002+0000 7f9b44c00640 1 -- 172.21.15.12:0/1064200775 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3800f6b0 con 0x7f9b480ffdf0 2024-03-23T23:02:32.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.002+0000 7f9b4d600640 1 -- 172.21.15.12:0/1064200775 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b481a5d70 con 0x7f9b480ffdf0 2024-03-23T23:02:32.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.003+0000 7f9b4d600640 1 -- 172.21.15.12:0/1064200775 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b481a6180 con 0x7f9b480ffdf0 2024-03-23T23:02:32.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.006+0000 7f9b44c00640 1 -- 172.21.15.12:0/1064200775 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9b380109d0 con 0x7f9b480ffdf0 2024-03-23T23:02:32.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.006+0000 7f9b51843740 1 Processor -- start 2024-03-23T23:02:32.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.006+0000 7f9b51843740 1 -- start start 2024-03-23T23:02:32.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.006+0000 7f9b51843740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55836fc2e050 0x55836fc2e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:32.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.006+0000 7f9b51843740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55836fc2ea40 0x55836fc33540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:32.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.006+0000 7f9b51843740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55836fc33bb0 0x55836fc36040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:32.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.006+0000 7f9b51843740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55836fb93250 con 0x55836fc2ea40 2024-03-23T23:02:32.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.006+0000 7f9b51843740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55836fb3ba30 con 0x55836fc33bb0 2024-03-23T23:02:32.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.006+0000 7f9b51843740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55836fb8f2f0 con 0x55836fc2e050 2024-03-23T23:02:32.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.006+0000 7f9b4cc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55836fc2e050 0x55836fc2e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:32.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.006+0000 7f9b4cc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55836fc2e050 0x55836fc2e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:40426/0 (socket says 172.21.15.12:40426) 2024-03-23T23:02:32.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.006+0000 7f9b4cc00640 1 -- 172.21.15.12:0/1018937478 learned_addr learned my addr 172.21.15.12:0/1018937478 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:32.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.006+0000 7f9b47400640 1 --2- 172.21.15.12:0/1018937478 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55836fc33bb0 0x55836fc36040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:32.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.006+0000 7f9b47e00640 1 --2- 172.21.15.12:0/1018937478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55836fc2ea40 0x55836fc33540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:32.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.007+0000 7f9b4cc00640 1 -- 172.21.15.12:0/1018937478 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55836fc33bb0 msgr2=0x55836fc36040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:32.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.007+0000 7f9b4cc00640 1 --2- 172.21.15.12:0/1018937478 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55836fc33bb0 0x55836fc36040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.007+0000 7f9b4cc00640 1 -- 172.21.15.12:0/1018937478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55836fc2ea40 msgr2=0x55836fc33540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:32.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.007+0000 7f9b4cc00640 1 --2- 172.21.15.12:0/1018937478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55836fc2ea40 0x55836fc33540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.007+0000 7f9b4cc00640 1 -- 172.21.15.12:0/1018937478 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55836fb26690 con 0x55836fc2e050 2024-03-23T23:02:32.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.007+0000 7f9b4cc00640 1 --2- 172.21.15.12:0/1018937478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55836fc2e050 0x55836fc2e470 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9b400039f0 tx=0x7f9b40003eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4398859b16763bbd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:32.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.007+0000 7f9b44c00640 1 --2- 172.21.15.12:0/1064200775 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9b2c0a2800 0x7f9b2c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:32.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.007+0000 7f9b3d200640 1 -- 172.21.15.12:0/1018937478 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4001a470 con 0x55836fc2e050 2024-03-23T23:02:32.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.007+0000 7f9b3d200640 1 -- 172.21.15.12:0/1018937478 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b40015070 con 0x55836fc2e050 2024-03-23T23:02:32.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.007+0000 7f9b3d200640 1 -- 172.21.15.12:0/1018937478 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b400108a0 con 0x55836fc2e050 2024-03-23T23:02:32.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.007+0000 7f9b44c00640 1 -- 172.21.15.12:0/1064200775 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9b380ce670 con 0x7f9b480ffdf0 2024-03-23T23:02:32.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.008+0000 7f9b47e00640 1 --2- 172.21.15.12:0/1064200775 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9b2c0a2800 0x7f9b2c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:32.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.008+0000 7f9b51843740 1 -- 172.21.15.12:0/1018937478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55836fc2e050 msgr2=0x55836fc2e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:32.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.008+0000 7f9b51843740 1 --2- 172.21.15.12:0/1018937478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55836fc2e050 0x55836fc2e470 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9b400039f0 tx=0x7f9b40003eb0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.008+0000 7f9b51843740 1 -- 172.21.15.12:0/1018937478 shutdown_connections 2024-03-23T23:02:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.008+0000 7f9b51843740 1 --2- 172.21.15.12:0/1018937478 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55836fc33bb0 0x55836fc36040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.008+0000 7f9b51843740 1 --2- 172.21.15.12:0/1018937478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55836fc2ea40 0x55836fc33540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.008+0000 7f9b51843740 1 --2- 172.21.15.12:0/1018937478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55836fc2e050 0x55836fc2e470 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.008+0000 7f9b51843740 1 -- 172.21.15.12:0/1018937478 >> 172.21.15.12:0/1018937478 conn(0x55836fbf6200 msgr2=0x55836fc299c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.008+0000 7f9b51843740 1 -- 172.21.15.12:0/1018937478 shutdown_connections 2024-03-23T23:02:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.008+0000 7f9b47e00640 1 --2- 172.21.15.12:0/1064200775 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9b2c0a2800 0x7f9b2c0a4cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9b380cc4f0 tx=0x7f9b380bb040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.008+0000 7f9b51843740 1 -- 172.21.15.12:0/1018937478 wait complete. 2024-03-23T23:02:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.009+0000 7f9b51843740 1 -- start start 2024-03-23T23:02:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.009+0000 7f9b51843740 1 Processor -- start 2024-03-23T23:02:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.009+0000 7f9b51843740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55836fc2e050 0x55836fc38030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.009+0000 7f9b51843740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55836fc2ea40 0x55836fc38570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.009+0000 7f9b51843740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55836fc33bb0 0x55836fc3f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.009+0000 7f9b51843740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55836fc36b60 con 0x55836fc2e050 2024-03-23T23:02:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.009+0000 7f9b51843740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55836fb4ee80 con 0x55836fc2ea40 2024-03-23T23:02:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.009+0000 7f9b51843740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55836fc36ce0 con 0x55836fc33bb0 2024-03-23T23:02:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.009+0000 7f9b51843740 10 client.0 _ll_register_callbacks cb 0x55836fb61130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:02:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.010+0000 7f9b47400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55836fc2ea40 0x55836fc38570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.010+0000 7f9b47400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55836fc2ea40 0x55836fc38570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57536/0 (socket says 172.21.15.12:57536) 2024-03-23T23:02:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.010+0000 7f9b47400640 1 -- 172.21.15.12:0/2204831670 learned_addr learned my addr 172.21.15.12:0/2204831670 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.010+0000 7f9b4cc00640 1 --2- 172.21.15.12:0/2204831670 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55836fc2e050 0x55836fc38030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.010+0000 7f9b4cc00640 1 --2- 172.21.15.12:0/2204831670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55836fc33bb0 0x55836fc3f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.010+0000 7f9b47400640 1 -- 172.21.15.12:0/2204831670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55836fc33bb0 msgr2=0x55836fc3f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.010+0000 7f9b47400640 1 --2- 172.21.15.12:0/2204831670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55836fc33bb0 0x55836fc3f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.010+0000 7f9b47400640 1 -- 172.21.15.12:0/2204831670 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55836fc2e050 msgr2=0x55836fc38030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.010+0000 7f9b47400640 1 --2- 172.21.15.12:0/2204831670 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55836fc2e050 0x55836fc38030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.010+0000 7f9b47400640 1 -- 172.21.15.12:0/2204831670 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55836f9717d0 con 0x55836fc2ea40 2024-03-23T23:02:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.010+0000 7f9b4cc00640 1 --2- 172.21.15.12:0/2204831670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55836fc33bb0 0x55836fc3f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:02:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.010+0000 7f9b47400640 1 --2- 172.21.15.12:0/2204831670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55836fc2ea40 0x55836fc38570 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9b3002c7c0 tx=0x7f9b30004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16980856d43746fa server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.010+0000 7f9b35600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:02:32.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.011+0000 7f9b35600640 1 -- 172.21.15.12:0/2204831670 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b30005030 con 0x55836fc2ea40 2024-03-23T23:02:32.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.011+0000 7f9b35600640 1 -- 172.21.15.12:0/2204831670 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9b300051d0 con 0x55836fc2ea40 2024-03-23T23:02:32.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.011+0000 7f9b51843740 1 -- 172.21.15.12:0/2204831670 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9b40015070 con 0x55836fc2ea40 2024-03-23T23:02:32.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.011+0000 7f9b35600640 1 -- 172.21.15.12:0/2204831670 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b300053d0 con 0x55836fc2ea40 2024-03-23T23:02:32.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.011+0000 7f9b51843740 10 client.25496 fetch_fsmap learned FSMap version 23 2024-03-23T23:02:32.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.011+0000 7f9b51843740 1 -- 172.21.15.12:0/2204831670 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55836f8a73f0 con 0x55836fc2ea40 2024-03-23T23:02:32.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.011+0000 7f9b35600640 1 -- 172.21.15.12:0/2204831670 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b40015070 con 0x55836fc2ea40 2024-03-23T23:02:32.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.011+0000 7f9b35600640 1 -- 172.21.15.12:0/2204831670 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f9b30003820 con 0x55836fc2ea40 2024-03-23T23:02:32.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.011+0000 7f9b51843740 10 client.25496 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:02:32.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.011+0000 7f9b51843740 10 client.25496 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:02:32.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.011+0000 7f9b51843740 20 client.25496 populate_metadata read hostname 'smithi012' 2024-03-23T23:02:32.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.012+0000 7f9b51843740 1 --2- 172.21.15.12:0/2204831670 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55836fc43dc0 0x55836fc46220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:32.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.012+0000 7f9b51843740 4 client.25496 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:02:32.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.012+0000 7f9b51843740 1 -- 172.21.15.12:0/2204831670 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55836fba0790 con 0x55836fc43dc0 2024-03-23T23:02:32.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.012+0000 7f9b4cc00640 1 --2- 172.21.15.12:0/2204831670 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55836fc43dc0 0x55836fc46220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:32.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.012+0000 7f9b4cc00640 1 --2- 172.21.15.12:0/2204831670 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55836fc43dc0 0x55836fc46220 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86075cfd02b8ad28 server_cookie=5ed4a2e438d36b6c in_seq=0 out_seq=0 2024-03-23T23:02:32.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.012+0000 7f9b35600640 10 client.25496 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:32.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.013+0000 7f9b35600640 1 -- 172.21.15.12:0/2204831670 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55836fba0790 con 0x55836fc43dc0 2024-03-23T23:02:32.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.013+0000 7f9b35600640 10 client.25496 handle_command_reply: tid=0 2024-03-23T23:02:32.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.013+0000 7f9b51843740 1 client.25496 shutdown 2024-03-23T23:02:32.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.014+0000 7f9b51843740 1 -- 172.21.15.12:0/2204831670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55836fc2ea40 msgr2=0x55836fc38570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:32.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.014+0000 7f9b51843740 1 --2- 172.21.15.12:0/2204831670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55836fc2ea40 0x55836fc38570 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9b3002c7c0 tx=0x7f9b30004270 comp rx=0 tx=0).stop 2024-03-23T23:02:32.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.014+0000 7f9b51843740 1 -- 172.21.15.12:0/2204831670 shutdown_connections 2024-03-23T23:02:32.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.014+0000 7f9b51843740 1 --2- 172.21.15.12:0/2204831670 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55836fc43dc0 0x55836fc46220 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.014+0000 7f9b51843740 1 --2- 172.21.15.12:0/2204831670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55836fc33bb0 0x55836fc3f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.014+0000 7f9b51843740 1 --2- 172.21.15.12:0/2204831670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55836fc2ea40 0x55836fc38570 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.014+0000 7f9b35600640 0 client.25496 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:32.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.014+0000 7f9b51843740 1 --2- 172.21.15.12:0/2204831670 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55836fc2e050 0x55836fc38030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.014+0000 7f9b51843740 1 -- 172.21.15.12:0/2204831670 >> 172.21.15.12:0/2204831670 conn(0x55836fbf6200 msgr2=0x55836fc28bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:32.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.014+0000 7f9b51843740 1 -- 172.21.15.12:0/2204831670 shutdown_connections 2024-03-23T23:02:32.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.014+0000 7f9b51843740 1 -- 172.21.15.12:0/2204831670 wait complete. 2024-03-23T23:02:32.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.014+0000 7f9b51843740 20 client.25496 trim_cache size 0 max 16384 2024-03-23T23:02:32.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.032+0000 7f9b51843740 1 Processor -- start 2024-03-23T23:02:32.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.032+0000 7f9b51843740 1 -- start start 2024-03-23T23:02:32.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.032+0000 7f9b51843740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55836fc2e050 0x55836fc28a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:32.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.032+0000 7f9b51843740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55836fc2ea40 0x55836fc28f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:32.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.032+0000 7f9b51843740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55836fc33bb0 0x55836fc3f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:32.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.032+0000 7f9b51843740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b481b1360 con 0x55836fc2ea40 2024-03-23T23:02:32.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.032+0000 7f9b51843740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b481aee60 con 0x55836fc33bb0 2024-03-23T23:02:32.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.032+0000 7f9b51843740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b481acf00 con 0x55836fc2e050 2024-03-23T23:02:32.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.032+0000 7f9b47400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55836fc2ea40 0x55836fc28f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.033+0000 7f9b47400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55836fc2ea40 0x55836fc28f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54630/0 (socket says 172.21.15.12:54630) 2024-03-23T23:02:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.033+0000 7f9b47400640 1 -- 172.21.15.12:0/3903456700 learned_addr learned my addr 172.21.15.12:0/3903456700 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.033+0000 7f9b4cc00640 1 --2- 172.21.15.12:0/3903456700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55836fc2e050 0x55836fc28a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.033+0000 7f9b4cc00640 1 --2- 172.21.15.12:0/3903456700 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55836fc33bb0 0x55836fc3f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.033+0000 7f9b47400640 1 -- 172.21.15.12:0/3903456700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55836fc2e050 msgr2=0x55836fc28a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.033+0000 7f9b47400640 1 --2- 172.21.15.12:0/3903456700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55836fc2e050 0x55836fc28a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.033+0000 7f9b47400640 1 -- 172.21.15.12:0/3903456700 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55836fc33bb0 msgr2=0x55836fc3f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.033+0000 7f9b47400640 1 --2- 172.21.15.12:0/3903456700 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55836fc33bb0 0x55836fc3f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.033+0000 7f9b47400640 1 -- 172.21.15.12:0/3903456700 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55836f8670f0 con 0x55836fc2ea40 2024-03-23T23:02:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.033+0000 7f9b4cc00640 1 --2- 172.21.15.12:0/3903456700 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55836fc33bb0 0x55836fc3f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:02:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.033+0000 7f9b47400640 1 --2- 172.21.15.12:0/3903456700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55836fc2ea40 0x55836fc28f80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9b30001eb0 tx=0x7f9b300095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26e07eec9dd8610a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.034+0000 7f9b37e00640 1 -- 172.21.15.12:0/3903456700 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b30056070 con 0x55836fc2ea40 2024-03-23T23:02:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.034+0000 7f9b37e00640 1 -- 172.21.15.12:0/3903456700 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b30005570 con 0x55836fc2ea40 2024-03-23T23:02:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.034+0000 7f9b37e00640 1 -- 172.21.15.12:0/3903456700 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b30047a20 con 0x55836fc2ea40 2024-03-23T23:02:32.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.034+0000 7f9b51843740 1 -- 172.21.15.12:0/3903456700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55836fc2ea40 msgr2=0x55836fc28f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:32.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.034+0000 7f9b51843740 1 --2- 172.21.15.12:0/3903456700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55836fc2ea40 0x55836fc28f80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9b30001eb0 tx=0x7f9b300095a0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.035+0000 7f9b51843740 1 -- 172.21.15.12:0/3903456700 shutdown_connections 2024-03-23T23:02:32.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.035+0000 7f9b51843740 1 --2- 172.21.15.12:0/3903456700 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55836fc33bb0 0x55836fc3f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.035+0000 7f9b51843740 1 --2- 172.21.15.12:0/3903456700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55836fc2ea40 0x55836fc28f80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.035+0000 7f9b51843740 1 --2- 172.21.15.12:0/3903456700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55836fc2e050 0x55836fc28a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.035+0000 7f9b51843740 1 -- 172.21.15.12:0/3903456700 >> 172.21.15.12:0/3903456700 conn(0x55836fbf6200 msgr2=0x55836fc28330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:32.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.035+0000 7f9b51843740 1 -- 172.21.15.12:0/3903456700 shutdown_connections 2024-03-23T23:02:32.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.035+0000 7f9b51843740 1 -- 172.21.15.12:0/3903456700 wait complete. 2024-03-23T23:02:32.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.035+0000 7f9b51843740 1 -- start start 2024-03-23T23:02:32.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.035+0000 7f9b51843740 1 Processor -- start 2024-03-23T23:02:32.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.036+0000 7f9b51843740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55836fc2e050 0x55836fc3d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:32.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.036+0000 7f9b51843740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55836fc2ea40 0x55836fc3d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.036+0000 7f9b51843740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55836fc33bb0 0x55836fc3e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.036+0000 7f9b51843740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b481ad500 con 0x55836fc2ea40 2024-03-23T23:02:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.036+0000 7f9b51843740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b481ae800 con 0x55836fc33bb0 2024-03-23T23:02:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.036+0000 7f9b51843740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b481adb40 con 0x55836fc2e050 2024-03-23T23:02:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.036+0000 7f9b51843740 10 client.0 _ll_register_callbacks cb 0x7f9b30009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:02:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.036+0000 7f9b47400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55836fc2ea40 0x55836fc3d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.036+0000 7f9b4cc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55836fc2e050 0x55836fc3d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.036+0000 7f9b47400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55836fc2ea40 0x55836fc3d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54638/0 (socket says 172.21.15.12:54638) 2024-03-23T23:02:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.036+0000 7f9b47400640 1 -- 172.21.15.12:0/833601619 learned_addr learned my addr 172.21.15.12:0/833601619 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.036+0000 7f9b4cc00640 1 --2- 172.21.15.12:0/833601619 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55836fc33bb0 0x55836fc3e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.036+0000 7f9b47400640 1 -- 172.21.15.12:0/833601619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55836fc2e050 msgr2=0x55836fc3d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.036+0000 7f9b47400640 1 --2- 172.21.15.12:0/833601619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55836fc2e050 0x55836fc3d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.036+0000 7f9b47400640 1 -- 172.21.15.12:0/833601619 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55836fc33bb0 msgr2=0x55836fc3e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.036+0000 7f9b47400640 1 --2- 172.21.15.12:0/833601619 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55836fc33bb0 0x55836fc3e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.036+0000 7f9b47400640 1 -- 172.21.15.12:0/833601619 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55836fc3efe0 con 0x55836fc2ea40 2024-03-23T23:02:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.036+0000 7f9b4cc00640 1 --2- 172.21.15.12:0/833601619 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55836fc33bb0 0x55836fc3e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:02:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.037+0000 7f9b47400640 1 --2- 172.21.15.12:0/833601619 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55836fc2ea40 0x55836fc3d9e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9b3002c7c0 tx=0x7f9b30005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c78d9942a45eb1e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.037+0000 7f9b35600640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:02:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.037+0000 7f9b35600640 1 -- 172.21.15.12:0/833601619 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b30056070 con 0x55836fc2ea40 2024-03-23T23:02:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.037+0000 7f9b35600640 1 -- 172.21.15.12:0/833601619 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9b300037f0 con 0x55836fc2ea40 2024-03-23T23:02:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.037+0000 7f9b51843740 1 -- 172.21.15.12:0/833601619 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9b30005570 con 0x55836fc2ea40 2024-03-23T23:02:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.037+0000 7f9b35600640 1 -- 172.21.15.12:0/833601619 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b300039f0 con 0x55836fc2ea40 2024-03-23T23:02:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.037+0000 7f9b51843740 10 client.15369 fetch_fsmap learned FSMap version 23 2024-03-23T23:02:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.037+0000 7f9b51843740 1 -- 172.21.15.12:0/833601619 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55836fc9fe00 con 0x55836fc2ea40 2024-03-23T23:02:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.038+0000 7f9b35600640 1 -- 172.21.15.12:0/833601619 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b30005570 con 0x55836fc2ea40 2024-03-23T23:02:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.038+0000 7f9b35600640 1 -- 172.21.15.12:0/833601619 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f9b30034710 con 0x55836fc2ea40 2024-03-23T23:02:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.038+0000 7f9b51843740 10 client.15369 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:02:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.038+0000 7f9b51843740 10 client.15369 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:02:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.038+0000 7f9b51843740 20 client.15369 populate_metadata read hostname 'smithi012' 2024-03-23T23:02:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.038+0000 7f9b51843740 1 --2- 172.21.15.12:0/833601619 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55836fca0690 0x55836fca2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.038+0000 7f9b51843740 4 client.15369 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:02:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.038+0000 7f9b51843740 1 -- 172.21.15.12:0/833601619 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55836fb9aae0 con 0x55836fca0690 2024-03-23T23:02:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.038+0000 7f9b4cc00640 1 --2- 172.21.15.12:0/833601619 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55836fca0690 0x55836fca2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.039+0000 7f9b4cc00640 1 --2- 172.21.15.12:0/833601619 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55836fca0690 0x55836fca2ab0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe1ef308cb53b267 server_cookie=6a1e019c07810030 in_seq=0 out_seq=0 2024-03-23T23:02:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.039+0000 7f9b35600640 10 client.15369 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.039+0000 7f9b35600640 1 -- 172.21.15.12:0/833601619 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55836fb9aae0 con 0x55836fca0690 2024-03-23T23:02:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.039+0000 7f9b35600640 10 client.15369 handle_command_reply: tid=0 2024-03-23T23:02:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.039+0000 7f9b51843740 1 client.15369 shutdown 2024-03-23T23:02:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.039+0000 7f9b51843740 1 -- 172.21.15.12:0/833601619 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55836fc2ea40 msgr2=0x55836fc3d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.039+0000 7f9b51843740 1 --2- 172.21.15.12:0/833601619 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55836fc2ea40 0x55836fc3d9e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9b3002c7c0 tx=0x7f9b30005720 comp rx=0 tx=0).stop 2024-03-23T23:02:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.039+0000 7f9b51843740 1 -- 172.21.15.12:0/833601619 shutdown_connections 2024-03-23T23:02:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.039+0000 7f9b51843740 1 --2- 172.21.15.12:0/833601619 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55836fca0690 0x55836fca2ab0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.039+0000 7f9b51843740 1 --2- 172.21.15.12:0/833601619 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55836fc33bb0 0x55836fc3e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.039+0000 7f9b51843740 1 --2- 172.21.15.12:0/833601619 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55836fc2ea40 0x55836fc3d9e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.039+0000 7f9b51843740 1 --2- 172.21.15.12:0/833601619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55836fc2e050 0x55836fc3d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.039+0000 7f9b51843740 1 -- 172.21.15.12:0/833601619 >> 172.21.15.12:0/833601619 conn(0x55836fbf6200 msgr2=0x55836fc28bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.039+0000 7f9b35600640 0 client.15369 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.039+0000 7f9b51843740 1 -- 172.21.15.12:0/833601619 shutdown_connections 2024-03-23T23:02:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.039+0000 7f9b51843740 1 -- 172.21.15.12:0/833601619 wait complete. 2024-03-23T23:02:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.039+0000 7f9b51843740 20 client.15369 trim_cache size 0 max 16384 2024-03-23T23:02:32.043 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:02:32.043 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T23:02:32.043 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-03-23T23:02:32.043 INFO:teuthology.orchestra.run.smithi012.stdout: "1f69d79e-356d-488e-a311-fe221b87084a": { 2024-03-23T23:02:32.043 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-03-23T23:02:32.043 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "1f69d79e-356d-488e-a311-fe221b87084a", 2024-03-23T23:02:32.043 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-03-23T23:02:32.043 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:02:32.043 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:02:32.043 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:02:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.040+0000 7f9b4d600640 1 -- 172.21.15.12:0/1064200775 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9b2c0a2800 msgr2=0x7f9b2c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.040+0000 7f9b4d600640 1 --2- 172.21.15.12:0/1064200775 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9b2c0a2800 0x7f9b2c0a4cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9b380cc4f0 tx=0x7f9b380bb040 comp rx=0 tx=0).stop 2024-03-23T23:02:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.040+0000 7f9b4d600640 1 -- 172.21.15.12:0/1064200775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b480ffdf0 msgr2=0x7f9b4819de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.040+0000 7f9b4d600640 1 --2- 172.21.15.12:0/1064200775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b480ffdf0 0x7f9b4819de60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9b3800d950 tx=0x7f9b3800de10 comp rx=0 tx=0).stop 2024-03-23T23:02:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.040+0000 7f9b4d600640 1 -- 172.21.15.12:0/1064200775 shutdown_connections 2024-03-23T23:02:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.040+0000 7f9b4d600640 1 --2- 172.21.15.12:0/1064200775 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9b2c0a2800 0x7f9b2c0a4cc0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.040+0000 7f9b4d600640 1 --2- 172.21.15.12:0/1064200775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b48105110 0x7f9b481a5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.040+0000 7f9b4d600640 1 --2- 172.21.15.12:0/1064200775 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b48102760 0x7f9b4819e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.040+0000 7f9b4d600640 1 --2- 172.21.15.12:0/1064200775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b480ffdf0 0x7f9b4819de60 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.040+0000 7f9b4d600640 1 -- 172.21.15.12:0/1064200775 >> 172.21.15.12:0/1064200775 conn(0x7f9b480fba10 msgr2=0x7f9b48100770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.040+0000 7f9b4d600640 1 -- 172.21.15.12:0/1064200775 shutdown_connections 2024-03-23T23:02:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:02:32.040+0000 7f9b4d600640 1 -- 172.21.15.12:0/1064200775 wait complete. 2024-03-23T23:02:32.059 INFO:tasks.cephfs.filesystem:scrub status for tag:1f69d79e-356d-488e-a311-fe221b87084a - {'path': '/', 'tag': '1f69d79e-356d-488e-a311-fe221b87084a', 'options': 'recursive,force'} 2024-03-23T23:02:32.709 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:32 smithi203 ceph-mon[36693]: pgmap v412: 97 pgs: 97 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-23T23:02:32.709 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:32 smithi203 ceph-mon[36693]: scrub queued for path: / 2024-03-23T23:02:32.709 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:32 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:02:32.710 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:32 smithi203 ceph-mon[36693]: scrub summary: active paths [/] 2024-03-23T23:02:32.710 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:32 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:02:32.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:32 smithi012 ceph-mon[31695]: pgmap v412: 97 pgs: 97 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-23T23:02:32.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:32 smithi012 ceph-mon[31695]: scrub queued for path: / 2024-03-23T23:02:32.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:32 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:02:32.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:32 smithi012 ceph-mon[31695]: scrub summary: active paths [/] 2024-03-23T23:02:32.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:32 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:02:32.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:32 smithi173 ceph-mon[36116]: pgmap v412: 97 pgs: 97 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-23T23:02:32.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:32 smithi173 ceph-mon[36116]: scrub queued for path: / 2024-03-23T23:02:32.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:32 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:02:32.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:32 smithi173 ceph-mon[36116]: scrub summary: active paths [/] 2024-03-23T23:02:32.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:32 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:02:34.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:34 smithi012 ceph-mon[31695]: pgmap v413: 97 pgs: 97 active+clean; 662 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-23T23:02:34.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:34 smithi173 ceph-mon[36116]: pgmap v413: 97 pgs: 97 active+clean; 662 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-23T23:02:34.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:34 smithi203 ceph-mon[36693]: pgmap v413: 97 pgs: 97 active+clean; 662 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-23T23:02:36.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:36 smithi012 ceph-mon[31695]: pgmap v414: 97 pgs: 97 active+clean; 662 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-23T23:02:36.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:36 smithi012 ceph-mon[31695]: scrub summary: idle 2024-03-23T23:02:36.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:36 smithi173 ceph-mon[36116]: pgmap v414: 97 pgs: 97 active+clean; 662 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-23T23:02:36.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:36 smithi173 ceph-mon[36116]: scrub summary: idle 2024-03-23T23:02:36.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:36 smithi203 ceph-mon[36693]: pgmap v414: 97 pgs: 97 active+clean; 662 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-23T23:02:36.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:36 smithi203 ceph-mon[36693]: scrub summary: idle 2024-03-23T23:02:37.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.936+0000 7f1c76400640 1 -- 172.21.15.12:0/2932616652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c701021e0 msgr2=0x7f1c701045d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:37.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.936+0000 7f1c76400640 1 --2- 172.21.15.12:0/2932616652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c701021e0 0x7f1c701045d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1c60009900 tx=0x7f1c6002bc30 comp rx=0 tx=0).stop 2024-03-23T23:02:37.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.936+0000 7f1c76400640 1 -- 172.21.15.12:0/2932616652 shutdown_connections 2024-03-23T23:02:37.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.936+0000 7f1c76400640 1 --2- 172.21.15.12:0/2932616652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c70107440 0x7f1c70109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.936+0000 7f1c76400640 1 --2- 172.21.15.12:0/2932616652 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c70104b10 0x7f1c70106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.936+0000 7f1c76400640 1 --2- 172.21.15.12:0/2932616652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c701021e0 0x7f1c701045d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.936+0000 7f1c76400640 1 -- 172.21.15.12:0/2932616652 >> 172.21.15.12:0/2932616652 conn(0x7f1c700fbb30 msgr2=0x7f1c700fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.936+0000 7f1c76400640 1 -- 172.21.15.12:0/2932616652 shutdown_connections 2024-03-23T23:02:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.936+0000 7f1c76400640 1 -- 172.21.15.12:0/2932616652 wait complete. 2024-03-23T23:02:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.937+0000 7f1c76400640 1 Processor -- start 2024-03-23T23:02:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.938+0000 7f1c76400640 1 -- start start 2024-03-23T23:02:37.938 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.938+0000 7f1c76400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c701021e0 0x7f1c70197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:37.938 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.938+0000 7f1c76400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c70104b10 0x7f1c70197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:37.938 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.938+0000 7f1c76400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c70107440 0x7f1c7019eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:37.938 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.938+0000 7f1c76400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7010d380 con 0x7f1c70107440 2024-03-23T23:02:37.938 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.938+0000 7f1c76400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7010d200 con 0x7f1c70104b10 2024-03-23T23:02:37.938 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.938+0000 7f1c76400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7010d500 con 0x7f1c701021e0 2024-03-23T23:02:37.938 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.938+0000 7f1c6f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c701021e0 0x7f1c70197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:37.938 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.938+0000 7f1c66a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c70104b10 0x7f1c70197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:37.938 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.938+0000 7f1c6f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c701021e0 0x7f1c70197940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:42280/0 (socket says 172.21.15.12:42280) 2024-03-23T23:02:37.938 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.938+0000 7f1c6f400640 1 -- 172.21.15.12:0/3501483096 learned_addr learned my addr 172.21.15.12:0/3501483096 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:37.938 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.938+0000 7f1c6fe00640 1 --2- 172.21.15.12:0/3501483096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c70107440 0x7f1c7019eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.939+0000 7f1c6f400640 1 -- 172.21.15.12:0/3501483096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c70104b10 msgr2=0x7f1c70197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.939+0000 7f1c6f400640 1 --2- 172.21.15.12:0/3501483096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c70104b10 0x7f1c70197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.939+0000 7f1c6f400640 1 -- 172.21.15.12:0/3501483096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c70107440 msgr2=0x7f1c7019eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.939+0000 7f1c6f400640 1 --2- 172.21.15.12:0/3501483096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c70107440 0x7f1c7019eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.939+0000 7f1c6f400640 1 -- 172.21.15.12:0/3501483096 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c7019f560 con 0x7f1c701021e0 2024-03-23T23:02:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.939+0000 7f1c66a00640 1 --2- 172.21.15.12:0/3501483096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c70104b10 0x7f1c70197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:02:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.939+0000 7f1c6fe00640 1 --2- 172.21.15.12:0/3501483096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c70107440 0x7f1c7019eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:02:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.939+0000 7f1c6f400640 1 --2- 172.21.15.12:0/3501483096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c701021e0 0x7f1c70197940 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1c60005ab0 tx=0x7f1c6002c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.939+0000 7f1c6cc00640 1 -- 172.21.15.12:0/3501483096 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c60043020 con 0x7f1c701021e0 2024-03-23T23:02:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.940+0000 7f1c6cc00640 1 -- 172.21.15.12:0/3501483096 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c600054e0 con 0x7f1c701021e0 2024-03-23T23:02:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.940+0000 7f1c6cc00640 1 -- 172.21.15.12:0/3501483096 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c60003ce0 con 0x7f1c701021e0 2024-03-23T23:02:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.940+0000 7f1c76400640 1 -- 172.21.15.12:0/3501483096 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c7019f7f0 con 0x7f1c701021e0 2024-03-23T23:02:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.940+0000 7f1c76400640 1 -- 172.21.15.12:0/3501483096 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c7019fd40 con 0x7f1c701021e0 2024-03-23T23:02:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.943+0000 7f1c6cc00640 1 -- 172.21.15.12:0/3501483096 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1c60046020 con 0x7f1c701021e0 2024-03-23T23:02:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.943+0000 7f1c7919b740 1 Processor -- start 2024-03-23T23:02:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.943+0000 7f1c7919b740 1 -- start start 2024-03-23T23:02:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.944+0000 7f1c7919b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf71261e0 0x55fbf7126600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.944+0000 7f1c7919b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fbf7126bd0 0x55fbf712b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.944+0000 7f1c7919b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fbf712bd40 0x55fbf712e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.944+0000 7f1c7919b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbf70e93b0 con 0x55fbf712bd40 2024-03-23T23:02:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.944+0000 7f1c7919b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbf708b830 con 0x55fbf71261e0 2024-03-23T23:02:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.944+0000 7f1c7919b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbf70345d0 con 0x55fbf7126bd0 2024-03-23T23:02:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.944+0000 7f1c6fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf71261e0 0x55fbf7126600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.944+0000 7f1c6fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf71261e0 0x55fbf7126600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:52632/0 (socket says 172.21.15.12:52632) 2024-03-23T23:02:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.944+0000 7f1c6fe00640 1 -- 172.21.15.12:0/3374382976 learned_addr learned my addr 172.21.15.12:0/3374382976 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.944+0000 7f1c6f400640 1 --2- 172.21.15.12:0/3374382976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fbf7126bd0 0x55fbf712b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.944+0000 7f1c66a00640 1 --2- 172.21.15.12:0/3374382976 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fbf712bd40 0x55fbf712e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.945+0000 7f1c6fe00640 1 -- 172.21.15.12:0/3374382976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fbf7126bd0 msgr2=0x55fbf712b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.945+0000 7f1c6fe00640 1 --2- 172.21.15.12:0/3374382976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fbf7126bd0 0x55fbf712b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.945+0000 7f1c6fe00640 1 -- 172.21.15.12:0/3374382976 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fbf712bd40 msgr2=0x55fbf712e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.945+0000 7f1c6fe00640 1 --2- 172.21.15.12:0/3374382976 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fbf712bd40 0x55fbf712e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.945+0000 7f1c6fe00640 1 -- 172.21.15.12:0/3374382976 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fbf704c2c0 con 0x55fbf71261e0 2024-03-23T23:02:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.945+0000 7f1c66a00640 1 --2- 172.21.15.12:0/3374382976 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fbf712bd40 0x55fbf712e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:02:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.945+0000 7f1c6cc00640 1 --2- 172.21.15.12:0/3501483096 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1c540a28a0 0x7f1c540a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.945+0000 7f1c6fe00640 1 --2- 172.21.15.12:0/3374382976 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf71261e0 0x55fbf7126600 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1c68009fb0 tx=0x7f1c68010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3519835de799ccf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.945+0000 7f1c6cc00640 1 -- 172.21.15.12:0/3501483096 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1c600f0c50 con 0x7f1c701021e0 2024-03-23T23:02:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.945+0000 7f1c64c00640 1 -- 172.21.15.12:0/3374382976 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c68016070 con 0x55fbf71261e0 2024-03-23T23:02:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.945+0000 7f1c6f400640 1 --2- 172.21.15.12:0/3501483096 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1c540a28a0 0x7f1c540a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.945+0000 7f1c64c00640 1 -- 172.21.15.12:0/3374382976 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c68012bf0 con 0x55fbf71261e0 2024-03-23T23:02:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.945+0000 7f1c64c00640 1 -- 172.21.15.12:0/3374382976 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6801bd20 con 0x55fbf71261e0 2024-03-23T23:02:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.946+0000 7f1c7919b740 1 -- 172.21.15.12:0/3374382976 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf71261e0 msgr2=0x55fbf7126600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.946+0000 7f1c6f400640 1 --2- 172.21.15.12:0/3501483096 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1c540a28a0 0x7f1c540a4d60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1c600bd2a0 tx=0x7f1c600f4690 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.946+0000 7f1c7919b740 1 --2- 172.21.15.12:0/3374382976 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf71261e0 0x55fbf7126600 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1c68009fb0 tx=0x7f1c68010800 comp rx=0 tx=0).stop 2024-03-23T23:02:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.946+0000 7f1c7919b740 1 -- 172.21.15.12:0/3374382976 shutdown_connections 2024-03-23T23:02:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.946+0000 7f1c7919b740 1 --2- 172.21.15.12:0/3374382976 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fbf712bd40 0x55fbf712e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.946+0000 7f1c7919b740 1 --2- 172.21.15.12:0/3374382976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fbf7126bd0 0x55fbf712b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.946+0000 7f1c7919b740 1 --2- 172.21.15.12:0/3374382976 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf71261e0 0x55fbf7126600 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.946+0000 7f1c7919b740 1 -- 172.21.15.12:0/3374382976 >> 172.21.15.12:0/3374382976 conn(0x55fbf7121750 msgr2=0x55fbf7121b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.946+0000 7f1c7919b740 1 -- 172.21.15.12:0/3374382976 shutdown_connections 2024-03-23T23:02:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.946+0000 7f1c7919b740 1 -- 172.21.15.12:0/3374382976 wait complete. 2024-03-23T23:02:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.947+0000 7f1c7919b740 1 -- start start 2024-03-23T23:02:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.947+0000 7f1c7919b740 1 Processor -- start 2024-03-23T23:02:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.947+0000 7f1c7919b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fbf71261e0 0x55fbf7134800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.947+0000 7f1c7919b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf7126bd0 0x55fbf7134d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.947+0000 7f1c7919b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fbf712bd40 0x55fbf713bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.947+0000 7f1c7919b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbf7047a20 con 0x55fbf71261e0 2024-03-23T23:02:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.947+0000 7f1c7919b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbf70878d0 con 0x55fbf7126bd0 2024-03-23T23:02:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.947+0000 7f1c7919b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbf712ecf0 con 0x55fbf712bd40 2024-03-23T23:02:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.947+0000 7f1c7919b740 10 client.0 _ll_register_callbacks cb 0x55fbf70841c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:02:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.948+0000 7f1c66a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf7126bd0 0x55fbf7134d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.948+0000 7f1c66a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf7126bd0 0x55fbf7134d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:52636/0 (socket says 172.21.15.12:52636) 2024-03-23T23:02:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.948+0000 7f1c66a00640 1 -- 172.21.15.12:0/3174531734 learned_addr learned my addr 172.21.15.12:0/3174531734 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.948+0000 7f1c6fe00640 1 --2- 172.21.15.12:0/3174531734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fbf71261e0 0x55fbf7134800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.948+0000 7f1c6fe00640 1 --2- 172.21.15.12:0/3174531734 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fbf712bd40 0x55fbf713bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.948+0000 7f1c66a00640 1 -- 172.21.15.12:0/3174531734 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fbf712bd40 msgr2=0x55fbf713bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.948+0000 7f1c66a00640 1 --2- 172.21.15.12:0/3174531734 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fbf712bd40 0x55fbf713bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.948+0000 7f1c66a00640 1 -- 172.21.15.12:0/3174531734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fbf71261e0 msgr2=0x55fbf7134800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.948+0000 7f1c66a00640 1 --2- 172.21.15.12:0/3174531734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fbf71261e0 0x55fbf7134800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.948+0000 7f1c66a00640 1 -- 172.21.15.12:0/3174531734 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fbf6da0210 con 0x55fbf7126bd0 2024-03-23T23:02:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.948+0000 7f1c6fe00640 1 --2- 172.21.15.12:0/3174531734 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fbf712bd40 0x55fbf713bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:02:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.948+0000 7f1c66a00640 1 --2- 172.21.15.12:0/3174531734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf7126bd0 0x55fbf7134d40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1c58012a10 tx=0x7f1c58012ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87981ca108a1002b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.948+0000 7f1c5d600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:02:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.948+0000 7f1c5d600640 1 -- 172.21.15.12:0/3174531734 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5800fa60 con 0x55fbf7126bd0 2024-03-23T23:02:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.949+0000 7f1c5d600640 1 -- 172.21.15.12:0/3174531734 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1c5800fc00 con 0x55fbf7126bd0 2024-03-23T23:02:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.949+0000 7f1c7919b740 1 -- 172.21.15.12:0/3174531734 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1c68012bf0 con 0x55fbf7126bd0 2024-03-23T23:02:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.949+0000 7f1c5d600640 1 -- 172.21.15.12:0/3174531734 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c5800fe00 con 0x55fbf7126bd0 2024-03-23T23:02:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.949+0000 7f1c7919b740 10 client.25514 fetch_fsmap learned FSMap version 23 2024-03-23T23:02:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.949+0000 7f1c7919b740 1 -- 172.21.15.12:0/3174531734 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55fbf6e6c620 con 0x55fbf7126bd0 2024-03-23T23:02:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.949+0000 7f1c5d600640 1 -- 172.21.15.12:0/3174531734 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c68012bf0 con 0x55fbf7126bd0 2024-03-23T23:02:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.949+0000 7f1c5d600640 1 -- 172.21.15.12:0/3174531734 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f1c58010690 con 0x55fbf7126bd0 2024-03-23T23:02:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.949+0000 7f1c7919b740 10 client.25514 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:02:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.949+0000 7f1c7919b740 10 client.25514 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:02:37.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.949+0000 7f1c7919b740 20 client.25514 populate_metadata read hostname 'smithi012' 2024-03-23T23:02:37.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.949+0000 7f1c7919b740 1 --2- 172.21.15.12:0/3174531734 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fbf7140600 0x55fbf7142a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:37.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.949+0000 7f1c7919b740 4 client.25514 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:02:37.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.949+0000 7f1c7919b740 1 -- 172.21.15.12:0/3174531734 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fbf7098d70 con 0x55fbf7140600 2024-03-23T23:02:37.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.950+0000 7f1c6fe00640 1 --2- 172.21.15.12:0/3174531734 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fbf7140600 0x55fbf7142a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:37.953 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.950+0000 7f1c6fe00640 1 --2- 172.21.15.12:0/3174531734 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fbf7140600 0x55fbf7142a10 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57ec8439f209478d server_cookie=50224c40b6f3372 in_seq=0 out_seq=0 2024-03-23T23:02:37.953 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.950+0000 7f1c5d600640 10 client.25514 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:37.953 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.951+0000 7f1c5d600640 1 -- 172.21.15.12:0/3174531734 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55fbf7098d70 con 0x55fbf7140600 2024-03-23T23:02:37.953 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.951+0000 7f1c5d600640 10 client.25514 handle_command_reply: tid=0 2024-03-23T23:02:37.953 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.951+0000 7f1c7919b740 1 client.25514 shutdown 2024-03-23T23:02:37.953 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.951+0000 7f1c7919b740 1 -- 172.21.15.12:0/3174531734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf7126bd0 msgr2=0x55fbf7134d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:37.953 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.951+0000 7f1c7919b740 1 --2- 172.21.15.12:0/3174531734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf7126bd0 0x55fbf7134d40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1c58012a10 tx=0x7f1c58012ed0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.953 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.951+0000 7f1c7919b740 1 -- 172.21.15.12:0/3174531734 shutdown_connections 2024-03-23T23:02:37.953 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.951+0000 7f1c7919b740 1 --2- 172.21.15.12:0/3174531734 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fbf7140600 0x55fbf7142a10 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.951+0000 7f1c7919b740 1 --2- 172.21.15.12:0/3174531734 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fbf712bd40 0x55fbf713bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.951+0000 7f1c7919b740 1 --2- 172.21.15.12:0/3174531734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf7126bd0 0x55fbf7134d40 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.951+0000 7f1c5d600640 0 client.25514 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.951+0000 7f1c7919b740 1 --2- 172.21.15.12:0/3174531734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fbf71261e0 0x55fbf7134800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.951+0000 7f1c7919b740 1 -- 172.21.15.12:0/3174531734 >> 172.21.15.12:0/3174531734 conn(0x55fbf7121750 msgr2=0x55fbf7120950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.951+0000 7f1c7919b740 1 -- 172.21.15.12:0/3174531734 shutdown_connections 2024-03-23T23:02:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.951+0000 7f1c7919b740 1 -- 172.21.15.12:0/3174531734 wait complete. 2024-03-23T23:02:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.952+0000 7f1c7919b740 20 client.25514 trim_cache size 0 max 16384 2024-03-23T23:02:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.967+0000 7f1c7919b740 1 Processor -- start 2024-03-23T23:02:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.967+0000 7f1c7919b740 1 -- start start 2024-03-23T23:02:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.967+0000 7f1c7919b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fbf71261e0 0x55fbf712f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.967+0000 7f1c7919b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf7126bd0 0x55fbf713bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.967+0000 7f1c7919b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fbf712bd40 0x55fbf71200c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.967+0000 7f1c7919b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c701a8c30 con 0x55fbf71261e0 2024-03-23T23:02:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.967+0000 7f1c7919b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c701a6730 con 0x55fbf7126bd0 2024-03-23T23:02:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.967+0000 7f1c7919b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c701a47d0 con 0x55fbf712bd40 2024-03-23T23:02:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.967+0000 7f1c66a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf7126bd0 0x55fbf713bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.968+0000 7f1c66a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf7126bd0 0x55fbf713bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:52638/0 (socket says 172.21.15.12:52638) 2024-03-23T23:02:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.968+0000 7f1c66a00640 1 -- 172.21.15.12:0/2954441386 learned_addr learned my addr 172.21.15.12:0/2954441386 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.968+0000 7f1c6fe00640 1 --2- 172.21.15.12:0/2954441386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fbf71261e0 0x55fbf712f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.968+0000 7f1c66a00640 1 -- 172.21.15.12:0/2954441386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fbf712bd40 msgr2=0x55fbf71200c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T23:02:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.968+0000 7f1c66a00640 1 --2- 172.21.15.12:0/2954441386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fbf712bd40 0x55fbf71200c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.968+0000 7f1c66a00640 1 -- 172.21.15.12:0/2954441386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fbf71261e0 msgr2=0x55fbf712f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.968+0000 7f1c66a00640 1 --2- 172.21.15.12:0/2954441386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fbf71261e0 0x55fbf712f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.968+0000 7f1c66a00640 1 -- 172.21.15.12:0/2954441386 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fbf71224c0 con 0x55fbf7126bd0 2024-03-23T23:02:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.968+0000 7f1c6fe00640 1 --2- 172.21.15.12:0/2954441386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fbf71261e0 0x55fbf712f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:02:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.968+0000 7f1c66a00640 1 --2- 172.21.15.12:0/2954441386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf7126bd0 0x55fbf713bd40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1c58001eb0 tx=0x7f1c58009770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ec5fea8f68de222 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.969+0000 7f1c5fe00640 1 -- 172.21.15.12:0/2954441386 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c58011e80 con 0x55fbf7126bd0 2024-03-23T23:02:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.969+0000 7f1c5fe00640 1 -- 172.21.15.12:0/2954441386 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c5800fa60 con 0x55fbf7126bd0 2024-03-23T23:02:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.969+0000 7f1c5fe00640 1 -- 172.21.15.12:0/2954441386 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c580035f0 con 0x55fbf7126bd0 2024-03-23T23:02:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.969+0000 7f1c7919b740 1 -- 172.21.15.12:0/2954441386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf7126bd0 msgr2=0x55fbf713bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.969+0000 7f1c7919b740 1 --2- 172.21.15.12:0/2954441386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf7126bd0 0x55fbf713bd40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1c58001eb0 tx=0x7f1c58009770 comp rx=0 tx=0).stop 2024-03-23T23:02:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.969+0000 7f1c7919b740 1 -- 172.21.15.12:0/2954441386 shutdown_connections 2024-03-23T23:02:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.969+0000 7f1c7919b740 1 --2- 172.21.15.12:0/2954441386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fbf712bd40 0x55fbf71200c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.969+0000 7f1c7919b740 1 --2- 172.21.15.12:0/2954441386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf7126bd0 0x55fbf713bd40 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.969+0000 7f1c7919b740 1 --2- 172.21.15.12:0/2954441386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fbf71261e0 0x55fbf712f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.969+0000 7f1c7919b740 1 -- 172.21.15.12:0/2954441386 >> 172.21.15.12:0/2954441386 conn(0x55fbf7121750 msgr2=0x55fbf712f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.969+0000 7f1c7919b740 1 -- 172.21.15.12:0/2954441386 shutdown_connections 2024-03-23T23:02:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.969+0000 7f1c7919b740 1 -- 172.21.15.12:0/2954441386 wait complete. 2024-03-23T23:02:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.970+0000 7f1c7919b740 1 -- start start 2024-03-23T23:02:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.970+0000 7f1c7919b740 1 Processor -- start 2024-03-23T23:02:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.970+0000 7f1c7919b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fbf71261e0 0x55fbf71399d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.970+0000 7f1c7919b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fbf7126bd0 0x55fbf7139f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.970+0000 7f1c7919b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf712bd40 0x55fbf713af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.970+0000 7f1c7919b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c701a4dd0 con 0x55fbf7126bd0 2024-03-23T23:02:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.970+0000 7f1c7919b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c701a60d0 con 0x55fbf712bd40 2024-03-23T23:02:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.970+0000 7f1c7919b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c701a5410 con 0x55fbf71261e0 2024-03-23T23:02:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.971+0000 7f1c7919b740 10 client.0 _ll_register_callbacks cb 0x7f1c58009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:02:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.971+0000 7f1c6fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fbf71261e0 0x55fbf71399d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.971+0000 7f1c66a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fbf7126bd0 0x55fbf7139f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.971+0000 7f1c6fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fbf71261e0 0x55fbf71399d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:42334/0 (socket says 172.21.15.12:42334) 2024-03-23T23:02:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.971+0000 7f1c6fe00640 1 -- 172.21.15.12:0/87597361 learned_addr learned my addr 172.21.15.12:0/87597361 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.971+0000 7f1c6fe00640 1 --2- 172.21.15.12:0/87597361 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf712bd40 0x55fbf713af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.971+0000 7f1c6fe00640 1 -- 172.21.15.12:0/87597361 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf712bd40 msgr2=0x55fbf713af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.971+0000 7f1c6fe00640 1 --2- 172.21.15.12:0/87597361 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf712bd40 0x55fbf713af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.971+0000 7f1c6fe00640 1 -- 172.21.15.12:0/87597361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fbf7126bd0 msgr2=0x55fbf7139f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.971+0000 7f1c6fe00640 1 --2- 172.21.15.12:0/87597361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fbf7126bd0 0x55fbf7139f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.971+0000 7f1c6fe00640 1 -- 172.21.15.12:0/87597361 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fbf713b510 con 0x55fbf71261e0 2024-03-23T23:02:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.971+0000 7f1c66a00640 1 --2- 172.21.15.12:0/87597361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fbf7126bd0 0x55fbf7139f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:02:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.972+0000 7f1c6fe00640 1 --2- 172.21.15.12:0/87597361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fbf71261e0 0x55fbf71399d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1c68006010 tx=0x7f1c680098a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5dd233b42f39b683 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.972+0000 7f1c5d600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:02:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.972+0000 7f1c5d600640 1 -- 172.21.15.12:0/87597361 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c68016040 con 0x55fbf71261e0 2024-03-23T23:02:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.972+0000 7f1c5d600640 1 -- 172.21.15.12:0/87597361 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1c6801b830 con 0x55fbf71261e0 2024-03-23T23:02:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.972+0000 7f1c7919b740 1 -- 172.21.15.12:0/87597361 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1c5800fa60 con 0x55fbf71261e0 2024-03-23T23:02:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.972+0000 7f1c5d600640 1 -- 172.21.15.12:0/87597361 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c6802b310 con 0x55fbf71261e0 2024-03-23T23:02:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.972+0000 7f1c7919b740 10 client.25489 fetch_fsmap learned FSMap version 23 2024-03-23T23:02:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.972+0000 7f1c7919b740 1 -- 172.21.15.12:0/87597361 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55fbf719c3e0 con 0x55fbf71261e0 2024-03-23T23:02:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.972+0000 7f1c5d600640 1 -- 172.21.15.12:0/87597361 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c5800fa60 con 0x55fbf71261e0 2024-03-23T23:02:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.972+0000 7f1c5d600640 1 -- 172.21.15.12:0/87597361 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f1c58009ce0 con 0x55fbf71261e0 2024-03-23T23:02:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.973+0000 7f1c7919b740 10 client.25489 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:02:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.973+0000 7f1c7919b740 10 client.25489 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:02:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.973+0000 7f1c7919b740 20 client.25489 populate_metadata read hostname 'smithi012' 2024-03-23T23:02:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.973+0000 7f1c7919b740 1 --2- 172.21.15.12:0/87597361 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fbf719ca50 0x55fbf719ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.973+0000 7f1c7919b740 4 client.25489 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:02:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.973+0000 7f1c7919b740 1 -- 172.21.15.12:0/87597361 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55fbf711ca80 con 0x55fbf719ca50 2024-03-23T23:02:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.973+0000 7f1c66a00640 1 --2- 172.21.15.12:0/87597361 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fbf719ca50 0x55fbf719ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.973+0000 7f1c66a00640 1 --2- 172.21.15.12:0/87597361 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fbf719ca50 0x55fbf719ee40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40e014e1eb76b6d3 server_cookie=d1a6b8caeac4b338 in_seq=0 out_seq=0 2024-03-23T23:02:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.973+0000 7f1c5d600640 10 client.25489 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.980+0000 7f1c5d600640 1 -- 172.21.15.12:0/87597361 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55fbf711ca80 con 0x55fbf719ca50 2024-03-23T23:02:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.980+0000 7f1c5d600640 10 client.25489 handle_command_reply: tid=0 2024-03-23T23:02:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.980+0000 7f1c7919b740 1 client.25489 shutdown 2024-03-23T23:02:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.980+0000 7f1c7919b740 1 -- 172.21.15.12:0/87597361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fbf71261e0 msgr2=0x55fbf71399d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.980+0000 7f1c7919b740 1 --2- 172.21.15.12:0/87597361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fbf71261e0 0x55fbf71399d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1c68006010 tx=0x7f1c680098a0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.980+0000 7f1c7919b740 1 -- 172.21.15.12:0/87597361 shutdown_connections 2024-03-23T23:02:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.980+0000 7f1c7919b740 1 --2- 172.21.15.12:0/87597361 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fbf719ca50 0x55fbf719ee40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.980+0000 7f1c7919b740 1 --2- 172.21.15.12:0/87597361 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbf712bd40 0x55fbf713af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.980+0000 7f1c7919b740 1 --2- 172.21.15.12:0/87597361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fbf7126bd0 0x55fbf7139f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.980+0000 7f1c7919b740 1 --2- 172.21.15.12:0/87597361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fbf71261e0 0x55fbf71399d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.980+0000 7f1c5d600640 0 client.25489 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.980+0000 7f1c7919b740 1 -- 172.21.15.12:0/87597361 >> 172.21.15.12:0/87597361 conn(0x55fbf7121750 msgr2=0x55fbf71249b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.980+0000 7f1c7919b740 1 -- 172.21.15.12:0/87597361 shutdown_connections 2024-03-23T23:02:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.980+0000 7f1c7919b740 1 -- 172.21.15.12:0/87597361 wait complete. 2024-03-23T23:02:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.980+0000 7f1c7919b740 20 client.25489 trim_cache size 0 max 16384 2024-03-23T23:02:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:02:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:02:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:02:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:02:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.981+0000 7f1c76400640 1 -- 172.21.15.12:0/3501483096 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1c540a28a0 msgr2=0x7f1c540a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.981+0000 7f1c76400640 1 --2- 172.21.15.12:0/3501483096 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1c540a28a0 0x7f1c540a4d60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1c600bd2a0 tx=0x7f1c600f4690 comp rx=0 tx=0).stop 2024-03-23T23:02:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.981+0000 7f1c76400640 1 -- 172.21.15.12:0/3501483096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c701021e0 msgr2=0x7f1c70197940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.981+0000 7f1c76400640 1 --2- 172.21.15.12:0/3501483096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c701021e0 0x7f1c70197940 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1c60005ab0 tx=0x7f1c6002c6c0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.981+0000 7f1c76400640 1 -- 172.21.15.12:0/3501483096 shutdown_connections 2024-03-23T23:02:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.981+0000 7f1c76400640 1 --2- 172.21.15.12:0/3501483096 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1c540a28a0 0x7f1c540a4d60 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.981+0000 7f1c76400640 1 --2- 172.21.15.12:0/3501483096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c70107440 0x7f1c7019eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.981+0000 7f1c76400640 1 --2- 172.21.15.12:0/3501483096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c70104b10 0x7f1c70197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.981+0000 7f1c76400640 1 --2- 172.21.15.12:0/3501483096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c701021e0 0x7f1c70197940 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.981+0000 7f1c76400640 1 -- 172.21.15.12:0/3501483096 >> 172.21.15.12:0/3501483096 conn(0x7f1c700fbb30 msgr2=0x7f1c70102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.981+0000 7f1c76400640 1 -- 172.21.15.12:0/3501483096 shutdown_connections 2024-03-23T23:02:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:37.981+0000 7f1c76400640 1 -- 172.21.15.12:0/3501483096 wait complete. 2024-03-23T23:02:38.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:38 smithi012 ceph-mon[31695]: pgmap v415: 97 pgs: 97 active+clean; 662 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-03-23T23:02:38.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:38 smithi173 ceph-mon[36116]: pgmap v415: 97 pgs: 97 active+clean; 662 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-03-23T23:02:38.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:38 smithi203 ceph-mon[36693]: pgmap v415: 97 pgs: 97 active+clean; 662 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-03-23T23:02:40.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:40 smithi012 ceph-mon[31695]: pgmap v416: 97 pgs: 97 active+clean; 662 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-23T23:02:40.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:40 smithi173 ceph-mon[36116]: pgmap v416: 97 pgs: 97 active+clean; 662 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-23T23:02:40.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:40 smithi203 ceph-mon[36693]: pgmap v416: 97 pgs: 97 active+clean; 662 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-23T23:02:42.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:42 smithi012 ceph-mon[31695]: pgmap v417: 97 pgs: 97 active+clean; 662 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-03-23T23:02:42.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:42 smithi173 ceph-mon[36116]: pgmap v417: 97 pgs: 97 active+clean; 662 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-03-23T23:02:42.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:42 smithi203 ceph-mon[36693]: pgmap v417: 97 pgs: 97 active+clean; 662 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-03-23T23:02:44.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:44 smithi012 ceph-mon[31695]: pgmap v418: 97 pgs: 97 active+clean; 663 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-23T23:02:44.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:44 smithi173 ceph-mon[36116]: pgmap v418: 97 pgs: 97 active+clean; 663 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-23T23:02:44.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:44 smithi203 ceph-mon[36693]: pgmap v418: 97 pgs: 97 active+clean; 663 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-23T23:02:46.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:46 smithi012 ceph-mon[31695]: pgmap v419: 97 pgs: 97 active+clean; 663 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 852 B/s wr, 8 op/s 2024-03-23T23:02:46.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:46 smithi173 ceph-mon[36116]: pgmap v419: 97 pgs: 97 active+clean; 663 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 852 B/s wr, 8 op/s 2024-03-23T23:02:46.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:46 smithi203 ceph-mon[36693]: pgmap v419: 97 pgs: 97 active+clean; 663 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 852 B/s wr, 8 op/s 2024-03-23T23:02:48.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:48 smithi012 ceph-mon[31695]: pgmap v420: 97 pgs: 97 active+clean; 663 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 852 B/s wr, 7 op/s 2024-03-23T23:02:48.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:48 smithi173 ceph-mon[36116]: pgmap v420: 97 pgs: 97 active+clean; 663 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 852 B/s wr, 7 op/s 2024-03-23T23:02:48.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:48 smithi203 ceph-mon[36693]: pgmap v420: 97 pgs: 97 active+clean; 663 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 852 B/s wr, 7 op/s 2024-03-23T23:02:50.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:50 smithi012 ceph-mon[31695]: pgmap v421: 97 pgs: 97 active+clean; 663 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-23T23:02:50.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:50 smithi173 ceph-mon[36116]: pgmap v421: 97 pgs: 97 active+clean; 663 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-23T23:02:50.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:50 smithi203 ceph-mon[36693]: pgmap v421: 97 pgs: 97 active+clean; 663 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-23T23:02:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.128+0000 7fa8a1600640 1 -- 172.21.15.12:0/742318877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa89c100130 msgr2=0x7fa89c109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.128+0000 7fa8a1600640 1 --2- 172.21.15.12:0/742318877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa89c100130 0x7fa89c109ce0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa894009ca0 tx=0x7fa89402c100 comp rx=0 tx=0).stop 2024-03-23T23:02:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.128+0000 7fa8a1600640 1 -- 172.21.15.12:0/742318877 shutdown_connections 2024-03-23T23:02:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.128+0000 7fa8a1600640 1 --2- 172.21.15.12:0/742318877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa89c100130 0x7fa89c109ce0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.129 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.128+0000 7fa8a1600640 1 --2- 172.21.15.12:0/742318877 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa89c0ff770 0x7fa89c0ffbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.129 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.128+0000 7fa8a1600640 1 --2- 172.21.15.12:0/742318877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa89c102d00 0x7fa89c103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.129 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.128+0000 7fa8a1600640 1 -- 172.21.15.12:0/742318877 >> 172.21.15.12:0/742318877 conn(0x7fa89c0fb2f0 msgr2=0x7fa89c0fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:51.129 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.128+0000 7fa8a1600640 1 -- 172.21.15.12:0/742318877 shutdown_connections 2024-03-23T23:02:51.129 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.128+0000 7fa8a1600640 1 -- 172.21.15.12:0/742318877 wait complete. 2024-03-23T23:02:51.129 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.129+0000 7fa8a1600640 1 Processor -- start 2024-03-23T23:02:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.129+0000 7fa8a1600640 1 -- start start 2024-03-23T23:02:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.129+0000 7fa8a1600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa89c0ff770 0x7fa89c199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.130+0000 7fa8a1600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa89c100130 0x7fa89c19a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.130+0000 7fa8a1600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa89c102d00 0x7fa89c1a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.130+0000 7fa8a1600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c10d360 con 0x7fa89c100130 2024-03-23T23:02:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.130+0000 7fa8a1600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c10d1e0 con 0x7fa89c102d00 2024-03-23T23:02:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.130+0000 7fa8a1600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c10d4e0 con 0x7fa89c0ff770 2024-03-23T23:02:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.130+0000 7fa89aa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa89c0ff770 0x7fa89c199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.130+0000 7fa89a000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa89c100130 0x7fa89c19a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.130+0000 7fa89aa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa89c0ff770 0x7fa89c199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:46484/0 (socket says 172.21.15.12:46484) 2024-03-23T23:02:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.130+0000 7fa89aa00640 1 -- 172.21.15.12:0/4209549575 learned_addr learned my addr 172.21.15.12:0/4209549575 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.130+0000 7fa89b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa89c102d00 0x7fa89c1a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.130+0000 7fa89aa00640 1 -- 172.21.15.12:0/4209549575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa89c102d00 msgr2=0x7fa89c1a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.130+0000 7fa89aa00640 1 --2- 172.21.15.12:0/4209549575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa89c102d00 0x7fa89c1a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.136 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.130+0000 7fa89aa00640 1 -- 172.21.15.12:0/4209549575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa89c100130 msgr2=0x7fa89c19a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:51.136 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.130+0000 7fa89aa00640 1 --2- 172.21.15.12:0/4209549575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa89c100130 0x7fa89c19a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.136 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.130+0000 7fa89aa00640 1 -- 172.21.15.12:0/4209549575 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa89c1a1710 con 0x7fa89c0ff770 2024-03-23T23:02:51.136 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.130+0000 7fa89a000640 1 --2- 172.21.15.12:0/4209549575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa89c100130 0x7fa89c19a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:02:51.136 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.130+0000 7fa89b400640 1 --2- 172.21.15.12:0/4209549575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa89c102d00 0x7fa89c1a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:02:51.136 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.131+0000 7fa89aa00640 1 --2- 172.21.15.12:0/4209549575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa89c0ff770 0x7fa89c199af0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa88c00e9c0 tx=0x7fa88c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:51.137 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.131+0000 7fa893400640 1 -- 172.21.15.12:0/4209549575 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c00cde0 con 0x7fa89c0ff770 2024-03-23T23:02:51.137 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.131+0000 7fa893400640 1 -- 172.21.15.12:0/4209549575 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa88c010910 con 0x7fa89c0ff770 2024-03-23T23:02:51.137 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.131+0000 7fa893400640 1 -- 172.21.15.12:0/4209549575 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c018a00 con 0x7fa89c0ff770 2024-03-23T23:02:51.137 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.131+0000 7fa8a1600640 1 -- 172.21.15.12:0/4209549575 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa89c1a1a00 con 0x7fa89c0ff770 2024-03-23T23:02:51.137 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.131+0000 7fa8a1600640 1 -- 172.21.15.12:0/4209549575 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa89c1a1fa0 con 0x7fa89c0ff770 2024-03-23T23:02:51.137 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.134+0000 7fa893400640 1 -- 172.21.15.12:0/4209549575 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa88c01f020 con 0x7fa89c0ff770 2024-03-23T23:02:51.137 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.134+0000 7fa8a43ab740 1 Processor -- start 2024-03-23T23:02:51.137 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.134+0000 7fa8a43ab740 1 -- start start 2024-03-23T23:02:51.137 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.135+0000 7fa8a43ab740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56110a8471e0 0x56110a847600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:51.137 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.135+0000 7fa8a43ab740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56110a847bd0 0x56110a84c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:51.137 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.135+0000 7fa8a43ab740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a84cd40 0x56110a84f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:51.137 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.135+0000 7fa8a43ab740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56110a80a3b0 con 0x56110a847bd0 2024-03-23T23:02:51.138 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.135+0000 7fa8a43ab740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56110a7ac830 con 0x56110a84cd40 2024-03-23T23:02:51.138 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.135+0000 7fa8a43ab740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x56110a7555d0 con 0x56110a8471e0 2024-03-23T23:02:51.138 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.135+0000 7fa89aa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56110a847bd0 0x56110a84c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:51.138 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.135+0000 7fa89a000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a84cd40 0x56110a84f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:51.138 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.135+0000 7fa89a000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a84cd40 0x56110a84f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:45210/0 (socket says 172.21.15.12:45210) 2024-03-23T23:02:51.138 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.135+0000 7fa89a000640 1 -- 172.21.15.12:0/3681395604 learned_addr learned my addr 172.21.15.12:0/3681395604 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:51.138 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.135+0000 7fa89b400640 1 --2- 172.21.15.12:0/3681395604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56110a8471e0 0x56110a847600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.135+0000 7fa89a000640 1 -- 172.21.15.12:0/3681395604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56110a8471e0 msgr2=0x56110a847600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.135+0000 7fa89a000640 1 --2- 172.21.15.12:0/3681395604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56110a8471e0 0x56110a847600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.135+0000 7fa89a000640 1 -- 172.21.15.12:0/3681395604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56110a847bd0 msgr2=0x56110a84c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.135+0000 7fa89a000640 1 --2- 172.21.15.12:0/3681395604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56110a847bd0 0x56110a84c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.135+0000 7fa89a000640 1 -- 172.21.15.12:0/3681395604 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56110a76d2c0 con 0x56110a84cd40 2024-03-23T23:02:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.136+0000 7fa89a000640 1 --2- 172.21.15.12:0/3681395604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a84cd40 0x56110a84f1d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa8880114f0 tx=0x7fa8880119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c79c199b143d150b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.136+0000 7fa893400640 1 --2- 172.21.15.12:0/4209549575 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa8800a2920 0x7fa8800a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.136+0000 7fa887400640 1 -- 172.21.15.12:0/3681395604 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa888004020 con 0x56110a84cd40 2024-03-23T23:02:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.136+0000 7fa887400640 1 -- 172.21.15.12:0/3681395604 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8880041c0 con 0x56110a84cd40 2024-03-23T23:02:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.136+0000 7fa887400640 1 -- 172.21.15.12:0/3681395604 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88800f950 con 0x56110a84cd40 2024-03-23T23:02:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.136+0000 7fa893400640 1 -- 172.21.15.12:0/4209549575 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fa88c0cec10 con 0x7fa89c0ff770 2024-03-23T23:02:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.136+0000 7fa89b400640 1 --2- 172.21.15.12:0/4209549575 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa8800a2920 0x7fa8800a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.137+0000 7fa8a43ab740 1 -- 172.21.15.12:0/3681395604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a84cd40 msgr2=0x56110a84f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.137+0000 7fa8a43ab740 1 --2- 172.21.15.12:0/3681395604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a84cd40 0x56110a84f1d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa8880114f0 tx=0x7fa8880119b0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.137+0000 7fa8a43ab740 1 -- 172.21.15.12:0/3681395604 shutdown_connections 2024-03-23T23:02:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.137+0000 7fa8a43ab740 1 --2- 172.21.15.12:0/3681395604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a84cd40 0x56110a84f1d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.137+0000 7fa8a43ab740 1 --2- 172.21.15.12:0/3681395604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56110a847bd0 0x56110a84c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.137+0000 7fa8a43ab740 1 --2- 172.21.15.12:0/3681395604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56110a8471e0 0x56110a847600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.145 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.137+0000 7fa89b400640 1 --2- 172.21.15.12:0/4209549575 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa8800a2920 0x7fa8800a4de0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa89402c640 tx=0x7fa894032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:51.145 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.137+0000 7fa8a43ab740 1 -- 172.21.15.12:0/3681395604 >> 172.21.15.12:0/3681395604 conn(0x56110a842750 msgr2=0x56110a842b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:51.145 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.137+0000 7fa8a43ab740 1 -- 172.21.15.12:0/3681395604 shutdown_connections 2024-03-23T23:02:51.145 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.137+0000 7fa8a43ab740 1 -- 172.21.15.12:0/3681395604 wait complete. 2024-03-23T23:02:51.145 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.138+0000 7fa8a43ab740 1 -- start start 2024-03-23T23:02:51.145 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.138+0000 7fa8a43ab740 1 Processor -- start 2024-03-23T23:02:51.145 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.139+0000 7fa8a43ab740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56110a8471e0 0x56110a851330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:51.145 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.139+0000 7fa8a43ab740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a847bd0 0x56110a851870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:51.146 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.139+0000 7fa8a43ab740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56110a84cd40 0x56110a8588b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:51.146 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.139+0000 7fa8a43ab740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56110a768a20 con 0x56110a8471e0 2024-03-23T23:02:51.146 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.139+0000 7fa8a43ab740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56110a7a88d0 con 0x56110a847bd0 2024-03-23T23:02:51.146 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.139+0000 7fa8a43ab740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x56110a84fcf0 con 0x56110a84cd40 2024-03-23T23:02:51.146 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.139+0000 7fa8a43ab740 10 client.0 _ll_register_callbacks cb 0x56110a7a51c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:02:51.146 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.139+0000 7fa89aa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56110a8471e0 0x56110a851330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:51.146 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.139+0000 7fa89a000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a847bd0 0x56110a851870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:51.146 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.139+0000 7fa89aa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56110a8471e0 0x56110a851330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52988/0 (socket says 172.21.15.12:52988) 2024-03-23T23:02:51.146 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.139+0000 7fa89a000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a847bd0 0x56110a851870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:45220/0 (socket says 172.21.15.12:45220) 2024-03-23T23:02:51.146 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.139+0000 7fa89aa00640 1 -- 172.21.15.12:0/999390929 learned_addr learned my addr 172.21.15.12:0/999390929 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:51.146 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.139+0000 7fa89b400640 1 --2- 172.21.15.12:0/999390929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56110a84cd40 0x56110a8588b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:51.146 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.139+0000 7fa89a000640 1 -- 172.21.15.12:0/999390929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56110a84cd40 msgr2=0x56110a8588b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:51.147 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.139+0000 7fa89a000640 1 --2- 172.21.15.12:0/999390929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56110a84cd40 0x56110a8588b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.147 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.139+0000 7fa89a000640 1 -- 172.21.15.12:0/999390929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56110a8471e0 msgr2=0x56110a851330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:51.147 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.139+0000 7fa89a000640 1 --2- 172.21.15.12:0/999390929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56110a8471e0 0x56110a851330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.147 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.139+0000 7fa89a000640 1 -- 172.21.15.12:0/999390929 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56110a4c1210 con 0x56110a847bd0 2024-03-23T23:02:51.147 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.140+0000 7fa89a000640 1 --2- 172.21.15.12:0/999390929 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a847bd0 0x56110a851870 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa888011eb0 tx=0x7fa888003c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=253b1b1901f449b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:51.147 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.140+0000 7fa87be00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:02:51.147 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.140+0000 7fa87be00640 1 -- 172.21.15.12:0/999390929 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa888028020 con 0x56110a847bd0 2024-03-23T23:02:51.147 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.140+0000 7fa8a43ab740 1 -- 172.21.15.12:0/999390929 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa8880041c0 con 0x56110a847bd0 2024-03-23T23:02:51.147 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.140+0000 7fa87be00640 1 -- 172.21.15.12:0/999390929 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fa888016040 con 0x56110a847bd0 2024-03-23T23:02:51.147 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.140+0000 7fa87be00640 1 -- 172.21.15.12:0/999390929 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa888004000 con 0x56110a847bd0 2024-03-23T23:02:51.147 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.140+0000 7fa8a43ab740 10 client.25532 fetch_fsmap learned FSMap version 23 2024-03-23T23:02:51.147 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.140+0000 7fa8a43ab740 1 -- 172.21.15.12:0/999390929 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56110a58d620 con 0x56110a847bd0 2024-03-23T23:02:51.147 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.141+0000 7fa87be00640 1 -- 172.21.15.12:0/999390929 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8880041c0 con 0x56110a847bd0 2024-03-23T23:02:51.147 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.141+0000 7fa87be00640 1 -- 172.21.15.12:0/999390929 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fa88800f650 con 0x56110a847bd0 2024-03-23T23:02:51.147 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.141+0000 7fa8a43ab740 10 client.25532 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:02:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.141+0000 7fa8a43ab740 10 client.25532 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:02:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.141+0000 7fa8a43ab740 20 client.25532 populate_metadata read hostname 'smithi012' 2024-03-23T23:02:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.141+0000 7fa8a43ab740 1 --2- 172.21.15.12:0/999390929 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56110a85ceb0 0x56110a85f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.141+0000 7fa8a43ab740 4 client.25532 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:02:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.141+0000 7fa8a43ab740 1 -- 172.21.15.12:0/999390929 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56110a7b9d70 con 0x56110a85ceb0 2024-03-23T23:02:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.141+0000 7fa89aa00640 1 --2- 172.21.15.12:0/999390929 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56110a85ceb0 0x56110a85f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.142+0000 7fa89aa00640 1 --2- 172.21.15.12:0/999390929 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56110a85ceb0 0x56110a85f2a0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58bc3225741110f5 server_cookie=b4488a2dbbb0416a in_seq=0 out_seq=0 2024-03-23T23:02:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.142+0000 7fa87be00640 10 client.25532 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.142+0000 7fa87be00640 1 -- 172.21.15.12:0/999390929 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56110a7b9d70 con 0x56110a85ceb0 2024-03-23T23:02:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.142+0000 7fa87be00640 10 client.25532 handle_command_reply: tid=0 2024-03-23T23:02:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.143+0000 7fa8a43ab740 1 client.25532 shutdown 2024-03-23T23:02:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.143+0000 7fa8a43ab740 1 -- 172.21.15.12:0/999390929 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a847bd0 msgr2=0x56110a851870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:51.149 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.143+0000 7fa8a43ab740 1 --2- 172.21.15.12:0/999390929 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a847bd0 0x56110a851870 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa888011eb0 tx=0x7fa888003c80 comp rx=0 tx=0).stop 2024-03-23T23:02:51.149 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.143+0000 7fa8a43ab740 1 -- 172.21.15.12:0/999390929 shutdown_connections 2024-03-23T23:02:51.149 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.143+0000 7fa8a43ab740 1 --2- 172.21.15.12:0/999390929 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56110a85ceb0 0x56110a85f2a0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.149 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.143+0000 7fa8a43ab740 1 --2- 172.21.15.12:0/999390929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56110a84cd40 0x56110a8588b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.149 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.143+0000 7fa8a43ab740 1 --2- 172.21.15.12:0/999390929 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a847bd0 0x56110a851870 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.149 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.143+0000 7fa87be00640 0 client.25532 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:51.149 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.143+0000 7fa8a43ab740 1 --2- 172.21.15.12:0/999390929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56110a8471e0 0x56110a851330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.149 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.143+0000 7fa8a43ab740 1 -- 172.21.15.12:0/999390929 >> 172.21.15.12:0/999390929 conn(0x56110a842750 msgr2=0x56110a841950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:51.149 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.143+0000 7fa8a43ab740 1 -- 172.21.15.12:0/999390929 shutdown_connections 2024-03-23T23:02:51.149 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.143+0000 7fa8a43ab740 1 -- 172.21.15.12:0/999390929 wait complete. 2024-03-23T23:02:51.149 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.143+0000 7fa8a43ab740 20 client.25532 trim_cache size 0 max 16384 2024-03-23T23:02:51.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.159+0000 7fa8a43ab740 1 Processor -- start 2024-03-23T23:02:51.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.159+0000 7fa8a43ab740 1 -- start start 2024-03-23T23:02:51.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.159+0000 7fa8a43ab740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a8471e0 0x56110a8482c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:51.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.159+0000 7fa8a43ab740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56110a847bd0 0x56110a858870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:51.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.159+0000 7fa8a43ab740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56110a84cd40 0x56110a848e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:51.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.159+0000 7fa8a43ab740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c1ad0e0 con 0x56110a847bd0 2024-03-23T23:02:51.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.159+0000 7fa8a43ab740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c1aabe0 con 0x56110a8471e0 2024-03-23T23:02:51.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.159+0000 7fa8a43ab740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c1a8c80 con 0x56110a84cd40 2024-03-23T23:02:51.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.159+0000 7fa89aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a8471e0 0x56110a8482c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:51.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.159+0000 7fa89aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a8471e0 0x56110a8482c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:45234/0 (socket says 172.21.15.12:45234) 2024-03-23T23:02:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.159+0000 7fa89aa00640 1 -- 172.21.15.12:0/3595478543 learned_addr learned my addr 172.21.15.12:0/3595478543 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.159+0000 7fa89b400640 1 --2- 172.21.15.12:0/3595478543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56110a84cd40 0x56110a848e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.160+0000 7fa89a000640 1 --2- 172.21.15.12:0/3595478543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56110a847bd0 0x56110a858870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.160+0000 7fa89aa00640 1 -- 172.21.15.12:0/3595478543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56110a84cd40 msgr2=0x56110a848e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.160+0000 7fa89aa00640 1 --2- 172.21.15.12:0/3595478543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56110a84cd40 0x56110a848e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.160+0000 7fa89aa00640 1 -- 172.21.15.12:0/3595478543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56110a847bd0 msgr2=0x56110a858870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.160+0000 7fa89aa00640 1 --2- 172.21.15.12:0/3595478543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56110a847bd0 0x56110a858870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.160+0000 7fa89aa00640 1 -- 172.21.15.12:0/3595478543 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56110a85f610 con 0x56110a8471e0 2024-03-23T23:02:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.160+0000 7fa89a000640 1 --2- 172.21.15.12:0/3595478543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56110a847bd0 0x56110a858870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:02:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.160+0000 7fa89aa00640 1 --2- 172.21.15.12:0/3595478543 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a8471e0 0x56110a8482c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa88c09b260 tx=0x7fa88c0cc590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a817bd4d3cbdd8de server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.160+0000 7fa886a00640 1 -- 172.21.15.12:0/3595478543 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c09b0a0 con 0x56110a8471e0 2024-03-23T23:02:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.160+0000 7fa886a00640 1 -- 172.21.15.12:0/3595478543 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa88c0d3040 con 0x56110a8471e0 2024-03-23T23:02:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.160+0000 7fa886a00640 1 -- 172.21.15.12:0/3595478543 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c09aec0 con 0x56110a8471e0 2024-03-23T23:02:51.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.161+0000 7fa8a43ab740 1 -- 172.21.15.12:0/3595478543 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a8471e0 msgr2=0x56110a8482c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:51.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.161+0000 7fa8a43ab740 1 --2- 172.21.15.12:0/3595478543 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a8471e0 0x56110a8482c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa88c09b260 tx=0x7fa88c0cc590 comp rx=0 tx=0).stop 2024-03-23T23:02:51.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.161+0000 7fa8a43ab740 1 -- 172.21.15.12:0/3595478543 shutdown_connections 2024-03-23T23:02:51.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.161+0000 7fa8a43ab740 1 --2- 172.21.15.12:0/3595478543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56110a84cd40 0x56110a848e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.161+0000 7fa8a43ab740 1 --2- 172.21.15.12:0/3595478543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56110a847bd0 0x56110a858870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.161+0000 7fa8a43ab740 1 --2- 172.21.15.12:0/3595478543 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a8471e0 0x56110a8482c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.161+0000 7fa8a43ab740 1 -- 172.21.15.12:0/3595478543 >> 172.21.15.12:0/3595478543 conn(0x56110a842750 msgr2=0x56110a842b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.161+0000 7fa8a43ab740 1 -- 172.21.15.12:0/3595478543 shutdown_connections 2024-03-23T23:02:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.161+0000 7fa8a43ab740 1 -- 172.21.15.12:0/3595478543 wait complete. 2024-03-23T23:02:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.162+0000 7fa8a43ab740 1 -- start start 2024-03-23T23:02:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.162+0000 7fa8a43ab740 1 Processor -- start 2024-03-23T23:02:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.162+0000 7fa8a43ab740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56110a8471e0 0x56110a8560c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.162+0000 7fa8a43ab740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56110a847bd0 0x56110a856600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.162+0000 7fa8a43ab740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a84cd40 0x56110a857610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.162+0000 7fa8a43ab740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c1a9280 con 0x56110a8471e0 2024-03-23T23:02:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.162+0000 7fa8a43ab740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c1aa580 con 0x56110a84cd40 2024-03-23T23:02:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.162+0000 7fa8a43ab740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c1a98c0 con 0x56110a847bd0 2024-03-23T23:02:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.162+0000 7fa8a43ab740 10 client.0 _ll_register_callbacks cb 0x7fa88c0d2520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:02:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.163+0000 7fa89aa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56110a8471e0 0x56110a8560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.163+0000 7fa89aa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56110a8471e0 0x56110a8560c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53018/0 (socket says 172.21.15.12:53018) 2024-03-23T23:02:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.163+0000 7fa89aa00640 1 -- 172.21.15.12:0/1421693595 learned_addr learned my addr 172.21.15.12:0/1421693595 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:02:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.163+0000 7fa89b400640 1 --2- 172.21.15.12:0/1421693595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a84cd40 0x56110a857610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.163+0000 7fa89a000640 1 --2- 172.21.15.12:0/1421693595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56110a847bd0 0x56110a856600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.163+0000 7fa89aa00640 1 -- 172.21.15.12:0/1421693595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56110a847bd0 msgr2=0x56110a856600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.163+0000 7fa89aa00640 1 --2- 172.21.15.12:0/1421693595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56110a847bd0 0x56110a856600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.163+0000 7fa89aa00640 1 -- 172.21.15.12:0/1421693595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a84cd40 msgr2=0x56110a857610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:02:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.163+0000 7fa89aa00640 1 --2- 172.21.15.12:0/1421693595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a84cd40 0x56110a857610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.163+0000 7fa89aa00640 1 -- 172.21.15.12:0/1421693595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56110a857c00 con 0x56110a8471e0 2024-03-23T23:02:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.163+0000 7fa89aa00640 1 --2- 172.21.15.12:0/1421693595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56110a8471e0 0x56110a8560c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa88c00c6a0 tx=0x7fa88c009490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa7713165533db0e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:02:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.163+0000 7fa87be00640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:02:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.164+0000 7fa87be00640 1 -- 172.21.15.12:0/1421693595 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c09b0a0 con 0x56110a8471e0 2024-03-23T23:02:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.164+0000 7fa87be00640 1 -- 172.21.15.12:0/1421693595 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fa88c097bd0 con 0x56110a8471e0 2024-03-23T23:02:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.164+0000 7fa8a43ab740 1 -- 172.21.15.12:0/1421693595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa88c0d3040 con 0x56110a8471e0 2024-03-23T23:02:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.164+0000 7fa87be00640 1 -- 172.21.15.12:0/1421693595 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa88c09f6b0 con 0x56110a8471e0 2024-03-23T23:02:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.164+0000 7fa8a43ab740 10 client.15402 fetch_fsmap learned FSMap version 23 2024-03-23T23:02:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.164+0000 7fa8a43ab740 1 -- 172.21.15.12:0/1421693595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56110a8b87e0 con 0x56110a8471e0 2024-03-23T23:02:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.164+0000 7fa87be00640 1 -- 172.21.15.12:0/1421693595 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa88c0d3040 con 0x56110a8471e0 2024-03-23T23:02:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.164+0000 7fa87be00640 1 -- 172.21.15.12:0/1421693595 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fa88c0a3870 con 0x56110a8471e0 2024-03-23T23:02:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.164+0000 7fa8a43ab740 10 client.15402 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:02:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.164+0000 7fa8a43ab740 10 client.15402 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:02:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.164+0000 7fa8a43ab740 20 client.15402 populate_metadata read hostname 'smithi012' 2024-03-23T23:02:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.164+0000 7fa8a43ab740 1 --2- 172.21.15.12:0/1421693595 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56110a8b90d0 0x56110a8bb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:02:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.164+0000 7fa8a43ab740 4 client.15402 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:02:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.164+0000 7fa8a43ab740 1 -- 172.21.15.12:0/1421693595 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56110a7b40c0 con 0x56110a8b90d0 2024-03-23T23:02:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.165+0000 7fa89a000640 1 --2- 172.21.15.12:0/1421693595 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56110a8b90d0 0x56110a8bb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:02:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.165+0000 7fa89a000640 1 --2- 172.21.15.12:0/1421693595 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56110a8b90d0 0x56110a8bb4c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4a836f7ccc498c8 server_cookie=a04dd72c2b9e6857 in_seq=0 out_seq=0 2024-03-23T23:02:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.165+0000 7fa87be00640 10 client.15402 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.168+0000 7fa87be00640 1 -- 172.21.15.12:0/1421693595 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56110a7b40c0 con 0x56110a8b90d0 2024-03-23T23:02:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.168+0000 7fa87be00640 10 client.15402 handle_command_reply: tid=0 2024-03-23T23:02:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.168+0000 7fa8a43ab740 1 client.15402 shutdown 2024-03-23T23:02:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.168+0000 7fa8a43ab740 1 -- 172.21.15.12:0/1421693595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56110a8471e0 msgr2=0x56110a8560c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.168+0000 7fa8a43ab740 1 --2- 172.21.15.12:0/1421693595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56110a8471e0 0x56110a8560c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa88c00c6a0 tx=0x7fa88c009490 comp rx=0 tx=0).stop 2024-03-23T23:02:51.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.169+0000 7fa8a43ab740 1 -- 172.21.15.12:0/1421693595 shutdown_connections 2024-03-23T23:02:51.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.169+0000 7fa8a43ab740 1 --2- 172.21.15.12:0/1421693595 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56110a8b90d0 0x56110a8bb4c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.169+0000 7fa8a43ab740 1 --2- 172.21.15.12:0/1421693595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56110a84cd40 0x56110a857610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.169+0000 7fa8a43ab740 1 --2- 172.21.15.12:0/1421693595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56110a847bd0 0x56110a856600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.169+0000 7fa8a43ab740 1 --2- 172.21.15.12:0/1421693595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56110a8471e0 0x56110a8560c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.169+0000 7fa87be00640 0 client.15402 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:02:51.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.169+0000 7fa8a43ab740 1 -- 172.21.15.12:0/1421693595 >> 172.21.15.12:0/1421693595 conn(0x56110a842750 msgr2=0x56110a8482c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:51.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.169+0000 7fa8a43ab740 1 -- 172.21.15.12:0/1421693595 shutdown_connections 2024-03-23T23:02:51.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.169+0000 7fa8a43ab740 1 -- 172.21.15.12:0/1421693595 wait complete. 2024-03-23T23:02:51.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.169+0000 7fa8a43ab740 20 client.15402 trim_cache size 0 max 16384 2024-03-23T23:02:51.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:02:51.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:02:51.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:02:51.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:02:51.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.170+0000 7fa8a1600640 1 -- 172.21.15.12:0/4209549575 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa8800a2920 msgr2=0x7fa8800a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:51.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.170+0000 7fa8a1600640 1 --2- 172.21.15.12:0/4209549575 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa8800a2920 0x7fa8800a4de0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa89402c640 tx=0x7fa894032590 comp rx=0 tx=0).stop 2024-03-23T23:02:51.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.170+0000 7fa8a1600640 1 -- 172.21.15.12:0/4209549575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa89c0ff770 msgr2=0x7fa89c199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:02:51.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.170+0000 7fa8a1600640 1 --2- 172.21.15.12:0/4209549575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa89c0ff770 0x7fa89c199af0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa88c00e9c0 tx=0x7fa88c00ee80 comp rx=0 tx=0).stop 2024-03-23T23:02:51.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.170+0000 7fa8a1600640 1 -- 172.21.15.12:0/4209549575 shutdown_connections 2024-03-23T23:02:51.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.170+0000 7fa8a1600640 1 --2- 172.21.15.12:0/4209549575 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa8800a2920 0x7fa8800a4de0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.170+0000 7fa8a1600640 1 --2- 172.21.15.12:0/4209549575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa89c102d00 0x7fa89c1a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.170+0000 7fa8a1600640 1 --2- 172.21.15.12:0/4209549575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa89c100130 0x7fa89c19a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.170+0000 7fa8a1600640 1 --2- 172.21.15.12:0/4209549575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa89c0ff770 0x7fa89c199af0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:02:51.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.170+0000 7fa8a1600640 1 -- 172.21.15.12:0/4209549575 >> 172.21.15.12:0/4209549575 conn(0x7fa89c0fb2f0 msgr2=0x7fa89c100880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:02:51.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.170+0000 7fa8a1600640 1 -- 172.21.15.12:0/4209549575 shutdown_connections 2024-03-23T23:02:51.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:02:51.170+0000 7fa8a1600640 1 -- 172.21.15.12:0/4209549575 wait complete. 2024-03-23T23:02:52.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:52 smithi012 ceph-mon[31695]: pgmap v422: 97 pgs: 97 active+clean; 663 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-23T23:02:52.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:52 smithi173 ceph-mon[36116]: pgmap v422: 97 pgs: 97 active+clean; 663 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-23T23:02:52.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:52 smithi203 ceph-mon[36693]: pgmap v422: 97 pgs: 97 active+clean; 663 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-23T23:02:54.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:54 smithi012 ceph-mon[31695]: pgmap v423: 97 pgs: 97 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 852 B/s wr, 7 op/s 2024-03-23T23:02:54.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:54 smithi173 ceph-mon[36116]: pgmap v423: 97 pgs: 97 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 852 B/s wr, 7 op/s 2024-03-23T23:02:54.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:54 smithi203 ceph-mon[36693]: pgmap v423: 97 pgs: 97 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 852 B/s wr, 7 op/s 2024-03-23T23:02:56.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:56 smithi012 ceph-mon[31695]: pgmap v424: 97 pgs: 97 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:02:56.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:56 smithi173 ceph-mon[36116]: pgmap v424: 97 pgs: 97 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:02:56.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:56 smithi203 ceph-mon[36693]: pgmap v424: 97 pgs: 97 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:02:58.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:58 smithi012 ceph-mon[31695]: pgmap v425: 97 pgs: 97 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-23T23:02:58.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:58 smithi173 ceph-mon[36116]: pgmap v425: 97 pgs: 97 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-23T23:02:58.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:58 smithi203 ceph-mon[36693]: pgmap v425: 97 pgs: 97 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-23T23:02:59.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:59 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:02:59.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:59 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:02:59.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:02:59 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:02:59.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:59 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:02:59.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:59 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:02:59.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:02:59 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:02:59.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:59 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:02:59.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:59 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:02:59.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:02:59 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:03:00.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:00 smithi012 ceph-mon[31695]: pgmap v426: 97 pgs: 97 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:03:00.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:00 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:03:00.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:00 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:03:00.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:00 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:03:00.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:00 smithi173 ceph-mon[36116]: pgmap v426: 97 pgs: 97 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:03:00.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:00 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:03:00.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:00 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:03:00.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:00 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:03:00.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:00 smithi203 ceph-mon[36693]: pgmap v426: 97 pgs: 97 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:03:00.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:00 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:03:00.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:00 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:03:00.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:00 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:03:02.060 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:03:02.060 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:03:02.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.268+0000 7f1cb8000640 1 -- 172.21.15.12:0/1718244989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cb01016d0 msgr2=0x7f1cb0101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:02.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.268+0000 7f1cb8000640 1 --2- 172.21.15.12:0/1718244989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cb01016d0 0x7f1cb0101af0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1ca4009900 tx=0x7f1ca402bc30 comp rx=0 tx=0).stop 2024-03-23T23:03:02.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.268+0000 7f1cb8000640 1 -- 172.21.15.12:0/1718244989 shutdown_connections 2024-03-23T23:03:02.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.268+0000 7f1cb8000640 1 --2- 172.21.15.12:0/1718244989 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1cb0103280 0x7f1cb0109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.268+0000 7f1cb8000640 1 --2- 172.21.15.12:0/1718244989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cb01028c0 0x7f1cb0102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.268+0000 7f1cb8000640 1 --2- 172.21.15.12:0/1718244989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cb01016d0 0x7f1cb0101af0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.268+0000 7f1cb8000640 1 -- 172.21.15.12:0/1718244989 >> 172.21.15.12:0/1718244989 conn(0x7f1cb00fcc40 msgr2=0x7f1cb00ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:02.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.268+0000 7f1cb8000640 1 -- 172.21.15.12:0/1718244989 shutdown_connections 2024-03-23T23:03:02.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.268+0000 7f1cb8000640 1 -- 172.21.15.12:0/1718244989 wait complete. 2024-03-23T23:03:02.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.269+0000 7f1cb8000640 1 Processor -- start 2024-03-23T23:03:02.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.269+0000 7f1cb8000640 1 -- start start 2024-03-23T23:03:02.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.269+0000 7f1cb8000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cb01016d0 0x7f1cb0199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:02.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.269+0000 7f1cb8000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1cb01028c0 0x7f1cb019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:02.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.269+0000 7f1cb8000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cb0103280 0x7f1cb01a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:02.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.270+0000 7f1cb8000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb010d3a0 con 0x7f1cb01016d0 2024-03-23T23:03:02.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.270+0000 7f1cb8000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb010d220 con 0x7f1cb01028c0 2024-03-23T23:03:02.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.270+0000 7f1cb8000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb010d520 con 0x7f1cb0103280 2024-03-23T23:03:02.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.270+0000 7f1cb4c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1cb01028c0 0x7f1cb019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:02.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.270+0000 7f1cb4c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1cb01028c0 0x7f1cb019a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:35978/0 (socket says 172.21.15.12:35978) 2024-03-23T23:03:02.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.270+0000 7f1cb4c00640 1 -- 172.21.15.12:0/2077246131 learned_addr learned my addr 172.21.15.12:0/2077246131 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:02.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.270+0000 7f1cb6000640 1 --2- 172.21.15.12:0/2077246131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cb0103280 0x7f1cb01a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:02.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.270+0000 7f1cb5600640 1 --2- 172.21.15.12:0/2077246131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cb01016d0 0x7f1cb0199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:02.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.270+0000 7f1cb4c00640 1 -- 172.21.15.12:0/2077246131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cb0103280 msgr2=0x7f1cb01a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:02.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.270+0000 7f1cb4c00640 1 --2- 172.21.15.12:0/2077246131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cb0103280 0x7f1cb01a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.270+0000 7f1cb4c00640 1 -- 172.21.15.12:0/2077246131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cb01016d0 msgr2=0x7f1cb0199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:02.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.270+0000 7f1cb4c00640 1 --2- 172.21.15.12:0/2077246131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cb01016d0 0x7f1cb0199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.270+0000 7f1cb4c00640 1 -- 172.21.15.12:0/2077246131 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb01a1850 con 0x7f1cb01028c0 2024-03-23T23:03:02.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.270+0000 7f1cb6000640 1 --2- 172.21.15.12:0/2077246131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cb0103280 0x7f1cb01a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:02.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.271+0000 7f1cb4c00640 1 --2- 172.21.15.12:0/2077246131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1cb01028c0 0x7f1cb019a180 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00e9f0 tx=0x7f1c9c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:02.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.271+0000 7f1cae000640 1 -- 172.21.15.12:0/2077246131 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c00cd80 con 0x7f1cb01028c0 2024-03-23T23:03:02.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.271+0000 7f1cb8000640 1 -- 172.21.15.12:0/2077246131 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb01a1b40 con 0x7f1cb01028c0 2024-03-23T23:03:02.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.271+0000 7f1cae000640 1 -- 172.21.15.12:0/2077246131 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c9c010910 con 0x7f1cb01028c0 2024-03-23T23:03:02.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.271+0000 7f1cae000640 1 -- 172.21.15.12:0/2077246131 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c0188f0 con 0x7f1cb01028c0 2024-03-23T23:03:02.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.271+0000 7f1cb8000640 1 -- 172.21.15.12:0/2077246131 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cb01a2060 con 0x7f1cb01028c0 2024-03-23T23:03:02.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.274+0000 7f1cae000640 1 -- 172.21.15.12:0/2077246131 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1c9c010ac0 con 0x7f1cb01028c0 2024-03-23T23:03:02.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.276+0000 7f1cbadae740 1 Processor -- start 2024-03-23T23:03:02.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.276+0000 7f1cbadae740 1 -- start start 2024-03-23T23:03:02.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.276+0000 7f1cbadae740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55faafd1ce80 0x55faafd1d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:02.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.276+0000 7f1cbadae740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55faafd1d870 0x55faafd22370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:02.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.277+0000 7f1cbadae740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55faafd229e0 0x55faafd24e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:02.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.277+0000 7f1cbadae740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faafc82250 con 0x55faafd1d870 2024-03-23T23:03:02.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.277+0000 7f1cbadae740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faafc2aa30 con 0x55faafd1ce80 2024-03-23T23:03:02.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.277+0000 7f1cbadae740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faafc7e2f0 con 0x55faafd229e0 2024-03-23T23:03:02.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.277+0000 7f1cae000640 1 --2- 172.21.15.12:0/2077246131 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1c980a2800 0x7f1c980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:02.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.277+0000 7f1cb4c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55faafd229e0 0x55faafd24e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:02.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.277+0000 7f1cb4c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55faafd229e0 0x55faafd24e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45890/0 (socket says 172.21.15.12:45890) 2024-03-23T23:03:02.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.277+0000 7f1cb4c00640 1 -- 172.21.15.12:0/1443119052 learned_addr learned my addr 172.21.15.12:0/1443119052 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:02.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.277+0000 7f1cb6000640 1 --2- 172.21.15.12:0/1443119052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55faafd1ce80 0x55faafd1d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:02.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.277+0000 7f1cb4c00640 1 -- 172.21.15.12:0/1443119052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55faafd1ce80 msgr2=0x55faafd1d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:02.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.277+0000 7f1cb4c00640 1 --2- 172.21.15.12:0/1443119052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55faafd1ce80 0x55faafd1d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.277+0000 7f1cb4c00640 1 -- 172.21.15.12:0/1443119052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55faafd1d870 msgr2=0x55faafd22370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:02.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.277+0000 7f1cb4c00640 1 --2- 172.21.15.12:0/1443119052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55faafd1d870 0x55faafd22370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.278+0000 7f1cb6000640 1 --2- 172.21.15.12:0/2077246131 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1c980a2800 0x7f1c980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:02.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.278+0000 7f1cb4c00640 1 -- 172.21.15.12:0/1443119052 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55faafc15690 con 0x55faafd229e0 2024-03-23T23:03:02.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.278+0000 7f1cae000640 1 -- 172.21.15.12:0/2077246131 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1c9c0cefa0 con 0x7f1cb01028c0 2024-03-23T23:03:02.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.278+0000 7f1cb4c00640 1 --2- 172.21.15.12:0/1443119052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55faafd229e0 0x55faafd24e70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c0cc9c0 tx=0x7f1c9c0666e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ce41c98549415ef server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:02.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.278+0000 7f1ca2600640 1 -- 172.21.15.12:0/1443119052 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c026030 con 0x55faafd229e0 2024-03-23T23:03:02.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.278+0000 7f1ca2600640 1 -- 172.21.15.12:0/1443119052 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c9c09bff0 con 0x55faafd229e0 2024-03-23T23:03:02.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.278+0000 7f1ca2600640 1 -- 172.21.15.12:0/1443119052 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c06eb50 con 0x55faafd229e0 2024-03-23T23:03:02.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.279+0000 7f1cbadae740 1 -- 172.21.15.12:0/1443119052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55faafd229e0 msgr2=0x55faafd24e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:02.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.279+0000 7f1cbadae740 1 --2- 172.21.15.12:0/1443119052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55faafd229e0 0x55faafd24e70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c0cc9c0 tx=0x7f1c9c0666e0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.279+0000 7f1cb6000640 1 --2- 172.21.15.12:0/2077246131 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1c980a2800 0x7f1c980a4cc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1ca80099e0 tx=0x7f1ca8013040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:02.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.279+0000 7f1cbadae740 1 -- 172.21.15.12:0/1443119052 shutdown_connections 2024-03-23T23:03:02.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.279+0000 7f1cbadae740 1 --2- 172.21.15.12:0/1443119052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55faafd229e0 0x55faafd24e70 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.279+0000 7f1cbadae740 1 --2- 172.21.15.12:0/1443119052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55faafd1d870 0x55faafd22370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.279+0000 7f1cbadae740 1 --2- 172.21.15.12:0/1443119052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55faafd1ce80 0x55faafd1d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.279+0000 7f1cbadae740 1 -- 172.21.15.12:0/1443119052 >> 172.21.15.12:0/1443119052 conn(0x55faafce5200 msgr2=0x55faafd187f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:02.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.280+0000 7f1cbadae740 1 -- 172.21.15.12:0/1443119052 shutdown_connections 2024-03-23T23:03:02.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.280+0000 7f1cbadae740 1 -- 172.21.15.12:0/1443119052 wait complete. 2024-03-23T23:03:02.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.280+0000 7f1cbadae740 1 -- start start 2024-03-23T23:03:02.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.281+0000 7f1cbadae740 1 Processor -- start 2024-03-23T23:03:02.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.281+0000 7f1cbadae740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55faafd1ce80 0x55faafd2f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.281+0000 7f1cbadae740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55faafd1d870 0x55faafd2fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.281+0000 7f1cbadae740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55faafd229e0 0x55faafd36aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.281+0000 7f1cbadae740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faafd258d0 con 0x55faafd229e0 2024-03-23T23:03:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.281+0000 7f1cbadae740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faafc3de80 con 0x55faafd1ce80 2024-03-23T23:03:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.281+0000 7f1cbadae740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faafd25a50 con 0x55faafd1d870 2024-03-23T23:03:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.281+0000 7f1cbadae740 10 client.0 _ll_register_callbacks cb 0x55faafc50130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.281+0000 7f1cb5600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55faafd1ce80 0x55faafd2f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.281+0000 7f1cb5600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55faafd1ce80 0x55faafd2f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:35988/0 (socket says 172.21.15.12:35988) 2024-03-23T23:03:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.281+0000 7f1cb5600640 1 -- 172.21.15.12:0/4167390386 learned_addr learned my addr 172.21.15.12:0/4167390386 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.282+0000 7f1cb4c00640 1 --2- 172.21.15.12:0/4167390386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55faafd1d870 0x55faafd2fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:02.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.282+0000 7f1cb5600640 1 -- 172.21.15.12:0/4167390386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55faafd1d870 msgr2=0x55faafd2fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:02.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.282+0000 7f1cb5600640 1 --2- 172.21.15.12:0/4167390386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55faafd1d870 0x55faafd2fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.282+0000 7f1cb5600640 1 -- 172.21.15.12:0/4167390386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55faafd229e0 msgr2=0x55faafd36aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:02.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.282+0000 7f1cb6000640 1 --2- 172.21.15.12:0/4167390386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55faafd229e0 0x55faafd36aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:02.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.282+0000 7f1cb5600640 1 --2- 172.21.15.12:0/4167390386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55faafd229e0 0x55faafd36aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.282+0000 7f1cb5600640 1 -- 172.21.15.12:0/4167390386 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55faafa607d0 con 0x55faafd1ce80 2024-03-23T23:03:02.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.282+0000 7f1cb6000640 1 --2- 172.21.15.12:0/4167390386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55faafd229e0 0x55faafd36aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:03:02.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.282+0000 7f1cb5600640 1 --2- 172.21.15.12:0/4167390386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55faafd1ce80 0x55faafd2f520 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1ca402c8e0 tx=0x7f1ca4004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce6da8eac926792a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:02.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.282+0000 7f1c96a00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:03:02.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.283+0000 7f1c96a00640 1 -- 172.21.15.12:0/4167390386 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4044020 con 0x55faafd1ce80 2024-03-23T23:03:02.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.283+0000 7f1c96a00640 1 -- 172.21.15.12:0/4167390386 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1ca4005240 con 0x55faafd1ce80 2024-03-23T23:03:02.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.283+0000 7f1cbadae740 1 -- 172.21.15.12:0/4167390386 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1c9c09bff0 con 0x55faafd1ce80 2024-03-23T23:03:02.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.283+0000 7f1c96a00640 1 -- 172.21.15.12:0/4167390386 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ca4005d50 con 0x55faafd1ce80 2024-03-23T23:03:02.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.283+0000 7f1cbadae740 10 client.25550 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:02.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.283+0000 7f1cbadae740 1 -- 172.21.15.12:0/4167390386 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55faaf9963f0 con 0x55faafd1ce80 2024-03-23T23:03:02.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.283+0000 7f1c96a00640 1 -- 172.21.15.12:0/4167390386 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c9c09bff0 con 0x55faafd1ce80 2024-03-23T23:03:02.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.284+0000 7f1c96a00640 1 -- 172.21.15.12:0/4167390386 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f1ca40035f0 con 0x55faafd1ce80 2024-03-23T23:03:02.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.284+0000 7f1cbadae740 10 client.25550 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:02.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.284+0000 7f1cbadae740 10 client.25550 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:02.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.284+0000 7f1cbadae740 20 client.25550 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:02.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.284+0000 7f1cbadae740 1 --2- 172.21.15.12:0/4167390386 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55faafd3b270 0x55faafd3d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:02.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.284+0000 7f1cbadae740 4 client.25550 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:03:02.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.284+0000 7f1cbadae740 1 -- 172.21.15.12:0/4167390386 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55faafc8f790 con 0x55faafd3b270 2024-03-23T23:03:02.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.284+0000 7f1cb4c00640 1 --2- 172.21.15.12:0/4167390386 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55faafd3b270 0x55faafd3d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:02.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.285+0000 7f1cb4c00640 1 --2- 172.21.15.12:0/4167390386 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55faafd3b270 0x55faafd3d6c0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1bfaf98b8473338 server_cookie=3b0010d516a2de09 in_seq=0 out_seq=0 2024-03-23T23:03:02.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.285+0000 7f1c96a00640 10 client.25550 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:02.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.286+0000 7f1c96a00640 1 -- 172.21.15.12:0/4167390386 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55faafc8f790 con 0x55faafd3b270 2024-03-23T23:03:02.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.286+0000 7f1c96a00640 10 client.25550 handle_command_reply: tid=0 2024-03-23T23:03:02.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.286+0000 7f1cbadae740 1 client.25550 shutdown 2024-03-23T23:03:02.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.286+0000 7f1cbadae740 1 -- 172.21.15.12:0/4167390386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55faafd1ce80 msgr2=0x55faafd2f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:02.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.286+0000 7f1cbadae740 1 --2- 172.21.15.12:0/4167390386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55faafd1ce80 0x55faafd2f520 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1ca402c8e0 tx=0x7f1ca4004270 comp rx=0 tx=0).stop 2024-03-23T23:03:02.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.286+0000 7f1cbadae740 1 -- 172.21.15.12:0/4167390386 shutdown_connections 2024-03-23T23:03:02.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.286+0000 7f1cbadae740 1 --2- 172.21.15.12:0/4167390386 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55faafd3b270 0x55faafd3d6c0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.286+0000 7f1cbadae740 1 --2- 172.21.15.12:0/4167390386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55faafd229e0 0x55faafd36aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.286+0000 7f1cbadae740 1 --2- 172.21.15.12:0/4167390386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55faafd1d870 0x55faafd2fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.286+0000 7f1cbadae740 1 --2- 172.21.15.12:0/4167390386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55faafd1ce80 0x55faafd2f520 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.286+0000 7f1cbadae740 1 -- 172.21.15.12:0/4167390386 >> 172.21.15.12:0/4167390386 conn(0x55faafce5200 msgr2=0x55faafd17a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:02.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.286+0000 7f1c96a00640 0 client.25550 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:02.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.287+0000 7f1cbadae740 1 -- 172.21.15.12:0/4167390386 shutdown_connections 2024-03-23T23:03:02.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.287+0000 7f1cbadae740 1 -- 172.21.15.12:0/4167390386 wait complete. 2024-03-23T23:03:02.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.287+0000 7f1cbadae740 20 client.25550 trim_cache size 0 max 16384 2024-03-23T23:03:02.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.321+0000 7f1cbadae740 1 Processor -- start 2024-03-23T23:03:02.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.321+0000 7f1cbadae740 1 -- start start 2024-03-23T23:03:02.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.321+0000 7f1cbadae740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55faafd1ce80 0x55faafd17cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:02.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.321+0000 7f1cbadae740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55faafd1d870 0x55faafd369c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:02.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.321+0000 7f1cbadae740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55faafd229e0 0x55faafd41f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:02.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.321+0000 7f1cbadae740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb01a8e70 con 0x55faafd1ce80 2024-03-23T23:03:02.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.321+0000 7f1cbadae740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb01a6970 con 0x55faafd229e0 2024-03-23T23:03:02.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.321+0000 7f1cbadae740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb01a4a30 con 0x55faafd1d870 2024-03-23T23:03:02.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.322+0000 7f1cb5600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55faafd1ce80 0x55faafd17cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:02.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.322+0000 7f1cb6000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55faafd229e0 0x55faafd41f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:02.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.322+0000 7f1cb5600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55faafd1ce80 0x55faafd17cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45062/0 (socket says 172.21.15.12:45062) 2024-03-23T23:03:02.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.322+0000 7f1cb5600640 1 -- 172.21.15.12:0/2524893095 learned_addr learned my addr 172.21.15.12:0/2524893095 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:02.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.322+0000 7f1cb4c00640 1 --2- 172.21.15.12:0/2524893095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55faafd1d870 0x55faafd369c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:02.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.322+0000 7f1cb5600640 1 -- 172.21.15.12:0/2524893095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55faafd1d870 msgr2=0x55faafd369c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:02.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.322+0000 7f1cb5600640 1 --2- 172.21.15.12:0/2524893095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55faafd1d870 0x55faafd369c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.322+0000 7f1cb5600640 1 -- 172.21.15.12:0/2524893095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55faafd229e0 msgr2=0x55faafd41f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:02.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.322+0000 7f1cb5600640 1 --2- 172.21.15.12:0/2524893095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55faafd229e0 0x55faafd41f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.322+0000 7f1cb5600640 1 -- 172.21.15.12:0/2524893095 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55faaf9560f0 con 0x55faafd1ce80 2024-03-23T23:03:02.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.322+0000 7f1cb6000640 1 --2- 172.21.15.12:0/2524893095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55faafd229e0 0x55faafd41f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:02.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.323+0000 7f1cb5600640 1 --2- 172.21.15.12:0/2524893095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55faafd1ce80 0x55faafd17cd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1ca404b040 tx=0x7f1ca402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d85206aa01225985 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:02.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.323+0000 7f1ca1c00640 1 -- 172.21.15.12:0/2524893095 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4056070 con 0x55faafd1ce80 2024-03-23T23:03:02.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.323+0000 7f1ca1c00640 1 -- 172.21.15.12:0/2524893095 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ca4009520 con 0x55faafd1ce80 2024-03-23T23:03:02.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.323+0000 7f1ca1c00640 1 -- 172.21.15.12:0/2524893095 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4044070 con 0x55faafd1ce80 2024-03-23T23:03:02.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.324+0000 7f1cbadae740 1 -- 172.21.15.12:0/2524893095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55faafd1ce80 msgr2=0x55faafd17cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:02.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.324+0000 7f1cbadae740 1 --2- 172.21.15.12:0/2524893095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55faafd1ce80 0x55faafd17cd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1ca404b040 tx=0x7f1ca402bc30 comp rx=0 tx=0).stop 2024-03-23T23:03:02.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.324+0000 7f1cbadae740 1 -- 172.21.15.12:0/2524893095 shutdown_connections 2024-03-23T23:03:02.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.324+0000 7f1cbadae740 1 --2- 172.21.15.12:0/2524893095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55faafd229e0 0x55faafd41f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.324+0000 7f1cbadae740 1 --2- 172.21.15.12:0/2524893095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55faafd1d870 0x55faafd369c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.324+0000 7f1cbadae740 1 --2- 172.21.15.12:0/2524893095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55faafd1ce80 0x55faafd17cd0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.324+0000 7f1cbadae740 1 -- 172.21.15.12:0/2524893095 >> 172.21.15.12:0/2524893095 conn(0x55faafce5200 msgr2=0x55faafd1c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:02.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.324+0000 7f1cbadae740 1 -- 172.21.15.12:0/2524893095 shutdown_connections 2024-03-23T23:03:02.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.324+0000 7f1cbadae740 1 -- 172.21.15.12:0/2524893095 wait complete. 2024-03-23T23:03:02.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.325+0000 7f1cbadae740 1 -- start start 2024-03-23T23:03:02.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.325+0000 7f1cbadae740 1 Processor -- start 2024-03-23T23:03:02.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.326+0000 7f1cbadae740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55faafd1ce80 0x55faafd34310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:02.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.326+0000 7f1cbadae740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55faafd1d870 0x55faafd34850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:02.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.326+0000 7f1cbadae740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55faafd229e0 0x55faafd35860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:02.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.326+0000 7f1cbadae740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb01a5030 con 0x55faafd229e0 2024-03-23T23:03:02.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.326+0000 7f1cbadae740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb01a6310 con 0x55faafd1ce80 2024-03-23T23:03:02.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.326+0000 7f1cbadae740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb01a5650 con 0x55faafd1d870 2024-03-23T23:03:02.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.326+0000 7f1cbadae740 10 client.0 _ll_register_callbacks cb 0x7f1c9c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:02.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.326+0000 7f1cb4c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55faafd1d870 0x55faafd34850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:02.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.326+0000 7f1cb5600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55faafd1ce80 0x55faafd34310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:02.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.326+0000 7f1cb4c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55faafd1d870 0x55faafd34850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45932/0 (socket says 172.21.15.12:45932) 2024-03-23T23:03:02.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.326+0000 7f1cb4c00640 1 -- 172.21.15.12:0/2976219114 learned_addr learned my addr 172.21.15.12:0/2976219114 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:02.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.327+0000 7f1cb6000640 1 --2- 172.21.15.12:0/2976219114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55faafd229e0 0x55faafd35860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:02.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.327+0000 7f1cb4c00640 1 -- 172.21.15.12:0/2976219114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55faafd1ce80 msgr2=0x55faafd34310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:02.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.327+0000 7f1cb4c00640 1 --2- 172.21.15.12:0/2976219114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55faafd1ce80 0x55faafd34310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.327+0000 7f1cb4c00640 1 -- 172.21.15.12:0/2976219114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55faafd229e0 msgr2=0x55faafd35860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:02.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.327+0000 7f1cb4c00640 1 --2- 172.21.15.12:0/2976219114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55faafd229e0 0x55faafd35860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.327+0000 7f1cb4c00640 1 -- 172.21.15.12:0/2976219114 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55faafd35e50 con 0x55faafd1d870 2024-03-23T23:03:02.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.327+0000 7f1cb5600640 1 --2- 172.21.15.12:0/2976219114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55faafd1ce80 0x55faafd34310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:02.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.327+0000 7f1cb4c00640 1 --2- 172.21.15.12:0/2976219114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55faafd1d870 0x55faafd34850 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c0a3b60 tx=0x7f1c9c098000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c064aa0aa01fdc5f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:02.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.327+0000 7f1c96a00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:03:02.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.328+0000 7f1c96a00640 1 -- 172.21.15.12:0/2976219114 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c099a30 con 0x55faafd1d870 2024-03-23T23:03:02.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.328+0000 7f1c96a00640 1 -- 172.21.15.12:0/2976219114 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1c9c0ccad0 con 0x55faafd1d870 2024-03-23T23:03:02.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.328+0000 7f1cbadae740 1 -- 172.21.15.12:0/2976219114 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1ca4009520 con 0x55faafd1d870 2024-03-23T23:03:02.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.328+0000 7f1c96a00640 1 -- 172.21.15.12:0/2976219114 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c9c09bff0 con 0x55faafd1d870 2024-03-23T23:03:02.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.328+0000 7f1cbadae740 10 client.25525 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:02.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.328+0000 7f1cbadae740 1 -- 172.21.15.12:0/2976219114 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55faafd96da0 con 0x55faafd1d870 2024-03-23T23:03:02.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.328+0000 7f1c96a00640 1 -- 172.21.15.12:0/2976219114 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ca4009520 con 0x55faafd1d870 2024-03-23T23:03:02.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.328+0000 7f1c96a00640 1 -- 172.21.15.12:0/2976219114 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f1c9c0860d0 con 0x55faafd1d870 2024-03-23T23:03:02.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.328+0000 7f1cbadae740 10 client.25525 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:02.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.328+0000 7f1cbadae740 10 client.25525 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:02.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.328+0000 7f1cbadae740 20 client.25525 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:02.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.328+0000 7f1cbadae740 1 --2- 172.21.15.12:0/2976219114 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55faafd972f0 0x55faafd99710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:02.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.328+0000 7f1cbadae740 4 client.25525 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:03:02.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.328+0000 7f1cbadae740 1 -- 172.21.15.12:0/2976219114 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55faafc89ae0 con 0x55faafd972f0 2024-03-23T23:03:02.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.329+0000 7f1cb5600640 1 --2- 172.21.15.12:0/2976219114 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55faafd972f0 0x55faafd99710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:02.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.329+0000 7f1cb5600640 1 --2- 172.21.15.12:0/2976219114 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55faafd972f0 0x55faafd99710 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d53dc5bdc00ece25 server_cookie=74489a3443e73033 in_seq=0 out_seq=0 2024-03-23T23:03:02.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.329+0000 7f1c96a00640 10 client.25525 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:02.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.329+0000 7f1c96a00640 1 -- 172.21.15.12:0/2976219114 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55faafc89ae0 con 0x55faafd972f0 2024-03-23T23:03:02.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.329+0000 7f1c96a00640 10 client.25525 handle_command_reply: tid=0 2024-03-23T23:03:02.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.329+0000 7f1cbadae740 1 client.25525 shutdown 2024-03-23T23:03:02.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.330+0000 7f1cbadae740 1 -- 172.21.15.12:0/2976219114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55faafd1d870 msgr2=0x55faafd34850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:02.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.330+0000 7f1cbadae740 1 --2- 172.21.15.12:0/2976219114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55faafd1d870 0x55faafd34850 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c0a3b60 tx=0x7f1c9c098000 comp rx=0 tx=0).stop 2024-03-23T23:03:02.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.330+0000 7f1cbadae740 1 -- 172.21.15.12:0/2976219114 shutdown_connections 2024-03-23T23:03:02.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.330+0000 7f1cbadae740 1 --2- 172.21.15.12:0/2976219114 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55faafd972f0 0x55faafd99710 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.330+0000 7f1cbadae740 1 --2- 172.21.15.12:0/2976219114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55faafd229e0 0x55faafd35860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.330+0000 7f1cbadae740 1 --2- 172.21.15.12:0/2976219114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55faafd1d870 0x55faafd34850 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.330+0000 7f1cbadae740 1 --2- 172.21.15.12:0/2976219114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55faafd1ce80 0x55faafd34310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.330+0000 7f1cbadae740 1 -- 172.21.15.12:0/2976219114 >> 172.21.15.12:0/2976219114 conn(0x55faafce5200 msgr2=0x55faafd1ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:02.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.330+0000 7f1c96a00640 0 client.25525 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:02.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.330+0000 7f1cbadae740 1 -- 172.21.15.12:0/2976219114 shutdown_connections 2024-03-23T23:03:02.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.330+0000 7f1cbadae740 1 -- 172.21.15.12:0/2976219114 wait complete. 2024-03-23T23:03:02.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.330+0000 7f1cbadae740 20 client.25525 trim_cache size 0 max 16384 2024-03-23T23:03:02.333 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:03:02.333 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-03-23T23:03:02.333 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-03-23T23:03:02.333 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:03:02.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.331+0000 7f1cb8000640 1 -- 172.21.15.12:0/2077246131 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1c980a2800 msgr2=0x7f1c980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:02.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.331+0000 7f1cb8000640 1 --2- 172.21.15.12:0/2077246131 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1c980a2800 0x7f1c980a4cc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1ca80099e0 tx=0x7f1ca8013040 comp rx=0 tx=0).stop 2024-03-23T23:03:02.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.331+0000 7f1cb8000640 1 -- 172.21.15.12:0/2077246131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1cb01028c0 msgr2=0x7f1cb019a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:02.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.331+0000 7f1cb8000640 1 --2- 172.21.15.12:0/2077246131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1cb01028c0 0x7f1cb019a180 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00e9f0 tx=0x7f1c9c00eeb0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.331+0000 7f1cb8000640 1 -- 172.21.15.12:0/2077246131 shutdown_connections 2024-03-23T23:03:02.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.331+0000 7f1cb8000640 1 --2- 172.21.15.12:0/2077246131 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1c980a2800 0x7f1c980a4cc0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.331+0000 7f1cb8000640 1 --2- 172.21.15.12:0/2077246131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cb0103280 0x7f1cb01a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.331+0000 7f1cb8000640 1 --2- 172.21.15.12:0/2077246131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1cb01028c0 0x7f1cb019a180 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.331+0000 7f1cb8000640 1 --2- 172.21.15.12:0/2077246131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1cb01016d0 0x7f1cb0199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.331+0000 7f1cb8000640 1 -- 172.21.15.12:0/2077246131 >> 172.21.15.12:0/2077246131 conn(0x7f1cb00fcc40 msgr2=0x7f1cb00738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:02.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.331+0000 7f1cb8000640 1 -- 172.21.15.12:0/2077246131 shutdown_connections 2024-03-23T23:03:02.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.332+0000 7f1cb8000640 1 -- 172.21.15.12:0/2077246131 wait complete. 2024-03-23T23:03:02.349 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T23:03:02.349 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:03:02.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.546+0000 7fa143200640 1 -- 172.21.15.12:0/1210749516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa13c1038e0 msgr2=0x7fa13c103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:02.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.546+0000 7fa143200640 1 --2- 172.21.15.12:0/1210749516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa13c1038e0 0x7fa13c103d00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa12c009900 tx=0x7fa12c02bc30 comp rx=0 tx=0).stop 2024-03-23T23:03:02.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.546+0000 7fa143200640 1 -- 172.21.15.12:0/1210749516 shutdown_connections 2024-03-23T23:03:02.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.546+0000 7fa143200640 1 --2- 172.21.15.12:0/1210749516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa13c105490 0x7fa13c10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.546+0000 7fa143200640 1 --2- 172.21.15.12:0/1210749516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa13c104ad0 0x7fa13c104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.546+0000 7fa143200640 1 --2- 172.21.15.12:0/1210749516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa13c1038e0 0x7fa13c103d00 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.546+0000 7fa143200640 1 -- 172.21.15.12:0/1210749516 >> 172.21.15.12:0/1210749516 conn(0x7fa13c0fee90 msgr2=0x7fa13c1012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:02.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.546+0000 7fa143200640 1 -- 172.21.15.12:0/1210749516 shutdown_connections 2024-03-23T23:03:02.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.547+0000 7fa143200640 1 -- 172.21.15.12:0/1210749516 wait complete. 2024-03-23T23:03:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.548+0000 7fa143200640 1 Processor -- start 2024-03-23T23:03:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.548+0000 7fa143200640 1 -- start start 2024-03-23T23:03:02.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.548+0000 7fa143200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa13c1038e0 0x7fa13c199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:02.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.549+0000 7fa143200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa13c104ad0 0x7fa13c19a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:02.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.549+0000 7fa143200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa13c105490 0x7fa13c1a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:02.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.549+0000 7fa143200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c10f580 con 0x7fa13c105490 2024-03-23T23:03:02.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.549+0000 7fa143200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c10f400 con 0x7fa13c1038e0 2024-03-23T23:03:02.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.549+0000 7fa143200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c10f700 con 0x7fa13c104ad0 2024-03-23T23:03:02.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.549+0000 7fa13be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa13c1038e0 0x7fa13c199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:02.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.549+0000 7fa13be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa13c1038e0 0x7fa13c199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36034/0 (socket says 172.21.15.12:36034) 2024-03-23T23:03:02.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.549+0000 7fa13be00640 1 -- 172.21.15.12:0/3390033410 learned_addr learned my addr 172.21.15.12:0/3390033410 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:02.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.549+0000 7fa13b400640 1 --2- 172.21.15.12:0/3390033410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa13c104ad0 0x7fa13c19a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:02.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.549+0000 7fa13be00640 1 -- 172.21.15.12:0/3390033410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa13c104ad0 msgr2=0x7fa13c19a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:02.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.549+0000 7fa13be00640 1 --2- 172.21.15.12:0/3390033410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa13c104ad0 0x7fa13c19a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.549+0000 7fa13be00640 1 -- 172.21.15.12:0/3390033410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa13c105490 msgr2=0x7fa13c1a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:02.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.549+0000 7fa141200640 1 --2- 172.21.15.12:0/3390033410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa13c105490 0x7fa13c1a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:02.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.549+0000 7fa13be00640 1 --2- 172.21.15.12:0/3390033410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa13c105490 0x7fa13c1a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.549+0000 7fa13be00640 1 -- 172.21.15.12:0/3390033410 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa13c1a1850 con 0x7fa13c1038e0 2024-03-23T23:03:02.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.549+0000 7fa141200640 1 --2- 172.21.15.12:0/3390033410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa13c105490 0x7fa13c1a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:03:02.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.550+0000 7fa13be00640 1 --2- 172.21.15.12:0/3390033410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa13c1038e0 0x7fa13c199c40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa12c000c00 tx=0x7fa12c02c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:02.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.550+0000 7fa138c00640 1 -- 172.21.15.12:0/3390033410 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12c043020 con 0x7fa13c1038e0 2024-03-23T23:03:02.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.550+0000 7fa138c00640 1 -- 172.21.15.12:0/3390033410 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa12c0054e0 con 0x7fa13c1038e0 2024-03-23T23:03:02.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.550+0000 7fa138c00640 1 -- 172.21.15.12:0/3390033410 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12c003ce0 con 0x7fa13c1038e0 2024-03-23T23:03:02.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.550+0000 7fa143200640 1 -- 172.21.15.12:0/3390033410 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa13c1a1ae0 con 0x7fa13c1038e0 2024-03-23T23:03:02.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.551+0000 7fa143200640 1 -- 172.21.15.12:0/3390033410 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa13c1a1fb0 con 0x7fa13c1038e0 2024-03-23T23:03:02.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.553+0000 7fa143200640 1 -- 172.21.15.12:0/3390033410 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa13c10ac00 con 0x7fa13c1038e0 2024-03-23T23:03:02.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.554+0000 7fa138c00640 1 -- 172.21.15.12:0/3390033410 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa12c046020 con 0x7fa13c1038e0 2024-03-23T23:03:02.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.556+0000 7fa138c00640 1 --2- 172.21.15.12:0/3390033410 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa1200a2880 0x7fa1200a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:02.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.556+0000 7fa138c00640 1 -- 172.21.15.12:0/3390033410 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fa12c0f0930 con 0x7fa13c1038e0 2024-03-23T23:03:02.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.556+0000 7fa13b400640 1 --2- 172.21.15.12:0/3390033410 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa1200a2880 0x7fa1200a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:02.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.557+0000 7fa13b400640 1 --2- 172.21.15.12:0/3390033410 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa1200a2880 0x7fa1200a4d40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa128009bd0 tx=0x7fa128005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:02.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.560+0000 7fa138c00640 1 -- 172.21.15.12:0/3390033410 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa12c04d030 con 0x7fa13c1038e0 2024-03-23T23:03:02.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:02 smithi012 ceph-mon[31695]: pgmap v427: 97 pgs: 97 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:03:02.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:02 smithi173 ceph-mon[36116]: pgmap v427: 97 pgs: 97 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:03:02.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.890+0000 7fa143200640 1 -- 172.21.15.12:0/3390033410 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa13c0622a0 con 0x7fa13c1038e0 2024-03-23T23:03:02.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.891+0000 7fa138c00640 1 -- 172.21.15.12:0/3390033410 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7fa12c0bd980 con 0x7fa13c1038e0 2024-03-23T23:03:02.891 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:03:02.891 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:03:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:03:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.894+0000 7fa143200640 1 -- 172.21.15.12:0/3390033410 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa1200a2880 msgr2=0x7fa1200a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.894+0000 7fa143200640 1 --2- 172.21.15.12:0/3390033410 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa1200a2880 0x7fa1200a4d40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa128009bd0 tx=0x7fa128005f90 comp rx=0 tx=0).stop 2024-03-23T23:03:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.894+0000 7fa143200640 1 -- 172.21.15.12:0/3390033410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa13c1038e0 msgr2=0x7fa13c199c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.894+0000 7fa143200640 1 --2- 172.21.15.12:0/3390033410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa13c1038e0 0x7fa13c199c40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa12c000c00 tx=0x7fa12c02c6c0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.894+0000 7fa143200640 1 -- 172.21.15.12:0/3390033410 shutdown_connections 2024-03-23T23:03:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.894+0000 7fa143200640 1 --2- 172.21.15.12:0/3390033410 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa1200a2880 0x7fa1200a4d40 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.894+0000 7fa143200640 1 --2- 172.21.15.12:0/3390033410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa13c105490 0x7fa13c1a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.894+0000 7fa143200640 1 --2- 172.21.15.12:0/3390033410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa13c104ad0 0x7fa13c19a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.894+0000 7fa143200640 1 --2- 172.21.15.12:0/3390033410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa13c1038e0 0x7fa13c199c40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.894+0000 7fa143200640 1 -- 172.21.15.12:0/3390033410 >> 172.21.15.12:0/3390033410 conn(0x7fa13c0fee90 msgr2=0x7fa13c107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.894+0000 7fa143200640 1 -- 172.21.15.12:0/3390033410 shutdown_connections 2024-03-23T23:03:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:02.894+0000 7fa143200640 1 -- 172.21.15.12:0/3390033410 wait complete. 2024-03-23T23:03:02.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:02 smithi203 ceph-mon[36693]: pgmap v427: 97 pgs: 97 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:03:02.919 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:03:02.920 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-23T23:03:03.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.101+0000 7f4042600640 1 -- 172.21.15.12:0/1463848977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f403c102790 msgr2=0x7f403c102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.101+0000 7f4042600640 1 --2- 172.21.15.12:0/1463848977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f403c102790 0x7f403c102c10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4028009900 tx=0x7f402802bc40 comp rx=0 tx=0).stop 2024-03-23T23:03:03.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.102+0000 7f4042600640 1 -- 172.21.15.12:0/1463848977 shutdown_connections 2024-03-23T23:03:03.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.102+0000 7f4042600640 1 --2- 172.21.15.12:0/1463848977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f403c103150 0x7f403c1099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.102+0000 7f4042600640 1 --2- 172.21.15.12:0/1463848977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f403c102790 0x7f403c102c10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.102+0000 7f4042600640 1 --2- 172.21.15.12:0/1463848977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f403c1015a0 0x7f403c1019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.102+0000 7f4042600640 1 -- 172.21.15.12:0/1463848977 >> 172.21.15.12:0/1463848977 conn(0x7f403c0fcb10 msgr2=0x7f403c0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:03.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.102+0000 7f4042600640 1 -- 172.21.15.12:0/1463848977 shutdown_connections 2024-03-23T23:03:03.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.102+0000 7f4042600640 1 -- 172.21.15.12:0/1463848977 wait complete. 2024-03-23T23:03:03.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.103+0000 7f4042600640 1 Processor -- start 2024-03-23T23:03:03.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.103+0000 7f4042600640 1 -- start start 2024-03-23T23:03:03.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.103+0000 7f4042600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f403c1015a0 0x7f403c197890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.103+0000 7f4042600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f403c102790 0x7f403c197dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.103+0000 7f4042600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f403c103150 0x7f403c19ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.103+0000 7f4042600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c10d250 con 0x7f403c1015a0 2024-03-23T23:03:03.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.103+0000 7f4042600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c10d0d0 con 0x7f403c103150 2024-03-23T23:03:03.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.103+0000 7f4042600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c10d3d0 con 0x7f403c102790 2024-03-23T23:03:03.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.103+0000 7f403be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f403c102790 0x7f403c197dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.103+0000 7f4041c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f403c103150 0x7f403c19ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.103+0000 7f403be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f403c102790 0x7f403c197dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45966/0 (socket says 172.21.15.12:45966) 2024-03-23T23:03:03.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.103+0000 7f403be00640 1 -- 172.21.15.12:0/1335128236 learned_addr learned my addr 172.21.15.12:0/1335128236 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:03.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.104+0000 7f4041200640 1 --2- 172.21.15.12:0/1335128236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f403c1015a0 0x7f403c197890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.104+0000 7f4041c00640 1 -- 172.21.15.12:0/1335128236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f403c102790 msgr2=0x7f403c197dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.104+0000 7f4041c00640 1 --2- 172.21.15.12:0/1335128236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f403c102790 0x7f403c197dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.104+0000 7f4041c00640 1 -- 172.21.15.12:0/1335128236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f403c1015a0 msgr2=0x7f403c197890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.104+0000 7f4041c00640 1 --2- 172.21.15.12:0/1335128236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f403c1015a0 0x7f403c197890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.104+0000 7f4041c00640 1 -- 172.21.15.12:0/1335128236 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403c073950 con 0x7f403c103150 2024-03-23T23:03:03.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.104+0000 7f4041c00640 1 --2- 172.21.15.12:0/1335128236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f403c103150 0x7f403c19ee40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f403400ec50 tx=0x7f403400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:03.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.105+0000 7f4039600640 1 -- 172.21.15.12:0/1335128236 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4034003b70 con 0x7f403c103150 2024-03-23T23:03:03.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.105+0000 7f4039600640 1 -- 172.21.15.12:0/1335128236 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4034003d10 con 0x7f403c103150 2024-03-23T23:03:03.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.105+0000 7f4039600640 1 -- 172.21.15.12:0/1335128236 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4034010890 con 0x7f403c103150 2024-03-23T23:03:03.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.105+0000 7f4042600640 1 -- 172.21.15.12:0/1335128236 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f403c073c40 con 0x7f403c103150 2024-03-23T23:03:03.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.105+0000 7f4042600640 1 -- 172.21.15.12:0/1335128236 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f403c074110 con 0x7f403c103150 2024-03-23T23:03:03.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.109+0000 7f404683d740 1 Processor -- start 2024-03-23T23:03:03.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.109+0000 7f4039600640 1 -- 172.21.15.12:0/1335128236 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4034003fc0 con 0x7f403c103150 2024-03-23T23:03:03.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.110+0000 7f404683d740 1 -- start start 2024-03-23T23:03:03.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.110+0000 7f404683d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bfc99c200 0x555bfc99c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.110+0000 7f404683d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555bfc99d3d0 0x555bfc99d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.110+0000 7f404683d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc99dd90 0x555bfc9a4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.110+0000 7f404683d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bfc8f82c0 con 0x555bfc99d3d0 2024-03-23T23:03:03.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.110+0000 7f404683d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bfc8a0c90 con 0x555bfc99dd90 2024-03-23T23:03:03.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.110+0000 7f404683d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bfc8f4360 con 0x555bfc99c200 2024-03-23T23:03:03.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.111+0000 7f403be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc99dd90 0x555bfc9a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.111+0000 7f4039600640 1 --2- 172.21.15.12:0/1335128236 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f40240a2880 0x7f40240a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.111+0000 7f4041200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555bfc99d3d0 0x555bfc99d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.111+0000 7f4041200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555bfc99d3d0 0x555bfc99d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45130/0 (socket says 172.21.15.12:45130) 2024-03-23T23:03:03.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.111+0000 7f4041200640 1 -- 172.21.15.12:0/211351411 learned_addr learned my addr 172.21.15.12:0/211351411 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:03.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.111+0000 7f4039600640 1 -- 172.21.15.12:0/1335128236 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f40340cefa0 con 0x7f403c103150 2024-03-23T23:03:03.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.111+0000 7f4041c00640 1 --2- 172.21.15.12:0/1335128236 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f40240a2880 0x7f40240a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.111+0000 7f403be00640 1 --2- 172.21.15.12:0/211351411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bfc99c200 0x555bfc99c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.112+0000 7f403be00640 1 -- 172.21.15.12:0/211351411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bfc99c200 msgr2=0x555bfc99c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:03.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.112+0000 7f403be00640 1 --2- 172.21.15.12:0/211351411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bfc99c200 0x555bfc99c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.112+0000 7f403be00640 1 -- 172.21.15.12:0/211351411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555bfc99d3d0 msgr2=0x555bfc99d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:03.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.112+0000 7f403be00640 1 --2- 172.21.15.12:0/211351411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555bfc99d3d0 0x555bfc99d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.112+0000 7f403be00640 1 -- 172.21.15.12:0/211351411 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555bfc8c5080 con 0x555bfc99dd90 2024-03-23T23:03:03.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.112+0000 7f4041200640 1 --2- 172.21.15.12:0/211351411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555bfc99d3d0 0x555bfc99d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:03.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.112+0000 7f403be00640 1 --2- 172.21.15.12:0/211351411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc99dd90 0x555bfc9a4620 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f402802c8f0 tx=0x7f4028005b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68743e9aada4db8c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:03.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.112+0000 7f402e000640 1 -- 172.21.15.12:0/211351411 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4028004240 con 0x555bfc99dd90 2024-03-23T23:03:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.113+0000 7f402e000640 1 -- 172.21.15.12:0/211351411 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40280043e0 con 0x555bfc99dd90 2024-03-23T23:03:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.113+0000 7f402e000640 1 -- 172.21.15.12:0/211351411 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40280054b0 con 0x555bfc99dd90 2024-03-23T23:03:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.113+0000 7f404683d740 1 -- 172.21.15.12:0/211351411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc99dd90 msgr2=0x555bfc9a4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.113+0000 7f404683d740 1 --2- 172.21.15.12:0/211351411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc99dd90 0x555bfc9a4620 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f402802c8f0 tx=0x7f4028005b70 comp rx=0 tx=0).stop 2024-03-23T23:03:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.113+0000 7f4041c00640 1 --2- 172.21.15.12:0/1335128236 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f40240a2880 0x7f40240a4d40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f403409b8e0 tx=0x7f40340046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.114+0000 7f404683d740 1 -- 172.21.15.12:0/211351411 shutdown_connections 2024-03-23T23:03:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.114+0000 7f404683d740 1 --2- 172.21.15.12:0/211351411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc99dd90 0x555bfc9a4620 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.114+0000 7f404683d740 1 --2- 172.21.15.12:0/211351411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555bfc99d3d0 0x555bfc99d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.114+0000 7f404683d740 1 --2- 172.21.15.12:0/211351411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bfc99c200 0x555bfc99c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.114+0000 7f404683d740 1 -- 172.21.15.12:0/211351411 >> 172.21.15.12:0/211351411 conn(0x555bfc98ede0 msgr2=0x555bfc98f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.114+0000 7f404683d740 1 -- 172.21.15.12:0/211351411 shutdown_connections 2024-03-23T23:03:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.114+0000 7f404683d740 1 -- 172.21.15.12:0/211351411 wait complete. 2024-03-23T23:03:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.114+0000 7f404683d740 1 -- start start 2024-03-23T23:03:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.114+0000 7f404683d740 1 Processor -- start 2024-03-23T23:03:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.115+0000 7f404683d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bfc99c200 0x555bfc994f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.115+0000 7f404683d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc99d3d0 0x555bfc995440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.115+0000 7f404683d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555bfc99dd90 0x555bfc99a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.115+0000 7f404683d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bfc993750 con 0x555bfc99dd90 2024-03-23T23:03:03.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.115+0000 7f404683d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bfc8b40d0 con 0x555bfc99d3d0 2024-03-23T23:03:03.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.115+0000 7f404683d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bfc9938d0 con 0x555bfc99c200 2024-03-23T23:03:03.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.115+0000 7f404683d740 10 client.0 _ll_register_callbacks cb 0x555bfc8b8e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.115+0000 7f4041200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc99d3d0 0x555bfc995440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.115+0000 7f4041200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc99d3d0 0x555bfc995440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36076/0 (socket says 172.21.15.12:36076) 2024-03-23T23:03:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.115+0000 7f4041200640 1 -- 172.21.15.12:0/3063504359 learned_addr learned my addr 172.21.15.12:0/3063504359 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.115+0000 7f403be00640 1 --2- 172.21.15.12:0/3063504359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bfc99c200 0x555bfc994f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.115+0000 7f4041200640 1 -- 172.21.15.12:0/3063504359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bfc99c200 msgr2=0x555bfc994f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.116+0000 7f4041200640 1 --2- 172.21.15.12:0/3063504359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bfc99c200 0x555bfc994f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.116+0000 7f4041200640 1 -- 172.21.15.12:0/3063504359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555bfc99dd90 msgr2=0x555bfc99a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.116+0000 7f403be00640 1 --2- 172.21.15.12:0/3063504359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555bfc99dd90 0x555bfc99a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.116+0000 7f4041200640 1 --2- 172.21.15.12:0/3063504359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555bfc99dd90 0x555bfc99a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.116+0000 7f4041200640 1 -- 172.21.15.12:0/3063504359 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555bfc60c3f0 con 0x555bfc99d3d0 2024-03-23T23:03:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.116+0000 7f403be00640 1 --2- 172.21.15.12:0/3063504359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555bfc99dd90 0x555bfc99a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:03:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.116+0000 7f4041200640 1 --2- 172.21.15.12:0/3063504359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc99d3d0 0x555bfc995440 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f403000bf90 tx=0x7f40300128e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdc2680d5574d2d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.116+0000 7f4022a00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:03:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.116+0000 7f4022a00640 1 -- 172.21.15.12:0/3063504359 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403001f060 con 0x555bfc99d3d0 2024-03-23T23:03:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.116+0000 7f404683d740 1 -- 172.21.15.12:0/3063504359 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f40280043e0 con 0x555bfc99d3d0 2024-03-23T23:03:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.116+0000 7f4022a00640 1 -- 172.21.15.12:0/3063504359 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4030010900 con 0x555bfc99d3d0 2024-03-23T23:03:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.117+0000 7f4022a00640 1 -- 172.21.15.12:0/3063504359 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4030010b00 con 0x555bfc99d3d0 2024-03-23T23:03:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.117+0000 7f404683d740 10 client.25574 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.117+0000 7f404683d740 1 -- 172.21.15.12:0/3063504359 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x555bfc6d67d0 con 0x555bfc99d3d0 2024-03-23T23:03:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.117+0000 7f4022a00640 1 -- 172.21.15.12:0/3063504359 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40280043e0 con 0x555bfc99d3d0 2024-03-23T23:03:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.117+0000 7f4022a00640 1 -- 172.21.15.12:0/3063504359 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f4030019030 con 0x555bfc99d3d0 2024-03-23T23:03:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.117+0000 7f404683d740 10 client.25574 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.117+0000 7f404683d740 10 client.25574 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:03:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.117+0000 7f404683d740 20 client.25574 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.117+0000 7f404683d740 1 --2- 172.21.15.12:0/3063504359 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x555bfc9ba770 0x555bfc9bcb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.117+0000 7f404683d740 4 client.25574 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:03:03.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.117+0000 7f404683d740 1 -- 172.21.15.12:0/3063504359 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555bfc905800 con 0x555bfc9ba770 2024-03-23T23:03:03.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.118+0000 7f403be00640 1 --2- 172.21.15.12:0/3063504359 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x555bfc9ba770 0x555bfc9bcb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.118+0000 7f403be00640 1 --2- 172.21.15.12:0/3063504359 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x555bfc9ba770 0x555bfc9bcb90 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=95c2a14562cb93c0 server_cookie=6c8785b3b89937f4 in_seq=0 out_seq=0 2024-03-23T23:03:03.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.118+0000 7f4022a00640 10 client.25574 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:03:03.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.119+0000 7f4022a00640 1 -- 172.21.15.12:0/3063504359 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555bfc905800 con 0x555bfc9ba770 2024-03-23T23:03:03.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.119+0000 7f4022a00640 10 client.25574 handle_command_reply: tid=0 2024-03-23T23:03:03.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.119+0000 7f404683d740 1 client.25574 shutdown 2024-03-23T23:03:03.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.119+0000 7f404683d740 1 -- 172.21.15.12:0/3063504359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc99d3d0 msgr2=0x555bfc995440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.119+0000 7f404683d740 1 --2- 172.21.15.12:0/3063504359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc99d3d0 0x555bfc995440 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f403000bf90 tx=0x7f40300128e0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.120+0000 7f404683d740 1 -- 172.21.15.12:0/3063504359 shutdown_connections 2024-03-23T23:03:03.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.120+0000 7f404683d740 1 --2- 172.21.15.12:0/3063504359 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x555bfc9ba770 0x555bfc9bcb90 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.120+0000 7f404683d740 1 --2- 172.21.15.12:0/3063504359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555bfc99dd90 0x555bfc99a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.120+0000 7f404683d740 1 --2- 172.21.15.12:0/3063504359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc99d3d0 0x555bfc995440 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.120+0000 7f404683d740 1 --2- 172.21.15.12:0/3063504359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bfc99c200 0x555bfc994f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.120+0000 7f4022a00640 0 client.25574 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:03:03.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.120+0000 7f404683d740 1 -- 172.21.15.12:0/3063504359 >> 172.21.15.12:0/3063504359 conn(0x555bfc98ede0 msgr2=0x555bfc98e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:03.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.120+0000 7f404683d740 1 -- 172.21.15.12:0/3063504359 shutdown_connections 2024-03-23T23:03:03.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.120+0000 7f404683d740 1 -- 172.21.15.12:0/3063504359 wait complete. 2024-03-23T23:03:03.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.120+0000 7f404683d740 20 client.25574 trim_cache size 0 max 16384 2024-03-23T23:03:03.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.133+0000 7f404683d740 1 Processor -- start 2024-03-23T23:03:03.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.133+0000 7f404683d740 1 -- start start 2024-03-23T23:03:03.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.134+0000 7f404683d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555bfc98ede0 0x555bfc98d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.134+0000 7f404683d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc99c200 0x555bfc98de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.134+0000 7f404683d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bfc99d3d0 0x555bfc98e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.134+0000 7f404683d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c1a8bc0 con 0x555bfc98ede0 2024-03-23T23:03:03.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.134+0000 7f404683d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c1a66c0 con 0x555bfc99c200 2024-03-23T23:03:03.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.134+0000 7f404683d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c1a4760 con 0x555bfc99d3d0 2024-03-23T23:03:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.134+0000 7f4041200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc99c200 0x555bfc98de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.134+0000 7f403be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555bfc98ede0 0x555bfc98d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.134+0000 7f4041200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc99c200 0x555bfc98de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36086/0 (socket says 172.21.15.12:36086) 2024-03-23T23:03:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.134+0000 7f4041200640 1 -- 172.21.15.12:0/4115491783 learned_addr learned my addr 172.21.15.12:0/4115491783 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.134+0000 7f403be00640 1 --2- 172.21.15.12:0/4115491783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bfc99d3d0 0x555bfc98e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.134+0000 7f4041200640 1 -- 172.21.15.12:0/4115491783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bfc99d3d0 msgr2=0x555bfc98e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.134+0000 7f4041200640 1 --2- 172.21.15.12:0/4115491783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bfc99d3d0 0x555bfc98e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.134+0000 7f4041200640 1 -- 172.21.15.12:0/4115491783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555bfc98ede0 msgr2=0x555bfc98d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.134+0000 7f4041200640 1 --2- 172.21.15.12:0/4115491783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555bfc98ede0 0x555bfc98d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.134+0000 7f4041200640 1 -- 172.21.15.12:0/4115491783 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555bfc9ba530 con 0x555bfc99c200 2024-03-23T23:03:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.135+0000 7f403be00640 1 --2- 172.21.15.12:0/4115491783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555bfc98ede0 0x555bfc98d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:03:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.135+0000 7f4041200640 1 --2- 172.21.15.12:0/4115491783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc99c200 0x555bfc98de10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4030000c00 tx=0x7f4030013df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48839a72ce148d93 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:03.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.135+0000 7f402d600640 1 -- 172.21.15.12:0/4115491783 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403002a030 con 0x555bfc99c200 2024-03-23T23:03:03.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.135+0000 7f402d600640 1 -- 172.21.15.12:0/4115491783 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f403002c070 con 0x555bfc99c200 2024-03-23T23:03:03.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.135+0000 7f402d600640 1 -- 172.21.15.12:0/4115491783 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40300048a0 con 0x555bfc99c200 2024-03-23T23:03:03.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.136+0000 7f404683d740 1 -- 172.21.15.12:0/4115491783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc99c200 msgr2=0x555bfc98de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.136+0000 7f404683d740 1 --2- 172.21.15.12:0/4115491783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc99c200 0x555bfc98de10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4030000c00 tx=0x7f4030013df0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.136+0000 7f404683d740 1 -- 172.21.15.12:0/4115491783 shutdown_connections 2024-03-23T23:03:03.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.136+0000 7f404683d740 1 --2- 172.21.15.12:0/4115491783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bfc99d3d0 0x555bfc98e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.136+0000 7f404683d740 1 --2- 172.21.15.12:0/4115491783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc99c200 0x555bfc98de10 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.136+0000 7f404683d740 1 --2- 172.21.15.12:0/4115491783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555bfc98ede0 0x555bfc98d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.136+0000 7f404683d740 1 -- 172.21.15.12:0/4115491783 >> 172.21.15.12:0/4115491783 conn(0x555bfc9a4740 msgr2=0x555bfc993c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.137+0000 7f404683d740 1 -- 172.21.15.12:0/4115491783 shutdown_connections 2024-03-23T23:03:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.137+0000 7f404683d740 1 -- 172.21.15.12:0/4115491783 wait complete. 2024-03-23T23:03:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.137+0000 7f404683d740 1 -- start start 2024-03-23T23:03:03.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.137+0000 7f404683d740 1 Processor -- start 2024-03-23T23:03:03.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.138+0000 7f404683d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc98ede0 0x555bfc9ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.138+0000 7f404683d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555bfc99c200 0x555bfc9c9820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.138+0000 7f404683d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bfc99d3d0 0x555bfc9c9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.138+0000 7f404683d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c1a4d60 con 0x555bfc99c200 2024-03-23T23:03:03.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.138+0000 7f404683d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c1a6060 con 0x555bfc98ede0 2024-03-23T23:03:03.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.138+0000 7f404683d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c1a53a0 con 0x555bfc99d3d0 2024-03-23T23:03:03.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.138+0000 7f404683d740 10 client.0 _ll_register_callbacks cb 0x7f403000b840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:03.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.139+0000 7f403be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc98ede0 0x555bfc9ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.139+0000 7f403be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc98ede0 0x555bfc9ccd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36094/0 (socket says 172.21.15.12:36094) 2024-03-23T23:03:03.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.139+0000 7f4041200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555bfc99c200 0x555bfc9c9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.139+0000 7f403be00640 1 -- 172.21.15.12:0/753008565 learned_addr learned my addr 172.21.15.12:0/753008565 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:03.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.139+0000 7f403be00640 1 --2- 172.21.15.12:0/753008565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bfc99d3d0 0x555bfc9c9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.139+0000 7f403be00640 1 -- 172.21.15.12:0/753008565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bfc99d3d0 msgr2=0x555bfc9c9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:03.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.139+0000 7f403be00640 1 --2- 172.21.15.12:0/753008565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bfc99d3d0 0x555bfc9c9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.139+0000 7f403be00640 1 -- 172.21.15.12:0/753008565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555bfc99c200 msgr2=0x555bfc9c9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:03.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.139+0000 7f403be00640 1 --2- 172.21.15.12:0/753008565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555bfc99c200 0x555bfc9c9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.139+0000 7f403be00640 1 -- 172.21.15.12:0/753008565 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555bfca15550 con 0x555bfc98ede0 2024-03-23T23:03:03.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.139+0000 7f4041200640 1 --2- 172.21.15.12:0/753008565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555bfc99c200 0x555bfc9c9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:03:03.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.139+0000 7f403be00640 1 --2- 172.21.15.12:0/753008565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc98ede0 0x555bfc9ccd30 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4028036040 tx=0x7f4028009660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b192e83d81515d9d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:03.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.139+0000 7f4022a00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:03:03.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.140+0000 7f4022a00640 1 -- 172.21.15.12:0/753008565 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4028055070 con 0x555bfc98ede0 2024-03-23T23:03:03.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.140+0000 7f4022a00640 1 -- 172.21.15.12:0/753008565 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4028004ef0 con 0x555bfc98ede0 2024-03-23T23:03:03.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.140+0000 7f404683d740 1 -- 172.21.15.12:0/753008565 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f403002c070 con 0x555bfc98ede0 2024-03-23T23:03:03.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.140+0000 7f4022a00640 1 -- 172.21.15.12:0/753008565 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40280050f0 con 0x555bfc98ede0 2024-03-23T23:03:03.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.140+0000 7f404683d740 10 client.25580 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:03.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.140+0000 7f404683d740 1 -- 172.21.15.12:0/753008565 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x555bfca168c0 con 0x555bfc98ede0 2024-03-23T23:03:03.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.141+0000 7f4022a00640 1 -- 172.21.15.12:0/753008565 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f403002c070 con 0x555bfc98ede0 2024-03-23T23:03:03.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.141+0000 7f4022a00640 1 -- 172.21.15.12:0/753008565 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f4028034710 con 0x555bfc98ede0 2024-03-23T23:03:03.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.141+0000 7f404683d740 10 client.25580 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:03.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.141+0000 7f404683d740 10 client.25580 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:03:03.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.141+0000 7f404683d740 20 client.25580 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:03.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.141+0000 7f404683d740 1 --2- 172.21.15.12:0/753008565 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x555bfc99dd90 0x555bfca18db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.141+0000 7f404683d740 4 client.25580 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:03:03.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.141+0000 7f404683d740 1 -- 172.21.15.12:0/753008565 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555bfc8ffb50 con 0x555bfc99dd90 2024-03-23T23:03:03.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.141+0000 7f4041200640 1 --2- 172.21.15.12:0/753008565 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x555bfc99dd90 0x555bfca18db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.142+0000 7f4041200640 1 --2- 172.21.15.12:0/753008565 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x555bfc99dd90 0x555bfca18db0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=336674a949512d85 server_cookie=73d1ffdaf0a2c55a in_seq=0 out_seq=0 2024-03-23T23:03:03.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.142+0000 7f4022a00640 10 client.25580 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:03:03.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.142+0000 7f4022a00640 1 -- 172.21.15.12:0/753008565 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555bfc8ffb50 con 0x555bfc99dd90 2024-03-23T23:03:03.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.142+0000 7f4022a00640 10 client.25580 handle_command_reply: tid=0 2024-03-23T23:03:03.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.142+0000 7f404683d740 1 client.25580 shutdown 2024-03-23T23:03:03.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.142+0000 7f404683d740 1 -- 172.21.15.12:0/753008565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc98ede0 msgr2=0x555bfc9ccd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.142+0000 7f404683d740 1 --2- 172.21.15.12:0/753008565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc98ede0 0x555bfc9ccd30 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4028036040 tx=0x7f4028009660 comp rx=0 tx=0).stop 2024-03-23T23:03:03.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.143+0000 7f404683d740 1 -- 172.21.15.12:0/753008565 shutdown_connections 2024-03-23T23:03:03.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.143+0000 7f404683d740 1 --2- 172.21.15.12:0/753008565 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x555bfc99dd90 0x555bfca18db0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.143+0000 7f404683d740 1 --2- 172.21.15.12:0/753008565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bfc99d3d0 0x555bfc9c9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.143+0000 7f404683d740 1 --2- 172.21.15.12:0/753008565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555bfc99c200 0x555bfc9c9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.143+0000 7f404683d740 1 --2- 172.21.15.12:0/753008565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555bfc98ede0 0x555bfc9ccd30 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.143+0000 7f404683d740 1 -- 172.21.15.12:0/753008565 >> 172.21.15.12:0/753008565 conn(0x555bfc9a4740 msgr2=0x555bfc992040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:03.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.143+0000 7f4022a00640 0 client.25580 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:03:03.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.143+0000 7f404683d740 1 -- 172.21.15.12:0/753008565 shutdown_connections 2024-03-23T23:03:03.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.143+0000 7f404683d740 1 -- 172.21.15.12:0/753008565 wait complete. 2024-03-23T23:03:03.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.143+0000 7f404683d740 20 client.25580 trim_cache size 0 max 16384 2024-03-23T23:03:03.145 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:03:03.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.144+0000 7f4042600640 1 -- 172.21.15.12:0/1335128236 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f40240a2880 msgr2=0x7f40240a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.144+0000 7f4042600640 1 --2- 172.21.15.12:0/1335128236 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f40240a2880 0x7f40240a4d40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f403409b8e0 tx=0x7f40340046c0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.144+0000 7f4042600640 1 -- 172.21.15.12:0/1335128236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f403c103150 msgr2=0x7f403c19ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.144+0000 7f4042600640 1 --2- 172.21.15.12:0/1335128236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f403c103150 0x7f403c19ee40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f403400ec50 tx=0x7f403400c6a0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.144+0000 7f4042600640 1 -- 172.21.15.12:0/1335128236 shutdown_connections 2024-03-23T23:03:03.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.144+0000 7f4042600640 1 --2- 172.21.15.12:0/1335128236 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f40240a2880 0x7f40240a4d40 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.144+0000 7f4042600640 1 --2- 172.21.15.12:0/1335128236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f403c103150 0x7f403c19ee40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.144+0000 7f4042600640 1 --2- 172.21.15.12:0/1335128236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f403c102790 0x7f403c197dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.144+0000 7f4042600640 1 --2- 172.21.15.12:0/1335128236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f403c1015a0 0x7f403c197890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.144+0000 7f4042600640 1 -- 172.21.15.12:0/1335128236 >> 172.21.15.12:0/1335128236 conn(0x7f403c0fcb10 msgr2=0x7f403c105a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:03.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.144+0000 7f4042600640 1 -- 172.21.15.12:0/1335128236 shutdown_connections 2024-03-23T23:03:03.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.144+0000 7f4042600640 1 -- 172.21.15.12:0/1335128236 wait complete. 2024-03-23T23:03:03.166 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:03:03.167 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-23T23:03:03.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.391+0000 7f4203e00640 1 -- 172.21.15.12:0/2530320064 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4204102760 msgr2=0x7f4204104bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.391+0000 7f4203e00640 1 --2- 172.21.15.12:0/2530320064 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4204102760 0x7f4204104bd0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f41f0009900 tx=0x7f41f002bc30 comp rx=0 tx=0).stop 2024-03-23T23:03:03.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.391+0000 7f4203e00640 1 -- 172.21.15.12:0/2530320064 shutdown_connections 2024-03-23T23:03:03.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.391+0000 7f4203e00640 1 --2- 172.21.15.12:0/2530320064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4204105110 0x7f4204105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.391+0000 7f4203e00640 1 --2- 172.21.15.12:0/2530320064 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4204102760 0x7f4204104bd0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.391+0000 7f4203e00640 1 --2- 172.21.15.12:0/2530320064 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42040ffdf0 0x7f4204102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.391+0000 7f4203e00640 1 -- 172.21.15.12:0/2530320064 >> 172.21.15.12:0/2530320064 conn(0x7f42040fba10 msgr2=0x7f42040fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:03.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.392+0000 7f4203e00640 1 -- 172.21.15.12:0/2530320064 shutdown_connections 2024-03-23T23:03:03.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.392+0000 7f4203e00640 1 -- 172.21.15.12:0/2530320064 wait complete. 2024-03-23T23:03:03.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.393+0000 7f4203e00640 1 Processor -- start 2024-03-23T23:03:03.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.393+0000 7f4203e00640 1 -- start start 2024-03-23T23:03:03.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.394+0000 7f4203e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42040ffdf0 0x7f420419de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.394+0000 7f4203e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4204102760 0x7f420419e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.394+0000 7f4203e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4204105110 0x7f42041a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.394+0000 7f4203e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4204113840 con 0x7f4204105110 2024-03-23T23:03:03.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.394+0000 7f4203e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42041136c0 con 0x7f4204102760 2024-03-23T23:03:03.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.394+0000 7f4203e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42041139c0 con 0x7f42040ffdf0 2024-03-23T23:03:03.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.394+0000 7f4202a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42040ffdf0 0x7f420419de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.394+0000 7f4202a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42040ffdf0 0x7f420419de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:46034/0 (socket says 172.21.15.12:46034) 2024-03-23T23:03:03.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.394+0000 7f4202a00640 1 -- 172.21.15.12:0/4136139068 learned_addr learned my addr 172.21.15.12:0/4136139068 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:03.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.394+0000 7f4202000640 1 --2- 172.21.15.12:0/4136139068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4204102760 0x7f420419e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.394+0000 7f4203400640 1 --2- 172.21.15.12:0/4136139068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4204105110 0x7f42041a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.394+0000 7f4202a00640 1 -- 172.21.15.12:0/4136139068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4204102760 msgr2=0x7f420419e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.394+0000 7f4202a00640 1 --2- 172.21.15.12:0/4136139068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4204102760 0x7f420419e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.394+0000 7f4202a00640 1 -- 172.21.15.12:0/4136139068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4204105110 msgr2=0x7f42041a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.394+0000 7f4202a00640 1 --2- 172.21.15.12:0/4136139068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4204105110 0x7f42041a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.394+0000 7f4202a00640 1 -- 172.21.15.12:0/4136139068 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42041a5aa0 con 0x7f42040ffdf0 2024-03-23T23:03:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.395+0000 7f4202000640 1 --2- 172.21.15.12:0/4136139068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4204102760 0x7f420419e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:03:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.395+0000 7f4202a00640 1 --2- 172.21.15.12:0/4136139068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42040ffdf0 0x7f420419de80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f41f400e9f0 tx=0x7f41f400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.395+0000 7f41fb400640 1 -- 172.21.15.12:0/4136139068 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f400cd80 con 0x7f42040ffdf0 2024-03-23T23:03:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.395+0000 7f41fb400640 1 -- 172.21.15.12:0/4136139068 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f41f4010910 con 0x7f42040ffdf0 2024-03-23T23:03:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.395+0000 7f41fb400640 1 -- 172.21.15.12:0/4136139068 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f4018990 con 0x7f42040ffdf0 2024-03-23T23:03:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.395+0000 7f4203e00640 1 -- 172.21.15.12:0/4136139068 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42041a5d90 con 0x7f42040ffdf0 2024-03-23T23:03:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.395+0000 7f4203e00640 1 -- 172.21.15.12:0/4136139068 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42041a61a0 con 0x7f42040ffdf0 2024-03-23T23:03:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.400+0000 7f420c7a6740 1 Processor -- start 2024-03-23T23:03:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.400+0000 7f420c7a6740 1 -- start start 2024-03-23T23:03:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.400+0000 7f41fb400640 1 -- 172.21.15.12:0/4136139068 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f41f4010ac0 con 0x7f42040ffdf0 2024-03-23T23:03:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.400+0000 7f420c7a6740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628b27689f0 0x5628b2768e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.400+0000 7f420c7a6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5628b2769bc0 0x5628b276a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.400+0000 7f420c7a6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628b276a580 0x5628b2770e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.400+0000 7f420c7a6740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628b26cd2c0 con 0x5628b27689f0 2024-03-23T23:03:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.400+0000 7f420c7a6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628b2675c90 con 0x5628b2769bc0 2024-03-23T23:03:03.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.400+0000 7f420c7a6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628b26c9360 con 0x5628b276a580 2024-03-23T23:03:03.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.401+0000 7f4202a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5628b2769bc0 0x5628b276a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.401+0000 7f4202a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5628b2769bc0 0x5628b276a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36114/0 (socket says 172.21.15.12:36114) 2024-03-23T23:03:03.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.401+0000 7f4202a00640 1 -- 172.21.15.12:0/1631666834 learned_addr learned my addr 172.21.15.12:0/1631666834 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:03.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.401+0000 7f4202000640 1 --2- 172.21.15.12:0/1631666834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628b276a580 0x5628b2770e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.401+0000 7f4203400640 1 --2- 172.21.15.12:0/1631666834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628b27689f0 0x5628b2768e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.401+0000 7f4202a00640 1 -- 172.21.15.12:0/1631666834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628b276a580 msgr2=0x5628b2770e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:03.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.401+0000 7f4202a00640 1 --2- 172.21.15.12:0/1631666834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628b276a580 0x5628b2770e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.401+0000 7f4202a00640 1 -- 172.21.15.12:0/1631666834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628b27689f0 msgr2=0x5628b2768e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:03.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.401+0000 7f4202a00640 1 --2- 172.21.15.12:0/1631666834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628b27689f0 0x5628b2768e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.401+0000 7f4202a00640 1 -- 172.21.15.12:0/1631666834 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5628b269a080 con 0x5628b2769bc0 2024-03-23T23:03:03.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.401+0000 7f4203400640 1 --2- 172.21.15.12:0/1631666834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628b27689f0 0x5628b2768e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:03:03.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.402+0000 7f4202a00640 1 --2- 172.21.15.12:0/1631666834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5628b2769bc0 0x5628b276a040 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f41f40d2f50 tx=0x7f41f40d7570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c218ded4d54f441 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:03.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.402+0000 7f41fb400640 1 --2- 172.21.15.12:0/4136139068 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f41e80a2800 0x7f41e80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.402+0000 7f41efe00640 1 -- 172.21.15.12:0/1631666834 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f40e2070 con 0x5628b2769bc0 2024-03-23T23:03:03.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.402+0000 7f41efe00640 1 -- 172.21.15.12:0/1631666834 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f41f4070390 con 0x5628b2769bc0 2024-03-23T23:03:03.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.402+0000 7f41efe00640 1 -- 172.21.15.12:0/1631666834 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f4084dc0 con 0x5628b2769bc0 2024-03-23T23:03:03.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.403+0000 7f41fb400640 1 -- 172.21.15.12:0/4136139068 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f41f40ceeb0 con 0x7f42040ffdf0 2024-03-23T23:03:03.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.403+0000 7f4203400640 1 --2- 172.21.15.12:0/4136139068 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f41e80a2800 0x7f41e80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.403+0000 7f420c7a6740 1 -- 172.21.15.12:0/1631666834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5628b2769bc0 msgr2=0x5628b276a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.403+0000 7f420c7a6740 1 --2- 172.21.15.12:0/1631666834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5628b2769bc0 0x5628b276a040 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f41f40d2f50 tx=0x7f41f40d7570 comp rx=0 tx=0).stop 2024-03-23T23:03:03.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.404+0000 7f420c7a6740 1 -- 172.21.15.12:0/1631666834 shutdown_connections 2024-03-23T23:03:03.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.404+0000 7f420c7a6740 1 --2- 172.21.15.12:0/1631666834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628b276a580 0x5628b2770e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.404+0000 7f420c7a6740 1 --2- 172.21.15.12:0/1631666834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5628b2769bc0 0x5628b276a040 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.404+0000 7f420c7a6740 1 --2- 172.21.15.12:0/1631666834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628b27689f0 0x5628b2768e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.404+0000 7f420c7a6740 1 -- 172.21.15.12:0/1631666834 >> 172.21.15.12:0/1631666834 conn(0x5628b2763de0 msgr2=0x5628b27641e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:03.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.404+0000 7f420c7a6740 1 -- 172.21.15.12:0/1631666834 shutdown_connections 2024-03-23T23:03:03.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.404+0000 7f420c7a6740 1 -- 172.21.15.12:0/1631666834 wait complete. 2024-03-23T23:03:03.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.404+0000 7f4203400640 1 --2- 172.21.15.12:0/4136139068 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f41e80a2800 0x7f41e80a4cc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x5628b2769ac0 tx=0x7f41fc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:03.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.404+0000 7f420c7a6740 1 -- start start 2024-03-23T23:03:03.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.405+0000 7f420c7a6740 1 Processor -- start 2024-03-23T23:03:03.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.405+0000 7f420c7a6740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628b27689f0 0x5628b2772bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.405+0000 7f420c7a6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628b2769bc0 0x5628b2773110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.405+0000 7f420c7a6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5628b276a580 0x5628b277a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.405+0000 7f420c7a6740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628b2771590 con 0x5628b27689f0 2024-03-23T23:03:03.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.405+0000 7f420c7a6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628b26890d0 con 0x5628b276a580 2024-03-23T23:03:03.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.405+0000 7f420c7a6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628b2771710 con 0x5628b2769bc0 2024-03-23T23:03:03.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.406+0000 7f420c7a6740 10 client.0 _ll_register_callbacks cb 0x5628b268de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:03.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.406+0000 7f4202000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628b2769bc0 0x5628b2773110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.406+0000 7f4202000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628b2769bc0 0x5628b2773110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:46062/0 (socket says 172.21.15.12:46062) 2024-03-23T23:03:03.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.406+0000 7f4202000640 1 -- 172.21.15.12:0/1090623743 learned_addr learned my addr 172.21.15.12:0/1090623743 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:03.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.406+0000 7f4202000640 1 -- 172.21.15.12:0/1090623743 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5628b276a580 msgr2=0x5628b277a150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T23:03:03.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.406+0000 7f4202a00640 1 --2- 172.21.15.12:0/1090623743 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628b27689f0 0x5628b2772bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.406+0000 7f4202000640 1 --2- 172.21.15.12:0/1090623743 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5628b276a580 0x5628b277a150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.406+0000 7f4202000640 1 -- 172.21.15.12:0/1090623743 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628b27689f0 msgr2=0x5628b2772bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:03.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.406+0000 7f4202000640 1 --2- 172.21.15.12:0/1090623743 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628b27689f0 0x5628b2772bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.406+0000 7f4202000640 1 -- 172.21.15.12:0/1090623743 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5628b23e13f0 con 0x5628b2769bc0 2024-03-23T23:03:03.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.406+0000 7f4202000640 1 --2- 172.21.15.12:0/1090623743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628b2769bc0 0x5628b2773110 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f41f0006010 tx=0x7f41f0004250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=395a67e2574bad1a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:03.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.407+0000 7f41ecc00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:03:03.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.407+0000 7f41ecc00640 1 -- 172.21.15.12:0/1090623743 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f0005d50 con 0x5628b2769bc0 2024-03-23T23:03:03.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.407+0000 7f41ecc00640 1 -- 172.21.15.12:0/1090623743 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f41f0005230 con 0x5628b2769bc0 2024-03-23T23:03:03.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.407+0000 7f420c7a6740 1 -- 172.21.15.12:0/1090623743 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f41f4070390 con 0x5628b2769bc0 2024-03-23T23:03:03.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.407+0000 7f41ecc00640 1 -- 172.21.15.12:0/1090623743 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41f0005430 con 0x5628b2769bc0 2024-03-23T23:03:03.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.407+0000 7f420c7a6740 10 client.25561 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:03.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.407+0000 7f420c7a6740 1 -- 172.21.15.12:0/1090623743 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5628b24ab7d0 con 0x5628b2769bc0 2024-03-23T23:03:03.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.407+0000 7f41ecc00640 1 -- 172.21.15.12:0/1090623743 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f41f4070390 con 0x5628b2769bc0 2024-03-23T23:03:03.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.408+0000 7f41ecc00640 1 -- 172.21.15.12:0/1090623743 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f41f00038f0 con 0x5628b2769bc0 2024-03-23T23:03:03.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.408+0000 7f420c7a6740 10 client.25561 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:03.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.408+0000 7f420c7a6740 10 client.25561 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:03:03.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.408+0000 7f420c7a6740 20 client.25561 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:03.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.408+0000 7f420c7a6740 1 --2- 172.21.15.12:0/1090623743 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5628b277e6d0 0x5628b2780b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.408+0000 7f420c7a6740 4 client.25561 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:03:03.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.408+0000 7f420c7a6740 1 -- 172.21.15.12:0/1090623743 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5628b26da800 con 0x5628b277e6d0 2024-03-23T23:03:03.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.408+0000 7f4202a00640 1 --2- 172.21.15.12:0/1090623743 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5628b277e6d0 0x5628b2780b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.409+0000 7f4202a00640 1 --2- 172.21.15.12:0/1090623743 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5628b277e6d0 0x5628b2780b70 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2fa018d8b0feac3b server_cookie=135c8acbf3c6d87b in_seq=0 out_seq=0 2024-03-23T23:03:03.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.409+0000 7f41ecc00640 10 client.25561 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:03:03.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.409+0000 7f41ecc00640 1 -- 172.21.15.12:0/1090623743 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5628b26da800 con 0x5628b277e6d0 2024-03-23T23:03:03.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.409+0000 7f41ecc00640 10 client.25561 handle_command_reply: tid=0 2024-03-23T23:03:03.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.409+0000 7f420c7a6740 1 client.25561 shutdown 2024-03-23T23:03:03.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.410+0000 7f420c7a6740 1 -- 172.21.15.12:0/1090623743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628b2769bc0 msgr2=0x5628b2773110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.410+0000 7f420c7a6740 1 --2- 172.21.15.12:0/1090623743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628b2769bc0 0x5628b2773110 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f41f0006010 tx=0x7f41f0004250 comp rx=0 tx=0).stop 2024-03-23T23:03:03.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.410+0000 7f420c7a6740 1 -- 172.21.15.12:0/1090623743 shutdown_connections 2024-03-23T23:03:03.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.410+0000 7f420c7a6740 1 --2- 172.21.15.12:0/1090623743 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5628b276a580 0x5628b277a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.410+0000 7f420c7a6740 1 --2- 172.21.15.12:0/1090623743 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5628b277e6d0 0x5628b2780b70 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.410+0000 7f420c7a6740 1 --2- 172.21.15.12:0/1090623743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628b2769bc0 0x5628b2773110 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.410+0000 7f41ecc00640 0 client.25561 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:03:03.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.410+0000 7f420c7a6740 1 --2- 172.21.15.12:0/1090623743 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628b27689f0 0x5628b2772bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.410+0000 7f420c7a6740 1 -- 172.21.15.12:0/1090623743 >> 172.21.15.12:0/1090623743 conn(0x5628b2763de0 msgr2=0x5628b2763160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:03.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.410+0000 7f420c7a6740 1 -- 172.21.15.12:0/1090623743 shutdown_connections 2024-03-23T23:03:03.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.410+0000 7f420c7a6740 1 -- 172.21.15.12:0/1090623743 wait complete. 2024-03-23T23:03:03.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.410+0000 7f420c7a6740 20 client.25561 trim_cache size 0 max 16384 2024-03-23T23:03:03.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.437+0000 7f420c7a6740 1 Processor -- start 2024-03-23T23:03:03.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.437+0000 7f420c7a6740 1 -- start start 2024-03-23T23:03:03.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.437+0000 7f420c7a6740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628b27689f0 0x5628b2762fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.437+0000 7f420c7a6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628b2769bc0 0x5628b277a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.437+0000 7f420c7a6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5628b276a580 0x5628b2785430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.437+0000 7f420c7a6740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42041b13e0 con 0x5628b27689f0 2024-03-23T23:03:03.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.437+0000 7f420c7a6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42041aeee0 con 0x5628b276a580 2024-03-23T23:03:03.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.437+0000 7f420c7a6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42041acf80 con 0x5628b2769bc0 2024-03-23T23:03:03.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.438+0000 7f4203400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5628b276a580 0x5628b2785430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.438+0000 7f4202000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628b2769bc0 0x5628b277a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.438+0000 7f4203400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5628b276a580 0x5628b2785430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36128/0 (socket says 172.21.15.12:36128) 2024-03-23T23:03:03.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.438+0000 7f4203400640 1 -- 172.21.15.12:0/4128795485 learned_addr learned my addr 172.21.15.12:0/4128795485 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:03.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.438+0000 7f4202a00640 1 --2- 172.21.15.12:0/4128795485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628b27689f0 0x5628b2762fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.438+0000 7f4203400640 1 -- 172.21.15.12:0/4128795485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628b2769bc0 msgr2=0x5628b277a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:03.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.438+0000 7f4203400640 1 --2- 172.21.15.12:0/4128795485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628b2769bc0 0x5628b277a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.438+0000 7f4203400640 1 -- 172.21.15.12:0/4128795485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628b27689f0 msgr2=0x5628b2762fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:03.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.438+0000 7f4203400640 1 --2- 172.21.15.12:0/4128795485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628b27689f0 0x5628b2762fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.438+0000 7f4203400640 1 -- 172.21.15.12:0/4128795485 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5628b2769640 con 0x5628b276a580 2024-03-23T23:03:03.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.438+0000 7f4203400640 1 --2- 172.21.15.12:0/4128795485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5628b276a580 0x5628b2785430 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f41fc003d50 tx=0x7f41fc017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5561b8c8afcfa348 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:03.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.439+0000 7f41ef400640 1 -- 172.21.15.12:0/4128795485 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc029480 con 0x5628b276a580 2024-03-23T23:03:03.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.439+0000 7f41ef400640 1 -- 172.21.15.12:0/4128795485 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f41fc029b00 con 0x5628b276a580 2024-03-23T23:03:03.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.439+0000 7f41ef400640 1 -- 172.21.15.12:0/4128795485 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc01fb70 con 0x5628b276a580 2024-03-23T23:03:03.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.440+0000 7f420c7a6740 1 -- 172.21.15.12:0/4128795485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5628b276a580 msgr2=0x5628b2785430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.440+0000 7f420c7a6740 1 --2- 172.21.15.12:0/4128795485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5628b276a580 0x5628b2785430 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f41fc003d50 tx=0x7f41fc017000 comp rx=0 tx=0).stop 2024-03-23T23:03:03.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.440+0000 7f420c7a6740 1 -- 172.21.15.12:0/4128795485 shutdown_connections 2024-03-23T23:03:03.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.440+0000 7f420c7a6740 1 --2- 172.21.15.12:0/4128795485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5628b276a580 0x5628b2785430 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.440+0000 7f420c7a6740 1 --2- 172.21.15.12:0/4128795485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628b2769bc0 0x5628b277a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.440+0000 7f420c7a6740 1 --2- 172.21.15.12:0/4128795485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628b27689f0 0x5628b2762fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.440+0000 7f420c7a6740 1 -- 172.21.15.12:0/4128795485 >> 172.21.15.12:0/4128795485 conn(0x5628b2763de0 msgr2=0x5628b27628d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:03.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.440+0000 7f420c7a6740 1 -- 172.21.15.12:0/4128795485 shutdown_connections 2024-03-23T23:03:03.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.440+0000 7f420c7a6740 1 -- 172.21.15.12:0/4128795485 wait complete. 2024-03-23T23:03:03.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.440+0000 7f420c7a6740 1 -- start start 2024-03-23T23:03:03.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.441+0000 7f420c7a6740 1 Processor -- start 2024-03-23T23:03:03.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.441+0000 7f420c7a6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5628b27689f0 0x5628b2779c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.441+0000 7f420c7a6740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628b2769bc0 0x5628b2778290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.441+0000 7f420c7a6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628b276a580 0x5628b27787d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.441+0000 7f420c7a6740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42041ad580 con 0x5628b2769bc0 2024-03-23T23:03:03.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.441+0000 7f420c7a6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42041ae880 con 0x5628b27689f0 2024-03-23T23:03:03.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.441+0000 7f420c7a6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42041adbc0 con 0x5628b276a580 2024-03-23T23:03:03.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.441+0000 7f420c7a6740 10 client.0 _ll_register_callbacks cb 0x7f41fc0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:03.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.442+0000 7f4202a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5628b27689f0 0x5628b2779c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.442+0000 7f4202a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5628b27689f0 0x5628b2779c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36134/0 (socket says 172.21.15.12:36134) 2024-03-23T23:03:03.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.442+0000 7f4202a00640 1 -- 172.21.15.12:0/2988822222 learned_addr learned my addr 172.21.15.12:0/2988822222 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:03.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.442+0000 7f4203400640 1 --2- 172.21.15.12:0/2988822222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628b276a580 0x5628b27787d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.442+0000 7f4202000640 1 --2- 172.21.15.12:0/2988822222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628b2769bc0 0x5628b2778290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.442+0000 7f4203400640 1 -- 172.21.15.12:0/2988822222 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5628b27689f0 msgr2=0x5628b2779c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:03.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.442+0000 7f4203400640 1 --2- 172.21.15.12:0/2988822222 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5628b27689f0 0x5628b2779c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.442+0000 7f4203400640 1 -- 172.21.15.12:0/2988822222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628b2769bc0 msgr2=0x5628b2778290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.442+0000 7f4203400640 1 --2- 172.21.15.12:0/2988822222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628b2769bc0 0x5628b2778290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.442+0000 7f4203400640 1 -- 172.21.15.12:0/2988822222 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5628b2778d10 con 0x5628b276a580 2024-03-23T23:03:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.442+0000 7f4203400640 1 --2- 172.21.15.12:0/2988822222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628b276a580 0x5628b27787d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f41fc019040 tx=0x7f41fc004f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a001ec0cbc5b526e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.442+0000 7f41ecc00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:03:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.443+0000 7f41ecc00640 1 -- 172.21.15.12:0/2988822222 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc029890 con 0x5628b276a580 2024-03-23T23:03:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.443+0000 7f41ecc00640 1 -- 172.21.15.12:0/2988822222 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f41fc01f8f0 con 0x5628b276a580 2024-03-23T23:03:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.443+0000 7f420c7a6740 1 -- 172.21.15.12:0/2988822222 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f41fc029b00 con 0x5628b276a580 2024-03-23T23:03:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.443+0000 7f41ecc00640 1 -- 172.21.15.12:0/2988822222 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41fc01faf0 con 0x5628b276a580 2024-03-23T23:03:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.443+0000 7f420c7a6740 10 client.25567 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.443+0000 7f420c7a6740 1 -- 172.21.15.12:0/2988822222 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5628b27dacc0 con 0x5628b276a580 2024-03-23T23:03:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.443+0000 7f41ecc00640 1 -- 172.21.15.12:0/2988822222 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f41fc029b00 con 0x5628b276a580 2024-03-23T23:03:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.443+0000 7f41ecc00640 1 -- 172.21.15.12:0/2988822222 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f41fc030020 con 0x5628b276a580 2024-03-23T23:03:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.444+0000 7f420c7a6740 10 client.25567 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:03.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.444+0000 7f420c7a6740 10 client.25567 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:03:03.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.444+0000 7f420c7a6740 20 client.25567 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:03.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.444+0000 7f420c7a6740 1 --2- 172.21.15.12:0/2988822222 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5628b27db550 0x5628b27dd970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.444+0000 7f420c7a6740 4 client.25567 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:03:03.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.444+0000 7f420c7a6740 1 -- 172.21.15.12:0/2988822222 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5628b26d4b50 con 0x5628b27db550 2024-03-23T23:03:03.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.444+0000 7f4202a00640 1 --2- 172.21.15.12:0/2988822222 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5628b27db550 0x5628b27dd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.445+0000 7f4202a00640 1 --2- 172.21.15.12:0/2988822222 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5628b27db550 0x5628b27dd970 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4d9c962110179ebf server_cookie=d83bde4d24162b6 in_seq=0 out_seq=0 2024-03-23T23:03:03.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.445+0000 7f41ecc00640 10 client.25567 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:03:03.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.445+0000 7f41ecc00640 1 -- 172.21.15.12:0/2988822222 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5628b26d4b50 con 0x5628b27db550 2024-03-23T23:03:03.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.445+0000 7f41ecc00640 10 client.25567 handle_command_reply: tid=0 2024-03-23T23:03:03.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.445+0000 7f420c7a6740 1 client.25567 shutdown 2024-03-23T23:03:03.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.445+0000 7f420c7a6740 1 -- 172.21.15.12:0/2988822222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628b276a580 msgr2=0x5628b27787d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.445+0000 7f420c7a6740 1 --2- 172.21.15.12:0/2988822222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628b276a580 0x5628b27787d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f41fc019040 tx=0x7f41fc004f40 comp rx=0 tx=0).stop 2024-03-23T23:03:03.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.445+0000 7f420c7a6740 1 -- 172.21.15.12:0/2988822222 shutdown_connections 2024-03-23T23:03:03.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.445+0000 7f420c7a6740 1 --2- 172.21.15.12:0/2988822222 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5628b27db550 0x5628b27dd970 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.445+0000 7f420c7a6740 1 --2- 172.21.15.12:0/2988822222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628b276a580 0x5628b27787d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.445+0000 7f420c7a6740 1 --2- 172.21.15.12:0/2988822222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628b2769bc0 0x5628b2778290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.445+0000 7f420c7a6740 1 --2- 172.21.15.12:0/2988822222 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5628b27689f0 0x5628b2779c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.445+0000 7f41ecc00640 0 client.25567 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:03:03.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.445+0000 7f420c7a6740 1 -- 172.21.15.12:0/2988822222 >> 172.21.15.12:0/2988822222 conn(0x5628b2763de0 msgr2=0x5628b2763160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:03.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.445+0000 7f420c7a6740 1 -- 172.21.15.12:0/2988822222 shutdown_connections 2024-03-23T23:03:03.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.446+0000 7f420c7a6740 1 -- 172.21.15.12:0/2988822222 wait complete. 2024-03-23T23:03:03.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.446+0000 7f420c7a6740 20 client.25567 trim_cache size 0 max 16384 2024-03-23T23:03:03.448 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:03:03.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.447+0000 7f4203e00640 1 -- 172.21.15.12:0/4136139068 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f41e80a2800 msgr2=0x7f41e80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.447+0000 7f4203e00640 1 --2- 172.21.15.12:0/4136139068 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f41e80a2800 0x7f41e80a4cc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x5628b2769ac0 tx=0x7f41fc009500 comp rx=0 tx=0).stop 2024-03-23T23:03:03.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.447+0000 7f4203e00640 1 -- 172.21.15.12:0/4136139068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42040ffdf0 msgr2=0x7f420419de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.447+0000 7f4203e00640 1 --2- 172.21.15.12:0/4136139068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42040ffdf0 0x7f420419de80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f41f400e9f0 tx=0x7f41f400eeb0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.447+0000 7f4203e00640 1 -- 172.21.15.12:0/4136139068 shutdown_connections 2024-03-23T23:03:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.447+0000 7f4203e00640 1 --2- 172.21.15.12:0/4136139068 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f41e80a2800 0x7f41e80a4cc0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.447+0000 7f4203e00640 1 --2- 172.21.15.12:0/4136139068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4204105110 0x7f42041a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.447+0000 7f4203e00640 1 --2- 172.21.15.12:0/4136139068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4204102760 0x7f420419e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.447+0000 7f4203e00640 1 --2- 172.21.15.12:0/4136139068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42040ffdf0 0x7f420419de80 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.447+0000 7f4203e00640 1 -- 172.21.15.12:0/4136139068 >> 172.21.15.12:0/4136139068 conn(0x7f42040fba10 msgr2=0x7f4204100770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.447+0000 7f4203e00640 1 -- 172.21.15.12:0/4136139068 shutdown_connections 2024-03-23T23:03:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.447+0000 7f4203e00640 1 -- 172.21.15.12:0/4136139068 wait complete. 2024-03-23T23:03:03.470 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:03:03.470 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-23T23:03:03.534 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:03 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3390033410' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:03:03.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.657+0000 7f382d200640 1 -- 172.21.15.12:0/4072316904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3828102790 msgr2=0x7f3828102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.657+0000 7f382d200640 1 --2- 172.21.15.12:0/4072316904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3828102790 0x7f3828102c10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3814009900 tx=0x7f381402bc30 comp rx=0 tx=0).stop 2024-03-23T23:03:03.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.657+0000 7f382d200640 1 -- 172.21.15.12:0/4072316904 shutdown_connections 2024-03-23T23:03:03.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.657+0000 7f382d200640 1 --2- 172.21.15.12:0/4072316904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3828103150 0x7f38281099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.657+0000 7f382d200640 1 --2- 172.21.15.12:0/4072316904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3828102790 0x7f3828102c10 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.657+0000 7f382d200640 1 --2- 172.21.15.12:0/4072316904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38281015a0 0x7f38281019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.657+0000 7f382d200640 1 -- 172.21.15.12:0/4072316904 >> 172.21.15.12:0/4072316904 conn(0x7f38280fcb10 msgr2=0x7f38280fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:03.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.657+0000 7f382d200640 1 -- 172.21.15.12:0/4072316904 shutdown_connections 2024-03-23T23:03:03.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.657+0000 7f382d200640 1 -- 172.21.15.12:0/4072316904 wait complete. 2024-03-23T23:03:03.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.659+0000 7f382d200640 1 Processor -- start 2024-03-23T23:03:03.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.659+0000 7f382d200640 1 -- start start 2024-03-23T23:03:03.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.659+0000 7f382d200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38281015a0 0x7f3828075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.659+0000 7f382d200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3828102790 0x7f3828073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.660+0000 7f3827400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38281015a0 0x7f3828075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.660+0000 7f382d200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3828103150 0x7f3828073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.660+0000 7f382d200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382810d270 con 0x7f3828102790 2024-03-23T23:03:03.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.660+0000 7f382d200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382810d0f0 con 0x7f38281015a0 2024-03-23T23:03:03.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.660+0000 7f382d200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382810d3f0 con 0x7f3828103150 2024-03-23T23:03:03.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.660+0000 7f3827400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38281015a0 0x7f3828075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36146/0 (socket says 172.21.15.12:36146) 2024-03-23T23:03:03.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.660+0000 7f3827400640 1 -- 172.21.15.12:0/129674147 learned_addr learned my addr 172.21.15.12:0/129674147 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:03.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.660+0000 7f3826a00640 1 --2- 172.21.15.12:0/129674147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3828102790 0x7f3828073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.660+0000 7f3827e00640 1 --2- 172.21.15.12:0/129674147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3828103150 0x7f3828073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.660+0000 7f3827400640 1 -- 172.21.15.12:0/129674147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3828103150 msgr2=0x7f3828073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.660+0000 7f3827400640 1 --2- 172.21.15.12:0/129674147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3828103150 0x7f3828073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.660+0000 7f3827400640 1 -- 172.21.15.12:0/129674147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3828102790 msgr2=0x7f3828073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.660+0000 7f3827400640 1 --2- 172.21.15.12:0/129674147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3828102790 0x7f3828073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.660+0000 7f3827400640 1 -- 172.21.15.12:0/129674147 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3828074860 con 0x7f38281015a0 2024-03-23T23:03:03.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.661+0000 7f3827e00640 1 --2- 172.21.15.12:0/129674147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3828103150 0x7f3828073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:03:03.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.661+0000 7f3827400640 1 --2- 172.21.15.12:0/129674147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38281015a0 0x7f3828075450 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f381800e9c0 tx=0x7f381800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:03.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.661+0000 7f381fe00640 1 -- 172.21.15.12:0/129674147 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381800cd80 con 0x7f38281015a0 2024-03-23T23:03:03.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.661+0000 7f381fe00640 1 -- 172.21.15.12:0/129674147 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38180035f0 con 0x7f38281015a0 2024-03-23T23:03:03.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.661+0000 7f381fe00640 1 -- 172.21.15.12:0/129674147 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3818010690 con 0x7f38281015a0 2024-03-23T23:03:03.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.661+0000 7f382d200640 1 -- 172.21.15.12:0/129674147 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38281a19a0 con 0x7f38281015a0 2024-03-23T23:03:03.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.661+0000 7f382d200640 1 -- 172.21.15.12:0/129674147 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38281a1e70 con 0x7f38281015a0 2024-03-23T23:03:03.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.664+0000 7f383144f740 1 Processor -- start 2024-03-23T23:03:03.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.665+0000 7f383144f740 1 -- start start 2024-03-23T23:03:03.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.665+0000 7f383144f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d2e9f0 0x564bb5d2ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.665+0000 7f383144f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb5d2fbc0 0x564bb5d30040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.665+0000 7f383144f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564bb5d30580 0x564bb5d36e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.665+0000 7f383144f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bb5c932c0 con 0x564bb5d30580 2024-03-23T23:03:03.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.665+0000 7f383144f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bb5c3bc90 con 0x564bb5d2fbc0 2024-03-23T23:03:03.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.665+0000 7f383144f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bb5c8f360 con 0x564bb5d2e9f0 2024-03-23T23:03:03.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.665+0000 7f3827e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d2e9f0 0x564bb5d2ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.666+0000 7f3827e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d2e9f0 0x564bb5d2ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:46102/0 (socket says 172.21.15.12:46102) 2024-03-23T23:03:03.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.666+0000 7f3827e00640 1 -- 172.21.15.12:0/2741838376 learned_addr learned my addr 172.21.15.12:0/2741838376 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:03.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.666+0000 7f381fe00640 1 -- 172.21.15.12:0/129674147 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f38180037a0 con 0x7f38281015a0 2024-03-23T23:03:03.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.666+0000 7f3826a00640 1 --2- 172.21.15.12:0/2741838376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564bb5d30580 0x564bb5d36e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.666+0000 7f3827e00640 1 -- 172.21.15.12:0/2741838376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb5d2fbc0 msgr2=0x564bb5d30040 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-23T23:03:03.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.666+0000 7f3827e00640 1 --2- 172.21.15.12:0/2741838376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb5d2fbc0 0x564bb5d30040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.666+0000 7f3827e00640 1 -- 172.21.15.12:0/2741838376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564bb5d30580 msgr2=0x564bb5d36e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:03.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.666+0000 7f3827e00640 1 --2- 172.21.15.12:0/2741838376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564bb5d30580 0x564bb5d36e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.666+0000 7f3827e00640 1 -- 172.21.15.12:0/2741838376 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564bb5c60080 con 0x564bb5d2e9f0 2024-03-23T23:03:03.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.666+0000 7f3826a00640 1 --2- 172.21.15.12:0/2741838376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564bb5d30580 0x564bb5d36e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:03:03.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.667+0000 7f3827e00640 1 --2- 172.21.15.12:0/2741838376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d2e9f0 0x564bb5d2ee10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3820009f90 tx=0x7f382000f7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f572955b9590ccda server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:03.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.667+0000 7f381cc00640 1 -- 172.21.15.12:0/2741838376 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3820003b70 con 0x564bb5d2e9f0 2024-03-23T23:03:03.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.667+0000 7f381cc00640 1 -- 172.21.15.12:0/2741838376 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3820003d10 con 0x564bb5d2e9f0 2024-03-23T23:03:03.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.667+0000 7f381cc00640 1 -- 172.21.15.12:0/2741838376 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3820011a30 con 0x564bb5d2e9f0 2024-03-23T23:03:03.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.668+0000 7f383144f740 1 -- 172.21.15.12:0/2741838376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d2e9f0 msgr2=0x564bb5d2ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.668+0000 7f383144f740 1 --2- 172.21.15.12:0/2741838376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d2e9f0 0x564bb5d2ee10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3820009f90 tx=0x7f382000f7f0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.668+0000 7f383144f740 1 -- 172.21.15.12:0/2741838376 shutdown_connections 2024-03-23T23:03:03.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.668+0000 7f383144f740 1 --2- 172.21.15.12:0/2741838376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564bb5d30580 0x564bb5d36e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.668+0000 7f383144f740 1 --2- 172.21.15.12:0/2741838376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb5d2fbc0 0x564bb5d30040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.668+0000 7f383144f740 1 --2- 172.21.15.12:0/2741838376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d2e9f0 0x564bb5d2ee10 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.668+0000 7f383144f740 1 -- 172.21.15.12:0/2741838376 >> 172.21.15.12:0/2741838376 conn(0x564bb5d29de0 msgr2=0x564bb5d2a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:03.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.668+0000 7f383144f740 1 -- 172.21.15.12:0/2741838376 shutdown_connections 2024-03-23T23:03:03.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.668+0000 7f383144f740 1 -- 172.21.15.12:0/2741838376 wait complete. 2024-03-23T23:03:03.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.668+0000 7f381fe00640 1 --2- 172.21.15.12:0/129674147 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f380c0a27a0 0x7f380c0a4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.668+0000 7f381fe00640 1 -- 172.21.15.12:0/129674147 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f38180ceef0 con 0x7f38281015a0 2024-03-23T23:03:03.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.669+0000 7f383144f740 1 -- start start 2024-03-23T23:03:03.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.669+0000 7f3826a00640 1 --2- 172.21.15.12:0/129674147 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f380c0a27a0 0x7f380c0a4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.669+0000 7f383144f740 1 Processor -- start 2024-03-23T23:03:03.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.669+0000 7f383144f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564bb5d2e9f0 0x564bb5d38c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.669+0000 7f383144f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb5d2fbc0 0x564bb5d391c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.670+0000 7f383144f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d30580 0x564bb5d40200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.670+0000 7f383144f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bb5d37590 con 0x564bb5d2e9f0 2024-03-23T23:03:03.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.670+0000 7f383144f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bb5c4f0d0 con 0x564bb5d2fbc0 2024-03-23T23:03:03.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.670+0000 7f383144f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bb5d37710 con 0x564bb5d30580 2024-03-23T23:03:03.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.670+0000 7f383144f740 10 client.0 _ll_register_callbacks cb 0x564bb5c53e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:03.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.670+0000 7f3827e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d30580 0x564bb5d40200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.670+0000 7f3827e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d30580 0x564bb5d40200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:46110/0 (socket says 172.21.15.12:46110) 2024-03-23T23:03:03.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.670+0000 7f3827e00640 1 -- 172.21.15.12:0/2084250416 learned_addr learned my addr 172.21.15.12:0/2084250416 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:03.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.670+0000 7f3827400640 1 --2- 172.21.15.12:0/2084250416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564bb5d2e9f0 0x564bb5d38c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.670+0000 7f3827e00640 1 -- 172.21.15.12:0/2084250416 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb5d2fbc0 msgr2=0x564bb5d391c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-23T23:03:03.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.670+0000 7f3827e00640 1 --2- 172.21.15.12:0/2084250416 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb5d2fbc0 0x564bb5d391c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.670+0000 7f3827e00640 1 -- 172.21.15.12:0/2084250416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564bb5d2e9f0 msgr2=0x564bb5d38c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:03.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.670+0000 7f3827e00640 1 --2- 172.21.15.12:0/2084250416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564bb5d2e9f0 0x564bb5d38c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.670+0000 7f3827e00640 1 -- 172.21.15.12:0/2084250416 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564bb59a73f0 con 0x564bb5d30580 2024-03-23T23:03:03.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.671+0000 7f3826a00640 1 --2- 172.21.15.12:0/129674147 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f380c0a27a0 0x7f380c0a4c60 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f381402c240 tx=0x7f381402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:03.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.671+0000 7f3827e00640 1 --2- 172.21.15.12:0/2084250416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d30580 0x564bb5d40200 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3820009370 tx=0x7f3820003ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6bd83542cffb119 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:03.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.671+0000 7f3811600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:03:03.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.671+0000 7f3811600640 1 -- 172.21.15.12:0/2084250416 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3820004c50 con 0x564bb5d30580 2024-03-23T23:03:03.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.671+0000 7f3811600640 1 -- 172.21.15.12:0/2084250416 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f3820004df0 con 0x564bb5d30580 2024-03-23T23:03:03.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.671+0000 7f383144f740 1 -- 172.21.15.12:0/2084250416 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3820003d10 con 0x564bb5d30580 2024-03-23T23:03:03.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.671+0000 7f3811600640 1 -- 172.21.15.12:0/2084250416 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38200117b0 con 0x564bb5d30580 2024-03-23T23:03:03.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.671+0000 7f383144f740 10 client.25576 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:03.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.671+0000 7f383144f740 1 -- 172.21.15.12:0/2084250416 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x564bb5a717d0 con 0x564bb5d30580 2024-03-23T23:03:03.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.672+0000 7f3811600640 1 -- 172.21.15.12:0/2084250416 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3820003d10 con 0x564bb5d30580 2024-03-23T23:03:03.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.673+0000 7f3811600640 1 -- 172.21.15.12:0/2084250416 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f3820003fe0 con 0x564bb5d30580 2024-03-23T23:03:03.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.673+0000 7f383144f740 10 client.25576 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:03.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.673+0000 7f383144f740 10 client.25576 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:03.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.673+0000 7f383144f740 20 client.25576 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:03.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.673+0000 7f383144f740 1 --2- 172.21.15.12:0/2084250416 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x564bb5d44320 0x564bb5d46710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.673+0000 7f383144f740 4 client.25576 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:03:03.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.673+0000 7f383144f740 1 -- 172.21.15.12:0/2084250416 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564bb5ca0800 con 0x564bb5d44320 2024-03-23T23:03:03.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.673+0000 7f3827400640 1 --2- 172.21.15.12:0/2084250416 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x564bb5d44320 0x564bb5d46710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.673+0000 7f3827400640 1 --2- 172.21.15.12:0/2084250416 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x564bb5d44320 0x564bb5d46710 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddd71b2d331997b2 server_cookie=4e55fd19a5608cd2 in_seq=0 out_seq=0 2024-03-23T23:03:03.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.673+0000 7f3811600640 10 client.25576 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:03.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.674+0000 7f3811600640 1 -- 172.21.15.12:0/2084250416 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564bb5ca0800 con 0x564bb5d44320 2024-03-23T23:03:03.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.674+0000 7f3811600640 10 client.25576 handle_command_reply: tid=0 2024-03-23T23:03:03.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.674+0000 7f383144f740 1 client.25576 shutdown 2024-03-23T23:03:03.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.674+0000 7f383144f740 1 -- 172.21.15.12:0/2084250416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d30580 msgr2=0x564bb5d40200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.674+0000 7f383144f740 1 --2- 172.21.15.12:0/2084250416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d30580 0x564bb5d40200 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3820009370 tx=0x7f3820003ba0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.675+0000 7f383144f740 1 -- 172.21.15.12:0/2084250416 shutdown_connections 2024-03-23T23:03:03.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.675+0000 7f383144f740 1 --2- 172.21.15.12:0/2084250416 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x564bb5d44320 0x564bb5d46710 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.675+0000 7f383144f740 1 --2- 172.21.15.12:0/2084250416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d30580 0x564bb5d40200 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.675+0000 7f383144f740 1 --2- 172.21.15.12:0/2084250416 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb5d2fbc0 0x564bb5d391c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.675+0000 7f383144f740 1 --2- 172.21.15.12:0/2084250416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564bb5d2e9f0 0x564bb5d38c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.675+0000 7f383144f740 1 -- 172.21.15.12:0/2084250416 >> 172.21.15.12:0/2084250416 conn(0x564bb5d29de0 msgr2=0x564bb5d29160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:03.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.675+0000 7f3811600640 0 client.25576 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:03.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.675+0000 7f383144f740 1 -- 172.21.15.12:0/2084250416 shutdown_connections 2024-03-23T23:03:03.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.675+0000 7f383144f740 1 -- 172.21.15.12:0/2084250416 wait complete. 2024-03-23T23:03:03.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.675+0000 7f383144f740 20 client.25576 trim_cache size 0 max 16384 2024-03-23T23:03:03.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.688+0000 7f383144f740 1 Processor -- start 2024-03-23T23:03:03.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.688+0000 7f383144f740 1 -- start start 2024-03-23T23:03:03.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.689+0000 7f383144f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d2e9f0 0x564bb5d36ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.689+0000 7f383144f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564bb5d2fbc0 0x564bb5d401c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.689+0000 7f383144f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb5d30580 0x564bb5d2d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.689+0000 7f383144f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38281acff0 con 0x564bb5d2fbc0 2024-03-23T23:03:03.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.689+0000 7f383144f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38281aaaf0 con 0x564bb5d30580 2024-03-23T23:03:03.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.689+0000 7f383144f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38281a8b90 con 0x564bb5d2e9f0 2024-03-23T23:03:03.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.689+0000 7f3827400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d2e9f0 0x564bb5d36ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.689+0000 7f3827400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d2e9f0 0x564bb5d36ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:46114/0 (socket says 172.21.15.12:46114) 2024-03-23T23:03:03.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.689+0000 7f3827400640 1 -- 172.21.15.12:0/3303554683 learned_addr learned my addr 172.21.15.12:0/3303554683 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:03.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.689+0000 7f3827e00640 1 --2- 172.21.15.12:0/3303554683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb5d30580 0x564bb5d2d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.689+0000 7f3826a00640 1 --2- 172.21.15.12:0/3303554683 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564bb5d2fbc0 0x564bb5d401c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.689+0000 7f3827400640 1 -- 172.21.15.12:0/3303554683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb5d30580 msgr2=0x564bb5d2d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:03.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.689+0000 7f3827400640 1 --2- 172.21.15.12:0/3303554683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb5d30580 0x564bb5d2d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.689+0000 7f3827400640 1 -- 172.21.15.12:0/3303554683 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564bb5d2fbc0 msgr2=0x564bb5d401c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:03.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.689+0000 7f3827400640 1 --2- 172.21.15.12:0/3303554683 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564bb5d2fbc0 0x564bb5d401c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.689+0000 7f3827400640 1 -- 172.21.15.12:0/3303554683 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564bb5d46cf0 con 0x564bb5d2e9f0 2024-03-23T23:03:03.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.690+0000 7f3827e00640 1 --2- 172.21.15.12:0/3303554683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb5d30580 0x564bb5d2d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:03.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.690+0000 7f3826a00640 1 --2- 172.21.15.12:0/3303554683 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564bb5d2fbc0 0x564bb5d401c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:03:03.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.690+0000 7f3827400640 1 --2- 172.21.15.12:0/3303554683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d2e9f0 0x564bb5d36ad0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3818003ee0 tx=0x7f3818086040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16f82fa8a4219b11 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:03.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.690+0000 7f3813e00640 1 -- 172.21.15.12:0/3303554683 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38180b3030 con 0x564bb5d2e9f0 2024-03-23T23:03:03.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.690+0000 7f3813e00640 1 -- 172.21.15.12:0/3303554683 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f381809f990 con 0x564bb5d2e9f0 2024-03-23T23:03:03.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.690+0000 7f3813e00640 1 -- 172.21.15.12:0/3303554683 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38180cce60 con 0x564bb5d2e9f0 2024-03-23T23:03:03.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.691+0000 7f383144f740 1 -- 172.21.15.12:0/3303554683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d2e9f0 msgr2=0x564bb5d36ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.691+0000 7f383144f740 1 --2- 172.21.15.12:0/3303554683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d2e9f0 0x564bb5d36ad0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3818003ee0 tx=0x7f3818086040 comp rx=0 tx=0).stop 2024-03-23T23:03:03.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.691+0000 7f383144f740 1 -- 172.21.15.12:0/3303554683 shutdown_connections 2024-03-23T23:03:03.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.691+0000 7f383144f740 1 --2- 172.21.15.12:0/3303554683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb5d30580 0x564bb5d2d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.691+0000 7f383144f740 1 --2- 172.21.15.12:0/3303554683 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564bb5d2fbc0 0x564bb5d401c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.691+0000 7f383144f740 1 --2- 172.21.15.12:0/3303554683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d2e9f0 0x564bb5d36ad0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.691+0000 7f383144f740 1 -- 172.21.15.12:0/3303554683 >> 172.21.15.12:0/3303554683 conn(0x564bb5d29de0 msgr2=0x564bb5d2d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:03.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.691+0000 7f383144f740 1 -- 172.21.15.12:0/3303554683 shutdown_connections 2024-03-23T23:03:03.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.691+0000 7f383144f740 1 -- 172.21.15.12:0/3303554683 wait complete. 2024-03-23T23:03:03.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.691+0000 7f383144f740 1 -- start start 2024-03-23T23:03:03.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.691+0000 7f383144f740 1 Processor -- start 2024-03-23T23:03:03.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.692+0000 7f383144f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564bb5d2e9f0 0x564bb5d3ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.692+0000 7f383144f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d2fbc0 0x564bb5d3bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.692+0000 7f383144f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb5d30580 0x564bb5d3efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.692+0000 7f383144f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38281a9190 con 0x564bb5d2e9f0 2024-03-23T23:03:03.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.692+0000 7f383144f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38281aa490 con 0x564bb5d30580 2024-03-23T23:03:03.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.692+0000 7f383144f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38281a97d0 con 0x564bb5d2fbc0 2024-03-23T23:03:03.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.692+0000 7f383144f740 10 client.0 _ll_register_callbacks cb 0x7f3820015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:03.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.692+0000 7f3827e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb5d30580 0x564bb5d3efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.692+0000 7f3826a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d2fbc0 0x564bb5d3bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.692+0000 7f3827e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb5d30580 0x564bb5d3efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:36194/0 (socket says 172.21.15.12:36194) 2024-03-23T23:03:03.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.692+0000 7f3827e00640 1 -- 172.21.15.12:0/2417884518 learned_addr learned my addr 172.21.15.12:0/2417884518 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:03.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.692+0000 7f3827400640 1 --2- 172.21.15.12:0/2417884518 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564bb5d2e9f0 0x564bb5d3ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.693+0000 7f3827e00640 1 -- 172.21.15.12:0/2417884518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d2fbc0 msgr2=0x564bb5d3bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:03.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.693+0000 7f3827e00640 1 --2- 172.21.15.12:0/2417884518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d2fbc0 0x564bb5d3bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.693+0000 7f3827e00640 1 -- 172.21.15.12:0/2417884518 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564bb5d2e9f0 msgr2=0x564bb5d3ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:03.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.693+0000 7f3827e00640 1 --2- 172.21.15.12:0/2417884518 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564bb5d2e9f0 0x564bb5d3ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.693+0000 7f3827e00640 1 -- 172.21.15.12:0/2417884518 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564bb5d3f5d0 con 0x564bb5d30580 2024-03-23T23:03:03.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.693+0000 7f3827400640 1 --2- 172.21.15.12:0/2417884518 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564bb5d2e9f0 0x564bb5d3ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:03:03.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.693+0000 7f3827e00640 1 --2- 172.21.15.12:0/2417884518 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb5d30580 0x564bb5d3efe0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3820009760 tx=0x7f3820004700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6904273f7b749a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:03.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.693+0000 7f3811600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:03:03.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.693+0000 7f3811600640 1 -- 172.21.15.12:0/2417884518 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3820034070 con 0x564bb5d30580 2024-03-23T23:03:03.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.693+0000 7f3811600640 1 -- 172.21.15.12:0/2417884518 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f3820011950 con 0x564bb5d30580 2024-03-23T23:03:03.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.693+0000 7f383144f740 1 -- 172.21.15.12:0/2417884518 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f381809f990 con 0x564bb5d30580 2024-03-23T23:03:03.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.693+0000 7f3811600640 1 -- 172.21.15.12:0/2417884518 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3820009e60 con 0x564bb5d30580 2024-03-23T23:03:03.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.694+0000 7f383144f740 10 client.25607 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:03.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.694+0000 7f383144f740 1 -- 172.21.15.12:0/2417884518 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x564bb5d9fca0 con 0x564bb5d30580 2024-03-23T23:03:03.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.694+0000 7f3811600640 1 -- 172.21.15.12:0/2417884518 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f381809f990 con 0x564bb5d30580 2024-03-23T23:03:03.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.694+0000 7f3811600640 1 -- 172.21.15.12:0/2417884518 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f3820041030 con 0x564bb5d30580 2024-03-23T23:03:03.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.694+0000 7f383144f740 10 client.25607 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:03.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.694+0000 7f383144f740 10 client.25607 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:03.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.695+0000 7f383144f740 20 client.25607 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:03.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.695+0000 7f383144f740 1 --2- 172.21.15.12:0/2417884518 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x564bb5da0510 0x564bb5da2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:03.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.695+0000 7f383144f740 4 client.25607 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:03:03.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.695+0000 7f383144f740 1 -- 172.21.15.12:0/2417884518 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564bb5c9ab50 con 0x564bb5da0510 2024-03-23T23:03:03.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.695+0000 7f3827400640 1 --2- 172.21.15.12:0/2417884518 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x564bb5da0510 0x564bb5da2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:03.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.695+0000 7f3811600640 10 client.25607 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:03.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.695+0000 7f3827400640 1 --2- 172.21.15.12:0/2417884518 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x564bb5da0510 0x564bb5da2930 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bebe241e6f8e406 server_cookie=66cb6cd9cf82750e in_seq=0 out_seq=0 2024-03-23T23:03:03.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.696+0000 7f3811600640 1 -- 172.21.15.12:0/2417884518 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564bb5c9ab50 con 0x564bb5da0510 2024-03-23T23:03:03.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.696+0000 7f3811600640 10 client.25607 handle_command_reply: tid=0 2024-03-23T23:03:03.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.696+0000 7f383144f740 1 client.25607 shutdown 2024-03-23T23:03:03.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.696+0000 7f383144f740 1 -- 172.21.15.12:0/2417884518 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb5d30580 msgr2=0x564bb5d3efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.696+0000 7f383144f740 1 --2- 172.21.15.12:0/2417884518 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb5d30580 0x564bb5d3efe0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3820009760 tx=0x7f3820004700 comp rx=0 tx=0).stop 2024-03-23T23:03:03.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.696+0000 7f383144f740 1 -- 172.21.15.12:0/2417884518 shutdown_connections 2024-03-23T23:03:03.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.696+0000 7f383144f740 1 --2- 172.21.15.12:0/2417884518 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x564bb5da0510 0x564bb5da2930 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.696+0000 7f383144f740 1 --2- 172.21.15.12:0/2417884518 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb5d30580 0x564bb5d3efe0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.696+0000 7f383144f740 1 --2- 172.21.15.12:0/2417884518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb5d2fbc0 0x564bb5d3bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.696+0000 7f383144f740 1 --2- 172.21.15.12:0/2417884518 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564bb5d2e9f0 0x564bb5d3ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.696+0000 7f383144f740 1 -- 172.21.15.12:0/2417884518 >> 172.21.15.12:0/2417884518 conn(0x564bb5d29de0 msgr2=0x564bb5d2d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:03.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.696+0000 7f3811600640 0 client.25607 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:03.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.696+0000 7f383144f740 1 -- 172.21.15.12:0/2417884518 shutdown_connections 2024-03-23T23:03:03.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.696+0000 7f383144f740 1 -- 172.21.15.12:0/2417884518 wait complete. 2024-03-23T23:03:03.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.696+0000 7f383144f740 20 client.25607 trim_cache size 0 max 16384 2024-03-23T23:03:03.699 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:03:03.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.697+0000 7f382d200640 1 -- 172.21.15.12:0/129674147 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f380c0a27a0 msgr2=0x7f380c0a4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.697+0000 7f382d200640 1 --2- 172.21.15.12:0/129674147 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f380c0a27a0 0x7f380c0a4c60 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f381402c240 tx=0x7f381402df90 comp rx=0 tx=0).stop 2024-03-23T23:03:03.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.697+0000 7f382d200640 1 -- 172.21.15.12:0/129674147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38281015a0 msgr2=0x7f3828075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:03.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.697+0000 7f382d200640 1 --2- 172.21.15.12:0/129674147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38281015a0 0x7f3828075450 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f381800e9c0 tx=0x7f381800ee80 comp rx=0 tx=0).stop 2024-03-23T23:03:03.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.697+0000 7f382d200640 1 -- 172.21.15.12:0/129674147 shutdown_connections 2024-03-23T23:03:03.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.697+0000 7f382d200640 1 --2- 172.21.15.12:0/129674147 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f380c0a27a0 0x7f380c0a4c60 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.697+0000 7f382d200640 1 --2- 172.21.15.12:0/129674147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3828103150 0x7f3828073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.697+0000 7f382d200640 1 --2- 172.21.15.12:0/129674147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3828102790 0x7f3828073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.697+0000 7f382d200640 1 --2- 172.21.15.12:0/129674147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38281015a0 0x7f3828075450 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:03.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.697+0000 7f382d200640 1 -- 172.21.15.12:0/129674147 >> 172.21.15.12:0/129674147 conn(0x7f38280fcb10 msgr2=0x7f3828105a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:03.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.697+0000 7f382d200640 1 -- 172.21.15.12:0/129674147 shutdown_connections 2024-03-23T23:03:03.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:03.697+0000 7f382d200640 1 -- 172.21.15.12:0/129674147 wait complete. 2024-03-23T23:03:03.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:03 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3390033410' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:03:03.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:03 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3390033410' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:03:04.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.332+0000 7ff8b1a00640 1 -- 172.21.15.12:0/20423414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8ac101520 msgr2=0x7ff8ac101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:04.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.332+0000 7ff8b1a00640 1 --2- 172.21.15.12:0/20423414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8ac101520 0x7ff8ac101940 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff89c009900 tx=0x7ff89c02bc30 comp rx=0 tx=0).stop 2024-03-23T23:03:04.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.332+0000 7ff8b1a00640 1 -- 172.21.15.12:0/20423414 shutdown_connections 2024-03-23T23:03:04.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.332+0000 7ff8b1a00640 1 --2- 172.21.15.12:0/20423414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8ac1030d0 0x7ff8ac109960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.332+0000 7ff8b1a00640 1 --2- 172.21.15.12:0/20423414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8ac102710 0x7ff8ac102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.332+0000 7ff8b1a00640 1 --2- 172.21.15.12:0/20423414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8ac101520 0x7ff8ac101940 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.332+0000 7ff8b1a00640 1 -- 172.21.15.12:0/20423414 >> 172.21.15.12:0/20423414 conn(0x7ff8ac0fcab0 msgr2=0x7ff8ac0feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:04.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.332+0000 7ff8b1a00640 1 -- 172.21.15.12:0/20423414 shutdown_connections 2024-03-23T23:03:04.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.332+0000 7ff8b1a00640 1 -- 172.21.15.12:0/20423414 wait complete. 2024-03-23T23:03:04.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.333+0000 7ff8b1a00640 1 Processor -- start 2024-03-23T23:03:04.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.334+0000 7ff8b1a00640 1 -- start start 2024-03-23T23:03:04.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.334+0000 7ff8b1a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8ac101520 0x7ff8ac197880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.334+0000 7ff8b1a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8ac102710 0x7ff8ac197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.334+0000 7ff8b1a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8ac1030d0 0x7ff8ac19ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.334+0000 7ff8b1a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac10d200 con 0x7ff8ac102710 2024-03-23T23:03:04.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.334+0000 7ff8b1a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac10d080 con 0x7ff8ac101520 2024-03-23T23:03:04.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.334+0000 7ff8b1a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac10d380 con 0x7ff8ac1030d0 2024-03-23T23:03:04.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.334+0000 7ff8abe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8ac101520 0x7ff8ac197880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.334+0000 7ff8abe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8ac101520 0x7ff8ac197880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:58974/0 (socket says 172.21.15.12:58974) 2024-03-23T23:03:04.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.335+0000 7ff8abe00640 1 -- 172.21.15.12:0/1679677145 learned_addr learned my addr 172.21.15.12:0/1679677145 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:04.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.335+0000 7ff8ab400640 1 --2- 172.21.15.12:0/1679677145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8ac102710 0x7ff8ac197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.335+0000 7ff8b1000640 1 --2- 172.21.15.12:0/1679677145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8ac1030d0 0x7ff8ac19ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.335+0000 7ff8abe00640 1 -- 172.21.15.12:0/1679677145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8ac1030d0 msgr2=0x7ff8ac19ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:04.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.335+0000 7ff8abe00640 1 --2- 172.21.15.12:0/1679677145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8ac1030d0 0x7ff8ac19ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.335+0000 7ff8abe00640 1 -- 172.21.15.12:0/1679677145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8ac102710 msgr2=0x7ff8ac197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:04.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.335+0000 7ff8abe00640 1 --2- 172.21.15.12:0/1679677145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8ac102710 0x7ff8ac197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.335+0000 7ff8abe00640 1 -- 172.21.15.12:0/1679677145 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ac0738d0 con 0x7ff8ac101520 2024-03-23T23:03:04.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.335+0000 7ff8abe00640 1 --2- 172.21.15.12:0/1679677145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8ac101520 0x7ff8ac197880 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff89c000c00 tx=0x7ff89c02c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:04.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.336+0000 7ff8a8c00640 1 -- 172.21.15.12:0/1679677145 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89c043020 con 0x7ff8ac101520 2024-03-23T23:03:04.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.336+0000 7ff8a8c00640 1 -- 172.21.15.12:0/1679677145 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff89c0054e0 con 0x7ff8ac101520 2024-03-23T23:03:04.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.336+0000 7ff8a8c00640 1 -- 172.21.15.12:0/1679677145 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89c003ce0 con 0x7ff8ac101520 2024-03-23T23:03:04.337 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.336+0000 7ff8b1a00640 1 -- 172.21.15.12:0/1679677145 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8ac073b60 con 0x7ff8ac101520 2024-03-23T23:03:04.337 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.336+0000 7ff8b1a00640 1 -- 172.21.15.12:0/1679677145 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8ac074030 con 0x7ff8ac101520 2024-03-23T23:03:04.339 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.339+0000 7ff8b5e0b740 1 Processor -- start 2024-03-23T23:03:04.339 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.339+0000 7ff8b5e0b740 1 -- start start 2024-03-23T23:03:04.339 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.340+0000 7ff8b5e0b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565492d663f0 0x565492d66810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.339 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.340+0000 7ff8b5e0b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565492d66de0 0x565492d6b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.339 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.340+0000 7ff8b5e0b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565492d6bf50 0x565492d6e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.340+0000 7ff8b5e0b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x565492d293b0 con 0x565492d663f0 2024-03-23T23:03:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.340+0000 7ff8b5e0b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x565492ccb830 con 0x565492d66de0 2024-03-23T23:03:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.340+0000 7ff8b5e0b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x565492c745d0 con 0x565492d6bf50 2024-03-23T23:03:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.340+0000 7ff8b1000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565492d663f0 0x565492d66810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.340+0000 7ff8ab400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565492d6bf50 0x565492d6e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.340+0000 7ff8b1000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565492d663f0 0x565492d66810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38352/0 (socket says 172.21.15.12:38352) 2024-03-23T23:03:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.340+0000 7ff8b1000640 1 -- 172.21.15.12:0/3242256090 learned_addr learned my addr 172.21.15.12:0/3242256090 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.340+0000 7ff8b1000640 1 -- 172.21.15.12:0/3242256090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565492d6bf50 msgr2=0x565492d6e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.340+0000 7ff8b1000640 1 --2- 172.21.15.12:0/3242256090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565492d6bf50 0x565492d6e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.340+0000 7ff8b1000640 1 -- 172.21.15.12:0/3242256090 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565492d66de0 msgr2=0x565492d6b8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T23:03:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.340+0000 7ff8b1000640 1 --2- 172.21.15.12:0/3242256090 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565492d66de0 0x565492d6b8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.341+0000 7ff8b1000640 1 -- 172.21.15.12:0/3242256090 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565492c8c2c0 con 0x565492d663f0 2024-03-23T23:03:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.341+0000 7ff8b1000640 1 --2- 172.21.15.12:0/3242256090 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565492d663f0 0x565492d66810 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff8a40039f0 tx=0x7ff8a4003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2526f0fb2a92c2f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.341+0000 7ff8a8c00640 1 -- 172.21.15.12:0/1679677145 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff89c046020 con 0x7ff8ac101520 2024-03-23T23:03:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.341+0000 7ff8a1600640 1 -- 172.21.15.12:0/3242256090 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a401a470 con 0x565492d663f0 2024-03-23T23:03:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.342+0000 7ff8a1600640 1 -- 172.21.15.12:0/3242256090 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8a4015070 con 0x565492d663f0 2024-03-23T23:03:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.342+0000 7ff8a1600640 1 -- 172.21.15.12:0/3242256090 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a4010900 con 0x565492d663f0 2024-03-23T23:03:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.343+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/3242256090 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565492d663f0 msgr2=0x565492d66810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.343+0000 7ff8b5e0b740 1 --2- 172.21.15.12:0/3242256090 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565492d663f0 0x565492d66810 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff8a40039f0 tx=0x7ff8a4003eb0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.343+0000 7ff8a8c00640 1 --2- 172.21.15.12:0/1679677145 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff8900a28f0 0x7ff8900a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.343+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/3242256090 shutdown_connections 2024-03-23T23:03:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.343+0000 7ff8b5e0b740 1 --2- 172.21.15.12:0/3242256090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565492d6bf50 0x565492d6e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.343+0000 7ff8b5e0b740 1 --2- 172.21.15.12:0/3242256090 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565492d66de0 0x565492d6b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.343+0000 7ff8b5e0b740 1 --2- 172.21.15.12:0/3242256090 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565492d663f0 0x565492d66810 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.343+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/3242256090 >> 172.21.15.12:0/3242256090 conn(0x565492d61960 msgr2=0x565492d61d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.343+0000 7ff8a8c00640 1 -- 172.21.15.12:0/1679677145 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff89c0f0b20 con 0x7ff8ac101520 2024-03-23T23:03:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.343+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/3242256090 shutdown_connections 2024-03-23T23:03:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.344+0000 7ff8b1000640 1 --2- 172.21.15.12:0/1679677145 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff8900a28f0 0x7ff8900a4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.344+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/3242256090 wait complete. 2024-03-23T23:03:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.344+0000 7ff8b5e0b740 1 -- start start 2024-03-23T23:03:04.345 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.345+0000 7ff8b5e0b740 1 Processor -- start 2024-03-23T23:03:04.346 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.345+0000 7ff8b1000640 1 --2- 172.21.15.12:0/1679677145 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff8900a28f0 0x7ff8900a4db0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4005fd0 tx=0x7ff8a4009fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:04.346 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.346+0000 7ff8b5e0b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565492d663f0 0x565492d78e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.346 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.346+0000 7ff8b5e0b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565492d66de0 0x565492d793a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.346 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.346+0000 7ff8b5e0b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565492d6bf50 0x565492d803e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.346 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.346+0000 7ff8b5e0b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x565492c87a20 con 0x565492d663f0 2024-03-23T23:03:04.346 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.346+0000 7ff8b5e0b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x565492cc78d0 con 0x565492d6bf50 2024-03-23T23:03:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.346+0000 7ff8b5e0b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x565492d6ef00 con 0x565492d66de0 2024-03-23T23:03:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.346+0000 7ff8b5e0b740 10 client.0 _ll_register_callbacks cb 0x565492cc41c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.346+0000 7ff8ab400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565492d66de0 0x565492d793a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.346+0000 7ff8ab400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565492d66de0 0x565492d793a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:39848/0 (socket says 172.21.15.12:39848) 2024-03-23T23:03:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.346+0000 7ff8ab400640 1 -- 172.21.15.12:0/550706467 learned_addr learned my addr 172.21.15.12:0/550706467 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.346+0000 7ff8abe00640 1 --2- 172.21.15.12:0/550706467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565492d663f0 0x565492d78e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.346+0000 7ff8b1000640 1 --2- 172.21.15.12:0/550706467 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565492d6bf50 0x565492d803e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.346+0000 7ff8ab400640 1 -- 172.21.15.12:0/550706467 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565492d6bf50 msgr2=0x565492d803e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.346+0000 7ff8ab400640 1 --2- 172.21.15.12:0/550706467 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565492d6bf50 0x565492d803e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.346+0000 7ff8ab400640 1 -- 172.21.15.12:0/550706467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565492d663f0 msgr2=0x565492d78e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.346+0000 7ff8ab400640 1 --2- 172.21.15.12:0/550706467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565492d663f0 0x565492d78e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.347+0000 7ff8ab400640 1 -- 172.21.15.12:0/550706467 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5654929e0210 con 0x565492d66de0 2024-03-23T23:03:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.347+0000 7ff8abe00640 1 --2- 172.21.15.12:0/550706467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565492d663f0 0x565492d78e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:03:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.347+0000 7ff8ab400640 1 --2- 172.21.15.12:0/550706467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565492d66de0 0x565492d793a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff8980126e0 tx=0x7ff898012ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28f881fe025b3abf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.347+0000 7ff896000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:03:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.347+0000 7ff896000640 1 -- 172.21.15.12:0/550706467 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89800f8c0 con 0x565492d66de0 2024-03-23T23:03:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.347+0000 7ff896000640 1 -- 172.21.15.12:0/550706467 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff89800fa60 con 0x565492d66de0 2024-03-23T23:03:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.347+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/550706467 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff8a4015070 con 0x565492d66de0 2024-03-23T23:03:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.347+0000 7ff896000640 1 -- 172.21.15.12:0/550706467 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff89800fc60 con 0x565492d66de0 2024-03-23T23:03:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.347+0000 7ff8b5e0b740 10 client.25594 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.347+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/550706467 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x565492aac620 con 0x565492d66de0 2024-03-23T23:03:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.348+0000 7ff896000640 1 -- 172.21.15.12:0/550706467 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8a4015070 con 0x565492d66de0 2024-03-23T23:03:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.348+0000 7ff896000640 1 -- 172.21.15.12:0/550706467 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ff89801c4b0 con 0x565492d66de0 2024-03-23T23:03:04.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.348+0000 7ff8b5e0b740 10 client.25594 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:04.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.348+0000 7ff8b5e0b740 10 client.25594 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:04.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.348+0000 7ff8b5e0b740 20 client.25594 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:04.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.348+0000 7ff8b5e0b740 1 --2- 172.21.15.12:0/550706467 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x565492d84d90 0x565492d871f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.348+0000 7ff8b5e0b740 4 client.25594 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:03:04.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.348+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/550706467 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565492cd8d70 con 0x565492d84d90 2024-03-23T23:03:04.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.348+0000 7ff8abe00640 1 --2- 172.21.15.12:0/550706467 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x565492d84d90 0x565492d871f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.349+0000 7ff8abe00640 1 --2- 172.21.15.12:0/550706467 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x565492d84d90 0x565492d871f0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c4f5b2b8a9964d4 server_cookie=ee8329bf904ad6a5 in_seq=0 out_seq=0 2024-03-23T23:03:04.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.349+0000 7ff896000640 10 client.25594 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:04.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.350+0000 7ff896000640 1 -- 172.21.15.12:0/550706467 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x565492cd8d70 con 0x565492d84d90 2024-03-23T23:03:04.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.350+0000 7ff896000640 10 client.25594 handle_command_reply: tid=0 2024-03-23T23:03:04.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.350+0000 7ff8b5e0b740 1 client.25594 shutdown 2024-03-23T23:03:04.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.350+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/550706467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565492d66de0 msgr2=0x565492d793a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:04.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.350+0000 7ff8b5e0b740 1 --2- 172.21.15.12:0/550706467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565492d66de0 0x565492d793a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff8980126e0 tx=0x7ff898012ba0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.350+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/550706467 shutdown_connections 2024-03-23T23:03:04.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.350+0000 7ff8b5e0b740 1 --2- 172.21.15.12:0/550706467 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x565492d84d90 0x565492d871f0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.350+0000 7ff8b5e0b740 1 --2- 172.21.15.12:0/550706467 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565492d6bf50 0x565492d803e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.350+0000 7ff8b5e0b740 1 --2- 172.21.15.12:0/550706467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565492d66de0 0x565492d793a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.350+0000 7ff8b5e0b740 1 --2- 172.21.15.12:0/550706467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565492d663f0 0x565492d78e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.350+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/550706467 >> 172.21.15.12:0/550706467 conn(0x565492d61960 msgr2=0x565492d60b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:04.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.350+0000 7ff896000640 0 client.25594 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:04.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.351+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/550706467 shutdown_connections 2024-03-23T23:03:04.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.351+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/550706467 wait complete. 2024-03-23T23:03:04.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.351+0000 7ff8b5e0b740 20 client.25594 trim_cache size 0 max 16384 2024-03-23T23:03:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.380+0000 7ff8b5e0b740 1 Processor -- start 2024-03-23T23:03:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.380+0000 7ff8b5e0b740 1 -- start start 2024-03-23T23:03:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.380+0000 7ff8b5e0b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565492d663f0 0x565492d78e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.380+0000 7ff8b5e0b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565492d66de0 0x565492d793a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.380+0000 7ff8b5e0b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565492d6bf50 0x565492d803a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.380+0000 7ff8b5e0b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac1a8b60 con 0x565492d6bf50 2024-03-23T23:03:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.380+0000 7ff8b5e0b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac1a6660 con 0x565492d66de0 2024-03-23T23:03:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.380+0000 7ff8b5e0b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac1a4700 con 0x565492d663f0 2024-03-23T23:03:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.381+0000 7ff8ab400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565492d66de0 0x565492d793a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.381+0000 7ff8ab400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565492d66de0 0x565492d793a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59002/0 (socket says 172.21.15.12:59002) 2024-03-23T23:03:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.381+0000 7ff8abe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565492d663f0 0x565492d78e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.381+0000 7ff8ab400640 1 -- 172.21.15.12:0/3255238816 learned_addr learned my addr 172.21.15.12:0/3255238816 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.381+0000 7ff8b1000640 1 --2- 172.21.15.12:0/3255238816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565492d6bf50 0x565492d803a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.381+0000 7ff8ab400640 1 -- 172.21.15.12:0/3255238816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565492d663f0 msgr2=0x565492d78e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.381+0000 7ff8ab400640 1 --2- 172.21.15.12:0/3255238816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565492d663f0 0x565492d78e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.381+0000 7ff8ab400640 1 -- 172.21.15.12:0/3255238816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565492d6bf50 msgr2=0x565492d803a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.381+0000 7ff8ab400640 1 --2- 172.21.15.12:0/3255238816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565492d6bf50 0x565492d803a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.381+0000 7ff8ab400640 1 -- 172.21.15.12:0/3255238816 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565492d626d0 con 0x565492d66de0 2024-03-23T23:03:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.381+0000 7ff8abe00640 1 --2- 172.21.15.12:0/3255238816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565492d663f0 0x565492d78e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.382+0000 7ff8ab400640 1 --2- 172.21.15.12:0/3255238816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565492d66de0 0x565492d793a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff898000c00 tx=0x7ff89800fe80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f20e391d3f5a6574 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.382+0000 7ff8a0c00640 1 -- 172.21.15.12:0/3255238816 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89801cb10 con 0x565492d66de0 2024-03-23T23:03:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.382+0000 7ff8a0c00640 1 -- 172.21.15.12:0/3255238816 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff89802b020 con 0x565492d66de0 2024-03-23T23:03:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.382+0000 7ff8a0c00640 1 -- 172.21.15.12:0/3255238816 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89800f9a0 con 0x565492d66de0 2024-03-23T23:03:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.383+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/3255238816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565492d66de0 msgr2=0x565492d793a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.383+0000 7ff8b5e0b740 1 --2- 172.21.15.12:0/3255238816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565492d66de0 0x565492d793a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff898000c00 tx=0x7ff89800fe80 comp rx=0 tx=0).stop 2024-03-23T23:03:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.383+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/3255238816 shutdown_connections 2024-03-23T23:03:04.386 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.383+0000 7ff8b5e0b740 1 --2- 172.21.15.12:0/3255238816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565492d6bf50 0x565492d803a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.386 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.383+0000 7ff8b5e0b740 1 --2- 172.21.15.12:0/3255238816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565492d66de0 0x565492d793a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.386 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.383+0000 7ff8b5e0b740 1 --2- 172.21.15.12:0/3255238816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565492d663f0 0x565492d78e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.386 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.383+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/3255238816 >> 172.21.15.12:0/3255238816 conn(0x565492d61960 msgr2=0x565492d61d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:04.386 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.384+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/3255238816 shutdown_connections 2024-03-23T23:03:04.386 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.384+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/3255238816 wait complete. 2024-03-23T23:03:04.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.384+0000 7ff8b5e0b740 1 -- start start 2024-03-23T23:03:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.385+0000 7ff8b5e0b740 1 Processor -- start 2024-03-23T23:03:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.385+0000 7ff8b5e0b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565492d663f0 0x565492d7dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.385+0000 7ff8b5e0b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565492d66de0 0x565492d7e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.385+0000 7ff8b5e0b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565492d6bf50 0x565492d7f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.385+0000 7ff8b5e0b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac1a4d00 con 0x565492d663f0 2024-03-23T23:03:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.385+0000 7ff8b5e0b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac1a6000 con 0x565492d6bf50 2024-03-23T23:03:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.385+0000 7ff8b5e0b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac1a5340 con 0x565492d66de0 2024-03-23T23:03:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.385+0000 7ff8b5e0b740 10 client.0 _ll_register_callbacks cb 0x7ff898009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.385+0000 7ff8abe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565492d663f0 0x565492d7dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.386+0000 7ff8b1000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565492d6bf50 0x565492d7f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.386+0000 7ff8ab400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565492d66de0 0x565492d7e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.386+0000 7ff8abe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565492d663f0 0x565492d7dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38364/0 (socket says 172.21.15.12:38364) 2024-03-23T23:03:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.386+0000 7ff8b1000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565492d6bf50 0x565492d7f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59014/0 (socket says 172.21.15.12:59014) 2024-03-23T23:03:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.386+0000 7ff8abe00640 1 -- 172.21.15.12:0/254782871 learned_addr learned my addr 172.21.15.12:0/254782871 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.386+0000 7ff8ab400640 1 -- 172.21.15.12:0/254782871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565492d6bf50 msgr2=0x565492d7f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.386+0000 7ff8ab400640 1 --2- 172.21.15.12:0/254782871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565492d6bf50 0x565492d7f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.386+0000 7ff8ab400640 1 -- 172.21.15.12:0/254782871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565492d663f0 msgr2=0x565492d7dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.386+0000 7ff8ab400640 1 --2- 172.21.15.12:0/254782871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565492d663f0 0x565492d7dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.386+0000 7ff8ab400640 1 -- 172.21.15.12:0/254782871 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565492d7f810 con 0x565492d66de0 2024-03-23T23:03:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.386+0000 7ff8abe00640 1 --2- 172.21.15.12:0/254782871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565492d663f0 0x565492d7dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.386+0000 7ff8ab400640 1 --2- 172.21.15.12:0/254782871 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565492d66de0 0x565492d7e210 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff898005b90 tx=0x7ff898005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5796843a572d88da server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.386+0000 7ff896000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:03:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.387+0000 7ff896000640 1 -- 172.21.15.12:0/254782871 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff898034020 con 0x565492d66de0 2024-03-23T23:03:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.387+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/254782871 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff89802b020 con 0x565492d66de0 2024-03-23T23:03:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.387+0000 7ff896000640 1 -- 172.21.15.12:0/254782871 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff898018040 con 0x565492d66de0 2024-03-23T23:03:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.387+0000 7ff896000640 1 -- 172.21.15.12:0/254782871 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff89801c420 con 0x565492d66de0 2024-03-23T23:03:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.387+0000 7ff8b5e0b740 10 client.25600 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.387+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/254782871 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x565492de0900 con 0x565492d66de0 2024-03-23T23:03:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.388+0000 7ff896000640 1 -- 172.21.15.12:0/254782871 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff89802b020 con 0x565492d66de0 2024-03-23T23:03:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.388+0000 7ff896000640 1 -- 172.21.15.12:0/254782871 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ff898003900 con 0x565492d66de0 2024-03-23T23:03:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.388+0000 7ff8b5e0b740 10 client.25600 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.388+0000 7ff8b5e0b740 10 client.25600 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.388+0000 7ff8b5e0b740 20 client.25600 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.388+0000 7ff8b5e0b740 1 --2- 172.21.15.12:0/254782871 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x565492de1280 0x565492de36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.388+0000 7ff8b5e0b740 4 client.25600 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:03:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.388+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/254782871 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x565492cd30c0 con 0x565492de1280 2024-03-23T23:03:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.388+0000 7ff8abe00640 1 --2- 172.21.15.12:0/254782871 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x565492de1280 0x565492de36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.389+0000 7ff8abe00640 1 --2- 172.21.15.12:0/254782871 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x565492de1280 0x565492de36a0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80e1c609499c2958 server_cookie=bd4a64e6c9262f21 in_seq=0 out_seq=0 2024-03-23T23:03:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.389+0000 7ff896000640 10 client.25600 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:04.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.392+0000 7ff896000640 1 -- 172.21.15.12:0/254782871 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x565492cd30c0 con 0x565492de1280 2024-03-23T23:03:04.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.392+0000 7ff896000640 10 client.25600 handle_command_reply: tid=0 2024-03-23T23:03:04.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.392+0000 7ff8b5e0b740 1 client.25600 shutdown 2024-03-23T23:03:04.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.392+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/254782871 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565492d66de0 msgr2=0x565492d7e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:04.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.392+0000 7ff8b5e0b740 1 --2- 172.21.15.12:0/254782871 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565492d66de0 0x565492d7e210 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff898005b90 tx=0x7ff898005f90 comp rx=0 tx=0).stop 2024-03-23T23:03:04.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.392+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/254782871 shutdown_connections 2024-03-23T23:03:04.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.392+0000 7ff8b5e0b740 1 --2- 172.21.15.12:0/254782871 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x565492de1280 0x565492de36a0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.392+0000 7ff8b5e0b740 1 --2- 172.21.15.12:0/254782871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565492d6bf50 0x565492d7f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.392+0000 7ff8b5e0b740 1 --2- 172.21.15.12:0/254782871 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565492d66de0 0x565492d7e210 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.392+0000 7ff8b5e0b740 1 --2- 172.21.15.12:0/254782871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565492d663f0 0x565492d7dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.392+0000 7ff896000640 0 client.25600 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:04.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.392+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/254782871 >> 172.21.15.12:0/254782871 conn(0x565492d61960 msgr2=0x565492d78e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:04.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.393+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/254782871 shutdown_connections 2024-03-23T23:03:04.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.393+0000 7ff8b5e0b740 1 -- 172.21.15.12:0/254782871 wait complete. 2024-03-23T23:03:04.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.393+0000 7ff8b5e0b740 20 client.25600 trim_cache size 0 max 16384 2024-03-23T23:03:04.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:03:04.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:03:04.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:03:04.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:03:04.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.393+0000 7ff8b1a00640 1 -- 172.21.15.12:0/1679677145 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff8900a28f0 msgr2=0x7ff8900a4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:04.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.393+0000 7ff8b1a00640 1 --2- 172.21.15.12:0/1679677145 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff8900a28f0 0x7ff8900a4db0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4005fd0 tx=0x7ff8a4009fb0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.393+0000 7ff8b1a00640 1 -- 172.21.15.12:0/1679677145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8ac101520 msgr2=0x7ff8ac197880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:04.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.393+0000 7ff8b1a00640 1 --2- 172.21.15.12:0/1679677145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8ac101520 0x7ff8ac197880 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff89c000c00 tx=0x7ff89c02c6c0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.394+0000 7ff8b1a00640 1 -- 172.21.15.12:0/1679677145 shutdown_connections 2024-03-23T23:03:04.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.394+0000 7ff8b1a00640 1 --2- 172.21.15.12:0/1679677145 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff8900a28f0 0x7ff8900a4db0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.394+0000 7ff8b1a00640 1 --2- 172.21.15.12:0/1679677145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8ac1030d0 0x7ff8ac19ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.394+0000 7ff8b1a00640 1 --2- 172.21.15.12:0/1679677145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8ac102710 0x7ff8ac197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.394+0000 7ff8b1a00640 1 --2- 172.21.15.12:0/1679677145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8ac101520 0x7ff8ac197880 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.394+0000 7ff8b1a00640 1 -- 172.21.15.12:0/1679677145 >> 172.21.15.12:0/1679677145 conn(0x7ff8ac0fcab0 msgr2=0x7ff8ac105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:04.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.394+0000 7ff8b1a00640 1 -- 172.21.15.12:0/1679677145 shutdown_connections 2024-03-23T23:03:04.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:04.394+0000 7ff8b1a00640 1 -- 172.21.15.12:0/1679677145 wait complete. 2024-03-23T23:03:04.714 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-23T23:03:04.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:03:04.714 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-23T23:03:04.783 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:04 smithi012 ceph-mon[31695]: pgmap v428: 97 pgs: 97 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:03:04.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:04 smithi173 ceph-mon[36116]: pgmap v428: 97 pgs: 97 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:03:04.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.905+0000 7fbaa3200640 1 -- 172.21.15.12:0/993166534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba9c0772a0 msgr2=0x7fba9c077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:04.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.905+0000 7fbaa3200640 1 --2- 172.21.15.12:0/993166534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba9c0772a0 0x7fba9c077700 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fba88009900 tx=0x7fba8802bc30 comp rx=0 tx=0).stop 2024-03-23T23:03:04.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.906+0000 7fbaa3200640 1 -- 172.21.15.12:0/993166534 shutdown_connections 2024-03-23T23:03:04.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.906+0000 7fbaa3200640 1 --2- 172.21.15.12:0/993166534 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba9c077c40 0x7fba9c0784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.906+0000 7fbaa3200640 1 --2- 172.21.15.12:0/993166534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba9c0772a0 0x7fba9c077700 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.906+0000 7fbaa3200640 1 --2- 172.21.15.12:0/993166534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba9c076060 0x7fba9c076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.906+0000 7fbaa3200640 1 -- 172.21.15.12:0/993166534 >> 172.21.15.12:0/993166534 conn(0x7fba9c0fee50 msgr2=0x7fba9c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:04.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.906+0000 7fbaa3200640 1 -- 172.21.15.12:0/993166534 shutdown_connections 2024-03-23T23:03:04.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.906+0000 7fbaa3200640 1 -- 172.21.15.12:0/993166534 wait complete. 2024-03-23T23:03:04.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:04 smithi203 ceph-mon[36693]: pgmap v428: 97 pgs: 97 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:03:04.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.907+0000 7fbaa3200640 1 Processor -- start 2024-03-23T23:03:04.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.907+0000 7fbaa3200640 1 -- start start 2024-03-23T23:03:04.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.907+0000 7fbaa3200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba9c076060 0x7fba9c19dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.907+0000 7fbaa3200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba9c0772a0 0x7fba9c19e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.907+0000 7fbaa3200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba9c077c40 0x7fba9c1a55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.907+0000 7fbaa3200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c113970 con 0x7fba9c0772a0 2024-03-23T23:03:04.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.907+0000 7fbaa3200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c1137f0 con 0x7fba9c077c40 2024-03-23T23:03:04.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.907+0000 7fbaa3200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c113af0 con 0x7fba9c076060 2024-03-23T23:03:04.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.908+0000 7fba9be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba9c076060 0x7fba9c19dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.908+0000 7fba9be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba9c076060 0x7fba9c19dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:39904/0 (socket says 172.21.15.12:39904) 2024-03-23T23:03:04.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.908+0000 7fba9be00640 1 -- 172.21.15.12:0/3397994720 learned_addr learned my addr 172.21.15.12:0/3397994720 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:04.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.908+0000 7fba9b400640 1 --2- 172.21.15.12:0/3397994720 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba9c0772a0 0x7fba9c19e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.908+0000 7fbaa1200640 1 --2- 172.21.15.12:0/3397994720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba9c077c40 0x7fba9c1a55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.908+0000 7fba9be00640 1 -- 172.21.15.12:0/3397994720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba9c077c40 msgr2=0x7fba9c1a55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:04.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.908+0000 7fba9be00640 1 --2- 172.21.15.12:0/3397994720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba9c077c40 0x7fba9c1a55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.908+0000 7fba9be00640 1 -- 172.21.15.12:0/3397994720 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba9c0772a0 msgr2=0x7fba9c19e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:04.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.908+0000 7fba9be00640 1 --2- 172.21.15.12:0/3397994720 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba9c0772a0 0x7fba9c19e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.909+0000 7fba9be00640 1 -- 172.21.15.12:0/3397994720 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba9c1a5c10 con 0x7fba9c076060 2024-03-23T23:03:04.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.909+0000 7fbaa1200640 1 --2- 172.21.15.12:0/3397994720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba9c077c40 0x7fba9c1a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:03:04.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.909+0000 7fba9b400640 1 --2- 172.21.15.12:0/3397994720 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba9c0772a0 0x7fba9c19e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:04.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.909+0000 7fba9be00640 1 --2- 172.21.15.12:0/3397994720 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba9c076060 0x7fba9c19dff0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fba8c00d950 tx=0x7fba8c00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:04.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.909+0000 7fba98c00640 1 -- 172.21.15.12:0/3397994720 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c017070 con 0x7fba9c076060 2024-03-23T23:03:04.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.909+0000 7fba98c00640 1 -- 172.21.15.12:0/3397994720 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba8c009c30 con 0x7fba9c076060 2024-03-23T23:03:04.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.909+0000 7fbaa3200640 1 -- 172.21.15.12:0/3397994720 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba9c1a5f00 con 0x7fba9c076060 2024-03-23T23:03:04.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.909+0000 7fba98c00640 1 -- 172.21.15.12:0/3397994720 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c00f6b0 con 0x7fba9c076060 2024-03-23T23:03:04.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.909+0000 7fbaa3200640 1 -- 172.21.15.12:0/3397994720 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba9c1a6370 con 0x7fba9c076060 2024-03-23T23:03:04.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.914+0000 7fbaa5b22740 1 Processor -- start 2024-03-23T23:03:04.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.914+0000 7fba98c00640 1 -- 172.21.15.12:0/3397994720 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fba8c0109d0 con 0x7fba9c076060 2024-03-23T23:03:04.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.914+0000 7fbaa5b22740 1 -- start start 2024-03-23T23:03:04.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.914+0000 7fbaa5b22740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595cf14e0a0 0x5595cf14e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.914+0000 7fbaa5b22740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595cf14ea90 0x5595cf153590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.914+0000 7fbaa5b22740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595cf153c00 0x5595cf156090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.914+0000 7fbaa5b22740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595cf0b3eb0 con 0x5595cf14ea90 2024-03-23T23:03:04.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.914+0000 7fbaa5b22740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595cf05ca60 con 0x5595cf153c00 2024-03-23T23:03:04.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.914+0000 7fbaa5b22740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595cf0aff50 con 0x5595cf14e0a0 2024-03-23T23:03:04.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.915+0000 7fba9b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595cf153c00 0x5595cf156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.915+0000 7fba9b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595cf153c00 0x5595cf156090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59034/0 (socket says 172.21.15.12:59034) 2024-03-23T23:03:04.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.915+0000 7fba9b400640 1 -- 172.21.15.12:0/1766505091 learned_addr learned my addr 172.21.15.12:0/1766505091 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:04.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.915+0000 7fba9be00640 1 --2- 172.21.15.12:0/1766505091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595cf14ea90 0x5595cf153590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.915+0000 7fbaa1200640 1 --2- 172.21.15.12:0/1766505091 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595cf14e0a0 0x5595cf14e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.915+0000 7fba9b400640 1 -- 172.21.15.12:0/1766505091 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595cf14e0a0 msgr2=0x5595cf14e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:04.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.915+0000 7fba9b400640 1 --2- 172.21.15.12:0/1766505091 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595cf14e0a0 0x5595cf14e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.915+0000 7fba9b400640 1 -- 172.21.15.12:0/1766505091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595cf14ea90 msgr2=0x5595cf153590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:04.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.915+0000 7fba9b400640 1 --2- 172.21.15.12:0/1766505091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595cf14ea90 0x5595cf153590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.915+0000 7fba9b400640 1 -- 172.21.15.12:0/1766505091 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595cee907e0 con 0x5595cf153c00 2024-03-23T23:03:04.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.915+0000 7fbaa1200640 1 --2- 172.21.15.12:0/1766505091 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595cf14e0a0 0x5595cf14e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:03:04.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.916+0000 7fba9b400640 1 --2- 172.21.15.12:0/1766505091 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595cf153c00 0x5595cf156090 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fba88005ab0 tx=0x7fba880095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67f7db36e2d435ca server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:04.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.916+0000 7fba91600640 1 -- 172.21.15.12:0/1766505091 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba88045070 con 0x5595cf153c00 2024-03-23T23:03:04.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.916+0000 7fba91600640 1 -- 172.21.15.12:0/1766505091 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba8802cd30 con 0x5595cf153c00 2024-03-23T23:03:04.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.916+0000 7fba91600640 1 -- 172.21.15.12:0/1766505091 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba88005130 con 0x5595cf153c00 2024-03-23T23:03:04.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.916+0000 7fba98c00640 1 --2- 172.21.15.12:0/3397994720 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fba800a2880 0x7fba800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.916+0000 7fba98c00640 1 -- 172.21.15.12:0/3397994720 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fba8c0ce990 con 0x7fba9c076060 2024-03-23T23:03:04.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.917+0000 7fbaa1200640 1 --2- 172.21.15.12:0/3397994720 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fba800a2880 0x7fba800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.917+0000 7fbaa5b22740 1 -- 172.21.15.12:0/1766505091 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595cf153c00 msgr2=0x5595cf156090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:04.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.917+0000 7fbaa5b22740 1 --2- 172.21.15.12:0/1766505091 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595cf153c00 0x5595cf156090 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fba88005ab0 tx=0x7fba880095e0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.917+0000 7fbaa5b22740 1 -- 172.21.15.12:0/1766505091 shutdown_connections 2024-03-23T23:03:04.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.917+0000 7fbaa5b22740 1 --2- 172.21.15.12:0/1766505091 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595cf153c00 0x5595cf156090 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.917+0000 7fbaa5b22740 1 --2- 172.21.15.12:0/1766505091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595cf14ea90 0x5595cf153590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.917+0000 7fbaa5b22740 1 --2- 172.21.15.12:0/1766505091 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595cf14e0a0 0x5595cf14e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.917+0000 7fbaa5b22740 1 -- 172.21.15.12:0/1766505091 >> 172.21.15.12:0/1766505091 conn(0x5595cf149610 msgr2=0x5595cf149a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:04.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.918+0000 7fbaa1200640 1 --2- 172.21.15.12:0/3397994720 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fba800a2880 0x7fba800a4d40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x5595cf14f2f0 tx=0x7fba94009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:04.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.918+0000 7fbaa5b22740 1 -- 172.21.15.12:0/1766505091 shutdown_connections 2024-03-23T23:03:04.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.918+0000 7fbaa5b22740 1 -- 172.21.15.12:0/1766505091 wait complete. 2024-03-23T23:03:04.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.918+0000 7fbaa5b22740 1 -- start start 2024-03-23T23:03:04.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.918+0000 7fbaa5b22740 1 Processor -- start 2024-03-23T23:03:04.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.919+0000 7fbaa5b22740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595cf14e0a0 0x5595cf1581f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.919+0000 7fbaa5b22740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595cf14ea90 0x5595cf158730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.919+0000 7fbaa5b22740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595cf153c00 0x5595cf15f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.919+0000 7fbaa5b22740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595cf156b50 con 0x5595cf14ea90 2024-03-23T23:03:04.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.919+0000 7fbaa5b22740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595cf06feb0 con 0x5595cf153c00 2024-03-23T23:03:04.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.919+0000 7fbaa5b22740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595cf156cd0 con 0x5595cf14e0a0 2024-03-23T23:03:04.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.919+0000 7fbaa5b22740 10 client.0 _ll_register_callbacks cb 0x5595cf064150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:04.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.919+0000 7fba9be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595cf14e0a0 0x5595cf1581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.919+0000 7fba9be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595cf14e0a0 0x5595cf1581f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:39928/0 (socket says 172.21.15.12:39928) 2024-03-23T23:03:04.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.919+0000 7fba9be00640 1 -- 172.21.15.12:0/2432160414 learned_addr learned my addr 172.21.15.12:0/2432160414 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:04.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.919+0000 7fba9b400640 1 --2- 172.21.15.12:0/2432160414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595cf14ea90 0x5595cf158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.919+0000 7fbaa1200640 1 --2- 172.21.15.12:0/2432160414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595cf153c00 0x5595cf15f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.920+0000 7fba9be00640 1 -- 172.21.15.12:0/2432160414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595cf153c00 msgr2=0x5595cf15f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:04.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.920+0000 7fba9be00640 1 --2- 172.21.15.12:0/2432160414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595cf153c00 0x5595cf15f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.920+0000 7fba9be00640 1 -- 172.21.15.12:0/2432160414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595cf14ea90 msgr2=0x5595cf158730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:04.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.920+0000 7fba9be00640 1 --2- 172.21.15.12:0/2432160414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595cf14ea90 0x5595cf158730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.920+0000 7fba9be00640 1 -- 172.21.15.12:0/2432160414 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5595ceef0f70 con 0x5595cf14e0a0 2024-03-23T23:03:04.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.920+0000 7fba9be00640 1 --2- 172.21.15.12:0/2432160414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595cf14e0a0 0x5595cf1581f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fba8c009610 tx=0x7fba8c009de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88c28da816bc4263 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:04.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.920+0000 7fba86000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:03:04.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.920+0000 7fba86000640 1 -- 172.21.15.12:0/2432160414 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c020070 con 0x5595cf14e0a0 2024-03-23T23:03:04.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.920+0000 7fba86000640 1 -- 172.21.15.12:0/2432160414 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fba8c0979b0 con 0x5595cf14e0a0 2024-03-23T23:03:04.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.921+0000 7fbaa5b22740 1 -- 172.21.15.12:0/2432160414 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fba8802cd30 con 0x5595cf14e0a0 2024-03-23T23:03:04.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.921+0000 7fba86000640 1 -- 172.21.15.12:0/2432160414 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba8c020070 con 0x5595cf14e0a0 2024-03-23T23:03:04.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.921+0000 7fbaa5b22740 10 client.25609 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:04.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.921+0000 7fbaa5b22740 1 -- 172.21.15.12:0/2432160414 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5595ced88170 con 0x5595cf14e0a0 2024-03-23T23:03:04.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.921+0000 7fba86000640 1 -- 172.21.15.12:0/2432160414 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba8802cd30 con 0x5595cf14e0a0 2024-03-23T23:03:04.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.921+0000 7fba86000640 1 -- 172.21.15.12:0/2432160414 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fba8c0bb0b0 con 0x5595cf14e0a0 2024-03-23T23:03:04.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.921+0000 7fbaa5b22740 10 client.25609 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:04.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.921+0000 7fbaa5b22740 10 client.25609 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:04.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.921+0000 7fbaa5b22740 20 client.25609 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:04.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.921+0000 7fbaa5b22740 1 --2- 172.21.15.12:0/2432160414 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5595cf175000 0x5595cf1774a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.921+0000 7fbaa5b22740 4 client.25609 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:03:04.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.921+0000 7fbaa5b22740 1 -- 172.21.15.12:0/2432160414 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5595cf0c13f0 con 0x5595cf175000 2024-03-23T23:03:04.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.922+0000 7fba9b400640 1 --2- 172.21.15.12:0/2432160414 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5595cf175000 0x5595cf1774a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.922+0000 7fba86000640 10 client.25609 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:04.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.922+0000 7fba9b400640 1 --2- 172.21.15.12:0/2432160414 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5595cf175000 0x5595cf1774a0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce6f052cfbc0ea3f server_cookie=b0a62b6a9842ceac in_seq=0 out_seq=0 2024-03-23T23:03:04.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.923+0000 7fba86000640 1 -- 172.21.15.12:0/2432160414 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5595cf0c13f0 con 0x5595cf175000 2024-03-23T23:03:04.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.923+0000 7fba86000640 10 client.25609 handle_command_reply: tid=0 2024-03-23T23:03:04.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.923+0000 7fbaa5b22740 1 client.25609 shutdown 2024-03-23T23:03:04.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.923+0000 7fbaa5b22740 1 -- 172.21.15.12:0/2432160414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595cf14e0a0 msgr2=0x5595cf1581f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:04.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.923+0000 7fbaa5b22740 1 --2- 172.21.15.12:0/2432160414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595cf14e0a0 0x5595cf1581f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fba8c009610 tx=0x7fba8c009de0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.923+0000 7fbaa5b22740 1 -- 172.21.15.12:0/2432160414 shutdown_connections 2024-03-23T23:03:04.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.923+0000 7fbaa5b22740 1 --2- 172.21.15.12:0/2432160414 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5595cf175000 0x5595cf1774a0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.923+0000 7fbaa5b22740 1 --2- 172.21.15.12:0/2432160414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595cf153c00 0x5595cf15f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.923+0000 7fbaa5b22740 1 --2- 172.21.15.12:0/2432160414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595cf14ea90 0x5595cf158730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.923+0000 7fbaa5b22740 1 --2- 172.21.15.12:0/2432160414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595cf14e0a0 0x5595cf1581f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.923+0000 7fba86000640 0 client.25609 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:04.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.923+0000 7fbaa5b22740 1 -- 172.21.15.12:0/2432160414 >> 172.21.15.12:0/2432160414 conn(0x5595cf149610 msgr2=0x5595cf148830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:04.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.924+0000 7fbaa5b22740 1 -- 172.21.15.12:0/2432160414 shutdown_connections 2024-03-23T23:03:04.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.924+0000 7fbaa5b22740 1 -- 172.21.15.12:0/2432160414 wait complete. 2024-03-23T23:03:04.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.924+0000 7fbaa5b22740 20 client.25609 trim_cache size 0 max 16384 2024-03-23T23:03:04.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.941+0000 7fbaa5b22740 1 Processor -- start 2024-03-23T23:03:04.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.941+0000 7fbaa5b22740 1 -- start start 2024-03-23T23:03:04.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.942+0000 7fbaa5b22740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595cf14e0a0 0x5595cf1486b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.942+0000 7fbaa5b22740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595cf14ea90 0x5595cf148bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.942+0000 7fbaa5b22740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595cf153c00 0x5595cf15f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.942+0000 7fbaa5b22740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c1a8d00 con 0x5595cf14e0a0 2024-03-23T23:03:04.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.942+0000 7fbaa5b22740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c1a6800 con 0x5595cf14ea90 2024-03-23T23:03:04.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.942+0000 7fbaa5b22740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c071bc0 con 0x5595cf153c00 2024-03-23T23:03:04.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.942+0000 7fba9be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595cf14e0a0 0x5595cf1486b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.942+0000 7fba9b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595cf14ea90 0x5595cf148bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.942+0000 7fba9b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595cf14ea90 0x5595cf148bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59052/0 (socket says 172.21.15.12:59052) 2024-03-23T23:03:04.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.942+0000 7fba9b400640 1 -- 172.21.15.12:0/2105923653 learned_addr learned my addr 172.21.15.12:0/2105923653 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:04.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.942+0000 7fbaa1200640 1 --2- 172.21.15.12:0/2105923653 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595cf153c00 0x5595cf15f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.942+0000 7fba9b400640 1 -- 172.21.15.12:0/2105923653 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595cf153c00 msgr2=0x5595cf15f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:04.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.942+0000 7fba9b400640 1 --2- 172.21.15.12:0/2105923653 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595cf153c00 0x5595cf15f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.943+0000 7fba9b400640 1 -- 172.21.15.12:0/2105923653 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595cf14e0a0 msgr2=0x5595cf1486b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:04.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.943+0000 7fba9b400640 1 --2- 172.21.15.12:0/2105923653 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595cf14e0a0 0x5595cf1486b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.943+0000 7fba9b400640 1 -- 172.21.15.12:0/2105923653 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595cf144c20 con 0x5595cf14ea90 2024-03-23T23:03:04.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.943+0000 7fba9b400640 1 --2- 172.21.15.12:0/2105923653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595cf14ea90 0x5595cf148bf0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fba88009900 tx=0x7fba8802bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9587fa3a4e5ca956 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:04.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.943+0000 7fba90c00640 1 -- 172.21.15.12:0/2105923653 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba88052070 con 0x5595cf14ea90 2024-03-23T23:03:04.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.943+0000 7fba90c00640 1 -- 172.21.15.12:0/2105923653 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba8802cd30 con 0x5595cf14ea90 2024-03-23T23:03:04.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.943+0000 7fba90c00640 1 -- 172.21.15.12:0/2105923653 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba88045040 con 0x5595cf14ea90 2024-03-23T23:03:04.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.944+0000 7fbaa5b22740 1 -- 172.21.15.12:0/2105923653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595cf14ea90 msgr2=0x5595cf148bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:04.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.944+0000 7fbaa5b22740 1 --2- 172.21.15.12:0/2105923653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595cf14ea90 0x5595cf148bf0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fba88009900 tx=0x7fba8802bc30 comp rx=0 tx=0).stop 2024-03-23T23:03:04.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.944+0000 7fbaa5b22740 1 -- 172.21.15.12:0/2105923653 shutdown_connections 2024-03-23T23:03:04.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.944+0000 7fbaa5b22740 1 --2- 172.21.15.12:0/2105923653 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595cf153c00 0x5595cf15f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.944+0000 7fbaa5b22740 1 --2- 172.21.15.12:0/2105923653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595cf14ea90 0x5595cf148bf0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.944+0000 7fbaa5b22740 1 --2- 172.21.15.12:0/2105923653 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595cf14e0a0 0x5595cf1486b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.944+0000 7fbaa5b22740 1 -- 172.21.15.12:0/2105923653 >> 172.21.15.12:0/2105923653 conn(0x5595cf149610 msgr2=0x5595cf147fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:04.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.944+0000 7fbaa5b22740 1 -- 172.21.15.12:0/2105923653 shutdown_connections 2024-03-23T23:03:04.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.944+0000 7fbaa5b22740 1 -- 172.21.15.12:0/2105923653 wait complete. 2024-03-23T23:03:04.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.945+0000 7fbaa5b22740 1 -- start start 2024-03-23T23:03:04.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.945+0000 7fbaa5b22740 1 Processor -- start 2024-03-23T23:03:04.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.945+0000 7fbaa5b22740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595cf14e0a0 0x5595cf15f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.945+0000 7fbaa5b22740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595cf14ea90 0x5595cf15f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.946+0000 7fbaa5b22740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595cf153c00 0x5595cf15d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.946+0000 7fbaa5b22740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c0721c0 con 0x5595cf14e0a0 2024-03-23T23:03:04.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.946+0000 7fbaa5b22740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c0734c0 con 0x5595cf153c00 2024-03-23T23:03:04.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.946+0000 7fbaa5b22740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c072800 con 0x5595cf14ea90 2024-03-23T23:03:04.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.946+0000 7fbaa5b22740 10 client.0 _ll_register_callbacks cb 0x7fba8c09ae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:04.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.946+0000 7fba9be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595cf14e0a0 0x5595cf15f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.946+0000 7fba9b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595cf14ea90 0x5595cf15f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.946+0000 7fba9be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595cf14e0a0 0x5595cf15f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38436/0 (socket says 172.21.15.12:38436) 2024-03-23T23:03:04.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.946+0000 7fba9be00640 1 -- 172.21.15.12:0/164047109 learned_addr learned my addr 172.21.15.12:0/164047109 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:04.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.946+0000 7fbaa1200640 1 --2- 172.21.15.12:0/164047109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595cf153c00 0x5595cf15d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.946+0000 7fba9b400640 1 -- 172.21.15.12:0/164047109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595cf153c00 msgr2=0x5595cf15d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:04.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.946+0000 7fba9b400640 1 --2- 172.21.15.12:0/164047109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595cf153c00 0x5595cf15d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.946+0000 7fba9b400640 1 -- 172.21.15.12:0/164047109 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595cf14e0a0 msgr2=0x5595cf15f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:04.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.946+0000 7fba9b400640 1 --2- 172.21.15.12:0/164047109 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595cf14e0a0 0x5595cf15f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.946+0000 7fba9b400640 1 -- 172.21.15.12:0/164047109 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5595cf14a080 con 0x5595cf14ea90 2024-03-23T23:03:04.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.946+0000 7fba9be00640 1 --2- 172.21.15.12:0/164047109 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595cf14e0a0 0x5595cf15f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:04.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.947+0000 7fba9b400640 1 --2- 172.21.15.12:0/164047109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595cf14ea90 0x5595cf15f580 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x5595cf1531a0 tx=0x7fba88004890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=874b42871ecea658 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:04.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.947+0000 7fba86000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:03:04.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.947+0000 7fba86000640 1 -- 172.21.15.12:0/164047109 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba88052070 con 0x5595cf14ea90 2024-03-23T23:03:04.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.947+0000 7fbaa5b22740 1 -- 172.21.15.12:0/164047109 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fba8802cd30 con 0x5595cf14ea90 2024-03-23T23:03:04.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.947+0000 7fba86000640 1 -- 172.21.15.12:0/164047109 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fba88004240 con 0x5595cf14ea90 2024-03-23T23:03:04.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.948+0000 7fba86000640 1 -- 172.21.15.12:0/164047109 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba8805e050 con 0x5595cf14ea90 2024-03-23T23:03:04.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.948+0000 7fbaa5b22740 10 client.25615 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:04.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.948+0000 7fbaa5b22740 1 -- 172.21.15.12:0/164047109 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5595cf1d1210 con 0x5595cf14ea90 2024-03-23T23:03:04.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.948+0000 7fba86000640 1 -- 172.21.15.12:0/164047109 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba8802cd30 con 0x5595cf14ea90 2024-03-23T23:03:04.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.948+0000 7fba86000640 1 -- 172.21.15.12:0/164047109 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fba88003a30 con 0x5595cf14ea90 2024-03-23T23:03:04.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.948+0000 7fbaa5b22740 10 client.25615 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:04.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.948+0000 7fbaa5b22740 10 client.25615 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:04.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.948+0000 7fbaa5b22740 20 client.25615 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:04.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.948+0000 7fbaa5b22740 1 --2- 172.21.15.12:0/164047109 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5595cf1d1660 0x5595cf1d3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:04.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.948+0000 7fbaa5b22740 4 client.25615 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-23T23:03:04.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.948+0000 7fbaa5b22740 1 -- 172.21.15.12:0/164047109 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5595cf0bb740 con 0x5595cf1d1660 2024-03-23T23:03:04.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.948+0000 7fba9be00640 1 --2- 172.21.15.12:0/164047109 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5595cf1d1660 0x5595cf1d3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:04.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.949+0000 7fba9be00640 1 --2- 172.21.15.12:0/164047109 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5595cf1d1660 0x5595cf1d3a50 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec5190828c8a51c3 server_cookie=45b16f8b715feee9 in_seq=0 out_seq=0 2024-03-23T23:03:04.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.949+0000 7fba86000640 10 client.25615 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:04.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.950+0000 7fba86000640 1 -- 172.21.15.12:0/164047109 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5595cf0bb740 con 0x5595cf1d1660 2024-03-23T23:03:04.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.950+0000 7fba86000640 10 client.25615 handle_command_reply: tid=0 2024-03-23T23:03:04.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.950+0000 7fbaa5b22740 1 client.25615 shutdown 2024-03-23T23:03:04.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.950+0000 7fbaa5b22740 1 -- 172.21.15.12:0/164047109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595cf14ea90 msgr2=0x5595cf15f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:04.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.950+0000 7fbaa5b22740 1 --2- 172.21.15.12:0/164047109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595cf14ea90 0x5595cf15f580 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x5595cf1531a0 tx=0x7fba88004890 comp rx=0 tx=0).stop 2024-03-23T23:03:04.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.950+0000 7fbaa5b22740 1 -- 172.21.15.12:0/164047109 shutdown_connections 2024-03-23T23:03:04.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.950+0000 7fbaa5b22740 1 --2- 172.21.15.12:0/164047109 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5595cf1d1660 0x5595cf1d3a50 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.950+0000 7fbaa5b22740 1 --2- 172.21.15.12:0/164047109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595cf153c00 0x5595cf15d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.950+0000 7fbaa5b22740 1 --2- 172.21.15.12:0/164047109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595cf14ea90 0x5595cf15f580 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.950+0000 7fbaa5b22740 1 --2- 172.21.15.12:0/164047109 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595cf14e0a0 0x5595cf15f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.950+0000 7fbaa5b22740 1 -- 172.21.15.12:0/164047109 >> 172.21.15.12:0/164047109 conn(0x5595cf149610 msgr2=0x5595cf148830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:04.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.950+0000 7fba86000640 0 client.25615 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:04.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.950+0000 7fbaa5b22740 1 -- 172.21.15.12:0/164047109 shutdown_connections 2024-03-23T23:03:04.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.951+0000 7fbaa5b22740 1 -- 172.21.15.12:0/164047109 wait complete. 2024-03-23T23:03:04.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.951+0000 7fbaa5b22740 20 client.25615 trim_cache size 0 max 16384 2024-03-23T23:03:04.953 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:03:04.953 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-03-23T23:03:04.953 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "6024a9d1-efbc-48b5-be02-05c2c0782468", 2024-03-23T23:03:04.953 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-03-23T23:03:04.953 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:03:04.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.951+0000 7fbaa3200640 1 -- 172.21.15.12:0/3397994720 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fba800a2880 msgr2=0x7fba800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:04.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.951+0000 7fbaa3200640 1 --2- 172.21.15.12:0/3397994720 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fba800a2880 0x7fba800a4d40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x5595cf14f2f0 tx=0x7fba94009500 comp rx=0 tx=0).stop 2024-03-23T23:03:04.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.951+0000 7fbaa3200640 1 -- 172.21.15.12:0/3397994720 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba9c076060 msgr2=0x7fba9c19dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:04.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.951+0000 7fbaa3200640 1 --2- 172.21.15.12:0/3397994720 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba9c076060 0x7fba9c19dff0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fba8c00d950 tx=0x7fba8c00de10 comp rx=0 tx=0).stop 2024-03-23T23:03:04.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.952+0000 7fbaa3200640 1 -- 172.21.15.12:0/3397994720 shutdown_connections 2024-03-23T23:03:04.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.952+0000 7fbaa3200640 1 --2- 172.21.15.12:0/3397994720 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fba800a2880 0x7fba800a4d40 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.952+0000 7fbaa3200640 1 --2- 172.21.15.12:0/3397994720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba9c077c40 0x7fba9c1a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.952+0000 7fbaa3200640 1 --2- 172.21.15.12:0/3397994720 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba9c0772a0 0x7fba9c19e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.952+0000 7fbaa3200640 1 --2- 172.21.15.12:0/3397994720 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba9c076060 0x7fba9c19dff0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:04.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.952+0000 7fbaa3200640 1 -- 172.21.15.12:0/3397994720 >> 172.21.15.12:0/3397994720 conn(0x7fba9c0fee50 msgr2=0x7fba9c100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:04.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.952+0000 7fbaa3200640 1 -- 172.21.15.12:0/3397994720 shutdown_connections 2024-03-23T23:03:04.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:04.952+0000 7fbaa3200640 1 -- 172.21.15.12:0/3397994720 wait complete. 2024-03-23T23:03:04.969 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:03:04.969 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:03:05.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.161+0000 7f98d7200640 1 -- 172.21.15.12:0/2700614334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98d00ffdf0 msgr2=0x7f98d0102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:05.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.161+0000 7f98d7200640 1 --2- 172.21.15.12:0/2700614334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98d00ffdf0 0x7f98d0102220 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f98c4009900 tx=0x7f98c402bc40 comp rx=0 tx=0).stop 2024-03-23T23:03:05.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.161+0000 7f98d7200640 1 -- 172.21.15.12:0/2700614334 shutdown_connections 2024-03-23T23:03:05.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.161+0000 7f98d7200640 1 --2- 172.21.15.12:0/2700614334 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98d0105110 0x7f98d0105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.161+0000 7f98d7200640 1 --2- 172.21.15.12:0/2700614334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98d0102760 0x7f98d0104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.161+0000 7f98d7200640 1 --2- 172.21.15.12:0/2700614334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98d00ffdf0 0x7f98d0102220 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.161+0000 7f98d7200640 1 -- 172.21.15.12:0/2700614334 >> 172.21.15.12:0/2700614334 conn(0x7f98d00fba10 msgr2=0x7f98d00fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:05.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.161+0000 7f98d7200640 1 -- 172.21.15.12:0/2700614334 shutdown_connections 2024-03-23T23:03:05.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.162+0000 7f98d7200640 1 -- 172.21.15.12:0/2700614334 wait complete. 2024-03-23T23:03:05.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.163+0000 7f98d7200640 1 Processor -- start 2024-03-23T23:03:05.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.164+0000 7f98d7200640 1 -- start start 2024-03-23T23:03:05.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.164+0000 7f98d7200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98d00ffdf0 0x7f98d019de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:05.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.164+0000 7f98d7200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98d0102760 0x7f98d019e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:05.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.164+0000 7f98d7200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98d0105110 0x7f98d01a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:05.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.164+0000 7f98d7200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d0113820 con 0x7f98d0102760 2024-03-23T23:03:05.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.164+0000 7f98d7200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d01136a0 con 0x7f98d0105110 2024-03-23T23:03:05.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.164+0000 7f98d7200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d01139a0 con 0x7f98d00ffdf0 2024-03-23T23:03:05.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.164+0000 7f98d5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98d00ffdf0 0x7f98d019de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:05.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.164+0000 7f98d5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98d00ffdf0 0x7f98d019de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:39966/0 (socket says 172.21.15.12:39966) 2024-03-23T23:03:05.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.164+0000 7f98d5e00640 1 -- 172.21.15.12:0/2980966224 learned_addr learned my addr 172.21.15.12:0/2980966224 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:05.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.165+0000 7f98d5400640 1 --2- 172.21.15.12:0/2980966224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98d0102760 0x7f98d019e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:05.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.165+0000 7f98d5e00640 1 -- 172.21.15.12:0/2980966224 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98d0105110 msgr2=0x7f98d01a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:05.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.165+0000 7f98d5e00640 1 --2- 172.21.15.12:0/2980966224 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98d0105110 0x7f98d01a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.165+0000 7f98d5e00640 1 -- 172.21.15.12:0/2980966224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98d0102760 msgr2=0x7f98d019e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:05.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.165+0000 7f98d5e00640 1 --2- 172.21.15.12:0/2980966224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98d0102760 0x7f98d019e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.165+0000 7f98d5e00640 1 -- 172.21.15.12:0/2980966224 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d01a5aa0 con 0x7f98d00ffdf0 2024-03-23T23:03:05.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.165+0000 7f98d5400640 1 --2- 172.21.15.12:0/2980966224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98d0102760 0x7f98d019e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:03:05.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.165+0000 7f98d5e00640 1 --2- 172.21.15.12:0/2980966224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98d00ffdf0 0x7f98d019de80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f98c402c8f0 tx=0x7f98c402c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:05.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.165+0000 7f98caa00640 1 -- 172.21.15.12:0/2980966224 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4041070 con 0x7f98d00ffdf0 2024-03-23T23:03:05.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.166+0000 7f98caa00640 1 -- 172.21.15.12:0/2980966224 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f98c4004450 con 0x7f98d00ffdf0 2024-03-23T23:03:05.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.166+0000 7f98caa00640 1 -- 172.21.15.12:0/2980966224 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4034440 con 0x7f98d00ffdf0 2024-03-23T23:03:05.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.166+0000 7f98d7200640 1 -- 172.21.15.12:0/2980966224 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d01a5d30 con 0x7f98d00ffdf0 2024-03-23T23:03:05.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.166+0000 7f98d7200640 1 -- 172.21.15.12:0/2980966224 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d01a61d0 con 0x7f98d00ffdf0 2024-03-23T23:03:05.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.170+0000 7f98db59d740 1 Processor -- start 2024-03-23T23:03:05.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.170+0000 7f98db59d740 1 -- start start 2024-03-23T23:03:05.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.171+0000 7f98db59d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582d50b8050 0x5582d50b8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:05.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.171+0000 7f98db59d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582d50b8a40 0x5582d50bd540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:05.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.171+0000 7f98db59d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50bdbb0 0x5582d50c0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:05.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.171+0000 7f98db59d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582d501d250 con 0x5582d50b8050 2024-03-23T23:03:05.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.171+0000 7f98db59d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582d4fc5a30 con 0x5582d50bdbb0 2024-03-23T23:03:05.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.171+0000 7f98db59d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582d50192f0 con 0x5582d50b8a40 2024-03-23T23:03:05.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.171+0000 7f98caa00640 1 -- 172.21.15.12:0/2980966224 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f98c4044070 con 0x7f98d00ffdf0 2024-03-23T23:03:05.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.171+0000 7f98d5400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50bdbb0 0x5582d50c0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:05.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.171+0000 7f98d5400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50bdbb0 0x5582d50c0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59100/0 (socket says 172.21.15.12:59100) 2024-03-23T23:03:05.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.171+0000 7f98d5400640 1 -- 172.21.15.12:0/2337349941 learned_addr learned my addr 172.21.15.12:0/2337349941 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:05.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.172+0000 7f98d5e00640 1 --2- 172.21.15.12:0/2337349941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582d50b8a40 0x5582d50bd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:05.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.172+0000 7f98d6800640 1 --2- 172.21.15.12:0/2337349941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582d50b8050 0x5582d50b8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:05.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.172+0000 7f98d5e00640 1 -- 172.21.15.12:0/2337349941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50bdbb0 msgr2=0x5582d50c0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:05.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.172+0000 7f98d5e00640 1 --2- 172.21.15.12:0/2337349941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50bdbb0 0x5582d50c0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.172+0000 7f98d5e00640 1 -- 172.21.15.12:0/2337349941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582d50b8050 msgr2=0x5582d50b8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:05.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.172+0000 7f98d5e00640 1 --2- 172.21.15.12:0/2337349941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582d50b8050 0x5582d50b8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.172+0000 7f98d5e00640 1 -- 172.21.15.12:0/2337349941 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5582d4fb0690 con 0x5582d50b8a40 2024-03-23T23:03:05.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.172+0000 7f98d5400640 1 --2- 172.21.15.12:0/2337349941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50bdbb0 0x5582d50c0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:05.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.172+0000 7f98d5e00640 1 --2- 172.21.15.12:0/2337349941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582d50b8a40 0x5582d50bd540 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f98c40edc40 tx=0x7f98c40f8270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62f7f3117d0fe552 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:05.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.173+0000 7f98bf000640 1 -- 172.21.15.12:0/2337349941 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c40f8a70 con 0x5582d50b8a40 2024-03-23T23:03:05.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.173+0000 7f98bf000640 1 -- 172.21.15.12:0/2337349941 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f98c40fd030 con 0x5582d50b8a40 2024-03-23T23:03:05.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.173+0000 7f98bf000640 1 -- 172.21.15.12:0/2337349941 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c41015d0 con 0x5582d50b8a40 2024-03-23T23:03:05.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.173+0000 7f98caa00640 1 --2- 172.21.15.12:0/2980966224 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f98b80a2880 0x7f98b80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:05.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.174+0000 7f98db59d740 1 -- 172.21.15.12:0/2337349941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582d50b8a40 msgr2=0x5582d50bd540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:05.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.174+0000 7f98db59d740 1 --2- 172.21.15.12:0/2337349941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582d50b8a40 0x5582d50bd540 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f98c40edc40 tx=0x7f98c40f8270 comp rx=0 tx=0).stop 2024-03-23T23:03:05.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.174+0000 7f98caa00640 1 -- 172.21.15.12:0/2980966224 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f98c40f01b0 con 0x7f98d00ffdf0 2024-03-23T23:03:05.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.174+0000 7f98d6800640 1 --2- 172.21.15.12:0/2980966224 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f98b80a2880 0x7f98b80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:05.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.174+0000 7f98db59d740 1 -- 172.21.15.12:0/2337349941 shutdown_connections 2024-03-23T23:03:05.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.174+0000 7f98db59d740 1 --2- 172.21.15.12:0/2337349941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50bdbb0 0x5582d50c0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.174+0000 7f98db59d740 1 --2- 172.21.15.12:0/2337349941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582d50b8a40 0x5582d50bd540 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.174+0000 7f98db59d740 1 --2- 172.21.15.12:0/2337349941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582d50b8050 0x5582d50b8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.174+0000 7f98db59d740 1 -- 172.21.15.12:0/2337349941 >> 172.21.15.12:0/2337349941 conn(0x5582d5080200 msgr2=0x5582d50b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:05.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.175+0000 7f98db59d740 1 -- 172.21.15.12:0/2337349941 shutdown_connections 2024-03-23T23:03:05.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.175+0000 7f98db59d740 1 -- 172.21.15.12:0/2337349941 wait complete. 2024-03-23T23:03:05.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.175+0000 7f98d6800640 1 --2- 172.21.15.12:0/2980966224 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f98b80a2880 0x7f98b80a4d40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x5582d50b92a0 tx=0x7f98cc009200 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:05.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.176+0000 7f98db59d740 1 -- start start 2024-03-23T23:03:05.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.176+0000 7f98db59d740 1 Processor -- start 2024-03-23T23:03:05.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.176+0000 7f98db59d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50b8050 0x5582d50c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:05.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.177+0000 7f98db59d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582d50b8a40 0x5582d50c2290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:05.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.177+0000 7f98db59d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582d50bdbb0 0x5582d50c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:05.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.177+0000 7f98db59d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582d50c0920 con 0x5582d50bdbb0 2024-03-23T23:03:05.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.177+0000 7f98db59d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582d4fd8e80 con 0x5582d50b8050 2024-03-23T23:03:05.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.177+0000 7f98db59d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582d50c0aa0 con 0x5582d50b8a40 2024-03-23T23:03:05.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.177+0000 7f98db59d740 10 client.0 _ll_register_callbacks cb 0x5582d4feb130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:05.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.177+0000 7f98d6800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582d50bdbb0 0x5582d50c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:05.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.177+0000 7f98d5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50b8050 0x5582d50c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:05.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.177+0000 7f98d5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50b8050 0x5582d50c1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59108/0 (socket says 172.21.15.12:59108) 2024-03-23T23:03:05.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.177+0000 7f98d5e00640 1 -- 172.21.15.12:0/3737987918 learned_addr learned my addr 172.21.15.12:0/3737987918 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:05.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.177+0000 7f98d5400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582d50b8a40 0x5582d50c2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:05.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.178+0000 7f98d5e00640 1 -- 172.21.15.12:0/3737987918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582d50b8a40 msgr2=0x5582d50c2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:05.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.178+0000 7f98d5e00640 1 --2- 172.21.15.12:0/3737987918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582d50b8a40 0x5582d50c2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.178+0000 7f98d5e00640 1 -- 172.21.15.12:0/3737987918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582d50bdbb0 msgr2=0x5582d50c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:05.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.178+0000 7f98d5e00640 1 --2- 172.21.15.12:0/3737987918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582d50bdbb0 0x5582d50c92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.178+0000 7f98d5e00640 1 -- 172.21.15.12:0/3737987918 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5582d4dfb7d0 con 0x5582d50b8050 2024-03-23T23:03:05.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.178+0000 7f98d5400640 1 --2- 172.21.15.12:0/3737987918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582d50b8a40 0x5582d50c2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:05.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.178+0000 7f98d6800640 1 --2- 172.21.15.12:0/3737987918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582d50bdbb0 0x5582d50c92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:05.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.178+0000 7f98d5e00640 1 --2- 172.21.15.12:0/3737987918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50b8050 0x5582d50c1d50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f98c40b9510 tx=0x7f98c402c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ed8799350ca527d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:05.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.178+0000 7f98b3400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:03:05.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.179+0000 7f98b3400640 1 -- 172.21.15.12:0/3737987918 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c40b8de0 con 0x5582d50b8050 2024-03-23T23:03:05.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.179+0000 7f98b3400640 1 -- 172.21.15.12:0/3737987918 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f98c40eddb0 con 0x5582d50b8050 2024-03-23T23:03:05.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.179+0000 7f98db59d740 1 -- 172.21.15.12:0/3737987918 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f98c40fd030 con 0x5582d50b8050 2024-03-23T23:03:05.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.179+0000 7f98b3400640 1 -- 172.21.15.12:0/3737987918 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f98c40bc4a0 con 0x5582d50b8050 2024-03-23T23:03:05.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.179+0000 7f98db59d740 10 client.25649 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:05.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.179+0000 7f98db59d740 1 -- 172.21.15.12:0/3737987918 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5582d4d313f0 con 0x5582d50b8050 2024-03-23T23:03:05.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.179+0000 7f98b3400640 1 -- 172.21.15.12:0/3737987918 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f98c40fd030 con 0x5582d50b8050 2024-03-23T23:03:05.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.180+0000 7f98b3400640 1 -- 172.21.15.12:0/3737987918 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f98c40035f0 con 0x5582d50b8050 2024-03-23T23:03:05.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.180+0000 7f98db59d740 10 client.25649 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:05.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.180+0000 7f98db59d740 10 client.25649 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:05.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.180+0000 7f98db59d740 20 client.25649 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:05.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.180+0000 7f98db59d740 1 --2- 172.21.15.12:0/3737987918 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5582d50cd810 0x5582d50cfc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:05.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.180+0000 7f98db59d740 4 client.25649 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:03:05.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.180+0000 7f98db59d740 1 -- 172.21.15.12:0/3737987918 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5582d502a790 con 0x5582d50cd810 2024-03-23T23:03:05.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.181+0000 7f98d5400640 1 --2- 172.21.15.12:0/3737987918 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5582d50cd810 0x5582d50cfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:05.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.182+0000 7f98d5400640 1 --2- 172.21.15.12:0/3737987918 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5582d50cd810 0x5582d50cfc00 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bf7183726d1d54b server_cookie=cd2a365f6809f90b in_seq=0 out_seq=0 2024-03-23T23:03:05.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.182+0000 7f98b3400640 10 client.25649 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:05.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.183+0000 7f98b3400640 1 -- 172.21.15.12:0/3737987918 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5582d502a790 con 0x5582d50cd810 2024-03-23T23:03:05.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.183+0000 7f98b3400640 10 client.25649 handle_command_reply: tid=0 2024-03-23T23:03:05.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.183+0000 7f98db59d740 1 client.25649 shutdown 2024-03-23T23:03:05.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.183+0000 7f98db59d740 1 -- 172.21.15.12:0/3737987918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50b8050 msgr2=0x5582d50c1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:05.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.183+0000 7f98db59d740 1 --2- 172.21.15.12:0/3737987918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50b8050 0x5582d50c1d50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f98c40b9510 tx=0x7f98c402c480 comp rx=0 tx=0).stop 2024-03-23T23:03:05.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.184+0000 7f98db59d740 1 -- 172.21.15.12:0/3737987918 shutdown_connections 2024-03-23T23:03:05.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.184+0000 7f98db59d740 1 --2- 172.21.15.12:0/3737987918 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5582d50cd810 0x5582d50cfc00 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.184+0000 7f98db59d740 1 --2- 172.21.15.12:0/3737987918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582d50bdbb0 0x5582d50c92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.184+0000 7f98db59d740 1 --2- 172.21.15.12:0/3737987918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582d50b8a40 0x5582d50c2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.184+0000 7f98db59d740 1 --2- 172.21.15.12:0/3737987918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50b8050 0x5582d50c1d50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.184+0000 7f98db59d740 1 -- 172.21.15.12:0/3737987918 >> 172.21.15.12:0/3737987918 conn(0x5582d5080200 msgr2=0x5582d50b2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:05.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.184+0000 7f98b3400640 0 client.25649 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:05.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.184+0000 7f98db59d740 1 -- 172.21.15.12:0/3737987918 shutdown_connections 2024-03-23T23:03:05.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.184+0000 7f98db59d740 1 -- 172.21.15.12:0/3737987918 wait complete. 2024-03-23T23:03:05.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.184+0000 7f98db59d740 20 client.25649 trim_cache size 0 max 16384 2024-03-23T23:03:05.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.218+0000 7f98db59d740 1 Processor -- start 2024-03-23T23:03:05.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.218+0000 7f98db59d740 1 -- start start 2024-03-23T23:03:05.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.218+0000 7f98db59d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50b8050 0x5582d50c21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:05.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.218+0000 7f98db59d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582d50b8a40 0x5582d50c9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:05.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.218+0000 7f98db59d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582d50bdbb0 0x5582d50b9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:05.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.218+0000 7f98db59d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d01b1390 con 0x5582d50b8a40 2024-03-23T23:03:05.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.218+0000 7f98db59d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d01aee90 con 0x5582d50b8050 2024-03-23T23:03:05.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.218+0000 7f98db59d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d01acf30 con 0x5582d50bdbb0 2024-03-23T23:03:05.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.219+0000 7f98d6800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582d50bdbb0 0x5582d50b9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:05.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.219+0000 7f98d5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50b8050 0x5582d50c21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:05.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.219+0000 7f98d6800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582d50bdbb0 0x5582d50b9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:39990/0 (socket says 172.21.15.12:39990) 2024-03-23T23:03:05.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.219+0000 7f98d5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50b8050 0x5582d50c21f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59116/0 (socket says 172.21.15.12:59116) 2024-03-23T23:03:05.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.219+0000 7f98d6800640 1 -- 172.21.15.12:0/3881472790 learned_addr learned my addr 172.21.15.12:0/3881472790 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:05.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.219+0000 7f98d5400640 1 --2- 172.21.15.12:0/3881472790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582d50b8a40 0x5582d50c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:05.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.219+0000 7f98d5e00640 1 -- 172.21.15.12:0/3881472790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582d50bdbb0 msgr2=0x5582d50b9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:05.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.219+0000 7f98d5e00640 1 --2- 172.21.15.12:0/3881472790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582d50bdbb0 0x5582d50b9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.219+0000 7f98d5e00640 1 -- 172.21.15.12:0/3881472790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582d50b8a40 msgr2=0x5582d50c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:05.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.219+0000 7f98d5e00640 1 --2- 172.21.15.12:0/3881472790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582d50b8a40 0x5582d50c9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.219+0000 7f98d5e00640 1 -- 172.21.15.12:0/3881472790 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5582d4cf10f0 con 0x5582d50b8050 2024-03-23T23:03:05.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.219+0000 7f98d5e00640 1 --2- 172.21.15.12:0/3881472790 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50b8050 0x5582d50c21f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f98c40badf0 tx=0x7f98c40f8270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52644dd1a4d20e2b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:05.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.220+0000 7f98be600640 1 -- 172.21.15.12:0/3881472790 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c40bc680 con 0x5582d50b8050 2024-03-23T23:03:05.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.220+0000 7f98be600640 1 -- 172.21.15.12:0/3881472790 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f98c40fd070 con 0x5582d50b8050 2024-03-23T23:03:05.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.220+0000 7f98be600640 1 -- 172.21.15.12:0/3881472790 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c40bc910 con 0x5582d50b8050 2024-03-23T23:03:05.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.220+0000 7f98db59d740 1 -- 172.21.15.12:0/3881472790 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50b8050 msgr2=0x5582d50c21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:05.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.220+0000 7f98db59d740 1 --2- 172.21.15.12:0/3881472790 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50b8050 0x5582d50c21f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f98c40badf0 tx=0x7f98c40f8270 comp rx=0 tx=0).stop 2024-03-23T23:03:05.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.220+0000 7f98db59d740 1 -- 172.21.15.12:0/3881472790 shutdown_connections 2024-03-23T23:03:05.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.220+0000 7f98db59d740 1 --2- 172.21.15.12:0/3881472790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582d50bdbb0 0x5582d50b9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.221+0000 7f98db59d740 1 --2- 172.21.15.12:0/3881472790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582d50b8a40 0x5582d50c9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.221+0000 7f98db59d740 1 --2- 172.21.15.12:0/3881472790 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50b8050 0x5582d50c21f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.221+0000 7f98db59d740 1 -- 172.21.15.12:0/3881472790 >> 172.21.15.12:0/3881472790 conn(0x5582d5080200 msgr2=0x5582d50b94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:05.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.221+0000 7f98db59d740 1 -- 172.21.15.12:0/3881472790 shutdown_connections 2024-03-23T23:03:05.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.221+0000 7f98db59d740 1 -- 172.21.15.12:0/3881472790 wait complete. 2024-03-23T23:03:05.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.221+0000 7f98db59d740 1 -- start start 2024-03-23T23:03:05.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.221+0000 7f98db59d740 1 Processor -- start 2024-03-23T23:03:05.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.222+0000 7f98db59d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582d50b8050 0x5582d50c5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:05.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.222+0000 7f98db59d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582d50b8a40 0x5582d50c6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:05.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.222+0000 7f98db59d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50bdbb0 0x5582d5127080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:05.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.222+0000 7f98db59d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d01ad530 con 0x5582d50b8050 2024-03-23T23:03:05.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.222+0000 7f98db59d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d01ae830 con 0x5582d50bdbb0 2024-03-23T23:03:05.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.222+0000 7f98db59d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d01adb70 con 0x5582d50b8a40 2024-03-23T23:03:05.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.222+0000 7f98db59d740 10 client.0 _ll_register_callbacks cb 0x7f98cc009820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:05.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.222+0000 7f98d5e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582d50b8050 0x5582d50c5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:05.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.222+0000 7f98d6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50bdbb0 0x5582d5127080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:05.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.222+0000 7f98d5400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582d50b8a40 0x5582d50c6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:05.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.222+0000 7f98d6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50bdbb0 0x5582d5127080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59118/0 (socket says 172.21.15.12:59118) 2024-03-23T23:03:05.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.222+0000 7f98d6800640 1 -- 172.21.15.12:0/2248096399 learned_addr learned my addr 172.21.15.12:0/2248096399 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:05.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.222+0000 7f98d6800640 1 -- 172.21.15.12:0/2248096399 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582d50b8a40 msgr2=0x5582d50c6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:05.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.222+0000 7f98d6800640 1 --2- 172.21.15.12:0/2248096399 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582d50b8a40 0x5582d50c6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.222+0000 7f98d6800640 1 -- 172.21.15.12:0/2248096399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582d50b8050 msgr2=0x5582d50c5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:05.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.222+0000 7f98d6800640 1 --2- 172.21.15.12:0/2248096399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582d50b8050 0x5582d50c5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.222+0000 7f98d6800640 1 -- 172.21.15.12:0/2248096399 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5582d50c6b90 con 0x5582d50bdbb0 2024-03-23T23:03:05.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.223+0000 7f98d5400640 1 --2- 172.21.15.12:0/2248096399 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582d50b8a40 0x5582d50c6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:05.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.223+0000 7f98d6800640 1 --2- 172.21.15.12:0/2248096399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50bdbb0 0x5582d5127080 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f98cc004840 tx=0x7f98cc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a77efef4e72fcad2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:05.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.223+0000 7f98b3400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:03:05.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.223+0000 7f98b3400640 1 -- 172.21.15.12:0/2248096399 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc004b20 con 0x5582d50bdbb0 2024-03-23T23:03:05.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.223+0000 7f98b3400640 1 -- 172.21.15.12:0/2248096399 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f98cc004cc0 con 0x5582d50bdbb0 2024-03-23T23:03:05.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.223+0000 7f98db59d740 1 -- 172.21.15.12:0/2248096399 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f98c40fd070 con 0x5582d50bdbb0 2024-03-23T23:03:05.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.224+0000 7f98b3400640 1 -- 172.21.15.12:0/2248096399 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f98cc005020 con 0x5582d50bdbb0 2024-03-23T23:03:05.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.224+0000 7f98db59d740 10 client.25655 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:05.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.224+0000 7f98db59d740 1 -- 172.21.15.12:0/2248096399 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5582d51287a0 con 0x5582d50bdbb0 2024-03-23T23:03:05.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.224+0000 7f98b3400640 1 -- 172.21.15.12:0/2248096399 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f98c40fd070 con 0x5582d50bdbb0 2024-03-23T23:03:05.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.225+0000 7f98b3400640 1 -- 172.21.15.12:0/2248096399 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f98cc0051c0 con 0x5582d50bdbb0 2024-03-23T23:03:05.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.225+0000 7f98db59d740 10 client.25655 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:05.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.225+0000 7f98db59d740 10 client.25655 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:05.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.225+0000 7f98db59d740 20 client.25655 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:05.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.225+0000 7f98db59d740 1 --2- 172.21.15.12:0/2248096399 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5582d51290e0 0x5582d512b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:05.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.225+0000 7f98db59d740 4 client.25655 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:03:05.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.225+0000 7f98db59d740 1 -- 172.21.15.12:0/2248096399 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5582d5024ae0 con 0x5582d51290e0 2024-03-23T23:03:05.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.225+0000 7f98d5e00640 1 --2- 172.21.15.12:0/2248096399 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5582d51290e0 0x5582d512b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:05.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.226+0000 7f98d5e00640 1 --2- 172.21.15.12:0/2248096399 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5582d51290e0 0x5582d512b4d0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa242c90d147aca server_cookie=723131602e7e2320 in_seq=0 out_seq=0 2024-03-23T23:03:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.226+0000 7f98b3400640 10 client.25655 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.226+0000 7f98b3400640 1 -- 172.21.15.12:0/2248096399 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5582d5024ae0 con 0x5582d51290e0 2024-03-23T23:03:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.226+0000 7f98b3400640 10 client.25655 handle_command_reply: tid=0 2024-03-23T23:03:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.226+0000 7f98db59d740 1 client.25655 shutdown 2024-03-23T23:03:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.226+0000 7f98db59d740 1 -- 172.21.15.12:0/2248096399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50bdbb0 msgr2=0x5582d5127080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.226+0000 7f98db59d740 1 --2- 172.21.15.12:0/2248096399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50bdbb0 0x5582d5127080 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f98cc004840 tx=0x7f98cc004a40 comp rx=0 tx=0).stop 2024-03-23T23:03:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.226+0000 7f98db59d740 1 -- 172.21.15.12:0/2248096399 shutdown_connections 2024-03-23T23:03:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.226+0000 7f98db59d740 1 --2- 172.21.15.12:0/2248096399 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5582d51290e0 0x5582d512b4d0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.226+0000 7f98db59d740 1 --2- 172.21.15.12:0/2248096399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582d50bdbb0 0x5582d5127080 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.226+0000 7f98db59d740 1 --2- 172.21.15.12:0/2248096399 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582d50b8a40 0x5582d50c6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.226+0000 7f98b3400640 0 client.25655 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.226+0000 7f98db59d740 1 --2- 172.21.15.12:0/2248096399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582d50b8050 0x5582d50c5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.226+0000 7f98db59d740 1 -- 172.21.15.12:0/2248096399 >> 172.21.15.12:0/2248096399 conn(0x5582d5080200 msgr2=0x5582d50b9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.227+0000 7f98db59d740 1 -- 172.21.15.12:0/2248096399 shutdown_connections 2024-03-23T23:03:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.227+0000 7f98db59d740 1 -- 172.21.15.12:0/2248096399 wait complete. 2024-03-23T23:03:05.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.227+0000 7f98db59d740 20 client.25655 trim_cache size 0 max 16384 2024-03-23T23:03:05.230 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:03:05.230 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T23:03:05.230 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-03-23T23:03:05.230 INFO:teuthology.orchestra.run.smithi012.stdout: "6024a9d1-efbc-48b5-be02-05c2c0782468": { 2024-03-23T23:03:05.230 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-03-23T23:03:05.230 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "6024a9d1-efbc-48b5-be02-05c2c0782468", 2024-03-23T23:03:05.230 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-03-23T23:03:05.230 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:03:05.230 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:03:05.231 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:03:05.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.228+0000 7f98d7200640 1 -- 172.21.15.12:0/2980966224 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f98b80a2880 msgr2=0x7f98b80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:05.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.228+0000 7f98d7200640 1 --2- 172.21.15.12:0/2980966224 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f98b80a2880 0x7f98b80a4d40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x5582d50b92a0 tx=0x7f98cc009200 comp rx=0 tx=0).stop 2024-03-23T23:03:05.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.228+0000 7f98d7200640 1 -- 172.21.15.12:0/2980966224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98d00ffdf0 msgr2=0x7f98d019de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:05.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.228+0000 7f98d7200640 1 --2- 172.21.15.12:0/2980966224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98d00ffdf0 0x7f98d019de80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f98c402c8f0 tx=0x7f98c402c660 comp rx=0 tx=0).stop 2024-03-23T23:03:05.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.228+0000 7f98d7200640 1 -- 172.21.15.12:0/2980966224 shutdown_connections 2024-03-23T23:03:05.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.228+0000 7f98d7200640 1 --2- 172.21.15.12:0/2980966224 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f98b80a2880 0x7f98b80a4d40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.228+0000 7f98d7200640 1 --2- 172.21.15.12:0/2980966224 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98d0105110 0x7f98d01a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.228+0000 7f98d7200640 1 --2- 172.21.15.12:0/2980966224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98d0102760 0x7f98d019e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.228+0000 7f98d7200640 1 --2- 172.21.15.12:0/2980966224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98d00ffdf0 0x7f98d019de80 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:05.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.228+0000 7f98d7200640 1 -- 172.21.15.12:0/2980966224 >> 172.21.15.12:0/2980966224 conn(0x7f98d00fba10 msgr2=0x7f98d01030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:05.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.228+0000 7f98d7200640 1 -- 172.21.15.12:0/2980966224 shutdown_connections 2024-03-23T23:03:05.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:05.228+0000 7f98d7200640 1 -- 172.21.15.12:0/2980966224 wait complete. 2024-03-23T23:03:05.252 INFO:tasks.cephfs.filesystem:scrub status for tag:6024a9d1-efbc-48b5-be02-05c2c0782468 - {'path': '/', 'tag': '6024a9d1-efbc-48b5-be02-05c2c0782468', 'options': 'recursive,force'} 2024-03-23T23:03:06.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:06 smithi012 ceph-mon[31695]: scrub queued for path: / 2024-03-23T23:03:06.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:06 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:03:06.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:06 smithi012 ceph-mon[31695]: scrub summary: active paths [/] 2024-03-23T23:03:06.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:06 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:03:06.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:06 smithi012 ceph-mon[31695]: pgmap v429: 97 pgs: 97 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-23T23:03:06.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:06 smithi012 ceph-mon[31695]: scrub summary: idle 2024-03-23T23:03:06.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:06 smithi173 ceph-mon[36116]: scrub queued for path: / 2024-03-23T23:03:06.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:06 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:03:06.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:06 smithi173 ceph-mon[36116]: scrub summary: active paths [/] 2024-03-23T23:03:06.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:06 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:03:06.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:06 smithi173 ceph-mon[36116]: pgmap v429: 97 pgs: 97 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-23T23:03:06.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:06 smithi173 ceph-mon[36116]: scrub summary: idle 2024-03-23T23:03:06.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:06 smithi203 ceph-mon[36693]: scrub queued for path: / 2024-03-23T23:03:06.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:06 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:03:06.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:06 smithi203 ceph-mon[36693]: scrub summary: active paths [/] 2024-03-23T23:03:06.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:06 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:03:06.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:06 smithi203 ceph-mon[36693]: pgmap v429: 97 pgs: 97 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-23T23:03:06.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:06 smithi203 ceph-mon[36693]: scrub summary: idle 2024-03-23T23:03:08.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:08 smithi012 ceph-mon[31695]: pgmap v430: 97 pgs: 97 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-23T23:03:08.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:08 smithi173 ceph-mon[36116]: pgmap v430: 97 pgs: 97 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-23T23:03:08.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:08 smithi203 ceph-mon[36693]: pgmap v430: 97 pgs: 97 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-23T23:03:10.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:10 smithi012 ceph-mon[31695]: pgmap v431: 97 pgs: 97 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-23T23:03:10.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:10 smithi173 ceph-mon[36116]: pgmap v431: 97 pgs: 97 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-23T23:03:10.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:10 smithi203 ceph-mon[36693]: pgmap v431: 97 pgs: 97 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-23T23:03:12.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:12 smithi012 ceph-mon[31695]: pgmap v432: 97 pgs: 97 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-23T23:03:12.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:12 smithi173 ceph-mon[36116]: pgmap v432: 97 pgs: 97 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-23T23:03:12.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:12 smithi203 ceph-mon[36693]: pgmap v432: 97 pgs: 97 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-23T23:03:14.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:14 smithi012 ceph-mon[31695]: pgmap v433: 97 pgs: 97 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-23T23:03:14.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:14 smithi173 ceph-mon[36116]: pgmap v433: 97 pgs: 97 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-23T23:03:14.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:14 smithi203 ceph-mon[36693]: pgmap v433: 97 pgs: 97 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-23T23:03:16.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:16 smithi012 ceph-mon[31695]: pgmap v434: 97 pgs: 97 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-23T23:03:16.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:16 smithi173 ceph-mon[36116]: pgmap v434: 97 pgs: 97 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-23T23:03:16.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:16 smithi203 ceph-mon[36693]: pgmap v434: 97 pgs: 97 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-23T23:03:17.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.580+0000 7ffb85c00640 1 -- 172.21.15.12:0/308953203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb80103740 msgr2=0x7ffb80103b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:17.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.580+0000 7ffb85c00640 1 --2- 172.21.15.12:0/308953203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb80103740 0x7ffb80103b60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ffb70009900 tx=0x7ffb7002bc40 comp rx=0 tx=0).stop 2024-03-23T23:03:17.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.580+0000 7ffb85c00640 1 -- 172.21.15.12:0/308953203 shutdown_connections 2024-03-23T23:03:17.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.580+0000 7ffb85c00640 1 --2- 172.21.15.12:0/308953203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb801052f0 0x7ffb8010bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.580+0000 7ffb85c00640 1 --2- 172.21.15.12:0/308953203 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb80104930 0x7ffb80104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.580+0000 7ffb85c00640 1 --2- 172.21.15.12:0/308953203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb80103740 0x7ffb80103b60 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.580+0000 7ffb85c00640 1 -- 172.21.15.12:0/308953203 >> 172.21.15.12:0/308953203 conn(0x7ffb800fecb0 msgr2=0x7ffb801010f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:17.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.580+0000 7ffb85c00640 1 -- 172.21.15.12:0/308953203 shutdown_connections 2024-03-23T23:03:17.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.580+0000 7ffb85c00640 1 -- 172.21.15.12:0/308953203 wait complete. 2024-03-23T23:03:17.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.581+0000 7ffb85c00640 1 Processor -- start 2024-03-23T23:03:17.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.582+0000 7ffb85c00640 1 -- start start 2024-03-23T23:03:17.582 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.582+0000 7ffb85c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb80103740 0x7ffb80199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:17.582 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.582+0000 7ffb85c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb80104930 0x7ffb80199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:17.582 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.582+0000 7ffb85c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb801052f0 0x7ffb801a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:17.582 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.582+0000 7ffb85c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8010f400 con 0x7ffb80104930 2024-03-23T23:03:17.582 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.582+0000 7ffb85c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8010f280 con 0x7ffb801052f0 2024-03-23T23:03:17.582 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.582+0000 7ffb85c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8010f580 con 0x7ffb80103740 2024-03-23T23:03:17.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.582+0000 7ffb85200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb801052f0 0x7ffb801a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:17.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.582+0000 7ffb7fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb80103740 0x7ffb80199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:17.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.582+0000 7ffb7fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb80103740 0x7ffb80199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:59008/0 (socket says 172.21.15.12:59008) 2024-03-23T23:03:17.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.582+0000 7ffb7fe00640 1 -- 172.21.15.12:0/2018525732 learned_addr learned my addr 172.21.15.12:0/2018525732 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:17.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.583+0000 7ffb7f400640 1 --2- 172.21.15.12:0/2018525732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb80104930 0x7ffb80199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:17.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.583+0000 7ffb85200640 1 -- 172.21.15.12:0/2018525732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb80103740 msgr2=0x7ffb80199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:17.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.583+0000 7ffb85200640 1 --2- 172.21.15.12:0/2018525732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb80103740 0x7ffb80199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.583+0000 7ffb85200640 1 -- 172.21.15.12:0/2018525732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb80104930 msgr2=0x7ffb80199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.583+0000 7ffb85200640 1 --2- 172.21.15.12:0/2018525732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb80104930 0x7ffb80199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.583+0000 7ffb85200640 1 -- 172.21.15.12:0/2018525732 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb801a16a0 con 0x7ffb801052f0 2024-03-23T23:03:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.583+0000 7ffb7f400640 1 --2- 172.21.15.12:0/2018525732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb80104930 0x7ffb80199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:03:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.583+0000 7ffb85200640 1 --2- 172.21.15.12:0/2018525732 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb801052f0 0x7ffb801a1030 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ffb7800da10 tx=0x7ffb7800ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.584+0000 7ffb7cc00640 1 -- 172.21.15.12:0/2018525732 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7800bd70 con 0x7ffb801052f0 2024-03-23T23:03:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.584+0000 7ffb7cc00640 1 -- 172.21.15.12:0/2018525732 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb78003e50 con 0x7ffb801052f0 2024-03-23T23:03:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.584+0000 7ffb85c00640 1 -- 172.21.15.12:0/2018525732 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb801a1990 con 0x7ffb801052f0 2024-03-23T23:03:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.584+0000 7ffb7cc00640 1 -- 172.21.15.12:0/2018525732 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb780106a0 con 0x7ffb801052f0 2024-03-23T23:03:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.584+0000 7ffb85c00640 1 -- 172.21.15.12:0/2018525732 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb801a1ee0 con 0x7ffb801052f0 2024-03-23T23:03:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.587+0000 7ffb7cc00640 1 -- 172.21.15.12:0/2018525732 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ffb78003970 con 0x7ffb801052f0 2024-03-23T23:03:17.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.588+0000 7ffb7cc00640 1 --2- 172.21.15.12:0/2018525732 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ffb640a28f0 0x7ffb640a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:17.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.588+0000 7ffb89e0d740 1 Processor -- start 2024-03-23T23:03:17.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.588+0000 7ffb7cc00640 1 -- 172.21.15.12:0/2018525732 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ffb780ce650 con 0x7ffb801052f0 2024-03-23T23:03:17.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.589+0000 7ffb7fe00640 1 --2- 172.21.15.12:0/2018525732 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ffb640a28f0 0x7ffb640a4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:17.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.589+0000 7ffb89e0d740 1 -- start start 2024-03-23T23:03:17.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.589+0000 7ffb89e0d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55db3d73ec00 0x55db3d73f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:17.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.589+0000 7ffb89e0d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55db3d73f5f0 0x55db3d7440f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:17.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.589+0000 7ffb89e0d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d744760 0x55db3d746bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:17.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.589+0000 7ffb89e0d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3d6f93b0 con 0x55db3d73ec00 2024-03-23T23:03:17.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.589+0000 7ffb89e0d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3d69b830 con 0x55db3d73f5f0 2024-03-23T23:03:17.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.589+0000 7ffb89e0d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3d6445d0 con 0x55db3d744760 2024-03-23T23:03:17.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.590+0000 7ffb85200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d744760 0x55db3d746bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:17.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.590+0000 7ffb85200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d744760 0x55db3d746bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:59018/0 (socket says 172.21.15.12:59018) 2024-03-23T23:03:17.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.590+0000 7ffb85200640 1 -- 172.21.15.12:0/595979704 learned_addr learned my addr 172.21.15.12:0/595979704 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:17.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.590+0000 7ffb7fe00640 1 --2- 172.21.15.12:0/595979704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55db3d73ec00 0x55db3d73f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:17.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.590+0000 7ffb85200640 1 -- 172.21.15.12:0/595979704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55db3d73f5f0 msgr2=0x55db3d7440f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-23T23:03:17.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.590+0000 7ffb7f400640 1 --2- 172.21.15.12:0/595979704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55db3d73f5f0 0x55db3d7440f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:17.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.590+0000 7ffb85200640 1 --2- 172.21.15.12:0/595979704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55db3d73f5f0 0x55db3d7440f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.590+0000 7ffb85200640 1 -- 172.21.15.12:0/595979704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55db3d73ec00 msgr2=0x55db3d73f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:17.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.590+0000 7ffb85200640 1 --2- 172.21.15.12:0/595979704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55db3d73ec00 0x55db3d73f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.590+0000 7ffb85200640 1 -- 172.21.15.12:0/595979704 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db3d65c2c0 con 0x55db3d744760 2024-03-23T23:03:17.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.590+0000 7ffb7fe00640 1 --2- 172.21.15.12:0/595979704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55db3d73ec00 0x55db3d73f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:03:17.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.591+0000 7ffb7fe00640 1 --2- 172.21.15.12:0/2018525732 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ffb640a28f0 0x7ffb640a4db0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ffb70005ab0 tx=0x7ffb7002cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:17.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.591+0000 7ffb85200640 1 --2- 172.21.15.12:0/595979704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d744760 0x55db3d746bf0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ffb78097970 tx=0x7ffb7809aba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e631e2e6118b06be server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:17.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.591+0000 7ffb75600640 1 -- 172.21.15.12:0/595979704 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7809afa0 con 0x55db3d744760 2024-03-23T23:03:17.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.591+0000 7ffb75600640 1 -- 172.21.15.12:0/595979704 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb7809f0f0 con 0x55db3d744760 2024-03-23T23:03:17.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.591+0000 7ffb75600640 1 -- 172.21.15.12:0/595979704 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb78097610 con 0x55db3d744760 2024-03-23T23:03:17.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.592+0000 7ffb89e0d740 1 -- 172.21.15.12:0/595979704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d744760 msgr2=0x55db3d746bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:17.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.592+0000 7ffb89e0d740 1 --2- 172.21.15.12:0/595979704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d744760 0x55db3d746bf0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ffb78097970 tx=0x7ffb7809aba0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.592+0000 7ffb89e0d740 1 -- 172.21.15.12:0/595979704 shutdown_connections 2024-03-23T23:03:17.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.592+0000 7ffb89e0d740 1 --2- 172.21.15.12:0/595979704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d744760 0x55db3d746bf0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.592+0000 7ffb89e0d740 1 --2- 172.21.15.12:0/595979704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55db3d73f5f0 0x55db3d7440f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.592+0000 7ffb89e0d740 1 --2- 172.21.15.12:0/595979704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55db3d73ec00 0x55db3d73f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.592+0000 7ffb89e0d740 1 -- 172.21.15.12:0/595979704 >> 172.21.15.12:0/595979704 conn(0x55db3d73a170 msgr2=0x55db3d73a570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:17.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.592+0000 7ffb89e0d740 1 -- 172.21.15.12:0/595979704 shutdown_connections 2024-03-23T23:03:17.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.592+0000 7ffb89e0d740 1 -- 172.21.15.12:0/595979704 wait complete. 2024-03-23T23:03:17.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.593+0000 7ffb89e0d740 1 -- start start 2024-03-23T23:03:17.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.593+0000 7ffb89e0d740 1 Processor -- start 2024-03-23T23:03:17.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.594+0000 7ffb89e0d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55db3d73ec00 0x55db3d748db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:17.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.594+0000 7ffb89e0d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55db3d73f5f0 0x55db3d7492f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:17.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.594+0000 7ffb89e0d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d744760 0x55db3d750330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:17.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.594+0000 7ffb89e0d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3d657a20 con 0x55db3d73ec00 2024-03-23T23:03:17.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.594+0000 7ffb89e0d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3d6978d0 con 0x55db3d73f5f0 2024-03-23T23:03:17.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.594+0000 7ffb89e0d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3d747710 con 0x55db3d744760 2024-03-23T23:03:17.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.594+0000 7ffb89e0d740 10 client.0 _ll_register_callbacks cb 0x55db3d6941c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:17.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.594+0000 7ffb7fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55db3d73ec00 0x55db3d748db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:17.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.594+0000 7ffb7f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55db3d73f5f0 0x55db3d7492f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:17.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.594+0000 7ffb7fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55db3d73ec00 0x55db3d748db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38870/0 (socket says 172.21.15.12:38870) 2024-03-23T23:03:17.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.594+0000 7ffb7fe00640 1 -- 172.21.15.12:0/1565491104 learned_addr learned my addr 172.21.15.12:0/1565491104 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:17.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.594+0000 7ffb85200640 1 --2- 172.21.15.12:0/1565491104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d744760 0x55db3d750330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:17.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.594+0000 7ffb85200640 1 -- 172.21.15.12:0/1565491104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55db3d73f5f0 msgr2=0x55db3d7492f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:17.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.594+0000 7ffb85200640 1 --2- 172.21.15.12:0/1565491104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55db3d73f5f0 0x55db3d7492f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.594+0000 7ffb85200640 1 -- 172.21.15.12:0/1565491104 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55db3d73ec00 msgr2=0x55db3d748db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:17.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.594+0000 7ffb85200640 1 --2- 172.21.15.12:0/1565491104 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55db3d73ec00 0x55db3d748db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.594+0000 7ffb85200640 1 -- 172.21.15.12:0/1565491104 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55db3d3b0210 con 0x55db3d744760 2024-03-23T23:03:17.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.595+0000 7ffb85200640 1 --2- 172.21.15.12:0/1565491104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d744760 0x55db3d750330 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ffb780a3210 tx=0x7ffb78005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1822106a683f98b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:17.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.595+0000 7ffb6a000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:03:17.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.595+0000 7ffb6a000640 1 -- 172.21.15.12:0/1565491104 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb780992a0 con 0x55db3d744760 2024-03-23T23:03:17.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.595+0000 7ffb89e0d740 1 -- 172.21.15.12:0/1565491104 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb7809f0f0 con 0x55db3d744760 2024-03-23T23:03:17.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.595+0000 7ffb6a000640 1 -- 172.21.15.12:0/1565491104 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ffb78097610 con 0x55db3d744760 2024-03-23T23:03:17.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.595+0000 7ffb6a000640 1 -- 172.21.15.12:0/1565491104 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb780a32b0 con 0x55db3d744760 2024-03-23T23:03:17.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.595+0000 7ffb89e0d740 10 client.25645 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:17.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.595+0000 7ffb89e0d740 1 -- 172.21.15.12:0/1565491104 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55db3d47c620 con 0x55db3d744760 2024-03-23T23:03:17.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.595+0000 7ffb6a000640 1 -- 172.21.15.12:0/1565491104 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb7809f0f0 con 0x55db3d744760 2024-03-23T23:03:17.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.596+0000 7ffb6a000640 1 -- 172.21.15.12:0/1565491104 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ffb7809a3f0 con 0x55db3d744760 2024-03-23T23:03:17.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.596+0000 7ffb89e0d740 10 client.25645 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:17.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.596+0000 7ffb89e0d740 10 client.25645 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:17.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.596+0000 7ffb89e0d740 20 client.25645 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:17.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.596+0000 7ffb89e0d740 1 --2- 172.21.15.12:0/1565491104 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55db3d754790 0x55db3d756b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:17.598 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.596+0000 7ffb89e0d740 4 client.25645 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:03:17.598 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.596+0000 7ffb89e0d740 1 -- 172.21.15.12:0/1565491104 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55db3d6a8d70 con 0x55db3d754790 2024-03-23T23:03:17.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.596+0000 7ffb7fe00640 1 --2- 172.21.15.12:0/1565491104 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55db3d754790 0x55db3d756b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:17.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.597+0000 7ffb7fe00640 1 --2- 172.21.15.12:0/1565491104 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55db3d754790 0x55db3d756b80 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=864cf30cb29ffc6a server_cookie=ac934c0c4280ddad in_seq=0 out_seq=0 2024-03-23T23:03:17.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.597+0000 7ffb6a000640 10 client.25645 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:17.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.597+0000 7ffb6a000640 1 -- 172.21.15.12:0/1565491104 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55db3d6a8d70 con 0x55db3d754790 2024-03-23T23:03:17.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.597+0000 7ffb6a000640 10 client.25645 handle_command_reply: tid=0 2024-03-23T23:03:17.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.597+0000 7ffb89e0d740 1 client.25645 shutdown 2024-03-23T23:03:17.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.598+0000 7ffb89e0d740 1 -- 172.21.15.12:0/1565491104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d744760 msgr2=0x55db3d750330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:17.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.598+0000 7ffb89e0d740 1 --2- 172.21.15.12:0/1565491104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d744760 0x55db3d750330 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ffb780a3210 tx=0x7ffb78005f80 comp rx=0 tx=0).stop 2024-03-23T23:03:17.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.598+0000 7ffb89e0d740 1 -- 172.21.15.12:0/1565491104 shutdown_connections 2024-03-23T23:03:17.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.598+0000 7ffb89e0d740 1 --2- 172.21.15.12:0/1565491104 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55db3d754790 0x55db3d756b80 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.598+0000 7ffb89e0d740 1 --2- 172.21.15.12:0/1565491104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d744760 0x55db3d750330 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.598+0000 7ffb89e0d740 1 --2- 172.21.15.12:0/1565491104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55db3d73f5f0 0x55db3d7492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.598+0000 7ffb89e0d740 1 --2- 172.21.15.12:0/1565491104 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55db3d73ec00 0x55db3d748db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.598+0000 7ffb89e0d740 1 -- 172.21.15.12:0/1565491104 >> 172.21.15.12:0/1565491104 conn(0x55db3d73a170 msgr2=0x55db3d730b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:17.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.598+0000 7ffb6a000640 0 client.25645 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:17.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.598+0000 7ffb89e0d740 1 -- 172.21.15.12:0/1565491104 shutdown_connections 2024-03-23T23:03:17.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.598+0000 7ffb89e0d740 1 -- 172.21.15.12:0/1565491104 wait complete. 2024-03-23T23:03:17.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.598+0000 7ffb89e0d740 20 client.25645 trim_cache size 0 max 16384 2024-03-23T23:03:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.627+0000 7ffb89e0d740 1 Processor -- start 2024-03-23T23:03:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.627+0000 7ffb89e0d740 1 -- start start 2024-03-23T23:03:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.628+0000 7ffb89e0d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d73ec00 0x55db3d748db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.628+0000 7ffb89e0d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55db3d73f5f0 0x55db3d7492f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.628+0000 7ffb89e0d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55db3d744760 0x55db3d7502f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.628+0000 7ffb89e0d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb801a8c60 con 0x55db3d73f5f0 2024-03-23T23:03:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.628+0000 7ffb89e0d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb801a6760 con 0x55db3d744760 2024-03-23T23:03:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.628+0000 7ffb89e0d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb801a4800 con 0x55db3d73ec00 2024-03-23T23:03:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.628+0000 7ffb85200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55db3d744760 0x55db3d7502f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.628+0000 7ffb7fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d73ec00 0x55db3d748db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.628+0000 7ffb7fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d73ec00 0x55db3d748db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:59034/0 (socket says 172.21.15.12:59034) 2024-03-23T23:03:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.628+0000 7ffb85200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55db3d744760 0x55db3d7502f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:49586/0 (socket says 172.21.15.12:49586) 2024-03-23T23:03:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.628+0000 7ffb7fe00640 1 -- 172.21.15.12:0/3741219343 learned_addr learned my addr 172.21.15.12:0/3741219343 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.628+0000 7ffb7f400640 1 --2- 172.21.15.12:0/3741219343 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55db3d73f5f0 0x55db3d7492f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.628+0000 7ffb85200640 1 -- 172.21.15.12:0/3741219343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d73ec00 msgr2=0x55db3d748db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.628+0000 7ffb85200640 1 --2- 172.21.15.12:0/3741219343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d73ec00 0x55db3d748db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.629+0000 7ffb85200640 1 -- 172.21.15.12:0/3741219343 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55db3d73f5f0 msgr2=0x55db3d7492f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.629+0000 7ffb85200640 1 --2- 172.21.15.12:0/3741219343 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55db3d73f5f0 0x55db3d7492f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.629+0000 7ffb85200640 1 -- 172.21.15.12:0/3741219343 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db3d740400 con 0x55db3d744760 2024-03-23T23:03:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.629+0000 7ffb85200640 1 --2- 172.21.15.12:0/3741219343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55db3d744760 0x55db3d7502f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ffb7800b630 tx=0x7ffb7800bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56791f02f9e2df2a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.629+0000 7ffb74c00640 1 -- 172.21.15.12:0/3741219343 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb78097070 con 0x55db3d744760 2024-03-23T23:03:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.629+0000 7ffb74c00640 1 -- 172.21.15.12:0/3741219343 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb78004000 con 0x55db3d744760 2024-03-23T23:03:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.629+0000 7ffb74c00640 1 -- 172.21.15.12:0/3741219343 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb780a3450 con 0x55db3d744760 2024-03-23T23:03:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.630+0000 7ffb89e0d740 1 -- 172.21.15.12:0/3741219343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55db3d744760 msgr2=0x55db3d7502f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.630+0000 7ffb89e0d740 1 --2- 172.21.15.12:0/3741219343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55db3d744760 0x55db3d7502f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ffb7800b630 tx=0x7ffb7800bf10 comp rx=0 tx=0).stop 2024-03-23T23:03:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.630+0000 7ffb89e0d740 1 -- 172.21.15.12:0/3741219343 shutdown_connections 2024-03-23T23:03:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.630+0000 7ffb89e0d740 1 --2- 172.21.15.12:0/3741219343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55db3d744760 0x55db3d7502f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.630+0000 7ffb89e0d740 1 --2- 172.21.15.12:0/3741219343 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55db3d73f5f0 0x55db3d7492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.630+0000 7ffb89e0d740 1 --2- 172.21.15.12:0/3741219343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d73ec00 0x55db3d748db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.630+0000 7ffb89e0d740 1 -- 172.21.15.12:0/3741219343 >> 172.21.15.12:0/3741219343 conn(0x55db3d73a170 msgr2=0x55db3d7483d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.630+0000 7ffb89e0d740 1 -- 172.21.15.12:0/3741219343 shutdown_connections 2024-03-23T23:03:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.630+0000 7ffb89e0d740 1 -- 172.21.15.12:0/3741219343 wait complete. 2024-03-23T23:03:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.631+0000 7ffb89e0d740 1 -- start start 2024-03-23T23:03:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.631+0000 7ffb89e0d740 1 Processor -- start 2024-03-23T23:03:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.631+0000 7ffb89e0d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55db3d73ec00 0x55db3d74d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.631+0000 7ffb89e0d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d73f5f0 0x55db3d74d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:17.633 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.631+0000 7ffb89e0d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55db3d744760 0x55db3d7adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:17.633 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.632+0000 7ffb89e0d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb801a4e00 con 0x55db3d73ec00 2024-03-23T23:03:17.633 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.632+0000 7ffb89e0d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb801a6100 con 0x55db3d744760 2024-03-23T23:03:17.633 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.632+0000 7ffb89e0d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb801a5440 con 0x55db3d73f5f0 2024-03-23T23:03:17.633 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.632+0000 7ffb89e0d740 10 client.0 _ll_register_callbacks cb 0x7ffb7809ae00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:17.633 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.632+0000 7ffb7f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d73f5f0 0x55db3d74d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:17.633 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.632+0000 7ffb7f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d73f5f0 0x55db3d74d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:59040/0 (socket says 172.21.15.12:59040) 2024-03-23T23:03:17.633 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.632+0000 7ffb7fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55db3d73ec00 0x55db3d74d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:17.633 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.632+0000 7ffb7f400640 1 -- 172.21.15.12:0/3871890606 learned_addr learned my addr 172.21.15.12:0/3871890606 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:17.633 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.632+0000 7ffb85200640 1 --2- 172.21.15.12:0/3871890606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55db3d744760 0x55db3d7adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:17.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.632+0000 7ffb7f400640 1 -- 172.21.15.12:0/3871890606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55db3d744760 msgr2=0x55db3d7adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:17.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.632+0000 7ffb7f400640 1 --2- 172.21.15.12:0/3871890606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55db3d744760 0x55db3d7adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.632+0000 7ffb7f400640 1 -- 172.21.15.12:0/3871890606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55db3d73ec00 msgr2=0x55db3d74d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:17.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.632+0000 7ffb7f400640 1 --2- 172.21.15.12:0/3871890606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55db3d73ec00 0x55db3d74d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.632+0000 7ffb7f400640 1 -- 172.21.15.12:0/3871890606 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55db3d7ae4f0 con 0x55db3d73f5f0 2024-03-23T23:03:17.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.632+0000 7ffb85200640 1 --2- 172.21.15.12:0/3871890606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55db3d744760 0x55db3d7adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:03:17.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.633+0000 7ffb7f400640 1 --2- 172.21.15.12:0/3871890606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d73f5f0 0x55db3d74d750 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x55db3d740060 tx=0x7ffb6c011b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97e1df2dfc8174a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:17.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.633+0000 7ffb6a000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:03:17.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.633+0000 7ffb6a000640 1 -- 172.21.15.12:0/3871890606 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6c010ae0 con 0x55db3d73f5f0 2024-03-23T23:03:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.633+0000 7ffb6a000640 1 -- 172.21.15.12:0/3871890606 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ffb6c010c80 con 0x55db3d73f5f0 2024-03-23T23:03:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.633+0000 7ffb89e0d740 1 -- 172.21.15.12:0/3871890606 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb78004000 con 0x55db3d73f5f0 2024-03-23T23:03:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.633+0000 7ffb6a000640 1 -- 172.21.15.12:0/3871890606 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb6c010e80 con 0x55db3d73f5f0 2024-03-23T23:03:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.633+0000 7ffb89e0d740 10 client.25651 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.633+0000 7ffb89e0d740 1 -- 172.21.15.12:0/3871890606 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55db3d7af8d0 con 0x55db3d73f5f0 2024-03-23T23:03:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.634+0000 7ffb6a000640 1 -- 172.21.15.12:0/3871890606 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb78004000 con 0x55db3d73f5f0 2024-03-23T23:03:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.634+0000 7ffb6a000640 1 -- 172.21.15.12:0/3871890606 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ffb6c00e540 con 0x55db3d73f5f0 2024-03-23T23:03:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.634+0000 7ffb89e0d740 10 client.25651 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.634+0000 7ffb89e0d740 10 client.25651 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.634+0000 7ffb89e0d740 20 client.25651 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.634+0000 7ffb89e0d740 1 --2- 172.21.15.12:0/3871890606 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55db3d7b02e0 0x55db3d7b2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.634+0000 7ffb89e0d740 4 client.25651 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:03:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.634+0000 7ffb89e0d740 1 -- 172.21.15.12:0/3871890606 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55db3d6a30c0 con 0x55db3d7b02e0 2024-03-23T23:03:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.634+0000 7ffb7fe00640 1 --2- 172.21.15.12:0/3871890606 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55db3d7b02e0 0x55db3d7b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.635+0000 7ffb7fe00640 1 --2- 172.21.15.12:0/3871890606 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55db3d7b02e0 0x55db3d7b2700 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba2e682610b85926 server_cookie=21cc563d5532ca99 in_seq=0 out_seq=0 2024-03-23T23:03:17.637 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.635+0000 7ffb6a000640 10 client.25651 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:17.639 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.640+0000 7ffb6a000640 1 -- 172.21.15.12:0/3871890606 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55db3d6a30c0 con 0x55db3d7b02e0 2024-03-23T23:03:17.639 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.640+0000 7ffb6a000640 10 client.25651 handle_command_reply: tid=0 2024-03-23T23:03:17.640 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.640+0000 7ffb89e0d740 1 client.25651 shutdown 2024-03-23T23:03:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.641+0000 7ffb89e0d740 1 -- 172.21.15.12:0/3871890606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d73f5f0 msgr2=0x55db3d74d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.641+0000 7ffb89e0d740 1 --2- 172.21.15.12:0/3871890606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d73f5f0 0x55db3d74d750 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x55db3d740060 tx=0x7ffb6c011b50 comp rx=0 tx=0).stop 2024-03-23T23:03:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.641+0000 7ffb89e0d740 1 -- 172.21.15.12:0/3871890606 shutdown_connections 2024-03-23T23:03:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.641+0000 7ffb89e0d740 1 --2- 172.21.15.12:0/3871890606 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55db3d7b02e0 0x55db3d7b2700 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.641+0000 7ffb89e0d740 1 --2- 172.21.15.12:0/3871890606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55db3d744760 0x55db3d7adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.641+0000 7ffb89e0d740 1 --2- 172.21.15.12:0/3871890606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db3d73f5f0 0x55db3d74d750 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.641+0000 7ffb89e0d740 1 --2- 172.21.15.12:0/3871890606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55db3d73ec00 0x55db3d74d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.641+0000 7ffb89e0d740 1 -- 172.21.15.12:0/3871890606 >> 172.21.15.12:0/3871890606 conn(0x55db3d73a170 msgr2=0x55db3d73d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.641+0000 7ffb6a000640 0 client.25651 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.641+0000 7ffb89e0d740 1 -- 172.21.15.12:0/3871890606 shutdown_connections 2024-03-23T23:03:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.641+0000 7ffb89e0d740 1 -- 172.21.15.12:0/3871890606 wait complete. 2024-03-23T23:03:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.641+0000 7ffb89e0d740 20 client.25651 trim_cache size 0 max 16384 2024-03-23T23:03:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:03:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:03:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:03:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:03:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.642+0000 7ffb85c00640 1 -- 172.21.15.12:0/2018525732 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ffb640a28f0 msgr2=0x7ffb640a4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.642+0000 7ffb85c00640 1 --2- 172.21.15.12:0/2018525732 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ffb640a28f0 0x7ffb640a4db0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ffb70005ab0 tx=0x7ffb7002cf90 comp rx=0 tx=0).stop 2024-03-23T23:03:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.642+0000 7ffb85c00640 1 -- 172.21.15.12:0/2018525732 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb801052f0 msgr2=0x7ffb801a1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.642+0000 7ffb85c00640 1 --2- 172.21.15.12:0/2018525732 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb801052f0 0x7ffb801a1030 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ffb7800da10 tx=0x7ffb7800ded0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.642+0000 7ffb85c00640 1 -- 172.21.15.12:0/2018525732 shutdown_connections 2024-03-23T23:03:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.642+0000 7ffb85c00640 1 --2- 172.21.15.12:0/2018525732 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ffb640a28f0 0x7ffb640a4db0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.642+0000 7ffb85c00640 1 --2- 172.21.15.12:0/2018525732 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb801052f0 0x7ffb801a1030 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.642+0000 7ffb85c00640 1 --2- 172.21.15.12:0/2018525732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb80104930 0x7ffb80199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.642+0000 7ffb85c00640 1 --2- 172.21.15.12:0/2018525732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb80103740 0x7ffb80199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.643+0000 7ffb85c00640 1 -- 172.21.15.12:0/2018525732 >> 172.21.15.12:0/2018525732 conn(0x7ffb800fecb0 msgr2=0x7ffb80107c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.643+0000 7ffb85c00640 1 -- 172.21.15.12:0/2018525732 shutdown_connections 2024-03-23T23:03:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:17.643+0000 7ffb85c00640 1 -- 172.21.15.12:0/2018525732 wait complete. 2024-03-23T23:03:18.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:18 smithi012 ceph-mon[31695]: pgmap v435: 97 pgs: 97 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:03:18.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:18 smithi173 ceph-mon[36116]: pgmap v435: 97 pgs: 97 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:03:18.910 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:18 smithi203 ceph-mon[36693]: pgmap v435: 97 pgs: 97 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:03:20.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:20 smithi012 ceph-mon[31695]: pgmap v436: 97 pgs: 97 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-23T23:03:20.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:20 smithi173 ceph-mon[36116]: pgmap v436: 97 pgs: 97 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-23T23:03:20.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:20 smithi203 ceph-mon[36693]: pgmap v436: 97 pgs: 97 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-23T23:03:22.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:22 smithi012 ceph-mon[31695]: pgmap v437: 97 pgs: 97 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-23T23:03:22.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:22 smithi173 ceph-mon[36116]: pgmap v437: 97 pgs: 97 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-23T23:03:22.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:22 smithi203 ceph-mon[36693]: pgmap v437: 97 pgs: 97 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-23T23:03:24.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:24 smithi012 ceph-mon[31695]: pgmap v438: 97 pgs: 97 active+clean; 666 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-23T23:03:24.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:24 smithi173 ceph-mon[36116]: pgmap v438: 97 pgs: 97 active+clean; 666 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-23T23:03:24.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:24 smithi203 ceph-mon[36693]: pgmap v438: 97 pgs: 97 active+clean; 666 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-23T23:03:26.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:26 smithi012 ceph-mon[31695]: pgmap v439: 97 pgs: 97 active+clean; 666 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-23T23:03:26.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:26 smithi173 ceph-mon[36116]: pgmap v439: 97 pgs: 97 active+clean; 666 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-23T23:03:26.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:26 smithi203 ceph-mon[36693]: pgmap v439: 97 pgs: 97 active+clean; 666 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-23T23:03:28.388 INFO:tasks.workunit.client.0.smithi012.stderr:Updating files: 59% (7244/12271) Updating files: 60% (7363/12271) Updating files: 61% (7486/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 98% (12029/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-23T23:03:28.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:28 smithi012 ceph-mon[31695]: pgmap v440: 97 pgs: 97 active+clean; 666 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-23T23:03:28.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:28 smithi173 ceph-mon[36116]: pgmap v440: 97 pgs: 97 active+clean; 666 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-23T23:03:28.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:28 smithi203 ceph-mon[36693]: pgmap v440: 97 pgs: 97 active+clean; 666 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-23T23:03:29.031 INFO:tasks.workunit.client.0.smithi012.stderr:Note: switching to '3be9a5c9bc793e11e2800a8c0c696e8b46742033'. 2024-03-23T23:03:29.032 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-03-23T23:03:29.032 INFO:tasks.workunit.client.0.smithi012.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-23T23:03:29.032 INFO:tasks.workunit.client.0.smithi012.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-23T23:03:29.032 INFO:tasks.workunit.client.0.smithi012.stderr:state without impacting any branches by switching back to a branch. 2024-03-23T23:03:29.032 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-03-23T23:03:29.032 INFO:tasks.workunit.client.0.smithi012.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-23T23:03:29.032 INFO:tasks.workunit.client.0.smithi012.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-23T23:03:29.032 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-03-23T23:03:29.032 INFO:tasks.workunit.client.0.smithi012.stderr: git switch -c 2024-03-23T23:03:29.032 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-03-23T23:03:29.032 INFO:tasks.workunit.client.0.smithi012.stderr:Or undo this operation with: 2024-03-23T23:03:29.032 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-03-23T23:03:29.032 INFO:tasks.workunit.client.0.smithi012.stderr: git switch - 2024-03-23T23:03:29.033 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-03-23T23:03:29.033 INFO:tasks.workunit.client.0.smithi012.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-23T23:03:29.033 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-03-23T23:03:29.033 INFO:tasks.workunit.client.0.smithi012.stderr:HEAD is now at 3be9a5c9bc7 dummy commit #1 2024-03-23T23:03:29.042 DEBUG:teuthology.orchestra.run.smithi012:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-23T23:03:29.105 INFO:tasks.workunit.client.0.smithi012.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-23T23:03:29.112 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-23T23:03:29.112 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-23T23:03:30.020 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-23T23:03:30.182 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-23T23:03:30.272 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-23T23:03:30.278 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-23T23:03:30.278 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-23T23:03:30.366 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-23T23:03:30.377 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-23T23:03:30.377 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-23T23:03:30.432 INFO:tasks.workunit:Running workunits matching fs/test_o_trunc.sh on client.0... 2024-03-23T23:03:30.434 INFO:tasks.workunit:Running workunit fs/test_o_trunc.sh... 2024-03-23T23:03:30.434 DEBUG:teuthology.orchestra.run.smithi012:workunit test fs/test_o_trunc.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_o_trunc.sh 2024-03-23T23:03:30.510 INFO:tasks.workunit.client.0.smithi012.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_o_trunc.sh 2024-03-23T23:03:30.513 INFO:tasks.workunit.client.0.smithi012.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs 2024-03-23T23:03:30.513 INFO:tasks.workunit.client.0.smithi012.stderr:+ /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_o_trunc trunc.foo 600 2024-03-23T23:03:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.793+0000 7f3a55000640 1 -- 172.21.15.12:0/783158554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a500756e0 msgr2=0x7f3a50073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.793+0000 7f3a55000640 1 --2- 172.21.15.12:0/783158554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a500756e0 0x7f3a50073b30 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3a40009900 tx=0x7f3a4002bc40 comp rx=0 tx=0).stop 2024-03-23T23:03:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.793+0000 7f3a55000640 1 -- 172.21.15.12:0/783158554 shutdown_connections 2024-03-23T23:03:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.793+0000 7f3a55000640 1 --2- 172.21.15.12:0/783158554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a501093d0 0x7f3a5010b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.793+0000 7f3a55000640 1 --2- 172.21.15.12:0/783158554 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a50074100 0x7f3a50074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.793+0000 7f3a55000640 1 --2- 172.21.15.12:0/783158554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a500756e0 0x7f3a50073b30 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.793+0000 7f3a55000640 1 -- 172.21.15.12:0/783158554 >> 172.21.15.12:0/783158554 conn(0x7f3a500fca80 msgr2=0x7f3a500feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.793+0000 7f3a55000640 1 -- 172.21.15.12:0/783158554 shutdown_connections 2024-03-23T23:03:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.793+0000 7f3a55000640 1 -- 172.21.15.12:0/783158554 wait complete. 2024-03-23T23:03:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.795+0000 7f3a55000640 1 Processor -- start 2024-03-23T23:03:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.795+0000 7f3a55000640 1 -- start start 2024-03-23T23:03:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.795+0000 7f3a55000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a50074100 0x7f3a501999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.795+0000 7f3a55000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a500756e0 0x7f3a50199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.795+0000 7f3a55000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a501093d0 0x7f3a501a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.795+0000 7f3a55000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5010f3a0 con 0x7f3a500756e0 2024-03-23T23:03:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.795+0000 7f3a55000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5010f220 con 0x7f3a50074100 2024-03-23T23:03:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.795+0000 7f3a55000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5010f520 con 0x7f3a501093d0 2024-03-23T23:03:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.796+0000 7f3a4f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a50074100 0x7f3a501999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.796+0000 7f3a4f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a50074100 0x7f3a501999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:52654/0 (socket says 172.21.15.12:52654) 2024-03-23T23:03:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.796+0000 7f3a4f400640 1 -- 172.21.15.12:0/3703375136 learned_addr learned my addr 172.21.15.12:0/3703375136 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.796+0000 7f3a4ea00640 1 --2- 172.21.15.12:0/3703375136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a500756e0 0x7f3a50199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.796+0000 7f3a4fe00640 1 --2- 172.21.15.12:0/3703375136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a501093d0 0x7f3a501a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.796+0000 7f3a4f400640 1 -- 172.21.15.12:0/3703375136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a501093d0 msgr2=0x7f3a501a0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.796+0000 7f3a4f400640 1 --2- 172.21.15.12:0/3703375136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a501093d0 0x7f3a501a0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.796+0000 7f3a4f400640 1 -- 172.21.15.12:0/3703375136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a500756e0 msgr2=0x7f3a50199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.796+0000 7f3a4f400640 1 --2- 172.21.15.12:0/3703375136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a500756e0 0x7f3a50199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.796+0000 7f3a4f400640 1 -- 172.21.15.12:0/3703375136 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a501a1610 con 0x7f3a50074100 2024-03-23T23:03:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.796+0000 7f3a4ea00640 1 --2- 172.21.15.12:0/3703375136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a500756e0 0x7f3a50199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:03:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.796+0000 7f3a4f400640 1 --2- 172.21.15.12:0/3703375136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a50074100 0x7f3a501999f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3a4002c8f0 tx=0x7f3a40005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.797+0000 7f3a47e00640 1 -- 172.21.15.12:0/3703375136 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a40041070 con 0x7f3a50074100 2024-03-23T23:03:30.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.797+0000 7f3a47e00640 1 -- 172.21.15.12:0/3703375136 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a400044d0 con 0x7f3a50074100 2024-03-23T23:03:30.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.797+0000 7f3a47e00640 1 -- 172.21.15.12:0/3703375136 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a40034440 con 0x7f3a50074100 2024-03-23T23:03:30.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.797+0000 7f3a55000640 1 -- 172.21.15.12:0/3703375136 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a501a18a0 con 0x7f3a50074100 2024-03-23T23:03:30.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.797+0000 7f3a55000640 1 -- 172.21.15.12:0/3703375136 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a501a1d90 con 0x7f3a50074100 2024-03-23T23:03:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.800+0000 7f3a47e00640 1 -- 172.21.15.12:0/3703375136 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3a40044070 con 0x7f3a50074100 2024-03-23T23:03:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.800+0000 7f3a59393740 1 Processor -- start 2024-03-23T23:03:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.800+0000 7f3a59393740 1 -- start start 2024-03-23T23:03:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.801+0000 7f3a59393740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ecad99a3f0 0x55ecad99a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.801+0000 7f3a59393740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ecad99ade0 0x55ecad99f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.801+0000 7f3a59393740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ecad99ff50 0x55ecad9a23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.801+0000 7f3a59393740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecad95d3b0 con 0x55ecad99ade0 2024-03-23T23:03:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.801+0000 7f3a59393740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecad8ff830 con 0x55ecad99a3f0 2024-03-23T23:03:30.803 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.801+0000 7f3a59393740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecad8a85d0 con 0x55ecad99ff50 2024-03-23T23:03:30.803 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.801+0000 7f3a4fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ecad99a3f0 0x55ecad99a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:30.803 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.801+0000 7f3a4fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ecad99a3f0 0x55ecad99a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:52670/0 (socket says 172.21.15.12:52670) 2024-03-23T23:03:30.803 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.801+0000 7f3a4fe00640 1 -- 172.21.15.12:0/267763565 learned_addr learned my addr 172.21.15.12:0/267763565 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:30.803 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.801+0000 7f3a4f400640 1 --2- 172.21.15.12:0/267763565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ecad99ade0 0x55ecad99f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:30.803 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.801+0000 7f3a4ea00640 1 --2- 172.21.15.12:0/267763565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ecad99ff50 0x55ecad9a23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:30.803 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.801+0000 7f3a4fe00640 1 -- 172.21.15.12:0/267763565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ecad99ff50 msgr2=0x55ecad9a23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:30.803 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.801+0000 7f3a4fe00640 1 --2- 172.21.15.12:0/267763565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ecad99ff50 0x55ecad9a23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.803 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.801+0000 7f3a4fe00640 1 -- 172.21.15.12:0/267763565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ecad99ade0 msgr2=0x55ecad99f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:30.804 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.801+0000 7f3a4fe00640 1 --2- 172.21.15.12:0/267763565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ecad99ade0 0x55ecad99f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.804 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.801+0000 7f3a4fe00640 1 -- 172.21.15.12:0/267763565 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ecad8c02c0 con 0x55ecad99a3f0 2024-03-23T23:03:30.804 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.801+0000 7f3a4ea00640 1 --2- 172.21.15.12:0/267763565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ecad99ff50 0x55ecad9a23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:30.804 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.802+0000 7f3a47e00640 1 --2- 172.21.15.12:0/3703375136 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3a340a27f0 0x7f3a340a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:30.804 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.802+0000 7f3a4fe00640 1 --2- 172.21.15.12:0/267763565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ecad99a3f0 0x55ecad99a810 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3a48009f50 tx=0x7f3a480118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee0d170002916c51 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:30.804 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.802+0000 7f3a47e00640 1 -- 172.21.15.12:0/3703375136 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f3a400f06f0 con 0x7f3a50074100 2024-03-23T23:03:30.804 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.802+0000 7f3a4f400640 1 --2- 172.21.15.12:0/3703375136 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3a340a27f0 0x7f3a340a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:30.804 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.802+0000 7f3a44c00640 1 -- 172.21.15.12:0/267763565 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a48004d20 con 0x55ecad99a3f0 2024-03-23T23:03:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.802+0000 7f3a44c00640 1 -- 172.21.15.12:0/267763565 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a48004050 con 0x55ecad99a3f0 2024-03-23T23:03:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.802+0000 7f3a44c00640 1 -- 172.21.15.12:0/267763565 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a48005280 con 0x55ecad99a3f0 2024-03-23T23:03:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.803+0000 7f3a59393740 1 -- 172.21.15.12:0/267763565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ecad99a3f0 msgr2=0x55ecad99a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.803+0000 7f3a4f400640 1 --2- 172.21.15.12:0/3703375136 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3a340a27f0 0x7f3a340a4cb0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3a400bcfd0 tx=0x7f3a400dd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.803+0000 7f3a59393740 1 --2- 172.21.15.12:0/267763565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ecad99a3f0 0x55ecad99a810 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3a48009f50 tx=0x7f3a480118d0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.803+0000 7f3a59393740 1 -- 172.21.15.12:0/267763565 shutdown_connections 2024-03-23T23:03:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.803+0000 7f3a59393740 1 --2- 172.21.15.12:0/267763565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ecad99ff50 0x55ecad9a23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.803+0000 7f3a59393740 1 --2- 172.21.15.12:0/267763565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ecad99ade0 0x55ecad99f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.803+0000 7f3a59393740 1 --2- 172.21.15.12:0/267763565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ecad99a3f0 0x55ecad99a810 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.803+0000 7f3a59393740 1 -- 172.21.15.12:0/267763565 >> 172.21.15.12:0/267763565 conn(0x55ecad995960 msgr2=0x55ecad995d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.803+0000 7f3a59393740 1 -- 172.21.15.12:0/267763565 shutdown_connections 2024-03-23T23:03:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.803+0000 7f3a59393740 1 -- 172.21.15.12:0/267763565 wait complete. 2024-03-23T23:03:30.807 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:30 smithi012 ceph-mon[31695]: pgmap v441: 97 pgs: 97 active+clean; 666 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-23T23:03:30.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.803+0000 7f3a59393740 1 -- start start 2024-03-23T23:03:30.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.804+0000 7f3a59393740 1 Processor -- start 2024-03-23T23:03:30.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.804+0000 7f3a59393740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ecad99a3f0 0x55ecad9a45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:30.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.804+0000 7f3a59393740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ecad99ade0 0x55ecad9a4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.804+0000 7f3a59393740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ecad99ff50 0x55ecad9abb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.804+0000 7f3a59393740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecad8bba20 con 0x55ecad99a3f0 2024-03-23T23:03:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.804+0000 7f3a59393740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecad8fb8d0 con 0x55ecad99ff50 2024-03-23T23:03:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.804+0000 7f3a59393740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecad9a2f00 con 0x55ecad99ade0 2024-03-23T23:03:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.804+0000 7f3a59393740 10 client.0 _ll_register_callbacks cb 0x55ecad8f81c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.805+0000 7f3a4fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ecad99a3f0 0x55ecad9a45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.805+0000 7f3a4ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ecad99ade0 0x55ecad9a4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.805+0000 7f3a4ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ecad99ade0 0x55ecad9a4b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:40884/0 (socket says 172.21.15.12:40884) 2024-03-23T23:03:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.805+0000 7f3a4ea00640 1 -- 172.21.15.12:0/2727913667 learned_addr learned my addr 172.21.15.12:0/2727913667 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.805+0000 7f3a4fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ecad99a3f0 0x55ecad9a45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37456/0 (socket says 172.21.15.12:37456) 2024-03-23T23:03:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.805+0000 7f3a4fe00640 1 --2- 172.21.15.12:0/2727913667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ecad99ff50 0x55ecad9abb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.805+0000 7f3a4ea00640 1 -- 172.21.15.12:0/2727913667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ecad99ff50 msgr2=0x55ecad9abb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.805+0000 7f3a4ea00640 1 --2- 172.21.15.12:0/2727913667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ecad99ff50 0x55ecad9abb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.805+0000 7f3a4ea00640 1 -- 172.21.15.12:0/2727913667 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ecad99a3f0 msgr2=0x55ecad9a45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.805+0000 7f3a4ea00640 1 --2- 172.21.15.12:0/2727913667 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ecad99a3f0 0x55ecad9a45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.805+0000 7f3a4ea00640 1 -- 172.21.15.12:0/2727913667 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ecad614210 con 0x55ecad99ade0 2024-03-23T23:03:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.805+0000 7f3a4ea00640 1 --2- 172.21.15.12:0/2727913667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ecad99ade0 0x55ecad9a4b10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c00bf70 tx=0x7f3a3c0128e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62dfdf65a4093181 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.805+0000 7f3a39600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:03:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.806+0000 7f3a39600640 1 -- 172.21.15.12:0/2727913667 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c004230 con 0x55ecad99ade0 2024-03-23T23:03:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.806+0000 7f3a39600640 1 -- 172.21.15.12:0/2727913667 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f3a3c0043d0 con 0x55ecad99ade0 2024-03-23T23:03:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.806+0000 7f3a59393740 1 -- 172.21.15.12:0/2727913667 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3a48004050 con 0x55ecad99ade0 2024-03-23T23:03:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.806+0000 7f3a39600640 1 -- 172.21.15.12:0/2727913667 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a3c0106a0 con 0x55ecad99ade0 2024-03-23T23:03:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.806+0000 7f3a59393740 10 client.25663 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.806+0000 7f3a59393740 1 -- 172.21.15.12:0/2727913667 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55ecad6e0620 con 0x55ecad99ade0 2024-03-23T23:03:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.806+0000 7f3a39600640 1 -- 172.21.15.12:0/2727913667 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a48004050 con 0x55ecad99ade0 2024-03-23T23:03:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.806+0000 7f3a39600640 1 -- 172.21.15.12:0/2727913667 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f3a3c01b030 con 0x55ecad99ade0 2024-03-23T23:03:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.806+0000 7f3a59393740 10 client.25663 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.806+0000 7f3a59393740 10 client.25663 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.807+0000 7f3a59393740 20 client.25663 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.807+0000 7f3a59393740 1 --2- 172.21.15.12:0/2727913667 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55ecad9b0440 0x55ecad9b2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:30.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.807+0000 7f3a59393740 4 client.25663 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:03:30.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.807+0000 7f3a59393740 1 -- 172.21.15.12:0/2727913667 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ecad90cd70 con 0x55ecad9b0440 2024-03-23T23:03:30.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.807+0000 7f3a4fe00640 1 --2- 172.21.15.12:0/2727913667 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55ecad9b0440 0x55ecad9b2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:30.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.807+0000 7f3a4fe00640 1 --2- 172.21.15.12:0/2727913667 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55ecad9b0440 0x55ecad9b2850 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b81547cf99e8ba7e server_cookie=70028300a9c4dac8 in_seq=0 out_seq=0 2024-03-23T23:03:30.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.807+0000 7f3a39600640 10 client.25663 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:30.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.808+0000 7f3a39600640 1 -- 172.21.15.12:0/2727913667 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ecad90cd70 con 0x55ecad9b0440 2024-03-23T23:03:30.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.808+0000 7f3a39600640 10 client.25663 handle_command_reply: tid=0 2024-03-23T23:03:30.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.808+0000 7f3a59393740 1 client.25663 shutdown 2024-03-23T23:03:30.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.808+0000 7f3a59393740 1 -- 172.21.15.12:0/2727913667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ecad99ade0 msgr2=0x55ecad9a4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:30.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.808+0000 7f3a59393740 1 --2- 172.21.15.12:0/2727913667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ecad99ade0 0x55ecad9a4b10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c00bf70 tx=0x7f3a3c0128e0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.808+0000 7f3a59393740 1 -- 172.21.15.12:0/2727913667 shutdown_connections 2024-03-23T23:03:30.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.808+0000 7f3a59393740 1 --2- 172.21.15.12:0/2727913667 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55ecad9b0440 0x55ecad9b2850 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.808+0000 7f3a59393740 1 --2- 172.21.15.12:0/2727913667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ecad99ff50 0x55ecad9abb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.808+0000 7f3a59393740 1 --2- 172.21.15.12:0/2727913667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ecad99ade0 0x55ecad9a4b10 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.808+0000 7f3a39600640 0 client.25663 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:30.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.808+0000 7f3a59393740 1 --2- 172.21.15.12:0/2727913667 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ecad99a3f0 0x55ecad9a45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.808+0000 7f3a59393740 1 -- 172.21.15.12:0/2727913667 >> 172.21.15.12:0/2727913667 conn(0x55ecad995960 msgr2=0x55ecad994b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:30.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.809+0000 7f3a59393740 1 -- 172.21.15.12:0/2727913667 shutdown_connections 2024-03-23T23:03:30.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.809+0000 7f3a59393740 1 -- 172.21.15.12:0/2727913667 wait complete. 2024-03-23T23:03:30.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.809+0000 7f3a59393740 20 client.25663 trim_cache size 0 max 16384 2024-03-23T23:03:30.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.824+0000 7f3a59393740 1 Processor -- start 2024-03-23T23:03:30.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.824+0000 7f3a59393740 1 -- start start 2024-03-23T23:03:30.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.824+0000 7f3a59393740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ecad99a3f0 0x55ecad9b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:30.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.824+0000 7f3a59393740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ecad99ade0 0x55ecad9abb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:30.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.824+0000 7f3a59393740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ecad99ff50 0x55ecad9a38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:30.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.824+0000 7f3a59393740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a501acf50 con 0x55ecad99ff50 2024-03-23T23:03:30.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.824+0000 7f3a59393740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a501aaa50 con 0x55ecad99ade0 2024-03-23T23:03:30.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.824+0000 7f3a59393740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a501a8af0 con 0x55ecad99a3f0 2024-03-23T23:03:30.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.825+0000 7f3a4ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ecad99ade0 0x55ecad9abb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:30.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.825+0000 7f3a4fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ecad99a3f0 0x55ecad9b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:30.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.825+0000 7f3a4ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ecad99ade0 0x55ecad9abb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:52692/0 (socket says 172.21.15.12:52692) 2024-03-23T23:03:30.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.825+0000 7f3a4ea00640 1 -- 172.21.15.12:0/14932785 learned_addr learned my addr 172.21.15.12:0/14932785 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:30.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.825+0000 7f3a4fe00640 1 --2- 172.21.15.12:0/14932785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ecad99ff50 0x55ecad9a38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:30.826 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.825+0000 7f3a4ea00640 1 -- 172.21.15.12:0/14932785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ecad99a3f0 msgr2=0x55ecad9b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:30.826 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.825+0000 7f3a4ea00640 1 --2- 172.21.15.12:0/14932785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ecad99a3f0 0x55ecad9b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.826 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.825+0000 7f3a4ea00640 1 -- 172.21.15.12:0/14932785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ecad99ff50 msgr2=0x55ecad9a38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:30.826 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.825+0000 7f3a4ea00640 1 --2- 172.21.15.12:0/14932785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ecad99ff50 0x55ecad9a38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.826 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.825+0000 7f3a4ea00640 1 -- 172.21.15.12:0/14932785 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ecad99a7d0 con 0x55ecad99ade0 2024-03-23T23:03:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.826+0000 7f3a4ea00640 1 --2- 172.21.15.12:0/14932785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ecad99ade0 0x55ecad9abb10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c000c00 tx=0x7f3a3c013e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=553b5d9a84107c2f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.826+0000 7f3a3be00640 1 -- 172.21.15.12:0/14932785 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c0037f0 con 0x55ecad99ade0 2024-03-23T23:03:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.826+0000 7f3a3be00640 1 -- 172.21.15.12:0/14932785 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a3c0055b0 con 0x55ecad99ade0 2024-03-23T23:03:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.826+0000 7f3a3be00640 1 -- 172.21.15.12:0/14932785 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c030480 con 0x55ecad99ade0 2024-03-23T23:03:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.827+0000 7f3a59393740 1 -- 172.21.15.12:0/14932785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ecad99ade0 msgr2=0x55ecad9abb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.827+0000 7f3a59393740 1 --2- 172.21.15.12:0/14932785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ecad99ade0 0x55ecad9abb10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c000c00 tx=0x7f3a3c013e70 comp rx=0 tx=0).stop 2024-03-23T23:03:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.827+0000 7f3a59393740 1 -- 172.21.15.12:0/14932785 shutdown_connections 2024-03-23T23:03:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.827+0000 7f3a59393740 1 --2- 172.21.15.12:0/14932785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ecad99ff50 0x55ecad9a38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.827+0000 7f3a59393740 1 --2- 172.21.15.12:0/14932785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ecad99ade0 0x55ecad9abb10 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.827+0000 7f3a59393740 1 --2- 172.21.15.12:0/14932785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ecad99a3f0 0x55ecad9b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.827+0000 7f3a59393740 1 -- 172.21.15.12:0/14932785 >> 172.21.15.12:0/14932785 conn(0x55ecad995960 msgr2=0x55ecad995d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.827+0000 7f3a59393740 1 -- 172.21.15.12:0/14932785 shutdown_connections 2024-03-23T23:03:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.827+0000 7f3a59393740 1 -- 172.21.15.12:0/14932785 wait complete. 2024-03-23T23:03:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.828+0000 7f3a59393740 1 -- start start 2024-03-23T23:03:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.828+0000 7f3a59393740 1 Processor -- start 2024-03-23T23:03:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.828+0000 7f3a59393740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ecad99a3f0 0x55ecad9a9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.829+0000 7f3a59393740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ecad99ade0 0x55ecad9a9870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.829+0000 7f3a59393740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ecad99ff50 0x55ecad9aa880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.829+0000 7f3a59393740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a501a90f0 con 0x55ecad99a3f0 2024-03-23T23:03:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.829+0000 7f3a59393740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a501aa3f0 con 0x55ecad99ade0 2024-03-23T23:03:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.829+0000 7f3a59393740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a501a9730 con 0x55ecad99ff50 2024-03-23T23:03:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.829+0000 7f3a59393740 10 client.0 _ll_register_callbacks cb 0x7f3a3c00b800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.829+0000 7f3a4fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ecad99a3f0 0x55ecad9a9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.829+0000 7f3a4fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ecad99a3f0 0x55ecad9a9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37486/0 (socket says 172.21.15.12:37486) 2024-03-23T23:03:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.829+0000 7f3a4fe00640 1 -- 172.21.15.12:0/521236491 learned_addr learned my addr 172.21.15.12:0/521236491 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.829+0000 7f3a4fe00640 1 --2- 172.21.15.12:0/521236491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ecad99ff50 0x55ecad9aa880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.829+0000 7f3a4ea00640 1 --2- 172.21.15.12:0/521236491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ecad99ade0 0x55ecad9a9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.829+0000 7f3a4fe00640 1 -- 172.21.15.12:0/521236491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ecad99ff50 msgr2=0x55ecad9aa880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.829+0000 7f3a4fe00640 1 --2- 172.21.15.12:0/521236491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ecad99ff50 0x55ecad9aa880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.829+0000 7f3a4fe00640 1 -- 172.21.15.12:0/521236491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ecad99ade0 msgr2=0x55ecad9a9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.829+0000 7f3a4fe00640 1 --2- 172.21.15.12:0/521236491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ecad99ade0 0x55ecad9a9870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.829+0000 7f3a4fe00640 1 -- 172.21.15.12:0/521236491 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ecad9aae70 con 0x55ecad99a3f0 2024-03-23T23:03:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.829+0000 7f3a4fe00640 1 --2- 172.21.15.12:0/521236491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ecad99a3f0 0x55ecad9a9330 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3a48006010 tx=0x7f3a4802f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0b5214f246ac677 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.829+0000 7f3a39600640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:03:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.830+0000 7f3a39600640 1 -- 172.21.15.12:0/521236491 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a48003de0 con 0x55ecad99a3f0 2024-03-23T23:03:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.830+0000 7f3a39600640 1 -- 172.21.15.12:0/521236491 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f3a48004000 con 0x55ecad99a3f0 2024-03-23T23:03:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.830+0000 7f3a59393740 1 -- 172.21.15.12:0/521236491 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3a3c0055b0 con 0x55ecad99a3f0 2024-03-23T23:03:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.830+0000 7f3a39600640 1 -- 172.21.15.12:0/521236491 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a48004200 con 0x55ecad99a3f0 2024-03-23T23:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.830+0000 7f3a59393740 10 client.15540 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.830+0000 7f3a59393740 1 -- 172.21.15.12:0/521236491 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55ecada0be60 con 0x55ecad99a3f0 2024-03-23T23:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.830+0000 7f3a39600640 1 -- 172.21.15.12:0/521236491 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a3c0055b0 con 0x55ecad99a3f0 2024-03-23T23:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.831+0000 7f3a39600640 1 -- 172.21.15.12:0/521236491 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f3a48039070 con 0x55ecad99a3f0 2024-03-23T23:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.831+0000 7f3a59393740 10 client.15540 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.831+0000 7f3a59393740 10 client.15540 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.831+0000 7f3a59393740 20 client.15540 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.831+0000 7f3a59393740 1 --2- 172.21.15.12:0/521236491 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55ecada0c610 0x55ecada0ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.831+0000 7f3a59393740 4 client.15540 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.831+0000 7f3a59393740 1 -- 172.21.15.12:0/521236491 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ecad9070c0 con 0x55ecada0c610 2024-03-23T23:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.831+0000 7f3a4ea00640 1 --2- 172.21.15.12:0/521236491 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55ecada0c610 0x55ecada0ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.832+0000 7f3a4ea00640 1 --2- 172.21.15.12:0/521236491 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55ecada0c610 0x55ecada0ea90 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=866cf4309c75bcf3 server_cookie=3deeaea48a5dad86 in_seq=0 out_seq=0 2024-03-23T23:03:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.832+0000 7f3a39600640 10 client.15540 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.840+0000 7f3a39600640 1 -- 172.21.15.12:0/521236491 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ecad9070c0 con 0x55ecada0c610 2024-03-23T23:03:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.840+0000 7f3a39600640 10 client.15540 handle_command_reply: tid=0 2024-03-23T23:03:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.840+0000 7f3a59393740 1 client.15540 shutdown 2024-03-23T23:03:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.841+0000 7f3a59393740 1 -- 172.21.15.12:0/521236491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ecad99a3f0 msgr2=0x55ecad9a9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.841+0000 7f3a59393740 1 --2- 172.21.15.12:0/521236491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ecad99a3f0 0x55ecad9a9330 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3a48006010 tx=0x7f3a4802f040 comp rx=0 tx=0).stop 2024-03-23T23:03:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.841+0000 7f3a59393740 1 -- 172.21.15.12:0/521236491 shutdown_connections 2024-03-23T23:03:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.841+0000 7f3a59393740 1 --2- 172.21.15.12:0/521236491 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55ecada0c610 0x55ecada0ea90 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.841+0000 7f3a59393740 1 --2- 172.21.15.12:0/521236491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ecad99ff50 0x55ecad9aa880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.841+0000 7f3a59393740 1 --2- 172.21.15.12:0/521236491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ecad99ade0 0x55ecad9a9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.841+0000 7f3a59393740 1 --2- 172.21.15.12:0/521236491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ecad99a3f0 0x55ecad9a9330 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.841+0000 7f3a39600640 0 client.15540 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.841+0000 7f3a59393740 1 -- 172.21.15.12:0/521236491 >> 172.21.15.12:0/521236491 conn(0x55ecad995960 msgr2=0x55ecad9a38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.841+0000 7f3a59393740 1 -- 172.21.15.12:0/521236491 shutdown_connections 2024-03-23T23:03:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.841+0000 7f3a59393740 1 -- 172.21.15.12:0/521236491 wait complete. 2024-03-23T23:03:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.841+0000 7f3a59393740 20 client.15540 trim_cache size 0 max 16384 2024-03-23T23:03:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:03:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:03:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:03:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:03:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.842+0000 7f3a55000640 1 -- 172.21.15.12:0/3703375136 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3a340a27f0 msgr2=0x7f3a340a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.842+0000 7f3a55000640 1 --2- 172.21.15.12:0/3703375136 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3a340a27f0 0x7f3a340a4cb0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3a400bcfd0 tx=0x7f3a400dd040 comp rx=0 tx=0).stop 2024-03-23T23:03:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.842+0000 7f3a55000640 1 -- 172.21.15.12:0/3703375136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a50074100 msgr2=0x7f3a501999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.842+0000 7f3a55000640 1 --2- 172.21.15.12:0/3703375136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a50074100 0x7f3a501999f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3a4002c8f0 tx=0x7f3a40005d50 comp rx=0 tx=0).stop 2024-03-23T23:03:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.842+0000 7f3a55000640 1 -- 172.21.15.12:0/3703375136 shutdown_connections 2024-03-23T23:03:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.842+0000 7f3a55000640 1 --2- 172.21.15.12:0/3703375136 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3a340a27f0 0x7f3a340a4cb0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.842+0000 7f3a55000640 1 --2- 172.21.15.12:0/3703375136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a501093d0 0x7f3a501a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.842+0000 7f3a55000640 1 --2- 172.21.15.12:0/3703375136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a500756e0 0x7f3a50199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.842+0000 7f3a55000640 1 --2- 172.21.15.12:0/3703375136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a50074100 0x7f3a501999f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.842+0000 7f3a55000640 1 -- 172.21.15.12:0/3703375136 >> 172.21.15.12:0/3703375136 conn(0x7f3a500fca80 msgr2=0x7f3a501052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.842+0000 7f3a55000640 1 -- 172.21.15.12:0/3703375136 shutdown_connections 2024-03-23T23:03:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:30.842+0000 7f3a55000640 1 -- 172.21.15.12:0/3703375136 wait complete. 2024-03-23T23:03:30.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:30 smithi173 ceph-mon[36116]: pgmap v441: 97 pgs: 97 active+clean; 666 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-23T23:03:30.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:30 smithi203 ceph-mon[36693]: pgmap v441: 97 pgs: 97 active+clean; 666 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-23T23:03:32.806 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:32 smithi012 ceph-mon[31695]: pgmap v442: 97 pgs: 97 active+clean; 666 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-23T23:03:32.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:32 smithi173 ceph-mon[36116]: pgmap v442: 97 pgs: 97 active+clean; 666 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-23T23:03:32.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:32 smithi203 ceph-mon[36693]: pgmap v442: 97 pgs: 97 active+clean; 666 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-23T23:03:34.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:34 smithi012 ceph-mon[31695]: pgmap v443: 97 pgs: 97 active+clean; 698 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 4.3 KiB/s wr, 9 op/s 2024-03-23T23:03:34.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:34 smithi173 ceph-mon[36116]: pgmap v443: 97 pgs: 97 active+clean; 698 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 4.3 KiB/s wr, 9 op/s 2024-03-23T23:03:34.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:34 smithi203 ceph-mon[36693]: pgmap v443: 97 pgs: 97 active+clean; 698 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 4.3 KiB/s wr, 9 op/s 2024-03-23T23:03:35.252 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:03:35.253 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:03:35.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.485+0000 7f5567e00640 1 -- 172.21.15.12:0/2474167114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55681037c0 msgr2=0x7f5568103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:35.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.485+0000 7f5567e00640 1 --2- 172.21.15.12:0/2474167114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55681037c0 0x7f5568103be0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5558009900 tx=0x7f555802bc30 comp rx=0 tx=0).stop 2024-03-23T23:03:35.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.486+0000 7f5567e00640 1 -- 172.21.15.12:0/2474167114 shutdown_connections 2024-03-23T23:03:35.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.486+0000 7f5567e00640 1 --2- 172.21.15.12:0/2474167114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5568105370 0x7f556810bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.486+0000 7f5567e00640 1 --2- 172.21.15.12:0/2474167114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55681049b0 0x7f5568104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.486+0000 7f5567e00640 1 --2- 172.21.15.12:0/2474167114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55681037c0 0x7f5568103be0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.486+0000 7f5567e00640 1 -- 172.21.15.12:0/2474167114 >> 172.21.15.12:0/2474167114 conn(0x7f55680fed50 msgr2=0x7f5568101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:35.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.486+0000 7f5567e00640 1 -- 172.21.15.12:0/2474167114 shutdown_connections 2024-03-23T23:03:35.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.486+0000 7f5567e00640 1 -- 172.21.15.12:0/2474167114 wait complete. 2024-03-23T23:03:35.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.487+0000 7f5567e00640 1 Processor -- start 2024-03-23T23:03:35.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.487+0000 7f5567e00640 1 -- start start 2024-03-23T23:03:35.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.488+0000 7f5567e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55681037c0 0x7f5568199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:35.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.488+0000 7f5567e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55681049b0 0x7f556819a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:35.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.488+0000 7f5567e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5568105370 0x7f55681a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:35.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.488+0000 7f5567e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556810f480 con 0x7f5568105370 2024-03-23T23:03:35.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.488+0000 7f5567e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556810f300 con 0x7f55681037c0 2024-03-23T23:03:35.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.488+0000 7f5567e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556810f600 con 0x7f55681049b0 2024-03-23T23:03:35.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.488+0000 7f5566000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55681049b0 0x7f556819a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:35.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.488+0000 7f5566a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55681037c0 0x7f5568199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:35.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.488+0000 7f5566000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55681049b0 0x7f556819a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35260/0 (socket says 172.21.15.12:35260) 2024-03-23T23:03:35.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.488+0000 7f5566000640 1 -- 172.21.15.12:0/9117516 learned_addr learned my addr 172.21.15.12:0/9117516 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:35.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.488+0000 7f5567400640 1 --2- 172.21.15.12:0/9117516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5568105370 0x7f55681a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:35.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.489+0000 7f5566a00640 1 -- 172.21.15.12:0/9117516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55681049b0 msgr2=0x7f556819a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:35.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.489+0000 7f5566a00640 1 --2- 172.21.15.12:0/9117516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55681049b0 0x7f556819a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.489+0000 7f5566a00640 1 -- 172.21.15.12:0/9117516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5568105370 msgr2=0x7f55681a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:35.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.489+0000 7f5566a00640 1 --2- 172.21.15.12:0/9117516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5568105370 0x7f55681a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.489+0000 7f5566a00640 1 -- 172.21.15.12:0/9117516 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55681a1720 con 0x7f55681037c0 2024-03-23T23:03:35.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.489+0000 7f5566000640 1 --2- 172.21.15.12:0/9117516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55681049b0 0x7f556819a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:35.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.489+0000 7f5566a00640 1 --2- 172.21.15.12:0/9117516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55681037c0 0x7f5568199b00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5558009230 tx=0x7f555802c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:35.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.489+0000 7f555f400640 1 -- 172.21.15.12:0/9117516 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5558043020 con 0x7f55681037c0 2024-03-23T23:03:35.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.489+0000 7f555f400640 1 -- 172.21.15.12:0/9117516 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f55580054e0 con 0x7f55681037c0 2024-03-23T23:03:35.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.489+0000 7f5567e00640 1 -- 172.21.15.12:0/9117516 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55681a19b0 con 0x7f55681037c0 2024-03-23T23:03:35.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.489+0000 7f555f400640 1 -- 172.21.15.12:0/9117516 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5558003c60 con 0x7f55681037c0 2024-03-23T23:03:35.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.490+0000 7f5567e00640 1 -- 172.21.15.12:0/9117516 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55681a1ed0 con 0x7f55681037c0 2024-03-23T23:03:35.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.493+0000 7f555f400640 1 -- 172.21.15.12:0/9117516 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5558046020 con 0x7f55681037c0 2024-03-23T23:03:35.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.493+0000 7f557042d740 1 Processor -- start 2024-03-23T23:03:35.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.493+0000 7f557042d740 1 -- start start 2024-03-23T23:03:35.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.493+0000 7f557042d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587e50de050 0x5587e50de470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:35.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.493+0000 7f557042d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 0x5587e50e3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:35.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.493+0000 7f557042d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587e50e3bb0 0x5587e50e6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:35.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.493+0000 7f557042d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587e5043250 con 0x5587e50de050 2024-03-23T23:03:35.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.493+0000 7f557042d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587e4feba30 con 0x5587e50dea40 2024-03-23T23:03:35.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.493+0000 7f557042d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587e503f2f0 con 0x5587e50e3bb0 2024-03-23T23:03:35.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.494+0000 7f5566a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 0x5587e50e3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:35.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.494+0000 7f5567400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587e50de050 0x5587e50de470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:35.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.494+0000 7f5567400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587e50de050 0x5587e50de470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48402/0 (socket says 172.21.15.12:48402) 2024-03-23T23:03:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.494+0000 7f5566a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 0x5587e50e3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50392/0 (socket says 172.21.15.12:50392) 2024-03-23T23:03:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.494+0000 7f5567400640 1 -- 172.21.15.12:0/1589544094 learned_addr learned my addr 172.21.15.12:0/1589544094 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.494+0000 7f5566000640 1 --2- 172.21.15.12:0/1589544094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587e50e3bb0 0x5587e50e6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.494+0000 7f5566a00640 1 -- 172.21.15.12:0/1589544094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587e50e3bb0 msgr2=0x5587e50e6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.494+0000 7f5566a00640 1 --2- 172.21.15.12:0/1589544094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587e50e3bb0 0x5587e50e6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.494+0000 7f5566a00640 1 -- 172.21.15.12:0/1589544094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587e50de050 msgr2=0x5587e50de470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.494+0000 7f5566a00640 1 --2- 172.21.15.12:0/1589544094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587e50de050 0x5587e50de470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.494+0000 7f5566a00640 1 -- 172.21.15.12:0/1589544094 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5587e4fd6690 con 0x5587e50dea40 2024-03-23T23:03:35.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.494+0000 7f5566a00640 1 --2- 172.21.15.12:0/1589544094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 0x5587e50e3540 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f55580ee610 tx=0x7f55580f4bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f56c32674f17e6d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:35.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.494+0000 7f555f400640 1 --2- 172.21.15.12:0/9117516 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f554c0a2800 0x7f554c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:35.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.495+0000 7f5557400640 1 -- 172.21.15.12:0/1589544094 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55580b97b0 con 0x5587e50dea40 2024-03-23T23:03:35.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.495+0000 7f5567400640 1 --2- 172.21.15.12:0/9117516 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f554c0a2800 0x7f554c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:35.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.495+0000 7f5557400640 1 -- 172.21.15.12:0/1589544094 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5558102070 con 0x5587e50dea40 2024-03-23T23:03:35.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.495+0000 7f5557400640 1 -- 172.21.15.12:0/1589544094 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55580c4510 con 0x5587e50dea40 2024-03-23T23:03:35.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.495+0000 7f555f400640 1 -- 172.21.15.12:0/9117516 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f55580f0bc0 con 0x7f55681037c0 2024-03-23T23:03:35.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.495+0000 7f557042d740 1 -- 172.21.15.12:0/1589544094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 msgr2=0x5587e50e3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:35.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.496+0000 7f557042d740 1 --2- 172.21.15.12:0/1589544094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 0x5587e50e3540 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f55580ee610 tx=0x7f55580f4bf0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.496+0000 7f5567400640 1 --2- 172.21.15.12:0/9117516 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f554c0a2800 0x7f554c0a4cc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x5587e50df2a0 tx=0x7f55600098a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:35.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.496+0000 7f557042d740 1 -- 172.21.15.12:0/1589544094 shutdown_connections 2024-03-23T23:03:35.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.496+0000 7f557042d740 1 --2- 172.21.15.12:0/1589544094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587e50e3bb0 0x5587e50e6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.496+0000 7f557042d740 1 --2- 172.21.15.12:0/1589544094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 0x5587e50e3540 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.496+0000 7f557042d740 1 --2- 172.21.15.12:0/1589544094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587e50de050 0x5587e50de470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.496+0000 7f557042d740 1 -- 172.21.15.12:0/1589544094 >> 172.21.15.12:0/1589544094 conn(0x5587e50a6200 msgr2=0x5587e50d99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:35.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.496+0000 7f557042d740 1 -- 172.21.15.12:0/1589544094 shutdown_connections 2024-03-23T23:03:35.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.496+0000 7f557042d740 1 -- 172.21.15.12:0/1589544094 wait complete. 2024-03-23T23:03:35.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.496+0000 7f557042d740 1 -- start start 2024-03-23T23:03:35.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.496+0000 7f557042d740 1 Processor -- start 2024-03-23T23:03:35.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.497+0000 7f557042d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587e50de050 0x5587e50e7d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:35.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.497+0000 7f557042d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 0x5587e50e8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:35.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.497+0000 7f557042d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587e50e3bb0 0x5587e50ef230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:35.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.497+0000 7f557042d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587e50e6920 con 0x5587e50de050 2024-03-23T23:03:35.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.497+0000 7f557042d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587e4ffee80 con 0x5587e50dea40 2024-03-23T23:03:35.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.497+0000 7f557042d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587e50e6aa0 con 0x5587e50e3bb0 2024-03-23T23:03:35.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.497+0000 7f557042d740 10 client.0 _ll_register_callbacks cb 0x5587e5011130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:35.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.497+0000 7f5566000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 0x5587e50e8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:35.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.497+0000 7f5566a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587e50de050 0x5587e50e7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:35.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.497+0000 7f5566000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 0x5587e50e8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50404/0 (socket says 172.21.15.12:50404) 2024-03-23T23:03:35.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.497+0000 7f5567400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587e50e3bb0 0x5587e50ef230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:35.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.497+0000 7f5566000640 1 -- 172.21.15.12:0/3820097546 learned_addr learned my addr 172.21.15.12:0/3820097546 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:35.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.498+0000 7f5566a00640 1 -- 172.21.15.12:0/3820097546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587e50e3bb0 msgr2=0x5587e50ef230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:35.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.498+0000 7f5566a00640 1 --2- 172.21.15.12:0/3820097546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587e50e3bb0 0x5587e50ef230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.498+0000 7f5566a00640 1 -- 172.21.15.12:0/3820097546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 msgr2=0x5587e50e8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:35.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.498+0000 7f5566a00640 1 --2- 172.21.15.12:0/3820097546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 0x5587e50e8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.498+0000 7f5566a00640 1 -- 172.21.15.12:0/3820097546 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5587e4e217d0 con 0x5587e50de050 2024-03-23T23:03:35.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.498+0000 7f5566000640 1 --2- 172.21.15.12:0/3820097546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 0x5587e50e8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:35.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.498+0000 7f5566a00640 1 --2- 172.21.15.12:0/3820097546 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587e50de050 0x5587e50e7d20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f55580ee610 tx=0x7f5558003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=261b5e7a8542a94c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:35.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.498+0000 7f554be00640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:03:35.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.498+0000 7f554be00640 1 -- 172.21.15.12:0/3820097546 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55580c4770 con 0x5587e50de050 2024-03-23T23:03:35.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.498+0000 7f554be00640 1 -- 172.21.15.12:0/3820097546 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5558003e00 con 0x5587e50de050 2024-03-23T23:03:35.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.498+0000 7f557042d740 1 -- 172.21.15.12:0/3820097546 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5558102070 con 0x5587e50de050 2024-03-23T23:03:35.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.498+0000 7f554be00640 1 -- 172.21.15.12:0/3820097546 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f55580c49b0 con 0x5587e50de050 2024-03-23T23:03:35.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.498+0000 7f557042d740 10 client.15552 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:35.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.498+0000 7f557042d740 1 -- 172.21.15.12:0/3820097546 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5587e4d573f0 con 0x5587e50de050 2024-03-23T23:03:35.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.499+0000 7f554be00640 1 -- 172.21.15.12:0/3820097546 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5558102070 con 0x5587e50de050 2024-03-23T23:03:35.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.499+0000 7f554be00640 1 -- 172.21.15.12:0/3820097546 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f55580bd770 con 0x5587e50de050 2024-03-23T23:03:35.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.499+0000 7f557042d740 10 client.15552 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:35.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.499+0000 7f557042d740 10 client.15552 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:35.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.499+0000 7f557042d740 20 client.15552 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:35.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.499+0000 7f557042d740 1 --2- 172.21.15.12:0/3820097546 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5587e50f3720 0x5587e50f5b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:35.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.499+0000 7f557042d740 4 client.15552 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:03:35.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.499+0000 7f557042d740 1 -- 172.21.15.12:0/3820097546 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5587e5050790 con 0x5587e50f3720 2024-03-23T23:03:35.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.499+0000 7f5566000640 1 --2- 172.21.15.12:0/3820097546 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5587e50f3720 0x5587e50f5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:35.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.500+0000 7f5566000640 1 --2- 172.21.15.12:0/3820097546 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5587e50f3720 0x5587e50f5b10 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=111d0601cc7715c9 server_cookie=241f749109be8d44 in_seq=0 out_seq=0 2024-03-23T23:03:35.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.500+0000 7f554be00640 10 client.15552 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:35.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.501+0000 7f554be00640 1 -- 172.21.15.12:0/3820097546 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5587e5050790 con 0x5587e50f3720 2024-03-23T23:03:35.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.501+0000 7f554be00640 10 client.15552 handle_command_reply: tid=0 2024-03-23T23:03:35.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.501+0000 7f557042d740 1 client.15552 shutdown 2024-03-23T23:03:35.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.501+0000 7f557042d740 1 -- 172.21.15.12:0/3820097546 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587e50de050 msgr2=0x5587e50e7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:35.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.501+0000 7f557042d740 1 --2- 172.21.15.12:0/3820097546 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587e50de050 0x5587e50e7d20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f55580ee610 tx=0x7f5558003760 comp rx=0 tx=0).stop 2024-03-23T23:03:35.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.501+0000 7f557042d740 1 -- 172.21.15.12:0/3820097546 shutdown_connections 2024-03-23T23:03:35.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.501+0000 7f557042d740 1 --2- 172.21.15.12:0/3820097546 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5587e50f3720 0x5587e50f5b10 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.501+0000 7f557042d740 1 --2- 172.21.15.12:0/3820097546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587e50e3bb0 0x5587e50ef230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.501+0000 7f557042d740 1 --2- 172.21.15.12:0/3820097546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 0x5587e50e8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.501+0000 7f557042d740 1 --2- 172.21.15.12:0/3820097546 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587e50de050 0x5587e50e7d20 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.501+0000 7f554be00640 0 client.15552 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:35.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.501+0000 7f557042d740 1 -- 172.21.15.12:0/3820097546 >> 172.21.15.12:0/3820097546 conn(0x5587e50a6200 msgr2=0x5587e50d8bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:35.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.501+0000 7f557042d740 1 -- 172.21.15.12:0/3820097546 shutdown_connections 2024-03-23T23:03:35.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.501+0000 7f557042d740 1 -- 172.21.15.12:0/3820097546 wait complete. 2024-03-23T23:03:35.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.501+0000 7f557042d740 20 client.15552 trim_cache size 0 max 16384 2024-03-23T23:03:35.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.519+0000 7f557042d740 1 Processor -- start 2024-03-23T23:03:35.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.519+0000 7f557042d740 1 -- start start 2024-03-23T23:03:35.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.519+0000 7f557042d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587e50de050 0x5587e50dfd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:35.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.519+0000 7f557042d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 0x5587e50ef1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:35.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.519+0000 7f557042d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587e50e3bb0 0x5587e50d8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:35.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.519+0000 7f557042d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55681acee0 con 0x5587e50e3bb0 2024-03-23T23:03:35.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.519+0000 7f557042d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55681aa9e0 con 0x5587e50dea40 2024-03-23T23:03:35.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.519+0000 7f557042d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55681a8a80 con 0x5587e50de050 2024-03-23T23:03:35.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.520+0000 7f5566000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 0x5587e50ef1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:35.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.520+0000 7f5566a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587e50de050 0x5587e50dfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:35.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.520+0000 7f5566000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 0x5587e50ef1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50418/0 (socket says 172.21.15.12:50418) 2024-03-23T23:03:35.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.520+0000 7f5566000640 1 -- 172.21.15.12:0/36385219 learned_addr learned my addr 172.21.15.12:0/36385219 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:35.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.520+0000 7f5567400640 1 --2- 172.21.15.12:0/36385219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587e50e3bb0 0x5587e50d8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:35.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.520+0000 7f5566000640 1 -- 172.21.15.12:0/36385219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587e50de050 msgr2=0x5587e50dfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:35.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.520+0000 7f5566000640 1 --2- 172.21.15.12:0/36385219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587e50de050 0x5587e50dfd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.520+0000 7f5566000640 1 -- 172.21.15.12:0/36385219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587e50e3bb0 msgr2=0x5587e50d8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:35.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.520+0000 7f5566000640 1 --2- 172.21.15.12:0/36385219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587e50e3bb0 0x5587e50d8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.520+0000 7f5566000640 1 -- 172.21.15.12:0/36385219 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5587e50d51f0 con 0x5587e50dea40 2024-03-23T23:03:35.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.520+0000 7f5566a00640 1 --2- 172.21.15.12:0/36385219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587e50de050 0x5587e50dfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:35.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.520+0000 7f5567400640 1 --2- 172.21.15.12:0/36385219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587e50e3bb0 0x5587e50d8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:03:35.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.520+0000 7f5566000640 1 --2- 172.21.15.12:0/36385219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 0x5587e50ef1f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5550004700 tx=0x7f5550005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39916e0e0f5c3bc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:35.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.521+0000 7f5556a00640 1 -- 172.21.15.12:0/36385219 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5550012110 con 0x5587e50dea40 2024-03-23T23:03:35.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.521+0000 7f5556a00640 1 -- 172.21.15.12:0/36385219 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5550005500 con 0x5587e50dea40 2024-03-23T23:03:35.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.521+0000 7f5556a00640 1 -- 172.21.15.12:0/36385219 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5550010430 con 0x5587e50dea40 2024-03-23T23:03:35.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.521+0000 7f557042d740 1 -- 172.21.15.12:0/36385219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 msgr2=0x5587e50ef1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:35.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.521+0000 7f557042d740 1 --2- 172.21.15.12:0/36385219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 0x5587e50ef1f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5550004700 tx=0x7f5550005f90 comp rx=0 tx=0).stop 2024-03-23T23:03:35.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.521+0000 7f557042d740 1 -- 172.21.15.12:0/36385219 shutdown_connections 2024-03-23T23:03:35.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.521+0000 7f557042d740 1 --2- 172.21.15.12:0/36385219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587e50e3bb0 0x5587e50d8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.521+0000 7f557042d740 1 --2- 172.21.15.12:0/36385219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 0x5587e50ef1f0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.521+0000 7f557042d740 1 --2- 172.21.15.12:0/36385219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587e50de050 0x5587e50dfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.521+0000 7f557042d740 1 -- 172.21.15.12:0/36385219 >> 172.21.15.12:0/36385219 conn(0x5587e50a6200 msgr2=0x5587e50d8330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:35.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.521+0000 7f557042d740 1 -- 172.21.15.12:0/36385219 shutdown_connections 2024-03-23T23:03:35.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.521+0000 7f557042d740 1 -- 172.21.15.12:0/36385219 wait complete. 2024-03-23T23:03:35.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.522+0000 7f557042d740 1 -- start start 2024-03-23T23:03:35.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.522+0000 7f557042d740 1 Processor -- start 2024-03-23T23:03:35.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.522+0000 7f557042d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587e50de050 0x5587e50ece80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:35.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.522+0000 7f557042d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 0x5587e50ed3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:35.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.522+0000 7f557042d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587e50e3bb0 0x5587e50ee3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:35.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.522+0000 7f557042d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55681a9080 con 0x5587e50e3bb0 2024-03-23T23:03:35.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.522+0000 7f557042d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55681aa380 con 0x5587e50dea40 2024-03-23T23:03:35.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.522+0000 7f557042d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55681a96c0 con 0x5587e50de050 2024-03-23T23:03:35.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.522+0000 7f557042d740 10 client.0 _ll_register_callbacks cb 0x7f55580b9040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:35.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.523+0000 7f5566000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 0x5587e50ed3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:35.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.523+0000 7f5566a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587e50de050 0x5587e50ece80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:35.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.523+0000 7f5566000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 0x5587e50ed3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50428/0 (socket says 172.21.15.12:50428) 2024-03-23T23:03:35.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.523+0000 7f5566000640 1 -- 172.21.15.12:0/3466943129 learned_addr learned my addr 172.21.15.12:0/3466943129 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:35.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.523+0000 7f5567400640 1 --2- 172.21.15.12:0/3466943129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587e50e3bb0 0x5587e50ee3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:35.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.523+0000 7f5566000640 1 -- 172.21.15.12:0/3466943129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587e50de050 msgr2=0x5587e50ece80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:35.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.523+0000 7f5566000640 1 --2- 172.21.15.12:0/3466943129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587e50de050 0x5587e50ece80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.523+0000 7f5566000640 1 -- 172.21.15.12:0/3466943129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587e50e3bb0 msgr2=0x5587e50ee3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:35.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.523+0000 7f5566000640 1 --2- 172.21.15.12:0/3466943129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587e50e3bb0 0x5587e50ee3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.523+0000 7f5566000640 1 -- 172.21.15.12:0/3466943129 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5587e4d170f0 con 0x5587e50dea40 2024-03-23T23:03:35.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.523+0000 7f5566a00640 1 --2- 172.21.15.12:0/3466943129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587e50de050 0x5587e50ece80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:35.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.523+0000 7f5566000640 1 --2- 172.21.15.12:0/3466943129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 0x5587e50ed3c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5550012920 tx=0x7f5550012950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d32528160c44416e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:35.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.523+0000 7f554be00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:03:35.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.524+0000 7f554be00640 1 -- 172.21.15.12:0/3466943129 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555002f030 con 0x5587e50dea40 2024-03-23T23:03:35.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.524+0000 7f557042d740 1 -- 172.21.15.12:0/3466943129 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5550005500 con 0x5587e50dea40 2024-03-23T23:03:35.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.524+0000 7f554be00640 1 -- 172.21.15.12:0/3466943129 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5550012110 con 0x5587e50dea40 2024-03-23T23:03:35.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.524+0000 7f554be00640 1 -- 172.21.15.12:0/3466943129 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5550022170 con 0x5587e50dea40 2024-03-23T23:03:35.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.524+0000 7f557042d740 10 client.25706 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:35.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.524+0000 7f557042d740 1 -- 172.21.15.12:0/3466943129 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5587e514f240 con 0x5587e50dea40 2024-03-23T23:03:35.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.524+0000 7f554be00640 1 -- 172.21.15.12:0/3466943129 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5550005500 con 0x5587e50dea40 2024-03-23T23:03:35.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.524+0000 7f554be00640 1 -- 172.21.15.12:0/3466943129 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f555002b750 con 0x5587e50dea40 2024-03-23T23:03:35.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.524+0000 7f557042d740 10 client.25706 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:35.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.524+0000 7f557042d740 10 client.25706 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:35.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.524+0000 7f557042d740 20 client.25706 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:35.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.524+0000 7f557042d740 1 --2- 172.21.15.12:0/3466943129 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5587e514fb40 0x5587e5151f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:35.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.524+0000 7f557042d740 4 client.25706 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:03:35.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.524+0000 7f557042d740 1 -- 172.21.15.12:0/3466943129 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5587e504aae0 con 0x5587e514fb40 2024-03-23T23:03:35.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.525+0000 7f5566a00640 1 --2- 172.21.15.12:0/3466943129 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5587e514fb40 0x5587e5151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:35.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.525+0000 7f5566a00640 1 --2- 172.21.15.12:0/3466943129 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5587e514fb40 0x5587e5151f60 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=859fb050d7647d60 server_cookie=85c213917a1f714e in_seq=0 out_seq=0 2024-03-23T23:03:35.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.525+0000 7f554be00640 10 client.25706 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:35.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.525+0000 7f554be00640 1 -- 172.21.15.12:0/3466943129 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5587e504aae0 con 0x5587e514fb40 2024-03-23T23:03:35.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.525+0000 7f554be00640 10 client.25706 handle_command_reply: tid=0 2024-03-23T23:03:35.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.525+0000 7f557042d740 1 client.25706 shutdown 2024-03-23T23:03:35.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.526+0000 7f557042d740 1 -- 172.21.15.12:0/3466943129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 msgr2=0x5587e50ed3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:35.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.526+0000 7f557042d740 1 --2- 172.21.15.12:0/3466943129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 0x5587e50ed3c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5550012920 tx=0x7f5550012950 comp rx=0 tx=0).stop 2024-03-23T23:03:35.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.526+0000 7f557042d740 1 -- 172.21.15.12:0/3466943129 shutdown_connections 2024-03-23T23:03:35.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.526+0000 7f557042d740 1 --2- 172.21.15.12:0/3466943129 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5587e514fb40 0x5587e5151f60 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.526+0000 7f557042d740 1 --2- 172.21.15.12:0/3466943129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587e50e3bb0 0x5587e50ee3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.526+0000 7f557042d740 1 --2- 172.21.15.12:0/3466943129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587e50dea40 0x5587e50ed3c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.526+0000 7f557042d740 1 --2- 172.21.15.12:0/3466943129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587e50de050 0x5587e50ece80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.526+0000 7f557042d740 1 -- 172.21.15.12:0/3466943129 >> 172.21.15.12:0/3466943129 conn(0x5587e50a6200 msgr2=0x5587e50d8330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:35.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.526+0000 7f554be00640 0 client.25706 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:35.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.526+0000 7f557042d740 1 -- 172.21.15.12:0/3466943129 shutdown_connections 2024-03-23T23:03:35.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.526+0000 7f557042d740 1 -- 172.21.15.12:0/3466943129 wait complete. 2024-03-23T23:03:35.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.526+0000 7f557042d740 20 client.25706 trim_cache size 0 max 16384 2024-03-23T23:03:35.535 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:03:35.535 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-03-23T23:03:35.535 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-03-23T23:03:35.535 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:03:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.527+0000 7f5567e00640 1 -- 172.21.15.12:0/9117516 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f554c0a2800 msgr2=0x7f554c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.527+0000 7f5567e00640 1 --2- 172.21.15.12:0/9117516 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f554c0a2800 0x7f554c0a4cc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x5587e50df2a0 tx=0x7f55600098a0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.527+0000 7f5567e00640 1 -- 172.21.15.12:0/9117516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55681037c0 msgr2=0x7f5568199b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.527+0000 7f5567e00640 1 --2- 172.21.15.12:0/9117516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55681037c0 0x7f5568199b00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5558009230 tx=0x7f555802c6c0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.527+0000 7f5567e00640 1 -- 172.21.15.12:0/9117516 shutdown_connections 2024-03-23T23:03:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.527+0000 7f5567e00640 1 --2- 172.21.15.12:0/9117516 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f554c0a2800 0x7f554c0a4cc0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.527+0000 7f5567e00640 1 --2- 172.21.15.12:0/9117516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5568105370 0x7f55681a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.527+0000 7f5567e00640 1 --2- 172.21.15.12:0/9117516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55681049b0 0x7f556819a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.527+0000 7f5567e00640 1 --2- 172.21.15.12:0/9117516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55681037c0 0x7f5568199b00 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.527+0000 7f5567e00640 1 -- 172.21.15.12:0/9117516 >> 172.21.15.12:0/9117516 conn(0x7f55680fed50 msgr2=0x7f5568107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.527+0000 7f5567e00640 1 -- 172.21.15.12:0/9117516 shutdown_connections 2024-03-23T23:03:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.527+0000 7f5567e00640 1 -- 172.21.15.12:0/9117516 wait complete. 2024-03-23T23:03:35.545 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T23:03:35.545 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:03:35.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.769+0000 7f4fd1e00640 1 -- 172.21.15.12:0/189887890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fcc1048e0 msgr2=0x7f4fcc104d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:35.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.769+0000 7f4fd1e00640 1 --2- 172.21.15.12:0/189887890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fcc1048e0 0x7f4fcc104d60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8009900 tx=0x7f4fb802bc50 comp rx=0 tx=0).stop 2024-03-23T23:03:35.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.769+0000 7f4fd1e00640 1 -- 172.21.15.12:0/189887890 shutdown_connections 2024-03-23T23:03:35.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.769+0000 7f4fd1e00640 1 --2- 172.21.15.12:0/189887890 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4fcc1052a0 0x7f4fcc10bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.769+0000 7f4fd1e00640 1 --2- 172.21.15.12:0/189887890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fcc1048e0 0x7f4fcc104d60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.769+0000 7f4fd1e00640 1 --2- 172.21.15.12:0/189887890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fcc1036f0 0x7f4fcc103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.769+0000 7f4fd1e00640 1 -- 172.21.15.12:0/189887890 >> 172.21.15.12:0/189887890 conn(0x7f4fcc0fec80 msgr2=0x7f4fcc1010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:35.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.770+0000 7f4fd1e00640 1 -- 172.21.15.12:0/189887890 shutdown_connections 2024-03-23T23:03:35.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.770+0000 7f4fd1e00640 1 -- 172.21.15.12:0/189887890 wait complete. 2024-03-23T23:03:35.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.771+0000 7f4fd1e00640 1 Processor -- start 2024-03-23T23:03:35.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.771+0000 7f4fd1e00640 1 -- start start 2024-03-23T23:03:35.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.771+0000 7f4fd1e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4fcc1036f0 0x7f4fcc199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:35.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.771+0000 7f4fd1e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fcc1048e0 0x7f4fcc199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:35.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.771+0000 7f4fd1e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fcc1052a0 0x7f4fcc1a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:35.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.771+0000 7f4fd1e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fcc10f350 con 0x7f4fcc1052a0 2024-03-23T23:03:35.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.771+0000 7f4fd1e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fcc10f1d0 con 0x7f4fcc1036f0 2024-03-23T23:03:35.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.771+0000 7f4fd1e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fcc10f4d0 con 0x7f4fcc1048e0 2024-03-23T23:03:35.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.772+0000 7f4fcbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fcc1048e0 0x7f4fcc199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:35.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.772+0000 7f4fd0a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4fcc1036f0 0x7f4fcc199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:35.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.772+0000 7f4fcbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fcc1048e0 0x7f4fcc199fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35320/0 (socket says 172.21.15.12:35320) 2024-03-23T23:03:35.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.772+0000 7f4fcbe00640 1 -- 172.21.15.12:0/1130346103 learned_addr learned my addr 172.21.15.12:0/1130346103 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:35.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.772+0000 7f4fd1400640 1 --2- 172.21.15.12:0/1130346103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fcc1052a0 0x7f4fcc1a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:35.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.772+0000 7f4fd0a00640 1 -- 172.21.15.12:0/1130346103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fcc1048e0 msgr2=0x7f4fcc199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:35.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.772+0000 7f4fd0a00640 1 --2- 172.21.15.12:0/1130346103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fcc1048e0 0x7f4fcc199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.772+0000 7f4fd0a00640 1 -- 172.21.15.12:0/1130346103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fcc1052a0 msgr2=0x7f4fcc1a1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:35.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.772+0000 7f4fd0a00640 1 --2- 172.21.15.12:0/1130346103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fcc1052a0 0x7f4fcc1a1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:35.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.772+0000 7f4fd0a00640 1 -- 172.21.15.12:0/1130346103 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fcc1a1690 con 0x7f4fcc1036f0 2024-03-23T23:03:35.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.772+0000 7f4fd0a00640 1 --2- 172.21.15.12:0/1130346103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4fcc1036f0 0x7f4fcc199a70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4fc000e9c0 tx=0x7f4fc000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:35.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.773+0000 7f4fc9600640 1 -- 172.21.15.12:0/1130346103 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc000ccc0 con 0x7f4fcc1036f0 2024-03-23T23:03:35.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.773+0000 7f4fc9600640 1 -- 172.21.15.12:0/1130346103 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4fc00035f0 con 0x7f4fcc1036f0 2024-03-23T23:03:35.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.773+0000 7f4fc9600640 1 -- 172.21.15.12:0/1130346103 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc000ce60 con 0x7f4fcc1036f0 2024-03-23T23:03:35.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.773+0000 7f4fd1e00640 1 -- 172.21.15.12:0/1130346103 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fcc1a1980 con 0x7f4fcc1036f0 2024-03-23T23:03:35.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.773+0000 7f4fd1e00640 1 -- 172.21.15.12:0/1130346103 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fcc1a1e50 con 0x7f4fcc1036f0 2024-03-23T23:03:35.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.774+0000 7f4fd1e00640 1 -- 172.21.15.12:0/1130346103 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fcc068800 con 0x7f4fcc1036f0 2024-03-23T23:03:35.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.776+0000 7f4fc9600640 1 -- 172.21.15.12:0/1130346103 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4fc00037a0 con 0x7f4fcc1036f0 2024-03-23T23:03:35.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.777+0000 7f4fc9600640 1 --2- 172.21.15.12:0/1130346103 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4fb40a2880 0x7f4fb40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:35.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.778+0000 7f4fc9600640 1 -- 172.21.15.12:0/1130346103 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4fc00cee40 con 0x7f4fcc1036f0 2024-03-23T23:03:35.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.778+0000 7f4fcbe00640 1 --2- 172.21.15.12:0/1130346103 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4fb40a2880 0x7f4fb40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:35.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.779+0000 7f4fcbe00640 1 --2- 172.21.15.12:0/1130346103 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4fb40a2880 0x7f4fb40a4d40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8000c00 tx=0x7f4fb802c8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:35.782+0000 7f4fc9600640 1 -- 172.21.15.12:0/1130346103 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4fc009be90 con 0x7f4fcc1036f0 2024-03-23T23:03:36.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.027+0000 7f4fd1e00640 1 -- 172.21.15.12:0/1130346103 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4fcc10aa90 con 0x7f4fcc1036f0 2024-03-23T23:03:36.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.029+0000 7f4fc9600640 1 -- 172.21.15.12:0/1130346103 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f4fc009b790 con 0x7f4fcc1036f0 2024-03-23T23:03:36.029 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:03:36.029 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:03:36.029 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:03:36.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.032+0000 7f4fd1e00640 1 -- 172.21.15.12:0/1130346103 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4fb40a2880 msgr2=0x7f4fb40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.032+0000 7f4fd1e00640 1 --2- 172.21.15.12:0/1130346103 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4fb40a2880 0x7f4fb40a4d40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8000c00 tx=0x7f4fb802c8a0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.032+0000 7f4fd1e00640 1 -- 172.21.15.12:0/1130346103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4fcc1036f0 msgr2=0x7f4fcc199a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.032+0000 7f4fd1e00640 1 --2- 172.21.15.12:0/1130346103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4fcc1036f0 0x7f4fcc199a70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4fc000e9c0 tx=0x7f4fc000ee80 comp rx=0 tx=0).stop 2024-03-23T23:03:36.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.032+0000 7f4fd1e00640 1 -- 172.21.15.12:0/1130346103 shutdown_connections 2024-03-23T23:03:36.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.032+0000 7f4fd1e00640 1 --2- 172.21.15.12:0/1130346103 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4fb40a2880 0x7f4fb40a4d40 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.032+0000 7f4fd1e00640 1 --2- 172.21.15.12:0/1130346103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fcc1052a0 0x7f4fcc1a1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.032+0000 7f4fd1e00640 1 --2- 172.21.15.12:0/1130346103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fcc1048e0 0x7f4fcc199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.032+0000 7f4fd1e00640 1 --2- 172.21.15.12:0/1130346103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4fcc1036f0 0x7f4fcc199a70 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.032+0000 7f4fd1e00640 1 -- 172.21.15.12:0/1130346103 >> 172.21.15.12:0/1130346103 conn(0x7f4fcc0fec80 msgr2=0x7f4fcc107c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:36.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.032+0000 7f4fd1e00640 1 -- 172.21.15.12:0/1130346103 shutdown_connections 2024-03-23T23:03:36.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.032+0000 7f4fd1e00640 1 -- 172.21.15.12:0/1130346103 wait complete. 2024-03-23T23:03:36.052 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:03:36.052 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-23T23:03:36.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.277+0000 7fa81ae00640 1 -- 172.21.15.12:0/2094019217 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8141037c0 msgr2=0x7fa814103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.277+0000 7fa81ae00640 1 --2- 172.21.15.12:0/2094019217 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8141037c0 0x7fa814103be0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa804009900 tx=0x7fa80402bc60 comp rx=0 tx=0).stop 2024-03-23T23:03:36.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.277+0000 7fa81ae00640 1 -- 172.21.15.12:0/2094019217 shutdown_connections 2024-03-23T23:03:36.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.277+0000 7fa81ae00640 1 --2- 172.21.15.12:0/2094019217 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa814105370 0x7fa81410bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.277+0000 7fa81ae00640 1 --2- 172.21.15.12:0/2094019217 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8141049b0 0x7fa814104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.277+0000 7fa81ae00640 1 --2- 172.21.15.12:0/2094019217 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8141037c0 0x7fa814103be0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.277+0000 7fa81ae00640 1 -- 172.21.15.12:0/2094019217 >> 172.21.15.12:0/2094019217 conn(0x7fa8140fed50 msgr2=0x7fa814101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:36.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.277+0000 7fa81ae00640 1 -- 172.21.15.12:0/2094019217 shutdown_connections 2024-03-23T23:03:36.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.277+0000 7fa81ae00640 1 -- 172.21.15.12:0/2094019217 wait complete. 2024-03-23T23:03:36.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.279+0000 7fa81ae00640 1 Processor -- start 2024-03-23T23:03:36.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.279+0000 7fa81ae00640 1 -- start start 2024-03-23T23:03:36.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.279+0000 7fa81ae00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8141037c0 0x7fa814199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.279+0000 7fa81ae00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8141049b0 0x7fa81419a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.279+0000 7fa81ae00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa814105370 0x7fa8141a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.279+0000 7fa81ae00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81410f500 con 0x7fa814105370 2024-03-23T23:03:36.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.279+0000 7fa81ae00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81410f380 con 0x7fa8141049b0 2024-03-23T23:03:36.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.279+0000 7fa81ae00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81410f680 con 0x7fa8141037c0 2024-03-23T23:03:36.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.279+0000 7fa819a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8141037c0 0x7fa814199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.279+0000 7fa819a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8141037c0 0x7fa814199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35326/0 (socket says 172.21.15.12:35326) 2024-03-23T23:03:36.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.279+0000 7fa819a00640 1 -- 172.21.15.12:0/3716826017 learned_addr learned my addr 172.21.15.12:0/3716826017 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:36.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.279+0000 7fa81a400640 1 --2- 172.21.15.12:0/3716826017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa814105370 0x7fa8141a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.280+0000 7fa819000640 1 --2- 172.21.15.12:0/3716826017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8141049b0 0x7fa81419a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.280+0000 7fa819a00640 1 -- 172.21.15.12:0/3716826017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8141049b0 msgr2=0x7fa81419a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.280+0000 7fa819a00640 1 --2- 172.21.15.12:0/3716826017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8141049b0 0x7fa81419a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.280+0000 7fa819a00640 1 -- 172.21.15.12:0/3716826017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa814105370 msgr2=0x7fa8141a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.280+0000 7fa819a00640 1 --2- 172.21.15.12:0/3716826017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa814105370 0x7fa8141a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.280+0000 7fa819a00640 1 -- 172.21.15.12:0/3716826017 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8141a1770 con 0x7fa8141037c0 2024-03-23T23:03:36.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.280+0000 7fa819a00640 1 --2- 172.21.15.12:0/3716826017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8141037c0 0x7fa814199b50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa80402efd0 tx=0x7fa804005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:36.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.281+0000 7fa812000640 1 -- 172.21.15.12:0/3716826017 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa804041070 con 0x7fa8141037c0 2024-03-23T23:03:36.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.281+0000 7fa812000640 1 -- 172.21.15.12:0/3716826017 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8040044d0 con 0x7fa8141037c0 2024-03-23T23:03:36.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.281+0000 7fa812000640 1 -- 172.21.15.12:0/3716826017 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa804034440 con 0x7fa8141037c0 2024-03-23T23:03:36.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.281+0000 7fa81ae00640 1 -- 172.21.15.12:0/3716826017 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8141a1a00 con 0x7fa8141037c0 2024-03-23T23:03:36.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.281+0000 7fa81ae00640 1 -- 172.21.15.12:0/3716826017 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8141a1e70 con 0x7fa8141037c0 2024-03-23T23:03:36.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.285+0000 7fa81f1cd740 1 Processor -- start 2024-03-23T23:03:36.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.286+0000 7fa812000640 1 -- 172.21.15.12:0/3716826017 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa804044070 con 0x7fa8141037c0 2024-03-23T23:03:36.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.286+0000 7fa81f1cd740 1 -- start start 2024-03-23T23:03:36.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.286+0000 7fa81f1cd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641233aa9f0 0x5641233aae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.286+0000 7fa81f1cd740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5641233abbc0 0x5641233ac040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.286+0000 7fa81f1cd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641233ac580 0x5641233b2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.286+0000 7fa81f1cd740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56412330f2c0 con 0x5641233abbc0 2024-03-23T23:03:36.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.286+0000 7fa81f1cd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641232b7c90 con 0x5641233ac580 2024-03-23T23:03:36.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.286+0000 7fa81f1cd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x56412330b360 con 0x5641233aa9f0 2024-03-23T23:03:36.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.286+0000 7fa81a400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641233aa9f0 0x5641233aae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.286+0000 7fa819a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5641233abbc0 0x5641233ac040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.286+0000 7fa81a400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641233aa9f0 0x5641233aae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35342/0 (socket says 172.21.15.12:35342) 2024-03-23T23:03:36.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.286+0000 7fa81a400640 1 -- 172.21.15.12:0/3131049039 learned_addr learned my addr 172.21.15.12:0/3131049039 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:36.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.287+0000 7fa819000640 1 --2- 172.21.15.12:0/3131049039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641233ac580 0x5641233b2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.287+0000 7fa81a400640 1 -- 172.21.15.12:0/3131049039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641233ac580 msgr2=0x5641233b2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:36.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.287+0000 7fa81a400640 1 --2- 172.21.15.12:0/3131049039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641233ac580 0x5641233b2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.287+0000 7fa81a400640 1 -- 172.21.15.12:0/3131049039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5641233abbc0 msgr2=0x5641233ac040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:36.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.287+0000 7fa81a400640 1 --2- 172.21.15.12:0/3131049039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5641233abbc0 0x5641233ac040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.287+0000 7fa81a400640 1 -- 172.21.15.12:0/3131049039 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641232dc080 con 0x5641233aa9f0 2024-03-23T23:03:36.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.287+0000 7fa81a400640 1 --2- 172.21.15.12:0/3131049039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641233aa9f0 0x5641233aae10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa80c00a4d0 tx=0x7fa80c009650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb289ef8831a7e36 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:36.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.287+0000 7fa812000640 1 --2- 172.21.15.12:0/3716826017 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa7fc0a2800 0x7fa7fc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.287+0000 7fa802a00640 1 -- 172.21.15.12:0/3131049039 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80c018020 con 0x5641233aa9f0 2024-03-23T23:03:36.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.287+0000 7fa802a00640 1 -- 172.21.15.12:0/3131049039 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa80c014cf0 con 0x5641233aa9f0 2024-03-23T23:03:36.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.287+0000 7fa802a00640 1 -- 172.21.15.12:0/3131049039 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80c01a2c0 con 0x5641233aa9f0 2024-03-23T23:03:36.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.287+0000 7fa812000640 1 -- 172.21.15.12:0/3716826017 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fa8040f0990 con 0x7fa8141037c0 2024-03-23T23:03:36.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.288+0000 7fa819a00640 1 --2- 172.21.15.12:0/3716826017 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa7fc0a2800 0x7fa7fc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.288+0000 7fa81f1cd740 1 -- 172.21.15.12:0/3131049039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641233aa9f0 msgr2=0x5641233aae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.288+0000 7fa81f1cd740 1 --2- 172.21.15.12:0/3131049039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641233aa9f0 0x5641233aae10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa80c00a4d0 tx=0x7fa80c009650 comp rx=0 tx=0).stop 2024-03-23T23:03:36.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.288+0000 7fa819a00640 1 --2- 172.21.15.12:0/3716826017 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa7fc0a2800 0x7fa7fc0a4cc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa8040b9d60 tx=0x7fa8040dd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:36.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.288+0000 7fa81f1cd740 1 -- 172.21.15.12:0/3131049039 shutdown_connections 2024-03-23T23:03:36.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.288+0000 7fa81f1cd740 1 --2- 172.21.15.12:0/3131049039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641233ac580 0x5641233b2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.288+0000 7fa81f1cd740 1 --2- 172.21.15.12:0/3131049039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5641233abbc0 0x5641233ac040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.288+0000 7fa81f1cd740 1 --2- 172.21.15.12:0/3131049039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641233aa9f0 0x5641233aae10 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.288+0000 7fa81f1cd740 1 -- 172.21.15.12:0/3131049039 >> 172.21.15.12:0/3131049039 conn(0x5641233a5de0 msgr2=0x5641233a61e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:36.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.288+0000 7fa81f1cd740 1 -- 172.21.15.12:0/3131049039 shutdown_connections 2024-03-23T23:03:36.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.289+0000 7fa81f1cd740 1 -- 172.21.15.12:0/3131049039 wait complete. 2024-03-23T23:03:36.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.289+0000 7fa81f1cd740 1 -- start start 2024-03-23T23:03:36.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.289+0000 7fa81f1cd740 1 Processor -- start 2024-03-23T23:03:36.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.290+0000 7fa81f1cd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641233aa9f0 0x5641233b4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.290+0000 7fa81f1cd740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5641233abbc0 0x5641233b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.290+0000 7fa81f1cd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641233ac580 0x5641233bc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.290+0000 7fa81f1cd740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641233b3590 con 0x5641233abbc0 2024-03-23T23:03:36.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.290+0000 7fa81f1cd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641232cb0d0 con 0x5641233ac580 2024-03-23T23:03:36.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.290+0000 7fa81f1cd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641233b3710 con 0x5641233aa9f0 2024-03-23T23:03:36.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.290+0000 7fa81f1cd740 10 client.0 _ll_register_callbacks cb 0x5641232cfe80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:36.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.290+0000 7fa81a400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641233aa9f0 0x5641233b4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.290+0000 7fa81a400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641233aa9f0 0x5641233b4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35344/0 (socket says 172.21.15.12:35344) 2024-03-23T23:03:36.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.290+0000 7fa81a400640 1 -- 172.21.15.12:0/1407091834 learned_addr learned my addr 172.21.15.12:0/1407091834 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:36.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.290+0000 7fa819000640 1 --2- 172.21.15.12:0/1407091834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5641233abbc0 0x5641233b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.290+0000 7fa81a400640 1 --2- 172.21.15.12:0/1407091834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641233ac580 0x5641233bc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.290+0000 7fa81a400640 1 -- 172.21.15.12:0/1407091834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641233aa9f0 msgr2=0x5641233b4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:36.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.290+0000 7fa81a400640 1 --2- 172.21.15.12:0/1407091834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641233aa9f0 0x5641233b4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.291+0000 7fa81a400640 1 -- 172.21.15.12:0/1407091834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5641233abbc0 msgr2=0x5641233b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:36.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.291+0000 7fa81a400640 1 --2- 172.21.15.12:0/1407091834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5641233abbc0 0x5641233b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.291+0000 7fa81a400640 1 -- 172.21.15.12:0/1407091834 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5641230233f0 con 0x5641233ac580 2024-03-23T23:03:36.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.291+0000 7fa819000640 1 --2- 172.21.15.12:0/1407091834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5641233abbc0 0x5641233b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:36.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.291+0000 7fa81a400640 1 --2- 172.21.15.12:0/1407091834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641233ac580 0x5641233bc1e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa80c009a40 tx=0x7fa80c009c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe73ebc55f2c0ebb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:36.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.291+0000 7fa7f7400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:03:36.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.291+0000 7fa7f7400640 1 -- 172.21.15.12:0/1407091834 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80c018030 con 0x5641233ac580 2024-03-23T23:03:36.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.291+0000 7fa7f7400640 1 -- 172.21.15.12:0/1407091834 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fa80c014a50 con 0x5641233ac580 2024-03-23T23:03:36.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.291+0000 7fa81f1cd740 1 -- 172.21.15.12:0/1407091834 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa80c014cf0 con 0x5641233ac580 2024-03-23T23:03:36.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.291+0000 7fa7f7400640 1 -- 172.21.15.12:0/1407091834 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa80c01e430 con 0x5641233ac580 2024-03-23T23:03:36.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.291+0000 7fa81f1cd740 10 client.25724 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:36.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.291+0000 7fa81f1cd740 1 -- 172.21.15.12:0/1407091834 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5641230ed7d0 con 0x5641233ac580 2024-03-23T23:03:36.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.292+0000 7fa7f7400640 1 -- 172.21.15.12:0/1407091834 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa80c014cf0 con 0x5641233ac580 2024-03-23T23:03:36.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.292+0000 7fa7f7400640 1 -- 172.21.15.12:0/1407091834 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fa80c01d4b0 con 0x5641233ac580 2024-03-23T23:03:36.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.292+0000 7fa81f1cd740 10 client.25724 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:36.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.292+0000 7fa81f1cd740 10 client.25724 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:03:36.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.292+0000 7fa81f1cd740 20 client.25724 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:36.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.292+0000 7fa81f1cd740 1 --2- 172.21.15.12:0/1407091834 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5641233c03b0 0x5641233c27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.292+0000 7fa81f1cd740 4 client.25724 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:03:36.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.292+0000 7fa81f1cd740 1 -- 172.21.15.12:0/1407091834 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56412331c800 con 0x5641233c03b0 2024-03-23T23:03:36.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.292+0000 7fa819000640 1 --2- 172.21.15.12:0/1407091834 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5641233c03b0 0x5641233c27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.293+0000 7fa819000640 1 --2- 172.21.15.12:0/1407091834 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5641233c03b0 0x5641233c27d0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=85dbf0973db3257f server_cookie=d3c4e743f9460f5e in_seq=0 out_seq=0 2024-03-23T23:03:36.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.293+0000 7fa7f7400640 10 client.25724 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:03:36.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.294+0000 7fa7f7400640 1 -- 172.21.15.12:0/1407091834 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56412331c800 con 0x5641233c03b0 2024-03-23T23:03:36.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.294+0000 7fa7f7400640 10 client.25724 handle_command_reply: tid=0 2024-03-23T23:03:36.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.294+0000 7fa81f1cd740 1 client.25724 shutdown 2024-03-23T23:03:36.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.294+0000 7fa81f1cd740 1 -- 172.21.15.12:0/1407091834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641233ac580 msgr2=0x5641233bc1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.294+0000 7fa81f1cd740 1 --2- 172.21.15.12:0/1407091834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641233ac580 0x5641233bc1e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa80c009a40 tx=0x7fa80c009c60 comp rx=0 tx=0).stop 2024-03-23T23:03:36.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.294+0000 7fa81f1cd740 1 -- 172.21.15.12:0/1407091834 shutdown_connections 2024-03-23T23:03:36.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.294+0000 7fa81f1cd740 1 --2- 172.21.15.12:0/1407091834 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5641233c03b0 0x5641233c27d0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.294+0000 7fa81f1cd740 1 --2- 172.21.15.12:0/1407091834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641233ac580 0x5641233bc1e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.294+0000 7fa81f1cd740 1 --2- 172.21.15.12:0/1407091834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5641233abbc0 0x5641233b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.294+0000 7fa81f1cd740 1 --2- 172.21.15.12:0/1407091834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641233aa9f0 0x5641233b4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.294+0000 7fa7f7400640 0 client.25724 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:03:36.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.294+0000 7fa81f1cd740 1 -- 172.21.15.12:0/1407091834 >> 172.21.15.12:0/1407091834 conn(0x5641233a5de0 msgr2=0x5641233a5160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:36.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.294+0000 7fa81f1cd740 1 -- 172.21.15.12:0/1407091834 shutdown_connections 2024-03-23T23:03:36.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.294+0000 7fa81f1cd740 1 -- 172.21.15.12:0/1407091834 wait complete. 2024-03-23T23:03:36.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.294+0000 7fa81f1cd740 20 client.25724 trim_cache size 0 max 16384 2024-03-23T23:03:36.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.308+0000 7fa81f1cd740 1 Processor -- start 2024-03-23T23:03:36.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.308+0000 7fa81f1cd740 1 -- start start 2024-03-23T23:03:36.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.309+0000 7fa81f1cd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641233aa9f0 0x5641233aec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.309+0000 7fa81f1cd740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5641233abbc0 0x5641233bc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.309+0000 7fa81f1cd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641233ac580 0x5641233af1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.309+0000 7fa81f1cd740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8141acf90 con 0x5641233abbc0 2024-03-23T23:03:36.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.309+0000 7fa81f1cd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8141aaa90 con 0x5641233aa9f0 2024-03-23T23:03:36.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.309+0000 7fa81f1cd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8141a8b30 con 0x5641233ac580 2024-03-23T23:03:36.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.309+0000 7fa819000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5641233abbc0 0x5641233bc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.309+0000 7fa81a400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641233aa9f0 0x5641233aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.309+0000 7fa81a400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641233aa9f0 0x5641233aec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50496/0 (socket says 172.21.15.12:50496) 2024-03-23T23:03:36.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.309+0000 7fa81a400640 1 -- 172.21.15.12:0/2739999450 learned_addr learned my addr 172.21.15.12:0/2739999450 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:36.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.309+0000 7fa81a400640 1 --2- 172.21.15.12:0/2739999450 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641233ac580 0x5641233af1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.310+0000 7fa81a400640 1 -- 172.21.15.12:0/2739999450 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641233ac580 msgr2=0x5641233af1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:36.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.310+0000 7fa81a400640 1 --2- 172.21.15.12:0/2739999450 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641233ac580 0x5641233af1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.310+0000 7fa81a400640 1 -- 172.21.15.12:0/2739999450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5641233abbc0 msgr2=0x5641233bc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:36.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.310+0000 7fa81a400640 1 --2- 172.21.15.12:0/2739999450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5641233abbc0 0x5641233bc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.310+0000 7fa81a400640 1 -- 172.21.15.12:0/2739999450 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641233ab3e0 con 0x5641233aa9f0 2024-03-23T23:03:36.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.310+0000 7fa81a400640 1 --2- 172.21.15.12:0/2739999450 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641233aa9f0 0x5641233aec60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa80c013fb0 tx=0x7fa80c009990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5a24811cbcd76d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:36.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.310+0000 7fa802000640 1 -- 172.21.15.12:0/2739999450 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80c018040 con 0x5641233aa9f0 2024-03-23T23:03:36.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.310+0000 7fa802000640 1 -- 172.21.15.12:0/2739999450 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa80c00ae20 con 0x5641233aa9f0 2024-03-23T23:03:36.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.310+0000 7fa802000640 1 -- 172.21.15.12:0/2739999450 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80c01d420 con 0x5641233aa9f0 2024-03-23T23:03:36.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.311+0000 7fa81f1cd740 1 -- 172.21.15.12:0/2739999450 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641233aa9f0 msgr2=0x5641233aec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.311+0000 7fa81f1cd740 1 --2- 172.21.15.12:0/2739999450 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641233aa9f0 0x5641233aec60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa80c013fb0 tx=0x7fa80c009990 comp rx=0 tx=0).stop 2024-03-23T23:03:36.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.311+0000 7fa81f1cd740 1 -- 172.21.15.12:0/2739999450 shutdown_connections 2024-03-23T23:03:36.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.311+0000 7fa81f1cd740 1 --2- 172.21.15.12:0/2739999450 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641233ac580 0x5641233af1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.311+0000 7fa81f1cd740 1 --2- 172.21.15.12:0/2739999450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5641233abbc0 0x5641233bc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.311+0000 7fa81f1cd740 1 --2- 172.21.15.12:0/2739999450 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641233aa9f0 0x5641233aec60 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.311+0000 7fa81f1cd740 1 -- 172.21.15.12:0/2739999450 >> 172.21.15.12:0/2739999450 conn(0x5641233a5de0 msgr2=0x5641233a98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:36.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.311+0000 7fa81f1cd740 1 -- 172.21.15.12:0/2739999450 shutdown_connections 2024-03-23T23:03:36.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.311+0000 7fa81f1cd740 1 -- 172.21.15.12:0/2739999450 wait complete. 2024-03-23T23:03:36.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.312+0000 7fa81f1cd740 1 -- start start 2024-03-23T23:03:36.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.312+0000 7fa81f1cd740 1 Processor -- start 2024-03-23T23:03:36.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.312+0000 7fa81f1cd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641233aa9f0 0x5641233b7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.312+0000 7fa81f1cd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641233abbc0 0x5641233b80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.312+0000 7fa81f1cd740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5641233ac580 0x5641233bb100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.312+0000 7fa81f1cd740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8141a9130 con 0x5641233ac580 2024-03-23T23:03:36.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.312+0000 7fa81f1cd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8141aa430 con 0x5641233aa9f0 2024-03-23T23:03:36.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.312+0000 7fa81f1cd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8141a9770 con 0x5641233abbc0 2024-03-23T23:03:36.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.312+0000 7fa81f1cd740 10 client.0 _ll_register_callbacks cb 0x7fa80c009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:36.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.313+0000 7fa819000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641233abbc0 0x5641233b80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.313+0000 7fa819000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641233abbc0 0x5641233b80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35364/0 (socket says 172.21.15.12:35364) 2024-03-23T23:03:36.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.313+0000 7fa819000640 1 -- 172.21.15.12:0/1402283903 learned_addr learned my addr 172.21.15.12:0/1402283903 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:36.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.313+0000 7fa81a400640 1 --2- 172.21.15.12:0/1402283903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641233aa9f0 0x5641233b7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.313+0000 7fa81a400640 1 --2- 172.21.15.12:0/1402283903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5641233ac580 0x5641233bb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.313+0000 7fa819000640 1 -- 172.21.15.12:0/1402283903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641233aa9f0 msgr2=0x5641233b7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:36.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.313+0000 7fa819000640 1 --2- 172.21.15.12:0/1402283903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641233aa9f0 0x5641233b7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.313+0000 7fa819000640 1 -- 172.21.15.12:0/1402283903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5641233ac580 msgr2=0x5641233bb100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:36.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.313+0000 7fa819000640 1 --2- 172.21.15.12:0/1402283903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5641233ac580 0x5641233bb100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.313+0000 7fa819000640 1 -- 172.21.15.12:0/1402283903 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5641233b5510 con 0x5641233abbc0 2024-03-23T23:03:36.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.313+0000 7fa81a400640 1 --2- 172.21.15.12:0/1402283903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641233aa9f0 0x5641233b7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:03:36.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.313+0000 7fa819000640 1 --2- 172.21.15.12:0/1402283903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641233abbc0 0x5641233b80e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x5641233c3040 tx=0x7fa808022d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=848f6b34b5532e31 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:36.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.313+0000 7fa7f7400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:03:36.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.314+0000 7fa7f7400640 1 -- 172.21.15.12:0/1402283903 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8080222b0 con 0x5641233abbc0 2024-03-23T23:03:36.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.314+0000 7fa7f7400640 1 -- 172.21.15.12:0/1402283903 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fa808012a20 con 0x5641233abbc0 2024-03-23T23:03:36.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.314+0000 7fa81f1cd740 1 -- 172.21.15.12:0/1402283903 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa80c00ae20 con 0x5641233abbc0 2024-03-23T23:03:36.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.314+0000 7fa7f7400640 1 -- 172.21.15.12:0/1402283903 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa808032050 con 0x5641233abbc0 2024-03-23T23:03:36.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.314+0000 7fa81f1cd740 10 client.25708 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:36.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.314+0000 7fa81f1cd740 1 -- 172.21.15.12:0/1402283903 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56412341be50 con 0x5641233abbc0 2024-03-23T23:03:36.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.314+0000 7fa7f7400640 1 -- 172.21.15.12:0/1402283903 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa80c00ae20 con 0x5641233abbc0 2024-03-23T23:03:36.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.314+0000 7fa7f7400640 1 -- 172.21.15.12:0/1402283903 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fa808003950 con 0x5641233abbc0 2024-03-23T23:03:36.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.314+0000 7fa81f1cd740 10 client.25708 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:36.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.314+0000 7fa81f1cd740 10 client.25708 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:03:36.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.314+0000 7fa81f1cd740 20 client.25708 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:36.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.315+0000 7fa81f1cd740 1 --2- 172.21.15.12:0/1402283903 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x56412341c770 0x56412341ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.315+0000 7fa81f1cd740 4 client.25708 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:03:36.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.315+0000 7fa81f1cd740 1 -- 172.21.15.12:0/1402283903 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564123316b50 con 0x56412341c770 2024-03-23T23:03:36.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.315+0000 7fa81a400640 1 --2- 172.21.15.12:0/1402283903 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x56412341c770 0x56412341ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.315+0000 7fa81a400640 1 --2- 172.21.15.12:0/1402283903 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x56412341c770 0x56412341ebf0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e8dbd409cd0fe83c server_cookie=e51e7de5d54091fe in_seq=0 out_seq=0 2024-03-23T23:03:36.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.315+0000 7fa7f7400640 10 client.25708 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:03:36.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.315+0000 7fa7f7400640 1 -- 172.21.15.12:0/1402283903 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564123316b50 con 0x56412341c770 2024-03-23T23:03:36.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.315+0000 7fa7f7400640 10 client.25708 handle_command_reply: tid=0 2024-03-23T23:03:36.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.315+0000 7fa81f1cd740 1 client.25708 shutdown 2024-03-23T23:03:36.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.316+0000 7fa81f1cd740 1 -- 172.21.15.12:0/1402283903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641233abbc0 msgr2=0x5641233b80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.316+0000 7fa81f1cd740 1 --2- 172.21.15.12:0/1402283903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641233abbc0 0x5641233b80e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x5641233c3040 tx=0x7fa808022d70 comp rx=0 tx=0).stop 2024-03-23T23:03:36.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.316+0000 7fa81f1cd740 1 -- 172.21.15.12:0/1402283903 shutdown_connections 2024-03-23T23:03:36.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.316+0000 7fa81f1cd740 1 --2- 172.21.15.12:0/1402283903 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x56412341c770 0x56412341ebf0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.316+0000 7fa81f1cd740 1 --2- 172.21.15.12:0/1402283903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5641233ac580 0x5641233bb100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.316+0000 7fa81f1cd740 1 --2- 172.21.15.12:0/1402283903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641233abbc0 0x5641233b80e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.316+0000 7fa81f1cd740 1 --2- 172.21.15.12:0/1402283903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641233aa9f0 0x5641233b7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.316+0000 7fa7f7400640 0 client.25708 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:03:36.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.316+0000 7fa81f1cd740 1 -- 172.21.15.12:0/1402283903 >> 172.21.15.12:0/1402283903 conn(0x5641233a5de0 msgr2=0x5641233a48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:36.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.316+0000 7fa81f1cd740 1 -- 172.21.15.12:0/1402283903 shutdown_connections 2024-03-23T23:03:36.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.316+0000 7fa81f1cd740 1 -- 172.21.15.12:0/1402283903 wait complete. 2024-03-23T23:03:36.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.316+0000 7fa81f1cd740 20 client.25708 trim_cache size 0 max 16384 2024-03-23T23:03:36.320 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:03:36.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.317+0000 7fa81ae00640 1 -- 172.21.15.12:0/3716826017 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa7fc0a2800 msgr2=0x7fa7fc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.317+0000 7fa81ae00640 1 --2- 172.21.15.12:0/3716826017 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa7fc0a2800 0x7fa7fc0a4cc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa8040b9d60 tx=0x7fa8040dd040 comp rx=0 tx=0).stop 2024-03-23T23:03:36.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.317+0000 7fa81ae00640 1 -- 172.21.15.12:0/3716826017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8141037c0 msgr2=0x7fa814199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.317+0000 7fa81ae00640 1 --2- 172.21.15.12:0/3716826017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8141037c0 0x7fa814199b50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa80402efd0 tx=0x7fa804005d50 comp rx=0 tx=0).stop 2024-03-23T23:03:36.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.317+0000 7fa81ae00640 1 -- 172.21.15.12:0/3716826017 shutdown_connections 2024-03-23T23:03:36.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.317+0000 7fa81ae00640 1 --2- 172.21.15.12:0/3716826017 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa7fc0a2800 0x7fa7fc0a4cc0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.317+0000 7fa81ae00640 1 --2- 172.21.15.12:0/3716826017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa814105370 0x7fa8141a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.317+0000 7fa81ae00640 1 --2- 172.21.15.12:0/3716826017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8141049b0 0x7fa81419a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.317+0000 7fa81ae00640 1 --2- 172.21.15.12:0/3716826017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8141037c0 0x7fa814199b50 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.317+0000 7fa81ae00640 1 -- 172.21.15.12:0/3716826017 >> 172.21.15.12:0/3716826017 conn(0x7fa8140fed50 msgr2=0x7fa814107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:36.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.317+0000 7fa81ae00640 1 -- 172.21.15.12:0/3716826017 shutdown_connections 2024-03-23T23:03:36.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.317+0000 7fa81ae00640 1 -- 172.21.15.12:0/3716826017 wait complete. 2024-03-23T23:03:36.334 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:03:36.334 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-23T23:03:36.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.556+0000 7f6ffb800640 1 -- 172.21.15.12:0/2907507923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ff41028a0 msgr2=0x7f6ff4102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.556+0000 7f6ffb800640 1 --2- 172.21.15.12:0/2907507923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ff41028a0 0x7f6ff4102d20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc009900 tx=0x7f6fdc02bc30 comp rx=0 tx=0).stop 2024-03-23T23:03:36.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.556+0000 7f6ffb800640 1 -- 172.21.15.12:0/2907507923 shutdown_connections 2024-03-23T23:03:36.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.556+0000 7f6ffb800640 1 --2- 172.21.15.12:0/2907507923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ff4103260 0x7f6ff4109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.556+0000 7f6ffb800640 1 --2- 172.21.15.12:0/2907507923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ff41028a0 0x7f6ff4102d20 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.556+0000 7f6ffb800640 1 --2- 172.21.15.12:0/2907507923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ff41016b0 0x7f6ff4101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.556+0000 7f6ffb800640 1 -- 172.21.15.12:0/2907507923 >> 172.21.15.12:0/2907507923 conn(0x7f6ff40fcc40 msgr2=0x7f6ff40ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:36.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.556+0000 7f6ffb800640 1 -- 172.21.15.12:0/2907507923 shutdown_connections 2024-03-23T23:03:36.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.556+0000 7f6ffb800640 1 -- 172.21.15.12:0/2907507923 wait complete. 2024-03-23T23:03:36.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.557+0000 7f6ffb800640 1 Processor -- start 2024-03-23T23:03:36.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.557+0000 7f6ffb800640 1 -- start start 2024-03-23T23:03:36.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.558+0000 7f6ffb800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ff41016b0 0x7f6ff4199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.558+0000 7f6ffb800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ff41028a0 0x7f6ff419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.558+0000 7f6ffb800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ff4103260 0x7f6ff41a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.558+0000 7f6ffb800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff410d360 con 0x7f6ff41016b0 2024-03-23T23:03:36.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.558+0000 7f6ffb800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff410d1e0 con 0x7f6ff4103260 2024-03-23T23:03:36.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.558+0000 7f6ffb800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff410d4e0 con 0x7f6ff41028a0 2024-03-23T23:03:36.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.558+0000 7f6ff8e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ff41016b0 0x7f6ff4199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.558+0000 7f6febe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ff41028a0 0x7f6ff419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.558+0000 7f6febe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ff41028a0 0x7f6ff419a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35382/0 (socket says 172.21.15.12:35382) 2024-03-23T23:03:36.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.558+0000 7f6ff8e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ff41016b0 0x7f6ff4199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48504/0 (socket says 172.21.15.12:48504) 2024-03-23T23:03:36.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.558+0000 7f6febe00640 1 -- 172.21.15.12:0/1452151845 learned_addr learned my addr 172.21.15.12:0/1452151845 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:36.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.558+0000 7f6ff9800640 1 --2- 172.21.15.12:0/1452151845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ff4103260 0x7f6ff41a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.558+0000 7f6febe00640 1 -- 172.21.15.12:0/1452151845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ff4103260 msgr2=0x7f6ff41a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.558+0000 7f6febe00640 1 --2- 172.21.15.12:0/1452151845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ff4103260 0x7f6ff41a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.558+0000 7f6febe00640 1 -- 172.21.15.12:0/1452151845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ff41016b0 msgr2=0x7f6ff4199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.558+0000 7f6febe00640 1 --2- 172.21.15.12:0/1452151845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ff41016b0 0x7f6ff4199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.559+0000 7f6febe00640 1 -- 172.21.15.12:0/1452151845 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff41a1830 con 0x7f6ff41028a0 2024-03-23T23:03:36.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.559+0000 7f6ff8e00640 1 --2- 172.21.15.12:0/1452151845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ff41016b0 0x7f6ff4199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:36.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.559+0000 7f6febe00640 1 --2- 172.21.15.12:0/1452151845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ff41028a0 0x7f6ff419a160 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc000c00 tx=0x7f6fdc02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:36.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.559+0000 7f6ff2000640 1 -- 172.21.15.12:0/1452151845 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc043020 con 0x7f6ff41028a0 2024-03-23T23:03:36.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.559+0000 7f6ff2000640 1 -- 172.21.15.12:0/1452151845 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6fdc0054e0 con 0x7f6ff41028a0 2024-03-23T23:03:36.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.559+0000 7f6ff2000640 1 -- 172.21.15.12:0/1452151845 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc003bc0 con 0x7f6ff41028a0 2024-03-23T23:03:36.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.559+0000 7f6ffb800640 1 -- 172.21.15.12:0/1452151845 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff41a1ac0 con 0x7f6ff41028a0 2024-03-23T23:03:36.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.559+0000 7f6ffb800640 1 -- 172.21.15.12:0/1452151845 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff41a2060 con 0x7f6ff41028a0 2024-03-23T23:03:36.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.564+0000 7f6ffe44a740 1 Processor -- start 2024-03-23T23:03:36.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.564+0000 7f6ff2000640 1 -- 172.21.15.12:0/1452151845 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6fdc046020 con 0x7f6ff41028a0 2024-03-23T23:03:36.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.564+0000 7f6ffe44a740 1 -- start start 2024-03-23T23:03:36.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.564+0000 7f6ffe44a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55881ce2f810 0x55881ce2fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.564+0000 7f6ffe44a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55881ce309e0 0x55881ce30e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.564+0000 7f6ffe44a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55881ce313a0 0x55881ce37c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.564+0000 7f6ffe44a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55881cd942c0 con 0x55881ce2f810 2024-03-23T23:03:36.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.565+0000 7f6ffe44a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55881cd3cc90 con 0x55881ce309e0 2024-03-23T23:03:36.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.565+0000 7f6ffe44a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55881cd90360 con 0x55881ce313a0 2024-03-23T23:03:36.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.565+0000 7f6ff8e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55881ce309e0 0x55881ce30e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.565+0000 7f6ff8e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55881ce309e0 0x55881ce30e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50534/0 (socket says 172.21.15.12:50534) 2024-03-23T23:03:36.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.565+0000 7f6ff8e00640 1 -- 172.21.15.12:0/540135839 learned_addr learned my addr 172.21.15.12:0/540135839 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:36.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.565+0000 7f6ff9800640 1 --2- 172.21.15.12:0/540135839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55881ce2f810 0x55881ce2fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.565+0000 7f6febe00640 1 --2- 172.21.15.12:0/540135839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55881ce313a0 0x55881ce37c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.565+0000 7f6ff8e00640 1 -- 172.21.15.12:0/540135839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55881ce313a0 msgr2=0x55881ce37c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:36.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.565+0000 7f6ff8e00640 1 --2- 172.21.15.12:0/540135839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55881ce313a0 0x55881ce37c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.565+0000 7f6ff8e00640 1 -- 172.21.15.12:0/540135839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55881ce2f810 msgr2=0x55881ce2fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:36.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.565+0000 7f6ff8e00640 1 --2- 172.21.15.12:0/540135839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55881ce2f810 0x55881ce2fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.565+0000 7f6ff8e00640 1 -- 172.21.15.12:0/540135839 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55881cd61080 con 0x55881ce309e0 2024-03-23T23:03:36.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.566+0000 7f6ff8e00640 1 --2- 172.21.15.12:0/540135839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55881ce309e0 0x55881ce30e60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6fe40114f0 tx=0x7f6fe40119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71bd2f02afbf9b22 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:36.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.566+0000 7f6ff2000640 1 --2- 172.21.15.12:0/1452151845 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6fd80a2880 0x7f6fd80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.566+0000 7f6fea000640 1 -- 172.21.15.12:0/540135839 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4004310 con 0x55881ce309e0 2024-03-23T23:03:36.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.566+0000 7f6ff2000640 1 -- 172.21.15.12:0/1452151845 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f6fdc0f0af0 con 0x7f6ff41028a0 2024-03-23T23:03:36.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.566+0000 7f6fea000640 1 -- 172.21.15.12:0/540135839 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6fe40044b0 con 0x55881ce309e0 2024-03-23T23:03:36.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.566+0000 7f6fea000640 1 -- 172.21.15.12:0/540135839 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe40035f0 con 0x55881ce309e0 2024-03-23T23:03:36.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.566+0000 7f6ff9800640 1 --2- 172.21.15.12:0/1452151845 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6fd80a2880 0x7f6fd80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.566+0000 7f6ffe44a740 1 -- 172.21.15.12:0/540135839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55881ce309e0 msgr2=0x55881ce30e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.567+0000 7f6ffe44a740 1 --2- 172.21.15.12:0/540135839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55881ce309e0 0x55881ce30e60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6fe40114f0 tx=0x7f6fe40119b0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.567+0000 7f6ff9800640 1 --2- 172.21.15.12:0/1452151845 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6fd80a2880 0x7f6fd80a4d40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x55881ce308e0 tx=0x7f6fec009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:36.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.567+0000 7f6ffe44a740 1 -- 172.21.15.12:0/540135839 shutdown_connections 2024-03-23T23:03:36.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.567+0000 7f6ffe44a740 1 --2- 172.21.15.12:0/540135839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55881ce313a0 0x55881ce37c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.567+0000 7f6ffe44a740 1 --2- 172.21.15.12:0/540135839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55881ce309e0 0x55881ce30e60 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.567+0000 7f6ffe44a740 1 --2- 172.21.15.12:0/540135839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55881ce2f810 0x55881ce2fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.567+0000 7f6ffe44a740 1 -- 172.21.15.12:0/540135839 >> 172.21.15.12:0/540135839 conn(0x55881ce2ac00 msgr2=0x55881ce2b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:36.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.567+0000 7f6ffe44a740 1 -- 172.21.15.12:0/540135839 shutdown_connections 2024-03-23T23:03:36.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.567+0000 7f6ffe44a740 1 -- 172.21.15.12:0/540135839 wait complete. 2024-03-23T23:03:36.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.567+0000 7f6ffe44a740 1 -- start start 2024-03-23T23:03:36.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.567+0000 7f6ffe44a740 1 Processor -- start 2024-03-23T23:03:36.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.568+0000 7f6ffe44a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55881ce2f810 0x55881ce39a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.568+0000 7f6ffe44a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55881ce309e0 0x55881ce39f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.568+0000 7f6ffe44a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55881ce313a0 0x55881ce40f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.568+0000 7f6ffe44a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55881ce383b0 con 0x55881ce2f810 2024-03-23T23:03:36.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.568+0000 7f6ffe44a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55881cd500d0 con 0x55881ce309e0 2024-03-23T23:03:36.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.568+0000 7f6ffe44a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55881ce38530 con 0x55881ce313a0 2024-03-23T23:03:36.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.568+0000 7f6ffe44a740 10 client.0 _ll_register_callbacks cb 0x55881cd54e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:36.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.568+0000 7f6febe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55881ce309e0 0x55881ce39f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.568+0000 7f6ff9800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55881ce313a0 0x55881ce40f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.568+0000 7f6febe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55881ce309e0 0x55881ce39f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50550/0 (socket says 172.21.15.12:50550) 2024-03-23T23:03:36.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.568+0000 7f6febe00640 1 -- 172.21.15.12:0/3100240192 learned_addr learned my addr 172.21.15.12:0/3100240192 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:36.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.568+0000 7f6ff8e00640 1 --2- 172.21.15.12:0/3100240192 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55881ce2f810 0x55881ce39a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.569+0000 7f6febe00640 1 -- 172.21.15.12:0/3100240192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55881ce313a0 msgr2=0x55881ce40f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:36.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.569+0000 7f6febe00640 1 --2- 172.21.15.12:0/3100240192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55881ce313a0 0x55881ce40f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.569+0000 7f6febe00640 1 -- 172.21.15.12:0/3100240192 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55881ce2f810 msgr2=0x55881ce39a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:36.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.569+0000 7f6febe00640 1 --2- 172.21.15.12:0/3100240192 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55881ce2f810 0x55881ce39a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.569+0000 7f6febe00640 1 -- 172.21.15.12:0/3100240192 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55881caa83f0 con 0x55881ce309e0 2024-03-23T23:03:36.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.569+0000 7f6febe00640 1 --2- 172.21.15.12:0/3100240192 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55881ce309e0 0x55881ce39f40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc0bd140 tx=0x7f6fdc0f44b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a47aad2c5216489f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:36.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.569+0000 7f6fe2a00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:03:36.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.569+0000 7f6fe2a00640 1 -- 172.21.15.12:0/3100240192 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc0fa040 con 0x55881ce309e0 2024-03-23T23:03:36.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.569+0000 7f6fe2a00640 1 -- 172.21.15.12:0/3100240192 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f6fdc0043e0 con 0x55881ce309e0 2024-03-23T23:03:36.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.569+0000 7f6ffe44a740 1 -- 172.21.15.12:0/3100240192 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6fe40044b0 con 0x55881ce309e0 2024-03-23T23:03:36.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.569+0000 7f6fe2a00640 1 -- 172.21.15.12:0/3100240192 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6fdc0b9650 con 0x55881ce309e0 2024-03-23T23:03:36.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.569+0000 7f6ffe44a740 10 client.25742 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:36.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.569+0000 7f6ffe44a740 1 -- 172.21.15.12:0/3100240192 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55881cb727d0 con 0x55881ce309e0 2024-03-23T23:03:36.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.570+0000 7f6fe2a00640 1 -- 172.21.15.12:0/3100240192 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6fe40044b0 con 0x55881ce309e0 2024-03-23T23:03:36.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.570+0000 7f6fe2a00640 1 -- 172.21.15.12:0/3100240192 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f6fdc0c1100 con 0x55881ce309e0 2024-03-23T23:03:36.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.570+0000 7f6ffe44a740 10 client.25742 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:36.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.570+0000 7f6ffe44a740 10 client.25742 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:03:36.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.570+0000 7f6ffe44a740 20 client.25742 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:36.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.570+0000 7f6ffe44a740 1 --2- 172.21.15.12:0/3100240192 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55881ce453b0 0x55881ce47830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.570+0000 7f6ffe44a740 4 client.25742 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:03:36.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.570+0000 7f6ffe44a740 1 -- 172.21.15.12:0/3100240192 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55881cda1800 con 0x55881ce453b0 2024-03-23T23:03:36.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.570+0000 7f6ff8e00640 1 --2- 172.21.15.12:0/3100240192 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55881ce453b0 0x55881ce47830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.571+0000 7f6ff8e00640 1 --2- 172.21.15.12:0/3100240192 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55881ce453b0 0x55881ce47830 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=74dc777418dca3a5 server_cookie=d186dc8893d9005e in_seq=0 out_seq=0 2024-03-23T23:03:36.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.571+0000 7f6fe2a00640 10 client.25742 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:03:36.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.572+0000 7f6fe2a00640 1 -- 172.21.15.12:0/3100240192 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55881cda1800 con 0x55881ce453b0 2024-03-23T23:03:36.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.572+0000 7f6fe2a00640 10 client.25742 handle_command_reply: tid=0 2024-03-23T23:03:36.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.572+0000 7f6ffe44a740 1 client.25742 shutdown 2024-03-23T23:03:36.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.572+0000 7f6ffe44a740 1 -- 172.21.15.12:0/3100240192 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55881ce309e0 msgr2=0x55881ce39f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.572+0000 7f6ffe44a740 1 --2- 172.21.15.12:0/3100240192 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55881ce309e0 0x55881ce39f40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc0bd140 tx=0x7f6fdc0f44b0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.572+0000 7f6ffe44a740 1 -- 172.21.15.12:0/3100240192 shutdown_connections 2024-03-23T23:03:36.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.572+0000 7f6ffe44a740 1 --2- 172.21.15.12:0/3100240192 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55881ce453b0 0x55881ce47830 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.572+0000 7f6ffe44a740 1 --2- 172.21.15.12:0/3100240192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55881ce313a0 0x55881ce40f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.572+0000 7f6ffe44a740 1 --2- 172.21.15.12:0/3100240192 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55881ce309e0 0x55881ce39f40 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.572+0000 7f6fe2a00640 0 client.25742 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:03:36.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.572+0000 7f6ffe44a740 1 --2- 172.21.15.12:0/3100240192 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55881ce2f810 0x55881ce39a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.572+0000 7f6ffe44a740 1 -- 172.21.15.12:0/3100240192 >> 172.21.15.12:0/3100240192 conn(0x55881ce2ac00 msgr2=0x55881ce29eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:36.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.573+0000 7f6ffe44a740 1 -- 172.21.15.12:0/3100240192 shutdown_connections 2024-03-23T23:03:36.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.573+0000 7f6ffe44a740 1 -- 172.21.15.12:0/3100240192 wait complete. 2024-03-23T23:03:36.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.573+0000 7f6ffe44a740 20 client.25742 trim_cache size 0 max 16384 2024-03-23T23:03:36.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.587+0000 7f6ffe44a740 1 Processor -- start 2024-03-23T23:03:36.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.587+0000 7f6ffe44a740 1 -- start start 2024-03-23T23:03:36.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.587+0000 7f6ffe44a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55881ce2f810 0x55881ce339e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.587+0000 7f6ffe44a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55881ce309e0 0x55881ce40f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.587+0000 7f6ffe44a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55881ce313a0 0x55881ce33f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.587+0000 7f6ffe44a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff41ad0e0 con 0x55881ce2f810 2024-03-23T23:03:36.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.587+0000 7f6ffe44a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff41aabe0 con 0x55881ce313a0 2024-03-23T23:03:36.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.587+0000 7f6ffe44a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff41a8c80 con 0x55881ce309e0 2024-03-23T23:03:36.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.587+0000 7f6febe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55881ce309e0 0x55881ce40f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.587+0000 7f6ff8e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55881ce2f810 0x55881ce339e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.587+0000 7f6febe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55881ce309e0 0x55881ce40f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35414/0 (socket says 172.21.15.12:35414) 2024-03-23T23:03:36.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.587+0000 7f6febe00640 1 -- 172.21.15.12:0/1341145941 learned_addr learned my addr 172.21.15.12:0/1341145941 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:36.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.588+0000 7f6febe00640 1 -- 172.21.15.12:0/1341145941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55881ce313a0 msgr2=0x55881ce33f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:36.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.588+0000 7f6febe00640 1 --2- 172.21.15.12:0/1341145941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55881ce313a0 0x55881ce33f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.588+0000 7f6febe00640 1 -- 172.21.15.12:0/1341145941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55881ce2f810 msgr2=0x55881ce339e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:36.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.588+0000 7f6febe00640 1 --2- 172.21.15.12:0/1341145941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55881ce2f810 0x55881ce339e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.588+0000 7f6febe00640 1 -- 172.21.15.12:0/1341145941 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55881ce2b970 con 0x55881ce309e0 2024-03-23T23:03:36.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.588+0000 7f6febe00640 1 --2- 172.21.15.12:0/1341145941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55881ce309e0 0x55881ce40f40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc0b9810 tx=0x7f6fdc02bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd67424816c711ee server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:36.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.588+0000 7f6fe9600640 1 -- 172.21.15.12:0/1341145941 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc0bb580 con 0x55881ce309e0 2024-03-23T23:03:36.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.588+0000 7f6fe9600640 1 -- 172.21.15.12:0/1341145941 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6fdc005690 con 0x55881ce309e0 2024-03-23T23:03:36.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.588+0000 7f6fe9600640 1 -- 172.21.15.12:0/1341145941 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc0bcda0 con 0x55881ce309e0 2024-03-23T23:03:36.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.589+0000 7f6ffe44a740 1 -- 172.21.15.12:0/1341145941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55881ce309e0 msgr2=0x55881ce40f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.589+0000 7f6ffe44a740 1 --2- 172.21.15.12:0/1341145941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55881ce309e0 0x55881ce40f40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc0b9810 tx=0x7f6fdc02bc30 comp rx=0 tx=0).stop 2024-03-23T23:03:36.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.589+0000 7f6ffe44a740 1 -- 172.21.15.12:0/1341145941 shutdown_connections 2024-03-23T23:03:36.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.589+0000 7f6ffe44a740 1 --2- 172.21.15.12:0/1341145941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55881ce313a0 0x55881ce33f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.589+0000 7f6ffe44a740 1 --2- 172.21.15.12:0/1341145941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55881ce309e0 0x55881ce40f40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.589+0000 7f6ffe44a740 1 --2- 172.21.15.12:0/1341145941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55881ce2f810 0x55881ce339e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.589+0000 7f6ffe44a740 1 -- 172.21.15.12:0/1341145941 >> 172.21.15.12:0/1341145941 conn(0x55881ce2ac00 msgr2=0x55881ce2e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:36.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.589+0000 7f6ffe44a740 1 -- 172.21.15.12:0/1341145941 shutdown_connections 2024-03-23T23:03:36.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.589+0000 7f6ffe44a740 1 -- 172.21.15.12:0/1341145941 wait complete. 2024-03-23T23:03:36.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.589+0000 7f6ffe44a740 1 -- start start 2024-03-23T23:03:36.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.590+0000 7f6ffe44a740 1 Processor -- start 2024-03-23T23:03:36.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.590+0000 7f6ffe44a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55881ce2f810 0x55881ce40860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.590+0000 7f6ffe44a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55881ce309e0 0x55881ce40da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.590+0000 7f6ffe44a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55881ce313a0 0x55881ce3d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.590+0000 7f6ffe44a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff41a9280 con 0x55881ce309e0 2024-03-23T23:03:36.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.590+0000 7f6ffe44a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff41aa580 con 0x55881ce2f810 2024-03-23T23:03:36.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.590+0000 7f6ffe44a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff41a98c0 con 0x55881ce313a0 2024-03-23T23:03:36.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.590+0000 7f6ffe44a740 10 client.0 _ll_register_callbacks cb 0x7f6fdc0035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:36.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.590+0000 7f6febe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55881ce309e0 0x55881ce40da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.590+0000 7f6febe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55881ce309e0 0x55881ce40da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48546/0 (socket says 172.21.15.12:48546) 2024-03-23T23:03:36.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.590+0000 7f6febe00640 1 -- 172.21.15.12:0/3324750246 learned_addr learned my addr 172.21.15.12:0/3324750246 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:36.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.590+0000 7f6ff9800640 1 --2- 172.21.15.12:0/3324750246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55881ce313a0 0x55881ce3d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.591+0000 7f6ff8e00640 1 --2- 172.21.15.12:0/3324750246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55881ce2f810 0x55881ce40860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.591+0000 7f6febe00640 1 -- 172.21.15.12:0/3324750246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55881ce313a0 msgr2=0x55881ce3d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:36.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.591+0000 7f6febe00640 1 --2- 172.21.15.12:0/3324750246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55881ce313a0 0x55881ce3d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.591+0000 7f6febe00640 1 -- 172.21.15.12:0/3324750246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55881ce2f810 msgr2=0x55881ce40860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:36.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.591+0000 7f6febe00640 1 --2- 172.21.15.12:0/3324750246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55881ce2f810 0x55881ce40860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.591+0000 7f6febe00640 1 -- 172.21.15.12:0/3324750246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55881ce3da90 con 0x55881ce309e0 2024-03-23T23:03:36.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.591+0000 7f6ff8e00640 1 --2- 172.21.15.12:0/3324750246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55881ce2f810 0x55881ce40860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:03:36.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.591+0000 7f6febe00640 1 --2- 172.21.15.12:0/3324750246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55881ce309e0 0x55881ce40da0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc004e90 tx=0x7f6fdc009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc64abd2790930c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:36.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.591+0000 7f6fe2a00640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:03:36.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.591+0000 7f6fe2a00640 1 -- 172.21.15.12:0/3324750246 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc0bcf80 con 0x55881ce309e0 2024-03-23T23:03:36.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.591+0000 7f6fe2a00640 1 -- 172.21.15.12:0/3324750246 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f6fdc0d6130 con 0x55881ce309e0 2024-03-23T23:03:36.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.591+0000 7f6ffe44a740 1 -- 172.21.15.12:0/3324750246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55881cd9bb50 con 0x55881ce309e0 2024-03-23T23:03:36.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.591+0000 7f6fe2a00640 1 -- 172.21.15.12:0/3324750246 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6fdc0bd210 con 0x55881ce309e0 2024-03-23T23:03:36.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.591+0000 7f6ffe44a740 10 client.15600 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:36.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.592+0000 7f6ffe44a740 1 -- 172.21.15.12:0/3324750246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55881ce3da90 con 0x55881ce309e0 2024-03-23T23:03:36.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.592+0000 7f6fe2a00640 1 -- 172.21.15.12:0/3324750246 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55881cd9bb50 con 0x55881ce309e0 2024-03-23T23:03:36.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.592+0000 7f6fe2a00640 1 -- 172.21.15.12:0/3324750246 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f6fdc09f080 con 0x55881ce309e0 2024-03-23T23:03:36.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.592+0000 7f6ffe44a740 10 client.15600 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:36.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.592+0000 7f6ffe44a740 10 client.15600 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:03:36.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.592+0000 7f6ffe44a740 20 client.15600 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:36.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.592+0000 7f6ffe44a740 1 --2- 172.21.15.12:0/3324750246 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55881cea1e20 0x55881cea4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.592+0000 7f6ffe44a740 4 client.15600 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:03:36.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.592+0000 7f6ffe44a740 1 -- 172.21.15.12:0/3324750246 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55881cd19500 con 0x55881cea1e20 2024-03-23T23:03:36.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.593+0000 7f6ff8e00640 1 --2- 172.21.15.12:0/3324750246 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55881cea1e20 0x55881cea4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.593+0000 7f6ff8e00640 1 --2- 172.21.15.12:0/3324750246 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55881cea1e20 0x55881cea4210 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2655b9c872d9f814 server_cookie=8ef85517e7563f22 in_seq=0 out_seq=0 2024-03-23T23:03:36.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.593+0000 7f6fe2a00640 10 client.15600 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:03:36.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.593+0000 7f6fe2a00640 1 -- 172.21.15.12:0/3324750246 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55881cd19500 con 0x55881cea1e20 2024-03-23T23:03:36.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.593+0000 7f6fe2a00640 10 client.15600 handle_command_reply: tid=0 2024-03-23T23:03:36.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.593+0000 7f6ffe44a740 1 client.15600 shutdown 2024-03-23T23:03:36.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.594+0000 7f6ffe44a740 1 -- 172.21.15.12:0/3324750246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55881ce309e0 msgr2=0x55881ce40da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.594+0000 7f6ffe44a740 1 --2- 172.21.15.12:0/3324750246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55881ce309e0 0x55881ce40da0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc004e90 tx=0x7f6fdc009260 comp rx=0 tx=0).stop 2024-03-23T23:03:36.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.594+0000 7f6ffe44a740 1 -- 172.21.15.12:0/3324750246 shutdown_connections 2024-03-23T23:03:36.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.594+0000 7f6ffe44a740 1 --2- 172.21.15.12:0/3324750246 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55881cea1e20 0x55881cea4210 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.594+0000 7f6ffe44a740 1 --2- 172.21.15.12:0/3324750246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55881ce313a0 0x55881ce3d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.594+0000 7f6fe2a00640 0 client.15600 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:03:36.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.594+0000 7f6ffe44a740 1 --2- 172.21.15.12:0/3324750246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55881ce309e0 0x55881ce40da0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.594+0000 7f6ffe44a740 1 --2- 172.21.15.12:0/3324750246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55881ce2f810 0x55881ce40860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.594+0000 7f6ffe44a740 1 -- 172.21.15.12:0/3324750246 >> 172.21.15.12:0/3324750246 conn(0x55881ce2ac00 msgr2=0x55881ce29620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:36.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.594+0000 7f6ffe44a740 1 -- 172.21.15.12:0/3324750246 shutdown_connections 2024-03-23T23:03:36.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.594+0000 7f6ffe44a740 1 -- 172.21.15.12:0/3324750246 wait complete. 2024-03-23T23:03:36.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.594+0000 7f6ffe44a740 20 client.15600 trim_cache size 0 max 16384 2024-03-23T23:03:36.598 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:03:36.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.595+0000 7f6ffb800640 1 -- 172.21.15.12:0/1452151845 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6fd80a2880 msgr2=0x7f6fd80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.595+0000 7f6ffb800640 1 --2- 172.21.15.12:0/1452151845 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6fd80a2880 0x7f6fd80a4d40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x55881ce308e0 tx=0x7f6fec009500 comp rx=0 tx=0).stop 2024-03-23T23:03:36.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.595+0000 7f6ffb800640 1 -- 172.21.15.12:0/1452151845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ff41028a0 msgr2=0x7f6ff419a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.595+0000 7f6ffb800640 1 --2- 172.21.15.12:0/1452151845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ff41028a0 0x7f6ff419a160 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc000c00 tx=0x7f6fdc02c650 comp rx=0 tx=0).stop 2024-03-23T23:03:36.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.595+0000 7f6ffb800640 1 -- 172.21.15.12:0/1452151845 shutdown_connections 2024-03-23T23:03:36.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.595+0000 7f6ffb800640 1 --2- 172.21.15.12:0/1452151845 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6fd80a2880 0x7f6fd80a4d40 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.595+0000 7f6ffb800640 1 --2- 172.21.15.12:0/1452151845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ff4103260 0x7f6ff41a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.595+0000 7f6ffb800640 1 --2- 172.21.15.12:0/1452151845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ff41028a0 0x7f6ff419a160 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.595+0000 7f6ffb800640 1 --2- 172.21.15.12:0/1452151845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ff41016b0 0x7f6ff4199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.595+0000 7f6ffb800640 1 -- 172.21.15.12:0/1452151845 >> 172.21.15.12:0/1452151845 conn(0x7f6ff40fcc40 msgr2=0x7f6ff40738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:36.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.595+0000 7f6ffb800640 1 -- 172.21.15.12:0/1452151845 shutdown_connections 2024-03-23T23:03:36.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.595+0000 7f6ffb800640 1 -- 172.21.15.12:0/1452151845 wait complete. 2024-03-23T23:03:36.610 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:36 smithi012 ceph-mon[31695]: pgmap v444: 97 pgs: 97 active+clean; 698 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 4.1 KiB/s wr, 9 op/s 2024-03-23T23:03:36.610 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:36 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1130346103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:03:36.613 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:03:36.613 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-23T23:03:36.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.831+0000 7ff8bdc00640 1 -- 172.21.15.12:0/521743065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8b81038c0 msgr2=0x7ff8b8103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.831+0000 7ff8bdc00640 1 --2- 172.21.15.12:0/521743065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8b81038c0 0x7ff8b8103ce0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8009900 tx=0x7ff8a802bc30 comp rx=0 tx=0).stop 2024-03-23T23:03:36.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.831+0000 7ff8bdc00640 1 -- 172.21.15.12:0/521743065 shutdown_connections 2024-03-23T23:03:36.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.831+0000 7ff8bdc00640 1 --2- 172.21.15.12:0/521743065 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8b8105470 0x7ff8b810bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.831+0000 7ff8bdc00640 1 --2- 172.21.15.12:0/521743065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8b8104ab0 0x7ff8b8104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.832+0000 7ff8bdc00640 1 --2- 172.21.15.12:0/521743065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8b81038c0 0x7ff8b8103ce0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.832+0000 7ff8bdc00640 1 -- 172.21.15.12:0/521743065 >> 172.21.15.12:0/521743065 conn(0x7ff8b80fee50 msgr2=0x7ff8b8101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:36.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.832+0000 7ff8bdc00640 1 -- 172.21.15.12:0/521743065 shutdown_connections 2024-03-23T23:03:36.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.832+0000 7ff8bdc00640 1 -- 172.21.15.12:0/521743065 wait complete. 2024-03-23T23:03:36.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.833+0000 7ff8bdc00640 1 Processor -- start 2024-03-23T23:03:36.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.833+0000 7ff8bdc00640 1 -- start start 2024-03-23T23:03:36.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.833+0000 7ff8bdc00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8b81038c0 0x7ff8b8199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.833+0000 7ff8bdc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8b8104ab0 0x7ff8b819a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.833+0000 7ff8bdc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8b8105470 0x7ff8b81a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.833+0000 7ff8bdc00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b810f560 con 0x7ff8b81038c0 2024-03-23T23:03:36.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.833+0000 7ff8bdc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b810f3e0 con 0x7ff8b8104ab0 2024-03-23T23:03:36.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.833+0000 7ff8bdc00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b810f6e0 con 0x7ff8b8105470 2024-03-23T23:03:36.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.834+0000 7ff8b6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8b81038c0 0x7ff8b8199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.834+0000 7ff8b6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8b81038c0 0x7ff8b8199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48564/0 (socket says 172.21.15.12:48564) 2024-03-23T23:03:36.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.834+0000 7ff8b6a00640 1 -- 172.21.15.12:0/2001242728 learned_addr learned my addr 172.21.15.12:0/2001242728 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:36.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.834+0000 7ff8b6a00640 1 -- 172.21.15.12:0/2001242728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8b8105470 msgr2=0x7ff8b81a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.834+0000 7ff8b6a00640 1 --2- 172.21.15.12:0/2001242728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8b8105470 0x7ff8b81a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.834+0000 7ff8b6a00640 1 -- 172.21.15.12:0/2001242728 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8b8104ab0 msgr2=0x7ff8b819a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.834+0000 7ff8b6a00640 1 --2- 172.21.15.12:0/2001242728 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8b8104ab0 0x7ff8b819a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.834+0000 7ff8b6a00640 1 -- 172.21.15.12:0/2001242728 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b81a17f0 con 0x7ff8b81038c0 2024-03-23T23:03:36.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.834+0000 7ff8b6a00640 1 --2- 172.21.15.12:0/2001242728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8b81038c0 0x7ff8b8199bd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8000c00 tx=0x7ff8a802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:36.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.835+0000 7ff8af400640 1 -- 172.21.15.12:0/2001242728 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a8043020 con 0x7ff8b81038c0 2024-03-23T23:03:36.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.835+0000 7ff8af400640 1 -- 172.21.15.12:0/2001242728 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8a80054e0 con 0x7ff8b81038c0 2024-03-23T23:03:36.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.835+0000 7ff8bdc00640 1 -- 172.21.15.12:0/2001242728 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b81a1a80 con 0x7ff8b81038c0 2024-03-23T23:03:36.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.835+0000 7ff8af400640 1 -- 172.21.15.12:0/2001242728 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a8003ce0 con 0x7ff8b81038c0 2024-03-23T23:03:36.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.835+0000 7ff8bdc00640 1 -- 172.21.15.12:0/2001242728 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8b81a1fa0 con 0x7ff8b81038c0 2024-03-23T23:03:36.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.838+0000 7ff8c0884740 1 Processor -- start 2024-03-23T23:03:36.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.838+0000 7ff8af400640 1 -- 172.21.15.12:0/2001242728 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff8a8046020 con 0x7ff8b81038c0 2024-03-23T23:03:36.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.838+0000 7ff8c0884740 1 -- start start 2024-03-23T23:03:36.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.839+0000 7ff8c0884740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558437488810 0x558437488c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.839+0000 7ff8c0884740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584374899e0 0x558437489e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.839+0000 7ff8c0884740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55843748a3a0 0x558437490c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.839+0000 7ff8c0884740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584373ed2c0 con 0x55843748a3a0 2024-03-23T23:03:36.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.839+0000 7ff8c0884740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x558437395c90 con 0x5584374899e0 2024-03-23T23:03:36.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.839+0000 7ff8c0884740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584373e9360 con 0x558437488810 2024-03-23T23:03:36.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.839+0000 7ff8b6a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584374899e0 0x558437489e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.839+0000 7ff8b6a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584374899e0 0x558437489e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50614/0 (socket says 172.21.15.12:50614) 2024-03-23T23:03:36.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.839+0000 7ff8b6a00640 1 -- 172.21.15.12:0/1841000086 learned_addr learned my addr 172.21.15.12:0/1841000086 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:36.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.839+0000 7ff8afe00640 1 --2- 172.21.15.12:0/1841000086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55843748a3a0 0x558437490c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.839+0000 7ff8b6a00640 1 -- 172.21.15.12:0/1841000086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558437488810 msgr2=0x558437488c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:36.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.839+0000 7ff8b6a00640 1 --2- 172.21.15.12:0/1841000086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558437488810 0x558437488c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.839+0000 7ff8b6a00640 1 -- 172.21.15.12:0/1841000086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55843748a3a0 msgr2=0x558437490c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:36.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.839+0000 7ff8b6a00640 1 --2- 172.21.15.12:0/1841000086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55843748a3a0 0x558437490c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.839+0000 7ff8b6a00640 1 -- 172.21.15.12:0/1841000086 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5584373ba080 con 0x5584374899e0 2024-03-23T23:03:36.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.840+0000 7ff8b6a00640 1 --2- 172.21.15.12:0/1841000086 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584374899e0 0x558437489e60 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff8a80f4ce0 tx=0x7ff8a80f4dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=450c62824bfe4d00 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:36.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.840+0000 7ff8af400640 1 --2- 172.21.15.12:0/2001242728 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff89c0a2800 0x7ff89c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.840+0000 7ff8a7e00640 1 -- 172.21.15.12:0/1841000086 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a80b9930 con 0x5584374899e0 2024-03-23T23:03:36.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.840+0000 7ff8a7e00640 1 -- 172.21.15.12:0/1841000086 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8a80fe070 con 0x5584374899e0 2024-03-23T23:03:36.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.840+0000 7ff8a7e00640 1 -- 172.21.15.12:0/1841000086 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a80bce10 con 0x5584374899e0 2024-03-23T23:03:36.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.840+0000 7ff8b7400640 1 --2- 172.21.15.12:0/2001242728 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff89c0a2800 0x7ff89c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.840+0000 7ff8af400640 1 -- 172.21.15.12:0/2001242728 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff8a80f0d40 con 0x7ff8b81038c0 2024-03-23T23:03:36.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.841+0000 7ff8c0884740 1 -- 172.21.15.12:0/1841000086 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584374899e0 msgr2=0x558437489e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.841+0000 7ff8c0884740 1 --2- 172.21.15.12:0/1841000086 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584374899e0 0x558437489e60 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff8a80f4ce0 tx=0x7ff8a80f4dd0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.841+0000 7ff8b7400640 1 --2- 172.21.15.12:0/2001242728 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff89c0a2800 0x7ff89c0a4cc0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x5584374898e0 tx=0x7ff8b00093c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:36.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.841+0000 7ff8c0884740 1 -- 172.21.15.12:0/1841000086 shutdown_connections 2024-03-23T23:03:36.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.841+0000 7ff8c0884740 1 --2- 172.21.15.12:0/1841000086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55843748a3a0 0x558437490c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.841+0000 7ff8c0884740 1 --2- 172.21.15.12:0/1841000086 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584374899e0 0x558437489e60 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.841+0000 7ff8c0884740 1 --2- 172.21.15.12:0/1841000086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558437488810 0x558437488c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.841+0000 7ff8c0884740 1 -- 172.21.15.12:0/1841000086 >> 172.21.15.12:0/1841000086 conn(0x558437483c00 msgr2=0x558437484000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:36.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.841+0000 7ff8c0884740 1 -- 172.21.15.12:0/1841000086 shutdown_connections 2024-03-23T23:03:36.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.841+0000 7ff8c0884740 1 -- 172.21.15.12:0/1841000086 wait complete. 2024-03-23T23:03:36.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.842+0000 7ff8c0884740 1 -- start start 2024-03-23T23:03:36.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.842+0000 7ff8c0884740 1 Processor -- start 2024-03-23T23:03:36.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.842+0000 7ff8c0884740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558437488810 0x558437492780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.842+0000 7ff8c0884740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584374899e0 0x558437492cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.842+0000 7ff8c0884740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843748a3a0 0x558437499d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.842+0000 7ff8c0884740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x558437491170 con 0x5584374899e0 2024-03-23T23:03:36.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.842+0000 7ff8c0884740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584373a90d0 con 0x55843748a3a0 2024-03-23T23:03:36.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.842+0000 7ff8c0884740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584374912f0 con 0x558437488810 2024-03-23T23:03:36.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.842+0000 7ff8c0884740 10 client.0 _ll_register_callbacks cb 0x5584373ade80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:36.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.843+0000 7ff8afe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584374899e0 0x558437492cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.843+0000 7ff8afe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584374899e0 0x558437492cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48576/0 (socket says 172.21.15.12:48576) 2024-03-23T23:03:36.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.843+0000 7ff8afe00640 1 -- 172.21.15.12:0/56392479 learned_addr learned my addr 172.21.15.12:0/56392479 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:36.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.843+0000 7ff8afe00640 1 -- 172.21.15.12:0/56392479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558437488810 msgr2=0x558437492780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:36.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.843+0000 7ff8afe00640 1 --2- 172.21.15.12:0/56392479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558437488810 0x558437492780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.843+0000 7ff8afe00640 1 -- 172.21.15.12:0/56392479 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843748a3a0 msgr2=0x558437499d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:36.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.843+0000 7ff8afe00640 1 --2- 172.21.15.12:0/56392479 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843748a3a0 0x558437499d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.843+0000 7ff8afe00640 1 -- 172.21.15.12:0/56392479 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5584371013f0 con 0x5584374899e0 2024-03-23T23:03:36.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.843+0000 7ff8afe00640 1 --2- 172.21.15.12:0/56392479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584374899e0 0x558437492cc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff8a00116f0 tx=0x7ff8a0011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa1b330396fa4530 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:36.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.843+0000 7ff8a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:03:36.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.844+0000 7ff8a4c00640 1 -- 172.21.15.12:0/56392479 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a0004460 con 0x5584374899e0 2024-03-23T23:03:36.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.844+0000 7ff8a4c00640 1 -- 172.21.15.12:0/56392479 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff8a000f8c0 con 0x5584374899e0 2024-03-23T23:03:36.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.844+0000 7ff8c0884740 1 -- 172.21.15.12:0/56392479 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff8a80fe070 con 0x5584374899e0 2024-03-23T23:03:36.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.844+0000 7ff8a4c00640 1 -- 172.21.15.12:0/56392479 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8a000fac0 con 0x5584374899e0 2024-03-23T23:03:36.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.844+0000 7ff8c0884740 10 client.15612 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:36.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.844+0000 7ff8c0884740 1 -- 172.21.15.12:0/56392479 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5584371cb7d0 con 0x5584374899e0 2024-03-23T23:03:36.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.844+0000 7ff8a4c00640 1 -- 172.21.15.12:0/56392479 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8a80fe070 con 0x5584374899e0 2024-03-23T23:03:36.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.844+0000 7ff8a4c00640 1 -- 172.21.15.12:0/56392479 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ff8a0018070 con 0x5584374899e0 2024-03-23T23:03:36.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.845+0000 7ff8c0884740 10 client.15612 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:36.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.845+0000 7ff8c0884740 10 client.15612 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:36.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.845+0000 7ff8c0884740 20 client.15612 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:36.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.845+0000 7ff8c0884740 1 --2- 172.21.15.12:0/56392479 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55843749e270 0x5584374a0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.845+0000 7ff8c0884740 4 client.15612 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:03:36.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.845+0000 7ff8c0884740 1 -- 172.21.15.12:0/56392479 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5584373fa800 con 0x55843749e270 2024-03-23T23:03:36.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.845+0000 7ff8b6a00640 1 --2- 172.21.15.12:0/56392479 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55843749e270 0x5584374a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.847+0000 7ff8b6a00640 1 --2- 172.21.15.12:0/56392479 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55843749e270 0x5584374a0710 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa3890b24668fde6 server_cookie=8b3f341097ebd8a5 in_seq=0 out_seq=0 2024-03-23T23:03:36.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.847+0000 7ff8a4c00640 10 client.15612 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:36.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.848+0000 7ff8a4c00640 1 -- 172.21.15.12:0/56392479 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5584373fa800 con 0x55843749e270 2024-03-23T23:03:36.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.848+0000 7ff8a4c00640 10 client.15612 handle_command_reply: tid=0 2024-03-23T23:03:36.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.848+0000 7ff8c0884740 1 client.15612 shutdown 2024-03-23T23:03:36.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.848+0000 7ff8c0884740 1 -- 172.21.15.12:0/56392479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584374899e0 msgr2=0x558437492cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.848+0000 7ff8c0884740 1 --2- 172.21.15.12:0/56392479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584374899e0 0x558437492cc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff8a00116f0 tx=0x7ff8a0011bb0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.849+0000 7ff8c0884740 1 -- 172.21.15.12:0/56392479 shutdown_connections 2024-03-23T23:03:36.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.849+0000 7ff8c0884740 1 --2- 172.21.15.12:0/56392479 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55843749e270 0x5584374a0710 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.849+0000 7ff8c0884740 1 --2- 172.21.15.12:0/56392479 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843748a3a0 0x558437499d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.849+0000 7ff8c0884740 1 --2- 172.21.15.12:0/56392479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584374899e0 0x558437492cc0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.849+0000 7ff8a4c00640 0 client.15612 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:36.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.849+0000 7ff8c0884740 1 --2- 172.21.15.12:0/56392479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558437488810 0x558437492780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.849+0000 7ff8c0884740 1 -- 172.21.15.12:0/56392479 >> 172.21.15.12:0/56392479 conn(0x558437483c00 msgr2=0x558437482eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:36.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.849+0000 7ff8c0884740 1 -- 172.21.15.12:0/56392479 shutdown_connections 2024-03-23T23:03:36.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.849+0000 7ff8c0884740 1 -- 172.21.15.12:0/56392479 wait complete. 2024-03-23T23:03:36.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.849+0000 7ff8c0884740 20 client.15612 trim_cache size 0 max 16384 2024-03-23T23:03:36.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.863+0000 7ff8c0884740 1 Processor -- start 2024-03-23T23:03:36.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.863+0000 7ff8c0884740 1 -- start start 2024-03-23T23:03:36.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.863+0000 7ff8c0884740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558437488810 0x558437487570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.863+0000 7ff8c0884740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584374899e0 0x558437499cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.863+0000 7ff8c0884740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55843748a3a0 0x55843748ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.863+0000 7ff8c0884740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b81ad1c0 con 0x55843748a3a0 2024-03-23T23:03:36.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.863+0000 7ff8c0884740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b81aacc0 con 0x5584374899e0 2024-03-23T23:03:36.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.863+0000 7ff8c0884740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b81a8d60 con 0x558437488810 2024-03-23T23:03:36.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.863+0000 7ff8afe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584374899e0 0x558437499cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.863+0000 7ff8afe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584374899e0 0x558437499cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50632/0 (socket says 172.21.15.12:50632) 2024-03-23T23:03:36.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.863+0000 7ff8afe00640 1 -- 172.21.15.12:0/3143322814 learned_addr learned my addr 172.21.15.12:0/3143322814 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:36.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.863+0000 7ff8b6a00640 1 --2- 172.21.15.12:0/3143322814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558437488810 0x558437487570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.863+0000 7ff8b7400640 1 --2- 172.21.15.12:0/3143322814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55843748a3a0 0x55843748ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.864+0000 7ff8afe00640 1 -- 172.21.15.12:0/3143322814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558437488810 msgr2=0x558437487570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:36.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.864+0000 7ff8afe00640 1 --2- 172.21.15.12:0/3143322814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558437488810 0x558437487570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.864+0000 7ff8afe00640 1 -- 172.21.15.12:0/3143322814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55843748a3a0 msgr2=0x55843748ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:36.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.864+0000 7ff8afe00640 1 --2- 172.21.15.12:0/3143322814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55843748a3a0 0x55843748ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.864+0000 7ff8afe00640 1 -- 172.21.15.12:0/3143322814 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558437484970 con 0x5584374899e0 2024-03-23T23:03:36.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.864+0000 7ff8b6a00640 1 --2- 172.21.15.12:0/3143322814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558437488810 0x558437487570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:36.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.864+0000 7ff8afe00640 1 --2- 172.21.15.12:0/3143322814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584374899e0 0x558437499cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0000c00 tx=0x7ff8a000f9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8205004c24d8119 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:36.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.865+0000 7ff8a7400640 1 -- 172.21.15.12:0/3143322814 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a000fe60 con 0x5584374899e0 2024-03-23T23:03:36.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.865+0000 7ff8a7400640 1 -- 172.21.15.12:0/3143322814 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8a0020030 con 0x5584374899e0 2024-03-23T23:03:36.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.865+0000 7ff8a7400640 1 -- 172.21.15.12:0/3143322814 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a00039c0 con 0x5584374899e0 2024-03-23T23:03:36.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.865+0000 7ff8c0884740 1 -- 172.21.15.12:0/3143322814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584374899e0 msgr2=0x558437499cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.865+0000 7ff8c0884740 1 --2- 172.21.15.12:0/3143322814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584374899e0 0x558437499cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0000c00 tx=0x7ff8a000f9a0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.865+0000 7ff8c0884740 1 -- 172.21.15.12:0/3143322814 shutdown_connections 2024-03-23T23:03:36.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.865+0000 7ff8c0884740 1 --2- 172.21.15.12:0/3143322814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55843748a3a0 0x55843748ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.865+0000 7ff8c0884740 1 --2- 172.21.15.12:0/3143322814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584374899e0 0x558437499cc0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.865+0000 7ff8c0884740 1 --2- 172.21.15.12:0/3143322814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558437488810 0x558437487570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.865+0000 7ff8c0884740 1 -- 172.21.15.12:0/3143322814 >> 172.21.15.12:0/3143322814 conn(0x558437483c00 msgr2=0x558437486e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:36.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.865+0000 7ff8c0884740 1 -- 172.21.15.12:0/3143322814 shutdown_connections 2024-03-23T23:03:36.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.865+0000 7ff8c0884740 1 -- 172.21.15.12:0/3143322814 wait complete. 2024-03-23T23:03:36.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.866+0000 7ff8c0884740 1 -- start start 2024-03-23T23:03:36.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.866+0000 7ff8c0884740 1 Processor -- start 2024-03-23T23:03:36.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.866+0000 7ff8c0884740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558437488810 0x558437495830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.867+0000 7ff8c0884740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584374899e0 0x558437495d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.867+0000 7ff8c0884740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55843748a3a0 0x558437498d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.867+0000 7ff8c0884740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b81a9360 con 0x558437488810 2024-03-23T23:03:36.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.867+0000 7ff8c0884740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b81aa660 con 0x5584374899e0 2024-03-23T23:03:36.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.867+0000 7ff8c0884740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b81a99a0 con 0x55843748a3a0 2024-03-23T23:03:36.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.867+0000 7ff8c0884740 10 client.0 _ll_register_callbacks cb 0x7ff8a0009710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:36.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.867+0000 7ff8afe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584374899e0 0x558437495d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.867+0000 7ff8b6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558437488810 0x558437495830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.867+0000 7ff8afe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584374899e0 0x558437495d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50640/0 (socket says 172.21.15.12:50640) 2024-03-23T23:03:36.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.867+0000 7ff8afe00640 1 -- 172.21.15.12:0/2891009588 learned_addr learned my addr 172.21.15.12:0/2891009588 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:36.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.867+0000 7ff8b7400640 1 --2- 172.21.15.12:0/2891009588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55843748a3a0 0x558437498d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.867+0000 7ff8afe00640 1 -- 172.21.15.12:0/2891009588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55843748a3a0 msgr2=0x558437498d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:36.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.867+0000 7ff8afe00640 1 --2- 172.21.15.12:0/2891009588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55843748a3a0 0x558437498d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.867+0000 7ff8afe00640 1 -- 172.21.15.12:0/2891009588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558437488810 msgr2=0x558437495830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:36.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.867+0000 7ff8afe00640 1 --2- 172.21.15.12:0/2891009588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558437488810 0x558437495830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.867+0000 7ff8afe00640 1 -- 172.21.15.12:0/2891009588 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558437499380 con 0x5584374899e0 2024-03-23T23:03:36.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.867+0000 7ff8b6a00640 1 --2- 172.21.15.12:0/2891009588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558437488810 0x558437495830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:36.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.868+0000 7ff8afe00640 1 --2- 172.21.15.12:0/2891009588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584374899e0 0x558437495d70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff8a001a570 tx=0x7ff8a000f820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bb5fbdc227f0e81 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:36.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.868+0000 7ff8a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:03:36.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.868+0000 7ff8a4c00640 1 -- 172.21.15.12:0/2891009588 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a001a790 con 0x5584374899e0 2024-03-23T23:03:36.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.868+0000 7ff8a4c00640 1 -- 172.21.15.12:0/2891009588 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff8a001a930 con 0x5584374899e0 2024-03-23T23:03:36.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.868+0000 7ff8c0884740 1 -- 172.21.15.12:0/2891009588 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff8a0020030 con 0x5584374899e0 2024-03-23T23:03:36.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.868+0000 7ff8a4c00640 1 -- 172.21.15.12:0/2891009588 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8a001ab30 con 0x5584374899e0 2024-03-23T23:03:36.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.868+0000 7ff8c0884740 10 client.25754 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:36.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.868+0000 7ff8c0884740 1 -- 172.21.15.12:0/2891009588 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5584374fa020 con 0x5584374899e0 2024-03-23T23:03:36.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.868+0000 7ff8a4c00640 1 -- 172.21.15.12:0/2891009588 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8a0020030 con 0x5584374899e0 2024-03-23T23:03:36.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.868+0000 7ff8a4c00640 1 -- 172.21.15.12:0/2891009588 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ff8a0018040 con 0x5584374899e0 2024-03-23T23:03:36.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.869+0000 7ff8c0884740 10 client.25754 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:36.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.869+0000 7ff8c0884740 10 client.25754 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:36.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.869+0000 7ff8c0884740 20 client.25754 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:36.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.869+0000 7ff8c0884740 1 --2- 172.21.15.12:0/2891009588 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5584374fa8e0 0x5584374fcd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:36.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.869+0000 7ff8c0884740 4 client.25754 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:03:36.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.869+0000 7ff8c0884740 1 -- 172.21.15.12:0/2891009588 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5584373f4b50 con 0x5584374fa8e0 2024-03-23T23:03:36.872 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:36 smithi173 ceph-mon[36116]: pgmap v444: 97 pgs: 97 active+clean; 698 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 4.1 KiB/s wr, 9 op/s 2024-03-23T23:03:36.872 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:36 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1130346103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:03:36.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.869+0000 7ff8b6a00640 1 --2- 172.21.15.12:0/2891009588 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5584374fa8e0 0x5584374fcd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:36.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.869+0000 7ff8b6a00640 1 --2- 172.21.15.12:0/2891009588 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5584374fa8e0 0x5584374fcd00 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f4900a90672663f server_cookie=2313cc684552f557 in_seq=0 out_seq=0 2024-03-23T23:03:36.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.869+0000 7ff8a4c00640 10 client.25754 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:36.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.869+0000 7ff8a4c00640 1 -- 172.21.15.12:0/2891009588 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5584373f4b50 con 0x5584374fa8e0 2024-03-23T23:03:36.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.870+0000 7ff8a4c00640 10 client.25754 handle_command_reply: tid=0 2024-03-23T23:03:36.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.870+0000 7ff8c0884740 1 client.25754 shutdown 2024-03-23T23:03:36.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.870+0000 7ff8c0884740 1 -- 172.21.15.12:0/2891009588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584374899e0 msgr2=0x558437495d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.870+0000 7ff8c0884740 1 --2- 172.21.15.12:0/2891009588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584374899e0 0x558437495d70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff8a001a570 tx=0x7ff8a000f820 comp rx=0 tx=0).stop 2024-03-23T23:03:36.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.870+0000 7ff8c0884740 1 -- 172.21.15.12:0/2891009588 shutdown_connections 2024-03-23T23:03:36.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.870+0000 7ff8c0884740 1 --2- 172.21.15.12:0/2891009588 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5584374fa8e0 0x5584374fcd00 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.870+0000 7ff8c0884740 1 --2- 172.21.15.12:0/2891009588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55843748a3a0 0x558437498d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.870+0000 7ff8c0884740 1 --2- 172.21.15.12:0/2891009588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584374899e0 0x558437495d70 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.870+0000 7ff8c0884740 1 --2- 172.21.15.12:0/2891009588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558437488810 0x558437495830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.870+0000 7ff8c0884740 1 -- 172.21.15.12:0/2891009588 >> 172.21.15.12:0/2891009588 conn(0x558437483c00 msgr2=0x5584374876f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:36.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.870+0000 7ff8a4c00640 0 client.25754 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:36.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.870+0000 7ff8c0884740 1 -- 172.21.15.12:0/2891009588 shutdown_connections 2024-03-23T23:03:36.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.870+0000 7ff8c0884740 1 -- 172.21.15.12:0/2891009588 wait complete. 2024-03-23T23:03:36.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.870+0000 7ff8c0884740 20 client.25754 trim_cache size 0 max 16384 2024-03-23T23:03:36.875 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:03:36.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.871+0000 7ff8bdc00640 1 -- 172.21.15.12:0/2001242728 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff89c0a2800 msgr2=0x7ff89c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.871+0000 7ff8bdc00640 1 --2- 172.21.15.12:0/2001242728 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff89c0a2800 0x7ff89c0a4cc0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x5584374898e0 tx=0x7ff8b00093c0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.871+0000 7ff8bdc00640 1 -- 172.21.15.12:0/2001242728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8b81038c0 msgr2=0x7ff8b8199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:36.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.871+0000 7ff8bdc00640 1 --2- 172.21.15.12:0/2001242728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8b81038c0 0x7ff8b8199bd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8000c00 tx=0x7ff8a802c6c0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.871+0000 7ff8bdc00640 1 -- 172.21.15.12:0/2001242728 shutdown_connections 2024-03-23T23:03:36.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.871+0000 7ff8bdc00640 1 --2- 172.21.15.12:0/2001242728 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff89c0a2800 0x7ff89c0a4cc0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.871+0000 7ff8bdc00640 1 --2- 172.21.15.12:0/2001242728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8b8105470 0x7ff8b81a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.871+0000 7ff8bdc00640 1 --2- 172.21.15.12:0/2001242728 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8b8104ab0 0x7ff8b819a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.871+0000 7ff8bdc00640 1 --2- 172.21.15.12:0/2001242728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8b81038c0 0x7ff8b8199bd0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:36.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.871+0000 7ff8bdc00640 1 -- 172.21.15.12:0/2001242728 >> 172.21.15.12:0/2001242728 conn(0x7ff8b80fee50 msgr2=0x7ff8b8107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:36.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.871+0000 7ff8bdc00640 1 -- 172.21.15.12:0/2001242728 shutdown_connections 2024-03-23T23:03:36.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:36.871+0000 7ff8bdc00640 1 -- 172.21.15.12:0/2001242728 wait complete. 2024-03-23T23:03:36.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:36 smithi203 ceph-mon[36693]: pgmap v444: 97 pgs: 97 active+clean; 698 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 4.1 KiB/s wr, 9 op/s 2024-03-23T23:03:36.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:36 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1130346103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:03:37.892 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-23T23:03:37.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:03:37.893 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-23T23:03:38.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.137+0000 7f145c000640 1 -- 172.21.15.12:0/3543171929 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14540772a0 msgr2=0x7f1454077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:38.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.137+0000 7f145c000640 1 --2- 172.21.15.12:0/3543171929 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14540772a0 0x7f1454077700 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1440009860 tx=0x7f144002bb70 comp rx=0 tx=0).stop 2024-03-23T23:03:38.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.137+0000 7f145c000640 1 -- 172.21.15.12:0/3543171929 shutdown_connections 2024-03-23T23:03:38.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.137+0000 7f145c000640 1 --2- 172.21.15.12:0/3543171929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1454077c40 0x7f14540784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.137+0000 7f145c000640 1 --2- 172.21.15.12:0/3543171929 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14540772a0 0x7f1454077700 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.137+0000 7f145c000640 1 --2- 172.21.15.12:0/3543171929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1454076060 0x7f1454076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.137+0000 7f145c000640 1 -- 172.21.15.12:0/3543171929 >> 172.21.15.12:0/3543171929 conn(0x7f14540fee50 msgr2=0x7f1454101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:38.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.138+0000 7f145c000640 1 -- 172.21.15.12:0/3543171929 shutdown_connections 2024-03-23T23:03:38.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.138+0000 7f145c000640 1 -- 172.21.15.12:0/3543171929 wait complete. 2024-03-23T23:03:38.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.139+0000 7f145c000640 1 Processor -- start 2024-03-23T23:03:38.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.139+0000 7f145c000640 1 -- start start 2024-03-23T23:03:38.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.139+0000 7f145c000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1454076060 0x7f145419e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.139+0000 7f145c000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14540772a0 0x7f145419e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.139+0000 7f145c000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1454077c40 0x7f14541a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.139+0000 7f145c000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1454113930 con 0x7f1454077c40 2024-03-23T23:03:38.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.139+0000 7f145c000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14541137b0 con 0x7f14540772a0 2024-03-23T23:03:38.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.139+0000 7f145c000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1454113ab0 con 0x7f1454076060 2024-03-23T23:03:38.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.139+0000 7f145a000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1454077c40 0x7f14541a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.139+0000 7f1458c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14540772a0 0x7f145419e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.139+0000 7f1459600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1454076060 0x7f145419e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.139+0000 7f145a000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1454077c40 0x7f14541a55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48604/0 (socket says 172.21.15.12:48604) 2024-03-23T23:03:38.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.139+0000 7f145a000640 1 -- 172.21.15.12:0/640310962 learned_addr learned my addr 172.21.15.12:0/640310962 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:38.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.140+0000 7f1458c00640 1 -- 172.21.15.12:0/640310962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1454076060 msgr2=0x7f145419e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:38.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.140+0000 7f1458c00640 1 --2- 172.21.15.12:0/640310962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1454076060 0x7f145419e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.140+0000 7f1458c00640 1 -- 172.21.15.12:0/640310962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1454077c40 msgr2=0x7f14541a55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:38.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.140+0000 7f1458c00640 1 --2- 172.21.15.12:0/640310962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1454077c40 0x7f14541a55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.140+0000 7f1458c00640 1 -- 172.21.15.12:0/640310962 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14541a5c30 con 0x7f14540772a0 2024-03-23T23:03:38.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.140+0000 7f145a000640 1 --2- 172.21.15.12:0/640310962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1454077c40 0x7f14541a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:38.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.140+0000 7f1458c00640 1 --2- 172.21.15.12:0/640310962 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14540772a0 0x7f145419e550 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f144002c0b0 tx=0x7f14400042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:38.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.140+0000 7f1452000640 1 -- 172.21.15.12:0/640310962 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1440041020 con 0x7f14540772a0 2024-03-23T23:03:38.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.141+0000 7f1452000640 1 -- 172.21.15.12:0/640310962 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14400323b0 con 0x7f14540772a0 2024-03-23T23:03:38.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.141+0000 7f1452000640 1 -- 172.21.15.12:0/640310962 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1440039040 con 0x7f14540772a0 2024-03-23T23:03:38.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.141+0000 7f145c000640 1 -- 172.21.15.12:0/640310962 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14541a5ec0 con 0x7f14540772a0 2024-03-23T23:03:38.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.141+0000 7f145c000640 1 -- 172.21.15.12:0/640310962 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14541a6330 con 0x7f14540772a0 2024-03-23T23:03:38.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.144+0000 7f1452000640 1 -- 172.21.15.12:0/640310962 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1440044020 con 0x7f14540772a0 2024-03-23T23:03:38.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.144+0000 7f145ed80740 1 Processor -- start 2024-03-23T23:03:38.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.144+0000 7f145ed80740 1 -- start start 2024-03-23T23:03:38.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.144+0000 7f145ed80740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7729fa0a0 0x55b7729fa4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.144+0000 7f145ed80740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7729faa90 0x55b7729ff590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.144+0000 7f145ed80740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b7729ffc00 0x55b772a02090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.144+0000 7f145ed80740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b77295feb0 con 0x55b7729ffc00 2024-03-23T23:03:38.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.144+0000 7f145ed80740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b772908a60 con 0x55b7729fa0a0 2024-03-23T23:03:38.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.144+0000 7f145ed80740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b77295bf50 con 0x55b7729faa90 2024-03-23T23:03:38.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.145+0000 7f145a000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7729fa0a0 0x55b7729fa4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.145+0000 7f145a000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7729fa0a0 0x55b7729fa4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50666/0 (socket says 172.21.15.12:50666) 2024-03-23T23:03:38.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.145+0000 7f145a000640 1 -- 172.21.15.12:0/3296250240 learned_addr learned my addr 172.21.15.12:0/3296250240 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:38.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.145+0000 7f1459600640 1 --2- 172.21.15.12:0/3296250240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7729faa90 0x55b7729ff590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.145+0000 7f1458c00640 1 --2- 172.21.15.12:0/3296250240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b7729ffc00 0x55b772a02090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.145+0000 7f145a000640 1 -- 172.21.15.12:0/3296250240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7729faa90 msgr2=0x55b7729ff590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:38.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.145+0000 7f145a000640 1 --2- 172.21.15.12:0/3296250240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7729faa90 0x55b7729ff590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.145+0000 7f145a000640 1 -- 172.21.15.12:0/3296250240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b7729ffc00 msgr2=0x55b772a02090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:38.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.145+0000 7f145a000640 1 --2- 172.21.15.12:0/3296250240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b7729ffc00 0x55b772a02090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.145+0000 7f145a000640 1 -- 172.21.15.12:0/3296250240 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b77273c7e0 con 0x55b7729fa0a0 2024-03-23T23:03:38.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.145+0000 7f145a000640 1 --2- 172.21.15.12:0/3296250240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7729fa0a0 0x55b7729fa4c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f144c00eaf0 tx=0x7f144c009ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10d038eee95314fc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:38.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.145+0000 7f1446600640 1 -- 172.21.15.12:0/3296250240 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144c00cc60 con 0x55b7729fa0a0 2024-03-23T23:03:38.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.145+0000 7f1446600640 1 -- 172.21.15.12:0/3296250240 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f144c00ce00 con 0x55b7729fa0a0 2024-03-23T23:03:38.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.145+0000 7f1446600640 1 -- 172.21.15.12:0/3296250240 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144c0183f0 con 0x55b7729fa0a0 2024-03-23T23:03:38.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.146+0000 7f1452000640 1 --2- 172.21.15.12:0/640310962 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f143c0a2880 0x7f143c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.146+0000 7f145ed80740 1 -- 172.21.15.12:0/3296250240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7729fa0a0 msgr2=0x55b7729fa4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:38.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.146+0000 7f145ed80740 1 --2- 172.21.15.12:0/3296250240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7729fa0a0 0x55b7729fa4c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f144c00eaf0 tx=0x7f144c009ee0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.146+0000 7f1452000640 1 -- 172.21.15.12:0/640310962 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f14400f02a0 con 0x7f14540772a0 2024-03-23T23:03:38.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.146+0000 7f145ed80740 1 -- 172.21.15.12:0/3296250240 shutdown_connections 2024-03-23T23:03:38.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.146+0000 7f145ed80740 1 --2- 172.21.15.12:0/3296250240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b7729ffc00 0x55b772a02090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.146+0000 7f145ed80740 1 --2- 172.21.15.12:0/3296250240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7729faa90 0x55b7729ff590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.146+0000 7f145ed80740 1 --2- 172.21.15.12:0/3296250240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7729fa0a0 0x55b7729fa4c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.146+0000 7f145ed80740 1 -- 172.21.15.12:0/3296250240 >> 172.21.15.12:0/3296250240 conn(0x55b7729f5610 msgr2=0x55b7729f5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:38.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.146+0000 7f1459600640 1 --2- 172.21.15.12:0/640310962 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f143c0a2880 0x7f143c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.146+0000 7f145ed80740 1 -- 172.21.15.12:0/3296250240 shutdown_connections 2024-03-23T23:03:38.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.146+0000 7f145ed80740 1 -- 172.21.15.12:0/3296250240 wait complete. 2024-03-23T23:03:38.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.147+0000 7f145ed80740 1 -- start start 2024-03-23T23:03:38.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.147+0000 7f145ed80740 1 Processor -- start 2024-03-23T23:03:38.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.147+0000 7f1459600640 1 --2- 172.21.15.12:0/640310962 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f143c0a2880 0x7f143c0a4d40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x55b7729f6580 tx=0x7f144800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:38.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.147+0000 7f145ed80740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7729fa0a0 0x55b772a042c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.147+0000 7f145ed80740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b7729faa90 0x55b772a04800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.148+0000 7f145ed80740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7729ffc00 0x55b772a0b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.148+0000 7f145ed80740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b772a02bb0 con 0x55b7729faa90 2024-03-23T23:03:38.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.148+0000 7f145ed80740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b77291beb0 con 0x55b7729fa0a0 2024-03-23T23:03:38.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.148+0000 7f145ed80740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b772a02d30 con 0x55b7729ffc00 2024-03-23T23:03:38.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.148+0000 7f145ed80740 10 client.0 _ll_register_callbacks cb 0x55b772910150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:38.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.148+0000 7f1459600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7729fa0a0 0x55b772a042c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.148+0000 7f1459600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7729fa0a0 0x55b772a042c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50676/0 (socket says 172.21.15.12:50676) 2024-03-23T23:03:38.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.148+0000 7f1459600640 1 -- 172.21.15.12:0/160207393 learned_addr learned my addr 172.21.15.12:0/160207393 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:38.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.148+0000 7f1458c00640 1 --2- 172.21.15.12:0/160207393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b7729faa90 0x55b772a04800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.148+0000 7f145a000640 1 --2- 172.21.15.12:0/160207393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7729ffc00 0x55b772a0b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.148+0000 7f1459600640 1 -- 172.21.15.12:0/160207393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7729ffc00 msgr2=0x55b772a0b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:38.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.148+0000 7f1459600640 1 --2- 172.21.15.12:0/160207393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7729ffc00 0x55b772a0b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.148+0000 7f1459600640 1 -- 172.21.15.12:0/160207393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b7729faa90 msgr2=0x55b772a04800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:38.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.148+0000 7f1459600640 1 --2- 172.21.15.12:0/160207393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b7729faa90 0x55b772a04800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.148+0000 7f1459600640 1 -- 172.21.15.12:0/160207393 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b77279cf70 con 0x55b7729fa0a0 2024-03-23T23:03:38.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.149+0000 7f1459600640 1 --2- 172.21.15.12:0/160207393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7729fa0a0 0x55b772a042c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1448003c00 tx=0x7f1448004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5574b5116c8ff9d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:38.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.149+0000 7f143aa00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:03:38.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.149+0000 7f143aa00640 1 -- 172.21.15.12:0/160207393 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144802c060 con 0x55b7729fa0a0 2024-03-23T23:03:38.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.149+0000 7f143aa00640 1 -- 172.21.15.12:0/160207393 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1448017030 con 0x55b7729fa0a0 2024-03-23T23:03:38.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.149+0000 7f145ed80740 1 -- 172.21.15.12:0/160207393 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f144c00ce00 con 0x55b7729fa0a0 2024-03-23T23:03:38.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.149+0000 7f143aa00640 1 -- 172.21.15.12:0/160207393 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f144801fd40 con 0x55b7729fa0a0 2024-03-23T23:03:38.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.149+0000 7f145ed80740 10 client.25766 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:38.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.149+0000 7f145ed80740 1 -- 172.21.15.12:0/160207393 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55b772634170 con 0x55b7729fa0a0 2024-03-23T23:03:38.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.149+0000 7f143aa00640 1 -- 172.21.15.12:0/160207393 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f144c00ce00 con 0x55b7729fa0a0 2024-03-23T23:03:38.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.149+0000 7f143aa00640 1 -- 172.21.15.12:0/160207393 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f144801c480 con 0x55b7729fa0a0 2024-03-23T23:03:38.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.150+0000 7f145ed80740 10 client.25766 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:38.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.150+0000 7f145ed80740 10 client.25766 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:38.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.150+0000 7f145ed80740 20 client.25766 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:38.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.150+0000 7f145ed80740 1 --2- 172.21.15.12:0/160207393 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b772a10150 0x55b772a125d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.150+0000 7f145ed80740 4 client.25766 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:03:38.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.150+0000 7f145ed80740 1 -- 172.21.15.12:0/160207393 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b77296d3f0 con 0x55b772a10150 2024-03-23T23:03:38.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.150+0000 7f1458c00640 1 --2- 172.21.15.12:0/160207393 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b772a10150 0x55b772a125d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.150+0000 7f1458c00640 1 --2- 172.21.15.12:0/160207393 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b772a10150 0x55b772a125d0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5e031966c31b477 server_cookie=4df600c56b82a028 in_seq=0 out_seq=0 2024-03-23T23:03:38.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.150+0000 7f143aa00640 10 client.25766 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.151+0000 7f143aa00640 1 -- 172.21.15.12:0/160207393 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b77296d3f0 con 0x55b772a10150 2024-03-23T23:03:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.151+0000 7f143aa00640 10 client.25766 handle_command_reply: tid=0 2024-03-23T23:03:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.151+0000 7f145ed80740 1 client.25766 shutdown 2024-03-23T23:03:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.151+0000 7f145ed80740 1 -- 172.21.15.12:0/160207393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7729fa0a0 msgr2=0x55b772a042c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.151+0000 7f145ed80740 1 --2- 172.21.15.12:0/160207393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7729fa0a0 0x55b772a042c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1448003c00 tx=0x7f1448004730 comp rx=0 tx=0).stop 2024-03-23T23:03:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.151+0000 7f145ed80740 1 -- 172.21.15.12:0/160207393 shutdown_connections 2024-03-23T23:03:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.151+0000 7f145ed80740 1 --2- 172.21.15.12:0/160207393 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b772a10150 0x55b772a125d0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.151+0000 7f145ed80740 1 --2- 172.21.15.12:0/160207393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7729ffc00 0x55b772a0b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.152+0000 7f143aa00640 0 client.25766 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.151+0000 7f145ed80740 1 --2- 172.21.15.12:0/160207393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b7729faa90 0x55b772a04800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.152+0000 7f145ed80740 1 --2- 172.21.15.12:0/160207393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7729fa0a0 0x55b772a042c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.152+0000 7f145ed80740 1 -- 172.21.15.12:0/160207393 >> 172.21.15.12:0/160207393 conn(0x55b7729f5610 msgr2=0x55b7729f4830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.152+0000 7f145ed80740 1 -- 172.21.15.12:0/160207393 shutdown_connections 2024-03-23T23:03:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.152+0000 7f145ed80740 1 -- 172.21.15.12:0/160207393 wait complete. 2024-03-23T23:03:38.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.152+0000 7f145ed80740 20 client.25766 trim_cache size 0 max 16384 2024-03-23T23:03:38.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.169+0000 7f145ed80740 1 Processor -- start 2024-03-23T23:03:38.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.170+0000 7f145ed80740 1 -- start start 2024-03-23T23:03:38.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.170+0000 7f145ed80740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7729fa0a0 0x55b7729fbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.170+0000 7f145ed80740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7729faa90 0x55b772a0b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.170+0000 7f145ed80740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b7729ffc00 0x55b7729fc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.170+0000 7f145ed80740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14541b1440 con 0x55b7729ffc00 2024-03-23T23:03:38.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.170+0000 7f145ed80740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14541aef40 con 0x55b7729faa90 2024-03-23T23:03:38.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.170+0000 7f145ed80740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14541acfe0 con 0x55b7729fa0a0 2024-03-23T23:03:38.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.170+0000 7f1459600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7729fa0a0 0x55b7729fbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.170+0000 7f1458c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7729faa90 0x55b772a0b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.170+0000 7f1459600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7729fa0a0 0x55b7729fbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35526/0 (socket says 172.21.15.12:35526) 2024-03-23T23:03:38.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.170+0000 7f1459600640 1 -- 172.21.15.12:0/632607178 learned_addr learned my addr 172.21.15.12:0/632607178 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:38.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.170+0000 7f145a000640 1 --2- 172.21.15.12:0/632607178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b7729ffc00 0x55b7729fc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.171+0000 7f1459600640 1 -- 172.21.15.12:0/632607178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7729faa90 msgr2=0x55b772a0b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:38.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.171+0000 7f1459600640 1 --2- 172.21.15.12:0/632607178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7729faa90 0x55b772a0b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.171+0000 7f1459600640 1 -- 172.21.15.12:0/632607178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b7729ffc00 msgr2=0x55b7729fc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:38.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.171+0000 7f1459600640 1 --2- 172.21.15.12:0/632607178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b7729ffc00 0x55b7729fc0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.171+0000 7f1459600640 1 -- 172.21.15.12:0/632607178 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b772a02a90 con 0x55b7729fa0a0 2024-03-23T23:03:38.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.171+0000 7f1459600640 1 --2- 172.21.15.12:0/632607178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7729fa0a0 0x55b7729fbba0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1448018040 tx=0x7f144800be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba0c0ae8f1132e61 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:38.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.171+0000 7f1445c00640 1 -- 172.21.15.12:0/632607178 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1448037030 con 0x55b7729fa0a0 2024-03-23T23:03:38.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.171+0000 7f1445c00640 1 -- 172.21.15.12:0/632607178 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f144801f800 con 0x55b7729fa0a0 2024-03-23T23:03:38.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.171+0000 7f1445c00640 1 -- 172.21.15.12:0/632607178 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1448033c60 con 0x55b7729fa0a0 2024-03-23T23:03:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.172+0000 7f145ed80740 1 -- 172.21.15.12:0/632607178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7729fa0a0 msgr2=0x55b7729fbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.172+0000 7f145ed80740 1 --2- 172.21.15.12:0/632607178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7729fa0a0 0x55b7729fbba0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1448018040 tx=0x7f144800be80 comp rx=0 tx=0).stop 2024-03-23T23:03:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.172+0000 7f145ed80740 1 -- 172.21.15.12:0/632607178 shutdown_connections 2024-03-23T23:03:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.172+0000 7f145ed80740 1 --2- 172.21.15.12:0/632607178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b7729ffc00 0x55b7729fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.172+0000 7f145ed80740 1 --2- 172.21.15.12:0/632607178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7729faa90 0x55b772a0b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.172+0000 7f145ed80740 1 --2- 172.21.15.12:0/632607178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7729fa0a0 0x55b7729fbba0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.172+0000 7f145ed80740 1 -- 172.21.15.12:0/632607178 >> 172.21.15.12:0/632607178 conn(0x55b7729f5610 msgr2=0x55b7729fb490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.172+0000 7f145ed80740 1 -- 172.21.15.12:0/632607178 shutdown_connections 2024-03-23T23:03:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.172+0000 7f145ed80740 1 -- 172.21.15.12:0/632607178 wait complete. 2024-03-23T23:03:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.173+0000 7f145ed80740 1 -- start start 2024-03-23T23:03:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.173+0000 7f145ed80740 1 Processor -- start 2024-03-23T23:03:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.173+0000 7f145ed80740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b7729fa0a0 0x55b772a0b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.173+0000 7f145ed80740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7729faa90 0x55b772a09d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.173+0000 7f145ed80740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7729ffc00 0x55b772a0a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.173+0000 7f145ed80740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14541ad5e0 con 0x55b7729fa0a0 2024-03-23T23:03:38.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.173+0000 7f145ed80740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14541ae8e0 con 0x55b7729ffc00 2024-03-23T23:03:38.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.173+0000 7f145ed80740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14541adc20 con 0x55b7729faa90 2024-03-23T23:03:38.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.174+0000 7f145ed80740 10 client.0 _ll_register_callbacks cb 0x7f1448004200 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:38.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.174+0000 7f1458c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7729faa90 0x55b772a09d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.174+0000 7f1459600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b7729fa0a0 0x55b772a0b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.174+0000 7f1458c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7729faa90 0x55b772a09d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35528/0 (socket says 172.21.15.12:35528) 2024-03-23T23:03:38.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.174+0000 7f1458c00640 1 -- 172.21.15.12:0/4245803289 learned_addr learned my addr 172.21.15.12:0/4245803289 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:38.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.174+0000 7f1459600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b7729fa0a0 0x55b772a0b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48638/0 (socket says 172.21.15.12:48638) 2024-03-23T23:03:38.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.174+0000 7f145a000640 1 --2- 172.21.15.12:0/4245803289 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7729ffc00 0x55b772a0a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.174+0000 7f1458c00640 1 -- 172.21.15.12:0/4245803289 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7729ffc00 msgr2=0x55b772a0a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:38.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.174+0000 7f1458c00640 1 --2- 172.21.15.12:0/4245803289 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7729ffc00 0x55b772a0a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.174+0000 7f1458c00640 1 -- 172.21.15.12:0/4245803289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b7729fa0a0 msgr2=0x55b772a0b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:38.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.174+0000 7f1458c00640 1 --2- 172.21.15.12:0/4245803289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b7729fa0a0 0x55b772a0b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.174+0000 7f1458c00640 1 -- 172.21.15.12:0/4245803289 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b772a0ab80 con 0x55b7729faa90 2024-03-23T23:03:38.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.174+0000 7f1459600640 1 --2- 172.21.15.12:0/4245803289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b7729fa0a0 0x55b772a0b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:38.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.174+0000 7f1458c00640 1 --2- 172.21.15.12:0/4245803289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7729faa90 0x55b772a09d90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f144002c180 tx=0x7f1440005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7031aaf9d8dce13 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:38.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.174+0000 7f143aa00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:03:38.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.175+0000 7f143aa00640 1 -- 172.21.15.12:0/4245803289 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14400bc730 con 0x55b7729faa90 2024-03-23T23:03:38.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.175+0000 7f143aa00640 1 -- 172.21.15.12:0/4245803289 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f14400b9260 con 0x55b7729faa90 2024-03-23T23:03:38.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.175+0000 7f145ed80740 1 -- 172.21.15.12:0/4245803289 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f144801f800 con 0x55b7729faa90 2024-03-23T23:03:38.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.175+0000 7f143aa00640 1 -- 172.21.15.12:0/4245803289 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14400bc550 con 0x55b7729faa90 2024-03-23T23:03:38.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.175+0000 7f145ed80740 10 client.25753 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:38.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.175+0000 7f145ed80740 1 -- 172.21.15.12:0/4245803289 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55b772a6cc90 con 0x55b7729faa90 2024-03-23T23:03:38.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.175+0000 7f143aa00640 1 -- 172.21.15.12:0/4245803289 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f144801f800 con 0x55b7729faa90 2024-03-23T23:03:38.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.175+0000 7f143aa00640 1 -- 172.21.15.12:0/4245803289 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f144003c030 con 0x55b7729faa90 2024-03-23T23:03:38.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.175+0000 7f145ed80740 10 client.25753 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:38.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.175+0000 7f145ed80740 10 client.25753 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:38.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.176+0000 7f145ed80740 20 client.25753 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.176+0000 7f145ed80740 1 --2- 172.21.15.12:0/4245803289 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b772a6d520 0x55b772a6f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.176+0000 7f145ed80740 4 client.25753 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-23T23:03:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.176+0000 7f145ed80740 1 -- 172.21.15.12:0/4245803289 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55b772967740 con 0x55b772a6d520 2024-03-23T23:03:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.176+0000 7f1459600640 1 --2- 172.21.15.12:0/4245803289 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b772a6d520 0x55b772a6f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.176+0000 7f1459600640 1 --2- 172.21.15.12:0/4245803289 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b772a6d520 0x55b772a6f940 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3100f41aa730d4b9 server_cookie=379192ac0e232604 in_seq=0 out_seq=0 2024-03-23T23:03:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.176+0000 7f143aa00640 10 client.25753 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.177+0000 7f143aa00640 1 -- 172.21.15.12:0/4245803289 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55b772967740 con 0x55b772a6d520 2024-03-23T23:03:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.177+0000 7f143aa00640 10 client.25753 handle_command_reply: tid=0 2024-03-23T23:03:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.177+0000 7f145ed80740 1 client.25753 shutdown 2024-03-23T23:03:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.178+0000 7f145ed80740 1 -- 172.21.15.12:0/4245803289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7729faa90 msgr2=0x55b772a09d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.178+0000 7f145ed80740 1 --2- 172.21.15.12:0/4245803289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7729faa90 0x55b772a09d90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f144002c180 tx=0x7f1440005f90 comp rx=0 tx=0).stop 2024-03-23T23:03:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.178+0000 7f145ed80740 1 -- 172.21.15.12:0/4245803289 shutdown_connections 2024-03-23T23:03:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.178+0000 7f145ed80740 1 --2- 172.21.15.12:0/4245803289 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b772a6d520 0x55b772a6f940 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.178+0000 7f145ed80740 1 --2- 172.21.15.12:0/4245803289 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7729ffc00 0x55b772a0a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.178+0000 7f145ed80740 1 --2- 172.21.15.12:0/4245803289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7729faa90 0x55b772a09d90 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.178+0000 7f145ed80740 1 --2- 172.21.15.12:0/4245803289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b7729fa0a0 0x55b772a0b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.178+0000 7f145ed80740 1 -- 172.21.15.12:0/4245803289 >> 172.21.15.12:0/4245803289 conn(0x55b7729f5610 msgr2=0x55b7729fbd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.178+0000 7f143aa00640 0 client.25753 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.178+0000 7f145ed80740 1 -- 172.21.15.12:0/4245803289 shutdown_connections 2024-03-23T23:03:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.178+0000 7f145ed80740 1 -- 172.21.15.12:0/4245803289 wait complete. 2024-03-23T23:03:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.178+0000 7f145ed80740 20 client.25753 trim_cache size 0 max 16384 2024-03-23T23:03:38.183 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:03:38.184 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-03-23T23:03:38.184 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "8551d2c0-914e-43df-a336-98e10888bd7c", 2024-03-23T23:03:38.184 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-03-23T23:03:38.184 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:03:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.179+0000 7f145c000640 1 -- 172.21.15.12:0/640310962 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f143c0a2880 msgr2=0x7f143c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.179+0000 7f145c000640 1 --2- 172.21.15.12:0/640310962 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f143c0a2880 0x7f143c0a4d40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x55b7729f6580 tx=0x7f144800b390 comp rx=0 tx=0).stop 2024-03-23T23:03:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.179+0000 7f145c000640 1 -- 172.21.15.12:0/640310962 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14540772a0 msgr2=0x7f145419e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.179+0000 7f145c000640 1 --2- 172.21.15.12:0/640310962 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14540772a0 0x7f145419e550 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f144002c0b0 tx=0x7f14400042f0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.179+0000 7f145c000640 1 -- 172.21.15.12:0/640310962 shutdown_connections 2024-03-23T23:03:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.179+0000 7f145c000640 1 --2- 172.21.15.12:0/640310962 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f143c0a2880 0x7f143c0a4d40 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.179+0000 7f145c000640 1 --2- 172.21.15.12:0/640310962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1454077c40 0x7f14541a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.179+0000 7f145c000640 1 --2- 172.21.15.12:0/640310962 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14540772a0 0x7f145419e550 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.179+0000 7f145c000640 1 --2- 172.21.15.12:0/640310962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1454076060 0x7f145419e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.179+0000 7f145c000640 1 -- 172.21.15.12:0/640310962 >> 172.21.15.12:0/640310962 conn(0x7f14540fee50 msgr2=0x7f14541005b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.179+0000 7f145c000640 1 -- 172.21.15.12:0/640310962 shutdown_connections 2024-03-23T23:03:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.179+0000 7f145c000640 1 -- 172.21.15.12:0/640310962 wait complete. 2024-03-23T23:03:38.197 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:03:38.197 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:03:38.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.419+0000 7f8f16600640 1 -- 172.21.15.12:0/2704398602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f100756e0 msgr2=0x7f8f10073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:38.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.419+0000 7f8f16600640 1 --2- 172.21.15.12:0/2704398602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f100756e0 0x7f8f10073b30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8f00009cd0 tx=0x7f8f0002c0e0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.419+0000 7f8f16600640 1 -- 172.21.15.12:0/2704398602 shutdown_connections 2024-03-23T23:03:38.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.419+0000 7f8f16600640 1 --2- 172.21.15.12:0/2704398602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f1010db70 0x7f8f1010ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.419+0000 7f8f16600640 1 --2- 172.21.15.12:0/2704398602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f10074100 0x7f8f10074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.419+0000 7f8f16600640 1 --2- 172.21.15.12:0/2704398602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f100756e0 0x7f8f10073b30 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.419+0000 7f8f16600640 1 -- 172.21.15.12:0/2704398602 >> 172.21.15.12:0/2704398602 conn(0x7f8f100fcb10 msgr2=0x7f8f100fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.419+0000 7f8f16600640 1 -- 172.21.15.12:0/2704398602 shutdown_connections 2024-03-23T23:03:38.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.419+0000 7f8f16600640 1 -- 172.21.15.12:0/2704398602 wait complete. 2024-03-23T23:03:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.420+0000 7f8f16600640 1 Processor -- start 2024-03-23T23:03:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.420+0000 7f8f16600640 1 -- start start 2024-03-23T23:03:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.420+0000 7f8f16600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f10074100 0x7f8f1019dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.420+0000 7f8f16600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f100756e0 0x7f8f1019e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.420+0000 7f8f16600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f1010db70 0x7f8f101a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.420+0000 7f8f16600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f10113810 con 0x7f8f1010db70 2024-03-23T23:03:38.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.420+0000 7f8f16600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f10113690 con 0x7f8f100756e0 2024-03-23T23:03:38.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.420+0000 7f8f16600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f10113990 con 0x7f8f10074100 2024-03-23T23:03:38.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.420+0000 7f8f15200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f10074100 0x7f8f1019dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.421+0000 7f8f15200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f10074100 0x7f8f1019dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35552/0 (socket says 172.21.15.12:35552) 2024-03-23T23:03:38.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.421+0000 7f8f15200640 1 -- 172.21.15.12:0/3594732146 learned_addr learned my addr 172.21.15.12:0/3594732146 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:38.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.421+0000 7f8f0fe00640 1 --2- 172.21.15.12:0/3594732146 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f100756e0 0x7f8f1019e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.421+0000 7f8f15c00640 1 --2- 172.21.15.12:0/3594732146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f1010db70 0x7f8f101a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.421+0000 7f8f15200640 1 -- 172.21.15.12:0/3594732146 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f100756e0 msgr2=0x7f8f1019e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:38.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.421+0000 7f8f15200640 1 --2- 172.21.15.12:0/3594732146 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f100756e0 0x7f8f1019e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.421+0000 7f8f15200640 1 -- 172.21.15.12:0/3594732146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f1010db70 msgr2=0x7f8f101a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:38.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.421+0000 7f8f15200640 1 --2- 172.21.15.12:0/3594732146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f1010db70 0x7f8f101a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.421+0000 7f8f15200640 1 -- 172.21.15.12:0/3594732146 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f101a5b00 con 0x7f8f10074100 2024-03-23T23:03:38.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.421+0000 7f8f0fe00640 1 --2- 172.21.15.12:0/3594732146 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f100756e0 0x7f8f1019e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:38.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.421+0000 7f8f15c00640 1 --2- 172.21.15.12:0/3594732146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f1010db70 0x7f8f101a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:03:38.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.421+0000 7f8f15200640 1 --2- 172.21.15.12:0/3594732146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f10074100 0x7f8f1019dee0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8f00000f80 tx=0x7f8f00035d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:38.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.422+0000 7f8f0d600640 1 -- 172.21.15.12:0/3594732146 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f00039040 con 0x7f8f10074100 2024-03-23T23:03:38.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.422+0000 7f8f0d600640 1 -- 172.21.15.12:0/3594732146 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f00044070 con 0x7f8f10074100 2024-03-23T23:03:38.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.422+0000 7f8f0d600640 1 -- 172.21.15.12:0/3594732146 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f000044a0 con 0x7f8f10074100 2024-03-23T23:03:38.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.422+0000 7f8f16600640 1 -- 172.21.15.12:0/3594732146 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f101a5d90 con 0x7f8f10074100 2024-03-23T23:03:38.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.422+0000 7f8f16600640 1 -- 172.21.15.12:0/3594732146 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f101a6210 con 0x7f8f10074100 2024-03-23T23:03:38.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.425+0000 7f8f0d600640 1 -- 172.21.15.12:0/3594732146 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8f00004640 con 0x7f8f10074100 2024-03-23T23:03:38.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.425+0000 7f8f1a868740 1 Processor -- start 2024-03-23T23:03:38.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.425+0000 7f8f1a868740 1 -- start start 2024-03-23T23:03:38.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.425+0000 7f8f1a868740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563f90252050 0x563f90252470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.425+0000 7f8f1a868740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f90252a40 0x563f90257540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.425+0000 7f8f1a868740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563f90257bb0 0x563f9025a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.425+0000 7f8f1a868740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f901b7250 con 0x563f90252050 2024-03-23T23:03:38.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.425+0000 7f8f1a868740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f9015fa30 con 0x563f90252a40 2024-03-23T23:03:38.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.425+0000 7f8f1a868740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f901b32f0 con 0x563f90257bb0 2024-03-23T23:03:38.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.426+0000 7f8f15200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f90252a40 0x563f90257540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.426+0000 7f8f15c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563f90252050 0x563f90252470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.426+0000 7f8f0fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563f90257bb0 0x563f9025a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.426+0000 7f8f15200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f90252a40 0x563f90257540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50722/0 (socket says 172.21.15.12:50722) 2024-03-23T23:03:38.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.426+0000 7f8f15200640 1 -- 172.21.15.12:0/1437636315 learned_addr learned my addr 172.21.15.12:0/1437636315 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:38.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.426+0000 7f8f15200640 1 -- 172.21.15.12:0/1437636315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563f90257bb0 msgr2=0x563f9025a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:38.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.426+0000 7f8f15200640 1 --2- 172.21.15.12:0/1437636315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563f90257bb0 0x563f9025a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.426+0000 7f8f15200640 1 -- 172.21.15.12:0/1437636315 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563f90252050 msgr2=0x563f90252470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:38.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.426+0000 7f8f15200640 1 --2- 172.21.15.12:0/1437636315 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563f90252050 0x563f90252470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.426+0000 7f8f15200640 1 -- 172.21.15.12:0/1437636315 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f9014a690 con 0x563f90252a40 2024-03-23T23:03:38.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.426+0000 7f8f15200640 1 --2- 172.21.15.12:0/1437636315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f90252a40 0x563f90257540 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8f000f9190 tx=0x7f8f000f4f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1a98dff1eb052bb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:38.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.427+0000 7f8f0d600640 1 --2- 172.21.15.12:0/3594732146 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8ef40a2880 0x7f8ef40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.427+0000 7f8f06000640 1 -- 172.21.15.12:0/1437636315 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f000f9670 con 0x563f90252a40 2024-03-23T23:03:38.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.427+0000 7f8f06000640 1 -- 172.21.15.12:0/1437636315 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f000fe030 con 0x563f90252a40 2024-03-23T23:03:38.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.427+0000 7f8f06000640 1 -- 172.21.15.12:0/1437636315 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f000f1d90 con 0x563f90252a40 2024-03-23T23:03:38.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.427+0000 7f8f0d600640 1 -- 172.21.15.12:0/3594732146 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8f000f1130 con 0x7f8f10074100 2024-03-23T23:03:38.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.427+0000 7f8f15c00640 1 --2- 172.21.15.12:0/3594732146 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8ef40a2880 0x7f8ef40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.427+0000 7f8f1a868740 1 -- 172.21.15.12:0/1437636315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f90252a40 msgr2=0x563f90257540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:38.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.427+0000 7f8f1a868740 1 --2- 172.21.15.12:0/1437636315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f90252a40 0x563f90257540 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8f000f9190 tx=0x7f8f000f4f90 comp rx=0 tx=0).stop 2024-03-23T23:03:38.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.428+0000 7f8f1a868740 1 -- 172.21.15.12:0/1437636315 shutdown_connections 2024-03-23T23:03:38.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.428+0000 7f8f1a868740 1 --2- 172.21.15.12:0/1437636315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563f90257bb0 0x563f9025a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.428+0000 7f8f1a868740 1 --2- 172.21.15.12:0/1437636315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f90252a40 0x563f90257540 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.428+0000 7f8f1a868740 1 --2- 172.21.15.12:0/1437636315 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563f90252050 0x563f90252470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.428+0000 7f8f1a868740 1 -- 172.21.15.12:0/1437636315 >> 172.21.15.12:0/1437636315 conn(0x563f9021a200 msgr2=0x563f9024d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:38.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.428+0000 7f8f1a868740 1 -- 172.21.15.12:0/1437636315 shutdown_connections 2024-03-23T23:03:38.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.428+0000 7f8f15c00640 1 --2- 172.21.15.12:0/3594732146 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8ef40a2880 0x7f8ef40a4d40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x563f902532a0 tx=0x7f8f08005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:38.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.428+0000 7f8f1a868740 1 -- 172.21.15.12:0/1437636315 wait complete. 2024-03-23T23:03:38.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.428+0000 7f8f1a868740 1 -- start start 2024-03-23T23:03:38.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.429+0000 7f8f1a868740 1 Processor -- start 2024-03-23T23:03:38.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.429+0000 7f8f1a868740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563f90252050 0x563f9025bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.429+0000 7f8f1a868740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563f90252a40 0x563f9025c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.429+0000 7f8f1a868740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f90257bb0 0x563f90263300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.429+0000 7f8f1a868740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f9025a920 con 0x563f90252a40 2024-03-23T23:03:38.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.429+0000 7f8f1a868740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f90172e80 con 0x563f90257bb0 2024-03-23T23:03:38.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.429+0000 7f8f1a868740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f9025aaa0 con 0x563f90252050 2024-03-23T23:03:38.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.429+0000 7f8f1a868740 10 client.0 _ll_register_callbacks cb 0x563f90185130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:38.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.429+0000 7f8f15200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563f90252050 0x563f9025bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.429+0000 7f8f15c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f90257bb0 0x563f90263300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.429+0000 7f8f0fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563f90252a40 0x563f9025c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.429+0000 7f8f15200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563f90252050 0x563f9025bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35570/0 (socket says 172.21.15.12:35570) 2024-03-23T23:03:38.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.429+0000 7f8f15200640 1 -- 172.21.15.12:0/3566488124 learned_addr learned my addr 172.21.15.12:0/3566488124 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:38.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.430+0000 7f8f15200640 1 -- 172.21.15.12:0/3566488124 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f90257bb0 msgr2=0x563f90263300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:38.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.430+0000 7f8f15200640 1 --2- 172.21.15.12:0/3566488124 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f90257bb0 0x563f90263300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.430+0000 7f8f15200640 1 -- 172.21.15.12:0/3566488124 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563f90252a40 msgr2=0x563f9025c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:38.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.430+0000 7f8f15200640 1 --2- 172.21.15.12:0/3566488124 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563f90252a40 0x563f9025c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.430+0000 7f8f15200640 1 -- 172.21.15.12:0/3566488124 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563f8ff957d0 con 0x563f90252050 2024-03-23T23:03:38.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.430+0000 7f8f15200640 1 --2- 172.21.15.12:0/3566488124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563f90252050 0x563f9025bd80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8f00009430 tx=0x7f8f0002c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=766bd092e6e74535 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:38.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.430+0000 7f8efaa00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:03:38.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.430+0000 7f8efaa00640 1 -- 172.21.15.12:0/3566488124 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f000b9d20 con 0x563f90252050 2024-03-23T23:03:38.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.430+0000 7f8efaa00640 1 -- 172.21.15.12:0/3566488124 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8f000eecf0 con 0x563f90252050 2024-03-23T23:03:38.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.430+0000 7f8f1a868740 1 -- 172.21.15.12:0/3566488124 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8f000fe030 con 0x563f90252050 2024-03-23T23:03:38.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.431+0000 7f8efaa00640 1 -- 172.21.15.12:0/3566488124 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f000bd3e0 con 0x563f90252050 2024-03-23T23:03:38.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.431+0000 7f8f1a868740 10 client.25765 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:38.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.431+0000 7f8f1a868740 1 -- 172.21.15.12:0/3566488124 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563f8fecb3f0 con 0x563f90252050 2024-03-23T23:03:38.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.431+0000 7f8efaa00640 1 -- 172.21.15.12:0/3566488124 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f000fe030 con 0x563f90252050 2024-03-23T23:03:38.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.431+0000 7f8efaa00640 1 -- 172.21.15.12:0/3566488124 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8f0003c030 con 0x563f90252050 2024-03-23T23:03:38.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.431+0000 7f8f1a868740 10 client.25765 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:38.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.431+0000 7f8f1a868740 10 client.25765 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:38.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.431+0000 7f8f1a868740 20 client.25765 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:38.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.431+0000 7f8f1a868740 1 --2- 172.21.15.12:0/3566488124 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563f90267900 0x563f90269d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.431+0000 7f8f1a868740 4 client.25765 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:03:38.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.431+0000 7f8f1a868740 1 -- 172.21.15.12:0/3566488124 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563f901c4790 con 0x563f90267900 2024-03-23T23:03:38.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.431+0000 7f8f0fe00640 1 --2- 172.21.15.12:0/3566488124 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563f90267900 0x563f90269d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.432+0000 7f8f0fe00640 1 --2- 172.21.15.12:0/3566488124 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563f90267900 0x563f90269d10 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6f8dea83de35f15 server_cookie=ec7e12235fbd2d7b in_seq=0 out_seq=0 2024-03-23T23:03:38.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.432+0000 7f8efaa00640 10 client.25765 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:38.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.433+0000 7f8efaa00640 1 -- 172.21.15.12:0/3566488124 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563f901c4790 con 0x563f90267900 2024-03-23T23:03:38.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.433+0000 7f8efaa00640 10 client.25765 handle_command_reply: tid=0 2024-03-23T23:03:38.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.433+0000 7f8f1a868740 1 client.25765 shutdown 2024-03-23T23:03:38.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.433+0000 7f8f1a868740 1 -- 172.21.15.12:0/3566488124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563f90252050 msgr2=0x563f9025bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:38.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.433+0000 7f8f1a868740 1 --2- 172.21.15.12:0/3566488124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563f90252050 0x563f9025bd80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8f00009430 tx=0x7f8f0002c900 comp rx=0 tx=0).stop 2024-03-23T23:03:38.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.433+0000 7f8f1a868740 1 -- 172.21.15.12:0/3566488124 shutdown_connections 2024-03-23T23:03:38.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.433+0000 7f8f1a868740 1 --2- 172.21.15.12:0/3566488124 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563f90267900 0x563f90269d10 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.433+0000 7f8f1a868740 1 --2- 172.21.15.12:0/3566488124 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f90257bb0 0x563f90263300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.433+0000 7f8f1a868740 1 --2- 172.21.15.12:0/3566488124 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563f90252a40 0x563f9025c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.433+0000 7f8efaa00640 0 client.25765 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:38.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.433+0000 7f8f1a868740 1 --2- 172.21.15.12:0/3566488124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563f90252050 0x563f9025bd80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.433+0000 7f8f1a868740 1 -- 172.21.15.12:0/3566488124 >> 172.21.15.12:0/3566488124 conn(0x563f9021a200 msgr2=0x563f9024cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:38.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.433+0000 7f8f1a868740 1 -- 172.21.15.12:0/3566488124 shutdown_connections 2024-03-23T23:03:38.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.433+0000 7f8f1a868740 1 -- 172.21.15.12:0/3566488124 wait complete. 2024-03-23T23:03:38.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.434+0000 7f8f1a868740 20 client.25765 trim_cache size 0 max 16384 2024-03-23T23:03:38.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.451+0000 7f8f1a868740 1 Processor -- start 2024-03-23T23:03:38.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.451+0000 7f8f1a868740 1 -- start start 2024-03-23T23:03:38.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.452+0000 7f8f1a868740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563f90252050 0x563f90251490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.452+0000 7f8f1a868740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f90252a40 0x563f90263290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.452+0000 7f8f1a868740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563f90257bb0 0x563f90253bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.452+0000 7f8f1a868740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f101b11f0 con 0x563f90257bb0 2024-03-23T23:03:38.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.452+0000 7f8f1a868740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f101aecf0 con 0x563f90252a40 2024-03-23T23:03:38.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.452+0000 7f8f1a868740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f101acd90 con 0x563f90252050 2024-03-23T23:03:38.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.452+0000 7f8f15c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563f90257bb0 0x563f90253bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.452+0000 7f8f15c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563f90257bb0 0x563f90253bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48678/0 (socket says 172.21.15.12:48678) 2024-03-23T23:03:38.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.452+0000 7f8f15c00640 1 -- 172.21.15.12:0/1038794531 learned_addr learned my addr 172.21.15.12:0/1038794531 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:38.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.452+0000 7f8f15200640 1 --2- 172.21.15.12:0/1038794531 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563f90252050 0x563f90251490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.452+0000 7f8f0fe00640 1 --2- 172.21.15.12:0/1038794531 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f90252a40 0x563f90263290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.453+0000 7f8f15c00640 1 -- 172.21.15.12:0/1038794531 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563f90252050 msgr2=0x563f90251490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:38.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.453+0000 7f8f15c00640 1 --2- 172.21.15.12:0/1038794531 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563f90252050 0x563f90251490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.453+0000 7f8f15c00640 1 -- 172.21.15.12:0/1038794531 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f90252a40 msgr2=0x563f90263290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:38.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.453+0000 7f8f15c00640 1 --2- 172.21.15.12:0/1038794531 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f90252a40 0x563f90263290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.453+0000 7f8f15c00640 1 -- 172.21.15.12:0/1038794531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f8fe8b0f0 con 0x563f90257bb0 2024-03-23T23:03:38.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.453+0000 7f8f15200640 1 --2- 172.21.15.12:0/1038794531 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563f90252050 0x563f90251490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:38.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.453+0000 7f8f0fe00640 1 --2- 172.21.15.12:0/1038794531 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f90252a40 0x563f90263290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:38.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.453+0000 7f8f15c00640 1 --2- 172.21.15.12:0/1038794531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563f90257bb0 0x563f90253bc0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8f0801f740 tx=0x7f8f0801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf010b2d5f08e87a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:38.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.453+0000 7f8f05600640 1 -- 172.21.15.12:0/1038794531 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f08011580 con 0x563f90257bb0 2024-03-23T23:03:38.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.453+0000 7f8f05600640 1 -- 172.21.15.12:0/1038794531 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f08011c00 con 0x563f90257bb0 2024-03-23T23:03:38.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.453+0000 7f8f05600640 1 -- 172.21.15.12:0/1038794531 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f08029de0 con 0x563f90257bb0 2024-03-23T23:03:38.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.454+0000 7f8f1a868740 1 -- 172.21.15.12:0/1038794531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563f90257bb0 msgr2=0x563f90253bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:38.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.454+0000 7f8f1a868740 1 --2- 172.21.15.12:0/1038794531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563f90257bb0 0x563f90253bc0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8f0801f740 tx=0x7f8f0801a000 comp rx=0 tx=0).stop 2024-03-23T23:03:38.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.454+0000 7f8f1a868740 1 -- 172.21.15.12:0/1038794531 shutdown_connections 2024-03-23T23:03:38.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.454+0000 7f8f1a868740 1 --2- 172.21.15.12:0/1038794531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563f90257bb0 0x563f90253bc0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.454+0000 7f8f1a868740 1 --2- 172.21.15.12:0/1038794531 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f90252a40 0x563f90263290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.454+0000 7f8f1a868740 1 --2- 172.21.15.12:0/1038794531 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563f90252050 0x563f90251490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.454+0000 7f8f1a868740 1 -- 172.21.15.12:0/1038794531 >> 172.21.15.12:0/1038794531 conn(0x563f9021a200 msgr2=0x563f902534b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:38.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.454+0000 7f8f1a868740 1 -- 172.21.15.12:0/1038794531 shutdown_connections 2024-03-23T23:03:38.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.454+0000 7f8f1a868740 1 -- 172.21.15.12:0/1038794531 wait complete. 2024-03-23T23:03:38.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.454+0000 7f8f1a868740 1 -- start start 2024-03-23T23:03:38.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.455+0000 7f8f1a868740 1 Processor -- start 2024-03-23T23:03:38.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.455+0000 7f8f1a868740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563f90252050 0x563f90260b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.455+0000 7f8f1a868740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563f90252a40 0x563f90261060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.455+0000 7f8f1a868740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f90257bb0 0x563f90262070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.455+0000 7f8f1a868740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f101ad390 con 0x563f90252050 2024-03-23T23:03:38.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.455+0000 7f8f1a868740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f101ae690 con 0x563f90257bb0 2024-03-23T23:03:38.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.455+0000 7f8f1a868740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f101ad9d0 con 0x563f90252a40 2024-03-23T23:03:38.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.455+0000 7f8f1a868740 10 client.0 _ll_register_callbacks cb 0x7f8f080095d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:38.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.455+0000 7f8f0fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563f90252a40 0x563f90261060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.455+0000 7f8f15c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f90257bb0 0x563f90262070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.455+0000 7f8f15c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f90257bb0 0x563f90262070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50764/0 (socket says 172.21.15.12:50764) 2024-03-23T23:03:38.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.455+0000 7f8f15c00640 1 -- 172.21.15.12:0/3464321838 learned_addr learned my addr 172.21.15.12:0/3464321838 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:38.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.455+0000 7f8f0fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563f90252a40 0x563f90261060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35596/0 (socket says 172.21.15.12:35596) 2024-03-23T23:03:38.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.455+0000 7f8f15200640 1 --2- 172.21.15.12:0/3464321838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563f90252050 0x563f90260b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.456+0000 7f8f15c00640 1 -- 172.21.15.12:0/3464321838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563f90252a40 msgr2=0x563f90261060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:38.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.456+0000 7f8f15c00640 1 --2- 172.21.15.12:0/3464321838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563f90252a40 0x563f90261060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.456+0000 7f8f15c00640 1 -- 172.21.15.12:0/3464321838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563f90252050 msgr2=0x563f90260b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:38.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.456+0000 7f8f15c00640 1 --2- 172.21.15.12:0/3464321838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563f90252050 0x563f90260b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.456+0000 7f8f15c00640 1 -- 172.21.15.12:0/3464321838 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563f90262660 con 0x563f90257bb0 2024-03-23T23:03:38.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.456+0000 7f8f0fe00640 1 --2- 172.21.15.12:0/3464321838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563f90252a40 0x563f90261060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:38.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.456+0000 7f8f15c00640 1 --2- 172.21.15.12:0/3464321838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f90257bb0 0x563f90262070 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8f08009b10 tx=0x7f8f0801f4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ed360e4b3afa2e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:38.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.456+0000 7f8efaa00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:03:38.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.456+0000 7f8efaa00640 1 -- 172.21.15.12:0/3464321838 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f08029d60 con 0x563f90257bb0 2024-03-23T23:03:38.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.456+0000 7f8efaa00640 1 -- 172.21.15.12:0/3464321838 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8f08011580 con 0x563f90257bb0 2024-03-23T23:03:38.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.456+0000 7f8f1a868740 1 -- 172.21.15.12:0/3464321838 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8f08011c00 con 0x563f90257bb0 2024-03-23T23:03:38.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.457+0000 7f8efaa00640 1 -- 172.21.15.12:0/3464321838 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f08011780 con 0x563f90257bb0 2024-03-23T23:03:38.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.457+0000 7f8f1a868740 10 client.25790 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:38.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.457+0000 7f8f1a868740 1 -- 172.21.15.12:0/3464321838 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563f902c32f0 con 0x563f90257bb0 2024-03-23T23:03:38.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.457+0000 7f8efaa00640 1 -- 172.21.15.12:0/3464321838 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f08011c00 con 0x563f90257bb0 2024-03-23T23:03:38.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.457+0000 7f8efaa00640 1 -- 172.21.15.12:0/3464321838 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8f08004750 con 0x563f90257bb0 2024-03-23T23:03:38.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.457+0000 7f8f1a868740 10 client.25790 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:38.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.457+0000 7f8f1a868740 10 client.25790 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:38.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.457+0000 7f8f1a868740 20 client.25790 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:38.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.457+0000 7f8f1a868740 1 --2- 172.21.15.12:0/3464321838 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563f902c3bf0 0x563f902c6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:38.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.457+0000 7f8f1a868740 4 client.25790 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:03:38.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.457+0000 7f8f1a868740 1 -- 172.21.15.12:0/3464321838 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563f901beae0 con 0x563f902c3bf0 2024-03-23T23:03:38.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.457+0000 7f8f15200640 1 --2- 172.21.15.12:0/3464321838 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563f902c3bf0 0x563f902c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:38.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.458+0000 7f8f15200640 1 --2- 172.21.15.12:0/3464321838 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563f902c3bf0 0x563f902c6010 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0dcf62671d8415c server_cookie=7911d9f5a1b3a78b in_seq=0 out_seq=0 2024-03-23T23:03:38.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.458+0000 7f8efaa00640 10 client.25790 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:38.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.458+0000 7f8efaa00640 1 -- 172.21.15.12:0/3464321838 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x563f901beae0 con 0x563f902c3bf0 2024-03-23T23:03:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.458+0000 7f8efaa00640 10 client.25790 handle_command_reply: tid=0 2024-03-23T23:03:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.458+0000 7f8f1a868740 1 client.25790 shutdown 2024-03-23T23:03:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.458+0000 7f8f1a868740 1 -- 172.21.15.12:0/3464321838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f90257bb0 msgr2=0x563f90262070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.458+0000 7f8f1a868740 1 --2- 172.21.15.12:0/3464321838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f90257bb0 0x563f90262070 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8f08009b10 tx=0x7f8f0801f4a0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.458+0000 7f8f1a868740 1 -- 172.21.15.12:0/3464321838 shutdown_connections 2024-03-23T23:03:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.458+0000 7f8f1a868740 1 --2- 172.21.15.12:0/3464321838 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563f902c3bf0 0x563f902c6010 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.458+0000 7f8f1a868740 1 --2- 172.21.15.12:0/3464321838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f90257bb0 0x563f90262070 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.458+0000 7f8f1a868740 1 --2- 172.21.15.12:0/3464321838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563f90252a40 0x563f90261060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.458+0000 7f8f1a868740 1 --2- 172.21.15.12:0/3464321838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563f90252050 0x563f90260b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.458+0000 7f8efaa00640 0 client.25790 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.458+0000 7f8f1a868740 1 -- 172.21.15.12:0/3464321838 >> 172.21.15.12:0/3464321838 conn(0x563f9021a200 msgr2=0x563f90253cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.458+0000 7f8f1a868740 1 -- 172.21.15.12:0/3464321838 shutdown_connections 2024-03-23T23:03:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.458+0000 7f8f1a868740 1 -- 172.21.15.12:0/3464321838 wait complete. 2024-03-23T23:03:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.458+0000 7f8f1a868740 20 client.25790 trim_cache size 0 max 16384 2024-03-23T23:03:38.467 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:03:38.467 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T23:03:38.467 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-03-23T23:03:38.467 INFO:teuthology.orchestra.run.smithi012.stdout: "8551d2c0-914e-43df-a336-98e10888bd7c": { 2024-03-23T23:03:38.467 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-03-23T23:03:38.467 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "8551d2c0-914e-43df-a336-98e10888bd7c", 2024-03-23T23:03:38.467 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-03-23T23:03:38.467 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:03:38.467 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:03:38.467 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:03:38.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.459+0000 7f8f16600640 1 -- 172.21.15.12:0/3594732146 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8ef40a2880 msgr2=0x7f8ef40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:38.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.459+0000 7f8f16600640 1 --2- 172.21.15.12:0/3594732146 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8ef40a2880 0x7f8ef40a4d40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x563f902532a0 tx=0x7f8f08005f90 comp rx=0 tx=0).stop 2024-03-23T23:03:38.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.459+0000 7f8f16600640 1 -- 172.21.15.12:0/3594732146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f10074100 msgr2=0x7f8f1019dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:38.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.459+0000 7f8f16600640 1 --2- 172.21.15.12:0/3594732146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f10074100 0x7f8f1019dee0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8f00000f80 tx=0x7f8f00035d90 comp rx=0 tx=0).stop 2024-03-23T23:03:38.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.459+0000 7f8f16600640 1 -- 172.21.15.12:0/3594732146 shutdown_connections 2024-03-23T23:03:38.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.459+0000 7f8f16600640 1 --2- 172.21.15.12:0/3594732146 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8ef40a2880 0x7f8ef40a4d40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.459+0000 7f8f16600640 1 --2- 172.21.15.12:0/3594732146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f1010db70 0x7f8f101a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.459+0000 7f8f16600640 1 --2- 172.21.15.12:0/3594732146 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f100756e0 0x7f8f1019e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.459+0000 7f8f16600640 1 --2- 172.21.15.12:0/3594732146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f10074100 0x7f8f1019dee0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:38.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.459+0000 7f8f16600640 1 -- 172.21.15.12:0/3594732146 >> 172.21.15.12:0/3594732146 conn(0x7f8f100fcb10 msgr2=0x7f8f10100f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:38.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.459+0000 7f8f16600640 1 -- 172.21.15.12:0/3594732146 shutdown_connections 2024-03-23T23:03:38.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:03:38.459+0000 7f8f16600640 1 -- 172.21.15.12:0/3594732146 wait complete. 2024-03-23T23:03:38.479 INFO:tasks.cephfs.filesystem:scrub status for tag:8551d2c0-914e-43df-a336-98e10888bd7c - {'path': '/', 'tag': '8551d2c0-914e-43df-a336-98e10888bd7c', 'options': 'recursive,force'} 2024-03-23T23:03:38.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:38 smithi012 ceph-mon[31695]: pgmap v445: 97 pgs: 97 active+clean; 698 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 4.1 KiB/s wr, 9 op/s 2024-03-23T23:03:38.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:38 smithi173 ceph-mon[36116]: pgmap v445: 97 pgs: 97 active+clean; 698 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 4.1 KiB/s wr, 9 op/s 2024-03-23T23:03:38.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:38 smithi203 ceph-mon[36693]: pgmap v445: 97 pgs: 97 active+clean; 698 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 4.1 KiB/s wr, 9 op/s 2024-03-23T23:03:39.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:39 smithi012 ceph-mon[31695]: scrub queued for path: / 2024-03-23T23:03:39.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:39 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:03:39.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:39 smithi012 ceph-mon[31695]: scrub summary: active paths [/] 2024-03-23T23:03:39.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:39 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:03:39.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:39 smithi173 ceph-mon[36116]: scrub queued for path: / 2024-03-23T23:03:39.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:39 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:03:39.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:39 smithi173 ceph-mon[36116]: scrub summary: active paths [/] 2024-03-23T23:03:39.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:39 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:03:39.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:39 smithi203 ceph-mon[36693]: scrub queued for path: / 2024-03-23T23:03:39.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:39 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:03:39.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:39 smithi203 ceph-mon[36693]: scrub summary: active paths [/] 2024-03-23T23:03:39.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:39 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:03:40.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:40 smithi012 ceph-mon[31695]: pgmap v446: 97 pgs: 97 active+clean; 755 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.8 KiB/s wr, 11 op/s 2024-03-23T23:03:40.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:40 smithi173 ceph-mon[36116]: pgmap v446: 97 pgs: 97 active+clean; 755 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.8 KiB/s wr, 11 op/s 2024-03-23T23:03:40.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:40 smithi203 ceph-mon[36693]: pgmap v446: 97 pgs: 97 active+clean; 755 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.8 KiB/s wr, 11 op/s 2024-03-23T23:03:41.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:41 smithi173 ceph-mon[36116]: scrub summary: idle 2024-03-23T23:03:41.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:41 smithi203 ceph-mon[36693]: scrub summary: idle 2024-03-23T23:03:42.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:41 smithi012 ceph-mon[31695]: scrub summary: idle 2024-03-23T23:03:42.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:42 smithi173 ceph-mon[36116]: pgmap v447: 97 pgs: 97 active+clean; 755 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.9 KiB/s wr, 12 op/s 2024-03-23T23:03:43.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:42 smithi012 ceph-mon[31695]: pgmap v447: 97 pgs: 97 active+clean; 755 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.9 KiB/s wr, 12 op/s 2024-03-23T23:03:43.116 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:42 smithi203 ceph-mon[36693]: pgmap v447: 97 pgs: 97 active+clean; 755 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.9 KiB/s wr, 12 op/s 2024-03-23T23:03:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.990+0000 7f1c32a00640 1 -- 172.21.15.12:0/1738767624 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c2c10dca0 msgr2=0x7f1c2c110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.990+0000 7f1c32a00640 1 --2- 172.21.15.12:0/1738767624 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c2c10dca0 0x7f1c2c110090 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1c24009cd0 tx=0x7f1c2402c0e0 comp rx=0 tx=0).stop 2024-03-23T23:03:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.990+0000 7f1c32a00640 1 -- 172.21.15.12:0/1738767624 shutdown_connections 2024-03-23T23:03:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.990+0000 7f1c32a00640 1 --2- 172.21.15.12:0/1738767624 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c2c10dca0 0x7f1c2c110090 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.990+0000 7f1c32a00640 1 --2- 172.21.15.12:0/1738767624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c2c074100 0x7f1c2c074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.990+0000 7f1c32a00640 1 --2- 172.21.15.12:0/1738767624 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c2c0756e0 0x7f1c2c073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.990+0000 7f1c32a00640 1 -- 172.21.15.12:0/1738767624 >> 172.21.15.12:0/1738767624 conn(0x7f1c2c0fcc60 msgr2=0x7f1c2c0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.990+0000 7f1c32a00640 1 -- 172.21.15.12:0/1738767624 shutdown_connections 2024-03-23T23:03:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.990+0000 7f1c32a00640 1 -- 172.21.15.12:0/1738767624 wait complete. 2024-03-23T23:03:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.991+0000 7f1c32a00640 1 Processor -- start 2024-03-23T23:03:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.991+0000 7f1c32a00640 1 -- start start 2024-03-23T23:03:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.991+0000 7f1c32a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c2c074100 0x7f1c2c19dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.991+0000 7f1c32a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c2c0756e0 0x7f1c2c19e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.991+0000 7f1c32a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c2c10dca0 0x7f1c2c1a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.991+0000 7f1c32a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c113920 con 0x7f1c2c10dca0 2024-03-23T23:03:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.991+0000 7f1c32a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c1137a0 con 0x7f1c2c0756e0 2024-03-23T23:03:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.991+0000 7f1c32a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c113aa0 con 0x7f1c2c074100 2024-03-23T23:03:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.992+0000 7f1c2be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c2c074100 0x7f1c2c19dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.992+0000 7f1c2be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c2c074100 0x7f1c2c19dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:57998/0 (socket says 172.21.15.12:57998) 2024-03-23T23:03:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.992+0000 7f1c2b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c2c0756e0 0x7f1c2c19e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.992+0000 7f1c2be00640 1 -- 172.21.15.12:0/1047964346 learned_addr learned my addr 172.21.15.12:0/1047964346 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.992+0000 7f1c30a00640 1 --2- 172.21.15.12:0/1047964346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c2c10dca0 0x7f1c2c1a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.992+0000 7f1c2be00640 1 -- 172.21.15.12:0/1047964346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c2c0756e0 msgr2=0x7f1c2c19e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.992+0000 7f1c2be00640 1 --2- 172.21.15.12:0/1047964346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c2c0756e0 0x7f1c2c19e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.992+0000 7f1c2be00640 1 -- 172.21.15.12:0/1047964346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c2c10dca0 msgr2=0x7f1c2c1a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.992+0000 7f1c2be00640 1 --2- 172.21.15.12:0/1047964346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c2c10dca0 0x7f1c2c1a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.992+0000 7f1c2be00640 1 -- 172.21.15.12:0/1047964346 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2c1a5b00 con 0x7f1c2c074100 2024-03-23T23:03:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.992+0000 7f1c2be00640 1 --2- 172.21.15.12:0/1047964346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c2c074100 0x7f1c2c19dee0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c00e9f0 tx=0x7f1c1c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.993+0000 7f1c28c00640 1 -- 172.21.15.12:0/1047964346 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c00cd80 con 0x7f1c2c074100 2024-03-23T23:03:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.993+0000 7f1c28c00640 1 -- 172.21.15.12:0/1047964346 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c1c010910 con 0x7f1c2c074100 2024-03-23T23:03:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.993+0000 7f1c28c00640 1 -- 172.21.15.12:0/1047964346 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c018990 con 0x7f1c2c074100 2024-03-23T23:03:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.993+0000 7f1c32a00640 1 -- 172.21.15.12:0/1047964346 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c2c1a5df0 con 0x7f1c2c074100 2024-03-23T23:03:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.993+0000 7f1c32a00640 1 -- 172.21.15.12:0/1047964346 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c2c1a62b0 con 0x7f1c2c074100 2024-03-23T23:03:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.996+0000 7f1c28c00640 1 -- 172.21.15.12:0/1047964346 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1c1c010ac0 con 0x7f1c2c074100 2024-03-23T23:03:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.996+0000 7f1c357e9740 1 Processor -- start 2024-03-23T23:03:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.996+0000 7f1c357e9740 1 -- start start 2024-03-23T23:03:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.997+0000 7f1c357e9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d533b1d1e0 0x55d533b1d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.997+0000 7f1c357e9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d533b1dbd0 0x55d533b226d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.997+0000 7f1c357e9740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d533b22d40 0x55d533b251d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.997+0000 7f1c357e9740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d533ae03b0 con 0x55d533b22d40 2024-03-23T23:03:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.997+0000 7f1c357e9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d533a82830 con 0x55d533b1dbd0 2024-03-23T23:03:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.997+0000 7f1c357e9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d533a2b5d0 con 0x55d533b1d1e0 2024-03-23T23:03:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.997+0000 7f1c2be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d533b1dbd0 0x55d533b226d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.997+0000 7f1c30a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d533b1d1e0 0x55d533b1d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.997+0000 7f1c30a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d533b1d1e0 0x55d533b1d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58000/0 (socket says 172.21.15.12:58000) 2024-03-23T23:03:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.997+0000 7f1c30a00640 1 -- 172.21.15.12:0/3002920346 learned_addr learned my addr 172.21.15.12:0/3002920346 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.997+0000 7f1c2b400640 1 --2- 172.21.15.12:0/3002920346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d533b22d40 0x55d533b251d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.997+0000 7f1c30a00640 1 -- 172.21.15.12:0/3002920346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d533b1dbd0 msgr2=0x55d533b226d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.997+0000 7f1c30a00640 1 --2- 172.21.15.12:0/3002920346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d533b1dbd0 0x55d533b226d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.998+0000 7f1c30a00640 1 -- 172.21.15.12:0/3002920346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d533b22d40 msgr2=0x55d533b251d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.998+0000 7f1c30a00640 1 --2- 172.21.15.12:0/3002920346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d533b22d40 0x55d533b251d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.998+0000 7f1c30a00640 1 -- 172.21.15.12:0/3002920346 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d533a432c0 con 0x55d533b1d1e0 2024-03-23T23:03:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.998+0000 7f1c2b400640 1 --2- 172.21.15.12:0/3002920346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d533b22d40 0x55d533b251d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:03:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.998+0000 7f1c30a00640 1 --2- 172.21.15.12:0/3002920346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d533b1d1e0 0x55d533b1d600 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1c240325d0 tx=0x7f1c24005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d3ed30b7a2a4040 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.998+0000 7f1c21200640 1 -- 172.21.15.12:0/3002920346 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c24043020 con 0x55d533b1d1e0 2024-03-23T23:03:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.998+0000 7f1c28c00640 1 --2- 172.21.15.12:0/1047964346 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1c100a2920 0x7f1c100a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.998+0000 7f1c21200640 1 -- 172.21.15.12:0/3002920346 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c24032e70 con 0x55d533b1d1e0 2024-03-23T23:03:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.998+0000 7f1c21200640 1 -- 172.21.15.12:0/3002920346 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c24039040 con 0x55d533b1d1e0 2024-03-23T23:03:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.998+0000 7f1c28c00640 1 -- 172.21.15.12:0/1047964346 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1c1c0cee10 con 0x7f1c2c074100 2024-03-23T23:03:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.998+0000 7f1c2be00640 1 --2- 172.21.15.12:0/1047964346 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1c100a2920 0x7f1c100a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.999+0000 7f1c357e9740 1 -- 172.21.15.12:0/3002920346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d533b1d1e0 msgr2=0x55d533b1d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.999+0000 7f1c357e9740 1 --2- 172.21.15.12:0/3002920346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d533b1d1e0 0x55d533b1d600 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1c240325d0 tx=0x7f1c24005f80 comp rx=0 tx=0).stop 2024-03-23T23:03:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.999+0000 7f1c357e9740 1 -- 172.21.15.12:0/3002920346 shutdown_connections 2024-03-23T23:03:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.999+0000 7f1c357e9740 1 --2- 172.21.15.12:0/3002920346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d533b22d40 0x55d533b251d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.999+0000 7f1c357e9740 1 --2- 172.21.15.12:0/3002920346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d533b1dbd0 0x55d533b226d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.999+0000 7f1c357e9740 1 --2- 172.21.15.12:0/3002920346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d533b1d1e0 0x55d533b1d600 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.999+0000 7f1c357e9740 1 -- 172.21.15.12:0/3002920346 >> 172.21.15.12:0/3002920346 conn(0x55d533b18750 msgr2=0x55d533b18b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.999+0000 7f1c2be00640 1 --2- 172.21.15.12:0/1047964346 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1c100a2920 0x7f1c100a4de0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c097bc0 tx=0x7f1c1c0d2850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.999+0000 7f1c357e9740 1 -- 172.21.15.12:0/3002920346 shutdown_connections 2024-03-23T23:03:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:43.999+0000 7f1c357e9740 1 -- 172.21.15.12:0/3002920346 wait complete. 2024-03-23T23:03:44.006 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.000+0000 7f1c357e9740 1 -- start start 2024-03-23T23:03:44.006 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.000+0000 7f1c357e9740 1 Processor -- start 2024-03-23T23:03:44.006 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.000+0000 7f1c357e9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d533b1d1e0 0x55d533b273c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:44.006 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.000+0000 7f1c357e9740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d533b1dbd0 0x55d533b27900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.000+0000 7f1c357e9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d533b22d40 0x55d533b2e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.000+0000 7f1c357e9740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d533a3ea20 con 0x55d533b1dbd0 2024-03-23T23:03:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.000+0000 7f1c357e9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d533a7e8d0 con 0x55d533b1d1e0 2024-03-23T23:03:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.000+0000 7f1c357e9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d533b25c90 con 0x55d533b22d40 2024-03-23T23:03:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.001+0000 7f1c357e9740 10 client.0 _ll_register_callbacks cb 0x55d533a7b1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.001+0000 7f1c30a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d533b1d1e0 0x55d533b273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.001+0000 7f1c30a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d533b1d1e0 0x55d533b273c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57632/0 (socket says 172.21.15.12:57632) 2024-03-23T23:03:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.001+0000 7f1c30a00640 1 -- 172.21.15.12:0/3265266248 learned_addr learned my addr 172.21.15.12:0/3265266248 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.001+0000 7f1c30a00640 1 --2- 172.21.15.12:0/3265266248 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d533b22d40 0x55d533b2e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.001+0000 7f1c2b400640 1 --2- 172.21.15.12:0/3265266248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d533b1dbd0 0x55d533b27900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.001+0000 7f1c30a00640 1 -- 172.21.15.12:0/3265266248 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d533b22d40 msgr2=0x55d533b2e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.001+0000 7f1c30a00640 1 --2- 172.21.15.12:0/3265266248 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d533b22d40 0x55d533b2e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.001+0000 7f1c30a00640 1 -- 172.21.15.12:0/3265266248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d533b1dbd0 msgr2=0x55d533b27900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.001+0000 7f1c30a00640 1 --2- 172.21.15.12:0/3265266248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d533b1dbd0 0x55d533b27900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.001+0000 7f1c30a00640 1 -- 172.21.15.12:0/3265266248 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d533797210 con 0x55d533b1d1e0 2024-03-23T23:03:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.001+0000 7f1c30a00640 1 --2- 172.21.15.12:0/3265266248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d533b1d1e0 0x55d533b273c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1c24009a60 tx=0x7f1c2402e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41608d254e24055a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.001+0000 7f1c15600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:03:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.002+0000 7f1c15600640 1 -- 172.21.15.12:0/3265266248 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2404d020 con 0x55d533b1d1e0 2024-03-23T23:03:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.002+0000 7f1c15600640 1 -- 172.21.15.12:0/3265266248 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1c240041c0 con 0x55d533b1d1e0 2024-03-23T23:03:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.002+0000 7f1c357e9740 1 -- 172.21.15.12:0/3265266248 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d533a8fd70 con 0x55d533b1d1e0 2024-03-23T23:03:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.002+0000 7f1c15600640 1 -- 172.21.15.12:0/3265266248 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c24043070 con 0x55d533b1d1e0 2024-03-23T23:03:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.002+0000 7f1c357e9740 10 client.25802 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.002+0000 7f1c357e9740 1 -- 172.21.15.12:0/3265266248 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55d533863620 con 0x55d533b1d1e0 2024-03-23T23:03:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.002+0000 7f1c15600640 1 -- 172.21.15.12:0/3265266248 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d533a8fd70 con 0x55d533b1d1e0 2024-03-23T23:03:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.002+0000 7f1c15600640 1 -- 172.21.15.12:0/3265266248 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f1c2403faa0 con 0x55d533b1d1e0 2024-03-23T23:03:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.002+0000 7f1c357e9740 10 client.25802 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.002+0000 7f1c357e9740 10 client.25802 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.002+0000 7f1c357e9740 20 client.25802 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.003+0000 7f1c357e9740 1 --2- 172.21.15.12:0/3265266248 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d533b32e10 0x55d533b35200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.003+0000 7f1c357e9740 4 client.25802 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:03:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.003+0000 7f1c357e9740 1 -- 172.21.15.12:0/3265266248 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d533a8a0c0 con 0x55d533b32e10 2024-03-23T23:03:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.003+0000 7f1c2b400640 1 --2- 172.21.15.12:0/3265266248 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d533b32e10 0x55d533b35200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.003+0000 7f1c2b400640 1 --2- 172.21.15.12:0/3265266248 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d533b32e10 0x55d533b35200 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea96074d951e87aa server_cookie=7ab3581ebdf11c11 in_seq=0 out_seq=0 2024-03-23T23:03:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.003+0000 7f1c15600640 10 client.25802 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.004+0000 7f1c15600640 1 -- 172.21.15.12:0/3265266248 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d533a8a0c0 con 0x55d533b32e10 2024-03-23T23:03:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.004+0000 7f1c15600640 10 client.25802 handle_command_reply: tid=0 2024-03-23T23:03:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.004+0000 7f1c357e9740 1 client.25802 shutdown 2024-03-23T23:03:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.005+0000 7f1c357e9740 1 -- 172.21.15.12:0/3265266248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d533b1d1e0 msgr2=0x55d533b273c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.005+0000 7f1c357e9740 1 --2- 172.21.15.12:0/3265266248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d533b1d1e0 0x55d533b273c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1c24009a60 tx=0x7f1c2402e040 comp rx=0 tx=0).stop 2024-03-23T23:03:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.005+0000 7f1c357e9740 1 -- 172.21.15.12:0/3265266248 shutdown_connections 2024-03-23T23:03:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.005+0000 7f1c357e9740 1 --2- 172.21.15.12:0/3265266248 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d533b32e10 0x55d533b35200 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.005+0000 7f1c357e9740 1 --2- 172.21.15.12:0/3265266248 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d533b22d40 0x55d533b2e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.005+0000 7f1c357e9740 1 --2- 172.21.15.12:0/3265266248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d533b1dbd0 0x55d533b27900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.005+0000 7f1c357e9740 1 --2- 172.21.15.12:0/3265266248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d533b1d1e0 0x55d533b273c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.005+0000 7f1c15600640 0 client.25802 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.005+0000 7f1c357e9740 1 -- 172.21.15.12:0/3265266248 >> 172.21.15.12:0/3265266248 conn(0x55d533b18750 msgr2=0x55d533b17950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.005+0000 7f1c357e9740 1 -- 172.21.15.12:0/3265266248 shutdown_connections 2024-03-23T23:03:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.005+0000 7f1c357e9740 1 -- 172.21.15.12:0/3265266248 wait complete. 2024-03-23T23:03:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.005+0000 7f1c357e9740 20 client.25802 trim_cache size 0 max 16384 2024-03-23T23:03:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.020+0000 7f1c357e9740 1 Processor -- start 2024-03-23T23:03:44.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.020+0000 7f1c357e9740 1 -- start start 2024-03-23T23:03:44.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.021+0000 7f1c357e9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d533b1d1e0 0x55d533b27860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:44.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.021+0000 7f1c357e9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d533b1dbd0 0x55d533b2e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:44.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.021+0000 7f1c357e9740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d533b22d40 0x55d533b1c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:44.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.021+0000 7f1c357e9740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c1b1490 con 0x55d533b22d40 2024-03-23T23:03:44.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.021+0000 7f1c357e9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c1aef90 con 0x55d533b1d1e0 2024-03-23T23:03:44.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.021+0000 7f1c357e9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c1ad030 con 0x55d533b1dbd0 2024-03-23T23:03:44.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.021+0000 7f1c2b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d533b1dbd0 0x55d533b2e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:44.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.021+0000 7f1c30a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d533b1d1e0 0x55d533b27860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:44.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.021+0000 7f1c2b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d533b1dbd0 0x55d533b2e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58018/0 (socket says 172.21.15.12:58018) 2024-03-23T23:03:44.023 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.021+0000 7f1c2b400640 1 -- 172.21.15.12:0/3901020123 learned_addr learned my addr 172.21.15.12:0/3901020123 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:44.023 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.021+0000 7f1c30a00640 1 --2- 172.21.15.12:0/3901020123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d533b22d40 0x55d533b1c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:44.023 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.021+0000 7f1c2b400640 1 -- 172.21.15.12:0/3901020123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d533b1d1e0 msgr2=0x55d533b27860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:44.023 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.021+0000 7f1c2b400640 1 --2- 172.21.15.12:0/3901020123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d533b1d1e0 0x55d533b27860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.023 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.021+0000 7f1c2b400640 1 -- 172.21.15.12:0/3901020123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d533b22d40 msgr2=0x55d533b1c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:44.023 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.021+0000 7f1c2b400640 1 --2- 172.21.15.12:0/3901020123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d533b22d40 0x55d533b1c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.023 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.021+0000 7f1c2b400640 1 -- 172.21.15.12:0/3901020123 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d533b1e430 con 0x55d533b1dbd0 2024-03-23T23:03:44.023 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.022+0000 7f1c2b400640 1 --2- 172.21.15.12:0/3901020123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d533b1dbd0 0x55d533b2e900 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x55d533b196c0 tx=0x7f1c18005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ac1c86d2595be51 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:44.023 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.022+0000 7f1c17e00640 1 -- 172.21.15.12:0/3901020123 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c18020310 con 0x55d533b1dbd0 2024-03-23T23:03:44.023 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.022+0000 7f1c17e00640 1 -- 172.21.15.12:0/3901020123 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c18011e50 con 0x55d533b1dbd0 2024-03-23T23:03:44.023 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.022+0000 7f1c17e00640 1 -- 172.21.15.12:0/3901020123 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1801ed20 con 0x55d533b1dbd0 2024-03-23T23:03:44.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.022+0000 7f1c357e9740 1 -- 172.21.15.12:0/3901020123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d533b1dbd0 msgr2=0x55d533b2e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:44.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.022+0000 7f1c357e9740 1 --2- 172.21.15.12:0/3901020123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d533b1dbd0 0x55d533b2e900 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x55d533b196c0 tx=0x7f1c18005f90 comp rx=0 tx=0).stop 2024-03-23T23:03:44.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.022+0000 7f1c357e9740 1 -- 172.21.15.12:0/3901020123 shutdown_connections 2024-03-23T23:03:44.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.022+0000 7f1c357e9740 1 --2- 172.21.15.12:0/3901020123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d533b22d40 0x55d533b1c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.022+0000 7f1c357e9740 1 --2- 172.21.15.12:0/3901020123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d533b1dbd0 0x55d533b2e900 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.022+0000 7f1c357e9740 1 --2- 172.21.15.12:0/3901020123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d533b1d1e0 0x55d533b27860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.022+0000 7f1c357e9740 1 -- 172.21.15.12:0/3901020123 >> 172.21.15.12:0/3901020123 conn(0x55d533b18750 msgr2=0x55d533b1b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:44.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.023+0000 7f1c357e9740 1 -- 172.21.15.12:0/3901020123 shutdown_connections 2024-03-23T23:03:44.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.023+0000 7f1c357e9740 1 -- 172.21.15.12:0/3901020123 wait complete. 2024-03-23T23:03:44.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.023+0000 7f1c357e9740 1 -- start start 2024-03-23T23:03:44.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.023+0000 7f1c357e9740 1 Processor -- start 2024-03-23T23:03:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.024+0000 7f1c357e9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d533b1d1e0 0x55d533b2c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.024+0000 7f1c357e9740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d533b1dbd0 0x55d533b2ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.024+0000 7f1c357e9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d533b22d40 0x55d533b2ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.024+0000 7f1c357e9740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c1ad630 con 0x55d533b1dbd0 2024-03-23T23:03:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.024+0000 7f1c357e9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c1ae930 con 0x55d533b1d1e0 2024-03-23T23:03:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.024+0000 7f1c357e9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c1adc70 con 0x55d533b22d40 2024-03-23T23:03:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.024+0000 7f1c357e9740 10 client.0 _ll_register_callbacks cb 0x7f1c1c0d2720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.024+0000 7f1c30a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d533b1d1e0 0x55d533b2c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.024+0000 7f1c2b400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d533b1dbd0 0x55d533b2ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.024+0000 7f1c30a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d533b1d1e0 0x55d533b2c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57650/0 (socket says 172.21.15.12:57650) 2024-03-23T23:03:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.024+0000 7f1c30a00640 1 -- 172.21.15.12:0/3472619804 learned_addr learned my addr 172.21.15.12:0/3472619804 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.024+0000 7f1c30a00640 1 --2- 172.21.15.12:0/3472619804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d533b22d40 0x55d533b2ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.024+0000 7f1c30a00640 1 -- 172.21.15.12:0/3472619804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d533b22d40 msgr2=0x55d533b2ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.024+0000 7f1c30a00640 1 --2- 172.21.15.12:0/3472619804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d533b22d40 0x55d533b2ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.024+0000 7f1c30a00640 1 -- 172.21.15.12:0/3472619804 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d533b1dbd0 msgr2=0x55d533b2ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.024+0000 7f1c30a00640 1 --2- 172.21.15.12:0/3472619804 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d533b1dbd0 0x55d533b2ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.024+0000 7f1c30a00640 1 -- 172.21.15.12:0/3472619804 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d533b2e3e0 con 0x55d533b1d1e0 2024-03-23T23:03:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.024+0000 7f1c2b400640 1 --2- 172.21.15.12:0/3472619804 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d533b1dbd0 0x55d533b2ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.025+0000 7f1c30a00640 1 --2- 172.21.15.12:0/3472619804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d533b1d1e0 0x55d533b2c940 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1c2403fe90 tx=0x7f1c24034590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bff6e18bd2caa891 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.025+0000 7f1c15600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:03:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.025+0000 7f1c15600640 1 -- 172.21.15.12:0/3472619804 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2404d040 con 0x55d533b1d1e0 2024-03-23T23:03:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.025+0000 7f1c357e9740 1 -- 172.21.15.12:0/3472619804 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1c18011e50 con 0x55d533b1d1e0 2024-03-23T23:03:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.025+0000 7f1c15600640 1 -- 172.21.15.12:0/3472619804 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1c24034a40 con 0x55d533b1d1e0 2024-03-23T23:03:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.025+0000 7f1c15600640 1 -- 172.21.15.12:0/3472619804 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c24039040 con 0x55d533b1d1e0 2024-03-23T23:03:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.025+0000 7f1c357e9740 10 client.25808 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.025+0000 7f1c357e9740 1 -- 172.21.15.12:0/3472619804 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55d533b8ee30 con 0x55d533b1d1e0 2024-03-23T23:03:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.025+0000 7f1c15600640 1 -- 172.21.15.12:0/3472619804 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c18011e50 con 0x55d533b1d1e0 2024-03-23T23:03:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.026+0000 7f1c15600640 1 -- 172.21.15.12:0/3472619804 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f1c24071020 con 0x55d533b1d1e0 2024-03-23T23:03:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.026+0000 7f1c357e9740 10 client.25808 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.026+0000 7f1c357e9740 10 client.25808 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.026+0000 7f1c357e9740 20 client.25808 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.026+0000 7f1c357e9740 1 --2- 172.21.15.12:0/3472619804 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d533b8f680 0x55d533b91ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.026+0000 7f1c357e9740 4 client.25808 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:03:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.026+0000 7f1c357e9740 1 -- 172.21.15.12:0/3472619804 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d533b13a80 con 0x55d533b8f680 2024-03-23T23:03:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.026+0000 7f1c2b400640 1 --2- 172.21.15.12:0/3472619804 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d533b8f680 0x55d533b91ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.026+0000 7f1c2b400640 1 --2- 172.21.15.12:0/3472619804 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d533b8f680 0x55d533b91ae0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bd6aa0ce5143a27 server_cookie=c8db22dfb233d19f in_seq=0 out_seq=0 2024-03-23T23:03:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.026+0000 7f1c15600640 10 client.25808 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.037+0000 7f1c15600640 1 -- 172.21.15.12:0/3472619804 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d533b13a80 con 0x55d533b8f680 2024-03-23T23:03:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.037+0000 7f1c15600640 10 client.25808 handle_command_reply: tid=0 2024-03-23T23:03:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.037+0000 7f1c357e9740 1 client.25808 shutdown 2024-03-23T23:03:44.038 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.037+0000 7f1c357e9740 1 -- 172.21.15.12:0/3472619804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d533b1d1e0 msgr2=0x55d533b2c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:44.038 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.038+0000 7f1c357e9740 1 --2- 172.21.15.12:0/3472619804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d533b1d1e0 0x55d533b2c940 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1c2403fe90 tx=0x7f1c24034590 comp rx=0 tx=0).stop 2024-03-23T23:03:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.038+0000 7f1c357e9740 1 -- 172.21.15.12:0/3472619804 shutdown_connections 2024-03-23T23:03:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.038+0000 7f1c357e9740 1 --2- 172.21.15.12:0/3472619804 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d533b8f680 0x55d533b91ae0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.038+0000 7f1c357e9740 1 --2- 172.21.15.12:0/3472619804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d533b22d40 0x55d533b2ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.038+0000 7f1c357e9740 1 --2- 172.21.15.12:0/3472619804 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d533b1dbd0 0x55d533b2ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.038+0000 7f1c357e9740 1 --2- 172.21.15.12:0/3472619804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d533b1d1e0 0x55d533b2c940 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.038+0000 7f1c357e9740 1 -- 172.21.15.12:0/3472619804 >> 172.21.15.12:0/3472619804 conn(0x55d533b18750 msgr2=0x55d533b276e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.038+0000 7f1c15600640 0 client.25808 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.038+0000 7f1c357e9740 1 -- 172.21.15.12:0/3472619804 shutdown_connections 2024-03-23T23:03:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.038+0000 7f1c357e9740 1 -- 172.21.15.12:0/3472619804 wait complete. 2024-03-23T23:03:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.038+0000 7f1c357e9740 20 client.25808 trim_cache size 0 max 16384 2024-03-23T23:03:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:03:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:03:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:03:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:03:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.039+0000 7f1c32a00640 1 -- 172.21.15.12:0/1047964346 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1c100a2920 msgr2=0x7f1c100a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.039+0000 7f1c32a00640 1 --2- 172.21.15.12:0/1047964346 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1c100a2920 0x7f1c100a4de0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c097bc0 tx=0x7f1c1c0d2850 comp rx=0 tx=0).stop 2024-03-23T23:03:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.039+0000 7f1c32a00640 1 -- 172.21.15.12:0/1047964346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c2c074100 msgr2=0x7f1c2c19dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.039+0000 7f1c32a00640 1 --2- 172.21.15.12:0/1047964346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c2c074100 0x7f1c2c19dee0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c00e9f0 tx=0x7f1c1c00eeb0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.039+0000 7f1c32a00640 1 -- 172.21.15.12:0/1047964346 shutdown_connections 2024-03-23T23:03:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.039+0000 7f1c32a00640 1 --2- 172.21.15.12:0/1047964346 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1c100a2920 0x7f1c100a4de0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.039+0000 7f1c32a00640 1 --2- 172.21.15.12:0/1047964346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c2c10dca0 0x7f1c2c1a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.039+0000 7f1c32a00640 1 --2- 172.21.15.12:0/1047964346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c2c0756e0 0x7f1c2c19e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.039+0000 7f1c32a00640 1 --2- 172.21.15.12:0/1047964346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c2c074100 0x7f1c2c19dee0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.039+0000 7f1c32a00640 1 -- 172.21.15.12:0/1047964346 >> 172.21.15.12:0/1047964346 conn(0x7f1c2c0fcc60 msgr2=0x7f1c2c101040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.039+0000 7f1c32a00640 1 -- 172.21.15.12:0/1047964346 shutdown_connections 2024-03-23T23:03:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:44.039+0000 7f1c32a00640 1 -- 172.21.15.12:0/1047964346 wait complete. 2024-03-23T23:03:44.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:44 smithi173 ceph-mon[36116]: pgmap v448: 97 pgs: 97 active+clean; 813 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 15 op/s 2024-03-23T23:03:44.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:44 smithi203 ceph-mon[36693]: pgmap v448: 97 pgs: 97 active+clean; 813 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 15 op/s 2024-03-23T23:03:45.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:44 smithi012 ceph-mon[31695]: pgmap v448: 97 pgs: 97 active+clean; 813 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 15 op/s 2024-03-23T23:03:46.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:46 smithi173 ceph-mon[36116]: pgmap v449: 97 pgs: 97 active+clean; 813 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-23T23:03:46.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:46 smithi203 ceph-mon[36693]: pgmap v449: 97 pgs: 97 active+clean; 813 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-23T23:03:47.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:46 smithi012 ceph-mon[31695]: pgmap v449: 97 pgs: 97 active+clean; 813 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-23T23:03:48.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:48 smithi173 ceph-mon[36116]: pgmap v450: 97 pgs: 97 active+clean; 813 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-23T23:03:48.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:48 smithi203 ceph-mon[36693]: pgmap v450: 97 pgs: 97 active+clean; 813 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-23T23:03:49.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:48 smithi012 ceph-mon[31695]: pgmap v450: 97 pgs: 97 active+clean; 813 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-23T23:03:50.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:50 smithi173 ceph-mon[36116]: pgmap v451: 97 pgs: 97 active+clean; 870 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 16 op/s 2024-03-23T23:03:50.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:50 smithi203 ceph-mon[36693]: pgmap v451: 97 pgs: 97 active+clean; 870 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 16 op/s 2024-03-23T23:03:51.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:50 smithi012 ceph-mon[31695]: pgmap v451: 97 pgs: 97 active+clean; 870 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 16 op/s 2024-03-23T23:03:52.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:52 smithi173 ceph-mon[36116]: pgmap v452: 97 pgs: 97 active+clean; 870 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-03-23T23:03:52.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:52 smithi203 ceph-mon[36693]: pgmap v452: 97 pgs: 97 active+clean; 870 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-03-23T23:03:53.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:52 smithi012 ceph-mon[31695]: pgmap v452: 97 pgs: 97 active+clean; 870 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-03-23T23:03:54.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:54 smithi173 ceph-mon[36116]: pgmap v453: 97 pgs: 97 active+clean; 929 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 15 op/s 2024-03-23T23:03:54.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:54 smithi203 ceph-mon[36693]: pgmap v453: 97 pgs: 97 active+clean; 929 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 15 op/s 2024-03-23T23:03:55.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:54 smithi012 ceph-mon[31695]: pgmap v453: 97 pgs: 97 active+clean; 929 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 15 op/s 2024-03-23T23:03:56.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:56 smithi173 ceph-mon[36116]: pgmap v454: 97 pgs: 97 active+clean; 929 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:03:56.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:56 smithi203 ceph-mon[36693]: pgmap v454: 97 pgs: 97 active+clean; 929 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:03:57.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:56 smithi012 ceph-mon[31695]: pgmap v454: 97 pgs: 97 active+clean; 929 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:03:57.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.185+0000 7f1069e00640 1 -- 172.21.15.12:0/2190053384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10641016e0 msgr2=0x7f1064101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:57.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.185+0000 7f1069e00640 1 --2- 172.21.15.12:0/2190053384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10641016e0 0x7f1064101b00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1054009900 tx=0x7f105402bc50 comp rx=0 tx=0).stop 2024-03-23T23:03:57.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.185+0000 7f1069e00640 1 -- 172.21.15.12:0/2190053384 shutdown_connections 2024-03-23T23:03:57.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.185+0000 7f1069e00640 1 --2- 172.21.15.12:0/2190053384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1064103290 0x7f1064109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.185+0000 7f1069e00640 1 --2- 172.21.15.12:0/2190053384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10641028d0 0x7f1064102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.185+0000 7f1069e00640 1 --2- 172.21.15.12:0/2190053384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10641016e0 0x7f1064101b00 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.185+0000 7f1069e00640 1 -- 172.21.15.12:0/2190053384 >> 172.21.15.12:0/2190053384 conn(0x7f10640fcc30 msgr2=0x7f10640ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:57.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.186+0000 7f1069e00640 1 -- 172.21.15.12:0/2190053384 shutdown_connections 2024-03-23T23:03:57.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.186+0000 7f1069e00640 1 -- 172.21.15.12:0/2190053384 wait complete. 2024-03-23T23:03:57.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.188+0000 7f1069e00640 1 Processor -- start 2024-03-23T23:03:57.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.188+0000 7f1069e00640 1 -- start start 2024-03-23T23:03:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.188+0000 7f1069e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10641016e0 0x7f10641979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.188+0000 7f1069e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10641028d0 0x7f1064197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.188+0000 7f1069e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1064103290 0x7f106419ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.188+0000 7f1069e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106410d3b0 con 0x7f1064103290 2024-03-23T23:03:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.188+0000 7f1069e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106410d230 con 0x7f10641016e0 2024-03-23T23:03:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.188+0000 7f1069e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106410d530 con 0x7f10641028d0 2024-03-23T23:03:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.188+0000 7f1062a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10641016e0 0x7f10641979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.188+0000 7f1062a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10641016e0 0x7f10641979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:40884/0 (socket says 172.21.15.12:40884) 2024-03-23T23:03:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.188+0000 7f1062a00640 1 -- 172.21.15.12:0/2712633311 learned_addr learned my addr 172.21.15.12:0/2712633311 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.188+0000 7f105a000640 1 --2- 172.21.15.12:0/2712633311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10641028d0 0x7f1064197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.188+0000 7f1063400640 1 --2- 172.21.15.12:0/2712633311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1064103290 0x7f106419ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:57.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.188+0000 7f1062a00640 1 -- 172.21.15.12:0/2712633311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10641028d0 msgr2=0x7f1064197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:57.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.188+0000 7f1062a00640 1 --2- 172.21.15.12:0/2712633311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10641028d0 0x7f1064197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.188+0000 7f1062a00640 1 -- 172.21.15.12:0/2712633311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1064103290 msgr2=0x7f106419ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:57.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.188+0000 7f1062a00640 1 --2- 172.21.15.12:0/2712633311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1064103290 0x7f106419ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.188+0000 7f1062a00640 1 -- 172.21.15.12:0/2712633311 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106419f600 con 0x7f10641016e0 2024-03-23T23:03:57.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.188+0000 7f105a000640 1 --2- 172.21.15.12:0/2712633311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10641028d0 0x7f1064197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:57.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.189+0000 7f1062a00640 1 --2- 172.21.15.12:0/2712633311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10641016e0 0x7f10641979e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1054005ab0 tx=0x7f105402c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:57.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.189+0000 7f1068a00640 1 -- 172.21.15.12:0/2712633311 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1054041070 con 0x7f10641016e0 2024-03-23T23:03:57.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.189+0000 7f1068a00640 1 -- 172.21.15.12:0/2712633311 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1054004450 con 0x7f10641016e0 2024-03-23T23:03:57.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.189+0000 7f1068a00640 1 -- 172.21.15.12:0/2712633311 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1054034440 con 0x7f10641016e0 2024-03-23T23:03:57.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.189+0000 7f1069e00640 1 -- 172.21.15.12:0/2712633311 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f106419f890 con 0x7f10641016e0 2024-03-23T23:03:57.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.189+0000 7f1069e00640 1 -- 172.21.15.12:0/2712633311 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f106419fde0 con 0x7f10641016e0 2024-03-23T23:03:57.192 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.193+0000 7f106cb6c740 1 Processor -- start 2024-03-23T23:03:57.192 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.193+0000 7f106cb6c740 1 -- start start 2024-03-23T23:03:57.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.193+0000 7f106cb6c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33df1e0 0x55a2e33df600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:57.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.193+0000 7f106cb6c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e33dfbd0 0x55a2e33e46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:57.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.193+0000 7f106cb6c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2e33e4d40 0x55a2e33e71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:57.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.193+0000 7f106cb6c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2e33a23b0 con 0x55a2e33e4d40 2024-03-23T23:03:57.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.193+0000 7f106cb6c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2e3344830 con 0x55a2e33df1e0 2024-03-23T23:03:57.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.193+0000 7f106cb6c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2e32ed5d0 con 0x55a2e33dfbd0 2024-03-23T23:03:57.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.193+0000 7f1063400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33df1e0 0x55a2e33df600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:57.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.193+0000 7f1063400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33df1e0 0x55a2e33df600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:40898/0 (socket says 172.21.15.12:40898) 2024-03-23T23:03:57.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.193+0000 7f1063400640 1 -- 172.21.15.12:0/2238226789 learned_addr learned my addr 172.21.15.12:0/2238226789 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:57.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.193+0000 7f1068a00640 1 -- 172.21.15.12:0/2712633311 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1054044070 con 0x7f10641016e0 2024-03-23T23:03:57.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.193+0000 7f105a000640 1 --2- 172.21.15.12:0/2238226789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2e33e4d40 0x55a2e33e71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:57.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.194+0000 7f1063400640 1 -- 172.21.15.12:0/2238226789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e33dfbd0 msgr2=0x55a2e33e46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:57.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.194+0000 7f1063400640 1 --2- 172.21.15.12:0/2238226789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e33dfbd0 0x55a2e33e46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.194+0000 7f1063400640 1 -- 172.21.15.12:0/2238226789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2e33e4d40 msgr2=0x55a2e33e71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:57.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.194+0000 7f1063400640 1 --2- 172.21.15.12:0/2238226789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2e33e4d40 0x55a2e33e71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.194+0000 7f1063400640 1 -- 172.21.15.12:0/2238226789 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a2e33052c0 con 0x55a2e33df1e0 2024-03-23T23:03:57.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.194+0000 7f1063400640 1 --2- 172.21.15.12:0/2238226789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33df1e0 0x55a2e33df600 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f105c009f50 tx=0x7f105c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8632d1fa4c25f576 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:57.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.194+0000 7f1053e00640 1 -- 172.21.15.12:0/2238226789 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c004b20 con 0x55a2e33df1e0 2024-03-23T23:03:57.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.194+0000 7f1053e00640 1 -- 172.21.15.12:0/2238226789 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f105c004cc0 con 0x55a2e33df1e0 2024-03-23T23:03:57.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.194+0000 7f1053e00640 1 -- 172.21.15.12:0/2238226789 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c005450 con 0x55a2e33df1e0 2024-03-23T23:03:57.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.195+0000 7f106cb6c740 1 -- 172.21.15.12:0/2238226789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33df1e0 msgr2=0x55a2e33df600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:57.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.195+0000 7f106cb6c740 1 --2- 172.21.15.12:0/2238226789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33df1e0 0x55a2e33df600 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f105c009f50 tx=0x7f105c0118d0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.195+0000 7f106cb6c740 1 -- 172.21.15.12:0/2238226789 shutdown_connections 2024-03-23T23:03:57.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.195+0000 7f106cb6c740 1 --2- 172.21.15.12:0/2238226789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2e33e4d40 0x55a2e33e71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.195+0000 7f106cb6c740 1 --2- 172.21.15.12:0/2238226789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e33dfbd0 0x55a2e33e46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.195+0000 7f106cb6c740 1 --2- 172.21.15.12:0/2238226789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33df1e0 0x55a2e33df600 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.195+0000 7f106cb6c740 1 -- 172.21.15.12:0/2238226789 >> 172.21.15.12:0/2238226789 conn(0x55a2e33da750 msgr2=0x55a2e33dab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:57.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.195+0000 7f106cb6c740 1 -- 172.21.15.12:0/2238226789 shutdown_connections 2024-03-23T23:03:57.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.195+0000 7f106cb6c740 1 -- 172.21.15.12:0/2238226789 wait complete. 2024-03-23T23:03:57.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.195+0000 7f1068a00640 1 --2- 172.21.15.12:0/2712633311 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f10480a27b0 0x7f10480a4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:57.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.197+0000 7f106cb6c740 1 -- start start 2024-03-23T23:03:57.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.197+0000 7f1068a00640 1 -- 172.21.15.12:0/2712633311 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f10540f0640 con 0x7f10641016e0 2024-03-23T23:03:57.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.197+0000 7f1063400640 1 --2- 172.21.15.12:0/2712633311 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f10480a27b0 0x7f10480a4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:57.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.197+0000 7f106cb6c740 1 Processor -- start 2024-03-23T23:03:57.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f106cb6c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e33df1e0 0x55a2e33e9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:57.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f106cb6c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2e33dfbd0 0x55a2e33e9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:57.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f106cb6c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33e4d40 0x55a2e33f0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:57.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f106cb6c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2e3300a20 con 0x55a2e33dfbd0 2024-03-23T23:03:57.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f106cb6c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2e33408d0 con 0x55a2e33e4d40 2024-03-23T23:03:57.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f106cb6c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2e33e7cf0 con 0x55a2e33df1e0 2024-03-23T23:03:57.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f106cb6c740 10 client.0 _ll_register_callbacks cb 0x55a2e333d1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:57.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f1063400640 1 --2- 172.21.15.12:0/2712633311 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f10480a27b0 0x7f10480a4c70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f105c00ff10 tx=0x7f105c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:57.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f1062a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e33df1e0 0x55a2e33e9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:57.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f1063400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33e4d40 0x55a2e33f0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:57.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f1062a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e33df1e0 0x55a2e33e9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:57702/0 (socket says 172.21.15.12:57702) 2024-03-23T23:03:57.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f1062a00640 1 -- 172.21.15.12:0/3203715816 learned_addr learned my addr 172.21.15.12:0/3203715816 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:57.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f105a000640 1 --2- 172.21.15.12:0/3203715816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2e33dfbd0 0x55a2e33e9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:57.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f1063400640 1 -- 172.21.15.12:0/3203715816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e33df1e0 msgr2=0x55a2e33e9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f1063400640 1 --2- 172.21.15.12:0/3203715816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e33df1e0 0x55a2e33e9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f1063400640 1 -- 172.21.15.12:0/3203715816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2e33dfbd0 msgr2=0x55a2e33e9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f1063400640 1 --2- 172.21.15.12:0/3203715816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2e33dfbd0 0x55a2e33e9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f1063400640 1 -- 172.21.15.12:0/3203715816 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a2e3059210 con 0x55a2e33e4d40 2024-03-23T23:03:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f1062a00640 1 --2- 172.21.15.12:0/3203715816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e33df1e0 0x55a2e33e9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:03:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f1063400640 1 --2- 172.21.15.12:0/3203715816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33e4d40 0x55a2e33f0970 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f105c009f20 tx=0x7f105c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21d93abcf1da1601 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f1050c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:03:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f1050c00640 1 -- 172.21.15.12:0/3203715816 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c032070 con 0x55a2e33e4d40 2024-03-23T23:03:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f1050c00640 1 -- 172.21.15.12:0/3203715816 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f105c024650 con 0x55a2e33e4d40 2024-03-23T23:03:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f106cb6c740 1 -- 172.21.15.12:0/3203715816 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f105c004cc0 con 0x55a2e33e4d40 2024-03-23T23:03:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f1050c00640 1 -- 172.21.15.12:0/3203715816 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f105c03e050 con 0x55a2e33e4d40 2024-03-23T23:03:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f106cb6c740 10 client.25820 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.198+0000 7f106cb6c740 1 -- 172.21.15.12:0/3203715816 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a2e3125620 con 0x55a2e33e4d40 2024-03-23T23:03:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.199+0000 7f1050c00640 1 -- 172.21.15.12:0/3203715816 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f105c004cc0 con 0x55a2e33e4d40 2024-03-23T23:03:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.199+0000 7f1050c00640 1 -- 172.21.15.12:0/3203715816 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f105c02d070 con 0x55a2e33e4d40 2024-03-23T23:03:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.199+0000 7f106cb6c740 10 client.25820 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.199+0000 7f106cb6c740 10 client.25820 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.199+0000 7f106cb6c740 20 client.25820 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.199+0000 7f106cb6c740 1 --2- 172.21.15.12:0/3203715816 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a2e33f5020 0x55a2e33f7410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.199+0000 7f106cb6c740 4 client.25820 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:03:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.199+0000 7f106cb6c740 1 -- 172.21.15.12:0/3203715816 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a2e3351d70 con 0x55a2e33f5020 2024-03-23T23:03:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.199+0000 7f1062a00640 1 --2- 172.21.15.12:0/3203715816 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a2e33f5020 0x55a2e33f7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.200+0000 7f1062a00640 1 --2- 172.21.15.12:0/3203715816 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a2e33f5020 0x55a2e33f7410 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3de570204cb3cdaf server_cookie=364e2c62334ff6fa in_seq=0 out_seq=0 2024-03-23T23:03:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.200+0000 7f1050c00640 10 client.25820 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.200+0000 7f1050c00640 1 -- 172.21.15.12:0/3203715816 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a2e3351d70 con 0x55a2e33f5020 2024-03-23T23:03:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.200+0000 7f1050c00640 10 client.25820 handle_command_reply: tid=0 2024-03-23T23:03:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.200+0000 7f106cb6c740 1 client.25820 shutdown 2024-03-23T23:03:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.201+0000 7f106cb6c740 1 -- 172.21.15.12:0/3203715816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33e4d40 msgr2=0x55a2e33f0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.201+0000 7f106cb6c740 1 --2- 172.21.15.12:0/3203715816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33e4d40 0x55a2e33f0970 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f105c009f20 tx=0x7f105c0118d0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.201+0000 7f106cb6c740 1 -- 172.21.15.12:0/3203715816 shutdown_connections 2024-03-23T23:03:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.201+0000 7f106cb6c740 1 --2- 172.21.15.12:0/3203715816 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a2e33f5020 0x55a2e33f7410 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.201+0000 7f106cb6c740 1 --2- 172.21.15.12:0/3203715816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33e4d40 0x55a2e33f0970 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.201+0000 7f106cb6c740 1 --2- 172.21.15.12:0/3203715816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2e33dfbd0 0x55a2e33e9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.201+0000 7f106cb6c740 1 --2- 172.21.15.12:0/3203715816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e33df1e0 0x55a2e33e9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.201+0000 7f106cb6c740 1 -- 172.21.15.12:0/3203715816 >> 172.21.15.12:0/3203715816 conn(0x55a2e33da750 msgr2=0x55a2e33d9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.201+0000 7f1050c00640 0 client.25820 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:57.206 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.201+0000 7f106cb6c740 1 -- 172.21.15.12:0/3203715816 shutdown_connections 2024-03-23T23:03:57.206 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.201+0000 7f106cb6c740 1 -- 172.21.15.12:0/3203715816 wait complete. 2024-03-23T23:03:57.206 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.201+0000 7f106cb6c740 20 client.25820 trim_cache size 0 max 16384 2024-03-23T23:03:57.217 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.216+0000 7f106cb6c740 1 Processor -- start 2024-03-23T23:03:57.217 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.216+0000 7f106cb6c740 1 -- start start 2024-03-23T23:03:57.217 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.216+0000 7f106cb6c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2e33df1e0 0x55a2e33e0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:57.217 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.216+0000 7f106cb6c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33dfbd0 0x55a2e33f0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:57.217 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.216+0000 7f106cb6c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e33e4d40 0x55a2e33e0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:57.217 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.216+0000 7f106cb6c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10641aaf80 con 0x55a2e33df1e0 2024-03-23T23:03:57.217 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.217+0000 7f106cb6c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10641a8a80 con 0x55a2e33dfbd0 2024-03-23T23:03:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.217+0000 7f106cb6c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10641a6b20 con 0x55a2e33e4d40 2024-03-23T23:03:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.217+0000 7f105a000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33dfbd0 0x55a2e33f0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.217+0000 7f1062a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2e33df1e0 0x55a2e33e0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.217+0000 7f105a000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33dfbd0 0x55a2e33f0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:40918/0 (socket says 172.21.15.12:40918) 2024-03-23T23:03:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.217+0000 7f105a000640 1 -- 172.21.15.12:0/2893911383 learned_addr learned my addr 172.21.15.12:0/2893911383 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.217+0000 7f1063400640 1 --2- 172.21.15.12:0/2893911383 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e33e4d40 0x55a2e33e0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.217+0000 7f105a000640 1 -- 172.21.15.12:0/2893911383 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e33e4d40 msgr2=0x55a2e33e0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.217+0000 7f105a000640 1 --2- 172.21.15.12:0/2893911383 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e33e4d40 0x55a2e33e0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.217+0000 7f105a000640 1 -- 172.21.15.12:0/2893911383 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2e33df1e0 msgr2=0x55a2e33e0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.217+0000 7f105a000640 1 --2- 172.21.15.12:0/2893911383 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2e33df1e0 0x55a2e33e0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.217+0000 7f105a000640 1 -- 172.21.15.12:0/2893911383 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a2e33e9cf0 con 0x55a2e33dfbd0 2024-03-23T23:03:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.219+0000 7f105a000640 1 --2- 172.21.15.12:0/2893911383 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33dfbd0 0x55a2e33f0930 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f104c011760 tx=0x7f104c011c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df0b6a36d0aa4e2d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.219+0000 7f1053400640 1 -- 172.21.15.12:0/2893911383 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c011d50 con 0x55a2e33dfbd0 2024-03-23T23:03:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.219+0000 7f1053400640 1 -- 172.21.15.12:0/2893911383 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f104c010990 con 0x55a2e33dfbd0 2024-03-23T23:03:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.219+0000 7f1053400640 1 -- 172.21.15.12:0/2893911383 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c005580 con 0x55a2e33dfbd0 2024-03-23T23:03:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.219+0000 7f106cb6c740 1 -- 172.21.15.12:0/2893911383 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33dfbd0 msgr2=0x55a2e33f0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.219+0000 7f106cb6c740 1 --2- 172.21.15.12:0/2893911383 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33dfbd0 0x55a2e33f0930 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f104c011760 tx=0x7f104c011c20 comp rx=0 tx=0).stop 2024-03-23T23:03:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.219+0000 7f106cb6c740 1 -- 172.21.15.12:0/2893911383 shutdown_connections 2024-03-23T23:03:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.219+0000 7f106cb6c740 1 --2- 172.21.15.12:0/2893911383 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e33e4d40 0x55a2e33e0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.219+0000 7f106cb6c740 1 --2- 172.21.15.12:0/2893911383 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33dfbd0 0x55a2e33f0930 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.219+0000 7f106cb6c740 1 --2- 172.21.15.12:0/2893911383 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2e33df1e0 0x55a2e33e0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.219+0000 7f106cb6c740 1 -- 172.21.15.12:0/2893911383 >> 172.21.15.12:0/2893911383 conn(0x55a2e33da750 msgr2=0x55a2e33dab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.219+0000 7f106cb6c740 1 -- 172.21.15.12:0/2893911383 shutdown_connections 2024-03-23T23:03:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.219+0000 7f106cb6c740 1 -- 172.21.15.12:0/2893911383 wait complete. 2024-03-23T23:03:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.219+0000 7f106cb6c740 1 -- start start 2024-03-23T23:03:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.219+0000 7f106cb6c740 1 Processor -- start 2024-03-23T23:03:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.221+0000 7f106cb6c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33df1e0 0x55a2e33edf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.221+0000 7f106cb6c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2e33dfbd0 0x55a2e33ee480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.221+0000 7f106cb6c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e33e4d40 0x55a2e33ef300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.221+0000 7f106cb6c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10641a7120 con 0x55a2e33dfbd0 2024-03-23T23:03:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.221+0000 7f106cb6c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10641a8420 con 0x55a2e33df1e0 2024-03-23T23:03:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.221+0000 7f106cb6c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10641a7760 con 0x55a2e33e4d40 2024-03-23T23:03:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.221+0000 7f106cb6c740 10 client.0 _ll_register_callbacks cb 0x7f104c00bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:03:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.221+0000 7f1062a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33df1e0 0x55a2e33edf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.221+0000 7f1062a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33df1e0 0x55a2e33edf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:40932/0 (socket says 172.21.15.12:40932) 2024-03-23T23:03:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.221+0000 7f1062a00640 1 -- 172.21.15.12:0/3053629993 learned_addr learned my addr 172.21.15.12:0/3053629993 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:03:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.221+0000 7f1063400640 1 --2- 172.21.15.12:0/3053629993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e33e4d40 0x55a2e33ef300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.221+0000 7f105a000640 1 --2- 172.21.15.12:0/3053629993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2e33dfbd0 0x55a2e33ee480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.221+0000 7f1062a00640 1 -- 172.21.15.12:0/3053629993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e33e4d40 msgr2=0x55a2e33ef300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.221+0000 7f1062a00640 1 --2- 172.21.15.12:0/3053629993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e33e4d40 0x55a2e33ef300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.221+0000 7f1062a00640 1 -- 172.21.15.12:0/3053629993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2e33dfbd0 msgr2=0x55a2e33ee480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:03:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.221+0000 7f1062a00640 1 --2- 172.21.15.12:0/3053629993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2e33dfbd0 0x55a2e33ee480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.221+0000 7f1062a00640 1 -- 172.21.15.12:0/3053629993 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a2e33ef8f0 con 0x55a2e33df1e0 2024-03-23T23:03:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.222+0000 7f1062a00640 1 --2- 172.21.15.12:0/3053629993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33df1e0 0x55a2e33edf40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f10540bcfd0 tx=0x7f10540035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5eec3a0751fc04a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:03:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.222+0000 7f1050c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:03:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.222+0000 7f1050c00640 1 -- 172.21.15.12:0/3053629993 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105404b040 con 0x55a2e33df1e0 2024-03-23T23:03:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.222+0000 7f106cb6c740 1 -- 172.21.15.12:0/3053629993 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f104c010990 con 0x55a2e33df1e0 2024-03-23T23:03:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.222+0000 7f1050c00640 1 -- 172.21.15.12:0/3053629993 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1054003680 con 0x55a2e33df1e0 2024-03-23T23:03:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.222+0000 7f1050c00640 1 -- 172.21.15.12:0/3053629993 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f10540db070 con 0x55a2e33df1e0 2024-03-23T23:03:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.222+0000 7f106cb6c740 10 client.25826 fetch_fsmap learned FSMap version 23 2024-03-23T23:03:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.222+0000 7f106cb6c740 1 -- 172.21.15.12:0/3053629993 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a2e33f0c70 con 0x55a2e33df1e0 2024-03-23T23:03:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.222+0000 7f1050c00640 1 -- 172.21.15.12:0/3053629993 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f104c010990 con 0x55a2e33df1e0 2024-03-23T23:03:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.222+0000 7f1050c00640 1 -- 172.21.15.12:0/3053629993 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f10540b7bb0 con 0x55a2e33df1e0 2024-03-23T23:03:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.222+0000 7f106cb6c740 10 client.25826 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:03:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.222+0000 7f106cb6c740 10 client.25826 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:03:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.222+0000 7f106cb6c740 20 client.25826 populate_metadata read hostname 'smithi012' 2024-03-23T23:03:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.222+0000 7f106cb6c740 1 --2- 172.21.15.12:0/3053629993 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a2e3450f60 0x55a2e3453350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:03:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.222+0000 7f106cb6c740 4 client.25826 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:03:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.222+0000 7f106cb6c740 1 -- 172.21.15.12:0/3053629993 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a2e334c0c0 con 0x55a2e3450f60 2024-03-23T23:03:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.223+0000 7f105a000640 1 --2- 172.21.15.12:0/3053629993 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a2e3450f60 0x55a2e3453350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:03:57.226 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.223+0000 7f105a000640 1 --2- 172.21.15.12:0/3053629993 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a2e3450f60 0x55a2e3453350 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4fb290b569e8d40 server_cookie=c23f88b4a955396a in_seq=0 out_seq=0 2024-03-23T23:03:57.226 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.223+0000 7f1050c00640 10 client.25826 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.233+0000 7f1050c00640 1 -- 172.21.15.12:0/3053629993 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a2e334c0c0 con 0x55a2e3450f60 2024-03-23T23:03:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.233+0000 7f1050c00640 10 client.25826 handle_command_reply: tid=0 2024-03-23T23:03:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.233+0000 7f106cb6c740 1 client.25826 shutdown 2024-03-23T23:03:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.233+0000 7f106cb6c740 1 -- 172.21.15.12:0/3053629993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33df1e0 msgr2=0x55a2e33edf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.233+0000 7f106cb6c740 1 --2- 172.21.15.12:0/3053629993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33df1e0 0x55a2e33edf40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f10540bcfd0 tx=0x7f10540035f0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.233+0000 7f106cb6c740 1 -- 172.21.15.12:0/3053629993 shutdown_connections 2024-03-23T23:03:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.233+0000 7f106cb6c740 1 --2- 172.21.15.12:0/3053629993 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a2e3450f60 0x55a2e3453350 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.233+0000 7f106cb6c740 1 --2- 172.21.15.12:0/3053629993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e33e4d40 0x55a2e33ef300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.233+0000 7f106cb6c740 1 --2- 172.21.15.12:0/3053629993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2e33dfbd0 0x55a2e33ee480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.233+0000 7f106cb6c740 1 --2- 172.21.15.12:0/3053629993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2e33df1e0 0x55a2e33edf40 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.233+0000 7f1050c00640 0 client.25826 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:03:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.233+0000 7f106cb6c740 1 -- 172.21.15.12:0/3053629993 >> 172.21.15.12:0/3053629993 conn(0x55a2e33da750 msgr2=0x55a2e33e0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.233+0000 7f106cb6c740 1 -- 172.21.15.12:0/3053629993 shutdown_connections 2024-03-23T23:03:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.233+0000 7f106cb6c740 1 -- 172.21.15.12:0/3053629993 wait complete. 2024-03-23T23:03:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.233+0000 7f106cb6c740 20 client.25826 trim_cache size 0 max 16384 2024-03-23T23:03:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:03:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:03:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:03:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:03:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.235+0000 7f1069e00640 1 -- 172.21.15.12:0/2712633311 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f10480a27b0 msgr2=0x7f10480a4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.235+0000 7f1069e00640 1 --2- 172.21.15.12:0/2712633311 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f10480a27b0 0x7f10480a4c70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f105c00ff10 tx=0x7f105c005fd0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.235+0000 7f1069e00640 1 -- 172.21.15.12:0/2712633311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10641016e0 msgr2=0x7f10641979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:03:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.235+0000 7f1069e00640 1 --2- 172.21.15.12:0/2712633311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10641016e0 0x7f10641979e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1054005ab0 tx=0x7f105402c6e0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.235+0000 7f1069e00640 1 -- 172.21.15.12:0/2712633311 shutdown_connections 2024-03-23T23:03:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.235+0000 7f1069e00640 1 --2- 172.21.15.12:0/2712633311 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f10480a27b0 0x7f10480a4c70 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.235+0000 7f1069e00640 1 --2- 172.21.15.12:0/2712633311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1064103290 0x7f106419ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.235+0000 7f1069e00640 1 --2- 172.21.15.12:0/2712633311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10641028d0 0x7f1064197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.235+0000 7f1069e00640 1 --2- 172.21.15.12:0/2712633311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10641016e0 0x7f10641979e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:03:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.235+0000 7f1069e00640 1 -- 172.21.15.12:0/2712633311 >> 172.21.15.12:0/2712633311 conn(0x7f10640fcc30 msgr2=0x7f1064105b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:03:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.235+0000 7f1069e00640 1 -- 172.21.15.12:0/2712633311 shutdown_connections 2024-03-23T23:03:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:03:57.235+0000 7f1069e00640 1 -- 172.21.15.12:0/2712633311 wait complete. 2024-03-23T23:03:58.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:03:58 smithi173 ceph-mon[36116]: pgmap v455: 97 pgs: 97 active+clean; 929 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:03:58.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:03:58 smithi203 ceph-mon[36693]: pgmap v455: 97 pgs: 97 active+clean; 929 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:03:59.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:03:58 smithi012 ceph-mon[31695]: pgmap v455: 97 pgs: 97 active+clean; 929 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:04:00.855 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:00 smithi012 ceph-mon[31695]: pgmap v456: 97 pgs: 97 active+clean; 1000 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:04:00.855 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:00 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:04:00.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:00 smithi173 ceph-mon[36116]: pgmap v456: 97 pgs: 97 active+clean; 1000 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:04:00.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:00 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:04:00.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:00 smithi203 ceph-mon[36693]: pgmap v456: 97 pgs: 97 active+clean; 1000 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:04:00.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:00 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:04:01.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:01 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:04:01.984 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:01 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:04:01.984 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:01 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:04:01.985 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:01 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:04:01.985 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:01 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:04:01.985 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:01 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:04:02.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:01 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:04:02.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:01 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:04:02.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:01 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:04:02.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:02 smithi173 ceph-mon[36116]: pgmap v457: 97 pgs: 97 active+clean; 1000 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:04:02.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:02 smithi203 ceph-mon[36693]: pgmap v457: 97 pgs: 97 active+clean; 1000 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:04:03.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:02 smithi012 ceph-mon[31695]: pgmap v457: 97 pgs: 97 active+clean; 1000 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:04:04.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:04 smithi173 ceph-mon[36116]: pgmap v458: 97 pgs: 97 active+clean; 1.0 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:04:04.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:04 smithi203 ceph-mon[36693]: pgmap v458: 97 pgs: 97 active+clean; 1.0 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:04:05.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:04 smithi012 ceph-mon[31695]: pgmap v458: 97 pgs: 97 active+clean; 1.0 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:04:06.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:06 smithi203 ceph-mon[36693]: pgmap v459: 97 pgs: 97 active+clean; 1.0 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:04:07.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:06 smithi012 ceph-mon[31695]: pgmap v459: 97 pgs: 97 active+clean; 1.0 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:04:07.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:06 smithi173 ceph-mon[36116]: pgmap v459: 97 pgs: 97 active+clean; 1.0 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:04:08.481 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:04:08.481 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:04:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.716+0000 7f92dea00640 1 -- 172.21.15.12:0/2758895438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92d8100480 msgr2=0x7f92d81008a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.716+0000 7f92dea00640 1 --2- 172.21.15.12:0/2758895438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92d8100480 0x7f92d81008a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f92c8009900 tx=0x7f92c802bc30 comp rx=0 tx=0).stop 2024-03-23T23:04:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.716+0000 7f92dea00640 1 -- 172.21.15.12:0/2758895438 shutdown_connections 2024-03-23T23:04:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.716+0000 7f92dea00640 1 --2- 172.21.15.12:0/2758895438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92d8102030 0x7f92d81088c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.716+0000 7f92dea00640 1 --2- 172.21.15.12:0/2758895438 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92d8101670 0x7f92d8101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.716+0000 7f92dea00640 1 --2- 172.21.15.12:0/2758895438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92d8100480 0x7f92d81008a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.716+0000 7f92dea00640 1 -- 172.21.15.12:0/2758895438 >> 172.21.15.12:0/2758895438 conn(0x7f92d80fba10 msgr2=0x7f92d80fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.716+0000 7f92dea00640 1 -- 172.21.15.12:0/2758895438 shutdown_connections 2024-03-23T23:04:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.717+0000 7f92dea00640 1 -- 172.21.15.12:0/2758895438 wait complete. 2024-03-23T23:04:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.717+0000 7f92dea00640 1 Processor -- start 2024-03-23T23:04:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.718+0000 7f92dea00640 1 -- start start 2024-03-23T23:04:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.718+0000 7f92dea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92d8100480 0x7f92d8104410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.718+0000 7f92dea00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92d8101670 0x7f92d8102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.718+0000 7f92dea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92d8102030 0x7f92d8102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.718+0000 7f92dea00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d810c130 con 0x7f92d8101670 2024-03-23T23:04:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.718+0000 7f92dea00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d810bfb0 con 0x7f92d8102030 2024-03-23T23:04:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.718+0000 7f92dea00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d810c2b0 con 0x7f92d8100480 2024-03-23T23:04:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.718+0000 7f92dd600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92d8100480 0x7f92d8104410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.718+0000 7f92de000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92d8102030 0x7f92d8102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.718+0000 7f92dcc00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92d8101670 0x7f92d8102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.718+0000 7f92dd600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92d8100480 0x7f92d8104410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:51596/0 (socket says 172.21.15.12:51596) 2024-03-23T23:04:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.718+0000 7f92dd600640 1 -- 172.21.15.12:0/2288318539 learned_addr learned my addr 172.21.15.12:0/2288318539 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.719+0000 7f92de000640 1 -- 172.21.15.12:0/2288318539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92d8100480 msgr2=0x7f92d8104410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.719+0000 7f92de000640 1 --2- 172.21.15.12:0/2288318539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92d8100480 0x7f92d8104410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.719+0000 7f92de000640 1 -- 172.21.15.12:0/2288318539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92d8101670 msgr2=0x7f92d8102a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.719+0000 7f92de000640 1 --2- 172.21.15.12:0/2288318539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92d8101670 0x7f92d8102a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.719+0000 7f92de000640 1 -- 172.21.15.12:0/2288318539 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d8103820 con 0x7f92d8102030 2024-03-23T23:04:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.719+0000 7f92dcc00640 1 --2- 172.21.15.12:0/2288318539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92d8101670 0x7f92d8102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.719+0000 7f92de000640 1 --2- 172.21.15.12:0/2288318539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92d8102030 0x7f92d8102f90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f92d000e940 tx=0x7f92d000ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.719+0000 7f92d6000640 1 -- 172.21.15.12:0/2288318539 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d0004d20 con 0x7f92d8102030 2024-03-23T23:04:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.720+0000 7f92d6000640 1 -- 172.21.15.12:0/2288318539 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f92d0005020 con 0x7f92d8102030 2024-03-23T23:04:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.720+0000 7f92d6000640 1 -- 172.21.15.12:0/2288318539 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d00116b0 con 0x7f92d8102030 2024-03-23T23:04:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.720+0000 7f92dea00640 1 -- 172.21.15.12:0/2288318539 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92d81a19b0 con 0x7f92d8102030 2024-03-23T23:04:08.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.720+0000 7f92dea00640 1 -- 172.21.15.12:0/2288318539 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92d81a1f00 con 0x7f92d8102030 2024-03-23T23:04:08.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.723+0000 7f92d6000640 1 -- 172.21.15.12:0/2288318539 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f92d00054d0 con 0x7f92d8102030 2024-03-23T23:04:08.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.723+0000 7f92e2d9b740 1 Processor -- start 2024-03-23T23:04:08.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.723+0000 7f92e2d9b740 1 -- start start 2024-03-23T23:04:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.723+0000 7f92e2d9b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa560eb050 0x55fa560eb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.723+0000 7f92e2d9b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa560eba40 0x55fa560f0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.723+0000 7f92e2d9b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa560f0bb0 0x55fa560f3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.723+0000 7f92e2d9b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa56050250 con 0x55fa560eba40 2024-03-23T23:04:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.723+0000 7f92e2d9b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa55ff8a30 con 0x55fa560eb050 2024-03-23T23:04:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.723+0000 7f92e2d9b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa5604c2f0 con 0x55fa560f0bb0 2024-03-23T23:04:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.724+0000 7f92dcc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa560eb050 0x55fa560eb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.724+0000 7f92dcc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa560eb050 0x55fa560eb470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37582/0 (socket says 172.21.15.12:37582) 2024-03-23T23:04:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.724+0000 7f92dcc00640 1 -- 172.21.15.12:0/1968950460 learned_addr learned my addr 172.21.15.12:0/1968950460 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.724+0000 7f92dd600640 1 --2- 172.21.15.12:0/1968950460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa560eba40 0x55fa560f0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.724+0000 7f92dcc00640 1 --2- 172.21.15.12:0/1968950460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa560f0bb0 0x55fa560f3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.724+0000 7f92dcc00640 1 -- 172.21.15.12:0/1968950460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa560f0bb0 msgr2=0x55fa560f3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.724+0000 7f92dcc00640 1 --2- 172.21.15.12:0/1968950460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa560f0bb0 0x55fa560f3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.724+0000 7f92dcc00640 1 -- 172.21.15.12:0/1968950460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa560eba40 msgr2=0x55fa560f0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.724+0000 7f92dcc00640 1 --2- 172.21.15.12:0/1968950460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa560eba40 0x55fa560f0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.724+0000 7f92dcc00640 1 -- 172.21.15.12:0/1968950460 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa55fe3690 con 0x55fa560eb050 2024-03-23T23:04:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.724+0000 7f92d6000640 1 --2- 172.21.15.12:0/2288318539 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f92c00a2880 0x7f92c00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.724+0000 7f92dcc00640 1 --2- 172.21.15.12:0/1968950460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa560eb050 0x55fa560eb470 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f92cc005ca0 tx=0x7f92cc00bbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da5c7ded73305fa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.725+0000 7f92d6000640 1 -- 172.21.15.12:0/2288318539 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f92d00cea80 con 0x7f92d8102030 2024-03-23T23:04:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.725+0000 7f92dd600640 1 --2- 172.21.15.12:0/2288318539 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f92c00a2880 0x7f92c00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.725+0000 7f92c6600640 1 -- 172.21.15.12:0/1968950460 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc012cd0 con 0x55fa560eb050 2024-03-23T23:04:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.725+0000 7f92c6600640 1 -- 172.21.15.12:0/1968950460 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f92cc010c00 con 0x55fa560eb050 2024-03-23T23:04:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.725+0000 7f92c6600640 1 -- 172.21.15.12:0/1968950460 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc003c00 con 0x55fa560eb050 2024-03-23T23:04:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.726+0000 7f92e2d9b740 1 -- 172.21.15.12:0/1968950460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa560eb050 msgr2=0x55fa560eb470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.726+0000 7f92e2d9b740 1 --2- 172.21.15.12:0/1968950460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa560eb050 0x55fa560eb470 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f92cc005ca0 tx=0x7f92cc00bbf0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.726+0000 7f92dd600640 1 --2- 172.21.15.12:0/2288318539 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f92c00a2880 0x7f92c00a4d40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f92c8006010 tx=0x7f92c802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.726+0000 7f92e2d9b740 1 -- 172.21.15.12:0/1968950460 shutdown_connections 2024-03-23T23:04:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.726+0000 7f92e2d9b740 1 --2- 172.21.15.12:0/1968950460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa560f0bb0 0x55fa560f3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.726+0000 7f92e2d9b740 1 --2- 172.21.15.12:0/1968950460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa560eba40 0x55fa560f0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.726+0000 7f92e2d9b740 1 --2- 172.21.15.12:0/1968950460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa560eb050 0x55fa560eb470 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.726+0000 7f92e2d9b740 1 -- 172.21.15.12:0/1968950460 >> 172.21.15.12:0/1968950460 conn(0x55fa560b3200 msgr2=0x55fa560e69c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.726+0000 7f92e2d9b740 1 -- 172.21.15.12:0/1968950460 shutdown_connections 2024-03-23T23:04:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.726+0000 7f92e2d9b740 1 -- 172.21.15.12:0/1968950460 wait complete. 2024-03-23T23:04:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.726+0000 7f92e2d9b740 1 -- start start 2024-03-23T23:04:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.727+0000 7f92e2d9b740 1 Processor -- start 2024-03-23T23:04:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.727+0000 7f92e2d9b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa560eb050 0x55fa560f5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.727+0000 7f92e2d9b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa560eba40 0x55fa560f5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.727+0000 7f92e2d9b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa560f0bb0 0x55fa560fc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.727+0000 7f92e2d9b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa560f3b60 con 0x55fa560eba40 2024-03-23T23:04:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.727+0000 7f92e2d9b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa5600be80 con 0x55fa560f0bb0 2024-03-23T23:04:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.727+0000 7f92e2d9b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa560f3ce0 con 0x55fa560eb050 2024-03-23T23:04:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.727+0000 7f92dd600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa560eb050 0x55fa560f5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.727+0000 7f92e2d9b740 10 client.0 _ll_register_callbacks cb 0x55fa5601e130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.727+0000 7f92de000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa560f0bb0 0x55fa560fc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.727+0000 7f92dd600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa560eb050 0x55fa560f5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:51606/0 (socket says 172.21.15.12:51606) 2024-03-23T23:04:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.727+0000 7f92dd600640 1 -- 172.21.15.12:0/3425026111 learned_addr learned my addr 172.21.15.12:0/3425026111 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.727+0000 7f92dcc00640 1 --2- 172.21.15.12:0/3425026111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa560eba40 0x55fa560f5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.728+0000 7f92de000640 1 -- 172.21.15.12:0/3425026111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa560eb050 msgr2=0x55fa560f5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.728+0000 7f92de000640 1 --2- 172.21.15.12:0/3425026111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa560eb050 0x55fa560f5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.728+0000 7f92de000640 1 -- 172.21.15.12:0/3425026111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa560eba40 msgr2=0x55fa560f5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.728+0000 7f92de000640 1 --2- 172.21.15.12:0/3425026111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa560eba40 0x55fa560f5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.728+0000 7f92de000640 1 -- 172.21.15.12:0/3425026111 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fa55e2e7d0 con 0x55fa560f0bb0 2024-03-23T23:04:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.728+0000 7f92dd600640 1 --2- 172.21.15.12:0/3425026111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa560eb050 0x55fa560f5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.728+0000 7f92de000640 1 --2- 172.21.15.12:0/3425026111 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa560f0bb0 0x55fa560fc5b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f92d0005c40 tx=0x7f92d009aea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d0608b15a32ad6a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.728+0000 7f92baa00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:04:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.728+0000 7f92baa00640 1 -- 172.21.15.12:0/3425026111 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d001f040 con 0x55fa560f0bb0 2024-03-23T23:04:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.728+0000 7f92baa00640 1 -- 172.21.15.12:0/3425026111 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f92d00b4180 con 0x55fa560f0bb0 2024-03-23T23:04:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.728+0000 7f92e2d9b740 1 -- 172.21.15.12:0/3425026111 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f92cc010c00 con 0x55fa560f0bb0 2024-03-23T23:04:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.729+0000 7f92baa00640 1 -- 172.21.15.12:0/3425026111 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f92d009f120 con 0x55fa560f0bb0 2024-03-23T23:04:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.729+0000 7f92e2d9b740 10 client.25838 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.729+0000 7f92e2d9b740 1 -- 172.21.15.12:0/3425026111 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55fa55d643f0 con 0x55fa560f0bb0 2024-03-23T23:04:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.729+0000 7f92baa00640 1 -- 172.21.15.12:0/3425026111 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f92cc010c00 con 0x55fa560f0bb0 2024-03-23T23:04:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.729+0000 7f92baa00640 1 -- 172.21.15.12:0/3425026111 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f92d000f040 con 0x55fa560f0bb0 2024-03-23T23:04:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.729+0000 7f92e2d9b740 10 client.25838 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.729+0000 7f92e2d9b740 10 client.25838 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:04:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.729+0000 7f92e2d9b740 20 client.25838 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.730+0000 7f92e2d9b740 1 --2- 172.21.15.12:0/3425026111 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fa56100e30 0x55fa56103280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.730+0000 7f92e2d9b740 4 client.25838 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:04:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.730+0000 7f92e2d9b740 1 -- 172.21.15.12:0/3425026111 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fa5605d790 con 0x55fa56100e30 2024-03-23T23:04:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.730+0000 7f92dd600640 1 --2- 172.21.15.12:0/3425026111 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fa56100e30 0x55fa56103280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.730+0000 7f92dd600640 1 --2- 172.21.15.12:0/3425026111 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fa56100e30 0x55fa56103280 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7209e62907a54012 server_cookie=a7febc1f69e9391e in_seq=0 out_seq=0 2024-03-23T23:04:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.730+0000 7f92baa00640 10 client.25838 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.732+0000 7f92baa00640 1 -- 172.21.15.12:0/3425026111 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55fa5605d790 con 0x55fa56100e30 2024-03-23T23:04:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.732+0000 7f92baa00640 10 client.25838 handle_command_reply: tid=0 2024-03-23T23:04:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.732+0000 7f92e2d9b740 1 client.25838 shutdown 2024-03-23T23:04:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.732+0000 7f92e2d9b740 1 -- 172.21.15.12:0/3425026111 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa560f0bb0 msgr2=0x55fa560fc5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.732+0000 7f92e2d9b740 1 --2- 172.21.15.12:0/3425026111 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa560f0bb0 0x55fa560fc5b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f92d0005c40 tx=0x7f92d009aea0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.732+0000 7f92e2d9b740 1 -- 172.21.15.12:0/3425026111 shutdown_connections 2024-03-23T23:04:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.732+0000 7f92e2d9b740 1 --2- 172.21.15.12:0/3425026111 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fa56100e30 0x55fa56103280 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.732+0000 7f92e2d9b740 1 --2- 172.21.15.12:0/3425026111 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa560f0bb0 0x55fa560fc5b0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.732+0000 7f92e2d9b740 1 --2- 172.21.15.12:0/3425026111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa560eba40 0x55fa560f5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.732+0000 7f92baa00640 0 client.25838 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.732+0000 7f92e2d9b740 1 --2- 172.21.15.12:0/3425026111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa560eb050 0x55fa560f5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.732+0000 7f92e2d9b740 1 -- 172.21.15.12:0/3425026111 >> 172.21.15.12:0/3425026111 conn(0x55fa560b3200 msgr2=0x55fa560e5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.732+0000 7f92e2d9b740 1 -- 172.21.15.12:0/3425026111 shutdown_connections 2024-03-23T23:04:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.732+0000 7f92e2d9b740 1 -- 172.21.15.12:0/3425026111 wait complete. 2024-03-23T23:04:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.732+0000 7f92e2d9b740 20 client.25838 trim_cache size 0 max 16384 2024-03-23T23:04:08.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.750+0000 7f92e2d9b740 1 Processor -- start 2024-03-23T23:04:08.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.750+0000 7f92e2d9b740 1 -- start start 2024-03-23T23:04:08.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.750+0000 7f92e2d9b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa560eb050 0x55fa560ea310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:08.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.750+0000 7f92e2d9b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa560eba40 0x55fa560fc570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:08.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.750+0000 7f92e2d9b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa560f0bb0 0x55fa560ea850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:08.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.750+0000 7f92e2d9b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d81acee0 con 0x55fa560eb050 2024-03-23T23:04:08.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.750+0000 7f92e2d9b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d81aa9e0 con 0x55fa560f0bb0 2024-03-23T23:04:08.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.750+0000 7f92e2d9b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d81a8a80 con 0x55fa560eba40 2024-03-23T23:04:08.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.751+0000 7f92dd600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa560eb050 0x55fa560ea310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:08.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.751+0000 7f92de000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa560f0bb0 0x55fa560ea850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:08.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.751+0000 7f92de000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa560f0bb0 0x55fa560ea850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37610/0 (socket says 172.21.15.12:37610) 2024-03-23T23:04:08.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.751+0000 7f92de000640 1 -- 172.21.15.12:0/1325740431 learned_addr learned my addr 172.21.15.12:0/1325740431 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:08.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.751+0000 7f92dcc00640 1 --2- 172.21.15.12:0/1325740431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa560eba40 0x55fa560fc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:08.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.751+0000 7f92dd600640 1 -- 172.21.15.12:0/1325740431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa560eba40 msgr2=0x55fa560fc570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:08.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.751+0000 7f92dd600640 1 --2- 172.21.15.12:0/1325740431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa560eba40 0x55fa560fc570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.751+0000 7f92dd600640 1 -- 172.21.15.12:0/1325740431 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa560f0bb0 msgr2=0x55fa560ea850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:08.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.751+0000 7f92dd600640 1 --2- 172.21.15.12:0/1325740431 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa560f0bb0 0x55fa560ea850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.751+0000 7f92dd600640 1 -- 172.21.15.12:0/1325740431 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa55d240f0 con 0x55fa560eb050 2024-03-23T23:04:08.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.751+0000 7f92dcc00640 1 --2- 172.21.15.12:0/1325740431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa560eba40 0x55fa560fc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:04:08.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.751+0000 7f92de000640 1 --2- 172.21.15.12:0/1325740431 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa560f0bb0 0x55fa560ea850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:08.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.752+0000 7f92dd600640 1 --2- 172.21.15.12:0/1325740431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa560eb050 0x55fa560ea310 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f92c802c8e0 tx=0x7f92c8042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45ba22b56791f570 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:08.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.752+0000 7f92c5c00640 1 -- 172.21.15.12:0/1325740431 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c8046070 con 0x55fa560eb050 2024-03-23T23:04:08.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.752+0000 7f92c5c00640 1 -- 172.21.15.12:0/1325740431 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f92c8004c60 con 0x55fa560eb050 2024-03-23T23:04:08.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.752+0000 7f92c5c00640 1 -- 172.21.15.12:0/1325740431 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c8004ef0 con 0x55fa560eb050 2024-03-23T23:04:08.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.753+0000 7f92e2d9b740 1 -- 172.21.15.12:0/1325740431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa560eb050 msgr2=0x55fa560ea310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:08.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.753+0000 7f92e2d9b740 1 --2- 172.21.15.12:0/1325740431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa560eb050 0x55fa560ea310 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f92c802c8e0 tx=0x7f92c8042000 comp rx=0 tx=0).stop 2024-03-23T23:04:08.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.753+0000 7f92e2d9b740 1 -- 172.21.15.12:0/1325740431 shutdown_connections 2024-03-23T23:04:08.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.753+0000 7f92e2d9b740 1 --2- 172.21.15.12:0/1325740431 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa560f0bb0 0x55fa560ea850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.753+0000 7f92e2d9b740 1 --2- 172.21.15.12:0/1325740431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa560eba40 0x55fa560fc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.753+0000 7f92e2d9b740 1 --2- 172.21.15.12:0/1325740431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa560eb050 0x55fa560ea310 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.753+0000 7f92e2d9b740 1 -- 172.21.15.12:0/1325740431 >> 172.21.15.12:0/1325740431 conn(0x55fa560b3200 msgr2=0x55fa560e9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:08.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.753+0000 7f92e2d9b740 1 -- 172.21.15.12:0/1325740431 shutdown_connections 2024-03-23T23:04:08.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.753+0000 7f92e2d9b740 1 -- 172.21.15.12:0/1325740431 wait complete. 2024-03-23T23:04:08.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.753+0000 7f92e2d9b740 1 -- start start 2024-03-23T23:04:08.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.753+0000 7f92e2d9b740 1 Processor -- start 2024-03-23T23:04:08.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.754+0000 7f92e2d9b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa560eb050 0x55fa560fa410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:08.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.754+0000 7f92e2d9b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa560eba40 0x55fa560fa950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:08.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.754+0000 7f92e2d9b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa560f0bb0 0x55fa560fb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:08.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.754+0000 7f92e2d9b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d81a9080 con 0x55fa560eb050 2024-03-23T23:04:08.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.754+0000 7f92e2d9b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d81aa380 con 0x55fa560f0bb0 2024-03-23T23:04:08.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.754+0000 7f92e2d9b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d81a96c0 con 0x55fa560eba40 2024-03-23T23:04:08.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.754+0000 7f92e2d9b740 10 client.0 _ll_register_callbacks cb 0x7f92d0004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:08.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.754+0000 7f92dd600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa560eb050 0x55fa560fa410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:08.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.754+0000 7f92dcc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa560eba40 0x55fa560fa950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:08.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.754+0000 7f92dd600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa560eb050 0x55fa560fa410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53680/0 (socket says 172.21.15.12:53680) 2024-03-23T23:04:08.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.754+0000 7f92dd600640 1 -- 172.21.15.12:0/2102612600 learned_addr learned my addr 172.21.15.12:0/2102612600 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:08.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.754+0000 7f92de000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa560f0bb0 0x55fa560fb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:08.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.754+0000 7f92dcc00640 1 -- 172.21.15.12:0/2102612600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa560f0bb0 msgr2=0x55fa560fb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:08.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.754+0000 7f92dcc00640 1 --2- 172.21.15.12:0/2102612600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa560f0bb0 0x55fa560fb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.754+0000 7f92dcc00640 1 -- 172.21.15.12:0/2102612600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa560eb050 msgr2=0x55fa560fa410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:08.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.754+0000 7f92dcc00640 1 --2- 172.21.15.12:0/2102612600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa560eb050 0x55fa560fa410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.754+0000 7f92dcc00640 1 -- 172.21.15.12:0/2102612600 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fa560fbf50 con 0x55fa560eba40 2024-03-23T23:04:08.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.755+0000 7f92dd600640 1 --2- 172.21.15.12:0/2102612600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa560eb050 0x55fa560fa410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:08.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.755+0000 7f92dcc00640 1 --2- 172.21.15.12:0/2102612600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa560eba40 0x55fa560fa950 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f92cc00bb00 tx=0x7f92cc012f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=484eeaaac8e6658d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:08.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.755+0000 7f92baa00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:04:08.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.755+0000 7f92baa00640 1 -- 172.21.15.12:0/2102612600 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc010de0 con 0x55fa560eba40 2024-03-23T23:04:08.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.755+0000 7f92e2d9b740 1 -- 172.21.15.12:0/2102612600 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f92c8004c60 con 0x55fa560eba40 2024-03-23T23:04:08.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.755+0000 7f92baa00640 1 -- 172.21.15.12:0/2102612600 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f92cc005040 con 0x55fa560eba40 2024-03-23T23:04:08.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.755+0000 7f92baa00640 1 -- 172.21.15.12:0/2102612600 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f92cc005260 con 0x55fa560eba40 2024-03-23T23:04:08.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.755+0000 7f92e2d9b740 10 client.25822 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:08.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.755+0000 7f92e2d9b740 1 -- 172.21.15.12:0/2102612600 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55fa5615ce90 con 0x55fa560eba40 2024-03-23T23:04:08.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.756+0000 7f92baa00640 1 -- 172.21.15.12:0/2102612600 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f92c8004c60 con 0x55fa560eba40 2024-03-23T23:04:08.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.756+0000 7f92baa00640 1 -- 172.21.15.12:0/2102612600 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f92cc01f470 con 0x55fa560eba40 2024-03-23T23:04:08.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.756+0000 7f92e2d9b740 10 client.25822 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:08.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.756+0000 7f92e2d9b740 10 client.25822 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:04:08.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.756+0000 7f92e2d9b740 20 client.25822 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:08.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.756+0000 7f92e2d9b740 1 --2- 172.21.15.12:0/2102612600 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fa5615d580 0x55fa5615f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:08.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.756+0000 7f92e2d9b740 4 client.25822 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:04:08.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.756+0000 7f92e2d9b740 1 -- 172.21.15.12:0/2102612600 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55fa56057ae0 con 0x55fa5615d580 2024-03-23T23:04:08.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.756+0000 7f92dd600640 1 --2- 172.21.15.12:0/2102612600 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fa5615d580 0x55fa5615f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:08.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.757+0000 7f92dd600640 1 --2- 172.21.15.12:0/2102612600 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fa5615d580 0x55fa5615f9a0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e896872a42d7868 server_cookie=d9a5f23854ee5d93 in_seq=0 out_seq=0 2024-03-23T23:04:08.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.757+0000 7f92baa00640 10 client.25822 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:08.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.757+0000 7f92baa00640 1 -- 172.21.15.12:0/2102612600 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55fa56057ae0 con 0x55fa5615d580 2024-03-23T23:04:08.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.757+0000 7f92baa00640 10 client.25822 handle_command_reply: tid=0 2024-03-23T23:04:08.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.757+0000 7f92e2d9b740 1 client.25822 shutdown 2024-03-23T23:04:08.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.757+0000 7f92e2d9b740 1 -- 172.21.15.12:0/2102612600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa560eba40 msgr2=0x55fa560fa950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:08.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.757+0000 7f92e2d9b740 1 --2- 172.21.15.12:0/2102612600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa560eba40 0x55fa560fa950 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f92cc00bb00 tx=0x7f92cc012f00 comp rx=0 tx=0).stop 2024-03-23T23:04:08.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.757+0000 7f92e2d9b740 1 -- 172.21.15.12:0/2102612600 shutdown_connections 2024-03-23T23:04:08.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.757+0000 7f92e2d9b740 1 --2- 172.21.15.12:0/2102612600 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fa5615d580 0x55fa5615f9a0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.757+0000 7f92e2d9b740 1 --2- 172.21.15.12:0/2102612600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa560f0bb0 0x55fa560fb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.757+0000 7f92e2d9b740 1 --2- 172.21.15.12:0/2102612600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa560eba40 0x55fa560fa950 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.757+0000 7f92e2d9b740 1 --2- 172.21.15.12:0/2102612600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa560eb050 0x55fa560fa410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.757+0000 7f92baa00640 0 client.25822 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:08.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.757+0000 7f92e2d9b740 1 -- 172.21.15.12:0/2102612600 >> 172.21.15.12:0/2102612600 conn(0x55fa560b3200 msgr2=0x55fa560ea490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:08.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.757+0000 7f92e2d9b740 1 -- 172.21.15.12:0/2102612600 shutdown_connections 2024-03-23T23:04:08.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.757+0000 7f92e2d9b740 1 -- 172.21.15.12:0/2102612600 wait complete. 2024-03-23T23:04:08.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.757+0000 7f92e2d9b740 20 client.25822 trim_cache size 0 max 16384 2024-03-23T23:04:08.762 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:04:08.762 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-03-23T23:04:08.762 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-03-23T23:04:08.762 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:04:08.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.758+0000 7f92dea00640 1 -- 172.21.15.12:0/2288318539 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f92c00a2880 msgr2=0x7f92c00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:08.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.758+0000 7f92dea00640 1 --2- 172.21.15.12:0/2288318539 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f92c00a2880 0x7f92c00a4d40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f92c8006010 tx=0x7f92c802df90 comp rx=0 tx=0).stop 2024-03-23T23:04:08.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.758+0000 7f92dea00640 1 -- 172.21.15.12:0/2288318539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92d8102030 msgr2=0x7f92d8102f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:08.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.758+0000 7f92dea00640 1 --2- 172.21.15.12:0/2288318539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92d8102030 0x7f92d8102f90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f92d000e940 tx=0x7f92d000ee00 comp rx=0 tx=0).stop 2024-03-23T23:04:08.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.758+0000 7f92dea00640 1 -- 172.21.15.12:0/2288318539 shutdown_connections 2024-03-23T23:04:08.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.758+0000 7f92dea00640 1 --2- 172.21.15.12:0/2288318539 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f92c00a2880 0x7f92c00a4d40 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.758+0000 7f92dea00640 1 --2- 172.21.15.12:0/2288318539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92d8102030 0x7f92d8102f90 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.758+0000 7f92dea00640 1 --2- 172.21.15.12:0/2288318539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92d8101670 0x7f92d8102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.758+0000 7f92dea00640 1 --2- 172.21.15.12:0/2288318539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92d8100480 0x7f92d8104410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:08.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.758+0000 7f92dea00640 1 -- 172.21.15.12:0/2288318539 >> 172.21.15.12:0/2288318539 conn(0x7f92d80fba10 msgr2=0x7f92d80fc090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:08.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.758+0000 7f92dea00640 1 -- 172.21.15.12:0/2288318539 shutdown_connections 2024-03-23T23:04:08.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:08.758+0000 7f92dea00640 1 -- 172.21.15.12:0/2288318539 wait complete. 2024-03-23T23:04:08.777 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T23:04:08.778 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:04:08.864 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:08 smithi012 ceph-mon[31695]: pgmap v460: 97 pgs: 97 active+clean; 1.0 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:04:08.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:08 smithi203 ceph-mon[36693]: pgmap v460: 97 pgs: 97 active+clean; 1.0 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:04:09.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.005+0000 7f0549c00640 1 -- 172.21.15.12:0/2887105769 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05441048e0 msgr2=0x7f0544104d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:09.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.005+0000 7f0549c00640 1 --2- 172.21.15.12:0/2887105769 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05441048e0 0x7f0544104d60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f052c009900 tx=0x7f052c02bc40 comp rx=0 tx=0).stop 2024-03-23T23:04:09.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.006+0000 7f0549c00640 1 -- 172.21.15.12:0/2887105769 shutdown_connections 2024-03-23T23:04:09.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.006+0000 7f0549c00640 1 --2- 172.21.15.12:0/2887105769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05441052a0 0x7f054410bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.006+0000 7f0549c00640 1 --2- 172.21.15.12:0/2887105769 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05441048e0 0x7f0544104d60 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.006+0000 7f0549c00640 1 --2- 172.21.15.12:0/2887105769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05441036f0 0x7f0544103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.006+0000 7f0549c00640 1 -- 172.21.15.12:0/2887105769 >> 172.21.15.12:0/2887105769 conn(0x7f05440fec80 msgr2=0x7f05441010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:09.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.006+0000 7f0549c00640 1 -- 172.21.15.12:0/2887105769 shutdown_connections 2024-03-23T23:04:09.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.006+0000 7f0549c00640 1 -- 172.21.15.12:0/2887105769 wait complete. 2024-03-23T23:04:09.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.007+0000 7f0549c00640 1 Processor -- start 2024-03-23T23:04:09.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.007+0000 7f0549c00640 1 -- start start 2024-03-23T23:04:09.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.007+0000 7f0549c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05441036f0 0x7f05441999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.008+0000 7f0549c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05441048e0 0x7f0544199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.008+0000 7f0549c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05441052a0 0x7f05441a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.008+0000 7f0549c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054410f3a0 con 0x7f05441052a0 2024-03-23T23:04:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.008+0000 7f0549c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054410f220 con 0x7f05441036f0 2024-03-23T23:04:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.008+0000 7f0549c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054410f520 con 0x7f05441048e0 2024-03-23T23:04:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.008+0000 7f0543e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05441036f0 0x7f05441999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.008+0000 7f0549200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05441052a0 0x7f05441a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.008+0000 7f0543400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05441048e0 0x7f0544199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.008+0000 7f0543400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05441048e0 0x7f0544199ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:51636/0 (socket says 172.21.15.12:51636) 2024-03-23T23:04:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.008+0000 7f0543400640 1 -- 172.21.15.12:0/3284722715 learned_addr learned my addr 172.21.15.12:0/3284722715 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.008+0000 7f0543e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05441036f0 0x7f05441999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37632/0 (socket says 172.21.15.12:37632) 2024-03-23T23:04:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.008+0000 7f0543e00640 1 -- 172.21.15.12:0/3284722715 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05441048e0 msgr2=0x7f0544199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.008+0000 7f0543e00640 1 --2- 172.21.15.12:0/3284722715 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05441048e0 0x7f0544199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.008+0000 7f0543e00640 1 -- 172.21.15.12:0/3284722715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05441052a0 msgr2=0x7f05441a0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.008+0000 7f0543e00640 1 --2- 172.21.15.12:0/3284722715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05441052a0 0x7f05441a0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.008+0000 7f0543e00640 1 -- 172.21.15.12:0/3284722715 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05441a15d0 con 0x7f05441036f0 2024-03-23T23:04:09.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.008+0000 7f0543400640 1 --2- 172.21.15.12:0/3284722715 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05441048e0 0x7f0544199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:09.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.009+0000 7f0543e00640 1 --2- 172.21.15.12:0/3284722715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05441036f0 0x7f05441999b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0534005fd0 tx=0x7f053400ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:09.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.009+0000 7f0540c00640 1 -- 172.21.15.12:0/3284722715 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0534003970 con 0x7f05441036f0 2024-03-23T23:04:09.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.009+0000 7f0540c00640 1 -- 172.21.15.12:0/3284722715 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0534004030 con 0x7f05441036f0 2024-03-23T23:04:09.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.009+0000 7f0549c00640 1 -- 172.21.15.12:0/3284722715 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05441a18c0 con 0x7f05441036f0 2024-03-23T23:04:09.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.009+0000 7f0540c00640 1 -- 172.21.15.12:0/3284722715 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05340107f0 con 0x7f05441036f0 2024-03-23T23:04:09.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.009+0000 7f0549c00640 1 -- 172.21.15.12:0/3284722715 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05441a1de0 con 0x7f05441036f0 2024-03-23T23:04:09.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.010+0000 7f0549c00640 1 -- 172.21.15.12:0/3284722715 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f054410a9b0 con 0x7f05441036f0 2024-03-23T23:04:09.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.012+0000 7f0540c00640 1 -- 172.21.15.12:0/3284722715 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f05340041e0 con 0x7f05441036f0 2024-03-23T23:04:09.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.013+0000 7f0540c00640 1 --2- 172.21.15.12:0/3284722715 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f05280a2800 0x7f05280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.013+0000 7f0540c00640 1 -- 172.21.15.12:0/3284722715 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f05340cf2b0 con 0x7f05441036f0 2024-03-23T23:04:09.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.014+0000 7f0543400640 1 --2- 172.21.15.12:0/3284722715 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f05280a2800 0x7f05280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.014+0000 7f0543400640 1 --2- 172.21.15.12:0/3284722715 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f05280a2800 0x7f05280a4cc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f052c006010 tx=0x7f052c02cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:09.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.018+0000 7f0540c00640 1 -- 172.21.15.12:0/3284722715 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f053409c300 con 0x7f05441036f0 2024-03-23T23:04:09.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:08 smithi173 ceph-mon[36116]: pgmap v460: 97 pgs: 97 active+clean; 1.0 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:04:09.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.265+0000 7f0549c00640 1 -- 172.21.15.12:0/3284722715 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f054402d070 con 0x7f05441036f0 2024-03-23T23:04:09.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.266+0000 7f0540c00640 1 -- 172.21.15.12:0/3284722715 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f053409bc00 con 0x7f05441036f0 2024-03-23T23:04:09.266 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:04:09.266 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:04:09.266 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:04:09.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.269+0000 7f0549c00640 1 -- 172.21.15.12:0/3284722715 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f05280a2800 msgr2=0x7f05280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:09.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.269+0000 7f0549c00640 1 --2- 172.21.15.12:0/3284722715 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f05280a2800 0x7f05280a4cc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f052c006010 tx=0x7f052c02cf90 comp rx=0 tx=0).stop 2024-03-23T23:04:09.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.269+0000 7f0549c00640 1 -- 172.21.15.12:0/3284722715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05441036f0 msgr2=0x7f05441999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:09.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.269+0000 7f0549c00640 1 --2- 172.21.15.12:0/3284722715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05441036f0 0x7f05441999b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0534005fd0 tx=0x7f053400ef90 comp rx=0 tx=0).stop 2024-03-23T23:04:09.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.269+0000 7f0549c00640 1 -- 172.21.15.12:0/3284722715 shutdown_connections 2024-03-23T23:04:09.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.269+0000 7f0549c00640 1 --2- 172.21.15.12:0/3284722715 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f05280a2800 0x7f05280a4cc0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.269+0000 7f0549c00640 1 --2- 172.21.15.12:0/3284722715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05441052a0 0x7f05441a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.269+0000 7f0549c00640 1 --2- 172.21.15.12:0/3284722715 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05441048e0 0x7f0544199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.269+0000 7f0549c00640 1 --2- 172.21.15.12:0/3284722715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05441036f0 0x7f05441999b0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.269+0000 7f0549c00640 1 -- 172.21.15.12:0/3284722715 >> 172.21.15.12:0/3284722715 conn(0x7f05440fec80 msgr2=0x7f0544107b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:09.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.269+0000 7f0549c00640 1 -- 172.21.15.12:0/3284722715 shutdown_connections 2024-03-23T23:04:09.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.270+0000 7f0549c00640 1 -- 172.21.15.12:0/3284722715 wait complete. 2024-03-23T23:04:09.287 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:04:09.287 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-23T23:04:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.514+0000 7fc26dc00640 1 -- 172.21.15.12:0/2560706492 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc268102790 msgr2=0x7fc268102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.514+0000 7fc26dc00640 1 --2- 172.21.15.12:0/2560706492 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc268102790 0x7fc268102c10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc258009be0 tx=0x7fc25802bef0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.515+0000 7fc26dc00640 1 -- 172.21.15.12:0/2560706492 shutdown_connections 2024-03-23T23:04:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.515+0000 7fc26dc00640 1 --2- 172.21.15.12:0/2560706492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc268103150 0x7fc2681099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.515+0000 7fc26dc00640 1 --2- 172.21.15.12:0/2560706492 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc268102790 0x7fc268102c10 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.515+0000 7fc26dc00640 1 --2- 172.21.15.12:0/2560706492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2681015a0 0x7fc2681019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.515+0000 7fc26dc00640 1 -- 172.21.15.12:0/2560706492 >> 172.21.15.12:0/2560706492 conn(0x7fc2680fcb10 msgr2=0x7fc2680fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.515+0000 7fc26dc00640 1 -- 172.21.15.12:0/2560706492 shutdown_connections 2024-03-23T23:04:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.515+0000 7fc26dc00640 1 -- 172.21.15.12:0/2560706492 wait complete. 2024-03-23T23:04:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.516+0000 7fc26dc00640 1 Processor -- start 2024-03-23T23:04:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.516+0000 7fc26dc00640 1 -- start start 2024-03-23T23:04:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.517+0000 7fc26dc00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2681015a0 0x7fc268199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.517+0000 7fc26dc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc268102790 0x7fc268199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.517+0000 7fc26dc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc268103150 0x7fc2681a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.517+0000 7fc26dc00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26810d270 con 0x7fc2681015a0 2024-03-23T23:04:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.517+0000 7fc26dc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26810d0f0 con 0x7fc268102790 2024-03-23T23:04:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.517+0000 7fc26dc00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26810d3f0 con 0x7fc268103150 2024-03-23T23:04:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.517+0000 7fc267e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2681015a0 0x7fc268199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.517+0000 7fc267400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc268102790 0x7fc268199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.517+0000 7fc267e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2681015a0 0x7fc268199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53712/0 (socket says 172.21.15.12:53712) 2024-03-23T23:04:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.517+0000 7fc267e00640 1 -- 172.21.15.12:0/3008498625 learned_addr learned my addr 172.21.15.12:0/3008498625 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.517+0000 7fc26d200640 1 --2- 172.21.15.12:0/3008498625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc268103150 0x7fc2681a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.517+0000 7fc267400640 1 -- 172.21.15.12:0/3008498625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc268103150 msgr2=0x7fc2681a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.517+0000 7fc267400640 1 --2- 172.21.15.12:0/3008498625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc268103150 0x7fc2681a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.517+0000 7fc267400640 1 -- 172.21.15.12:0/3008498625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2681015a0 msgr2=0x7fc268199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:09.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.517+0000 7fc267400640 1 --2- 172.21.15.12:0/3008498625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2681015a0 0x7fc268199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.517+0000 7fc267400640 1 -- 172.21.15.12:0/3008498625 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2681a16d0 con 0x7fc268102790 2024-03-23T23:04:09.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.518+0000 7fc267400640 1 --2- 172.21.15.12:0/3008498625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc268102790 0x7fc268199ff0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc25802c500 tx=0x7fc258033580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:09.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.518+0000 7fc264c00640 1 -- 172.21.15.12:0/3008498625 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc258043020 con 0x7fc268102790 2024-03-23T23:04:09.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.518+0000 7fc264c00640 1 -- 172.21.15.12:0/3008498625 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc258003e50 con 0x7fc268102790 2024-03-23T23:04:09.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.518+0000 7fc264c00640 1 -- 172.21.15.12:0/3008498625 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc258034690 con 0x7fc268102790 2024-03-23T23:04:09.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.518+0000 7fc26dc00640 1 -- 172.21.15.12:0/3008498625 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2681a1960 con 0x7fc268102790 2024-03-23T23:04:09.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.518+0000 7fc26dc00640 1 -- 172.21.15.12:0/3008498625 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2681a1e60 con 0x7fc268102790 2024-03-23T23:04:09.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.521+0000 7fc264c00640 1 -- 172.21.15.12:0/3008498625 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc258046020 con 0x7fc268102790 2024-03-23T23:04:09.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.522+0000 7fc271f80740 1 Processor -- start 2024-03-23T23:04:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.522+0000 7fc271f80740 1 -- start start 2024-03-23T23:04:09.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.522+0000 7fc271f80740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55816b99f9f0 0x55816b99fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.522+0000 7fc271f80740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55816b9a0bc0 0x55816b9a1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.522+0000 7fc271f80740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55816b9a1580 0x55816b9a7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.522+0000 7fc271f80740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55816b9042c0 con 0x55816b9a0bc0 2024-03-23T23:04:09.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.522+0000 7fc271f80740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55816b8acc90 con 0x55816b99f9f0 2024-03-23T23:04:09.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.522+0000 7fc271f80740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55816b900360 con 0x55816b9a1580 2024-03-23T23:04:09.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.523+0000 7fc26d200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55816b99f9f0 0x55816b99fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.523+0000 7fc26d200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55816b99f9f0 0x55816b99fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37652/0 (socket says 172.21.15.12:37652) 2024-03-23T23:04:09.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.523+0000 7fc26d200640 1 -- 172.21.15.12:0/3407320921 learned_addr learned my addr 172.21.15.12:0/3407320921 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:09.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.523+0000 7fc267e00640 1 --2- 172.21.15.12:0/3407320921 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55816b9a0bc0 0x55816b9a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.523+0000 7fc267400640 1 --2- 172.21.15.12:0/3407320921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55816b9a1580 0x55816b9a7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.523+0000 7fc264c00640 1 --2- 172.21.15.12:0/3008498625 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc24c0a2880 0x7fc24c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.523+0000 7fc264c00640 1 -- 172.21.15.12:0/3008498625 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fc2580f1080 con 0x7fc268102790 2024-03-23T23:04:09.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.523+0000 7fc26d200640 1 -- 172.21.15.12:0/3407320921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55816b9a1580 msgr2=0x55816b9a7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:09.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.523+0000 7fc26d200640 1 --2- 172.21.15.12:0/3407320921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55816b9a1580 0x55816b9a7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.523+0000 7fc26d200640 1 -- 172.21.15.12:0/3407320921 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55816b9a0bc0 msgr2=0x55816b9a1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:09.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.523+0000 7fc26d200640 1 --2- 172.21.15.12:0/3407320921 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55816b9a0bc0 0x55816b9a1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.523+0000 7fc26d200640 1 -- 172.21.15.12:0/3407320921 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55816b8d1080 con 0x55816b99f9f0 2024-03-23T23:04:09.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.523+0000 7fc26d200640 1 --2- 172.21.15.12:0/3008498625 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc24c0a2880 0x7fc24c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.524+0000 7fc26d200640 1 --2- 172.21.15.12:0/3407320921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55816b99f9f0 0x55816b99fe10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc260009e20 tx=0x7fc260011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2cb7d9270175a80 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:09.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.524+0000 7fc25d600640 1 -- 172.21.15.12:0/3407320921 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc260017050 con 0x55816b99f9f0 2024-03-23T23:04:09.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.524+0000 7fc25d600640 1 -- 172.21.15.12:0/3407320921 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc260009e50 con 0x55816b99f9f0 2024-03-23T23:04:09.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.524+0000 7fc25d600640 1 -- 172.21.15.12:0/3407320921 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26000e3b0 con 0x55816b99f9f0 2024-03-23T23:04:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.525+0000 7fc271f80740 1 -- 172.21.15.12:0/3407320921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55816b99f9f0 msgr2=0x55816b99fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.525+0000 7fc271f80740 1 --2- 172.21.15.12:0/3407320921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55816b99f9f0 0x55816b99fe10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc260009e20 tx=0x7fc260011810 comp rx=0 tx=0).stop 2024-03-23T23:04:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.525+0000 7fc26d200640 1 --2- 172.21.15.12:0/3008498625 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc24c0a2880 0x7fc24c0a4d40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc260023000 tx=0x7fc260024000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.525+0000 7fc271f80740 1 -- 172.21.15.12:0/3407320921 shutdown_connections 2024-03-23T23:04:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.525+0000 7fc271f80740 1 --2- 172.21.15.12:0/3407320921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55816b9a1580 0x55816b9a7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.525+0000 7fc271f80740 1 --2- 172.21.15.12:0/3407320921 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55816b9a0bc0 0x55816b9a1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.525+0000 7fc271f80740 1 --2- 172.21.15.12:0/3407320921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55816b99f9f0 0x55816b99fe10 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.525+0000 7fc271f80740 1 -- 172.21.15.12:0/3407320921 >> 172.21.15.12:0/3407320921 conn(0x55816b99ade0 msgr2=0x55816b99b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.525+0000 7fc271f80740 1 -- 172.21.15.12:0/3407320921 shutdown_connections 2024-03-23T23:04:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.525+0000 7fc271f80740 1 -- 172.21.15.12:0/3407320921 wait complete. 2024-03-23T23:04:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.525+0000 7fc271f80740 1 -- start start 2024-03-23T23:04:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.526+0000 7fc271f80740 1 Processor -- start 2024-03-23T23:04:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.526+0000 7fc271f80740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55816b99f9f0 0x55816b9a9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.526+0000 7fc271f80740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55816b9a0bc0 0x55816b9aa190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.526+0000 7fc271f80740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55816b9a1580 0x55816b9b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.526+0000 7fc271f80740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55816b9a8590 con 0x55816b99f9f0 2024-03-23T23:04:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.526+0000 7fc271f80740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55816b8c00d0 con 0x55816b9a1580 2024-03-23T23:04:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.526+0000 7fc271f80740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55816b9a8710 con 0x55816b9a0bc0 2024-03-23T23:04:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.526+0000 7fc271f80740 10 client.0 _ll_register_callbacks cb 0x55816b8c4e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.526+0000 7fc267400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55816b9a0bc0 0x55816b9aa190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.526+0000 7fc267400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55816b9a0bc0 0x55816b9aa190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:51658/0 (socket says 172.21.15.12:51658) 2024-03-23T23:04:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.526+0000 7fc26d200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55816b9a1580 0x55816b9b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.526+0000 7fc267400640 1 -- 172.21.15.12:0/669287058 learned_addr learned my addr 172.21.15.12:0/669287058 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.526+0000 7fc267e00640 1 --2- 172.21.15.12:0/669287058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55816b99f9f0 0x55816b9a9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.527+0000 7fc267400640 1 -- 172.21.15.12:0/669287058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55816b9a1580 msgr2=0x55816b9b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.527+0000 7fc267400640 1 --2- 172.21.15.12:0/669287058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55816b9a1580 0x55816b9b11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.527+0000 7fc267400640 1 -- 172.21.15.12:0/669287058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55816b99f9f0 msgr2=0x55816b9a9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.527+0000 7fc267400640 1 --2- 172.21.15.12:0/669287058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55816b99f9f0 0x55816b9a9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.527+0000 7fc267400640 1 -- 172.21.15.12:0/669287058 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55816b6183f0 con 0x55816b9a0bc0 2024-03-23T23:04:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.527+0000 7fc267400640 1 --2- 172.21.15.12:0/669287058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55816b9a0bc0 0x55816b9aa190 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc2580b9e30 tx=0x7fc258009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f16831d419a1ad0a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.527+0000 7fc256000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:04:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.527+0000 7fc256000640 1 -- 172.21.15.12:0/669287058 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2580c1b20 con 0x55816b9a0bc0 2024-03-23T23:04:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.527+0000 7fc271f80740 1 -- 172.21.15.12:0/669287058 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc260009e50 con 0x55816b9a0bc0 2024-03-23T23:04:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.527+0000 7fc256000640 1 -- 172.21.15.12:0/669287058 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fc2580042d0 con 0x55816b9a0bc0 2024-03-23T23:04:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.528+0000 7fc256000640 1 -- 172.21.15.12:0/669287058 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2580c6170 con 0x55816b9a0bc0 2024-03-23T23:04:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.528+0000 7fc271f80740 10 client.25840 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.528+0000 7fc271f80740 1 -- 172.21.15.12:0/669287058 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55816b6e27d0 con 0x55816b9a0bc0 2024-03-23T23:04:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.528+0000 7fc256000640 1 -- 172.21.15.12:0/669287058 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc260009e50 con 0x55816b9a0bc0 2024-03-23T23:04:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.528+0000 7fc256000640 1 -- 172.21.15.12:0/669287058 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fc2580bce20 con 0x55816b9a0bc0 2024-03-23T23:04:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.528+0000 7fc271f80740 10 client.25840 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.528+0000 7fc271f80740 10 client.25840 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:04:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.528+0000 7fc271f80740 20 client.25840 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.528+0000 7fc271f80740 1 --2- 172.21.15.12:0/669287058 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55816b9b5600 0x55816b9b7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.528+0000 7fc271f80740 4 client.25840 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:04:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.528+0000 7fc271f80740 1 -- 172.21.15.12:0/669287058 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55816b911800 con 0x55816b9b5600 2024-03-23T23:04:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.528+0000 7fc267e00640 1 --2- 172.21.15.12:0/669287058 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55816b9b5600 0x55816b9b7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.529+0000 7fc267e00640 1 --2- 172.21.15.12:0/669287058 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55816b9b5600 0x55816b9b7a60 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bd63776a24ded3be server_cookie=c38f459bd80c0a88 in_seq=0 out_seq=0 2024-03-23T23:04:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.529+0000 7fc256000640 10 client.25840 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:04:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.530+0000 7fc256000640 1 -- 172.21.15.12:0/669287058 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55816b911800 con 0x55816b9b5600 2024-03-23T23:04:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.530+0000 7fc256000640 10 client.25840 handle_command_reply: tid=0 2024-03-23T23:04:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.530+0000 7fc271f80740 1 client.25840 shutdown 2024-03-23T23:04:09.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.530+0000 7fc271f80740 1 -- 172.21.15.12:0/669287058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55816b9a0bc0 msgr2=0x55816b9aa190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:09.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.530+0000 7fc271f80740 1 --2- 172.21.15.12:0/669287058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55816b9a0bc0 0x55816b9aa190 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc2580b9e30 tx=0x7fc258009500 comp rx=0 tx=0).stop 2024-03-23T23:04:09.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.530+0000 7fc271f80740 1 -- 172.21.15.12:0/669287058 shutdown_connections 2024-03-23T23:04:09.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.530+0000 7fc271f80740 1 --2- 172.21.15.12:0/669287058 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55816b9b5600 0x55816b9b7a60 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.530+0000 7fc271f80740 1 --2- 172.21.15.12:0/669287058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55816b9a1580 0x55816b9b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.530+0000 7fc271f80740 1 --2- 172.21.15.12:0/669287058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55816b9a0bc0 0x55816b9aa190 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.530+0000 7fc271f80740 1 --2- 172.21.15.12:0/669287058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55816b99f9f0 0x55816b9a9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.530+0000 7fc256000640 0 client.25840 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:04:09.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.530+0000 7fc271f80740 1 -- 172.21.15.12:0/669287058 >> 172.21.15.12:0/669287058 conn(0x55816b99ade0 msgr2=0x55816b99a160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:09.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.530+0000 7fc271f80740 1 -- 172.21.15.12:0/669287058 shutdown_connections 2024-03-23T23:04:09.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.530+0000 7fc271f80740 1 -- 172.21.15.12:0/669287058 wait complete. 2024-03-23T23:04:09.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.530+0000 7fc271f80740 20 client.25840 trim_cache size 0 max 16384 2024-03-23T23:04:09.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.544+0000 7fc271f80740 1 Processor -- start 2024-03-23T23:04:09.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.544+0000 7fc271f80740 1 -- start start 2024-03-23T23:04:09.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.544+0000 7fc271f80740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55816b99f9f0 0x55816b99e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.544+0000 7fc271f80740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55816b9a0bc0 0x55816b9b1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.544+0000 7fc271f80740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55816b9a1580 0x55816b9a3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.544+0000 7fc271f80740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2681acf80 con 0x55816b9a0bc0 2024-03-23T23:04:09.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.544+0000 7fc271f80740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2681aaa80 con 0x55816b9a1580 2024-03-23T23:04:09.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.544+0000 7fc271f80740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2681a8b20 con 0x55816b99f9f0 2024-03-23T23:04:09.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.545+0000 7fc26d200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55816b9a1580 0x55816b9a3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.545+0000 7fc267400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55816b9a0bc0 0x55816b9b1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.545+0000 7fc26d200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55816b9a1580 0x55816b9a3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37672/0 (socket says 172.21.15.12:37672) 2024-03-23T23:04:09.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.545+0000 7fc26d200640 1 -- 172.21.15.12:0/3433194774 learned_addr learned my addr 172.21.15.12:0/3433194774 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:09.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.545+0000 7fc267e00640 1 --2- 172.21.15.12:0/3433194774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55816b99f9f0 0x55816b99e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.545+0000 7fc26d200640 1 -- 172.21.15.12:0/3433194774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55816b99f9f0 msgr2=0x55816b99e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:09.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.545+0000 7fc26d200640 1 --2- 172.21.15.12:0/3433194774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55816b99f9f0 0x55816b99e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.545+0000 7fc26d200640 1 -- 172.21.15.12:0/3433194774 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55816b9a0bc0 msgr2=0x55816b9b1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:09.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.545+0000 7fc26d200640 1 --2- 172.21.15.12:0/3433194774 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55816b9a0bc0 0x55816b9b1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.545+0000 7fc26d200640 1 -- 172.21.15.12:0/3433194774 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55816b9b80e0 con 0x55816b9a1580 2024-03-23T23:04:09.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.545+0000 7fc267400640 1 --2- 172.21.15.12:0/3433194774 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55816b9a0bc0 0x55816b9b1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:09.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.545+0000 7fc267e00640 1 --2- 172.21.15.12:0/3433194774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55816b99f9f0 0x55816b99e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:09.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.546+0000 7fc26d200640 1 --2- 172.21.15.12:0/3433194774 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55816b9a1580 0x55816b9a3df0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc260014040 tx=0x7fc26001e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf8662412124056 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:09.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.546+0000 7fc25cc00640 1 -- 172.21.15.12:0/3433194774 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc260038070 con 0x55816b9a1580 2024-03-23T23:04:09.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.546+0000 7fc25cc00640 1 -- 172.21.15.12:0/3433194774 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc260009780 con 0x55816b9a1580 2024-03-23T23:04:09.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.546+0000 7fc25cc00640 1 -- 172.21.15.12:0/3433194774 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc260004540 con 0x55816b9a1580 2024-03-23T23:04:09.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.547+0000 7fc271f80740 1 -- 172.21.15.12:0/3433194774 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55816b9a1580 msgr2=0x55816b9a3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:09.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.547+0000 7fc271f80740 1 --2- 172.21.15.12:0/3433194774 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55816b9a1580 0x55816b9a3df0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc260014040 tx=0x7fc26001e000 comp rx=0 tx=0).stop 2024-03-23T23:04:09.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.547+0000 7fc271f80740 1 -- 172.21.15.12:0/3433194774 shutdown_connections 2024-03-23T23:04:09.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.547+0000 7fc271f80740 1 --2- 172.21.15.12:0/3433194774 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55816b9a1580 0x55816b9a3df0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.547+0000 7fc271f80740 1 --2- 172.21.15.12:0/3433194774 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55816b9a0bc0 0x55816b9b1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.547+0000 7fc271f80740 1 --2- 172.21.15.12:0/3433194774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55816b99f9f0 0x55816b99e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.547+0000 7fc271f80740 1 -- 172.21.15.12:0/3433194774 >> 172.21.15.12:0/3433194774 conn(0x55816b99ade0 msgr2=0x55816b99e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:09.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.547+0000 7fc271f80740 1 -- 172.21.15.12:0/3433194774 shutdown_connections 2024-03-23T23:04:09.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.547+0000 7fc271f80740 1 -- 172.21.15.12:0/3433194774 wait complete. 2024-03-23T23:04:09.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.547+0000 7fc271f80740 1 -- start start 2024-03-23T23:04:09.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.548+0000 7fc271f80740 1 Processor -- start 2024-03-23T23:04:09.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.548+0000 7fc271f80740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55816b99f9f0 0x55816b9af150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.548+0000 7fc271f80740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55816b9a0bc0 0x55816b9af690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.548+0000 7fc271f80740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55816b9a1580 0x55816b9b06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.548+0000 7fc271f80740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2681a9120 con 0x55816b9a0bc0 2024-03-23T23:04:09.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.548+0000 7fc271f80740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2681aa420 con 0x55816b9a1580 2024-03-23T23:04:09.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.548+0000 7fc271f80740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2681a9760 con 0x55816b99f9f0 2024-03-23T23:04:09.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.548+0000 7fc271f80740 10 client.0 _ll_register_callbacks cb 0x7fc258009840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:09.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.548+0000 7fc267e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55816b99f9f0 0x55816b9af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.548+0000 7fc267e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55816b99f9f0 0x55816b9af150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:51684/0 (socket says 172.21.15.12:51684) 2024-03-23T23:04:09.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.548+0000 7fc267e00640 1 -- 172.21.15.12:0/739062932 learned_addr learned my addr 172.21.15.12:0/739062932 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:09.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.548+0000 7fc267400640 1 --2- 172.21.15.12:0/739062932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55816b9a0bc0 0x55816b9af690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.549+0000 7fc26d200640 1 --2- 172.21.15.12:0/739062932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55816b9a1580 0x55816b9b06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.549+0000 7fc267e00640 1 -- 172.21.15.12:0/739062932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55816b9a1580 msgr2=0x55816b9b06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:09.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.549+0000 7fc267e00640 1 --2- 172.21.15.12:0/739062932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55816b9a1580 0x55816b9b06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.549+0000 7fc267e00640 1 -- 172.21.15.12:0/739062932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55816b9a0bc0 msgr2=0x55816b9af690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:09.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.549+0000 7fc267e00640 1 --2- 172.21.15.12:0/739062932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55816b9a0bc0 0x55816b9af690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.549+0000 7fc267e00640 1 -- 172.21.15.12:0/739062932 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55816b9b0c60 con 0x55816b99f9f0 2024-03-23T23:04:09.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.549+0000 7fc267e00640 1 --2- 172.21.15.12:0/739062932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55816b99f9f0 0x55816b9af150 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x55816b9b82b0 tx=0x7fc2500035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=917fb0686630d87f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:09.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.549+0000 7fc256000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:04:09.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.549+0000 7fc256000640 1 -- 172.21.15.12:0/739062932 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc250023070 con 0x55816b99f9f0 2024-03-23T23:04:09.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.549+0000 7fc256000640 1 -- 172.21.15.12:0/739062932 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fc25001f270 con 0x55816b99f9f0 2024-03-23T23:04:09.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.549+0000 7fc271f80740 1 -- 172.21.15.12:0/739062932 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc260009780 con 0x55816b99f9f0 2024-03-23T23:04:09.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.550+0000 7fc256000640 1 -- 172.21.15.12:0/739062932 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc25002e050 con 0x55816b99f9f0 2024-03-23T23:04:09.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.550+0000 7fc271f80740 10 client.25846 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:09.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.550+0000 7fc271f80740 1 -- 172.21.15.12:0/739062932 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55816ba118c0 con 0x55816b99f9f0 2024-03-23T23:04:09.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.550+0000 7fc256000640 1 -- 172.21.15.12:0/739062932 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc260009780 con 0x55816b99f9f0 2024-03-23T23:04:09.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.550+0000 7fc256000640 1 -- 172.21.15.12:0/739062932 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fc2500109e0 con 0x55816b99f9f0 2024-03-23T23:04:09.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.550+0000 7fc271f80740 10 client.25846 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:09.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.550+0000 7fc271f80740 10 client.25846 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:04:09.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.550+0000 7fc271f80740 20 client.25846 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:09.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.550+0000 7fc271f80740 1 --2- 172.21.15.12:0/739062932 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55816ba12170 0x55816ba14590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.550+0000 7fc271f80740 4 client.25846 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:04:09.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.550+0000 7fc271f80740 1 -- 172.21.15.12:0/739062932 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55816b90bb50 con 0x55816ba12170 2024-03-23T23:04:09.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.550+0000 7fc267400640 1 --2- 172.21.15.12:0/739062932 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55816ba12170 0x55816ba14590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.551+0000 7fc267400640 1 --2- 172.21.15.12:0/739062932 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55816ba12170 0x55816ba14590 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fa3c3de5291a27a8 server_cookie=90b96e2b38fc396c in_seq=0 out_seq=0 2024-03-23T23:04:09.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.551+0000 7fc256000640 10 client.25846 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:04:09.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.551+0000 7fc256000640 1 -- 172.21.15.12:0/739062932 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55816b90bb50 con 0x55816ba12170 2024-03-23T23:04:09.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.551+0000 7fc256000640 10 client.25846 handle_command_reply: tid=0 2024-03-23T23:04:09.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.551+0000 7fc271f80740 1 client.25846 shutdown 2024-03-23T23:04:09.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.551+0000 7fc271f80740 1 -- 172.21.15.12:0/739062932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55816b99f9f0 msgr2=0x55816b9af150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:09.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.551+0000 7fc271f80740 1 --2- 172.21.15.12:0/739062932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55816b99f9f0 0x55816b9af150 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x55816b9b82b0 tx=0x7fc2500035f0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.551+0000 7fc271f80740 1 -- 172.21.15.12:0/739062932 shutdown_connections 2024-03-23T23:04:09.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.551+0000 7fc271f80740 1 --2- 172.21.15.12:0/739062932 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55816ba12170 0x55816ba14590 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.551+0000 7fc271f80740 1 --2- 172.21.15.12:0/739062932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55816b9a1580 0x55816b9b06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.551+0000 7fc271f80740 1 --2- 172.21.15.12:0/739062932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55816b9a0bc0 0x55816b9af690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.551+0000 7fc271f80740 1 --2- 172.21.15.12:0/739062932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55816b99f9f0 0x55816b9af150 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.551+0000 7fc256000640 0 client.25846 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:04:09.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.551+0000 7fc271f80740 1 -- 172.21.15.12:0/739062932 >> 172.21.15.12:0/739062932 conn(0x55816b99ade0 msgr2=0x55816b99e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:09.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.552+0000 7fc271f80740 1 -- 172.21.15.12:0/739062932 shutdown_connections 2024-03-23T23:04:09.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.552+0000 7fc271f80740 1 -- 172.21.15.12:0/739062932 wait complete. 2024-03-23T23:04:09.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.552+0000 7fc271f80740 20 client.25846 trim_cache size 0 max 16384 2024-03-23T23:04:09.556 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:04:09.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.552+0000 7fc26dc00640 1 -- 172.21.15.12:0/3008498625 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc24c0a2880 msgr2=0x7fc24c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:09.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.552+0000 7fc26dc00640 1 --2- 172.21.15.12:0/3008498625 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc24c0a2880 0x7fc24c0a4d40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc260023000 tx=0x7fc260024000 comp rx=0 tx=0).stop 2024-03-23T23:04:09.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.552+0000 7fc26dc00640 1 -- 172.21.15.12:0/3008498625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc268102790 msgr2=0x7fc268199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:09.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.552+0000 7fc26dc00640 1 --2- 172.21.15.12:0/3008498625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc268102790 0x7fc268199ff0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc25802c500 tx=0x7fc258033580 comp rx=0 tx=0).stop 2024-03-23T23:04:09.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.552+0000 7fc26dc00640 1 -- 172.21.15.12:0/3008498625 shutdown_connections 2024-03-23T23:04:09.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.552+0000 7fc26dc00640 1 --2- 172.21.15.12:0/3008498625 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc24c0a2880 0x7fc24c0a4d40 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.552+0000 7fc26dc00640 1 --2- 172.21.15.12:0/3008498625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc268103150 0x7fc2681a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.552+0000 7fc26dc00640 1 --2- 172.21.15.12:0/3008498625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc268102790 0x7fc268199ff0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.552+0000 7fc26dc00640 1 --2- 172.21.15.12:0/3008498625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2681015a0 0x7fc268199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.552+0000 7fc26dc00640 1 -- 172.21.15.12:0/3008498625 >> 172.21.15.12:0/3008498625 conn(0x7fc2680fcb10 msgr2=0x7fc2680738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:09.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.553+0000 7fc26dc00640 1 -- 172.21.15.12:0/3008498625 shutdown_connections 2024-03-23T23:04:09.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.553+0000 7fc26dc00640 1 -- 172.21.15.12:0/3008498625 wait complete. 2024-03-23T23:04:09.572 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:04:09.572 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-23T23:04:09.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.800+0000 7f6b51600640 1 -- 172.21.15.12:0/2884267170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b4c1037a0 msgr2=0x7f6b4c103bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:09.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.800+0000 7f6b51600640 1 --2- 172.21.15.12:0/2884267170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b4c1037a0 0x7f6b4c103bc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c009be0 tx=0x7f6b3c02bef0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.801+0000 7f6b51600640 1 -- 172.21.15.12:0/2884267170 shutdown_connections 2024-03-23T23:04:09.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.801+0000 7f6b51600640 1 --2- 172.21.15.12:0/2884267170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b4c105350 0x7f6b4c10bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.801+0000 7f6b51600640 1 --2- 172.21.15.12:0/2884267170 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c104990 0x7f6b4c104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.801+0000 7f6b51600640 1 --2- 172.21.15.12:0/2884267170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b4c1037a0 0x7f6b4c103bc0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.801+0000 7f6b51600640 1 -- 172.21.15.12:0/2884267170 >> 172.21.15.12:0/2884267170 conn(0x7f6b4c0fed50 msgr2=0x7f6b4c101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:09.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.801+0000 7f6b51600640 1 -- 172.21.15.12:0/2884267170 shutdown_connections 2024-03-23T23:04:09.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.801+0000 7f6b51600640 1 -- 172.21.15.12:0/2884267170 wait complete. 2024-03-23T23:04:09.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.802+0000 7f6b51600640 1 Processor -- start 2024-03-23T23:04:09.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.802+0000 7f6b51600640 1 -- start start 2024-03-23T23:04:09.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.803+0000 7f6b51600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b4c1037a0 0x7f6b4c199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.803+0000 7f6b51600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c104990 0x7f6b4c19a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.803+0000 7f6b51600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b4c105350 0x7f6b4c1a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.803+0000 7f6b51600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c10f440 con 0x7f6b4c104990 2024-03-23T23:04:09.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.803+0000 7f6b51600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c10f2c0 con 0x7f6b4c105350 2024-03-23T23:04:09.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.803+0000 7f6b51600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c10f5c0 con 0x7f6b4c1037a0 2024-03-23T23:04:09.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.803+0000 7f6b50c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b4c105350 0x7f6b4c1a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.803+0000 7f6b4be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b4c1037a0 0x7f6b4c199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.803+0000 7f6b50c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b4c105350 0x7f6b4c1a1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37702/0 (socket says 172.21.15.12:37702) 2024-03-23T23:04:09.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.803+0000 7f6b50c00640 1 -- 172.21.15.12:0/4284593711 learned_addr learned my addr 172.21.15.12:0/4284593711 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:09.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.803+0000 7f6b4b400640 1 --2- 172.21.15.12:0/4284593711 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c104990 0x7f6b4c19a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.803+0000 7f6b4be00640 1 -- 172.21.15.12:0/4284593711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b4c105350 msgr2=0x7f6b4c1a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:09.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.803+0000 7f6b4be00640 1 --2- 172.21.15.12:0/4284593711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b4c105350 0x7f6b4c1a1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.803+0000 7f6b4be00640 1 -- 172.21.15.12:0/4284593711 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c104990 msgr2=0x7f6b4c19a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:09.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.803+0000 7f6b4be00640 1 --2- 172.21.15.12:0/4284593711 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c104990 0x7f6b4c19a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.803+0000 7f6b4be00640 1 -- 172.21.15.12:0/4284593711 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b4c1a16e0 con 0x7f6b4c1037a0 2024-03-23T23:04:09.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.804+0000 7f6b4be00640 1 --2- 172.21.15.12:0/4284593711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b4c1037a0 0x7f6b4c199ac0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c02c430 tx=0x7f6b3c033580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:09.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.804+0000 7f6b48c00640 1 -- 172.21.15.12:0/4284593711 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c043020 con 0x7f6b4c1037a0 2024-03-23T23:04:09.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.804+0000 7f6b48c00640 1 -- 172.21.15.12:0/4284593711 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b3c003e50 con 0x7f6b4c1037a0 2024-03-23T23:04:09.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.804+0000 7f6b51600640 1 -- 172.21.15.12:0/4284593711 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b4c1a1970 con 0x7f6b4c1037a0 2024-03-23T23:04:09.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.804+0000 7f6b48c00640 1 -- 172.21.15.12:0/4284593711 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c034690 con 0x7f6b4c1037a0 2024-03-23T23:04:09.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.804+0000 7f6b51600640 1 -- 172.21.15.12:0/4284593711 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b4c1a1ee0 con 0x7f6b4c1037a0 2024-03-23T23:04:09.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.808+0000 7f6b55978740 1 Processor -- start 2024-03-23T23:04:09.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.808+0000 7f6b48c00640 1 -- 172.21.15.12:0/4284593711 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6b3c046020 con 0x7f6b4c1037a0 2024-03-23T23:04:09.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.808+0000 7f6b55978740 1 -- start start 2024-03-23T23:04:09.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.808+0000 7f6b55978740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f279f0 0x5635b9f27e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.808+0000 7f6b55978740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635b9f28bc0 0x5635b9f29040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.808+0000 7f6b55978740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635b9f29580 0x5635b9f2fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.808+0000 7f6b55978740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635b9e8c2c0 con 0x5635b9f29580 2024-03-23T23:04:09.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.808+0000 7f6b55978740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635b9e34c90 con 0x5635b9f279f0 2024-03-23T23:04:09.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.808+0000 7f6b55978740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635b9e88360 con 0x5635b9f28bc0 2024-03-23T23:04:09.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.809+0000 7f6b50c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f279f0 0x5635b9f27e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.809+0000 7f6b50c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f279f0 0x5635b9f27e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37708/0 (socket says 172.21.15.12:37708) 2024-03-23T23:04:09.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.809+0000 7f6b50c00640 1 -- 172.21.15.12:0/2283439230 learned_addr learned my addr 172.21.15.12:0/2283439230 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:09.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.809+0000 7f6b4be00640 1 --2- 172.21.15.12:0/2283439230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635b9f28bc0 0x5635b9f29040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.809+0000 7f6b4b400640 1 --2- 172.21.15.12:0/2283439230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635b9f29580 0x5635b9f2fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.809+0000 7f6b50c00640 1 -- 172.21.15.12:0/2283439230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635b9f28bc0 msgr2=0x5635b9f29040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:09.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.809+0000 7f6b50c00640 1 --2- 172.21.15.12:0/2283439230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635b9f28bc0 0x5635b9f29040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.809+0000 7f6b50c00640 1 -- 172.21.15.12:0/2283439230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635b9f29580 msgr2=0x5635b9f2fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:09.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.809+0000 7f6b50c00640 1 --2- 172.21.15.12:0/2283439230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635b9f29580 0x5635b9f2fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.809+0000 7f6b50c00640 1 -- 172.21.15.12:0/2283439230 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635b9e59080 con 0x5635b9f279f0 2024-03-23T23:04:09.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.809+0000 7f6b4be00640 1 --2- 172.21.15.12:0/2283439230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635b9f28bc0 0x5635b9f29040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:04:09.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.809+0000 7f6b50c00640 1 --2- 172.21.15.12:0/2283439230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f279f0 0x5635b9f27e10 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6b44009d90 tx=0x7f6b44011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db9f89bd9f9d7cb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:09.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.809+0000 7f6b41200640 1 -- 172.21.15.12:0/2283439230 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4401e070 con 0x5635b9f279f0 2024-03-23T23:04:09.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.810+0000 7f6b41200640 1 -- 172.21.15.12:0/2283439230 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b44019b10 con 0x5635b9f279f0 2024-03-23T23:04:09.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.810+0000 7f6b41200640 1 -- 172.21.15.12:0/2283439230 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4400fbd0 con 0x5635b9f279f0 2024-03-23T23:04:09.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.810+0000 7f6b48c00640 1 --2- 172.21.15.12:0/4284593711 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6b300a2880 0x7f6b300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.810+0000 7f6b48c00640 1 -- 172.21.15.12:0/4284593711 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f6b3c0f1060 con 0x7f6b4c1037a0 2024-03-23T23:04:09.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.810+0000 7f6b4be00640 1 --2- 172.21.15.12:0/4284593711 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6b300a2880 0x7f6b300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.810+0000 7f6b55978740 1 -- 172.21.15.12:0/2283439230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f279f0 msgr2=0x5635b9f27e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:09.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.810+0000 7f6b55978740 1 --2- 172.21.15.12:0/2283439230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f279f0 0x5635b9f27e10 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6b44009d90 tx=0x7f6b44011810 comp rx=0 tx=0).stop 2024-03-23T23:04:09.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.810+0000 7f6b55978740 1 -- 172.21.15.12:0/2283439230 shutdown_connections 2024-03-23T23:04:09.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.810+0000 7f6b55978740 1 --2- 172.21.15.12:0/2283439230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635b9f29580 0x5635b9f2fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.810+0000 7f6b55978740 1 --2- 172.21.15.12:0/2283439230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635b9f28bc0 0x5635b9f29040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.810+0000 7f6b55978740 1 --2- 172.21.15.12:0/2283439230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f279f0 0x5635b9f27e10 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.810+0000 7f6b55978740 1 -- 172.21.15.12:0/2283439230 >> 172.21.15.12:0/2283439230 conn(0x5635b9f22de0 msgr2=0x5635b9f231e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:09.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.810+0000 7f6b55978740 1 -- 172.21.15.12:0/2283439230 shutdown_connections 2024-03-23T23:04:09.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.811+0000 7f6b55978740 1 -- 172.21.15.12:0/2283439230 wait complete. 2024-03-23T23:04:09.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.811+0000 7f6b4be00640 1 --2- 172.21.15.12:0/4284593711 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6b300a2880 0x7f6b300a4d40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c0ba380 tx=0x7f6b3c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:09.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.811+0000 7f6b55978740 1 -- start start 2024-03-23T23:04:09.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.811+0000 7f6b55978740 1 Processor -- start 2024-03-23T23:04:09.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.812+0000 7f6b55978740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f279f0 0x5635b9f31cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.812+0000 7f6b55978740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635b9f28bc0 0x5635b9f32230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.812+0000 7f6b55978740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635b9f29580 0x5635b9f39200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.812+0000 7f6b55978740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635b9f30590 con 0x5635b9f28bc0 2024-03-23T23:04:09.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.812+0000 7f6b55978740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635b9e480d0 con 0x5635b9f279f0 2024-03-23T23:04:09.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.812+0000 7f6b55978740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635b9f30710 con 0x5635b9f29580 2024-03-23T23:04:09.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.812+0000 7f6b55978740 10 client.0 _ll_register_callbacks cb 0x5635b9e4ce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:09.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.812+0000 7f6b50c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f279f0 0x5635b9f31cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.812+0000 7f6b50c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f279f0 0x5635b9f31cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37722/0 (socket says 172.21.15.12:37722) 2024-03-23T23:04:09.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.812+0000 7f6b50c00640 1 -- 172.21.15.12:0/3405303356 learned_addr learned my addr 172.21.15.12:0/3405303356 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:09.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.812+0000 7f6b4b400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635b9f28bc0 0x5635b9f32230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.812+0000 7f6b50c00640 1 --2- 172.21.15.12:0/3405303356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635b9f29580 0x5635b9f39200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.812+0000 7f6b50c00640 1 -- 172.21.15.12:0/3405303356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635b9f29580 msgr2=0x5635b9f39200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:09.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.812+0000 7f6b50c00640 1 --2- 172.21.15.12:0/3405303356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635b9f29580 0x5635b9f39200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.812+0000 7f6b50c00640 1 -- 172.21.15.12:0/3405303356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635b9f28bc0 msgr2=0x5635b9f32230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:09.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.812+0000 7f6b50c00640 1 --2- 172.21.15.12:0/3405303356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635b9f28bc0 0x5635b9f32230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.812+0000 7f6b50c00640 1 -- 172.21.15.12:0/3405303356 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5635b9ba03f0 con 0x5635b9f279f0 2024-03-23T23:04:09.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.812+0000 7f6b50c00640 1 --2- 172.21.15.12:0/3405303356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f279f0 0x5635b9f31cf0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6b44011d10 tx=0x7f6b4400f3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60554d8cff593cd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:09.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.813+0000 7f6b39600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:04:09.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.813+0000 7f6b39600640 1 -- 172.21.15.12:0/3405303356 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4401e030 con 0x5635b9f279f0 2024-03-23T23:04:09.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.813+0000 7f6b39600640 1 -- 172.21.15.12:0/3405303356 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f6b4400fd60 con 0x5635b9f279f0 2024-03-23T23:04:09.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.813+0000 7f6b55978740 1 -- 172.21.15.12:0/3405303356 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6b44019b10 con 0x5635b9f279f0 2024-03-23T23:04:09.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.813+0000 7f6b39600640 1 -- 172.21.15.12:0/3405303356 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b44005050 con 0x5635b9f279f0 2024-03-23T23:04:09.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.813+0000 7f6b55978740 10 client.25880 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:09.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.813+0000 7f6b55978740 1 -- 172.21.15.12:0/3405303356 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5635b9c6a7d0 con 0x5635b9f279f0 2024-03-23T23:04:09.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.813+0000 7f6b39600640 1 -- 172.21.15.12:0/3405303356 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b44019b10 con 0x5635b9f279f0 2024-03-23T23:04:09.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.813+0000 7f6b39600640 1 -- 172.21.15.12:0/3405303356 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f6b44004b80 con 0x5635b9f279f0 2024-03-23T23:04:09.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.813+0000 7f6b55978740 10 client.25880 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:09.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.813+0000 7f6b55978740 10 client.25880 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:04:09.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.814+0000 7f6b55978740 20 client.25880 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:09.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.814+0000 7f6b55978740 1 --2- 172.21.15.12:0/3405303356 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5635b9f3d510 0x5635b9f3f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.814+0000 7f6b55978740 4 client.25880 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:04:09.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.814+0000 7f6b55978740 1 -- 172.21.15.12:0/3405303356 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5635b9e99800 con 0x5635b9f3d510 2024-03-23T23:04:09.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.814+0000 7f6b4b400640 1 --2- 172.21.15.12:0/3405303356 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5635b9f3d510 0x5635b9f3f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.814+0000 7f6b4b400640 1 --2- 172.21.15.12:0/3405303356 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5635b9f3d510 0x5635b9f3f900 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=92c1245c41723a51 server_cookie=1c5e863d9204e6ad in_seq=0 out_seq=0 2024-03-23T23:04:09.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.814+0000 7f6b39600640 10 client.25880 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:04:09.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.815+0000 7f6b39600640 1 -- 172.21.15.12:0/3405303356 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5635b9e99800 con 0x5635b9f3d510 2024-03-23T23:04:09.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.815+0000 7f6b39600640 10 client.25880 handle_command_reply: tid=0 2024-03-23T23:04:09.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.815+0000 7f6b55978740 1 client.25880 shutdown 2024-03-23T23:04:09.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.815+0000 7f6b55978740 1 -- 172.21.15.12:0/3405303356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f279f0 msgr2=0x5635b9f31cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:09.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.815+0000 7f6b55978740 1 --2- 172.21.15.12:0/3405303356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f279f0 0x5635b9f31cf0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6b44011d10 tx=0x7f6b4400f3c0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.815+0000 7f6b55978740 1 -- 172.21.15.12:0/3405303356 shutdown_connections 2024-03-23T23:04:09.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.815+0000 7f6b55978740 1 --2- 172.21.15.12:0/3405303356 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5635b9f3d510 0x5635b9f3f900 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.815+0000 7f6b55978740 1 --2- 172.21.15.12:0/3405303356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635b9f29580 0x5635b9f39200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.815+0000 7f6b55978740 1 --2- 172.21.15.12:0/3405303356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635b9f28bc0 0x5635b9f32230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.815+0000 7f6b55978740 1 --2- 172.21.15.12:0/3405303356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f279f0 0x5635b9f31cf0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.815+0000 7f6b55978740 1 -- 172.21.15.12:0/3405303356 >> 172.21.15.12:0/3405303356 conn(0x5635b9f22de0 msgr2=0x5635b9f22160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:09.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.815+0000 7f6b39600640 0 client.25880 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:04:09.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.816+0000 7f6b55978740 1 -- 172.21.15.12:0/3405303356 shutdown_connections 2024-03-23T23:04:09.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.816+0000 7f6b55978740 1 -- 172.21.15.12:0/3405303356 wait complete. 2024-03-23T23:04:09.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.816+0000 7f6b55978740 20 client.25880 trim_cache size 0 max 16384 2024-03-23T23:04:09.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.829+0000 7f6b55978740 1 Processor -- start 2024-03-23T23:04:09.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.830+0000 7f6b55978740 1 -- start start 2024-03-23T23:04:09.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.830+0000 7f6b55978740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f279f0 0x5635b9f26750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.830+0000 7f6b55978740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635b9f28bc0 0x5635b9f39190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.830+0000 7f6b55978740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635b9f29580 0x5635b9f2bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.830+0000 7f6b55978740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c1acf30 con 0x5635b9f29580 2024-03-23T23:04:09.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.830+0000 7f6b55978740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c1aaa30 con 0x5635b9f279f0 2024-03-23T23:04:09.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.830+0000 7f6b55978740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c1a8ad0 con 0x5635b9f28bc0 2024-03-23T23:04:09.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.830+0000 7f6b50c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f279f0 0x5635b9f26750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.830+0000 7f6b50c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f279f0 0x5635b9f26750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37728/0 (socket says 172.21.15.12:37728) 2024-03-23T23:04:09.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.830+0000 7f6b50c00640 1 -- 172.21.15.12:0/2990419908 learned_addr learned my addr 172.21.15.12:0/2990419908 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:09.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.830+0000 7f6b50c00640 1 --2- 172.21.15.12:0/2990419908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635b9f29580 0x5635b9f2bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.830+0000 7f6b4b400640 1 --2- 172.21.15.12:0/2990419908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635b9f28bc0 0x5635b9f39190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.831+0000 7f6b50c00640 1 -- 172.21.15.12:0/2990419908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635b9f28bc0 msgr2=0x5635b9f39190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:09.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.831+0000 7f6b50c00640 1 --2- 172.21.15.12:0/2990419908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635b9f28bc0 0x5635b9f39190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.831+0000 7f6b50c00640 1 -- 172.21.15.12:0/2990419908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635b9f29580 msgr2=0x5635b9f2bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:09.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.831+0000 7f6b50c00640 1 --2- 172.21.15.12:0/2990419908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635b9f29580 0x5635b9f2bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.831+0000 7f6b50c00640 1 -- 172.21.15.12:0/2990419908 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635b9f1e8b0 con 0x5635b9f279f0 2024-03-23T23:04:09.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.831+0000 7f6b50c00640 1 --2- 172.21.15.12:0/2990419908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f279f0 0x5635b9f26750 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6b440197c0 tx=0x7f6b44011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f259cb78012fe35 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:09.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.831+0000 7f6b3be00640 1 -- 172.21.15.12:0/2990419908 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4401e030 con 0x5635b9f279f0 2024-03-23T23:04:09.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.831+0000 7f6b3be00640 1 -- 172.21.15.12:0/2990419908 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b4402d070 con 0x5635b9f279f0 2024-03-23T23:04:09.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.831+0000 7f6b3be00640 1 -- 172.21.15.12:0/2990419908 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b44005450 con 0x5635b9f279f0 2024-03-23T23:04:09.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.832+0000 7f6b55978740 1 -- 172.21.15.12:0/2990419908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f279f0 msgr2=0x5635b9f26750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:09.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.832+0000 7f6b55978740 1 --2- 172.21.15.12:0/2990419908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f279f0 0x5635b9f26750 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6b440197c0 tx=0x7f6b44011810 comp rx=0 tx=0).stop 2024-03-23T23:04:09.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.832+0000 7f6b55978740 1 -- 172.21.15.12:0/2990419908 shutdown_connections 2024-03-23T23:04:09.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.832+0000 7f6b55978740 1 --2- 172.21.15.12:0/2990419908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635b9f29580 0x5635b9f2bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.832+0000 7f6b55978740 1 --2- 172.21.15.12:0/2990419908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635b9f28bc0 0x5635b9f39190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.832+0000 7f6b55978740 1 --2- 172.21.15.12:0/2990419908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f279f0 0x5635b9f26750 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.832+0000 7f6b55978740 1 -- 172.21.15.12:0/2990419908 >> 172.21.15.12:0/2990419908 conn(0x5635b9f22de0 msgr2=0x5635b9f26040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:09.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.832+0000 7f6b55978740 1 -- 172.21.15.12:0/2990419908 shutdown_connections 2024-03-23T23:04:09.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.832+0000 7f6b55978740 1 -- 172.21.15.12:0/2990419908 wait complete. 2024-03-23T23:04:09.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.832+0000 7f6b55978740 1 -- start start 2024-03-23T23:04:09.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.833+0000 7f6b55978740 1 Processor -- start 2024-03-23T23:04:09.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.833+0000 7f6b55978740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635b9f279f0 0x5635b9f34ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.833+0000 7f6b55978740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f28bc0 0x5635b9f35020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.833+0000 7f6b55978740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635b9f29580 0x5635b9f38040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.833+0000 7f6b55978740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c1a90d0 con 0x5635b9f279f0 2024-03-23T23:04:09.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.833+0000 7f6b55978740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c1aa3d0 con 0x5635b9f28bc0 2024-03-23T23:04:09.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.833+0000 7f6b55978740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c1a9710 con 0x5635b9f29580 2024-03-23T23:04:09.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.833+0000 7f6b55978740 10 client.0 _ll_register_callbacks cb 0x7f6b4401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:09.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.833+0000 7f6b4b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f28bc0 0x5635b9f35020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.833+0000 7f6b50c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635b9f279f0 0x5635b9f34ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.833+0000 7f6b4b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f28bc0 0x5635b9f35020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37740/0 (socket says 172.21.15.12:37740) 2024-03-23T23:04:09.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.833+0000 7f6b4b400640 1 -- 172.21.15.12:0/3278624278 learned_addr learned my addr 172.21.15.12:0/3278624278 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:09.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.833+0000 7f6b50c00640 1 --2- 172.21.15.12:0/3278624278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635b9f29580 0x5635b9f38040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.834+0000 7f6b4b400640 1 -- 172.21.15.12:0/3278624278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635b9f29580 msgr2=0x5635b9f38040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:09.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.834+0000 7f6b4b400640 1 --2- 172.21.15.12:0/3278624278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635b9f29580 0x5635b9f38040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.834+0000 7f6b4b400640 1 -- 172.21.15.12:0/3278624278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635b9f279f0 msgr2=0x5635b9f34ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:09.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.834+0000 7f6b4b400640 1 --2- 172.21.15.12:0/3278624278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635b9f279f0 0x5635b9f34ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.834+0000 7f6b4b400640 1 -- 172.21.15.12:0/3278624278 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5635b9f38630 con 0x5635b9f28bc0 2024-03-23T23:04:09.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.834+0000 7f6b50c00640 1 --2- 172.21.15.12:0/3278624278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635b9f279f0 0x5635b9f34ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:09.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.834+0000 7f6b4b400640 1 --2- 172.21.15.12:0/3278624278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f28bc0 0x5635b9f35020 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x5635b9f23d50 tx=0x7f6b34009f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=490aba699a7f3084 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:09.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.834+0000 7f6b39600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:04:09.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.834+0000 7f6b39600640 1 -- 172.21.15.12:0/3278624278 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b34005220 con 0x5635b9f28bc0 2024-03-23T23:04:09.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.834+0000 7f6b39600640 1 -- 172.21.15.12:0/3278624278 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f6b340053c0 con 0x5635b9f28bc0 2024-03-23T23:04:09.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.834+0000 7f6b55978740 1 -- 172.21.15.12:0/3278624278 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6b4402d070 con 0x5635b9f28bc0 2024-03-23T23:04:09.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.835+0000 7f6b39600640 1 -- 172.21.15.12:0/3278624278 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b34030050 con 0x5635b9f28bc0 2024-03-23T23:04:09.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.835+0000 7f6b55978740 10 client.25886 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:09.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.835+0000 7f6b55978740 1 -- 172.21.15.12:0/3278624278 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5635b9f98fa0 con 0x5635b9f28bc0 2024-03-23T23:04:09.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.835+0000 7f6b39600640 1 -- 172.21.15.12:0/3278624278 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b4402d070 con 0x5635b9f28bc0 2024-03-23T23:04:09.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.835+0000 7f6b39600640 1 -- 172.21.15.12:0/3278624278 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f6b34003680 con 0x5635b9f28bc0 2024-03-23T23:04:09.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.835+0000 7f6b55978740 10 client.25886 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:09.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.835+0000 7f6b55978740 10 client.25886 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:04:09.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.835+0000 7f6b55978740 20 client.25886 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:09.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.835+0000 7f6b55978740 1 --2- 172.21.15.12:0/3278624278 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5635b9f998f0 0x5635b9f9bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:09.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.835+0000 7f6b55978740 4 client.25886 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:04:09.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.835+0000 7f6b55978740 1 -- 172.21.15.12:0/3278624278 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5635b9e93b50 con 0x5635b9f998f0 2024-03-23T23:04:09.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.836+0000 7f6b50c00640 1 --2- 172.21.15.12:0/3278624278 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5635b9f998f0 0x5635b9f9bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:09.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.836+0000 7f6b50c00640 1 --2- 172.21.15.12:0/3278624278 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5635b9f998f0 0x5635b9f9bd30 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d6225673782d8406 server_cookie=a6ee31c42454672 in_seq=0 out_seq=0 2024-03-23T23:04:09.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.836+0000 7f6b39600640 10 client.25886 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:04:09.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.836+0000 7f6b39600640 1 -- 172.21.15.12:0/3278624278 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5635b9e93b50 con 0x5635b9f998f0 2024-03-23T23:04:09.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.836+0000 7f6b39600640 10 client.25886 handle_command_reply: tid=0 2024-03-23T23:04:09.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.836+0000 7f6b55978740 1 client.25886 shutdown 2024-03-23T23:04:09.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.837+0000 7f6b55978740 1 -- 172.21.15.12:0/3278624278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f28bc0 msgr2=0x5635b9f35020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:09.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.837+0000 7f6b55978740 1 --2- 172.21.15.12:0/3278624278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f28bc0 0x5635b9f35020 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x5635b9f23d50 tx=0x7f6b34009f50 comp rx=0 tx=0).stop 2024-03-23T23:04:09.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.837+0000 7f6b55978740 1 -- 172.21.15.12:0/3278624278 shutdown_connections 2024-03-23T23:04:09.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.837+0000 7f6b55978740 1 --2- 172.21.15.12:0/3278624278 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5635b9f998f0 0x5635b9f9bd30 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.837+0000 7f6b55978740 1 --2- 172.21.15.12:0/3278624278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635b9f29580 0x5635b9f38040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.837+0000 7f6b55978740 1 --2- 172.21.15.12:0/3278624278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635b9f28bc0 0x5635b9f35020 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.837+0000 7f6b55978740 1 --2- 172.21.15.12:0/3278624278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635b9f279f0 0x5635b9f34ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.837+0000 7f6b39600640 0 client.25886 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:04:09.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.837+0000 7f6b55978740 1 -- 172.21.15.12:0/3278624278 >> 172.21.15.12:0/3278624278 conn(0x5635b9f22de0 msgr2=0x5635b9f268d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:09.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.837+0000 7f6b55978740 1 -- 172.21.15.12:0/3278624278 shutdown_connections 2024-03-23T23:04:09.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.837+0000 7f6b55978740 1 -- 172.21.15.12:0/3278624278 wait complete. 2024-03-23T23:04:09.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.837+0000 7f6b55978740 20 client.25886 trim_cache size 0 max 16384 2024-03-23T23:04:09.840 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:04:09.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.838+0000 7f6b51600640 1 -- 172.21.15.12:0/4284593711 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6b300a2880 msgr2=0x7f6b300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:09.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.838+0000 7f6b51600640 1 --2- 172.21.15.12:0/4284593711 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6b300a2880 0x7f6b300a4d40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c0ba380 tx=0x7f6b3c009500 comp rx=0 tx=0).stop 2024-03-23T23:04:09.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.838+0000 7f6b51600640 1 -- 172.21.15.12:0/4284593711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b4c1037a0 msgr2=0x7f6b4c199ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:09.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.838+0000 7f6b51600640 1 --2- 172.21.15.12:0/4284593711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b4c1037a0 0x7f6b4c199ac0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c02c430 tx=0x7f6b3c033580 comp rx=0 tx=0).stop 2024-03-23T23:04:09.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.838+0000 7f6b51600640 1 -- 172.21.15.12:0/4284593711 shutdown_connections 2024-03-23T23:04:09.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.838+0000 7f6b51600640 1 --2- 172.21.15.12:0/4284593711 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6b300a2880 0x7f6b300a4d40 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.838+0000 7f6b51600640 1 --2- 172.21.15.12:0/4284593711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b4c105350 0x7f6b4c1a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.838+0000 7f6b51600640 1 --2- 172.21.15.12:0/4284593711 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c104990 0x7f6b4c19a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.838+0000 7f6b51600640 1 --2- 172.21.15.12:0/4284593711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b4c1037a0 0x7f6b4c199ac0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:09.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.838+0000 7f6b51600640 1 -- 172.21.15.12:0/4284593711 >> 172.21.15.12:0/4284593711 conn(0x7f6b4c0fed50 msgr2=0x7f6b4c107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:09.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.838+0000 7f6b51600640 1 -- 172.21.15.12:0/4284593711 shutdown_connections 2024-03-23T23:04:09.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:09.838+0000 7f6b51600640 1 -- 172.21.15.12:0/4284593711 wait complete. 2024-03-23T23:04:09.852 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:09 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3284722715' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:04:09.854 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:04:09.854 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-23T23:04:09.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:09 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3284722715' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:04:10.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.084+0000 7f78b2000640 1 -- 172.21.15.12:0/336600567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78ac109770 msgr2=0x7f78ac110000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:10.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.084+0000 7f78b2000640 1 --2- 172.21.15.12:0/336600567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78ac109770 0x7f78ac110000 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f78a4009be0 tx=0x7f78a402bef0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.084+0000 7f78b2000640 1 -- 172.21.15.12:0/336600567 shutdown_connections 2024-03-23T23:04:10.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.084+0000 7f78b2000640 1 --2- 172.21.15.12:0/336600567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78ac109770 0x7f78ac110000 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.084+0000 7f78b2000640 1 --2- 172.21.15.12:0/336600567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78ac108db0 0x7f78ac109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.084+0000 7f78b2000640 1 --2- 172.21.15.12:0/336600567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78ac107bc0 0x7f78ac107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.084+0000 7f78b2000640 1 -- 172.21.15.12:0/336600567 >> 172.21.15.12:0/336600567 conn(0x7f78ac076b70 msgr2=0x7f78ac078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:10.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.084+0000 7f78b2000640 1 -- 172.21.15.12:0/336600567 shutdown_connections 2024-03-23T23:04:10.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.084+0000 7f78b2000640 1 -- 172.21.15.12:0/336600567 wait complete. 2024-03-23T23:04:10.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.085+0000 7f78b2000640 1 Processor -- start 2024-03-23T23:04:10.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.085+0000 7f78b2000640 1 -- start start 2024-03-23T23:04:10.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.085+0000 7f78b2000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78ac107bc0 0x7f78ac19ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.085+0000 7f78b2000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78ac108db0 0x7f78ac19e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.085+0000 7f78b2000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78ac109770 0x7f78ac1a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.085+0000 7f78b2000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac113870 con 0x7f78ac107bc0 2024-03-23T23:04:10.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.085+0000 7f78b2000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac1136f0 con 0x7f78ac109770 2024-03-23T23:04:10.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.085+0000 7f78b2000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac1139f0 con 0x7f78ac108db0 2024-03-23T23:04:10.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.086+0000 7f78b0c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78ac107bc0 0x7f78ac19ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.086+0000 7f78abe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78ac108db0 0x7f78ac19e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.086+0000 7f78b1600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78ac109770 0x7f78ac1a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.086+0000 7f78abe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78ac108db0 0x7f78ac19e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:51778/0 (socket says 172.21.15.12:51778) 2024-03-23T23:04:10.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.086+0000 7f78abe00640 1 -- 172.21.15.12:0/2316314464 learned_addr learned my addr 172.21.15.12:0/2316314464 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:10.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.086+0000 7f78b0c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78ac107bc0 0x7f78ac19ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53834/0 (socket says 172.21.15.12:53834) 2024-03-23T23:04:10.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.086+0000 7f78abe00640 1 -- 172.21.15.12:0/2316314464 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78ac109770 msgr2=0x7f78ac1a53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:10.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.086+0000 7f78abe00640 1 --2- 172.21.15.12:0/2316314464 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78ac109770 0x7f78ac1a53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.086+0000 7f78abe00640 1 -- 172.21.15.12:0/2316314464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78ac107bc0 msgr2=0x7f78ac19ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:10.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.086+0000 7f78abe00640 1 --2- 172.21.15.12:0/2316314464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78ac107bc0 0x7f78ac19ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.086+0000 7f78abe00640 1 -- 172.21.15.12:0/2316314464 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78ac1a5a10 con 0x7f78ac108db0 2024-03-23T23:04:10.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.086+0000 7f78abe00640 1 --2- 172.21.15.12:0/2316314464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78ac108db0 0x7f78ac19e330 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f789800e9c0 tx=0x7f789800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:10.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.087+0000 7f78a9600640 1 -- 172.21.15.12:0/2316314464 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789800cde0 con 0x7f78ac108db0 2024-03-23T23:04:10.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.087+0000 7f78a9600640 1 -- 172.21.15.12:0/2316314464 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78980035f0 con 0x7f78ac108db0 2024-03-23T23:04:10.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.087+0000 7f78a9600640 1 -- 172.21.15.12:0/2316314464 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7898010690 con 0x7f78ac108db0 2024-03-23T23:04:10.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.087+0000 7f78b2000640 1 -- 172.21.15.12:0/2316314464 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78ac1a5d00 con 0x7f78ac108db0 2024-03-23T23:04:10.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.087+0000 7f78b2000640 1 -- 172.21.15.12:0/2316314464 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78ac1a61a0 con 0x7f78ac108db0 2024-03-23T23:04:10.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.090+0000 7f78a9600640 1 -- 172.21.15.12:0/2316314464 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f78980037a0 con 0x7f78ac108db0 2024-03-23T23:04:10.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.090+0000 7f78b6379740 1 Processor -- start 2024-03-23T23:04:10.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.090+0000 7f78b6379740 1 -- start start 2024-03-23T23:04:10.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.090+0000 7f78b6379740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639a5ffe9f0 0x5639a5ffee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.090+0000 7f78b6379740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5fffbc0 0x5639a6000040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.090+0000 7f78b6379740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639a6000580 0x5639a6006e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.090+0000 7f78b6379740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639a5f632c0 con 0x5639a5ffe9f0 2024-03-23T23:04:10.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.090+0000 7f78b6379740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639a5f0bc90 con 0x5639a5fffbc0 2024-03-23T23:04:10.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.090+0000 7f78b6379740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639a5f5f360 con 0x5639a6000580 2024-03-23T23:04:10.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.091+0000 7f78b0c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5fffbc0 0x5639a6000040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.091+0000 7f78b0c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5fffbc0 0x5639a6000040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37764/0 (socket says 172.21.15.12:37764) 2024-03-23T23:04:10.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.091+0000 7f78b1600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639a5ffe9f0 0x5639a5ffee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.091+0000 7f78b0c00640 1 -- 172.21.15.12:0/3373338007 learned_addr learned my addr 172.21.15.12:0/3373338007 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:10.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.091+0000 7f78abe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639a6000580 0x5639a6006e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.091+0000 7f78b0c00640 1 -- 172.21.15.12:0/3373338007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639a6000580 msgr2=0x5639a6006e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:10.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.091+0000 7f78b0c00640 1 --2- 172.21.15.12:0/3373338007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639a6000580 0x5639a6006e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.091+0000 7f78b0c00640 1 -- 172.21.15.12:0/3373338007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639a5ffe9f0 msgr2=0x5639a5ffee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:10.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.091+0000 7f78b0c00640 1 --2- 172.21.15.12:0/3373338007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639a5ffe9f0 0x5639a5ffee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.091+0000 7f78b0c00640 1 -- 172.21.15.12:0/3373338007 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5639a5f30080 con 0x5639a5fffbc0 2024-03-23T23:04:10.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.091+0000 7f78b0c00640 1 --2- 172.21.15.12:0/3373338007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5fffbc0 0x5639a6000040 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f78a000b7d0 tx=0x7f78a0004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88419c3c75ec764c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:10.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.092+0000 7f789dc00640 1 -- 172.21.15.12:0/3373338007 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a0003dd0 con 0x5639a5fffbc0 2024-03-23T23:04:10.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.092+0000 7f789dc00640 1 -- 172.21.15.12:0/3373338007 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78a0015070 con 0x5639a5fffbc0 2024-03-23T23:04:10.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.092+0000 7f78a9600640 1 --2- 172.21.15.12:0/2316314464 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f78940a2880 0x7f78940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.092+0000 7f789dc00640 1 -- 172.21.15.12:0/3373338007 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a00117c0 con 0x5639a5fffbc0 2024-03-23T23:04:10.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.092+0000 7f78a9600640 1 -- 172.21.15.12:0/2316314464 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f78980cecf0 con 0x7f78ac108db0 2024-03-23T23:04:10.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.092+0000 7f78b1600640 1 --2- 172.21.15.12:0/2316314464 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f78940a2880 0x7f78940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.092+0000 7f78b6379740 1 -- 172.21.15.12:0/3373338007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5fffbc0 msgr2=0x5639a6000040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:10.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.092+0000 7f78b6379740 1 --2- 172.21.15.12:0/3373338007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5fffbc0 0x5639a6000040 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f78a000b7d0 tx=0x7f78a0004610 comp rx=0 tx=0).stop 2024-03-23T23:04:10.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.093+0000 7f78b6379740 1 -- 172.21.15.12:0/3373338007 shutdown_connections 2024-03-23T23:04:10.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.093+0000 7f78b6379740 1 --2- 172.21.15.12:0/3373338007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639a6000580 0x5639a6006e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.093+0000 7f78b6379740 1 --2- 172.21.15.12:0/3373338007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5fffbc0 0x5639a6000040 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.093+0000 7f78b6379740 1 --2- 172.21.15.12:0/3373338007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639a5ffe9f0 0x5639a5ffee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.093+0000 7f78b6379740 1 -- 172.21.15.12:0/3373338007 >> 172.21.15.12:0/3373338007 conn(0x5639a5ff9de0 msgr2=0x5639a5ffa1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:10.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.093+0000 7f78b6379740 1 -- 172.21.15.12:0/3373338007 shutdown_connections 2024-03-23T23:04:10.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.093+0000 7f78b1600640 1 --2- 172.21.15.12:0/2316314464 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f78940a2880 0x7f78940a4d40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f78a4006010 tx=0x7f78a4004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:10.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.093+0000 7f78b6379740 1 -- 172.21.15.12:0/3373338007 wait complete. 2024-03-23T23:04:10.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.093+0000 7f78b6379740 1 -- start start 2024-03-23T23:04:10.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.093+0000 7f78b6379740 1 Processor -- start 2024-03-23T23:04:10.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.094+0000 7f78b6379740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5ffe9f0 0x5639a6008c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.094+0000 7f78b6379740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639a5fffbc0 0x5639a6009170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.094+0000 7f78b6379740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639a6000580 0x5639a60101b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.094+0000 7f78b6379740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639a6007590 con 0x5639a6000580 2024-03-23T23:04:10.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.094+0000 7f78b6379740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639a5f1f0d0 con 0x5639a5ffe9f0 2024-03-23T23:04:10.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.094+0000 7f78b6379740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639a6007710 con 0x5639a5fffbc0 2024-03-23T23:04:10.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.094+0000 7f78b6379740 10 client.0 _ll_register_callbacks cb 0x5639a5f23e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:10.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.094+0000 7f78abe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639a5fffbc0 0x5639a6009170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.094+0000 7f78b0c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5ffe9f0 0x5639a6008c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.094+0000 7f78abe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639a5fffbc0 0x5639a6009170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:51798/0 (socket says 172.21.15.12:51798) 2024-03-23T23:04:10.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.094+0000 7f78abe00640 1 -- 172.21.15.12:0/206014626 learned_addr learned my addr 172.21.15.12:0/206014626 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:10.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.094+0000 7f78b1600640 1 --2- 172.21.15.12:0/206014626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639a6000580 0x5639a60101b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.095+0000 7f78abe00640 1 -- 172.21.15.12:0/206014626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5ffe9f0 msgr2=0x5639a6008c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:10.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.095+0000 7f78abe00640 1 --2- 172.21.15.12:0/206014626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5ffe9f0 0x5639a6008c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.095+0000 7f78abe00640 1 -- 172.21.15.12:0/206014626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639a6000580 msgr2=0x5639a60101b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:10.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.095+0000 7f78abe00640 1 --2- 172.21.15.12:0/206014626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639a6000580 0x5639a60101b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.095+0000 7f78abe00640 1 -- 172.21.15.12:0/206014626 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5639a5c773f0 con 0x5639a5fffbc0 2024-03-23T23:04:10.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.095+0000 7f78abe00640 1 --2- 172.21.15.12:0/206014626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639a5fffbc0 0x5639a6009170 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7898098010 tx=0x7f78980972a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=536c55c38d35312e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:10.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.095+0000 7f7892000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:04:10.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.095+0000 7f7892000640 1 -- 172.21.15.12:0/206014626 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789809b410 con 0x5639a5fffbc0 2024-03-23T23:04:10.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.095+0000 7f7892000640 1 -- 172.21.15.12:0/206014626 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f78980a3950 con 0x5639a5fffbc0 2024-03-23T23:04:10.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.095+0000 7f78b6379740 1 -- 172.21.15.12:0/206014626 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f78a0015070 con 0x5639a5fffbc0 2024-03-23T23:04:10.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.095+0000 7f7892000640 1 -- 172.21.15.12:0/206014626 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f789809afa0 con 0x5639a5fffbc0 2024-03-23T23:04:10.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.095+0000 7f78b6379740 10 client.25876 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:10.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.095+0000 7f78b6379740 1 -- 172.21.15.12:0/206014626 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5639a5d417d0 con 0x5639a5fffbc0 2024-03-23T23:04:10.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.095+0000 7f7892000640 1 -- 172.21.15.12:0/206014626 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78a0015070 con 0x5639a5fffbc0 2024-03-23T23:04:10.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.096+0000 7f7892000640 1 -- 172.21.15.12:0/206014626 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f78980d6040 con 0x5639a5fffbc0 2024-03-23T23:04:10.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.096+0000 7f78b6379740 10 client.25876 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:10.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.096+0000 7f78b6379740 10 client.25876 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:04:10.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.096+0000 7f78b6379740 20 client.25876 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:10.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.096+0000 7f78b6379740 1 --2- 172.21.15.12:0/206014626 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5639a6014750 0x5639a6016b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.096+0000 7f78b6379740 4 client.25876 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:04:10.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.096+0000 7f78b6379740 1 -- 172.21.15.12:0/206014626 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5639a5f70800 con 0x5639a6014750 2024-03-23T23:04:10.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.096+0000 7f78b0c00640 1 --2- 172.21.15.12:0/206014626 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5639a6014750 0x5639a6016b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.097+0000 7f78b0c00640 1 --2- 172.21.15.12:0/206014626 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5639a6014750 0x5639a6016b90 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19a6382578c7c74d server_cookie=dad6eef23e6a9a90 in_seq=0 out_seq=0 2024-03-23T23:04:10.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.097+0000 7f7892000640 10 client.25876 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:10.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.097+0000 7f7892000640 1 -- 172.21.15.12:0/206014626 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5639a5f70800 con 0x5639a6014750 2024-03-23T23:04:10.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.097+0000 7f7892000640 10 client.25876 handle_command_reply: tid=0 2024-03-23T23:04:10.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.097+0000 7f78b6379740 1 client.25876 shutdown 2024-03-23T23:04:10.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.098+0000 7f78b6379740 1 -- 172.21.15.12:0/206014626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639a5fffbc0 msgr2=0x5639a6009170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:10.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.098+0000 7f78b6379740 1 --2- 172.21.15.12:0/206014626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639a5fffbc0 0x5639a6009170 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7898098010 tx=0x7f78980972a0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.098+0000 7f78b6379740 1 -- 172.21.15.12:0/206014626 shutdown_connections 2024-03-23T23:04:10.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.098+0000 7f78b6379740 1 --2- 172.21.15.12:0/206014626 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5639a6014750 0x5639a6016b90 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.098+0000 7f78b6379740 1 --2- 172.21.15.12:0/206014626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639a6000580 0x5639a60101b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.098+0000 7f78b6379740 1 --2- 172.21.15.12:0/206014626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639a5fffbc0 0x5639a6009170 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.098+0000 7f78b6379740 1 --2- 172.21.15.12:0/206014626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5ffe9f0 0x5639a6008c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.098+0000 7f7892000640 0 client.25876 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:10.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.098+0000 7f78b6379740 1 -- 172.21.15.12:0/206014626 >> 172.21.15.12:0/206014626 conn(0x5639a5ff9de0 msgr2=0x5639a5ff9160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:10.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.098+0000 7f78b6379740 1 -- 172.21.15.12:0/206014626 shutdown_connections 2024-03-23T23:04:10.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.098+0000 7f78b6379740 1 -- 172.21.15.12:0/206014626 wait complete. 2024-03-23T23:04:10.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.098+0000 7f78b6379740 20 client.25876 trim_cache size 0 max 16384 2024-03-23T23:04:10.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.112+0000 7f78b6379740 1 Processor -- start 2024-03-23T23:04:10.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.112+0000 7f78b6379740 1 -- start start 2024-03-23T23:04:10.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.112+0000 7f78b6379740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639a5ffe9f0 0x5639a6002c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.112+0000 7f78b6379740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5fffbc0 0x5639a6010170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.112+0000 7f78b6379740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639a6000580 0x5639a60031a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.112+0000 7f78b6379740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac1b1310 con 0x5639a5ffe9f0 2024-03-23T23:04:10.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.112+0000 7f78b6379740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac1aee10 con 0x5639a5fffbc0 2024-03-23T23:04:10.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.112+0000 7f78b6379740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac1aceb0 con 0x5639a6000580 2024-03-23T23:04:10.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.112+0000 7f78abe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5fffbc0 0x5639a6010170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.112+0000 7f78b1600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639a6000580 0x5639a60031a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.112+0000 7f78b0c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639a5ffe9f0 0x5639a6002c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.112+0000 7f78abe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5fffbc0 0x5639a6010170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37790/0 (socket says 172.21.15.12:37790) 2024-03-23T23:04:10.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.112+0000 7f78b1600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639a6000580 0x5639a60031a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:51804/0 (socket says 172.21.15.12:51804) 2024-03-23T23:04:10.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.112+0000 7f78abe00640 1 -- 172.21.15.12:0/2217813913 learned_addr learned my addr 172.21.15.12:0/2217813913 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:10.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.113+0000 7f78abe00640 1 -- 172.21.15.12:0/2217813913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639a6000580 msgr2=0x5639a60031a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:10.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.113+0000 7f78abe00640 1 --2- 172.21.15.12:0/2217813913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639a6000580 0x5639a60031a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.113+0000 7f78abe00640 1 -- 172.21.15.12:0/2217813913 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639a5ffe9f0 msgr2=0x5639a6002c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:10.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.113+0000 7f78abe00640 1 --2- 172.21.15.12:0/2217813913 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639a5ffe9f0 0x5639a6002c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.113+0000 7f78abe00640 1 -- 172.21.15.12:0/2217813913 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5639a5fff640 con 0x5639a5fffbc0 2024-03-23T23:04:10.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.113+0000 7f78b1600640 1 --2- 172.21.15.12:0/2217813913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639a6000580 0x5639a60031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:10.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.113+0000 7f78abe00640 1 --2- 172.21.15.12:0/2217813913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5fffbc0 0x5639a6010170 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7898000c00 tx=0x7f78980cef70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c925bb5273ecf1d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:10.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.113+0000 7f789d200640 1 -- 172.21.15.12:0/2217813913 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789809b180 con 0x5639a5fffbc0 2024-03-23T23:04:10.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.113+0000 7f789d200640 1 -- 172.21.15.12:0/2217813913 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f789809bd40 con 0x5639a5fffbc0 2024-03-23T23:04:10.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.113+0000 7f789d200640 1 -- 172.21.15.12:0/2217813913 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789809aa90 con 0x5639a5fffbc0 2024-03-23T23:04:10.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.114+0000 7f78b6379740 1 -- 172.21.15.12:0/2217813913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5fffbc0 msgr2=0x5639a6010170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:10.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.114+0000 7f78b6379740 1 --2- 172.21.15.12:0/2217813913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5fffbc0 0x5639a6010170 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7898000c00 tx=0x7f78980cef70 comp rx=0 tx=0).stop 2024-03-23T23:04:10.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.114+0000 7f78b6379740 1 -- 172.21.15.12:0/2217813913 shutdown_connections 2024-03-23T23:04:10.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.114+0000 7f78b6379740 1 --2- 172.21.15.12:0/2217813913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639a6000580 0x5639a60031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.114+0000 7f78b6379740 1 --2- 172.21.15.12:0/2217813913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5fffbc0 0x5639a6010170 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.114+0000 7f78b6379740 1 --2- 172.21.15.12:0/2217813913 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639a5ffe9f0 0x5639a6002c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.114+0000 7f78b6379740 1 -- 172.21.15.12:0/2217813913 >> 172.21.15.12:0/2217813913 conn(0x5639a5ff9de0 msgr2=0x5639a5ffd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:10.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.114+0000 7f78b6379740 1 -- 172.21.15.12:0/2217813913 shutdown_connections 2024-03-23T23:04:10.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.114+0000 7f78b6379740 1 -- 172.21.15.12:0/2217813913 wait complete. 2024-03-23T23:04:10.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.115+0000 7f78b6379740 1 -- start start 2024-03-23T23:04:10.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.115+0000 7f78b6379740 1 Processor -- start 2024-03-23T23:04:10.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.115+0000 7f78b6379740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5ffe9f0 0x5639a600c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.115+0000 7f78b6379740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639a5fffbc0 0x5639a600c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.115+0000 7f78b6379740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639a6000580 0x5639a600f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.115+0000 7f78b6379740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac1ad4b0 con 0x5639a5fffbc0 2024-03-23T23:04:10.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.115+0000 7f78b6379740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac1ae7b0 con 0x5639a5ffe9f0 2024-03-23T23:04:10.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.115+0000 7f78b6379740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac1adaf0 con 0x5639a6000580 2024-03-23T23:04:10.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.115+0000 7f78b6379740 10 client.0 _ll_register_callbacks cb 0x7f7898003e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:10.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.115+0000 7f78b0c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5ffe9f0 0x5639a600c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.115+0000 7f78b0c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5ffe9f0 0x5639a600c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37806/0 (socket says 172.21.15.12:37806) 2024-03-23T23:04:10.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.115+0000 7f78b0c00640 1 -- 172.21.15.12:0/2220876974 learned_addr learned my addr 172.21.15.12:0/2220876974 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:10.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.116+0000 7f78b1600640 1 --2- 172.21.15.12:0/2220876974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639a6000580 0x5639a600f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.116+0000 7f78abe00640 1 --2- 172.21.15.12:0/2220876974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639a5fffbc0 0x5639a600c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.116+0000 7f78b0c00640 1 -- 172.21.15.12:0/2220876974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639a6000580 msgr2=0x5639a600f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:10.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.116+0000 7f78b0c00640 1 --2- 172.21.15.12:0/2220876974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639a6000580 0x5639a600f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.116+0000 7f78b0c00640 1 -- 172.21.15.12:0/2220876974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639a5fffbc0 msgr2=0x5639a600c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:10.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.116+0000 7f78b0c00640 1 --2- 172.21.15.12:0/2220876974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639a5fffbc0 0x5639a600c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.116+0000 7f78b0c00640 1 -- 172.21.15.12:0/2220876974 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5639a600fcb0 con 0x5639a5ffe9f0 2024-03-23T23:04:10.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.116+0000 7f78abe00640 1 --2- 172.21.15.12:0/2220876974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639a5fffbc0 0x5639a600c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:04:10.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.116+0000 7f78b0c00640 1 --2- 172.21.15.12:0/2220876974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5ffe9f0 0x5639a600c1d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f78a0005940 tx=0x7f78a0003c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f220f334da01090 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:10.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.116+0000 7f7892000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:04:10.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.116+0000 7f7892000640 1 -- 172.21.15.12:0/2220876974 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a0003f60 con 0x5639a5ffe9f0 2024-03-23T23:04:10.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.116+0000 7f7892000640 1 -- 172.21.15.12:0/2220876974 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f78a0004100 con 0x5639a5ffe9f0 2024-03-23T23:04:10.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.116+0000 7f78b6379740 1 -- 172.21.15.12:0/2220876974 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f789809bd40 con 0x5639a5ffe9f0 2024-03-23T23:04:10.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.117+0000 7f7892000640 1 -- 172.21.15.12:0/2220876974 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78a002c870 con 0x5639a5ffe9f0 2024-03-23T23:04:10.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.117+0000 7f78b6379740 10 client.25904 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:10.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.117+0000 7f78b6379740 1 -- 172.21.15.12:0/2220876974 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5639a6070990 con 0x5639a5ffe9f0 2024-03-23T23:04:10.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.117+0000 7f7892000640 1 -- 172.21.15.12:0/2220876974 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f789809bd40 con 0x5639a5ffe9f0 2024-03-23T23:04:10.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.117+0000 7f7892000640 1 -- 172.21.15.12:0/2220876974 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f78a0004680 con 0x5639a5ffe9f0 2024-03-23T23:04:10.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.117+0000 7f78b6379740 10 client.25904 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:10.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.117+0000 7f78b6379740 10 client.25904 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:04:10.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.117+0000 7f78b6379740 20 client.25904 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:10.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.117+0000 7f78b6379740 1 --2- 172.21.15.12:0/2220876974 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5639a6070f80 0x5639a60733a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.117+0000 7f78b6379740 4 client.25904 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:04:10.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.117+0000 7f78b6379740 1 -- 172.21.15.12:0/2220876974 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5639a5f6ab50 con 0x5639a6070f80 2024-03-23T23:04:10.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.117+0000 7f78abe00640 1 --2- 172.21.15.12:0/2220876974 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5639a6070f80 0x5639a60733a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.118+0000 7f78abe00640 1 --2- 172.21.15.12:0/2220876974 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5639a6070f80 0x5639a60733a0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b53d7d445ae30a66 server_cookie=710693d0898c27f3 in_seq=0 out_seq=0 2024-03-23T23:04:10.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.118+0000 7f7892000640 10 client.25904 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:10.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.118+0000 7f7892000640 1 -- 172.21.15.12:0/2220876974 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5639a5f6ab50 con 0x5639a6070f80 2024-03-23T23:04:10.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.118+0000 7f7892000640 10 client.25904 handle_command_reply: tid=0 2024-03-23T23:04:10.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.118+0000 7f78b6379740 1 client.25904 shutdown 2024-03-23T23:04:10.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.118+0000 7f78b6379740 1 -- 172.21.15.12:0/2220876974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5ffe9f0 msgr2=0x5639a600c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:10.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.118+0000 7f78b6379740 1 --2- 172.21.15.12:0/2220876974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5ffe9f0 0x5639a600c1d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f78a0005940 tx=0x7f78a0003c70 comp rx=0 tx=0).stop 2024-03-23T23:04:10.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.118+0000 7f78b6379740 1 -- 172.21.15.12:0/2220876974 shutdown_connections 2024-03-23T23:04:10.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.118+0000 7f78b6379740 1 --2- 172.21.15.12:0/2220876974 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5639a6070f80 0x5639a60733a0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.118+0000 7f78b6379740 1 --2- 172.21.15.12:0/2220876974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639a6000580 0x5639a600f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.118+0000 7f78b6379740 1 --2- 172.21.15.12:0/2220876974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639a5fffbc0 0x5639a600c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.118+0000 7f78b6379740 1 --2- 172.21.15.12:0/2220876974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639a5ffe9f0 0x5639a600c1d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.118+0000 7f78b6379740 1 -- 172.21.15.12:0/2220876974 >> 172.21.15.12:0/2220876974 conn(0x5639a5ff9de0 msgr2=0x5639a5ff88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:10.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.118+0000 7f7892000640 0 client.25904 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:10.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.118+0000 7f78b6379740 1 -- 172.21.15.12:0/2220876974 shutdown_connections 2024-03-23T23:04:10.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.118+0000 7f78b6379740 1 -- 172.21.15.12:0/2220876974 wait complete. 2024-03-23T23:04:10.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.119+0000 7f78b6379740 20 client.25904 trim_cache size 0 max 16384 2024-03-23T23:04:10.122 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:04:10.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.119+0000 7f78b2000640 1 -- 172.21.15.12:0/2316314464 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f78940a2880 msgr2=0x7f78940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:10.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.119+0000 7f78b2000640 1 --2- 172.21.15.12:0/2316314464 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f78940a2880 0x7f78940a4d40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f78a4006010 tx=0x7f78a4004a40 comp rx=0 tx=0).stop 2024-03-23T23:04:10.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.119+0000 7f78b2000640 1 -- 172.21.15.12:0/2316314464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78ac108db0 msgr2=0x7f78ac19e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:10.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.119+0000 7f78b2000640 1 --2- 172.21.15.12:0/2316314464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78ac108db0 0x7f78ac19e330 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f789800e9c0 tx=0x7f789800ee80 comp rx=0 tx=0).stop 2024-03-23T23:04:10.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.119+0000 7f78b2000640 1 -- 172.21.15.12:0/2316314464 shutdown_connections 2024-03-23T23:04:10.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.119+0000 7f78b2000640 1 --2- 172.21.15.12:0/2316314464 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f78940a2880 0x7f78940a4d40 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.119+0000 7f78b2000640 1 --2- 172.21.15.12:0/2316314464 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78ac109770 0x7f78ac1a53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.119+0000 7f78b2000640 1 --2- 172.21.15.12:0/2316314464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78ac108db0 0x7f78ac19e330 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.119+0000 7f78b2000640 1 --2- 172.21.15.12:0/2316314464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78ac107bc0 0x7f78ac19ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.119+0000 7f78b2000640 1 -- 172.21.15.12:0/2316314464 >> 172.21.15.12:0/2316314464 conn(0x7f78ac076b70 msgr2=0x7f78ac10c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:10.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.119+0000 7f78b2000640 1 -- 172.21.15.12:0/2316314464 shutdown_connections 2024-03-23T23:04:10.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:10.119+0000 7f78b2000640 1 -- 172.21.15.12:0/2316314464 wait complete. 2024-03-23T23:04:10.124 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:09 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3284722715' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:04:10.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.378+0000 7f536be00640 1 -- 172.21.15.12:0/2922793544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f536c102710 msgr2=0x7f536c102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:10.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.378+0000 7f536be00640 1 --2- 172.21.15.12:0/2922793544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f536c102710 0x7f536c102b90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5354009900 tx=0x7f535402bc40 comp rx=0 tx=0).stop 2024-03-23T23:04:10.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.378+0000 7f536be00640 1 -- 172.21.15.12:0/2922793544 shutdown_connections 2024-03-23T23:04:10.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.378+0000 7f536be00640 1 --2- 172.21.15.12:0/2922793544 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f536c1030d0 0x7f536c109960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.378 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.378+0000 7f536be00640 1 --2- 172.21.15.12:0/2922793544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f536c102710 0x7f536c102b90 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.378 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.378+0000 7f536be00640 1 --2- 172.21.15.12:0/2922793544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f536c101520 0x7f536c101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.378 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.378+0000 7f536be00640 1 -- 172.21.15.12:0/2922793544 >> 172.21.15.12:0/2922793544 conn(0x7f536c0fcab0 msgr2=0x7f536c0feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:10.378 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.378+0000 7f536be00640 1 -- 172.21.15.12:0/2922793544 shutdown_connections 2024-03-23T23:04:10.378 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.378+0000 7f536be00640 1 -- 172.21.15.12:0/2922793544 wait complete. 2024-03-23T23:04:10.378 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.380+0000 7f536be00640 1 Processor -- start 2024-03-23T23:04:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.380+0000 7f536be00640 1 -- start start 2024-03-23T23:04:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.380+0000 7f536be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f536c101520 0x7f536c075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.380+0000 7f536be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f536c102710 0x7f536c073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.380+0000 7f536be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f536c1030d0 0x7f536c073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.380+0000 7f536be00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c10d200 con 0x7f536c1030d0 2024-03-23T23:04:10.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.381+0000 7f536be00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c10d080 con 0x7f536c101520 2024-03-23T23:04:10.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.381+0000 7f536be00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c10d380 con 0x7f536c102710 2024-03-23T23:04:10.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.381+0000 7f536aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f536c101520 0x7f536c075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.381+0000 7f536b400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f536c1030d0 0x7f536c073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.381+0000 7f5362000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f536c102710 0x7f536c073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.381+0000 7f536aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f536c101520 0x7f536c075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37828/0 (socket says 172.21.15.12:37828) 2024-03-23T23:04:10.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.381+0000 7f536aa00640 1 -- 172.21.15.12:0/407169766 learned_addr learned my addr 172.21.15.12:0/407169766 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:10.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.381+0000 7f536aa00640 1 -- 172.21.15.12:0/407169766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f536c102710 msgr2=0x7f536c073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:10.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.381+0000 7f536aa00640 1 --2- 172.21.15.12:0/407169766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f536c102710 0x7f536c073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.381+0000 7f536aa00640 1 -- 172.21.15.12:0/407169766 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f536c1030d0 msgr2=0x7f536c073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:10.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.381+0000 7f536aa00640 1 --2- 172.21.15.12:0/407169766 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f536c1030d0 0x7f536c073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.381+0000 7f536aa00640 1 -- 172.21.15.12:0/407169766 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536c074860 con 0x7f536c101520 2024-03-23T23:04:10.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.382+0000 7f536aa00640 1 --2- 172.21.15.12:0/407169766 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f536c101520 0x7f536c075450 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f535c00ea80 tx=0x7f535c00ef40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:10.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.382+0000 7f5363e00640 1 -- 172.21.15.12:0/407169766 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535c00ce40 con 0x7f536c101520 2024-03-23T23:04:10.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.382+0000 7f5363e00640 1 -- 172.21.15.12:0/407169766 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f535c011950 con 0x7f536c101520 2024-03-23T23:04:10.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.382+0000 7f5363e00640 1 -- 172.21.15.12:0/407169766 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535c01ab80 con 0x7f536c101520 2024-03-23T23:04:10.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.382+0000 7f536be00640 1 -- 172.21.15.12:0/407169766 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f536c1a1910 con 0x7f536c101520 2024-03-23T23:04:10.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.382+0000 7f536be00640 1 -- 172.21.15.12:0/407169766 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f536c1a1de0 con 0x7f536c101520 2024-03-23T23:04:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.385+0000 7f5363e00640 1 -- 172.21.15.12:0/407169766 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f535c011b00 con 0x7f536c101520 2024-03-23T23:04:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.386+0000 7f53745bd740 1 Processor -- start 2024-03-23T23:04:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.386+0000 7f53745bd740 1 -- start start 2024-03-23T23:04:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.386+0000 7f53745bd740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5555701d13f0 0x5555701d1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.386+0000 7f53745bd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5555701d1de0 0x5555701d68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.386+0000 7f53745bd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555701d6f50 0x5555701d93e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.386+0000 7f53745bd740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555701943b0 con 0x5555701d13f0 2024-03-23T23:04:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.386+0000 7f53745bd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x555570136830 con 0x5555701d1de0 2024-03-23T23:04:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.387+0000 7f53745bd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555700df5d0 con 0x5555701d6f50 2024-03-23T23:04:10.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.387+0000 7f536aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5555701d1de0 0x5555701d68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.387+0000 7f5362000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555701d6f50 0x5555701d93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.387+0000 7f536aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5555701d1de0 0x5555701d68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37842/0 (socket says 172.21.15.12:37842) 2024-03-23T23:04:10.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.387+0000 7f5362000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555701d6f50 0x5555701d93e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:51840/0 (socket says 172.21.15.12:51840) 2024-03-23T23:04:10.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.387+0000 7f536aa00640 1 -- 172.21.15.12:0/168866804 learned_addr learned my addr 172.21.15.12:0/168866804 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:10.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.387+0000 7f536b400640 1 --2- 172.21.15.12:0/168866804 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5555701d13f0 0x5555701d1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.387+0000 7f536aa00640 1 -- 172.21.15.12:0/168866804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555701d6f50 msgr2=0x5555701d93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:10.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.387+0000 7f536aa00640 1 --2- 172.21.15.12:0/168866804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555701d6f50 0x5555701d93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.387+0000 7f536aa00640 1 -- 172.21.15.12:0/168866804 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5555701d13f0 msgr2=0x5555701d1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:10.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.387+0000 7f536aa00640 1 --2- 172.21.15.12:0/168866804 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5555701d13f0 0x5555701d1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.387+0000 7f536aa00640 1 -- 172.21.15.12:0/168866804 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5555700f72c0 con 0x5555701d1de0 2024-03-23T23:04:10.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.388+0000 7f5363e00640 1 --2- 172.21.15.12:0/407169766 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f53500a2870 0x7f53500a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.388+0000 7f536aa00640 1 --2- 172.21.15.12:0/168866804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5555701d1de0 0x5555701d68e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f535c0d7260 tx=0x7f535c0d7720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6ce7636fe2f199a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:10.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.388+0000 7f5363e00640 1 -- 172.21.15.12:0/407169766 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f535c0cefa0 con 0x7f536c101520 2024-03-23T23:04:10.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.388+0000 7f536b400640 1 --2- 172.21.15.12:0/407169766 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f53500a2870 0x7f53500a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.388+0000 7f535be00640 1 -- 172.21.15.12:0/168866804 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535c097c90 con 0x5555701d1de0 2024-03-23T23:04:10.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.388+0000 7f535be00640 1 -- 172.21.15.12:0/168866804 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f535c09c0f0 con 0x5555701d1de0 2024-03-23T23:04:10.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.388+0000 7f535be00640 1 -- 172.21.15.12:0/168866804 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535c09b350 con 0x5555701d1de0 2024-03-23T23:04:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.389+0000 7f53745bd740 1 -- 172.21.15.12:0/168866804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5555701d1de0 msgr2=0x5555701d68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.389+0000 7f536b400640 1 --2- 172.21.15.12:0/407169766 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f53500a2870 0x7f53500a4d30 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x5555701d2640 tx=0x7f5364005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.389+0000 7f53745bd740 1 --2- 172.21.15.12:0/168866804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5555701d1de0 0x5555701d68e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f535c0d7260 tx=0x7f535c0d7720 comp rx=0 tx=0).stop 2024-03-23T23:04:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.389+0000 7f53745bd740 1 -- 172.21.15.12:0/168866804 shutdown_connections 2024-03-23T23:04:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.389+0000 7f53745bd740 1 --2- 172.21.15.12:0/168866804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555701d6f50 0x5555701d93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.389+0000 7f53745bd740 1 --2- 172.21.15.12:0/168866804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5555701d1de0 0x5555701d68e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.389+0000 7f53745bd740 1 --2- 172.21.15.12:0/168866804 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5555701d13f0 0x5555701d1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.389+0000 7f53745bd740 1 -- 172.21.15.12:0/168866804 >> 172.21.15.12:0/168866804 conn(0x5555701cc960 msgr2=0x5555701ccd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.389+0000 7f53745bd740 1 -- 172.21.15.12:0/168866804 shutdown_connections 2024-03-23T23:04:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.389+0000 7f53745bd740 1 -- 172.21.15.12:0/168866804 wait complete. 2024-03-23T23:04:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.389+0000 7f53745bd740 1 -- start start 2024-03-23T23:04:10.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.390+0000 7f53745bd740 1 Processor -- start 2024-03-23T23:04:10.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.390+0000 7f53745bd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5555701d13f0 0x5555701db580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.390+0000 7f53745bd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555701d1de0 0x5555701dbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.390+0000 7f53745bd740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5555701d6f50 0x5555701e2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.390+0000 7f53745bd740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555700f2a20 con 0x5555701d6f50 2024-03-23T23:04:10.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.390+0000 7f53745bd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555701328d0 con 0x5555701d13f0 2024-03-23T23:04:10.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.390+0000 7f53745bd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555701d9f00 con 0x5555701d1de0 2024-03-23T23:04:10.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.390+0000 7f53745bd740 10 client.0 _ll_register_callbacks cb 0x55557012f1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:10.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.390+0000 7f5362000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555701d1de0 0x5555701dbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.390+0000 7f536aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5555701d13f0 0x5555701db580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.390+0000 7f5362000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555701d1de0 0x5555701dbac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:51846/0 (socket says 172.21.15.12:51846) 2024-03-23T23:04:10.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.390+0000 7f536b400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5555701d6f50 0x5555701e2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.390+0000 7f5362000640 1 -- 172.21.15.12:0/190825836 learned_addr learned my addr 172.21.15.12:0/190825836 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:10.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.391+0000 7f536aa00640 1 -- 172.21.15.12:0/190825836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555701d1de0 msgr2=0x5555701dbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:10.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.391+0000 7f536aa00640 1 --2- 172.21.15.12:0/190825836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555701d1de0 0x5555701dbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.391+0000 7f536aa00640 1 -- 172.21.15.12:0/190825836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5555701d6f50 msgr2=0x5555701e2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:10.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.391+0000 7f536aa00640 1 --2- 172.21.15.12:0/190825836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5555701d6f50 0x5555701e2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.391+0000 7f536aa00640 1 -- 172.21.15.12:0/190825836 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55556fe4b210 con 0x5555701d13f0 2024-03-23T23:04:10.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.391+0000 7f536aa00640 1 --2- 172.21.15.12:0/190825836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5555701d13f0 0x5555701db580 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x5555701d2850 tx=0x7f535c0d2a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9cac56877a6211 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:10.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.391+0000 7f5358c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:04:10.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.391+0000 7f5358c00640 1 -- 172.21.15.12:0/190825836 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535c09b530 con 0x5555701d13f0 2024-03-23T23:04:10.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.391+0000 7f5358c00640 1 -- 172.21.15.12:0/190825836 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f535c099cf0 con 0x5555701d13f0 2024-03-23T23:04:10.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.391+0000 7f53745bd740 1 -- 172.21.15.12:0/190825836 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f535c09c0f0 con 0x5555701d13f0 2024-03-23T23:04:10.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.392+0000 7f5358c00640 1 -- 172.21.15.12:0/190825836 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f535c09b7c0 con 0x5555701d13f0 2024-03-23T23:04:10.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.392+0000 7f53745bd740 10 client.25916 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.392+0000 7f53745bd740 1 -- 172.21.15.12:0/190825836 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55556ff17620 con 0x5555701d13f0 2024-03-23T23:04:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.392+0000 7f5358c00640 1 -- 172.21.15.12:0/190825836 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f535c09c0f0 con 0x5555701d13f0 2024-03-23T23:04:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.392+0000 7f5358c00640 1 -- 172.21.15.12:0/190825836 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f535c022030 con 0x5555701d13f0 2024-03-23T23:04:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.392+0000 7f53745bd740 10 client.25916 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.392+0000 7f53745bd740 10 client.25916 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:04:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.392+0000 7f53745bd740 20 client.25916 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.392+0000 7f53745bd740 1 --2- 172.21.15.12:0/190825836 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5555701e6ff0 0x5555701e93e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.392+0000 7f53745bd740 4 client.25916 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:04:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.392+0000 7f53745bd740 1 -- 172.21.15.12:0/190825836 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555570143d70 con 0x5555701e6ff0 2024-03-23T23:04:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.392+0000 7f5362000640 1 --2- 172.21.15.12:0/190825836 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5555701e6ff0 0x5555701e93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.393+0000 7f5362000640 1 --2- 172.21.15.12:0/190825836 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5555701e6ff0 0x5555701e93e0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ec7c5401faf675f server_cookie=ac9d4b37e2f82470 in_seq=0 out_seq=0 2024-03-23T23:04:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.393+0000 7f5358c00640 10 client.25916 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.394+0000 7f5358c00640 1 -- 172.21.15.12:0/190825836 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555570143d70 con 0x5555701e6ff0 2024-03-23T23:04:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.394+0000 7f5358c00640 10 client.25916 handle_command_reply: tid=0 2024-03-23T23:04:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.394+0000 7f53745bd740 1 client.25916 shutdown 2024-03-23T23:04:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.394+0000 7f53745bd740 1 -- 172.21.15.12:0/190825836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5555701d13f0 msgr2=0x5555701db580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.394+0000 7f53745bd740 1 --2- 172.21.15.12:0/190825836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5555701d13f0 0x5555701db580 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x5555701d2850 tx=0x7f535c0d2a90 comp rx=0 tx=0).stop 2024-03-23T23:04:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.394+0000 7f53745bd740 1 -- 172.21.15.12:0/190825836 shutdown_connections 2024-03-23T23:04:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.394+0000 7f53745bd740 1 --2- 172.21.15.12:0/190825836 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5555701e6ff0 0x5555701e93e0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.394+0000 7f53745bd740 1 --2- 172.21.15.12:0/190825836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5555701d6f50 0x5555701e2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.394+0000 7f53745bd740 1 --2- 172.21.15.12:0/190825836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555701d1de0 0x5555701dbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.394+0000 7f5358c00640 0 client.25916 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.394+0000 7f53745bd740 1 --2- 172.21.15.12:0/190825836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5555701d13f0 0x5555701db580 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.398 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.394+0000 7f53745bd740 1 -- 172.21.15.12:0/190825836 >> 172.21.15.12:0/190825836 conn(0x5555701cc960 msgr2=0x5555701cbb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:10.398 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.394+0000 7f53745bd740 1 -- 172.21.15.12:0/190825836 shutdown_connections 2024-03-23T23:04:10.398 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.394+0000 7f53745bd740 1 -- 172.21.15.12:0/190825836 wait complete. 2024-03-23T23:04:10.398 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.394+0000 7f53745bd740 20 client.25916 trim_cache size 0 max 16384 2024-03-23T23:04:10.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.410+0000 7f53745bd740 1 Processor -- start 2024-03-23T23:04:10.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.410+0000 7f53745bd740 1 -- start start 2024-03-23T23:04:10.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.410+0000 7f53745bd740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5555701d13f0 0x5555701edac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.410+0000 7f53745bd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555701d1de0 0x5555701ee000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.410+0000 7f53745bd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5555701d6f50 0x5555701ee540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.410+0000 7f53745bd740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c1ad000 con 0x5555701d13f0 2024-03-23T23:04:10.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.410+0000 7f53745bd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c1aab00 con 0x5555701d6f50 2024-03-23T23:04:10.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.410+0000 7f53745bd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c1a8ba0 con 0x5555701d1de0 2024-03-23T23:04:10.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.410+0000 7f5362000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555701d1de0 0x5555701ee000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.410+0000 7f536aa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5555701d13f0 0x5555701edac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.410+0000 7f536aa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5555701d13f0 0x5555701edac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53926/0 (socket says 172.21.15.12:53926) 2024-03-23T23:04:10.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.410+0000 7f536aa00640 1 -- 172.21.15.12:0/3828331832 learned_addr learned my addr 172.21.15.12:0/3828331832 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:10.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.410+0000 7f536b400640 1 --2- 172.21.15.12:0/3828331832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5555701d6f50 0x5555701ee540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.411+0000 7f5362000640 1 -- 172.21.15.12:0/3828331832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5555701d6f50 msgr2=0x5555701ee540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:10.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.411+0000 7f5362000640 1 --2- 172.21.15.12:0/3828331832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5555701d6f50 0x5555701ee540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.411+0000 7f5362000640 1 -- 172.21.15.12:0/3828331832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5555701d13f0 msgr2=0x5555701edac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:10.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.411+0000 7f5362000640 1 --2- 172.21.15.12:0/3828331832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5555701d13f0 0x5555701edac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.411+0000 7f5362000640 1 -- 172.21.15.12:0/3828331832 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5555701d2bf0 con 0x5555701d1de0 2024-03-23T23:04:10.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.411+0000 7f536aa00640 1 --2- 172.21.15.12:0/3828331832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5555701d13f0 0x5555701edac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:10.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.411+0000 7f5362000640 1 --2- 172.21.15.12:0/3828331832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555701d1de0 0x5555701ee000 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f535402c8f0 tx=0x7f535402cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee2f4bcc8d795511 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:10.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.411+0000 7f535b400640 1 -- 172.21.15.12:0/3828331832 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5354050070 con 0x5555701d1de0 2024-03-23T23:04:10.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.411+0000 7f535b400640 1 -- 172.21.15.12:0/3828331832 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f535404a110 con 0x5555701d1de0 2024-03-23T23:04:10.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.411+0000 7f535b400640 1 -- 172.21.15.12:0/3828331832 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535404a6f0 con 0x5555701d1de0 2024-03-23T23:04:10.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.412+0000 7f53745bd740 1 -- 172.21.15.12:0/3828331832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555701d1de0 msgr2=0x5555701ee000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:10.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.412+0000 7f53745bd740 1 --2- 172.21.15.12:0/3828331832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555701d1de0 0x5555701ee000 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f535402c8f0 tx=0x7f535402cf90 comp rx=0 tx=0).stop 2024-03-23T23:04:10.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.412+0000 7f53745bd740 1 -- 172.21.15.12:0/3828331832 shutdown_connections 2024-03-23T23:04:10.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.412+0000 7f53745bd740 1 --2- 172.21.15.12:0/3828331832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5555701d6f50 0x5555701ee540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.412+0000 7f53745bd740 1 --2- 172.21.15.12:0/3828331832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555701d1de0 0x5555701ee000 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.412+0000 7f53745bd740 1 --2- 172.21.15.12:0/3828331832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5555701d13f0 0x5555701edac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.412+0000 7f53745bd740 1 -- 172.21.15.12:0/3828331832 >> 172.21.15.12:0/3828331832 conn(0x5555701cc960 msgr2=0x5555701d3100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:10.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.412+0000 7f53745bd740 1 -- 172.21.15.12:0/3828331832 shutdown_connections 2024-03-23T23:04:10.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.412+0000 7f53745bd740 1 -- 172.21.15.12:0/3828331832 wait complete. 2024-03-23T23:04:10.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.413+0000 7f53745bd740 1 -- start start 2024-03-23T23:04:10.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.413+0000 7f53745bd740 1 Processor -- start 2024-03-23T23:04:10.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.413+0000 7f53745bd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5555701d13f0 0x5555701e1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.413+0000 7f53745bd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555701d1de0 0x5555701e2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.413+0000 7f53745bd740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5555701d6f50 0x5555701e0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.413+0000 7f53745bd740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c1a91a0 con 0x5555701d6f50 2024-03-23T23:04:10.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.413+0000 7f53745bd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c1aa4a0 con 0x5555701d13f0 2024-03-23T23:04:10.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.413+0000 7f53745bd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c1a97e0 con 0x5555701d1de0 2024-03-23T23:04:10.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.413+0000 7f53745bd740 10 client.0 _ll_register_callbacks cb 0x7f5364009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:10.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.414+0000 7f536aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5555701d13f0 0x5555701e1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.414+0000 7f5362000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555701d1de0 0x5555701e2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.414+0000 7f5362000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555701d1de0 0x5555701e2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:51866/0 (socket says 172.21.15.12:51866) 2024-03-23T23:04:10.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.414+0000 7f536aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5555701d13f0 0x5555701e1c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37866/0 (socket says 172.21.15.12:37866) 2024-03-23T23:04:10.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.414+0000 7f5362000640 1 -- 172.21.15.12:0/2884111863 learned_addr learned my addr 172.21.15.12:0/2884111863 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:10.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.414+0000 7f536b400640 1 --2- 172.21.15.12:0/2884111863 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5555701d6f50 0x5555701e0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.414+0000 7f5362000640 1 -- 172.21.15.12:0/2884111863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5555701d13f0 msgr2=0x5555701e1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.414+0000 7f5362000640 1 --2- 172.21.15.12:0/2884111863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5555701d13f0 0x5555701e1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.414+0000 7f5362000640 1 -- 172.21.15.12:0/2884111863 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5555701d6f50 msgr2=0x5555701e0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.414+0000 7f5362000640 1 --2- 172.21.15.12:0/2884111863 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5555701d6f50 0x5555701e0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.414+0000 7f5362000640 1 -- 172.21.15.12:0/2884111863 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5555701e0a80 con 0x5555701d1de0 2024-03-23T23:04:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.414+0000 7f536b400640 1 --2- 172.21.15.12:0/2884111863 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5555701d6f50 0x5555701e0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:04:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.414+0000 7f5362000640 1 --2- 172.21.15.12:0/2884111863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555701d1de0 0x5555701e2180 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f535402c180 tx=0x7f535402c9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e9729f8122d60e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.414+0000 7f5358c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:04:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.414+0000 7f5358c00640 1 -- 172.21.15.12:0/2884111863 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5354050070 con 0x5555701d1de0 2024-03-23T23:04:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.414+0000 7f53745bd740 1 -- 172.21.15.12:0/2884111863 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f535404a110 con 0x5555701d1de0 2024-03-23T23:04:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.414+0000 7f5358c00640 1 -- 172.21.15.12:0/2884111863 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5354004b60 con 0x5555701d1de0 2024-03-23T23:04:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.415+0000 7f5358c00640 1 -- 172.21.15.12:0/2884111863 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f535404a6f0 con 0x5555701d1de0 2024-03-23T23:04:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.415+0000 7f53745bd740 10 client.25900 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.415+0000 7f53745bd740 1 -- 172.21.15.12:0/2884111863 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x555570243180 con 0x5555701d1de0 2024-03-23T23:04:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.415+0000 7f5358c00640 1 -- 172.21.15.12:0/2884111863 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f535404a110 con 0x5555701d1de0 2024-03-23T23:04:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.415+0000 7f5358c00640 1 -- 172.21.15.12:0/2884111863 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f5354047070 con 0x5555701d1de0 2024-03-23T23:04:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.415+0000 7f53745bd740 10 client.25900 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.415+0000 7f53745bd740 10 client.25900 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:04:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.415+0000 7f53745bd740 20 client.25900 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.415+0000 7f53745bd740 1 --2- 172.21.15.12:0/2884111863 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x555570243a00 0x555570245e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.415+0000 7f53745bd740 4 client.25900 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:04:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.415+0000 7f53745bd740 1 -- 172.21.15.12:0/2884111863 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55557013e0c0 con 0x555570243a00 2024-03-23T23:04:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.415+0000 7f536aa00640 1 --2- 172.21.15.12:0/2884111863 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x555570243a00 0x555570245e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.416+0000 7f536aa00640 1 --2- 172.21.15.12:0/2884111863 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x555570243a00 0x555570245e20 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=664802636a0b4c6c server_cookie=8a32c03eeb0e2883 in_seq=0 out_seq=0 2024-03-23T23:04:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.416+0000 7f5358c00640 10 client.25900 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:10.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.428+0000 7f5358c00640 1 -- 172.21.15.12:0/2884111863 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55557013e0c0 con 0x555570243a00 2024-03-23T23:04:10.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.428+0000 7f5358c00640 10 client.25900 handle_command_reply: tid=0 2024-03-23T23:04:10.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.428+0000 7f53745bd740 1 client.25900 shutdown 2024-03-23T23:04:10.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.428+0000 7f53745bd740 1 -- 172.21.15.12:0/2884111863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555701d1de0 msgr2=0x5555701e2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:10.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.428+0000 7f53745bd740 1 --2- 172.21.15.12:0/2884111863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555701d1de0 0x5555701e2180 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f535402c180 tx=0x7f535402c9b0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.428+0000 7f53745bd740 1 -- 172.21.15.12:0/2884111863 shutdown_connections 2024-03-23T23:04:10.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.428+0000 7f53745bd740 1 --2- 172.21.15.12:0/2884111863 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x555570243a00 0x555570245e20 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.428+0000 7f53745bd740 1 --2- 172.21.15.12:0/2884111863 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5555701d6f50 0x5555701e0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.428+0000 7f53745bd740 1 --2- 172.21.15.12:0/2884111863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555701d1de0 0x5555701e2180 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.428+0000 7f5358c00640 0 client.25900 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:10.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.428+0000 7f53745bd740 1 --2- 172.21.15.12:0/2884111863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5555701d13f0 0x5555701e1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.428+0000 7f53745bd740 1 -- 172.21.15.12:0/2884111863 >> 172.21.15.12:0/2884111863 conn(0x5555701cc960 msgr2=0x5555701cb2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:10.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.428+0000 7f53745bd740 1 -- 172.21.15.12:0/2884111863 shutdown_connections 2024-03-23T23:04:10.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.428+0000 7f53745bd740 1 -- 172.21.15.12:0/2884111863 wait complete. 2024-03-23T23:04:10.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.428+0000 7f53745bd740 20 client.25900 trim_cache size 0 max 16384 2024-03-23T23:04:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:04:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:04:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:04:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:04:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.429+0000 7f536be00640 1 -- 172.21.15.12:0/407169766 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f53500a2870 msgr2=0x7f53500a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.429+0000 7f536be00640 1 --2- 172.21.15.12:0/407169766 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f53500a2870 0x7f53500a4d30 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x5555701d2640 tx=0x7f5364005f90 comp rx=0 tx=0).stop 2024-03-23T23:04:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.429+0000 7f536be00640 1 -- 172.21.15.12:0/407169766 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f536c101520 msgr2=0x7f536c075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.429+0000 7f536be00640 1 --2- 172.21.15.12:0/407169766 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f536c101520 0x7f536c075450 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f535c00ea80 tx=0x7f535c00ef40 comp rx=0 tx=0).stop 2024-03-23T23:04:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.429+0000 7f536be00640 1 -- 172.21.15.12:0/407169766 shutdown_connections 2024-03-23T23:04:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.429+0000 7f536be00640 1 --2- 172.21.15.12:0/407169766 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f53500a2870 0x7f53500a4d30 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.429+0000 7f536be00640 1 --2- 172.21.15.12:0/407169766 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f536c1030d0 0x7f536c073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.429+0000 7f536be00640 1 --2- 172.21.15.12:0/407169766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f536c102710 0x7f536c073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.429+0000 7f536be00640 1 --2- 172.21.15.12:0/407169766 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f536c101520 0x7f536c075450 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.429+0000 7f536be00640 1 -- 172.21.15.12:0/407169766 >> 172.21.15.12:0/407169766 conn(0x7f536c0fcab0 msgr2=0x7f536c105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.430+0000 7f536be00640 1 -- 172.21.15.12:0/407169766 shutdown_connections 2024-03-23T23:04:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:10.430+0000 7f536be00640 1 -- 172.21.15.12:0/407169766 wait complete. 2024-03-23T23:04:10.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:10 smithi203 ceph-mon[36693]: pgmap v461: 97 pgs: 97 active+clean; 1.1 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:04:11.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:10 smithi012 ceph-mon[31695]: pgmap v461: 97 pgs: 97 active+clean; 1.1 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:04:11.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:10 smithi173 ceph-mon[36116]: pgmap v461: 97 pgs: 97 active+clean; 1.1 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:04:11.137 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-23T23:04:11.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:04:11.137 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-23T23:04:11.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.377+0000 7f9b36e00640 1 -- 172.21.15.12:0/3793333632 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b30109870 msgr2=0x7f9b30110100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:11.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.377+0000 7f9b36e00640 1 --2- 172.21.15.12:0/3793333632 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b30109870 0x7f9b30110100 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9b28009ca0 tx=0x7f9b2802c100 comp rx=0 tx=0).stop 2024-03-23T23:04:11.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.377+0000 7f9b36e00640 1 -- 172.21.15.12:0/3793333632 shutdown_connections 2024-03-23T23:04:11.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.377+0000 7f9b36e00640 1 --2- 172.21.15.12:0/3793333632 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b30109870 0x7f9b30110100 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.377+0000 7f9b36e00640 1 --2- 172.21.15.12:0/3793333632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b30108eb0 0x7f9b30109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.377+0000 7f9b36e00640 1 --2- 172.21.15.12:0/3793333632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b30107cc0 0x7f9b301080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.377+0000 7f9b36e00640 1 -- 172.21.15.12:0/3793333632 >> 172.21.15.12:0/3793333632 conn(0x7f9b30076c60 msgr2=0x7f9b300790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:11.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.378+0000 7f9b36e00640 1 -- 172.21.15.12:0/3793333632 shutdown_connections 2024-03-23T23:04:11.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.378+0000 7f9b36e00640 1 -- 172.21.15.12:0/3793333632 wait complete. 2024-03-23T23:04:11.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.379+0000 7f9b36e00640 1 Processor -- start 2024-03-23T23:04:11.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.379+0000 7f9b36e00640 1 -- start start 2024-03-23T23:04:11.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.379+0000 7f9b36e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b30107cc0 0x7f9b3019df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.379+0000 7f9b36e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b30108eb0 0x7f9b3019e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.379+0000 7f9b36e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b30109870 0x7f9b301a5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.379+0000 7f9b36e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b30113940 con 0x7f9b30108eb0 2024-03-23T23:04:11.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.379+0000 7f9b36e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b301137c0 con 0x7f9b30109870 2024-03-23T23:04:11.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.379+0000 7f9b36e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b30113ac0 con 0x7f9b30107cc0 2024-03-23T23:04:11.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.379+0000 7f9b2fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b30107cc0 0x7f9b3019df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.379+0000 7f9b2fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b30107cc0 0x7f9b3019df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:51892/0 (socket says 172.21.15.12:51892) 2024-03-23T23:04:11.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.379+0000 7f9b2fe00640 1 -- 172.21.15.12:0/382669392 learned_addr learned my addr 172.21.15.12:0/382669392 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:11.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.379+0000 7f9b27e00640 1 --2- 172.21.15.12:0/382669392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b30108eb0 0x7f9b3019e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.379+0000 7f9b34e00640 1 --2- 172.21.15.12:0/382669392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b30109870 0x7f9b301a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.380+0000 7f9b2fe00640 1 -- 172.21.15.12:0/382669392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b30109870 msgr2=0x7f9b301a5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:11.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.380+0000 7f9b2fe00640 1 --2- 172.21.15.12:0/382669392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b30109870 0x7f9b301a5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.380+0000 7f9b2fe00640 1 -- 172.21.15.12:0/382669392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b30108eb0 msgr2=0x7f9b3019e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:11.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.380+0000 7f9b2fe00640 1 --2- 172.21.15.12:0/382669392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b30108eb0 0x7f9b3019e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.380+0000 7f9b2fe00640 1 -- 172.21.15.12:0/382669392 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b301a5ba0 con 0x7f9b30107cc0 2024-03-23T23:04:11.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.380+0000 7f9b2fe00640 1 --2- 172.21.15.12:0/382669392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b30107cc0 0x7f9b3019df80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9b2000d770 tx=0x7f9b2000dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:11.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.380+0000 7f9b2d600640 1 -- 172.21.15.12:0/382669392 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b20017070 con 0x7f9b30107cc0 2024-03-23T23:04:11.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.380+0000 7f9b2d600640 1 -- 172.21.15.12:0/382669392 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b20009c30 con 0x7f9b30107cc0 2024-03-23T23:04:11.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.380+0000 7f9b2d600640 1 -- 172.21.15.12:0/382669392 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b20003dd0 con 0x7f9b30107cc0 2024-03-23T23:04:11.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.381+0000 7f9b36e00640 1 -- 172.21.15.12:0/382669392 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b301a5e90 con 0x7f9b30107cc0 2024-03-23T23:04:11.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.381+0000 7f9b36e00640 1 -- 172.21.15.12:0/382669392 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b301a62a0 con 0x7f9b30107cc0 2024-03-23T23:04:11.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.385+0000 7f9b39bf5740 1 Processor -- start 2024-03-23T23:04:11.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.385+0000 7f9b39bf5740 1 -- start start 2024-03-23T23:04:11.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.385+0000 7f9b2d600640 1 -- 172.21.15.12:0/382669392 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9b20003f70 con 0x7f9b30107cc0 2024-03-23T23:04:11.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.386+0000 7f9b39bf5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d840c0a0 0x5561d840c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.386+0000 7f9b39bf5740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561d840ca90 0x5561d8411590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.386+0000 7f9b39bf5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561d8411c00 0x5561d8414090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.386+0000 7f9b39bf5740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561d8371eb0 con 0x5561d840ca90 2024-03-23T23:04:11.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.386+0000 7f9b39bf5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561d831aa60 con 0x5561d840c0a0 2024-03-23T23:04:11.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.386+0000 7f9b39bf5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561d836df50 con 0x5561d8411c00 2024-03-23T23:04:11.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.386+0000 7f9b34e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d840c0a0 0x5561d840c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.386+0000 7f9b34e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d840c0a0 0x5561d840c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37886/0 (socket says 172.21.15.12:37886) 2024-03-23T23:04:11.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.386+0000 7f9b34e00640 1 -- 172.21.15.12:0/1881012510 learned_addr learned my addr 172.21.15.12:0/1881012510 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:11.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.386+0000 7f9b2fe00640 1 --2- 172.21.15.12:0/1881012510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561d840ca90 0x5561d8411590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.386+0000 7f9b27e00640 1 --2- 172.21.15.12:0/1881012510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561d8411c00 0x5561d8414090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.386+0000 7f9b34e00640 1 -- 172.21.15.12:0/1881012510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561d8411c00 msgr2=0x5561d8414090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:11.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.386+0000 7f9b34e00640 1 --2- 172.21.15.12:0/1881012510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561d8411c00 0x5561d8414090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.386+0000 7f9b34e00640 1 -- 172.21.15.12:0/1881012510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561d840ca90 msgr2=0x5561d8411590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:11.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.386+0000 7f9b34e00640 1 --2- 172.21.15.12:0/1881012510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561d840ca90 0x5561d8411590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.386+0000 7f9b34e00640 1 -- 172.21.15.12:0/1881012510 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5561d814e7e0 con 0x5561d840c0a0 2024-03-23T23:04:11.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.386+0000 7f9b2fe00640 1 --2- 172.21.15.12:0/1881012510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561d840ca90 0x5561d8411590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:04:11.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.387+0000 7f9b34e00640 1 --2- 172.21.15.12:0/1881012510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d840c0a0 0x5561d840c4c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9b2802c710 tx=0x7f9b280323d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a412fb3fa410d458 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:11.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.387+0000 7f9b25600640 1 -- 172.21.15.12:0/1881012510 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b28043070 con 0x5561d840c0a0 2024-03-23T23:04:11.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.387+0000 7f9b25600640 1 -- 172.21.15.12:0/1881012510 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b28032c90 con 0x5561d840c0a0 2024-03-23T23:04:11.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.387+0000 7f9b25600640 1 -- 172.21.15.12:0/1881012510 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b28039040 con 0x5561d840c0a0 2024-03-23T23:04:11.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.387+0000 7f9b2d600640 1 --2- 172.21.15.12:0/382669392 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9b140a2880 0x7f9b140a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.387+0000 7f9b2d600640 1 -- 172.21.15.12:0/382669392 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9b200ce300 con 0x7f9b30107cc0 2024-03-23T23:04:11.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.387+0000 7f9b2fe00640 1 --2- 172.21.15.12:0/382669392 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9b140a2880 0x7f9b140a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.388+0000 7f9b39bf5740 1 -- 172.21.15.12:0/1881012510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d840c0a0 msgr2=0x5561d840c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:11.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.388+0000 7f9b39bf5740 1 --2- 172.21.15.12:0/1881012510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d840c0a0 0x5561d840c4c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9b2802c710 tx=0x7f9b280323d0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.388+0000 7f9b39bf5740 1 -- 172.21.15.12:0/1881012510 shutdown_connections 2024-03-23T23:04:11.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.388+0000 7f9b39bf5740 1 --2- 172.21.15.12:0/1881012510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561d8411c00 0x5561d8414090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.388+0000 7f9b39bf5740 1 --2- 172.21.15.12:0/1881012510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561d840ca90 0x5561d8411590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.388+0000 7f9b39bf5740 1 --2- 172.21.15.12:0/1881012510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d840c0a0 0x5561d840c4c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.388+0000 7f9b39bf5740 1 -- 172.21.15.12:0/1881012510 >> 172.21.15.12:0/1881012510 conn(0x5561d8407610 msgr2=0x5561d8407a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:11.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.388+0000 7f9b39bf5740 1 -- 172.21.15.12:0/1881012510 shutdown_connections 2024-03-23T23:04:11.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.388+0000 7f9b39bf5740 1 -- 172.21.15.12:0/1881012510 wait complete. 2024-03-23T23:04:11.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.388+0000 7f9b2fe00640 1 --2- 172.21.15.12:0/382669392 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9b140a2880 0x7f9b140a4d40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9b2009aa00 tx=0x7f9b20004590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:11.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.388+0000 7f9b39bf5740 1 -- start start 2024-03-23T23:04:11.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.389+0000 7f9b39bf5740 1 Processor -- start 2024-03-23T23:04:11.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.389+0000 7f9b39bf5740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561d840c0a0 0x5561d8416250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.389+0000 7f9b39bf5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d840ca90 0x5561d8416790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.389+0000 7f9b39bf5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561d8411c00 0x5561d841d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.389+0000 7f9b39bf5740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561d8414b50 con 0x5561d840c0a0 2024-03-23T23:04:11.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.389+0000 7f9b39bf5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561d832deb0 con 0x5561d840ca90 2024-03-23T23:04:11.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.389+0000 7f9b39bf5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561d8414cd0 con 0x5561d8411c00 2024-03-23T23:04:11.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.389+0000 7f9b39bf5740 10 client.0 _ll_register_callbacks cb 0x5561d8322150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:11.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.389+0000 7f9b27e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d840ca90 0x5561d8416790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.389+0000 7f9b27e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d840ca90 0x5561d8416790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37900/0 (socket says 172.21.15.12:37900) 2024-03-23T23:04:11.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.389+0000 7f9b27e00640 1 -- 172.21.15.12:0/2491569628 learned_addr learned my addr 172.21.15.12:0/2491569628 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:11.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.389+0000 7f9b34e00640 1 --2- 172.21.15.12:0/2491569628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561d840c0a0 0x5561d8416250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.390+0000 7f9b34e00640 1 --2- 172.21.15.12:0/2491569628 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561d8411c00 0x5561d841d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.390+0000 7f9b27e00640 1 -- 172.21.15.12:0/2491569628 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561d8411c00 msgr2=0x5561d841d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:11.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.390+0000 7f9b27e00640 1 --2- 172.21.15.12:0/2491569628 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561d8411c00 0x5561d841d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.390+0000 7f9b27e00640 1 -- 172.21.15.12:0/2491569628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561d840c0a0 msgr2=0x5561d8416250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:11.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.390+0000 7f9b27e00640 1 --2- 172.21.15.12:0/2491569628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561d840c0a0 0x5561d8416250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.390+0000 7f9b27e00640 1 -- 172.21.15.12:0/2491569628 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5561d81aef70 con 0x5561d840ca90 2024-03-23T23:04:11.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.390+0000 7f9b27e00640 1 --2- 172.21.15.12:0/2491569628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d840ca90 0x5561d8416790 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9b18012720 tx=0x7f9b18012be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9afdbdb6a56fe66d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:11.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.390+0000 7f9b1e000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:04:11.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.390+0000 7f9b1e000640 1 -- 172.21.15.12:0/2491569628 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1800f920 con 0x5561d840ca90 2024-03-23T23:04:11.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.390+0000 7f9b39bf5740 1 -- 172.21.15.12:0/2491569628 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9b28032c90 con 0x5561d840ca90 2024-03-23T23:04:11.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.390+0000 7f9b1e000640 1 -- 172.21.15.12:0/2491569628 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9b1800fac0 con 0x5561d840ca90 2024-03-23T23:04:11.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.391+0000 7f9b1e000640 1 -- 172.21.15.12:0/2491569628 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b1800fcc0 con 0x5561d840ca90 2024-03-23T23:04:11.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.391+0000 7f9b39bf5740 10 client.25934 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:11.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.391+0000 7f9b39bf5740 1 -- 172.21.15.12:0/2491569628 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5561d8046170 con 0x5561d840ca90 2024-03-23T23:04:11.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.391+0000 7f9b1e000640 1 -- 172.21.15.12:0/2491569628 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b28032c90 con 0x5561d840ca90 2024-03-23T23:04:11.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.391+0000 7f9b1e000640 1 -- 172.21.15.12:0/2491569628 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f9b1801c4b0 con 0x5561d840ca90 2024-03-23T23:04:11.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.391+0000 7f9b39bf5740 10 client.25934 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:11.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.391+0000 7f9b39bf5740 10 client.25934 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:04:11.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.391+0000 7f9b39bf5740 20 client.25934 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:11.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.391+0000 7f9b39bf5740 1 --2- 172.21.15.12:0/2491569628 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5561d8421fa0 0x5561d8424420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.391+0000 7f9b39bf5740 4 client.25934 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:04:11.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.391+0000 7f9b39bf5740 1 -- 172.21.15.12:0/2491569628 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5561d837f3f0 con 0x5561d8421fa0 2024-03-23T23:04:11.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.392+0000 7f9b34e00640 1 --2- 172.21.15.12:0/2491569628 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5561d8421fa0 0x5561d8424420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.392+0000 7f9b34e00640 1 --2- 172.21.15.12:0/2491569628 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5561d8421fa0 0x5561d8424420 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24db8076fa30b09 server_cookie=62785be00dd2ae3 in_seq=0 out_seq=0 2024-03-23T23:04:11.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.392+0000 7f9b1e000640 10 client.25934 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:11.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.393+0000 7f9b1e000640 1 -- 172.21.15.12:0/2491569628 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5561d837f3f0 con 0x5561d8421fa0 2024-03-23T23:04:11.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.393+0000 7f9b1e000640 10 client.25934 handle_command_reply: tid=0 2024-03-23T23:04:11.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.393+0000 7f9b39bf5740 1 client.25934 shutdown 2024-03-23T23:04:11.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.393+0000 7f9b39bf5740 1 -- 172.21.15.12:0/2491569628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d840ca90 msgr2=0x5561d8416790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:11.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.393+0000 7f9b39bf5740 1 --2- 172.21.15.12:0/2491569628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d840ca90 0x5561d8416790 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9b18012720 tx=0x7f9b18012be0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.393+0000 7f9b39bf5740 1 -- 172.21.15.12:0/2491569628 shutdown_connections 2024-03-23T23:04:11.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.393+0000 7f9b39bf5740 1 --2- 172.21.15.12:0/2491569628 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5561d8421fa0 0x5561d8424420 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.393+0000 7f9b39bf5740 1 --2- 172.21.15.12:0/2491569628 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561d8411c00 0x5561d841d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.393+0000 7f9b39bf5740 1 --2- 172.21.15.12:0/2491569628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d840ca90 0x5561d8416790 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.393+0000 7f9b39bf5740 1 --2- 172.21.15.12:0/2491569628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561d840c0a0 0x5561d8416250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.393+0000 7f9b1e000640 0 client.25934 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:11.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.393+0000 7f9b39bf5740 1 -- 172.21.15.12:0/2491569628 >> 172.21.15.12:0/2491569628 conn(0x5561d8407610 msgr2=0x5561d8406830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:11.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.393+0000 7f9b39bf5740 1 -- 172.21.15.12:0/2491569628 shutdown_connections 2024-03-23T23:04:11.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.394+0000 7f9b39bf5740 1 -- 172.21.15.12:0/2491569628 wait complete. 2024-03-23T23:04:11.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.394+0000 7f9b39bf5740 20 client.25934 trim_cache size 0 max 16384 2024-03-23T23:04:11.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.412+0000 7f9b39bf5740 1 Processor -- start 2024-03-23T23:04:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.412+0000 7f9b39bf5740 1 -- start start 2024-03-23T23:04:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.412+0000 7f9b39bf5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d840c0a0 0x5561d840d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.412+0000 7f9b39bf5740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561d840ca90 0x5561d841d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.412+0000 7f9b39bf5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561d8411c00 0x5561d840da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.412+0000 7f9b39bf5740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b301b1520 con 0x5561d840ca90 2024-03-23T23:04:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.412+0000 7f9b39bf5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b301af020 con 0x5561d840c0a0 2024-03-23T23:04:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.412+0000 7f9b39bf5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b301ad0c0 con 0x5561d8411c00 2024-03-23T23:04:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.412+0000 7f9b27e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561d840ca90 0x5561d841d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.412+0000 7f9b34e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d840c0a0 0x5561d840d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.412+0000 7f9b27e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561d840ca90 0x5561d841d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53984/0 (socket says 172.21.15.12:53984) 2024-03-23T23:04:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.412+0000 7f9b34e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d840c0a0 0x5561d840d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37914/0 (socket says 172.21.15.12:37914) 2024-03-23T23:04:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.412+0000 7f9b27e00640 1 -- 172.21.15.12:0/1437067156 learned_addr learned my addr 172.21.15.12:0/1437067156 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.412+0000 7f9b34e00640 1 --2- 172.21.15.12:0/1437067156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561d8411c00 0x5561d840da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.413+0000 7f9b34e00640 1 -- 172.21.15.12:0/1437067156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561d8411c00 msgr2=0x5561d840da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:11.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.413+0000 7f9b34e00640 1 --2- 172.21.15.12:0/1437067156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561d8411c00 0x5561d840da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.413+0000 7f9b34e00640 1 -- 172.21.15.12:0/1437067156 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561d840ca90 msgr2=0x5561d841d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:11.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.413+0000 7f9b34e00640 1 --2- 172.21.15.12:0/1437067156 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561d840ca90 0x5561d841d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.413+0000 7f9b34e00640 1 -- 172.21.15.12:0/1437067156 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5561d84166b0 con 0x5561d840c0a0 2024-03-23T23:04:11.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.413+0000 7f9b27e00640 1 --2- 172.21.15.12:0/1437067156 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561d840ca90 0x5561d841d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:11.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.413+0000 7f9b34e00640 1 --2- 172.21.15.12:0/1437067156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d840c0a0 0x5561d840d530 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9b28009ca0 tx=0x7f9b280095a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4cfa279c42cd00c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.413+0000 7f9b24c00640 1 -- 172.21.15.12:0/1437067156 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b28051040 con 0x5561d840c0a0 2024-03-23T23:04:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.413+0000 7f9b24c00640 1 -- 172.21.15.12:0/1437067156 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b28003cd0 con 0x5561d840c0a0 2024-03-23T23:04:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.413+0000 7f9b24c00640 1 -- 172.21.15.12:0/1437067156 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b28039040 con 0x5561d840c0a0 2024-03-23T23:04:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.414+0000 7f9b39bf5740 1 -- 172.21.15.12:0/1437067156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d840c0a0 msgr2=0x5561d840d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.414+0000 7f9b39bf5740 1 --2- 172.21.15.12:0/1437067156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d840c0a0 0x5561d840d530 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9b28009ca0 tx=0x7f9b280095a0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.414+0000 7f9b39bf5740 1 -- 172.21.15.12:0/1437067156 shutdown_connections 2024-03-23T23:04:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.414+0000 7f9b39bf5740 1 --2- 172.21.15.12:0/1437067156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561d8411c00 0x5561d840da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.414+0000 7f9b39bf5740 1 --2- 172.21.15.12:0/1437067156 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561d840ca90 0x5561d841d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.414+0000 7f9b39bf5740 1 --2- 172.21.15.12:0/1437067156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d840c0a0 0x5561d840d530 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.414+0000 7f9b39bf5740 1 -- 172.21.15.12:0/1437067156 >> 172.21.15.12:0/1437067156 conn(0x5561d8407610 msgr2=0x5561d840b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.414+0000 7f9b39bf5740 1 -- 172.21.15.12:0/1437067156 shutdown_connections 2024-03-23T23:04:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.414+0000 7f9b39bf5740 1 -- 172.21.15.12:0/1437067156 wait complete. 2024-03-23T23:04:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.415+0000 7f9b39bf5740 1 -- start start 2024-03-23T23:04:11.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.415+0000 7f9b39bf5740 1 Processor -- start 2024-03-23T23:04:11.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.415+0000 7f9b39bf5740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561d840c0a0 0x5561d841d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.415+0000 7f9b39bf5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561d840ca90 0x5561d841b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.415+0000 7f9b39bf5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d8411c00 0x5561d841bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.415+0000 7f9b39bf5740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b301ad6c0 con 0x5561d840c0a0 2024-03-23T23:04:11.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.415+0000 7f9b39bf5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b301ae9c0 con 0x5561d8411c00 2024-03-23T23:04:11.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.415+0000 7f9b39bf5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b301add00 con 0x5561d840ca90 2024-03-23T23:04:11.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.415+0000 7f9b39bf5740 10 client.0 _ll_register_callbacks cb 0x7f9b18009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:11.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.415+0000 7f9b27e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561d840ca90 0x5561d841b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.416+0000 7f9b27e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561d840ca90 0x5561d841b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:51926/0 (socket says 172.21.15.12:51926) 2024-03-23T23:04:11.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.416+0000 7f9b27e00640 1 -- 172.21.15.12:0/1293667547 learned_addr learned my addr 172.21.15.12:0/1293667547 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:11.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.416+0000 7f9b34e00640 1 --2- 172.21.15.12:0/1293667547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561d840c0a0 0x5561d841d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.416+0000 7f9b34e00640 1 --2- 172.21.15.12:0/1293667547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d8411c00 0x5561d841bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.416+0000 7f9b27e00640 1 -- 172.21.15.12:0/1293667547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d8411c00 msgr2=0x5561d841bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:11.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.416+0000 7f9b27e00640 1 --2- 172.21.15.12:0/1293667547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d8411c00 0x5561d841bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.416+0000 7f9b27e00640 1 -- 172.21.15.12:0/1293667547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561d840c0a0 msgr2=0x5561d841d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:11.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.416+0000 7f9b27e00640 1 --2- 172.21.15.12:0/1293667547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561d840c0a0 0x5561d841d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.416+0000 7f9b27e00640 1 -- 172.21.15.12:0/1293667547 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5561d841c440 con 0x5561d840ca90 2024-03-23T23:04:11.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.416+0000 7f9b34e00640 1 --2- 172.21.15.12:0/1293667547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d8411c00 0x5561d841bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:04:11.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.416+0000 7f9b27e00640 1 --2- 172.21.15.12:0/1293667547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561d840ca90 0x5561d841b9c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9b180098e0 tx=0x7f9b1800f820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf74d80822559d89 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:11.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.416+0000 7f9b1e000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:04:11.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.416+0000 7f9b1e000640 1 -- 172.21.15.12:0/1293667547 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1801c420 con 0x5561d840ca90 2024-03-23T23:04:11.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.417+0000 7f9b39bf5740 1 -- 172.21.15.12:0/1293667547 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9b28003cd0 con 0x5561d840ca90 2024-03-23T23:04:11.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.417+0000 7f9b1e000640 1 -- 172.21.15.12:0/1293667547 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9b1801c5c0 con 0x5561d840ca90 2024-03-23T23:04:11.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.417+0000 7f9b1e000640 1 -- 172.21.15.12:0/1293667547 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b18005040 con 0x5561d840ca90 2024-03-23T23:04:11.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.417+0000 7f9b39bf5740 10 client.25918 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:11.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.417+0000 7f9b39bf5740 1 -- 172.21.15.12:0/1293667547 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5561d847e650 con 0x5561d840ca90 2024-03-23T23:04:11.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.417+0000 7f9b1e000640 1 -- 172.21.15.12:0/1293667547 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b28003cd0 con 0x5561d840ca90 2024-03-23T23:04:11.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.417+0000 7f9b1e000640 1 -- 172.21.15.12:0/1293667547 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f9b18004c10 con 0x5561d840ca90 2024-03-23T23:04:11.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.417+0000 7f9b39bf5740 10 client.25918 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:11.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.417+0000 7f9b39bf5740 10 client.25918 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:04:11.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.417+0000 7f9b39bf5740 20 client.25918 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:11.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.417+0000 7f9b39bf5740 1 --2- 172.21.15.12:0/1293667547 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5561d847ef80 0x5561d84813a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.417+0000 7f9b39bf5740 4 client.25918 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-23T23:04:11.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.417+0000 7f9b39bf5740 1 -- 172.21.15.12:0/1293667547 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5561d8379740 con 0x5561d847ef80 2024-03-23T23:04:11.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.418+0000 7f9b34e00640 1 --2- 172.21.15.12:0/1293667547 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5561d847ef80 0x5561d84813a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.418+0000 7f9b34e00640 1 --2- 172.21.15.12:0/1293667547 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5561d847ef80 0x5561d84813a0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec42bc12718b996 server_cookie=576afb3b837d71f7 in_seq=0 out_seq=0 2024-03-23T23:04:11.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.418+0000 7f9b1e000640 10 client.25918 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.419+0000 7f9b1e000640 1 -- 172.21.15.12:0/1293667547 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5561d8379740 con 0x5561d847ef80 2024-03-23T23:04:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.419+0000 7f9b1e000640 10 client.25918 handle_command_reply: tid=0 2024-03-23T23:04:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.419+0000 7f9b39bf5740 1 client.25918 shutdown 2024-03-23T23:04:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.419+0000 7f9b39bf5740 1 -- 172.21.15.12:0/1293667547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561d840ca90 msgr2=0x5561d841b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.419+0000 7f9b39bf5740 1 --2- 172.21.15.12:0/1293667547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561d840ca90 0x5561d841b9c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9b180098e0 tx=0x7f9b1800f820 comp rx=0 tx=0).stop 2024-03-23T23:04:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.420+0000 7f9b39bf5740 1 -- 172.21.15.12:0/1293667547 shutdown_connections 2024-03-23T23:04:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.420+0000 7f9b39bf5740 1 --2- 172.21.15.12:0/1293667547 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5561d847ef80 0x5561d84813a0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.420+0000 7f9b39bf5740 1 --2- 172.21.15.12:0/1293667547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561d8411c00 0x5561d841bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.420+0000 7f9b39bf5740 1 --2- 172.21.15.12:0/1293667547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561d840ca90 0x5561d841b9c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.420+0000 7f9b39bf5740 1 --2- 172.21.15.12:0/1293667547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561d840c0a0 0x5561d841d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.420+0000 7f9b39bf5740 1 -- 172.21.15.12:0/1293667547 >> 172.21.15.12:0/1293667547 conn(0x5561d8407610 msgr2=0x5561d8405fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.420+0000 7f9b1e000640 0 client.25918 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.420+0000 7f9b39bf5740 1 -- 172.21.15.12:0/1293667547 shutdown_connections 2024-03-23T23:04:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.420+0000 7f9b39bf5740 1 -- 172.21.15.12:0/1293667547 wait complete. 2024-03-23T23:04:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.420+0000 7f9b39bf5740 20 client.25918 trim_cache size 0 max 16384 2024-03-23T23:04:11.423 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:04:11.423 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-03-23T23:04:11.423 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "22685496-b5b7-441a-9e68-8806a05a51b9", 2024-03-23T23:04:11.423 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-03-23T23:04:11.423 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:04:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.421+0000 7f9b36e00640 1 -- 172.21.15.12:0/382669392 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9b140a2880 msgr2=0x7f9b140a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.421+0000 7f9b36e00640 1 --2- 172.21.15.12:0/382669392 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9b140a2880 0x7f9b140a4d40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9b2009aa00 tx=0x7f9b20004590 comp rx=0 tx=0).stop 2024-03-23T23:04:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.421+0000 7f9b36e00640 1 -- 172.21.15.12:0/382669392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b30107cc0 msgr2=0x7f9b3019df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.421+0000 7f9b36e00640 1 --2- 172.21.15.12:0/382669392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b30107cc0 0x7f9b3019df80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9b2000d770 tx=0x7f9b2000dc30 comp rx=0 tx=0).stop 2024-03-23T23:04:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.421+0000 7f9b36e00640 1 -- 172.21.15.12:0/382669392 shutdown_connections 2024-03-23T23:04:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.421+0000 7f9b36e00640 1 --2- 172.21.15.12:0/382669392 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9b140a2880 0x7f9b140a4d40 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.421+0000 7f9b36e00640 1 --2- 172.21.15.12:0/382669392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b30109870 0x7f9b301a5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.421+0000 7f9b36e00640 1 --2- 172.21.15.12:0/382669392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b30108eb0 0x7f9b3019e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.421+0000 7f9b36e00640 1 --2- 172.21.15.12:0/382669392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b30107cc0 0x7f9b3019df80 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.421+0000 7f9b36e00640 1 -- 172.21.15.12:0/382669392 >> 172.21.15.12:0/382669392 conn(0x7f9b30076c60 msgr2=0x7f9b3010c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:11.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.421+0000 7f9b36e00640 1 -- 172.21.15.12:0/382669392 shutdown_connections 2024-03-23T23:04:11.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.421+0000 7f9b36e00640 1 -- 172.21.15.12:0/382669392 wait complete. 2024-03-23T23:04:11.438 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:04:11.438 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:04:11.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.656+0000 7f253ec00640 1 -- 172.21.15.12:0/2534454776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2538103130 msgr2=0x7f25381099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:11.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.656+0000 7f253ec00640 1 --2- 172.21.15.12:0/2534454776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2538103130 0x7f25381099c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2530009ca0 tx=0x7f253002c0f0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.656+0000 7f253ec00640 1 -- 172.21.15.12:0/2534454776 shutdown_connections 2024-03-23T23:04:11.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.656+0000 7f253ec00640 1 --2- 172.21.15.12:0/2534454776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2538103130 0x7f25381099c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.656+0000 7f253ec00640 1 --2- 172.21.15.12:0/2534454776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2538102770 0x7f2538102bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.656+0000 7f253ec00640 1 --2- 172.21.15.12:0/2534454776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2538101580 0x7f25381019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.656+0000 7f253ec00640 1 -- 172.21.15.12:0/2534454776 >> 172.21.15.12:0/2534454776 conn(0x7f25380fcb10 msgr2=0x7f25380fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:11.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.656+0000 7f253ec00640 1 -- 172.21.15.12:0/2534454776 shutdown_connections 2024-03-23T23:04:11.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.656+0000 7f253ec00640 1 -- 172.21.15.12:0/2534454776 wait complete. 2024-03-23T23:04:11.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.657+0000 7f253ec00640 1 Processor -- start 2024-03-23T23:04:11.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.657+0000 7f253ec00640 1 -- start start 2024-03-23T23:04:11.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.658+0000 7f253ec00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2538101580 0x7f25381999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.658+0000 7f253ec00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2538102770 0x7f2538199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.658+0000 7f253ec00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2538103130 0x7f25381a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.658+0000 7f253ec00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253810d210 con 0x7f2538102770 2024-03-23T23:04:11.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.658+0000 7f253ec00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253810d090 con 0x7f2538101580 2024-03-23T23:04:11.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.658+0000 7f253ec00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253810d390 con 0x7f2538103130 2024-03-23T23:04:11.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.658+0000 7f253ce00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2538102770 0x7f2538199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.658+0000 7f253ce00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2538102770 0x7f2538199f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54014/0 (socket says 172.21.15.12:54014) 2024-03-23T23:04:11.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.658+0000 7f253ce00640 1 -- 172.21.15.12:0/588713173 learned_addr learned my addr 172.21.15.12:0/588713173 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:11.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.658+0000 7f253d800640 1 --2- 172.21.15.12:0/588713173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2538101580 0x7f25381999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.658+0000 7f253e200640 1 --2- 172.21.15.12:0/588713173 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2538103130 0x7f25381a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.659+0000 7f253ce00640 1 -- 172.21.15.12:0/588713173 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2538103130 msgr2=0x7f25381a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:11.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.659+0000 7f253ce00640 1 --2- 172.21.15.12:0/588713173 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2538103130 0x7f25381a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.659+0000 7f253ce00640 1 -- 172.21.15.12:0/588713173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2538101580 msgr2=0x7f25381999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:11.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.659+0000 7f253ce00640 1 --2- 172.21.15.12:0/588713173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2538101580 0x7f25381999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.659+0000 7f253ce00640 1 -- 172.21.15.12:0/588713173 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25381a1600 con 0x7f2538102770 2024-03-23T23:04:11.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.659+0000 7f253d800640 1 --2- 172.21.15.12:0/588713173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2538101580 0x7f25381999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:11.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.659+0000 7f253ce00640 1 --2- 172.21.15.12:0/588713173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2538102770 0x7f2538199f20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f252c00ea60 tx=0x7f252c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:11.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.659+0000 7f2536000640 1 -- 172.21.15.12:0/588713173 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f252c00cde0 con 0x7f2538102770 2024-03-23T23:04:11.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.659+0000 7f2536000640 1 -- 172.21.15.12:0/588713173 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f252c0035f0 con 0x7f2538102770 2024-03-23T23:04:11.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.659+0000 7f253ec00640 1 -- 172.21.15.12:0/588713173 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25381a18f0 con 0x7f2538102770 2024-03-23T23:04:11.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.659+0000 7f2536000640 1 -- 172.21.15.12:0/588713173 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f252c010670 con 0x7f2538102770 2024-03-23T23:04:11.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.660+0000 7f253ec00640 1 -- 172.21.15.12:0/588713173 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25381a1e40 con 0x7f2538102770 2024-03-23T23:04:11.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.663+0000 7f2542f6d740 1 Processor -- start 2024-03-23T23:04:11.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.663+0000 7f2536000640 1 -- 172.21.15.12:0/588713173 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f252c0037a0 con 0x7f2538102770 2024-03-23T23:04:11.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.663+0000 7f2542f6d740 1 -- start start 2024-03-23T23:04:11.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.663+0000 7f2542f6d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589526e9050 0x5589526e9470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.663+0000 7f2542f6d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589526e9a40 0x5589526ee540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.663+0000 7f2542f6d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589526eebb0 0x5589526f1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.663+0000 7f2542f6d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895264e250 con 0x5589526e9a40 2024-03-23T23:04:11.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.663+0000 7f2542f6d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589525f6a30 con 0x5589526eebb0 2024-03-23T23:04:11.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.663+0000 7f2542f6d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895264a2f0 con 0x5589526e9050 2024-03-23T23:04:11.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.664+0000 7f253ce00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589526eebb0 0x5589526f1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.664+0000 7f253e200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589526e9050 0x5589526e9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.664+0000 7f253d800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589526e9a40 0x5589526ee540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.664+0000 7f253e200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589526e9050 0x5589526e9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:51966/0 (socket says 172.21.15.12:51966) 2024-03-23T23:04:11.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.664+0000 7f253e200640 1 -- 172.21.15.12:0/2556128883 learned_addr learned my addr 172.21.15.12:0/2556128883 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:11.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.664+0000 7f253d800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589526e9a40 0x5589526ee540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54026/0 (socket says 172.21.15.12:54026) 2024-03-23T23:04:11.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.664+0000 7f253ce00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589526eebb0 0x5589526f1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37942/0 (socket says 172.21.15.12:37942) 2024-03-23T23:04:11.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.664+0000 7f253e200640 1 -- 172.21.15.12:0/2556128883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589526eebb0 msgr2=0x5589526f1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:11.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.664+0000 7f253e200640 1 --2- 172.21.15.12:0/2556128883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589526eebb0 0x5589526f1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.664+0000 7f253e200640 1 -- 172.21.15.12:0/2556128883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589526e9a40 msgr2=0x5589526ee540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:11.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.664+0000 7f253e200640 1 --2- 172.21.15.12:0/2556128883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589526e9a40 0x5589526ee540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.664+0000 7f253e200640 1 -- 172.21.15.12:0/2556128883 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5589525e1690 con 0x5589526e9050 2024-03-23T23:04:11.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.665+0000 7f253e200640 1 --2- 172.21.15.12:0/2556128883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589526e9050 0x5589526e9470 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f253002c700 tx=0x7f25300323d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49a6123e45585210 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:11.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.665+0000 7f2536000640 1 --2- 172.21.15.12:0/588713173 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f25200a2800 0x7f25200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.665+0000 7f2536000640 1 -- 172.21.15.12:0/588713173 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f252c0ceda0 con 0x7f2538102770 2024-03-23T23:04:11.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.665+0000 7f252aa00640 1 -- 172.21.15.12:0/2556128883 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2530043070 con 0x5589526e9050 2024-03-23T23:04:11.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.665+0000 7f252aa00640 1 -- 172.21.15.12:0/2556128883 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f253002cca0 con 0x5589526e9050 2024-03-23T23:04:11.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.665+0000 7f252aa00640 1 -- 172.21.15.12:0/2556128883 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2530039040 con 0x5589526e9050 2024-03-23T23:04:11.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.665+0000 7f253d800640 1 --2- 172.21.15.12:0/588713173 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f25200a2800 0x7f25200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.666+0000 7f2542f6d740 1 -- 172.21.15.12:0/2556128883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589526e9050 msgr2=0x5589526e9470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:11.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.666+0000 7f2542f6d740 1 --2- 172.21.15.12:0/2556128883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589526e9050 0x5589526e9470 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f253002c700 tx=0x7f25300323d0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.666+0000 7f253d800640 1 --2- 172.21.15.12:0/588713173 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f25200a2800 0x7f25200a4cc0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x5589526e9e40 tx=0x7f2524005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:11.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.666+0000 7f2542f6d740 1 -- 172.21.15.12:0/2556128883 shutdown_connections 2024-03-23T23:04:11.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.666+0000 7f2542f6d740 1 --2- 172.21.15.12:0/2556128883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589526eebb0 0x5589526f1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.666+0000 7f2542f6d740 1 --2- 172.21.15.12:0/2556128883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589526e9a40 0x5589526ee540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.666+0000 7f2542f6d740 1 --2- 172.21.15.12:0/2556128883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589526e9050 0x5589526e9470 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.666+0000 7f2542f6d740 1 -- 172.21.15.12:0/2556128883 >> 172.21.15.12:0/2556128883 conn(0x5589526b1200 msgr2=0x5589526e49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:11.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.666+0000 7f2542f6d740 1 -- 172.21.15.12:0/2556128883 shutdown_connections 2024-03-23T23:04:11.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.666+0000 7f2542f6d740 1 -- 172.21.15.12:0/2556128883 wait complete. 2024-03-23T23:04:11.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.666+0000 7f2542f6d740 1 -- start start 2024-03-23T23:04:11.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.667+0000 7f2542f6d740 1 Processor -- start 2024-03-23T23:04:11.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.667+0000 7f2542f6d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589526e9050 0x5589526f3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.667+0000 7f2542f6d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589526e9a40 0x5589526f35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.667+0000 7f2542f6d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589526eebb0 0x5589526fa600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.667+0000 7f2542f6d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589526f1b00 con 0x5589526e9050 2024-03-23T23:04:11.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.667+0000 7f2542f6d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x558952609e80 con 0x5589526eebb0 2024-03-23T23:04:11.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.667+0000 7f2542f6d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589526f1c80 con 0x5589526e9a40 2024-03-23T23:04:11.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.667+0000 7f2542f6d740 10 client.0 _ll_register_callbacks cb 0x55895261c130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:11.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.667+0000 7f253e200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589526eebb0 0x5589526fa600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.667+0000 7f253ce00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589526e9a40 0x5589526f35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.667+0000 7f253e200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589526eebb0 0x5589526fa600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37952/0 (socket says 172.21.15.12:37952) 2024-03-23T23:04:11.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.667+0000 7f253e200640 1 -- 172.21.15.12:0/2724642428 learned_addr learned my addr 172.21.15.12:0/2724642428 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:11.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.667+0000 7f253d800640 1 --2- 172.21.15.12:0/2724642428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589526e9050 0x5589526f3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.668+0000 7f253e200640 1 -- 172.21.15.12:0/2724642428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589526e9a40 msgr2=0x5589526f35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.668+0000 7f253e200640 1 --2- 172.21.15.12:0/2724642428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589526e9a40 0x5589526f35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.668+0000 7f253e200640 1 -- 172.21.15.12:0/2724642428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589526e9050 msgr2=0x5589526f3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.668+0000 7f253e200640 1 --2- 172.21.15.12:0/2724642428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589526e9050 0x5589526f3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.668+0000 7f253e200640 1 -- 172.21.15.12:0/2724642428 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55895242c7d0 con 0x5589526eebb0 2024-03-23T23:04:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.668+0000 7f253ce00640 1 --2- 172.21.15.12:0/2724642428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589526e9a40 0x5589526f35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.668+0000 7f253e200640 1 --2- 172.21.15.12:0/2724642428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589526eebb0 0x5589526fa600 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2530009bf0 tx=0x7f2530003a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32e907a77647a470 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.668+0000 7f251f400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:04:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.668+0000 7f251f400640 1 -- 172.21.15.12:0/2724642428 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253004c030 con 0x5589526eebb0 2024-03-23T23:04:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.668+0000 7f2542f6d740 1 -- 172.21.15.12:0/2724642428 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f253002cca0 con 0x5589526eebb0 2024-03-23T23:04:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.668+0000 7f251f400640 1 -- 172.21.15.12:0/2724642428 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2530032960 con 0x5589526eebb0 2024-03-23T23:04:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.669+0000 7f251f400640 1 -- 172.21.15.12:0/2724642428 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2530039070 con 0x5589526eebb0 2024-03-23T23:04:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.669+0000 7f2542f6d740 10 client.25949 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.669+0000 7f2542f6d740 1 -- 172.21.15.12:0/2724642428 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5589523623f0 con 0x5589526eebb0 2024-03-23T23:04:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.669+0000 7f251f400640 1 -- 172.21.15.12:0/2724642428 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f253002cca0 con 0x5589526eebb0 2024-03-23T23:04:11.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.669+0000 7f251f400640 1 -- 172.21.15.12:0/2724642428 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f253003f560 con 0x5589526eebb0 2024-03-23T23:04:11.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.669+0000 7f2542f6d740 10 client.25949 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:11.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.669+0000 7f2542f6d740 10 client.25949 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:04:11.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.669+0000 7f2542f6d740 20 client.25949 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:11.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.669+0000 7f2542f6d740 1 --2- 172.21.15.12:0/2724642428 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5589526feaf0 0x558952700ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.669+0000 7f2542f6d740 4 client.25949 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:04:11.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.669+0000 7f2542f6d740 1 -- 172.21.15.12:0/2724642428 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55895265b790 con 0x5589526feaf0 2024-03-23T23:04:11.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.669+0000 7f253d800640 1 --2- 172.21.15.12:0/2724642428 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5589526feaf0 0x558952700ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.670+0000 7f253d800640 1 --2- 172.21.15.12:0/2724642428 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5589526feaf0 0x558952700ee0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d85faf08023d1ebf server_cookie=e580768813cd98dc in_seq=0 out_seq=0 2024-03-23T23:04:11.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.670+0000 7f251f400640 10 client.25949 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:11.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.671+0000 7f251f400640 1 -- 172.21.15.12:0/2724642428 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55895265b790 con 0x5589526feaf0 2024-03-23T23:04:11.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.671+0000 7f251f400640 10 client.25949 handle_command_reply: tid=0 2024-03-23T23:04:11.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.671+0000 7f2542f6d740 1 client.25949 shutdown 2024-03-23T23:04:11.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.671+0000 7f2542f6d740 1 -- 172.21.15.12:0/2724642428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589526eebb0 msgr2=0x5589526fa600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:11.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.671+0000 7f2542f6d740 1 --2- 172.21.15.12:0/2724642428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589526eebb0 0x5589526fa600 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2530009bf0 tx=0x7f2530003a00 comp rx=0 tx=0).stop 2024-03-23T23:04:11.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.671+0000 7f2542f6d740 1 -- 172.21.15.12:0/2724642428 shutdown_connections 2024-03-23T23:04:11.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.671+0000 7f2542f6d740 1 --2- 172.21.15.12:0/2724642428 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5589526feaf0 0x558952700ee0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.671+0000 7f2542f6d740 1 --2- 172.21.15.12:0/2724642428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589526eebb0 0x5589526fa600 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.671+0000 7f2542f6d740 1 --2- 172.21.15.12:0/2724642428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589526e9a40 0x5589526f35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.671+0000 7f2542f6d740 1 --2- 172.21.15.12:0/2724642428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589526e9050 0x5589526f3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.671+0000 7f2542f6d740 1 -- 172.21.15.12:0/2724642428 >> 172.21.15.12:0/2724642428 conn(0x5589526b1200 msgr2=0x5589526e3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:11.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.671+0000 7f251f400640 0 client.25949 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:11.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.671+0000 7f2542f6d740 1 -- 172.21.15.12:0/2724642428 shutdown_connections 2024-03-23T23:04:11.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.671+0000 7f2542f6d740 1 -- 172.21.15.12:0/2724642428 wait complete. 2024-03-23T23:04:11.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.671+0000 7f2542f6d740 20 client.25949 trim_cache size 0 max 16384 2024-03-23T23:04:11.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.689+0000 7f2542f6d740 1 Processor -- start 2024-03-23T23:04:11.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.689+0000 7f2542f6d740 1 -- start start 2024-03-23T23:04:11.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.689+0000 7f2542f6d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589526e9050 0x5589526e7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.689+0000 7f2542f6d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589526e9a40 0x5589526fa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.689+0000 7f2542f6d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589526eebb0 0x5589526e8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.689+0000 7f2542f6d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25381aced0 con 0x5589526eebb0 2024-03-23T23:04:11.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.689+0000 7f2542f6d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25381aa9d0 con 0x5589526e9a40 2024-03-23T23:04:11.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.689+0000 7f2542f6d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25381a8a70 con 0x5589526e9050 2024-03-23T23:04:11.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.689+0000 7f253d800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589526e9050 0x5589526e7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.689+0000 7f253d800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589526e9050 0x5589526e7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:51980/0 (socket says 172.21.15.12:51980) 2024-03-23T23:04:11.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.689+0000 7f253d800640 1 -- 172.21.15.12:0/2272209301 learned_addr learned my addr 172.21.15.12:0/2272209301 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:11.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.689+0000 7f253ce00640 1 --2- 172.21.15.12:0/2272209301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589526e9a40 0x5589526fa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.689+0000 7f253e200640 1 --2- 172.21.15.12:0/2272209301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589526eebb0 0x5589526e8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.690+0000 7f253d800640 1 -- 172.21.15.12:0/2272209301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589526e9a40 msgr2=0x5589526fa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:11.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.690+0000 7f253d800640 1 --2- 172.21.15.12:0/2272209301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589526e9a40 0x5589526fa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.690+0000 7f253d800640 1 -- 172.21.15.12:0/2272209301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589526eebb0 msgr2=0x5589526e8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:11.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.690+0000 7f253d800640 1 --2- 172.21.15.12:0/2272209301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589526eebb0 0x5589526e8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.690+0000 7f253d800640 1 -- 172.21.15.12:0/2272209301 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5589523220f0 con 0x5589526e9050 2024-03-23T23:04:11.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.690+0000 7f253ce00640 1 --2- 172.21.15.12:0/2272209301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589526e9a40 0x5589526fa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:04:11.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.690+0000 7f253d800640 1 --2- 172.21.15.12:0/2272209301 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589526e9050 0x5589526e7c00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x5589526ea4b0 tx=0x7f252401b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb8058bce08631bf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:11.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.690+0000 7f252a000640 1 -- 172.21.15.12:0/2272209301 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f252402b680 con 0x5589526e9050 2024-03-23T23:04:11.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.690+0000 7f252a000640 1 -- 172.21.15.12:0/2272209301 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2524030070 con 0x5589526e9050 2024-03-23T23:04:11.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.690+0000 7f252a000640 1 -- 172.21.15.12:0/2272209301 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f252400fc40 con 0x5589526e9050 2024-03-23T23:04:11.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.691+0000 7f2542f6d740 1 -- 172.21.15.12:0/2272209301 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589526e9050 msgr2=0x5589526e7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:11.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.691+0000 7f2542f6d740 1 --2- 172.21.15.12:0/2272209301 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589526e9050 0x5589526e7c00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x5589526ea4b0 tx=0x7f252401b000 comp rx=0 tx=0).stop 2024-03-23T23:04:11.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.691+0000 7f2542f6d740 1 -- 172.21.15.12:0/2272209301 shutdown_connections 2024-03-23T23:04:11.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.691+0000 7f2542f6d740 1 --2- 172.21.15.12:0/2272209301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589526eebb0 0x5589526e8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.691+0000 7f2542f6d740 1 --2- 172.21.15.12:0/2272209301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589526e9a40 0x5589526fa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.691+0000 7f2542f6d740 1 --2- 172.21.15.12:0/2272209301 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589526e9050 0x5589526e7c00 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.691+0000 7f2542f6d740 1 -- 172.21.15.12:0/2272209301 >> 172.21.15.12:0/2272209301 conn(0x5589526b1200 msgr2=0x5589526ead70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:11.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.691+0000 7f2542f6d740 1 -- 172.21.15.12:0/2272209301 shutdown_connections 2024-03-23T23:04:11.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.691+0000 7f2542f6d740 1 -- 172.21.15.12:0/2272209301 wait complete. 2024-03-23T23:04:11.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.691+0000 7f2542f6d740 1 -- start start 2024-03-23T23:04:11.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.692+0000 7f2542f6d740 1 Processor -- start 2024-03-23T23:04:11.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.692+0000 7f2542f6d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589526e9050 0x5589526f7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.692+0000 7f2542f6d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589526e9a40 0x5589526f7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.692+0000 7f2542f6d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589526eebb0 0x5589526f8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.692+0000 7f2542f6d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25381a9070 con 0x5589526e9a40 2024-03-23T23:04:11.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.692+0000 7f2542f6d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25381aa370 con 0x5589526eebb0 2024-03-23T23:04:11.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.692+0000 7f2542f6d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25381a96b0 con 0x5589526e9050 2024-03-23T23:04:11.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.692+0000 7f2542f6d740 10 client.0 _ll_register_callbacks cb 0x7f2524009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:11.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.692+0000 7f253ce00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589526e9a40 0x5589526f7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.692+0000 7f253d800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589526e9050 0x5589526f7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.692+0000 7f253ce00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589526e9a40 0x5589526f7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54048/0 (socket says 172.21.15.12:54048) 2024-03-23T23:04:11.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.692+0000 7f253ce00640 1 -- 172.21.15.12:0/903238099 learned_addr learned my addr 172.21.15.12:0/903238099 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:11.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.692+0000 7f253e200640 1 --2- 172.21.15.12:0/903238099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589526eebb0 0x5589526f8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.693+0000 7f253ce00640 1 -- 172.21.15.12:0/903238099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589526e9050 msgr2=0x5589526f7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.693+0000 7f253ce00640 1 --2- 172.21.15.12:0/903238099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589526e9050 0x5589526f7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.693+0000 7f253ce00640 1 -- 172.21.15.12:0/903238099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589526eebb0 msgr2=0x5589526f8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.693+0000 7f253ce00640 1 --2- 172.21.15.12:0/903238099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589526eebb0 0x5589526f8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.693+0000 7f253ce00640 1 -- 172.21.15.12:0/903238099 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5589526f9580 con 0x5589526e9a40 2024-03-23T23:04:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.693+0000 7f253ce00640 1 --2- 172.21.15.12:0/903238099 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589526e9a40 0x5589526f7f80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f252c0cc850 tx=0x7f252c097aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d252b17b1f710a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.693+0000 7f251f400640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:04:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.693+0000 7f251f400640 1 -- 172.21.15.12:0/903238099 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f252c09b050 con 0x5589526e9a40 2024-03-23T23:04:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.693+0000 7f251f400640 1 -- 172.21.15.12:0/903238099 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f252c0cc960 con 0x5589526e9a40 2024-03-23T23:04:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.693+0000 7f2542f6d740 1 -- 172.21.15.12:0/903238099 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2524030070 con 0x5589526e9a40 2024-03-23T23:04:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.694+0000 7f251f400640 1 -- 172.21.15.12:0/903238099 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f252c09bdf0 con 0x5589526e9a40 2024-03-23T23:04:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.694+0000 7f2542f6d740 10 client.15822 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.694+0000 7f2542f6d740 1 -- 172.21.15.12:0/903238099 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5589526fa960 con 0x5589526e9a40 2024-03-23T23:04:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.694+0000 7f251f400640 1 -- 172.21.15.12:0/903238099 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2524030070 con 0x5589526e9a40 2024-03-23T23:04:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.694+0000 7f251f400640 1 -- 172.21.15.12:0/903238099 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f252c0a3a00 con 0x5589526e9a40 2024-03-23T23:04:11.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.694+0000 7f2542f6d740 10 client.15822 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:11.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.694+0000 7f2542f6d740 10 client.15822 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:04:11.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.694+0000 7f2542f6d740 20 client.15822 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:11.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.694+0000 7f2542f6d740 1 --2- 172.21.15.12:0/903238099 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55895275a990 0x55895275cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:11.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.694+0000 7f2542f6d740 4 client.15822 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:04:11.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.694+0000 7f2542f6d740 1 -- 172.21.15.12:0/903238099 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558952655ae0 con 0x55895275a990 2024-03-23T23:04:11.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.694+0000 7f253d800640 1 --2- 172.21.15.12:0/903238099 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55895275a990 0x55895275cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:11.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.695+0000 7f253d800640 1 --2- 172.21.15.12:0/903238099 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55895275a990 0x55895275cdb0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=997b843e12625ce6 server_cookie=92b601108b2a224d in_seq=0 out_seq=0 2024-03-23T23:04:11.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.695+0000 7f251f400640 10 client.15822 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:11.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.695+0000 7f251f400640 1 -- 172.21.15.12:0/903238099 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x558952655ae0 con 0x55895275a990 2024-03-23T23:04:11.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.695+0000 7f251f400640 10 client.15822 handle_command_reply: tid=0 2024-03-23T23:04:11.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.695+0000 7f2542f6d740 1 client.15822 shutdown 2024-03-23T23:04:11.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.695+0000 7f2542f6d740 1 -- 172.21.15.12:0/903238099 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589526e9a40 msgr2=0x5589526f7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:11.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.695+0000 7f2542f6d740 1 --2- 172.21.15.12:0/903238099 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589526e9a40 0x5589526f7f80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f252c0cc850 tx=0x7f252c097aa0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.695+0000 7f2542f6d740 1 -- 172.21.15.12:0/903238099 shutdown_connections 2024-03-23T23:04:11.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.695+0000 7f2542f6d740 1 --2- 172.21.15.12:0/903238099 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55895275a990 0x55895275cdb0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.695+0000 7f2542f6d740 1 --2- 172.21.15.12:0/903238099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589526eebb0 0x5589526f8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.695+0000 7f2542f6d740 1 --2- 172.21.15.12:0/903238099 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589526e9a40 0x5589526f7f80 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.695+0000 7f2542f6d740 1 --2- 172.21.15.12:0/903238099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589526e9050 0x5589526f7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.695+0000 7f251f400640 0 client.15822 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:11.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.695+0000 7f2542f6d740 1 -- 172.21.15.12:0/903238099 >> 172.21.15.12:0/903238099 conn(0x5589526b1200 msgr2=0x5589526e3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:11.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.696+0000 7f2542f6d740 1 -- 172.21.15.12:0/903238099 shutdown_connections 2024-03-23T23:04:11.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.696+0000 7f2542f6d740 1 -- 172.21.15.12:0/903238099 wait complete. 2024-03-23T23:04:11.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.696+0000 7f2542f6d740 20 client.15822 trim_cache size 0 max 16384 2024-03-23T23:04:11.698 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:04:11.699 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T23:04:11.699 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-03-23T23:04:11.699 INFO:teuthology.orchestra.run.smithi012.stdout: "22685496-b5b7-441a-9e68-8806a05a51b9": { 2024-03-23T23:04:11.699 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-03-23T23:04:11.699 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "22685496-b5b7-441a-9e68-8806a05a51b9", 2024-03-23T23:04:11.699 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-03-23T23:04:11.699 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:04:11.699 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:04:11.699 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:04:11.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.696+0000 7f253ec00640 1 -- 172.21.15.12:0/588713173 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f25200a2800 msgr2=0x7f25200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:11.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.696+0000 7f253ec00640 1 --2- 172.21.15.12:0/588713173 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f25200a2800 0x7f25200a4cc0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x5589526e9e40 tx=0x7f2524005f90 comp rx=0 tx=0).stop 2024-03-23T23:04:11.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.696+0000 7f253ec00640 1 -- 172.21.15.12:0/588713173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2538102770 msgr2=0x7f2538199f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:11.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.696+0000 7f253ec00640 1 --2- 172.21.15.12:0/588713173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2538102770 0x7f2538199f20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f252c00ea60 tx=0x7f252c00ef20 comp rx=0 tx=0).stop 2024-03-23T23:04:11.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.697+0000 7f253ec00640 1 -- 172.21.15.12:0/588713173 shutdown_connections 2024-03-23T23:04:11.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.697+0000 7f253ec00640 1 --2- 172.21.15.12:0/588713173 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f25200a2800 0x7f25200a4cc0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.697+0000 7f253ec00640 1 --2- 172.21.15.12:0/588713173 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2538103130 0x7f25381a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.697+0000 7f253ec00640 1 --2- 172.21.15.12:0/588713173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2538102770 0x7f2538199f20 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.697+0000 7f253ec00640 1 --2- 172.21.15.12:0/588713173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2538101580 0x7f25381999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:11.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.697+0000 7f253ec00640 1 -- 172.21.15.12:0/588713173 >> 172.21.15.12:0/588713173 conn(0x7f25380fcb10 msgr2=0x7f25380738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:11.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.697+0000 7f253ec00640 1 -- 172.21.15.12:0/588713173 shutdown_connections 2024-03-23T23:04:11.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:11.697+0000 7f253ec00640 1 -- 172.21.15.12:0/588713173 wait complete. 2024-03-23T23:04:11.715 INFO:tasks.cephfs.filesystem:scrub status for tag:22685496-b5b7-441a-9e68-8806a05a51b9 - {'path': '/', 'tag': '22685496-b5b7-441a-9e68-8806a05a51b9', 'options': 'recursive,force'} 2024-03-23T23:04:12.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:12 smithi203 ceph-mon[36693]: pgmap v462: 97 pgs: 97 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-23T23:04:12.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:12 smithi203 ceph-mon[36693]: scrub queued for path: / 2024-03-23T23:04:12.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:12 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:04:12.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:12 smithi203 ceph-mon[36693]: scrub summary: active paths [/] 2024-03-23T23:04:12.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:12 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:04:13.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:12 smithi012 ceph-mon[31695]: pgmap v462: 97 pgs: 97 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-23T23:04:13.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:12 smithi012 ceph-mon[31695]: scrub queued for path: / 2024-03-23T23:04:13.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:12 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:04:13.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:12 smithi012 ceph-mon[31695]: scrub summary: active paths [/] 2024-03-23T23:04:13.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:12 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:04:13.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:12 smithi173 ceph-mon[36116]: pgmap v462: 97 pgs: 97 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-23T23:04:13.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:12 smithi173 ceph-mon[36116]: scrub queued for path: / 2024-03-23T23:04:13.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:12 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:04:13.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:12 smithi173 ceph-mon[36116]: scrub summary: active paths [/] 2024-03-23T23:04:13.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:12 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:04:14.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:14 smithi203 ceph-mon[36693]: pgmap v463: 97 pgs: 97 active+clean; 1.2 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:04:15.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:14 smithi012 ceph-mon[31695]: pgmap v463: 97 pgs: 97 active+clean; 1.2 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:04:15.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:14 smithi173 ceph-mon[36116]: pgmap v463: 97 pgs: 97 active+clean; 1.2 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:04:16.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:16 smithi203 ceph-mon[36693]: pgmap v464: 97 pgs: 97 active+clean; 1.2 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:04:16.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:16 smithi203 ceph-mon[36693]: scrub summary: idle 2024-03-23T23:04:17.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:16 smithi012 ceph-mon[31695]: pgmap v464: 97 pgs: 97 active+clean; 1.2 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:04:17.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:16 smithi012 ceph-mon[31695]: scrub summary: idle 2024-03-23T23:04:17.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:16 smithi173 ceph-mon[36116]: pgmap v464: 97 pgs: 97 active+clean; 1.2 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:04:17.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:16 smithi173 ceph-mon[36116]: scrub summary: idle 2024-03-23T23:04:18.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:18 smithi203 ceph-mon[36693]: pgmap v465: 97 pgs: 97 active+clean; 1.2 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:04:19.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:18 smithi012 ceph-mon[31695]: pgmap v465: 97 pgs: 97 active+clean; 1.2 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:04:19.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:18 smithi173 ceph-mon[36116]: pgmap v465: 97 pgs: 97 active+clean; 1.2 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:04:20.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:20 smithi203 ceph-mon[36693]: pgmap v466: 97 pgs: 97 active+clean; 1.2 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:04:21.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:20 smithi012 ceph-mon[31695]: pgmap v466: 97 pgs: 97 active+clean; 1.2 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:04:21.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:20 smithi173 ceph-mon[36116]: pgmap v466: 97 pgs: 97 active+clean; 1.2 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:04:23.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:22 smithi012 ceph-mon[31695]: pgmap v467: 97 pgs: 97 active+clean; 1.2 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-03-23T23:04:23.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:22 smithi173 ceph-mon[36116]: pgmap v467: 97 pgs: 97 active+clean; 1.2 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-03-23T23:04:23.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:22 smithi203 ceph-mon[36693]: pgmap v467: 97 pgs: 97 active+clean; 1.2 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-03-23T23:04:23.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.576+0000 7f0be3200640 1 -- 172.21.15.12:0/3695006976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0bdc0ffd80 msgr2=0x7f0bdc1021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:23.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.576+0000 7f0be3200640 1 --2- 172.21.15.12:0/3695006976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0bdc0ffd80 0x7f0bdc1021b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0009900 tx=0x7f0bd002bc30 comp rx=0 tx=0).stop 2024-03-23T23:04:23.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.576+0000 7f0be3200640 1 -- 172.21.15.12:0/3695006976 shutdown_connections 2024-03-23T23:04:23.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.576+0000 7f0be3200640 1 --2- 172.21.15.12:0/3695006976 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bdc1050a0 0x7f0bdc105520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.576+0000 7f0be3200640 1 --2- 172.21.15.12:0/3695006976 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bdc1026f0 0x7f0bdc104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.576+0000 7f0be3200640 1 --2- 172.21.15.12:0/3695006976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0bdc0ffd80 0x7f0bdc1021b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.576+0000 7f0be3200640 1 -- 172.21.15.12:0/3695006976 >> 172.21.15.12:0/3695006976 conn(0x7f0bdc0fb9a0 msgr2=0x7f0bdc0fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:23.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.576+0000 7f0be3200640 1 -- 172.21.15.12:0/3695006976 shutdown_connections 2024-03-23T23:04:23.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.576+0000 7f0be3200640 1 -- 172.21.15.12:0/3695006976 wait complete. 2024-03-23T23:04:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.577+0000 7f0be3200640 1 Processor -- start 2024-03-23T23:04:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.578+0000 7f0be3200640 1 -- start start 2024-03-23T23:04:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.578+0000 7f0be3200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0bdc0ffd80 0x7f0bdc19de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.578+0000 7f0be3200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bdc1026f0 0x7f0bdc19e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.578+0000 7f0be3200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bdc1050a0 0x7f0bdc1a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.578+0000 7f0be3200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc1137d0 con 0x7f0bdc1026f0 2024-03-23T23:04:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.578+0000 7f0be3200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc113650 con 0x7f0bdc1050a0 2024-03-23T23:04:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.578+0000 7f0be3200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc113950 con 0x7f0bdc0ffd80 2024-03-23T23:04:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.578+0000 7f0be1e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0bdc0ffd80 0x7f0bdc19de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.578+0000 7f0be1e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0bdc0ffd80 0x7f0bdc19de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:48802/0 (socket says 172.21.15.12:48802) 2024-03-23T23:04:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.578+0000 7f0be1e00640 1 -- 172.21.15.12:0/2891714394 learned_addr learned my addr 172.21.15.12:0/2891714394 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.578+0000 7f0be1400640 1 --2- 172.21.15.12:0/2891714394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bdc1026f0 0x7f0bdc19e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.578+0000 7f0be2800640 1 --2- 172.21.15.12:0/2891714394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bdc1050a0 0x7f0bdc1a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.579+0000 7f0be1e00640 1 -- 172.21.15.12:0/2891714394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bdc1050a0 msgr2=0x7f0bdc1a53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.579+0000 7f0be1e00640 1 --2- 172.21.15.12:0/2891714394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bdc1050a0 0x7f0bdc1a53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.579+0000 7f0be1e00640 1 -- 172.21.15.12:0/2891714394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bdc1026f0 msgr2=0x7f0bdc19e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.579+0000 7f0be1e00640 1 --2- 172.21.15.12:0/2891714394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bdc1026f0 0x7f0bdc19e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.579+0000 7f0be1e00640 1 -- 172.21.15.12:0/2891714394 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bdc1a5a30 con 0x7f0bdc0ffd80 2024-03-23T23:04:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.579+0000 7f0be1e00640 1 --2- 172.21.15.12:0/2891714394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0bdc0ffd80 0x7f0bdc19de10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0000c00 tx=0x7f0bd002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.579+0000 7f0bd6a00640 1 -- 172.21.15.12:0/2891714394 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd0043020 con 0x7f0bdc0ffd80 2024-03-23T23:04:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.579+0000 7f0bd6a00640 1 -- 172.21.15.12:0/2891714394 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0bd00054e0 con 0x7f0bdc0ffd80 2024-03-23T23:04:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.579+0000 7f0bd6a00640 1 -- 172.21.15.12:0/2891714394 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd0003c40 con 0x7f0bdc0ffd80 2024-03-23T23:04:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.579+0000 7f0be3200640 1 -- 172.21.15.12:0/2891714394 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bdc1a5cc0 con 0x7f0bdc0ffd80 2024-03-23T23:04:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.580+0000 7f0be3200640 1 -- 172.21.15.12:0/2891714394 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bdc1a61c0 con 0x7f0bdc0ffd80 2024-03-23T23:04:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.583+0000 7f0bd6a00640 1 -- 172.21.15.12:0/2891714394 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0bd0046020 con 0x7f0bdc0ffd80 2024-03-23T23:04:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.583+0000 7f0be7422740 1 Processor -- start 2024-03-23T23:04:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.583+0000 7f0be7422740 1 -- start start 2024-03-23T23:04:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.583+0000 7f0be7422740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f1623f0 0x55711f162810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.583+0000 7f0be7422740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55711f162de0 0x55711f1678e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.584+0000 7f0be7422740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f167f50 0x55711f16a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.584+0000 7f0be7422740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55711f1253b0 con 0x55711f162de0 2024-03-23T23:04:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.584+0000 7f0be7422740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55711f0c7830 con 0x55711f167f50 2024-03-23T23:04:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.584+0000 7f0be7422740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55711f0705d0 con 0x55711f1623f0 2024-03-23T23:04:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.584+0000 7f0be2800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f1623f0 0x55711f162810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.584+0000 7f0be1e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55711f162de0 0x55711f1678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.584+0000 7f0be1400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f167f50 0x55711f16a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.584+0000 7f0be2800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f1623f0 0x55711f162810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:48812/0 (socket says 172.21.15.12:48812) 2024-03-23T23:04:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.584+0000 7f0be2800640 1 -- 172.21.15.12:0/3474770377 learned_addr learned my addr 172.21.15.12:0/3474770377 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.584+0000 7f0be2800640 1 -- 172.21.15.12:0/3474770377 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f167f50 msgr2=0x55711f16a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.584+0000 7f0be2800640 1 --2- 172.21.15.12:0/3474770377 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f167f50 0x55711f16a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.584+0000 7f0bd6a00640 1 --2- 172.21.15.12:0/2891714394 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0bc40a27f0 0x7f0bc40a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.584+0000 7f0be2800640 1 -- 172.21.15.12:0/3474770377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55711f162de0 msgr2=0x55711f1678e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.584+0000 7f0be2800640 1 --2- 172.21.15.12:0/3474770377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55711f162de0 0x55711f1678e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.584+0000 7f0be2800640 1 -- 172.21.15.12:0/3474770377 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55711f0882c0 con 0x55711f1623f0 2024-03-23T23:04:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.584+0000 7f0be1400640 1 --2- 172.21.15.12:0/3474770377 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f167f50 0x55711f16a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.585+0000 7f0bd6a00640 1 -- 172.21.15.12:0/2891714394 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0bd00f0af0 con 0x7f0bdc0ffd80 2024-03-23T23:04:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.585+0000 7f0be2800640 1 --2- 172.21.15.12:0/2891714394 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0bc40a27f0 0x7f0bc40a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.585+0000 7f0be2800640 1 --2- 172.21.15.12:0/3474770377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f1623f0 0x55711f162810 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8009fd0 tx=0x7f0bd80118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee1c324241e21374 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:23.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.585+0000 7f0bcb000640 1 -- 172.21.15.12:0/3474770377 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd8004250 con 0x55711f1623f0 2024-03-23T23:04:23.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.585+0000 7f0bcb000640 1 -- 172.21.15.12:0/3474770377 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0bd800f4e0 con 0x55711f1623f0 2024-03-23T23:04:23.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.585+0000 7f0bcb000640 1 -- 172.21.15.12:0/3474770377 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd801b690 con 0x55711f1623f0 2024-03-23T23:04:23.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.586+0000 7f0be7422740 1 -- 172.21.15.12:0/3474770377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f1623f0 msgr2=0x55711f162810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:23.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.586+0000 7f0be7422740 1 --2- 172.21.15.12:0/3474770377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f1623f0 0x55711f162810 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8009fd0 tx=0x7f0bd80118d0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.586+0000 7f0be2800640 1 --2- 172.21.15.12:0/2891714394 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0bc40a27f0 0x7f0bc40a4cb0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8020000 tx=0x7f0bd801bd90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:23.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.586+0000 7f0be7422740 1 -- 172.21.15.12:0/3474770377 shutdown_connections 2024-03-23T23:04:23.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.586+0000 7f0be7422740 1 --2- 172.21.15.12:0/3474770377 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f167f50 0x55711f16a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.586+0000 7f0be7422740 1 --2- 172.21.15.12:0/3474770377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55711f162de0 0x55711f1678e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.586+0000 7f0be7422740 1 --2- 172.21.15.12:0/3474770377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f1623f0 0x55711f162810 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.586+0000 7f0be7422740 1 -- 172.21.15.12:0/3474770377 >> 172.21.15.12:0/3474770377 conn(0x55711f15d960 msgr2=0x55711f15dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:23.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.586+0000 7f0be7422740 1 -- 172.21.15.12:0/3474770377 shutdown_connections 2024-03-23T23:04:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.586+0000 7f0be7422740 1 -- 172.21.15.12:0/3474770377 wait complete. 2024-03-23T23:04:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.587+0000 7f0be7422740 1 -- start start 2024-03-23T23:04:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.587+0000 7f0be7422740 1 Processor -- start 2024-03-23T23:04:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.587+0000 7f0be7422740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f1623f0 0x55711f16c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.587+0000 7f0be7422740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55711f162de0 0x55711f16cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.587+0000 7f0be7422740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f167f50 0x55711f173b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.587+0000 7f0be7422740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55711f083a20 con 0x55711f162de0 2024-03-23T23:04:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.587+0000 7f0be7422740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55711f0c38d0 con 0x55711f167f50 2024-03-23T23:04:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.587+0000 7f0be7422740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55711f16af00 con 0x55711f1623f0 2024-03-23T23:04:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.587+0000 7f0be7422740 10 client.0 _ll_register_callbacks cb 0x55711f0c01c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.588+0000 7f0be1400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55711f162de0 0x55711f16cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.588+0000 7f0be2800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f167f50 0x55711f173b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:23.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.588+0000 7f0be1e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f1623f0 0x55711f16c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:23.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.588+0000 7f0be2800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f167f50 0x55711f173b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59724/0 (socket says 172.21.15.12:59724) 2024-03-23T23:04:23.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.588+0000 7f0be2800640 1 -- 172.21.15.12:0/868492179 learned_addr learned my addr 172.21.15.12:0/868492179 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:23.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.588+0000 7f0be2800640 1 -- 172.21.15.12:0/868492179 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f1623f0 msgr2=0x55711f16c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:23.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.588+0000 7f0be2800640 1 --2- 172.21.15.12:0/868492179 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f1623f0 0x55711f16c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.588+0000 7f0be2800640 1 -- 172.21.15.12:0/868492179 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55711f162de0 msgr2=0x55711f16cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:23.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.588+0000 7f0be2800640 1 --2- 172.21.15.12:0/868492179 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55711f162de0 0x55711f16cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.588+0000 7f0be2800640 1 -- 172.21.15.12:0/868492179 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55711eddc210 con 0x55711f167f50 2024-03-23T23:04:23.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.588+0000 7f0be1e00640 1 --2- 172.21.15.12:0/868492179 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f1623f0 0x55711f16c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:23.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.588+0000 7f0be2800640 1 --2- 172.21.15.12:0/868492179 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f167f50 0x55711f173b40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0bd801b660 tx=0x7f0bd8017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fca816230bee246a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:23.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.588+0000 7f0bbf400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:04:23.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.589+0000 7f0bbf400640 1 -- 172.21.15.12:0/868492179 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd800f9e0 con 0x55711f167f50 2024-03-23T23:04:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.589+0000 7f0bbf400640 1 -- 172.21.15.12:0/868492179 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0bd8019950 con 0x55711f167f50 2024-03-23T23:04:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.589+0000 7f0be7422740 1 -- 172.21.15.12:0/868492179 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0bd800f4e0 con 0x55711f167f50 2024-03-23T23:04:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.589+0000 7f0bbf400640 1 -- 172.21.15.12:0/868492179 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0bd8019b50 con 0x55711f167f50 2024-03-23T23:04:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.589+0000 7f0be7422740 10 client.25967 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.589+0000 7f0be7422740 1 -- 172.21.15.12:0/868492179 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55711eea8620 con 0x55711f167f50 2024-03-23T23:04:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.589+0000 7f0bbf400640 1 -- 172.21.15.12:0/868492179 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0bd800f4e0 con 0x55711f167f50 2024-03-23T23:04:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.590+0000 7f0bbf400640 1 -- 172.21.15.12:0/868492179 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f0bd80045a0 con 0x55711f167f50 2024-03-23T23:04:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.590+0000 7f0be7422740 10 client.25967 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.590+0000 7f0be7422740 10 client.25967 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:04:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.590+0000 7f0be7422740 20 client.25967 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.590+0000 7f0be7422740 1 --2- 172.21.15.12:0/868492179 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55711f178340 0x55711f17a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.590+0000 7f0be7422740 4 client.25967 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:04:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.590+0000 7f0be7422740 1 -- 172.21.15.12:0/868492179 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55711f0d4d70 con 0x55711f178340 2024-03-23T23:04:23.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.590+0000 7f0be1e00640 1 --2- 172.21.15.12:0/868492179 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55711f178340 0x55711f17a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:23.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.590+0000 7f0be1e00640 1 --2- 172.21.15.12:0/868492179 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55711f178340 0x55711f17a780 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e27e3c5e91658947 server_cookie=59276113ec10add4 in_seq=0 out_seq=0 2024-03-23T23:04:23.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.590+0000 7f0bbf400640 10 client.25967 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.591+0000 7f0bbf400640 1 -- 172.21.15.12:0/868492179 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55711f0d4d70 con 0x55711f178340 2024-03-23T23:04:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.591+0000 7f0bbf400640 10 client.25967 handle_command_reply: tid=0 2024-03-23T23:04:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.591+0000 7f0be7422740 1 client.25967 shutdown 2024-03-23T23:04:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.591+0000 7f0be7422740 1 -- 172.21.15.12:0/868492179 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f167f50 msgr2=0x55711f173b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.591+0000 7f0be7422740 1 --2- 172.21.15.12:0/868492179 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f167f50 0x55711f173b40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0bd801b660 tx=0x7f0bd8017000 comp rx=0 tx=0).stop 2024-03-23T23:04:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.592+0000 7f0be7422740 1 -- 172.21.15.12:0/868492179 shutdown_connections 2024-03-23T23:04:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.592+0000 7f0be7422740 1 --2- 172.21.15.12:0/868492179 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55711f178340 0x55711f17a780 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.592+0000 7f0be7422740 1 --2- 172.21.15.12:0/868492179 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f167f50 0x55711f173b40 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.592+0000 7f0be7422740 1 --2- 172.21.15.12:0/868492179 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55711f162de0 0x55711f16cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.592+0000 7f0be7422740 1 --2- 172.21.15.12:0/868492179 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f1623f0 0x55711f16c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.592+0000 7f0be7422740 1 -- 172.21.15.12:0/868492179 >> 172.21.15.12:0/868492179 conn(0x55711f15d960 msgr2=0x55711f15cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.592+0000 7f0bbf400640 0 client.25967 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.592+0000 7f0be7422740 1 -- 172.21.15.12:0/868492179 shutdown_connections 2024-03-23T23:04:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.592+0000 7f0be7422740 1 -- 172.21.15.12:0/868492179 wait complete. 2024-03-23T23:04:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.592+0000 7f0be7422740 20 client.25967 trim_cache size 0 max 16384 2024-03-23T23:04:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.608+0000 7f0be7422740 1 Processor -- start 2024-03-23T23:04:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.608+0000 7f0be7422740 1 -- start start 2024-03-23T23:04:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.608+0000 7f0be7422740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f1623f0 0x55711f164140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.608+0000 7f0be7422740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55711f162de0 0x55711f164680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.608+0000 7f0be7422740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f167f50 0x55711f173b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.608+0000 7f0be7422740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc1b1280 con 0x55711f162de0 2024-03-23T23:04:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.608+0000 7f0be7422740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc1aed80 con 0x55711f167f50 2024-03-23T23:04:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.608+0000 7f0be7422740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc1ace20 con 0x55711f1623f0 2024-03-23T23:04:23.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.608+0000 7f0be2800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f167f50 0x55711f173b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:23.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.609+0000 7f0be1e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f1623f0 0x55711f164140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:23.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.609+0000 7f0be2800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f167f50 0x55711f173b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59738/0 (socket says 172.21.15.12:59738) 2024-03-23T23:04:23.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.609+0000 7f0be2800640 1 -- 172.21.15.12:0/4255359432 learned_addr learned my addr 172.21.15.12:0/4255359432 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:23.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.609+0000 7f0be1400640 1 --2- 172.21.15.12:0/4255359432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55711f162de0 0x55711f164680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:23.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.609+0000 7f0be2800640 1 -- 172.21.15.12:0/4255359432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f1623f0 msgr2=0x55711f164140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:23.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.609+0000 7f0be2800640 1 --2- 172.21.15.12:0/4255359432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f1623f0 0x55711f164140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.609+0000 7f0be2800640 1 -- 172.21.15.12:0/4255359432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55711f162de0 msgr2=0x55711f164680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:23.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.609+0000 7f0be2800640 1 --2- 172.21.15.12:0/4255359432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55711f162de0 0x55711f164680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.609+0000 7f0be2800640 1 -- 172.21.15.12:0/4255359432 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55711f1627d0 con 0x55711f167f50 2024-03-23T23:04:23.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.609+0000 7f0be1400640 1 --2- 172.21.15.12:0/4255359432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55711f162de0 0x55711f164680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:04:23.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.609+0000 7f0be1e00640 1 --2- 172.21.15.12:0/4255359432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f1623f0 0x55711f164140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:23.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.609+0000 7f0be2800640 1 --2- 172.21.15.12:0/4255359432 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f167f50 0x55711f173b00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8011910 tx=0x7f0bd80096b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=704cf000d50f9eba server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.609+0000 7f0bca600640 1 -- 172.21.15.12:0/4255359432 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd800f4e0 con 0x55711f167f50 2024-03-23T23:04:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.610+0000 7f0bca600640 1 -- 172.21.15.12:0/4255359432 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0bd802c040 con 0x55711f167f50 2024-03-23T23:04:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.610+0000 7f0bca600640 1 -- 172.21.15.12:0/4255359432 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd8019cf0 con 0x55711f167f50 2024-03-23T23:04:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.610+0000 7f0be7422740 1 -- 172.21.15.12:0/4255359432 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f167f50 msgr2=0x55711f173b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.610+0000 7f0be7422740 1 --2- 172.21.15.12:0/4255359432 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f167f50 0x55711f173b00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8011910 tx=0x7f0bd80096b0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.610+0000 7f0be7422740 1 -- 172.21.15.12:0/4255359432 shutdown_connections 2024-03-23T23:04:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.610+0000 7f0be7422740 1 --2- 172.21.15.12:0/4255359432 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f167f50 0x55711f173b00 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.610+0000 7f0be7422740 1 --2- 172.21.15.12:0/4255359432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55711f162de0 0x55711f164680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.610+0000 7f0be7422740 1 --2- 172.21.15.12:0/4255359432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f1623f0 0x55711f164140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.610+0000 7f0be7422740 1 -- 172.21.15.12:0/4255359432 >> 172.21.15.12:0/4255359432 conn(0x55711f15d960 msgr2=0x55711f16c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.610+0000 7f0be7422740 1 -- 172.21.15.12:0/4255359432 shutdown_connections 2024-03-23T23:04:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.610+0000 7f0be7422740 1 -- 172.21.15.12:0/4255359432 wait complete. 2024-03-23T23:04:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.611+0000 7f0be7422740 1 -- start start 2024-03-23T23:04:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.611+0000 7f0be7422740 1 Processor -- start 2024-03-23T23:04:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.612+0000 7f0be7422740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55711f1623f0 0x55711f170bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.612+0000 7f0be7422740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f162de0 0x55711f171130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.612+0000 7f0be7422740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f167f50 0x55711f1720f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.612+0000 7f0be7422740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc1ad420 con 0x55711f1623f0 2024-03-23T23:04:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.612+0000 7f0be7422740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc1ae720 con 0x55711f162de0 2024-03-23T23:04:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.612+0000 7f0be7422740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc1ada60 con 0x55711f167f50 2024-03-23T23:04:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.612+0000 7f0be7422740 10 client.0 _ll_register_callbacks cb 0x7f0bd00f4400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.612+0000 7f0be2800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f167f50 0x55711f1720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.612+0000 7f0be1400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f162de0 0x55711f171130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.612+0000 7f0be1e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55711f1623f0 0x55711f170bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:23.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.612+0000 7f0be2800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f167f50 0x55711f1720f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:48832/0 (socket says 172.21.15.12:48832) 2024-03-23T23:04:23.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.612+0000 7f0be2800640 1 -- 172.21.15.12:0/3494366094 learned_addr learned my addr 172.21.15.12:0/3494366094 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:23.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.612+0000 7f0be1400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f162de0 0x55711f171130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59740/0 (socket says 172.21.15.12:59740) 2024-03-23T23:04:23.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.612+0000 7f0be2800640 1 -- 172.21.15.12:0/3494366094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f162de0 msgr2=0x55711f171130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:23.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.612+0000 7f0be2800640 1 --2- 172.21.15.12:0/3494366094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f162de0 0x55711f171130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.612+0000 7f0be2800640 1 -- 172.21.15.12:0/3494366094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55711f1623f0 msgr2=0x55711f170bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:23.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.612+0000 7f0be2800640 1 --2- 172.21.15.12:0/3494366094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55711f1623f0 0x55711f170bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.612+0000 7f0be2800640 1 -- 172.21.15.12:0/3494366094 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55711f1726e0 con 0x55711f167f50 2024-03-23T23:04:23.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.613+0000 7f0be2800640 1 --2- 172.21.15.12:0/3494366094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f167f50 0x55711f1720f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8019820 tx=0x7f0bd800f4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b2a55c6042387f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:23.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.613+0000 7f0bbf400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:04:23.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.613+0000 7f0bbf400640 1 -- 172.21.15.12:0/3494366094 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd8055060 con 0x55711f167f50 2024-03-23T23:04:23.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.613+0000 7f0be7422740 1 -- 172.21.15.12:0/3494366094 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0bd802c040 con 0x55711f167f50 2024-03-23T23:04:23.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.613+0000 7f0bbf400640 1 -- 172.21.15.12:0/3494366094 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0bd8019950 con 0x55711f167f50 2024-03-23T23:04:23.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.613+0000 7f0bbf400640 1 -- 172.21.15.12:0/3494366094 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0bd8009c40 con 0x55711f167f50 2024-03-23T23:04:23.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.613+0000 7f0be7422740 10 client.25954 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:23.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.613+0000 7f0be7422740 1 -- 172.21.15.12:0/3494366094 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55711f173a20 con 0x55711f167f50 2024-03-23T23:04:23.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.613+0000 7f0bbf400640 1 -- 172.21.15.12:0/3494366094 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0bd802c040 con 0x55711f167f50 2024-03-23T23:04:23.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.614+0000 7f0bbf400640 1 -- 172.21.15.12:0/3494366094 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f0bd804f070 con 0x55711f167f50 2024-03-23T23:04:23.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.614+0000 7f0be7422740 10 client.25954 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:23.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.614+0000 7f0be7422740 10 client.25954 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:04:23.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.614+0000 7f0be7422740 20 client.25954 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:23.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.614+0000 7f0be7422740 1 --2- 172.21.15.12:0/3494366094 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55711f1d3f00 0x55711f1d62f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:23.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.614+0000 7f0be7422740 4 client.25954 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:04:23.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.614+0000 7f0be7422740 1 -- 172.21.15.12:0/3494366094 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55711f0cf0c0 con 0x55711f1d3f00 2024-03-23T23:04:23.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.614+0000 7f0be1e00640 1 --2- 172.21.15.12:0/3494366094 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55711f1d3f00 0x55711f1d62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:23.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.614+0000 7f0be1e00640 1 --2- 172.21.15.12:0/3494366094 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55711f1d3f00 0x55711f1d62f0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8e74a024dbd7714 server_cookie=f1b38b203e1ba2f5 in_seq=0 out_seq=0 2024-03-23T23:04:23.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.614+0000 7f0bbf400640 10 client.25954 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.626+0000 7f0bbf400640 1 -- 172.21.15.12:0/3494366094 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55711f0cf0c0 con 0x55711f1d3f00 2024-03-23T23:04:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.626+0000 7f0bbf400640 10 client.25954 handle_command_reply: tid=0 2024-03-23T23:04:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.626+0000 7f0be7422740 1 client.25954 shutdown 2024-03-23T23:04:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.627+0000 7f0be7422740 1 -- 172.21.15.12:0/3494366094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f167f50 msgr2=0x55711f1720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.627+0000 7f0be7422740 1 --2- 172.21.15.12:0/3494366094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f167f50 0x55711f1720f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8019820 tx=0x7f0bd800f4e0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.627+0000 7f0be7422740 1 -- 172.21.15.12:0/3494366094 shutdown_connections 2024-03-23T23:04:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.627+0000 7f0be7422740 1 --2- 172.21.15.12:0/3494366094 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55711f1d3f00 0x55711f1d62f0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.627+0000 7f0be7422740 1 --2- 172.21.15.12:0/3494366094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55711f167f50 0x55711f1720f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.627+0000 7f0be7422740 1 --2- 172.21.15.12:0/3494366094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55711f162de0 0x55711f171130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.627+0000 7f0be7422740 1 --2- 172.21.15.12:0/3494366094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55711f1623f0 0x55711f170bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.627+0000 7f0be7422740 1 -- 172.21.15.12:0/3494366094 >> 172.21.15.12:0/3494366094 conn(0x55711f15d960 msgr2=0x55711f16c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.627+0000 7f0bbf400640 0 client.25954 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.627+0000 7f0be7422740 1 -- 172.21.15.12:0/3494366094 shutdown_connections 2024-03-23T23:04:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.627+0000 7f0be7422740 1 -- 172.21.15.12:0/3494366094 wait complete. 2024-03-23T23:04:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.627+0000 7f0be7422740 20 client.25954 trim_cache size 0 max 16384 2024-03-23T23:04:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:04:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:04:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:04:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:04:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.628+0000 7f0be3200640 1 -- 172.21.15.12:0/2891714394 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0bc40a27f0 msgr2=0x7f0bc40a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.628+0000 7f0be3200640 1 --2- 172.21.15.12:0/2891714394 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0bc40a27f0 0x7f0bc40a4cb0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8020000 tx=0x7f0bd801bd90 comp rx=0 tx=0).stop 2024-03-23T23:04:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.628+0000 7f0be3200640 1 -- 172.21.15.12:0/2891714394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0bdc0ffd80 msgr2=0x7f0bdc19de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:23.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.628+0000 7f0be3200640 1 --2- 172.21.15.12:0/2891714394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0bdc0ffd80 0x7f0bdc19de10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0000c00 tx=0x7f0bd002c650 comp rx=0 tx=0).stop 2024-03-23T23:04:23.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.628+0000 7f0be3200640 1 -- 172.21.15.12:0/2891714394 shutdown_connections 2024-03-23T23:04:23.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.628+0000 7f0be3200640 1 --2- 172.21.15.12:0/2891714394 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0bc40a27f0 0x7f0bc40a4cb0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.628+0000 7f0be3200640 1 --2- 172.21.15.12:0/2891714394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bdc1050a0 0x7f0bdc1a53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.628+0000 7f0be3200640 1 --2- 172.21.15.12:0/2891714394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bdc1026f0 0x7f0bdc19e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.628+0000 7f0be3200640 1 --2- 172.21.15.12:0/2891714394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0bdc0ffd80 0x7f0bdc19de10 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:23.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.628+0000 7f0be3200640 1 -- 172.21.15.12:0/2891714394 >> 172.21.15.12:0/2891714394 conn(0x7f0bdc0fb9a0 msgr2=0x7f0bdc102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:23.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.628+0000 7f0be3200640 1 -- 172.21.15.12:0/2891714394 shutdown_connections 2024-03-23T23:04:23.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:23.628+0000 7f0be3200640 1 -- 172.21.15.12:0/2891714394 wait complete. 2024-03-23T23:04:25.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:24 smithi012 ceph-mon[31695]: pgmap v468: 97 pgs: 97 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:04:25.425 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:24 smithi203 ceph-mon[36693]: pgmap v468: 97 pgs: 97 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:04:25.426 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:24 smithi173 ceph-mon[36116]: pgmap v468: 97 pgs: 97 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:04:27.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:26 smithi012 ceph-mon[31695]: pgmap v469: 97 pgs: 97 active+clean; 1.3 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:04:27.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:26 smithi173 ceph-mon[36116]: pgmap v469: 97 pgs: 97 active+clean; 1.3 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:04:27.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:26 smithi203 ceph-mon[36693]: pgmap v469: 97 pgs: 97 active+clean; 1.3 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:04:29.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:28 smithi012 ceph-mon[31695]: pgmap v470: 97 pgs: 97 active+clean; 1.3 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:04:29.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:28 smithi173 ceph-mon[36116]: pgmap v470: 97 pgs: 97 active+clean; 1.3 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:04:29.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:28 smithi203 ceph-mon[36693]: pgmap v470: 97 pgs: 97 active+clean; 1.3 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:04:31.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:30 smithi012 ceph-mon[31695]: pgmap v471: 97 pgs: 97 active+clean; 1.3 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:04:31.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:30 smithi173 ceph-mon[36116]: pgmap v471: 97 pgs: 97 active+clean; 1.3 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:04:31.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:30 smithi203 ceph-mon[36693]: pgmap v471: 97 pgs: 97 active+clean; 1.3 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:04:32.959 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:32 smithi203 ceph-mon[36693]: pgmap v472: 97 pgs: 97 active+clean; 1.3 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:04:33.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:32 smithi012 ceph-mon[31695]: pgmap v472: 97 pgs: 97 active+clean; 1.3 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:04:33.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:32 smithi173 ceph-mon[36116]: pgmap v472: 97 pgs: 97 active+clean; 1.3 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:04:33.970 INFO:tasks.workunit.client.0.smithi012.stdout:1/600: open fd = 3 2024-03-23T23:04:33.970 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.971 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.971 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.971 INFO:tasks.workunit.client.0.smithi012.stdout:2/600: open fd = 3 2024-03-23T23:04:33.971 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.971 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.972 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.972 INFO:tasks.workunit.client.0.smithi012.stdout:3/600: open fd = 3 2024-03-23T23:04:33.972 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.972 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.972 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.972 INFO:tasks.workunit.client.0.smithi012.stdout:4/600: open fd = 3 2024-03-23T23:04:33.972 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.973 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.973 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.973 INFO:tasks.workunit.client.0.smithi012.stdout:5/600: open fd = 3 2024-03-23T23:04:33.973 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.973 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.973 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.974 INFO:tasks.workunit.client.0.smithi012.stdout:6/600: open fd = 3 2024-03-23T23:04:33.974 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.974 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.974 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.974 INFO:tasks.workunit.client.0.smithi012.stdout:7/600: open fd = 3 2024-03-23T23:04:33.974 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.974 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.975 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.975 INFO:tasks.workunit.client.0.smithi012.stdout:8/600: open fd = 3 2024-03-23T23:04:33.975 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.975 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.975 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.975 INFO:tasks.workunit.client.0.smithi012.stdout:9/600: open fd = 3 2024-03-23T23:04:33.975 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.975 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.976 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.976 INFO:tasks.workunit.client.0.smithi012.stdout:10/600: open fd = 3 2024-03-23T23:04:33.976 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.976 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.976 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.976 INFO:tasks.workunit.client.0.smithi012.stdout:11/600: open fd = 3 2024-03-23T23:04:33.976 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.976 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.976 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.977 INFO:tasks.workunit.client.0.smithi012.stdout:12/600: open fd = 3 2024-03-23T23:04:33.977 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.977 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.977 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.977 INFO:tasks.workunit.client.0.smithi012.stdout:13/600: open fd = 3 2024-03-23T23:04:33.977 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.977 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.978 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.978 INFO:tasks.workunit.client.0.smithi012.stdout:14/600: open fd = 3 2024-03-23T23:04:33.978 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.978 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.978 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.978 INFO:tasks.workunit.client.0.smithi012.stdout:15/600: open fd = 3 2024-03-23T23:04:33.978 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.978 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.978 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.979 INFO:tasks.workunit.client.0.smithi012.stdout:16/600: open fd = 3 2024-03-23T23:04:33.979 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.979 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.979 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.979 INFO:tasks.workunit.client.0.smithi012.stdout:17/600: open fd = 3 2024-03-23T23:04:33.979 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.979 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.979 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.979 INFO:tasks.workunit.client.0.smithi012.stdout:18/600: open fd = 3 2024-03-23T23:04:33.979 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.979 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.979 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.979 INFO:tasks.workunit.client.0.smithi012.stdout:19/600: open fd = 3 2024-03-23T23:04:33.979 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.979 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.980 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.980 INFO:tasks.workunit.client.0.smithi012.stdout:20/600: open fd = 3 2024-03-23T23:04:33.980 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.980 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.980 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.980 INFO:tasks.workunit.client.0.smithi012.stdout:21/600: open fd = 3 2024-03-23T23:04:33.980 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.980 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.980 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.980 INFO:tasks.workunit.client.0.smithi012.stdout:22/600: open fd = 3 2024-03-23T23:04:33.980 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.980 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.980 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.980 INFO:tasks.workunit.client.0.smithi012.stdout:23/600: open fd = 3 2024-03-23T23:04:33.980 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.981 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.981 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.981 INFO:tasks.workunit.client.0.smithi012.stdout:24/600: open fd = 3 2024-03-23T23:04:33.981 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.981 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.981 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.981 INFO:tasks.workunit.client.0.smithi012.stdout:25/600: open fd = 3 2024-03-23T23:04:33.981 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.981 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.981 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.981 INFO:tasks.workunit.client.0.smithi012.stdout:26/600: open fd = 3 2024-03-23T23:04:33.981 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.981 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.981 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.982 INFO:tasks.workunit.client.0.smithi012.stdout:27/600: open fd = 3 2024-03-23T23:04:33.982 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.982 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.982 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.982 INFO:tasks.workunit.client.0.smithi012.stdout:28/600: open fd = 3 2024-03-23T23:04:33.982 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.982 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.982 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.982 INFO:tasks.workunit.client.0.smithi012.stdout:29/600: open fd = 3 2024-03-23T23:04:33.982 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.982 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.982 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.982 INFO:tasks.workunit.client.0.smithi012.stdout:30/600: open fd = 3 2024-03-23T23:04:33.982 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.983 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.983 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.983 INFO:tasks.workunit.client.0.smithi012.stdout:31/600: open fd = 3 2024-03-23T23:04:33.983 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.983 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.983 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.983 INFO:tasks.workunit.client.0.smithi012.stdout:32/600: open fd = 3 2024-03-23T23:04:33.983 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.983 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.983 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.984 INFO:tasks.workunit.client.0.smithi012.stdout:33/600: open fd = 3 2024-03-23T23:04:33.984 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.984 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.984 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.984 INFO:tasks.workunit.client.0.smithi012.stdout:34/600: open fd = 3 2024-03-23T23:04:33.984 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.984 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.984 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.985 INFO:tasks.workunit.client.0.smithi012.stdout:35/600: open fd = 3 2024-03-23T23:04:33.985 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.985 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.985 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.985 INFO:tasks.workunit.client.0.smithi012.stdout:36/600: open fd = 3 2024-03-23T23:04:33.985 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.985 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.985 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.985 INFO:tasks.workunit.client.0.smithi012.stdout:37/600: open fd = 3 2024-03-23T23:04:33.985 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.985 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.985 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.985 INFO:tasks.workunit.client.0.smithi012.stdout:38/600: open fd = 3 2024-03-23T23:04:33.986 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.986 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.986 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.986 INFO:tasks.workunit.client.0.smithi012.stdout:39/600: open fd = 3 2024-03-23T23:04:33.986 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.986 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.986 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.986 INFO:tasks.workunit.client.0.smithi012.stdout:40/600: open fd = 3 2024-03-23T23:04:33.986 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.986 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.986 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.986 INFO:tasks.workunit.client.0.smithi012.stdout:41/600: open fd = 3 2024-03-23T23:04:33.986 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.987 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.987 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.987 INFO:tasks.workunit.client.0.smithi012.stdout:42/600: open fd = 3 2024-03-23T23:04:33.987 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.987 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.987 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.987 INFO:tasks.workunit.client.0.smithi012.stdout:43/600: open fd = 3 2024-03-23T23:04:33.987 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.987 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.987 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.987 INFO:tasks.workunit.client.0.smithi012.stdout:44/600: open fd = 3 2024-03-23T23:04:33.987 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.987 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.987 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.988 INFO:tasks.workunit.client.0.smithi012.stdout:45/600: open fd = 3 2024-03-23T23:04:33.988 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.988 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.988 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.988 INFO:tasks.workunit.client.0.smithi012.stdout:46/600: open fd = 3 2024-03-23T23:04:33.988 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.988 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.988 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.988 INFO:tasks.workunit.client.0.smithi012.stdout:47/600: open fd = 3 2024-03-23T23:04:33.988 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.988 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.988 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.988 INFO:tasks.workunit.client.0.smithi012.stdout:48/600: open fd = 3 2024-03-23T23:04:33.988 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.989 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.989 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.989 INFO:tasks.workunit.client.0.smithi012.stdout:49/600: open fd = 3 2024-03-23T23:04:33.989 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.989 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.989 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.989 INFO:tasks.workunit.client.0.smithi012.stdout:50/600: open fd = 3 2024-03-23T23:04:33.989 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.989 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.989 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.989 INFO:tasks.workunit.client.0.smithi012.stdout:51/600: open fd = 3 2024-03-23T23:04:33.989 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.989 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.990 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.990 INFO:tasks.workunit.client.0.smithi012.stdout:52/600: open fd = 3 2024-03-23T23:04:33.990 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.990 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.990 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.990 INFO:tasks.workunit.client.0.smithi012.stdout:53/600: open fd = 3 2024-03-23T23:04:33.990 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.990 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.990 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.990 INFO:tasks.workunit.client.0.smithi012.stdout:54/600: open fd = 3 2024-03-23T23:04:33.990 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.990 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.990 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.991 INFO:tasks.workunit.client.0.smithi012.stdout:55/600: open fd = 3 2024-03-23T23:04:33.991 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.991 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.991 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.991 INFO:tasks.workunit.client.0.smithi012.stdout:56/600: open fd = 3 2024-03-23T23:04:33.991 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.991 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.991 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.991 INFO:tasks.workunit.client.0.smithi012.stdout:57/600: open fd = 3 2024-03-23T23:04:33.991 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.991 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.991 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.991 INFO:tasks.workunit.client.0.smithi012.stdout:58/600: open fd = 3 2024-03-23T23:04:33.991 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.991 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.992 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.992 INFO:tasks.workunit.client.0.smithi012.stdout:59/600: open fd = 3 2024-03-23T23:04:33.992 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.992 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.992 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.992 INFO:tasks.workunit.client.0.smithi012.stdout:60/600: open fd = 3 2024-03-23T23:04:33.992 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.992 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.992 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.992 INFO:tasks.workunit.client.0.smithi012.stdout:61/600: open fd = 3 2024-03-23T23:04:33.992 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.992 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.992 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.992 INFO:tasks.workunit.client.0.smithi012.stdout:62/600: open fd = 3 2024-03-23T23:04:33.992 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.993 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.993 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:33.993 INFO:tasks.workunit.client.0.smithi012.stdout:63/600: open fd = 3 2024-03-23T23:04:33.993 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.993 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:04:33.993 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:04:35.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:34 smithi012 ceph-mon[31695]: pgmap v473: 97 pgs: 97 active+clean; 1.4 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:04:35.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:34 smithi173 ceph-mon[36116]: pgmap v473: 97 pgs: 97 active+clean; 1.4 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:04:35.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:34 smithi203 ceph-mon[36693]: pgmap v473: 97 pgs: 97 active+clean; 1.4 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:04:36.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.779+0000 7f2c26000640 1 -- 172.21.15.12:0/3501059091 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c20103900 msgr2=0x7f2c20103d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:36.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.779+0000 7f2c26000640 1 --2- 172.21.15.12:0/3501059091 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c20103900 0x7f2c20103d20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2c10009900 tx=0x7f2c1002bc30 comp rx=0 tx=0).stop 2024-03-23T23:04:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.779+0000 7f2c26000640 1 -- 172.21.15.12:0/3501059091 shutdown_connections 2024-03-23T23:04:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.779+0000 7f2c26000640 1 --2- 172.21.15.12:0/3501059091 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c201054b0 0x7f2c2010bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.779+0000 7f2c26000640 1 --2- 172.21.15.12:0/3501059091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c20104af0 0x7f2c20104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.779+0000 7f2c26000640 1 --2- 172.21.15.12:0/3501059091 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c20103900 0x7f2c20103d20 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.779+0000 7f2c26000640 1 -- 172.21.15.12:0/3501059091 >> 172.21.15.12:0/3501059091 conn(0x7f2c200fee90 msgr2=0x7f2c201012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.779+0000 7f2c26000640 1 -- 172.21.15.12:0/3501059091 shutdown_connections 2024-03-23T23:04:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.779+0000 7f2c26000640 1 -- 172.21.15.12:0/3501059091 wait complete. 2024-03-23T23:04:36.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.780+0000 7f2c26000640 1 Processor -- start 2024-03-23T23:04:36.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.780+0000 7f2c26000640 1 -- start start 2024-03-23T23:04:36.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.781+0000 7f2c26000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c20103900 0x7f2c20199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:36.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.781+0000 7f2c26000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c20104af0 0x7f2c2019a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:36.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.781+0000 7f2c26000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c201054b0 0x7f2c201a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:36.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.781+0000 7f2c26000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2010f5a0 con 0x7f2c20103900 2024-03-23T23:04:36.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.781+0000 7f2c26000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2010f420 con 0x7f2c201054b0 2024-03-23T23:04:36.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.781+0000 7f2c26000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2010f720 con 0x7f2c20104af0 2024-03-23T23:04:36.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.781+0000 7f2c1ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c20104af0 0x7f2c2019a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:36.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.781+0000 7f2c1ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c20104af0 0x7f2c2019a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54786/0 (socket says 172.21.15.12:54786) 2024-03-23T23:04:36.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.781+0000 7f2c1fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c201054b0 0x7f2c201a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:36.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.781+0000 7f2c1ea00640 1 -- 172.21.15.12:0/4129293093 learned_addr learned my addr 172.21.15.12:0/4129293093 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:36.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.781+0000 7f2c1f400640 1 --2- 172.21.15.12:0/4129293093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c20103900 0x7f2c20199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:36.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.781+0000 7f2c1ea00640 1 -- 172.21.15.12:0/4129293093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c201054b0 msgr2=0x7f2c201a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:36.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.781+0000 7f2c1ea00640 1 --2- 172.21.15.12:0/4129293093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c201054b0 0x7f2c201a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.781+0000 7f2c1ea00640 1 -- 172.21.15.12:0/4129293093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c20103900 msgr2=0x7f2c20199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:36.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.781+0000 7f2c1ea00640 1 --2- 172.21.15.12:0/4129293093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c20103900 0x7f2c20199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.781+0000 7f2c1ea00640 1 -- 172.21.15.12:0/4129293093 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c201a17a0 con 0x7f2c20104af0 2024-03-23T23:04:36.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.781+0000 7f2c1fe00640 1 --2- 172.21.15.12:0/4129293093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c201054b0 0x7f2c201a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.782+0000 7f2c1ea00640 1 --2- 172.21.15.12:0/4129293093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c20104af0 0x7f2c2019a0c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c00e9f0 tx=0x7f2c0c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.782+0000 7f2c17e00640 1 -- 172.21.15.12:0/4129293093 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c00cd80 con 0x7f2c20104af0 2024-03-23T23:04:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.782+0000 7f2c17e00640 1 -- 172.21.15.12:0/4129293093 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c0c010910 con 0x7f2c20104af0 2024-03-23T23:04:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.782+0000 7f2c26000640 1 -- 172.21.15.12:0/4129293093 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c201a1a90 con 0x7f2c20104af0 2024-03-23T23:04:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.782+0000 7f2c17e00640 1 -- 172.21.15.12:0/4129293093 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c018990 con 0x7f2c20104af0 2024-03-23T23:04:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.782+0000 7f2c26000640 1 -- 172.21.15.12:0/4129293093 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c201a2030 con 0x7f2c20104af0 2024-03-23T23:04:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.787+0000 7f2c28e38740 1 Processor -- start 2024-03-23T23:04:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.787+0000 7f2c28e38740 1 -- start start 2024-03-23T23:04:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.787+0000 7f2c28e38740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612ce72e1e0 0x5612ce72e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.787+0000 7f2c28e38740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612ce72ebd0 0x5612ce7336d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.787+0000 7f2c28e38740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612ce733d40 0x5612ce7361d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.787+0000 7f2c28e38740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612ce6f13b0 con 0x5612ce72e1e0 2024-03-23T23:04:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.787+0000 7f2c28e38740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612ce693830 con 0x5612ce72ebd0 2024-03-23T23:04:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.787+0000 7f2c28e38740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612ce63c5d0 con 0x5612ce733d40 2024-03-23T23:04:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.788+0000 7f2c1f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612ce72ebd0 0x5612ce7336d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.788+0000 7f2c1fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612ce72e1e0 0x5612ce72e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.788+0000 7f2c1f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612ce72ebd0 0x5612ce7336d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48160/0 (socket says 172.21.15.12:48160) 2024-03-23T23:04:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.788+0000 7f2c1f400640 1 -- 172.21.15.12:0/2879769127 learned_addr learned my addr 172.21.15.12:0/2879769127 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.788+0000 7f2c17e00640 1 -- 172.21.15.12:0/4129293093 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2c0c010ac0 con 0x7f2c20104af0 2024-03-23T23:04:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.788+0000 7f2c1f400640 1 -- 172.21.15.12:0/2879769127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612ce733d40 msgr2=0x5612ce7361d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-23T23:04:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.788+0000 7f2c1f400640 1 --2- 172.21.15.12:0/2879769127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612ce733d40 0x5612ce7361d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.788+0000 7f2c1f400640 1 -- 172.21.15.12:0/2879769127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612ce72e1e0 msgr2=0x5612ce72e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.788+0000 7f2c1f400640 1 --2- 172.21.15.12:0/2879769127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612ce72e1e0 0x5612ce72e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.788+0000 7f2c1f400640 1 -- 172.21.15.12:0/2879769127 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5612ce6542c0 con 0x5612ce72ebd0 2024-03-23T23:04:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.788+0000 7f2c1fe00640 1 --2- 172.21.15.12:0/2879769127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612ce72e1e0 0x5612ce72e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.788+0000 7f2c1f400640 1 --2- 172.21.15.12:0/2879769127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612ce72ebd0 0x5612ce7336d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2c10000c00 tx=0x7f2c10009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e4a25fc32e097ce server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.789+0000 7f2c0be00640 1 -- 172.21.15.12:0/2879769127 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1002c980 con 0x5612ce72ebd0 2024-03-23T23:04:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.789+0000 7f2c0be00640 1 -- 172.21.15.12:0/2879769127 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c1002cb20 con 0x5612ce72ebd0 2024-03-23T23:04:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.789+0000 7f2c0be00640 1 -- 172.21.15.12:0/2879769127 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c10004920 con 0x5612ce72ebd0 2024-03-23T23:04:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.789+0000 7f2c17e00640 1 --2- 172.21.15.12:0/4129293093 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2c040a2920 0x7f2c040a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.789+0000 7f2c28e38740 1 -- 172.21.15.12:0/2879769127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612ce72ebd0 msgr2=0x5612ce7336d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.789+0000 7f2c28e38740 1 --2- 172.21.15.12:0/2879769127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612ce72ebd0 0x5612ce7336d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2c10000c00 tx=0x7f2c10009560 comp rx=0 tx=0).stop 2024-03-23T23:04:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.789+0000 7f2c17e00640 1 -- 172.21.15.12:0/4129293093 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2c0c0ced40 con 0x7f2c20104af0 2024-03-23T23:04:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.790+0000 7f2c28e38740 1 -- 172.21.15.12:0/2879769127 shutdown_connections 2024-03-23T23:04:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.790+0000 7f2c28e38740 1 --2- 172.21.15.12:0/2879769127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612ce733d40 0x5612ce7361d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.790+0000 7f2c28e38740 1 --2- 172.21.15.12:0/2879769127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612ce72ebd0 0x5612ce7336d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.790+0000 7f2c28e38740 1 --2- 172.21.15.12:0/2879769127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612ce72e1e0 0x5612ce72e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.790+0000 7f2c28e38740 1 -- 172.21.15.12:0/2879769127 >> 172.21.15.12:0/2879769127 conn(0x5612ce729750 msgr2=0x5612ce729b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.790+0000 7f2c1fe00640 1 --2- 172.21.15.12:0/4129293093 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2c040a2920 0x7f2c040a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.790+0000 7f2c28e38740 1 -- 172.21.15.12:0/2879769127 shutdown_connections 2024-03-23T23:04:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.790+0000 7f2c28e38740 1 -- 172.21.15.12:0/2879769127 wait complete. 2024-03-23T23:04:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.790+0000 7f2c28e38740 1 -- start start 2024-03-23T23:04:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.790+0000 7f2c1fe00640 1 --2- 172.21.15.12:0/4129293093 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2c040a2920 0x7f2c040a4de0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x5612ce72f430 tx=0x7f2c18009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.791+0000 7f2c28e38740 1 Processor -- start 2024-03-23T23:04:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.791+0000 7f2c28e38740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612ce72e1e0 0x5612ce738370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.791+0000 7f2c28e38740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612ce72ebd0 0x5612ce7388b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.791+0000 7f2c28e38740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612ce733d40 0x5612ce73f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.791+0000 7f2c28e38740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612ce64fa20 con 0x5612ce733d40 2024-03-23T23:04:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.791+0000 7f2c28e38740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612ce68f8d0 con 0x5612ce72e1e0 2024-03-23T23:04:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.791+0000 7f2c28e38740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612ce736c90 con 0x5612ce72ebd0 2024-03-23T23:04:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.791+0000 7f2c28e38740 10 client.0 _ll_register_callbacks cb 0x5612ce68c1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.791+0000 7f2c1f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612ce72e1e0 0x5612ce738370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.791+0000 7f2c1f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612ce72e1e0 0x5612ce738370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48168/0 (socket says 172.21.15.12:48168) 2024-03-23T23:04:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.791+0000 7f2c1ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612ce72ebd0 0x5612ce7388b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.791+0000 7f2c1f400640 1 -- 172.21.15.12:0/238606524 learned_addr learned my addr 172.21.15.12:0/238606524 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.791+0000 7f2c1fe00640 1 --2- 172.21.15.12:0/238606524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612ce733d40 0x5612ce73f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.792+0000 7f2c1f400640 1 -- 172.21.15.12:0/238606524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612ce72ebd0 msgr2=0x5612ce7388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.792+0000 7f2c1f400640 1 --2- 172.21.15.12:0/238606524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612ce72ebd0 0x5612ce7388b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.792+0000 7f2c1f400640 1 -- 172.21.15.12:0/238606524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612ce733d40 msgr2=0x5612ce73f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.792+0000 7f2c1f400640 1 --2- 172.21.15.12:0/238606524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612ce733d40 0x5612ce73f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.792+0000 7f2c1f400640 1 -- 172.21.15.12:0/238606524 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5612ce3a8210 con 0x5612ce72e1e0 2024-03-23T23:04:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.792+0000 7f2c1fe00640 1 --2- 172.21.15.12:0/238606524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612ce733d40 0x5612ce73f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:04:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.792+0000 7f2c1f400640 1 --2- 172.21.15.12:0/238606524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612ce72e1e0 0x5612ce738370 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2c1002c240 tx=0x7f2c100045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e492ae87a1b98643 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.792+0000 7f2c08c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:04:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.792+0000 7f2c08c00640 1 -- 172.21.15.12:0/238606524 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1004d070 con 0x5612ce72e1e0 2024-03-23T23:04:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.792+0000 7f2c08c00640 1 -- 172.21.15.12:0/238606524 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2c10003a00 con 0x5612ce72e1e0 2024-03-23T23:04:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.792+0000 7f2c28e38740 1 -- 172.21.15.12:0/238606524 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2c1002cb20 con 0x5612ce72e1e0 2024-03-23T23:04:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.793+0000 7f2c08c00640 1 -- 172.21.15.12:0/238606524 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c10059050 con 0x5612ce72e1e0 2024-03-23T23:04:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.793+0000 7f2c28e38740 10 client.25985 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.793+0000 7f2c28e38740 1 -- 172.21.15.12:0/238606524 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5612ce474620 con 0x5612ce72e1e0 2024-03-23T23:04:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.793+0000 7f2c08c00640 1 -- 172.21.15.12:0/238606524 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c1002cb20 con 0x5612ce72e1e0 2024-03-23T23:04:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.793+0000 7f2c08c00640 1 -- 172.21.15.12:0/238606524 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f2c1003fb80 con 0x5612ce72e1e0 2024-03-23T23:04:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.793+0000 7f2c28e38740 10 client.25985 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.793+0000 7f2c28e38740 10 client.25985 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:04:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.793+0000 7f2c28e38740 20 client.25985 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.793+0000 7f2c28e38740 1 --2- 172.21.15.12:0/238606524 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5612ce743de0 0x5612ce7461d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.793+0000 7f2c28e38740 4 client.25985 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:04:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.793+0000 7f2c28e38740 1 -- 172.21.15.12:0/238606524 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5612ce6a0d70 con 0x5612ce743de0 2024-03-23T23:04:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.794+0000 7f2c1ea00640 1 --2- 172.21.15.12:0/238606524 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5612ce743de0 0x5612ce7461d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.794+0000 7f2c1ea00640 1 --2- 172.21.15.12:0/238606524 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5612ce743de0 0x5612ce7461d0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7bc778cadaad17b server_cookie=3179f8a6b77fc9ad in_seq=0 out_seq=0 2024-03-23T23:04:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.794+0000 7f2c08c00640 10 client.25985 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.795+0000 7f2c08c00640 1 -- 172.21.15.12:0/238606524 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5612ce6a0d70 con 0x5612ce743de0 2024-03-23T23:04:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.795+0000 7f2c08c00640 10 client.25985 handle_command_reply: tid=0 2024-03-23T23:04:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.795+0000 7f2c28e38740 1 client.25985 shutdown 2024-03-23T23:04:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.795+0000 7f2c28e38740 1 -- 172.21.15.12:0/238606524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612ce72e1e0 msgr2=0x5612ce738370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.795+0000 7f2c28e38740 1 --2- 172.21.15.12:0/238606524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612ce72e1e0 0x5612ce738370 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2c1002c240 tx=0x7f2c100045e0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.795+0000 7f2c28e38740 1 -- 172.21.15.12:0/238606524 shutdown_connections 2024-03-23T23:04:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.795+0000 7f2c28e38740 1 --2- 172.21.15.12:0/238606524 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5612ce743de0 0x5612ce7461d0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.795+0000 7f2c28e38740 1 --2- 172.21.15.12:0/238606524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612ce733d40 0x5612ce73f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.795+0000 7f2c28e38740 1 --2- 172.21.15.12:0/238606524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612ce72ebd0 0x5612ce7388b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.795+0000 7f2c28e38740 1 --2- 172.21.15.12:0/238606524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612ce72e1e0 0x5612ce738370 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.795+0000 7f2c08c00640 0 client.25985 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.795+0000 7f2c28e38740 1 -- 172.21.15.12:0/238606524 >> 172.21.15.12:0/238606524 conn(0x5612ce729750 msgr2=0x5612ce728950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.795+0000 7f2c28e38740 1 -- 172.21.15.12:0/238606524 shutdown_connections 2024-03-23T23:04:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.795+0000 7f2c28e38740 1 -- 172.21.15.12:0/238606524 wait complete. 2024-03-23T23:04:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.795+0000 7f2c28e38740 20 client.25985 trim_cache size 0 max 16384 2024-03-23T23:04:36.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.811+0000 7f2c28e38740 1 Processor -- start 2024-03-23T23:04:36.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.811+0000 7f2c28e38740 1 -- start start 2024-03-23T23:04:36.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.811+0000 7f2c28e38740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612ce72e1e0 0x5612ce72d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:36.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.811+0000 7f2c28e38740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612ce72ebd0 0x5612ce73f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:36.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.811+0000 7f2c28e38740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612ce733d40 0x5612ce74a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:36.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.811+0000 7f2c28e38740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c201ad090 con 0x5612ce72e1e0 2024-03-23T23:04:36.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.811+0000 7f2c28e38740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c201aab90 con 0x5612ce72ebd0 2024-03-23T23:04:36.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.811+0000 7f2c28e38740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c201a8c30 con 0x5612ce733d40 2024-03-23T23:04:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.811+0000 7f2c1fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612ce733d40 0x5612ce74a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.811+0000 7f2c1f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612ce72e1e0 0x5612ce72d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.811+0000 7f2c1fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612ce733d40 0x5612ce74a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54806/0 (socket says 172.21.15.12:54806) 2024-03-23T23:04:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.811+0000 7f2c1fe00640 1 -- 172.21.15.12:0/3627640216 learned_addr learned my addr 172.21.15.12:0/3627640216 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.811+0000 7f2c1ea00640 1 --2- 172.21.15.12:0/3627640216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612ce72ebd0 0x5612ce73f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.812+0000 7f2c1fe00640 1 -- 172.21.15.12:0/3627640216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612ce72ebd0 msgr2=0x5612ce73f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.812+0000 7f2c1fe00640 1 --2- 172.21.15.12:0/3627640216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612ce72ebd0 0x5612ce73f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.812+0000 7f2c1fe00640 1 -- 172.21.15.12:0/3627640216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612ce72e1e0 msgr2=0x5612ce72d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.812+0000 7f2c1fe00640 1 --2- 172.21.15.12:0/3627640216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612ce72e1e0 0x5612ce72d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.812+0000 7f2c1fe00640 1 -- 172.21.15.12:0/3627640216 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5612ce72a1a0 con 0x5612ce733d40 2024-03-23T23:04:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.812+0000 7f2c1f400640 1 --2- 172.21.15.12:0/3627640216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612ce72e1e0 0x5612ce72d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.812+0000 7f2c1fe00640 1 --2- 172.21.15.12:0/3627640216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612ce733d40 0x5612ce74a9a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2c18009420 tx=0x7f2c18016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc07839652856ec7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.812+0000 7f2c0b400640 1 -- 172.21.15.12:0/3627640216 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c18004b20 con 0x5612ce733d40 2024-03-23T23:04:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.813+0000 7f2c0b400640 1 -- 172.21.15.12:0/3627640216 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c18004cc0 con 0x5612ce733d40 2024-03-23T23:04:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.813+0000 7f2c0b400640 1 -- 172.21.15.12:0/3627640216 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1801c520 con 0x5612ce733d40 2024-03-23T23:04:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.813+0000 7f2c28e38740 1 -- 172.21.15.12:0/3627640216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612ce733d40 msgr2=0x5612ce74a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.813+0000 7f2c28e38740 1 --2- 172.21.15.12:0/3627640216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612ce733d40 0x5612ce74a9a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2c18009420 tx=0x7f2c18016000 comp rx=0 tx=0).stop 2024-03-23T23:04:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.813+0000 7f2c28e38740 1 -- 172.21.15.12:0/3627640216 shutdown_connections 2024-03-23T23:04:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.813+0000 7f2c28e38740 1 --2- 172.21.15.12:0/3627640216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612ce733d40 0x5612ce74a9a0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.813+0000 7f2c28e38740 1 --2- 172.21.15.12:0/3627640216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612ce72ebd0 0x5612ce73f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.813+0000 7f2c28e38740 1 --2- 172.21.15.12:0/3627640216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612ce72e1e0 0x5612ce72d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.813+0000 7f2c28e38740 1 -- 172.21.15.12:0/3627640216 >> 172.21.15.12:0/3627640216 conn(0x5612ce729750 msgr2=0x5612ce72c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.813+0000 7f2c28e38740 1 -- 172.21.15.12:0/3627640216 shutdown_connections 2024-03-23T23:04:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.813+0000 7f2c28e38740 1 -- 172.21.15.12:0/3627640216 wait complete. 2024-03-23T23:04:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.814+0000 7f2c28e38740 1 -- start start 2024-03-23T23:04:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.814+0000 7f2c28e38740 1 Processor -- start 2024-03-23T23:04:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.814+0000 7f2c28e38740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612ce72e1e0 0x5612ce73cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.814+0000 7f2c28e38740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612ce72ebd0 0x5612ce73d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.814+0000 7f2c28e38740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612ce733d40 0x5612ce73e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.814+0000 7f2c28e38740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c201a9230 con 0x5612ce72e1e0 2024-03-23T23:04:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.814+0000 7f2c28e38740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c201aa530 con 0x5612ce72ebd0 2024-03-23T23:04:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.814+0000 7f2c28e38740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c201a9870 con 0x5612ce733d40 2024-03-23T23:04:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.814+0000 7f2c28e38740 10 client.0 _ll_register_callbacks cb 0x7f2c18009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.814+0000 7f2c1fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612ce733d40 0x5612ce73e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.814+0000 7f2c1f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612ce72e1e0 0x5612ce73cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.814+0000 7f2c1ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612ce72ebd0 0x5612ce73d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.814+0000 7f2c1fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612ce733d40 0x5612ce73e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54822/0 (socket says 172.21.15.12:54822) 2024-03-23T23:04:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.815+0000 7f2c1fe00640 1 -- 172.21.15.12:0/3606331642 learned_addr learned my addr 172.21.15.12:0/3606331642 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.815+0000 7f2c1fe00640 1 -- 172.21.15.12:0/3606331642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612ce72ebd0 msgr2=0x5612ce73d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.815+0000 7f2c1fe00640 1 --2- 172.21.15.12:0/3606331642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612ce72ebd0 0x5612ce73d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.815+0000 7f2c1fe00640 1 -- 172.21.15.12:0/3606331642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612ce72e1e0 msgr2=0x5612ce73cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.815+0000 7f2c1fe00640 1 --2- 172.21.15.12:0/3606331642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612ce72e1e0 0x5612ce73cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.815+0000 7f2c1fe00640 1 -- 172.21.15.12:0/3606331642 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5612ce73e8d0 con 0x5612ce733d40 2024-03-23T23:04:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.815+0000 7f2c1ea00640 1 --2- 172.21.15.12:0/3606331642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612ce72ebd0 0x5612ce73d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.815+0000 7f2c1fe00640 1 --2- 172.21.15.12:0/3606331642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612ce733d40 0x5612ce73e2e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2c18018040 tx=0x7f2c18005350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9042ce5762e8cb50 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.815+0000 7f2c08c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:04:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.815+0000 7f2c08c00640 1 -- 172.21.15.12:0/3606331642 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c18005540 con 0x5612ce733d40 2024-03-23T23:04:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.816+0000 7f2c08c00640 1 -- 172.21.15.12:0/3606331642 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2c1801c480 con 0x5612ce733d40 2024-03-23T23:04:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.816+0000 7f2c28e38740 1 -- 172.21.15.12:0/3606331642 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2c18004cc0 con 0x5612ce733d40 2024-03-23T23:04:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.816+0000 7f2c08c00640 1 -- 172.21.15.12:0/3606331642 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c1801c680 con 0x5612ce733d40 2024-03-23T23:04:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.816+0000 7f2c28e38740 10 client.25969 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.816+0000 7f2c28e38740 1 -- 172.21.15.12:0/3606331642 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5612ce73fbe0 con 0x5612ce733d40 2024-03-23T23:04:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.816+0000 7f2c08c00640 1 -- 172.21.15.12:0/3606331642 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c18004cc0 con 0x5612ce733d40 2024-03-23T23:04:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.816+0000 7f2c08c00640 1 -- 172.21.15.12:0/3606331642 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f2c180314e0 con 0x5612ce733d40 2024-03-23T23:04:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.816+0000 7f2c28e38740 10 client.25969 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.816+0000 7f2c28e38740 10 client.25969 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:04:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.816+0000 7f2c28e38740 20 client.25969 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.816+0000 7f2c28e38740 1 --2- 172.21.15.12:0/3606331642 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5612ce79fd70 0x5612ce7a2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.816+0000 7f2c28e38740 4 client.25969 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:04:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.816+0000 7f2c28e38740 1 -- 172.21.15.12:0/3606331642 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5612ce69b0c0 con 0x5612ce79fd70 2024-03-23T23:04:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.817+0000 7f2c1f400640 1 --2- 172.21.15.12:0/3606331642 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5612ce79fd70 0x5612ce7a2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.817+0000 7f2c1f400640 1 --2- 172.21.15.12:0/3606331642 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5612ce79fd70 0x5612ce7a2160 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=698b6208d5766476 server_cookie=589349a7f945a404 in_seq=0 out_seq=0 2024-03-23T23:04:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.817+0000 7f2c08c00640 10 client.25969 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:36.828 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.829+0000 7f2c08c00640 1 -- 172.21.15.12:0/3606331642 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5612ce69b0c0 con 0x5612ce79fd70 2024-03-23T23:04:36.828 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.829+0000 7f2c08c00640 10 client.25969 handle_command_reply: tid=0 2024-03-23T23:04:36.828 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.829+0000 7f2c28e38740 1 client.25969 shutdown 2024-03-23T23:04:36.828 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.829+0000 7f2c28e38740 1 -- 172.21.15.12:0/3606331642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612ce733d40 msgr2=0x5612ce73e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:36.828 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.829+0000 7f2c28e38740 1 --2- 172.21.15.12:0/3606331642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612ce733d40 0x5612ce73e2e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2c18018040 tx=0x7f2c18005350 comp rx=0 tx=0).stop 2024-03-23T23:04:36.828 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.829+0000 7f2c28e38740 1 -- 172.21.15.12:0/3606331642 shutdown_connections 2024-03-23T23:04:36.829 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.829+0000 7f2c28e38740 1 --2- 172.21.15.12:0/3606331642 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5612ce79fd70 0x5612ce7a2160 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.829 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.829+0000 7f2c28e38740 1 --2- 172.21.15.12:0/3606331642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612ce733d40 0x5612ce73e2e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.829 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.829+0000 7f2c28e38740 1 --2- 172.21.15.12:0/3606331642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612ce72ebd0 0x5612ce73d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.829 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.829+0000 7f2c28e38740 1 --2- 172.21.15.12:0/3606331642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612ce72e1e0 0x5612ce73cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.829 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.829+0000 7f2c08c00640 0 client.25969 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:36.829 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.829+0000 7f2c28e38740 1 -- 172.21.15.12:0/3606331642 >> 172.21.15.12:0/3606331642 conn(0x5612ce729750 msgr2=0x5612ce72d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:36.829 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.829+0000 7f2c28e38740 1 -- 172.21.15.12:0/3606331642 shutdown_connections 2024-03-23T23:04:36.829 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.829+0000 7f2c28e38740 1 -- 172.21.15.12:0/3606331642 wait complete. 2024-03-23T23:04:36.829 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.830+0000 7f2c28e38740 20 client.25969 trim_cache size 0 max 16384 2024-03-23T23:04:36.829 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:04:36.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:04:36.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:04:36.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:04:36.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.830+0000 7f2c26000640 1 -- 172.21.15.12:0/4129293093 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2c040a2920 msgr2=0x7f2c040a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:36.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.830+0000 7f2c26000640 1 --2- 172.21.15.12:0/4129293093 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2c040a2920 0x7f2c040a4de0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x5612ce72f430 tx=0x7f2c18009500 comp rx=0 tx=0).stop 2024-03-23T23:04:36.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.830+0000 7f2c26000640 1 -- 172.21.15.12:0/4129293093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c20104af0 msgr2=0x7f2c2019a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:36.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.830+0000 7f2c26000640 1 --2- 172.21.15.12:0/4129293093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c20104af0 0x7f2c2019a0c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c00e9f0 tx=0x7f2c0c00eeb0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.830+0000 7f2c26000640 1 -- 172.21.15.12:0/4129293093 shutdown_connections 2024-03-23T23:04:36.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.830+0000 7f2c26000640 1 --2- 172.21.15.12:0/4129293093 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2c040a2920 0x7f2c040a4de0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.830+0000 7f2c26000640 1 --2- 172.21.15.12:0/4129293093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c201054b0 0x7f2c201a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.830+0000 7f2c26000640 1 --2- 172.21.15.12:0/4129293093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c20104af0 0x7f2c2019a0c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.830+0000 7f2c26000640 1 --2- 172.21.15.12:0/4129293093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c20103900 0x7f2c20199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:36.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.830+0000 7f2c26000640 1 -- 172.21.15.12:0/4129293093 >> 172.21.15.12:0/4129293093 conn(0x7f2c200fee90 msgr2=0x7f2c20107df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:36.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.831+0000 7f2c26000640 1 -- 172.21.15.12:0/4129293093 shutdown_connections 2024-03-23T23:04:36.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:36.831+0000 7f2c26000640 1 -- 172.21.15.12:0/4129293093 wait complete. 2024-03-23T23:04:37.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:36 smithi012 ceph-mon[31695]: pgmap v474: 97 pgs: 97 active+clean; 1.4 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:04:37.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:36 smithi173 ceph-mon[36116]: pgmap v474: 97 pgs: 97 active+clean; 1.4 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:04:37.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:36 smithi203 ceph-mon[36693]: pgmap v474: 97 pgs: 97 active+clean; 1.4 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:04:39.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:38 smithi012 ceph-mon[31695]: pgmap v475: 97 pgs: 97 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:04:39.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:38 smithi173 ceph-mon[36116]: pgmap v475: 97 pgs: 97 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:04:39.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:38 smithi203 ceph-mon[36693]: pgmap v475: 97 pgs: 97 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:04:41.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:40 smithi012 ceph-mon[31695]: pgmap v476: 97 pgs: 97 active+clean; 1.5 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 15 op/s 2024-03-23T23:04:41.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:40 smithi173 ceph-mon[36116]: pgmap v476: 97 pgs: 97 active+clean; 1.5 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 15 op/s 2024-03-23T23:04:41.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:40 smithi203 ceph-mon[36693]: pgmap v476: 97 pgs: 97 active+clean; 1.5 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 15 op/s 2024-03-23T23:04:41.716 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:04:41.717 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:04:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.959+0000 7f2061e00640 1 -- 172.21.15.12:0/838566212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f205c105350 msgr2=0x7f205c10bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:41.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.959+0000 7f2061e00640 1 --2- 172.21.15.12:0/838566212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f205c105350 0x7f205c10bbe0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f204c009900 tx=0x7f204c02bc40 comp rx=0 tx=0).stop 2024-03-23T23:04:41.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.959+0000 7f2061e00640 1 -- 172.21.15.12:0/838566212 shutdown_connections 2024-03-23T23:04:41.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.959+0000 7f2061e00640 1 --2- 172.21.15.12:0/838566212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f205c105350 0x7f205c10bbe0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:41.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.959+0000 7f2061e00640 1 --2- 172.21.15.12:0/838566212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f205c104990 0x7f205c104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:41.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.959+0000 7f2061e00640 1 --2- 172.21.15.12:0/838566212 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f205c1037a0 0x7f205c103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:41.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.959+0000 7f2061e00640 1 -- 172.21.15.12:0/838566212 >> 172.21.15.12:0/838566212 conn(0x7f205c0fed10 msgr2=0x7f205c101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:41.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.959+0000 7f2061e00640 1 -- 172.21.15.12:0/838566212 shutdown_connections 2024-03-23T23:04:41.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.959+0000 7f2061e00640 1 -- 172.21.15.12:0/838566212 wait complete. 2024-03-23T23:04:41.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.960+0000 7f2061e00640 1 Processor -- start 2024-03-23T23:04:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.960+0000 7f2061e00640 1 -- start start 2024-03-23T23:04:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.961+0000 7f2061e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f205c1037a0 0x7f205c199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.961+0000 7f2061e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f205c104990 0x7f205c199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.961+0000 7f2061e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f205c105350 0x7f205c1a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.961+0000 7f2061e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205c10f450 con 0x7f205c1037a0 2024-03-23T23:04:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.961+0000 7f2061e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205c10f2d0 con 0x7f205c105350 2024-03-23T23:04:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.961+0000 7f2061e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205c10f5d0 con 0x7f205c104990 2024-03-23T23:04:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.961+0000 7f2060a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f205c1037a0 0x7f205c199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.961+0000 7f2060a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f205c1037a0 0x7f205c199a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40014/0 (socket says 172.21.15.12:40014) 2024-03-23T23:04:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.961+0000 7f205be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f205c104990 0x7f205c199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.961+0000 7f2060a00640 1 -- 172.21.15.12:0/3292763677 learned_addr learned my addr 172.21.15.12:0/3292763677 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.961+0000 7f2061400640 1 --2- 172.21.15.12:0/3292763677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f205c105350 0x7f205c1a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.962+0000 7f205be00640 1 -- 172.21.15.12:0/3292763677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f205c105350 msgr2=0x7f205c1a0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:41.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.962+0000 7f205be00640 1 --2- 172.21.15.12:0/3292763677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f205c105350 0x7f205c1a0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:41.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.962+0000 7f205be00640 1 -- 172.21.15.12:0/3292763677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f205c1037a0 msgr2=0x7f205c199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:41.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.962+0000 7f205be00640 1 --2- 172.21.15.12:0/3292763677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f205c1037a0 0x7f205c199a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:41.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.962+0000 7f205be00640 1 -- 172.21.15.12:0/3292763677 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205c1a1650 con 0x7f205c104990 2024-03-23T23:04:41.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.962+0000 7f205be00640 1 --2- 172.21.15.12:0/3292763677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f205c104990 0x7f205c199f70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f204400ec80 tx=0x7f204400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:41.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.962+0000 7f2059600640 1 -- 172.21.15.12:0/3292763677 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2044003b70 con 0x7f205c104990 2024-03-23T23:04:41.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.962+0000 7f2059600640 1 -- 172.21.15.12:0/3292763677 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2044003d10 con 0x7f205c104990 2024-03-23T23:04:41.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.962+0000 7f2061e00640 1 -- 172.21.15.12:0/3292763677 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f205c1a1940 con 0x7f205c104990 2024-03-23T23:04:41.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.962+0000 7f2059600640 1 -- 172.21.15.12:0/3292763677 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2044010a30 con 0x7f205c104990 2024-03-23T23:04:41.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.962+0000 7f2061e00640 1 -- 172.21.15.12:0/3292763677 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f205c1a1ee0 con 0x7f205c104990 2024-03-23T23:04:41.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.967+0000 7f20661c7740 1 Processor -- start 2024-03-23T23:04:41.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.967+0000 7f2059600640 1 -- 172.21.15.12:0/3292763677 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2044003ec0 con 0x7f205c104990 2024-03-23T23:04:41.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.967+0000 7f20661c7740 1 -- start start 2024-03-23T23:04:41.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.967+0000 7f20661c7740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561901fa7050 0x561901fa7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:41.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.967+0000 7f20661c7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561901fa7a40 0x561901fac540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:41.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.967+0000 7f20661c7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561901facbb0 0x561901faf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:41.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.967+0000 7f20661c7740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561901f0c250 con 0x561901fa7050 2024-03-23T23:04:41.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.967+0000 7f20661c7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x561901eb4a30 con 0x561901fa7a40 2024-03-23T23:04:41.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.967+0000 7f20661c7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x561901f082f0 con 0x561901facbb0 2024-03-23T23:04:41.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.968+0000 7f2060a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561901fa7a40 0x561901fac540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:41.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.968+0000 7f205be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561901facbb0 0x561901faf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:41.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.968+0000 7f2060a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561901fa7a40 0x561901fac540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48230/0 (socket says 172.21.15.12:48230) 2024-03-23T23:04:41.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.968+0000 7f2060a00640 1 -- 172.21.15.12:0/2700704586 learned_addr learned my addr 172.21.15.12:0/2700704586 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:41.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.968+0000 7f2061400640 1 --2- 172.21.15.12:0/2700704586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561901fa7050 0x561901fa7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:41.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.968+0000 7f2060a00640 1 -- 172.21.15.12:0/2700704586 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561901facbb0 msgr2=0x561901faf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:41.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.968+0000 7f2060a00640 1 --2- 172.21.15.12:0/2700704586 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561901facbb0 0x561901faf040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:41.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.968+0000 7f2060a00640 1 -- 172.21.15.12:0/2700704586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561901fa7050 msgr2=0x561901fa7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:41.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.968+0000 7f2060a00640 1 --2- 172.21.15.12:0/2700704586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561901fa7050 0x561901fa7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:41.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.968+0000 7f2060a00640 1 -- 172.21.15.12:0/2700704586 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561901e9f690 con 0x561901fa7a40 2024-03-23T23:04:41.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.968+0000 7f2061400640 1 --2- 172.21.15.12:0/2700704586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561901fa7050 0x561901fa7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:04:41.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.968+0000 7f2060a00640 1 --2- 172.21.15.12:0/2700704586 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561901fa7a40 0x561901fac540 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f2054009d60 tx=0x7f2054011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bac91336e30037c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:41.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.969+0000 7f2059600640 1 --2- 172.21.15.12:0/3292763677 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f20400a2880 0x7f20400a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:41.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.969+0000 7f2051c00640 1 -- 172.21.15.12:0/2700704586 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2054004230 con 0x561901fa7a40 2024-03-23T23:04:41.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.969+0000 7f2051c00640 1 -- 172.21.15.12:0/2700704586 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f20540043d0 con 0x561901fa7a40 2024-03-23T23:04:41.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.969+0000 7f2051c00640 1 -- 172.21.15.12:0/2700704586 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20540198f0 con 0x561901fa7a40 2024-03-23T23:04:41.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.969+0000 7f2059600640 1 -- 172.21.15.12:0/3292763677 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f20440cf080 con 0x7f205c104990 2024-03-23T23:04:41.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.969+0000 7f2061400640 1 --2- 172.21.15.12:0/3292763677 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f20400a2880 0x7f20400a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:41.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.969+0000 7f20661c7740 1 -- 172.21.15.12:0/2700704586 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561901fa7a40 msgr2=0x561901fac540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:41.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.969+0000 7f20661c7740 1 --2- 172.21.15.12:0/2700704586 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561901fa7a40 0x561901fac540 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f2054009d60 tx=0x7f2054011810 comp rx=0 tx=0).stop 2024-03-23T23:04:41.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.969+0000 7f20661c7740 1 -- 172.21.15.12:0/2700704586 shutdown_connections 2024-03-23T23:04:41.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.969+0000 7f20661c7740 1 --2- 172.21.15.12:0/2700704586 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561901facbb0 0x561901faf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:41.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.969+0000 7f20661c7740 1 --2- 172.21.15.12:0/2700704586 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561901fa7a40 0x561901fac540 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:41.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.969+0000 7f20661c7740 1 --2- 172.21.15.12:0/2700704586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561901fa7050 0x561901fa7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:41.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.969+0000 7f20661c7740 1 -- 172.21.15.12:0/2700704586 >> 172.21.15.12:0/2700704586 conn(0x561901f6f200 msgr2=0x561901fa29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:41.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.969+0000 7f20661c7740 1 -- 172.21.15.12:0/2700704586 shutdown_connections 2024-03-23T23:04:41.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.970+0000 7f20661c7740 1 -- 172.21.15.12:0/2700704586 wait complete. 2024-03-23T23:04:41.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.970+0000 7f2061400640 1 --2- 172.21.15.12:0/3292763677 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f20400a2880 0x7f20400a4d40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f204c036040 tx=0x7f204c02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:41.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.970+0000 7f20661c7740 1 -- start start 2024-03-23T23:04:41.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.970+0000 7f20661c7740 1 Processor -- start 2024-03-23T23:04:41.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.971+0000 7f20661c7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561901fa7050 0x561901fb1060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:41.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.971+0000 7f20661c7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561901fa7a40 0x561901fb15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:41.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.971+0000 7f20661c7740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561901facbb0 0x561901fb85e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:41.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.971+0000 7f20661c7740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561901fafb60 con 0x561901facbb0 2024-03-23T23:04:41.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.971+0000 7f20661c7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x561901ec7e80 con 0x561901fa7050 2024-03-23T23:04:41.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.971+0000 7f20661c7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x561901fafce0 con 0x561901fa7a40 2024-03-23T23:04:41.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.971+0000 7f20661c7740 10 client.0 _ll_register_callbacks cb 0x561901eda130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:41.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.971+0000 7f2060a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561901fa7050 0x561901fb1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:41.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.971+0000 7f2060a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561901fa7050 0x561901fb1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48240/0 (socket says 172.21.15.12:48240) 2024-03-23T23:04:41.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.971+0000 7f2060a00640 1 -- 172.21.15.12:0/600563810 learned_addr learned my addr 172.21.15.12:0/600563810 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:41.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.971+0000 7f205be00640 1 --2- 172.21.15.12:0/600563810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561901fa7a40 0x561901fb15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:41.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.971+0000 7f2061400640 1 --2- 172.21.15.12:0/600563810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561901facbb0 0x561901fb85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:41.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.971+0000 7f2060a00640 1 -- 172.21.15.12:0/600563810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561901fa7a40 msgr2=0x561901fb15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:41.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.971+0000 7f2060a00640 1 --2- 172.21.15.12:0/600563810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561901fa7a40 0x561901fb15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:41.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.971+0000 7f2060a00640 1 -- 172.21.15.12:0/600563810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561901facbb0 msgr2=0x561901fb85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:41.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.971+0000 7f2060a00640 1 --2- 172.21.15.12:0/600563810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561901facbb0 0x561901fb85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:41.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.972+0000 7f2060a00640 1 -- 172.21.15.12:0/600563810 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561901cea7d0 con 0x561901fa7050 2024-03-23T23:04:41.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.972+0000 7f205be00640 1 --2- 172.21.15.12:0/600563810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561901fa7a40 0x561901fb15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:41.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.972+0000 7f2060a00640 1 --2- 172.21.15.12:0/600563810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561901fa7050 0x561901fb1060 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2054000c00 tx=0x7f2054009ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81433c8045674d22 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:41.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.972+0000 7f204a000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:04:41.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.972+0000 7f204a000640 1 -- 172.21.15.12:0/600563810 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2054028070 con 0x561901fa7050 2024-03-23T23:04:41.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.972+0000 7f204a000640 1 -- 172.21.15.12:0/600563810 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f20540053d0 con 0x561901fa7050 2024-03-23T23:04:41.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.972+0000 7f20661c7740 1 -- 172.21.15.12:0/600563810 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f20540043d0 con 0x561901fa7050 2024-03-23T23:04:41.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.972+0000 7f204a000640 1 -- 172.21.15.12:0/600563810 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2054034050 con 0x561901fa7050 2024-03-23T23:04:41.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.973+0000 7f20661c7740 10 client.26003 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:41.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.973+0000 7f20661c7740 1 -- 172.21.15.12:0/600563810 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561901c203f0 con 0x561901fa7050 2024-03-23T23:04:41.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.973+0000 7f204a000640 1 -- 172.21.15.12:0/600563810 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f20540043d0 con 0x561901fa7050 2024-03-23T23:04:41.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.973+0000 7f204a000640 1 -- 172.21.15.12:0/600563810 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f2054004ab0 con 0x561901fa7050 2024-03-23T23:04:41.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.973+0000 7f20661c7740 10 client.26003 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:41.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.973+0000 7f20661c7740 10 client.26003 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:04:41.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.973+0000 7f20661c7740 20 client.26003 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:41.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.973+0000 7f20661c7740 1 --2- 172.21.15.12:0/600563810 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561901fbcba0 0x561901fbefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:41.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.973+0000 7f20661c7740 4 client.26003 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:04:41.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.973+0000 7f20661c7740 1 -- 172.21.15.12:0/600563810 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561901f19790 con 0x561901fbcba0 2024-03-23T23:04:41.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.974+0000 7f205be00640 1 --2- 172.21.15.12:0/600563810 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561901fbcba0 0x561901fbefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:41.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.974+0000 7f205be00640 1 --2- 172.21.15.12:0/600563810 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561901fbcba0 0x561901fbefd0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98b1993884c09c00 server_cookie=35cb3a880bb33dca in_seq=0 out_seq=0 2024-03-23T23:04:41.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.974+0000 7f204a000640 10 client.26003 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:41.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.975+0000 7f204a000640 1 -- 172.21.15.12:0/600563810 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561901f19790 con 0x561901fbcba0 2024-03-23T23:04:41.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.975+0000 7f204a000640 10 client.26003 handle_command_reply: tid=0 2024-03-23T23:04:41.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.975+0000 7f20661c7740 1 client.26003 shutdown 2024-03-23T23:04:41.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.975+0000 7f20661c7740 1 -- 172.21.15.12:0/600563810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561901fa7050 msgr2=0x561901fb1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:41.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.975+0000 7f20661c7740 1 --2- 172.21.15.12:0/600563810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561901fa7050 0x561901fb1060 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2054000c00 tx=0x7f2054009ae0 comp rx=0 tx=0).stop 2024-03-23T23:04:41.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.975+0000 7f20661c7740 1 -- 172.21.15.12:0/600563810 shutdown_connections 2024-03-23T23:04:41.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.975+0000 7f20661c7740 1 --2- 172.21.15.12:0/600563810 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561901fbcba0 0x561901fbefd0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:41.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.975+0000 7f20661c7740 1 --2- 172.21.15.12:0/600563810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561901facbb0 0x561901fb85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:41.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.975+0000 7f20661c7740 1 --2- 172.21.15.12:0/600563810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561901fa7a40 0x561901fb15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:41.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.975+0000 7f20661c7740 1 --2- 172.21.15.12:0/600563810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561901fa7050 0x561901fb1060 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:41.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.975+0000 7f20661c7740 1 -- 172.21.15.12:0/600563810 >> 172.21.15.12:0/600563810 conn(0x561901f6f200 msgr2=0x561901fa1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:41.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.975+0000 7f204a000640 0 client.26003 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:41.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.975+0000 7f20661c7740 1 -- 172.21.15.12:0/600563810 shutdown_connections 2024-03-23T23:04:41.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.975+0000 7f20661c7740 1 -- 172.21.15.12:0/600563810 wait complete. 2024-03-23T23:04:41.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.975+0000 7f20661c7740 20 client.26003 trim_cache size 0 max 16384 2024-03-23T23:04:42.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.993+0000 7f20661c7740 1 Processor -- start 2024-03-23T23:04:42.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.993+0000 7f20661c7740 1 -- start start 2024-03-23T23:04:42.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.994+0000 7f20661c7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561901fa7050 0x561901fa84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.994+0000 7f20661c7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561901fa7a40 0x561901fb85a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.994+0000 7f20661c7740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561901facbb0 0x561901fc3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.994+0000 7f20661c7740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205c1acef0 con 0x561901facbb0 2024-03-23T23:04:42.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.994+0000 7f20661c7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205c1aa9f0 con 0x561901fa7a40 2024-03-23T23:04:42.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.994+0000 7f20661c7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205c1a8a90 con 0x561901fa7050 2024-03-23T23:04:42.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.994+0000 7f2060a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561901fa7050 0x561901fa84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.994+0000 7f2060a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561901fa7050 0x561901fa84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54900/0 (socket says 172.21.15.12:54900) 2024-03-23T23:04:42.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.994+0000 7f2060a00640 1 -- 172.21.15.12:0/1322322750 learned_addr learned my addr 172.21.15.12:0/1322322750 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:42.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.994+0000 7f2061400640 1 --2- 172.21.15.12:0/1322322750 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561901facbb0 0x561901fc3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.994+0000 7f205be00640 1 --2- 172.21.15.12:0/1322322750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561901fa7a40 0x561901fb85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.994+0000 7f2060a00640 1 -- 172.21.15.12:0/1322322750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561901fa7a40 msgr2=0x561901fb85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:42.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.994+0000 7f2060a00640 1 --2- 172.21.15.12:0/1322322750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561901fa7a40 0x561901fb85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.994+0000 7f2060a00640 1 -- 172.21.15.12:0/1322322750 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561901facbb0 msgr2=0x561901fc3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:42.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.994+0000 7f2060a00640 1 --2- 172.21.15.12:0/1322322750 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561901facbb0 0x561901fc3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.994+0000 7f2060a00640 1 -- 172.21.15.12:0/1322322750 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561901be00f0 con 0x561901fa7050 2024-03-23T23:04:42.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.994+0000 7f205be00640 1 --2- 172.21.15.12:0/1322322750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561901fa7a40 0x561901fb85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:04:42.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.995+0000 7f2060a00640 1 --2- 172.21.15.12:0/1322322750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561901fa7050 0x561901fa84b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f20540058c0 tx=0x7f2054001e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8438a2ff33c9ede1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:42.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.995+0000 7f2051200640 1 -- 172.21.15.12:0/1322322750 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2054028070 con 0x561901fa7050 2024-03-23T23:04:42.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.995+0000 7f2051200640 1 -- 172.21.15.12:0/1322322750 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2054019800 con 0x561901fa7050 2024-03-23T23:04:42.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.995+0000 7f2051200640 1 -- 172.21.15.12:0/1322322750 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20540048d0 con 0x561901fa7050 2024-03-23T23:04:42.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.995+0000 7f20661c7740 1 -- 172.21.15.12:0/1322322750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561901fa7050 msgr2=0x561901fa84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:42.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.995+0000 7f20661c7740 1 --2- 172.21.15.12:0/1322322750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561901fa7050 0x561901fa84b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f20540058c0 tx=0x7f2054001e50 comp rx=0 tx=0).stop 2024-03-23T23:04:42.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.995+0000 7f20661c7740 1 -- 172.21.15.12:0/1322322750 shutdown_connections 2024-03-23T23:04:42.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.995+0000 7f20661c7740 1 --2- 172.21.15.12:0/1322322750 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561901facbb0 0x561901fc3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.995+0000 7f20661c7740 1 --2- 172.21.15.12:0/1322322750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561901fa7a40 0x561901fb85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.995+0000 7f20661c7740 1 --2- 172.21.15.12:0/1322322750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561901fa7050 0x561901fa84b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.995+0000 7f20661c7740 1 -- 172.21.15.12:0/1322322750 >> 172.21.15.12:0/1322322750 conn(0x561901f6f200 msgr2=0x561901fa1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:42.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.996+0000 7f20661c7740 1 -- 172.21.15.12:0/1322322750 shutdown_connections 2024-03-23T23:04:42.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.996+0000 7f20661c7740 1 -- 172.21.15.12:0/1322322750 wait complete. 2024-03-23T23:04:42.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.996+0000 7f20661c7740 1 -- start start 2024-03-23T23:04:42.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.996+0000 7f20661c7740 1 Processor -- start 2024-03-23T23:04:42.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.997+0000 7f20661c7740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561901fa7050 0x561901fb58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.997+0000 7f20661c7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561901fa7a40 0x561901fb5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.997+0000 7f20661c7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561901facbb0 0x561901fb6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.997+0000 7f20661c7740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205c1a9090 con 0x561901fa7050 2024-03-23T23:04:42.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.997+0000 7f20661c7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205c1aa390 con 0x561901fa7a40 2024-03-23T23:04:42.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.997+0000 7f20661c7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205c1a96d0 con 0x561901facbb0 2024-03-23T23:04:42.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.997+0000 7f20661c7740 10 client.0 _ll_register_callbacks cb 0x7f20440d2a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:42.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.997+0000 7f2060a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561901fa7050 0x561901fb58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.997+0000 7f2061400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561901facbb0 0x561901fb6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.997+0000 7f2060a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561901fa7050 0x561901fb58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40044/0 (socket says 172.21.15.12:40044) 2024-03-23T23:04:42.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.997+0000 7f2060a00640 1 -- 172.21.15.12:0/2558393168 learned_addr learned my addr 172.21.15.12:0/2558393168 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:42.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.997+0000 7f205be00640 1 --2- 172.21.15.12:0/2558393168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561901fa7a40 0x561901fb5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.997+0000 7f2060a00640 1 -- 172.21.15.12:0/2558393168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561901facbb0 msgr2=0x561901fb6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:42.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.997+0000 7f2060a00640 1 --2- 172.21.15.12:0/2558393168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561901facbb0 0x561901fb6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.997+0000 7f2060a00640 1 -- 172.21.15.12:0/2558393168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561901fa7a40 msgr2=0x561901fb5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:42.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.997+0000 7f2060a00640 1 --2- 172.21.15.12:0/2558393168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561901fa7a40 0x561901fb5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.997+0000 7f2060a00640 1 -- 172.21.15.12:0/2558393168 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561901fb73f0 con 0x561901fa7050 2024-03-23T23:04:42.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.997+0000 7f2061400640 1 --2- 172.21.15.12:0/2558393168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561901facbb0 0x561901fb6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:42.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.998+0000 7f2060a00640 1 --2- 172.21.15.12:0/2558393168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561901fa7050 0x561901fb58b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2054014010 tx=0x7f2054019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c11f3420cabd7fe server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.998+0000 7f204a000640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:04:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.998+0000 7f204a000640 1 -- 172.21.15.12:0/2558393168 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2054045060 con 0x561901fa7050 2024-03-23T23:04:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.998+0000 7f204a000640 1 -- 172.21.15.12:0/2558393168 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2054019a90 con 0x561901fa7050 2024-03-23T23:04:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.998+0000 7f20661c7740 1 -- 172.21.15.12:0/2558393168 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2054019800 con 0x561901fa7050 2024-03-23T23:04:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.998+0000 7f204a000640 1 -- 172.21.15.12:0/2558393168 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2054028040 con 0x561901fa7050 2024-03-23T23:04:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.998+0000 7f20661c7740 10 client.15876 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.998+0000 7f20661c7740 1 -- 172.21.15.12:0/2558393168 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561901fb8700 con 0x561901fa7050 2024-03-23T23:04:42.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.999+0000 7f204a000640 1 -- 172.21.15.12:0/2558393168 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2054019800 con 0x561901fa7050 2024-03-23T23:04:42.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.999+0000 7f204a000640 1 -- 172.21.15.12:0/2558393168 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f2054003990 con 0x561901fa7050 2024-03-23T23:04:42.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.999+0000 7f20661c7740 10 client.15876 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:42.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.999+0000 7f20661c7740 10 client.15876 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:04:42.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.999+0000 7f20661c7740 20 client.15876 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:42.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.999+0000 7f20661c7740 1 --2- 172.21.15.12:0/2558393168 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561902018990 0x56190201adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.999+0000 7f20661c7740 4 client.15876 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:04:42.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.999+0000 7f20661c7740 1 -- 172.21.15.12:0/2558393168 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561901f13ae0 con 0x561902018990 2024-03-23T23:04:42.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:41.999+0000 7f205be00640 1 --2- 172.21.15.12:0/2558393168 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561902018990 0x56190201adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.000+0000 7f205be00640 1 --2- 172.21.15.12:0/2558393168 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561902018990 0x56190201adb0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5013d5b2cc17b77b server_cookie=8ced84924781644d in_seq=0 out_seq=0 2024-03-23T23:04:42.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.000+0000 7f204a000640 10 client.15876 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:42.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.000+0000 7f204a000640 1 -- 172.21.15.12:0/2558393168 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x561901f13ae0 con 0x561902018990 2024-03-23T23:04:42.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.000+0000 7f204a000640 10 client.15876 handle_command_reply: tid=0 2024-03-23T23:04:42.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.000+0000 7f20661c7740 1 client.15876 shutdown 2024-03-23T23:04:42.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.000+0000 7f20661c7740 1 -- 172.21.15.12:0/2558393168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561901fa7050 msgr2=0x561901fb58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:42.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.000+0000 7f20661c7740 1 --2- 172.21.15.12:0/2558393168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561901fa7050 0x561901fb58b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2054014010 tx=0x7f2054019680 comp rx=0 tx=0).stop 2024-03-23T23:04:42.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.000+0000 7f20661c7740 1 -- 172.21.15.12:0/2558393168 shutdown_connections 2024-03-23T23:04:42.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.000+0000 7f20661c7740 1 --2- 172.21.15.12:0/2558393168 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561902018990 0x56190201adb0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.000+0000 7f20661c7740 1 --2- 172.21.15.12:0/2558393168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561901facbb0 0x561901fb6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.000+0000 7f20661c7740 1 --2- 172.21.15.12:0/2558393168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561901fa7a40 0x561901fb5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.000+0000 7f20661c7740 1 --2- 172.21.15.12:0/2558393168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561901fa7050 0x561901fb58b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.000+0000 7f204a000640 0 client.15876 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:42.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.000+0000 7f20661c7740 1 -- 172.21.15.12:0/2558393168 >> 172.21.15.12:0/2558393168 conn(0x561901f6f200 msgr2=0x561901fa1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:42.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.000+0000 7f20661c7740 1 -- 172.21.15.12:0/2558393168 shutdown_connections 2024-03-23T23:04:42.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.000+0000 7f20661c7740 1 -- 172.21.15.12:0/2558393168 wait complete. 2024-03-23T23:04:42.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.000+0000 7f20661c7740 20 client.15876 trim_cache size 0 max 16384 2024-03-23T23:04:42.017 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:04:42.017 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-03-23T23:04:42.017 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-03-23T23:04:42.017 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:04:42.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.001+0000 7f2061e00640 1 -- 172.21.15.12:0/3292763677 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f20400a2880 msgr2=0x7f20400a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:42.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.001+0000 7f2061e00640 1 --2- 172.21.15.12:0/3292763677 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f20400a2880 0x7f20400a4d40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f204c036040 tx=0x7f204c02e040 comp rx=0 tx=0).stop 2024-03-23T23:04:42.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.001+0000 7f2061e00640 1 -- 172.21.15.12:0/3292763677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f205c104990 msgr2=0x7f205c199f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:42.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.001+0000 7f2061e00640 1 --2- 172.21.15.12:0/3292763677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f205c104990 0x7f205c199f70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f204400ec80 tx=0x7f204400c6a0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.001+0000 7f2061e00640 1 -- 172.21.15.12:0/3292763677 shutdown_connections 2024-03-23T23:04:42.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.001+0000 7f2061e00640 1 --2- 172.21.15.12:0/3292763677 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f20400a2880 0x7f20400a4d40 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.001+0000 7f2061e00640 1 --2- 172.21.15.12:0/3292763677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f205c105350 0x7f205c1a0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.001+0000 7f2061e00640 1 --2- 172.21.15.12:0/3292763677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f205c104990 0x7f205c199f70 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.001+0000 7f2061e00640 1 --2- 172.21.15.12:0/3292763677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f205c1037a0 0x7f205c199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.001+0000 7f2061e00640 1 -- 172.21.15.12:0/3292763677 >> 172.21.15.12:0/3292763677 conn(0x7f205c0fed10 msgr2=0x7f205c107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:42.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.001+0000 7f2061e00640 1 -- 172.21.15.12:0/3292763677 shutdown_connections 2024-03-23T23:04:42.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.001+0000 7f2061e00640 1 -- 172.21.15.12:0/3292763677 wait complete. 2024-03-23T23:04:42.022 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T23:04:42.022 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:04:42.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.246+0000 7f1336800640 1 -- 172.21.15.12:0/3262443724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13301052a0 msgr2=0x7f133010bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:42.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.246+0000 7f1336800640 1 --2- 172.21.15.12:0/3262443724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13301052a0 0x7f133010bb30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1328009900 tx=0x7f132802bc30 comp rx=0 tx=0).stop 2024-03-23T23:04:42.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.246+0000 7f1336800640 1 -- 172.21.15.12:0/3262443724 shutdown_connections 2024-03-23T23:04:42.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.246+0000 7f1336800640 1 --2- 172.21.15.12:0/3262443724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13301052a0 0x7f133010bb30 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.246+0000 7f1336800640 1 --2- 172.21.15.12:0/3262443724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13301048e0 0x7f1330104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.246+0000 7f1336800640 1 --2- 172.21.15.12:0/3262443724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13301036f0 0x7f1330103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.246+0000 7f1336800640 1 -- 172.21.15.12:0/3262443724 >> 172.21.15.12:0/3262443724 conn(0x7f13300fec80 msgr2=0x7f13301010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:42.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.246+0000 7f1336800640 1 -- 172.21.15.12:0/3262443724 shutdown_connections 2024-03-23T23:04:42.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.246+0000 7f1336800640 1 -- 172.21.15.12:0/3262443724 wait complete. 2024-03-23T23:04:42.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.248+0000 7f1336800640 1 Processor -- start 2024-03-23T23:04:42.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.248+0000 7f1336800640 1 -- start start 2024-03-23T23:04:42.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.248+0000 7f1336800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13301036f0 0x7f1330199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.248+0000 7f1336800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13301048e0 0x7f1330199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.248+0000 7f1336800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13301052a0 0x7f13301a0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.248+0000 7f1336800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133010f350 con 0x7f13301048e0 2024-03-23T23:04:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.248+0000 7f1336800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133010f1d0 con 0x7f13301052a0 2024-03-23T23:04:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.248+0000 7f1336800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133010f4d0 con 0x7f13301036f0 2024-03-23T23:04:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.249+0000 7f1335400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13301036f0 0x7f1330199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.249+0000 7f1335400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13301036f0 0x7f1330199960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54938/0 (socket says 172.21.15.12:54938) 2024-03-23T23:04:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.249+0000 7f1335400640 1 -- 172.21.15.12:0/4138086759 learned_addr learned my addr 172.21.15.12:0/4138086759 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.249+0000 7f1334a00640 1 --2- 172.21.15.12:0/4138086759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13301048e0 0x7f1330199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.249+0000 7f1335e00640 1 --2- 172.21.15.12:0/4138086759 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13301052a0 0x7f13301a0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.249+0000 7f1335400640 1 -- 172.21.15.12:0/4138086759 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13301052a0 msgr2=0x7f13301a0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.249+0000 7f1335400640 1 --2- 172.21.15.12:0/4138086759 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13301052a0 0x7f13301a0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.249+0000 7f1335400640 1 -- 172.21.15.12:0/4138086759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13301048e0 msgr2=0x7f1330199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.249+0000 7f1335400640 1 --2- 172.21.15.12:0/4138086759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13301048e0 0x7f1330199ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.249+0000 7f1335400640 1 -- 172.21.15.12:0/4138086759 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13301a1580 con 0x7f13301036f0 2024-03-23T23:04:42.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.249+0000 7f1334a00640 1 --2- 172.21.15.12:0/4138086759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13301048e0 0x7f1330199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:42.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.249+0000 7f1335400640 1 --2- 172.21.15.12:0/4138086759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13301036f0 0x7f1330199960 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f132400ebc0 tx=0x7f132400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:42.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.250+0000 7f132e000640 1 -- 172.21.15.12:0/4138086759 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1324003b70 con 0x7f13301036f0 2024-03-23T23:04:42.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.250+0000 7f132e000640 1 -- 172.21.15.12:0/4138086759 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1324003d10 con 0x7f13301036f0 2024-03-23T23:04:42.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.250+0000 7f1336800640 1 -- 172.21.15.12:0/4138086759 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13301a1870 con 0x7f13301036f0 2024-03-23T23:04:42.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.250+0000 7f132e000640 1 -- 172.21.15.12:0/4138086759 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1324019980 con 0x7f13301036f0 2024-03-23T23:04:42.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.250+0000 7f1336800640 1 -- 172.21.15.12:0/4138086759 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13301a1dc0 con 0x7f13301036f0 2024-03-23T23:04:42.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.251+0000 7f1336800640 1 -- 172.21.15.12:0/4138086759 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f133010a9b0 con 0x7f13301036f0 2024-03-23T23:04:42.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.253+0000 7f132e000640 1 -- 172.21.15.12:0/4138086759 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1324019b20 con 0x7f13301036f0 2024-03-23T23:04:42.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.254+0000 7f132e000640 1 --2- 172.21.15.12:0/4138086759 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f13180a2800 0x7f13180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.254+0000 7f132e000640 1 -- 172.21.15.12:0/4138086759 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f13240cf290 con 0x7f13301036f0 2024-03-23T23:04:42.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.254+0000 7f1334a00640 1 --2- 172.21.15.12:0/4138086759 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f13180a2800 0x7f13180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.255+0000 7f1334a00640 1 --2- 172.21.15.12:0/4138086759 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f13180a2800 0x7f13180a4cc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f133019ae70 tx=0x7f131c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:42.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.258+0000 7f132e000640 1 -- 172.21.15.12:0/4138086759 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f132409c2e0 con 0x7f13301036f0 2024-03-23T23:04:42.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.504+0000 7f1336800640 1 -- 172.21.15.12:0/4138086759 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f133002d070 con 0x7f13301036f0 2024-03-23T23:04:42.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.505+0000 7f132e000640 1 -- 172.21.15.12:0/4138086759 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f132409bbe0 con 0x7f13301036f0 2024-03-23T23:04:42.504 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:04:42.504 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:04:42.504 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:04:42.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.507+0000 7f1336800640 1 -- 172.21.15.12:0/4138086759 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f13180a2800 msgr2=0x7f13180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:42.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.507+0000 7f1336800640 1 --2- 172.21.15.12:0/4138086759 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f13180a2800 0x7f13180a4cc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f133019ae70 tx=0x7f131c005f90 comp rx=0 tx=0).stop 2024-03-23T23:04:42.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.507+0000 7f1336800640 1 -- 172.21.15.12:0/4138086759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13301036f0 msgr2=0x7f1330199960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:42.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.507+0000 7f1336800640 1 --2- 172.21.15.12:0/4138086759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13301036f0 0x7f1330199960 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f132400ebc0 tx=0x7f132400c6a0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.507+0000 7f1336800640 1 -- 172.21.15.12:0/4138086759 shutdown_connections 2024-03-23T23:04:42.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.507+0000 7f1336800640 1 --2- 172.21.15.12:0/4138086759 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f13180a2800 0x7f13180a4cc0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.508+0000 7f1336800640 1 --2- 172.21.15.12:0/4138086759 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13301052a0 0x7f13301a0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.508+0000 7f1336800640 1 --2- 172.21.15.12:0/4138086759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13301048e0 0x7f1330199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.508+0000 7f1336800640 1 --2- 172.21.15.12:0/4138086759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13301036f0 0x7f1330199960 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.508+0000 7f1336800640 1 -- 172.21.15.12:0/4138086759 >> 172.21.15.12:0/4138086759 conn(0x7f13300fec80 msgr2=0x7f1330107b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:42.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.508+0000 7f1336800640 1 -- 172.21.15.12:0/4138086759 shutdown_connections 2024-03-23T23:04:42.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.508+0000 7f1336800640 1 -- 172.21.15.12:0/4138086759 wait complete. 2024-03-23T23:04:42.527 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:04:42.527 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-23T23:04:42.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.755+0000 7f6a56e00640 1 -- 172.21.15.12:0/3103895072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a500756e0 msgr2=0x7f6a50073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:42.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.755+0000 7f6a56e00640 1 --2- 172.21.15.12:0/3103895072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a500756e0 0x7f6a50073b30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6a44009900 tx=0x7f6a4402bc30 comp rx=0 tx=0).stop 2024-03-23T23:04:42.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.755+0000 7f6a56e00640 1 -- 172.21.15.12:0/3103895072 shutdown_connections 2024-03-23T23:04:42.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.755+0000 7f6a56e00640 1 --2- 172.21.15.12:0/3103895072 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6a5010db50 0x7f6a5010ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.755+0000 7f6a56e00640 1 --2- 172.21.15.12:0/3103895072 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a50074100 0x7f6a50074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.755+0000 7f6a56e00640 1 --2- 172.21.15.12:0/3103895072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a500756e0 0x7f6a50073b30 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.755+0000 7f6a56e00640 1 -- 172.21.15.12:0/3103895072 >> 172.21.15.12:0/3103895072 conn(0x7f6a500fcb10 msgr2=0x7f6a500fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:42.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.755+0000 7f6a56e00640 1 -- 172.21.15.12:0/3103895072 shutdown_connections 2024-03-23T23:04:42.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.755+0000 7f6a56e00640 1 -- 172.21.15.12:0/3103895072 wait complete. 2024-03-23T23:04:42.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.756+0000 7f6a56e00640 1 Processor -- start 2024-03-23T23:04:42.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.756+0000 7f6a56e00640 1 -- start start 2024-03-23T23:04:42.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.757+0000 7f6a56e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a50074100 0x7f6a5019de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.757+0000 7f6a56e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a500756e0 0x7f6a5019e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.757+0000 7f6a56e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6a5010db50 0x7f6a501a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.757+0000 7f6a56e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a50113820 con 0x7f6a500756e0 2024-03-23T23:04:42.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.757+0000 7f6a56e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a501136a0 con 0x7f6a5010db50 2024-03-23T23:04:42.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.757+0000 7f6a56e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a501139a0 con 0x7f6a50074100 2024-03-23T23:04:42.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.757+0000 7f6a55a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a50074100 0x7f6a5019de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.757+0000 7f6a55a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a50074100 0x7f6a5019de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54952/0 (socket says 172.21.15.12:54952) 2024-03-23T23:04:42.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.757+0000 7f6a55a00640 1 -- 172.21.15.12:0/868665571 learned_addr learned my addr 172.21.15.12:0/868665571 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:42.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.757+0000 7f6a55000640 1 --2- 172.21.15.12:0/868665571 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a500756e0 0x7f6a5019e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.757+0000 7f6a56400640 1 --2- 172.21.15.12:0/868665571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6a5010db50 0x7f6a501a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.757+0000 7f6a55a00640 1 -- 172.21.15.12:0/868665571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6a5010db50 msgr2=0x7f6a501a5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:42.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.757+0000 7f6a55a00640 1 --2- 172.21.15.12:0/868665571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6a5010db50 0x7f6a501a5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.757+0000 7f6a55a00640 1 -- 172.21.15.12:0/868665571 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a500756e0 msgr2=0x7f6a5019e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:42.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.757+0000 7f6a55a00640 1 --2- 172.21.15.12:0/868665571 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a500756e0 0x7f6a5019e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.757+0000 7f6a55a00640 1 -- 172.21.15.12:0/868665571 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a501a5a80 con 0x7f6a50074100 2024-03-23T23:04:42.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.758+0000 7f6a55a00640 1 --2- 172.21.15.12:0/868665571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a50074100 0x7f6a5019de60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6a44005ab0 tx=0x7f6a4402c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:42.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.758+0000 7f6a4e000640 1 -- 172.21.15.12:0/868665571 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a44041070 con 0x7f6a50074100 2024-03-23T23:04:42.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.758+0000 7f6a4e000640 1 -- 172.21.15.12:0/868665571 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a44004450 con 0x7f6a50074100 2024-03-23T23:04:42.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.758+0000 7f6a56e00640 1 -- 172.21.15.12:0/868665571 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a501a5d10 con 0x7f6a50074100 2024-03-23T23:04:42.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.758+0000 7f6a4e000640 1 -- 172.21.15.12:0/868665571 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a44003da0 con 0x7f6a50074100 2024-03-23T23:04:42.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.758+0000 7f6a56e00640 1 -- 172.21.15.12:0/868665571 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a501a61d0 con 0x7f6a50074100 2024-03-23T23:04:42.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.761+0000 7f6a5b164740 1 Processor -- start 2024-03-23T23:04:42.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.761+0000 7f6a5b164740 1 -- start start 2024-03-23T23:04:42.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.762+0000 7f6a4e000640 1 -- 172.21.15.12:0/868665571 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6a44044070 con 0x7f6a50074100 2024-03-23T23:04:42.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.762+0000 7f6a5b164740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b319f0 0x556f36b31e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.762+0000 7f6a5b164740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f36b32bc0 0x556f36b33040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.762+0000 7f6a5b164740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f36b33580 0x556f36b39e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.762+0000 7f6a5b164740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f36a962c0 con 0x556f36b32bc0 2024-03-23T23:04:42.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.762+0000 7f6a5b164740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f36a3ec90 con 0x556f36b33580 2024-03-23T23:04:42.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.762+0000 7f6a5b164740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f36a92360 con 0x556f36b319f0 2024-03-23T23:04:42.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.762+0000 7f6a55a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f36b32bc0 0x556f36b33040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.762+0000 7f6a56400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b319f0 0x556f36b31e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.762+0000 7f6a55000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f36b33580 0x556f36b39e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.762+0000 7f6a56400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b319f0 0x556f36b31e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54964/0 (socket says 172.21.15.12:54964) 2024-03-23T23:04:42.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.762+0000 7f6a56400640 1 -- 172.21.15.12:0/4045286910 learned_addr learned my addr 172.21.15.12:0/4045286910 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:42.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.762+0000 7f6a56400640 1 -- 172.21.15.12:0/4045286910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f36b33580 msgr2=0x556f36b39e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:42.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.762+0000 7f6a56400640 1 --2- 172.21.15.12:0/4045286910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f36b33580 0x556f36b39e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.762+0000 7f6a56400640 1 -- 172.21.15.12:0/4045286910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f36b32bc0 msgr2=0x556f36b33040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:42.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.762+0000 7f6a56400640 1 --2- 172.21.15.12:0/4045286910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f36b32bc0 0x556f36b33040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.763+0000 7f6a56400640 1 -- 172.21.15.12:0/4045286910 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f36a63080 con 0x556f36b319f0 2024-03-23T23:04:42.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.763+0000 7f6a55000640 1 --2- 172.21.15.12:0/4045286910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f36b33580 0x556f36b39e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:42.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.763+0000 7f6a56400640 1 --2- 172.21.15.12:0/4045286910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b319f0 0x556f36b31e10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6a48009fd0 tx=0x7f6a480118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b6e7c71b7a015cf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:42.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.763+0000 7f6a42a00640 1 -- 172.21.15.12:0/4045286910 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a48004d20 con 0x556f36b319f0 2024-03-23T23:04:42.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.763+0000 7f6a42a00640 1 -- 172.21.15.12:0/4045286910 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a48004050 con 0x556f36b319f0 2024-03-23T23:04:42.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.763+0000 7f6a42a00640 1 -- 172.21.15.12:0/4045286910 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a48005220 con 0x556f36b319f0 2024-03-23T23:04:42.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.763+0000 7f6a4e000640 1 --2- 172.21.15.12:0/868665571 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6a380a2880 0x7f6a380a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.764+0000 7f6a4e000640 1 -- 172.21.15.12:0/868665571 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f6a440f0530 con 0x7f6a50074100 2024-03-23T23:04:42.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.764+0000 7f6a55a00640 1 --2- 172.21.15.12:0/868665571 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6a380a2880 0x7f6a380a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.764+0000 7f6a5b164740 1 -- 172.21.15.12:0/4045286910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b319f0 msgr2=0x556f36b31e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:42.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.764+0000 7f6a5b164740 1 --2- 172.21.15.12:0/4045286910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b319f0 0x556f36b31e10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6a48009fd0 tx=0x7f6a480118d0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.764+0000 7f6a5b164740 1 -- 172.21.15.12:0/4045286910 shutdown_connections 2024-03-23T23:04:42.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.764+0000 7f6a5b164740 1 --2- 172.21.15.12:0/4045286910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f36b33580 0x556f36b39e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.764+0000 7f6a5b164740 1 --2- 172.21.15.12:0/4045286910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f36b32bc0 0x556f36b33040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.764+0000 7f6a5b164740 1 --2- 172.21.15.12:0/4045286910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b319f0 0x556f36b31e10 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.764+0000 7f6a5b164740 1 -- 172.21.15.12:0/4045286910 >> 172.21.15.12:0/4045286910 conn(0x556f36b2cde0 msgr2=0x556f36b2d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:42.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.764+0000 7f6a5b164740 1 -- 172.21.15.12:0/4045286910 shutdown_connections 2024-03-23T23:04:42.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.764+0000 7f6a5b164740 1 -- 172.21.15.12:0/4045286910 wait complete. 2024-03-23T23:04:42.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.764+0000 7f6a55a00640 1 --2- 172.21.15.12:0/868665571 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6a380a2880 0x7f6a380a4d40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6a440bcf20 tx=0x7f6a440f6060 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:42.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.765+0000 7f6a5b164740 1 -- start start 2024-03-23T23:04:42.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.765+0000 7f6a5b164740 1 Processor -- start 2024-03-23T23:04:42.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.765+0000 7f6a5b164740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f36b319f0 0x556f36b3bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.765+0000 7f6a5b164740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f36b32bc0 0x556f36b3c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.765+0000 7f6a5b164740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b33580 0x556f36b431e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.765+0000 7f6a5b164740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f36b3a590 con 0x556f36b319f0 2024-03-23T23:04:42.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.765+0000 7f6a5b164740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f36a520d0 con 0x556f36b32bc0 2024-03-23T23:04:42.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.765+0000 7f6a5b164740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f36b3a710 con 0x556f36b33580 2024-03-23T23:04:42.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.765+0000 7f6a5b164740 10 client.0 _ll_register_callbacks cb 0x556f36a56e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:42.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.766+0000 7f6a55000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f36b32bc0 0x556f36b3c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.766+0000 7f6a55000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f36b32bc0 0x556f36b3c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48324/0 (socket says 172.21.15.12:48324) 2024-03-23T23:04:42.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.766+0000 7f6a55000640 1 -- 172.21.15.12:0/4154600285 learned_addr learned my addr 172.21.15.12:0/4154600285 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:42.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.766+0000 7f6a56400640 1 --2- 172.21.15.12:0/4154600285 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f36b319f0 0x556f36b3bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.766+0000 7f6a56400640 1 --2- 172.21.15.12:0/4154600285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b33580 0x556f36b431e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.766+0000 7f6a55000640 1 -- 172.21.15.12:0/4154600285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b33580 msgr2=0x556f36b431e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:42.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.766+0000 7f6a55000640 1 --2- 172.21.15.12:0/4154600285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b33580 0x556f36b431e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.766+0000 7f6a55000640 1 -- 172.21.15.12:0/4154600285 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f36b319f0 msgr2=0x556f36b3bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:42.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.766+0000 7f6a55000640 1 --2- 172.21.15.12:0/4154600285 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f36b319f0 0x556f36b3bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.766+0000 7f6a55000640 1 -- 172.21.15.12:0/4154600285 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556f367aa3f0 con 0x556f36b32bc0 2024-03-23T23:04:42.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.766+0000 7f6a56400640 1 --2- 172.21.15.12:0/4154600285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b33580 0x556f36b431e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:04:42.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.766+0000 7f6a55000640 1 --2- 172.21.15.12:0/4154600285 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f36b32bc0 0x556f36b3c1a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c012700 tx=0x7f6a3c012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bd689d3148bd251 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:42.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.766+0000 7f6a37400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:04:42.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.766+0000 7f6a37400640 1 -- 172.21.15.12:0/4154600285 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c004030 con 0x556f36b32bc0 2024-03-23T23:04:42.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.766+0000 7f6a37400640 1 -- 172.21.15.12:0/4154600285 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f6a3c0041d0 con 0x556f36b32bc0 2024-03-23T23:04:42.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.766+0000 7f6a5b164740 1 -- 172.21.15.12:0/4154600285 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6a48004050 con 0x556f36b32bc0 2024-03-23T23:04:42.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.767+0000 7f6a37400640 1 -- 172.21.15.12:0/4154600285 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a3c0043d0 con 0x556f36b32bc0 2024-03-23T23:04:42.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.767+0000 7f6a5b164740 10 client.26027 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:42.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.767+0000 7f6a5b164740 1 -- 172.21.15.12:0/4154600285 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556f368747d0 con 0x556f36b32bc0 2024-03-23T23:04:42.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.767+0000 7f6a37400640 1 -- 172.21.15.12:0/4154600285 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a48004050 con 0x556f36b32bc0 2024-03-23T23:04:42.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.767+0000 7f6a37400640 1 -- 172.21.15.12:0/4154600285 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f6a3c003b80 con 0x556f36b32bc0 2024-03-23T23:04:42.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.767+0000 7f6a5b164740 10 client.26027 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:42.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.767+0000 7f6a5b164740 10 client.26027 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:04:42.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.767+0000 7f6a5b164740 20 client.26027 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:42.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.767+0000 7f6a5b164740 1 --2- 172.21.15.12:0/4154600285 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x556f36b476c0 0x556f36b49b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.767+0000 7f6a5b164740 4 client.26027 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:04:42.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.767+0000 7f6a5b164740 1 -- 172.21.15.12:0/4154600285 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556f36aa3800 con 0x556f36b476c0 2024-03-23T23:04:42.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.768+0000 7f6a56400640 1 --2- 172.21.15.12:0/4154600285 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x556f36b476c0 0x556f36b49b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.768+0000 7f6a56400640 1 --2- 172.21.15.12:0/4154600285 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x556f36b476c0 0x556f36b49b00 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a0d140b7109d0c0c server_cookie=63cde29534776cb0 in_seq=0 out_seq=0 2024-03-23T23:04:42.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.768+0000 7f6a37400640 10 client.26027 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:04:42.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.769+0000 7f6a37400640 1 -- 172.21.15.12:0/4154600285 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556f36aa3800 con 0x556f36b476c0 2024-03-23T23:04:42.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.769+0000 7f6a37400640 10 client.26027 handle_command_reply: tid=0 2024-03-23T23:04:42.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.769+0000 7f6a5b164740 1 client.26027 shutdown 2024-03-23T23:04:42.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.770+0000 7f6a5b164740 1 -- 172.21.15.12:0/4154600285 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f36b32bc0 msgr2=0x556f36b3c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:42.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.770+0000 7f6a5b164740 1 --2- 172.21.15.12:0/4154600285 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f36b32bc0 0x556f36b3c1a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c012700 tx=0x7f6a3c012bc0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.770+0000 7f6a5b164740 1 -- 172.21.15.12:0/4154600285 shutdown_connections 2024-03-23T23:04:42.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.770+0000 7f6a5b164740 1 --2- 172.21.15.12:0/4154600285 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x556f36b476c0 0x556f36b49b00 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.770+0000 7f6a5b164740 1 --2- 172.21.15.12:0/4154600285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b33580 0x556f36b431e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.770+0000 7f6a5b164740 1 --2- 172.21.15.12:0/4154600285 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f36b32bc0 0x556f36b3c1a0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.770+0000 7f6a5b164740 1 --2- 172.21.15.12:0/4154600285 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f36b319f0 0x556f36b3bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.770+0000 7f6a37400640 0 client.26027 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:04:42.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.770+0000 7f6a5b164740 1 -- 172.21.15.12:0/4154600285 >> 172.21.15.12:0/4154600285 conn(0x556f36b2cde0 msgr2=0x556f36b2c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:42.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.770+0000 7f6a5b164740 1 -- 172.21.15.12:0/4154600285 shutdown_connections 2024-03-23T23:04:42.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.770+0000 7f6a5b164740 1 -- 172.21.15.12:0/4154600285 wait complete. 2024-03-23T23:04:42.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.770+0000 7f6a5b164740 20 client.26027 trim_cache size 0 max 16384 2024-03-23T23:04:42.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.783+0000 7f6a5b164740 1 Processor -- start 2024-03-23T23:04:42.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.784+0000 7f6a5b164740 1 -- start start 2024-03-23T23:04:42.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.784+0000 7f6a5b164740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f36b319f0 0x556f36b35df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.784+0000 7f6a5b164740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b32bc0 0x556f36b431a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.784+0000 7f6a5b164740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f36b33580 0x556f36b36330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.784+0000 7f6a5b164740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a501b1320 con 0x556f36b33580 2024-03-23T23:04:42.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.784+0000 7f6a5b164740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a501aee20 con 0x556f36b319f0 2024-03-23T23:04:42.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.784+0000 7f6a5b164740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a501acec0 con 0x556f36b32bc0 2024-03-23T23:04:42.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.784+0000 7f6a56400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f36b319f0 0x556f36b35df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.784+0000 7f6a56400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f36b319f0 0x556f36b35df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48330/0 (socket says 172.21.15.12:48330) 2024-03-23T23:04:42.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.784+0000 7f6a56400640 1 -- 172.21.15.12:0/3568882139 learned_addr learned my addr 172.21.15.12:0/3568882139 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:42.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.784+0000 7f6a56400640 1 --2- 172.21.15.12:0/3568882139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f36b33580 0x556f36b36330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.784+0000 7f6a55000640 1 --2- 172.21.15.12:0/3568882139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b32bc0 0x556f36b431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.784+0000 7f6a56400640 1 -- 172.21.15.12:0/3568882139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b32bc0 msgr2=0x556f36b431a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:42.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.784+0000 7f6a56400640 1 --2- 172.21.15.12:0/3568882139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b32bc0 0x556f36b431a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.784+0000 7f6a56400640 1 -- 172.21.15.12:0/3568882139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f36b33580 msgr2=0x556f36b36330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:42.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.784+0000 7f6a56400640 1 --2- 172.21.15.12:0/3568882139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f36b33580 0x556f36b36330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.785+0000 7f6a56400640 1 -- 172.21.15.12:0/3568882139 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f36b2cb50 con 0x556f36b319f0 2024-03-23T23:04:42.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.785+0000 7f6a55000640 1 --2- 172.21.15.12:0/3568882139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b32bc0 0x556f36b431a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:42.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.785+0000 7f6a56400640 1 --2- 172.21.15.12:0/3568882139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f36b319f0 0x556f36b35df0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6a48014040 tx=0x7f6a48009a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7265660df48a67a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:42.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.785+0000 7f6a42000640 1 -- 172.21.15.12:0/3568882139 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a48004050 con 0x556f36b319f0 2024-03-23T23:04:42.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.785+0000 7f6a42000640 1 -- 172.21.15.12:0/3568882139 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a480046d0 con 0x556f36b319f0 2024-03-23T23:04:42.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.785+0000 7f6a42000640 1 -- 172.21.15.12:0/3568882139 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4800fc70 con 0x556f36b319f0 2024-03-23T23:04:42.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.785+0000 7f6a5b164740 1 -- 172.21.15.12:0/3568882139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f36b319f0 msgr2=0x556f36b35df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:42.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.786+0000 7f6a5b164740 1 --2- 172.21.15.12:0/3568882139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f36b319f0 0x556f36b35df0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6a48014040 tx=0x7f6a48009a50 comp rx=0 tx=0).stop 2024-03-23T23:04:42.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.786+0000 7f6a5b164740 1 -- 172.21.15.12:0/3568882139 shutdown_connections 2024-03-23T23:04:42.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.786+0000 7f6a5b164740 1 --2- 172.21.15.12:0/3568882139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f36b33580 0x556f36b36330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.786+0000 7f6a5b164740 1 --2- 172.21.15.12:0/3568882139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b32bc0 0x556f36b431a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.786+0000 7f6a5b164740 1 --2- 172.21.15.12:0/3568882139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f36b319f0 0x556f36b35df0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.786+0000 7f6a5b164740 1 -- 172.21.15.12:0/3568882139 >> 172.21.15.12:0/3568882139 conn(0x556f36b2cde0 msgr2=0x556f36b308d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:42.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.786+0000 7f6a5b164740 1 -- 172.21.15.12:0/3568882139 shutdown_connections 2024-03-23T23:04:42.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.786+0000 7f6a5b164740 1 -- 172.21.15.12:0/3568882139 wait complete. 2024-03-23T23:04:42.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.786+0000 7f6a5b164740 1 -- start start 2024-03-23T23:04:42.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.786+0000 7f6a5b164740 1 Processor -- start 2024-03-23T23:04:42.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.787+0000 7f6a5b164740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b319f0 0x556f36b411b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.787+0000 7f6a5b164740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f36b32bc0 0x556f36b416f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.787+0000 7f6a5b164740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f36b33580 0x556f36b42700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.787+0000 7f6a5b164740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a501ad4c0 con 0x556f36b32bc0 2024-03-23T23:04:42.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.787+0000 7f6a5b164740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a501ae7c0 con 0x556f36b33580 2024-03-23T23:04:42.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.787+0000 7f6a5b164740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a501adb00 con 0x556f36b319f0 2024-03-23T23:04:42.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.787+0000 7f6a5b164740 10 client.0 _ll_register_callbacks cb 0x7f6a3c009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:42.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.787+0000 7f6a55000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f36b32bc0 0x556f36b416f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.787+0000 7f6a56400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b319f0 0x556f36b411b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.787+0000 7f6a56400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b319f0 0x556f36b411b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54992/0 (socket says 172.21.15.12:54992) 2024-03-23T23:04:42.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.787+0000 7f6a56400640 1 -- 172.21.15.12:0/913774070 learned_addr learned my addr 172.21.15.12:0/913774070 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:42.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.787+0000 7f6a56400640 1 --2- 172.21.15.12:0/913774070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f36b33580 0x556f36b42700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.787+0000 7f6a55000640 1 -- 172.21.15.12:0/913774070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b319f0 msgr2=0x556f36b411b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:42.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.787+0000 7f6a55000640 1 --2- 172.21.15.12:0/913774070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b319f0 0x556f36b411b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.787+0000 7f6a55000640 1 -- 172.21.15.12:0/913774070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f36b33580 msgr2=0x556f36b42700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:42.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.787+0000 7f6a55000640 1 --2- 172.21.15.12:0/913774070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f36b33580 0x556f36b42700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.787+0000 7f6a55000640 1 -- 172.21.15.12:0/913774070 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556f36b42cc0 con 0x556f36b32bc0 2024-03-23T23:04:42.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.787+0000 7f6a56400640 1 --2- 172.21.15.12:0/913774070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b319f0 0x556f36b411b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:42.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.788+0000 7f6a55000640 1 --2- 172.21.15.12:0/913774070 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f36b32bc0 0x556f36b416f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c009900 tx=0x7f6a3c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84edb3208f61bcca server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:42.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.788+0000 7f6a37400640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:04:42.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.788+0000 7f6a37400640 1 -- 172.21.15.12:0/913774070 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c010900 con 0x556f36b32bc0 2024-03-23T23:04:42.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.788+0000 7f6a37400640 1 -- 172.21.15.12:0/913774070 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f6a3c018040 con 0x556f36b32bc0 2024-03-23T23:04:42.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.788+0000 7f6a5b164740 1 -- 172.21.15.12:0/913774070 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6a480046d0 con 0x556f36b32bc0 2024-03-23T23:04:42.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.788+0000 7f6a37400640 1 -- 172.21.15.12:0/913774070 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a3c010aa0 con 0x556f36b32bc0 2024-03-23T23:04:42.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.788+0000 7f6a5b164740 10 client.15900 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:42.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.788+0000 7f6a5b164740 1 -- 172.21.15.12:0/913774070 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556f36ba3900 con 0x556f36b32bc0 2024-03-23T23:04:42.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.789+0000 7f6a37400640 1 -- 172.21.15.12:0/913774070 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a480046d0 con 0x556f36b32bc0 2024-03-23T23:04:42.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.789+0000 7f6a37400640 1 -- 172.21.15.12:0/913774070 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f6a3c003680 con 0x556f36b32bc0 2024-03-23T23:04:42.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.789+0000 7f6a5b164740 10 client.15900 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:42.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.789+0000 7f6a5b164740 10 client.15900 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:04:42.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.789+0000 7f6a5b164740 20 client.15900 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:42.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.789+0000 7f6a5b164740 1 --2- 172.21.15.12:0/913774070 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x556f36ba4230 0x556f36ba6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:42.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.789+0000 7f6a5b164740 4 client.15900 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:04:42.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.789+0000 7f6a5b164740 1 -- 172.21.15.12:0/913774070 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556f36a9db50 con 0x556f36ba4230 2024-03-23T23:04:42.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.789+0000 7f6a56400640 1 --2- 172.21.15.12:0/913774070 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x556f36ba4230 0x556f36ba6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:42.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.790+0000 7f6a56400640 1 --2- 172.21.15.12:0/913774070 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x556f36ba4230 0x556f36ba6650 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f8aea0c43ec2d786 server_cookie=378fd7d6ab08258 in_seq=0 out_seq=0 2024-03-23T23:04:42.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.790+0000 7f6a37400640 10 client.15900 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:04:42.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.790+0000 7f6a37400640 1 -- 172.21.15.12:0/913774070 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556f36a9db50 con 0x556f36ba4230 2024-03-23T23:04:42.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.790+0000 7f6a37400640 10 client.15900 handle_command_reply: tid=0 2024-03-23T23:04:42.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.790+0000 7f6a5b164740 1 client.15900 shutdown 2024-03-23T23:04:42.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.790+0000 7f6a5b164740 1 -- 172.21.15.12:0/913774070 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f36b32bc0 msgr2=0x556f36b416f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:42.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.790+0000 7f6a5b164740 1 --2- 172.21.15.12:0/913774070 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f36b32bc0 0x556f36b416f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c009900 tx=0x7f6a3c009f80 comp rx=0 tx=0).stop 2024-03-23T23:04:42.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.790+0000 7f6a5b164740 1 -- 172.21.15.12:0/913774070 shutdown_connections 2024-03-23T23:04:42.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.790+0000 7f6a5b164740 1 --2- 172.21.15.12:0/913774070 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x556f36ba4230 0x556f36ba6650 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.790+0000 7f6a5b164740 1 --2- 172.21.15.12:0/913774070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f36b33580 0x556f36b42700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.790+0000 7f6a5b164740 1 --2- 172.21.15.12:0/913774070 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f36b32bc0 0x556f36b416f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.790+0000 7f6a5b164740 1 --2- 172.21.15.12:0/913774070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f36b319f0 0x556f36b411b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.790+0000 7f6a5b164740 1 -- 172.21.15.12:0/913774070 >> 172.21.15.12:0/913774070 conn(0x556f36b2cde0 msgr2=0x556f36b2b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:42.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.790+0000 7f6a37400640 0 client.15900 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:04:42.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.790+0000 7f6a5b164740 1 -- 172.21.15.12:0/913774070 shutdown_connections 2024-03-23T23:04:42.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.790+0000 7f6a5b164740 1 -- 172.21.15.12:0/913774070 wait complete. 2024-03-23T23:04:42.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.790+0000 7f6a5b164740 20 client.15900 trim_cache size 0 max 16384 2024-03-23T23:04:42.794 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:04:42.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.791+0000 7f6a56e00640 1 -- 172.21.15.12:0/868665571 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6a380a2880 msgr2=0x7f6a380a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:42.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.791+0000 7f6a56e00640 1 --2- 172.21.15.12:0/868665571 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6a380a2880 0x7f6a380a4d40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6a440bcf20 tx=0x7f6a440f6060 comp rx=0 tx=0).stop 2024-03-23T23:04:42.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.791+0000 7f6a56e00640 1 -- 172.21.15.12:0/868665571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a50074100 msgr2=0x7f6a5019de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:42.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.791+0000 7f6a56e00640 1 --2- 172.21.15.12:0/868665571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a50074100 0x7f6a5019de60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6a44005ab0 tx=0x7f6a4402c6c0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.791+0000 7f6a56e00640 1 -- 172.21.15.12:0/868665571 shutdown_connections 2024-03-23T23:04:42.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.791+0000 7f6a56e00640 1 --2- 172.21.15.12:0/868665571 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6a380a2880 0x7f6a380a4d40 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.791+0000 7f6a56e00640 1 --2- 172.21.15.12:0/868665571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6a5010db50 0x7f6a501a5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.791+0000 7f6a56e00640 1 --2- 172.21.15.12:0/868665571 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a500756e0 0x7f6a5019e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.791+0000 7f6a56e00640 1 --2- 172.21.15.12:0/868665571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a50074100 0x7f6a5019de60 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:42.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.791+0000 7f6a56e00640 1 -- 172.21.15.12:0/868665571 >> 172.21.15.12:0/868665571 conn(0x7f6a500fcb10 msgr2=0x7f6a50100ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:42.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.791+0000 7f6a56e00640 1 -- 172.21.15.12:0/868665571 shutdown_connections 2024-03-23T23:04:42.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:42.791+0000 7f6a56e00640 1 -- 172.21.15.12:0/868665571 wait complete. 2024-03-23T23:04:42.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:42 smithi012 ceph-mon[31695]: pgmap v477: 97 pgs: 97 active+clean; 1.5 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:04:42.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:42 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/4138086759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:04:42.810 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:04:42.810 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-23T23:04:43.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.023+0000 7f52f2c00640 1 -- 172.21.15.12:0/3779269250 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52ec102770 msgr2=0x7f52ec102bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:43.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.023+0000 7f52f2c00640 1 --2- 172.21.15.12:0/3779269250 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52ec102770 0x7f52ec102bf0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f52d8009900 tx=0x7f52d802bc50 comp rx=0 tx=0).stop 2024-03-23T23:04:43.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.023+0000 7f52f2c00640 1 -- 172.21.15.12:0/3779269250 shutdown_connections 2024-03-23T23:04:43.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.023+0000 7f52f2c00640 1 --2- 172.21.15.12:0/3779269250 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52ec103130 0x7f52ec1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.023+0000 7f52f2c00640 1 --2- 172.21.15.12:0/3779269250 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52ec102770 0x7f52ec102bf0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.023+0000 7f52f2c00640 1 --2- 172.21.15.12:0/3779269250 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52ec101580 0x7f52ec1019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.023+0000 7f52f2c00640 1 -- 172.21.15.12:0/3779269250 >> 172.21.15.12:0/3779269250 conn(0x7f52ec0fcb10 msgr2=0x7f52ec0fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:43.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.023+0000 7f52f2c00640 1 -- 172.21.15.12:0/3779269250 shutdown_connections 2024-03-23T23:04:43.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.023+0000 7f52f2c00640 1 -- 172.21.15.12:0/3779269250 wait complete. 2024-03-23T23:04:43.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.024+0000 7f52f2c00640 1 Processor -- start 2024-03-23T23:04:43.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.024+0000 7f52f2c00640 1 -- start start 2024-03-23T23:04:43.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.025+0000 7f52f2c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52ec101580 0x7f52ec075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.025+0000 7f52f2c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52ec102770 0x7f52ec073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.025+0000 7f52f2c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52ec103130 0x7f52ec073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.025+0000 7f52f2c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec10d230 con 0x7f52ec102770 2024-03-23T23:04:43.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.025+0000 7f52f2c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec10d0b0 con 0x7f52ec101580 2024-03-23T23:04:43.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.025+0000 7f52f2c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec10d3b0 con 0x7f52ec103130 2024-03-23T23:04:43.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.025+0000 7f52f1800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52ec101580 0x7f52ec075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.025+0000 7f52f0e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52ec102770 0x7f52ec073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.025+0000 7f52f1800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52ec101580 0x7f52ec075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48362/0 (socket says 172.21.15.12:48362) 2024-03-23T23:04:43.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.025+0000 7f52f1800640 1 -- 172.21.15.12:0/2187844150 learned_addr learned my addr 172.21.15.12:0/2187844150 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:43.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.025+0000 7f52f2200640 1 --2- 172.21.15.12:0/2187844150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52ec103130 0x7f52ec073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.025+0000 7f52f1800640 1 -- 172.21.15.12:0/2187844150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52ec103130 msgr2=0x7f52ec073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:43.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.025+0000 7f52f1800640 1 --2- 172.21.15.12:0/2187844150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52ec103130 0x7f52ec073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.026+0000 7f52f1800640 1 -- 172.21.15.12:0/2187844150 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52ec102770 msgr2=0x7f52ec073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:43.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.026+0000 7f52f1800640 1 --2- 172.21.15.12:0/2187844150 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52ec102770 0x7f52ec073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.026+0000 7f52f1800640 1 -- 172.21.15.12:0/2187844150 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ec074860 con 0x7f52ec101580 2024-03-23T23:04:43.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.026+0000 7f52f1800640 1 --2- 172.21.15.12:0/2187844150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52ec101580 0x7f52ec075450 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f52e000e9c0 tx=0x7f52e000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:43.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.026+0000 7f52ea000640 1 -- 172.21.15.12:0/2187844150 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e000cde0 con 0x7f52ec101580 2024-03-23T23:04:43.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.026+0000 7f52ea000640 1 -- 172.21.15.12:0/2187844150 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52e0010910 con 0x7f52ec101580 2024-03-23T23:04:43.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.026+0000 7f52ea000640 1 -- 172.21.15.12:0/2187844150 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e00189c0 con 0x7f52ec101580 2024-03-23T23:04:43.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.026+0000 7f52f2c00640 1 -- 172.21.15.12:0/2187844150 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52ec1a1a00 con 0x7f52ec101580 2024-03-23T23:04:43.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.026+0000 7f52f2c00640 1 -- 172.21.15.12:0/2187844150 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52ec1a1f70 con 0x7f52ec101580 2024-03-23T23:04:43.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.031+0000 7f52f6e72740 1 Processor -- start 2024-03-23T23:04:43.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.031+0000 7f52f6e72740 1 -- start start 2024-03-23T23:04:43.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.031+0000 7f52ea000640 1 -- 172.21.15.12:0/2187844150 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f52e0010ac0 con 0x7f52ec101580 2024-03-23T23:04:43.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.031+0000 7f52f6e72740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b67b04e9f0 0x55b67b04ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.031+0000 7f52f6e72740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b04fbc0 0x55b67b050040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.031+0000 7f52f6e72740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b67b050580 0x55b67b056e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.031+0000 7f52f6e72740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b67afb32c0 con 0x55b67b04e9f0 2024-03-23T23:04:43.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.031+0000 7f52f6e72740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b67af5bc90 con 0x55b67b050580 2024-03-23T23:04:43.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.031+0000 7f52f6e72740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b67afaf360 con 0x55b67b04fbc0 2024-03-23T23:04:43.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.032+0000 7f52f1800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b04fbc0 0x55b67b050040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.032+0000 7f52f0e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b67b050580 0x55b67b056e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.032+0000 7f52f2200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b67b04e9f0 0x55b67b04ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.032+0000 7f52f1800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b04fbc0 0x55b67b050040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55026/0 (socket says 172.21.15.12:55026) 2024-03-23T23:04:43.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.032+0000 7f52f0e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b67b050580 0x55b67b056e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48378/0 (socket says 172.21.15.12:48378) 2024-03-23T23:04:43.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.032+0000 7f52f1800640 1 -- 172.21.15.12:0/1418246339 learned_addr learned my addr 172.21.15.12:0/1418246339 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:43.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.032+0000 7f52f1800640 1 -- 172.21.15.12:0/1418246339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b67b050580 msgr2=0x55b67b056e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:43.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.032+0000 7f52f1800640 1 --2- 172.21.15.12:0/1418246339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b67b050580 0x55b67b056e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.032+0000 7f52f1800640 1 -- 172.21.15.12:0/1418246339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b67b04e9f0 msgr2=0x55b67b04ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:43.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.032+0000 7f52f1800640 1 --2- 172.21.15.12:0/1418246339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b67b04e9f0 0x55b67b04ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.032+0000 7f52f1800640 1 -- 172.21.15.12:0/1418246339 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b67af80080 con 0x55b67b04fbc0 2024-03-23T23:04:43.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.032+0000 7f52f1800640 1 --2- 172.21.15.12:0/1418246339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b04fbc0 0x55b67b050040 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f52e00d70e0 tx=0x7f52e00d75a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d331cdcd88d378d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:43.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.033+0000 7f52dea00640 1 -- 172.21.15.12:0/1418246339 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e00e2070 con 0x55b67b04fbc0 2024-03-23T23:04:43.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.033+0000 7f52dea00640 1 -- 172.21.15.12:0/1418246339 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52e00ddae0 con 0x55b67b04fbc0 2024-03-23T23:04:43.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.033+0000 7f52dea00640 1 -- 172.21.15.12:0/1418246339 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e00dcc40 con 0x55b67b04fbc0 2024-03-23T23:04:43.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.033+0000 7f52ea000640 1 --2- 172.21.15.12:0/2187844150 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f52d40a2880 0x7f52d40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.033+0000 7f52ea000640 1 -- 172.21.15.12:0/2187844150 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f52e00ceef0 con 0x7f52ec101580 2024-03-23T23:04:43.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.033+0000 7f52f2200640 1 --2- 172.21.15.12:0/2187844150 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f52d40a2880 0x7f52d40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.033+0000 7f52f6e72740 1 -- 172.21.15.12:0/1418246339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b04fbc0 msgr2=0x55b67b050040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:43.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.033+0000 7f52f6e72740 1 --2- 172.21.15.12:0/1418246339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b04fbc0 0x55b67b050040 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f52e00d70e0 tx=0x7f52e00d75a0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.033+0000 7f52f6e72740 1 -- 172.21.15.12:0/1418246339 shutdown_connections 2024-03-23T23:04:43.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.033+0000 7f52f6e72740 1 --2- 172.21.15.12:0/1418246339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b67b050580 0x55b67b056e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.033+0000 7f52f6e72740 1 --2- 172.21.15.12:0/1418246339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b04fbc0 0x55b67b050040 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.033+0000 7f52f6e72740 1 --2- 172.21.15.12:0/1418246339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b67b04e9f0 0x55b67b04ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.033+0000 7f52f6e72740 1 -- 172.21.15.12:0/1418246339 >> 172.21.15.12:0/1418246339 conn(0x55b67b049de0 msgr2=0x55b67b04a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:43.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.033+0000 7f52f6e72740 1 -- 172.21.15.12:0/1418246339 shutdown_connections 2024-03-23T23:04:43.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.034+0000 7f52f6e72740 1 -- 172.21.15.12:0/1418246339 wait complete. 2024-03-23T23:04:43.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.034+0000 7f52f2200640 1 --2- 172.21.15.12:0/2187844150 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f52d40a2880 0x7f52d40a4d40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x55b67b04fac0 tx=0x7f52e4009600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:43.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.034+0000 7f52f6e72740 1 -- start start 2024-03-23T23:04:43.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.034+0000 7f52f6e72740 1 Processor -- start 2024-03-23T23:04:43.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.035+0000 7f52f6e72740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b04e9f0 0x55b67b058c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.035+0000 7f52f6e72740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b67b04fbc0 0x55b67b059170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.035+0000 7f52f6e72740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b67b050580 0x55b67b0601b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.035+0000 7f52f6e72740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b67b057590 con 0x55b67b04fbc0 2024-03-23T23:04:43.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.035+0000 7f52f6e72740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b67af6f0d0 con 0x55b67b050580 2024-03-23T23:04:43.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.035+0000 7f52f6e72740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b67b057710 con 0x55b67b04e9f0 2024-03-23T23:04:43.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.035+0000 7f52f6e72740 10 client.0 _ll_register_callbacks cb 0x55b67af73e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:43.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.035+0000 7f52f1800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b04e9f0 0x55b67b058c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.035+0000 7f52f1800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b04e9f0 0x55b67b058c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55036/0 (socket says 172.21.15.12:55036) 2024-03-23T23:04:43.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.035+0000 7f52f1800640 1 -- 172.21.15.12:0/3051167691 learned_addr learned my addr 172.21.15.12:0/3051167691 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:43.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.035+0000 7f52f2200640 1 --2- 172.21.15.12:0/3051167691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b67b050580 0x55b67b0601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.035+0000 7f52f0e00640 1 --2- 172.21.15.12:0/3051167691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b67b04fbc0 0x55b67b059170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.035+0000 7f52f1800640 1 -- 172.21.15.12:0/3051167691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b67b050580 msgr2=0x55b67b0601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:43.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.035+0000 7f52f1800640 1 --2- 172.21.15.12:0/3051167691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b67b050580 0x55b67b0601b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.035+0000 7f52f1800640 1 -- 172.21.15.12:0/3051167691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b67b04fbc0 msgr2=0x55b67b059170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:43.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.035+0000 7f52f1800640 1 --2- 172.21.15.12:0/3051167691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b67b04fbc0 0x55b67b059170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.035+0000 7f52f1800640 1 -- 172.21.15.12:0/3051167691 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b67acc73f0 con 0x55b67b04e9f0 2024-03-23T23:04:43.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.036+0000 7f52f1800640 1 --2- 172.21.15.12:0/3051167691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b04e9f0 0x55b67b058c30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f52e0000c00 tx=0x7f52e00cf1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=112bc4505c1c4409 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:43.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.036+0000 7f52d3400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:04:43.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.036+0000 7f52d3400640 1 -- 172.21.15.12:0/3051167691 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e0097b40 con 0x55b67b04e9f0 2024-03-23T23:04:43.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.036+0000 7f52d3400640 1 -- 172.21.15.12:0/3051167691 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f52e0099ba0 con 0x55b67b04e9f0 2024-03-23T23:04:43.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.036+0000 7f52f6e72740 1 -- 172.21.15.12:0/3051167691 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f52e00ddae0 con 0x55b67b04e9f0 2024-03-23T23:04:43.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.036+0000 7f52d3400640 1 -- 172.21.15.12:0/3051167691 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52e009b200 con 0x55b67b04e9f0 2024-03-23T23:04:43.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.036+0000 7f52f6e72740 10 client.26020 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:43.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.036+0000 7f52f6e72740 1 -- 172.21.15.12:0/3051167691 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55b67ad917d0 con 0x55b67b04e9f0 2024-03-23T23:04:43.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.037+0000 7f52d3400640 1 -- 172.21.15.12:0/3051167691 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52e00ddae0 con 0x55b67b04e9f0 2024-03-23T23:04:43.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.037+0000 7f52d3400640 1 -- 172.21.15.12:0/3051167691 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f52e00860f0 con 0x55b67b04e9f0 2024-03-23T23:04:43.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.037+0000 7f52f6e72740 10 client.26020 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:43.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.037+0000 7f52f6e72740 10 client.26020 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:04:43.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.037+0000 7f52f6e72740 20 client.26020 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:43.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.037+0000 7f52f6e72740 1 --2- 172.21.15.12:0/3051167691 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55b67b064310 0x55b67b066700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.037+0000 7f52f6e72740 4 client.26020 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:04:43.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.037+0000 7f52f6e72740 1 -- 172.21.15.12:0/3051167691 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b67afc0800 con 0x55b67b064310 2024-03-23T23:04:43.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.037+0000 7f52f0e00640 1 --2- 172.21.15.12:0/3051167691 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55b67b064310 0x55b67b066700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.038+0000 7f52f0e00640 1 --2- 172.21.15.12:0/3051167691 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55b67b064310 0x55b67b066700 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f80161aee5669a1d server_cookie=69d74c1a442f0253 in_seq=0 out_seq=0 2024-03-23T23:04:43.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.038+0000 7f52d3400640 10 client.26020 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:04:43.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.039+0000 7f52d3400640 1 -- 172.21.15.12:0/3051167691 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b67afc0800 con 0x55b67b064310 2024-03-23T23:04:43.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.039+0000 7f52d3400640 10 client.26020 handle_command_reply: tid=0 2024-03-23T23:04:43.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.039+0000 7f52f6e72740 1 client.26020 shutdown 2024-03-23T23:04:43.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.039+0000 7f52f6e72740 1 -- 172.21.15.12:0/3051167691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b04e9f0 msgr2=0x55b67b058c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:43.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.039+0000 7f52f6e72740 1 --2- 172.21.15.12:0/3051167691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b04e9f0 0x55b67b058c30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f52e0000c00 tx=0x7f52e00cf1f0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.039+0000 7f52f6e72740 1 -- 172.21.15.12:0/3051167691 shutdown_connections 2024-03-23T23:04:43.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.039+0000 7f52f6e72740 1 --2- 172.21.15.12:0/3051167691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b67b050580 0x55b67b0601b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.039+0000 7f52f6e72740 1 --2- 172.21.15.12:0/3051167691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b67b04fbc0 0x55b67b059170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.039+0000 7f52f6e72740 1 --2- 172.21.15.12:0/3051167691 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55b67b064310 0x55b67b066700 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.039+0000 7f52f6e72740 1 --2- 172.21.15.12:0/3051167691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b04e9f0 0x55b67b058c30 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.039+0000 7f52f6e72740 1 -- 172.21.15.12:0/3051167691 >> 172.21.15.12:0/3051167691 conn(0x55b67b049de0 msgr2=0x55b67b049160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:43.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.039+0000 7f52d3400640 0 client.26020 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:04:43.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.039+0000 7f52f6e72740 1 -- 172.21.15.12:0/3051167691 shutdown_connections 2024-03-23T23:04:43.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.039+0000 7f52f6e72740 1 -- 172.21.15.12:0/3051167691 wait complete. 2024-03-23T23:04:43.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.039+0000 7f52f6e72740 20 client.26020 trim_cache size 0 max 16384 2024-03-23T23:04:43.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.053+0000 7f52f6e72740 1 Processor -- start 2024-03-23T23:04:43.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.053+0000 7f52f6e72740 1 -- start start 2024-03-23T23:04:43.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.053+0000 7f52f6e72740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b67b04e9f0 0x55b67b052df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.053+0000 7f52f6e72740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b67b04fbc0 0x55b67b060170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.053+0000 7f52f6e72740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b050580 0x55b67b053330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.053+0000 7f52f6e72740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec1acfe0 con 0x55b67b04fbc0 2024-03-23T23:04:43.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.053+0000 7f52f6e72740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec1aaae0 con 0x55b67b04e9f0 2024-03-23T23:04:43.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.053+0000 7f52f6e72740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec1a8b80 con 0x55b67b050580 2024-03-23T23:04:43.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.053+0000 7f52f1800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b67b04e9f0 0x55b67b052df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.053+0000 7f52f1800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b67b04e9f0 0x55b67b052df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48390/0 (socket says 172.21.15.12:48390) 2024-03-23T23:04:43.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.053+0000 7f52f1800640 1 -- 172.21.15.12:0/2066554917 learned_addr learned my addr 172.21.15.12:0/2066554917 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:43.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.053+0000 7f52f2200640 1 --2- 172.21.15.12:0/2066554917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b050580 0x55b67b053330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.053+0000 7f52f0e00640 1 --2- 172.21.15.12:0/2066554917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b67b04fbc0 0x55b67b060170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.054+0000 7f52f1800640 1 -- 172.21.15.12:0/2066554917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b050580 msgr2=0x55b67b053330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:43.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.054+0000 7f52f1800640 1 --2- 172.21.15.12:0/2066554917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b050580 0x55b67b053330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.054+0000 7f52f1800640 1 -- 172.21.15.12:0/2066554917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b67b04fbc0 msgr2=0x55b67b060170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:43.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.054+0000 7f52f1800640 1 --2- 172.21.15.12:0/2066554917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b67b04fbc0 0x55b67b060170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.054+0000 7f52f1800640 1 -- 172.21.15.12:0/2066554917 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b67b04f640 con 0x55b67b04e9f0 2024-03-23T23:04:43.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.054+0000 7f52f0e00640 1 --2- 172.21.15.12:0/2066554917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b67b04fbc0 0x55b67b060170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:04:43.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.054+0000 7f52f1800640 1 --2- 172.21.15.12:0/2066554917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b67b04e9f0 0x55b67b052df0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f52e000c6a0 tx=0x7f52e00cf260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a906d79adc515e60 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:43.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.054+0000 7f52de000640 1 -- 172.21.15.12:0/2066554917 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e009b3e0 con 0x55b67b04e9f0 2024-03-23T23:04:43.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.054+0000 7f52de000640 1 -- 172.21.15.12:0/2066554917 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52e009bfa0 con 0x55b67b04e9f0 2024-03-23T23:04:43.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.054+0000 7f52de000640 1 -- 172.21.15.12:0/2066554917 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e009b670 con 0x55b67b04e9f0 2024-03-23T23:04:43.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.055+0000 7f52f6e72740 1 -- 172.21.15.12:0/2066554917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b67b04e9f0 msgr2=0x55b67b052df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:43.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.055+0000 7f52f6e72740 1 --2- 172.21.15.12:0/2066554917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b67b04e9f0 0x55b67b052df0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f52e000c6a0 tx=0x7f52e00cf260 comp rx=0 tx=0).stop 2024-03-23T23:04:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.055+0000 7f52f6e72740 1 -- 172.21.15.12:0/2066554917 shutdown_connections 2024-03-23T23:04:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.055+0000 7f52f6e72740 1 --2- 172.21.15.12:0/2066554917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b050580 0x55b67b053330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.055+0000 7f52f6e72740 1 --2- 172.21.15.12:0/2066554917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b67b04fbc0 0x55b67b060170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.055+0000 7f52f6e72740 1 --2- 172.21.15.12:0/2066554917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b67b04e9f0 0x55b67b052df0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.055+0000 7f52f6e72740 1 -- 172.21.15.12:0/2066554917 >> 172.21.15.12:0/2066554917 conn(0x55b67b049de0 msgr2=0x55b67b04d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.055+0000 7f52f6e72740 1 -- 172.21.15.12:0/2066554917 shutdown_connections 2024-03-23T23:04:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.055+0000 7f52f6e72740 1 -- 172.21.15.12:0/2066554917 wait complete. 2024-03-23T23:04:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.055+0000 7f52f6e72740 1 -- start start 2024-03-23T23:04:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.055+0000 7f52f6e72740 1 Processor -- start 2024-03-23T23:04:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.056+0000 7f52f6e72740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b04e9f0 0x55b67b05fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.056+0000 7f52f6e72740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b67b04fbc0 0x55b67b05e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.056+0000 7f52f6e72740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b67b050580 0x55b67b05e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.056+0000 7f52f6e72740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec1a9180 con 0x55b67b04fbc0 2024-03-23T23:04:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.056+0000 7f52f6e72740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec1aa480 con 0x55b67b050580 2024-03-23T23:04:43.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.056+0000 7f52f6e72740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec1a97c0 con 0x55b67b04e9f0 2024-03-23T23:04:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.056+0000 7f52f6e72740 10 client.0 _ll_register_callbacks cb 0x7f52e00e0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.056+0000 7f52f1800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b04e9f0 0x55b67b05fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.056+0000 7f52f1800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b04e9f0 0x55b67b05fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55060/0 (socket says 172.21.15.12:55060) 2024-03-23T23:04:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.056+0000 7f52f1800640 1 -- 172.21.15.12:0/2824028267 learned_addr learned my addr 172.21.15.12:0/2824028267 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.056+0000 7f52f2200640 1 --2- 172.21.15.12:0/2824028267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b67b050580 0x55b67b05e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.056+0000 7f52f0e00640 1 --2- 172.21.15.12:0/2824028267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b67b04fbc0 0x55b67b05e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.056+0000 7f52f1800640 1 -- 172.21.15.12:0/2824028267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b67b050580 msgr2=0x55b67b05e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.056+0000 7f52f1800640 1 --2- 172.21.15.12:0/2824028267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b67b050580 0x55b67b05e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.056+0000 7f52f1800640 1 -- 172.21.15.12:0/2824028267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b67b04fbc0 msgr2=0x55b67b05e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.056+0000 7f52f1800640 1 --2- 172.21.15.12:0/2824028267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b67b04fbc0 0x55b67b05e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.056+0000 7f52f1800640 1 -- 172.21.15.12:0/2824028267 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b67b05ed60 con 0x55b67b04e9f0 2024-03-23T23:04:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.057+0000 7f52f1800640 1 --2- 172.21.15.12:0/2824028267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b04e9f0 0x55b67b05fcd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f52e00d7aa0 tx=0x7f52e009faf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77b266996413f7e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.057+0000 7f52d3400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:04:43.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.057+0000 7f52d3400640 1 -- 172.21.15.12:0/2824028267 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e009b670 con 0x55b67b04e9f0 2024-03-23T23:04:43.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.057+0000 7f52d3400640 1 -- 172.21.15.12:0/2824028267 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f52e00a3bb0 con 0x55b67b04e9f0 2024-03-23T23:04:43.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.057+0000 7f52f6e72740 1 -- 172.21.15.12:0/2824028267 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f52e009bfa0 con 0x55b67b04e9f0 2024-03-23T23:04:43.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.057+0000 7f52d3400640 1 -- 172.21.15.12:0/2824028267 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52e0097b40 con 0x55b67b04e9f0 2024-03-23T23:04:43.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.057+0000 7f52f6e72740 10 client.26026 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:43.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.057+0000 7f52f6e72740 1 -- 172.21.15.12:0/2824028267 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55b67b05ed60 con 0x55b67b04e9f0 2024-03-23T23:04:43.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.057+0000 7f52d3400640 1 -- 172.21.15.12:0/2824028267 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52e009bfa0 con 0x55b67b04e9f0 2024-03-23T23:04:43.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.057+0000 7f52d3400640 1 -- 172.21.15.12:0/2824028267 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f52e009cbd0 con 0x55b67b04e9f0 2024-03-23T23:04:43.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.057+0000 7f52f6e72740 10 client.26026 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:43.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.058+0000 7f52f6e72740 10 client.26026 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:04:43.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.058+0000 7f52f6e72740 20 client.26026 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:43.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.058+0000 7f52f6e72740 1 --2- 172.21.15.12:0/2824028267 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55b67b0c91e0 0x55b67b0cb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.058+0000 7f52f6e72740 4 client.26026 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:04:43.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.058+0000 7f52f6e72740 1 -- 172.21.15.12:0/2824028267 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b67afbab50 con 0x55b67b0c91e0 2024-03-23T23:04:43.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.058+0000 7f52f0e00640 1 --2- 172.21.15.12:0/2824028267 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55b67b0c91e0 0x55b67b0cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.058+0000 7f52f0e00640 1 --2- 172.21.15.12:0/2824028267 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55b67b0c91e0 0x55b67b0cb600 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=aa3d82242f62c638 server_cookie=86fb83f5b93d1b77 in_seq=0 out_seq=0 2024-03-23T23:04:43.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.058+0000 7f52d3400640 10 client.26026 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:04:43.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.058+0000 7f52d3400640 1 -- 172.21.15.12:0/2824028267 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b67afbab50 con 0x55b67b0c91e0 2024-03-23T23:04:43.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.058+0000 7f52d3400640 10 client.26026 handle_command_reply: tid=0 2024-03-23T23:04:43.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.058+0000 7f52f6e72740 1 client.26026 shutdown 2024-03-23T23:04:43.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.059+0000 7f52f6e72740 1 -- 172.21.15.12:0/2824028267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b04e9f0 msgr2=0x55b67b05fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:43.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.059+0000 7f52f6e72740 1 --2- 172.21.15.12:0/2824028267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b04e9f0 0x55b67b05fcd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f52e00d7aa0 tx=0x7f52e009faf0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.059+0000 7f52f6e72740 1 -- 172.21.15.12:0/2824028267 shutdown_connections 2024-03-23T23:04:43.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.059+0000 7f52f6e72740 1 --2- 172.21.15.12:0/2824028267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b67b050580 0x55b67b05e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.059+0000 7f52f6e72740 1 --2- 172.21.15.12:0/2824028267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b67b04fbc0 0x55b67b05e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.059+0000 7f52f6e72740 1 --2- 172.21.15.12:0/2824028267 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55b67b0c91e0 0x55b67b0cb600 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.059+0000 7f52f6e72740 1 --2- 172.21.15.12:0/2824028267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b67b04e9f0 0x55b67b05fcd0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.059+0000 7f52f6e72740 1 -- 172.21.15.12:0/2824028267 >> 172.21.15.12:0/2824028267 conn(0x55b67b049de0 msgr2=0x55b67b0488d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:43.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.059+0000 7f52d3400640 0 client.26026 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:04:43.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.059+0000 7f52f6e72740 1 -- 172.21.15.12:0/2824028267 shutdown_connections 2024-03-23T23:04:43.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.059+0000 7f52f6e72740 1 -- 172.21.15.12:0/2824028267 wait complete. 2024-03-23T23:04:43.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.059+0000 7f52f6e72740 20 client.26026 trim_cache size 0 max 16384 2024-03-23T23:04:43.063 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:04:43.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.059+0000 7f52f2c00640 1 -- 172.21.15.12:0/2187844150 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f52d40a2880 msgr2=0x7f52d40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:43.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.059+0000 7f52f2c00640 1 --2- 172.21.15.12:0/2187844150 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f52d40a2880 0x7f52d40a4d40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x55b67b04fac0 tx=0x7f52e4009600 comp rx=0 tx=0).stop 2024-03-23T23:04:43.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.059+0000 7f52f2c00640 1 -- 172.21.15.12:0/2187844150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52ec101580 msgr2=0x7f52ec075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:43.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.060+0000 7f52f2c00640 1 --2- 172.21.15.12:0/2187844150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52ec101580 0x7f52ec075450 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f52e000e9c0 tx=0x7f52e000ee80 comp rx=0 tx=0).stop 2024-03-23T23:04:43.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.060+0000 7f52f2c00640 1 -- 172.21.15.12:0/2187844150 shutdown_connections 2024-03-23T23:04:43.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.060+0000 7f52f2c00640 1 --2- 172.21.15.12:0/2187844150 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f52d40a2880 0x7f52d40a4d40 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.060+0000 7f52f2c00640 1 --2- 172.21.15.12:0/2187844150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52ec103130 0x7f52ec073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.060+0000 7f52f2c00640 1 --2- 172.21.15.12:0/2187844150 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52ec102770 0x7f52ec073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.060+0000 7f52f2c00640 1 --2- 172.21.15.12:0/2187844150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52ec101580 0x7f52ec075450 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.060+0000 7f52f2c00640 1 -- 172.21.15.12:0/2187844150 >> 172.21.15.12:0/2187844150 conn(0x7f52ec0fcb10 msgr2=0x7f52ec105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:43.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.060+0000 7f52f2c00640 1 -- 172.21.15.12:0/2187844150 shutdown_connections 2024-03-23T23:04:43.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.060+0000 7f52f2c00640 1 -- 172.21.15.12:0/2187844150 wait complete. 2024-03-23T23:04:43.075 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:04:43.076 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-23T23:04:43.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:42 smithi173 ceph-mon[36116]: pgmap v477: 97 pgs: 97 active+clean; 1.5 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:04:43.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:42 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/4138086759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:04:43.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:42 smithi203 ceph-mon[36693]: pgmap v477: 97 pgs: 97 active+clean; 1.5 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:04:43.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:42 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/4138086759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:04:43.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.285+0000 7ff26fe00640 1 -- 172.21.15.12:0/2237629267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2701037c0 msgr2=0x7ff270103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:43.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.285+0000 7ff26fe00640 1 --2- 172.21.15.12:0/2237629267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2701037c0 0x7ff270103be0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff260009900 tx=0x7ff26002bc40 comp rx=0 tx=0).stop 2024-03-23T23:04:43.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.285+0000 7ff26fe00640 1 -- 172.21.15.12:0/2237629267 shutdown_connections 2024-03-23T23:04:43.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.285+0000 7ff26fe00640 1 --2- 172.21.15.12:0/2237629267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff270105370 0x7ff27010bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.285+0000 7ff26fe00640 1 --2- 172.21.15.12:0/2237629267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff2701049b0 0x7ff270104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.285+0000 7ff26fe00640 1 --2- 172.21.15.12:0/2237629267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2701037c0 0x7ff270103be0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.286+0000 7ff26fe00640 1 -- 172.21.15.12:0/2237629267 >> 172.21.15.12:0/2237629267 conn(0x7ff2700fed50 msgr2=0x7ff270101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:43.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.286+0000 7ff26fe00640 1 -- 172.21.15.12:0/2237629267 shutdown_connections 2024-03-23T23:04:43.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.286+0000 7ff26fe00640 1 -- 172.21.15.12:0/2237629267 wait complete. 2024-03-23T23:04:43.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.287+0000 7ff26fe00640 1 Processor -- start 2024-03-23T23:04:43.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.287+0000 7ff26fe00640 1 -- start start 2024-03-23T23:04:43.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.287+0000 7ff26fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff2701037c0 0x7ff270199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.288+0000 7ff26fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2701049b0 0x7ff27019a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.288+0000 7ff26fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff270105370 0x7ff2701a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.288+0000 7ff26fe00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27010f480 con 0x7ff270105370 2024-03-23T23:04:43.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.288+0000 7ff26fe00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27010f300 con 0x7ff2701037c0 2024-03-23T23:04:43.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.288+0000 7ff26fe00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27010f600 con 0x7ff2701049b0 2024-03-23T23:04:43.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.288+0000 7ff26ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff2701037c0 0x7ff270199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.288+0000 7ff26e000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2701049b0 0x7ff27019a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.288+0000 7ff26ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff2701037c0 0x7ff270199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48418/0 (socket says 172.21.15.12:48418) 2024-03-23T23:04:43.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.288+0000 7ff26ea00640 1 -- 172.21.15.12:0/2576397246 learned_addr learned my addr 172.21.15.12:0/2576397246 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:43.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.288+0000 7ff26f400640 1 --2- 172.21.15.12:0/2576397246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff270105370 0x7ff2701a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.288+0000 7ff26ea00640 1 -- 172.21.15.12:0/2576397246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2701049b0 msgr2=0x7ff27019a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:43.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.288+0000 7ff26ea00640 1 --2- 172.21.15.12:0/2576397246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2701049b0 0x7ff27019a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.288+0000 7ff26ea00640 1 -- 172.21.15.12:0/2576397246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff270105370 msgr2=0x7ff2701a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:43.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.288+0000 7ff26ea00640 1 --2- 172.21.15.12:0/2576397246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff270105370 0x7ff2701a1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.288+0000 7ff26ea00640 1 -- 172.21.15.12:0/2576397246 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2701a16f0 con 0x7ff2701037c0 2024-03-23T23:04:43.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.288+0000 7ff26e000640 1 --2- 172.21.15.12:0/2576397246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2701049b0 0x7ff27019a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:43.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.289+0000 7ff26ea00640 1 --2- 172.21.15.12:0/2576397246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff2701037c0 0x7ff270199ad0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff26002efd0 tx=0x7ff260005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:43.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.289+0000 7ff267400640 1 -- 172.21.15.12:0/2576397246 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff260041070 con 0x7ff2701037c0 2024-03-23T23:04:43.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.289+0000 7ff26fe00640 1 -- 172.21.15.12:0/2576397246 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2701a1980 con 0x7ff2701037c0 2024-03-23T23:04:43.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.289+0000 7ff267400640 1 -- 172.21.15.12:0/2576397246 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff2600044d0 con 0x7ff2701037c0 2024-03-23T23:04:43.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.289+0000 7ff267400640 1 -- 172.21.15.12:0/2576397246 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff260034440 con 0x7ff2701037c0 2024-03-23T23:04:43.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.289+0000 7ff26fe00640 1 -- 172.21.15.12:0/2576397246 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2701a1e40 con 0x7ff2701037c0 2024-03-23T23:04:43.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.292+0000 7ff267400640 1 -- 172.21.15.12:0/2576397246 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff260044070 con 0x7ff2701037c0 2024-03-23T23:04:43.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.292+0000 7ff2789ce740 1 Processor -- start 2024-03-23T23:04:43.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.292+0000 7ff2789ce740 1 -- start start 2024-03-23T23:04:43.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.293+0000 7ff2789ce740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eae9f0 0x558275eaee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.293+0000 7ff2789ce740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558275eafbc0 0x558275eb0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.293+0000 7ff2789ce740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558275eb0580 0x558275eb6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.293+0000 7ff2789ce740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x558275e132c0 con 0x558275eafbc0 2024-03-23T23:04:43.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.293+0000 7ff2789ce740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x558275dbbc90 con 0x558275eae9f0 2024-03-23T23:04:43.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.293+0000 7ff2789ce740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x558275e0f360 con 0x558275eb0580 2024-03-23T23:04:43.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.293+0000 7ff26f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eae9f0 0x558275eaee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.293+0000 7ff26f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eae9f0 0x558275eaee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48420/0 (socket says 172.21.15.12:48420) 2024-03-23T23:04:43.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.293+0000 7ff26f400640 1 -- 172.21.15.12:0/2623668739 learned_addr learned my addr 172.21.15.12:0/2623668739 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:43.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.293+0000 7ff26e000640 1 --2- 172.21.15.12:0/2623668739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558275eb0580 0x558275eb6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.293+0000 7ff26ea00640 1 --2- 172.21.15.12:0/2623668739 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558275eafbc0 0x558275eb0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.293+0000 7ff26f400640 1 -- 172.21.15.12:0/2623668739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558275eb0580 msgr2=0x558275eb6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:43.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.293+0000 7ff26f400640 1 --2- 172.21.15.12:0/2623668739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558275eb0580 0x558275eb6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.293+0000 7ff26f400640 1 -- 172.21.15.12:0/2623668739 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558275eafbc0 msgr2=0x558275eb0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:43.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.293+0000 7ff26f400640 1 --2- 172.21.15.12:0/2623668739 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558275eafbc0 0x558275eb0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.293+0000 7ff26f400640 1 -- 172.21.15.12:0/2623668739 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558275de0080 con 0x558275eae9f0 2024-03-23T23:04:43.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.294+0000 7ff267400640 1 --2- 172.21.15.12:0/2576397246 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff2540a2800 0x7ff2540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.294+0000 7ff26f400640 1 --2- 172.21.15.12:0/2623668739 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eae9f0 0x558275eaee10 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff268010370 tx=0x7ff268010830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c844bcdb908647dd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:43.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.294+0000 7ff267400640 1 -- 172.21.15.12:0/2576397246 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff2600f0600 con 0x7ff2701037c0 2024-03-23T23:04:43.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.294+0000 7ff25be00640 1 -- 172.21.15.12:0/2623668739 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff268016070 con 0x558275eae9f0 2024-03-23T23:04:43.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.294+0000 7ff25be00640 1 -- 172.21.15.12:0/2623668739 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff268004d20 con 0x558275eae9f0 2024-03-23T23:04:43.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.294+0000 7ff25be00640 1 -- 172.21.15.12:0/2623668739 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff268005480 con 0x558275eae9f0 2024-03-23T23:04:43.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.294+0000 7ff26ea00640 1 --2- 172.21.15.12:0/2576397246 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff2540a2800 0x7ff2540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.294+0000 7ff2789ce740 1 -- 172.21.15.12:0/2623668739 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eae9f0 msgr2=0x558275eaee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:43.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.294+0000 7ff2789ce740 1 --2- 172.21.15.12:0/2623668739 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eae9f0 0x558275eaee10 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff268010370 tx=0x7ff268010830 comp rx=0 tx=0).stop 2024-03-23T23:04:43.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.295+0000 7ff2789ce740 1 -- 172.21.15.12:0/2623668739 shutdown_connections 2024-03-23T23:04:43.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.295+0000 7ff2789ce740 1 --2- 172.21.15.12:0/2623668739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558275eb0580 0x558275eb6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.295+0000 7ff2789ce740 1 --2- 172.21.15.12:0/2623668739 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558275eafbc0 0x558275eb0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.295+0000 7ff2789ce740 1 --2- 172.21.15.12:0/2623668739 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eae9f0 0x558275eaee10 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.295+0000 7ff2789ce740 1 -- 172.21.15.12:0/2623668739 >> 172.21.15.12:0/2623668739 conn(0x558275ea9de0 msgr2=0x558275eaa1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:43.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.295+0000 7ff2789ce740 1 -- 172.21.15.12:0/2623668739 shutdown_connections 2024-03-23T23:04:43.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.295+0000 7ff26ea00640 1 --2- 172.21.15.12:0/2576397246 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff2540a2800 0x7ff2540a4cc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff2600c5200 tx=0x7ff260009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:43.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.295+0000 7ff2789ce740 1 -- 172.21.15.12:0/2623668739 wait complete. 2024-03-23T23:04:43.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.295+0000 7ff2789ce740 1 -- start start 2024-03-23T23:04:43.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.295+0000 7ff2789ce740 1 Processor -- start 2024-03-23T23:04:43.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.296+0000 7ff2789ce740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558275eae9f0 0x558275eb8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.296+0000 7ff2789ce740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eafbc0 0x558275eb91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.296+0000 7ff2789ce740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558275eb0580 0x558275ec01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.296+0000 7ff2789ce740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x558275eb7590 con 0x558275eae9f0 2024-03-23T23:04:43.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.296+0000 7ff2789ce740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x558275dcf0d0 con 0x558275eafbc0 2024-03-23T23:04:43.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.296+0000 7ff2789ce740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x558275eb7710 con 0x558275eb0580 2024-03-23T23:04:43.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.296+0000 7ff2789ce740 10 client.0 _ll_register_callbacks cb 0x558275dd3e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:43.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.296+0000 7ff26e000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eafbc0 0x558275eb91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.296+0000 7ff26e000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eafbc0 0x558275eb91a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48428/0 (socket says 172.21.15.12:48428) 2024-03-23T23:04:43.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.296+0000 7ff26e000640 1 -- 172.21.15.12:0/1236723560 learned_addr learned my addr 172.21.15.12:0/1236723560 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:43.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.296+0000 7ff26f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558275eae9f0 0x558275eb8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.296+0000 7ff26f400640 1 --2- 172.21.15.12:0/1236723560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558275eb0580 0x558275ec01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.296+0000 7ff26e000640 1 -- 172.21.15.12:0/1236723560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558275eb0580 msgr2=0x558275ec01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:43.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.296+0000 7ff26e000640 1 --2- 172.21.15.12:0/1236723560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558275eb0580 0x558275ec01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.296+0000 7ff26e000640 1 -- 172.21.15.12:0/1236723560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558275eae9f0 msgr2=0x558275eb8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:43.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.296+0000 7ff26e000640 1 --2- 172.21.15.12:0/1236723560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558275eae9f0 0x558275eb8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.296+0000 7ff26e000640 1 -- 172.21.15.12:0/1236723560 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558275b273f0 con 0x558275eafbc0 2024-03-23T23:04:43.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.296+0000 7ff26f400640 1 --2- 172.21.15.12:0/1236723560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558275eae9f0 0x558275eb8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:04:43.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.297+0000 7ff26e000640 1 --2- 172.21.15.12:0/1236723560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eafbc0 0x558275eb91a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff25c012d20 tx=0x7ff25c00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3217865b048e1dfc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:43.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.297+0000 7ff258c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:04:43.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.297+0000 7ff258c00640 1 -- 172.21.15.12:0/1236723560 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c012110 con 0x558275eafbc0 2024-03-23T23:04:43.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.297+0000 7ff258c00640 1 -- 172.21.15.12:0/1236723560 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff25c004030 con 0x558275eafbc0 2024-03-23T23:04:43.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.297+0000 7ff2789ce740 1 -- 172.21.15.12:0/1236723560 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff268004d20 con 0x558275eafbc0 2024-03-23T23:04:43.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.297+0000 7ff258c00640 1 -- 172.21.15.12:0/1236723560 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff25c0122b0 con 0x558275eafbc0 2024-03-23T23:04:43.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.297+0000 7ff2789ce740 10 client.26063 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:43.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.297+0000 7ff2789ce740 1 -- 172.21.15.12:0/1236723560 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x558275bf17d0 con 0x558275eafbc0 2024-03-23T23:04:43.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.297+0000 7ff258c00640 1 -- 172.21.15.12:0/1236723560 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff268004d20 con 0x558275eafbc0 2024-03-23T23:04:43.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.297+0000 7ff258c00640 1 -- 172.21.15.12:0/1236723560 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ff25c010430 con 0x558275eafbc0 2024-03-23T23:04:43.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.298+0000 7ff2789ce740 10 client.26063 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:43.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.298+0000 7ff2789ce740 10 client.26063 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:04:43.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.298+0000 7ff2789ce740 20 client.26063 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:43.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.298+0000 7ff2789ce740 1 --2- 172.21.15.12:0/1236723560 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558275ec46c0 0x558275ec6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.298+0000 7ff2789ce740 4 client.26063 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:04:43.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.298+0000 7ff2789ce740 1 -- 172.21.15.12:0/1236723560 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558275e20800 con 0x558275ec46c0 2024-03-23T23:04:43.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.298+0000 7ff26f400640 1 --2- 172.21.15.12:0/1236723560 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558275ec46c0 0x558275ec6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.298+0000 7ff26f400640 1 --2- 172.21.15.12:0/1236723560 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558275ec46c0 0x558275ec6b20 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcf95ea63b875ea9 server_cookie=bc169dccf14c5261 in_seq=0 out_seq=0 2024-03-23T23:04:43.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.298+0000 7ff258c00640 10 client.26063 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:43.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.299+0000 7ff258c00640 1 -- 172.21.15.12:0/1236723560 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558275e20800 con 0x558275ec46c0 2024-03-23T23:04:43.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.299+0000 7ff258c00640 10 client.26063 handle_command_reply: tid=0 2024-03-23T23:04:43.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.299+0000 7ff2789ce740 1 client.26063 shutdown 2024-03-23T23:04:43.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.299+0000 7ff2789ce740 1 -- 172.21.15.12:0/1236723560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eafbc0 msgr2=0x558275eb91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:43.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.299+0000 7ff2789ce740 1 --2- 172.21.15.12:0/1236723560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eafbc0 0x558275eb91a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff25c012d20 tx=0x7ff25c00f790 comp rx=0 tx=0).stop 2024-03-23T23:04:43.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.299+0000 7ff2789ce740 1 -- 172.21.15.12:0/1236723560 shutdown_connections 2024-03-23T23:04:43.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.299+0000 7ff2789ce740 1 --2- 172.21.15.12:0/1236723560 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558275ec46c0 0x558275ec6b20 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.299+0000 7ff2789ce740 1 --2- 172.21.15.12:0/1236723560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558275eb0580 0x558275ec01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.299+0000 7ff2789ce740 1 --2- 172.21.15.12:0/1236723560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eafbc0 0x558275eb91a0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.299+0000 7ff2789ce740 1 --2- 172.21.15.12:0/1236723560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558275eae9f0 0x558275eb8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.299+0000 7ff2789ce740 1 -- 172.21.15.12:0/1236723560 >> 172.21.15.12:0/1236723560 conn(0x558275ea9de0 msgr2=0x558275ea9160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:43.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.299+0000 7ff258c00640 0 client.26063 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:43.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.299+0000 7ff2789ce740 1 -- 172.21.15.12:0/1236723560 shutdown_connections 2024-03-23T23:04:43.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.300+0000 7ff2789ce740 1 -- 172.21.15.12:0/1236723560 wait complete. 2024-03-23T23:04:43.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.300+0000 7ff2789ce740 20 client.26063 trim_cache size 0 max 16384 2024-03-23T23:04:43.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.313+0000 7ff2789ce740 1 Processor -- start 2024-03-23T23:04:43.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.313+0000 7ff2789ce740 1 -- start start 2024-03-23T23:04:43.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.313+0000 7ff2789ce740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558275eae9f0 0x558275eb2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.313+0000 7ff2789ce740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eafbc0 0x558275ec01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.313+0000 7ff2789ce740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558275eb0580 0x558275eb31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.313+0000 7ff2789ce740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2701acf10 con 0x558275eb0580 2024-03-23T23:04:43.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.313+0000 7ff2789ce740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2701aaa10 con 0x558275eafbc0 2024-03-23T23:04:43.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.313+0000 7ff2789ce740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2701a8ab0 con 0x558275eae9f0 2024-03-23T23:04:43.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.313+0000 7ff26e000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eafbc0 0x558275ec01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.314+0000 7ff26e000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eafbc0 0x558275ec01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48436/0 (socket says 172.21.15.12:48436) 2024-03-23T23:04:43.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.314+0000 7ff26e000640 1 -- 172.21.15.12:0/1617425716 learned_addr learned my addr 172.21.15.12:0/1617425716 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:43.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.314+0000 7ff26f400640 1 --2- 172.21.15.12:0/1617425716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558275eae9f0 0x558275eb2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.314+0000 7ff26f400640 1 --2- 172.21.15.12:0/1617425716 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558275eb0580 0x558275eb31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.314+0000 7ff26e000640 1 -- 172.21.15.12:0/1617425716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558275eae9f0 msgr2=0x558275eb2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:43.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.314+0000 7ff26e000640 1 --2- 172.21.15.12:0/1617425716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558275eae9f0 0x558275eb2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.314+0000 7ff26e000640 1 -- 172.21.15.12:0/1617425716 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558275eb0580 msgr2=0x558275eb31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:43.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.314+0000 7ff26e000640 1 --2- 172.21.15.12:0/1617425716 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558275eb0580 0x558275eb31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.314+0000 7ff26e000640 1 -- 172.21.15.12:0/1617425716 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558275eb6750 con 0x558275eafbc0 2024-03-23T23:04:43.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.314+0000 7ff26f400640 1 --2- 172.21.15.12:0/1617425716 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558275eb0580 0x558275eb31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:04:43.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.314+0000 7ff26e000640 1 --2- 172.21.15.12:0/1617425716 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eafbc0 0x558275ec01a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff25c0058c0 tx=0x7ff25c005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2beb23fb8562c98c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:43.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.314+0000 7ff25b400640 1 -- 172.21.15.12:0/1617425716 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c010a90 con 0x558275eafbc0 2024-03-23T23:04:43.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.315+0000 7ff25b400640 1 -- 172.21.15.12:0/1617425716 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff25c02b020 con 0x558275eafbc0 2024-03-23T23:04:43.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.315+0000 7ff25b400640 1 -- 172.21.15.12:0/1617425716 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c0035f0 con 0x558275eafbc0 2024-03-23T23:04:43.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.315+0000 7ff2789ce740 1 -- 172.21.15.12:0/1617425716 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eafbc0 msgr2=0x558275ec01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:43.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.315+0000 7ff2789ce740 1 --2- 172.21.15.12:0/1617425716 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eafbc0 0x558275ec01a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff25c0058c0 tx=0x7ff25c005850 comp rx=0 tx=0).stop 2024-03-23T23:04:43.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.315+0000 7ff2789ce740 1 -- 172.21.15.12:0/1617425716 shutdown_connections 2024-03-23T23:04:43.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.315+0000 7ff2789ce740 1 --2- 172.21.15.12:0/1617425716 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558275eb0580 0x558275eb31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.315+0000 7ff2789ce740 1 --2- 172.21.15.12:0/1617425716 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eafbc0 0x558275ec01a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.315+0000 7ff2789ce740 1 --2- 172.21.15.12:0/1617425716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558275eae9f0 0x558275eb2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.315+0000 7ff2789ce740 1 -- 172.21.15.12:0/1617425716 >> 172.21.15.12:0/1617425716 conn(0x558275ea9de0 msgr2=0x558275ead8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:43.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.315+0000 7ff2789ce740 1 -- 172.21.15.12:0/1617425716 shutdown_connections 2024-03-23T23:04:43.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.315+0000 7ff2789ce740 1 -- 172.21.15.12:0/1617425716 wait complete. 2024-03-23T23:04:43.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.316+0000 7ff2789ce740 1 -- start start 2024-03-23T23:04:43.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.316+0000 7ff2789ce740 1 Processor -- start 2024-03-23T23:04:43.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.316+0000 7ff2789ce740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558275eae9f0 0x558275ebc370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.316+0000 7ff2789ce740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eafbc0 0x558275ebc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.316+0000 7ff2789ce740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558275eb0580 0x558275ebf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.316+0000 7ff2789ce740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2701a90b0 con 0x558275eae9f0 2024-03-23T23:04:43.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.316+0000 7ff2789ce740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2701aa3b0 con 0x558275eafbc0 2024-03-23T23:04:43.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.316+0000 7ff2789ce740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2701a96f0 con 0x558275eb0580 2024-03-23T23:04:43.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.316+0000 7ff2789ce740 10 client.0 _ll_register_callbacks cb 0x7ff25c009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:43.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.316+0000 7ff26e000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eafbc0 0x558275ebc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.316+0000 7ff26e000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eafbc0 0x558275ebc8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48448/0 (socket says 172.21.15.12:48448) 2024-03-23T23:04:43.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.316+0000 7ff26f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558275eae9f0 0x558275ebc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.316+0000 7ff26e000640 1 -- 172.21.15.12:0/3765616157 learned_addr learned my addr 172.21.15.12:0/3765616157 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:43.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.317+0000 7ff26f400640 1 --2- 172.21.15.12:0/3765616157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558275eb0580 0x558275ebf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.317+0000 7ff26e000640 1 -- 172.21.15.12:0/3765616157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558275eb0580 msgr2=0x558275ebf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:43.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.317+0000 7ff26e000640 1 --2- 172.21.15.12:0/3765616157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558275eb0580 0x558275ebf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.317+0000 7ff26e000640 1 -- 172.21.15.12:0/3765616157 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558275eae9f0 msgr2=0x558275ebc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:43.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.317+0000 7ff26e000640 1 --2- 172.21.15.12:0/3765616157 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558275eae9f0 0x558275ebc370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.317+0000 7ff26e000640 1 -- 172.21.15.12:0/3765616157 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558275ebfec0 con 0x558275eafbc0 2024-03-23T23:04:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.317+0000 7ff26f400640 1 --2- 172.21.15.12:0/3765616157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558275eb0580 0x558275ebf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:04:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.317+0000 7ff26e000640 1 --2- 172.21.15.12:0/3765616157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eafbc0 0x558275ebc8b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff25c010970 tx=0x7ff25c011f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbe1ec338b300302 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.317+0000 7ff258c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:04:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.317+0000 7ff258c00640 1 -- 172.21.15.12:0/3765616157 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c005400 con 0x558275eafbc0 2024-03-23T23:04:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.317+0000 7ff258c00640 1 -- 172.21.15.12:0/3765616157 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff25c018040 con 0x558275eafbc0 2024-03-23T23:04:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.317+0000 7ff2789ce740 1 -- 172.21.15.12:0/3765616157 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff25c02b020 con 0x558275eafbc0 2024-03-23T23:04:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.318+0000 7ff258c00640 1 -- 172.21.15.12:0/3765616157 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff25c0055a0 con 0x558275eafbc0 2024-03-23T23:04:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.318+0000 7ff2789ce740 10 client.26069 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.318+0000 7ff2789ce740 1 -- 172.21.15.12:0/3765616157 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x558275f20a40 con 0x558275eafbc0 2024-03-23T23:04:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.318+0000 7ff258c00640 1 -- 172.21.15.12:0/3765616157 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff25c02b020 con 0x558275eafbc0 2024-03-23T23:04:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.318+0000 7ff258c00640 1 -- 172.21.15.12:0/3765616157 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ff25c030810 con 0x558275eafbc0 2024-03-23T23:04:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.318+0000 7ff2789ce740 10 client.26069 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.318+0000 7ff2789ce740 10 client.26069 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:04:43.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.318+0000 7ff2789ce740 20 client.26069 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:43.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.318+0000 7ff2789ce740 1 --2- 172.21.15.12:0/3765616157 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558275f212d0 0x558275f236f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:43.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.318+0000 7ff2789ce740 4 client.26069 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:04:43.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.318+0000 7ff2789ce740 1 -- 172.21.15.12:0/3765616157 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558275e1ab50 con 0x558275f212d0 2024-03-23T23:04:43.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.318+0000 7ff26f400640 1 --2- 172.21.15.12:0/3765616157 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558275f212d0 0x558275f236f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:43.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.319+0000 7ff26f400640 1 --2- 172.21.15.12:0/3765616157 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558275f212d0 0x558275f236f0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ced199b90a6974 server_cookie=1fba0ed8e161b7ec in_seq=0 out_seq=0 2024-03-23T23:04:43.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.319+0000 7ff258c00640 10 client.26069 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:43.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.319+0000 7ff258c00640 1 -- 172.21.15.12:0/3765616157 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558275e1ab50 con 0x558275f212d0 2024-03-23T23:04:43.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.319+0000 7ff258c00640 10 client.26069 handle_command_reply: tid=0 2024-03-23T23:04:43.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.319+0000 7ff2789ce740 1 client.26069 shutdown 2024-03-23T23:04:43.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.319+0000 7ff2789ce740 1 -- 172.21.15.12:0/3765616157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eafbc0 msgr2=0x558275ebc8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:43.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.319+0000 7ff2789ce740 1 --2- 172.21.15.12:0/3765616157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eafbc0 0x558275ebc8b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff25c010970 tx=0x7ff25c011f00 comp rx=0 tx=0).stop 2024-03-23T23:04:43.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.319+0000 7ff2789ce740 1 -- 172.21.15.12:0/3765616157 shutdown_connections 2024-03-23T23:04:43.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.319+0000 7ff2789ce740 1 --2- 172.21.15.12:0/3765616157 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558275f212d0 0x558275f236f0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.319+0000 7ff2789ce740 1 --2- 172.21.15.12:0/3765616157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558275eb0580 0x558275ebf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.319+0000 7ff2789ce740 1 --2- 172.21.15.12:0/3765616157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558275eafbc0 0x558275ebc8b0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.319+0000 7ff2789ce740 1 --2- 172.21.15.12:0/3765616157 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558275eae9f0 0x558275ebc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.319+0000 7ff258c00640 0 client.26069 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:43.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.319+0000 7ff2789ce740 1 -- 172.21.15.12:0/3765616157 >> 172.21.15.12:0/3765616157 conn(0x558275ea9de0 msgr2=0x558275ea88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:43.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.320+0000 7ff2789ce740 1 -- 172.21.15.12:0/3765616157 shutdown_connections 2024-03-23T23:04:43.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.320+0000 7ff2789ce740 1 -- 172.21.15.12:0/3765616157 wait complete. 2024-03-23T23:04:43.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.320+0000 7ff2789ce740 20 client.26069 trim_cache size 0 max 16384 2024-03-23T23:04:43.324 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:04:43.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.320+0000 7ff26fe00640 1 -- 172.21.15.12:0/2576397246 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff2540a2800 msgr2=0x7ff2540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:43.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.320+0000 7ff26fe00640 1 --2- 172.21.15.12:0/2576397246 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff2540a2800 0x7ff2540a4cc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff2600c5200 tx=0x7ff260009260 comp rx=0 tx=0).stop 2024-03-23T23:04:43.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.320+0000 7ff26fe00640 1 -- 172.21.15.12:0/2576397246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff2701037c0 msgr2=0x7ff270199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:43.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.320+0000 7ff26fe00640 1 --2- 172.21.15.12:0/2576397246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff2701037c0 0x7ff270199ad0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff26002efd0 tx=0x7ff260005d50 comp rx=0 tx=0).stop 2024-03-23T23:04:43.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.320+0000 7ff26fe00640 1 -- 172.21.15.12:0/2576397246 shutdown_connections 2024-03-23T23:04:43.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.320+0000 7ff26fe00640 1 --2- 172.21.15.12:0/2576397246 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff2540a2800 0x7ff2540a4cc0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.320+0000 7ff26fe00640 1 --2- 172.21.15.12:0/2576397246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff270105370 0x7ff2701a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.320+0000 7ff26fe00640 1 --2- 172.21.15.12:0/2576397246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2701049b0 0x7ff27019a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.320+0000 7ff26fe00640 1 --2- 172.21.15.12:0/2576397246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff2701037c0 0x7ff270199ad0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:43.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.320+0000 7ff26fe00640 1 -- 172.21.15.12:0/2576397246 >> 172.21.15.12:0/2576397246 conn(0x7ff2700fed50 msgr2=0x7ff270107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:43.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.321+0000 7ff26fe00640 1 -- 172.21.15.12:0/2576397246 shutdown_connections 2024-03-23T23:04:43.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:43.321+0000 7ff26fe00640 1 -- 172.21.15.12:0/2576397246 wait complete. 2024-03-23T23:04:44.336 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-23T23:04:44.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:04:44.337 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-23T23:04:44.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.578+0000 7f7a03000640 1 -- 172.21.15.12:0/3056176096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79fc108eb0 msgr2=0x7f79fc109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:44.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.578+0000 7f7a03000640 1 --2- 172.21.15.12:0/3056176096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79fc108eb0 0x7f79fc109330 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f79e4009900 tx=0x7f79e402bc30 comp rx=0 tx=0).stop 2024-03-23T23:04:44.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.578+0000 7f7a03000640 1 -- 172.21.15.12:0/3056176096 shutdown_connections 2024-03-23T23:04:44.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.578+0000 7f7a03000640 1 --2- 172.21.15.12:0/3056176096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79fc109870 0x7f79fc110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.578+0000 7f7a03000640 1 --2- 172.21.15.12:0/3056176096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79fc108eb0 0x7f79fc109330 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.578+0000 7f7a03000640 1 --2- 172.21.15.12:0/3056176096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79fc107cc0 0x7f79fc1080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.578+0000 7f7a03000640 1 -- 172.21.15.12:0/3056176096 >> 172.21.15.12:0/3056176096 conn(0x7f79fc076c60 msgr2=0x7f79fc0790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:44.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.578+0000 7f7a03000640 1 -- 172.21.15.12:0/3056176096 shutdown_connections 2024-03-23T23:04:44.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.579+0000 7f7a03000640 1 -- 172.21.15.12:0/3056176096 wait complete. 2024-03-23T23:04:44.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.580+0000 7f7a03000640 1 Processor -- start 2024-03-23T23:04:44.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.580+0000 7f7a03000640 1 -- start start 2024-03-23T23:04:44.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.580+0000 7f7a03000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79fc107cc0 0x7f79fc19df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.580+0000 7f7a03000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79fc108eb0 0x7f79fc19e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.580+0000 7f7a03000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79fc109870 0x7f79fc1a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.580+0000 7f7a03000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc113970 con 0x7f79fc109870 2024-03-23T23:04:44.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.580+0000 7f7a03000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc1137f0 con 0x7f79fc107cc0 2024-03-23T23:04:44.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.580+0000 7f7a03000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc113af0 con 0x7f79fc108eb0 2024-03-23T23:04:44.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.581+0000 7f79fbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79fc107cc0 0x7f79fc19df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.581+0000 7f79f3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79fc108eb0 0x7f79fc19e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.581+0000 7f79fbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79fc107cc0 0x7f79fc19df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:56216/0 (socket says 172.21.15.12:56216) 2024-03-23T23:04:44.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.581+0000 7f79fbe00640 1 -- 172.21.15.12:0/2987536857 learned_addr learned my addr 172.21.15.12:0/2987536857 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:44.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.581+0000 7f7a01000640 1 --2- 172.21.15.12:0/2987536857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79fc109870 0x7f79fc1a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.581+0000 7f79fbe00640 1 -- 172.21.15.12:0/2987536857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79fc108eb0 msgr2=0x7f79fc19e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:44.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.581+0000 7f79fbe00640 1 --2- 172.21.15.12:0/2987536857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79fc108eb0 0x7f79fc19e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.581+0000 7f79fbe00640 1 -- 172.21.15.12:0/2987536857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79fc109870 msgr2=0x7f79fc1a5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:44.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.581+0000 7f79fbe00640 1 --2- 172.21.15.12:0/2987536857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79fc109870 0x7f79fc1a5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.581+0000 7f79fbe00640 1 -- 172.21.15.12:0/2987536857 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79fc1a5bb0 con 0x7f79fc107cc0 2024-03-23T23:04:44.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.581+0000 7f7a01000640 1 --2- 172.21.15.12:0/2987536857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79fc109870 0x7f79fc1a5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:04:44.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.581+0000 7f79fbe00640 1 --2- 172.21.15.12:0/2987536857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79fc107cc0 0x7f79fc19df90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f79ec00e9c0 tx=0x7f79ec00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:44.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.582+0000 7f79f9600640 1 -- 172.21.15.12:0/2987536857 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec00cde0 con 0x7f79fc107cc0 2024-03-23T23:04:44.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.582+0000 7f79f9600640 1 -- 172.21.15.12:0/2987536857 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79ec010910 con 0x7f79fc107cc0 2024-03-23T23:04:44.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.582+0000 7f7a03000640 1 -- 172.21.15.12:0/2987536857 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79fc1a5ea0 con 0x7f79fc107cc0 2024-03-23T23:04:44.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.582+0000 7f79f9600640 1 -- 172.21.15.12:0/2987536857 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec018a00 con 0x7f79fc107cc0 2024-03-23T23:04:44.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.582+0000 7f7a03000640 1 -- 172.21.15.12:0/2987536857 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79fc1a6310 con 0x7f79fc107cc0 2024-03-23T23:04:44.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.585+0000 7f7a05da4740 1 Processor -- start 2024-03-23T23:04:44.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.585+0000 7f7a05da4740 1 -- start start 2024-03-23T23:04:44.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.586+0000 7f7a05da4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c93aa0a0 0x5576c93aa4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.586+0000 7f7a05da4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576c93aaa90 0x5576c93af590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.586+0000 7f7a05da4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93afc00 0x5576c93b2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.586+0000 7f7a05da4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576c930feb0 con 0x5576c93aaa90 2024-03-23T23:04:44.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.586+0000 7f7a05da4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576c92b8a60 con 0x5576c93afc00 2024-03-23T23:04:44.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.586+0000 7f7a05da4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576c930bf50 con 0x5576c93aa0a0 2024-03-23T23:04:44.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.586+0000 7f7a01000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c93aa0a0 0x5576c93aa4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.586+0000 7f79f3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93afc00 0x5576c93b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.586+0000 7f7a01000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c93aa0a0 0x5576c93aa4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58226/0 (socket says 172.21.15.12:58226) 2024-03-23T23:04:44.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.586+0000 7f7a01000640 1 -- 172.21.15.12:0/1461301266 learned_addr learned my addr 172.21.15.12:0/1461301266 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:44.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.586+0000 7f79f9600640 1 -- 172.21.15.12:0/2987536857 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f79ec01f020 con 0x7f79fc107cc0 2024-03-23T23:04:44.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.586+0000 7f7a01000640 1 -- 172.21.15.12:0/1461301266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93afc00 msgr2=0x5576c93b2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:44.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.586+0000 7f7a01000640 1 --2- 172.21.15.12:0/1461301266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93afc00 0x5576c93b2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.586+0000 7f7a01000640 1 -- 172.21.15.12:0/1461301266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576c93aaa90 msgr2=0x5576c93af590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T23:04:44.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.587+0000 7f7a01000640 1 --2- 172.21.15.12:0/1461301266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576c93aaa90 0x5576c93af590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.587+0000 7f7a01000640 1 -- 172.21.15.12:0/1461301266 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576c90ec7e0 con 0x5576c93aa0a0 2024-03-23T23:04:44.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.587+0000 7f7a01000640 1 --2- 172.21.15.12:0/1461301266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c93aa0a0 0x5576c93aa4c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f79f4009fd0 tx=0x7f79f40108e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4f00b432329d48 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:44.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.587+0000 7f79f1600640 1 -- 172.21.15.12:0/1461301266 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f4011760 con 0x5576c93aa0a0 2024-03-23T23:04:44.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.587+0000 7f79f1600640 1 -- 172.21.15.12:0/1461301266 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79f4011de0 con 0x5576c93aa0a0 2024-03-23T23:04:44.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.587+0000 7f79f1600640 1 -- 172.21.15.12:0/1461301266 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f400e3e0 con 0x5576c93aa0a0 2024-03-23T23:04:44.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.588+0000 7f79f9600640 1 --2- 172.21.15.12:0/2987536857 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f79e00a2880 0x7f79e00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.588+0000 7f79f9600640 1 -- 172.21.15.12:0/2987536857 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f79ec0cec00 con 0x7f79fc107cc0 2024-03-23T23:04:44.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.588+0000 7f7a05da4740 1 -- 172.21.15.12:0/1461301266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c93aa0a0 msgr2=0x5576c93aa4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:44.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.588+0000 7f7a05da4740 1 --2- 172.21.15.12:0/1461301266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c93aa0a0 0x5576c93aa4c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f79f4009fd0 tx=0x7f79f40108e0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.588+0000 7f79fbe00640 1 --2- 172.21.15.12:0/2987536857 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f79e00a2880 0x7f79e00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.588+0000 7f7a05da4740 1 -- 172.21.15.12:0/1461301266 shutdown_connections 2024-03-23T23:04:44.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.588+0000 7f7a05da4740 1 --2- 172.21.15.12:0/1461301266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93afc00 0x5576c93b2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.588+0000 7f7a05da4740 1 --2- 172.21.15.12:0/1461301266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576c93aaa90 0x5576c93af590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.588+0000 7f7a05da4740 1 --2- 172.21.15.12:0/1461301266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c93aa0a0 0x5576c93aa4c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.588+0000 7f7a05da4740 1 -- 172.21.15.12:0/1461301266 >> 172.21.15.12:0/1461301266 conn(0x5576c93a5610 msgr2=0x5576c93a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:44.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.588+0000 7f7a05da4740 1 -- 172.21.15.12:0/1461301266 shutdown_connections 2024-03-23T23:04:44.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.588+0000 7f7a05da4740 1 -- 172.21.15.12:0/1461301266 wait complete. 2024-03-23T23:04:44.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.589+0000 7f79fbe00640 1 --2- 172.21.15.12:0/2987536857 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f79e00a2880 0x7f79e00a4d40 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f79ec0a3820 tx=0x7f79ec010ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:44.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.589+0000 7f7a05da4740 1 -- start start 2024-03-23T23:04:44.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.589+0000 7f7a05da4740 1 Processor -- start 2024-03-23T23:04:44.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.589+0000 7f7a05da4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c93aa0a0 0x5576c93b4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.590+0000 7f7a05da4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93aaa90 0x5576c93b4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.590+0000 7f7a05da4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576c93afc00 0x5576c93bb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.590+0000 7f7a05da4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576c93b2bb0 con 0x5576c93afc00 2024-03-23T23:04:44.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.590+0000 7f7a05da4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576c92cbeb0 con 0x5576c93aaa90 2024-03-23T23:04:44.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.590+0000 7f7a05da4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576c93b2d30 con 0x5576c93aa0a0 2024-03-23T23:04:44.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.590+0000 7f79f3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93aaa90 0x5576c93b4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.590+0000 7f79f3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93aaa90 0x5576c93b4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:56220/0 (socket says 172.21.15.12:56220) 2024-03-23T23:04:44.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.590+0000 7f79f3e00640 1 -- 172.21.15.12:0/829590160 learned_addr learned my addr 172.21.15.12:0/829590160 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:44.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.590+0000 7f7a05da4740 10 client.0 _ll_register_callbacks cb 0x5576c92c0150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:44.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.590+0000 7f7a01000640 1 --2- 172.21.15.12:0/829590160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c93aa0a0 0x5576c93b4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.590+0000 7f7a01000640 1 --2- 172.21.15.12:0/829590160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576c93afc00 0x5576c93bb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.590+0000 7f79f3e00640 1 -- 172.21.15.12:0/829590160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c93aa0a0 msgr2=0x5576c93b4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:44.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.590+0000 7f79f3e00640 1 --2- 172.21.15.12:0/829590160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c93aa0a0 0x5576c93b4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.590+0000 7f79f3e00640 1 -- 172.21.15.12:0/829590160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576c93afc00 msgr2=0x5576c93bb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:44.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.590+0000 7f79f3e00640 1 --2- 172.21.15.12:0/829590160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576c93afc00 0x5576c93bb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.590+0000 7f79f3e00640 1 -- 172.21.15.12:0/829590160 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5576c914cf70 con 0x5576c93aaa90 2024-03-23T23:04:44.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.591+0000 7f79f3e00640 1 --2- 172.21.15.12:0/829590160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93aaa90 0x5576c93b4780 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f79e4005ab0 tx=0x7f79e4004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f4c6628a45ef1cc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:44.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.591+0000 7f79ea000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:04:44.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.591+0000 7f79ea000640 1 -- 172.21.15.12:0/829590160 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e4005d50 con 0x5576c93aaa90 2024-03-23T23:04:44.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.591+0000 7f79ea000640 1 -- 172.21.15.12:0/829590160 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f79e40046c0 con 0x5576c93aaa90 2024-03-23T23:04:44.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.591+0000 7f7a05da4740 1 -- 172.21.15.12:0/829590160 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f79f4011de0 con 0x5576c93aaa90 2024-03-23T23:04:44.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.591+0000 7f79ea000640 1 -- 172.21.15.12:0/829590160 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79e40048c0 con 0x5576c93aaa90 2024-03-23T23:04:44.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.591+0000 7f7a05da4740 10 client.26081 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:44.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.591+0000 7f7a05da4740 1 -- 172.21.15.12:0/829590160 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5576c8fe4170 con 0x5576c93aaa90 2024-03-23T23:04:44.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.592+0000 7f79ea000640 1 -- 172.21.15.12:0/829590160 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79f4011de0 con 0x5576c93aaa90 2024-03-23T23:04:44.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.592+0000 7f79ea000640 1 -- 172.21.15.12:0/829590160 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f79e40038f0 con 0x5576c93aaa90 2024-03-23T23:04:44.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.592+0000 7f7a05da4740 10 client.26081 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:44.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.592+0000 7f7a05da4740 10 client.26081 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:04:44.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.592+0000 7f7a05da4740 20 client.26081 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:44.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.592+0000 7f7a05da4740 1 --2- 172.21.15.12:0/829590160 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5576c93c0070 0x5576c93c2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.592+0000 7f7a05da4740 4 client.26081 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:04:44.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.592+0000 7f7a05da4740 1 -- 172.21.15.12:0/829590160 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5576c931d3f0 con 0x5576c93c0070 2024-03-23T23:04:44.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.592+0000 7f7a01000640 1 --2- 172.21.15.12:0/829590160 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5576c93c0070 0x5576c93c2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.593+0000 7f7a01000640 1 --2- 172.21.15.12:0/829590160 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5576c93c0070 0x5576c93c2490 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e403d9b9618a567f server_cookie=2c244a58548977a9 in_seq=0 out_seq=0 2024-03-23T23:04:44.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.593+0000 7f79ea000640 10 client.26081 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:44.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.593+0000 7f79ea000640 1 -- 172.21.15.12:0/829590160 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5576c931d3f0 con 0x5576c93c0070 2024-03-23T23:04:44.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.593+0000 7f79ea000640 10 client.26081 handle_command_reply: tid=0 2024-03-23T23:04:44.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.594+0000 7f7a05da4740 1 client.26081 shutdown 2024-03-23T23:04:44.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.594+0000 7f7a05da4740 1 -- 172.21.15.12:0/829590160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93aaa90 msgr2=0x5576c93b4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:44.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.594+0000 7f7a05da4740 1 --2- 172.21.15.12:0/829590160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93aaa90 0x5576c93b4780 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f79e4005ab0 tx=0x7f79e4004270 comp rx=0 tx=0).stop 2024-03-23T23:04:44.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.594+0000 7f7a05da4740 1 -- 172.21.15.12:0/829590160 shutdown_connections 2024-03-23T23:04:44.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.594+0000 7f7a05da4740 1 --2- 172.21.15.12:0/829590160 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5576c93c0070 0x5576c93c2490 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.594+0000 7f7a05da4740 1 --2- 172.21.15.12:0/829590160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576c93afc00 0x5576c93bb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.594+0000 7f79ea000640 0 client.26081 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:44.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.594+0000 7f7a05da4740 1 --2- 172.21.15.12:0/829590160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93aaa90 0x5576c93b4780 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.594+0000 7f7a05da4740 1 --2- 172.21.15.12:0/829590160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c93aa0a0 0x5576c93b4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.594+0000 7f7a05da4740 1 -- 172.21.15.12:0/829590160 >> 172.21.15.12:0/829590160 conn(0x5576c93a5610 msgr2=0x5576c93a4830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:44.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.594+0000 7f7a05da4740 1 -- 172.21.15.12:0/829590160 shutdown_connections 2024-03-23T23:04:44.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.594+0000 7f7a05da4740 1 -- 172.21.15.12:0/829590160 wait complete. 2024-03-23T23:04:44.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.594+0000 7f7a05da4740 20 client.26081 trim_cache size 0 max 16384 2024-03-23T23:04:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.614+0000 7f7a05da4740 1 Processor -- start 2024-03-23T23:04:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.614+0000 7f7a05da4740 1 -- start start 2024-03-23T23:04:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.614+0000 7f7a05da4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576c93a5610 0x5576c93ab530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.614+0000 7f7a05da4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93aa0a0 0x5576c93bb780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.614+0000 7f7a05da4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c93aaa90 0x5576c93aba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.614+0000 7f7a05da4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc1b1510 con 0x5576c93a5610 2024-03-23T23:04:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.614+0000 7f7a05da4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc1af010 con 0x5576c93aa0a0 2024-03-23T23:04:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.614+0000 7f7a05da4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc1ad0b0 con 0x5576c93aaa90 2024-03-23T23:04:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.615+0000 7f79f3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93aa0a0 0x5576c93bb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.615+0000 7f79f3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93aa0a0 0x5576c93bb780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:56226/0 (socket says 172.21.15.12:56226) 2024-03-23T23:04:44.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.615+0000 7f79f3e00640 1 -- 172.21.15.12:0/1184080164 learned_addr learned my addr 172.21.15.12:0/1184080164 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:44.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.615+0000 7f7a01000640 1 --2- 172.21.15.12:0/1184080164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576c93a5610 0x5576c93ab530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.615+0000 7f79f3e00640 1 -- 172.21.15.12:0/1184080164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c93aaa90 msgr2=0x5576c93aba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:44.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.615+0000 7f7a01000640 1 --2- 172.21.15.12:0/1184080164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c93aaa90 0x5576c93aba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.615+0000 7f79f3e00640 1 --2- 172.21.15.12:0/1184080164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c93aaa90 0x5576c93aba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.615+0000 7f79f3e00640 1 -- 172.21.15.12:0/1184080164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576c93a5610 msgr2=0x5576c93ab530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:44.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.615+0000 7f79f3e00640 1 --2- 172.21.15.12:0/1184080164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576c93a5610 0x5576c93ab530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.615+0000 7f79f3e00640 1 -- 172.21.15.12:0/1184080164 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576c93c2800 con 0x5576c93aa0a0 2024-03-23T23:04:44.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.615+0000 7f7a01000640 1 --2- 172.21.15.12:0/1184080164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c93aaa90 0x5576c93aba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:04:44.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.615+0000 7f79f3e00640 1 --2- 172.21.15.12:0/1184080164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93aa0a0 0x5576c93bb780 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f79e402c170 tx=0x7f79e402bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8257d0679d0985a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:44.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.616+0000 7f79f0c00640 1 -- 172.21.15.12:0/1184080164 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e4005d50 con 0x5576c93aa0a0 2024-03-23T23:04:44.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.616+0000 7f79f0c00640 1 -- 172.21.15.12:0/1184080164 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79e405a070 con 0x5576c93aa0a0 2024-03-23T23:04:44.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.616+0000 7f79f0c00640 1 -- 172.21.15.12:0/1184080164 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e404baf0 con 0x5576c93aa0a0 2024-03-23T23:04:44.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.616+0000 7f7a05da4740 1 -- 172.21.15.12:0/1184080164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93aa0a0 msgr2=0x5576c93bb780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:44.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.616+0000 7f7a05da4740 1 --2- 172.21.15.12:0/1184080164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93aa0a0 0x5576c93bb780 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f79e402c170 tx=0x7f79e402bc30 comp rx=0 tx=0).stop 2024-03-23T23:04:44.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.616+0000 7f7a05da4740 1 -- 172.21.15.12:0/1184080164 shutdown_connections 2024-03-23T23:04:44.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.616+0000 7f7a05da4740 1 --2- 172.21.15.12:0/1184080164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c93aaa90 0x5576c93aba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.616+0000 7f7a05da4740 1 --2- 172.21.15.12:0/1184080164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93aa0a0 0x5576c93bb780 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.616+0000 7f7a05da4740 1 --2- 172.21.15.12:0/1184080164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576c93a5610 0x5576c93ab530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.616+0000 7f7a05da4740 1 -- 172.21.15.12:0/1184080164 >> 172.21.15.12:0/1184080164 conn(0x5576c93a94a0 msgr2=0x5576c93b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:44.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.617+0000 7f7a05da4740 1 -- 172.21.15.12:0/1184080164 shutdown_connections 2024-03-23T23:04:44.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.617+0000 7f7a05da4740 1 -- 172.21.15.12:0/1184080164 wait complete. 2024-03-23T23:04:44.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.617+0000 7f7a05da4740 1 -- start start 2024-03-23T23:04:44.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.617+0000 7f7a05da4740 1 Processor -- start 2024-03-23T23:04:44.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.618+0000 7f7a05da4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c93a5610 0x5576c93bb300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.618+0000 7f7a05da4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93aa0a0 0x5576c93b9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.618+0000 7f7a05da4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576c93aaa90 0x5576c93b9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.618+0000 7f7a05da4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc1ad6b0 con 0x5576c93aaa90 2024-03-23T23:04:44.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.618+0000 7f7a05da4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc1ae9b0 con 0x5576c93aa0a0 2024-03-23T23:04:44.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.618+0000 7f7a05da4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc1adcf0 con 0x5576c93a5610 2024-03-23T23:04:44.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.618+0000 7f7a05da4740 10 client.0 _ll_register_callbacks cb 0x7f79e4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:44.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.618+0000 7f79f3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93aa0a0 0x5576c93b9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.618+0000 7f7a01000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c93a5610 0x5576c93bb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.618+0000 7f79f3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93aa0a0 0x5576c93b9910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:56236/0 (socket says 172.21.15.12:56236) 2024-03-23T23:04:44.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.618+0000 7f79f3e00640 1 -- 172.21.15.12:0/2770975148 learned_addr learned my addr 172.21.15.12:0/2770975148 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:44.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.618+0000 7f7a01000640 1 --2- 172.21.15.12:0/2770975148 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576c93aaa90 0x5576c93b9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.618+0000 7f79f3e00640 1 -- 172.21.15.12:0/2770975148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c93a5610 msgr2=0x5576c93bb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:44.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.618+0000 7f79f3e00640 1 --2- 172.21.15.12:0/2770975148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c93a5610 0x5576c93bb300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.618+0000 7f79f3e00640 1 -- 172.21.15.12:0/2770975148 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576c93aaa90 msgr2=0x5576c93b9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:44.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.618+0000 7f79f3e00640 1 --2- 172.21.15.12:0/2770975148 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576c93aaa90 0x5576c93b9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.618+0000 7f79f3e00640 1 -- 172.21.15.12:0/2770975148 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5576c93ba390 con 0x5576c93aa0a0 2024-03-23T23:04:44.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.619+0000 7f79f3e00640 1 --2- 172.21.15.12:0/2770975148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93aa0a0 0x5576c93b9910 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f79e4009850 tx=0x7f79e4005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f222215d74f6d89 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:44.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.619+0000 7f79ea000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:04:44.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.619+0000 7f79ea000640 1 -- 172.21.15.12:0/2770975148 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e4062070 con 0x5576c93aa0a0 2024-03-23T23:04:44.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.619+0000 7f79ea000640 1 -- 172.21.15.12:0/2770975148 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f79e404ba20 con 0x5576c93aa0a0 2024-03-23T23:04:44.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.619+0000 7f7a05da4740 1 -- 172.21.15.12:0/2770975148 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f79e405a070 con 0x5576c93aa0a0 2024-03-23T23:04:44.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.619+0000 7f79ea000640 1 -- 172.21.15.12:0/2770975148 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79e406e050 con 0x5576c93aa0a0 2024-03-23T23:04:44.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.619+0000 7f7a05da4740 10 client.26087 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:44.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.619+0000 7f7a05da4740 1 -- 172.21.15.12:0/2770975148 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5576c941c550 con 0x5576c93aa0a0 2024-03-23T23:04:44.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.619+0000 7f79ea000640 1 -- 172.21.15.12:0/2770975148 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79e405a070 con 0x5576c93aa0a0 2024-03-23T23:04:44.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.620+0000 7f79ea000640 1 -- 172.21.15.12:0/2770975148 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f79e4005420 con 0x5576c93aa0a0 2024-03-23T23:04:44.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.620+0000 7f7a05da4740 10 client.26087 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:44.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.620+0000 7f7a05da4740 10 client.26087 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:04:44.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.620+0000 7f7a05da4740 20 client.26087 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:44.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.620+0000 7f7a05da4740 1 --2- 172.21.15.12:0/2770975148 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5576c93afc00 0x5576c941eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.620+0000 7f7a05da4740 4 client.26087 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-23T23:04:44.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.620+0000 7f7a05da4740 1 -- 172.21.15.12:0/2770975148 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5576c9317740 con 0x5576c93afc00 2024-03-23T23:04:44.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.620+0000 7f7a01000640 1 --2- 172.21.15.12:0/2770975148 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5576c93afc00 0x5576c941eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.620+0000 7f7a01000640 1 --2- 172.21.15.12:0/2770975148 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5576c93afc00 0x5576c941eec0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd2635e5daa639f server_cookie=240c371dabbcaf4d in_seq=0 out_seq=0 2024-03-23T23:04:44.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.620+0000 7f79ea000640 10 client.26087 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:44.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.622+0000 7f79ea000640 1 -- 172.21.15.12:0/2770975148 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5576c9317740 con 0x5576c93afc00 2024-03-23T23:04:44.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.622+0000 7f79ea000640 10 client.26087 handle_command_reply: tid=0 2024-03-23T23:04:44.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.622+0000 7f7a05da4740 1 client.26087 shutdown 2024-03-23T23:04:44.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.622+0000 7f7a05da4740 1 -- 172.21.15.12:0/2770975148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93aa0a0 msgr2=0x5576c93b9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:44.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.622+0000 7f7a05da4740 1 --2- 172.21.15.12:0/2770975148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93aa0a0 0x5576c93b9910 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f79e4009850 tx=0x7f79e4005a20 comp rx=0 tx=0).stop 2024-03-23T23:04:44.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.622+0000 7f7a05da4740 1 -- 172.21.15.12:0/2770975148 shutdown_connections 2024-03-23T23:04:44.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.622+0000 7f7a05da4740 1 --2- 172.21.15.12:0/2770975148 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5576c93afc00 0x5576c941eec0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.622+0000 7f7a05da4740 1 --2- 172.21.15.12:0/2770975148 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576c93aaa90 0x5576c93b9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.622+0000 7f7a05da4740 1 --2- 172.21.15.12:0/2770975148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c93aa0a0 0x5576c93b9910 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.622+0000 7f7a05da4740 1 --2- 172.21.15.12:0/2770975148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c93a5610 0x5576c93bb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.622+0000 7f79ea000640 0 client.26087 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:44.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.622+0000 7f7a05da4740 1 -- 172.21.15.12:0/2770975148 >> 172.21.15.12:0/2770975148 conn(0x5576c93a94a0 msgr2=0x5576c93b3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:44.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.622+0000 7f7a05da4740 1 -- 172.21.15.12:0/2770975148 shutdown_connections 2024-03-23T23:04:44.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.622+0000 7f7a05da4740 1 -- 172.21.15.12:0/2770975148 wait complete. 2024-03-23T23:04:44.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.622+0000 7f7a05da4740 20 client.26087 trim_cache size 0 max 16384 2024-03-23T23:04:44.625 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:04:44.626 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-03-23T23:04:44.626 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "21daee86-0251-46c9-b8ec-8d59e79642a3", 2024-03-23T23:04:44.626 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-03-23T23:04:44.626 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:04:44.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.623+0000 7f7a03000640 1 -- 172.21.15.12:0/2987536857 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f79e00a2880 msgr2=0x7f79e00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:44.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.623+0000 7f7a03000640 1 --2- 172.21.15.12:0/2987536857 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f79e00a2880 0x7f79e00a4d40 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f79ec0a3820 tx=0x7f79ec010ed0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.623+0000 7f7a03000640 1 -- 172.21.15.12:0/2987536857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79fc107cc0 msgr2=0x7f79fc19df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:44.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.623+0000 7f7a03000640 1 --2- 172.21.15.12:0/2987536857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79fc107cc0 0x7f79fc19df90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f79ec00e9c0 tx=0x7f79ec00ee80 comp rx=0 tx=0).stop 2024-03-23T23:04:44.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.623+0000 7f7a03000640 1 -- 172.21.15.12:0/2987536857 shutdown_connections 2024-03-23T23:04:44.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.623+0000 7f7a03000640 1 --2- 172.21.15.12:0/2987536857 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f79e00a2880 0x7f79e00a4d40 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.623+0000 7f7a03000640 1 --2- 172.21.15.12:0/2987536857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79fc109870 0x7f79fc1a5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.623+0000 7f7a03000640 1 --2- 172.21.15.12:0/2987536857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79fc108eb0 0x7f79fc19e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.623+0000 7f7a03000640 1 --2- 172.21.15.12:0/2987536857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79fc107cc0 0x7f79fc19df90 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.624+0000 7f7a03000640 1 -- 172.21.15.12:0/2987536857 >> 172.21.15.12:0/2987536857 conn(0x7f79fc076c60 msgr2=0x7f79fc10c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:44.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.624+0000 7f7a03000640 1 -- 172.21.15.12:0/2987536857 shutdown_connections 2024-03-23T23:04:44.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.624+0000 7f7a03000640 1 -- 172.21.15.12:0/2987536857 wait complete. 2024-03-23T23:04:44.644 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:04:44.644 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:04:44.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.870+0000 7fbdab200640 1 -- 172.21.15.12:0/1467491178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdac107bc0 msgr2=0x7fbdac107fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:44.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.870+0000 7fbdab200640 1 --2- 172.21.15.12:0/1467491178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdac107bc0 0x7fbdac107fe0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fbd94009900 tx=0x7fbd9402bc30 comp rx=0 tx=0).stop 2024-03-23T23:04:44.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.870+0000 7fbdab200640 1 -- 172.21.15.12:0/1467491178 shutdown_connections 2024-03-23T23:04:44.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.870+0000 7fbdab200640 1 --2- 172.21.15.12:0/1467491178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbdac109770 0x7fbdac110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.870+0000 7fbdab200640 1 --2- 172.21.15.12:0/1467491178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdac108db0 0x7fbdac109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.870+0000 7fbdab200640 1 --2- 172.21.15.12:0/1467491178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdac107bc0 0x7fbdac107fe0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.870+0000 7fbdab200640 1 -- 172.21.15.12:0/1467491178 >> 172.21.15.12:0/1467491178 conn(0x7fbdac076b70 msgr2=0x7fbdac078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:44.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.870+0000 7fbdab200640 1 -- 172.21.15.12:0/1467491178 shutdown_connections 2024-03-23T23:04:44.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.870+0000 7fbdab200640 1 -- 172.21.15.12:0/1467491178 wait complete. 2024-03-23T23:04:44.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.872+0000 7fbdab200640 1 Processor -- start 2024-03-23T23:04:44.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.872+0000 7fbdab200640 1 -- start start 2024-03-23T23:04:44.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.872+0000 7fbdab200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdac107bc0 0x7fbdac19df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.872+0000 7fbdab200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbdac108db0 0x7fbdac19e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.872+0000 7fbdab200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdac109770 0x7fbdac1a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.872+0000 7fbdab200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdac113890 con 0x7fbdac107bc0 2024-03-23T23:04:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.872+0000 7fbdab200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdac113710 con 0x7fbdac108db0 2024-03-23T23:04:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.872+0000 7fbdab200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdac113a10 con 0x7fbdac109770 2024-03-23T23:04:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.872+0000 7fbda9400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbdac108db0 0x7fbdac19e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.872+0000 7fbda9400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbdac108db0 0x7fbdac19e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:56250/0 (socket says 172.21.15.12:56250) 2024-03-23T23:04:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.872+0000 7fbda9400640 1 -- 172.21.15.12:0/3993993637 learned_addr learned my addr 172.21.15.12:0/3993993637 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.873+0000 7fbda9e00640 1 --2- 172.21.15.12:0/3993993637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdac107bc0 0x7fbdac19df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.873+0000 7fbdaa800640 1 --2- 172.21.15.12:0/3993993637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdac109770 0x7fbdac1a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.873+0000 7fbda9400640 1 -- 172.21.15.12:0/3993993637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdac109770 msgr2=0x7fbdac1a54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.873+0000 7fbda9400640 1 --2- 172.21.15.12:0/3993993637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdac109770 0x7fbdac1a54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.873+0000 7fbda9400640 1 -- 172.21.15.12:0/3993993637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdac107bc0 msgr2=0x7fbdac19df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.873+0000 7fbda9400640 1 --2- 172.21.15.12:0/3993993637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdac107bc0 0x7fbdac19df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.873+0000 7fbda9400640 1 -- 172.21.15.12:0/3993993637 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdac1a5b20 con 0x7fbdac108db0 2024-03-23T23:04:44.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.873+0000 7fbda9400640 1 --2- 172.21.15.12:0/3993993637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbdac108db0 0x7fbdac19e440 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c009d50 tx=0x7fbd9c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:44.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.873+0000 7fbda2a00640 1 -- 172.21.15.12:0/3993993637 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c003b70 con 0x7fbdac108db0 2024-03-23T23:04:44.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.874+0000 7fbda2a00640 1 -- 172.21.15.12:0/3993993637 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd9c003d10 con 0x7fbdac108db0 2024-03-23T23:04:44.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.874+0000 7fbda2a00640 1 -- 172.21.15.12:0/3993993637 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c010a20 con 0x7fbdac108db0 2024-03-23T23:04:44.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.874+0000 7fbdab200640 1 -- 172.21.15.12:0/3993993637 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdac1a5e10 con 0x7fbdac108db0 2024-03-23T23:04:44.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.874+0000 7fbdab200640 1 -- 172.21.15.12:0/3993993637 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdac1a6320 con 0x7fbdac108db0 2024-03-23T23:04:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.877+0000 7fbda2a00640 1 -- 172.21.15.12:0/3993993637 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbd9c003ec0 con 0x7fbdac108db0 2024-03-23T23:04:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.877+0000 7fbdb3a6c740 1 Processor -- start 2024-03-23T23:04:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.877+0000 7fbdb3a6c740 1 -- start start 2024-03-23T23:04:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.877+0000 7fbdb3a6c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475da050 0x55c3475da470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.877+0000 7fbdb3a6c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475daa40 0x55c3475df540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.877+0000 7fbdb3a6c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3475dfbb0 0x55c3475e2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.877+0000 7fbdb3a6c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c34753f250 con 0x55c3475dfbb0 2024-03-23T23:04:44.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.877+0000 7fbdb3a6c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3474e7a30 con 0x55c3475daa40 2024-03-23T23:04:44.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.877+0000 7fbdb3a6c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c34753b2f0 con 0x55c3475da050 2024-03-23T23:04:44.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.878+0000 7fbda9e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475daa40 0x55c3475df540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.878+0000 7fbdaa800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475da050 0x55c3475da470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.878+0000 7fbda9e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475daa40 0x55c3475df540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:56254/0 (socket says 172.21.15.12:56254) 2024-03-23T23:04:44.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.878+0000 7fbda9e00640 1 -- 172.21.15.12:0/3983901046 learned_addr learned my addr 172.21.15.12:0/3983901046 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:44.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.878+0000 7fbdaa800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475da050 0x55c3475da470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58288/0 (socket says 172.21.15.12:58288) 2024-03-23T23:04:44.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.878+0000 7fbda9400640 1 --2- 172.21.15.12:0/3983901046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3475dfbb0 0x55c3475e2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.878+0000 7fbdaa800640 1 -- 172.21.15.12:0/3983901046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475daa40 msgr2=0x55c3475df540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:44.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.878+0000 7fbdaa800640 1 --2- 172.21.15.12:0/3983901046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475daa40 0x55c3475df540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.878+0000 7fbdaa800640 1 -- 172.21.15.12:0/3983901046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3475dfbb0 msgr2=0x55c3475e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:44.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.878+0000 7fbdaa800640 1 --2- 172.21.15.12:0/3983901046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3475dfbb0 0x55c3475e2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.878+0000 7fbdaa800640 1 -- 172.21.15.12:0/3983901046 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3474d2690 con 0x55c3475da050 2024-03-23T23:04:44.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.878+0000 7fbda9e00640 1 --2- 172.21.15.12:0/3983901046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475daa40 0x55c3475df540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:44.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.879+0000 7fbdaa800640 1 --2- 172.21.15.12:0/3983901046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475da050 0x55c3475da470 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009b80 tx=0x7fbda4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3bff4e4efd162e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:44.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.879+0000 7fbd9b400640 1 -- 172.21.15.12:0/3983901046 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda4004030 con 0x55c3475da050 2024-03-23T23:04:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.879+0000 7fbd9b400640 1 -- 172.21.15.12:0/3983901046 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbda4005500 con 0x55c3475da050 2024-03-23T23:04:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.879+0000 7fbd9b400640 1 -- 172.21.15.12:0/3983901046 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda4004dd0 con 0x55c3475da050 2024-03-23T23:04:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.879+0000 7fbda2a00640 1 --2- 172.21.15.12:0/3993993637 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbd900a2880 0x7fbd900a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.879+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/3983901046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475da050 msgr2=0x55c3475da470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.879+0000 7fbdb3a6c740 1 --2- 172.21.15.12:0/3983901046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475da050 0x55c3475da470 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009b80 tx=0x7fbda4011810 comp rx=0 tx=0).stop 2024-03-23T23:04:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.879+0000 7fbda2a00640 1 -- 172.21.15.12:0/3993993637 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fbd9c0cf0a0 con 0x7fbdac108db0 2024-03-23T23:04:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.880+0000 7fbda9e00640 1 --2- 172.21.15.12:0/3993993637 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbd900a2880 0x7fbd900a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.880+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/3983901046 shutdown_connections 2024-03-23T23:04:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.880+0000 7fbdb3a6c740 1 --2- 172.21.15.12:0/3983901046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3475dfbb0 0x55c3475e2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.880+0000 7fbdb3a6c740 1 --2- 172.21.15.12:0/3983901046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475daa40 0x55c3475df540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.880+0000 7fbdb3a6c740 1 --2- 172.21.15.12:0/3983901046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475da050 0x55c3475da470 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.880+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/3983901046 >> 172.21.15.12:0/3983901046 conn(0x55c3475a2200 msgr2=0x55c3475d59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.880+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/3983901046 shutdown_connections 2024-03-23T23:04:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.880+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/3983901046 wait complete. 2024-03-23T23:04:44.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.880+0000 7fbdb3a6c740 1 -- start start 2024-03-23T23:04:44.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.880+0000 7fbda9e00640 1 --2- 172.21.15.12:0/3993993637 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbd900a2880 0x7fbd900a4d40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x55c3475db4b0 tx=0x7fbd9402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:44.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.881+0000 7fbdb3a6c740 1 Processor -- start 2024-03-23T23:04:44.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.881+0000 7fbdb3a6c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3475da050 0x55c3475e4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.881+0000 7fbdb3a6c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475daa40 0x55c3475e45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.881+0000 7fbdb3a6c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475dfbb0 0x55c3475eb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.881+0000 7fbdb3a6c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3475e2b60 con 0x55c3475da050 2024-03-23T23:04:44.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.881+0000 7fbdb3a6c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3474fae80 con 0x55c3475dfbb0 2024-03-23T23:04:44.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.881+0000 7fbdb3a6c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3475e2ce0 con 0x55c3475daa40 2024-03-23T23:04:44.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.881+0000 7fbdb3a6c740 10 client.0 _ll_register_callbacks cb 0x55c34750d130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:44.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.881+0000 7fbda9400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475daa40 0x55c3475e45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.881+0000 7fbdaa800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475dfbb0 0x55c3475eb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.881+0000 7fbda9e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3475da050 0x55c3475e4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.881+0000 7fbda9400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475daa40 0x55c3475e45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58298/0 (socket says 172.21.15.12:58298) 2024-03-23T23:04:44.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.881+0000 7fbdaa800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475dfbb0 0x55c3475eb5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:56264/0 (socket says 172.21.15.12:56264) 2024-03-23T23:04:44.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.881+0000 7fbda9400640 1 -- 172.21.15.12:0/2982202016 learned_addr learned my addr 172.21.15.12:0/2982202016 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:44.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.882+0000 7fbdaa800640 1 -- 172.21.15.12:0/2982202016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475daa40 msgr2=0x55c3475e45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:44.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.882+0000 7fbdaa800640 1 --2- 172.21.15.12:0/2982202016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475daa40 0x55c3475e45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.882+0000 7fbdaa800640 1 -- 172.21.15.12:0/2982202016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3475da050 msgr2=0x55c3475e4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:44.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.882+0000 7fbdaa800640 1 --2- 172.21.15.12:0/2982202016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3475da050 0x55c3475e4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.882+0000 7fbdaa800640 1 -- 172.21.15.12:0/2982202016 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c34731d7d0 con 0x55c3475dfbb0 2024-03-23T23:04:44.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.882+0000 7fbda9400640 1 --2- 172.21.15.12:0/2982202016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475daa40 0x55c3475e45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:44.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.882+0000 7fbdaa800640 1 --2- 172.21.15.12:0/2982202016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475dfbb0 0x55c3475eb5e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009b50 tx=0x7fbda4005170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d92318a739c64623 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:44.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.882+0000 7fbd8fe00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:04:44.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.882+0000 7fbd8fe00640 1 -- 172.21.15.12:0/2982202016 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda4028020 con 0x55c3475dfbb0 2024-03-23T23:04:44.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.883+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/2982202016 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbda4005500 con 0x55c3475dfbb0 2024-03-23T23:04:44.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.883+0000 7fbd8fe00640 1 -- 172.21.15.12:0/2982202016 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fbda4024620 con 0x55c3475dfbb0 2024-03-23T23:04:44.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.883+0000 7fbd8fe00640 1 -- 172.21.15.12:0/2982202016 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbda4005360 con 0x55c3475dfbb0 2024-03-23T23:04:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.883+0000 7fbdb3a6c740 10 client.26099 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.883+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/2982202016 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c3472533f0 con 0x55c3475dfbb0 2024-03-23T23:04:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.883+0000 7fbd8fe00640 1 -- 172.21.15.12:0/2982202016 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbda4005500 con 0x55c3475dfbb0 2024-03-23T23:04:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.883+0000 7fbd8fe00640 1 -- 172.21.15.12:0/2982202016 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fbda4018070 con 0x55c3475dfbb0 2024-03-23T23:04:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.883+0000 7fbdb3a6c740 10 client.26099 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.883+0000 7fbdb3a6c740 10 client.26099 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:04:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.883+0000 7fbdb3a6c740 20 client.26099 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.883+0000 7fbdb3a6c740 1 --2- 172.21.15.12:0/2982202016 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c3475efb70 0x55c3475f1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.883+0000 7fbdb3a6c740 4 client.26099 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:04:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.883+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/2982202016 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c34754c790 con 0x55c3475efb70 2024-03-23T23:04:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.884+0000 7fbda9e00640 1 --2- 172.21.15.12:0/2982202016 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c3475efb70 0x55c3475f1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.884+0000 7fbda9e00640 1 --2- 172.21.15.12:0/2982202016 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c3475efb70 0x55c3475f1f60 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7dd285aa9942edbd server_cookie=c3a5c89a639623b0 in_seq=0 out_seq=0 2024-03-23T23:04:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.884+0000 7fbd8fe00640 10 client.26099 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.885+0000 7fbd8fe00640 1 -- 172.21.15.12:0/2982202016 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c34754c790 con 0x55c3475efb70 2024-03-23T23:04:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.885+0000 7fbd8fe00640 10 client.26099 handle_command_reply: tid=0 2024-03-23T23:04:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.885+0000 7fbdb3a6c740 1 client.26099 shutdown 2024-03-23T23:04:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.885+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/2982202016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475dfbb0 msgr2=0x55c3475eb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.885+0000 7fbdb3a6c740 1 --2- 172.21.15.12:0/2982202016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475dfbb0 0x55c3475eb5e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009b50 tx=0x7fbda4005170 comp rx=0 tx=0).stop 2024-03-23T23:04:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.885+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/2982202016 shutdown_connections 2024-03-23T23:04:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.885+0000 7fbdb3a6c740 1 --2- 172.21.15.12:0/2982202016 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c3475efb70 0x55c3475f1f60 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.886+0000 7fbdb3a6c740 1 --2- 172.21.15.12:0/2982202016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475dfbb0 0x55c3475eb5e0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.886+0000 7fbdb3a6c740 1 --2- 172.21.15.12:0/2982202016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475daa40 0x55c3475e45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.886+0000 7fbdb3a6c740 1 --2- 172.21.15.12:0/2982202016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3475da050 0x55c3475e4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.886+0000 7fbd8fe00640 0 client.26099 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.886+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/2982202016 >> 172.21.15.12:0/2982202016 conn(0x55c3475a2200 msgr2=0x55c3475d4bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.886+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/2982202016 shutdown_connections 2024-03-23T23:04:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.886+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/2982202016 wait complete. 2024-03-23T23:04:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.886+0000 7fbdb3a6c740 20 client.26099 trim_cache size 0 max 16384 2024-03-23T23:04:44.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.904+0000 7fbdb3a6c740 1 Processor -- start 2024-03-23T23:04:44.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.904+0000 7fbdb3a6c740 1 -- start start 2024-03-23T23:04:44.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.905+0000 7fbdb3a6c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3475da050 0x55c3475e3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.905+0000 7fbdb3a6c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475daa40 0x55c3475eb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.905+0000 7fbdb3a6c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475dfbb0 0x55c3475d9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.905+0000 7fbdb3a6c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdac1b1470 con 0x55c3475da050 2024-03-23T23:04:44.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.905+0000 7fbdb3a6c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdac1aef70 con 0x55c3475dfbb0 2024-03-23T23:04:44.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.905+0000 7fbdb3a6c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdac1ad010 con 0x55c3475daa40 2024-03-23T23:04:44.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.905+0000 7fbda9400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475daa40 0x55c3475eb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.905+0000 7fbdaa800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475dfbb0 0x55c3475d9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.905+0000 7fbda9400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475daa40 0x55c3475eb5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58304/0 (socket says 172.21.15.12:58304) 2024-03-23T23:04:44.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.905+0000 7fbda9400640 1 -- 172.21.15.12:0/3900426890 learned_addr learned my addr 172.21.15.12:0/3900426890 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:44.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.905+0000 7fbda9e00640 1 --2- 172.21.15.12:0/3900426890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3475da050 0x55c3475e3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.905+0000 7fbdaa800640 1 -- 172.21.15.12:0/3900426890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475daa40 msgr2=0x55c3475eb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:44.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.905+0000 7fbdaa800640 1 --2- 172.21.15.12:0/3900426890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475daa40 0x55c3475eb5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.906+0000 7fbdaa800640 1 -- 172.21.15.12:0/3900426890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3475da050 msgr2=0x55c3475e3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:44.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.906+0000 7fbdaa800640 1 --2- 172.21.15.12:0/3900426890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3475da050 0x55c3475e3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.906+0000 7fbdaa800640 1 -- 172.21.15.12:0/3900426890 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3472130f0 con 0x55c3475dfbb0 2024-03-23T23:04:44.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.906+0000 7fbda9400640 1 --2- 172.21.15.12:0/3900426890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475daa40 0x55c3475eb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:44.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.906+0000 7fbda9e00640 1 --2- 172.21.15.12:0/3900426890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3475da050 0x55c3475e3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:04:44.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.906+0000 7fbdaa800640 1 --2- 172.21.15.12:0/3900426890 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475dfbb0 0x55c3475d9310 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009b80 tx=0x7fbda4009a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=680479ef5bb23f06 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:44.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.906+0000 7fbd9aa00640 1 -- 172.21.15.12:0/3900426890 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda4028040 con 0x55c3475dfbb0 2024-03-23T23:04:44.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.906+0000 7fbd9aa00640 1 -- 172.21.15.12:0/3900426890 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbda401f040 con 0x55c3475dfbb0 2024-03-23T23:04:44.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.906+0000 7fbd9aa00640 1 -- 172.21.15.12:0/3900426890 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda4003ad0 con 0x55c3475dfbb0 2024-03-23T23:04:44.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.907+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/3900426890 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475dfbb0 msgr2=0x55c3475d9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:44.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.907+0000 7fbdb3a6c740 1 --2- 172.21.15.12:0/3900426890 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475dfbb0 0x55c3475d9310 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009b80 tx=0x7fbda4009a50 comp rx=0 tx=0).stop 2024-03-23T23:04:44.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.907+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/3900426890 shutdown_connections 2024-03-23T23:04:44.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.907+0000 7fbdb3a6c740 1 --2- 172.21.15.12:0/3900426890 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475dfbb0 0x55c3475d9310 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.907+0000 7fbdb3a6c740 1 --2- 172.21.15.12:0/3900426890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475daa40 0x55c3475eb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.907+0000 7fbdb3a6c740 1 --2- 172.21.15.12:0/3900426890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3475da050 0x55c3475e3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.907+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/3900426890 >> 172.21.15.12:0/3900426890 conn(0x55c3475a2200 msgr2=0x55c3475d8c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.907+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/3900426890 shutdown_connections 2024-03-23T23:04:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.907+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/3900426890 wait complete. 2024-03-23T23:04:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.907+0000 7fbdb3a6c740 1 -- start start 2024-03-23T23:04:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.907+0000 7fbdb3a6c740 1 Processor -- start 2024-03-23T23:04:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.908+0000 7fbdb3a6c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3475da050 0x55c3475e8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.908+0000 7fbdb3a6c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475daa40 0x55c3475e8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.908+0000 7fbdb3a6c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475dfbb0 0x55c3475e9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.908+0000 7fbdb3a6c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdac1ad610 con 0x55c3475da050 2024-03-23T23:04:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.908+0000 7fbdb3a6c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdac1ae910 con 0x55c3475daa40 2024-03-23T23:04:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.908+0000 7fbdb3a6c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdac1adc50 con 0x55c3475dfbb0 2024-03-23T23:04:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.908+0000 7fbdb3a6c740 10 client.0 _ll_register_callbacks cb 0x7fbd9c0d29b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.908+0000 7fbda9e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3475da050 0x55c3475e8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.908+0000 7fbda9400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475daa40 0x55c3475e8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.908+0000 7fbda9400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475daa40 0x55c3475e8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:56272/0 (socket says 172.21.15.12:56272) 2024-03-23T23:04:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.908+0000 7fbda9400640 1 -- 172.21.15.12:0/1299147898 learned_addr learned my addr 172.21.15.12:0/1299147898 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.908+0000 7fbda9e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3475da050 0x55c3475e8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48628/0 (socket says 172.21.15.12:48628) 2024-03-23T23:04:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.908+0000 7fbdaa800640 1 --2- 172.21.15.12:0/1299147898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475dfbb0 0x55c3475e9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.908+0000 7fbda9400640 1 -- 172.21.15.12:0/1299147898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475dfbb0 msgr2=0x55c3475e9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.908+0000 7fbda9400640 1 --2- 172.21.15.12:0/1299147898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475dfbb0 0x55c3475e9f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.908+0000 7fbda9400640 1 -- 172.21.15.12:0/1299147898 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3475da050 msgr2=0x55c3475e8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.908+0000 7fbda9400640 1 --2- 172.21.15.12:0/1299147898 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3475da050 0x55c3475e8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.909+0000 7fbda9400640 1 -- 172.21.15.12:0/1299147898 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c3475ea500 con 0x55c3475daa40 2024-03-23T23:04:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.909+0000 7fbda9e00640 1 --2- 172.21.15.12:0/1299147898 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3475da050 0x55c3475e8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.909+0000 7fbda9400640 1 --2- 172.21.15.12:0/1299147898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475daa40 0x55c3475e8fa0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c097e50 tx=0x7fbd9c009480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49a9c48b2aa74356 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.909+0000 7fbd8fe00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:04:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.909+0000 7fbd8fe00640 1 -- 172.21.15.12:0/1299147898 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c009820 con 0x55c3475daa40 2024-03-23T23:04:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.909+0000 7fbd8fe00640 1 -- 172.21.15.12:0/1299147898 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fbd9c0d2c50 con 0x55c3475daa40 2024-03-23T23:04:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.909+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/1299147898 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbda401f040 con 0x55c3475daa40 2024-03-23T23:04:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.909+0000 7fbd8fe00640 1 -- 172.21.15.12:0/1299147898 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd9c0d2e50 con 0x55c3475daa40 2024-03-23T23:04:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.909+0000 7fbdb3a6c740 10 client.26105 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.909+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/1299147898 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c3475eb880 con 0x55c3475daa40 2024-03-23T23:04:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.909+0000 7fbd8fe00640 1 -- 172.21.15.12:0/1299147898 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbda401f040 con 0x55c3475daa40 2024-03-23T23:04:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.910+0000 7fbd8fe00640 1 -- 172.21.15.12:0/1299147898 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fbd9c0d1180 con 0x55c3475daa40 2024-03-23T23:04:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.910+0000 7fbdb3a6c740 10 client.26105 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.910+0000 7fbdb3a6c740 10 client.26105 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:04:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.910+0000 7fbdb3a6c740 20 client.26105 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.910+0000 7fbdb3a6c740 1 --2- 172.21.15.12:0/1299147898 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c34764bb40 0x55c34764df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.910+0000 7fbdb3a6c740 4 client.26105 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:04:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.910+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/1299147898 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c347546ae0 con 0x55c34764bb40 2024-03-23T23:04:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.910+0000 7fbda9e00640 1 --2- 172.21.15.12:0/1299147898 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c34764bb40 0x55c34764df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.910+0000 7fbda9e00640 1 --2- 172.21.15.12:0/1299147898 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c34764bb40 0x55c34764df60 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49ebf8b51e7bd5be server_cookie=67449a4474c75779 in_seq=0 out_seq=0 2024-03-23T23:04:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.910+0000 7fbd8fe00640 10 client.26105 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.911+0000 7fbd8fe00640 1 -- 172.21.15.12:0/1299147898 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55c347546ae0 con 0x55c34764bb40 2024-03-23T23:04:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.911+0000 7fbd8fe00640 10 client.26105 handle_command_reply: tid=0 2024-03-23T23:04:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.911+0000 7fbdb3a6c740 1 client.26105 shutdown 2024-03-23T23:04:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.911+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/1299147898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475daa40 msgr2=0x55c3475e8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.911+0000 7fbdb3a6c740 1 --2- 172.21.15.12:0/1299147898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475daa40 0x55c3475e8fa0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c097e50 tx=0x7fbd9c009480 comp rx=0 tx=0).stop 2024-03-23T23:04:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.911+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/1299147898 shutdown_connections 2024-03-23T23:04:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.911+0000 7fbdb3a6c740 1 --2- 172.21.15.12:0/1299147898 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c34764bb40 0x55c34764df60 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.911+0000 7fbdb3a6c740 1 --2- 172.21.15.12:0/1299147898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3475dfbb0 0x55c3475e9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.911+0000 7fbdb3a6c740 1 --2- 172.21.15.12:0/1299147898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3475daa40 0x55c3475e8fa0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.911+0000 7fbdb3a6c740 1 --2- 172.21.15.12:0/1299147898 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3475da050 0x55c3475e8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.911+0000 7fbd8fe00640 0 client.26105 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.911+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/1299147898 >> 172.21.15.12:0/1299147898 conn(0x55c3475a2200 msgr2=0x55c3475d9490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.911+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/1299147898 shutdown_connections 2024-03-23T23:04:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.911+0000 7fbdb3a6c740 1 -- 172.21.15.12:0/1299147898 wait complete. 2024-03-23T23:04:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.911+0000 7fbdb3a6c740 20 client.26105 trim_cache size 0 max 16384 2024-03-23T23:04:44.916 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:04:44.916 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T23:04:44.916 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-03-23T23:04:44.916 INFO:teuthology.orchestra.run.smithi012.stdout: "21daee86-0251-46c9-b8ec-8d59e79642a3": { 2024-03-23T23:04:44.916 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-03-23T23:04:44.916 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "21daee86-0251-46c9-b8ec-8d59e79642a3", 2024-03-23T23:04:44.916 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-03-23T23:04:44.916 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:04:44.916 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:04:44.916 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:04:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.912+0000 7fbdab200640 1 -- 172.21.15.12:0/3993993637 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbd900a2880 msgr2=0x7fbd900a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.912+0000 7fbdab200640 1 --2- 172.21.15.12:0/3993993637 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbd900a2880 0x7fbd900a4d40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x55c3475db4b0 tx=0x7fbd9402df90 comp rx=0 tx=0).stop 2024-03-23T23:04:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.912+0000 7fbdab200640 1 -- 172.21.15.12:0/3993993637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbdac108db0 msgr2=0x7fbdac19e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.912+0000 7fbdab200640 1 --2- 172.21.15.12:0/3993993637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbdac108db0 0x7fbdac19e440 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c009d50 tx=0x7fbd9c00c6a0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.912+0000 7fbdab200640 1 -- 172.21.15.12:0/3993993637 shutdown_connections 2024-03-23T23:04:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.912+0000 7fbdab200640 1 --2- 172.21.15.12:0/3993993637 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbd900a2880 0x7fbd900a4d40 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.912+0000 7fbdab200640 1 --2- 172.21.15.12:0/3993993637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdac109770 0x7fbdac1a54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.912+0000 7fbdab200640 1 --2- 172.21.15.12:0/3993993637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbdac108db0 0x7fbdac19e440 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.912+0000 7fbdab200640 1 --2- 172.21.15.12:0/3993993637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdac107bc0 0x7fbdac19df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:44.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.912+0000 7fbdab200640 1 -- 172.21.15.12:0/3993993637 >> 172.21.15.12:0/3993993637 conn(0x7fbdac076b70 msgr2=0x7fbdac10c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:44.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.912+0000 7fbdab200640 1 -- 172.21.15.12:0/3993993637 shutdown_connections 2024-03-23T23:04:44.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:04:44.912+0000 7fbdab200640 1 -- 172.21.15.12:0/3993993637 wait complete. 2024-03-23T23:04:44.929 INFO:tasks.cephfs.filesystem:scrub status for tag:21daee86-0251-46c9-b8ec-8d59e79642a3 - {'path': '/', 'tag': '21daee86-0251-46c9-b8ec-8d59e79642a3', 'options': 'recursive,force'} 2024-03-23T23:04:45.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:44 smithi012 ceph-mon[31695]: pgmap v478: 97 pgs: 97 active+clean; 1.5 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:04:45.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:44 smithi173 ceph-mon[36116]: pgmap v478: 97 pgs: 97 active+clean; 1.5 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:04:45.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:44 smithi203 ceph-mon[36693]: pgmap v478: 97 pgs: 97 active+clean; 1.5 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:04:46.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:45 smithi012 ceph-mon[31695]: scrub queued for path: / 2024-03-23T23:04:46.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:45 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:04:46.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:45 smithi012 ceph-mon[31695]: scrub summary: active paths [/] 2024-03-23T23:04:46.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:45 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:04:46.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:45 smithi173 ceph-mon[36116]: scrub queued for path: / 2024-03-23T23:04:46.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:45 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:04:46.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:45 smithi173 ceph-mon[36116]: scrub summary: active paths [/] 2024-03-23T23:04:46.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:45 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:04:46.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:45 smithi203 ceph-mon[36693]: scrub queued for path: / 2024-03-23T23:04:46.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:45 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:04:46.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:45 smithi203 ceph-mon[36693]: scrub summary: active paths [/] 2024-03-23T23:04:46.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:45 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:04:47.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:46 smithi012 ceph-mon[31695]: pgmap v479: 97 pgs: 97 active+clean; 1.5 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:04:47.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:46 smithi012 ceph-mon[31695]: scrub summary: idle 2024-03-23T23:04:47.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:46 smithi173 ceph-mon[36116]: pgmap v479: 97 pgs: 97 active+clean; 1.5 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:04:47.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:46 smithi173 ceph-mon[36116]: scrub summary: idle 2024-03-23T23:04:47.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:46 smithi203 ceph-mon[36693]: pgmap v479: 97 pgs: 97 active+clean; 1.5 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:04:47.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:46 smithi203 ceph-mon[36693]: scrub summary: idle 2024-03-23T23:04:49.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:48 smithi012 ceph-mon[31695]: pgmap v480: 97 pgs: 97 active+clean; 1.5 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:04:49.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:48 smithi173 ceph-mon[36116]: pgmap v480: 97 pgs: 97 active+clean; 1.5 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:04:49.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:48 smithi203 ceph-mon[36693]: pgmap v480: 97 pgs: 97 active+clean; 1.5 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:04:49.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.978+0000 7ff21f600640 1 -- 172.21.15.12:0/1448852618 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff218104af0 msgr2=0x7ff218104f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:49.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.978+0000 7ff21f600640 1 --2- 172.21.15.12:0/1448852618 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff218104af0 0x7ff218104f70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff208009900 tx=0x7ff20802bc30 comp rx=0 tx=0).stop 2024-03-23T23:04:49.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.978+0000 7ff21f600640 1 -- 172.21.15.12:0/1448852618 shutdown_connections 2024-03-23T23:04:49.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.978+0000 7ff21f600640 1 --2- 172.21.15.12:0/1448852618 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2181054b0 0x7ff21810bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:49.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.978+0000 7ff21f600640 1 --2- 172.21.15.12:0/1448852618 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff218104af0 0x7ff218104f70 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:49.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.978+0000 7ff21f600640 1 --2- 172.21.15.12:0/1448852618 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff218103900 0x7ff218103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:49.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.978+0000 7ff21f600640 1 -- 172.21.15.12:0/1448852618 >> 172.21.15.12:0/1448852618 conn(0x7ff2180fee90 msgr2=0x7ff2181012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:49.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.979+0000 7ff21f600640 1 -- 172.21.15.12:0/1448852618 shutdown_connections 2024-03-23T23:04:49.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.979+0000 7ff21f600640 1 -- 172.21.15.12:0/1448852618 wait complete. 2024-03-23T23:04:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.980+0000 7ff21f600640 1 Processor -- start 2024-03-23T23:04:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.980+0000 7ff21f600640 1 -- start start 2024-03-23T23:04:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.980+0000 7ff21f600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff218103900 0x7ff218079450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.980+0000 7ff21f600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff218104af0 0x7ff218079990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.980+0000 7ff21f600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2181054b0 0x7ff218075fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.980+0000 7ff21f600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21810f5d0 con 0x7ff2181054b0 2024-03-23T23:04:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.980+0000 7ff21f600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21810f450 con 0x7ff218103900 2024-03-23T23:04:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.980+0000 7ff21f600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21810f750 con 0x7ff218104af0 2024-03-23T23:04:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.981+0000 7ff21cc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff218103900 0x7ff218079450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.981+0000 7ff21cc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff218103900 0x7ff218079450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:56294/0 (socket says 172.21.15.12:56294) 2024-03-23T23:04:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.981+0000 7ff21cc00640 1 -- 172.21.15.12:0/1939022183 learned_addr learned my addr 172.21.15.12:0/1939022183 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.981+0000 7ff21d600640 1 --2- 172.21.15.12:0/1939022183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2181054b0 0x7ff218075fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.981+0000 7ff217e00640 1 --2- 172.21.15.12:0/1939022183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff218104af0 0x7ff218079990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.981+0000 7ff21cc00640 1 -- 172.21.15.12:0/1939022183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff218104af0 msgr2=0x7ff218079990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.981+0000 7ff21cc00640 1 --2- 172.21.15.12:0/1939022183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff218104af0 0x7ff218079990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.981+0000 7ff21cc00640 1 -- 172.21.15.12:0/1939022183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2181054b0 msgr2=0x7ff218075fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.981+0000 7ff21cc00640 1 --2- 172.21.15.12:0/1939022183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2181054b0 0x7ff218075fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.981+0000 7ff21cc00640 1 -- 172.21.15.12:0/1939022183 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff218076880 con 0x7ff218103900 2024-03-23T23:04:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.981+0000 7ff21cc00640 1 --2- 172.21.15.12:0/1939022183 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff218103900 0x7ff218079450 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00e9c0 tx=0x7ff20c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.982+0000 7ff215600640 1 -- 172.21.15.12:0/1939022183 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c00cd80 con 0x7ff218103900 2024-03-23T23:04:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.982+0000 7ff215600640 1 -- 172.21.15.12:0/1939022183 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff20c0035f0 con 0x7ff218103900 2024-03-23T23:04:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.982+0000 7ff21f600640 1 -- 172.21.15.12:0/1939022183 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2181a5fd0 con 0x7ff218103900 2024-03-23T23:04:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.982+0000 7ff215600640 1 -- 172.21.15.12:0/1939022183 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c010690 con 0x7ff218103900 2024-03-23T23:04:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.982+0000 7ff21f600640 1 -- 172.21.15.12:0/1939022183 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2181a6380 con 0x7ff218103900 2024-03-23T23:04:49.984 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.985+0000 7ff215600640 1 -- 172.21.15.12:0/1939022183 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff20c0037a0 con 0x7ff218103900 2024-03-23T23:04:49.985 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.985+0000 7ff2223e1740 1 Processor -- start 2024-03-23T23:04:49.985 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.985+0000 7ff2223e1740 1 -- start start 2024-03-23T23:04:49.986 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.986+0000 7ff2223e1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91ca1e0 0x55d5e91ca600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:49.986 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.986+0000 7ff2223e1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5e91cabd0 0x55d5e91cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:49.986 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.986+0000 7ff2223e1740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5e91cfd40 0x55d5e91d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:49.986 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.986+0000 7ff2223e1740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5e918d3b0 con 0x55d5e91cfd40 2024-03-23T23:04:49.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.986+0000 7ff2223e1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5e912f830 con 0x55d5e91cabd0 2024-03-23T23:04:49.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.986+0000 7ff2223e1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5e90d85d0 con 0x55d5e91ca1e0 2024-03-23T23:04:49.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.986+0000 7ff21cc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5e91cabd0 0x55d5e91cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:49.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.986+0000 7ff21d600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91ca1e0 0x55d5e91ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:49.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.986+0000 7ff21cc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5e91cabd0 0x55d5e91cf6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:56308/0 (socket says 172.21.15.12:56308) 2024-03-23T23:04:49.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.986+0000 7ff21cc00640 1 -- 172.21.15.12:0/3636606873 learned_addr learned my addr 172.21.15.12:0/3636606873 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:49.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.986+0000 7ff217e00640 1 --2- 172.21.15.12:0/3636606873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5e91cfd40 0x55d5e91d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:49.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.986+0000 7ff21cc00640 1 -- 172.21.15.12:0/3636606873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91ca1e0 msgr2=0x55d5e91ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:49.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.986+0000 7ff21cc00640 1 --2- 172.21.15.12:0/3636606873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91ca1e0 0x55d5e91ca600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:49.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.986+0000 7ff21cc00640 1 -- 172.21.15.12:0/3636606873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5e91cfd40 msgr2=0x55d5e91d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:49.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.986+0000 7ff21cc00640 1 --2- 172.21.15.12:0/3636606873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5e91cfd40 0x55d5e91d21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:49.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.987+0000 7ff21cc00640 1 -- 172.21.15.12:0/3636606873 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d5e90f02c0 con 0x55d5e91cabd0 2024-03-23T23:04:49.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.987+0000 7ff217e00640 1 --2- 172.21.15.12:0/3636606873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5e91cfd40 0x55d5e91d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:04:49.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.987+0000 7ff21cc00640 1 --2- 172.21.15.12:0/3636606873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5e91cabd0 0x55d5e91cf6d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0d2d20 tx=0x7ff20c0d2e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51cadc94f3855c91 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:49.991 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.987+0000 7ff205c00640 1 -- 172.21.15.12:0/3636606873 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c0d0030 con 0x55d5e91cabd0 2024-03-23T23:04:49.991 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.987+0000 7ff205c00640 1 -- 172.21.15.12:0/3636606873 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff20c0d5960 con 0x55d5e91cabd0 2024-03-23T23:04:49.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.987+0000 7ff205c00640 1 -- 172.21.15.12:0/3636606873 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c0df920 con 0x55d5e91cabd0 2024-03-23T23:04:49.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.987+0000 7ff215600640 1 --2- 172.21.15.12:0/1939022183 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff2000a2920 0x7ff2000a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:49.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.987+0000 7ff215600640 1 -- 172.21.15.12:0/1939022183 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff20c0ced80 con 0x7ff218103900 2024-03-23T23:04:49.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.988+0000 7ff21d600640 1 --2- 172.21.15.12:0/1939022183 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff2000a2920 0x7ff2000a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:49.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.988+0000 7ff2223e1740 1 -- 172.21.15.12:0/3636606873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5e91cabd0 msgr2=0x55d5e91cf6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:49.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.988+0000 7ff2223e1740 1 --2- 172.21.15.12:0/3636606873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5e91cabd0 0x55d5e91cf6d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0d2d20 tx=0x7ff20c0d2e10 comp rx=0 tx=0).stop 2024-03-23T23:04:49.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.988+0000 7ff2223e1740 1 -- 172.21.15.12:0/3636606873 shutdown_connections 2024-03-23T23:04:49.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.988+0000 7ff2223e1740 1 --2- 172.21.15.12:0/3636606873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5e91cfd40 0x55d5e91d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:49.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.988+0000 7ff2223e1740 1 --2- 172.21.15.12:0/3636606873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5e91cabd0 0x55d5e91cf6d0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:49.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.988+0000 7ff2223e1740 1 --2- 172.21.15.12:0/3636606873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91ca1e0 0x55d5e91ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:49.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.988+0000 7ff2223e1740 1 -- 172.21.15.12:0/3636606873 >> 172.21.15.12:0/3636606873 conn(0x55d5e91c5750 msgr2=0x55d5e91c5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:49.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.988+0000 7ff2223e1740 1 -- 172.21.15.12:0/3636606873 shutdown_connections 2024-03-23T23:04:49.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.988+0000 7ff2223e1740 1 -- 172.21.15.12:0/3636606873 wait complete. 2024-03-23T23:04:49.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.988+0000 7ff21d600640 1 --2- 172.21.15.12:0/1939022183 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff2000a2920 0x7ff2000a4de0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x55d5e91cb430 tx=0x7ff2100096a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:49.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.989+0000 7ff2223e1740 1 -- start start 2024-03-23T23:04:49.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.989+0000 7ff2223e1740 1 Processor -- start 2024-03-23T23:04:49.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.989+0000 7ff2223e1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91ca1e0 0x55d5e91d4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:49.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.990+0000 7ff2223e1740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5e91cabd0 0x55d5e91d4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:49.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.990+0000 7ff2223e1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5e91cfd40 0x55d5e91db960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:49.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.990+0000 7ff2223e1740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5e90eba20 con 0x55d5e91cabd0 2024-03-23T23:04:49.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.990+0000 7ff2223e1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5e912b8d0 con 0x55d5e91cfd40 2024-03-23T23:04:49.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.990+0000 7ff2223e1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5e91d2cf0 con 0x55d5e91ca1e0 2024-03-23T23:04:49.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.990+0000 7ff2223e1740 10 client.0 _ll_register_callbacks cb 0x55d5e91281c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:49.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.990+0000 7ff21cc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91ca1e0 0x55d5e91d4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:49.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.990+0000 7ff217e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5e91cabd0 0x55d5e91d4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:49.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.990+0000 7ff21d600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5e91cfd40 0x55d5e91db960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:49.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.990+0000 7ff21cc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91ca1e0 0x55d5e91d4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58346/0 (socket says 172.21.15.12:58346) 2024-03-23T23:04:49.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.990+0000 7ff21cc00640 1 -- 172.21.15.12:0/1938261649 learned_addr learned my addr 172.21.15.12:0/1938261649 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:49.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.990+0000 7ff21cc00640 1 -- 172.21.15.12:0/1938261649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5e91cfd40 msgr2=0x55d5e91db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:49.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.990+0000 7ff21cc00640 1 --2- 172.21.15.12:0/1938261649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5e91cfd40 0x55d5e91db960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:49.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.990+0000 7ff21cc00640 1 -- 172.21.15.12:0/1938261649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5e91cabd0 msgr2=0x55d5e91d4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:49.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.990+0000 7ff21cc00640 1 --2- 172.21.15.12:0/1938261649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5e91cabd0 0x55d5e91d4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:49.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.990+0000 7ff21cc00640 1 -- 172.21.15.12:0/1938261649 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d5e8e44210 con 0x55d5e91ca1e0 2024-03-23T23:04:49.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.991+0000 7ff21cc00640 1 --2- 172.21.15.12:0/1938261649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91ca1e0 0x55d5e91d4450 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0a3940 tx=0x7ff20c0045f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67c7ad85ad852bc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:49.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.991+0000 7ff1fa000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:04:49.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.991+0000 7ff1fa000640 1 -- 172.21.15.12:0/1938261649 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c0d0060 con 0x55d5e91ca1e0 2024-03-23T23:04:49.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.991+0000 7ff1fa000640 1 -- 172.21.15.12:0/1938261649 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff20c0cc940 con 0x55d5e91ca1e0 2024-03-23T23:04:49.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.991+0000 7ff2223e1740 1 -- 172.21.15.12:0/1938261649 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff20c0d5960 con 0x55d5e91ca1e0 2024-03-23T23:04:49.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.991+0000 7ff1fa000640 1 -- 172.21.15.12:0/1938261649 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff20c097970 con 0x55d5e91ca1e0 2024-03-23T23:04:49.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.991+0000 7ff2223e1740 10 client.26086 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:49.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.991+0000 7ff2223e1740 1 -- 172.21.15.12:0/1938261649 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55d5e8f10620 con 0x55d5e91ca1e0 2024-03-23T23:04:49.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.991+0000 7ff1fa000640 1 -- 172.21.15.12:0/1938261649 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff20c0d5960 con 0x55d5e91ca1e0 2024-03-23T23:04:49.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.991+0000 7ff1fa000640 1 -- 172.21.15.12:0/1938261649 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ff20c0de070 con 0x55d5e91ca1e0 2024-03-23T23:04:49.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.991+0000 7ff2223e1740 10 client.26086 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:49.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.991+0000 7ff2223e1740 10 client.26086 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:04:49.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.991+0000 7ff2223e1740 20 client.26086 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:49.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.992+0000 7ff2223e1740 1 --2- 172.21.15.12:0/1938261649 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d5e91dfe50 0x55d5e91e2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:49.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.992+0000 7ff2223e1740 4 client.26086 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:04:49.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.992+0000 7ff2223e1740 1 -- 172.21.15.12:0/1938261649 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d5e913cd70 con 0x55d5e91dfe50 2024-03-23T23:04:49.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.992+0000 7ff217e00640 1 --2- 172.21.15.12:0/1938261649 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d5e91dfe50 0x55d5e91e2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:49.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.992+0000 7ff217e00640 1 --2- 172.21.15.12:0/1938261649 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d5e91dfe50 0x55d5e91e2240 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3c99f18e4b531f5 server_cookie=15baa7534b947094 in_seq=0 out_seq=0 2024-03-23T23:04:49.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.992+0000 7ff1fa000640 10 client.26086 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:49.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.993+0000 7ff1fa000640 1 -- 172.21.15.12:0/1938261649 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d5e913cd70 con 0x55d5e91dfe50 2024-03-23T23:04:49.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.993+0000 7ff1fa000640 10 client.26086 handle_command_reply: tid=0 2024-03-23T23:04:49.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.993+0000 7ff2223e1740 1 client.26086 shutdown 2024-03-23T23:04:49.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.994+0000 7ff2223e1740 1 -- 172.21.15.12:0/1938261649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91ca1e0 msgr2=0x55d5e91d4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:49.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.994+0000 7ff2223e1740 1 --2- 172.21.15.12:0/1938261649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91ca1e0 0x55d5e91d4450 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0a3940 tx=0x7ff20c0045f0 comp rx=0 tx=0).stop 2024-03-23T23:04:49.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.994+0000 7ff2223e1740 1 -- 172.21.15.12:0/1938261649 shutdown_connections 2024-03-23T23:04:49.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.994+0000 7ff2223e1740 1 --2- 172.21.15.12:0/1938261649 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d5e91dfe50 0x55d5e91e2240 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:49.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.994+0000 7ff2223e1740 1 --2- 172.21.15.12:0/1938261649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5e91cfd40 0x55d5e91db960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:49.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.994+0000 7ff2223e1740 1 --2- 172.21.15.12:0/1938261649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5e91cabd0 0x55d5e91d4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:49.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.994+0000 7ff2223e1740 1 --2- 172.21.15.12:0/1938261649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91ca1e0 0x55d5e91d4450 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:49.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.994+0000 7ff1fa000640 0 client.26086 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:49.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.994+0000 7ff2223e1740 1 -- 172.21.15.12:0/1938261649 >> 172.21.15.12:0/1938261649 conn(0x55d5e91c5750 msgr2=0x55d5e91c4950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:49.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.994+0000 7ff2223e1740 1 -- 172.21.15.12:0/1938261649 shutdown_connections 2024-03-23T23:04:49.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.994+0000 7ff2223e1740 1 -- 172.21.15.12:0/1938261649 wait complete. 2024-03-23T23:04:49.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:49.994+0000 7ff2223e1740 20 client.26086 trim_cache size 0 max 16384 2024-03-23T23:04:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.009+0000 7ff2223e1740 1 Processor -- start 2024-03-23T23:04:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.009+0000 7ff2223e1740 1 -- start start 2024-03-23T23:04:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.009+0000 7ff2223e1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91ca1e0 0x55d5e91d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.009+0000 7ff2223e1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5e91cabd0 0x55d5e91db920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.009+0000 7ff2223e1740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5e91cfd40 0x55d5e91d4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:50.010 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.010+0000 7ff2223e1740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2181b1530 con 0x55d5e91cfd40 2024-03-23T23:04:50.010 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.010+0000 7ff2223e1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2181af030 con 0x55d5e91cabd0 2024-03-23T23:04:50.010 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.010+0000 7ff2223e1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2181ad0d0 con 0x55d5e91ca1e0 2024-03-23T23:04:50.011 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.010+0000 7ff21cc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91ca1e0 0x55d5e91d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:50.011 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.010+0000 7ff21d600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5e91cfd40 0x55d5e91d4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.010+0000 7ff217e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5e91cabd0 0x55d5e91db920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.010+0000 7ff21cc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91ca1e0 0x55d5e91d2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58362/0 (socket says 172.21.15.12:58362) 2024-03-23T23:04:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.010+0000 7ff21cc00640 1 -- 172.21.15.12:0/2748157933 learned_addr learned my addr 172.21.15.12:0/2748157933 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.010+0000 7ff21cc00640 1 -- 172.21.15.12:0/2748157933 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5e91cabd0 msgr2=0x55d5e91db920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.010+0000 7ff21cc00640 1 --2- 172.21.15.12:0/2748157933 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5e91cabd0 0x55d5e91db920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.010+0000 7ff21cc00640 1 -- 172.21.15.12:0/2748157933 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5e91cfd40 msgr2=0x55d5e91d4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.010+0000 7ff21cc00640 1 --2- 172.21.15.12:0/2748157933 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5e91cfd40 0x55d5e91d4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.010+0000 7ff21cc00640 1 -- 172.21.15.12:0/2748157933 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d5e91c3d40 con 0x55d5e91ca1e0 2024-03-23T23:04:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.010+0000 7ff217e00640 1 --2- 172.21.15.12:0/2748157933 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5e91cabd0 0x55d5e91db920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.010+0000 7ff21cc00640 1 --2- 172.21.15.12:0/2748157933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91ca1e0 0x55d5e91d2130 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0a3fe0 tx=0x7ff20c099be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee53d0d6c81c29c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.011+0000 7ff205200640 1 -- 172.21.15.12:0/2748157933 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c09b030 con 0x55d5e91ca1e0 2024-03-23T23:04:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.011+0000 7ff205200640 1 -- 172.21.15.12:0/2748157933 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff20c09bdd0 con 0x55d5e91ca1e0 2024-03-23T23:04:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.011+0000 7ff205200640 1 -- 172.21.15.12:0/2748157933 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c09b210 con 0x55d5e91ca1e0 2024-03-23T23:04:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.011+0000 7ff2223e1740 1 -- 172.21.15.12:0/2748157933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91ca1e0 msgr2=0x55d5e91d2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.011+0000 7ff2223e1740 1 --2- 172.21.15.12:0/2748157933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91ca1e0 0x55d5e91d2130 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0a3fe0 tx=0x7ff20c099be0 comp rx=0 tx=0).stop 2024-03-23T23:04:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.011+0000 7ff2223e1740 1 -- 172.21.15.12:0/2748157933 shutdown_connections 2024-03-23T23:04:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.011+0000 7ff2223e1740 1 --2- 172.21.15.12:0/2748157933 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5e91cfd40 0x55d5e91d4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.011+0000 7ff2223e1740 1 --2- 172.21.15.12:0/2748157933 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5e91cabd0 0x55d5e91db920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.011+0000 7ff2223e1740 1 --2- 172.21.15.12:0/2748157933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91ca1e0 0x55d5e91d2130 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.011+0000 7ff2223e1740 1 -- 172.21.15.12:0/2748157933 >> 172.21.15.12:0/2748157933 conn(0x55d5e91c5750 msgr2=0x55d5e91d4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.011+0000 7ff2223e1740 1 -- 172.21.15.12:0/2748157933 shutdown_connections 2024-03-23T23:04:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.011+0000 7ff2223e1740 1 -- 172.21.15.12:0/2748157933 wait complete. 2024-03-23T23:04:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.012+0000 7ff2223e1740 1 -- start start 2024-03-23T23:04:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.012+0000 7ff2223e1740 1 Processor -- start 2024-03-23T23:04:50.016 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.012+0000 7ff2223e1740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5e91ca1e0 0x55d5e91d9520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:50.016 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.012+0000 7ff2223e1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91cabd0 0x55d5e91d9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:50.016 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.012+0000 7ff2223e1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5e91cfd40 0x55d5e91daa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:50.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.012+0000 7ff2223e1740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2181ad6d0 con 0x55d5e91ca1e0 2024-03-23T23:04:50.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.012+0000 7ff2223e1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2181ae9d0 con 0x55d5e91cfd40 2024-03-23T23:04:50.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.012+0000 7ff2223e1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2181add10 con 0x55d5e91cabd0 2024-03-23T23:04:50.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.012+0000 7ff2223e1740 10 client.0 _ll_register_callbacks cb 0x7ff210009ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:04:50.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.013+0000 7ff217e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91cabd0 0x55d5e91d9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:50.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.013+0000 7ff21d600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5e91cfd40 0x55d5e91daa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:50.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.013+0000 7ff21cc00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5e91ca1e0 0x55d5e91d9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:50.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.013+0000 7ff217e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91cabd0 0x55d5e91d9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58372/0 (socket says 172.21.15.12:58372) 2024-03-23T23:04:50.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.013+0000 7ff217e00640 1 -- 172.21.15.12:0/253541127 learned_addr learned my addr 172.21.15.12:0/253541127 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:04:50.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.013+0000 7ff21d600640 1 -- 172.21.15.12:0/253541127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91cabd0 msgr2=0x55d5e91d9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:50.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.013+0000 7ff21d600640 1 --2- 172.21.15.12:0/253541127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91cabd0 0x55d5e91d9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:50.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.013+0000 7ff21d600640 1 -- 172.21.15.12:0/253541127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5e91ca1e0 msgr2=0x55d5e91d9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:04:50.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.013+0000 7ff21d600640 1 --2- 172.21.15.12:0/253541127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5e91ca1e0 0x55d5e91d9520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:50.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.013+0000 7ff21d600640 1 -- 172.21.15.12:0/253541127 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d5e91db060 con 0x55d5e91cfd40 2024-03-23T23:04:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.013+0000 7ff217e00640 1 --2- 172.21.15.12:0/253541127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91cabd0 0x55d5e91d9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.013+0000 7ff21cc00640 1 --2- 172.21.15.12:0/253541127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5e91ca1e0 0x55d5e91d9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:04:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.013+0000 7ff21d600640 1 --2- 172.21.15.12:0/253541127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5e91cfd40 0x55d5e91daa70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff210004010 tx=0x7ff210004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4243cb3058939671 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:04:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.013+0000 7ff1fa000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:04:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.014+0000 7ff1fa000640 1 -- 172.21.15.12:0/253541127 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21002e060 con 0x55d5e91cfd40 2024-03-23T23:04:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.014+0000 7ff2223e1740 1 -- 172.21.15.12:0/253541127 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff20c09bdd0 con 0x55d5e91cfd40 2024-03-23T23:04:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.014+0000 7ff1fa000640 1 -- 172.21.15.12:0/253541127 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff21001d410 con 0x55d5e91cfd40 2024-03-23T23:04:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.014+0000 7ff1fa000640 1 -- 172.21.15.12:0/253541127 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff21001d610 con 0x55d5e91cfd40 2024-03-23T23:04:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.014+0000 7ff2223e1740 10 client.26123 fetch_fsmap learned FSMap version 23 2024-03-23T23:04:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.014+0000 7ff2223e1740 1 -- 172.21.15.12:0/253541127 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55d5e923ba10 con 0x55d5e91cfd40 2024-03-23T23:04:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.014+0000 7ff1fa000640 1 -- 172.21.15.12:0/253541127 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff20c09bdd0 con 0x55d5e91cfd40 2024-03-23T23:04:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.014+0000 7ff1fa000640 1 -- 172.21.15.12:0/253541127 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ff2100299a0 con 0x55d5e91cfd40 2024-03-23T23:04:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.014+0000 7ff2223e1740 10 client.26123 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:04:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.014+0000 7ff2223e1740 10 client.26123 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:04:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.014+0000 7ff2223e1740 20 client.26123 populate_metadata read hostname 'smithi012' 2024-03-23T23:04:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.014+0000 7ff2223e1740 1 --2- 172.21.15.12:0/253541127 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d5e923c350 0x55d5e923e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:04:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.014+0000 7ff2223e1740 4 client.26123 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:04:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.014+0000 7ff2223e1740 1 -- 172.21.15.12:0/253541127 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d5e91c0a80 con 0x55d5e923c350 2024-03-23T23:04:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.015+0000 7ff21cc00640 1 --2- 172.21.15.12:0/253541127 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d5e923c350 0x55d5e923e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:04:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.015+0000 7ff21cc00640 1 --2- 172.21.15.12:0/253541127 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d5e923c350 0x55d5e923e770 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b38a4025a4cc80b1 server_cookie=811b71e1dd379fa8 in_seq=0 out_seq=0 2024-03-23T23:04:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.015+0000 7ff1fa000640 10 client.26123 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.027+0000 7ff1fa000640 1 -- 172.21.15.12:0/253541127 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d5e91c0a80 con 0x55d5e923c350 2024-03-23T23:04:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.027+0000 7ff1fa000640 10 client.26123 handle_command_reply: tid=0 2024-03-23T23:04:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.027+0000 7ff2223e1740 1 client.26123 shutdown 2024-03-23T23:04:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.027+0000 7ff2223e1740 1 -- 172.21.15.12:0/253541127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5e91cfd40 msgr2=0x55d5e91daa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.027+0000 7ff2223e1740 1 --2- 172.21.15.12:0/253541127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5e91cfd40 0x55d5e91daa70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff210004010 tx=0x7ff210004a40 comp rx=0 tx=0).stop 2024-03-23T23:04:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.027+0000 7ff2223e1740 1 -- 172.21.15.12:0/253541127 shutdown_connections 2024-03-23T23:04:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.027+0000 7ff2223e1740 1 --2- 172.21.15.12:0/253541127 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d5e923c350 0x55d5e923e770 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.027+0000 7ff2223e1740 1 --2- 172.21.15.12:0/253541127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5e91cfd40 0x55d5e91daa70 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.027+0000 7ff2223e1740 1 --2- 172.21.15.12:0/253541127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5e91cabd0 0x55d5e91d9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.027+0000 7ff2223e1740 1 --2- 172.21.15.12:0/253541127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5e91ca1e0 0x55d5e91d9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.027+0000 7ff2223e1740 1 -- 172.21.15.12:0/253541127 >> 172.21.15.12:0/253541127 conn(0x55d5e91c5750 msgr2=0x55d5e91d4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.027+0000 7ff1fa000640 0 client.26123 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:04:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.027+0000 7ff2223e1740 1 -- 172.21.15.12:0/253541127 shutdown_connections 2024-03-23T23:04:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.028+0000 7ff2223e1740 1 -- 172.21.15.12:0/253541127 wait complete. 2024-03-23T23:04:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.028+0000 7ff2223e1740 20 client.26123 trim_cache size 0 max 16384 2024-03-23T23:04:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:04:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:04:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:04:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:04:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.028+0000 7ff21f600640 1 -- 172.21.15.12:0/1939022183 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff2000a2920 msgr2=0x7ff2000a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.028+0000 7ff21f600640 1 --2- 172.21.15.12:0/1939022183 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff2000a2920 0x7ff2000a4de0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x55d5e91cb430 tx=0x7ff2100096a0 comp rx=0 tx=0).stop 2024-03-23T23:04:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.028+0000 7ff21f600640 1 -- 172.21.15.12:0/1939022183 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff218103900 msgr2=0x7ff218079450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:04:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.028+0000 7ff21f600640 1 --2- 172.21.15.12:0/1939022183 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff218103900 0x7ff218079450 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00e9c0 tx=0x7ff20c00ee80 comp rx=0 tx=0).stop 2024-03-23T23:04:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.028+0000 7ff21f600640 1 -- 172.21.15.12:0/1939022183 shutdown_connections 2024-03-23T23:04:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.028+0000 7ff21f600640 1 --2- 172.21.15.12:0/1939022183 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff2000a2920 0x7ff2000a4de0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.028+0000 7ff21f600640 1 --2- 172.21.15.12:0/1939022183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2181054b0 0x7ff218075fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.028+0000 7ff21f600640 1 --2- 172.21.15.12:0/1939022183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff218104af0 0x7ff218079990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.028+0000 7ff21f600640 1 --2- 172.21.15.12:0/1939022183 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff218103900 0x7ff218079450 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:04:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.028+0000 7ff21f600640 1 -- 172.21.15.12:0/1939022183 >> 172.21.15.12:0/1939022183 conn(0x7ff2180fee90 msgr2=0x7ff218107d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:04:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.029+0000 7ff21f600640 1 -- 172.21.15.12:0/1939022183 shutdown_connections 2024-03-23T23:04:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:04:50.029+0000 7ff21f600640 1 -- 172.21.15.12:0/1939022183 wait complete. 2024-03-23T23:04:51.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:50 smithi012 ceph-mon[31695]: pgmap v481: 97 pgs: 97 active+clean; 1.6 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:04:51.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:50 smithi173 ceph-mon[36116]: pgmap v481: 97 pgs: 97 active+clean; 1.6 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:04:51.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:50 smithi203 ceph-mon[36693]: pgmap v481: 97 pgs: 97 active+clean; 1.6 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:04:53.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:52 smithi012 ceph-mon[31695]: pgmap v482: 97 pgs: 97 active+clean; 1.6 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-03-23T23:04:53.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:52 smithi173 ceph-mon[36116]: pgmap v482: 97 pgs: 97 active+clean; 1.6 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-03-23T23:04:53.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:52 smithi203 ceph-mon[36693]: pgmap v482: 97 pgs: 97 active+clean; 1.6 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-03-23T23:04:55.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:54 smithi012 ceph-mon[31695]: pgmap v483: 97 pgs: 97 active+clean; 1.7 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:04:55.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:54 smithi173 ceph-mon[36116]: pgmap v483: 97 pgs: 97 active+clean; 1.7 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:04:55.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:54 smithi203 ceph-mon[36693]: pgmap v483: 97 pgs: 97 active+clean; 1.7 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:04:57.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:56 smithi012 ceph-mon[31695]: pgmap v484: 97 pgs: 97 active+clean; 1.7 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:04:57.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:56 smithi173 ceph-mon[36116]: pgmap v484: 97 pgs: 97 active+clean; 1.7 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:04:57.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:56 smithi203 ceph-mon[36693]: pgmap v484: 97 pgs: 97 active+clean; 1.7 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:04:59.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:04:58 smithi012 ceph-mon[31695]: pgmap v485: 97 pgs: 97 active+clean; 1.7 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:04:59.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:04:58 smithi173 ceph-mon[36116]: pgmap v485: 97 pgs: 97 active+clean; 1.7 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:04:59.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:04:58 smithi203 ceph-mon[36693]: pgmap v485: 97 pgs: 97 active+clean; 1.7 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:05:01.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:00 smithi012 ceph-mon[31695]: pgmap v486: 97 pgs: 97 active+clean; 1.7 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:05:01.120 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:00 smithi173 ceph-mon[36116]: pgmap v486: 97 pgs: 97 active+clean; 1.7 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:05:01.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:00 smithi203 ceph-mon[36693]: pgmap v486: 97 pgs: 97 active+clean; 1.7 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:05:03.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:02 smithi012 ceph-mon[31695]: pgmap v487: 97 pgs: 97 active+clean; 1.7 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:05:03.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:02 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:05:03.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:02 smithi173 ceph-mon[36116]: pgmap v487: 97 pgs: 97 active+clean; 1.7 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:05:03.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:02 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:05:03.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:02 smithi203 ceph-mon[36693]: pgmap v487: 97 pgs: 97 active+clean; 1.7 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:05:03.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:02 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:05:03.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.167+0000 7f8d76400640 1 -- 172.21.15.12:0/222471814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d701054b0 msgr2=0x7f8d7010bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:03.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.167+0000 7f8d76400640 1 --2- 172.21.15.12:0/222471814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d701054b0 0x7f8d7010bd40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8d68009900 tx=0x7f8d6802bc40 comp rx=0 tx=0).stop 2024-03-23T23:05:03.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.167+0000 7f8d76400640 1 -- 172.21.15.12:0/222471814 shutdown_connections 2024-03-23T23:05:03.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.167+0000 7f8d76400640 1 --2- 172.21.15.12:0/222471814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d701054b0 0x7f8d7010bd40 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.167+0000 7f8d76400640 1 --2- 172.21.15.12:0/222471814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d70104af0 0x7f8d70104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.167+0000 7f8d76400640 1 --2- 172.21.15.12:0/222471814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d70103900 0x7f8d70103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.167+0000 7f8d76400640 1 -- 172.21.15.12:0/222471814 >> 172.21.15.12:0/222471814 conn(0x7f8d700fee90 msgr2=0x7f8d701012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:03.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.167+0000 7f8d76400640 1 -- 172.21.15.12:0/222471814 shutdown_connections 2024-03-23T23:05:03.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.167+0000 7f8d76400640 1 -- 172.21.15.12:0/222471814 wait complete. 2024-03-23T23:05:03.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.168+0000 7f8d76400640 1 Processor -- start 2024-03-23T23:05:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.169+0000 7f8d76400640 1 -- start start 2024-03-23T23:05:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.169+0000 7f8d76400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d70103900 0x7f8d70199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.169+0000 7f8d76400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d70104af0 0x7f8d70199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.169+0000 7f8d76400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d701054b0 0x7f8d701a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.169+0000 7f8d76400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7010f5b0 con 0x7f8d70104af0 2024-03-23T23:05:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.169+0000 7f8d76400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7010f430 con 0x7f8d701054b0 2024-03-23T23:05:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.169+0000 7f8d76400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7010f730 con 0x7f8d70103900 2024-03-23T23:05:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.169+0000 7f8d6f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d70103900 0x7f8d70199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.169+0000 7f8d6fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d701054b0 0x7f8d701a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.169+0000 7f8d66a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d70104af0 0x7f8d70199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.169+0000 7f8d6fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d701054b0 0x7f8d701a1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:42792/0 (socket says 172.21.15.12:42792) 2024-03-23T23:05:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.169+0000 7f8d6fe00640 1 -- 172.21.15.12:0/1924866067 learned_addr learned my addr 172.21.15.12:0/1924866067 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.169+0000 7f8d6f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d70103900 0x7f8d70199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:40358/0 (socket says 172.21.15.12:40358) 2024-03-23T23:05:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.170+0000 7f8d6fe00640 1 -- 172.21.15.12:0/1924866067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d70103900 msgr2=0x7f8d70199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.170+0000 7f8d6fe00640 1 --2- 172.21.15.12:0/1924866067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d70103900 0x7f8d70199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.170+0000 7f8d6fe00640 1 -- 172.21.15.12:0/1924866067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d70104af0 msgr2=0x7f8d70199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.170+0000 7f8d6fe00640 1 --2- 172.21.15.12:0/1924866067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d70104af0 0x7f8d70199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.170+0000 7f8d6fe00640 1 -- 172.21.15.12:0/1924866067 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d701a16d0 con 0x7f8d701054b0 2024-03-23T23:05:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.170+0000 7f8d6fe00640 1 --2- 172.21.15.12:0/1924866067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d701054b0 0x7f8d701a1060 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8d6802c180 tx=0x7f8d6802c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.170+0000 7f8d6cc00640 1 -- 172.21.15.12:0/1924866067 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d68041070 con 0x7f8d701054b0 2024-03-23T23:05:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.170+0000 7f8d6cc00640 1 -- 172.21.15.12:0/1924866067 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d680054e0 con 0x7f8d701054b0 2024-03-23T23:05:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.170+0000 7f8d6cc00640 1 -- 172.21.15.12:0/1924866067 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d68034440 con 0x7f8d701054b0 2024-03-23T23:05:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.170+0000 7f8d76400640 1 -- 172.21.15.12:0/1924866067 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d701a1960 con 0x7f8d701054b0 2024-03-23T23:05:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.171+0000 7f8d76400640 1 -- 172.21.15.12:0/1924866067 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d701a1e00 con 0x7f8d701054b0 2024-03-23T23:05:03.183 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.174+0000 7f8d6cc00640 1 -- 172.21.15.12:0/1924866067 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8d68044070 con 0x7f8d701054b0 2024-03-23T23:05:03.183 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.175+0000 7f8d791ed740 1 Processor -- start 2024-03-23T23:05:03.183 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.175+0000 7f8d791ed740 1 -- start start 2024-03-23T23:05:03.183 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.175+0000 7f8d791ed740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557eb1b91e0 0x5557eb1b9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.175+0000 7f8d791ed740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5557eb1b9bd0 0x5557eb1be6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.175+0000 7f8d6cc00640 1 --2- 172.21.15.12:0/1924866067 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8d540a2920 0x7f8d540a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.175+0000 7f8d791ed740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557eb1bed40 0x5557eb1c11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.175+0000 7f8d791ed740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557eb17c3b0 con 0x5557eb1b9bd0 2024-03-23T23:05:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.175+0000 7f8d791ed740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557eb11e830 con 0x5557eb1b91e0 2024-03-23T23:05:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.175+0000 7f8d791ed740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557eb0c75d0 con 0x5557eb1bed40 2024-03-23T23:05:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.176+0000 7f8d6fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557eb1bed40 0x5557eb1c11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.176+0000 7f8d66a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557eb1b91e0 0x5557eb1b9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.176+0000 7f8d6f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5557eb1b9bd0 0x5557eb1be6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.176+0000 7f8d6fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557eb1bed40 0x5557eb1c11d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:40374/0 (socket says 172.21.15.12:40374) 2024-03-23T23:05:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.176+0000 7f8d6fe00640 1 -- 172.21.15.12:0/142139318 learned_addr learned my addr 172.21.15.12:0/142139318 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.176+0000 7f8d66a00640 1 --2- 172.21.15.12:0/1924866067 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8d540a2920 0x7f8d540a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.176+0000 7f8d6cc00640 1 -- 172.21.15.12:0/1924866067 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8d680f05c0 con 0x7f8d701054b0 2024-03-23T23:05:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.176+0000 7f8d6fe00640 1 -- 172.21.15.12:0/142139318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557eb1b91e0 msgr2=0x5557eb1b9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.176+0000 7f8d6fe00640 1 --2- 172.21.15.12:0/142139318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557eb1b91e0 0x5557eb1b9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.176+0000 7f8d6fe00640 1 -- 172.21.15.12:0/142139318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5557eb1b9bd0 msgr2=0x5557eb1be6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.176+0000 7f8d6fe00640 1 --2- 172.21.15.12:0/142139318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5557eb1b9bd0 0x5557eb1be6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.176+0000 7f8d6fe00640 1 -- 172.21.15.12:0/142139318 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5557eb0df2c0 con 0x5557eb1bed40 2024-03-23T23:05:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.176+0000 7f8d6fe00640 1 --2- 172.21.15.12:0/142139318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557eb1bed40 0x5557eb1c11d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8d6802f020 tx=0x7f8d68003990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9775a2547a1490d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.177+0000 7f8d66a00640 1 --2- 172.21.15.12:0/1924866067 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8d540a2920 0x7f8d540a4de0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x5557eb1ba430 tx=0x7f8d58005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.177+0000 7f8d64c00640 1 -- 172.21.15.12:0/142139318 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6804b070 con 0x5557eb1bed40 2024-03-23T23:05:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.177+0000 7f8d64c00640 1 -- 172.21.15.12:0/142139318 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d68005690 con 0x5557eb1bed40 2024-03-23T23:05:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.177+0000 7f8d64c00640 1 -- 172.21.15.12:0/142139318 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d680bcfc0 con 0x5557eb1bed40 2024-03-23T23:05:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.177+0000 7f8d791ed740 1 -- 172.21.15.12:0/142139318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557eb1bed40 msgr2=0x5557eb1c11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.177+0000 7f8d791ed740 1 --2- 172.21.15.12:0/142139318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557eb1bed40 0x5557eb1c11d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8d6802f020 tx=0x7f8d68003990 comp rx=0 tx=0).stop 2024-03-23T23:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.177+0000 7f8d791ed740 1 -- 172.21.15.12:0/142139318 shutdown_connections 2024-03-23T23:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.177+0000 7f8d791ed740 1 --2- 172.21.15.12:0/142139318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557eb1bed40 0x5557eb1c11d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.177+0000 7f8d791ed740 1 --2- 172.21.15.12:0/142139318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5557eb1b9bd0 0x5557eb1be6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.177+0000 7f8d791ed740 1 --2- 172.21.15.12:0/142139318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557eb1b91e0 0x5557eb1b9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.177+0000 7f8d791ed740 1 -- 172.21.15.12:0/142139318 >> 172.21.15.12:0/142139318 conn(0x5557eb1b4750 msgr2=0x5557eb1b4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.178+0000 7f8d791ed740 1 -- 172.21.15.12:0/142139318 shutdown_connections 2024-03-23T23:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.178+0000 7f8d791ed740 1 -- 172.21.15.12:0/142139318 wait complete. 2024-03-23T23:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.178+0000 7f8d791ed740 1 -- start start 2024-03-23T23:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.178+0000 7f8d791ed740 1 Processor -- start 2024-03-23T23:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.179+0000 7f8d791ed740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557eb1b91e0 0x5557eb1c3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.179+0000 7f8d791ed740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5557eb1b9bd0 0x5557eb1c3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.179+0000 7f8d791ed740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557eb1bed40 0x5557eb1ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.179+0000 7f8d791ed740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557eb0daa20 con 0x5557eb1b9bd0 2024-03-23T23:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.179+0000 7f8d791ed740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557eb11a8d0 con 0x5557eb1bed40 2024-03-23T23:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.179+0000 7f8d791ed740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557eb0c75d0 con 0x5557eb1b91e0 2024-03-23T23:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.179+0000 7f8d791ed740 10 client.0 _ll_register_callbacks cb 0x5557eb1171c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.179+0000 7f8d6f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557eb1b91e0 0x5557eb1c3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.179+0000 7f8d6fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557eb1bed40 0x5557eb1ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.179+0000 7f8d6f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557eb1b91e0 0x5557eb1c3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:40376/0 (socket says 172.21.15.12:40376) 2024-03-23T23:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.179+0000 7f8d6f400640 1 -- 172.21.15.12:0/2949621165 learned_addr learned my addr 172.21.15.12:0/2949621165 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.179+0000 7f8d66a00640 1 --2- 172.21.15.12:0/2949621165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5557eb1b9bd0 0x5557eb1c3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.180+0000 7f8d6fe00640 1 -- 172.21.15.12:0/2949621165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557eb1b91e0 msgr2=0x5557eb1c3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.180+0000 7f8d6fe00640 1 --2- 172.21.15.12:0/2949621165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557eb1b91e0 0x5557eb1c3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.180+0000 7f8d6fe00640 1 -- 172.21.15.12:0/2949621165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5557eb1b9bd0 msgr2=0x5557eb1c3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.180+0000 7f8d6fe00640 1 --2- 172.21.15.12:0/2949621165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5557eb1b9bd0 0x5557eb1c3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.180+0000 7f8d6fe00640 1 -- 172.21.15.12:0/2949621165 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5557eae33210 con 0x5557eb1bed40 2024-03-23T23:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.180+0000 7f8d6f400640 1 --2- 172.21.15.12:0/2949621165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557eb1b91e0 0x5557eb1c3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.180+0000 7f8d6fe00640 1 --2- 172.21.15.12:0/2949621165 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557eb1bed40 0x5557eb1ca580 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8d6802c220 tx=0x7f8d680bc9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5f9ca75ecd5057 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.180+0000 7f8d5d600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.180+0000 7f8d5d600640 1 -- 172.21.15.12:0/2949621165 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6804b070 con 0x5557eb1bed40 2024-03-23T23:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.180+0000 7f8d5d600640 1 -- 172.21.15.12:0/2949621165 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8d680be240 con 0x5557eb1bed40 2024-03-23T23:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.180+0000 7f8d791ed740 1 -- 172.21.15.12:0/2949621165 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5557eb12bd70 con 0x5557eb1bed40 2024-03-23T23:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.180+0000 7f8d5d600640 1 -- 172.21.15.12:0/2949621165 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d680c1060 con 0x5557eb1bed40 2024-03-23T23:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.180+0000 7f8d791ed740 10 client.26135 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.180+0000 7f8d791ed740 1 -- 172.21.15.12:0/2949621165 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5557eaeff620 con 0x5557eb1bed40 2024-03-23T23:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.181+0000 7f8d5d600640 1 -- 172.21.15.12:0/2949621165 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5557eb12bd70 con 0x5557eb1bed40 2024-03-23T23:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.181+0000 7f8d5d600640 1 -- 172.21.15.12:0/2949621165 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8d680d2070 con 0x5557eb1bed40 2024-03-23T23:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.181+0000 7f8d791ed740 10 client.26135 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.181+0000 7f8d791ed740 10 client.26135 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.181+0000 7f8d791ed740 20 client.26135 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.181+0000 7f8d791ed740 1 --2- 172.21.15.12:0/2949621165 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5557eb1ceb10 0x5557eb1d0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.181+0000 7f8d791ed740 4 client.26135 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.181+0000 7f8d791ed740 1 -- 172.21.15.12:0/2949621165 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5557eb1260c0 con 0x5557eb1ceb10 2024-03-23T23:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.181+0000 7f8d6f400640 1 --2- 172.21.15.12:0/2949621165 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5557eb1ceb10 0x5557eb1d0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.182+0000 7f8d6f400640 1 --2- 172.21.15.12:0/2949621165 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5557eb1ceb10 0x5557eb1d0f00 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8586543055eb3a61 server_cookie=4d0a401b83c5f012 in_seq=0 out_seq=0 2024-03-23T23:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.182+0000 7f8d5d600640 10 client.26135 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.182+0000 7f8d5d600640 1 -- 172.21.15.12:0/2949621165 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5557eb1260c0 con 0x5557eb1ceb10 2024-03-23T23:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.182+0000 7f8d5d600640 10 client.26135 handle_command_reply: tid=0 2024-03-23T23:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.182+0000 7f8d791ed740 1 client.26135 shutdown 2024-03-23T23:05:03.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.183+0000 7f8d791ed740 1 -- 172.21.15.12:0/2949621165 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557eb1bed40 msgr2=0x5557eb1ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:03.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.183+0000 7f8d791ed740 1 --2- 172.21.15.12:0/2949621165 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557eb1bed40 0x5557eb1ca580 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8d6802c220 tx=0x7f8d680bc9a0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.183+0000 7f8d791ed740 1 -- 172.21.15.12:0/2949621165 shutdown_connections 2024-03-23T23:05:03.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.183+0000 7f8d791ed740 1 --2- 172.21.15.12:0/2949621165 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5557eb1ceb10 0x5557eb1d0f00 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.183+0000 7f8d791ed740 1 --2- 172.21.15.12:0/2949621165 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557eb1bed40 0x5557eb1ca580 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.183+0000 7f8d791ed740 1 --2- 172.21.15.12:0/2949621165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5557eb1b9bd0 0x5557eb1c3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.183+0000 7f8d791ed740 1 --2- 172.21.15.12:0/2949621165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557eb1b91e0 0x5557eb1c3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.183+0000 7f8d791ed740 1 -- 172.21.15.12:0/2949621165 >> 172.21.15.12:0/2949621165 conn(0x5557eb1b4750 msgr2=0x5557eb1b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:03.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.183+0000 7f8d5d600640 0 client.26135 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:03.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.183+0000 7f8d791ed740 1 -- 172.21.15.12:0/2949621165 shutdown_connections 2024-03-23T23:05:03.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.183+0000 7f8d791ed740 1 -- 172.21.15.12:0/2949621165 wait complete. 2024-03-23T23:05:03.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.183+0000 7f8d791ed740 20 client.26135 trim_cache size 0 max 16384 2024-03-23T23:05:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.198+0000 7f8d791ed740 1 Processor -- start 2024-03-23T23:05:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.199+0000 7f8d791ed740 1 -- start start 2024-03-23T23:05:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.199+0000 7f8d791ed740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557eb1b91e0 0x5557eb1c2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.199+0000 7f8d791ed740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557eb1b9bd0 0x5557eb1ca540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.199+0000 7f8d791ed740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5557eb1bed40 0x5557eb1c3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.199+0000 7f8d791ed740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d701acfc0 con 0x5557eb1bed40 2024-03-23T23:05:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.199+0000 7f8d791ed740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d701aaac0 con 0x5557eb1b91e0 2024-03-23T23:05:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.199+0000 7f8d791ed740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d701a8b60 con 0x5557eb1b9bd0 2024-03-23T23:05:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.199+0000 7f8d6f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557eb1b91e0 0x5557eb1c2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.199+0000 7f8d6f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557eb1b91e0 0x5557eb1c2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:42822/0 (socket says 172.21.15.12:42822) 2024-03-23T23:05:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.199+0000 7f8d6f400640 1 -- 172.21.15.12:0/3100702280 learned_addr learned my addr 172.21.15.12:0/3100702280 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.199+0000 7f8d66a00640 1 --2- 172.21.15.12:0/3100702280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557eb1b9bd0 0x5557eb1ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.199+0000 7f8d6fe00640 1 --2- 172.21.15.12:0/3100702280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5557eb1bed40 0x5557eb1c3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.200+0000 7f8d6f400640 1 -- 172.21.15.12:0/3100702280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557eb1b9bd0 msgr2=0x5557eb1ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.200+0000 7f8d6f400640 1 --2- 172.21.15.12:0/3100702280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557eb1b9bd0 0x5557eb1ca540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.200+0000 7f8d6f400640 1 -- 172.21.15.12:0/3100702280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5557eb1bed40 msgr2=0x5557eb1c3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:03.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.200+0000 7f8d6f400640 1 --2- 172.21.15.12:0/3100702280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5557eb1bed40 0x5557eb1c3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.200+0000 7f8d6f400640 1 -- 172.21.15.12:0/3100702280 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5557eaeff620 con 0x5557eb1b91e0 2024-03-23T23:05:03.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.200+0000 7f8d6f400640 1 --2- 172.21.15.12:0/3100702280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557eb1b91e0 0x5557eb1c2620 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8d60009ac0 tx=0x7f8d60009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed172c25644db8ac server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:03.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.200+0000 7f8d5fe00640 1 -- 172.21.15.12:0/3100702280 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d60004b20 con 0x5557eb1b91e0 2024-03-23T23:05:03.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.200+0000 7f8d5fe00640 1 -- 172.21.15.12:0/3100702280 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d60024030 con 0x5557eb1b91e0 2024-03-23T23:05:03.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.200+0000 7f8d5fe00640 1 -- 172.21.15.12:0/3100702280 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d600055f0 con 0x5557eb1b91e0 2024-03-23T23:05:03.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.201+0000 7f8d791ed740 1 -- 172.21.15.12:0/3100702280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557eb1b91e0 msgr2=0x5557eb1c2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:03.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.201+0000 7f8d791ed740 1 --2- 172.21.15.12:0/3100702280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557eb1b91e0 0x5557eb1c2620 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8d60009ac0 tx=0x7f8d60009500 comp rx=0 tx=0).stop 2024-03-23T23:05:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.201+0000 7f8d791ed740 1 -- 172.21.15.12:0/3100702280 shutdown_connections 2024-03-23T23:05:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.201+0000 7f8d791ed740 1 --2- 172.21.15.12:0/3100702280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5557eb1bed40 0x5557eb1c3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.201+0000 7f8d791ed740 1 --2- 172.21.15.12:0/3100702280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557eb1b9bd0 0x5557eb1ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.201+0000 7f8d791ed740 1 --2- 172.21.15.12:0/3100702280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557eb1b91e0 0x5557eb1c2620 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.201+0000 7f8d791ed740 1 -- 172.21.15.12:0/3100702280 >> 172.21.15.12:0/3100702280 conn(0x5557eb1b4750 msgr2=0x5557eb1c1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.201+0000 7f8d791ed740 1 -- 172.21.15.12:0/3100702280 shutdown_connections 2024-03-23T23:05:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.201+0000 7f8d791ed740 1 -- 172.21.15.12:0/3100702280 wait complete. 2024-03-23T23:05:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.202+0000 7f8d791ed740 1 -- start start 2024-03-23T23:05:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.202+0000 7f8d791ed740 1 Processor -- start 2024-03-23T23:05:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.202+0000 7f8d791ed740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557eb1b91e0 0x5557eb1c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.202+0000 7f8d791ed740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5557eb1b9bd0 0x5557eb1c7b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.202+0000 7f8d791ed740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557eb1bed40 0x5557eb1c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.202+0000 7f8d791ed740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d701a9160 con 0x5557eb1b9bd0 2024-03-23T23:05:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.202+0000 7f8d791ed740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d701aa460 con 0x5557eb1bed40 2024-03-23T23:05:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.202+0000 7f8d791ed740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d701a97a0 con 0x5557eb1b91e0 2024-03-23T23:05:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.202+0000 7f8d791ed740 10 client.0 _ll_register_callbacks cb 0x7f8d58009b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.203+0000 7f8d6fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557eb1bed40 0x5557eb1c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.203+0000 7f8d6fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557eb1bed40 0x5557eb1c8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:42824/0 (socket says 172.21.15.12:42824) 2024-03-23T23:05:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.203+0000 7f8d6fe00640 1 -- 172.21.15.12:0/3719011369 learned_addr learned my addr 172.21.15.12:0/3719011369 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.203+0000 7f8d6f400640 1 --2- 172.21.15.12:0/3719011369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557eb1b91e0 0x5557eb1c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.203+0000 7f8d66a00640 1 --2- 172.21.15.12:0/3719011369 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5557eb1b9bd0 0x5557eb1c7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.203+0000 7f8d6fe00640 1 -- 172.21.15.12:0/3719011369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557eb1b91e0 msgr2=0x5557eb1c75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.203+0000 7f8d6fe00640 1 --2- 172.21.15.12:0/3719011369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557eb1b91e0 0x5557eb1c75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.203+0000 7f8d6fe00640 1 -- 172.21.15.12:0/3719011369 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5557eb1b9bd0 msgr2=0x5557eb1c7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.203+0000 7f8d6fe00640 1 --2- 172.21.15.12:0/3719011369 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5557eb1b9bd0 0x5557eb1c7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.203+0000 7f8d6fe00640 1 -- 172.21.15.12:0/3719011369 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5557eb1c9100 con 0x5557eb1bed40 2024-03-23T23:05:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.204+0000 7f8d6fe00640 1 --2- 172.21.15.12:0/3719011369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557eb1bed40 0x5557eb1c8b10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8d68004650 tx=0x7f8d680ee110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a1e1476d12ce6f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.204+0000 7f8d5d600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:05:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.204+0000 7f8d5d600640 1 -- 172.21.15.12:0/3719011369 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6804b070 con 0x5557eb1bed40 2024-03-23T23:05:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.204+0000 7f8d5d600640 1 -- 172.21.15.12:0/3719011369 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8d680b7d30 con 0x5557eb1bed40 2024-03-23T23:05:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.204+0000 7f8d791ed740 1 -- 172.21.15.12:0/3719011369 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8d60024030 con 0x5557eb1bed40 2024-03-23T23:05:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.204+0000 7f8d5d600640 1 -- 172.21.15.12:0/3719011369 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d680b9260 con 0x5557eb1bed40 2024-03-23T23:05:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.204+0000 7f8d791ed740 10 client.26141 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.204+0000 7f8d791ed740 1 -- 172.21.15.12:0/3719011369 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5557eb1c9100 con 0x5557eb1bed40 2024-03-23T23:05:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.205+0000 7f8d5d600640 1 -- 172.21.15.12:0/3719011369 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d60024030 con 0x5557eb1bed40 2024-03-23T23:05:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.205+0000 7f8d5d600640 1 -- 172.21.15.12:0/3719011369 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8d680d2070 con 0x5557eb1bed40 2024-03-23T23:05:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.205+0000 7f8d791ed740 10 client.26141 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.205+0000 7f8d791ed740 10 client.26141 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.205+0000 7f8d791ed740 20 client.26141 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.205+0000 7f8d791ed740 1 --2- 172.21.15.12:0/3719011369 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5557eb22a360 0x5557eb22c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.205+0000 7f8d791ed740 4 client.26141 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:05:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.205+0000 7f8d791ed740 1 -- 172.21.15.12:0/3719011369 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5557eb0a45d0 con 0x5557eb22a360 2024-03-23T23:05:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.205+0000 7f8d6f400640 1 --2- 172.21.15.12:0/3719011369 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5557eb22a360 0x5557eb22c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.206+0000 7f8d6f400640 1 --2- 172.21.15.12:0/3719011369 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5557eb22a360 0x5557eb22c750 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66a2e392858b715f server_cookie=5010e7c54b7e6851 in_seq=0 out_seq=0 2024-03-23T23:05:03.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.206+0000 7f8d5d600640 10 client.26141 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.219+0000 7f8d5d600640 1 -- 172.21.15.12:0/3719011369 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5557eb0a45d0 con 0x5557eb22a360 2024-03-23T23:05:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.219+0000 7f8d5d600640 10 client.26141 handle_command_reply: tid=0 2024-03-23T23:05:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.220+0000 7f8d791ed740 1 client.26141 shutdown 2024-03-23T23:05:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.220+0000 7f8d791ed740 1 -- 172.21.15.12:0/3719011369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557eb1bed40 msgr2=0x5557eb1c8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.220+0000 7f8d791ed740 1 --2- 172.21.15.12:0/3719011369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557eb1bed40 0x5557eb1c8b10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8d68004650 tx=0x7f8d680ee110 comp rx=0 tx=0).stop 2024-03-23T23:05:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.220+0000 7f8d791ed740 1 -- 172.21.15.12:0/3719011369 shutdown_connections 2024-03-23T23:05:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.220+0000 7f8d791ed740 1 --2- 172.21.15.12:0/3719011369 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5557eb22a360 0x5557eb22c750 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.220+0000 7f8d791ed740 1 --2- 172.21.15.12:0/3719011369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557eb1bed40 0x5557eb1c8b10 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.220+0000 7f8d791ed740 1 --2- 172.21.15.12:0/3719011369 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5557eb1b9bd0 0x5557eb1c7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.220+0000 7f8d791ed740 1 --2- 172.21.15.12:0/3719011369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557eb1b91e0 0x5557eb1c75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.220+0000 7f8d5d600640 0 client.26141 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.220+0000 7f8d791ed740 1 -- 172.21.15.12:0/3719011369 >> 172.21.15.12:0/3719011369 conn(0x5557eb1b4750 msgr2=0x5557eb1baef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.220+0000 7f8d791ed740 1 -- 172.21.15.12:0/3719011369 shutdown_connections 2024-03-23T23:05:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.220+0000 7f8d791ed740 1 -- 172.21.15.12:0/3719011369 wait complete. 2024-03-23T23:05:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.220+0000 7f8d791ed740 20 client.26141 trim_cache size 0 max 16384 2024-03-23T23:05:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:05:03.222 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:05:03.222 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:05:03.222 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:05:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.221+0000 7f8d76400640 1 -- 172.21.15.12:0/1924866067 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8d540a2920 msgr2=0x7f8d540a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.221+0000 7f8d76400640 1 --2- 172.21.15.12:0/1924866067 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8d540a2920 0x7f8d540a4de0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x5557eb1ba430 tx=0x7f8d58005f90 comp rx=0 tx=0).stop 2024-03-23T23:05:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.221+0000 7f8d76400640 1 -- 172.21.15.12:0/1924866067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d701054b0 msgr2=0x7f8d701a1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.221+0000 7f8d76400640 1 --2- 172.21.15.12:0/1924866067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d701054b0 0x7f8d701a1060 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8d6802c180 tx=0x7f8d6802c660 comp rx=0 tx=0).stop 2024-03-23T23:05:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.221+0000 7f8d76400640 1 -- 172.21.15.12:0/1924866067 shutdown_connections 2024-03-23T23:05:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.221+0000 7f8d76400640 1 --2- 172.21.15.12:0/1924866067 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8d540a2920 0x7f8d540a4de0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.221+0000 7f8d76400640 1 --2- 172.21.15.12:0/1924866067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d701054b0 0x7f8d701a1060 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.221+0000 7f8d76400640 1 --2- 172.21.15.12:0/1924866067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d70104af0 0x7f8d70199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.221+0000 7f8d76400640 1 --2- 172.21.15.12:0/1924866067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d70103900 0x7f8d70199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.221+0000 7f8d76400640 1 -- 172.21.15.12:0/1924866067 >> 172.21.15.12:0/1924866067 conn(0x7f8d700fee90 msgr2=0x7f8d70107d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.221+0000 7f8d76400640 1 -- 172.21.15.12:0/1924866067 shutdown_connections 2024-03-23T23:05:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:03.221+0000 7f8d76400640 1 -- 172.21.15.12:0/1924866067 wait complete. 2024-03-23T23:05:04.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:05:04.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:05:04.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:05:04.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:05:04.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:05:04.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:05:04.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:05:04.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:05:04.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:05:05.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:05 smithi203 ceph-mon[36693]: pgmap v488: 97 pgs: 97 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:05:05.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:05 smithi012 ceph-mon[31695]: pgmap v488: 97 pgs: 97 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:05:05.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:05 smithi173 ceph-mon[36116]: pgmap v488: 97 pgs: 97 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:05:07.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:07 smithi203 ceph-mon[36693]: pgmap v489: 97 pgs: 97 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:05:07.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:07 smithi012 ceph-mon[31695]: pgmap v489: 97 pgs: 97 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:05:07.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:07 smithi173 ceph-mon[36116]: pgmap v489: 97 pgs: 97 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:05:09.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:09 smithi203 ceph-mon[36693]: pgmap v490: 97 pgs: 97 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:05:09.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:09 smithi012 ceph-mon[31695]: pgmap v490: 97 pgs: 97 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:05:09.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:09 smithi173 ceph-mon[36116]: pgmap v490: 97 pgs: 97 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:05:11.409 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:11 smithi203 ceph-mon[36693]: pgmap v491: 97 pgs: 97 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:05:11.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:11 smithi012 ceph-mon[31695]: pgmap v491: 97 pgs: 97 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:05:11.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:11 smithi173 ceph-mon[36116]: pgmap v491: 97 pgs: 97 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:05:13.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:13 smithi203 ceph-mon[36693]: pgmap v492: 97 pgs: 97 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:05:13.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:13 smithi012 ceph-mon[31695]: pgmap v492: 97 pgs: 97 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:05:13.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:13 smithi173 ceph-mon[36116]: pgmap v492: 97 pgs: 97 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:05:14.930 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:05:14.931 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:05:15.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.173+0000 7fdd3be00640 1 -- 172.21.15.12:0/513335534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd3c077c40 msgr2=0x7fdd3c0784e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:15.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.173+0000 7fdd3be00640 1 --2- 172.21.15.12:0/513335534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd3c077c40 0x7fdd3c0784e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdd34009ca0 tx=0x7fdd3402c100 comp rx=0 tx=0).stop 2024-03-23T23:05:15.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.173+0000 7fdd3be00640 1 -- 172.21.15.12:0/513335534 shutdown_connections 2024-03-23T23:05:15.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.173+0000 7fdd3be00640 1 --2- 172.21.15.12:0/513335534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd3c077c40 0x7fdd3c0784e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.173+0000 7fdd3be00640 1 --2- 172.21.15.12:0/513335534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd3c0772a0 0x7fdd3c077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.173+0000 7fdd3be00640 1 --2- 172.21.15.12:0/513335534 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd3c076060 0x7fdd3c076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.173+0000 7fdd3be00640 1 -- 172.21.15.12:0/513335534 >> 172.21.15.12:0/513335534 conn(0x7fdd3c0fed10 msgr2=0x7fdd3c101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:15.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.173+0000 7fdd3be00640 1 -- 172.21.15.12:0/513335534 shutdown_connections 2024-03-23T23:05:15.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.173+0000 7fdd3be00640 1 -- 172.21.15.12:0/513335534 wait complete. 2024-03-23T23:05:15.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.174+0000 7fdd3be00640 1 Processor -- start 2024-03-23T23:05:15.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.175+0000 7fdd3be00640 1 -- start start 2024-03-23T23:05:15.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.175+0000 7fdd3be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd3c076060 0x7fdd3c19de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.175+0000 7fdd3be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd3c0772a0 0x7fdd3c19e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.175+0000 7fdd3be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd3c077c40 0x7fdd3c1a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.175+0000 7fdd3be00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c113830 con 0x7fdd3c0772a0 2024-03-23T23:05:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.175+0000 7fdd3be00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c1136b0 con 0x7fdd3c076060 2024-03-23T23:05:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.175+0000 7fdd3be00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c1139b0 con 0x7fdd3c077c40 2024-03-23T23:05:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.175+0000 7fdd3aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd3c076060 0x7fdd3c19de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.175+0000 7fdd3b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd3c077c40 0x7fdd3c1a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.175+0000 7fdd3aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd3c076060 0x7fdd3c19de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59344/0 (socket says 172.21.15.12:59344) 2024-03-23T23:05:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.175+0000 7fdd3aa00640 1 -- 172.21.15.12:0/1256943667 learned_addr learned my addr 172.21.15.12:0/1256943667 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.175+0000 7fdd3a000640 1 --2- 172.21.15.12:0/1256943667 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd3c0772a0 0x7fdd3c19e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.176+0000 7fdd3b400640 1 -- 172.21.15.12:0/1256943667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd3c076060 msgr2=0x7fdd3c19de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.176+0000 7fdd3b400640 1 --2- 172.21.15.12:0/1256943667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd3c076060 0x7fdd3c19de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.176+0000 7fdd3b400640 1 -- 172.21.15.12:0/1256943667 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd3c0772a0 msgr2=0x7fdd3c19e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.176+0000 7fdd3b400640 1 --2- 172.21.15.12:0/1256943667 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd3c0772a0 0x7fdd3c19e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.176+0000 7fdd3b400640 1 -- 172.21.15.12:0/1256943667 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd3c1a5a20 con 0x7fdd3c077c40 2024-03-23T23:05:15.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.176+0000 7fdd3b400640 1 --2- 172.21.15.12:0/1256943667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd3c077c40 0x7fdd3c1a53b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fdd3402c640 tx=0x7fdd34035d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:15.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.176+0000 7fdd33400640 1 -- 172.21.15.12:0/1256943667 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd34039040 con 0x7fdd3c077c40 2024-03-23T23:05:15.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.176+0000 7fdd33400640 1 -- 172.21.15.12:0/1256943667 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd34044070 con 0x7fdd3c077c40 2024-03-23T23:05:15.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.176+0000 7fdd33400640 1 -- 172.21.15.12:0/1256943667 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd340043e0 con 0x7fdd3c077c40 2024-03-23T23:05:15.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.176+0000 7fdd3be00640 1 -- 172.21.15.12:0/1256943667 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd3c1a5cb0 con 0x7fdd3c077c40 2024-03-23T23:05:15.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.177+0000 7fdd3be00640 1 -- 172.21.15.12:0/1256943667 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd3c1a6120 con 0x7fdd3c077c40 2024-03-23T23:05:15.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.179+0000 7fdd33400640 1 -- 172.21.15.12:0/1256943667 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fdd34004580 con 0x7fdd3c077c40 2024-03-23T23:05:15.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.180+0000 7fdd4458e740 1 Processor -- start 2024-03-23T23:05:15.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.180+0000 7fdd4458e740 1 -- start start 2024-03-23T23:05:15.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.180+0000 7fdd4458e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce18e42050 0x55ce18e42470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.180+0000 7fdd4458e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce18e42a40 0x55ce18e47540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.180+0000 7fdd4458e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce18e47bb0 0x55ce18e4a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.180+0000 7fdd4458e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce18da7250 con 0x55ce18e42a40 2024-03-23T23:05:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.180+0000 7fdd4458e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce18d4fa30 con 0x55ce18e47bb0 2024-03-23T23:05:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.180+0000 7fdd4458e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce18da32f0 con 0x55ce18e42050 2024-03-23T23:05:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.180+0000 7fdd3a000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce18e42050 0x55ce18e42470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.181+0000 7fdd3aa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce18e42a40 0x55ce18e47540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.181+0000 7fdd3a000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce18e42050 0x55ce18e42470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58786/0 (socket says 172.21.15.12:58786) 2024-03-23T23:05:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.181+0000 7fdd3aa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce18e42a40 0x55ce18e47540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40614/0 (socket says 172.21.15.12:40614) 2024-03-23T23:05:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.181+0000 7fdd3a000640 1 -- 172.21.15.12:0/2074104703 learned_addr learned my addr 172.21.15.12:0/2074104703 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.181+0000 7fdd3a000640 1 --2- 172.21.15.12:0/2074104703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce18e47bb0 0x55ce18e4a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.181+0000 7fdd33400640 1 --2- 172.21.15.12:0/1256943667 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdd200a2880 0x7fdd200a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.181+0000 7fdd3aa00640 1 -- 172.21.15.12:0/2074104703 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce18e42050 msgr2=0x55ce18e42470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.181+0000 7fdd3aa00640 1 --2- 172.21.15.12:0/2074104703 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce18e42050 0x55ce18e42470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.181+0000 7fdd3aa00640 1 -- 172.21.15.12:0/2074104703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce18e47bb0 msgr2=0x55ce18e4a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:15.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.181+0000 7fdd3aa00640 1 --2- 172.21.15.12:0/2074104703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce18e47bb0 0x55ce18e4a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.181+0000 7fdd3aa00640 1 -- 172.21.15.12:0/2074104703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce18d3a690 con 0x55ce18e42a40 2024-03-23T23:05:15.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.181+0000 7fdd3a000640 1 --2- 172.21.15.12:0/2074104703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce18e47bb0 0x55ce18e4a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:15.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.181+0000 7fdd33400640 1 -- 172.21.15.12:0/1256943667 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fdd340f1110 con 0x7fdd3c077c40 2024-03-23T23:05:15.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.181+0000 7fdd3b400640 1 --2- 172.21.15.12:0/1256943667 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdd200a2880 0x7fdd200a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.181+0000 7fdd3aa00640 1 --2- 172.21.15.12:0/2074104703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce18e42a40 0x55ce18e47540 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c009d20 tx=0x7fdd2c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ea19a966cd410da server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:15.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.182+0000 7fdd27e00640 1 -- 172.21.15.12:0/2074104703 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c005020 con 0x55ce18e42a40 2024-03-23T23:05:15.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.182+0000 7fdd27e00640 1 -- 172.21.15.12:0/2074104703 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd2c0047a0 con 0x55ce18e42a40 2024-03-23T23:05:15.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.182+0000 7fdd27e00640 1 -- 172.21.15.12:0/2074104703 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c019670 con 0x55ce18e42a40 2024-03-23T23:05:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.182+0000 7fdd4458e740 1 -- 172.21.15.12:0/2074104703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce18e42a40 msgr2=0x55ce18e47540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.182+0000 7fdd4458e740 1 --2- 172.21.15.12:0/2074104703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce18e42a40 0x55ce18e47540 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c009d20 tx=0x7fdd2c011810 comp rx=0 tx=0).stop 2024-03-23T23:05:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.182+0000 7fdd3b400640 1 --2- 172.21.15.12:0/1256943667 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdd200a2880 0x7fdd200a4d40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fdd340eebc0 tx=0x7fdd3402c100 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.183+0000 7fdd4458e740 1 -- 172.21.15.12:0/2074104703 shutdown_connections 2024-03-23T23:05:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.183+0000 7fdd4458e740 1 --2- 172.21.15.12:0/2074104703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce18e47bb0 0x55ce18e4a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.183+0000 7fdd4458e740 1 --2- 172.21.15.12:0/2074104703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce18e42a40 0x55ce18e47540 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.183+0000 7fdd4458e740 1 --2- 172.21.15.12:0/2074104703 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce18e42050 0x55ce18e42470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.183+0000 7fdd4458e740 1 -- 172.21.15.12:0/2074104703 >> 172.21.15.12:0/2074104703 conn(0x55ce18e0a200 msgr2=0x55ce18e3d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.183+0000 7fdd4458e740 1 -- 172.21.15.12:0/2074104703 shutdown_connections 2024-03-23T23:05:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.183+0000 7fdd4458e740 1 -- 172.21.15.12:0/2074104703 wait complete. 2024-03-23T23:05:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.183+0000 7fdd4458e740 1 -- start start 2024-03-23T23:05:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.183+0000 7fdd4458e740 1 Processor -- start 2024-03-23T23:05:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.184+0000 7fdd4458e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce18e42050 0x55ce18e4c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.184+0000 7fdd4458e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce18e42a40 0x55ce18e4c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.184+0000 7fdd4458e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce18e47bb0 0x55ce18e535c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.184+0000 7fdd4458e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce18e4ab60 con 0x55ce18e47bb0 2024-03-23T23:05:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.184+0000 7fdd4458e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce18d62e80 con 0x55ce18e42050 2024-03-23T23:05:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.184+0000 7fdd4458e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce18e4ace0 con 0x55ce18e42a40 2024-03-23T23:05:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.184+0000 7fdd4458e740 10 client.0 _ll_register_callbacks cb 0x55ce18d75130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.184+0000 7fdd3a000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce18e42050 0x55ce18e4c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.184+0000 7fdd3a000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce18e47bb0 0x55ce18e535c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.184+0000 7fdd3a000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce18e42050 0x55ce18e4c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59364/0 (socket says 172.21.15.12:59364) 2024-03-23T23:05:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.184+0000 7fdd3a000640 1 -- 172.21.15.12:0/2238684854 learned_addr learned my addr 172.21.15.12:0/2238684854 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.184+0000 7fdd3a000640 1 -- 172.21.15.12:0/2238684854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce18e42a40 msgr2=0x55ce18e4c5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-23T23:05:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.184+0000 7fdd3a000640 1 --2- 172.21.15.12:0/2238684854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce18e42a40 0x55ce18e4c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.185+0000 7fdd3a000640 1 -- 172.21.15.12:0/2238684854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce18e47bb0 msgr2=0x55ce18e535c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.185+0000 7fdd3a000640 1 --2- 172.21.15.12:0/2238684854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce18e47bb0 0x55ce18e535c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.185+0000 7fdd3a000640 1 -- 172.21.15.12:0/2238684854 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ce18b857d0 con 0x55ce18e42050 2024-03-23T23:05:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.185+0000 7fdd3a000640 1 --2- 172.21.15.12:0/2238684854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce18e42050 0x55ce18e4c0b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fdd28014f50 tx=0x7fdd280046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0a4fe98bdfd49bc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.185+0000 7fdd24c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:05:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.185+0000 7fdd24c00640 1 -- 172.21.15.12:0/2238684854 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2801b040 con 0x55ce18e42050 2024-03-23T23:05:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.185+0000 7fdd24c00640 1 -- 172.21.15.12:0/2238684854 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fdd28014310 con 0x55ce18e42050 2024-03-23T23:05:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.185+0000 7fdd4458e740 1 -- 172.21.15.12:0/2238684854 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdd2c0047a0 con 0x55ce18e42050 2024-03-23T23:05:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.185+0000 7fdd24c00640 1 -- 172.21.15.12:0/2238684854 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd28029050 con 0x55ce18e42050 2024-03-23T23:05:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.185+0000 7fdd4458e740 10 client.26153 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.185+0000 7fdd4458e740 1 -- 172.21.15.12:0/2238684854 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55ce18abb3f0 con 0x55ce18e42050 2024-03-23T23:05:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.186+0000 7fdd24c00640 1 -- 172.21.15.12:0/2238684854 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd2c0047a0 con 0x55ce18e42050 2024-03-23T23:05:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.186+0000 7fdd24c00640 1 -- 172.21.15.12:0/2238684854 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fdd2800f6e0 con 0x55ce18e42050 2024-03-23T23:05:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.186+0000 7fdd4458e740 10 client.26153 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.186+0000 7fdd4458e740 10 client.26153 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.186+0000 7fdd4458e740 20 client.26153 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:15.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.186+0000 7fdd4458e740 1 --2- 172.21.15.12:0/2238684854 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55ce18e57db0 0x55ce18e5a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.186+0000 7fdd4458e740 4 client.26153 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:05:15.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.186+0000 7fdd4458e740 1 -- 172.21.15.12:0/2238684854 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ce18db4790 con 0x55ce18e57db0 2024-03-23T23:05:15.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.186+0000 7fdd3aa00640 1 --2- 172.21.15.12:0/2238684854 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55ce18e57db0 0x55ce18e5a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.187+0000 7fdd3aa00640 1 --2- 172.21.15.12:0/2238684854 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55ce18e57db0 0x55ce18e5a250 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ab7642131418a3f server_cookie=841ce42ef92055a5 in_seq=0 out_seq=0 2024-03-23T23:05:15.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.187+0000 7fdd24c00640 10 client.26153 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:15.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.188+0000 7fdd24c00640 1 -- 172.21.15.12:0/2238684854 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ce18db4790 con 0x55ce18e57db0 2024-03-23T23:05:15.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.188+0000 7fdd24c00640 10 client.26153 handle_command_reply: tid=0 2024-03-23T23:05:15.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.188+0000 7fdd4458e740 1 client.26153 shutdown 2024-03-23T23:05:15.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.188+0000 7fdd4458e740 1 -- 172.21.15.12:0/2238684854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce18e42050 msgr2=0x55ce18e4c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:15.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.188+0000 7fdd4458e740 1 --2- 172.21.15.12:0/2238684854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce18e42050 0x55ce18e4c0b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fdd28014f50 tx=0x7fdd280046c0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.188+0000 7fdd4458e740 1 -- 172.21.15.12:0/2238684854 shutdown_connections 2024-03-23T23:05:15.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.188+0000 7fdd4458e740 1 --2- 172.21.15.12:0/2238684854 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55ce18e57db0 0x55ce18e5a250 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.188+0000 7fdd4458e740 1 --2- 172.21.15.12:0/2238684854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce18e47bb0 0x55ce18e535c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.188+0000 7fdd4458e740 1 --2- 172.21.15.12:0/2238684854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce18e42a40 0x55ce18e4c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.188+0000 7fdd4458e740 1 --2- 172.21.15.12:0/2238684854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce18e42050 0x55ce18e4c0b0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.188+0000 7fdd4458e740 1 -- 172.21.15.12:0/2238684854 >> 172.21.15.12:0/2238684854 conn(0x55ce18e0a200 msgr2=0x55ce18e3cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:15.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.188+0000 7fdd24c00640 0 client.26153 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:15.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.189+0000 7fdd4458e740 1 -- 172.21.15.12:0/2238684854 shutdown_connections 2024-03-23T23:05:15.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.189+0000 7fdd4458e740 1 -- 172.21.15.12:0/2238684854 wait complete. 2024-03-23T23:05:15.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.189+0000 7fdd4458e740 20 client.26153 trim_cache size 0 max 16384 2024-03-23T23:05:15.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.207+0000 7fdd4458e740 1 Processor -- start 2024-03-23T23:05:15.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.207+0000 7fdd4458e740 1 -- start start 2024-03-23T23:05:15.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.207+0000 7fdd4458e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce18e42050 0x55ce18e5eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.207+0000 7fdd4458e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce18e42a40 0x55ce18e53580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.207+0000 7fdd4458e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce18e47bb0 0x55ce18e5f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.207+0000 7fdd4458e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c1b12b0 con 0x55ce18e47bb0 2024-03-23T23:05:15.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.207+0000 7fdd4458e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c1aedb0 con 0x55ce18e42050 2024-03-23T23:05:15.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.207+0000 7fdd4458e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c1ace50 con 0x55ce18e42a40 2024-03-23T23:05:15.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.207+0000 7fdd3a000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce18e47bb0 0x55ce18e5f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.208+0000 7fdd3a000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce18e42050 0x55ce18e5eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.208+0000 7fdd3aa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce18e42a40 0x55ce18e53580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.208+0000 7fdd3a000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce18e42050 0x55ce18e5eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59368/0 (socket says 172.21.15.12:59368) 2024-03-23T23:05:15.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.208+0000 7fdd3a000640 1 -- 172.21.15.12:0/1811955869 learned_addr learned my addr 172.21.15.12:0/1811955869 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:15.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.208+0000 7fdd3a000640 1 -- 172.21.15.12:0/1811955869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce18e42a40 msgr2=0x55ce18e53580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:15.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.208+0000 7fdd3a000640 1 --2- 172.21.15.12:0/1811955869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce18e42a40 0x55ce18e53580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.208+0000 7fdd3a000640 1 -- 172.21.15.12:0/1811955869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce18e42050 msgr2=0x55ce18e5eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.208+0000 7fdd3a000640 1 --2- 172.21.15.12:0/1811955869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce18e42050 0x55ce18e5eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.208+0000 7fdd3a000640 1 -- 172.21.15.12:0/1811955869 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce18a7b0f0 con 0x55ce18e47bb0 2024-03-23T23:05:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.208+0000 7fdd3a000640 1 --2- 172.21.15.12:0/1811955869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce18e47bb0 0x55ce18e5f050 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x55ce18e54b20 tx=0x7fdd28005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8bcfc5a3bc65e69 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.209+0000 7fdd27400640 1 -- 172.21.15.12:0/1811955869 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd28031070 con 0x55ce18e47bb0 2024-03-23T23:05:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.209+0000 7fdd27400640 1 -- 172.21.15.12:0/1811955869 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd28014110 con 0x55ce18e47bb0 2024-03-23T23:05:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.209+0000 7fdd27400640 1 -- 172.21.15.12:0/1811955869 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd28004280 con 0x55ce18e47bb0 2024-03-23T23:05:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.209+0000 7fdd4458e740 1 -- 172.21.15.12:0/1811955869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce18e47bb0 msgr2=0x55ce18e5f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.209+0000 7fdd4458e740 1 --2- 172.21.15.12:0/1811955869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce18e47bb0 0x55ce18e5f050 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x55ce18e54b20 tx=0x7fdd28005f90 comp rx=0 tx=0).stop 2024-03-23T23:05:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.209+0000 7fdd4458e740 1 -- 172.21.15.12:0/1811955869 shutdown_connections 2024-03-23T23:05:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.209+0000 7fdd4458e740 1 --2- 172.21.15.12:0/1811955869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce18e47bb0 0x55ce18e5f050 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.209+0000 7fdd4458e740 1 --2- 172.21.15.12:0/1811955869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce18e42a40 0x55ce18e53580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.209+0000 7fdd4458e740 1 --2- 172.21.15.12:0/1811955869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce18e42050 0x55ce18e5eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.209+0000 7fdd4458e740 1 -- 172.21.15.12:0/1811955869 >> 172.21.15.12:0/1811955869 conn(0x55ce18e0a200 msgr2=0x55ce18e43d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.209+0000 7fdd4458e740 1 -- 172.21.15.12:0/1811955869 shutdown_connections 2024-03-23T23:05:15.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.209+0000 7fdd4458e740 1 -- 172.21.15.12:0/1811955869 wait complete. 2024-03-23T23:05:15.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.210+0000 7fdd4458e740 1 -- start start 2024-03-23T23:05:15.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.210+0000 7fdd4458e740 1 Processor -- start 2024-03-23T23:05:15.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.210+0000 7fdd4458e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce18e42050 0x55ce18e50df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.211+0000 7fdd4458e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce18e42a40 0x55ce18e51330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.211+0000 7fdd4458e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce18e47bb0 0x55ce18e52340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.211+0000 7fdd4458e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c1ad450 con 0x55ce18e47bb0 2024-03-23T23:05:15.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.211+0000 7fdd4458e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c1ae750 con 0x55ce18e42a40 2024-03-23T23:05:15.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.211+0000 7fdd4458e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c1ada90 con 0x55ce18e42050 2024-03-23T23:05:15.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.211+0000 7fdd4458e740 10 client.0 _ll_register_callbacks cb 0x7fdd28014e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:15.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.211+0000 7fdd3a000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce18e47bb0 0x55ce18e52340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.211+0000 7fdd3a000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce18e47bb0 0x55ce18e52340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40640/0 (socket says 172.21.15.12:40640) 2024-03-23T23:05:15.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.211+0000 7fdd3a000640 1 -- 172.21.15.12:0/3821354351 learned_addr learned my addr 172.21.15.12:0/3821354351 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:15.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.211+0000 7fdd3aa00640 1 --2- 172.21.15.12:0/3821354351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce18e42a40 0x55ce18e51330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.211+0000 7fdd3a000640 1 -- 172.21.15.12:0/3821354351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce18e42050 msgr2=0x55ce18e50df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:15.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.211+0000 7fdd3a000640 1 --2- 172.21.15.12:0/3821354351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce18e42050 0x55ce18e50df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.211+0000 7fdd3a000640 1 -- 172.21.15.12:0/3821354351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce18e42a40 msgr2=0x55ce18e51330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:15.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.211+0000 7fdd3a000640 1 --2- 172.21.15.12:0/3821354351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce18e42a40 0x55ce18e51330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.211+0000 7fdd3a000640 1 -- 172.21.15.12:0/3821354351 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ce18e52930 con 0x55ce18e47bb0 2024-03-23T23:05:15.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.211+0000 7fdd3aa00640 1 --2- 172.21.15.12:0/3821354351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce18e42a40 0x55ce18e51330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:05:15.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.212+0000 7fdd3a000640 1 --2- 172.21.15.12:0/3821354351 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce18e47bb0 0x55ce18e52340 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fdd2800fb70 tx=0x7fdd28009370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83a2e895ea1e0e48 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:15.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.212+0000 7fdd24c00640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:05:15.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.212+0000 7fdd24c00640 1 -- 172.21.15.12:0/3821354351 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd28031040 con 0x55ce18e47bb0 2024-03-23T23:05:15.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.212+0000 7fdd24c00640 1 -- 172.21.15.12:0/3821354351 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fdd2800fc40 con 0x55ce18e47bb0 2024-03-23T23:05:15.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.212+0000 7fdd4458e740 1 -- 172.21.15.12:0/3821354351 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdd28014110 con 0x55ce18e47bb0 2024-03-23T23:05:15.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.212+0000 7fdd24c00640 1 -- 172.21.15.12:0/3821354351 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd28014750 con 0x55ce18e47bb0 2024-03-23T23:05:15.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.212+0000 7fdd4458e740 10 client.16020 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:15.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.212+0000 7fdd4458e740 1 -- 172.21.15.12:0/3821354351 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55ce18eb3820 con 0x55ce18e47bb0 2024-03-23T23:05:15.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.213+0000 7fdd24c00640 1 -- 172.21.15.12:0/3821354351 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd28014110 con 0x55ce18e47bb0 2024-03-23T23:05:15.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.213+0000 7fdd24c00640 1 -- 172.21.15.12:0/3821354351 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fdd28011440 con 0x55ce18e47bb0 2024-03-23T23:05:15.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.213+0000 7fdd4458e740 10 client.16020 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:15.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.213+0000 7fdd4458e740 10 client.16020 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:15.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.213+0000 7fdd4458e740 20 client.16020 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:15.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.213+0000 7fdd4458e740 1 --2- 172.21.15.12:0/3821354351 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55ce18eb41b0 0x55ce18eb65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.213+0000 7fdd4458e740 4 client.16020 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:05:15.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.213+0000 7fdd4458e740 1 -- 172.21.15.12:0/3821354351 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ce18daeae0 con 0x55ce18eb41b0 2024-03-23T23:05:15.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.213+0000 7fdd3aa00640 1 --2- 172.21.15.12:0/3821354351 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55ce18eb41b0 0x55ce18eb65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.214+0000 7fdd3aa00640 1 --2- 172.21.15.12:0/3821354351 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55ce18eb41b0 0x55ce18eb65d0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b5c9ae84c0c2b9d server_cookie=d9bc1ba332881798 in_seq=0 out_seq=0 2024-03-23T23:05:15.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.214+0000 7fdd24c00640 10 client.16020 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:15.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.214+0000 7fdd24c00640 1 -- 172.21.15.12:0/3821354351 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55ce18daeae0 con 0x55ce18eb41b0 2024-03-23T23:05:15.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.214+0000 7fdd24c00640 10 client.16020 handle_command_reply: tid=0 2024-03-23T23:05:15.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.214+0000 7fdd4458e740 1 client.16020 shutdown 2024-03-23T23:05:15.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.214+0000 7fdd4458e740 1 -- 172.21.15.12:0/3821354351 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce18e47bb0 msgr2=0x55ce18e52340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:15.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.214+0000 7fdd4458e740 1 --2- 172.21.15.12:0/3821354351 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce18e47bb0 0x55ce18e52340 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fdd2800fb70 tx=0x7fdd28009370 comp rx=0 tx=0).stop 2024-03-23T23:05:15.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.214+0000 7fdd4458e740 1 -- 172.21.15.12:0/3821354351 shutdown_connections 2024-03-23T23:05:15.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.214+0000 7fdd4458e740 1 --2- 172.21.15.12:0/3821354351 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55ce18eb41b0 0x55ce18eb65d0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.214+0000 7fdd4458e740 1 --2- 172.21.15.12:0/3821354351 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce18e47bb0 0x55ce18e52340 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.214+0000 7fdd4458e740 1 --2- 172.21.15.12:0/3821354351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce18e42a40 0x55ce18e51330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.214+0000 7fdd4458e740 1 --2- 172.21.15.12:0/3821354351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce18e42050 0x55ce18e50df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.214+0000 7fdd24c00640 0 client.16020 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:15.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.214+0000 7fdd4458e740 1 -- 172.21.15.12:0/3821354351 >> 172.21.15.12:0/3821354351 conn(0x55ce18e0a200 msgr2=0x55ce18e40c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:15.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.214+0000 7fdd4458e740 1 -- 172.21.15.12:0/3821354351 shutdown_connections 2024-03-23T23:05:15.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.214+0000 7fdd4458e740 1 -- 172.21.15.12:0/3821354351 wait complete. 2024-03-23T23:05:15.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.214+0000 7fdd4458e740 20 client.16020 trim_cache size 0 max 16384 2024-03-23T23:05:15.218 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:05:15.218 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-03-23T23:05:15.218 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-03-23T23:05:15.219 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:05:15.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.215+0000 7fdd3be00640 1 -- 172.21.15.12:0/1256943667 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdd200a2880 msgr2=0x7fdd200a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:15.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.215+0000 7fdd3be00640 1 --2- 172.21.15.12:0/1256943667 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdd200a2880 0x7fdd200a4d40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fdd340eebc0 tx=0x7fdd3402c100 comp rx=0 tx=0).stop 2024-03-23T23:05:15.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.215+0000 7fdd3be00640 1 -- 172.21.15.12:0/1256943667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd3c077c40 msgr2=0x7fdd3c1a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:15.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.215+0000 7fdd3be00640 1 --2- 172.21.15.12:0/1256943667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd3c077c40 0x7fdd3c1a53b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fdd3402c640 tx=0x7fdd34035d90 comp rx=0 tx=0).stop 2024-03-23T23:05:15.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.215+0000 7fdd3be00640 1 -- 172.21.15.12:0/1256943667 shutdown_connections 2024-03-23T23:05:15.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.215+0000 7fdd3be00640 1 --2- 172.21.15.12:0/1256943667 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdd200a2880 0x7fdd200a4d40 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.215+0000 7fdd3be00640 1 --2- 172.21.15.12:0/1256943667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd3c077c40 0x7fdd3c1a53b0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.215+0000 7fdd3be00640 1 --2- 172.21.15.12:0/1256943667 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd3c0772a0 0x7fdd3c19e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.215+0000 7fdd3be00640 1 --2- 172.21.15.12:0/1256943667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd3c076060 0x7fdd3c19de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.215+0000 7fdd3be00640 1 -- 172.21.15.12:0/1256943667 >> 172.21.15.12:0/1256943667 conn(0x7fdd3c0fed10 msgr2=0x7fdd3c100290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:15.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.215+0000 7fdd3be00640 1 -- 172.21.15.12:0/1256943667 shutdown_connections 2024-03-23T23:05:15.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.215+0000 7fdd3be00640 1 -- 172.21.15.12:0/1256943667 wait complete. 2024-03-23T23:05:15.233 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T23:05:15.233 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:05:15.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:15 smithi012 ceph-mon[31695]: pgmap v493: 97 pgs: 97 active+clean; 1.9 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:05:15.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:15 smithi203 ceph-mon[36693]: pgmap v493: 97 pgs: 97 active+clean; 1.9 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:05:15.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.453+0000 7f5731200640 1 -- 172.21.15.12:0/2592652100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f572c1028b0 msgr2=0x7f572c102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:15.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.453+0000 7f5731200640 1 --2- 172.21.15.12:0/2592652100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f572c1028b0 0x7f572c102d30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5718009900 tx=0x7f571802bc30 comp rx=0 tx=0).stop 2024-03-23T23:05:15.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.453+0000 7f5731200640 1 -- 172.21.15.12:0/2592652100 shutdown_connections 2024-03-23T23:05:15.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.453+0000 7f5731200640 1 --2- 172.21.15.12:0/2592652100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f572c103270 0x7f572c109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.453+0000 7f5731200640 1 --2- 172.21.15.12:0/2592652100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f572c1028b0 0x7f572c102d30 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.453+0000 7f5731200640 1 --2- 172.21.15.12:0/2592652100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f572c1016c0 0x7f572c101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.453+0000 7f5731200640 1 -- 172.21.15.12:0/2592652100 >> 172.21.15.12:0/2592652100 conn(0x7f572c0fcc50 msgr2=0x7f572c0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:15.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.453+0000 7f5731200640 1 -- 172.21.15.12:0/2592652100 shutdown_connections 2024-03-23T23:05:15.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.453+0000 7f5731200640 1 -- 172.21.15.12:0/2592652100 wait complete. 2024-03-23T23:05:15.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.454+0000 7f5731200640 1 Processor -- start 2024-03-23T23:05:15.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.454+0000 7f5731200640 1 -- start start 2024-03-23T23:05:15.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.455+0000 7f5731200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f572c1016c0 0x7f572c199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.455+0000 7f5731200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f572c1028b0 0x7f572c19a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.455+0000 7f5731200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f572c103270 0x7f572c1a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.455+0000 7f5731200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572c10d390 con 0x7f572c103270 2024-03-23T23:05:15.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.455+0000 7f5731200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572c10d210 con 0x7f572c1028b0 2024-03-23T23:05:15.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.455+0000 7f5731200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572c10d510 con 0x7f572c1016c0 2024-03-23T23:05:15.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.455+0000 7f5729400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f572c1028b0 0x7f572c19a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.455+0000 7f5729e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f572c1016c0 0x7f572c199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.455+0000 7f572a800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f572c103270 0x7f572c1a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.455+0000 7f5729400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f572c1028b0 0x7f572c19a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59404/0 (socket says 172.21.15.12:59404) 2024-03-23T23:05:15.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.455+0000 7f5729400640 1 -- 172.21.15.12:0/621819747 learned_addr learned my addr 172.21.15.12:0/621819747 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:15.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.455+0000 7f5729400640 1 -- 172.21.15.12:0/621819747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f572c1016c0 msgr2=0x7f572c199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:15.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.455+0000 7f5729400640 1 --2- 172.21.15.12:0/621819747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f572c1016c0 0x7f572c199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.455+0000 7f5729400640 1 -- 172.21.15.12:0/621819747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f572c103270 msgr2=0x7f572c1a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:15.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.455+0000 7f5729400640 1 --2- 172.21.15.12:0/621819747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f572c103270 0x7f572c1a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.455+0000 7f5729400640 1 -- 172.21.15.12:0/621819747 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572c1a17e0 con 0x7f572c1028b0 2024-03-23T23:05:15.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.455+0000 7f5729e00640 1 --2- 172.21.15.12:0/621819747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f572c1016c0 0x7f572c199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:15.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.456+0000 7f5729400640 1 --2- 172.21.15.12:0/621819747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f572c1028b0 0x7f572c19a100 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5718005ab0 tx=0x7f571802c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:15.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.456+0000 7f5722a00640 1 -- 172.21.15.12:0/621819747 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5718043020 con 0x7f572c1028b0 2024-03-23T23:05:15.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.456+0000 7f5731200640 1 -- 172.21.15.12:0/621819747 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f572c1a1a70 con 0x7f572c1028b0 2024-03-23T23:05:15.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.456+0000 7f5722a00640 1 -- 172.21.15.12:0/621819747 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57180054e0 con 0x7f572c1028b0 2024-03-23T23:05:15.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.456+0000 7f5722a00640 1 -- 172.21.15.12:0/621819747 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5718003ce0 con 0x7f572c1028b0 2024-03-23T23:05:15.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.456+0000 7f5731200640 1 -- 172.21.15.12:0/621819747 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f572c1a2010 con 0x7f572c1028b0 2024-03-23T23:05:15.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.457+0000 7f5731200640 1 -- 172.21.15.12:0/621819747 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f572c1a22a0 con 0x7f572c1028b0 2024-03-23T23:05:15.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.459+0000 7f5722a00640 1 -- 172.21.15.12:0/621819747 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5718046020 con 0x7f572c1028b0 2024-03-23T23:05:15.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.461+0000 7f5722a00640 1 --2- 172.21.15.12:0/621819747 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f57100a2800 0x7f57100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.461+0000 7f5722a00640 1 -- 172.21.15.12:0/621819747 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f57180f0f40 con 0x7f572c1028b0 2024-03-23T23:05:15.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.461+0000 7f5729e00640 1 --2- 172.21.15.12:0/621819747 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f57100a2800 0x7f57100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.462+0000 7f5729e00640 1 --2- 172.21.15.12:0/621819747 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f57100a2800 0x7f57100a4cc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f572c102710 tx=0x7f571c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:15.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.468+0000 7f5722a00640 1 -- 172.21.15.12:0/621819747 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f571804d020 con 0x7f572c1028b0 2024-03-23T23:05:15.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:15 smithi173 ceph-mon[36116]: pgmap v493: 97 pgs: 97 active+clean; 1.9 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:05:15.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.714+0000 7f5731200640 1 -- 172.21.15.12:0/621819747 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f572c1a2470 con 0x7f572c1028b0 2024-03-23T23:05:15.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.715+0000 7f5722a00640 1 -- 172.21.15.12:0/621819747 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f57180a6e20 con 0x7f572c1028b0 2024-03-23T23:05:15.715 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:05:15.715 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:05:15.716 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:05:15.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.718+0000 7f5731200640 1 -- 172.21.15.12:0/621819747 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f57100a2800 msgr2=0x7f57100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:15.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.718+0000 7f5731200640 1 --2- 172.21.15.12:0/621819747 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f57100a2800 0x7f57100a4cc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f572c102710 tx=0x7f571c00b390 comp rx=0 tx=0).stop 2024-03-23T23:05:15.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.719+0000 7f5731200640 1 -- 172.21.15.12:0/621819747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f572c1028b0 msgr2=0x7f572c19a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:15.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.719+0000 7f5731200640 1 --2- 172.21.15.12:0/621819747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f572c1028b0 0x7f572c19a100 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5718005ab0 tx=0x7f571802c6c0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.719+0000 7f5731200640 1 -- 172.21.15.12:0/621819747 shutdown_connections 2024-03-23T23:05:15.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.719+0000 7f5731200640 1 --2- 172.21.15.12:0/621819747 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f57100a2800 0x7f57100a4cc0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.719+0000 7f5731200640 1 --2- 172.21.15.12:0/621819747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f572c103270 0x7f572c1a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.719+0000 7f5731200640 1 --2- 172.21.15.12:0/621819747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f572c1028b0 0x7f572c19a100 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.719+0000 7f5731200640 1 --2- 172.21.15.12:0/621819747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f572c1016c0 0x7f572c199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.719+0000 7f5731200640 1 -- 172.21.15.12:0/621819747 >> 172.21.15.12:0/621819747 conn(0x7f572c0fcc50 msgr2=0x7f572c0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:15.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.719+0000 7f5731200640 1 -- 172.21.15.12:0/621819747 shutdown_connections 2024-03-23T23:05:15.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.719+0000 7f5731200640 1 -- 172.21.15.12:0/621819747 wait complete. 2024-03-23T23:05:15.739 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:05:15.739 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-23T23:05:15.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.966+0000 7f7b43200640 1 -- 172.21.15.12:0/3555620517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b3c109870 msgr2=0x7f7b3c110100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:15.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.966+0000 7f7b43200640 1 --2- 172.21.15.12:0/3555620517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b3c109870 0x7f7b3c110100 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7b34009860 tx=0x7f7b3402bb70 comp rx=0 tx=0).stop 2024-03-23T23:05:15.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.966+0000 7f7b43200640 1 -- 172.21.15.12:0/3555620517 shutdown_connections 2024-03-23T23:05:15.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.966+0000 7f7b43200640 1 --2- 172.21.15.12:0/3555620517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b3c109870 0x7f7b3c110100 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.966+0000 7f7b43200640 1 --2- 172.21.15.12:0/3555620517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b3c108eb0 0x7f7b3c109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.966+0000 7f7b43200640 1 --2- 172.21.15.12:0/3555620517 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b3c107cc0 0x7f7b3c1080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.966+0000 7f7b43200640 1 -- 172.21.15.12:0/3555620517 >> 172.21.15.12:0/3555620517 conn(0x7f7b3c076c60 msgr2=0x7f7b3c0790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:15.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.966+0000 7f7b43200640 1 -- 172.21.15.12:0/3555620517 shutdown_connections 2024-03-23T23:05:15.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.966+0000 7f7b43200640 1 -- 172.21.15.12:0/3555620517 wait complete. 2024-03-23T23:05:15.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.967+0000 7f7b43200640 1 Processor -- start 2024-03-23T23:05:15.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.967+0000 7f7b43200640 1 -- start start 2024-03-23T23:05:15.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.967+0000 7f7b43200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b3c107cc0 0x7f7b3c19df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.967+0000 7f7b43200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b3c108eb0 0x7f7b3c19e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.967+0000 7f7b43200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b3c109870 0x7f7b3c1a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.967+0000 7f7b43200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c113920 con 0x7f7b3c107cc0 2024-03-23T23:05:15.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.967+0000 7f7b43200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c1137a0 con 0x7f7b3c108eb0 2024-03-23T23:05:15.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.967+0000 7f7b43200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c113aa0 con 0x7f7b3c109870 2024-03-23T23:05:15.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.968+0000 7f7b41200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b3c109870 0x7f7b3c1a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.968+0000 7f7b3be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b3c107cc0 0x7f7b3c19df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.968+0000 7f7b41200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b3c109870 0x7f7b3c1a54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58832/0 (socket says 172.21.15.12:58832) 2024-03-23T23:05:15.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.968+0000 7f7b41200640 1 -- 172.21.15.12:0/1859679968 learned_addr learned my addr 172.21.15.12:0/1859679968 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:15.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.968+0000 7f7b33400640 1 --2- 172.21.15.12:0/1859679968 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b3c108eb0 0x7f7b3c19e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.968+0000 7f7b41200640 1 -- 172.21.15.12:0/1859679968 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b3c108eb0 msgr2=0x7f7b3c19e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:15.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.968+0000 7f7b41200640 1 --2- 172.21.15.12:0/1859679968 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b3c108eb0 0x7f7b3c19e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.968+0000 7f7b41200640 1 -- 172.21.15.12:0/1859679968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b3c107cc0 msgr2=0x7f7b3c19df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:15.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.968+0000 7f7b41200640 1 --2- 172.21.15.12:0/1859679968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b3c107cc0 0x7f7b3c19df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.968+0000 7f7b41200640 1 -- 172.21.15.12:0/1859679968 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b3c1a5b60 con 0x7f7b3c109870 2024-03-23T23:05:15.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.968+0000 7f7b41200640 1 --2- 172.21.15.12:0/1859679968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b3c109870 0x7f7b3c1a54f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7b3402c180 tx=0x7f7b34032460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:15.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.969+0000 7f7b39600640 1 -- 172.21.15.12:0/1859679968 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b34041020 con 0x7f7b3c109870 2024-03-23T23:05:15.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.969+0000 7f7b39600640 1 -- 172.21.15.12:0/1859679968 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b3402c9f0 con 0x7f7b3c109870 2024-03-23T23:05:15.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.969+0000 7f7b39600640 1 -- 172.21.15.12:0/1859679968 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b34039040 con 0x7f7b3c109870 2024-03-23T23:05:15.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.969+0000 7f7b43200640 1 -- 172.21.15.12:0/1859679968 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b3c1a5df0 con 0x7f7b3c109870 2024-03-23T23:05:15.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.969+0000 7f7b43200640 1 -- 172.21.15.12:0/1859679968 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b3c1a6230 con 0x7f7b3c109870 2024-03-23T23:05:15.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.972+0000 7f7b39600640 1 -- 172.21.15.12:0/1859679968 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7b34044020 con 0x7f7b3c109870 2024-03-23T23:05:15.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.972+0000 7f7b46020740 1 Processor -- start 2024-03-23T23:05:15.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.972+0000 7f7b46020740 1 -- start start 2024-03-23T23:05:15.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.973+0000 7f7b46020740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56001b676810 0x56001b676c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.973+0000 7f7b46020740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56001b6779e0 0x56001b677e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.973+0000 7f7b46020740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b6783a0 0x56001b67ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.973+0000 7f7b46020740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56001b5db2c0 con 0x56001b676810 2024-03-23T23:05:15.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.973+0000 7f7b46020740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56001b583c90 con 0x56001b6783a0 2024-03-23T23:05:15.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.973+0000 7f7b46020740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x56001b5d7360 con 0x56001b6779e0 2024-03-23T23:05:15.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.973+0000 7f7b3be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56001b6779e0 0x56001b677e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.973+0000 7f7b3be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56001b6779e0 0x56001b677e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58848/0 (socket says 172.21.15.12:58848) 2024-03-23T23:05:15.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.973+0000 7f7b3be00640 1 -- 172.21.15.12:0/3283464980 learned_addr learned my addr 172.21.15.12:0/3283464980 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:15.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.973+0000 7f7b33400640 1 --2- 172.21.15.12:0/3283464980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56001b676810 0x56001b676c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.973+0000 7f7b33400640 1 --2- 172.21.15.12:0/3283464980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b6783a0 0x56001b67ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.973+0000 7f7b3be00640 1 -- 172.21.15.12:0/3283464980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b6783a0 msgr2=0x56001b67ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:15.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.973+0000 7f7b3be00640 1 --2- 172.21.15.12:0/3283464980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b6783a0 0x56001b67ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.973+0000 7f7b3be00640 1 -- 172.21.15.12:0/3283464980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56001b676810 msgr2=0x56001b676c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:15.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.973+0000 7f7b3be00640 1 --2- 172.21.15.12:0/3283464980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56001b676810 0x56001b676c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.973+0000 7f7b3be00640 1 -- 172.21.15.12:0/3283464980 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56001b5a8080 con 0x56001b6779e0 2024-03-23T23:05:15.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.973+0000 7f7b33400640 1 --2- 172.21.15.12:0/3283464980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b6783a0 0x56001b67ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:05:15.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.974+0000 7f7b3be00640 1 --2- 172.21.15.12:0/3283464980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56001b6779e0 0x56001b677e60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c009c80 tx=0x7f7b2c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a6861d9ec413088 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:15.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.974+0000 7f7b39600640 1 --2- 172.21.15.12:0/1859679968 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7b200a2880 0x7f7b200a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.974+0000 7f7b31600640 1 -- 172.21.15.12:0/3283464980 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c005220 con 0x56001b6779e0 2024-03-23T23:05:15.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.974+0000 7f7b31600640 1 -- 172.21.15.12:0/3283464980 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b2c009e30 con 0x56001b6779e0 2024-03-23T23:05:15.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.974+0000 7f7b31600640 1 -- 172.21.15.12:0/3283464980 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c019890 con 0x56001b6779e0 2024-03-23T23:05:15.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.974+0000 7f7b39600640 1 -- 172.21.15.12:0/1859679968 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7b340f0460 con 0x7f7b3c109870 2024-03-23T23:05:15.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.974+0000 7f7b33400640 1 --2- 172.21.15.12:0/1859679968 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7b200a2880 0x7f7b200a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.974+0000 7f7b46020740 1 -- 172.21.15.12:0/3283464980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56001b6779e0 msgr2=0x56001b677e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:15.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.974+0000 7f7b46020740 1 --2- 172.21.15.12:0/3283464980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56001b6779e0 0x56001b677e60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c009c80 tx=0x7f7b2c011810 comp rx=0 tx=0).stop 2024-03-23T23:05:15.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.975+0000 7f7b46020740 1 -- 172.21.15.12:0/3283464980 shutdown_connections 2024-03-23T23:05:15.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.975+0000 7f7b46020740 1 --2- 172.21.15.12:0/3283464980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b6783a0 0x56001b67ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.975+0000 7f7b46020740 1 --2- 172.21.15.12:0/3283464980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56001b6779e0 0x56001b677e60 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.975+0000 7f7b46020740 1 --2- 172.21.15.12:0/3283464980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56001b676810 0x56001b676c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.975+0000 7f7b46020740 1 -- 172.21.15.12:0/3283464980 >> 172.21.15.12:0/3283464980 conn(0x56001b671c00 msgr2=0x56001b672000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:15.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.975+0000 7f7b46020740 1 -- 172.21.15.12:0/3283464980 shutdown_connections 2024-03-23T23:05:15.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.975+0000 7f7b46020740 1 -- 172.21.15.12:0/3283464980 wait complete. 2024-03-23T23:05:15.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.975+0000 7f7b33400640 1 --2- 172.21.15.12:0/1859679968 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7b200a2880 0x7f7b200a4d40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x56001b6778e0 tx=0x7f7b24014040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:15.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.975+0000 7f7b46020740 1 -- start start 2024-03-23T23:05:15.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.975+0000 7f7b46020740 1 Processor -- start 2024-03-23T23:05:15.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.976+0000 7f7b46020740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b676810 0x56001b680a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.976+0000 7f7b46020740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56001b6779e0 0x56001b680f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.976+0000 7f7b46020740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56001b6783a0 0x56001b687fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.976+0000 7f7b46020740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56001b67f3b0 con 0x56001b6779e0 2024-03-23T23:05:15.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.976+0000 7f7b46020740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56001b5970d0 con 0x56001b676810 2024-03-23T23:05:15.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.976+0000 7f7b46020740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x56001b67f530 con 0x56001b6783a0 2024-03-23T23:05:15.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.976+0000 7f7b46020740 10 client.0 _ll_register_callbacks cb 0x56001b59be80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:15.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.976+0000 7f7b3be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b676810 0x56001b680a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.976+0000 7f7b3be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b676810 0x56001b680a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59436/0 (socket says 172.21.15.12:59436) 2024-03-23T23:05:15.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.976+0000 7f7b3be00640 1 -- 172.21.15.12:0/1851544014 learned_addr learned my addr 172.21.15.12:0/1851544014 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:15.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.976+0000 7f7b41200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56001b6783a0 0x56001b687fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.976+0000 7f7b33400640 1 --2- 172.21.15.12:0/1851544014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56001b6779e0 0x56001b680f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.977+0000 7f7b3be00640 1 -- 172.21.15.12:0/1851544014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56001b6783a0 msgr2=0x56001b687fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.977+0000 7f7b3be00640 1 --2- 172.21.15.12:0/1851544014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56001b6783a0 0x56001b687fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.977+0000 7f7b3be00640 1 -- 172.21.15.12:0/1851544014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56001b6779e0 msgr2=0x56001b680f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.977+0000 7f7b3be00640 1 --2- 172.21.15.12:0/1851544014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56001b6779e0 0x56001b680f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.977+0000 7f7b3be00640 1 -- 172.21.15.12:0/1851544014 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56001b2ef3f0 con 0x56001b676810 2024-03-23T23:05:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.977+0000 7f7b41200640 1 --2- 172.21.15.12:0/1851544014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56001b6783a0 0x56001b687fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.977+0000 7f7b3be00640 1 --2- 172.21.15.12:0/1851544014 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b676810 0x56001b680a30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c01b040 tx=0x7f7b2c019d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=286252e9fcda625d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.977+0000 7f7b2a000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:05:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.977+0000 7f7b2a000640 1 -- 172.21.15.12:0/1851544014 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c029070 con 0x56001b676810 2024-03-23T23:05:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.977+0000 7f7b2a000640 1 -- 172.21.15.12:0/1851544014 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7b2c023840 con 0x56001b676810 2024-03-23T23:05:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.977+0000 7f7b46020740 1 -- 172.21.15.12:0/1851544014 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7b2c009e30 con 0x56001b676810 2024-03-23T23:05:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.978+0000 7f7b2a000640 1 -- 172.21.15.12:0/1851544014 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b2c035050 con 0x56001b676810 2024-03-23T23:05:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.978+0000 7f7b46020740 10 client.26171 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:15.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.978+0000 7f7b46020740 1 -- 172.21.15.12:0/1851544014 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56001b3b97d0 con 0x56001b676810 2024-03-23T23:05:15.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.978+0000 7f7b2a000640 1 -- 172.21.15.12:0/1851544014 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b2c009e30 con 0x56001b676810 2024-03-23T23:05:15.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.978+0000 7f7b2a000640 1 -- 172.21.15.12:0/1851544014 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f7b2c022760 con 0x56001b676810 2024-03-23T23:05:15.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.978+0000 7f7b46020740 10 client.26171 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:15.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.978+0000 7f7b46020740 10 client.26171 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:05:15.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.978+0000 7f7b46020740 20 client.26171 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:15.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.978+0000 7f7b46020740 1 --2- 172.21.15.12:0/1851544014 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x56001b68c0f0 0x56001b68e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.978+0000 7f7b46020740 4 client.26171 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:05:15.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.978+0000 7f7b46020740 1 -- 172.21.15.12:0/1851544014 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56001b5e8800 con 0x56001b68c0f0 2024-03-23T23:05:15.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.979+0000 7f7b33400640 1 --2- 172.21.15.12:0/1851544014 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x56001b68c0f0 0x56001b68e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.979+0000 7f7b33400640 1 --2- 172.21.15.12:0/1851544014 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x56001b68c0f0 0x56001b68e570 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9d4d4c298653ad64 server_cookie=c3356b244347bd21 in_seq=0 out_seq=0 2024-03-23T23:05:15.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.979+0000 7f7b2a000640 10 client.26171 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:05:15.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.980+0000 7f7b2a000640 1 -- 172.21.15.12:0/1851544014 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56001b5e8800 con 0x56001b68c0f0 2024-03-23T23:05:15.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.980+0000 7f7b2a000640 10 client.26171 handle_command_reply: tid=0 2024-03-23T23:05:15.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.980+0000 7f7b46020740 1 client.26171 shutdown 2024-03-23T23:05:15.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.980+0000 7f7b46020740 1 -- 172.21.15.12:0/1851544014 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b676810 msgr2=0x56001b680a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:15.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.980+0000 7f7b46020740 1 --2- 172.21.15.12:0/1851544014 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b676810 0x56001b680a30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c01b040 tx=0x7f7b2c019d90 comp rx=0 tx=0).stop 2024-03-23T23:05:15.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.980+0000 7f7b46020740 1 -- 172.21.15.12:0/1851544014 shutdown_connections 2024-03-23T23:05:15.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.980+0000 7f7b46020740 1 --2- 172.21.15.12:0/1851544014 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x56001b68c0f0 0x56001b68e570 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.980+0000 7f7b46020740 1 --2- 172.21.15.12:0/1851544014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56001b6783a0 0x56001b687fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.980+0000 7f7b46020740 1 --2- 172.21.15.12:0/1851544014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56001b6779e0 0x56001b680f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.980+0000 7f7b2a000640 0 client.26171 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:05:15.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.980+0000 7f7b46020740 1 --2- 172.21.15.12:0/1851544014 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b676810 0x56001b680a30 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.980+0000 7f7b46020740 1 -- 172.21.15.12:0/1851544014 >> 172.21.15.12:0/1851544014 conn(0x56001b671c00 msgr2=0x56001b670eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:15.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.980+0000 7f7b46020740 1 -- 172.21.15.12:0/1851544014 shutdown_connections 2024-03-23T23:05:15.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.980+0000 7f7b46020740 1 -- 172.21.15.12:0/1851544014 wait complete. 2024-03-23T23:05:15.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.980+0000 7f7b46020740 20 client.26171 trim_cache size 0 max 16384 2024-03-23T23:05:15.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.994+0000 7f7b46020740 1 Processor -- start 2024-03-23T23:05:15.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.994+0000 7f7b46020740 1 -- start start 2024-03-23T23:05:15.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.994+0000 7f7b46020740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b676810 0x56001b680050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.994+0000 7f7b46020740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56001b6779e0 0x56001b687ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.995+0000 7f7b46020740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56001b6783a0 0x56001b680590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:15.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.995+0000 7f7b46020740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c1b1530 con 0x56001b6783a0 2024-03-23T23:05:15.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.995+0000 7f7b46020740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c1af030 con 0x56001b676810 2024-03-23T23:05:15.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.995+0000 7f7b46020740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c1ad0d0 con 0x56001b6779e0 2024-03-23T23:05:15.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.995+0000 7f7b33400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56001b6779e0 0x56001b687ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.995+0000 7f7b33400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56001b6779e0 0x56001b687ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58866/0 (socket says 172.21.15.12:58866) 2024-03-23T23:05:15.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.995+0000 7f7b3be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b676810 0x56001b680050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.995+0000 7f7b33400640 1 -- 172.21.15.12:0/1504745862 learned_addr learned my addr 172.21.15.12:0/1504745862 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:15.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.995+0000 7f7b41200640 1 --2- 172.21.15.12:0/1504745862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56001b6783a0 0x56001b680590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:15.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.995+0000 7f7b33400640 1 -- 172.21.15.12:0/1504745862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b676810 msgr2=0x56001b680050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:15.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.995+0000 7f7b33400640 1 --2- 172.21.15.12:0/1504745862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b676810 0x56001b680050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.995+0000 7f7b33400640 1 -- 172.21.15.12:0/1504745862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56001b6783a0 msgr2=0x56001b680590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:15.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.995+0000 7f7b33400640 1 --2- 172.21.15.12:0/1504745862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56001b6783a0 0x56001b680590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:15.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.995+0000 7f7b33400640 1 -- 172.21.15.12:0/1504745862 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56001b672670 con 0x56001b6779e0 2024-03-23T23:05:15.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.995+0000 7f7b3be00640 1 --2- 172.21.15.12:0/1504745862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b676810 0x56001b680050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:16.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.996+0000 7f7b33400640 1 --2- 172.21.15.12:0/1504745862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56001b6779e0 0x56001b687ed0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x56001b68ede0 tx=0x7f7b24004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f769915f89ebb406 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:16.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.996+0000 7f7b30c00640 1 -- 172.21.15.12:0/1504745862 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b24010c70 con 0x56001b6779e0 2024-03-23T23:05:16.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.996+0000 7f7b30c00640 1 -- 172.21.15.12:0/1504745862 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b24010e10 con 0x56001b6779e0 2024-03-23T23:05:16.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.996+0000 7f7b30c00640 1 -- 172.21.15.12:0/1504745862 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b240043f0 con 0x56001b6779e0 2024-03-23T23:05:16.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.996+0000 7f7b46020740 1 -- 172.21.15.12:0/1504745862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56001b6779e0 msgr2=0x56001b687ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.996+0000 7f7b46020740 1 --2- 172.21.15.12:0/1504745862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56001b6779e0 0x56001b687ed0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x56001b68ede0 tx=0x7f7b24004a40 comp rx=0 tx=0).stop 2024-03-23T23:05:16.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.996+0000 7f7b46020740 1 -- 172.21.15.12:0/1504745862 shutdown_connections 2024-03-23T23:05:16.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.996+0000 7f7b46020740 1 --2- 172.21.15.12:0/1504745862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56001b6783a0 0x56001b680590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.996+0000 7f7b46020740 1 --2- 172.21.15.12:0/1504745862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56001b6779e0 0x56001b687ed0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.996+0000 7f7b46020740 1 --2- 172.21.15.12:0/1504745862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b676810 0x56001b680050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.996+0000 7f7b46020740 1 -- 172.21.15.12:0/1504745862 >> 172.21.15.12:0/1504745862 conn(0x56001b671c00 msgr2=0x56001b670eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:16.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.996+0000 7f7b46020740 1 -- 172.21.15.12:0/1504745862 shutdown_connections 2024-03-23T23:05:16.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.997+0000 7f7b46020740 1 -- 172.21.15.12:0/1504745862 wait complete. 2024-03-23T23:05:16.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.997+0000 7f7b46020740 1 -- start start 2024-03-23T23:05:16.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.997+0000 7f7b46020740 1 Processor -- start 2024-03-23T23:05:16.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.998+0000 7f7b46020740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b676810 0x56001b683850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.998+0000 7f7b46020740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56001b6779e0 0x56001b683d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.998+0000 7f7b46020740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56001b6783a0 0x56001b686db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.998+0000 7f7b46020740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c1ad6d0 con 0x56001b6779e0 2024-03-23T23:05:16.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.998+0000 7f7b46020740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c1ae9d0 con 0x56001b676810 2024-03-23T23:05:16.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.998+0000 7f7b46020740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c1add10 con 0x56001b6783a0 2024-03-23T23:05:16.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.998+0000 7f7b46020740 10 client.0 _ll_register_callbacks cb 0x7f7b2c009ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:16.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.998+0000 7f7b3be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b676810 0x56001b683850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.998+0000 7f7b3be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b676810 0x56001b683850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59450/0 (socket says 172.21.15.12:59450) 2024-03-23T23:05:16.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.998+0000 7f7b3be00640 1 -- 172.21.15.12:0/1216537594 learned_addr learned my addr 172.21.15.12:0/1216537594 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.998+0000 7f7b33400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56001b6779e0 0x56001b683d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.998+0000 7f7b41200640 1 --2- 172.21.15.12:0/1216537594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56001b6783a0 0x56001b686db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.998+0000 7f7b3be00640 1 -- 172.21.15.12:0/1216537594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56001b6783a0 msgr2=0x56001b686db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.998+0000 7f7b3be00640 1 --2- 172.21.15.12:0/1216537594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56001b6783a0 0x56001b686db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.998+0000 7f7b3be00640 1 -- 172.21.15.12:0/1216537594 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56001b6779e0 msgr2=0x56001b683d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.998+0000 7f7b3be00640 1 --2- 172.21.15.12:0/1216537594 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56001b6779e0 0x56001b683d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.998+0000 7f7b3be00640 1 -- 172.21.15.12:0/1216537594 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56001b6873a0 con 0x56001b676810 2024-03-23T23:05:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.998+0000 7f7b3be00640 1 --2- 172.21.15.12:0/1216537594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b676810 0x56001b683850 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c00f3c0 tx=0x7f7b2c005970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2174a1c8da810312 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.998+0000 7f7b2a000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:05:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.999+0000 7f7b2a000640 1 -- 172.21.15.12:0/1216537594 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c03e060 con 0x56001b676810 2024-03-23T23:05:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.999+0000 7f7b2a000640 1 -- 172.21.15.12:0/1216537594 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7b2c022390 con 0x56001b676810 2024-03-23T23:05:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.999+0000 7f7b46020740 1 -- 172.21.15.12:0/1216537594 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7b24010e10 con 0x56001b676810 2024-03-23T23:05:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.999+0000 7f7b2a000640 1 -- 172.21.15.12:0/1216537594 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b2c029030 con 0x56001b676810 2024-03-23T23:05:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.999+0000 7f7b46020740 10 client.26177 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:16.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.999+0000 7f7b46020740 1 -- 172.21.15.12:0/1216537594 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56001b6e7cc0 con 0x56001b676810 2024-03-23T23:05:16.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.999+0000 7f7b2a000640 1 -- 172.21.15.12:0/1216537594 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b24010e10 con 0x56001b676810 2024-03-23T23:05:16.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.999+0000 7f7b2a000640 1 -- 172.21.15.12:0/1216537594 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f7b2c0053a0 con 0x56001b676810 2024-03-23T23:05:16.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.999+0000 7f7b46020740 10 client.26177 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:16.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.999+0000 7f7b46020740 10 client.26177 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:05:16.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.999+0000 7f7b46020740 20 client.26177 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:16.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.999+0000 7f7b46020740 1 --2- 172.21.15.12:0/1216537594 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x56001b6e8560 0x56001b6ea980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.999+0000 7f7b46020740 4 client.26177 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:05:16.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:15.999+0000 7f7b46020740 1 -- 172.21.15.12:0/1216537594 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56001b5e2b50 con 0x56001b6e8560 2024-03-23T23:05:16.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.000+0000 7f7b33400640 1 --2- 172.21.15.12:0/1216537594 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x56001b6e8560 0x56001b6ea980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.000+0000 7f7b33400640 1 --2- 172.21.15.12:0/1216537594 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x56001b6e8560 0x56001b6ea980 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1ad3f08ab55b065a server_cookie=aa169015b643625f in_seq=0 out_seq=0 2024-03-23T23:05:16.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.000+0000 7f7b2a000640 10 client.26177 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:05:16.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.000+0000 7f7b2a000640 1 -- 172.21.15.12:0/1216537594 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56001b5e2b50 con 0x56001b6e8560 2024-03-23T23:05:16.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.000+0000 7f7b2a000640 10 client.26177 handle_command_reply: tid=0 2024-03-23T23:05:16.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.000+0000 7f7b46020740 1 client.26177 shutdown 2024-03-23T23:05:16.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.001+0000 7f7b46020740 1 -- 172.21.15.12:0/1216537594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b676810 msgr2=0x56001b683850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.001+0000 7f7b46020740 1 --2- 172.21.15.12:0/1216537594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b676810 0x56001b683850 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c00f3c0 tx=0x7f7b2c005970 comp rx=0 tx=0).stop 2024-03-23T23:05:16.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.001+0000 7f7b46020740 1 -- 172.21.15.12:0/1216537594 shutdown_connections 2024-03-23T23:05:16.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.001+0000 7f7b46020740 1 --2- 172.21.15.12:0/1216537594 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x56001b6e8560 0x56001b6ea980 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.001+0000 7f7b46020740 1 --2- 172.21.15.12:0/1216537594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56001b6783a0 0x56001b686db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.001+0000 7f7b46020740 1 --2- 172.21.15.12:0/1216537594 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56001b6779e0 0x56001b683d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.001+0000 7f7b46020740 1 --2- 172.21.15.12:0/1216537594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56001b676810 0x56001b683850 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.001+0000 7f7b46020740 1 -- 172.21.15.12:0/1216537594 >> 172.21.15.12:0/1216537594 conn(0x56001b671c00 msgr2=0x56001b67ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:16.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.001+0000 7f7b2a000640 0 client.26177 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:05:16.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.001+0000 7f7b46020740 1 -- 172.21.15.12:0/1216537594 shutdown_connections 2024-03-23T23:05:16.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.001+0000 7f7b46020740 1 -- 172.21.15.12:0/1216537594 wait complete. 2024-03-23T23:05:16.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.001+0000 7f7b46020740 20 client.26177 trim_cache size 0 max 16384 2024-03-23T23:05:16.006 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:05:16.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.002+0000 7f7b43200640 1 -- 172.21.15.12:0/1859679968 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7b200a2880 msgr2=0x7f7b200a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.002+0000 7f7b43200640 1 --2- 172.21.15.12:0/1859679968 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7b200a2880 0x7f7b200a4d40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x56001b6778e0 tx=0x7f7b24014040 comp rx=0 tx=0).stop 2024-03-23T23:05:16.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.002+0000 7f7b43200640 1 -- 172.21.15.12:0/1859679968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b3c109870 msgr2=0x7f7b3c1a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.002+0000 7f7b43200640 1 --2- 172.21.15.12:0/1859679968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b3c109870 0x7f7b3c1a54f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7b3402c180 tx=0x7f7b34032460 comp rx=0 tx=0).stop 2024-03-23T23:05:16.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.002+0000 7f7b43200640 1 -- 172.21.15.12:0/1859679968 shutdown_connections 2024-03-23T23:05:16.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.002+0000 7f7b43200640 1 --2- 172.21.15.12:0/1859679968 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7b200a2880 0x7f7b200a4d40 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.002+0000 7f7b43200640 1 --2- 172.21.15.12:0/1859679968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b3c109870 0x7f7b3c1a54f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.002+0000 7f7b43200640 1 --2- 172.21.15.12:0/1859679968 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b3c108eb0 0x7f7b3c19e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.002+0000 7f7b43200640 1 --2- 172.21.15.12:0/1859679968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b3c107cc0 0x7f7b3c19df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.002+0000 7f7b43200640 1 -- 172.21.15.12:0/1859679968 >> 172.21.15.12:0/1859679968 conn(0x7f7b3c076c60 msgr2=0x7f7b3c10c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:16.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.002+0000 7f7b43200640 1 -- 172.21.15.12:0/1859679968 shutdown_connections 2024-03-23T23:05:16.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.002+0000 7f7b43200640 1 -- 172.21.15.12:0/1859679968 wait complete. 2024-03-23T23:05:16.019 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:05:16.019 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-23T23:05:16.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.239+0000 7fa1fe800640 1 -- 172.21.15.12:0/614781168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1f81049b0 msgr2=0x7fa1f8104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.239+0000 7fa1fe800640 1 --2- 172.21.15.12:0/614781168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1f81049b0 0x7fa1f8104e30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009900 tx=0x7fa1e402bc30 comp rx=0 tx=0).stop 2024-03-23T23:05:16.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.239+0000 7fa1fe800640 1 -- 172.21.15.12:0/614781168 shutdown_connections 2024-03-23T23:05:16.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.239+0000 7fa1fe800640 1 --2- 172.21.15.12:0/614781168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1f8105370 0x7fa1f810bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.239+0000 7fa1fe800640 1 --2- 172.21.15.12:0/614781168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1f81049b0 0x7fa1f8104e30 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.239+0000 7fa1fe800640 1 --2- 172.21.15.12:0/614781168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa1f81037c0 0x7fa1f8103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.239+0000 7fa1fe800640 1 -- 172.21.15.12:0/614781168 >> 172.21.15.12:0/614781168 conn(0x7fa1f80fed50 msgr2=0x7fa1f8101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:16.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.239+0000 7fa1fe800640 1 -- 172.21.15.12:0/614781168 shutdown_connections 2024-03-23T23:05:16.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.239+0000 7fa1fe800640 1 -- 172.21.15.12:0/614781168 wait complete. 2024-03-23T23:05:16.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.240+0000 7fa1fe800640 1 Processor -- start 2024-03-23T23:05:16.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.240+0000 7fa1fe800640 1 -- start start 2024-03-23T23:05:16.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.240+0000 7fa1fe800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1f81037c0 0x7fa1f8199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.240+0000 7fa1fe800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa1f81049b0 0x7fa1f819a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.241+0000 7fa1fe800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1f8105370 0x7fa1f81a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.241+0000 7fa1fe800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f810f490 con 0x7fa1f8105370 2024-03-23T23:05:16.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.241+0000 7fa1fe800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f810f310 con 0x7fa1f81049b0 2024-03-23T23:05:16.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.241+0000 7fa1fe800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f810f610 con 0x7fa1f81037c0 2024-03-23T23:05:16.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.241+0000 7fa1fca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa1f81049b0 0x7fa1f819a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.241+0000 7fa1fca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa1f81049b0 0x7fa1f819a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59478/0 (socket says 172.21.15.12:59478) 2024-03-23T23:05:16.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.241+0000 7fa1fca00640 1 -- 172.21.15.12:0/3214521312 learned_addr learned my addr 172.21.15.12:0/3214521312 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:16.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.241+0000 7fa1fd400640 1 --2- 172.21.15.12:0/3214521312 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1f81037c0 0x7fa1f8199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.241+0000 7fa1fde00640 1 --2- 172.21.15.12:0/3214521312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1f8105370 0x7fa1f81a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.241+0000 7fa1fca00640 1 -- 172.21.15.12:0/3214521312 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1f81037c0 msgr2=0x7fa1f8199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.241+0000 7fa1fca00640 1 --2- 172.21.15.12:0/3214521312 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1f81037c0 0x7fa1f8199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.241+0000 7fa1fca00640 1 -- 172.21.15.12:0/3214521312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1f8105370 msgr2=0x7fa1f81a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.241+0000 7fa1fca00640 1 --2- 172.21.15.12:0/3214521312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1f8105370 0x7fa1f81a1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.241+0000 7fa1fca00640 1 -- 172.21.15.12:0/3214521312 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f81a16f0 con 0x7fa1f81049b0 2024-03-23T23:05:16.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.241+0000 7fa1fd400640 1 --2- 172.21.15.12:0/3214521312 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1f81037c0 0x7fa1f8199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:16.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.242+0000 7fa1fca00640 1 --2- 172.21.15.12:0/3214521312 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa1f81049b0 0x7fa1f819a010 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4000c00 tx=0x7fa1e402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:16.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.242+0000 7fa1f6000640 1 -- 172.21.15.12:0/3214521312 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4043020 con 0x7fa1f81049b0 2024-03-23T23:05:16.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.242+0000 7fa1f6000640 1 -- 172.21.15.12:0/3214521312 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1e40054e0 con 0x7fa1f81049b0 2024-03-23T23:05:16.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.242+0000 7fa1f6000640 1 -- 172.21.15.12:0/3214521312 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4003c40 con 0x7fa1f81049b0 2024-03-23T23:05:16.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.242+0000 7fa1fe800640 1 -- 172.21.15.12:0/3214521312 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f81a1980 con 0x7fa1f81049b0 2024-03-23T23:05:16.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.242+0000 7fa1fe800640 1 -- 172.21.15.12:0/3214521312 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f81a1e80 con 0x7fa1f81049b0 2024-03-23T23:05:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.245+0000 7fa1f6000640 1 -- 172.21.15.12:0/3214521312 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa1e4046020 con 0x7fa1f81049b0 2024-03-23T23:05:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.246+0000 7fa202a27740 1 Processor -- start 2024-03-23T23:05:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.246+0000 7fa202a27740 1 -- start start 2024-03-23T23:05:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.246+0000 7fa202a27740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5648781019f0 0x564878101e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.246+0000 7fa202a27740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564878102bc0 0x564878103040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.246+0000 7fa202a27740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564878103580 0x564878109e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.246+0000 7fa202a27740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648780662c0 con 0x564878102bc0 2024-03-23T23:05:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.246+0000 7fa202a27740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56487800ec90 con 0x5648781019f0 2024-03-23T23:05:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.246+0000 7fa202a27740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x564878062360 con 0x564878103580 2024-03-23T23:05:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.246+0000 7fa1fde00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5648781019f0 0x564878101e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.246+0000 7fa1fca00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564878103580 0x564878109e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.246+0000 7fa1fde00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5648781019f0 0x564878101e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59488/0 (socket says 172.21.15.12:59488) 2024-03-23T23:05:16.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.246+0000 7fa1fde00640 1 -- 172.21.15.12:0/1084660982 learned_addr learned my addr 172.21.15.12:0/1084660982 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:16.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.247+0000 7fa1fd400640 1 --2- 172.21.15.12:0/1084660982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564878102bc0 0x564878103040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.247+0000 7fa1fde00640 1 -- 172.21.15.12:0/1084660982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564878103580 msgr2=0x564878109e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:16.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.247+0000 7fa1fde00640 1 --2- 172.21.15.12:0/1084660982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564878103580 0x564878109e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.247+0000 7fa1fde00640 1 -- 172.21.15.12:0/1084660982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564878102bc0 msgr2=0x564878103040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:16.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.247+0000 7fa1fde00640 1 --2- 172.21.15.12:0/1084660982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564878102bc0 0x564878103040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.247+0000 7fa1fde00640 1 -- 172.21.15.12:0/1084660982 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564878033080 con 0x5648781019f0 2024-03-23T23:05:16.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.247+0000 7fa1fd400640 1 --2- 172.21.15.12:0/1084660982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564878102bc0 0x564878103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:05:16.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.247+0000 7fa1fca00640 1 --2- 172.21.15.12:0/1084660982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564878103580 0x564878109e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:16.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.247+0000 7fa1f6000640 1 --2- 172.21.15.12:0/3214521312 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa1e00a2950 0x7fa1e00a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.247+0000 7fa1fde00640 1 --2- 172.21.15.12:0/1084660982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5648781019f0 0x564878101e10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa1f0009fd0 tx=0x7fa1f00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca95316d676eb271 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.247+0000 7fa1ea600640 1 -- 172.21.15.12:0/1084660982 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0004d20 con 0x5648781019f0 2024-03-23T23:05:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.247+0000 7fa1f6000640 1 -- 172.21.15.12:0/3214521312 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fa1e40f09c0 con 0x7fa1f81049b0 2024-03-23T23:05:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.248+0000 7fa1ea600640 1 -- 172.21.15.12:0/1084660982 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1f0004050 con 0x5648781019f0 2024-03-23T23:05:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.248+0000 7fa1ea600640 1 -- 172.21.15.12:0/1084660982 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0005260 con 0x5648781019f0 2024-03-23T23:05:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.248+0000 7fa1fd400640 1 --2- 172.21.15.12:0/3214521312 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa1e00a2950 0x7fa1e00a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.248+0000 7fa202a27740 1 -- 172.21.15.12:0/1084660982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5648781019f0 msgr2=0x564878101e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.248+0000 7fa202a27740 1 --2- 172.21.15.12:0/1084660982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5648781019f0 0x564878101e10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa1f0009fd0 tx=0x7fa1f00118d0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.249+0000 7fa1fd400640 1 --2- 172.21.15.12:0/3214521312 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa1e00a2950 0x7fa1e00a4e10 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8104810 tx=0x7fa1ec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.249+0000 7fa202a27740 1 -- 172.21.15.12:0/1084660982 shutdown_connections 2024-03-23T23:05:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.249+0000 7fa202a27740 1 --2- 172.21.15.12:0/1084660982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564878103580 0x564878109e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.249+0000 7fa202a27740 1 --2- 172.21.15.12:0/1084660982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564878102bc0 0x564878103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.249+0000 7fa202a27740 1 --2- 172.21.15.12:0/1084660982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5648781019f0 0x564878101e10 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.249+0000 7fa202a27740 1 -- 172.21.15.12:0/1084660982 >> 172.21.15.12:0/1084660982 conn(0x5648780fcde0 msgr2=0x5648780fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:16.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.249+0000 7fa202a27740 1 -- 172.21.15.12:0/1084660982 shutdown_connections 2024-03-23T23:05:16.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.249+0000 7fa202a27740 1 -- 172.21.15.12:0/1084660982 wait complete. 2024-03-23T23:05:16.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.249+0000 7fa202a27740 1 -- start start 2024-03-23T23:05:16.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.249+0000 7fa202a27740 1 Processor -- start 2024-03-23T23:05:16.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.250+0000 7fa202a27740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648781019f0 0x56487810bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.250+0000 7fa202a27740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564878102bc0 0x56487810c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.250+0000 7fa202a27740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564878103580 0x564878113240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.250+0000 7fa202a27740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56487810a590 con 0x564878102bc0 2024-03-23T23:05:16.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.250+0000 7fa202a27740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648780220d0 con 0x564878103580 2024-03-23T23:05:16.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.250+0000 7fa202a27740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x56487810a710 con 0x5648781019f0 2024-03-23T23:05:16.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.250+0000 7fa202a27740 10 client.0 _ll_register_callbacks cb 0x564878026e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:16.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.250+0000 7fa1fca00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564878102bc0 0x56487810c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.250+0000 7fa1fd400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648781019f0 0x56487810bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.250+0000 7fa1fd400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648781019f0 0x56487810bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58908/0 (socket says 172.21.15.12:58908) 2024-03-23T23:05:16.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.250+0000 7fa1fd400640 1 -- 172.21.15.12:0/3193853398 learned_addr learned my addr 172.21.15.12:0/3193853398 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:16.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.250+0000 7fa1fde00640 1 --2- 172.21.15.12:0/3193853398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564878103580 0x564878113240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.251+0000 7fa1fd400640 1 -- 172.21.15.12:0/3193853398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564878103580 msgr2=0x564878113240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:16.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.251+0000 7fa1fd400640 1 --2- 172.21.15.12:0/3193853398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564878103580 0x564878113240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.251+0000 7fa1fd400640 1 -- 172.21.15.12:0/3193853398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564878102bc0 msgr2=0x56487810c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:16.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.251+0000 7fa1fd400640 1 --2- 172.21.15.12:0/3193853398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564878102bc0 0x56487810c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.251+0000 7fa1fd400640 1 -- 172.21.15.12:0/3193853398 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564877d7a3f0 con 0x5648781019f0 2024-03-23T23:05:16.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.251+0000 7fa1fca00640 1 --2- 172.21.15.12:0/3193853398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564878102bc0 0x56487810c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:16.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.251+0000 7fa1fd400640 1 --2- 172.21.15.12:0/3193853398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648781019f0 0x56487810bcc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec00fb30 tx=0x7fa1ec004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6beb0480753ec596 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:16.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.251+0000 7fa1dea00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:05:16.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.251+0000 7fa1dea00640 1 -- 172.21.15.12:0/3193853398 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec02e060 con 0x5648781019f0 2024-03-23T23:05:16.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.251+0000 7fa1dea00640 1 -- 172.21.15.12:0/3193853398 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fa1ec018030 con 0x5648781019f0 2024-03-23T23:05:16.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.251+0000 7fa202a27740 1 -- 172.21.15.12:0/3193853398 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa1f0004050 con 0x5648781019f0 2024-03-23T23:05:16.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.252+0000 7fa1dea00640 1 -- 172.21.15.12:0/3193853398 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1ec01e5a0 con 0x5648781019f0 2024-03-23T23:05:16.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.252+0000 7fa202a27740 10 client.26158 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:16.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.252+0000 7fa202a27740 1 -- 172.21.15.12:0/3193853398 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x564877e447d0 con 0x5648781019f0 2024-03-23T23:05:16.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.252+0000 7fa1dea00640 1 -- 172.21.15.12:0/3193853398 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1f0004050 con 0x5648781019f0 2024-03-23T23:05:16.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.252+0000 7fa1dea00640 1 -- 172.21.15.12:0/3193853398 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fa1ec034820 con 0x5648781019f0 2024-03-23T23:05:16.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.252+0000 7fa202a27740 10 client.26158 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:16.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.252+0000 7fa202a27740 10 client.26158 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:05:16.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.252+0000 7fa202a27740 20 client.26158 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:16.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.252+0000 7fa202a27740 1 --2- 172.21.15.12:0/3193853398 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5648781177d0 0x564878119c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.252+0000 7fa202a27740 4 client.26158 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:05:16.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.252+0000 7fa202a27740 1 -- 172.21.15.12:0/3193853398 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564878073800 con 0x5648781177d0 2024-03-23T23:05:16.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.253+0000 7fa1fca00640 1 --2- 172.21.15.12:0/3193853398 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5648781177d0 0x564878119c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.253+0000 7fa1fca00640 1 --2- 172.21.15.12:0/3193853398 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5648781177d0 0x564878119c50 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4e4a21d490bedbd4 server_cookie=4db7ee9b96472a1 in_seq=0 out_seq=0 2024-03-23T23:05:16.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.253+0000 7fa1dea00640 10 client.26158 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:05:16.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.254+0000 7fa1dea00640 1 -- 172.21.15.12:0/3193853398 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564878073800 con 0x5648781177d0 2024-03-23T23:05:16.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.254+0000 7fa1dea00640 10 client.26158 handle_command_reply: tid=0 2024-03-23T23:05:16.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.254+0000 7fa202a27740 1 client.26158 shutdown 2024-03-23T23:05:16.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.254+0000 7fa202a27740 1 -- 172.21.15.12:0/3193853398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648781019f0 msgr2=0x56487810bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.254+0000 7fa202a27740 1 --2- 172.21.15.12:0/3193853398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648781019f0 0x56487810bcc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec00fb30 tx=0x7fa1ec004730 comp rx=0 tx=0).stop 2024-03-23T23:05:16.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.254+0000 7fa202a27740 1 -- 172.21.15.12:0/3193853398 shutdown_connections 2024-03-23T23:05:16.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.254+0000 7fa202a27740 1 --2- 172.21.15.12:0/3193853398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564878103580 0x564878113240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.254+0000 7fa202a27740 1 --2- 172.21.15.12:0/3193853398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564878102bc0 0x56487810c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.254+0000 7fa202a27740 1 --2- 172.21.15.12:0/3193853398 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5648781177d0 0x564878119c50 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.254+0000 7fa202a27740 1 --2- 172.21.15.12:0/3193853398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648781019f0 0x56487810bcc0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.254+0000 7fa1dea00640 0 client.26158 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:05:16.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.254+0000 7fa202a27740 1 -- 172.21.15.12:0/3193853398 >> 172.21.15.12:0/3193853398 conn(0x5648780fcde0 msgr2=0x5648780fc160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:16.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.254+0000 7fa202a27740 1 -- 172.21.15.12:0/3193853398 shutdown_connections 2024-03-23T23:05:16.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.254+0000 7fa202a27740 1 -- 172.21.15.12:0/3193853398 wait complete. 2024-03-23T23:05:16.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.254+0000 7fa202a27740 20 client.26158 trim_cache size 0 max 16384 2024-03-23T23:05:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.269+0000 7fa202a27740 1 Processor -- start 2024-03-23T23:05:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.269+0000 7fa202a27740 1 -- start start 2024-03-23T23:05:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.269+0000 7fa202a27740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648781019f0 0x5648780fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.269+0000 7fa202a27740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564878102bc0 0x564878113200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.269+0000 7fa202a27740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564878103580 0x564878105df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.269+0000 7fa202a27740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f81acf80 con 0x5648781019f0 2024-03-23T23:05:16.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.269+0000 7fa202a27740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f81aaa80 con 0x564878103580 2024-03-23T23:05:16.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.269+0000 7fa202a27740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f81a8b20 con 0x564878102bc0 2024-03-23T23:05:16.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.270+0000 7fa1fca00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564878102bc0 0x564878113200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.270+0000 7fa1fde00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564878103580 0x564878105df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.270+0000 7fa1fde00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564878103580 0x564878105df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59510/0 (socket says 172.21.15.12:59510) 2024-03-23T23:05:16.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.270+0000 7fa1fde00640 1 -- 172.21.15.12:0/1133513062 learned_addr learned my addr 172.21.15.12:0/1133513062 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:16.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.270+0000 7fa1fd400640 1 --2- 172.21.15.12:0/1133513062 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648781019f0 0x5648780fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.270+0000 7fa1fca00640 1 -- 172.21.15.12:0/1133513062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564878103580 msgr2=0x564878105df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:16.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.270+0000 7fa1fca00640 1 --2- 172.21.15.12:0/1133513062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564878103580 0x564878105df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.270+0000 7fa1fca00640 1 -- 172.21.15.12:0/1133513062 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648781019f0 msgr2=0x5648780fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:16.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.270+0000 7fa1fca00640 1 --2- 172.21.15.12:0/1133513062 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648781019f0 0x5648780fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.270+0000 7fa1fca00640 1 -- 172.21.15.12:0/1133513062 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5648780fcb50 con 0x564878102bc0 2024-03-23T23:05:16.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.270+0000 7fa1fca00640 1 --2- 172.21.15.12:0/1133513062 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564878102bc0 0x564878113200 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa1e40ee520 tx=0x7fa1e40f0040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=781bb85f675ded8d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:16.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.271+0000 7fa1e9c00640 1 -- 172.21.15.12:0/1133513062 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e40bd140 con 0x564878102bc0 2024-03-23T23:05:16.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.271+0000 7fa1e9c00640 1 -- 172.21.15.12:0/1133513062 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1e4004250 con 0x564878102bc0 2024-03-23T23:05:16.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.271+0000 7fa1e9c00640 1 -- 172.21.15.12:0/1133513062 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e40bccd0 con 0x564878102bc0 2024-03-23T23:05:16.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.271+0000 7fa202a27740 1 -- 172.21.15.12:0/1133513062 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564878102bc0 msgr2=0x564878113200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.271+0000 7fa202a27740 1 --2- 172.21.15.12:0/1133513062 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564878102bc0 0x564878113200 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa1e40ee520 tx=0x7fa1e40f0040 comp rx=0 tx=0).stop 2024-03-23T23:05:16.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.272+0000 7fa202a27740 1 -- 172.21.15.12:0/1133513062 shutdown_connections 2024-03-23T23:05:16.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.272+0000 7fa202a27740 1 --2- 172.21.15.12:0/1133513062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564878103580 0x564878105df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.272+0000 7fa202a27740 1 --2- 172.21.15.12:0/1133513062 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564878102bc0 0x564878113200 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.272+0000 7fa202a27740 1 --2- 172.21.15.12:0/1133513062 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648781019f0 0x5648780fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.272+0000 7fa202a27740 1 -- 172.21.15.12:0/1133513062 >> 172.21.15.12:0/1133513062 conn(0x5648780fcde0 msgr2=0x5648781008d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:16.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.272+0000 7fa202a27740 1 -- 172.21.15.12:0/1133513062 shutdown_connections 2024-03-23T23:05:16.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.272+0000 7fa202a27740 1 -- 172.21.15.12:0/1133513062 wait complete. 2024-03-23T23:05:16.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.272+0000 7fa202a27740 1 -- start start 2024-03-23T23:05:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.273+0000 7fa202a27740 1 Processor -- start 2024-03-23T23:05:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.273+0000 7fa202a27740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648781019f0 0x564878112c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.273+0000 7fa202a27740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564878102bc0 0x564878113180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.273+0000 7fa202a27740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564878103580 0x564878111250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.273+0000 7fa202a27740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f81a9120 con 0x564878102bc0 2024-03-23T23:05:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.273+0000 7fa202a27740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f81aa420 con 0x564878103580 2024-03-23T23:05:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.273+0000 7fa202a27740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f81a9760 con 0x5648781019f0 2024-03-23T23:05:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.273+0000 7fa202a27740 10 client.0 _ll_register_callbacks cb 0x7fa1ec009e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.273+0000 7fa1fd400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648781019f0 0x564878112c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.273+0000 7fa1fd400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648781019f0 0x564878112c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58920/0 (socket says 172.21.15.12:58920) 2024-03-23T23:05:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.273+0000 7fa1fd400640 1 -- 172.21.15.12:0/4013078840 learned_addr learned my addr 172.21.15.12:0/4013078840 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.273+0000 7fa1fca00640 1 --2- 172.21.15.12:0/4013078840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564878102bc0 0x564878113180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.274+0000 7fa1fde00640 1 --2- 172.21.15.12:0/4013078840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564878103580 0x564878111250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.274+0000 7fa1fd400640 1 -- 172.21.15.12:0/4013078840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564878103580 msgr2=0x564878111250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.274+0000 7fa1fd400640 1 --2- 172.21.15.12:0/4013078840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564878103580 0x564878111250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.274+0000 7fa1fd400640 1 -- 172.21.15.12:0/4013078840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564878102bc0 msgr2=0x564878113180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.274+0000 7fa1fd400640 1 --2- 172.21.15.12:0/4013078840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564878102bc0 0x564878113180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.274+0000 7fa1fd400640 1 -- 172.21.15.12:0/4013078840 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564878111a80 con 0x5648781019f0 2024-03-23T23:05:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.274+0000 7fa1fca00640 1 --2- 172.21.15.12:0/4013078840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564878102bc0 0x564878113180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:05:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.274+0000 7fa1fd400640 1 --2- 172.21.15.12:0/4013078840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648781019f0 0x564878112c40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec009f80 tx=0x7fa1ec0122f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f82644c65045458e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.274+0000 7fa1dea00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:05:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.274+0000 7fa1dea00640 1 -- 172.21.15.12:0/4013078840 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec042060 con 0x5648781019f0 2024-03-23T23:05:16.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.274+0000 7fa1dea00640 1 -- 172.21.15.12:0/4013078840 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fa1ec0345e0 con 0x5648781019f0 2024-03-23T23:05:16.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.274+0000 7fa202a27740 1 -- 172.21.15.12:0/4013078840 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56487806db50 con 0x5648781019f0 2024-03-23T23:05:16.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.275+0000 7fa1dea00640 1 -- 172.21.15.12:0/4013078840 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1ec039050 con 0x5648781019f0 2024-03-23T23:05:16.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.275+0000 7fa202a27740 10 client.26164 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:16.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.275+0000 7fa202a27740 1 -- 172.21.15.12:0/4013078840 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x564878173b80 con 0x5648781019f0 2024-03-23T23:05:16.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.275+0000 7fa1dea00640 1 -- 172.21.15.12:0/4013078840 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56487806db50 con 0x5648781019f0 2024-03-23T23:05:16.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.275+0000 7fa1dea00640 1 -- 172.21.15.12:0/4013078840 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fa1ec01ec20 con 0x5648781019f0 2024-03-23T23:05:16.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.275+0000 7fa202a27740 10 client.26164 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:16.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.275+0000 7fa202a27740 10 client.26164 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:05:16.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.275+0000 7fa202a27740 20 client.26164 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:16.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.275+0000 7fa202a27740 1 --2- 172.21.15.12:0/4013078840 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5648781744b0 0x5648781768d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.276+0000 7fa202a27740 4 client.26164 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:05:16.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.276+0000 7fa202a27740 1 -- 172.21.15.12:0/4013078840 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564877feb500 con 0x5648781744b0 2024-03-23T23:05:16.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.276+0000 7fa1fca00640 1 --2- 172.21.15.12:0/4013078840 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5648781744b0 0x5648781768d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.276+0000 7fa1fca00640 1 --2- 172.21.15.12:0/4013078840 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5648781744b0 0x5648781768d0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d8304727b26177cf server_cookie=144563e75e2eaddd in_seq=0 out_seq=0 2024-03-23T23:05:16.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.276+0000 7fa1dea00640 10 client.26164 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:05:16.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.277+0000 7fa1dea00640 1 -- 172.21.15.12:0/4013078840 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564877feb500 con 0x5648781744b0 2024-03-23T23:05:16.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.277+0000 7fa1dea00640 10 client.26164 handle_command_reply: tid=0 2024-03-23T23:05:16.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.277+0000 7fa202a27740 1 client.26164 shutdown 2024-03-23T23:05:16.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.277+0000 7fa202a27740 1 -- 172.21.15.12:0/4013078840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648781019f0 msgr2=0x564878112c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.277+0000 7fa202a27740 1 --2- 172.21.15.12:0/4013078840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648781019f0 0x564878112c40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec009f80 tx=0x7fa1ec0122f0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.277+0000 7fa202a27740 1 -- 172.21.15.12:0/4013078840 shutdown_connections 2024-03-23T23:05:16.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.277+0000 7fa202a27740 1 --2- 172.21.15.12:0/4013078840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564878103580 0x564878111250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.277+0000 7fa202a27740 1 --2- 172.21.15.12:0/4013078840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564878102bc0 0x564878113180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.277+0000 7fa202a27740 1 --2- 172.21.15.12:0/4013078840 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5648781744b0 0x5648781768d0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.277+0000 7fa202a27740 1 --2- 172.21.15.12:0/4013078840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648781019f0 0x564878112c40 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.277+0000 7fa202a27740 1 -- 172.21.15.12:0/4013078840 >> 172.21.15.12:0/4013078840 conn(0x5648780fcde0 msgr2=0x5648781008d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:16.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.277+0000 7fa1dea00640 0 client.26164 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:05:16.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.277+0000 7fa202a27740 1 -- 172.21.15.12:0/4013078840 shutdown_connections 2024-03-23T23:05:16.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.277+0000 7fa202a27740 1 -- 172.21.15.12:0/4013078840 wait complete. 2024-03-23T23:05:16.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.277+0000 7fa202a27740 20 client.26164 trim_cache size 0 max 16384 2024-03-23T23:05:16.283 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:05:16.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.278+0000 7fa1fe800640 1 -- 172.21.15.12:0/3214521312 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa1e00a2950 msgr2=0x7fa1e00a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.278+0000 7fa1fe800640 1 --2- 172.21.15.12:0/3214521312 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa1e00a2950 0x7fa1e00a4e10 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8104810 tx=0x7fa1ec005f90 comp rx=0 tx=0).stop 2024-03-23T23:05:16.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.278+0000 7fa1fe800640 1 -- 172.21.15.12:0/3214521312 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa1f81049b0 msgr2=0x7fa1f819a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.278+0000 7fa1fe800640 1 --2- 172.21.15.12:0/3214521312 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa1f81049b0 0x7fa1f819a010 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4000c00 tx=0x7fa1e402c650 comp rx=0 tx=0).stop 2024-03-23T23:05:16.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.278+0000 7fa1fe800640 1 -- 172.21.15.12:0/3214521312 shutdown_connections 2024-03-23T23:05:16.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.278+0000 7fa1fe800640 1 --2- 172.21.15.12:0/3214521312 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa1e00a2950 0x7fa1e00a4e10 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.278+0000 7fa1fe800640 1 --2- 172.21.15.12:0/3214521312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1f8105370 0x7fa1f81a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.278+0000 7fa1fe800640 1 --2- 172.21.15.12:0/3214521312 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa1f81049b0 0x7fa1f819a010 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.278+0000 7fa1fe800640 1 --2- 172.21.15.12:0/3214521312 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1f81037c0 0x7fa1f8199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.278+0000 7fa1fe800640 1 -- 172.21.15.12:0/3214521312 >> 172.21.15.12:0/3214521312 conn(0x7fa1f80fed50 msgr2=0x7fa1f8107d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:16.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.278+0000 7fa1fe800640 1 -- 172.21.15.12:0/3214521312 shutdown_connections 2024-03-23T23:05:16.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.278+0000 7fa1fe800640 1 -- 172.21.15.12:0/3214521312 wait complete. 2024-03-23T23:05:16.301 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:16 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/621819747' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:05:16.305 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:05:16.305 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-23T23:05:16.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.367+0000 7fda0da00640 1 -- 172.21.15.12:0/1425800296 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda080fff50 msgr2=0x7fda08102380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.368+0000 7fda0da00640 1 --2- 172.21.15.12:0/1425800296 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda080fff50 0x7fda08102380 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0009900 tx=0x7fd9f002bc30 comp rx=0 tx=0).stop 2024-03-23T23:05:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.368+0000 7fda0da00640 1 -- 172.21.15.12:0/1425800296 shutdown_connections 2024-03-23T23:05:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.368+0000 7fda0da00640 1 --2- 172.21.15.12:0/1425800296 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda08105270 0x7fda081056f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.368+0000 7fda0da00640 1 --2- 172.21.15.12:0/1425800296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda081028c0 0x7fda08104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.368+0000 7fda0da00640 1 --2- 172.21.15.12:0/1425800296 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda080fff50 0x7fda08102380 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.368+0000 7fda0da00640 1 -- 172.21.15.12:0/1425800296 >> 172.21.15.12:0/1425800296 conn(0x7fda080fbb50 msgr2=0x7fda080fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.368+0000 7fda0da00640 1 -- 172.21.15.12:0/1425800296 shutdown_connections 2024-03-23T23:05:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.368+0000 7fda0da00640 1 -- 172.21.15.12:0/1425800296 wait complete. 2024-03-23T23:05:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.369+0000 7fda0da00640 1 Processor -- start 2024-03-23T23:05:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.369+0000 7fda0da00640 1 -- start start 2024-03-23T23:05:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.370+0000 7fda0da00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda080fff50 0x7fda0819ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.370+0000 7fda0da00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda081028c0 0x7fda0819e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.370+0000 7fda0da00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda08105270 0x7fda081a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.370+0000 7fda0da00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda081139a0 con 0x7fda080fff50 2024-03-23T23:05:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.370+0000 7fda0da00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda08113820 con 0x7fda08105270 2024-03-23T23:05:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.370+0000 7fda0da00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda08113b20 con 0x7fda081028c0 2024-03-23T23:05:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.370+0000 7fda06a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda080fff50 0x7fda0819ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.370+0000 7fda07400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda08105270 0x7fda081a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.370+0000 7fda06a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda080fff50 0x7fda0819ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40788/0 (socket says 172.21.15.12:40788) 2024-03-23T23:05:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.370+0000 7fda06a00640 1 -- 172.21.15.12:0/1580201520 learned_addr learned my addr 172.21.15.12:0/1580201520 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.370+0000 7fda06000640 1 --2- 172.21.15.12:0/1580201520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda081028c0 0x7fda0819e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.370+0000 7fda07400640 1 -- 172.21.15.12:0/1580201520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda081028c0 msgr2=0x7fda0819e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.370+0000 7fda07400640 1 --2- 172.21.15.12:0/1580201520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda081028c0 0x7fda0819e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.370+0000 7fda07400640 1 -- 172.21.15.12:0/1580201520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda080fff50 msgr2=0x7fda0819ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.370+0000 7fda07400640 1 --2- 172.21.15.12:0/1580201520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda080fff50 0x7fda0819ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.370+0000 7fda07400640 1 -- 172.21.15.12:0/1580201520 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda081a5af0 con 0x7fda08105270 2024-03-23T23:05:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.371+0000 7fda07400640 1 --2- 172.21.15.12:0/1580201520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda08105270 0x7fda081a5480 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fda00009fd0 tx=0x7fda0000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.371+0000 7fd9ff400640 1 -- 172.21.15.12:0/1580201520 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0000ce20 con 0x7fda08105270 2024-03-23T23:05:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.371+0000 7fd9ff400640 1 -- 172.21.15.12:0/1580201520 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda00005020 con 0x7fda08105270 2024-03-23T23:05:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.371+0000 7fd9ff400640 1 -- 172.21.15.12:0/1580201520 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda00010720 con 0x7fda08105270 2024-03-23T23:05:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.371+0000 7fda0da00640 1 -- 172.21.15.12:0/1580201520 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda081a5de0 con 0x7fda08105270 2024-03-23T23:05:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.371+0000 7fda0da00640 1 -- 172.21.15.12:0/1580201520 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda081a6250 con 0x7fda08105270 2024-03-23T23:05:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.375+0000 7fda1076c740 1 Processor -- start 2024-03-23T23:05:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.376+0000 7fd9ff400640 1 -- 172.21.15.12:0/1580201520 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fda00005540 con 0x7fda08105270 2024-03-23T23:05:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.377+0000 7fda1076c740 1 -- start start 2024-03-23T23:05:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.377+0000 7fda1076c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a523c9d1e0 0x55a523c9d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.377+0000 7fda1076c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a523c9dbd0 0x55a523ca26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.377+0000 7fda1076c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a523ca2d40 0x55a523ca51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.377+0000 7fda1076c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a523c603b0 con 0x55a523c9d1e0 2024-03-23T23:05:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.377+0000 7fda1076c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a523c02830 con 0x55a523ca2d40 2024-03-23T23:05:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.377+0000 7fda1076c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a523bab5d0 con 0x55a523c9dbd0 2024-03-23T23:05:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.377+0000 7fda06a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a523c9dbd0 0x55a523ca26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.377+0000 7fda06a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a523c9dbd0 0x55a523ca26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58936/0 (socket says 172.21.15.12:58936) 2024-03-23T23:05:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.377+0000 7fda06a00640 1 -- 172.21.15.12:0/1142296129 learned_addr learned my addr 172.21.15.12:0/1142296129 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.377+0000 7fda06000640 1 --2- 172.21.15.12:0/1142296129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a523c9d1e0 0x55a523c9d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.377+0000 7fda06000640 1 --2- 172.21.15.12:0/1142296129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a523ca2d40 0x55a523ca51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.378+0000 7fda06a00640 1 -- 172.21.15.12:0/1142296129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a523ca2d40 msgr2=0x55a523ca51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.378+0000 7fda06a00640 1 --2- 172.21.15.12:0/1142296129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a523ca2d40 0x55a523ca51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.378+0000 7fda06a00640 1 -- 172.21.15.12:0/1142296129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a523c9d1e0 msgr2=0x55a523c9d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.378+0000 7fda06a00640 1 --2- 172.21.15.12:0/1142296129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a523c9d1e0 0x55a523c9d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.378+0000 7fda06a00640 1 -- 172.21.15.12:0/1142296129 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a523bc32c0 con 0x55a523c9dbd0 2024-03-23T23:05:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.378+0000 7fda06000640 1 --2- 172.21.15.12:0/1142296129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a523ca2d40 0x55a523ca51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:05:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.378+0000 7fd9ff400640 1 --2- 172.21.15.12:0/1580201520 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd9ec0a2920 0x7fd9ec0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.378+0000 7fda06a00640 1 --2- 172.21.15.12:0/1142296129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a523c9dbd0 0x55a523ca26d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd9f002c170 tx=0x7fd9f00095e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e07c771814ef6d57 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.378+0000 7fd9ff400640 1 -- 172.21.15.12:0/1580201520 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fda000ceff0 con 0x7fda08105270 2024-03-23T23:05:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.378+0000 7fda06000640 1 --2- 172.21.15.12:0/1580201520 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd9ec0a2920 0x7fd9ec0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.378+0000 7fd9f7e00640 1 -- 172.21.15.12:0/1142296129 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0045070 con 0x55a523c9dbd0 2024-03-23T23:05:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.378+0000 7fd9f7e00640 1 -- 172.21.15.12:0/1142296129 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9f0004040 con 0x55a523c9dbd0 2024-03-23T23:05:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.378+0000 7fd9f7e00640 1 -- 172.21.15.12:0/1142296129 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0004ef0 con 0x55a523c9dbd0 2024-03-23T23:05:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.379+0000 7fda06000640 1 --2- 172.21.15.12:0/1580201520 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd9ec0a2920 0x7fd9ec0a4de0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x55a523c9e430 tx=0x7fd9f8005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.379+0000 7fda1076c740 1 -- 172.21.15.12:0/1142296129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a523c9dbd0 msgr2=0x55a523ca26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.379+0000 7fda1076c740 1 --2- 172.21.15.12:0/1142296129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a523c9dbd0 0x55a523ca26d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd9f002c170 tx=0x7fd9f00095e0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.379+0000 7fda1076c740 1 -- 172.21.15.12:0/1142296129 shutdown_connections 2024-03-23T23:05:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.379+0000 7fda1076c740 1 --2- 172.21.15.12:0/1142296129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a523ca2d40 0x55a523ca51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.379+0000 7fda1076c740 1 --2- 172.21.15.12:0/1142296129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a523c9dbd0 0x55a523ca26d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.379+0000 7fda1076c740 1 --2- 172.21.15.12:0/1142296129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a523c9d1e0 0x55a523c9d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.379+0000 7fda1076c740 1 -- 172.21.15.12:0/1142296129 >> 172.21.15.12:0/1142296129 conn(0x55a523c98750 msgr2=0x55a523c98b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.380+0000 7fda1076c740 1 -- 172.21.15.12:0/1142296129 shutdown_connections 2024-03-23T23:05:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.380+0000 7fda1076c740 1 -- 172.21.15.12:0/1142296129 wait complete. 2024-03-23T23:05:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.380+0000 7fda1076c740 1 -- start start 2024-03-23T23:05:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.381+0000 7fda1076c740 1 Processor -- start 2024-03-23T23:05:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.381+0000 7fda1076c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a523c9d1e0 0x55a523ca7300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.381+0000 7fda1076c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a523c9dbd0 0x55a523ca7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.381+0000 7fda1076c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a523ca2d40 0x55a523cae880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.381+0000 7fda1076c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a523bbea20 con 0x55a523ca2d40 2024-03-23T23:05:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.381+0000 7fda1076c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a523bfe8d0 con 0x55a523c9dbd0 2024-03-23T23:05:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.381+0000 7fda1076c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a523ca5c90 con 0x55a523c9d1e0 2024-03-23T23:05:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.381+0000 7fda06a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a523c9d1e0 0x55a523ca7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.381+0000 7fda06a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a523c9d1e0 0x55a523ca7300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58950/0 (socket says 172.21.15.12:58950) 2024-03-23T23:05:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.381+0000 7fda06a00640 1 -- 172.21.15.12:0/3806017594 learned_addr learned my addr 172.21.15.12:0/3806017594 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.381+0000 7fda1076c740 10 client.0 _ll_register_callbacks cb 0x55a523bfb1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.381+0000 7fda06000640 1 --2- 172.21.15.12:0/3806017594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a523c9dbd0 0x55a523ca7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.381+0000 7fda07400640 1 --2- 172.21.15.12:0/3806017594 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a523ca2d40 0x55a523cae880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.382+0000 7fda06a00640 1 -- 172.21.15.12:0/3806017594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a523c9dbd0 msgr2=0x55a523ca7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.382+0000 7fda06a00640 1 --2- 172.21.15.12:0/3806017594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a523c9dbd0 0x55a523ca7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.382+0000 7fda06a00640 1 -- 172.21.15.12:0/3806017594 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a523ca2d40 msgr2=0x55a523cae880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.382+0000 7fda06a00640 1 --2- 172.21.15.12:0/3806017594 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a523ca2d40 0x55a523cae880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.382+0000 7fda06a00640 1 -- 172.21.15.12:0/3806017594 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a523917210 con 0x55a523c9d1e0 2024-03-23T23:05:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.382+0000 7fda06a00640 1 --2- 172.21.15.12:0/3806017594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a523c9d1e0 0x55a523ca7300 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd9f002c8e0 tx=0x7fd9f002bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54846da032622f30 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.382+0000 7fd9f4c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:05:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.382+0000 7fd9f4c00640 1 -- 172.21.15.12:0/3806017594 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0045040 con 0x55a523c9d1e0 2024-03-23T23:05:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.383+0000 7fd9f4c00640 1 -- 172.21.15.12:0/3806017594 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd9f0004ef0 con 0x55a523c9d1e0 2024-03-23T23:05:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.383+0000 7fda1076c740 1 -- 172.21.15.12:0/3806017594 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd9f0004040 con 0x55a523c9d1e0 2024-03-23T23:05:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.383+0000 7fd9f4c00640 1 -- 172.21.15.12:0/3806017594 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9f005b050 con 0x55a523c9d1e0 2024-03-23T23:05:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.383+0000 7fda1076c740 10 client.26176 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.383+0000 7fda1076c740 1 -- 172.21.15.12:0/3806017594 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a5239e3620 con 0x55a523c9d1e0 2024-03-23T23:05:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.383+0000 7fd9f4c00640 1 -- 172.21.15.12:0/3806017594 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9f0004040 con 0x55a523c9d1e0 2024-03-23T23:05:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.383+0000 7fd9f4c00640 1 -- 172.21.15.12:0/3806017594 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fd9f0003ba0 con 0x55a523c9d1e0 2024-03-23T23:05:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.384+0000 7fda1076c740 10 client.26176 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.384+0000 7fda1076c740 10 client.26176 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.384+0000 7fda1076c740 20 client.26176 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.384+0000 7fda1076c740 1 --2- 172.21.15.12:0/3806017594 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a523cb2d70 0x55a523cb5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.384+0000 7fda1076c740 4 client.26176 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:05:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.384+0000 7fda1076c740 1 -- 172.21.15.12:0/3806017594 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a523c0fd70 con 0x55a523cb2d70 2024-03-23T23:05:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.384+0000 7fda06000640 1 --2- 172.21.15.12:0/3806017594 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a523cb2d70 0x55a523cb5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.384+0000 7fda06000640 1 --2- 172.21.15.12:0/3806017594 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a523cb2d70 0x55a523cb5160 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e46b748583b3bb87 server_cookie=8d01ad851e4993a0 in_seq=0 out_seq=0 2024-03-23T23:05:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.384+0000 7fd9f4c00640 10 client.26176 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.385+0000 7fd9f4c00640 1 -- 172.21.15.12:0/3806017594 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a523c0fd70 con 0x55a523cb2d70 2024-03-23T23:05:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.385+0000 7fd9f4c00640 10 client.26176 handle_command_reply: tid=0 2024-03-23T23:05:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.385+0000 7fda1076c740 1 client.26176 shutdown 2024-03-23T23:05:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.385+0000 7fda1076c740 1 -- 172.21.15.12:0/3806017594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a523c9d1e0 msgr2=0x55a523ca7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.385+0000 7fda1076c740 1 --2- 172.21.15.12:0/3806017594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a523c9d1e0 0x55a523ca7300 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd9f002c8e0 tx=0x7fd9f002bc30 comp rx=0 tx=0).stop 2024-03-23T23:05:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.386+0000 7fda1076c740 1 -- 172.21.15.12:0/3806017594 shutdown_connections 2024-03-23T23:05:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.386+0000 7fda1076c740 1 --2- 172.21.15.12:0/3806017594 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a523cb2d70 0x55a523cb5160 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.386+0000 7fda1076c740 1 --2- 172.21.15.12:0/3806017594 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a523ca2d40 0x55a523cae880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.386+0000 7fda1076c740 1 --2- 172.21.15.12:0/3806017594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a523c9dbd0 0x55a523ca7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.386+0000 7fd9f4c00640 0 client.26176 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.386+0000 7fda1076c740 1 --2- 172.21.15.12:0/3806017594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a523c9d1e0 0x55a523ca7300 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.386+0000 7fda1076c740 1 -- 172.21.15.12:0/3806017594 >> 172.21.15.12:0/3806017594 conn(0x55a523c98750 msgr2=0x55a523c97950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:16.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.386+0000 7fda1076c740 1 -- 172.21.15.12:0/3806017594 shutdown_connections 2024-03-23T23:05:16.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.386+0000 7fda1076c740 1 -- 172.21.15.12:0/3806017594 wait complete. 2024-03-23T23:05:16.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.386+0000 7fda1076c740 20 client.26176 trim_cache size 0 max 16384 2024-03-23T23:05:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.401+0000 7fda1076c740 1 Processor -- start 2024-03-23T23:05:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.401+0000 7fda1076c740 1 -- start start 2024-03-23T23:05:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.402+0000 7fda1076c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a523c9d1e0 0x55a523ca6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.402+0000 7fda1076c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a523c9dbd0 0x55a523cae7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.402+0000 7fda1076c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a523ca2d40 0x55a523c9b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.402+0000 7fda1076c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda081b1440 con 0x55a523c9dbd0 2024-03-23T23:05:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.402+0000 7fda1076c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda081aef40 con 0x55a523ca2d40 2024-03-23T23:05:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.402+0000 7fda1076c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda081acfe0 con 0x55a523c9d1e0 2024-03-23T23:05:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.402+0000 7fda07400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a523ca2d40 0x55a523c9b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.402+0000 7fda06000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a523c9dbd0 0x55a523cae7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.402+0000 7fda06a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a523c9d1e0 0x55a523ca6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.402+0000 7fda07400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a523ca2d40 0x55a523c9b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59560/0 (socket says 172.21.15.12:59560) 2024-03-23T23:05:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.402+0000 7fda07400640 1 -- 172.21.15.12:0/225696046 learned_addr learned my addr 172.21.15.12:0/225696046 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.402+0000 7fda07400640 1 -- 172.21.15.12:0/225696046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a523c9d1e0 msgr2=0x55a523ca6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.402+0000 7fda07400640 1 --2- 172.21.15.12:0/225696046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a523c9d1e0 0x55a523ca6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.402+0000 7fda07400640 1 -- 172.21.15.12:0/225696046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a523c9dbd0 msgr2=0x55a523cae7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.402+0000 7fda07400640 1 --2- 172.21.15.12:0/225696046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a523c9dbd0 0x55a523cae7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.402+0000 7fda07400640 1 -- 172.21.15.12:0/225696046 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a523c994c0 con 0x55a523ca2d40 2024-03-23T23:05:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.403+0000 7fda07400640 1 --2- 172.21.15.12:0/225696046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a523ca2d40 0x55a523c9b9b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fda00097de0 tx=0x7fda000054b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0d0922b571945a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.403+0000 7fd9f7400640 1 -- 172.21.15.12:0/225696046 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda000a4120 con 0x55a523ca2d40 2024-03-23T23:05:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.403+0000 7fd9f7400640 1 -- 172.21.15.12:0/225696046 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda00099a20 con 0x55a523ca2d40 2024-03-23T23:05:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.403+0000 7fd9f7400640 1 -- 172.21.15.12:0/225696046 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda000a3ef0 con 0x55a523ca2d40 2024-03-23T23:05:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.404+0000 7fda1076c740 1 -- 172.21.15.12:0/225696046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a523ca2d40 msgr2=0x55a523c9b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.404+0000 7fda1076c740 1 --2- 172.21.15.12:0/225696046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a523ca2d40 0x55a523c9b9b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fda00097de0 tx=0x7fda000054b0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.404+0000 7fda1076c740 1 -- 172.21.15.12:0/225696046 shutdown_connections 2024-03-23T23:05:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.404+0000 7fda1076c740 1 --2- 172.21.15.12:0/225696046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a523ca2d40 0x55a523c9b9b0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.404+0000 7fda1076c740 1 --2- 172.21.15.12:0/225696046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a523c9dbd0 0x55a523cae7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.404+0000 7fda1076c740 1 --2- 172.21.15.12:0/225696046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a523c9d1e0 0x55a523ca6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.404+0000 7fda1076c740 1 -- 172.21.15.12:0/225696046 >> 172.21.15.12:0/225696046 conn(0x55a523c98750 msgr2=0x55a523c9ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.404+0000 7fda1076c740 1 -- 172.21.15.12:0/225696046 shutdown_connections 2024-03-23T23:05:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.404+0000 7fda1076c740 1 -- 172.21.15.12:0/225696046 wait complete. 2024-03-23T23:05:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.404+0000 7fda1076c740 1 -- start start 2024-03-23T23:05:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.405+0000 7fda1076c740 1 Processor -- start 2024-03-23T23:05:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.405+0000 7fda1076c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a523c9d1e0 0x55a523cac750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.405+0000 7fda1076c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a523c9dbd0 0x55a523cacc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.405+0000 7fda1076c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a523ca2d40 0x55a523cadc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.405+0000 7fda1076c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda081ad5e0 con 0x55a523c9d1e0 2024-03-23T23:05:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.405+0000 7fda1076c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda081ae8e0 con 0x55a523c9dbd0 2024-03-23T23:05:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.405+0000 7fda1076c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda081adc20 con 0x55a523ca2d40 2024-03-23T23:05:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.405+0000 7fda1076c740 10 client.0 _ll_register_callbacks cb 0x7fd9f800ba10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.405+0000 7fda06000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a523c9dbd0 0x55a523cacc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.405+0000 7fda06000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a523c9dbd0 0x55a523cacc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59566/0 (socket says 172.21.15.12:59566) 2024-03-23T23:05:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.405+0000 7fda06000640 1 -- 172.21.15.12:0/2147315756 learned_addr learned my addr 172.21.15.12:0/2147315756 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.405+0000 7fda07400640 1 --2- 172.21.15.12:0/2147315756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a523ca2d40 0x55a523cadc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.405+0000 7fda06a00640 1 --2- 172.21.15.12:0/2147315756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a523c9d1e0 0x55a523cac750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.406+0000 7fda06000640 1 -- 172.21.15.12:0/2147315756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a523ca2d40 msgr2=0x55a523cadc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.406+0000 7fda06000640 1 --2- 172.21.15.12:0/2147315756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a523ca2d40 0x55a523cadc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.406+0000 7fda06000640 1 -- 172.21.15.12:0/2147315756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a523c9d1e0 msgr2=0x55a523cac750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.406+0000 7fda06000640 1 --2- 172.21.15.12:0/2147315756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a523c9d1e0 0x55a523cac750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.406+0000 7fda06000640 1 -- 172.21.15.12:0/2147315756 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a523cae240 con 0x55a523c9dbd0 2024-03-23T23:05:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.406+0000 7fda06a00640 1 --2- 172.21.15.12:0/2147315756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a523c9d1e0 0x55a523cac750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:05:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.406+0000 7fda07400640 1 --2- 172.21.15.12:0/2147315756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a523ca2d40 0x55a523cadc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:05:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.406+0000 7fda06000640 1 --2- 172.21.15.12:0/2147315756 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a523c9dbd0 0x55a523cacc90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd9f800bfa0 tx=0x7fd9f8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b4395b377247e01 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.406+0000 7fd9f4c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:05:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.406+0000 7fd9f4c00640 1 -- 172.21.15.12:0/2147315756 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f802f070 con 0x55a523c9dbd0 2024-03-23T23:05:16.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.406+0000 7fd9f4c00640 1 -- 172.21.15.12:0/2147315756 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd9f802b720 con 0x55a523c9dbd0 2024-03-23T23:05:16.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.406+0000 7fda1076c740 1 -- 172.21.15.12:0/2147315756 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fda00099a20 con 0x55a523c9dbd0 2024-03-23T23:05:16.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.407+0000 7fd9f4c00640 1 -- 172.21.15.12:0/2147315756 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9f803b050 con 0x55a523c9dbd0 2024-03-23T23:05:16.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.407+0000 7fda1076c740 10 client.26213 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:16.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.407+0000 7fda1076c740 1 -- 172.21.15.12:0/2147315756 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a523d0ec90 con 0x55a523c9dbd0 2024-03-23T23:05:16.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.407+0000 7fd9f4c00640 1 -- 172.21.15.12:0/2147315756 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda00099a20 con 0x55a523c9dbd0 2024-03-23T23:05:16.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.407+0000 7fd9f4c00640 1 -- 172.21.15.12:0/2147315756 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fd9f8012a30 con 0x55a523c9dbd0 2024-03-23T23:05:16.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.407+0000 7fda1076c740 10 client.26213 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:16.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.407+0000 7fda1076c740 10 client.26213 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:16.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.407+0000 7fda1076c740 20 client.26213 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:16.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.407+0000 7fda1076c740 1 --2- 172.21.15.12:0/2147315756 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a523d0f520 0x55a523d11940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.407+0000 7fda1076c740 4 client.26213 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:05:16.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.407+0000 7fda1076c740 1 -- 172.21.15.12:0/2147315756 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a523c93a80 con 0x55a523d0f520 2024-03-23T23:05:16.411 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:16 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/621819747' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:05:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.408+0000 7fda06a00640 1 --2- 172.21.15.12:0/2147315756 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a523d0f520 0x55a523d11940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.408+0000 7fd9f4c00640 10 client.26213 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.408+0000 7fda06a00640 1 --2- 172.21.15.12:0/2147315756 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a523d0f520 0x55a523d11940 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f72286931adbef20 server_cookie=e22e7aa72a386f0 in_seq=0 out_seq=0 2024-03-23T23:05:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.420+0000 7fd9f4c00640 1 -- 172.21.15.12:0/2147315756 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a523c93a80 con 0x55a523d0f520 2024-03-23T23:05:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.420+0000 7fd9f4c00640 10 client.26213 handle_command_reply: tid=0 2024-03-23T23:05:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.420+0000 7fda1076c740 1 client.26213 shutdown 2024-03-23T23:05:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.420+0000 7fda1076c740 1 -- 172.21.15.12:0/2147315756 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a523c9dbd0 msgr2=0x55a523cacc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.420+0000 7fda1076c740 1 --2- 172.21.15.12:0/2147315756 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a523c9dbd0 0x55a523cacc90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd9f800bfa0 tx=0x7fd9f8004730 comp rx=0 tx=0).stop 2024-03-23T23:05:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.420+0000 7fda1076c740 1 -- 172.21.15.12:0/2147315756 shutdown_connections 2024-03-23T23:05:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.420+0000 7fda1076c740 1 --2- 172.21.15.12:0/2147315756 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a523d0f520 0x55a523d11940 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.420+0000 7fda1076c740 1 --2- 172.21.15.12:0/2147315756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a523ca2d40 0x55a523cadc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.420+0000 7fda1076c740 1 --2- 172.21.15.12:0/2147315756 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a523c9dbd0 0x55a523cacc90 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.420+0000 7fda1076c740 1 --2- 172.21.15.12:0/2147315756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a523c9d1e0 0x55a523cac750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.420+0000 7fd9f4c00640 0 client.26213 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.420+0000 7fda1076c740 1 -- 172.21.15.12:0/2147315756 >> 172.21.15.12:0/2147315756 conn(0x55a523c98750 msgr2=0x55a523c9b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.420+0000 7fda1076c740 1 -- 172.21.15.12:0/2147315756 shutdown_connections 2024-03-23T23:05:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.420+0000 7fda1076c740 1 -- 172.21.15.12:0/2147315756 wait complete. 2024-03-23T23:05:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.420+0000 7fda1076c740 20 client.26213 trim_cache size 0 max 16384 2024-03-23T23:05:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:05:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:05:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:05:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:05:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.421+0000 7fda0da00640 1 -- 172.21.15.12:0/1580201520 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd9ec0a2920 msgr2=0x7fd9ec0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.421+0000 7fda0da00640 1 --2- 172.21.15.12:0/1580201520 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd9ec0a2920 0x7fd9ec0a4de0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x55a523c9e430 tx=0x7fd9f8005c90 comp rx=0 tx=0).stop 2024-03-23T23:05:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.421+0000 7fda0da00640 1 -- 172.21.15.12:0/1580201520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda08105270 msgr2=0x7fda081a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.421+0000 7fda0da00640 1 --2- 172.21.15.12:0/1580201520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda08105270 0x7fda081a5480 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fda00009fd0 tx=0x7fda0000ef10 comp rx=0 tx=0).stop 2024-03-23T23:05:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.421+0000 7fda0da00640 1 -- 172.21.15.12:0/1580201520 shutdown_connections 2024-03-23T23:05:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.421+0000 7fda0da00640 1 --2- 172.21.15.12:0/1580201520 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd9ec0a2920 0x7fd9ec0a4de0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.421+0000 7fda0da00640 1 --2- 172.21.15.12:0/1580201520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda08105270 0x7fda081a5480 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.421+0000 7fda0da00640 1 --2- 172.21.15.12:0/1580201520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda081028c0 0x7fda0819e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.421+0000 7fda0da00640 1 --2- 172.21.15.12:0/1580201520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda080fff50 0x7fda0819ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.421+0000 7fda0da00640 1 -- 172.21.15.12:0/1580201520 >> 172.21.15.12:0/1580201520 conn(0x7fda080fbb50 msgr2=0x7fda08103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.421+0000 7fda0da00640 1 -- 172.21.15.12:0/1580201520 shutdown_connections 2024-03-23T23:05:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:16.421+0000 7fda0da00640 1 -- 172.21.15.12:0/1580201520 wait complete. 2024-03-23T23:05:16.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.522+0000 7f2ed9800640 1 -- 172.21.15.12:0/2926322611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ed4107bc0 msgr2=0x7f2ed4107fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.522+0000 7f2ed9800640 1 --2- 172.21.15.12:0/2926322611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ed4107bc0 0x7f2ed4107fe0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4009900 tx=0x7f2ec402bc30 comp rx=0 tx=0).stop 2024-03-23T23:05:16.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.522+0000 7f2ed9800640 1 -- 172.21.15.12:0/2926322611 shutdown_connections 2024-03-23T23:05:16.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.522+0000 7f2ed9800640 1 --2- 172.21.15.12:0/2926322611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ed4109770 0x7f2ed4110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.522+0000 7f2ed9800640 1 --2- 172.21.15.12:0/2926322611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ed4108db0 0x7f2ed4109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.522+0000 7f2ed9800640 1 --2- 172.21.15.12:0/2926322611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ed4107bc0 0x7f2ed4107fe0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.522+0000 7f2ed9800640 1 -- 172.21.15.12:0/2926322611 >> 172.21.15.12:0/2926322611 conn(0x7f2ed4076b70 msgr2=0x7f2ed4078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:16.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.522+0000 7f2ed9800640 1 -- 172.21.15.12:0/2926322611 shutdown_connections 2024-03-23T23:05:16.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.522+0000 7f2ed9800640 1 -- 172.21.15.12:0/2926322611 wait complete. 2024-03-23T23:05:16.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.523+0000 7f2ed9800640 1 Processor -- start 2024-03-23T23:05:16.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.523+0000 7f2ed9800640 1 -- start start 2024-03-23T23:05:16.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.523+0000 7f2ed9800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ed4107bc0 0x7f2ed419df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.523+0000 7f2ed9800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ed4108db0 0x7f2ed419e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.523+0000 7f2ed9800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ed4109770 0x7f2ed41a54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.523+0000 7f2ed9800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed4113880 con 0x7f2ed4107bc0 2024-03-23T23:05:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.523+0000 7f2ed9800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed4113700 con 0x7f2ed4108db0 2024-03-23T23:05:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.523+0000 7f2ed9800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed4113a00 con 0x7f2ed4109770 2024-03-23T23:05:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.524+0000 7f2ed3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ed4107bc0 0x7f2ed419df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.524+0000 7f2ed3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ed4108db0 0x7f2ed419e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.524+0000 7f2ed3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ed4108db0 0x7f2ed419e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59582/0 (socket says 172.21.15.12:59582) 2024-03-23T23:05:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.524+0000 7f2ed3400640 1 -- 172.21.15.12:0/3860389526 learned_addr learned my addr 172.21.15.12:0/3860389526 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.524+0000 7f2ed3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ed4107bc0 0x7f2ed419df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40842/0 (socket says 172.21.15.12:40842) 2024-03-23T23:05:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.524+0000 7f2ed8e00640 1 --2- 172.21.15.12:0/3860389526 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ed4109770 0x7f2ed41a54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.524+0000 7f2ed3400640 1 -- 172.21.15.12:0/3860389526 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ed4109770 msgr2=0x7f2ed41a54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.524+0000 7f2ed3400640 1 --2- 172.21.15.12:0/3860389526 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ed4109770 0x7f2ed41a54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.524+0000 7f2ed3400640 1 -- 172.21.15.12:0/3860389526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ed4107bc0 msgr2=0x7f2ed419df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.524+0000 7f2ed3400640 1 --2- 172.21.15.12:0/3860389526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ed4107bc0 0x7f2ed419df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.524+0000 7f2ed3400640 1 -- 172.21.15.12:0/3860389526 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed41a5b40 con 0x7f2ed4108db0 2024-03-23T23:05:16.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.524+0000 7f2ed3400640 1 --2- 172.21.15.12:0/3860389526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ed4108db0 0x7f2ed419e460 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2ec000eb90 tx=0x7f2ec000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:16.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.525+0000 7f2ed0c00640 1 -- 172.21.15.12:0/3860389526 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec0004d20 con 0x7f2ed4108db0 2024-03-23T23:05:16.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.525+0000 7f2ed0c00640 1 -- 172.21.15.12:0/3860389526 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ec000ee90 con 0x7f2ed4108db0 2024-03-23T23:05:16.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.525+0000 7f2ed0c00640 1 -- 172.21.15.12:0/3860389526 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec0010830 con 0x7f2ed4108db0 2024-03-23T23:05:16.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.525+0000 7f2ed9800640 1 -- 172.21.15.12:0/3860389526 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed41a5e30 con 0x7f2ed4108db0 2024-03-23T23:05:16.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.525+0000 7f2ed9800640 1 -- 172.21.15.12:0/3860389526 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed41a62a0 con 0x7f2ed4108db0 2024-03-23T23:05:16.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.529+0000 7f2edda49740 1 Processor -- start 2024-03-23T23:05:16.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.529+0000 7f2edda49740 1 -- start start 2024-03-23T23:05:16.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.530+0000 7f2edda49740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55af7ddf69f0 0x55af7ddf6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.530+0000 7f2edda49740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af7ddf7bc0 0x55af7ddf8040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.530+0000 7f2edda49740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af7ddf8580 0x55af7ddfee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.530+0000 7f2edda49740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af7dd5b2c0 con 0x55af7ddf69f0 2024-03-23T23:05:16.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.530+0000 7f2edda49740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af7dd03c90 con 0x55af7ddf7bc0 2024-03-23T23:05:16.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.530+0000 7f2edda49740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af7dd57360 con 0x55af7ddf8580 2024-03-23T23:05:16.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.530+0000 7f2ed3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af7ddf7bc0 0x55af7ddf8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.530+0000 7f2ed8e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55af7ddf69f0 0x55af7ddf6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.530+0000 7f2ed3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af7ddf7bc0 0x55af7ddf8040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59584/0 (socket says 172.21.15.12:59584) 2024-03-23T23:05:16.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.530+0000 7f2ed3e00640 1 -- 172.21.15.12:0/289962715 learned_addr learned my addr 172.21.15.12:0/289962715 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:16.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.530+0000 7f2ed0c00640 1 -- 172.21.15.12:0/3860389526 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2ec0005020 con 0x7f2ed4108db0 2024-03-23T23:05:16.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.531+0000 7f2ed3e00640 1 -- 172.21.15.12:0/289962715 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af7ddf8580 msgr2=0x55af7ddfee10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-23T23:05:16.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.531+0000 7f2ed3e00640 1 --2- 172.21.15.12:0/289962715 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af7ddf8580 0x55af7ddfee10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.531+0000 7f2ed3e00640 1 -- 172.21.15.12:0/289962715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55af7ddf69f0 msgr2=0x55af7ddf6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:16.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.531+0000 7f2ed3e00640 1 --2- 172.21.15.12:0/289962715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55af7ddf69f0 0x55af7ddf6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.531+0000 7f2ed3e00640 1 -- 172.21.15.12:0/289962715 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af7dd28080 con 0x55af7ddf7bc0 2024-03-23T23:05:16.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.531+0000 7f2ed8e00640 1 --2- 172.21.15.12:0/289962715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55af7ddf69f0 0x55af7ddf6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:16.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.531+0000 7f2ed3e00640 1 --2- 172.21.15.12:0/289962715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af7ddf7bc0 0x55af7ddf8040 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4000c00 tx=0x7f2ec40095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=873c804d8d27152a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:16.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.532+0000 7f2ec9600640 1 -- 172.21.15.12:0/289962715 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec402c980 con 0x55af7ddf7bc0 2024-03-23T23:05:16.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.532+0000 7f2ec9600640 1 -- 172.21.15.12:0/289962715 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ec402cb20 con 0x55af7ddf7bc0 2024-03-23T23:05:16.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.532+0000 7f2ec9600640 1 -- 172.21.15.12:0/289962715 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec4004920 con 0x55af7ddf7bc0 2024-03-23T23:05:16.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.532+0000 7f2edda49740 1 -- 172.21.15.12:0/289962715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af7ddf7bc0 msgr2=0x55af7ddf8040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.532+0000 7f2edda49740 1 --2- 172.21.15.12:0/289962715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af7ddf7bc0 0x55af7ddf8040 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4000c00 tx=0x7f2ec40095e0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.532+0000 7f2edda49740 1 -- 172.21.15.12:0/289962715 shutdown_connections 2024-03-23T23:05:16.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.532+0000 7f2edda49740 1 --2- 172.21.15.12:0/289962715 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af7ddf8580 0x55af7ddfee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.532+0000 7f2edda49740 1 --2- 172.21.15.12:0/289962715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af7ddf7bc0 0x55af7ddf8040 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.532+0000 7f2edda49740 1 --2- 172.21.15.12:0/289962715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55af7ddf69f0 0x55af7ddf6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.532+0000 7f2edda49740 1 -- 172.21.15.12:0/289962715 >> 172.21.15.12:0/289962715 conn(0x55af7ddf1de0 msgr2=0x55af7ddf21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:16.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.532+0000 7f2edda49740 1 -- 172.21.15.12:0/289962715 shutdown_connections 2024-03-23T23:05:16.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.533+0000 7f2edda49740 1 -- 172.21.15.12:0/289962715 wait complete. 2024-03-23T23:05:16.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.533+0000 7f2ed0c00640 1 --2- 172.21.15.12:0/3860389526 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2eb80a2790 0x7f2eb80a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.533+0000 7f2ed0c00640 1 -- 172.21.15.12:0/3860389526 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2ec00cefc0 con 0x7f2ed4108db0 2024-03-23T23:05:16.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.533+0000 7f2edda49740 1 -- start start 2024-03-23T23:05:16.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.533+0000 7f2ed3e00640 1 --2- 172.21.15.12:0/3860389526 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2eb80a2790 0x7f2eb80a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.533+0000 7f2edda49740 1 Processor -- start 2024-03-23T23:05:16.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.534+0000 7f2ed3e00640 1 --2- 172.21.15.12:0/3860389526 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2eb80a2790 0x7f2eb80a4c50 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2ec402c7b0 tx=0x7f2ec402bc30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:16.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.534+0000 7f2edda49740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af7ddf69f0 0x55af7de00c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.534+0000 7f2edda49740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55af7ddf7bc0 0x55af7de011a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.534+0000 7f2edda49740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af7ddf8580 0x55af7de08190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.534+0000 7f2edda49740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af7ddff530 con 0x55af7ddf7bc0 2024-03-23T23:05:16.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.534+0000 7f2edda49740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af7dd170d0 con 0x55af7ddf69f0 2024-03-23T23:05:16.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.534+0000 7f2edda49740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af7ddff6b0 con 0x55af7ddf8580 2024-03-23T23:05:16.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.534+0000 7f2edda49740 10 client.0 _ll_register_callbacks cb 0x55af7dd1be80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:16.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.534+0000 7f2ed3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af7ddf69f0 0x55af7de00c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.534+0000 7f2ed3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af7ddf69f0 0x55af7de00c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59596/0 (socket says 172.21.15.12:59596) 2024-03-23T23:05:16.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.534+0000 7f2ed3e00640 1 -- 172.21.15.12:0/2487837617 learned_addr learned my addr 172.21.15.12:0/2487837617 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:16.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.534+0000 7f2ed8e00640 1 --2- 172.21.15.12:0/2487837617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af7ddf8580 0x55af7de08190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.534+0000 7f2ed3400640 1 --2- 172.21.15.12:0/2487837617 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55af7ddf7bc0 0x55af7de011a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.535+0000 7f2ed3e00640 1 -- 172.21.15.12:0/2487837617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af7ddf8580 msgr2=0x55af7de08190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:16.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.535+0000 7f2ed3e00640 1 --2- 172.21.15.12:0/2487837617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af7ddf8580 0x55af7de08190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.535+0000 7f2ed3e00640 1 -- 172.21.15.12:0/2487837617 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55af7ddf7bc0 msgr2=0x55af7de011a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:16.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.535+0000 7f2ed3e00640 1 --2- 172.21.15.12:0/2487837617 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55af7ddf7bc0 0x55af7de011a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.535+0000 7f2ed3e00640 1 -- 172.21.15.12:0/2487837617 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55af7da6f3f0 con 0x55af7ddf69f0 2024-03-23T23:05:16.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.535+0000 7f2ed3400640 1 --2- 172.21.15.12:0/2487837617 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55af7ddf7bc0 0x55af7de011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:05:16.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.535+0000 7f2ed3e00640 1 --2- 172.21.15.12:0/2487837617 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af7ddf69f0 0x55af7de00c60 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4006010 tx=0x7f2ec40095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=280bb355102701f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:16.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.535+0000 7f2ebe000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:05:16.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.535+0000 7f2ebe000640 1 -- 172.21.15.12:0/2487837617 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec4058020 con 0x55af7ddf69f0 2024-03-23T23:05:16.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.535+0000 7f2ebe000640 1 -- 172.21.15.12:0/2487837617 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2ec4003d00 con 0x55af7ddf69f0 2024-03-23T23:05:16.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.535+0000 7f2edda49740 1 -- 172.21.15.12:0/2487837617 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2ec402cb20 con 0x55af7ddf69f0 2024-03-23T23:05:16.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.536+0000 7f2ebe000640 1 -- 172.21.15.12:0/2487837617 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ec403f440 con 0x55af7ddf69f0 2024-03-23T23:05:16.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.536+0000 7f2edda49740 10 client.26225 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:16.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.536+0000 7f2edda49740 1 -- 172.21.15.12:0/2487837617 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55af7db397d0 con 0x55af7ddf69f0 2024-03-23T23:05:16.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.536+0000 7f2ebe000640 1 -- 172.21.15.12:0/2487837617 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ec402cb20 con 0x55af7ddf69f0 2024-03-23T23:05:16.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.536+0000 7f2ebe000640 1 -- 172.21.15.12:0/2487837617 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f2ec40346a0 con 0x55af7ddf69f0 2024-03-23T23:05:16.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.536+0000 7f2edda49740 10 client.26225 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:16.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.536+0000 7f2edda49740 10 client.26225 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:16.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.536+0000 7f2edda49740 20 client.26225 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:16.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.536+0000 7f2edda49740 1 --2- 172.21.15.12:0/2487837617 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55af7de0c310 0x55af7de0e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.536+0000 7f2edda49740 4 client.26225 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:05:16.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.536+0000 7f2edda49740 1 -- 172.21.15.12:0/2487837617 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55af7dd68800 con 0x55af7de0c310 2024-03-23T23:05:16.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.536+0000 7f2ed3400640 1 --2- 172.21.15.12:0/2487837617 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55af7de0c310 0x55af7de0e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.537+0000 7f2ed3400640 1 --2- 172.21.15.12:0/2487837617 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55af7de0c310 0x55af7de0e700 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16151e8330b874d1 server_cookie=7528baadb54ec0b8 in_seq=0 out_seq=0 2024-03-23T23:05:16.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.537+0000 7f2ebe000640 10 client.26225 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:16.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.538+0000 7f2ebe000640 1 -- 172.21.15.12:0/2487837617 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55af7dd68800 con 0x55af7de0c310 2024-03-23T23:05:16.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.538+0000 7f2ebe000640 10 client.26225 handle_command_reply: tid=0 2024-03-23T23:05:16.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.538+0000 7f2edda49740 1 client.26225 shutdown 2024-03-23T23:05:16.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.538+0000 7f2edda49740 1 -- 172.21.15.12:0/2487837617 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af7ddf69f0 msgr2=0x55af7de00c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.538+0000 7f2edda49740 1 --2- 172.21.15.12:0/2487837617 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af7ddf69f0 0x55af7de00c60 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4006010 tx=0x7f2ec40095e0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.538+0000 7f2edda49740 1 -- 172.21.15.12:0/2487837617 shutdown_connections 2024-03-23T23:05:16.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.538+0000 7f2edda49740 1 --2- 172.21.15.12:0/2487837617 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55af7de0c310 0x55af7de0e700 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.538+0000 7f2edda49740 1 --2- 172.21.15.12:0/2487837617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af7ddf8580 0x55af7de08190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.538+0000 7f2edda49740 1 --2- 172.21.15.12:0/2487837617 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55af7ddf7bc0 0x55af7de011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.538+0000 7f2ebe000640 0 client.26225 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:16.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.538+0000 7f2edda49740 1 --2- 172.21.15.12:0/2487837617 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af7ddf69f0 0x55af7de00c60 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.538+0000 7f2edda49740 1 -- 172.21.15.12:0/2487837617 >> 172.21.15.12:0/2487837617 conn(0x55af7ddf1de0 msgr2=0x55af7ddf1160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:16.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.539+0000 7f2edda49740 1 -- 172.21.15.12:0/2487837617 shutdown_connections 2024-03-23T23:05:16.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.539+0000 7f2edda49740 1 -- 172.21.15.12:0/2487837617 wait complete. 2024-03-23T23:05:16.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.539+0000 7f2edda49740 20 client.26225 trim_cache size 0 max 16384 2024-03-23T23:05:16.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.552+0000 7f2edda49740 1 Processor -- start 2024-03-23T23:05:16.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.552+0000 7f2edda49740 1 -- start start 2024-03-23T23:05:16.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.553+0000 7f2edda49740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55af7ddf69f0 0x55af7ddfac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.553+0000 7f2edda49740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af7ddf7bc0 0x55af7de08150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.553+0000 7f2edda49740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af7ddf8580 0x55af7ddfb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.553+0000 7f2edda49740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed41b1390 con 0x55af7ddf69f0 2024-03-23T23:05:16.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.553+0000 7f2edda49740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed41aee90 con 0x55af7ddf8580 2024-03-23T23:05:16.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.553+0000 7f2edda49740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed41acf30 con 0x55af7ddf7bc0 2024-03-23T23:05:16.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.553+0000 7f2ed3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af7ddf7bc0 0x55af7de08150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.553+0000 7f2ed8e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af7ddf8580 0x55af7ddfb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.553+0000 7f2ed3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55af7ddf69f0 0x55af7ddfac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.553+0000 7f2ed3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af7ddf7bc0 0x55af7de08150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:59024/0 (socket says 172.21.15.12:59024) 2024-03-23T23:05:16.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.553+0000 7f2ed8e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af7ddf8580 0x55af7ddfb1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59610/0 (socket says 172.21.15.12:59610) 2024-03-23T23:05:16.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.553+0000 7f2ed3400640 1 -- 172.21.15.12:0/1778393152 learned_addr learned my addr 172.21.15.12:0/1778393152 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:16.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.554+0000 7f2ed3400640 1 -- 172.21.15.12:0/1778393152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af7ddf8580 msgr2=0x55af7ddfb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:16.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.554+0000 7f2ed3400640 1 --2- 172.21.15.12:0/1778393152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af7ddf8580 0x55af7ddfb1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.554+0000 7f2ed3400640 1 -- 172.21.15.12:0/1778393152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55af7ddf69f0 msgr2=0x55af7ddfac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:16.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.554+0000 7f2ed3400640 1 --2- 172.21.15.12:0/1778393152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55af7ddf69f0 0x55af7ddfac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.554+0000 7f2ed3400640 1 -- 172.21.15.12:0/1778393152 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af7ddf7640 con 0x55af7ddf7bc0 2024-03-23T23:05:16.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.554+0000 7f2ed3400640 1 --2- 172.21.15.12:0/1778393152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af7ddf7bc0 0x55af7de08150 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2ec00cce40 tx=0x7f2ec0086040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=286a0c877311a26c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:16.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.554+0000 7f2ec8c00640 1 -- 172.21.15.12:0/1778393152 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec009b970 con 0x55af7ddf7bc0 2024-03-23T23:05:16.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.554+0000 7f2ec8c00640 1 -- 172.21.15.12:0/1778393152 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ec009fac0 con 0x55af7ddf7bc0 2024-03-23T23:05:16.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.554+0000 7f2ec8c00640 1 -- 172.21.15.12:0/1778393152 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec0097fe0 con 0x55af7ddf7bc0 2024-03-23T23:05:16.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.555+0000 7f2edda49740 1 -- 172.21.15.12:0/1778393152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af7ddf7bc0 msgr2=0x55af7de08150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.555+0000 7f2edda49740 1 --2- 172.21.15.12:0/1778393152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af7ddf7bc0 0x55af7de08150 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2ec00cce40 tx=0x7f2ec0086040 comp rx=0 tx=0).stop 2024-03-23T23:05:16.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.555+0000 7f2edda49740 1 -- 172.21.15.12:0/1778393152 shutdown_connections 2024-03-23T23:05:16.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.555+0000 7f2edda49740 1 --2- 172.21.15.12:0/1778393152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af7ddf8580 0x55af7ddfb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.555+0000 7f2edda49740 1 --2- 172.21.15.12:0/1778393152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af7ddf7bc0 0x55af7de08150 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.555+0000 7f2edda49740 1 --2- 172.21.15.12:0/1778393152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55af7ddf69f0 0x55af7ddfac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.555+0000 7f2edda49740 1 -- 172.21.15.12:0/1778393152 >> 172.21.15.12:0/1778393152 conn(0x55af7ddf1de0 msgr2=0x55af7ddf58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:16.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.555+0000 7f2edda49740 1 -- 172.21.15.12:0/1778393152 shutdown_connections 2024-03-23T23:05:16.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.555+0000 7f2edda49740 1 -- 172.21.15.12:0/1778393152 wait complete. 2024-03-23T23:05:16.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.555+0000 7f2edda49740 1 -- start start 2024-03-23T23:05:16.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.556+0000 7f2edda49740 1 Processor -- start 2024-03-23T23:05:16.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.556+0000 7f2edda49740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af7ddf69f0 0x55af7de04380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.556+0000 7f2edda49740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55af7ddf7bc0 0x55af7de048c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.556+0000 7f2edda49740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af7ddf8580 0x55af7de07820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.556+0000 7f2edda49740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed41ad530 con 0x55af7ddf7bc0 2024-03-23T23:05:16.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.556+0000 7f2edda49740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed41ae830 con 0x55af7ddf8580 2024-03-23T23:05:16.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.556+0000 7f2edda49740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed41adb70 con 0x55af7ddf69f0 2024-03-23T23:05:16.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.556+0000 7f2edda49740 10 client.0 _ll_register_callbacks cb 0x7f2ecc004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:16.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.556+0000 7f2ed3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af7ddf69f0 0x55af7de04380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.556+0000 7f2ed8e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af7ddf8580 0x55af7de07820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.556+0000 7f2ed3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af7ddf69f0 0x55af7de04380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:59030/0 (socket says 172.21.15.12:59030) 2024-03-23T23:05:16.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.556+0000 7f2ed3400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55af7ddf7bc0 0x55af7de048c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.556+0000 7f2ed3e00640 1 -- 172.21.15.12:0/2255604043 learned_addr learned my addr 172.21.15.12:0/2255604043 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:16.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.557+0000 7f2ed3e00640 1 -- 172.21.15.12:0/2255604043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af7ddf8580 msgr2=0x55af7de07820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:16.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.557+0000 7f2ed3e00640 1 --2- 172.21.15.12:0/2255604043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af7ddf8580 0x55af7de07820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.557+0000 7f2ed3e00640 1 -- 172.21.15.12:0/2255604043 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55af7ddf7bc0 msgr2=0x55af7de048c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:16.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.557+0000 7f2ed3e00640 1 --2- 172.21.15.12:0/2255604043 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55af7ddf7bc0 0x55af7de048c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.557+0000 7f2ed3e00640 1 -- 172.21.15.12:0/2255604043 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55af7de07e10 con 0x55af7ddf69f0 2024-03-23T23:05:16.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.557+0000 7f2ed3e00640 1 --2- 172.21.15.12:0/2255604043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af7ddf69f0 0x55af7de04380 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4004650 tx=0x7f2ec4005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc86edd24ab6e90b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:16.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.557+0000 7f2ebe000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:05:16.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.557+0000 7f2ebe000640 1 -- 172.21.15.12:0/2255604043 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec4058070 con 0x55af7ddf69f0 2024-03-23T23:05:16.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.557+0000 7f2edda49740 1 -- 172.21.15.12:0/2255604043 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2ec009fac0 con 0x55af7ddf69f0 2024-03-23T23:05:16.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.557+0000 7f2ebe000640 1 -- 172.21.15.12:0/2255604043 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2ec4034440 con 0x55af7ddf69f0 2024-03-23T23:05:16.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.557+0000 7f2ebe000640 1 -- 172.21.15.12:0/2255604043 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ec4034640 con 0x55af7ddf69f0 2024-03-23T23:05:16.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.557+0000 7f2edda49740 10 client.26197 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:16.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.557+0000 7f2edda49740 1 -- 172.21.15.12:0/2255604043 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55af7de684b0 con 0x55af7ddf69f0 2024-03-23T23:05:16.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.558+0000 7f2ebe000640 1 -- 172.21.15.12:0/2255604043 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ec009fac0 con 0x55af7ddf69f0 2024-03-23T23:05:16.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.558+0000 7f2ebe000640 1 -- 172.21.15.12:0/2255604043 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f2ec4068430 con 0x55af7ddf69f0 2024-03-23T23:05:16.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.558+0000 7f2edda49740 10 client.26197 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:16.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.558+0000 7f2edda49740 10 client.26197 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:16.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.558+0000 7f2edda49740 20 client.26197 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:16.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.558+0000 7f2edda49740 1 --2- 172.21.15.12:0/2255604043 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55af7de68d70 0x55af7de6b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:16.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.558+0000 7f2edda49740 4 client.26197 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:05:16.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.558+0000 7f2edda49740 1 -- 172.21.15.12:0/2255604043 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55af7dd62b50 con 0x55af7de68d70 2024-03-23T23:05:16.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.558+0000 7f2ed3400640 1 --2- 172.21.15.12:0/2255604043 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55af7de68d70 0x55af7de6b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:16.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.559+0000 7f2ed3400640 1 --2- 172.21.15.12:0/2255604043 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55af7de68d70 0x55af7de6b190 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6392646b31b5764f server_cookie=e9a7000a7f4f2457 in_seq=0 out_seq=0 2024-03-23T23:05:16.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.559+0000 7f2ebe000640 10 client.26197 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:16.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.559+0000 7f2ebe000640 1 -- 172.21.15.12:0/2255604043 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55af7dd62b50 con 0x55af7de68d70 2024-03-23T23:05:16.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.559+0000 7f2ebe000640 10 client.26197 handle_command_reply: tid=0 2024-03-23T23:05:16.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.559+0000 7f2edda49740 1 client.26197 shutdown 2024-03-23T23:05:16.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.559+0000 7f2edda49740 1 -- 172.21.15.12:0/2255604043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af7ddf69f0 msgr2=0x55af7de04380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.559+0000 7f2edda49740 1 --2- 172.21.15.12:0/2255604043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af7ddf69f0 0x55af7de04380 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4004650 tx=0x7f2ec4005dd0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.559+0000 7f2edda49740 1 -- 172.21.15.12:0/2255604043 shutdown_connections 2024-03-23T23:05:16.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.559+0000 7f2edda49740 1 --2- 172.21.15.12:0/2255604043 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55af7de68d70 0x55af7de6b190 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.559+0000 7f2edda49740 1 --2- 172.21.15.12:0/2255604043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af7ddf8580 0x55af7de07820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.559+0000 7f2edda49740 1 --2- 172.21.15.12:0/2255604043 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55af7ddf7bc0 0x55af7de048c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.559+0000 7f2edda49740 1 --2- 172.21.15.12:0/2255604043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af7ddf69f0 0x55af7de04380 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.559+0000 7f2ebe000640 0 client.26197 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:16.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.559+0000 7f2edda49740 1 -- 172.21.15.12:0/2255604043 >> 172.21.15.12:0/2255604043 conn(0x55af7ddf1de0 msgr2=0x55af7ddf08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:16.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.559+0000 7f2edda49740 1 -- 172.21.15.12:0/2255604043 shutdown_connections 2024-03-23T23:05:16.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.559+0000 7f2edda49740 1 -- 172.21.15.12:0/2255604043 wait complete. 2024-03-23T23:05:16.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.559+0000 7f2edda49740 20 client.26197 trim_cache size 0 max 16384 2024-03-23T23:05:16.563 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:05:16.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.560+0000 7f2ed9800640 1 -- 172.21.15.12:0/3860389526 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2eb80a2790 msgr2=0x7f2eb80a4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.560+0000 7f2ed9800640 1 --2- 172.21.15.12:0/3860389526 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2eb80a2790 0x7f2eb80a4c50 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2ec402c7b0 tx=0x7f2ec402bc30 comp rx=0 tx=0).stop 2024-03-23T23:05:16.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.560+0000 7f2ed9800640 1 -- 172.21.15.12:0/3860389526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ed4108db0 msgr2=0x7f2ed419e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:16.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.560+0000 7f2ed9800640 1 --2- 172.21.15.12:0/3860389526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ed4108db0 0x7f2ed419e460 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2ec000eb90 tx=0x7f2ec000c6a0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.560+0000 7f2ed9800640 1 -- 172.21.15.12:0/3860389526 shutdown_connections 2024-03-23T23:05:16.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.560+0000 7f2ed9800640 1 --2- 172.21.15.12:0/3860389526 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2eb80a2790 0x7f2eb80a4c50 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.560+0000 7f2ed9800640 1 --2- 172.21.15.12:0/3860389526 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ed4109770 0x7f2ed41a54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.560+0000 7f2ed9800640 1 --2- 172.21.15.12:0/3860389526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ed4108db0 0x7f2ed419e460 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.560+0000 7f2ed9800640 1 --2- 172.21.15.12:0/3860389526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ed4107bc0 0x7f2ed419df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:16.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.560+0000 7f2ed9800640 1 -- 172.21.15.12:0/3860389526 >> 172.21.15.12:0/3860389526 conn(0x7f2ed4076b70 msgr2=0x7f2ed410c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:16.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.560+0000 7f2ed9800640 1 -- 172.21.15.12:0/3860389526 shutdown_connections 2024-03-23T23:05:16.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:16.560+0000 7f2ed9800640 1 -- 172.21.15.12:0/3860389526 wait complete. 2024-03-23T23:05:16.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:16 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/621819747' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:05:17.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:17 smithi203 ceph-mon[36693]: pgmap v494: 97 pgs: 97 active+clean; 1.9 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:05:17.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:17 smithi012 ceph-mon[31695]: pgmap v494: 97 pgs: 97 active+clean; 1.9 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:05:17.578 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-23T23:05:17.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:05:17.579 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-23T23:05:17.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:17 smithi173 ceph-mon[36116]: pgmap v494: 97 pgs: 97 active+clean; 1.9 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:05:17.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.817+0000 7f9b83200640 1 -- 172.21.15.12:0/1930548114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b84102000 msgr2=0x7f9b841043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:17.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.817+0000 7f9b83200640 1 --2- 172.21.15.12:0/1930548114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b84102000 0x7f9b841043f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f9b74009900 tx=0x7f9b7402bc30 comp rx=0 tx=0).stop 2024-03-23T23:05:17.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.817+0000 7f9b83200640 1 -- 172.21.15.12:0/1930548114 shutdown_connections 2024-03-23T23:05:17.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.817+0000 7f9b83200640 1 --2- 172.21.15.12:0/1930548114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b84107260 0x7f9b841096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.817+0000 7f9b83200640 1 --2- 172.21.15.12:0/1930548114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b84104930 0x7f9b84106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.817+0000 7f9b83200640 1 --2- 172.21.15.12:0/1930548114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b84102000 0x7f9b841043f0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.817+0000 7f9b83200640 1 -- 172.21.15.12:0/1930548114 >> 172.21.15.12:0/1930548114 conn(0x7f9b840fba10 msgr2=0x7f9b840fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:17.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.817+0000 7f9b83200640 1 -- 172.21.15.12:0/1930548114 shutdown_connections 2024-03-23T23:05:17.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.817+0000 7f9b83200640 1 -- 172.21.15.12:0/1930548114 wait complete. 2024-03-23T23:05:17.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.818+0000 7f9b83200640 1 Processor -- start 2024-03-23T23:05:17.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.819+0000 7f9b83200640 1 -- start start 2024-03-23T23:05:17.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.819+0000 7f9b83200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b84102000 0x7f9b84199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:17.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.819+0000 7f9b83200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b84104930 0x7f9b8419a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:17.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.819+0000 7f9b83200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b84107260 0x7f9b841a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:17.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.819+0000 7f9b83200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8410d260 con 0x7f9b84104930 2024-03-23T23:05:17.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.819+0000 7f9b83200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8410d0e0 con 0x7f9b84107260 2024-03-23T23:05:17.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.819+0000 7f9b83200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8410d3e0 con 0x7f9b84102000 2024-03-23T23:05:17.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.819+0000 7f9b81e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b84102000 0x7f9b84199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:17.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.819+0000 7f9b81e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b84102000 0x7f9b84199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:59048/0 (socket says 172.21.15.12:59048) 2024-03-23T23:05:17.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.819+0000 7f9b81e00640 1 -- 172.21.15.12:0/291908565 learned_addr learned my addr 172.21.15.12:0/291908565 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:17.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.819+0000 7f9b81400640 1 --2- 172.21.15.12:0/291908565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b84104930 0x7f9b8419a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:17.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.819+0000 7f9b82800640 1 --2- 172.21.15.12:0/291908565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b84107260 0x7f9b841a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:17.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.820+0000 7f9b81e00640 1 -- 172.21.15.12:0/291908565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b84107260 msgr2=0x7f9b841a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:17.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.820+0000 7f9b81e00640 1 --2- 172.21.15.12:0/291908565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b84107260 0x7f9b841a1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.820+0000 7f9b81e00640 1 -- 172.21.15.12:0/291908565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b84104930 msgr2=0x7f9b8419a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:17.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.820+0000 7f9b81e00640 1 --2- 172.21.15.12:0/291908565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b84104930 0x7f9b8419a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.820+0000 7f9b81e00640 1 -- 172.21.15.12:0/291908565 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b841a16f0 con 0x7f9b84102000 2024-03-23T23:05:17.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.820+0000 7f9b82800640 1 --2- 172.21.15.12:0/291908565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b84107260 0x7f9b841a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:05:17.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.820+0000 7f9b81e00640 1 --2- 172.21.15.12:0/291908565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b84102000 0x7f9b84199ad0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9b74005ab0 tx=0x7f9b7402c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:17.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.820+0000 7f9b7aa00640 1 -- 172.21.15.12:0/291908565 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b74043020 con 0x7f9b84102000 2024-03-23T23:05:17.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.820+0000 7f9b7aa00640 1 -- 172.21.15.12:0/291908565 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b740054e0 con 0x7f9b84102000 2024-03-23T23:05:17.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.820+0000 7f9b83200640 1 -- 172.21.15.12:0/291908565 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b841a1980 con 0x7f9b84102000 2024-03-23T23:05:17.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.820+0000 7f9b7aa00640 1 -- 172.21.15.12:0/291908565 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b74003ce0 con 0x7f9b84102000 2024-03-23T23:05:17.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.821+0000 7f9b83200640 1 -- 172.21.15.12:0/291908565 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b841a1f20 con 0x7f9b84102000 2024-03-23T23:05:17.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.824+0000 7f9b7aa00640 1 -- 172.21.15.12:0/291908565 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9b74046020 con 0x7f9b84102000 2024-03-23T23:05:17.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.824+0000 7f9b8b7c1740 1 Processor -- start 2024-03-23T23:05:17.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.824+0000 7f9b8b7c1740 1 -- start start 2024-03-23T23:05:17.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.824+0000 7f9b8b7c1740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56231e1f72b0 0x56231e1f76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:17.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.824+0000 7f9b8b7c1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56231e1f7ca0 0x56231e1fc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:17.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.824+0000 7f9b8b7c1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56231e1fce10 0x56231e1ff2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:17.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.824+0000 7f9b8b7c1740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56231e15ceb0 con 0x56231e1f72b0 2024-03-23T23:05:17.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.824+0000 7f9b8b7c1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56231e105a60 con 0x56231e1fce10 2024-03-23T23:05:17.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.824+0000 7f9b8b7c1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x56231e158f50 con 0x56231e1f7ca0 2024-03-23T23:05:17.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.825+0000 7f9b81400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56231e1fce10 0x56231e1ff2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:17.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.825+0000 7f9b81400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56231e1fce10 0x56231e1ff2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59646/0 (socket says 172.21.15.12:59646) 2024-03-23T23:05:17.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.825+0000 7f9b81e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56231e1f7ca0 0x56231e1fc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:17.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.825+0000 7f9b81400640 1 -- 172.21.15.12:0/2582905935 learned_addr learned my addr 172.21.15.12:0/2582905935 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:17.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.825+0000 7f9b82800640 1 --2- 172.21.15.12:0/2582905935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56231e1f72b0 0x56231e1f76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:17.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.825+0000 7f9b81400640 1 -- 172.21.15.12:0/2582905935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56231e1f7ca0 msgr2=0x56231e1fc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:17.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.825+0000 7f9b81400640 1 --2- 172.21.15.12:0/2582905935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56231e1f7ca0 0x56231e1fc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.825+0000 7f9b81400640 1 -- 172.21.15.12:0/2582905935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56231e1f72b0 msgr2=0x56231e1f76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:17.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.825+0000 7f9b81400640 1 --2- 172.21.15.12:0/2582905935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56231e1f72b0 0x56231e1f76d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.825+0000 7f9b81400640 1 -- 172.21.15.12:0/2582905935 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56231df397e0 con 0x56231e1fce10 2024-03-23T23:05:17.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.825+0000 7f9b82800640 1 --2- 172.21.15.12:0/2582905935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56231e1f72b0 0x56231e1f76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:05:17.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.825+0000 7f9b81e00640 1 --2- 172.21.15.12:0/2582905935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56231e1f7ca0 0x56231e1fc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:17.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.826+0000 7f9b81400640 1 --2- 172.21.15.12:0/2582905935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56231e1fce10 0x56231e1ff2a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c009fc0 tx=0x7f9b6c010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33e60e7a39e99772 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:17.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.826+0000 7f9b7aa00640 1 --2- 172.21.15.12:0/291908565 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9b640a2880 0x7f9b640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:17.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.826+0000 7f9b73000640 1 -- 172.21.15.12:0/2582905935 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c004d20 con 0x56231e1fce10 2024-03-23T23:05:17.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.826+0000 7f9b73000640 1 -- 172.21.15.12:0/2582905935 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b6c003e50 con 0x56231e1fce10 2024-03-23T23:05:17.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.826+0000 7f9b73000640 1 -- 172.21.15.12:0/2582905935 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c011760 con 0x56231e1fce10 2024-03-23T23:05:17.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.826+0000 7f9b7aa00640 1 -- 172.21.15.12:0/291908565 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9b740f0cc0 con 0x7f9b84102000 2024-03-23T23:05:17.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.826+0000 7f9b82800640 1 --2- 172.21.15.12:0/291908565 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9b640a2880 0x7f9b640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:17.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.827+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/2582905935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56231e1fce10 msgr2=0x56231e1ff2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:17.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.827+0000 7f9b8b7c1740 1 --2- 172.21.15.12:0/2582905935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56231e1fce10 0x56231e1ff2a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c009fc0 tx=0x7f9b6c010860 comp rx=0 tx=0).stop 2024-03-23T23:05:17.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.827+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/2582905935 shutdown_connections 2024-03-23T23:05:17.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.827+0000 7f9b8b7c1740 1 --2- 172.21.15.12:0/2582905935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56231e1fce10 0x56231e1ff2a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.827+0000 7f9b8b7c1740 1 --2- 172.21.15.12:0/2582905935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56231e1f7ca0 0x56231e1fc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.827+0000 7f9b8b7c1740 1 --2- 172.21.15.12:0/2582905935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56231e1f72b0 0x56231e1f76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.827+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/2582905935 >> 172.21.15.12:0/2582905935 conn(0x56231e1f2820 msgr2=0x56231e1f2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:17.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.827+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/2582905935 shutdown_connections 2024-03-23T23:05:17.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.827+0000 7f9b82800640 1 --2- 172.21.15.12:0/291908565 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9b640a2880 0x7f9b640a4d40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x56231e1f8500 tx=0x7f9b7c005f30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:17.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.827+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/2582905935 wait complete. 2024-03-23T23:05:17.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.827+0000 7f9b8b7c1740 1 -- start start 2024-03-23T23:05:17.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.828+0000 7f9b8b7c1740 1 Processor -- start 2024-03-23T23:05:17.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.828+0000 7f9b8b7c1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56231e1f72b0 0x56231e201490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:17.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.828+0000 7f9b8b7c1740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56231e1f7ca0 0x56231e2019d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:17.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.828+0000 7f9b8b7c1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56231e1fce10 0x56231e208a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:17.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.828+0000 7f9b8b7c1740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56231e1ffdc0 con 0x56231e1f7ca0 2024-03-23T23:05:17.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.828+0000 7f9b8b7c1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56231e118eb0 con 0x56231e1fce10 2024-03-23T23:05:17.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.828+0000 7f9b8b7c1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x56231e1fff40 con 0x56231e1f72b0 2024-03-23T23:05:17.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.828+0000 7f9b8b7c1740 10 client.0 _ll_register_callbacks cb 0x56231e10d150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:17.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.829+0000 7f9b82800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56231e1fce10 0x56231e208a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:17.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.829+0000 7f9b81400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56231e1f7ca0 0x56231e2019d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:17.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.829+0000 7f9b81e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56231e1f72b0 0x56231e201490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:17.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.829+0000 7f9b82800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56231e1fce10 0x56231e208a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59654/0 (socket says 172.21.15.12:59654) 2024-03-23T23:05:17.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.829+0000 7f9b82800640 1 -- 172.21.15.12:0/3838184132 learned_addr learned my addr 172.21.15.12:0/3838184132 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:17.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.829+0000 7f9b81e00640 1 -- 172.21.15.12:0/3838184132 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56231e1fce10 msgr2=0x56231e208a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:17.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.829+0000 7f9b81e00640 1 --2- 172.21.15.12:0/3838184132 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56231e1fce10 0x56231e208a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.829+0000 7f9b81e00640 1 -- 172.21.15.12:0/3838184132 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56231e1f7ca0 msgr2=0x56231e2019d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.829+0000 7f9b81e00640 1 --2- 172.21.15.12:0/3838184132 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56231e1f7ca0 0x56231e2019d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.829+0000 7f9b81e00640 1 -- 172.21.15.12:0/3838184132 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56231df99f70 con 0x56231e1f72b0 2024-03-23T23:05:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.829+0000 7f9b81e00640 1 --2- 172.21.15.12:0/3838184132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56231e1f72b0 0x56231e201490 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x56231e1f8710 tx=0x7f9b740b92b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6809b272891fdf8f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.829+0000 7f9b6b400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:05:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.829+0000 7f9b6b400640 1 -- 172.21.15.12:0/3838184132 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b740bd3e0 con 0x56231e1f72b0 2024-03-23T23:05:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.830+0000 7f9b6b400640 1 -- 172.21.15.12:0/3838184132 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9b740ee880 con 0x56231e1f72b0 2024-03-23T23:05:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.830+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/3838184132 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9b6c003e50 con 0x56231e1f72b0 2024-03-23T23:05:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.830+0000 7f9b6b400640 1 -- 172.21.15.12:0/3838184132 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b740bcf70 con 0x56231e1f72b0 2024-03-23T23:05:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.830+0000 7f9b8b7c1740 10 client.26209 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.830+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/3838184132 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56231de31170 con 0x56231e1f72b0 2024-03-23T23:05:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.830+0000 7f9b6b400640 1 -- 172.21.15.12:0/3838184132 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b6c003e50 con 0x56231e1f72b0 2024-03-23T23:05:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.830+0000 7f9b6b400640 1 -- 172.21.15.12:0/3838184132 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f9b740bd870 con 0x56231e1f72b0 2024-03-23T23:05:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.830+0000 7f9b8b7c1740 10 client.26209 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:17.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.830+0000 7f9b8b7c1740 10 client.26209 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:17.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.830+0000 7f9b8b7c1740 20 client.26209 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:17.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.830+0000 7f9b8b7c1740 1 --2- 172.21.15.12:0/3838184132 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56231e20d2e0 0x56231e20f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:17.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.830+0000 7f9b8b7c1740 4 client.26209 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:05:17.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.830+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/3838184132 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56231e16a3f0 con 0x56231e20d2e0 2024-03-23T23:05:17.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.831+0000 7f9b81400640 1 --2- 172.21.15.12:0/3838184132 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56231e20d2e0 0x56231e20f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:17.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.831+0000 7f9b81400640 1 --2- 172.21.15.12:0/3838184132 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56231e20d2e0 0x56231e20f750 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6979a5643ccdaf57 server_cookie=6d4d59fcad69f0d9 in_seq=0 out_seq=0 2024-03-23T23:05:17.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.831+0000 7f9b6b400640 10 client.26209 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:17.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.832+0000 7f9b6b400640 1 -- 172.21.15.12:0/3838184132 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56231e16a3f0 con 0x56231e20d2e0 2024-03-23T23:05:17.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.832+0000 7f9b6b400640 10 client.26209 handle_command_reply: tid=0 2024-03-23T23:05:17.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.832+0000 7f9b8b7c1740 1 client.26209 shutdown 2024-03-23T23:05:17.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.832+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/3838184132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56231e1f72b0 msgr2=0x56231e201490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:17.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.832+0000 7f9b8b7c1740 1 --2- 172.21.15.12:0/3838184132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56231e1f72b0 0x56231e201490 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x56231e1f8710 tx=0x7f9b740b92b0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.832+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/3838184132 shutdown_connections 2024-03-23T23:05:17.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.832+0000 7f9b8b7c1740 1 --2- 172.21.15.12:0/3838184132 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56231e20d2e0 0x56231e20f750 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.832+0000 7f9b8b7c1740 1 --2- 172.21.15.12:0/3838184132 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56231e1fce10 0x56231e208a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.832+0000 7f9b8b7c1740 1 --2- 172.21.15.12:0/3838184132 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56231e1f7ca0 0x56231e2019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.832+0000 7f9b8b7c1740 1 --2- 172.21.15.12:0/3838184132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56231e1f72b0 0x56231e201490 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.832+0000 7f9b6b400640 0 client.26209 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:17.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.832+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/3838184132 >> 172.21.15.12:0/3838184132 conn(0x56231e1f2820 msgr2=0x56231e1f1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:17.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.833+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/3838184132 shutdown_connections 2024-03-23T23:05:17.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.833+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/3838184132 wait complete. 2024-03-23T23:05:17.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.833+0000 7f9b8b7c1740 20 client.26209 trim_cache size 0 max 16384 2024-03-23T23:05:17.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.850+0000 7f9b8b7c1740 1 Processor -- start 2024-03-23T23:05:17.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.851+0000 7f9b8b7c1740 1 -- start start 2024-03-23T23:05:17.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.851+0000 7f9b8b7c1740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56231e1f72b0 0x56231e1f6190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:17.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.851+0000 7f9b8b7c1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56231e1f7ca0 0x56231e2089a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:17.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.851+0000 7f9b8b7c1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56231e1fce10 0x56231e213fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:17.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.851+0000 7f9b8b7c1740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b841acec0 con 0x56231e1f72b0 2024-03-23T23:05:17.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.851+0000 7f9b8b7c1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b841aa9c0 con 0x56231e1fce10 2024-03-23T23:05:17.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.851+0000 7f9b8b7c1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b841a8a60 con 0x56231e1f7ca0 2024-03-23T23:05:17.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.851+0000 7f9b81e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56231e1f72b0 0x56231e1f6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:17.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.851+0000 7f9b82800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56231e1fce10 0x56231e213fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:17.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.851+0000 7f9b81e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56231e1f72b0 0x56231e1f6190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40928/0 (socket says 172.21.15.12:40928) 2024-03-23T23:05:17.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.851+0000 7f9b81e00640 1 -- 172.21.15.12:0/938252738 learned_addr learned my addr 172.21.15.12:0/938252738 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:17.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.851+0000 7f9b81e00640 1 -- 172.21.15.12:0/938252738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56231e1f7ca0 msgr2=0x56231e2089a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:17.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.851+0000 7f9b81e00640 1 --2- 172.21.15.12:0/938252738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56231e1f7ca0 0x56231e2089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.851+0000 7f9b81e00640 1 -- 172.21.15.12:0/938252738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56231e1fce10 msgr2=0x56231e213fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:17.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.851+0000 7f9b81e00640 1 --2- 172.21.15.12:0/938252738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56231e1fce10 0x56231e213fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.851+0000 7f9b81e00640 1 -- 172.21.15.12:0/938252738 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56231e1f8530 con 0x56231e1f72b0 2024-03-23T23:05:17.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.852+0000 7f9b81e00640 1 --2- 172.21.15.12:0/938252738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56231e1f72b0 0x56231e1f6190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9b740b9a70 tx=0x7f9b740bd790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c82eda6f57554ee9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:17.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.852+0000 7f9b72600640 1 -- 172.21.15.12:0/938252738 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b740bd150 con 0x56231e1f72b0 2024-03-23T23:05:17.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.852+0000 7f9b72600640 1 -- 172.21.15.12:0/938252738 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b740f4020 con 0x56231e1f72b0 2024-03-23T23:05:17.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.852+0000 7f9b72600640 1 -- 172.21.15.12:0/938252738 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b740044a0 con 0x56231e1f72b0 2024-03-23T23:05:17.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.853+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/938252738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56231e1f72b0 msgr2=0x56231e1f6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:17.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.853+0000 7f9b8b7c1740 1 --2- 172.21.15.12:0/938252738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56231e1f72b0 0x56231e1f6190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9b740b9a70 tx=0x7f9b740bd790 comp rx=0 tx=0).stop 2024-03-23T23:05:17.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.853+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/938252738 shutdown_connections 2024-03-23T23:05:17.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.853+0000 7f9b8b7c1740 1 --2- 172.21.15.12:0/938252738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56231e1fce10 0x56231e213fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.853+0000 7f9b8b7c1740 1 --2- 172.21.15.12:0/938252738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56231e1f7ca0 0x56231e2089a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.853+0000 7f9b8b7c1740 1 --2- 172.21.15.12:0/938252738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56231e1f72b0 0x56231e1f6190 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.853+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/938252738 >> 172.21.15.12:0/938252738 conn(0x56231e1f2820 msgr2=0x56231e1f5a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:17.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.853+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/938252738 shutdown_connections 2024-03-23T23:05:17.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.853+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/938252738 wait complete. 2024-03-23T23:05:17.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.853+0000 7f9b8b7c1740 1 -- start start 2024-03-23T23:05:17.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.853+0000 7f9b8b7c1740 1 Processor -- start 2024-03-23T23:05:17.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.854+0000 7f9b8b7c1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56231e1f72b0 0x56231e2078a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:17.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.854+0000 7f9b8b7c1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56231e1f7ca0 0x56231e207de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:17.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.854+0000 7f9b8b7c1740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56231e1fce10 0x56231e271390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:17.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.854+0000 7f9b8b7c1740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b841a9060 con 0x56231e1fce10 2024-03-23T23:05:17.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.854+0000 7f9b8b7c1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b841aa360 con 0x56231e1f72b0 2024-03-23T23:05:17.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.854+0000 7f9b8b7c1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b841a96a0 con 0x56231e1f7ca0 2024-03-23T23:05:17.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.854+0000 7f9b8b7c1740 10 client.0 _ll_register_callbacks cb 0x7f9b74005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:17.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.854+0000 7f9b81400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56231e1f7ca0 0x56231e207de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:17.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.854+0000 7f9b81e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56231e1f72b0 0x56231e2078a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:17.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.854+0000 7f9b82800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56231e1fce10 0x56231e271390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:17.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.854+0000 7f9b81400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56231e1f7ca0 0x56231e207de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:59086/0 (socket says 172.21.15.12:59086) 2024-03-23T23:05:17.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.854+0000 7f9b81400640 1 -- 172.21.15.12:0/1853772671 learned_addr learned my addr 172.21.15.12:0/1853772671 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:17.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.854+0000 7f9b81400640 1 -- 172.21.15.12:0/1853772671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56231e1f72b0 msgr2=0x56231e2078a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:17.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.854+0000 7f9b81400640 1 --2- 172.21.15.12:0/1853772671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56231e1f72b0 0x56231e2078a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.854+0000 7f9b81400640 1 -- 172.21.15.12:0/1853772671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56231e1fce10 msgr2=0x56231e271390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:17.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.854+0000 7f9b81400640 1 --2- 172.21.15.12:0/1853772671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56231e1fce10 0x56231e271390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.855+0000 7f9b81400640 1 -- 172.21.15.12:0/1853772671 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56231e2718d0 con 0x56231e1f7ca0 2024-03-23T23:05:17.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.855+0000 7f9b81400640 1 --2- 172.21.15.12:0/1853772671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56231e1f7ca0 0x56231e207de0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x56231e1f3790 tx=0x7f9b6c009be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab61c339112b5676 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:17.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.855+0000 7f9b6b400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:05:17.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.855+0000 7f9b6b400640 1 -- 172.21.15.12:0/1853772671 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c030060 con 0x56231e1f7ca0 2024-03-23T23:05:17.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.855+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/1853772671 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9b740f4020 con 0x56231e1f7ca0 2024-03-23T23:05:17.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.855+0000 7f9b6b400640 1 -- 172.21.15.12:0/1853772671 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9b6c004bd0 con 0x56231e1f7ca0 2024-03-23T23:05:17.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.855+0000 7f9b6b400640 1 -- 172.21.15.12:0/1853772671 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b6c004dd0 con 0x56231e1f7ca0 2024-03-23T23:05:17.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.855+0000 7f9b8b7c1740 10 client.26212 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:17.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.855+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/1853772671 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56231e272ad0 con 0x56231e1f7ca0 2024-03-23T23:05:17.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.855+0000 7f9b6b400640 1 -- 172.21.15.12:0/1853772671 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b740f4020 con 0x56231e1f7ca0 2024-03-23T23:05:17.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.856+0000 7f9b6b400640 1 -- 172.21.15.12:0/1853772671 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f9b6c0118f0 con 0x56231e1f7ca0 2024-03-23T23:05:17.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.856+0000 7f9b8b7c1740 10 client.26212 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:17.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.856+0000 7f9b8b7c1740 10 client.26212 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:17.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.856+0000 7f9b8b7c1740 20 client.26212 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:17.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.856+0000 7f9b8b7c1740 1 --2- 172.21.15.12:0/1853772671 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56231e273150 0x56231e275570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:17.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.856+0000 7f9b8b7c1740 4 client.26212 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-23T23:05:17.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.856+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/1853772671 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56231e164740 con 0x56231e273150 2024-03-23T23:05:17.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.856+0000 7f9b81e00640 1 --2- 172.21.15.12:0/1853772671 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56231e273150 0x56231e275570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:17.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.856+0000 7f9b81e00640 1 --2- 172.21.15.12:0/1853772671 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56231e273150 0x56231e275570 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c52e6abe400d6fab server_cookie=a6e878d1b508d974 in_seq=0 out_seq=0 2024-03-23T23:05:17.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.856+0000 7f9b6b400640 10 client.26212 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:17.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.857+0000 7f9b6b400640 1 -- 172.21.15.12:0/1853772671 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56231e164740 con 0x56231e273150 2024-03-23T23:05:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.857+0000 7f9b6b400640 10 client.26212 handle_command_reply: tid=0 2024-03-23T23:05:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.857+0000 7f9b8b7c1740 1 client.26212 shutdown 2024-03-23T23:05:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.858+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/1853772671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56231e1f7ca0 msgr2=0x56231e207de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.858+0000 7f9b8b7c1740 1 --2- 172.21.15.12:0/1853772671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56231e1f7ca0 0x56231e207de0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x56231e1f3790 tx=0x7f9b6c009be0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.858+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/1853772671 shutdown_connections 2024-03-23T23:05:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.858+0000 7f9b8b7c1740 1 --2- 172.21.15.12:0/1853772671 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56231e273150 0x56231e275570 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.858+0000 7f9b8b7c1740 1 --2- 172.21.15.12:0/1853772671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56231e1fce10 0x56231e271390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.858+0000 7f9b8b7c1740 1 --2- 172.21.15.12:0/1853772671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56231e1f7ca0 0x56231e207de0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.858+0000 7f9b8b7c1740 1 --2- 172.21.15.12:0/1853772671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56231e1f72b0 0x56231e2078a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.858+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/1853772671 >> 172.21.15.12:0/1853772671 conn(0x56231e1f2820 msgr2=0x56231e1f6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.858+0000 7f9b6b400640 0 client.26212 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.858+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/1853772671 shutdown_connections 2024-03-23T23:05:17.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.858+0000 7f9b8b7c1740 1 -- 172.21.15.12:0/1853772671 wait complete. 2024-03-23T23:05:17.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.858+0000 7f9b8b7c1740 20 client.26212 trim_cache size 0 max 16384 2024-03-23T23:05:17.862 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:05:17.862 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-03-23T23:05:17.862 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "5532c8a8-d7f2-4301-ae93-370bee29f82a", 2024-03-23T23:05:17.862 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-03-23T23:05:17.862 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:05:17.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.859+0000 7f9b83200640 1 -- 172.21.15.12:0/291908565 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9b640a2880 msgr2=0x7f9b640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.859+0000 7f9b83200640 1 --2- 172.21.15.12:0/291908565 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9b640a2880 0x7f9b640a4d40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x56231e1f8500 tx=0x7f9b7c005f30 comp rx=0 tx=0).stop 2024-03-23T23:05:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.859+0000 7f9b83200640 1 -- 172.21.15.12:0/291908565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b84102000 msgr2=0x7f9b84199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.859+0000 7f9b83200640 1 --2- 172.21.15.12:0/291908565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b84102000 0x7f9b84199ad0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9b74005ab0 tx=0x7f9b7402c6c0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.859+0000 7f9b83200640 1 -- 172.21.15.12:0/291908565 shutdown_connections 2024-03-23T23:05:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.859+0000 7f9b83200640 1 --2- 172.21.15.12:0/291908565 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9b640a2880 0x7f9b640a4d40 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.859+0000 7f9b83200640 1 --2- 172.21.15.12:0/291908565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b84107260 0x7f9b841a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.859+0000 7f9b83200640 1 --2- 172.21.15.12:0/291908565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b84104930 0x7f9b8419a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.859+0000 7f9b83200640 1 --2- 172.21.15.12:0/291908565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b84102000 0x7f9b84199ad0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.859+0000 7f9b83200640 1 -- 172.21.15.12:0/291908565 >> 172.21.15.12:0/291908565 conn(0x7f9b840fba10 msgr2=0x7f9b84101700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.859+0000 7f9b83200640 1 -- 172.21.15.12:0/291908565 shutdown_connections 2024-03-23T23:05:17.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:17.859+0000 7f9b83200640 1 -- 172.21.15.12:0/291908565 wait complete. 2024-03-23T23:05:17.876 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:05:17.876 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:05:18.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.101+0000 7f4eb2e00640 1 -- 172.21.15.12:0/2329956027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4eac108db0 msgr2=0x7f4eac109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:18.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.101+0000 7f4eb2e00640 1 --2- 172.21.15.12:0/2329956027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4eac108db0 0x7f4eac109230 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4e98009860 tx=0x7f4e9802bb70 comp rx=0 tx=0).stop 2024-03-23T23:05:18.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.101+0000 7f4eb2e00640 1 -- 172.21.15.12:0/2329956027 shutdown_connections 2024-03-23T23:05:18.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.101+0000 7f4eb2e00640 1 --2- 172.21.15.12:0/2329956027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4eac109770 0x7f4eac110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.101+0000 7f4eb2e00640 1 --2- 172.21.15.12:0/2329956027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4eac108db0 0x7f4eac109230 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.101+0000 7f4eb2e00640 1 --2- 172.21.15.12:0/2329956027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4eac107bc0 0x7f4eac107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.101+0000 7f4eb2e00640 1 -- 172.21.15.12:0/2329956027 >> 172.21.15.12:0/2329956027 conn(0x7f4eac076b70 msgr2=0x7f4eac078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:18.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.102+0000 7f4eb2e00640 1 -- 172.21.15.12:0/2329956027 shutdown_connections 2024-03-23T23:05:18.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.102+0000 7f4eb2e00640 1 -- 172.21.15.12:0/2329956027 wait complete. 2024-03-23T23:05:18.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.103+0000 7f4eb2e00640 1 Processor -- start 2024-03-23T23:05:18.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.103+0000 7f4eb2e00640 1 -- start start 2024-03-23T23:05:18.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.103+0000 7f4eb2e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4eac107bc0 0x7f4eac19def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:18.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.103+0000 7f4eb2e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4eac108db0 0x7f4eac19e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:18.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.103+0000 7f4eb2e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4eac109770 0x7f4eac1a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:18.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.103+0000 7f4eb2e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac113820 con 0x7f4eac107bc0 2024-03-23T23:05:18.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.103+0000 7f4eb2e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac1136a0 con 0x7f4eac108db0 2024-03-23T23:05:18.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.103+0000 7f4eb2e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac1139a0 con 0x7f4eac109770 2024-03-23T23:05:18.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.103+0000 7f4eb1a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4eac107bc0 0x7f4eac19def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:18.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.103+0000 7f4eb2400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4eac109770 0x7f4eac1a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:18.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.103+0000 7f4eb1000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4eac108db0 0x7f4eac19e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:18.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.103+0000 7f4eb2400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4eac109770 0x7f4eac1a5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:59092/0 (socket says 172.21.15.12:59092) 2024-03-23T23:05:18.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.103+0000 7f4eb1000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4eac108db0 0x7f4eac19e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59676/0 (socket says 172.21.15.12:59676) 2024-03-23T23:05:18.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.103+0000 7f4eb2400640 1 -- 172.21.15.12:0/4086919151 learned_addr learned my addr 172.21.15.12:0/4086919151 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:18.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.104+0000 7f4eb2400640 1 -- 172.21.15.12:0/4086919151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4eac108db0 msgr2=0x7f4eac19e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:18.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.104+0000 7f4eb2400640 1 --2- 172.21.15.12:0/4086919151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4eac108db0 0x7f4eac19e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.104+0000 7f4eb2400640 1 -- 172.21.15.12:0/4086919151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4eac107bc0 msgr2=0x7f4eac19def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:18.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.104+0000 7f4eb2400640 1 --2- 172.21.15.12:0/4086919151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4eac107bc0 0x7f4eac19def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.104+0000 7f4eb2400640 1 -- 172.21.15.12:0/4086919151 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eac1a5a70 con 0x7f4eac109770 2024-03-23T23:05:18.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.104+0000 7f4eb1000640 1 --2- 172.21.15.12:0/4086919151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4eac108db0 0x7f4eac19e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:18.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.104+0000 7f4eb2400640 1 --2- 172.21.15.12:0/4086919151 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4eac109770 0x7f4eac1a5400 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4ea400eb10 tx=0x7f4ea400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:18.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.104+0000 7f4eaa000640 1 -- 172.21.15.12:0/4086919151 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea400ce20 con 0x7f4eac109770 2024-03-23T23:05:18.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.104+0000 7f4eaa000640 1 -- 172.21.15.12:0/4086919151 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ea4005020 con 0x7f4eac109770 2024-03-23T23:05:18.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.105+0000 7f4eb2e00640 1 -- 172.21.15.12:0/4086919151 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eac1a5d60 con 0x7f4eac109770 2024-03-23T23:05:18.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.105+0000 7f4eaa000640 1 -- 172.21.15.12:0/4086919151 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea4010690 con 0x7f4eac109770 2024-03-23T23:05:18.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.105+0000 7f4eb2e00640 1 -- 172.21.15.12:0/4086919151 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eac1a6220 con 0x7f4eac109770 2024-03-23T23:05:18.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.109+0000 7f4eb71cd740 1 Processor -- start 2024-03-23T23:05:18.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.109+0000 7f4eaa000640 1 -- 172.21.15.12:0/4086919151 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4ea4005540 con 0x7f4eac109770 2024-03-23T23:05:18.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.109+0000 7f4eb71cd740 1 -- start start 2024-03-23T23:05:18.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.110+0000 7f4eb71cd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56272b960050 0x56272b960470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:18.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.110+0000 7f4eb71cd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960a40 0x56272b965540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:18.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.110+0000 7f4eb71cd740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56272b965bb0 0x56272b968040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:18.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.110+0000 7f4eb71cd740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56272b8c5250 con 0x56272b965bb0 2024-03-23T23:05:18.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.110+0000 7f4eb71cd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56272b86da30 con 0x56272b960a40 2024-03-23T23:05:18.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.110+0000 7f4eb71cd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x56272b8c12f0 con 0x56272b960050 2024-03-23T23:05:18.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.110+0000 7f4eb1a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960a40 0x56272b965540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:18.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.110+0000 7f4eb1000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56272b960050 0x56272b960470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:18.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.110+0000 7f4eb1a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960a40 0x56272b965540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59688/0 (socket says 172.21.15.12:59688) 2024-03-23T23:05:18.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.110+0000 7f4eb1000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56272b960050 0x56272b960470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:59108/0 (socket says 172.21.15.12:59108) 2024-03-23T23:05:18.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.110+0000 7f4eb1a00640 1 -- 172.21.15.12:0/711598568 learned_addr learned my addr 172.21.15.12:0/711598568 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:18.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.110+0000 7f4eb1000640 1 --2- 172.21.15.12:0/711598568 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56272b965bb0 0x56272b968040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:18.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.111+0000 7f4eb1a00640 1 -- 172.21.15.12:0/711598568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56272b960050 msgr2=0x56272b960470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:18.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.111+0000 7f4eb1a00640 1 --2- 172.21.15.12:0/711598568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56272b960050 0x56272b960470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.111+0000 7f4eb1a00640 1 -- 172.21.15.12:0/711598568 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56272b965bb0 msgr2=0x56272b968040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:18.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.111+0000 7f4eb1a00640 1 --2- 172.21.15.12:0/711598568 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56272b965bb0 0x56272b968040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.111+0000 7f4eb1a00640 1 -- 172.21.15.12:0/711598568 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56272b858690 con 0x56272b960a40 2024-03-23T23:05:18.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.111+0000 7f4eb1000640 1 --2- 172.21.15.12:0/711598568 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56272b965bb0 0x56272b968040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:05:18.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.111+0000 7f4eaa000640 1 --2- 172.21.15.12:0/4086919151 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4e940a2880 0x7f4e940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:18.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.111+0000 7f4eb1a00640 1 --2- 172.21.15.12:0/711598568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960a40 0x56272b965540 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0009ca0 tx=0x7f4ea0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1226daeba1421ed8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:18.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.111+0000 7f4eaa000640 1 -- 172.21.15.12:0/4086919151 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4ea40cef90 con 0x7f4eac109770 2024-03-23T23:05:18.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.111+0000 7f4eb1000640 1 --2- 172.21.15.12:0/4086919151 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4e940a2880 0x7f4e940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:18.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.111+0000 7f4e9ea00640 1 -- 172.21.15.12:0/711598568 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0004230 con 0x56272b960a40 2024-03-23T23:05:18.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.111+0000 7f4e9ea00640 1 -- 172.21.15.12:0/711598568 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ea00043d0 con 0x56272b960a40 2024-03-23T23:05:18.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.111+0000 7f4e9ea00640 1 -- 172.21.15.12:0/711598568 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0004e50 con 0x56272b960a40 2024-03-23T23:05:18.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.112+0000 7f4eb1000640 1 --2- 172.21.15.12:0/4086919151 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4e940a2880 0x7f4e940a4d40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4e98009830 tx=0x7f4e9802e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:18.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.112+0000 7f4eb71cd740 1 -- 172.21.15.12:0/711598568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960a40 msgr2=0x56272b965540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:18.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.112+0000 7f4eb71cd740 1 --2- 172.21.15.12:0/711598568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960a40 0x56272b965540 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0009ca0 tx=0x7f4ea0011810 comp rx=0 tx=0).stop 2024-03-23T23:05:18.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.112+0000 7f4eb71cd740 1 -- 172.21.15.12:0/711598568 shutdown_connections 2024-03-23T23:05:18.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.112+0000 7f4eb71cd740 1 --2- 172.21.15.12:0/711598568 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56272b965bb0 0x56272b968040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.112+0000 7f4eb71cd740 1 --2- 172.21.15.12:0/711598568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960a40 0x56272b965540 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.112+0000 7f4eb71cd740 1 --2- 172.21.15.12:0/711598568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56272b960050 0x56272b960470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.112+0000 7f4eb71cd740 1 -- 172.21.15.12:0/711598568 >> 172.21.15.12:0/711598568 conn(0x56272b928200 msgr2=0x56272b95b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:18.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.112+0000 7f4eb71cd740 1 -- 172.21.15.12:0/711598568 shutdown_connections 2024-03-23T23:05:18.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.112+0000 7f4eb71cd740 1 -- 172.21.15.12:0/711598568 wait complete. 2024-03-23T23:05:18.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.113+0000 7f4eb71cd740 1 -- start start 2024-03-23T23:05:18.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.113+0000 7f4eb71cd740 1 Processor -- start 2024-03-23T23:05:18.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.113+0000 7f4eb71cd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960050 0x56272b96a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:18.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.113+0000 7f4eb71cd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56272b960a40 0x56272b96a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:18.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.113+0000 7f4eb71cd740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56272b965bb0 0x56272b9715b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:18.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.113+0000 7f4eb71cd740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56272b968b60 con 0x56272b965bb0 2024-03-23T23:05:18.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.113+0000 7f4eb71cd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56272b880e80 con 0x56272b960050 2024-03-23T23:05:18.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.113+0000 7f4eb71cd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x56272b968ce0 con 0x56272b960a40 2024-03-23T23:05:18.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.114+0000 7f4eb71cd740 10 client.0 _ll_register_callbacks cb 0x56272b893130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:18.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.114+0000 7f4eb1a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960050 0x56272b96a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:18.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.114+0000 7f4eb2400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56272b965bb0 0x56272b9715b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:18.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.114+0000 7f4eb1000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56272b960a40 0x56272b96a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:18.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.114+0000 7f4eb1a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960050 0x56272b96a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59690/0 (socket says 172.21.15.12:59690) 2024-03-23T23:05:18.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.114+0000 7f4eb1a00640 1 -- 172.21.15.12:0/3106799744 learned_addr learned my addr 172.21.15.12:0/3106799744 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:18.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.114+0000 7f4eb1a00640 1 -- 172.21.15.12:0/3106799744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56272b960a40 msgr2=0x56272b96a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:18.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.114+0000 7f4eb1a00640 1 --2- 172.21.15.12:0/3106799744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56272b960a40 0x56272b96a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.114+0000 7f4eb1a00640 1 -- 172.21.15.12:0/3106799744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56272b965bb0 msgr2=0x56272b9715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:18.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.114+0000 7f4eb1a00640 1 --2- 172.21.15.12:0/3106799744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56272b965bb0 0x56272b9715b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.114+0000 7f4eb1a00640 1 -- 172.21.15.12:0/3106799744 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56272b6a37d0 con 0x56272b960050 2024-03-23T23:05:18.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.114+0000 7f4eb1a00640 1 --2- 172.21.15.12:0/3106799744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960050 0x56272b96a030 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0011d10 tx=0x7f4ea0019b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba497916f23691ec server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:18.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.114+0000 7f4e93400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:05:18.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.115+0000 7f4e93400640 1 -- 172.21.15.12:0/3106799744 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0028020 con 0x56272b960050 2024-03-23T23:05:18.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.115+0000 7f4e93400640 1 -- 172.21.15.12:0/3106799744 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4ea0024420 con 0x56272b960050 2024-03-23T23:05:18.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.115+0000 7f4eb71cd740 1 -- 172.21.15.12:0/3106799744 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4ea00043d0 con 0x56272b960050 2024-03-23T23:05:18.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.115+0000 7f4e93400640 1 -- 172.21.15.12:0/3106799744 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ea0009890 con 0x56272b960050 2024-03-23T23:05:18.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.115+0000 7f4eb71cd740 10 client.26261 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:18.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.115+0000 7f4eb71cd740 1 -- 172.21.15.12:0/3106799744 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56272b5d93f0 con 0x56272b960050 2024-03-23T23:05:18.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.115+0000 7f4e93400640 1 -- 172.21.15.12:0/3106799744 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ea00043d0 con 0x56272b960050 2024-03-23T23:05:18.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.115+0000 7f4e93400640 1 -- 172.21.15.12:0/3106799744 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f4ea0004e80 con 0x56272b960050 2024-03-23T23:05:18.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.115+0000 7f4eb71cd740 10 client.26261 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:18.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.115+0000 7f4eb71cd740 10 client.26261 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:18.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.115+0000 7f4eb71cd740 20 client.26261 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:18.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.116+0000 7f4eb71cd740 1 --2- 172.21.15.12:0/3106799744 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56272b975b60 0x56272b977f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:18.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.116+0000 7f4eb71cd740 4 client.26261 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:05:18.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.116+0000 7f4eb71cd740 1 -- 172.21.15.12:0/3106799744 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56272b8d2790 con 0x56272b975b60 2024-03-23T23:05:18.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.116+0000 7f4eb1000640 1 --2- 172.21.15.12:0/3106799744 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56272b975b60 0x56272b977f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:18.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.116+0000 7f4eb1000640 1 --2- 172.21.15.12:0/3106799744 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56272b975b60 0x56272b977f90 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1b8ac42df8ad7db server_cookie=a037cf605f42c348 in_seq=0 out_seq=0 2024-03-23T23:05:18.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.116+0000 7f4e93400640 10 client.26261 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:18.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.117+0000 7f4e93400640 1 -- 172.21.15.12:0/3106799744 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56272b8d2790 con 0x56272b975b60 2024-03-23T23:05:18.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.117+0000 7f4e93400640 10 client.26261 handle_command_reply: tid=0 2024-03-23T23:05:18.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.117+0000 7f4eb71cd740 1 client.26261 shutdown 2024-03-23T23:05:18.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.117+0000 7f4eb71cd740 1 -- 172.21.15.12:0/3106799744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960050 msgr2=0x56272b96a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:18.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.117+0000 7f4eb71cd740 1 --2- 172.21.15.12:0/3106799744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960050 0x56272b96a030 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0011d10 tx=0x7f4ea0019b60 comp rx=0 tx=0).stop 2024-03-23T23:05:18.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.117+0000 7f4eb71cd740 1 -- 172.21.15.12:0/3106799744 shutdown_connections 2024-03-23T23:05:18.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.117+0000 7f4eb71cd740 1 --2- 172.21.15.12:0/3106799744 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56272b975b60 0x56272b977f90 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.117+0000 7f4eb71cd740 1 --2- 172.21.15.12:0/3106799744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56272b965bb0 0x56272b9715b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.117+0000 7f4eb71cd740 1 --2- 172.21.15.12:0/3106799744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56272b960a40 0x56272b96a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.117+0000 7f4eb71cd740 1 --2- 172.21.15.12:0/3106799744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960050 0x56272b96a030 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.117+0000 7f4e93400640 0 client.26261 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:18.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.117+0000 7f4eb71cd740 1 -- 172.21.15.12:0/3106799744 >> 172.21.15.12:0/3106799744 conn(0x56272b928200 msgr2=0x56272b95abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:18.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.118+0000 7f4eb71cd740 1 -- 172.21.15.12:0/3106799744 shutdown_connections 2024-03-23T23:05:18.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.118+0000 7f4eb71cd740 1 -- 172.21.15.12:0/3106799744 wait complete. 2024-03-23T23:05:18.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.118+0000 7f4eb71cd740 20 client.26261 trim_cache size 0 max 16384 2024-03-23T23:05:18.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.135+0000 7f4eb71cd740 1 Processor -- start 2024-03-23T23:05:18.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.136+0000 7f4eb71cd740 1 -- start start 2024-03-23T23:05:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.136+0000 7f4eb71cd740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56272b960050 0x56272b9614b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.136+0000 7f4eb71cd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960a40 0x56272b971570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.136+0000 7f4eb71cd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56272b965bb0 0x56272b97c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.136+0000 7f4eb71cd740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac1b1290 con 0x56272b960050 2024-03-23T23:05:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.136+0000 7f4eb71cd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac1aed90 con 0x56272b960a40 2024-03-23T23:05:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.136+0000 7f4eb71cd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac1ace30 con 0x56272b965bb0 2024-03-23T23:05:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.136+0000 7f4eb1a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56272b960050 0x56272b9614b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.136+0000 7f4eb1a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56272b960050 0x56272b9614b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40974/0 (socket says 172.21.15.12:40974) 2024-03-23T23:05:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.136+0000 7f4eb1a00640 1 -- 172.21.15.12:0/3158694179 learned_addr learned my addr 172.21.15.12:0/3158694179 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.136+0000 7f4eb1000640 1 --2- 172.21.15.12:0/3158694179 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960a40 0x56272b971570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.136+0000 7f4eb2400640 1 --2- 172.21.15.12:0/3158694179 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56272b965bb0 0x56272b97c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.136+0000 7f4eb1a00640 1 -- 172.21.15.12:0/3158694179 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56272b965bb0 msgr2=0x56272b97c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:18.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.136+0000 7f4eb1a00640 1 --2- 172.21.15.12:0/3158694179 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56272b965bb0 0x56272b97c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.136+0000 7f4eb1a00640 1 -- 172.21.15.12:0/3158694179 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960a40 msgr2=0x56272b971570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:18.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.136+0000 7f4eb1a00640 1 --2- 172.21.15.12:0/3158694179 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960a40 0x56272b971570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.136+0000 7f4eb1a00640 1 -- 172.21.15.12:0/3158694179 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56272b5990f0 con 0x56272b960050 2024-03-23T23:05:18.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.137+0000 7f4eb1000640 1 --2- 172.21.15.12:0/3158694179 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960a40 0x56272b971570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:18.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.137+0000 7f4eb1a00640 1 --2- 172.21.15.12:0/3158694179 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56272b960050 0x56272b9614b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0006010 tx=0x7f4ea0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df3a76bb71cc4b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:18.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.137+0000 7f4e9e000640 1 -- 172.21.15.12:0/3158694179 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea003c070 con 0x56272b960050 2024-03-23T23:05:18.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.137+0000 7f4e9e000640 1 -- 172.21.15.12:0/3158694179 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ea001f040 con 0x56272b960050 2024-03-23T23:05:18.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.137+0000 7f4e9e000640 1 -- 172.21.15.12:0/3158694179 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0028040 con 0x56272b960050 2024-03-23T23:05:18.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.138+0000 7f4eb71cd740 1 -- 172.21.15.12:0/3158694179 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56272b960050 msgr2=0x56272b9614b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:18.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.138+0000 7f4eb71cd740 1 --2- 172.21.15.12:0/3158694179 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56272b960050 0x56272b9614b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0006010 tx=0x7f4ea0009260 comp rx=0 tx=0).stop 2024-03-23T23:05:18.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.138+0000 7f4eb71cd740 1 -- 172.21.15.12:0/3158694179 shutdown_connections 2024-03-23T23:05:18.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.138+0000 7f4eb71cd740 1 --2- 172.21.15.12:0/3158694179 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56272b965bb0 0x56272b97c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.138+0000 7f4eb71cd740 1 --2- 172.21.15.12:0/3158694179 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960a40 0x56272b971570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.138+0000 7f4eb71cd740 1 --2- 172.21.15.12:0/3158694179 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56272b960050 0x56272b9614b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.138+0000 7f4eb71cd740 1 -- 172.21.15.12:0/3158694179 >> 172.21.15.12:0/3158694179 conn(0x56272b928200 msgr2=0x56272b95abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:18.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.138+0000 7f4eb71cd740 1 -- 172.21.15.12:0/3158694179 shutdown_connections 2024-03-23T23:05:18.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.138+0000 7f4eb71cd740 1 -- 172.21.15.12:0/3158694179 wait complete. 2024-03-23T23:05:18.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.138+0000 7f4eb71cd740 1 -- start start 2024-03-23T23:05:18.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.138+0000 7f4eb71cd740 1 Processor -- start 2024-03-23T23:05:18.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.139+0000 7f4eb71cd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960050 0x56272b96e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:18.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.139+0000 7f4eb71cd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56272b960a40 0x56272b96eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:18.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.139+0000 7f4eb71cd740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56272b965bb0 0x56272b96fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.139+0000 7f4eb71cd740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac1ad430 con 0x56272b965bb0 2024-03-23T23:05:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.139+0000 7f4eb71cd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac1ae730 con 0x56272b960050 2024-03-23T23:05:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.139+0000 7f4eb71cd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac1ada70 con 0x56272b960a40 2024-03-23T23:05:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.139+0000 7f4eb71cd740 10 client.0 _ll_register_callbacks cb 0x7f4ea0005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.139+0000 7f4eb1000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56272b960a40 0x56272b96eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.139+0000 7f4eb2400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56272b965bb0 0x56272b96fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.139+0000 7f4eb1000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56272b960a40 0x56272b96eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:59132/0 (socket says 172.21.15.12:59132) 2024-03-23T23:05:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.139+0000 7f4eb1000640 1 -- 172.21.15.12:0/1132180038 learned_addr learned my addr 172.21.15.12:0/1132180038 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.139+0000 7f4eb1a00640 1 --2- 172.21.15.12:0/1132180038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960050 0x56272b96e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.140+0000 7f4eb1000640 1 -- 172.21.15.12:0/1132180038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960050 msgr2=0x56272b96e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.140+0000 7f4eb1000640 1 --2- 172.21.15.12:0/1132180038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960050 0x56272b96e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.140+0000 7f4eb1000640 1 -- 172.21.15.12:0/1132180038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56272b965bb0 msgr2=0x56272b96fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.140+0000 7f4eb1000640 1 --2- 172.21.15.12:0/1132180038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56272b965bb0 0x56272b96fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.140+0000 7f4eb1000640 1 -- 172.21.15.12:0/1132180038 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56272b9701a0 con 0x56272b960a40 2024-03-23T23:05:18.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.140+0000 7f4eb1a00640 1 --2- 172.21.15.12:0/1132180038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960050 0x56272b96e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:05:18.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.140+0000 7f4eb2400640 1 --2- 172.21.15.12:0/1132180038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56272b965bb0 0x56272b96fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:18.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.140+0000 7f4eb1000640 1 --2- 172.21.15.12:0/1132180038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56272b960a40 0x56272b96eba0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4e980057c0 tx=0x7f4e98034ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=953283b2fa010f53 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:18.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.140+0000 7f4e93400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:05:18.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.140+0000 7f4e93400640 1 -- 172.21.15.12:0/1132180038 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e98039040 con 0x56272b960a40 2024-03-23T23:05:18.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.140+0000 7f4eb71cd740 1 -- 172.21.15.12:0/1132180038 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4ea001f040 con 0x56272b960a40 2024-03-23T23:05:18.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.140+0000 7f4e93400640 1 -- 172.21.15.12:0/1132180038 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4e98057d30 con 0x56272b960a40 2024-03-23T23:05:18.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.140+0000 7f4e93400640 1 -- 172.21.15.12:0/1132180038 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e98057300 con 0x56272b960a40 2024-03-23T23:05:18.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.140+0000 7f4eb71cd740 10 client.26230 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:18.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.140+0000 7f4eb71cd740 1 -- 172.21.15.12:0/1132180038 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56272b971550 con 0x56272b960a40 2024-03-23T23:05:18.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.141+0000 7f4e93400640 1 -- 172.21.15.12:0/1132180038 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ea001f040 con 0x56272b960a40 2024-03-23T23:05:18.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.141+0000 7f4e93400640 1 -- 172.21.15.12:0/1132180038 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f4e98032b50 con 0x56272b960a40 2024-03-23T23:05:18.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.141+0000 7f4eb71cd740 10 client.26230 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:18.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.141+0000 7f4eb71cd740 10 client.26230 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:18.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.141+0000 7f4eb71cd740 20 client.26230 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:18.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.141+0000 7f4eb71cd740 1 --2- 172.21.15.12:0/1132180038 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56272b9d17b0 0x56272b9d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:18.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.141+0000 7f4eb71cd740 4 client.26230 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:05:18.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.141+0000 7f4eb71cd740 1 -- 172.21.15.12:0/1132180038 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56272b962360 con 0x56272b9d17b0 2024-03-23T23:05:18.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.141+0000 7f4eb1a00640 1 --2- 172.21.15.12:0/1132180038 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56272b9d17b0 0x56272b9d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:18.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.142+0000 7f4eb1a00640 1 --2- 172.21.15.12:0/1132180038 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56272b9d17b0 0x56272b9d3bd0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43a2f0b26e45f56d server_cookie=c831bf6ed35a3f47 in_seq=0 out_seq=0 2024-03-23T23:05:18.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.142+0000 7f4e93400640 10 client.26230 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:18.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.142+0000 7f4e93400640 1 -- 172.21.15.12:0/1132180038 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56272b962360 con 0x56272b9d17b0 2024-03-23T23:05:18.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.142+0000 7f4e93400640 10 client.26230 handle_command_reply: tid=0 2024-03-23T23:05:18.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.142+0000 7f4eb71cd740 1 client.26230 shutdown 2024-03-23T23:05:18.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.142+0000 7f4eb71cd740 1 -- 172.21.15.12:0/1132180038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56272b960a40 msgr2=0x56272b96eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:18.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.142+0000 7f4eb71cd740 1 --2- 172.21.15.12:0/1132180038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56272b960a40 0x56272b96eba0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4e980057c0 tx=0x7f4e98034ab0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.142+0000 7f4eb71cd740 1 -- 172.21.15.12:0/1132180038 shutdown_connections 2024-03-23T23:05:18.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.142+0000 7f4eb71cd740 1 --2- 172.21.15.12:0/1132180038 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56272b9d17b0 0x56272b9d3bd0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.142+0000 7f4eb71cd740 1 --2- 172.21.15.12:0/1132180038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56272b965bb0 0x56272b96fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.142+0000 7f4eb71cd740 1 --2- 172.21.15.12:0/1132180038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56272b960a40 0x56272b96eba0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.142+0000 7f4eb71cd740 1 --2- 172.21.15.12:0/1132180038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56272b960050 0x56272b96e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.142+0000 7f4e93400640 0 client.26230 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:18.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.142+0000 7f4eb71cd740 1 -- 172.21.15.12:0/1132180038 >> 172.21.15.12:0/1132180038 conn(0x56272b928200 msgr2=0x56272b95abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:18.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.142+0000 7f4eb71cd740 1 -- 172.21.15.12:0/1132180038 shutdown_connections 2024-03-23T23:05:18.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.142+0000 7f4eb71cd740 1 -- 172.21.15.12:0/1132180038 wait complete. 2024-03-23T23:05:18.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.142+0000 7f4eb71cd740 20 client.26230 trim_cache size 0 max 16384 2024-03-23T23:05:18.148 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:05:18.148 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T23:05:18.149 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-03-23T23:05:18.149 INFO:teuthology.orchestra.run.smithi012.stdout: "5532c8a8-d7f2-4301-ae93-370bee29f82a": { 2024-03-23T23:05:18.149 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-03-23T23:05:18.149 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "5532c8a8-d7f2-4301-ae93-370bee29f82a", 2024-03-23T23:05:18.149 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-03-23T23:05:18.149 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:05:18.149 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:05:18.149 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:05:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.143+0000 7f4eb2e00640 1 -- 172.21.15.12:0/4086919151 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4e940a2880 msgr2=0x7f4e940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.143+0000 7f4eb2e00640 1 --2- 172.21.15.12:0/4086919151 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4e940a2880 0x7f4e940a4d40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4e98009830 tx=0x7f4e9802e040 comp rx=0 tx=0).stop 2024-03-23T23:05:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.143+0000 7f4eb2e00640 1 -- 172.21.15.12:0/4086919151 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4eac109770 msgr2=0x7f4eac1a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.143+0000 7f4eb2e00640 1 --2- 172.21.15.12:0/4086919151 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4eac109770 0x7f4eac1a5400 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4ea400eb10 tx=0x7f4ea400ef10 comp rx=0 tx=0).stop 2024-03-23T23:05:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.143+0000 7f4eb2e00640 1 -- 172.21.15.12:0/4086919151 shutdown_connections 2024-03-23T23:05:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.143+0000 7f4eb2e00640 1 --2- 172.21.15.12:0/4086919151 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4e940a2880 0x7f4e940a4d40 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.143+0000 7f4eb2e00640 1 --2- 172.21.15.12:0/4086919151 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4eac109770 0x7f4eac1a5400 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.143+0000 7f4eb2e00640 1 --2- 172.21.15.12:0/4086919151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4eac108db0 0x7f4eac19e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.143+0000 7f4eb2e00640 1 --2- 172.21.15.12:0/4086919151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4eac107bc0 0x7f4eac19def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.143+0000 7f4eb2e00640 1 -- 172.21.15.12:0/4086919151 >> 172.21.15.12:0/4086919151 conn(0x7f4eac076b70 msgr2=0x7f4eac10c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.143+0000 7f4eb2e00640 1 -- 172.21.15.12:0/4086919151 shutdown_connections 2024-03-23T23:05:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:18.143+0000 7f4eb2e00640 1 -- 172.21.15.12:0/4086919151 wait complete. 2024-03-23T23:05:18.161 INFO:tasks.cephfs.filesystem:scrub status for tag:5532c8a8-d7f2-4301-ae93-370bee29f82a - {'path': '/', 'tag': '5532c8a8-d7f2-4301-ae93-370bee29f82a', 'options': 'recursive,force'} 2024-03-23T23:05:18.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:18 smithi012 ceph-mon[31695]: scrub queued for path: / 2024-03-23T23:05:18.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:18 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:05:18.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:18 smithi012 ceph-mon[31695]: scrub summary: active paths [/] 2024-03-23T23:05:18.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:18 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:05:18.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:18 smithi173 ceph-mon[36116]: scrub queued for path: / 2024-03-23T23:05:18.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:18 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:05:18.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:18 smithi173 ceph-mon[36116]: scrub summary: active paths [/] 2024-03-23T23:05:18.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:18 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:05:18.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:18 smithi203 ceph-mon[36693]: scrub queued for path: / 2024-03-23T23:05:18.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:18 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:05:18.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:18 smithi203 ceph-mon[36693]: scrub summary: active paths [/] 2024-03-23T23:05:18.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:18 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:05:19.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:19 smithi012 ceph-mon[31695]: pgmap v495: 97 pgs: 97 active+clean; 1.9 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:05:19.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:19 smithi173 ceph-mon[36116]: pgmap v495: 97 pgs: 97 active+clean; 1.9 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:05:19.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:19 smithi203 ceph-mon[36693]: pgmap v495: 97 pgs: 97 active+clean; 1.9 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:05:21.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:21 smithi012 ceph-mon[31695]: pgmap v496: 97 pgs: 97 active+clean; 2.0 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 16 op/s 2024-03-23T23:05:21.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:21 smithi012 ceph-mon[31695]: scrub summary: idle 2024-03-23T23:05:21.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:21 smithi173 ceph-mon[36116]: pgmap v496: 97 pgs: 97 active+clean; 2.0 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 16 op/s 2024-03-23T23:05:21.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:21 smithi173 ceph-mon[36116]: scrub summary: idle 2024-03-23T23:05:21.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:21 smithi203 ceph-mon[36693]: pgmap v496: 97 pgs: 97 active+clean; 2.0 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 16 op/s 2024-03-23T23:05:21.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:21 smithi203 ceph-mon[36693]: scrub summary: idle 2024-03-23T23:05:23.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:23 smithi012 ceph-mon[31695]: pgmap v497: 97 pgs: 97 active+clean; 2.0 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:05:23.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:23 smithi173 ceph-mon[36116]: pgmap v497: 97 pgs: 97 active+clean; 2.0 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:05:23.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:23 smithi203 ceph-mon[36693]: pgmap v497: 97 pgs: 97 active+clean; 2.0 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:05:25.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:25 smithi012 ceph-mon[31695]: pgmap v498: 97 pgs: 97 active+clean; 2.0 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:05:25.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:25 smithi173 ceph-mon[36116]: pgmap v498: 97 pgs: 97 active+clean; 2.0 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:05:25.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:25 smithi203 ceph-mon[36693]: pgmap v498: 97 pgs: 97 active+clean; 2.0 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:05:27.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:27 smithi012 ceph-mon[31695]: pgmap v499: 97 pgs: 97 active+clean; 2.0 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:05:28.248 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:27 smithi203 ceph-mon[36693]: pgmap v499: 97 pgs: 97 active+clean; 2.0 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:05:28.248 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:27 smithi173 ceph-mon[36116]: pgmap v499: 97 pgs: 97 active+clean; 2.0 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:05:29.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:29 smithi012 ceph-mon[31695]: pgmap v500: 97 pgs: 97 active+clean; 2.0 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:05:29.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.564+0000 7f4436a00640 1 -- 172.21.15.12:0/433380204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44301048a0 msgr2=0x7f4430106c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:29.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.564+0000 7f4436a00640 1 --2- 172.21.15.12:0/433380204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44301048a0 0x7f4430106c90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f441c009900 tx=0x7f441c02bc40 comp rx=0 tx=0).stop 2024-03-23T23:05:29.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.564+0000 7f4436a00640 1 -- 172.21.15.12:0/433380204 shutdown_connections 2024-03-23T23:05:29.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.564+0000 7f4436a00640 1 --2- 172.21.15.12:0/433380204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44301071d0 0x7f4430109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.564+0000 7f4436a00640 1 --2- 172.21.15.12:0/433380204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44301048a0 0x7f4430106c90 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.564+0000 7f4436a00640 1 --2- 172.21.15.12:0/433380204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4430101f70 0x7f4430104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.564+0000 7f4436a00640 1 -- 172.21.15.12:0/433380204 >> 172.21.15.12:0/433380204 conn(0x7f44300fb980 msgr2=0x7f44300fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:29.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.564+0000 7f4436a00640 1 -- 172.21.15.12:0/433380204 shutdown_connections 2024-03-23T23:05:29.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.564+0000 7f4436a00640 1 -- 172.21.15.12:0/433380204 wait complete. 2024-03-23T23:05:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.565+0000 7f4436a00640 1 Processor -- start 2024-03-23T23:05:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.565+0000 7f4436a00640 1 -- start start 2024-03-23T23:05:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.565+0000 7f4436a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4430101f70 0x7f4430199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.565+0000 7f4436a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44301048a0 0x7f4430199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.565+0000 7f4436a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44301071d0 0x7f44301a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.565+0000 7f4436a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443010d1d0 con 0x7f4430101f70 2024-03-23T23:05:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.565+0000 7f4436a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443010d050 con 0x7f44301071d0 2024-03-23T23:05:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.565+0000 7f4436a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443010d350 con 0x7f44301048a0 2024-03-23T23:05:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.566+0000 7f4435600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4430101f70 0x7f4430199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.566+0000 7f4436000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44301071d0 0x7f44301a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.566+0000 7f4436000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44301071d0 0x7f44301a0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50774/0 (socket says 172.21.15.12:50774) 2024-03-23T23:05:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.566+0000 7f4436000640 1 -- 172.21.15.12:0/2661370460 learned_addr learned my addr 172.21.15.12:0/2661370460 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.566+0000 7f4434c00640 1 --2- 172.21.15.12:0/2661370460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44301048a0 0x7f4430199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:29.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.566+0000 7f4436000640 1 -- 172.21.15.12:0/2661370460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44301048a0 msgr2=0x7f4430199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:29.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.566+0000 7f4436000640 1 --2- 172.21.15.12:0/2661370460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44301048a0 0x7f4430199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.566+0000 7f4436000640 1 -- 172.21.15.12:0/2661370460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4430101f70 msgr2=0x7f4430199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:29.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.566+0000 7f4436000640 1 --2- 172.21.15.12:0/2661370460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4430101f70 0x7f4430199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.566+0000 7f4436000640 1 -- 172.21.15.12:0/2661370460 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44301a1640 con 0x7f44301071d0 2024-03-23T23:05:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.566+0000 7f4436000640 1 --2- 172.21.15.12:0/2661370460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44301071d0 0x7f44301a0fd0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f442800e970 tx=0x7f442800ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.567+0000 7f442e000640 1 -- 172.21.15.12:0/2661370460 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442800ccc0 con 0x7f44301071d0 2024-03-23T23:05:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.567+0000 7f442e000640 1 -- 172.21.15.12:0/2661370460 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f442800ce60 con 0x7f44301071d0 2024-03-23T23:05:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.567+0000 7f4436a00640 1 -- 172.21.15.12:0/2661370460 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44301a1930 con 0x7f44301071d0 2024-03-23T23:05:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.567+0000 7f442e000640 1 -- 172.21.15.12:0/2661370460 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4428010490 con 0x7f44301071d0 2024-03-23T23:05:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.567+0000 7f4436a00640 1 -- 172.21.15.12:0/2661370460 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44301a1ea0 con 0x7f44301071d0 2024-03-23T23:05:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.570+0000 7f443ad84740 1 Processor -- start 2024-03-23T23:05:29.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.570+0000 7f442e000640 1 -- 172.21.15.12:0/2661370460 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4428005020 con 0x7f44301071d0 2024-03-23T23:05:29.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.571+0000 7f443ad84740 1 -- start start 2024-03-23T23:05:29.572 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.571+0000 7f443ad84740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561fdb683f0 0x5561fdb68810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:29.572 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.571+0000 7f443ad84740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561fdb68de0 0x5561fdb6d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:29.572 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.571+0000 7f443ad84740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561fdb6df50 0x5561fdb703e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:29.572 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.571+0000 7f443ad84740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561fdb2b3b0 con 0x5561fdb68de0 2024-03-23T23:05:29.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.571+0000 7f443ad84740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561fdacd830 con 0x5561fdb683f0 2024-03-23T23:05:29.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.571+0000 7f443ad84740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561fda765d0 con 0x5561fdb6df50 2024-03-23T23:05:29.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.571+0000 7f4434c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561fdb683f0 0x5561fdb68810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:29.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.571+0000 7f4435600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561fdb68de0 0x5561fdb6d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:29.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.571+0000 7f4434c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561fdb683f0 0x5561fdb68810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50790/0 (socket says 172.21.15.12:50790) 2024-03-23T23:05:29.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.571+0000 7f4434c00640 1 -- 172.21.15.12:0/938923716 learned_addr learned my addr 172.21.15.12:0/938923716 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:29.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.571+0000 7f4434c00640 1 --2- 172.21.15.12:0/938923716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561fdb6df50 0x5561fdb703e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:29.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.572+0000 7f4435600640 1 -- 172.21.15.12:0/938923716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561fdb6df50 msgr2=0x5561fdb703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:29.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.572+0000 7f4435600640 1 --2- 172.21.15.12:0/938923716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561fdb6df50 0x5561fdb703e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.572+0000 7f4435600640 1 -- 172.21.15.12:0/938923716 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561fdb683f0 msgr2=0x5561fdb68810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:29.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.572+0000 7f4435600640 1 --2- 172.21.15.12:0/938923716 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561fdb683f0 0x5561fdb68810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.572+0000 7f4435600640 1 -- 172.21.15.12:0/938923716 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5561fda8e2c0 con 0x5561fdb68de0 2024-03-23T23:05:29.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.572+0000 7f4435600640 1 --2- 172.21.15.12:0/938923716 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561fdb68de0 0x5561fdb6d8e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4424009fa0 tx=0x7f44240118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a800575ebcd15655 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:29.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.572+0000 7f4422600640 1 -- 172.21.15.12:0/938923716 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4424003c30 con 0x5561fdb68de0 2024-03-23T23:05:29.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.573+0000 7f4422600640 1 -- 172.21.15.12:0/938923716 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4424003dd0 con 0x5561fdb68de0 2024-03-23T23:05:29.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.573+0000 7f4422600640 1 -- 172.21.15.12:0/938923716 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442400fac0 con 0x5561fdb68de0 2024-03-23T23:05:29.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.573+0000 7f442e000640 1 --2- 172.21.15.12:0/2661370460 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f44180a2870 0x7f44180a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:29.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.573+0000 7f442e000640 1 -- 172.21.15.12:0/2661370460 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f44280cee30 con 0x7f44301071d0 2024-03-23T23:05:29.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.573+0000 7f4434c00640 1 --2- 172.21.15.12:0/2661370460 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f44180a2870 0x7f44180a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:29.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.573+0000 7f443ad84740 1 -- 172.21.15.12:0/938923716 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561fdb68de0 msgr2=0x5561fdb6d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:29.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.573+0000 7f443ad84740 1 --2- 172.21.15.12:0/938923716 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561fdb68de0 0x5561fdb6d8e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4424009fa0 tx=0x7f44240118d0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.573+0000 7f443ad84740 1 -- 172.21.15.12:0/938923716 shutdown_connections 2024-03-23T23:05:29.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.573+0000 7f443ad84740 1 --2- 172.21.15.12:0/938923716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561fdb6df50 0x5561fdb703e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.573+0000 7f443ad84740 1 --2- 172.21.15.12:0/938923716 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561fdb68de0 0x5561fdb6d8e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.573+0000 7f443ad84740 1 --2- 172.21.15.12:0/938923716 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561fdb683f0 0x5561fdb68810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.573+0000 7f443ad84740 1 -- 172.21.15.12:0/938923716 >> 172.21.15.12:0/938923716 conn(0x5561fdb63960 msgr2=0x5561fdb63d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:29.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.574+0000 7f443ad84740 1 -- 172.21.15.12:0/938923716 shutdown_connections 2024-03-23T23:05:29.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.574+0000 7f443ad84740 1 -- 172.21.15.12:0/938923716 wait complete. 2024-03-23T23:05:29.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.574+0000 7f4434c00640 1 --2- 172.21.15.12:0/2661370460 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f44180a2870 0x7f44180a4d30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x5561fdb69640 tx=0x7f441c043040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:29.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.574+0000 7f443ad84740 1 -- start start 2024-03-23T23:05:29.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.574+0000 7f443ad84740 1 Processor -- start 2024-03-23T23:05:29.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.575+0000 7f443ad84740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561fdb683f0 0x5561fdb72550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:29.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.575+0000 7f443ad84740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561fdb68de0 0x5561fdb72a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:29.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.575+0000 7f443ad84740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561fdb6df50 0x5561fdb79ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:29.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.575+0000 7f443ad84740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561fda89a20 con 0x5561fdb6df50 2024-03-23T23:05:29.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.575+0000 7f443ad84740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561fdac98d0 con 0x5561fdb683f0 2024-03-23T23:05:29.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.575+0000 7f443ad84740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561fdb70f00 con 0x5561fdb68de0 2024-03-23T23:05:29.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.575+0000 7f443ad84740 10 client.0 _ll_register_callbacks cb 0x5561fdac61c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:29.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.575+0000 7f4436000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561fdb6df50 0x5561fdb79ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:29.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.575+0000 7f4435600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561fdb683f0 0x5561fdb72550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:29.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.575+0000 7f4435600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561fdb683f0 0x5561fdb72550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50800/0 (socket says 172.21.15.12:50800) 2024-03-23T23:05:29.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.575+0000 7f4434c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561fdb68de0 0x5561fdb72a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:29.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.575+0000 7f4435600640 1 -- 172.21.15.12:0/3458861820 learned_addr learned my addr 172.21.15.12:0/3458861820 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.575+0000 7f4436000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561fdb6df50 0x5561fdb79ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45794/0 (socket says 172.21.15.12:45794) 2024-03-23T23:05:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.575+0000 7f4436000640 1 -- 172.21.15.12:0/3458861820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561fdb68de0 msgr2=0x5561fdb72a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.575+0000 7f4436000640 1 --2- 172.21.15.12:0/3458861820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561fdb68de0 0x5561fdb72a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.575+0000 7f4436000640 1 -- 172.21.15.12:0/3458861820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561fdb683f0 msgr2=0x5561fdb72550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.575+0000 7f4436000640 1 --2- 172.21.15.12:0/3458861820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561fdb683f0 0x5561fdb72550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.575+0000 7f4436000640 1 -- 172.21.15.12:0/3458861820 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5561fd7e2210 con 0x5561fdb6df50 2024-03-23T23:05:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.575+0000 7f4435600640 1 --2- 172.21.15.12:0/3458861820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561fdb683f0 0x5561fdb72550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.576+0000 7f4436000640 1 --2- 172.21.15.12:0/3458861820 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561fdb6df50 0x5561fdb79ad0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4428004fd0 tx=0x7f442809bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62852a769f3a970e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.576+0000 7f4416a00640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:05:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.576+0000 7f4416a00640 1 -- 172.21.15.12:0/3458861820 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44280b3020 con 0x5561fdb6df50 2024-03-23T23:05:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.576+0000 7f4416a00640 1 -- 172.21.15.12:0/3458861820 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4428097df0 con 0x5561fdb6df50 2024-03-23T23:05:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.576+0000 7f443ad84740 1 -- 172.21.15.12:0/3458861820 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4424003dd0 con 0x5561fdb6df50 2024-03-23T23:05:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.576+0000 7f4416a00640 1 -- 172.21.15.12:0/3458861820 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f44280a3f20 con 0x5561fdb6df50 2024-03-23T23:05:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.576+0000 7f443ad84740 10 client.16146 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.576+0000 7f443ad84740 1 -- 172.21.15.12:0/3458861820 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5561fd8ae620 con 0x5561fdb6df50 2024-03-23T23:05:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.577+0000 7f4416a00640 1 -- 172.21.15.12:0/3458861820 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4424003dd0 con 0x5561fdb6df50 2024-03-23T23:05:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.577+0000 7f4416a00640 1 -- 172.21.15.12:0/3458861820 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f4428097040 con 0x5561fdb6df50 2024-03-23T23:05:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.577+0000 7f443ad84740 10 client.16146 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.577+0000 7f443ad84740 10 client.16146 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.577+0000 7f443ad84740 20 client.16146 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.577+0000 7f443ad84740 1 --2- 172.21.15.12:0/3458861820 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5561fdb7e460 0x5561fdb80850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.577+0000 7f443ad84740 4 client.16146 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:05:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.577+0000 7f443ad84740 1 -- 172.21.15.12:0/3458861820 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5561fdadad70 con 0x5561fdb7e460 2024-03-23T23:05:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.577+0000 7f4435600640 1 --2- 172.21.15.12:0/3458861820 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5561fdb7e460 0x5561fdb80850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.578+0000 7f4435600640 1 --2- 172.21.15.12:0/3458861820 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5561fdb7e460 0x5561fdb80850 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c6383cf2137a75 server_cookie=61a1f7f21ab42e58 in_seq=0 out_seq=0 2024-03-23T23:05:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.578+0000 7f4416a00640 10 client.16146 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.578+0000 7f4416a00640 1 -- 172.21.15.12:0/3458861820 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5561fdadad70 con 0x5561fdb7e460 2024-03-23T23:05:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.578+0000 7f4416a00640 10 client.16146 handle_command_reply: tid=0 2024-03-23T23:05:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.578+0000 7f443ad84740 1 client.16146 shutdown 2024-03-23T23:05:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.579+0000 7f443ad84740 1 -- 172.21.15.12:0/3458861820 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561fdb6df50 msgr2=0x5561fdb79ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.579+0000 7f443ad84740 1 --2- 172.21.15.12:0/3458861820 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561fdb6df50 0x5561fdb79ad0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4428004fd0 tx=0x7f442809bf40 comp rx=0 tx=0).stop 2024-03-23T23:05:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.579+0000 7f443ad84740 1 -- 172.21.15.12:0/3458861820 shutdown_connections 2024-03-23T23:05:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.579+0000 7f443ad84740 1 --2- 172.21.15.12:0/3458861820 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5561fdb7e460 0x5561fdb80850 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.579+0000 7f443ad84740 1 --2- 172.21.15.12:0/3458861820 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561fdb6df50 0x5561fdb79ad0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.579+0000 7f443ad84740 1 --2- 172.21.15.12:0/3458861820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561fdb68de0 0x5561fdb72a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.579+0000 7f443ad84740 1 --2- 172.21.15.12:0/3458861820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561fdb683f0 0x5561fdb72550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.579+0000 7f4416a00640 0 client.16146 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.579+0000 7f443ad84740 1 -- 172.21.15.12:0/3458861820 >> 172.21.15.12:0/3458861820 conn(0x5561fdb63960 msgr2=0x5561fdb62b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.579+0000 7f443ad84740 1 -- 172.21.15.12:0/3458861820 shutdown_connections 2024-03-23T23:05:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.579+0000 7f443ad84740 1 -- 172.21.15.12:0/3458861820 wait complete. 2024-03-23T23:05:29.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.579+0000 7f443ad84740 20 client.16146 trim_cache size 0 max 16384 2024-03-23T23:05:29.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.594+0000 7f443ad84740 1 Processor -- start 2024-03-23T23:05:29.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.594+0000 7f443ad84740 1 -- start start 2024-03-23T23:05:29.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.594+0000 7f443ad84740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561fdb683f0 0x5561fdb85110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:29.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.594+0000 7f443ad84740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561fdb68de0 0x5561fdb79a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:29.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.594+0000 7f443ad84740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561fdb6df50 0x5561fdb85650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:29.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.594+0000 7f443ad84740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44301ad010 con 0x5561fdb6df50 2024-03-23T23:05:29.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.594+0000 7f443ad84740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44301aab10 con 0x5561fdb68de0 2024-03-23T23:05:29.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.594+0000 7f443ad84740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44301a8bb0 con 0x5561fdb683f0 2024-03-23T23:05:29.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.595+0000 7f4435600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561fdb683f0 0x5561fdb85110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:29.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.595+0000 7f4434c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561fdb68de0 0x5561fdb79a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:29.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.595+0000 7f4435600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561fdb683f0 0x5561fdb85110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:51290/0 (socket says 172.21.15.12:51290) 2024-03-23T23:05:29.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.595+0000 7f4435600640 1 -- 172.21.15.12:0/1278008738 learned_addr learned my addr 172.21.15.12:0/1278008738 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:29.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.595+0000 7f4436000640 1 --2- 172.21.15.12:0/1278008738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561fdb6df50 0x5561fdb85650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:29.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.595+0000 7f4435600640 1 -- 172.21.15.12:0/1278008738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561fdb68de0 msgr2=0x5561fdb79a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:29.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.595+0000 7f4435600640 1 --2- 172.21.15.12:0/1278008738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561fdb68de0 0x5561fdb79a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.595+0000 7f4435600640 1 -- 172.21.15.12:0/1278008738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561fdb6df50 msgr2=0x5561fdb85650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:29.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.595+0000 7f4435600640 1 --2- 172.21.15.12:0/1278008738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561fdb6df50 0x5561fdb85650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.595+0000 7f4435600640 1 -- 172.21.15.12:0/1278008738 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5561fdb80eb0 con 0x5561fdb683f0 2024-03-23T23:05:29.598 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.595+0000 7f4435600640 1 --2- 172.21.15.12:0/1278008738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561fdb683f0 0x5561fdb85110 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4424009d60 tx=0x7f44240118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c9156454129f68d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:29.598 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.595+0000 7f4421c00640 1 -- 172.21.15.12:0/1278008738 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4424030030 con 0x5561fdb683f0 2024-03-23T23:05:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.596+0000 7f4421c00640 1 -- 172.21.15.12:0/1278008738 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4424003dd0 con 0x5561fdb683f0 2024-03-23T23:05:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.596+0000 7f4421c00640 1 -- 172.21.15.12:0/1278008738 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442400f8c0 con 0x5561fdb683f0 2024-03-23T23:05:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.596+0000 7f443ad84740 1 -- 172.21.15.12:0/1278008738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561fdb683f0 msgr2=0x5561fdb85110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.596+0000 7f443ad84740 1 --2- 172.21.15.12:0/1278008738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561fdb683f0 0x5561fdb85110 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4424009d60 tx=0x7f44240118d0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.596+0000 7f443ad84740 1 -- 172.21.15.12:0/1278008738 shutdown_connections 2024-03-23T23:05:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.596+0000 7f443ad84740 1 --2- 172.21.15.12:0/1278008738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561fdb6df50 0x5561fdb85650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.596+0000 7f443ad84740 1 --2- 172.21.15.12:0/1278008738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561fdb68de0 0x5561fdb79a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.596+0000 7f443ad84740 1 --2- 172.21.15.12:0/1278008738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561fdb683f0 0x5561fdb85110 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.596+0000 7f443ad84740 1 -- 172.21.15.12:0/1278008738 >> 172.21.15.12:0/1278008738 conn(0x5561fdb63960 msgr2=0x5561fdb66bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.596+0000 7f443ad84740 1 -- 172.21.15.12:0/1278008738 shutdown_connections 2024-03-23T23:05:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.596+0000 7f443ad84740 1 -- 172.21.15.12:0/1278008738 wait complete. 2024-03-23T23:05:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.597+0000 7f443ad84740 1 -- start start 2024-03-23T23:05:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.597+0000 7f443ad84740 1 Processor -- start 2024-03-23T23:05:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.597+0000 7f443ad84740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561fdb683f0 0x5561fdb77850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.597+0000 7f443ad84740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561fdb68de0 0x5561fdb77d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.597+0000 7f443ad84740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561fdb6df50 0x5561fdb78d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.597+0000 7f443ad84740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44301a91b0 con 0x5561fdb6df50 2024-03-23T23:05:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.597+0000 7f443ad84740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44301aa4b0 con 0x5561fdb68de0 2024-03-23T23:05:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.597+0000 7f443ad84740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44301a97f0 con 0x5561fdb683f0 2024-03-23T23:05:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.597+0000 7f443ad84740 10 client.0 _ll_register_callbacks cb 0x7f4428005690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.598+0000 7f4434c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561fdb68de0 0x5561fdb77d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.598+0000 7f4435600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561fdb683f0 0x5561fdb77850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.598+0000 7f4436000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561fdb6df50 0x5561fdb78d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.598+0000 7f4435600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561fdb683f0 0x5561fdb77850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:51296/0 (socket says 172.21.15.12:51296) 2024-03-23T23:05:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.598+0000 7f4434c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561fdb68de0 0x5561fdb77d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50820/0 (socket says 172.21.15.12:50820) 2024-03-23T23:05:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.598+0000 7f4435600640 1 -- 172.21.15.12:0/510063685 learned_addr learned my addr 172.21.15.12:0/510063685 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.598+0000 7f4435600640 1 -- 172.21.15.12:0/510063685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561fdb68de0 msgr2=0x5561fdb77d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.598+0000 7f4435600640 1 --2- 172.21.15.12:0/510063685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561fdb68de0 0x5561fdb77d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.598+0000 7f4435600640 1 -- 172.21.15.12:0/510063685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561fdb6df50 msgr2=0x5561fdb78d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.598+0000 7f4435600640 1 --2- 172.21.15.12:0/510063685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561fdb6df50 0x5561fdb78d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.598+0000 7f4435600640 1 -- 172.21.15.12:0/510063685 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5561fdb79340 con 0x5561fdb683f0 2024-03-23T23:05:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.598+0000 7f4434c00640 1 --2- 172.21.15.12:0/510063685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561fdb68de0 0x5561fdb77d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.598+0000 7f4435600640 1 --2- 172.21.15.12:0/510063685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561fdb683f0 0x5561fdb77850 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4424014040 tx=0x7f4424004500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=476e61ca21f5ff8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.598+0000 7f4416a00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:05:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.599+0000 7f4416a00640 1 -- 172.21.15.12:0/510063685 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4424030040 con 0x5561fdb683f0 2024-03-23T23:05:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.599+0000 7f4416a00640 1 -- 172.21.15.12:0/510063685 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f442400f840 con 0x5561fdb683f0 2024-03-23T23:05:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.599+0000 7f443ad84740 1 -- 172.21.15.12:0/510063685 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4424003dd0 con 0x5561fdb683f0 2024-03-23T23:05:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.599+0000 7f4416a00640 1 -- 172.21.15.12:0/510063685 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4424043050 con 0x5561fdb683f0 2024-03-23T23:05:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.599+0000 7f443ad84740 10 client.26248 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.599+0000 7f443ad84740 1 -- 172.21.15.12:0/510063685 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5561fdbda310 con 0x5561fdb683f0 2024-03-23T23:05:29.602 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.599+0000 7f4416a00640 1 -- 172.21.15.12:0/510063685 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4424003dd0 con 0x5561fdb683f0 2024-03-23T23:05:29.602 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.599+0000 7f4416a00640 1 -- 172.21.15.12:0/510063685 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f4428086050 con 0x5561fdb683f0 2024-03-23T23:05:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.599+0000 7f443ad84740 10 client.26248 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.599+0000 7f443ad84740 10 client.26248 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.599+0000 7f443ad84740 20 client.26248 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.599+0000 7f443ad84740 1 --2- 172.21.15.12:0/510063685 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5561fdbda9f0 0x5561fdbdce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.599+0000 7f443ad84740 4 client.26248 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:05:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.599+0000 7f443ad84740 1 -- 172.21.15.12:0/510063685 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5561fdad50c0 con 0x5561fdbda9f0 2024-03-23T23:05:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.600+0000 7f4434c00640 1 --2- 172.21.15.12:0/510063685 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5561fdbda9f0 0x5561fdbdce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.600+0000 7f4434c00640 1 --2- 172.21.15.12:0/510063685 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5561fdbda9f0 0x5561fdbdce10 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79b1ccc829319088 server_cookie=4f912a30c88e9d61 in_seq=0 out_seq=0 2024-03-23T23:05:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.600+0000 7f4416a00640 10 client.26248 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:29.612 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.613+0000 7f4416a00640 1 -- 172.21.15.12:0/510063685 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5561fdad50c0 con 0x5561fdbda9f0 2024-03-23T23:05:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.613+0000 7f4416a00640 10 client.26248 handle_command_reply: tid=0 2024-03-23T23:05:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.613+0000 7f443ad84740 1 client.26248 shutdown 2024-03-23T23:05:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.613+0000 7f443ad84740 1 -- 172.21.15.12:0/510063685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561fdb683f0 msgr2=0x5561fdb77850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.613+0000 7f443ad84740 1 --2- 172.21.15.12:0/510063685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561fdb683f0 0x5561fdb77850 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4424014040 tx=0x7f4424004500 comp rx=0 tx=0).stop 2024-03-23T23:05:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.613+0000 7f443ad84740 1 -- 172.21.15.12:0/510063685 shutdown_connections 2024-03-23T23:05:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.613+0000 7f443ad84740 1 --2- 172.21.15.12:0/510063685 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5561fdbda9f0 0x5561fdbdce10 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.613+0000 7f443ad84740 1 --2- 172.21.15.12:0/510063685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561fdb6df50 0x5561fdb78d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.613+0000 7f4416a00640 0 client.26248 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.613+0000 7f443ad84740 1 --2- 172.21.15.12:0/510063685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5561fdb68de0 0x5561fdb77d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.613+0000 7f443ad84740 1 --2- 172.21.15.12:0/510063685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5561fdb683f0 0x5561fdb77850 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.613+0000 7f443ad84740 1 -- 172.21.15.12:0/510063685 >> 172.21.15.12:0/510063685 conn(0x5561fdb63960 msgr2=0x5561fdb622e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.613+0000 7f443ad84740 1 -- 172.21.15.12:0/510063685 shutdown_connections 2024-03-23T23:05:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.613+0000 7f443ad84740 1 -- 172.21.15.12:0/510063685 wait complete. 2024-03-23T23:05:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.613+0000 7f443ad84740 20 client.26248 trim_cache size 0 max 16384 2024-03-23T23:05:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:05:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:05:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:05:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:05:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.613+0000 7f4436a00640 1 -- 172.21.15.12:0/2661370460 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f44180a2870 msgr2=0x7f44180a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.614+0000 7f4436a00640 1 --2- 172.21.15.12:0/2661370460 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f44180a2870 0x7f44180a4d30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x5561fdb69640 tx=0x7f441c043040 comp rx=0 tx=0).stop 2024-03-23T23:05:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.614+0000 7f4436a00640 1 -- 172.21.15.12:0/2661370460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44301071d0 msgr2=0x7f44301a0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.614+0000 7f4436a00640 1 --2- 172.21.15.12:0/2661370460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44301071d0 0x7f44301a0fd0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f442800e970 tx=0x7f442800ed70 comp rx=0 tx=0).stop 2024-03-23T23:05:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.614+0000 7f4436a00640 1 -- 172.21.15.12:0/2661370460 shutdown_connections 2024-03-23T23:05:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.614+0000 7f4436a00640 1 --2- 172.21.15.12:0/2661370460 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f44180a2870 0x7f44180a4d30 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.614+0000 7f4436a00640 1 --2- 172.21.15.12:0/2661370460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44301071d0 0x7f44301a0fd0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.614+0000 7f4436a00640 1 --2- 172.21.15.12:0/2661370460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44301048a0 0x7f4430199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.614+0000 7f4436a00640 1 --2- 172.21.15.12:0/2661370460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4430101f70 0x7f4430199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.614+0000 7f4436a00640 1 -- 172.21.15.12:0/2661370460 >> 172.21.15.12:0/2661370460 conn(0x7f44300fb980 msgr2=0x7f4430101700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.614+0000 7f4436a00640 1 -- 172.21.15.12:0/2661370460 shutdown_connections 2024-03-23T23:05:29.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:29.614+0000 7f4436a00640 1 -- 172.21.15.12:0/2661370460 wait complete. 2024-03-23T23:05:29.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:29 smithi173 ceph-mon[36116]: pgmap v500: 97 pgs: 97 active+clean; 2.0 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:05:29.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:29 smithi203 ceph-mon[36693]: pgmap v500: 97 pgs: 97 active+clean; 2.0 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:05:31.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:31 smithi012 ceph-mon[31695]: pgmap v501: 97 pgs: 97 active+clean; 2.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-03-23T23:05:31.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:31 smithi173 ceph-mon[36116]: pgmap v501: 97 pgs: 97 active+clean; 2.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-03-23T23:05:31.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:31 smithi203 ceph-mon[36693]: pgmap v501: 97 pgs: 97 active+clean; 2.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-03-23T23:05:33.459 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:33 smithi203 ceph-mon[36693]: pgmap v502: 97 pgs: 97 active+clean; 2.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-23T23:05:33.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:33 smithi012 ceph-mon[31695]: pgmap v502: 97 pgs: 97 active+clean; 2.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-23T23:05:33.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:33 smithi173 ceph-mon[36116]: pgmap v502: 97 pgs: 97 active+clean; 2.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-23T23:05:35.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:35 smithi012 ceph-mon[31695]: pgmap v503: 97 pgs: 97 active+clean; 2.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:05:35.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:35 smithi173 ceph-mon[36116]: pgmap v503: 97 pgs: 97 active+clean; 2.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:05:35.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:35 smithi203 ceph-mon[36693]: pgmap v503: 97 pgs: 97 active+clean; 2.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:05:37.419 INFO:tasks.workunit.client.0.smithi012.stdout:64/600: open fd = 3 2024-03-23T23:05:37.419 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.419 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.419 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.420 INFO:tasks.workunit.client.0.smithi012.stdout:65/600: open fd = 3 2024-03-23T23:05:37.420 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.420 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.420 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.420 INFO:tasks.workunit.client.0.smithi012.stdout:66/600: open fd = 3 2024-03-23T23:05:37.420 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.420 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.420 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.420 INFO:tasks.workunit.client.0.smithi012.stdout:67/600: open fd = 3 2024-03-23T23:05:37.420 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.420 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.420 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.420 INFO:tasks.workunit.client.0.smithi012.stdout:68/600: open fd = 3 2024-03-23T23:05:37.420 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.421 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.421 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.422 INFO:tasks.workunit.client.0.smithi012.stdout:69/600: open fd = 3 2024-03-23T23:05:37.422 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.422 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.422 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.422 INFO:tasks.workunit.client.0.smithi012.stdout:70/600: open fd = 3 2024-03-23T23:05:37.422 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.422 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.422 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.422 INFO:tasks.workunit.client.0.smithi012.stdout:71/600: open fd = 3 2024-03-23T23:05:37.422 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.422 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.422 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.423 INFO:tasks.workunit.client.0.smithi012.stdout:72/600: open fd = 3 2024-03-23T23:05:37.423 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.423 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.423 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.423 INFO:tasks.workunit.client.0.smithi012.stdout:73/600: open fd = 3 2024-03-23T23:05:37.423 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.423 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.423 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.423 INFO:tasks.workunit.client.0.smithi012.stdout:74/600: open fd = 3 2024-03-23T23:05:37.423 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.423 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.423 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.423 INFO:tasks.workunit.client.0.smithi012.stdout:75/600: open fd = 3 2024-03-23T23:05:37.423 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.423 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.423 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.424 INFO:tasks.workunit.client.0.smithi012.stdout:76/600: open fd = 3 2024-03-23T23:05:37.424 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.424 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.424 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.424 INFO:tasks.workunit.client.0.smithi012.stdout:77/600: open fd = 3 2024-03-23T23:05:37.424 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.424 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.424 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.424 INFO:tasks.workunit.client.0.smithi012.stdout:78/600: open fd = 3 2024-03-23T23:05:37.424 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.424 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.424 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.424 INFO:tasks.workunit.client.0.smithi012.stdout:79/600: open fd = 3 2024-03-23T23:05:37.424 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.424 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.425 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.425 INFO:tasks.workunit.client.0.smithi012.stdout:80/600: open fd = 3 2024-03-23T23:05:37.425 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.425 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.425 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.425 INFO:tasks.workunit.client.0.smithi012.stdout:81/600: open fd = 3 2024-03-23T23:05:37.425 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.425 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.425 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.425 INFO:tasks.workunit.client.0.smithi012.stdout:82/600: open fd = 3 2024-03-23T23:05:37.425 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.425 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.425 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.425 INFO:tasks.workunit.client.0.smithi012.stdout:83/600: open fd = 3 2024-03-23T23:05:37.425 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.425 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.426 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.426 INFO:tasks.workunit.client.0.smithi012.stdout:84/600: open fd = 3 2024-03-23T23:05:37.426 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.426 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.426 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.426 INFO:tasks.workunit.client.0.smithi012.stdout:85/600: open fd = 3 2024-03-23T23:05:37.426 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.426 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.426 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.426 INFO:tasks.workunit.client.0.smithi012.stdout:86/600: open fd = 3 2024-03-23T23:05:37.426 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.426 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.427 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.427 INFO:tasks.workunit.client.0.smithi012.stdout:87/600: open fd = 3 2024-03-23T23:05:37.427 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.427 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.427 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.427 INFO:tasks.workunit.client.0.smithi012.stdout:88/600: open fd = 3 2024-03-23T23:05:37.427 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.427 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.427 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.428 INFO:tasks.workunit.client.0.smithi012.stdout:89/600: open fd = 3 2024-03-23T23:05:37.428 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.428 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.428 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.428 INFO:tasks.workunit.client.0.smithi012.stdout:90/600: open fd = 3 2024-03-23T23:05:37.428 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.428 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.428 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.428 INFO:tasks.workunit.client.0.smithi012.stdout:91/600: open fd = 3 2024-03-23T23:05:37.428 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.429 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.429 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.429 INFO:tasks.workunit.client.0.smithi012.stdout:92/600: open fd = 3 2024-03-23T23:05:37.429 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.429 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.429 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.429 INFO:tasks.workunit.client.0.smithi012.stdout:93/600: open fd = 3 2024-03-23T23:05:37.429 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.429 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.429 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.429 INFO:tasks.workunit.client.0.smithi012.stdout:94/600: open fd = 3 2024-03-23T23:05:37.429 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.430 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.430 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.430 INFO:tasks.workunit.client.0.smithi012.stdout:95/600: open fd = 3 2024-03-23T23:05:37.430 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.430 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.430 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.430 INFO:tasks.workunit.client.0.smithi012.stdout:96/600: open fd = 3 2024-03-23T23:05:37.430 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.430 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.430 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.430 INFO:tasks.workunit.client.0.smithi012.stdout:97/600: open fd = 3 2024-03-23T23:05:37.430 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.430 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.430 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.430 INFO:tasks.workunit.client.0.smithi012.stdout:98/600: open fd = 3 2024-03-23T23:05:37.431 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.431 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.431 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.431 INFO:tasks.workunit.client.0.smithi012.stdout:99/600: open fd = 3 2024-03-23T23:05:37.431 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.431 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.431 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.431 INFO:tasks.workunit.client.0.smithi012.stdout:100/600: open fd = 3 2024-03-23T23:05:37.431 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.431 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.431 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.431 INFO:tasks.workunit.client.0.smithi012.stdout:101/600: open fd = 3 2024-03-23T23:05:37.431 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.431 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.431 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.432 INFO:tasks.workunit.client.0.smithi012.stdout:102/600: open fd = 3 2024-03-23T23:05:37.432 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.432 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.432 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.432 INFO:tasks.workunit.client.0.smithi012.stdout:103/600: open fd = 3 2024-03-23T23:05:37.432 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.432 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.432 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.432 INFO:tasks.workunit.client.0.smithi012.stdout:104/600: open fd = 3 2024-03-23T23:05:37.432 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.432 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.432 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.432 INFO:tasks.workunit.client.0.smithi012.stdout:105/600: open fd = 3 2024-03-23T23:05:37.432 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.432 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.432 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.433 INFO:tasks.workunit.client.0.smithi012.stdout:106/600: open fd = 3 2024-03-23T23:05:37.433 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.433 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.433 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.433 INFO:tasks.workunit.client.0.smithi012.stdout:107/600: open fd = 3 2024-03-23T23:05:37.433 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.433 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.433 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.433 INFO:tasks.workunit.client.0.smithi012.stdout:108/600: open fd = 3 2024-03-23T23:05:37.433 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.433 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.433 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.433 INFO:tasks.workunit.client.0.smithi012.stdout:109/600: open fd = 3 2024-03-23T23:05:37.433 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.433 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.433 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.434 INFO:tasks.workunit.client.0.smithi012.stdout:110/600: open fd = 3 2024-03-23T23:05:37.434 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.434 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.434 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.434 INFO:tasks.workunit.client.0.smithi012.stdout:111/600: open fd = 3 2024-03-23T23:05:37.434 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.434 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.434 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.434 INFO:tasks.workunit.client.0.smithi012.stdout:112/600: open fd = 3 2024-03-23T23:05:37.434 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.434 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.434 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.434 INFO:tasks.workunit.client.0.smithi012.stdout:113/600: open fd = 3 2024-03-23T23:05:37.434 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.434 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.435 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.435 INFO:tasks.workunit.client.0.smithi012.stdout:114/600: open fd = 3 2024-03-23T23:05:37.435 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.435 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.435 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.435 INFO:tasks.workunit.client.0.smithi012.stdout:115/600: open fd = 3 2024-03-23T23:05:37.435 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.435 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.435 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.435 INFO:tasks.workunit.client.0.smithi012.stdout:116/600: open fd = 3 2024-03-23T23:05:37.435 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.436 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.436 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.436 INFO:tasks.workunit.client.0.smithi012.stdout:117/600: open fd = 3 2024-03-23T23:05:37.436 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.436 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.436 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.436 INFO:tasks.workunit.client.0.smithi012.stdout:118/600: open fd = 3 2024-03-23T23:05:37.436 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.436 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.436 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.436 INFO:tasks.workunit.client.0.smithi012.stdout:119/600: open fd = 3 2024-03-23T23:05:37.436 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.436 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.436 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.436 INFO:tasks.workunit.client.0.smithi012.stdout:120/600: open fd = 3 2024-03-23T23:05:37.437 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.437 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.437 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.437 INFO:tasks.workunit.client.0.smithi012.stdout:121/600: open fd = 3 2024-03-23T23:05:37.437 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.437 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.437 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.437 INFO:tasks.workunit.client.0.smithi012.stdout:122/600: open fd = 3 2024-03-23T23:05:37.437 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.437 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.437 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.437 INFO:tasks.workunit.client.0.smithi012.stdout:123/600: open fd = 3 2024-03-23T23:05:37.437 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.437 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.437 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.438 INFO:tasks.workunit.client.0.smithi012.stdout:124/600: open fd = 3 2024-03-23T23:05:37.438 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.438 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.438 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.438 INFO:tasks.workunit.client.0.smithi012.stdout:125/600: open fd = 3 2024-03-23T23:05:37.438 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.438 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.438 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:05:37.438 INFO:tasks.workunit.client.0.smithi012.stdout:126/600: open fd = 3 2024-03-23T23:05:37.438 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:05:37.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:37 smithi012 ceph-mon[31695]: pgmap v504: 97 pgs: 97 active+clean; 2.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:05:37.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:37 smithi173 ceph-mon[36116]: pgmap v504: 97 pgs: 97 active+clean; 2.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:05:37.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:37 smithi203 ceph-mon[36693]: pgmap v504: 97 pgs: 97 active+clean; 2.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:05:39.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:39 smithi012 ceph-mon[31695]: pgmap v505: 97 pgs: 97 active+clean; 2.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:05:39.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:39 smithi173 ceph-mon[36116]: pgmap v505: 97 pgs: 97 active+clean; 2.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:05:39.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:39 smithi203 ceph-mon[36693]: pgmap v505: 97 pgs: 97 active+clean; 2.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:05:41.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:41 smithi012 ceph-mon[31695]: pgmap v506: 97 pgs: 97 active+clean; 2.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:05:41.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:41 smithi173 ceph-mon[36116]: pgmap v506: 97 pgs: 97 active+clean; 2.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:05:41.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:41 smithi203 ceph-mon[36693]: pgmap v506: 97 pgs: 97 active+clean; 2.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:05:42.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.763+0000 7f0078800640 1 -- 172.21.15.12:0/207148894 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0070103900 msgr2=0x7f0070103d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:42.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.763+0000 7f0078800640 1 --2- 172.21.15.12:0/207148894 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0070103900 0x7f0070103d20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f0064009900 tx=0x7f006402bc40 comp rx=0 tx=0).stop 2024-03-23T23:05:42.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.764+0000 7f0078800640 1 -- 172.21.15.12:0/207148894 shutdown_connections 2024-03-23T23:05:42.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.764+0000 7f0078800640 1 --2- 172.21.15.12:0/207148894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00701054b0 0x7f007010bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.764+0000 7f0078800640 1 --2- 172.21.15.12:0/207148894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0070104af0 0x7f0070104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.764+0000 7f0078800640 1 --2- 172.21.15.12:0/207148894 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0070103900 0x7f0070103d20 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.764+0000 7f0078800640 1 -- 172.21.15.12:0/207148894 >> 172.21.15.12:0/207148894 conn(0x7f00700fee90 msgr2=0x7f00701012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:42.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.764+0000 7f0078800640 1 -- 172.21.15.12:0/207148894 shutdown_connections 2024-03-23T23:05:42.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.764+0000 7f0078800640 1 -- 172.21.15.12:0/207148894 wait complete. 2024-03-23T23:05:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.765+0000 7f0078800640 1 Processor -- start 2024-03-23T23:05:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.765+0000 7f0078800640 1 -- start start 2024-03-23T23:05:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.765+0000 7f0078800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0070103900 0x7f0070199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.765+0000 7f0078800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0070104af0 0x7f007019a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.765+0000 7f0078800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00701054b0 0x7f00701a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.766+0000 7f0078800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007010f5d0 con 0x7f0070103900 2024-03-23T23:05:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.766+0000 7f0078800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007010f450 con 0x7f00701054b0 2024-03-23T23:05:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.766+0000 7f0078800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007010f750 con 0x7f0070104af0 2024-03-23T23:05:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.766+0000 7f0075400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0070104af0 0x7f007019a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.766+0000 7f0075e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0070103900 0x7f0070199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.766+0000 7f0075400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0070104af0 0x7f007019a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55682/0 (socket says 172.21.15.12:55682) 2024-03-23T23:05:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.766+0000 7f0075400640 1 -- 172.21.15.12:0/3802650944 learned_addr learned my addr 172.21.15.12:0/3802650944 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.766+0000 7f0076800640 1 --2- 172.21.15.12:0/3802650944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00701054b0 0x7f00701a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.766+0000 7f0075400640 1 -- 172.21.15.12:0/3802650944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00701054b0 msgr2=0x7f00701a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.766+0000 7f0075400640 1 --2- 172.21.15.12:0/3802650944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00701054b0 0x7f00701a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.766+0000 7f0075400640 1 -- 172.21.15.12:0/3802650944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0070103900 msgr2=0x7f0070199b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.766+0000 7f0075400640 1 --2- 172.21.15.12:0/3802650944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0070103900 0x7f0070199b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.766+0000 7f0075400640 1 -- 172.21.15.12:0/3802650944 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00701a1770 con 0x7f0070104af0 2024-03-23T23:05:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.766+0000 7f0075e00640 1 --2- 172.21.15.12:0/3802650944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0070103900 0x7f0070199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.767+0000 7f0075400640 1 --2- 172.21.15.12:0/3802650944 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0070104af0 0x7f007019a090 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f006000e9f0 tx=0x7f006000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.767+0000 7f006aa00640 1 -- 172.21.15.12:0/3802650944 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006000cd80 con 0x7f0070104af0 2024-03-23T23:05:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.767+0000 7f006aa00640 1 -- 172.21.15.12:0/3802650944 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0060010910 con 0x7f0070104af0 2024-03-23T23:05:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.767+0000 7f006aa00640 1 -- 172.21.15.12:0/3802650944 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0060018990 con 0x7f0070104af0 2024-03-23T23:05:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.767+0000 7f0078800640 1 -- 172.21.15.12:0/3802650944 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00701a1a60 con 0x7f0070104af0 2024-03-23T23:05:42.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.767+0000 7f0078800640 1 -- 172.21.15.12:0/3802650944 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00701a1f60 con 0x7f0070104af0 2024-03-23T23:05:42.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.770+0000 7f006aa00640 1 -- 172.21.15.12:0/3802650944 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0060010ac0 con 0x7f0070104af0 2024-03-23T23:05:42.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.771+0000 7f007b3ff740 1 Processor -- start 2024-03-23T23:05:42.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.771+0000 7f007b3ff740 1 -- start start 2024-03-23T23:05:42.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.771+0000 7f007b3ff740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5575888731e0 0x557588873600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:42.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.771+0000 7f007b3ff740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557588873bd0 0x5575888786d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:42.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.771+0000 7f007b3ff740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557588878d40 0x55758887b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:42.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.771+0000 7f007b3ff740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575888363b0 con 0x557588878d40 2024-03-23T23:05:42.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.771+0000 7f007b3ff740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575887d8830 con 0x557588873bd0 2024-03-23T23:05:42.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.771+0000 7f007b3ff740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575887815d0 con 0x5575888731e0 2024-03-23T23:05:42.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.772+0000 7f0075e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557588873bd0 0x5575888786d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:42.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.772+0000 7f0075e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557588873bd0 0x5575888786d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:43712/0 (socket says 172.21.15.12:43712) 2024-03-23T23:05:42.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.772+0000 7f0075e00640 1 -- 172.21.15.12:0/3281672788 learned_addr learned my addr 172.21.15.12:0/3281672788 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:42.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.772+0000 7f0076800640 1 --2- 172.21.15.12:0/3281672788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5575888731e0 0x557588873600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:42.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.772+0000 7f0075400640 1 --2- 172.21.15.12:0/3281672788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557588878d40 0x55758887b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:42.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.772+0000 7f0075e00640 1 -- 172.21.15.12:0/3281672788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5575888731e0 msgr2=0x557588873600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:42.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.772+0000 7f0075e00640 1 --2- 172.21.15.12:0/3281672788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5575888731e0 0x557588873600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.772+0000 7f0075e00640 1 -- 172.21.15.12:0/3281672788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557588878d40 msgr2=0x55758887b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:42.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.772+0000 7f0075e00640 1 --2- 172.21.15.12:0/3281672788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557588878d40 0x55758887b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.772+0000 7f0075e00640 1 -- 172.21.15.12:0/3281672788 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5575887992c0 con 0x557588873bd0 2024-03-23T23:05:42.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.772+0000 7f006aa00640 1 --2- 172.21.15.12:0/3802650944 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f00580a28a0 0x7f00580a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:42.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.772+0000 7f0075e00640 1 --2- 172.21.15.12:0/3281672788 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557588873bd0 0x5575888786d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f006402efd0 tx=0x7f0064000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb7a6491d2da95b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:42.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.772+0000 7f006aa00640 1 -- 172.21.15.12:0/3802650944 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f00600cee00 con 0x7f0070104af0 2024-03-23T23:05:42.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.773+0000 7f0076800640 1 --2- 172.21.15.12:0/3802650944 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f00580a28a0 0x7f00580a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:42.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.773+0000 7f005f000640 1 -- 172.21.15.12:0/3281672788 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064042070 con 0x557588873bd0 2024-03-23T23:05:42.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.773+0000 7f005f000640 1 -- 172.21.15.12:0/3281672788 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f006402cd40 con 0x557588873bd0 2024-03-23T23:05:42.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.773+0000 7f005f000640 1 -- 172.21.15.12:0/3281672788 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00640052d0 con 0x557588873bd0 2024-03-23T23:05:42.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.773+0000 7f007b3ff740 1 -- 172.21.15.12:0/3281672788 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557588873bd0 msgr2=0x5575888786d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:42.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.773+0000 7f0076800640 1 --2- 172.21.15.12:0/3802650944 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f00580a28a0 0x7f00580a4d60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x557588874430 tx=0x7f006c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:42.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.773+0000 7f007b3ff740 1 --2- 172.21.15.12:0/3281672788 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557588873bd0 0x5575888786d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f006402efd0 tx=0x7f0064000bc0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.774+0000 7f007b3ff740 1 -- 172.21.15.12:0/3281672788 shutdown_connections 2024-03-23T23:05:42.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.774+0000 7f007b3ff740 1 --2- 172.21.15.12:0/3281672788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557588878d40 0x55758887b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.774+0000 7f007b3ff740 1 --2- 172.21.15.12:0/3281672788 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557588873bd0 0x5575888786d0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.774+0000 7f007b3ff740 1 --2- 172.21.15.12:0/3281672788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5575888731e0 0x557588873600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.774+0000 7f007b3ff740 1 -- 172.21.15.12:0/3281672788 >> 172.21.15.12:0/3281672788 conn(0x55758886e750 msgr2=0x55758886eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:42.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.774+0000 7f007b3ff740 1 -- 172.21.15.12:0/3281672788 shutdown_connections 2024-03-23T23:05:42.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.774+0000 7f007b3ff740 1 -- 172.21.15.12:0/3281672788 wait complete. 2024-03-23T23:05:42.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.774+0000 7f007b3ff740 1 -- start start 2024-03-23T23:05:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.774+0000 7f007b3ff740 1 Processor -- start 2024-03-23T23:05:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.775+0000 7f007b3ff740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575888731e0 0x55758887d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.775+0000 7f007b3ff740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557588873bd0 0x55758887d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.775+0000 7f007b3ff740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557588878d40 0x557588884910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.775+0000 7f007b3ff740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x557588794a20 con 0x557588878d40 2024-03-23T23:05:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.775+0000 7f007b3ff740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575887d48d0 con 0x5575888731e0 2024-03-23T23:05:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.775+0000 7f007b3ff740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55758887bc90 con 0x557588873bd0 2024-03-23T23:05:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.775+0000 7f007b3ff740 10 client.0 _ll_register_callbacks cb 0x5575887d11c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.775+0000 7f0075e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575888731e0 0x55758887d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.775+0000 7f0075e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575888731e0 0x55758887d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:43718/0 (socket says 172.21.15.12:43718) 2024-03-23T23:05:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.775+0000 7f0075e00640 1 -- 172.21.15.12:0/2107396879 learned_addr learned my addr 172.21.15.12:0/2107396879 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.775+0000 7f0075400640 1 --2- 172.21.15.12:0/2107396879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557588873bd0 0x55758887d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.775+0000 7f0076800640 1 --2- 172.21.15.12:0/2107396879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557588878d40 0x557588884910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.775+0000 7f0075e00640 1 -- 172.21.15.12:0/2107396879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557588873bd0 msgr2=0x55758887d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.775+0000 7f0075e00640 1 --2- 172.21.15.12:0/2107396879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557588873bd0 0x55758887d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.775+0000 7f0075e00640 1 -- 172.21.15.12:0/2107396879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557588878d40 msgr2=0x557588884910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.775+0000 7f0075e00640 1 --2- 172.21.15.12:0/2107396879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557588878d40 0x557588884910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.775+0000 7f0075e00640 1 -- 172.21.15.12:0/2107396879 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5575884ed210 con 0x5575888731e0 2024-03-23T23:05:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.776+0000 7f0075e00640 1 --2- 172.21.15.12:0/2107396879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575888731e0 0x55758887d390 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f0064005ab0 tx=0x7f0064004da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddd7215836436120 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.776+0000 7f0053400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:05:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.776+0000 7f0053400640 1 -- 172.21.15.12:0/2107396879 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064042040 con 0x5575888731e0 2024-03-23T23:05:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.776+0000 7f0053400640 1 -- 172.21.15.12:0/2107396879 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f00640054e0 con 0x5575888731e0 2024-03-23T23:05:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.776+0000 7f007b3ff740 1 -- 172.21.15.12:0/2107396879 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f006402cd40 con 0x5575888731e0 2024-03-23T23:05:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.776+0000 7f0053400640 1 -- 172.21.15.12:0/2107396879 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0064047460 con 0x5575888731e0 2024-03-23T23:05:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.776+0000 7f007b3ff740 10 client.26294 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.776+0000 7f007b3ff740 1 -- 172.21.15.12:0/2107396879 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5575885b9620 con 0x5575888731e0 2024-03-23T23:05:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.777+0000 7f0053400640 1 -- 172.21.15.12:0/2107396879 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f006402cd40 con 0x5575888731e0 2024-03-23T23:05:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.777+0000 7f0053400640 1 -- 172.21.15.12:0/2107396879 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f0064034710 con 0x5575888731e0 2024-03-23T23:05:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.777+0000 7f007b3ff740 10 client.26294 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.777+0000 7f007b3ff740 10 client.26294 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.777+0000 7f007b3ff740 20 client.26294 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.777+0000 7f007b3ff740 1 --2- 172.21.15.12:0/2107396879 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557588888e00 0x55758888b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.777+0000 7f007b3ff740 4 client.26294 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:05:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.777+0000 7f007b3ff740 1 -- 172.21.15.12:0/2107396879 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5575887e5d70 con 0x557588888e00 2024-03-23T23:05:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.777+0000 7f0075400640 1 --2- 172.21.15.12:0/2107396879 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557588888e00 0x55758888b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.778+0000 7f0075400640 1 --2- 172.21.15.12:0/2107396879 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557588888e00 0x55758888b1f0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33e77017ef43677a server_cookie=fb15ddc4c4ed2f in_seq=0 out_seq=0 2024-03-23T23:05:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.778+0000 7f0053400640 10 client.26294 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:42.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.779+0000 7f0053400640 1 -- 172.21.15.12:0/2107396879 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5575887e5d70 con 0x557588888e00 2024-03-23T23:05:42.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.779+0000 7f0053400640 10 client.26294 handle_command_reply: tid=0 2024-03-23T23:05:42.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.779+0000 7f007b3ff740 1 client.26294 shutdown 2024-03-23T23:05:42.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.779+0000 7f007b3ff740 1 -- 172.21.15.12:0/2107396879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575888731e0 msgr2=0x55758887d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:42.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.779+0000 7f007b3ff740 1 --2- 172.21.15.12:0/2107396879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575888731e0 0x55758887d390 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f0064005ab0 tx=0x7f0064004da0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.779+0000 7f007b3ff740 1 -- 172.21.15.12:0/2107396879 shutdown_connections 2024-03-23T23:05:42.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.779+0000 7f007b3ff740 1 --2- 172.21.15.12:0/2107396879 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557588888e00 0x55758888b1f0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.779+0000 7f007b3ff740 1 --2- 172.21.15.12:0/2107396879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557588878d40 0x557588884910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.779+0000 7f007b3ff740 1 --2- 172.21.15.12:0/2107396879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557588873bd0 0x55758887d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.779+0000 7f007b3ff740 1 --2- 172.21.15.12:0/2107396879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575888731e0 0x55758887d390 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.779+0000 7f007b3ff740 1 -- 172.21.15.12:0/2107396879 >> 172.21.15.12:0/2107396879 conn(0x55758886e750 msgr2=0x55758886d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:42.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.779+0000 7f0053400640 0 client.26294 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:42.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.779+0000 7f007b3ff740 1 -- 172.21.15.12:0/2107396879 shutdown_connections 2024-03-23T23:05:42.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.779+0000 7f007b3ff740 1 -- 172.21.15.12:0/2107396879 wait complete. 2024-03-23T23:05:42.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.779+0000 7f007b3ff740 20 client.26294 trim_cache size 0 max 16384 2024-03-23T23:05:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.795+0000 7f007b3ff740 1 Processor -- start 2024-03-23T23:05:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.795+0000 7f007b3ff740 1 -- start start 2024-03-23T23:05:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.795+0000 7f007b3ff740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55758886e750 0x55758888f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.795+0000 7f007b3ff740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575888731e0 0x5575888848d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.795+0000 7f007b3ff740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557588873bd0 0x55758888fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.795+0000 7f007b3ff740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00701ad030 con 0x5575888731e0 2024-03-23T23:05:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.795+0000 7f007b3ff740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00701aab30 con 0x55758886e750 2024-03-23T23:05:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.795+0000 7f007b3ff740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00701a8bd0 con 0x557588873bd0 2024-03-23T23:05:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.795+0000 7f0075e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55758886e750 0x55758888f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.795+0000 7f0075e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55758886e750 0x55758888f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:43724/0 (socket says 172.21.15.12:43724) 2024-03-23T23:05:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.795+0000 7f0075e00640 1 -- 172.21.15.12:0/2514978106 learned_addr learned my addr 172.21.15.12:0/2514978106 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.795+0000 7f0076800640 1 --2- 172.21.15.12:0/2514978106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557588873bd0 0x55758888fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.795+0000 7f0075400640 1 --2- 172.21.15.12:0/2514978106 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575888731e0 0x5575888848d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.796+0000 7f0075e00640 1 -- 172.21.15.12:0/2514978106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557588873bd0 msgr2=0x55758888fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.796+0000 7f0075e00640 1 --2- 172.21.15.12:0/2514978106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557588873bd0 0x55758888fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.796+0000 7f0075e00640 1 -- 172.21.15.12:0/2514978106 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575888731e0 msgr2=0x5575888848d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.796+0000 7f0075e00640 1 --2- 172.21.15.12:0/2514978106 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575888731e0 0x5575888848d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.796+0000 7f0075e00640 1 -- 172.21.15.12:0/2514978106 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557588869a10 con 0x55758886e750 2024-03-23T23:05:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.796+0000 7f0075e00640 1 --2- 172.21.15.12:0/2514978106 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55758886e750 0x55758888f990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f006402c8f0 tx=0x7f0064005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b53fa095f393f477 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.796+0000 7f005e600640 1 -- 172.21.15.12:0/2514978106 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006405f070 con 0x55758886e750 2024-03-23T23:05:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.796+0000 7f005e600640 1 -- 172.21.15.12:0/2514978106 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f006402cd40 con 0x55758886e750 2024-03-23T23:05:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.796+0000 7f005e600640 1 -- 172.21.15.12:0/2514978106 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064042070 con 0x55758886e750 2024-03-23T23:05:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.797+0000 7f007b3ff740 1 -- 172.21.15.12:0/2514978106 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55758886e750 msgr2=0x55758888f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.797+0000 7f007b3ff740 1 --2- 172.21.15.12:0/2514978106 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55758886e750 0x55758888f990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f006402c8f0 tx=0x7f0064005b50 comp rx=0 tx=0).stop 2024-03-23T23:05:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.797+0000 7f007b3ff740 1 -- 172.21.15.12:0/2514978106 shutdown_connections 2024-03-23T23:05:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.797+0000 7f007b3ff740 1 --2- 172.21.15.12:0/2514978106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557588873bd0 0x55758888fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.797+0000 7f007b3ff740 1 --2- 172.21.15.12:0/2514978106 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575888731e0 0x5575888848d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.797+0000 7f007b3ff740 1 --2- 172.21.15.12:0/2514978106 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55758886e750 0x55758888f990 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.797+0000 7f007b3ff740 1 -- 172.21.15.12:0/2514978106 >> 172.21.15.12:0/2514978106 conn(0x55758888b560 msgr2=0x557588872240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.797+0000 7f007b3ff740 1 -- 172.21.15.12:0/2514978106 shutdown_connections 2024-03-23T23:05:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.797+0000 7f007b3ff740 1 -- 172.21.15.12:0/2514978106 wait complete. 2024-03-23T23:05:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.797+0000 7f007b3ff740 1 -- start start 2024-03-23T23:05:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.797+0000 7f007b3ff740 1 Processor -- start 2024-03-23T23:05:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.798+0000 7f007b3ff740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55758886e750 0x557588884260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.798+0000 7f007b3ff740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575888731e0 0x5575888847a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.798+0000 7f007b3ff740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557588873bd0 0x557588882870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.798+0000 7f007b3ff740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00701a91d0 con 0x557588873bd0 2024-03-23T23:05:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.798+0000 7f007b3ff740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00701aa4d0 con 0x5575888731e0 2024-03-23T23:05:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.798+0000 7f007b3ff740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00701a9810 con 0x55758886e750 2024-03-23T23:05:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.798+0000 7f007b3ff740 10 client.0 _ll_register_callbacks cb 0x7f006c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.798+0000 7f0075e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55758886e750 0x557588884260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.798+0000 7f0075400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575888731e0 0x5575888847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.798+0000 7f0075e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55758886e750 0x557588884260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55716/0 (socket says 172.21.15.12:55716) 2024-03-23T23:05:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.798+0000 7f0075e00640 1 -- 172.21.15.12:0/2417381580 learned_addr learned my addr 172.21.15.12:0/2417381580 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.798+0000 7f0076800640 1 --2- 172.21.15.12:0/2417381580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557588873bd0 0x557588882870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.798+0000 7f0075e00640 1 -- 172.21.15.12:0/2417381580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575888731e0 msgr2=0x5575888847a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.798+0000 7f0075e00640 1 --2- 172.21.15.12:0/2417381580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575888731e0 0x5575888847a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.799+0000 7f0075e00640 1 -- 172.21.15.12:0/2417381580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557588873bd0 msgr2=0x557588882870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.799+0000 7f0075e00640 1 --2- 172.21.15.12:0/2417381580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557588873bd0 0x557588882870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.799+0000 7f0075e00640 1 -- 172.21.15.12:0/2417381580 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5575888830a0 con 0x55758886e750 2024-03-23T23:05:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.799+0000 7f0076800640 1 --2- 172.21.15.12:0/2417381580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557588873bd0 0x557588882870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:05:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.799+0000 7f0075e00640 1 --2- 172.21.15.12:0/2417381580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55758886e750 0x557588884260 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0064009800 tx=0x7f0064005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc43c38518a9f8e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.799+0000 7f0053400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:05:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.799+0000 7f0053400640 1 -- 172.21.15.12:0/2417381580 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006405f030 con 0x55758886e750 2024-03-23T23:05:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.799+0000 7f0053400640 1 -- 172.21.15.12:0/2417381580 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0064034650 con 0x55758886e750 2024-03-23T23:05:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.799+0000 7f007b3ff740 1 -- 172.21.15.12:0/2417381580 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f006402cd40 con 0x55758886e750 2024-03-23T23:05:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.799+0000 7f0053400640 1 -- 172.21.15.12:0/2417381580 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0064042040 con 0x55758886e750 2024-03-23T23:05:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.799+0000 7f007b3ff740 10 client.26266 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.799+0000 7f007b3ff740 1 -- 172.21.15.12:0/2417381580 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5575888ed860 con 0x55758886e750 2024-03-23T23:05:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.800+0000 7f0053400640 1 -- 172.21.15.12:0/2417381580 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f006402cd40 con 0x55758886e750 2024-03-23T23:05:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.800+0000 7f0053400640 1 -- 172.21.15.12:0/2417381580 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f00640507e0 con 0x55758886e750 2024-03-23T23:05:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.800+0000 7f007b3ff740 10 client.26266 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.800+0000 7f007b3ff740 10 client.26266 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.800+0000 7f007b3ff740 20 client.26266 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.800+0000 7f007b3ff740 1 --2- 172.21.15.12:0/2417381580 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557588878d40 0x5575888f0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.800+0000 7f007b3ff740 4 client.26266 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:05:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.800+0000 7f007b3ff740 1 -- 172.21.15.12:0/2417381580 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5575887e00c0 con 0x557588878d40 2024-03-23T23:05:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.800+0000 7f0075400640 1 --2- 172.21.15.12:0/2417381580 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557588878d40 0x5575888f0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.801+0000 7f0075400640 1 --2- 172.21.15.12:0/2417381580 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557588878d40 0x5575888f0140 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a38459dbfe40429 server_cookie=e42691d92fcbefe3 in_seq=0 out_seq=0 2024-03-23T23:05:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.801+0000 7f0053400640 10 client.26266 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.812+0000 7f0053400640 1 -- 172.21.15.12:0/2417381580 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5575887e00c0 con 0x557588878d40 2024-03-23T23:05:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.812+0000 7f0053400640 10 client.26266 handle_command_reply: tid=0 2024-03-23T23:05:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.812+0000 7f007b3ff740 1 client.26266 shutdown 2024-03-23T23:05:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.812+0000 7f007b3ff740 1 -- 172.21.15.12:0/2417381580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55758886e750 msgr2=0x557588884260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.812+0000 7f007b3ff740 1 --2- 172.21.15.12:0/2417381580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55758886e750 0x557588884260 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0064009800 tx=0x7f0064005a20 comp rx=0 tx=0).stop 2024-03-23T23:05:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.813+0000 7f007b3ff740 1 -- 172.21.15.12:0/2417381580 shutdown_connections 2024-03-23T23:05:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.813+0000 7f007b3ff740 1 --2- 172.21.15.12:0/2417381580 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557588878d40 0x5575888f0140 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.813+0000 7f007b3ff740 1 --2- 172.21.15.12:0/2417381580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557588873bd0 0x557588882870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.813+0000 7f007b3ff740 1 --2- 172.21.15.12:0/2417381580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575888731e0 0x5575888847a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.813+0000 7f007b3ff740 1 --2- 172.21.15.12:0/2417381580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55758886e750 0x557588884260 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.813+0000 7f007b3ff740 1 -- 172.21.15.12:0/2417381580 >> 172.21.15.12:0/2417381580 conn(0x55758888b560 msgr2=0x55758886d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.813+0000 7f0053400640 0 client.26266 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.813+0000 7f007b3ff740 1 -- 172.21.15.12:0/2417381580 shutdown_connections 2024-03-23T23:05:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.813+0000 7f007b3ff740 1 -- 172.21.15.12:0/2417381580 wait complete. 2024-03-23T23:05:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.813+0000 7f007b3ff740 20 client.26266 trim_cache size 0 max 16384 2024-03-23T23:05:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:05:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:05:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:05:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:05:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.814+0000 7f0078800640 1 -- 172.21.15.12:0/3802650944 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f00580a28a0 msgr2=0x7f00580a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.814+0000 7f0078800640 1 --2- 172.21.15.12:0/3802650944 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f00580a28a0 0x7f00580a4d60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x557588874430 tx=0x7f006c009500 comp rx=0 tx=0).stop 2024-03-23T23:05:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.814+0000 7f0078800640 1 -- 172.21.15.12:0/3802650944 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0070104af0 msgr2=0x7f007019a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.814+0000 7f0078800640 1 --2- 172.21.15.12:0/3802650944 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0070104af0 0x7f007019a090 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f006000e9f0 tx=0x7f006000eeb0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.814+0000 7f0078800640 1 -- 172.21.15.12:0/3802650944 shutdown_connections 2024-03-23T23:05:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.814+0000 7f0078800640 1 --2- 172.21.15.12:0/3802650944 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f00580a28a0 0x7f00580a4d60 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.814+0000 7f0078800640 1 --2- 172.21.15.12:0/3802650944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00701054b0 0x7f00701a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.814+0000 7f0078800640 1 --2- 172.21.15.12:0/3802650944 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0070104af0 0x7f007019a090 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.814+0000 7f0078800640 1 --2- 172.21.15.12:0/3802650944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0070103900 0x7f0070199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.814+0000 7f0078800640 1 -- 172.21.15.12:0/3802650944 >> 172.21.15.12:0/3802650944 conn(0x7f00700fee90 msgr2=0x7f0070107d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.814+0000 7f0078800640 1 -- 172.21.15.12:0/3802650944 shutdown_connections 2024-03-23T23:05:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:42.814+0000 7f0078800640 1 -- 172.21.15.12:0/3802650944 wait complete. 2024-03-23T23:05:43.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:43 smithi012 ceph-mon[31695]: pgmap v507: 97 pgs: 97 active+clean; 2.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:05:43.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:43 smithi173 ceph-mon[36116]: pgmap v507: 97 pgs: 97 active+clean; 2.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:05:43.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:43 smithi203 ceph-mon[36693]: pgmap v507: 97 pgs: 97 active+clean; 2.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:05:45.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:45 smithi012 ceph-mon[31695]: pgmap v508: 97 pgs: 97 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:05:45.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:45 smithi173 ceph-mon[36116]: pgmap v508: 97 pgs: 97 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:05:45.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:45 smithi203 ceph-mon[36693]: pgmap v508: 97 pgs: 97 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:05:47.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:47 smithi012 ceph-mon[31695]: pgmap v509: 97 pgs: 97 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:05:47.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:47 smithi173 ceph-mon[36116]: pgmap v509: 97 pgs: 97 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:05:47.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:47 smithi203 ceph-mon[36693]: pgmap v509: 97 pgs: 97 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:05:48.161 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:05:48.162 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:05:48.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.400+0000 7fcbdd000640 1 -- 172.21.15.12:0/1189711652 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbd8075680 msgr2=0x7fcbd8073ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:48.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.400+0000 7fcbdd000640 1 --2- 172.21.15.12:0/1189711652 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbd8075680 0x7fcbd8073ad0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fcbc8009900 tx=0x7fcbc802bcd0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.400+0000 7fcbdd000640 1 -- 172.21.15.12:0/1189711652 shutdown_connections 2024-03-23T23:05:48.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.400+0000 7fcbdd000640 1 --2- 172.21.15.12:0/1189711652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbd8109540 0x7fcbd810b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.400+0000 7fcbdd000640 1 --2- 172.21.15.12:0/1189711652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbd80740a0 0x7fcbd8074500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.400+0000 7fcbdd000640 1 --2- 172.21.15.12:0/1189711652 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbd8075680 0x7fcbd8073ad0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.400+0000 7fcbdd000640 1 -- 172.21.15.12:0/1189711652 >> 172.21.15.12:0/1189711652 conn(0x7fcbd80fcc40 msgr2=0x7fcbd80ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:48.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.400+0000 7fcbdd000640 1 -- 172.21.15.12:0/1189711652 shutdown_connections 2024-03-23T23:05:48.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.400+0000 7fcbdd000640 1 -- 172.21.15.12:0/1189711652 wait complete. 2024-03-23T23:05:48.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.401+0000 7fcbdd000640 1 Processor -- start 2024-03-23T23:05:48.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.402+0000 7fcbdd000640 1 -- start start 2024-03-23T23:05:48.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.402+0000 7fcbdd000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbd80740a0 0x7fcbd8199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:48.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.402+0000 7fcbdd000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbd8075680 0x7fcbd819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:48.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.402+0000 7fcbdd000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbd8109540 0x7fcbd81a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:48.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.402+0000 7fcbdd000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd810f520 con 0x7fcbd8075680 2024-03-23T23:05:48.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.402+0000 7fcbdd000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd810f3a0 con 0x7fcbd8109540 2024-03-23T23:05:48.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.402+0000 7fcbdd000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd810f6a0 con 0x7fcbd80740a0 2024-03-23T23:05:48.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.402+0000 7fcbd6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbd8109540 0x7fcbd81a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:48.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.402+0000 7fcbcd400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbd8075680 0x7fcbd819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:48.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.402+0000 7fcbd6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbd8109540 0x7fcbd81a1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50010/0 (socket says 172.21.15.12:50010) 2024-03-23T23:05:48.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.402+0000 7fcbd6800640 1 -- 172.21.15.12:0/1802614945 learned_addr learned my addr 172.21.15.12:0/1802614945 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:48.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.402+0000 7fcbd5e00640 1 --2- 172.21.15.12:0/1802614945 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbd80740a0 0x7fcbd8199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:48.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.403+0000 7fcbd6800640 1 -- 172.21.15.12:0/1802614945 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbd80740a0 msgr2=0x7fcbd8199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:48.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.403+0000 7fcbd6800640 1 --2- 172.21.15.12:0/1802614945 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbd80740a0 0x7fcbd8199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.403+0000 7fcbd6800640 1 -- 172.21.15.12:0/1802614945 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbd8075680 msgr2=0x7fcbd819a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:48.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.403+0000 7fcbd6800640 1 --2- 172.21.15.12:0/1802614945 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbd8075680 0x7fcbd819a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.403+0000 7fcbd6800640 1 -- 172.21.15.12:0/1802614945 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd81a1830 con 0x7fcbd8109540 2024-03-23T23:05:48.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.403+0000 7fcbcd400640 1 --2- 172.21.15.12:0/1802614945 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbd8075680 0x7fcbd819a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:48.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.403+0000 7fcbd6800640 1 --2- 172.21.15.12:0/1802614945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbd8109540 0x7fcbd81a1190 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fcbd000efd0 tx=0x7fcbd000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:48.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.403+0000 7fcbcf400640 1 -- 172.21.15.12:0/1802614945 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd0003b70 con 0x7fcbd8109540 2024-03-23T23:05:48.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.403+0000 7fcbcf400640 1 -- 172.21.15.12:0/1802614945 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbd0003d10 con 0x7fcbd8109540 2024-03-23T23:05:48.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.403+0000 7fcbdd000640 1 -- 172.21.15.12:0/1802614945 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbd81a1b20 con 0x7fcbd8109540 2024-03-23T23:05:48.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.403+0000 7fcbcf400640 1 -- 172.21.15.12:0/1802614945 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd00109f0 con 0x7fcbd8109540 2024-03-23T23:05:48.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.403+0000 7fcbdd000640 1 -- 172.21.15.12:0/1802614945 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbd81a2070 con 0x7fcbd8109540 2024-03-23T23:05:48.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.406+0000 7fcbcf400640 1 -- 172.21.15.12:0/1802614945 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcbd0004020 con 0x7fcbd8109540 2024-03-23T23:05:48.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.407+0000 7fcbdfd7f740 1 Processor -- start 2024-03-23T23:05:48.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.407+0000 7fcbdfd7f740 1 -- start start 2024-03-23T23:05:48.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.407+0000 7fcbdfd7f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdfd475e80 0x55fdfd4762a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:48.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.407+0000 7fcbdfd7f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd476870 0x55fdfd47b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.407+0000 7fcbdfd7f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fdfd47b9e0 0x55fdfd47de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.407+0000 7fcbdfd7f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdfd3db250 con 0x55fdfd47b9e0 2024-03-23T23:05:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.407+0000 7fcbdfd7f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdfd383a30 con 0x55fdfd475e80 2024-03-23T23:05:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.407+0000 7fcbdfd7f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdfd3d72f0 con 0x55fdfd476870 2024-03-23T23:05:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.407+0000 7fcbd5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd476870 0x55fdfd47b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.407+0000 7fcbd5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd476870 0x55fdfd47b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45660/0 (socket says 172.21.15.12:45660) 2024-03-23T23:05:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.407+0000 7fcbd5e00640 1 -- 172.21.15.12:0/4050193071 learned_addr learned my addr 172.21.15.12:0/4050193071 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.408+0000 7fcbcd400640 1 --2- 172.21.15.12:0/4050193071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdfd475e80 0x55fdfd4762a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.408+0000 7fcbcd400640 1 --2- 172.21.15.12:0/4050193071 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fdfd47b9e0 0x55fdfd47de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.408+0000 7fcbd5e00640 1 -- 172.21.15.12:0/4050193071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdfd475e80 msgr2=0x55fdfd4762a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.408+0000 7fcbd5e00640 1 --2- 172.21.15.12:0/4050193071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdfd475e80 0x55fdfd4762a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.408+0000 7fcbd5e00640 1 -- 172.21.15.12:0/4050193071 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fdfd47b9e0 msgr2=0x55fdfd47de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.408+0000 7fcbd5e00640 1 --2- 172.21.15.12:0/4050193071 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fdfd47b9e0 0x55fdfd47de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.408+0000 7fcbd5e00640 1 -- 172.21.15.12:0/4050193071 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fdfd36e690 con 0x55fdfd476870 2024-03-23T23:05:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.408+0000 7fcbcd400640 1 --2- 172.21.15.12:0/4050193071 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fdfd47b9e0 0x55fdfd47de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:05:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.408+0000 7fcbd5e00640 1 --2- 172.21.15.12:0/4050193071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd476870 0x55fdfd47b370 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fcbc8036040 tx=0x7fcbc8009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8424a1a5f150ff13 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.408+0000 7fcbcf400640 1 --2- 172.21.15.12:0/1802614945 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcbbc0a2880 0x7fcbbc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.408+0000 7fcbcf400640 1 -- 172.21.15.12:0/1802614945 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fcbd00ceed0 con 0x7fcbd8109540 2024-03-23T23:05:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.408+0000 7fcbc7400640 1 -- 172.21.15.12:0/4050193071 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc802cc20 con 0x55fdfd476870 2024-03-23T23:05:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.409+0000 7fcbc7400640 1 -- 172.21.15.12:0/4050193071 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbc8005040 con 0x55fdfd476870 2024-03-23T23:05:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.409+0000 7fcbc7400640 1 -- 172.21.15.12:0/4050193071 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc8004a80 con 0x55fdfd476870 2024-03-23T23:05:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.409+0000 7fcbcd400640 1 --2- 172.21.15.12:0/1802614945 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcbbc0a2880 0x7fcbbc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.409+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/4050193071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd476870 msgr2=0x55fdfd47b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.409+0000 7fcbdfd7f740 1 --2- 172.21.15.12:0/4050193071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd476870 0x55fdfd47b370 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fcbc8036040 tx=0x7fcbc8009560 comp rx=0 tx=0).stop 2024-03-23T23:05:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.409+0000 7fcbcd400640 1 --2- 172.21.15.12:0/1802614945 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcbbc0a2880 0x7fcbbc0a4d40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x55fdfd4770d0 tx=0x7fcbc0004730 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.409+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/4050193071 shutdown_connections 2024-03-23T23:05:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.410+0000 7fcbdfd7f740 1 --2- 172.21.15.12:0/4050193071 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fdfd47b9e0 0x55fdfd47de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.410+0000 7fcbdfd7f740 1 --2- 172.21.15.12:0/4050193071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd476870 0x55fdfd47b370 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.410+0000 7fcbdfd7f740 1 --2- 172.21.15.12:0/4050193071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdfd475e80 0x55fdfd4762a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.410+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/4050193071 >> 172.21.15.12:0/4050193071 conn(0x55fdfd43e200 msgr2=0x55fdfd4717f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.410+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/4050193071 shutdown_connections 2024-03-23T23:05:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.410+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/4050193071 wait complete. 2024-03-23T23:05:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.410+0000 7fcbdfd7f740 1 -- start start 2024-03-23T23:05:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.410+0000 7fcbdfd7f740 1 Processor -- start 2024-03-23T23:05:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.411+0000 7fcbdfd7f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd475e80 0x55fdfd47fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.411+0000 7fcbdfd7f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdfd476870 0x55fdfd480310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.411+0000 7fcbdfd7f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fdfd47b9e0 0x55fdfd487350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.411+0000 7fcbdfd7f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdfd47e930 con 0x55fdfd47b9e0 2024-03-23T23:05:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.411+0000 7fcbdfd7f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdfd396e80 con 0x55fdfd476870 2024-03-23T23:05:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.411+0000 7fcbdfd7f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdfd47eab0 con 0x55fdfd475e80 2024-03-23T23:05:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.411+0000 7fcbdfd7f740 10 client.0 _ll_register_callbacks cb 0x55fdfd3a9130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.411+0000 7fcbd5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd475e80 0x55fdfd47fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.411+0000 7fcbcd400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdfd476870 0x55fdfd480310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.411+0000 7fcbd5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd475e80 0x55fdfd47fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45666/0 (socket says 172.21.15.12:45666) 2024-03-23T23:05:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.411+0000 7fcbd6800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fdfd47b9e0 0x55fdfd487350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.411+0000 7fcbd5e00640 1 -- 172.21.15.12:0/687631807 learned_addr learned my addr 172.21.15.12:0/687631807 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.411+0000 7fcbd5e00640 1 -- 172.21.15.12:0/687631807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdfd476870 msgr2=0x55fdfd480310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.411+0000 7fcbd5e00640 1 --2- 172.21.15.12:0/687631807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdfd476870 0x55fdfd480310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.411+0000 7fcbd5e00640 1 -- 172.21.15.12:0/687631807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fdfd47b9e0 msgr2=0x55fdfd487350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.411+0000 7fcbd5e00640 1 --2- 172.21.15.12:0/687631807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fdfd47b9e0 0x55fdfd487350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.411+0000 7fcbd5e00640 1 -- 172.21.15.12:0/687631807 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fdfd1b97d0 con 0x55fdfd475e80 2024-03-23T23:05:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.412+0000 7fcbd5e00640 1 --2- 172.21.15.12:0/687631807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd475e80 0x55fdfd47fdd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fcbc802c850 tx=0x7fcbc8005420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f30ba5cf373d787 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.412+0000 7fcbbbe00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:05:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.412+0000 7fcbbbe00640 1 -- 172.21.15.12:0/687631807 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc8004450 con 0x55fdfd475e80 2024-03-23T23:05:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.412+0000 7fcbbbe00640 1 -- 172.21.15.12:0/687631807 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fcbc80047c0 con 0x55fdfd475e80 2024-03-23T23:05:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.412+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/687631807 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcbc8005040 con 0x55fdfd475e80 2024-03-23T23:05:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.412+0000 7fcbbbe00640 1 -- 172.21.15.12:0/687631807 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcbc80049c0 con 0x55fdfd475e80 2024-03-23T23:05:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.412+0000 7fcbdfd7f740 10 client.26278 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.412+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/687631807 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55fdfd0ef3f0 con 0x55fdfd475e80 2024-03-23T23:05:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.413+0000 7fcbbbe00640 1 -- 172.21.15.12:0/687631807 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbc8005040 con 0x55fdfd475e80 2024-03-23T23:05:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.413+0000 7fcbbbe00640 1 -- 172.21.15.12:0/687631807 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fcbc8046480 con 0x55fdfd475e80 2024-03-23T23:05:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.413+0000 7fcbdfd7f740 10 client.26278 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.413+0000 7fcbdfd7f740 10 client.26278 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.413+0000 7fcbdfd7f740 20 client.26278 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.413+0000 7fcbdfd7f740 1 --2- 172.21.15.12:0/687631807 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fdfd48b7b0 0x55fdfd48dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.413+0000 7fcbdfd7f740 4 client.26278 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:05:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.413+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/687631807 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fdfd3e8790 con 0x55fdfd48b7b0 2024-03-23T23:05:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.413+0000 7fcbcd400640 1 --2- 172.21.15.12:0/687631807 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fdfd48b7b0 0x55fdfd48dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.414+0000 7fcbcd400640 1 --2- 172.21.15.12:0/687631807 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fdfd48b7b0 0x55fdfd48dba0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29c99b64becc06b6 server_cookie=ecffbbcf5f0b3db8 in_seq=0 out_seq=0 2024-03-23T23:05:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.414+0000 7fcbbbe00640 10 client.26278 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.415+0000 7fcbbbe00640 1 -- 172.21.15.12:0/687631807 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55fdfd3e8790 con 0x55fdfd48b7b0 2024-03-23T23:05:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.415+0000 7fcbbbe00640 10 client.26278 handle_command_reply: tid=0 2024-03-23T23:05:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.415+0000 7fcbdfd7f740 1 client.26278 shutdown 2024-03-23T23:05:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.415+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/687631807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd475e80 msgr2=0x55fdfd47fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.415+0000 7fcbdfd7f740 1 --2- 172.21.15.12:0/687631807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd475e80 0x55fdfd47fdd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fcbc802c850 tx=0x7fcbc8005420 comp rx=0 tx=0).stop 2024-03-23T23:05:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.415+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/687631807 shutdown_connections 2024-03-23T23:05:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.415+0000 7fcbdfd7f740 1 --2- 172.21.15.12:0/687631807 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fdfd48b7b0 0x55fdfd48dba0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.415+0000 7fcbdfd7f740 1 --2- 172.21.15.12:0/687631807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fdfd47b9e0 0x55fdfd487350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.415+0000 7fcbdfd7f740 1 --2- 172.21.15.12:0/687631807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdfd476870 0x55fdfd480310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.415+0000 7fcbdfd7f740 1 --2- 172.21.15.12:0/687631807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd475e80 0x55fdfd47fdd0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.415+0000 7fcbbbe00640 0 client.26278 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:48.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.415+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/687631807 >> 172.21.15.12:0/687631807 conn(0x55fdfd43e200 msgr2=0x55fdfd470a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:48.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.416+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/687631807 shutdown_connections 2024-03-23T23:05:48.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.416+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/687631807 wait complete. 2024-03-23T23:05:48.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.416+0000 7fcbdfd7f740 20 client.26278 trim_cache size 0 max 16384 2024-03-23T23:05:48.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.433+0000 7fcbdfd7f740 1 Processor -- start 2024-03-23T23:05:48.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.433+0000 7fcbdfd7f740 1 -- start start 2024-03-23T23:05:48.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.434+0000 7fcbdfd7f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdfd475e80 0x55fdfd477ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:48.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.434+0000 7fcbdfd7f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd476870 0x55fdfd487310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:48.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.434+0000 7fcbdfd7f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fdfd47b9e0 0x55fdfd492370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:48.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.434+0000 7fcbdfd7f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd81ad000 con 0x55fdfd47b9e0 2024-03-23T23:05:48.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.434+0000 7fcbdfd7f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd81aab00 con 0x55fdfd475e80 2024-03-23T23:05:48.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.434+0000 7fcbdfd7f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd81a8ba0 con 0x55fdfd476870 2024-03-23T23:05:48.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.434+0000 7fcbcd400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd476870 0x55fdfd487310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:48.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.434+0000 7fcbcd400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd476870 0x55fdfd487310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45674/0 (socket says 172.21.15.12:45674) 2024-03-23T23:05:48.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.434+0000 7fcbcd400640 1 -- 172.21.15.12:0/3320344674 learned_addr learned my addr 172.21.15.12:0/3320344674 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:48.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.434+0000 7fcbd6800640 1 --2- 172.21.15.12:0/3320344674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fdfd47b9e0 0x55fdfd492370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:48.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.434+0000 7fcbd5e00640 1 --2- 172.21.15.12:0/3320344674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdfd475e80 0x55fdfd477ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:48.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.434+0000 7fcbcd400640 1 -- 172.21.15.12:0/3320344674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdfd475e80 msgr2=0x55fdfd477ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:48.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.434+0000 7fcbcd400640 1 --2- 172.21.15.12:0/3320344674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdfd475e80 0x55fdfd477ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.434+0000 7fcbcd400640 1 -- 172.21.15.12:0/3320344674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fdfd47b9e0 msgr2=0x55fdfd492370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:48.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.434+0000 7fcbcd400640 1 --2- 172.21.15.12:0/3320344674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fdfd47b9e0 0x55fdfd492370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.434+0000 7fcbcd400640 1 -- 172.21.15.12:0/3320344674 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fdfd0af0f0 con 0x55fdfd476870 2024-03-23T23:05:48.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.434+0000 7fcbd6800640 1 --2- 172.21.15.12:0/3320344674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fdfd47b9e0 0x55fdfd492370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:05:48.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.435+0000 7fcbcd400640 1 --2- 172.21.15.12:0/3320344674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd476870 0x55fdfd487310 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fcbc000bd60 tx=0x7fcbc0015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b5d2d25170ba840 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:48.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.435+0000 7fcbc6a00640 1 -- 172.21.15.12:0/3320344674 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc002a250 con 0x55fdfd476870 2024-03-23T23:05:48.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.435+0000 7fcbc6a00640 1 -- 172.21.15.12:0/3320344674 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbc0005500 con 0x55fdfd476870 2024-03-23T23:05:48.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.435+0000 7fcbc6a00640 1 -- 172.21.15.12:0/3320344674 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc002a950 con 0x55fdfd476870 2024-03-23T23:05:48.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.435+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/3320344674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd476870 msgr2=0x55fdfd487310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:48.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.435+0000 7fcbdfd7f740 1 --2- 172.21.15.12:0/3320344674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd476870 0x55fdfd487310 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fcbc000bd60 tx=0x7fcbc0015000 comp rx=0 tx=0).stop 2024-03-23T23:05:48.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.436+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/3320344674 shutdown_connections 2024-03-23T23:05:48.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.436+0000 7fcbdfd7f740 1 --2- 172.21.15.12:0/3320344674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fdfd47b9e0 0x55fdfd492370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.436+0000 7fcbdfd7f740 1 --2- 172.21.15.12:0/3320344674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd476870 0x55fdfd487310 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.436+0000 7fcbdfd7f740 1 --2- 172.21.15.12:0/3320344674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdfd475e80 0x55fdfd477ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.436+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/3320344674 >> 172.21.15.12:0/3320344674 conn(0x55fdfd43e200 msgr2=0x55fdfd470a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:48.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.436+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/3320344674 shutdown_connections 2024-03-23T23:05:48.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.436+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/3320344674 wait complete. 2024-03-23T23:05:48.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.436+0000 7fcbdfd7f740 1 -- start start 2024-03-23T23:05:48.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.436+0000 7fcbdfd7f740 1 Processor -- start 2024-03-23T23:05:48.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.437+0000 7fcbdfd7f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd475e80 0x55fdfd484960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:48.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.437+0000 7fcbdfd7f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fdfd476870 0x55fdfd484ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:48.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.437+0000 7fcbdfd7f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdfd47b9e0 0x55fdfd485df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:48.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.437+0000 7fcbdfd7f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd81a91a0 con 0x55fdfd476870 2024-03-23T23:05:48.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.437+0000 7fcbdfd7f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd81aa4a0 con 0x55fdfd47b9e0 2024-03-23T23:05:48.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.437+0000 7fcbdfd7f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd81a97e0 con 0x55fdfd475e80 2024-03-23T23:05:48.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.437+0000 7fcbdfd7f740 10 client.0 _ll_register_callbacks cb 0x7fcbc802c700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:48.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.437+0000 7fcbcd400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fdfd476870 0x55fdfd484ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:48.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.437+0000 7fcbd5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd475e80 0x55fdfd484960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:48.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.437+0000 7fcbd6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdfd47b9e0 0x55fdfd485df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:48.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.437+0000 7fcbcd400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fdfd476870 0x55fdfd484ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52574/0 (socket says 172.21.15.12:52574) 2024-03-23T23:05:48.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.437+0000 7fcbd5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd475e80 0x55fdfd484960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45682/0 (socket says 172.21.15.12:45682) 2024-03-23T23:05:48.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.437+0000 7fcbcd400640 1 -- 172.21.15.12:0/4070592284 learned_addr learned my addr 172.21.15.12:0/4070592284 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:48.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.438+0000 7fcbd6800640 1 -- 172.21.15.12:0/4070592284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd475e80 msgr2=0x55fdfd484960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:48.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.438+0000 7fcbd6800640 1 --2- 172.21.15.12:0/4070592284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd475e80 0x55fdfd484960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.438+0000 7fcbd6800640 1 -- 172.21.15.12:0/4070592284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fdfd476870 msgr2=0x55fdfd484ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:48.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.438+0000 7fcbd6800640 1 --2- 172.21.15.12:0/4070592284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fdfd476870 0x55fdfd484ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.438+0000 7fcbd6800640 1 -- 172.21.15.12:0/4070592284 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fdfd4863e0 con 0x55fdfd47b9e0 2024-03-23T23:05:48.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.438+0000 7fcbd5e00640 1 --2- 172.21.15.12:0/4070592284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd475e80 0x55fdfd484960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:48.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.438+0000 7fcbd6800640 1 --2- 172.21.15.12:0/4070592284 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdfd47b9e0 0x55fdfd485df0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fcbd00cc9e0 tx=0x7fcbd0004920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a080dde456ce81bb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:48.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.438+0000 7fcbbbe00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:05:48.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.438+0000 7fcbbbe00640 1 -- 172.21.15.12:0/4070592284 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd00b3070 con 0x55fdfd47b9e0 2024-03-23T23:05:48.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.438+0000 7fcbbbe00640 1 -- 172.21.15.12:0/4070592284 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fcbd0097e60 con 0x55fdfd47b9e0 2024-03-23T23:05:48.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.438+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/4070592284 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcbc0005500 con 0x55fdfd47b9e0 2024-03-23T23:05:48.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.438+0000 7fcbbbe00640 1 -- 172.21.15.12:0/4070592284 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcbd009bf80 con 0x55fdfd47b9e0 2024-03-23T23:05:48.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.438+0000 7fcbdfd7f740 10 client.26318 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:48.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.438+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/4070592284 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55fdfd4e6f40 con 0x55fdfd47b9e0 2024-03-23T23:05:48.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.439+0000 7fcbbbe00640 1 -- 172.21.15.12:0/4070592284 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbc0005500 con 0x55fdfd47b9e0 2024-03-23T23:05:48.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.439+0000 7fcbbbe00640 1 -- 172.21.15.12:0/4070592284 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fcbd001f020 con 0x55fdfd47b9e0 2024-03-23T23:05:48.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.439+0000 7fcbdfd7f740 10 client.26318 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:48.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.439+0000 7fcbdfd7f740 10 client.26318 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:48.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.439+0000 7fcbdfd7f740 20 client.26318 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:48.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.439+0000 7fcbdfd7f740 1 --2- 172.21.15.12:0/4070592284 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fdfd4e7680 0x55fdfd4e9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:48.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.439+0000 7fcbdfd7f740 4 client.26318 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:05:48.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.439+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/4070592284 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55fdfd3e2ae0 con 0x55fdfd4e7680 2024-03-23T23:05:48.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.439+0000 7fcbd5e00640 1 --2- 172.21.15.12:0/4070592284 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fdfd4e7680 0x55fdfd4e9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:48.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.439+0000 7fcbd5e00640 1 --2- 172.21.15.12:0/4070592284 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fdfd4e7680 0x55fdfd4e9b00 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcb63f118ef7e417 server_cookie=f50361578dcf413 in_seq=0 out_seq=0 2024-03-23T23:05:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.440+0000 7fcbbbe00640 10 client.26318 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.440+0000 7fcbbbe00640 1 -- 172.21.15.12:0/4070592284 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55fdfd3e2ae0 con 0x55fdfd4e7680 2024-03-23T23:05:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.440+0000 7fcbbbe00640 10 client.26318 handle_command_reply: tid=0 2024-03-23T23:05:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.440+0000 7fcbdfd7f740 1 client.26318 shutdown 2024-03-23T23:05:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.440+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/4070592284 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdfd47b9e0 msgr2=0x55fdfd485df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.440+0000 7fcbdfd7f740 1 --2- 172.21.15.12:0/4070592284 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdfd47b9e0 0x55fdfd485df0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fcbd00cc9e0 tx=0x7fcbd0004920 comp rx=0 tx=0).stop 2024-03-23T23:05:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.440+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/4070592284 shutdown_connections 2024-03-23T23:05:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.440+0000 7fcbdfd7f740 1 --2- 172.21.15.12:0/4070592284 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fdfd4e7680 0x55fdfd4e9b00 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.440+0000 7fcbdfd7f740 1 --2- 172.21.15.12:0/4070592284 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdfd47b9e0 0x55fdfd485df0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.440+0000 7fcbdfd7f740 1 --2- 172.21.15.12:0/4070592284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fdfd476870 0x55fdfd484ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.440+0000 7fcbbbe00640 0 client.26318 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.440+0000 7fcbdfd7f740 1 --2- 172.21.15.12:0/4070592284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdfd475e80 0x55fdfd484960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.440+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/4070592284 >> 172.21.15.12:0/4070592284 conn(0x55fdfd43e200 msgr2=0x55fdfd470a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:48.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.440+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/4070592284 shutdown_connections 2024-03-23T23:05:48.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.440+0000 7fcbdfd7f740 1 -- 172.21.15.12:0/4070592284 wait complete. 2024-03-23T23:05:48.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.440+0000 7fcbdfd7f740 20 client.26318 trim_cache size 0 max 16384 2024-03-23T23:05:48.444 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:05:48.444 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-03-23T23:05:48.444 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-03-23T23:05:48.444 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:05:48.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.441+0000 7fcbdd000640 1 -- 172.21.15.12:0/1802614945 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcbbc0a2880 msgr2=0x7fcbbc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:48.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.441+0000 7fcbdd000640 1 --2- 172.21.15.12:0/1802614945 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcbbc0a2880 0x7fcbbc0a4d40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x55fdfd4770d0 tx=0x7fcbc0004730 comp rx=0 tx=0).stop 2024-03-23T23:05:48.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.441+0000 7fcbdd000640 1 -- 172.21.15.12:0/1802614945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbd8109540 msgr2=0x7fcbd81a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:48.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.441+0000 7fcbdd000640 1 --2- 172.21.15.12:0/1802614945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbd8109540 0x7fcbd81a1190 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fcbd000efd0 tx=0x7fcbd000c6a0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.441+0000 7fcbdd000640 1 -- 172.21.15.12:0/1802614945 shutdown_connections 2024-03-23T23:05:48.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.441+0000 7fcbdd000640 1 --2- 172.21.15.12:0/1802614945 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcbbc0a2880 0x7fcbbc0a4d40 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.441+0000 7fcbdd000640 1 --2- 172.21.15.12:0/1802614945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbd8109540 0x7fcbd81a1190 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.441+0000 7fcbdd000640 1 --2- 172.21.15.12:0/1802614945 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbd8075680 0x7fcbd819a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.441+0000 7fcbdd000640 1 --2- 172.21.15.12:0/1802614945 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbd80740a0 0x7fcbd8199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.441+0000 7fcbdd000640 1 -- 172.21.15.12:0/1802614945 >> 172.21.15.12:0/1802614945 conn(0x7fcbd80fcc40 msgr2=0x7fcbd81054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:48.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.441+0000 7fcbdd000640 1 -- 172.21.15.12:0/1802614945 shutdown_connections 2024-03-23T23:05:48.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.441+0000 7fcbdd000640 1 -- 172.21.15.12:0/1802614945 wait complete. 2024-03-23T23:05:48.457 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T23:05:48.457 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:05:48.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.682+0000 7f2516a00640 1 -- 172.21.15.12:0/1390901862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2510105490 msgr2=0x7f251010bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:48.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.682+0000 7f2516a00640 1 --2- 172.21.15.12:0/1390901862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2510105490 0x7f251010bd20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f2508009ca0 tx=0x7f250802c0f0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.682+0000 7f2516a00640 1 -- 172.21.15.12:0/1390901862 shutdown_connections 2024-03-23T23:05:48.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.682+0000 7f2516a00640 1 --2- 172.21.15.12:0/1390901862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2510105490 0x7f251010bd20 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.682+0000 7f2516a00640 1 --2- 172.21.15.12:0/1390901862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2510104ad0 0x7f2510104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.682+0000 7f2516a00640 1 --2- 172.21.15.12:0/1390901862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25101038e0 0x7f2510103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.682+0000 7f2516a00640 1 -- 172.21.15.12:0/1390901862 >> 172.21.15.12:0/1390901862 conn(0x7f25100fee90 msgr2=0x7f25101012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:48.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.682+0000 7f2516a00640 1 -- 172.21.15.12:0/1390901862 shutdown_connections 2024-03-23T23:05:48.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.682+0000 7f2516a00640 1 -- 172.21.15.12:0/1390901862 wait complete. 2024-03-23T23:05:48.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.683+0000 7f2516a00640 1 Processor -- start 2024-03-23T23:05:48.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.683+0000 7f2516a00640 1 -- start start 2024-03-23T23:05:48.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.683+0000 7f2516a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25101038e0 0x7f2510199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:48.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.683+0000 7f2516a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2510104ad0 0x7f251019a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:48.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.683+0000 7f2516a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2510105490 0x7f25101a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:48.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.683+0000 7f2516a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251010f540 con 0x7f2510104ad0 2024-03-23T23:05:48.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.683+0000 7f2516a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251010f3c0 con 0x7f2510105490 2024-03-23T23:05:48.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.683+0000 7f2516a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251010f6c0 con 0x7f25101038e0 2024-03-23T23:05:48.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.684+0000 7f2507400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2510104ad0 0x7f251019a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:48.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.684+0000 7f250fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25101038e0 0x7f2510199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:48.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.684+0000 7f250fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25101038e0 0x7f2510199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45704/0 (socket says 172.21.15.12:45704) 2024-03-23T23:05:48.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.684+0000 7f250fe00640 1 -- 172.21.15.12:0/1622021157 learned_addr learned my addr 172.21.15.12:0/1622021157 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:48.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.684+0000 7f2514a00640 1 --2- 172.21.15.12:0/1622021157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2510105490 0x7f25101a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:48.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.684+0000 7f250fe00640 1 -- 172.21.15.12:0/1622021157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2510105490 msgr2=0x7f25101a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:48.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.684+0000 7f250fe00640 1 --2- 172.21.15.12:0/1622021157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2510105490 0x7f25101a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.684+0000 7f250fe00640 1 -- 172.21.15.12:0/1622021157 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2510104ad0 msgr2=0x7f251019a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:48.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.684+0000 7f250fe00640 1 --2- 172.21.15.12:0/1622021157 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2510104ad0 0x7f251019a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.684+0000 7f250fe00640 1 -- 172.21.15.12:0/1622021157 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25101a1770 con 0x7f25101038e0 2024-03-23T23:05:48.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.684+0000 7f250fe00640 1 --2- 172.21.15.12:0/1622021157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25101038e0 0x7f2510199b50 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f250000e9c0 tx=0x7f250000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:48.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.685+0000 7f250d600640 1 -- 172.21.15.12:0/1622021157 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250000cd80 con 0x7f25101038e0 2024-03-23T23:05:48.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.685+0000 7f250d600640 1 -- 172.21.15.12:0/1622021157 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f25000035f0 con 0x7f25101038e0 2024-03-23T23:05:48.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.685+0000 7f2516a00640 1 -- 172.21.15.12:0/1622021157 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25101a1a60 con 0x7f25101038e0 2024-03-23T23:05:48.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.685+0000 7f250d600640 1 -- 172.21.15.12:0/1622021157 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2500010690 con 0x7f25101038e0 2024-03-23T23:05:48.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.685+0000 7f2516a00640 1 -- 172.21.15.12:0/1622021157 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25101a1fb0 con 0x7f25101038e0 2024-03-23T23:05:48.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.686+0000 7f2516a00640 1 -- 172.21.15.12:0/1622021157 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25100688e0 con 0x7f25101038e0 2024-03-23T23:05:48.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.688+0000 7f250d600640 1 -- 172.21.15.12:0/1622021157 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f25000037a0 con 0x7f25101038e0 2024-03-23T23:05:48.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.689+0000 7f250d600640 1 --2- 172.21.15.12:0/1622021157 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f24f40a2800 0x7f24f40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:48.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.690+0000 7f250d600640 1 -- 172.21.15.12:0/1622021157 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f25000cee60 con 0x7f25101038e0 2024-03-23T23:05:48.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.690+0000 7f2507400640 1 --2- 172.21.15.12:0/1622021157 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f24f40a2800 0x7f24f40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:48.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.690+0000 7f2507400640 1 --2- 172.21.15.12:0/1622021157 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f24f40a2800 0x7f24f40a4cc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f24f80095f0 tx=0x7f24f8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:48.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.693+0000 7f250d600640 1 -- 172.21.15.12:0/1622021157 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f250009beb0 con 0x7f25101038e0 2024-03-23T23:05:48.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.938+0000 7f2516a00640 1 -- 172.21.15.12:0/1622021157 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f251010ac00 con 0x7f25101038e0 2024-03-23T23:05:48.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.940+0000 7f250d600640 1 -- 172.21.15.12:0/1622021157 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f250009b7b0 con 0x7f25101038e0 2024-03-23T23:05:48.939 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:05:48.939 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:05:48.939 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:05:48.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.943+0000 7f2516a00640 1 -- 172.21.15.12:0/1622021157 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f24f40a2800 msgr2=0x7f24f40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:48.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.943+0000 7f2516a00640 1 --2- 172.21.15.12:0/1622021157 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f24f40a2800 0x7f24f40a4cc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f24f80095f0 tx=0x7f24f8005f90 comp rx=0 tx=0).stop 2024-03-23T23:05:48.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.943+0000 7f2516a00640 1 -- 172.21.15.12:0/1622021157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25101038e0 msgr2=0x7f2510199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:48.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.943+0000 7f2516a00640 1 --2- 172.21.15.12:0/1622021157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25101038e0 0x7f2510199b50 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f250000e9c0 tx=0x7f250000ee80 comp rx=0 tx=0).stop 2024-03-23T23:05:48.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.943+0000 7f2516a00640 1 -- 172.21.15.12:0/1622021157 shutdown_connections 2024-03-23T23:05:48.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.943+0000 7f2516a00640 1 --2- 172.21.15.12:0/1622021157 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f24f40a2800 0x7f24f40a4cc0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.943+0000 7f2516a00640 1 --2- 172.21.15.12:0/1622021157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2510105490 0x7f25101a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.943+0000 7f2516a00640 1 --2- 172.21.15.12:0/1622021157 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2510104ad0 0x7f251019a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.943+0000 7f2516a00640 1 --2- 172.21.15.12:0/1622021157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25101038e0 0x7f2510199b50 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:48.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.943+0000 7f2516a00640 1 -- 172.21.15.12:0/1622021157 >> 172.21.15.12:0/1622021157 conn(0x7f25100fee90 msgr2=0x7f2510107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:48.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.943+0000 7f2516a00640 1 -- 172.21.15.12:0/1622021157 shutdown_connections 2024-03-23T23:05:48.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:48.943+0000 7f2516a00640 1 -- 172.21.15.12:0/1622021157 wait complete. 2024-03-23T23:05:48.962 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:05:48.962 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-23T23:05:49.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.189+0000 7fd749a00640 1 -- 172.21.15.12:0/1070555071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7441038e0 msgr2=0x7fd744103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.189+0000 7fd749a00640 1 --2- 172.21.15.12:0/1070555071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7441038e0 0x7fd744103d00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd72c009900 tx=0x7fd72c02bc50 comp rx=0 tx=0).stop 2024-03-23T23:05:49.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.189+0000 7fd749a00640 1 -- 172.21.15.12:0/1070555071 shutdown_connections 2024-03-23T23:05:49.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.189+0000 7fd749a00640 1 --2- 172.21.15.12:0/1070555071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd744105490 0x7fd74410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.189+0000 7fd749a00640 1 --2- 172.21.15.12:0/1070555071 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd744104ad0 0x7fd744104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.189+0000 7fd749a00640 1 --2- 172.21.15.12:0/1070555071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7441038e0 0x7fd744103d00 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.189+0000 7fd749a00640 1 -- 172.21.15.12:0/1070555071 >> 172.21.15.12:0/1070555071 conn(0x7fd7440fee50 msgr2=0x7fd744101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:49.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.189+0000 7fd749a00640 1 -- 172.21.15.12:0/1070555071 shutdown_connections 2024-03-23T23:05:49.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.190+0000 7fd749a00640 1 -- 172.21.15.12:0/1070555071 wait complete. 2024-03-23T23:05:49.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.191+0000 7fd749a00640 1 Processor -- start 2024-03-23T23:05:49.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.191+0000 7fd749a00640 1 -- start start 2024-03-23T23:05:49.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.191+0000 7fd749a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7441038e0 0x7fd744199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.191+0000 7fd749a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd744104ad0 0x7fd74419a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.191+0000 7fd749a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd744105490 0x7fd7441a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.191+0000 7fd749a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74410f5b0 con 0x7fd744105490 2024-03-23T23:05:49.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.191+0000 7fd749a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74410f430 con 0x7fd744104ad0 2024-03-23T23:05:49.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.191+0000 7fd749a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74410f730 con 0x7fd7441038e0 2024-03-23T23:05:49.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.191+0000 7fd742a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7441038e0 0x7fd744199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.191+0000 7fd742000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd744104ad0 0x7fd74419a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.191+0000 7fd742a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7441038e0 0x7fd744199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45728/0 (socket says 172.21.15.12:45728) 2024-03-23T23:05:49.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.191+0000 7fd742a00640 1 -- 172.21.15.12:0/3424970896 learned_addr learned my addr 172.21.15.12:0/3424970896 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:49.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.191+0000 7fd743400640 1 --2- 172.21.15.12:0/3424970896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd744105490 0x7fd7441a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.192+0000 7fd742a00640 1 -- 172.21.15.12:0/3424970896 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd744104ad0 msgr2=0x7fd74419a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.192+0000 7fd742a00640 1 --2- 172.21.15.12:0/3424970896 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd744104ad0 0x7fd74419a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.192+0000 7fd742a00640 1 -- 172.21.15.12:0/3424970896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd744105490 msgr2=0x7fd7441a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.192+0000 7fd742a00640 1 --2- 172.21.15.12:0/3424970896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd744105490 0x7fd7441a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.192+0000 7fd742a00640 1 -- 172.21.15.12:0/3424970896 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7441a1810 con 0x7fd7441038e0 2024-03-23T23:05:49.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.192+0000 7fd742000640 1 --2- 172.21.15.12:0/3424970896 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd744104ad0 0x7fd74419a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:49.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.192+0000 7fd743400640 1 --2- 172.21.15.12:0/3424970896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd744105490 0x7fd7441a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:49.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.192+0000 7fd742a00640 1 --2- 172.21.15.12:0/3424970896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7441038e0 0x7fd744199bf0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd72c005ab0 tx=0x7fd72c02c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:49.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.192+0000 7fd73b400640 1 -- 172.21.15.12:0/3424970896 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72c041070 con 0x7fd7441038e0 2024-03-23T23:05:49.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.193+0000 7fd73b400640 1 -- 172.21.15.12:0/3424970896 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd72c004450 con 0x7fd7441038e0 2024-03-23T23:05:49.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.193+0000 7fd749a00640 1 -- 172.21.15.12:0/3424970896 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7441a1aa0 con 0x7fd7441038e0 2024-03-23T23:05:49.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.193+0000 7fd73b400640 1 -- 172.21.15.12:0/3424970896 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72c034440 con 0x7fd7441038e0 2024-03-23T23:05:49.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.193+0000 7fd749a00640 1 -- 172.21.15.12:0/3424970896 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7441a2040 con 0x7fd7441038e0 2024-03-23T23:05:49.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.196+0000 7fd74c83a740 1 Processor -- start 2024-03-23T23:05:49.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.196+0000 7fd73b400640 1 -- 172.21.15.12:0/3424970896 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd72c044070 con 0x7fd7441038e0 2024-03-23T23:05:49.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.196+0000 7fd74c83a740 1 -- start start 2024-03-23T23:05:49.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.196+0000 7fd74c83a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc95810 0x55c80cc95c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.196+0000 7fd74c83a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c80cc969e0 0x55c80cc96e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.196+0000 7fd74c83a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c80cc973a0 0x55c80cc9dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.196+0000 7fd74c83a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c80cbfa2c0 con 0x55c80cc969e0 2024-03-23T23:05:49.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.196+0000 7fd74c83a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c80cba2c90 con 0x55c80cc95810 2024-03-23T23:05:49.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.196+0000 7fd74c83a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c80cbf6360 con 0x55c80cc973a0 2024-03-23T23:05:49.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.197+0000 7fd743400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc95810 0x55c80cc95c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.197+0000 7fd743400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc95810 0x55c80cc95c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50092/0 (socket says 172.21.15.12:50092) 2024-03-23T23:05:49.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.197+0000 7fd743400640 1 -- 172.21.15.12:0/3293732635 learned_addr learned my addr 172.21.15.12:0/3293732635 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:49.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.197+0000 7fd742a00640 1 --2- 172.21.15.12:0/3293732635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c80cc969e0 0x55c80cc96e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.197+0000 7fd742000640 1 --2- 172.21.15.12:0/3293732635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c80cc973a0 0x55c80cc9dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.197+0000 7fd743400640 1 -- 172.21.15.12:0/3293732635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c80cc973a0 msgr2=0x55c80cc9dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:49.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.197+0000 7fd743400640 1 --2- 172.21.15.12:0/3293732635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c80cc973a0 0x55c80cc9dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.197+0000 7fd743400640 1 -- 172.21.15.12:0/3293732635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c80cc969e0 msgr2=0x55c80cc96e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:49.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.197+0000 7fd743400640 1 --2- 172.21.15.12:0/3293732635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c80cc969e0 0x55c80cc96e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.197+0000 7fd743400640 1 -- 172.21.15.12:0/3293732635 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c80cbc7080 con 0x55c80cc95810 2024-03-23T23:05:49.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.197+0000 7fd742000640 1 --2- 172.21.15.12:0/3293732635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c80cc973a0 0x55c80cc9dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:05:49.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.197+0000 7fd743400640 1 --2- 172.21.15.12:0/3293732635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc95810 0x55c80cc95c30 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd73c011440 tx=0x7fd73c011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=921e9ad46615e05a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:49.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.198+0000 7fd733e00640 1 -- 172.21.15.12:0/3293732635 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73c004020 con 0x55c80cc95810 2024-03-23T23:05:49.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.198+0000 7fd73b400640 1 --2- 172.21.15.12:0/3424970896 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd7280a2880 0x7fd7280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.198+0000 7fd733e00640 1 -- 172.21.15.12:0/3293732635 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd73c0041c0 con 0x55c80cc95810 2024-03-23T23:05:49.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.198+0000 7fd733e00640 1 -- 172.21.15.12:0/3293732635 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73c00f940 con 0x55c80cc95810 2024-03-23T23:05:49.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.198+0000 7fd73b400640 1 -- 172.21.15.12:0/3424970896 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd72c0f0590 con 0x7fd7441038e0 2024-03-23T23:05:49.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.198+0000 7fd742a00640 1 --2- 172.21.15.12:0/3424970896 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd7280a2880 0x7fd7280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.198+0000 7fd74c83a740 1 -- 172.21.15.12:0/3293732635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc95810 msgr2=0x55c80cc95c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.198+0000 7fd74c83a740 1 --2- 172.21.15.12:0/3293732635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc95810 0x55c80cc95c30 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd73c011440 tx=0x7fd73c011900 comp rx=0 tx=0).stop 2024-03-23T23:05:49.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.198+0000 7fd74c83a740 1 -- 172.21.15.12:0/3293732635 shutdown_connections 2024-03-23T23:05:49.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.198+0000 7fd74c83a740 1 --2- 172.21.15.12:0/3293732635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c80cc973a0 0x55c80cc9dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.198+0000 7fd74c83a740 1 --2- 172.21.15.12:0/3293732635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c80cc969e0 0x55c80cc96e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.198+0000 7fd74c83a740 1 --2- 172.21.15.12:0/3293732635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc95810 0x55c80cc95c30 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.198+0000 7fd74c83a740 1 -- 172.21.15.12:0/3293732635 >> 172.21.15.12:0/3293732635 conn(0x55c80cc90c00 msgr2=0x55c80cc91000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:49.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.199+0000 7fd74c83a740 1 -- 172.21.15.12:0/3293732635 shutdown_connections 2024-03-23T23:05:49.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.199+0000 7fd74c83a740 1 -- 172.21.15.12:0/3293732635 wait complete. 2024-03-23T23:05:49.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.199+0000 7fd742a00640 1 --2- 172.21.15.12:0/3424970896 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd7280a2880 0x7fd7280a4d40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd72c0b9960 tx=0x7fd72c003f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:49.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.199+0000 7fd74c83a740 1 -- start start 2024-03-23T23:05:49.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.199+0000 7fd74c83a740 1 Processor -- start 2024-03-23T23:05:49.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.200+0000 7fd74c83a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c80cc95810 0x55c80cc9fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.200+0000 7fd74c83a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc969e0 0x55c80cca0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.200+0000 7fd74c83a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c80cc973a0 0x55c80cca7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.200+0000 7fd74c83a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c80cc9e3b0 con 0x55c80cc973a0 2024-03-23T23:05:49.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.200+0000 7fd74c83a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c80cbb60d0 con 0x55c80cc969e0 2024-03-23T23:05:49.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.200+0000 7fd74c83a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c80cc9e530 con 0x55c80cc95810 2024-03-23T23:05:49.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.200+0000 7fd74c83a740 10 client.0 _ll_register_callbacks cb 0x55c80cbbae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:49.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.200+0000 7fd742000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc969e0 0x55c80cca0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.200+0000 7fd742000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc969e0 0x55c80cca0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50106/0 (socket says 172.21.15.12:50106) 2024-03-23T23:05:49.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.200+0000 7fd742000640 1 -- 172.21.15.12:0/1201282904 learned_addr learned my addr 172.21.15.12:0/1201282904 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:49.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.200+0000 7fd743400640 1 --2- 172.21.15.12:0/1201282904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c80cc95810 0x55c80cc9fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.200+0000 7fd743400640 1 --2- 172.21.15.12:0/1201282904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c80cc973a0 0x55c80cca7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.201+0000 7fd742000640 1 -- 172.21.15.12:0/1201282904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c80cc95810 msgr2=0x55c80cc9fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:49.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.201+0000 7fd742000640 1 --2- 172.21.15.12:0/1201282904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c80cc95810 0x55c80cc9fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.201+0000 7fd742000640 1 -- 172.21.15.12:0/1201282904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c80cc973a0 msgr2=0x55c80cca7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:49.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.201+0000 7fd742000640 1 --2- 172.21.15.12:0/1201282904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c80cc973a0 0x55c80cca7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.201+0000 7fd742000640 1 -- 172.21.15.12:0/1201282904 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c80c90e3f0 con 0x55c80cc969e0 2024-03-23T23:05:49.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.201+0000 7fd743400640 1 --2- 172.21.15.12:0/1201282904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c80cc95810 0x55c80cc9fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:05:49.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.201+0000 7fd742000640 1 --2- 172.21.15.12:0/1201282904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc969e0 0x55c80cca0000 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd734012a70 tx=0x7fd734012f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e2c88fac2f8cca1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:49.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.201+0000 7fd730c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:05:49.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.201+0000 7fd730c00640 1 -- 172.21.15.12:0/1201282904 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73400ee60 con 0x55c80cc969e0 2024-03-23T23:05:49.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.201+0000 7fd730c00640 1 -- 172.21.15.12:0/1201282904 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd734004d20 con 0x55c80cc969e0 2024-03-23T23:05:49.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.201+0000 7fd74c83a740 1 -- 172.21.15.12:0/1201282904 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd73c0041c0 con 0x55c80cc969e0 2024-03-23T23:05:49.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.201+0000 7fd730c00640 1 -- 172.21.15.12:0/1201282904 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd734003970 con 0x55c80cc969e0 2024-03-23T23:05:49.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.202+0000 7fd74c83a740 10 client.26336 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:49.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.202+0000 7fd74c83a740 1 -- 172.21.15.12:0/1201282904 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c80c9d87d0 con 0x55c80cc969e0 2024-03-23T23:05:49.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.202+0000 7fd730c00640 1 -- 172.21.15.12:0/1201282904 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd73c0041c0 con 0x55c80cc969e0 2024-03-23T23:05:49.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.202+0000 7fd730c00640 1 -- 172.21.15.12:0/1201282904 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fd734019070 con 0x55c80cc969e0 2024-03-23T23:05:49.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.202+0000 7fd74c83a740 10 client.26336 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:49.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.202+0000 7fd74c83a740 10 client.26336 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:05:49.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.202+0000 7fd74c83a740 20 client.26336 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:49.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.202+0000 7fd74c83a740 1 --2- 172.21.15.12:0/1201282904 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55c80ccab550 0x55c80ccad970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.202+0000 7fd74c83a740 4 client.26336 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:05:49.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.202+0000 7fd74c83a740 1 -- 172.21.15.12:0/1201282904 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c80cc07800 con 0x55c80ccab550 2024-03-23T23:05:49.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.202+0000 7fd743400640 1 --2- 172.21.15.12:0/1201282904 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55c80ccab550 0x55c80ccad970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.203+0000 7fd743400640 1 --2- 172.21.15.12:0/1201282904 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55c80ccab550 0x55c80ccad970 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=acb024c80dc1c56 server_cookie=a8b0c59d383b309b in_seq=0 out_seq=0 2024-03-23T23:05:49.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.203+0000 7fd730c00640 10 client.26336 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:05:49.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.204+0000 7fd730c00640 1 -- 172.21.15.12:0/1201282904 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c80cc07800 con 0x55c80ccab550 2024-03-23T23:05:49.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.204+0000 7fd730c00640 10 client.26336 handle_command_reply: tid=0 2024-03-23T23:05:49.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.204+0000 7fd74c83a740 1 client.26336 shutdown 2024-03-23T23:05:49.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.204+0000 7fd74c83a740 1 -- 172.21.15.12:0/1201282904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc969e0 msgr2=0x55c80cca0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.204+0000 7fd74c83a740 1 --2- 172.21.15.12:0/1201282904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc969e0 0x55c80cca0000 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd734012a70 tx=0x7fd734012f30 comp rx=0 tx=0).stop 2024-03-23T23:05:49.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.204+0000 7fd74c83a740 1 -- 172.21.15.12:0/1201282904 shutdown_connections 2024-03-23T23:05:49.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.204+0000 7fd74c83a740 1 --2- 172.21.15.12:0/1201282904 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55c80ccab550 0x55c80ccad970 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.204+0000 7fd74c83a740 1 --2- 172.21.15.12:0/1201282904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c80cc973a0 0x55c80cca7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.204+0000 7fd74c83a740 1 --2- 172.21.15.12:0/1201282904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc969e0 0x55c80cca0000 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.204+0000 7fd730c00640 0 client.26336 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:05:49.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.204+0000 7fd74c83a740 1 --2- 172.21.15.12:0/1201282904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c80cc95810 0x55c80cc9fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.204+0000 7fd74c83a740 1 -- 172.21.15.12:0/1201282904 >> 172.21.15.12:0/1201282904 conn(0x55c80cc90c00 msgr2=0x55c80cc8feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:49.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.204+0000 7fd74c83a740 1 -- 172.21.15.12:0/1201282904 shutdown_connections 2024-03-23T23:05:49.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.204+0000 7fd74c83a740 1 -- 172.21.15.12:0/1201282904 wait complete. 2024-03-23T23:05:49.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.204+0000 7fd74c83a740 20 client.26336 trim_cache size 0 max 16384 2024-03-23T23:05:49.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.218+0000 7fd74c83a740 1 Processor -- start 2024-03-23T23:05:49.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.218+0000 7fd74c83a740 1 -- start start 2024-03-23T23:05:49.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.218+0000 7fd74c83a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc95810 0x55c80cc94570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.218+0000 7fd74c83a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c80cc969e0 0x55c80cca7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.218+0000 7fd74c83a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c80cc973a0 0x55c80cc99a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.218+0000 7fd74c83a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7441ad120 con 0x55c80cc973a0 2024-03-23T23:05:49.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.218+0000 7fd74c83a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7441aac20 con 0x55c80cc95810 2024-03-23T23:05:49.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.218+0000 7fd74c83a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7441a8cc0 con 0x55c80cc969e0 2024-03-23T23:05:49.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.219+0000 7fd742000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c80cc969e0 0x55c80cca7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.219+0000 7fd742000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c80cc969e0 0x55c80cca7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45744/0 (socket says 172.21.15.12:45744) 2024-03-23T23:05:49.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.219+0000 7fd742000640 1 -- 172.21.15.12:0/982450583 learned_addr learned my addr 172.21.15.12:0/982450583 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:49.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.219+0000 7fd743400640 1 --2- 172.21.15.12:0/982450583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc95810 0x55c80cc94570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.219+0000 7fd743400640 1 --2- 172.21.15.12:0/982450583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c80cc973a0 0x55c80cc99a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.219+0000 7fd742000640 1 -- 172.21.15.12:0/982450583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc95810 msgr2=0x55c80cc94570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:49.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.219+0000 7fd742000640 1 --2- 172.21.15.12:0/982450583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc95810 0x55c80cc94570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.219+0000 7fd742000640 1 -- 172.21.15.12:0/982450583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c80cc973a0 msgr2=0x55c80cc99a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:49.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.219+0000 7fd742000640 1 --2- 172.21.15.12:0/982450583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c80cc973a0 0x55c80cc99a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.219+0000 7fd742000640 1 -- 172.21.15.12:0/982450583 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c80cc95410 con 0x55c80cc969e0 2024-03-23T23:05:49.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.219+0000 7fd743400640 1 --2- 172.21.15.12:0/982450583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c80cc973a0 0x55c80cc99a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:05:49.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.219+0000 7fd742000640 1 --2- 172.21.15.12:0/982450583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c80cc969e0 0x55c80cca7000 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd734006010 tx=0x7fd73400e8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2f6af2bb6f58f9b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:49.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.220+0000 7fd733400640 1 -- 172.21.15.12:0/982450583 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd734003dd0 con 0x55c80cc969e0 2024-03-23T23:05:49.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.220+0000 7fd733400640 1 -- 172.21.15.12:0/982450583 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd734020030 con 0x55c80cc969e0 2024-03-23T23:05:49.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.220+0000 7fd733400640 1 -- 172.21.15.12:0/982450583 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7340057b0 con 0x55c80cc969e0 2024-03-23T23:05:49.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.220+0000 7fd74c83a740 1 -- 172.21.15.12:0/982450583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c80cc969e0 msgr2=0x55c80cca7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.220+0000 7fd74c83a740 1 --2- 172.21.15.12:0/982450583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c80cc969e0 0x55c80cca7000 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd734006010 tx=0x7fd73400e8c0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.220+0000 7fd74c83a740 1 -- 172.21.15.12:0/982450583 shutdown_connections 2024-03-23T23:05:49.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.220+0000 7fd74c83a740 1 --2- 172.21.15.12:0/982450583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c80cc973a0 0x55c80cc99a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.220+0000 7fd74c83a740 1 --2- 172.21.15.12:0/982450583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c80cc969e0 0x55c80cca7000 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.220+0000 7fd74c83a740 1 --2- 172.21.15.12:0/982450583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc95810 0x55c80cc94570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.220+0000 7fd74c83a740 1 -- 172.21.15.12:0/982450583 >> 172.21.15.12:0/982450583 conn(0x55c80cc90c00 msgr2=0x55c80cc93e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:49.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.220+0000 7fd74c83a740 1 -- 172.21.15.12:0/982450583 shutdown_connections 2024-03-23T23:05:49.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.220+0000 7fd74c83a740 1 -- 172.21.15.12:0/982450583 wait complete. 2024-03-23T23:05:49.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.221+0000 7fd74c83a740 1 -- start start 2024-03-23T23:05:49.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.221+0000 7fd74c83a740 1 Processor -- start 2024-03-23T23:05:49.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.221+0000 7fd74c83a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc95810 0x55c80cca2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.221+0000 7fd74c83a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c80cc969e0 0x55c80cca3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.221+0000 7fd74c83a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c80cc973a0 0x55c80cca6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.221+0000 7fd74c83a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7441a92c0 con 0x55c80cc969e0 2024-03-23T23:05:49.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.221+0000 7fd74c83a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7441aa5c0 con 0x55c80cc95810 2024-03-23T23:05:49.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.221+0000 7fd74c83a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7441a9900 con 0x55c80cc973a0 2024-03-23T23:05:49.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.222+0000 7fd74c83a740 10 client.0 _ll_register_callbacks cb 0x7fd734009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:49.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.222+0000 7fd743400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc95810 0x55c80cca2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.222+0000 7fd742000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c80cc969e0 0x55c80cca3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.222+0000 7fd743400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc95810 0x55c80cca2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50120/0 (socket says 172.21.15.12:50120) 2024-03-23T23:05:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.222+0000 7fd743400640 1 -- 172.21.15.12:0/3334629604 learned_addr learned my addr 172.21.15.12:0/3334629604 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.222+0000 7fd743400640 1 --2- 172.21.15.12:0/3334629604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c80cc973a0 0x55c80cca6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.222+0000 7fd743400640 1 -- 172.21.15.12:0/3334629604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c80cc973a0 msgr2=0x55c80cca6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.222+0000 7fd743400640 1 --2- 172.21.15.12:0/3334629604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c80cc973a0 0x55c80cca6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.222+0000 7fd743400640 1 -- 172.21.15.12:0/3334629604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c80cc969e0 msgr2=0x55c80cca3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.222+0000 7fd743400640 1 --2- 172.21.15.12:0/3334629604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c80cc969e0 0x55c80cca3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.222+0000 7fd743400640 1 -- 172.21.15.12:0/3334629604 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c80cca6a80 con 0x55c80cc95810 2024-03-23T23:05:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.222+0000 7fd743400640 1 --2- 172.21.15.12:0/3334629604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc95810 0x55c80cca2f30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd73c009e20 tx=0x7fd73c011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aba5a9ef63ddb9e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.222+0000 7fd730c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:05:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.223+0000 7fd730c00640 1 -- 172.21.15.12:0/3334629604 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73c01d040 con 0x55c80cc95810 2024-03-23T23:05:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.223+0000 7fd730c00640 1 -- 172.21.15.12:0/3334629604 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd73c016040 con 0x55c80cc95810 2024-03-23T23:05:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.223+0000 7fd74c83a740 1 -- 172.21.15.12:0/3334629604 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd734020030 con 0x55c80cc95810 2024-03-23T23:05:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.223+0000 7fd730c00640 1 -- 172.21.15.12:0/3334629604 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd73c02b180 con 0x55c80cc95810 2024-03-23T23:05:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.223+0000 7fd74c83a740 10 client.26342 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.223+0000 7fd74c83a740 1 -- 172.21.15.12:0/3334629604 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c80cd07620 con 0x55c80cc95810 2024-03-23T23:05:49.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.223+0000 7fd730c00640 1 -- 172.21.15.12:0/3334629604 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd734020030 con 0x55c80cc95810 2024-03-23T23:05:49.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.223+0000 7fd730c00640 1 -- 172.21.15.12:0/3334629604 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fd73c023040 con 0x55c80cc95810 2024-03-23T23:05:49.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.223+0000 7fd74c83a740 10 client.26342 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:49.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.223+0000 7fd74c83a740 10 client.26342 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:05:49.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.223+0000 7fd74c83a740 20 client.26342 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:49.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.223+0000 7fd74c83a740 1 --2- 172.21.15.12:0/3334629604 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55c80cd07d30 0x55c80cd0a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.223+0000 7fd74c83a740 4 client.26342 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:05:49.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.223+0000 7fd74c83a740 1 -- 172.21.15.12:0/3334629604 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c80cc01b50 con 0x55c80cd07d30 2024-03-23T23:05:49.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.224+0000 7fd742000640 1 --2- 172.21.15.12:0/3334629604 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55c80cd07d30 0x55c80cd0a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.224+0000 7fd742000640 1 --2- 172.21.15.12:0/3334629604 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55c80cd07d30 0x55c80cd0a150 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f8f6c719bf3bad07 server_cookie=faf68ac4b44abf82 in_seq=0 out_seq=0 2024-03-23T23:05:49.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.224+0000 7fd730c00640 10 client.26342 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:05:49.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.224+0000 7fd730c00640 1 -- 172.21.15.12:0/3334629604 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c80cc01b50 con 0x55c80cd07d30 2024-03-23T23:05:49.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.224+0000 7fd730c00640 10 client.26342 handle_command_reply: tid=0 2024-03-23T23:05:49.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.224+0000 7fd74c83a740 1 client.26342 shutdown 2024-03-23T23:05:49.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.224+0000 7fd74c83a740 1 -- 172.21.15.12:0/3334629604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc95810 msgr2=0x55c80cca2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.224+0000 7fd74c83a740 1 --2- 172.21.15.12:0/3334629604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc95810 0x55c80cca2f30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd73c009e20 tx=0x7fd73c011900 comp rx=0 tx=0).stop 2024-03-23T23:05:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.225+0000 7fd74c83a740 1 -- 172.21.15.12:0/3334629604 shutdown_connections 2024-03-23T23:05:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.225+0000 7fd74c83a740 1 --2- 172.21.15.12:0/3334629604 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55c80cd07d30 0x55c80cd0a150 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.225+0000 7fd74c83a740 1 --2- 172.21.15.12:0/3334629604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c80cc973a0 0x55c80cca6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.225+0000 7fd74c83a740 1 --2- 172.21.15.12:0/3334629604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c80cc969e0 0x55c80cca3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.225+0000 7fd730c00640 0 client.26342 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:05:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.225+0000 7fd74c83a740 1 --2- 172.21.15.12:0/3334629604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c80cc95810 0x55c80cca2f30 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.225+0000 7fd74c83a740 1 -- 172.21.15.12:0/3334629604 >> 172.21.15.12:0/3334629604 conn(0x55c80cc90c00 msgr2=0x55c80cc946f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.225+0000 7fd74c83a740 1 -- 172.21.15.12:0/3334629604 shutdown_connections 2024-03-23T23:05:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.225+0000 7fd74c83a740 1 -- 172.21.15.12:0/3334629604 wait complete. 2024-03-23T23:05:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.225+0000 7fd74c83a740 20 client.26342 trim_cache size 0 max 16384 2024-03-23T23:05:49.231 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:05:49.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.225+0000 7fd749a00640 1 -- 172.21.15.12:0/3424970896 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd7280a2880 msgr2=0x7fd7280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.225+0000 7fd749a00640 1 --2- 172.21.15.12:0/3424970896 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd7280a2880 0x7fd7280a4d40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd72c0b9960 tx=0x7fd72c003f90 comp rx=0 tx=0).stop 2024-03-23T23:05:49.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.225+0000 7fd749a00640 1 -- 172.21.15.12:0/3424970896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7441038e0 msgr2=0x7fd744199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.225+0000 7fd749a00640 1 --2- 172.21.15.12:0/3424970896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7441038e0 0x7fd744199bf0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd72c005ab0 tx=0x7fd72c02c6e0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.226+0000 7fd749a00640 1 -- 172.21.15.12:0/3424970896 shutdown_connections 2024-03-23T23:05:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.226+0000 7fd749a00640 1 --2- 172.21.15.12:0/3424970896 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd7280a2880 0x7fd7280a4d40 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.226+0000 7fd749a00640 1 --2- 172.21.15.12:0/3424970896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd744105490 0x7fd7441a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.226+0000 7fd749a00640 1 --2- 172.21.15.12:0/3424970896 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd744104ad0 0x7fd74419a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.226+0000 7fd749a00640 1 --2- 172.21.15.12:0/3424970896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7441038e0 0x7fd744199bf0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.226+0000 7fd749a00640 1 -- 172.21.15.12:0/3424970896 >> 172.21.15.12:0/3424970896 conn(0x7fd7440fee50 msgr2=0x7fd744107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.226+0000 7fd749a00640 1 -- 172.21.15.12:0/3424970896 shutdown_connections 2024-03-23T23:05:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.226+0000 7fd749a00640 1 -- 172.21.15.12:0/3424970896 wait complete. 2024-03-23T23:05:49.242 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:05:49.242 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-23T23:05:49.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.454+0000 7fd8af200640 1 -- 172.21.15.12:0/2069189595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8b0109430 msgr2=0x7fd8b010b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.454+0000 7fd8af200640 1 --2- 172.21.15.12:0/2069189595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8b0109430 0x7fd8b010b8c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009ca0 tx=0x7fd8a802c190 comp rx=0 tx=0).stop 2024-03-23T23:05:49.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.454+0000 7fd8af200640 1 -- 172.21.15.12:0/2069189595 shutdown_connections 2024-03-23T23:05:49.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.454+0000 7fd8af200640 1 --2- 172.21.15.12:0/2069189595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8b0109430 0x7fd8b010b8c0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.454+0000 7fd8af200640 1 --2- 172.21.15.12:0/2069189595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8b0074080 0x7fd8b00744e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.454+0000 7fd8af200640 1 --2- 172.21.15.12:0/2069189595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8b0075660 0x7fd8b0073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.454+0000 7fd8af200640 1 -- 172.21.15.12:0/2069189595 >> 172.21.15.12:0/2069189595 conn(0x7fd8b00fcb10 msgr2=0x7fd8b00fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:49.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.454+0000 7fd8af200640 1 -- 172.21.15.12:0/2069189595 shutdown_connections 2024-03-23T23:05:49.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.454+0000 7fd8af200640 1 -- 172.21.15.12:0/2069189595 wait complete. 2024-03-23T23:05:49.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.455+0000 7fd8af200640 1 Processor -- start 2024-03-23T23:05:49.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.456+0000 7fd8af200640 1 -- start start 2024-03-23T23:05:49.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.456+0000 7fd8af200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8b0074080 0x7fd8b0199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.456+0000 7fd8af200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8b0075660 0x7fd8b0199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.456+0000 7fd8af200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8b0109430 0x7fd8b01a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.456+0000 7fd8af200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b010f3f0 con 0x7fd8b0075660 2024-03-23T23:05:49.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.456+0000 7fd8af200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b010f270 con 0x7fd8b0109430 2024-03-23T23:05:49.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.456+0000 7fd8af200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b010f570 con 0x7fd8b0074080 2024-03-23T23:05:49.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.456+0000 7fd8ae800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8b0109430 0x7fd8b01a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.456+0000 7fd8ad400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8b0075660 0x7fd8b0199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.456+0000 7fd8ae800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8b0109430 0x7fd8b01a0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50142/0 (socket says 172.21.15.12:50142) 2024-03-23T23:05:49.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.456+0000 7fd8ae800640 1 -- 172.21.15.12:0/217700460 learned_addr learned my addr 172.21.15.12:0/217700460 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:49.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.457+0000 7fd8ae800640 1 -- 172.21.15.12:0/217700460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8b0074080 msgr2=0x7fd8b0199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.457+0000 7fd8ae800640 1 --2- 172.21.15.12:0/217700460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8b0074080 0x7fd8b0199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.457+0000 7fd8ae800640 1 -- 172.21.15.12:0/217700460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8b0075660 msgr2=0x7fd8b0199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.457+0000 7fd8ae800640 1 --2- 172.21.15.12:0/217700460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8b0075660 0x7fd8b0199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.457+0000 7fd8ae800640 1 -- 172.21.15.12:0/217700460 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b01a1620 con 0x7fd8b0109430 2024-03-23T23:05:49.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.457+0000 7fd8ae800640 1 --2- 172.21.15.12:0/217700460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8b0109430 0x7fd8b01a0fb0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd8a802c6d0 tx=0x7fd8a8003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:49.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.457+0000 7fd8a6a00640 1 -- 172.21.15.12:0/217700460 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8039040 con 0x7fd8b0109430 2024-03-23T23:05:49.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.457+0000 7fd8a6a00640 1 -- 172.21.15.12:0/217700460 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8a8044070 con 0x7fd8b0109430 2024-03-23T23:05:49.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.457+0000 7fd8af200640 1 -- 172.21.15.12:0/217700460 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b01a18b0 con 0x7fd8b0109430 2024-03-23T23:05:49.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.458+0000 7fd8a6a00640 1 -- 172.21.15.12:0/217700460 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8004330 con 0x7fd8b0109430 2024-03-23T23:05:49.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.458+0000 7fd8af200640 1 -- 172.21.15.12:0/217700460 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b01a1df0 con 0x7fd8b0109430 2024-03-23T23:05:49.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.461+0000 7fd8a6a00640 1 -- 172.21.15.12:0/217700460 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd8a80044d0 con 0x7fd8b0109430 2024-03-23T23:05:49.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.461+0000 7fd8b7838740 1 Processor -- start 2024-03-23T23:05:49.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.461+0000 7fd8b7838740 1 -- start start 2024-03-23T23:05:49.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.461+0000 7fd8b7838740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558afe3d59f0 0x558afe3d5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.462+0000 7fd8b7838740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558afe3d6bc0 0x558afe3d7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.462+0000 7fd8b7838740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d7580 0x558afe3dde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.462+0000 7fd8b7838740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x558afe33a2c0 con 0x558afe3d59f0 2024-03-23T23:05:49.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.462+0000 7fd8b7838740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x558afe2e2c90 con 0x558afe3d7580 2024-03-23T23:05:49.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.462+0000 7fd8b7838740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x558afe336360 con 0x558afe3d6bc0 2024-03-23T23:05:49.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.462+0000 7fd8ade00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558afe3d6bc0 0x558afe3d7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.462+0000 7fd8ade00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558afe3d6bc0 0x558afe3d7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45770/0 (socket says 172.21.15.12:45770) 2024-03-23T23:05:49.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.462+0000 7fd8ade00640 1 -- 172.21.15.12:0/2900468395 learned_addr learned my addr 172.21.15.12:0/2900468395 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:49.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.462+0000 7fd8ad400640 1 --2- 172.21.15.12:0/2900468395 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558afe3d59f0 0x558afe3d5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.462+0000 7fd8ade00640 1 -- 172.21.15.12:0/2900468395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d7580 msgr2=0x558afe3dde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:49.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.462+0000 7fd8ad400640 1 --2- 172.21.15.12:0/2900468395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d7580 0x558afe3dde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.462+0000 7fd8ade00640 1 --2- 172.21.15.12:0/2900468395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d7580 0x558afe3dde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.462+0000 7fd8ade00640 1 -- 172.21.15.12:0/2900468395 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558afe3d59f0 msgr2=0x558afe3d5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:49.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.462+0000 7fd8ade00640 1 --2- 172.21.15.12:0/2900468395 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558afe3d59f0 0x558afe3d5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.462+0000 7fd8ade00640 1 -- 172.21.15.12:0/2900468395 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558afe307080 con 0x558afe3d6bc0 2024-03-23T23:05:49.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.462+0000 7fd8ad400640 1 --2- 172.21.15.12:0/2900468395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d7580 0x558afe3dde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:05:49.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.463+0000 7fd8ade00640 1 --2- 172.21.15.12:0/2900468395 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558afe3d6bc0 0x558afe3d7040 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000bb10 tx=0x7fd8a000d2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f83da4d3cd2df20d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:49.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.463+0000 7fd89b000640 1 -- 172.21.15.12:0/2900468395 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0016070 con 0x558afe3d6bc0 2024-03-23T23:05:49.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.463+0000 7fd8a6a00640 1 --2- 172.21.15.12:0/217700460 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd8940a2880 0x7fd8940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.463+0000 7fd89b000640 1 -- 172.21.15.12:0/2900468395 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8a0005500 con 0x558afe3d6bc0 2024-03-23T23:05:49.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.463+0000 7fd89b000640 1 -- 172.21.15.12:0/2900468395 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0010460 con 0x558afe3d6bc0 2024-03-23T23:05:49.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.463+0000 7fd8a6a00640 1 -- 172.21.15.12:0/217700460 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd8a80f1070 con 0x7fd8b0109430 2024-03-23T23:05:49.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.463+0000 7fd8ad400640 1 --2- 172.21.15.12:0/217700460 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd8940a2880 0x7fd8940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.463+0000 7fd8b7838740 1 -- 172.21.15.12:0/2900468395 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558afe3d6bc0 msgr2=0x558afe3d7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.463+0000 7fd8b7838740 1 --2- 172.21.15.12:0/2900468395 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558afe3d6bc0 0x558afe3d7040 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000bb10 tx=0x7fd8a000d2e0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.464+0000 7fd8b7838740 1 -- 172.21.15.12:0/2900468395 shutdown_connections 2024-03-23T23:05:49.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.464+0000 7fd8b7838740 1 --2- 172.21.15.12:0/2900468395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d7580 0x558afe3dde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.464+0000 7fd8b7838740 1 --2- 172.21.15.12:0/2900468395 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558afe3d6bc0 0x558afe3d7040 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.464+0000 7fd8b7838740 1 --2- 172.21.15.12:0/2900468395 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558afe3d59f0 0x558afe3d5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.464+0000 7fd8b7838740 1 -- 172.21.15.12:0/2900468395 >> 172.21.15.12:0/2900468395 conn(0x558afe3d0de0 msgr2=0x558afe3d11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:49.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.464+0000 7fd8b7838740 1 -- 172.21.15.12:0/2900468395 shutdown_connections 2024-03-23T23:05:49.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.464+0000 7fd8b7838740 1 -- 172.21.15.12:0/2900468395 wait complete. 2024-03-23T23:05:49.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.464+0000 7fd8ad400640 1 --2- 172.21.15.12:0/217700460 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd8940a2880 0x7fd8940a4d40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x558afe3d6ac0 tx=0x7fd89c009770 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:49.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.464+0000 7fd8b7838740 1 -- start start 2024-03-23T23:05:49.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.465+0000 7fd8b7838740 1 Processor -- start 2024-03-23T23:05:49.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.465+0000 7fd8b7838740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558afe3d59f0 0x558afe3dfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.465+0000 7fd8b7838740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d6bc0 0x558afe3e01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.465+0000 7fd8b7838740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558afe3d7580 0x558afe3e7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.465+0000 7fd8b7838740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x558afe3de590 con 0x558afe3d59f0 2024-03-23T23:05:49.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.465+0000 7fd8b7838740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x558afe2f60d0 con 0x558afe3d6bc0 2024-03-23T23:05:49.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.465+0000 7fd8b7838740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x558afe3de710 con 0x558afe3d7580 2024-03-23T23:05:49.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.465+0000 7fd8b7838740 10 client.0 _ll_register_callbacks cb 0x558afe2fae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:49.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.466+0000 7fd8ade00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558afe3d59f0 0x558afe3dfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.466+0000 7fd8ad400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d6bc0 0x558afe3e01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.466+0000 7fd8ad400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d6bc0 0x558afe3e01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50164/0 (socket says 172.21.15.12:50164) 2024-03-23T23:05:49.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.466+0000 7fd8ade00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558afe3d59f0 0x558afe3dfca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52678/0 (socket says 172.21.15.12:52678) 2024-03-23T23:05:49.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.466+0000 7fd8ad400640 1 -- 172.21.15.12:0/3932576529 learned_addr learned my addr 172.21.15.12:0/3932576529 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:49.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.466+0000 7fd8ade00640 1 -- 172.21.15.12:0/3932576529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558afe3d7580 msgr2=0x558afe3e7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:49.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.466+0000 7fd8ade00640 1 --2- 172.21.15.12:0/3932576529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558afe3d7580 0x558afe3e7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.466+0000 7fd8ade00640 1 -- 172.21.15.12:0/3932576529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d6bc0 msgr2=0x558afe3e01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:49.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.466+0000 7fd8ade00640 1 --2- 172.21.15.12:0/3932576529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d6bc0 0x558afe3e01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.466+0000 7fd8ade00640 1 -- 172.21.15.12:0/3932576529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558afe04e3f0 con 0x558afe3d59f0 2024-03-23T23:05:49.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.466+0000 7fd8ad400640 1 --2- 172.21.15.12:0/3932576529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d6bc0 0x558afe3e01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:49.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.466+0000 7fd8ade00640 1 --2- 172.21.15.12:0/3932576529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558afe3d59f0 0x558afe3dfca0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000bae0 tx=0x7fd8a000b6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=280a51b95acdf6d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:49.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.466+0000 7fd88f400640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:05:49.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.467+0000 7fd88f400640 1 -- 172.21.15.12:0/3932576529 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0016070 con 0x558afe3d59f0 2024-03-23T23:05:49.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.467+0000 7fd88f400640 1 -- 172.21.15.12:0/3932576529 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd8a0005220 con 0x558afe3d59f0 2024-03-23T23:05:49.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.467+0000 7fd8b7838740 1 -- 172.21.15.12:0/3932576529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd8a0005500 con 0x558afe3d59f0 2024-03-23T23:05:49.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.467+0000 7fd88f400640 1 -- 172.21.15.12:0/3932576529 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8a000dae0 con 0x558afe3d59f0 2024-03-23T23:05:49.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.467+0000 7fd8b7838740 10 client.16221 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:49.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.467+0000 7fd8b7838740 1 -- 172.21.15.12:0/3932576529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x558afe1187d0 con 0x558afe3d59f0 2024-03-23T23:05:49.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.467+0000 7fd88f400640 1 -- 172.21.15.12:0/3932576529 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8a0005500 con 0x558afe3d59f0 2024-03-23T23:05:49.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.468+0000 7fd88f400640 1 -- 172.21.15.12:0/3932576529 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fd8a00038f0 con 0x558afe3d59f0 2024-03-23T23:05:49.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.468+0000 7fd8b7838740 10 client.16221 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:49.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.468+0000 7fd8b7838740 10 client.16221 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:05:49.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.468+0000 7fd8b7838740 20 client.16221 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:49.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.468+0000 7fd8b7838740 1 --2- 172.21.15.12:0/3932576529 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x558afe3eb400 0x558afe3ed7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.468+0000 7fd8b7838740 4 client.16221 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:05:49.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.468+0000 7fd8b7838740 1 -- 172.21.15.12:0/3932576529 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558afe347800 con 0x558afe3eb400 2024-03-23T23:05:49.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.468+0000 7fd8ad400640 1 --2- 172.21.15.12:0/3932576529 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x558afe3eb400 0x558afe3ed7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.469+0000 7fd8ad400640 1 --2- 172.21.15.12:0/3932576529 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x558afe3eb400 0x558afe3ed7f0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bece19d01ecbf8ae server_cookie=481ab128005ffe1f in_seq=0 out_seq=0 2024-03-23T23:05:49.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.469+0000 7fd88f400640 10 client.16221 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:05:49.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.470+0000 7fd88f400640 1 -- 172.21.15.12:0/3932576529 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558afe347800 con 0x558afe3eb400 2024-03-23T23:05:49.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.470+0000 7fd88f400640 10 client.16221 handle_command_reply: tid=0 2024-03-23T23:05:49.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.470+0000 7fd8b7838740 1 client.16221 shutdown 2024-03-23T23:05:49.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.470+0000 7fd8b7838740 1 -- 172.21.15.12:0/3932576529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558afe3d59f0 msgr2=0x558afe3dfca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.470+0000 7fd8b7838740 1 --2- 172.21.15.12:0/3932576529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558afe3d59f0 0x558afe3dfca0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000bae0 tx=0x7fd8a000b6e0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.470+0000 7fd8b7838740 1 -- 172.21.15.12:0/3932576529 shutdown_connections 2024-03-23T23:05:49.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.470+0000 7fd8b7838740 1 --2- 172.21.15.12:0/3932576529 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x558afe3eb400 0x558afe3ed7f0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.470+0000 7fd8b7838740 1 --2- 172.21.15.12:0/3932576529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558afe3d7580 0x558afe3e7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.470+0000 7fd8b7838740 1 --2- 172.21.15.12:0/3932576529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d6bc0 0x558afe3e01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.470+0000 7fd8b7838740 1 --2- 172.21.15.12:0/3932576529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558afe3d59f0 0x558afe3dfca0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.470+0000 7fd88f400640 0 client.16221 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:05:49.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.470+0000 7fd8b7838740 1 -- 172.21.15.12:0/3932576529 >> 172.21.15.12:0/3932576529 conn(0x558afe3d0de0 msgr2=0x558afe3d0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:49.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.470+0000 7fd8b7838740 1 -- 172.21.15.12:0/3932576529 shutdown_connections 2024-03-23T23:05:49.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.470+0000 7fd8b7838740 1 -- 172.21.15.12:0/3932576529 wait complete. 2024-03-23T23:05:49.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.471+0000 7fd8b7838740 20 client.16221 trim_cache size 0 max 16384 2024-03-23T23:05:49.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.484+0000 7fd8b7838740 1 Processor -- start 2024-03-23T23:05:49.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.484+0000 7fd8b7838740 1 -- start start 2024-03-23T23:05:49.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.485+0000 7fd8b7838740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d59f0 0x558afe3cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.485+0000 7fd8b7838740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558afe3d6bc0 0x558afe3e71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.485+0000 7fd8b7838740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558afe3d7580 0x558afe3f2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.485+0000 7fd8b7838740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b01acf30 con 0x558afe3d7580 2024-03-23T23:05:49.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.485+0000 7fd8b7838740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b01aaa30 con 0x558afe3d59f0 2024-03-23T23:05:49.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.485+0000 7fd8b7838740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b01a8ad0 con 0x558afe3d6bc0 2024-03-23T23:05:49.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.485+0000 7fd8ad400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558afe3d6bc0 0x558afe3e71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.485+0000 7fd8ade00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d59f0 0x558afe3cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.485+0000 7fd8ae800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558afe3d7580 0x558afe3f2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.485+0000 7fd8ad400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558afe3d6bc0 0x558afe3e71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45792/0 (socket says 172.21.15.12:45792) 2024-03-23T23:05:49.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.485+0000 7fd8ade00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d59f0 0x558afe3cffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50166/0 (socket says 172.21.15.12:50166) 2024-03-23T23:05:49.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.485+0000 7fd8ad400640 1 -- 172.21.15.12:0/2158499116 learned_addr learned my addr 172.21.15.12:0/2158499116 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:49.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.485+0000 7fd8ad400640 1 -- 172.21.15.12:0/2158499116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d59f0 msgr2=0x558afe3cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:49.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.485+0000 7fd8ad400640 1 --2- 172.21.15.12:0/2158499116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d59f0 0x558afe3cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.485+0000 7fd8ad400640 1 -- 172.21.15.12:0/2158499116 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558afe3d7580 msgr2=0x558afe3f2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:49.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.485+0000 7fd8ad400640 1 --2- 172.21.15.12:0/2158499116 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558afe3d7580 0x558afe3f2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.485+0000 7fd8ad400640 1 -- 172.21.15.12:0/2158499116 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558afe3d1830 con 0x558afe3d6bc0 2024-03-23T23:05:49.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.486+0000 7fd8ade00640 1 --2- 172.21.15.12:0/2158499116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d59f0 0x558afe3cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:49.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.486+0000 7fd8ad400640 1 --2- 172.21.15.12:0/2158499116 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558afe3d6bc0 0x558afe3e71b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x558afe3ee010 tx=0x7fd89c018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15c57074539e7729 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:49.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.486+0000 7fd89a600640 1 -- 172.21.15.12:0/2158499116 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c02e0d0 con 0x558afe3d6bc0 2024-03-23T23:05:49.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.486+0000 7fd89a600640 1 -- 172.21.15.12:0/2158499116 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd89c020e50 con 0x558afe3d6bc0 2024-03-23T23:05:49.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.486+0000 7fd89a600640 1 -- 172.21.15.12:0/2158499116 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c01e4c0 con 0x558afe3d6bc0 2024-03-23T23:05:49.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.487+0000 7fd8b7838740 1 -- 172.21.15.12:0/2158499116 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558afe3d6bc0 msgr2=0x558afe3e71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.487+0000 7fd8b7838740 1 --2- 172.21.15.12:0/2158499116 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558afe3d6bc0 0x558afe3e71b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x558afe3ee010 tx=0x7fd89c018000 comp rx=0 tx=0).stop 2024-03-23T23:05:49.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.487+0000 7fd8b7838740 1 -- 172.21.15.12:0/2158499116 shutdown_connections 2024-03-23T23:05:49.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.487+0000 7fd8b7838740 1 --2- 172.21.15.12:0/2158499116 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558afe3d7580 0x558afe3f2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.487+0000 7fd8b7838740 1 --2- 172.21.15.12:0/2158499116 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558afe3d6bc0 0x558afe3e71b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.487+0000 7fd8b7838740 1 --2- 172.21.15.12:0/2158499116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d59f0 0x558afe3cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.487+0000 7fd8b7838740 1 -- 172.21.15.12:0/2158499116 >> 172.21.15.12:0/2158499116 conn(0x558afe3d0de0 msgr2=0x558afe3cf8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:49.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.487+0000 7fd8b7838740 1 -- 172.21.15.12:0/2158499116 shutdown_connections 2024-03-23T23:05:49.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.487+0000 7fd8b7838740 1 -- 172.21.15.12:0/2158499116 wait complete. 2024-03-23T23:05:49.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.487+0000 7fd8b7838740 1 -- start start 2024-03-23T23:05:49.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.487+0000 7fd8b7838740 1 Processor -- start 2024-03-23T23:05:49.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.488+0000 7fd8b7838740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d59f0 0x558afe3e2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.488+0000 7fd8b7838740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558afe3d6bc0 0x558afe3e3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.488+0000 7fd8b7838740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558afe3d7580 0x558afe3e6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.488+0000 7fd8b7838740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b01a90d0 con 0x558afe3d6bc0 2024-03-23T23:05:49.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.488+0000 7fd8b7838740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b01aa3d0 con 0x558afe3d59f0 2024-03-23T23:05:49.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.488+0000 7fd8b7838740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b01a9710 con 0x558afe3d7580 2024-03-23T23:05:49.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.488+0000 7fd8b7838740 10 client.0 _ll_register_callbacks cb 0x7fd8a000b960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:49.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.488+0000 7fd8ade00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d59f0 0x558afe3e2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.488+0000 7fd8ad400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558afe3d6bc0 0x558afe3e3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.488+0000 7fd8ade00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d59f0 0x558afe3e2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50168/0 (socket says 172.21.15.12:50168) 2024-03-23T23:05:49.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.488+0000 7fd8ade00640 1 -- 172.21.15.12:0/1980828003 learned_addr learned my addr 172.21.15.12:0/1980828003 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:49.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.488+0000 7fd8ae800640 1 --2- 172.21.15.12:0/1980828003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558afe3d7580 0x558afe3e6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.488+0000 7fd8ade00640 1 -- 172.21.15.12:0/1980828003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558afe3d7580 msgr2=0x558afe3e6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:49.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.488+0000 7fd8ade00640 1 --2- 172.21.15.12:0/1980828003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558afe3d7580 0x558afe3e6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.488+0000 7fd8ade00640 1 -- 172.21.15.12:0/1980828003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558afe3d6bc0 msgr2=0x558afe3e3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:49.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.488+0000 7fd8ade00640 1 --2- 172.21.15.12:0/1980828003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558afe3d6bc0 0x558afe3e3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.488+0000 7fd8ade00640 1 -- 172.21.15.12:0/1980828003 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558afe3e6730 con 0x558afe3d59f0 2024-03-23T23:05:49.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.488+0000 7fd8ad400640 1 --2- 172.21.15.12:0/1980828003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558afe3d6bc0 0x558afe3e3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:05:49.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.489+0000 7fd8ade00640 1 --2- 172.21.15.12:0/1980828003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d59f0 0x558afe3e2be0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000bb10 tx=0x7fd8a000deb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=753b6eaf4cd38dba server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:49.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.489+0000 7fd88f400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:05:49.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.489+0000 7fd88f400640 1 -- 172.21.15.12:0/1980828003 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0016040 con 0x558afe3d59f0 2024-03-23T23:05:49.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.489+0000 7fd88f400640 1 -- 172.21.15.12:0/1980828003 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd8a00037f0 con 0x558afe3d59f0 2024-03-23T23:05:49.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.489+0000 7fd8b7838740 1 -- 172.21.15.12:0/1980828003 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd89c020e50 con 0x558afe3d59f0 2024-03-23T23:05:49.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.489+0000 7fd88f400640 1 -- 172.21.15.12:0/1980828003 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8a0005850 con 0x558afe3d59f0 2024-03-23T23:05:49.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.489+0000 7fd8b7838740 10 client.26357 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:49.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.489+0000 7fd8b7838740 1 -- 172.21.15.12:0/1980828003 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x558afe446f60 con 0x558afe3d59f0 2024-03-23T23:05:49.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.490+0000 7fd88f400640 1 -- 172.21.15.12:0/1980828003 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd89c020e50 con 0x558afe3d59f0 2024-03-23T23:05:49.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.490+0000 7fd88f400640 1 -- 172.21.15.12:0/1980828003 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fd8a0036030 con 0x558afe3d59f0 2024-03-23T23:05:49.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.490+0000 7fd8b7838740 10 client.26357 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:49.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.490+0000 7fd8b7838740 10 client.26357 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:05:49.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.490+0000 7fd8b7838740 20 client.26357 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:49.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.490+0000 7fd8b7838740 1 --2- 172.21.15.12:0/1980828003 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x558afe447780 0x558afe449c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.490+0000 7fd8b7838740 4 client.26357 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:05:49.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.490+0000 7fd8b7838740 1 -- 172.21.15.12:0/1980828003 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558afe341b50 con 0x558afe447780 2024-03-23T23:05:49.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.490+0000 7fd8ad400640 1 --2- 172.21.15.12:0/1980828003 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x558afe447780 0x558afe449c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.491+0000 7fd8ad400640 1 --2- 172.21.15.12:0/1980828003 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x558afe447780 0x558afe449c40 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7001412e84b95e13 server_cookie=e929e0fa543eb791 in_seq=0 out_seq=0 2024-03-23T23:05:49.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.491+0000 7fd88f400640 10 client.26357 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:05:49.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.491+0000 7fd88f400640 1 -- 172.21.15.12:0/1980828003 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558afe341b50 con 0x558afe447780 2024-03-23T23:05:49.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.491+0000 7fd88f400640 10 client.26357 handle_command_reply: tid=0 2024-03-23T23:05:49.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.491+0000 7fd8b7838740 1 client.26357 shutdown 2024-03-23T23:05:49.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.491+0000 7fd8b7838740 1 -- 172.21.15.12:0/1980828003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d59f0 msgr2=0x558afe3e2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.491+0000 7fd8b7838740 1 --2- 172.21.15.12:0/1980828003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d59f0 0x558afe3e2be0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000bb10 tx=0x7fd8a000deb0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.491+0000 7fd8b7838740 1 -- 172.21.15.12:0/1980828003 shutdown_connections 2024-03-23T23:05:49.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.491+0000 7fd8b7838740 1 --2- 172.21.15.12:0/1980828003 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x558afe447780 0x558afe449c40 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.491+0000 7fd8b7838740 1 --2- 172.21.15.12:0/1980828003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558afe3d7580 0x558afe3e6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.491+0000 7fd8b7838740 1 --2- 172.21.15.12:0/1980828003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558afe3d6bc0 0x558afe3e3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.491+0000 7fd88f400640 0 client.26357 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:05:49.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.491+0000 7fd8b7838740 1 --2- 172.21.15.12:0/1980828003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558afe3d59f0 0x558afe3e2be0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.491+0000 7fd8b7838740 1 -- 172.21.15.12:0/1980828003 >> 172.21.15.12:0/1980828003 conn(0x558afe3d0de0 msgr2=0x558afe3d0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:49.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.491+0000 7fd8b7838740 1 -- 172.21.15.12:0/1980828003 shutdown_connections 2024-03-23T23:05:49.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.491+0000 7fd8b7838740 1 -- 172.21.15.12:0/1980828003 wait complete. 2024-03-23T23:05:49.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.492+0000 7fd8b7838740 20 client.26357 trim_cache size 0 max 16384 2024-03-23T23:05:49.495 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:05:49.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.492+0000 7fd8af200640 1 -- 172.21.15.12:0/217700460 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd8940a2880 msgr2=0x7fd8940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.492+0000 7fd8af200640 1 --2- 172.21.15.12:0/217700460 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd8940a2880 0x7fd8940a4d40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x558afe3d6ac0 tx=0x7fd89c009770 comp rx=0 tx=0).stop 2024-03-23T23:05:49.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.492+0000 7fd8af200640 1 -- 172.21.15.12:0/217700460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8b0109430 msgr2=0x7fd8b01a0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.492+0000 7fd8af200640 1 --2- 172.21.15.12:0/217700460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8b0109430 0x7fd8b01a0fb0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd8a802c6d0 tx=0x7fd8a8003ba0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.492+0000 7fd8af200640 1 -- 172.21.15.12:0/217700460 shutdown_connections 2024-03-23T23:05:49.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.492+0000 7fd8af200640 1 --2- 172.21.15.12:0/217700460 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd8940a2880 0x7fd8940a4d40 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.492+0000 7fd8af200640 1 --2- 172.21.15.12:0/217700460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8b0109430 0x7fd8b01a0fb0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.492+0000 7fd8af200640 1 --2- 172.21.15.12:0/217700460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8b0075660 0x7fd8b0199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.492+0000 7fd8af200640 1 --2- 172.21.15.12:0/217700460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8b0074080 0x7fd8b0199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.492+0000 7fd8af200640 1 -- 172.21.15.12:0/217700460 >> 172.21.15.12:0/217700460 conn(0x7fd8b00fcb10 msgr2=0x7fd8b01053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:49.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.492+0000 7fd8af200640 1 -- 172.21.15.12:0/217700460 shutdown_connections 2024-03-23T23:05:49.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.492+0000 7fd8af200640 1 -- 172.21.15.12:0/217700460 wait complete. 2024-03-23T23:05:49.506 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:49 smithi012 ceph-mon[31695]: pgmap v510: 97 pgs: 97 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:05:49.506 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:49 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1622021157' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:05:49.508 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:05:49.508 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-23T23:05:49.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:49 smithi173 ceph-mon[36116]: pgmap v510: 97 pgs: 97 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:05:49.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:49 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1622021157' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:05:49.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:49 smithi203 ceph-mon[36693]: pgmap v510: 97 pgs: 97 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:05:49.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:49 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1622021157' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:05:49.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.741+0000 7f789ee00640 1 -- 172.21.15.12:0/662347443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7898105240 msgr2=0x7f78981056c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.741+0000 7f789ee00640 1 --2- 172.21.15.12:0/662347443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7898105240 0x7f78981056c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7890009900 tx=0x7f789002bc30 comp rx=0 tx=0).stop 2024-03-23T23:05:49.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.741+0000 7f789ee00640 1 -- 172.21.15.12:0/662347443 shutdown_connections 2024-03-23T23:05:49.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.741+0000 7f789ee00640 1 --2- 172.21.15.12:0/662347443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7898105240 0x7f78981056c0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.741+0000 7f789ee00640 1 --2- 172.21.15.12:0/662347443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7898102890 0x7f7898104d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.741+0000 7f789ee00640 1 --2- 172.21.15.12:0/662347443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78980fff20 0x7f7898102350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.741+0000 7f789ee00640 1 -- 172.21.15.12:0/662347443 >> 172.21.15.12:0/662347443 conn(0x7f78980fbb40 msgr2=0x7f78980fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:49.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.741+0000 7f789ee00640 1 -- 172.21.15.12:0/662347443 shutdown_connections 2024-03-23T23:05:49.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.741+0000 7f789ee00640 1 -- 172.21.15.12:0/662347443 wait complete. 2024-03-23T23:05:49.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.742+0000 7f789ee00640 1 Processor -- start 2024-03-23T23:05:49.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.743+0000 7f789ee00640 1 -- start start 2024-03-23T23:05:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.743+0000 7f789ee00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78980fff20 0x7f789819df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.743+0000 7f789ee00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7898102890 0x7f789819e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.743+0000 7f789ee00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7898105240 0x7f78981a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.743+0000 7f789ee00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7898113970 con 0x7f7898102890 2024-03-23T23:05:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.743+0000 7f789ee00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78981137f0 con 0x7f78980fff20 2024-03-23T23:05:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.743+0000 7f789ee00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7898113af0 con 0x7f7898105240 2024-03-23T23:05:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.743+0000 7f7897400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7898102890 0x7f789819e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.743+0000 7f7897400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7898102890 0x7f789819e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52708/0 (socket says 172.21.15.12:52708) 2024-03-23T23:05:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.743+0000 7f7897400640 1 -- 172.21.15.12:0/4199152810 learned_addr learned my addr 172.21.15.12:0/4199152810 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.744+0000 7f7897e00640 1 --2- 172.21.15.12:0/4199152810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78980fff20 0x7f789819df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.744+0000 7f789ce00640 1 --2- 172.21.15.12:0/4199152810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7898105240 0x7f78981a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.744+0000 7f7897400640 1 -- 172.21.15.12:0/4199152810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7898105240 msgr2=0x7f78981a5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.744+0000 7f7897400640 1 --2- 172.21.15.12:0/4199152810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7898105240 0x7f78981a5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.744+0000 7f7897400640 1 -- 172.21.15.12:0/4199152810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78980fff20 msgr2=0x7f789819df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.744+0000 7f7897400640 1 --2- 172.21.15.12:0/4199152810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78980fff20 0x7f789819df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.744+0000 7f7897400640 1 -- 172.21.15.12:0/4199152810 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78981a5bb0 con 0x7f7898102890 2024-03-23T23:05:49.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.744+0000 7f7897400640 1 --2- 172.21.15.12:0/4199152810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7898102890 0x7f789819e4d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f788000e9f0 tx=0x7f788000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:49.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.744+0000 7f7894c00640 1 -- 172.21.15.12:0/4199152810 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788000cd80 con 0x7f7898102890 2024-03-23T23:05:49.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.745+0000 7f7894c00640 1 -- 172.21.15.12:0/4199152810 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7880005500 con 0x7f7898102890 2024-03-23T23:05:49.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.745+0000 7f789ee00640 1 -- 172.21.15.12:0/4199152810 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78981a5ea0 con 0x7f7898102890 2024-03-23T23:05:49.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.745+0000 7f7894c00640 1 -- 172.21.15.12:0/4199152810 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7880010690 con 0x7f7898102890 2024-03-23T23:05:49.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.745+0000 7f789ee00640 1 -- 172.21.15.12:0/4199152810 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78981a6310 con 0x7f7898102890 2024-03-23T23:05:49.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.748+0000 7f78a1b9d740 1 Processor -- start 2024-03-23T23:05:49.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.748+0000 7f78a1b9d740 1 -- start start 2024-03-23T23:05:49.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.748+0000 7f7894c00640 1 -- 172.21.15.12:0/4199152810 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f78800046c0 con 0x7f7898102890 2024-03-23T23:05:49.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.748+0000 7f78a1b9d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2593d810 0x561b2593dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.748+0000 7f78a1b9d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561b2593e9e0 0x561b2593ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.748+0000 7f78a1b9d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b2593f3a0 0x561b25945c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.749+0000 7f78a1b9d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b258a22c0 con 0x561b2593e9e0 2024-03-23T23:05:49.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.749+0000 7f78a1b9d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b2584ac90 con 0x561b2593f3a0 2024-03-23T23:05:49.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.749+0000 7f78a1b9d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b2589e360 con 0x561b2593d810 2024-03-23T23:05:49.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.749+0000 7f789ce00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2593d810 0x561b2593dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.749+0000 7f789ce00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2593d810 0x561b2593dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45834/0 (socket says 172.21.15.12:45834) 2024-03-23T23:05:49.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.749+0000 7f7897400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b2593f3a0 0x561b25945c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.749+0000 7f789ce00640 1 -- 172.21.15.12:0/4159881326 learned_addr learned my addr 172.21.15.12:0/4159881326 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:49.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.749+0000 7f7897e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561b2593e9e0 0x561b2593ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.749+0000 7f789ce00640 1 -- 172.21.15.12:0/4159881326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b2593f3a0 msgr2=0x561b25945c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:49.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.749+0000 7f789ce00640 1 --2- 172.21.15.12:0/4159881326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b2593f3a0 0x561b25945c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.749+0000 7f789ce00640 1 -- 172.21.15.12:0/4159881326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561b2593e9e0 msgr2=0x561b2593ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:49.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.749+0000 7f789ce00640 1 --2- 172.21.15.12:0/4159881326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561b2593e9e0 0x561b2593ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.749+0000 7f789ce00640 1 -- 172.21.15.12:0/4159881326 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b2586f080 con 0x561b2593d810 2024-03-23T23:05:49.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.750+0000 7f789ce00640 1 --2- 172.21.15.12:0/4159881326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2593d810 0x561b2593dc30 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f789002c240 tx=0x7f789002c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7bb63059fd9cd74 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:49.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.750+0000 7f7894c00640 1 --2- 172.21.15.12:0/4199152810 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f787c0a2800 0x7f787c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.750+0000 7f788d600640 1 -- 172.21.15.12:0/4159881326 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7890044020 con 0x561b2593d810 2024-03-23T23:05:49.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.750+0000 7f788d600640 1 -- 172.21.15.12:0/4159881326 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f789002cb80 con 0x561b2593d810 2024-03-23T23:05:49.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.750+0000 7f788d600640 1 -- 172.21.15.12:0/4159881326 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7890004920 con 0x561b2593d810 2024-03-23T23:05:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.750+0000 7f7894c00640 1 -- 172.21.15.12:0/4199152810 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f78800cef30 con 0x7f7898102890 2024-03-23T23:05:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.750+0000 7f7897e00640 1 --2- 172.21.15.12:0/4199152810 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f787c0a2800 0x7f787c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.751+0000 7f78a1b9d740 1 -- 172.21.15.12:0/4159881326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2593d810 msgr2=0x561b2593dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.751+0000 7f78a1b9d740 1 --2- 172.21.15.12:0/4159881326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2593d810 0x561b2593dc30 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f789002c240 tx=0x7f789002c680 comp rx=0 tx=0).stop 2024-03-23T23:05:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.751+0000 7f78a1b9d740 1 -- 172.21.15.12:0/4159881326 shutdown_connections 2024-03-23T23:05:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.751+0000 7f78a1b9d740 1 --2- 172.21.15.12:0/4159881326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b2593f3a0 0x561b25945c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.751+0000 7f78a1b9d740 1 --2- 172.21.15.12:0/4159881326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561b2593e9e0 0x561b2593ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.751+0000 7f78a1b9d740 1 --2- 172.21.15.12:0/4159881326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2593d810 0x561b2593dc30 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.751+0000 7f78a1b9d740 1 -- 172.21.15.12:0/4159881326 >> 172.21.15.12:0/4159881326 conn(0x561b25938c00 msgr2=0x561b25939000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.751+0000 7f78a1b9d740 1 -- 172.21.15.12:0/4159881326 shutdown_connections 2024-03-23T23:05:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.751+0000 7f7897e00640 1 --2- 172.21.15.12:0/4199152810 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f787c0a2800 0x7f787c0a4cc0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7888005fd0 tx=0x7f7888009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.751+0000 7f78a1b9d740 1 -- 172.21.15.12:0/4159881326 wait complete. 2024-03-23T23:05:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.752+0000 7f78a1b9d740 1 -- start start 2024-03-23T23:05:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.752+0000 7f78a1b9d740 1 Processor -- start 2024-03-23T23:05:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.752+0000 7f78a1b9d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561b2593d810 0x561b25947a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.752+0000 7f78a1b9d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2593e9e0 0x561b25947fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.752+0000 7f78a1b9d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b2593f3a0 0x561b2594f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.752+0000 7f78a1b9d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b25946350 con 0x561b2593d810 2024-03-23T23:05:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.752+0000 7f78a1b9d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b2585e0d0 con 0x561b2593f3a0 2024-03-23T23:05:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.752+0000 7f78a1b9d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b259464d0 con 0x561b2593e9e0 2024-03-23T23:05:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.752+0000 7f78a1b9d740 10 client.0 _ll_register_callbacks cb 0x561b25862e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.752+0000 7f7897e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561b2593d810 0x561b25947a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.753+0000 7f7897e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561b2593d810 0x561b25947a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52720/0 (socket says 172.21.15.12:52720) 2024-03-23T23:05:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.753+0000 7f7897e00640 1 -- 172.21.15.12:0/1294741013 learned_addr learned my addr 172.21.15.12:0/1294741013 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.753+0000 7f7897400640 1 --2- 172.21.15.12:0/1294741013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2593e9e0 0x561b25947fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.753+0000 7f789ce00640 1 --2- 172.21.15.12:0/1294741013 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b2593f3a0 0x561b2594f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.753+0000 7f7897e00640 1 -- 172.21.15.12:0/1294741013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2593e9e0 msgr2=0x561b25947fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.753+0000 7f7897e00640 1 --2- 172.21.15.12:0/1294741013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2593e9e0 0x561b25947fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.753+0000 7f7897e00640 1 -- 172.21.15.12:0/1294741013 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b2593f3a0 msgr2=0x561b2594f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.753+0000 7f7897e00640 1 --2- 172.21.15.12:0/1294741013 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b2593f3a0 0x561b2594f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.753+0000 7f7897e00640 1 -- 172.21.15.12:0/1294741013 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561b255b63f0 con 0x561b2593d810 2024-03-23T23:05:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.753+0000 7f7897400640 1 --2- 172.21.15.12:0/1294741013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2593e9e0 0x561b25947fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.753+0000 7f789ce00640 1 --2- 172.21.15.12:0/1294741013 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b2593f3a0 0x561b2594f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.753+0000 7f7897e00640 1 --2- 172.21.15.12:0/1294741013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561b2593d810 0x561b25947a80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7888003fc0 tx=0x7f7888004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89365ebdfc34f536 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.753+0000 7f7886000640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:05:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.754+0000 7f7886000640 1 -- 172.21.15.12:0/1294741013 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788800f970 con 0x561b2593d810 2024-03-23T23:05:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.754+0000 7f7886000640 1 -- 172.21.15.12:0/1294741013 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f788800fb10 con 0x561b2593d810 2024-03-23T23:05:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.754+0000 7f78a1b9d740 1 -- 172.21.15.12:0/1294741013 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f789002cb80 con 0x561b2593d810 2024-03-23T23:05:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.754+0000 7f7886000640 1 -- 172.21.15.12:0/1294741013 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f788800fd10 con 0x561b2593d810 2024-03-23T23:05:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.754+0000 7f78a1b9d740 10 client.16239 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.754+0000 7f78a1b9d740 1 -- 172.21.15.12:0/1294741013 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561b256807d0 con 0x561b2593d810 2024-03-23T23:05:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.754+0000 7f7886000640 1 -- 172.21.15.12:0/1294741013 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f789002cb80 con 0x561b2593d810 2024-03-23T23:05:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.754+0000 7f7886000640 1 -- 172.21.15.12:0/1294741013 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f7888026070 con 0x561b2593d810 2024-03-23T23:05:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.754+0000 7f78a1b9d740 10 client.16239 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.754+0000 7f78a1b9d740 10 client.16239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.754+0000 7f78a1b9d740 20 client.16239 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.755+0000 7f78a1b9d740 1 --2- 172.21.15.12:0/1294741013 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561b25953550 0x561b259559a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.755+0000 7f78a1b9d740 4 client.16239 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:05:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.755+0000 7f78a1b9d740 1 -- 172.21.15.12:0/1294741013 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561b258af800 con 0x561b25953550 2024-03-23T23:05:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.755+0000 7f7897400640 1 --2- 172.21.15.12:0/1294741013 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561b25953550 0x561b259559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.755+0000 7f7897400640 1 --2- 172.21.15.12:0/1294741013 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561b25953550 0x561b259559a0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=467fed86d787cb4a server_cookie=d4a0c28aba08c7c6 in_seq=0 out_seq=0 2024-03-23T23:05:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.755+0000 7f7886000640 10 client.16239 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.756+0000 7f7886000640 1 -- 172.21.15.12:0/1294741013 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561b258af800 con 0x561b25953550 2024-03-23T23:05:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.756+0000 7f7886000640 10 client.16239 handle_command_reply: tid=0 2024-03-23T23:05:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.756+0000 7f78a1b9d740 1 client.16239 shutdown 2024-03-23T23:05:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.756+0000 7f78a1b9d740 1 -- 172.21.15.12:0/1294741013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561b2593d810 msgr2=0x561b25947a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.756+0000 7f78a1b9d740 1 --2- 172.21.15.12:0/1294741013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561b2593d810 0x561b25947a80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7888003fc0 tx=0x7f7888004ab0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.756+0000 7f78a1b9d740 1 -- 172.21.15.12:0/1294741013 shutdown_connections 2024-03-23T23:05:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.756+0000 7f78a1b9d740 1 --2- 172.21.15.12:0/1294741013 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561b25953550 0x561b259559a0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.756+0000 7f78a1b9d740 1 --2- 172.21.15.12:0/1294741013 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b2593f3a0 0x561b2594f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.756+0000 7f78a1b9d740 1 --2- 172.21.15.12:0/1294741013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2593e9e0 0x561b25947fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.756+0000 7f7886000640 0 client.16239 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:49.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.756+0000 7f78a1b9d740 1 --2- 172.21.15.12:0/1294741013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561b2593d810 0x561b25947a80 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.756+0000 7f78a1b9d740 1 -- 172.21.15.12:0/1294741013 >> 172.21.15.12:0/1294741013 conn(0x561b25938c00 msgr2=0x561b25937eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:49.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.756+0000 7f78a1b9d740 1 -- 172.21.15.12:0/1294741013 shutdown_connections 2024-03-23T23:05:49.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.757+0000 7f78a1b9d740 1 -- 172.21.15.12:0/1294741013 wait complete. 2024-03-23T23:05:49.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.757+0000 7f78a1b9d740 20 client.16239 trim_cache size 0 max 16384 2024-03-23T23:05:49.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.770+0000 7f78a1b9d740 1 Processor -- start 2024-03-23T23:05:49.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.770+0000 7f78a1b9d740 1 -- start start 2024-03-23T23:05:49.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.771+0000 7f78a1b9d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561b2593d810 0x561b25941a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.771+0000 7f78a1b9d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b2593e9e0 0x561b2594efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.771+0000 7f78a1b9d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2593f3a0 0x561b25941fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.771+0000 7f78a1b9d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78981b15d0 con 0x561b2593d810 2024-03-23T23:05:49.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.771+0000 7f78a1b9d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78981af0d0 con 0x561b2593e9e0 2024-03-23T23:05:49.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.771+0000 7f78a1b9d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78981ad170 con 0x561b2593f3a0 2024-03-23T23:05:49.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.771+0000 7f7897400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b2593e9e0 0x561b2594efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.771+0000 7f7897e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561b2593d810 0x561b25941a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.771+0000 7f789ce00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2593f3a0 0x561b25941fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.771+0000 7f7897400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b2593e9e0 0x561b2594efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50218/0 (socket says 172.21.15.12:50218) 2024-03-23T23:05:49.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.771+0000 7f7897400640 1 -- 172.21.15.12:0/1044838431 learned_addr learned my addr 172.21.15.12:0/1044838431 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:49.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.771+0000 7f7897400640 1 -- 172.21.15.12:0/1044838431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2593f3a0 msgr2=0x561b25941fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:49.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.771+0000 7f7897400640 1 --2- 172.21.15.12:0/1044838431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2593f3a0 0x561b25941fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.771+0000 7f7897400640 1 -- 172.21.15.12:0/1044838431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561b2593d810 msgr2=0x561b25941a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:49.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.771+0000 7f7897400640 1 --2- 172.21.15.12:0/1044838431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561b2593d810 0x561b25941a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.771+0000 7f7897400640 1 -- 172.21.15.12:0/1044838431 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b259332c0 con 0x561b2593e9e0 2024-03-23T23:05:49.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.772+0000 7f7897400640 1 --2- 172.21.15.12:0/1044838431 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b2593e9e0 0x561b2594efc0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7880097ce0 tx=0x7f78800d1380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=383e13e58620edd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:49.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.772+0000 7f788cc00640 1 -- 172.21.15.12:0/1044838431 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788009b3c0 con 0x561b2593e9e0 2024-03-23T23:05:49.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.772+0000 7f788cc00640 1 -- 172.21.15.12:0/1044838431 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f788009bf80 con 0x561b2593e9e0 2024-03-23T23:05:49.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.772+0000 7f788cc00640 1 -- 172.21.15.12:0/1044838431 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788009b1e0 con 0x561b2593e9e0 2024-03-23T23:05:49.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.772+0000 7f78a1b9d740 1 -- 172.21.15.12:0/1044838431 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b2593e9e0 msgr2=0x561b2594efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.772+0000 7f78a1b9d740 1 --2- 172.21.15.12:0/1044838431 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b2593e9e0 0x561b2594efc0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7880097ce0 tx=0x7f78800d1380 comp rx=0 tx=0).stop 2024-03-23T23:05:49.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.773+0000 7f78a1b9d740 1 -- 172.21.15.12:0/1044838431 shutdown_connections 2024-03-23T23:05:49.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.773+0000 7f78a1b9d740 1 --2- 172.21.15.12:0/1044838431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2593f3a0 0x561b25941fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.773+0000 7f78a1b9d740 1 --2- 172.21.15.12:0/1044838431 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b2593e9e0 0x561b2594efc0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.773+0000 7f78a1b9d740 1 --2- 172.21.15.12:0/1044838431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561b2593d810 0x561b25941a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.773+0000 7f78a1b9d740 1 -- 172.21.15.12:0/1044838431 >> 172.21.15.12:0/1044838431 conn(0x561b25938c00 msgr2=0x561b2593c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:49.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.773+0000 7f78a1b9d740 1 -- 172.21.15.12:0/1044838431 shutdown_connections 2024-03-23T23:05:49.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.773+0000 7f78a1b9d740 1 -- 172.21.15.12:0/1044838431 wait complete. 2024-03-23T23:05:49.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.773+0000 7f78a1b9d740 1 -- start start 2024-03-23T23:05:49.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.773+0000 7f78a1b9d740 1 Processor -- start 2024-03-23T23:05:49.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.774+0000 7f78a1b9d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b2593d810 0x561b2594a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.774+0000 7f78a1b9d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561b2593e9e0 0x561b2594abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.774+0000 7f78a1b9d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2593f3a0 0x561b2594dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.774+0000 7f78a1b9d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78981ad770 con 0x561b2593e9e0 2024-03-23T23:05:49.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.774+0000 7f78a1b9d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78981aea70 con 0x561b2593d810 2024-03-23T23:05:49.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.774+0000 7f78a1b9d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78981addb0 con 0x561b2593f3a0 2024-03-23T23:05:49.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.774+0000 7f78a1b9d740 10 client.0 _ll_register_callbacks cb 0x7f7888003e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:49.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.774+0000 7f789ce00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2593f3a0 0x561b2594dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.774+0000 7f7897e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b2593d810 0x561b2594a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.774+0000 7f789ce00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2593f3a0 0x561b2594dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45868/0 (socket says 172.21.15.12:45868) 2024-03-23T23:05:49.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.774+0000 7f789ce00640 1 -- 172.21.15.12:0/1592262002 learned_addr learned my addr 172.21.15.12:0/1592262002 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:49.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.774+0000 7f7897400640 1 --2- 172.21.15.12:0/1592262002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561b2593e9e0 0x561b2594abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.774+0000 7f789ce00640 1 -- 172.21.15.12:0/1592262002 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b2593d810 msgr2=0x561b2594a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:49.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.774+0000 7f789ce00640 1 --2- 172.21.15.12:0/1592262002 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b2593d810 0x561b2594a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.774+0000 7f789ce00640 1 -- 172.21.15.12:0/1592262002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561b2593e9e0 msgr2=0x561b2594abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:49.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.774+0000 7f789ce00640 1 --2- 172.21.15.12:0/1592262002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561b2593e9e0 0x561b2594abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.774+0000 7f789ce00640 1 -- 172.21.15.12:0/1592262002 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561b2594e1f0 con 0x561b2593f3a0 2024-03-23T23:05:49.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.775+0000 7f789ce00640 1 --2- 172.21.15.12:0/1592262002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2593f3a0 0x561b2594dc00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f789002c240 tx=0x7f789002cbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d242d334796852d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:49.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.775+0000 7f7886000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:05:49.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.775+0000 7f7886000640 1 -- 172.21.15.12:0/1592262002 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7890051020 con 0x561b2593f3a0 2024-03-23T23:05:49.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.775+0000 7f7886000640 1 -- 172.21.15.12:0/1592262002 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7890005450 con 0x561b2593f3a0 2024-03-23T23:05:49.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.775+0000 7f78a1b9d740 1 -- 172.21.15.12:0/1592262002 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f788009bf80 con 0x561b2593f3a0 2024-03-23T23:05:49.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.775+0000 7f7886000640 1 -- 172.21.15.12:0/1592262002 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7890044070 con 0x561b2593f3a0 2024-03-23T23:05:49.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.775+0000 7f78a1b9d740 10 client.26338 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:49.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.775+0000 7f78a1b9d740 1 -- 172.21.15.12:0/1592262002 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561b259aee30 con 0x561b2593f3a0 2024-03-23T23:05:49.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.775+0000 7f7886000640 1 -- 172.21.15.12:0/1592262002 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f788009bf80 con 0x561b2593f3a0 2024-03-23T23:05:49.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.775+0000 7f7886000640 1 -- 172.21.15.12:0/1592262002 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f78900035f0 con 0x561b2593f3a0 2024-03-23T23:05:49.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.776+0000 7f78a1b9d740 10 client.26338 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:49.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.776+0000 7f78a1b9d740 10 client.26338 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:49.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.776+0000 7f78a1b9d740 20 client.26338 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:49.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.776+0000 7f78a1b9d740 1 --2- 172.21.15.12:0/1592262002 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561b259af210 0x561b259b1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:49.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.776+0000 7f78a1b9d740 4 client.26338 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:05:49.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.776+0000 7f78a1b9d740 1 -- 172.21.15.12:0/1592262002 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561b258a9b50 con 0x561b259af210 2024-03-23T23:05:49.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.776+0000 7f7897e00640 1 --2- 172.21.15.12:0/1592262002 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561b259af210 0x561b259b1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:49.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.776+0000 7f7897e00640 1 --2- 172.21.15.12:0/1592262002 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561b259af210 0x561b259b1680 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4820aa9390f9e70 server_cookie=74975af0a74b1adb in_seq=0 out_seq=0 2024-03-23T23:05:49.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.776+0000 7f7886000640 10 client.26338 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:49.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.777+0000 7f7886000640 1 -- 172.21.15.12:0/1592262002 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561b258a9b50 con 0x561b259af210 2024-03-23T23:05:49.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.777+0000 7f7886000640 10 client.26338 handle_command_reply: tid=0 2024-03-23T23:05:49.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.777+0000 7f78a1b9d740 1 client.26338 shutdown 2024-03-23T23:05:49.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.777+0000 7f78a1b9d740 1 -- 172.21.15.12:0/1592262002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2593f3a0 msgr2=0x561b2594dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.777+0000 7f78a1b9d740 1 --2- 172.21.15.12:0/1592262002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2593f3a0 0x561b2594dc00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f789002c240 tx=0x7f789002cbb0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.777+0000 7f78a1b9d740 1 -- 172.21.15.12:0/1592262002 shutdown_connections 2024-03-23T23:05:49.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.777+0000 7f78a1b9d740 1 --2- 172.21.15.12:0/1592262002 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561b259af210 0x561b259b1680 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.777+0000 7f78a1b9d740 1 --2- 172.21.15.12:0/1592262002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2593f3a0 0x561b2594dc00 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.777+0000 7f78a1b9d740 1 --2- 172.21.15.12:0/1592262002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561b2593e9e0 0x561b2594abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.777+0000 7f78a1b9d740 1 --2- 172.21.15.12:0/1592262002 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b2593d810 0x561b2594a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.777+0000 7f78a1b9d740 1 -- 172.21.15.12:0/1592262002 >> 172.21.15.12:0/1592262002 conn(0x561b25938c00 msgr2=0x561b25937620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:49.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.777+0000 7f7886000640 0 client.26338 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:49.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.777+0000 7f78a1b9d740 1 -- 172.21.15.12:0/1592262002 shutdown_connections 2024-03-23T23:05:49.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.777+0000 7f78a1b9d740 1 -- 172.21.15.12:0/1592262002 wait complete. 2024-03-23T23:05:49.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.777+0000 7f78a1b9d740 20 client.26338 trim_cache size 0 max 16384 2024-03-23T23:05:49.780 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:05:49.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.778+0000 7f789ee00640 1 -- 172.21.15.12:0/4199152810 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f787c0a2800 msgr2=0x7f787c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.778+0000 7f789ee00640 1 --2- 172.21.15.12:0/4199152810 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f787c0a2800 0x7f787c0a4cc0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7888005fd0 tx=0x7f7888009500 comp rx=0 tx=0).stop 2024-03-23T23:05:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.778+0000 7f789ee00640 1 -- 172.21.15.12:0/4199152810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7898102890 msgr2=0x7f789819e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.778+0000 7f789ee00640 1 --2- 172.21.15.12:0/4199152810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7898102890 0x7f789819e4d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f788000e9f0 tx=0x7f788000edf0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.778+0000 7f789ee00640 1 -- 172.21.15.12:0/4199152810 shutdown_connections 2024-03-23T23:05:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.778+0000 7f789ee00640 1 --2- 172.21.15.12:0/4199152810 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f787c0a2800 0x7f787c0a4cc0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.778+0000 7f789ee00640 1 --2- 172.21.15.12:0/4199152810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7898105240 0x7f78981a5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.778+0000 7f789ee00640 1 --2- 172.21.15.12:0/4199152810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7898102890 0x7f789819e4d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.778+0000 7f789ee00640 1 --2- 172.21.15.12:0/4199152810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78980fff20 0x7f789819df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.778+0000 7f789ee00640 1 -- 172.21.15.12:0/4199152810 >> 172.21.15.12:0/4199152810 conn(0x7f78980fbb40 msgr2=0x7f78981008a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.778+0000 7f789ee00640 1 -- 172.21.15.12:0/4199152810 shutdown_connections 2024-03-23T23:05:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:49.778+0000 7f789ee00640 1 -- 172.21.15.12:0/4199152810 wait complete. 2024-03-23T23:05:50.795 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-23T23:05:50.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:05:50.795 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-23T23:05:51.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.039+0000 7f5fe8800640 1 -- 172.21.15.12:0/3765402272 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fe0104ad0 msgr2=0x7f5fe0104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.039+0000 7f5fe8800640 1 --2- 172.21.15.12:0/3765402272 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fe0104ad0 0x7f5fe0104f50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0009900 tx=0x7f5fd002bc30 comp rx=0 tx=0).stop 2024-03-23T23:05:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.039+0000 7f5fe8800640 1 -- 172.21.15.12:0/3765402272 shutdown_connections 2024-03-23T23:05:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.039+0000 7f5fe8800640 1 --2- 172.21.15.12:0/3765402272 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fe0105490 0x7f5fe010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.039+0000 7f5fe8800640 1 --2- 172.21.15.12:0/3765402272 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fe0104ad0 0x7f5fe0104f50 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.039+0000 7f5fe8800640 1 --2- 172.21.15.12:0/3765402272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fe01038e0 0x7f5fe0103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.039+0000 7f5fe8800640 1 -- 172.21.15.12:0/3765402272 >> 172.21.15.12:0/3765402272 conn(0x7f5fe00fee50 msgr2=0x7f5fe0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.039+0000 7f5fe8800640 1 -- 172.21.15.12:0/3765402272 shutdown_connections 2024-03-23T23:05:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.039+0000 7f5fe8800640 1 -- 172.21.15.12:0/3765402272 wait complete. 2024-03-23T23:05:51.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.040+0000 7f5fe8800640 1 Processor -- start 2024-03-23T23:05:51.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.040+0000 7f5fe8800640 1 -- start start 2024-03-23T23:05:51.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.041+0000 7f5fe8800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fe01038e0 0x7f5fe0199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.041+0000 7f5fe8800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fe0104ad0 0x7f5fe019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.041+0000 7f5fe8800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fe0105490 0x7f5fe01a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.041+0000 7f5fe8800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe010f5b0 con 0x7f5fe0105490 2024-03-23T23:05:51.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.041+0000 7f5fe8800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe010f430 con 0x7f5fe0104ad0 2024-03-23T23:05:51.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.041+0000 7f5fe8800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe010f730 con 0x7f5fe01038e0 2024-03-23T23:05:51.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.041+0000 7f5fe5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fe01038e0 0x7f5fe0199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.041+0000 7f5fe5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fe01038e0 0x7f5fe0199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45896/0 (socket says 172.21.15.12:45896) 2024-03-23T23:05:51.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.041+0000 7f5fe5e00640 1 -- 172.21.15.12:0/3423927129 learned_addr learned my addr 172.21.15.12:0/3423927129 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:51.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.041+0000 7f5fe6800640 1 --2- 172.21.15.12:0/3423927129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fe0105490 0x7f5fe01a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.041+0000 7f5fe5400640 1 --2- 172.21.15.12:0/3423927129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fe0104ad0 0x7f5fe019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.041+0000 7f5fe5e00640 1 -- 172.21.15.12:0/3423927129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fe0104ad0 msgr2=0x7f5fe019a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:51.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.041+0000 7f5fe5e00640 1 --2- 172.21.15.12:0/3423927129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fe0104ad0 0x7f5fe019a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.041+0000 7f5fe5e00640 1 -- 172.21.15.12:0/3423927129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fe0105490 msgr2=0x7f5fe01a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:51.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.041+0000 7f5fe5e00640 1 --2- 172.21.15.12:0/3423927129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fe0105490 0x7f5fe01a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.041+0000 7f5fe5e00640 1 -- 172.21.15.12:0/3423927129 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe01a1850 con 0x7f5fe01038e0 2024-03-23T23:05:51.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.041+0000 7f5fe5400640 1 --2- 172.21.15.12:0/3423927129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fe0104ad0 0x7f5fe019a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:05:51.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.042+0000 7f5fe5e00640 1 --2- 172.21.15.12:0/3423927129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fe01038e0 0x7f5fe0199c40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5fd800eb70 tx=0x7f5fd800ef70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:51.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.042+0000 7f5fd6a00640 1 -- 172.21.15.12:0/3423927129 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd8003970 con 0x7f5fe01038e0 2024-03-23T23:05:51.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.042+0000 7f5fd6a00640 1 -- 172.21.15.12:0/3423927129 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fd8004020 con 0x7f5fe01038e0 2024-03-23T23:05:51.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.042+0000 7f5fe8800640 1 -- 172.21.15.12:0/3423927129 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe01a1b40 con 0x7f5fe01038e0 2024-03-23T23:05:51.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.042+0000 7f5fd6a00640 1 -- 172.21.15.12:0/3423927129 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd8010670 con 0x7f5fe01038e0 2024-03-23T23:05:51.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.042+0000 7f5fe8800640 1 -- 172.21.15.12:0/3423927129 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe01a2040 con 0x7f5fe01038e0 2024-03-23T23:05:51.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.047+0000 7f5feb470740 1 Processor -- start 2024-03-23T23:05:51.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.047+0000 7f5feb470740 1 -- start start 2024-03-23T23:05:51.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.047+0000 7f5fd6a00640 1 -- 172.21.15.12:0/3423927129 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5fd80041d0 con 0x7f5fe01038e0 2024-03-23T23:05:51.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.047+0000 7f5feb470740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd884f98b0 0x55cd884f9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.047+0000 7f5feb470740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884fa2a0 0x55cd884feda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.047+0000 7f5feb470740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd884ff410 0x55cd885018a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.047+0000 7f5feb470740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd88456eb0 con 0x55cd884ff410 2024-03-23T23:05:51.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.047+0000 7f5feb470740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd883ffa60 con 0x55cd884f98b0 2024-03-23T23:05:51.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.047+0000 7f5feb470740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd88452f50 con 0x55cd884fa2a0 2024-03-23T23:05:51.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.048+0000 7f5fe5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884fa2a0 0x55cd884feda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.048+0000 7f5fe6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd884f98b0 0x55cd884f9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.048+0000 7f5fe5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884fa2a0 0x55cd884feda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45908/0 (socket says 172.21.15.12:45908) 2024-03-23T23:05:51.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.048+0000 7f5fe5e00640 1 -- 172.21.15.12:0/2307604365 learned_addr learned my addr 172.21.15.12:0/2307604365 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:51.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.048+0000 7f5fe5400640 1 --2- 172.21.15.12:0/2307604365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd884ff410 0x55cd885018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.048+0000 7f5fe5e00640 1 -- 172.21.15.12:0/2307604365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd884f98b0 msgr2=0x55cd884f9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:51.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.048+0000 7f5fe5e00640 1 --2- 172.21.15.12:0/2307604365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd884f98b0 0x55cd884f9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.048+0000 7f5fe5e00640 1 -- 172.21.15.12:0/2307604365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd884ff410 msgr2=0x55cd885018a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:51.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.048+0000 7f5fe5e00640 1 --2- 172.21.15.12:0/2307604365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd884ff410 0x55cd885018a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.048+0000 7f5fe5e00640 1 -- 172.21.15.12:0/2307604365 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd882337e0 con 0x55cd884fa2a0 2024-03-23T23:05:51.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.048+0000 7f5fe5e00640 1 --2- 172.21.15.12:0/2307604365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884fa2a0 0x55cd884feda0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80d72f0 tx=0x7f5fd80d73e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a71dabb77d754227 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:51.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.049+0000 7f5fcb000640 1 -- 172.21.15.12:0/2307604365 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd80e2070 con 0x55cd884fa2a0 2024-03-23T23:05:51.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.049+0000 7f5fcb000640 1 -- 172.21.15.12:0/2307604365 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fd80d7e20 con 0x55cd884fa2a0 2024-03-23T23:05:51.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.049+0000 7f5fcb000640 1 -- 172.21.15.12:0/2307604365 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd80dd640 con 0x55cd884fa2a0 2024-03-23T23:05:51.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.049+0000 7f5fd6a00640 1 --2- 172.21.15.12:0/3423927129 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5fcc0a2800 0x7f5fcc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.049+0000 7f5fd6a00640 1 -- 172.21.15.12:0/3423927129 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5fd80cf270 con 0x7f5fe01038e0 2024-03-23T23:05:51.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.049+0000 7f5fe6800640 1 --2- 172.21.15.12:0/3423927129 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5fcc0a2800 0x7f5fcc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.049+0000 7f5feb470740 1 -- 172.21.15.12:0/2307604365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884fa2a0 msgr2=0x55cd884feda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:51.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.049+0000 7f5feb470740 1 --2- 172.21.15.12:0/2307604365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884fa2a0 0x55cd884feda0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80d72f0 tx=0x7f5fd80d73e0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.049+0000 7f5feb470740 1 -- 172.21.15.12:0/2307604365 shutdown_connections 2024-03-23T23:05:51.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.049+0000 7f5feb470740 1 --2- 172.21.15.12:0/2307604365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd884ff410 0x55cd885018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.049+0000 7f5feb470740 1 --2- 172.21.15.12:0/2307604365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884fa2a0 0x55cd884feda0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.049+0000 7f5feb470740 1 --2- 172.21.15.12:0/2307604365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd884f98b0 0x55cd884f9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.049+0000 7f5feb470740 1 -- 172.21.15.12:0/2307604365 >> 172.21.15.12:0/2307604365 conn(0x55cd884ec610 msgr2=0x55cd884eca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:51.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.049+0000 7f5feb470740 1 -- 172.21.15.12:0/2307604365 shutdown_connections 2024-03-23T23:05:51.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.050+0000 7f5feb470740 1 -- 172.21.15.12:0/2307604365 wait complete. 2024-03-23T23:05:51.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.050+0000 7f5fe6800640 1 --2- 172.21.15.12:0/3423927129 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5fcc0a2800 0x7f5fcc0a4cc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x55cd884fab00 tx=0x7f5fdc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:51.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.050+0000 7f5feb470740 1 -- start start 2024-03-23T23:05:51.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.050+0000 7f5feb470740 1 Processor -- start 2024-03-23T23:05:51.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.051+0000 7f5feb470740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd884f98b0 0x55cd88503ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.051+0000 7f5feb470740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884fa2a0 0x55cd88504000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.051+0000 7f5feb470740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd884ff410 0x55cd8850b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.051+0000 7f5feb470740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd885023c0 con 0x55cd884ff410 2024-03-23T23:05:51.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.051+0000 7f5feb470740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd88412eb0 con 0x55cd884f98b0 2024-03-23T23:05:51.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.051+0000 7f5feb470740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd88502540 con 0x55cd884fa2a0 2024-03-23T23:05:51.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.051+0000 7f5feb470740 10 client.0 _ll_register_callbacks cb 0x55cd88407150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:51.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.051+0000 7f5fe5400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884fa2a0 0x55cd88504000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.051+0000 7f5fe5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd884f98b0 0x55cd88503ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.051+0000 7f5fe5400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884fa2a0 0x55cd88504000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45912/0 (socket says 172.21.15.12:45912) 2024-03-23T23:05:51.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.051+0000 7f5fe6800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd884ff410 0x55cd8850b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.051+0000 7f5fe5400640 1 -- 172.21.15.12:0/263220671 learned_addr learned my addr 172.21.15.12:0/263220671 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:51.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.051+0000 7f5fe5400640 1 -- 172.21.15.12:0/263220671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd884f98b0 msgr2=0x55cd88503ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:51.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.051+0000 7f5fe5400640 1 --2- 172.21.15.12:0/263220671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd884f98b0 0x55cd88503ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.051+0000 7f5fe5400640 1 -- 172.21.15.12:0/263220671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd884ff410 msgr2=0x55cd8850b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:51.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.051+0000 7f5fe5400640 1 --2- 172.21.15.12:0/263220671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd884ff410 0x55cd8850b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.051+0000 7f5fe5400640 1 -- 172.21.15.12:0/263220671 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cd88293f70 con 0x55cd884fa2a0 2024-03-23T23:05:51.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.051+0000 7f5fe5e00640 1 --2- 172.21.15.12:0/263220671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd884f98b0 0x55cd88503ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:51.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.052+0000 7f5fe5400640 1 --2- 172.21.15.12:0/263220671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884fa2a0 0x55cd88504000 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0006010 tx=0x7f5fd0004250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acc7f98cc683c688 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:51.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.052+0000 7f5fbf400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:05:51.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.052+0000 7f5fbf400640 1 -- 172.21.15.12:0/263220671 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd0005d50 con 0x55cd884fa2a0 2024-03-23T23:05:51.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.052+0000 7f5fbf400640 1 -- 172.21.15.12:0/263220671 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5fd0005230 con 0x55cd884fa2a0 2024-03-23T23:05:51.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.052+0000 7f5feb470740 1 -- 172.21.15.12:0/263220671 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5fd80d7e20 con 0x55cd884fa2a0 2024-03-23T23:05:51.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.052+0000 7f5fbf400640 1 -- 172.21.15.12:0/263220671 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fd0005430 con 0x55cd884fa2a0 2024-03-23T23:05:51.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.052+0000 7f5feb470740 10 client.26350 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:51.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.052+0000 7f5feb470740 1 -- 172.21.15.12:0/263220671 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55cd8812b170 con 0x55cd884fa2a0 2024-03-23T23:05:51.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.052+0000 7f5fbf400640 1 -- 172.21.15.12:0/263220671 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fd80d7e20 con 0x55cd884fa2a0 2024-03-23T23:05:51.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.053+0000 7f5fbf400640 1 -- 172.21.15.12:0/263220671 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f5fd00037b0 con 0x55cd884fa2a0 2024-03-23T23:05:51.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.053+0000 7f5feb470740 10 client.26350 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:51.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.053+0000 7f5feb470740 10 client.26350 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:51.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.053+0000 7f5feb470740 20 client.26350 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:51.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.053+0000 7f5feb470740 1 --2- 172.21.15.12:0/263220671 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cd8850fa10 0x55cd88511e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.053+0000 7f5feb470740 4 client.26350 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:05:51.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.053+0000 7f5feb470740 1 -- 172.21.15.12:0/263220671 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cd884643f0 con 0x55cd8850fa10 2024-03-23T23:05:51.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.053+0000 7f5fe5e00640 1 --2- 172.21.15.12:0/263220671 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cd8850fa10 0x55cd88511e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.054+0000 7f5fbf400640 10 client.26350 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:51.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.053+0000 7f5fe5e00640 1 --2- 172.21.15.12:0/263220671 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cd8850fa10 0x55cd88511e70 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40684612fa010182 server_cookie=41352e048fece449 in_seq=0 out_seq=0 2024-03-23T23:05:51.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.054+0000 7f5fbf400640 1 -- 172.21.15.12:0/263220671 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cd884643f0 con 0x55cd8850fa10 2024-03-23T23:05:51.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.054+0000 7f5fbf400640 10 client.26350 handle_command_reply: tid=0 2024-03-23T23:05:51.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.054+0000 7f5feb470740 1 client.26350 shutdown 2024-03-23T23:05:51.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.054+0000 7f5feb470740 1 -- 172.21.15.12:0/263220671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884fa2a0 msgr2=0x55cd88504000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:51.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.054+0000 7f5feb470740 1 --2- 172.21.15.12:0/263220671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884fa2a0 0x55cd88504000 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0006010 tx=0x7f5fd0004250 comp rx=0 tx=0).stop 2024-03-23T23:05:51.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.055+0000 7f5feb470740 1 -- 172.21.15.12:0/263220671 shutdown_connections 2024-03-23T23:05:51.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.055+0000 7f5feb470740 1 --2- 172.21.15.12:0/263220671 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cd8850fa10 0x55cd88511e70 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.055+0000 7f5feb470740 1 --2- 172.21.15.12:0/263220671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd884ff410 0x55cd8850b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.055+0000 7f5feb470740 1 --2- 172.21.15.12:0/263220671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884fa2a0 0x55cd88504000 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.055+0000 7f5feb470740 1 --2- 172.21.15.12:0/263220671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd884f98b0 0x55cd88503ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.055+0000 7f5fbf400640 0 client.26350 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:51.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.055+0000 7f5feb470740 1 -- 172.21.15.12:0/263220671 >> 172.21.15.12:0/263220671 conn(0x55cd884ec610 msgr2=0x55cd884eb830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:51.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.055+0000 7f5feb470740 1 -- 172.21.15.12:0/263220671 shutdown_connections 2024-03-23T23:05:51.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.055+0000 7f5feb470740 1 -- 172.21.15.12:0/263220671 wait complete. 2024-03-23T23:05:51.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.055+0000 7f5feb470740 20 client.26350 trim_cache size 0 max 16384 2024-03-23T23:05:51.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.072+0000 7f5feb470740 1 Processor -- start 2024-03-23T23:05:51.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.073+0000 7f5feb470740 1 -- start start 2024-03-23T23:05:51.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.073+0000 7f5feb470740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884ec610 0x55cd884efee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.073+0000 7f5feb470740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd884f98b0 0x55cd8850b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.073+0000 7f5feb470740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd884fa2a0 0x55cd88516730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.073+0000 7f5feb470740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe01a8ca0 con 0x55cd884fa2a0 2024-03-23T23:05:51.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.073+0000 7f5feb470740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe01a67a0 con 0x55cd884f98b0 2024-03-23T23:05:51.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.073+0000 7f5feb470740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe01a4840 con 0x55cd884ec610 2024-03-23T23:05:51.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.073+0000 7f5fe5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884ec610 0x55cd884efee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.073+0000 7f5fe5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884ec610 0x55cd884efee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45924/0 (socket says 172.21.15.12:45924) 2024-03-23T23:05:51.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.073+0000 7f5fe5e00640 1 -- 172.21.15.12:0/2975802257 learned_addr learned my addr 172.21.15.12:0/2975802257 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:51.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.073+0000 7f5fe5400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd884f98b0 0x55cd8850b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.073+0000 7f5fe6800640 1 --2- 172.21.15.12:0/2975802257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd884fa2a0 0x55cd88516730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.073+0000 7f5fe5e00640 1 -- 172.21.15.12:0/2975802257 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd884f98b0 msgr2=0x55cd8850b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:51.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.073+0000 7f5fe5e00640 1 --2- 172.21.15.12:0/2975802257 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd884f98b0 0x55cd8850b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.073+0000 7f5fe5e00640 1 -- 172.21.15.12:0/2975802257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd884fa2a0 msgr2=0x55cd88516730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:51.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.073+0000 7f5fe5e00640 1 --2- 172.21.15.12:0/2975802257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd884fa2a0 0x55cd88516730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.073+0000 7f5fe5e00640 1 -- 172.21.15.12:0/2975802257 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd884e7c20 con 0x55cd884ec610 2024-03-23T23:05:51.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.074+0000 7f5fe5e00640 1 --2- 172.21.15.12:0/2975802257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884ec610 0x55cd884efee0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5fd809bbb0 tx=0x7f5fd80d7990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10fc0f80e4cbd15d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:51.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.074+0000 7f5fca600640 1 -- 172.21.15.12:0/2975802257 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd809b9f0 con 0x55cd884ec610 2024-03-23T23:05:51.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.074+0000 7f5fca600640 1 -- 172.21.15.12:0/2975802257 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fd809c320 con 0x55cd884ec610 2024-03-23T23:05:51.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.074+0000 7f5fca600640 1 -- 172.21.15.12:0/2975802257 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd8099d60 con 0x55cd884ec610 2024-03-23T23:05:51.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.075+0000 7f5feb470740 1 -- 172.21.15.12:0/2975802257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884ec610 msgr2=0x55cd884efee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:51.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.075+0000 7f5feb470740 1 --2- 172.21.15.12:0/2975802257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884ec610 0x55cd884efee0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5fd809bbb0 tx=0x7f5fd80d7990 comp rx=0 tx=0).stop 2024-03-23T23:05:51.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.075+0000 7f5feb470740 1 -- 172.21.15.12:0/2975802257 shutdown_connections 2024-03-23T23:05:51.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.075+0000 7f5feb470740 1 --2- 172.21.15.12:0/2975802257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd884fa2a0 0x55cd88516730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.075+0000 7f5feb470740 1 --2- 172.21.15.12:0/2975802257 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd884f98b0 0x55cd8850b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.075+0000 7f5feb470740 1 --2- 172.21.15.12:0/2975802257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884ec610 0x55cd884efee0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.075+0000 7f5feb470740 1 -- 172.21.15.12:0/2975802257 >> 172.21.15.12:0/2975802257 conn(0x55cd88503680 msgr2=0x55cd884eb830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:51.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.075+0000 7f5feb470740 1 -- 172.21.15.12:0/2975802257 shutdown_connections 2024-03-23T23:05:51.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.075+0000 7f5feb470740 1 -- 172.21.15.12:0/2975802257 wait complete. 2024-03-23T23:05:51.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.075+0000 7f5feb470740 1 -- start start 2024-03-23T23:05:51.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.075+0000 7f5feb470740 1 Processor -- start 2024-03-23T23:05:51.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.076+0000 7f5feb470740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884ec610 0x55cd88509cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.076+0000 7f5feb470740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd884f98b0 0x55cd8850a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.076+0000 7f5feb470740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd884fa2a0 0x55cd8856b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.076+0000 7f5feb470740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe01a4e40 con 0x55cd884fa2a0 2024-03-23T23:05:51.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.076+0000 7f5feb470740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe01a6140 con 0x55cd884f98b0 2024-03-23T23:05:51.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.076+0000 7f5feb470740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe01a5480 con 0x55cd884ec610 2024-03-23T23:05:51.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.076+0000 7f5feb470740 10 client.0 _ll_register_callbacks cb 0x7f5fd80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:51.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.076+0000 7f5fe5400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd884f98b0 0x55cd8850a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.076+0000 7f5fe5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884ec610 0x55cd88509cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.076+0000 7f5fe5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884ec610 0x55cd88509cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45932/0 (socket says 172.21.15.12:45932) 2024-03-23T23:05:51.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.076+0000 7f5fe5e00640 1 -- 172.21.15.12:0/1359655328 learned_addr learned my addr 172.21.15.12:0/1359655328 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:51.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.076+0000 7f5fe5400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd884f98b0 0x55cd8850a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50282/0 (socket says 172.21.15.12:50282) 2024-03-23T23:05:51.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.076+0000 7f5fe6800640 1 --2- 172.21.15.12:0/1359655328 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd884fa2a0 0x55cd8856b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.076+0000 7f5fe5e00640 1 -- 172.21.15.12:0/1359655328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd884f98b0 msgr2=0x55cd8850a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:51.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.076+0000 7f5fe5e00640 1 --2- 172.21.15.12:0/1359655328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd884f98b0 0x55cd8850a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.076+0000 7f5fe5e00640 1 -- 172.21.15.12:0/1359655328 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd884fa2a0 msgr2=0x55cd8856b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:51.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.076+0000 7f5fe5e00640 1 --2- 172.21.15.12:0/1359655328 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd884fa2a0 0x55cd8856b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.076+0000 7f5fe5e00640 1 -- 172.21.15.12:0/1359655328 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cd8856b840 con 0x55cd884ec610 2024-03-23T23:05:51.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.076+0000 7f5fe5400640 1 --2- 172.21.15.12:0/1359655328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd884f98b0 0x55cd8850a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:51.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.077+0000 7f5fe5e00640 1 --2- 172.21.15.12:0/1359655328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884ec610 0x55cd88509cd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5fd809b920 tx=0x7f5fd80cf600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5dff4298cb087d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:51.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.077+0000 7f5fbf400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:05:51.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.077+0000 7f5fbf400640 1 -- 172.21.15.12:0/1359655328 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd80b3070 con 0x55cd884ec610 2024-03-23T23:05:51.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.077+0000 7f5fbf400640 1 -- 172.21.15.12:0/1359655328 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5fd8099f20 con 0x55cd884ec610 2024-03-23T23:05:51.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.077+0000 7f5feb470740 1 -- 172.21.15.12:0/1359655328 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5fd809c320 con 0x55cd884ec610 2024-03-23T23:05:51.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.077+0000 7f5fbf400640 1 -- 172.21.15.12:0/1359655328 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fd809b580 con 0x55cd884ec610 2024-03-23T23:05:51.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.077+0000 7f5feb470740 10 client.26356 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:51.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.077+0000 7f5feb470740 1 -- 172.21.15.12:0/1359655328 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55cd8856ca40 con 0x55cd884ec610 2024-03-23T23:05:51.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.077+0000 7f5fbf400640 1 -- 172.21.15.12:0/1359655328 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fd809c320 con 0x55cd884ec610 2024-03-23T23:05:51.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.077+0000 7f5fbf400640 1 -- 172.21.15.12:0/1359655328 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f5fd80b4ce0 con 0x55cd884ec610 2024-03-23T23:05:51.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.078+0000 7f5feb470740 10 client.26356 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:51.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.078+0000 7f5feb470740 10 client.26356 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:51.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.078+0000 7f5feb470740 20 client.26356 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:51.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.078+0000 7f5feb470740 1 --2- 172.21.15.12:0/1359655328 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cd884ff410 0x55cd8856ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.078+0000 7f5feb470740 4 client.26356 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-23T23:05:51.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.078+0000 7f5feb470740 1 -- 172.21.15.12:0/1359655328 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55cd8845e740 con 0x55cd884ff410 2024-03-23T23:05:51.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.078+0000 7f5fe5400640 1 --2- 172.21.15.12:0/1359655328 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cd884ff410 0x55cd8856ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.078+0000 7f5fe5400640 1 --2- 172.21.15.12:0/1359655328 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cd884ff410 0x55cd8856ef40 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef1760b713436148 server_cookie=48eb561b01180633 in_seq=0 out_seq=0 2024-03-23T23:05:51.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.078+0000 7f5fbf400640 10 client.26356 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:51.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.080+0000 7f5fbf400640 1 -- 172.21.15.12:0/1359655328 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55cd8845e740 con 0x55cd884ff410 2024-03-23T23:05:51.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.080+0000 7f5fbf400640 10 client.26356 handle_command_reply: tid=0 2024-03-23T23:05:51.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.080+0000 7f5feb470740 1 client.26356 shutdown 2024-03-23T23:05:51.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.080+0000 7f5feb470740 1 -- 172.21.15.12:0/1359655328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884ec610 msgr2=0x55cd88509cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:51.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.080+0000 7f5feb470740 1 --2- 172.21.15.12:0/1359655328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884ec610 0x55cd88509cd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5fd809b920 tx=0x7f5fd80cf600 comp rx=0 tx=0).stop 2024-03-23T23:05:51.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.080+0000 7f5feb470740 1 -- 172.21.15.12:0/1359655328 shutdown_connections 2024-03-23T23:05:51.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.080+0000 7f5feb470740 1 --2- 172.21.15.12:0/1359655328 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cd884ff410 0x55cd8856ef40 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.080+0000 7f5feb470740 1 --2- 172.21.15.12:0/1359655328 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd884fa2a0 0x55cd8856b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.080+0000 7f5feb470740 1 --2- 172.21.15.12:0/1359655328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd884f98b0 0x55cd8850a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.080+0000 7f5feb470740 1 --2- 172.21.15.12:0/1359655328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd884ec610 0x55cd88509cd0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.080+0000 7f5feb470740 1 -- 172.21.15.12:0/1359655328 >> 172.21.15.12:0/1359655328 conn(0x55cd88503680 msgr2=0x55cd884fac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:51.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.080+0000 7f5fbf400640 0 client.26356 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:51.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.080+0000 7f5feb470740 1 -- 172.21.15.12:0/1359655328 shutdown_connections 2024-03-23T23:05:51.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.080+0000 7f5feb470740 1 -- 172.21.15.12:0/1359655328 wait complete. 2024-03-23T23:05:51.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.080+0000 7f5feb470740 20 client.26356 trim_cache size 0 max 16384 2024-03-23T23:05:51.083 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:05:51.083 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-03-23T23:05:51.083 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "77fb7428-74ab-4e22-8ce3-b849b98ea931", 2024-03-23T23:05:51.083 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-03-23T23:05:51.083 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:05:51.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.081+0000 7f5fe8800640 1 -- 172.21.15.12:0/3423927129 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5fcc0a2800 msgr2=0x7f5fcc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:51.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.081+0000 7f5fe8800640 1 --2- 172.21.15.12:0/3423927129 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5fcc0a2800 0x7f5fcc0a4cc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x55cd884fab00 tx=0x7f5fdc005f90 comp rx=0 tx=0).stop 2024-03-23T23:05:51.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.081+0000 7f5fe8800640 1 -- 172.21.15.12:0/3423927129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fe01038e0 msgr2=0x7f5fe0199c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:51.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.081+0000 7f5fe8800640 1 --2- 172.21.15.12:0/3423927129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fe01038e0 0x7f5fe0199c40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5fd800eb70 tx=0x7f5fd800ef70 comp rx=0 tx=0).stop 2024-03-23T23:05:51.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.081+0000 7f5fe8800640 1 -- 172.21.15.12:0/3423927129 shutdown_connections 2024-03-23T23:05:51.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.081+0000 7f5fe8800640 1 --2- 172.21.15.12:0/3423927129 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5fcc0a2800 0x7f5fcc0a4cc0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.081+0000 7f5fe8800640 1 --2- 172.21.15.12:0/3423927129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fe0105490 0x7f5fe01a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.081+0000 7f5fe8800640 1 --2- 172.21.15.12:0/3423927129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fe0104ad0 0x7f5fe019a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.081+0000 7f5fe8800640 1 --2- 172.21.15.12:0/3423927129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fe01038e0 0x7f5fe0199c40 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.081+0000 7f5fe8800640 1 -- 172.21.15.12:0/3423927129 >> 172.21.15.12:0/3423927129 conn(0x7f5fe00fee50 msgr2=0x7f5fe0107e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:51.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.081+0000 7f5fe8800640 1 -- 172.21.15.12:0/3423927129 shutdown_connections 2024-03-23T23:05:51.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.081+0000 7f5fe8800640 1 -- 172.21.15.12:0/3423927129 wait complete. 2024-03-23T23:05:51.098 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:05:51.098 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:05:51.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.320+0000 7fe092800640 1 -- 172.21.15.12:0/412923596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe08c109770 msgr2=0x7fe08c110000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:51.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.320+0000 7fe092800640 1 --2- 172.21.15.12:0/412923596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe08c109770 0x7fe08c110000 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe084009ca0 tx=0x7fe08402c100 comp rx=0 tx=0).stop 2024-03-23T23:05:51.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.320+0000 7fe092800640 1 -- 172.21.15.12:0/412923596 shutdown_connections 2024-03-23T23:05:51.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.320+0000 7fe092800640 1 --2- 172.21.15.12:0/412923596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe08c109770 0x7fe08c110000 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.320+0000 7fe092800640 1 --2- 172.21.15.12:0/412923596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe08c108db0 0x7fe08c109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.321+0000 7fe092800640 1 --2- 172.21.15.12:0/412923596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe08c107bc0 0x7fe08c107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.321+0000 7fe092800640 1 -- 172.21.15.12:0/412923596 >> 172.21.15.12:0/412923596 conn(0x7fe08c076b70 msgr2=0x7fe08c078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:51.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.321+0000 7fe092800640 1 -- 172.21.15.12:0/412923596 shutdown_connections 2024-03-23T23:05:51.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.321+0000 7fe092800640 1 -- 172.21.15.12:0/412923596 wait complete. 2024-03-23T23:05:51.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.322+0000 7fe092800640 1 Processor -- start 2024-03-23T23:05:51.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.322+0000 7fe092800640 1 -- start start 2024-03-23T23:05:51.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.322+0000 7fe092800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe08c107bc0 0x7fe08c19de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.322+0000 7fe092800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe08c108db0 0x7fe08c19e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.322+0000 7fe092800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe08c109770 0x7fe08c1a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.322+0000 7fe092800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c113840 con 0x7fe08c108db0 2024-03-23T23:05:51.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.322+0000 7fe092800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c1136c0 con 0x7fe08c107bc0 2024-03-23T23:05:51.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.322+0000 7fe092800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c1139c0 con 0x7fe08c109770 2024-03-23T23:05:51.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.322+0000 7fe091e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe08c109770 0x7fe08c1a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.323+0000 7fe091e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe08c109770 0x7fe08c1a53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45952/0 (socket says 172.21.15.12:45952) 2024-03-23T23:05:51.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.323+0000 7fe091e00640 1 -- 172.21.15.12:0/2762310556 learned_addr learned my addr 172.21.15.12:0/2762310556 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:51.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.323+0000 7fe091400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe08c107bc0 0x7fe08c19de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.323+0000 7fe090a00640 1 --2- 172.21.15.12:0/2762310556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe08c108db0 0x7fe08c19e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.323+0000 7fe091e00640 1 -- 172.21.15.12:0/2762310556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe08c107bc0 msgr2=0x7fe08c19de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:51.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.323+0000 7fe091e00640 1 --2- 172.21.15.12:0/2762310556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe08c107bc0 0x7fe08c19de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.323+0000 7fe091e00640 1 -- 172.21.15.12:0/2762310556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe08c108db0 msgr2=0x7fe08c19e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:51.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.323+0000 7fe091e00640 1 --2- 172.21.15.12:0/2762310556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe08c108db0 0x7fe08c19e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.323+0000 7fe091e00640 1 -- 172.21.15.12:0/2762310556 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe08c1a5a40 con 0x7fe08c109770 2024-03-23T23:05:51.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.323+0000 7fe090a00640 1 --2- 172.21.15.12:0/2762310556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe08c108db0 0x7fe08c19e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:05:51.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.323+0000 7fe091e00640 1 --2- 172.21.15.12:0/2762310556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe08c109770 0x7fe08c1a53d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe08402c640 tx=0x7fe084035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:51.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.323+0000 7fe08a000640 1 -- 172.21.15.12:0/2762310556 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe084039040 con 0x7fe08c109770 2024-03-23T23:05:51.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.323+0000 7fe08a000640 1 -- 172.21.15.12:0/2762310556 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe084044070 con 0x7fe08c109770 2024-03-23T23:05:51.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.323+0000 7fe08a000640 1 -- 172.21.15.12:0/2762310556 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe084004460 con 0x7fe08c109770 2024-03-23T23:05:51.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.324+0000 7fe092800640 1 -- 172.21.15.12:0/2762310556 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe08c1a5cd0 con 0x7fe08c109770 2024-03-23T23:05:51.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.324+0000 7fe092800640 1 -- 172.21.15.12:0/2762310556 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe08c1a6170 con 0x7fe08c109770 2024-03-23T23:05:51.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.327+0000 7fe08a000640 1 -- 172.21.15.12:0/2762310556 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe084004600 con 0x7fe08c109770 2024-03-23T23:05:51.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.327+0000 7fe096bbd740 1 Processor -- start 2024-03-23T23:05:51.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.327+0000 7fe096bbd740 1 -- start start 2024-03-23T23:05:51.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.327+0000 7fe096bbd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563607730050 0x563607730470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.327+0000 7fe096bbd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730a40 0x563607735540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.327+0000 7fe096bbd740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563607735bb0 0x563607738040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.327+0000 7fe096bbd740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563607695250 con 0x563607735bb0 2024-03-23T23:05:51.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.327+0000 7fe096bbd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56360763da30 con 0x563607730a40 2024-03-23T23:05:51.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.327+0000 7fe096bbd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636076912f0 con 0x563607730050 2024-03-23T23:05:51.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.328+0000 7fe090a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563607730050 0x563607730470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.328+0000 7fe091400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730a40 0x563607735540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.328+0000 7fe091400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730a40 0x563607735540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50308/0 (socket says 172.21.15.12:50308) 2024-03-23T23:05:51.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.328+0000 7fe091400640 1 -- 172.21.15.12:0/1915860169 learned_addr learned my addr 172.21.15.12:0/1915860169 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:51.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.328+0000 7fe090a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563607730050 0x563607730470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45966/0 (socket says 172.21.15.12:45966) 2024-03-23T23:05:51.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.328+0000 7fe090a00640 1 --2- 172.21.15.12:0/1915860169 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563607735bb0 0x563607738040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.328+0000 7fe091400640 1 -- 172.21.15.12:0/1915860169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563607730050 msgr2=0x563607730470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:51.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.328+0000 7fe091400640 1 --2- 172.21.15.12:0/1915860169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563607730050 0x563607730470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.328+0000 7fe091400640 1 -- 172.21.15.12:0/1915860169 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563607735bb0 msgr2=0x563607738040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:51.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.328+0000 7fe091400640 1 --2- 172.21.15.12:0/1915860169 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563607735bb0 0x563607738040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.328+0000 7fe091400640 1 -- 172.21.15.12:0/1915860169 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563607628690 con 0x563607730a40 2024-03-23T23:05:51.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.328+0000 7fe090a00640 1 --2- 172.21.15.12:0/1915860169 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563607735bb0 0x563607738040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:05:51.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.329+0000 7fe091400640 1 --2- 172.21.15.12:0/1915860169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730a40 0x563607735540 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe08000b7d0 tx=0x7fe080004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4650410ce47d81d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:51.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.329+0000 7fe08a000640 1 --2- 172.21.15.12:0/2762310556 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe0740a2880 0x7fe0740a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.329+0000 7fe07e600640 1 -- 172.21.15.12:0/1915860169 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe080003dd0 con 0x563607730a40 2024-03-23T23:05:51.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.329+0000 7fe07e600640 1 -- 172.21.15.12:0/1915860169 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe080015070 con 0x563607730a40 2024-03-23T23:05:51.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.329+0000 7fe07e600640 1 -- 172.21.15.12:0/1915860169 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0800117c0 con 0x563607730a40 2024-03-23T23:05:51.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.329+0000 7fe08a000640 1 -- 172.21.15.12:0/2762310556 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe0840f10f0 con 0x7fe08c109770 2024-03-23T23:05:51.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.329+0000 7fe090a00640 1 --2- 172.21.15.12:0/2762310556 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe0740a2880 0x7fe0740a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.330+0000 7fe096bbd740 1 -- 172.21.15.12:0/1915860169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730a40 msgr2=0x563607735540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:51.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.330+0000 7fe096bbd740 1 --2- 172.21.15.12:0/1915860169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730a40 0x563607735540 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe08000b7d0 tx=0x7fe080004610 comp rx=0 tx=0).stop 2024-03-23T23:05:51.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.330+0000 7fe096bbd740 1 -- 172.21.15.12:0/1915860169 shutdown_connections 2024-03-23T23:05:51.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.330+0000 7fe096bbd740 1 --2- 172.21.15.12:0/1915860169 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563607735bb0 0x563607738040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.330+0000 7fe096bbd740 1 --2- 172.21.15.12:0/1915860169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730a40 0x563607735540 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.330+0000 7fe096bbd740 1 --2- 172.21.15.12:0/1915860169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563607730050 0x563607730470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.330+0000 7fe096bbd740 1 -- 172.21.15.12:0/1915860169 >> 172.21.15.12:0/1915860169 conn(0x5636076f8200 msgr2=0x56360772b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:51.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.330+0000 7fe090a00640 1 --2- 172.21.15.12:0/2762310556 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe0740a2880 0x7fe0740a4d40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x5636077312a0 tx=0x7fe078005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:51.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.330+0000 7fe096bbd740 1 -- 172.21.15.12:0/1915860169 shutdown_connections 2024-03-23T23:05:51.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.330+0000 7fe096bbd740 1 -- 172.21.15.12:0/1915860169 wait complete. 2024-03-23T23:05:51.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.330+0000 7fe096bbd740 1 -- start start 2024-03-23T23:05:51.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.331+0000 7fe096bbd740 1 Processor -- start 2024-03-23T23:05:51.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.331+0000 7fe096bbd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730050 0x56360773a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.331+0000 7fe096bbd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563607730a40 0x56360773a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.331+0000 7fe096bbd740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563607735bb0 0x5636077415b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.331+0000 7fe096bbd740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563607738b60 con 0x563607735bb0 2024-03-23T23:05:51.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.331+0000 7fe096bbd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x563607650e80 con 0x563607730050 2024-03-23T23:05:51.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.331+0000 7fe096bbd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x563607738ce0 con 0x563607730a40 2024-03-23T23:05:51.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.331+0000 7fe096bbd740 10 client.0 _ll_register_callbacks cb 0x563607663130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:51.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.331+0000 7fe091400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730050 0x56360773a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.331+0000 7fe090a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563607730a40 0x56360773a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.331+0000 7fe091400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730050 0x56360773a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50324/0 (socket says 172.21.15.12:50324) 2024-03-23T23:05:51.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.331+0000 7fe091400640 1 -- 172.21.15.12:0/2171234490 learned_addr learned my addr 172.21.15.12:0/2171234490 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:51.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.331+0000 7fe091e00640 1 --2- 172.21.15.12:0/2171234490 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563607735bb0 0x5636077415b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.332+0000 7fe091400640 1 -- 172.21.15.12:0/2171234490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563607730a40 msgr2=0x56360773a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:51.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.332+0000 7fe091400640 1 --2- 172.21.15.12:0/2171234490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563607730a40 0x56360773a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.332+0000 7fe091400640 1 -- 172.21.15.12:0/2171234490 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563607735bb0 msgr2=0x5636077415b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:51.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.332+0000 7fe091400640 1 --2- 172.21.15.12:0/2171234490 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563607735bb0 0x5636077415b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.332+0000 7fe091400640 1 -- 172.21.15.12:0/2171234490 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5636074737d0 con 0x563607730050 2024-03-23T23:05:51.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.332+0000 7fe091400640 1 --2- 172.21.15.12:0/2171234490 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730050 0x56360773a030 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fe080005c90 tx=0x7fe080003e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3aa8101ef50ba2f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:51.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.332+0000 7fe072a00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:05:51.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.332+0000 7fe072a00640 1 -- 172.21.15.12:0/2171234490 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe080027020 con 0x563607730050 2024-03-23T23:05:51.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.333+0000 7fe072a00640 1 -- 172.21.15.12:0/2171234490 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe080005450 con 0x563607730050 2024-03-23T23:05:51.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.333+0000 7fe096bbd740 1 -- 172.21.15.12:0/2171234490 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe080015070 con 0x563607730050 2024-03-23T23:05:51.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.333+0000 7fe072a00640 1 -- 172.21.15.12:0/2171234490 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe080005650 con 0x563607730050 2024-03-23T23:05:51.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.333+0000 7fe096bbd740 10 client.26405 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:51.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.333+0000 7fe096bbd740 1 -- 172.21.15.12:0/2171234490 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5636073a93f0 con 0x563607730050 2024-03-23T23:05:51.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.333+0000 7fe072a00640 1 -- 172.21.15.12:0/2171234490 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe080015070 con 0x563607730050 2024-03-23T23:05:51.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.333+0000 7fe072a00640 1 -- 172.21.15.12:0/2171234490 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe080004a50 con 0x563607730050 2024-03-23T23:05:51.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.333+0000 7fe096bbd740 10 client.26405 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:51.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.333+0000 7fe096bbd740 10 client.26405 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:51.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.333+0000 7fe096bbd740 20 client.26405 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:51.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.333+0000 7fe096bbd740 1 --2- 172.21.15.12:0/2171234490 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563607745aa0 0x563607747e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.333+0000 7fe096bbd740 4 client.26405 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:05:51.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.333+0000 7fe096bbd740 1 -- 172.21.15.12:0/2171234490 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5636076a2790 con 0x563607745aa0 2024-03-23T23:05:51.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.334+0000 7fe090a00640 1 --2- 172.21.15.12:0/2171234490 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563607745aa0 0x563607747e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.334+0000 7fe090a00640 1 --2- 172.21.15.12:0/2171234490 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563607745aa0 0x563607747e90 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ef75886a5cd1dda server_cookie=532b9e155c820ea0 in_seq=0 out_seq=0 2024-03-23T23:05:51.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.334+0000 7fe072a00640 10 client.26405 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:51.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.335+0000 7fe072a00640 1 -- 172.21.15.12:0/2171234490 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5636076a2790 con 0x563607745aa0 2024-03-23T23:05:51.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.335+0000 7fe072a00640 10 client.26405 handle_command_reply: tid=0 2024-03-23T23:05:51.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.335+0000 7fe096bbd740 1 client.26405 shutdown 2024-03-23T23:05:51.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.335+0000 7fe096bbd740 1 -- 172.21.15.12:0/2171234490 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730050 msgr2=0x56360773a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:51.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.335+0000 7fe096bbd740 1 --2- 172.21.15.12:0/2171234490 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730050 0x56360773a030 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fe080005c90 tx=0x7fe080003e00 comp rx=0 tx=0).stop 2024-03-23T23:05:51.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.335+0000 7fe096bbd740 1 -- 172.21.15.12:0/2171234490 shutdown_connections 2024-03-23T23:05:51.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.335+0000 7fe096bbd740 1 --2- 172.21.15.12:0/2171234490 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563607745aa0 0x563607747e90 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.335+0000 7fe096bbd740 1 --2- 172.21.15.12:0/2171234490 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563607735bb0 0x5636077415b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.335+0000 7fe096bbd740 1 --2- 172.21.15.12:0/2171234490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563607730a40 0x56360773a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.335+0000 7fe072a00640 0 client.26405 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:51.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.335+0000 7fe096bbd740 1 --2- 172.21.15.12:0/2171234490 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730050 0x56360773a030 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.335+0000 7fe096bbd740 1 -- 172.21.15.12:0/2171234490 >> 172.21.15.12:0/2171234490 conn(0x5636076f8200 msgr2=0x56360772abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:51.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.336+0000 7fe096bbd740 1 -- 172.21.15.12:0/2171234490 shutdown_connections 2024-03-23T23:05:51.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.336+0000 7fe096bbd740 1 -- 172.21.15.12:0/2171234490 wait complete. 2024-03-23T23:05:51.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.336+0000 7fe096bbd740 20 client.26405 trim_cache size 0 max 16384 2024-03-23T23:05:51.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.353+0000 7fe096bbd740 1 Processor -- start 2024-03-23T23:05:51.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.354+0000 7fe096bbd740 1 -- start start 2024-03-23T23:05:51.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.354+0000 7fe096bbd740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563607730050 0x5636077314b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.354+0000 7fe096bbd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730a40 0x5636077414d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.354+0000 7fe096bbd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563607735bb0 0x56360774c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.354+0000 7fe096bbd740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c1b1360 con 0x563607730050 2024-03-23T23:05:51.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.354+0000 7fe096bbd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c1aee60 con 0x563607730a40 2024-03-23T23:05:51.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.354+0000 7fe096bbd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c1acf00 con 0x563607735bb0 2024-03-23T23:05:51.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.354+0000 7fe091400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563607730050 0x5636077314b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.354+0000 7fe091400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563607730050 0x5636077314b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52844/0 (socket says 172.21.15.12:52844) 2024-03-23T23:05:51.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.354+0000 7fe091400640 1 -- 172.21.15.12:0/141218764 learned_addr learned my addr 172.21.15.12:0/141218764 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:51.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.354+0000 7fe090a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730a40 0x5636077414d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.354+0000 7fe091e00640 1 --2- 172.21.15.12:0/141218764 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563607735bb0 0x56360774c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.355+0000 7fe091400640 1 -- 172.21.15.12:0/141218764 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563607735bb0 msgr2=0x56360774c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:51.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.355+0000 7fe091400640 1 --2- 172.21.15.12:0/141218764 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563607735bb0 0x56360774c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.355+0000 7fe091400640 1 -- 172.21.15.12:0/141218764 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730a40 msgr2=0x5636077414d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:51.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.355+0000 7fe091400640 1 --2- 172.21.15.12:0/141218764 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730a40 0x5636077414d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.355+0000 7fe091400640 1 -- 172.21.15.12:0/141218764 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636073690f0 con 0x563607730050 2024-03-23T23:05:51.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.355+0000 7fe091e00640 1 --2- 172.21.15.12:0/141218764 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563607735bb0 0x56360774c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:51.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.355+0000 7fe090a00640 1 --2- 172.21.15.12:0/141218764 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730a40 0x5636077414d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:51.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.355+0000 7fe091400640 1 --2- 172.21.15.12:0/141218764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563607730050 0x5636077314b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe080004e70 tx=0x7fe08000bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87976574aab047cd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:51.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.355+0000 7fe07dc00640 1 -- 172.21.15.12:0/141218764 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe080027070 con 0x563607730050 2024-03-23T23:05:51.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.355+0000 7fe07dc00640 1 -- 172.21.15.12:0/141218764 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe080015040 con 0x563607730050 2024-03-23T23:05:51.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.355+0000 7fe07dc00640 1 -- 172.21.15.12:0/141218764 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08001a6d0 con 0x563607730050 2024-03-23T23:05:51.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.356+0000 7fe096bbd740 1 -- 172.21.15.12:0/141218764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563607730050 msgr2=0x5636077314b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:51.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.356+0000 7fe096bbd740 1 --2- 172.21.15.12:0/141218764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563607730050 0x5636077314b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe080004e70 tx=0x7fe08000bfb0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.356+0000 7fe096bbd740 1 -- 172.21.15.12:0/141218764 shutdown_connections 2024-03-23T23:05:51.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.356+0000 7fe096bbd740 1 --2- 172.21.15.12:0/141218764 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563607735bb0 0x56360774c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.356+0000 7fe096bbd740 1 --2- 172.21.15.12:0/141218764 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730a40 0x5636077414d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.356+0000 7fe096bbd740 1 --2- 172.21.15.12:0/141218764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563607730050 0x5636077314b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.356+0000 7fe096bbd740 1 -- 172.21.15.12:0/141218764 >> 172.21.15.12:0/141218764 conn(0x5636076f8200 msgr2=0x56360772abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:51.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.356+0000 7fe096bbd740 1 -- 172.21.15.12:0/141218764 shutdown_connections 2024-03-23T23:05:51.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.356+0000 7fe096bbd740 1 -- 172.21.15.12:0/141218764 wait complete. 2024-03-23T23:05:51.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.356+0000 7fe096bbd740 1 -- start start 2024-03-23T23:05:51.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.357+0000 7fe096bbd740 1 Processor -- start 2024-03-23T23:05:51.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.357+0000 7fe096bbd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730050 0x56360773eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.357+0000 7fe096bbd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563607730a40 0x56360773f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.357+0000 7fe096bbd740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563607735bb0 0x563607740060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.357+0000 7fe096bbd740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c1ad500 con 0x563607735bb0 2024-03-23T23:05:51.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.357+0000 7fe096bbd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c1ae800 con 0x563607730050 2024-03-23T23:05:51.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.357+0000 7fe096bbd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c1adb40 con 0x563607730a40 2024-03-23T23:05:51.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.357+0000 7fe096bbd740 10 client.0 _ll_register_callbacks cb 0x7fe0780099b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:51.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.357+0000 7fe091400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730050 0x56360773eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.357+0000 7fe091400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730050 0x56360773eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:50348/0 (socket says 172.21.15.12:50348) 2024-03-23T23:05:51.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.357+0000 7fe091400640 1 -- 172.21.15.12:0/3971417047 learned_addr learned my addr 172.21.15.12:0/3971417047 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:51.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.357+0000 7fe091e00640 1 --2- 172.21.15.12:0/3971417047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563607735bb0 0x563607740060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.357+0000 7fe090a00640 1 --2- 172.21.15.12:0/3971417047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563607730a40 0x56360773f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.358+0000 7fe091400640 1 -- 172.21.15.12:0/3971417047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563607730a40 msgr2=0x56360773f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:51.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.358+0000 7fe091400640 1 --2- 172.21.15.12:0/3971417047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563607730a40 0x56360773f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.358+0000 7fe091400640 1 -- 172.21.15.12:0/3971417047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563607735bb0 msgr2=0x563607740060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:51.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.358+0000 7fe091400640 1 --2- 172.21.15.12:0/3971417047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563607735bb0 0x563607740060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.358+0000 7fe091400640 1 -- 172.21.15.12:0/3971417047 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563607740650 con 0x563607730050 2024-03-23T23:05:51.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.358+0000 7fe091e00640 1 --2- 172.21.15.12:0/3971417047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563607735bb0 0x563607740060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:05:51.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.358+0000 7fe090a00640 1 --2- 172.21.15.12:0/3971417047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563607730a40 0x56360773f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:05:51.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.358+0000 7fe091400640 1 --2- 172.21.15.12:0/3971417047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730050 0x56360773eb60 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe080000c00 tx=0x7fe080005bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f92fd6c582bef754 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:51.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.358+0000 7fe072a00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:05:51.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.358+0000 7fe072a00640 1 -- 172.21.15.12:0/3971417047 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe080027070 con 0x563607730050 2024-03-23T23:05:51.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.358+0000 7fe072a00640 1 -- 172.21.15.12:0/3971417047 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe080011d50 con 0x563607730050 2024-03-23T23:05:51.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.358+0000 7fe096bbd740 1 -- 172.21.15.12:0/3971417047 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe080015040 con 0x563607730050 2024-03-23T23:05:51.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.359+0000 7fe072a00640 1 -- 172.21.15.12:0/3971417047 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe080047050 con 0x563607730050 2024-03-23T23:05:51.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.359+0000 7fe096bbd740 10 client.26411 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:51.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.359+0000 7fe096bbd740 1 -- 172.21.15.12:0/3971417047 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563607741960 con 0x563607730050 2024-03-23T23:05:51.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.359+0000 7fe072a00640 1 -- 172.21.15.12:0/3971417047 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe080015040 con 0x563607730050 2024-03-23T23:05:51.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.359+0000 7fe072a00640 1 -- 172.21.15.12:0/3971417047 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe08003f020 con 0x563607730050 2024-03-23T23:05:51.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.359+0000 7fe096bbd740 10 client.26411 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:51.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.359+0000 7fe096bbd740 10 client.26411 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:51.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.359+0000 7fe096bbd740 20 client.26411 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:51.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.359+0000 7fe096bbd740 1 --2- 172.21.15.12:0/3971417047 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5636077a1900 0x5636077a3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:51.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.359+0000 7fe096bbd740 4 client.26411 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:05:51.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.359+0000 7fe096bbd740 1 -- 172.21.15.12:0/3971417047 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563607732360 con 0x5636077a1900 2024-03-23T23:05:51.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.359+0000 7fe090a00640 1 --2- 172.21.15.12:0/3971417047 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5636077a1900 0x5636077a3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:51.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.360+0000 7fe090a00640 1 --2- 172.21.15.12:0/3971417047 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5636077a1900 0x5636077a3d20 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2162c1272f94ac6 server_cookie=5fd55b2661527587 in_seq=0 out_seq=0 2024-03-23T23:05:51.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.360+0000 7fe072a00640 10 client.26411 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:51.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.360+0000 7fe072a00640 1 -- 172.21.15.12:0/3971417047 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x563607732360 con 0x5636077a1900 2024-03-23T23:05:51.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.360+0000 7fe072a00640 10 client.26411 handle_command_reply: tid=0 2024-03-23T23:05:51.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.360+0000 7fe096bbd740 1 client.26411 shutdown 2024-03-23T23:05:51.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.360+0000 7fe096bbd740 1 -- 172.21.15.12:0/3971417047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730050 msgr2=0x56360773eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:51.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.360+0000 7fe096bbd740 1 --2- 172.21.15.12:0/3971417047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730050 0x56360773eb60 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe080000c00 tx=0x7fe080005bf0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.360+0000 7fe096bbd740 1 -- 172.21.15.12:0/3971417047 shutdown_connections 2024-03-23T23:05:51.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.360+0000 7fe096bbd740 1 --2- 172.21.15.12:0/3971417047 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5636077a1900 0x5636077a3d20 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.360+0000 7fe096bbd740 1 --2- 172.21.15.12:0/3971417047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563607735bb0 0x563607740060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.360+0000 7fe096bbd740 1 --2- 172.21.15.12:0/3971417047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563607730a40 0x56360773f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.360+0000 7fe072a00640 0 client.26411 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:51.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.360+0000 7fe096bbd740 1 --2- 172.21.15.12:0/3971417047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563607730050 0x56360773eb60 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.360+0000 7fe096bbd740 1 -- 172.21.15.12:0/3971417047 >> 172.21.15.12:0/3971417047 conn(0x5636076f8200 msgr2=0x56360772abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:51.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.360+0000 7fe096bbd740 1 -- 172.21.15.12:0/3971417047 shutdown_connections 2024-03-23T23:05:51.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.360+0000 7fe096bbd740 1 -- 172.21.15.12:0/3971417047 wait complete. 2024-03-23T23:05:51.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.361+0000 7fe096bbd740 20 client.26411 trim_cache size 0 max 16384 2024-03-23T23:05:51.365 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:05:51.365 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T23:05:51.365 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-03-23T23:05:51.365 INFO:teuthology.orchestra.run.smithi012.stdout: "77fb7428-74ab-4e22-8ce3-b849b98ea931": { 2024-03-23T23:05:51.365 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-03-23T23:05:51.365 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "77fb7428-74ab-4e22-8ce3-b849b98ea931", 2024-03-23T23:05:51.365 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-03-23T23:05:51.365 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:05:51.365 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:05:51.365 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:05:51.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.361+0000 7fe092800640 1 -- 172.21.15.12:0/2762310556 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe0740a2880 msgr2=0x7fe0740a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:51.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.361+0000 7fe092800640 1 --2- 172.21.15.12:0/2762310556 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe0740a2880 0x7fe0740a4d40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x5636077312a0 tx=0x7fe078005f90 comp rx=0 tx=0).stop 2024-03-23T23:05:51.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.361+0000 7fe092800640 1 -- 172.21.15.12:0/2762310556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe08c109770 msgr2=0x7fe08c1a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:51.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.361+0000 7fe092800640 1 --2- 172.21.15.12:0/2762310556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe08c109770 0x7fe08c1a53d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe08402c640 tx=0x7fe084035d10 comp rx=0 tx=0).stop 2024-03-23T23:05:51.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.361+0000 7fe092800640 1 -- 172.21.15.12:0/2762310556 shutdown_connections 2024-03-23T23:05:51.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.361+0000 7fe092800640 1 --2- 172.21.15.12:0/2762310556 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe0740a2880 0x7fe0740a4d40 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.361+0000 7fe092800640 1 --2- 172.21.15.12:0/2762310556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe08c109770 0x7fe08c1a53d0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.361+0000 7fe092800640 1 --2- 172.21.15.12:0/2762310556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe08c108db0 0x7fe08c19e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.361+0000 7fe092800640 1 --2- 172.21.15.12:0/2762310556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe08c107bc0 0x7fe08c19de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:51.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.361+0000 7fe092800640 1 -- 172.21.15.12:0/2762310556 >> 172.21.15.12:0/2762310556 conn(0x7fe08c076b70 msgr2=0x7fe08c10c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:51.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.361+0000 7fe092800640 1 -- 172.21.15.12:0/2762310556 shutdown_connections 2024-03-23T23:05:51.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:05:51.361+0000 7fe092800640 1 -- 172.21.15.12:0/2762310556 wait complete. 2024-03-23T23:05:51.378 INFO:tasks.cephfs.filesystem:scrub status for tag:77fb7428-74ab-4e22-8ce3-b849b98ea931 - {'path': '/', 'tag': '77fb7428-74ab-4e22-8ce3-b849b98ea931', 'options': 'recursive,force'} 2024-03-23T23:05:51.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:51 smithi012 ceph-mon[31695]: pgmap v511: 97 pgs: 97 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:05:51.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:51 smithi173 ceph-mon[36116]: pgmap v511: 97 pgs: 97 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:05:51.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:51 smithi203 ceph-mon[36693]: pgmap v511: 97 pgs: 97 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:05:52.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:52 smithi012 ceph-mon[31695]: scrub queued for path: / 2024-03-23T23:05:52.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:52 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:05:52.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:52 smithi012 ceph-mon[31695]: scrub summary: active paths [/] 2024-03-23T23:05:52.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:52 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:05:52.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:52 smithi173 ceph-mon[36116]: scrub queued for path: / 2024-03-23T23:05:52.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:52 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:05:52.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:52 smithi173 ceph-mon[36116]: scrub summary: active paths [/] 2024-03-23T23:05:52.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:52 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:05:52.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:52 smithi203 ceph-mon[36693]: scrub queued for path: / 2024-03-23T23:05:52.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:52 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:05:52.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:52 smithi203 ceph-mon[36693]: scrub summary: active paths [/] 2024-03-23T23:05:52.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:52 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:05:53.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:53 smithi012 ceph-mon[31695]: pgmap v512: 97 pgs: 97 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:05:53.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:53 smithi173 ceph-mon[36116]: pgmap v512: 97 pgs: 97 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:05:53.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:53 smithi203 ceph-mon[36693]: pgmap v512: 97 pgs: 97 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:05:54.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:54 smithi012 ceph-mon[31695]: pgmap v513: 97 pgs: 97 active+clean; 2.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:05:54.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:54 smithi173 ceph-mon[36116]: pgmap v513: 97 pgs: 97 active+clean; 2.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:05:54.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:54 smithi203 ceph-mon[36693]: pgmap v513: 97 pgs: 97 active+clean; 2.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:05:55.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.964+0000 7fea5fe00640 1 -- 172.21.15.12:0/2246290094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea5810dca0 msgr2=0x7fea58110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:55.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.964+0000 7fea5fe00640 1 --2- 172.21.15.12:0/2246290094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea5810dca0 0x7fea58110090 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fea50009ca0 tx=0x7fea5002c0f0 comp rx=0 tx=0).stop 2024-03-23T23:05:55.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.964+0000 7fea5fe00640 1 -- 172.21.15.12:0/2246290094 shutdown_connections 2024-03-23T23:05:55.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.964+0000 7fea5fe00640 1 --2- 172.21.15.12:0/2246290094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea5810dca0 0x7fea58110090 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:55.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.964+0000 7fea5fe00640 1 --2- 172.21.15.12:0/2246290094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea58074100 0x7fea58074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:55.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.964+0000 7fea5fe00640 1 --2- 172.21.15.12:0/2246290094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea580756e0 0x7fea58073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:55.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.964+0000 7fea5fe00640 1 -- 172.21.15.12:0/2246290094 >> 172.21.15.12:0/2246290094 conn(0x7fea580fcc60 msgr2=0x7fea580ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:55.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.964+0000 7fea5fe00640 1 -- 172.21.15.12:0/2246290094 shutdown_connections 2024-03-23T23:05:55.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.964+0000 7fea5fe00640 1 -- 172.21.15.12:0/2246290094 wait complete. 2024-03-23T23:05:55.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.965+0000 7fea5fe00640 1 Processor -- start 2024-03-23T23:05:55.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.965+0000 7fea5fe00640 1 -- start start 2024-03-23T23:05:55.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.965+0000 7fea5fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea58074100 0x7fea5819dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:55.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.965+0000 7fea5fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea580756e0 0x7fea5819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:55.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.965+0000 7fea5fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea5810dca0 0x7fea581a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:55.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.965+0000 7fea5fe00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea58113920 con 0x7fea58074100 2024-03-23T23:05:55.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.965+0000 7fea5fe00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea581137a0 con 0x7fea5810dca0 2024-03-23T23:05:55.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.965+0000 7fea5fe00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea58113aa0 con 0x7fea580756e0 2024-03-23T23:05:55.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.966+0000 7fea5de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea5810dca0 0x7fea581a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:55.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.966+0000 7fea5de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea5810dca0 0x7fea581a5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:47480/0 (socket says 172.21.15.12:47480) 2024-03-23T23:05:55.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.966+0000 7fea5ca00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea580756e0 0x7fea5819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:55.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.966+0000 7fea5de00640 1 -- 172.21.15.12:0/1474783415 learned_addr learned my addr 172.21.15.12:0/1474783415 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:55.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.966+0000 7fea5d400640 1 --2- 172.21.15.12:0/1474783415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea58074100 0x7fea5819dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:55.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.966+0000 7fea5de00640 1 -- 172.21.15.12:0/1474783415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea580756e0 msgr2=0x7fea5819e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:55.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.966+0000 7fea5de00640 1 --2- 172.21.15.12:0/1474783415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea580756e0 0x7fea5819e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:55.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.966+0000 7fea5de00640 1 -- 172.21.15.12:0/1474783415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea58074100 msgr2=0x7fea5819dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:55.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.966+0000 7fea5de00640 1 --2- 172.21.15.12:0/1474783415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea58074100 0x7fea5819dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:55.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.966+0000 7fea5de00640 1 -- 172.21.15.12:0/1474783415 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea581a5ac0 con 0x7fea5810dca0 2024-03-23T23:05:55.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.966+0000 7fea5d400640 1 --2- 172.21.15.12:0/1474783415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea58074100 0x7fea5819dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:05:55.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.966+0000 7fea5de00640 1 --2- 172.21.15.12:0/1474783415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea5810dca0 0x7fea581a5450 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fea5002c630 tx=0x7fea50032ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:55.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.967+0000 7fea56000640 1 -- 172.21.15.12:0/1474783415 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea50042020 con 0x7fea5810dca0 2024-03-23T23:05:55.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.967+0000 7fea56000640 1 -- 172.21.15.12:0/1474783415 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fea50034640 con 0x7fea5810dca0 2024-03-23T23:05:55.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.967+0000 7fea5fe00640 1 -- 172.21.15.12:0/1474783415 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea581a5d50 con 0x7fea5810dca0 2024-03-23T23:05:55.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.967+0000 7fea56000640 1 -- 172.21.15.12:0/1474783415 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea50039040 con 0x7fea5810dca0 2024-03-23T23:05:55.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.967+0000 7fea5fe00640 1 -- 172.21.15.12:0/1474783415 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea581a61e0 con 0x7fea5810dca0 2024-03-23T23:05:55.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.971+0000 7fea62b91740 1 Processor -- start 2024-03-23T23:05:55.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.972+0000 7fea56000640 1 -- 172.21.15.12:0/1474783415 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fea50045020 con 0x7fea5810dca0 2024-03-23T23:05:55.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.972+0000 7fea62b91740 1 -- start start 2024-03-23T23:05:55.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.972+0000 7fea62b91740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709db1e0 0x55c4709db600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:55.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.972+0000 7fea62b91740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4709dbbd0 0x55c4709e06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:55.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.972+0000 7fea62b91740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c4709e0d40 0x55c4709e31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:55.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.972+0000 7fea62b91740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c47099e3b0 con 0x55c4709e0d40 2024-03-23T23:05:55.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.972+0000 7fea62b91740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c470940830 con 0x55c4709db1e0 2024-03-23T23:05:55.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.972+0000 7fea62b91740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4708e95d0 con 0x55c4709dbbd0 2024-03-23T23:05:55.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.972+0000 7fea5ca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709db1e0 0x55c4709db600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.972+0000 7fea5d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4709dbbd0 0x55c4709e06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.972+0000 7fea5d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4709dbbd0 0x55c4709e06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:38596/0 (socket says 172.21.15.12:38596) 2024-03-23T23:05:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.972+0000 7fea5d400640 1 -- 172.21.15.12:0/2830523989 learned_addr learned my addr 172.21.15.12:0/2830523989 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.972+0000 7fea5ca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709db1e0 0x55c4709db600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:47482/0 (socket says 172.21.15.12:47482) 2024-03-23T23:05:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.973+0000 7fea5ca00640 1 --2- 172.21.15.12:0/2830523989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c4709e0d40 0x55c4709e31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.973+0000 7fea5d400640 1 -- 172.21.15.12:0/2830523989 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709db1e0 msgr2=0x55c4709db600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.973+0000 7fea5d400640 1 --2- 172.21.15.12:0/2830523989 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709db1e0 0x55c4709db600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.973+0000 7fea5d400640 1 -- 172.21.15.12:0/2830523989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c4709e0d40 msgr2=0x55c4709e31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.973+0000 7fea5d400640 1 --2- 172.21.15.12:0/2830523989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c4709e0d40 0x55c4709e31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.973+0000 7fea5d400640 1 -- 172.21.15.12:0/2830523989 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4709012c0 con 0x55c4709dbbd0 2024-03-23T23:05:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.973+0000 7fea5ca00640 1 --2- 172.21.15.12:0/2830523989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c4709e0d40 0x55c4709e31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:05:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.973+0000 7fea5d400640 1 --2- 172.21.15.12:0/2830523989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4709dbbd0 0x55c4709e06d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fea4c009d50 tx=0x7fea4c0117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b157931f13b548a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.973+0000 7fea4a600640 1 -- 172.21.15.12:0/2830523989 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c012650 con 0x55c4709dbbd0 2024-03-23T23:05:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.973+0000 7fea4a600640 1 -- 172.21.15.12:0/2830523989 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fea4c009e40 con 0x55c4709dbbd0 2024-03-23T23:05:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.973+0000 7fea4a600640 1 -- 172.21.15.12:0/2830523989 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c00fc00 con 0x55c4709dbbd0 2024-03-23T23:05:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.973+0000 7fea56000640 1 --2- 172.21.15.12:0/1474783415 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fea400a28a0 0x7fea400a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.974+0000 7fea56000640 1 -- 172.21.15.12:0/1474783415 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fea500f08d0 con 0x7fea5810dca0 2024-03-23T23:05:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.974+0000 7fea5ca00640 1 --2- 172.21.15.12:0/1474783415 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fea400a28a0 0x7fea400a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.974+0000 7fea62b91740 1 -- 172.21.15.12:0/2830523989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4709dbbd0 msgr2=0x55c4709e06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.974+0000 7fea62b91740 1 --2- 172.21.15.12:0/2830523989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4709dbbd0 0x55c4709e06d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fea4c009d50 tx=0x7fea4c0117b0 comp rx=0 tx=0).stop 2024-03-23T23:05:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.974+0000 7fea62b91740 1 -- 172.21.15.12:0/2830523989 shutdown_connections 2024-03-23T23:05:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.974+0000 7fea62b91740 1 --2- 172.21.15.12:0/2830523989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c4709e0d40 0x55c4709e31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.974+0000 7fea62b91740 1 --2- 172.21.15.12:0/2830523989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4709dbbd0 0x55c4709e06d0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.974+0000 7fea62b91740 1 --2- 172.21.15.12:0/2830523989 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709db1e0 0x55c4709db600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.974+0000 7fea62b91740 1 -- 172.21.15.12:0/2830523989 >> 172.21.15.12:0/2830523989 conn(0x55c4709d6750 msgr2=0x55c4709d6b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.974+0000 7fea62b91740 1 -- 172.21.15.12:0/2830523989 shutdown_connections 2024-03-23T23:05:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.974+0000 7fea62b91740 1 -- 172.21.15.12:0/2830523989 wait complete. 2024-03-23T23:05:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.975+0000 7fea5ca00640 1 --2- 172.21.15.12:0/1474783415 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fea400a28a0 0x7fea400a4d60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x55c4709dc430 tx=0x7fea44013040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:55.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.975+0000 7fea62b91740 1 -- start start 2024-03-23T23:05:55.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.975+0000 7fea62b91740 1 Processor -- start 2024-03-23T23:05:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.976+0000 7fea62b91740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4709db1e0 0x55c4709e5360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.976+0000 7fea62b91740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709dbbd0 0x55c4709e58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.976+0000 7fea62b91740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c4709e0d40 0x55c4709ec8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.976+0000 7fea62b91740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4708fca20 con 0x55c4709e0d40 2024-03-23T23:05:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.976+0000 7fea62b91740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c47093c8d0 con 0x55c4709dbbd0 2024-03-23T23:05:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.976+0000 7fea62b91740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4709e3cf0 con 0x55c4709db1e0 2024-03-23T23:05:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.976+0000 7fea62b91740 10 client.0 _ll_register_callbacks cb 0x55c4709391c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.976+0000 7fea5ca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709dbbd0 0x55c4709e58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.976+0000 7fea5ca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709dbbd0 0x55c4709e58a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:47494/0 (socket says 172.21.15.12:47494) 2024-03-23T23:05:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.976+0000 7fea5d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4709db1e0 0x55c4709e5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.976+0000 7fea5ca00640 1 -- 172.21.15.12:0/858732394 learned_addr learned my addr 172.21.15.12:0/858732394 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.976+0000 7fea5de00640 1 --2- 172.21.15.12:0/858732394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c4709e0d40 0x55c4709ec8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.976+0000 7fea5d400640 1 -- 172.21.15.12:0/858732394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709dbbd0 msgr2=0x55c4709e58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.976+0000 7fea5d400640 1 --2- 172.21.15.12:0/858732394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709dbbd0 0x55c4709e58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.976+0000 7fea5d400640 1 -- 172.21.15.12:0/858732394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c4709e0d40 msgr2=0x55c4709ec8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.976+0000 7fea5d400640 1 --2- 172.21.15.12:0/858732394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c4709e0d40 0x55c4709ec8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.976+0000 7fea5d400640 1 -- 172.21.15.12:0/858732394 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c470655210 con 0x55c4709db1e0 2024-03-23T23:05:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.977+0000 7fea5d400640 1 --2- 172.21.15.12:0/858732394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4709db1e0 0x55c4709e5360 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x55c4709d76c0 tx=0x7fea4c0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5eaad48e4827c465 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.977+0000 7fea3ea00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:05:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.977+0000 7fea3ea00640 1 -- 172.21.15.12:0/858732394 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c027020 con 0x55c4709db1e0 2024-03-23T23:05:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.977+0000 7fea3ea00640 1 -- 172.21.15.12:0/858732394 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fea4c00fdd0 con 0x55c4709db1e0 2024-03-23T23:05:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.977+0000 7fea62b91740 1 -- 172.21.15.12:0/858732394 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c47094dd70 con 0x55c4709db1e0 2024-03-23T23:05:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.977+0000 7fea3ea00640 1 -- 172.21.15.12:0/858732394 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea4c0044e0 con 0x55c4709db1e0 2024-03-23T23:05:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.977+0000 7fea62b91740 10 client.26386 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.977+0000 7fea62b91740 1 -- 172.21.15.12:0/858732394 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c470721620 con 0x55c4709db1e0 2024-03-23T23:05:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.978+0000 7fea3ea00640 1 -- 172.21.15.12:0/858732394 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c47094dd70 con 0x55c4709db1e0 2024-03-23T23:05:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.978+0000 7fea3ea00640 1 -- 172.21.15.12:0/858732394 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fea4c004c50 con 0x55c4709db1e0 2024-03-23T23:05:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.978+0000 7fea62b91740 10 client.26386 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.978+0000 7fea62b91740 10 client.26386 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.978+0000 7fea62b91740 20 client.26386 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.978+0000 7fea62b91740 1 --2- 172.21.15.12:0/858732394 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c4709f0e90 0x55c4709f3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.978+0000 7fea62b91740 4 client.26386 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:05:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.978+0000 7fea62b91740 1 -- 172.21.15.12:0/858732394 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c4709480c0 con 0x55c4709f0e90 2024-03-23T23:05:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.978+0000 7fea5ca00640 1 --2- 172.21.15.12:0/858732394 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c4709f0e90 0x55c4709f3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.979+0000 7fea5ca00640 1 --2- 172.21.15.12:0/858732394 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c4709f0e90 0x55c4709f3280 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb1db17b686ae109 server_cookie=777684212de118f8 in_seq=0 out_seq=0 2024-03-23T23:05:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.979+0000 7fea3ea00640 10 client.26386 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.979+0000 7fea3ea00640 1 -- 172.21.15.12:0/858732394 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c4709480c0 con 0x55c4709f0e90 2024-03-23T23:05:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.979+0000 7fea3ea00640 10 client.26386 handle_command_reply: tid=0 2024-03-23T23:05:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.980+0000 7fea62b91740 1 client.26386 shutdown 2024-03-23T23:05:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.980+0000 7fea62b91740 1 -- 172.21.15.12:0/858732394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4709db1e0 msgr2=0x55c4709e5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.980+0000 7fea62b91740 1 --2- 172.21.15.12:0/858732394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4709db1e0 0x55c4709e5360 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x55c4709d76c0 tx=0x7fea4c0042f0 comp rx=0 tx=0).stop 2024-03-23T23:05:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.980+0000 7fea62b91740 1 -- 172.21.15.12:0/858732394 shutdown_connections 2024-03-23T23:05:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.980+0000 7fea62b91740 1 --2- 172.21.15.12:0/858732394 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c4709f0e90 0x55c4709f3280 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.980+0000 7fea62b91740 1 --2- 172.21.15.12:0/858732394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c4709e0d40 0x55c4709ec8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.980+0000 7fea62b91740 1 --2- 172.21.15.12:0/858732394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709dbbd0 0x55c4709e58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.980+0000 7fea62b91740 1 --2- 172.21.15.12:0/858732394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4709db1e0 0x55c4709e5360 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.980+0000 7fea3ea00640 0 client.26386 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.980+0000 7fea62b91740 1 -- 172.21.15.12:0/858732394 >> 172.21.15.12:0/858732394 conn(0x55c4709d6750 msgr2=0x55c4709d5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.980+0000 7fea62b91740 1 -- 172.21.15.12:0/858732394 shutdown_connections 2024-03-23T23:05:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.980+0000 7fea62b91740 1 -- 172.21.15.12:0/858732394 wait complete. 2024-03-23T23:05:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.980+0000 7fea62b91740 20 client.26386 trim_cache size 0 max 16384 2024-03-23T23:05:55.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.996+0000 7fea62b91740 1 Processor -- start 2024-03-23T23:05:55.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.996+0000 7fea62b91740 1 -- start start 2024-03-23T23:05:55.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.996+0000 7fea62b91740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709d6750 0x55c4709e4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:55.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.996+0000 7fea62b91740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4709db1e0 0x55c4709ec8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:55.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.996+0000 7fea62b91740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c4709dbbd0 0x55c4709d57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:55.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.996+0000 7fea62b91740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea581b1440 con 0x55c4709dbbd0 2024-03-23T23:05:55.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.996+0000 7fea62b91740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea581aef40 con 0x55c4709d6750 2024-03-23T23:05:55.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.996+0000 7fea62b91740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea581acfe0 con 0x55c4709db1e0 2024-03-23T23:05:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.996+0000 7fea5d400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709d6750 0x55c4709e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.996+0000 7fea5d400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709d6750 0x55c4709e4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:47498/0 (socket says 172.21.15.12:47498) 2024-03-23T23:05:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.996+0000 7fea5d400640 1 -- 172.21.15.12:0/4253877893 learned_addr learned my addr 172.21.15.12:0/4253877893 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.996+0000 7fea5ca00640 1 --2- 172.21.15.12:0/4253877893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4709db1e0 0x55c4709ec8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.996+0000 7fea5de00640 1 --2- 172.21.15.12:0/4253877893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c4709dbbd0 0x55c4709d57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.997+0000 7fea5d400640 1 -- 172.21.15.12:0/4253877893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4709db1e0 msgr2=0x55c4709ec8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.997+0000 7fea5d400640 1 --2- 172.21.15.12:0/4253877893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4709db1e0 0x55c4709ec8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.997+0000 7fea5d400640 1 -- 172.21.15.12:0/4253877893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c4709dbbd0 msgr2=0x55c4709d57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.997+0000 7fea5d400640 1 --2- 172.21.15.12:0/4253877893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c4709dbbd0 0x55c4709d57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.997+0000 7fea5d400640 1 -- 172.21.15.12:0/4253877893 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4709d74c0 con 0x55c4709d6750 2024-03-23T23:05:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.997+0000 7fea5d400640 1 --2- 172.21.15.12:0/4253877893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709d6750 0x55c4709e4090 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fea4c003640 tx=0x7fea4c009ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df7ea7adea6df809 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.997+0000 7fea49c00640 1 -- 172.21.15.12:0/4253877893 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c03b070 con 0x55c4709d6750 2024-03-23T23:05:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.997+0000 7fea49c00640 1 -- 172.21.15.12:0/4253877893 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fea4c00f750 con 0x55c4709d6750 2024-03-23T23:05:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.997+0000 7fea49c00640 1 -- 172.21.15.12:0/4253877893 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c027040 con 0x55c4709d6750 2024-03-23T23:05:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.998+0000 7fea62b91740 1 -- 172.21.15.12:0/4253877893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709d6750 msgr2=0x55c4709e4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.998+0000 7fea62b91740 1 --2- 172.21.15.12:0/4253877893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709d6750 0x55c4709e4090 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fea4c003640 tx=0x7fea4c009ba0 comp rx=0 tx=0).stop 2024-03-23T23:05:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.998+0000 7fea62b91740 1 -- 172.21.15.12:0/4253877893 shutdown_connections 2024-03-23T23:05:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.998+0000 7fea62b91740 1 --2- 172.21.15.12:0/4253877893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c4709dbbd0 0x55c4709d57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.998+0000 7fea62b91740 1 --2- 172.21.15.12:0/4253877893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4709db1e0 0x55c4709ec8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.998+0000 7fea62b91740 1 --2- 172.21.15.12:0/4253877893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709d6750 0x55c4709e4090 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.998+0000 7fea62b91740 1 -- 172.21.15.12:0/4253877893 >> 172.21.15.12:0/4253877893 conn(0x55c4709e0880 msgr2=0x55c4709d50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.998+0000 7fea62b91740 1 -- 172.21.15.12:0/4253877893 shutdown_connections 2024-03-23T23:05:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.998+0000 7fea62b91740 1 -- 172.21.15.12:0/4253877893 wait complete. 2024-03-23T23:05:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.998+0000 7fea62b91740 1 -- start start 2024-03-23T23:05:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.999+0000 7fea62b91740 1 Processor -- start 2024-03-23T23:05:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.999+0000 7fea62b91740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c4709d6750 0x55c4709ea7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.999+0000 7fea62b91740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4709db1e0 0x55c4709eace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.999+0000 7fea62b91740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709dbbd0 0x55c4709ebcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.999+0000 7fea62b91740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea581ad5e0 con 0x55c4709d6750 2024-03-23T23:05:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.999+0000 7fea62b91740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea581ae8e0 con 0x55c4709dbbd0 2024-03-23T23:05:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.999+0000 7fea62b91740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea581adc20 con 0x55c4709db1e0 2024-03-23T23:05:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.999+0000 7fea62b91740 10 client.0 _ll_register_callbacks cb 0x7fea44009a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:05:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.999+0000 7fea5de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709dbbd0 0x55c4709ebcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.999+0000 7fea5d400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c4709d6750 0x55c4709ea7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.999+0000 7fea5ca00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4709db1e0 0x55c4709eace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.999+0000 7fea5de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709dbbd0 0x55c4709ebcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:47504/0 (socket says 172.21.15.12:47504) 2024-03-23T23:05:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.999+0000 7fea5de00640 1 -- 172.21.15.12:0/3182290670 learned_addr learned my addr 172.21.15.12:0/3182290670 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:05:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:55.999+0000 7fea5ca00640 1 --2- 172.21.15.12:0/3182290670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4709db1e0 0x55c4709eace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:38630/0 (socket says 172.21.15.12:38630) 2024-03-23T23:05:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.000+0000 7fea5ca00640 1 -- 172.21.15.12:0/3182290670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709dbbd0 msgr2=0x55c4709ebcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.000+0000 7fea5ca00640 1 --2- 172.21.15.12:0/3182290670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709dbbd0 0x55c4709ebcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.000+0000 7fea5ca00640 1 -- 172.21.15.12:0/3182290670 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c4709d6750 msgr2=0x55c4709ea7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:05:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.000+0000 7fea5ca00640 1 --2- 172.21.15.12:0/3182290670 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c4709d6750 0x55c4709ea7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.000+0000 7fea5ca00640 1 -- 172.21.15.12:0/3182290670 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c4709ec2e0 con 0x55c4709db1e0 2024-03-23T23:05:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.000+0000 7fea5de00640 1 --2- 172.21.15.12:0/3182290670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709dbbd0 0x55c4709ebcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.000+0000 7fea5d400640 1 --2- 172.21.15.12:0/3182290670 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c4709d6750 0x55c4709ea7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:05:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.000+0000 7fea5ca00640 1 --2- 172.21.15.12:0/3182290670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4709db1e0 0x55c4709eace0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x55c4708fcb70 tx=0x7fea44004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c42ee340b3db0bde server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:05:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.000+0000 7fea3ea00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:05:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.000+0000 7fea3ea00640 1 -- 172.21.15.12:0/3182290670 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4402d890 con 0x55c4709db1e0 2024-03-23T23:05:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.000+0000 7fea3ea00640 1 -- 172.21.15.12:0/3182290670 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fea4401a070 con 0x55c4709db1e0 2024-03-23T23:05:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.000+0000 7fea62b91740 1 -- 172.21.15.12:0/3182290670 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fea4c00f750 con 0x55c4709db1e0 2024-03-23T23:05:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.001+0000 7fea3ea00640 1 -- 172.21.15.12:0/3182290670 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea4403c050 con 0x55c4709db1e0 2024-03-23T23:05:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.001+0000 7fea62b91740 10 client.26392 fetch_fsmap learned FSMap version 23 2024-03-23T23:05:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.001+0000 7fea62b91740 1 -- 172.21.15.12:0/3182290670 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c470a4cf20 con 0x55c4709db1e0 2024-03-23T23:05:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.001+0000 7fea3ea00640 1 -- 172.21.15.12:0/3182290670 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fea4c00f750 con 0x55c4709db1e0 2024-03-23T23:05:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.001+0000 7fea3ea00640 1 -- 172.21.15.12:0/3182290670 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fea4400fbd0 con 0x55c4709db1e0 2024-03-23T23:05:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.001+0000 7fea62b91740 10 client.26392 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:05:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.001+0000 7fea62b91740 10 client.26392 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:05:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.001+0000 7fea62b91740 20 client.26392 populate_metadata read hostname 'smithi012' 2024-03-23T23:05:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.001+0000 7fea62b91740 1 --2- 172.21.15.12:0/3182290670 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c4709e0d40 0x55c470a4f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:05:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.001+0000 7fea62b91740 4 client.26392 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:05:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.001+0000 7fea62b91740 1 -- 172.21.15.12:0/3182290670 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c4708c65d0 con 0x55c4709e0d40 2024-03-23T23:05:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.001+0000 7fea5d400640 1 --2- 172.21.15.12:0/3182290670 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c4709e0d40 0x55c470a4f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:05:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.002+0000 7fea5d400640 1 --2- 172.21.15.12:0/3182290670 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c4709e0d40 0x55c470a4f8f0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8e31e730a3e60ab server_cookie=743e825f16ca0bf1 in_seq=0 out_seq=0 2024-03-23T23:05:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.002+0000 7fea3ea00640 10 client.26392 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:56.014 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.014+0000 7fea3ea00640 1 -- 172.21.15.12:0/3182290670 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c4708c65d0 con 0x55c4709e0d40 2024-03-23T23:05:56.014 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.014+0000 7fea3ea00640 10 client.26392 handle_command_reply: tid=0 2024-03-23T23:05:56.014 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.014+0000 7fea62b91740 1 client.26392 shutdown 2024-03-23T23:05:56.014 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.014+0000 7fea62b91740 1 -- 172.21.15.12:0/3182290670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4709db1e0 msgr2=0x55c4709eace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:56.014 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.014+0000 7fea62b91740 1 --2- 172.21.15.12:0/3182290670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4709db1e0 0x55c4709eace0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x55c4708fcb70 tx=0x7fea44004730 comp rx=0 tx=0).stop 2024-03-23T23:05:56.015 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.014+0000 7fea62b91740 1 -- 172.21.15.12:0/3182290670 shutdown_connections 2024-03-23T23:05:56.015 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.014+0000 7fea62b91740 1 --2- 172.21.15.12:0/3182290670 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c4709e0d40 0x55c470a4f8f0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:56.015 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.014+0000 7fea62b91740 1 --2- 172.21.15.12:0/3182290670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4709dbbd0 0x55c4709ebcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:56.015 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.014+0000 7fea62b91740 1 --2- 172.21.15.12:0/3182290670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4709db1e0 0x55c4709eace0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:56.016 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.014+0000 7fea62b91740 1 --2- 172.21.15.12:0/3182290670 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c4709d6750 0x55c4709ea7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:56.016 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.014+0000 7fea62b91740 1 -- 172.21.15.12:0/3182290670 >> 172.21.15.12:0/3182290670 conn(0x55c4709e0880 msgr2=0x55c4709d5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:56.016 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.014+0000 7fea3ea00640 0 client.26392 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:05:56.016 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.014+0000 7fea62b91740 1 -- 172.21.15.12:0/3182290670 shutdown_connections 2024-03-23T23:05:56.016 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.014+0000 7fea62b91740 1 -- 172.21.15.12:0/3182290670 wait complete. 2024-03-23T23:05:56.016 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.015+0000 7fea62b91740 20 client.26392 trim_cache size 0 max 16384 2024-03-23T23:05:56.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:05:56.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:05:56.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:05:56.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:05:56.018 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.015+0000 7fea5fe00640 1 -- 172.21.15.12:0/1474783415 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fea400a28a0 msgr2=0x7fea400a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:56.019 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.015+0000 7fea5fe00640 1 --2- 172.21.15.12:0/1474783415 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fea400a28a0 0x7fea400a4d60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x55c4709dc430 tx=0x7fea44013040 comp rx=0 tx=0).stop 2024-03-23T23:05:56.019 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.015+0000 7fea5fe00640 1 -- 172.21.15.12:0/1474783415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea5810dca0 msgr2=0x7fea581a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:05:56.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.015+0000 7fea5fe00640 1 --2- 172.21.15.12:0/1474783415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea5810dca0 0x7fea581a5450 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fea5002c630 tx=0x7fea50032ea0 comp rx=0 tx=0).stop 2024-03-23T23:05:56.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.015+0000 7fea5fe00640 1 -- 172.21.15.12:0/1474783415 shutdown_connections 2024-03-23T23:05:56.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.015+0000 7fea5fe00640 1 --2- 172.21.15.12:0/1474783415 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fea400a28a0 0x7fea400a4d60 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:56.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.015+0000 7fea5fe00640 1 --2- 172.21.15.12:0/1474783415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea5810dca0 0x7fea581a5450 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:56.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.015+0000 7fea5fe00640 1 --2- 172.21.15.12:0/1474783415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea580756e0 0x7fea5819e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:56.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.015+0000 7fea5fe00640 1 --2- 172.21.15.12:0/1474783415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea58074100 0x7fea5819dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:05:56.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.015+0000 7fea5fe00640 1 -- 172.21.15.12:0/1474783415 >> 172.21.15.12:0/1474783415 conn(0x7fea580fcc60 msgr2=0x7fea58101040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:05:56.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.016+0000 7fea5fe00640 1 -- 172.21.15.12:0/1474783415 shutdown_connections 2024-03-23T23:05:56.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:05:56.016+0000 7fea5fe00640 1 -- 172.21.15.12:0/1474783415 wait complete. 2024-03-23T23:05:56.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:56 smithi012 ceph-mon[31695]: pgmap v514: 97 pgs: 97 active+clean; 2.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:05:56.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:56 smithi012 ceph-mon[31695]: scrub summary: idle 2024-03-23T23:05:56.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:56 smithi173 ceph-mon[36116]: pgmap v514: 97 pgs: 97 active+clean; 2.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:05:56.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:56 smithi173 ceph-mon[36116]: scrub summary: idle 2024-03-23T23:05:56.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:56 smithi203 ceph-mon[36693]: pgmap v514: 97 pgs: 97 active+clean; 2.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:05:56.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:56 smithi203 ceph-mon[36693]: scrub summary: idle 2024-03-23T23:05:58.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:05:58 smithi012 ceph-mon[31695]: pgmap v515: 97 pgs: 97 active+clean; 2.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:05:58.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:05:58 smithi173 ceph-mon[36116]: pgmap v515: 97 pgs: 97 active+clean; 2.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:05:58.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:05:58 smithi203 ceph-mon[36693]: pgmap v515: 97 pgs: 97 active+clean; 2.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:06:00.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:00 smithi012 ceph-mon[31695]: pgmap v516: 97 pgs: 97 active+clean; 2.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-03-23T23:06:00.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:00 smithi173 ceph-mon[36116]: pgmap v516: 97 pgs: 97 active+clean; 2.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-03-23T23:06:00.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:00 smithi203 ceph-mon[36693]: pgmap v516: 97 pgs: 97 active+clean; 2.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-03-23T23:06:02.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:02 smithi012 ceph-mon[31695]: pgmap v517: 97 pgs: 97 active+clean; 2.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-03-23T23:06:02.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:02 smithi173 ceph-mon[36116]: pgmap v517: 97 pgs: 97 active+clean; 2.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-03-23T23:06:02.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:02 smithi203 ceph-mon[36693]: pgmap v517: 97 pgs: 97 active+clean; 2.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-03-23T23:06:03.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:03 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:06:03.808 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:03 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:06:03.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:03 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:06:04.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:04 smithi012 ceph-mon[31695]: pgmap v518: 97 pgs: 97 active+clean; 2.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:06:04.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:04 smithi173 ceph-mon[36116]: pgmap v518: 97 pgs: 97 active+clean; 2.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:06:04.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:04 smithi203 ceph-mon[36693]: pgmap v518: 97 pgs: 97 active+clean; 2.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:06:05.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:05 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:06:05.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:05 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:06:05.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:05 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:06:05.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:05 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:06:05.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:05 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:06:05.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:05 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:06:05.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:05 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:06:05.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:05 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:06:05.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:05 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:06:07.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:06 smithi012 ceph-mon[31695]: pgmap v519: 97 pgs: 97 active+clean; 2.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:06:07.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:06 smithi173 ceph-mon[36116]: pgmap v519: 97 pgs: 97 active+clean; 2.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:06:07.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:06 smithi203 ceph-mon[36693]: pgmap v519: 97 pgs: 97 active+clean; 2.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:06:09.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:08 smithi012 ceph-mon[31695]: pgmap v520: 97 pgs: 97 active+clean; 2.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:06:09.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:08 smithi173 ceph-mon[36116]: pgmap v520: 97 pgs: 97 active+clean; 2.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:06:09.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:08 smithi203 ceph-mon[36693]: pgmap v520: 97 pgs: 97 active+clean; 2.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:06:09.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.161+0000 7f7d9f200640 1 -- 172.21.15.12:0/1924760542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7da0101510 msgr2=0x7f7da0101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:09.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.161+0000 7f7d9f200640 1 --2- 172.21.15.12:0/1924760542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7da0101510 0x7f7da0101930 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7d90009900 tx=0x7f7d9002bc30 comp rx=0 tx=0).stop 2024-03-23T23:06:09.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.161+0000 7f7d9f200640 1 -- 172.21.15.12:0/1924760542 shutdown_connections 2024-03-23T23:06:09.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.161+0000 7f7d9f200640 1 --2- 172.21.15.12:0/1924760542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7da01030c0 0x7f7da0109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.161+0000 7f7d9f200640 1 --2- 172.21.15.12:0/1924760542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7da0102700 0x7f7da0102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.161+0000 7f7d9f200640 1 --2- 172.21.15.12:0/1924760542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7da0101510 0x7f7da0101930 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.161+0000 7f7d9f200640 1 -- 172.21.15.12:0/1924760542 >> 172.21.15.12:0/1924760542 conn(0x7f7da00fca80 msgr2=0x7f7da00feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:09.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.161+0000 7f7d9f200640 1 -- 172.21.15.12:0/1924760542 shutdown_connections 2024-03-23T23:06:09.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.161+0000 7f7d9f200640 1 -- 172.21.15.12:0/1924760542 wait complete. 2024-03-23T23:06:09.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.162+0000 7f7d9f200640 1 Processor -- start 2024-03-23T23:06:09.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.162+0000 7f7d9f200640 1 -- start start 2024-03-23T23:06:09.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.162+0000 7f7d9f200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7da0101510 0x7f7da0197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:09.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.163+0000 7f7d9f200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7da0102700 0x7f7da0197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:09.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.163+0000 7f7d9f200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7da01030c0 0x7f7da019ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:09.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.163+0000 7f7d9f200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da010d1c0 con 0x7f7da0101510 2024-03-23T23:06:09.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.163+0000 7f7d9f200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da010d040 con 0x7f7da0102700 2024-03-23T23:06:09.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.163+0000 7f7d9f200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da010d340 con 0x7f7da01030c0 2024-03-23T23:06:09.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.163+0000 7f7d95400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7da0102700 0x7f7da0197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:09.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.163+0000 7f7d9de00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7da0101510 0x7f7da0197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:09.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.163+0000 7f7d95400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7da0102700 0x7f7da0197da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:55974/0 (socket says 172.21.15.12:55974) 2024-03-23T23:06:09.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.163+0000 7f7d95400640 1 -- 172.21.15.12:0/392019243 learned_addr learned my addr 172.21.15.12:0/392019243 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:09.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.163+0000 7f7d9e800640 1 --2- 172.21.15.12:0/392019243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7da01030c0 0x7f7da019ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:09.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.163+0000 7f7d95400640 1 -- 172.21.15.12:0/392019243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7da01030c0 msgr2=0x7f7da019ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:09.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.163+0000 7f7d95400640 1 --2- 172.21.15.12:0/392019243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7da01030c0 0x7f7da019ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.163+0000 7f7d95400640 1 -- 172.21.15.12:0/392019243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7da0101510 msgr2=0x7f7da0197860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:09.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.163+0000 7f7d95400640 1 --2- 172.21.15.12:0/392019243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7da0101510 0x7f7da0197860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.163+0000 7f7d95400640 1 -- 172.21.15.12:0/392019243 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da019f480 con 0x7f7da0102700 2024-03-23T23:06:09.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.164+0000 7f7d95400640 1 --2- 172.21.15.12:0/392019243 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7da0102700 0x7f7da0197da0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f7d8800e9f0 tx=0x7f7d8800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:09.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.164+0000 7f7d97400640 1 -- 172.21.15.12:0/392019243 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8800cd80 con 0x7f7da0102700 2024-03-23T23:06:09.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.164+0000 7f7d9f200640 1 -- 172.21.15.12:0/392019243 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7da019f770 con 0x7f7da0102700 2024-03-23T23:06:09.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.164+0000 7f7d97400640 1 -- 172.21.15.12:0/392019243 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d88010910 con 0x7f7da0102700 2024-03-23T23:06:09.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.164+0000 7f7d97400640 1 -- 172.21.15.12:0/392019243 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d880188f0 con 0x7f7da0102700 2024-03-23T23:06:09.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.164+0000 7f7d9f200640 1 -- 172.21.15.12:0/392019243 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7da019fcc0 con 0x7f7da0102700 2024-03-23T23:06:09.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.167+0000 7f7d97400640 1 -- 172.21.15.12:0/392019243 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7d88010ac0 con 0x7f7da0102700 2024-03-23T23:06:09.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.167+0000 7f7da79bc740 1 Processor -- start 2024-03-23T23:06:09.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.167+0000 7f7da79bc740 1 -- start start 2024-03-23T23:06:09.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.168+0000 7f7da79bc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3c6e0e3f0 0x55a3c6e0e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:09.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.168+0000 7f7da79bc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3c6e0ede0 0x55a3c6e138e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:09.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.168+0000 7f7da79bc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c6e13f50 0x55a3c6e163e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:09.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.168+0000 7f7da79bc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3c6dd13b0 con 0x55a3c6e0e3f0 2024-03-23T23:06:09.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.168+0000 7f7da79bc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3c6d73830 con 0x55a3c6e0ede0 2024-03-23T23:06:09.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.168+0000 7f7da79bc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3c6d1c5d0 con 0x55a3c6e13f50 2024-03-23T23:06:09.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.168+0000 7f7d9de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3c6e0ede0 0x55a3c6e138e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:09.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.168+0000 7f7d9de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3c6e0ede0 0x55a3c6e138e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:55990/0 (socket says 172.21.15.12:55990) 2024-03-23T23:06:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.168+0000 7f7d9de00640 1 -- 172.21.15.12:0/938889845 learned_addr learned my addr 172.21.15.12:0/938889845 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.168+0000 7f7d9e800640 1 --2- 172.21.15.12:0/938889845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3c6e0e3f0 0x55a3c6e0e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.168+0000 7f7d95400640 1 --2- 172.21.15.12:0/938889845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c6e13f50 0x55a3c6e163e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.168+0000 7f7d95400640 1 -- 172.21.15.12:0/938889845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3c6e0ede0 msgr2=0x55a3c6e138e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.168+0000 7f7d95400640 1 --2- 172.21.15.12:0/938889845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3c6e0ede0 0x55a3c6e138e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.168+0000 7f7d95400640 1 -- 172.21.15.12:0/938889845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3c6e0e3f0 msgr2=0x55a3c6e0e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.168+0000 7f7d95400640 1 --2- 172.21.15.12:0/938889845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3c6e0e3f0 0x55a3c6e0e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.168+0000 7f7d95400640 1 -- 172.21.15.12:0/938889845 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a3c6d342c0 con 0x55a3c6e13f50 2024-03-23T23:06:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.169+0000 7f7d9de00640 1 --2- 172.21.15.12:0/938889845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3c6e0ede0 0x55a3c6e138e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.169+0000 7f7d97400640 1 --2- 172.21.15.12:0/392019243 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7d840a27f0 0x7f7d840a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.169+0000 7f7d95400640 1 --2- 172.21.15.12:0/938889845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c6e13f50 0x55a3c6e163e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7d880cc8b0 tx=0x7f7d880d5830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86cac310386fe86e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.169+0000 7f7d97400640 1 -- 172.21.15.12:0/392019243 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7d880cee60 con 0x7f7da0102700 2024-03-23T23:06:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.169+0000 7f7d9e800640 1 --2- 172.21.15.12:0/392019243 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7d840a27f0 0x7f7d840a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.169+0000 7f7d8f000640 1 -- 172.21.15.12:0/938889845 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d88097b00 con 0x55a3c6e13f50 2024-03-23T23:06:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.169+0000 7f7d8f000640 1 -- 172.21.15.12:0/938889845 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d8809b860 con 0x55a3c6e13f50 2024-03-23T23:06:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.169+0000 7f7d8f000640 1 -- 172.21.15.12:0/938889845 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8809b1c0 con 0x55a3c6e13f50 2024-03-23T23:06:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.170+0000 7f7da79bc740 1 -- 172.21.15.12:0/938889845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c6e13f50 msgr2=0x55a3c6e163e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.170+0000 7f7da79bc740 1 --2- 172.21.15.12:0/938889845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c6e13f50 0x55a3c6e163e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7d880cc8b0 tx=0x7f7d880d5830 comp rx=0 tx=0).stop 2024-03-23T23:06:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.170+0000 7f7d9e800640 1 --2- 172.21.15.12:0/392019243 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7d840a27f0 0x7f7d840a4cb0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x55a3c6e0f640 tx=0x7f7d98009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.170+0000 7f7da79bc740 1 -- 172.21.15.12:0/938889845 shutdown_connections 2024-03-23T23:06:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.170+0000 7f7da79bc740 1 --2- 172.21.15.12:0/938889845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c6e13f50 0x55a3c6e163e0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.170+0000 7f7da79bc740 1 --2- 172.21.15.12:0/938889845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3c6e0ede0 0x55a3c6e138e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.170+0000 7f7da79bc740 1 --2- 172.21.15.12:0/938889845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3c6e0e3f0 0x55a3c6e0e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.170+0000 7f7da79bc740 1 -- 172.21.15.12:0/938889845 >> 172.21.15.12:0/938889845 conn(0x55a3c6e09960 msgr2=0x55a3c6e09d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.170+0000 7f7da79bc740 1 -- 172.21.15.12:0/938889845 shutdown_connections 2024-03-23T23:06:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.170+0000 7f7da79bc740 1 -- 172.21.15.12:0/938889845 wait complete. 2024-03-23T23:06:09.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.170+0000 7f7da79bc740 1 -- start start 2024-03-23T23:06:09.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.171+0000 7f7da79bc740 1 Processor -- start 2024-03-23T23:06:09.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.171+0000 7f7da79bc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3c6e0e3f0 0x55a3c6e18550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:09.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.171+0000 7f7da79bc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3c6e0ede0 0x55a3c6e18a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:09.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.171+0000 7f7da79bc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c6e13f50 0x55a3c6e1fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:09.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.171+0000 7f7da79bc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3c6d2fa20 con 0x55a3c6e0ede0 2024-03-23T23:06:09.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.171+0000 7f7da79bc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3c6d6f8d0 con 0x55a3c6e0e3f0 2024-03-23T23:06:09.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.171+0000 7f7da79bc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3c6e16f00 con 0x55a3c6e13f50 2024-03-23T23:06:09.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.171+0000 7f7da79bc740 10 client.0 _ll_register_callbacks cb 0x55a3c6d6c1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:09.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.171+0000 7f7d9de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3c6e0e3f0 0x55a3c6e18550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:09.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.171+0000 7f7d95400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3c6e0ede0 0x55a3c6e18a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:09.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.171+0000 7f7d9de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3c6e0e3f0 0x55a3c6e18550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:56000/0 (socket says 172.21.15.12:56000) 2024-03-23T23:06:09.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.171+0000 7f7d9de00640 1 -- 172.21.15.12:0/2273848454 learned_addr learned my addr 172.21.15.12:0/2273848454 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:09.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.171+0000 7f7d9e800640 1 --2- 172.21.15.12:0/2273848454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c6e13f50 0x55a3c6e1fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:09.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.172+0000 7f7d9de00640 1 -- 172.21.15.12:0/2273848454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c6e13f50 msgr2=0x55a3c6e1fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:09.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.172+0000 7f7d9de00640 1 --2- 172.21.15.12:0/2273848454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c6e13f50 0x55a3c6e1fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.172+0000 7f7d9de00640 1 -- 172.21.15.12:0/2273848454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3c6e0ede0 msgr2=0x55a3c6e18a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:09.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.172+0000 7f7d9de00640 1 --2- 172.21.15.12:0/2273848454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3c6e0ede0 0x55a3c6e18a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.172+0000 7f7d9de00640 1 -- 172.21.15.12:0/2273848454 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a3c6a88210 con 0x55a3c6e0e3f0 2024-03-23T23:06:09.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.172+0000 7f7d95400640 1 --2- 172.21.15.12:0/2273848454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3c6e0ede0 0x55a3c6e18a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:09.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.172+0000 7f7d9de00640 1 --2- 172.21.15.12:0/2273848454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3c6e0e3f0 0x55a3c6e18550 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7d90036040 tx=0x7f7d90004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d379d45178e07d78 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:09.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.172+0000 7f7d83400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:06:09.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.173+0000 7f7d83400640 1 -- 172.21.15.12:0/2273848454 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d90005d50 con 0x55a3c6e0e3f0 2024-03-23T23:06:09.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.173+0000 7f7d83400640 1 -- 172.21.15.12:0/2273848454 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7d90005030 con 0x55a3c6e0e3f0 2024-03-23T23:06:09.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.173+0000 7f7da79bc740 1 -- 172.21.15.12:0/2273848454 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7d8809b860 con 0x55a3c6e0e3f0 2024-03-23T23:06:09.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.173+0000 7f7d83400640 1 -- 172.21.15.12:0/2273848454 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d90052050 con 0x55a3c6e0e3f0 2024-03-23T23:06:09.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.173+0000 7f7da79bc740 10 client.26441 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:09.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.173+0000 7f7da79bc740 1 -- 172.21.15.12:0/2273848454 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a3c6b54620 con 0x55a3c6e0e3f0 2024-03-23T23:06:09.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.173+0000 7f7d83400640 1 -- 172.21.15.12:0/2273848454 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d8809b860 con 0x55a3c6e0e3f0 2024-03-23T23:06:09.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.174+0000 7f7d83400640 1 -- 172.21.15.12:0/2273848454 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f7d90003850 con 0x55a3c6e0e3f0 2024-03-23T23:06:09.178 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.174+0000 7f7da79bc740 10 client.26441 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:09.178 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.174+0000 7f7da79bc740 10 client.26441 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:06:09.178 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.174+0000 7f7da79bc740 20 client.26441 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:09.178 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.174+0000 7f7da79bc740 1 --2- 172.21.15.12:0/2273848454 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a3c6e24370 0x55a3c6e26760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:09.178 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.174+0000 7f7da79bc740 4 client.26441 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:06:09.178 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.174+0000 7f7da79bc740 1 -- 172.21.15.12:0/2273848454 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a3c6d80d70 con 0x55a3c6e24370 2024-03-23T23:06:09.178 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.174+0000 7f7d95400640 1 --2- 172.21.15.12:0/2273848454 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a3c6e24370 0x55a3c6e26760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:09.178 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.174+0000 7f7d95400640 1 --2- 172.21.15.12:0/2273848454 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a3c6e24370 0x55a3c6e26760 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca7f94c7520636b9 server_cookie=9b6f918685ba4f5c in_seq=0 out_seq=0 2024-03-23T23:06:09.178 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.174+0000 7f7d83400640 10 client.26441 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:09.178 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.175+0000 7f7d83400640 1 -- 172.21.15.12:0/2273848454 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a3c6d80d70 con 0x55a3c6e24370 2024-03-23T23:06:09.178 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.175+0000 7f7d83400640 10 client.26441 handle_command_reply: tid=0 2024-03-23T23:06:09.178 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.175+0000 7f7da79bc740 1 client.26441 shutdown 2024-03-23T23:06:09.178 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.176+0000 7f7da79bc740 1 -- 172.21.15.12:0/2273848454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3c6e0e3f0 msgr2=0x55a3c6e18550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:09.179 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.176+0000 7f7da79bc740 1 --2- 172.21.15.12:0/2273848454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3c6e0e3f0 0x55a3c6e18550 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7d90036040 tx=0x7f7d90004270 comp rx=0 tx=0).stop 2024-03-23T23:06:09.179 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.176+0000 7f7da79bc740 1 -- 172.21.15.12:0/2273848454 shutdown_connections 2024-03-23T23:06:09.179 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.176+0000 7f7da79bc740 1 --2- 172.21.15.12:0/2273848454 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a3c6e24370 0x55a3c6e26760 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.179 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.176+0000 7f7da79bc740 1 --2- 172.21.15.12:0/2273848454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c6e13f50 0x55a3c6e1fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.179 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.176+0000 7f7da79bc740 1 --2- 172.21.15.12:0/2273848454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3c6e0ede0 0x55a3c6e18a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.179 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.176+0000 7f7da79bc740 1 --2- 172.21.15.12:0/2273848454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3c6e0e3f0 0x55a3c6e18550 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.179 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.176+0000 7f7da79bc740 1 -- 172.21.15.12:0/2273848454 >> 172.21.15.12:0/2273848454 conn(0x55a3c6e09960 msgr2=0x55a3c6e08b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:09.179 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.176+0000 7f7d83400640 0 client.26441 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:09.179 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.176+0000 7f7da79bc740 1 -- 172.21.15.12:0/2273848454 shutdown_connections 2024-03-23T23:06:09.179 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.176+0000 7f7da79bc740 1 -- 172.21.15.12:0/2273848454 wait complete. 2024-03-23T23:06:09.179 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.176+0000 7f7da79bc740 20 client.26441 trim_cache size 0 max 16384 2024-03-23T23:06:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.191+0000 7f7da79bc740 1 Processor -- start 2024-03-23T23:06:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.191+0000 7f7da79bc740 1 -- start start 2024-03-23T23:06:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.192+0000 7f7da79bc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3c6e0e3f0 0x55a3c6e09d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.192+0000 7f7da79bc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c6e0ede0 0x55a3c6e0cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.192+0000 7f7da79bc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3c6e13f50 0x55a3c6e0d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.192+0000 7f7da79bc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da01aaea0 con 0x55a3c6e13f50 2024-03-23T23:06:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.192+0000 7f7da79bc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da01a89a0 con 0x55a3c6e0e3f0 2024-03-23T23:06:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.192+0000 7f7da79bc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da01a6a40 con 0x55a3c6e0ede0 2024-03-23T23:06:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.192+0000 7f7d9e800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3c6e13f50 0x55a3c6e0d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.192+0000 7f7d95400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c6e0ede0 0x55a3c6e0cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.192+0000 7f7d95400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c6e0ede0 0x55a3c6e0cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55010/0 (socket says 172.21.15.12:55010) 2024-03-23T23:06:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.192+0000 7f7d95400640 1 -- 172.21.15.12:0/897886397 learned_addr learned my addr 172.21.15.12:0/897886397 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.193+0000 7f7d95400640 1 -- 172.21.15.12:0/897886397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3c6e0e3f0 msgr2=0x55a3c6e09d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.193+0000 7f7d95400640 1 --2- 172.21.15.12:0/897886397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3c6e0e3f0 0x55a3c6e09d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.193+0000 7f7d95400640 1 -- 172.21.15.12:0/897886397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3c6e13f50 msgr2=0x55a3c6e0d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.193+0000 7f7d95400640 1 --2- 172.21.15.12:0/897886397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3c6e13f50 0x55a3c6e0d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.193+0000 7f7d95400640 1 -- 172.21.15.12:0/897886397 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a3c6e17040 con 0x55a3c6e0ede0 2024-03-23T23:06:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.193+0000 7f7d95400640 1 --2- 172.21.15.12:0/897886397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c6e0ede0 0x55a3c6e0cbc0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7d88098230 tx=0x7f7d880d5830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9180b800ed601415 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.193+0000 7f7d8e600640 1 -- 172.21.15.12:0/897886397 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8809b3a0 con 0x55a3c6e0ede0 2024-03-23T23:06:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.193+0000 7f7d8e600640 1 -- 172.21.15.12:0/897886397 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d8809b860 con 0x55a3c6e0ede0 2024-03-23T23:06:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.193+0000 7f7d8e600640 1 -- 172.21.15.12:0/897886397 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8809b630 con 0x55a3c6e0ede0 2024-03-23T23:06:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.194+0000 7f7da79bc740 1 -- 172.21.15.12:0/897886397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c6e0ede0 msgr2=0x55a3c6e0cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.194+0000 7f7da79bc740 1 --2- 172.21.15.12:0/897886397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c6e0ede0 0x55a3c6e0cbc0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7d88098230 tx=0x7f7d880d5830 comp rx=0 tx=0).stop 2024-03-23T23:06:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.194+0000 7f7da79bc740 1 -- 172.21.15.12:0/897886397 shutdown_connections 2024-03-23T23:06:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.194+0000 7f7da79bc740 1 --2- 172.21.15.12:0/897886397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3c6e13f50 0x55a3c6e0d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.194+0000 7f7da79bc740 1 --2- 172.21.15.12:0/897886397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c6e0ede0 0x55a3c6e0cbc0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.194+0000 7f7da79bc740 1 --2- 172.21.15.12:0/897886397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3c6e0e3f0 0x55a3c6e09d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.194+0000 7f7da79bc740 1 -- 172.21.15.12:0/897886397 >> 172.21.15.12:0/897886397 conn(0x55a3c6e09960 msgr2=0x55a3c6e13700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:09.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.194+0000 7f7da79bc740 1 -- 172.21.15.12:0/897886397 shutdown_connections 2024-03-23T23:06:09.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.194+0000 7f7da79bc740 1 -- 172.21.15.12:0/897886397 wait complete. 2024-03-23T23:06:09.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.194+0000 7f7da79bc740 1 -- start start 2024-03-23T23:06:09.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.194+0000 7f7da79bc740 1 Processor -- start 2024-03-23T23:06:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.195+0000 7f7da79bc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c6e0e3f0 0x55a3c6e1cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.195+0000 7f7da79bc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3c6e0ede0 0x55a3c6e1d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.195+0000 7f7da79bc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3c6e13f50 0x55a3c6e1e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.195+0000 7f7da79bc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da01a7040 con 0x55a3c6e13f50 2024-03-23T23:06:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.195+0000 7f7da79bc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da01a8340 con 0x55a3c6e0ede0 2024-03-23T23:06:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.195+0000 7f7da79bc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da01a7680 con 0x55a3c6e0e3f0 2024-03-23T23:06:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.195+0000 7f7da79bc740 10 client.0 _ll_register_callbacks cb 0x7f7d90009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.195+0000 7f7d95400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3c6e0ede0 0x55a3c6e1d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.195+0000 7f7d95400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3c6e0ede0 0x55a3c6e1d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:56008/0 (socket says 172.21.15.12:56008) 2024-03-23T23:06:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.195+0000 7f7d95400640 1 -- 172.21.15.12:0/1693316359 learned_addr learned my addr 172.21.15.12:0/1693316359 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.195+0000 7f7d9de00640 1 --2- 172.21.15.12:0/1693316359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c6e0e3f0 0x55a3c6e1cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.195+0000 7f7d9e800640 1 --2- 172.21.15.12:0/1693316359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3c6e13f50 0x55a3c6e1e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.195+0000 7f7d95400640 1 -- 172.21.15.12:0/1693316359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c6e0e3f0 msgr2=0x55a3c6e1cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.195+0000 7f7d95400640 1 --2- 172.21.15.12:0/1693316359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c6e0e3f0 0x55a3c6e1cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.195+0000 7f7d95400640 1 -- 172.21.15.12:0/1693316359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3c6e13f50 msgr2=0x55a3c6e1e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.195+0000 7f7d95400640 1 --2- 172.21.15.12:0/1693316359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3c6e13f50 0x55a3c6e1e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.196+0000 7f7d95400640 1 -- 172.21.15.12:0/1693316359 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a3c6e1e890 con 0x55a3c6e0ede0 2024-03-23T23:06:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.196+0000 7f7d9de00640 1 --2- 172.21.15.12:0/1693316359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c6e0e3f0 0x55a3c6e1cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.196+0000 7f7d95400640 1 --2- 172.21.15.12:0/1693316359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3c6e0ede0 0x55a3c6e1d300 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7d880a3a20 tx=0x7f7d880a3f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b73a8b7aec4f0bb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.196+0000 7f7d83400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:06:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.196+0000 7f7d83400640 1 -- 172.21.15.12:0/1693316359 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8809b630 con 0x55a3c6e0ede0 2024-03-23T23:06:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.196+0000 7f7d83400640 1 -- 172.21.15.12:0/1693316359 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7d880cca20 con 0x55a3c6e0ede0 2024-03-23T23:06:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.196+0000 7f7da79bc740 1 -- 172.21.15.12:0/1693316359 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7d8809b860 con 0x55a3c6e0ede0 2024-03-23T23:06:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.197+0000 7f7d83400640 1 -- 172.21.15.12:0/1693316359 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d880d2530 con 0x55a3c6e0ede0 2024-03-23T23:06:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.197+0000 7f7da79bc740 10 client.26444 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.197+0000 7f7da79bc740 1 -- 172.21.15.12:0/1693316359 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a3c6e843a0 con 0x55a3c6e0ede0 2024-03-23T23:06:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.197+0000 7f7d83400640 1 -- 172.21.15.12:0/1693316359 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d8809b860 con 0x55a3c6e0ede0 2024-03-23T23:06:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.197+0000 7f7d83400640 1 -- 172.21.15.12:0/1693316359 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f7d880d1070 con 0x55a3c6e0ede0 2024-03-23T23:06:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.197+0000 7f7da79bc740 10 client.26444 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.197+0000 7f7da79bc740 10 client.26444 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:06:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.197+0000 7f7da79bc740 20 client.26444 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.197+0000 7f7da79bc740 1 --2- 172.21.15.12:0/1693316359 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a3c6e846b0 0x55a3c6e86aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.197+0000 7f7da79bc740 4 client.26444 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:06:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.197+0000 7f7da79bc740 1 -- 172.21.15.12:0/1693316359 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a3c6d7b0c0 con 0x55a3c6e846b0 2024-03-23T23:06:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.198+0000 7f7d9de00640 1 --2- 172.21.15.12:0/1693316359 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a3c6e846b0 0x55a3c6e86aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.200+0000 7f7d9de00640 1 --2- 172.21.15.12:0/1693316359 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a3c6e846b0 0x55a3c6e86aa0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=858a769cf4baf4e8 server_cookie=c5eb32ec910f195e in_seq=0 out_seq=0 2024-03-23T23:06:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.200+0000 7f7d83400640 10 client.26444 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.213+0000 7f7d83400640 1 -- 172.21.15.12:0/1693316359 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a3c6d7b0c0 con 0x55a3c6e846b0 2024-03-23T23:06:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.213+0000 7f7d83400640 10 client.26444 handle_command_reply: tid=0 2024-03-23T23:06:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.213+0000 7f7da79bc740 1 client.26444 shutdown 2024-03-23T23:06:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.213+0000 7f7da79bc740 1 -- 172.21.15.12:0/1693316359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3c6e0ede0 msgr2=0x55a3c6e1d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.213+0000 7f7da79bc740 1 --2- 172.21.15.12:0/1693316359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3c6e0ede0 0x55a3c6e1d300 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7d880a3a20 tx=0x7f7d880a3f50 comp rx=0 tx=0).stop 2024-03-23T23:06:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.214+0000 7f7da79bc740 1 -- 172.21.15.12:0/1693316359 shutdown_connections 2024-03-23T23:06:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.214+0000 7f7da79bc740 1 --2- 172.21.15.12:0/1693316359 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a3c6e846b0 0x55a3c6e86aa0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.214+0000 7f7da79bc740 1 --2- 172.21.15.12:0/1693316359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3c6e13f50 0x55a3c6e1e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.214+0000 7f7da79bc740 1 --2- 172.21.15.12:0/1693316359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3c6e0ede0 0x55a3c6e1d300 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.214+0000 7f7da79bc740 1 --2- 172.21.15.12:0/1693316359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c6e0e3f0 0x55a3c6e1cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.214+0000 7f7d83400640 0 client.26444 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.214+0000 7f7da79bc740 1 -- 172.21.15.12:0/1693316359 >> 172.21.15.12:0/1693316359 conn(0x55a3c6e09960 msgr2=0x55a3c6e09d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.214+0000 7f7da79bc740 1 -- 172.21.15.12:0/1693316359 shutdown_connections 2024-03-23T23:06:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.214+0000 7f7da79bc740 1 -- 172.21.15.12:0/1693316359 wait complete. 2024-03-23T23:06:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.214+0000 7f7da79bc740 20 client.26444 trim_cache size 0 max 16384 2024-03-23T23:06:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:06:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:06:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:06:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:06:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.215+0000 7f7d9f200640 1 -- 172.21.15.12:0/392019243 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7d840a27f0 msgr2=0x7f7d840a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.215+0000 7f7d9f200640 1 --2- 172.21.15.12:0/392019243 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7d840a27f0 0x7f7d840a4cb0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x55a3c6e0f640 tx=0x7f7d98009500 comp rx=0 tx=0).stop 2024-03-23T23:06:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.215+0000 7f7d9f200640 1 -- 172.21.15.12:0/392019243 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7da0102700 msgr2=0x7f7da0197da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:09.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.215+0000 7f7d9f200640 1 --2- 172.21.15.12:0/392019243 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7da0102700 0x7f7da0197da0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f7d8800e9f0 tx=0x7f7d8800eeb0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.215+0000 7f7d9f200640 1 -- 172.21.15.12:0/392019243 shutdown_connections 2024-03-23T23:06:09.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.215+0000 7f7d9f200640 1 --2- 172.21.15.12:0/392019243 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7d840a27f0 0x7f7d840a4cb0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.215+0000 7f7d9f200640 1 --2- 172.21.15.12:0/392019243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7da01030c0 0x7f7da019ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.215+0000 7f7d9f200640 1 --2- 172.21.15.12:0/392019243 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7da0102700 0x7f7da0197da0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.215+0000 7f7d9f200640 1 --2- 172.21.15.12:0/392019243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7da0101510 0x7f7da0197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:09.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.215+0000 7f7d9f200640 1 -- 172.21.15.12:0/392019243 >> 172.21.15.12:0/392019243 conn(0x7f7da00fca80 msgr2=0x7f7da0105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:09.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.216+0000 7f7d9f200640 1 -- 172.21.15.12:0/392019243 shutdown_connections 2024-03-23T23:06:09.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:09.216+0000 7f7d9f200640 1 -- 172.21.15.12:0/392019243 wait complete. 2024-03-23T23:06:11.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:10 smithi012 ceph-mon[31695]: pgmap v521: 97 pgs: 97 active+clean; 2.6 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 14 op/s 2024-03-23T23:06:11.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:10 smithi173 ceph-mon[36116]: pgmap v521: 97 pgs: 97 active+clean; 2.6 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 14 op/s 2024-03-23T23:06:11.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:10 smithi203 ceph-mon[36693]: pgmap v521: 97 pgs: 97 active+clean; 2.6 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 14 op/s 2024-03-23T23:06:13.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:12 smithi012 ceph-mon[31695]: pgmap v522: 97 pgs: 97 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-03-23T23:06:13.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:12 smithi173 ceph-mon[36116]: pgmap v522: 97 pgs: 97 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-03-23T23:06:13.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:12 smithi203 ceph-mon[36693]: pgmap v522: 97 pgs: 97 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-03-23T23:06:15.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:14 smithi012 ceph-mon[31695]: pgmap v523: 97 pgs: 97 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:06:15.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:14 smithi173 ceph-mon[36116]: pgmap v523: 97 pgs: 97 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:06:15.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:14 smithi203 ceph-mon[36693]: pgmap v523: 97 pgs: 97 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:06:17.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:16 smithi012 ceph-mon[31695]: pgmap v524: 97 pgs: 97 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:06:17.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:16 smithi173 ceph-mon[36116]: pgmap v524: 97 pgs: 97 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:06:17.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:16 smithi203 ceph-mon[36693]: pgmap v524: 97 pgs: 97 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:06:18.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:18 smithi012 ceph-mon[31695]: pgmap v525: 97 pgs: 97 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:06:18.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:18 smithi173 ceph-mon[36116]: pgmap v525: 97 pgs: 97 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:06:18.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:18 smithi203 ceph-mon[36693]: pgmap v525: 97 pgs: 97 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:06:20.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:20 smithi012 ceph-mon[31695]: pgmap v526: 97 pgs: 97 active+clean; 2.7 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:06:20.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:20 smithi173 ceph-mon[36116]: pgmap v526: 97 pgs: 97 active+clean; 2.7 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:06:20.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:20 smithi203 ceph-mon[36693]: pgmap v526: 97 pgs: 97 active+clean; 2.7 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:06:21.379 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:06:21.380 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:06:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.634+0000 7f5e00a00640 1 -- 172.21.15.12:0/1787579428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5dfc105370 msgr2=0x7f5dfc10bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.634+0000 7f5e00a00640 1 --2- 172.21.15.12:0/1787579428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5dfc105370 0x7f5dfc10bc00 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5df4009ca0 tx=0x7f5df402c100 comp rx=0 tx=0).stop 2024-03-23T23:06:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.634+0000 7f5e00a00640 1 -- 172.21.15.12:0/1787579428 shutdown_connections 2024-03-23T23:06:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.634+0000 7f5e00a00640 1 --2- 172.21.15.12:0/1787579428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5dfc105370 0x7f5dfc10bc00 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.634+0000 7f5e00a00640 1 --2- 172.21.15.12:0/1787579428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5dfc1049b0 0x7f5dfc104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.634+0000 7f5e00a00640 1 --2- 172.21.15.12:0/1787579428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dfc1037c0 0x7f5dfc103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.634+0000 7f5e00a00640 1 -- 172.21.15.12:0/1787579428 >> 172.21.15.12:0/1787579428 conn(0x7f5dfc0fed50 msgr2=0x7f5dfc101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:21.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.634+0000 7f5e00a00640 1 -- 172.21.15.12:0/1787579428 shutdown_connections 2024-03-23T23:06:21.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.634+0000 7f5e00a00640 1 -- 172.21.15.12:0/1787579428 wait complete. 2024-03-23T23:06:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.635+0000 7f5e00a00640 1 Processor -- start 2024-03-23T23:06:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.635+0000 7f5e00a00640 1 -- start start 2024-03-23T23:06:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.635+0000 7f5e00a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5dfc1037c0 0x7f5dfc199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.635+0000 7f5e00a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5dfc1049b0 0x7f5dfc199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.635+0000 7f5e00a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dfc105370 0x7f5dfc1a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.635+0000 7f5e00a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc10f420 con 0x7f5dfc105370 2024-03-23T23:06:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.635+0000 7f5e00a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc10f2a0 con 0x7f5dfc1049b0 2024-03-23T23:06:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.635+0000 7f5e00a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc10f5a0 con 0x7f5dfc1037c0 2024-03-23T23:06:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.636+0000 7f5dfaa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5dfc1049b0 0x7f5dfc199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.636+0000 7f5dfb400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5dfc1037c0 0x7f5dfc199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.636+0000 7f5dfaa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5dfc1049b0 0x7f5dfc199f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60682/0 (socket says 172.21.15.12:60682) 2024-03-23T23:06:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.636+0000 7f5dfaa00640 1 -- 172.21.15.12:0/663730017 learned_addr learned my addr 172.21.15.12:0/663730017 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.636+0000 7f5dfbe00640 1 --2- 172.21.15.12:0/663730017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dfc105370 0x7f5dfc1a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.636+0000 7f5dfaa00640 1 -- 172.21.15.12:0/663730017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5dfc1037c0 msgr2=0x7f5dfc199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.636+0000 7f5dfaa00640 1 --2- 172.21.15.12:0/663730017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5dfc1037c0 0x7f5dfc199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.636+0000 7f5dfaa00640 1 -- 172.21.15.12:0/663730017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dfc105370 msgr2=0x7f5dfc1a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.636+0000 7f5dfaa00640 1 --2- 172.21.15.12:0/663730017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dfc105370 0x7f5dfc1a0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.636+0000 7f5dfaa00640 1 -- 172.21.15.12:0/663730017 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dfc1a1660 con 0x7f5dfc1049b0 2024-03-23T23:06:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.636+0000 7f5dfbe00640 1 --2- 172.21.15.12:0/663730017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dfc105370 0x7f5dfc1a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:06:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.636+0000 7f5dfb400640 1 --2- 172.21.15.12:0/663730017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5dfc1037c0 0x7f5dfc199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.636+0000 7f5dfaa00640 1 --2- 172.21.15.12:0/663730017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5dfc1049b0 0x7f5dfc199f80 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f5de400e9f0 tx=0x7f5de400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.637+0000 7f5df3e00640 1 -- 172.21.15.12:0/663730017 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de400cd80 con 0x7f5dfc1049b0 2024-03-23T23:06:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.637+0000 7f5df3e00640 1 -- 172.21.15.12:0/663730017 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5de4010910 con 0x7f5dfc1049b0 2024-03-23T23:06:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.637+0000 7f5e00a00640 1 -- 172.21.15.12:0/663730017 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dfc1a1950 con 0x7f5dfc1049b0 2024-03-23T23:06:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.637+0000 7f5df3e00640 1 -- 172.21.15.12:0/663730017 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de4018990 con 0x7f5dfc1049b0 2024-03-23T23:06:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.637+0000 7f5e00a00640 1 -- 172.21.15.12:0/663730017 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dfc1a1ef0 con 0x7f5dfc1049b0 2024-03-23T23:06:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.640+0000 7f5e04e35740 1 Processor -- start 2024-03-23T23:06:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.640+0000 7f5df3e00640 1 -- 172.21.15.12:0/663730017 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5de4010ac0 con 0x7f5dfc1049b0 2024-03-23T23:06:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.640+0000 7f5e04e35740 1 -- start start 2024-03-23T23:06:21.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.640+0000 7f5e04e35740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fcc5050 0x558f5fcc5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:21.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.641+0000 7f5e04e35740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5fcc5a40 0x558f5fcca540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:21.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.641+0000 7f5e04e35740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f5fccabb0 0x558f5fccd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:21.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.641+0000 7f5e04e35740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f5fc2a250 con 0x558f5fccabb0 2024-03-23T23:06:21.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.641+0000 7f5e04e35740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f5fbd2a30 con 0x558f5fcc5a40 2024-03-23T23:06:21.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.641+0000 7f5e04e35740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f5fc262f0 con 0x558f5fcc5050 2024-03-23T23:06:21.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.641+0000 7f5dfbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fcc5050 0x558f5fcc5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:21.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.641+0000 7f5dfbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fcc5050 0x558f5fcc5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55370/0 (socket says 172.21.15.12:55370) 2024-03-23T23:06:21.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.641+0000 7f5dfbe00640 1 -- 172.21.15.12:0/1371723785 learned_addr learned my addr 172.21.15.12:0/1371723785 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:21.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.641+0000 7f5dfaa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f5fccabb0 0x558f5fccd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:21.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.641+0000 7f5dfbe00640 1 -- 172.21.15.12:0/1371723785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5fcc5a40 msgr2=0x558f5fcca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:21.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.641+0000 7f5dfbe00640 1 --2- 172.21.15.12:0/1371723785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5fcc5a40 0x558f5fcca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.641+0000 7f5dfbe00640 1 -- 172.21.15.12:0/1371723785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f5fccabb0 msgr2=0x558f5fccd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:21.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.641+0000 7f5dfbe00640 1 --2- 172.21.15.12:0/1371723785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f5fccabb0 0x558f5fccd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.641+0000 7f5dfbe00640 1 -- 172.21.15.12:0/1371723785 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f5fbbd690 con 0x558f5fcc5050 2024-03-23T23:06:21.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.641+0000 7f5dfaa00640 1 --2- 172.21.15.12:0/1371723785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f5fccabb0 0x558f5fccd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:06:21.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.642+0000 7f5dfbe00640 1 --2- 172.21.15.12:0/1371723785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fcc5050 0x558f5fcc5470 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5df4009ca0 tx=0x7f5df4009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=583c49705b05ff5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:21.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.642+0000 7f5debe00640 1 -- 172.21.15.12:0/1371723785 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df4043020 con 0x558f5fcc5050 2024-03-23T23:06:21.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.642+0000 7f5df3e00640 1 --2- 172.21.15.12:0/663730017 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5de00a2880 0x7f5de00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:21.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.642+0000 7f5debe00640 1 -- 172.21.15.12:0/1371723785 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5df4003970 con 0x558f5fcc5050 2024-03-23T23:06:21.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.642+0000 7f5debe00640 1 -- 172.21.15.12:0/1371723785 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df4039040 con 0x558f5fcc5050 2024-03-23T23:06:21.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.642+0000 7f5df3e00640 1 -- 172.21.15.12:0/663730017 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5de40cee10 con 0x7f5dfc1049b0 2024-03-23T23:06:21.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.642+0000 7f5dfb400640 1 --2- 172.21.15.12:0/663730017 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5de00a2880 0x7f5de00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:21.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.643+0000 7f5e04e35740 1 -- 172.21.15.12:0/1371723785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fcc5050 msgr2=0x558f5fcc5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:21.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.643+0000 7f5e04e35740 1 --2- 172.21.15.12:0/1371723785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fcc5050 0x558f5fcc5470 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5df4009ca0 tx=0x7f5df4009800 comp rx=0 tx=0).stop 2024-03-23T23:06:21.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.643+0000 7f5e04e35740 1 -- 172.21.15.12:0/1371723785 shutdown_connections 2024-03-23T23:06:21.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.643+0000 7f5e04e35740 1 --2- 172.21.15.12:0/1371723785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f5fccabb0 0x558f5fccd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.643+0000 7f5e04e35740 1 --2- 172.21.15.12:0/1371723785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5fcc5a40 0x558f5fcca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.643+0000 7f5e04e35740 1 --2- 172.21.15.12:0/1371723785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fcc5050 0x558f5fcc5470 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.643+0000 7f5e04e35740 1 -- 172.21.15.12:0/1371723785 >> 172.21.15.12:0/1371723785 conn(0x558f5fc8d200 msgr2=0x558f5fcc09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:21.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.643+0000 7f5dfb400640 1 --2- 172.21.15.12:0/663730017 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5de00a2880 0x7f5de00a4d40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x558f5fcc5e40 tx=0x7f5dec009c30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:21.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.643+0000 7f5e04e35740 1 -- 172.21.15.12:0/1371723785 shutdown_connections 2024-03-23T23:06:21.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.643+0000 7f5e04e35740 1 -- 172.21.15.12:0/1371723785 wait complete. 2024-03-23T23:06:21.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.643+0000 7f5e04e35740 1 -- start start 2024-03-23T23:06:21.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.644+0000 7f5e04e35740 1 Processor -- start 2024-03-23T23:06:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.644+0000 7f5e04e35740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fcc5050 0x558f5fccf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.644+0000 7f5e04e35740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f5fcc5a40 0x558f5fccf570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.644+0000 7f5e04e35740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5fccabb0 0x558f5fcd65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.644+0000 7f5e04e35740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f5fccdb00 con 0x558f5fcc5a40 2024-03-23T23:06:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.644+0000 7f5e04e35740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f5fbe5e80 con 0x558f5fccabb0 2024-03-23T23:06:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.644+0000 7f5e04e35740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f5fccdc80 con 0x558f5fcc5050 2024-03-23T23:06:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.644+0000 7f5e04e35740 10 client.0 _ll_register_callbacks cb 0x558f5fbf8130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.644+0000 7f5dfb400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fcc5050 0x558f5fccf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.644+0000 7f5dfaa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f5fcc5a40 0x558f5fccf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.644+0000 7f5dfb400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fcc5050 0x558f5fccf030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55376/0 (socket says 172.21.15.12:55376) 2024-03-23T23:06:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.645+0000 7f5dfb400640 1 -- 172.21.15.12:0/3805148113 learned_addr learned my addr 172.21.15.12:0/3805148113 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.644+0000 7f5dfaa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f5fcc5a40 0x558f5fccf570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37926/0 (socket says 172.21.15.12:37926) 2024-03-23T23:06:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.645+0000 7f5dfbe00640 1 --2- 172.21.15.12:0/3805148113 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5fccabb0 0x558f5fcd65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.645+0000 7f5dfb400640 1 -- 172.21.15.12:0/3805148113 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5fccabb0 msgr2=0x558f5fcd65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.645+0000 7f5dfb400640 1 --2- 172.21.15.12:0/3805148113 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5fccabb0 0x558f5fcd65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.645+0000 7f5dfb400640 1 -- 172.21.15.12:0/3805148113 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f5fcc5a40 msgr2=0x558f5fccf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.645+0000 7f5dfb400640 1 --2- 172.21.15.12:0/3805148113 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f5fcc5a40 0x558f5fccf570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.645+0000 7f5dfb400640 1 -- 172.21.15.12:0/3805148113 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558f5fa087d0 con 0x558f5fcc5050 2024-03-23T23:06:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.645+0000 7f5dfbe00640 1 --2- 172.21.15.12:0/3805148113 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5fccabb0 0x558f5fcd65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.645+0000 7f5dfaa00640 1 --2- 172.21.15.12:0/3805148113 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f5fcc5a40 0x558f5fccf570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.645+0000 7f5dfb400640 1 --2- 172.21.15.12:0/3805148113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fcc5050 0x558f5fccf030 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5dec01ba20 tx=0x7f5dec005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d9fceec9715da05 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.645+0000 7f5de8c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:06:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.646+0000 7f5de8c00640 1 -- 172.21.15.12:0/3805148113 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dec02c060 con 0x558f5fcc5050 2024-03-23T23:06:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.646+0000 7f5de8c00640 1 -- 172.21.15.12:0/3805148113 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5dec01d8a0 con 0x558f5fcc5050 2024-03-23T23:06:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.646+0000 7f5e04e35740 1 -- 172.21.15.12:0/3805148113 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5df4003970 con 0x558f5fcc5050 2024-03-23T23:06:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.646+0000 7f5de8c00640 1 -- 172.21.15.12:0/3805148113 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5dec01daa0 con 0x558f5fcc5050 2024-03-23T23:06:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.646+0000 7f5e04e35740 10 client.26422 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.646+0000 7f5e04e35740 1 -- 172.21.15.12:0/3805148113 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x558f5f93e3f0 con 0x558f5fcc5050 2024-03-23T23:06:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.646+0000 7f5de8c00640 1 -- 172.21.15.12:0/3805148113 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5df4003970 con 0x558f5fcc5050 2024-03-23T23:06:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.646+0000 7f5de8c00640 1 -- 172.21.15.12:0/3805148113 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f5dec0323d0 con 0x558f5fcc5050 2024-03-23T23:06:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.646+0000 7f5e04e35740 10 client.26422 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.646+0000 7f5e04e35740 10 client.26422 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:06:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.647+0000 7f5e04e35740 20 client.26422 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.647+0000 7f5e04e35740 1 --2- 172.21.15.12:0/3805148113 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558f5fcdaed0 0x558f5fcdd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.647+0000 7f5e04e35740 4 client.26422 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:06:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.647+0000 7f5e04e35740 1 -- 172.21.15.12:0/3805148113 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558f5fc37790 con 0x558f5fcdaed0 2024-03-23T23:06:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.647+0000 7f5dfaa00640 1 --2- 172.21.15.12:0/3805148113 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558f5fcdaed0 0x558f5fcdd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.647+0000 7f5dfaa00640 1 --2- 172.21.15.12:0/3805148113 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558f5fcdaed0 0x558f5fcdd330 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b88fb18ce182730 server_cookie=9ac7586d3fe20276 in_seq=0 out_seq=0 2024-03-23T23:06:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.647+0000 7f5de8c00640 10 client.26422 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.648+0000 7f5de8c00640 1 -- 172.21.15.12:0/3805148113 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558f5fc37790 con 0x558f5fcdaed0 2024-03-23T23:06:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.648+0000 7f5de8c00640 10 client.26422 handle_command_reply: tid=0 2024-03-23T23:06:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.648+0000 7f5e04e35740 1 client.26422 shutdown 2024-03-23T23:06:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.648+0000 7f5e04e35740 1 -- 172.21.15.12:0/3805148113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fcc5050 msgr2=0x558f5fccf030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:21.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.648+0000 7f5e04e35740 1 --2- 172.21.15.12:0/3805148113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fcc5050 0x558f5fccf030 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5dec01ba20 tx=0x7f5dec005f90 comp rx=0 tx=0).stop 2024-03-23T23:06:21.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.648+0000 7f5e04e35740 1 -- 172.21.15.12:0/3805148113 shutdown_connections 2024-03-23T23:06:21.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.648+0000 7f5e04e35740 1 --2- 172.21.15.12:0/3805148113 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558f5fcdaed0 0x558f5fcdd330 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.648+0000 7f5e04e35740 1 --2- 172.21.15.12:0/3805148113 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5fccabb0 0x558f5fcd65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.648+0000 7f5e04e35740 1 --2- 172.21.15.12:0/3805148113 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f5fcc5a40 0x558f5fccf570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.648+0000 7f5e04e35740 1 --2- 172.21.15.12:0/3805148113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fcc5050 0x558f5fccf030 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.648+0000 7f5de8c00640 0 client.26422 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:21.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.648+0000 7f5e04e35740 1 -- 172.21.15.12:0/3805148113 >> 172.21.15.12:0/3805148113 conn(0x558f5fc8d200 msgr2=0x558f5fcbfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:21.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.649+0000 7f5e04e35740 1 -- 172.21.15.12:0/3805148113 shutdown_connections 2024-03-23T23:06:21.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.649+0000 7f5e04e35740 1 -- 172.21.15.12:0/3805148113 wait complete. 2024-03-23T23:06:21.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.649+0000 7f5e04e35740 20 client.26422 trim_cache size 0 max 16384 2024-03-23T23:06:21.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.669+0000 7f5e04e35740 1 Processor -- start 2024-03-23T23:06:21.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.669+0000 7f5e04e35740 1 -- start start 2024-03-23T23:06:21.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.670+0000 7f5e04e35740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f5fcc5050 0x558f5fcc3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:21.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.670+0000 7f5e04e35740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5fcc5a40 0x558f5fcd6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:21.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.670+0000 7f5e04e35740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fccabb0 0x558f5fcc4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:21.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.670+0000 7f5e04e35740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc1acfa0 con 0x558f5fcc5050 2024-03-23T23:06:21.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.670+0000 7f5e04e35740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc1aaaa0 con 0x558f5fcc5a40 2024-03-23T23:06:21.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.670+0000 7f5e04e35740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc1a8b40 con 0x558f5fccabb0 2024-03-23T23:06:21.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.670+0000 7f5dfaa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5fcc5a40 0x558f5fcd6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:21.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.670+0000 7f5dfaa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5fcc5a40 0x558f5fcd6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60718/0 (socket says 172.21.15.12:60718) 2024-03-23T23:06:21.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.670+0000 7f5dfaa00640 1 -- 172.21.15.12:0/1313248878 learned_addr learned my addr 172.21.15.12:0/1313248878 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:21.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.670+0000 7f5dfbe00640 1 --2- 172.21.15.12:0/1313248878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fccabb0 0x558f5fcc4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:21.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.670+0000 7f5dfb400640 1 --2- 172.21.15.12:0/1313248878 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f5fcc5050 0x558f5fcc3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:21.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.670+0000 7f5dfaa00640 1 -- 172.21.15.12:0/1313248878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fccabb0 msgr2=0x558f5fcc4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:21.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.671+0000 7f5dfaa00640 1 --2- 172.21.15.12:0/1313248878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fccabb0 0x558f5fcc4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.671+0000 7f5dfaa00640 1 -- 172.21.15.12:0/1313248878 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f5fcc5050 msgr2=0x558f5fcc3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:21.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.671+0000 7f5dfaa00640 1 --2- 172.21.15.12:0/1313248878 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f5fcc5050 0x558f5fcc3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.671+0000 7f5dfaa00640 1 -- 172.21.15.12:0/1313248878 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f5f8fe0f0 con 0x558f5fcc5a40 2024-03-23T23:06:21.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.671+0000 7f5dfbe00640 1 --2- 172.21.15.12:0/1313248878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fccabb0 0x558f5fcc4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:21.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.671+0000 7f5dfaa00640 1 --2- 172.21.15.12:0/1313248878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5fcc5a40 0x558f5fcd6570 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f5de40980e0 tx=0x7f5de40d1260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=288b2cfaf0401e71 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:21.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.671+0000 7f5deb400640 1 -- 172.21.15.12:0/1313248878 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de409f8b0 con 0x558f5fcc5a40 2024-03-23T23:06:21.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.671+0000 7f5deb400640 1 -- 172.21.15.12:0/1313248878 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5de40a3f00 con 0x558f5fcc5a40 2024-03-23T23:06:21.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.671+0000 7f5deb400640 1 -- 172.21.15.12:0/1313248878 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de4018e20 con 0x558f5fcc5a40 2024-03-23T23:06:21.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.672+0000 7f5e04e35740 1 -- 172.21.15.12:0/1313248878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5fcc5a40 msgr2=0x558f5fcd6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:21.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.672+0000 7f5e04e35740 1 --2- 172.21.15.12:0/1313248878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5fcc5a40 0x558f5fcd6570 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f5de40980e0 tx=0x7f5de40d1260 comp rx=0 tx=0).stop 2024-03-23T23:06:21.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.672+0000 7f5e04e35740 1 -- 172.21.15.12:0/1313248878 shutdown_connections 2024-03-23T23:06:21.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.672+0000 7f5e04e35740 1 --2- 172.21.15.12:0/1313248878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fccabb0 0x558f5fcc4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.672+0000 7f5e04e35740 1 --2- 172.21.15.12:0/1313248878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5fcc5a40 0x558f5fcd6570 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.672+0000 7f5e04e35740 1 --2- 172.21.15.12:0/1313248878 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f5fcc5050 0x558f5fcc3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.672+0000 7f5e04e35740 1 -- 172.21.15.12:0/1313248878 >> 172.21.15.12:0/1313248878 conn(0x558f5fc8d200 msgr2=0x558f5fcc6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:21.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.672+0000 7f5e04e35740 1 -- 172.21.15.12:0/1313248878 shutdown_connections 2024-03-23T23:06:21.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.672+0000 7f5e04e35740 1 -- 172.21.15.12:0/1313248878 wait complete. 2024-03-23T23:06:21.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.672+0000 7f5e04e35740 1 -- start start 2024-03-23T23:06:21.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.673+0000 7f5e04e35740 1 Processor -- start 2024-03-23T23:06:21.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.673+0000 7f5e04e35740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5fcc5050 0x558f5fcd4130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:21.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.673+0000 7f5e04e35740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fcc5a40 0x558f5fcd4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:21.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.673+0000 7f5e04e35740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f5fccabb0 0x558f5fcd5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:21.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.673+0000 7f5e04e35740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc1a9140 con 0x558f5fccabb0 2024-03-23T23:06:21.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.673+0000 7f5e04e35740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc1aa440 con 0x558f5fcc5050 2024-03-23T23:06:21.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.673+0000 7f5e04e35740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc1a9780 con 0x558f5fcc5a40 2024-03-23T23:06:21.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.673+0000 7f5e04e35740 10 client.0 _ll_register_callbacks cb 0x7f5dec01b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:21.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.673+0000 7f5dfb400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5fcc5050 0x558f5fcd4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:21.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.673+0000 7f5dfaa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fcc5a40 0x558f5fcd4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:21.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.673+0000 7f5dfbe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f5fccabb0 0x558f5fcd5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:21.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.673+0000 7f5dfb400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5fcc5050 0x558f5fcd4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60724/0 (socket says 172.21.15.12:60724) 2024-03-23T23:06:21.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.673+0000 7f5dfaa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fcc5a40 0x558f5fcd4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55402/0 (socket says 172.21.15.12:55402) 2024-03-23T23:06:21.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.673+0000 7f5dfb400640 1 -- 172.21.15.12:0/2731164002 learned_addr learned my addr 172.21.15.12:0/2731164002 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:21.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.674+0000 7f5dfaa00640 1 -- 172.21.15.12:0/2731164002 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5fcc5050 msgr2=0x558f5fcd4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:21.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.674+0000 7f5dfaa00640 1 --2- 172.21.15.12:0/2731164002 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5fcc5050 0x558f5fcd4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.674+0000 7f5dfaa00640 1 -- 172.21.15.12:0/2731164002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f5fccabb0 msgr2=0x558f5fcd5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:21.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.674+0000 7f5dfaa00640 1 --2- 172.21.15.12:0/2731164002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f5fccabb0 0x558f5fcd5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.674+0000 7f5dfaa00640 1 -- 172.21.15.12:0/2731164002 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558f5fcd5c70 con 0x558f5fcc5a40 2024-03-23T23:06:21.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.674+0000 7f5dfaa00640 1 --2- 172.21.15.12:0/2731164002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fcc5a40 0x558f5fcd4670 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5de4098130 tx=0x7f5de409b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9e5cf4755824d0b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:21.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.674+0000 7f5de8c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:06:21.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.674+0000 7f5de8c00640 1 -- 172.21.15.12:0/2731164002 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de40af060 con 0x558f5fcc5a40 2024-03-23T23:06:21.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.674+0000 7f5de8c00640 1 -- 172.21.15.12:0/2731164002 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5de4018ba0 con 0x558f5fcc5a40 2024-03-23T23:06:21.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.674+0000 7f5e04e35740 1 -- 172.21.15.12:0/2731164002 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5de40a3f00 con 0x558f5fcc5a40 2024-03-23T23:06:21.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.674+0000 7f5de8c00640 1 -- 172.21.15.12:0/2731164002 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5de409f8b0 con 0x558f5fcc5a40 2024-03-23T23:06:21.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.674+0000 7f5e04e35740 10 client.26428 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:21.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.674+0000 7f5e04e35740 1 -- 172.21.15.12:0/2731164002 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x558f5fd36d00 con 0x558f5fcc5a40 2024-03-23T23:06:21.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.675+0000 7f5de8c00640 1 -- 172.21.15.12:0/2731164002 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5de40a3f00 con 0x558f5fcc5a40 2024-03-23T23:06:21.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.675+0000 7f5de8c00640 1 -- 172.21.15.12:0/2731164002 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f5de409ac40 con 0x558f5fcc5a40 2024-03-23T23:06:21.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.675+0000 7f5e04e35740 10 client.26428 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:21.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.675+0000 7f5e04e35740 10 client.26428 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:06:21.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.675+0000 7f5e04e35740 20 client.26428 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:21.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.675+0000 7f5e04e35740 1 --2- 172.21.15.12:0/2731164002 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558f5fd37590 0x558f5fd399b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:21.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.675+0000 7f5e04e35740 4 client.26428 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:06:21.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.675+0000 7f5e04e35740 1 -- 172.21.15.12:0/2731164002 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558f5fc31ae0 con 0x558f5fd37590 2024-03-23T23:06:21.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.675+0000 7f5dfb400640 1 --2- 172.21.15.12:0/2731164002 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558f5fd37590 0x558f5fd399b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:21.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.676+0000 7f5dfb400640 1 --2- 172.21.15.12:0/2731164002 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558f5fd37590 0x558f5fd399b0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b9151e214e95360 server_cookie=f020801a0af61df6 in_seq=0 out_seq=0 2024-03-23T23:06:21.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.676+0000 7f5de8c00640 10 client.26428 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:21.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.676+0000 7f5de8c00640 1 -- 172.21.15.12:0/2731164002 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x558f5fc31ae0 con 0x558f5fd37590 2024-03-23T23:06:21.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.676+0000 7f5de8c00640 10 client.26428 handle_command_reply: tid=0 2024-03-23T23:06:21.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.676+0000 7f5e04e35740 1 client.26428 shutdown 2024-03-23T23:06:21.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.676+0000 7f5e04e35740 1 -- 172.21.15.12:0/2731164002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fcc5a40 msgr2=0x558f5fcd4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:21.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.676+0000 7f5e04e35740 1 --2- 172.21.15.12:0/2731164002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fcc5a40 0x558f5fcd4670 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5de4098130 tx=0x7f5de409b8e0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.676+0000 7f5e04e35740 1 -- 172.21.15.12:0/2731164002 shutdown_connections 2024-03-23T23:06:21.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.676+0000 7f5e04e35740 1 --2- 172.21.15.12:0/2731164002 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558f5fd37590 0x558f5fd399b0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.676+0000 7f5e04e35740 1 --2- 172.21.15.12:0/2731164002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f5fccabb0 0x558f5fcd5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.676+0000 7f5e04e35740 1 --2- 172.21.15.12:0/2731164002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f5fcc5a40 0x558f5fcd4670 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.676+0000 7f5e04e35740 1 --2- 172.21.15.12:0/2731164002 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5fcc5050 0x558f5fcd4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.676+0000 7f5de8c00640 0 client.26428 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:21.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.676+0000 7f5e04e35740 1 -- 172.21.15.12:0/2731164002 >> 172.21.15.12:0/2731164002 conn(0x558f5fc8d200 msgr2=0x558f5fcbf330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:21.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.676+0000 7f5e04e35740 1 -- 172.21.15.12:0/2731164002 shutdown_connections 2024-03-23T23:06:21.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.676+0000 7f5e04e35740 1 -- 172.21.15.12:0/2731164002 wait complete. 2024-03-23T23:06:21.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.676+0000 7f5e04e35740 20 client.26428 trim_cache size 0 max 16384 2024-03-23T23:06:21.681 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:06:21.681 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-03-23T23:06:21.681 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-03-23T23:06:21.681 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:06:21.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.677+0000 7f5e00a00640 1 -- 172.21.15.12:0/663730017 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5de00a2880 msgr2=0x7f5de00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:21.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.677+0000 7f5e00a00640 1 --2- 172.21.15.12:0/663730017 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5de00a2880 0x7f5de00a4d40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x558f5fcc5e40 tx=0x7f5dec009c30 comp rx=0 tx=0).stop 2024-03-23T23:06:21.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.677+0000 7f5e00a00640 1 -- 172.21.15.12:0/663730017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5dfc1049b0 msgr2=0x7f5dfc199f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:21.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.677+0000 7f5e00a00640 1 --2- 172.21.15.12:0/663730017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5dfc1049b0 0x7f5dfc199f80 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f5de400e9f0 tx=0x7f5de400eeb0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.677+0000 7f5e00a00640 1 -- 172.21.15.12:0/663730017 shutdown_connections 2024-03-23T23:06:21.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.677+0000 7f5e00a00640 1 --2- 172.21.15.12:0/663730017 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5de00a2880 0x7f5de00a4d40 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.677+0000 7f5e00a00640 1 --2- 172.21.15.12:0/663730017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dfc105370 0x7f5dfc1a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.677+0000 7f5e00a00640 1 --2- 172.21.15.12:0/663730017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5dfc1049b0 0x7f5dfc199f80 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.677+0000 7f5e00a00640 1 --2- 172.21.15.12:0/663730017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5dfc1037c0 0x7f5dfc199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.677+0000 7f5e00a00640 1 -- 172.21.15.12:0/663730017 >> 172.21.15.12:0/663730017 conn(0x7f5dfc0fed50 msgr2=0x7f5dfc107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:21.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.677+0000 7f5e00a00640 1 -- 172.21.15.12:0/663730017 shutdown_connections 2024-03-23T23:06:21.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.677+0000 7f5e00a00640 1 -- 172.21.15.12:0/663730017 wait complete. 2024-03-23T23:06:21.694 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T23:06:21.695 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:06:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.903+0000 7f2efee00640 1 -- 172.21.15.12:0/3714402285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ef8075680 msgr2=0x7f2ef8073ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.903+0000 7f2efee00640 1 --2- 172.21.15.12:0/3714402285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ef8075680 0x7f2ef8073ad0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2eec009900 tx=0x7f2eec02bcc0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.904+0000 7f2efee00640 1 -- 172.21.15.12:0/3714402285 shutdown_connections 2024-03-23T23:06:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.904+0000 7f2efee00640 1 --2- 172.21.15.12:0/3714402285 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ef8109390 0x7f2ef810b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.904+0000 7f2efee00640 1 --2- 172.21.15.12:0/3714402285 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ef80740a0 0x7f2ef8074500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.904+0000 7f2efee00640 1 --2- 172.21.15.12:0/3714402285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ef8075680 0x7f2ef8073ad0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.904+0000 7f2efee00640 1 -- 172.21.15.12:0/3714402285 >> 172.21.15.12:0/3714402285 conn(0x7f2ef80fca70 msgr2=0x7f2ef80feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:21.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.904+0000 7f2efee00640 1 -- 172.21.15.12:0/3714402285 shutdown_connections 2024-03-23T23:06:21.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.904+0000 7f2efee00640 1 -- 172.21.15.12:0/3714402285 wait complete. 2024-03-23T23:06:21.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.905+0000 7f2efee00640 1 Processor -- start 2024-03-23T23:06:21.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.905+0000 7f2efee00640 1 -- start start 2024-03-23T23:06:21.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.905+0000 7f2efee00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ef80740a0 0x7f2ef8199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:21.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.905+0000 7f2efee00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ef8075680 0x7f2ef8199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:21.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.905+0000 7f2efee00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ef8109390 0x7f2ef81a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:21.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.905+0000 7f2efee00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef810f420 con 0x7f2ef80740a0 2024-03-23T23:06:21.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.905+0000 7f2efee00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef810f2a0 con 0x7f2ef8109390 2024-03-23T23:06:21.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.905+0000 7f2efee00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef810f5a0 con 0x7f2ef8075680 2024-03-23T23:06:21.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.906+0000 7f2efda00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ef80740a0 0x7f2ef8199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.906+0000 7f2efe400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ef8109390 0x7f2ef81a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.906+0000 7f2efd000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ef8075680 0x7f2ef8199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.906+0000 7f2efda00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ef80740a0 0x7f2ef8199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37958/0 (socket says 172.21.15.12:37958) 2024-03-23T23:06:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.906+0000 7f2efe400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ef8109390 0x7f2ef81a1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60732/0 (socket says 172.21.15.12:60732) 2024-03-23T23:06:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.906+0000 7f2efda00640 1 -- 172.21.15.12:0/637741123 learned_addr learned my addr 172.21.15.12:0/637741123 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.906+0000 7f2efe400640 1 -- 172.21.15.12:0/637741123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ef8075680 msgr2=0x7f2ef8199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.906+0000 7f2efe400640 1 --2- 172.21.15.12:0/637741123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ef8075680 0x7f2ef8199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.906+0000 7f2efe400640 1 -- 172.21.15.12:0/637741123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ef80740a0 msgr2=0x7f2ef8199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.906+0000 7f2efe400640 1 --2- 172.21.15.12:0/637741123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ef80740a0 0x7f2ef8199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.906+0000 7f2efe400640 1 -- 172.21.15.12:0/637741123 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef81a1690 con 0x7f2ef8109390 2024-03-23T23:06:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.906+0000 7f2efda00640 1 --2- 172.21.15.12:0/637741123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ef80740a0 0x7f2ef8199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.906+0000 7f2efd000640 1 --2- 172.21.15.12:0/637741123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ef8075680 0x7f2ef8199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.907+0000 7f2efe400640 1 --2- 172.21.15.12:0/637741123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ef8109390 0x7f2ef81a1020 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2ef000da10 tx=0x7f2ef000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.907+0000 7f2ef6000640 1 -- 172.21.15.12:0/637741123 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef000bd10 con 0x7f2ef8109390 2024-03-23T23:06:21.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.907+0000 7f2ef6000640 1 -- 172.21.15.12:0/637741123 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ef0003e50 con 0x7f2ef8109390 2024-03-23T23:06:21.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.907+0000 7f2efee00640 1 -- 172.21.15.12:0/637741123 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef81a1980 con 0x7f2ef8109390 2024-03-23T23:06:21.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.907+0000 7f2ef6000640 1 -- 172.21.15.12:0/637741123 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef00106c0 con 0x7f2ef8109390 2024-03-23T23:06:21.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.907+0000 7f2efee00640 1 -- 172.21.15.12:0/637741123 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef81a1e80 con 0x7f2ef8109390 2024-03-23T23:06:21.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.908+0000 7f2efee00640 1 -- 172.21.15.12:0/637741123 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef8108150 con 0x7f2ef8109390 2024-03-23T23:06:21.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.910+0000 7f2ef6000640 1 -- 172.21.15.12:0/637741123 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2ef0003970 con 0x7f2ef8109390 2024-03-23T23:06:21.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.912+0000 7f2ef6000640 1 --2- 172.21.15.12:0/637741123 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ee00a2800 0x7f2ee00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:21.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.912+0000 7f2ef6000640 1 -- 172.21.15.12:0/637741123 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2ef00cee10 con 0x7f2ef8109390 2024-03-23T23:06:21.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.912+0000 7f2efda00640 1 --2- 172.21.15.12:0/637741123 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ee00a2800 0x7f2ee00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:21.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.913+0000 7f2efda00640 1 --2- 172.21.15.12:0/637741123 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ee00a2800 0x7f2ee00a4cc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2eec006010 tx=0x7f2eec02c910 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:21.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:21.919+0000 7f2ef6000640 1 -- 172.21.15.12:0/637741123 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2ef00d2030 con 0x7f2ef8109390 2024-03-23T23:06:22.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.162+0000 7f2efee00640 1 -- 172.21.15.12:0/637741123 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ef802d070 con 0x7f2ef8109390 2024-03-23T23:06:22.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.164+0000 7f2ef6000640 1 -- 172.21.15.12:0/637741123 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f2ef009be60 con 0x7f2ef8109390 2024-03-23T23:06:22.163 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:06:22.163 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:06:22.163 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:06:22.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.166+0000 7f2efee00640 1 -- 172.21.15.12:0/637741123 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ee00a2800 msgr2=0x7f2ee00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.166+0000 7f2efee00640 1 --2- 172.21.15.12:0/637741123 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ee00a2800 0x7f2ee00a4cc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2eec006010 tx=0x7f2eec02c910 comp rx=0 tx=0).stop 2024-03-23T23:06:22.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.166+0000 7f2efee00640 1 -- 172.21.15.12:0/637741123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ef8109390 msgr2=0x7f2ef81a1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.167+0000 7f2efee00640 1 --2- 172.21.15.12:0/637741123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ef8109390 0x7f2ef81a1020 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2ef000da10 tx=0x7f2ef000de10 comp rx=0 tx=0).stop 2024-03-23T23:06:22.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.167+0000 7f2efee00640 1 -- 172.21.15.12:0/637741123 shutdown_connections 2024-03-23T23:06:22.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.167+0000 7f2efee00640 1 --2- 172.21.15.12:0/637741123 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ee00a2800 0x7f2ee00a4cc0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.167+0000 7f2efee00640 1 --2- 172.21.15.12:0/637741123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ef8109390 0x7f2ef81a1020 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.167+0000 7f2efee00640 1 --2- 172.21.15.12:0/637741123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ef8075680 0x7f2ef8199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.167+0000 7f2efee00640 1 --2- 172.21.15.12:0/637741123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ef80740a0 0x7f2ef8199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.167+0000 7f2efee00640 1 -- 172.21.15.12:0/637741123 >> 172.21.15.12:0/637741123 conn(0x7f2ef80fca70 msgr2=0x7f2ef8105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:22.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.167+0000 7f2efee00640 1 -- 172.21.15.12:0/637741123 shutdown_connections 2024-03-23T23:06:22.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.167+0000 7f2efee00640 1 -- 172.21.15.12:0/637741123 wait complete. 2024-03-23T23:06:22.186 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:06:22.186 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-23T23:06:22.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.356+0000 7f0315800640 1 -- 172.21.15.12:0/2956587498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0310102170 msgr2=0x7f0310108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.356+0000 7f0315800640 1 --2- 172.21.15.12:0/2956587498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0310102170 0x7f0310108a00 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0308009be0 tx=0x7f030802bef0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.356+0000 7f0315800640 1 -- 172.21.15.12:0/2956587498 shutdown_connections 2024-03-23T23:06:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.356+0000 7f0315800640 1 --2- 172.21.15.12:0/2956587498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0310102170 0x7f0310108a00 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.356+0000 7f0315800640 1 --2- 172.21.15.12:0/2956587498 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03101017b0 0x7f0310101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.356+0000 7f0315800640 1 --2- 172.21.15.12:0/2956587498 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03101005c0 0x7f03101009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.356+0000 7f0315800640 1 -- 172.21.15.12:0/2956587498 >> 172.21.15.12:0/2956587498 conn(0x7f03100fbb30 msgr2=0x7f03100fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.356+0000 7f0315800640 1 -- 172.21.15.12:0/2956587498 shutdown_connections 2024-03-23T23:06:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.357+0000 7f0315800640 1 -- 172.21.15.12:0/2956587498 wait complete. 2024-03-23T23:06:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.358+0000 7f0315800640 1 Processor -- start 2024-03-23T23:06:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.358+0000 7f0315800640 1 -- start start 2024-03-23T23:06:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.358+0000 7f0315800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03101005c0 0x7f0310197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.358+0000 7f0315800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03101017b0 0x7f0310197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.358+0000 7f0315800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0310102170 0x7f031019eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.358+0000 7f0315800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031010c2a0 con 0x7f0310102170 2024-03-23T23:06:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.358+0000 7f0315800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031010c120 con 0x7f03101005c0 2024-03-23T23:06:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.358+0000 7f0315800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031010c420 con 0x7f03101017b0 2024-03-23T23:06:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.359+0000 7f030ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03101005c0 0x7f0310197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.359+0000 7f030ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03101005c0 0x7f0310197940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60750/0 (socket says 172.21.15.12:60750) 2024-03-23T23:06:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.359+0000 7f030ea00640 1 -- 172.21.15.12:0/1578099628 learned_addr learned my addr 172.21.15.12:0/1578099628 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:22.359 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.359+0000 7f0307e00640 1 --2- 172.21.15.12:0/1578099628 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03101017b0 0x7f0310197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.359 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.359+0000 7f030f400640 1 --2- 172.21.15.12:0/1578099628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0310102170 0x7f031019eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.359 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.359+0000 7f030ea00640 1 -- 172.21.15.12:0/1578099628 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03101017b0 msgr2=0x7f0310197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.359 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.359+0000 7f030ea00640 1 --2- 172.21.15.12:0/1578099628 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03101017b0 0x7f0310197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.359 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.359+0000 7f030ea00640 1 -- 172.21.15.12:0/1578099628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0310102170 msgr2=0x7f031019eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.359 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.359+0000 7f030ea00640 1 --2- 172.21.15.12:0/1578099628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0310102170 0x7f031019eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.359 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.359+0000 7f030ea00640 1 -- 172.21.15.12:0/1578099628 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031019f560 con 0x7f03101005c0 2024-03-23T23:06:22.360 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.360+0000 7f030ea00640 1 --2- 172.21.15.12:0/1578099628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03101005c0 0x7f0310197940 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f030000e9f0 tx=0x7f030000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:22.360 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.360+0000 7f0307400640 1 -- 172.21.15.12:0/1578099628 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030000ccc0 con 0x7f03101005c0 2024-03-23T23:06:22.360 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.360+0000 7f0307400640 1 -- 172.21.15.12:0/1578099628 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f03000035f0 con 0x7f03101005c0 2024-03-23T23:06:22.360 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.360+0000 7f0315800640 1 -- 172.21.15.12:0/1578099628 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f031019f850 con 0x7f03101005c0 2024-03-23T23:06:22.360 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.360+0000 7f0307400640 1 -- 172.21.15.12:0/1578099628 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030000ce60 con 0x7f03101005c0 2024-03-23T23:06:22.360 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.360+0000 7f0315800640 1 -- 172.21.15.12:0/1578099628 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f031019fda0 con 0x7f03101005c0 2024-03-23T23:06:22.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.363+0000 7f0307400640 1 -- 172.21.15.12:0/1578099628 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f03000037a0 con 0x7f03101005c0 2024-03-23T23:06:22.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.363+0000 7f0318438740 1 Processor -- start 2024-03-23T23:06:22.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.364+0000 7f0318438740 1 -- start start 2024-03-23T23:06:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.364+0000 7f0318438740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b189c35f0 0x564b189c3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.364+0000 7f0318438740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564b189c3fe0 0x564b189c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.364+0000 7f0318438740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564b189c9150 0x564b189cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.364+0000 7f0318438740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b189823b0 con 0x564b189c9150 2024-03-23T23:06:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.364+0000 7f0318438740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b18924830 con 0x564b189c3fe0 2024-03-23T23:06:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.364+0000 7f0318438740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b188cd5d0 con 0x564b189c35f0 2024-03-23T23:06:22.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.364+0000 7f030ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564b189c3fe0 0x564b189c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.364+0000 7f030f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b189c35f0 0x564b189c3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.364+0000 7f030f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b189c35f0 0x564b189c3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55452/0 (socket says 172.21.15.12:55452) 2024-03-23T23:06:22.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.364+0000 7f030f400640 1 -- 172.21.15.12:0/1938101333 learned_addr learned my addr 172.21.15.12:0/1938101333 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:22.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.364+0000 7f0307e00640 1 --2- 172.21.15.12:0/1938101333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564b189c9150 0x564b189cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.365+0000 7f030f400640 1 -- 172.21.15.12:0/1938101333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564b189c3fe0 msgr2=0x564b189c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.365+0000 7f030f400640 1 --2- 172.21.15.12:0/1938101333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564b189c3fe0 0x564b189c8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.365+0000 7f030f400640 1 -- 172.21.15.12:0/1938101333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564b189c9150 msgr2=0x564b189cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.365+0000 7f030f400640 1 --2- 172.21.15.12:0/1938101333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564b189c9150 0x564b189cb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.365+0000 7f030f400640 1 -- 172.21.15.12:0/1938101333 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564b188e52c0 con 0x564b189c35f0 2024-03-23T23:06:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.365+0000 7f030f400640 1 --2- 172.21.15.12:0/1938101333 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b189c35f0 0x564b189c3a10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0308009be0 tx=0x7f0308009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d36eb2194325f13d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.366+0000 7f02ffe00640 1 -- 172.21.15.12:0/1938101333 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0308043070 con 0x564b189c35f0 2024-03-23T23:06:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.366+0000 7f02ffe00640 1 -- 172.21.15.12:0/1938101333 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0308033bc0 con 0x564b189c35f0 2024-03-23T23:06:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.366+0000 7f02ffe00640 1 -- 172.21.15.12:0/1938101333 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030802caf0 con 0x564b189c35f0 2024-03-23T23:06:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.366+0000 7f0307400640 1 --2- 172.21.15.12:0/1578099628 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f02f40a28a0 0x7f02f40a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.366+0000 7f030ea00640 1 --2- 172.21.15.12:0/1578099628 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f02f40a28a0 0x7f02f40a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.366+0000 7f0307400640 1 -- 172.21.15.12:0/1578099628 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f03000cedc0 con 0x7f03101005c0 2024-03-23T23:06:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.366+0000 7f0318438740 1 -- 172.21.15.12:0/1938101333 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b189c35f0 msgr2=0x564b189c3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.366+0000 7f0318438740 1 --2- 172.21.15.12:0/1938101333 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b189c35f0 0x564b189c3a10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0308009be0 tx=0x7f0308009310 comp rx=0 tx=0).stop 2024-03-23T23:06:22.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.367+0000 7f0318438740 1 -- 172.21.15.12:0/1938101333 shutdown_connections 2024-03-23T23:06:22.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.367+0000 7f0318438740 1 --2- 172.21.15.12:0/1938101333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564b189c9150 0x564b189cb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.367+0000 7f0318438740 1 --2- 172.21.15.12:0/1938101333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564b189c3fe0 0x564b189c8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.367+0000 7f0318438740 1 --2- 172.21.15.12:0/1938101333 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b189c35f0 0x564b189c3a10 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.367+0000 7f0318438740 1 -- 172.21.15.12:0/1938101333 >> 172.21.15.12:0/1938101333 conn(0x564b189ba750 msgr2=0x564b189bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.367+0000 7f0318438740 1 -- 172.21.15.12:0/1938101333 shutdown_connections 2024-03-23T23:06:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.367+0000 7f0318438740 1 -- 172.21.15.12:0/1938101333 wait complete. 2024-03-23T23:06:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.367+0000 7f030ea00640 1 --2- 172.21.15.12:0/1578099628 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f02f40a28a0 0x7f02f40a4d60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f030000e9c0 tx=0x7f03000d2800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.367+0000 7f0318438740 1 -- start start 2024-03-23T23:06:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.367+0000 7f0318438740 1 Processor -- start 2024-03-23T23:06:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.368+0000 7f0318438740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564b189c35f0 0x564b189cd720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.368+0000 7f0318438740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b189c3fe0 0x564b189cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.368+0000 7f0318438740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564b189c9150 0x564b189d4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.368+0000 7f0318438740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b188e0a20 con 0x564b189c35f0 2024-03-23T23:06:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.368+0000 7f0318438740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b189208d0 con 0x564b189c9150 2024-03-23T23:06:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.368+0000 7f0318438740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b189cc040 con 0x564b189c3fe0 2024-03-23T23:06:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.368+0000 7f0318438740 10 client.0 _ll_register_callbacks cb 0x564b1891d1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.368+0000 7f030f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564b189c35f0 0x564b189cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.368+0000 7f0307e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b189c3fe0 0x564b189cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.368+0000 7f0307e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b189c3fe0 0x564b189cdc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55458/0 (socket says 172.21.15.12:55458) 2024-03-23T23:06:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.368+0000 7f030f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564b189c35f0 0x564b189cd720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37992/0 (socket says 172.21.15.12:37992) 2024-03-23T23:06:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.368+0000 7f0307e00640 1 -- 172.21.15.12:0/315278582 learned_addr learned my addr 172.21.15.12:0/315278582 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.368+0000 7f030f400640 1 --2- 172.21.15.12:0/315278582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564b189c9150 0x564b189d4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.369+0000 7f0307e00640 1 -- 172.21.15.12:0/315278582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564b189c9150 msgr2=0x564b189d4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.369+0000 7f0307e00640 1 --2- 172.21.15.12:0/315278582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564b189c9150 0x564b189d4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.369+0000 7f0307e00640 1 -- 172.21.15.12:0/315278582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564b189c35f0 msgr2=0x564b189cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.369+0000 7f0307e00640 1 --2- 172.21.15.12:0/315278582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564b189c35f0 0x564b189cd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.369+0000 7f0307e00640 1 -- 172.21.15.12:0/315278582 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564b18639210 con 0x564b189c3fe0 2024-03-23T23:06:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.369+0000 7f030f400640 1 --2- 172.21.15.12:0/315278582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564b189c35f0 0x564b189cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:06:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.369+0000 7f0307e00640 1 --2- 172.21.15.12:0/315278582 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b189c3fe0 0x564b189cdc60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f02f8012720 tx=0x7f02f8012be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=182223fdf249bbd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.369+0000 7f02fcc00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:06:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.369+0000 7f02fcc00640 1 -- 172.21.15.12:0/315278582 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f800f8c0 con 0x564b189c3fe0 2024-03-23T23:06:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.369+0000 7f02fcc00640 1 -- 172.21.15.12:0/315278582 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f02f800fa60 con 0x564b189c3fe0 2024-03-23T23:06:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.369+0000 7f0318438740 1 -- 172.21.15.12:0/315278582 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0308033bc0 con 0x564b189c3fe0 2024-03-23T23:06:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.370+0000 7f02fcc00640 1 -- 172.21.15.12:0/315278582 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02f800fc60 con 0x564b189c3fe0 2024-03-23T23:06:22.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.370+0000 7f0318438740 10 client.26446 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:22.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.370+0000 7f0318438740 1 -- 172.21.15.12:0/315278582 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x564b18705620 con 0x564b189c3fe0 2024-03-23T23:06:22.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.370+0000 7f02fcc00640 1 -- 172.21.15.12:0/315278582 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0308033bc0 con 0x564b189c3fe0 2024-03-23T23:06:22.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.370+0000 7f02fcc00640 1 -- 172.21.15.12:0/315278582 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f02f801c420 con 0x564b189c3fe0 2024-03-23T23:06:22.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.370+0000 7f0318438740 10 client.26446 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:22.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.370+0000 7f0318438740 10 client.26446 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:06:22.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.370+0000 7f0318438740 20 client.26446 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:22.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.370+0000 7f0318438740 1 --2- 172.21.15.12:0/315278582 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x564b189d9610 0x564b189dba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.370+0000 7f0318438740 4 client.26446 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:06:22.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.370+0000 7f0318438740 1 -- 172.21.15.12:0/315278582 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564b18931d70 con 0x564b189d9610 2024-03-23T23:06:22.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.371+0000 7f030f400640 1 --2- 172.21.15.12:0/315278582 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x564b189d9610 0x564b189dba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.371+0000 7f030f400640 1 --2- 172.21.15.12:0/315278582 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x564b189d9610 0x564b189dba70 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c116536c3d327d0 server_cookie=bd973780af1fddf4 in_seq=0 out_seq=0 2024-03-23T23:06:22.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.371+0000 7f02fcc00640 10 client.26446 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.372+0000 7f02fcc00640 1 -- 172.21.15.12:0/315278582 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564b18931d70 con 0x564b189d9610 2024-03-23T23:06:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.372+0000 7f02fcc00640 10 client.26446 handle_command_reply: tid=0 2024-03-23T23:06:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.372+0000 7f0318438740 1 client.26446 shutdown 2024-03-23T23:06:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.372+0000 7f0318438740 1 -- 172.21.15.12:0/315278582 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b189c3fe0 msgr2=0x564b189cdc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.372+0000 7f0318438740 1 --2- 172.21.15.12:0/315278582 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b189c3fe0 0x564b189cdc60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f02f8012720 tx=0x7f02f8012be0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.372+0000 7f0318438740 1 -- 172.21.15.12:0/315278582 shutdown_connections 2024-03-23T23:06:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.372+0000 7f0318438740 1 --2- 172.21.15.12:0/315278582 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x564b189d9610 0x564b189dba70 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.372+0000 7f0318438740 1 --2- 172.21.15.12:0/315278582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564b189c9150 0x564b189d4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.372+0000 7f0318438740 1 --2- 172.21.15.12:0/315278582 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b189c3fe0 0x564b189cdc60 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.372+0000 7f0318438740 1 --2- 172.21.15.12:0/315278582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564b189c35f0 0x564b189cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.372+0000 7f0318438740 1 -- 172.21.15.12:0/315278582 >> 172.21.15.12:0/315278582 conn(0x564b189ba750 msgr2=0x564b189b9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.372+0000 7f02fcc00640 0 client.26446 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.372+0000 7f0318438740 1 -- 172.21.15.12:0/315278582 shutdown_connections 2024-03-23T23:06:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.372+0000 7f0318438740 1 -- 172.21.15.12:0/315278582 wait complete. 2024-03-23T23:06:22.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.373+0000 7f0318438740 20 client.26446 trim_cache size 0 max 16384 2024-03-23T23:06:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.388+0000 7f0318438740 1 Processor -- start 2024-03-23T23:06:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.388+0000 7f0318438740 1 -- start start 2024-03-23T23:06:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.388+0000 7f0318438740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564b189c35f0 0x564b189cd720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.388+0000 7f0318438740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564b189c3fe0 0x564b189d4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.388+0000 7f0318438740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b189c9150 0x564b189cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.388+0000 7f0318438740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03101a8d40 con 0x564b189c3fe0 2024-03-23T23:06:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.388+0000 7f0318438740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03101a6840 con 0x564b189c35f0 2024-03-23T23:06:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.388+0000 7f0318438740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03101a48e0 con 0x564b189c9150 2024-03-23T23:06:22.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.388+0000 7f030f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564b189c35f0 0x564b189cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.388+0000 7f0307e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564b189c3fe0 0x564b189d4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.388+0000 7f030f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564b189c35f0 0x564b189cd720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60778/0 (socket says 172.21.15.12:60778) 2024-03-23T23:06:22.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.388+0000 7f030f400640 1 -- 172.21.15.12:0/2623461939 learned_addr learned my addr 172.21.15.12:0/2623461939 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:22.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.388+0000 7f030f400640 1 --2- 172.21.15.12:0/2623461939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b189c9150 0x564b189cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.389+0000 7f0307e00640 1 -- 172.21.15.12:0/2623461939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b189c9150 msgr2=0x564b189cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.389+0000 7f0307e00640 1 --2- 172.21.15.12:0/2623461939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b189c9150 0x564b189cdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.389+0000 7f0307e00640 1 -- 172.21.15.12:0/2623461939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564b189c35f0 msgr2=0x564b189cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.389+0000 7f0307e00640 1 --2- 172.21.15.12:0/2623461939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564b189c35f0 0x564b189cd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.389+0000 7f0307e00640 1 -- 172.21.15.12:0/2623461939 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564b189bb4c0 con 0x564b189c3fe0 2024-03-23T23:06:22.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.389+0000 7f030f400640 1 --2- 172.21.15.12:0/2623461939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564b189c35f0 0x564b189cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:22.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.389+0000 7f0307e00640 1 --2- 172.21.15.12:0/2623461939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564b189c3fe0 0x564b189d4c60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f02f80059d0 tx=0x7f02f80056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ded778d6ad3c0d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:22.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.389+0000 7f02ff400640 1 -- 172.21.15.12:0/2623461939 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f802b070 con 0x564b189c3fe0 2024-03-23T23:06:22.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.389+0000 7f02ff400640 1 -- 172.21.15.12:0/2623461939 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f02f8022030 con 0x564b189c3fe0 2024-03-23T23:06:22.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.389+0000 7f02ff400640 1 -- 172.21.15.12:0/2623461939 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f8010430 con 0x564b189c3fe0 2024-03-23T23:06:22.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.390+0000 7f0318438740 1 -- 172.21.15.12:0/2623461939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564b189c3fe0 msgr2=0x564b189d4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.390+0000 7f0318438740 1 --2- 172.21.15.12:0/2623461939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564b189c3fe0 0x564b189d4c60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f02f80059d0 tx=0x7f02f80056d0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.390+0000 7f0318438740 1 -- 172.21.15.12:0/2623461939 shutdown_connections 2024-03-23T23:06:22.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.390+0000 7f0318438740 1 --2- 172.21.15.12:0/2623461939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b189c9150 0x564b189cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.390+0000 7f0318438740 1 --2- 172.21.15.12:0/2623461939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564b189c3fe0 0x564b189d4c60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.390+0000 7f0318438740 1 --2- 172.21.15.12:0/2623461939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564b189c35f0 0x564b189cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.390+0000 7f0318438740 1 -- 172.21.15.12:0/2623461939 >> 172.21.15.12:0/2623461939 conn(0x564b189ba750 msgr2=0x564b189c53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:22.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.390+0000 7f0318438740 1 -- 172.21.15.12:0/2623461939 shutdown_connections 2024-03-23T23:06:22.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.390+0000 7f0318438740 1 -- 172.21.15.12:0/2623461939 wait complete. 2024-03-23T23:06:22.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.391+0000 7f0318438740 1 -- start start 2024-03-23T23:06:22.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.391+0000 7f0318438740 1 Processor -- start 2024-03-23T23:06:22.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.391+0000 7f0318438740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564b189c35f0 0x564b189d2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.391+0000 7f0318438740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b189c3fe0 0x564b189d2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.391+0000 7f0318438740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564b189c9150 0x564b189d3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.391+0000 7f0318438740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03101a4ee0 con 0x564b189c35f0 2024-03-23T23:06:22.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.391+0000 7f0318438740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03101a61e0 con 0x564b189c9150 2024-03-23T23:06:22.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.391+0000 7f0318438740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03101a5520 con 0x564b189c3fe0 2024-03-23T23:06:22.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.391+0000 7f0318438740 10 client.0 _ll_register_callbacks cb 0x7f02f8009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:22.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.392+0000 7f030f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564b189c35f0 0x564b189d2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.392+0000 7f030f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564b189c35f0 0x564b189d2880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38000/0 (socket says 172.21.15.12:38000) 2024-03-23T23:06:22.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.392+0000 7f030f400640 1 -- 172.21.15.12:0/2948158029 learned_addr learned my addr 172.21.15.12:0/2948158029 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:22.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.392+0000 7f0307e00640 1 --2- 172.21.15.12:0/2948158029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b189c3fe0 0x564b189d2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.392+0000 7f030f400640 1 --2- 172.21.15.12:0/2948158029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564b189c9150 0x564b189d3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.392+0000 7f030f400640 1 -- 172.21.15.12:0/2948158029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b189c3fe0 msgr2=0x564b189d2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.392+0000 7f030f400640 1 --2- 172.21.15.12:0/2948158029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b189c3fe0 0x564b189d2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.392+0000 7f030f400640 1 -- 172.21.15.12:0/2948158029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564b189c9150 msgr2=0x564b189d3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.392+0000 7f030f400640 1 --2- 172.21.15.12:0/2948158029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564b189c9150 0x564b189d3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.392+0000 7f030f400640 1 -- 172.21.15.12:0/2948158029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564b189d4350 con 0x564b189c35f0 2024-03-23T23:06:22.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.392+0000 7f030f400640 1 --2- 172.21.15.12:0/2948158029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564b189c35f0 0x564b189d2880 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0308036040 tx=0x7f030804b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b71517fcb761abe server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:22.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.392+0000 7f02fcc00640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:06:22.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.393+0000 7f02fcc00640 1 -- 172.21.15.12:0/2948158029 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0308052070 con 0x564b189c35f0 2024-03-23T23:06:22.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.393+0000 7f02fcc00640 1 -- 172.21.15.12:0/2948158029 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0308033580 con 0x564b189c35f0 2024-03-23T23:06:22.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.393+0000 7f0318438740 1 -- 172.21.15.12:0/2948158029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f02f8022030 con 0x564b189c35f0 2024-03-23T23:06:22.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.393+0000 7f02fcc00640 1 -- 172.21.15.12:0/2948158029 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0308043040 con 0x564b189c35f0 2024-03-23T23:06:22.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.393+0000 7f0318438740 10 client.16359 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:22.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.393+0000 7f0318438740 1 -- 172.21.15.12:0/2948158029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x564b18a352d0 con 0x564b189c35f0 2024-03-23T23:06:22.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.393+0000 7f02fcc00640 1 -- 172.21.15.12:0/2948158029 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f02f8022030 con 0x564b189c35f0 2024-03-23T23:06:22.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.393+0000 7f02fcc00640 1 -- 172.21.15.12:0/2948158029 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f03080536b0 con 0x564b189c35f0 2024-03-23T23:06:22.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.394+0000 7f0318438740 10 client.16359 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:22.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.394+0000 7f0318438740 10 client.16359 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:06:22.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.394+0000 7f0318438740 20 client.16359 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:22.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.394+0000 7f0318438740 1 --2- 172.21.15.12:0/2948158029 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x564b18a358e0 0x564b18a37cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.394+0000 7f0318438740 4 client.16359 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:06:22.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.394+0000 7f0318438740 1 -- 172.21.15.12:0/2948158029 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x564b1892c0c0 con 0x564b18a358e0 2024-03-23T23:06:22.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.394+0000 7f0307e00640 1 --2- 172.21.15.12:0/2948158029 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x564b18a358e0 0x564b18a37cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.394+0000 7f0307e00640 1 --2- 172.21.15.12:0/2948158029 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x564b18a358e0 0x564b18a37cd0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4248b9045fb3d1be server_cookie=6dcfe3949f8e7f4f in_seq=0 out_seq=0 2024-03-23T23:06:22.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.394+0000 7f02fcc00640 10 client.16359 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.407+0000 7f02fcc00640 1 -- 172.21.15.12:0/2948158029 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x564b1892c0c0 con 0x564b18a358e0 2024-03-23T23:06:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.407+0000 7f02fcc00640 10 client.16359 handle_command_reply: tid=0 2024-03-23T23:06:22.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.407+0000 7f0318438740 1 client.16359 shutdown 2024-03-23T23:06:22.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.407+0000 7f0318438740 1 -- 172.21.15.12:0/2948158029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564b189c35f0 msgr2=0x564b189d2880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.407+0000 7f0318438740 1 --2- 172.21.15.12:0/2948158029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564b189c35f0 0x564b189d2880 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0308036040 tx=0x7f030804b040 comp rx=0 tx=0).stop 2024-03-23T23:06:22.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.407+0000 7f0318438740 1 -- 172.21.15.12:0/2948158029 shutdown_connections 2024-03-23T23:06:22.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.407+0000 7f0318438740 1 --2- 172.21.15.12:0/2948158029 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x564b18a358e0 0x564b18a37cd0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.407+0000 7f0318438740 1 --2- 172.21.15.12:0/2948158029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564b189c9150 0x564b189d3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.407+0000 7f0318438740 1 --2- 172.21.15.12:0/2948158029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b189c3fe0 0x564b189d2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.407+0000 7f0318438740 1 --2- 172.21.15.12:0/2948158029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564b189c35f0 0x564b189d2880 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.407+0000 7f02fcc00640 0 client.16359 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:22.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.407+0000 7f0318438740 1 -- 172.21.15.12:0/2948158029 >> 172.21.15.12:0/2948158029 conn(0x564b189ba750 msgr2=0x564b189cd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:22.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.407+0000 7f0318438740 1 -- 172.21.15.12:0/2948158029 shutdown_connections 2024-03-23T23:06:22.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.408+0000 7f0318438740 1 -- 172.21.15.12:0/2948158029 wait complete. 2024-03-23T23:06:22.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.408+0000 7f0318438740 20 client.16359 trim_cache size 0 max 16384 2024-03-23T23:06:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:06:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:06:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:06:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:06:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.408+0000 7f0315800640 1 -- 172.21.15.12:0/1578099628 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f02f40a28a0 msgr2=0x7f02f40a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.408+0000 7f0315800640 1 --2- 172.21.15.12:0/1578099628 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f02f40a28a0 0x7f02f40a4d60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f030000e9c0 tx=0x7f03000d2800 comp rx=0 tx=0).stop 2024-03-23T23:06:22.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.408+0000 7f0315800640 1 -- 172.21.15.12:0/1578099628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03101005c0 msgr2=0x7f0310197940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.408+0000 7f0315800640 1 --2- 172.21.15.12:0/1578099628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03101005c0 0x7f0310197940 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f030000e9f0 tx=0x7f030000eeb0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.409+0000 7f0315800640 1 -- 172.21.15.12:0/1578099628 shutdown_connections 2024-03-23T23:06:22.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.409+0000 7f0315800640 1 --2- 172.21.15.12:0/1578099628 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f02f40a28a0 0x7f02f40a4d60 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.409+0000 7f0315800640 1 --2- 172.21.15.12:0/1578099628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0310102170 0x7f031019eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.409+0000 7f0315800640 1 --2- 172.21.15.12:0/1578099628 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03101017b0 0x7f0310197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.409+0000 7f0315800640 1 --2- 172.21.15.12:0/1578099628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03101005c0 0x7f0310197940 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.409+0000 7f0315800640 1 -- 172.21.15.12:0/1578099628 >> 172.21.15.12:0/1578099628 conn(0x7f03100fbb30 msgr2=0x7f0310068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:22.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.409+0000 7f0315800640 1 -- 172.21.15.12:0/1578099628 shutdown_connections 2024-03-23T23:06:22.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:22.409+0000 7f0315800640 1 -- 172.21.15.12:0/1578099628 wait complete. 2024-03-23T23:06:22.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.412+0000 7fa6b4000640 1 -- 172.21.15.12:0/3886300123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6ac104ad0 msgr2=0x7fa6ac104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.412+0000 7fa6b4000640 1 --2- 172.21.15.12:0/3886300123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6ac104ad0 0x7fa6ac104f50 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa698009900 tx=0x7fa69802bc30 comp rx=0 tx=0).stop 2024-03-23T23:06:22.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.412+0000 7fa6b4000640 1 -- 172.21.15.12:0/3886300123 shutdown_connections 2024-03-23T23:06:22.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.412+0000 7fa6b4000640 1 --2- 172.21.15.12:0/3886300123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6ac105490 0x7fa6ac10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.412+0000 7fa6b4000640 1 --2- 172.21.15.12:0/3886300123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6ac104ad0 0x7fa6ac104f50 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.413+0000 7fa6b4000640 1 --2- 172.21.15.12:0/3886300123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6ac1038e0 0x7fa6ac103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.413+0000 7fa6b4000640 1 -- 172.21.15.12:0/3886300123 >> 172.21.15.12:0/3886300123 conn(0x7fa6ac0fee50 msgr2=0x7fa6ac101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:22.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.413+0000 7fa6b4000640 1 -- 172.21.15.12:0/3886300123 shutdown_connections 2024-03-23T23:06:22.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.413+0000 7fa6b4000640 1 -- 172.21.15.12:0/3886300123 wait complete. 2024-03-23T23:06:22.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.414+0000 7fa6b4000640 1 Processor -- start 2024-03-23T23:06:22.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.414+0000 7fa6b4000640 1 -- start start 2024-03-23T23:06:22.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.414+0000 7fa6b4000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6ac1038e0 0x7fa6ac199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.414+0000 7fa6b4000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6ac104ad0 0x7fa6ac19a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.414+0000 7fa6b4000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6ac105490 0x7fa6ac1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.414+0000 7fa6b4000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac10f590 con 0x7fa6ac1038e0 2024-03-23T23:06:22.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.414+0000 7fa6b4000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac10f410 con 0x7fa6ac105490 2024-03-23T23:06:22.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.414+0000 7fa6b4000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac10f710 con 0x7fa6ac104ad0 2024-03-23T23:06:22.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.414+0000 7fa6b1600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6ac1038e0 0x7fa6ac199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.414+0000 7fa6b2000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6ac105490 0x7fa6ac1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.414+0000 7fa6b0c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6ac104ad0 0x7fa6ac19a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.414+0000 7fa6b1600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6ac1038e0 0x7fa6ac199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38012/0 (socket says 172.21.15.12:38012) 2024-03-23T23:06:22.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.414+0000 7fa6b2000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6ac105490 0x7fa6ac1a1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60806/0 (socket says 172.21.15.12:60806) 2024-03-23T23:06:22.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.414+0000 7fa6b0c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6ac104ad0 0x7fa6ac19a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55504/0 (socket says 172.21.15.12:55504) 2024-03-23T23:06:22.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.414+0000 7fa6b1600640 1 -- 172.21.15.12:0/3059828836 learned_addr learned my addr 172.21.15.12:0/3059828836 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:22.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.415+0000 7fa6b2000640 1 -- 172.21.15.12:0/3059828836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6ac104ad0 msgr2=0x7fa6ac19a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.415+0000 7fa6b2000640 1 --2- 172.21.15.12:0/3059828836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6ac104ad0 0x7fa6ac19a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.415+0000 7fa6b2000640 1 -- 172.21.15.12:0/3059828836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6ac1038e0 msgr2=0x7fa6ac199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.415+0000 7fa6b2000640 1 --2- 172.21.15.12:0/3059828836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6ac1038e0 0x7fa6ac199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.415+0000 7fa6b2000640 1 -- 172.21.15.12:0/3059828836 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ac1a1830 con 0x7fa6ac105490 2024-03-23T23:06:22.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.415+0000 7fa6b2000640 1 --2- 172.21.15.12:0/3059828836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6ac105490 0x7fa6ac1a1190 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400ec50 tx=0x7fa6a400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:22.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.415+0000 7fa6aa000640 1 -- 172.21.15.12:0/3059828836 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a4003b70 con 0x7fa6ac105490 2024-03-23T23:06:22.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.415+0000 7fa6aa000640 1 -- 172.21.15.12:0/3059828836 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6a4003d10 con 0x7fa6ac105490 2024-03-23T23:06:22.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.415+0000 7fa6b4000640 1 -- 172.21.15.12:0/3059828836 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6ac1a1b20 con 0x7fa6ac105490 2024-03-23T23:06:22.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.415+0000 7fa6aa000640 1 -- 172.21.15.12:0/3059828836 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a4010890 con 0x7fa6ac105490 2024-03-23T23:06:22.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.416+0000 7fa6b4000640 1 -- 172.21.15.12:0/3059828836 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6ac1a2020 con 0x7fa6ac105490 2024-03-23T23:06:22.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.420+0000 7fa6b6dc3740 1 Processor -- start 2024-03-23T23:06:22.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.420+0000 7fa6aa000640 1 -- 172.21.15.12:0/3059828836 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa6a4010a30 con 0x7fa6ac105490 2024-03-23T23:06:22.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.420+0000 7fa6b6dc3740 1 -- start start 2024-03-23T23:06:22.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.421+0000 7fa6b6dc3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a4810 0x556e7e1a4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.421+0000 7fa6b6dc3740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556e7e1a59e0 0x556e7e1a5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.421+0000 7fa6b6dc3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e7e1a63a0 0x556e7e1acc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.421+0000 7fa6b6dc3740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e7e1092c0 con 0x556e7e1a59e0 2024-03-23T23:06:22.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.421+0000 7fa6b6dc3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e7e0b1c90 con 0x556e7e1a4810 2024-03-23T23:06:22.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.421+0000 7fa6b6dc3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e7e105360 con 0x556e7e1a63a0 2024-03-23T23:06:22.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.421+0000 7fa6b0c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a4810 0x556e7e1a4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.421+0000 7fa6b0c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a4810 0x556e7e1a4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60820/0 (socket says 172.21.15.12:60820) 2024-03-23T23:06:22.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.421+0000 7fa6b0c00640 1 -- 172.21.15.12:0/443206884 learned_addr learned my addr 172.21.15.12:0/443206884 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:22.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.421+0000 7fa6b0c00640 1 --2- 172.21.15.12:0/443206884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e7e1a63a0 0x556e7e1acc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.421+0000 7fa6b1600640 1 --2- 172.21.15.12:0/443206884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556e7e1a59e0 0x556e7e1a5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.422+0000 7fa6b0c00640 1 -- 172.21.15.12:0/443206884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e7e1a63a0 msgr2=0x556e7e1acc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.422+0000 7fa6b0c00640 1 --2- 172.21.15.12:0/443206884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e7e1a63a0 0x556e7e1acc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.422+0000 7fa6b0c00640 1 -- 172.21.15.12:0/443206884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556e7e1a59e0 msgr2=0x556e7e1a5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.422+0000 7fa6b0c00640 1 --2- 172.21.15.12:0/443206884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556e7e1a59e0 0x556e7e1a5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.422+0000 7fa6b0c00640 1 -- 172.21.15.12:0/443206884 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556e7e0d6080 con 0x556e7e1a4810 2024-03-23T23:06:22.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.422+0000 7fa6b1600640 1 --2- 172.21.15.12:0/443206884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556e7e1a59e0 0x556e7e1a5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:06:22.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.422+0000 7fa6aa000640 1 --2- 172.21.15.12:0/3059828836 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa6940a2800 0x7fa6940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.423+0000 7fa6b0c00640 1 --2- 172.21.15.12:0/443206884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a4810 0x556e7e1a4c30 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa69802c8e0 tx=0x7fa698005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f035146a98f422d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:22.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.423+0000 7fa6aa000640 1 -- 172.21.15.12:0/3059828836 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fa6a40cf020 con 0x7fa6ac105490 2024-03-23T23:06:22.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.423+0000 7fa69e600640 1 -- 172.21.15.12:0/443206884 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa698004030 con 0x556e7e1a4810 2024-03-23T23:06:22.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.423+0000 7fa69e600640 1 -- 172.21.15.12:0/443206884 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa698004ba0 con 0x556e7e1a4810 2024-03-23T23:06:22.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.423+0000 7fa69e600640 1 -- 172.21.15.12:0/443206884 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa698003970 con 0x556e7e1a4810 2024-03-23T23:06:22.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.423+0000 7fa6b1600640 1 --2- 172.21.15.12:0/3059828836 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa6940a2800 0x7fa6940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.424+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/443206884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a4810 msgr2=0x556e7e1a4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.424+0000 7fa6b6dc3740 1 --2- 172.21.15.12:0/443206884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a4810 0x556e7e1a4c30 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa69802c8e0 tx=0x7fa698005fd0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.424+0000 7fa6b1600640 1 --2- 172.21.15.12:0/3059828836 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa6940a2800 0x7fa6940a4cc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac19b0f0 tx=0x7fa6a000b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:22.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.424+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/443206884 shutdown_connections 2024-03-23T23:06:22.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.424+0000 7fa6b6dc3740 1 --2- 172.21.15.12:0/443206884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e7e1a63a0 0x556e7e1acc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.424+0000 7fa6b6dc3740 1 --2- 172.21.15.12:0/443206884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556e7e1a59e0 0x556e7e1a5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.424+0000 7fa6b6dc3740 1 --2- 172.21.15.12:0/443206884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a4810 0x556e7e1a4c30 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.424+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/443206884 >> 172.21.15.12:0/443206884 conn(0x556e7e19fc00 msgr2=0x556e7e1a0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:22.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.424+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/443206884 shutdown_connections 2024-03-23T23:06:22.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.424+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/443206884 wait complete. 2024-03-23T23:06:22.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.425+0000 7fa6b6dc3740 1 -- start start 2024-03-23T23:06:22.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.425+0000 7fa6b6dc3740 1 Processor -- start 2024-03-23T23:06:22.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.425+0000 7fa6b6dc3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e7e1a4810 0x556e7e1aeab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.425+0000 7fa6b6dc3740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556e7e1a59e0 0x556e7e1aeff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.425+0000 7fa6b6dc3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a63a0 0x556e7e1b6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.425+0000 7fa6b6dc3740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e7e1ad350 con 0x556e7e1a59e0 2024-03-23T23:06:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.425+0000 7fa6b6dc3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e7e0c50d0 con 0x556e7e1a63a0 2024-03-23T23:06:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.425+0000 7fa6b6dc3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e7e1ad4d0 con 0x556e7e1a4810 2024-03-23T23:06:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.425+0000 7fa6b6dc3740 10 client.0 _ll_register_callbacks cb 0x556e7e0c9e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.426+0000 7fa6b1600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e7e1a4810 0x556e7e1aeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.426+0000 7fa6b2000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a63a0 0x556e7e1b6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.426+0000 7fa6b1600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e7e1a4810 0x556e7e1aeab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55526/0 (socket says 172.21.15.12:55526) 2024-03-23T23:06:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.426+0000 7fa6b2000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a63a0 0x556e7e1b6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60834/0 (socket says 172.21.15.12:60834) 2024-03-23T23:06:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.426+0000 7fa6b0c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556e7e1a59e0 0x556e7e1aeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.426+0000 7fa6b1600640 1 -- 172.21.15.12:0/3973370427 learned_addr learned my addr 172.21.15.12:0/3973370427 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.426+0000 7fa6b2000640 1 -- 172.21.15.12:0/3973370427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e7e1a4810 msgr2=0x556e7e1aeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.426+0000 7fa6b2000640 1 --2- 172.21.15.12:0/3973370427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e7e1a4810 0x556e7e1aeab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.426+0000 7fa6b2000640 1 -- 172.21.15.12:0/3973370427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556e7e1a59e0 msgr2=0x556e7e1aeff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.426+0000 7fa6b2000640 1 --2- 172.21.15.12:0/3973370427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556e7e1a59e0 0x556e7e1aeff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.426+0000 7fa6b2000640 1 -- 172.21.15.12:0/3973370427 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556e7de1d3f0 con 0x556e7e1a63a0 2024-03-23T23:06:22.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.426+0000 7fa6b1600640 1 --2- 172.21.15.12:0/3973370427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e7e1a4810 0x556e7e1aeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:22.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.426+0000 7fa6b0c00640 1 --2- 172.21.15.12:0/3973370427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556e7e1a59e0 0x556e7e1aeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:22.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.426+0000 7fa6b2000640 1 --2- 172.21.15.12:0/3973370427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a63a0 0x556e7e1b6030 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa6a40ccba0 tx=0x7fa6a409b890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3be01bb2fd1f69d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:22.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.426+0000 7fa692a00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:06:22.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.427+0000 7fa692a00640 1 -- 172.21.15.12:0/3973370427 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a40b4060 con 0x556e7e1a63a0 2024-03-23T23:06:22.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.427+0000 7fa692a00640 1 -- 172.21.15.12:0/3973370427 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fa6a4004b40 con 0x556e7e1a63a0 2024-03-23T23:06:22.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.427+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/3973370427 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa698004ba0 con 0x556e7e1a63a0 2024-03-23T23:06:22.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.427+0000 7fa692a00640 1 -- 172.21.15.12:0/3973370427 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6a4009820 con 0x556e7e1a63a0 2024-03-23T23:06:22.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.427+0000 7fa6b6dc3740 10 client.26492 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:22.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.427+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/3973370427 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556e7dee77d0 con 0x556e7e1a63a0 2024-03-23T23:06:22.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.427+0000 7fa692a00640 1 -- 172.21.15.12:0/3973370427 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa698004ba0 con 0x556e7e1a63a0 2024-03-23T23:06:22.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.427+0000 7fa692a00640 1 -- 172.21.15.12:0/3973370427 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fa6a409ae40 con 0x556e7e1a63a0 2024-03-23T23:06:22.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.427+0000 7fa6b6dc3740 10 client.26492 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:22.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.427+0000 7fa6b6dc3740 10 client.26492 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:06:22.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.428+0000 7fa6b6dc3740 20 client.26492 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:22.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.428+0000 7fa6b6dc3740 1 --2- 172.21.15.12:0/3973370427 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x556e7e1ba470 0x556e7e1bc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.428+0000 7fa6b6dc3740 4 client.26492 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:06:22.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.428+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/3973370427 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556e7e116800 con 0x556e7e1ba470 2024-03-23T23:06:22.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.428+0000 7fa6b1600640 1 --2- 172.21.15.12:0/3973370427 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x556e7e1ba470 0x556e7e1bc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.428+0000 7fa6b1600640 1 --2- 172.21.15.12:0/3973370427 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x556e7e1ba470 0x556e7e1bc8b0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8a06718931fa115 server_cookie=c49e3b18cdfdf8c0 in_seq=0 out_seq=0 2024-03-23T23:06:22.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.428+0000 7fa692a00640 10 client.26492 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:06:22.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.429+0000 7fa692a00640 1 -- 172.21.15.12:0/3973370427 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556e7e116800 con 0x556e7e1ba470 2024-03-23T23:06:22.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.429+0000 7fa692a00640 10 client.26492 handle_command_reply: tid=0 2024-03-23T23:06:22.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.429+0000 7fa6b6dc3740 1 client.26492 shutdown 2024-03-23T23:06:22.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.429+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/3973370427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a63a0 msgr2=0x556e7e1b6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.429+0000 7fa6b6dc3740 1 --2- 172.21.15.12:0/3973370427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a63a0 0x556e7e1b6030 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa6a40ccba0 tx=0x7fa6a409b890 comp rx=0 tx=0).stop 2024-03-23T23:06:22.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.429+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/3973370427 shutdown_connections 2024-03-23T23:06:22.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.429+0000 7fa6b6dc3740 1 --2- 172.21.15.12:0/3973370427 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x556e7e1ba470 0x556e7e1bc8b0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.429+0000 7fa6b6dc3740 1 --2- 172.21.15.12:0/3973370427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a63a0 0x556e7e1b6030 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.429+0000 7fa6b6dc3740 1 --2- 172.21.15.12:0/3973370427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556e7e1a59e0 0x556e7e1aeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.429+0000 7fa6b6dc3740 1 --2- 172.21.15.12:0/3973370427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e7e1a4810 0x556e7e1aeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.429+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/3973370427 >> 172.21.15.12:0/3973370427 conn(0x556e7e19fc00 msgr2=0x556e7e19eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:22.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.429+0000 7fa692a00640 0 client.26492 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:06:22.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.429+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/3973370427 shutdown_connections 2024-03-23T23:06:22.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.430+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/3973370427 wait complete. 2024-03-23T23:06:22.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.430+0000 7fa6b6dc3740 20 client.26492 trim_cache size 0 max 16384 2024-03-23T23:06:22.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.444+0000 7fa6b6dc3740 1 Processor -- start 2024-03-23T23:06:22.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.444+0000 7fa6b6dc3740 1 -- start start 2024-03-23T23:06:22.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.444+0000 7fa6b6dc3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a4810 0x556e7e1a8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.444+0000 7fa6b6dc3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e7e1a59e0 0x556e7e1b5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.444+0000 7fa6b6dc3740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556e7e1a63a0 0x556e7e1a8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.444+0000 7fa6b6dc3740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac1ad1a0 con 0x556e7e1a63a0 2024-03-23T23:06:22.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.444+0000 7fa6b6dc3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac1aaca0 con 0x556e7e1a4810 2024-03-23T23:06:22.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.444+0000 7fa6b6dc3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac1a8d40 con 0x556e7e1a59e0 2024-03-23T23:06:22.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.444+0000 7fa6b1600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a4810 0x556e7e1a8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.444+0000 7fa6b1600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a4810 0x556e7e1a8a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60848/0 (socket says 172.21.15.12:60848) 2024-03-23T23:06:22.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.444+0000 7fa6b1600640 1 -- 172.21.15.12:0/2967593066 learned_addr learned my addr 172.21.15.12:0/2967593066 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:22.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.444+0000 7fa6b0c00640 1 --2- 172.21.15.12:0/2967593066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e7e1a59e0 0x556e7e1b5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.444+0000 7fa6b2000640 1 --2- 172.21.15.12:0/2967593066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556e7e1a63a0 0x556e7e1a8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.445+0000 7fa6b1600640 1 -- 172.21.15.12:0/2967593066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e7e1a59e0 msgr2=0x556e7e1b5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.445+0000 7fa6b1600640 1 --2- 172.21.15.12:0/2967593066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e7e1a59e0 0x556e7e1b5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.445+0000 7fa6b1600640 1 -- 172.21.15.12:0/2967593066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556e7e1a63a0 msgr2=0x556e7e1a8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.445+0000 7fa6b1600640 1 --2- 172.21.15.12:0/2967593066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556e7e1a63a0 0x556e7e1a8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.445+0000 7fa6b1600640 1 -- 172.21.15.12:0/2967593066 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556e7e19a2c0 con 0x556e7e1a4810 2024-03-23T23:06:22.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.445+0000 7fa6b2000640 1 --2- 172.21.15.12:0/2967593066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556e7e1a63a0 0x556e7e1a8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:06:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.445+0000 7fa6b1600640 1 --2- 172.21.15.12:0/2967593066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a4810 0x556e7e1a8a80 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x556e7e1bd120 tx=0x7fa6a0015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97fe60b31b05f431 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.445+0000 7fa69dc00640 1 -- 172.21.15.12:0/2967593066 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a0021db0 con 0x556e7e1a4810 2024-03-23T23:06:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.445+0000 7fa69dc00640 1 -- 172.21.15.12:0/2967593066 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6a002b050 con 0x556e7e1a4810 2024-03-23T23:06:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.445+0000 7fa69dc00640 1 -- 172.21.15.12:0/2967593066 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a001f810 con 0x556e7e1a4810 2024-03-23T23:06:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.446+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/2967593066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a4810 msgr2=0x556e7e1a8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.446+0000 7fa6b6dc3740 1 --2- 172.21.15.12:0/2967593066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a4810 0x556e7e1a8a80 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x556e7e1bd120 tx=0x7fa6a0015000 comp rx=0 tx=0).stop 2024-03-23T23:06:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.446+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/2967593066 shutdown_connections 2024-03-23T23:06:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.446+0000 7fa6b6dc3740 1 --2- 172.21.15.12:0/2967593066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556e7e1a63a0 0x556e7e1a8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.446+0000 7fa6b6dc3740 1 --2- 172.21.15.12:0/2967593066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e7e1a59e0 0x556e7e1b5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.446+0000 7fa6b6dc3740 1 --2- 172.21.15.12:0/2967593066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a4810 0x556e7e1a8a80 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.446+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/2967593066 >> 172.21.15.12:0/2967593066 conn(0x556e7e19fc00 msgr2=0x556e7e1a36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.446+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/2967593066 shutdown_connections 2024-03-23T23:06:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.446+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/2967593066 wait complete. 2024-03-23T23:06:22.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.446+0000 7fa6b6dc3740 1 -- start start 2024-03-23T23:06:22.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.447+0000 7fa6b6dc3740 1 Processor -- start 2024-03-23T23:06:22.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.447+0000 7fa6b6dc3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e7e1a4810 0x556e7e1b1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.447+0000 7fa6b6dc3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a59e0 0x556e7e1b2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.447+0000 7fa6b6dc3740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556e7e1a63a0 0x556e7e1b4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.447+0000 7fa6b6dc3740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac1a9340 con 0x556e7e1a63a0 2024-03-23T23:06:22.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.447+0000 7fa6b6dc3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac1aa640 con 0x556e7e1a59e0 2024-03-23T23:06:22.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.447+0000 7fa6b6dc3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac1a9980 con 0x556e7e1a4810 2024-03-23T23:06:22.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.447+0000 7fa6b6dc3740 10 client.0 _ll_register_callbacks cb 0x7fa6a4003ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.447+0000 7fa6b1600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e7e1a4810 0x556e7e1b1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.447+0000 7fa6b1600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e7e1a4810 0x556e7e1b1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55532/0 (socket says 172.21.15.12:55532) 2024-03-23T23:06:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.447+0000 7fa6b1600640 1 -- 172.21.15.12:0/574378567 learned_addr learned my addr 172.21.15.12:0/574378567 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.447+0000 7fa6b0c00640 1 --2- 172.21.15.12:0/574378567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a59e0 0x556e7e1b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.448+0000 7fa6b2000640 1 --2- 172.21.15.12:0/574378567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556e7e1a63a0 0x556e7e1b4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.448+0000 7fa6b1600640 1 -- 172.21.15.12:0/574378567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a59e0 msgr2=0x556e7e1b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.448+0000 7fa6b1600640 1 --2- 172.21.15.12:0/574378567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a59e0 0x556e7e1b2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.448+0000 7fa6b1600640 1 -- 172.21.15.12:0/574378567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556e7e1a63a0 msgr2=0x556e7e1b4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.448+0000 7fa6b1600640 1 --2- 172.21.15.12:0/574378567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556e7e1a63a0 0x556e7e1b4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.448+0000 7fa6b1600640 1 -- 172.21.15.12:0/574378567 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556e7e1b5590 con 0x556e7e1a4810 2024-03-23T23:06:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.448+0000 7fa6b0c00640 1 --2- 172.21.15.12:0/574378567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a59e0 0x556e7e1b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.448+0000 7fa6b1600640 1 --2- 172.21.15.12:0/574378567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e7e1a4810 0x556e7e1b1ae0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa6a00057a0 tx=0x7fa6a0021db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78a5c06fb164b460 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.448+0000 7fa692a00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:06:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.448+0000 7fa692a00640 1 -- 172.21.15.12:0/574378567 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a003a020 con 0x556e7e1a4810 2024-03-23T23:06:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.448+0000 7fa692a00640 1 -- 172.21.15.12:0/574378567 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fa6a002b200 con 0x556e7e1a4810 2024-03-23T23:06:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.448+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/574378567 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa6a002b050 con 0x556e7e1a4810 2024-03-23T23:06:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.449+0000 7fa692a00640 1 -- 172.21.15.12:0/574378567 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6a001f810 con 0x556e7e1a4810 2024-03-23T23:06:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.449+0000 7fa6b6dc3740 10 client.26470 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.449+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/574378567 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556e7e215fa0 con 0x556e7e1a4810 2024-03-23T23:06:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.449+0000 7fa692a00640 1 -- 172.21.15.12:0/574378567 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6a002b050 con 0x556e7e1a4810 2024-03-23T23:06:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.449+0000 7fa692a00640 1 -- 172.21.15.12:0/574378567 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fa6a0041420 con 0x556e7e1a4810 2024-03-23T23:06:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.449+0000 7fa6b6dc3740 10 client.26470 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.449+0000 7fa6b6dc3740 10 client.26470 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:06:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.449+0000 7fa6b6dc3740 20 client.26470 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.449+0000 7fa6b6dc3740 1 --2- 172.21.15.12:0/574378567 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x556e7e2168f0 0x556e7e218d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.449+0000 7fa6b6dc3740 4 client.26470 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:06:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.449+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/574378567 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556e7e110b50 con 0x556e7e2168f0 2024-03-23T23:06:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.450+0000 7fa6b0c00640 1 --2- 172.21.15.12:0/574378567 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x556e7e2168f0 0x556e7e218d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.450+0000 7fa6b0c00640 1 --2- 172.21.15.12:0/574378567 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x556e7e2168f0 0x556e7e218d90 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bad57fcbc21f21ce server_cookie=607f3e10018bcbda in_seq=0 out_seq=0 2024-03-23T23:06:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.450+0000 7fa692a00640 10 client.26470 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:06:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.450+0000 7fa692a00640 1 -- 172.21.15.12:0/574378567 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556e7e110b50 con 0x556e7e2168f0 2024-03-23T23:06:22.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.450+0000 7fa692a00640 10 client.26470 handle_command_reply: tid=0 2024-03-23T23:06:22.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.450+0000 7fa6b6dc3740 1 client.26470 shutdown 2024-03-23T23:06:22.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.451+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/574378567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e7e1a4810 msgr2=0x556e7e1b1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.451+0000 7fa6b6dc3740 1 --2- 172.21.15.12:0/574378567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e7e1a4810 0x556e7e1b1ae0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa6a00057a0 tx=0x7fa6a0021db0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.451+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/574378567 shutdown_connections 2024-03-23T23:06:22.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.451+0000 7fa6b6dc3740 1 --2- 172.21.15.12:0/574378567 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x556e7e2168f0 0x556e7e218d90 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.451+0000 7fa6b6dc3740 1 --2- 172.21.15.12:0/574378567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556e7e1a63a0 0x556e7e1b4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.451+0000 7fa6b6dc3740 1 --2- 172.21.15.12:0/574378567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e7e1a59e0 0x556e7e1b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.451+0000 7fa6b6dc3740 1 --2- 172.21.15.12:0/574378567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e7e1a4810 0x556e7e1b1ae0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.451+0000 7fa692a00640 0 client.26470 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:06:22.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.451+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/574378567 >> 172.21.15.12:0/574378567 conn(0x556e7e19fc00 msgr2=0x556e7e19e620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:22.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.451+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/574378567 shutdown_connections 2024-03-23T23:06:22.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.451+0000 7fa6b6dc3740 1 -- 172.21.15.12:0/574378567 wait complete. 2024-03-23T23:06:22.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.451+0000 7fa6b6dc3740 20 client.26470 trim_cache size 0 max 16384 2024-03-23T23:06:22.454 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:06:22.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.452+0000 7fa6b4000640 1 -- 172.21.15.12:0/3059828836 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa6940a2800 msgr2=0x7fa6940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.452+0000 7fa6b4000640 1 --2- 172.21.15.12:0/3059828836 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa6940a2800 0x7fa6940a4cc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac19b0f0 tx=0x7fa6a000b4a0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.452+0000 7fa6b4000640 1 -- 172.21.15.12:0/3059828836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6ac105490 msgr2=0x7fa6ac1a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.452+0000 7fa6b4000640 1 --2- 172.21.15.12:0/3059828836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6ac105490 0x7fa6ac1a1190 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400ec50 tx=0x7fa6a400c6a0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.452+0000 7fa6b4000640 1 -- 172.21.15.12:0/3059828836 shutdown_connections 2024-03-23T23:06:22.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.452+0000 7fa6b4000640 1 --2- 172.21.15.12:0/3059828836 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa6940a2800 0x7fa6940a4cc0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.452+0000 7fa6b4000640 1 --2- 172.21.15.12:0/3059828836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6ac105490 0x7fa6ac1a1190 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.452+0000 7fa6b4000640 1 --2- 172.21.15.12:0/3059828836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6ac104ad0 0x7fa6ac19a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.452+0000 7fa6b4000640 1 --2- 172.21.15.12:0/3059828836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6ac1038e0 0x7fa6ac199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.452+0000 7fa6b4000640 1 -- 172.21.15.12:0/3059828836 >> 172.21.15.12:0/3059828836 conn(0x7fa6ac0fee50 msgr2=0x7fa6ac107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:22.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.452+0000 7fa6b4000640 1 -- 172.21.15.12:0/3059828836 shutdown_connections 2024-03-23T23:06:22.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.452+0000 7fa6b4000640 1 -- 172.21.15.12:0/3059828836 wait complete. 2024-03-23T23:06:22.468 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:06:22.469 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-23T23:06:22.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.684+0000 7f8725a00640 1 -- 172.21.15.12:0/515782269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87201037c0 msgr2=0x7f8720103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.684+0000 7f8725a00640 1 --2- 172.21.15.12:0/515782269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87201037c0 0x7f8720103be0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f8710009900 tx=0x7f871002bc30 comp rx=0 tx=0).stop 2024-03-23T23:06:22.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.684+0000 7f8725a00640 1 -- 172.21.15.12:0/515782269 shutdown_connections 2024-03-23T23:06:22.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.684+0000 7f8725a00640 1 --2- 172.21.15.12:0/515782269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8720105370 0x7f872010bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.684+0000 7f8725a00640 1 --2- 172.21.15.12:0/515782269 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87201049b0 0x7f8720104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.684+0000 7f8725a00640 1 --2- 172.21.15.12:0/515782269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87201037c0 0x7f8720103be0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.684+0000 7f8725a00640 1 -- 172.21.15.12:0/515782269 >> 172.21.15.12:0/515782269 conn(0x7f87200fed50 msgr2=0x7f8720101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:22.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.684+0000 7f8725a00640 1 -- 172.21.15.12:0/515782269 shutdown_connections 2024-03-23T23:06:22.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.684+0000 7f8725a00640 1 -- 172.21.15.12:0/515782269 wait complete. 2024-03-23T23:06:22.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.685+0000 7f8725a00640 1 Processor -- start 2024-03-23T23:06:22.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.686+0000 7f8725a00640 1 -- start start 2024-03-23T23:06:22.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.686+0000 7f8725a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87201037c0 0x7f8720199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.686+0000 7f8725a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87201049b0 0x7f8720199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.686+0000 7f8725a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8720105370 0x7f87201a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.686+0000 7f8725a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872010f460 con 0x7f87201049b0 2024-03-23T23:06:22.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.686+0000 7f8725a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872010f2e0 con 0x7f87201037c0 2024-03-23T23:06:22.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.686+0000 7f8725a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872010f5e0 con 0x7f8720105370 2024-03-23T23:06:22.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.686+0000 7f8717400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87201049b0 0x7f8720199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.686+0000 7f871fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87201037c0 0x7f8720199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.686+0000 7f8725000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8720105370 0x7f87201a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.686+0000 7f871fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87201037c0 0x7f8720199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60866/0 (socket says 172.21.15.12:60866) 2024-03-23T23:06:22.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.686+0000 7f8725000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8720105370 0x7f87201a1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55546/0 (socket says 172.21.15.12:55546) 2024-03-23T23:06:22.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.686+0000 7f8717400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87201049b0 0x7f8720199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38070/0 (socket says 172.21.15.12:38070) 2024-03-23T23:06:22.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.686+0000 7f871fe00640 1 -- 172.21.15.12:0/4282462668 learned_addr learned my addr 172.21.15.12:0/4282462668 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:22.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.687+0000 7f871fe00640 1 -- 172.21.15.12:0/4282462668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8720105370 msgr2=0x7f87201a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.687+0000 7f871fe00640 1 --2- 172.21.15.12:0/4282462668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8720105370 0x7f87201a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.687+0000 7f871fe00640 1 -- 172.21.15.12:0/4282462668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87201049b0 msgr2=0x7f8720199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.687+0000 7f871fe00640 1 --2- 172.21.15.12:0/4282462668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87201049b0 0x7f8720199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.687+0000 7f871fe00640 1 -- 172.21.15.12:0/4282462668 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87201a16d0 con 0x7f87201037c0 2024-03-23T23:06:22.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.687+0000 7f8717400640 1 --2- 172.21.15.12:0/4282462668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87201049b0 0x7f8720199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:22.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.687+0000 7f8725000640 1 --2- 172.21.15.12:0/4282462668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8720105370 0x7f87201a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:22.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.687+0000 7f871fe00640 1 --2- 172.21.15.12:0/4282462668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87201037c0 0x7f8720199ab0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8710000c00 tx=0x7f871002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.688+0000 7f871d600640 1 -- 172.21.15.12:0/4282462668 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8710043020 con 0x7f87201037c0 2024-03-23T23:06:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.688+0000 7f871d600640 1 -- 172.21.15.12:0/4282462668 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87100054e0 con 0x7f87201037c0 2024-03-23T23:06:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.688+0000 7f8725a00640 1 -- 172.21.15.12:0/4282462668 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87201a1960 con 0x7f87201037c0 2024-03-23T23:06:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.688+0000 7f871d600640 1 -- 172.21.15.12:0/4282462668 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8710003c40 con 0x7f87201037c0 2024-03-23T23:06:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.688+0000 7f8725a00640 1 -- 172.21.15.12:0/4282462668 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87201a1f00 con 0x7f87201037c0 2024-03-23T23:06:22.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.691+0000 7f8729c69740 1 Processor -- start 2024-03-23T23:06:22.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.691+0000 7f8729c69740 1 -- start start 2024-03-23T23:06:22.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.692+0000 7f8729c69740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a339f0 0x560033a33e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.692+0000 7f8729c69740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560033a34bc0 0x560033a35040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.692+0000 7f8729c69740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560033a35580 0x560033a3be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.692+0000 7f8729c69740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600339982c0 con 0x560033a35580 2024-03-23T23:06:22.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.692+0000 7f8729c69740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x560033940c90 con 0x560033a339f0 2024-03-23T23:06:22.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.692+0000 7f8729c69740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x560033994360 con 0x560033a34bc0 2024-03-23T23:06:22.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.692+0000 7f871d600640 1 -- 172.21.15.12:0/4282462668 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8710046020 con 0x7f87201037c0 2024-03-23T23:06:22.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.692+0000 7f8725000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a339f0 0x560033a33e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.692+0000 7f8717400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560033a35580 0x560033a3be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.692+0000 7f8725000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a339f0 0x560033a33e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60868/0 (socket says 172.21.15.12:60868) 2024-03-23T23:06:22.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.692+0000 7f8725000640 1 -- 172.21.15.12:0/772403893 learned_addr learned my addr 172.21.15.12:0/772403893 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:22.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.692+0000 7f871fe00640 1 --2- 172.21.15.12:0/772403893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560033a34bc0 0x560033a35040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.692+0000 7f8725000640 1 -- 172.21.15.12:0/772403893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560033a34bc0 msgr2=0x560033a35040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.692+0000 7f8725000640 1 --2- 172.21.15.12:0/772403893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560033a34bc0 0x560033a35040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.692+0000 7f8725000640 1 -- 172.21.15.12:0/772403893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560033a35580 msgr2=0x560033a3be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.692+0000 7f8725000640 1 --2- 172.21.15.12:0/772403893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560033a35580 0x560033a3be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.692+0000 7f8725000640 1 -- 172.21.15.12:0/772403893 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560033965080 con 0x560033a339f0 2024-03-23T23:06:22.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.693+0000 7f8717400640 1 --2- 172.21.15.12:0/772403893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560033a35580 0x560033a3be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:22.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.693+0000 7f871fe00640 1 --2- 172.21.15.12:0/772403893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560033a34bc0 0x560033a35040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:06:22.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.693+0000 7f8725000640 1 --2- 172.21.15.12:0/772403893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a339f0 0x560033a33e10 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f87180117e0 tx=0x7f8718011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=481892f96a53c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:22.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.693+0000 7f8715600640 1 -- 172.21.15.12:0/772403893 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8718011310 con 0x560033a339f0 2024-03-23T23:06:22.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.693+0000 7f8715600640 1 -- 172.21.15.12:0/772403893 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8718003e50 con 0x560033a339f0 2024-03-23T23:06:22.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.693+0000 7f8715600640 1 -- 172.21.15.12:0/772403893 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87180045b0 con 0x560033a339f0 2024-03-23T23:06:22.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.694+0000 7f871d600640 1 --2- 172.21.15.12:0/4282462668 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f87040a2880 0x7f87040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.694+0000 7f8729c69740 1 -- 172.21.15.12:0/772403893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a339f0 msgr2=0x560033a33e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.694+0000 7f8729c69740 1 --2- 172.21.15.12:0/772403893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a339f0 0x560033a33e10 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f87180117e0 tx=0x7f8718011ca0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.694+0000 7f8729c69740 1 -- 172.21.15.12:0/772403893 shutdown_connections 2024-03-23T23:06:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.694+0000 7f8729c69740 1 --2- 172.21.15.12:0/772403893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560033a35580 0x560033a3be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.694+0000 7f8729c69740 1 --2- 172.21.15.12:0/772403893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560033a34bc0 0x560033a35040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.694+0000 7f8729c69740 1 --2- 172.21.15.12:0/772403893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a339f0 0x560033a33e10 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.694+0000 7f871d600640 1 -- 172.21.15.12:0/4282462668 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f87100f09b0 con 0x7f87201037c0 2024-03-23T23:06:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.694+0000 7f8729c69740 1 -- 172.21.15.12:0/772403893 >> 172.21.15.12:0/772403893 conn(0x560033a2ede0 msgr2=0x560033a2f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.694+0000 7f8729c69740 1 -- 172.21.15.12:0/772403893 shutdown_connections 2024-03-23T23:06:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.694+0000 7f871fe00640 1 --2- 172.21.15.12:0/4282462668 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f87040a2880 0x7f87040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.694+0000 7f8729c69740 1 -- 172.21.15.12:0/772403893 wait complete. 2024-03-23T23:06:22.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.695+0000 7f8729c69740 1 -- start start 2024-03-23T23:06:22.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.695+0000 7f871fe00640 1 --2- 172.21.15.12:0/4282462668 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f87040a2880 0x7f87040a4d40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f87100b9d30 tx=0x7f87100dd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:22.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.695+0000 7f8729c69740 1 Processor -- start 2024-03-23T23:06:22.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.695+0000 7f8729c69740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560033a339f0 0x560033a3dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.695+0000 7f8729c69740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a34bc0 0x560033a3e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.695+0000 7f8729c69740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560033a35580 0x560033a45220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.695+0000 7f8729c69740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x560033a3c590 con 0x560033a339f0 2024-03-23T23:06:22.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.695+0000 7f8729c69740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600339540d0 con 0x560033a34bc0 2024-03-23T23:06:22.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.695+0000 7f8729c69740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x560033a3c710 con 0x560033a35580 2024-03-23T23:06:22.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.696+0000 7f8729c69740 10 client.0 _ll_register_callbacks cb 0x560033958e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:22.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.696+0000 7f8717400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a34bc0 0x560033a3e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.696+0000 7f8725000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560033a339f0 0x560033a3dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.696+0000 7f8717400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a34bc0 0x560033a3e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60880/0 (socket says 172.21.15.12:60880) 2024-03-23T23:06:22.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.696+0000 7f8717400640 1 -- 172.21.15.12:0/1839294971 learned_addr learned my addr 172.21.15.12:0/1839294971 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:22.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.696+0000 7f8725000640 1 --2- 172.21.15.12:0/1839294971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560033a35580 0x560033a45220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.696+0000 7f8717400640 1 -- 172.21.15.12:0/1839294971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560033a35580 msgr2=0x560033a45220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.696+0000 7f8717400640 1 --2- 172.21.15.12:0/1839294971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560033a35580 0x560033a45220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.696+0000 7f8717400640 1 -- 172.21.15.12:0/1839294971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560033a339f0 msgr2=0x560033a3dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.696+0000 7f8717400640 1 --2- 172.21.15.12:0/1839294971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560033a339f0 0x560033a3dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.696+0000 7f8717400640 1 -- 172.21.15.12:0/1839294971 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5600336ac3f0 con 0x560033a34bc0 2024-03-23T23:06:22.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.696+0000 7f8717400640 1 --2- 172.21.15.12:0/1839294971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a34bc0 0x560033a3e1e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f8708012d00 tx=0x7f870800f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23f6d6ad35353d32 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:22.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.696+0000 7f870e000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:06:22.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.696+0000 7f870e000640 1 -- 172.21.15.12:0/1839294971 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8708009d90 con 0x560033a34bc0 2024-03-23T23:06:22.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.697+0000 7f870e000640 1 -- 172.21.15.12:0/1839294971 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f87080120d0 con 0x560033a34bc0 2024-03-23T23:06:22.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.697+0000 7f8729c69740 1 -- 172.21.15.12:0/1839294971 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8718003e50 con 0x560033a34bc0 2024-03-23T23:06:22.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.697+0000 7f870e000640 1 -- 172.21.15.12:0/1839294971 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8708005020 con 0x560033a34bc0 2024-03-23T23:06:22.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.697+0000 7f8729c69740 10 client.26510 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:22.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.697+0000 7f8729c69740 1 -- 172.21.15.12:0/1839294971 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5600337767d0 con 0x560033a34bc0 2024-03-23T23:06:22.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.697+0000 7f870e000640 1 -- 172.21.15.12:0/1839294971 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8718003e50 con 0x560033a34bc0 2024-03-23T23:06:22.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.697+0000 7f870e000640 1 -- 172.21.15.12:0/1839294971 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8708010430 con 0x560033a34bc0 2024-03-23T23:06:22.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.697+0000 7f8729c69740 10 client.26510 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:22.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.697+0000 7f8729c69740 10 client.26510 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:06:22.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.697+0000 7f8729c69740 20 client.26510 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:22.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.697+0000 7f8729c69740 1 --2- 172.21.15.12:0/1839294971 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x560033a49610 0x560033a4ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.698+0000 7f8729c69740 4 client.26510 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:06:22.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.698+0000 7f8729c69740 1 -- 172.21.15.12:0/1839294971 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5600339a5800 con 0x560033a49610 2024-03-23T23:06:22.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.698+0000 7f8725000640 1 --2- 172.21.15.12:0/1839294971 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x560033a49610 0x560033a4ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.698+0000 7f8725000640 1 --2- 172.21.15.12:0/1839294971 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x560033a49610 0x560033a4ba50 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c4c22a0e3ba8ac5c server_cookie=de7de008ba0ad250 in_seq=0 out_seq=0 2024-03-23T23:06:22.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.698+0000 7f870e000640 10 client.26510 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:06:22.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.699+0000 7f870e000640 1 -- 172.21.15.12:0/1839294971 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5600339a5800 con 0x560033a49610 2024-03-23T23:06:22.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.699+0000 7f870e000640 10 client.26510 handle_command_reply: tid=0 2024-03-23T23:06:22.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.699+0000 7f8729c69740 1 client.26510 shutdown 2024-03-23T23:06:22.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.699+0000 7f8729c69740 1 -- 172.21.15.12:0/1839294971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a34bc0 msgr2=0x560033a3e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.699+0000 7f8729c69740 1 --2- 172.21.15.12:0/1839294971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a34bc0 0x560033a3e1e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f8708012d00 tx=0x7f870800f790 comp rx=0 tx=0).stop 2024-03-23T23:06:22.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.699+0000 7f8729c69740 1 -- 172.21.15.12:0/1839294971 shutdown_connections 2024-03-23T23:06:22.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.699+0000 7f8729c69740 1 --2- 172.21.15.12:0/1839294971 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x560033a49610 0x560033a4ba50 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.699+0000 7f8729c69740 1 --2- 172.21.15.12:0/1839294971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560033a35580 0x560033a45220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.699+0000 7f8729c69740 1 --2- 172.21.15.12:0/1839294971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a34bc0 0x560033a3e1e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.699+0000 7f8729c69740 1 --2- 172.21.15.12:0/1839294971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560033a339f0 0x560033a3dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.699+0000 7f870e000640 0 client.26510 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:06:22.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.699+0000 7f8729c69740 1 -- 172.21.15.12:0/1839294971 >> 172.21.15.12:0/1839294971 conn(0x560033a2ede0 msgr2=0x560033a2e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:22.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.700+0000 7f8729c69740 1 -- 172.21.15.12:0/1839294971 shutdown_connections 2024-03-23T23:06:22.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.700+0000 7f8729c69740 1 -- 172.21.15.12:0/1839294971 wait complete. 2024-03-23T23:06:22.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.700+0000 7f8729c69740 20 client.26510 trim_cache size 0 max 16384 2024-03-23T23:06:22.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.713+0000 7f8729c69740 1 Processor -- start 2024-03-23T23:06:22.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.714+0000 7f8729c69740 1 -- start start 2024-03-23T23:06:22.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.714+0000 7f8729c69740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a339f0 0x560033a32b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.714+0000 7f8729c69740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560033a34bc0 0x560033a451e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.714+0000 7f8729c69740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560033a35580 0x560033a37c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.714+0000 7f8729c69740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87201acfa0 con 0x560033a34bc0 2024-03-23T23:06:22.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.714+0000 7f8729c69740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87201aaaa0 con 0x560033a339f0 2024-03-23T23:06:22.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.714+0000 7f8729c69740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87201a8b40 con 0x560033a35580 2024-03-23T23:06:22.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.714+0000 7f8725000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a339f0 0x560033a32b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.714+0000 7f8717400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560033a34bc0 0x560033a451e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.714+0000 7f8725000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a339f0 0x560033a32b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60892/0 (socket says 172.21.15.12:60892) 2024-03-23T23:06:22.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.714+0000 7f8725000640 1 -- 172.21.15.12:0/1852206274 learned_addr learned my addr 172.21.15.12:0/1852206274 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:22.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.714+0000 7f8717400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560033a34bc0 0x560033a451e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38094/0 (socket says 172.21.15.12:38094) 2024-03-23T23:06:22.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.714+0000 7f8725000640 1 --2- 172.21.15.12:0/1852206274 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560033a35580 0x560033a37c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.715+0000 7f8725000640 1 -- 172.21.15.12:0/1852206274 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560033a35580 msgr2=0x560033a37c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.715+0000 7f8725000640 1 --2- 172.21.15.12:0/1852206274 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560033a35580 0x560033a37c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.715+0000 7f8725000640 1 -- 172.21.15.12:0/1852206274 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560033a34bc0 msgr2=0x560033a451e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.715+0000 7f8725000640 1 --2- 172.21.15.12:0/1852206274 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560033a34bc0 0x560033a451e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.715+0000 7f8725000640 1 -- 172.21.15.12:0/1852206274 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560033a2f850 con 0x560033a339f0 2024-03-23T23:06:22.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.715+0000 7f8717400640 1 --2- 172.21.15.12:0/1852206274 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560033a34bc0 0x560033a451e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:22.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.715+0000 7f8725000640 1 --2- 172.21.15.12:0/1852206274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a339f0 0x560033a32b10 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f871800a010 tx=0x7f8718005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16b1f0da9dafc797 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:22.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.715+0000 7f8714c00640 1 -- 172.21.15.12:0/1852206274 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8718022040 con 0x560033a339f0 2024-03-23T23:06:22.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.715+0000 7f8714c00640 1 -- 172.21.15.12:0/1852206274 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8718011190 con 0x560033a339f0 2024-03-23T23:06:22.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.715+0000 7f8714c00640 1 -- 172.21.15.12:0/1852206274 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8718005070 con 0x560033a339f0 2024-03-23T23:06:22.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.716+0000 7f8729c69740 1 -- 172.21.15.12:0/1852206274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a339f0 msgr2=0x560033a32b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.716+0000 7f8729c69740 1 --2- 172.21.15.12:0/1852206274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a339f0 0x560033a32b10 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f871800a010 tx=0x7f8718005bd0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.716+0000 7f8729c69740 1 -- 172.21.15.12:0/1852206274 shutdown_connections 2024-03-23T23:06:22.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.716+0000 7f8729c69740 1 --2- 172.21.15.12:0/1852206274 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560033a35580 0x560033a37c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.716+0000 7f8729c69740 1 --2- 172.21.15.12:0/1852206274 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560033a34bc0 0x560033a451e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.716+0000 7f8729c69740 1 --2- 172.21.15.12:0/1852206274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a339f0 0x560033a32b10 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.716+0000 7f8729c69740 1 -- 172.21.15.12:0/1852206274 >> 172.21.15.12:0/1852206274 conn(0x560033a2ede0 msgr2=0x560033a2e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:22.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.716+0000 7f8729c69740 1 -- 172.21.15.12:0/1852206274 shutdown_connections 2024-03-23T23:06:22.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.716+0000 7f8729c69740 1 -- 172.21.15.12:0/1852206274 wait complete. 2024-03-23T23:06:22.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.716+0000 7f8729c69740 1 -- start start 2024-03-23T23:06:22.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.717+0000 7f8729c69740 1 Processor -- start 2024-03-23T23:06:22.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.717+0000 7f8729c69740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560033a339f0 0x560033a41220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.717+0000 7f8729c69740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a34bc0 0x560033a41760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.717+0000 7f8729c69740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560033a35580 0x560033a44780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.717+0000 7f8729c69740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87201a9140 con 0x560033a339f0 2024-03-23T23:06:22.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.717+0000 7f8729c69740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87201aa440 con 0x560033a34bc0 2024-03-23T23:06:22.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.717+0000 7f8729c69740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87201a9780 con 0x560033a35580 2024-03-23T23:06:22.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.717+0000 7f8729c69740 10 client.0 _ll_register_callbacks cb 0x7f8708009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:22.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.717+0000 7f8725000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560033a339f0 0x560033a41220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.717+0000 7f8725000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560033a339f0 0x560033a41220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38106/0 (socket says 172.21.15.12:38106) 2024-03-23T23:06:22.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.717+0000 7f8717400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a34bc0 0x560033a41760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.717+0000 7f8725000640 1 -- 172.21.15.12:0/3992080812 learned_addr learned my addr 172.21.15.12:0/3992080812 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:22.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.717+0000 7f8725000640 1 --2- 172.21.15.12:0/3992080812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560033a35580 0x560033a44780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.718+0000 7f8717400640 1 -- 172.21.15.12:0/3992080812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560033a35580 msgr2=0x560033a44780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.718+0000 7f8717400640 1 --2- 172.21.15.12:0/3992080812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560033a35580 0x560033a44780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.718+0000 7f8717400640 1 -- 172.21.15.12:0/3992080812 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560033a339f0 msgr2=0x560033a41220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.718+0000 7f8717400640 1 --2- 172.21.15.12:0/3992080812 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560033a339f0 0x560033a41220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.718+0000 7f8717400640 1 -- 172.21.15.12:0/3992080812 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560033a44d70 con 0x560033a34bc0 2024-03-23T23:06:22.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.718+0000 7f8725000640 1 --2- 172.21.15.12:0/3992080812 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560033a339f0 0x560033a41220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:22.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.718+0000 7f8717400640 1 --2- 172.21.15.12:0/3992080812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a34bc0 0x560033a41760 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f87080108f0 tx=0x7f8708012860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce9c4da3b4ec802e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:22.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.718+0000 7f870e000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:06:22.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.719+0000 7f870e000640 1 -- 172.21.15.12:0/3992080812 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87080120d0 con 0x560033a34bc0 2024-03-23T23:06:22.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.719+0000 7f870e000640 1 -- 172.21.15.12:0/3992080812 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f87080051c0 con 0x560033a34bc0 2024-03-23T23:06:22.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.719+0000 7f8729c69740 1 -- 172.21.15.12:0/3992080812 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8718011190 con 0x560033a34bc0 2024-03-23T23:06:22.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.719+0000 7f870e000640 1 -- 172.21.15.12:0/3992080812 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8708012270 con 0x560033a34bc0 2024-03-23T23:06:22.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.719+0000 7f8729c69740 10 client.26516 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:22.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.719+0000 7f8729c69740 1 -- 172.21.15.12:0/3992080812 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x560033aa58f0 con 0x560033a34bc0 2024-03-23T23:06:22.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.719+0000 7f870e000640 1 -- 172.21.15.12:0/3992080812 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8718011190 con 0x560033a34bc0 2024-03-23T23:06:22.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.719+0000 7f870e000640 1 -- 172.21.15.12:0/3992080812 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f870801c470 con 0x560033a34bc0 2024-03-23T23:06:22.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.719+0000 7f8729c69740 10 client.26516 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:22.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.719+0000 7f8729c69740 10 client.26516 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:06:22.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.719+0000 7f8729c69740 20 client.26516 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:22.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.719+0000 7f8729c69740 1 --2- 172.21.15.12:0/3992080812 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x560033aa6210 0x560033aa8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.719+0000 7f8729c69740 4 client.26516 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:06:22.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.719+0000 7f8729c69740 1 -- 172.21.15.12:0/3992080812 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56003399fb50 con 0x560033aa6210 2024-03-23T23:06:22.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.720+0000 7f8725000640 1 --2- 172.21.15.12:0/3992080812 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x560033aa6210 0x560033aa8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.720+0000 7f8725000640 1 --2- 172.21.15.12:0/3992080812 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x560033aa6210 0x560033aa8630 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=db53d634ccdc7bc9 server_cookie=3d5c0050d0c24ca7 in_seq=0 out_seq=0 2024-03-23T23:06:22.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.720+0000 7f870e000640 10 client.26516 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:06:22.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.720+0000 7f870e000640 1 -- 172.21.15.12:0/3992080812 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56003399fb50 con 0x560033aa6210 2024-03-23T23:06:22.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.720+0000 7f870e000640 10 client.26516 handle_command_reply: tid=0 2024-03-23T23:06:22.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.720+0000 7f8729c69740 1 client.26516 shutdown 2024-03-23T23:06:22.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.721+0000 7f8729c69740 1 -- 172.21.15.12:0/3992080812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a34bc0 msgr2=0x560033a41760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.721+0000 7f8729c69740 1 --2- 172.21.15.12:0/3992080812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a34bc0 0x560033a41760 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f87080108f0 tx=0x7f8708012860 comp rx=0 tx=0).stop 2024-03-23T23:06:22.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.721+0000 7f8729c69740 1 -- 172.21.15.12:0/3992080812 shutdown_connections 2024-03-23T23:06:22.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.721+0000 7f8729c69740 1 --2- 172.21.15.12:0/3992080812 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x560033aa6210 0x560033aa8630 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.721+0000 7f8729c69740 1 --2- 172.21.15.12:0/3992080812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560033a35580 0x560033a44780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.721+0000 7f8729c69740 1 --2- 172.21.15.12:0/3992080812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560033a34bc0 0x560033a41760 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.721+0000 7f870e000640 0 client.26516 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:06:22.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.721+0000 7f8729c69740 1 --2- 172.21.15.12:0/3992080812 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560033a339f0 0x560033a41220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.721+0000 7f8729c69740 1 -- 172.21.15.12:0/3992080812 >> 172.21.15.12:0/3992080812 conn(0x560033a2ede0 msgr2=0x560033a3cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:22.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.721+0000 7f8729c69740 1 -- 172.21.15.12:0/3992080812 shutdown_connections 2024-03-23T23:06:22.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.721+0000 7f8729c69740 1 -- 172.21.15.12:0/3992080812 wait complete. 2024-03-23T23:06:22.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.721+0000 7f8729c69740 20 client.26516 trim_cache size 0 max 16384 2024-03-23T23:06:22.724 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:06:22.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.722+0000 7f8725a00640 1 -- 172.21.15.12:0/4282462668 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f87040a2880 msgr2=0x7f87040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.722+0000 7f8725a00640 1 --2- 172.21.15.12:0/4282462668 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f87040a2880 0x7f87040a4d40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f87100b9d30 tx=0x7f87100dd040 comp rx=0 tx=0).stop 2024-03-23T23:06:22.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.722+0000 7f8725a00640 1 -- 172.21.15.12:0/4282462668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87201037c0 msgr2=0x7f8720199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.722+0000 7f8725a00640 1 --2- 172.21.15.12:0/4282462668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87201037c0 0x7f8720199ab0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8710000c00 tx=0x7f871002c650 comp rx=0 tx=0).stop 2024-03-23T23:06:22.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.722+0000 7f8725a00640 1 -- 172.21.15.12:0/4282462668 shutdown_connections 2024-03-23T23:06:22.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.722+0000 7f8725a00640 1 --2- 172.21.15.12:0/4282462668 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f87040a2880 0x7f87040a4d40 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.722+0000 7f8725a00640 1 --2- 172.21.15.12:0/4282462668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8720105370 0x7f87201a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.722+0000 7f8725a00640 1 --2- 172.21.15.12:0/4282462668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87201049b0 0x7f8720199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.722+0000 7f8725a00640 1 --2- 172.21.15.12:0/4282462668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87201037c0 0x7f8720199ab0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.722+0000 7f8725a00640 1 -- 172.21.15.12:0/4282462668 >> 172.21.15.12:0/4282462668 conn(0x7f87200fed50 msgr2=0x7f8720107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:22.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.722+0000 7f8725a00640 1 -- 172.21.15.12:0/4282462668 shutdown_connections 2024-03-23T23:06:22.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.722+0000 7f8725a00640 1 -- 172.21.15.12:0/4282462668 wait complete. 2024-03-23T23:06:22.737 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:22 smithi012 ceph-mon[31695]: pgmap v527: 97 pgs: 97 active+clean; 2.7 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:06:22.737 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:22 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/637741123' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:06:22.740 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:06:22.740 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-23T23:06:22.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:22 smithi173 ceph-mon[36116]: pgmap v527: 97 pgs: 97 active+clean; 2.7 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:06:22.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:22 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/637741123' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:06:22.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:22 smithi203 ceph-mon[36693]: pgmap v527: 97 pgs: 97 active+clean; 2.7 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:06:22.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:22 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/637741123' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:06:22.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.966+0000 7fd5bcc00640 1 -- 172.21.15.12:0/3730380121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5b8105490 msgr2=0x7fd5b810bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.966+0000 7fd5bcc00640 1 --2- 172.21.15.12:0/3730380121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5b8105490 0x7fd5b810bd20 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0009ca0 tx=0x7fd5b002c100 comp rx=0 tx=0).stop 2024-03-23T23:06:22.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.966+0000 7fd5bcc00640 1 -- 172.21.15.12:0/3730380121 shutdown_connections 2024-03-23T23:06:22.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.966+0000 7fd5bcc00640 1 --2- 172.21.15.12:0/3730380121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5b8105490 0x7fd5b810bd20 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.966+0000 7fd5bcc00640 1 --2- 172.21.15.12:0/3730380121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5b8104ad0 0x7fd5b8104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.966+0000 7fd5bcc00640 1 --2- 172.21.15.12:0/3730380121 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5b81038e0 0x7fd5b8103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.966+0000 7fd5bcc00640 1 -- 172.21.15.12:0/3730380121 >> 172.21.15.12:0/3730380121 conn(0x7fd5b80fee50 msgr2=0x7fd5b8101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:22.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.966+0000 7fd5bcc00640 1 -- 172.21.15.12:0/3730380121 shutdown_connections 2024-03-23T23:06:22.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.966+0000 7fd5bcc00640 1 -- 172.21.15.12:0/3730380121 wait complete. 2024-03-23T23:06:22.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.967+0000 7fd5bcc00640 1 Processor -- start 2024-03-23T23:06:22.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.967+0000 7fd5bcc00640 1 -- start start 2024-03-23T23:06:22.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.968+0000 7fd5bcc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5b81038e0 0x7fd5b8199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.968+0000 7fd5bcc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5b8104ad0 0x7fd5b819a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.968+0000 7fd5bcc00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5b8105490 0x7fd5b81a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.968+0000 7fd5bcc00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b810f560 con 0x7fd5b8105490 2024-03-23T23:06:22.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.968+0000 7fd5bcc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b810f3e0 con 0x7fd5b8104ad0 2024-03-23T23:06:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.968+0000 7fd5bcc00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b810f6e0 con 0x7fd5b81038e0 2024-03-23T23:06:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.968+0000 7fd5b5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5b81038e0 0x7fd5b8199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.968+0000 7fd5afe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5b8104ad0 0x7fd5b819a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.968+0000 7fd5b5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5b81038e0 0x7fd5b8199b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55596/0 (socket says 172.21.15.12:55596) 2024-03-23T23:06:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.968+0000 7fd5b5e00640 1 -- 172.21.15.12:0/4137113273 learned_addr learned my addr 172.21.15.12:0/4137113273 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.968+0000 7fd5b6800640 1 --2- 172.21.15.12:0/4137113273 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5b8105490 0x7fd5b81a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.969+0000 7fd5b5e00640 1 -- 172.21.15.12:0/4137113273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5b8104ad0 msgr2=0x7fd5b819a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.969+0000 7fd5b5e00640 1 --2- 172.21.15.12:0/4137113273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5b8104ad0 0x7fd5b819a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.969+0000 7fd5b5e00640 1 -- 172.21.15.12:0/4137113273 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5b8105490 msgr2=0x7fd5b81a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.969+0000 7fd5b5e00640 1 --2- 172.21.15.12:0/4137113273 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5b8105490 0x7fd5b81a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.969+0000 7fd5b5e00640 1 -- 172.21.15.12:0/4137113273 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b81a1790 con 0x7fd5b81038e0 2024-03-23T23:06:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.969+0000 7fd5afe00640 1 --2- 172.21.15.12:0/4137113273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5b8104ad0 0x7fd5b819a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.969+0000 7fd5b5e00640 1 --2- 172.21.15.12:0/4137113273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5b81038e0 0x7fd5b8199b70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800e9c0 tx=0x7fd5a800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.969+0000 7fd5aea00640 1 -- 172.21.15.12:0/4137113273 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a800cde0 con 0x7fd5b81038e0 2024-03-23T23:06:22.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.969+0000 7fd5aea00640 1 -- 172.21.15.12:0/4137113273 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5a8010910 con 0x7fd5b81038e0 2024-03-23T23:06:22.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.969+0000 7fd5aea00640 1 -- 172.21.15.12:0/4137113273 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8018a00 con 0x7fd5b81038e0 2024-03-23T23:06:22.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.969+0000 7fd5bcc00640 1 -- 172.21.15.12:0/4137113273 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b81a1a80 con 0x7fd5b81038e0 2024-03-23T23:06:22.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.969+0000 7fd5bcc00640 1 -- 172.21.15.12:0/4137113273 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b81a2020 con 0x7fd5b81038e0 2024-03-23T23:06:22.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.972+0000 7fd5bf981740 1 Processor -- start 2024-03-23T23:06:22.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.973+0000 7fd5aea00640 1 -- 172.21.15.12:0/4137113273 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd5a801f020 con 0x7fd5b81038e0 2024-03-23T23:06:22.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.973+0000 7fd5bf981740 1 -- start start 2024-03-23T23:06:22.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.973+0000 7fd5bf981740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab7810 0x5644e9ab7c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.973+0000 7fd5bf981740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644e9ab89e0 0x5644e9ab8e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.973+0000 7fd5bf981740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644e9ab93a0 0x5644e9abfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.973+0000 7fd5bf981740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644e9a1c2c0 con 0x5644e9ab89e0 2024-03-23T23:06:22.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.973+0000 7fd5bf981740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644e99c4c90 con 0x5644e9ab7810 2024-03-23T23:06:22.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.973+0000 7fd5bf981740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644e9a18360 con 0x5644e9ab93a0 2024-03-23T23:06:22.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.973+0000 7fd5b6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab7810 0x5644e9ab7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.973+0000 7fd5b5e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644e9ab89e0 0x5644e9ab8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.973+0000 7fd5afe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644e9ab93a0 0x5644e9abfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.973+0000 7fd5b6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab7810 0x5644e9ab7c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60940/0 (socket says 172.21.15.12:60940) 2024-03-23T23:06:22.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.973+0000 7fd5b6800640 1 -- 172.21.15.12:0/1739024965 learned_addr learned my addr 172.21.15.12:0/1739024965 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:22.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.974+0000 7fd5b6800640 1 -- 172.21.15.12:0/1739024965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644e9ab93a0 msgr2=0x5644e9abfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.974+0000 7fd5b6800640 1 --2- 172.21.15.12:0/1739024965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644e9ab93a0 0x5644e9abfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.974+0000 7fd5b6800640 1 -- 172.21.15.12:0/1739024965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644e9ab89e0 msgr2=0x5644e9ab8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.974+0000 7fd5b6800640 1 --2- 172.21.15.12:0/1739024965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644e9ab89e0 0x5644e9ab8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.974+0000 7fd5b6800640 1 -- 172.21.15.12:0/1739024965 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5644e99e9080 con 0x5644e9ab7810 2024-03-23T23:06:22.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.974+0000 7fd5afe00640 1 --2- 172.21.15.12:0/1739024965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644e9ab93a0 0x5644e9abfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:22.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.974+0000 7fd5b6800640 1 --2- 172.21.15.12:0/1739024965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab7810 0x5644e9ab7c30 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd5b002c710 tx=0x7fd5b00323d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba7f99f234ac6417 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:22.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.975+0000 7fd5a7000640 1 -- 172.21.15.12:0/1739024965 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b0043070 con 0x5644e9ab7810 2024-03-23T23:06:22.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.975+0000 7fd5a7000640 1 -- 172.21.15.12:0/1739024965 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5b0032c90 con 0x5644e9ab7810 2024-03-23T23:06:22.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.975+0000 7fd5a7000640 1 -- 172.21.15.12:0/1739024965 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b0039040 con 0x5644e9ab7810 2024-03-23T23:06:22.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.975+0000 7fd5aea00640 1 --2- 172.21.15.12:0/4137113273 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd59c0a2880 0x7fd59c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.975+0000 7fd5aea00640 1 -- 172.21.15.12:0/4137113273 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd5a80cec10 con 0x7fd5b81038e0 2024-03-23T23:06:22.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.975+0000 7fd5b5e00640 1 --2- 172.21.15.12:0/4137113273 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd59c0a2880 0x7fd59c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.975+0000 7fd5bf981740 1 -- 172.21.15.12:0/1739024965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab7810 msgr2=0x5644e9ab7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.975+0000 7fd5bf981740 1 --2- 172.21.15.12:0/1739024965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab7810 0x5644e9ab7c30 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd5b002c710 tx=0x7fd5b00323d0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.976+0000 7fd5bf981740 1 -- 172.21.15.12:0/1739024965 shutdown_connections 2024-03-23T23:06:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.976+0000 7fd5bf981740 1 --2- 172.21.15.12:0/1739024965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644e9ab93a0 0x5644e9abfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.976+0000 7fd5bf981740 1 --2- 172.21.15.12:0/1739024965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644e9ab89e0 0x5644e9ab8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.976+0000 7fd5bf981740 1 --2- 172.21.15.12:0/1739024965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab7810 0x5644e9ab7c30 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.976+0000 7fd5bf981740 1 -- 172.21.15.12:0/1739024965 >> 172.21.15.12:0/1739024965 conn(0x5644e9ab2c00 msgr2=0x5644e9ab3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.976+0000 7fd5bf981740 1 -- 172.21.15.12:0/1739024965 shutdown_connections 2024-03-23T23:06:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.976+0000 7fd5bf981740 1 -- 172.21.15.12:0/1739024965 wait complete. 2024-03-23T23:06:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.976+0000 7fd5b5e00640 1 --2- 172.21.15.12:0/4137113273 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd59c0a2880 0x7fd59c0a4d40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8000c00 tx=0x7fd5a80d2650 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.976+0000 7fd5bf981740 1 -- start start 2024-03-23T23:06:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.976+0000 7fd5bf981740 1 Processor -- start 2024-03-23T23:06:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.977+0000 7fd5bf981740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab7810 0x5644e9ac19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.977+0000 7fd5bf981740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644e9ab89e0 0x5644e9ac1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.977+0000 7fd5bf981740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644e9ab93a0 0x5644e9ac8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.977+0000 7fd5bf981740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644e9ac0350 con 0x5644e9ab89e0 2024-03-23T23:06:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.977+0000 7fd5bf981740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644e99d80d0 con 0x5644e9ab7810 2024-03-23T23:06:22.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.977+0000 7fd5bf981740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644e9ac04d0 con 0x5644e9ab93a0 2024-03-23T23:06:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.977+0000 7fd5bf981740 10 client.0 _ll_register_callbacks cb 0x5644e99dce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.977+0000 7fd5b6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab7810 0x5644e9ac19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.977+0000 7fd5afe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644e9ab89e0 0x5644e9ac1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.977+0000 7fd5b6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab7810 0x5644e9ac19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60942/0 (socket says 172.21.15.12:60942) 2024-03-23T23:06:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.977+0000 7fd5b6800640 1 -- 172.21.15.12:0/1395760334 learned_addr learned my addr 172.21.15.12:0/1395760334 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.977+0000 7fd5b6800640 1 --2- 172.21.15.12:0/1395760334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644e9ab93a0 0x5644e9ac8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.978+0000 7fd5b6800640 1 -- 172.21.15.12:0/1395760334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644e9ab93a0 msgr2=0x5644e9ac8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.978+0000 7fd5b6800640 1 --2- 172.21.15.12:0/1395760334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644e9ab93a0 0x5644e9ac8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.978+0000 7fd5b6800640 1 -- 172.21.15.12:0/1395760334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644e9ab89e0 msgr2=0x5644e9ac1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.978+0000 7fd5b6800640 1 --2- 172.21.15.12:0/1395760334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644e9ab89e0 0x5644e9ac1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.978+0000 7fd5b6800640 1 -- 172.21.15.12:0/1395760334 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5644e97303f0 con 0x5644e9ab7810 2024-03-23T23:06:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.978+0000 7fd5b6800640 1 --2- 172.21.15.12:0/1395760334 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab7810 0x5644e9ac19f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0009ca0 tx=0x7fd5b002e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f551b8dbbb519d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.978+0000 7fd59b400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:06:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.978+0000 7fd59b400640 1 -- 172.21.15.12:0/1395760334 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b0039070 con 0x5644e9ab7810 2024-03-23T23:06:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.978+0000 7fd59b400640 1 -- 172.21.15.12:0/1395760334 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd5b0004360 con 0x5644e9ab7810 2024-03-23T23:06:22.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.978+0000 7fd5bf981740 1 -- 172.21.15.12:0/1395760334 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd5b0032c90 con 0x5644e9ab7810 2024-03-23T23:06:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.978+0000 7fd59b400640 1 -- 172.21.15.12:0/1395760334 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5b0043040 con 0x5644e9ab7810 2024-03-23T23:06:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.978+0000 7fd5bf981740 10 client.26528 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.978+0000 7fd5bf981740 1 -- 172.21.15.12:0/1395760334 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5644e97fa7d0 con 0x5644e9ab7810 2024-03-23T23:06:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.979+0000 7fd59b400640 1 -- 172.21.15.12:0/1395760334 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5b0032c90 con 0x5644e9ab7810 2024-03-23T23:06:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.979+0000 7fd59b400640 1 -- 172.21.15.12:0/1395760334 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fd5b0048ad0 con 0x5644e9ab7810 2024-03-23T23:06:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.979+0000 7fd5bf981740 10 client.26528 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.979+0000 7fd5bf981740 10 client.26528 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:06:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.979+0000 7fd5bf981740 20 client.26528 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.979+0000 7fd5bf981740 1 --2- 172.21.15.12:0/1395760334 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5644e9acd0d0 0x5644e9acf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.979+0000 7fd5bf981740 4 client.26528 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:06:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.979+0000 7fd5bf981740 1 -- 172.21.15.12:0/1395760334 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5644e9a29800 con 0x5644e9acd0d0 2024-03-23T23:06:22.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.979+0000 7fd5afe00640 1 --2- 172.21.15.12:0/1395760334 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5644e9acd0d0 0x5644e9acf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.980+0000 7fd5afe00640 1 --2- 172.21.15.12:0/1395760334 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5644e9acd0d0 0x5644e9acf4c0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55dd2b0ffef9014c server_cookie=45aa9b97f0f10fff in_seq=0 out_seq=0 2024-03-23T23:06:22.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.980+0000 7fd59b400640 10 client.26528 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:22.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.980+0000 7fd59b400640 1 -- 172.21.15.12:0/1395760334 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5644e9a29800 con 0x5644e9acd0d0 2024-03-23T23:06:22.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.980+0000 7fd59b400640 10 client.26528 handle_command_reply: tid=0 2024-03-23T23:06:22.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.981+0000 7fd5bf981740 1 client.26528 shutdown 2024-03-23T23:06:22.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.981+0000 7fd5bf981740 1 -- 172.21.15.12:0/1395760334 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab7810 msgr2=0x5644e9ac19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.981+0000 7fd5bf981740 1 --2- 172.21.15.12:0/1395760334 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab7810 0x5644e9ac19f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0009ca0 tx=0x7fd5b002e040 comp rx=0 tx=0).stop 2024-03-23T23:06:22.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.981+0000 7fd5bf981740 1 -- 172.21.15.12:0/1395760334 shutdown_connections 2024-03-23T23:06:22.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.981+0000 7fd5bf981740 1 --2- 172.21.15.12:0/1395760334 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5644e9acd0d0 0x5644e9acf4c0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.981+0000 7fd5bf981740 1 --2- 172.21.15.12:0/1395760334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644e9ab93a0 0x5644e9ac8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.981+0000 7fd5bf981740 1 --2- 172.21.15.12:0/1395760334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644e9ab89e0 0x5644e9ac1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.981+0000 7fd5bf981740 1 --2- 172.21.15.12:0/1395760334 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab7810 0x5644e9ac19f0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.981+0000 7fd59b400640 0 client.26528 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:22.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.981+0000 7fd5bf981740 1 -- 172.21.15.12:0/1395760334 >> 172.21.15.12:0/1395760334 conn(0x5644e9ab2c00 msgr2=0x5644e9ab1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:22.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.981+0000 7fd5bf981740 1 -- 172.21.15.12:0/1395760334 shutdown_connections 2024-03-23T23:06:22.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.981+0000 7fd5bf981740 1 -- 172.21.15.12:0/1395760334 wait complete. 2024-03-23T23:06:22.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.981+0000 7fd5bf981740 20 client.26528 trim_cache size 0 max 16384 2024-03-23T23:06:22.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.995+0000 7fd5bf981740 1 Processor -- start 2024-03-23T23:06:22.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.995+0000 7fd5bf981740 1 -- start start 2024-03-23T23:06:22.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.995+0000 7fd5bf981740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644e9ab7810 0x5644e9ab6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.995+0000 7fd5bf981740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644e9ab89e0 0x5644e9ac8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.995+0000 7fd5bf981740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab93a0 0x5644e9ab1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:22.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.995+0000 7fd5bf981740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b81ad140 con 0x5644e9ab7810 2024-03-23T23:06:22.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.995+0000 7fd5bf981740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b81aac40 con 0x5644e9ab93a0 2024-03-23T23:06:22.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.995+0000 7fd5bf981740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b81a8ce0 con 0x5644e9ab89e0 2024-03-23T23:06:22.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.995+0000 7fd5afe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644e9ab89e0 0x5644e9ac8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.996+0000 7fd5afe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644e9ab89e0 0x5644e9ac8f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55636/0 (socket says 172.21.15.12:55636) 2024-03-23T23:06:22.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.996+0000 7fd5afe00640 1 -- 172.21.15.12:0/120791670 learned_addr learned my addr 172.21.15.12:0/120791670 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:22.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.996+0000 7fd5b6800640 1 --2- 172.21.15.12:0/120791670 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644e9ab7810 0x5644e9ab6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.996+0000 7fd5b6800640 1 --2- 172.21.15.12:0/120791670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab93a0 0x5644e9ab1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:22.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.996+0000 7fd5afe00640 1 -- 172.21.15.12:0/120791670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab93a0 msgr2=0x5644e9ab1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.996+0000 7fd5afe00640 1 --2- 172.21.15.12:0/120791670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab93a0 0x5644e9ab1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.996+0000 7fd5afe00640 1 -- 172.21.15.12:0/120791670 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644e9ab7810 msgr2=0x5644e9ab6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:22.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.996+0000 7fd5afe00640 1 --2- 172.21.15.12:0/120791670 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644e9ab7810 0x5644e9ab6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.996+0000 7fd5afe00640 1 -- 172.21.15.12:0/120791670 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5644e9acf830 con 0x5644e9ab89e0 2024-03-23T23:06:22.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.996+0000 7fd5b6800640 1 --2- 172.21.15.12:0/120791670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab93a0 0x5644e9ab1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:06:22.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.996+0000 7fd5afe00640 1 --2- 172.21.15.12:0/120791670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644e9ab89e0 0x5644e9ac8f30 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x5644e9ab3b70 tx=0x7fd5a0005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8aff4956d0904f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:22.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.996+0000 7fd5a6600640 1 -- 172.21.15.12:0/120791670 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a0012310 con 0x5644e9ab89e0 2024-03-23T23:06:22.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.997+0000 7fd5a6600640 1 -- 172.21.15.12:0/120791670 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5a0025070 con 0x5644e9ab89e0 2024-03-23T23:06:22.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.997+0000 7fd5a6600640 1 -- 172.21.15.12:0/120791670 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a00038e0 con 0x5644e9ab89e0 2024-03-23T23:06:22.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.997+0000 7fd5bf981740 1 -- 172.21.15.12:0/120791670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644e9ab89e0 msgr2=0x5644e9ac8f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:22.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.997+0000 7fd5bf981740 1 --2- 172.21.15.12:0/120791670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644e9ab89e0 0x5644e9ac8f30 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x5644e9ab3b70 tx=0x7fd5a0005f90 comp rx=0 tx=0).stop 2024-03-23T23:06:22.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.997+0000 7fd5bf981740 1 -- 172.21.15.12:0/120791670 shutdown_connections 2024-03-23T23:06:22.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.997+0000 7fd5bf981740 1 --2- 172.21.15.12:0/120791670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab93a0 0x5644e9ab1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.997+0000 7fd5bf981740 1 --2- 172.21.15.12:0/120791670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644e9ab89e0 0x5644e9ac8f30 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.997+0000 7fd5bf981740 1 --2- 172.21.15.12:0/120791670 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644e9ab7810 0x5644e9ab6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:22.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.997+0000 7fd5bf981740 1 -- 172.21.15.12:0/120791670 >> 172.21.15.12:0/120791670 conn(0x5644e9ab2c00 msgr2=0x5644e9abc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:22.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.997+0000 7fd5bf981740 1 -- 172.21.15.12:0/120791670 shutdown_connections 2024-03-23T23:06:22.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.997+0000 7fd5bf981740 1 -- 172.21.15.12:0/120791670 wait complete. 2024-03-23T23:06:23.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.998+0000 7fd5bf981740 1 -- start start 2024-03-23T23:06:23.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.998+0000 7fd5bf981740 1 Processor -- start 2024-03-23T23:06:23.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.998+0000 7fd5bf981740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab7810 0x5644e9ac6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:23.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.998+0000 7fd5bf981740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644e9ab89e0 0x5644e9ac7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:23.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.998+0000 7fd5bf981740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644e9ab93a0 0x5644e9ac83e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:23.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.998+0000 7fd5bf981740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b81a92e0 con 0x5644e9ab89e0 2024-03-23T23:06:23.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.998+0000 7fd5bf981740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b81aa5e0 con 0x5644e9ab7810 2024-03-23T23:06:23.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.998+0000 7fd5bf981740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b81a9920 con 0x5644e9ab93a0 2024-03-23T23:06:23.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.998+0000 7fd5bf981740 10 client.0 _ll_register_callbacks cb 0x7fd5a80d2520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:23.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.998+0000 7fd5afe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644e9ab89e0 0x5644e9ac7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:23.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.999+0000 7fd5b6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab7810 0x5644e9ac6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:23.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.999+0000 7fd5afe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644e9ab89e0 0x5644e9ac7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38150/0 (socket says 172.21.15.12:38150) 2024-03-23T23:06:23.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.999+0000 7fd5afe00640 1 -- 172.21.15.12:0/2430305051 learned_addr learned my addr 172.21.15.12:0/2430305051 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:23.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.999+0000 7fd5b6800640 1 --2- 172.21.15.12:0/2430305051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644e9ab93a0 0x5644e9ac83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:23.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.999+0000 7fd5afe00640 1 -- 172.21.15.12:0/2430305051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644e9ab93a0 msgr2=0x5644e9ac83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:23.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.999+0000 7fd5afe00640 1 --2- 172.21.15.12:0/2430305051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644e9ab93a0 0x5644e9ac83e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:23.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.999+0000 7fd5afe00640 1 -- 172.21.15.12:0/2430305051 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab7810 msgr2=0x5644e9ac6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:23.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.999+0000 7fd5afe00640 1 --2- 172.21.15.12:0/2430305051 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab7810 0x5644e9ac6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:23.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.999+0000 7fd5afe00640 1 -- 172.21.15.12:0/2430305051 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5644e9ac89d0 con 0x5644e9ab89e0 2024-03-23T23:06:23.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.999+0000 7fd5b6800640 1 --2- 172.21.15.12:0/2430305051 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab7810 0x5644e9ac6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:06:23.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.999+0000 7fd5afe00640 1 --2- 172.21.15.12:0/2430305051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644e9ab89e0 0x5644e9ac7470 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0024000 tx=0x7fd5a00200f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45d8e5e7ae5939a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:23.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.999+0000 7fd59b400640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:06:23.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:22.999+0000 7fd59b400640 1 -- 172.21.15.12:0/2430305051 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a002d030 con 0x5644e9ab89e0 2024-03-23T23:06:23.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.000+0000 7fd59b400640 1 -- 172.21.15.12:0/2430305051 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd5a0020770 con 0x5644e9ab89e0 2024-03-23T23:06:23.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.000+0000 7fd5bf981740 1 -- 172.21.15.12:0/2430305051 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd5a0025070 con 0x5644e9ab89e0 2024-03-23T23:06:23.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.000+0000 7fd59b400640 1 -- 172.21.15.12:0/2430305051 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5a0020970 con 0x5644e9ab89e0 2024-03-23T23:06:23.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.000+0000 7fd5bf981740 10 client.16410 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:23.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.000+0000 7fd5bf981740 1 -- 172.21.15.12:0/2430305051 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5644e9b29060 con 0x5644e9ab89e0 2024-03-23T23:06:23.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.000+0000 7fd59b400640 1 -- 172.21.15.12:0/2430305051 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5a0025070 con 0x5644e9ab89e0 2024-03-23T23:06:23.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.000+0000 7fd59b400640 1 -- 172.21.15.12:0/2430305051 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fd5a0010430 con 0x5644e9ab89e0 2024-03-23T23:06:23.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.000+0000 7fd5bf981740 10 client.16410 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:23.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.000+0000 7fd5bf981740 10 client.16410 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:06:23.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.000+0000 7fd5bf981740 20 client.16410 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:23.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.000+0000 7fd5bf981740 1 --2- 172.21.15.12:0/2430305051 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5644e9b299c0 0x5644e9b2bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:23.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.000+0000 7fd5bf981740 4 client.16410 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:06:23.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.000+0000 7fd5bf981740 1 -- 172.21.15.12:0/2430305051 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5644e9a23b50 con 0x5644e9b299c0 2024-03-23T23:06:23.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.001+0000 7fd5b6800640 1 --2- 172.21.15.12:0/2430305051 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5644e9b299c0 0x5644e9b2bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:23.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.001+0000 7fd5b6800640 1 --2- 172.21.15.12:0/2430305051 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5644e9b299c0 0x5644e9b2bde0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5b8b3ba4b5fcf20 server_cookie=138bbba3a7693d76 in_seq=0 out_seq=0 2024-03-23T23:06:23.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.001+0000 7fd59b400640 10 client.16410 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:23.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.001+0000 7fd59b400640 1 -- 172.21.15.12:0/2430305051 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5644e9a23b50 con 0x5644e9b299c0 2024-03-23T23:06:23.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.001+0000 7fd59b400640 10 client.16410 handle_command_reply: tid=0 2024-03-23T23:06:23.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.001+0000 7fd5bf981740 1 client.16410 shutdown 2024-03-23T23:06:23.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.002+0000 7fd5bf981740 1 -- 172.21.15.12:0/2430305051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644e9ab89e0 msgr2=0x5644e9ac7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:23.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.002+0000 7fd5bf981740 1 --2- 172.21.15.12:0/2430305051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644e9ab89e0 0x5644e9ac7470 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0024000 tx=0x7fd5a00200f0 comp rx=0 tx=0).stop 2024-03-23T23:06:23.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.002+0000 7fd5bf981740 1 -- 172.21.15.12:0/2430305051 shutdown_connections 2024-03-23T23:06:23.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.002+0000 7fd5bf981740 1 --2- 172.21.15.12:0/2430305051 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5644e9b299c0 0x5644e9b2bde0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:23.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.002+0000 7fd5bf981740 1 --2- 172.21.15.12:0/2430305051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644e9ab93a0 0x5644e9ac83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:23.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.002+0000 7fd5bf981740 1 --2- 172.21.15.12:0/2430305051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644e9ab89e0 0x5644e9ac7470 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:23.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.002+0000 7fd59b400640 0 client.16410 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:23.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.002+0000 7fd5bf981740 1 --2- 172.21.15.12:0/2430305051 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5644e9ab7810 0x5644e9ac6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:23.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.002+0000 7fd5bf981740 1 -- 172.21.15.12:0/2430305051 >> 172.21.15.12:0/2430305051 conn(0x5644e9ab2c00 msgr2=0x5644e9ad3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:23.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.002+0000 7fd5bf981740 1 -- 172.21.15.12:0/2430305051 shutdown_connections 2024-03-23T23:06:23.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.002+0000 7fd5bf981740 1 -- 172.21.15.12:0/2430305051 wait complete. 2024-03-23T23:06:23.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.002+0000 7fd5bf981740 20 client.16410 trim_cache size 0 max 16384 2024-03-23T23:06:23.005 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:06:23.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.003+0000 7fd5bcc00640 1 -- 172.21.15.12:0/4137113273 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd59c0a2880 msgr2=0x7fd59c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:23.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.003+0000 7fd5bcc00640 1 --2- 172.21.15.12:0/4137113273 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd59c0a2880 0x7fd59c0a4d40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8000c00 tx=0x7fd5a80d2650 comp rx=0 tx=0).stop 2024-03-23T23:06:23.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.003+0000 7fd5bcc00640 1 -- 172.21.15.12:0/4137113273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5b81038e0 msgr2=0x7fd5b8199b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:23.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.003+0000 7fd5bcc00640 1 --2- 172.21.15.12:0/4137113273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5b81038e0 0x7fd5b8199b70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800e9c0 tx=0x7fd5a800ee80 comp rx=0 tx=0).stop 2024-03-23T23:06:23.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.003+0000 7fd5bcc00640 1 -- 172.21.15.12:0/4137113273 shutdown_connections 2024-03-23T23:06:23.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.003+0000 7fd5bcc00640 1 --2- 172.21.15.12:0/4137113273 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd59c0a2880 0x7fd59c0a4d40 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:23.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.003+0000 7fd5bcc00640 1 --2- 172.21.15.12:0/4137113273 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5b8105490 0x7fd5b81a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:23.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.003+0000 7fd5bcc00640 1 --2- 172.21.15.12:0/4137113273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5b8104ad0 0x7fd5b819a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:23.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.003+0000 7fd5bcc00640 1 --2- 172.21.15.12:0/4137113273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5b81038e0 0x7fd5b8199b70 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:23.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.003+0000 7fd5bcc00640 1 -- 172.21.15.12:0/4137113273 >> 172.21.15.12:0/4137113273 conn(0x7fd5b80fee50 msgr2=0x7fd5b8107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:23.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.003+0000 7fd5bcc00640 1 -- 172.21.15.12:0/4137113273 shutdown_connections 2024-03-23T23:06:23.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:23.003+0000 7fd5bcc00640 1 -- 172.21.15.12:0/4137113273 wait complete. 2024-03-23T23:06:24.020 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-23T23:06:24.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:06:24.021 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-23T23:06:24.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.265+0000 7fe825000640 1 -- 172.21.15.12:0/2491680398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe820101580 msgr2=0x7fe8201019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:24.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.265+0000 7fe825000640 1 --2- 172.21.15.12:0/2491680398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe820101580 0x7fe8201019a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fe810009900 tx=0x7fe81002bc30 comp rx=0 tx=0).stop 2024-03-23T23:06:24.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.265+0000 7fe825000640 1 -- 172.21.15.12:0/2491680398 shutdown_connections 2024-03-23T23:06:24.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.265+0000 7fe825000640 1 --2- 172.21.15.12:0/2491680398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe820103130 0x7fe8201099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.265+0000 7fe825000640 1 --2- 172.21.15.12:0/2491680398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe820102770 0x7fe820102bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.265+0000 7fe825000640 1 --2- 172.21.15.12:0/2491680398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe820101580 0x7fe8201019a0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.265+0000 7fe825000640 1 -- 172.21.15.12:0/2491680398 >> 172.21.15.12:0/2491680398 conn(0x7fe8200fcb10 msgr2=0x7fe8200fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:24.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.265+0000 7fe825000640 1 -- 172.21.15.12:0/2491680398 shutdown_connections 2024-03-23T23:06:24.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.265+0000 7fe825000640 1 -- 172.21.15.12:0/2491680398 wait complete. 2024-03-23T23:06:24.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.266+0000 7fe825000640 1 Processor -- start 2024-03-23T23:06:24.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.266+0000 7fe825000640 1 -- start start 2024-03-23T23:06:24.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.267+0000 7fe825000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe820101580 0x7fe8201978b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.267+0000 7fe825000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe820102770 0x7fe820197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.267+0000 7fe825000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe820103130 0x7fe82019ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.267+0000 7fe825000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82010d240 con 0x7fe820101580 2024-03-23T23:06:24.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.267+0000 7fe825000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82010d0c0 con 0x7fe820103130 2024-03-23T23:06:24.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.267+0000 7fe825000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82010d3c0 con 0x7fe820102770 2024-03-23T23:06:24.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.267+0000 7fe817e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe820102770 0x7fe820197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.267+0000 7fe81f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe820101580 0x7fe8201978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.267+0000 7fe817e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe820102770 0x7fe820197df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54800/0 (socket says 172.21.15.12:54800) 2024-03-23T23:06:24.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.267+0000 7fe81fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe820103130 0x7fe82019ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.267+0000 7fe817e00640 1 -- 172.21.15.12:0/1392860850 learned_addr learned my addr 172.21.15.12:0/1392860850 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:24.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.267+0000 7fe817e00640 1 -- 172.21.15.12:0/1392860850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe820103130 msgr2=0x7fe82019ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:24.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.267+0000 7fe817e00640 1 --2- 172.21.15.12:0/1392860850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe820103130 0x7fe82019ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.267+0000 7fe817e00640 1 -- 172.21.15.12:0/1392860850 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe820101580 msgr2=0x7fe8201978b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:24.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.267+0000 7fe817e00640 1 --2- 172.21.15.12:0/1392860850 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe820101580 0x7fe8201978b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.267+0000 7fe817e00640 1 -- 172.21.15.12:0/1392860850 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe82019f4d0 con 0x7fe820102770 2024-03-23T23:06:24.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.267+0000 7fe81fe00640 1 --2- 172.21.15.12:0/1392860850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe820103130 0x7fe82019ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:24.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.268+0000 7fe817e00640 1 --2- 172.21.15.12:0/1392860850 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe820102770 0x7fe820197df0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe80800e9f0 tx=0x7fe80800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:24.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.268+0000 7fe81cc00640 1 -- 172.21.15.12:0/1392860850 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80800cd80 con 0x7fe820102770 2024-03-23T23:06:24.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.268+0000 7fe825000640 1 -- 172.21.15.12:0/1392860850 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe82019f7c0 con 0x7fe820102770 2024-03-23T23:06:24.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.268+0000 7fe81cc00640 1 -- 172.21.15.12:0/1392860850 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe808005500 con 0x7fe820102770 2024-03-23T23:06:24.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.268+0000 7fe81cc00640 1 -- 172.21.15.12:0/1392860850 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe808010690 con 0x7fe820102770 2024-03-23T23:06:24.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.268+0000 7fe825000640 1 -- 172.21.15.12:0/1392860850 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe82019fcc0 con 0x7fe820102770 2024-03-23T23:06:24.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.271+0000 7fe829207740 1 Processor -- start 2024-03-23T23:06:24.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.271+0000 7fe829207740 1 -- start start 2024-03-23T23:06:24.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.271+0000 7fe81cc00640 1 -- 172.21.15.12:0/1392860850 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe8080046c0 con 0x7fe820102770 2024-03-23T23:06:24.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.272+0000 7fe829207740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55582b53c2b0 0x55582b53c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.272+0000 7fe829207740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55582b53cca0 0x55582b5417a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.272+0000 7fe829207740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55582b541e10 0x55582b5442a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.272+0000 7fe829207740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582b4a1eb0 con 0x55582b541e10 2024-03-23T23:06:24.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.272+0000 7fe829207740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582b44aa60 con 0x55582b53c2b0 2024-03-23T23:06:24.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.272+0000 7fe829207740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582b49df50 con 0x55582b53cca0 2024-03-23T23:06:24.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.272+0000 7fe81fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55582b53c2b0 0x55582b53c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.272+0000 7fe817e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55582b541e10 0x55582b5442a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.272+0000 7fe81f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55582b53cca0 0x55582b5417a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.272+0000 7fe81fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55582b53c2b0 0x55582b53c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59540/0 (socket says 172.21.15.12:59540) 2024-03-23T23:06:24.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.272+0000 7fe81fe00640 1 -- 172.21.15.12:0/3736592081 learned_addr learned my addr 172.21.15.12:0/3736592081 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:24.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.272+0000 7fe81f400640 1 -- 172.21.15.12:0/3736592081 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55582b53c2b0 msgr2=0x55582b53c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:24.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.272+0000 7fe81f400640 1 --2- 172.21.15.12:0/3736592081 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55582b53c2b0 0x55582b53c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.272+0000 7fe81f400640 1 -- 172.21.15.12:0/3736592081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55582b541e10 msgr2=0x55582b5442a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:24.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.272+0000 7fe81f400640 1 --2- 172.21.15.12:0/3736592081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55582b541e10 0x55582b5442a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.272+0000 7fe81f400640 1 -- 172.21.15.12:0/3736592081 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55582b27e7e0 con 0x55582b53cca0 2024-03-23T23:06:24.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.273+0000 7fe81f400640 1 --2- 172.21.15.12:0/3736592081 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55582b53cca0 0x55582b5417a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe810009230 tx=0x7fe810009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=139a53092319f15c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:24.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.273+0000 7fe814c00640 1 -- 172.21.15.12:0/3736592081 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81002c980 con 0x55582b53cca0 2024-03-23T23:06:24.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.273+0000 7fe814c00640 1 -- 172.21.15.12:0/3736592081 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe81002cb20 con 0x55582b53cca0 2024-03-23T23:06:24.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.273+0000 7fe814c00640 1 -- 172.21.15.12:0/3736592081 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe810004920 con 0x55582b53cca0 2024-03-23T23:06:24.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.273+0000 7fe81cc00640 1 --2- 172.21.15.12:0/1392860850 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe8040a2800 0x7fe8040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.273+0000 7fe81cc00640 1 -- 172.21.15.12:0/1392860850 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe8080cee40 con 0x7fe820102770 2024-03-23T23:06:24.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.274+0000 7fe81fe00640 1 --2- 172.21.15.12:0/1392860850 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe8040a2800 0x7fe8040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.274+0000 7fe829207740 1 -- 172.21.15.12:0/3736592081 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55582b53cca0 msgr2=0x55582b5417a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:24.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.274+0000 7fe829207740 1 --2- 172.21.15.12:0/3736592081 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55582b53cca0 0x55582b5417a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe810009230 tx=0x7fe810009560 comp rx=0 tx=0).stop 2024-03-23T23:06:24.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.274+0000 7fe829207740 1 -- 172.21.15.12:0/3736592081 shutdown_connections 2024-03-23T23:06:24.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.274+0000 7fe829207740 1 --2- 172.21.15.12:0/3736592081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55582b541e10 0x55582b5442a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.274+0000 7fe829207740 1 --2- 172.21.15.12:0/3736592081 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55582b53cca0 0x55582b5417a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.274+0000 7fe829207740 1 --2- 172.21.15.12:0/3736592081 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55582b53c2b0 0x55582b53c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.274+0000 7fe829207740 1 -- 172.21.15.12:0/3736592081 >> 172.21.15.12:0/3736592081 conn(0x55582b537820 msgr2=0x55582b537c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:24.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.274+0000 7fe829207740 1 -- 172.21.15.12:0/3736592081 shutdown_connections 2024-03-23T23:06:24.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.274+0000 7fe829207740 1 -- 172.21.15.12:0/3736592081 wait complete. 2024-03-23T23:06:24.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.274+0000 7fe81fe00640 1 --2- 172.21.15.12:0/1392860850 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe8040a2800 0x7fe8040a4cc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x55582b53d500 tx=0x7fe818009600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:24.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.275+0000 7fe829207740 1 -- start start 2024-03-23T23:06:24.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.275+0000 7fe829207740 1 Processor -- start 2024-03-23T23:06:24.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.275+0000 7fe829207740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55582b53c2b0 0x55582b546520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.275+0000 7fe829207740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55582b53cca0 0x55582b546a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.275+0000 7fe829207740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55582b541e10 0x55582b54daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.275+0000 7fe829207740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582b544d60 con 0x55582b53c2b0 2024-03-23T23:06:24.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.275+0000 7fe829207740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582b45deb0 con 0x55582b53cca0 2024-03-23T23:06:24.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.275+0000 7fe829207740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582b544ee0 con 0x55582b541e10 2024-03-23T23:06:24.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.275+0000 7fe829207740 10 client.0 _ll_register_callbacks cb 0x55582b452150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:24.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.276+0000 7fe817e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55582b53cca0 0x55582b546a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.276+0000 7fe81f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55582b53c2b0 0x55582b546520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.276+0000 7fe81fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55582b541e10 0x55582b54daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.276+0000 7fe817e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55582b53cca0 0x55582b546a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59548/0 (socket says 172.21.15.12:59548) 2024-03-23T23:06:24.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.276+0000 7fe817e00640 1 -- 172.21.15.12:0/4168984880 learned_addr learned my addr 172.21.15.12:0/4168984880 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:24.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.276+0000 7fe817e00640 1 -- 172.21.15.12:0/4168984880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55582b541e10 msgr2=0x55582b54daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:24.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.276+0000 7fe817e00640 1 --2- 172.21.15.12:0/4168984880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55582b541e10 0x55582b54daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.276+0000 7fe817e00640 1 -- 172.21.15.12:0/4168984880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55582b53c2b0 msgr2=0x55582b546520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:24.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.276+0000 7fe817e00640 1 --2- 172.21.15.12:0/4168984880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55582b53c2b0 0x55582b546520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.276+0000 7fe817e00640 1 -- 172.21.15.12:0/4168984880 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55582b2def70 con 0x55582b53cca0 2024-03-23T23:06:24.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.276+0000 7fe817e00640 1 --2- 172.21.15.12:0/4168984880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55582b53cca0 0x55582b546a60 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe80800c6a0 tx=0x7fe808097490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6536e28ad7cc0b99 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:24.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.276+0000 7fe80d600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:06:24.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.276+0000 7fe80d600640 1 -- 172.21.15.12:0/4168984880 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80809b330 con 0x55582b53cca0 2024-03-23T23:06:24.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.277+0000 7fe80d600640 1 -- 172.21.15.12:0/4168984880 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe8080a3b00 con 0x55582b53cca0 2024-03-23T23:06:24.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.277+0000 7fe829207740 1 -- 172.21.15.12:0/4168984880 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe81002cb20 con 0x55582b53cca0 2024-03-23T23:06:24.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.277+0000 7fe80d600640 1 -- 172.21.15.12:0/4168984880 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe80809b150 con 0x55582b53cca0 2024-03-23T23:06:24.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.277+0000 7fe829207740 10 client.26543 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:24.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.277+0000 7fe829207740 1 -- 172.21.15.12:0/4168984880 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55582b176170 con 0x55582b53cca0 2024-03-23T23:06:24.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.277+0000 7fe80d600640 1 -- 172.21.15.12:0/4168984880 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe81002cb20 con 0x55582b53cca0 2024-03-23T23:06:24.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.277+0000 7fe80d600640 1 -- 172.21.15.12:0/4168984880 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe8080d0070 con 0x55582b53cca0 2024-03-23T23:06:24.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.277+0000 7fe829207740 10 client.26543 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:24.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.277+0000 7fe829207740 10 client.26543 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:06:24.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.277+0000 7fe829207740 20 client.26543 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:24.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.277+0000 7fe829207740 1 --2- 172.21.15.12:0/4168984880 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55582b552370 0x55582b5547c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.277+0000 7fe829207740 4 client.26543 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:06:24.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.277+0000 7fe829207740 1 -- 172.21.15.12:0/4168984880 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55582b4af3f0 con 0x55582b552370 2024-03-23T23:06:24.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.278+0000 7fe81f400640 1 --2- 172.21.15.12:0/4168984880 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55582b552370 0x55582b5547c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.278+0000 7fe81f400640 1 --2- 172.21.15.12:0/4168984880 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55582b552370 0x55582b5547c0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d9fb76f8b0a18e3 server_cookie=e48516e329328328 in_seq=0 out_seq=0 2024-03-23T23:06:24.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.278+0000 7fe80d600640 10 client.26543 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:24.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.279+0000 7fe80d600640 1 -- 172.21.15.12:0/4168984880 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55582b4af3f0 con 0x55582b552370 2024-03-23T23:06:24.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.279+0000 7fe80d600640 10 client.26543 handle_command_reply: tid=0 2024-03-23T23:06:24.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.279+0000 7fe829207740 1 client.26543 shutdown 2024-03-23T23:06:24.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.279+0000 7fe829207740 1 -- 172.21.15.12:0/4168984880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55582b53cca0 msgr2=0x55582b546a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:24.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.279+0000 7fe829207740 1 --2- 172.21.15.12:0/4168984880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55582b53cca0 0x55582b546a60 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe80800c6a0 tx=0x7fe808097490 comp rx=0 tx=0).stop 2024-03-23T23:06:24.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.279+0000 7fe829207740 1 -- 172.21.15.12:0/4168984880 shutdown_connections 2024-03-23T23:06:24.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.279+0000 7fe829207740 1 --2- 172.21.15.12:0/4168984880 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55582b552370 0x55582b5547c0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.279+0000 7fe829207740 1 --2- 172.21.15.12:0/4168984880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55582b541e10 0x55582b54daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.279+0000 7fe829207740 1 --2- 172.21.15.12:0/4168984880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55582b53cca0 0x55582b546a60 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.279+0000 7fe829207740 1 --2- 172.21.15.12:0/4168984880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55582b53c2b0 0x55582b546520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.279+0000 7fe80d600640 0 client.26543 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:24.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.279+0000 7fe829207740 1 -- 172.21.15.12:0/4168984880 >> 172.21.15.12:0/4168984880 conn(0x55582b537820 msgr2=0x55582b536a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:24.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.279+0000 7fe829207740 1 -- 172.21.15.12:0/4168984880 shutdown_connections 2024-03-23T23:06:24.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.280+0000 7fe829207740 1 -- 172.21.15.12:0/4168984880 wait complete. 2024-03-23T23:06:24.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.280+0000 7fe829207740 20 client.26543 trim_cache size 0 max 16384 2024-03-23T23:06:24.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.298+0000 7fe829207740 1 Processor -- start 2024-03-23T23:06:24.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.298+0000 7fe829207740 1 -- start start 2024-03-23T23:06:24.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.298+0000 7fe829207740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55582b537820 0x55582b536900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.298+0000 7fe829207740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55582b53c2b0 0x55582b54da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.298+0000 7fe829207740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55582b53cca0 0x55582b559050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.298+0000 7fe829207740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8201aac50 con 0x55582b53c2b0 2024-03-23T23:06:24.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.298+0000 7fe829207740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8201a8750 con 0x55582b53cca0 2024-03-23T23:06:24.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.298+0000 7fe829207740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8201a67f0 con 0x55582b537820 2024-03-23T23:06:24.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.298+0000 7fe81fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55582b53cca0 0x55582b559050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.298+0000 7fe81fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55582b53cca0 0x55582b559050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59550/0 (socket says 172.21.15.12:59550) 2024-03-23T23:06:24.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.298+0000 7fe81fe00640 1 -- 172.21.15.12:0/883384728 learned_addr learned my addr 172.21.15.12:0/883384728 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:24.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.298+0000 7fe81f400640 1 --2- 172.21.15.12:0/883384728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55582b537820 0x55582b536900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.298+0000 7fe817e00640 1 --2- 172.21.15.12:0/883384728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55582b53c2b0 0x55582b54da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.299+0000 7fe81fe00640 1 -- 172.21.15.12:0/883384728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55582b537820 msgr2=0x55582b536900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:24.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.299+0000 7fe81fe00640 1 --2- 172.21.15.12:0/883384728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55582b537820 0x55582b536900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.299+0000 7fe81fe00640 1 -- 172.21.15.12:0/883384728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55582b53c2b0 msgr2=0x55582b54da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:24.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.299+0000 7fe81fe00640 1 --2- 172.21.15.12:0/883384728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55582b53c2b0 0x55582b54da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.299+0000 7fe81fe00640 1 -- 172.21.15.12:0/883384728 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55582b538590 con 0x55582b53cca0 2024-03-23T23:06:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.299+0000 7fe81f400640 1 --2- 172.21.15.12:0/883384728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55582b537820 0x55582b536900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:06:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.299+0000 7fe81fe00640 1 --2- 172.21.15.12:0/883384728 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55582b53cca0 0x55582b559050 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe818013dd0 tx=0x7fe81800b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76ec163b2fa0a1ee server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.299+0000 7fe80fe00640 1 -- 172.21.15.12:0/883384728 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81802c070 con 0x55582b53cca0 2024-03-23T23:06:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.299+0000 7fe80fe00640 1 -- 172.21.15.12:0/883384728 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe81801c940 con 0x55582b53cca0 2024-03-23T23:06:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.299+0000 7fe80fe00640 1 -- 172.21.15.12:0/883384728 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe818027a00 con 0x55582b53cca0 2024-03-23T23:06:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.300+0000 7fe829207740 1 -- 172.21.15.12:0/883384728 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55582b53cca0 msgr2=0x55582b559050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.300+0000 7fe829207740 1 --2- 172.21.15.12:0/883384728 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55582b53cca0 0x55582b559050 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe818013dd0 tx=0x7fe81800b000 comp rx=0 tx=0).stop 2024-03-23T23:06:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.300+0000 7fe829207740 1 -- 172.21.15.12:0/883384728 shutdown_connections 2024-03-23T23:06:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.300+0000 7fe829207740 1 --2- 172.21.15.12:0/883384728 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55582b53cca0 0x55582b559050 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.300+0000 7fe829207740 1 --2- 172.21.15.12:0/883384728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55582b53c2b0 0x55582b54da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.300+0000 7fe829207740 1 --2- 172.21.15.12:0/883384728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55582b537820 0x55582b536900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.300+0000 7fe829207740 1 -- 172.21.15.12:0/883384728 >> 172.21.15.12:0/883384728 conn(0x55582b541910 msgr2=0x55582b5361f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.300+0000 7fe829207740 1 -- 172.21.15.12:0/883384728 shutdown_connections 2024-03-23T23:06:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.300+0000 7fe829207740 1 -- 172.21.15.12:0/883384728 wait complete. 2024-03-23T23:06:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.300+0000 7fe829207740 1 -- start start 2024-03-23T23:06:24.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.300+0000 7fe829207740 1 Processor -- start 2024-03-23T23:06:24.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.301+0000 7fe829207740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55582b537820 0x55582b54d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.301+0000 7fe829207740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55582b53c2b0 0x55582b54da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.301+0000 7fe829207740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55582b53cca0 0x55582b54bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.301+0000 7fe829207740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8201a6df0 con 0x55582b53c2b0 2024-03-23T23:06:24.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.301+0000 7fe829207740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8201a80f0 con 0x55582b53cca0 2024-03-23T23:06:24.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.301+0000 7fe829207740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8201a7430 con 0x55582b537820 2024-03-23T23:06:24.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.301+0000 7fe829207740 10 client.0 _ll_register_callbacks cb 0x7fe8080d2610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:24.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.301+0000 7fe81f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55582b537820 0x55582b54d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.301+0000 7fe81f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55582b537820 0x55582b54d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54818/0 (socket says 172.21.15.12:54818) 2024-03-23T23:06:24.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.301+0000 7fe81f400640 1 -- 172.21.15.12:0/1098225599 learned_addr learned my addr 172.21.15.12:0/1098225599 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:24.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.301+0000 7fe81fe00640 1 --2- 172.21.15.12:0/1098225599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55582b53cca0 0x55582b54bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.301+0000 7fe817e00640 1 --2- 172.21.15.12:0/1098225599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55582b53c2b0 0x55582b54da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.302+0000 7fe81f400640 1 -- 172.21.15.12:0/1098225599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55582b53cca0 msgr2=0x55582b54bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:24.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.302+0000 7fe81f400640 1 --2- 172.21.15.12:0/1098225599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55582b53cca0 0x55582b54bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.302+0000 7fe81f400640 1 -- 172.21.15.12:0/1098225599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55582b53c2b0 msgr2=0x55582b54da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:24.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.302+0000 7fe81f400640 1 --2- 172.21.15.12:0/1098225599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55582b53c2b0 0x55582b54da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.302+0000 7fe81f400640 1 -- 172.21.15.12:0/1098225599 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55582b54c500 con 0x55582b537820 2024-03-23T23:06:24.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.302+0000 7fe81f400640 1 --2- 172.21.15.12:0/1098225599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55582b537820 0x55582b54d510 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe810005ab0 tx=0x7fe81002bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79d58ee87fde0dec server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:24.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.302+0000 7fe80d600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:06:24.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.302+0000 7fe80d600640 1 -- 172.21.15.12:0/1098225599 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe810005d50 con 0x55582b537820 2024-03-23T23:06:24.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.302+0000 7fe80d600640 1 -- 172.21.15.12:0/1098225599 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe810004240 con 0x55582b537820 2024-03-23T23:06:24.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.302+0000 7fe829207740 1 -- 172.21.15.12:0/1098225599 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe81801c940 con 0x55582b537820 2024-03-23T23:06:24.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.302+0000 7fe80d600640 1 -- 172.21.15.12:0/1098225599 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe81002cb20 con 0x55582b537820 2024-03-23T23:06:24.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.302+0000 7fe829207740 10 client.26521 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:24.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.302+0000 7fe829207740 1 -- 172.21.15.12:0/1098225599 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55582b5b70f0 con 0x55582b537820 2024-03-23T23:06:24.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.303+0000 7fe80d600640 1 -- 172.21.15.12:0/1098225599 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe81801c940 con 0x55582b537820 2024-03-23T23:06:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.303+0000 7fe80d600640 1 -- 172.21.15.12:0/1098225599 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe810034560 con 0x55582b537820 2024-03-23T23:06:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.303+0000 7fe829207740 10 client.26521 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.303+0000 7fe829207740 10 client.26521 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:06:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.303+0000 7fe829207740 20 client.26521 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.303+0000 7fe829207740 1 --2- 172.21.15.12:0/1098225599 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55582b541e10 0x55582b5b96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.303+0000 7fe829207740 4 client.26521 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-23T23:06:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.303+0000 7fe829207740 1 -- 172.21.15.12:0/1098225599 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55582b4a9740 con 0x55582b541e10 2024-03-23T23:06:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.303+0000 7fe817e00640 1 --2- 172.21.15.12:0/1098225599 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55582b541e10 0x55582b5b96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.304+0000 7fe817e00640 1 --2- 172.21.15.12:0/1098225599 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55582b541e10 0x55582b5b96c0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=330ce8265c87c80 server_cookie=5d13dd1e11e5ec64 in_seq=0 out_seq=0 2024-03-23T23:06:24.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.304+0000 7fe80d600640 10 client.26521 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:24.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.306+0000 7fe80d600640 1 -- 172.21.15.12:0/1098225599 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55582b4a9740 con 0x55582b541e10 2024-03-23T23:06:24.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.306+0000 7fe80d600640 10 client.26521 handle_command_reply: tid=0 2024-03-23T23:06:24.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.306+0000 7fe829207740 1 client.26521 shutdown 2024-03-23T23:06:24.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.306+0000 7fe829207740 1 -- 172.21.15.12:0/1098225599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55582b537820 msgr2=0x55582b54d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:24.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.306+0000 7fe829207740 1 --2- 172.21.15.12:0/1098225599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55582b537820 0x55582b54d510 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe810005ab0 tx=0x7fe81002bc30 comp rx=0 tx=0).stop 2024-03-23T23:06:24.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.306+0000 7fe829207740 1 -- 172.21.15.12:0/1098225599 shutdown_connections 2024-03-23T23:06:24.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.306+0000 7fe829207740 1 --2- 172.21.15.12:0/1098225599 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55582b541e10 0x55582b5b96c0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.306+0000 7fe829207740 1 --2- 172.21.15.12:0/1098225599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55582b53cca0 0x55582b54bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.306+0000 7fe829207740 1 --2- 172.21.15.12:0/1098225599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55582b53c2b0 0x55582b54da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.306+0000 7fe80d600640 0 client.26521 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:24.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.306+0000 7fe829207740 1 --2- 172.21.15.12:0/1098225599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55582b537820 0x55582b54d510 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.306+0000 7fe829207740 1 -- 172.21.15.12:0/1098225599 >> 172.21.15.12:0/1098225599 conn(0x55582b541910 msgr2=0x55582b536a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:24.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.306+0000 7fe829207740 1 -- 172.21.15.12:0/1098225599 shutdown_connections 2024-03-23T23:06:24.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.306+0000 7fe829207740 1 -- 172.21.15.12:0/1098225599 wait complete. 2024-03-23T23:06:24.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.306+0000 7fe829207740 20 client.26521 trim_cache size 0 max 16384 2024-03-23T23:06:24.307 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:06:24.308 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-03-23T23:06:24.308 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "db57d0c9-1bf9-40ab-a58b-2eb50a1280a4", 2024-03-23T23:06:24.308 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-03-23T23:06:24.308 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:06:24.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.307+0000 7fe825000640 1 -- 172.21.15.12:0/1392860850 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe8040a2800 msgr2=0x7fe8040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:24.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.307+0000 7fe825000640 1 --2- 172.21.15.12:0/1392860850 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe8040a2800 0x7fe8040a4cc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x55582b53d500 tx=0x7fe818009600 comp rx=0 tx=0).stop 2024-03-23T23:06:24.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.307+0000 7fe825000640 1 -- 172.21.15.12:0/1392860850 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe820102770 msgr2=0x7fe820197df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:24.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.307+0000 7fe825000640 1 --2- 172.21.15.12:0/1392860850 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe820102770 0x7fe820197df0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe80800e9f0 tx=0x7fe80800edf0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.307+0000 7fe825000640 1 -- 172.21.15.12:0/1392860850 shutdown_connections 2024-03-23T23:06:24.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.307+0000 7fe825000640 1 --2- 172.21.15.12:0/1392860850 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe8040a2800 0x7fe8040a4cc0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.307+0000 7fe825000640 1 --2- 172.21.15.12:0/1392860850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe820103130 0x7fe82019ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.307+0000 7fe825000640 1 --2- 172.21.15.12:0/1392860850 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe820102770 0x7fe820197df0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.307+0000 7fe825000640 1 --2- 172.21.15.12:0/1392860850 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe820101580 0x7fe8201978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.307+0000 7fe825000640 1 -- 172.21.15.12:0/1392860850 >> 172.21.15.12:0/1392860850 conn(0x7fe8200fcb10 msgr2=0x7fe820105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:24.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.307+0000 7fe825000640 1 -- 172.21.15.12:0/1392860850 shutdown_connections 2024-03-23T23:06:24.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.307+0000 7fe825000640 1 -- 172.21.15.12:0/1392860850 wait complete. 2024-03-23T23:06:24.324 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:06:24.325 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:06:24.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.553+0000 7f2a97e00640 1 -- 172.21.15.12:0/1620750835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a98102790 msgr2=0x7f2a98102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:24.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.553+0000 7f2a97e00640 1 --2- 172.21.15.12:0/1620750835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a98102790 0x7f2a98102c10 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2a80009900 tx=0x7f2a8002bc30 comp rx=0 tx=0).stop 2024-03-23T23:06:24.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.553+0000 7f2a97e00640 1 -- 172.21.15.12:0/1620750835 shutdown_connections 2024-03-23T23:06:24.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.553+0000 7f2a97e00640 1 --2- 172.21.15.12:0/1620750835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a98103150 0x7f2a981099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.553+0000 7f2a97e00640 1 --2- 172.21.15.12:0/1620750835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a98102790 0x7f2a98102c10 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.553+0000 7f2a97e00640 1 --2- 172.21.15.12:0/1620750835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a981015a0 0x7f2a981019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.553+0000 7f2a97e00640 1 -- 172.21.15.12:0/1620750835 >> 172.21.15.12:0/1620750835 conn(0x7f2a980fcb10 msgr2=0x7f2a980fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.553+0000 7f2a97e00640 1 -- 172.21.15.12:0/1620750835 shutdown_connections 2024-03-23T23:06:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.553+0000 7f2a97e00640 1 -- 172.21.15.12:0/1620750835 wait complete. 2024-03-23T23:06:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.555+0000 7f2a97e00640 1 Processor -- start 2024-03-23T23:06:24.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.555+0000 7f2a97e00640 1 -- start start 2024-03-23T23:06:24.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.555+0000 7f2a97e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a981015a0 0x7f2a98197890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.555+0000 7f2a97e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a98102790 0x7f2a98197dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.555+0000 7f2a97e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a98103150 0x7f2a9819ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.555+0000 7f2a97e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9810d250 con 0x7f2a98103150 2024-03-23T23:06:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.555+0000 7f2a97e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9810d0d0 con 0x7f2a981015a0 2024-03-23T23:06:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.555+0000 7f2a97e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9810d3d0 con 0x7f2a98102790 2024-03-23T23:06:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.555+0000 7f2a8e000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a98102790 0x7f2a98197dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.556+0000 7f2a8e000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a98102790 0x7f2a98197dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54838/0 (socket says 172.21.15.12:54838) 2024-03-23T23:06:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.556+0000 7f2a8e000640 1 -- 172.21.15.12:0/2935252582 learned_addr learned my addr 172.21.15.12:0/2935252582 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.556+0000 7f2a96a00640 1 --2- 172.21.15.12:0/2935252582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a981015a0 0x7f2a98197890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.556+0000 7f2a97400640 1 --2- 172.21.15.12:0/2935252582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a98103150 0x7f2a9819ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.556+0000 7f2a8e000640 1 -- 172.21.15.12:0/2935252582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a981015a0 msgr2=0x7f2a98197890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.556+0000 7f2a8e000640 1 --2- 172.21.15.12:0/2935252582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a981015a0 0x7f2a98197890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.556+0000 7f2a8e000640 1 -- 172.21.15.12:0/2935252582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a98103150 msgr2=0x7f2a9819ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.556+0000 7f2a8e000640 1 --2- 172.21.15.12:0/2935252582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a98103150 0x7f2a9819ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.556+0000 7f2a8e000640 1 -- 172.21.15.12:0/2935252582 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9819f4b0 con 0x7f2a98102790 2024-03-23T23:06:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.556+0000 7f2a97400640 1 --2- 172.21.15.12:0/2935252582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a98103150 0x7f2a9819ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:06:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.556+0000 7f2a8e000640 1 --2- 172.21.15.12:0/2935252582 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a98102790 0x7f2a98197dd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f2a80005ab0 tx=0x7f2a8002c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.557+0000 7f2a9ca00640 1 -- 172.21.15.12:0/2935252582 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a80043020 con 0x7f2a98102790 2024-03-23T23:06:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.557+0000 7f2a9ca00640 1 -- 172.21.15.12:0/2935252582 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2a800054e0 con 0x7f2a98102790 2024-03-23T23:06:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.557+0000 7f2a9ca00640 1 -- 172.21.15.12:0/2935252582 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a80003ce0 con 0x7f2a98102790 2024-03-23T23:06:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.557+0000 7f2a97e00640 1 -- 172.21.15.12:0/2935252582 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a9819f740 con 0x7f2a98102790 2024-03-23T23:06:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.557+0000 7f2a97e00640 1 -- 172.21.15.12:0/2935252582 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a9819fc40 con 0x7f2a98102790 2024-03-23T23:06:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.560+0000 7f2a9ca00640 1 -- 172.21.15.12:0/2935252582 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2a80046020 con 0x7f2a98102790 2024-03-23T23:06:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.560+0000 7f2aa0bec740 1 Processor -- start 2024-03-23T23:06:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.560+0000 7f2aa0bec740 1 -- start start 2024-03-23T23:06:24.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.561+0000 7f2aa0bec740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f311050 0x559c9f311470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.561+0000 7f2aa0bec740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559c9f311a40 0x559c9f316540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.561+0000 7f2aa0bec740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559c9f316bb0 0x559c9f319040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.561+0000 7f2aa0bec740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c9f276250 con 0x559c9f311a40 2024-03-23T23:06:24.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.561+0000 7f2aa0bec740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c9f21ea30 con 0x559c9f311050 2024-03-23T23:06:24.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.561+0000 7f2aa0bec740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c9f2722f0 con 0x559c9f316bb0 2024-03-23T23:06:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.561+0000 7f2a97400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f311050 0x559c9f311470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.561+0000 7f2a97400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f311050 0x559c9f311470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59588/0 (socket says 172.21.15.12:59588) 2024-03-23T23:06:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.561+0000 7f2a97400640 1 -- 172.21.15.12:0/2984466282 learned_addr learned my addr 172.21.15.12:0/2984466282 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.561+0000 7f2a96a00640 1 --2- 172.21.15.12:0/2984466282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559c9f311a40 0x559c9f316540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.561+0000 7f2a8e000640 1 --2- 172.21.15.12:0/2984466282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559c9f316bb0 0x559c9f319040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.562+0000 7f2a97400640 1 -- 172.21.15.12:0/2984466282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559c9f316bb0 msgr2=0x559c9f319040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.562+0000 7f2a97400640 1 --2- 172.21.15.12:0/2984466282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559c9f316bb0 0x559c9f319040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.562+0000 7f2a97400640 1 -- 172.21.15.12:0/2984466282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559c9f311a40 msgr2=0x559c9f316540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.562+0000 7f2a97400640 1 --2- 172.21.15.12:0/2984466282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559c9f311a40 0x559c9f316540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.562+0000 7f2a97400640 1 -- 172.21.15.12:0/2984466282 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c9f209690 con 0x559c9f311050 2024-03-23T23:06:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.562+0000 7f2a8e000640 1 --2- 172.21.15.12:0/2984466282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559c9f316bb0 0x559c9f319040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:06:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.562+0000 7f2a9ca00640 1 --2- 172.21.15.12:0/2935252582 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2a7c0a2880 0x7f2a7c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.562+0000 7f2a97400640 1 --2- 172.21.15.12:0/2984466282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f311050 0x559c9f311470 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2a90009f10 tx=0x7f2a9000e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4f153ef4bb99f1e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:24.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.562+0000 7f2a9ca00640 1 -- 172.21.15.12:0/2935252582 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2a800f0be0 con 0x7f2a98102790 2024-03-23T23:06:24.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.562+0000 7f2a96a00640 1 --2- 172.21.15.12:0/2935252582 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2a7c0a2880 0x7f2a7c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.562+0000 7f2a87e00640 1 -- 172.21.15.12:0/2984466282 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a90016070 con 0x559c9f311050 2024-03-23T23:06:24.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.562+0000 7f2a87e00640 1 -- 172.21.15.12:0/2984466282 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2a90004d20 con 0x559c9f311050 2024-03-23T23:06:24.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.562+0000 7f2a87e00640 1 -- 172.21.15.12:0/2984466282 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a900118d0 con 0x559c9f311050 2024-03-23T23:06:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.563+0000 7f2aa0bec740 1 -- 172.21.15.12:0/2984466282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f311050 msgr2=0x559c9f311470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.563+0000 7f2a96a00640 1 --2- 172.21.15.12:0/2935252582 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2a7c0a2880 0x7f2a7c0a4d40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x559c9f311e40 tx=0x7f2a88005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.563+0000 7f2aa0bec740 1 --2- 172.21.15.12:0/2984466282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f311050 0x559c9f311470 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2a90009f10 tx=0x7f2a9000e6c0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.563+0000 7f2aa0bec740 1 -- 172.21.15.12:0/2984466282 shutdown_connections 2024-03-23T23:06:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.563+0000 7f2aa0bec740 1 --2- 172.21.15.12:0/2984466282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559c9f316bb0 0x559c9f319040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.563+0000 7f2aa0bec740 1 --2- 172.21.15.12:0/2984466282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559c9f311a40 0x559c9f316540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.563+0000 7f2aa0bec740 1 --2- 172.21.15.12:0/2984466282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f311050 0x559c9f311470 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.563+0000 7f2aa0bec740 1 -- 172.21.15.12:0/2984466282 >> 172.21.15.12:0/2984466282 conn(0x559c9f2d9200 msgr2=0x559c9f30c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.563+0000 7f2aa0bec740 1 -- 172.21.15.12:0/2984466282 shutdown_connections 2024-03-23T23:06:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.563+0000 7f2aa0bec740 1 -- 172.21.15.12:0/2984466282 wait complete. 2024-03-23T23:06:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.564+0000 7f2aa0bec740 1 -- start start 2024-03-23T23:06:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.564+0000 7f2aa0bec740 1 Processor -- start 2024-03-23T23:06:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.564+0000 7f2aa0bec740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f311050 0x559c9f31b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.564+0000 7f2aa0bec740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559c9f311a40 0x559c9f31b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.564+0000 7f2aa0bec740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559c9f316bb0 0x559c9f322610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.564+0000 7f2aa0bec740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c9f319b60 con 0x559c9f311a40 2024-03-23T23:06:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.564+0000 7f2aa0bec740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c9f231e80 con 0x559c9f311050 2024-03-23T23:06:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.564+0000 7f2aa0bec740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c9f319ce0 con 0x559c9f316bb0 2024-03-23T23:06:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.564+0000 7f2aa0bec740 10 client.0 _ll_register_callbacks cb 0x559c9f244130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.565+0000 7f2a96a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f311050 0x559c9f31b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.565+0000 7f2a97400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559c9f316bb0 0x559c9f322610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.565+0000 7f2a8e000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559c9f311a40 0x559c9f31b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.565+0000 7f2a97400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559c9f316bb0 0x559c9f322610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54846/0 (socket says 172.21.15.12:54846) 2024-03-23T23:06:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.565+0000 7f2a96a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f311050 0x559c9f31b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59600/0 (socket says 172.21.15.12:59600) 2024-03-23T23:06:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.565+0000 7f2a97400640 1 -- 172.21.15.12:0/74667309 learned_addr learned my addr 172.21.15.12:0/74667309 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.565+0000 7f2a97400640 1 -- 172.21.15.12:0/74667309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f311050 msgr2=0x559c9f31b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.565+0000 7f2a97400640 1 --2- 172.21.15.12:0/74667309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f311050 0x559c9f31b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.565+0000 7f2a97400640 1 -- 172.21.15.12:0/74667309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559c9f311a40 msgr2=0x559c9f31b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.565+0000 7f2a97400640 1 --2- 172.21.15.12:0/74667309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559c9f311a40 0x559c9f31b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.565+0000 7f2a97400640 1 -- 172.21.15.12:0/74667309 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559c9f0547d0 con 0x559c9f316bb0 2024-03-23T23:06:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.565+0000 7f2a97400640 1 --2- 172.21.15.12:0/74667309 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559c9f316bb0 0x559c9f322610 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2a90005100 tx=0x7f2a90005130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=576821bd0d5ed7e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.565+0000 7f2a84c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:06:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.566+0000 7f2a84c00640 1 -- 172.21.15.12:0/74667309 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a90028060 con 0x559c9f316bb0 2024-03-23T23:06:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.566+0000 7f2a84c00640 1 -- 172.21.15.12:0/74667309 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2a90005560 con 0x559c9f316bb0 2024-03-23T23:06:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.566+0000 7f2aa0bec740 1 -- 172.21.15.12:0/74667309 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2a90004d20 con 0x559c9f316bb0 2024-03-23T23:06:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.566+0000 7f2a84c00640 1 -- 172.21.15.12:0/74667309 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a90016040 con 0x559c9f316bb0 2024-03-23T23:06:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.566+0000 7f2aa0bec740 10 client.26533 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.566+0000 7f2aa0bec740 1 -- 172.21.15.12:0/74667309 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x559c9ef8a3f0 con 0x559c9f316bb0 2024-03-23T23:06:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.566+0000 7f2a84c00640 1 -- 172.21.15.12:0/74667309 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2a90004d20 con 0x559c9f316bb0 2024-03-23T23:06:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.566+0000 7f2a84c00640 1 -- 172.21.15.12:0/74667309 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f2a9000f040 con 0x559c9f316bb0 2024-03-23T23:06:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.567+0000 7f2aa0bec740 10 client.26533 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.567+0000 7f2aa0bec740 10 client.26533 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:06:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.567+0000 7f2aa0bec740 20 client.26533 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.567+0000 7f2aa0bec740 1 --2- 172.21.15.12:0/74667309 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x559c9f326c00 0x559c9f328ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.567+0000 7f2aa0bec740 4 client.26533 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:06:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.567+0000 7f2aa0bec740 1 -- 172.21.15.12:0/74667309 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559c9f283790 con 0x559c9f326c00 2024-03-23T23:06:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.567+0000 7f2a96a00640 1 --2- 172.21.15.12:0/74667309 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x559c9f326c00 0x559c9f328ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.567+0000 7f2a96a00640 1 --2- 172.21.15.12:0/74667309 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x559c9f326c00 0x559c9f328ff0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e3e3c9736ff7b61 server_cookie=46f83c798ab4f704 in_seq=0 out_seq=0 2024-03-23T23:06:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.567+0000 7f2a84c00640 10 client.26533 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:24.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.568+0000 7f2a84c00640 1 -- 172.21.15.12:0/74667309 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559c9f283790 con 0x559c9f326c00 2024-03-23T23:06:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.568+0000 7f2a84c00640 10 client.26533 handle_command_reply: tid=0 2024-03-23T23:06:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.568+0000 7f2aa0bec740 1 client.26533 shutdown 2024-03-23T23:06:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.568+0000 7f2aa0bec740 1 -- 172.21.15.12:0/74667309 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559c9f316bb0 msgr2=0x559c9f322610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.568+0000 7f2aa0bec740 1 --2- 172.21.15.12:0/74667309 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559c9f316bb0 0x559c9f322610 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2a90005100 tx=0x7f2a90005130 comp rx=0 tx=0).stop 2024-03-23T23:06:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.568+0000 7f2aa0bec740 1 -- 172.21.15.12:0/74667309 shutdown_connections 2024-03-23T23:06:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.568+0000 7f2aa0bec740 1 --2- 172.21.15.12:0/74667309 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x559c9f326c00 0x559c9f328ff0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.569+0000 7f2aa0bec740 1 --2- 172.21.15.12:0/74667309 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559c9f316bb0 0x559c9f322610 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.569+0000 7f2aa0bec740 1 --2- 172.21.15.12:0/74667309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559c9f311a40 0x559c9f31b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.569+0000 7f2aa0bec740 1 --2- 172.21.15.12:0/74667309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f311050 0x559c9f31b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.569+0000 7f2a84c00640 0 client.26533 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.569+0000 7f2aa0bec740 1 -- 172.21.15.12:0/74667309 >> 172.21.15.12:0/74667309 conn(0x559c9f2d9200 msgr2=0x559c9f30bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.569+0000 7f2aa0bec740 1 -- 172.21.15.12:0/74667309 shutdown_connections 2024-03-23T23:06:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.569+0000 7f2aa0bec740 1 -- 172.21.15.12:0/74667309 wait complete. 2024-03-23T23:06:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.569+0000 7f2aa0bec740 20 client.26533 trim_cache size 0 max 16384 2024-03-23T23:06:24.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.587+0000 7f2aa0bec740 1 Processor -- start 2024-03-23T23:06:24.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.587+0000 7f2aa0bec740 1 -- start start 2024-03-23T23:06:24.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.587+0000 7f2aa0bec740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559c9f311050 0x559c9f312bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.587+0000 7f2aa0bec740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559c9f311a40 0x559c9f3225d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.587+0000 7f2aa0bec740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f316bb0 0x559c9f30b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.587+0000 7f2aa0bec740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a981aad40 con 0x559c9f311a40 2024-03-23T23:06:24.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.587+0000 7f2aa0bec740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a981a8840 con 0x559c9f316bb0 2024-03-23T23:06:24.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.587+0000 7f2aa0bec740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a981a68e0 con 0x559c9f311050 2024-03-23T23:06:24.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.587+0000 7f2a97400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f316bb0 0x559c9f30b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.587+0000 7f2a96a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559c9f311050 0x559c9f312bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.587+0000 7f2a97400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f316bb0 0x559c9f30b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59612/0 (socket says 172.21.15.12:59612) 2024-03-23T23:06:24.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.587+0000 7f2a8e000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559c9f311a40 0x559c9f3225d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.587+0000 7f2a97400640 1 -- 172.21.15.12:0/1598430868 learned_addr learned my addr 172.21.15.12:0/1598430868 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:24.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.588+0000 7f2a97400640 1 -- 172.21.15.12:0/1598430868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559c9f311050 msgr2=0x559c9f312bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:24.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.588+0000 7f2a97400640 1 --2- 172.21.15.12:0/1598430868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559c9f311050 0x559c9f312bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.588+0000 7f2a97400640 1 -- 172.21.15.12:0/1598430868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559c9f311a40 msgr2=0x559c9f3225d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:24.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.588+0000 7f2a97400640 1 --2- 172.21.15.12:0/1598430868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559c9f311a40 0x559c9f3225d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.588+0000 7f2a97400640 1 -- 172.21.15.12:0/1598430868 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c9ef4a0f0 con 0x559c9f316bb0 2024-03-23T23:06:24.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.588+0000 7f2a8e000640 1 --2- 172.21.15.12:0/1598430868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559c9f311a40 0x559c9f3225d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:06:24.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.588+0000 7f2a96a00640 1 --2- 172.21.15.12:0/1598430868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559c9f311050 0x559c9f312bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.588+0000 7f2a97400640 1 --2- 172.21.15.12:0/1598430868 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f316bb0 0x559c9f30b330 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f2a90009740 tx=0x7f2a9000ecf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7b478d9f4064d3c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.588+0000 7f2a87400640 1 -- 172.21.15.12:0/1598430868 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a90033070 con 0x559c9f316bb0 2024-03-23T23:06:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.588+0000 7f2a87400640 1 -- 172.21.15.12:0/1598430868 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2a900051a0 con 0x559c9f316bb0 2024-03-23T23:06:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.588+0000 7f2a87400640 1 -- 172.21.15.12:0/1598430868 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a90028070 con 0x559c9f316bb0 2024-03-23T23:06:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.589+0000 7f2aa0bec740 1 -- 172.21.15.12:0/1598430868 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f316bb0 msgr2=0x559c9f30b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.589+0000 7f2aa0bec740 1 --2- 172.21.15.12:0/1598430868 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f316bb0 0x559c9f30b330 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f2a90009740 tx=0x7f2a9000ecf0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.589+0000 7f2aa0bec740 1 -- 172.21.15.12:0/1598430868 shutdown_connections 2024-03-23T23:06:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.589+0000 7f2aa0bec740 1 --2- 172.21.15.12:0/1598430868 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f316bb0 0x559c9f30b330 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.589+0000 7f2aa0bec740 1 --2- 172.21.15.12:0/1598430868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559c9f311a40 0x559c9f3225d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.589+0000 7f2aa0bec740 1 --2- 172.21.15.12:0/1598430868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559c9f311050 0x559c9f312bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.589+0000 7f2aa0bec740 1 -- 172.21.15.12:0/1598430868 >> 172.21.15.12:0/1598430868 conn(0x559c9f2d9200 msgr2=0x559c9f3124e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.589+0000 7f2aa0bec740 1 -- 172.21.15.12:0/1598430868 shutdown_connections 2024-03-23T23:06:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.589+0000 7f2aa0bec740 1 -- 172.21.15.12:0/1598430868 wait complete. 2024-03-23T23:06:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.589+0000 7f2aa0bec740 1 -- start start 2024-03-23T23:06:24.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.589+0000 7f2aa0bec740 1 Processor -- start 2024-03-23T23:06:24.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.590+0000 7f2aa0bec740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f311050 0x559c9f320160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.590+0000 7f2aa0bec740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559c9f311a40 0x559c9f3206a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.590+0000 7f2aa0bec740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559c9f316bb0 0x559c9f3216b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.590+0000 7f2aa0bec740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a981a6ee0 con 0x559c9f311a40 2024-03-23T23:06:24.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.590+0000 7f2aa0bec740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a981a81e0 con 0x559c9f311050 2024-03-23T23:06:24.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.590+0000 7f2aa0bec740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a981a7520 con 0x559c9f316bb0 2024-03-23T23:06:24.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.590+0000 7f2aa0bec740 10 client.0 _ll_register_callbacks cb 0x7f2a880097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:24.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.590+0000 7f2a96a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f311050 0x559c9f320160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.590+0000 7f2a8e000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559c9f311a40 0x559c9f3206a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.590+0000 7f2a96a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f311050 0x559c9f320160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59628/0 (socket says 172.21.15.12:59628) 2024-03-23T23:06:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.590+0000 7f2a96a00640 1 -- 172.21.15.12:0/1096109977 learned_addr learned my addr 172.21.15.12:0/1096109977 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.590+0000 7f2a97400640 1 --2- 172.21.15.12:0/1096109977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559c9f316bb0 0x559c9f3216b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.590+0000 7f2a96a00640 1 -- 172.21.15.12:0/1096109977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559c9f316bb0 msgr2=0x559c9f3216b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.590+0000 7f2a96a00640 1 --2- 172.21.15.12:0/1096109977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559c9f316bb0 0x559c9f3216b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.590+0000 7f2a96a00640 1 -- 172.21.15.12:0/1096109977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559c9f311a40 msgr2=0x559c9f3206a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.590+0000 7f2a96a00640 1 --2- 172.21.15.12:0/1096109977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559c9f311a40 0x559c9f3206a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.591+0000 7f2a96a00640 1 -- 172.21.15.12:0/1096109977 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559c9f321ca0 con 0x559c9f311050 2024-03-23T23:06:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.591+0000 7f2a8e000640 1 --2- 172.21.15.12:0/1096109977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559c9f311a40 0x559c9f3206a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.591+0000 7f2a96a00640 1 --2- 172.21.15.12:0/1096109977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f311050 0x559c9f320160 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x559c9f30d910 tx=0x7f2a880046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ba3745061c9286 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.591+0000 7f2a84c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:06:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.591+0000 7f2a84c00640 1 -- 172.21.15.12:0/1096109977 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a88003d00 con 0x559c9f311050 2024-03-23T23:06:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.591+0000 7f2a84c00640 1 -- 172.21.15.12:0/1096109977 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2a8802b680 con 0x559c9f311050 2024-03-23T23:06:24.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.591+0000 7f2aa0bec740 1 -- 172.21.15.12:0/1096109977 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2a900051a0 con 0x559c9f311050 2024-03-23T23:06:24.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.591+0000 7f2a84c00640 1 -- 172.21.15.12:0/1096109977 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a88003ea0 con 0x559c9f311050 2024-03-23T23:06:24.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.591+0000 7f2aa0bec740 10 client.26567 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:24.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.592+0000 7f2aa0bec740 1 -- 172.21.15.12:0/1096109977 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x559c9f3828c0 con 0x559c9f311050 2024-03-23T23:06:24.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.592+0000 7f2a84c00640 1 -- 172.21.15.12:0/1096109977 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2a900051a0 con 0x559c9f311050 2024-03-23T23:06:24.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.592+0000 7f2a84c00640 1 -- 172.21.15.12:0/1096109977 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f2a90004a00 con 0x559c9f311050 2024-03-23T23:06:24.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.592+0000 7f2aa0bec740 10 client.26567 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:24.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.592+0000 7f2aa0bec740 10 client.26567 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:06:24.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.592+0000 7f2aa0bec740 20 client.26567 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:24.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.592+0000 7f2aa0bec740 1 --2- 172.21.15.12:0/1096109977 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x559c9f383200 0x559c9f385620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:24.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.592+0000 7f2aa0bec740 4 client.26567 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:06:24.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.592+0000 7f2aa0bec740 1 -- 172.21.15.12:0/1096109977 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559c9f27dae0 con 0x559c9f383200 2024-03-23T23:06:24.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.592+0000 7f2a8e000640 1 --2- 172.21.15.12:0/1096109977 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x559c9f383200 0x559c9f385620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:24.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.593+0000 7f2a8e000640 1 --2- 172.21.15.12:0/1096109977 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x559c9f383200 0x559c9f385620 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=529210001c69fb4b server_cookie=b2f6c2c381011bc6 in_seq=0 out_seq=0 2024-03-23T23:06:24.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.593+0000 7f2a84c00640 10 client.26567 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.593+0000 7f2a84c00640 1 -- 172.21.15.12:0/1096109977 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x559c9f27dae0 con 0x559c9f383200 2024-03-23T23:06:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.593+0000 7f2a84c00640 10 client.26567 handle_command_reply: tid=0 2024-03-23T23:06:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.593+0000 7f2aa0bec740 1 client.26567 shutdown 2024-03-23T23:06:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.593+0000 7f2aa0bec740 1 -- 172.21.15.12:0/1096109977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f311050 msgr2=0x559c9f320160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.593+0000 7f2aa0bec740 1 --2- 172.21.15.12:0/1096109977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f311050 0x559c9f320160 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x559c9f30d910 tx=0x7f2a880046c0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.593+0000 7f2aa0bec740 1 -- 172.21.15.12:0/1096109977 shutdown_connections 2024-03-23T23:06:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.593+0000 7f2aa0bec740 1 --2- 172.21.15.12:0/1096109977 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x559c9f383200 0x559c9f385620 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.593+0000 7f2aa0bec740 1 --2- 172.21.15.12:0/1096109977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559c9f316bb0 0x559c9f3216b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.593+0000 7f2aa0bec740 1 --2- 172.21.15.12:0/1096109977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559c9f311a40 0x559c9f3206a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.593+0000 7f2a84c00640 0 client.26567 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.593+0000 7f2aa0bec740 1 --2- 172.21.15.12:0/1096109977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559c9f311050 0x559c9f320160 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.593+0000 7f2aa0bec740 1 -- 172.21.15.12:0/1096109977 >> 172.21.15.12:0/1096109977 conn(0x559c9f2d9200 msgr2=0x559c9f312d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.593+0000 7f2aa0bec740 1 -- 172.21.15.12:0/1096109977 shutdown_connections 2024-03-23T23:06:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.594+0000 7f2aa0bec740 1 -- 172.21.15.12:0/1096109977 wait complete. 2024-03-23T23:06:24.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.594+0000 7f2aa0bec740 20 client.26567 trim_cache size 0 max 16384 2024-03-23T23:06:24.597 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:06:24.597 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T23:06:24.597 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-03-23T23:06:24.597 INFO:teuthology.orchestra.run.smithi012.stdout: "db57d0c9-1bf9-40ab-a58b-2eb50a1280a4": { 2024-03-23T23:06:24.597 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-03-23T23:06:24.597 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "db57d0c9-1bf9-40ab-a58b-2eb50a1280a4", 2024-03-23T23:06:24.597 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-03-23T23:06:24.597 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:06:24.597 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:06:24.597 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:06:24.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.594+0000 7f2a97e00640 1 -- 172.21.15.12:0/2935252582 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2a7c0a2880 msgr2=0x7f2a7c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:24.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.594+0000 7f2a97e00640 1 --2- 172.21.15.12:0/2935252582 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2a7c0a2880 0x7f2a7c0a4d40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x559c9f311e40 tx=0x7f2a88005f90 comp rx=0 tx=0).stop 2024-03-23T23:06:24.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.594+0000 7f2a97e00640 1 -- 172.21.15.12:0/2935252582 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a98102790 msgr2=0x7f2a98197dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:24.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.594+0000 7f2a97e00640 1 --2- 172.21.15.12:0/2935252582 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a98102790 0x7f2a98197dd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f2a80005ab0 tx=0x7f2a8002c6c0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.594+0000 7f2a97e00640 1 -- 172.21.15.12:0/2935252582 shutdown_connections 2024-03-23T23:06:24.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.594+0000 7f2a97e00640 1 --2- 172.21.15.12:0/2935252582 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2a7c0a2880 0x7f2a7c0a4d40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.594+0000 7f2a97e00640 1 --2- 172.21.15.12:0/2935252582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a98103150 0x7f2a9819ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.594+0000 7f2a97e00640 1 --2- 172.21.15.12:0/2935252582 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a98102790 0x7f2a98197dd0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.594+0000 7f2a97e00640 1 --2- 172.21.15.12:0/2935252582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a981015a0 0x7f2a98197890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:24.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.594+0000 7f2a97e00640 1 -- 172.21.15.12:0/2935252582 >> 172.21.15.12:0/2935252582 conn(0x7f2a980fcb10 msgr2=0x7f2a98105a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:24.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.595+0000 7f2a97e00640 1 -- 172.21.15.12:0/2935252582 shutdown_connections 2024-03-23T23:06:24.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:24.595+0000 7f2a97e00640 1 -- 172.21.15.12:0/2935252582 wait complete. 2024-03-23T23:06:24.611 INFO:tasks.cephfs.filesystem:scrub status for tag:db57d0c9-1bf9-40ab-a58b-2eb50a1280a4 - {'path': '/', 'tag': 'db57d0c9-1bf9-40ab-a58b-2eb50a1280a4', 'options': 'recursive,force'} 2024-03-23T23:06:24.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:24 smithi012 ceph-mon[31695]: pgmap v528: 97 pgs: 97 active+clean; 2.8 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 16 op/s 2024-03-23T23:06:24.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:24 smithi173 ceph-mon[36116]: pgmap v528: 97 pgs: 97 active+clean; 2.8 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 16 op/s 2024-03-23T23:06:24.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:24 smithi203 ceph-mon[36693]: pgmap v528: 97 pgs: 97 active+clean; 2.8 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 16 op/s 2024-03-23T23:06:25.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:25 smithi012 ceph-mon[31695]: scrub queued for path: / 2024-03-23T23:06:25.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:25 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:06:25.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:25 smithi012 ceph-mon[31695]: scrub summary: active paths [/] 2024-03-23T23:06:25.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:25 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:06:25.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:25 smithi173 ceph-mon[36116]: scrub queued for path: / 2024-03-23T23:06:25.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:25 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:06:25.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:25 smithi173 ceph-mon[36116]: scrub summary: active paths [/] 2024-03-23T23:06:25.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:25 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:06:25.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:25 smithi203 ceph-mon[36693]: scrub queued for path: / 2024-03-23T23:06:25.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:25 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:06:25.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:25 smithi203 ceph-mon[36693]: scrub summary: active paths [/] 2024-03-23T23:06:25.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:25 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:06:26.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:26 smithi012 ceph-mon[31695]: pgmap v529: 97 pgs: 97 active+clean; 2.8 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:06:26.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:26 smithi012 ceph-mon[31695]: scrub summary: idle 2024-03-23T23:06:26.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:26 smithi173 ceph-mon[36116]: pgmap v529: 97 pgs: 97 active+clean; 2.8 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:06:26.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:26 smithi173 ceph-mon[36116]: scrub summary: idle 2024-03-23T23:06:26.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:26 smithi203 ceph-mon[36693]: pgmap v529: 97 pgs: 97 active+clean; 2.8 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:06:26.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:26 smithi203 ceph-mon[36693]: scrub summary: idle 2024-03-23T23:06:28.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:28 smithi012 ceph-mon[31695]: pgmap v530: 97 pgs: 97 active+clean; 2.8 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:06:28.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:28 smithi173 ceph-mon[36116]: pgmap v530: 97 pgs: 97 active+clean; 2.8 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:06:28.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:28 smithi203 ceph-mon[36693]: pgmap v530: 97 pgs: 97 active+clean; 2.8 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:06:30.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:30 smithi012 ceph-mon[31695]: pgmap v531: 97 pgs: 97 active+clean; 2.8 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:06:30.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:30 smithi173 ceph-mon[36116]: pgmap v531: 97 pgs: 97 active+clean; 2.8 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:06:30.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:30 smithi203 ceph-mon[36693]: pgmap v531: 97 pgs: 97 active+clean; 2.8 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:06:32.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:32 smithi012 ceph-mon[31695]: pgmap v532: 97 pgs: 97 active+clean; 2.8 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-03-23T23:06:32.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:32 smithi173 ceph-mon[36116]: pgmap v532: 97 pgs: 97 active+clean; 2.8 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-03-23T23:06:32.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:32 smithi203 ceph-mon[36693]: pgmap v532: 97 pgs: 97 active+clean; 2.8 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-03-23T23:06:34.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:34 smithi012 ceph-mon[31695]: pgmap v533: 97 pgs: 97 active+clean; 2.9 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:06:34.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:34 smithi173 ceph-mon[36116]: pgmap v533: 97 pgs: 97 active+clean; 2.9 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:06:34.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:34 smithi203 ceph-mon[36693]: pgmap v533: 97 pgs: 97 active+clean; 2.9 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:06:35.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.557+0000 7f46c7e00640 1 -- 172.21.15.12:0/2167643864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46c8107b20 msgr2=0x7f46c8107f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:35.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.557+0000 7f46c7e00640 1 --2- 172.21.15.12:0/2167643864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46c8107b20 0x7f46c8107f40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f46b8009ca0 tx=0x7f46b802c100 comp rx=0 tx=0).stop 2024-03-23T23:06:35.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.557+0000 7f46c7e00640 1 -- 172.21.15.12:0/2167643864 shutdown_connections 2024-03-23T23:06:35.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.557+0000 7f46c7e00640 1 --2- 172.21.15.12:0/2167643864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46c81096d0 0x7f46c810ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.557+0000 7f46c7e00640 1 --2- 172.21.15.12:0/2167643864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46c8108d10 0x7f46c8109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.557+0000 7f46c7e00640 1 --2- 172.21.15.12:0/2167643864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46c8107b20 0x7f46c8107f40 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.557+0000 7f46c7e00640 1 -- 172.21.15.12:0/2167643864 >> 172.21.15.12:0/2167643864 conn(0x7f46c8076ae0 msgr2=0x7f46c8078f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:35.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.557+0000 7f46c7e00640 1 -- 172.21.15.12:0/2167643864 shutdown_connections 2024-03-23T23:06:35.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.557+0000 7f46c7e00640 1 -- 172.21.15.12:0/2167643864 wait complete. 2024-03-23T23:06:35.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.558+0000 7f46c7e00640 1 Processor -- start 2024-03-23T23:06:35.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.558+0000 7f46c7e00640 1 -- start start 2024-03-23T23:06:35.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.559+0000 7f46c7e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46c8107b20 0x7f46c819dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:35.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.559+0000 7f46c7e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46c8108d10 0x7f46c819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:35.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.559+0000 7f46c7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46c81096d0 0x7f46c81a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:35.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.559+0000 7f46c7e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c8113790 con 0x7f46c8107b20 2024-03-23T23:06:35.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.559+0000 7f46c7e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c8113610 con 0x7f46c81096d0 2024-03-23T23:06:35.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.559+0000 7f46c7e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c8113910 con 0x7f46c8108d10 2024-03-23T23:06:35.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.559+0000 7f46c6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46c8107b20 0x7f46c819dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:35.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.559+0000 7f46c6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46c8107b20 0x7f46c819dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32896/0 (socket says 172.21.15.12:32896) 2024-03-23T23:06:35.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.559+0000 7f46c6a00640 1 -- 172.21.15.12:0/2416372848 learned_addr learned my addr 172.21.15.12:0/2416372848 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:35.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.559+0000 7f46c7400640 1 --2- 172.21.15.12:0/2416372848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46c81096d0 0x7f46c81a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:35.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.559+0000 7f46c6000640 1 --2- 172.21.15.12:0/2416372848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46c8108d10 0x7f46c819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:35.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.559+0000 7f46c6a00640 1 -- 172.21.15.12:0/2416372848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46c8108d10 msgr2=0x7f46c819e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:35.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.559+0000 7f46c6a00640 1 --2- 172.21.15.12:0/2416372848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46c8108d10 0x7f46c819e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.560+0000 7f46c6a00640 1 -- 172.21.15.12:0/2416372848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46c81096d0 msgr2=0x7f46c81a5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:35.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.560+0000 7f46c6a00640 1 --2- 172.21.15.12:0/2416372848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46c81096d0 0x7f46c81a5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.560+0000 7f46c6a00640 1 -- 172.21.15.12:0/2416372848 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c81a5a70 con 0x7f46c8107b20 2024-03-23T23:06:35.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.560+0000 7f46c6a00640 1 --2- 172.21.15.12:0/2416372848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46c8107b20 0x7f46c819dea0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f46b8000f80 tx=0x7f46b8035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:35.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.560+0000 7f46bf400640 1 -- 172.21.15.12:0/2416372848 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b8039040 con 0x7f46c8107b20 2024-03-23T23:06:35.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.560+0000 7f46bf400640 1 -- 172.21.15.12:0/2416372848 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46b8044070 con 0x7f46c8107b20 2024-03-23T23:06:35.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.560+0000 7f46c7e00640 1 -- 172.21.15.12:0/2416372848 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c81a5d00 con 0x7f46c8107b20 2024-03-23T23:06:35.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.560+0000 7f46bf400640 1 -- 172.21.15.12:0/2416372848 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b80045a0 con 0x7f46c8107b20 2024-03-23T23:06:35.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.560+0000 7f46c7e00640 1 -- 172.21.15.12:0/2416372848 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c81a61c0 con 0x7f46c8107b20 2024-03-23T23:06:35.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.564+0000 7f46bf400640 1 -- 172.21.15.12:0/2416372848 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f46b8032890 con 0x7f46c8107b20 2024-03-23T23:06:35.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.564+0000 7f46d05bd740 1 Processor -- start 2024-03-23T23:06:35.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.564+0000 7f46d05bd740 1 -- start start 2024-03-23T23:06:35.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.564+0000 7f46d05bd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646b3d483f0 0x5646b3d48810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:35.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.564+0000 7f46d05bd740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5646b3d48de0 0x5646b3d4d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:35.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.564+0000 7f46d05bd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d4df50 0x5646b3d503e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:35.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.564+0000 7f46d05bd740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646b3d0b3b0 con 0x5646b3d48de0 2024-03-23T23:06:35.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.564+0000 7f46d05bd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646b3cad830 con 0x5646b3d4df50 2024-03-23T23:06:35.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.564+0000 7f46d05bd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646b3c565d0 con 0x5646b3d483f0 2024-03-23T23:06:35.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.564+0000 7f46c6000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d4df50 0x5646b3d503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:35.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.564+0000 7f46c7400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646b3d483f0 0x5646b3d48810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:35.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.564+0000 7f46c6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5646b3d48de0 0x5646b3d4d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:35.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.564+0000 7f46c6000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d4df50 0x5646b3d503e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:55938/0 (socket says 172.21.15.12:55938) 2024-03-23T23:06:35.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.564+0000 7f46c7400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646b3d483f0 0x5646b3d48810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:46280/0 (socket says 172.21.15.12:46280) 2024-03-23T23:06:35.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.564+0000 7f46c6000640 1 -- 172.21.15.12:0/3042570688 learned_addr learned my addr 172.21.15.12:0/3042570688 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:35.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.565+0000 7f46c7400640 1 -- 172.21.15.12:0/3042570688 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d4df50 msgr2=0x5646b3d503e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:35.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.565+0000 7f46c7400640 1 --2- 172.21.15.12:0/3042570688 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d4df50 0x5646b3d503e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.565+0000 7f46c7400640 1 -- 172.21.15.12:0/3042570688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5646b3d48de0 msgr2=0x5646b3d4d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:35.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.565+0000 7f46c7400640 1 --2- 172.21.15.12:0/3042570688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5646b3d48de0 0x5646b3d4d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.565+0000 7f46c7400640 1 -- 172.21.15.12:0/3042570688 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5646b3c6e2c0 con 0x5646b3d483f0 2024-03-23T23:06:35.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.565+0000 7f46c6000640 1 --2- 172.21.15.12:0/3042570688 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d4df50 0x5646b3d503e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:35.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.565+0000 7f46c7400640 1 --2- 172.21.15.12:0/3042570688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646b3d483f0 0x5646b3d48810 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f46c00045b0 tx=0x7f46c000bb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6479b936d4f81c8f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:35.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.565+0000 7f46bf400640 1 --2- 172.21.15.12:0/2416372848 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f46ac0a27f0 0x7f46ac0a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:35.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.565+0000 7f46b3e00640 1 -- 172.21.15.12:0/3042570688 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c0003bd0 con 0x5646b3d483f0 2024-03-23T23:06:35.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.565+0000 7f46b3e00640 1 -- 172.21.15.12:0/3042570688 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46c0015070 con 0x5646b3d483f0 2024-03-23T23:06:35.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.566+0000 7f46b3e00640 1 -- 172.21.15.12:0/3042570688 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c0011730 con 0x5646b3d483f0 2024-03-23T23:06:35.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.566+0000 7f46bf400640 1 -- 172.21.15.12:0/2416372848 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f46b80f0f30 con 0x7f46c8107b20 2024-03-23T23:06:35.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.566+0000 7f46c6a00640 1 --2- 172.21.15.12:0/2416372848 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f46ac0a27f0 0x7f46ac0a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:35.572 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.566+0000 7f46d05bd740 1 -- 172.21.15.12:0/3042570688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646b3d483f0 msgr2=0x5646b3d48810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:35.572 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.566+0000 7f46d05bd740 1 --2- 172.21.15.12:0/3042570688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646b3d483f0 0x5646b3d48810 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f46c00045b0 tx=0x7f46c000bb10 comp rx=0 tx=0).stop 2024-03-23T23:06:35.572 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.566+0000 7f46c6a00640 1 --2- 172.21.15.12:0/2416372848 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f46ac0a27f0 0x7f46ac0a4cb0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f46b802c710 tx=0x7f46b80f4970 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:35.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.566+0000 7f46d05bd740 1 -- 172.21.15.12:0/3042570688 shutdown_connections 2024-03-23T23:06:35.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.566+0000 7f46d05bd740 1 --2- 172.21.15.12:0/3042570688 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d4df50 0x5646b3d503e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.566+0000 7f46d05bd740 1 --2- 172.21.15.12:0/3042570688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5646b3d48de0 0x5646b3d4d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.566+0000 7f46d05bd740 1 --2- 172.21.15.12:0/3042570688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646b3d483f0 0x5646b3d48810 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.566+0000 7f46d05bd740 1 -- 172.21.15.12:0/3042570688 >> 172.21.15.12:0/3042570688 conn(0x5646b3d43960 msgr2=0x5646b3d43d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:35.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.566+0000 7f46d05bd740 1 -- 172.21.15.12:0/3042570688 shutdown_connections 2024-03-23T23:06:35.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.567+0000 7f46d05bd740 1 -- 172.21.15.12:0/3042570688 wait complete. 2024-03-23T23:06:35.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.567+0000 7f46d05bd740 1 -- start start 2024-03-23T23:06:35.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.567+0000 7f46d05bd740 1 Processor -- start 2024-03-23T23:06:35.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.568+0000 7f46d05bd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646b3d483f0 0x5646b3d52660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:35.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.568+0000 7f46d05bd740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5646b3d48de0 0x5646b3d52ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:35.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.568+0000 7f46d05bd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d4df50 0x5646b3d59b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:35.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.568+0000 7f46d05bd740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646b3c69a20 con 0x5646b3d48de0 2024-03-23T23:06:35.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.568+0000 7f46d05bd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646b3ca98d0 con 0x5646b3d4df50 2024-03-23T23:06:35.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.568+0000 7f46d05bd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646b3d50f00 con 0x5646b3d483f0 2024-03-23T23:06:35.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.568+0000 7f46d05bd740 10 client.0 _ll_register_callbacks cb 0x5646b3ca61c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:35.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.568+0000 7f46c7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d4df50 0x5646b3d59b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:35.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.568+0000 7f46c6000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5646b3d48de0 0x5646b3d52ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:35.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.568+0000 7f46c6000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5646b3d48de0 0x5646b3d52ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32920/0 (socket says 172.21.15.12:32920) 2024-03-23T23:06:35.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.568+0000 7f46c7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d4df50 0x5646b3d59b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:55954/0 (socket says 172.21.15.12:55954) 2024-03-23T23:06:35.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.568+0000 7f46c6000640 1 -- 172.21.15.12:0/3593436121 learned_addr learned my addr 172.21.15.12:0/3593436121 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:35.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.568+0000 7f46c7400640 1 --2- 172.21.15.12:0/3593436121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646b3d483f0 0x5646b3d52660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:35.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.568+0000 7f46c6000640 1 -- 172.21.15.12:0/3593436121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646b3d483f0 msgr2=0x5646b3d52660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:35.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.568+0000 7f46c6000640 1 --2- 172.21.15.12:0/3593436121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646b3d483f0 0x5646b3d52660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.569+0000 7f46c6000640 1 -- 172.21.15.12:0/3593436121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d4df50 msgr2=0x5646b3d59b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:35.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.569+0000 7f46c6000640 1 --2- 172.21.15.12:0/3593436121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d4df50 0x5646b3d59b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.569+0000 7f46c6000640 1 -- 172.21.15.12:0/3593436121 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5646b39c2210 con 0x5646b3d48de0 2024-03-23T23:06:35.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.569+0000 7f46c6000640 1 --2- 172.21.15.12:0/3593436121 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5646b3d48de0 0x5646b3d52ba0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f46b400b990 tx=0x7f46b400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b120995461e552ce server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:35.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.569+0000 7f46b0c00640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:06:35.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.569+0000 7f46b0c00640 1 -- 172.21.15.12:0/3593436121 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b4003da0 con 0x5646b3d48de0 2024-03-23T23:06:35.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.569+0000 7f46b0c00640 1 -- 172.21.15.12:0/3593436121 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f46b4005020 con 0x5646b3d48de0 2024-03-23T23:06:35.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.569+0000 7f46d05bd740 1 -- 172.21.15.12:0/3593436121 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f46c0015070 con 0x5646b3d48de0 2024-03-23T23:06:35.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.569+0000 7f46b0c00640 1 -- 172.21.15.12:0/3593436121 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46b4005220 con 0x5646b3d48de0 2024-03-23T23:06:35.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.569+0000 7f46d05bd740 10 client.16455 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:35.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.569+0000 7f46d05bd740 1 -- 172.21.15.12:0/3593436121 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5646b3a8e620 con 0x5646b3d48de0 2024-03-23T23:06:35.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.570+0000 7f46b0c00640 1 -- 172.21.15.12:0/3593436121 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46c0015070 con 0x5646b3d48de0 2024-03-23T23:06:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.570+0000 7f46b0c00640 1 -- 172.21.15.12:0/3593436121 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f46b401b590 con 0x5646b3d48de0 2024-03-23T23:06:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.570+0000 7f46d05bd740 10 client.16455 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.570+0000 7f46d05bd740 10 client.16455 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:06:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.570+0000 7f46d05bd740 20 client.16455 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.570+0000 7f46d05bd740 1 --2- 172.21.15.12:0/3593436121 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5646b3d5e580 0x5646b3d609a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.570+0000 7f46d05bd740 4 client.16455 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:06:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.570+0000 7f46d05bd740 1 -- 172.21.15.12:0/3593436121 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5646b3cbad70 con 0x5646b3d5e580 2024-03-23T23:06:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.570+0000 7f46c7400640 1 --2- 172.21.15.12:0/3593436121 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5646b3d5e580 0x5646b3d609a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.571+0000 7f46c7400640 1 --2- 172.21.15.12:0/3593436121 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5646b3d5e580 0x5646b3d609a0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60389d89c8fc9130 server_cookie=11df85a70c016b42 in_seq=0 out_seq=0 2024-03-23T23:06:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.571+0000 7f46b0c00640 10 client.16455 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.572+0000 7f46b0c00640 1 -- 172.21.15.12:0/3593436121 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5646b3cbad70 con 0x5646b3d5e580 2024-03-23T23:06:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.572+0000 7f46b0c00640 10 client.16455 handle_command_reply: tid=0 2024-03-23T23:06:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.572+0000 7f46d05bd740 1 client.16455 shutdown 2024-03-23T23:06:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.572+0000 7f46d05bd740 1 -- 172.21.15.12:0/3593436121 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5646b3d48de0 msgr2=0x5646b3d52ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.572+0000 7f46d05bd740 1 --2- 172.21.15.12:0/3593436121 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5646b3d48de0 0x5646b3d52ba0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f46b400b990 tx=0x7f46b400be50 comp rx=0 tx=0).stop 2024-03-23T23:06:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.572+0000 7f46d05bd740 1 -- 172.21.15.12:0/3593436121 shutdown_connections 2024-03-23T23:06:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.572+0000 7f46d05bd740 1 --2- 172.21.15.12:0/3593436121 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5646b3d5e580 0x5646b3d609a0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.572+0000 7f46d05bd740 1 --2- 172.21.15.12:0/3593436121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d4df50 0x5646b3d59b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.572+0000 7f46d05bd740 1 --2- 172.21.15.12:0/3593436121 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5646b3d48de0 0x5646b3d52ba0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.572+0000 7f46d05bd740 1 --2- 172.21.15.12:0/3593436121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646b3d483f0 0x5646b3d52660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.572+0000 7f46b0c00640 0 client.16455 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.572+0000 7f46d05bd740 1 -- 172.21.15.12:0/3593436121 >> 172.21.15.12:0/3593436121 conn(0x5646b3d43960 msgr2=0x5646b3d42b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.573+0000 7f46d05bd740 1 -- 172.21.15.12:0/3593436121 shutdown_connections 2024-03-23T23:06:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.573+0000 7f46d05bd740 1 -- 172.21.15.12:0/3593436121 wait complete. 2024-03-23T23:06:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.573+0000 7f46d05bd740 20 client.16455 trim_cache size 0 max 16384 2024-03-23T23:06:35.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.588+0000 7f46d05bd740 1 Processor -- start 2024-03-23T23:06:35.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.588+0000 7f46d05bd740 1 -- start start 2024-03-23T23:06:35.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.588+0000 7f46d05bd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646b3d483f0 0x5646b3d4a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:35.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.588+0000 7f46d05bd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d48de0 0x5646b3d4a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:35.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.588+0000 7f46d05bd740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5646b3d4df50 0x5646b3d59b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:35.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.588+0000 7f46d05bd740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c81b13a0 con 0x5646b3d4df50 2024-03-23T23:06:35.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.588+0000 7f46d05bd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c81aeea0 con 0x5646b3d48de0 2024-03-23T23:06:35.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.588+0000 7f46d05bd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c81acf40 con 0x5646b3d483f0 2024-03-23T23:06:35.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.589+0000 7f46c6000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d48de0 0x5646b3d4a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:35.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.589+0000 7f46c7400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646b3d483f0 0x5646b3d4a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:35.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.589+0000 7f46c6000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d48de0 0x5646b3d4a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:55962/0 (socket says 172.21.15.12:55962) 2024-03-23T23:06:35.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.589+0000 7f46c6000640 1 -- 172.21.15.12:0/4092653412 learned_addr learned my addr 172.21.15.12:0/4092653412 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:35.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.589+0000 7f46c7400640 1 --2- 172.21.15.12:0/4092653412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5646b3d4df50 0x5646b3d59b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:35.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.589+0000 7f46c6000640 1 -- 172.21.15.12:0/4092653412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646b3d483f0 msgr2=0x5646b3d4a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:35.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.589+0000 7f46c6000640 1 --2- 172.21.15.12:0/4092653412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646b3d483f0 0x5646b3d4a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.589+0000 7f46c6000640 1 -- 172.21.15.12:0/4092653412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5646b3d4df50 msgr2=0x5646b3d59b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:35.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.589+0000 7f46c6000640 1 --2- 172.21.15.12:0/4092653412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5646b3d4df50 0x5646b3d59b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.589+0000 7f46c6000640 1 -- 172.21.15.12:0/4092653412 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5646b3d61000 con 0x5646b3d48de0 2024-03-23T23:06:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.589+0000 7f46c7400640 1 --2- 172.21.15.12:0/4092653412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5646b3d4df50 0x5646b3d59b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:06:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.590+0000 7f46c6000640 1 --2- 172.21.15.12:0/4092653412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d48de0 0x5646b3d4a680 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f46b4006010 tx=0x7f46b4005c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83259b7927702e0b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.590+0000 7f46b3400640 1 -- 172.21.15.12:0/4092653412 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b401bbc0 con 0x5646b3d48de0 2024-03-23T23:06:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.590+0000 7f46b3400640 1 -- 172.21.15.12:0/4092653412 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46b40053c0 con 0x5646b3d48de0 2024-03-23T23:06:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.590+0000 7f46b3400640 1 -- 172.21.15.12:0/4092653412 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b4026cf0 con 0x5646b3d48de0 2024-03-23T23:06:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.590+0000 7f46d05bd740 1 -- 172.21.15.12:0/4092653412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d48de0 msgr2=0x5646b3d4a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.590+0000 7f46d05bd740 1 --2- 172.21.15.12:0/4092653412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d48de0 0x5646b3d4a680 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f46b4006010 tx=0x7f46b4005c70 comp rx=0 tx=0).stop 2024-03-23T23:06:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.591+0000 7f46d05bd740 1 -- 172.21.15.12:0/4092653412 shutdown_connections 2024-03-23T23:06:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.591+0000 7f46d05bd740 1 --2- 172.21.15.12:0/4092653412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5646b3d4df50 0x5646b3d59b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.591+0000 7f46d05bd740 1 --2- 172.21.15.12:0/4092653412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d48de0 0x5646b3d4a680 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.591+0000 7f46d05bd740 1 --2- 172.21.15.12:0/4092653412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646b3d483f0 0x5646b3d4a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.591+0000 7f46d05bd740 1 -- 172.21.15.12:0/4092653412 >> 172.21.15.12:0/4092653412 conn(0x5646b3d43960 msgr2=0x5646b3d3e6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:35.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.591+0000 7f46d05bd740 1 -- 172.21.15.12:0/4092653412 shutdown_connections 2024-03-23T23:06:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.591+0000 7f46d05bd740 1 -- 172.21.15.12:0/4092653412 wait complete. 2024-03-23T23:06:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.591+0000 7f46d05bd740 1 -- start start 2024-03-23T23:06:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.591+0000 7f46d05bd740 1 Processor -- start 2024-03-23T23:06:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.592+0000 7f46d05bd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d483f0 0x5646b3d57790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.592+0000 7f46d05bd740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5646b3d48de0 0x5646b3d57cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.592+0000 7f46d05bd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646b3d4df50 0x5646b3d58ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.592+0000 7f46d05bd740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c81ad540 con 0x5646b3d48de0 2024-03-23T23:06:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.592+0000 7f46d05bd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c81ae840 con 0x5646b3d483f0 2024-03-23T23:06:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.592+0000 7f46d05bd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c81adb80 con 0x5646b3d4df50 2024-03-23T23:06:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.592+0000 7f46d05bd740 10 client.0 _ll_register_callbacks cb 0x7f46b4004210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.592+0000 7f46c6000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5646b3d48de0 0x5646b3d57cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.592+0000 7f46c7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d483f0 0x5646b3d57790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.592+0000 7f46c6000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5646b3d48de0 0x5646b3d57cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32928/0 (socket says 172.21.15.12:32928) 2024-03-23T23:06:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.592+0000 7f46c6000640 1 -- 172.21.15.12:0/1444788860 learned_addr learned my addr 172.21.15.12:0/1444788860 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:35.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.592+0000 7f46c7400640 1 --2- 172.21.15.12:0/1444788860 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646b3d4df50 0x5646b3d58ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:35.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.592+0000 7f46c6000640 1 -- 172.21.15.12:0/1444788860 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646b3d4df50 msgr2=0x5646b3d58ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:35.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.592+0000 7f46c6000640 1 --2- 172.21.15.12:0/1444788860 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646b3d4df50 0x5646b3d58ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.592+0000 7f46c6000640 1 -- 172.21.15.12:0/1444788860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d483f0 msgr2=0x5646b3d57790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:35.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.592+0000 7f46c6000640 1 --2- 172.21.15.12:0/1444788860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d483f0 0x5646b3d57790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.592+0000 7f46c6000640 1 -- 172.21.15.12:0/1444788860 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5646b3d592d0 con 0x5646b3d48de0 2024-03-23T23:06:35.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.592+0000 7f46c7400640 1 --2- 172.21.15.12:0/1444788860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d483f0 0x5646b3d57790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:35.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.593+0000 7f46c6000640 1 --2- 172.21.15.12:0/1444788860 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5646b3d48de0 0x5646b3d57cd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f46b401cf00 tx=0x7f46b400b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b6bbe310f797189 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:35.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.593+0000 7f46b0c00640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:06:35.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.593+0000 7f46b0c00640 1 -- 172.21.15.12:0/1444788860 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b401b470 con 0x5646b3d48de0 2024-03-23T23:06:35.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.593+0000 7f46d05bd740 1 -- 172.21.15.12:0/1444788860 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f46b40053c0 con 0x5646b3d48de0 2024-03-23T23:06:35.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.593+0000 7f46b0c00640 1 -- 172.21.15.12:0/1444788860 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f46b4026450 con 0x5646b3d48de0 2024-03-23T23:06:35.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.593+0000 7f46b0c00640 1 -- 172.21.15.12:0/1444788860 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46b4040050 con 0x5646b3d48de0 2024-03-23T23:06:35.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.593+0000 7f46d05bd740 10 client.16461 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:35.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.593+0000 7f46d05bd740 1 -- 172.21.15.12:0/1444788860 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5646b3dba270 con 0x5646b3d48de0 2024-03-23T23:06:35.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.594+0000 7f46b0c00640 1 -- 172.21.15.12:0/1444788860 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46b40053c0 con 0x5646b3d48de0 2024-03-23T23:06:35.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.594+0000 7f46b0c00640 1 -- 172.21.15.12:0/1444788860 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f46b4037070 con 0x5646b3d48de0 2024-03-23T23:06:35.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.594+0000 7f46d05bd740 10 client.16461 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:35.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.594+0000 7f46d05bd740 10 client.16461 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:06:35.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.594+0000 7f46d05bd740 20 client.16461 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:35.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.594+0000 7f46d05bd740 1 --2- 172.21.15.12:0/1444788860 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5646b3dbab10 0x5646b3dbcf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:35.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.594+0000 7f46d05bd740 4 client.16461 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:06:35.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.594+0000 7f46d05bd740 1 -- 172.21.15.12:0/1444788860 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5646b3cb50c0 con 0x5646b3dbab10 2024-03-23T23:06:35.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.594+0000 7f46c7400640 1 --2- 172.21.15.12:0/1444788860 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5646b3dbab10 0x5646b3dbcf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:35.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.595+0000 7f46c7400640 1 --2- 172.21.15.12:0/1444788860 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5646b3dbab10 0x5646b3dbcf30 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b25738f62f83041 server_cookie=fc829ee5551533fd in_seq=0 out_seq=0 2024-03-23T23:06:35.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.595+0000 7f46b0c00640 10 client.16461 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:35.607 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.607+0000 7f46b0c00640 1 -- 172.21.15.12:0/1444788860 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5646b3cb50c0 con 0x5646b3dbab10 2024-03-23T23:06:35.607 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.607+0000 7f46b0c00640 10 client.16461 handle_command_reply: tid=0 2024-03-23T23:06:35.607 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.607+0000 7f46d05bd740 1 client.16461 shutdown 2024-03-23T23:06:35.607 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.607+0000 7f46d05bd740 1 -- 172.21.15.12:0/1444788860 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5646b3d48de0 msgr2=0x5646b3d57cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:35.607 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.607+0000 7f46d05bd740 1 --2- 172.21.15.12:0/1444788860 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5646b3d48de0 0x5646b3d57cd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f46b401cf00 tx=0x7f46b400b390 comp rx=0 tx=0).stop 2024-03-23T23:06:35.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.607+0000 7f46d05bd740 1 -- 172.21.15.12:0/1444788860 shutdown_connections 2024-03-23T23:06:35.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.607+0000 7f46d05bd740 1 --2- 172.21.15.12:0/1444788860 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5646b3dbab10 0x5646b3dbcf30 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.607+0000 7f46d05bd740 1 --2- 172.21.15.12:0/1444788860 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646b3d4df50 0x5646b3d58ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.607+0000 7f46d05bd740 1 --2- 172.21.15.12:0/1444788860 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5646b3d48de0 0x5646b3d57cd0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.607+0000 7f46d05bd740 1 --2- 172.21.15.12:0/1444788860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646b3d483f0 0x5646b3d57790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.607+0000 7f46b0c00640 0 client.16461 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:35.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.607+0000 7f46d05bd740 1 -- 172.21.15.12:0/1444788860 >> 172.21.15.12:0/1444788860 conn(0x5646b3d43960 msgr2=0x5646b3d52600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:35.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.607+0000 7f46d05bd740 1 -- 172.21.15.12:0/1444788860 shutdown_connections 2024-03-23T23:06:35.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.607+0000 7f46d05bd740 1 -- 172.21.15.12:0/1444788860 wait complete. 2024-03-23T23:06:35.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.607+0000 7f46d05bd740 20 client.16461 trim_cache size 0 max 16384 2024-03-23T23:06:35.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:06:35.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:06:35.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:06:35.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:06:35.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.608+0000 7f46c7e00640 1 -- 172.21.15.12:0/2416372848 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f46ac0a27f0 msgr2=0x7f46ac0a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:35.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.608+0000 7f46c7e00640 1 --2- 172.21.15.12:0/2416372848 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f46ac0a27f0 0x7f46ac0a4cb0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f46b802c710 tx=0x7f46b80f4970 comp rx=0 tx=0).stop 2024-03-23T23:06:35.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.608+0000 7f46c7e00640 1 -- 172.21.15.12:0/2416372848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46c8107b20 msgr2=0x7f46c819dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.608+0000 7f46c7e00640 1 --2- 172.21.15.12:0/2416372848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46c8107b20 0x7f46c819dea0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f46b8000f80 tx=0x7f46b8035d10 comp rx=0 tx=0).stop 2024-03-23T23:06:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.608+0000 7f46c7e00640 1 -- 172.21.15.12:0/2416372848 shutdown_connections 2024-03-23T23:06:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.608+0000 7f46c7e00640 1 --2- 172.21.15.12:0/2416372848 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f46ac0a27f0 0x7f46ac0a4cb0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.608+0000 7f46c7e00640 1 --2- 172.21.15.12:0/2416372848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46c81096d0 0x7f46c81a5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.608+0000 7f46c7e00640 1 --2- 172.21.15.12:0/2416372848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46c8108d10 0x7f46c819e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.608+0000 7f46c7e00640 1 --2- 172.21.15.12:0/2416372848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46c8107b20 0x7f46c819dea0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.608+0000 7f46c7e00640 1 -- 172.21.15.12:0/2416372848 >> 172.21.15.12:0/2416372848 conn(0x7f46c8076ae0 msgr2=0x7f46c810c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.609+0000 7f46c7e00640 1 -- 172.21.15.12:0/2416372848 shutdown_connections 2024-03-23T23:06:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:35.609+0000 7f46c7e00640 1 -- 172.21.15.12:0/2416372848 wait complete. 2024-03-23T23:06:36.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:36 smithi012 ceph-mon[31695]: pgmap v534: 97 pgs: 97 active+clean; 2.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-23T23:06:36.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:36 smithi173 ceph-mon[36116]: pgmap v534: 97 pgs: 97 active+clean; 2.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-23T23:06:36.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:36 smithi203 ceph-mon[36693]: pgmap v534: 97 pgs: 97 active+clean; 2.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-23T23:06:38.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:38 smithi012 ceph-mon[31695]: pgmap v535: 97 pgs: 97 active+clean; 2.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-23T23:06:38.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:38 smithi173 ceph-mon[36116]: pgmap v535: 97 pgs: 97 active+clean; 2.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-23T23:06:38.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:38 smithi203 ceph-mon[36693]: pgmap v535: 97 pgs: 97 active+clean; 2.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-23T23:06:39.869 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.869 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.869 INFO:tasks.workunit.client.0.smithi012.stdout:127/600: open fd = 3 2024-03-23T23:06:39.869 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.869 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.869 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.869 INFO:tasks.workunit.client.0.smithi012.stdout:128/600: open fd = 3 2024-03-23T23:06:39.869 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.869 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.870 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.870 INFO:tasks.workunit.client.0.smithi012.stdout:129/600: open fd = 3 2024-03-23T23:06:39.870 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.870 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.870 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.870 INFO:tasks.workunit.client.0.smithi012.stdout:130/600: open fd = 3 2024-03-23T23:06:39.870 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.870 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.870 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.870 INFO:tasks.workunit.client.0.smithi012.stdout:131/600: open fd = 3 2024-03-23T23:06:39.870 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.870 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.870 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.870 INFO:tasks.workunit.client.0.smithi012.stdout:132/600: open fd = 3 2024-03-23T23:06:39.871 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.871 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.871 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.871 INFO:tasks.workunit.client.0.smithi012.stdout:133/600: open fd = 3 2024-03-23T23:06:39.871 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.871 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.871 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.871 INFO:tasks.workunit.client.0.smithi012.stdout:134/600: open fd = 3 2024-03-23T23:06:39.871 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.871 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.871 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.871 INFO:tasks.workunit.client.0.smithi012.stdout:135/600: open fd = 3 2024-03-23T23:06:39.871 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.871 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.871 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.872 INFO:tasks.workunit.client.0.smithi012.stdout:136/600: open fd = 3 2024-03-23T23:06:39.872 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.872 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.872 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.872 INFO:tasks.workunit.client.0.smithi012.stdout:137/600: open fd = 3 2024-03-23T23:06:39.872 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.872 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.872 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.872 INFO:tasks.workunit.client.0.smithi012.stdout:138/600: open fd = 3 2024-03-23T23:06:39.872 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.872 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.872 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.872 INFO:tasks.workunit.client.0.smithi012.stdout:139/600: open fd = 3 2024-03-23T23:06:39.872 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.872 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.872 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.872 INFO:tasks.workunit.client.0.smithi012.stdout:140/600: open fd = 3 2024-03-23T23:06:39.873 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.873 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.873 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.873 INFO:tasks.workunit.client.0.smithi012.stdout:141/600: open fd = 3 2024-03-23T23:06:39.873 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.873 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.873 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.873 INFO:tasks.workunit.client.0.smithi012.stdout:142/600: open fd = 3 2024-03-23T23:06:39.873 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.873 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.873 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.873 INFO:tasks.workunit.client.0.smithi012.stdout:143/600: open fd = 3 2024-03-23T23:06:39.873 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.873 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.873 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.873 INFO:tasks.workunit.client.0.smithi012.stdout:144/600: open fd = 3 2024-03-23T23:06:39.874 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.874 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.874 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.874 INFO:tasks.workunit.client.0.smithi012.stdout:145/600: open fd = 3 2024-03-23T23:06:39.874 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.874 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.874 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.874 INFO:tasks.workunit.client.0.smithi012.stdout:146/600: open fd = 3 2024-03-23T23:06:39.874 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.874 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.874 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.874 INFO:tasks.workunit.client.0.smithi012.stdout:147/600: open fd = 3 2024-03-23T23:06:39.874 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.874 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.874 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.875 INFO:tasks.workunit.client.0.smithi012.stdout:148/600: open fd = 3 2024-03-23T23:06:39.875 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.875 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.875 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.875 INFO:tasks.workunit.client.0.smithi012.stdout:149/600: open fd = 3 2024-03-23T23:06:39.875 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.875 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.875 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.875 INFO:tasks.workunit.client.0.smithi012.stdout:150/600: open fd = 3 2024-03-23T23:06:39.875 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.875 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.875 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.875 INFO:tasks.workunit.client.0.smithi012.stdout:151/600: open fd = 3 2024-03-23T23:06:39.875 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.875 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.875 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.876 INFO:tasks.workunit.client.0.smithi012.stdout:152/600: open fd = 3 2024-03-23T23:06:39.876 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.876 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.876 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.876 INFO:tasks.workunit.client.0.smithi012.stdout:153/600: open fd = 3 2024-03-23T23:06:39.876 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.876 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.876 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.876 INFO:tasks.workunit.client.0.smithi012.stdout:154/600: open fd = 3 2024-03-23T23:06:39.876 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.876 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.876 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.876 INFO:tasks.workunit.client.0.smithi012.stdout:155/600: open fd = 3 2024-03-23T23:06:39.876 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.876 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.876 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.876 INFO:tasks.workunit.client.0.smithi012.stdout:156/600: open fd = 3 2024-03-23T23:06:39.876 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.877 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.877 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.877 INFO:tasks.workunit.client.0.smithi012.stdout:157/600: open fd = 3 2024-03-23T23:06:39.877 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.877 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.877 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.877 INFO:tasks.workunit.client.0.smithi012.stdout:158/600: open fd = 3 2024-03-23T23:06:39.877 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.877 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.877 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.877 INFO:tasks.workunit.client.0.smithi012.stdout:159/600: open fd = 3 2024-03-23T23:06:39.877 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.877 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.877 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.877 INFO:tasks.workunit.client.0.smithi012.stdout:160/600: open fd = 3 2024-03-23T23:06:39.877 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.877 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.878 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.878 INFO:tasks.workunit.client.0.smithi012.stdout:161/600: open fd = 3 2024-03-23T23:06:39.878 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.878 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.878 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.878 INFO:tasks.workunit.client.0.smithi012.stdout:162/600: open fd = 3 2024-03-23T23:06:39.878 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.878 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.878 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.878 INFO:tasks.workunit.client.0.smithi012.stdout:163/600: open fd = 3 2024-03-23T23:06:39.878 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.878 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.878 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.878 INFO:tasks.workunit.client.0.smithi012.stdout:164/600: open fd = 3 2024-03-23T23:06:39.878 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.878 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.879 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.879 INFO:tasks.workunit.client.0.smithi012.stdout:165/600: open fd = 3 2024-03-23T23:06:39.879 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.879 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.879 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.879 INFO:tasks.workunit.client.0.smithi012.stdout:166/600: open fd = 3 2024-03-23T23:06:39.879 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.879 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.879 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.879 INFO:tasks.workunit.client.0.smithi012.stdout:167/600: open fd = 3 2024-03-23T23:06:39.879 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.879 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.879 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.879 INFO:tasks.workunit.client.0.smithi012.stdout:168/600: open fd = 3 2024-03-23T23:06:39.879 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.879 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.880 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.880 INFO:tasks.workunit.client.0.smithi012.stdout:169/600: open fd = 3 2024-03-23T23:06:39.880 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.880 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.880 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.880 INFO:tasks.workunit.client.0.smithi012.stdout:170/600: open fd = 3 2024-03-23T23:06:39.880 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.880 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.880 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.880 INFO:tasks.workunit.client.0.smithi012.stdout:171/600: open fd = 3 2024-03-23T23:06:39.880 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.880 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.880 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.880 INFO:tasks.workunit.client.0.smithi012.stdout:172/600: open fd = 3 2024-03-23T23:06:39.880 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.880 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.881 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.881 INFO:tasks.workunit.client.0.smithi012.stdout:173/600: open fd = 3 2024-03-23T23:06:39.881 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.881 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.881 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.881 INFO:tasks.workunit.client.0.smithi012.stdout:174/600: open fd = 3 2024-03-23T23:06:39.881 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.881 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.881 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.881 INFO:tasks.workunit.client.0.smithi012.stdout:175/600: open fd = 3 2024-03-23T23:06:39.881 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.881 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.881 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.881 INFO:tasks.workunit.client.0.smithi012.stdout:176/600: open fd = 3 2024-03-23T23:06:39.881 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.881 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.882 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.882 INFO:tasks.workunit.client.0.smithi012.stdout:177/600: open fd = 3 2024-03-23T23:06:39.882 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.882 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.882 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.882 INFO:tasks.workunit.client.0.smithi012.stdout:178/600: open fd = 3 2024-03-23T23:06:39.882 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.882 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.882 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.882 INFO:tasks.workunit.client.0.smithi012.stdout:179/600: open fd = 3 2024-03-23T23:06:39.882 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.882 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.882 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.882 INFO:tasks.workunit.client.0.smithi012.stdout:180/600: open fd = 3 2024-03-23T23:06:39.882 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.883 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.883 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.883 INFO:tasks.workunit.client.0.smithi012.stdout:181/600: open fd = 3 2024-03-23T23:06:39.883 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.883 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.883 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.883 INFO:tasks.workunit.client.0.smithi012.stdout:182/600: open fd = 3 2024-03-23T23:06:39.883 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.883 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.883 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.883 INFO:tasks.workunit.client.0.smithi012.stdout:183/600: open fd = 3 2024-03-23T23:06:39.883 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.883 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.883 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.883 INFO:tasks.workunit.client.0.smithi012.stdout:184/600: open fd = 3 2024-03-23T23:06:39.883 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.883 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.884 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.884 INFO:tasks.workunit.client.0.smithi012.stdout:185/600: open fd = 3 2024-03-23T23:06:39.884 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.884 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.884 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.884 INFO:tasks.workunit.client.0.smithi012.stdout:186/600: open fd = 3 2024-03-23T23:06:39.884 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.884 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.884 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.884 INFO:tasks.workunit.client.0.smithi012.stdout:187/600: open fd = 3 2024-03-23T23:06:39.884 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.884 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.884 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:06:39.884 INFO:tasks.workunit.client.0.smithi012.stdout:188/600: open fd = 3 2024-03-23T23:06:39.884 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:39.884 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:06:40.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:40 smithi012 ceph-mon[31695]: pgmap v536: 97 pgs: 97 active+clean; 2.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:06:40.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:40 smithi173 ceph-mon[36116]: pgmap v536: 97 pgs: 97 active+clean; 2.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:06:40.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:40 smithi203 ceph-mon[36693]: pgmap v536: 97 pgs: 97 active+clean; 2.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:06:42.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:42 smithi173 ceph-mon[36116]: pgmap v537: 97 pgs: 97 active+clean; 2.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:06:42.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:42 smithi203 ceph-mon[36693]: pgmap v537: 97 pgs: 97 active+clean; 2.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:06:43.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:42 smithi012 ceph-mon[31695]: pgmap v537: 97 pgs: 97 active+clean; 2.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:06:44.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:44 smithi173 ceph-mon[36116]: pgmap v538: 97 pgs: 97 active+clean; 3.0 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:06:44.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:44 smithi203 ceph-mon[36693]: pgmap v538: 97 pgs: 97 active+clean; 3.0 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:06:45.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:44 smithi012 ceph-mon[31695]: pgmap v538: 97 pgs: 97 active+clean; 3.0 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:06:46.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:46 smithi173 ceph-mon[36116]: pgmap v539: 97 pgs: 97 active+clean; 3.0 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:06:46.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:46 smithi203 ceph-mon[36693]: pgmap v539: 97 pgs: 97 active+clean; 3.0 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:06:47.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:46 smithi012 ceph-mon[31695]: pgmap v539: 97 pgs: 97 active+clean; 3.0 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:06:48.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.757+0000 7f58bb200640 1 -- 172.21.15.12:0/573828432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58bc101600 msgr2=0x7f58bc101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:48.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.757+0000 7f58bb200640 1 --2- 172.21.15.12:0/573828432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58bc101600 0x7f58bc101a80 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f58a8009900 tx=0x7f58a802bc30 comp rx=0 tx=0).stop 2024-03-23T23:06:48.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.757+0000 7f58bb200640 1 -- 172.21.15.12:0/573828432 shutdown_connections 2024-03-23T23:06:48.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.757+0000 7f58bb200640 1 --2- 172.21.15.12:0/573828432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58bc101fc0 0x7f58bc108850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.757+0000 7f58bb200640 1 --2- 172.21.15.12:0/573828432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58bc101600 0x7f58bc101a80 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.757+0000 7f58bb200640 1 --2- 172.21.15.12:0/573828432 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58bc100410 0x7f58bc100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.757+0000 7f58bb200640 1 -- 172.21.15.12:0/573828432 >> 172.21.15.12:0/573828432 conn(0x7f58bc0fb9a0 msgr2=0x7f58bc0fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:48.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.757+0000 7f58bb200640 1 -- 172.21.15.12:0/573828432 shutdown_connections 2024-03-23T23:06:48.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.757+0000 7f58bb200640 1 -- 172.21.15.12:0/573828432 wait complete. 2024-03-23T23:06:48.760 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.758+0000 7f58bb200640 1 Processor -- start 2024-03-23T23:06:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.759+0000 7f58bb200640 1 -- start start 2024-03-23T23:06:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.759+0000 7f58bb200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58bc100410 0x7f58bc1978b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.759+0000 7f58bb200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58bc101600 0x7f58bc197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.759+0000 7f58bb200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58bc101fc0 0x7f58bc19ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.759+0000 7f58bb200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc10c0f0 con 0x7f58bc101600 2024-03-23T23:06:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.759+0000 7f58bb200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc10bf70 con 0x7f58bc100410 2024-03-23T23:06:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.759+0000 7f58bb200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc10c270 con 0x7f58bc101fc0 2024-03-23T23:06:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.759+0000 7f58b9e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58bc100410 0x7f58bc1978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.759+0000 7f58b9400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58bc101600 0x7f58bc197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.759+0000 7f58ba800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58bc101fc0 0x7f58bc19ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.759+0000 7f58b9e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58bc100410 0x7f58bc1978b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59132/0 (socket says 172.21.15.12:59132) 2024-03-23T23:06:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.759+0000 7f58b9e00640 1 -- 172.21.15.12:0/3677348393 learned_addr learned my addr 172.21.15.12:0/3677348393 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.760+0000 7f58ba800640 1 -- 172.21.15.12:0/3677348393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58bc100410 msgr2=0x7f58bc1978b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.760+0000 7f58ba800640 1 --2- 172.21.15.12:0/3677348393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58bc100410 0x7f58bc1978b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.760+0000 7f58ba800640 1 -- 172.21.15.12:0/3677348393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58bc101600 msgr2=0x7f58bc197df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.760+0000 7f58ba800640 1 --2- 172.21.15.12:0/3677348393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58bc101600 0x7f58bc197df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.760+0000 7f58ba800640 1 -- 172.21.15.12:0/3677348393 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc19f480 con 0x7f58bc101fc0 2024-03-23T23:06:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.760+0000 7f58b9e00640 1 --2- 172.21.15.12:0/3677348393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58bc100410 0x7f58bc1978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.760+0000 7f58ba800640 1 --2- 172.21.15.12:0/3677348393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58bc101fc0 0x7f58bc19ee10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f58b4009fd0 tx=0x7f58b400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.760+0000 7f58b2a00640 1 -- 172.21.15.12:0/3677348393 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b400ce20 con 0x7f58bc101fc0 2024-03-23T23:06:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.760+0000 7f58b2a00640 1 -- 172.21.15.12:0/3677348393 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f58b4005020 con 0x7f58bc101fc0 2024-03-23T23:06:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.760+0000 7f58b2a00640 1 -- 172.21.15.12:0/3677348393 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b4010720 con 0x7f58bc101fc0 2024-03-23T23:06:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.760+0000 7f58bb200640 1 -- 172.21.15.12:0/3677348393 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58bc19f770 con 0x7f58bc101fc0 2024-03-23T23:06:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.761+0000 7f58bb200640 1 -- 172.21.15.12:0/3677348393 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58bc19fc40 con 0x7f58bc101fc0 2024-03-23T23:06:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.764+0000 7f58b2a00640 1 -- 172.21.15.12:0/3677348393 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f58b4005540 con 0x7f58bc101fc0 2024-03-23T23:06:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.764+0000 7f58c3872740 1 Processor -- start 2024-03-23T23:06:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.764+0000 7f58c3872740 1 -- start start 2024-03-23T23:06:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.764+0000 7f58c3872740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601bad0f800 0x5601bad0fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.764+0000 7f58c3872740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601bad101f0 0x5601bad14cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.764+0000 7f58c3872740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5601bad15360 0x5601bad177f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.764+0000 7f58c3872740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bacce3b0 con 0x5601bad15360 2024-03-23T23:06:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.764+0000 7f58c3872740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bac70830 con 0x5601bad101f0 2024-03-23T23:06:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.764+0000 7f58c3872740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bac195d0 con 0x5601bad0f800 2024-03-23T23:06:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.765+0000 7f58b9400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601bad0f800 0x5601bad0fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.765+0000 7f58b9400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601bad0f800 0x5601bad0fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:50468/0 (socket says 172.21.15.12:50468) 2024-03-23T23:06:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.765+0000 7f58b9e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601bad101f0 0x5601bad14cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.765+0000 7f58b9400640 1 -- 172.21.15.12:0/479456095 learned_addr learned my addr 172.21.15.12:0/479456095 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.765+0000 7f58b9400640 1 --2- 172.21.15.12:0/479456095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5601bad15360 0x5601bad177f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.765+0000 7f58b9e00640 1 -- 172.21.15.12:0/479456095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601bad0f800 msgr2=0x5601bad0fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.765+0000 7f58b9e00640 1 --2- 172.21.15.12:0/479456095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601bad0f800 0x5601bad0fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.765+0000 7f58b9e00640 1 -- 172.21.15.12:0/479456095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5601bad15360 msgr2=0x5601bad177f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.765+0000 7f58b9e00640 1 --2- 172.21.15.12:0/479456095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5601bad15360 0x5601bad177f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.765+0000 7f58b9e00640 1 -- 172.21.15.12:0/479456095 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601bac312c0 con 0x5601bad101f0 2024-03-23T23:06:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.765+0000 7f58b9400640 1 --2- 172.21.15.12:0/479456095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5601bad15360 0x5601bad177f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:06:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.765+0000 7f58b2a00640 1 --2- 172.21.15.12:0/3677348393 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f58a00a2870 0x7f58a00a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.766+0000 7f58b9e00640 1 --2- 172.21.15.12:0/479456095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601bad101f0 0x5601bad14cf0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f58ac0058c0 tx=0x7f58ac011950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3e44e2ad40265bd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.766+0000 7f58b2a00640 1 -- 172.21.15.12:0/3677348393 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f58b40cf140 con 0x7f58bc101fc0 2024-03-23T23:06:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.766+0000 7f58a7000640 1 -- 172.21.15.12:0/479456095 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac01e070 con 0x5601bad101f0 2024-03-23T23:06:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.766+0000 7f58b9400640 1 --2- 172.21.15.12:0/3677348393 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f58a00a2870 0x7f58a00a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.766+0000 7f58a7000640 1 -- 172.21.15.12:0/479456095 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f58ac0035f0 con 0x5601bad101f0 2024-03-23T23:06:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.766+0000 7f58a7000640 1 -- 172.21.15.12:0/479456095 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac00f7a0 con 0x5601bad101f0 2024-03-23T23:06:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.767+0000 7f58c3872740 1 -- 172.21.15.12:0/479456095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601bad101f0 msgr2=0x5601bad14cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.767+0000 7f58c3872740 1 --2- 172.21.15.12:0/479456095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601bad101f0 0x5601bad14cf0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f58ac0058c0 tx=0x7f58ac011950 comp rx=0 tx=0).stop 2024-03-23T23:06:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.767+0000 7f58b9400640 1 --2- 172.21.15.12:0/3677348393 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f58a00a2870 0x7f58a00a4d30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f58a8000c00 tx=0x7f58a802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.767+0000 7f58c3872740 1 -- 172.21.15.12:0/479456095 shutdown_connections 2024-03-23T23:06:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.767+0000 7f58c3872740 1 --2- 172.21.15.12:0/479456095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5601bad15360 0x5601bad177f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.767+0000 7f58c3872740 1 --2- 172.21.15.12:0/479456095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601bad101f0 0x5601bad14cf0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.767+0000 7f58c3872740 1 --2- 172.21.15.12:0/479456095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601bad0f800 0x5601bad0fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.767+0000 7f58c3872740 1 -- 172.21.15.12:0/479456095 >> 172.21.15.12:0/479456095 conn(0x5601bad06960 msgr2=0x5601bad06d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.767+0000 7f58c3872740 1 -- 172.21.15.12:0/479456095 shutdown_connections 2024-03-23T23:06:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.767+0000 7f58c3872740 1 -- 172.21.15.12:0/479456095 wait complete. 2024-03-23T23:06:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.767+0000 7f58c3872740 1 -- start start 2024-03-23T23:06:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.767+0000 7f58c3872740 1 Processor -- start 2024-03-23T23:06:48.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.768+0000 7f58c3872740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5601bad0f800 0x5601bad19960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:48.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.768+0000 7f58c3872740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601bad101f0 0x5601bad19ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:48.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.768+0000 7f58c3872740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601bad15360 0x5601bad20ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.768+0000 7f58c3872740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bac2ca20 con 0x5601bad0f800 2024-03-23T23:06:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.768+0000 7f58c3872740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bac6c8d0 con 0x5601bad101f0 2024-03-23T23:06:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.768+0000 7f58c3872740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bad18310 con 0x5601bad15360 2024-03-23T23:06:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.768+0000 7f58c3872740 10 client.0 _ll_register_callbacks cb 0x5601bac691c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.768+0000 7f58b9e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5601bad0f800 0x5601bad19960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.768+0000 7f58b9e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5601bad0f800 0x5601bad19960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35646/0 (socket says 172.21.15.12:35646) 2024-03-23T23:06:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.768+0000 7f58b9e00640 1 -- 172.21.15.12:0/1255836075 learned_addr learned my addr 172.21.15.12:0/1255836075 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.768+0000 7f58b9400640 1 --2- 172.21.15.12:0/1255836075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601bad101f0 0x5601bad19ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.768+0000 7f58ba800640 1 --2- 172.21.15.12:0/1255836075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601bad15360 0x5601bad20ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.769+0000 7f58b9e00640 1 -- 172.21.15.12:0/1255836075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601bad15360 msgr2=0x5601bad20ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.769+0000 7f58b9e00640 1 --2- 172.21.15.12:0/1255836075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601bad15360 0x5601bad20ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.769+0000 7f58b9e00640 1 -- 172.21.15.12:0/1255836075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601bad101f0 msgr2=0x5601bad19ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.769+0000 7f58b9e00640 1 --2- 172.21.15.12:0/1255836075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601bad101f0 0x5601bad19ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.769+0000 7f58b9e00640 1 -- 172.21.15.12:0/1255836075 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5601ba985210 con 0x5601bad0f800 2024-03-23T23:06:48.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.769+0000 7f58b9e00640 1 --2- 172.21.15.12:0/1255836075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5601bad0f800 0x5601bad19960 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f58ac005e00 tx=0x7f58ac0042b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd7c638a0ce7c728 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:48.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.769+0000 7f589b400640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:06:48.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.769+0000 7f589b400640 1 -- 172.21.15.12:0/1255836075 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac027020 con 0x5601bad0f800 2024-03-23T23:06:48.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.769+0000 7f58c3872740 1 -- 172.21.15.12:0/1255836075 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f58ac0035f0 con 0x5601bad0f800 2024-03-23T23:06:48.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.769+0000 7f589b400640 1 -- 172.21.15.12:0/1255836075 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f58ac005040 con 0x5601bad0f800 2024-03-23T23:06:48.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.770+0000 7f589b400640 1 -- 172.21.15.12:0/1255836075 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58ac01e040 con 0x5601bad0f800 2024-03-23T23:06:48.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.770+0000 7f58c3872740 10 client.16473 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:48.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.770+0000 7f58c3872740 1 -- 172.21.15.12:0/1255836075 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5601baa51620 con 0x5601bad0f800 2024-03-23T23:06:48.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.770+0000 7f589b400640 1 -- 172.21.15.12:0/1255836075 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f58ac0035f0 con 0x5601bad0f800 2024-03-23T23:06:48.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.770+0000 7f589b400640 1 -- 172.21.15.12:0/1255836075 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f58ac00f540 con 0x5601bad0f800 2024-03-23T23:06:48.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.770+0000 7f58c3872740 10 client.16473 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:48.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.770+0000 7f58c3872740 10 client.16473 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:06:48.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.770+0000 7f58c3872740 20 client.16473 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:48.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.770+0000 7f58c3872740 1 --2- 172.21.15.12:0/1255836075 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5601bad254d0 0x5601bad278c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:48.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.770+0000 7f58c3872740 4 client.16473 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:06:48.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.770+0000 7f58c3872740 1 -- 172.21.15.12:0/1255836075 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5601bac7dd70 con 0x5601bad254d0 2024-03-23T23:06:48.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.771+0000 7f58b9400640 1 --2- 172.21.15.12:0/1255836075 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5601bad254d0 0x5601bad278c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:48.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.771+0000 7f58b9400640 1 --2- 172.21.15.12:0/1255836075 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5601bad254d0 0x5601bad278c0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=846da1c6e17baa42 server_cookie=b77128c26e941914 in_seq=0 out_seq=0 2024-03-23T23:06:48.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.771+0000 7f589b400640 10 client.16473 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:48.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.772+0000 7f589b400640 1 -- 172.21.15.12:0/1255836075 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5601bac7dd70 con 0x5601bad254d0 2024-03-23T23:06:48.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.772+0000 7f589b400640 10 client.16473 handle_command_reply: tid=0 2024-03-23T23:06:48.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.772+0000 7f58c3872740 1 client.16473 shutdown 2024-03-23T23:06:48.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.772+0000 7f58c3872740 1 -- 172.21.15.12:0/1255836075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5601bad0f800 msgr2=0x5601bad19960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:48.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.772+0000 7f58c3872740 1 --2- 172.21.15.12:0/1255836075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5601bad0f800 0x5601bad19960 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f58ac005e00 tx=0x7f58ac0042b0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.772+0000 7f58c3872740 1 -- 172.21.15.12:0/1255836075 shutdown_connections 2024-03-23T23:06:48.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.772+0000 7f58c3872740 1 --2- 172.21.15.12:0/1255836075 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5601bad254d0 0x5601bad278c0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.772+0000 7f58c3872740 1 --2- 172.21.15.12:0/1255836075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601bad15360 0x5601bad20ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.772+0000 7f58c3872740 1 --2- 172.21.15.12:0/1255836075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601bad101f0 0x5601bad19ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.772+0000 7f589b400640 0 client.16473 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:48.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.772+0000 7f58c3872740 1 --2- 172.21.15.12:0/1255836075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5601bad0f800 0x5601bad19960 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.772+0000 7f58c3872740 1 -- 172.21.15.12:0/1255836075 >> 172.21.15.12:0/1255836075 conn(0x5601bad06960 msgr2=0x5601bad05b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:48.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.773+0000 7f58c3872740 1 -- 172.21.15.12:0/1255836075 shutdown_connections 2024-03-23T23:06:48.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.773+0000 7f58c3872740 1 -- 172.21.15.12:0/1255836075 wait complete. 2024-03-23T23:06:48.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.773+0000 7f58c3872740 20 client.16473 trim_cache size 0 max 16384 2024-03-23T23:06:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.788+0000 7f58c3872740 1 Processor -- start 2024-03-23T23:06:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.788+0000 7f58c3872740 1 -- start start 2024-03-23T23:06:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.788+0000 7f58c3872740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601bad0f800 0x5601bad0a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.788+0000 7f58c3872740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601bad101f0 0x5601bad20ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.789+0000 7f58c3872740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5601bad15360 0x5601bad18c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.789+0000 7f58c3872740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc1a8b30 con 0x5601bad15360 2024-03-23T23:06:48.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.789+0000 7f58c3872740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc1a6630 con 0x5601bad0f800 2024-03-23T23:06:48.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.789+0000 7f58c3872740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc1a46d0 con 0x5601bad101f0 2024-03-23T23:06:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.789+0000 7f58b9e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601bad0f800 0x5601bad0a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.789+0000 7f58b9400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601bad101f0 0x5601bad20ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.789+0000 7f58b9e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601bad0f800 0x5601bad0a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:59158/0 (socket says 172.21.15.12:59158) 2024-03-23T23:06:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.789+0000 7f58b9400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601bad101f0 0x5601bad20ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:50486/0 (socket says 172.21.15.12:50486) 2024-03-23T23:06:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.789+0000 7f58b9e00640 1 -- 172.21.15.12:0/2687901621 learned_addr learned my addr 172.21.15.12:0/2687901621 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.789+0000 7f58ba800640 1 --2- 172.21.15.12:0/2687901621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5601bad15360 0x5601bad18c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.789+0000 7f58b9e00640 1 -- 172.21.15.12:0/2687901621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601bad101f0 msgr2=0x5601bad20ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.789+0000 7f58b9e00640 1 --2- 172.21.15.12:0/2687901621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601bad101f0 0x5601bad20ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.789+0000 7f58b9e00640 1 -- 172.21.15.12:0/2687901621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5601bad15360 msgr2=0x5601bad18c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.789+0000 7f58b9e00640 1 --2- 172.21.15.12:0/2687901621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5601bad15360 0x5601bad18c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.789+0000 7f58b9e00640 1 -- 172.21.15.12:0/2687901621 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601bad062a0 con 0x5601bad0f800 2024-03-23T23:06:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.789+0000 7f58b9400640 1 --2- 172.21.15.12:0/2687901621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601bad101f0 0x5601bad20ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.790+0000 7f58b9e00640 1 --2- 172.21.15.12:0/2687901621 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601bad0f800 0x5601bad0a2d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f58ac030040 tx=0x7f58ac005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63a22729c6eb3cf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:48.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.790+0000 7f58a6600640 1 -- 172.21.15.12:0/2687901621 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac027030 con 0x5601bad0f800 2024-03-23T23:06:48.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.790+0000 7f58a6600640 1 -- 172.21.15.12:0/2687901621 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f58ac016030 con 0x5601bad0f800 2024-03-23T23:06:48.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.790+0000 7f58a6600640 1 -- 172.21.15.12:0/2687901621 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac0053a0 con 0x5601bad0f800 2024-03-23T23:06:48.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.790+0000 7f58c3872740 1 -- 172.21.15.12:0/2687901621 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601bad0f800 msgr2=0x5601bad0a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:48.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.790+0000 7f58c3872740 1 --2- 172.21.15.12:0/2687901621 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601bad0f800 0x5601bad0a2d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f58ac030040 tx=0x7f58ac005f90 comp rx=0 tx=0).stop 2024-03-23T23:06:48.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.790+0000 7f58c3872740 1 -- 172.21.15.12:0/2687901621 shutdown_connections 2024-03-23T23:06:48.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.790+0000 7f58c3872740 1 --2- 172.21.15.12:0/2687901621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5601bad15360 0x5601bad18c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.790+0000 7f58c3872740 1 --2- 172.21.15.12:0/2687901621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601bad101f0 0x5601bad20ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.790+0000 7f58c3872740 1 --2- 172.21.15.12:0/2687901621 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601bad0f800 0x5601bad0a2d0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.790+0000 7f58c3872740 1 -- 172.21.15.12:0/2687901621 >> 172.21.15.12:0/2687901621 conn(0x5601bad06960 msgr2=0x5601bad09bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:48.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.791+0000 7f58c3872740 1 -- 172.21.15.12:0/2687901621 shutdown_connections 2024-03-23T23:06:48.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.791+0000 7f58c3872740 1 -- 172.21.15.12:0/2687901621 wait complete. 2024-03-23T23:06:48.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.791+0000 7f58c3872740 1 -- start start 2024-03-23T23:06:48.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.791+0000 7f58c3872740 1 Processor -- start 2024-03-23T23:06:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.791+0000 7f58c3872740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5601bad0f800 0x5601bad1ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.791+0000 7f58c3872740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601bad101f0 0x5601bad1f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.792+0000 7f58c3872740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601bad15360 0x5601bad201f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.792+0000 7f58c3872740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc1a4cd0 con 0x5601bad0f800 2024-03-23T23:06:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.792+0000 7f58c3872740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc1a5fd0 con 0x5601bad101f0 2024-03-23T23:06:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.792+0000 7f58c3872740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc1a5310 con 0x5601bad15360 2024-03-23T23:06:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.792+0000 7f58c3872740 10 client.0 _ll_register_callbacks cb 0x7f58ac01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.792+0000 7f58b9e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5601bad0f800 0x5601bad1ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.792+0000 7f58b9400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601bad101f0 0x5601bad1f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.792+0000 7f58b9e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5601bad0f800 0x5601bad1ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35652/0 (socket says 172.21.15.12:35652) 2024-03-23T23:06:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.792+0000 7f58b9e00640 1 -- 172.21.15.12:0/3570523662 learned_addr learned my addr 172.21.15.12:0/3570523662 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.792+0000 7f58ba800640 1 --2- 172.21.15.12:0/3570523662 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601bad15360 0x5601bad201f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.792+0000 7f58b9e00640 1 -- 172.21.15.12:0/3570523662 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601bad15360 msgr2=0x5601bad201f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.792+0000 7f58b9e00640 1 --2- 172.21.15.12:0/3570523662 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601bad15360 0x5601bad201f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.792+0000 7f58b9e00640 1 -- 172.21.15.12:0/3570523662 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601bad101f0 msgr2=0x5601bad1f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.792+0000 7f58b9e00640 1 --2- 172.21.15.12:0/3570523662 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601bad101f0 0x5601bad1f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.792+0000 7f58b9e00640 1 -- 172.21.15.12:0/3570523662 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5601bad207e0 con 0x5601bad0f800 2024-03-23T23:06:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.792+0000 7f58b9e00640 1 --2- 172.21.15.12:0/3570523662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5601bad0f800 0x5601bad1ecf0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f58ac00fb20 tx=0x7f58ac005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fce2e0cfb2a341a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.792+0000 7f589b400640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:06:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.793+0000 7f589b400640 1 -- 172.21.15.12:0/3570523662 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac027040 con 0x5601bad0f800 2024-03-23T23:06:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.793+0000 7f589b400640 1 -- 172.21.15.12:0/3570523662 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f58ac0055a0 con 0x5601bad0f800 2024-03-23T23:06:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.793+0000 7f58c3872740 1 -- 172.21.15.12:0/3570523662 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f58ac016030 con 0x5601bad0f800 2024-03-23T23:06:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.793+0000 7f589b400640 1 -- 172.21.15.12:0/3570523662 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58ac047050 con 0x5601bad0f800 2024-03-23T23:06:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.793+0000 7f58c3872740 10 client.16479 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.793+0000 7f58c3872740 1 -- 172.21.15.12:0/3570523662 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5601bad81390 con 0x5601bad0f800 2024-03-23T23:06:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.793+0000 7f589b400640 1 -- 172.21.15.12:0/3570523662 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f58ac016030 con 0x5601bad0f800 2024-03-23T23:06:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.793+0000 7f589b400640 1 -- 172.21.15.12:0/3570523662 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f58ac035020 con 0x5601bad0f800 2024-03-23T23:06:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.794+0000 7f58c3872740 10 client.16479 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:48.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.794+0000 7f58c3872740 10 client.16479 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:06:48.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.794+0000 7f58c3872740 20 client.16479 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:48.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.794+0000 7f58c3872740 1 --2- 172.21.15.12:0/3570523662 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5601bad81c00 0x5601bad84020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:48.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.794+0000 7f58c3872740 4 client.16479 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:06:48.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.794+0000 7f58c3872740 1 -- 172.21.15.12:0/3570523662 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5601bac780c0 con 0x5601bad81c00 2024-03-23T23:06:48.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.794+0000 7f58b9400640 1 --2- 172.21.15.12:0/3570523662 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5601bad81c00 0x5601bad84020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:48.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.794+0000 7f58b9400640 1 --2- 172.21.15.12:0/3570523662 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5601bad81c00 0x5601bad84020 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d218af925cfe1fd8 server_cookie=e6d2885074a057ac in_seq=0 out_seq=0 2024-03-23T23:06:48.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.794+0000 7f589b400640 10 client.16479 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:48.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.807+0000 7f589b400640 1 -- 172.21.15.12:0/3570523662 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5601bac780c0 con 0x5601bad81c00 2024-03-23T23:06:48.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.807+0000 7f589b400640 10 client.16479 handle_command_reply: tid=0 2024-03-23T23:06:48.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.807+0000 7f58c3872740 1 client.16479 shutdown 2024-03-23T23:06:48.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.808+0000 7f58c3872740 1 -- 172.21.15.12:0/3570523662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5601bad0f800 msgr2=0x5601bad1ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:48.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.808+0000 7f58c3872740 1 --2- 172.21.15.12:0/3570523662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5601bad0f800 0x5601bad1ecf0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f58ac00fb20 tx=0x7f58ac005d30 comp rx=0 tx=0).stop 2024-03-23T23:06:48.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.808+0000 7f58c3872740 1 -- 172.21.15.12:0/3570523662 shutdown_connections 2024-03-23T23:06:48.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.808+0000 7f58c3872740 1 --2- 172.21.15.12:0/3570523662 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5601bad81c00 0x5601bad84020 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.808+0000 7f58c3872740 1 --2- 172.21.15.12:0/3570523662 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601bad15360 0x5601bad201f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.808+0000 7f58c3872740 1 --2- 172.21.15.12:0/3570523662 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601bad101f0 0x5601bad1f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.808+0000 7f58c3872740 1 --2- 172.21.15.12:0/3570523662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5601bad0f800 0x5601bad1ecf0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.808+0000 7f58c3872740 1 -- 172.21.15.12:0/3570523662 >> 172.21.15.12:0/3570523662 conn(0x5601bad06960 msgr2=0x5601bad0a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:48.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.808+0000 7f589b400640 0 client.16479 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:48.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.808+0000 7f58c3872740 1 -- 172.21.15.12:0/3570523662 shutdown_connections 2024-03-23T23:06:48.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.808+0000 7f58c3872740 1 -- 172.21.15.12:0/3570523662 wait complete. 2024-03-23T23:06:48.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.808+0000 7f58c3872740 20 client.16479 trim_cache size 0 max 16384 2024-03-23T23:06:48.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:06:48.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:06:48.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:06:48.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:06:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.809+0000 7f58bb200640 1 -- 172.21.15.12:0/3677348393 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f58a00a2870 msgr2=0x7f58a00a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.809+0000 7f58bb200640 1 --2- 172.21.15.12:0/3677348393 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f58a00a2870 0x7f58a00a4d30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f58a8000c00 tx=0x7f58a802df90 comp rx=0 tx=0).stop 2024-03-23T23:06:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.809+0000 7f58bb200640 1 -- 172.21.15.12:0/3677348393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58bc101fc0 msgr2=0x7f58bc19ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.809+0000 7f58bb200640 1 --2- 172.21.15.12:0/3677348393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58bc101fc0 0x7f58bc19ee10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f58b4009fd0 tx=0x7f58b400ef10 comp rx=0 tx=0).stop 2024-03-23T23:06:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.809+0000 7f58bb200640 1 -- 172.21.15.12:0/3677348393 shutdown_connections 2024-03-23T23:06:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.809+0000 7f58bb200640 1 --2- 172.21.15.12:0/3677348393 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f58a00a2870 0x7f58a00a4d30 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.809+0000 7f58bb200640 1 --2- 172.21.15.12:0/3677348393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58bc101fc0 0x7f58bc19ee10 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.809+0000 7f58bb200640 1 --2- 172.21.15.12:0/3677348393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58bc101600 0x7f58bc197df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.809+0000 7f58bb200640 1 --2- 172.21.15.12:0/3677348393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58bc100410 0x7f58bc1978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.809+0000 7f58bb200640 1 -- 172.21.15.12:0/3677348393 >> 172.21.15.12:0/3677348393 conn(0x7f58bc0fb9a0 msgr2=0x7f58bc069b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:48.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.810+0000 7f58bb200640 1 -- 172.21.15.12:0/3677348393 shutdown_connections 2024-03-23T23:06:48.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:06:48.810+0000 7f58bb200640 1 -- 172.21.15.12:0/3677348393 wait complete. 2024-03-23T23:06:48.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:48 smithi173 ceph-mon[36116]: pgmap v540: 97 pgs: 97 active+clean; 3.0 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:06:48.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:48 smithi203 ceph-mon[36693]: pgmap v540: 97 pgs: 97 active+clean; 3.0 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:06:49.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:48 smithi012 ceph-mon[31695]: pgmap v540: 97 pgs: 97 active+clean; 3.0 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:06:50.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:50 smithi173 ceph-mon[36116]: pgmap v541: 97 pgs: 97 active+clean; 3.1 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 13 op/s 2024-03-23T23:06:50.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:50 smithi203 ceph-mon[36693]: pgmap v541: 97 pgs: 97 active+clean; 3.1 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 13 op/s 2024-03-23T23:06:51.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:50 smithi012 ceph-mon[31695]: pgmap v541: 97 pgs: 97 active+clean; 3.1 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 13 op/s 2024-03-23T23:06:52.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:52 smithi173 ceph-mon[36116]: pgmap v542: 97 pgs: 97 active+clean; 3.1 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-23T23:06:52.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:52 smithi203 ceph-mon[36693]: pgmap v542: 97 pgs: 97 active+clean; 3.1 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-23T23:06:53.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:52 smithi012 ceph-mon[31695]: pgmap v542: 97 pgs: 97 active+clean; 3.1 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-23T23:06:54.612 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:06:54.612 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:06:54.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.852+0000 7fb3ae400640 1 -- 172.21.15.12:0/1589945101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3a8105490 msgr2=0x7fb3a810bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:54.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.852+0000 7fb3ae400640 1 --2- 172.21.15.12:0/1589945101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3a8105490 0x7fb3a810bd20 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0009ca0 tx=0x7fb3a002c100 comp rx=0 tx=0).stop 2024-03-23T23:06:54.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.852+0000 7fb3ae400640 1 -- 172.21.15.12:0/1589945101 shutdown_connections 2024-03-23T23:06:54.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.852+0000 7fb3ae400640 1 --2- 172.21.15.12:0/1589945101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3a8105490 0x7fb3a810bd20 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.852+0000 7fb3ae400640 1 --2- 172.21.15.12:0/1589945101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3a8104ad0 0x7fb3a8104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.852+0000 7fb3ae400640 1 --2- 172.21.15.12:0/1589945101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3a81038e0 0x7fb3a8103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.852+0000 7fb3ae400640 1 -- 172.21.15.12:0/1589945101 >> 172.21.15.12:0/1589945101 conn(0x7fb3a80fee50 msgr2=0x7fb3a8101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:54.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.852+0000 7fb3ae400640 1 -- 172.21.15.12:0/1589945101 shutdown_connections 2024-03-23T23:06:54.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.852+0000 7fb3ae400640 1 -- 172.21.15.12:0/1589945101 wait complete. 2024-03-23T23:06:54.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.853+0000 7fb3ae400640 1 Processor -- start 2024-03-23T23:06:54.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.853+0000 7fb3ae400640 1 -- start start 2024-03-23T23:06:54.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.854+0000 7fb3ae400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3a81038e0 0x7fb3a8199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:54.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.854+0000 7fb3ae400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3a8104ad0 0x7fb3a819a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:54.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.854+0000 7fb3ae400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3a8105490 0x7fb3a81a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:54.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.854+0000 7fb3ae400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a810f540 con 0x7fb3a8104ad0 2024-03-23T23:06:54.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.854+0000 7fb3ae400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a810f3c0 con 0x7fb3a81038e0 2024-03-23T23:06:54.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.854+0000 7fb3ae400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a810f6c0 con 0x7fb3a8105490 2024-03-23T23:06:54.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.854+0000 7fb3a7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3a81038e0 0x7fb3a8199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:54.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.854+0000 7fb3a7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3a81038e0 0x7fb3a8199b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:53432/0 (socket says 172.21.15.12:53432) 2024-03-23T23:06:54.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.854+0000 7fb3a7400640 1 -- 172.21.15.12:0/1626646769 learned_addr learned my addr 172.21.15.12:0/1626646769 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:54.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.854+0000 7fb3a7e00640 1 --2- 172.21.15.12:0/1626646769 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3a8105490 0x7fb3a81a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:54.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.854+0000 7fb3a6a00640 1 --2- 172.21.15.12:0/1626646769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3a8104ad0 0x7fb3a819a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:54.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.855+0000 7fb3a7400640 1 -- 172.21.15.12:0/1626646769 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3a8105490 msgr2=0x7fb3a81a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:54.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.855+0000 7fb3a7400640 1 --2- 172.21.15.12:0/1626646769 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3a8105490 0x7fb3a81a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.855+0000 7fb3a7400640 1 -- 172.21.15.12:0/1626646769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3a8104ad0 msgr2=0x7fb3a819a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:54.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.855+0000 7fb3a7400640 1 --2- 172.21.15.12:0/1626646769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3a8104ad0 0x7fb3a819a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.855+0000 7fb3a7400640 1 -- 172.21.15.12:0/1626646769 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a81a17b0 con 0x7fb3a81038e0 2024-03-23T23:06:54.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.855+0000 7fb3a6a00640 1 --2- 172.21.15.12:0/1626646769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3a8104ad0 0x7fb3a819a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:06:54.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.855+0000 7fb3a7e00640 1 --2- 172.21.15.12:0/1626646769 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3a8105490 0x7fb3a81a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:54.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.855+0000 7fb3a7400640 1 --2- 172.21.15.12:0/1626646769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3a81038e0 0x7fb3a8199b90 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb39800e9c0 tx=0x7fb39800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:54.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.855+0000 7fb39fe00640 1 -- 172.21.15.12:0/1626646769 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39800cd80 con 0x7fb3a81038e0 2024-03-23T23:06:54.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.855+0000 7fb39fe00640 1 -- 172.21.15.12:0/1626646769 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3980035f0 con 0x7fb3a81038e0 2024-03-23T23:06:54.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.855+0000 7fb3ae400640 1 -- 172.21.15.12:0/1626646769 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a81a1aa0 con 0x7fb3a81038e0 2024-03-23T23:06:54.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.855+0000 7fb39fe00640 1 -- 172.21.15.12:0/1626646769 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb398010690 con 0x7fb3a81038e0 2024-03-23T23:06:54.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.855+0000 7fb3ae400640 1 -- 172.21.15.12:0/1626646769 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a81a1ff0 con 0x7fb3a81038e0 2024-03-23T23:06:54.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.858+0000 7fb3b11c7740 1 Processor -- start 2024-03-23T23:06:54.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.859+0000 7fb3b11c7740 1 -- start start 2024-03-23T23:06:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.859+0000 7fb3b11c7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56373c9b6e80 0x56373c9b72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.859+0000 7fb3b11c7740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56373c9b7870 0x56373c9bc370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.859+0000 7fb3b11c7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56373c9bc9e0 0x56373c9bee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.859+0000 7fb3b11c7740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56373c91c250 con 0x56373c9b7870 2024-03-23T23:06:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.859+0000 7fb3b11c7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56373c8c4a30 con 0x56373c9b6e80 2024-03-23T23:06:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.859+0000 7fb3b11c7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x56373c9182f0 con 0x56373c9bc9e0 2024-03-23T23:06:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.859+0000 7fb3a7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56373c9b6e80 0x56373c9b72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.859+0000 7fb3a6a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56373c9bc9e0 0x56373c9bee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.859+0000 7fb3a6a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56373c9bc9e0 0x56373c9bee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47630/0 (socket says 172.21.15.12:47630) 2024-03-23T23:06:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.859+0000 7fb3a6a00640 1 -- 172.21.15.12:0/53577125 learned_addr learned my addr 172.21.15.12:0/53577125 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.860+0000 7fb3a6a00640 1 -- 172.21.15.12:0/53577125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56373c9b6e80 msgr2=0x56373c9b72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.860+0000 7fb3a6a00640 1 --2- 172.21.15.12:0/53577125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56373c9b6e80 0x56373c9b72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.860+0000 7fb3a6a00640 1 -- 172.21.15.12:0/53577125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56373c9b7870 msgr2=0x56373c9bc370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T23:06:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.860+0000 7fb3a6a00640 1 --2- 172.21.15.12:0/53577125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56373c9b7870 0x56373c9bc370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.860+0000 7fb3a6a00640 1 -- 172.21.15.12:0/53577125 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56373c8af690 con 0x56373c9bc9e0 2024-03-23T23:06:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.860+0000 7fb3a6a00640 1 --2- 172.21.15.12:0/53577125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56373c9bc9e0 0x56373c9bee70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb39400bcb0 tx=0x7fb39400f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c84f48dff9390eb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.860+0000 7fb39fe00640 1 -- 172.21.15.12:0/1626646769 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb3980037a0 con 0x7fb3a81038e0 2024-03-23T23:06:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.860+0000 7fb39cc00640 1 -- 172.21.15.12:0/53577125 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb394011440 con 0x56373c9bc9e0 2024-03-23T23:06:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.860+0000 7fb39cc00640 1 -- 172.21.15.12:0/53577125 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb394011ac0 con 0x56373c9bc9e0 2024-03-23T23:06:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.860+0000 7fb39cc00640 1 -- 172.21.15.12:0/53577125 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39401bb60 con 0x56373c9bc9e0 2024-03-23T23:06:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.861+0000 7fb3b11c7740 1 -- 172.21.15.12:0/53577125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56373c9bc9e0 msgr2=0x56373c9bee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.861+0000 7fb3b11c7740 1 --2- 172.21.15.12:0/53577125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56373c9bc9e0 0x56373c9bee70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb39400bcb0 tx=0x7fb39400f730 comp rx=0 tx=0).stop 2024-03-23T23:06:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.861+0000 7fb3b11c7740 1 -- 172.21.15.12:0/53577125 shutdown_connections 2024-03-23T23:06:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.861+0000 7fb3b11c7740 1 --2- 172.21.15.12:0/53577125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56373c9bc9e0 0x56373c9bee70 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.861+0000 7fb3b11c7740 1 --2- 172.21.15.12:0/53577125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56373c9b7870 0x56373c9bc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.861+0000 7fb3b11c7740 1 --2- 172.21.15.12:0/53577125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56373c9b6e80 0x56373c9b72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.861+0000 7fb3b11c7740 1 -- 172.21.15.12:0/53577125 >> 172.21.15.12:0/53577125 conn(0x56373c97f200 msgr2=0x56373c9b27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.861+0000 7fb3b11c7740 1 -- 172.21.15.12:0/53577125 shutdown_connections 2024-03-23T23:06:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.861+0000 7fb3b11c7740 1 -- 172.21.15.12:0/53577125 wait complete. 2024-03-23T23:06:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.862+0000 7fb3b11c7740 1 -- start start 2024-03-23T23:06:54.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.862+0000 7fb3b11c7740 1 Processor -- start 2024-03-23T23:06:54.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.862+0000 7fb39fe00640 1 --2- 172.21.15.12:0/1626646769 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb38c0a2710 0x7fb38c0a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.863+0000 7fb39fe00640 1 -- 172.21.15.12:0/1626646769 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb3980cef10 con 0x7fb3a81038e0 2024-03-23T23:06:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.863+0000 7fb3a7e00640 1 --2- 172.21.15.12:0/1626646769 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb38c0a2710 0x7fb38c0a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.863+0000 7fb3b11c7740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56373c9b6e80 0x56373c9c0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.863+0000 7fb3b11c7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56373c9b7870 0x56373c9c13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.863+0000 7fb3b11c7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56373c9bc9e0 0x56373c9c83e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.863+0000 7fb3b11c7740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56373c9bf990 con 0x56373c9b6e80 2024-03-23T23:06:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.863+0000 7fb3b11c7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56373c8d7e80 con 0x56373c9b7870 2024-03-23T23:06:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.863+0000 7fb3b11c7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x56373c9bfb10 con 0x56373c9bc9e0 2024-03-23T23:06:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.863+0000 7fb3b11c7740 10 client.0 _ll_register_callbacks cb 0x56373c8ea130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.863+0000 7fb3a6a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56373c9b7870 0x56373c9c13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.863+0000 7fb3a7400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56373c9b6e80 0x56373c9c0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.863+0000 7fb3a7400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56373c9b6e80 0x56373c9c0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37584/0 (socket says 172.21.15.12:37584) 2024-03-23T23:06:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.863+0000 7fb3a7400640 1 -- 172.21.15.12:0/567529757 learned_addr learned my addr 172.21.15.12:0/567529757 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.864+0000 7fb3a7e00640 1 --2- 172.21.15.12:0/567529757 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56373c9bc9e0 0x56373c9c83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.864+0000 7fb3a6a00640 1 -- 172.21.15.12:0/567529757 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56373c9bc9e0 msgr2=0x56373c9c83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.864+0000 7fb3a6a00640 1 --2- 172.21.15.12:0/567529757 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56373c9bc9e0 0x56373c9c83e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.864+0000 7fb3a6a00640 1 -- 172.21.15.12:0/567529757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56373c9b6e80 msgr2=0x56373c9c0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.864+0000 7fb3a6a00640 1 --2- 172.21.15.12:0/567529757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56373c9b6e80 0x56373c9c0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.864+0000 7fb3a6a00640 1 -- 172.21.15.12:0/567529757 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56373c6fa7d0 con 0x56373c9b7870 2024-03-23T23:06:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.864+0000 7fb3a7e00640 1 --2- 172.21.15.12:0/567529757 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56373c9bc9e0 0x56373c9c83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:06:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.864+0000 7fb3a7400640 1 --2- 172.21.15.12:0/567529757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56373c9b6e80 0x56373c9c0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.864+0000 7fb3a7e00640 1 --2- 172.21.15.12:0/1626646769 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb38c0a2710 0x7fb38c0a4bd0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb3a002c640 tx=0x7fb3a0009970 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.864+0000 7fb3a6a00640 1 --2- 172.21.15.12:0/567529757 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56373c9b7870 0x56373c9c13a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x56373c9b3740 tx=0x7fb39400bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f3e06ee657bc6cb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.864+0000 7fb391600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:06:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.864+0000 7fb391600640 1 -- 172.21.15.12:0/567529757 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb394011850 con 0x56373c9b7870 2024-03-23T23:06:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.865+0000 7fb391600640 1 -- 172.21.15.12:0/567529757 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb39401b8e0 con 0x56373c9b7870 2024-03-23T23:06:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.865+0000 7fb3b11c7740 1 -- 172.21.15.12:0/567529757 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb394011ac0 con 0x56373c9b7870 2024-03-23T23:06:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.865+0000 7fb391600640 1 -- 172.21.15.12:0/567529757 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb39401bae0 con 0x56373c9b7870 2024-03-23T23:06:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.865+0000 7fb3b11c7740 10 client.26615 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:54.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.865+0000 7fb3b11c7740 1 -- 172.21.15.12:0/567529757 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56373c6303f0 con 0x56373c9b7870 2024-03-23T23:06:54.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.865+0000 7fb391600640 1 -- 172.21.15.12:0/567529757 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb394011ac0 con 0x56373c9b7870 2024-03-23T23:06:54.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.865+0000 7fb391600640 1 -- 172.21.15.12:0/567529757 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fb39402d030 con 0x56373c9b7870 2024-03-23T23:06:54.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.865+0000 7fb3b11c7740 10 client.26615 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:54.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.865+0000 7fb3b11c7740 10 client.26615 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:06:54.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.865+0000 7fb3b11c7740 20 client.26615 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:54.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.865+0000 7fb3b11c7740 1 --2- 172.21.15.12:0/567529757 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56373c9cc8d0 0x56373c9cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:54.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.865+0000 7fb3b11c7740 4 client.26615 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:06:54.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.865+0000 7fb3b11c7740 1 -- 172.21.15.12:0/567529757 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56373c929790 con 0x56373c9cc8d0 2024-03-23T23:06:54.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.866+0000 7fb3a7400640 1 --2- 172.21.15.12:0/567529757 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56373c9cc8d0 0x56373c9cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:54.869 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:54 smithi173 ceph-mon[36116]: pgmap v543: 97 pgs: 97 active+clean; 3.1 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:06:54.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.866+0000 7fb3a7400640 1 --2- 172.21.15.12:0/567529757 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56373c9cc8d0 0x56373c9cecc0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b4383fe6e11cddf server_cookie=21ce359d4b0b7c5b in_seq=0 out_seq=0 2024-03-23T23:06:54.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.866+0000 7fb391600640 10 client.26615 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:54.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.867+0000 7fb391600640 1 -- 172.21.15.12:0/567529757 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56373c929790 con 0x56373c9cc8d0 2024-03-23T23:06:54.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.867+0000 7fb391600640 10 client.26615 handle_command_reply: tid=0 2024-03-23T23:06:54.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.867+0000 7fb3b11c7740 1 client.26615 shutdown 2024-03-23T23:06:54.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.867+0000 7fb3b11c7740 1 -- 172.21.15.12:0/567529757 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56373c9b7870 msgr2=0x56373c9c13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:54.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.867+0000 7fb3b11c7740 1 --2- 172.21.15.12:0/567529757 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56373c9b7870 0x56373c9c13a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x56373c9b3740 tx=0x7fb39400bfd0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.867+0000 7fb3b11c7740 1 -- 172.21.15.12:0/567529757 shutdown_connections 2024-03-23T23:06:54.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.867+0000 7fb3b11c7740 1 --2- 172.21.15.12:0/567529757 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56373c9cc8d0 0x56373c9cecc0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.867+0000 7fb3b11c7740 1 --2- 172.21.15.12:0/567529757 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56373c9bc9e0 0x56373c9c83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.867+0000 7fb3b11c7740 1 --2- 172.21.15.12:0/567529757 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56373c9b7870 0x56373c9c13a0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.867+0000 7fb3b11c7740 1 --2- 172.21.15.12:0/567529757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56373c9b6e80 0x56373c9c0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.867+0000 7fb391600640 0 client.26615 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:54.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.867+0000 7fb3b11c7740 1 -- 172.21.15.12:0/567529757 >> 172.21.15.12:0/567529757 conn(0x56373c97f200 msgr2=0x56373c9b1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:54.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.867+0000 7fb3b11c7740 1 -- 172.21.15.12:0/567529757 shutdown_connections 2024-03-23T23:06:54.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.867+0000 7fb3b11c7740 1 -- 172.21.15.12:0/567529757 wait complete. 2024-03-23T23:06:54.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.868+0000 7fb3b11c7740 20 client.26615 trim_cache size 0 max 16384 2024-03-23T23:06:54.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.885+0000 7fb3b11c7740 1 Processor -- start 2024-03-23T23:06:54.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.885+0000 7fb3b11c7740 1 -- start start 2024-03-23T23:06:54.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.886+0000 7fb3b11c7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56373c9b6e80 0x56373c9c0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:54.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.886+0000 7fb3b11c7740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56373c9b7870 0x56373c9c83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:54.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.886+0000 7fb3b11c7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56373c9bc9e0 0x56373c9b89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:54.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.886+0000 7fb3b11c7740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a81ad160 con 0x56373c9b7870 2024-03-23T23:06:54.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.886+0000 7fb3b11c7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a81aac60 con 0x56373c9b6e80 2024-03-23T23:06:54.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.886+0000 7fb3b11c7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a81a8d00 con 0x56373c9bc9e0 2024-03-23T23:06:54.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.886+0000 7fb3a6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56373c9b7870 0x56373c9c83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:54.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.886+0000 7fb3a7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56373c9b6e80 0x56373c9c0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:54.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.886+0000 7fb3a6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56373c9b7870 0x56373c9c83a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37590/0 (socket says 172.21.15.12:37590) 2024-03-23T23:06:54.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.886+0000 7fb3a6a00640 1 -- 172.21.15.12:0/820431684 learned_addr learned my addr 172.21.15.12:0/820431684 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:54.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.886+0000 7fb3a7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56373c9b6e80 0x56373c9c0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:53454/0 (socket says 172.21.15.12:53454) 2024-03-23T23:06:54.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.886+0000 7fb3a7e00640 1 --2- 172.21.15.12:0/820431684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56373c9bc9e0 0x56373c9b89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:54.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.886+0000 7fb3a6a00640 1 -- 172.21.15.12:0/820431684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56373c9bc9e0 msgr2=0x56373c9b89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:54.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.886+0000 7fb3a6a00640 1 --2- 172.21.15.12:0/820431684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56373c9bc9e0 0x56373c9b89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.886+0000 7fb3a6a00640 1 -- 172.21.15.12:0/820431684 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56373c9b6e80 msgr2=0x56373c9c0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:54.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.886+0000 7fb3a6a00640 1 --2- 172.21.15.12:0/820431684 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56373c9b6e80 0x56373c9c0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.886+0000 7fb3a6a00640 1 -- 172.21.15.12:0/820431684 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56373c5f00f0 con 0x56373c9b7870 2024-03-23T23:06:54.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.887+0000 7fb3a6a00640 1 --2- 172.21.15.12:0/820431684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56373c9b7870 0x56373c9c83a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb394014040 tx=0x7fb39400fc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8602cd1c2bfd924 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:54.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.887+0000 7fb393e00640 1 -- 172.21.15.12:0/820431684 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39401b840 con 0x56373c9b7870 2024-03-23T23:06:54.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.887+0000 7fb393e00640 1 -- 172.21.15.12:0/820431684 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb394004f90 con 0x56373c9b7870 2024-03-23T23:06:54.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.887+0000 7fb393e00640 1 -- 172.21.15.12:0/820431684 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb394004290 con 0x56373c9b7870 2024-03-23T23:06:54.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.888+0000 7fb3b11c7740 1 -- 172.21.15.12:0/820431684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56373c9b7870 msgr2=0x56373c9c83a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:54.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.888+0000 7fb3b11c7740 1 --2- 172.21.15.12:0/820431684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56373c9b7870 0x56373c9c83a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb394014040 tx=0x7fb39400fc70 comp rx=0 tx=0).stop 2024-03-23T23:06:54.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.888+0000 7fb3b11c7740 1 -- 172.21.15.12:0/820431684 shutdown_connections 2024-03-23T23:06:54.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.888+0000 7fb3b11c7740 1 --2- 172.21.15.12:0/820431684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56373c9bc9e0 0x56373c9b89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.888+0000 7fb3b11c7740 1 --2- 172.21.15.12:0/820431684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56373c9b7870 0x56373c9c83a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.888+0000 7fb3b11c7740 1 --2- 172.21.15.12:0/820431684 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56373c9b6e80 0x56373c9c0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.888+0000 7fb3b11c7740 1 -- 172.21.15.12:0/820431684 >> 172.21.15.12:0/820431684 conn(0x56373c97f200 msgr2=0x56373c9b82e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:54.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.888+0000 7fb3b11c7740 1 -- 172.21.15.12:0/820431684 shutdown_connections 2024-03-23T23:06:54.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.888+0000 7fb3b11c7740 1 -- 172.21.15.12:0/820431684 wait complete. 2024-03-23T23:06:54.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.888+0000 7fb3b11c7740 1 -- start start 2024-03-23T23:06:54.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.888+0000 7fb3b11c7740 1 Processor -- start 2024-03-23T23:06:54.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.889+0000 7fb3b11c7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56373c9b6e80 0x56373c9c59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:54.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.889+0000 7fb3b11c7740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56373c9b7870 0x56373c9c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:54.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.889+0000 7fb3b11c7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56373c9bc9e0 0x56373c9c6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:54.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.889+0000 7fb3b11c7740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a81a9300 con 0x56373c9b7870 2024-03-23T23:06:54.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.889+0000 7fb3b11c7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a81aa600 con 0x56373c9bc9e0 2024-03-23T23:06:54.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.889+0000 7fb3b11c7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a81a9940 con 0x56373c9b6e80 2024-03-23T23:06:54.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.889+0000 7fb3b11c7740 10 client.0 _ll_register_callbacks cb 0x7fb394015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:54.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.889+0000 7fb3a7400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56373c9b6e80 0x56373c9c59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:54.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.889+0000 7fb3a6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56373c9b7870 0x56373c9c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:54.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.889+0000 7fb3a7400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56373c9b6e80 0x56373c9c59e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47652/0 (socket says 172.21.15.12:47652) 2024-03-23T23:06:54.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.889+0000 7fb3a7400640 1 -- 172.21.15.12:0/2945876118 learned_addr learned my addr 172.21.15.12:0/2945876118 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:54.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.889+0000 7fb3a7e00640 1 --2- 172.21.15.12:0/2945876118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56373c9bc9e0 0x56373c9c6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:54.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.889+0000 7fb3a7400640 1 -- 172.21.15.12:0/2945876118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56373c9bc9e0 msgr2=0x56373c9c6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:54.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.889+0000 7fb3a7400640 1 --2- 172.21.15.12:0/2945876118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56373c9bc9e0 0x56373c9c6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.889+0000 7fb3a7400640 1 -- 172.21.15.12:0/2945876118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56373c9b7870 msgr2=0x56373c9c5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:54.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.889+0000 7fb3a7400640 1 --2- 172.21.15.12:0/2945876118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56373c9b7870 0x56373c9c5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.890+0000 7fb3a7400640 1 -- 172.21.15.12:0/2945876118 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56373c9c7520 con 0x56373c9b6e80 2024-03-23T23:06:54.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.890+0000 7fb3a6a00640 1 --2- 172.21.15.12:0/2945876118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56373c9b7870 0x56373c9c5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:54.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.890+0000 7fb3a7400640 1 --2- 172.21.15.12:0/2945876118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56373c9b6e80 0x56373c9c59e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb398097cc0 tx=0x7fb3980046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c2ed268db3654fa server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:54.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.890+0000 7fb391600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:06:54.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.890+0000 7fb391600640 1 -- 172.21.15.12:0/2945876118 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39809f9b0 con 0x56373c9b6e80 2024-03-23T23:06:54.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.890+0000 7fb391600640 1 -- 172.21.15.12:0/2945876118 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb3980ccad0 con 0x56373c9b6e80 2024-03-23T23:06:54.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.890+0000 7fb3b11c7740 1 -- 172.21.15.12:0/2945876118 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb394004f90 con 0x56373c9b6e80 2024-03-23T23:06:54.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.890+0000 7fb391600640 1 -- 172.21.15.12:0/2945876118 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3980a4000 con 0x56373c9b6e80 2024-03-23T23:06:54.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.890+0000 7fb3b11c7740 10 client.26593 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:54.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.890+0000 7fb3b11c7740 1 -- 172.21.15.12:0/2945876118 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56373ca280a0 con 0x56373c9b6e80 2024-03-23T23:06:54.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.890+0000 7fb391600640 1 -- 172.21.15.12:0/2945876118 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb394004f90 con 0x56373c9b6e80 2024-03-23T23:06:54.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.891+0000 7fb391600640 1 -- 172.21.15.12:0/2945876118 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fb39809cb90 con 0x56373c9b6e80 2024-03-23T23:06:54.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.891+0000 7fb3b11c7740 10 client.26593 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:54.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.891+0000 7fb3b11c7740 10 client.26593 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:06:54.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.891+0000 7fb3b11c7740 20 client.26593 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:54.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.891+0000 7fb3b11c7740 1 --2- 172.21.15.12:0/2945876118 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56373ca28a00 0x56373ca2ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:54.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.891+0000 7fb3b11c7740 4 client.26593 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:06:54.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.891+0000 7fb3b11c7740 1 -- 172.21.15.12:0/2945876118 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56373c9c0c50 con 0x56373ca28a00 2024-03-23T23:06:54.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.891+0000 7fb3a6a00640 1 --2- 172.21.15.12:0/2945876118 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56373ca28a00 0x56373ca2ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:54.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.891+0000 7fb3a6a00640 1 --2- 172.21.15.12:0/2945876118 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56373ca28a00 0x56373ca2ae20 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31bcbe1df5d904fd server_cookie=214ec59aefe3d39f in_seq=0 out_seq=0 2024-03-23T23:06:54.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.891+0000 7fb391600640 10 client.26593 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:54.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.892+0000 7fb391600640 1 -- 172.21.15.12:0/2945876118 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56373c9c0c50 con 0x56373ca28a00 2024-03-23T23:06:54.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.892+0000 7fb391600640 10 client.26593 handle_command_reply: tid=0 2024-03-23T23:06:54.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.892+0000 7fb3b11c7740 1 client.26593 shutdown 2024-03-23T23:06:54.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.892+0000 7fb3b11c7740 1 -- 172.21.15.12:0/2945876118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56373c9b6e80 msgr2=0x56373c9c59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:54.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.892+0000 7fb3b11c7740 1 --2- 172.21.15.12:0/2945876118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56373c9b6e80 0x56373c9c59e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb398097cc0 tx=0x7fb3980046c0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.892+0000 7fb3b11c7740 1 -- 172.21.15.12:0/2945876118 shutdown_connections 2024-03-23T23:06:54.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.892+0000 7fb3b11c7740 1 --2- 172.21.15.12:0/2945876118 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56373ca28a00 0x56373ca2ae20 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.892+0000 7fb3b11c7740 1 --2- 172.21.15.12:0/2945876118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56373c9bc9e0 0x56373c9c6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.892+0000 7fb3b11c7740 1 --2- 172.21.15.12:0/2945876118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56373c9b7870 0x56373c9c5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.892+0000 7fb391600640 0 client.26593 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:54.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.892+0000 7fb3b11c7740 1 --2- 172.21.15.12:0/2945876118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56373c9b6e80 0x56373c9c59e0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.892+0000 7fb3b11c7740 1 -- 172.21.15.12:0/2945876118 >> 172.21.15.12:0/2945876118 conn(0x56373c97f200 msgr2=0x56373c9b1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:54.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.892+0000 7fb3b11c7740 1 -- 172.21.15.12:0/2945876118 shutdown_connections 2024-03-23T23:06:54.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.892+0000 7fb3b11c7740 1 -- 172.21.15.12:0/2945876118 wait complete. 2024-03-23T23:06:54.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.892+0000 7fb3b11c7740 20 client.26593 trim_cache size 0 max 16384 2024-03-23T23:06:54.896 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:06:54.896 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-03-23T23:06:54.897 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-03-23T23:06:54.897 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:06:54.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.893+0000 7fb3ae400640 1 -- 172.21.15.12:0/1626646769 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb38c0a2710 msgr2=0x7fb38c0a4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:54.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.893+0000 7fb3ae400640 1 --2- 172.21.15.12:0/1626646769 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb38c0a2710 0x7fb38c0a4bd0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb3a002c640 tx=0x7fb3a0009970 comp rx=0 tx=0).stop 2024-03-23T23:06:54.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.893+0000 7fb3ae400640 1 -- 172.21.15.12:0/1626646769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3a81038e0 msgr2=0x7fb3a8199b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:54.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.893+0000 7fb3ae400640 1 --2- 172.21.15.12:0/1626646769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3a81038e0 0x7fb3a8199b90 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb39800e9c0 tx=0x7fb39800ee80 comp rx=0 tx=0).stop 2024-03-23T23:06:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.893+0000 7fb3ae400640 1 -- 172.21.15.12:0/1626646769 shutdown_connections 2024-03-23T23:06:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.893+0000 7fb3ae400640 1 --2- 172.21.15.12:0/1626646769 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb38c0a2710 0x7fb38c0a4bd0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.893+0000 7fb3ae400640 1 --2- 172.21.15.12:0/1626646769 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3a8105490 0x7fb3a81a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.893+0000 7fb3ae400640 1 --2- 172.21.15.12:0/1626646769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3a8104ad0 0x7fb3a819a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.893+0000 7fb3ae400640 1 --2- 172.21.15.12:0/1626646769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3a81038e0 0x7fb3a8199b90 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.893+0000 7fb3ae400640 1 -- 172.21.15.12:0/1626646769 >> 172.21.15.12:0/1626646769 conn(0x7fb3a80fee50 msgr2=0x7fb3a8107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.893+0000 7fb3ae400640 1 -- 172.21.15.12:0/1626646769 shutdown_connections 2024-03-23T23:06:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:54.893+0000 7fb3ae400640 1 -- 172.21.15.12:0/1626646769 wait complete. 2024-03-23T23:06:54.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:54 smithi203 ceph-mon[36693]: pgmap v543: 97 pgs: 97 active+clean; 3.1 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:06:54.908 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:54 smithi012 ceph-mon[31695]: pgmap v543: 97 pgs: 97 active+clean; 3.1 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:06:54.911 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T23:06:54.912 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:06:55.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.139+0000 7f05b4400640 1 -- 172.21.15.12:0/698555991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05ac107ce0 msgr2=0x7f05ac108100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:55.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.139+0000 7f05b4400640 1 --2- 172.21.15.12:0/698555991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05ac107ce0 0x7f05ac108100 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f05a0009900 tx=0x7f05a002bc50 comp rx=0 tx=0).stop 2024-03-23T23:06:55.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.139+0000 7f05b4400640 1 -- 172.21.15.12:0/698555991 shutdown_connections 2024-03-23T23:06:55.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.139+0000 7f05b4400640 1 --2- 172.21.15.12:0/698555991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05ac109890 0x7f05ac110120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.139+0000 7f05b4400640 1 --2- 172.21.15.12:0/698555991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05ac108ed0 0x7f05ac109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.139+0000 7f05b4400640 1 --2- 172.21.15.12:0/698555991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05ac107ce0 0x7f05ac108100 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.139+0000 7f05b4400640 1 -- 172.21.15.12:0/698555991 >> 172.21.15.12:0/698555991 conn(0x7f05ac076cb0 msgr2=0x7f05ac0790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:55.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.139+0000 7f05b4400640 1 -- 172.21.15.12:0/698555991 shutdown_connections 2024-03-23T23:06:55.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.139+0000 7f05b4400640 1 -- 172.21.15.12:0/698555991 wait complete. 2024-03-23T23:06:55.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.140+0000 7f05b4400640 1 Processor -- start 2024-03-23T23:06:55.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.141+0000 7f05b4400640 1 -- start start 2024-03-23T23:06:55.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.141+0000 7f05b4400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05ac107ce0 0x7f05ac19dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.141+0000 7f05b4400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05ac108ed0 0x7f05ac19e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.141+0000 7f05b4400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05ac109890 0x7f05ac1a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.141+0000 7f05b4400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac1139b0 con 0x7f05ac107ce0 2024-03-23T23:06:55.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.141+0000 7f05b4400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac113830 con 0x7f05ac109890 2024-03-23T23:06:55.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.141+0000 7f05b4400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac113b30 con 0x7f05ac108ed0 2024-03-23T23:06:55.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.141+0000 7f05b1a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05ac107ce0 0x7f05ac19dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.141+0000 7f05b1000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05ac108ed0 0x7f05ac19e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.141+0000 7f05b2400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05ac109890 0x7f05ac1a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.141+0000 7f05b1a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05ac107ce0 0x7f05ac19dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37618/0 (socket says 172.21.15.12:37618) 2024-03-23T23:06:55.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.141+0000 7f05b1a00640 1 -- 172.21.15.12:0/203522427 learned_addr learned my addr 172.21.15.12:0/203522427 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:55.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.142+0000 7f05b2400640 1 -- 172.21.15.12:0/203522427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05ac108ed0 msgr2=0x7f05ac19e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:55.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.142+0000 7f05b2400640 1 --2- 172.21.15.12:0/203522427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05ac108ed0 0x7f05ac19e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.142+0000 7f05b2400640 1 -- 172.21.15.12:0/203522427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05ac107ce0 msgr2=0x7f05ac19dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:55.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.142+0000 7f05b2400640 1 --2- 172.21.15.12:0/203522427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05ac107ce0 0x7f05ac19dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.142+0000 7f05b2400640 1 -- 172.21.15.12:0/203522427 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ac1a5be0 con 0x7f05ac109890 2024-03-23T23:06:55.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.142+0000 7f05b1000640 1 --2- 172.21.15.12:0/203522427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05ac108ed0 0x7f05ac19e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:55.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.142+0000 7f05b2400640 1 --2- 172.21.15.12:0/203522427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05ac109890 0x7f05ac1a5570 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f05a4009fd0 tx=0x7f05a400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:55.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.142+0000 7f05aa000640 1 -- 172.21.15.12:0/203522427 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a400ce20 con 0x7f05ac109890 2024-03-23T23:06:55.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.142+0000 7f05aa000640 1 -- 172.21.15.12:0/203522427 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05a4005020 con 0x7f05ac109890 2024-03-23T23:06:55.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.142+0000 7f05b4400640 1 -- 172.21.15.12:0/203522427 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05ac1a5ed0 con 0x7f05ac109890 2024-03-23T23:06:55.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.142+0000 7f05aa000640 1 -- 172.21.15.12:0/203522427 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a4010720 con 0x7f05ac109890 2024-03-23T23:06:55.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.143+0000 7f05b4400640 1 -- 172.21.15.12:0/203522427 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05ac1a6390 con 0x7f05ac109890 2024-03-23T23:06:55.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.144+0000 7f05b4400640 1 -- 172.21.15.12:0/203522427 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05ac10efa0 con 0x7f05ac109890 2024-03-23T23:06:55.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.146+0000 7f05aa000640 1 -- 172.21.15.12:0/203522427 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f05a4005540 con 0x7f05ac109890 2024-03-23T23:06:55.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.147+0000 7f05aa000640 1 --2- 172.21.15.12:0/203522427 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f05940a2800 0x7f05940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.147+0000 7f05aa000640 1 -- 172.21.15.12:0/203522427 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f05a40cf060 con 0x7f05ac109890 2024-03-23T23:06:55.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.148+0000 7f05b1a00640 1 --2- 172.21.15.12:0/203522427 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f05940a2800 0x7f05940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.148+0000 7f05b1a00640 1 --2- 172.21.15.12:0/203522427 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f05940a2800 0x7f05940a4cc0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f05a002c900 tx=0x7f05a0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:55.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.154+0000 7f05aa000640 1 -- 172.21.15.12:0/203522427 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f05a409c0b0 con 0x7f05ac109890 2024-03-23T23:06:55.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.405+0000 7f05b4400640 1 -- 172.21.15.12:0/203522427 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05ac02d2c0 con 0x7f05ac109890 2024-03-23T23:06:55.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.407+0000 7f05aa000640 1 -- 172.21.15.12:0/203522427 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f05a409b9b0 con 0x7f05ac109890 2024-03-23T23:06:55.407 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:06:55.407 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:06:55.407 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:06:55.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.410+0000 7f05b4400640 1 -- 172.21.15.12:0/203522427 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f05940a2800 msgr2=0x7f05940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:55.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.410+0000 7f05b4400640 1 --2- 172.21.15.12:0/203522427 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f05940a2800 0x7f05940a4cc0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f05a002c900 tx=0x7f05a0000bc0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.410+0000 7f05b4400640 1 -- 172.21.15.12:0/203522427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05ac109890 msgr2=0x7f05ac1a5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:55.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.410+0000 7f05b4400640 1 --2- 172.21.15.12:0/203522427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05ac109890 0x7f05ac1a5570 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f05a4009fd0 tx=0x7f05a400ef10 comp rx=0 tx=0).stop 2024-03-23T23:06:55.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.410+0000 7f05b4400640 1 -- 172.21.15.12:0/203522427 shutdown_connections 2024-03-23T23:06:55.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.410+0000 7f05b4400640 1 --2- 172.21.15.12:0/203522427 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f05940a2800 0x7f05940a4cc0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.410+0000 7f05b4400640 1 --2- 172.21.15.12:0/203522427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05ac109890 0x7f05ac1a5570 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.410+0000 7f05b4400640 1 --2- 172.21.15.12:0/203522427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05ac108ed0 0x7f05ac19e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.410+0000 7f05b4400640 1 --2- 172.21.15.12:0/203522427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05ac107ce0 0x7f05ac19dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.410+0000 7f05b4400640 1 -- 172.21.15.12:0/203522427 >> 172.21.15.12:0/203522427 conn(0x7f05ac076cb0 msgr2=0x7f05ac10c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:55.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.410+0000 7f05b4400640 1 -- 172.21.15.12:0/203522427 shutdown_connections 2024-03-23T23:06:55.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.410+0000 7f05b4400640 1 -- 172.21.15.12:0/203522427 wait complete. 2024-03-23T23:06:55.431 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:06:55.431 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-23T23:06:55.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.662+0000 7f67c7000640 1 -- 172.21.15.12:0/2274397299 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67c0074100 msgr2=0x7f67c0074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:55.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.662+0000 7f67c7000640 1 --2- 172.21.15.12:0/2274397299 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67c0074100 0x7f67c0074580 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f67b4009890 tx=0x7f67b402bb70 comp rx=0 tx=0).stop 2024-03-23T23:06:55.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.663+0000 7f67c7000640 1 -- 172.21.15.12:0/2274397299 shutdown_connections 2024-03-23T23:06:55.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.663+0000 7f67c7000640 1 --2- 172.21.15.12:0/2274397299 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67c010db50 0x7f67c010ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.663+0000 7f67c7000640 1 --2- 172.21.15.12:0/2274397299 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67c0074100 0x7f67c0074580 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.663+0000 7f67c7000640 1 --2- 172.21.15.12:0/2274397299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67c00756e0 0x7f67c0073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.663+0000 7f67c7000640 1 -- 172.21.15.12:0/2274397299 >> 172.21.15.12:0/2274397299 conn(0x7f67c00fcb10 msgr2=0x7f67c00fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:55.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.663+0000 7f67c7000640 1 -- 172.21.15.12:0/2274397299 shutdown_connections 2024-03-23T23:06:55.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.663+0000 7f67c7000640 1 -- 172.21.15.12:0/2274397299 wait complete. 2024-03-23T23:06:55.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.664+0000 7f67c7000640 1 Processor -- start 2024-03-23T23:06:55.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.664+0000 7f67c7000640 1 -- start start 2024-03-23T23:06:55.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.664+0000 7f67c7000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67c0074100 0x7f67c019dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.664+0000 7f67c7000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67c00756e0 0x7f67c019e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.664+0000 7f67c7000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67c010db50 0x7f67c01a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.664+0000 7f67c7000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c01137f0 con 0x7f67c00756e0 2024-03-23T23:06:55.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.664+0000 7f67c7000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c0113670 con 0x7f67c010db50 2024-03-23T23:06:55.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.664+0000 7f67c7000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c0113970 con 0x7f67c0074100 2024-03-23T23:06:55.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.664+0000 7f67c6600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67c010db50 0x7f67c01a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.664+0000 7f67c6600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67c010db50 0x7f67c01a5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:53500/0 (socket says 172.21.15.12:53500) 2024-03-23T23:06:55.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.664+0000 7f67c6600640 1 -- 172.21.15.12:0/1222618281 learned_addr learned my addr 172.21.15.12:0/1222618281 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:55.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.665+0000 7f67c5c00640 1 --2- 172.21.15.12:0/1222618281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67c0074100 0x7f67c019dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.665+0000 7f67c5200640 1 --2- 172.21.15.12:0/1222618281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67c00756e0 0x7f67c019e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.665+0000 7f67c6600640 1 -- 172.21.15.12:0/1222618281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67c0074100 msgr2=0x7f67c019dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:55.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.665+0000 7f67c6600640 1 --2- 172.21.15.12:0/1222618281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67c0074100 0x7f67c019dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.665+0000 7f67c6600640 1 -- 172.21.15.12:0/1222618281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67c00756e0 msgr2=0x7f67c019e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:55.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.665+0000 7f67c6600640 1 --2- 172.21.15.12:0/1222618281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67c00756e0 0x7f67c019e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.665+0000 7f67c6600640 1 -- 172.21.15.12:0/1222618281 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c01a5ab0 con 0x7f67c010db50 2024-03-23T23:06:55.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.665+0000 7f67c6600640 1 --2- 172.21.15.12:0/1222618281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67c010db50 0x7f67c01a5440 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f67b800efc0 tx=0x7f67b800c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:55.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.665+0000 7f67be000640 1 -- 172.21.15.12:0/1222618281 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b8003970 con 0x7f67c010db50 2024-03-23T23:06:55.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.666+0000 7f67be000640 1 -- 172.21.15.12:0/1222618281 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67b8004020 con 0x7f67c010db50 2024-03-23T23:06:55.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.666+0000 7f67c7000640 1 -- 172.21.15.12:0/1222618281 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c01a5da0 con 0x7f67c010db50 2024-03-23T23:06:55.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.666+0000 7f67be000640 1 -- 172.21.15.12:0/1222618281 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b800b9f0 con 0x7f67c010db50 2024-03-23T23:06:55.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.666+0000 7f67c7000640 1 -- 172.21.15.12:0/1222618281 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c01a6210 con 0x7f67c010db50 2024-03-23T23:06:55.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.668+0000 7f67be000640 1 -- 172.21.15.12:0/1222618281 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f67b801d020 con 0x7f67c010db50 2024-03-23T23:06:55.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.669+0000 7f67cb3cc740 1 Processor -- start 2024-03-23T23:06:55.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.669+0000 7f67cb3cc740 1 -- start start 2024-03-23T23:06:55.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.669+0000 7f67cb3cc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a902a699f0 0x55a902a69e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.670+0000 7f67cb3cc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a6abc0 0x55a902a6b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.670+0000 7f67cb3cc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a902a6b580 0x55a902a71e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.670+0000 7f67cb3cc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9029ce2c0 con 0x55a902a6b580 2024-03-23T23:06:55.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.670+0000 7f67cb3cc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a902976c90 con 0x55a902a6abc0 2024-03-23T23:06:55.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.670+0000 7f67cb3cc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9029ca360 con 0x55a902a699f0 2024-03-23T23:06:55.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.670+0000 7f67c5200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a902a699f0 0x55a902a69e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.670+0000 7f67c5c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a6abc0 0x55a902a6b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.670+0000 7f67c5200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a902a699f0 0x55a902a69e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47698/0 (socket says 172.21.15.12:47698) 2024-03-23T23:06:55.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.670+0000 7f67c5c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a6abc0 0x55a902a6b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:53502/0 (socket says 172.21.15.12:53502) 2024-03-23T23:06:55.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.670+0000 7f67c5200640 1 -- 172.21.15.12:0/2343968555 learned_addr learned my addr 172.21.15.12:0/2343968555 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:55.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.670+0000 7f67c5200640 1 --2- 172.21.15.12:0/2343968555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a902a6b580 0x55a902a71e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.670+0000 7f67c5c00640 1 -- 172.21.15.12:0/2343968555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a902a699f0 msgr2=0x55a902a69e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:55.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.670+0000 7f67c5c00640 1 --2- 172.21.15.12:0/2343968555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a902a699f0 0x55a902a69e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.670+0000 7f67c5c00640 1 -- 172.21.15.12:0/2343968555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a902a6b580 msgr2=0x55a902a71e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:55.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.670+0000 7f67c5c00640 1 --2- 172.21.15.12:0/2343968555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a902a6b580 0x55a902a71e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.670+0000 7f67c5c00640 1 -- 172.21.15.12:0/2343968555 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a90299b080 con 0x55a902a6abc0 2024-03-23T23:06:55.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.670+0000 7f67c5200640 1 --2- 172.21.15.12:0/2343968555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a902a6b580 0x55a902a71e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:06:55.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.671+0000 7f67c5c00640 1 --2- 172.21.15.12:0/2343968555 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a6abc0 0x55a902a6b040 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009c00 tx=0x7f67ac011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9616567ed0f5a00 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:55.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.671+0000 7f67be000640 1 --2- 172.21.15.12:0/1222618281 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f67a80a2800 0x7f67a80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.671+0000 7f67b2a00640 1 -- 172.21.15.12:0/2343968555 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ac004230 con 0x55a902a6abc0 2024-03-23T23:06:55.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.671+0000 7f67b2a00640 1 -- 172.21.15.12:0/2343968555 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67ac0043d0 con 0x55a902a6abc0 2024-03-23T23:06:55.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.671+0000 7f67b2a00640 1 -- 172.21.15.12:0/2343968555 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ac004dd0 con 0x55a902a6abc0 2024-03-23T23:06:55.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.671+0000 7f67be000640 1 -- 172.21.15.12:0/1222618281 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f67b80ce930 con 0x7f67c010db50 2024-03-23T23:06:55.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.671+0000 7f67c5200640 1 --2- 172.21.15.12:0/1222618281 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f67a80a2800 0x7f67a80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.672+0000 7f67cb3cc740 1 -- 172.21.15.12:0/2343968555 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a6abc0 msgr2=0x55a902a6b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:55.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.672+0000 7f67cb3cc740 1 --2- 172.21.15.12:0/2343968555 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a6abc0 0x55a902a6b040 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009c00 tx=0x7f67ac011810 comp rx=0 tx=0).stop 2024-03-23T23:06:55.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.672+0000 7f67cb3cc740 1 -- 172.21.15.12:0/2343968555 shutdown_connections 2024-03-23T23:06:55.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.672+0000 7f67cb3cc740 1 --2- 172.21.15.12:0/2343968555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a902a6b580 0x55a902a71e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.672+0000 7f67cb3cc740 1 --2- 172.21.15.12:0/2343968555 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a6abc0 0x55a902a6b040 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.672+0000 7f67cb3cc740 1 --2- 172.21.15.12:0/2343968555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a902a699f0 0x55a902a69e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.672+0000 7f67cb3cc740 1 -- 172.21.15.12:0/2343968555 >> 172.21.15.12:0/2343968555 conn(0x55a902a64de0 msgr2=0x55a902a651e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:55.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.672+0000 7f67cb3cc740 1 -- 172.21.15.12:0/2343968555 shutdown_connections 2024-03-23T23:06:55.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.672+0000 7f67c5200640 1 --2- 172.21.15.12:0/1222618281 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f67a80a2800 0x7f67a80a4cc0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f67b4000c00 tx=0x7f67b402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:55.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.672+0000 7f67cb3cc740 1 -- 172.21.15.12:0/2343968555 wait complete. 2024-03-23T23:06:55.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.672+0000 7f67cb3cc740 1 -- start start 2024-03-23T23:06:55.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.673+0000 7f67cb3cc740 1 Processor -- start 2024-03-23T23:06:55.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.673+0000 7f67cb3cc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a902a699f0 0x55a902a73c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.673+0000 7f67cb3cc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a902a6abc0 0x55a902a74170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.673+0000 7f67cb3cc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a6b580 0x55a902a7b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.673+0000 7f67cb3cc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a902a72590 con 0x55a902a6abc0 2024-03-23T23:06:55.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.673+0000 7f67cb3cc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a90298a0d0 con 0x55a902a6b580 2024-03-23T23:06:55.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.673+0000 7f67cb3cc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a902a72710 con 0x55a902a699f0 2024-03-23T23:06:55.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.673+0000 7f67cb3cc740 10 client.0 _ll_register_callbacks cb 0x55a90298ee80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:55.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.673+0000 7f67c5c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a902a699f0 0x55a902a73c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.673+0000 7f67c6600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a6b580 0x55a902a7b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.673+0000 7f67c5c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a902a699f0 0x55a902a73c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47706/0 (socket says 172.21.15.12:47706) 2024-03-23T23:06:55.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.673+0000 7f67c6600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a6b580 0x55a902a7b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:53516/0 (socket says 172.21.15.12:53516) 2024-03-23T23:06:55.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.673+0000 7f67c5c00640 1 -- 172.21.15.12:0/1989781483 learned_addr learned my addr 172.21.15.12:0/1989781483 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:55.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.673+0000 7f67c5200640 1 --2- 172.21.15.12:0/1989781483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a902a6abc0 0x55a902a74170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.674+0000 7f67c6600640 1 -- 172.21.15.12:0/1989781483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a902a699f0 msgr2=0x55a902a73c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:55.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.674+0000 7f67c6600640 1 --2- 172.21.15.12:0/1989781483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a902a699f0 0x55a902a73c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.674+0000 7f67c6600640 1 -- 172.21.15.12:0/1989781483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a902a6abc0 msgr2=0x55a902a74170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:55.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.674+0000 7f67c6600640 1 --2- 172.21.15.12:0/1989781483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a902a6abc0 0x55a902a74170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.674+0000 7f67c6600640 1 -- 172.21.15.12:0/1989781483 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a9026e23f0 con 0x55a902a6b580 2024-03-23T23:06:55.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.674+0000 7f67c5c00640 1 --2- 172.21.15.12:0/1989781483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a902a699f0 0x55a902a73c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:55.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.674+0000 7f67c6600640 1 --2- 172.21.15.12:0/1989781483 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a6b580 0x55a902a7b1b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f67b809af80 tx=0x7f67b809bac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f94a64af210abb14 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:55.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.674+0000 7f67a7400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:06:55.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.674+0000 7f67a7400640 1 -- 172.21.15.12:0/1989781483 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b80a3a20 con 0x55a902a6b580 2024-03-23T23:06:55.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.674+0000 7f67a7400640 1 -- 172.21.15.12:0/1989781483 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f67b8004540 con 0x55a902a6b580 2024-03-23T23:06:55.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.674+0000 7f67cb3cc740 1 -- 172.21.15.12:0/1989781483 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f67ac0043d0 con 0x55a902a6b580 2024-03-23T23:06:55.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.674+0000 7f67a7400640 1 -- 172.21.15.12:0/1989781483 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67b80bb0c0 con 0x55a902a6b580 2024-03-23T23:06:55.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.674+0000 7f67cb3cc740 10 client.26639 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:55.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.675+0000 7f67cb3cc740 1 -- 172.21.15.12:0/1989781483 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a9027ac7d0 con 0x55a902a6b580 2024-03-23T23:06:55.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.675+0000 7f67a7400640 1 -- 172.21.15.12:0/1989781483 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67ac0043d0 con 0x55a902a6b580 2024-03-23T23:06:55.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.675+0000 7f67a7400640 1 -- 172.21.15.12:0/1989781483 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f67b800b040 con 0x55a902a6b580 2024-03-23T23:06:55.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.675+0000 7f67cb3cc740 10 client.26639 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:55.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.675+0000 7f67cb3cc740 10 client.26639 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:06:55.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.675+0000 7f67cb3cc740 20 client.26639 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:55.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.675+0000 7f67cb3cc740 1 --2- 172.21.15.12:0/1989781483 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55a902a7f5f0 0x55a902a81a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.675+0000 7f67cb3cc740 4 client.26639 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:06:55.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.675+0000 7f67cb3cc740 1 -- 172.21.15.12:0/1989781483 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a9029db800 con 0x55a902a7f5f0 2024-03-23T23:06:55.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.675+0000 7f67c5c00640 1 --2- 172.21.15.12:0/1989781483 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55a902a7f5f0 0x55a902a81a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.676+0000 7f67c5c00640 1 --2- 172.21.15.12:0/1989781483 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55a902a7f5f0 0x55a902a81a60 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=15a2258bba15d650 server_cookie=21b3727511719ab0 in_seq=0 out_seq=0 2024-03-23T23:06:55.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.676+0000 7f67a7400640 10 client.26639 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:06:55.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.677+0000 7f67a7400640 1 -- 172.21.15.12:0/1989781483 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a9029db800 con 0x55a902a7f5f0 2024-03-23T23:06:55.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.677+0000 7f67a7400640 10 client.26639 handle_command_reply: tid=0 2024-03-23T23:06:55.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.677+0000 7f67cb3cc740 1 client.26639 shutdown 2024-03-23T23:06:55.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.677+0000 7f67cb3cc740 1 -- 172.21.15.12:0/1989781483 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a6b580 msgr2=0x55a902a7b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:55.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.677+0000 7f67cb3cc740 1 --2- 172.21.15.12:0/1989781483 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a6b580 0x55a902a7b1b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f67b809af80 tx=0x7f67b809bac0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.677+0000 7f67cb3cc740 1 -- 172.21.15.12:0/1989781483 shutdown_connections 2024-03-23T23:06:55.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.677+0000 7f67cb3cc740 1 --2- 172.21.15.12:0/1989781483 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55a902a7f5f0 0x55a902a81a60 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.677+0000 7f67cb3cc740 1 --2- 172.21.15.12:0/1989781483 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a6b580 0x55a902a7b1b0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.677+0000 7f67cb3cc740 1 --2- 172.21.15.12:0/1989781483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a902a6abc0 0x55a902a74170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.677+0000 7f67cb3cc740 1 --2- 172.21.15.12:0/1989781483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a902a699f0 0x55a902a73c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.677+0000 7f67a7400640 0 client.26639 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:06:55.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.677+0000 7f67cb3cc740 1 -- 172.21.15.12:0/1989781483 >> 172.21.15.12:0/1989781483 conn(0x55a902a64de0 msgr2=0x55a902a64160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:55.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.677+0000 7f67cb3cc740 1 -- 172.21.15.12:0/1989781483 shutdown_connections 2024-03-23T23:06:55.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.677+0000 7f67cb3cc740 1 -- 172.21.15.12:0/1989781483 wait complete. 2024-03-23T23:06:55.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.677+0000 7f67cb3cc740 20 client.26639 trim_cache size 0 max 16384 2024-03-23T23:06:55.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.691+0000 7f67cb3cc740 1 Processor -- start 2024-03-23T23:06:55.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.692+0000 7f67cb3cc740 1 -- start start 2024-03-23T23:06:55.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.692+0000 7f67cb3cc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a699f0 0x55a902a68750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.692+0000 7f67cb3cc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a902a6abc0 0x55a902a7b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.692+0000 7f67cb3cc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a902a6b580 0x55a902a862c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.692+0000 7f67cb3cc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c01b1310 con 0x55a902a6abc0 2024-03-23T23:06:55.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.692+0000 7f67cb3cc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c01aee10 con 0x55a902a699f0 2024-03-23T23:06:55.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.692+0000 7f67cb3cc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c01aceb0 con 0x55a902a6b580 2024-03-23T23:06:55.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.692+0000 7f67c5c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a699f0 0x55a902a68750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.692+0000 7f67c6600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a902a6b580 0x55a902a862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.692+0000 7f67c5c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a699f0 0x55a902a68750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:53532/0 (socket says 172.21.15.12:53532) 2024-03-23T23:06:55.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.692+0000 7f67c5c00640 1 -- 172.21.15.12:0/1473742190 learned_addr learned my addr 172.21.15.12:0/1473742190 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:55.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.692+0000 7f67c5200640 1 --2- 172.21.15.12:0/1473742190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a902a6abc0 0x55a902a7b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.692+0000 7f67c5c00640 1 -- 172.21.15.12:0/1473742190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a902a6b580 msgr2=0x55a902a862c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:55.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.692+0000 7f67c5c00640 1 --2- 172.21.15.12:0/1473742190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a902a6b580 0x55a902a862c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.692+0000 7f67c5c00640 1 -- 172.21.15.12:0/1473742190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a902a6abc0 msgr2=0x55a902a7b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:55.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.692+0000 7f67c5c00640 1 --2- 172.21.15.12:0/1473742190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a902a6abc0 0x55a902a7b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.692+0000 7f67c5c00640 1 -- 172.21.15.12:0/1473742190 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a902a74ad0 con 0x55a902a699f0 2024-03-23T23:06:55.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.692+0000 7f67c6600640 1 --2- 172.21.15.12:0/1473742190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a902a6b580 0x55a902a862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:55.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.692+0000 7f67c5200640 1 --2- 172.21.15.12:0/1473742190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a902a6abc0 0x55a902a7b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:06:55.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.693+0000 7f67c5c00640 1 --2- 172.21.15.12:0/1473742190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a699f0 0x55a902a68750 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f67ac0058c0 tx=0x7f67ac0045c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=feae9b86f812e166 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:55.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.693+0000 7f67b2000640 1 -- 172.21.15.12:0/1473742190 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ac032070 con 0x55a902a699f0 2024-03-23T23:06:55.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.693+0000 7f67b2000640 1 -- 172.21.15.12:0/1473742190 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67ac02a070 con 0x55a902a699f0 2024-03-23T23:06:55.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.693+0000 7f67b2000640 1 -- 172.21.15.12:0/1473742190 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ac01e040 con 0x55a902a699f0 2024-03-23T23:06:55.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.694+0000 7f67cb3cc740 1 -- 172.21.15.12:0/1473742190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a699f0 msgr2=0x55a902a68750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:55.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.694+0000 7f67cb3cc740 1 --2- 172.21.15.12:0/1473742190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a699f0 0x55a902a68750 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f67ac0058c0 tx=0x7f67ac0045c0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.694+0000 7f67cb3cc740 1 -- 172.21.15.12:0/1473742190 shutdown_connections 2024-03-23T23:06:55.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.694+0000 7f67cb3cc740 1 --2- 172.21.15.12:0/1473742190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a902a6b580 0x55a902a862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.694+0000 7f67cb3cc740 1 --2- 172.21.15.12:0/1473742190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a902a6abc0 0x55a902a7b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.694+0000 7f67cb3cc740 1 --2- 172.21.15.12:0/1473742190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a699f0 0x55a902a68750 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.694+0000 7f67cb3cc740 1 -- 172.21.15.12:0/1473742190 >> 172.21.15.12:0/1473742190 conn(0x55a902a64de0 msgr2=0x55a902a68040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:55.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.694+0000 7f67cb3cc740 1 -- 172.21.15.12:0/1473742190 shutdown_connections 2024-03-23T23:06:55.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.694+0000 7f67cb3cc740 1 -- 172.21.15.12:0/1473742190 wait complete. 2024-03-23T23:06:55.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.694+0000 7f67cb3cc740 1 -- start start 2024-03-23T23:06:55.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.694+0000 7f67cb3cc740 1 Processor -- start 2024-03-23T23:06:55.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.695+0000 7f67cb3cc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a699f0 0x55a902a7afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.695+0000 7f67cb3cc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a902a6abc0 0x55a902a77a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.695+0000 7f67cb3cc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a902a6b580 0x55a902a77fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.695+0000 7f67cb3cc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c01ad4b0 con 0x55a902a6b580 2024-03-23T23:06:55.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.695+0000 7f67cb3cc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c01ae7b0 con 0x55a902a699f0 2024-03-23T23:06:55.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.695+0000 7f67cb3cc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c01adaf0 con 0x55a902a6abc0 2024-03-23T23:06:55.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.695+0000 7f67cb3cc740 10 client.0 _ll_register_callbacks cb 0x7f67b8096e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:55.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.695+0000 7f67c5c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a699f0 0x55a902a7afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.695+0000 7f67c5c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a699f0 0x55a902a7afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:53540/0 (socket says 172.21.15.12:53540) 2024-03-23T23:06:55.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.695+0000 7f67c5c00640 1 -- 172.21.15.12:0/1565278000 learned_addr learned my addr 172.21.15.12:0/1565278000 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:55.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.695+0000 7f67c5200640 1 --2- 172.21.15.12:0/1565278000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a902a6abc0 0x55a902a77a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.695+0000 7f67c6600640 1 --2- 172.21.15.12:0/1565278000 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a902a6b580 0x55a902a77fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.695+0000 7f67c5c00640 1 -- 172.21.15.12:0/1565278000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a902a6abc0 msgr2=0x55a902a77a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:55.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.695+0000 7f67c5c00640 1 --2- 172.21.15.12:0/1565278000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a902a6abc0 0x55a902a77a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.695+0000 7f67c5c00640 1 -- 172.21.15.12:0/1565278000 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a902a6b580 msgr2=0x55a902a77fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:55.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.695+0000 7f67c5c00640 1 --2- 172.21.15.12:0/1565278000 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a902a6b580 0x55a902a77fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.695+0000 7f67c5c00640 1 -- 172.21.15.12:0/1565278000 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a902adae90 con 0x55a902a699f0 2024-03-23T23:06:55.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.696+0000 7f67c5c00640 1 --2- 172.21.15.12:0/1565278000 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a699f0 0x55a902a7afd0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f67ac005890 tx=0x7f67ac009fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23ee0048acf1d69c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:55.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.696+0000 7f67a7400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:06:55.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.696+0000 7f67a7400640 1 -- 172.21.15.12:0/1565278000 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ac032070 con 0x55a902a699f0 2024-03-23T23:06:55.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.696+0000 7f67a7400640 1 -- 172.21.15.12:0/1565278000 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f67ac0269d0 con 0x55a902a699f0 2024-03-23T23:06:55.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.696+0000 7f67cb3cc740 1 -- 172.21.15.12:0/1565278000 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f67ac02a070 con 0x55a902a699f0 2024-03-23T23:06:55.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.696+0000 7f67a7400640 1 -- 172.21.15.12:0/1565278000 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67ac03e050 con 0x55a902a699f0 2024-03-23T23:06:55.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.696+0000 7f67cb3cc740 10 client.26645 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:55.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.696+0000 7f67cb3cc740 1 -- 172.21.15.12:0/1565278000 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a902adc0a0 con 0x55a902a699f0 2024-03-23T23:06:55.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.697+0000 7f67a7400640 1 -- 172.21.15.12:0/1565278000 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67ac02a070 con 0x55a902a699f0 2024-03-23T23:06:55.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.697+0000 7f67a7400640 1 -- 172.21.15.12:0/1565278000 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f67ac024040 con 0x55a902a699f0 2024-03-23T23:06:55.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.697+0000 7f67cb3cc740 10 client.26645 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:55.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.697+0000 7f67cb3cc740 10 client.26645 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:06:55.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.697+0000 7f67cb3cc740 20 client.26645 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:55.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.697+0000 7f67cb3cc740 1 --2- 172.21.15.12:0/1565278000 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55a902adc810 0x55a902adec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.697+0000 7f67cb3cc740 4 client.26645 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:06:55.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.697+0000 7f67cb3cc740 1 -- 172.21.15.12:0/1565278000 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a9029d5b50 con 0x55a902adc810 2024-03-23T23:06:55.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.697+0000 7f67c5200640 1 --2- 172.21.15.12:0/1565278000 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55a902adc810 0x55a902adec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.698+0000 7f67c5200640 1 --2- 172.21.15.12:0/1565278000 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55a902adc810 0x55a902adec30 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=95c344b5b69d63c6 server_cookie=f67ce5b5a9aa5795 in_seq=0 out_seq=0 2024-03-23T23:06:55.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.698+0000 7f67a7400640 10 client.26645 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:06:55.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.698+0000 7f67a7400640 1 -- 172.21.15.12:0/1565278000 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a9029d5b50 con 0x55a902adc810 2024-03-23T23:06:55.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.698+0000 7f67a7400640 10 client.26645 handle_command_reply: tid=0 2024-03-23T23:06:55.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.698+0000 7f67cb3cc740 1 client.26645 shutdown 2024-03-23T23:06:55.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.698+0000 7f67cb3cc740 1 -- 172.21.15.12:0/1565278000 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a699f0 msgr2=0x55a902a7afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:55.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.698+0000 7f67cb3cc740 1 --2- 172.21.15.12:0/1565278000 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a699f0 0x55a902a7afd0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f67ac005890 tx=0x7f67ac009fa0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.698+0000 7f67cb3cc740 1 -- 172.21.15.12:0/1565278000 shutdown_connections 2024-03-23T23:06:55.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.698+0000 7f67cb3cc740 1 --2- 172.21.15.12:0/1565278000 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55a902adc810 0x55a902adec30 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.698+0000 7f67cb3cc740 1 --2- 172.21.15.12:0/1565278000 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a902a6b580 0x55a902a77fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.698+0000 7f67cb3cc740 1 --2- 172.21.15.12:0/1565278000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a902a6abc0 0x55a902a77a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.698+0000 7f67a7400640 0 client.26645 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:06:55.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.698+0000 7f67cb3cc740 1 --2- 172.21.15.12:0/1565278000 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a902a699f0 0x55a902a7afd0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.698+0000 7f67cb3cc740 1 -- 172.21.15.12:0/1565278000 >> 172.21.15.12:0/1565278000 conn(0x55a902a64de0 msgr2=0x55a902a688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:55.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.698+0000 7f67cb3cc740 1 -- 172.21.15.12:0/1565278000 shutdown_connections 2024-03-23T23:06:55.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.698+0000 7f67cb3cc740 1 -- 172.21.15.12:0/1565278000 wait complete. 2024-03-23T23:06:55.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.698+0000 7f67cb3cc740 20 client.26645 trim_cache size 0 max 16384 2024-03-23T23:06:55.703 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:06:55.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.699+0000 7f67c7000640 1 -- 172.21.15.12:0/1222618281 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f67a80a2800 msgr2=0x7f67a80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:55.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.699+0000 7f67c7000640 1 --2- 172.21.15.12:0/1222618281 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f67a80a2800 0x7f67a80a4cc0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f67b4000c00 tx=0x7f67b402e040 comp rx=0 tx=0).stop 2024-03-23T23:06:55.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.699+0000 7f67c7000640 1 -- 172.21.15.12:0/1222618281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67c010db50 msgr2=0x7f67c01a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:55.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.699+0000 7f67c7000640 1 --2- 172.21.15.12:0/1222618281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67c010db50 0x7f67c01a5440 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f67b800efc0 tx=0x7f67b800c800 comp rx=0 tx=0).stop 2024-03-23T23:06:55.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.699+0000 7f67c7000640 1 -- 172.21.15.12:0/1222618281 shutdown_connections 2024-03-23T23:06:55.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.699+0000 7f67c7000640 1 --2- 172.21.15.12:0/1222618281 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f67a80a2800 0x7f67a80a4cc0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.699+0000 7f67c7000640 1 --2- 172.21.15.12:0/1222618281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67c010db50 0x7f67c01a5440 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.699+0000 7f67c7000640 1 --2- 172.21.15.12:0/1222618281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67c00756e0 0x7f67c019e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.699+0000 7f67c7000640 1 --2- 172.21.15.12:0/1222618281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67c0074100 0x7f67c019dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.699+0000 7f67c7000640 1 -- 172.21.15.12:0/1222618281 >> 172.21.15.12:0/1222618281 conn(0x7f67c00fcb10 msgr2=0x7f67c0100de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:55.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.699+0000 7f67c7000640 1 -- 172.21.15.12:0/1222618281 shutdown_connections 2024-03-23T23:06:55.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.699+0000 7f67c7000640 1 -- 172.21.15.12:0/1222618281 wait complete. 2024-03-23T23:06:55.714 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:55 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/203522427' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:06:55.717 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:06:55.717 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-23T23:06:55.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:55 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/203522427' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:06:55.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:55 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/203522427' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:06:55.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.939+0000 7f2ab8a00640 1 -- 172.21.15.12:0/3443147966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ab4105370 msgr2=0x7f2ab410bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:55.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.939+0000 7f2ab8a00640 1 --2- 172.21.15.12:0/3443147966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ab4105370 0x7f2ab410bc00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f2aac009ca0 tx=0x7f2aac02c100 comp rx=0 tx=0).stop 2024-03-23T23:06:55.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.940+0000 7f2ab8a00640 1 -- 172.21.15.12:0/3443147966 shutdown_connections 2024-03-23T23:06:55.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.940+0000 7f2ab8a00640 1 --2- 172.21.15.12:0/3443147966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ab4105370 0x7f2ab410bc00 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.940+0000 7f2ab8a00640 1 --2- 172.21.15.12:0/3443147966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ab41049b0 0x7f2ab4104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.940+0000 7f2ab8a00640 1 --2- 172.21.15.12:0/3443147966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ab41037c0 0x7f2ab4103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.940+0000 7f2ab8a00640 1 -- 172.21.15.12:0/3443147966 >> 172.21.15.12:0/3443147966 conn(0x7f2ab40fed50 msgr2=0x7f2ab4101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:55.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.940+0000 7f2ab8a00640 1 -- 172.21.15.12:0/3443147966 shutdown_connections 2024-03-23T23:06:55.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.940+0000 7f2ab8a00640 1 -- 172.21.15.12:0/3443147966 wait complete. 2024-03-23T23:06:55.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.941+0000 7f2ab8a00640 1 Processor -- start 2024-03-23T23:06:55.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.941+0000 7f2ab8a00640 1 -- start start 2024-03-23T23:06:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.941+0000 7f2ab8a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ab41037c0 0x7f2ab4199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.941+0000 7f2ab8a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ab41049b0 0x7f2ab4199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.941+0000 7f2ab8a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ab4105370 0x7f2ab41a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.941+0000 7f2ab8a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab410f420 con 0x7f2ab4105370 2024-03-23T23:06:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.941+0000 7f2ab8a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab410f2a0 con 0x7f2ab41049b0 2024-03-23T23:06:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.941+0000 7f2ab8a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab410f5a0 con 0x7f2ab41037c0 2024-03-23T23:06:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.941+0000 7f2ab3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ab4105370 0x7f2ab41a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.941+0000 7f2ab3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ab41037c0 0x7f2ab4199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.941+0000 7f2ab2a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ab41049b0 0x7f2ab4199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.941+0000 7f2ab3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ab4105370 0x7f2ab41a1030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37696/0 (socket says 172.21.15.12:37696) 2024-03-23T23:06:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.941+0000 7f2ab3e00640 1 -- 172.21.15.12:0/241952147 learned_addr learned my addr 172.21.15.12:0/241952147 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.942+0000 7f2ab2a00640 1 -- 172.21.15.12:0/241952147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ab41037c0 msgr2=0x7f2ab4199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.942+0000 7f2ab2a00640 1 --2- 172.21.15.12:0/241952147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ab41037c0 0x7f2ab4199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.942+0000 7f2ab2a00640 1 -- 172.21.15.12:0/241952147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ab4105370 msgr2=0x7f2ab41a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.942+0000 7f2ab2a00640 1 --2- 172.21.15.12:0/241952147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ab4105370 0x7f2ab41a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.942+0000 7f2ab2a00640 1 -- 172.21.15.12:0/241952147 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab41a16a0 con 0x7f2ab41049b0 2024-03-23T23:06:55.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.942+0000 7f2ab3e00640 1 --2- 172.21.15.12:0/241952147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ab4105370 0x7f2ab41a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:55.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.942+0000 7f2ab2a00640 1 --2- 172.21.15.12:0/241952147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ab41049b0 0x7f2ab4199fc0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2aa000e9f0 tx=0x7f2aa000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:55.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.942+0000 7f2aabe00640 1 -- 172.21.15.12:0/241952147 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa000cd80 con 0x7f2ab41049b0 2024-03-23T23:06:55.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.942+0000 7f2ab8a00640 1 -- 172.21.15.12:0/241952147 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab41a1990 con 0x7f2ab41049b0 2024-03-23T23:06:55.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.942+0000 7f2aabe00640 1 -- 172.21.15.12:0/241952147 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2aa00035f0 con 0x7f2ab41049b0 2024-03-23T23:06:55.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.942+0000 7f2aabe00640 1 -- 172.21.15.12:0/241952147 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0010670 con 0x7f2ab41049b0 2024-03-23T23:06:55.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.943+0000 7f2ab8a00640 1 -- 172.21.15.12:0/241952147 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab41a1f30 con 0x7f2ab41049b0 2024-03-23T23:06:55.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.946+0000 7f2aabe00640 1 -- 172.21.15.12:0/241952147 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2aa00037a0 con 0x7f2ab41049b0 2024-03-23T23:06:55.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.946+0000 7f2abcdeb740 1 Processor -- start 2024-03-23T23:06:55.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.946+0000 7f2abcdeb740 1 -- start start 2024-03-23T23:06:55.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.946+0000 7f2abcdeb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563118aed9f0 0x563118aede10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.946+0000 7f2abcdeb740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563118aeebc0 0x563118aef040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.946+0000 7f2abcdeb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aef580 0x563118af5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.946+0000 7f2abcdeb740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563118a522c0 con 0x563118aeebc0 2024-03-23T23:06:55.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.946+0000 7f2abcdeb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631189fac90 con 0x563118aef580 2024-03-23T23:06:55.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.946+0000 7f2abcdeb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x563118a4e360 con 0x563118aed9f0 2024-03-23T23:06:55.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.947+0000 7f2ab3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563118aed9f0 0x563118aede10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.947+0000 7f2ab3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563118aed9f0 0x563118aede10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47748/0 (socket says 172.21.15.12:47748) 2024-03-23T23:06:55.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.947+0000 7f2ab3e00640 1 -- 172.21.15.12:0/1152612633 learned_addr learned my addr 172.21.15.12:0/1152612633 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:55.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.947+0000 7f2ab2a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aef580 0x563118af5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.947+0000 7f2ab3400640 1 --2- 172.21.15.12:0/1152612633 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563118aeebc0 0x563118aef040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.947+0000 7f2ab3e00640 1 -- 172.21.15.12:0/1152612633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aef580 msgr2=0x563118af5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:55.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.947+0000 7f2ab3e00640 1 --2- 172.21.15.12:0/1152612633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aef580 0x563118af5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.947+0000 7f2ab3e00640 1 -- 172.21.15.12:0/1152612633 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563118aeebc0 msgr2=0x563118aef040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:55.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.947+0000 7f2ab3e00640 1 --2- 172.21.15.12:0/1152612633 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563118aeebc0 0x563118aef040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.947+0000 7f2ab3e00640 1 -- 172.21.15.12:0/1152612633 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563118a1f080 con 0x563118aed9f0 2024-03-23T23:06:55.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.948+0000 7f2ab3e00640 1 --2- 172.21.15.12:0/1152612633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563118aed9f0 0x563118aede10 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2aac032370 tx=0x7f2aac0323a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86bb6d765b335bda server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:55.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.948+0000 7f2a9fe00640 1 -- 172.21.15.12:0/1152612633 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aac043020 con 0x563118aed9f0 2024-03-23T23:06:55.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.948+0000 7f2aabe00640 1 --2- 172.21.15.12:0/241952147 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2a980a2800 0x7f2a980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.948+0000 7f2a9fe00640 1 -- 172.21.15.12:0/1152612633 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2aac003970 con 0x563118aed9f0 2024-03-23T23:06:55.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.948+0000 7f2a9fe00640 1 -- 172.21.15.12:0/1152612633 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aac039040 con 0x563118aed9f0 2024-03-23T23:06:55.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.948+0000 7f2aabe00640 1 -- 172.21.15.12:0/241952147 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2aa00cecf0 con 0x7f2ab41049b0 2024-03-23T23:06:55.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.948+0000 7f2ab3400640 1 --2- 172.21.15.12:0/241952147 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2a980a2800 0x7f2a980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.949+0000 7f2abcdeb740 1 -- 172.21.15.12:0/1152612633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563118aed9f0 msgr2=0x563118aede10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.949+0000 7f2abcdeb740 1 --2- 172.21.15.12:0/1152612633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563118aed9f0 0x563118aede10 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2aac032370 tx=0x7f2aac0323a0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.949+0000 7f2abcdeb740 1 -- 172.21.15.12:0/1152612633 shutdown_connections 2024-03-23T23:06:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.949+0000 7f2abcdeb740 1 --2- 172.21.15.12:0/1152612633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aef580 0x563118af5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.949+0000 7f2abcdeb740 1 --2- 172.21.15.12:0/1152612633 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563118aeebc0 0x563118aef040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.949+0000 7f2abcdeb740 1 --2- 172.21.15.12:0/1152612633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563118aed9f0 0x563118aede10 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.949+0000 7f2abcdeb740 1 -- 172.21.15.12:0/1152612633 >> 172.21.15.12:0/1152612633 conn(0x563118ae8de0 msgr2=0x563118ae91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.949+0000 7f2abcdeb740 1 -- 172.21.15.12:0/1152612633 shutdown_connections 2024-03-23T23:06:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.949+0000 7f2abcdeb740 1 -- 172.21.15.12:0/1152612633 wait complete. 2024-03-23T23:06:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.949+0000 7f2ab3400640 1 --2- 172.21.15.12:0/241952147 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2a980a2800 0x7f2a980a4cc0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4104810 tx=0x7f2aa4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.949+0000 7f2abcdeb740 1 -- start start 2024-03-23T23:06:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.950+0000 7f2abcdeb740 1 Processor -- start 2024-03-23T23:06:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.950+0000 7f2abcdeb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aed9f0 0x563118af7c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.950+0000 7f2abcdeb740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563118aeebc0 0x563118af8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.950+0000 7f2abcdeb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563118aef580 0x563118aff1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.950+0000 7f2abcdeb740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563118af6530 con 0x563118aeebc0 2024-03-23T23:06:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.950+0000 7f2abcdeb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x563118a0e0d0 con 0x563118aed9f0 2024-03-23T23:06:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.950+0000 7f2abcdeb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x563118af66b0 con 0x563118aef580 2024-03-23T23:06:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.950+0000 7f2abcdeb740 10 client.0 _ll_register_callbacks cb 0x563118a12e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.950+0000 7f2ab2a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563118aeebc0 0x563118af8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.950+0000 7f2ab3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aed9f0 0x563118af7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.951+0000 7f2ab3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563118aef580 0x563118aff1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.951+0000 7f2ab3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aed9f0 0x563118af7c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:53582/0 (socket says 172.21.15.12:53582) 2024-03-23T23:06:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.951+0000 7f2ab3400640 1 -- 172.21.15.12:0/361690509 learned_addr learned my addr 172.21.15.12:0/361690509 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.951+0000 7f2ab3e00640 1 -- 172.21.15.12:0/361690509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aed9f0 msgr2=0x563118af7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.951+0000 7f2ab3e00640 1 --2- 172.21.15.12:0/361690509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aed9f0 0x563118af7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.951+0000 7f2ab3e00640 1 -- 172.21.15.12:0/361690509 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563118aeebc0 msgr2=0x563118af8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.951+0000 7f2ab3e00640 1 --2- 172.21.15.12:0/361690509 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563118aeebc0 0x563118af8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.951+0000 7f2ab3e00640 1 -- 172.21.15.12:0/361690509 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5631187663f0 con 0x563118aef580 2024-03-23T23:06:55.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.951+0000 7f2ab3400640 1 --2- 172.21.15.12:0/361690509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aed9f0 0x563118af7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:55.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.951+0000 7f2ab2a00640 1 --2- 172.21.15.12:0/361690509 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563118aeebc0 0x563118af8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:55.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.951+0000 7f2ab3e00640 1 --2- 172.21.15.12:0/361690509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563118aef580 0x563118aff1b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2aac009880 tx=0x7f2aac003f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=def6676a8a291b20 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:55.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.951+0000 7f2a9cc00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:06:55.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.952+0000 7f2a9cc00640 1 -- 172.21.15.12:0/361690509 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aac04c030 con 0x563118aef580 2024-03-23T23:06:55.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.952+0000 7f2a9cc00640 1 -- 172.21.15.12:0/361690509 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2aac032690 con 0x563118aef580 2024-03-23T23:06:55.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.952+0000 7f2abcdeb740 1 -- 172.21.15.12:0/361690509 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2aac003970 con 0x563118aef580 2024-03-23T23:06:55.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.952+0000 7f2a9cc00640 1 -- 172.21.15.12:0/361690509 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2aac043050 con 0x563118aef580 2024-03-23T23:06:55.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.952+0000 7f2abcdeb740 10 client.26629 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:55.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.952+0000 7f2abcdeb740 1 -- 172.21.15.12:0/361690509 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5631188307d0 con 0x563118aef580 2024-03-23T23:06:55.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.952+0000 7f2a9cc00640 1 -- 172.21.15.12:0/361690509 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2aac003970 con 0x563118aef580 2024-03-23T23:06:55.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.952+0000 7f2a9cc00640 1 -- 172.21.15.12:0/361690509 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f2aac004500 con 0x563118aef580 2024-03-23T23:06:55.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.952+0000 7f2abcdeb740 10 client.26629 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:55.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.952+0000 7f2abcdeb740 10 client.26629 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:06:55.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.952+0000 7f2abcdeb740 20 client.26629 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:55.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.952+0000 7f2abcdeb740 1 --2- 172.21.15.12:0/361690509 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x563118b03310 0x563118b05700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.952+0000 7f2abcdeb740 4 client.26629 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:06:55.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.952+0000 7f2abcdeb740 1 -- 172.21.15.12:0/361690509 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563118a5f800 con 0x563118b03310 2024-03-23T23:06:55.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.953+0000 7f2ab3400640 1 --2- 172.21.15.12:0/361690509 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x563118b03310 0x563118b05700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.953+0000 7f2ab3400640 1 --2- 172.21.15.12:0/361690509 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x563118b03310 0x563118b05700 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f45b30694c25c7c8 server_cookie=3c2e6a2225bb1681 in_seq=0 out_seq=0 2024-03-23T23:06:55.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.953+0000 7f2a9cc00640 10 client.26629 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:06:55.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.954+0000 7f2a9cc00640 1 -- 172.21.15.12:0/361690509 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563118a5f800 con 0x563118b03310 2024-03-23T23:06:55.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.954+0000 7f2a9cc00640 10 client.26629 handle_command_reply: tid=0 2024-03-23T23:06:55.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.954+0000 7f2abcdeb740 1 client.26629 shutdown 2024-03-23T23:06:55.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.954+0000 7f2abcdeb740 1 -- 172.21.15.12:0/361690509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563118aef580 msgr2=0x563118aff1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:55.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.954+0000 7f2abcdeb740 1 --2- 172.21.15.12:0/361690509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563118aef580 0x563118aff1b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2aac009880 tx=0x7f2aac003f00 comp rx=0 tx=0).stop 2024-03-23T23:06:55.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.954+0000 7f2abcdeb740 1 -- 172.21.15.12:0/361690509 shutdown_connections 2024-03-23T23:06:55.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.954+0000 7f2abcdeb740 1 --2- 172.21.15.12:0/361690509 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x563118b03310 0x563118b05700 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.954+0000 7f2abcdeb740 1 --2- 172.21.15.12:0/361690509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563118aef580 0x563118aff1b0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.954+0000 7f2a9cc00640 0 client.26629 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:06:55.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.954+0000 7f2abcdeb740 1 --2- 172.21.15.12:0/361690509 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563118aeebc0 0x563118af8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.954+0000 7f2abcdeb740 1 --2- 172.21.15.12:0/361690509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aed9f0 0x563118af7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.954+0000 7f2abcdeb740 1 -- 172.21.15.12:0/361690509 >> 172.21.15.12:0/361690509 conn(0x563118ae8de0 msgr2=0x563118ae8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:55.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.954+0000 7f2abcdeb740 1 -- 172.21.15.12:0/361690509 shutdown_connections 2024-03-23T23:06:55.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.954+0000 7f2abcdeb740 1 -- 172.21.15.12:0/361690509 wait complete. 2024-03-23T23:06:55.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.955+0000 7f2abcdeb740 20 client.26629 trim_cache size 0 max 16384 2024-03-23T23:06:55.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.968+0000 7f2abcdeb740 1 Processor -- start 2024-03-23T23:06:55.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.969+0000 7f2abcdeb740 1 -- start start 2024-03-23T23:06:55.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.969+0000 7f2abcdeb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563118aed9f0 0x563118aec750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.969+0000 7f2abcdeb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aeebc0 0x563118aff170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.969+0000 7f2abcdeb740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563118aef580 0x563118af1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.969+0000 7f2abcdeb740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab41ace90 con 0x563118aef580 2024-03-23T23:06:55.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.969+0000 7f2abcdeb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab41aa990 con 0x563118aeebc0 2024-03-23T23:06:55.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.969+0000 7f2abcdeb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab41a8a30 con 0x563118aed9f0 2024-03-23T23:06:55.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.969+0000 7f2ab2a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aeebc0 0x563118aff170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.969+0000 7f2ab3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563118aed9f0 0x563118aec750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.969+0000 7f2ab2a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aeebc0 0x563118aff170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:53586/0 (socket says 172.21.15.12:53586) 2024-03-23T23:06:55.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.969+0000 7f2ab3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563118aef580 0x563118af1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.969+0000 7f2ab2a00640 1 -- 172.21.15.12:0/116390689 learned_addr learned my addr 172.21.15.12:0/116390689 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:55.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.969+0000 7f2ab2a00640 1 -- 172.21.15.12:0/116390689 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563118aed9f0 msgr2=0x563118aec750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:55.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.969+0000 7f2ab2a00640 1 --2- 172.21.15.12:0/116390689 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563118aed9f0 0x563118aec750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.969+0000 7f2ab2a00640 1 -- 172.21.15.12:0/116390689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563118aef580 msgr2=0x563118af1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:55.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.969+0000 7f2ab2a00640 1 --2- 172.21.15.12:0/116390689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563118aef580 0x563118af1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.970+0000 7f2ab2a00640 1 -- 172.21.15.12:0/116390689 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563118aee3e0 con 0x563118aeebc0 2024-03-23T23:06:55.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.970+0000 7f2ab2a00640 1 --2- 172.21.15.12:0/116390689 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aeebc0 0x563118aff170 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2aa009b630 tx=0x7f2aa00d2760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d0c8b44d0e4f065 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:55.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.970+0000 7f2a9f400640 1 -- 172.21.15.12:0/116390689 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa009b1e0 con 0x563118aeebc0 2024-03-23T23:06:55.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.970+0000 7f2a9f400640 1 -- 172.21.15.12:0/116390689 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2aa009bda0 con 0x563118aeebc0 2024-03-23T23:06:55.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.970+0000 7f2a9f400640 1 -- 172.21.15.12:0/116390689 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa009b000 con 0x563118aeebc0 2024-03-23T23:06:55.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.971+0000 7f2abcdeb740 1 -- 172.21.15.12:0/116390689 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aeebc0 msgr2=0x563118aff170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:55.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.971+0000 7f2abcdeb740 1 --2- 172.21.15.12:0/116390689 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aeebc0 0x563118aff170 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2aa009b630 tx=0x7f2aa00d2760 comp rx=0 tx=0).stop 2024-03-23T23:06:55.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.971+0000 7f2abcdeb740 1 -- 172.21.15.12:0/116390689 shutdown_connections 2024-03-23T23:06:55.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.971+0000 7f2abcdeb740 1 --2- 172.21.15.12:0/116390689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563118aef580 0x563118af1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.971+0000 7f2abcdeb740 1 --2- 172.21.15.12:0/116390689 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aeebc0 0x563118aff170 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.971+0000 7f2abcdeb740 1 --2- 172.21.15.12:0/116390689 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563118aed9f0 0x563118aec750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.971+0000 7f2abcdeb740 1 -- 172.21.15.12:0/116390689 >> 172.21.15.12:0/116390689 conn(0x563118ae8de0 msgr2=0x563118aec040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:55.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.971+0000 7f2abcdeb740 1 -- 172.21.15.12:0/116390689 shutdown_connections 2024-03-23T23:06:55.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.971+0000 7f2abcdeb740 1 -- 172.21.15.12:0/116390689 wait complete. 2024-03-23T23:06:55.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.971+0000 7f2abcdeb740 1 -- start start 2024-03-23T23:06:55.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.971+0000 7f2abcdeb740 1 Processor -- start 2024-03-23T23:06:55.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.972+0000 7f2abcdeb740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563118aed9f0 0x563118afabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.972+0000 7f2abcdeb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aeebc0 0x563118afb110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.972+0000 7f2abcdeb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563118aef580 0x563118afe130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.972+0000 7f2abcdeb740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab41a9030 con 0x563118aed9f0 2024-03-23T23:06:55.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.972+0000 7f2abcdeb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab41aa330 con 0x563118aeebc0 2024-03-23T23:06:55.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.972+0000 7f2abcdeb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab41a9670 con 0x563118aef580 2024-03-23T23:06:55.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.972+0000 7f2abcdeb740 10 client.0 _ll_register_callbacks cb 0x7f2aa40097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:55.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.972+0000 7f2ab2a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aeebc0 0x563118afb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.972+0000 7f2ab2a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aeebc0 0x563118afb110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:53588/0 (socket says 172.21.15.12:53588) 2024-03-23T23:06:55.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.972+0000 7f2ab2a00640 1 -- 172.21.15.12:0/514274492 learned_addr learned my addr 172.21.15.12:0/514274492 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:55.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.972+0000 7f2ab3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563118aef580 0x563118afe130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.972+0000 7f2ab3400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563118aed9f0 0x563118afabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.972+0000 7f2ab2a00640 1 -- 172.21.15.12:0/514274492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563118aef580 msgr2=0x563118afe130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:55.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.972+0000 7f2ab2a00640 1 --2- 172.21.15.12:0/514274492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563118aef580 0x563118afe130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.973+0000 7f2ab2a00640 1 -- 172.21.15.12:0/514274492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563118aed9f0 msgr2=0x563118afabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:55.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.973+0000 7f2ab2a00640 1 --2- 172.21.15.12:0/514274492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563118aed9f0 0x563118afabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.973+0000 7f2ab2a00640 1 -- 172.21.15.12:0/514274492 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563118afe720 con 0x563118aeebc0 2024-03-23T23:06:55.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.973+0000 7f2ab3e00640 1 --2- 172.21.15.12:0/514274492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563118aef580 0x563118afe130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:55.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.973+0000 7f2ab3400640 1 --2- 172.21.15.12:0/514274492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563118aed9f0 0x563118afabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:55.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.973+0000 7f2ab2a00640 1 --2- 172.21.15.12:0/514274492 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aeebc0 0x563118afb110 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2aa000e9c0 tx=0x7f2aa0097340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=453b88bafa2e84ab server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:55.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.973+0000 7f2a9cc00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:06:55.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.973+0000 7f2a9cc00640 1 -- 172.21.15.12:0/514274492 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa009b000 con 0x563118aeebc0 2024-03-23T23:06:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.973+0000 7f2a9cc00640 1 -- 172.21.15.12:0/514274492 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2aa0097d10 con 0x563118aeebc0 2024-03-23T23:06:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.973+0000 7f2abcdeb740 1 -- 172.21.15.12:0/514274492 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2aa009bda0 con 0x563118aeebc0 2024-03-23T23:06:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.973+0000 7f2a9cc00640 1 -- 172.21.15.12:0/514274492 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2aa009b1e0 con 0x563118aeebc0 2024-03-23T23:06:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.973+0000 7f2abcdeb740 10 client.26663 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.973+0000 7f2abcdeb740 1 -- 172.21.15.12:0/514274492 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563118b5ee00 con 0x563118aeebc0 2024-03-23T23:06:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.973+0000 7f2a9cc00640 1 -- 172.21.15.12:0/514274492 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2aa009bda0 con 0x563118aeebc0 2024-03-23T23:06:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.974+0000 7f2a9cc00640 1 -- 172.21.15.12:0/514274492 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f2aa009aaf0 con 0x563118aeebc0 2024-03-23T23:06:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.974+0000 7f2abcdeb740 10 client.26663 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.974+0000 7f2abcdeb740 10 client.26663 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:06:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.974+0000 7f2abcdeb740 20 client.26663 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.974+0000 7f2abcdeb740 1 --2- 172.21.15.12:0/514274492 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x563118b5f3d0 0x563118b61850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.974+0000 7f2abcdeb740 4 client.26663 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:06:55.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.974+0000 7f2abcdeb740 1 -- 172.21.15.12:0/514274492 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563118a59b50 con 0x563118b5f3d0 2024-03-23T23:06:55.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.974+0000 7f2ab3400640 1 --2- 172.21.15.12:0/514274492 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x563118b5f3d0 0x563118b61850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:55.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.974+0000 7f2ab3400640 1 --2- 172.21.15.12:0/514274492 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x563118b5f3d0 0x563118b61850 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c126b567a407bd08 server_cookie=8f817d4e3e8f814a in_seq=0 out_seq=0 2024-03-23T23:06:55.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.974+0000 7f2a9cc00640 10 client.26663 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:06:55.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.975+0000 7f2a9cc00640 1 -- 172.21.15.12:0/514274492 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563118a59b50 con 0x563118b5f3d0 2024-03-23T23:06:55.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.975+0000 7f2a9cc00640 10 client.26663 handle_command_reply: tid=0 2024-03-23T23:06:55.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.975+0000 7f2abcdeb740 1 client.26663 shutdown 2024-03-23T23:06:55.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.975+0000 7f2abcdeb740 1 -- 172.21.15.12:0/514274492 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aeebc0 msgr2=0x563118afb110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:55.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.975+0000 7f2abcdeb740 1 --2- 172.21.15.12:0/514274492 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aeebc0 0x563118afb110 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2aa000e9c0 tx=0x7f2aa0097340 comp rx=0 tx=0).stop 2024-03-23T23:06:55.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.975+0000 7f2abcdeb740 1 -- 172.21.15.12:0/514274492 shutdown_connections 2024-03-23T23:06:55.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.975+0000 7f2abcdeb740 1 --2- 172.21.15.12:0/514274492 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x563118b5f3d0 0x563118b61850 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.975+0000 7f2abcdeb740 1 --2- 172.21.15.12:0/514274492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563118aef580 0x563118afe130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.975+0000 7f2abcdeb740 1 --2- 172.21.15.12:0/514274492 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563118aeebc0 0x563118afb110 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.975+0000 7f2abcdeb740 1 --2- 172.21.15.12:0/514274492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563118aed9f0 0x563118afabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.975+0000 7f2abcdeb740 1 -- 172.21.15.12:0/514274492 >> 172.21.15.12:0/514274492 conn(0x563118ae8de0 msgr2=0x563118aec8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:55.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.975+0000 7f2a9cc00640 0 client.26663 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:06:55.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.975+0000 7f2abcdeb740 1 -- 172.21.15.12:0/514274492 shutdown_connections 2024-03-23T23:06:55.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.975+0000 7f2abcdeb740 1 -- 172.21.15.12:0/514274492 wait complete. 2024-03-23T23:06:55.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.975+0000 7f2abcdeb740 20 client.26663 trim_cache size 0 max 16384 2024-03-23T23:06:55.981 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:06:55.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.976+0000 7f2ab8a00640 1 -- 172.21.15.12:0/241952147 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2a980a2800 msgr2=0x7f2a980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:55.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.976+0000 7f2ab8a00640 1 --2- 172.21.15.12:0/241952147 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2a980a2800 0x7f2a980a4cc0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4104810 tx=0x7f2aa4005f90 comp rx=0 tx=0).stop 2024-03-23T23:06:55.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.976+0000 7f2ab8a00640 1 -- 172.21.15.12:0/241952147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ab41049b0 msgr2=0x7f2ab4199fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:55.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.976+0000 7f2ab8a00640 1 --2- 172.21.15.12:0/241952147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ab41049b0 0x7f2ab4199fc0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2aa000e9f0 tx=0x7f2aa000eeb0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.976+0000 7f2ab8a00640 1 -- 172.21.15.12:0/241952147 shutdown_connections 2024-03-23T23:06:55.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.976+0000 7f2ab8a00640 1 --2- 172.21.15.12:0/241952147 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2a980a2800 0x7f2a980a4cc0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.976+0000 7f2ab8a00640 1 --2- 172.21.15.12:0/241952147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ab4105370 0x7f2ab41a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.976+0000 7f2ab8a00640 1 --2- 172.21.15.12:0/241952147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ab41049b0 0x7f2ab4199fc0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.976+0000 7f2ab8a00640 1 --2- 172.21.15.12:0/241952147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ab41037c0 0x7f2ab4199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:55.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.976+0000 7f2ab8a00640 1 -- 172.21.15.12:0/241952147 >> 172.21.15.12:0/241952147 conn(0x7f2ab40fed50 msgr2=0x7f2ab4107c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:55.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.976+0000 7f2ab8a00640 1 -- 172.21.15.12:0/241952147 shutdown_connections 2024-03-23T23:06:55.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:55.976+0000 7f2ab8a00640 1 -- 172.21.15.12:0/241952147 wait complete. 2024-03-23T23:06:55.993 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:06:55.993 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-23T23:06:56.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.202+0000 7f744be00640 1 -- 172.21.15.12:0/1706980113 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f744c107bc0 msgr2=0x7f744c107fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:56.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.202+0000 7f744be00640 1 --2- 172.21.15.12:0/1706980113 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f744c107bc0 0x7f744c107fe0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f743c009ca0 tx=0x7f743c02c0f0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.202+0000 7f744be00640 1 -- 172.21.15.12:0/1706980113 shutdown_connections 2024-03-23T23:06:56.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.202+0000 7f744be00640 1 --2- 172.21.15.12:0/1706980113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f744c109770 0x7f744c110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.202+0000 7f744be00640 1 --2- 172.21.15.12:0/1706980113 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f744c108db0 0x7f744c109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.202+0000 7f744be00640 1 --2- 172.21.15.12:0/1706980113 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f744c107bc0 0x7f744c107fe0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.202+0000 7f744be00640 1 -- 172.21.15.12:0/1706980113 >> 172.21.15.12:0/1706980113 conn(0x7f744c076b70 msgr2=0x7f744c078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:56.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.203+0000 7f744be00640 1 -- 172.21.15.12:0/1706980113 shutdown_connections 2024-03-23T23:06:56.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.203+0000 7f744be00640 1 -- 172.21.15.12:0/1706980113 wait complete. 2024-03-23T23:06:56.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.204+0000 7f744be00640 1 Processor -- start 2024-03-23T23:06:56.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.204+0000 7f744be00640 1 -- start start 2024-03-23T23:06:56.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.204+0000 7f744be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f744c107bc0 0x7f744c19dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:56.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.204+0000 7f744be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f744c108db0 0x7f744c19e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:56.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.204+0000 7f744be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f744c109770 0x7f744c1a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:56.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.204+0000 7f744be00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744c113840 con 0x7f744c109770 2024-03-23T23:06:56.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.204+0000 7f744be00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744c1136c0 con 0x7f744c108db0 2024-03-23T23:06:56.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.204+0000 7f744be00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744c1139c0 con 0x7f744c107bc0 2024-03-23T23:06:56.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.205+0000 7f744aa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f744c107bc0 0x7f744c19dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:56.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.205+0000 7f744a000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f744c108db0 0x7f744c19e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:56.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.205+0000 7f744aa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f744c107bc0 0x7f744c19dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47798/0 (socket says 172.21.15.12:47798) 2024-03-23T23:06:56.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.205+0000 7f744aa00640 1 -- 172.21.15.12:0/3142694845 learned_addr learned my addr 172.21.15.12:0/3142694845 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:56.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.205+0000 7f744b400640 1 --2- 172.21.15.12:0/3142694845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f744c109770 0x7f744c1a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:56.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.205+0000 7f744aa00640 1 -- 172.21.15.12:0/3142694845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f744c108db0 msgr2=0x7f744c19e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:56.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.205+0000 7f744aa00640 1 --2- 172.21.15.12:0/3142694845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f744c108db0 0x7f744c19e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.205+0000 7f744aa00640 1 -- 172.21.15.12:0/3142694845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f744c109770 msgr2=0x7f744c1a5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:56.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.205+0000 7f744aa00640 1 --2- 172.21.15.12:0/3142694845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f744c109770 0x7f744c1a5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.205+0000 7f744aa00640 1 -- 172.21.15.12:0/3142694845 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f744c1a5ae0 con 0x7f744c107bc0 2024-03-23T23:06:56.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.205+0000 7f744a000640 1 --2- 172.21.15.12:0/3142694845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f744c108db0 0x7f744c19e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:56.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.205+0000 7f744aa00640 1 --2- 172.21.15.12:0/3142694845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f744c107bc0 0x7f744c19dec0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f743c000f80 tx=0x7f743c032ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:56.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.206+0000 7f7450a00640 1 -- 172.21.15.12:0/3142694845 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743c041020 con 0x7f744c107bc0 2024-03-23T23:06:56.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.206+0000 7f7450a00640 1 -- 172.21.15.12:0/3142694845 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f743c034cd0 con 0x7f744c107bc0 2024-03-23T23:06:56.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.206+0000 7f744be00640 1 -- 172.21.15.12:0/3142694845 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f744c1a5d70 con 0x7f744c107bc0 2024-03-23T23:06:56.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.206+0000 7f7450a00640 1 -- 172.21.15.12:0/3142694845 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743c039040 con 0x7f744c107bc0 2024-03-23T23:06:56.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.206+0000 7f744be00640 1 -- 172.21.15.12:0/3142694845 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f744c1a6180 con 0x7f744c107bc0 2024-03-23T23:06:56.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.209+0000 7f7450a00640 1 -- 172.21.15.12:0/3142694845 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f743c044020 con 0x7f744c107bc0 2024-03-23T23:06:56.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.209+0000 7f7454bd9740 1 Processor -- start 2024-03-23T23:06:56.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.209+0000 7f7454bd9740 1 -- start start 2024-03-23T23:06:56.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.210+0000 7f7454bd9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59b19f0 0x55bbf59b1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:56.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.210+0000 7f7454bd9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bbf59b2bc0 0x55bbf59b3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:56.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.210+0000 7f7454bd9740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bbf59b3580 0x55bbf59b9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:56.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.210+0000 7f7454bd9740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbf59162c0 con 0x55bbf59b3580 2024-03-23T23:06:56.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.210+0000 7f7454bd9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbf58bec90 con 0x55bbf59b19f0 2024-03-23T23:06:56.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.210+0000 7f7454bd9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbf5912360 con 0x55bbf59b2bc0 2024-03-23T23:06:56.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.210+0000 7f744aa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bbf59b2bc0 0x55bbf59b3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:56.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.210+0000 7f744b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59b19f0 0x55bbf59b1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:56.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.210+0000 7f744a000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bbf59b3580 0x55bbf59b9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:56.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.210+0000 7f744aa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bbf59b2bc0 0x55bbf59b3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47806/0 (socket says 172.21.15.12:47806) 2024-03-23T23:06:56.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.210+0000 7f744b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59b19f0 0x55bbf59b1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:53614/0 (socket says 172.21.15.12:53614) 2024-03-23T23:06:56.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.210+0000 7f744aa00640 1 -- 172.21.15.12:0/2852213135 learned_addr learned my addr 172.21.15.12:0/2852213135 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:56.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.210+0000 7f744aa00640 1 -- 172.21.15.12:0/2852213135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59b19f0 msgr2=0x55bbf59b1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:56.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.210+0000 7f744aa00640 1 --2- 172.21.15.12:0/2852213135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59b19f0 0x55bbf59b1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.210+0000 7f744aa00640 1 -- 172.21.15.12:0/2852213135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bbf59b3580 msgr2=0x55bbf59b9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:56.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.210+0000 7f744aa00640 1 --2- 172.21.15.12:0/2852213135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bbf59b3580 0x55bbf59b9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.210+0000 7f744aa00640 1 -- 172.21.15.12:0/2852213135 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bbf58e3080 con 0x55bbf59b2bc0 2024-03-23T23:06:56.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.211+0000 7f7450a00640 1 --2- 172.21.15.12:0/3142694845 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f74300a2800 0x7f74300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:56.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.211+0000 7f744aa00640 1 --2- 172.21.15.12:0/2852213135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bbf59b2bc0 0x55bbf59b3040 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f743c0f6a90 tx=0x7f743c0f6b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6a660232f4d616e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:56.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.211+0000 7f744b400640 1 --2- 172.21.15.12:0/3142694845 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f74300a2800 0x7f74300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:56.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.211+0000 7f7450a00640 1 -- 172.21.15.12:0/3142694845 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f743c0f08f0 con 0x7f744c107bc0 2024-03-23T23:06:56.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.211+0000 7f7437e00640 1 -- 172.21.15.12:0/2852213135 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743c0f3e60 con 0x55bbf59b2bc0 2024-03-23T23:06:56.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.211+0000 7f7437e00640 1 -- 172.21.15.12:0/2852213135 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f743c0c1160 con 0x55bbf59b2bc0 2024-03-23T23:06:56.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.211+0000 7f7437e00640 1 -- 172.21.15.12:0/2852213135 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743c0b94e0 con 0x55bbf59b2bc0 2024-03-23T23:06:56.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.212+0000 7f7454bd9740 1 -- 172.21.15.12:0/2852213135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bbf59b2bc0 msgr2=0x55bbf59b3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:56.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.212+0000 7f7454bd9740 1 --2- 172.21.15.12:0/2852213135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bbf59b2bc0 0x55bbf59b3040 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f743c0f6a90 tx=0x7f743c0f6b80 comp rx=0 tx=0).stop 2024-03-23T23:06:56.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.212+0000 7f744b400640 1 --2- 172.21.15.12:0/3142694845 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f74300a2800 0x7f74300a4cc0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x55bbf59b2ac0 tx=0x7f7444005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:56.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.212+0000 7f7454bd9740 1 -- 172.21.15.12:0/2852213135 shutdown_connections 2024-03-23T23:06:56.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.212+0000 7f7454bd9740 1 --2- 172.21.15.12:0/2852213135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bbf59b3580 0x55bbf59b9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.212+0000 7f7454bd9740 1 --2- 172.21.15.12:0/2852213135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bbf59b2bc0 0x55bbf59b3040 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.212+0000 7f7454bd9740 1 --2- 172.21.15.12:0/2852213135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59b19f0 0x55bbf59b1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.212+0000 7f7454bd9740 1 -- 172.21.15.12:0/2852213135 >> 172.21.15.12:0/2852213135 conn(0x55bbf59acde0 msgr2=0x55bbf59ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:56.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.212+0000 7f7454bd9740 1 -- 172.21.15.12:0/2852213135 shutdown_connections 2024-03-23T23:06:56.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.212+0000 7f7454bd9740 1 -- 172.21.15.12:0/2852213135 wait complete. 2024-03-23T23:06:56.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.212+0000 7f7454bd9740 1 -- start start 2024-03-23T23:06:56.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.213+0000 7f7454bd9740 1 Processor -- start 2024-03-23T23:06:56.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.213+0000 7f7454bd9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59b19f0 0x55bbf59bb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:56.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.213+0000 7f7454bd9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bbf59b2bc0 0x55bbf59bbea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:56.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.213+0000 7f7454bd9740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bbf59b3580 0x55bbf59c2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:56.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.213+0000 7f7454bd9740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbf59ba350 con 0x55bbf59b3580 2024-03-23T23:06:56.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.213+0000 7f7454bd9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbf58d20d0 con 0x55bbf59b19f0 2024-03-23T23:06:56.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.213+0000 7f7454bd9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbf59ba4d0 con 0x55bbf59b2bc0 2024-03-23T23:06:56.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.213+0000 7f7454bd9740 10 client.0 _ll_register_callbacks cb 0x55bbf58d6e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:56.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.213+0000 7f744aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59b19f0 0x55bbf59bb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:56.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.213+0000 7f744a000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bbf59b2bc0 0x55bbf59bbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:56.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.213+0000 7f744aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59b19f0 0x55bbf59bb960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:53616/0 (socket says 172.21.15.12:53616) 2024-03-23T23:06:56.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.213+0000 7f744aa00640 1 -- 172.21.15.12:0/659167684 learned_addr learned my addr 172.21.15.12:0/659167684 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:56.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.213+0000 7f744b400640 1 --2- 172.21.15.12:0/659167684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bbf59b3580 0x55bbf59c2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:56.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.214+0000 7f744aa00640 1 -- 172.21.15.12:0/659167684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bbf59b2bc0 msgr2=0x55bbf59bbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:56.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.214+0000 7f744aa00640 1 --2- 172.21.15.12:0/659167684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bbf59b2bc0 0x55bbf59bbea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.214+0000 7f744aa00640 1 -- 172.21.15.12:0/659167684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bbf59b3580 msgr2=0x55bbf59c2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:56.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.214+0000 7f744aa00640 1 --2- 172.21.15.12:0/659167684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bbf59b3580 0x55bbf59c2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.214+0000 7f744aa00640 1 -- 172.21.15.12:0/659167684 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bbf562a3f0 con 0x55bbf59b19f0 2024-03-23T23:06:56.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.214+0000 7f744aa00640 1 --2- 172.21.15.12:0/659167684 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59b19f0 0x55bbf59bb960 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f743c009620 tx=0x7f743c005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dd0805a8ff46e03 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:56.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.214+0000 7f7434c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:06:56.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.214+0000 7f7434c00640 1 -- 172.21.15.12:0/659167684 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743c0f3e60 con 0x55bbf59b19f0 2024-03-23T23:06:56.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.214+0000 7f7434c00640 1 -- 172.21.15.12:0/659167684 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f743c0dd0c0 con 0x55bbf59b19f0 2024-03-23T23:06:56.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.214+0000 7f7454bd9740 1 -- 172.21.15.12:0/659167684 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f743c0c1160 con 0x55bbf59b19f0 2024-03-23T23:06:56.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.214+0000 7f7434c00640 1 -- 172.21.15.12:0/659167684 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f743c0bcba0 con 0x55bbf59b19f0 2024-03-23T23:06:56.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.214+0000 7f7454bd9740 10 client.26675 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:56.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.214+0000 7f7454bd9740 1 -- 172.21.15.12:0/659167684 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55bbf56f47d0 con 0x55bbf59b19f0 2024-03-23T23:06:56.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.215+0000 7f7434c00640 1 -- 172.21.15.12:0/659167684 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f743c0c1160 con 0x55bbf59b19f0 2024-03-23T23:06:56.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.215+0000 7f7434c00640 1 -- 172.21.15.12:0/659167684 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f743c0f0ce0 con 0x55bbf59b19f0 2024-03-23T23:06:56.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.215+0000 7f7454bd9740 10 client.26675 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:56.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.215+0000 7f7454bd9740 10 client.26675 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:06:56.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.215+0000 7f7454bd9740 20 client.26675 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:56.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.215+0000 7f7454bd9740 1 --2- 172.21.15.12:0/659167684 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55bbf59c7100 0x55bbf59c94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:56.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.215+0000 7f7454bd9740 4 client.26675 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:06:56.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.215+0000 7f7454bd9740 1 -- 172.21.15.12:0/659167684 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bbf5923800 con 0x55bbf59c7100 2024-03-23T23:06:56.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.215+0000 7f744a000640 1 --2- 172.21.15.12:0/659167684 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55bbf59c7100 0x55bbf59c94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:56.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.216+0000 7f744a000640 1 --2- 172.21.15.12:0/659167684 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55bbf59c7100 0x55bbf59c94f0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc649829aef4ecb3 server_cookie=93dd5a98835a9ccc in_seq=0 out_seq=0 2024-03-23T23:06:56.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.216+0000 7f7434c00640 10 client.26675 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:56.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.216+0000 7f7434c00640 1 -- 172.21.15.12:0/659167684 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55bbf5923800 con 0x55bbf59c7100 2024-03-23T23:06:56.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.217+0000 7f7434c00640 10 client.26675 handle_command_reply: tid=0 2024-03-23T23:06:56.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.217+0000 7f7454bd9740 1 client.26675 shutdown 2024-03-23T23:06:56.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.217+0000 7f7454bd9740 1 -- 172.21.15.12:0/659167684 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59b19f0 msgr2=0x55bbf59bb960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:56.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.217+0000 7f7454bd9740 1 --2- 172.21.15.12:0/659167684 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59b19f0 0x55bbf59bb960 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f743c009620 tx=0x7f743c005ef0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.217+0000 7f7454bd9740 1 -- 172.21.15.12:0/659167684 shutdown_connections 2024-03-23T23:06:56.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.217+0000 7f7454bd9740 1 --2- 172.21.15.12:0/659167684 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55bbf59c7100 0x55bbf59c94f0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.217+0000 7f7454bd9740 1 --2- 172.21.15.12:0/659167684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bbf59b3580 0x55bbf59c2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.217+0000 7f7454bd9740 1 --2- 172.21.15.12:0/659167684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bbf59b2bc0 0x55bbf59bbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.217+0000 7f7434c00640 0 client.26675 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:56.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.217+0000 7f7454bd9740 1 --2- 172.21.15.12:0/659167684 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59b19f0 0x55bbf59bb960 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.217+0000 7f7454bd9740 1 -- 172.21.15.12:0/659167684 >> 172.21.15.12:0/659167684 conn(0x55bbf59acde0 msgr2=0x55bbf59ac160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:56.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.217+0000 7f7454bd9740 1 -- 172.21.15.12:0/659167684 shutdown_connections 2024-03-23T23:06:56.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.217+0000 7f7454bd9740 1 -- 172.21.15.12:0/659167684 wait complete. 2024-03-23T23:06:56.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.217+0000 7f7454bd9740 20 client.26675 trim_cache size 0 max 16384 2024-03-23T23:06:56.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.231+0000 7f7454bd9740 1 Processor -- start 2024-03-23T23:06:56.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.231+0000 7f7454bd9740 1 -- start start 2024-03-23T23:06:56.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.231+0000 7f7454bd9740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bbf59acde0 0x55bbf59baf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:56.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.231+0000 7f7454bd9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59b19f0 0x55bbf59c2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:56.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.231+0000 7f7454bd9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bbf59b2bc0 0x55bbf59bb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:56.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.231+0000 7f7454bd9740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744c1b13b0 con 0x55bbf59acde0 2024-03-23T23:06:56.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.231+0000 7f7454bd9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744c1aeeb0 con 0x55bbf59b19f0 2024-03-23T23:06:56.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.231+0000 7f7454bd9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744c1acf50 con 0x55bbf59b2bc0 2024-03-23T23:06:56.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.232+0000 7f744a000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59b19f0 0x55bbf59c2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:56.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.232+0000 7f744b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bbf59b2bc0 0x55bbf59bb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:56.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.232+0000 7f744a000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59b19f0 0x55bbf59c2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:53626/0 (socket says 172.21.15.12:53626) 2024-03-23T23:06:56.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.232+0000 7f744aa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bbf59acde0 0x55bbf59baf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:56.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.232+0000 7f744a000640 1 -- 172.21.15.12:0/2061877643 learned_addr learned my addr 172.21.15.12:0/2061877643 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:56.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.232+0000 7f744a000640 1 -- 172.21.15.12:0/2061877643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bbf59b2bc0 msgr2=0x55bbf59bb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:56.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.232+0000 7f744a000640 1 --2- 172.21.15.12:0/2061877643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bbf59b2bc0 0x55bbf59bb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.232+0000 7f744a000640 1 -- 172.21.15.12:0/2061877643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bbf59acde0 msgr2=0x55bbf59baf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:56.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.232+0000 7f744a000640 1 --2- 172.21.15.12:0/2061877643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bbf59acde0 0x55bbf59baf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.232+0000 7f744a000640 1 -- 172.21.15.12:0/2061877643 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bbf59b9710 con 0x55bbf59b19f0 2024-03-23T23:06:56.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.232+0000 7f744a000640 1 --2- 172.21.15.12:0/2061877643 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59b19f0 0x55bbf59c2e70 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f743800b6e0 tx=0x7f743800b410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd8ab967c6fc9fad server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:56.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.233+0000 7f7437400640 1 -- 172.21.15.12:0/2061877643 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7438020230 con 0x55bbf59b19f0 2024-03-23T23:06:56.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.233+0000 7f7437400640 1 -- 172.21.15.12:0/2061877643 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7438024020 con 0x55bbf59b19f0 2024-03-23T23:06:56.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.233+0000 7f7437400640 1 -- 172.21.15.12:0/2061877643 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7438010690 con 0x55bbf59b19f0 2024-03-23T23:06:56.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.233+0000 7f7454bd9740 1 -- 172.21.15.12:0/2061877643 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59b19f0 msgr2=0x55bbf59c2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:56.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.233+0000 7f7454bd9740 1 --2- 172.21.15.12:0/2061877643 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59b19f0 0x55bbf59c2e70 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f743800b6e0 tx=0x7f743800b410 comp rx=0 tx=0).stop 2024-03-23T23:06:56.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.233+0000 7f7454bd9740 1 -- 172.21.15.12:0/2061877643 shutdown_connections 2024-03-23T23:06:56.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.233+0000 7f7454bd9740 1 --2- 172.21.15.12:0/2061877643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bbf59b2bc0 0x55bbf59bb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.233+0000 7f7454bd9740 1 --2- 172.21.15.12:0/2061877643 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59b19f0 0x55bbf59c2e70 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.233+0000 7f7454bd9740 1 --2- 172.21.15.12:0/2061877643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bbf59acde0 0x55bbf59baf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.233+0000 7f7454bd9740 1 -- 172.21.15.12:0/2061877643 >> 172.21.15.12:0/2061877643 conn(0x55bbf59b0c70 msgr2=0x55bbf59b6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.233+0000 7f7454bd9740 1 -- 172.21.15.12:0/2061877643 shutdown_connections 2024-03-23T23:06:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.233+0000 7f7454bd9740 1 -- 172.21.15.12:0/2061877643 wait complete. 2024-03-23T23:06:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.234+0000 7f7454bd9740 1 -- start start 2024-03-23T23:06:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.234+0000 7f7454bd9740 1 Processor -- start 2024-03-23T23:06:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.234+0000 7f7454bd9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59acde0 0x55bbf59c28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.234+0000 7f7454bd9740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bbf59b19f0 0x55bbf59c2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.234+0000 7f7454bd9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bbf59b2bc0 0x55bbf59bf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.234+0000 7f7454bd9740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744c1ad550 con 0x55bbf59b19f0 2024-03-23T23:06:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.234+0000 7f7454bd9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744c1ae850 con 0x55bbf59acde0 2024-03-23T23:06:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.234+0000 7f7454bd9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744c1adb90 con 0x55bbf59b2bc0 2024-03-23T23:06:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.235+0000 7f7454bd9740 10 client.0 _ll_register_callbacks cb 0x7f743800bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.235+0000 7f744aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59acde0 0x55bbf59c28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.235+0000 7f744aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59acde0 0x55bbf59c28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:53634/0 (socket says 172.21.15.12:53634) 2024-03-23T23:06:56.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.235+0000 7f744aa00640 1 -- 172.21.15.12:0/450321900 learned_addr learned my addr 172.21.15.12:0/450321900 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:56.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.235+0000 7f744b400640 1 --2- 172.21.15.12:0/450321900 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bbf59b2bc0 0x55bbf59bf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:56.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.235+0000 7f744a000640 1 --2- 172.21.15.12:0/450321900 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bbf59b19f0 0x55bbf59c2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:56.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.235+0000 7f744aa00640 1 -- 172.21.15.12:0/450321900 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bbf59b2bc0 msgr2=0x55bbf59bf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:56.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.235+0000 7f744aa00640 1 --2- 172.21.15.12:0/450321900 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bbf59b2bc0 0x55bbf59bf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.235+0000 7f744aa00640 1 -- 172.21.15.12:0/450321900 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bbf59b19f0 msgr2=0x55bbf59c2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:56.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.235+0000 7f744aa00640 1 --2- 172.21.15.12:0/450321900 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bbf59b19f0 0x55bbf59c2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.235+0000 7f744aa00640 1 -- 172.21.15.12:0/450321900 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bbf59bfc70 con 0x55bbf59acde0 2024-03-23T23:06:56.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.235+0000 7f744a000640 1 --2- 172.21.15.12:0/450321900 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bbf59b19f0 0x55bbf59c2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:06:56.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.235+0000 7f744aa00640 1 --2- 172.21.15.12:0/450321900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59acde0 0x55bbf59c28d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f743c0bcf40 tx=0x7f743c003970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e7402f5ab0e2498 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:56.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.235+0000 7f7434c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:06:56.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.235+0000 7f7434c00640 1 -- 172.21.15.12:0/450321900 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743c0f3e60 con 0x55bbf59acde0 2024-03-23T23:06:56.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.235+0000 7f7434c00640 1 -- 172.21.15.12:0/450321900 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f743c0f6220 con 0x55bbf59acde0 2024-03-23T23:06:56.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.236+0000 7f7454bd9740 1 -- 172.21.15.12:0/450321900 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7438024020 con 0x55bbf59acde0 2024-03-23T23:06:56.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.236+0000 7f7434c00640 1 -- 172.21.15.12:0/450321900 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f743c0bcd80 con 0x55bbf59acde0 2024-03-23T23:06:56.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.236+0000 7f7454bd9740 10 client.26681 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:56.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.236+0000 7f7454bd9740 1 -- 172.21.15.12:0/450321900 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55bbf59bfc70 con 0x55bbf59acde0 2024-03-23T23:06:56.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.236+0000 7f7434c00640 1 -- 172.21.15.12:0/450321900 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7438024020 con 0x55bbf59acde0 2024-03-23T23:06:56.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.236+0000 7f7434c00640 1 -- 172.21.15.12:0/450321900 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f743c03c030 con 0x55bbf59acde0 2024-03-23T23:06:56.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.236+0000 7f7454bd9740 10 client.26681 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:56.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.236+0000 7f7454bd9740 10 client.26681 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:06:56.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.236+0000 7f7454bd9740 20 client.26681 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:56.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.236+0000 7f7454bd9740 1 --2- 172.21.15.12:0/450321900 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55bbf59b3580 0x55bbf5a25d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:56.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.236+0000 7f7454bd9740 4 client.26681 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:06:56.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.236+0000 7f7454bd9740 1 -- 172.21.15.12:0/450321900 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55bbf591db50 con 0x55bbf59b3580 2024-03-23T23:06:56.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.236+0000 7f744a000640 1 --2- 172.21.15.12:0/450321900 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55bbf59b3580 0x55bbf5a25d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:56.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.237+0000 7f744a000640 1 --2- 172.21.15.12:0/450321900 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55bbf59b3580 0x55bbf5a25d10 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3da5ae301a07ee0e server_cookie=c4c0d04f0a57fe25 in_seq=0 out_seq=0 2024-03-23T23:06:56.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.237+0000 7f7434c00640 10 client.26681 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:56.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.237+0000 7f7434c00640 1 -- 172.21.15.12:0/450321900 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55bbf591db50 con 0x55bbf59b3580 2024-03-23T23:06:56.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.237+0000 7f7434c00640 10 client.26681 handle_command_reply: tid=0 2024-03-23T23:06:56.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.237+0000 7f7454bd9740 1 client.26681 shutdown 2024-03-23T23:06:56.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.237+0000 7f7454bd9740 1 -- 172.21.15.12:0/450321900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59acde0 msgr2=0x55bbf59c28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:56.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.237+0000 7f7454bd9740 1 --2- 172.21.15.12:0/450321900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59acde0 0x55bbf59c28d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f743c0bcf40 tx=0x7f743c003970 comp rx=0 tx=0).stop 2024-03-23T23:06:56.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.237+0000 7f7454bd9740 1 -- 172.21.15.12:0/450321900 shutdown_connections 2024-03-23T23:06:56.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.237+0000 7f7454bd9740 1 --2- 172.21.15.12:0/450321900 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55bbf59b3580 0x55bbf5a25d10 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.237+0000 7f7454bd9740 1 --2- 172.21.15.12:0/450321900 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bbf59b2bc0 0x55bbf59bf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.237+0000 7f7454bd9740 1 --2- 172.21.15.12:0/450321900 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bbf59b19f0 0x55bbf59c2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.237+0000 7f7454bd9740 1 --2- 172.21.15.12:0/450321900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bbf59acde0 0x55bbf59c28d0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.237+0000 7f7434c00640 0 client.26681 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:56.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.237+0000 7f7454bd9740 1 -- 172.21.15.12:0/450321900 >> 172.21.15.12:0/450321900 conn(0x55bbf59b0c70 msgr2=0x55bbf59baf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:56.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.237+0000 7f7454bd9740 1 -- 172.21.15.12:0/450321900 shutdown_connections 2024-03-23T23:06:56.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.238+0000 7f7454bd9740 1 -- 172.21.15.12:0/450321900 wait complete. 2024-03-23T23:06:56.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.238+0000 7f7454bd9740 20 client.26681 trim_cache size 0 max 16384 2024-03-23T23:06:56.242 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:06:56.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.238+0000 7f744be00640 1 -- 172.21.15.12:0/3142694845 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f74300a2800 msgr2=0x7f74300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:56.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.238+0000 7f744be00640 1 --2- 172.21.15.12:0/3142694845 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f74300a2800 0x7f74300a4cc0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x55bbf59b2ac0 tx=0x7f7444005f90 comp rx=0 tx=0).stop 2024-03-23T23:06:56.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.238+0000 7f744be00640 1 -- 172.21.15.12:0/3142694845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f744c107bc0 msgr2=0x7f744c19dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:56.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.238+0000 7f744be00640 1 --2- 172.21.15.12:0/3142694845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f744c107bc0 0x7f744c19dec0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f743c000f80 tx=0x7f743c032ec0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.238+0000 7f744be00640 1 -- 172.21.15.12:0/3142694845 shutdown_connections 2024-03-23T23:06:56.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.238+0000 7f744be00640 1 --2- 172.21.15.12:0/3142694845 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f74300a2800 0x7f74300a4cc0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.238+0000 7f744be00640 1 --2- 172.21.15.12:0/3142694845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f744c109770 0x7f744c1a5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.238+0000 7f744be00640 1 --2- 172.21.15.12:0/3142694845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f744c108db0 0x7f744c19e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.238+0000 7f744be00640 1 --2- 172.21.15.12:0/3142694845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f744c107bc0 0x7f744c19dec0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:56.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.238+0000 7f744be00640 1 -- 172.21.15.12:0/3142694845 >> 172.21.15.12:0/3142694845 conn(0x7f744c076b70 msgr2=0x7f744c10c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:56.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.239+0000 7f744be00640 1 -- 172.21.15.12:0/3142694845 shutdown_connections 2024-03-23T23:06:56.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:56.239+0000 7f744be00640 1 -- 172.21.15.12:0/3142694845 wait complete. 2024-03-23T23:06:56.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:56 smithi173 ceph-mon[36116]: pgmap v544: 97 pgs: 97 active+clean; 3.1 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:06:56.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:56 smithi203 ceph-mon[36693]: pgmap v544: 97 pgs: 97 active+clean; 3.1 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:06:57.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:56 smithi012 ceph-mon[31695]: pgmap v544: 97 pgs: 97 active+clean; 3.1 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:06:57.258 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-23T23:06:57.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:06:57.259 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-23T23:06:57.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.497+0000 7f0651400640 1 -- 172.21.15.12:0/1526374986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f064c1038c0 msgr2=0x7f064c103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:57.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.497+0000 7f0651400640 1 --2- 172.21.15.12:0/1526374986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f064c1038c0 0x7f064c103ce0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f063c009900 tx=0x7f063c02bc40 comp rx=0 tx=0).stop 2024-03-23T23:06:57.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.498+0000 7f0651400640 1 -- 172.21.15.12:0/1526374986 shutdown_connections 2024-03-23T23:06:57.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.498+0000 7f0651400640 1 --2- 172.21.15.12:0/1526374986 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f064c105470 0x7f064c10bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.498+0000 7f0651400640 1 --2- 172.21.15.12:0/1526374986 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f064c104ab0 0x7f064c104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.498+0000 7f0651400640 1 --2- 172.21.15.12:0/1526374986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f064c1038c0 0x7f064c103ce0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.498+0000 7f0651400640 1 -- 172.21.15.12:0/1526374986 >> 172.21.15.12:0/1526374986 conn(0x7f064c0fee50 msgr2=0x7f064c101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:57.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.498+0000 7f0651400640 1 -- 172.21.15.12:0/1526374986 shutdown_connections 2024-03-23T23:06:57.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.498+0000 7f0651400640 1 -- 172.21.15.12:0/1526374986 wait complete. 2024-03-23T23:06:57.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.499+0000 7f0651400640 1 Processor -- start 2024-03-23T23:06:57.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.499+0000 7f0651400640 1 -- start start 2024-03-23T23:06:57.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.499+0000 7f0651400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f064c1038c0 0x7f064c199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.499+0000 7f0651400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f064c104ab0 0x7f064c19a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.499+0000 7f0651400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f064c105470 0x7f064c1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.499+0000 7f0651400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064c10f530 con 0x7f064c105470 2024-03-23T23:06:57.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.499+0000 7f0651400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064c10f3b0 con 0x7f064c104ab0 2024-03-23T23:06:57.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.499+0000 7f0651400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064c10f6b0 con 0x7f064c1038c0 2024-03-23T23:06:57.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.499+0000 7f064a800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f064c1038c0 0x7f064c199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.499+0000 7f0641e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f064c104ab0 0x7f064c19a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.499+0000 7f064b200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f064c105470 0x7f064c1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.499+0000 7f064a800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f064c1038c0 0x7f064c199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47840/0 (socket says 172.21.15.12:47840) 2024-03-23T23:06:57.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.499+0000 7f064a800640 1 -- 172.21.15.12:0/3812450887 learned_addr learned my addr 172.21.15.12:0/3812450887 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:57.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.500+0000 7f0641e00640 1 -- 172.21.15.12:0/3812450887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f064c1038c0 msgr2=0x7f064c199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:57.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.500+0000 7f0641e00640 1 --2- 172.21.15.12:0/3812450887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f064c1038c0 0x7f064c199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.500+0000 7f0641e00640 1 -- 172.21.15.12:0/3812450887 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f064c105470 msgr2=0x7f064c1a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:57.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.500+0000 7f0641e00640 1 --2- 172.21.15.12:0/3812450887 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f064c105470 0x7f064c1a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.500+0000 7f0641e00640 1 -- 172.21.15.12:0/3812450887 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f064c1a1830 con 0x7f064c104ab0 2024-03-23T23:06:57.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.500+0000 7f0641e00640 1 --2- 172.21.15.12:0/3812450887 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f064c104ab0 0x7f064c19a160 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f063400e9f0 tx=0x7f063400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:57.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.500+0000 7f0643e00640 1 -- 172.21.15.12:0/3812450887 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063400ccc0 con 0x7f064c104ab0 2024-03-23T23:06:57.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.500+0000 7f0643e00640 1 -- 172.21.15.12:0/3812450887 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0634005020 con 0x7f064c104ab0 2024-03-23T23:06:57.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.500+0000 7f0651400640 1 -- 172.21.15.12:0/3812450887 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f064c1a1b20 con 0x7f064c104ab0 2024-03-23T23:06:57.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.501+0000 7f0643e00640 1 -- 172.21.15.12:0/3812450887 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0634004a00 con 0x7f064c104ab0 2024-03-23T23:06:57.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.501+0000 7f0651400640 1 -- 172.21.15.12:0/3812450887 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f064c1a2020 con 0x7f064c104ab0 2024-03-23T23:06:57.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.504+0000 7f0643e00640 1 -- 172.21.15.12:0/3812450887 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f06340051d0 con 0x7f064c104ab0 2024-03-23T23:06:57.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.504+0000 7f065418c740 1 Processor -- start 2024-03-23T23:06:57.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.504+0000 7f065418c740 1 -- start start 2024-03-23T23:06:57.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.504+0000 7f065418c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0a84010a0 0x55d0a84014c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.504+0000 7f065418c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0a8401a90 0x55d0a8406590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.504+0000 7f065418c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0a8406c00 0x55d0a8409090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.504+0000 7f065418c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0a8366eb0 con 0x55d0a8406c00 2024-03-23T23:06:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.504+0000 7f065418c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0a830fa60 con 0x55d0a8401a90 2024-03-23T23:06:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.504+0000 7f065418c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0a8362f50 con 0x55d0a84010a0 2024-03-23T23:06:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.504+0000 7f064b200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0a84010a0 0x55d0a84014c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.505+0000 7f064b200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0a84010a0 0x55d0a84014c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47850/0 (socket says 172.21.15.12:47850) 2024-03-23T23:06:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.505+0000 7f064a800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0a8401a90 0x55d0a8406590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.505+0000 7f064b200640 1 -- 172.21.15.12:0/3606195082 learned_addr learned my addr 172.21.15.12:0/3606195082 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.505+0000 7f0641e00640 1 --2- 172.21.15.12:0/3606195082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0a8406c00 0x55d0a8409090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.505+0000 7f064b200640 1 -- 172.21.15.12:0/3606195082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0a8401a90 msgr2=0x55d0a8406590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.505+0000 7f064b200640 1 --2- 172.21.15.12:0/3606195082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0a8401a90 0x55d0a8406590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.505+0000 7f064b200640 1 -- 172.21.15.12:0/3606195082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0a8406c00 msgr2=0x55d0a8409090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:57.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.505+0000 7f064b200640 1 --2- 172.21.15.12:0/3606195082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0a8406c00 0x55d0a8409090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.505+0000 7f064b200640 1 -- 172.21.15.12:0/3606195082 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0a81437e0 con 0x55d0a84010a0 2024-03-23T23:06:57.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.505+0000 7f0643e00640 1 --2- 172.21.15.12:0/3812450887 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f06300a2800 0x7f06300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.505+0000 7f064b200640 1 --2- 172.21.15.12:0/3606195082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0a84010a0 0x55d0a84014c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0644011440 tx=0x7f0644011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8f01560505a60e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:57.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.505+0000 7f0643e00640 1 -- 172.21.15.12:0/3812450887 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f06340cea50 con 0x7f064c104ab0 2024-03-23T23:06:57.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.505+0000 7f064a800640 1 --2- 172.21.15.12:0/3812450887 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f06300a2800 0x7f06300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.506+0000 7f063b400640 1 -- 172.21.15.12:0/3606195082 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0644003c30 con 0x55d0a84010a0 2024-03-23T23:06:57.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.506+0000 7f063b400640 1 -- 172.21.15.12:0/3606195082 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0644003dd0 con 0x55d0a84010a0 2024-03-23T23:06:57.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.506+0000 7f063b400640 1 -- 172.21.15.12:0/3606195082 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064400fb50 con 0x55d0a84010a0 2024-03-23T23:06:57.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.506+0000 7f065418c740 1 -- 172.21.15.12:0/3606195082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0a84010a0 msgr2=0x55d0a84014c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:57.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.506+0000 7f065418c740 1 --2- 172.21.15.12:0/3606195082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0a84010a0 0x55d0a84014c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0644011440 tx=0x7f0644011900 comp rx=0 tx=0).stop 2024-03-23T23:06:57.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.506+0000 7f064a800640 1 --2- 172.21.15.12:0/3812450887 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f06300a2800 0x7f06300a4cc0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f063c006010 tx=0x7f063c02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:57.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.506+0000 7f065418c740 1 -- 172.21.15.12:0/3606195082 shutdown_connections 2024-03-23T23:06:57.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.506+0000 7f065418c740 1 --2- 172.21.15.12:0/3606195082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0a8406c00 0x55d0a8409090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.506+0000 7f065418c740 1 --2- 172.21.15.12:0/3606195082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0a8401a90 0x55d0a8406590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.506+0000 7f065418c740 1 --2- 172.21.15.12:0/3606195082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0a84010a0 0x55d0a84014c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.506+0000 7f065418c740 1 -- 172.21.15.12:0/3606195082 >> 172.21.15.12:0/3606195082 conn(0x55d0a83fc610 msgr2=0x55d0a83fca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:57.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.506+0000 7f065418c740 1 -- 172.21.15.12:0/3606195082 shutdown_connections 2024-03-23T23:06:57.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.507+0000 7f065418c740 1 -- 172.21.15.12:0/3606195082 wait complete. 2024-03-23T23:06:57.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.507+0000 7f065418c740 1 -- start start 2024-03-23T23:06:57.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.507+0000 7f065418c740 1 Processor -- start 2024-03-23T23:06:57.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.508+0000 7f065418c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0a84010a0 0x55d0a840b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.508+0000 7f065418c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0a8401a90 0x55d0a840b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.508+0000 7f065418c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0a8406c00 0x55d0a8412830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.508+0000 7f065418c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0a8409bb0 con 0x55d0a84010a0 2024-03-23T23:06:57.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.508+0000 7f065418c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0a8322eb0 con 0x55d0a8401a90 2024-03-23T23:06:57.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.508+0000 7f065418c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0a8409d30 con 0x55d0a8406c00 2024-03-23T23:06:57.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.508+0000 7f065418c740 10 client.0 _ll_register_callbacks cb 0x55d0a8317150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:57.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.508+0000 7f0641e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0a8401a90 0x55d0a840b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.508+0000 7f064a800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0a84010a0 0x55d0a840b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.508+0000 7f0641e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0a8401a90 0x55d0a840b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:53654/0 (socket says 172.21.15.12:53654) 2024-03-23T23:06:57.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.508+0000 7f0641e00640 1 -- 172.21.15.12:0/1595728785 learned_addr learned my addr 172.21.15.12:0/1595728785 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:57.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.508+0000 7f064b200640 1 --2- 172.21.15.12:0/1595728785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0a8406c00 0x55d0a8412830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.508+0000 7f0641e00640 1 -- 172.21.15.12:0/1595728785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0a8406c00 msgr2=0x55d0a8412830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:57.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.508+0000 7f0641e00640 1 --2- 172.21.15.12:0/1595728785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0a8406c00 0x55d0a8412830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.508+0000 7f0641e00640 1 -- 172.21.15.12:0/1595728785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0a84010a0 msgr2=0x55d0a840b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:57.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.508+0000 7f0641e00640 1 --2- 172.21.15.12:0/1595728785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0a84010a0 0x55d0a840b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.508+0000 7f0641e00640 1 -- 172.21.15.12:0/1595728785 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d0a81a3f70 con 0x55d0a8401a90 2024-03-23T23:06:57.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.509+0000 7f0641e00640 1 --2- 172.21.15.12:0/1595728785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0a8401a90 0x55d0a840b7f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f063409b330 tx=0x7f063409adc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fb0408a39fba4d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:57.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.509+0000 7f062fe00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:06:57.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.509+0000 7f062fe00640 1 -- 172.21.15.12:0/1595728785 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063401c040 con 0x55d0a8401a90 2024-03-23T23:06:57.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.509+0000 7f062fe00640 1 -- 172.21.15.12:0/1595728785 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f06340a38b0 con 0x55d0a8401a90 2024-03-23T23:06:57.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.509+0000 7f065418c740 1 -- 172.21.15.12:0/1595728785 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0644003dd0 con 0x55d0a8401a90 2024-03-23T23:06:57.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.509+0000 7f062fe00640 1 -- 172.21.15.12:0/1595728785 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f063409b3a0 con 0x55d0a8401a90 2024-03-23T23:06:57.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.509+0000 7f065418c740 10 client.26693 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:57.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.509+0000 7f065418c740 1 -- 172.21.15.12:0/1595728785 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55d0a803b170 con 0x55d0a8401a90 2024-03-23T23:06:57.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.509+0000 7f062fe00640 1 -- 172.21.15.12:0/1595728785 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0644003dd0 con 0x55d0a8401a90 2024-03-23T23:06:57.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.509+0000 7f062fe00640 1 -- 172.21.15.12:0/1595728785 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f0634004ba0 con 0x55d0a8401a90 2024-03-23T23:06:57.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.510+0000 7f065418c740 10 client.26693 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:57.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.510+0000 7f065418c740 10 client.26693 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:06:57.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.510+0000 7f065418c740 20 client.26693 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:57.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.510+0000 7f065418c740 1 --2- 172.21.15.12:0/1595728785 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d0a8417070 0x55d0a84194d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.510+0000 7f065418c740 4 client.26693 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:06:57.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.510+0000 7f065418c740 1 -- 172.21.15.12:0/1595728785 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d0a83743f0 con 0x55d0a8417070 2024-03-23T23:06:57.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.510+0000 7f064a800640 1 --2- 172.21.15.12:0/1595728785 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d0a8417070 0x55d0a84194d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.510+0000 7f064a800640 1 --2- 172.21.15.12:0/1595728785 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d0a8417070 0x55d0a84194d0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19d992974ee84276 server_cookie=d41ff8c82d8fabe3 in_seq=0 out_seq=0 2024-03-23T23:06:57.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.510+0000 7f062fe00640 10 client.26693 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:57.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.511+0000 7f062fe00640 1 -- 172.21.15.12:0/1595728785 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d0a83743f0 con 0x55d0a8417070 2024-03-23T23:06:57.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.511+0000 7f062fe00640 10 client.26693 handle_command_reply: tid=0 2024-03-23T23:06:57.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.511+0000 7f065418c740 1 client.26693 shutdown 2024-03-23T23:06:57.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.511+0000 7f065418c740 1 -- 172.21.15.12:0/1595728785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0a8401a90 msgr2=0x55d0a840b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:57.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.511+0000 7f065418c740 1 --2- 172.21.15.12:0/1595728785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0a8401a90 0x55d0a840b7f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f063409b330 tx=0x7f063409adc0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.511+0000 7f065418c740 1 -- 172.21.15.12:0/1595728785 shutdown_connections 2024-03-23T23:06:57.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.511+0000 7f065418c740 1 --2- 172.21.15.12:0/1595728785 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d0a8417070 0x55d0a84194d0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.511+0000 7f065418c740 1 --2- 172.21.15.12:0/1595728785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0a8406c00 0x55d0a8412830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.511+0000 7f065418c740 1 --2- 172.21.15.12:0/1595728785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0a8401a90 0x55d0a840b7f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.511+0000 7f065418c740 1 --2- 172.21.15.12:0/1595728785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0a84010a0 0x55d0a840b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.511+0000 7f065418c740 1 -- 172.21.15.12:0/1595728785 >> 172.21.15.12:0/1595728785 conn(0x55d0a83fc610 msgr2=0x55d0a83fb830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:57.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.511+0000 7f062fe00640 0 client.26693 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:57.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.512+0000 7f065418c740 1 -- 172.21.15.12:0/1595728785 shutdown_connections 2024-03-23T23:06:57.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.512+0000 7f065418c740 1 -- 172.21.15.12:0/1595728785 wait complete. 2024-03-23T23:06:57.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.512+0000 7f065418c740 20 client.26693 trim_cache size 0 max 16384 2024-03-23T23:06:57.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.530+0000 7f065418c740 1 Processor -- start 2024-03-23T23:06:57.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.530+0000 7f065418c740 1 -- start start 2024-03-23T23:06:57.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.530+0000 7f065418c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0a84010a0 0x55d0a83ffee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.530+0000 7f065418c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0a8401a90 0x55d0a84127f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.530+0000 7f065418c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0a8406c00 0x55d0a841dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.530+0000 7f065418c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064c1ad120 con 0x55d0a84010a0 2024-03-23T23:06:57.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.530+0000 7f065418c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064c1aac20 con 0x55d0a8406c00 2024-03-23T23:06:57.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.530+0000 7f065418c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064c1a8cc0 con 0x55d0a8401a90 2024-03-23T23:06:57.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.530+0000 7f064b200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0a8406c00 0x55d0a841dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.530+0000 7f0641e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0a8401a90 0x55d0a84127f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.530+0000 7f064a800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0a84010a0 0x55d0a83ffee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.530+0000 7f064b200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0a8406c00 0x55d0a841dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:53670/0 (socket says 172.21.15.12:53670) 2024-03-23T23:06:57.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.530+0000 7f0641e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0a8401a90 0x55d0a84127f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47866/0 (socket says 172.21.15.12:47866) 2024-03-23T23:06:57.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.530+0000 7f064b200640 1 -- 172.21.15.12:0/2988726170 learned_addr learned my addr 172.21.15.12:0/2988726170 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:57.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.531+0000 7f064b200640 1 -- 172.21.15.12:0/2988726170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0a8401a90 msgr2=0x55d0a84127f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.531+0000 7f064b200640 1 --2- 172.21.15.12:0/2988726170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0a8401a90 0x55d0a84127f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.531+0000 7f064b200640 1 -- 172.21.15.12:0/2988726170 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0a84010a0 msgr2=0x55d0a83ffee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.531+0000 7f064b200640 1 --2- 172.21.15.12:0/2988726170 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0a84010a0 0x55d0a83ffee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.531+0000 7f064b200640 1 -- 172.21.15.12:0/2988726170 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0a8401e70 con 0x55d0a8406c00 2024-03-23T23:06:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.531+0000 7f064b200640 1 --2- 172.21.15.12:0/2988726170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0a8406c00 0x55d0a841dd90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x55d0a84022f0 tx=0x7f0644005ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d77d1ec350ec5c8a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.531+0000 7f063aa00640 1 -- 172.21.15.12:0/2988726170 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0644003bf0 con 0x55d0a8406c00 2024-03-23T23:06:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.532+0000 7f063aa00640 1 -- 172.21.15.12:0/2988726170 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0644003d90 con 0x55d0a8406c00 2024-03-23T23:06:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.532+0000 7f063aa00640 1 -- 172.21.15.12:0/2988726170 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0644005580 con 0x55d0a8406c00 2024-03-23T23:06:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.532+0000 7f065418c740 1 -- 172.21.15.12:0/2988726170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0a8406c00 msgr2=0x55d0a841dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.532+0000 7f065418c740 1 --2- 172.21.15.12:0/2988726170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0a8406c00 0x55d0a841dd90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x55d0a84022f0 tx=0x7f0644005ee0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.532+0000 7f065418c740 1 -- 172.21.15.12:0/2988726170 shutdown_connections 2024-03-23T23:06:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.532+0000 7f065418c740 1 --2- 172.21.15.12:0/2988726170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0a8406c00 0x55d0a841dd90 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.532+0000 7f065418c740 1 --2- 172.21.15.12:0/2988726170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0a8401a90 0x55d0a84127f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.532+0000 7f065418c740 1 --2- 172.21.15.12:0/2988726170 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0a84010a0 0x55d0a83ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.532+0000 7f065418c740 1 -- 172.21.15.12:0/2988726170 >> 172.21.15.12:0/2988726170 conn(0x55d0a83fc610 msgr2=0x55d0a83fb830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:57.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.532+0000 7f065418c740 1 -- 172.21.15.12:0/2988726170 shutdown_connections 2024-03-23T23:06:57.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.532+0000 7f065418c740 1 -- 172.21.15.12:0/2988726170 wait complete. 2024-03-23T23:06:57.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.533+0000 7f065418c740 1 -- start start 2024-03-23T23:06:57.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.533+0000 7f065418c740 1 Processor -- start 2024-03-23T23:06:57.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.533+0000 7f065418c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0a84010a0 0x55d0a8410ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.533+0000 7f065418c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0a8401a90 0x55d0a8411430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.534+0000 7f065418c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0a8406c00 0x55d0a8411970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.534+0000 7f065418c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064c1a92c0 con 0x55d0a8401a90 2024-03-23T23:06:57.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.534+0000 7f065418c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064c1aa5c0 con 0x55d0a84010a0 2024-03-23T23:06:57.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.534+0000 7f065418c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064c1a9900 con 0x55d0a8406c00 2024-03-23T23:06:57.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.534+0000 7f065418c740 10 client.0 _ll_register_callbacks cb 0x7f0634009f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:57.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.534+0000 7f0641e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0a8401a90 0x55d0a8411430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.534+0000 7f064a800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0a84010a0 0x55d0a8410ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.534+0000 7f0641e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0a8401a90 0x55d0a8411430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37808/0 (socket says 172.21.15.12:37808) 2024-03-23T23:06:57.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.534+0000 7f0641e00640 1 -- 172.21.15.12:0/3531925246 learned_addr learned my addr 172.21.15.12:0/3531925246 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:57.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.534+0000 7f064b200640 1 --2- 172.21.15.12:0/3531925246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0a8406c00 0x55d0a8411970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.534+0000 7f064a800640 1 -- 172.21.15.12:0/3531925246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0a8406c00 msgr2=0x55d0a8411970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:57.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.534+0000 7f064a800640 1 --2- 172.21.15.12:0/3531925246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0a8406c00 0x55d0a8411970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.534+0000 7f064a800640 1 -- 172.21.15.12:0/3531925246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0a8401a90 msgr2=0x55d0a8411430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:57.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.534+0000 7f064a800640 1 --2- 172.21.15.12:0/3531925246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0a8401a90 0x55d0a8411430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.534+0000 7f064a800640 1 -- 172.21.15.12:0/3531925246 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d0a847b170 con 0x55d0a84010a0 2024-03-23T23:06:57.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.534+0000 7f0641e00640 1 --2- 172.21.15.12:0/3531925246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0a8401a90 0x55d0a8411430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.535+0000 7f064a800640 1 --2- 172.21.15.12:0/3531925246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0a84010a0 0x55d0a8410ef0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f064c19b0f0 tx=0x7f063c02bc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7db290da35bf77d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.535+0000 7f062fe00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:06:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.535+0000 7f062fe00640 1 -- 172.21.15.12:0/3531925246 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063c062060 con 0x55d0a84010a0 2024-03-23T23:06:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.535+0000 7f062fe00640 1 -- 172.21.15.12:0/3531925246 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f063c0048c0 con 0x55d0a84010a0 2024-03-23T23:06:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.535+0000 7f065418c740 1 -- 172.21.15.12:0/3531925246 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0644003d90 con 0x55d0a84010a0 2024-03-23T23:06:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.535+0000 7f062fe00640 1 -- 172.21.15.12:0/3531925246 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f063c004ae0 con 0x55d0a84010a0 2024-03-23T23:06:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.535+0000 7f065418c740 10 client.26699 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.535+0000 7f065418c740 1 -- 172.21.15.12:0/3531925246 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55d0a847c450 con 0x55d0a84010a0 2024-03-23T23:06:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.535+0000 7f062fe00640 1 -- 172.21.15.12:0/3531925246 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0644003d90 con 0x55d0a84010a0 2024-03-23T23:06:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.536+0000 7f062fe00640 1 -- 172.21.15.12:0/3531925246 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f063c05c030 con 0x55d0a84010a0 2024-03-23T23:06:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.536+0000 7f065418c740 10 client.26699 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.536+0000 7f065418c740 10 client.26699 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:06:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.536+0000 7f065418c740 20 client.26699 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.536+0000 7f065418c740 1 --2- 172.21.15.12:0/3531925246 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d0a847cd30 0x55d0a847f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.536+0000 7f065418c740 4 client.26699 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-23T23:06:57.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.536+0000 7f065418c740 1 -- 172.21.15.12:0/3531925246 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55d0a836e740 con 0x55d0a847cd30 2024-03-23T23:06:57.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.536+0000 7f0641e00640 1 --2- 172.21.15.12:0/3531925246 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d0a847cd30 0x55d0a847f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.536+0000 7f062fe00640 10 client.26699 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:57.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.536+0000 7f0641e00640 1 --2- 172.21.15.12:0/3531925246 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d0a847cd30 0x55d0a847f150 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8fbe2e4cdead283f server_cookie=65170084fdebe562 in_seq=0 out_seq=0 2024-03-23T23:06:57.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.538+0000 7f062fe00640 1 -- 172.21.15.12:0/3531925246 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55d0a836e740 con 0x55d0a847cd30 2024-03-23T23:06:57.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.538+0000 7f062fe00640 10 client.26699 handle_command_reply: tid=0 2024-03-23T23:06:57.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.538+0000 7f065418c740 1 client.26699 shutdown 2024-03-23T23:06:57.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.538+0000 7f065418c740 1 -- 172.21.15.12:0/3531925246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0a84010a0 msgr2=0x55d0a8410ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:57.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.538+0000 7f065418c740 1 --2- 172.21.15.12:0/3531925246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0a84010a0 0x55d0a8410ef0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f064c19b0f0 tx=0x7f063c02bc40 comp rx=0 tx=0).stop 2024-03-23T23:06:57.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.538+0000 7f065418c740 1 -- 172.21.15.12:0/3531925246 shutdown_connections 2024-03-23T23:06:57.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.538+0000 7f065418c740 1 --2- 172.21.15.12:0/3531925246 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d0a847cd30 0x55d0a847f150 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.538+0000 7f065418c740 1 --2- 172.21.15.12:0/3531925246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0a8406c00 0x55d0a8411970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.538+0000 7f065418c740 1 --2- 172.21.15.12:0/3531925246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0a8401a90 0x55d0a8411430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.538+0000 7f065418c740 1 --2- 172.21.15.12:0/3531925246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0a84010a0 0x55d0a8410ef0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.538+0000 7f062fe00640 0 client.26699 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:57.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.538+0000 7f065418c740 1 -- 172.21.15.12:0/3531925246 >> 172.21.15.12:0/3531925246 conn(0x55d0a83fc610 msgr2=0x55d0a8402490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:57.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.538+0000 7f065418c740 1 -- 172.21.15.12:0/3531925246 shutdown_connections 2024-03-23T23:06:57.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.538+0000 7f065418c740 1 -- 172.21.15.12:0/3531925246 wait complete. 2024-03-23T23:06:57.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.538+0000 7f065418c740 20 client.26699 trim_cache size 0 max 16384 2024-03-23T23:06:57.541 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:06:57.541 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-03-23T23:06:57.541 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "a1cd91b6-eb38-42ce-9921-7a41b1bb9946", 2024-03-23T23:06:57.541 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-03-23T23:06:57.542 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:06:57.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.539+0000 7f0651400640 1 -- 172.21.15.12:0/3812450887 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f06300a2800 msgr2=0x7f06300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:57.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.539+0000 7f0651400640 1 --2- 172.21.15.12:0/3812450887 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f06300a2800 0x7f06300a4cc0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f063c006010 tx=0x7f063c02e040 comp rx=0 tx=0).stop 2024-03-23T23:06:57.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.539+0000 7f0651400640 1 -- 172.21.15.12:0/3812450887 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f064c104ab0 msgr2=0x7f064c19a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:57.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.539+0000 7f0651400640 1 --2- 172.21.15.12:0/3812450887 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f064c104ab0 0x7f064c19a160 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f063400e9f0 tx=0x7f063400edf0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.539+0000 7f0651400640 1 -- 172.21.15.12:0/3812450887 shutdown_connections 2024-03-23T23:06:57.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.539+0000 7f0651400640 1 --2- 172.21.15.12:0/3812450887 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f06300a2800 0x7f06300a4cc0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.539+0000 7f0651400640 1 --2- 172.21.15.12:0/3812450887 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f064c105470 0x7f064c1a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.539+0000 7f0651400640 1 --2- 172.21.15.12:0/3812450887 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f064c104ab0 0x7f064c19a160 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.539+0000 7f0651400640 1 --2- 172.21.15.12:0/3812450887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f064c1038c0 0x7f064c199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.539+0000 7f0651400640 1 -- 172.21.15.12:0/3812450887 >> 172.21.15.12:0/3812450887 conn(0x7f064c0fee50 msgr2=0x7f064c107e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:57.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.539+0000 7f0651400640 1 -- 172.21.15.12:0/3812450887 shutdown_connections 2024-03-23T23:06:57.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.539+0000 7f0651400640 1 -- 172.21.15.12:0/3812450887 wait complete. 2024-03-23T23:06:57.559 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:06:57.559 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:06:57.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.790+0000 7fe52f400640 1 -- 172.21.15.12:0/433302645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5280772a0 msgr2=0x7fe528077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:57.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.790+0000 7fe52f400640 1 --2- 172.21.15.12:0/433302645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5280772a0 0x7fe528077700 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe514009900 tx=0x7fe51402bc40 comp rx=0 tx=0).stop 2024-03-23T23:06:57.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.790+0000 7fe52f400640 1 -- 172.21.15.12:0/433302645 shutdown_connections 2024-03-23T23:06:57.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.790+0000 7fe52f400640 1 --2- 172.21.15.12:0/433302645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe528077c40 0x7fe5280784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.790+0000 7fe52f400640 1 --2- 172.21.15.12:0/433302645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5280772a0 0x7fe528077700 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.790+0000 7fe52f400640 1 --2- 172.21.15.12:0/433302645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe528076060 0x7fe528076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.790+0000 7fe52f400640 1 -- 172.21.15.12:0/433302645 >> 172.21.15.12:0/433302645 conn(0x7fe5280fee50 msgr2=0x7fe528101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:57.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.790+0000 7fe52f400640 1 -- 172.21.15.12:0/433302645 shutdown_connections 2024-03-23T23:06:57.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.790+0000 7fe52f400640 1 -- 172.21.15.12:0/433302645 wait complete. 2024-03-23T23:06:57.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.791+0000 7fe52f400640 1 Processor -- start 2024-03-23T23:06:57.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.791+0000 7fe52f400640 1 -- start start 2024-03-23T23:06:57.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.791+0000 7fe52f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe528076060 0x7fe52819e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.791+0000 7fe52f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5280772a0 0x7fe52819e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.792+0000 7fe52f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe528077c40 0x7fe5281a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.792+0000 7fe52f400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe528113970 con 0x7fe528076060 2024-03-23T23:06:57.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.792+0000 7fe52f400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5281137f0 con 0x7fe5280772a0 2024-03-23T23:06:57.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.792+0000 7fe52f400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe528113af0 con 0x7fe528077c40 2024-03-23T23:06:57.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.792+0000 7fe52ca00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe528076060 0x7fe52819e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.792+0000 7fe527e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5280772a0 0x7fe52819e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.792+0000 7fe52ca00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe528076060 0x7fe52819e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37832/0 (socket says 172.21.15.12:37832) 2024-03-23T23:06:57.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.792+0000 7fe52ca00640 1 -- 172.21.15.12:0/3892020992 learned_addr learned my addr 172.21.15.12:0/3892020992 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:57.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.792+0000 7fe52d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe528077c40 0x7fe5281a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.792+0000 7fe527e00640 1 --2- 172.21.15.12:0/3892020992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5280772a0 0x7fe52819e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:53696/0 (socket says 172.21.15.12:53696) 2024-03-23T23:06:57.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.792+0000 7fe527e00640 1 -- 172.21.15.12:0/3892020992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe528077c40 msgr2=0x7fe5281a55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:57.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.792+0000 7fe527e00640 1 --2- 172.21.15.12:0/3892020992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe528077c40 0x7fe5281a55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.792+0000 7fe527e00640 1 -- 172.21.15.12:0/3892020992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe528076060 msgr2=0x7fe52819e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:57.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.792+0000 7fe527e00640 1 --2- 172.21.15.12:0/3892020992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe528076060 0x7fe52819e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.792+0000 7fe527e00640 1 -- 172.21.15.12:0/3892020992 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5281a5c30 con 0x7fe5280772a0 2024-03-23T23:06:57.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.793+0000 7fe527e00640 1 --2- 172.21.15.12:0/3892020992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5280772a0 0x7fe52819e550 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe514036040 tx=0x7fe514005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:57.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.793+0000 7fe525600640 1 -- 172.21.15.12:0/3892020992 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe514041070 con 0x7fe5280772a0 2024-03-23T23:06:57.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.793+0000 7fe525600640 1 -- 172.21.15.12:0/3892020992 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe5140044d0 con 0x7fe5280772a0 2024-03-23T23:06:57.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.793+0000 7fe525600640 1 -- 172.21.15.12:0/3892020992 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe514034440 con 0x7fe5280772a0 2024-03-23T23:06:57.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.793+0000 7fe52f400640 1 -- 172.21.15.12:0/3892020992 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5281a5ec0 con 0x7fe5280772a0 2024-03-23T23:06:57.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.793+0000 7fe52f400640 1 -- 172.21.15.12:0/3892020992 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5281a62d0 con 0x7fe5280772a0 2024-03-23T23:06:57.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.796+0000 7fe532002740 1 Processor -- start 2024-03-23T23:06:57.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.797+0000 7fe532002740 1 -- start start 2024-03-23T23:06:57.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.797+0000 7fe525600640 1 -- 172.21.15.12:0/3892020992 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe514044070 con 0x7fe5280772a0 2024-03-23T23:06:57.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.797+0000 7fe532002740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654166ace80 0x5654166ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.797+0000 7fe532002740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654166ad870 0x5654166b2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.797+0000 7fe532002740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166b29e0 0x5654166b4e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.797+0000 7fe532002740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x565416612250 con 0x5654166ace80 2024-03-23T23:06:57.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.797+0000 7fe532002740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654165baa30 con 0x5654166b29e0 2024-03-23T23:06:57.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.797+0000 7fe532002740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x56541660e2f0 con 0x5654166ad870 2024-03-23T23:06:57.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.797+0000 7fe527e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166b29e0 0x5654166b4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.797+0000 7fe52d400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654166ace80 0x5654166ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.797+0000 7fe527e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166b29e0 0x5654166b4e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:53700/0 (socket says 172.21.15.12:53700) 2024-03-23T23:06:57.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.797+0000 7fe527e00640 1 -- 172.21.15.12:0/2533269631 learned_addr learned my addr 172.21.15.12:0/2533269631 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:57.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.797+0000 7fe52ca00640 1 --2- 172.21.15.12:0/2533269631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654166ad870 0x5654166b2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.798+0000 7fe527e00640 1 -- 172.21.15.12:0/2533269631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654166ad870 msgr2=0x5654166b2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:57.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.798+0000 7fe527e00640 1 --2- 172.21.15.12:0/2533269631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654166ad870 0x5654166b2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.798+0000 7fe527e00640 1 -- 172.21.15.12:0/2533269631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654166ace80 msgr2=0x5654166ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:57.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.798+0000 7fe527e00640 1 --2- 172.21.15.12:0/2533269631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654166ace80 0x5654166ad2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.798+0000 7fe527e00640 1 -- 172.21.15.12:0/2533269631 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654165a5690 con 0x5654166b29e0 2024-03-23T23:06:57.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.798+0000 7fe52d400640 1 --2- 172.21.15.12:0/2533269631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654166ace80 0x5654166ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:06:57.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.798+0000 7fe527e00640 1 --2- 172.21.15.12:0/2533269631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166b29e0 0x5654166b4e70 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe5140f6460 tx=0x7fe5140f6550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc58e0b66cb22d3c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:57.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.798+0000 7fe51dc00640 1 -- 172.21.15.12:0/2533269631 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5140f0ae0 con 0x5654166b29e0 2024-03-23T23:06:57.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.798+0000 7fe51dc00640 1 -- 172.21.15.12:0/2533269631 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe5140fd030 con 0x5654166b29e0 2024-03-23T23:06:57.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.798+0000 7fe51dc00640 1 -- 172.21.15.12:0/2533269631 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe514102820 con 0x5654166b29e0 2024-03-23T23:06:57.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.798+0000 7fe525600640 1 --2- 172.21.15.12:0/3892020992 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe50c0a2880 0x7fe50c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.799+0000 7fe525600640 1 -- 172.21.15.12:0/3892020992 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe5140f04f0 con 0x7fe5280772a0 2024-03-23T23:06:57.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.799+0000 7fe52d400640 1 --2- 172.21.15.12:0/3892020992 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe50c0a2880 0x7fe50c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.799+0000 7fe532002740 1 -- 172.21.15.12:0/2533269631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166b29e0 msgr2=0x5654166b4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:57.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.799+0000 7fe532002740 1 --2- 172.21.15.12:0/2533269631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166b29e0 0x5654166b4e70 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe5140f6460 tx=0x7fe5140f6550 comp rx=0 tx=0).stop 2024-03-23T23:06:57.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.799+0000 7fe532002740 1 -- 172.21.15.12:0/2533269631 shutdown_connections 2024-03-23T23:06:57.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.799+0000 7fe532002740 1 --2- 172.21.15.12:0/2533269631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166b29e0 0x5654166b4e70 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.799+0000 7fe532002740 1 --2- 172.21.15.12:0/2533269631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654166ad870 0x5654166b2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.799+0000 7fe532002740 1 --2- 172.21.15.12:0/2533269631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654166ace80 0x5654166ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.799+0000 7fe532002740 1 -- 172.21.15.12:0/2533269631 >> 172.21.15.12:0/2533269631 conn(0x565416675200 msgr2=0x5654166a87f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:57.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.799+0000 7fe532002740 1 -- 172.21.15.12:0/2533269631 shutdown_connections 2024-03-23T23:06:57.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.799+0000 7fe532002740 1 -- 172.21.15.12:0/2533269631 wait complete. 2024-03-23T23:06:57.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.800+0000 7fe52d400640 1 --2- 172.21.15.12:0/3892020992 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe50c0a2880 0x7fe50c0a4d40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x5654166ae0d0 tx=0x7fe5200096a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:57.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.800+0000 7fe532002740 1 -- start start 2024-03-23T23:06:57.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.800+0000 7fe532002740 1 Processor -- start 2024-03-23T23:06:57.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.800+0000 7fe532002740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166ace80 0x5654166b6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.800+0000 7fe532002740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654166ad870 0x5654166b70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.800+0000 7fe532002740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654166b29e0 0x5654166be080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.800+0000 7fe532002740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654166b5750 con 0x5654166ad870 2024-03-23T23:06:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.800+0000 7fe532002740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654165cde80 con 0x5654166ace80 2024-03-23T23:06:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.800+0000 7fe532002740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654166b58d0 con 0x5654166b29e0 2024-03-23T23:06:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.800+0000 7fe532002740 10 client.0 _ll_register_callbacks cb 0x5654165e0130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.801+0000 7fe527e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654166ad870 0x5654166b70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.801+0000 7fe52ca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166ace80 0x5654166b6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.801+0000 7fe52d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654166b29e0 0x5654166be080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.801+0000 7fe52ca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166ace80 0x5654166b6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:53702/0 (socket says 172.21.15.12:53702) 2024-03-23T23:06:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.801+0000 7fe52d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654166b29e0 0x5654166be080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47898/0 (socket says 172.21.15.12:47898) 2024-03-23T23:06:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.801+0000 7fe52ca00640 1 -- 172.21.15.12:0/3016287029 learned_addr learned my addr 172.21.15.12:0/3016287029 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.801+0000 7fe52d400640 1 -- 172.21.15.12:0/3016287029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166ace80 msgr2=0x5654166b6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.801+0000 7fe52d400640 1 --2- 172.21.15.12:0/3016287029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166ace80 0x5654166b6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.801+0000 7fe52d400640 1 -- 172.21.15.12:0/3016287029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654166ad870 msgr2=0x5654166b70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.801+0000 7fe52d400640 1 --2- 172.21.15.12:0/3016287029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654166ad870 0x5654166b70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.801+0000 7fe52d400640 1 -- 172.21.15.12:0/3016287029 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5654163f07d0 con 0x5654166b29e0 2024-03-23T23:06:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.801+0000 7fe52d400640 1 --2- 172.21.15.12:0/3016287029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654166b29e0 0x5654166be080 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fe520009e70 tx=0x7fe520004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6837be5da68892ac server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.801+0000 7fe512000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:06:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.802+0000 7fe512000640 1 -- 172.21.15.12:0/3016287029 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe520010930 con 0x5654166b29e0 2024-03-23T23:06:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.802+0000 7fe512000640 1 -- 172.21.15.12:0/3016287029 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe520010ad0 con 0x5654166b29e0 2024-03-23T23:06:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.802+0000 7fe532002740 1 -- 172.21.15.12:0/3016287029 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe5140fd030 con 0x5654166b29e0 2024-03-23T23:06:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.802+0000 7fe512000640 1 -- 172.21.15.12:0/3016287029 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe520010cd0 con 0x5654166b29e0 2024-03-23T23:06:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.802+0000 7fe532002740 10 client.26683 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.802+0000 7fe532002740 1 -- 172.21.15.12:0/3016287029 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5654163263f0 con 0x5654166b29e0 2024-03-23T23:06:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.802+0000 7fe512000640 1 -- 172.21.15.12:0/3016287029 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe5140fd030 con 0x5654166b29e0 2024-03-23T23:06:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.802+0000 7fe512000640 1 -- 172.21.15.12:0/3016287029 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe520024070 con 0x5654166b29e0 2024-03-23T23:06:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.802+0000 7fe532002740 10 client.26683 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.802+0000 7fe532002740 10 client.26683 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:06:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.802+0000 7fe532002740 20 client.26683 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.803+0000 7fe532002740 1 --2- 172.21.15.12:0/3016287029 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5654166c2980 0x5654166c4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.803+0000 7fe532002740 4 client.26683 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:06:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.803+0000 7fe532002740 1 -- 172.21.15.12:0/3016287029 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56541661f790 con 0x5654166c2980 2024-03-23T23:06:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.803+0000 7fe52ca00640 1 --2- 172.21.15.12:0/3016287029 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5654166c2980 0x5654166c4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.803+0000 7fe52ca00640 1 --2- 172.21.15.12:0/3016287029 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5654166c2980 0x5654166c4de0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3525ed8bc5860849 server_cookie=62194ca3cd897ea1 in_seq=0 out_seq=0 2024-03-23T23:06:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.803+0000 7fe512000640 10 client.26683 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.804+0000 7fe512000640 1 -- 172.21.15.12:0/3016287029 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56541661f790 con 0x5654166c2980 2024-03-23T23:06:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.804+0000 7fe512000640 10 client.26683 handle_command_reply: tid=0 2024-03-23T23:06:57.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.805+0000 7fe532002740 1 client.26683 shutdown 2024-03-23T23:06:57.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.805+0000 7fe532002740 1 -- 172.21.15.12:0/3016287029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654166b29e0 msgr2=0x5654166be080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:57.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.805+0000 7fe532002740 1 --2- 172.21.15.12:0/3016287029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654166b29e0 0x5654166be080 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fe520009e70 tx=0x7fe520004a40 comp rx=0 tx=0).stop 2024-03-23T23:06:57.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.805+0000 7fe532002740 1 -- 172.21.15.12:0/3016287029 shutdown_connections 2024-03-23T23:06:57.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.805+0000 7fe532002740 1 --2- 172.21.15.12:0/3016287029 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5654166c2980 0x5654166c4de0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.805+0000 7fe532002740 1 --2- 172.21.15.12:0/3016287029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654166b29e0 0x5654166be080 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.805+0000 7fe532002740 1 --2- 172.21.15.12:0/3016287029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654166ad870 0x5654166b70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.805+0000 7fe532002740 1 --2- 172.21.15.12:0/3016287029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166ace80 0x5654166b6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.805+0000 7fe532002740 1 -- 172.21.15.12:0/3016287029 >> 172.21.15.12:0/3016287029 conn(0x565416675200 msgr2=0x5654166a7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:57.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.805+0000 7fe512000640 0 client.26683 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:57.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.805+0000 7fe532002740 1 -- 172.21.15.12:0/3016287029 shutdown_connections 2024-03-23T23:06:57.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.805+0000 7fe532002740 1 -- 172.21.15.12:0/3016287029 wait complete. 2024-03-23T23:06:57.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.805+0000 7fe532002740 20 client.26683 trim_cache size 0 max 16384 2024-03-23T23:06:57.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.823+0000 7fe532002740 1 Processor -- start 2024-03-23T23:06:57.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.823+0000 7fe532002740 1 -- start start 2024-03-23T23:06:57.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.823+0000 7fe532002740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166ace80 0x5654166aba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.823+0000 7fe532002740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654166ad870 0x5654166be040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.823+0000 7fe532002740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654166b29e0 0x5654166abf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.823+0000 7fe532002740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5281b15a0 con 0x5654166b29e0 2024-03-23T23:06:57.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.823+0000 7fe532002740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5281af0a0 con 0x5654166ace80 2024-03-23T23:06:57.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.823+0000 7fe532002740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5281ad140 con 0x5654166ad870 2024-03-23T23:06:57.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.823+0000 7fe52ca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166ace80 0x5654166aba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.823+0000 7fe527e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654166ad870 0x5654166be040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.823+0000 7fe52d400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654166b29e0 0x5654166abf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.823+0000 7fe527e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654166ad870 0x5654166be040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47904/0 (socket says 172.21.15.12:47904) 2024-03-23T23:06:57.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.823+0000 7fe527e00640 1 -- 172.21.15.12:0/4013719715 learned_addr learned my addr 172.21.15.12:0/4013719715 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:57.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.824+0000 7fe527e00640 1 -- 172.21.15.12:0/4013719715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166ace80 msgr2=0x5654166aba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:57.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.824+0000 7fe527e00640 1 --2- 172.21.15.12:0/4013719715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166ace80 0x5654166aba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.824+0000 7fe527e00640 1 -- 172.21.15.12:0/4013719715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654166b29e0 msgr2=0x5654166abf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:57.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.824+0000 7fe527e00640 1 --2- 172.21.15.12:0/4013719715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654166b29e0 0x5654166abf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.824+0000 7fe527e00640 1 -- 172.21.15.12:0/4013719715 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654162e60f0 con 0x5654166ad870 2024-03-23T23:06:57.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.824+0000 7fe52ca00640 1 --2- 172.21.15.12:0/4013719715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166ace80 0x5654166aba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:06:57.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.824+0000 7fe527e00640 1 --2- 172.21.15.12:0/4013719715 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654166ad870 0x5654166be040 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe514006010 tx=0x7fe5140f0d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82647a169b871f4c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:57.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.824+0000 7fe51d200640 1 -- 172.21.15.12:0/4013719715 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51404b040 con 0x5654166ad870 2024-03-23T23:06:57.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.824+0000 7fe51d200640 1 -- 172.21.15.12:0/4013719715 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe5140fd070 con 0x5654166ad870 2024-03-23T23:06:57.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.824+0000 7fe51d200640 1 -- 172.21.15.12:0/4013719715 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5140bc850 con 0x5654166ad870 2024-03-23T23:06:57.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.825+0000 7fe532002740 1 -- 172.21.15.12:0/4013719715 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654166ad870 msgr2=0x5654166be040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:57.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.825+0000 7fe532002740 1 --2- 172.21.15.12:0/4013719715 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654166ad870 0x5654166be040 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe514006010 tx=0x7fe5140f0d30 comp rx=0 tx=0).stop 2024-03-23T23:06:57.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.825+0000 7fe532002740 1 -- 172.21.15.12:0/4013719715 shutdown_connections 2024-03-23T23:06:57.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.825+0000 7fe532002740 1 --2- 172.21.15.12:0/4013719715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654166b29e0 0x5654166abf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.825+0000 7fe532002740 1 --2- 172.21.15.12:0/4013719715 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654166ad870 0x5654166be040 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.825+0000 7fe532002740 1 --2- 172.21.15.12:0/4013719715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166ace80 0x5654166aba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.825+0000 7fe532002740 1 -- 172.21.15.12:0/4013719715 >> 172.21.15.12:0/4013719715 conn(0x565416675200 msgr2=0x5654166b6b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:57.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.825+0000 7fe532002740 1 -- 172.21.15.12:0/4013719715 shutdown_connections 2024-03-23T23:06:57.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.825+0000 7fe532002740 1 -- 172.21.15.12:0/4013719715 wait complete. 2024-03-23T23:06:57.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.825+0000 7fe532002740 1 -- start start 2024-03-23T23:06:57.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.826+0000 7fe532002740 1 Processor -- start 2024-03-23T23:06:57.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.826+0000 7fe532002740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166ace80 0x5654166bb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.826+0000 7fe532002740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654166b29e0 0x5654166bbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.826+0000 7fe532002740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654166bcac0 0x5654166bcf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.826+0000 7fe532002740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5281ad740 con 0x5654166b29e0 2024-03-23T23:06:57.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.826+0000 7fe532002740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5281aea40 con 0x5654166ace80 2024-03-23T23:06:57.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.826+0000 7fe532002740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5281add80 con 0x5654166bcac0 2024-03-23T23:06:57.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.826+0000 7fe532002740 10 client.0 _ll_register_callbacks cb 0x7fe520003f90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:06:57.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.826+0000 7fe52ca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166ace80 0x5654166bb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.826+0000 7fe52d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654166bcac0 0x5654166bcf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.826+0000 7fe52ca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166ace80 0x5654166bb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:53732/0 (socket says 172.21.15.12:53732) 2024-03-23T23:06:57.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.826+0000 7fe52ca00640 1 -- 172.21.15.12:0/2499444811 learned_addr learned my addr 172.21.15.12:0/2499444811 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:06:57.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.826+0000 7fe527e00640 1 --2- 172.21.15.12:0/2499444811 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654166b29e0 0x5654166bbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.827+0000 7fe52ca00640 1 -- 172.21.15.12:0/2499444811 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654166bcac0 msgr2=0x5654166bcf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:57.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.827+0000 7fe52ca00640 1 --2- 172.21.15.12:0/2499444811 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654166bcac0 0x5654166bcf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.827+0000 7fe52ca00640 1 -- 172.21.15.12:0/2499444811 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654166b29e0 msgr2=0x5654166bbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:06:57.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.827+0000 7fe52ca00640 1 --2- 172.21.15.12:0/2499444811 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654166b29e0 0x5654166bbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.827+0000 7fe52ca00640 1 -- 172.21.15.12:0/2499444811 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5654166bd480 con 0x5654166ace80 2024-03-23T23:06:57.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.827+0000 7fe527e00640 1 --2- 172.21.15.12:0/2499444811 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654166b29e0 0x5654166bbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:06:57.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.827+0000 7fe52ca00640 1 --2- 172.21.15.12:0/2499444811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166ace80 0x5654166bb570 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fe518009f70 tx=0x7fe5180039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e7b84200451f8c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:06:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.827+0000 7fe512000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:06:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.827+0000 7fe512000640 1 -- 172.21.15.12:0/2499444811 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe518022780 con 0x5654166ace80 2024-03-23T23:06:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.827+0000 7fe532002740 1 -- 172.21.15.12:0/2499444811 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe5140fd070 con 0x5654166ace80 2024-03-23T23:06:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.827+0000 7fe512000640 1 -- 172.21.15.12:0/2499444811 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe518022920 con 0x5654166ace80 2024-03-23T23:06:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.827+0000 7fe512000640 1 -- 172.21.15.12:0/2499444811 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe518022b20 con 0x5654166ace80 2024-03-23T23:06:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.827+0000 7fe532002740 10 client.26717 fetch_fsmap learned FSMap version 23 2024-03-23T23:06:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.827+0000 7fe532002740 1 -- 172.21.15.12:0/2499444811 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56541671e550 con 0x5654166ace80 2024-03-23T23:06:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.828+0000 7fe512000640 1 -- 172.21.15.12:0/2499444811 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe5140fd070 con 0x5654166ace80 2024-03-23T23:06:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.828+0000 7fe512000640 1 -- 172.21.15.12:0/2499444811 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe518012b40 con 0x5654166ace80 2024-03-23T23:06:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.828+0000 7fe532002740 10 client.26717 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:06:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.828+0000 7fe532002740 10 client.26717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:06:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.828+0000 7fe532002740 20 client.26717 populate_metadata read hostname 'smithi012' 2024-03-23T23:06:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.828+0000 7fe532002740 1 --2- 172.21.15.12:0/2499444811 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56541671eea0 0x5654167212c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:06:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.828+0000 7fe532002740 4 client.26717 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:06:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.828+0000 7fe532002740 1 -- 172.21.15.12:0/2499444811 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x565416619ae0 con 0x56541671eea0 2024-03-23T23:06:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.828+0000 7fe527e00640 1 --2- 172.21.15.12:0/2499444811 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56541671eea0 0x5654167212c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:06:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.829+0000 7fe527e00640 1 --2- 172.21.15.12:0/2499444811 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56541671eea0 0x5654167212c0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34516f574b84aec2 server_cookie=7bbb3ed315482acd in_seq=0 out_seq=0 2024-03-23T23:06:57.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.829+0000 7fe512000640 10 client.26717 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:57.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.829+0000 7fe512000640 1 -- 172.21.15.12:0/2499444811 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x565416619ae0 con 0x56541671eea0 2024-03-23T23:06:57.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.829+0000 7fe512000640 10 client.26717 handle_command_reply: tid=0 2024-03-23T23:06:57.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.829+0000 7fe532002740 1 client.26717 shutdown 2024-03-23T23:06:57.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.829+0000 7fe532002740 1 -- 172.21.15.12:0/2499444811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166ace80 msgr2=0x5654166bb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:57.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.829+0000 7fe532002740 1 --2- 172.21.15.12:0/2499444811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166ace80 0x5654166bb570 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fe518009f70 tx=0x7fe5180039c0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.829+0000 7fe532002740 1 -- 172.21.15.12:0/2499444811 shutdown_connections 2024-03-23T23:06:57.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.829+0000 7fe532002740 1 --2- 172.21.15.12:0/2499444811 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56541671eea0 0x5654167212c0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.829+0000 7fe532002740 1 --2- 172.21.15.12:0/2499444811 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654166bcac0 0x5654166bcf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.829+0000 7fe532002740 1 --2- 172.21.15.12:0/2499444811 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654166b29e0 0x5654166bbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.829+0000 7fe532002740 1 --2- 172.21.15.12:0/2499444811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654166ace80 0x5654166bb570 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.829+0000 7fe532002740 1 -- 172.21.15.12:0/2499444811 >> 172.21.15.12:0/2499444811 conn(0x565416675200 msgr2=0x5654166a7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:57.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.829+0000 7fe512000640 0 client.26717 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:06:57.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.829+0000 7fe532002740 1 -- 172.21.15.12:0/2499444811 shutdown_connections 2024-03-23T23:06:57.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.829+0000 7fe532002740 1 -- 172.21.15.12:0/2499444811 wait complete. 2024-03-23T23:06:57.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.830+0000 7fe532002740 20 client.26717 trim_cache size 0 max 16384 2024-03-23T23:06:57.834 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:06:57.834 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T23:06:57.834 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-03-23T23:06:57.834 INFO:teuthology.orchestra.run.smithi012.stdout: "a1cd91b6-eb38-42ce-9921-7a41b1bb9946": { 2024-03-23T23:06:57.834 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-03-23T23:06:57.834 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "a1cd91b6-eb38-42ce-9921-7a41b1bb9946", 2024-03-23T23:06:57.834 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-03-23T23:06:57.834 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:06:57.834 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:06:57.835 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:06:57.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.830+0000 7fe52f400640 1 -- 172.21.15.12:0/3892020992 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe50c0a2880 msgr2=0x7fe50c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:57.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.830+0000 7fe52f400640 1 --2- 172.21.15.12:0/3892020992 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe50c0a2880 0x7fe50c0a4d40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x5654166ae0d0 tx=0x7fe5200096a0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.830+0000 7fe52f400640 1 -- 172.21.15.12:0/3892020992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5280772a0 msgr2=0x7fe52819e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:06:57.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.830+0000 7fe52f400640 1 --2- 172.21.15.12:0/3892020992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5280772a0 0x7fe52819e550 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe514036040 tx=0x7fe514005d50 comp rx=0 tx=0).stop 2024-03-23T23:06:57.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.830+0000 7fe52f400640 1 -- 172.21.15.12:0/3892020992 shutdown_connections 2024-03-23T23:06:57.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.830+0000 7fe52f400640 1 --2- 172.21.15.12:0/3892020992 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe50c0a2880 0x7fe50c0a4d40 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.830+0000 7fe52f400640 1 --2- 172.21.15.12:0/3892020992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe528077c40 0x7fe5281a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.830+0000 7fe52f400640 1 --2- 172.21.15.12:0/3892020992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5280772a0 0x7fe52819e550 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.830+0000 7fe52f400640 1 --2- 172.21.15.12:0/3892020992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe528076060 0x7fe52819e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:06:57.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.830+0000 7fe52f400640 1 -- 172.21.15.12:0/3892020992 >> 172.21.15.12:0/3892020992 conn(0x7fe5280fee50 msgr2=0x7fe5281005a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:06:57.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.830+0000 7fe52f400640 1 -- 172.21.15.12:0/3892020992 shutdown_connections 2024-03-23T23:06:57.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:06:57.830+0000 7fe52f400640 1 -- 172.21.15.12:0/3892020992 wait complete. 2024-03-23T23:06:57.848 INFO:tasks.cephfs.filesystem:scrub status for tag:a1cd91b6-eb38-42ce-9921-7a41b1bb9946 - {'path': '/', 'tag': 'a1cd91b6-eb38-42ce-9921-7a41b1bb9946', 'options': 'recursive,force'} 2024-03-23T23:06:58.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:58 smithi173 ceph-mon[36116]: pgmap v545: 97 pgs: 97 active+clean; 3.1 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:06:58.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:58 smithi173 ceph-mon[36116]: scrub queued for path: / 2024-03-23T23:06:58.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:58 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:06:58.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:58 smithi173 ceph-mon[36116]: scrub summary: active paths [/] 2024-03-23T23:06:58.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:06:58 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:06:58.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:58 smithi203 ceph-mon[36693]: pgmap v545: 97 pgs: 97 active+clean; 3.1 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:06:58.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:58 smithi203 ceph-mon[36693]: scrub queued for path: / 2024-03-23T23:06:58.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:58 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:06:58.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:58 smithi203 ceph-mon[36693]: scrub summary: active paths [/] 2024-03-23T23:06:58.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:06:58 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:06:59.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:58 smithi012 ceph-mon[31695]: pgmap v545: 97 pgs: 97 active+clean; 3.1 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:06:59.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:58 smithi012 ceph-mon[31695]: scrub queued for path: / 2024-03-23T23:06:59.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:58 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:06:59.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:58 smithi012 ceph-mon[31695]: scrub summary: active paths [/] 2024-03-23T23:06:59.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:06:58 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:07:00.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:00 smithi173 ceph-mon[36116]: pgmap v546: 97 pgs: 97 active+clean; 3.2 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:07:00.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:00 smithi203 ceph-mon[36693]: pgmap v546: 97 pgs: 97 active+clean; 3.2 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:07:01.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:00 smithi012 ceph-mon[31695]: pgmap v546: 97 pgs: 97 active+clean; 3.2 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:07:01.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:01 smithi173 ceph-mon[36116]: scrub summary: idle 2024-03-23T23:07:01.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:01 smithi203 ceph-mon[36693]: scrub summary: idle 2024-03-23T23:07:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.957+0000 7f5887e00640 1 -- 172.21.15.12:0/2640756367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58881052d0 msgr2=0x7f588810bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.957+0000 7f5887e00640 1 --2- 172.21.15.12:0/2640756367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58881052d0 0x7f588810bb60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5870009900 tx=0x7f587002bc30 comp rx=0 tx=0).stop 2024-03-23T23:07:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.957+0000 7f5887e00640 1 -- 172.21.15.12:0/2640756367 shutdown_connections 2024-03-23T23:07:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.957+0000 7f5887e00640 1 --2- 172.21.15.12:0/2640756367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58881052d0 0x7f588810bb60 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.957+0000 7f5887e00640 1 --2- 172.21.15.12:0/2640756367 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5888104910 0x7f5888104d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.957+0000 7f5887e00640 1 --2- 172.21.15.12:0/2640756367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5888103720 0x7f5888103b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.957+0000 7f5887e00640 1 -- 172.21.15.12:0/2640756367 >> 172.21.15.12:0/2640756367 conn(0x7f58880fecb0 msgr2=0x7f58881010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.957+0000 7f5887e00640 1 -- 172.21.15.12:0/2640756367 shutdown_connections 2024-03-23T23:07:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.957+0000 7f5887e00640 1 -- 172.21.15.12:0/2640756367 wait complete. 2024-03-23T23:07:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.958+0000 7f5887e00640 1 Processor -- start 2024-03-23T23:07:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.959+0000 7f5887e00640 1 -- start start 2024-03-23T23:07:01.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.959+0000 7f5887e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5888103720 0x7f5888199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:01.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.959+0000 7f5887e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5888104910 0x7f5888199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:01.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.959+0000 7f5887e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58881052d0 0x7f58881a0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:01.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.959+0000 7f5887e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588810f3d0 con 0x7f5888103720 2024-03-23T23:07:01.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.959+0000 7f5887e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588810f250 con 0x7f5888104910 2024-03-23T23:07:01.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.959+0000 7f5887e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588810f550 con 0x7f58881052d0 2024-03-23T23:07:01.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.959+0000 7f5887400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58881052d0 0x7f58881a0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:01.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.959+0000 7f5887400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58881052d0 0x7f58881a0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47926/0 (socket says 172.21.15.12:47926) 2024-03-23T23:07:01.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.959+0000 7f5887400640 1 -- 172.21.15.12:0/4277762692 learned_addr learned my addr 172.21.15.12:0/4277762692 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:01.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.959+0000 7f5886000640 1 --2- 172.21.15.12:0/4277762692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5888104910 0x7f5888199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:01.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.959+0000 7f5886a00640 1 --2- 172.21.15.12:0/4277762692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5888103720 0x7f5888199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:01.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.960+0000 7f5887400640 1 -- 172.21.15.12:0/4277762692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5888104910 msgr2=0x7f5888199ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:01.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.960+0000 7f5887400640 1 --2- 172.21.15.12:0/4277762692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5888104910 0x7f5888199ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:01.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.960+0000 7f5887400640 1 -- 172.21.15.12:0/4277762692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5888103720 msgr2=0x7f5888199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:01.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.960+0000 7f5887400640 1 --2- 172.21.15.12:0/4277762692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5888103720 0x7f5888199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:01.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.960+0000 7f5887400640 1 -- 172.21.15.12:0/4277762692 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58881a15b0 con 0x7f58881052d0 2024-03-23T23:07:01.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.960+0000 7f5886a00640 1 --2- 172.21.15.12:0/4277762692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5888103720 0x7f5888199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:07:01.961 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.960+0000 7f5887400640 1 --2- 172.21.15.12:0/4277762692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58881052d0 0x7f58881a0f40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f587002c8e0 tx=0x7f587002c640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:01.961 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.960+0000 7f587f400640 1 -- 172.21.15.12:0/4277762692 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5870043020 con 0x7f58881052d0 2024-03-23T23:07:01.961 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.960+0000 7f587f400640 1 -- 172.21.15.12:0/4277762692 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f58700054e0 con 0x7f58881052d0 2024-03-23T23:07:01.961 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.960+0000 7f587f400640 1 -- 172.21.15.12:0/4277762692 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5870003d00 con 0x7f58881052d0 2024-03-23T23:07:01.961 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.961+0000 7f5887e00640 1 -- 172.21.15.12:0/4277762692 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58881a1840 con 0x7f58881052d0 2024-03-23T23:07:01.961 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.961+0000 7f5887e00640 1 -- 172.21.15.12:0/4277762692 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58881a1d40 con 0x7f58881052d0 2024-03-23T23:07:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.964+0000 7f587f400640 1 -- 172.21.15.12:0/4277762692 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5870046020 con 0x7f58881052d0 2024-03-23T23:07:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.964+0000 7f58905cc740 1 Processor -- start 2024-03-23T23:07:01.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.964+0000 7f58905cc740 1 -- start start 2024-03-23T23:07:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.964+0000 7f58905cc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ab1a8c3f0 0x562ab1a8c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.965+0000 7f58905cc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ab1a8cde0 0x562ab1a918e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.965+0000 7f58905cc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ab1a91f50 0x562ab1a943e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.965+0000 7f58905cc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ab1a4f3b0 con 0x562ab1a91f50 2024-03-23T23:07:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.965+0000 7f58905cc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ab19f1830 con 0x562ab1a8c3f0 2024-03-23T23:07:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.965+0000 7f58905cc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ab199a5d0 con 0x562ab1a8cde0 2024-03-23T23:07:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.965+0000 7f5886a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ab1a8cde0 0x562ab1a918e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.965+0000 7f5886a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ab1a8cde0 0x562ab1a918e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47938/0 (socket says 172.21.15.12:47938) 2024-03-23T23:07:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.965+0000 7f5886a00640 1 -- 172.21.15.12:0/1876545800 learned_addr learned my addr 172.21.15.12:0/1876545800 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.965+0000 7f5886000640 1 --2- 172.21.15.12:0/1876545800 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ab1a8c3f0 0x562ab1a8c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.965+0000 7f5886000640 1 --2- 172.21.15.12:0/1876545800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ab1a91f50 0x562ab1a943e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.965+0000 7f5886a00640 1 -- 172.21.15.12:0/1876545800 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ab1a8c3f0 msgr2=0x562ab1a8c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.965+0000 7f5886a00640 1 --2- 172.21.15.12:0/1876545800 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ab1a8c3f0 0x562ab1a8c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.965+0000 7f5886a00640 1 -- 172.21.15.12:0/1876545800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ab1a91f50 msgr2=0x562ab1a943e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:01.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.965+0000 7f5886a00640 1 --2- 172.21.15.12:0/1876545800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ab1a91f50 0x562ab1a943e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:01.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.965+0000 7f5886a00640 1 -- 172.21.15.12:0/1876545800 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ab19b22c0 con 0x562ab1a8cde0 2024-03-23T23:07:01.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.965+0000 7f5886000640 1 --2- 172.21.15.12:0/1876545800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ab1a91f50 0x562ab1a943e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:07:01.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.966+0000 7f5886a00640 1 --2- 172.21.15.12:0/1876545800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ab1a8cde0 0x562ab1a918e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f5880009cd0 tx=0x7f58800117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38be5c8650604fe2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:01.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.966+0000 7f587f400640 1 --2- 172.21.15.12:0/4277762692 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f586c0a2870 0x7f586c0a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:01.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.966+0000 7f5877e00640 1 -- 172.21.15.12:0/1876545800 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5880012450 con 0x562ab1a8cde0 2024-03-23T23:07:01.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.966+0000 7f587f400640 1 -- 172.21.15.12:0/4277762692 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f58700f0b00 con 0x7f58881052d0 2024-03-23T23:07:01.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.966+0000 7f5877e00640 1 -- 172.21.15.12:0/1876545800 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5880012ad0 con 0x562ab1a8cde0 2024-03-23T23:07:01.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.966+0000 7f5877e00640 1 -- 172.21.15.12:0/1876545800 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588000fca0 con 0x562ab1a8cde0 2024-03-23T23:07:01.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.966+0000 7f5886000640 1 --2- 172.21.15.12:0/4277762692 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f586c0a2870 0x7f586c0a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:01.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.967+0000 7f58905cc740 1 -- 172.21.15.12:0/1876545800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ab1a8cde0 msgr2=0x562ab1a918e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:01.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.967+0000 7f58905cc740 1 --2- 172.21.15.12:0/1876545800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ab1a8cde0 0x562ab1a918e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f5880009cd0 tx=0x7f58800117b0 comp rx=0 tx=0).stop 2024-03-23T23:07:01.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.967+0000 7f5886000640 1 --2- 172.21.15.12:0/4277762692 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f586c0a2870 0x7f586c0a4d30 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x562ab1a8d640 tx=0x7f5878013040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.967+0000 7f58905cc740 1 -- 172.21.15.12:0/1876545800 shutdown_connections 2024-03-23T23:07:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.967+0000 7f58905cc740 1 --2- 172.21.15.12:0/1876545800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ab1a91f50 0x562ab1a943e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.967+0000 7f58905cc740 1 --2- 172.21.15.12:0/1876545800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ab1a8cde0 0x562ab1a918e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.967+0000 7f58905cc740 1 --2- 172.21.15.12:0/1876545800 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ab1a8c3f0 0x562ab1a8c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.967+0000 7f58905cc740 1 -- 172.21.15.12:0/1876545800 >> 172.21.15.12:0/1876545800 conn(0x562ab1a87960 msgr2=0x562ab1a87d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.967+0000 7f58905cc740 1 -- 172.21.15.12:0/1876545800 shutdown_connections 2024-03-23T23:07:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.967+0000 7f58905cc740 1 -- 172.21.15.12:0/1876545800 wait complete. 2024-03-23T23:07:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.967+0000 7f58905cc740 1 -- start start 2024-03-23T23:07:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.968+0000 7f58905cc740 1 Processor -- start 2024-03-23T23:07:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.968+0000 7f58905cc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ab1a8c3f0 0x562ab1a965d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.968+0000 7f58905cc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ab1a8cde0 0x562ab1a96b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.968+0000 7f58905cc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ab1a91f50 0x562ab1a9db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.968+0000 7f58905cc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ab19ada20 con 0x562ab1a8c3f0 2024-03-23T23:07:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.968+0000 7f58905cc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ab19ed8d0 con 0x562ab1a8cde0 2024-03-23T23:07:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.968+0000 7f58905cc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ab1a94f00 con 0x562ab1a91f50 2024-03-23T23:07:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.968+0000 7f58905cc740 10 client.0 _ll_register_callbacks cb 0x562ab19ea1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.968+0000 7f5886000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ab1a8cde0 0x562ab1a96b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.968+0000 7f5886a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ab1a8c3f0 0x562ab1a965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.968+0000 7f5887400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ab1a91f50 0x562ab1a9db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.968+0000 7f5886000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ab1a8cde0 0x562ab1a96b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:53784/0 (socket says 172.21.15.12:53784) 2024-03-23T23:07:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.968+0000 7f5886a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ab1a8c3f0 0x562ab1a965d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37900/0 (socket says 172.21.15.12:37900) 2024-03-23T23:07:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.968+0000 7f5886000640 1 -- 172.21.15.12:0/1145680659 learned_addr learned my addr 172.21.15.12:0/1145680659 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.969+0000 7f5887400640 1 -- 172.21.15.12:0/1145680659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ab1a8cde0 msgr2=0x562ab1a96b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.969+0000 7f5887400640 1 --2- 172.21.15.12:0/1145680659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ab1a8cde0 0x562ab1a96b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.969+0000 7f5887400640 1 -- 172.21.15.12:0/1145680659 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ab1a8c3f0 msgr2=0x562ab1a965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.969+0000 7f5887400640 1 --2- 172.21.15.12:0/1145680659 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ab1a8c3f0 0x562ab1a965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.969+0000 7f5887400640 1 -- 172.21.15.12:0/1145680659 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562ab1706210 con 0x562ab1a91f50 2024-03-23T23:07:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.969+0000 7f5887400640 1 --2- 172.21.15.12:0/1145680659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ab1a91f50 0x562ab1a9db50 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f58700ee5b0 tx=0x7f58700bd2b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7043691972462945 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.969+0000 7f5874c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:07:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.969+0000 7f5874c00640 1 -- 172.21.15.12:0/1145680659 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5870004470 con 0x562ab1a91f50 2024-03-23T23:07:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.969+0000 7f5874c00640 1 -- 172.21.15.12:0/1145680659 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f58700d6180 con 0x562ab1a91f50 2024-03-23T23:07:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.969+0000 7f58905cc740 1 -- 172.21.15.12:0/1145680659 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5880012ad0 con 0x562ab1a91f50 2024-03-23T23:07:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.970+0000 7f5874c00640 1 -- 172.21.15.12:0/1145680659 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5870005690 con 0x562ab1a91f50 2024-03-23T23:07:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.970+0000 7f58905cc740 10 client.26701 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.970+0000 7f58905cc740 1 -- 172.21.15.12:0/1145680659 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x562ab17d2620 con 0x562ab1a91f50 2024-03-23T23:07:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.970+0000 7f5874c00640 1 -- 172.21.15.12:0/1145680659 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5880012ad0 con 0x562ab1a91f50 2024-03-23T23:07:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.970+0000 7f5874c00640 1 -- 172.21.15.12:0/1145680659 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f58700c5760 con 0x562ab1a91f50 2024-03-23T23:07:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.970+0000 7f58905cc740 10 client.26701 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.970+0000 7f58905cc740 10 client.26701 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:07:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.970+0000 7f58905cc740 20 client.26701 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.970+0000 7f58905cc740 1 --2- 172.21.15.12:0/1145680659 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562ab1aa2420 0x562ab1aa4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.970+0000 7f58905cc740 4 client.26701 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:07:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.970+0000 7f58905cc740 1 -- 172.21.15.12:0/1145680659 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562ab19fed70 con 0x562ab1aa2420 2024-03-23T23:07:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.970+0000 7f5886a00640 1 --2- 172.21.15.12:0/1145680659 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562ab1aa2420 0x562ab1aa4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.971+0000 7f5886a00640 1 --2- 172.21.15.12:0/1145680659 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562ab1aa2420 0x562ab1aa4810 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=430ba29455900626 server_cookie=b868074b09d04706 in_seq=0 out_seq=0 2024-03-23T23:07:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.971+0000 7f5874c00640 10 client.26701 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.972+0000 7f5874c00640 1 -- 172.21.15.12:0/1145680659 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562ab19fed70 con 0x562ab1aa2420 2024-03-23T23:07:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.972+0000 7f5874c00640 10 client.26701 handle_command_reply: tid=0 2024-03-23T23:07:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.972+0000 7f58905cc740 1 client.26701 shutdown 2024-03-23T23:07:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.972+0000 7f58905cc740 1 -- 172.21.15.12:0/1145680659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ab1a91f50 msgr2=0x562ab1a9db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.972+0000 7f58905cc740 1 --2- 172.21.15.12:0/1145680659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ab1a91f50 0x562ab1a9db50 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f58700ee5b0 tx=0x7f58700bd2b0 comp rx=0 tx=0).stop 2024-03-23T23:07:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.972+0000 7f58905cc740 1 -- 172.21.15.12:0/1145680659 shutdown_connections 2024-03-23T23:07:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.972+0000 7f58905cc740 1 --2- 172.21.15.12:0/1145680659 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562ab1aa2420 0x562ab1aa4810 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.972+0000 7f58905cc740 1 --2- 172.21.15.12:0/1145680659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ab1a91f50 0x562ab1a9db50 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.972+0000 7f58905cc740 1 --2- 172.21.15.12:0/1145680659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ab1a8cde0 0x562ab1a96b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.972+0000 7f5874c00640 0 client.26701 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.972+0000 7f58905cc740 1 --2- 172.21.15.12:0/1145680659 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ab1a8c3f0 0x562ab1a965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.972+0000 7f58905cc740 1 -- 172.21.15.12:0/1145680659 >> 172.21.15.12:0/1145680659 conn(0x562ab1a87960 msgr2=0x562ab1a86b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.972+0000 7f58905cc740 1 -- 172.21.15.12:0/1145680659 shutdown_connections 2024-03-23T23:07:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.973+0000 7f58905cc740 1 -- 172.21.15.12:0/1145680659 wait complete. 2024-03-23T23:07:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.973+0000 7f58905cc740 20 client.26701 trim_cache size 0 max 16384 2024-03-23T23:07:01.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.988+0000 7f58905cc740 1 Processor -- start 2024-03-23T23:07:01.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.988+0000 7f58905cc740 1 -- start start 2024-03-23T23:07:01.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.989+0000 7f58905cc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ab1a8c3f0 0x562ab1a95fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:01.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.989+0000 7f58905cc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ab1a8cde0 0x562ab1a9db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:01.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.989+0000 7f58905cc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ab1a91f50 0x562ab1a862e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:01.989 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.989+0000 7f58905cc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58881ace90 con 0x562ab1a8cde0 2024-03-23T23:07:01.989 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.989+0000 7f58905cc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58881aa990 con 0x562ab1a8c3f0 2024-03-23T23:07:01.989 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.989+0000 7f58905cc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58881a8a30 con 0x562ab1a91f50 2024-03-23T23:07:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.989+0000 7f5886a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ab1a8c3f0 0x562ab1a95fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.989+0000 7f5886a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ab1a8c3f0 0x562ab1a95fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:53800/0 (socket says 172.21.15.12:53800) 2024-03-23T23:07:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.989+0000 7f5886a00640 1 -- 172.21.15.12:0/4171875822 learned_addr learned my addr 172.21.15.12:0/4171875822 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.989+0000 7f5886000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ab1a8cde0 0x562ab1a9db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.989+0000 7f5887400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ab1a91f50 0x562ab1a862e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.989+0000 7f5886a00640 1 -- 172.21.15.12:0/4171875822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ab1a91f50 msgr2=0x562ab1a862e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.989+0000 7f5886a00640 1 --2- 172.21.15.12:0/4171875822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ab1a91f50 0x562ab1a862e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.989+0000 7f5886a00640 1 -- 172.21.15.12:0/4171875822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ab1a8cde0 msgr2=0x562ab1a9db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.989+0000 7f5886a00640 1 --2- 172.21.15.12:0/4171875822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ab1a8cde0 0x562ab1a9db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.989+0000 7f5886a00640 1 -- 172.21.15.12:0/4171875822 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ab1a82d10 con 0x562ab1a8c3f0 2024-03-23T23:07:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.990+0000 7f5886a00640 1 --2- 172.21.15.12:0/4171875822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ab1a8c3f0 0x562ab1a95fb0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5880009ca0 tx=0x7f5880014040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79ccf96b18e3e0c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.990+0000 7f5877400640 1 -- 172.21.15.12:0/4171875822 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58800098c0 con 0x562ab1a8c3f0 2024-03-23T23:07:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.990+0000 7f5877400640 1 -- 172.21.15.12:0/4171875822 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f588002e030 con 0x562ab1a8c3f0 2024-03-23T23:07:01.991 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.990+0000 7f5877400640 1 -- 172.21.15.12:0/4171875822 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58800052f0 con 0x562ab1a8c3f0 2024-03-23T23:07:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.990+0000 7f58905cc740 1 -- 172.21.15.12:0/4171875822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ab1a8c3f0 msgr2=0x562ab1a95fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.990+0000 7f58905cc740 1 --2- 172.21.15.12:0/4171875822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ab1a8c3f0 0x562ab1a95fb0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5880009ca0 tx=0x7f5880014040 comp rx=0 tx=0).stop 2024-03-23T23:07:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.991+0000 7f58905cc740 1 -- 172.21.15.12:0/4171875822 shutdown_connections 2024-03-23T23:07:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.991+0000 7f58905cc740 1 --2- 172.21.15.12:0/4171875822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ab1a91f50 0x562ab1a862e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.991+0000 7f58905cc740 1 --2- 172.21.15.12:0/4171875822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ab1a8cde0 0x562ab1a9db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.991+0000 7f58905cc740 1 --2- 172.21.15.12:0/4171875822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ab1a8c3f0 0x562ab1a95fb0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.991+0000 7f58905cc740 1 -- 172.21.15.12:0/4171875822 >> 172.21.15.12:0/4171875822 conn(0x562ab1a87960 msgr2=0x562ab1a958a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.991+0000 7f58905cc740 1 -- 172.21.15.12:0/4171875822 shutdown_connections 2024-03-23T23:07:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.991+0000 7f58905cc740 1 -- 172.21.15.12:0/4171875822 wait complete. 2024-03-23T23:07:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.991+0000 7f58905cc740 1 -- start start 2024-03-23T23:07:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.991+0000 7f58905cc740 1 Processor -- start 2024-03-23T23:07:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.992+0000 7f58905cc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ab1a8c3f0 0x562ab1a9b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.992+0000 7f58905cc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ab1a8cde0 0x562ab1a9bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.992+0000 7f58905cc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ab1a91f50 0x562ab1a9cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.992+0000 7f58905cc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58881a9030 con 0x562ab1a91f50 2024-03-23T23:07:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.992+0000 7f58905cc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58881aa330 con 0x562ab1a8cde0 2024-03-23T23:07:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.992+0000 7f58905cc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58881a9670 con 0x562ab1a8c3f0 2024-03-23T23:07:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.992+0000 7f58905cc740 10 client.0 _ll_register_callbacks cb 0x7f58700095c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.992+0000 7f5886a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ab1a8c3f0 0x562ab1a9b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.992+0000 7f5886000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ab1a8cde0 0x562ab1a9bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.992+0000 7f5886a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ab1a8c3f0 0x562ab1a9b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47958/0 (socket says 172.21.15.12:47958) 2024-03-23T23:07:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.992+0000 7f5886a00640 1 -- 172.21.15.12:0/3377380226 learned_addr learned my addr 172.21.15.12:0/3377380226 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.992+0000 7f5887400640 1 --2- 172.21.15.12:0/3377380226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ab1a91f50 0x562ab1a9cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:01.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.992+0000 7f5886000640 1 -- 172.21.15.12:0/3377380226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ab1a8c3f0 msgr2=0x562ab1a9b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:01.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.992+0000 7f5886000640 1 --2- 172.21.15.12:0/3377380226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ab1a8c3f0 0x562ab1a9b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:01.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.992+0000 7f5886000640 1 -- 172.21.15.12:0/3377380226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ab1a91f50 msgr2=0x562ab1a9cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:01.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.992+0000 7f5886000640 1 --2- 172.21.15.12:0/3377380226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ab1a91f50 0x562ab1a9cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:01.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.992+0000 7f5886000640 1 -- 172.21.15.12:0/3377380226 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562ab1a9d240 con 0x562ab1a8cde0 2024-03-23T23:07:01.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.993+0000 7f5886000640 1 --2- 172.21.15.12:0/3377380226 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ab1a8cde0 0x562ab1a9bc40 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f5878013010 tx=0x7f5878004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67ff05890f8e6785 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:01.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.993+0000 7f5874c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:07:01.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.993+0000 7f5874c00640 1 -- 172.21.15.12:0/3377380226 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5878031060 con 0x562ab1a8cde0 2024-03-23T23:07:01.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.993+0000 7f5874c00640 1 -- 172.21.15.12:0/3377380226 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5878009040 con 0x562ab1a8cde0 2024-03-23T23:07:01.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.993+0000 7f58905cc740 1 -- 172.21.15.12:0/3377380226 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f588002e030 con 0x562ab1a8cde0 2024-03-23T23:07:01.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.993+0000 7f5874c00640 1 -- 172.21.15.12:0/3377380226 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5878021b80 con 0x562ab1a8cde0 2024-03-23T23:07:01.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.993+0000 7f58905cc740 10 client.26735 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:01.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.993+0000 7f58905cc740 1 -- 172.21.15.12:0/3377380226 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x562ab1afe250 con 0x562ab1a8cde0 2024-03-23T23:07:01.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.994+0000 7f5874c00640 1 -- 172.21.15.12:0/3377380226 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f588002e030 con 0x562ab1a8cde0 2024-03-23T23:07:01.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.994+0000 7f5874c00640 1 -- 172.21.15.12:0/3377380226 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f5878020410 con 0x562ab1a8cde0 2024-03-23T23:07:01.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.994+0000 7f58905cc740 10 client.26735 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:01.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.994+0000 7f58905cc740 10 client.26735 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:07:01.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.994+0000 7f58905cc740 20 client.26735 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:01.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.994+0000 7f58905cc740 1 --2- 172.21.15.12:0/3377380226 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562ab1afec30 0x562ab1b01050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:01.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.994+0000 7f58905cc740 4 client.26735 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:07:01.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.994+0000 7f58905cc740 1 -- 172.21.15.12:0/3377380226 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562ab19f90c0 con 0x562ab1afec30 2024-03-23T23:07:01.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.994+0000 7f5886a00640 1 --2- 172.21.15.12:0/3377380226 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562ab1afec30 0x562ab1b01050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:01.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.995+0000 7f5886a00640 1 --2- 172.21.15.12:0/3377380226 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562ab1afec30 0x562ab1b01050 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fc9e648213523b9 server_cookie=59e6e92f2614190c in_seq=0 out_seq=0 2024-03-23T23:07:01.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:01.995+0000 7f5874c00640 10 client.26735 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.007+0000 7f5874c00640 1 -- 172.21.15.12:0/3377380226 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562ab19f90c0 con 0x562ab1afec30 2024-03-23T23:07:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.007+0000 7f5874c00640 10 client.26735 handle_command_reply: tid=0 2024-03-23T23:07:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.007+0000 7f58905cc740 1 client.26735 shutdown 2024-03-23T23:07:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.007+0000 7f58905cc740 1 -- 172.21.15.12:0/3377380226 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ab1a8cde0 msgr2=0x562ab1a9bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.007+0000 7f58905cc740 1 --2- 172.21.15.12:0/3377380226 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ab1a8cde0 0x562ab1a9bc40 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f5878013010 tx=0x7f5878004a40 comp rx=0 tx=0).stop 2024-03-23T23:07:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.007+0000 7f58905cc740 1 -- 172.21.15.12:0/3377380226 shutdown_connections 2024-03-23T23:07:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.007+0000 7f58905cc740 1 --2- 172.21.15.12:0/3377380226 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562ab1afec30 0x562ab1b01050 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.007+0000 7f58905cc740 1 --2- 172.21.15.12:0/3377380226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ab1a91f50 0x562ab1a9cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.007+0000 7f58905cc740 1 --2- 172.21.15.12:0/3377380226 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ab1a8cde0 0x562ab1a9bc40 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.007+0000 7f58905cc740 1 --2- 172.21.15.12:0/3377380226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ab1a8c3f0 0x562ab1a9b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.007+0000 7f5874c00640 0 client.26735 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.007+0000 7f58905cc740 1 -- 172.21.15.12:0/3377380226 >> 172.21.15.12:0/3377380226 conn(0x562ab1a87960 msgr2=0x562ab1a8abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.007+0000 7f58905cc740 1 -- 172.21.15.12:0/3377380226 shutdown_connections 2024-03-23T23:07:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.007+0000 7f58905cc740 1 -- 172.21.15.12:0/3377380226 wait complete. 2024-03-23T23:07:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.008+0000 7f58905cc740 20 client.26735 trim_cache size 0 max 16384 2024-03-23T23:07:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:07:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:07:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:07:02.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:07:02.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.008+0000 7f5887e00640 1 -- 172.21.15.12:0/4277762692 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f586c0a2870 msgr2=0x7f586c0a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:02.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.008+0000 7f5887e00640 1 --2- 172.21.15.12:0/4277762692 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f586c0a2870 0x7f586c0a4d30 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x562ab1a8d640 tx=0x7f5878013040 comp rx=0 tx=0).stop 2024-03-23T23:07:02.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.008+0000 7f5887e00640 1 -- 172.21.15.12:0/4277762692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58881052d0 msgr2=0x7f58881a0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:02.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.008+0000 7f5887e00640 1 --2- 172.21.15.12:0/4277762692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58881052d0 0x7f58881a0f40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f587002c8e0 tx=0x7f587002c640 comp rx=0 tx=0).stop 2024-03-23T23:07:02.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.008+0000 7f5887e00640 1 -- 172.21.15.12:0/4277762692 shutdown_connections 2024-03-23T23:07:02.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.008+0000 7f5887e00640 1 --2- 172.21.15.12:0/4277762692 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f586c0a2870 0x7f586c0a4d30 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:02.009 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.008+0000 7f5887e00640 1 --2- 172.21.15.12:0/4277762692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58881052d0 0x7f58881a0f40 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:02.010 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.008+0000 7f5887e00640 1 --2- 172.21.15.12:0/4277762692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5888104910 0x7f5888199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:02.010 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.008+0000 7f5887e00640 1 --2- 172.21.15.12:0/4277762692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5888103720 0x7f5888199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:02.010 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.008+0000 7f5887e00640 1 -- 172.21.15.12:0/4277762692 >> 172.21.15.12:0/4277762692 conn(0x7f58880fecb0 msgr2=0x7f5888107bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:02.010 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.009+0000 7f5887e00640 1 -- 172.21.15.12:0/4277762692 shutdown_connections 2024-03-23T23:07:02.010 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:02.009+0000 7f5887e00640 1 -- 172.21.15.12:0/4277762692 wait complete. 2024-03-23T23:07:02.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:01 smithi012 ceph-mon[31695]: scrub summary: idle 2024-03-23T23:07:02.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:02 smithi173 ceph-mon[36116]: pgmap v547: 97 pgs: 97 active+clean; 3.2 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-03-23T23:07:02.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:02 smithi203 ceph-mon[36693]: pgmap v547: 97 pgs: 97 active+clean; 3.2 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-03-23T23:07:02.955 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:02 smithi012 ceph-mon[31695]: pgmap v547: 97 pgs: 97 active+clean; 3.2 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-03-23T23:07:04.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:04 smithi203 ceph-mon[36693]: pgmap v548: 97 pgs: 97 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-03-23T23:07:05.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:04 smithi012 ceph-mon[31695]: pgmap v548: 97 pgs: 97 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-03-23T23:07:05.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:04 smithi173 ceph-mon[36116]: pgmap v548: 97 pgs: 97 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-03-23T23:07:05.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:05 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:07:06.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:05 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:07:06.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:05 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:07:06.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:06 smithi203 ceph-mon[36693]: pgmap v549: 97 pgs: 97 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:07:06.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:06 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:07:06.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:06 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:07:06.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:06 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:07:07.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:06 smithi012 ceph-mon[31695]: pgmap v549: 97 pgs: 97 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:07:07.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:06 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:07:07.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:06 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:07:07.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:06 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:07:07.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:06 smithi173 ceph-mon[36116]: pgmap v549: 97 pgs: 97 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:07:07.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:06 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:07:07.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:06 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:07:07.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:06 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:07:08.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:08 smithi203 ceph-mon[36693]: pgmap v550: 97 pgs: 97 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:07:09.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:08 smithi012 ceph-mon[31695]: pgmap v550: 97 pgs: 97 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:07:09.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:08 smithi173 ceph-mon[36116]: pgmap v550: 97 pgs: 97 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:07:10.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:10 smithi203 ceph-mon[36693]: pgmap v551: 97 pgs: 97 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:07:11.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:10 smithi012 ceph-mon[31695]: pgmap v551: 97 pgs: 97 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:07:11.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:10 smithi173 ceph-mon[36116]: pgmap v551: 97 pgs: 97 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:07:12.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:12 smithi203 ceph-mon[36693]: pgmap v552: 97 pgs: 97 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:07:13.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:12 smithi012 ceph-mon[31695]: pgmap v552: 97 pgs: 97 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:07:13.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:12 smithi173 ceph-mon[36116]: pgmap v552: 97 pgs: 97 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:07:14.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:14 smithi203 ceph-mon[36693]: pgmap v553: 97 pgs: 97 active+clean; 3.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:07:15.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:14 smithi012 ceph-mon[31695]: pgmap v553: 97 pgs: 97 active+clean; 3.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:07:15.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:14 smithi173 ceph-mon[36116]: pgmap v553: 97 pgs: 97 active+clean; 3.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:07:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.158+0000 7fa517200640 1 -- 172.21.15.12:0/2608586991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa518104910 msgr2=0x7fa518104d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.158+0000 7fa517200640 1 --2- 172.21.15.12:0/2608586991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa518104910 0x7fa518104d90 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fa504009900 tx=0x7fa50402bc30 comp rx=0 tx=0).stop 2024-03-23T23:07:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.158+0000 7fa517200640 1 -- 172.21.15.12:0/2608586991 shutdown_connections 2024-03-23T23:07:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.158+0000 7fa517200640 1 --2- 172.21.15.12:0/2608586991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5181052d0 0x7fa51810bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.158+0000 7fa517200640 1 --2- 172.21.15.12:0/2608586991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa518104910 0x7fa518104d90 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.158+0000 7fa517200640 1 --2- 172.21.15.12:0/2608586991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa518103720 0x7fa518103b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.158+0000 7fa517200640 1 -- 172.21.15.12:0/2608586991 >> 172.21.15.12:0/2608586991 conn(0x7fa5180fecb0 msgr2=0x7fa5181010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.158+0000 7fa517200640 1 -- 172.21.15.12:0/2608586991 shutdown_connections 2024-03-23T23:07:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.158+0000 7fa517200640 1 -- 172.21.15.12:0/2608586991 wait complete. 2024-03-23T23:07:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.159+0000 7fa517200640 1 Processor -- start 2024-03-23T23:07:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.159+0000 7fa517200640 1 -- start start 2024-03-23T23:07:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.160+0000 7fa517200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa518103720 0x7fa518199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.160+0000 7fa517200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa518104910 0x7fa518199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.160+0000 7fa517200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5181052d0 0x7fa5181a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.160+0000 7fa517200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51810f3d0 con 0x7fa518103720 2024-03-23T23:07:15.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.160+0000 7fa517200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51810f250 con 0x7fa5181052d0 2024-03-23T23:07:15.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.160+0000 7fa517200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51810f550 con 0x7fa518104910 2024-03-23T23:07:15.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.160+0000 7fa515400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa518104910 0x7fa518199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:15.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.160+0000 7fa515e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa518103720 0x7fa518199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:15.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.160+0000 7fa515400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa518104910 0x7fa518199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58576/0 (socket says 172.21.15.12:58576) 2024-03-23T23:07:15.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.160+0000 7fa515400640 1 -- 172.21.15.12:0/541754346 learned_addr learned my addr 172.21.15.12:0/541754346 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:15.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.160+0000 7fa516800640 1 --2- 172.21.15.12:0/541754346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5181052d0 0x7fa5181a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:15.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.160+0000 7fa515400640 1 -- 172.21.15.12:0/541754346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5181052d0 msgr2=0x7fa5181a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:15.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.160+0000 7fa515400640 1 --2- 172.21.15.12:0/541754346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5181052d0 0x7fa5181a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.160+0000 7fa515400640 1 -- 172.21.15.12:0/541754346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa518103720 msgr2=0x7fa518199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:15.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.160+0000 7fa515400640 1 --2- 172.21.15.12:0/541754346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa518103720 0x7fa518199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.160+0000 7fa515400640 1 -- 172.21.15.12:0/541754346 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5181a1630 con 0x7fa518104910 2024-03-23T23:07:15.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.161+0000 7fa515400640 1 --2- 172.21.15.12:0/541754346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa518104910 0x7fa518199f50 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa504000c00 tx=0x7fa504005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.161+0000 7fa50ea00640 1 -- 172.21.15.12:0/541754346 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa504043020 con 0x7fa518104910 2024-03-23T23:07:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.161+0000 7fa50ea00640 1 -- 172.21.15.12:0/541754346 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5040054e0 con 0x7fa518104910 2024-03-23T23:07:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.161+0000 7fa517200640 1 -- 172.21.15.12:0/541754346 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5181a18c0 con 0x7fa518104910 2024-03-23T23:07:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.161+0000 7fa50ea00640 1 -- 172.21.15.12:0/541754346 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa504003df0 con 0x7fa518104910 2024-03-23T23:07:15.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.161+0000 7fa517200640 1 -- 172.21.15.12:0/541754346 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5181a1e10 con 0x7fa518104910 2024-03-23T23:07:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.164+0000 7fa50ea00640 1 -- 172.21.15.12:0/541754346 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa504046020 con 0x7fa518104910 2024-03-23T23:07:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.165+0000 7fa51fc30740 1 Processor -- start 2024-03-23T23:07:15.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.165+0000 7fa51fc30740 1 -- start start 2024-03-23T23:07:15.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.165+0000 7fa51fc30740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56440cb693f0 0x56440cb69810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.165+0000 7fa51fc30740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb69de0 0x56440cb6e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.165+0000 7fa51fc30740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56440cb6ef50 0x56440cb713e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.165+0000 7fa51fc30740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56440cb2c3b0 con 0x56440cb693f0 2024-03-23T23:07:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.165+0000 7fa51fc30740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56440cace830 con 0x56440cb69de0 2024-03-23T23:07:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.165+0000 7fa51fc30740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x56440ca775d0 con 0x56440cb6ef50 2024-03-23T23:07:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.166+0000 7fa515e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb69de0 0x56440cb6e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.166+0000 7fa515400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56440cb6ef50 0x56440cb713e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.166+0000 7fa515e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb69de0 0x56440cb6e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60782/0 (socket says 172.21.15.12:60782) 2024-03-23T23:07:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.166+0000 7fa515e00640 1 -- 172.21.15.12:0/999260419 learned_addr learned my addr 172.21.15.12:0/999260419 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.166+0000 7fa516800640 1 --2- 172.21.15.12:0/999260419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56440cb693f0 0x56440cb69810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.166+0000 7fa515e00640 1 -- 172.21.15.12:0/999260419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56440cb6ef50 msgr2=0x56440cb713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.166+0000 7fa515e00640 1 --2- 172.21.15.12:0/999260419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56440cb6ef50 0x56440cb713e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.166+0000 7fa515e00640 1 -- 172.21.15.12:0/999260419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56440cb693f0 msgr2=0x56440cb69810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.166+0000 7fa515e00640 1 --2- 172.21.15.12:0/999260419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56440cb693f0 0x56440cb69810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.166+0000 7fa515e00640 1 -- 172.21.15.12:0/999260419 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56440ca8f2c0 con 0x56440cb69de0 2024-03-23T23:07:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.166+0000 7fa50ea00640 1 --2- 172.21.15.12:0/541754346 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa4fc0a27f0 0x7fa4fc0a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.166+0000 7fa515e00640 1 --2- 172.21.15.12:0/999260419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb69de0 0x56440cb6e8e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fa50800bcb0 tx=0x7fa5080107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25b127c30c8ef154 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.167+0000 7fa50ea00640 1 -- 172.21.15.12:0/541754346 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fa5040f0c80 con 0x7fa518104910 2024-03-23T23:07:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.167+0000 7fa503400640 1 -- 172.21.15.12:0/999260419 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa508004020 con 0x56440cb69de0 2024-03-23T23:07:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.167+0000 7fa503400640 1 -- 172.21.15.12:0/999260419 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa508005500 con 0x56440cb69de0 2024-03-23T23:07:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.167+0000 7fa516800640 1 --2- 172.21.15.12:0/541754346 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa4fc0a27f0 0x7fa4fc0a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.167+0000 7fa503400640 1 -- 172.21.15.12:0/999260419 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa508004df0 con 0x56440cb69de0 2024-03-23T23:07:15.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.167+0000 7fa51fc30740 1 -- 172.21.15.12:0/999260419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb69de0 msgr2=0x56440cb6e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:15.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.167+0000 7fa51fc30740 1 --2- 172.21.15.12:0/999260419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb69de0 0x56440cb6e8e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fa50800bcb0 tx=0x7fa5080107a0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.167+0000 7fa516800640 1 --2- 172.21.15.12:0/541754346 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa4fc0a27f0 0x7fa4fc0a4cb0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x56440cb6a640 tx=0x7fa510009600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:15.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.168+0000 7fa51fc30740 1 -- 172.21.15.12:0/999260419 shutdown_connections 2024-03-23T23:07:15.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.168+0000 7fa51fc30740 1 --2- 172.21.15.12:0/999260419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56440cb6ef50 0x56440cb713e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.168+0000 7fa51fc30740 1 --2- 172.21.15.12:0/999260419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb69de0 0x56440cb6e8e0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.168+0000 7fa51fc30740 1 --2- 172.21.15.12:0/999260419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56440cb693f0 0x56440cb69810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.168+0000 7fa51fc30740 1 -- 172.21.15.12:0/999260419 >> 172.21.15.12:0/999260419 conn(0x56440cb64960 msgr2=0x56440cb64d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:15.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.168+0000 7fa51fc30740 1 -- 172.21.15.12:0/999260419 shutdown_connections 2024-03-23T23:07:15.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.168+0000 7fa51fc30740 1 -- 172.21.15.12:0/999260419 wait complete. 2024-03-23T23:07:15.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.168+0000 7fa51fc30740 1 -- start start 2024-03-23T23:07:15.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.168+0000 7fa51fc30740 1 Processor -- start 2024-03-23T23:07:15.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.169+0000 7fa51fc30740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb693f0 0x56440cb735c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:15.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.169+0000 7fa51fc30740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56440cb69de0 0x56440cb73b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:15.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.169+0000 7fa51fc30740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56440cb6ef50 0x56440cb7ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:15.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.169+0000 7fa51fc30740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56440ca8aa20 con 0x56440cb6ef50 2024-03-23T23:07:15.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.169+0000 7fa51fc30740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56440caca8d0 con 0x56440cb693f0 2024-03-23T23:07:15.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.169+0000 7fa51fc30740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x56440cb71f00 con 0x56440cb69de0 2024-03-23T23:07:15.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.169+0000 7fa515e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb693f0 0x56440cb735c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:15.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.169+0000 7fa515e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb693f0 0x56440cb735c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60792/0 (socket says 172.21.15.12:60792) 2024-03-23T23:07:15.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.169+0000 7fa51fc30740 10 client.0 _ll_register_callbacks cb 0x56440cac71c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:15.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.169+0000 7fa515e00640 1 -- 172.21.15.12:0/594347007 learned_addr learned my addr 172.21.15.12:0/594347007 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:15.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.169+0000 7fa515400640 1 --2- 172.21.15.12:0/594347007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56440cb69de0 0x56440cb73b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:15.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.169+0000 7fa516800640 1 --2- 172.21.15.12:0/594347007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56440cb6ef50 0x56440cb7ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:15.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.169+0000 7fa515e00640 1 -- 172.21.15.12:0/594347007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56440cb69de0 msgr2=0x56440cb73b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:15.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.169+0000 7fa515e00640 1 --2- 172.21.15.12:0/594347007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56440cb69de0 0x56440cb73b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.169+0000 7fa515e00640 1 -- 172.21.15.12:0/594347007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56440cb6ef50 msgr2=0x56440cb7ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:15.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.169+0000 7fa515e00640 1 --2- 172.21.15.12:0/594347007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56440cb6ef50 0x56440cb7ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.169+0000 7fa515e00640 1 -- 172.21.15.12:0/594347007 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56440c7e3210 con 0x56440cb693f0 2024-03-23T23:07:15.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.169+0000 7fa516800640 1 --2- 172.21.15.12:0/594347007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56440cb6ef50 0x56440cb7ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:07:15.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.170+0000 7fa515e00640 1 --2- 172.21.15.12:0/594347007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb693f0 0x56440cb735c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fa50801a000 tx=0x7fa50800bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=351cbe8c0fd1723f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:15.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.170+0000 7fa4f7e00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:07:15.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.170+0000 7fa4f7e00640 1 -- 172.21.15.12:0/594347007 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50802b060 con 0x56440cb693f0 2024-03-23T23:07:15.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.170+0000 7fa4f7e00640 1 -- 172.21.15.12:0/594347007 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fa508004d00 con 0x56440cb693f0 2024-03-23T23:07:15.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.170+0000 7fa51fc30740 1 -- 172.21.15.12:0/594347007 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa508005500 con 0x56440cb693f0 2024-03-23T23:07:15.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.170+0000 7fa4f7e00640 1 -- 172.21.15.12:0/594347007 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5080046c0 con 0x56440cb693f0 2024-03-23T23:07:15.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.170+0000 7fa51fc30740 10 client.26747 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:15.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.170+0000 7fa51fc30740 1 -- 172.21.15.12:0/594347007 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56440c8af620 con 0x56440cb693f0 2024-03-23T23:07:15.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.171+0000 7fa4f7e00640 1 -- 172.21.15.12:0/594347007 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa508005500 con 0x56440cb693f0 2024-03-23T23:07:15.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.171+0000 7fa4f7e00640 1 -- 172.21.15.12:0/594347007 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fa508011a10 con 0x56440cb693f0 2024-03-23T23:07:15.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.171+0000 7fa51fc30740 10 client.26747 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:15.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.171+0000 7fa51fc30740 10 client.26747 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:07:15.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.171+0000 7fa51fc30740 20 client.26747 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:15.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.171+0000 7fa51fc30740 1 --2- 172.21.15.12:0/594347007 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56440cb7f130 0x56440cb81520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:15.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.171+0000 7fa51fc30740 4 client.26747 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:07:15.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.171+0000 7fa51fc30740 1 -- 172.21.15.12:0/594347007 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56440cadbd70 con 0x56440cb7f130 2024-03-23T23:07:15.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.171+0000 7fa515400640 1 --2- 172.21.15.12:0/594347007 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56440cb7f130 0x56440cb81520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:15.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.172+0000 7fa515400640 1 --2- 172.21.15.12:0/594347007 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56440cb7f130 0x56440cb81520 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0a96e0dc44ec30b server_cookie=1fb99d622731482b in_seq=0 out_seq=0 2024-03-23T23:07:15.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.172+0000 7fa4f7e00640 10 client.26747 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:15.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.172+0000 7fa4f7e00640 1 -- 172.21.15.12:0/594347007 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56440cadbd70 con 0x56440cb7f130 2024-03-23T23:07:15.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.172+0000 7fa4f7e00640 10 client.26747 handle_command_reply: tid=0 2024-03-23T23:07:15.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.172+0000 7fa51fc30740 1 client.26747 shutdown 2024-03-23T23:07:15.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.172+0000 7fa51fc30740 1 -- 172.21.15.12:0/594347007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb693f0 msgr2=0x56440cb735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:15.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.172+0000 7fa51fc30740 1 --2- 172.21.15.12:0/594347007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb693f0 0x56440cb735c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fa50801a000 tx=0x7fa50800bfd0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.173+0000 7fa51fc30740 1 -- 172.21.15.12:0/594347007 shutdown_connections 2024-03-23T23:07:15.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.173+0000 7fa51fc30740 1 --2- 172.21.15.12:0/594347007 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56440cb7f130 0x56440cb81520 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.173+0000 7fa51fc30740 1 --2- 172.21.15.12:0/594347007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56440cb6ef50 0x56440cb7ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.173+0000 7fa51fc30740 1 --2- 172.21.15.12:0/594347007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56440cb69de0 0x56440cb73b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.173+0000 7fa51fc30740 1 --2- 172.21.15.12:0/594347007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb693f0 0x56440cb735c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.173+0000 7fa4f7e00640 0 client.26747 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:15.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.173+0000 7fa51fc30740 1 -- 172.21.15.12:0/594347007 >> 172.21.15.12:0/594347007 conn(0x56440cb64960 msgr2=0x56440cb63b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:15.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.173+0000 7fa51fc30740 1 -- 172.21.15.12:0/594347007 shutdown_connections 2024-03-23T23:07:15.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.173+0000 7fa51fc30740 1 -- 172.21.15.12:0/594347007 wait complete. 2024-03-23T23:07:15.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.173+0000 7fa51fc30740 20 client.26747 trim_cache size 0 max 16384 2024-03-23T23:07:15.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.188+0000 7fa51fc30740 1 Processor -- start 2024-03-23T23:07:15.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.188+0000 7fa51fc30740 1 -- start start 2024-03-23T23:07:15.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.188+0000 7fa51fc30740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56440cb693f0 0x56440cb639f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:15.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.188+0000 7fa51fc30740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56440cb69de0 0x56440cb7ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:15.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.189+0000 7fa51fc30740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb6ef50 0x56440cb85cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:15.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.189+0000 7fa51fc30740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5181acf10 con 0x56440cb693f0 2024-03-23T23:07:15.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.189+0000 7fa51fc30740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5181aaa10 con 0x56440cb6ef50 2024-03-23T23:07:15.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.189+0000 7fa51fc30740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5181a8ab0 con 0x56440cb69de0 2024-03-23T23:07:15.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.189+0000 7fa515400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56440cb69de0 0x56440cb7ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:15.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.189+0000 7fa515400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56440cb69de0 0x56440cb7ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58600/0 (socket says 172.21.15.12:58600) 2024-03-23T23:07:15.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.189+0000 7fa515400640 1 -- 172.21.15.12:0/448132655 learned_addr learned my addr 172.21.15.12:0/448132655 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:15.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.189+0000 7fa516800640 1 --2- 172.21.15.12:0/448132655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb6ef50 0x56440cb85cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:15.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.189+0000 7fa515e00640 1 --2- 172.21.15.12:0/448132655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56440cb693f0 0x56440cb639f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:15.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.189+0000 7fa515400640 1 -- 172.21.15.12:0/448132655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb6ef50 msgr2=0x56440cb85cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:15.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.189+0000 7fa515400640 1 --2- 172.21.15.12:0/448132655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb6ef50 0x56440cb85cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.189+0000 7fa515400640 1 -- 172.21.15.12:0/448132655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56440cb693f0 msgr2=0x56440cb639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.189+0000 7fa515400640 1 --2- 172.21.15.12:0/448132655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56440cb693f0 0x56440cb639f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.189+0000 7fa515400640 1 -- 172.21.15.12:0/448132655 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56440cb642a0 con 0x56440cb69de0 2024-03-23T23:07:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.189+0000 7fa515400640 1 --2- 172.21.15.12:0/448132655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56440cb69de0 0x56440cb7ab00 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa5040b9a30 tx=0x7fa5040f30d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77418d970468f538 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.190+0000 7fa502a00640 1 -- 172.21.15.12:0/448132655 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5040b9870 con 0x56440cb69de0 2024-03-23T23:07:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.190+0000 7fa502a00640 1 -- 172.21.15.12:0/448132655 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa504004c80 con 0x56440cb69de0 2024-03-23T23:07:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.190+0000 7fa502a00640 1 -- 172.21.15.12:0/448132655 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5040bcf30 con 0x56440cb69de0 2024-03-23T23:07:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.190+0000 7fa51fc30740 1 -- 172.21.15.12:0/448132655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56440cb69de0 msgr2=0x56440cb7ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.190+0000 7fa51fc30740 1 --2- 172.21.15.12:0/448132655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56440cb69de0 0x56440cb7ab00 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa5040b9a30 tx=0x7fa5040f30d0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.191+0000 7fa51fc30740 1 -- 172.21.15.12:0/448132655 shutdown_connections 2024-03-23T23:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.191+0000 7fa51fc30740 1 --2- 172.21.15.12:0/448132655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb6ef50 0x56440cb85cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.191+0000 7fa51fc30740 1 --2- 172.21.15.12:0/448132655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56440cb69de0 0x56440cb7ab00 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.191+0000 7fa51fc30740 1 --2- 172.21.15.12:0/448132655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56440cb693f0 0x56440cb639f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.191+0000 7fa51fc30740 1 -- 172.21.15.12:0/448132655 >> 172.21.15.12:0/448132655 conn(0x56440cb64960 msgr2=0x56440cb632e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.191+0000 7fa51fc30740 1 -- 172.21.15.12:0/448132655 shutdown_connections 2024-03-23T23:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.191+0000 7fa51fc30740 1 -- 172.21.15.12:0/448132655 wait complete. 2024-03-23T23:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.191+0000 7fa51fc30740 1 -- start start 2024-03-23T23:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.191+0000 7fa51fc30740 1 Processor -- start 2024-03-23T23:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.192+0000 7fa51fc30740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56440cb693f0 0x56440cb78380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.192+0000 7fa51fc30740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb69de0 0x56440cb788c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.192+0000 7fa51fc30740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56440cb6ef50 0x56440cb798d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.192+0000 7fa51fc30740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5181a90b0 con 0x56440cb693f0 2024-03-23T23:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.192+0000 7fa51fc30740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5181aa3b0 con 0x56440cb69de0 2024-03-23T23:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.192+0000 7fa51fc30740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5181a96f0 con 0x56440cb6ef50 2024-03-23T23:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.192+0000 7fa51fc30740 10 client.0 _ll_register_callbacks cb 0x7fa510009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.192+0000 7fa515e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56440cb693f0 0x56440cb78380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.192+0000 7fa515400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb69de0 0x56440cb788c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.192+0000 7fa515400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb69de0 0x56440cb788c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60804/0 (socket says 172.21.15.12:60804) 2024-03-23T23:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.192+0000 7fa515400640 1 -- 172.21.15.12:0/2729147534 learned_addr learned my addr 172.21.15.12:0/2729147534 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.192+0000 7fa515e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56440cb693f0 0x56440cb78380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47452/0 (socket says 172.21.15.12:47452) 2024-03-23T23:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.192+0000 7fa516800640 1 --2- 172.21.15.12:0/2729147534 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56440cb6ef50 0x56440cb798d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.192+0000 7fa515400640 1 -- 172.21.15.12:0/2729147534 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56440cb6ef50 msgr2=0x56440cb798d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.192+0000 7fa515400640 1 --2- 172.21.15.12:0/2729147534 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56440cb6ef50 0x56440cb798d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.192+0000 7fa515400640 1 -- 172.21.15.12:0/2729147534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56440cb693f0 msgr2=0x56440cb78380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.192+0000 7fa515400640 1 --2- 172.21.15.12:0/2729147534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56440cb693f0 0x56440cb78380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.192+0000 7fa515400640 1 -- 172.21.15.12:0/2729147534 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56440cb79ec0 con 0x56440cb69de0 2024-03-23T23:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.193+0000 7fa515400640 1 --2- 172.21.15.12:0/2729147534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb69de0 0x56440cb788c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa5040bd2d0 tx=0x7fa5040f44c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=338110cceee87a63 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.193+0000 7fa4f7e00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.193+0000 7fa4f7e00640 1 -- 172.21.15.12:0/2729147534 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5040d5070 con 0x56440cb69de0 2024-03-23T23:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.193+0000 7fa4f7e00640 1 -- 172.21.15.12:0/2729147534 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fa5040ee840 con 0x56440cb69de0 2024-03-23T23:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.193+0000 7fa51fc30740 1 -- 172.21.15.12:0/2729147534 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa504004c80 con 0x56440cb69de0 2024-03-23T23:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.193+0000 7fa4f7e00640 1 -- 172.21.15.12:0/2729147534 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5040bd110 con 0x56440cb69de0 2024-03-23T23:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.193+0000 7fa51fc30740 10 client.26753 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.193+0000 7fa51fc30740 1 -- 172.21.15.12:0/2729147534 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56440cbda9e0 con 0x56440cb69de0 2024-03-23T23:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.193+0000 7fa4f7e00640 1 -- 172.21.15.12:0/2729147534 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa504004c80 con 0x56440cb69de0 2024-03-23T23:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.193+0000 7fa4f7e00640 1 -- 172.21.15.12:0/2729147534 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fa5040f9040 con 0x56440cb69de0 2024-03-23T23:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.194+0000 7fa51fc30740 10 client.26753 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.194+0000 7fa51fc30740 10 client.26753 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:07:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.194+0000 7fa51fc30740 20 client.26753 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:15.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.194+0000 7fa51fc30740 1 --2- 172.21.15.12:0/2729147534 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56440cbdb320 0x56440cbdd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:15.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.194+0000 7fa51fc30740 4 client.26753 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:07:15.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.194+0000 7fa51fc30740 1 -- 172.21.15.12:0/2729147534 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56440cad60c0 con 0x56440cbdb320 2024-03-23T23:07:15.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.194+0000 7fa515e00640 1 --2- 172.21.15.12:0/2729147534 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56440cbdb320 0x56440cbdd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:15.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.194+0000 7fa515e00640 1 --2- 172.21.15.12:0/2729147534 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56440cbdb320 0x56440cbdd740 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf6cfa15b97caff8 server_cookie=d3d2a92507763d76 in_seq=0 out_seq=0 2024-03-23T23:07:15.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.194+0000 7fa4f7e00640 10 client.26753 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:15.207 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.207+0000 7fa4f7e00640 1 -- 172.21.15.12:0/2729147534 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56440cad60c0 con 0x56440cbdb320 2024-03-23T23:07:15.207 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.207+0000 7fa4f7e00640 10 client.26753 handle_command_reply: tid=0 2024-03-23T23:07:15.207 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.207+0000 7fa51fc30740 1 client.26753 shutdown 2024-03-23T23:07:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.207+0000 7fa51fc30740 1 -- 172.21.15.12:0/2729147534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb69de0 msgr2=0x56440cb788c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.207+0000 7fa51fc30740 1 --2- 172.21.15.12:0/2729147534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb69de0 0x56440cb788c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa5040bd2d0 tx=0x7fa5040f44c0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.207+0000 7fa51fc30740 1 -- 172.21.15.12:0/2729147534 shutdown_connections 2024-03-23T23:07:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.207+0000 7fa51fc30740 1 --2- 172.21.15.12:0/2729147534 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56440cbdb320 0x56440cbdd740 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.207+0000 7fa51fc30740 1 --2- 172.21.15.12:0/2729147534 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56440cb6ef50 0x56440cb798d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.207+0000 7fa51fc30740 1 --2- 172.21.15.12:0/2729147534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56440cb69de0 0x56440cb788c0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.207+0000 7fa51fc30740 1 --2- 172.21.15.12:0/2729147534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56440cb693f0 0x56440cb78380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.207+0000 7fa51fc30740 1 -- 172.21.15.12:0/2729147534 >> 172.21.15.12:0/2729147534 conn(0x56440cb64960 msgr2=0x56440cb63b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.207+0000 7fa4f7e00640 0 client.26753 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.207+0000 7fa51fc30740 1 -- 172.21.15.12:0/2729147534 shutdown_connections 2024-03-23T23:07:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.207+0000 7fa51fc30740 1 -- 172.21.15.12:0/2729147534 wait complete. 2024-03-23T23:07:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.208+0000 7fa51fc30740 20 client.26753 trim_cache size 0 max 16384 2024-03-23T23:07:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:07:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:07:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:07:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:07:15.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.208+0000 7fa517200640 1 -- 172.21.15.12:0/541754346 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa4fc0a27f0 msgr2=0x7fa4fc0a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:15.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.208+0000 7fa517200640 1 --2- 172.21.15.12:0/541754346 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa4fc0a27f0 0x7fa4fc0a4cb0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x56440cb6a640 tx=0x7fa510009600 comp rx=0 tx=0).stop 2024-03-23T23:07:15.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.208+0000 7fa517200640 1 -- 172.21.15.12:0/541754346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa518104910 msgr2=0x7fa518199f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:15.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.208+0000 7fa517200640 1 --2- 172.21.15.12:0/541754346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa518104910 0x7fa518199f50 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa504000c00 tx=0x7fa504005dd0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.208+0000 7fa517200640 1 -- 172.21.15.12:0/541754346 shutdown_connections 2024-03-23T23:07:15.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.208+0000 7fa517200640 1 --2- 172.21.15.12:0/541754346 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa4fc0a27f0 0x7fa4fc0a4cb0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.208+0000 7fa517200640 1 --2- 172.21.15.12:0/541754346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5181052d0 0x7fa5181a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.208+0000 7fa517200640 1 --2- 172.21.15.12:0/541754346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa518104910 0x7fa518199f50 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.209+0000 7fa517200640 1 --2- 172.21.15.12:0/541754346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa518103720 0x7fa518199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:15.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.209+0000 7fa517200640 1 -- 172.21.15.12:0/541754346 >> 172.21.15.12:0/541754346 conn(0x7fa5180fecb0 msgr2=0x7fa518107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:15.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.209+0000 7fa517200640 1 -- 172.21.15.12:0/541754346 shutdown_connections 2024-03-23T23:07:15.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:15.209+0000 7fa517200640 1 -- 172.21.15.12:0/541754346 wait complete. 2024-03-23T23:07:16.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:16 smithi203 ceph-mon[36693]: pgmap v554: 97 pgs: 97 active+clean; 3.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-23T23:07:17.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:16 smithi012 ceph-mon[31695]: pgmap v554: 97 pgs: 97 active+clean; 3.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-23T23:07:17.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:16 smithi173 ceph-mon[36116]: pgmap v554: 97 pgs: 97 active+clean; 3.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-23T23:07:18.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:18 smithi203 ceph-mon[36693]: pgmap v555: 97 pgs: 97 active+clean; 3.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:07:19.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:18 smithi012 ceph-mon[31695]: pgmap v555: 97 pgs: 97 active+clean; 3.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:07:19.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:18 smithi173 ceph-mon[36116]: pgmap v555: 97 pgs: 97 active+clean; 3.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:07:21.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:20 smithi012 ceph-mon[31695]: pgmap v556: 97 pgs: 97 active+clean; 3.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:07:21.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:20 smithi173 ceph-mon[36116]: pgmap v556: 97 pgs: 97 active+clean; 3.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:07:21.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:20 smithi203 ceph-mon[36693]: pgmap v556: 97 pgs: 97 active+clean; 3.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:07:23.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:22 smithi012 ceph-mon[31695]: pgmap v557: 97 pgs: 97 active+clean; 3.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:07:23.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:22 smithi173 ceph-mon[36116]: pgmap v557: 97 pgs: 97 active+clean; 3.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:07:23.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:22 smithi203 ceph-mon[36693]: pgmap v557: 97 pgs: 97 active+clean; 3.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:07:25.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:24 smithi012 ceph-mon[31695]: pgmap v558: 97 pgs: 97 active+clean; 3.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:07:25.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:24 smithi173 ceph-mon[36116]: pgmap v558: 97 pgs: 97 active+clean; 3.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:07:25.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:24 smithi203 ceph-mon[36693]: pgmap v558: 97 pgs: 97 active+clean; 3.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:07:27.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:26 smithi012 ceph-mon[31695]: pgmap v559: 97 pgs: 97 active+clean; 3.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:07:27.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:26 smithi173 ceph-mon[36116]: pgmap v559: 97 pgs: 97 active+clean; 3.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:07:27.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:26 smithi203 ceph-mon[36693]: pgmap v559: 97 pgs: 97 active+clean; 3.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:07:27.849 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:07:27.849 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:07:28.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.084+0000 7f7cbb000640 1 -- 172.21.15.12:0/753393539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cb41049b0 msgr2=0x7f7cb4104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.084+0000 7f7cbb000640 1 --2- 172.21.15.12:0/753393539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cb41049b0 0x7f7cb4104e30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0009900 tx=0x7f7ca002bc30 comp rx=0 tx=0).stop 2024-03-23T23:07:28.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.084+0000 7f7cbb000640 1 -- 172.21.15.12:0/753393539 shutdown_connections 2024-03-23T23:07:28.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.084+0000 7f7cbb000640 1 --2- 172.21.15.12:0/753393539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cb4105370 0x7f7cb410bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.084+0000 7f7cbb000640 1 --2- 172.21.15.12:0/753393539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cb41049b0 0x7f7cb4104e30 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.084+0000 7f7cbb000640 1 --2- 172.21.15.12:0/753393539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cb41037c0 0x7f7cb4103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.084+0000 7f7cbb000640 1 -- 172.21.15.12:0/753393539 >> 172.21.15.12:0/753393539 conn(0x7f7cb40fed50 msgr2=0x7f7cb4101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:28.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.085+0000 7f7cbb000640 1 -- 172.21.15.12:0/753393539 shutdown_connections 2024-03-23T23:07:28.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.085+0000 7f7cbb000640 1 -- 172.21.15.12:0/753393539 wait complete. 2024-03-23T23:07:28.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.085+0000 7f7cbb000640 1 Processor -- start 2024-03-23T23:07:28.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.086+0000 7f7cbb000640 1 -- start start 2024-03-23T23:07:28.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.086+0000 7f7cbb000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cb41037c0 0x7f7cb4199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.086+0000 7f7cbb000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cb41049b0 0x7f7cb419a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.086+0000 7f7cbb000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cb4105370 0x7f7cb41a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.086+0000 7f7cbb000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb410f490 con 0x7f7cb41037c0 2024-03-23T23:07:28.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.086+0000 7f7cbb000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb410f310 con 0x7f7cb4105370 2024-03-23T23:07:28.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.086+0000 7f7cbb000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb410f610 con 0x7f7cb41049b0 2024-03-23T23:07:28.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.086+0000 7f7cb9c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cb41037c0 0x7f7cb4199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.086+0000 7f7cb9200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cb41049b0 0x7f7cb419a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.086+0000 7f7cba600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cb4105370 0x7f7cb41a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.086+0000 7f7cb9200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cb41049b0 0x7f7cb419a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45654/0 (socket says 172.21.15.12:45654) 2024-03-23T23:07:28.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.086+0000 7f7cba600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cb4105370 0x7f7cb41a10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:43932/0 (socket says 172.21.15.12:43932) 2024-03-23T23:07:28.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.086+0000 7f7cb9200640 1 -- 172.21.15.12:0/2425657944 learned_addr learned my addr 172.21.15.12:0/2425657944 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:28.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.087+0000 7f7cb9c00640 1 -- 172.21.15.12:0/2425657944 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cb41049b0 msgr2=0x7f7cb419a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.087+0000 7f7cb9c00640 1 --2- 172.21.15.12:0/2425657944 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cb41049b0 0x7f7cb419a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.087+0000 7f7cb9c00640 1 -- 172.21.15.12:0/2425657944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cb4105370 msgr2=0x7f7cb41a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.087+0000 7f7cb9c00640 1 --2- 172.21.15.12:0/2425657944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cb4105370 0x7f7cb41a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.087+0000 7f7cb9c00640 1 -- 172.21.15.12:0/2425657944 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb41a1710 con 0x7f7cb41037c0 2024-03-23T23:07:28.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.087+0000 7f7cba600640 1 --2- 172.21.15.12:0/2425657944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cb4105370 0x7f7cb41a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:07:28.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.087+0000 7f7cb9c00640 1 --2- 172.21.15.12:0/2425657944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cb41037c0 0x7f7cb4199af0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7ca800e9c0 tx=0x7f7ca800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:28.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.087+0000 7f7cb2000640 1 -- 172.21.15.12:0/2425657944 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca800ccc0 con 0x7f7cb41037c0 2024-03-23T23:07:28.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.087+0000 7f7cb2000640 1 -- 172.21.15.12:0/2425657944 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ca80035f0 con 0x7f7cb41037c0 2024-03-23T23:07:28.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.087+0000 7f7cbb000640 1 -- 172.21.15.12:0/2425657944 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb41a1a00 con 0x7f7cb41037c0 2024-03-23T23:07:28.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.087+0000 7f7cb2000640 1 -- 172.21.15.12:0/2425657944 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca800ce60 con 0x7f7cb41037c0 2024-03-23T23:07:28.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.088+0000 7f7cbb000640 1 -- 172.21.15.12:0/2425657944 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb41a1f50 con 0x7f7cb41037c0 2024-03-23T23:07:28.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.091+0000 7f7cb2000640 1 -- 172.21.15.12:0/2425657944 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7ca80037a0 con 0x7f7cb41037c0 2024-03-23T23:07:28.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.091+0000 7f7cbf393740 1 Processor -- start 2024-03-23T23:07:28.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.091+0000 7f7cbf393740 1 -- start start 2024-03-23T23:07:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.092+0000 7f7cbf393740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e420f0050 0x561e420f0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.092+0000 7f7cbf393740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e420f0a40 0x561e420f5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.092+0000 7f7cbf393740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e420f5bb0 0x561e420f8040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.092+0000 7f7cbf393740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e42055250 con 0x561e420f0a40 2024-03-23T23:07:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.092+0000 7f7cbf393740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e41ffda30 con 0x561e420f0050 2024-03-23T23:07:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.092+0000 7f7cbf393740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e420512f0 con 0x561e420f5bb0 2024-03-23T23:07:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.092+0000 7f7cb9c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e420f0a40 0x561e420f5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.092+0000 7f7cba600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e420f0050 0x561e420f0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.092+0000 7f7cba600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e420f0050 0x561e420f0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:43946/0 (socket says 172.21.15.12:43946) 2024-03-23T23:07:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.092+0000 7f7cba600640 1 -- 172.21.15.12:0/2257994164 learned_addr learned my addr 172.21.15.12:0/2257994164 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.092+0000 7f7cb9200640 1 --2- 172.21.15.12:0/2257994164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e420f5bb0 0x561e420f8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.092+0000 7f7cba600640 1 -- 172.21.15.12:0/2257994164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e420f5bb0 msgr2=0x561e420f8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.092+0000 7f7cba600640 1 --2- 172.21.15.12:0/2257994164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e420f5bb0 0x561e420f8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.092+0000 7f7cba600640 1 -- 172.21.15.12:0/2257994164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e420f0a40 msgr2=0x561e420f5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.092+0000 7f7cba600640 1 --2- 172.21.15.12:0/2257994164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e420f0a40 0x561e420f5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.092+0000 7f7cba600640 1 -- 172.21.15.12:0/2257994164 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e41fe8690 con 0x561e420f0050 2024-03-23T23:07:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.093+0000 7f7cb2000640 1 --2- 172.21.15.12:0/2425657944 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7c9c0a2800 0x7f7c9c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.093+0000 7f7cba600640 1 --2- 172.21.15.12:0/2257994164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e420f0050 0x561e420f0470 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f7cac010750 tx=0x7f7cac010c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=907eea5e36a038ee server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:28.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.093+0000 7f7cb2000640 1 -- 172.21.15.12:0/2425657944 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7ca80cedf0 con 0x7f7cb41037c0 2024-03-23T23:07:28.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.093+0000 7f7cb9c00640 1 --2- 172.21.15.12:0/2425657944 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7c9c0a2800 0x7f7c9c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.093+0000 7f7ca6a00640 1 -- 172.21.15.12:0/2257994164 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac00ead0 con 0x561e420f0050 2024-03-23T23:07:28.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.093+0000 7f7ca6a00640 1 -- 172.21.15.12:0/2257994164 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7cac010e60 con 0x561e420f0050 2024-03-23T23:07:28.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.093+0000 7f7ca6a00640 1 -- 172.21.15.12:0/2257994164 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac011640 con 0x561e420f0050 2024-03-23T23:07:28.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.094+0000 7f7cbf393740 1 -- 172.21.15.12:0/2257994164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e420f0050 msgr2=0x561e420f0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.094+0000 7f7cbf393740 1 --2- 172.21.15.12:0/2257994164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e420f0050 0x561e420f0470 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f7cac010750 tx=0x7f7cac010c10 comp rx=0 tx=0).stop 2024-03-23T23:07:28.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.094+0000 7f7cb9c00640 1 --2- 172.21.15.12:0/2425657944 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7c9c0a2800 0x7f7c9c0a4cc0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7ca80cc8a0 tx=0x7f7ca80d2830 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:28.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.094+0000 7f7cbf393740 1 -- 172.21.15.12:0/2257994164 shutdown_connections 2024-03-23T23:07:28.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.094+0000 7f7cbf393740 1 --2- 172.21.15.12:0/2257994164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e420f5bb0 0x561e420f8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.094+0000 7f7cbf393740 1 --2- 172.21.15.12:0/2257994164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e420f0a40 0x561e420f5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.094+0000 7f7cbf393740 1 --2- 172.21.15.12:0/2257994164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e420f0050 0x561e420f0470 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.094+0000 7f7cbf393740 1 -- 172.21.15.12:0/2257994164 >> 172.21.15.12:0/2257994164 conn(0x561e420b8200 msgr2=0x561e420eb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:28.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.094+0000 7f7cbf393740 1 -- 172.21.15.12:0/2257994164 shutdown_connections 2024-03-23T23:07:28.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.094+0000 7f7cbf393740 1 -- 172.21.15.12:0/2257994164 wait complete. 2024-03-23T23:07:28.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.095+0000 7f7cbf393740 1 -- start start 2024-03-23T23:07:28.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.095+0000 7f7cbf393740 1 Processor -- start 2024-03-23T23:07:28.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.095+0000 7f7cbf393740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e420f0050 0x561e420fa030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.095+0000 7f7cbf393740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e420f0a40 0x561e420fa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.095+0000 7f7cbf393740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e420f5bb0 0x561e421015b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.095+0000 7f7cbf393740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e420f8b60 con 0x561e420f0a40 2024-03-23T23:07:28.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.095+0000 7f7cbf393740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e42010e80 con 0x561e420f5bb0 2024-03-23T23:07:28.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.095+0000 7f7cbf393740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e420f8ce0 con 0x561e420f0050 2024-03-23T23:07:28.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.095+0000 7f7cbf393740 10 client.0 _ll_register_callbacks cb 0x561e42023130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:28.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.096+0000 7f7cba600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e420f0050 0x561e420fa030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.096+0000 7f7cba600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e420f0050 0x561e420fa030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45678/0 (socket says 172.21.15.12:45678) 2024-03-23T23:07:28.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.096+0000 7f7cb9200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e420f0a40 0x561e420fa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.096+0000 7f7cba600640 1 -- 172.21.15.12:0/3300486372 learned_addr learned my addr 172.21.15.12:0/3300486372 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:28.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.096+0000 7f7cba600640 1 --2- 172.21.15.12:0/3300486372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e420f5bb0 0x561e421015b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.096+0000 7f7cb9200640 1 -- 172.21.15.12:0/3300486372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e420f0050 msgr2=0x561e420fa030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:28.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.096+0000 7f7cb9200640 1 --2- 172.21.15.12:0/3300486372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e420f0050 0x561e420fa030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.096+0000 7f7cb9200640 1 -- 172.21.15.12:0/3300486372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e420f5bb0 msgr2=0x561e421015b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:28.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.096+0000 7f7cb9200640 1 --2- 172.21.15.12:0/3300486372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e420f5bb0 0x561e421015b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.096+0000 7f7cb9200640 1 -- 172.21.15.12:0/3300486372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561e41e337d0 con 0x561e420f0a40 2024-03-23T23:07:28.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.096+0000 7f7cb9200640 1 --2- 172.21.15.12:0/3300486372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e420f0a40 0x561e420fa570 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0005ab0 tx=0x7f7ca0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da8842e965a444a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:28.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.096+0000 7f7c9b400640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:07:28.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.097+0000 7f7c9b400640 1 -- 172.21.15.12:0/3300486372 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0005d50 con 0x561e420f0a40 2024-03-23T23:07:28.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.097+0000 7f7c9b400640 1 -- 172.21.15.12:0/3300486372 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7ca00046c0 con 0x561e420f0a40 2024-03-23T23:07:28.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.097+0000 7f7cbf393740 1 -- 172.21.15.12:0/3300486372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7cac010e60 con 0x561e420f0a40 2024-03-23T23:07:28.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.097+0000 7f7c9b400640 1 -- 172.21.15.12:0/3300486372 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ca00048c0 con 0x561e420f0a40 2024-03-23T23:07:28.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.097+0000 7f7cbf393740 10 client.16638 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:28.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.097+0000 7f7cbf393740 1 -- 172.21.15.12:0/3300486372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561e41d693f0 con 0x561e420f0a40 2024-03-23T23:07:28.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.097+0000 7f7c9b400640 1 -- 172.21.15.12:0/3300486372 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7cac010e60 con 0x561e420f0a40 2024-03-23T23:07:28.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.097+0000 7f7c9b400640 1 -- 172.21.15.12:0/3300486372 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f7ca00035f0 con 0x561e420f0a40 2024-03-23T23:07:28.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.097+0000 7f7cbf393740 10 client.16638 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:28.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.097+0000 7f7cbf393740 10 client.16638 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:07:28.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.098+0000 7f7cbf393740 20 client.16638 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:28.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.098+0000 7f7cbf393740 1 --2- 172.21.15.12:0/3300486372 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561e42105ec0 0x561e421082e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.098+0000 7f7cbf393740 4 client.16638 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:07:28.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.098+0000 7f7cbf393740 1 -- 172.21.15.12:0/3300486372 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561e42062790 con 0x561e42105ec0 2024-03-23T23:07:28.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.098+0000 7f7cba600640 1 --2- 172.21.15.12:0/3300486372 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561e42105ec0 0x561e421082e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.098+0000 7f7cba600640 1 --2- 172.21.15.12:0/3300486372 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561e42105ec0 0x561e421082e0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbf389c473e82e48 server_cookie=244d90b1af775058 in_seq=0 out_seq=0 2024-03-23T23:07:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.098+0000 7f7c9b400640 10 client.16638 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.099+0000 7f7c9b400640 1 -- 172.21.15.12:0/3300486372 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561e42062790 con 0x561e42105ec0 2024-03-23T23:07:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.099+0000 7f7c9b400640 10 client.16638 handle_command_reply: tid=0 2024-03-23T23:07:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.099+0000 7f7cbf393740 1 client.16638 shutdown 2024-03-23T23:07:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.100+0000 7f7cbf393740 1 -- 172.21.15.12:0/3300486372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e420f0a40 msgr2=0x561e420fa570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.100+0000 7f7cbf393740 1 --2- 172.21.15.12:0/3300486372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e420f0a40 0x561e420fa570 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0005ab0 tx=0x7f7ca0004270 comp rx=0 tx=0).stop 2024-03-23T23:07:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.100+0000 7f7cbf393740 1 -- 172.21.15.12:0/3300486372 shutdown_connections 2024-03-23T23:07:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.100+0000 7f7cbf393740 1 --2- 172.21.15.12:0/3300486372 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561e42105ec0 0x561e421082e0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.100+0000 7f7cbf393740 1 --2- 172.21.15.12:0/3300486372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e420f5bb0 0x561e421015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.100+0000 7f7cbf393740 1 --2- 172.21.15.12:0/3300486372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e420f0a40 0x561e420fa570 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.100+0000 7f7c9b400640 0 client.16638 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.100+0000 7f7cbf393740 1 --2- 172.21.15.12:0/3300486372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e420f0050 0x561e420fa030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.100+0000 7f7cbf393740 1 -- 172.21.15.12:0/3300486372 >> 172.21.15.12:0/3300486372 conn(0x561e420b8200 msgr2=0x561e420eabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:28.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.100+0000 7f7cbf393740 1 -- 172.21.15.12:0/3300486372 shutdown_connections 2024-03-23T23:07:28.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.100+0000 7f7cbf393740 1 -- 172.21.15.12:0/3300486372 wait complete. 2024-03-23T23:07:28.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.100+0000 7f7cbf393740 20 client.16638 trim_cache size 0 max 16384 2024-03-23T23:07:28.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.118+0000 7f7cbf393740 1 Processor -- start 2024-03-23T23:07:28.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.118+0000 7f7cbf393740 1 -- start start 2024-03-23T23:07:28.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.118+0000 7f7cbf393740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e420f0050 0x561e420e6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.118+0000 7f7cbf393740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e420f0a40 0x561e42101570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.118+0000 7f7cbf393740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e420f5bb0 0x561e420f1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.118+0000 7f7cbf393740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb41ad000 con 0x561e420f0050 2024-03-23T23:07:28.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.118+0000 7f7cbf393740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb41aab00 con 0x561e420f5bb0 2024-03-23T23:07:28.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.118+0000 7f7cbf393740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb41a8ba0 con 0x561e420f0a40 2024-03-23T23:07:28.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.118+0000 7f7cba600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e420f0050 0x561e420e6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.118+0000 7f7cb9200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e420f0a40 0x561e42101570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.118+0000 7f7cba600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e420f0050 0x561e420e6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60474/0 (socket says 172.21.15.12:60474) 2024-03-23T23:07:28.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.118+0000 7f7cba600640 1 -- 172.21.15.12:0/2633113538 learned_addr learned my addr 172.21.15.12:0/2633113538 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:28.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.119+0000 7f7cba600640 1 --2- 172.21.15.12:0/2633113538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e420f5bb0 0x561e420f1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.119+0000 7f7cba600640 1 -- 172.21.15.12:0/2633113538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e420f0a40 msgr2=0x561e42101570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:28.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.119+0000 7f7cba600640 1 --2- 172.21.15.12:0/2633113538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e420f0a40 0x561e42101570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.119+0000 7f7cba600640 1 -- 172.21.15.12:0/2633113538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e420f5bb0 msgr2=0x561e420f1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:28.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.119+0000 7f7cba600640 1 --2- 172.21.15.12:0/2633113538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e420f5bb0 0x561e420f1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.119+0000 7f7cba600640 1 -- 172.21.15.12:0/2633113538 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e41d290f0 con 0x561e420f0050 2024-03-23T23:07:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.119+0000 7f7cba600640 1 --2- 172.21.15.12:0/2633113538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e420f0050 0x561e420e6920 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7cac010750 tx=0x7f7cac005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b72b870838c3602 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.119+0000 7f7ca6000640 1 -- 172.21.15.12:0/2633113538 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac025040 con 0x561e420f0050 2024-03-23T23:07:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.119+0000 7f7ca6000640 1 -- 172.21.15.12:0/2633113538 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7cac012d00 con 0x561e420f0050 2024-03-23T23:07:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.119+0000 7f7ca6000640 1 -- 172.21.15.12:0/2633113538 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac00ec50 con 0x561e420f0050 2024-03-23T23:07:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.120+0000 7f7cbf393740 1 -- 172.21.15.12:0/2633113538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e420f0050 msgr2=0x561e420e6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.120+0000 7f7cbf393740 1 --2- 172.21.15.12:0/2633113538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e420f0050 0x561e420e6920 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7cac010750 tx=0x7f7cac005a50 comp rx=0 tx=0).stop 2024-03-23T23:07:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.120+0000 7f7cbf393740 1 -- 172.21.15.12:0/2633113538 shutdown_connections 2024-03-23T23:07:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.120+0000 7f7cbf393740 1 --2- 172.21.15.12:0/2633113538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e420f5bb0 0x561e420f1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.120+0000 7f7cbf393740 1 --2- 172.21.15.12:0/2633113538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e420f0a40 0x561e42101570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.120+0000 7f7cbf393740 1 --2- 172.21.15.12:0/2633113538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e420f0050 0x561e420e6920 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.120+0000 7f7cbf393740 1 -- 172.21.15.12:0/2633113538 >> 172.21.15.12:0/2633113538 conn(0x561e420b8200 msgr2=0x561e420f14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.120+0000 7f7cbf393740 1 -- 172.21.15.12:0/2633113538 shutdown_connections 2024-03-23T23:07:28.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.120+0000 7f7cbf393740 1 -- 172.21.15.12:0/2633113538 wait complete. 2024-03-23T23:07:28.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.120+0000 7f7cbf393740 1 -- start start 2024-03-23T23:07:28.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.121+0000 7f7cbf393740 1 Processor -- start 2024-03-23T23:07:28.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.121+0000 7f7cbf393740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e420f0050 0x561e420ff0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.121+0000 7f7cbf393740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e420f0a40 0x561e420ff600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.121+0000 7f7cbf393740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e420f5bb0 0x561e42100610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.121+0000 7f7cbf393740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb41a91a0 con 0x561e420f0050 2024-03-23T23:07:28.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.121+0000 7f7cbf393740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb41aa4a0 con 0x561e420f0a40 2024-03-23T23:07:28.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.121+0000 7f7cbf393740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb41a97e0 con 0x561e420f5bb0 2024-03-23T23:07:28.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.121+0000 7f7cbf393740 10 client.0 _ll_register_callbacks cb 0x7f7ca002c5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:28.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.121+0000 7f7cb9200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e420f0a40 0x561e420ff600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.121+0000 7f7cba600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e420f0050 0x561e420ff0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.121+0000 7f7cb9200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e420f0a40 0x561e420ff600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:43976/0 (socket says 172.21.15.12:43976) 2024-03-23T23:07:28.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.121+0000 7f7cb9200640 1 -- 172.21.15.12:0/3253578214 learned_addr learned my addr 172.21.15.12:0/3253578214 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:28.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.122+0000 7f7cba600640 1 --2- 172.21.15.12:0/3253578214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e420f5bb0 0x561e42100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.122+0000 7f7cb9200640 1 -- 172.21.15.12:0/3253578214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e420f5bb0 msgr2=0x561e42100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:28.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.122+0000 7f7cb9200640 1 --2- 172.21.15.12:0/3253578214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e420f5bb0 0x561e42100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.122+0000 7f7cb9200640 1 -- 172.21.15.12:0/3253578214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e420f0050 msgr2=0x561e420ff0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:28.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.122+0000 7f7cb9200640 1 --2- 172.21.15.12:0/3253578214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e420f0050 0x561e420ff0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.122+0000 7f7cb9200640 1 -- 172.21.15.12:0/3253578214 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561e42100c00 con 0x561e420f0a40 2024-03-23T23:07:28.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.122+0000 7f7cba600640 1 --2- 172.21.15.12:0/3253578214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e420f0050 0x561e420ff0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:07:28.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.122+0000 7f7cb9200640 1 --2- 172.21.15.12:0/3253578214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e420f0a40 0x561e420ff600 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0009850 tx=0x7f7ca0003e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3731725a4209cc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:28.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.122+0000 7f7c9b400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:07:28.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.122+0000 7f7c9b400640 1 -- 172.21.15.12:0/3253578214 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0005d50 con 0x561e420f0a40 2024-03-23T23:07:28.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.122+0000 7f7cbf393740 1 -- 172.21.15.12:0/3253578214 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7cac012d00 con 0x561e420f0a40 2024-03-23T23:07:28.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.122+0000 7f7c9b400640 1 -- 172.21.15.12:0/3253578214 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7ca0004c70 con 0x561e420f0a40 2024-03-23T23:07:28.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.123+0000 7f7c9b400640 1 -- 172.21.15.12:0/3253578214 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ca0009520 con 0x561e420f0a40 2024-03-23T23:07:28.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.123+0000 7f7cbf393740 10 client.26771 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:28.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.123+0000 7f7cbf393740 1 -- 172.21.15.12:0/3253578214 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561e42161bd0 con 0x561e420f0a40 2024-03-23T23:07:28.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.123+0000 7f7c9b400640 1 -- 172.21.15.12:0/3253578214 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7cac012d00 con 0x561e420f0a40 2024-03-23T23:07:28.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.123+0000 7f7c9b400640 1 -- 172.21.15.12:0/3253578214 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f7ca0066030 con 0x561e420f0a40 2024-03-23T23:07:28.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.123+0000 7f7cbf393740 10 client.26771 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:28.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.123+0000 7f7cbf393740 10 client.26771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:07:28.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.123+0000 7f7cbf393740 20 client.26771 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:28.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.123+0000 7f7cbf393740 1 --2- 172.21.15.12:0/3253578214 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561e421624c0 0x561e421648e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.123+0000 7f7cbf393740 4 client.26771 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:07:28.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.123+0000 7f7cbf393740 1 -- 172.21.15.12:0/3253578214 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561e4205cae0 con 0x561e421624c0 2024-03-23T23:07:28.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.124+0000 7f7cba600640 1 --2- 172.21.15.12:0/3253578214 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561e421624c0 0x561e421648e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.124+0000 7f7cba600640 1 --2- 172.21.15.12:0/3253578214 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561e421624c0 0x561e421648e0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbe5bd1c8031a160 server_cookie=9dff59ba56bccd05 in_seq=0 out_seq=0 2024-03-23T23:07:28.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.124+0000 7f7c9b400640 10 client.26771 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:28.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.124+0000 7f7c9b400640 1 -- 172.21.15.12:0/3253578214 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x561e4205cae0 con 0x561e421624c0 2024-03-23T23:07:28.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.124+0000 7f7c9b400640 10 client.26771 handle_command_reply: tid=0 2024-03-23T23:07:28.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.124+0000 7f7cbf393740 1 client.26771 shutdown 2024-03-23T23:07:28.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.124+0000 7f7cbf393740 1 -- 172.21.15.12:0/3253578214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e420f0a40 msgr2=0x561e420ff600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.124+0000 7f7cbf393740 1 --2- 172.21.15.12:0/3253578214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e420f0a40 0x561e420ff600 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0009850 tx=0x7f7ca0003e00 comp rx=0 tx=0).stop 2024-03-23T23:07:28.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.124+0000 7f7cbf393740 1 -- 172.21.15.12:0/3253578214 shutdown_connections 2024-03-23T23:07:28.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.124+0000 7f7cbf393740 1 --2- 172.21.15.12:0/3253578214 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561e421624c0 0x561e421648e0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.124+0000 7f7cbf393740 1 --2- 172.21.15.12:0/3253578214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e420f5bb0 0x561e42100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.124+0000 7f7cbf393740 1 --2- 172.21.15.12:0/3253578214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e420f0a40 0x561e420ff600 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.124+0000 7f7c9b400640 0 client.26771 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:28.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.124+0000 7f7cbf393740 1 --2- 172.21.15.12:0/3253578214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e420f0050 0x561e420ff0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.124+0000 7f7cbf393740 1 -- 172.21.15.12:0/3253578214 >> 172.21.15.12:0/3253578214 conn(0x561e420b8200 msgr2=0x561e420eabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:28.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.125+0000 7f7cbf393740 1 -- 172.21.15.12:0/3253578214 shutdown_connections 2024-03-23T23:07:28.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.125+0000 7f7cbf393740 1 -- 172.21.15.12:0/3253578214 wait complete. 2024-03-23T23:07:28.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.125+0000 7f7cbf393740 20 client.26771 trim_cache size 0 max 16384 2024-03-23T23:07:28.131 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:07:28.131 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-03-23T23:07:28.131 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-03-23T23:07:28.131 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:07:28.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.125+0000 7f7cbb000640 1 -- 172.21.15.12:0/2425657944 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7c9c0a2800 msgr2=0x7f7c9c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.125+0000 7f7cbb000640 1 --2- 172.21.15.12:0/2425657944 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7c9c0a2800 0x7f7c9c0a4cc0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7ca80cc8a0 tx=0x7f7ca80d2830 comp rx=0 tx=0).stop 2024-03-23T23:07:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.125+0000 7f7cbb000640 1 -- 172.21.15.12:0/2425657944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cb41037c0 msgr2=0x7f7cb4199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.125+0000 7f7cbb000640 1 --2- 172.21.15.12:0/2425657944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cb41037c0 0x7f7cb4199af0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7ca800e9c0 tx=0x7f7ca800ee80 comp rx=0 tx=0).stop 2024-03-23T23:07:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.125+0000 7f7cbb000640 1 -- 172.21.15.12:0/2425657944 shutdown_connections 2024-03-23T23:07:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.125+0000 7f7cbb000640 1 --2- 172.21.15.12:0/2425657944 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7c9c0a2800 0x7f7c9c0a4cc0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.125+0000 7f7cbb000640 1 --2- 172.21.15.12:0/2425657944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cb4105370 0x7f7cb41a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.125+0000 7f7cbb000640 1 --2- 172.21.15.12:0/2425657944 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cb41049b0 0x7f7cb419a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.125+0000 7f7cbb000640 1 --2- 172.21.15.12:0/2425657944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cb41037c0 0x7f7cb4199af0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.125+0000 7f7cbb000640 1 -- 172.21.15.12:0/2425657944 >> 172.21.15.12:0/2425657944 conn(0x7f7cb40fed50 msgr2=0x7f7cb4107c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.126+0000 7f7cbb000640 1 -- 172.21.15.12:0/2425657944 shutdown_connections 2024-03-23T23:07:28.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.126+0000 7f7cbb000640 1 -- 172.21.15.12:0/2425657944 wait complete. 2024-03-23T23:07:28.145 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T23:07:28.145 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:07:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.353+0000 7fe6bcc00640 1 -- 172.21.15.12:0/63850893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6b8102160 msgr2=0x7fe6b8104550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.353+0000 7fe6bcc00640 1 --2- 172.21.15.12:0/63850893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6b8102160 0x7fe6b8104550 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe6a8009900 tx=0x7fe6a802bc30 comp rx=0 tx=0).stop 2024-03-23T23:07:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.353+0000 7fe6bcc00640 1 -- 172.21.15.12:0/63850893 shutdown_connections 2024-03-23T23:07:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.353+0000 7fe6bcc00640 1 --2- 172.21.15.12:0/63850893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6b81073c0 0x7fe6b8109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.353+0000 7fe6bcc00640 1 --2- 172.21.15.12:0/63850893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6b8104a90 0x7fe6b8106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.353+0000 7fe6bcc00640 1 --2- 172.21.15.12:0/63850893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6b8102160 0x7fe6b8104550 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.353+0000 7fe6bcc00640 1 -- 172.21.15.12:0/63850893 >> 172.21.15.12:0/63850893 conn(0x7fe6b80fbb50 msgr2=0x7fe6b80fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.353+0000 7fe6bcc00640 1 -- 172.21.15.12:0/63850893 shutdown_connections 2024-03-23T23:07:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.353+0000 7fe6bcc00640 1 -- 172.21.15.12:0/63850893 wait complete. 2024-03-23T23:07:28.354 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.354+0000 7fe6bcc00640 1 Processor -- start 2024-03-23T23:07:28.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.355+0000 7fe6bcc00640 1 -- start start 2024-03-23T23:07:28.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.355+0000 7fe6bcc00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6b8102160 0x7fe6b8199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.355+0000 7fe6bcc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6b8104a90 0x7fe6b819a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.355+0000 7fe6bcc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6b81073c0 0x7fe6b81a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.355+0000 7fe6bcc00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b810d3d0 con 0x7fe6b8102160 2024-03-23T23:07:28.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.355+0000 7fe6bcc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b810d250 con 0x7fe6b8104a90 2024-03-23T23:07:28.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.355+0000 7fe6bcc00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b810d550 con 0x7fe6b81073c0 2024-03-23T23:07:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.355+0000 7fe6b5400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6b8104a90 0x7fe6b819a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.355+0000 7fe6b5e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6b8102160 0x7fe6b8199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.355+0000 7fe6b6800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6b81073c0 0x7fe6b81a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.355+0000 7fe6b5400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6b8104a90 0x7fe6b819a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:43982/0 (socket says 172.21.15.12:43982) 2024-03-23T23:07:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.355+0000 7fe6b5400640 1 -- 172.21.15.12:0/2242131114 learned_addr learned my addr 172.21.15.12:0/2242131114 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.356+0000 7fe6b5400640 1 -- 172.21.15.12:0/2242131114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6b81073c0 msgr2=0x7fe6b81a1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.356+0000 7fe6b5400640 1 --2- 172.21.15.12:0/2242131114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6b81073c0 0x7fe6b81a1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.356+0000 7fe6b5400640 1 -- 172.21.15.12:0/2242131114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6b8102160 msgr2=0x7fe6b8199b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.356+0000 7fe6b5400640 1 --2- 172.21.15.12:0/2242131114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6b8102160 0x7fe6b8199b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.356+0000 7fe6b5400640 1 -- 172.21.15.12:0/2242131114 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b81a1780 con 0x7fe6b8104a90 2024-03-23T23:07:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.356+0000 7fe6b5400640 1 --2- 172.21.15.12:0/2242131114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6b8104a90 0x7fe6b819a0a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe6a000e9c0 tx=0x7fe6a000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.357+0000 7fe6aea00640 1 -- 172.21.15.12:0/2242131114 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a000cd80 con 0x7fe6b8104a90 2024-03-23T23:07:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.357+0000 7fe6aea00640 1 -- 172.21.15.12:0/2242131114 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6a0005500 con 0x7fe6b8104a90 2024-03-23T23:07:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.357+0000 7fe6bcc00640 1 -- 172.21.15.12:0/2242131114 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b81a1a70 con 0x7fe6b8104a90 2024-03-23T23:07:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.357+0000 7fe6bcc00640 1 -- 172.21.15.12:0/2242131114 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b81a2010 con 0x7fe6b8104a90 2024-03-23T23:07:28.358 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.357+0000 7fe6aea00640 1 -- 172.21.15.12:0/2242131114 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a0010690 con 0x7fe6b8104a90 2024-03-23T23:07:28.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.360+0000 7f14e9800640 1 -- 172.21.15.12:0/3122430735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14e4075660 msgr2=0x7f14e4073ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.360+0000 7f14e9800640 1 --2- 172.21.15.12:0/3122430735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14e4075660 0x7f14e4073ab0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f14d4009900 tx=0x7f14d402bcc0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.360+0000 7f14e9800640 1 -- 172.21.15.12:0/3122430735 shutdown_connections 2024-03-23T23:07:28.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.360+0000 7f14e9800640 1 --2- 172.21.15.12:0/3122430735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e41093b0 0x7f14e410b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.360+0000 7f14e9800640 1 --2- 172.21.15.12:0/3122430735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14e4074080 0x7f14e40744e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.360+0000 7f14e9800640 1 --2- 172.21.15.12:0/3122430735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14e4075660 0x7f14e4073ab0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.360+0000 7f14e9800640 1 -- 172.21.15.12:0/3122430735 >> 172.21.15.12:0/3122430735 conn(0x7f14e40fca70 msgr2=0x7f14e40feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:28.360 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.360+0000 7fe6aea00640 1 -- 172.21.15.12:0/2242131114 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe6a0005020 con 0x7fe6b8104a90 2024-03-23T23:07:28.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.360+0000 7f14e9800640 1 -- 172.21.15.12:0/3122430735 shutdown_connections 2024-03-23T23:07:28.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.360+0000 7f14e9800640 1 -- 172.21.15.12:0/3122430735 wait complete. 2024-03-23T23:07:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.361+0000 7fe6bfa0c740 1 Processor -- start 2024-03-23T23:07:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.361+0000 7fe6aea00640 1 --2- 172.21.15.12:0/2242131114 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe69c0a2920 0x7fe69c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.362+0000 7f14e9800640 1 Processor -- start 2024-03-23T23:07:28.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.362+0000 7fe6bfa0c740 1 -- start start 2024-03-23T23:07:28.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.362+0000 7fe6aea00640 1 -- 172.21.15.12:0/2242131114 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe6a00cebf0 con 0x7fe6b8104a90 2024-03-23T23:07:28.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.362+0000 7f14e9800640 1 -- start start 2024-03-23T23:07:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.362+0000 7fe6b6800640 1 --2- 172.21.15.12:0/2242131114 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe69c0a2920 0x7fe69c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.362+0000 7fe6bfa0c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563cc9d141e0 0x563cc9d14600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.362+0000 7fe6bfa0c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cc9d14bd0 0x563cc9d196d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.362+0000 7fe6bfa0c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563cc9d19d40 0x563cc9d1c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.362+0000 7fe6bfa0c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cc9cd73b0 con 0x563cc9d141e0 2024-03-23T23:07:28.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.362+0000 7fe6bfa0c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cc9c79830 con 0x563cc9d14bd0 2024-03-23T23:07:28.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.362+0000 7fe6bfa0c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cc9c225d0 con 0x563cc9d19d40 2024-03-23T23:07:28.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.362+0000 7f14e9800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e4074080 0x7f14e4199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.363+0000 7fe6b5e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563cc9d141e0 0x563cc9d14600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.363+0000 7fe6b5e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563cc9d141e0 0x563cc9d14600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60526/0 (socket says 172.21.15.12:60526) 2024-03-23T23:07:28.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.363+0000 7fe6b5e00640 1 -- 172.21.15.12:0/956203993 learned_addr learned my addr 172.21.15.12:0/956203993 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:28.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.363+0000 7fe6b5400640 1 --2- 172.21.15.12:0/956203993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cc9d14bd0 0x563cc9d196d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.362+0000 7f14e9800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14e4075660 0x7f14e4199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.362+0000 7f14e9800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14e41093b0 0x7f14e41a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.362+0000 7f14e9800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e410f3b0 con 0x7f14e4074080 2024-03-23T23:07:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.362+0000 7f14e9800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e410f230 con 0x7f14e4075660 2024-03-23T23:07:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.362+0000 7f14e9800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e410f530 con 0x7f14e41093b0 2024-03-23T23:07:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.363+0000 7f14e3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14e4075660 0x7f14e4199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.363+0000 7f14e3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14e4075660 0x7f14e4199f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44014/0 (socket says 172.21.15.12:44014) 2024-03-23T23:07:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.363+0000 7f14e3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e4074080 0x7f14e4199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.363+0000 7f14e3400640 1 -- 172.21.15.12:0/1135864511 learned_addr learned my addr 172.21.15.12:0/1135864511 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.363+0000 7f14e8e00640 1 --2- 172.21.15.12:0/1135864511 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14e41093b0 0x7f14e41a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.363+0000 7fe6b6800640 1 --2- 172.21.15.12:0/2242131114 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe69c0a2920 0x7fe69c0a4de0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0005fd0 tx=0x7fe6b0011810 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:28.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.363+0000 7fe6b6800640 1 --2- 172.21.15.12:0/956203993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563cc9d19d40 0x563cc9d1c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.363+0000 7f14e3400640 1 -- 172.21.15.12:0/1135864511 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14e41093b0 msgr2=0x7f14e41a0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.363+0000 7f14e3400640 1 --2- 172.21.15.12:0/1135864511 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14e41093b0 0x7f14e41a0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.363+0000 7fe6b5400640 1 -- 172.21.15.12:0/956203993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563cc9d19d40 msgr2=0x563cc9d1c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:28.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.363+0000 7fe6b5400640 1 --2- 172.21.15.12:0/956203993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563cc9d19d40 0x563cc9d1c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.363+0000 7fe6b5400640 1 -- 172.21.15.12:0/956203993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563cc9d141e0 msgr2=0x563cc9d14600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:28.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.363+0000 7fe6b5400640 1 --2- 172.21.15.12:0/956203993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563cc9d141e0 0x563cc9d14600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.363+0000 7fe6b5400640 1 -- 172.21.15.12:0/956203993 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563cc9c3a2c0 con 0x563cc9d14bd0 2024-03-23T23:07:28.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.363+0000 7f14e3400640 1 -- 172.21.15.12:0/1135864511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e4074080 msgr2=0x7f14e4199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.363+0000 7f14e3400640 1 --2- 172.21.15.12:0/1135864511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e4074080 0x7f14e4199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.363+0000 7f14e3400640 1 -- 172.21.15.12:0/1135864511 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e41a1620 con 0x7f14e4075660 2024-03-23T23:07:28.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.363+0000 7f14e3e00640 1 --2- 172.21.15.12:0/1135864511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e4074080 0x7f14e4199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:07:28.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.363+0000 7fe6b5e00640 1 --2- 172.21.15.12:0/956203993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563cc9d141e0 0x563cc9d14600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:07:28.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.363+0000 7fe6b6800640 1 --2- 172.21.15.12:0/956203993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563cc9d19d40 0x563cc9d1c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:07:28.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.363+0000 7fe6b5400640 1 --2- 172.21.15.12:0/956203993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cc9d14bd0 0x563cc9d196d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0004c40 tx=0x7fe6a0097900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=787ba21f97b4bf6f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:28.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.364+0000 7f14e3400640 1 --2- 172.21.15.12:0/1135864511 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14e4075660 0x7f14e4199f40 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f14d000b2f0 tx=0x7f14d000b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:28.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.364+0000 7fe6a7000640 1 -- 172.21.15.12:0/956203993 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a00b3020 con 0x563cc9d14bd0 2024-03-23T23:07:28.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.364+0000 7fe6a7000640 1 -- 172.21.15.12:0/956203993 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6a009bca0 con 0x563cc9d14bd0 2024-03-23T23:07:28.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.364+0000 7fe6a7000640 1 -- 172.21.15.12:0/956203993 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a0099930 con 0x563cc9d14bd0 2024-03-23T23:07:28.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.364+0000 7f14e0c00640 1 -- 172.21.15.12:0/1135864511 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d0022070 con 0x7f14e4075660 2024-03-23T23:07:28.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.364+0000 7f14e0c00640 1 -- 172.21.15.12:0/1135864511 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14d001ba00 con 0x7f14e4075660 2024-03-23T23:07:28.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.364+0000 7f14e0c00640 1 -- 172.21.15.12:0/1135864511 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d001dba0 con 0x7f14e4075660 2024-03-23T23:07:28.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.364+0000 7f14e9800640 1 -- 172.21.15.12:0/1135864511 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e4100f90 con 0x7f14e4075660 2024-03-23T23:07:28.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.364+0000 7f14e9800640 1 -- 172.21.15.12:0/1135864511 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e4101490 con 0x7f14e4075660 2024-03-23T23:07:28.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.364+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/956203993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cc9d14bd0 msgr2=0x563cc9d196d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.364+0000 7fe6bfa0c740 1 --2- 172.21.15.12:0/956203993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cc9d14bd0 0x563cc9d196d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0004c40 tx=0x7fe6a0097900 comp rx=0 tx=0).stop 2024-03-23T23:07:28.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.365+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/956203993 shutdown_connections 2024-03-23T23:07:28.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.365+0000 7fe6bfa0c740 1 --2- 172.21.15.12:0/956203993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563cc9d19d40 0x563cc9d1c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.365+0000 7fe6bfa0c740 1 --2- 172.21.15.12:0/956203993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cc9d14bd0 0x563cc9d196d0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.365+0000 7fe6bfa0c740 1 --2- 172.21.15.12:0/956203993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563cc9d141e0 0x563cc9d14600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.365+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/956203993 >> 172.21.15.12:0/956203993 conn(0x563cc9d0f750 msgr2=0x563cc9d0fb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:28.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.365+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/956203993 shutdown_connections 2024-03-23T23:07:28.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.365+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/956203993 wait complete. 2024-03-23T23:07:28.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.365+0000 7f14e9800640 1 -- 172.21.15.12:0/1135864511 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e4108170 con 0x7f14e4075660 2024-03-23T23:07:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.365+0000 7fe6bfa0c740 1 -- start start 2024-03-23T23:07:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.366+0000 7fe6bfa0c740 1 Processor -- start 2024-03-23T23:07:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.366+0000 7fe6bfa0c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563cc9d141e0 0x563cc9d26b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.366+0000 7fe6bfa0c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cc9d14bd0 0x563cc9d270b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.366+0000 7fe6bfa0c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563cc9d19d40 0x563cc9d2e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.366+0000 7fe6bfa0c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cc9c35a20 con 0x563cc9d141e0 2024-03-23T23:07:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.366+0000 7fe6bfa0c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cc9c758d0 con 0x563cc9d14bd0 2024-03-23T23:07:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.366+0000 7fe6bfa0c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cc9d1ccf0 con 0x563cc9d19d40 2024-03-23T23:07:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.366+0000 7fe6bfa0c740 10 client.0 _ll_register_callbacks cb 0x563cc9c721c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.366+0000 7fe6b5400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cc9d14bd0 0x563cc9d270b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.366+0000 7fe6b5e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563cc9d141e0 0x563cc9d26b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.366+0000 7fe6b6800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563cc9d19d40 0x563cc9d2e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.366+0000 7fe6b5400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cc9d14bd0 0x563cc9d270b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44024/0 (socket says 172.21.15.12:44024) 2024-03-23T23:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.366+0000 7fe6b5400640 1 -- 172.21.15.12:0/1511252222 learned_addr learned my addr 172.21.15.12:0/1511252222 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.367+0000 7fe6b6800640 1 -- 172.21.15.12:0/1511252222 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cc9d14bd0 msgr2=0x563cc9d270b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.367+0000 7fe6b6800640 1 --2- 172.21.15.12:0/1511252222 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cc9d14bd0 0x563cc9d270b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.367+0000 7fe6b6800640 1 -- 172.21.15.12:0/1511252222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563cc9d141e0 msgr2=0x563cc9d26b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.367+0000 7fe6b6800640 1 --2- 172.21.15.12:0/1511252222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563cc9d141e0 0x563cc9d26b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.367+0000 7fe6b6800640 1 -- 172.21.15.12:0/1511252222 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563cc998e210 con 0x563cc9d19d40 2024-03-23T23:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.367+0000 7fe6b5400640 1 --2- 172.21.15.12:0/1511252222 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cc9d14bd0 0x563cc9d270b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.367+0000 7fe6b5e00640 1 --2- 172.21.15.12:0/1511252222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563cc9d141e0 0x563cc9d26b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.367+0000 7fe6b6800640 1 --2- 172.21.15.12:0/1511252222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563cc9d19d40 0x563cc9d2e0f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe6b000fc50 tx=0x7fe6b0004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b427879e4dd9ee1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.367+0000 7fe69b400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.367+0000 7fe69b400640 1 -- 172.21.15.12:0/1511252222 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0017030 con 0x563cc9d19d40 2024-03-23T23:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.368+0000 7fe69b400640 1 -- 172.21.15.12:0/1511252222 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe6b0018070 con 0x563cc9d19d40 2024-03-23T23:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.368+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/1511252222 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe6a009bca0 con 0x563cc9d19d40 2024-03-23T23:07:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.368+0000 7fe69b400640 1 -- 172.21.15.12:0/1511252222 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6b001de90 con 0x563cc9d19d40 2024-03-23T23:07:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.368+0000 7fe6bfa0c740 10 client.26758 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.368+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/1511252222 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563cc9a5a620 con 0x563cc9d19d40 2024-03-23T23:07:28.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.368+0000 7f14e0c00640 1 -- 172.21.15.12:0/1135864511 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f14d0025070 con 0x7f14e4075660 2024-03-23T23:07:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.368+0000 7fe69b400640 1 -- 172.21.15.12:0/1511252222 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6a009bca0 con 0x563cc9d19d40 2024-03-23T23:07:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.368+0000 7fe69b400640 1 -- 172.21.15.12:0/1511252222 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe6b00265b0 con 0x563cc9d19d40 2024-03-23T23:07:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.368+0000 7fe6bfa0c740 10 client.26758 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.368+0000 7fe6bfa0c740 10 client.26758 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:07:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.368+0000 7fe6bfa0c740 20 client.26758 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.368+0000 7fe6bfa0c740 1 --2- 172.21.15.12:0/1511252222 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563cc9d32a50 0x563cc9d34ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.368+0000 7fe6bfa0c740 4 client.26758 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:07:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.368+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/1511252222 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563cc9c86d70 con 0x563cc9d32a50 2024-03-23T23:07:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.369+0000 7fe6b5e00640 1 --2- 172.21.15.12:0/1511252222 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563cc9d32a50 0x563cc9d34ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.369+0000 7fe6b5e00640 1 --2- 172.21.15.12:0/1511252222 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563cc9d32a50 0x563cc9d34ec0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c58761495ff5036 server_cookie=5bf2f5c7e801c8ba in_seq=0 out_seq=0 2024-03-23T23:07:28.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.369+0000 7fe69b400640 10 client.26758 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.370+0000 7f14e0c00640 1 --2- 172.21.15.12:0/1135864511 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f14c80a2880 0x7f14c80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.370+0000 7f14e0c00640 1 -- 172.21.15.12:0/1135864511 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f14d00d6650 con 0x7f14e4075660 2024-03-23T23:07:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.370+0000 7f14e3e00640 1 --2- 172.21.15.12:0/1135864511 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f14c80a2880 0x7f14c80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.370+0000 7fe69b400640 1 -- 172.21.15.12:0/1511252222 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563cc9c86d70 con 0x563cc9d32a50 2024-03-23T23:07:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.370+0000 7fe69b400640 10 client.26758 handle_command_reply: tid=0 2024-03-23T23:07:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.370+0000 7fe6bfa0c740 1 client.26758 shutdown 2024-03-23T23:07:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.371+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/1511252222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563cc9d19d40 msgr2=0x563cc9d2e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.371+0000 7fe6bfa0c740 1 --2- 172.21.15.12:0/1511252222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563cc9d19d40 0x563cc9d2e0f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe6b000fc50 tx=0x7fe6b0004ab0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.371+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/1511252222 shutdown_connections 2024-03-23T23:07:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.371+0000 7fe6bfa0c740 1 --2- 172.21.15.12:0/1511252222 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563cc9d32a50 0x563cc9d34ec0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.371+0000 7fe6bfa0c740 1 --2- 172.21.15.12:0/1511252222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563cc9d19d40 0x563cc9d2e0f0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.371+0000 7fe6bfa0c740 1 --2- 172.21.15.12:0/1511252222 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cc9d14bd0 0x563cc9d270b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.371+0000 7fe6bfa0c740 1 --2- 172.21.15.12:0/1511252222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563cc9d141e0 0x563cc9d26b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.371+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/1511252222 >> 172.21.15.12:0/1511252222 conn(0x563cc9d0f750 msgr2=0x563cc9d0e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:28.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.371+0000 7fe69b400640 0 client.26758 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:28.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.371+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/1511252222 shutdown_connections 2024-03-23T23:07:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.371+0000 7f14e3e00640 1 --2- 172.21.15.12:0/1135864511 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f14c80a2880 0x7f14c80a4d40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f14d402c2d0 tx=0x7f14d4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:28.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.371+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/1511252222 wait complete. 2024-03-23T23:07:28.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.371+0000 7fe6bfa0c740 20 client.26758 trim_cache size 0 max 16384 2024-03-23T23:07:28.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.374+0000 7f14e0c00640 1 -- 172.21.15.12:0/1135864511 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f14d00a3050 con 0x7f14e4075660 2024-03-23T23:07:28.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.387+0000 7fe6bfa0c740 1 Processor -- start 2024-03-23T23:07:28.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.387+0000 7fe6bfa0c740 1 -- start start 2024-03-23T23:07:28.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.387+0000 7fe6bfa0c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563cc9d141e0 0x563cc9d26b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.387+0000 7fe6bfa0c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563cc9d14bd0 0x563cc9d270b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.387+0000 7fe6bfa0c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cc9d19d40 0x563cc9d2e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.387+0000 7fe6bfa0c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6940107f0 con 0x563cc9d14bd0 2024-03-23T23:07:28.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.387+0000 7fe6bfa0c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69400e2f0 con 0x563cc9d19d40 2024-03-23T23:07:28.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.387+0000 7fe6bfa0c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69400c390 con 0x563cc9d141e0 2024-03-23T23:07:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.388+0000 7fe6b6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cc9d19d40 0x563cc9d2e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.388+0000 7fe6b5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563cc9d141e0 0x563cc9d26b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.388+0000 7fe6b5400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563cc9d14bd0 0x563cc9d270b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.388+0000 7fe6b5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563cc9d141e0 0x563cc9d26b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45758/0 (socket says 172.21.15.12:45758) 2024-03-23T23:07:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.388+0000 7fe6b5e00640 1 -- 172.21.15.12:0/1328372404 learned_addr learned my addr 172.21.15.12:0/1328372404 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.388+0000 7fe6b5e00640 1 -- 172.21.15.12:0/1328372404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cc9d19d40 msgr2=0x563cc9d2e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.388+0000 7fe6b5e00640 1 --2- 172.21.15.12:0/1328372404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cc9d19d40 0x563cc9d2e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.388+0000 7fe6b5e00640 1 -- 172.21.15.12:0/1328372404 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563cc9d14bd0 msgr2=0x563cc9d270b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.388+0000 7fe6b5e00640 1 --2- 172.21.15.12:0/1328372404 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563cc9d14bd0 0x563cc9d270b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.388+0000 7fe6b5e00640 1 -- 172.21.15.12:0/1328372404 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563cc9d101a0 con 0x563cc9d141e0 2024-03-23T23:07:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.388+0000 7fe6b5e00640 1 --2- 172.21.15.12:0/1328372404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563cc9d141e0 0x563cc9d26b70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe6a8009230 tx=0x7fe6a8009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1e1498b13c51d03 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.389+0000 7fe6a6600640 1 -- 172.21.15.12:0/1328372404 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a8005d50 con 0x563cc9d141e0 2024-03-23T23:07:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.389+0000 7fe6a6600640 1 -- 172.21.15.12:0/1328372404 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6a804e070 con 0x563cc9d141e0 2024-03-23T23:07:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.389+0000 7fe6a6600640 1 -- 172.21.15.12:0/1328372404 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a8005110 con 0x563cc9d141e0 2024-03-23T23:07:28.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.389+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/1328372404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563cc9d141e0 msgr2=0x563cc9d26b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.389+0000 7fe6bfa0c740 1 --2- 172.21.15.12:0/1328372404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563cc9d141e0 0x563cc9d26b70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe6a8009230 tx=0x7fe6a8009260 comp rx=0 tx=0).stop 2024-03-23T23:07:28.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.389+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/1328372404 shutdown_connections 2024-03-23T23:07:28.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.389+0000 7fe6bfa0c740 1 --2- 172.21.15.12:0/1328372404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cc9d19d40 0x563cc9d2e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.389+0000 7fe6bfa0c740 1 --2- 172.21.15.12:0/1328372404 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563cc9d14bd0 0x563cc9d270b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.389+0000 7fe6bfa0c740 1 --2- 172.21.15.12:0/1328372404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563cc9d141e0 0x563cc9d26b70 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.389+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/1328372404 >> 172.21.15.12:0/1328372404 conn(0x563cc9d0f750 msgr2=0x563cc9d0a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:28.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.390+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/1328372404 shutdown_connections 2024-03-23T23:07:28.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.390+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/1328372404 wait complete. 2024-03-23T23:07:28.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.390+0000 7fe6bfa0c740 1 -- start start 2024-03-23T23:07:28.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.390+0000 7fe6bfa0c740 1 Processor -- start 2024-03-23T23:07:28.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.391+0000 7fe6bfa0c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563cc9d141e0 0x563cc9d2b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.391+0000 7fe6bfa0c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563cc9d14bd0 0x563cc9d2bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.391+0000 7fe6bfa0c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cc9d19d40 0x563cc9d2ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.391+0000 7fe6bfa0c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69400c990 con 0x563cc9d14bd0 2024-03-23T23:07:28.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.391+0000 7fe6bfa0c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69400dc90 con 0x563cc9d19d40 2024-03-23T23:07:28.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.391+0000 7fe6bfa0c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69400cfd0 con 0x563cc9d141e0 2024-03-23T23:07:28.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.391+0000 7fe6bfa0c740 10 client.0 _ll_register_callbacks cb 0x7fe6b000f3c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:28.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.391+0000 7fe6b5400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563cc9d14bd0 0x563cc9d2bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.391+0000 7fe6b6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cc9d19d40 0x563cc9d2ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.391+0000 7fe6b5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563cc9d141e0 0x563cc9d2b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.391+0000 7fe6b6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cc9d19d40 0x563cc9d2ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44040/0 (socket says 172.21.15.12:44040) 2024-03-23T23:07:28.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.391+0000 7fe6b6800640 1 -- 172.21.15.12:0/112946650 learned_addr learned my addr 172.21.15.12:0/112946650 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:28.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.391+0000 7fe6b5400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563cc9d14bd0 0x563cc9d2bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60572/0 (socket says 172.21.15.12:60572) 2024-03-23T23:07:28.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.391+0000 7fe6b5e00640 1 -- 172.21.15.12:0/112946650 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cc9d19d40 msgr2=0x563cc9d2ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:28.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.391+0000 7fe6b5e00640 1 --2- 172.21.15.12:0/112946650 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cc9d19d40 0x563cc9d2ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.391+0000 7fe6b5e00640 1 -- 172.21.15.12:0/112946650 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563cc9d14bd0 msgr2=0x563cc9d2bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:28.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.391+0000 7fe6b5e00640 1 --2- 172.21.15.12:0/112946650 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563cc9d14bd0 0x563cc9d2bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.391+0000 7fe6b5e00640 1 -- 172.21.15.12:0/112946650 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563cc9d2d060 con 0x563cc9d141e0 2024-03-23T23:07:28.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.391+0000 7fe6b6800640 1 --2- 172.21.15.12:0/112946650 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cc9d19d40 0x563cc9d2ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:07:28.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.392+0000 7fe6b5e00640 1 --2- 172.21.15.12:0/112946650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563cc9d141e0 0x563cc9d2b590 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe6a802c170 tx=0x7fe6a8004e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c97ee699aeb2940b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:28.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.392+0000 7fe69b400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:07:28.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.392+0000 7fe69b400640 1 -- 172.21.15.12:0/112946650 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a8004020 con 0x563cc9d141e0 2024-03-23T23:07:28.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.392+0000 7fe69b400640 1 -- 172.21.15.12:0/112946650 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe6a80041c0 con 0x563cc9d141e0 2024-03-23T23:07:28.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.392+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/112946650 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe6a804e070 con 0x563cc9d141e0 2024-03-23T23:07:28.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.392+0000 7fe69b400640 1 -- 172.21.15.12:0/112946650 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6a80043c0 con 0x563cc9d141e0 2024-03-23T23:07:28.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.392+0000 7fe6bfa0c740 10 client.26764 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:28.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.392+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/112946650 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563cc9d2e370 con 0x563cc9d141e0 2024-03-23T23:07:28.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.392+0000 7fe69b400640 1 -- 172.21.15.12:0/112946650 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6a804e070 con 0x563cc9d141e0 2024-03-23T23:07:28.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.393+0000 7fe69b400640 1 -- 172.21.15.12:0/112946650 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe6a804a970 con 0x563cc9d141e0 2024-03-23T23:07:28.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.393+0000 7fe6bfa0c740 10 client.26764 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:28.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.393+0000 7fe6bfa0c740 10 client.26764 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:07:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.393+0000 7fe6bfa0c740 20 client.26764 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.393+0000 7fe6bfa0c740 1 --2- 172.21.15.12:0/112946650 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563cc9d8ea30 0x563cc9d90e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.393+0000 7fe6bfa0c740 4 client.26764 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:07:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.393+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/112946650 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563cc9c810c0 con 0x563cc9d8ea30 2024-03-23T23:07:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.393+0000 7fe6b5400640 1 --2- 172.21.15.12:0/112946650 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563cc9d8ea30 0x563cc9d90e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.393+0000 7fe6b5400640 1 --2- 172.21.15.12:0/112946650 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563cc9d8ea30 0x563cc9d90e50 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46197d70ce902285 server_cookie=c3bafff6450c385e in_seq=0 out_seq=0 2024-03-23T23:07:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.393+0000 7fe69b400640 10 client.26764 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.406+0000 7fe69b400640 1 -- 172.21.15.12:0/112946650 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563cc9c810c0 con 0x563cc9d8ea30 2024-03-23T23:07:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.406+0000 7fe69b400640 10 client.26764 handle_command_reply: tid=0 2024-03-23T23:07:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.406+0000 7fe6bfa0c740 1 client.26764 shutdown 2024-03-23T23:07:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.407+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/112946650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563cc9d141e0 msgr2=0x563cc9d2b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.407+0000 7fe6bfa0c740 1 --2- 172.21.15.12:0/112946650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563cc9d141e0 0x563cc9d2b590 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe6a802c170 tx=0x7fe6a8004e00 comp rx=0 tx=0).stop 2024-03-23T23:07:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.407+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/112946650 shutdown_connections 2024-03-23T23:07:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.407+0000 7fe6bfa0c740 1 --2- 172.21.15.12:0/112946650 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563cc9d8ea30 0x563cc9d90e50 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.407+0000 7fe6bfa0c740 1 --2- 172.21.15.12:0/112946650 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cc9d19d40 0x563cc9d2ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.407+0000 7fe6bfa0c740 1 --2- 172.21.15.12:0/112946650 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563cc9d14bd0 0x563cc9d2bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.407+0000 7fe6bfa0c740 1 --2- 172.21.15.12:0/112946650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563cc9d141e0 0x563cc9d2b590 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.407+0000 7fe69b400640 0 client.26764 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.407+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/112946650 >> 172.21.15.12:0/112946650 conn(0x563cc9d0f750 msgr2=0x563cc9d190b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.407+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/112946650 shutdown_connections 2024-03-23T23:07:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.407+0000 7fe6bfa0c740 1 -- 172.21.15.12:0/112946650 wait complete. 2024-03-23T23:07:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.407+0000 7fe6bfa0c740 20 client.26764 trim_cache size 0 max 16384 2024-03-23T23:07:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:07:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:07:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:07:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:07:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.408+0000 7fe6bcc00640 1 -- 172.21.15.12:0/2242131114 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe69c0a2920 msgr2=0x7fe69c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.408+0000 7fe6bcc00640 1 --2- 172.21.15.12:0/2242131114 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe69c0a2920 0x7fe69c0a4de0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0005fd0 tx=0x7fe6b0011810 comp rx=0 tx=0).stop 2024-03-23T23:07:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.408+0000 7fe6bcc00640 1 -- 172.21.15.12:0/2242131114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6b8104a90 msgr2=0x7fe6b819a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.408+0000 7fe6bcc00640 1 --2- 172.21.15.12:0/2242131114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6b8104a90 0x7fe6b819a0a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe6a000e9c0 tx=0x7fe6a000ee80 comp rx=0 tx=0).stop 2024-03-23T23:07:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.408+0000 7fe6bcc00640 1 -- 172.21.15.12:0/2242131114 shutdown_connections 2024-03-23T23:07:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.408+0000 7fe6bcc00640 1 --2- 172.21.15.12:0/2242131114 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe69c0a2920 0x7fe69c0a4de0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.408+0000 7fe6bcc00640 1 --2- 172.21.15.12:0/2242131114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6b81073c0 0x7fe6b81a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.408+0000 7fe6bcc00640 1 --2- 172.21.15.12:0/2242131114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6b8104a90 0x7fe6b819a0a0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.408+0000 7fe6bcc00640 1 --2- 172.21.15.12:0/2242131114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6b8102160 0x7fe6b8199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.408+0000 7fe6bcc00640 1 -- 172.21.15.12:0/2242131114 >> 172.21.15.12:0/2242131114 conn(0x7fe6b80fbb50 msgr2=0x7fe6b81017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.408+0000 7fe6bcc00640 1 -- 172.21.15.12:0/2242131114 shutdown_connections 2024-03-23T23:07:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:28.408+0000 7fe6bcc00640 1 -- 172.21.15.12:0/2242131114 wait complete. 2024-03-23T23:07:28.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.626+0000 7f14e9800640 1 -- 172.21.15.12:0/1135864511 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14e402d070 con 0x7f14e4075660 2024-03-23T23:07:28.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.627+0000 7f14e0c00640 1 -- 172.21.15.12:0/1135864511 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f14d00a70f0 con 0x7f14e4075660 2024-03-23T23:07:28.628 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:07:28.628 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:07:28.629 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:07:28.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.630+0000 7f14e9800640 1 -- 172.21.15.12:0/1135864511 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f14c80a2880 msgr2=0x7f14c80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.630+0000 7f14e9800640 1 --2- 172.21.15.12:0/1135864511 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f14c80a2880 0x7f14c80a4d40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f14d402c2d0 tx=0x7f14d4005fd0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.630+0000 7f14e9800640 1 -- 172.21.15.12:0/1135864511 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14e4075660 msgr2=0x7f14e4199f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.630+0000 7f14e9800640 1 --2- 172.21.15.12:0/1135864511 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14e4075660 0x7f14e4199f40 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f14d000b2f0 tx=0x7f14d000b7b0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.630+0000 7f14e9800640 1 -- 172.21.15.12:0/1135864511 shutdown_connections 2024-03-23T23:07:28.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.630+0000 7f14e9800640 1 --2- 172.21.15.12:0/1135864511 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f14c80a2880 0x7f14c80a4d40 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.630+0000 7f14e9800640 1 --2- 172.21.15.12:0/1135864511 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14e41093b0 0x7f14e41a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.630+0000 7f14e9800640 1 --2- 172.21.15.12:0/1135864511 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14e4075660 0x7f14e4199f40 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.630+0000 7f14e9800640 1 --2- 172.21.15.12:0/1135864511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e4074080 0x7f14e4199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.630+0000 7f14e9800640 1 -- 172.21.15.12:0/1135864511 >> 172.21.15.12:0/1135864511 conn(0x7f14e40fca70 msgr2=0x7f14e4105340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:28.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.630+0000 7f14e9800640 1 -- 172.21.15.12:0/1135864511 shutdown_connections 2024-03-23T23:07:28.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.631+0000 7f14e9800640 1 -- 172.21.15.12:0/1135864511 wait complete. 2024-03-23T23:07:28.652 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:07:28.652 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-23T23:07:28.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.881+0000 7fb71a800640 1 -- 172.21.15.12:0/851105555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb714102000 msgr2=0x7fb7141043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.881+0000 7fb71a800640 1 --2- 172.21.15.12:0/851105555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb714102000 0x7fb7141043f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb704009900 tx=0x7fb70402bc30 comp rx=0 tx=0).stop 2024-03-23T23:07:28.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.881+0000 7fb71a800640 1 -- 172.21.15.12:0/851105555 shutdown_connections 2024-03-23T23:07:28.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.881+0000 7fb71a800640 1 --2- 172.21.15.12:0/851105555 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb714107260 0x7fb7141096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.881+0000 7fb71a800640 1 --2- 172.21.15.12:0/851105555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb714104930 0x7fb714106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.881+0000 7fb71a800640 1 --2- 172.21.15.12:0/851105555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb714102000 0x7fb7141043f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.881+0000 7fb71a800640 1 -- 172.21.15.12:0/851105555 >> 172.21.15.12:0/851105555 conn(0x7fb7140fba10 msgr2=0x7fb7140fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:28.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.881+0000 7fb71a800640 1 -- 172.21.15.12:0/851105555 shutdown_connections 2024-03-23T23:07:28.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.881+0000 7fb71a800640 1 -- 172.21.15.12:0/851105555 wait complete. 2024-03-23T23:07:28.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.882+0000 7fb71a800640 1 Processor -- start 2024-03-23T23:07:28.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.883+0000 7fb71a800640 1 -- start start 2024-03-23T23:07:28.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.883+0000 7fb71a800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb714102000 0x7fb714199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.883+0000 7fb71a800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb714104930 0x7fb714199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.883+0000 7fb71a800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb714107260 0x7fb7141a1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.883+0000 7fb71a800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71410d260 con 0x7fb714104930 2024-03-23T23:07:28.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.883+0000 7fb71a800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71410d0e0 con 0x7fb714107260 2024-03-23T23:07:28.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.883+0000 7fb71a800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71410d3e0 con 0x7fb714102000 2024-03-23T23:07:28.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.883+0000 7fb718a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb714104930 0x7fb714199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.883+0000 7fb719400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb714102000 0x7fb714199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.883+0000 7fb719400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb714102000 0x7fb714199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45786/0 (socket says 172.21.15.12:45786) 2024-03-23T23:07:28.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.883+0000 7fb719400640 1 -- 172.21.15.12:0/1805908354 learned_addr learned my addr 172.21.15.12:0/1805908354 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:28.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.883+0000 7fb719e00640 1 --2- 172.21.15.12:0/1805908354 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb714107260 0x7fb7141a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.884+0000 7fb719400640 1 -- 172.21.15.12:0/1805908354 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb714107260 msgr2=0x7fb7141a1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.884+0000 7fb719400640 1 --2- 172.21.15.12:0/1805908354 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb714107260 0x7fb7141a1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.884+0000 7fb719400640 1 -- 172.21.15.12:0/1805908354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb714104930 msgr2=0x7fb714199fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.884+0000 7fb719400640 1 --2- 172.21.15.12:0/1805908354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb714104930 0x7fb714199fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.884+0000 7fb719400640 1 -- 172.21.15.12:0/1805908354 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7141a16b0 con 0x7fb714102000 2024-03-23T23:07:28.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.884+0000 7fb719400640 1 --2- 172.21.15.12:0/1805908354 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb714102000 0x7fb714199a90 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb704005ab0 tx=0x7fb70402c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:28.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.884+0000 7fb712000640 1 -- 172.21.15.12:0/1805908354 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb704043020 con 0x7fb714102000 2024-03-23T23:07:28.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.884+0000 7fb712000640 1 -- 172.21.15.12:0/1805908354 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7040054e0 con 0x7fb714102000 2024-03-23T23:07:28.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.884+0000 7fb712000640 1 -- 172.21.15.12:0/1805908354 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb704003ce0 con 0x7fb714102000 2024-03-23T23:07:28.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.884+0000 7fb71a800640 1 -- 172.21.15.12:0/1805908354 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7141a1940 con 0x7fb714102000 2024-03-23T23:07:28.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.885+0000 7fb71a800640 1 -- 172.21.15.12:0/1805908354 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7141a1ee0 con 0x7fb714102000 2024-03-23T23:07:28.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.888+0000 7fb712000640 1 -- 172.21.15.12:0/1805908354 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb704046020 con 0x7fb714102000 2024-03-23T23:07:28.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.888+0000 7fb71ea74740 1 Processor -- start 2024-03-23T23:07:28.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.888+0000 7fb71ea74740 1 -- start start 2024-03-23T23:07:28.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.888+0000 7fb71ea74740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5610fcf129f0 0x5610fcf12e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.888+0000 7fb71ea74740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610fcf13bc0 0x5610fcf14040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.888+0000 7fb71ea74740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610fcf14580 0x5610fcf1ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.888+0000 7fb71ea74740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610fce772c0 con 0x5610fcf129f0 2024-03-23T23:07:28.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.888+0000 7fb71ea74740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610fce1fc90 con 0x5610fcf13bc0 2024-03-23T23:07:28.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.888+0000 7fb71ea74740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610fce73360 con 0x5610fcf14580 2024-03-23T23:07:28.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.889+0000 7fb719e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5610fcf129f0 0x5610fcf12e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.889+0000 7fb718a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610fcf14580 0x5610fcf1ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.889+0000 7fb719e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5610fcf129f0 0x5610fcf12e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60590/0 (socket says 172.21.15.12:60590) 2024-03-23T23:07:28.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.889+0000 7fb719e00640 1 -- 172.21.15.12:0/2598547463 learned_addr learned my addr 172.21.15.12:0/2598547463 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:28.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.889+0000 7fb719400640 1 --2- 172.21.15.12:0/2598547463 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610fcf13bc0 0x5610fcf14040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.889+0000 7fb718a00640 1 -- 172.21.15.12:0/2598547463 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610fcf13bc0 msgr2=0x5610fcf14040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:28.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.889+0000 7fb718a00640 1 --2- 172.21.15.12:0/2598547463 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610fcf13bc0 0x5610fcf14040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.889+0000 7fb718a00640 1 -- 172.21.15.12:0/2598547463 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5610fcf129f0 msgr2=0x5610fcf12e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:28.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.889+0000 7fb718a00640 1 --2- 172.21.15.12:0/2598547463 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5610fcf129f0 0x5610fcf12e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.889+0000 7fb718a00640 1 -- 172.21.15.12:0/2598547463 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610fce44080 con 0x5610fcf14580 2024-03-23T23:07:28.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.889+0000 7fb718a00640 1 --2- 172.21.15.12:0/2598547463 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610fcf14580 0x5610fcf1ae10 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb714069360 tx=0x7fb700003e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=644778c18f759b8a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:28.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.890+0000 7fb70a600640 1 -- 172.21.15.12:0/2598547463 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb700011440 con 0x5610fcf14580 2024-03-23T23:07:28.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.890+0000 7fb70a600640 1 -- 172.21.15.12:0/2598547463 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb700015030 con 0x5610fcf14580 2024-03-23T23:07:28.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.890+0000 7fb70a600640 1 -- 172.21.15.12:0/2598547463 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb700010ab0 con 0x5610fcf14580 2024-03-23T23:07:28.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.890+0000 7fb712000640 1 --2- 172.21.15.12:0/1805908354 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb6f80a2880 0x7fb6f80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.890+0000 7fb712000640 1 -- 172.21.15.12:0/1805908354 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb7040f0b00 con 0x7fb714102000 2024-03-23T23:07:28.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.890+0000 7fb719e00640 1 --2- 172.21.15.12:0/1805908354 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb6f80a2880 0x7fb6f80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.890+0000 7fb71ea74740 1 -- 172.21.15.12:0/2598547463 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610fcf14580 msgr2=0x5610fcf1ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.890+0000 7fb71ea74740 1 --2- 172.21.15.12:0/2598547463 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610fcf14580 0x5610fcf1ae10 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb714069360 tx=0x7fb700003e80 comp rx=0 tx=0).stop 2024-03-23T23:07:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.891+0000 7fb71ea74740 1 -- 172.21.15.12:0/2598547463 shutdown_connections 2024-03-23T23:07:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.891+0000 7fb71ea74740 1 --2- 172.21.15.12:0/2598547463 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610fcf14580 0x5610fcf1ae10 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.891+0000 7fb71ea74740 1 --2- 172.21.15.12:0/2598547463 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610fcf13bc0 0x5610fcf14040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.891+0000 7fb71ea74740 1 --2- 172.21.15.12:0/2598547463 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5610fcf129f0 0x5610fcf12e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.891+0000 7fb71ea74740 1 -- 172.21.15.12:0/2598547463 >> 172.21.15.12:0/2598547463 conn(0x5610fcf0dde0 msgr2=0x5610fcf0e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.891+0000 7fb71ea74740 1 -- 172.21.15.12:0/2598547463 shutdown_connections 2024-03-23T23:07:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.891+0000 7fb71ea74740 1 -- 172.21.15.12:0/2598547463 wait complete. 2024-03-23T23:07:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.891+0000 7fb719e00640 1 --2- 172.21.15.12:0/1805908354 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb6f80a2880 0x7fb6f80a4d40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x5610fcf13ac0 tx=0x7fb70c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.891+0000 7fb71ea74740 1 -- start start 2024-03-23T23:07:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.891+0000 7fb71ea74740 1 Processor -- start 2024-03-23T23:07:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.892+0000 7fb71ea74740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610fcf129f0 0x5610fcf1cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.892+0000 7fb71ea74740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610fcf13bc0 0x5610fcf1d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.892+0000 7fb71ea74740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5610fcf14580 0x5610fcf241c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.892+0000 7fb71ea74740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610fcf1b590 con 0x5610fcf14580 2024-03-23T23:07:28.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.892+0000 7fb71ea74740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610fce330d0 con 0x5610fcf13bc0 2024-03-23T23:07:28.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.892+0000 7fb71ea74740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610fcf1b710 con 0x5610fcf129f0 2024-03-23T23:07:28.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.892+0000 7fb71ea74740 10 client.0 _ll_register_callbacks cb 0x5610fce37e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:28.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.892+0000 7fb718a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610fcf13bc0 0x5610fcf1d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.892+0000 7fb719e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5610fcf14580 0x5610fcf241c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.892+0000 7fb719400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610fcf129f0 0x5610fcf1cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.892+0000 7fb718a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610fcf13bc0 0x5610fcf1d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44082/0 (socket says 172.21.15.12:44082) 2024-03-23T23:07:28.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.892+0000 7fb719e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5610fcf14580 0x5610fcf241c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60592/0 (socket says 172.21.15.12:60592) 2024-03-23T23:07:28.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.892+0000 7fb718a00640 1 -- 172.21.15.12:0/3525127545 learned_addr learned my addr 172.21.15.12:0/3525127545 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:28.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.892+0000 7fb718a00640 1 -- 172.21.15.12:0/3525127545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610fcf129f0 msgr2=0x5610fcf1cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:28.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.893+0000 7fb718a00640 1 --2- 172.21.15.12:0/3525127545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610fcf129f0 0x5610fcf1cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.893+0000 7fb718a00640 1 -- 172.21.15.12:0/3525127545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5610fcf14580 msgr2=0x5610fcf241c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:28.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.893+0000 7fb718a00640 1 --2- 172.21.15.12:0/3525127545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5610fcf14580 0x5610fcf241c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.893+0000 7fb718a00640 1 -- 172.21.15.12:0/3525127545 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5610fcb8b3f0 con 0x5610fcf13bc0 2024-03-23T23:07:28.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.893+0000 7fb719400640 1 --2- 172.21.15.12:0/3525127545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610fcf129f0 0x5610fcf1cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:07:28.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.893+0000 7fb718a00640 1 --2- 172.21.15.12:0/3525127545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610fcf13bc0 0x5610fcf1d180 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fb700000f80 tx=0x7fb7000104d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf3a4244b6c39a5a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:28.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.893+0000 7fb6fea00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:07:28.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.893+0000 7fb6fea00640 1 -- 172.21.15.12:0/3525127545 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb700029060 con 0x5610fcf13bc0 2024-03-23T23:07:28.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.893+0000 7fb6fea00640 1 -- 172.21.15.12:0/3525127545 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb700010a00 con 0x5610fcf13bc0 2024-03-23T23:07:28.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.893+0000 7fb71ea74740 1 -- 172.21.15.12:0/3525127545 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb700015030 con 0x5610fcf13bc0 2024-03-23T23:07:28.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.894+0000 7fb6fea00640 1 -- 172.21.15.12:0/3525127545 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb700010c00 con 0x5610fcf13bc0 2024-03-23T23:07:28.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.894+0000 7fb71ea74740 10 client.26807 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:28.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.894+0000 7fb71ea74740 1 -- 172.21.15.12:0/3525127545 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5610fcc557d0 con 0x5610fcf13bc0 2024-03-23T23:07:28.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.894+0000 7fb6fea00640 1 -- 172.21.15.12:0/3525127545 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb700015030 con 0x5610fcf13bc0 2024-03-23T23:07:28.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.894+0000 7fb6fea00640 1 -- 172.21.15.12:0/3525127545 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fb700023070 con 0x5610fcf13bc0 2024-03-23T23:07:28.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.894+0000 7fb71ea74740 10 client.26807 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:28.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.894+0000 7fb71ea74740 10 client.26807 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:07:28.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.894+0000 7fb71ea74740 20 client.26807 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:28.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.894+0000 7fb71ea74740 1 --2- 172.21.15.12:0/3525127545 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5610fcf283d0 0x5610fcf2a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.894+0000 7fb71ea74740 4 client.26807 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:07:28.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.894+0000 7fb71ea74740 1 -- 172.21.15.12:0/3525127545 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5610fce84800 con 0x5610fcf283d0 2024-03-23T23:07:28.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.894+0000 7fb719400640 1 --2- 172.21.15.12:0/3525127545 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5610fcf283d0 0x5610fcf2a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.895+0000 7fb719400640 1 --2- 172.21.15.12:0/3525127545 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5610fcf283d0 0x5610fcf2a7c0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1787f97e61fa33db server_cookie=bddb48ee97b83206 in_seq=0 out_seq=0 2024-03-23T23:07:28.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.895+0000 7fb6fea00640 10 client.26807 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:07:28.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.896+0000 7fb6fea00640 1 -- 172.21.15.12:0/3525127545 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5610fce84800 con 0x5610fcf283d0 2024-03-23T23:07:28.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.896+0000 7fb6fea00640 10 client.26807 handle_command_reply: tid=0 2024-03-23T23:07:28.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.896+0000 7fb71ea74740 1 client.26807 shutdown 2024-03-23T23:07:28.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.896+0000 7fb71ea74740 1 -- 172.21.15.12:0/3525127545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610fcf13bc0 msgr2=0x5610fcf1d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.896+0000 7fb71ea74740 1 --2- 172.21.15.12:0/3525127545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610fcf13bc0 0x5610fcf1d180 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fb700000f80 tx=0x7fb7000104d0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.896+0000 7fb71ea74740 1 -- 172.21.15.12:0/3525127545 shutdown_connections 2024-03-23T23:07:28.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.896+0000 7fb71ea74740 1 --2- 172.21.15.12:0/3525127545 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5610fcf283d0 0x5610fcf2a7c0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.896+0000 7fb71ea74740 1 --2- 172.21.15.12:0/3525127545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5610fcf14580 0x5610fcf241c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.896+0000 7fb71ea74740 1 --2- 172.21.15.12:0/3525127545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610fcf13bc0 0x5610fcf1d180 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.896+0000 7fb6fea00640 0 client.26807 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:07:28.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.896+0000 7fb71ea74740 1 --2- 172.21.15.12:0/3525127545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610fcf129f0 0x5610fcf1cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.896+0000 7fb71ea74740 1 -- 172.21.15.12:0/3525127545 >> 172.21.15.12:0/3525127545 conn(0x5610fcf0dde0 msgr2=0x5610fcf0d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:28.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.896+0000 7fb71ea74740 1 -- 172.21.15.12:0/3525127545 shutdown_connections 2024-03-23T23:07:28.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.896+0000 7fb71ea74740 1 -- 172.21.15.12:0/3525127545 wait complete. 2024-03-23T23:07:28.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.896+0000 7fb71ea74740 20 client.26807 trim_cache size 0 max 16384 2024-03-23T23:07:28.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.910+0000 7fb71ea74740 1 Processor -- start 2024-03-23T23:07:28.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.910+0000 7fb71ea74740 1 -- start start 2024-03-23T23:07:28.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.910+0000 7fb71ea74740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610fcf129f0 0x5610fcf0cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.910+0000 7fb71ea74740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610fcf13bc0 0x5610fcf24180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.910+0000 7fb71ea74740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5610fcf14580 0x5610fcf2efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.910+0000 7fb71ea74740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7141acf10 con 0x5610fcf14580 2024-03-23T23:07:28.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.910+0000 7fb71ea74740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7141aaa10 con 0x5610fcf13bc0 2024-03-23T23:07:28.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.910+0000 7fb71ea74740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7141a8ab0 con 0x5610fcf129f0 2024-03-23T23:07:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.911+0000 7fb719400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610fcf129f0 0x5610fcf0cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.911+0000 7fb719e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5610fcf14580 0x5610fcf2efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.911+0000 7fb719400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610fcf129f0 0x5610fcf0cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45808/0 (socket says 172.21.15.12:45808) 2024-03-23T23:07:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.911+0000 7fb719400640 1 -- 172.21.15.12:0/180886118 learned_addr learned my addr 172.21.15.12:0/180886118 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.911+0000 7fb718a00640 1 --2- 172.21.15.12:0/180886118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610fcf13bc0 0x5610fcf24180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.911+0000 7fb719400640 1 -- 172.21.15.12:0/180886118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610fcf13bc0 msgr2=0x5610fcf24180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.911+0000 7fb719400640 1 --2- 172.21.15.12:0/180886118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610fcf13bc0 0x5610fcf24180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.911+0000 7fb719400640 1 -- 172.21.15.12:0/180886118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5610fcf14580 msgr2=0x5610fcf2efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.911+0000 7fb719400640 1 --2- 172.21.15.12:0/180886118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5610fcf14580 0x5610fcf2efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.911+0000 7fb719400640 1 -- 172.21.15.12:0/180886118 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610fcf1dc30 con 0x5610fcf129f0 2024-03-23T23:07:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.911+0000 7fb719400640 1 --2- 172.21.15.12:0/180886118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610fcf129f0 0x5610fcf0cfe0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fb7040ee5b0 tx=0x7fb7040dd040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e82698225b052017 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.911+0000 7fb709c00640 1 -- 172.21.15.12:0/180886118 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7040bcf90 con 0x5610fcf129f0 2024-03-23T23:07:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.912+0000 7fb709c00640 1 -- 172.21.15.12:0/180886118 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb704004c80 con 0x5610fcf129f0 2024-03-23T23:07:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.912+0000 7fb709c00640 1 -- 172.21.15.12:0/180886118 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7040bcdb0 con 0x5610fcf129f0 2024-03-23T23:07:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.912+0000 7fb71ea74740 1 -- 172.21.15.12:0/180886118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610fcf129f0 msgr2=0x5610fcf0cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.912+0000 7fb71ea74740 1 --2- 172.21.15.12:0/180886118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610fcf129f0 0x5610fcf0cfe0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fb7040ee5b0 tx=0x7fb7040dd040 comp rx=0 tx=0).stop 2024-03-23T23:07:28.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.912+0000 7fb71ea74740 1 -- 172.21.15.12:0/180886118 shutdown_connections 2024-03-23T23:07:28.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.912+0000 7fb71ea74740 1 --2- 172.21.15.12:0/180886118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5610fcf14580 0x5610fcf2efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.912+0000 7fb71ea74740 1 --2- 172.21.15.12:0/180886118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610fcf13bc0 0x5610fcf24180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.912+0000 7fb71ea74740 1 --2- 172.21.15.12:0/180886118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610fcf129f0 0x5610fcf0cfe0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.912+0000 7fb71ea74740 1 -- 172.21.15.12:0/180886118 >> 172.21.15.12:0/180886118 conn(0x5610fcf0dde0 msgr2=0x5610fcf0c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:28.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.912+0000 7fb71ea74740 1 -- 172.21.15.12:0/180886118 shutdown_connections 2024-03-23T23:07:28.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.912+0000 7fb71ea74740 1 -- 172.21.15.12:0/180886118 wait complete. 2024-03-23T23:07:28.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.913+0000 7fb71ea74740 1 -- start start 2024-03-23T23:07:28.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.913+0000 7fb71ea74740 1 Processor -- start 2024-03-23T23:07:28.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.913+0000 7fb71ea74740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5610fcf129f0 0x5610fcf24060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.913+0000 7fb71ea74740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610fcf13bc0 0x5610fcf22670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.913+0000 7fb71ea74740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610fcf14580 0x5610fcf22bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.913+0000 7fb71ea74740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7141a90b0 con 0x5610fcf129f0 2024-03-23T23:07:28.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.913+0000 7fb71ea74740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7141aa3b0 con 0x5610fcf13bc0 2024-03-23T23:07:28.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.913+0000 7fb71ea74740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7141a96f0 con 0x5610fcf14580 2024-03-23T23:07:28.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.913+0000 7fb71ea74740 10 client.0 _ll_register_callbacks cb 0x7fb70c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:28.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.913+0000 7fb719400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5610fcf129f0 0x5610fcf24060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.914+0000 7fb719400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5610fcf129f0 0x5610fcf24060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60616/0 (socket says 172.21.15.12:60616) 2024-03-23T23:07:28.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.914+0000 7fb719400640 1 -- 172.21.15.12:0/3385372047 learned_addr learned my addr 172.21.15.12:0/3385372047 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:28.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.914+0000 7fb718a00640 1 --2- 172.21.15.12:0/3385372047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610fcf13bc0 0x5610fcf22670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.914+0000 7fb719e00640 1 --2- 172.21.15.12:0/3385372047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610fcf14580 0x5610fcf22bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.914+0000 7fb719400640 1 -- 172.21.15.12:0/3385372047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610fcf14580 msgr2=0x5610fcf22bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:28.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.914+0000 7fb719400640 1 --2- 172.21.15.12:0/3385372047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610fcf14580 0x5610fcf22bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.914+0000 7fb719400640 1 -- 172.21.15.12:0/3385372047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610fcf13bc0 msgr2=0x5610fcf22670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:28.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.914+0000 7fb719400640 1 --2- 172.21.15.12:0/3385372047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610fcf13bc0 0x5610fcf22670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.914+0000 7fb719400640 1 -- 172.21.15.12:0/3385372047 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5610fcf233e0 con 0x5610fcf129f0 2024-03-23T23:07:28.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.914+0000 7fb719400640 1 --2- 172.21.15.12:0/3385372047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5610fcf129f0 0x5610fcf24060 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb704000c00 tx=0x7fb7040bce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=517a61be2d65d518 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:28.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.914+0000 7fb6fea00640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:07:28.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.915+0000 7fb6fea00640 1 -- 172.21.15.12:0/3385372047 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb704005690 con 0x5610fcf129f0 2024-03-23T23:07:28.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.915+0000 7fb6fea00640 1 -- 172.21.15.12:0/3385372047 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb7040035f0 con 0x5610fcf129f0 2024-03-23T23:07:28.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.915+0000 7fb71ea74740 1 -- 172.21.15.12:0/3385372047 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb704004c80 con 0x5610fcf129f0 2024-03-23T23:07:28.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.915+0000 7fb6fea00640 1 -- 172.21.15.12:0/3385372047 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7040bdb50 con 0x5610fcf129f0 2024-03-23T23:07:28.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.915+0000 7fb71ea74740 10 client.16686 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:28.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.915+0000 7fb71ea74740 1 -- 172.21.15.12:0/3385372047 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5610fcf84bd0 con 0x5610fcf129f0 2024-03-23T23:07:28.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.915+0000 7fb6fea00640 1 -- 172.21.15.12:0/3385372047 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb704004c80 con 0x5610fcf129f0 2024-03-23T23:07:28.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.915+0000 7fb6fea00640 1 -- 172.21.15.12:0/3385372047 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fb7040c5760 con 0x5610fcf129f0 2024-03-23T23:07:28.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.915+0000 7fb71ea74740 10 client.16686 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:28.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.915+0000 7fb71ea74740 10 client.16686 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:07:28.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.915+0000 7fb71ea74740 20 client.16686 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:28.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.915+0000 7fb71ea74740 1 --2- 172.21.15.12:0/3385372047 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5610fcf85460 0x5610fcf87880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:28.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.915+0000 7fb71ea74740 4 client.16686 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:07:28.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.915+0000 7fb71ea74740 1 -- 172.21.15.12:0/3385372047 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5610fcf1ca30 con 0x5610fcf85460 2024-03-23T23:07:28.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.916+0000 7fb718a00640 1 --2- 172.21.15.12:0/3385372047 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5610fcf85460 0x5610fcf87880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:28.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.916+0000 7fb718a00640 1 --2- 172.21.15.12:0/3385372047 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5610fcf85460 0x5610fcf87880 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d7f4315de1dccd2d server_cookie=d90e008e685883d3 in_seq=0 out_seq=0 2024-03-23T23:07:28.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.916+0000 7fb6fea00640 10 client.16686 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:07:28.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.916+0000 7fb6fea00640 1 -- 172.21.15.12:0/3385372047 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5610fcf1ca30 con 0x5610fcf85460 2024-03-23T23:07:28.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.916+0000 7fb6fea00640 10 client.16686 handle_command_reply: tid=0 2024-03-23T23:07:28.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.916+0000 7fb71ea74740 1 client.16686 shutdown 2024-03-23T23:07:28.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.917+0000 7fb71ea74740 1 -- 172.21.15.12:0/3385372047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5610fcf129f0 msgr2=0x5610fcf24060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.917+0000 7fb71ea74740 1 --2- 172.21.15.12:0/3385372047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5610fcf129f0 0x5610fcf24060 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb704000c00 tx=0x7fb7040bce30 comp rx=0 tx=0).stop 2024-03-23T23:07:28.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.917+0000 7fb71ea74740 1 -- 172.21.15.12:0/3385372047 shutdown_connections 2024-03-23T23:07:28.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.917+0000 7fb71ea74740 1 --2- 172.21.15.12:0/3385372047 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5610fcf85460 0x5610fcf87880 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.917+0000 7fb71ea74740 1 --2- 172.21.15.12:0/3385372047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610fcf14580 0x5610fcf22bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.917+0000 7fb71ea74740 1 --2- 172.21.15.12:0/3385372047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610fcf13bc0 0x5610fcf22670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.917+0000 7fb71ea74740 1 --2- 172.21.15.12:0/3385372047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5610fcf129f0 0x5610fcf24060 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.917+0000 7fb6fea00640 0 client.16686 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:07:28.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.917+0000 7fb71ea74740 1 -- 172.21.15.12:0/3385372047 >> 172.21.15.12:0/3385372047 conn(0x5610fcf0dde0 msgr2=0x5610fcf17680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:28.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.917+0000 7fb71ea74740 1 -- 172.21.15.12:0/3385372047 shutdown_connections 2024-03-23T23:07:28.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.917+0000 7fb71ea74740 1 -- 172.21.15.12:0/3385372047 wait complete. 2024-03-23T23:07:28.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.917+0000 7fb71ea74740 20 client.16686 trim_cache size 0 max 16384 2024-03-23T23:07:28.921 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:07:28.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.918+0000 7fb71a800640 1 -- 172.21.15.12:0/1805908354 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb6f80a2880 msgr2=0x7fb6f80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.918+0000 7fb71a800640 1 --2- 172.21.15.12:0/1805908354 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb6f80a2880 0x7fb6f80a4d40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x5610fcf13ac0 tx=0x7fb70c005f90 comp rx=0 tx=0).stop 2024-03-23T23:07:28.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.918+0000 7fb71a800640 1 -- 172.21.15.12:0/1805908354 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb714102000 msgr2=0x7fb714199a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:28.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.918+0000 7fb71a800640 1 --2- 172.21.15.12:0/1805908354 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb714102000 0x7fb714199a90 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb704005ab0 tx=0x7fb70402c6c0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.918+0000 7fb71a800640 1 -- 172.21.15.12:0/1805908354 shutdown_connections 2024-03-23T23:07:28.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.918+0000 7fb71a800640 1 --2- 172.21.15.12:0/1805908354 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb6f80a2880 0x7fb6f80a4d40 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.918+0000 7fb71a800640 1 --2- 172.21.15.12:0/1805908354 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb714107260 0x7fb7141a1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.918+0000 7fb71a800640 1 --2- 172.21.15.12:0/1805908354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb714104930 0x7fb714199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.918+0000 7fb71a800640 1 --2- 172.21.15.12:0/1805908354 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb714102000 0x7fb714199a90 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:28.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.918+0000 7fb71a800640 1 -- 172.21.15.12:0/1805908354 >> 172.21.15.12:0/1805908354 conn(0x7fb7140fba10 msgr2=0x7fb7141016b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:28.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.918+0000 7fb71a800640 1 -- 172.21.15.12:0/1805908354 shutdown_connections 2024-03-23T23:07:28.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:28.918+0000 7fb71a800640 1 -- 172.21.15.12:0/1805908354 wait complete. 2024-03-23T23:07:28.933 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:28 smithi012 ceph-mon[31695]: pgmap v560: 97 pgs: 97 active+clean; 3.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:07:28.933 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:28 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1135864511' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:07:28.936 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:07:28.936 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-23T23:07:29.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:28 smithi173 ceph-mon[36116]: pgmap v560: 97 pgs: 97 active+clean; 3.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:07:29.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:28 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1135864511' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:07:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.146+0000 7f8d9a400640 1 -- 172.21.15.12:0/3507960624 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d94108eb0 msgr2=0x7f8d94109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.146+0000 7f8d9a400640 1 --2- 172.21.15.12:0/3507960624 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d94108eb0 0x7f8d94109330 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8d84009900 tx=0x7f8d8402bc30 comp rx=0 tx=0).stop 2024-03-23T23:07:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.147+0000 7f8d9a400640 1 -- 172.21.15.12:0/3507960624 shutdown_connections 2024-03-23T23:07:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.147+0000 7f8d9a400640 1 --2- 172.21.15.12:0/3507960624 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d94109870 0x7f8d94110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.147+0000 7f8d9a400640 1 --2- 172.21.15.12:0/3507960624 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d94108eb0 0x7f8d94109330 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.147+0000 7f8d9a400640 1 --2- 172.21.15.12:0/3507960624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d94107cc0 0x7f8d941080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.147+0000 7f8d9a400640 1 -- 172.21.15.12:0/3507960624 >> 172.21.15.12:0/3507960624 conn(0x7f8d94076c60 msgr2=0x7f8d940790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.147+0000 7f8d9a400640 1 -- 172.21.15.12:0/3507960624 shutdown_connections 2024-03-23T23:07:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.147+0000 7f8d9a400640 1 -- 172.21.15.12:0/3507960624 wait complete. 2024-03-23T23:07:29.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.148+0000 7f8d9a400640 1 Processor -- start 2024-03-23T23:07:29.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.148+0000 7f8d9a400640 1 -- start start 2024-03-23T23:07:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.149+0000 7f8d9a400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d94107cc0 0x7f8d9419df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.149+0000 7f8d9a400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d94108eb0 0x7f8d9419e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.149+0000 7f8d9a400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d94109870 0x7f8d941a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.149+0000 7f8d9a400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d94113970 con 0x7f8d94108eb0 2024-03-23T23:07:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.149+0000 7f8d9a400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d941137f0 con 0x7f8d94109870 2024-03-23T23:07:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.149+0000 7f8d9a400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d94113af0 con 0x7f8d94107cc0 2024-03-23T23:07:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.149+0000 7f8d93400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d94107cc0 0x7f8d9419df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.149+0000 7f8d93400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d94107cc0 0x7f8d9419df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45832/0 (socket says 172.21.15.12:45832) 2024-03-23T23:07:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.149+0000 7f8d93400640 1 -- 172.21.15.12:0/1074100953 learned_addr learned my addr 172.21.15.12:0/1074100953 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.149+0000 7f8d92a00640 1 --2- 172.21.15.12:0/1074100953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d94108eb0 0x7f8d9419e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.149+0000 7f8d93e00640 1 --2- 172.21.15.12:0/1074100953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d94109870 0x7f8d941a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.149+0000 7f8d93400640 1 -- 172.21.15.12:0/1074100953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d94109870 msgr2=0x7f8d941a5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.149+0000 7f8d93400640 1 --2- 172.21.15.12:0/1074100953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d94109870 0x7f8d941a5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.149+0000 7f8d93400640 1 -- 172.21.15.12:0/1074100953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d94108eb0 msgr2=0x7f8d9419e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.149+0000 7f8d93400640 1 --2- 172.21.15.12:0/1074100953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d94108eb0 0x7f8d9419e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.149+0000 7f8d93400640 1 -- 172.21.15.12:0/1074100953 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d941a5bb0 con 0x7f8d94107cc0 2024-03-23T23:07:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.150+0000 7f8d93400640 1 --2- 172.21.15.12:0/1074100953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d94107cc0 0x7f8d9419df90 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c00d770 tx=0x7f8d7c00dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.150+0000 7f8d8be00640 1 -- 172.21.15.12:0/1074100953 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c017070 con 0x7f8d94107cc0 2024-03-23T23:07:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.150+0000 7f8d8be00640 1 -- 172.21.15.12:0/1074100953 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d7c009c30 con 0x7f8d94107cc0 2024-03-23T23:07:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.150+0000 7f8d8be00640 1 -- 172.21.15.12:0/1074100953 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c003dd0 con 0x7f8d94107cc0 2024-03-23T23:07:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.150+0000 7f8d9a400640 1 -- 172.21.15.12:0/1074100953 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d941a5ea0 con 0x7f8d94107cc0 2024-03-23T23:07:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.150+0000 7f8d9a400640 1 -- 172.21.15.12:0/1074100953 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d941a62b0 con 0x7f8d94107cc0 2024-03-23T23:07:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.153+0000 7f8d9d061740 1 Processor -- start 2024-03-23T23:07:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.154+0000 7f8d9d061740 1 -- start start 2024-03-23T23:07:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.154+0000 7f8d8be00640 1 -- 172.21.15.12:0/1074100953 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8d7c003f70 con 0x7f8d94107cc0 2024-03-23T23:07:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.154+0000 7f8d9d061740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589eb823810 0x5589eb823c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.154+0000 7f8d9d061740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589eb8249e0 0x5589eb824e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.154+0000 7f8d9d061740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589eb8253a0 0x5589eb82bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.154+0000 7f8d9d061740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589eb7882c0 con 0x5589eb8249e0 2024-03-23T23:07:29.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.154+0000 7f8d9d061740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589eb730c90 con 0x5589eb8253a0 2024-03-23T23:07:29.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.154+0000 7f8d9d061740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589eb784360 con 0x5589eb823810 2024-03-23T23:07:29.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.154+0000 7f8d93e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589eb823810 0x5589eb823c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.154+0000 7f8d92a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589eb8253a0 0x5589eb82bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.154+0000 7f8d93e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589eb823810 0x5589eb823c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45836/0 (socket says 172.21.15.12:45836) 2024-03-23T23:07:29.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.154+0000 7f8d93e00640 1 -- 172.21.15.12:0/152247139 learned_addr learned my addr 172.21.15.12:0/152247139 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:29.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.154+0000 7f8d93400640 1 --2- 172.21.15.12:0/152247139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589eb8249e0 0x5589eb824e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.155+0000 7f8d93e00640 1 -- 172.21.15.12:0/152247139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589eb8253a0 msgr2=0x5589eb82bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:29.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.155+0000 7f8d93e00640 1 --2- 172.21.15.12:0/152247139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589eb8253a0 0x5589eb82bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.155+0000 7f8d93e00640 1 -- 172.21.15.12:0/152247139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589eb8249e0 msgr2=0x5589eb824e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:29.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.155+0000 7f8d93e00640 1 --2- 172.21.15.12:0/152247139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589eb8249e0 0x5589eb824e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.155+0000 7f8d93e00640 1 -- 172.21.15.12:0/152247139 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5589eb755080 con 0x5589eb823810 2024-03-23T23:07:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.155+0000 7f8d93e00640 1 --2- 172.21.15.12:0/152247139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589eb823810 0x5589eb823c30 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c010370 tx=0x7f8d8c010830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa7fc25329616112 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.155+0000 7f8d88c00640 1 -- 172.21.15.12:0/152247139 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c016070 con 0x5589eb823810 2024-03-23T23:07:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.155+0000 7f8d88c00640 1 -- 172.21.15.12:0/152247139 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d8c004d20 con 0x5589eb823810 2024-03-23T23:07:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.155+0000 7f8d88c00640 1 -- 172.21.15.12:0/152247139 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c005480 con 0x5589eb823810 2024-03-23T23:07:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.156+0000 7f8d8be00640 1 --2- 172.21.15.12:0/1074100953 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8d780a2800 0x7f8d780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.156+0000 7f8d9d061740 1 -- 172.21.15.12:0/152247139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589eb823810 msgr2=0x5589eb823c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.156+0000 7f8d93400640 1 --2- 172.21.15.12:0/1074100953 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8d780a2800 0x7f8d780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.156+0000 7f8d9d061740 1 --2- 172.21.15.12:0/152247139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589eb823810 0x5589eb823c30 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c010370 tx=0x7f8d8c010830 comp rx=0 tx=0).stop 2024-03-23T23:07:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.156+0000 7f8d8be00640 1 -- 172.21.15.12:0/1074100953 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8d7c0ce380 con 0x7f8d94107cc0 2024-03-23T23:07:29.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.156+0000 7f8d9d061740 1 -- 172.21.15.12:0/152247139 shutdown_connections 2024-03-23T23:07:29.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.156+0000 7f8d9d061740 1 --2- 172.21.15.12:0/152247139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589eb8253a0 0x5589eb82bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.156+0000 7f8d9d061740 1 --2- 172.21.15.12:0/152247139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589eb8249e0 0x5589eb824e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.156+0000 7f8d9d061740 1 --2- 172.21.15.12:0/152247139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589eb823810 0x5589eb823c30 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.156+0000 7f8d9d061740 1 -- 172.21.15.12:0/152247139 >> 172.21.15.12:0/152247139 conn(0x5589eb81ec00 msgr2=0x5589eb81f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:29.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.156+0000 7f8d9d061740 1 -- 172.21.15.12:0/152247139 shutdown_connections 2024-03-23T23:07:29.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.156+0000 7f8d9d061740 1 -- 172.21.15.12:0/152247139 wait complete. 2024-03-23T23:07:29.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.157+0000 7f8d9d061740 1 -- start start 2024-03-23T23:07:29.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.157+0000 7f8d93400640 1 --2- 172.21.15.12:0/1074100953 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8d780a2800 0x7f8d780a4cc0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c09ad10 tx=0x7f8d7c003870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:29.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.157+0000 7f8d9d061740 1 Processor -- start 2024-03-23T23:07:29.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.157+0000 7f8d9d061740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589eb823810 0x5589eb82da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.157+0000 7f8d9d061740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589eb8249e0 0x5589eb82df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.157+0000 7f8d9d061740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589eb8253a0 0x5589eb834fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.157+0000 7f8d9d061740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589eb82c3b0 con 0x5589eb8253a0 2024-03-23T23:07:29.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.158+0000 7f8d9d061740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589eb7440d0 con 0x5589eb8249e0 2024-03-23T23:07:29.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.158+0000 7f8d9d061740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589eb82c530 con 0x5589eb823810 2024-03-23T23:07:29.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.158+0000 7f8d92a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589eb8249e0 0x5589eb82df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.158+0000 7f8d92a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589eb8249e0 0x5589eb82df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44150/0 (socket says 172.21.15.12:44150) 2024-03-23T23:07:29.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.158+0000 7f8d92a00640 1 -- 172.21.15.12:0/2637388572 learned_addr learned my addr 172.21.15.12:0/2637388572 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:29.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.158+0000 7f8d9d061740 10 client.0 _ll_register_callbacks cb 0x5589eb748e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:29.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.158+0000 7f8d93e00640 1 --2- 172.21.15.12:0/2637388572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589eb823810 0x5589eb82da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.158+0000 7f8d93e00640 1 --2- 172.21.15.12:0/2637388572 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589eb8253a0 0x5589eb834fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.158+0000 7f8d92a00640 1 -- 172.21.15.12:0/2637388572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589eb823810 msgr2=0x5589eb82da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:29.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.158+0000 7f8d92a00640 1 --2- 172.21.15.12:0/2637388572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589eb823810 0x5589eb82da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.158+0000 7f8d92a00640 1 -- 172.21.15.12:0/2637388572 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589eb8253a0 msgr2=0x5589eb834fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:29.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.158+0000 7f8d92a00640 1 --2- 172.21.15.12:0/2637388572 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589eb8253a0 0x5589eb834fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.158+0000 7f8d92a00640 1 -- 172.21.15.12:0/2637388572 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5589eb49c3f0 con 0x5589eb8249e0 2024-03-23T23:07:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.158+0000 7f8d93e00640 1 --2- 172.21.15.12:0/2637388572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589eb823810 0x5589eb82da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:07:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.159+0000 7f8d92a00640 1 --2- 172.21.15.12:0/2637388572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589eb8249e0 0x5589eb82df70 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f8d84005c40 tx=0x7f8d84004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea5a8a1d12dc85e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.159+0000 7f8d81600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:07:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.159+0000 7f8d81600640 1 -- 172.21.15.12:0/2637388572 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d840046c0 con 0x5589eb8249e0 2024-03-23T23:07:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.159+0000 7f8d81600640 1 -- 172.21.15.12:0/2637388572 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8d84004860 con 0x5589eb8249e0 2024-03-23T23:07:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.159+0000 7f8d9d061740 1 -- 172.21.15.12:0/2637388572 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8d8c004d20 con 0x5589eb8249e0 2024-03-23T23:07:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.159+0000 7f8d81600640 1 -- 172.21.15.12:0/2637388572 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d84004a60 con 0x5589eb8249e0 2024-03-23T23:07:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.159+0000 7f8d9d061740 10 client.26825 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.159+0000 7f8d9d061740 1 -- 172.21.15.12:0/2637388572 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5589eb5667d0 con 0x5589eb8249e0 2024-03-23T23:07:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.159+0000 7f8d81600640 1 -- 172.21.15.12:0/2637388572 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d8c004d20 con 0x5589eb8249e0 2024-03-23T23:07:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.160+0000 7f8d81600640 1 -- 172.21.15.12:0/2637388572 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8d840039f0 con 0x5589eb8249e0 2024-03-23T23:07:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.160+0000 7f8d9d061740 10 client.26825 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.160+0000 7f8d9d061740 10 client.26825 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:07:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.160+0000 7f8d9d061740 20 client.26825 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.160+0000 7f8d9d061740 1 --2- 172.21.15.12:0/2637388572 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5589eb8394b0 0x5589eb83b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.160+0000 7f8d9d061740 4 client.26825 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:07:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.160+0000 7f8d9d061740 1 -- 172.21.15.12:0/2637388572 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5589eb795800 con 0x5589eb8394b0 2024-03-23T23:07:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.160+0000 7f8d93e00640 1 --2- 172.21.15.12:0/2637388572 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5589eb8394b0 0x5589eb83b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.165 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:28 smithi203 ceph-mon[36693]: pgmap v560: 97 pgs: 97 active+clean; 3.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:07:29.165 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:28 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1135864511' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:07:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.161+0000 7f8d93e00640 1 --2- 172.21.15.12:0/2637388572 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5589eb8394b0 0x5589eb83b8f0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=76acfde0d3ed4804 server_cookie=b6a786f88680f5e1 in_seq=0 out_seq=0 2024-03-23T23:07:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.161+0000 7f8d81600640 10 client.26825 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:07:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.162+0000 7f8d81600640 1 -- 172.21.15.12:0/2637388572 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5589eb795800 con 0x5589eb8394b0 2024-03-23T23:07:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.162+0000 7f8d81600640 10 client.26825 handle_command_reply: tid=0 2024-03-23T23:07:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.162+0000 7f8d9d061740 1 client.26825 shutdown 2024-03-23T23:07:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.162+0000 7f8d9d061740 1 -- 172.21.15.12:0/2637388572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589eb8249e0 msgr2=0x5589eb82df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.162+0000 7f8d9d061740 1 --2- 172.21.15.12:0/2637388572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589eb8249e0 0x5589eb82df70 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f8d84005c40 tx=0x7f8d84004270 comp rx=0 tx=0).stop 2024-03-23T23:07:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.162+0000 7f8d9d061740 1 -- 172.21.15.12:0/2637388572 shutdown_connections 2024-03-23T23:07:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.162+0000 7f8d9d061740 1 --2- 172.21.15.12:0/2637388572 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589eb8253a0 0x5589eb834fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.162+0000 7f8d9d061740 1 --2- 172.21.15.12:0/2637388572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589eb8249e0 0x5589eb82df70 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.162+0000 7f8d9d061740 1 --2- 172.21.15.12:0/2637388572 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5589eb8394b0 0x5589eb83b8f0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.162+0000 7f8d81600640 0 client.26825 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:07:29.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.162+0000 7f8d9d061740 1 --2- 172.21.15.12:0/2637388572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589eb823810 0x5589eb82da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.162+0000 7f8d9d061740 1 -- 172.21.15.12:0/2637388572 >> 172.21.15.12:0/2637388572 conn(0x5589eb81ec00 msgr2=0x5589eb81deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:29.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.162+0000 7f8d9d061740 1 -- 172.21.15.12:0/2637388572 shutdown_connections 2024-03-23T23:07:29.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.162+0000 7f8d9d061740 1 -- 172.21.15.12:0/2637388572 wait complete. 2024-03-23T23:07:29.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.162+0000 7f8d9d061740 20 client.26825 trim_cache size 0 max 16384 2024-03-23T23:07:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.176+0000 7f8d9d061740 1 Processor -- start 2024-03-23T23:07:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.176+0000 7f8d9d061740 1 -- start start 2024-03-23T23:07:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.176+0000 7f8d9d061740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589eb823810 0x5589eb82d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.176+0000 7f8d9d061740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589eb8249e0 0x5589eb834f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.176+0000 7f8d9d061740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589eb8253a0 0x5589eb827c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.176+0000 7f8d9d061740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d941b1590 con 0x5589eb823810 2024-03-23T23:07:29.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.176+0000 7f8d9d061740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d941af090 con 0x5589eb8253a0 2024-03-23T23:07:29.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.176+0000 7f8d9d061740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d941ad130 con 0x5589eb8249e0 2024-03-23T23:07:29.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.177+0000 7f8d93e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589eb823810 0x5589eb82d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.177+0000 7f8d92a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589eb8249e0 0x5589eb834f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.177+0000 7f8d93e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589eb823810 0x5589eb82d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60640/0 (socket says 172.21.15.12:60640) 2024-03-23T23:07:29.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.177+0000 7f8d93e00640 1 -- 172.21.15.12:0/2455659081 learned_addr learned my addr 172.21.15.12:0/2455659081 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:29.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.177+0000 7f8d93e00640 1 --2- 172.21.15.12:0/2455659081 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589eb8253a0 0x5589eb827c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.177+0000 7f8d93e00640 1 -- 172.21.15.12:0/2455659081 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589eb8249e0 msgr2=0x5589eb834f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:29.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.177+0000 7f8d93e00640 1 --2- 172.21.15.12:0/2455659081 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589eb8249e0 0x5589eb834f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.177+0000 7f8d93e00640 1 -- 172.21.15.12:0/2455659081 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589eb8253a0 msgr2=0x5589eb827c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:29.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.177+0000 7f8d93e00640 1 --2- 172.21.15.12:0/2455659081 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589eb8253a0 0x5589eb827c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.177+0000 7f8d93e00640 1 -- 172.21.15.12:0/2455659081 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5589eb82ea20 con 0x5589eb823810 2024-03-23T23:07:29.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.177+0000 7f8d93e00640 1 --2- 172.21.15.12:0/2455659081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589eb823810 0x5589eb82d050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c005fd0 tx=0x7f8d8c009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce7369d53c87fad3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:29.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.178+0000 7f8d83e00640 1 -- 172.21.15.12:0/2455659081 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c016040 con 0x5589eb823810 2024-03-23T23:07:29.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.178+0000 7f8d83e00640 1 -- 172.21.15.12:0/2455659081 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d8c0099e0 con 0x5589eb823810 2024-03-23T23:07:29.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.178+0000 7f8d83e00640 1 -- 172.21.15.12:0/2455659081 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c0052a0 con 0x5589eb823810 2024-03-23T23:07:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.178+0000 7f8d9d061740 1 -- 172.21.15.12:0/2455659081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589eb823810 msgr2=0x5589eb82d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.178+0000 7f8d9d061740 1 --2- 172.21.15.12:0/2455659081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589eb823810 0x5589eb82d050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c005fd0 tx=0x7f8d8c009600 comp rx=0 tx=0).stop 2024-03-23T23:07:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.178+0000 7f8d9d061740 1 -- 172.21.15.12:0/2455659081 shutdown_connections 2024-03-23T23:07:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.178+0000 7f8d9d061740 1 --2- 172.21.15.12:0/2455659081 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589eb8253a0 0x5589eb827c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.178+0000 7f8d9d061740 1 --2- 172.21.15.12:0/2455659081 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589eb8249e0 0x5589eb834f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.178+0000 7f8d9d061740 1 --2- 172.21.15.12:0/2455659081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589eb823810 0x5589eb82d050 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.178+0000 7f8d9d061740 1 -- 172.21.15.12:0/2455659081 >> 172.21.15.12:0/2455659081 conn(0x5589eb81ec00 msgr2=0x5589eb8226f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.178+0000 7f8d9d061740 1 -- 172.21.15.12:0/2455659081 shutdown_connections 2024-03-23T23:07:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.179+0000 7f8d9d061740 1 -- 172.21.15.12:0/2455659081 wait complete. 2024-03-23T23:07:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.179+0000 7f8d9d061740 1 -- start start 2024-03-23T23:07:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.179+0000 7f8d9d061740 1 Processor -- start 2024-03-23T23:07:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.179+0000 7f8d9d061740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589eb823810 0x5589eb834ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.179+0000 7f8d9d061740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589eb8249e0 0x5589eb8331b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.180+0000 7f8d9d061740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589eb8253a0 0x5589eb8336f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.180+0000 7f8d9d061740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d941ad730 con 0x5589eb8249e0 2024-03-23T23:07:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.180+0000 7f8d9d061740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d941aea30 con 0x5589eb8253a0 2024-03-23T23:07:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.180+0000 7f8d9d061740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d941add70 con 0x5589eb823810 2024-03-23T23:07:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.180+0000 7f8d9d061740 10 client.0 _ll_register_callbacks cb 0x7f8d84009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.180+0000 7f8d92a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589eb8249e0 0x5589eb8331b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.180+0000 7f8d93e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589eb8253a0 0x5589eb8336f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.180+0000 7f8d92a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589eb8249e0 0x5589eb8331b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60654/0 (socket says 172.21.15.12:60654) 2024-03-23T23:07:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.180+0000 7f8d92a00640 1 -- 172.21.15.12:0/3036268861 learned_addr learned my addr 172.21.15.12:0/3036268861 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.180+0000 7f8d93e00640 1 --2- 172.21.15.12:0/3036268861 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589eb823810 0x5589eb834ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.180+0000 7f8d92a00640 1 -- 172.21.15.12:0/3036268861 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589eb823810 msgr2=0x5589eb834ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.180+0000 7f8d92a00640 1 --2- 172.21.15.12:0/3036268861 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589eb823810 0x5589eb834ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.180+0000 7f8d92a00640 1 -- 172.21.15.12:0/3036268861 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589eb8253a0 msgr2=0x5589eb8336f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.180+0000 7f8d92a00640 1 --2- 172.21.15.12:0/3036268861 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589eb8253a0 0x5589eb8336f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.180+0000 7f8d92a00640 1 -- 172.21.15.12:0/3036268861 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5589eb833c30 con 0x5589eb8249e0 2024-03-23T23:07:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.181+0000 7f8d92a00640 1 --2- 172.21.15.12:0/3036268861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589eb8249e0 0x5589eb8331b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8d84009900 tx=0x7f8d840048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b1d884db34a107c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:29.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.181+0000 7f8d81600640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:07:29.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.181+0000 7f8d81600640 1 -- 172.21.15.12:0/3036268861 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d84000e60 con 0x5589eb8249e0 2024-03-23T23:07:29.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.181+0000 7f8d81600640 1 -- 172.21.15.12:0/3036268861 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8d84003800 con 0x5589eb8249e0 2024-03-23T23:07:29.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.181+0000 7f8d9d061740 1 -- 172.21.15.12:0/3036268861 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8d8c0099e0 con 0x5589eb8249e0 2024-03-23T23:07:29.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.181+0000 7f8d81600640 1 -- 172.21.15.12:0/3036268861 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d8402cd20 con 0x5589eb8249e0 2024-03-23T23:07:29.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.181+0000 7f8d9d061740 10 client.16704 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:29.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.181+0000 7f8d9d061740 1 -- 172.21.15.12:0/3036268861 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5589eb895a80 con 0x5589eb8249e0 2024-03-23T23:07:29.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.182+0000 7f8d81600640 1 -- 172.21.15.12:0/3036268861 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d8c0099e0 con 0x5589eb8249e0 2024-03-23T23:07:29.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.182+0000 7f8d81600640 1 -- 172.21.15.12:0/3036268861 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8d8405e030 con 0x5589eb8249e0 2024-03-23T23:07:29.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.182+0000 7f8d9d061740 10 client.16704 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:29.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.182+0000 7f8d9d061740 10 client.16704 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:07:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.182+0000 7f8d9d061740 20 client.16704 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.182+0000 7f8d9d061740 1 --2- 172.21.15.12:0/3036268861 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5589eb8394b0 0x5589eb898440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.182+0000 7f8d9d061740 4 client.16704 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:07:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.182+0000 7f8d9d061740 1 -- 172.21.15.12:0/3036268861 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5589eb82d820 con 0x5589eb8394b0 2024-03-23T23:07:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.182+0000 7f8d93e00640 1 --2- 172.21.15.12:0/3036268861 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5589eb8394b0 0x5589eb898440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.183+0000 7f8d93e00640 1 --2- 172.21.15.12:0/3036268861 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5589eb8394b0 0x5589eb898440 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dd64a5fe000c6901 server_cookie=9b36516ba96037f9 in_seq=0 out_seq=0 2024-03-23T23:07:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.183+0000 7f8d81600640 10 client.16704 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:07:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.183+0000 7f8d81600640 1 -- 172.21.15.12:0/3036268861 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5589eb82d820 con 0x5589eb8394b0 2024-03-23T23:07:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.183+0000 7f8d81600640 10 client.16704 handle_command_reply: tid=0 2024-03-23T23:07:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.183+0000 7f8d9d061740 1 client.16704 shutdown 2024-03-23T23:07:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.183+0000 7f8d9d061740 1 -- 172.21.15.12:0/3036268861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589eb8249e0 msgr2=0x5589eb8331b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.183+0000 7f8d9d061740 1 --2- 172.21.15.12:0/3036268861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589eb8249e0 0x5589eb8331b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8d84009900 tx=0x7f8d840048f0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.183+0000 7f8d9d061740 1 -- 172.21.15.12:0/3036268861 shutdown_connections 2024-03-23T23:07:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.183+0000 7f8d9d061740 1 --2- 172.21.15.12:0/3036268861 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5589eb8253a0 0x5589eb8336f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.183+0000 7f8d9d061740 1 --2- 172.21.15.12:0/3036268861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5589eb8249e0 0x5589eb8331b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.183+0000 7f8d9d061740 1 --2- 172.21.15.12:0/3036268861 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5589eb8394b0 0x5589eb898440 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.183+0000 7f8d9d061740 1 --2- 172.21.15.12:0/3036268861 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5589eb823810 0x5589eb834ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.183+0000 7f8d81600640 0 client.16704 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:07:29.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.183+0000 7f8d9d061740 1 -- 172.21.15.12:0/3036268861 >> 172.21.15.12:0/3036268861 conn(0x5589eb81ec00 msgr2=0x5589eb8226f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:29.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.183+0000 7f8d9d061740 1 -- 172.21.15.12:0/3036268861 shutdown_connections 2024-03-23T23:07:29.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.183+0000 7f8d9d061740 1 -- 172.21.15.12:0/3036268861 wait complete. 2024-03-23T23:07:29.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.183+0000 7f8d9d061740 20 client.16704 trim_cache size 0 max 16384 2024-03-23T23:07:29.187 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:07:29.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.184+0000 7f8d9a400640 1 -- 172.21.15.12:0/1074100953 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8d780a2800 msgr2=0x7f8d780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:29.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.184+0000 7f8d9a400640 1 --2- 172.21.15.12:0/1074100953 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8d780a2800 0x7f8d780a4cc0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c09ad10 tx=0x7f8d7c003870 comp rx=0 tx=0).stop 2024-03-23T23:07:29.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.184+0000 7f8d9a400640 1 -- 172.21.15.12:0/1074100953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d94107cc0 msgr2=0x7f8d9419df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:29.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.184+0000 7f8d9a400640 1 --2- 172.21.15.12:0/1074100953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d94107cc0 0x7f8d9419df90 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c00d770 tx=0x7f8d7c00dc30 comp rx=0 tx=0).stop 2024-03-23T23:07:29.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.184+0000 7f8d9a400640 1 -- 172.21.15.12:0/1074100953 shutdown_connections 2024-03-23T23:07:29.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.184+0000 7f8d9a400640 1 --2- 172.21.15.12:0/1074100953 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8d780a2800 0x7f8d780a4cc0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.184+0000 7f8d9a400640 1 --2- 172.21.15.12:0/1074100953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d94109870 0x7f8d941a5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.184+0000 7f8d9a400640 1 --2- 172.21.15.12:0/1074100953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d94108eb0 0x7f8d9419e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.184+0000 7f8d9a400640 1 --2- 172.21.15.12:0/1074100953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d94107cc0 0x7f8d9419df90 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.184+0000 7f8d9a400640 1 -- 172.21.15.12:0/1074100953 >> 172.21.15.12:0/1074100953 conn(0x7f8d94076c60 msgr2=0x7f8d9410c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:29.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.184+0000 7f8d9a400640 1 -- 172.21.15.12:0/1074100953 shutdown_connections 2024-03-23T23:07:29.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.184+0000 7f8d9a400640 1 -- 172.21.15.12:0/1074100953 wait complete. 2024-03-23T23:07:29.204 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:07:29.204 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-23T23:07:29.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.428+0000 7fbbb9200640 1 -- 172.21.15.12:0/2872281386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbbb4102000 msgr2=0x7fbbb41043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:29.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.428+0000 7fbbb9200640 1 --2- 172.21.15.12:0/2872281386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbbb4102000 0x7fbbb41043f0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fbba4009900 tx=0x7fbba402bc50 comp rx=0 tx=0).stop 2024-03-23T23:07:29.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.428+0000 7fbbb9200640 1 -- 172.21.15.12:0/2872281386 shutdown_connections 2024-03-23T23:07:29.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.428+0000 7fbbb9200640 1 --2- 172.21.15.12:0/2872281386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbb4107260 0x7fbbb41096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.428+0000 7fbbb9200640 1 --2- 172.21.15.12:0/2872281386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb4104930 0x7fbbb4106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.428+0000 7fbbb9200640 1 --2- 172.21.15.12:0/2872281386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbbb4102000 0x7fbbb41043f0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.428+0000 7fbbb9200640 1 -- 172.21.15.12:0/2872281386 >> 172.21.15.12:0/2872281386 conn(0x7fbbb40fba10 msgr2=0x7fbbb40fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:29.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.429+0000 7fbbb9200640 1 -- 172.21.15.12:0/2872281386 shutdown_connections 2024-03-23T23:07:29.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.429+0000 7fbbb9200640 1 -- 172.21.15.12:0/2872281386 wait complete. 2024-03-23T23:07:29.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.430+0000 7fbbb9200640 1 Processor -- start 2024-03-23T23:07:29.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.430+0000 7fbbb9200640 1 -- start start 2024-03-23T23:07:29.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.430+0000 7fbbb9200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb4102000 0x7fbbb4199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.430+0000 7fbbb9200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbb4104930 0x7fbbb419a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.430+0000 7fbbb9200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbbb4107260 0x7fbbb41a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.431+0000 7fbbb9200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb41002a0 con 0x7fbbb4102000 2024-03-23T23:07:29.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.431+0000 7fbbb9200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb4100120 con 0x7fbbb4107260 2024-03-23T23:07:29.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.431+0000 7fbbb9200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb4100420 con 0x7fbbb4104930 2024-03-23T23:07:29.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.431+0000 7fbbb2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbb4104930 0x7fbbb419a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.431+0000 7fbbb3400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb4102000 0x7fbbb4199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.431+0000 7fbbb3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbbb4107260 0x7fbbb41a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.431+0000 7fbbb2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbb4104930 0x7fbbb419a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45884/0 (socket says 172.21.15.12:45884) 2024-03-23T23:07:29.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.431+0000 7fbbb2a00640 1 -- 172.21.15.12:0/3274594940 learned_addr learned my addr 172.21.15.12:0/3274594940 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:29.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.431+0000 7fbbb3400640 1 -- 172.21.15.12:0/3274594940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbb4104930 msgr2=0x7fbbb419a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:29.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.431+0000 7fbbb3400640 1 --2- 172.21.15.12:0/3274594940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbb4104930 0x7fbbb419a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.431+0000 7fbbb3400640 1 -- 172.21.15.12:0/3274594940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbbb4107260 msgr2=0x7fbbb41a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:29.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.431+0000 7fbbb3400640 1 --2- 172.21.15.12:0/3274594940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbbb4107260 0x7fbbb41a1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.431+0000 7fbbb3400640 1 -- 172.21.15.12:0/3274594940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb41a16f0 con 0x7fbbb4102000 2024-03-23T23:07:29.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.431+0000 7fbbb2a00640 1 --2- 172.21.15.12:0/3274594940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbb4104930 0x7fbbb419a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:07:29.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.432+0000 7fbbb3400640 1 --2- 172.21.15.12:0/3274594940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb4102000 0x7fbbb4199ad0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbba4036040 tx=0x7fbba402c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:29.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.432+0000 7fbbabe00640 1 -- 172.21.15.12:0/3274594940 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba4041070 con 0x7fbbb4102000 2024-03-23T23:07:29.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.432+0000 7fbbabe00640 1 -- 172.21.15.12:0/3274594940 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbba4004450 con 0x7fbbb4102000 2024-03-23T23:07:29.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.432+0000 7fbbabe00640 1 -- 172.21.15.12:0/3274594940 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba4003e60 con 0x7fbbb4102000 2024-03-23T23:07:29.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.432+0000 7fbbb9200640 1 -- 172.21.15.12:0/3274594940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb41a1980 con 0x7fbbb4102000 2024-03-23T23:07:29.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.432+0000 7fbbb9200640 1 -- 172.21.15.12:0/3274594940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbb41a1e50 con 0x7fbbb4102000 2024-03-23T23:07:29.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.436+0000 7fbbbd5dc740 1 Processor -- start 2024-03-23T23:07:29.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.436+0000 7fbbabe00640 1 -- 172.21.15.12:0/3274594940 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbba4044070 con 0x7fbbb4102000 2024-03-23T23:07:29.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.436+0000 7fbbbd5dc740 1 -- start start 2024-03-23T23:07:29.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.436+0000 7fbbbd5dc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562557f659f0 0x562557f65e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.436+0000 7fbbbd5dc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562557f66bc0 0x562557f67040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.436+0000 7fbbbd5dc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562557f67580 0x562557f6de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.436+0000 7fbbbd5dc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x562557eca2c0 con 0x562557f66bc0 2024-03-23T23:07:29.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.436+0000 7fbbbd5dc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x562557e72c90 con 0x562557f67580 2024-03-23T23:07:29.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.436+0000 7fbbbd5dc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x562557ec6360 con 0x562557f659f0 2024-03-23T23:07:29.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.437+0000 7fbbb2a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562557f67580 0x562557f6de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.437+0000 7fbbb3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562557f659f0 0x562557f65e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.437+0000 7fbbb3400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562557f66bc0 0x562557f67040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.437+0000 7fbbb3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562557f659f0 0x562557f65e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45886/0 (socket says 172.21.15.12:45886) 2024-03-23T23:07:29.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.437+0000 7fbbb2a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562557f67580 0x562557f6de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44202/0 (socket says 172.21.15.12:44202) 2024-03-23T23:07:29.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.437+0000 7fbbb3e00640 1 -- 172.21.15.12:0/3876665596 learned_addr learned my addr 172.21.15.12:0/3876665596 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:29.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.437+0000 7fbbb2a00640 1 -- 172.21.15.12:0/3876665596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562557f659f0 msgr2=0x562557f65e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:29.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.437+0000 7fbbb2a00640 1 --2- 172.21.15.12:0/3876665596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562557f659f0 0x562557f65e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.437+0000 7fbbb2a00640 1 -- 172.21.15.12:0/3876665596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562557f66bc0 msgr2=0x562557f67040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:29.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.437+0000 7fbbb2a00640 1 --2- 172.21.15.12:0/3876665596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562557f66bc0 0x562557f67040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.437+0000 7fbbb2a00640 1 -- 172.21.15.12:0/3876665596 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562557e97080 con 0x562557f67580 2024-03-23T23:07:29.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.437+0000 7fbbb3e00640 1 --2- 172.21.15.12:0/3876665596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562557f659f0 0x562557f65e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:07:29.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.438+0000 7fbbb2a00640 1 --2- 172.21.15.12:0/3876665596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562557f67580 0x562557f6de10 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fbbb419afe0 tx=0x7fbb9c0119a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bb8d90b5a8166fa server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:29.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.438+0000 7fbba8c00640 1 -- 172.21.15.12:0/3876665596 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9c01e070 con 0x562557f67580 2024-03-23T23:07:29.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.438+0000 7fbba8c00640 1 -- 172.21.15.12:0/3876665596 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb9c0035f0 con 0x562557f67580 2024-03-23T23:07:29.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.438+0000 7fbba8c00640 1 -- 172.21.15.12:0/3876665596 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9c00f850 con 0x562557f67580 2024-03-23T23:07:29.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.438+0000 7fbbabe00640 1 --2- 172.21.15.12:0/3274594940 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbb980a2880 0x7fbb980a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.438+0000 7fbbabe00640 1 -- 172.21.15.12:0/3274594940 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fbba40f0400 con 0x7fbbb4102000 2024-03-23T23:07:29.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.438+0000 7fbbb3e00640 1 --2- 172.21.15.12:0/3274594940 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbb980a2880 0x7fbb980a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.439+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/3876665596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562557f67580 msgr2=0x562557f6de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:29.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.439+0000 7fbbbd5dc740 1 --2- 172.21.15.12:0/3876665596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562557f67580 0x562557f6de10 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fbbb419afe0 tx=0x7fbb9c0119a0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.439+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/3876665596 shutdown_connections 2024-03-23T23:07:29.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.439+0000 7fbbbd5dc740 1 --2- 172.21.15.12:0/3876665596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562557f67580 0x562557f6de10 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.439+0000 7fbbbd5dc740 1 --2- 172.21.15.12:0/3876665596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562557f66bc0 0x562557f67040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.439+0000 7fbbbd5dc740 1 --2- 172.21.15.12:0/3876665596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562557f659f0 0x562557f65e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.439+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/3876665596 >> 172.21.15.12:0/3876665596 conn(0x562557f60de0 msgr2=0x562557f611e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:29.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.439+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/3876665596 shutdown_connections 2024-03-23T23:07:29.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.439+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/3876665596 wait complete. 2024-03-23T23:07:29.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.439+0000 7fbbb3e00640 1 --2- 172.21.15.12:0/3274594940 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbb980a2880 0x7fbb980a4d40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x562557f66ac0 tx=0x7fbbac009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:29.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.439+0000 7fbbbd5dc740 1 -- start start 2024-03-23T23:07:29.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.440+0000 7fbbbd5dc740 1 Processor -- start 2024-03-23T23:07:29.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.440+0000 7fbbbd5dc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562557f659f0 0x562557f6fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.440+0000 7fbbbd5dc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562557f66bc0 0x562557f701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.440+0000 7fbbbd5dc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562557f67580 0x562557f771e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.440+0000 7fbbbd5dc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x562557f6e590 con 0x562557f66bc0 2024-03-23T23:07:29.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.440+0000 7fbbbd5dc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x562557e860d0 con 0x562557f659f0 2024-03-23T23:07:29.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.440+0000 7fbbbd5dc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x562557f6e710 con 0x562557f67580 2024-03-23T23:07:29.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.440+0000 7fbbbd5dc740 10 client.0 _ll_register_callbacks cb 0x562557e8ae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:29.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.440+0000 7fbbb2a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562557f66bc0 0x562557f701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.440+0000 7fbbb3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562557f67580 0x562557f771e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.440+0000 7fbbb3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562557f67580 0x562557f771e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45894/0 (socket says 172.21.15.12:45894) 2024-03-23T23:07:29.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.440+0000 7fbbb2a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562557f66bc0 0x562557f701a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60694/0 (socket says 172.21.15.12:60694) 2024-03-23T23:07:29.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.440+0000 7fbbb3e00640 1 -- 172.21.15.12:0/3094877306 learned_addr learned my addr 172.21.15.12:0/3094877306 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:29.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.441+0000 7fbbb3e00640 1 -- 172.21.15.12:0/3094877306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562557f659f0 msgr2=0x562557f6fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:29.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.441+0000 7fbbb3e00640 1 --2- 172.21.15.12:0/3094877306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562557f659f0 0x562557f6fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.441+0000 7fbbb3e00640 1 -- 172.21.15.12:0/3094877306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562557f66bc0 msgr2=0x562557f701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:29.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.441+0000 7fbbb3e00640 1 --2- 172.21.15.12:0/3094877306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562557f66bc0 0x562557f701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.441+0000 7fbbb3e00640 1 -- 172.21.15.12:0/3094877306 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562557bde3f0 con 0x562557f67580 2024-03-23T23:07:29.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.441+0000 7fbbb3e00640 1 --2- 172.21.15.12:0/3094877306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562557f67580 0x562557f771e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fbbac012010 tx=0x7fbbac004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a10cc93018aa778 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:29.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.441+0000 7fbba1600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:07:29.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.442+0000 7fbba1600640 1 -- 172.21.15.12:0/3094877306 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbac004300 con 0x562557f67580 2024-03-23T23:07:29.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.442+0000 7fbba1600640 1 -- 172.21.15.12:0/3094877306 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fbbac0044a0 con 0x562557f67580 2024-03-23T23:07:29.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.442+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/3094877306 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbb9c0035f0 con 0x562557f67580 2024-03-23T23:07:29.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.442+0000 7fbba1600640 1 -- 172.21.15.12:0/3094877306 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbbac0046a0 con 0x562557f67580 2024-03-23T23:07:29.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.442+0000 7fbbbd5dc740 10 client.26812 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:29.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.442+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/3094877306 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x562557ca87d0 con 0x562557f67580 2024-03-23T23:07:29.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.442+0000 7fbba1600640 1 -- 172.21.15.12:0/3094877306 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb9c0035f0 con 0x562557f67580 2024-03-23T23:07:29.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.442+0000 7fbba1600640 1 -- 172.21.15.12:0/3094877306 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fbbac026070 con 0x562557f67580 2024-03-23T23:07:29.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.442+0000 7fbbbd5dc740 10 client.26812 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:29.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.442+0000 7fbbbd5dc740 10 client.26812 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:07:29.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.442+0000 7fbbbd5dc740 20 client.26812 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:29.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.443+0000 7fbbbd5dc740 1 --2- 172.21.15.12:0/3094877306 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562557f7b750 0x562557f7dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.443+0000 7fbbbd5dc740 4 client.26812 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:07:29.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.443+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/3094877306 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562557ed7800 con 0x562557f7b750 2024-03-23T23:07:29.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.443+0000 7fbbb3400640 1 --2- 172.21.15.12:0/3094877306 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562557f7b750 0x562557f7dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.443+0000 7fbbb3400640 1 --2- 172.21.15.12:0/3094877306 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562557f7b750 0x562557f7dbd0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86260c123f2f0859 server_cookie=fdd1bd61b28dbad2 in_seq=0 out_seq=0 2024-03-23T23:07:29.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.443+0000 7fbba1600640 10 client.26812 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:29.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.444+0000 7fbba1600640 1 -- 172.21.15.12:0/3094877306 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562557ed7800 con 0x562557f7b750 2024-03-23T23:07:29.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.444+0000 7fbba1600640 10 client.26812 handle_command_reply: tid=0 2024-03-23T23:07:29.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.444+0000 7fbbbd5dc740 1 client.26812 shutdown 2024-03-23T23:07:29.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.444+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/3094877306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562557f67580 msgr2=0x562557f771e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:29.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.444+0000 7fbbbd5dc740 1 --2- 172.21.15.12:0/3094877306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562557f67580 0x562557f771e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fbbac012010 tx=0x7fbbac004ab0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.444+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/3094877306 shutdown_connections 2024-03-23T23:07:29.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.444+0000 7fbbbd5dc740 1 --2- 172.21.15.12:0/3094877306 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562557f7b750 0x562557f7dbd0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.444+0000 7fbbbd5dc740 1 --2- 172.21.15.12:0/3094877306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562557f67580 0x562557f771e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.444+0000 7fbbbd5dc740 1 --2- 172.21.15.12:0/3094877306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562557f66bc0 0x562557f701a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.444+0000 7fbbbd5dc740 1 --2- 172.21.15.12:0/3094877306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562557f659f0 0x562557f6fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.444+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/3094877306 >> 172.21.15.12:0/3094877306 conn(0x562557f60de0 msgr2=0x562557f60160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:29.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.444+0000 7fbba1600640 0 client.26812 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:29.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.445+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/3094877306 shutdown_connections 2024-03-23T23:07:29.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.445+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/3094877306 wait complete. 2024-03-23T23:07:29.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.445+0000 7fbbbd5dc740 20 client.26812 trim_cache size 0 max 16384 2024-03-23T23:07:29.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.458+0000 7fbbbd5dc740 1 Processor -- start 2024-03-23T23:07:29.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.458+0000 7fbbbd5dc740 1 -- start start 2024-03-23T23:07:29.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.459+0000 7fbbbd5dc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562557f659f0 0x562557f64750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.459+0000 7fbbbd5dc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562557f66bc0 0x562557f771a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.459+0000 7fbbbd5dc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562557f67580 0x562557f69df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.459+0000 7fbbbd5dc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb41acf80 con 0x562557f659f0 2024-03-23T23:07:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.459+0000 7fbbbd5dc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb41aaa80 con 0x562557f67580 2024-03-23T23:07:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.459+0000 7fbbbd5dc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb41a8b20 con 0x562557f66bc0 2024-03-23T23:07:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.459+0000 7fbbb3400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562557f659f0 0x562557f64750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.459+0000 7fbbb3400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562557f659f0 0x562557f64750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60710/0 (socket says 172.21.15.12:60710) 2024-03-23T23:07:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.459+0000 7fbbb3400640 1 -- 172.21.15.12:0/4100370626 learned_addr learned my addr 172.21.15.12:0/4100370626 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.459+0000 7fbbb2a00640 1 --2- 172.21.15.12:0/4100370626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562557f66bc0 0x562557f771a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.459+0000 7fbbb3e00640 1 --2- 172.21.15.12:0/4100370626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562557f67580 0x562557f69df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.459+0000 7fbbb3400640 1 -- 172.21.15.12:0/4100370626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562557f66bc0 msgr2=0x562557f771a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.459+0000 7fbbb3400640 1 --2- 172.21.15.12:0/4100370626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562557f66bc0 0x562557f771a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.459+0000 7fbbb3400640 1 -- 172.21.15.12:0/4100370626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562557f67580 msgr2=0x562557f69df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.459+0000 7fbbb3400640 1 --2- 172.21.15.12:0/4100370626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562557f67580 0x562557f69df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.459+0000 7fbbb3400640 1 -- 172.21.15.12:0/4100370626 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562557f66640 con 0x562557f659f0 2024-03-23T23:07:29.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.460+0000 7fbbb3400640 1 --2- 172.21.15.12:0/4100370626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562557f659f0 0x562557f64750 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbba40edf10 tx=0x7fbba40f3ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3e4417e4d816fc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:29.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.460+0000 7fbba3e00640 1 -- 172.21.15.12:0/4100370626 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba404b050 con 0x562557f659f0 2024-03-23T23:07:29.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.460+0000 7fbba3e00640 1 -- 172.21.15.12:0/4100370626 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbba4004c60 con 0x562557f659f0 2024-03-23T23:07:29.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.460+0000 7fbba3e00640 1 -- 172.21.15.12:0/4100370626 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba40c5550 con 0x562557f659f0 2024-03-23T23:07:29.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.461+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/4100370626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562557f659f0 msgr2=0x562557f64750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:29.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.461+0000 7fbbbd5dc740 1 --2- 172.21.15.12:0/4100370626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562557f659f0 0x562557f64750 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbba40edf10 tx=0x7fbba40f3ee0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.461+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/4100370626 shutdown_connections 2024-03-23T23:07:29.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.461+0000 7fbbbd5dc740 1 --2- 172.21.15.12:0/4100370626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562557f67580 0x562557f69df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.461+0000 7fbbbd5dc740 1 --2- 172.21.15.12:0/4100370626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562557f66bc0 0x562557f771a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.461+0000 7fbbbd5dc740 1 --2- 172.21.15.12:0/4100370626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562557f659f0 0x562557f64750 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.461+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/4100370626 >> 172.21.15.12:0/4100370626 conn(0x562557f60de0 msgr2=0x562557f64040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:29.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.461+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/4100370626 shutdown_connections 2024-03-23T23:07:29.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.461+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/4100370626 wait complete. 2024-03-23T23:07:29.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.461+0000 7fbbbd5dc740 1 -- start start 2024-03-23T23:07:29.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.461+0000 7fbbbd5dc740 1 Processor -- start 2024-03-23T23:07:29.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.462+0000 7fbbbd5dc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562557f659f0 0x562557f74f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.462+0000 7fbbbd5dc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562557f66bc0 0x562557f754a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.462+0000 7fbbbd5dc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562557f67580 0x562557f764b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.462+0000 7fbbbd5dc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb41a9120 con 0x562557f67580 2024-03-23T23:07:29.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.462+0000 7fbbbd5dc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb41aa420 con 0x562557f66bc0 2024-03-23T23:07:29.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.462+0000 7fbbbd5dc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb41a9760 con 0x562557f659f0 2024-03-23T23:07:29.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.462+0000 7fbbbd5dc740 10 client.0 _ll_register_callbacks cb 0x7fbbac009dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:29.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.462+0000 7fbbb3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562557f67580 0x562557f764b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.462+0000 7fbbb2a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562557f66bc0 0x562557f754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.462+0000 7fbbb3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562557f67580 0x562557f764b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60720/0 (socket says 172.21.15.12:60720) 2024-03-23T23:07:29.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.462+0000 7fbbb3e00640 1 -- 172.21.15.12:0/2051617413 learned_addr learned my addr 172.21.15.12:0/2051617413 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:29.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.462+0000 7fbbb3400640 1 --2- 172.21.15.12:0/2051617413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562557f659f0 0x562557f74f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.463+0000 7fbbb3e00640 1 -- 172.21.15.12:0/2051617413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562557f659f0 msgr2=0x562557f74f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:29.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.463+0000 7fbbb3e00640 1 --2- 172.21.15.12:0/2051617413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562557f659f0 0x562557f74f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.463+0000 7fbbb3e00640 1 -- 172.21.15.12:0/2051617413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562557f66bc0 msgr2=0x562557f754a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:29.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.463+0000 7fbbb3e00640 1 --2- 172.21.15.12:0/2051617413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562557f66bc0 0x562557f754a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.463+0000 7fbbb3e00640 1 -- 172.21.15.12:0/2051617413 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562557f76aa0 con 0x562557f67580 2024-03-23T23:07:29.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.463+0000 7fbbb3400640 1 --2- 172.21.15.12:0/2051617413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562557f659f0 0x562557f74f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:07:29.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.463+0000 7fbbb3e00640 1 --2- 172.21.15.12:0/2051617413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562557f67580 0x562557f764b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbbac0285d0 tx=0x7fbbac003aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27a2f622590f37c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:29.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.463+0000 7fbba1600640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:07:29.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.463+0000 7fbba1600640 1 -- 172.21.15.12:0/2051617413 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbac0287c0 con 0x562557f67580 2024-03-23T23:07:29.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.463+0000 7fbba1600640 1 -- 172.21.15.12:0/2051617413 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fbbac028960 con 0x562557f67580 2024-03-23T23:07:29.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.463+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/2051617413 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbba4004c60 con 0x562557f67580 2024-03-23T23:07:29.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.463+0000 7fbba1600640 1 -- 172.21.15.12:0/2051617413 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbbac041050 con 0x562557f67580 2024-03-23T23:07:29.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.463+0000 7fbbbd5dc740 10 client.16722 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:29.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.463+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/2051617413 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x562557fd7670 con 0x562557f67580 2024-03-23T23:07:29.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.464+0000 7fbba1600640 1 -- 172.21.15.12:0/2051617413 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbba4004c60 con 0x562557f67580 2024-03-23T23:07:29.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.464+0000 7fbba1600640 1 -- 172.21.15.12:0/2051617413 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fbbac026050 con 0x562557f67580 2024-03-23T23:07:29.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.464+0000 7fbbbd5dc740 10 client.16722 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:29.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.464+0000 7fbbbd5dc740 10 client.16722 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:07:29.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.464+0000 7fbbbd5dc740 20 client.16722 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:29.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.464+0000 7fbbbd5dc740 1 --2- 172.21.15.12:0/2051617413 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562557fd7fb0 0x562557fda3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:29.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.464+0000 7fbbbd5dc740 4 client.16722 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:07:29.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.464+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/2051617413 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562557ed1b50 con 0x562557fd7fb0 2024-03-23T23:07:29.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.464+0000 7fbbb3400640 1 --2- 172.21.15.12:0/2051617413 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562557fd7fb0 0x562557fda3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:29.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.465+0000 7fbbb3400640 1 --2- 172.21.15.12:0/2051617413 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562557fd7fb0 0x562557fda3d0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64b609ffc3be51e7 server_cookie=86d11d62fa7e7d73 in_seq=0 out_seq=0 2024-03-23T23:07:29.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.465+0000 7fbba1600640 10 client.16722 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:29.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.465+0000 7fbba1600640 1 -- 172.21.15.12:0/2051617413 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562557ed1b50 con 0x562557fd7fb0 2024-03-23T23:07:29.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.465+0000 7fbba1600640 10 client.16722 handle_command_reply: tid=0 2024-03-23T23:07:29.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.465+0000 7fbbbd5dc740 1 client.16722 shutdown 2024-03-23T23:07:29.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.465+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/2051617413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562557f67580 msgr2=0x562557f764b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:29.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.465+0000 7fbbbd5dc740 1 --2- 172.21.15.12:0/2051617413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562557f67580 0x562557f764b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbbac0285d0 tx=0x7fbbac003aa0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.465+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/2051617413 shutdown_connections 2024-03-23T23:07:29.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.465+0000 7fbbbd5dc740 1 --2- 172.21.15.12:0/2051617413 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562557fd7fb0 0x562557fda3d0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.465+0000 7fbbbd5dc740 1 --2- 172.21.15.12:0/2051617413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562557f67580 0x562557f764b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.465+0000 7fbbbd5dc740 1 --2- 172.21.15.12:0/2051617413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562557f66bc0 0x562557f754a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.465+0000 7fbbbd5dc740 1 --2- 172.21.15.12:0/2051617413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562557f659f0 0x562557f74f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.465+0000 7fbba1600640 0 client.16722 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:29.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.465+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/2051617413 >> 172.21.15.12:0/2051617413 conn(0x562557f60de0 msgr2=0x562557f648d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:29.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.465+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/2051617413 shutdown_connections 2024-03-23T23:07:29.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.465+0000 7fbbbd5dc740 1 -- 172.21.15.12:0/2051617413 wait complete. 2024-03-23T23:07:29.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.465+0000 7fbbbd5dc740 20 client.16722 trim_cache size 0 max 16384 2024-03-23T23:07:29.470 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:07:29.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.466+0000 7fbbb9200640 1 -- 172.21.15.12:0/3274594940 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbb980a2880 msgr2=0x7fbb980a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:29.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.466+0000 7fbbb9200640 1 --2- 172.21.15.12:0/3274594940 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbb980a2880 0x7fbb980a4d40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x562557f66ac0 tx=0x7fbbac009500 comp rx=0 tx=0).stop 2024-03-23T23:07:29.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.466+0000 7fbbb9200640 1 -- 172.21.15.12:0/3274594940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb4102000 msgr2=0x7fbbb4199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:29.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.466+0000 7fbbb9200640 1 --2- 172.21.15.12:0/3274594940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb4102000 0x7fbbb4199ad0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbba4036040 tx=0x7fbba402c670 comp rx=0 tx=0).stop 2024-03-23T23:07:29.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.466+0000 7fbbb9200640 1 -- 172.21.15.12:0/3274594940 shutdown_connections 2024-03-23T23:07:29.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.466+0000 7fbbb9200640 1 --2- 172.21.15.12:0/3274594940 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbb980a2880 0x7fbb980a4d40 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.466+0000 7fbbb9200640 1 --2- 172.21.15.12:0/3274594940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbbb4107260 0x7fbbb41a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.466+0000 7fbbb9200640 1 --2- 172.21.15.12:0/3274594940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbb4104930 0x7fbbb419a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.466+0000 7fbbb9200640 1 --2- 172.21.15.12:0/3274594940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb4102000 0x7fbbb4199ad0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:29.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.466+0000 7fbbb9200640 1 -- 172.21.15.12:0/3274594940 >> 172.21.15.12:0/3274594940 conn(0x7fbbb40fba10 msgr2=0x7fbbb4104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:29.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.466+0000 7fbbb9200640 1 -- 172.21.15.12:0/3274594940 shutdown_connections 2024-03-23T23:07:29.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:29.466+0000 7fbbb9200640 1 -- 172.21.15.12:0/3274594940 wait complete. 2024-03-23T23:07:30.484 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-23T23:07:30.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:07:30.484 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-23T23:07:30.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.728+0000 7f11d3200640 1 -- 172.21.15.12:0/2906760678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11d4108db0 msgr2=0x7f11d4109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:30.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.728+0000 7f11d3200640 1 --2- 172.21.15.12:0/2906760678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11d4108db0 0x7f11d4109230 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f11c0009900 tx=0x7f11c002bc30 comp rx=0 tx=0).stop 2024-03-23T23:07:30.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.728+0000 7f11d3200640 1 -- 172.21.15.12:0/2906760678 shutdown_connections 2024-03-23T23:07:30.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.728+0000 7f11d3200640 1 --2- 172.21.15.12:0/2906760678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11d4109770 0x7f11d4110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.728+0000 7f11d3200640 1 --2- 172.21.15.12:0/2906760678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11d4108db0 0x7f11d4109230 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.728+0000 7f11d3200640 1 --2- 172.21.15.12:0/2906760678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11d4107bc0 0x7f11d4107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.728+0000 7f11d3200640 1 -- 172.21.15.12:0/2906760678 >> 172.21.15.12:0/2906760678 conn(0x7f11d4076b70 msgr2=0x7f11d4078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:30.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.728+0000 7f11d3200640 1 -- 172.21.15.12:0/2906760678 shutdown_connections 2024-03-23T23:07:30.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.728+0000 7f11d3200640 1 -- 172.21.15.12:0/2906760678 wait complete. 2024-03-23T23:07:30.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.729+0000 7f11d3200640 1 Processor -- start 2024-03-23T23:07:30.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.729+0000 7f11d3200640 1 -- start start 2024-03-23T23:07:30.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.729+0000 7f11d3200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11d4107bc0 0x7f11d419deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:30.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.729+0000 7f11d3200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11d4108db0 0x7f11d419e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:30.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.729+0000 7f11d3200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11d4109770 0x7f11d41a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:30.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.729+0000 7f11d3200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d4113890 con 0x7f11d4108db0 2024-03-23T23:07:30.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.729+0000 7f11d3200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d4113710 con 0x7f11d4107bc0 2024-03-23T23:07:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.729+0000 7f11d3200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d4113a10 con 0x7f11d4109770 2024-03-23T23:07:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.730+0000 7f11d1400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11d4108db0 0x7f11d419e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.730+0000 7f11d2800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11d4109770 0x7f11d41a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.730+0000 7f11d1400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11d4108db0 0x7f11d419e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60740/0 (socket says 172.21.15.12:60740) 2024-03-23T23:07:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.730+0000 7f11d2800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11d4109770 0x7f11d41a5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45936/0 (socket says 172.21.15.12:45936) 2024-03-23T23:07:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.730+0000 7f11d1400640 1 -- 172.21.15.12:0/237106472 learned_addr learned my addr 172.21.15.12:0/237106472 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.730+0000 7f11d2800640 1 -- 172.21.15.12:0/237106472 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11d4107bc0 msgr2=0x7f11d419deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.730+0000 7f11d2800640 1 --2- 172.21.15.12:0/237106472 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11d4107bc0 0x7f11d419deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.730+0000 7f11d2800640 1 -- 172.21.15.12:0/237106472 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11d4108db0 msgr2=0x7f11d419e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.730+0000 7f11d2800640 1 --2- 172.21.15.12:0/237106472 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11d4108db0 0x7f11d419e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.730+0000 7f11d2800640 1 -- 172.21.15.12:0/237106472 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d41a5ad0 con 0x7f11d4109770 2024-03-23T23:07:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.730+0000 7f11d2800640 1 --2- 172.21.15.12:0/237106472 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11d4109770 0x7f11d41a5460 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f11cc00d680 tx=0x7f11cc00db40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.731+0000 7f11caa00640 1 -- 172.21.15.12:0/237106472 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11cc017070 con 0x7f11d4109770 2024-03-23T23:07:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.731+0000 7f11caa00640 1 -- 172.21.15.12:0/237106472 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11cc009820 con 0x7f11d4109770 2024-03-23T23:07:30.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.731+0000 7f11caa00640 1 -- 172.21.15.12:0/237106472 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11cc004e00 con 0x7f11d4109770 2024-03-23T23:07:30.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.731+0000 7f11d3200640 1 -- 172.21.15.12:0/237106472 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d41a5dc0 con 0x7f11d4109770 2024-03-23T23:07:30.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.731+0000 7f11d3200640 1 -- 172.21.15.12:0/237106472 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d41a6230 con 0x7f11d4109770 2024-03-23T23:07:30.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.735+0000 7f11dc011740 1 Processor -- start 2024-03-23T23:07:30.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.735+0000 7f11caa00640 1 -- 172.21.15.12:0/237106472 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f11cc0099d0 con 0x7f11d4109770 2024-03-23T23:07:30.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.736+0000 7f11dc011740 1 -- start start 2024-03-23T23:07:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.736+0000 7f11dc011740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e7ecc2a2b0 0x55e7ecc2a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.736+0000 7f11dc011740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7ecc2aca0 0x55e7ecc2f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.736+0000 7f11dc011740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7ecc2fe10 0x55e7ecc322a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:30.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.736+0000 7f11dc011740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7ecb8feb0 con 0x55e7ecc2a2b0 2024-03-23T23:07:30.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.736+0000 7f11dc011740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7ecb38a60 con 0x55e7ecc2aca0 2024-03-23T23:07:30.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.736+0000 7f11dc011740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7ecb8bf50 con 0x55e7ecc2fe10 2024-03-23T23:07:30.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.736+0000 7f11d1e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7ecc2aca0 0x55e7ecc2f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:30.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.736+0000 7f11d1400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e7ecc2a2b0 0x55e7ecc2a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:30.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.736+0000 7f11d1e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7ecc2aca0 0x55e7ecc2f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44254/0 (socket says 172.21.15.12:44254) 2024-03-23T23:07:30.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.736+0000 7f11d1e00640 1 -- 172.21.15.12:0/1488330730 learned_addr learned my addr 172.21.15.12:0/1488330730 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:30.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.736+0000 7f11d1400640 1 --2- 172.21.15.12:0/1488330730 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7ecc2fe10 0x55e7ecc322a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:30.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.737+0000 7f11d1e00640 1 -- 172.21.15.12:0/1488330730 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7ecc2fe10 msgr2=0x55e7ecc322a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:30.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.737+0000 7f11d1e00640 1 --2- 172.21.15.12:0/1488330730 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7ecc2fe10 0x55e7ecc322a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.737+0000 7f11d1e00640 1 -- 172.21.15.12:0/1488330730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e7ecc2a2b0 msgr2=0x55e7ecc2a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:30.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.737+0000 7f11d1e00640 1 --2- 172.21.15.12:0/1488330730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e7ecc2a2b0 0x55e7ecc2a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.737+0000 7f11d1e00640 1 -- 172.21.15.12:0/1488330730 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e7ec96c7e0 con 0x55e7ecc2aca0 2024-03-23T23:07:30.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.737+0000 7f11d1e00640 1 --2- 172.21.15.12:0/1488330730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7ecc2aca0 0x55e7ecc2f7a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f11c4009ea0 tx=0x7f11c4009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89d2280a613f4b73 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:30.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.737+0000 7f11caa00640 1 --2- 172.21.15.12:0/237106472 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f11b80a2880 0x7f11b80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:30.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.737+0000 7f11bf400640 1 -- 172.21.15.12:0/1488330730 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c4010630 con 0x55e7ecc2aca0 2024-03-23T23:07:30.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.737+0000 7f11bf400640 1 -- 172.21.15.12:0/1488330730 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11c4010cb0 con 0x55e7ecc2aca0 2024-03-23T23:07:30.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.737+0000 7f11bf400640 1 -- 172.21.15.12:0/1488330730 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c4018d30 con 0x55e7ecc2aca0 2024-03-23T23:07:30.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.737+0000 7f11caa00640 1 -- 172.21.15.12:0/237106472 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f11cc0ce2f0 con 0x7f11d4109770 2024-03-23T23:07:30.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.737+0000 7f11d1400640 1 --2- 172.21.15.12:0/237106472 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f11b80a2880 0x7f11b80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:30.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.738+0000 7f11dc011740 1 -- 172.21.15.12:0/1488330730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7ecc2aca0 msgr2=0x55e7ecc2f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:30.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.738+0000 7f11dc011740 1 --2- 172.21.15.12:0/1488330730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7ecc2aca0 0x55e7ecc2f7a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f11c4009ea0 tx=0x7f11c4009f90 comp rx=0 tx=0).stop 2024-03-23T23:07:30.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.738+0000 7f11dc011740 1 -- 172.21.15.12:0/1488330730 shutdown_connections 2024-03-23T23:07:30.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.738+0000 7f11dc011740 1 --2- 172.21.15.12:0/1488330730 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7ecc2fe10 0x55e7ecc322a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.738+0000 7f11dc011740 1 --2- 172.21.15.12:0/1488330730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7ecc2aca0 0x55e7ecc2f7a0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.738+0000 7f11dc011740 1 --2- 172.21.15.12:0/1488330730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e7ecc2a2b0 0x55e7ecc2a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.738+0000 7f11dc011740 1 -- 172.21.15.12:0/1488330730 >> 172.21.15.12:0/1488330730 conn(0x55e7ecc25820 msgr2=0x55e7ecc25c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:30.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.738+0000 7f11d1400640 1 --2- 172.21.15.12:0/237106472 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f11b80a2880 0x7f11b80a4d40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f11c002c8e0 tx=0x7f11c0005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:30.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.738+0000 7f11dc011740 1 -- 172.21.15.12:0/1488330730 shutdown_connections 2024-03-23T23:07:30.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.738+0000 7f11dc011740 1 -- 172.21.15.12:0/1488330730 wait complete. 2024-03-23T23:07:30.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.739+0000 7f11dc011740 1 -- start start 2024-03-23T23:07:30.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.739+0000 7f11dc011740 1 Processor -- start 2024-03-23T23:07:30.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.739+0000 7f11dc011740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7ecc2a2b0 0x55e7ecc344f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:30.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.739+0000 7f11dc011740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7ecc2aca0 0x55e7ecc34a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:30.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.739+0000 7f11dc011740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e7ecc2fe10 0x55e7ecc3ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:30.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.739+0000 7f11dc011740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7ecc32dc0 con 0x55e7ecc2fe10 2024-03-23T23:07:30.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.740+0000 7f11dc011740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7ecb4beb0 con 0x55e7ecc2aca0 2024-03-23T23:07:30.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.740+0000 7f11dc011740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7ecc32f40 con 0x55e7ecc2a2b0 2024-03-23T23:07:30.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.740+0000 7f11dc011740 10 client.0 _ll_register_callbacks cb 0x55e7ecb40150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:30.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.740+0000 7f11d1e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7ecc2a2b0 0x55e7ecc344f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:30.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.740+0000 7f11d1e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7ecc2a2b0 0x55e7ecc344f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:45952/0 (socket says 172.21.15.12:45952) 2024-03-23T23:07:30.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.740+0000 7f11d1e00640 1 -- 172.21.15.12:0/2616420606 learned_addr learned my addr 172.21.15.12:0/2616420606 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:30.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.740+0000 7f11d1400640 1 --2- 172.21.15.12:0/2616420606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7ecc2aca0 0x55e7ecc34a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:30.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.740+0000 7f11d2800640 1 --2- 172.21.15.12:0/2616420606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e7ecc2fe10 0x55e7ecc3ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:30.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.740+0000 7f11d1e00640 1 -- 172.21.15.12:0/2616420606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7ecc2aca0 msgr2=0x55e7ecc34a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:30.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.740+0000 7f11d1e00640 1 --2- 172.21.15.12:0/2616420606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7ecc2aca0 0x55e7ecc34a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.740+0000 7f11d1e00640 1 -- 172.21.15.12:0/2616420606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e7ecc2fe10 msgr2=0x55e7ecc3ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:30.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.740+0000 7f11d1e00640 1 --2- 172.21.15.12:0/2616420606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e7ecc2fe10 0x55e7ecc3ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.740+0000 7f11d1e00640 1 -- 172.21.15.12:0/2616420606 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e7ec9ccf70 con 0x55e7ecc2a2b0 2024-03-23T23:07:30.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.740+0000 7f11d1400640 1 --2- 172.21.15.12:0/2616420606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7ecc2aca0 0x55e7ecc34a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:07:30.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.741+0000 7f11d1e00640 1 --2- 172.21.15.12:0/2616420606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7ecc2a2b0 0x55e7ecc344f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f11c400ebb0 tx=0x7f11c4018650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8cc9119196486f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:30.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.741+0000 7f11b3e00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:07:30.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.741+0000 7f11b3e00640 1 -- 172.21.15.12:0/2616420606 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c4010e60 con 0x55e7ecc2a2b0 2024-03-23T23:07:30.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.741+0000 7f11b3e00640 1 -- 172.21.15.12:0/2616420606 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f11c4010430 con 0x55e7ecc2a2b0 2024-03-23T23:07:30.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.741+0000 7f11dc011740 1 -- 172.21.15.12:0/2616420606 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f11c4010cb0 con 0x55e7ecc2a2b0 2024-03-23T23:07:30.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.741+0000 7f11b3e00640 1 -- 172.21.15.12:0/2616420606 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11c4031050 con 0x55e7ecc2a2b0 2024-03-23T23:07:30.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.741+0000 7f11dc011740 10 client.26827 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:30.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.741+0000 7f11dc011740 1 -- 172.21.15.12:0/2616420606 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55e7ec864170 con 0x55e7ecc2a2b0 2024-03-23T23:07:30.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.741+0000 7f11b3e00640 1 -- 172.21.15.12:0/2616420606 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11c4010cb0 con 0x55e7ecc2a2b0 2024-03-23T23:07:30.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.742+0000 7f11b3e00640 1 -- 172.21.15.12:0/2616420606 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f11c4022960 con 0x55e7ecc2a2b0 2024-03-23T23:07:30.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.742+0000 7f11dc011740 10 client.26827 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:30.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.742+0000 7f11dc011740 10 client.26827 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:07:30.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.742+0000 7f11dc011740 20 client.26827 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:30.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.742+0000 7f11dc011740 1 --2- 172.21.15.12:0/2616420606 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55e7ecc40000 0x55e7ecc423f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:30.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.742+0000 7f11dc011740 4 client.26827 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:07:30.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.742+0000 7f11dc011740 1 -- 172.21.15.12:0/2616420606 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e7ecb9d3f0 con 0x55e7ecc40000 2024-03-23T23:07:30.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.742+0000 7f11d1400640 1 --2- 172.21.15.12:0/2616420606 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55e7ecc40000 0x55e7ecc423f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:30.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.743+0000 7f11d1400640 1 --2- 172.21.15.12:0/2616420606 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55e7ecc40000 0x55e7ecc423f0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a60cac227bc806f server_cookie=612694f2d360d17e in_seq=0 out_seq=0 2024-03-23T23:07:30.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.743+0000 7f11b3e00640 10 client.26827 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:30.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.743+0000 7f11b3e00640 1 -- 172.21.15.12:0/2616420606 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e7ecb9d3f0 con 0x55e7ecc40000 2024-03-23T23:07:30.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.743+0000 7f11b3e00640 10 client.26827 handle_command_reply: tid=0 2024-03-23T23:07:30.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.743+0000 7f11dc011740 1 client.26827 shutdown 2024-03-23T23:07:30.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.744+0000 7f11dc011740 1 -- 172.21.15.12:0/2616420606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7ecc2a2b0 msgr2=0x55e7ecc344f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:30.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.744+0000 7f11dc011740 1 --2- 172.21.15.12:0/2616420606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7ecc2a2b0 0x55e7ecc344f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f11c400ebb0 tx=0x7f11c4018650 comp rx=0 tx=0).stop 2024-03-23T23:07:30.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.744+0000 7f11dc011740 1 -- 172.21.15.12:0/2616420606 shutdown_connections 2024-03-23T23:07:30.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.744+0000 7f11dc011740 1 --2- 172.21.15.12:0/2616420606 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55e7ecc40000 0x55e7ecc423f0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.744+0000 7f11dc011740 1 --2- 172.21.15.12:0/2616420606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e7ecc2fe10 0x55e7ecc3ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.744+0000 7f11dc011740 1 --2- 172.21.15.12:0/2616420606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7ecc2aca0 0x55e7ecc34a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.744+0000 7f11dc011740 1 --2- 172.21.15.12:0/2616420606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7ecc2a2b0 0x55e7ecc344f0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.744+0000 7f11dc011740 1 -- 172.21.15.12:0/2616420606 >> 172.21.15.12:0/2616420606 conn(0x55e7ecc25820 msgr2=0x55e7ecc24a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:30.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.744+0000 7f11b3e00640 0 client.26827 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:30.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.744+0000 7f11dc011740 1 -- 172.21.15.12:0/2616420606 shutdown_connections 2024-03-23T23:07:30.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.744+0000 7f11dc011740 1 -- 172.21.15.12:0/2616420606 wait complete. 2024-03-23T23:07:30.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.744+0000 7f11dc011740 20 client.26827 trim_cache size 0 max 16384 2024-03-23T23:07:30.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.762+0000 7f11dc011740 1 Processor -- start 2024-03-23T23:07:30.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.762+0000 7f11dc011740 1 -- start start 2024-03-23T23:07:30.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.762+0000 7f11dc011740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e7ecc2a2b0 0x55e7ecc24900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:30.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.762+0000 7f11dc011740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7ecc2aca0 0x55e7ecc46c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:30.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.762+0000 7f11dc011740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7ecc2fe10 0x55e7ecc47150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:30.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.762+0000 7f11dc011740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d41b1310 con 0x55e7ecc2a2b0 2024-03-23T23:07:30.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.762+0000 7f11dc011740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d41aee10 con 0x55e7ecc2fe10 2024-03-23T23:07:30.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.762+0000 7f11dc011740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d41aceb0 con 0x55e7ecc2aca0 2024-03-23T23:07:30.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.763+0000 7f11d1e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e7ecc2a2b0 0x55e7ecc24900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:30.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.763+0000 7f11d1e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e7ecc2a2b0 0x55e7ecc24900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60772/0 (socket says 172.21.15.12:60772) 2024-03-23T23:07:30.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.763+0000 7f11d1400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7ecc2aca0 0x55e7ecc46c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:30.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.763+0000 7f11d1e00640 1 -- 172.21.15.12:0/1802213999 learned_addr learned my addr 172.21.15.12:0/1802213999 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:30.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.763+0000 7f11d2800640 1 --2- 172.21.15.12:0/1802213999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7ecc2fe10 0x55e7ecc47150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:30.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.763+0000 7f11d1e00640 1 -- 172.21.15.12:0/1802213999 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7ecc2aca0 msgr2=0x55e7ecc46c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:30.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.763+0000 7f11d1e00640 1 --2- 172.21.15.12:0/1802213999 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7ecc2aca0 0x55e7ecc46c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.763+0000 7f11d1e00640 1 -- 172.21.15.12:0/1802213999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7ecc2fe10 msgr2=0x55e7ecc47150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:30.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.763+0000 7f11d1e00640 1 --2- 172.21.15.12:0/1802213999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7ecc2fe10 0x55e7ecc47150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.763+0000 7f11d1e00640 1 -- 172.21.15.12:0/1802213999 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e7ecc2f190 con 0x55e7ecc2a2b0 2024-03-23T23:07:30.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.763+0000 7f11d1400640 1 --2- 172.21.15.12:0/1802213999 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7ecc2aca0 0x55e7ecc46c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:07:30.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.764+0000 7f11d1e00640 1 --2- 172.21.15.12:0/1802213999 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e7ecc2a2b0 0x55e7ecc24900 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f11c4022d00 tx=0x7f11c4009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce88b5d428936f9f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:30.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.764+0000 7f11bea00640 1 -- 172.21.15.12:0/1802213999 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c4009ae0 con 0x55e7ecc2a2b0 2024-03-23T23:07:30.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.764+0000 7f11bea00640 1 -- 172.21.15.12:0/1802213999 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11c403a070 con 0x55e7ecc2a2b0 2024-03-23T23:07:30.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.764+0000 7f11bea00640 1 -- 172.21.15.12:0/1802213999 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c402aae0 con 0x55e7ecc2a2b0 2024-03-23T23:07:30.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.765+0000 7f11dc011740 1 -- 172.21.15.12:0/1802213999 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e7ecc2a2b0 msgr2=0x55e7ecc24900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:30.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.765+0000 7f11dc011740 1 --2- 172.21.15.12:0/1802213999 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e7ecc2a2b0 0x55e7ecc24900 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f11c4022d00 tx=0x7f11c4009610 comp rx=0 tx=0).stop 2024-03-23T23:07:30.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.765+0000 7f11dc011740 1 -- 172.21.15.12:0/1802213999 shutdown_connections 2024-03-23T23:07:30.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.765+0000 7f11dc011740 1 --2- 172.21.15.12:0/1802213999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7ecc2fe10 0x55e7ecc47150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.765+0000 7f11dc011740 1 --2- 172.21.15.12:0/1802213999 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7ecc2aca0 0x55e7ecc46c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.765+0000 7f11dc011740 1 --2- 172.21.15.12:0/1802213999 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e7ecc2a2b0 0x55e7ecc24900 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.765+0000 7f11dc011740 1 -- 172.21.15.12:0/1802213999 >> 172.21.15.12:0/1802213999 conn(0x55e7ecc25820 msgr2=0x55e7ecc241f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:30.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.765+0000 7f11dc011740 1 -- 172.21.15.12:0/1802213999 shutdown_connections 2024-03-23T23:07:30.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.765+0000 7f11dc011740 1 -- 172.21.15.12:0/1802213999 wait complete. 2024-03-23T23:07:30.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.765+0000 7f11dc011740 1 -- start start 2024-03-23T23:07:30.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.765+0000 7f11dc011740 1 Processor -- start 2024-03-23T23:07:30.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.766+0000 7f11dc011740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e7ecc2a2b0 0x55e7ecc38f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:30.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.766+0000 7f11dc011740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7ecc2aca0 0x55e7ecc39490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:30.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.766+0000 7f11dc011740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7ecc2fe10 0x55e7ecc3a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:30.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.766+0000 7f11dc011740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d41ad4b0 con 0x55e7ecc2a2b0 2024-03-23T23:07:30.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.766+0000 7f11dc011740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d41ae7b0 con 0x55e7ecc2aca0 2024-03-23T23:07:30.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.766+0000 7f11dc011740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d41adaf0 con 0x55e7ecc2fe10 2024-03-23T23:07:30.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.766+0000 7f11dc011740 10 client.0 _ll_register_callbacks cb 0x7f11c0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:30.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.766+0000 7f11d1e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e7ecc2a2b0 0x55e7ecc38f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:30.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.766+0000 7f11d1e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e7ecc2a2b0 0x55e7ecc38f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60786/0 (socket says 172.21.15.12:60786) 2024-03-23T23:07:30.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.766+0000 7f11d1e00640 1 -- 172.21.15.12:0/2387955270 learned_addr learned my addr 172.21.15.12:0/2387955270 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:30.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.766+0000 7f11d2800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7ecc2fe10 0x55e7ecc3a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:30.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.766+0000 7f11d1400640 1 --2- 172.21.15.12:0/2387955270 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7ecc2aca0 0x55e7ecc39490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:30.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.767+0000 7f11d1e00640 1 -- 172.21.15.12:0/2387955270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7ecc2fe10 msgr2=0x55e7ecc3a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:30.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.767+0000 7f11d1e00640 1 --2- 172.21.15.12:0/2387955270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7ecc2fe10 0x55e7ecc3a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.767+0000 7f11d1e00640 1 -- 172.21.15.12:0/2387955270 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7ecc2aca0 msgr2=0x55e7ecc39490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:30.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.767+0000 7f11d1e00640 1 --2- 172.21.15.12:0/2387955270 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7ecc2aca0 0x55e7ecc39490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.767+0000 7f11d1e00640 1 -- 172.21.15.12:0/2387955270 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e7ecc3aa40 con 0x55e7ecc2a2b0 2024-03-23T23:07:30.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.767+0000 7f11d2800640 1 --2- 172.21.15.12:0/2387955270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7ecc2fe10 0x55e7ecc3a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:07:30.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.767+0000 7f11d1e00640 1 --2- 172.21.15.12:0/2387955270 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e7ecc2a2b0 0x55e7ecc38f50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f11c400ebb0 tx=0x7f11c4009b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=175b79310f7574c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:30.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.767+0000 7f11b3e00640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:07:30.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.767+0000 7f11b3e00640 1 -- 172.21.15.12:0/2387955270 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c4042070 con 0x55e7ecc2a2b0 2024-03-23T23:07:30.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.767+0000 7f11dc011740 1 -- 172.21.15.12:0/2387955270 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f11c403a070 con 0x55e7ecc2a2b0 2024-03-23T23:07:30.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.767+0000 7f11b3e00640 1 -- 172.21.15.12:0/2387955270 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f11c402ac70 con 0x55e7ecc2a2b0 2024-03-23T23:07:30.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.768+0000 7f11b3e00640 1 -- 172.21.15.12:0/2387955270 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11c404e050 con 0x55e7ecc2a2b0 2024-03-23T23:07:30.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.768+0000 7f11dc011740 10 client.16740 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:30.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.768+0000 7f11dc011740 1 -- 172.21.15.12:0/2387955270 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55e7ecc9be40 con 0x55e7ecc2a2b0 2024-03-23T23:07:30.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.768+0000 7f11b3e00640 1 -- 172.21.15.12:0/2387955270 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11c403a070 con 0x55e7ecc2a2b0 2024-03-23T23:07:30.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.768+0000 7f11b3e00640 1 -- 172.21.15.12:0/2387955270 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f11c403e450 con 0x55e7ecc2a2b0 2024-03-23T23:07:30.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.768+0000 7f11dc011740 10 client.16740 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:30.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.768+0000 7f11dc011740 10 client.16740 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:07:30.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.768+0000 7f11dc011740 20 client.16740 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:30.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.768+0000 7f11dc011740 1 --2- 172.21.15.12:0/2387955270 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55e7ecc9c5e0 0x55e7ecc9ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:30.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.768+0000 7f11dc011740 4 client.16740 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-23T23:07:30.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.768+0000 7f11dc011740 1 -- 172.21.15.12:0/2387955270 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55e7ecb97740 con 0x55e7ecc9c5e0 2024-03-23T23:07:30.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.770+0000 7f11d1400640 1 --2- 172.21.15.12:0/2387955270 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55e7ecc9c5e0 0x55e7ecc9ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:30.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.773+0000 7f11d1400640 1 --2- 172.21.15.12:0/2387955270 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55e7ecc9c5e0 0x55e7ecc9ea80 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6c5463c47fb1240 server_cookie=6a5437bbd4dec402 in_seq=0 out_seq=0 2024-03-23T23:07:30.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.773+0000 7f11b3e00640 10 client.16740 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:30.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.774+0000 7f11b3e00640 1 -- 172.21.15.12:0/2387955270 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55e7ecb97740 con 0x55e7ecc9c5e0 2024-03-23T23:07:30.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.774+0000 7f11b3e00640 10 client.16740 handle_command_reply: tid=0 2024-03-23T23:07:30.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.774+0000 7f11dc011740 1 client.16740 shutdown 2024-03-23T23:07:30.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.774+0000 7f11dc011740 1 -- 172.21.15.12:0/2387955270 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e7ecc2a2b0 msgr2=0x55e7ecc38f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:30.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.774+0000 7f11dc011740 1 --2- 172.21.15.12:0/2387955270 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e7ecc2a2b0 0x55e7ecc38f50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f11c400ebb0 tx=0x7f11c4009b10 comp rx=0 tx=0).stop 2024-03-23T23:07:30.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.774+0000 7f11dc011740 1 -- 172.21.15.12:0/2387955270 shutdown_connections 2024-03-23T23:07:30.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.774+0000 7f11dc011740 1 --2- 172.21.15.12:0/2387955270 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55e7ecc9c5e0 0x55e7ecc9ea80 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.774+0000 7f11dc011740 1 --2- 172.21.15.12:0/2387955270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7ecc2fe10 0x55e7ecc3a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.774+0000 7f11dc011740 1 --2- 172.21.15.12:0/2387955270 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7ecc2aca0 0x55e7ecc39490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.774+0000 7f11dc011740 1 --2- 172.21.15.12:0/2387955270 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e7ecc2a2b0 0x55e7ecc38f50 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.774+0000 7f11dc011740 1 -- 172.21.15.12:0/2387955270 >> 172.21.15.12:0/2387955270 conn(0x55e7ecc25820 msgr2=0x55e7ecc24a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:30.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.774+0000 7f11b3e00640 0 client.16740 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:30.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.775+0000 7f11dc011740 1 -- 172.21.15.12:0/2387955270 shutdown_connections 2024-03-23T23:07:30.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.775+0000 7f11dc011740 1 -- 172.21.15.12:0/2387955270 wait complete. 2024-03-23T23:07:30.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.775+0000 7f11dc011740 20 client.16740 trim_cache size 0 max 16384 2024-03-23T23:07:30.776 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:07:30.776 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-03-23T23:07:30.776 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "43975097-4db0-4a5c-b7af-62652b78d883", 2024-03-23T23:07:30.776 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-03-23T23:07:30.776 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:07:30.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.775+0000 7f11d3200640 1 -- 172.21.15.12:0/237106472 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f11b80a2880 msgr2=0x7f11b80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:30.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.776+0000 7f11d3200640 1 --2- 172.21.15.12:0/237106472 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f11b80a2880 0x7f11b80a4d40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f11c002c8e0 tx=0x7f11c0005b50 comp rx=0 tx=0).stop 2024-03-23T23:07:30.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.776+0000 7f11d3200640 1 -- 172.21.15.12:0/237106472 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11d4109770 msgr2=0x7f11d41a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:30.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.776+0000 7f11d3200640 1 --2- 172.21.15.12:0/237106472 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11d4109770 0x7f11d41a5460 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f11cc00d680 tx=0x7f11cc00db40 comp rx=0 tx=0).stop 2024-03-23T23:07:30.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.776+0000 7f11d3200640 1 -- 172.21.15.12:0/237106472 shutdown_connections 2024-03-23T23:07:30.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.776+0000 7f11d3200640 1 --2- 172.21.15.12:0/237106472 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f11b80a2880 0x7f11b80a4d40 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.776+0000 7f11d3200640 1 --2- 172.21.15.12:0/237106472 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11d4109770 0x7f11d41a5460 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.776+0000 7f11d3200640 1 --2- 172.21.15.12:0/237106472 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11d4108db0 0x7f11d419e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.776+0000 7f11d3200640 1 --2- 172.21.15.12:0/237106472 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11d4107bc0 0x7f11d419deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:30.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.776+0000 7f11d3200640 1 -- 172.21.15.12:0/237106472 >> 172.21.15.12:0/237106472 conn(0x7f11d4076b70 msgr2=0x7f11d410c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:30.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.776+0000 7f11d3200640 1 -- 172.21.15.12:0/237106472 shutdown_connections 2024-03-23T23:07:30.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:30.776+0000 7f11d3200640 1 -- 172.21.15.12:0/237106472 wait complete. 2024-03-23T23:07:30.795 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:07:30.795 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:07:30.869 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:30 smithi012 ceph-mon[31695]: pgmap v561: 97 pgs: 97 active+clean; 3.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 13 op/s 2024-03-23T23:07:31.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.006+0000 7f708c800640 1 -- 172.21.15.12:0/3062068388 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7084109870 msgr2=0x7f7084110100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:31.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.006+0000 7f708c800640 1 --2- 172.21.15.12:0/3062068388 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7084109870 0x7f7084110100 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7080009be0 tx=0x7f708002bef0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.006+0000 7f708c800640 1 -- 172.21.15.12:0/3062068388 shutdown_connections 2024-03-23T23:07:31.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.006+0000 7f708c800640 1 --2- 172.21.15.12:0/3062068388 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7084109870 0x7f7084110100 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.006+0000 7f708c800640 1 --2- 172.21.15.12:0/3062068388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7084108eb0 0x7f7084109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.006+0000 7f708c800640 1 --2- 172.21.15.12:0/3062068388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7084107cc0 0x7f70841080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.006+0000 7f708c800640 1 -- 172.21.15.12:0/3062068388 >> 172.21.15.12:0/3062068388 conn(0x7f7084076c60 msgr2=0x7f70840790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:31.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.006+0000 7f708c800640 1 -- 172.21.15.12:0/3062068388 shutdown_connections 2024-03-23T23:07:31.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.006+0000 7f708c800640 1 -- 172.21.15.12:0/3062068388 wait complete. 2024-03-23T23:07:31.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.007+0000 7f708c800640 1 Processor -- start 2024-03-23T23:07:31.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.008+0000 7f708c800640 1 -- start start 2024-03-23T23:07:31.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.008+0000 7f708c800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7084107cc0 0x7f708419df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.008+0000 7f708c800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7084108eb0 0x7f708419e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.008+0000 7f708c800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7084109870 0x7f70841a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.008+0000 7f708c800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7084113920 con 0x7f7084108eb0 2024-03-23T23:07:31.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.008+0000 7f708c800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70841137a0 con 0x7f7084109870 2024-03-23T23:07:31.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.008+0000 7f708c800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7084113aa0 con 0x7f7084107cc0 2024-03-23T23:07:31.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.008+0000 7f7089400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7084108eb0 0x7f708419e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.008+0000 7f708a800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7084109870 0x7f70841a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.008+0000 7f7089e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7084107cc0 0x7f708419df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.008+0000 7f708a800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7084109870 0x7f70841a5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44280/0 (socket says 172.21.15.12:44280) 2024-03-23T23:07:31.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.008+0000 7f708a800640 1 -- 172.21.15.12:0/3454718712 learned_addr learned my addr 172.21.15.12:0/3454718712 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:31.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.009+0000 7f7089400640 1 -- 172.21.15.12:0/3454718712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7084107cc0 msgr2=0x7f708419df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:31.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.009+0000 7f7089400640 1 --2- 172.21.15.12:0/3454718712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7084107cc0 0x7f708419df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.009+0000 7f7089400640 1 -- 172.21.15.12:0/3454718712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7084109870 msgr2=0x7f70841a5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:31.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.009+0000 7f7089400640 1 --2- 172.21.15.12:0/3454718712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7084109870 0x7f70841a5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.009+0000 7f7089400640 1 -- 172.21.15.12:0/3454718712 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70841a5b80 con 0x7f7084108eb0 2024-03-23T23:07:31.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.009+0000 7f708a800640 1 --2- 172.21.15.12:0/3454718712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7084109870 0x7f70841a5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:07:31.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.009+0000 7f7089400640 1 --2- 172.21.15.12:0/3454718712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7084108eb0 0x7f708419e4a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f707400e790 tx=0x7f707400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:31.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.009+0000 7f707ea00640 1 -- 172.21.15.12:0/3454718712 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707400eea0 con 0x7f7084108eb0 2024-03-23T23:07:31.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.009+0000 7f707ea00640 1 -- 172.21.15.12:0/3454718712 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f707400ccc0 con 0x7f7084108eb0 2024-03-23T23:07:31.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.009+0000 7f707ea00640 1 -- 172.21.15.12:0/3454718712 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7074018780 con 0x7f7084108eb0 2024-03-23T23:07:31.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.009+0000 7f708c800640 1 -- 172.21.15.12:0/3454718712 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70841a5e70 con 0x7f7084108eb0 2024-03-23T23:07:31.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.010+0000 7f708c800640 1 -- 172.21.15.12:0/3454718712 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70841a62e0 con 0x7f7084108eb0 2024-03-23T23:07:31.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.013+0000 7f708f3e0740 1 Processor -- start 2024-03-23T23:07:31.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.013+0000 7f708f3e0740 1 -- start start 2024-03-23T23:07:31.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.013+0000 7f707ea00640 1 -- 172.21.15.12:0/3454718712 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7074010430 con 0x7f7084108eb0 2024-03-23T23:07:31.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.014+0000 7f708f3e0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f2f27be80 0x556f2f27c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.014+0000 7f708f3e0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f2f27c870 0x556f2f281370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.014+0000 7f708f3e0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f2f2819e0 0x556f2f283e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.014+0000 7f708f3e0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f2f1e1250 con 0x556f2f27be80 2024-03-23T23:07:31.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.014+0000 7f708f3e0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f2f189a30 con 0x556f2f27c870 2024-03-23T23:07:31.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.014+0000 7f708f3e0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f2f1dd2f0 con 0x556f2f2819e0 2024-03-23T23:07:31.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.014+0000 7f708a800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f2f27be80 0x556f2f27c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.014+0000 7f7089e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f2f27c870 0x556f2f281370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.014+0000 7f708a800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f2f27be80 0x556f2f27c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60810/0 (socket says 172.21.15.12:60810) 2024-03-23T23:07:31.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.014+0000 7f708a800640 1 -- 172.21.15.12:0/558266474 learned_addr learned my addr 172.21.15.12:0/558266474 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:31.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.014+0000 7f7089400640 1 --2- 172.21.15.12:0/558266474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f2f2819e0 0x556f2f283e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.014+0000 7f7089e00640 1 -- 172.21.15.12:0/558266474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f2f2819e0 msgr2=0x556f2f283e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:31.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.014+0000 7f7089e00640 1 --2- 172.21.15.12:0/558266474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f2f2819e0 0x556f2f283e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.014+0000 7f7089e00640 1 -- 172.21.15.12:0/558266474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f2f27be80 msgr2=0x556f2f27c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:31.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.014+0000 7f7089e00640 1 --2- 172.21.15.12:0/558266474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f2f27be80 0x556f2f27c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.014+0000 7f7089e00640 1 -- 172.21.15.12:0/558266474 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f2f174690 con 0x556f2f27c870 2024-03-23T23:07:31.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.014+0000 7f708a800640 1 --2- 172.21.15.12:0/558266474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f2f27be80 0x556f2f27c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:07:31.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.015+0000 7f7089e00640 1 --2- 172.21.15.12:0/558266474 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f2f27c870 0x556f2f281370 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f707800bbc0 tx=0x7f70780107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e784c8a9df147d1c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:31.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.015+0000 7f707ea00640 1 --2- 172.21.15.12:0/3454718712 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f706c0a2800 0x7f706c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.015+0000 7f7073000640 1 -- 172.21.15.12:0/558266474 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7078004030 con 0x556f2f27c870 2024-03-23T23:07:31.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.015+0000 7f7073000640 1 -- 172.21.15.12:0/558266474 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7078005500 con 0x556f2f27c870 2024-03-23T23:07:31.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.015+0000 7f7073000640 1 -- 172.21.15.12:0/558266474 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7078004cb0 con 0x556f2f27c870 2024-03-23T23:07:31.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.015+0000 7f707ea00640 1 -- 172.21.15.12:0/3454718712 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f70740ceb80 con 0x7f7084108eb0 2024-03-23T23:07:31.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.015+0000 7f708a800640 1 --2- 172.21.15.12:0/3454718712 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f706c0a2800 0x7f706c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.016+0000 7f708f3e0740 1 -- 172.21.15.12:0/558266474 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f2f27c870 msgr2=0x556f2f281370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:31.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.016+0000 7f708f3e0740 1 --2- 172.21.15.12:0/558266474 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f2f27c870 0x556f2f281370 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f707800bbc0 tx=0x7f70780107a0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.016+0000 7f708f3e0740 1 -- 172.21.15.12:0/558266474 shutdown_connections 2024-03-23T23:07:31.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.016+0000 7f708f3e0740 1 --2- 172.21.15.12:0/558266474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f2f2819e0 0x556f2f283e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.016+0000 7f708f3e0740 1 --2- 172.21.15.12:0/558266474 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f2f27c870 0x556f2f281370 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.016+0000 7f708f3e0740 1 --2- 172.21.15.12:0/558266474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f2f27be80 0x556f2f27c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.016+0000 7f708f3e0740 1 -- 172.21.15.12:0/558266474 >> 172.21.15.12:0/558266474 conn(0x556f2f244200 msgr2=0x556f2f2777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:31.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.016+0000 7f708f3e0740 1 -- 172.21.15.12:0/558266474 shutdown_connections 2024-03-23T23:07:31.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.016+0000 7f708a800640 1 --2- 172.21.15.12:0/3454718712 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f706c0a2800 0x7f706c0a4cc0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f70800099a0 tx=0x7f7080004a80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:31.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.016+0000 7f708f3e0740 1 -- 172.21.15.12:0/558266474 wait complete. 2024-03-23T23:07:31.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.017+0000 7f708f3e0740 1 -- start start 2024-03-23T23:07:31.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.017+0000 7f708f3e0740 1 Processor -- start 2024-03-23T23:07:31.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.017+0000 7f708f3e0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f2f27be80 0x556f2f285e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.017+0000 7f708f3e0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f2f27c870 0x556f2f2863d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.017+0000 7f708f3e0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f2f2819e0 0x556f2f28d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.017+0000 7f708f3e0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f2f284990 con 0x556f2f27c870 2024-03-23T23:07:31.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.017+0000 7f708f3e0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f2f19ce80 con 0x556f2f2819e0 2024-03-23T23:07:31.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.017+0000 7f708f3e0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f2f284b10 con 0x556f2f27be80 2024-03-23T23:07:31.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.017+0000 7f708f3e0740 10 client.0 _ll_register_callbacks cb 0x556f2f1af130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:31.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.017+0000 7f7089e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f2f27be80 0x556f2f285e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.017+0000 7f7089e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f2f27be80 0x556f2f285e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:46008/0 (socket says 172.21.15.12:46008) 2024-03-23T23:07:31.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.018+0000 7f7089e00640 1 -- 172.21.15.12:0/1483136344 learned_addr learned my addr 172.21.15.12:0/1483136344 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:31.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.018+0000 7f7089400640 1 --2- 172.21.15.12:0/1483136344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f2f27c870 0x556f2f2863d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.018+0000 7f708a800640 1 --2- 172.21.15.12:0/1483136344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f2f2819e0 0x556f2f28d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.018+0000 7f7089e00640 1 -- 172.21.15.12:0/1483136344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f2f2819e0 msgr2=0x556f2f28d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:31.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.018+0000 7f7089e00640 1 --2- 172.21.15.12:0/1483136344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f2f2819e0 0x556f2f28d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.018+0000 7f7089e00640 1 -- 172.21.15.12:0/1483136344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f2f27c870 msgr2=0x556f2f2863d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:31.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.018+0000 7f7089e00640 1 --2- 172.21.15.12:0/1483136344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f2f27c870 0x556f2f2863d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.018+0000 7f7089e00640 1 -- 172.21.15.12:0/1483136344 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556f2efbf7d0 con 0x556f2f27be80 2024-03-23T23:07:31.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.018+0000 7f708a800640 1 --2- 172.21.15.12:0/1483136344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f2f2819e0 0x556f2f28d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:07:31.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.018+0000 7f7089400640 1 --2- 172.21.15.12:0/1483136344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f2f27c870 0x556f2f2863d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:07:31.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.018+0000 7f7089e00640 1 --2- 172.21.15.12:0/1483136344 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f2f27be80 0x556f2f285e90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7078005cd0 tx=0x7f707800bfa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd6d408a299e5708 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:31.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.018+0000 7f7067400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:07:31.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.018+0000 7f7067400640 1 -- 172.21.15.12:0/1483136344 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707802b060 con 0x556f2f27be80 2024-03-23T23:07:31.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.018+0000 7f7067400640 1 -- 172.21.15.12:0/1483136344 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7078004bc0 con 0x556f2f27be80 2024-03-23T23:07:31.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.018+0000 7f708f3e0740 1 -- 172.21.15.12:0/1483136344 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7078005500 con 0x556f2f27be80 2024-03-23T23:07:31.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.019+0000 7f7067400640 1 -- 172.21.15.12:0/1483136344 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7078004440 con 0x556f2f27be80 2024-03-23T23:07:31.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.019+0000 7f708f3e0740 10 client.26845 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:31.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.019+0000 7f708f3e0740 1 -- 172.21.15.12:0/1483136344 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556f2eef53f0 con 0x556f2f27be80 2024-03-23T23:07:31.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.019+0000 7f7067400640 1 -- 172.21.15.12:0/1483136344 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7078005500 con 0x556f2f27be80 2024-03-23T23:07:31.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.019+0000 7f7067400640 1 -- 172.21.15.12:0/1483136344 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f7078011ac0 con 0x556f2f27be80 2024-03-23T23:07:31.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.020+0000 7f708f3e0740 10 client.26845 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:31.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.020+0000 7f708f3e0740 10 client.26845 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:07:31.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.020+0000 7f708f3e0740 20 client.26845 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:31.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.020+0000 7f708f3e0740 1 --2- 172.21.15.12:0/1483136344 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x556f2f291a60 0x556f2f293e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.020+0000 7f708f3e0740 4 client.26845 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:07:31.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.020+0000 7f708f3e0740 1 -- 172.21.15.12:0/1483136344 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556f2f1ee790 con 0x556f2f291a60 2024-03-23T23:07:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.020+0000 7f7089400640 1 --2- 172.21.15.12:0/1483136344 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x556f2f291a60 0x556f2f293e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.020+0000 7f7089400640 1 --2- 172.21.15.12:0/1483136344 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x556f2f291a60 0x556f2f293e50 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f55103475e1de0e server_cookie=ea12ac6b37a4189 in_seq=0 out_seq=0 2024-03-23T23:07:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.020+0000 7f7067400640 10 client.26845 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.021+0000 7f7067400640 1 -- 172.21.15.12:0/1483136344 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556f2f1ee790 con 0x556f2f291a60 2024-03-23T23:07:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.021+0000 7f7067400640 10 client.26845 handle_command_reply: tid=0 2024-03-23T23:07:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.021+0000 7f708f3e0740 1 client.26845 shutdown 2024-03-23T23:07:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.021+0000 7f708f3e0740 1 -- 172.21.15.12:0/1483136344 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f2f27be80 msgr2=0x556f2f285e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.021+0000 7f708f3e0740 1 --2- 172.21.15.12:0/1483136344 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f2f27be80 0x556f2f285e90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7078005cd0 tx=0x7f707800bfa0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.021+0000 7f708f3e0740 1 -- 172.21.15.12:0/1483136344 shutdown_connections 2024-03-23T23:07:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.021+0000 7f708f3e0740 1 --2- 172.21.15.12:0/1483136344 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x556f2f291a60 0x556f2f293e50 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.021+0000 7f708f3e0740 1 --2- 172.21.15.12:0/1483136344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f2f2819e0 0x556f2f28d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.021+0000 7f708f3e0740 1 --2- 172.21.15.12:0/1483136344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f2f27c870 0x556f2f2863d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.021+0000 7f7067400640 0 client.26845 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.021+0000 7f708f3e0740 1 --2- 172.21.15.12:0/1483136344 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f2f27be80 0x556f2f285e90 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.021+0000 7f708f3e0740 1 -- 172.21.15.12:0/1483136344 >> 172.21.15.12:0/1483136344 conn(0x556f2f244200 msgr2=0x556f2f276a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.022+0000 7f708f3e0740 1 -- 172.21.15.12:0/1483136344 shutdown_connections 2024-03-23T23:07:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.022+0000 7f708f3e0740 1 -- 172.21.15.12:0/1483136344 wait complete. 2024-03-23T23:07:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.022+0000 7f708f3e0740 20 client.26845 trim_cache size 0 max 16384 2024-03-23T23:07:31.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.039+0000 7f708f3e0740 1 Processor -- start 2024-03-23T23:07:31.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.040+0000 7f708f3e0740 1 -- start start 2024-03-23T23:07:31.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.040+0000 7f708f3e0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f2f244200 0x556f2f27d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.040+0000 7f708f3e0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f2f27be80 0x556f2f28d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.040+0000 7f708f3e0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f2f27c870 0x556f2f2985f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.040+0000 7f708f3e0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70841b1540 con 0x556f2f244200 2024-03-23T23:07:31.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.040+0000 7f708f3e0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70841af040 con 0x556f2f27c870 2024-03-23T23:07:31.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.040+0000 7f708f3e0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70841ad0e0 con 0x556f2f27be80 2024-03-23T23:07:31.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.040+0000 7f7089e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f2f244200 0x556f2f27d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.040+0000 7f7089e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f2f244200 0x556f2f27d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60828/0 (socket says 172.21.15.12:60828) 2024-03-23T23:07:31.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.040+0000 7f708a800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f2f27c870 0x556f2f2985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.040+0000 7f7089e00640 1 -- 172.21.15.12:0/2672359678 learned_addr learned my addr 172.21.15.12:0/2672359678 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:31.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.040+0000 7f7089400640 1 --2- 172.21.15.12:0/2672359678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f2f27be80 0x556f2f28d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.040+0000 7f7089e00640 1 -- 172.21.15.12:0/2672359678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f2f27be80 msgr2=0x556f2f28d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:31.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.040+0000 7f7089e00640 1 --2- 172.21.15.12:0/2672359678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f2f27be80 0x556f2f28d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.040+0000 7f7089e00640 1 -- 172.21.15.12:0/2672359678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f2f27c870 msgr2=0x556f2f2985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:31.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.040+0000 7f7089e00640 1 --2- 172.21.15.12:0/2672359678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f2f27c870 0x556f2f2985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.040+0000 7f7089e00640 1 -- 172.21.15.12:0/2672359678 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f2eeb50f0 con 0x556f2f244200 2024-03-23T23:07:31.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.041+0000 7f7089e00640 1 --2- 172.21.15.12:0/2672359678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f2f244200 0x556f2f27d2e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7078011010 tx=0x7f7078004910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8924d353fccb6a80 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:31.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.041+0000 7f7072600640 1 -- 172.21.15.12:0/2672359678 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7078036070 con 0x556f2f244200 2024-03-23T23:07:31.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.041+0000 7f7072600640 1 -- 172.21.15.12:0/2672359678 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f707801f070 con 0x556f2f244200 2024-03-23T23:07:31.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.041+0000 7f7072600640 1 -- 172.21.15.12:0/2672359678 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707802b030 con 0x556f2f244200 2024-03-23T23:07:31.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.042+0000 7f708f3e0740 1 -- 172.21.15.12:0/2672359678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f2f244200 msgr2=0x556f2f27d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:31.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.042+0000 7f708f3e0740 1 --2- 172.21.15.12:0/2672359678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f2f244200 0x556f2f27d2e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7078011010 tx=0x7f7078004910 comp rx=0 tx=0).stop 2024-03-23T23:07:31.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.042+0000 7f708f3e0740 1 -- 172.21.15.12:0/2672359678 shutdown_connections 2024-03-23T23:07:31.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.042+0000 7f708f3e0740 1 --2- 172.21.15.12:0/2672359678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f2f27c870 0x556f2f2985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.042+0000 7f708f3e0740 1 --2- 172.21.15.12:0/2672359678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f2f27be80 0x556f2f28d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.042+0000 7f708f3e0740 1 --2- 172.21.15.12:0/2672359678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f2f244200 0x556f2f27d2e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.042+0000 7f708f3e0740 1 -- 172.21.15.12:0/2672359678 >> 172.21.15.12:0/2672359678 conn(0x556f2f2941c0 msgr2=0x556f2f276a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:31.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.042+0000 7f708f3e0740 1 -- 172.21.15.12:0/2672359678 shutdown_connections 2024-03-23T23:07:31.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.042+0000 7f708f3e0740 1 -- 172.21.15.12:0/2672359678 wait complete. 2024-03-23T23:07:31.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.042+0000 7f708f3e0740 1 -- start start 2024-03-23T23:07:31.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.043+0000 7f708f3e0740 1 Processor -- start 2024-03-23T23:07:31.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.043+0000 7f708f3e0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f2f244200 0x556f2f28aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.043+0000 7f708f3e0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f2f27be80 0x556f2f28b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.043+0000 7f708f3e0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f2f27c870 0x556f2f28c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.043+0000 7f708f3e0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70841ad6e0 con 0x556f2f27c870 2024-03-23T23:07:31.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.043+0000 7f708f3e0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70841ae9e0 con 0x556f2f244200 2024-03-23T23:07:31.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.043+0000 7f708f3e0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70841add20 con 0x556f2f27be80 2024-03-23T23:07:31.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.043+0000 7f708f3e0740 10 client.0 _ll_register_callbacks cb 0x7f7080009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:31.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.043+0000 7f7089400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f2f27be80 0x556f2f28b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.043+0000 7f708a800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f2f27c870 0x556f2f28c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.043+0000 7f7089400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f2f27be80 0x556f2f28b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:46022/0 (socket says 172.21.15.12:46022) 2024-03-23T23:07:31.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.043+0000 7f7089400640 1 -- 172.21.15.12:0/535559414 learned_addr learned my addr 172.21.15.12:0/535559414 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:31.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.043+0000 7f7089e00640 1 --2- 172.21.15.12:0/535559414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f2f244200 0x556f2f28aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.044+0000 7f7089400640 1 -- 172.21.15.12:0/535559414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f2f244200 msgr2=0x556f2f28aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:31.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.044+0000 7f7089400640 1 --2- 172.21.15.12:0/535559414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f2f244200 0x556f2f28aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.044+0000 7f7089400640 1 -- 172.21.15.12:0/535559414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f2f27c870 msgr2=0x556f2f28c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:31.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.044+0000 7f7089400640 1 --2- 172.21.15.12:0/535559414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f2f27c870 0x556f2f28c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.044+0000 7f7089400640 1 -- 172.21.15.12:0/535559414 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556f2f28c9e0 con 0x556f2f27be80 2024-03-23T23:07:31.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.044+0000 7f7089400640 1 --2- 172.21.15.12:0/535559414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f2f27be80 0x556f2f28b3e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f70740972c0 tx=0x7f7074010e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bf640e79865d966 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:31.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.044+0000 7f7067400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:07:31.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.044+0000 7f7067400640 1 -- 172.21.15.12:0/535559414 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70740d3040 con 0x556f2f27be80 2024-03-23T23:07:31.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.044+0000 7f708f3e0740 1 -- 172.21.15.12:0/535559414 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f707801f070 con 0x556f2f27be80 2024-03-23T23:07:31.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.044+0000 7f7067400640 1 -- 172.21.15.12:0/535559414 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f70740998e0 con 0x556f2f27be80 2024-03-23T23:07:31.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.044+0000 7f7067400640 1 -- 172.21.15.12:0/535559414 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f707409b5e0 con 0x556f2f27be80 2024-03-23T23:07:31.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.044+0000 7f708f3e0740 10 client.26851 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:31.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.044+0000 7f708f3e0740 1 -- 172.21.15.12:0/535559414 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556f2f2ed550 con 0x556f2f27be80 2024-03-23T23:07:31.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.045+0000 7f7067400640 1 -- 172.21.15.12:0/535559414 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f707801f070 con 0x556f2f27be80 2024-03-23T23:07:31.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.045+0000 7f7067400640 1 -- 172.21.15.12:0/535559414 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f7074097c50 con 0x556f2f27be80 2024-03-23T23:07:31.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.045+0000 7f708f3e0740 10 client.26851 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:31.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.045+0000 7f708f3e0740 10 client.26851 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:07:31.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.045+0000 7f708f3e0740 20 client.26851 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:31.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.045+0000 7f708f3e0740 1 --2- 172.21.15.12:0/535559414 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x556f2f2819e0 0x556f2f2efe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.045+0000 7f708f3e0740 4 client.26851 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:07:31.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.045+0000 7f708f3e0740 1 -- 172.21.15.12:0/535559414 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556f2f1e8ae0 con 0x556f2f2819e0 2024-03-23T23:07:31.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.045+0000 7f7089e00640 1 --2- 172.21.15.12:0/535559414 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x556f2f2819e0 0x556f2f2efe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.045+0000 7f7067400640 10 client.26851 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:31.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.045+0000 7f7089e00640 1 --2- 172.21.15.12:0/535559414 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x556f2f2819e0 0x556f2f2efe90 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4d205b31da3fb97 server_cookie=774e47700b30b8ff in_seq=0 out_seq=0 2024-03-23T23:07:31.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.046+0000 7f7067400640 1 -- 172.21.15.12:0/535559414 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x556f2f1e8ae0 con 0x556f2f2819e0 2024-03-23T23:07:31.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.046+0000 7f7067400640 10 client.26851 handle_command_reply: tid=0 2024-03-23T23:07:31.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.046+0000 7f708f3e0740 1 client.26851 shutdown 2024-03-23T23:07:31.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.046+0000 7f708f3e0740 1 -- 172.21.15.12:0/535559414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f2f27be80 msgr2=0x556f2f28b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:31.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.046+0000 7f708f3e0740 1 --2- 172.21.15.12:0/535559414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f2f27be80 0x556f2f28b3e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f70740972c0 tx=0x7f7074010e30 comp rx=0 tx=0).stop 2024-03-23T23:07:31.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.046+0000 7f708f3e0740 1 -- 172.21.15.12:0/535559414 shutdown_connections 2024-03-23T23:07:31.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.046+0000 7f708f3e0740 1 --2- 172.21.15.12:0/535559414 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x556f2f2819e0 0x556f2f2efe90 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.046+0000 7f708f3e0740 1 --2- 172.21.15.12:0/535559414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556f2f27c870 0x556f2f28c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.046+0000 7f708f3e0740 1 --2- 172.21.15.12:0/535559414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f2f27be80 0x556f2f28b3e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.046+0000 7f708f3e0740 1 --2- 172.21.15.12:0/535559414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f2f244200 0x556f2f28aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.046+0000 7f7067400640 0 client.26851 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:31.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.046+0000 7f708f3e0740 1 -- 172.21.15.12:0/535559414 >> 172.21.15.12:0/535559414 conn(0x556f2f2941c0 msgr2=0x556f2f276a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:31.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.046+0000 7f708f3e0740 1 -- 172.21.15.12:0/535559414 shutdown_connections 2024-03-23T23:07:31.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.046+0000 7f708f3e0740 1 -- 172.21.15.12:0/535559414 wait complete. 2024-03-23T23:07:31.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.046+0000 7f708f3e0740 20 client.26851 trim_cache size 0 max 16384 2024-03-23T23:07:31.051 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:07:31.051 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-03-23T23:07:31.051 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-03-23T23:07:31.051 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:07:31.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.047+0000 7f708c800640 1 -- 172.21.15.12:0/3454718712 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f706c0a2800 msgr2=0x7f706c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:31.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.047+0000 7f708c800640 1 --2- 172.21.15.12:0/3454718712 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f706c0a2800 0x7f706c0a4cc0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f70800099a0 tx=0x7f7080004a80 comp rx=0 tx=0).stop 2024-03-23T23:07:31.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.047+0000 7f708c800640 1 -- 172.21.15.12:0/3454718712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7084108eb0 msgr2=0x7f708419e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:31.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.047+0000 7f708c800640 1 --2- 172.21.15.12:0/3454718712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7084108eb0 0x7f708419e4a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f707400e790 tx=0x7f707400ec50 comp rx=0 tx=0).stop 2024-03-23T23:07:31.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.047+0000 7f708c800640 1 -- 172.21.15.12:0/3454718712 shutdown_connections 2024-03-23T23:07:31.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.047+0000 7f708c800640 1 --2- 172.21.15.12:0/3454718712 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f706c0a2800 0x7f706c0a4cc0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.047+0000 7f708c800640 1 --2- 172.21.15.12:0/3454718712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7084109870 0x7f70841a5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.047+0000 7f708c800640 1 --2- 172.21.15.12:0/3454718712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7084108eb0 0x7f708419e4a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.047+0000 7f708c800640 1 --2- 172.21.15.12:0/3454718712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7084107cc0 0x7f708419df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.047+0000 7f708c800640 1 -- 172.21.15.12:0/3454718712 >> 172.21.15.12:0/3454718712 conn(0x7f7084076c60 msgr2=0x7f708410c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:31.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.047+0000 7f708c800640 1 -- 172.21.15.12:0/3454718712 shutdown_connections 2024-03-23T23:07:31.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.047+0000 7f708c800640 1 -- 172.21.15.12:0/3454718712 wait complete. 2024-03-23T23:07:31.064 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T23:07:31.065 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:07:31.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:30 smithi173 ceph-mon[36116]: pgmap v561: 97 pgs: 97 active+clean; 3.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 13 op/s 2024-03-23T23:07:31.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:30 smithi203 ceph-mon[36693]: pgmap v561: 97 pgs: 97 active+clean; 3.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 13 op/s 2024-03-23T23:07:31.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.270+0000 7fc12f000640 1 -- 172.21.15.12:0/869503257 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc128101500 msgr2=0x7fc128101920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:31.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.270+0000 7fc12f000640 1 --2- 172.21.15.12:0/869503257 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc128101500 0x7fc128101920 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fc11c009900 tx=0x7fc11c02bc40 comp rx=0 tx=0).stop 2024-03-23T23:07:31.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.270+0000 7fc12f000640 1 -- 172.21.15.12:0/869503257 shutdown_connections 2024-03-23T23:07:31.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.270+0000 7fc12f000640 1 --2- 172.21.15.12:0/869503257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1281030b0 0x7fc128109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.270+0000 7fc12f000640 1 --2- 172.21.15.12:0/869503257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1281026f0 0x7fc128102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.270+0000 7fc12f000640 1 --2- 172.21.15.12:0/869503257 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc128101500 0x7fc128101920 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.270+0000 7fc12f000640 1 -- 172.21.15.12:0/869503257 >> 172.21.15.12:0/869503257 conn(0x7fc1280fca70 msgr2=0x7fc1280feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:31.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.270+0000 7fc12f000640 1 -- 172.21.15.12:0/869503257 shutdown_connections 2024-03-23T23:07:31.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.270+0000 7fc12f000640 1 -- 172.21.15.12:0/869503257 wait complete. 2024-03-23T23:07:31.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.271+0000 7fc12f000640 1 Processor -- start 2024-03-23T23:07:31.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.271+0000 7fc12f000640 1 -- start start 2024-03-23T23:07:31.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.272+0000 7fc12f000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc128101500 0x7fc128199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.272+0000 7fc12f000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1281026f0 0x7fc128199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.272+0000 7fc12f000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1281030b0 0x7fc1281a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.272+0000 7fc12f000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12810d1e0 con 0x7fc1281030b0 2024-03-23T23:07:31.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.272+0000 7fc12f000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12810d060 con 0x7fc128101500 2024-03-23T23:07:31.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.272+0000 7fc12f000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12810d360 con 0x7fc1281026f0 2024-03-23T23:07:31.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.272+0000 7fc12e600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1281030b0 0x7fc1281a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.272+0000 7fc12dc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc128101500 0x7fc128199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.272+0000 7fc12dc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc128101500 0x7fc128199a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44336/0 (socket says 172.21.15.12:44336) 2024-03-23T23:07:31.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.272+0000 7fc12dc00640 1 -- 172.21.15.12:0/2885052451 learned_addr learned my addr 172.21.15.12:0/2885052451 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:31.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.272+0000 7fc12d200640 1 --2- 172.21.15.12:0/2885052451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1281026f0 0x7fc128199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.273+0000 7fc12e600640 1 -- 172.21.15.12:0/2885052451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1281026f0 msgr2=0x7fc128199f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:31.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.273+0000 7fc12e600640 1 --2- 172.21.15.12:0/2885052451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1281026f0 0x7fc128199f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.273+0000 7fc12e600640 1 -- 172.21.15.12:0/2885052451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc128101500 msgr2=0x7fc128199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:31.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.273+0000 7fc12e600640 1 --2- 172.21.15.12:0/2885052451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc128101500 0x7fc128199a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.273+0000 7fc12e600640 1 -- 172.21.15.12:0/2885052451 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1281a1650 con 0x7fc1281030b0 2024-03-23T23:07:31.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.273+0000 7fc12e600640 1 --2- 172.21.15.12:0/2885052451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1281030b0 0x7fc1281a0fe0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc120009fd0 tx=0x7fc12000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:31.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.273+0000 7fc126000640 1 -- 172.21.15.12:0/2885052451 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12000ce20 con 0x7fc1281030b0 2024-03-23T23:07:31.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.273+0000 7fc126000640 1 -- 172.21.15.12:0/2885052451 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc120005020 con 0x7fc1281030b0 2024-03-23T23:07:31.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.273+0000 7fc12f000640 1 -- 172.21.15.12:0/2885052451 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1281a1940 con 0x7fc1281030b0 2024-03-23T23:07:31.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.273+0000 7fc126000640 1 -- 172.21.15.12:0/2885052451 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc120010720 con 0x7fc1281030b0 2024-03-23T23:07:31.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.273+0000 7fc12f000640 1 -- 172.21.15.12:0/2885052451 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1281a1ee0 con 0x7fc1281030b0 2024-03-23T23:07:31.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.274+0000 7fc12f000640 1 -- 172.21.15.12:0/2885052451 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1281a2170 con 0x7fc1281030b0 2024-03-23T23:07:31.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.276+0000 7fc126000640 1 -- 172.21.15.12:0/2885052451 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc120005540 con 0x7fc1281030b0 2024-03-23T23:07:31.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.278+0000 7fc126000640 1 --2- 172.21.15.12:0/2885052451 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc1100a2880 0x7fc1100a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.279+0000 7fc126000640 1 -- 172.21.15.12:0/2885052451 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fc1200cf190 con 0x7fc1281030b0 2024-03-23T23:07:31.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.279+0000 7fc12dc00640 1 --2- 172.21.15.12:0/2885052451 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc1100a2880 0x7fc1100a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.279+0000 7fc12dc00640 1 --2- 172.21.15.12:0/2885052451 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc1100a2880 0x7fc1100a4d40 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc11c02efd0 tx=0x7fc11c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:31.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.283+0000 7fc126000640 1 -- 172.21.15.12:0/2885052451 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc12009c1e0 con 0x7fc1281030b0 2024-03-23T23:07:31.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.528+0000 7fc12f000640 1 -- 172.21.15.12:0/2885052451 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1281a2340 con 0x7fc1281030b0 2024-03-23T23:07:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.529+0000 7fc126000640 1 -- 172.21.15.12:0/2885052451 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7fc12009bae0 con 0x7fc1281030b0 2024-03-23T23:07:31.529 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:07:31.529 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:07:31.530 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:07:31.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.531+0000 7fc12f000640 1 -- 172.21.15.12:0/2885052451 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc1100a2880 msgr2=0x7fc1100a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:31.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.531+0000 7fc12f000640 1 --2- 172.21.15.12:0/2885052451 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc1100a2880 0x7fc1100a4d40 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc11c02efd0 tx=0x7fc11c000bc0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.531+0000 7fc12f000640 1 -- 172.21.15.12:0/2885052451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1281030b0 msgr2=0x7fc1281a0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:31.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.531+0000 7fc12f000640 1 --2- 172.21.15.12:0/2885052451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1281030b0 0x7fc1281a0fe0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc120009fd0 tx=0x7fc12000ef10 comp rx=0 tx=0).stop 2024-03-23T23:07:31.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.532+0000 7fc12f000640 1 -- 172.21.15.12:0/2885052451 shutdown_connections 2024-03-23T23:07:31.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.532+0000 7fc12f000640 1 --2- 172.21.15.12:0/2885052451 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc1100a2880 0x7fc1100a4d40 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.532+0000 7fc12f000640 1 --2- 172.21.15.12:0/2885052451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1281030b0 0x7fc1281a0fe0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.532+0000 7fc12f000640 1 --2- 172.21.15.12:0/2885052451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1281026f0 0x7fc128199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.532+0000 7fc12f000640 1 --2- 172.21.15.12:0/2885052451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc128101500 0x7fc128199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.532+0000 7fc12f000640 1 -- 172.21.15.12:0/2885052451 >> 172.21.15.12:0/2885052451 conn(0x7fc1280fca70 msgr2=0x7fc1280738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:31.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.532+0000 7fc12f000640 1 -- 172.21.15.12:0/2885052451 shutdown_connections 2024-03-23T23:07:31.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.532+0000 7fc12f000640 1 -- 172.21.15.12:0/2885052451 wait complete. 2024-03-23T23:07:31.552 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:07:31.552 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-23T23:07:31.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.776+0000 7fcf7ca00640 1 -- 172.21.15.12:0/1417834709 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf78109540 msgr2=0x7fcf7810b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:31.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.776+0000 7fcf7ca00640 1 --2- 172.21.15.12:0/1417834709 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf78109540 0x7fcf7810b9e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fcf70009ca0 tx=0x7fcf7002c190 comp rx=0 tx=0).stop 2024-03-23T23:07:31.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.776+0000 7fcf7ca00640 1 -- 172.21.15.12:0/1417834709 shutdown_connections 2024-03-23T23:07:31.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.776+0000 7fcf7ca00640 1 --2- 172.21.15.12:0/1417834709 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf78109540 0x7fcf7810b9e0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.776+0000 7fcf7ca00640 1 --2- 172.21.15.12:0/1417834709 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf780740a0 0x7fcf78074500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.776+0000 7fcf7ca00640 1 --2- 172.21.15.12:0/1417834709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf78075680 0x7fcf78073ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.776+0000 7fcf7ca00640 1 -- 172.21.15.12:0/1417834709 >> 172.21.15.12:0/1417834709 conn(0x7fcf780fcc40 msgr2=0x7fcf780ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:31.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.776+0000 7fcf7ca00640 1 -- 172.21.15.12:0/1417834709 shutdown_connections 2024-03-23T23:07:31.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.776+0000 7fcf7ca00640 1 -- 172.21.15.12:0/1417834709 wait complete. 2024-03-23T23:07:31.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.777+0000 7fcf7ca00640 1 Processor -- start 2024-03-23T23:07:31.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.777+0000 7fcf7ca00640 1 -- start start 2024-03-23T23:07:31.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.778+0000 7fcf7ca00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf780740a0 0x7fcf78199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.778+0000 7fcf7ca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf78075680 0x7fcf7819a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.778+0000 7fcf7ca00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf78109540 0x7fcf781a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.778+0000 7fcf7ca00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7810f510 con 0x7fcf78109540 2024-03-23T23:07:31.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.778+0000 7fcf7ca00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7810f390 con 0x7fcf78075680 2024-03-23T23:07:31.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.778+0000 7fcf7ca00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7810f690 con 0x7fcf780740a0 2024-03-23T23:07:31.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.778+0000 7fcf75e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf780740a0 0x7fcf78199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.778+0000 7fcf75e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf780740a0 0x7fcf78199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:46044/0 (socket says 172.21.15.12:46044) 2024-03-23T23:07:31.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.778+0000 7fcf75400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf78075680 0x7fcf7819a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.778+0000 7fcf75e00640 1 -- 172.21.15.12:0/3344046416 learned_addr learned my addr 172.21.15.12:0/3344046416 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:31.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.778+0000 7fcf76800640 1 --2- 172.21.15.12:0/3344046416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf78109540 0x7fcf781a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.778+0000 7fcf75400640 1 -- 172.21.15.12:0/3344046416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf780740a0 msgr2=0x7fcf78199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:31.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.778+0000 7fcf75400640 1 --2- 172.21.15.12:0/3344046416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf780740a0 0x7fcf78199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.778+0000 7fcf75400640 1 -- 172.21.15.12:0/3344046416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf78109540 msgr2=0x7fcf781a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:31.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.778+0000 7fcf75400640 1 --2- 172.21.15.12:0/3344046416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf78109540 0x7fcf781a10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.778+0000 7fcf75400640 1 -- 172.21.15.12:0/3344046416 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf781a1740 con 0x7fcf78075680 2024-03-23T23:07:31.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.778+0000 7fcf75e00640 1 --2- 172.21.15.12:0/3344046416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf780740a0 0x7fcf78199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:07:31.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.779+0000 7fcf75400640 1 --2- 172.21.15.12:0/3344046416 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf78075680 0x7fcf7819a060 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fcf6400e9f0 tx=0x7fcf6400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:31.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.779+0000 7fcf6ea00640 1 -- 172.21.15.12:0/3344046416 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6400ccc0 con 0x7fcf78075680 2024-03-23T23:07:31.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.779+0000 7fcf6ea00640 1 -- 172.21.15.12:0/3344046416 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf64010910 con 0x7fcf78075680 2024-03-23T23:07:31.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.779+0000 7fcf7ca00640 1 -- 172.21.15.12:0/3344046416 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf781a1a30 con 0x7fcf78075680 2024-03-23T23:07:31.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.779+0000 7fcf6ea00640 1 -- 172.21.15.12:0/3344046416 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf64018960 con 0x7fcf78075680 2024-03-23T23:07:31.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.779+0000 7fcf7ca00640 1 -- 172.21.15.12:0/3344046416 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf781a1fd0 con 0x7fcf78075680 2024-03-23T23:07:31.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.782+0000 7fcf6ea00640 1 -- 172.21.15.12:0/3344046416 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcf64010ac0 con 0x7fcf78075680 2024-03-23T23:07:31.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.783+0000 7fcf7f5f4740 1 Processor -- start 2024-03-23T23:07:31.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.783+0000 7fcf7f5f4740 1 -- start start 2024-03-23T23:07:31.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.783+0000 7fcf7f5f4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5606c8be1810 0x5606c8be1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.783+0000 7fcf7f5f4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606c8be29e0 0x5606c8be2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.783+0000 7fcf7f5f4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be33a0 0x5606c8be9c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.783+0000 7fcf7f5f4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606c8b462c0 con 0x5606c8be1810 2024-03-23T23:07:31.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.783+0000 7fcf7f5f4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606c8aeec90 con 0x5606c8be33a0 2024-03-23T23:07:31.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.783+0000 7fcf7f5f4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606c8b42360 con 0x5606c8be29e0 2024-03-23T23:07:31.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.783+0000 7fcf76800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5606c8be1810 0x5606c8be1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.783+0000 7fcf75e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606c8be29e0 0x5606c8be2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.783+0000 7fcf75e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606c8be29e0 0x5606c8be2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:46046/0 (socket says 172.21.15.12:46046) 2024-03-23T23:07:31.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.783+0000 7fcf75e00640 1 -- 172.21.15.12:0/18206052 learned_addr learned my addr 172.21.15.12:0/18206052 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:31.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.783+0000 7fcf76800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5606c8be1810 0x5606c8be1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60860/0 (socket says 172.21.15.12:60860) 2024-03-23T23:07:31.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.783+0000 7fcf75400640 1 --2- 172.21.15.12:0/18206052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be33a0 0x5606c8be9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.784+0000 7fcf6ea00640 1 --2- 172.21.15.12:0/3344046416 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcf5c0a2880 0x7fcf5c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.784+0000 7fcf75e00640 1 -- 172.21.15.12:0/18206052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be33a0 msgr2=0x5606c8be9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:31.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.784+0000 7fcf75e00640 1 --2- 172.21.15.12:0/18206052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be33a0 0x5606c8be9c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.784+0000 7fcf75e00640 1 -- 172.21.15.12:0/18206052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5606c8be1810 msgr2=0x5606c8be1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:31.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.784+0000 7fcf75e00640 1 --2- 172.21.15.12:0/18206052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5606c8be1810 0x5606c8be1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.784+0000 7fcf75e00640 1 -- 172.21.15.12:0/18206052 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606c8b13080 con 0x5606c8be29e0 2024-03-23T23:07:31.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.784+0000 7fcf75400640 1 --2- 172.21.15.12:0/18206052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be33a0 0x5606c8be9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:07:31.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.784+0000 7fcf6ea00640 1 -- 172.21.15.12:0/3344046416 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fcf640cedc0 con 0x7fcf78075680 2024-03-23T23:07:31.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.784+0000 7fcf76800640 1 --2- 172.21.15.12:0/3344046416 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcf5c0a2880 0x7fcf5c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.784+0000 7fcf75e00640 1 --2- 172.21.15.12:0/18206052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606c8be29e0 0x5606c8be2e60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fcf68010580 tx=0x7fcf68010a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f07c06a8fb4d1d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:31.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.784+0000 7fcf63000640 1 -- 172.21.15.12:0/18206052 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6800e890 con 0x5606c8be29e0 2024-03-23T23:07:31.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.784+0000 7fcf63000640 1 -- 172.21.15.12:0/18206052 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf6801c480 con 0x5606c8be29e0 2024-03-23T23:07:31.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.785+0000 7fcf63000640 1 -- 172.21.15.12:0/18206052 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf680116a0 con 0x5606c8be29e0 2024-03-23T23:07:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.785+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/18206052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606c8be29e0 msgr2=0x5606c8be2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.785+0000 7fcf76800640 1 --2- 172.21.15.12:0/3344046416 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcf5c0a2880 0x7fcf5c0a4d40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fcf7002c6d0 tx=0x7fcf7002e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.785+0000 7fcf7f5f4740 1 --2- 172.21.15.12:0/18206052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606c8be29e0 0x5606c8be2e60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fcf68010580 tx=0x7fcf68010a40 comp rx=0 tx=0).stop 2024-03-23T23:07:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.785+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/18206052 shutdown_connections 2024-03-23T23:07:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.785+0000 7fcf7f5f4740 1 --2- 172.21.15.12:0/18206052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be33a0 0x5606c8be9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.785+0000 7fcf7f5f4740 1 --2- 172.21.15.12:0/18206052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606c8be29e0 0x5606c8be2e60 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.785+0000 7fcf7f5f4740 1 --2- 172.21.15.12:0/18206052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5606c8be1810 0x5606c8be1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.785+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/18206052 >> 172.21.15.12:0/18206052 conn(0x5606c8bdcc00 msgr2=0x5606c8bdd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.785+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/18206052 shutdown_connections 2024-03-23T23:07:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.786+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/18206052 wait complete. 2024-03-23T23:07:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.786+0000 7fcf7f5f4740 1 -- start start 2024-03-23T23:07:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.786+0000 7fcf7f5f4740 1 Processor -- start 2024-03-23T23:07:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.787+0000 7fcf7f5f4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606c8be1810 0x5606c8beba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.787+0000 7fcf7f5f4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5606c8be29e0 0x5606c8bebf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.787+0000 7fcf7f5f4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be33a0 0x5606c8bf2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.787+0000 7fcf7f5f4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606c8bea3b0 con 0x5606c8be29e0 2024-03-23T23:07:31.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.787+0000 7fcf7f5f4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606c8b020d0 con 0x5606c8be33a0 2024-03-23T23:07:31.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.787+0000 7fcf7f5f4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606c8bea530 con 0x5606c8be1810 2024-03-23T23:07:31.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.787+0000 7fcf7f5f4740 10 client.0 _ll_register_callbacks cb 0x5606c8b06e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:31.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.787+0000 7fcf75e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606c8be1810 0x5606c8beba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.787+0000 7fcf75e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606c8be1810 0x5606c8beba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:46056/0 (socket says 172.21.15.12:46056) 2024-03-23T23:07:31.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.787+0000 7fcf75400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5606c8be29e0 0x5606c8bebf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.787+0000 7fcf75e00640 1 -- 172.21.15.12:0/68191641 learned_addr learned my addr 172.21.15.12:0/68191641 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:31.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.787+0000 7fcf76800640 1 --2- 172.21.15.12:0/68191641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be33a0 0x5606c8bf2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.787+0000 7fcf75e00640 1 -- 172.21.15.12:0/68191641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be33a0 msgr2=0x5606c8bf2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:31.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.787+0000 7fcf75e00640 1 --2- 172.21.15.12:0/68191641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be33a0 0x5606c8bf2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.787+0000 7fcf75e00640 1 -- 172.21.15.12:0/68191641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5606c8be29e0 msgr2=0x5606c8bebf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:31.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.787+0000 7fcf75e00640 1 --2- 172.21.15.12:0/68191641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5606c8be29e0 0x5606c8bebf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.787+0000 7fcf75e00640 1 -- 172.21.15.12:0/68191641 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5606c885a3f0 con 0x5606c8be1810 2024-03-23T23:07:31.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.787+0000 7fcf76800640 1 --2- 172.21.15.12:0/68191641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be33a0 0x5606c8bf2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:07:31.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.788+0000 7fcf75e00640 1 --2- 172.21.15.12:0/68191641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606c8be1810 0x5606c8beba30 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fcf6801c8e0 tx=0x7fcf6801c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6bb15cbee3f6fc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:31.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.788+0000 7fcf57400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:07:31.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.788+0000 7fcf57400640 1 -- 172.21.15.12:0/68191641 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf68004250 con 0x5606c8be1810 2024-03-23T23:07:31.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.788+0000 7fcf57400640 1 -- 172.21.15.12:0/68191641 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fcf680043f0 con 0x5606c8be1810 2024-03-23T23:07:31.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.788+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/68191641 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf6801c480 con 0x5606c8be1810 2024-03-23T23:07:31.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.788+0000 7fcf57400640 1 -- 172.21.15.12:0/68191641 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf6800e690 con 0x5606c8be1810 2024-03-23T23:07:31.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.788+0000 7fcf7f5f4740 10 client.26869 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:31.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.788+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/68191641 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5606c89247d0 con 0x5606c8be1810 2024-03-23T23:07:31.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.788+0000 7fcf57400640 1 -- 172.21.15.12:0/68191641 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf6801c480 con 0x5606c8be1810 2024-03-23T23:07:31.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.789+0000 7fcf57400640 1 -- 172.21.15.12:0/68191641 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fcf680038a0 con 0x5606c8be1810 2024-03-23T23:07:31.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.789+0000 7fcf7f5f4740 10 client.26869 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:31.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.789+0000 7fcf7f5f4740 10 client.26869 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:07:31.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.789+0000 7fcf7f5f4740 20 client.26869 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:31.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.789+0000 7fcf7f5f4740 1 --2- 172.21.15.12:0/68191641 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5606c8bf7230 0x5606c8bf9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.789+0000 7fcf7f5f4740 4 client.26869 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:07:31.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.789+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/68191641 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5606c8b53800 con 0x5606c8bf7230 2024-03-23T23:07:31.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.789+0000 7fcf75400640 1 --2- 172.21.15.12:0/68191641 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5606c8bf7230 0x5606c8bf9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.789+0000 7fcf75400640 1 --2- 172.21.15.12:0/68191641 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5606c8bf7230 0x5606c8bf9620 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2ac1098d23b098e9 server_cookie=42b40e94eb837f71 in_seq=0 out_seq=0 2024-03-23T23:07:31.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.789+0000 7fcf57400640 10 client.26869 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:07:31.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.790+0000 7fcf57400640 1 -- 172.21.15.12:0/68191641 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5606c8b53800 con 0x5606c8bf7230 2024-03-23T23:07:31.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.790+0000 7fcf57400640 10 client.26869 handle_command_reply: tid=0 2024-03-23T23:07:31.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.790+0000 7fcf7f5f4740 1 client.26869 shutdown 2024-03-23T23:07:31.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.790+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/68191641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606c8be1810 msgr2=0x5606c8beba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:31.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.790+0000 7fcf7f5f4740 1 --2- 172.21.15.12:0/68191641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606c8be1810 0x5606c8beba30 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fcf6801c8e0 tx=0x7fcf6801c910 comp rx=0 tx=0).stop 2024-03-23T23:07:31.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.790+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/68191641 shutdown_connections 2024-03-23T23:07:31.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.790+0000 7fcf7f5f4740 1 --2- 172.21.15.12:0/68191641 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5606c8bf7230 0x5606c8bf9620 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.790+0000 7fcf7f5f4740 1 --2- 172.21.15.12:0/68191641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be33a0 0x5606c8bf2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.790+0000 7fcf7f5f4740 1 --2- 172.21.15.12:0/68191641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5606c8be29e0 0x5606c8bebf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.790+0000 7fcf7f5f4740 1 --2- 172.21.15.12:0/68191641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606c8be1810 0x5606c8beba30 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.790+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/68191641 >> 172.21.15.12:0/68191641 conn(0x5606c8bdcc00 msgr2=0x5606c8bdbeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:31.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.790+0000 7fcf57400640 0 client.26869 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:07:31.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.791+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/68191641 shutdown_connections 2024-03-23T23:07:31.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.791+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/68191641 wait complete. 2024-03-23T23:07:31.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.791+0000 7fcf7f5f4740 20 client.26869 trim_cache size 0 max 16384 2024-03-23T23:07:31.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:31 smithi012 ceph-mon[31695]: scrub queued for path: / 2024-03-23T23:07:31.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:31 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:07:31.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:31 smithi012 ceph-mon[31695]: scrub summary: active paths [/] 2024-03-23T23:07:31.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:31 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:07:31.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:31 smithi012 ceph-mon[31695]: scrub summary: idle 2024-03-23T23:07:31.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:31 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2885052451' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:07:31.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.805+0000 7fcf7f5f4740 1 Processor -- start 2024-03-23T23:07:31.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.805+0000 7fcf7f5f4740 1 -- start start 2024-03-23T23:07:31.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.805+0000 7fcf7f5f4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be1810 0x5606c8beb050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.805+0000 7fcf7f5f4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5606c8be29e0 0x5606c8bf2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.805+0000 7fcf7f5f4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606c8be33a0 0x5606c8beb590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.805+0000 7fcf7f5f4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf781ad1b0 con 0x5606c8be29e0 2024-03-23T23:07:31.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.805+0000 7fcf7f5f4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf781aacb0 con 0x5606c8be1810 2024-03-23T23:07:31.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.805+0000 7fcf7f5f4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf781a8d50 con 0x5606c8be33a0 2024-03-23T23:07:31.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.806+0000 7fcf75e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be1810 0x5606c8beb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.806+0000 7fcf76800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606c8be33a0 0x5606c8beb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.806+0000 7fcf75e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be1810 0x5606c8beb050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44388/0 (socket says 172.21.15.12:44388) 2024-03-23T23:07:31.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.806+0000 7fcf75400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5606c8be29e0 0x5606c8bf2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.806+0000 7fcf75e00640 1 -- 172.21.15.12:0/3559250514 learned_addr learned my addr 172.21.15.12:0/3559250514 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:31.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.806+0000 7fcf75e00640 1 -- 172.21.15.12:0/3559250514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606c8be33a0 msgr2=0x5606c8beb590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:31.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.806+0000 7fcf75e00640 1 --2- 172.21.15.12:0/3559250514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606c8be33a0 0x5606c8beb590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.806+0000 7fcf75e00640 1 -- 172.21.15.12:0/3559250514 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5606c8be29e0 msgr2=0x5606c8bf2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:31.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.806+0000 7fcf75e00640 1 --2- 172.21.15.12:0/3559250514 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5606c8be29e0 0x5606c8bf2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.806+0000 7fcf75e00640 1 -- 172.21.15.12:0/3559250514 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606c8bdd670 con 0x5606c8be1810 2024-03-23T23:07:31.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.806+0000 7fcf75e00640 1 --2- 172.21.15.12:0/3559250514 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be1810 0x5606c8beb050 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fcf68001eb0 tx=0x7fcf68005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=436e12bcd1692bc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:31.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.807+0000 7fcf62600640 1 -- 172.21.15.12:0/3559250514 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6801c980 con 0x5606c8be1810 2024-03-23T23:07:31.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.807+0000 7fcf62600640 1 -- 172.21.15.12:0/3559250514 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf6802a070 con 0x5606c8be1810 2024-03-23T23:07:31.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.807+0000 7fcf62600640 1 -- 172.21.15.12:0/3559250514 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6800e830 con 0x5606c8be1810 2024-03-23T23:07:31.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.807+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/3559250514 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be1810 msgr2=0x5606c8beb050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:31.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.807+0000 7fcf7f5f4740 1 --2- 172.21.15.12:0/3559250514 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be1810 0x5606c8beb050 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fcf68001eb0 tx=0x7fcf68005880 comp rx=0 tx=0).stop 2024-03-23T23:07:31.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.807+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/3559250514 shutdown_connections 2024-03-23T23:07:31.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.807+0000 7fcf7f5f4740 1 --2- 172.21.15.12:0/3559250514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606c8be33a0 0x5606c8beb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.807+0000 7fcf7f5f4740 1 --2- 172.21.15.12:0/3559250514 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5606c8be29e0 0x5606c8bf2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.807+0000 7fcf7f5f4740 1 --2- 172.21.15.12:0/3559250514 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be1810 0x5606c8beb050 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.807+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/3559250514 >> 172.21.15.12:0/3559250514 conn(0x5606c8bdcc00 msgr2=0x5606c8bdbeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:31.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.807+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/3559250514 shutdown_connections 2024-03-23T23:07:31.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.807+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/3559250514 wait complete. 2024-03-23T23:07:31.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.808+0000 7fcf7f5f4740 1 -- start start 2024-03-23T23:07:31.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.808+0000 7fcf7f5f4740 1 Processor -- start 2024-03-23T23:07:31.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.808+0000 7fcf7f5f4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606c8be1810 0x5606c8beed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.808+0000 7fcf7f5f4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be29e0 0x5606c8bef2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.808+0000 7fcf7f5f4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5606c8be33a0 0x5606c8bf22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.808+0000 7fcf7f5f4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf781a9350 con 0x5606c8be33a0 2024-03-23T23:07:31.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.808+0000 7fcf7f5f4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf781aa650 con 0x5606c8be29e0 2024-03-23T23:07:31.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.808+0000 7fcf7f5f4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf781a9990 con 0x5606c8be1810 2024-03-23T23:07:31.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.808+0000 7fcf7f5f4740 10 client.0 _ll_register_callbacks cb 0x7fcf680103d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:31.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.809+0000 7fcf76800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5606c8be33a0 0x5606c8bf22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.809+0000 7fcf75e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606c8be1810 0x5606c8beed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.809+0000 7fcf75400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be29e0 0x5606c8bef2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.809+0000 7fcf75400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be29e0 0x5606c8bef2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44400/0 (socket says 172.21.15.12:44400) 2024-03-23T23:07:31.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.809+0000 7fcf75400640 1 -- 172.21.15.12:0/819020927 learned_addr learned my addr 172.21.15.12:0/819020927 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:31.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.809+0000 7fcf75e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606c8be1810 0x5606c8beed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:46070/0 (socket says 172.21.15.12:46070) 2024-03-23T23:07:31.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.809+0000 7fcf76800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5606c8be33a0 0x5606c8bf22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60886/0 (socket says 172.21.15.12:60886) 2024-03-23T23:07:31.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.809+0000 7fcf75400640 1 -- 172.21.15.12:0/819020927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606c8be1810 msgr2=0x5606c8beed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:31.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.809+0000 7fcf75400640 1 --2- 172.21.15.12:0/819020927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606c8be1810 0x5606c8beed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.809+0000 7fcf75400640 1 -- 172.21.15.12:0/819020927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5606c8be33a0 msgr2=0x5606c8bf22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:31.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.809+0000 7fcf75400640 1 --2- 172.21.15.12:0/819020927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5606c8be33a0 0x5606c8bf22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.809+0000 7fcf75400640 1 -- 172.21.15.12:0/819020927 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5606c8bf28d0 con 0x5606c8be29e0 2024-03-23T23:07:31.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.809+0000 7fcf75400640 1 --2- 172.21.15.12:0/819020927 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be29e0 0x5606c8bef2c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fcf640cc870 tx=0x7fcf64009700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=441b7ed4c1dab362 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:31.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.809+0000 7fcf57400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:07:31.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.810+0000 7fcf57400640 1 -- 172.21.15.12:0/819020927 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf640860b0 con 0x5606c8be29e0 2024-03-23T23:07:31.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.810+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/819020927 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf6802a070 con 0x5606c8be29e0 2024-03-23T23:07:31.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.810+0000 7fcf57400640 1 -- 172.21.15.12:0/819020927 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fcf64097e30 con 0x5606c8be29e0 2024-03-23T23:07:31.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.810+0000 7fcf57400640 1 -- 172.21.15.12:0/819020927 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf6409ac10 con 0x5606c8be29e0 2024-03-23T23:07:31.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.810+0000 7fcf7f5f4740 10 client.26903 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:31.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.810+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/819020927 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5606c8c531c0 con 0x5606c8be29e0 2024-03-23T23:07:31.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.810+0000 7fcf57400640 1 -- 172.21.15.12:0/819020927 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf6802a070 con 0x5606c8be29e0 2024-03-23T23:07:31.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.811+0000 7fcf57400640 1 -- 172.21.15.12:0/819020927 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fcf64018c00 con 0x5606c8be29e0 2024-03-23T23:07:31.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.811+0000 7fcf7f5f4740 10 client.26903 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:31.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.811+0000 7fcf7f5f4740 10 client.26903 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:07:31.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.811+0000 7fcf7f5f4740 20 client.26903 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:31.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.811+0000 7fcf7f5f4740 1 --2- 172.21.15.12:0/819020927 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5606c8c53ac0 0x5606c8c55ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:31.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.811+0000 7fcf7f5f4740 4 client.26903 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:07:31.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.811+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/819020927 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5606c8b4db50 con 0x5606c8c53ac0 2024-03-23T23:07:31.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.811+0000 7fcf75e00640 1 --2- 172.21.15.12:0/819020927 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5606c8c53ac0 0x5606c8c55ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:31.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.812+0000 7fcf75e00640 1 --2- 172.21.15.12:0/819020927 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5606c8c53ac0 0x5606c8c55ee0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3f2f8e559995eaf9 server_cookie=fe350e34219ecf91 in_seq=0 out_seq=0 2024-03-23T23:07:31.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.812+0000 7fcf57400640 10 client.26903 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:07:31.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.812+0000 7fcf57400640 1 -- 172.21.15.12:0/819020927 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5606c8b4db50 con 0x5606c8c53ac0 2024-03-23T23:07:31.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.812+0000 7fcf57400640 10 client.26903 handle_command_reply: tid=0 2024-03-23T23:07:31.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.812+0000 7fcf7f5f4740 1 client.26903 shutdown 2024-03-23T23:07:31.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.812+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/819020927 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be29e0 msgr2=0x5606c8bef2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:31.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.812+0000 7fcf7f5f4740 1 --2- 172.21.15.12:0/819020927 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be29e0 0x5606c8bef2c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fcf640cc870 tx=0x7fcf64009700 comp rx=0 tx=0).stop 2024-03-23T23:07:31.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.812+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/819020927 shutdown_connections 2024-03-23T23:07:31.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.812+0000 7fcf7f5f4740 1 --2- 172.21.15.12:0/819020927 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5606c8c53ac0 0x5606c8c55ee0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.812+0000 7fcf7f5f4740 1 --2- 172.21.15.12:0/819020927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5606c8be33a0 0x5606c8bf22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.812+0000 7fcf7f5f4740 1 --2- 172.21.15.12:0/819020927 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606c8be29e0 0x5606c8bef2c0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.812+0000 7fcf57400640 0 client.26903 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:07:31.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.812+0000 7fcf7f5f4740 1 --2- 172.21.15.12:0/819020927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606c8be1810 0x5606c8beed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.812+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/819020927 >> 172.21.15.12:0/819020927 conn(0x5606c8bdcc00 msgr2=0x5606c8be5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:31.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.812+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/819020927 shutdown_connections 2024-03-23T23:07:31.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.812+0000 7fcf7f5f4740 1 -- 172.21.15.12:0/819020927 wait complete. 2024-03-23T23:07:31.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.812+0000 7fcf7f5f4740 20 client.26903 trim_cache size 0 max 16384 2024-03-23T23:07:31.816 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:07:31.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.813+0000 7fcf7ca00640 1 -- 172.21.15.12:0/3344046416 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcf5c0a2880 msgr2=0x7fcf5c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:31.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.813+0000 7fcf7ca00640 1 --2- 172.21.15.12:0/3344046416 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcf5c0a2880 0x7fcf5c0a4d40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fcf7002c6d0 tx=0x7fcf7002e040 comp rx=0 tx=0).stop 2024-03-23T23:07:31.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.813+0000 7fcf7ca00640 1 -- 172.21.15.12:0/3344046416 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf78075680 msgr2=0x7fcf7819a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:31.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.813+0000 7fcf7ca00640 1 --2- 172.21.15.12:0/3344046416 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf78075680 0x7fcf7819a060 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fcf6400e9f0 tx=0x7fcf6400eeb0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.813+0000 7fcf7ca00640 1 -- 172.21.15.12:0/3344046416 shutdown_connections 2024-03-23T23:07:31.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.813+0000 7fcf7ca00640 1 --2- 172.21.15.12:0/3344046416 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcf5c0a2880 0x7fcf5c0a4d40 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.813+0000 7fcf7ca00640 1 --2- 172.21.15.12:0/3344046416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf78109540 0x7fcf781a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.813+0000 7fcf7ca00640 1 --2- 172.21.15.12:0/3344046416 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf78075680 0x7fcf7819a060 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.813+0000 7fcf7ca00640 1 --2- 172.21.15.12:0/3344046416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf780740a0 0x7fcf78199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:31.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.813+0000 7fcf7ca00640 1 -- 172.21.15.12:0/3344046416 >> 172.21.15.12:0/3344046416 conn(0x7fcf780fcc40 msgr2=0x7fcf781054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:31.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.813+0000 7fcf7ca00640 1 -- 172.21.15.12:0/3344046416 shutdown_connections 2024-03-23T23:07:31.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:31.813+0000 7fcf7ca00640 1 -- 172.21.15.12:0/3344046416 wait complete. 2024-03-23T23:07:31.830 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:07:31.830 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-23T23:07:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.036+0000 7f7268800640 1 -- 172.21.15.12:0/1490199168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7260100ae0 msgr2=0x7f7260100f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.036+0000 7f7268800640 1 --2- 172.21.15.12:0/1490199168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7260100ae0 0x7f7260100f60 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7254009900 tx=0x7f725402bc30 comp rx=0 tx=0).stop 2024-03-23T23:07:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.036+0000 7f7268800640 1 -- 172.21.15.12:0/1490199168 shutdown_connections 2024-03-23T23:07:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.036+0000 7f7268800640 1 --2- 172.21.15.12:0/1490199168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72601014a0 0x7f7260107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.036+0000 7f7268800640 1 --2- 172.21.15.12:0/1490199168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7260100ae0 0x7f7260100f60 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.036+0000 7f7268800640 1 --2- 172.21.15.12:0/1490199168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72600ff8f0 0x7f72600ffd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.036+0000 7f7268800640 1 -- 172.21.15.12:0/1490199168 >> 172.21.15.12:0/1490199168 conn(0x7f72600fae60 msgr2=0x7f72600fd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.036+0000 7f7268800640 1 -- 172.21.15.12:0/1490199168 shutdown_connections 2024-03-23T23:07:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.037+0000 7f7268800640 1 -- 172.21.15.12:0/1490199168 wait complete. 2024-03-23T23:07:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.038+0000 7f7268800640 1 Processor -- start 2024-03-23T23:07:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.038+0000 7f7268800640 1 -- start start 2024-03-23T23:07:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.039+0000 7f7268800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72600ff8f0 0x7f7260197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.039+0000 7f7268800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7260100ae0 0x7f7260197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.039+0000 7f7268800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72601014a0 0x7f726019efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.039+0000 7f7268800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726010b190 con 0x7f72600ff8f0 2024-03-23T23:07:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.039+0000 7f7268800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726010b010 con 0x7f72601014a0 2024-03-23T23:07:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.039+0000 7f7268800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726010b310 con 0x7f7260100ae0 2024-03-23T23:07:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.039+0000 7f7265400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7260100ae0 0x7f7260197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.039+0000 7f7265e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72600ff8f0 0x7f7260197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.039+0000 7f7265400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7260100ae0 0x7f7260197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:46084/0 (socket says 172.21.15.12:46084) 2024-03-23T23:07:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.039+0000 7f7265400640 1 -- 172.21.15.12:0/294915265 learned_addr learned my addr 172.21.15.12:0/294915265 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.039+0000 7f7266800640 1 --2- 172.21.15.12:0/294915265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72601014a0 0x7f726019efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.039+0000 7f7265400640 1 -- 172.21.15.12:0/294915265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72601014a0 msgr2=0x7f726019efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.039+0000 7f7265400640 1 --2- 172.21.15.12:0/294915265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72601014a0 0x7f726019efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.039+0000 7f7265400640 1 -- 172.21.15.12:0/294915265 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72600ff8f0 msgr2=0x7f7260197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.039+0000 7f7265400640 1 --2- 172.21.15.12:0/294915265 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72600ff8f0 0x7f7260197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.039+0000 7f7265400640 1 -- 172.21.15.12:0/294915265 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f726019f620 con 0x7f7260100ae0 2024-03-23T23:07:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.039+0000 7f7266800640 1 --2- 172.21.15.12:0/294915265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72601014a0 0x7f726019efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:07:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.040+0000 7f7265400640 1 --2- 172.21.15.12:0/294915265 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7260100ae0 0x7f7260197f40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7254005ab0 tx=0x7f725402c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.040+0000 7f725aa00640 1 -- 172.21.15.12:0/294915265 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7254043020 con 0x7f7260100ae0 2024-03-23T23:07:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.040+0000 7f725aa00640 1 -- 172.21.15.12:0/294915265 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72540054e0 con 0x7f7260100ae0 2024-03-23T23:07:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.040+0000 7f725aa00640 1 -- 172.21.15.12:0/294915265 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7254003ce0 con 0x7f7260100ae0 2024-03-23T23:07:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.040+0000 7f7268800640 1 -- 172.21.15.12:0/294915265 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f726019f8b0 con 0x7f7260100ae0 2024-03-23T23:07:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.040+0000 7f7268800640 1 -- 172.21.15.12:0/294915265 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f726019fdb0 con 0x7f7260100ae0 2024-03-23T23:07:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.043+0000 7f725aa00640 1 -- 172.21.15.12:0/294915265 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7254046020 con 0x7f7260100ae0 2024-03-23T23:07:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.044+0000 7f726b3f4740 1 Processor -- start 2024-03-23T23:07:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.044+0000 7f726b3f4740 1 -- start start 2024-03-23T23:07:32.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.044+0000 7f726b3f4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559643176c20 0x559643177040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.044+0000 7f726b3f4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559643177df0 0x559643178270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.044+0000 7f726b3f4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596431787b0 0x55964317f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.044+0000 7f726b3f4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596430d72c0 con 0x559643176c20 2024-03-23T23:07:32.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.044+0000 7f726b3f4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55964307fc90 con 0x559643177df0 2024-03-23T23:07:32.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.044+0000 7f726b3f4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596430d3360 con 0x5596431787b0 2024-03-23T23:07:32.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.045+0000 7f7265e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559643177df0 0x559643178270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.045+0000 7f7266800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559643176c20 0x559643177040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.045+0000 7f7265e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559643177df0 0x559643178270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44440/0 (socket says 172.21.15.12:44440) 2024-03-23T23:07:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.045+0000 7f7265400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596431787b0 0x55964317f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.045+0000 7f7265e00640 1 -- 172.21.15.12:0/3420539685 learned_addr learned my addr 172.21.15.12:0/3420539685 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.045+0000 7f725aa00640 1 --2- 172.21.15.12:0/294915265 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f72480a2880 0x7f72480a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.045+0000 7f7265e00640 1 -- 172.21.15.12:0/3420539685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596431787b0 msgr2=0x55964317f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.045+0000 7f7265e00640 1 --2- 172.21.15.12:0/3420539685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596431787b0 0x55964317f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.045+0000 7f7265e00640 1 -- 172.21.15.12:0/3420539685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559643176c20 msgr2=0x559643177040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.045+0000 7f7265e00640 1 --2- 172.21.15.12:0/3420539685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559643176c20 0x559643177040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.045+0000 7f7265e00640 1 -- 172.21.15.12:0/3420539685 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5596430a4080 con 0x559643177df0 2024-03-23T23:07:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.045+0000 7f725aa00640 1 -- 172.21.15.12:0/294915265 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f72540f0be0 con 0x7f7260100ae0 2024-03-23T23:07:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.045+0000 7f7266800640 1 --2- 172.21.15.12:0/294915265 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f72480a2880 0x7f72480a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.046+0000 7f7265e00640 1 --2- 172.21.15.12:0/3420539685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559643177df0 0x559643178270 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f724c00b7d0 tx=0x7f724c004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be224c6883065fed server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:32.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.046+0000 7f7253000640 1 -- 172.21.15.12:0/3420539685 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c003dd0 con 0x559643177df0 2024-03-23T23:07:32.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.046+0000 7f7253000640 1 -- 172.21.15.12:0/3420539685 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f724c015070 con 0x559643177df0 2024-03-23T23:07:32.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.046+0000 7f7253000640 1 -- 172.21.15.12:0/3420539685 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c0117c0 con 0x559643177df0 2024-03-23T23:07:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.047+0000 7f726b3f4740 1 -- 172.21.15.12:0/3420539685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559643177df0 msgr2=0x559643178270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.047+0000 7f7266800640 1 --2- 172.21.15.12:0/294915265 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f72480a2880 0x7f72480a4d40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x559643177cf0 tx=0x7f725c00e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.047+0000 7f726b3f4740 1 --2- 172.21.15.12:0/3420539685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559643177df0 0x559643178270 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f724c00b7d0 tx=0x7f724c004610 comp rx=0 tx=0).stop 2024-03-23T23:07:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.047+0000 7f726b3f4740 1 -- 172.21.15.12:0/3420539685 shutdown_connections 2024-03-23T23:07:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.047+0000 7f726b3f4740 1 --2- 172.21.15.12:0/3420539685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596431787b0 0x55964317f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.047+0000 7f726b3f4740 1 --2- 172.21.15.12:0/3420539685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559643177df0 0x559643178270 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.047+0000 7f726b3f4740 1 --2- 172.21.15.12:0/3420539685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559643176c20 0x559643177040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.047+0000 7f726b3f4740 1 -- 172.21.15.12:0/3420539685 >> 172.21.15.12:0/3420539685 conn(0x559643172010 msgr2=0x559643172410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.047+0000 7f726b3f4740 1 -- 172.21.15.12:0/3420539685 shutdown_connections 2024-03-23T23:07:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.047+0000 7f726b3f4740 1 -- 172.21.15.12:0/3420539685 wait complete. 2024-03-23T23:07:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.047+0000 7f726b3f4740 1 -- start start 2024-03-23T23:07:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.047+0000 7f726b3f4740 1 Processor -- start 2024-03-23T23:07:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.048+0000 7f726b3f4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559643176c20 0x559643180e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.048+0000 7f726b3f4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559643177df0 0x5596431813a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.048+0000 7f726b3f4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5596431787b0 0x5596431883e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.048+0000 7f726b3f4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55964317f7c0 con 0x5596431787b0 2024-03-23T23:07:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.048+0000 7f726b3f4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596430930d0 con 0x559643176c20 2024-03-23T23:07:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.048+0000 7f726b3f4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55964317f940 con 0x559643177df0 2024-03-23T23:07:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.048+0000 7f726b3f4740 10 client.0 _ll_register_callbacks cb 0x559643097e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.048+0000 7f7265e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559643176c20 0x559643180e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.048+0000 7f7265e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559643176c20 0x559643180e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44444/0 (socket says 172.21.15.12:44444) 2024-03-23T23:07:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.048+0000 7f7265e00640 1 -- 172.21.15.12:0/974235849 learned_addr learned my addr 172.21.15.12:0/974235849 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.048+0000 7f7265400640 1 --2- 172.21.15.12:0/974235849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559643177df0 0x5596431813a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.048+0000 7f7266800640 1 --2- 172.21.15.12:0/974235849 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5596431787b0 0x5596431883e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.049+0000 7f7265e00640 1 -- 172.21.15.12:0/974235849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559643177df0 msgr2=0x5596431813a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.049+0000 7f7265e00640 1 --2- 172.21.15.12:0/974235849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559643177df0 0x5596431813a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.049+0000 7f7265e00640 1 -- 172.21.15.12:0/974235849 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5596431787b0 msgr2=0x5596431883e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.049+0000 7f7265e00640 1 --2- 172.21.15.12:0/974235849 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5596431787b0 0x5596431883e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.049+0000 7f7265e00640 1 -- 172.21.15.12:0/974235849 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559642deb3f0 con 0x559643176c20 2024-03-23T23:07:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.049+0000 7f7265e00640 1 --2- 172.21.15.12:0/974235849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559643176c20 0x559643180e60 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f724c005c90 tx=0x7f724c003e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d20cc90c4296eb10 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.049+0000 7f7247400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:07:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.049+0000 7f7247400640 1 -- 172.21.15.12:0/974235849 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c027020 con 0x559643176c20 2024-03-23T23:07:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.049+0000 7f7247400640 1 -- 172.21.15.12:0/974235849 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f724c005450 con 0x559643176c20 2024-03-23T23:07:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.049+0000 7f726b3f4740 1 -- 172.21.15.12:0/974235849 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f724c015070 con 0x559643176c20 2024-03-23T23:07:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.049+0000 7f7247400640 1 -- 172.21.15.12:0/974235849 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f724c005650 con 0x559643176c20 2024-03-23T23:07:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.050+0000 7f726b3f4740 10 client.26915 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.050+0000 7f726b3f4740 1 -- 172.21.15.12:0/974235849 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x559642eb57d0 con 0x559643176c20 2024-03-23T23:07:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.050+0000 7f7247400640 1 -- 172.21.15.12:0/974235849 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f724c015070 con 0x559643176c20 2024-03-23T23:07:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.050+0000 7f7247400640 1 -- 172.21.15.12:0/974235849 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f724c004880 con 0x559643176c20 2024-03-23T23:07:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.050+0000 7f726b3f4740 10 client.26915 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.050+0000 7f726b3f4740 10 client.26915 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:07:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.050+0000 7f726b3f4740 20 client.26915 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.050+0000 7f726b3f4740 1 --2- 172.21.15.12:0/974235849 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55964318c540 0x55964318e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.050+0000 7f726b3f4740 4 client.26915 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:07:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.050+0000 7f726b3f4740 1 -- 172.21.15.12:0/974235849 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5596430e4800 con 0x55964318c540 2024-03-23T23:07:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.050+0000 7f7265400640 1 --2- 172.21.15.12:0/974235849 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55964318c540 0x55964318e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.051+0000 7f7265400640 1 --2- 172.21.15.12:0/974235849 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55964318c540 0x55964318e930 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=856c1e5f64cdb8dc server_cookie=bbde7d81288da22 in_seq=0 out_seq=0 2024-03-23T23:07:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.051+0000 7f7247400640 10 client.26915 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:07:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.051+0000 7f7247400640 1 -- 172.21.15.12:0/974235849 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5596430e4800 con 0x55964318c540 2024-03-23T23:07:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.052+0000 7f7247400640 10 client.26915 handle_command_reply: tid=0 2024-03-23T23:07:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.052+0000 7f726b3f4740 1 client.26915 shutdown 2024-03-23T23:07:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.052+0000 7f726b3f4740 1 -- 172.21.15.12:0/974235849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559643176c20 msgr2=0x559643180e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.052+0000 7f726b3f4740 1 --2- 172.21.15.12:0/974235849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559643176c20 0x559643180e60 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f724c005c90 tx=0x7f724c003e00 comp rx=0 tx=0).stop 2024-03-23T23:07:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.052+0000 7f726b3f4740 1 -- 172.21.15.12:0/974235849 shutdown_connections 2024-03-23T23:07:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.052+0000 7f726b3f4740 1 --2- 172.21.15.12:0/974235849 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5596431787b0 0x5596431883e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.052+0000 7f726b3f4740 1 --2- 172.21.15.12:0/974235849 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55964318c540 0x55964318e930 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.052+0000 7f726b3f4740 1 --2- 172.21.15.12:0/974235849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559643177df0 0x5596431813a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.052+0000 7f726b3f4740 1 --2- 172.21.15.12:0/974235849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559643176c20 0x559643180e60 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.052+0000 7f726b3f4740 1 -- 172.21.15.12:0/974235849 >> 172.21.15.12:0/974235849 conn(0x559643172010 msgr2=0x55964316ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.052+0000 7f7247400640 0 client.26915 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:07:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.052+0000 7f726b3f4740 1 -- 172.21.15.12:0/974235849 shutdown_connections 2024-03-23T23:07:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.052+0000 7f726b3f4740 1 -- 172.21.15.12:0/974235849 wait complete. 2024-03-23T23:07:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.052+0000 7f726b3f4740 20 client.26915 trim_cache size 0 max 16384 2024-03-23T23:07:32.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.066+0000 7f726b3f4740 1 Processor -- start 2024-03-23T23:07:32.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.066+0000 7f726b3f4740 1 -- start start 2024-03-23T23:07:32.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.066+0000 7f726b3f4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559643172760 0x559643193150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.066+0000 7f726b3f4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559643172010 0x559643188300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.066+0000 7f726b3f4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559643176c20 0x559643193690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.066+0000 7f726b3f4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72601a8d20 con 0x559643172760 2024-03-23T23:07:32.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.066+0000 7f726b3f4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72601a6820 con 0x559643172010 2024-03-23T23:07:32.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.066+0000 7f726b3f4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72601a48c0 con 0x559643176c20 2024-03-23T23:07:32.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.067+0000 7f7266800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559643176c20 0x559643193690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.067+0000 7f7265400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559643172010 0x559643188300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.067+0000 7f7266800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559643176c20 0x559643193690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:46100/0 (socket says 172.21.15.12:46100) 2024-03-23T23:07:32.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.067+0000 7f7266800640 1 -- 172.21.15.12:0/2849414475 learned_addr learned my addr 172.21.15.12:0/2849414475 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:32.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.067+0000 7f7265e00640 1 --2- 172.21.15.12:0/2849414475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559643172760 0x559643193150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.067+0000 7f7266800640 1 -- 172.21.15.12:0/2849414475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559643172010 msgr2=0x559643188300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:32.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.067+0000 7f7266800640 1 --2- 172.21.15.12:0/2849414475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559643172010 0x559643188300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.067+0000 7f7266800640 1 -- 172.21.15.12:0/2849414475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559643172760 msgr2=0x559643193150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:32.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.067+0000 7f7266800640 1 --2- 172.21.15.12:0/2849414475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559643172760 0x559643193150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.067+0000 7f7266800640 1 -- 172.21.15.12:0/2849414475 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559643177870 con 0x559643176c20 2024-03-23T23:07:32.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.067+0000 7f7266800640 1 --2- 172.21.15.12:0/2849414475 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559643176c20 0x559643193690 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f725c007dc0 tx=0x7f725c004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f70728d291f953df server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:32.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.068+0000 7f7252600640 1 -- 172.21.15.12:0/2849414475 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c0243f0 con 0x559643176c20 2024-03-23T23:07:32.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.068+0000 7f7252600640 1 -- 172.21.15.12:0/2849414475 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f725c029070 con 0x559643176c20 2024-03-23T23:07:32.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.068+0000 7f7252600640 1 -- 172.21.15.12:0/2849414475 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c0176b0 con 0x559643176c20 2024-03-23T23:07:32.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.068+0000 7f726b3f4740 1 -- 172.21.15.12:0/2849414475 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559643176c20 msgr2=0x559643193690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:32.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.068+0000 7f726b3f4740 1 --2- 172.21.15.12:0/2849414475 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559643176c20 0x559643193690 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f725c007dc0 tx=0x7f725c004ab0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.068+0000 7f726b3f4740 1 -- 172.21.15.12:0/2849414475 shutdown_connections 2024-03-23T23:07:32.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.068+0000 7f726b3f4740 1 --2- 172.21.15.12:0/2849414475 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559643176c20 0x559643193690 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.068+0000 7f726b3f4740 1 --2- 172.21.15.12:0/2849414475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559643172010 0x559643188300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.068+0000 7f726b3f4740 1 --2- 172.21.15.12:0/2849414475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559643172760 0x559643193150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.068+0000 7f726b3f4740 1 -- 172.21.15.12:0/2849414475 >> 172.21.15.12:0/2849414475 conn(0x559643175ea0 msgr2=0x55964316ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:32.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.068+0000 7f726b3f4740 1 -- 172.21.15.12:0/2849414475 shutdown_connections 2024-03-23T23:07:32.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.068+0000 7f726b3f4740 1 -- 172.21.15.12:0/2849414475 wait complete. 2024-03-23T23:07:32.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.069+0000 7f726b3f4740 1 -- start start 2024-03-23T23:07:32.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.069+0000 7f726b3f4740 1 Processor -- start 2024-03-23T23:07:32.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.069+0000 7f726b3f4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559643172010 0x559643184420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.069+0000 7f726b3f4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559643172760 0x559643184960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.069+0000 7f726b3f4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559643176c20 0x559643187930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.069+0000 7f726b3f4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72601a4ec0 con 0x559643172760 2024-03-23T23:07:32.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.069+0000 7f726b3f4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72601a61c0 con 0x559643176c20 2024-03-23T23:07:32.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.069+0000 7f726b3f4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72601a5500 con 0x559643172010 2024-03-23T23:07:32.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.069+0000 7f726b3f4740 10 client.0 _ll_register_callbacks cb 0x7f725c0074f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.070+0000 7f7265e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559643172010 0x559643184420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.070+0000 7f7266800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559643176c20 0x559643187930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.070+0000 7f7266800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559643176c20 0x559643187930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44460/0 (socket says 172.21.15.12:44460) 2024-03-23T23:07:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.070+0000 7f7266800640 1 -- 172.21.15.12:0/2395489276 learned_addr learned my addr 172.21.15.12:0/2395489276 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.070+0000 7f7265e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559643172010 0x559643184420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:46110/0 (socket says 172.21.15.12:46110) 2024-03-23T23:07:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.070+0000 7f7265400640 1 --2- 172.21.15.12:0/2395489276 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559643172760 0x559643184960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.070+0000 7f7265e00640 1 -- 172.21.15.12:0/2395489276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559643176c20 msgr2=0x559643187930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:32.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.070+0000 7f7265e00640 1 --2- 172.21.15.12:0/2395489276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559643176c20 0x559643187930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.070+0000 7f7265e00640 1 -- 172.21.15.12:0/2395489276 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559643172760 msgr2=0x559643184960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:32.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.070+0000 7f7265e00640 1 --2- 172.21.15.12:0/2395489276 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559643172760 0x559643184960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.070+0000 7f7265e00640 1 -- 172.21.15.12:0/2395489276 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559643187f20 con 0x559643172010 2024-03-23T23:07:32.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.070+0000 7f7265e00640 1 --2- 172.21.15.12:0/2395489276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559643172010 0x559643184420 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f724c000c00 tx=0x7f724c01adb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b9d084336a1071e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:32.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.070+0000 7f7247400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:07:32.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.071+0000 7f7247400640 1 -- 172.21.15.12:0/2395489276 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c027070 con 0x559643172010 2024-03-23T23:07:32.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.071+0000 7f7247400640 1 -- 172.21.15.12:0/2395489276 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f724c004080 con 0x559643172010 2024-03-23T23:07:32.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.071+0000 7f726b3f4740 1 -- 172.21.15.12:0/2395489276 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f725c029070 con 0x559643172010 2024-03-23T23:07:32.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.071+0000 7f7247400640 1 -- 172.21.15.12:0/2395489276 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f724c0042a0 con 0x559643172010 2024-03-23T23:07:32.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.071+0000 7f726b3f4740 10 client.26893 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:32.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.071+0000 7f726b3f4740 1 -- 172.21.15.12:0/2395489276 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5596431e8700 con 0x559643172010 2024-03-23T23:07:32.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.072+0000 7f7247400640 1 -- 172.21.15.12:0/2395489276 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f725c029070 con 0x559643172010 2024-03-23T23:07:32.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.072+0000 7f7247400640 1 -- 172.21.15.12:0/2395489276 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f724c011650 con 0x559643172010 2024-03-23T23:07:32.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.072+0000 7f726b3f4740 10 client.26893 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:32.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.072+0000 7f726b3f4740 10 client.26893 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:07:32.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.072+0000 7f726b3f4740 20 client.26893 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:32.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.072+0000 7f726b3f4740 1 --2- 172.21.15.12:0/2395489276 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x559643177df0 0x5596431eb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.072+0000 7f726b3f4740 4 client.26893 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:07:32.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.072+0000 7f726b3f4740 1 -- 172.21.15.12:0/2395489276 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5596430deb50 con 0x559643177df0 2024-03-23T23:07:32.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.072+0000 7f7265400640 1 --2- 172.21.15.12:0/2395489276 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x559643177df0 0x5596431eb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.073+0000 7f7265400640 1 --2- 172.21.15.12:0/2395489276 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x559643177df0 0x5596431eb0b0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=12ea97eb038d7ffc server_cookie=c061c3bd236fa175 in_seq=0 out_seq=0 2024-03-23T23:07:32.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.073+0000 7f7247400640 10 client.26893 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:07:32.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.073+0000 7f7247400640 1 -- 172.21.15.12:0/2395489276 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5596430deb50 con 0x559643177df0 2024-03-23T23:07:32.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.073+0000 7f7247400640 10 client.26893 handle_command_reply: tid=0 2024-03-23T23:07:32.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.073+0000 7f726b3f4740 1 client.26893 shutdown 2024-03-23T23:07:32.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.073+0000 7f726b3f4740 1 -- 172.21.15.12:0/2395489276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559643172010 msgr2=0x559643184420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:32.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.073+0000 7f726b3f4740 1 --2- 172.21.15.12:0/2395489276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559643172010 0x559643184420 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f724c000c00 tx=0x7f724c01adb0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.073+0000 7f726b3f4740 1 -- 172.21.15.12:0/2395489276 shutdown_connections 2024-03-23T23:07:32.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.073+0000 7f726b3f4740 1 --2- 172.21.15.12:0/2395489276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559643176c20 0x559643187930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.073+0000 7f726b3f4740 1 --2- 172.21.15.12:0/2395489276 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559643172760 0x559643184960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.073+0000 7f726b3f4740 1 --2- 172.21.15.12:0/2395489276 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x559643177df0 0x5596431eb0b0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.073+0000 7f726b3f4740 1 --2- 172.21.15.12:0/2395489276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559643172010 0x559643184420 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.073+0000 7f7247400640 0 client.26893 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:07:32.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.073+0000 7f726b3f4740 1 -- 172.21.15.12:0/2395489276 >> 172.21.15.12:0/2395489276 conn(0x559643175ea0 msgr2=0x559643193150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:32.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.073+0000 7f726b3f4740 1 -- 172.21.15.12:0/2395489276 shutdown_connections 2024-03-23T23:07:32.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.073+0000 7f726b3f4740 1 -- 172.21.15.12:0/2395489276 wait complete. 2024-03-23T23:07:32.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.074+0000 7f726b3f4740 20 client.26893 trim_cache size 0 max 16384 2024-03-23T23:07:32.079 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:07:32.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.074+0000 7f7268800640 1 -- 172.21.15.12:0/294915265 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f72480a2880 msgr2=0x7f72480a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:32.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.074+0000 7f7268800640 1 --2- 172.21.15.12:0/294915265 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f72480a2880 0x7f72480a4d40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x559643177cf0 tx=0x7f725c00e040 comp rx=0 tx=0).stop 2024-03-23T23:07:32.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.074+0000 7f7268800640 1 -- 172.21.15.12:0/294915265 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7260100ae0 msgr2=0x7f7260197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:32.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.074+0000 7f7268800640 1 --2- 172.21.15.12:0/294915265 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7260100ae0 0x7f7260197f40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7254005ab0 tx=0x7f725402c6c0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.074+0000 7f7268800640 1 -- 172.21.15.12:0/294915265 shutdown_connections 2024-03-23T23:07:32.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.074+0000 7f7268800640 1 --2- 172.21.15.12:0/294915265 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f72480a2880 0x7f72480a4d40 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.074+0000 7f7268800640 1 --2- 172.21.15.12:0/294915265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72601014a0 0x7f726019efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.074+0000 7f7268800640 1 --2- 172.21.15.12:0/294915265 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7260100ae0 0x7f7260197f40 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.074+0000 7f7268800640 1 --2- 172.21.15.12:0/294915265 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72600ff8f0 0x7f7260197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.074+0000 7f7268800640 1 -- 172.21.15.12:0/294915265 >> 172.21.15.12:0/294915265 conn(0x7f72600fae60 msgr2=0x7f7260105ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:32.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.074+0000 7f7268800640 1 -- 172.21.15.12:0/294915265 shutdown_connections 2024-03-23T23:07:32.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.075+0000 7f7268800640 1 -- 172.21.15.12:0/294915265 wait complete. 2024-03-23T23:07:32.094 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:07:32.094 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-23T23:07:32.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:31 smithi173 ceph-mon[36116]: scrub queued for path: / 2024-03-23T23:07:32.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:31 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:07:32.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:31 smithi173 ceph-mon[36116]: scrub summary: active paths [/] 2024-03-23T23:07:32.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:31 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:07:32.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:31 smithi173 ceph-mon[36116]: scrub summary: idle 2024-03-23T23:07:32.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:31 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2885052451' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:07:32.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:31 smithi203 ceph-mon[36693]: scrub queued for path: / 2024-03-23T23:07:32.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:31 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:07:32.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:31 smithi203 ceph-mon[36693]: scrub summary: active paths [/] 2024-03-23T23:07:32.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:31 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:07:32.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:31 smithi203 ceph-mon[36693]: scrub summary: idle 2024-03-23T23:07:32.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:31 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2885052451' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:07:32.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.313+0000 7f066d000640 1 -- 172.21.15.12:0/695501389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0668102130 msgr2=0x7f0668104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:32.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.313+0000 7f066d000640 1 --2- 172.21.15.12:0/695501389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0668102130 0x7f0668104520 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0658009ca0 tx=0x7f065802c100 comp rx=0 tx=0).stop 2024-03-23T23:07:32.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.313+0000 7f066d000640 1 -- 172.21.15.12:0/695501389 shutdown_connections 2024-03-23T23:07:32.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.313+0000 7f066d000640 1 --2- 172.21.15.12:0/695501389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0668107390 0x7f06681097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.313+0000 7f066d000640 1 --2- 172.21.15.12:0/695501389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0668104a60 0x7f0668106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.313+0000 7f066d000640 1 --2- 172.21.15.12:0/695501389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0668102130 0x7f0668104520 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.313+0000 7f066d000640 1 -- 172.21.15.12:0/695501389 >> 172.21.15.12:0/695501389 conn(0x7f06680fbb40 msgr2=0x7f06680fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:32.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.314+0000 7f066d000640 1 -- 172.21.15.12:0/695501389 shutdown_connections 2024-03-23T23:07:32.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.314+0000 7f066d000640 1 -- 172.21.15.12:0/695501389 wait complete. 2024-03-23T23:07:32.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.315+0000 7f066d000640 1 Processor -- start 2024-03-23T23:07:32.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.315+0000 7f066d000640 1 -- start start 2024-03-23T23:07:32.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.315+0000 7f066d000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0668102130 0x7f0668199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.315+0000 7f066d000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0668104a60 0x7f066819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.315+0000 7f066d000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0668107390 0x7f06681a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.315+0000 7f066d000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066810d340 con 0x7f0668107390 2024-03-23T23:07:32.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.315+0000 7f066d000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066810d1c0 con 0x7f0668104a60 2024-03-23T23:07:32.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.315+0000 7f066d000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066810d4c0 con 0x7f0668102130 2024-03-23T23:07:32.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.316+0000 7f065d400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0668104a60 0x7f066819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.316+0000 7f0665e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0668102130 0x7f0668199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.316+0000 7f065d400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0668104a60 0x7f066819a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44480/0 (socket says 172.21.15.12:44480) 2024-03-23T23:07:32.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.316+0000 7f065d400640 1 -- 172.21.15.12:0/2523540538 learned_addr learned my addr 172.21.15.12:0/2523540538 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:32.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.316+0000 7f0666800640 1 --2- 172.21.15.12:0/2523540538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0668107390 0x7f06681a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.316+0000 7f065d400640 1 -- 172.21.15.12:0/2523540538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0668102130 msgr2=0x7f0668199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:32.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.316+0000 7f065d400640 1 --2- 172.21.15.12:0/2523540538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0668102130 0x7f0668199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.316+0000 7f065d400640 1 -- 172.21.15.12:0/2523540538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0668107390 msgr2=0x7f06681a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:32.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.316+0000 7f065d400640 1 --2- 172.21.15.12:0/2523540538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0668107390 0x7f06681a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.316+0000 7f065d400640 1 -- 172.21.15.12:0/2523540538 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06681a1890 con 0x7f0668104a60 2024-03-23T23:07:32.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.316+0000 7f0666800640 1 --2- 172.21.15.12:0/2523540538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0668107390 0x7f06681a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:07:32.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.316+0000 7f065d400640 1 --2- 172.21.15.12:0/2523540538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0668104a60 0x7f066819a1c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f065000e9c0 tx=0x7f065000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:32.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.317+0000 7f065f400640 1 -- 172.21.15.12:0/2523540538 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065000cd80 con 0x7f0668104a60 2024-03-23T23:07:32.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.317+0000 7f065f400640 1 -- 172.21.15.12:0/2523540538 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0650005500 con 0x7f0668104a60 2024-03-23T23:07:32.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.317+0000 7f065f400640 1 -- 172.21.15.12:0/2523540538 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0650010690 con 0x7f0668104a60 2024-03-23T23:07:32.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.317+0000 7f066d000640 1 -- 172.21.15.12:0/2523540538 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06681a1b80 con 0x7f0668104a60 2024-03-23T23:07:32.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.317+0000 7f066d000640 1 -- 172.21.15.12:0/2523540538 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06681a20a0 con 0x7f0668104a60 2024-03-23T23:07:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.321+0000 7f066fc06740 1 Processor -- start 2024-03-23T23:07:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.321+0000 7f066fc06740 1 -- start start 2024-03-23T23:07:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.322+0000 7f066fc06740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6d0acc810 0x55a6d0accc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.322+0000 7f066fc06740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acd9e0 0x55a6d0acde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.322+0000 7f066fc06740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a6d0ace3a0 0x55a6d0ad4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.322+0000 7f066fc06740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6d0a312c0 con 0x55a6d0ace3a0 2024-03-23T23:07:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.322+0000 7f066fc06740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6d09d9c90 con 0x55a6d0acd9e0 2024-03-23T23:07:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.322+0000 7f066fc06740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6d0a2d360 con 0x55a6d0acc810 2024-03-23T23:07:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.322+0000 7f065f400640 1 -- 172.21.15.12:0/2523540538 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f06500046c0 con 0x7f0668104a60 2024-03-23T23:07:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.322+0000 7f0665e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acd9e0 0x55a6d0acde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.322+0000 7f0665e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acd9e0 0x55a6d0acde60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44482/0 (socket says 172.21.15.12:44482) 2024-03-23T23:07:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.322+0000 7f0665e00640 1 -- 172.21.15.12:0/463799232 learned_addr learned my addr 172.21.15.12:0/463799232 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.322+0000 7f0666800640 1 --2- 172.21.15.12:0/463799232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6d0acc810 0x55a6d0accc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.322+0000 7f065d400640 1 --2- 172.21.15.12:0/463799232 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a6d0ace3a0 0x55a6d0ad4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.323+0000 7f0665e00640 1 -- 172.21.15.12:0/463799232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6d0acc810 msgr2=0x55a6d0accc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:32.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.323+0000 7f0665e00640 1 --2- 172.21.15.12:0/463799232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6d0acc810 0x55a6d0accc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.323+0000 7f0665e00640 1 -- 172.21.15.12:0/463799232 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a6d0ace3a0 msgr2=0x55a6d0ad4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:32.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.323+0000 7f0665e00640 1 --2- 172.21.15.12:0/463799232 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a6d0ace3a0 0x55a6d0ad4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.323+0000 7f0665e00640 1 -- 172.21.15.12:0/463799232 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6d09fe080 con 0x55a6d0acd9e0 2024-03-23T23:07:32.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.323+0000 7f065d400640 1 --2- 172.21.15.12:0/463799232 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a6d0ace3a0 0x55a6d0ad4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:07:32.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.323+0000 7f0665e00640 1 --2- 172.21.15.12:0/463799232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acd9e0 0x55a6d0acde60 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0658005e30 tx=0x7f0658005f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98998a1b2e106686 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:32.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.323+0000 7f0657400640 1 -- 172.21.15.12:0/463799232 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0658043070 con 0x55a6d0acd9e0 2024-03-23T23:07:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.323+0000 7f0657400640 1 -- 172.21.15.12:0/463799232 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0658003970 con 0x55a6d0acd9e0 2024-03-23T23:07:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.324+0000 7f0657400640 1 -- 172.21.15.12:0/463799232 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0658039040 con 0x55a6d0acd9e0 2024-03-23T23:07:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.324+0000 7f065f400640 1 --2- 172.21.15.12:0/2523540538 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f064c0a2800 0x7f064c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.324+0000 7f065f400640 1 -- 172.21.15.12:0/2523540538 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f06500cef50 con 0x7f0668104a60 2024-03-23T23:07:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.324+0000 7f066fc06740 1 -- 172.21.15.12:0/463799232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acd9e0 msgr2=0x55a6d0acde60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.324+0000 7f066fc06740 1 --2- 172.21.15.12:0/463799232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acd9e0 0x55a6d0acde60 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0658005e30 tx=0x7f0658005f60 comp rx=0 tx=0).stop 2024-03-23T23:07:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.324+0000 7f0666800640 1 --2- 172.21.15.12:0/2523540538 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f064c0a2800 0x7f064c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.324+0000 7f066fc06740 1 -- 172.21.15.12:0/463799232 shutdown_connections 2024-03-23T23:07:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.324+0000 7f066fc06740 1 --2- 172.21.15.12:0/463799232 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a6d0ace3a0 0x55a6d0ad4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.324+0000 7f066fc06740 1 --2- 172.21.15.12:0/463799232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acd9e0 0x55a6d0acde60 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.324+0000 7f066fc06740 1 --2- 172.21.15.12:0/463799232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6d0acc810 0x55a6d0accc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.324+0000 7f066fc06740 1 -- 172.21.15.12:0/463799232 >> 172.21.15.12:0/463799232 conn(0x55a6d0ac7c00 msgr2=0x55a6d0ac8000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.324+0000 7f066fc06740 1 -- 172.21.15.12:0/463799232 shutdown_connections 2024-03-23T23:07:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.325+0000 7f066fc06740 1 -- 172.21.15.12:0/463799232 wait complete. 2024-03-23T23:07:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.325+0000 7f0666800640 1 --2- 172.21.15.12:0/2523540538 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f064c0a2800 0x7f064c0a4cc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x55a6d0acd8e0 tx=0x7f0660005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.325+0000 7f066fc06740 1 -- start start 2024-03-23T23:07:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.325+0000 7f066fc06740 1 Processor -- start 2024-03-23T23:07:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.326+0000 7f066fc06740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acc810 0x55a6d0ad6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.326+0000 7f066fc06740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a6d0acd9e0 0x55a6d0ad6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.326+0000 7f066fc06740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6d0ace3a0 0x55a6d0addfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.326+0000 7f066fc06740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6d0ad5350 con 0x55a6d0acd9e0 2024-03-23T23:07:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.326+0000 7f066fc06740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6d09ed0d0 con 0x55a6d0acc810 2024-03-23T23:07:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.326+0000 7f066fc06740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6d0ad54d0 con 0x55a6d0ace3a0 2024-03-23T23:07:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.326+0000 7f066fc06740 10 client.0 _ll_register_callbacks cb 0x55a6d09f1e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.326+0000 7f0665e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acc810 0x55a6d0ad6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.326+0000 7f0665e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acc810 0x55a6d0ad6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44484/0 (socket says 172.21.15.12:44484) 2024-03-23T23:07:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.326+0000 7f065d400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a6d0acd9e0 0x55a6d0ad6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.326+0000 7f0666800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6d0ace3a0 0x55a6d0addfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.326+0000 7f0665e00640 1 -- 172.21.15.12:0/1278906039 learned_addr learned my addr 172.21.15.12:0/1278906039 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.326+0000 7f0665e00640 1 -- 172.21.15.12:0/1278906039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6d0ace3a0 msgr2=0x55a6d0addfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.326+0000 7f0665e00640 1 --2- 172.21.15.12:0/1278906039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6d0ace3a0 0x55a6d0addfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.326+0000 7f0665e00640 1 -- 172.21.15.12:0/1278906039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a6d0acd9e0 msgr2=0x55a6d0ad6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:32.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.326+0000 7f0665e00640 1 --2- 172.21.15.12:0/1278906039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a6d0acd9e0 0x55a6d0ad6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.327+0000 7f0665e00640 1 -- 172.21.15.12:0/1278906039 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a6d07453f0 con 0x55a6d0acc810 2024-03-23T23:07:32.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.327+0000 7f0665e00640 1 --2- 172.21.15.12:0/1278906039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acc810 0x55a6d0ad6a20 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f065802c640 tx=0x7f0658032a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad855924e52c72e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:32.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.327+0000 7f064be00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:07:32.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.327+0000 7f064be00640 1 -- 172.21.15.12:0/1278906039 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065804c030 con 0x55a6d0acc810 2024-03-23T23:07:32.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.327+0000 7f064be00640 1 -- 172.21.15.12:0/1278906039 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0658003b20 con 0x55a6d0acc810 2024-03-23T23:07:32.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.327+0000 7f066fc06740 1 -- 172.21.15.12:0/1278906039 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0658003970 con 0x55a6d0acc810 2024-03-23T23:07:32.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.327+0000 7f064be00640 1 -- 172.21.15.12:0/1278906039 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0658039070 con 0x55a6d0acc810 2024-03-23T23:07:32.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.327+0000 7f066fc06740 10 client.26933 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:32.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.327+0000 7f066fc06740 1 -- 172.21.15.12:0/1278906039 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a6d080f7d0 con 0x55a6d0acc810 2024-03-23T23:07:32.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.328+0000 7f064be00640 1 -- 172.21.15.12:0/1278906039 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0658003970 con 0x55a6d0acc810 2024-03-23T23:07:32.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.328+0000 7f064be00640 1 -- 172.21.15.12:0/1278906039 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f0658048570 con 0x55a6d0acc810 2024-03-23T23:07:32.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.328+0000 7f066fc06740 10 client.26933 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:32.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.328+0000 7f066fc06740 10 client.26933 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:07:32.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.328+0000 7f066fc06740 20 client.26933 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:32.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.328+0000 7f066fc06740 1 --2- 172.21.15.12:0/1278906039 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a6d0ae2070 0x55a6d0ae4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.328+0000 7f066fc06740 4 client.26933 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:07:32.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.328+0000 7f066fc06740 1 -- 172.21.15.12:0/1278906039 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a6d0a3e800 con 0x55a6d0ae2070 2024-03-23T23:07:32.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.328+0000 7f065d400640 1 --2- 172.21.15.12:0/1278906039 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a6d0ae2070 0x55a6d0ae4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.329+0000 7f065d400640 1 --2- 172.21.15.12:0/1278906039 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a6d0ae2070 0x55a6d0ae4460 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfedc084e4412ba1 server_cookie=e14d8e95d06570f4 in_seq=0 out_seq=0 2024-03-23T23:07:32.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.329+0000 7f064be00640 10 client.26933 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:32.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.329+0000 7f064be00640 1 -- 172.21.15.12:0/1278906039 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a6d0a3e800 con 0x55a6d0ae2070 2024-03-23T23:07:32.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.329+0000 7f064be00640 10 client.26933 handle_command_reply: tid=0 2024-03-23T23:07:32.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.330+0000 7f066fc06740 1 client.26933 shutdown 2024-03-23T23:07:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.330+0000 7f066fc06740 1 -- 172.21.15.12:0/1278906039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acc810 msgr2=0x55a6d0ad6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.330+0000 7f066fc06740 1 --2- 172.21.15.12:0/1278906039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acc810 0x55a6d0ad6a20 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f065802c640 tx=0x7f0658032a70 comp rx=0 tx=0).stop 2024-03-23T23:07:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.330+0000 7f066fc06740 1 -- 172.21.15.12:0/1278906039 shutdown_connections 2024-03-23T23:07:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.330+0000 7f066fc06740 1 --2- 172.21.15.12:0/1278906039 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a6d0ae2070 0x55a6d0ae4460 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.330+0000 7f066fc06740 1 --2- 172.21.15.12:0/1278906039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6d0ace3a0 0x55a6d0addfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.330+0000 7f066fc06740 1 --2- 172.21.15.12:0/1278906039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a6d0acd9e0 0x55a6d0ad6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.330+0000 7f066fc06740 1 --2- 172.21.15.12:0/1278906039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acc810 0x55a6d0ad6a20 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.330+0000 7f066fc06740 1 -- 172.21.15.12:0/1278906039 >> 172.21.15.12:0/1278906039 conn(0x55a6d0ac7c00 msgr2=0x55a6d0ac6eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.330+0000 7f064be00640 0 client.26933 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.330+0000 7f066fc06740 1 -- 172.21.15.12:0/1278906039 shutdown_connections 2024-03-23T23:07:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.330+0000 7f066fc06740 1 -- 172.21.15.12:0/1278906039 wait complete. 2024-03-23T23:07:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.330+0000 7f066fc06740 20 client.26933 trim_cache size 0 max 16384 2024-03-23T23:07:32.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.344+0000 7f066fc06740 1 Processor -- start 2024-03-23T23:07:32.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.344+0000 7f066fc06740 1 -- start start 2024-03-23T23:07:32.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.344+0000 7f066fc06740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acc810 0x55a6d0ad09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.344+0000 7f066fc06740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6d0acd9e0 0x55a6d0ad0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.344+0000 7f066fc06740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a6d0ace3a0 0x55a6d0ad1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.344+0000 7f066fc06740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06681ad1d0 con 0x55a6d0ace3a0 2024-03-23T23:07:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.344+0000 7f066fc06740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06681aacd0 con 0x55a6d0acc810 2024-03-23T23:07:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.344+0000 7f066fc06740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06681a8d70 con 0x55a6d0acd9e0 2024-03-23T23:07:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.345+0000 7f0665e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acc810 0x55a6d0ad09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.345+0000 7f0665e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acc810 0x55a6d0ad09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44486/0 (socket says 172.21.15.12:44486) 2024-03-23T23:07:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.345+0000 7f0665e00640 1 -- 172.21.15.12:0/520196723 learned_addr learned my addr 172.21.15.12:0/520196723 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.345+0000 7f065d400640 1 --2- 172.21.15.12:0/520196723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6d0acd9e0 0x55a6d0ad0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.345+0000 7f0666800640 1 --2- 172.21.15.12:0/520196723 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a6d0ace3a0 0x55a6d0ad1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.345+0000 7f0665e00640 1 -- 172.21.15.12:0/520196723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6d0acd9e0 msgr2=0x55a6d0ad0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.345+0000 7f0665e00640 1 --2- 172.21.15.12:0/520196723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6d0acd9e0 0x55a6d0ad0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.345+0000 7f0665e00640 1 -- 172.21.15.12:0/520196723 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a6d0ace3a0 msgr2=0x55a6d0ad1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.345+0000 7f0665e00640 1 --2- 172.21.15.12:0/520196723 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a6d0ace3a0 0x55a6d0ad1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.345+0000 7f0665e00640 1 -- 172.21.15.12:0/520196723 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6d0ac8970 con 0x55a6d0acc810 2024-03-23T23:07:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.345+0000 7f0665e00640 1 --2- 172.21.15.12:0/520196723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acc810 0x55a6d0ad09e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f0658000f80 tx=0x7f0658009ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97e9fdaea67648f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.345+0000 7f0656a00640 1 -- 172.21.15.12:0/520196723 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065804c070 con 0x55a6d0acc810 2024-03-23T23:07:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.345+0000 7f0656a00640 1 -- 172.21.15.12:0/520196723 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0658048450 con 0x55a6d0acc810 2024-03-23T23:07:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.345+0000 7f0656a00640 1 -- 172.21.15.12:0/520196723 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0658043040 con 0x55a6d0acc810 2024-03-23T23:07:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.346+0000 7f066fc06740 1 -- 172.21.15.12:0/520196723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acc810 msgr2=0x55a6d0ad09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.346+0000 7f066fc06740 1 --2- 172.21.15.12:0/520196723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acc810 0x55a6d0ad09e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f0658000f80 tx=0x7f0658009ba0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.346+0000 7f066fc06740 1 -- 172.21.15.12:0/520196723 shutdown_connections 2024-03-23T23:07:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.346+0000 7f066fc06740 1 --2- 172.21.15.12:0/520196723 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a6d0ace3a0 0x55a6d0ad1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.346+0000 7f066fc06740 1 --2- 172.21.15.12:0/520196723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6d0acd9e0 0x55a6d0ad0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.346+0000 7f066fc06740 1 --2- 172.21.15.12:0/520196723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acc810 0x55a6d0ad09e0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.346+0000 7f066fc06740 1 -- 172.21.15.12:0/520196723 >> 172.21.15.12:0/520196723 conn(0x55a6d0ac7c00 msgr2=0x55a6d0acb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.346+0000 7f066fc06740 1 -- 172.21.15.12:0/520196723 shutdown_connections 2024-03-23T23:07:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.346+0000 7f066fc06740 1 -- 172.21.15.12:0/520196723 wait complete. 2024-03-23T23:07:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.347+0000 7f066fc06740 1 -- start start 2024-03-23T23:07:32.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.347+0000 7f066fc06740 1 Processor -- start 2024-03-23T23:07:32.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.347+0000 7f066fc06740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6d0acc810 0x55a6d0ad93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.347+0000 7f066fc06740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acd9e0 0x55a6d0ad98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.347+0000 7f066fc06740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a6d0ace3a0 0x55a6d0adc910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.347+0000 7f066fc06740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06681a9370 con 0x55a6d0ace3a0 2024-03-23T23:07:32.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.347+0000 7f066fc06740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06681aa670 con 0x55a6d0acd9e0 2024-03-23T23:07:32.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.347+0000 7f066fc06740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06681a99b0 con 0x55a6d0acc810 2024-03-23T23:07:32.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.347+0000 7f066fc06740 10 client.0 _ll_register_callbacks cb 0x7f06600095d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:32.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.348+0000 7f065d400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acd9e0 0x55a6d0ad98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.348+0000 7f065d400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acd9e0 0x55a6d0ad98f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44496/0 (socket says 172.21.15.12:44496) 2024-03-23T23:07:32.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.348+0000 7f0665e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6d0acc810 0x55a6d0ad93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.348+0000 7f065d400640 1 -- 172.21.15.12:0/1522971441 learned_addr learned my addr 172.21.15.12:0/1522971441 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:32.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.348+0000 7f0666800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a6d0ace3a0 0x55a6d0adc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.348+0000 7f065d400640 1 -- 172.21.15.12:0/1522971441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6d0acc810 msgr2=0x55a6d0ad93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:32.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.348+0000 7f065d400640 1 --2- 172.21.15.12:0/1522971441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6d0acc810 0x55a6d0ad93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.348+0000 7f065d400640 1 -- 172.21.15.12:0/1522971441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a6d0ace3a0 msgr2=0x55a6d0adc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:32.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.348+0000 7f065d400640 1 --2- 172.21.15.12:0/1522971441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a6d0ace3a0 0x55a6d0adc910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.348+0000 7f065d400640 1 -- 172.21.15.12:0/1522971441 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a6d0adcf00 con 0x55a6d0acd9e0 2024-03-23T23:07:32.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.348+0000 7f0665e00640 1 --2- 172.21.15.12:0/1522971441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6d0acc810 0x55a6d0ad93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:07:32.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.348+0000 7f065d400640 1 --2- 172.21.15.12:0/1522971441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acd9e0 0x55a6d0ad98f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f0650000c00 tx=0x7f0650086040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab38cd718f8f3421 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:32.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.348+0000 7f064be00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:07:32.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.348+0000 7f064be00640 1 -- 172.21.15.12:0/1522971441 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065009bfa0 con 0x55a6d0acd9e0 2024-03-23T23:07:32.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.349+0000 7f064be00640 1 -- 172.21.15.12:0/1522971441 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f06500860b0 con 0x55a6d0acd9e0 2024-03-23T23:07:32.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.349+0000 7f066fc06740 1 -- 172.21.15.12:0/1522971441 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0658048450 con 0x55a6d0acd9e0 2024-03-23T23:07:32.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.349+0000 7f064be00640 1 -- 172.21.15.12:0/1522971441 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f065009b8a0 con 0x55a6d0acd9e0 2024-03-23T23:07:32.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.349+0000 7f066fc06740 10 client.26939 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:32.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.349+0000 7f066fc06740 1 -- 172.21.15.12:0/1522971441 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a6d0b3dd10 con 0x55a6d0acd9e0 2024-03-23T23:07:32.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.349+0000 7f064be00640 1 -- 172.21.15.12:0/1522971441 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0658048450 con 0x55a6d0acd9e0 2024-03-23T23:07:32.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.349+0000 7f064be00640 1 -- 172.21.15.12:0/1522971441 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f065009cbd0 con 0x55a6d0acd9e0 2024-03-23T23:07:32.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.349+0000 7f066fc06740 10 client.26939 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:32.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.349+0000 7f066fc06740 10 client.26939 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:07:32.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.349+0000 7f066fc06740 20 client.26939 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:32.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.349+0000 7f066fc06740 1 --2- 172.21.15.12:0/1522971441 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a6d0b3e520 0x55a6d0b409a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:32.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.349+0000 7f066fc06740 4 client.26939 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:07:32.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.349+0000 7f066fc06740 1 -- 172.21.15.12:0/1522971441 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a6d0a38b50 con 0x55a6d0b3e520 2024-03-23T23:07:32.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.350+0000 7f0665e00640 1 --2- 172.21.15.12:0/1522971441 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a6d0b3e520 0x55a6d0b409a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:32.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.350+0000 7f0665e00640 1 --2- 172.21.15.12:0/1522971441 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a6d0b3e520 0x55a6d0b409a0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82e28887415addd2 server_cookie=d1987d721e49006f in_seq=0 out_seq=0 2024-03-23T23:07:32.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.350+0000 7f064be00640 10 client.26939 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:32.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.350+0000 7f064be00640 1 -- 172.21.15.12:0/1522971441 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a6d0a38b50 con 0x55a6d0b3e520 2024-03-23T23:07:32.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.350+0000 7f064be00640 10 client.26939 handle_command_reply: tid=0 2024-03-23T23:07:32.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.350+0000 7f066fc06740 1 client.26939 shutdown 2024-03-23T23:07:32.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.350+0000 7f066fc06740 1 -- 172.21.15.12:0/1522971441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acd9e0 msgr2=0x55a6d0ad98f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:32.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.350+0000 7f066fc06740 1 --2- 172.21.15.12:0/1522971441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acd9e0 0x55a6d0ad98f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f0650000c00 tx=0x7f0650086040 comp rx=0 tx=0).stop 2024-03-23T23:07:32.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.350+0000 7f066fc06740 1 -- 172.21.15.12:0/1522971441 shutdown_connections 2024-03-23T23:07:32.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.350+0000 7f066fc06740 1 --2- 172.21.15.12:0/1522971441 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a6d0b3e520 0x55a6d0b409a0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.350+0000 7f066fc06740 1 --2- 172.21.15.12:0/1522971441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a6d0ace3a0 0x55a6d0adc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.350+0000 7f066fc06740 1 --2- 172.21.15.12:0/1522971441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a6d0acd9e0 0x55a6d0ad98f0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.350+0000 7f066fc06740 1 --2- 172.21.15.12:0/1522971441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6d0acc810 0x55a6d0ad93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.350+0000 7f066fc06740 1 -- 172.21.15.12:0/1522971441 >> 172.21.15.12:0/1522971441 conn(0x55a6d0ac7c00 msgr2=0x55a6d0ac6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:32.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.350+0000 7f064be00640 0 client.26939 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:32.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.351+0000 7f066fc06740 1 -- 172.21.15.12:0/1522971441 shutdown_connections 2024-03-23T23:07:32.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.351+0000 7f066fc06740 1 -- 172.21.15.12:0/1522971441 wait complete. 2024-03-23T23:07:32.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.351+0000 7f066fc06740 20 client.26939 trim_cache size 0 max 16384 2024-03-23T23:07:32.356 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:07:32.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.351+0000 7f066d000640 1 -- 172.21.15.12:0/2523540538 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f064c0a2800 msgr2=0x7f064c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:32.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.351+0000 7f066d000640 1 --2- 172.21.15.12:0/2523540538 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f064c0a2800 0x7f064c0a4cc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x55a6d0acd8e0 tx=0x7f0660005f90 comp rx=0 tx=0).stop 2024-03-23T23:07:32.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.351+0000 7f066d000640 1 -- 172.21.15.12:0/2523540538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0668104a60 msgr2=0x7f066819a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:32.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.351+0000 7f066d000640 1 --2- 172.21.15.12:0/2523540538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0668104a60 0x7f066819a1c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f065000e9c0 tx=0x7f065000ee80 comp rx=0 tx=0).stop 2024-03-23T23:07:32.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.351+0000 7f066d000640 1 -- 172.21.15.12:0/2523540538 shutdown_connections 2024-03-23T23:07:32.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.351+0000 7f066d000640 1 --2- 172.21.15.12:0/2523540538 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f064c0a2800 0x7f064c0a4cc0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.351+0000 7f066d000640 1 --2- 172.21.15.12:0/2523540538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0668107390 0x7f06681a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.351+0000 7f066d000640 1 --2- 172.21.15.12:0/2523540538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0668104a60 0x7f066819a1c0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.351+0000 7f066d000640 1 --2- 172.21.15.12:0/2523540538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0668102130 0x7f0668199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.351+0000 7f066d000640 1 -- 172.21.15.12:0/2523540538 >> 172.21.15.12:0/2523540538 conn(0x7f06680fbb40 msgr2=0x7f06681018e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.352+0000 7f066d000640 1 -- 172.21.15.12:0/2523540538 shutdown_connections 2024-03-23T23:07:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:32.352+0000 7f066d000640 1 -- 172.21.15.12:0/2523540538 wait complete. 2024-03-23T23:07:33.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:32 smithi012 ceph-mon[31695]: pgmap v562: 97 pgs: 97 active+clean; 3.5 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-23T23:07:33.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:32 smithi173 ceph-mon[36116]: pgmap v562: 97 pgs: 97 active+clean; 3.5 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-23T23:07:33.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:32 smithi203 ceph-mon[36693]: pgmap v562: 97 pgs: 97 active+clean; 3.5 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-23T23:07:33.370 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-23T23:07:33.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:07:33.370 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-23T23:07:33.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.609+0000 7f19e3e00640 1 -- 172.21.15.12:0/2283476759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19e4102770 msgr2=0x7f19e4102bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:33.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.609+0000 7f19e3e00640 1 --2- 172.21.15.12:0/2283476759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19e4102770 0x7f19e4102bf0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f19d0009900 tx=0x7f19d002bc50 comp rx=0 tx=0).stop 2024-03-23T23:07:33.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.609+0000 7f19e3e00640 1 -- 172.21.15.12:0/2283476759 shutdown_connections 2024-03-23T23:07:33.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.609+0000 7f19e3e00640 1 --2- 172.21.15.12:0/2283476759 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19e4103130 0x7f19e41099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.609+0000 7f19e3e00640 1 --2- 172.21.15.12:0/2283476759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19e4102770 0x7f19e4102bf0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.609+0000 7f19e3e00640 1 --2- 172.21.15.12:0/2283476759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19e4101580 0x7f19e41019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.609+0000 7f19e3e00640 1 -- 172.21.15.12:0/2283476759 >> 172.21.15.12:0/2283476759 conn(0x7f19e40fcb10 msgr2=0x7f19e40fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:33.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.609+0000 7f19e3e00640 1 -- 172.21.15.12:0/2283476759 shutdown_connections 2024-03-23T23:07:33.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.609+0000 7f19e3e00640 1 -- 172.21.15.12:0/2283476759 wait complete. 2024-03-23T23:07:33.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.610+0000 7f19e3e00640 1 Processor -- start 2024-03-23T23:07:33.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.610+0000 7f19e3e00640 1 -- start start 2024-03-23T23:07:33.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.610+0000 7f19e3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19e4101580 0x7f19e4075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.610+0000 7f19e3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19e4102770 0x7f19e4073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.610+0000 7f19e3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19e4103130 0x7f19e4073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.610+0000 7f19e3e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e410d250 con 0x7f19e4102770 2024-03-23T23:07:33.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.610+0000 7f19e3e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e410d0d0 con 0x7f19e4103130 2024-03-23T23:07:33.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.610+0000 7f19e3e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e410d3d0 con 0x7f19e4101580 2024-03-23T23:07:33.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.611+0000 7f19e3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19e4103130 0x7f19e4073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.611+0000 7f19e2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19e4101580 0x7f19e4075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.611+0000 7f19e2000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19e4102770 0x7f19e4073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.611+0000 7f19e3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19e4103130 0x7f19e4073fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44506/0 (socket says 172.21.15.12:44506) 2024-03-23T23:07:33.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.611+0000 7f19e3400640 1 -- 172.21.15.12:0/1927166214 learned_addr learned my addr 172.21.15.12:0/1927166214 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:33.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.611+0000 7f19e3400640 1 -- 172.21.15.12:0/1927166214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19e4101580 msgr2=0x7f19e4075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:33.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.611+0000 7f19e3400640 1 --2- 172.21.15.12:0/1927166214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19e4101580 0x7f19e4075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.611+0000 7f19e3400640 1 -- 172.21.15.12:0/1927166214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19e4102770 msgr2=0x7f19e4073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:33.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.611+0000 7f19e3400640 1 --2- 172.21.15.12:0/1927166214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19e4102770 0x7f19e4073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.611+0000 7f19e3400640 1 -- 172.21.15.12:0/1927166214 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e4074860 con 0x7f19e4103130 2024-03-23T23:07:33.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.611+0000 7f19e3400640 1 --2- 172.21.15.12:0/1927166214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19e4103130 0x7f19e4073fd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f19dc00eb10 tx=0x7f19dc00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:33.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.612+0000 7f19db400640 1 -- 172.21.15.12:0/1927166214 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19dc004b20 con 0x7f19e4103130 2024-03-23T23:07:33.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.612+0000 7f19db400640 1 -- 172.21.15.12:0/1927166214 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19dc004cc0 con 0x7f19e4103130 2024-03-23T23:07:33.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.612+0000 7f19db400640 1 -- 172.21.15.12:0/1927166214 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19dc010760 con 0x7f19e4103130 2024-03-23T23:07:33.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.612+0000 7f19e3e00640 1 -- 172.21.15.12:0/1927166214 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e41a1980 con 0x7f19e4103130 2024-03-23T23:07:33.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.612+0000 7f19e3e00640 1 -- 172.21.15.12:0/1927166214 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e41a1e50 con 0x7f19e4103130 2024-03-23T23:07:33.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.615+0000 7f19ec576740 1 Processor -- start 2024-03-23T23:07:33.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.615+0000 7f19db400640 1 -- 172.21.15.12:0/1927166214 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f19dc005020 con 0x7f19e4103130 2024-03-23T23:07:33.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.616+0000 7f19ec576740 1 -- start start 2024-03-23T23:07:33.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.616+0000 7f19ec576740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f72a2232b0 0x55f72a2236d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.616+0000 7f19ec576740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a223ca0 0x55f72a2287a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.616+0000 7f19ec576740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f72a228e10 0x55f72a22b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.616+0000 7f19ec576740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f72a188eb0 con 0x55f72a2232b0 2024-03-23T23:07:33.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.616+0000 7f19ec576740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f72a131a60 con 0x55f72a223ca0 2024-03-23T23:07:33.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.616+0000 7f19ec576740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f72a184f50 con 0x55f72a228e10 2024-03-23T23:07:33.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.616+0000 7f19e2a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a223ca0 0x55f72a2287a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.616+0000 7f19e2a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a223ca0 0x55f72a2287a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44522/0 (socket says 172.21.15.12:44522) 2024-03-23T23:07:33.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.616+0000 7f19e2000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f72a2232b0 0x55f72a2236d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.616+0000 7f19e2a00640 1 -- 172.21.15.12:0/4067906597 learned_addr learned my addr 172.21.15.12:0/4067906597 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:33.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.616+0000 7f19e2000640 1 --2- 172.21.15.12:0/4067906597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f72a228e10 0x55f72a22b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.617+0000 7f19e2a00640 1 -- 172.21.15.12:0/4067906597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f72a228e10 msgr2=0x55f72a22b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:33.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.617+0000 7f19e2a00640 1 --2- 172.21.15.12:0/4067906597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f72a228e10 0x55f72a22b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.617+0000 7f19e2a00640 1 -- 172.21.15.12:0/4067906597 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f72a2232b0 msgr2=0x55f72a2236d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:33.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.617+0000 7f19e2a00640 1 --2- 172.21.15.12:0/4067906597 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f72a2232b0 0x55f72a2236d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.617+0000 7f19e2a00640 1 -- 172.21.15.12:0/4067906597 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f729f657e0 con 0x55f72a223ca0 2024-03-23T23:07:33.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.617+0000 7f19e2000640 1 --2- 172.21.15.12:0/4067906597 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f72a2232b0 0x55f72a2236d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:07:33.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.617+0000 7f19db400640 1 --2- 172.21.15.12:0/1927166214 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f19c80a2880 0x7f19c80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.617+0000 7f19e2a00640 1 --2- 172.21.15.12:0/4067906597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a223ca0 0x55f72a2287a0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f19d4009c80 tx=0x7f19d4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53a5b61d7a19196d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:33.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.617+0000 7f19db400640 1 -- 172.21.15.12:0/1927166214 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f19dc0cf150 con 0x7f19e4103130 2024-03-23T23:07:33.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.617+0000 7f19cfe00640 1 -- 172.21.15.12:0/4067906597 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d401e070 con 0x55f72a223ca0 2024-03-23T23:07:33.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.617+0000 7f19cfe00640 1 -- 172.21.15.12:0/4067906597 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19d4009e30 con 0x55f72a223ca0 2024-03-23T23:07:33.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.617+0000 7f19cfe00640 1 -- 172.21.15.12:0/4067906597 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d400ea80 con 0x55f72a223ca0 2024-03-23T23:07:33.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.617+0000 7f19e2000640 1 --2- 172.21.15.12:0/1927166214 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f19c80a2880 0x7f19c80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.618+0000 7f19ec576740 1 -- 172.21.15.12:0/4067906597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a223ca0 msgr2=0x55f72a2287a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:33.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.618+0000 7f19ec576740 1 --2- 172.21.15.12:0/4067906597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a223ca0 0x55f72a2287a0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f19d4009c80 tx=0x7f19d4011810 comp rx=0 tx=0).stop 2024-03-23T23:07:33.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.618+0000 7f19e2000640 1 --2- 172.21.15.12:0/1927166214 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f19c80a2880 0x7f19c80a4d40 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f19d002c260 tx=0x7f19d0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:33.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.618+0000 7f19ec576740 1 -- 172.21.15.12:0/4067906597 shutdown_connections 2024-03-23T23:07:33.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.618+0000 7f19ec576740 1 --2- 172.21.15.12:0/4067906597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f72a228e10 0x55f72a22b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.618+0000 7f19ec576740 1 --2- 172.21.15.12:0/4067906597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a223ca0 0x55f72a2287a0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.618+0000 7f19ec576740 1 --2- 172.21.15.12:0/4067906597 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f72a2232b0 0x55f72a2236d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.618+0000 7f19ec576740 1 -- 172.21.15.12:0/4067906597 >> 172.21.15.12:0/4067906597 conn(0x55f72a21e820 msgr2=0x55f72a21ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:33.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.618+0000 7f19ec576740 1 -- 172.21.15.12:0/4067906597 shutdown_connections 2024-03-23T23:07:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.618+0000 7f19ec576740 1 -- 172.21.15.12:0/4067906597 wait complete. 2024-03-23T23:07:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.619+0000 7f19ec576740 1 -- start start 2024-03-23T23:07:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.619+0000 7f19ec576740 1 Processor -- start 2024-03-23T23:07:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.619+0000 7f19ec576740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f72a2232b0 0x55f72a22d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.619+0000 7f19ec576740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a223ca0 0x55f72a22d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.619+0000 7f19ec576740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f72a228e10 0x55f72a2349f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.619+0000 7f19ec576740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f72a22bdc0 con 0x55f72a2232b0 2024-03-23T23:07:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.619+0000 7f19ec576740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f72a144eb0 con 0x55f72a223ca0 2024-03-23T23:07:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.619+0000 7f19ec576740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f72a22bf40 con 0x55f72a228e10 2024-03-23T23:07:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.620+0000 7f19ec576740 10 client.0 _ll_register_callbacks cb 0x55f72a139150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.620+0000 7f19e2000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a223ca0 0x55f72a22d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.620+0000 7f19e2000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a223ca0 0x55f72a22d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44530/0 (socket says 172.21.15.12:44530) 2024-03-23T23:07:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.620+0000 7f19e2000640 1 -- 172.21.15.12:0/3952752059 learned_addr learned my addr 172.21.15.12:0/3952752059 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.620+0000 7f19e3400640 1 --2- 172.21.15.12:0/3952752059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f72a228e10 0x55f72a2349f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.620+0000 7f19e2a00640 1 --2- 172.21.15.12:0/3952752059 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f72a2232b0 0x55f72a22d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.620+0000 7f19e2000640 1 -- 172.21.15.12:0/3952752059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f72a228e10 msgr2=0x55f72a2349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.620+0000 7f19e2000640 1 --2- 172.21.15.12:0/3952752059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f72a228e10 0x55f72a2349f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.620+0000 7f19e2000640 1 -- 172.21.15.12:0/3952752059 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f72a2232b0 msgr2=0x55f72a22d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.620+0000 7f19e2000640 1 --2- 172.21.15.12:0/3952752059 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f72a2232b0 0x55f72a22d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.620+0000 7f19e2000640 1 -- 172.21.15.12:0/3952752059 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f729fc5f70 con 0x55f72a223ca0 2024-03-23T23:07:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.620+0000 7f19e3400640 1 --2- 172.21.15.12:0/3952752059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f72a228e10 0x55f72a2349f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:07:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.620+0000 7f19e2000640 1 --2- 172.21.15.12:0/3952752059 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a223ca0 0x55f72a22d9b0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f19d002c900 tx=0x7f19d002bc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=322468209fcf71a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.620+0000 7f19ccc00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:07:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.621+0000 7f19ccc00640 1 -- 172.21.15.12:0/3952752059 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d0048070 con 0x55f72a223ca0 2024-03-23T23:07:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.621+0000 7f19ccc00640 1 -- 172.21.15.12:0/3952752059 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f19d00050f0 con 0x55f72a223ca0 2024-03-23T23:07:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.621+0000 7f19ec576740 1 -- 172.21.15.12:0/3952752059 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f19d4009e30 con 0x55f72a223ca0 2024-03-23T23:07:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.621+0000 7f19ccc00640 1 -- 172.21.15.12:0/3952752059 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f19d00052f0 con 0x55f72a223ca0 2024-03-23T23:07:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.621+0000 7f19ec576740 10 client.26951 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.621+0000 7f19ec576740 1 -- 172.21.15.12:0/3952752059 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f729e5d170 con 0x55f72a223ca0 2024-03-23T23:07:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.621+0000 7f19ccc00640 1 -- 172.21.15.12:0/3952752059 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19d4009e30 con 0x55f72a223ca0 2024-03-23T23:07:33.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.622+0000 7f19ccc00640 1 -- 172.21.15.12:0/3952752059 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f19d004f500 con 0x55f72a223ca0 2024-03-23T23:07:33.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.622+0000 7f19ec576740 10 client.26951 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:33.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.622+0000 7f19ec576740 10 client.26951 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:07:33.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.622+0000 7f19ec576740 20 client.26951 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:33.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.622+0000 7f19ec576740 1 --2- 172.21.15.12:0/3952752059 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f72a239310 0x55f72a23b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.622+0000 7f19ec576740 4 client.26951 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:07:33.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.622+0000 7f19ec576740 1 -- 172.21.15.12:0/3952752059 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f72a1963f0 con 0x55f72a239310 2024-03-23T23:07:33.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.623+0000 7f19e2a00640 1 --2- 172.21.15.12:0/3952752059 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f72a239310 0x55f72a23b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.623+0000 7f19e2a00640 1 --2- 172.21.15.12:0/3952752059 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f72a239310 0x55f72a23b740 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f9ad62c9e5cbb25 server_cookie=cfe6f04745bb3c8 in_seq=0 out_seq=0 2024-03-23T23:07:33.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.623+0000 7f19ccc00640 10 client.26951 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:33.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.624+0000 7f19ccc00640 1 -- 172.21.15.12:0/3952752059 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f72a1963f0 con 0x55f72a239310 2024-03-23T23:07:33.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.624+0000 7f19ccc00640 10 client.26951 handle_command_reply: tid=0 2024-03-23T23:07:33.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.624+0000 7f19ec576740 1 client.26951 shutdown 2024-03-23T23:07:33.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.624+0000 7f19ec576740 1 -- 172.21.15.12:0/3952752059 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a223ca0 msgr2=0x55f72a22d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:33.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.624+0000 7f19ec576740 1 --2- 172.21.15.12:0/3952752059 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a223ca0 0x55f72a22d9b0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f19d002c900 tx=0x7f19d002bc50 comp rx=0 tx=0).stop 2024-03-23T23:07:33.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.624+0000 7f19ec576740 1 -- 172.21.15.12:0/3952752059 shutdown_connections 2024-03-23T23:07:33.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.624+0000 7f19ec576740 1 --2- 172.21.15.12:0/3952752059 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f72a239310 0x55f72a23b740 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.624+0000 7f19ec576740 1 --2- 172.21.15.12:0/3952752059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f72a228e10 0x55f72a2349f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.624+0000 7f19ec576740 1 --2- 172.21.15.12:0/3952752059 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a223ca0 0x55f72a22d9b0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.624+0000 7f19ec576740 1 --2- 172.21.15.12:0/3952752059 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f72a2232b0 0x55f72a22d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.624+0000 7f19ccc00640 0 client.26951 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:33.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.624+0000 7f19ec576740 1 -- 172.21.15.12:0/3952752059 >> 172.21.15.12:0/3952752059 conn(0x55f72a21e820 msgr2=0x55f72a21da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:33.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.625+0000 7f19ec576740 1 -- 172.21.15.12:0/3952752059 shutdown_connections 2024-03-23T23:07:33.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.625+0000 7f19ec576740 1 -- 172.21.15.12:0/3952752059 wait complete. 2024-03-23T23:07:33.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.625+0000 7f19ec576740 20 client.26951 trim_cache size 0 max 16384 2024-03-23T23:07:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.643+0000 7f19ec576740 1 Processor -- start 2024-03-23T23:07:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.643+0000 7f19ec576740 1 -- start start 2024-03-23T23:07:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.643+0000 7f19ec576740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f72a2232b0 0x55f72a21d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.643+0000 7f19ec576740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f72a223ca0 0x55f72a2349b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.643+0000 7f19ec576740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a228e10 0x55f72a240000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.643+0000 7f19ec576740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e41acfa0 con 0x55f72a223ca0 2024-03-23T23:07:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.643+0000 7f19ec576740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e41aaaa0 con 0x55f72a228e10 2024-03-23T23:07:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.643+0000 7f19ec576740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e41a8b40 con 0x55f72a2232b0 2024-03-23T23:07:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.643+0000 7f19e3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a228e10 0x55f72a240000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.643+0000 7f19e3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a228e10 0x55f72a240000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44546/0 (socket says 172.21.15.12:44546) 2024-03-23T23:07:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.643+0000 7f19e2000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f72a223ca0 0x55f72a2349b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.643+0000 7f19e3400640 1 -- 172.21.15.12:0/3840246466 learned_addr learned my addr 172.21.15.12:0/3840246466 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:33.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.644+0000 7f19e2a00640 1 --2- 172.21.15.12:0/3840246466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f72a2232b0 0x55f72a21d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.644+0000 7f19e3400640 1 -- 172.21.15.12:0/3840246466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f72a2232b0 msgr2=0x55f72a21d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:33.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.644+0000 7f19e3400640 1 --2- 172.21.15.12:0/3840246466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f72a2232b0 0x55f72a21d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.644+0000 7f19e3400640 1 -- 172.21.15.12:0/3840246466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f72a223ca0 msgr2=0x55f72a2349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:33.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.644+0000 7f19e3400640 1 --2- 172.21.15.12:0/3840246466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f72a223ca0 0x55f72a2349b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.644+0000 7f19e3400640 1 -- 172.21.15.12:0/3840246466 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f72a21f290 con 0x55f72a228e10 2024-03-23T23:07:33.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.644+0000 7f19e2a00640 1 --2- 172.21.15.12:0/3840246466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f72a2232b0 0x55f72a21d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:07:33.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.644+0000 7f19e3400640 1 --2- 172.21.15.12:0/3840246466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a228e10 0x55f72a240000 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f19dc098470 tx=0x7f19dc09b5e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1312502fe257ebe server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:33.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.644+0000 7f19cf400640 1 -- 172.21.15.12:0/3840246466 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19dc0b3070 con 0x55f72a228e10 2024-03-23T23:07:33.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.644+0000 7f19cf400640 1 -- 172.21.15.12:0/3840246466 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19dc09baa0 con 0x55f72a228e10 2024-03-23T23:07:33.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.644+0000 7f19cf400640 1 -- 172.21.15.12:0/3840246466 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19dc09aef0 con 0x55f72a228e10 2024-03-23T23:07:33.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.645+0000 7f19ec576740 1 -- 172.21.15.12:0/3840246466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a228e10 msgr2=0x55f72a240000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:33.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.645+0000 7f19ec576740 1 --2- 172.21.15.12:0/3840246466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a228e10 0x55f72a240000 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f19dc098470 tx=0x7f19dc09b5e0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.645+0000 7f19ec576740 1 -- 172.21.15.12:0/3840246466 shutdown_connections 2024-03-23T23:07:33.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.645+0000 7f19ec576740 1 --2- 172.21.15.12:0/3840246466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a228e10 0x55f72a240000 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.645+0000 7f19ec576740 1 --2- 172.21.15.12:0/3840246466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f72a223ca0 0x55f72a2349b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.645+0000 7f19ec576740 1 --2- 172.21.15.12:0/3840246466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f72a2232b0 0x55f72a21d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.645+0000 7f19ec576740 1 -- 172.21.15.12:0/3840246466 >> 172.21.15.12:0/3840246466 conn(0x55f72a21e820 msgr2=0x55f72a21d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:33.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.645+0000 7f19ec576740 1 -- 172.21.15.12:0/3840246466 shutdown_connections 2024-03-23T23:07:33.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.645+0000 7f19ec576740 1 -- 172.21.15.12:0/3840246466 wait complete. 2024-03-23T23:07:33.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.646+0000 7f19ec576740 1 -- start start 2024-03-23T23:07:33.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.646+0000 7f19ec576740 1 Processor -- start 2024-03-23T23:07:33.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.646+0000 7f19ec576740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f72a2232b0 0x55f72a234600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.646+0000 7f19ec576740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f72a223ca0 0x55f72a232cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.646+0000 7f19ec576740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a228e10 0x55f72a2331f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.646+0000 7f19ec576740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e41a9140 con 0x55f72a223ca0 2024-03-23T23:07:33.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.646+0000 7f19ec576740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e41aa440 con 0x55f72a228e10 2024-03-23T23:07:33.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.646+0000 7f19ec576740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e41a9780 con 0x55f72a2232b0 2024-03-23T23:07:33.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.646+0000 7f19ec576740 10 client.0 _ll_register_callbacks cb 0x7f19dc0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:33.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.646+0000 7f19e2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f72a2232b0 0x55f72a234600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.647+0000 7f19e2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f72a2232b0 0x55f72a234600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:46222/0 (socket says 172.21.15.12:46222) 2024-03-23T23:07:33.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.647+0000 7f19e2a00640 1 -- 172.21.15.12:0/4114982141 learned_addr learned my addr 172.21.15.12:0/4114982141 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:33.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.647+0000 7f19e3400640 1 --2- 172.21.15.12:0/4114982141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a228e10 0x55f72a2331f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.647+0000 7f19e2000640 1 --2- 172.21.15.12:0/4114982141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f72a223ca0 0x55f72a232cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.647+0000 7f19e2a00640 1 -- 172.21.15.12:0/4114982141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a228e10 msgr2=0x55f72a2331f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:33.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.647+0000 7f19e2a00640 1 --2- 172.21.15.12:0/4114982141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a228e10 0x55f72a2331f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.647+0000 7f19e2a00640 1 -- 172.21.15.12:0/4114982141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f72a223ca0 msgr2=0x55f72a232cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:33.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.647+0000 7f19e2a00640 1 --2- 172.21.15.12:0/4114982141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f72a223ca0 0x55f72a232cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.647+0000 7f19e2a00640 1 -- 172.21.15.12:0/4114982141 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f72a233730 con 0x55f72a2232b0 2024-03-23T23:07:33.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.647+0000 7f19e3400640 1 --2- 172.21.15.12:0/4114982141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a228e10 0x55f72a2331f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:07:33.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.647+0000 7f19e2a00640 1 --2- 172.21.15.12:0/4114982141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f72a2232b0 0x55f72a234600 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f19d40058c0 tx=0x7f19d4009400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e697c4ec51cc9df server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:33.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.647+0000 7f19ccc00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:07:33.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.647+0000 7f19ccc00640 1 -- 172.21.15.12:0/4114982141 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d401e040 con 0x55f72a2232b0 2024-03-23T23:07:33.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.647+0000 7f19ccc00640 1 -- 172.21.15.12:0/4114982141 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f19d40035f0 con 0x55f72a2232b0 2024-03-23T23:07:33.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.647+0000 7f19ec576740 1 -- 172.21.15.12:0/4114982141 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f19dc09baa0 con 0x55f72a2232b0 2024-03-23T23:07:33.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.648+0000 7f19ccc00640 1 -- 172.21.15.12:0/4114982141 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f19d4003830 con 0x55f72a2232b0 2024-03-23T23:07:33.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.648+0000 7f19ec576740 10 client.26926 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:33.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.648+0000 7f19ec576740 1 -- 172.21.15.12:0/4114982141 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f72a2959a0 con 0x55f72a2232b0 2024-03-23T23:07:33.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.648+0000 7f19ccc00640 1 -- 172.21.15.12:0/4114982141 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19dc09baa0 con 0x55f72a2232b0 2024-03-23T23:07:33.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.648+0000 7f19ccc00640 1 -- 172.21.15.12:0/4114982141 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f19d4037020 con 0x55f72a2232b0 2024-03-23T23:07:33.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.648+0000 7f19ec576740 10 client.26926 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:33.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.648+0000 7f19ec576740 10 client.26926 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:07:33.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.648+0000 7f19ec576740 20 client.26926 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:33.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.648+0000 7f19ec576740 1 --2- 172.21.15.12:0/4114982141 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f72a296190 0x55f72a2985b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.648+0000 7f19ec576740 4 client.26926 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-23T23:07:33.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.648+0000 7f19ec576740 1 -- 172.21.15.12:0/4114982141 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f72a190740 con 0x55f72a296190 2024-03-23T23:07:33.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.648+0000 7f19e2000640 1 --2- 172.21.15.12:0/4114982141 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f72a296190 0x55f72a2985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.649+0000 7f19e2000640 1 --2- 172.21.15.12:0/4114982141 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f72a296190 0x55f72a2985b0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4981a0da14c95f03 server_cookie=3ca7efcdf8427d10 in_seq=0 out_seq=0 2024-03-23T23:07:33.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.649+0000 7f19ccc00640 10 client.26926 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:33.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.650+0000 7f19ccc00640 1 -- 172.21.15.12:0/4114982141 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f72a190740 con 0x55f72a296190 2024-03-23T23:07:33.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.650+0000 7f19ccc00640 10 client.26926 handle_command_reply: tid=0 2024-03-23T23:07:33.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.650+0000 7f19ec576740 1 client.26926 shutdown 2024-03-23T23:07:33.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.650+0000 7f19ec576740 1 -- 172.21.15.12:0/4114982141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f72a2232b0 msgr2=0x55f72a234600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:33.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.650+0000 7f19ec576740 1 --2- 172.21.15.12:0/4114982141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f72a2232b0 0x55f72a234600 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f19d40058c0 tx=0x7f19d4009400 comp rx=0 tx=0).stop 2024-03-23T23:07:33.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.650+0000 7f19ec576740 1 -- 172.21.15.12:0/4114982141 shutdown_connections 2024-03-23T23:07:33.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.650+0000 7f19ec576740 1 --2- 172.21.15.12:0/4114982141 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f72a296190 0x55f72a2985b0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.650+0000 7f19ec576740 1 --2- 172.21.15.12:0/4114982141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72a228e10 0x55f72a2331f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.650+0000 7f19ccc00640 0 client.26926 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:33.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.650+0000 7f19ec576740 1 --2- 172.21.15.12:0/4114982141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f72a223ca0 0x55f72a232cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.650+0000 7f19ec576740 1 --2- 172.21.15.12:0/4114982141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f72a2232b0 0x55f72a234600 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.650+0000 7f19ec576740 1 -- 172.21.15.12:0/4114982141 >> 172.21.15.12:0/4114982141 conn(0x55f72a21e820 msgr2=0x55f72a21da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:33.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.651+0000 7f19ec576740 1 -- 172.21.15.12:0/4114982141 shutdown_connections 2024-03-23T23:07:33.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.651+0000 7f19ec576740 1 -- 172.21.15.12:0/4114982141 wait complete. 2024-03-23T23:07:33.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.651+0000 7f19ec576740 20 client.26926 trim_cache size 0 max 16384 2024-03-23T23:07:33.655 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:07:33.656 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-03-23T23:07:33.656 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "5a45dc25-c387-486c-aebb-3b235e572c5d", 2024-03-23T23:07:33.656 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-03-23T23:07:33.656 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:07:33.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.651+0000 7f19e3e00640 1 -- 172.21.15.12:0/1927166214 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f19c80a2880 msgr2=0x7f19c80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:33.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.651+0000 7f19e3e00640 1 --2- 172.21.15.12:0/1927166214 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f19c80a2880 0x7f19c80a4d40 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f19d002c260 tx=0x7f19d0005fd0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.651+0000 7f19e3e00640 1 -- 172.21.15.12:0/1927166214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19e4103130 msgr2=0x7f19e4073fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:33.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.651+0000 7f19e3e00640 1 --2- 172.21.15.12:0/1927166214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19e4103130 0x7f19e4073fd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f19dc00eb10 tx=0x7f19dc00c2d0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.652+0000 7f19e3e00640 1 -- 172.21.15.12:0/1927166214 shutdown_connections 2024-03-23T23:07:33.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.652+0000 7f19e3e00640 1 --2- 172.21.15.12:0/1927166214 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f19c80a2880 0x7f19c80a4d40 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.652+0000 7f19e3e00640 1 --2- 172.21.15.12:0/1927166214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19e4103130 0x7f19e4073fd0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.652+0000 7f19e3e00640 1 --2- 172.21.15.12:0/1927166214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19e4102770 0x7f19e4073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.652+0000 7f19e3e00640 1 --2- 172.21.15.12:0/1927166214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19e4101580 0x7f19e4075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.652+0000 7f19e3e00640 1 -- 172.21.15.12:0/1927166214 >> 172.21.15.12:0/1927166214 conn(0x7f19e40fcb10 msgr2=0x7f19e4105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:33.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.652+0000 7f19e3e00640 1 -- 172.21.15.12:0/1927166214 shutdown_connections 2024-03-23T23:07:33.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.652+0000 7f19e3e00640 1 -- 172.21.15.12:0/1927166214 wait complete. 2024-03-23T23:07:33.669 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:07:33.669 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:07:33.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.887+0000 7f2cc0400640 1 -- 172.21.15.12:0/1212949486 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cb81038c0 msgr2=0x7f2cb8103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:33.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.887+0000 7f2cc0400640 1 --2- 172.21.15.12:0/1212949486 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cb81038c0 0x7f2cb8103ce0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f2cac009900 tx=0x7f2cac02bc40 comp rx=0 tx=0).stop 2024-03-23T23:07:33.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.888+0000 7f2cc0400640 1 -- 172.21.15.12:0/1212949486 shutdown_connections 2024-03-23T23:07:33.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.888+0000 7f2cc0400640 1 --2- 172.21.15.12:0/1212949486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cb8105470 0x7f2cb810bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.888+0000 7f2cc0400640 1 --2- 172.21.15.12:0/1212949486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cb8104ab0 0x7f2cb8104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.888+0000 7f2cc0400640 1 --2- 172.21.15.12:0/1212949486 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cb81038c0 0x7f2cb8103ce0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.888+0000 7f2cc0400640 1 -- 172.21.15.12:0/1212949486 >> 172.21.15.12:0/1212949486 conn(0x7f2cb80fee50 msgr2=0x7f2cb8101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:33.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.888+0000 7f2cc0400640 1 -- 172.21.15.12:0/1212949486 shutdown_connections 2024-03-23T23:07:33.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.888+0000 7f2cc0400640 1 -- 172.21.15.12:0/1212949486 wait complete. 2024-03-23T23:07:33.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.889+0000 7f2cc0400640 1 Processor -- start 2024-03-23T23:07:33.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.889+0000 7f2cc0400640 1 -- start start 2024-03-23T23:07:33.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.889+0000 7f2cc0400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cb81038c0 0x7f2cb8199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.889+0000 7f2cc0400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cb8104ab0 0x7f2cb819a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.889+0000 7f2cc0400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cb8105470 0x7f2cb81a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.889+0000 7f2cc0400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb810f590 con 0x7f2cb81038c0 2024-03-23T23:07:33.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.889+0000 7f2cc0400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb810f410 con 0x7f2cb8104ab0 2024-03-23T23:07:33.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.889+0000 7f2cc0400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb810f710 con 0x7f2cb8105470 2024-03-23T23:07:33.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.890+0000 7f2cbd000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cb8104ab0 0x7f2cb819a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.890+0000 7f2cbda00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cb81038c0 0x7f2cb8199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.890+0000 7f2cbd000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cb8104ab0 0x7f2cb819a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:43610/0 (socket says 172.21.15.12:43610) 2024-03-23T23:07:33.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.890+0000 7f2cbd000640 1 -- 172.21.15.12:0/2389922612 learned_addr learned my addr 172.21.15.12:0/2389922612 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:33.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.890+0000 7f2cbe400640 1 --2- 172.21.15.12:0/2389922612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cb8105470 0x7f2cb81a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.890+0000 7f2cbd000640 1 -- 172.21.15.12:0/2389922612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cb8105470 msgr2=0x7f2cb81a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:33.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.890+0000 7f2cbd000640 1 --2- 172.21.15.12:0/2389922612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cb8105470 0x7f2cb81a1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.890+0000 7f2cbd000640 1 -- 172.21.15.12:0/2389922612 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cb81038c0 msgr2=0x7f2cb8199bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:33.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.890+0000 7f2cbd000640 1 --2- 172.21.15.12:0/2389922612 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cb81038c0 0x7f2cb8199bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.890+0000 7f2cbd000640 1 -- 172.21.15.12:0/2389922612 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb81a17d0 con 0x7f2cb8104ab0 2024-03-23T23:07:33.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.891+0000 7f2cbd000640 1 --2- 172.21.15.12:0/2389922612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cb8104ab0 0x7f2cb819a0f0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f2ca400e9f0 tx=0x7f2ca400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:33.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.891+0000 7f2cb6000640 1 -- 172.21.15.12:0/2389922612 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca400cd80 con 0x7f2cb8104ab0 2024-03-23T23:07:33.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.891+0000 7f2cb6000640 1 -- 172.21.15.12:0/2389922612 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ca4010910 con 0x7f2cb8104ab0 2024-03-23T23:07:33.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.891+0000 7f2cb6000640 1 -- 172.21.15.12:0/2389922612 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca4018990 con 0x7f2cb8104ab0 2024-03-23T23:07:33.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.891+0000 7f2cc0400640 1 -- 172.21.15.12:0/2389922612 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb81a1ac0 con 0x7f2cb8104ab0 2024-03-23T23:07:33.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.891+0000 7f2cc0400640 1 -- 172.21.15.12:0/2389922612 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb81a2010 con 0x7f2cb8104ab0 2024-03-23T23:07:33.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.894+0000 7f2cc3231740 1 Processor -- start 2024-03-23T23:07:33.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.895+0000 7f2cc3231740 1 -- start start 2024-03-23T23:07:33.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.895+0000 7f2cc3231740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639349b4e80 0x5639349b52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.895+0000 7f2cb6000640 1 -- 172.21.15.12:0/2389922612 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2ca4010ac0 con 0x7f2cb8104ab0 2024-03-23T23:07:33.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.895+0000 7f2cc3231740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639349b5870 0x5639349ba370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.895+0000 7f2cc3231740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639349ba9e0 0x5639349bce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.895+0000 7f2cc3231740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56393491a250 con 0x5639349b5870 2024-03-23T23:07:33.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.895+0000 7f2cc3231740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639348c2a30 con 0x5639349ba9e0 2024-03-23T23:07:33.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.895+0000 7f2cc3231740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639349162f0 con 0x5639349b4e80 2024-03-23T23:07:33.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.895+0000 7f2cbe400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639349b4e80 0x5639349b52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.895+0000 7f2cbda00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639349b5870 0x5639349ba370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.895+0000 7f2cbd000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639349ba9e0 0x5639349bce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.895+0000 7f2cbe400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639349b4e80 0x5639349b52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47900/0 (socket says 172.21.15.12:47900) 2024-03-23T23:07:33.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.895+0000 7f2cbe400640 1 -- 172.21.15.12:0/2650317691 learned_addr learned my addr 172.21.15.12:0/2650317691 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:33.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.896+0000 7f2cbe400640 1 -- 172.21.15.12:0/2650317691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639349ba9e0 msgr2=0x5639349bce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:33.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.896+0000 7f2cbe400640 1 --2- 172.21.15.12:0/2650317691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639349ba9e0 0x5639349bce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.896+0000 7f2cbe400640 1 -- 172.21.15.12:0/2650317691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639349b5870 msgr2=0x5639349ba370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:33.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.896+0000 7f2cbe400640 1 --2- 172.21.15.12:0/2650317691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639349b5870 0x5639349ba370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.896+0000 7f2cbe400640 1 -- 172.21.15.12:0/2650317691 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5639348ad690 con 0x5639349b4e80 2024-03-23T23:07:33.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.896+0000 7f2cbe400640 1 --2- 172.21.15.12:0/2650317691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639349b4e80 0x5639349b52a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0009fd0 tx=0x7f2cb00118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d77504a1293b4ea server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:33.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.896+0000 7f2caaa00640 1 -- 172.21.15.12:0/2650317691 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0004d20 con 0x5639349b4e80 2024-03-23T23:07:33.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.896+0000 7f2caaa00640 1 -- 172.21.15.12:0/2650317691 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2cb0004050 con 0x5639349b4e80 2024-03-23T23:07:33.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.896+0000 7f2caaa00640 1 -- 172.21.15.12:0/2650317691 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0005220 con 0x5639349b4e80 2024-03-23T23:07:33.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.897+0000 7f2cb6000640 1 --2- 172.21.15.12:0/2389922612 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ca00a2950 0x7f2ca00a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.897+0000 7f2cc3231740 1 -- 172.21.15.12:0/2650317691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639349b4e80 msgr2=0x5639349b52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:33.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.897+0000 7f2cc3231740 1 --2- 172.21.15.12:0/2650317691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639349b4e80 0x5639349b52a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0009fd0 tx=0x7f2cb00118d0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.897+0000 7f2cb6000640 1 -- 172.21.15.12:0/2389922612 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2ca40cecb0 con 0x7f2cb8104ab0 2024-03-23T23:07:33.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.897+0000 7f2cc3231740 1 -- 172.21.15.12:0/2650317691 shutdown_connections 2024-03-23T23:07:33.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.897+0000 7f2cc3231740 1 --2- 172.21.15.12:0/2650317691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639349ba9e0 0x5639349bce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.897+0000 7f2cc3231740 1 --2- 172.21.15.12:0/2650317691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639349b5870 0x5639349ba370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.897+0000 7f2cc3231740 1 --2- 172.21.15.12:0/2650317691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639349b4e80 0x5639349b52a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.897+0000 7f2cc3231740 1 -- 172.21.15.12:0/2650317691 >> 172.21.15.12:0/2650317691 conn(0x56393497d200 msgr2=0x5639349b07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:33.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.897+0000 7f2cbda00640 1 --2- 172.21.15.12:0/2389922612 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ca00a2950 0x7f2ca00a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.897+0000 7f2cc3231740 1 -- 172.21.15.12:0/2650317691 shutdown_connections 2024-03-23T23:07:33.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.897+0000 7f2cc3231740 1 -- 172.21.15.12:0/2650317691 wait complete. 2024-03-23T23:07:33.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.898+0000 7f2cc3231740 1 -- start start 2024-03-23T23:07:33.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.898+0000 7f2cbda00640 1 --2- 172.21.15.12:0/2389922612 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ca00a2950 0x7f2ca00a4e10 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2cac02c180 tx=0x7f2cac02cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:33.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.898+0000 7f2cc3231740 1 Processor -- start 2024-03-23T23:07:33.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.899+0000 7f2cc3231740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639349b4e80 0x5639349bedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.899+0000 7f2cc3231740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639349b5870 0x5639349bf300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.899+0000 7f2cc3231740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639349ba9e0 0x5639349c6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.899+0000 7f2cc3231740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639349bd990 con 0x5639349b5870 2024-03-23T23:07:33.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.899+0000 7f2cc3231740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639348d5e80 con 0x5639349ba9e0 2024-03-23T23:07:33.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.899+0000 7f2cc3231740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639349bdb10 con 0x5639349b4e80 2024-03-23T23:07:33.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.899+0000 7f2cc3231740 10 client.0 _ll_register_callbacks cb 0x5639348e8130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:33.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.899+0000 7f2cbda00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639349b4e80 0x5639349bedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.899+0000 7f2cbd000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639349b5870 0x5639349bf300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.899+0000 7f2cbda00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639349b4e80 0x5639349bedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47906/0 (socket says 172.21.15.12:47906) 2024-03-23T23:07:33.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.899+0000 7f2cbda00640 1 -- 172.21.15.12:0/1059078132 learned_addr learned my addr 172.21.15.12:0/1059078132 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:33.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.899+0000 7f2cbe400640 1 --2- 172.21.15.12:0/1059078132 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639349ba9e0 0x5639349c6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.899+0000 7f2cbd000640 1 -- 172.21.15.12:0/1059078132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639349b4e80 msgr2=0x5639349bedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:33.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.899+0000 7f2cbd000640 1 --2- 172.21.15.12:0/1059078132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639349b4e80 0x5639349bedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.899+0000 7f2cbd000640 1 -- 172.21.15.12:0/1059078132 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639349ba9e0 msgr2=0x5639349c6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:33.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.899+0000 7f2cbd000640 1 --2- 172.21.15.12:0/1059078132 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639349ba9e0 0x5639349c6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.899+0000 7f2cbd000640 1 -- 172.21.15.12:0/1059078132 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5639346f87d0 con 0x5639349b5870 2024-03-23T23:07:33.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.900+0000 7f2cbd000640 1 --2- 172.21.15.12:0/1059078132 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639349b5870 0x5639349bf300 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2ca400e9c0 tx=0x7f2ca4097310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a1f08cfee7b5892 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:33.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.900+0000 7f2c9f400640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:07:33.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.900+0000 7f2c9f400640 1 -- 172.21.15.12:0/1059078132 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca40d2020 con 0x5639349b5870 2024-03-23T23:07:33.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.900+0000 7f2c9f400640 1 -- 172.21.15.12:0/1059078132 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2ca4018d30 con 0x5639349b5870 2024-03-23T23:07:33.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.900+0000 7f2cc3231740 1 -- 172.21.15.12:0/1059078132 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2cb0004050 con 0x5639349b5870 2024-03-23T23:07:33.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.900+0000 7f2c9f400640 1 -- 172.21.15.12:0/1059078132 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ca40b4050 con 0x5639349b5870 2024-03-23T23:07:33.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.900+0000 7f2cc3231740 10 client.16845 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:33.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.900+0000 7f2cc3231740 1 -- 172.21.15.12:0/1059078132 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56393462e3f0 con 0x5639349b5870 2024-03-23T23:07:33.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.900+0000 7f2c9f400640 1 -- 172.21.15.12:0/1059078132 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2cb0004050 con 0x5639349b5870 2024-03-23T23:07:33.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.901+0000 7f2c9f400640 1 -- 172.21.15.12:0/1059078132 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f2ca40b4280 con 0x5639349b5870 2024-03-23T23:07:33.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.901+0000 7f2cc3231740 10 client.16845 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:33.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.901+0000 7f2cc3231740 10 client.16845 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:07:33.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.901+0000 7f2cc3231740 20 client.16845 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:33.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.901+0000 7f2cc3231740 1 --2- 172.21.15.12:0/1059078132 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5639349cab60 0x5639349ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.901+0000 7f2cc3231740 4 client.16845 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:07:33.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.901+0000 7f2cc3231740 1 -- 172.21.15.12:0/1059078132 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563934927790 con 0x5639349cab60 2024-03-23T23:07:33.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.901+0000 7f2cbda00640 1 --2- 172.21.15.12:0/1059078132 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5639349cab60 0x5639349ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.902+0000 7f2cbda00640 1 --2- 172.21.15.12:0/1059078132 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5639349cab60 0x5639349ccfe0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f0cbf8a1fb4b60b server_cookie=7a4758505ad27e71 in_seq=0 out_seq=0 2024-03-23T23:07:33.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.902+0000 7f2c9f400640 10 client.16845 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:33.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.902+0000 7f2c9f400640 1 -- 172.21.15.12:0/1059078132 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563934927790 con 0x5639349cab60 2024-03-23T23:07:33.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.902+0000 7f2c9f400640 10 client.16845 handle_command_reply: tid=0 2024-03-23T23:07:33.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.902+0000 7f2cc3231740 1 client.16845 shutdown 2024-03-23T23:07:33.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.903+0000 7f2cc3231740 1 -- 172.21.15.12:0/1059078132 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639349b5870 msgr2=0x5639349bf300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:33.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.903+0000 7f2cc3231740 1 --2- 172.21.15.12:0/1059078132 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639349b5870 0x5639349bf300 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2ca400e9c0 tx=0x7f2ca4097310 comp rx=0 tx=0).stop 2024-03-23T23:07:33.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.903+0000 7f2cc3231740 1 -- 172.21.15.12:0/1059078132 shutdown_connections 2024-03-23T23:07:33.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.903+0000 7f2cc3231740 1 --2- 172.21.15.12:0/1059078132 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5639349cab60 0x5639349ccfe0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.903+0000 7f2cc3231740 1 --2- 172.21.15.12:0/1059078132 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639349ba9e0 0x5639349c6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.903+0000 7f2cc3231740 1 --2- 172.21.15.12:0/1059078132 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639349b5870 0x5639349bf300 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.903+0000 7f2cc3231740 1 --2- 172.21.15.12:0/1059078132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639349b4e80 0x5639349bedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.903+0000 7f2c9f400640 0 client.16845 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:33.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.903+0000 7f2cc3231740 1 -- 172.21.15.12:0/1059078132 >> 172.21.15.12:0/1059078132 conn(0x56393497d200 msgr2=0x5639349b1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:33.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.903+0000 7f2cc3231740 1 -- 172.21.15.12:0/1059078132 shutdown_connections 2024-03-23T23:07:33.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.903+0000 7f2cc3231740 1 -- 172.21.15.12:0/1059078132 wait complete. 2024-03-23T23:07:33.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.903+0000 7f2cc3231740 20 client.16845 trim_cache size 0 max 16384 2024-03-23T23:07:33.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.921+0000 7f2cc3231740 1 Processor -- start 2024-03-23T23:07:33.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.921+0000 7f2cc3231740 1 -- start start 2024-03-23T23:07:33.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.921+0000 7f2cc3231740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639349b4e80 0x5639349af910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.921+0000 7f2cc3231740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639349b5870 0x5639349c6300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.921+0000 7f2cc3231740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639349ba9e0 0x5639349b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.921+0000 7f2cc3231740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb81ad190 con 0x5639349b5870 2024-03-23T23:07:33.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.921+0000 7f2cc3231740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb81aac90 con 0x5639349ba9e0 2024-03-23T23:07:33.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.921+0000 7f2cc3231740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb81a8d30 con 0x5639349b4e80 2024-03-23T23:07:33.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.921+0000 7f2cbe400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639349ba9e0 0x5639349b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.921+0000 7f2cbda00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639349b4e80 0x5639349af910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.921+0000 7f2cbe400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639349ba9e0 0x5639349b3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:43638/0 (socket says 172.21.15.12:43638) 2024-03-23T23:07:33.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.921+0000 7f2cbe400640 1 -- 172.21.15.12:0/3712657964 learned_addr learned my addr 172.21.15.12:0/3712657964 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:33.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.922+0000 7f2cbd000640 1 --2- 172.21.15.12:0/3712657964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639349b5870 0x5639349c6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.922+0000 7f2cbda00640 1 -- 172.21.15.12:0/3712657964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639349ba9e0 msgr2=0x5639349b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:33.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.922+0000 7f2cbda00640 1 --2- 172.21.15.12:0/3712657964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639349ba9e0 0x5639349b3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.922+0000 7f2cbda00640 1 -- 172.21.15.12:0/3712657964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639349b5870 msgr2=0x5639349c6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:33.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.922+0000 7f2cbda00640 1 --2- 172.21.15.12:0/3712657964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639349b5870 0x5639349c6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.922+0000 7f2cbda00640 1 -- 172.21.15.12:0/3712657964 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5639345ee0f0 con 0x5639349b4e80 2024-03-23T23:07:33.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.922+0000 7f2cbe400640 1 --2- 172.21.15.12:0/3712657964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639349ba9e0 0x5639349b3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:07:33.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.922+0000 7f2cbda00640 1 --2- 172.21.15.12:0/3712657964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639349b4e80 0x5639349af910 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x5639349cd820 tx=0x7f2cac040040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7873c4108ecce211 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:33.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.923+0000 7f2caa000640 1 -- 172.21.15.12:0/3712657964 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cac053020 con 0x5639349b4e80 2024-03-23T23:07:33.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.923+0000 7f2caa000640 1 -- 172.21.15.12:0/3712657964 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2cac0531c0 con 0x5639349b4e80 2024-03-23T23:07:33.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.923+0000 7f2caa000640 1 -- 172.21.15.12:0/3712657964 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cac034860 con 0x5639349b4e80 2024-03-23T23:07:33.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.923+0000 7f2cc3231740 1 -- 172.21.15.12:0/3712657964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639349b4e80 msgr2=0x5639349af910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:33.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.923+0000 7f2cc3231740 1 --2- 172.21.15.12:0/3712657964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639349b4e80 0x5639349af910 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x5639349cd820 tx=0x7f2cac040040 comp rx=0 tx=0).stop 2024-03-23T23:07:33.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.923+0000 7f2cc3231740 1 -- 172.21.15.12:0/3712657964 shutdown_connections 2024-03-23T23:07:33.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.923+0000 7f2cc3231740 1 --2- 172.21.15.12:0/3712657964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639349ba9e0 0x5639349b3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.923+0000 7f2cc3231740 1 --2- 172.21.15.12:0/3712657964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639349b5870 0x5639349c6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.923+0000 7f2cc3231740 1 --2- 172.21.15.12:0/3712657964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639349b4e80 0x5639349af910 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.923+0000 7f2cc3231740 1 -- 172.21.15.12:0/3712657964 >> 172.21.15.12:0/3712657964 conn(0x56393497d200 msgr2=0x5639349af200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:33.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.923+0000 7f2cc3231740 1 -- 172.21.15.12:0/3712657964 shutdown_connections 2024-03-23T23:07:33.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.923+0000 7f2cc3231740 1 -- 172.21.15.12:0/3712657964 wait complete. 2024-03-23T23:07:33.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.924+0000 7f2cc3231740 1 -- start start 2024-03-23T23:07:33.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.924+0000 7f2cc3231740 1 Processor -- start 2024-03-23T23:07:33.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.924+0000 7f2cc3231740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639349b4e80 0x5639349c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.924+0000 7f2cc3231740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639349b5870 0x5639349c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.924+0000 7f2cc3231740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639349ba9e0 0x5639349c4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.924+0000 7f2cc3231740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb81a9330 con 0x5639349ba9e0 2024-03-23T23:07:33.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.924+0000 7f2cc3231740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb81aa630 con 0x5639349b5870 2024-03-23T23:07:33.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.924+0000 7f2cc3231740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb81a9970 con 0x5639349b4e80 2024-03-23T23:07:33.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.924+0000 7f2cc3231740 10 client.0 _ll_register_callbacks cb 0x7f2ca40d23e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:33.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.925+0000 7f2cbd000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639349b5870 0x5639349c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.925+0000 7f2cbd000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639349b5870 0x5639349c60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:43644/0 (socket says 172.21.15.12:43644) 2024-03-23T23:07:33.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.925+0000 7f2cbd000640 1 -- 172.21.15.12:0/1619409348 learned_addr learned my addr 172.21.15.12:0/1619409348 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:33.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.925+0000 7f2cbda00640 1 --2- 172.21.15.12:0/1619409348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639349b4e80 0x5639349c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.925+0000 7f2cbe400640 1 --2- 172.21.15.12:0/1619409348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639349ba9e0 0x5639349c4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.925+0000 7f2cbd000640 1 -- 172.21.15.12:0/1619409348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639349b4e80 msgr2=0x5639349c5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:33.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.925+0000 7f2cbd000640 1 --2- 172.21.15.12:0/1619409348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639349b4e80 0x5639349c5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.925+0000 7f2cbd000640 1 -- 172.21.15.12:0/1619409348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639349ba9e0 msgr2=0x5639349c4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:33.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.925+0000 7f2cbd000640 1 --2- 172.21.15.12:0/1619409348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639349ba9e0 0x5639349c4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.925+0000 7f2cbd000640 1 -- 172.21.15.12:0/1619409348 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5639349c4870 con 0x5639349b5870 2024-03-23T23:07:33.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.925+0000 7f2cbe400640 1 --2- 172.21.15.12:0/1619409348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639349ba9e0 0x5639349c4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:07:33.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.925+0000 7f2cbda00640 1 --2- 172.21.15.12:0/1619409348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639349b4e80 0x5639349c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:07:33.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.925+0000 7f2cbd000640 1 --2- 172.21.15.12:0/1619409348 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639349b5870 0x5639349c60b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f2ca4018460 tx=0x7f2ca409aba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=365908cf45434e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:33.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.925+0000 7f2c9f400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:07:33.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.926+0000 7f2c9f400640 1 -- 172.21.15.12:0/1619409348 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca409b480 con 0x5639349b5870 2024-03-23T23:07:33.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.926+0000 7f2c9f400640 1 -- 172.21.15.12:0/1619409348 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2ca4010630 con 0x5639349b5870 2024-03-23T23:07:33.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.926+0000 7f2cc3231740 1 -- 172.21.15.12:0/1619409348 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2cac0531c0 con 0x5639349b5870 2024-03-23T23:07:33.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.926+0000 7f2c9f400640 1 -- 172.21.15.12:0/1619409348 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ca409b6b0 con 0x5639349b5870 2024-03-23T23:07:33.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.926+0000 7f2cc3231740 10 client.26975 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:33.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.926+0000 7f2cc3231740 1 -- 172.21.15.12:0/1619409348 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563934a26eb0 con 0x5639349b5870 2024-03-23T23:07:33.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.926+0000 7f2c9f400640 1 -- 172.21.15.12:0/1619409348 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2cac0531c0 con 0x5639349b5870 2024-03-23T23:07:33.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.926+0000 7f2c9f400640 1 -- 172.21.15.12:0/1619409348 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f2ca409a700 con 0x5639349b5870 2024-03-23T23:07:33.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.926+0000 7f2cc3231740 10 client.26975 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:33.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.926+0000 7f2cc3231740 10 client.26975 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:07:33.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.926+0000 7f2cc3231740 20 client.26975 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:33.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.926+0000 7f2cc3231740 1 --2- 172.21.15.12:0/1619409348 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563934a27640 0x563934a29a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:33.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.926+0000 7f2cc3231740 4 client.26975 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:07:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.926+0000 7f2cc3231740 1 -- 172.21.15.12:0/1619409348 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563934921ae0 con 0x563934a27640 2024-03-23T23:07:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.927+0000 7f2cbda00640 1 --2- 172.21.15.12:0/1619409348 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563934a27640 0x563934a29a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.927+0000 7f2cbda00640 1 --2- 172.21.15.12:0/1619409348 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563934a27640 0x563934a29a60 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad858615fc38c38a server_cookie=b58b9bb3eb0ae571 in_seq=0 out_seq=0 2024-03-23T23:07:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.927+0000 7f2c9f400640 10 client.26975 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.927+0000 7f2c9f400640 1 -- 172.21.15.12:0/1619409348 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x563934921ae0 con 0x563934a27640 2024-03-23T23:07:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.927+0000 7f2c9f400640 10 client.26975 handle_command_reply: tid=0 2024-03-23T23:07:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.927+0000 7f2cc3231740 1 client.26975 shutdown 2024-03-23T23:07:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.927+0000 7f2cc3231740 1 -- 172.21.15.12:0/1619409348 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639349b5870 msgr2=0x5639349c60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.928+0000 7f2cc3231740 1 --2- 172.21.15.12:0/1619409348 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639349b5870 0x5639349c60b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f2ca4018460 tx=0x7f2ca409aba0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.928+0000 7f2cc3231740 1 -- 172.21.15.12:0/1619409348 shutdown_connections 2024-03-23T23:07:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.928+0000 7f2cc3231740 1 --2- 172.21.15.12:0/1619409348 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563934a27640 0x563934a29a60 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.928+0000 7f2cc3231740 1 --2- 172.21.15.12:0/1619409348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5639349ba9e0 0x5639349c4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.928+0000 7f2cc3231740 1 --2- 172.21.15.12:0/1619409348 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639349b5870 0x5639349c60b0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.928+0000 7f2cc3231740 1 --2- 172.21.15.12:0/1619409348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639349b4e80 0x5639349c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.928+0000 7f2c9f400640 0 client.26975 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.928+0000 7f2cc3231740 1 -- 172.21.15.12:0/1619409348 >> 172.21.15.12:0/1619409348 conn(0x56393497d200 msgr2=0x5639349b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:33.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.928+0000 7f2cc3231740 1 -- 172.21.15.12:0/1619409348 shutdown_connections 2024-03-23T23:07:33.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.928+0000 7f2cc3231740 1 -- 172.21.15.12:0/1619409348 wait complete. 2024-03-23T23:07:33.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.928+0000 7f2cc3231740 20 client.26975 trim_cache size 0 max 16384 2024-03-23T23:07:33.932 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:07:33.932 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T23:07:33.932 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-03-23T23:07:33.932 INFO:teuthology.orchestra.run.smithi012.stdout: "5a45dc25-c387-486c-aebb-3b235e572c5d": { 2024-03-23T23:07:33.932 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-03-23T23:07:33.932 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "5a45dc25-c387-486c-aebb-3b235e572c5d", 2024-03-23T23:07:33.932 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-03-23T23:07:33.932 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:07:33.932 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:07:33.933 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:07:33.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.929+0000 7f2cc0400640 1 -- 172.21.15.12:0/2389922612 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ca00a2950 msgr2=0x7f2ca00a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:33.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.929+0000 7f2cc0400640 1 --2- 172.21.15.12:0/2389922612 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ca00a2950 0x7f2ca00a4e10 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2cac02c180 tx=0x7f2cac02cf90 comp rx=0 tx=0).stop 2024-03-23T23:07:33.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.929+0000 7f2cc0400640 1 -- 172.21.15.12:0/2389922612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cb8104ab0 msgr2=0x7f2cb819a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:33.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.929+0000 7f2cc0400640 1 --2- 172.21.15.12:0/2389922612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cb8104ab0 0x7f2cb819a0f0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f2ca400e9f0 tx=0x7f2ca400eeb0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.929+0000 7f2cc0400640 1 -- 172.21.15.12:0/2389922612 shutdown_connections 2024-03-23T23:07:33.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.929+0000 7f2cc0400640 1 --2- 172.21.15.12:0/2389922612 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ca00a2950 0x7f2ca00a4e10 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.929+0000 7f2cc0400640 1 --2- 172.21.15.12:0/2389922612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cb8105470 0x7f2cb81a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.929+0000 7f2cc0400640 1 --2- 172.21.15.12:0/2389922612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cb8104ab0 0x7f2cb819a0f0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.929+0000 7f2cc0400640 1 --2- 172.21.15.12:0/2389922612 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cb81038c0 0x7f2cb8199bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:33.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.929+0000 7f2cc0400640 1 -- 172.21.15.12:0/2389922612 >> 172.21.15.12:0/2389922612 conn(0x7f2cb80fee50 msgr2=0x7f2cb8107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:33.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.929+0000 7f2cc0400640 1 -- 172.21.15.12:0/2389922612 shutdown_connections 2024-03-23T23:07:33.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:07:33.929+0000 7f2cc0400640 1 -- 172.21.15.12:0/2389922612 wait complete. 2024-03-23T23:07:33.947 INFO:tasks.cephfs.filesystem:scrub status for tag:5a45dc25-c387-486c-aebb-3b235e572c5d - {'path': '/', 'tag': '5a45dc25-c387-486c-aebb-3b235e572c5d', 'options': 'recursive,force'} 2024-03-23T23:07:34.958 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:34 smithi203 ceph-mon[36693]: pgmap v563: 97 pgs: 97 active+clean; 3.6 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-03-23T23:07:34.958 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:34 smithi203 ceph-mon[36693]: scrub queued for path: / 2024-03-23T23:07:34.959 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:34 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:07:34.959 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:34 smithi203 ceph-mon[36693]: scrub summary: active paths [/] 2024-03-23T23:07:34.959 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:34 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:07:35.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:34 smithi012 ceph-mon[31695]: pgmap v563: 97 pgs: 97 active+clean; 3.6 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-03-23T23:07:35.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:34 smithi012 ceph-mon[31695]: scrub queued for path: / 2024-03-23T23:07:35.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:34 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:07:35.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:34 smithi012 ceph-mon[31695]: scrub summary: active paths [/] 2024-03-23T23:07:35.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:34 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:07:35.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:34 smithi173 ceph-mon[36116]: pgmap v563: 97 pgs: 97 active+clean; 3.6 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-03-23T23:07:35.129 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:34 smithi173 ceph-mon[36116]: scrub queued for path: / 2024-03-23T23:07:35.129 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:34 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:07:35.129 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:34 smithi173 ceph-mon[36116]: scrub summary: active paths [/] 2024-03-23T23:07:35.129 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:34 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:07:37.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:36 smithi012 ceph-mon[31695]: pgmap v564: 97 pgs: 97 active+clean; 3.6 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-03-23T23:07:37.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:36 smithi012 ceph-mon[31695]: scrub summary: idle 2024-03-23T23:07:37.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:36 smithi173 ceph-mon[36116]: pgmap v564: 97 pgs: 97 active+clean; 3.6 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-03-23T23:07:37.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:36 smithi173 ceph-mon[36116]: scrub summary: idle 2024-03-23T23:07:37.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:36 smithi203 ceph-mon[36693]: pgmap v564: 97 pgs: 97 active+clean; 3.6 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-03-23T23:07:37.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:36 smithi203 ceph-mon[36693]: scrub summary: idle 2024-03-23T23:07:39.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:38 smithi012 ceph-mon[31695]: pgmap v565: 97 pgs: 97 active+clean; 3.6 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:07:39.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:38 smithi173 ceph-mon[36116]: pgmap v565: 97 pgs: 97 active+clean; 3.6 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:07:39.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:38 smithi203 ceph-mon[36693]: pgmap v565: 97 pgs: 97 active+clean; 3.6 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:07:41.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:40 smithi012 ceph-mon[31695]: pgmap v566: 97 pgs: 97 active+clean; 3.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 18 op/s 2024-03-23T23:07:41.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:40 smithi173 ceph-mon[36116]: pgmap v566: 97 pgs: 97 active+clean; 3.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 18 op/s 2024-03-23T23:07:41.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:40 smithi203 ceph-mon[36693]: pgmap v566: 97 pgs: 97 active+clean; 3.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 18 op/s 2024-03-23T23:07:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.554+0000 7f2d3e800640 1 -- 172.21.15.12:0/3605082963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d38107d00 msgr2=0x7f2d38108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.554+0000 7f2d3e800640 1 --2- 172.21.15.12:0/3605082963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d38107d00 0x7f2d38108120 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f2d28009900 tx=0x7f2d2802bc40 comp rx=0 tx=0).stop 2024-03-23T23:07:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.554+0000 7f2d3e800640 1 -- 172.21.15.12:0/3605082963 shutdown_connections 2024-03-23T23:07:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.555+0000 7f2d3e800640 1 --2- 172.21.15.12:0/3605082963 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2d381098b0 0x7f2d38110140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.555+0000 7f2d3e800640 1 --2- 172.21.15.12:0/3605082963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d38108ef0 0x7f2d38109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.555+0000 7f2d3e800640 1 --2- 172.21.15.12:0/3605082963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d38107d00 0x7f2d38108120 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.555+0000 7f2d3e800640 1 -- 172.21.15.12:0/3605082963 >> 172.21.15.12:0/3605082963 conn(0x7f2d38076cb0 msgr2=0x7f2d380790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.555+0000 7f2d3e800640 1 -- 172.21.15.12:0/3605082963 shutdown_connections 2024-03-23T23:07:41.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.555+0000 7f2d3e800640 1 -- 172.21.15.12:0/3605082963 wait complete. 2024-03-23T23:07:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.556+0000 7f2d3e800640 1 Processor -- start 2024-03-23T23:07:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.556+0000 7f2d3e800640 1 -- start start 2024-03-23T23:07:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.556+0000 7f2d3e800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d38107d00 0x7f2d3819dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.556+0000 7f2d3e800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d38108ef0 0x7f2d3819e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.556+0000 7f2d3e800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2d381098b0 0x7f2d381a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.556+0000 7f2d3e800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d381139a0 con 0x7f2d38107d00 2024-03-23T23:07:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.556+0000 7f2d3e800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d38113820 con 0x7f2d381098b0 2024-03-23T23:07:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.556+0000 7f2d3e800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d38113b20 con 0x7f2d38108ef0 2024-03-23T23:07:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.557+0000 7f2d36a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d38108ef0 0x7f2d3819e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.557+0000 7f2d37400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d38107d00 0x7f2d3819dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.557+0000 7f2d36a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d38108ef0 0x7f2d3819e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47940/0 (socket says 172.21.15.12:47940) 2024-03-23T23:07:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.557+0000 7f2d36a00640 1 -- 172.21.15.12:0/1139506548 learned_addr learned my addr 172.21.15.12:0/1139506548 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.557+0000 7f2d37e00640 1 --2- 172.21.15.12:0/1139506548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2d381098b0 0x7f2d381a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.557+0000 7f2d36a00640 1 -- 172.21.15.12:0/1139506548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2d381098b0 msgr2=0x7f2d381a5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.557+0000 7f2d36a00640 1 --2- 172.21.15.12:0/1139506548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2d381098b0 0x7f2d381a5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.557+0000 7f2d36a00640 1 -- 172.21.15.12:0/1139506548 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d38107d00 msgr2=0x7f2d3819dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.557+0000 7f2d36a00640 1 --2- 172.21.15.12:0/1139506548 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d38107d00 0x7f2d3819dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.557+0000 7f2d36a00640 1 -- 172.21.15.12:0/1139506548 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d381a5bf0 con 0x7f2d38108ef0 2024-03-23T23:07:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.557+0000 7f2d37400640 1 --2- 172.21.15.12:0/1139506548 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d38107d00 0x7f2d3819dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:07:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.558+0000 7f2d36a00640 1 --2- 172.21.15.12:0/1139506548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d38108ef0 0x7f2d3819e510 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2d2400ea60 tx=0x7f2d2400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.558+0000 7f2d2fe00640 1 -- 172.21.15.12:0/1139506548 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2400cde0 con 0x7f2d38108ef0 2024-03-23T23:07:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.558+0000 7f2d2fe00640 1 -- 172.21.15.12:0/1139506548 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d240035f0 con 0x7f2d38108ef0 2024-03-23T23:07:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.558+0000 7f2d3e800640 1 -- 172.21.15.12:0/1139506548 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d381a5ee0 con 0x7f2d38108ef0 2024-03-23T23:07:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.558+0000 7f2d2fe00640 1 -- 172.21.15.12:0/1139506548 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d24010670 con 0x7f2d38108ef0 2024-03-23T23:07:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.558+0000 7f2d3e800640 1 -- 172.21.15.12:0/1139506548 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d381a63f0 con 0x7f2d38108ef0 2024-03-23T23:07:41.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.561+0000 7f2d415b0740 1 Processor -- start 2024-03-23T23:07:41.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.561+0000 7f2d415b0740 1 -- start start 2024-03-23T23:07:41.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.562+0000 7f2d415b0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562afa5bf1e0 0x562afa5bf600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:41.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.562+0000 7f2d415b0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bfbd0 0x562afa5c46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.562+0000 7f2d415b0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562afa5c4d40 0x562afa5c71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.562+0000 7f2d415b0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x562afa5823b0 con 0x562afa5bf1e0 2024-03-23T23:07:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.562+0000 7f2d415b0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x562afa524830 con 0x562afa5c4d40 2024-03-23T23:07:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.562+0000 7f2d415b0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x562afa4cd5d0 con 0x562afa5bfbd0 2024-03-23T23:07:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.562+0000 7f2d37e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562afa5bf1e0 0x562afa5bf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.562+0000 7f2d37400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bfbd0 0x562afa5c46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.562+0000 7f2d37e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562afa5bf1e0 0x562afa5bf600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51028/0 (socket says 172.21.15.12:51028) 2024-03-23T23:07:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.562+0000 7f2d37400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bfbd0 0x562afa5c46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47942/0 (socket says 172.21.15.12:47942) 2024-03-23T23:07:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.562+0000 7f2d37e00640 1 -- 172.21.15.12:0/86157241 learned_addr learned my addr 172.21.15.12:0/86157241 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.562+0000 7f2d37400640 1 -- 172.21.15.12:0/86157241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562afa5c4d40 msgr2=0x562afa5c71d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T23:07:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.562+0000 7f2d37400640 1 --2- 172.21.15.12:0/86157241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562afa5c4d40 0x562afa5c71d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.563+0000 7f2d37400640 1 -- 172.21.15.12:0/86157241 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562afa5bf1e0 msgr2=0x562afa5bf600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.563+0000 7f2d37400640 1 --2- 172.21.15.12:0/86157241 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562afa5bf1e0 0x562afa5bf600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.563+0000 7f2d37400640 1 -- 172.21.15.12:0/86157241 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562afa4e52c0 con 0x562afa5bfbd0 2024-03-23T23:07:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.563+0000 7f2d2fe00640 1 -- 172.21.15.12:0/1139506548 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2d240037a0 con 0x7f2d38108ef0 2024-03-23T23:07:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.563+0000 7f2d37e00640 1 --2- 172.21.15.12:0/86157241 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562afa5bf1e0 0x562afa5bf600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:07:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.563+0000 7f2d37400640 1 --2- 172.21.15.12:0/86157241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bfbd0 0x562afa5c46d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2d28009230 tx=0x7f2d28000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16bd562cf5d448fc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.564+0000 7f2d2fe00640 1 --2- 172.21.15.12:0/1139506548 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2d1c0a2920 0x7f2d1c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.564+0000 7f2d2fe00640 1 -- 172.21.15.12:0/1139506548 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2d240cecf0 con 0x7f2d38108ef0 2024-03-23T23:07:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.564+0000 7f2d37e00640 1 --2- 172.21.15.12:0/1139506548 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2d1c0a2920 0x7f2d1c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.565+0000 7f2d2cc00640 1 -- 172.21.15.12:0/86157241 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d28042070 con 0x562afa5bfbd0 2024-03-23T23:07:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.565+0000 7f2d2cc00640 1 -- 172.21.15.12:0/86157241 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d2802cd40 con 0x562afa5bfbd0 2024-03-23T23:07:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.565+0000 7f2d2cc00640 1 -- 172.21.15.12:0/86157241 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d280052a0 con 0x562afa5bfbd0 2024-03-23T23:07:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.565+0000 7f2d37e00640 1 --2- 172.21.15.12:0/1139506548 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2d1c0a2920 0x7f2d1c0a4de0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2d30009fd0 tx=0x7f2d30009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.565+0000 7f2d415b0740 1 -- 172.21.15.12:0/86157241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bfbd0 msgr2=0x562afa5c46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.565+0000 7f2d415b0740 1 --2- 172.21.15.12:0/86157241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bfbd0 0x562afa5c46d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2d28009230 tx=0x7f2d28000bc0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.565+0000 7f2d415b0740 1 -- 172.21.15.12:0/86157241 shutdown_connections 2024-03-23T23:07:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.565+0000 7f2d415b0740 1 --2- 172.21.15.12:0/86157241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562afa5c4d40 0x562afa5c71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.565+0000 7f2d415b0740 1 --2- 172.21.15.12:0/86157241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bfbd0 0x562afa5c46d0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.565+0000 7f2d415b0740 1 --2- 172.21.15.12:0/86157241 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562afa5bf1e0 0x562afa5bf600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.565+0000 7f2d415b0740 1 -- 172.21.15.12:0/86157241 >> 172.21.15.12:0/86157241 conn(0x562afa5ba750 msgr2=0x562afa5bab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.565+0000 7f2d415b0740 1 -- 172.21.15.12:0/86157241 shutdown_connections 2024-03-23T23:07:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.566+0000 7f2d415b0740 1 -- 172.21.15.12:0/86157241 wait complete. 2024-03-23T23:07:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.566+0000 7f2d415b0740 1 -- start start 2024-03-23T23:07:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.566+0000 7f2d415b0740 1 Processor -- start 2024-03-23T23:07:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.567+0000 7f2d415b0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bf1e0 0x562afa5c9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.567+0000 7f2d415b0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562afa5bfbd0 0x562afa5c9970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.567+0000 7f2d415b0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562afa5c4d40 0x562afa5d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.567+0000 7f2d415b0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x562afa4e0a20 con 0x562afa5bfbd0 2024-03-23T23:07:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.567+0000 7f2d415b0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x562afa5208d0 con 0x562afa5c4d40 2024-03-23T23:07:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.567+0000 7f2d415b0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x562afa5c7c90 con 0x562afa5bf1e0 2024-03-23T23:07:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.567+0000 7f2d415b0740 10 client.0 _ll_register_callbacks cb 0x562afa51d1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.567+0000 7f2d37400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bf1e0 0x562afa5c9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.567+0000 7f2d37400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bf1e0 0x562afa5c9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47950/0 (socket says 172.21.15.12:47950) 2024-03-23T23:07:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.567+0000 7f2d37400640 1 -- 172.21.15.12:0/2720358460 learned_addr learned my addr 172.21.15.12:0/2720358460 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.567+0000 7f2d36a00640 1 --2- 172.21.15.12:0/2720358460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562afa5bfbd0 0x562afa5c9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.567+0000 7f2d37e00640 1 --2- 172.21.15.12:0/2720358460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562afa5c4d40 0x562afa5d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.567+0000 7f2d37400640 1 -- 172.21.15.12:0/2720358460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562afa5c4d40 msgr2=0x562afa5d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.567+0000 7f2d37400640 1 --2- 172.21.15.12:0/2720358460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562afa5c4d40 0x562afa5d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.567+0000 7f2d37400640 1 -- 172.21.15.12:0/2720358460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562afa5bfbd0 msgr2=0x562afa5c9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.567+0000 7f2d37400640 1 --2- 172.21.15.12:0/2720358460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562afa5bfbd0 0x562afa5c9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.568+0000 7f2d37400640 1 -- 172.21.15.12:0/2720358460 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562afa239210 con 0x562afa5bf1e0 2024-03-23T23:07:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.568+0000 7f2d37400640 1 --2- 172.21.15.12:0/2720358460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bf1e0 0x562afa5c9430 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2d2802c180 tx=0x7f2d28004cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=359b523df9fcbd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.568+0000 7f2d21600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:07:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.568+0000 7f2d21600640 1 -- 172.21.15.12:0/2720358460 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2804d070 con 0x562afa5bf1e0 2024-03-23T23:07:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.568+0000 7f2d21600640 1 -- 172.21.15.12:0/2720358460 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2d28005430 con 0x562afa5bf1e0 2024-03-23T23:07:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.568+0000 7f2d415b0740 1 -- 172.21.15.12:0/2720358460 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2d2802cd40 con 0x562afa5bf1e0 2024-03-23T23:07:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.569+0000 7f2d21600640 1 -- 172.21.15.12:0/2720358460 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d28042050 con 0x562afa5bf1e0 2024-03-23T23:07:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.569+0000 7f2d415b0740 10 client.26956 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.569+0000 7f2d415b0740 1 -- 172.21.15.12:0/2720358460 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x562afa305620 con 0x562afa5bf1e0 2024-03-23T23:07:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.569+0000 7f2d21600640 1 -- 172.21.15.12:0/2720358460 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d2802cd40 con 0x562afa5bf1e0 2024-03-23T23:07:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.569+0000 7f2d21600640 1 -- 172.21.15.12:0/2720358460 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f2d28034440 con 0x562afa5bf1e0 2024-03-23T23:07:41.571 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.569+0000 7f2d415b0740 10 client.26956 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:41.571 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.569+0000 7f2d415b0740 10 client.26956 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:07:41.571 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.569+0000 7f2d415b0740 20 client.26956 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:41.571 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.569+0000 7f2d415b0740 1 --2- 172.21.15.12:0/2720358460 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562afa5d4ea0 0x562afa5d7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:41.571 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.569+0000 7f2d415b0740 4 client.26956 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:07:41.571 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.569+0000 7f2d415b0740 1 -- 172.21.15.12:0/2720358460 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562afa531d70 con 0x562afa5d4ea0 2024-03-23T23:07:41.571 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.570+0000 7f2d36a00640 1 --2- 172.21.15.12:0/2720358460 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562afa5d4ea0 0x562afa5d7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:41.571 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.570+0000 7f2d36a00640 1 --2- 172.21.15.12:0/2720358460 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562afa5d4ea0 0x562afa5d7290 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db37a2dd566fdde1 server_cookie=49abd680ba7114cf in_seq=0 out_seq=0 2024-03-23T23:07:41.571 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.570+0000 7f2d21600640 10 client.26956 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:41.572 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.571+0000 7f2d21600640 1 -- 172.21.15.12:0/2720358460 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562afa531d70 con 0x562afa5d4ea0 2024-03-23T23:07:41.572 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.571+0000 7f2d21600640 10 client.26956 handle_command_reply: tid=0 2024-03-23T23:07:41.572 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.571+0000 7f2d415b0740 1 client.26956 shutdown 2024-03-23T23:07:41.572 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.571+0000 7f2d415b0740 1 -- 172.21.15.12:0/2720358460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bf1e0 msgr2=0x562afa5c9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:41.572 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.571+0000 7f2d415b0740 1 --2- 172.21.15.12:0/2720358460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bf1e0 0x562afa5c9430 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2d2802c180 tx=0x7f2d28004cf0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.572 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.571+0000 7f2d415b0740 1 -- 172.21.15.12:0/2720358460 shutdown_connections 2024-03-23T23:07:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.571+0000 7f2d415b0740 1 --2- 172.21.15.12:0/2720358460 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562afa5d4ea0 0x562afa5d7290 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.571+0000 7f2d415b0740 1 --2- 172.21.15.12:0/2720358460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562afa5c4d40 0x562afa5d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.571+0000 7f2d415b0740 1 --2- 172.21.15.12:0/2720358460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562afa5bfbd0 0x562afa5c9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.571+0000 7f2d415b0740 1 --2- 172.21.15.12:0/2720358460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bf1e0 0x562afa5c9430 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.571+0000 7f2d415b0740 1 -- 172.21.15.12:0/2720358460 >> 172.21.15.12:0/2720358460 conn(0x562afa5ba750 msgr2=0x562afa5b9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.571+0000 7f2d21600640 0 client.26956 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.571+0000 7f2d415b0740 1 -- 172.21.15.12:0/2720358460 shutdown_connections 2024-03-23T23:07:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.571+0000 7f2d415b0740 1 -- 172.21.15.12:0/2720358460 wait complete. 2024-03-23T23:07:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.571+0000 7f2d415b0740 20 client.26956 trim_cache size 0 max 16384 2024-03-23T23:07:41.586 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.587+0000 7f2d415b0740 1 Processor -- start 2024-03-23T23:07:41.586 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.587+0000 7f2d415b0740 1 -- start start 2024-03-23T23:07:41.586 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.588+0000 7f2d415b0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bf1e0 0x562afa5be0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.588+0000 7f2d415b0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562afa5bfbd0 0x562afa5d0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.588+0000 7f2d415b0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562afa5c4d40 0x562afa5dba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.588+0000 7f2d415b0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d381b1420 con 0x562afa5bfbd0 2024-03-23T23:07:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.588+0000 7f2d415b0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d381aef20 con 0x562afa5c4d40 2024-03-23T23:07:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.588+0000 7f2d415b0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d381acfc0 con 0x562afa5bf1e0 2024-03-23T23:07:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.588+0000 7f2d37400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bf1e0 0x562afa5be0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.588+0000 7f2d37400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bf1e0 0x562afa5be0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47956/0 (socket says 172.21.15.12:47956) 2024-03-23T23:07:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.588+0000 7f2d37400640 1 -- 172.21.15.12:0/687634095 learned_addr learned my addr 172.21.15.12:0/687634095 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.588+0000 7f2d37e00640 1 --2- 172.21.15.12:0/687634095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562afa5c4d40 0x562afa5dba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.588+0000 7f2d36a00640 1 --2- 172.21.15.12:0/687634095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562afa5bfbd0 0x562afa5d0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.588+0000 7f2d37400640 1 -- 172.21.15.12:0/687634095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562afa5c4d40 msgr2=0x562afa5dba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.588+0000 7f2d37400640 1 --2- 172.21.15.12:0/687634095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562afa5c4d40 0x562afa5dba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.588+0000 7f2d37400640 1 -- 172.21.15.12:0/687634095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562afa5bfbd0 msgr2=0x562afa5d0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.588+0000 7f2d37400640 1 --2- 172.21.15.12:0/687634095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562afa5bfbd0 0x562afa5d0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.588+0000 7f2d37400640 1 -- 172.21.15.12:0/687634095 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562afa5d7600 con 0x562afa5bf1e0 2024-03-23T23:07:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.588+0000 7f2d36a00640 1 --2- 172.21.15.12:0/687634095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562afa5bfbd0 0x562afa5d0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:07:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.589+0000 7f2d37400640 1 --2- 172.21.15.12:0/687634095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bf1e0 0x562afa5be0c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2d2802c8f0 tx=0x7f2d28009520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71e9f9ab28666985 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.589+0000 7f2d23e00640 1 -- 172.21.15.12:0/687634095 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2804d070 con 0x562afa5bf1e0 2024-03-23T23:07:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.589+0000 7f2d23e00640 1 -- 172.21.15.12:0/687634095 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d2802cd40 con 0x562afa5bf1e0 2024-03-23T23:07:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.589+0000 7f2d23e00640 1 -- 172.21.15.12:0/687634095 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d280042a0 con 0x562afa5bf1e0 2024-03-23T23:07:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.589+0000 7f2d415b0740 1 -- 172.21.15.12:0/687634095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bf1e0 msgr2=0x562afa5be0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.589+0000 7f2d415b0740 1 --2- 172.21.15.12:0/687634095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bf1e0 0x562afa5be0c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2d2802c8f0 tx=0x7f2d28009520 comp rx=0 tx=0).stop 2024-03-23T23:07:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.589+0000 7f2d415b0740 1 -- 172.21.15.12:0/687634095 shutdown_connections 2024-03-23T23:07:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.589+0000 7f2d415b0740 1 --2- 172.21.15.12:0/687634095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562afa5c4d40 0x562afa5dba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.589+0000 7f2d415b0740 1 --2- 172.21.15.12:0/687634095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562afa5bfbd0 0x562afa5d0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.589+0000 7f2d415b0740 1 --2- 172.21.15.12:0/687634095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bf1e0 0x562afa5be0c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.589+0000 7f2d415b0740 1 -- 172.21.15.12:0/687634095 >> 172.21.15.12:0/687634095 conn(0x562afa5ba750 msgr2=0x562afa5bd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.590+0000 7f2d415b0740 1 -- 172.21.15.12:0/687634095 shutdown_connections 2024-03-23T23:07:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.590+0000 7f2d415b0740 1 -- 172.21.15.12:0/687634095 wait complete. 2024-03-23T23:07:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.590+0000 7f2d415b0740 1 -- start start 2024-03-23T23:07:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.590+0000 7f2d415b0740 1 Processor -- start 2024-03-23T23:07:41.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.590+0000 7f2d415b0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562afa5bf1e0 0x562afa5cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:41.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.591+0000 7f2d415b0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bfbd0 0x562afa5ce380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:41.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.591+0000 7f2d415b0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562afa5c4d40 0x562afa5cf390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:41.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.591+0000 7f2d415b0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d381ad5c0 con 0x562afa5c4d40 2024-03-23T23:07:41.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.591+0000 7f2d415b0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d381ae8c0 con 0x562afa5bf1e0 2024-03-23T23:07:41.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.591+0000 7f2d415b0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d381adc00 con 0x562afa5bfbd0 2024-03-23T23:07:41.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.591+0000 7f2d415b0740 10 client.0 _ll_register_callbacks cb 0x7f2d30009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.591+0000 7f2d36a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bfbd0 0x562afa5ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.591+0000 7f2d37400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562afa5bf1e0 0x562afa5cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.591+0000 7f2d37400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562afa5bf1e0 0x562afa5cde40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:43680/0 (socket says 172.21.15.12:43680) 2024-03-23T23:07:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.591+0000 7f2d37400640 1 -- 172.21.15.12:0/1369985306 learned_addr learned my addr 172.21.15.12:0/1369985306 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.591+0000 7f2d36a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bfbd0 0x562afa5ce380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:47972/0 (socket says 172.21.15.12:47972) 2024-03-23T23:07:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.591+0000 7f2d37e00640 1 --2- 172.21.15.12:0/1369985306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562afa5c4d40 0x562afa5cf390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.591+0000 7f2d36a00640 1 -- 172.21.15.12:0/1369985306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562afa5bf1e0 msgr2=0x562afa5cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.591+0000 7f2d36a00640 1 --2- 172.21.15.12:0/1369985306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562afa5bf1e0 0x562afa5cde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.591+0000 7f2d36a00640 1 -- 172.21.15.12:0/1369985306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562afa5c4d40 msgr2=0x562afa5cf390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.591+0000 7f2d36a00640 1 --2- 172.21.15.12:0/1369985306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562afa5c4d40 0x562afa5cf390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.591+0000 7f2d36a00640 1 -- 172.21.15.12:0/1369985306 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562afa5cf980 con 0x562afa5bfbd0 2024-03-23T23:07:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.591+0000 7f2d37400640 1 --2- 172.21.15.12:0/1369985306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562afa5bf1e0 0x562afa5cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:07:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.592+0000 7f2d36a00640 1 --2- 172.21.15.12:0/1369985306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bfbd0 0x562afa5ce380 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2d2400c760 tx=0x7f2d24086040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab7b0add8823399d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.592+0000 7f2d21600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:07:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.592+0000 7f2d21600640 1 -- 172.21.15.12:0/1369985306 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2409f790 con 0x562afa5bfbd0 2024-03-23T23:07:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.592+0000 7f2d21600640 1 -- 172.21.15.12:0/1369985306 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2d24003b30 con 0x562afa5bfbd0 2024-03-23T23:07:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.592+0000 7f2d415b0740 1 -- 172.21.15.12:0/1369985306 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2d2802cd40 con 0x562afa5bfbd0 2024-03-23T23:07:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.592+0000 7f2d21600640 1 -- 172.21.15.12:0/1369985306 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d240a3de0 con 0x562afa5bfbd0 2024-03-23T23:07:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.592+0000 7f2d415b0740 10 client.26962 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.592+0000 7f2d415b0740 1 -- 172.21.15.12:0/1369985306 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x562afa630650 con 0x562afa5bfbd0 2024-03-23T23:07:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.592+0000 7f2d21600640 1 -- 172.21.15.12:0/1369985306 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d2802cd40 con 0x562afa5bfbd0 2024-03-23T23:07:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.592+0000 7f2d21600640 1 -- 172.21.15.12:0/1369985306 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f2d2409aa90 con 0x562afa5bfbd0 2024-03-23T23:07:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.592+0000 7f2d415b0740 10 client.26962 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.592+0000 7f2d415b0740 10 client.26962 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:07:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.592+0000 7f2d415b0740 20 client.26962 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.592+0000 7f2d415b0740 1 --2- 172.21.15.12:0/1369985306 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562afa630e40 0x562afa633230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.593+0000 7f2d415b0740 4 client.26962 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:07:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.593+0000 7f2d415b0740 1 -- 172.21.15.12:0/1369985306 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562afa52c0c0 con 0x562afa630e40 2024-03-23T23:07:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.593+0000 7f2d37400640 1 --2- 172.21.15.12:0/1369985306 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562afa630e40 0x562afa633230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.593+0000 7f2d37400640 1 --2- 172.21.15.12:0/1369985306 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562afa630e40 0x562afa633230 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae755cddfc753250 server_cookie=25bc9aa7eba95692 in_seq=0 out_seq=0 2024-03-23T23:07:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.593+0000 7f2d21600640 10 client.26962 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.606+0000 7f2d21600640 1 -- 172.21.15.12:0/1369985306 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562afa52c0c0 con 0x562afa630e40 2024-03-23T23:07:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.606+0000 7f2d21600640 10 client.26962 handle_command_reply: tid=0 2024-03-23T23:07:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.606+0000 7f2d415b0740 1 client.26962 shutdown 2024-03-23T23:07:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.606+0000 7f2d415b0740 1 -- 172.21.15.12:0/1369985306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bfbd0 msgr2=0x562afa5ce380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.606+0000 7f2d415b0740 1 --2- 172.21.15.12:0/1369985306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bfbd0 0x562afa5ce380 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2d2400c760 tx=0x7f2d24086040 comp rx=0 tx=0).stop 2024-03-23T23:07:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.606+0000 7f2d415b0740 1 -- 172.21.15.12:0/1369985306 shutdown_connections 2024-03-23T23:07:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.606+0000 7f2d415b0740 1 --2- 172.21.15.12:0/1369985306 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562afa630e40 0x562afa633230 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.606+0000 7f2d415b0740 1 --2- 172.21.15.12:0/1369985306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562afa5c4d40 0x562afa5cf390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.606+0000 7f2d415b0740 1 --2- 172.21.15.12:0/1369985306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562afa5bfbd0 0x562afa5ce380 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.606+0000 7f2d415b0740 1 --2- 172.21.15.12:0/1369985306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562afa5bf1e0 0x562afa5cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.606+0000 7f2d21600640 0 client.26962 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.606+0000 7f2d415b0740 1 -- 172.21.15.12:0/1369985306 >> 172.21.15.12:0/1369985306 conn(0x562afa5ba750 msgr2=0x562afa5be240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.606+0000 7f2d415b0740 1 -- 172.21.15.12:0/1369985306 shutdown_connections 2024-03-23T23:07:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.606+0000 7f2d415b0740 1 -- 172.21.15.12:0/1369985306 wait complete. 2024-03-23T23:07:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.607+0000 7f2d415b0740 20 client.26962 trim_cache size 0 max 16384 2024-03-23T23:07:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:07:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:07:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:07:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:07:41.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.607+0000 7f2d3e800640 1 -- 172.21.15.12:0/1139506548 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2d1c0a2920 msgr2=0x7f2d1c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:41.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.607+0000 7f2d3e800640 1 --2- 172.21.15.12:0/1139506548 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2d1c0a2920 0x7f2d1c0a4de0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2d30009fd0 tx=0x7f2d30009500 comp rx=0 tx=0).stop 2024-03-23T23:07:41.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.607+0000 7f2d3e800640 1 -- 172.21.15.12:0/1139506548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d38108ef0 msgr2=0x7f2d3819e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:41.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.608+0000 7f2d3e800640 1 --2- 172.21.15.12:0/1139506548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d38108ef0 0x7f2d3819e510 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2d2400ea60 tx=0x7f2d2400ef20 comp rx=0 tx=0).stop 2024-03-23T23:07:41.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.608+0000 7f2d3e800640 1 -- 172.21.15.12:0/1139506548 shutdown_connections 2024-03-23T23:07:41.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.608+0000 7f2d3e800640 1 --2- 172.21.15.12:0/1139506548 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2d1c0a2920 0x7f2d1c0a4de0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.608+0000 7f2d3e800640 1 --2- 172.21.15.12:0/1139506548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2d381098b0 0x7f2d381a5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.608+0000 7f2d3e800640 1 --2- 172.21.15.12:0/1139506548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d38108ef0 0x7f2d3819e510 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.608+0000 7f2d3e800640 1 --2- 172.21.15.12:0/1139506548 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d38107d00 0x7f2d3819dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:41.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.608+0000 7f2d3e800640 1 -- 172.21.15.12:0/1139506548 >> 172.21.15.12:0/1139506548 conn(0x7f2d38076cb0 msgr2=0x7f2d3810c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:41.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.608+0000 7f2d3e800640 1 -- 172.21.15.12:0/1139506548 shutdown_connections 2024-03-23T23:07:41.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:41.608+0000 7f2d3e800640 1 -- 172.21.15.12:0/1139506548 wait complete. 2024-03-23T23:07:42.338 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.338 INFO:tasks.workunit.client.0.smithi012.stdout:189/600: open fd = 3 2024-03-23T23:07:42.338 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.338 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.338 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.338 INFO:tasks.workunit.client.0.smithi012.stdout:190/600: open fd = 3 2024-03-23T23:07:42.338 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.338 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.338 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.338 INFO:tasks.workunit.client.0.smithi012.stdout:191/600: open fd = 3 2024-03-23T23:07:42.339 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.339 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.339 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.339 INFO:tasks.workunit.client.0.smithi012.stdout:192/600: open fd = 3 2024-03-23T23:07:42.339 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.339 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.339 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.339 INFO:tasks.workunit.client.0.smithi012.stdout:193/600: open fd = 3 2024-03-23T23:07:42.339 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.339 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.339 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.339 INFO:tasks.workunit.client.0.smithi012.stdout:194/600: open fd = 3 2024-03-23T23:07:42.339 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.339 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.339 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.340 INFO:tasks.workunit.client.0.smithi012.stdout:195/600: open fd = 3 2024-03-23T23:07:42.340 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.340 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.340 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.340 INFO:tasks.workunit.client.0.smithi012.stdout:196/600: open fd = 3 2024-03-23T23:07:42.340 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.340 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.340 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.340 INFO:tasks.workunit.client.0.smithi012.stdout:197/600: open fd = 3 2024-03-23T23:07:42.340 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.340 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.340 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.340 INFO:tasks.workunit.client.0.smithi012.stdout:198/600: open fd = 3 2024-03-23T23:07:42.340 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.341 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.341 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.341 INFO:tasks.workunit.client.0.smithi012.stdout:199/600: open fd = 3 2024-03-23T23:07:42.341 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.341 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.341 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.341 INFO:tasks.workunit.client.0.smithi012.stdout:200/600: open fd = 3 2024-03-23T23:07:42.341 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.341 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.341 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.341 INFO:tasks.workunit.client.0.smithi012.stdout:201/600: open fd = 3 2024-03-23T23:07:42.341 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.341 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.341 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.341 INFO:tasks.workunit.client.0.smithi012.stdout:202/600: open fd = 3 2024-03-23T23:07:42.341 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.342 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.342 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.342 INFO:tasks.workunit.client.0.smithi012.stdout:203/600: open fd = 3 2024-03-23T23:07:42.342 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.342 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.342 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.342 INFO:tasks.workunit.client.0.smithi012.stdout:204/600: open fd = 3 2024-03-23T23:07:42.342 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.342 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.342 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.342 INFO:tasks.workunit.client.0.smithi012.stdout:205/600: open fd = 3 2024-03-23T23:07:42.342 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.342 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.342 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.342 INFO:tasks.workunit.client.0.smithi012.stdout:206/600: open fd = 3 2024-03-23T23:07:42.342 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.342 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.343 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.343 INFO:tasks.workunit.client.0.smithi012.stdout:207/600: open fd = 3 2024-03-23T23:07:42.343 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.343 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.343 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.343 INFO:tasks.workunit.client.0.smithi012.stdout:208/600: open fd = 3 2024-03-23T23:07:42.343 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.343 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.343 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.343 INFO:tasks.workunit.client.0.smithi012.stdout:209/600: open fd = 3 2024-03-23T23:07:42.343 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.343 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.343 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.343 INFO:tasks.workunit.client.0.smithi012.stdout:210/600: open fd = 3 2024-03-23T23:07:42.343 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.343 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.344 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.344 INFO:tasks.workunit.client.0.smithi012.stdout:211/600: open fd = 3 2024-03-23T23:07:42.344 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.344 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.344 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.344 INFO:tasks.workunit.client.0.smithi012.stdout:212/600: open fd = 3 2024-03-23T23:07:42.344 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.344 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.344 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.344 INFO:tasks.workunit.client.0.smithi012.stdout:213/600: open fd = 3 2024-03-23T23:07:42.344 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.344 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.344 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.344 INFO:tasks.workunit.client.0.smithi012.stdout:214/600: open fd = 3 2024-03-23T23:07:42.344 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.344 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.344 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.345 INFO:tasks.workunit.client.0.smithi012.stdout:215/600: open fd = 3 2024-03-23T23:07:42.345 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.345 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.345 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.345 INFO:tasks.workunit.client.0.smithi012.stdout:216/600: open fd = 3 2024-03-23T23:07:42.345 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.345 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.345 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.345 INFO:tasks.workunit.client.0.smithi012.stdout:217/600: open fd = 3 2024-03-23T23:07:42.345 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.345 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.345 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.345 INFO:tasks.workunit.client.0.smithi012.stdout:218/600: open fd = 3 2024-03-23T23:07:42.345 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.345 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.345 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.346 INFO:tasks.workunit.client.0.smithi012.stdout:219/600: open fd = 3 2024-03-23T23:07:42.346 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.346 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.346 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.346 INFO:tasks.workunit.client.0.smithi012.stdout:220/600: open fd = 3 2024-03-23T23:07:42.346 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.346 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.346 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.346 INFO:tasks.workunit.client.0.smithi012.stdout:221/600: open fd = 3 2024-03-23T23:07:42.346 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.346 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.346 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.346 INFO:tasks.workunit.client.0.smithi012.stdout:222/600: open fd = 3 2024-03-23T23:07:42.346 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.347 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.347 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.347 INFO:tasks.workunit.client.0.smithi012.stdout:223/600: open fd = 3 2024-03-23T23:07:42.347 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.347 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.347 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.347 INFO:tasks.workunit.client.0.smithi012.stdout:224/600: open fd = 3 2024-03-23T23:07:42.347 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.347 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.347 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.347 INFO:tasks.workunit.client.0.smithi012.stdout:225/600: open fd = 3 2024-03-23T23:07:42.347 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.347 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.347 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.348 INFO:tasks.workunit.client.0.smithi012.stdout:226/600: open fd = 3 2024-03-23T23:07:42.348 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.348 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.348 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.348 INFO:tasks.workunit.client.0.smithi012.stdout:227/600: open fd = 3 2024-03-23T23:07:42.348 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.348 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.348 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.348 INFO:tasks.workunit.client.0.smithi012.stdout:228/600: open fd = 3 2024-03-23T23:07:42.348 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.348 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.348 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.348 INFO:tasks.workunit.client.0.smithi012.stdout:229/600: open fd = 3 2024-03-23T23:07:42.348 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.348 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.349 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.349 INFO:tasks.workunit.client.0.smithi012.stdout:230/600: open fd = 3 2024-03-23T23:07:42.349 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.349 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.349 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.349 INFO:tasks.workunit.client.0.smithi012.stdout:231/600: open fd = 3 2024-03-23T23:07:42.349 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.349 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.349 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.349 INFO:tasks.workunit.client.0.smithi012.stdout:232/600: open fd = 3 2024-03-23T23:07:42.349 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.349 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.349 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.349 INFO:tasks.workunit.client.0.smithi012.stdout:233/600: open fd = 3 2024-03-23T23:07:42.349 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.350 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.350 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.350 INFO:tasks.workunit.client.0.smithi012.stdout:234/600: open fd = 3 2024-03-23T23:07:42.350 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.350 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.350 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.350 INFO:tasks.workunit.client.0.smithi012.stdout:235/600: open fd = 3 2024-03-23T23:07:42.350 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.350 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.350 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.350 INFO:tasks.workunit.client.0.smithi012.stdout:236/600: open fd = 3 2024-03-23T23:07:42.350 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.350 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.350 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.351 INFO:tasks.workunit.client.0.smithi012.stdout:237/600: open fd = 3 2024-03-23T23:07:42.351 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.351 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.351 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.351 INFO:tasks.workunit.client.0.smithi012.stdout:238/600: open fd = 3 2024-03-23T23:07:42.351 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.351 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.351 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.351 INFO:tasks.workunit.client.0.smithi012.stdout:239/600: open fd = 3 2024-03-23T23:07:42.351 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.351 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.351 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.351 INFO:tasks.workunit.client.0.smithi012.stdout:240/600: open fd = 3 2024-03-23T23:07:42.351 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.351 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.351 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.352 INFO:tasks.workunit.client.0.smithi012.stdout:241/600: open fd = 3 2024-03-23T23:07:42.352 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.352 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.352 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.352 INFO:tasks.workunit.client.0.smithi012.stdout:242/600: open fd = 3 2024-03-23T23:07:42.352 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.352 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.352 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.352 INFO:tasks.workunit.client.0.smithi012.stdout:243/600: open fd = 3 2024-03-23T23:07:42.352 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.352 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.352 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.352 INFO:tasks.workunit.client.0.smithi012.stdout:244/600: open fd = 3 2024-03-23T23:07:42.352 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.352 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.352 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.353 INFO:tasks.workunit.client.0.smithi012.stdout:245/600: open fd = 3 2024-03-23T23:07:42.353 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.353 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.353 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.353 INFO:tasks.workunit.client.0.smithi012.stdout:246/600: open fd = 3 2024-03-23T23:07:42.353 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.353 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.353 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.353 INFO:tasks.workunit.client.0.smithi012.stdout:247/600: open fd = 3 2024-03-23T23:07:42.353 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.353 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.353 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.353 INFO:tasks.workunit.client.0.smithi012.stdout:248/600: open fd = 3 2024-03-23T23:07:42.353 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.354 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.354 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.354 INFO:tasks.workunit.client.0.smithi012.stdout:249/600: open fd = 3 2024-03-23T23:07:42.354 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.354 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.354 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:07:42.354 INFO:tasks.workunit.client.0.smithi012.stdout:250/600: open fd = 3 2024-03-23T23:07:42.354 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:42.354 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:07:43.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:42 smithi012 ceph-mon[31695]: pgmap v567: 97 pgs: 97 active+clean; 3.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-03-23T23:07:43.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:42 smithi173 ceph-mon[36116]: pgmap v567: 97 pgs: 97 active+clean; 3.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-03-23T23:07:43.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:42 smithi203 ceph-mon[36693]: pgmap v567: 97 pgs: 97 active+clean; 3.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-03-23T23:07:45.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:44 smithi012 ceph-mon[31695]: pgmap v568: 97 pgs: 97 active+clean; 3.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-23T23:07:45.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:44 smithi173 ceph-mon[36116]: pgmap v568: 97 pgs: 97 active+clean; 3.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-23T23:07:45.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:44 smithi203 ceph-mon[36693]: pgmap v568: 97 pgs: 97 active+clean; 3.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-23T23:07:47.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:46 smithi012 ceph-mon[31695]: pgmap v569: 97 pgs: 97 active+clean; 3.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-03-23T23:07:47.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:46 smithi173 ceph-mon[36116]: pgmap v569: 97 pgs: 97 active+clean; 3.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-03-23T23:07:47.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:46 smithi203 ceph-mon[36693]: pgmap v569: 97 pgs: 97 active+clean; 3.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-03-23T23:07:49.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:48 smithi012 ceph-mon[31695]: pgmap v570: 97 pgs: 97 active+clean; 3.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:07:49.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:48 smithi173 ceph-mon[36116]: pgmap v570: 97 pgs: 97 active+clean; 3.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:07:49.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:48 smithi203 ceph-mon[36693]: pgmap v570: 97 pgs: 97 active+clean; 3.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:07:51.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:50 smithi012 ceph-mon[31695]: pgmap v571: 97 pgs: 97 active+clean; 3.8 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:07:51.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:50 smithi173 ceph-mon[36116]: pgmap v571: 97 pgs: 97 active+clean; 3.8 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:07:51.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:50 smithi203 ceph-mon[36693]: pgmap v571: 97 pgs: 97 active+clean; 3.8 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:07:53.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:52 smithi012 ceph-mon[31695]: pgmap v572: 97 pgs: 97 active+clean; 3.8 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:07:53.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:52 smithi173 ceph-mon[36116]: pgmap v572: 97 pgs: 97 active+clean; 3.8 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:07:53.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:52 smithi203 ceph-mon[36693]: pgmap v572: 97 pgs: 97 active+clean; 3.8 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:07:54.754 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.755+0000 7f8832a00640 1 -- 172.21.15.12:0/4239049997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f882c1028c0 msgr2=0x7f882c102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.755+0000 7f8832a00640 1 --2- 172.21.15.12:0/4239049997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f882c1028c0 0x7f882c102d40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f8814009900 tx=0x7f881402bc30 comp rx=0 tx=0).stop 2024-03-23T23:07:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.755+0000 7f8832a00640 1 -- 172.21.15.12:0/4239049997 shutdown_connections 2024-03-23T23:07:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.755+0000 7f8832a00640 1 --2- 172.21.15.12:0/4239049997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f882c103280 0x7f882c109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.755+0000 7f8832a00640 1 --2- 172.21.15.12:0/4239049997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f882c1028c0 0x7f882c102d40 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.755+0000 7f8832a00640 1 --2- 172.21.15.12:0/4239049997 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f882c1016d0 0x7f882c101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.755+0000 7f8832a00640 1 -- 172.21.15.12:0/4239049997 >> 172.21.15.12:0/4239049997 conn(0x7f882c0fcc60 msgr2=0x7f882c0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.755+0000 7f8832a00640 1 -- 172.21.15.12:0/4239049997 shutdown_connections 2024-03-23T23:07:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.755+0000 7f8832a00640 1 -- 172.21.15.12:0/4239049997 wait complete. 2024-03-23T23:07:54.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.756+0000 7f8832a00640 1 Processor -- start 2024-03-23T23:07:54.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.756+0000 7f8832a00640 1 -- start start 2024-03-23T23:07:54.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.757+0000 7f8832a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f882c1016d0 0x7f882c199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:54.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.757+0000 7f8832a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f882c1028c0 0x7f882c19a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:54.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.757+0000 7f8832a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f882c103280 0x7f882c1a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:54.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.757+0000 7f8832a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882c10d380 con 0x7f882c1016d0 2024-03-23T23:07:54.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.757+0000 7f8832a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882c10d200 con 0x7f882c103280 2024-03-23T23:07:54.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.757+0000 7f8832a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882c10d500 con 0x7f882c1028c0 2024-03-23T23:07:54.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.757+0000 7f882be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f882c1016d0 0x7f882c199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:54.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.757+0000 7f8823e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f882c1028c0 0x7f882c19a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:54.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.757+0000 7f8830a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f882c103280 0x7f882c1a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:54.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.757+0000 7f882be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f882c1016d0 0x7f882c199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39294/0 (socket says 172.21.15.12:39294) 2024-03-23T23:07:54.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.757+0000 7f882be00640 1 -- 172.21.15.12:0/3113764975 learned_addr learned my addr 172.21.15.12:0/3113764975 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:54.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.757+0000 7f8823e00640 1 -- 172.21.15.12:0/3113764975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f882c103280 msgr2=0x7f882c1a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:54.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.757+0000 7f8823e00640 1 --2- 172.21.15.12:0/3113764975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f882c103280 0x7f882c1a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.757+0000 7f8823e00640 1 -- 172.21.15.12:0/3113764975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f882c1016d0 msgr2=0x7f882c199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:54.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.757+0000 7f8823e00640 1 --2- 172.21.15.12:0/3113764975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f882c1016d0 0x7f882c199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.757+0000 7f8823e00640 1 -- 172.21.15.12:0/3113764975 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882c1a1720 con 0x7f882c1028c0 2024-03-23T23:07:54.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.758+0000 7f8823e00640 1 --2- 172.21.15.12:0/3113764975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f882c1028c0 0x7f882c19a040 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f8814036040 tx=0x7f881402c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:54.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.758+0000 7f8829600640 1 -- 172.21.15.12:0/3113764975 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8814043020 con 0x7f882c1028c0 2024-03-23T23:07:54.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.758+0000 7f8829600640 1 -- 172.21.15.12:0/3113764975 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88140054e0 con 0x7f882c1028c0 2024-03-23T23:07:54.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.758+0000 7f8829600640 1 -- 172.21.15.12:0/3113764975 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8814003c80 con 0x7f882c1028c0 2024-03-23T23:07:54.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.758+0000 7f8832a00640 1 -- 172.21.15.12:0/3113764975 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f882c1a19b0 con 0x7f882c1028c0 2024-03-23T23:07:54.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.758+0000 7f8832a00640 1 -- 172.21.15.12:0/3113764975 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f882c1a1ed0 con 0x7f882c1028c0 2024-03-23T23:07:54.760 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.761+0000 7f8829600640 1 -- 172.21.15.12:0/3113764975 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8814046020 con 0x7f882c1028c0 2024-03-23T23:07:54.760 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.761+0000 7f883565b740 1 Processor -- start 2024-03-23T23:07:54.760 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.762+0000 7f883565b740 1 -- start start 2024-03-23T23:07:54.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.762+0000 7f883565b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a8878d1e0 0x556a8878d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:54.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.762+0000 7f883565b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556a8878dbd0 0x556a887926d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:54.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.762+0000 7f883565b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556a88792d40 0x556a887951d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:54.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.762+0000 7f883565b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a887503b0 con 0x556a8878dbd0 2024-03-23T23:07:54.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.762+0000 7f883565b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a886f2830 con 0x556a88792d40 2024-03-23T23:07:54.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.762+0000 7f883565b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a8869b5d0 con 0x556a8878d1e0 2024-03-23T23:07:54.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.762+0000 7f8830a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a8878d1e0 0x556a8878d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:54.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.762+0000 7f8823e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556a88792d40 0x556a887951d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:54.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.762+0000 7f882be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556a8878dbd0 0x556a887926d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:54.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.762+0000 7f8830a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a8878d1e0 0x556a8878d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:38748/0 (socket says 172.21.15.12:38748) 2024-03-23T23:07:54.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.762+0000 7f8830a00640 1 -- 172.21.15.12:0/624203181 learned_addr learned my addr 172.21.15.12:0/624203181 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:54.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.763+0000 7f8830a00640 1 -- 172.21.15.12:0/624203181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556a88792d40 msgr2=0x556a887951d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:54.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.763+0000 7f8830a00640 1 --2- 172.21.15.12:0/624203181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556a88792d40 0x556a887951d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.763+0000 7f8830a00640 1 -- 172.21.15.12:0/624203181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556a8878dbd0 msgr2=0x556a887926d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:54.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.763+0000 7f8830a00640 1 --2- 172.21.15.12:0/624203181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556a8878dbd0 0x556a887926d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.763+0000 7f8830a00640 1 -- 172.21.15.12:0/624203181 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556a886b32c0 con 0x556a8878d1e0 2024-03-23T23:07:54.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.763+0000 7f8829600640 1 --2- 172.21.15.12:0/3113764975 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f88100a28a0 0x7f88100a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:54.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.763+0000 7f8823e00640 1 --2- 172.21.15.12:0/624203181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556a88792d40 0x556a887951d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:07:54.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.763+0000 7f8829600640 1 -- 172.21.15.12:0/3113764975 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f88140f0af0 con 0x7f882c1028c0 2024-03-23T23:07:54.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.763+0000 7f8823e00640 1 --2- 172.21.15.12:0/3113764975 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f88100a28a0 0x7f88100a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:54.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.763+0000 7f8830a00640 1 --2- 172.21.15.12:0/624203181 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a8878d1e0 0x556a8878d600 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8824010370 tx=0x7f8824010830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3745eda6c310685 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:54.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.763+0000 7f8821200640 1 -- 172.21.15.12:0/624203181 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8824016070 con 0x556a8878d1e0 2024-03-23T23:07:54.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.763+0000 7f8821200640 1 -- 172.21.15.12:0/624203181 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8824004d20 con 0x556a8878d1e0 2024-03-23T23:07:54.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.763+0000 7f8821200640 1 -- 172.21.15.12:0/624203181 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8824005480 con 0x556a8878d1e0 2024-03-23T23:07:54.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.764+0000 7f883565b740 1 -- 172.21.15.12:0/624203181 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a8878d1e0 msgr2=0x556a8878d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:54.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.764+0000 7f8823e00640 1 --2- 172.21.15.12:0/3113764975 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f88100a28a0 0x7f88100a4d60 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8814005ab0 tx=0x7f88140b9040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:54.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.764+0000 7f883565b740 1 --2- 172.21.15.12:0/624203181 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a8878d1e0 0x556a8878d600 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8824010370 tx=0x7f8824010830 comp rx=0 tx=0).stop 2024-03-23T23:07:54.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.764+0000 7f883565b740 1 -- 172.21.15.12:0/624203181 shutdown_connections 2024-03-23T23:07:54.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.764+0000 7f883565b740 1 --2- 172.21.15.12:0/624203181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556a88792d40 0x556a887951d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.764+0000 7f883565b740 1 --2- 172.21.15.12:0/624203181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556a8878dbd0 0x556a887926d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.764+0000 7f883565b740 1 --2- 172.21.15.12:0/624203181 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a8878d1e0 0x556a8878d600 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.764+0000 7f883565b740 1 -- 172.21.15.12:0/624203181 >> 172.21.15.12:0/624203181 conn(0x556a88788750 msgr2=0x556a88788b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:54.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.764+0000 7f883565b740 1 -- 172.21.15.12:0/624203181 shutdown_connections 2024-03-23T23:07:54.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.764+0000 7f883565b740 1 -- 172.21.15.12:0/624203181 wait complete. 2024-03-23T23:07:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.765+0000 7f883565b740 1 -- start start 2024-03-23T23:07:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.765+0000 7f883565b740 1 Processor -- start 2024-03-23T23:07:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.765+0000 7f883565b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a8878d1e0 0x556a88797420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.765+0000 7f883565b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556a8878dbd0 0x556a88797960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:54.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.765+0000 7f883565b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556a88792d40 0x556a8879e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:54.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.765+0000 7f883565b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a886aea20 con 0x556a88792d40 2024-03-23T23:07:54.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.766+0000 7f883565b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a886ee8d0 con 0x556a8878dbd0 2024-03-23T23:07:54.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.766+0000 7f883565b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a88795cf0 con 0x556a8878d1e0 2024-03-23T23:07:54.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.766+0000 7f883565b740 10 client.0 _ll_register_callbacks cb 0x556a886eb1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:54.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.766+0000 7f8830a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a8878d1e0 0x556a88797420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:54.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.766+0000 7f8830a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a8878d1e0 0x556a88797420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:38758/0 (socket says 172.21.15.12:38758) 2024-03-23T23:07:54.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.766+0000 7f8830a00640 1 -- 172.21.15.12:0/4019660877 learned_addr learned my addr 172.21.15.12:0/4019660877 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:54.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.766+0000 7f882be00640 1 --2- 172.21.15.12:0/4019660877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556a8878dbd0 0x556a88797960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:54.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.766+0000 7f8830a00640 1 --2- 172.21.15.12:0/4019660877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556a88792d40 0x556a8879e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:54.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.766+0000 7f8830a00640 1 -- 172.21.15.12:0/4019660877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556a8878dbd0 msgr2=0x556a88797960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:54.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.766+0000 7f8830a00640 1 --2- 172.21.15.12:0/4019660877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556a8878dbd0 0x556a88797960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.766+0000 7f8830a00640 1 -- 172.21.15.12:0/4019660877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556a88792d40 msgr2=0x556a8879e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:54.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.766+0000 7f8830a00640 1 --2- 172.21.15.12:0/4019660877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556a88792d40 0x556a8879e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.766+0000 7f8830a00640 1 -- 172.21.15.12:0/4019660877 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556a88407210 con 0x556a8878d1e0 2024-03-23T23:07:54.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.766+0000 7f882be00640 1 --2- 172.21.15.12:0/4019660877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556a8878dbd0 0x556a88797960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:07:54.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.767+0000 7f8830a00640 1 --2- 172.21.15.12:0/4019660877 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a8878d1e0 0x556a88797420 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8824009db0 tx=0x7f8824009bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a0adb8b757d2b5f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:54.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.767+0000 7f8819600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:07:54.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.767+0000 7f8819600640 1 -- 172.21.15.12:0/4019660877 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8824016040 con 0x556a8878d1e0 2024-03-23T23:07:54.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.767+0000 7f8819600640 1 -- 172.21.15.12:0/4019660877 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f88240058b0 con 0x556a8878d1e0 2024-03-23T23:07:54.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.767+0000 7f883565b740 1 -- 172.21.15.12:0/4019660877 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8824004d20 con 0x556a8878d1e0 2024-03-23T23:07:54.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.767+0000 7f8819600640 1 -- 172.21.15.12:0/4019660877 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8824005ab0 con 0x556a8878d1e0 2024-03-23T23:07:54.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.767+0000 7f883565b740 10 client.26974 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:54.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.767+0000 7f883565b740 1 -- 172.21.15.12:0/4019660877 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556a884d3620 con 0x556a8878d1e0 2024-03-23T23:07:54.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.768+0000 7f8819600640 1 -- 172.21.15.12:0/4019660877 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8824004d20 con 0x556a8878d1e0 2024-03-23T23:07:54.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.768+0000 7f8819600640 1 -- 172.21.15.12:0/4019660877 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8824028040 con 0x556a8878d1e0 2024-03-23T23:07:54.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.768+0000 7f883565b740 10 client.26974 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:54.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.768+0000 7f883565b740 10 client.26974 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:07:54.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.768+0000 7f883565b740 20 client.26974 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:54.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.768+0000 7f883565b740 1 --2- 172.21.15.12:0/4019660877 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x556a887a2ff0 0x556a887a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:54.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.768+0000 7f883565b740 4 client.26974 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:07:54.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.768+0000 7f883565b740 1 -- 172.21.15.12:0/4019660877 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556a886ffd70 con 0x556a887a2ff0 2024-03-23T23:07:54.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.768+0000 7f882be00640 1 --2- 172.21.15.12:0/4019660877 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x556a887a2ff0 0x556a887a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:54.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.769+0000 7f882be00640 1 --2- 172.21.15.12:0/4019660877 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x556a887a2ff0 0x556a887a53e0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2196b22952bb209a server_cookie=801b3fd6798d44ce in_seq=0 out_seq=0 2024-03-23T23:07:54.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.769+0000 7f8819600640 10 client.26974 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:54.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.770+0000 7f8819600640 1 -- 172.21.15.12:0/4019660877 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556a886ffd70 con 0x556a887a2ff0 2024-03-23T23:07:54.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.770+0000 7f8819600640 10 client.26974 handle_command_reply: tid=0 2024-03-23T23:07:54.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.770+0000 7f883565b740 1 client.26974 shutdown 2024-03-23T23:07:54.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.770+0000 7f883565b740 1 -- 172.21.15.12:0/4019660877 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a8878d1e0 msgr2=0x556a88797420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.770+0000 7f883565b740 1 --2- 172.21.15.12:0/4019660877 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a8878d1e0 0x556a88797420 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8824009db0 tx=0x7f8824009bb0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.770+0000 7f883565b740 1 -- 172.21.15.12:0/4019660877 shutdown_connections 2024-03-23T23:07:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.770+0000 7f883565b740 1 --2- 172.21.15.12:0/4019660877 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x556a887a2ff0 0x556a887a53e0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.770+0000 7f883565b740 1 --2- 172.21.15.12:0/4019660877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556a88792d40 0x556a8879e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.770+0000 7f883565b740 1 --2- 172.21.15.12:0/4019660877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556a8878dbd0 0x556a88797960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.770+0000 7f8819600640 0 client.26974 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.770+0000 7f883565b740 1 --2- 172.21.15.12:0/4019660877 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a8878d1e0 0x556a88797420 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.770+0000 7f883565b740 1 -- 172.21.15.12:0/4019660877 >> 172.21.15.12:0/4019660877 conn(0x556a88788750 msgr2=0x556a88787950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.770+0000 7f883565b740 1 -- 172.21.15.12:0/4019660877 shutdown_connections 2024-03-23T23:07:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.770+0000 7f883565b740 1 -- 172.21.15.12:0/4019660877 wait complete. 2024-03-23T23:07:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.770+0000 7f883565b740 20 client.26974 trim_cache size 0 max 16384 2024-03-23T23:07:54.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.786+0000 7f883565b740 1 Processor -- start 2024-03-23T23:07:54.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.786+0000 7f883565b740 1 -- start start 2024-03-23T23:07:54.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.786+0000 7f883565b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556a8878d1e0 0x556a8878b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:54.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.786+0000 7f883565b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556a8878dbd0 0x556a8879e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:54.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.786+0000 7f883565b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a88792d40 0x556a8878bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:54.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.786+0000 7f883565b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882c1ad070 con 0x556a8878dbd0 2024-03-23T23:07:54.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.786+0000 7f883565b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882c1aab70 con 0x556a8878d1e0 2024-03-23T23:07:54.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.786+0000 7f883565b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882c1a8c10 con 0x556a88792d40 2024-03-23T23:07:54.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.786+0000 7f8830a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556a8878d1e0 0x556a8878b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:54.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.786+0000 7f8830a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556a8878d1e0 0x556a8878b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:39364/0 (socket says 172.21.15.12:39364) 2024-03-23T23:07:54.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.787+0000 7f8830a00640 1 -- 172.21.15.12:0/3798246347 learned_addr learned my addr 172.21.15.12:0/3798246347 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:54.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.787+0000 7f882be00640 1 --2- 172.21.15.12:0/3798246347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556a8878dbd0 0x556a8879e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.787+0000 7f8830a00640 1 --2- 172.21.15.12:0/3798246347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a88792d40 0x556a8878bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.787+0000 7f8830a00640 1 -- 172.21.15.12:0/3798246347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a88792d40 msgr2=0x556a8878bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.787+0000 7f8830a00640 1 --2- 172.21.15.12:0/3798246347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a88792d40 0x556a8878bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.787+0000 7f8830a00640 1 -- 172.21.15.12:0/3798246347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556a8878dbd0 msgr2=0x556a8879e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.787+0000 7f8830a00640 1 --2- 172.21.15.12:0/3798246347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556a8878dbd0 0x556a8879e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.787+0000 7f8830a00640 1 -- 172.21.15.12:0/3798246347 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556a887a5a40 con 0x556a8878d1e0 2024-03-23T23:07:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.787+0000 7f882be00640 1 --2- 172.21.15.12:0/3798246347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556a8878dbd0 0x556a8879e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:07:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.787+0000 7f8830a00640 1 --2- 172.21.15.12:0/3798246347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556a8878d1e0 0x556a8878b9b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8824004730 tx=0x7f8824010830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8dc9a2d52aca477 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.787+0000 7f881be00640 1 -- 172.21.15.12:0/3798246347 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882403a070 con 0x556a8878d1e0 2024-03-23T23:07:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.787+0000 7f881be00640 1 -- 172.21.15.12:0/3798246347 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f882403b040 con 0x556a8878d1e0 2024-03-23T23:07:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.787+0000 7f881be00640 1 -- 172.21.15.12:0/3798246347 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8824016030 con 0x556a8878d1e0 2024-03-23T23:07:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.788+0000 7f883565b740 1 -- 172.21.15.12:0/3798246347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556a8878d1e0 msgr2=0x556a8878b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.788+0000 7f883565b740 1 --2- 172.21.15.12:0/3798246347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556a8878d1e0 0x556a8878b9b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8824004730 tx=0x7f8824010830 comp rx=0 tx=0).stop 2024-03-23T23:07:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.788+0000 7f883565b740 1 -- 172.21.15.12:0/3798246347 shutdown_connections 2024-03-23T23:07:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.788+0000 7f883565b740 1 --2- 172.21.15.12:0/3798246347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a88792d40 0x556a8878bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.788+0000 7f883565b740 1 --2- 172.21.15.12:0/3798246347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556a8878dbd0 0x556a8879e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.788+0000 7f883565b740 1 --2- 172.21.15.12:0/3798246347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556a8878d1e0 0x556a8878b9b0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.788+0000 7f883565b740 1 -- 172.21.15.12:0/3798246347 >> 172.21.15.12:0/3798246347 conn(0x556a88788750 msgr2=0x556a8878ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:54.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.788+0000 7f883565b740 1 -- 172.21.15.12:0/3798246347 shutdown_connections 2024-03-23T23:07:54.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.788+0000 7f883565b740 1 -- 172.21.15.12:0/3798246347 wait complete. 2024-03-23T23:07:54.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.788+0000 7f883565b740 1 -- start start 2024-03-23T23:07:54.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.789+0000 7f883565b740 1 Processor -- start 2024-03-23T23:07:54.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.789+0000 7f883565b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a8878d1e0 0x556a8879bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:54.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.789+0000 7f883565b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556a8878dbd0 0x556a8879c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:54.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.789+0000 7f883565b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556a88792d40 0x556a8879d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:54.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.789+0000 7f883565b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882c1a9210 con 0x556a88792d40 2024-03-23T23:07:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.789+0000 7f883565b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882c1aa510 con 0x556a8878dbd0 2024-03-23T23:07:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.789+0000 7f883565b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882c1a9850 con 0x556a8878d1e0 2024-03-23T23:07:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.789+0000 7f883565b740 10 client.0 _ll_register_callbacks cb 0x7f881c009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:07:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.789+0000 7f8830a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a8878d1e0 0x556a8879bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.789+0000 7f8830a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a8878d1e0 0x556a8879bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:38784/0 (socket says 172.21.15.12:38784) 2024-03-23T23:07:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.789+0000 7f8830a00640 1 -- 172.21.15.12:0/3294727422 learned_addr learned my addr 172.21.15.12:0/3294727422 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:07:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.789+0000 7f882be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556a8878dbd0 0x556a8879c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.789+0000 7f8830a00640 1 --2- 172.21.15.12:0/3294727422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556a88792d40 0x556a8879d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.790+0000 7f882be00640 1 -- 172.21.15.12:0/3294727422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a8878d1e0 msgr2=0x556a8879bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.790+0000 7f882be00640 1 --2- 172.21.15.12:0/3294727422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a8878d1e0 0x556a8879bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.790+0000 7f882be00640 1 -- 172.21.15.12:0/3294727422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556a88792d40 msgr2=0x556a8879d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:07:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.790+0000 7f882be00640 1 --2- 172.21.15.12:0/3294727422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556a88792d40 0x556a8879d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.790+0000 7f882be00640 1 -- 172.21.15.12:0/3294727422 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556a8879da90 con 0x556a8878dbd0 2024-03-23T23:07:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.790+0000 7f882be00640 1 --2- 172.21.15.12:0/3294727422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556a8878dbd0 0x556a8879c500 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x556a887896c0 tx=0x7f881c012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9dc9b83dd5996951 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:07:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.790+0000 7f8819600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:07:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.790+0000 7f8819600640 1 -- 172.21.15.12:0/3294727422 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881c01fe30 con 0x556a8878dbd0 2024-03-23T23:07:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.790+0000 7f883565b740 1 -- 172.21.15.12:0/3294727422 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f882403b040 con 0x556a8878dbd0 2024-03-23T23:07:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.790+0000 7f8819600640 1 -- 172.21.15.12:0/3294727422 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f881c01f070 con 0x556a8878dbd0 2024-03-23T23:07:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.791+0000 7f8819600640 1 -- 172.21.15.12:0/3294727422 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f881c01f290 con 0x556a8878dbd0 2024-03-23T23:07:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.791+0000 7f883565b740 10 client.27008 fetch_fsmap learned FSMap version 23 2024-03-23T23:07:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.791+0000 7f883565b740 1 -- 172.21.15.12:0/3294727422 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556a887fe870 con 0x556a8878dbd0 2024-03-23T23:07:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.791+0000 7f8819600640 1 -- 172.21.15.12:0/3294727422 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f882403b040 con 0x556a8878dbd0 2024-03-23T23:07:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.791+0000 7f8819600640 1 -- 172.21.15.12:0/3294727422 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f881c028420 con 0x556a8878dbd0 2024-03-23T23:07:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.791+0000 7f883565b740 10 client.27008 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:07:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.791+0000 7f883565b740 10 client.27008 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:07:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.791+0000 7f883565b740 20 client.27008 populate_metadata read hostname 'smithi012' 2024-03-23T23:07:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.791+0000 7f883565b740 1 --2- 172.21.15.12:0/3294727422 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x556a887ff0b0 0x556a888014a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:07:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.791+0000 7f883565b740 4 client.27008 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:07:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.791+0000 7f883565b740 1 -- 172.21.15.12:0/3294727422 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x556a886fa0c0 con 0x556a887ff0b0 2024-03-23T23:07:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.792+0000 7f8830a00640 1 --2- 172.21.15.12:0/3294727422 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x556a887ff0b0 0x556a888014a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:07:54.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.792+0000 7f8830a00640 1 --2- 172.21.15.12:0/3294727422 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x556a887ff0b0 0x556a888014a0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f2f06b5d407d6b8 server_cookie=7b460530f6e759db in_seq=0 out_seq=0 2024-03-23T23:07:54.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.792+0000 7f8819600640 10 client.27008 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:54.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.806+0000 7f8819600640 1 -- 172.21.15.12:0/3294727422 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x556a886fa0c0 con 0x556a887ff0b0 2024-03-23T23:07:54.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.807+0000 7f8819600640 10 client.27008 handle_command_reply: tid=0 2024-03-23T23:07:54.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.807+0000 7f883565b740 1 client.27008 shutdown 2024-03-23T23:07:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.807+0000 7f883565b740 1 -- 172.21.15.12:0/3294727422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556a8878dbd0 msgr2=0x556a8879c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.807+0000 7f883565b740 1 --2- 172.21.15.12:0/3294727422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556a8878dbd0 0x556a8879c500 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x556a887896c0 tx=0x7f881c012bc0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.807+0000 7f883565b740 1 -- 172.21.15.12:0/3294727422 shutdown_connections 2024-03-23T23:07:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.807+0000 7f883565b740 1 --2- 172.21.15.12:0/3294727422 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x556a887ff0b0 0x556a888014a0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.807+0000 7f883565b740 1 --2- 172.21.15.12:0/3294727422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556a88792d40 0x556a8879d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.807+0000 7f883565b740 1 --2- 172.21.15.12:0/3294727422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556a8878dbd0 0x556a8879c500 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.807+0000 7f883565b740 1 --2- 172.21.15.12:0/3294727422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a8878d1e0 0x556a8879bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.807+0000 7f8819600640 0 client.27008 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:07:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.807+0000 7f883565b740 1 -- 172.21.15.12:0/3294727422 >> 172.21.15.12:0/3294727422 conn(0x556a88788750 msgr2=0x556a8878b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.807+0000 7f883565b740 1 -- 172.21.15.12:0/3294727422 shutdown_connections 2024-03-23T23:07:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.807+0000 7f883565b740 1 -- 172.21.15.12:0/3294727422 wait complete. 2024-03-23T23:07:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.808+0000 7f883565b740 20 client.27008 trim_cache size 0 max 16384 2024-03-23T23:07:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:07:54.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:07:54.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:07:54.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:07:54.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.808+0000 7f8832a00640 1 -- 172.21.15.12:0/3113764975 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f88100a28a0 msgr2=0x7f88100a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:54.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.808+0000 7f8832a00640 1 --2- 172.21.15.12:0/3113764975 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f88100a28a0 0x7f88100a4d60 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8814005ab0 tx=0x7f88140b9040 comp rx=0 tx=0).stop 2024-03-23T23:07:54.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.808+0000 7f8832a00640 1 -- 172.21.15.12:0/3113764975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f882c1028c0 msgr2=0x7f882c19a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:07:54.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.808+0000 7f8832a00640 1 --2- 172.21.15.12:0/3113764975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f882c1028c0 0x7f882c19a040 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f8814036040 tx=0x7f881402c6c0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.808+0000 7f8832a00640 1 -- 172.21.15.12:0/3113764975 shutdown_connections 2024-03-23T23:07:54.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.809+0000 7f8832a00640 1 --2- 172.21.15.12:0/3113764975 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f88100a28a0 0x7f88100a4d60 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.809+0000 7f8832a00640 1 --2- 172.21.15.12:0/3113764975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f882c103280 0x7f882c1a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.809+0000 7f8832a00640 1 --2- 172.21.15.12:0/3113764975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f882c1028c0 0x7f882c19a040 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.809+0000 7f8832a00640 1 --2- 172.21.15.12:0/3113764975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f882c1016d0 0x7f882c199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:07:54.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.809+0000 7f8832a00640 1 -- 172.21.15.12:0/3113764975 >> 172.21.15.12:0/3113764975 conn(0x7f882c0fcc60 msgr2=0x7f882c0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:07:54.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.809+0000 7f8832a00640 1 -- 172.21.15.12:0/3113764975 shutdown_connections 2024-03-23T23:07:54.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:07:54.809+0000 7f8832a00640 1 -- 172.21.15.12:0/3113764975 wait complete. 2024-03-23T23:07:55.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:54 smithi012 ceph-mon[31695]: pgmap v573: 97 pgs: 97 active+clean; 3.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:07:55.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:54 smithi173 ceph-mon[36116]: pgmap v573: 97 pgs: 97 active+clean; 3.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:07:55.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:54 smithi203 ceph-mon[36693]: pgmap v573: 97 pgs: 97 active+clean; 3.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:07:57.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:56 smithi012 ceph-mon[31695]: pgmap v574: 97 pgs: 97 active+clean; 3.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:07:57.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:56 smithi173 ceph-mon[36116]: pgmap v574: 97 pgs: 97 active+clean; 3.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:07:57.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:56 smithi203 ceph-mon[36693]: pgmap v574: 97 pgs: 97 active+clean; 3.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:07:59.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:07:58 smithi012 ceph-mon[31695]: pgmap v575: 97 pgs: 97 active+clean; 3.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-23T23:07:59.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:07:58 smithi173 ceph-mon[36116]: pgmap v575: 97 pgs: 97 active+clean; 3.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-23T23:07:59.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:07:58 smithi203 ceph-mon[36693]: pgmap v575: 97 pgs: 97 active+clean; 3.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-23T23:08:01.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:00 smithi012 ceph-mon[31695]: pgmap v576: 97 pgs: 97 active+clean; 3.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:08:01.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:00 smithi173 ceph-mon[36116]: pgmap v576: 97 pgs: 97 active+clean; 3.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:08:01.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:00 smithi203 ceph-mon[36693]: pgmap v576: 97 pgs: 97 active+clean; 3.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:08:03.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:02 smithi012 ceph-mon[31695]: pgmap v577: 97 pgs: 97 active+clean; 3.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:08:03.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:02 smithi173 ceph-mon[36116]: pgmap v577: 97 pgs: 97 active+clean; 3.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:08:03.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:02 smithi203 ceph-mon[36693]: pgmap v577: 97 pgs: 97 active+clean; 3.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:08:03.947 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:08:03.948 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:08:04.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.187+0000 7f12d1400640 1 -- 172.21.15.12:0/1753691631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12cc077c40 msgr2=0x7f12cc0784e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:04.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.187+0000 7f12d1400640 1 --2- 172.21.15.12:0/1753691631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12cc077c40 0x7f12cc0784e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f12c4009900 tx=0x7f12c402bc30 comp rx=0 tx=0).stop 2024-03-23T23:08:04.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.187+0000 7f12d1400640 1 -- 172.21.15.12:0/1753691631 shutdown_connections 2024-03-23T23:08:04.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.187+0000 7f12d1400640 1 --2- 172.21.15.12:0/1753691631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12cc077c40 0x7f12cc0784e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.187+0000 7f12d1400640 1 --2- 172.21.15.12:0/1753691631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12cc0772a0 0x7f12cc077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.187+0000 7f12d1400640 1 --2- 172.21.15.12:0/1753691631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12cc076060 0x7f12cc076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.187+0000 7f12d1400640 1 -- 172.21.15.12:0/1753691631 >> 172.21.15.12:0/1753691631 conn(0x7f12cc0fed10 msgr2=0x7f12cc101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:04.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.187+0000 7f12d1400640 1 -- 172.21.15.12:0/1753691631 shutdown_connections 2024-03-23T23:08:04.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.187+0000 7f12d1400640 1 -- 172.21.15.12:0/1753691631 wait complete. 2024-03-23T23:08:04.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.188+0000 7f12d1400640 1 Processor -- start 2024-03-23T23:08:04.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.188+0000 7f12d1400640 1 -- start start 2024-03-23T23:08:04.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.188+0000 7f12d1400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12cc076060 0x7f12cc19de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.188+0000 7f12d1400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12cc0772a0 0x7f12cc19e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.188+0000 7f12d1400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12cc077c40 0x7f12cc1a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.189+0000 7f12d1400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc113870 con 0x7f12cc0772a0 2024-03-23T23:08:04.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.189+0000 7f12d1400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc1136f0 con 0x7f12cc077c40 2024-03-23T23:08:04.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.189+0000 7f12d1400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc1139f0 con 0x7f12cc076060 2024-03-23T23:08:04.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.189+0000 7f12c3400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12cc0772a0 0x7f12cc19e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.189+0000 7f12c3400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12cc0772a0 0x7f12cc19e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59302/0 (socket says 172.21.15.12:59302) 2024-03-23T23:08:04.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.189+0000 7f12c3400640 1 -- 172.21.15.12:0/679539205 learned_addr learned my addr 172.21.15.12:0/679539205 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:04.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.189+0000 7f12cbe00640 1 --2- 172.21.15.12:0/679539205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12cc076060 0x7f12cc19de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.189+0000 7f12d0a00640 1 --2- 172.21.15.12:0/679539205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12cc077c40 0x7f12cc1a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.189+0000 7f12c3400640 1 -- 172.21.15.12:0/679539205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12cc076060 msgr2=0x7f12cc19de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:04.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.189+0000 7f12c3400640 1 --2- 172.21.15.12:0/679539205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12cc076060 0x7f12cc19de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.189+0000 7f12c3400640 1 -- 172.21.15.12:0/679539205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12cc077c40 msgr2=0x7f12cc1a53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:04.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.189+0000 7f12c3400640 1 --2- 172.21.15.12:0/679539205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12cc077c40 0x7f12cc1a53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.189+0000 7f12c3400640 1 -- 172.21.15.12:0/679539205 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12cc1a5a60 con 0x7f12cc0772a0 2024-03-23T23:08:04.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.189+0000 7f12d0a00640 1 --2- 172.21.15.12:0/679539205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12cc077c40 0x7f12cc1a53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:08:04.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.189+0000 7f12c3400640 1 --2- 172.21.15.12:0/679539205 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12cc0772a0 0x7f12cc19e380 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f12b400e9f0 tx=0x7f12b400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:04.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.190+0000 7f12c9600640 1 -- 172.21.15.12:0/679539205 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b400cd80 con 0x7f12cc0772a0 2024-03-23T23:08:04.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.190+0000 7f12c9600640 1 -- 172.21.15.12:0/679539205 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12b4010910 con 0x7f12cc0772a0 2024-03-23T23:08:04.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.190+0000 7f12d1400640 1 -- 172.21.15.12:0/679539205 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12cc1a5d50 con 0x7f12cc0772a0 2024-03-23T23:08:04.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.190+0000 7f12c9600640 1 -- 172.21.15.12:0/679539205 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b4018990 con 0x7f12cc0772a0 2024-03-23T23:08:04.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.190+0000 7f12d1400640 1 -- 172.21.15.12:0/679539205 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12cc1a6240 con 0x7f12cc0772a0 2024-03-23T23:08:04.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.193+0000 7f12c9600640 1 -- 172.21.15.12:0/679539205 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f12b4010ac0 con 0x7f12cc0772a0 2024-03-23T23:08:04.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.194+0000 7f12d577e740 1 Processor -- start 2024-03-23T23:08:04.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.194+0000 7f12d577e740 1 -- start start 2024-03-23T23:08:04.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.194+0000 7f12d577e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5d050 0x5623e4c5d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.194+0000 7f12d577e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c5da40 0x5623e4c62540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.194+0000 7f12d577e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623e4c62bb0 0x5623e4c65040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.194+0000 7f12d577e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623e4bc2250 con 0x5623e4c62bb0 2024-03-23T23:08:04.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.194+0000 7f12d577e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623e4b6aa30 con 0x5623e4c5d050 2024-03-23T23:08:04.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.194+0000 7f12d577e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623e4bbe2f0 con 0x5623e4c5da40 2024-03-23T23:08:04.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.194+0000 7f12cbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c5da40 0x5623e4c62540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.194+0000 7f12d0a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5d050 0x5623e4c5d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.194+0000 7f12cbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c5da40 0x5623e4c62540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44488/0 (socket says 172.21.15.12:44488) 2024-03-23T23:08:04.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.194+0000 7f12d0a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5d050 0x5623e4c5d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44782/0 (socket says 172.21.15.12:44782) 2024-03-23T23:08:04.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.194+0000 7f12cbe00640 1 -- 172.21.15.12:0/4069620934 learned_addr learned my addr 172.21.15.12:0/4069620934 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:04.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.194+0000 7f12c3400640 1 --2- 172.21.15.12:0/4069620934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623e4c62bb0 0x5623e4c65040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.195+0000 7f12d0a00640 1 -- 172.21.15.12:0/4069620934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c5da40 msgr2=0x5623e4c62540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:04.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.195+0000 7f12d0a00640 1 --2- 172.21.15.12:0/4069620934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c5da40 0x5623e4c62540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.195+0000 7f12d0a00640 1 -- 172.21.15.12:0/4069620934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623e4c62bb0 msgr2=0x5623e4c65040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:04.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.195+0000 7f12d0a00640 1 --2- 172.21.15.12:0/4069620934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623e4c62bb0 0x5623e4c65040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.195+0000 7f12d0a00640 1 -- 172.21.15.12:0/4069620934 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5623e4b55690 con 0x5623e4c5d050 2024-03-23T23:08:04.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.195+0000 7f12cbe00640 1 --2- 172.21.15.12:0/4069620934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c5da40 0x5623e4c62540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:04.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.195+0000 7f12c9600640 1 --2- 172.21.15.12:0/679539205 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f12b00a2800 0x7f12b00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.195+0000 7f12d0a00640 1 --2- 172.21.15.12:0/4069620934 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5d050 0x5623e4c5d470 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f12c402c170 tx=0x7f12c4009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eae40226930615b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:04.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.195+0000 7f12c9600640 1 -- 172.21.15.12:0/679539205 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f12b40cee30 con 0x7f12cc0772a0 2024-03-23T23:08:04.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.195+0000 7f12c1200640 1 -- 172.21.15.12:0/4069620934 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c4044020 con 0x5623e4c5d050 2024-03-23T23:08:04.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.196+0000 7f12c1200640 1 -- 172.21.15.12:0/4069620934 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12c402cb80 con 0x5623e4c5d050 2024-03-23T23:08:04.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.196+0000 7f12cbe00640 1 --2- 172.21.15.12:0/679539205 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f12b00a2800 0x7f12b00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.196+0000 7f12c1200640 1 -- 172.21.15.12:0/4069620934 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c4004920 con 0x5623e4c5d050 2024-03-23T23:08:04.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.196+0000 7f12d577e740 1 -- 172.21.15.12:0/4069620934 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5d050 msgr2=0x5623e4c5d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:04.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.196+0000 7f12d577e740 1 --2- 172.21.15.12:0/4069620934 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5d050 0x5623e4c5d470 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f12c402c170 tx=0x7f12c4009560 comp rx=0 tx=0).stop 2024-03-23T23:08:04.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.196+0000 7f12cbe00640 1 --2- 172.21.15.12:0/679539205 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f12b00a2800 0x7f12b00a4cc0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x5623e4c5de40 tx=0x7f12bc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:04.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.196+0000 7f12d577e740 1 -- 172.21.15.12:0/4069620934 shutdown_connections 2024-03-23T23:08:04.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.196+0000 7f12d577e740 1 --2- 172.21.15.12:0/4069620934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623e4c62bb0 0x5623e4c65040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.196+0000 7f12d577e740 1 --2- 172.21.15.12:0/4069620934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c5da40 0x5623e4c62540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.196+0000 7f12d577e740 1 --2- 172.21.15.12:0/4069620934 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5d050 0x5623e4c5d470 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.196+0000 7f12d577e740 1 -- 172.21.15.12:0/4069620934 >> 172.21.15.12:0/4069620934 conn(0x5623e4c25200 msgr2=0x5623e4c589c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:04.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.197+0000 7f12d577e740 1 -- 172.21.15.12:0/4069620934 shutdown_connections 2024-03-23T23:08:04.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.197+0000 7f12d577e740 1 -- 172.21.15.12:0/4069620934 wait complete. 2024-03-23T23:08:04.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.197+0000 7f12d577e740 1 -- start start 2024-03-23T23:08:04.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.197+0000 7f12d577e740 1 Processor -- start 2024-03-23T23:08:04.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.198+0000 7f12d577e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623e4c5d050 0x5623e4c67030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.198+0000 7f12d577e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5da40 0x5623e4c67570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.198+0000 7f12d577e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c62bb0 0x5623e4c6e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.198+0000 7f12d577e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623e4c65b00 con 0x5623e4c5d050 2024-03-23T23:08:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.198+0000 7f12d577e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623e4b7de80 con 0x5623e4c5da40 2024-03-23T23:08:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.198+0000 7f12d577e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623e4c65c80 con 0x5623e4c62bb0 2024-03-23T23:08:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.198+0000 7f12d577e740 10 client.0 _ll_register_callbacks cb 0x5623e4b90130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.198+0000 7f12cbe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623e4c5d050 0x5623e4c67030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.198+0000 7f12c3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5da40 0x5623e4c67570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.198+0000 7f12d0a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c62bb0 0x5623e4c6e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.198+0000 7f12cbe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623e4c5d050 0x5623e4c67030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59316/0 (socket says 172.21.15.12:59316) 2024-03-23T23:08:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.198+0000 7f12c3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5da40 0x5623e4c67570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44792/0 (socket says 172.21.15.12:44792) 2024-03-23T23:08:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.198+0000 7f12cbe00640 1 -- 172.21.15.12:0/1674835576 learned_addr learned my addr 172.21.15.12:0/1674835576 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.198+0000 7f12c3400640 1 -- 172.21.15.12:0/1674835576 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c62bb0 msgr2=0x5623e4c6e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.198+0000 7f12c3400640 1 --2- 172.21.15.12:0/1674835576 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c62bb0 0x5623e4c6e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.198+0000 7f12c3400640 1 -- 172.21.15.12:0/1674835576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623e4c5d050 msgr2=0x5623e4c67030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.198+0000 7f12c3400640 1 --2- 172.21.15.12:0/1674835576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623e4c5d050 0x5623e4c67030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.199+0000 7f12c3400640 1 -- 172.21.15.12:0/1674835576 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5623e49a07d0 con 0x5623e4c5da40 2024-03-23T23:08:04.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.199+0000 7f12d0a00640 1 --2- 172.21.15.12:0/1674835576 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c62bb0 0x5623e4c6e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:04.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.199+0000 7f12c3400640 1 --2- 172.21.15.12:0/1674835576 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5da40 0x5623e4c67570 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f12b40cc990 tx=0x7f12b4097480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16afae3774ef1b77 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:04.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.199+0000 7f12b9600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:08:04.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.199+0000 7f12b9600640 1 -- 172.21.15.12:0/1674835576 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b409b140 con 0x5623e4c5da40 2024-03-23T23:08:04.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.199+0000 7f12b9600640 1 -- 172.21.15.12:0/1674835576 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f12b40860e0 con 0x5623e4c5da40 2024-03-23T23:08:04.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.199+0000 7f12d577e740 1 -- 172.21.15.12:0/1674835576 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f12c402cb80 con 0x5623e4c5da40 2024-03-23T23:08:04.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.199+0000 7f12b9600640 1 -- 172.21.15.12:0/1674835576 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12b409bee0 con 0x5623e4c5da40 2024-03-23T23:08:04.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.199+0000 7f12d577e740 10 client.27020 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:04.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.199+0000 7f12d577e740 1 -- 172.21.15.12:0/1674835576 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5623e48d63f0 con 0x5623e4c5da40 2024-03-23T23:08:04.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.199+0000 7f12b9600640 1 -- 172.21.15.12:0/1674835576 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12c402cb80 con 0x5623e4c5da40 2024-03-23T23:08:04.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.200+0000 7f12b9600640 1 -- 172.21.15.12:0/1674835576 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f12b40afbc0 con 0x5623e4c5da40 2024-03-23T23:08:04.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.200+0000 7f12d577e740 10 client.27020 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:04.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.200+0000 7f12d577e740 10 client.27020 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:08:04.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.200+0000 7f12d577e740 20 client.27020 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:04.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.200+0000 7f12d577e740 1 --2- 172.21.15.12:0/1674835576 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5623e4c72d60 0x5623e4c751c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.200+0000 7f12d577e740 4 client.27020 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:08:04.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.200+0000 7f12d577e740 1 -- 172.21.15.12:0/1674835576 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5623e4bcf790 con 0x5623e4c72d60 2024-03-23T23:08:04.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.200+0000 7f12cbe00640 1 --2- 172.21.15.12:0/1674835576 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5623e4c72d60 0x5623e4c751c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.201+0000 7f12cbe00640 1 --2- 172.21.15.12:0/1674835576 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5623e4c72d60 0x5623e4c751c0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7f03e8b1b7705a3 server_cookie=2eceeb1fea220a90 in_seq=0 out_seq=0 2024-03-23T23:08:04.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.201+0000 7f12b9600640 10 client.27020 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:04.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.201+0000 7f12b9600640 1 -- 172.21.15.12:0/1674835576 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5623e4bcf790 con 0x5623e4c72d60 2024-03-23T23:08:04.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.201+0000 7f12b9600640 10 client.27020 handle_command_reply: tid=0 2024-03-23T23:08:04.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.201+0000 7f12d577e740 1 client.27020 shutdown 2024-03-23T23:08:04.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.202+0000 7f12d577e740 1 -- 172.21.15.12:0/1674835576 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5da40 msgr2=0x5623e4c67570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:04.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.202+0000 7f12d577e740 1 --2- 172.21.15.12:0/1674835576 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5da40 0x5623e4c67570 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f12b40cc990 tx=0x7f12b4097480 comp rx=0 tx=0).stop 2024-03-23T23:08:04.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.202+0000 7f12d577e740 1 -- 172.21.15.12:0/1674835576 shutdown_connections 2024-03-23T23:08:04.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.202+0000 7f12d577e740 1 --2- 172.21.15.12:0/1674835576 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5623e4c72d60 0x5623e4c751c0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.202+0000 7f12d577e740 1 --2- 172.21.15.12:0/1674835576 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c62bb0 0x5623e4c6e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.202+0000 7f12d577e740 1 --2- 172.21.15.12:0/1674835576 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5da40 0x5623e4c67570 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.202+0000 7f12d577e740 1 --2- 172.21.15.12:0/1674835576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623e4c5d050 0x5623e4c67030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.202+0000 7f12d577e740 1 -- 172.21.15.12:0/1674835576 >> 172.21.15.12:0/1674835576 conn(0x5623e4c25200 msgr2=0x5623e4c57bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:04.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.202+0000 7f12b9600640 0 client.27020 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:04.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.202+0000 7f12d577e740 1 -- 172.21.15.12:0/1674835576 shutdown_connections 2024-03-23T23:08:04.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.202+0000 7f12d577e740 1 -- 172.21.15.12:0/1674835576 wait complete. 2024-03-23T23:08:04.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.202+0000 7f12d577e740 20 client.27020 trim_cache size 0 max 16384 2024-03-23T23:08:04.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.220+0000 7f12d577e740 1 Processor -- start 2024-03-23T23:08:04.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.221+0000 7f12d577e740 1 -- start start 2024-03-23T23:08:04.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.221+0000 7f12d577e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5d050 0x5623e4c5bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.221+0000 7f12d577e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623e4c5da40 0x5623e4c6e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.221+0000 7f12d577e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c62bb0 0x5623e4c5c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.221+0000 7f12d577e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc1b1350 con 0x5623e4c5da40 2024-03-23T23:08:04.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.221+0000 7f12d577e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc1aee50 con 0x5623e4c5d050 2024-03-23T23:08:04.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.221+0000 7f12d577e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc1acef0 con 0x5623e4c62bb0 2024-03-23T23:08:04.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.221+0000 7f12c3400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623e4c5da40 0x5623e4c6e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.221+0000 7f12d0a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c62bb0 0x5623e4c5c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.221+0000 7f12d0a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c62bb0 0x5623e4c5c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44496/0 (socket says 172.21.15.12:44496) 2024-03-23T23:08:04.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.221+0000 7f12c3400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623e4c5da40 0x5623e4c6e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59322/0 (socket says 172.21.15.12:59322) 2024-03-23T23:08:04.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.221+0000 7f12d0a00640 1 -- 172.21.15.12:0/3684062994 learned_addr learned my addr 172.21.15.12:0/3684062994 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:04.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.221+0000 7f12cbe00640 1 --2- 172.21.15.12:0/3684062994 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5d050 0x5623e4c5bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.222+0000 7f12d0a00640 1 -- 172.21.15.12:0/3684062994 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5d050 msgr2=0x5623e4c5bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:04.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.222+0000 7f12d0a00640 1 --2- 172.21.15.12:0/3684062994 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5d050 0x5623e4c5bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.222+0000 7f12d0a00640 1 -- 172.21.15.12:0/3684062994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623e4c5da40 msgr2=0x5623e4c6e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:04.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.222+0000 7f12d0a00640 1 --2- 172.21.15.12:0/3684062994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623e4c5da40 0x5623e4c6e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.222+0000 7f12d0a00640 1 -- 172.21.15.12:0/3684062994 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5623e48960f0 con 0x5623e4c62bb0 2024-03-23T23:08:04.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.222+0000 7f12c3400640 1 --2- 172.21.15.12:0/3684062994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623e4c5da40 0x5623e4c6e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:04.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.222+0000 7f12d0a00640 1 --2- 172.21.15.12:0/3684062994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c62bb0 0x5623e4c5c140 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f12c402c170 tx=0x7f12c4000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce882a24e0f08209 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:04.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.222+0000 7f12bbe00640 1 -- 172.21.15.12:0/3684062994 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c4044040 con 0x5623e4c62bb0 2024-03-23T23:08:04.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.222+0000 7f12bbe00640 1 -- 172.21.15.12:0/3684062994 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12c4005030 con 0x5623e4c62bb0 2024-03-23T23:08:04.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.222+0000 7f12bbe00640 1 -- 172.21.15.12:0/3684062994 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c4004920 con 0x5623e4c62bb0 2024-03-23T23:08:04.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.223+0000 7f12d577e740 1 -- 172.21.15.12:0/3684062994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c62bb0 msgr2=0x5623e4c5c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:04.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.223+0000 7f12d577e740 1 --2- 172.21.15.12:0/3684062994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c62bb0 0x5623e4c5c140 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f12c402c170 tx=0x7f12c4000bc0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.223+0000 7f12d577e740 1 -- 172.21.15.12:0/3684062994 shutdown_connections 2024-03-23T23:08:04.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.223+0000 7f12d577e740 1 --2- 172.21.15.12:0/3684062994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c62bb0 0x5623e4c5c140 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.223+0000 7f12d577e740 1 --2- 172.21.15.12:0/3684062994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623e4c5da40 0x5623e4c6e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.223+0000 7f12d577e740 1 --2- 172.21.15.12:0/3684062994 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5d050 0x5623e4c5bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.223+0000 7f12d577e740 1 -- 172.21.15.12:0/3684062994 >> 172.21.15.12:0/3684062994 conn(0x5623e4c25200 msgr2=0x5623e4c5ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:04.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.223+0000 7f12d577e740 1 -- 172.21.15.12:0/3684062994 shutdown_connections 2024-03-23T23:08:04.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.223+0000 7f12d577e740 1 -- 172.21.15.12:0/3684062994 wait complete. 2024-03-23T23:08:04.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.223+0000 7f12d577e740 1 -- start start 2024-03-23T23:08:04.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.224+0000 7f12d577e740 1 Processor -- start 2024-03-23T23:08:04.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.224+0000 7f12d577e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5d050 0x5623e4c6bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.224+0000 7f12d577e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c5da40 0x5623e4c6c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.224+0000 7f12d577e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623e4c62bb0 0x5623e4c6d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.224+0000 7f12d577e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc1ad4f0 con 0x5623e4c62bb0 2024-03-23T23:08:04.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.224+0000 7f12d577e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc1ae7f0 con 0x5623e4c5d050 2024-03-23T23:08:04.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.224+0000 7f12d577e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc1adb30 con 0x5623e4c5da40 2024-03-23T23:08:04.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.224+0000 7f12d577e740 10 client.0 _ll_register_callbacks cb 0x7f12b40d2600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:04.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.224+0000 7f12c3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c5da40 0x5623e4c6c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.224+0000 7f12cbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5d050 0x5623e4c6bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.224+0000 7f12c3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c5da40 0x5623e4c6c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44500/0 (socket says 172.21.15.12:44500) 2024-03-23T23:08:04.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.224+0000 7f12cbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5d050 0x5623e4c6bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44808/0 (socket says 172.21.15.12:44808) 2024-03-23T23:08:04.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.224+0000 7f12c3400640 1 -- 172.21.15.12:0/3319467430 learned_addr learned my addr 172.21.15.12:0/3319467430 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:04.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.224+0000 7f12d0a00640 1 --2- 172.21.15.12:0/3319467430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623e4c62bb0 0x5623e4c6d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.225+0000 7f12c3400640 1 -- 172.21.15.12:0/3319467430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5d050 msgr2=0x5623e4c6bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:04.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.225+0000 7f12c3400640 1 --2- 172.21.15.12:0/3319467430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5d050 0x5623e4c6bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.225+0000 7f12c3400640 1 -- 172.21.15.12:0/3319467430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623e4c62bb0 msgr2=0x5623e4c6d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:04.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.225+0000 7f12c3400640 1 --2- 172.21.15.12:0/3319467430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623e4c62bb0 0x5623e4c6d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.225+0000 7f12c3400640 1 -- 172.21.15.12:0/3319467430 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5623e4c6d770 con 0x5623e4c5da40 2024-03-23T23:08:04.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.225+0000 7f12cbe00640 1 --2- 172.21.15.12:0/3319467430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5d050 0x5623e4c6bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:04.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.225+0000 7f12c3400640 1 --2- 172.21.15.12:0/3319467430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c5da40 0x5623e4c6c170 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f12b40cccb0 tx=0x7f12b400cf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef88039357294af9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:04.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.225+0000 7f12b9600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:08:04.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.225+0000 7f12b9600640 1 -- 172.21.15.12:0/3319467430 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b409f930 con 0x5623e4c5da40 2024-03-23T23:08:04.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.225+0000 7f12b9600640 1 -- 172.21.15.12:0/3319467430 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f12b40afbc0 con 0x5623e4c5da40 2024-03-23T23:08:04.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.225+0000 7f12d577e740 1 -- 172.21.15.12:0/3319467430 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f12c4005030 con 0x5623e4c5da40 2024-03-23T23:08:04.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.226+0000 7f12b9600640 1 -- 172.21.15.12:0/3319467430 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12b40a3f80 con 0x5623e4c5da40 2024-03-23T23:08:04.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.226+0000 7f12d577e740 10 client.26998 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:04.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.226+0000 7f12d577e740 1 -- 172.21.15.12:0/3319467430 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5623e4cce700 con 0x5623e4c5da40 2024-03-23T23:08:04.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.226+0000 7f12b9600640 1 -- 172.21.15.12:0/3319467430 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12c4005030 con 0x5623e4c5da40 2024-03-23T23:08:04.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.226+0000 7f12b9600640 1 -- 172.21.15.12:0/3319467430 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f12b4097530 con 0x5623e4c5da40 2024-03-23T23:08:04.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.226+0000 7f12d577e740 10 client.26998 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:04.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.226+0000 7f12d577e740 10 client.26998 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:08:04.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.226+0000 7f12d577e740 20 client.26998 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:04.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.226+0000 7f12d577e740 1 --2- 172.21.15.12:0/3319467430 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5623e4cceff0 0x5623e4cd1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.226+0000 7f12d577e740 4 client.26998 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:08:04.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.226+0000 7f12d577e740 1 -- 172.21.15.12:0/3319467430 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5623e4bc9ae0 con 0x5623e4cceff0 2024-03-23T23:08:04.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.226+0000 7f12cbe00640 1 --2- 172.21.15.12:0/3319467430 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5623e4cceff0 0x5623e4cd1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.227+0000 7f12cbe00640 1 --2- 172.21.15.12:0/3319467430 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5623e4cceff0 0x5623e4cd1410 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95f2defa2050c79e server_cookie=4b54d6593036d498 in_seq=0 out_seq=0 2024-03-23T23:08:04.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.227+0000 7f12b9600640 10 client.26998 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:04.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.227+0000 7f12b9600640 1 -- 172.21.15.12:0/3319467430 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5623e4bc9ae0 con 0x5623e4cceff0 2024-03-23T23:08:04.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.227+0000 7f12b9600640 10 client.26998 handle_command_reply: tid=0 2024-03-23T23:08:04.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.227+0000 7f12d577e740 1 client.26998 shutdown 2024-03-23T23:08:04.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.227+0000 7f12d577e740 1 -- 172.21.15.12:0/3319467430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c5da40 msgr2=0x5623e4c6c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:04.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.227+0000 7f12d577e740 1 --2- 172.21.15.12:0/3319467430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c5da40 0x5623e4c6c170 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f12b40cccb0 tx=0x7f12b400cf20 comp rx=0 tx=0).stop 2024-03-23T23:08:04.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.227+0000 7f12d577e740 1 -- 172.21.15.12:0/3319467430 shutdown_connections 2024-03-23T23:08:04.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.227+0000 7f12d577e740 1 --2- 172.21.15.12:0/3319467430 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5623e4cceff0 0x5623e4cd1410 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.227+0000 7f12d577e740 1 --2- 172.21.15.12:0/3319467430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623e4c62bb0 0x5623e4c6d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.227+0000 7f12d577e740 1 --2- 172.21.15.12:0/3319467430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e4c5da40 0x5623e4c6c170 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.227+0000 7f12b9600640 0 client.26998 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:04.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.227+0000 7f12d577e740 1 --2- 172.21.15.12:0/3319467430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e4c5d050 0x5623e4c6bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.227+0000 7f12d577e740 1 -- 172.21.15.12:0/3319467430 >> 172.21.15.12:0/3319467430 conn(0x5623e4c25200 msgr2=0x5623e4c57330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:04.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.227+0000 7f12d577e740 1 -- 172.21.15.12:0/3319467430 shutdown_connections 2024-03-23T23:08:04.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.228+0000 7f12d577e740 1 -- 172.21.15.12:0/3319467430 wait complete. 2024-03-23T23:08:04.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.228+0000 7f12d577e740 20 client.26998 trim_cache size 0 max 16384 2024-03-23T23:08:04.240 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:08:04.240 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-03-23T23:08:04.240 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-03-23T23:08:04.240 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:08:04.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.228+0000 7f12d1400640 1 -- 172.21.15.12:0/679539205 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f12b00a2800 msgr2=0x7f12b00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:04.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.228+0000 7f12d1400640 1 --2- 172.21.15.12:0/679539205 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f12b00a2800 0x7f12b00a4cc0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x5623e4c5de40 tx=0x7f12bc009500 comp rx=0 tx=0).stop 2024-03-23T23:08:04.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.228+0000 7f12d1400640 1 -- 172.21.15.12:0/679539205 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12cc0772a0 msgr2=0x7f12cc19e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:04.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.228+0000 7f12d1400640 1 --2- 172.21.15.12:0/679539205 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12cc0772a0 0x7f12cc19e380 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f12b400e9f0 tx=0x7f12b400eeb0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.228+0000 7f12d1400640 1 -- 172.21.15.12:0/679539205 shutdown_connections 2024-03-23T23:08:04.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.228+0000 7f12d1400640 1 --2- 172.21.15.12:0/679539205 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f12b00a2800 0x7f12b00a4cc0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.228+0000 7f12d1400640 1 --2- 172.21.15.12:0/679539205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12cc077c40 0x7f12cc1a53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.228+0000 7f12d1400640 1 --2- 172.21.15.12:0/679539205 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12cc0772a0 0x7f12cc19e380 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.228+0000 7f12d1400640 1 --2- 172.21.15.12:0/679539205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12cc076060 0x7f12cc19de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.228+0000 7f12d1400640 1 -- 172.21.15.12:0/679539205 >> 172.21.15.12:0/679539205 conn(0x7f12cc0fed10 msgr2=0x7f12cc100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:04.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.229+0000 7f12d1400640 1 -- 172.21.15.12:0/679539205 shutdown_connections 2024-03-23T23:08:04.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.229+0000 7f12d1400640 1 -- 172.21.15.12:0/679539205 wait complete. 2024-03-23T23:08:04.245 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T23:08:04.246 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:08:04.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.466+0000 7f91ce000640 1 -- 172.21.15.12:0/918765845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91c8103260 msgr2=0x7f91c8109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:04.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.466+0000 7f91ce000640 1 --2- 172.21.15.12:0/918765845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91c8103260 0x7f91c8109af0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f91c0009be0 tx=0x7f91c002bef0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.466+0000 7f91ce000640 1 -- 172.21.15.12:0/918765845 shutdown_connections 2024-03-23T23:08:04.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.466+0000 7f91ce000640 1 --2- 172.21.15.12:0/918765845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91c8103260 0x7f91c8109af0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.466+0000 7f91ce000640 1 --2- 172.21.15.12:0/918765845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91c81028a0 0x7f91c8102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.466+0000 7f91ce000640 1 --2- 172.21.15.12:0/918765845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91c81016b0 0x7f91c8101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.466+0000 7f91ce000640 1 -- 172.21.15.12:0/918765845 >> 172.21.15.12:0/918765845 conn(0x7f91c80fcc80 msgr2=0x7f91c80ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:04.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.467+0000 7f91ce000640 1 -- 172.21.15.12:0/918765845 shutdown_connections 2024-03-23T23:08:04.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.467+0000 7f91ce000640 1 -- 172.21.15.12:0/918765845 wait complete. 2024-03-23T23:08:04.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.468+0000 7f91ce000640 1 Processor -- start 2024-03-23T23:08:04.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.468+0000 7f91ce000640 1 -- start start 2024-03-23T23:08:04.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.469+0000 7f91ce000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91c81016b0 0x7f91c8199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.469+0000 7f91ce000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91c81028a0 0x7f91c819a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.469+0000 7f91ce000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91c8103260 0x7f91c81a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.469+0000 7f91ce000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c810d380 con 0x7f91c8103260 2024-03-23T23:08:04.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.469+0000 7f91ce000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c810d200 con 0x7f91c81028a0 2024-03-23T23:08:04.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.469+0000 7f91ce000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c810d500 con 0x7f91c81016b0 2024-03-23T23:08:04.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.469+0000 7f91c7400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91c81016b0 0x7f91c8199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.469+0000 7f91c7400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91c81016b0 0x7f91c8199b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44520/0 (socket says 172.21.15.12:44520) 2024-03-23T23:08:04.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.469+0000 7f91c7400640 1 -- 172.21.15.12:0/307805671 learned_addr learned my addr 172.21.15.12:0/307805671 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:04.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.469+0000 7f91c6a00640 1 --2- 172.21.15.12:0/307805671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91c81028a0 0x7f91c819a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.469+0000 7f91c7e00640 1 --2- 172.21.15.12:0/307805671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91c8103260 0x7f91c81a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.469+0000 7f91c7400640 1 -- 172.21.15.12:0/307805671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91c81028a0 msgr2=0x7f91c819a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:04.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.469+0000 7f91c7400640 1 --2- 172.21.15.12:0/307805671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91c81028a0 0x7f91c819a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.469+0000 7f91c7400640 1 -- 172.21.15.12:0/307805671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91c8103260 msgr2=0x7f91c81a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:04.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.469+0000 7f91c7400640 1 --2- 172.21.15.12:0/307805671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91c8103260 0x7f91c81a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.469+0000 7f91c7400640 1 -- 172.21.15.12:0/307805671 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c81a17b0 con 0x7f91c81016b0 2024-03-23T23:08:04.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.469+0000 7f91c7e00640 1 --2- 172.21.15.12:0/307805671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91c8103260 0x7f91c81a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:08:04.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.470+0000 7f91c7400640 1 --2- 172.21.15.12:0/307805671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91c81016b0 0x7f91c8199b90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f91b800e9c0 tx=0x7f91b800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:04.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.470+0000 7f91bfe00640 1 -- 172.21.15.12:0/307805671 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b800cd80 con 0x7f91c81016b0 2024-03-23T23:08:04.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.470+0000 7f91bfe00640 1 -- 172.21.15.12:0/307805671 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f91b80035f0 con 0x7f91c81016b0 2024-03-23T23:08:04.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.470+0000 7f91bfe00640 1 -- 172.21.15.12:0/307805671 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b8010690 con 0x7f91c81016b0 2024-03-23T23:08:04.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.470+0000 7f91ce000640 1 -- 172.21.15.12:0/307805671 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c81a1aa0 con 0x7f91c81016b0 2024-03-23T23:08:04.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.470+0000 7f91ce000640 1 -- 172.21.15.12:0/307805671 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c81a1fa0 con 0x7f91c81016b0 2024-03-23T23:08:04.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.471+0000 7f91ce000640 1 -- 172.21.15.12:0/307805671 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91c81a2230 con 0x7f91c81016b0 2024-03-23T23:08:04.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.473+0000 7f91bfe00640 1 -- 172.21.15.12:0/307805671 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f91b80037a0 con 0x7f91c81016b0 2024-03-23T23:08:04.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.474+0000 7f91bfe00640 1 --2- 172.21.15.12:0/307805671 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f91ac0a2880 0x7f91ac0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.474+0000 7f91bfe00640 1 -- 172.21.15.12:0/307805671 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f91b80cee30 con 0x7f91c81016b0 2024-03-23T23:08:04.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.475+0000 7f91c6a00640 1 --2- 172.21.15.12:0/307805671 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f91ac0a2880 0x7f91ac0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.475+0000 7f91c6a00640 1 --2- 172.21.15.12:0/307805671 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f91ac0a2880 0x7f91ac0a4d40 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f91c819b0a0 tx=0x7f91b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:04.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.478+0000 7f91bfe00640 1 -- 172.21.15.12:0/307805671 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f91b809be80 con 0x7f91c81016b0 2024-03-23T23:08:04.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.722+0000 7f91ce000640 1 -- 172.21.15.12:0/307805671 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91c81a29a0 con 0x7f91c81016b0 2024-03-23T23:08:04.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.723+0000 7f91bfe00640 1 -- 172.21.15.12:0/307805671 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f91b809b780 con 0x7f91c81016b0 2024-03-23T23:08:04.722 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:08:04.722 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:08:04.722 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:08:04.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.726+0000 7f91ce000640 1 -- 172.21.15.12:0/307805671 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f91ac0a2880 msgr2=0x7f91ac0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:04.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.726+0000 7f91ce000640 1 --2- 172.21.15.12:0/307805671 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f91ac0a2880 0x7f91ac0a4d40 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f91c819b0a0 tx=0x7f91b4009260 comp rx=0 tx=0).stop 2024-03-23T23:08:04.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.726+0000 7f91ce000640 1 -- 172.21.15.12:0/307805671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91c81016b0 msgr2=0x7f91c8199b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:04.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.726+0000 7f91ce000640 1 --2- 172.21.15.12:0/307805671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91c81016b0 0x7f91c8199b90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f91b800e9c0 tx=0x7f91b800ee80 comp rx=0 tx=0).stop 2024-03-23T23:08:04.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.726+0000 7f91ce000640 1 -- 172.21.15.12:0/307805671 shutdown_connections 2024-03-23T23:08:04.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.726+0000 7f91ce000640 1 --2- 172.21.15.12:0/307805671 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f91ac0a2880 0x7f91ac0a4d40 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.726+0000 7f91ce000640 1 --2- 172.21.15.12:0/307805671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91c8103260 0x7f91c81a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.726+0000 7f91ce000640 1 --2- 172.21.15.12:0/307805671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91c81028a0 0x7f91c819a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.726+0000 7f91ce000640 1 --2- 172.21.15.12:0/307805671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91c81016b0 0x7f91c8199b90 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.726+0000 7f91ce000640 1 -- 172.21.15.12:0/307805671 >> 172.21.15.12:0/307805671 conn(0x7f91c80fcc80 msgr2=0x7f91c80738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:04.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.726+0000 7f91ce000640 1 -- 172.21.15.12:0/307805671 shutdown_connections 2024-03-23T23:08:04.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.726+0000 7f91ce000640 1 -- 172.21.15.12:0/307805671 wait complete. 2024-03-23T23:08:04.746 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:08:04.746 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-23T23:08:04.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.964+0000 7ffb67e00640 1 -- 172.21.15.12:0/2680730946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb68105350 msgr2=0x7ffb6810bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:04.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.964+0000 7ffb67e00640 1 --2- 172.21.15.12:0/2680730946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb68105350 0x7ffb6810bbe0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ffb60009ca0 tx=0x7ffb6002c100 comp rx=0 tx=0).stop 2024-03-23T23:08:04.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.964+0000 7ffb67e00640 1 -- 172.21.15.12:0/2680730946 shutdown_connections 2024-03-23T23:08:04.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.964+0000 7ffb67e00640 1 --2- 172.21.15.12:0/2680730946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb68105350 0x7ffb6810bbe0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.964+0000 7ffb67e00640 1 --2- 172.21.15.12:0/2680730946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb68104990 0x7ffb68104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.964+0000 7ffb67e00640 1 --2- 172.21.15.12:0/2680730946 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb681037a0 0x7ffb68103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.964+0000 7ffb67e00640 1 -- 172.21.15.12:0/2680730946 >> 172.21.15.12:0/2680730946 conn(0x7ffb680fed10 msgr2=0x7ffb68101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:04.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.964+0000 7ffb67e00640 1 -- 172.21.15.12:0/2680730946 shutdown_connections 2024-03-23T23:08:04.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.964+0000 7ffb67e00640 1 -- 172.21.15.12:0/2680730946 wait complete. 2024-03-23T23:08:04.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.965+0000 7ffb67e00640 1 Processor -- start 2024-03-23T23:08:04.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.966+0000 7ffb67e00640 1 -- start start 2024-03-23T23:08:04.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.966+0000 7ffb67e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb681037a0 0x7ffb68199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.966+0000 7ffb67e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb68104990 0x7ffb68199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.966+0000 7ffb67e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb68105350 0x7ffb681a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.966+0000 7ffb67e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6810f420 con 0x7ffb68104990 2024-03-23T23:08:04.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.966+0000 7ffb67e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6810f2a0 con 0x7ffb68105350 2024-03-23T23:08:04.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.966+0000 7ffb67e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6810f5a0 con 0x7ffb681037a0 2024-03-23T23:08:04.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.966+0000 7ffb66a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb681037a0 0x7ffb68199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.966+0000 7ffb66a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb681037a0 0x7ffb68199a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44540/0 (socket says 172.21.15.12:44540) 2024-03-23T23:08:04.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.966+0000 7ffb66a00640 1 -- 172.21.15.12:0/2465536578 learned_addr learned my addr 172.21.15.12:0/2465536578 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:04.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.966+0000 7ffb66000640 1 --2- 172.21.15.12:0/2465536578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb68104990 0x7ffb68199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.967+0000 7ffb67400640 1 --2- 172.21.15.12:0/2465536578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb68105350 0x7ffb681a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.967+0000 7ffb66a00640 1 -- 172.21.15.12:0/2465536578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb68105350 msgr2=0x7ffb681a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:04.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.967+0000 7ffb66a00640 1 --2- 172.21.15.12:0/2465536578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb68105350 0x7ffb681a0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.967+0000 7ffb66a00640 1 -- 172.21.15.12:0/2465536578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb68104990 msgr2=0x7ffb68199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:04.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.967+0000 7ffb66a00640 1 --2- 172.21.15.12:0/2465536578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb68104990 0x7ffb68199f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.967+0000 7ffb66a00640 1 -- 172.21.15.12:0/2465536578 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb681a1660 con 0x7ffb681037a0 2024-03-23T23:08:04.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.967+0000 7ffb66a00640 1 --2- 172.21.15.12:0/2465536578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb681037a0 0x7ffb68199a40 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ffb5800d950 tx=0x7ffb5800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:04.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.967+0000 7ffb5f400640 1 -- 172.21.15.12:0/2465536578 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb58017020 con 0x7ffb681037a0 2024-03-23T23:08:04.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.967+0000 7ffb5f400640 1 -- 172.21.15.12:0/2465536578 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb58009c30 con 0x7ffb681037a0 2024-03-23T23:08:04.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.967+0000 7ffb5f400640 1 -- 172.21.15.12:0/2465536578 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5800f720 con 0x7ffb681037a0 2024-03-23T23:08:04.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.967+0000 7ffb67e00640 1 -- 172.21.15.12:0/2465536578 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb681a1950 con 0x7ffb681037a0 2024-03-23T23:08:04.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.968+0000 7ffb67e00640 1 -- 172.21.15.12:0/2465536578 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb681a1ef0 con 0x7ffb681037a0 2024-03-23T23:08:04.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.971+0000 7ffb5f400640 1 -- 172.21.15.12:0/2465536578 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ffb5801a020 con 0x7ffb681037a0 2024-03-23T23:08:04.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.971+0000 7ffb70782740 1 Processor -- start 2024-03-23T23:08:04.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.971+0000 7ffb70782740 1 -- start start 2024-03-23T23:08:04.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.971+0000 7ffb70782740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bc404549f0 0x55bc40454e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.971+0000 7ffb70782740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc40455bc0 0x55bc40456040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.971+0000 7ffb70782740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bc40456580 0x55bc4045ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.971+0000 7ffb70782740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc403b92c0 con 0x55bc40456580 2024-03-23T23:08:04.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.971+0000 7ffb70782740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc40361c90 con 0x55bc404549f0 2024-03-23T23:08:04.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.971+0000 7ffb70782740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc403b5360 con 0x55bc40455bc0 2024-03-23T23:08:04.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.972+0000 7ffb66a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc40455bc0 0x55bc40456040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.972+0000 7ffb66a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc40455bc0 0x55bc40456040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44544/0 (socket says 172.21.15.12:44544) 2024-03-23T23:08:04.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.972+0000 7ffb66a00640 1 -- 172.21.15.12:0/189746972 learned_addr learned my addr 172.21.15.12:0/189746972 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:04.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.972+0000 7ffb66000640 1 --2- 172.21.15.12:0/189746972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bc40456580 0x55bc4045ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.972+0000 7ffb67400640 1 --2- 172.21.15.12:0/189746972 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bc404549f0 0x55bc40454e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.972+0000 7ffb66a00640 1 -- 172.21.15.12:0/189746972 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bc404549f0 msgr2=0x55bc40454e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:04.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.972+0000 7ffb66a00640 1 --2- 172.21.15.12:0/189746972 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bc404549f0 0x55bc40454e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.972+0000 7ffb66a00640 1 -- 172.21.15.12:0/189746972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bc40456580 msgr2=0x55bc4045ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:04.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.972+0000 7ffb66a00640 1 --2- 172.21.15.12:0/189746972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bc40456580 0x55bc4045ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.972+0000 7ffb66a00640 1 -- 172.21.15.12:0/189746972 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc40386080 con 0x55bc40455bc0 2024-03-23T23:08:04.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.972+0000 7ffb5f400640 1 --2- 172.21.15.12:0/2465536578 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ffb4c0a2880 0x7ffb4c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.972+0000 7ffb66a00640 1 --2- 172.21.15.12:0/189746972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc40455bc0 0x55bc40456040 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ffb580d4c10 tx=0x7ffb580d6690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c35fd2b282a8acb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:04.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.973+0000 7ffb53e00640 1 -- 172.21.15.12:0/189746972 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb580974e0 con 0x55bc40455bc0 2024-03-23T23:08:04.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.973+0000 7ffb67400640 1 --2- 172.21.15.12:0/2465536578 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ffb4c0a2880 0x7ffb4c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.973+0000 7ffb5f400640 1 -- 172.21.15.12:0/2465536578 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ffb580ce890 con 0x7ffb681037a0 2024-03-23T23:08:04.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.973+0000 7ffb53e00640 1 -- 172.21.15.12:0/189746972 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb5809b940 con 0x55bc40455bc0 2024-03-23T23:08:04.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.973+0000 7ffb53e00640 1 -- 172.21.15.12:0/189746972 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb580c4fe0 con 0x55bc40455bc0 2024-03-23T23:08:04.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.973+0000 7ffb70782740 1 -- 172.21.15.12:0/189746972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc40455bc0 msgr2=0x55bc40456040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:04.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.973+0000 7ffb70782740 1 --2- 172.21.15.12:0/189746972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc40455bc0 0x55bc40456040 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ffb580d4c10 tx=0x7ffb580d6690 comp rx=0 tx=0).stop 2024-03-23T23:08:04.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.974+0000 7ffb67400640 1 --2- 172.21.15.12:0/2465536578 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ffb4c0a2880 0x7ffb4c0a4d40 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ffb6002c640 tx=0x7ffb60032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:04.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.974+0000 7ffb70782740 1 -- 172.21.15.12:0/189746972 shutdown_connections 2024-03-23T23:08:04.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.974+0000 7ffb70782740 1 --2- 172.21.15.12:0/189746972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bc40456580 0x55bc4045ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.974+0000 7ffb70782740 1 --2- 172.21.15.12:0/189746972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc40455bc0 0x55bc40456040 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.974+0000 7ffb70782740 1 --2- 172.21.15.12:0/189746972 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bc404549f0 0x55bc40454e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.974+0000 7ffb70782740 1 -- 172.21.15.12:0/189746972 >> 172.21.15.12:0/189746972 conn(0x55bc4044fde0 msgr2=0x55bc404501e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:04.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.974+0000 7ffb70782740 1 -- 172.21.15.12:0/189746972 shutdown_connections 2024-03-23T23:08:04.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.974+0000 7ffb70782740 1 -- 172.21.15.12:0/189746972 wait complete. 2024-03-23T23:08:04.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.974+0000 7ffb70782740 1 -- start start 2024-03-23T23:08:04.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.974+0000 7ffb70782740 1 Processor -- start 2024-03-23T23:08:04.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.975+0000 7ffb70782740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc404549f0 0x55bc4045eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.975+0000 7ffb70782740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bc40455bc0 0x55bc4045f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.975+0000 7ffb70782740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bc40456580 0x55bc40466100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.975+0000 7ffb70782740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc4045d530 con 0x55bc40456580 2024-03-23T23:08:04.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.975+0000 7ffb70782740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc403750d0 con 0x55bc40455bc0 2024-03-23T23:08:04.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.975+0000 7ffb70782740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc4045d6b0 con 0x55bc404549f0 2024-03-23T23:08:04.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.975+0000 7ffb70782740 10 client.0 _ll_register_callbacks cb 0x55bc40379e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:04.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.975+0000 7ffb66a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc404549f0 0x55bc4045eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.975+0000 7ffb66a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc404549f0 0x55bc4045eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44552/0 (socket says 172.21.15.12:44552) 2024-03-23T23:08:04.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.975+0000 7ffb66000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bc40455bc0 0x55bc4045f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.975+0000 7ffb66a00640 1 -- 172.21.15.12:0/2390458655 learned_addr learned my addr 172.21.15.12:0/2390458655 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:04.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.975+0000 7ffb67400640 1 --2- 172.21.15.12:0/2390458655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bc40456580 0x55bc40466100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.976+0000 7ffb66a00640 1 -- 172.21.15.12:0/2390458655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bc40455bc0 msgr2=0x55bc4045f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:04.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.976+0000 7ffb66a00640 1 --2- 172.21.15.12:0/2390458655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bc40455bc0 0x55bc4045f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.976+0000 7ffb66a00640 1 -- 172.21.15.12:0/2390458655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bc40456580 msgr2=0x55bc40466100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:04.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.976+0000 7ffb66a00640 1 --2- 172.21.15.12:0/2390458655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bc40456580 0x55bc40466100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.976+0000 7ffb66a00640 1 -- 172.21.15.12:0/2390458655 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bc400cd3f0 con 0x55bc404549f0 2024-03-23T23:08:04.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.976+0000 7ffb66000640 1 --2- 172.21.15.12:0/2390458655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bc40455bc0 0x55bc4045f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:04.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.976+0000 7ffb66a00640 1 --2- 172.21.15.12:0/2390458655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc404549f0 0x55bc4045eb80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ffb580cc710 tx=0x7ffb58004700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a791b6e000b635d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:04.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.976+0000 7ffb50c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:08:04.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.976+0000 7ffb50c00640 1 -- 172.21.15.12:0/2390458655 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb580c51b0 con 0x55bc404549f0 2024-03-23T23:08:04.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.976+0000 7ffb50c00640 1 -- 172.21.15.12:0/2390458655 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ffb58099540 con 0x55bc404549f0 2024-03-23T23:08:04.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.976+0000 7ffb70782740 1 -- 172.21.15.12:0/2390458655 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb5809b940 con 0x55bc404549f0 2024-03-23T23:08:04.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.976+0000 7ffb50c00640 1 -- 172.21.15.12:0/2390458655 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb580c2cc0 con 0x55bc404549f0 2024-03-23T23:08:04.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.976+0000 7ffb70782740 10 client.27016 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:04.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.976+0000 7ffb70782740 1 -- 172.21.15.12:0/2390458655 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55bc401977d0 con 0x55bc404549f0 2024-03-23T23:08:04.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.977+0000 7ffb50c00640 1 -- 172.21.15.12:0/2390458655 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb5809b940 con 0x55bc404549f0 2024-03-23T23:08:04.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.977+0000 7ffb50c00640 1 -- 172.21.15.12:0/2390458655 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ffb580bb080 con 0x55bc404549f0 2024-03-23T23:08:04.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.977+0000 7ffb70782740 10 client.27016 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:04.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.977+0000 7ffb70782740 10 client.27016 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:08:04.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.977+0000 7ffb70782740 20 client.27016 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:04.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.977+0000 7ffb70782740 1 --2- 172.21.15.12:0/2390458655 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55bc4046a260 0x55bc4046c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.977+0000 7ffb70782740 4 client.27016 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:08:04.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.977+0000 7ffb70782740 1 -- 172.21.15.12:0/2390458655 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bc403c6800 con 0x55bc4046a260 2024-03-23T23:08:04.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.977+0000 7ffb66000640 1 --2- 172.21.15.12:0/2390458655 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55bc4046a260 0x55bc4046c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.978+0000 7ffb66000640 1 --2- 172.21.15.12:0/2390458655 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55bc4046a260 0x55bc4046c650 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=33886f23365002ec server_cookie=6ca59bc2269503c4 in_seq=0 out_seq=0 2024-03-23T23:08:04.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.978+0000 7ffb50c00640 10 client.27016 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:08:04.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.978+0000 7ffb50c00640 1 -- 172.21.15.12:0/2390458655 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55bc403c6800 con 0x55bc4046a260 2024-03-23T23:08:04.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.978+0000 7ffb50c00640 10 client.27016 handle_command_reply: tid=0 2024-03-23T23:08:04.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.979+0000 7ffb70782740 1 client.27016 shutdown 2024-03-23T23:08:04.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.979+0000 7ffb70782740 1 -- 172.21.15.12:0/2390458655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc404549f0 msgr2=0x55bc4045eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:04.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.979+0000 7ffb70782740 1 --2- 172.21.15.12:0/2390458655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc404549f0 0x55bc4045eb80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ffb580cc710 tx=0x7ffb58004700 comp rx=0 tx=0).stop 2024-03-23T23:08:04.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.979+0000 7ffb70782740 1 -- 172.21.15.12:0/2390458655 shutdown_connections 2024-03-23T23:08:04.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.979+0000 7ffb70782740 1 --2- 172.21.15.12:0/2390458655 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55bc4046a260 0x55bc4046c650 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.979+0000 7ffb70782740 1 --2- 172.21.15.12:0/2390458655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bc40456580 0x55bc40466100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.979+0000 7ffb50c00640 0 client.27016 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:08:04.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.979+0000 7ffb70782740 1 --2- 172.21.15.12:0/2390458655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bc40455bc0 0x55bc4045f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.979+0000 7ffb70782740 1 --2- 172.21.15.12:0/2390458655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc404549f0 0x55bc4045eb80 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.979+0000 7ffb70782740 1 -- 172.21.15.12:0/2390458655 >> 172.21.15.12:0/2390458655 conn(0x55bc4044fde0 msgr2=0x55bc4044f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:04.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.979+0000 7ffb70782740 1 -- 172.21.15.12:0/2390458655 shutdown_connections 2024-03-23T23:08:04.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.979+0000 7ffb70782740 1 -- 172.21.15.12:0/2390458655 wait complete. 2024-03-23T23:08:04.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.979+0000 7ffb70782740 20 client.27016 trim_cache size 0 max 16384 2024-03-23T23:08:04.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.993+0000 7ffb70782740 1 Processor -- start 2024-03-23T23:08:04.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.993+0000 7ffb70782740 1 -- start start 2024-03-23T23:08:04.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.993+0000 7ffb70782740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc404549f0 0x55bc40458bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.993+0000 7ffb70782740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bc40455bc0 0x55bc40466090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.993+0000 7ffb70782740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bc40456580 0x55bc40459110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:04.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.993+0000 7ffb70782740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb681acfc0 con 0x55bc40456580 2024-03-23T23:08:04.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.993+0000 7ffb70782740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb681aaac0 con 0x55bc40455bc0 2024-03-23T23:08:04.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.993+0000 7ffb70782740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb681a8b60 con 0x55bc404549f0 2024-03-23T23:08:04.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.994+0000 7ffb66a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc404549f0 0x55bc40458bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.994+0000 7ffb66000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bc40455bc0 0x55bc40466090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.994+0000 7ffb66a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc404549f0 0x55bc40458bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44560/0 (socket says 172.21.15.12:44560) 2024-03-23T23:08:04.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.994+0000 7ffb66000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bc40455bc0 0x55bc40466090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44876/0 (socket says 172.21.15.12:44876) 2024-03-23T23:08:04.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.994+0000 7ffb66a00640 1 -- 172.21.15.12:0/673490379 learned_addr learned my addr 172.21.15.12:0/673490379 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:04.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.994+0000 7ffb67400640 1 --2- 172.21.15.12:0/673490379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bc40456580 0x55bc40459110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:04.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.994+0000 7ffb66a00640 1 -- 172.21.15.12:0/673490379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bc40455bc0 msgr2=0x55bc40466090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:04.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.994+0000 7ffb66a00640 1 --2- 172.21.15.12:0/673490379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bc40455bc0 0x55bc40466090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.994+0000 7ffb66a00640 1 -- 172.21.15.12:0/673490379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bc40456580 msgr2=0x55bc40459110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:04.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.994+0000 7ffb66a00640 1 --2- 172.21.15.12:0/673490379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bc40456580 0x55bc40459110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.994+0000 7ffb66a00640 1 -- 172.21.15.12:0/673490379 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc4045c6e0 con 0x55bc404549f0 2024-03-23T23:08:04.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.994+0000 7ffb66000640 1 --2- 172.21.15.12:0/673490379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bc40455bc0 0x55bc40466090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:04.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.994+0000 7ffb66a00640 1 --2- 172.21.15.12:0/673490379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc404549f0 0x55bc40458bd0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ffb580d4c10 tx=0x7ffb580d4550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5b738d9e91116aa server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:04.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.995+0000 7ffb53400640 1 -- 172.21.15.12:0/673490379 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb580c2e90 con 0x55bc404549f0 2024-03-23T23:08:04.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.995+0000 7ffb53400640 1 -- 172.21.15.12:0/673490379 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb5809b940 con 0x55bc404549f0 2024-03-23T23:08:04.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.995+0000 7ffb53400640 1 -- 172.21.15.12:0/673490379 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5809aba0 con 0x55bc404549f0 2024-03-23T23:08:04.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.995+0000 7ffb70782740 1 -- 172.21.15.12:0/673490379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc404549f0 msgr2=0x55bc40458bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:04.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.995+0000 7ffb70782740 1 --2- 172.21.15.12:0/673490379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc404549f0 0x55bc40458bd0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ffb580d4c10 tx=0x7ffb580d4550 comp rx=0 tx=0).stop 2024-03-23T23:08:04.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.995+0000 7ffb70782740 1 -- 172.21.15.12:0/673490379 shutdown_connections 2024-03-23T23:08:04.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.995+0000 7ffb70782740 1 --2- 172.21.15.12:0/673490379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bc40456580 0x55bc40459110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.995+0000 7ffb70782740 1 --2- 172.21.15.12:0/673490379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bc40455bc0 0x55bc40466090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.995+0000 7ffb70782740 1 --2- 172.21.15.12:0/673490379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc404549f0 0x55bc40458bd0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:04.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.995+0000 7ffb70782740 1 -- 172.21.15.12:0/673490379 >> 172.21.15.12:0/673490379 conn(0x55bc4044fde0 msgr2=0x55bc404538d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:04.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.996+0000 7ffb70782740 1 -- 172.21.15.12:0/673490379 shutdown_connections 2024-03-23T23:08:04.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.996+0000 7ffb70782740 1 -- 172.21.15.12:0/673490379 wait complete. 2024-03-23T23:08:04.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.996+0000 7ffb70782740 1 -- start start 2024-03-23T23:08:04.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.996+0000 7ffb70782740 1 Processor -- start 2024-03-23T23:08:05.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.996+0000 7ffb70782740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc404549f0 0x55bc40465860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.997+0000 7ffb70782740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bc40455bc0 0x55bc40465da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.997+0000 7ffb70782740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bc40456580 0x55bc40462370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.997+0000 7ffb70782740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb681a9160 con 0x55bc40456580 2024-03-23T23:08:05.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.997+0000 7ffb70782740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb681aa460 con 0x55bc40455bc0 2024-03-23T23:08:05.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.997+0000 7ffb70782740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb681a97a0 con 0x55bc404549f0 2024-03-23T23:08:05.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.997+0000 7ffb70782740 10 client.0 _ll_register_callbacks cb 0x7ffb54009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:05.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.997+0000 7ffb66a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc404549f0 0x55bc40465860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.997+0000 7ffb66a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc404549f0 0x55bc40465860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44574/0 (socket says 172.21.15.12:44574) 2024-03-23T23:08:05.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.997+0000 7ffb66a00640 1 -- 172.21.15.12:0/1090688692 learned_addr learned my addr 172.21.15.12:0/1090688692 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:05.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.997+0000 7ffb67400640 1 --2- 172.21.15.12:0/1090688692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bc40456580 0x55bc40462370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.997+0000 7ffb66000640 1 --2- 172.21.15.12:0/1090688692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bc40455bc0 0x55bc40465da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.997+0000 7ffb66a00640 1 -- 172.21.15.12:0/1090688692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bc40455bc0 msgr2=0x55bc40465da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:05.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.997+0000 7ffb66a00640 1 --2- 172.21.15.12:0/1090688692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bc40455bc0 0x55bc40465da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.997+0000 7ffb66a00640 1 -- 172.21.15.12:0/1090688692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bc40456580 msgr2=0x55bc40462370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:05.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.997+0000 7ffb66a00640 1 --2- 172.21.15.12:0/1090688692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bc40456580 0x55bc40462370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.997+0000 7ffb66a00640 1 -- 172.21.15.12:0/1090688692 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bc404629b0 con 0x55bc404549f0 2024-03-23T23:08:05.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.997+0000 7ffb66a00640 1 --2- 172.21.15.12:0/1090688692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc404549f0 0x55bc40465860 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ffb580ced10 tx=0x7ffb58004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=141391cbd143df37 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:05.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.997+0000 7ffb50c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:08:05.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.998+0000 7ffb50c00640 1 -- 172.21.15.12:0/1090688692 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb580c2e60 con 0x55bc404549f0 2024-03-23T23:08:05.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.998+0000 7ffb50c00640 1 -- 172.21.15.12:0/1090688692 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ffb580bb0b0 con 0x55bc404549f0 2024-03-23T23:08:05.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.998+0000 7ffb70782740 1 -- 172.21.15.12:0/1090688692 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb5809b940 con 0x55bc404549f0 2024-03-23T23:08:05.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.998+0000 7ffb50c00640 1 -- 172.21.15.12:0/1090688692 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb5809ad80 con 0x55bc404549f0 2024-03-23T23:08:05.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.998+0000 7ffb70782740 10 client.27022 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:05.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.998+0000 7ffb70782740 1 -- 172.21.15.12:0/1090688692 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55bc404629b0 con 0x55bc404549f0 2024-03-23T23:08:05.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.998+0000 7ffb50c00640 1 -- 172.21.15.12:0/1090688692 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb5809b940 con 0x55bc404549f0 2024-03-23T23:08:05.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.998+0000 7ffb50c00640 1 -- 172.21.15.12:0/1090688692 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ffb580a3550 con 0x55bc404549f0 2024-03-23T23:08:05.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.998+0000 7ffb70782740 10 client.27022 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:05.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.998+0000 7ffb70782740 10 client.27022 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:08:05.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.998+0000 7ffb70782740 20 client.27022 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:05.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.998+0000 7ffb70782740 1 --2- 172.21.15.12:0/1090688692 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55bc404c69e0 0x55bc404c8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.998+0000 7ffb70782740 4 client.27022 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:08:05.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.998+0000 7ffb70782740 1 -- 172.21.15.12:0/1090688692 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55bc403c0b50 con 0x55bc404c69e0 2024-03-23T23:08:05.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.999+0000 7ffb66000640 1 --2- 172.21.15.12:0/1090688692 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55bc404c69e0 0x55bc404c8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.999+0000 7ffb66000640 1 --2- 172.21.15.12:0/1090688692 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55bc404c69e0 0x55bc404c8e00 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ee824d919f5fa758 server_cookie=b898abda7c154733 in_seq=0 out_seq=0 2024-03-23T23:08:05.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.999+0000 7ffb50c00640 10 client.27022 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:08:05.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.999+0000 7ffb50c00640 1 -- 172.21.15.12:0/1090688692 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55bc403c0b50 con 0x55bc404c69e0 2024-03-23T23:08:05.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.999+0000 7ffb50c00640 10 client.27022 handle_command_reply: tid=0 2024-03-23T23:08:05.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.999+0000 7ffb70782740 1 client.27022 shutdown 2024-03-23T23:08:05.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.999+0000 7ffb70782740 1 -- 172.21.15.12:0/1090688692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc404549f0 msgr2=0x55bc40465860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:05.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:04.999+0000 7ffb70782740 1 --2- 172.21.15.12:0/1090688692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc404549f0 0x55bc40465860 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ffb580ced10 tx=0x7ffb58004730 comp rx=0 tx=0).stop 2024-03-23T23:08:05.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.000+0000 7ffb70782740 1 -- 172.21.15.12:0/1090688692 shutdown_connections 2024-03-23T23:08:05.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.000+0000 7ffb70782740 1 --2- 172.21.15.12:0/1090688692 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55bc404c69e0 0x55bc404c8e00 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.000+0000 7ffb70782740 1 --2- 172.21.15.12:0/1090688692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bc40456580 0x55bc40462370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.000+0000 7ffb70782740 1 --2- 172.21.15.12:0/1090688692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bc40455bc0 0x55bc40465da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.000+0000 7ffb50c00640 0 client.27022 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:08:05.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.000+0000 7ffb70782740 1 --2- 172.21.15.12:0/1090688692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc404549f0 0x55bc40465860 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.000+0000 7ffb70782740 1 -- 172.21.15.12:0/1090688692 >> 172.21.15.12:0/1090688692 conn(0x55bc4044fde0 msgr2=0x55bc4044e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:05.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.000+0000 7ffb70782740 1 -- 172.21.15.12:0/1090688692 shutdown_connections 2024-03-23T23:08:05.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.000+0000 7ffb70782740 1 -- 172.21.15.12:0/1090688692 wait complete. 2024-03-23T23:08:05.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.000+0000 7ffb70782740 20 client.27022 trim_cache size 0 max 16384 2024-03-23T23:08:05.004 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:08:05.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.000+0000 7ffb67e00640 1 -- 172.21.15.12:0/2465536578 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ffb4c0a2880 msgr2=0x7ffb4c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:05.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.000+0000 7ffb67e00640 1 --2- 172.21.15.12:0/2465536578 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ffb4c0a2880 0x7ffb4c0a4d40 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ffb6002c640 tx=0x7ffb60032590 comp rx=0 tx=0).stop 2024-03-23T23:08:05.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.000+0000 7ffb67e00640 1 -- 172.21.15.12:0/2465536578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb681037a0 msgr2=0x7ffb68199a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:05.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.000+0000 7ffb67e00640 1 --2- 172.21.15.12:0/2465536578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb681037a0 0x7ffb68199a40 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ffb5800d950 tx=0x7ffb5800de10 comp rx=0 tx=0).stop 2024-03-23T23:08:05.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.001+0000 7ffb67e00640 1 -- 172.21.15.12:0/2465536578 shutdown_connections 2024-03-23T23:08:05.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.001+0000 7ffb67e00640 1 --2- 172.21.15.12:0/2465536578 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ffb4c0a2880 0x7ffb4c0a4d40 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.001+0000 7ffb67e00640 1 --2- 172.21.15.12:0/2465536578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb68105350 0x7ffb681a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.001+0000 7ffb67e00640 1 --2- 172.21.15.12:0/2465536578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb68104990 0x7ffb68199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.001+0000 7ffb67e00640 1 --2- 172.21.15.12:0/2465536578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb681037a0 0x7ffb68199a40 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.001+0000 7ffb67e00640 1 -- 172.21.15.12:0/2465536578 >> 172.21.15.12:0/2465536578 conn(0x7ffb680fed10 msgr2=0x7ffb68107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:05.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.001+0000 7ffb67e00640 1 -- 172.21.15.12:0/2465536578 shutdown_connections 2024-03-23T23:08:05.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.001+0000 7ffb67e00640 1 -- 172.21.15.12:0/2465536578 wait complete. 2024-03-23T23:08:05.015 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:04 smithi012 ceph-mon[31695]: pgmap v578: 97 pgs: 97 active+clean; 4.0 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:08:05.015 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:04 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/307805671' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:08:05.018 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:08:05.018 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-23T23:08:05.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:04 smithi173 ceph-mon[36116]: pgmap v578: 97 pgs: 97 active+clean; 4.0 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:08:05.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:04 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/307805671' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:08:05.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:04 smithi203 ceph-mon[36693]: pgmap v578: 97 pgs: 97 active+clean; 4.0 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:08:05.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:04 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/307805671' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:08:05.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.242+0000 7f8fe5000640 1 -- 172.21.15.12:0/2900903421 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fe0074080 msgr2=0x7f8fe00744e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:05.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.242+0000 7f8fe5000640 1 --2- 172.21.15.12:0/2900903421 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fe0074080 0x7f8fe00744e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc009900 tx=0x7f8fcc02bcc0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.242+0000 7f8fe5000640 1 -- 172.21.15.12:0/2900903421 shutdown_connections 2024-03-23T23:08:05.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.242+0000 7f8fe5000640 1 --2- 172.21.15.12:0/2900903421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fe0109430 0x7f8fe010b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.242+0000 7f8fe5000640 1 --2- 172.21.15.12:0/2900903421 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fe0074080 0x7f8fe00744e0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.242+0000 7f8fe5000640 1 --2- 172.21.15.12:0/2900903421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8fe0075660 0x7f8fe0073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.242+0000 7f8fe5000640 1 -- 172.21.15.12:0/2900903421 >> 172.21.15.12:0/2900903421 conn(0x7f8fe00fcb10 msgr2=0x7f8fe00fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:05.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.242+0000 7f8fe5000640 1 -- 172.21.15.12:0/2900903421 shutdown_connections 2024-03-23T23:08:05.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.242+0000 7f8fe5000640 1 -- 172.21.15.12:0/2900903421 wait complete. 2024-03-23T23:08:05.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.243+0000 7f8fe5000640 1 Processor -- start 2024-03-23T23:08:05.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.243+0000 7f8fe5000640 1 -- start start 2024-03-23T23:08:05.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.244+0000 7f8fe5000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fe0074080 0x7f8fe01999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.244+0000 7f8fe5000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8fe0075660 0x7f8fe0199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.244+0000 7f8fe5000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fe0109430 0x7f8fe01a0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.244+0000 7f8fe5000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe010f430 con 0x7f8fe0075660 2024-03-23T23:08:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.244+0000 7f8fe5000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe010f2b0 con 0x7f8fe0074080 2024-03-23T23:08:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.244+0000 7f8fe5000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe010f5b0 con 0x7f8fe0109430 2024-03-23T23:08:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.244+0000 7f8fdf400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fe0074080 0x7f8fe01999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.244+0000 7f8fdfe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fe0109430 0x7f8fe01a0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.244+0000 7f8fdea00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8fe0075660 0x7f8fe0199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.244+0000 7f8fdf400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fe0074080 0x7f8fe01999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44902/0 (socket says 172.21.15.12:44902) 2024-03-23T23:08:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.244+0000 7f8fdf400640 1 -- 172.21.15.12:0/2274094623 learned_addr learned my addr 172.21.15.12:0/2274094623 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.245+0000 7f8fdf400640 1 -- 172.21.15.12:0/2274094623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fe0109430 msgr2=0x7f8fe01a0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.245+0000 7f8fdf400640 1 --2- 172.21.15.12:0/2274094623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fe0109430 0x7f8fe01a0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.245+0000 7f8fdf400640 1 -- 172.21.15.12:0/2274094623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8fe0075660 msgr2=0x7f8fe0199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.245+0000 7f8fdf400640 1 --2- 172.21.15.12:0/2274094623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8fe0075660 0x7f8fe0199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.245+0000 7f8fdf400640 1 -- 172.21.15.12:0/2274094623 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe01a15e0 con 0x7f8fe0074080 2024-03-23T23:08:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.245+0000 7f8fdfe00640 1 --2- 172.21.15.12:0/2274094623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fe0109430 0x7f8fe01a0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.245+0000 7f8fdf400640 1 --2- 172.21.15.12:0/2274094623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fe0074080 0x7f8fe01999c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f8fd000c920 tx=0x7f8fd000cde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:05.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.245+0000 7f8fd7e00640 1 -- 172.21.15.12:0/2274094623 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0007c70 con 0x7f8fe0074080 2024-03-23T23:08:05.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.245+0000 7f8fd7e00640 1 -- 172.21.15.12:0/2274094623 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8fd0007e10 con 0x7f8fe0074080 2024-03-23T23:08:05.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.245+0000 7f8fd7e00640 1 -- 172.21.15.12:0/2274094623 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0018890 con 0x7f8fe0074080 2024-03-23T23:08:05.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.245+0000 7f8fe5000640 1 -- 172.21.15.12:0/2274094623 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe01a18d0 con 0x7f8fe0074080 2024-03-23T23:08:05.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.245+0000 7f8fe5000640 1 -- 172.21.15.12:0/2274094623 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe01a1da0 con 0x7f8fe0074080 2024-03-23T23:08:05.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.248+0000 7f8fd7e00640 1 -- 172.21.15.12:0/2274094623 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8fd0018a70 con 0x7f8fe0074080 2024-03-23T23:08:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.249+0000 7f8fe9399740 1 Processor -- start 2024-03-23T23:08:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.249+0000 7f8fe9399740 1 -- start start 2024-03-23T23:08:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.249+0000 7f8fe9399740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5c220d9f0 0x55f5c220de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.250+0000 7f8fe9399740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5c220ebc0 0x55f5c220f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.250+0000 7f8fe9399740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220f580 0x55f5c2215e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.250+0000 7f8fe9399740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5c21722c0 con 0x55f5c220d9f0 2024-03-23T23:08:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.250+0000 7f8fe9399740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5c211ac90 con 0x55f5c220ebc0 2024-03-23T23:08:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.250+0000 7f8fe9399740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5c216e360 con 0x55f5c220f580 2024-03-23T23:08:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.250+0000 7f8fdf400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5c220ebc0 0x55f5c220f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.250+0000 7f8fdea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220f580 0x55f5c2215e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.250+0000 7f8fdfe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5c220d9f0 0x55f5c220de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.250+0000 7f8fdea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220f580 0x55f5c2215e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44600/0 (socket says 172.21.15.12:44600) 2024-03-23T23:08:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.250+0000 7f8fdea00640 1 -- 172.21.15.12:0/306176779 learned_addr learned my addr 172.21.15.12:0/306176779 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:05.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.250+0000 7f8fdea00640 1 -- 172.21.15.12:0/306176779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5c220ebc0 msgr2=0x55f5c220f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:05.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.250+0000 7f8fdea00640 1 --2- 172.21.15.12:0/306176779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5c220ebc0 0x55f5c220f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.250+0000 7f8fdea00640 1 -- 172.21.15.12:0/306176779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5c220d9f0 msgr2=0x55f5c220de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:05.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.250+0000 7f8fdea00640 1 --2- 172.21.15.12:0/306176779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5c220d9f0 0x55f5c220de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.250+0000 7f8fdea00640 1 -- 172.21.15.12:0/306176779 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5c213f080 con 0x55f5c220f580 2024-03-23T23:08:05.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.251+0000 7f8fd7e00640 1 --2- 172.21.15.12:0/2274094623 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8fc40a2880 0x7f8fc40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.251+0000 7f8fdea00640 1 --2- 172.21.15.12:0/306176779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220f580 0x55f5c2215e10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc000c00 tx=0x7f8fcc005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2458ddd1fb6e387 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:05.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.251+0000 7f8fd7e00640 1 -- 172.21.15.12:0/2274094623 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8fd00ceab0 con 0x7f8fe0074080 2024-03-23T23:08:05.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.251+0000 7f8fd4c00640 1 -- 172.21.15.12:0/306176779 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc043070 con 0x55f5c220f580 2024-03-23T23:08:05.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.251+0000 7f8fdfe00640 1 --2- 172.21.15.12:0/2274094623 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8fc40a2880 0x7f8fc40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.251+0000 7f8fd4c00640 1 -- 172.21.15.12:0/306176779 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8fcc02cb50 con 0x55f5c220f580 2024-03-23T23:08:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.251+0000 7f8fd4c00640 1 -- 172.21.15.12:0/306176779 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc004f10 con 0x55f5c220f580 2024-03-23T23:08:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.252+0000 7f8fe9399740 1 -- 172.21.15.12:0/306176779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220f580 msgr2=0x55f5c2215e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.252+0000 7f8fe9399740 1 --2- 172.21.15.12:0/306176779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220f580 0x55f5c2215e10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc000c00 tx=0x7f8fcc005d50 comp rx=0 tx=0).stop 2024-03-23T23:08:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.252+0000 7f8fdfe00640 1 --2- 172.21.15.12:0/2274094623 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8fc40a2880 0x7f8fc40a4d40 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x55f5c220eac0 tx=0x7f8fd8009600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.252+0000 7f8fe9399740 1 -- 172.21.15.12:0/306176779 shutdown_connections 2024-03-23T23:08:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.252+0000 7f8fe9399740 1 --2- 172.21.15.12:0/306176779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220f580 0x55f5c2215e10 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.252+0000 7f8fe9399740 1 --2- 172.21.15.12:0/306176779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5c220ebc0 0x55f5c220f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.252+0000 7f8fe9399740 1 --2- 172.21.15.12:0/306176779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5c220d9f0 0x55f5c220de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.252+0000 7f8fe9399740 1 -- 172.21.15.12:0/306176779 >> 172.21.15.12:0/306176779 conn(0x55f5c2208de0 msgr2=0x55f5c22091e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.252+0000 7f8fe9399740 1 -- 172.21.15.12:0/306176779 shutdown_connections 2024-03-23T23:08:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.252+0000 7f8fe9399740 1 -- 172.21.15.12:0/306176779 wait complete. 2024-03-23T23:08:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.253+0000 7f8fe9399740 1 -- start start 2024-03-23T23:08:05.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.253+0000 7f8fe9399740 1 Processor -- start 2024-03-23T23:08:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.253+0000 7f8fe9399740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220d9f0 0x55f5c2217c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.253+0000 7f8fe9399740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5c220ebc0 0x55f5c2218140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.253+0000 7f8fe9399740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5c220f580 0x55f5c221f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.253+0000 7f8fe9399740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5c2216530 con 0x55f5c220f580 2024-03-23T23:08:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.253+0000 7f8fe9399740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5c212e0d0 con 0x55f5c220ebc0 2024-03-23T23:08:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.253+0000 7f8fe9399740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5c22166b0 con 0x55f5c220d9f0 2024-03-23T23:08:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.254+0000 7f8fe9399740 10 client.0 _ll_register_callbacks cb 0x55f5c2132e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.254+0000 7f8fdf400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220d9f0 0x55f5c2217c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.254+0000 7f8fdf400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220d9f0 0x55f5c2217c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44604/0 (socket says 172.21.15.12:44604) 2024-03-23T23:08:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.254+0000 7f8fdf400640 1 -- 172.21.15.12:0/2601248131 learned_addr learned my addr 172.21.15.12:0/2601248131 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.254+0000 7f8fdea00640 1 --2- 172.21.15.12:0/2601248131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5c220ebc0 0x55f5c2218140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.254+0000 7f8fdfe00640 1 --2- 172.21.15.12:0/2601248131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5c220f580 0x55f5c221f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.254+0000 7f8fdf400640 1 -- 172.21.15.12:0/2601248131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5c220ebc0 msgr2=0x55f5c2218140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.254+0000 7f8fdf400640 1 --2- 172.21.15.12:0/2601248131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5c220ebc0 0x55f5c2218140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.254+0000 7f8fdf400640 1 -- 172.21.15.12:0/2601248131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5c220f580 msgr2=0x55f5c221f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.254+0000 7f8fdf400640 1 --2- 172.21.15.12:0/2601248131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5c220f580 0x55f5c221f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.254+0000 7f8fdf400640 1 -- 172.21.15.12:0/2601248131 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f5c1e863f0 con 0x55f5c220d9f0 2024-03-23T23:08:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.254+0000 7f8fdea00640 1 --2- 172.21.15.12:0/2601248131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5c220ebc0 0x55f5c2218140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.254+0000 7f8fdf400640 1 --2- 172.21.15.12:0/2601248131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220d9f0 0x55f5c2217c00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0097860 tx=0x7f8fd000a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=624ce3a9c2bb3892 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.255+0000 7f8fc9600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:08:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.255+0000 7f8fc9600640 1 -- 172.21.15.12:0/2601248131 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd009ad60 con 0x55f5c220d9f0 2024-03-23T23:08:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.255+0000 7f8fc9600640 1 -- 172.21.15.12:0/2601248131 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8fd000fb30 con 0x55f5c220d9f0 2024-03-23T23:08:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.255+0000 7f8fe9399740 1 -- 172.21.15.12:0/2601248131 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8fcc02cb50 con 0x55f5c220d9f0 2024-03-23T23:08:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.255+0000 7f8fc9600640 1 -- 172.21.15.12:0/2601248131 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8fd009bb00 con 0x55f5c220d9f0 2024-03-23T23:08:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.255+0000 7f8fe9399740 10 client.27034 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.255+0000 7f8fe9399740 1 -- 172.21.15.12:0/2601248131 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f5c1f507d0 con 0x55f5c220d9f0 2024-03-23T23:08:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.255+0000 7f8fc9600640 1 -- 172.21.15.12:0/2601248131 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8fcc02cb50 con 0x55f5c220d9f0 2024-03-23T23:08:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.256+0000 7f8fc9600640 1 -- 172.21.15.12:0/2601248131 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8fd009f080 con 0x55f5c220d9f0 2024-03-23T23:08:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.256+0000 7f8fe9399740 10 client.27034 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.256+0000 7f8fe9399740 10 client.27034 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:08:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.256+0000 7f8fe9399740 20 client.27034 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.256+0000 7f8fe9399740 1 --2- 172.21.15.12:0/2601248131 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55f5c2223620 0x55f5c2225a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.256+0000 7f8fe9399740 4 client.27034 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:08:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.256+0000 7f8fe9399740 1 -- 172.21.15.12:0/2601248131 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f5c217f800 con 0x55f5c2223620 2024-03-23T23:08:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.256+0000 7f8fdea00640 1 --2- 172.21.15.12:0/2601248131 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55f5c2223620 0x55f5c2225a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.257+0000 7f8fdea00640 1 --2- 172.21.15.12:0/2601248131 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55f5c2223620 0x55f5c2225a70 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ba147be44554899d server_cookie=8370a9cfa57896d9 in_seq=0 out_seq=0 2024-03-23T23:08:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.257+0000 7f8fc9600640 10 client.27034 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:08:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.258+0000 7f8fc9600640 1 -- 172.21.15.12:0/2601248131 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f5c217f800 con 0x55f5c2223620 2024-03-23T23:08:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.258+0000 7f8fc9600640 10 client.27034 handle_command_reply: tid=0 2024-03-23T23:08:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.258+0000 7f8fe9399740 1 client.27034 shutdown 2024-03-23T23:08:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.258+0000 7f8fe9399740 1 -- 172.21.15.12:0/2601248131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220d9f0 msgr2=0x55f5c2217c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.258+0000 7f8fe9399740 1 --2- 172.21.15.12:0/2601248131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220d9f0 0x55f5c2217c00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0097860 tx=0x7f8fd000a620 comp rx=0 tx=0).stop 2024-03-23T23:08:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.258+0000 7f8fe9399740 1 -- 172.21.15.12:0/2601248131 shutdown_connections 2024-03-23T23:08:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.258+0000 7f8fe9399740 1 --2- 172.21.15.12:0/2601248131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5c220f580 0x55f5c221f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.258+0000 7f8fe9399740 1 --2- 172.21.15.12:0/2601248131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5c220ebc0 0x55f5c2218140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.258+0000 7f8fe9399740 1 --2- 172.21.15.12:0/2601248131 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55f5c2223620 0x55f5c2225a70 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.258+0000 7f8fe9399740 1 --2- 172.21.15.12:0/2601248131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220d9f0 0x55f5c2217c00 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.258+0000 7f8fe9399740 1 -- 172.21.15.12:0/2601248131 >> 172.21.15.12:0/2601248131 conn(0x55f5c2208de0 msgr2=0x55f5c2208160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.258+0000 7f8fc9600640 0 client.27034 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:08:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.258+0000 7f8fe9399740 1 -- 172.21.15.12:0/2601248131 shutdown_connections 2024-03-23T23:08:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.258+0000 7f8fe9399740 1 -- 172.21.15.12:0/2601248131 wait complete. 2024-03-23T23:08:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.258+0000 7f8fe9399740 20 client.27034 trim_cache size 0 max 16384 2024-03-23T23:08:05.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.272+0000 7f8fe9399740 1 Processor -- start 2024-03-23T23:08:05.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.272+0000 7f8fe9399740 1 -- start start 2024-03-23T23:08:05.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.272+0000 7f8fe9399740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5c220d9f0 0x55f5c2211c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.272+0000 7f8fe9399740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5c220ebc0 0x55f5c221f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.273+0000 7f8fe9399740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220f580 0x55f5c22121a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.273+0000 7f8fe9399740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe01ace00 con 0x55f5c220ebc0 2024-03-23T23:08:05.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.273+0000 7f8fe9399740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe01aa900 con 0x55f5c220d9f0 2024-03-23T23:08:05.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.273+0000 7f8fe9399740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe01a89a0 con 0x55f5c220f580 2024-03-23T23:08:05.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.273+0000 7f8fdea00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5c220ebc0 0x55f5c221f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.273+0000 7f8fdf400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5c220d9f0 0x55f5c2211c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.273+0000 7f8fdea00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5c220ebc0 0x55f5c221f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59508/0 (socket says 172.21.15.12:59508) 2024-03-23T23:08:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.273+0000 7f8fdfe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220f580 0x55f5c22121a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.273+0000 7f8fdea00640 1 -- 172.21.15.12:0/190353871 learned_addr learned my addr 172.21.15.12:0/190353871 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.273+0000 7f8fdea00640 1 -- 172.21.15.12:0/190353871 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220f580 msgr2=0x55f5c22121a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.273+0000 7f8fdea00640 1 --2- 172.21.15.12:0/190353871 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220f580 0x55f5c22121a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.273+0000 7f8fdea00640 1 -- 172.21.15.12:0/190353871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5c220d9f0 msgr2=0x55f5c2211c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.273+0000 7f8fdea00640 1 --2- 172.21.15.12:0/190353871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5c220d9f0 0x55f5c2211c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.273+0000 7f8fdea00640 1 -- 172.21.15.12:0/190353871 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5c220e640 con 0x55f5c220ebc0 2024-03-23T23:08:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.273+0000 7f8fdf400640 1 --2- 172.21.15.12:0/190353871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5c220d9f0 0x55f5c2211c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.274+0000 7f8fdea00640 1 --2- 172.21.15.12:0/190353871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5c220ebc0 0x55f5c221f140 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc02c200 tx=0x7f8fcc02bcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fb579c367c71f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.274+0000 7f8fcbe00640 1 -- 172.21.15.12:0/190353871 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc043040 con 0x55f5c220ebc0 2024-03-23T23:08:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.274+0000 7f8fcbe00640 1 -- 172.21.15.12:0/190353871 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8fcc02ca10 con 0x55f5c220ebc0 2024-03-23T23:08:05.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.274+0000 7f8fcbe00640 1 -- 172.21.15.12:0/190353871 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc005330 con 0x55f5c220ebc0 2024-03-23T23:08:05.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.274+0000 7f8fe9399740 1 -- 172.21.15.12:0/190353871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5c220ebc0 msgr2=0x55f5c221f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.274+0000 7f8fe9399740 1 --2- 172.21.15.12:0/190353871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5c220ebc0 0x55f5c221f140 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc02c200 tx=0x7f8fcc02bcc0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.275+0000 7f8fe9399740 1 -- 172.21.15.12:0/190353871 shutdown_connections 2024-03-23T23:08:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.275+0000 7f8fe9399740 1 --2- 172.21.15.12:0/190353871 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220f580 0x55f5c22121a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.275+0000 7f8fe9399740 1 --2- 172.21.15.12:0/190353871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5c220ebc0 0x55f5c221f140 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.275+0000 7f8fe9399740 1 --2- 172.21.15.12:0/190353871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5c220d9f0 0x55f5c2211c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.275+0000 7f8fe9399740 1 -- 172.21.15.12:0/190353871 >> 172.21.15.12:0/190353871 conn(0x55f5c2208de0 msgr2=0x55f5c220c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.275+0000 7f8fe9399740 1 -- 172.21.15.12:0/190353871 shutdown_connections 2024-03-23T23:08:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.275+0000 7f8fe9399740 1 -- 172.21.15.12:0/190353871 wait complete. 2024-03-23T23:08:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.275+0000 7f8fe9399740 1 -- start start 2024-03-23T23:08:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.275+0000 7f8fe9399740 1 Processor -- start 2024-03-23T23:08:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.276+0000 7f8fe9399740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220d9f0 0x55f5c221abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.276+0000 7f8fe9399740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5c220ebc0 0x55f5c221b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.276+0000 7f8fe9399740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5c220f580 0x55f5c221e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.276+0000 7f8fe9399740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe01a8fa0 con 0x55f5c220f580 2024-03-23T23:08:05.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.276+0000 7f8fe9399740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe01aa2a0 con 0x55f5c220ebc0 2024-03-23T23:08:05.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.276+0000 7f8fe9399740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe01a95e0 con 0x55f5c220d9f0 2024-03-23T23:08:05.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.276+0000 7f8fe9399740 10 client.0 _ll_register_callbacks cb 0x7f8fd00d21e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:05.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.276+0000 7f8fdea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5c220ebc0 0x55f5c221b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.276+0000 7f8fdf400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220d9f0 0x55f5c221abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.276+0000 7f8fdea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5c220ebc0 0x55f5c221b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44940/0 (socket says 172.21.15.12:44940) 2024-03-23T23:08:05.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.276+0000 7f8fdf400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220d9f0 0x55f5c221abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44620/0 (socket says 172.21.15.12:44620) 2024-03-23T23:08:05.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.276+0000 7f8fdea00640 1 -- 172.21.15.12:0/965406178 learned_addr learned my addr 172.21.15.12:0/965406178 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:05.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.276+0000 7f8fdfe00640 1 --2- 172.21.15.12:0/965406178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5c220f580 0x55f5c221e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.276+0000 7f8fdea00640 1 -- 172.21.15.12:0/965406178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220d9f0 msgr2=0x55f5c221abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:05.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.276+0000 7f8fdea00640 1 --2- 172.21.15.12:0/965406178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220d9f0 0x55f5c221abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.276+0000 7f8fdea00640 1 -- 172.21.15.12:0/965406178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5c220f580 msgr2=0x55f5c221e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:05.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.276+0000 7f8fdea00640 1 --2- 172.21.15.12:0/965406178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5c220f580 0x55f5c221e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.276+0000 7f8fdea00640 1 -- 172.21.15.12:0/965406178 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f5c221e6c0 con 0x55f5c220ebc0 2024-03-23T23:08:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.276+0000 7f8fdf400640 1 --2- 172.21.15.12:0/965406178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220d9f0 0x55f5c221abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.277+0000 7f8fdea00640 1 --2- 172.21.15.12:0/965406178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5c220ebc0 0x55f5c221b120 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc0096c0 tx=0x7f8fcc004ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a9a2d8cd5e217c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.277+0000 7f8fc9600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:08:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.277+0000 7f8fc9600640 1 -- 172.21.15.12:0/965406178 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc043070 con 0x55f5c220ebc0 2024-03-23T23:08:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.277+0000 7f8fe9399740 1 -- 172.21.15.12:0/965406178 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8fcc02ca10 con 0x55f5c220ebc0 2024-03-23T23:08:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.277+0000 7f8fc9600640 1 -- 172.21.15.12:0/965406178 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8fcc004380 con 0x55f5c220ebc0 2024-03-23T23:08:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.277+0000 7f8fc9600640 1 -- 172.21.15.12:0/965406178 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8fcc0050b0 con 0x55f5c220ebc0 2024-03-23T23:08:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.277+0000 7f8fe9399740 10 client.27068 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.277+0000 7f8fe9399740 1 -- 172.21.15.12:0/965406178 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f5c227f1a0 con 0x55f5c220ebc0 2024-03-23T23:08:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.277+0000 7f8fc9600640 1 -- 172.21.15.12:0/965406178 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8fcc02ca10 con 0x55f5c220ebc0 2024-03-23T23:08:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.278+0000 7f8fc9600640 1 -- 172.21.15.12:0/965406178 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8fcc003d20 con 0x55f5c220ebc0 2024-03-23T23:08:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.278+0000 7f8fe9399740 10 client.27068 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.278+0000 7f8fe9399740 10 client.27068 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:08:05.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.278+0000 7f8fe9399740 20 client.27068 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:05.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.278+0000 7f8fe9399740 1 --2- 172.21.15.12:0/965406178 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55f5c227f780 0x55f5c2281b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.278+0000 7f8fe9399740 4 client.27068 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:08:05.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.278+0000 7f8fe9399740 1 -- 172.21.15.12:0/965406178 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f5c2179b50 con 0x55f5c227f780 2024-03-23T23:08:05.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.278+0000 7f8fdf400640 1 --2- 172.21.15.12:0/965406178 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55f5c227f780 0x55f5c2281b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.278+0000 7f8fdf400640 1 --2- 172.21.15.12:0/965406178 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55f5c227f780 0x55f5c2281b70 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fef6953861a6e37f server_cookie=2e905111cf345844 in_seq=0 out_seq=0 2024-03-23T23:08:05.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.278+0000 7f8fc9600640 10 client.27068 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:08:05.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.279+0000 7f8fc9600640 1 -- 172.21.15.12:0/965406178 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f5c2179b50 con 0x55f5c227f780 2024-03-23T23:08:05.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.279+0000 7f8fc9600640 10 client.27068 handle_command_reply: tid=0 2024-03-23T23:08:05.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.279+0000 7f8fe9399740 1 client.27068 shutdown 2024-03-23T23:08:05.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.279+0000 7f8fe9399740 1 -- 172.21.15.12:0/965406178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5c220ebc0 msgr2=0x55f5c221b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:05.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.279+0000 7f8fe9399740 1 --2- 172.21.15.12:0/965406178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5c220ebc0 0x55f5c221b120 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc0096c0 tx=0x7f8fcc004ae0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.279+0000 7f8fe9399740 1 -- 172.21.15.12:0/965406178 shutdown_connections 2024-03-23T23:08:05.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.279+0000 7f8fe9399740 1 --2- 172.21.15.12:0/965406178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5c220f580 0x55f5c221e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.279+0000 7f8fe9399740 1 --2- 172.21.15.12:0/965406178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5c220ebc0 0x55f5c221b120 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.279+0000 7f8fe9399740 1 --2- 172.21.15.12:0/965406178 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55f5c227f780 0x55f5c2281b70 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.279+0000 7f8fe9399740 1 --2- 172.21.15.12:0/965406178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5c220d9f0 0x55f5c221abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.279+0000 7f8fe9399740 1 -- 172.21.15.12:0/965406178 >> 172.21.15.12:0/965406178 conn(0x55f5c2208de0 msgr2=0x55f5c22078d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:05.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.279+0000 7f8fc9600640 0 client.27068 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:08:05.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.279+0000 7f8fe9399740 1 -- 172.21.15.12:0/965406178 shutdown_connections 2024-03-23T23:08:05.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.279+0000 7f8fe9399740 1 -- 172.21.15.12:0/965406178 wait complete. 2024-03-23T23:08:05.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.279+0000 7f8fe9399740 20 client.27068 trim_cache size 0 max 16384 2024-03-23T23:08:05.282 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:08:05.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.280+0000 7f8fe5000640 1 -- 172.21.15.12:0/2274094623 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8fc40a2880 msgr2=0x7f8fc40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:05.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.280+0000 7f8fe5000640 1 --2- 172.21.15.12:0/2274094623 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8fc40a2880 0x7f8fc40a4d40 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x55f5c220eac0 tx=0x7f8fd8009600 comp rx=0 tx=0).stop 2024-03-23T23:08:05.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.280+0000 7f8fe5000640 1 -- 172.21.15.12:0/2274094623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fe0074080 msgr2=0x7f8fe01999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:05.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.280+0000 7f8fe5000640 1 --2- 172.21.15.12:0/2274094623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fe0074080 0x7f8fe01999c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f8fd000c920 tx=0x7f8fd000cde0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.280+0000 7f8fe5000640 1 -- 172.21.15.12:0/2274094623 shutdown_connections 2024-03-23T23:08:05.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.280+0000 7f8fe5000640 1 --2- 172.21.15.12:0/2274094623 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8fc40a2880 0x7f8fc40a4d40 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.280+0000 7f8fe5000640 1 --2- 172.21.15.12:0/2274094623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fe0109430 0x7f8fe01a0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.280+0000 7f8fe5000640 1 --2- 172.21.15.12:0/2274094623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8fe0075660 0x7f8fe0199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.280+0000 7f8fe5000640 1 --2- 172.21.15.12:0/2274094623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fe0074080 0x7f8fe01999c0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.280+0000 7f8fe5000640 1 -- 172.21.15.12:0/2274094623 >> 172.21.15.12:0/2274094623 conn(0x7f8fe00fcb10 msgr2=0x7f8fe01053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:05.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.280+0000 7f8fe5000640 1 -- 172.21.15.12:0/2274094623 shutdown_connections 2024-03-23T23:08:05.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.280+0000 7f8fe5000640 1 -- 172.21.15.12:0/2274094623 wait complete. 2024-03-23T23:08:05.299 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:08:05.299 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-23T23:08:05.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.521+0000 7f364a400640 1 -- 172.21.15.12:0/1255756532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3644107260 msgr2=0x7f36441096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:05.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.521+0000 7f364a400640 1 --2- 172.21.15.12:0/1255756532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3644107260 0x7f36441096b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f363c009860 tx=0x7f363c02bb70 comp rx=0 tx=0).stop 2024-03-23T23:08:05.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.521+0000 7f364a400640 1 -- 172.21.15.12:0/1255756532 shutdown_connections 2024-03-23T23:08:05.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.521+0000 7f364a400640 1 --2- 172.21.15.12:0/1255756532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3644107260 0x7f36441096b0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.521+0000 7f364a400640 1 --2- 172.21.15.12:0/1255756532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3644104930 0x7f3644106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.521+0000 7f364a400640 1 --2- 172.21.15.12:0/1255756532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3644102000 0x7f36441043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.521+0000 7f364a400640 1 -- 172.21.15.12:0/1255756532 >> 172.21.15.12:0/1255756532 conn(0x7f36440fba10 msgr2=0x7f36440fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:05.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.521+0000 7f364a400640 1 -- 172.21.15.12:0/1255756532 shutdown_connections 2024-03-23T23:08:05.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.521+0000 7f364a400640 1 -- 172.21.15.12:0/1255756532 wait complete. 2024-03-23T23:08:05.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.522+0000 7f364a400640 1 Processor -- start 2024-03-23T23:08:05.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.522+0000 7f364a400640 1 -- start start 2024-03-23T23:08:05.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.523+0000 7f364a400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3644102000 0x7f3644199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.523+0000 7f364a400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3644104930 0x7f3644199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.523+0000 7f364a400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3644107260 0x7f36441a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.523+0000 7f364a400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364410d210 con 0x7f3644107260 2024-03-23T23:08:05.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.523+0000 7f364a400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364410d090 con 0x7f3644102000 2024-03-23T23:08:05.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.523+0000 7f364a400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364410d390 con 0x7f3644104930 2024-03-23T23:08:05.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.523+0000 7f3649000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3644102000 0x7f3644199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.523+0000 7f363be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3644104930 0x7f3644199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.523+0000 7f3649000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3644102000 0x7f3644199a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44950/0 (socket says 172.21.15.12:44950) 2024-03-23T23:08:05.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.523+0000 7f3649000640 1 -- 172.21.15.12:0/3588746468 learned_addr learned my addr 172.21.15.12:0/3588746468 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:05.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.523+0000 7f3649a00640 1 --2- 172.21.15.12:0/3588746468 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3644107260 0x7f36441a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.523+0000 7f3649000640 1 -- 172.21.15.12:0/3588746468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3644104930 msgr2=0x7f3644199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:05.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.523+0000 7f3649000640 1 --2- 172.21.15.12:0/3588746468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3644104930 0x7f3644199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.523+0000 7f3649000640 1 -- 172.21.15.12:0/3588746468 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3644107260 msgr2=0x7f36441a0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:05.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.523+0000 7f3649000640 1 --2- 172.21.15.12:0/3588746468 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3644107260 0x7f36441a0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.523+0000 7f3649000640 1 -- 172.21.15.12:0/3588746468 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36441a1620 con 0x7f3644102000 2024-03-23T23:08:05.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.524+0000 7f3649000640 1 --2- 172.21.15.12:0/3588746468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3644102000 0x7f3644199a00 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f363400ebc0 tx=0x7f363400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:05.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.524+0000 7f3642000640 1 -- 172.21.15.12:0/3588746468 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3634010630 con 0x7f3644102000 2024-03-23T23:08:05.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.524+0000 7f364a400640 1 -- 172.21.15.12:0/3588746468 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36441a1910 con 0x7f3644102000 2024-03-23T23:08:05.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.524+0000 7f3642000640 1 -- 172.21.15.12:0/3588746468 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3634010cb0 con 0x7f3644102000 2024-03-23T23:08:05.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.524+0000 7f3642000640 1 -- 172.21.15.12:0/3588746468 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3634018bd0 con 0x7f3644102000 2024-03-23T23:08:05.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.524+0000 7f364a400640 1 -- 172.21.15.12:0/3588746468 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36441a1e60 con 0x7f3644102000 2024-03-23T23:08:05.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.527+0000 7f3642000640 1 -- 172.21.15.12:0/3588746468 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f36340107d0 con 0x7f3644102000 2024-03-23T23:08:05.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.527+0000 7f364e684740 1 Processor -- start 2024-03-23T23:08:05.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.528+0000 7f364e684740 1 -- start start 2024-03-23T23:08:05.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.528+0000 7f364e684740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc3329f0 0x5582fc332e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.528+0000 7f364e684740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582fc333bc0 0x5582fc334040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.528+0000 7f364e684740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582fc334580 0x5582fc33ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.528+0000 7f364e684740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582fc2972c0 con 0x5582fc334580 2024-03-23T23:08:05.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.528+0000 7f364e684740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582fc23fc90 con 0x5582fc333bc0 2024-03-23T23:08:05.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.528+0000 7f364e684740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582fc293360 con 0x5582fc3329f0 2024-03-23T23:08:05.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.528+0000 7f3649000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582fc333bc0 0x5582fc334040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.528+0000 7f3649a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc3329f0 0x5582fc332e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.528+0000 7f3649a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc3329f0 0x5582fc332e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44644/0 (socket says 172.21.15.12:44644) 2024-03-23T23:08:05.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.528+0000 7f363be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582fc334580 0x5582fc33ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.528+0000 7f3649a00640 1 -- 172.21.15.12:0/2911502926 learned_addr learned my addr 172.21.15.12:0/2911502926 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:05.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.529+0000 7f3649a00640 1 -- 172.21.15.12:0/2911502926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582fc333bc0 msgr2=0x5582fc334040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:05.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.529+0000 7f3649a00640 1 --2- 172.21.15.12:0/2911502926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582fc333bc0 0x5582fc334040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.529+0000 7f3649a00640 1 -- 172.21.15.12:0/2911502926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582fc334580 msgr2=0x5582fc33ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:05.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.529+0000 7f3649a00640 1 --2- 172.21.15.12:0/2911502926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582fc334580 0x5582fc33ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.529+0000 7f3649a00640 1 -- 172.21.15.12:0/2911502926 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5582fc264080 con 0x5582fc3329f0 2024-03-23T23:08:05.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.529+0000 7f3642000640 1 --2- 172.21.15.12:0/3588746468 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f36280a2800 0x7f36280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.529+0000 7f3649a00640 1 --2- 172.21.15.12:0/2911502926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc3329f0 0x5582fc332e10 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f363c034b10 tx=0x7f363c005b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cc4b70e9a8a06e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:05.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.529+0000 7f3642000640 1 -- 172.21.15.12:0/3588746468 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f36340cf030 con 0x7f3644102000 2024-03-23T23:08:05.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.529+0000 7f3649000640 1 --2- 172.21.15.12:0/3588746468 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f36280a2800 0x7f36280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.529+0000 7f363a000640 1 -- 172.21.15.12:0/2911502926 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c043020 con 0x5582fc3329f0 2024-03-23T23:08:05.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.529+0000 7f363a000640 1 -- 172.21.15.12:0/2911502926 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f363c034db0 con 0x5582fc3329f0 2024-03-23T23:08:05.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.529+0000 7f363a000640 1 -- 172.21.15.12:0/2911502926 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c039040 con 0x5582fc3329f0 2024-03-23T23:08:05.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.530+0000 7f364e684740 1 -- 172.21.15.12:0/2911502926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc3329f0 msgr2=0x5582fc332e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:05.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.530+0000 7f364e684740 1 --2- 172.21.15.12:0/2911502926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc3329f0 0x5582fc332e10 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f363c034b10 tx=0x7f363c005b20 comp rx=0 tx=0).stop 2024-03-23T23:08:05.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.530+0000 7f3649000640 1 --2- 172.21.15.12:0/3588746468 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f36280a2800 0x7f36280a4cc0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f363400cba0 tx=0x7f36340d2b10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:05.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.530+0000 7f364e684740 1 -- 172.21.15.12:0/2911502926 shutdown_connections 2024-03-23T23:08:05.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.530+0000 7f364e684740 1 --2- 172.21.15.12:0/2911502926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582fc334580 0x5582fc33ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.530+0000 7f364e684740 1 --2- 172.21.15.12:0/2911502926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582fc333bc0 0x5582fc334040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.530+0000 7f364e684740 1 --2- 172.21.15.12:0/2911502926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc3329f0 0x5582fc332e10 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.530+0000 7f364e684740 1 -- 172.21.15.12:0/2911502926 >> 172.21.15.12:0/2911502926 conn(0x5582fc32dde0 msgr2=0x5582fc32e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:05.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.530+0000 7f364e684740 1 -- 172.21.15.12:0/2911502926 shutdown_connections 2024-03-23T23:08:05.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.530+0000 7f364e684740 1 -- 172.21.15.12:0/2911502926 wait complete. 2024-03-23T23:08:05.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.531+0000 7f364e684740 1 -- start start 2024-03-23T23:08:05.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.531+0000 7f364e684740 1 Processor -- start 2024-03-23T23:08:05.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.531+0000 7f364e684740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582fc3329f0 0x5582fc33cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.531+0000 7f364e684740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc333bc0 0x5582fc33d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.531+0000 7f364e684740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582fc334580 0x5582fc3441b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.531+0000 7f364e684740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582fc33b530 con 0x5582fc334580 2024-03-23T23:08:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.531+0000 7f364e684740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582fc2530d0 con 0x5582fc3329f0 2024-03-23T23:08:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.531+0000 7f364e684740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582fc33b6b0 con 0x5582fc333bc0 2024-03-23T23:08:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.531+0000 7f364e684740 10 client.0 _ll_register_callbacks cb 0x5582fc257e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.532+0000 7f363be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc333bc0 0x5582fc33d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.532+0000 7f363be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc333bc0 0x5582fc33d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44656/0 (socket says 172.21.15.12:44656) 2024-03-23T23:08:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.532+0000 7f363be00640 1 -- 172.21.15.12:0/2315398735 learned_addr learned my addr 172.21.15.12:0/2315398735 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.532+0000 7f3649a00640 1 --2- 172.21.15.12:0/2315398735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582fc3329f0 0x5582fc33cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.532+0000 7f3649a00640 1 --2- 172.21.15.12:0/2315398735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582fc334580 0x5582fc3441b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.532+0000 7f363be00640 1 -- 172.21.15.12:0/2315398735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582fc3329f0 msgr2=0x5582fc33cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.532+0000 7f363be00640 1 --2- 172.21.15.12:0/2315398735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582fc3329f0 0x5582fc33cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.532+0000 7f363be00640 1 -- 172.21.15.12:0/2315398735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582fc334580 msgr2=0x5582fc3441b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.532+0000 7f363be00640 1 --2- 172.21.15.12:0/2315398735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582fc334580 0x5582fc3441b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.532+0000 7f363be00640 1 -- 172.21.15.12:0/2315398735 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5582fbfab3f0 con 0x5582fc333bc0 2024-03-23T23:08:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.532+0000 7f363be00640 1 --2- 172.21.15.12:0/2315398735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc333bc0 0x5582fc33d170 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f362c0126d0 tx=0x7f362c012b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f42c359634374d9a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.532+0000 7f3632a00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:08:05.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.532+0000 7f3632a00640 1 -- 172.21.15.12:0/2315398735 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362c004020 con 0x5582fc333bc0 2024-03-23T23:08:05.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.533+0000 7f3632a00640 1 -- 172.21.15.12:0/2315398735 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f362c0041c0 con 0x5582fc333bc0 2024-03-23T23:08:05.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.533+0000 7f364e684740 1 -- 172.21.15.12:0/2315398735 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f363c034db0 con 0x5582fc333bc0 2024-03-23T23:08:05.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.533+0000 7f3632a00640 1 -- 172.21.15.12:0/2315398735 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f362c0043c0 con 0x5582fc333bc0 2024-03-23T23:08:05.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.533+0000 7f364e684740 10 client.27052 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:05.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.533+0000 7f364e684740 1 -- 172.21.15.12:0/2315398735 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5582fc0757d0 con 0x5582fc333bc0 2024-03-23T23:08:05.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.533+0000 7f3632a00640 1 -- 172.21.15.12:0/2315398735 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f363c034db0 con 0x5582fc333bc0 2024-03-23T23:08:05.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.533+0000 7f3632a00640 1 -- 172.21.15.12:0/2315398735 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f362c003ae0 con 0x5582fc333bc0 2024-03-23T23:08:05.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.533+0000 7f364e684740 10 client.27052 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:05.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.533+0000 7f364e684740 10 client.27052 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:08:05.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.533+0000 7f364e684740 20 client.27052 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:05.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.533+0000 7f364e684740 1 --2- 172.21.15.12:0/2315398735 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5582fc3487b0 0x5582fc34abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.533+0000 7f364e684740 4 client.27052 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:08:05.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.533+0000 7f364e684740 1 -- 172.21.15.12:0/2315398735 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5582fc2a4800 con 0x5582fc3487b0 2024-03-23T23:08:05.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.534+0000 7f3649a00640 1 --2- 172.21.15.12:0/2315398735 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5582fc3487b0 0x5582fc34abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.534+0000 7f3649a00640 1 --2- 172.21.15.12:0/2315398735 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5582fc3487b0 0x5582fc34abf0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a926f191d2f8d60 server_cookie=6ee790e88e1896b6 in_seq=0 out_seq=0 2024-03-23T23:08:05.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.534+0000 7f3632a00640 10 client.27052 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:05.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.535+0000 7f3632a00640 1 -- 172.21.15.12:0/2315398735 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5582fc2a4800 con 0x5582fc3487b0 2024-03-23T23:08:05.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.535+0000 7f3632a00640 10 client.27052 handle_command_reply: tid=0 2024-03-23T23:08:05.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.535+0000 7f364e684740 1 client.27052 shutdown 2024-03-23T23:08:05.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.535+0000 7f364e684740 1 -- 172.21.15.12:0/2315398735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc333bc0 msgr2=0x5582fc33d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:05.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.535+0000 7f364e684740 1 --2- 172.21.15.12:0/2315398735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc333bc0 0x5582fc33d170 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f362c0126d0 tx=0x7f362c012b90 comp rx=0 tx=0).stop 2024-03-23T23:08:05.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.535+0000 7f364e684740 1 -- 172.21.15.12:0/2315398735 shutdown_connections 2024-03-23T23:08:05.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.536+0000 7f364e684740 1 --2- 172.21.15.12:0/2315398735 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5582fc3487b0 0x5582fc34abf0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.536+0000 7f364e684740 1 --2- 172.21.15.12:0/2315398735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582fc334580 0x5582fc3441b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.536+0000 7f364e684740 1 --2- 172.21.15.12:0/2315398735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc333bc0 0x5582fc33d170 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.536+0000 7f364e684740 1 --2- 172.21.15.12:0/2315398735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582fc3329f0 0x5582fc33cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.536+0000 7f364e684740 1 -- 172.21.15.12:0/2315398735 >> 172.21.15.12:0/2315398735 conn(0x5582fc32dde0 msgr2=0x5582fc32d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:05.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.536+0000 7f3632a00640 0 client.27052 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:05.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.536+0000 7f364e684740 1 -- 172.21.15.12:0/2315398735 shutdown_connections 2024-03-23T23:08:05.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.536+0000 7f364e684740 1 -- 172.21.15.12:0/2315398735 wait complete. 2024-03-23T23:08:05.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.536+0000 7f364e684740 20 client.27052 trim_cache size 0 max 16384 2024-03-23T23:08:05.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.550+0000 7f364e684740 1 Processor -- start 2024-03-23T23:08:05.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.550+0000 7f364e684740 1 -- start start 2024-03-23T23:08:05.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.550+0000 7f364e684740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582fc3329f0 0x5582fc336bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.550+0000 7f364e684740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc333bc0 0x5582fc344170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.550+0000 7f364e684740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582fc334580 0x5582fc337100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.550+0000 7f364e684740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36441acee0 con 0x5582fc3329f0 2024-03-23T23:08:05.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.550+0000 7f364e684740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36441aa9e0 con 0x5582fc334580 2024-03-23T23:08:05.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.550+0000 7f364e684740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36441a8a80 con 0x5582fc333bc0 2024-03-23T23:08:05.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.551+0000 7f363be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc333bc0 0x5582fc344170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.551+0000 7f3649a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582fc3329f0 0x5582fc336bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.551+0000 7f363be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc333bc0 0x5582fc344170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44664/0 (socket says 172.21.15.12:44664) 2024-03-23T23:08:05.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.551+0000 7f363be00640 1 -- 172.21.15.12:0/1413982088 learned_addr learned my addr 172.21.15.12:0/1413982088 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:05.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.551+0000 7f3649a00640 1 --2- 172.21.15.12:0/1413982088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582fc334580 0x5582fc337100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.551+0000 7f3649a00640 1 -- 172.21.15.12:0/1413982088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc333bc0 msgr2=0x5582fc344170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:05.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.551+0000 7f3649a00640 1 --2- 172.21.15.12:0/1413982088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc333bc0 0x5582fc344170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.551+0000 7f3649a00640 1 -- 172.21.15.12:0/1413982088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582fc334580 msgr2=0x5582fc337100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:05.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.551+0000 7f3649a00640 1 --2- 172.21.15.12:0/1413982088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582fc334580 0x5582fc337100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.551+0000 7f3649a00640 1 -- 172.21.15.12:0/1413982088 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5582fc33c050 con 0x5582fc3329f0 2024-03-23T23:08:05.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.551+0000 7f3649a00640 1 --2- 172.21.15.12:0/1413982088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582fc3329f0 0x5582fc336bc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f363c037040 tx=0x7f363c02bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=885a16c503b3bb5a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:05.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.551+0000 7f3639600640 1 -- 172.21.15.12:0/1413982088 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c039070 con 0x5582fc3329f0 2024-03-23T23:08:05.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.552+0000 7f3639600640 1 -- 172.21.15.12:0/1413982088 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f363c034cd0 con 0x5582fc3329f0 2024-03-23T23:08:05.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.552+0000 7f3639600640 1 -- 172.21.15.12:0/1413982088 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c043040 con 0x5582fc3329f0 2024-03-23T23:08:05.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.552+0000 7f364e684740 1 -- 172.21.15.12:0/1413982088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582fc3329f0 msgr2=0x5582fc336bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:05.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.552+0000 7f364e684740 1 --2- 172.21.15.12:0/1413982088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582fc3329f0 0x5582fc336bc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f363c037040 tx=0x7f363c02bb70 comp rx=0 tx=0).stop 2024-03-23T23:08:05.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.552+0000 7f364e684740 1 -- 172.21.15.12:0/1413982088 shutdown_connections 2024-03-23T23:08:05.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.552+0000 7f364e684740 1 --2- 172.21.15.12:0/1413982088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582fc334580 0x5582fc337100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.552+0000 7f364e684740 1 --2- 172.21.15.12:0/1413982088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc333bc0 0x5582fc344170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.552+0000 7f364e684740 1 --2- 172.21.15.12:0/1413982088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582fc3329f0 0x5582fc336bc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.552+0000 7f364e684740 1 -- 172.21.15.12:0/1413982088 >> 172.21.15.12:0/1413982088 conn(0x5582fc32dde0 msgr2=0x5582fc32c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:05.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.552+0000 7f364e684740 1 -- 172.21.15.12:0/1413982088 shutdown_connections 2024-03-23T23:08:05.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.552+0000 7f364e684740 1 -- 172.21.15.12:0/1413982088 wait complete. 2024-03-23T23:08:05.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.553+0000 7f364e684740 1 -- start start 2024-03-23T23:08:05.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.553+0000 7f364e684740 1 Processor -- start 2024-03-23T23:08:05.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.553+0000 7f364e684740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582fc3329f0 0x5582fc3400e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.553+0000 7f364e684740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc333bc0 0x5582fc340620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.553+0000 7f364e684740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582fc334580 0x5582fc343640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.553+0000 7f364e684740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36441a9080 con 0x5582fc334580 2024-03-23T23:08:05.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.553+0000 7f364e684740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36441aa380 con 0x5582fc3329f0 2024-03-23T23:08:05.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.553+0000 7f364e684740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36441a96c0 con 0x5582fc333bc0 2024-03-23T23:08:05.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.553+0000 7f364e684740 10 client.0 _ll_register_callbacks cb 0x7f362c005b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:05.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.554+0000 7f363be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc333bc0 0x5582fc340620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.554+0000 7f363be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc333bc0 0x5582fc340620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44668/0 (socket says 172.21.15.12:44668) 2024-03-23T23:08:05.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.554+0000 7f363be00640 1 -- 172.21.15.12:0/3441925200 learned_addr learned my addr 172.21.15.12:0/3441925200 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:05.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.554+0000 7f3649a00640 1 --2- 172.21.15.12:0/3441925200 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582fc3329f0 0x5582fc3400e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.554+0000 7f3649a00640 1 --2- 172.21.15.12:0/3441925200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582fc334580 0x5582fc343640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.554+0000 7f363be00640 1 -- 172.21.15.12:0/3441925200 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582fc3329f0 msgr2=0x5582fc3400e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:05.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.554+0000 7f363be00640 1 --2- 172.21.15.12:0/3441925200 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582fc3329f0 0x5582fc3400e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.554+0000 7f363be00640 1 -- 172.21.15.12:0/3441925200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582fc334580 msgr2=0x5582fc343640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:05.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.554+0000 7f363be00640 1 --2- 172.21.15.12:0/3441925200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582fc334580 0x5582fc343640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.554+0000 7f363be00640 1 -- 172.21.15.12:0/3441925200 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5582fc343c30 con 0x5582fc333bc0 2024-03-23T23:08:05.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.554+0000 7f363be00640 1 --2- 172.21.15.12:0/3441925200 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc333bc0 0x5582fc340620 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f362c003f20 tx=0x7f362c009df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=765b45ec6b2bfd83 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:05.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.554+0000 7f3632a00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:08:05.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.555+0000 7f3632a00640 1 -- 172.21.15.12:0/3441925200 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362c035060 con 0x5582fc333bc0 2024-03-23T23:08:05.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.555+0000 7f364e684740 1 -- 172.21.15.12:0/3441925200 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f363c034cd0 con 0x5582fc333bc0 2024-03-23T23:08:05.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.555+0000 7f3632a00640 1 -- 172.21.15.12:0/3441925200 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f362c003800 con 0x5582fc333bc0 2024-03-23T23:08:05.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.555+0000 7f3632a00640 1 -- 172.21.15.12:0/3441925200 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f362c003a00 con 0x5582fc333bc0 2024-03-23T23:08:05.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.555+0000 7f364e684740 10 client.27058 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:05.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.555+0000 7f364e684740 1 -- 172.21.15.12:0/3441925200 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5582fc3a49a0 con 0x5582fc333bc0 2024-03-23T23:08:05.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.555+0000 7f3632a00640 1 -- 172.21.15.12:0/3441925200 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f363c034cd0 con 0x5582fc333bc0 2024-03-23T23:08:05.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.555+0000 7f3632a00640 1 -- 172.21.15.12:0/3441925200 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f362c01c680 con 0x5582fc333bc0 2024-03-23T23:08:05.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.555+0000 7f364e684740 10 client.27058 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:05.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.555+0000 7f364e684740 10 client.27058 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:08:05.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.555+0000 7f364e684740 20 client.27058 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:05.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.556+0000 7f364e684740 1 --2- 172.21.15.12:0/3441925200 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5582fc3a5320 0x5582fc3a7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:05.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.556+0000 7f364e684740 4 client.27058 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:08:05.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.556+0000 7f364e684740 1 -- 172.21.15.12:0/3441925200 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5582fc29eb50 con 0x5582fc3a5320 2024-03-23T23:08:05.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.556+0000 7f3649a00640 1 --2- 172.21.15.12:0/3441925200 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5582fc3a5320 0x5582fc3a7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:05.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.556+0000 7f3649a00640 1 --2- 172.21.15.12:0/3441925200 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5582fc3a5320 0x5582fc3a7740 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc06bff0ce2fa351 server_cookie=23718cb759d4f98d in_seq=0 out_seq=0 2024-03-23T23:08:05.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.556+0000 7f3632a00640 10 client.27058 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:05.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.556+0000 7f3632a00640 1 -- 172.21.15.12:0/3441925200 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5582fc29eb50 con 0x5582fc3a5320 2024-03-23T23:08:05.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.556+0000 7f3632a00640 10 client.27058 handle_command_reply: tid=0 2024-03-23T23:08:05.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.556+0000 7f364e684740 1 client.27058 shutdown 2024-03-23T23:08:05.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.557+0000 7f364e684740 1 -- 172.21.15.12:0/3441925200 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc333bc0 msgr2=0x5582fc340620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:05.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.557+0000 7f364e684740 1 --2- 172.21.15.12:0/3441925200 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc333bc0 0x5582fc340620 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f362c003f20 tx=0x7f362c009df0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.557+0000 7f364e684740 1 -- 172.21.15.12:0/3441925200 shutdown_connections 2024-03-23T23:08:05.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.557+0000 7f364e684740 1 --2- 172.21.15.12:0/3441925200 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5582fc3a5320 0x5582fc3a7740 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.557+0000 7f364e684740 1 --2- 172.21.15.12:0/3441925200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5582fc334580 0x5582fc343640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.557+0000 7f364e684740 1 --2- 172.21.15.12:0/3441925200 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582fc333bc0 0x5582fc340620 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.557+0000 7f3632a00640 0 client.27058 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:05.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.557+0000 7f364e684740 1 --2- 172.21.15.12:0/3441925200 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582fc3329f0 0x5582fc3400e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.557+0000 7f364e684740 1 -- 172.21.15.12:0/3441925200 >> 172.21.15.12:0/3441925200 conn(0x5582fc32dde0 msgr2=0x5582fc331040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:05.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.557+0000 7f364e684740 1 -- 172.21.15.12:0/3441925200 shutdown_connections 2024-03-23T23:08:05.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.557+0000 7f364e684740 1 -- 172.21.15.12:0/3441925200 wait complete. 2024-03-23T23:08:05.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.557+0000 7f364e684740 20 client.27058 trim_cache size 0 max 16384 2024-03-23T23:08:05.560 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:08:05.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.558+0000 7f364a400640 1 -- 172.21.15.12:0/3588746468 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f36280a2800 msgr2=0x7f36280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:05.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.558+0000 7f364a400640 1 --2- 172.21.15.12:0/3588746468 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f36280a2800 0x7f36280a4cc0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f363400cba0 tx=0x7f36340d2b10 comp rx=0 tx=0).stop 2024-03-23T23:08:05.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.558+0000 7f364a400640 1 -- 172.21.15.12:0/3588746468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3644102000 msgr2=0x7f3644199a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:05.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.558+0000 7f364a400640 1 --2- 172.21.15.12:0/3588746468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3644102000 0x7f3644199a00 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f363400ebc0 tx=0x7f363400c6a0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.558+0000 7f364a400640 1 -- 172.21.15.12:0/3588746468 shutdown_connections 2024-03-23T23:08:05.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.558+0000 7f364a400640 1 --2- 172.21.15.12:0/3588746468 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f36280a2800 0x7f36280a4cc0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.558+0000 7f364a400640 1 --2- 172.21.15.12:0/3588746468 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3644107260 0x7f36441a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.558+0000 7f364a400640 1 --2- 172.21.15.12:0/3588746468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3644104930 0x7f3644199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.558+0000 7f364a400640 1 --2- 172.21.15.12:0/3588746468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3644102000 0x7f3644199a00 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:05.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.558+0000 7f364a400640 1 -- 172.21.15.12:0/3588746468 >> 172.21.15.12:0/3588746468 conn(0x7f36440fba10 msgr2=0x7f36441016c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:05.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.558+0000 7f364a400640 1 -- 172.21.15.12:0/3588746468 shutdown_connections 2024-03-23T23:08:05.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:05.558+0000 7f364a400640 1 -- 172.21.15.12:0/3588746468 wait complete. 2024-03-23T23:08:06.575 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-23T23:08:06.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:08:06.576 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-23T23:08:06.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.814+0000 7fbbe7e00640 1 -- 172.21.15.12:0/732040219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbe01038e0 msgr2=0x7fbbe0103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:06.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.814+0000 7fbbe7e00640 1 --2- 172.21.15.12:0/732040219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbe01038e0 0x7fbbe0103d00 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc009900 tx=0x7fbbcc02bc30 comp rx=0 tx=0).stop 2024-03-23T23:08:06.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.814+0000 7fbbe7e00640 1 -- 172.21.15.12:0/732040219 shutdown_connections 2024-03-23T23:08:06.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.814+0000 7fbbe7e00640 1 --2- 172.21.15.12:0/732040219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbe0105490 0x7fbbe010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.814+0000 7fbbe7e00640 1 --2- 172.21.15.12:0/732040219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbbe0104ad0 0x7fbbe0104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.814+0000 7fbbe7e00640 1 --2- 172.21.15.12:0/732040219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbe01038e0 0x7fbbe0103d00 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.814+0000 7fbbe7e00640 1 -- 172.21.15.12:0/732040219 >> 172.21.15.12:0/732040219 conn(0x7fbbe00fee50 msgr2=0x7fbbe0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:06.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.814+0000 7fbbe7e00640 1 -- 172.21.15.12:0/732040219 shutdown_connections 2024-03-23T23:08:06.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.815+0000 7fbbe7e00640 1 -- 172.21.15.12:0/732040219 wait complete. 2024-03-23T23:08:06.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.816+0000 7fbbe7e00640 1 Processor -- start 2024-03-23T23:08:06.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.816+0000 7fbbe7e00640 1 -- start start 2024-03-23T23:08:06.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.816+0000 7fbbe7e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbe01038e0 0x7fbbe0199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:06.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.816+0000 7fbbe7e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbe0104ad0 0x7fbbe019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:06.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.816+0000 7fbbe7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbbe0105490 0x7fbbe01a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:06.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.816+0000 7fbbe7e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe010f580 con 0x7fbbe01038e0 2024-03-23T23:08:06.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.816+0000 7fbbe7e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe010f400 con 0x7fbbe0105490 2024-03-23T23:08:06.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.816+0000 7fbbe7e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe010f700 con 0x7fbbe0104ad0 2024-03-23T23:08:06.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.816+0000 7fbbe4a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbe0104ad0 0x7fbbe019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:06.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.816+0000 7fbbe4a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbe0104ad0 0x7fbbe019a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44682/0 (socket says 172.21.15.12:44682) 2024-03-23T23:08:06.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.816+0000 7fbbe4a00640 1 -- 172.21.15.12:0/2631215648 learned_addr learned my addr 172.21.15.12:0/2631215648 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:06.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.816+0000 7fbbe5400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbe01038e0 0x7fbbe0199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:06.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.816+0000 7fbbe5e00640 1 --2- 172.21.15.12:0/2631215648 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbbe0105490 0x7fbbe01a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:06.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.817+0000 7fbbe4a00640 1 -- 172.21.15.12:0/2631215648 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbbe0105490 msgr2=0x7fbbe01a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:06.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.817+0000 7fbbe4a00640 1 --2- 172.21.15.12:0/2631215648 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbbe0105490 0x7fbbe01a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.817+0000 7fbbe4a00640 1 -- 172.21.15.12:0/2631215648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbe01038e0 msgr2=0x7fbbe0199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:06.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.817+0000 7fbbe4a00640 1 --2- 172.21.15.12:0/2631215648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbe01038e0 0x7fbbe0199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.817+0000 7fbbe4a00640 1 -- 172.21.15.12:0/2631215648 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe01a1850 con 0x7fbbe0104ad0 2024-03-23T23:08:06.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.817+0000 7fbbe4a00640 1 --2- 172.21.15.12:0/2631215648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbe0104ad0 0x7fbbe019a180 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fbbd400ea60 tx=0x7fbbd400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:06.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.817+0000 7fbbde000640 1 -- 172.21.15.12:0/2631215648 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd400cde0 con 0x7fbbe0104ad0 2024-03-23T23:08:06.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.817+0000 7fbbe7e00640 1 -- 172.21.15.12:0/2631215648 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbe01a1b40 con 0x7fbbe0104ad0 2024-03-23T23:08:06.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.817+0000 7fbbde000640 1 -- 172.21.15.12:0/2631215648 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbbd40035f0 con 0x7fbbe0104ad0 2024-03-23T23:08:06.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.817+0000 7fbbde000640 1 -- 172.21.15.12:0/2631215648 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd4010670 con 0x7fbbe0104ad0 2024-03-23T23:08:06.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.818+0000 7fbbe7e00640 1 -- 172.21.15.12:0/2631215648 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbe01a2060 con 0x7fbbe0104ad0 2024-03-23T23:08:06.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.822+0000 7fbbeab85740 1 Processor -- start 2024-03-23T23:08:06.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.822+0000 7fbbeab85740 1 -- start start 2024-03-23T23:08:06.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.822+0000 7fbbeab85740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ff50a0 0x563de0ff54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:06.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.822+0000 7fbbeab85740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563de0ff5a90 0x563de0ffa590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:06.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.822+0000 7fbbde000640 1 -- 172.21.15.12:0/2631215648 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbbd40037a0 con 0x7fbbe0104ad0 2024-03-23T23:08:06.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.823+0000 7fbbeab85740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563de0ffac00 0x563de0ffd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:06.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.823+0000 7fbbeab85740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563de0f5aeb0 con 0x563de0ff5a90 2024-03-23T23:08:06.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.823+0000 7fbbeab85740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x563de0f03a60 con 0x563de0ff50a0 2024-03-23T23:08:06.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.823+0000 7fbbeab85740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x563de0f56f50 con 0x563de0ffac00 2024-03-23T23:08:06.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.823+0000 7fbbe5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ff50a0 0x563de0ff54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:06.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.823+0000 7fbbe5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ff50a0 0x563de0ff54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:45018/0 (socket says 172.21.15.12:45018) 2024-03-23T23:08:06.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.823+0000 7fbbe5e00640 1 -- 172.21.15.12:0/606626558 learned_addr learned my addr 172.21.15.12:0/606626558 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:06.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.823+0000 7fbbe5400640 1 --2- 172.21.15.12:0/606626558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563de0ff5a90 0x563de0ffa590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:06.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.823+0000 7fbbe4a00640 1 --2- 172.21.15.12:0/606626558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563de0ffac00 0x563de0ffd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:06.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.823+0000 7fbbe5e00640 1 -- 172.21.15.12:0/606626558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563de0ffac00 msgr2=0x563de0ffd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:06.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.823+0000 7fbbe5e00640 1 --2- 172.21.15.12:0/606626558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563de0ffac00 0x563de0ffd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.823+0000 7fbbe5e00640 1 -- 172.21.15.12:0/606626558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563de0ff5a90 msgr2=0x563de0ffa590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:06.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.823+0000 7fbbe5e00640 1 --2- 172.21.15.12:0/606626558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563de0ff5a90 0x563de0ffa590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.823+0000 7fbbe5e00640 1 -- 172.21.15.12:0/606626558 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563de0d377e0 con 0x563de0ff50a0 2024-03-23T23:08:06.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.824+0000 7fbbe5e00640 1 --2- 172.21.15.12:0/606626558 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ff50a0 0x563de0ff54c0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fbbd8009fd0 tx=0x7fbbd80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29fc6598caf315f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:06.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.824+0000 7fbbd2600640 1 -- 172.21.15.12:0/606626558 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd8004d20 con 0x563de0ff50a0 2024-03-23T23:08:06.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.824+0000 7fbbd2600640 1 -- 172.21.15.12:0/606626558 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbbd8004050 con 0x563de0ff50a0 2024-03-23T23:08:06.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.824+0000 7fbbd2600640 1 -- 172.21.15.12:0/606626558 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd8005240 con 0x563de0ff50a0 2024-03-23T23:08:06.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.824+0000 7fbbde000640 1 --2- 172.21.15.12:0/2631215648 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbbc80a2800 0x7fbbc80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:06.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.825+0000 7fbbde000640 1 -- 172.21.15.12:0/2631215648 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fbbd40cee30 con 0x7fbbe0104ad0 2024-03-23T23:08:06.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.825+0000 7fbbe5400640 1 --2- 172.21.15.12:0/2631215648 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbbc80a2800 0x7fbbc80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:06.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.825+0000 7fbbeab85740 1 -- 172.21.15.12:0/606626558 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ff50a0 msgr2=0x563de0ff54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:06.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.825+0000 7fbbeab85740 1 --2- 172.21.15.12:0/606626558 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ff50a0 0x563de0ff54c0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fbbd8009fd0 tx=0x7fbbd80118d0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.825+0000 7fbbeab85740 1 -- 172.21.15.12:0/606626558 shutdown_connections 2024-03-23T23:08:06.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.825+0000 7fbbeab85740 1 --2- 172.21.15.12:0/606626558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563de0ffac00 0x563de0ffd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.825+0000 7fbbeab85740 1 --2- 172.21.15.12:0/606626558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563de0ff5a90 0x563de0ffa590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.825+0000 7fbbeab85740 1 --2- 172.21.15.12:0/606626558 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ff50a0 0x563de0ff54c0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.825+0000 7fbbeab85740 1 -- 172.21.15.12:0/606626558 >> 172.21.15.12:0/606626558 conn(0x563de0ff0610 msgr2=0x563de0ff0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:06.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.825+0000 7fbbeab85740 1 -- 172.21.15.12:0/606626558 shutdown_connections 2024-03-23T23:08:06.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.825+0000 7fbbeab85740 1 -- 172.21.15.12:0/606626558 wait complete. 2024-03-23T23:08:06.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.826+0000 7fbbe5400640 1 --2- 172.21.15.12:0/2631215648 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbbc80a2800 0x7fbbc80a4cc0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc02c170 tx=0x7fbbcc02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:06.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.826+0000 7fbbeab85740 1 -- start start 2024-03-23T23:08:06.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.826+0000 7fbbeab85740 1 Processor -- start 2024-03-23T23:08:06.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.827+0000 7fbbeab85740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563de0ff50a0 0x563de0fff220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:06.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.827+0000 7fbbeab85740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563de0ff5a90 0x563de0fff760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:06.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.827+0000 7fbbeab85740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ffac00 0x563de10067a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:06.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.827+0000 7fbbeab85740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563de0ffdbb0 con 0x563de0ff5a90 2024-03-23T23:08:06.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.827+0000 7fbbeab85740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x563de0f16eb0 con 0x563de0ffac00 2024-03-23T23:08:06.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.827+0000 7fbbeab85740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x563de0ffdd30 con 0x563de0ff50a0 2024-03-23T23:08:06.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.827+0000 7fbbeab85740 10 client.0 _ll_register_callbacks cb 0x563de0f0b150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:06.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.827+0000 7fbbe5400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563de0ff50a0 0x563de0fff220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:06.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.827+0000 7fbbe5400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563de0ff50a0 0x563de0fff220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44702/0 (socket says 172.21.15.12:44702) 2024-03-23T23:08:06.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.827+0000 7fbbe5400640 1 -- 172.21.15.12:0/116068678 learned_addr learned my addr 172.21.15.12:0/116068678 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:06.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.827+0000 7fbbe4a00640 1 --2- 172.21.15.12:0/116068678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563de0ff5a90 0x563de0fff760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:06.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.827+0000 7fbbe5e00640 1 --2- 172.21.15.12:0/116068678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ffac00 0x563de10067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:06.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.827+0000 7fbbe5e00640 1 -- 172.21.15.12:0/116068678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563de0ff50a0 msgr2=0x563de0fff220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:06.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.827+0000 7fbbe5e00640 1 --2- 172.21.15.12:0/116068678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563de0ff50a0 0x563de0fff220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.827+0000 7fbbe5e00640 1 -- 172.21.15.12:0/116068678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563de0ff5a90 msgr2=0x563de0fff760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:06.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.827+0000 7fbbe5e00640 1 --2- 172.21.15.12:0/116068678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563de0ff5a90 0x563de0fff760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.827+0000 7fbbe5e00640 1 -- 172.21.15.12:0/116068678 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563de0d97f70 con 0x563de0ffac00 2024-03-23T23:08:06.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.827+0000 7fbbe5400640 1 --2- 172.21.15.12:0/116068678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563de0ff50a0 0x563de0fff220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:06.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.828+0000 7fbbe5e00640 1 --2- 172.21.15.12:0/116068678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ffac00 0x563de10067a0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fbbd801b040 tx=0x7fbbd8004430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=707e8cba4699d90c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:06.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.828+0000 7fbbc6a00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:08:06.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.828+0000 7fbbc6a00640 1 -- 172.21.15.12:0/116068678 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd8028020 con 0x563de0ffac00 2024-03-23T23:08:06.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.828+0000 7fbbeab85740 1 -- 172.21.15.12:0/116068678 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbbd8004050 con 0x563de0ffac00 2024-03-23T23:08:06.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.828+0000 7fbbc6a00640 1 -- 172.21.15.12:0/116068678 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fbbd8004c50 con 0x563de0ffac00 2024-03-23T23:08:06.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.829+0000 7fbbc6a00640 1 -- 172.21.15.12:0/116068678 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbbd8004e50 con 0x563de0ffac00 2024-03-23T23:08:06.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.829+0000 7fbbeab85740 10 client.27098 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:06.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.829+0000 7fbbeab85740 1 -- 172.21.15.12:0/116068678 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563de0c2f170 con 0x563de0ffac00 2024-03-23T23:08:06.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.829+0000 7fbbc6a00640 1 -- 172.21.15.12:0/116068678 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbbd8004050 con 0x563de0ffac00 2024-03-23T23:08:06.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.829+0000 7fbbc6a00640 1 -- 172.21.15.12:0/116068678 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fbbd800f570 con 0x563de0ffac00 2024-03-23T23:08:06.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.829+0000 7fbbeab85740 10 client.27098 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:06.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.829+0000 7fbbeab85740 10 client.27098 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:08:06.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.829+0000 7fbbeab85740 20 client.27098 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:06.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.829+0000 7fbbeab85740 1 --2- 172.21.15.12:0/116068678 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563de100ade0 0x563de100d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:06.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.829+0000 7fbbeab85740 4 client.27098 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:08:06.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.829+0000 7fbbeab85740 1 -- 172.21.15.12:0/116068678 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563de0f683f0 con 0x563de100ade0 2024-03-23T23:08:06.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.830+0000 7fbbe5400640 1 --2- 172.21.15.12:0/116068678 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563de100ade0 0x563de100d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:06.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.830+0000 7fbbe5400640 1 --2- 172.21.15.12:0/116068678 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563de100ade0 0x563de100d1d0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a25957bd25bbc75a server_cookie=8d5b147577ce6701 in_seq=0 out_seq=0 2024-03-23T23:08:06.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.830+0000 7fbbc6a00640 10 client.27098 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:06.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.831+0000 7fbbc6a00640 1 -- 172.21.15.12:0/116068678 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563de0f683f0 con 0x563de100ade0 2024-03-23T23:08:06.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.831+0000 7fbbc6a00640 10 client.27098 handle_command_reply: tid=0 2024-03-23T23:08:06.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.831+0000 7fbbeab85740 1 client.27098 shutdown 2024-03-23T23:08:06.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.832+0000 7fbbeab85740 1 -- 172.21.15.12:0/116068678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ffac00 msgr2=0x563de10067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:06.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.832+0000 7fbbeab85740 1 --2- 172.21.15.12:0/116068678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ffac00 0x563de10067a0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fbbd801b040 tx=0x7fbbd8004430 comp rx=0 tx=0).stop 2024-03-23T23:08:06.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.832+0000 7fbbeab85740 1 -- 172.21.15.12:0/116068678 shutdown_connections 2024-03-23T23:08:06.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.832+0000 7fbbeab85740 1 --2- 172.21.15.12:0/116068678 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563de100ade0 0x563de100d1d0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.832+0000 7fbbeab85740 1 --2- 172.21.15.12:0/116068678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ffac00 0x563de10067a0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.832+0000 7fbbeab85740 1 --2- 172.21.15.12:0/116068678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563de0ff5a90 0x563de0fff760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.832+0000 7fbbc6a00640 0 client.27098 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:06.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.832+0000 7fbbeab85740 1 --2- 172.21.15.12:0/116068678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563de0ff50a0 0x563de0fff220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.832+0000 7fbbeab85740 1 -- 172.21.15.12:0/116068678 >> 172.21.15.12:0/116068678 conn(0x563de0ff0610 msgr2=0x563de0fef830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:06.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.832+0000 7fbbeab85740 1 -- 172.21.15.12:0/116068678 shutdown_connections 2024-03-23T23:08:06.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.832+0000 7fbbeab85740 1 -- 172.21.15.12:0/116068678 wait complete. 2024-03-23T23:08:06.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.832+0000 7fbbeab85740 20 client.27098 trim_cache size 0 max 16384 2024-03-23T23:08:06.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.850+0000 7fbbeab85740 1 Processor -- start 2024-03-23T23:08:06.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.850+0000 7fbbeab85740 1 -- start start 2024-03-23T23:08:06.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.851+0000 7fbbeab85740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563de0ff50a0 0x563de0ff3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:06.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.851+0000 7fbbeab85740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ff5a90 0x563de10066c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:06.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.851+0000 7fbbeab85740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563de0ffac00 0x563de1011a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:06.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.851+0000 7fbbeab85740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe01ad250 con 0x563de0ff50a0 2024-03-23T23:08:06.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.851+0000 7fbbeab85740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe01aad50 con 0x563de0ff5a90 2024-03-23T23:08:06.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.851+0000 7fbbeab85740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe01a8df0 con 0x563de0ffac00 2024-03-23T23:08:06.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.851+0000 7fbbe4a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ff5a90 0x563de10066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:06.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.851+0000 7fbbe4a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ff5a90 0x563de10066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:45042/0 (socket says 172.21.15.12:45042) 2024-03-23T23:08:06.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.851+0000 7fbbe5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563de0ffac00 0x563de1011a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:06.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.851+0000 7fbbe4a00640 1 -- 172.21.15.12:0/2846985476 learned_addr learned my addr 172.21.15.12:0/2846985476 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:06.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.851+0000 7fbbe5400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563de0ff50a0 0x563de0ff3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:06.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.851+0000 7fbbe4a00640 1 -- 172.21.15.12:0/2846985476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563de0ffac00 msgr2=0x563de1011a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:06.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.851+0000 7fbbe4a00640 1 --2- 172.21.15.12:0/2846985476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563de0ffac00 0x563de1011a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.851+0000 7fbbe4a00640 1 -- 172.21.15.12:0/2846985476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563de0ff50a0 msgr2=0x563de0ff3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:06.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.851+0000 7fbbe4a00640 1 --2- 172.21.15.12:0/2846985476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563de0ff50a0 0x563de0ff3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.851+0000 7fbbe4a00640 1 -- 172.21.15.12:0/2846985476 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563de0ff0440 con 0x563de0ff5a90 2024-03-23T23:08:06.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.851+0000 7fbbe5e00640 1 --2- 172.21.15.12:0/2846985476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563de0ffac00 0x563de1011a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:06.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.852+0000 7fbbe4a00640 1 --2- 172.21.15.12:0/2846985476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ff5a90 0x563de10066c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fbbd40d2610 tx=0x7fbbd40d2460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16cf89834b5c6390 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:06.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.852+0000 7fbbd1c00640 1 -- 172.21.15.12:0/2846985476 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd409b550 con 0x563de0ff5a90 2024-03-23T23:08:06.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.852+0000 7fbbd1c00640 1 -- 172.21.15.12:0/2846985476 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbbd409be80 con 0x563de0ff5a90 2024-03-23T23:08:06.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.852+0000 7fbbd1c00640 1 -- 172.21.15.12:0/2846985476 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd40998c0 con 0x563de0ff5a90 2024-03-23T23:08:06.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.853+0000 7fbbeab85740 1 -- 172.21.15.12:0/2846985476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ff5a90 msgr2=0x563de10066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:06.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.853+0000 7fbbeab85740 1 --2- 172.21.15.12:0/2846985476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ff5a90 0x563de10066c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fbbd40d2610 tx=0x7fbbd40d2460 comp rx=0 tx=0).stop 2024-03-23T23:08:06.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.853+0000 7fbbeab85740 1 -- 172.21.15.12:0/2846985476 shutdown_connections 2024-03-23T23:08:06.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.853+0000 7fbbeab85740 1 --2- 172.21.15.12:0/2846985476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563de0ffac00 0x563de1011a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.853+0000 7fbbeab85740 1 --2- 172.21.15.12:0/2846985476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ff5a90 0x563de10066c0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.853+0000 7fbbeab85740 1 --2- 172.21.15.12:0/2846985476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563de0ff50a0 0x563de0ff3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.853+0000 7fbbeab85740 1 -- 172.21.15.12:0/2846985476 >> 172.21.15.12:0/2846985476 conn(0x563de0ff0610 msgr2=0x563de0fef830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:06.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.853+0000 7fbbeab85740 1 -- 172.21.15.12:0/2846985476 shutdown_connections 2024-03-23T23:08:06.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.853+0000 7fbbeab85740 1 -- 172.21.15.12:0/2846985476 wait complete. 2024-03-23T23:08:06.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.853+0000 7fbbeab85740 1 -- start start 2024-03-23T23:08:06.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.854+0000 7fbbeab85740 1 Processor -- start 2024-03-23T23:08:06.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.854+0000 7fbbeab85740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ff50a0 0x563de1006590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:06.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.854+0000 7fbbeab85740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563de0ff5a90 0x563de1004ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:06.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.854+0000 7fbbeab85740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563de0ffac00 0x563de10050e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:06.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.854+0000 7fbbeab85740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe01a93f0 con 0x563de0ffac00 2024-03-23T23:08:06.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.854+0000 7fbbeab85740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe01aa6f0 con 0x563de0ff50a0 2024-03-23T23:08:06.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.854+0000 7fbbeab85740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe01a9a30 con 0x563de0ff5a90 2024-03-23T23:08:06.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.854+0000 7fbbeab85740 10 client.0 _ll_register_callbacks cb 0x7fbbd4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:06.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.854+0000 7fbbe5400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ff50a0 0x563de1006590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:06.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.854+0000 7fbbe5400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ff50a0 0x563de1006590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:45056/0 (socket says 172.21.15.12:45056) 2024-03-23T23:08:06.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.854+0000 7fbbe5400640 1 -- 172.21.15.12:0/4015838767 learned_addr learned my addr 172.21.15.12:0/4015838767 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:06.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.854+0000 7fbbe4a00640 1 --2- 172.21.15.12:0/4015838767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563de0ff5a90 0x563de1004ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:06.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.855+0000 7fbbe5e00640 1 --2- 172.21.15.12:0/4015838767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563de0ffac00 0x563de10050e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:06.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.855+0000 7fbbe5400640 1 -- 172.21.15.12:0/4015838767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563de0ff5a90 msgr2=0x563de1004ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:06.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.855+0000 7fbbe5400640 1 --2- 172.21.15.12:0/4015838767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563de0ff5a90 0x563de1004ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.855+0000 7fbbe5400640 1 -- 172.21.15.12:0/4015838767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563de0ffac00 msgr2=0x563de10050e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:06.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.855+0000 7fbbe5400640 1 --2- 172.21.15.12:0/4015838767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563de0ffac00 0x563de10050e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.855+0000 7fbbe5400640 1 -- 172.21.15.12:0/4015838767 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563de0ff1080 con 0x563de0ff50a0 2024-03-23T23:08:06.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.855+0000 7fbbe5e00640 1 --2- 172.21.15.12:0/4015838767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563de0ffac00 0x563de10050e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:08:06.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.855+0000 7fbbe5400640 1 --2- 172.21.15.12:0/4015838767 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ff50a0 0x563de1006590 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc009900 tx=0x7fbbcc009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2aff8f153417ca79 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:06.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.855+0000 7fbbc6a00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:08:06.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.855+0000 7fbbc6a00640 1 -- 172.21.15.12:0/4015838767 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc064060 con 0x563de0ff50a0 2024-03-23T23:08:06.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.856+0000 7fbbc6a00640 1 -- 172.21.15.12:0/4015838767 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fbbcc0579c0 con 0x563de0ff50a0 2024-03-23T23:08:06.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.856+0000 7fbbeab85740 1 -- 172.21.15.12:0/4015838767 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbbd409be80 con 0x563de0ff50a0 2024-03-23T23:08:06.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.856+0000 7fbbc6a00640 1 -- 172.21.15.12:0/4015838767 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbbcc045070 con 0x563de0ff50a0 2024-03-23T23:08:06.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.856+0000 7fbbeab85740 10 client.27104 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:06.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.856+0000 7fbbeab85740 1 -- 172.21.15.12:0/4015838767 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563de0ffde70 con 0x563de0ff50a0 2024-03-23T23:08:06.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.856+0000 7fbbc6a00640 1 -- 172.21.15.12:0/4015838767 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbbd409be80 con 0x563de0ff50a0 2024-03-23T23:08:06.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.856+0000 7fbbc6a00640 1 -- 172.21.15.12:0/4015838767 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fbbcc05e030 con 0x563de0ff50a0 2024-03-23T23:08:06.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.856+0000 7fbbeab85740 10 client.27104 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:06.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.856+0000 7fbbeab85740 10 client.27104 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:08:06.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.856+0000 7fbbeab85740 20 client.27104 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:06.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.856+0000 7fbbeab85740 1 --2- 172.21.15.12:0/4015838767 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563de1067d00 0x563de106a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:06.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.856+0000 7fbbeab85740 4 client.27104 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-23T23:08:06.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.856+0000 7fbbeab85740 1 -- 172.21.15.12:0/4015838767 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x563de0f62740 con 0x563de1067d00 2024-03-23T23:08:06.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.857+0000 7fbbe4a00640 1 --2- 172.21.15.12:0/4015838767 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563de1067d00 0x563de106a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:06.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.857+0000 7fbbe4a00640 1 --2- 172.21.15.12:0/4015838767 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563de1067d00 0x563de106a120 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9850566ba374a700 server_cookie=be5b2ed2af545271 in_seq=0 out_seq=0 2024-03-23T23:08:06.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.857+0000 7fbbc6a00640 10 client.27104 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:06.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.859+0000 7fbbc6a00640 1 -- 172.21.15.12:0/4015838767 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x7fbbd40d7080 con 0x563de1067d00 2024-03-23T23:08:06.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.859+0000 7fbbc6a00640 10 client.27104 handle_command_reply: tid=0 2024-03-23T23:08:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.859+0000 7fbbeab85740 1 client.27104 shutdown 2024-03-23T23:08:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.859+0000 7fbbeab85740 1 -- 172.21.15.12:0/4015838767 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ff50a0 msgr2=0x563de1006590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.859+0000 7fbbeab85740 1 --2- 172.21.15.12:0/4015838767 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ff50a0 0x563de1006590 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc009900 tx=0x7fbbcc009260 comp rx=0 tx=0).stop 2024-03-23T23:08:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.859+0000 7fbbeab85740 1 -- 172.21.15.12:0/4015838767 shutdown_connections 2024-03-23T23:08:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.859+0000 7fbbeab85740 1 --2- 172.21.15.12:0/4015838767 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563de1067d00 0x563de106a120 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.859+0000 7fbbeab85740 1 --2- 172.21.15.12:0/4015838767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563de0ffac00 0x563de10050e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.859+0000 7fbbeab85740 1 --2- 172.21.15.12:0/4015838767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563de0ff5a90 0x563de1004ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.859+0000 7fbbeab85740 1 --2- 172.21.15.12:0/4015838767 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563de0ff50a0 0x563de1006590 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.859+0000 7fbbeab85740 1 -- 172.21.15.12:0/4015838767 >> 172.21.15.12:0/4015838767 conn(0x563de0ff0610 msgr2=0x563de1011a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.859+0000 7fbbc6a00640 0 client.27104 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.859+0000 7fbbeab85740 1 -- 172.21.15.12:0/4015838767 shutdown_connections 2024-03-23T23:08:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.859+0000 7fbbeab85740 1 -- 172.21.15.12:0/4015838767 wait complete. 2024-03-23T23:08:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.859+0000 7fbbeab85740 20 client.27104 trim_cache size 0 max 16384 2024-03-23T23:08:06.863 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:08:06.863 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-03-23T23:08:06.863 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "dbf24598-1d3b-4c8a-a1b0-d3f77890ec1d", 2024-03-23T23:08:06.863 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-03-23T23:08:06.863 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:08:06.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.860+0000 7fbbe7e00640 1 -- 172.21.15.12:0/2631215648 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbbc80a2800 msgr2=0x7fbbc80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:06.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.860+0000 7fbbe7e00640 1 --2- 172.21.15.12:0/2631215648 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbbc80a2800 0x7fbbc80a4cc0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc02c170 tx=0x7fbbcc02df90 comp rx=0 tx=0).stop 2024-03-23T23:08:06.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.860+0000 7fbbe7e00640 1 -- 172.21.15.12:0/2631215648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbe0104ad0 msgr2=0x7fbbe019a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:06.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.860+0000 7fbbe7e00640 1 --2- 172.21.15.12:0/2631215648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbe0104ad0 0x7fbbe019a180 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fbbd400ea60 tx=0x7fbbd400ef20 comp rx=0 tx=0).stop 2024-03-23T23:08:06.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.861+0000 7fbbe7e00640 1 -- 172.21.15.12:0/2631215648 shutdown_connections 2024-03-23T23:08:06.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.861+0000 7fbbe7e00640 1 --2- 172.21.15.12:0/2631215648 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbbc80a2800 0x7fbbc80a4cc0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.861+0000 7fbbe7e00640 1 --2- 172.21.15.12:0/2631215648 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbbe0105490 0x7fbbe01a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.861+0000 7fbbe7e00640 1 --2- 172.21.15.12:0/2631215648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbe0104ad0 0x7fbbe019a180 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.861+0000 7fbbe7e00640 1 --2- 172.21.15.12:0/2631215648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbe01038e0 0x7fbbe0199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:06.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.861+0000 7fbbe7e00640 1 -- 172.21.15.12:0/2631215648 >> 172.21.15.12:0/2631215648 conn(0x7fbbe00fee50 msgr2=0x7fbbe0107e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:06.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.861+0000 7fbbe7e00640 1 -- 172.21.15.12:0/2631215648 shutdown_connections 2024-03-23T23:08:06.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:06.861+0000 7fbbe7e00640 1 -- 172.21.15.12:0/2631215648 wait complete. 2024-03-23T23:08:06.877 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:08:06.877 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:08:06.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:06 smithi012 ceph-mon[31695]: pgmap v579: 97 pgs: 97 active+clean; 4.0 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:08:06.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:06 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:08:07.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.083+0000 7fcfe3e00640 1 -- 172.21.15.12:0/3628101732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfe4104930 msgr2=0x7fcfe4106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:07.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.083+0000 7fcfe3e00640 1 --2- 172.21.15.12:0/3628101732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfe4104930 0x7fcfe4106d20 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0009900 tx=0x7fcfd002bc50 comp rx=0 tx=0).stop 2024-03-23T23:08:07.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.083+0000 7fcfe3e00640 1 -- 172.21.15.12:0/3628101732 shutdown_connections 2024-03-23T23:08:07.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.083+0000 7fcfe3e00640 1 --2- 172.21.15.12:0/3628101732 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfe4107260 0x7fcfe41096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.083+0000 7fcfe3e00640 1 --2- 172.21.15.12:0/3628101732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfe4104930 0x7fcfe4106d20 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.083+0000 7fcfe3e00640 1 --2- 172.21.15.12:0/3628101732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfe4102000 0x7fcfe41043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.083+0000 7fcfe3e00640 1 -- 172.21.15.12:0/3628101732 >> 172.21.15.12:0/3628101732 conn(0x7fcfe40fba10 msgr2=0x7fcfe40fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:07.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.083+0000 7fcfe3e00640 1 -- 172.21.15.12:0/3628101732 shutdown_connections 2024-03-23T23:08:07.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.083+0000 7fcfe3e00640 1 -- 172.21.15.12:0/3628101732 wait complete. 2024-03-23T23:08:07.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.084+0000 7fcfe3e00640 1 Processor -- start 2024-03-23T23:08:07.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.085+0000 7fcfe3e00640 1 -- start start 2024-03-23T23:08:07.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.085+0000 7fcfe3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfe4102000 0x7fcfe4101970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.085+0000 7fcfe3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfe4104930 0x7fcfe40fffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.085+0000 7fcfe3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfe4107260 0x7fcfe41004f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.085+0000 7fcfe3e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe410d260 con 0x7fcfe4102000 2024-03-23T23:08:07.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.085+0000 7fcfe3e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe410d0e0 con 0x7fcfe4104930 2024-03-23T23:08:07.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.085+0000 7fcfe3e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe410d3e0 con 0x7fcfe4107260 2024-03-23T23:08:07.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.085+0000 7fcfe2000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfe4104930 0x7fcfe40fffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.085+0000 7fcfe2000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfe4104930 0x7fcfe40fffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:45066/0 (socket says 172.21.15.12:45066) 2024-03-23T23:08:07.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.085+0000 7fcfe3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfe4107260 0x7fcfe41004f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.085+0000 7fcfe2000640 1 -- 172.21.15.12:0/1306024126 learned_addr learned my addr 172.21.15.12:0/1306024126 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:07.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.085+0000 7fcfe2a00640 1 --2- 172.21.15.12:0/1306024126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfe4102000 0x7fcfe4101970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.086+0000 7fcfe2000640 1 -- 172.21.15.12:0/1306024126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfe4107260 msgr2=0x7fcfe41004f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:07.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.086+0000 7fcfe2000640 1 --2- 172.21.15.12:0/1306024126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfe4107260 0x7fcfe41004f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.086+0000 7fcfe2000640 1 -- 172.21.15.12:0/1306024126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfe4102000 msgr2=0x7fcfe4101970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:07.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.086+0000 7fcfe2000640 1 --2- 172.21.15.12:0/1306024126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfe4102000 0x7fcfe4101970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.086+0000 7fcfe2000640 1 -- 172.21.15.12:0/1306024126 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe4100d80 con 0x7fcfe4104930 2024-03-23T23:08:07.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.086+0000 7fcfe3400640 1 --2- 172.21.15.12:0/1306024126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfe4107260 0x7fcfe41004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:07.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.086+0000 7fcfe2a00640 1 --2- 172.21.15.12:0/1306024126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfe4102000 0x7fcfe4101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:07.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.086+0000 7fcfe2000640 1 --2- 172.21.15.12:0/1306024126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfe4104930 0x7fcfe40fffb0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fcfd002c260 tx=0x7fcfd002c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:07.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.087+0000 7fcfdb400640 1 -- 172.21.15.12:0/1306024126 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0041070 con 0x7fcfe4104930 2024-03-23T23:08:07.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.087+0000 7fcfe3e00640 1 -- 172.21.15.12:0/1306024126 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe41a19b0 con 0x7fcfe4104930 2024-03-23T23:08:07.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.087+0000 7fcfdb400640 1 -- 172.21.15.12:0/1306024126 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfd0004450 con 0x7fcfe4104930 2024-03-23T23:08:07.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.087+0000 7fcfdb400640 1 -- 172.21.15.12:0/1306024126 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0034440 con 0x7fcfe4104930 2024-03-23T23:08:07.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.087+0000 7fcfe3e00640 1 -- 172.21.15.12:0/1306024126 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe41a1f50 con 0x7fcfe4104930 2024-03-23T23:08:07.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.091+0000 7fcfec591740 1 Processor -- start 2024-03-23T23:08:07.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.091+0000 7fcfec591740 1 -- start start 2024-03-23T23:08:07.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.091+0000 7fcfec591740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622a7f5c050 0x5622a7f5c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.091+0000 7fcfec591740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622a7f5ca40 0x5622a7f61540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.091+0000 7fcfec591740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5622a7f61bb0 0x5622a7f64040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.092+0000 7fcfec591740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622a7ec1250 con 0x5622a7f61bb0 2024-03-23T23:08:07.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.092+0000 7fcfec591740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622a7e69a30 con 0x5622a7f5c050 2024-03-23T23:08:07.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.092+0000 7fcfec591740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622a7ebd2f0 con 0x5622a7f5ca40 2024-03-23T23:08:07.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.092+0000 7fcfe2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622a7f5ca40 0x5622a7f61540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.092+0000 7fcfe3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622a7f5c050 0x5622a7f5c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.092+0000 7fcfe2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622a7f5ca40 0x5622a7f61540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44736/0 (socket says 172.21.15.12:44736) 2024-03-23T23:08:07.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.092+0000 7fcfe2a00640 1 -- 172.21.15.12:0/4221797645 learned_addr learned my addr 172.21.15.12:0/4221797645 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:07.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.092+0000 7fcfdb400640 1 -- 172.21.15.12:0/1306024126 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcfd0044070 con 0x7fcfe4104930 2024-03-23T23:08:07.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.092+0000 7fcfe3400640 1 -- 172.21.15.12:0/4221797645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622a7f5ca40 msgr2=0x5622a7f61540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:07.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.092+0000 7fcfe3400640 1 --2- 172.21.15.12:0/4221797645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622a7f5ca40 0x5622a7f61540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.092+0000 7fcfe3400640 1 -- 172.21.15.12:0/4221797645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5622a7f61bb0 msgr2=0x5622a7f64040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:07.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.092+0000 7fcfe3400640 1 --2- 172.21.15.12:0/4221797645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5622a7f61bb0 0x5622a7f64040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.092+0000 7fcfe3400640 1 -- 172.21.15.12:0/4221797645 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622a7e54690 con 0x5622a7f5c050 2024-03-23T23:08:07.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.093+0000 7fcfe3400640 1 --2- 172.21.15.12:0/4221797645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622a7f5c050 0x5622a7f5c470 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc005fd0 tx=0x7fcfdc011bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7317dc077a7f2ef8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:07.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.093+0000 7fcfcfe00640 1 -- 172.21.15.12:0/4221797645 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc004d20 con 0x5622a7f5c050 2024-03-23T23:08:07.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.093+0000 7fcfcfe00640 1 -- 172.21.15.12:0/4221797645 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfdc00f9a0 con 0x5622a7f5c050 2024-03-23T23:08:07.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.093+0000 7fcfcfe00640 1 -- 172.21.15.12:0/4221797645 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc004510 con 0x5622a7f5c050 2024-03-23T23:08:07.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.093+0000 7fcfdb400640 1 --2- 172.21.15.12:0/1306024126 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcfc80a2800 0x7fcfc80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.094+0000 7fcfec591740 1 -- 172.21.15.12:0/4221797645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622a7f5c050 msgr2=0x5622a7f5c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:07.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.094+0000 7fcfec591740 1 --2- 172.21.15.12:0/4221797645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622a7f5c050 0x5622a7f5c470 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc005fd0 tx=0x7fcfdc011bf0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.094+0000 7fcfdb400640 1 -- 172.21.15.12:0/1306024126 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fcfd00f04f0 con 0x7fcfe4104930 2024-03-23T23:08:07.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.094+0000 7fcfec591740 1 -- 172.21.15.12:0/4221797645 shutdown_connections 2024-03-23T23:08:07.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.094+0000 7fcfec591740 1 --2- 172.21.15.12:0/4221797645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5622a7f61bb0 0x5622a7f64040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.094+0000 7fcfec591740 1 --2- 172.21.15.12:0/4221797645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622a7f5ca40 0x5622a7f61540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.094+0000 7fcfec591740 1 --2- 172.21.15.12:0/4221797645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622a7f5c050 0x5622a7f5c470 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.094+0000 7fcfec591740 1 -- 172.21.15.12:0/4221797645 >> 172.21.15.12:0/4221797645 conn(0x5622a7f24200 msgr2=0x5622a7f579c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:07.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.094+0000 7fcfe2a00640 1 --2- 172.21.15.12:0/1306024126 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcfc80a2800 0x7fcfc80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.094+0000 7fcfec591740 1 -- 172.21.15.12:0/4221797645 shutdown_connections 2024-03-23T23:08:07.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.094+0000 7fcfec591740 1 -- 172.21.15.12:0/4221797645 wait complete. 2024-03-23T23:08:07.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.095+0000 7fcfec591740 1 -- start start 2024-03-23T23:08:07.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.095+0000 7fcfe2a00640 1 --2- 172.21.15.12:0/1306024126 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcfc80a2800 0x7fcfc80a4cc0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x5622a7f58910 tx=0x7fcfd4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:07.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.095+0000 7fcfec591740 1 Processor -- start 2024-03-23T23:08:07.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.095+0000 7fcfec591740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622a7f5c050 0x5622a7f660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.095+0000 7fcfec591740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622a7f5ca40 0x5622a7f665f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.095+0000 7fcfec591740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5622a7f61bb0 0x5622a7f6d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.095+0000 7fcfec591740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622a7f64b60 con 0x5622a7f61bb0 2024-03-23T23:08:07.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.095+0000 7fcfec591740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622a7e7ce80 con 0x5622a7f5ca40 2024-03-23T23:08:07.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.095+0000 7fcfec591740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622a7f64ce0 con 0x5622a7f5c050 2024-03-23T23:08:07.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.096+0000 7fcfec591740 10 client.0 _ll_register_callbacks cb 0x5622a7e8f130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:07.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.096+0000 7fcfe2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622a7f5c050 0x5622a7f660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.096+0000 7fcfe2000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622a7f5ca40 0x5622a7f665f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.096+0000 7fcfe2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622a7f5c050 0x5622a7f660b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44744/0 (socket says 172.21.15.12:44744) 2024-03-23T23:08:07.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.096+0000 7fcfe2a00640 1 -- 172.21.15.12:0/3525202208 learned_addr learned my addr 172.21.15.12:0/3525202208 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:07.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.096+0000 7fcfe3400640 1 --2- 172.21.15.12:0/3525202208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5622a7f61bb0 0x5622a7f6d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.096+0000 7fcfe2000640 1 -- 172.21.15.12:0/3525202208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622a7f5c050 msgr2=0x5622a7f660b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:07.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.096+0000 7fcfe2000640 1 --2- 172.21.15.12:0/3525202208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622a7f5c050 0x5622a7f660b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.096+0000 7fcfe2000640 1 -- 172.21.15.12:0/3525202208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5622a7f61bb0 msgr2=0x5622a7f6d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:07.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.096+0000 7fcfe2000640 1 --2- 172.21.15.12:0/3525202208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5622a7f61bb0 0x5622a7f6d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.096+0000 7fcfe2000640 1 -- 172.21.15.12:0/3525202208 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5622a7c9f7d0 con 0x5622a7f5ca40 2024-03-23T23:08:07.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.096+0000 7fcfe3400640 1 --2- 172.21.15.12:0/3525202208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5622a7f61bb0 0x5622a7f6d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:08:07.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.096+0000 7fcfe2000640 1 --2- 172.21.15.12:0/3525202208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622a7f5ca40 0x5622a7f665f0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fcfd00ee000 tx=0x7fcfd00bcde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbb51454ebc22d4d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:07.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.096+0000 7fcfccc00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:08:07.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.097+0000 7fcfccc00640 1 -- 172.21.15.12:0/3525202208 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd004b040 con 0x5622a7f5ca40 2024-03-23T23:08:07.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.097+0000 7fcfccc00640 1 -- 172.21.15.12:0/3525202208 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fcfd00035f0 con 0x5622a7f5ca40 2024-03-23T23:08:07.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.097+0000 7fcfec591740 1 -- 172.21.15.12:0/3525202208 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcfdc00f9a0 con 0x5622a7f5ca40 2024-03-23T23:08:07.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.097+0000 7fcfccc00640 1 -- 172.21.15.12:0/3525202208 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcfd00bcf50 con 0x5622a7f5ca40 2024-03-23T23:08:07.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.097+0000 7fcfec591740 10 client.27116 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:07.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.097+0000 7fcfec591740 1 -- 172.21.15.12:0/3525202208 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5622a7bd53f0 con 0x5622a7f5ca40 2024-03-23T23:08:07.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.097+0000 7fcfccc00640 1 -- 172.21.15.12:0/3525202208 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfdc00f9a0 con 0x5622a7f5ca40 2024-03-23T23:08:07.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.097+0000 7fcfccc00640 1 -- 172.21.15.12:0/3525202208 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fcfd00c51b0 con 0x5622a7f5ca40 2024-03-23T23:08:07.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.097+0000 7fcfec591740 10 client.27116 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:07.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.097+0000 7fcfec591740 10 client.27116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:08:07.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.097+0000 7fcfec591740 20 client.27116 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:07.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.097+0000 7fcfec591740 1 --2- 172.21.15.12:0/3525202208 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5622a7f71f00 0x5622a7f742f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.097+0000 7fcfec591740 4 client.27116 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:08:07.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.097+0000 7fcfec591740 1 -- 172.21.15.12:0/3525202208 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5622a7ece790 con 0x5622a7f71f00 2024-03-23T23:08:07.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.098+0000 7fcfe2a00640 1 --2- 172.21.15.12:0/3525202208 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5622a7f71f00 0x5622a7f742f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.098+0000 7fcfe2a00640 1 --2- 172.21.15.12:0/3525202208 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5622a7f71f00 0x5622a7f742f0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31f599481e6edd5b server_cookie=4a49b2c92fcbdbe6 in_seq=0 out_seq=0 2024-03-23T23:08:07.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.098+0000 7fcfccc00640 10 client.27116 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:07.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.099+0000 7fcfccc00640 1 -- 172.21.15.12:0/3525202208 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5622a7ece790 con 0x5622a7f71f00 2024-03-23T23:08:07.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.099+0000 7fcfccc00640 10 client.27116 handle_command_reply: tid=0 2024-03-23T23:08:07.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.100+0000 7fcfec591740 1 client.27116 shutdown 2024-03-23T23:08:07.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.100+0000 7fcfec591740 1 -- 172.21.15.12:0/3525202208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622a7f5ca40 msgr2=0x5622a7f665f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:07.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.100+0000 7fcfec591740 1 --2- 172.21.15.12:0/3525202208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622a7f5ca40 0x5622a7f665f0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fcfd00ee000 tx=0x7fcfd00bcde0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.100+0000 7fcfec591740 1 -- 172.21.15.12:0/3525202208 shutdown_connections 2024-03-23T23:08:07.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.100+0000 7fcfec591740 1 --2- 172.21.15.12:0/3525202208 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5622a7f71f00 0x5622a7f742f0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.100+0000 7fcfec591740 1 --2- 172.21.15.12:0/3525202208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5622a7f61bb0 0x5622a7f6d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.100+0000 7fcfec591740 1 --2- 172.21.15.12:0/3525202208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622a7f5ca40 0x5622a7f665f0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.100+0000 7fcfec591740 1 --2- 172.21.15.12:0/3525202208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622a7f5c050 0x5622a7f660b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.100+0000 7fcfccc00640 0 client.27116 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:07.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.100+0000 7fcfec591740 1 -- 172.21.15.12:0/3525202208 >> 172.21.15.12:0/3525202208 conn(0x5622a7f24200 msgr2=0x5622a7f56bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:07.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.100+0000 7fcfec591740 1 -- 172.21.15.12:0/3525202208 shutdown_connections 2024-03-23T23:08:07.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.100+0000 7fcfec591740 1 -- 172.21.15.12:0/3525202208 wait complete. 2024-03-23T23:08:07.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.100+0000 7fcfec591740 20 client.27116 trim_cache size 0 max 16384 2024-03-23T23:08:07.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.118+0000 7fcfec591740 1 Processor -- start 2024-03-23T23:08:07.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.118+0000 7fcfec591740 1 -- start start 2024-03-23T23:08:07.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.118+0000 7fcfec591740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622a7f5c050 0x5622a7f78bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.118+0000 7fcfec591740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5622a7f5ca40 0x5622a7f6d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.118+0000 7fcfec591740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622a7f61bb0 0x5622a7f790f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.118+0000 7fcfec591740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe41ace30 con 0x5622a7f5ca40 2024-03-23T23:08:07.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.118+0000 7fcfec591740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe41aa930 con 0x5622a7f5c050 2024-03-23T23:08:07.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.118+0000 7fcfec591740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe41a89d0 con 0x5622a7f61bb0 2024-03-23T23:08:07.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.119+0000 7fcfe2a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622a7f5c050 0x5622a7f78bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.119+0000 7fcfe3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622a7f61bb0 0x5622a7f790f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.119+0000 7fcfe2a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622a7f5c050 0x5622a7f78bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:45098/0 (socket says 172.21.15.12:45098) 2024-03-23T23:08:07.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.119+0000 7fcfe2a00640 1 -- 172.21.15.12:0/165651376 learned_addr learned my addr 172.21.15.12:0/165651376 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:07.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.119+0000 7fcfe2000640 1 --2- 172.21.15.12:0/165651376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5622a7f5ca40 0x5622a7f6d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:06 smithi173 ceph-mon[36116]: pgmap v579: 97 pgs: 97 active+clean; 4.0 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:08:07.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:06 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:08:07.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.119+0000 7fcfe2a00640 1 -- 172.21.15.12:0/165651376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622a7f61bb0 msgr2=0x5622a7f790f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:07.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.119+0000 7fcfe2a00640 1 --2- 172.21.15.12:0/165651376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622a7f61bb0 0x5622a7f790f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.119+0000 7fcfe2a00640 1 -- 172.21.15.12:0/165651376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5622a7f5ca40 msgr2=0x5622a7f6d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:07.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.119+0000 7fcfe2a00640 1 --2- 172.21.15.12:0/165651376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5622a7f5ca40 0x5622a7f6d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.119+0000 7fcfe2a00640 1 -- 172.21.15.12:0/165651376 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622a7f74660 con 0x5622a7f5c050 2024-03-23T23:08:07.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.119+0000 7fcfe2a00640 1 --2- 172.21.15.12:0/165651376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622a7f5c050 0x5622a7f78bb0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4012290 tx=0x7fcfd401a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0e517705d3be64 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:07.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.120+0000 7fcfcf400640 1 -- 172.21.15.12:0/165651376 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4004340 con 0x5622a7f5c050 2024-03-23T23:08:07.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.120+0000 7fcfcf400640 1 -- 172.21.15.12:0/165651376 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfd4005020 con 0x5622a7f5c050 2024-03-23T23:08:07.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.120+0000 7fcfcf400640 1 -- 172.21.15.12:0/165651376 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4004bc0 con 0x5622a7f5c050 2024-03-23T23:08:07.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.120+0000 7fcfec591740 1 -- 172.21.15.12:0/165651376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622a7f5c050 msgr2=0x5622a7f78bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:07.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.120+0000 7fcfec591740 1 --2- 172.21.15.12:0/165651376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622a7f5c050 0x5622a7f78bb0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4012290 tx=0x7fcfd401a000 comp rx=0 tx=0).stop 2024-03-23T23:08:07.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.120+0000 7fcfec591740 1 -- 172.21.15.12:0/165651376 shutdown_connections 2024-03-23T23:08:07.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.120+0000 7fcfec591740 1 --2- 172.21.15.12:0/165651376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622a7f61bb0 0x5622a7f790f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.120+0000 7fcfec591740 1 --2- 172.21.15.12:0/165651376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5622a7f5ca40 0x5622a7f6d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.120+0000 7fcfec591740 1 --2- 172.21.15.12:0/165651376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622a7f5c050 0x5622a7f78bb0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.120+0000 7fcfec591740 1 -- 172.21.15.12:0/165651376 >> 172.21.15.12:0/165651376 conn(0x5622a7f24200 msgr2=0x5622a7f5dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:07.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.121+0000 7fcfec591740 1 -- 172.21.15.12:0/165651376 shutdown_connections 2024-03-23T23:08:07.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.121+0000 7fcfec591740 1 -- 172.21.15.12:0/165651376 wait complete. 2024-03-23T23:08:07.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.121+0000 7fcfec591740 1 -- start start 2024-03-23T23:08:07.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.121+0000 7fcfec591740 1 Processor -- start 2024-03-23T23:08:07.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.122+0000 7fcfec591740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622a7f5c050 0x5622a7f6b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.122+0000 7fcfec591740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622a7f5ca40 0x5622a7f6ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.122+0000 7fcfec591740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5622a7f61bb0 0x5622a7f6ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.122+0000 7fcfec591740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe41a8fd0 con 0x5622a7f61bb0 2024-03-23T23:08:07.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.122+0000 7fcfec591740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe41aa2d0 con 0x5622a7f5ca40 2024-03-23T23:08:07.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.122+0000 7fcfec591740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe41a9610 con 0x5622a7f5c050 2024-03-23T23:08:07.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.122+0000 7fcfec591740 10 client.0 _ll_register_callbacks cb 0x7fcfd0009580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:07.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.122+0000 7fcfe2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622a7f5c050 0x5622a7f6b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.122+0000 7fcfe2000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622a7f5ca40 0x5622a7f6ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.122+0000 7fcfe2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622a7f5c050 0x5622a7f6b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44758/0 (socket says 172.21.15.12:44758) 2024-03-23T23:08:07.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.122+0000 7fcfe2a00640 1 -- 172.21.15.12:0/3867400432 learned_addr learned my addr 172.21.15.12:0/3867400432 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:07.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.122+0000 7fcfe3400640 1 --2- 172.21.15.12:0/3867400432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5622a7f61bb0 0x5622a7f6ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.122+0000 7fcfe2a00640 1 -- 172.21.15.12:0/3867400432 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622a7f5ca40 msgr2=0x5622a7f6ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:07.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.122+0000 7fcfe2a00640 1 --2- 172.21.15.12:0/3867400432 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622a7f5ca40 0x5622a7f6ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.122+0000 7fcfe2a00640 1 -- 172.21.15.12:0/3867400432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5622a7f61bb0 msgr2=0x5622a7f6ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:07.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.122+0000 7fcfe2a00640 1 --2- 172.21.15.12:0/3867400432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5622a7f61bb0 0x5622a7f6ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.122+0000 7fcfe2a00640 1 -- 172.21.15.12:0/3867400432 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5622a7b950f0 con 0x5622a7f5c050 2024-03-23T23:08:07.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.122+0000 7fcfe2000640 1 --2- 172.21.15.12:0/3867400432 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622a7f5ca40 0x5622a7f6ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:07.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.123+0000 7fcfe2a00640 1 --2- 172.21.15.12:0/3867400432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622a7f5c050 0x5622a7f6b4d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fcfd40123d0 tx=0x7fcfd4005d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c578ef417767087b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:07.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.123+0000 7fcfccc00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:08:07.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.123+0000 7fcfccc00640 1 -- 172.21.15.12:0/3867400432 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4038020 con 0x5622a7f5c050 2024-03-23T23:08:07.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.123+0000 7fcfccc00640 1 -- 172.21.15.12:0/3867400432 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fcfd40051d0 con 0x5622a7f5c050 2024-03-23T23:08:07.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.123+0000 7fcfec591740 1 -- 172.21.15.12:0/3867400432 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcfd4005020 con 0x5622a7f5c050 2024-03-23T23:08:07.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.123+0000 7fcfccc00640 1 -- 172.21.15.12:0/3867400432 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcfd402f040 con 0x5622a7f5c050 2024-03-23T23:08:07.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.123+0000 7fcfec591740 10 client.27094 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:07.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.123+0000 7fcfec591740 1 -- 172.21.15.12:0/3867400432 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5622a7fd6590 con 0x5622a7f5c050 2024-03-23T23:08:07.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.124+0000 7fcfccc00640 1 -- 172.21.15.12:0/3867400432 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfd4005020 con 0x5622a7f5c050 2024-03-23T23:08:07.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.124+0000 7fcfccc00640 1 -- 172.21.15.12:0/3867400432 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fcfd402b680 con 0x5622a7f5c050 2024-03-23T23:08:07.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.124+0000 7fcfec591740 10 client.27094 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:07.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.124+0000 7fcfec591740 10 client.27094 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:08:07.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.124+0000 7fcfec591740 20 client.27094 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:07.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.124+0000 7fcfec591740 1 --2- 172.21.15.12:0/3867400432 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5622a7fd6ec0 0x5622a7fd92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.124+0000 7fcfec591740 4 client.27094 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:08:07.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.124+0000 7fcfec591740 1 -- 172.21.15.12:0/3867400432 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5622a7ec8ae0 con 0x5622a7fd6ec0 2024-03-23T23:08:07.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.124+0000 7fcfe2000640 1 --2- 172.21.15.12:0/3867400432 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5622a7fd6ec0 0x5622a7fd92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.125+0000 7fcfe2000640 1 --2- 172.21.15.12:0/3867400432 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5622a7fd6ec0 0x5622a7fd92e0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=883b6183c412d1cb server_cookie=63d7d1d74cc55bce in_seq=0 out_seq=0 2024-03-23T23:08:07.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.125+0000 7fcfccc00640 10 client.27094 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:07.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.125+0000 7fcfccc00640 1 -- 172.21.15.12:0/3867400432 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5622a7ec8ae0 con 0x5622a7fd6ec0 2024-03-23T23:08:07.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.125+0000 7fcfccc00640 10 client.27094 handle_command_reply: tid=0 2024-03-23T23:08:07.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.125+0000 7fcfec591740 1 client.27094 shutdown 2024-03-23T23:08:07.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.125+0000 7fcfec591740 1 -- 172.21.15.12:0/3867400432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622a7f5c050 msgr2=0x5622a7f6b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:07.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.125+0000 7fcfec591740 1 --2- 172.21.15.12:0/3867400432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622a7f5c050 0x5622a7f6b4d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fcfd40123d0 tx=0x7fcfd4005d30 comp rx=0 tx=0).stop 2024-03-23T23:08:07.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.125+0000 7fcfec591740 1 -- 172.21.15.12:0/3867400432 shutdown_connections 2024-03-23T23:08:07.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.125+0000 7fcfec591740 1 --2- 172.21.15.12:0/3867400432 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5622a7fd6ec0 0x5622a7fd92e0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.125+0000 7fcfec591740 1 --2- 172.21.15.12:0/3867400432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5622a7f61bb0 0x5622a7f6ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.125+0000 7fcfec591740 1 --2- 172.21.15.12:0/3867400432 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622a7f5ca40 0x5622a7f6ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.125+0000 7fcfccc00640 0 client.27094 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:07.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.125+0000 7fcfec591740 1 --2- 172.21.15.12:0/3867400432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622a7f5c050 0x5622a7f6b4d0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.125+0000 7fcfec591740 1 -- 172.21.15.12:0/3867400432 >> 172.21.15.12:0/3867400432 conn(0x5622a7f24200 msgr2=0x5622a7f5ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:07.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.125+0000 7fcfec591740 1 -- 172.21.15.12:0/3867400432 shutdown_connections 2024-03-23T23:08:07.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.125+0000 7fcfec591740 1 -- 172.21.15.12:0/3867400432 wait complete. 2024-03-23T23:08:07.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.126+0000 7fcfec591740 20 client.27094 trim_cache size 0 max 16384 2024-03-23T23:08:07.129 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:08:07.129 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T23:08:07.129 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-03-23T23:08:07.129 INFO:teuthology.orchestra.run.smithi012.stdout: "dbf24598-1d3b-4c8a-a1b0-d3f77890ec1d": { 2024-03-23T23:08:07.129 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-03-23T23:08:07.129 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "dbf24598-1d3b-4c8a-a1b0-d3f77890ec1d", 2024-03-23T23:08:07.129 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-03-23T23:08:07.129 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:08:07.129 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:08:07.130 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:08:07.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.126+0000 7fcfe3e00640 1 -- 172.21.15.12:0/1306024126 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcfc80a2800 msgr2=0x7fcfc80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:07.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.126+0000 7fcfe3e00640 1 --2- 172.21.15.12:0/1306024126 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcfc80a2800 0x7fcfc80a4cc0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x5622a7f58910 tx=0x7fcfd4005f90 comp rx=0 tx=0).stop 2024-03-23T23:08:07.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.126+0000 7fcfe3e00640 1 -- 172.21.15.12:0/1306024126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfe4104930 msgr2=0x7fcfe40fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:07.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.126+0000 7fcfe3e00640 1 --2- 172.21.15.12:0/1306024126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfe4104930 0x7fcfe40fffb0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fcfd002c260 tx=0x7fcfd002c670 comp rx=0 tx=0).stop 2024-03-23T23:08:07.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.126+0000 7fcfe3e00640 1 -- 172.21.15.12:0/1306024126 shutdown_connections 2024-03-23T23:08:07.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.126+0000 7fcfe3e00640 1 --2- 172.21.15.12:0/1306024126 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fcfc80a2800 0x7fcfc80a4cc0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.126+0000 7fcfe3e00640 1 --2- 172.21.15.12:0/1306024126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfe4107260 0x7fcfe41004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.126+0000 7fcfe3e00640 1 --2- 172.21.15.12:0/1306024126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfe4104930 0x7fcfe40fffb0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.126+0000 7fcfe3e00640 1 --2- 172.21.15.12:0/1306024126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfe4102000 0x7fcfe4101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.126+0000 7fcfe3e00640 1 -- 172.21.15.12:0/1306024126 >> 172.21.15.12:0/1306024126 conn(0x7fcfe40fba10 msgr2=0x7fcfe4102940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:07.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.126+0000 7fcfe3e00640 1 -- 172.21.15.12:0/1306024126 shutdown_connections 2024-03-23T23:08:07.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:07.127+0000 7fcfe3e00640 1 -- 172.21.15.12:0/1306024126 wait complete. 2024-03-23T23:08:07.144 INFO:tasks.cephfs.filesystem:scrub status for tag:dbf24598-1d3b-4c8a-a1b0-d3f77890ec1d - {'path': '/', 'tag': 'dbf24598-1d3b-4c8a-a1b0-d3f77890ec1d', 'options': 'recursive,force'} 2024-03-23T23:08:07.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:06 smithi203 ceph-mon[36693]: pgmap v579: 97 pgs: 97 active+clean; 4.0 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:08:07.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:06 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:08:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.957+0000 7fe287200640 1 -- 172.21.15.12:0/221636034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2801016d0 msgr2=0x7fe280101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.957+0000 7fe287200640 1 --2- 172.21.15.12:0/221636034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2801016d0 0x7fe280101af0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe270009900 tx=0x7fe27002bc30 comp rx=0 tx=0).stop 2024-03-23T23:08:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.957+0000 7fe287200640 1 -- 172.21.15.12:0/221636034 shutdown_connections 2024-03-23T23:08:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.957+0000 7fe287200640 1 --2- 172.21.15.12:0/221636034 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe280103280 0x7fe280109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.957+0000 7fe287200640 1 --2- 172.21.15.12:0/221636034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2801028c0 0x7fe280102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.957+0000 7fe287200640 1 --2- 172.21.15.12:0/221636034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2801016d0 0x7fe280101af0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.957+0000 7fe287200640 1 -- 172.21.15.12:0/221636034 >> 172.21.15.12:0/221636034 conn(0x7fe2800fcc60 msgr2=0x7fe2800ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.957+0000 7fe287200640 1 -- 172.21.15.12:0/221636034 shutdown_connections 2024-03-23T23:08:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.957+0000 7fe287200640 1 -- 172.21.15.12:0/221636034 wait complete. 2024-03-23T23:08:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.958+0000 7fe287200640 1 Processor -- start 2024-03-23T23:08:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.958+0000 7fe287200640 1 -- start start 2024-03-23T23:08:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.958+0000 7fe287200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2801016d0 0x7fe280107450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.959+0000 7fe287200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2801028c0 0x7fe280107990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.959+0000 7fe287200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe280103280 0x7fe280103fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.959+0000 7fe287200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28010d3a0 con 0x7fe2801028c0 2024-03-23T23:08:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.959+0000 7fe287200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28010d220 con 0x7fe280103280 2024-03-23T23:08:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.959+0000 7fe287200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28010d520 con 0x7fe2801016d0 2024-03-23T23:08:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.959+0000 7fe27fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2801016d0 0x7fe280107450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.959+0000 7fe277e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2801028c0 0x7fe280107990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.959+0000 7fe27fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2801016d0 0x7fe280107450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44782/0 (socket says 172.21.15.12:44782) 2024-03-23T23:08:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.959+0000 7fe27fe00640 1 -- 172.21.15.12:0/3177492258 learned_addr learned my addr 172.21.15.12:0/3177492258 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.959+0000 7fe285200640 1 --2- 172.21.15.12:0/3177492258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe280103280 0x7fe280103fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.959+0000 7fe277e00640 1 -- 172.21.15.12:0/3177492258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2801016d0 msgr2=0x7fe280107450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.959+0000 7fe277e00640 1 --2- 172.21.15.12:0/3177492258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2801016d0 0x7fe280107450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.959+0000 7fe277e00640 1 -- 172.21.15.12:0/3177492258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe280103280 msgr2=0x7fe280103fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.959+0000 7fe277e00640 1 --2- 172.21.15.12:0/3177492258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe280103280 0x7fe280103fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.959+0000 7fe277e00640 1 -- 172.21.15.12:0/3177492258 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe280104880 con 0x7fe2801028c0 2024-03-23T23:08:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.960+0000 7fe277e00640 1 --2- 172.21.15.12:0/3177492258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2801028c0 0x7fe280107990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe26800e9f0 tx=0x7fe26800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.960+0000 7fe27d600640 1 -- 172.21.15.12:0/3177492258 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe26800ccc0 con 0x7fe2801028c0 2024-03-23T23:08:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.960+0000 7fe27d600640 1 -- 172.21.15.12:0/3177492258 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe268010910 con 0x7fe2801028c0 2024-03-23T23:08:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.960+0000 7fe27d600640 1 -- 172.21.15.12:0/3177492258 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe268018960 con 0x7fe2801028c0 2024-03-23T23:08:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.960+0000 7fe287200640 1 -- 172.21.15.12:0/3177492258 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2801a5f90 con 0x7fe2801028c0 2024-03-23T23:08:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.960+0000 7fe287200640 1 -- 172.21.15.12:0/3177492258 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2801a6340 con 0x7fe2801028c0 2024-03-23T23:08:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.963+0000 7fe289e3e740 1 Processor -- start 2024-03-23T23:08:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.963+0000 7fe289e3e740 1 -- start start 2024-03-23T23:08:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.964+0000 7fe27d600640 1 -- 172.21.15.12:0/3177492258 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe268010ac0 con 0x7fe2801028c0 2024-03-23T23:08:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.964+0000 7fe289e3e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3829ab1e0 0x55c3829ab600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.964+0000 7fe289e3e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3829abbd0 0x55c3829b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.964+0000 7fe289e3e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3829b0d40 0x55c3829b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.964+0000 7fe289e3e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c38296e3b0 con 0x55c3829b0d40 2024-03-23T23:08:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.964+0000 7fe289e3e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c382910830 con 0x55c3829ab1e0 2024-03-23T23:08:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.964+0000 7fe289e3e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3828b95d0 con 0x55c3829abbd0 2024-03-23T23:08:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.964+0000 7fe285200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3829ab1e0 0x55c3829ab600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.964+0000 7fe27fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3829abbd0 0x55c3829b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.964+0000 7fe285200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3829ab1e0 0x55c3829ab600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:45132/0 (socket says 172.21.15.12:45132) 2024-03-23T23:08:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.964+0000 7fe285200640 1 -- 172.21.15.12:0/490150036 learned_addr learned my addr 172.21.15.12:0/490150036 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.964+0000 7fe277e00640 1 --2- 172.21.15.12:0/490150036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3829b0d40 0x55c3829b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.964+0000 7fe285200640 1 -- 172.21.15.12:0/490150036 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3829abbd0 msgr2=0x55c3829b06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.965+0000 7fe285200640 1 --2- 172.21.15.12:0/490150036 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3829abbd0 0x55c3829b06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.965+0000 7fe285200640 1 -- 172.21.15.12:0/490150036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3829b0d40 msgr2=0x55c3829b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.965+0000 7fe285200640 1 --2- 172.21.15.12:0/490150036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3829b0d40 0x55c3829b31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.965+0000 7fe285200640 1 -- 172.21.15.12:0/490150036 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3828d12c0 con 0x55c3829ab1e0 2024-03-23T23:08:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.965+0000 7fe285200640 1 --2- 172.21.15.12:0/490150036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3829ab1e0 0x55c3829ab600 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe278009fc0 tx=0x7fe278010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e462f49827c83857 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.965+0000 7fe275600640 1 -- 172.21.15.12:0/490150036 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe278003b70 con 0x55c3829ab1e0 2024-03-23T23:08:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.965+0000 7fe275600640 1 -- 172.21.15.12:0/490150036 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe278003d10 con 0x55c3829ab1e0 2024-03-23T23:08:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.965+0000 7fe275600640 1 -- 172.21.15.12:0/490150036 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe278011aa0 con 0x55c3829ab1e0 2024-03-23T23:08:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.965+0000 7fe27d600640 1 --2- 172.21.15.12:0/3177492258 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe2640a28a0 0x7fe2640a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.966+0000 7fe27d600640 1 -- 172.21.15.12:0/3177492258 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe2680cef10 con 0x7fe2801028c0 2024-03-23T23:08:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.966+0000 7fe27fe00640 1 --2- 172.21.15.12:0/3177492258 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe2640a28a0 0x7fe2640a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.966+0000 7fe289e3e740 1 -- 172.21.15.12:0/490150036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3829ab1e0 msgr2=0x55c3829ab600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.966+0000 7fe289e3e740 1 --2- 172.21.15.12:0/490150036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3829ab1e0 0x55c3829ab600 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe278009fc0 tx=0x7fe278010860 comp rx=0 tx=0).stop 2024-03-23T23:08:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.966+0000 7fe289e3e740 1 -- 172.21.15.12:0/490150036 shutdown_connections 2024-03-23T23:08:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.966+0000 7fe289e3e740 1 --2- 172.21.15.12:0/490150036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3829b0d40 0x55c3829b31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.966+0000 7fe289e3e740 1 --2- 172.21.15.12:0/490150036 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3829abbd0 0x55c3829b06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.966+0000 7fe289e3e740 1 --2- 172.21.15.12:0/490150036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3829ab1e0 0x55c3829ab600 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.966+0000 7fe289e3e740 1 -- 172.21.15.12:0/490150036 >> 172.21.15.12:0/490150036 conn(0x55c3829a6750 msgr2=0x55c3829a6b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:07.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.966+0000 7fe289e3e740 1 -- 172.21.15.12:0/490150036 shutdown_connections 2024-03-23T23:08:07.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.967+0000 7fe289e3e740 1 -- 172.21.15.12:0/490150036 wait complete. 2024-03-23T23:08:07.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.967+0000 7fe27fe00640 1 --2- 172.21.15.12:0/3177492258 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe2640a28a0 0x7fe2640a4d60 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe270006010 tx=0x7fe27002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:07.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.967+0000 7fe289e3e740 1 -- start start 2024-03-23T23:08:07.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.967+0000 7fe289e3e740 1 Processor -- start 2024-03-23T23:08:07.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.968+0000 7fe289e3e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3829ab1e0 0x55c3829b53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.968+0000 7fe289e3e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3829abbd0 0x55c3829b5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.968+0000 7fe289e3e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3829b0d40 0x55c3829bc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.968+0000 7fe289e3e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3828cca20 con 0x55c3829b0d40 2024-03-23T23:08:07.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.968+0000 7fe289e3e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c38290c8d0 con 0x55c3829ab1e0 2024-03-23T23:08:07.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.968+0000 7fe289e3e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3829b3cf0 con 0x55c3829abbd0 2024-03-23T23:08:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.968+0000 7fe289e3e740 10 client.0 _ll_register_callbacks cb 0x55c3829091c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.968+0000 7fe277e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3829abbd0 0x55c3829b5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.968+0000 7fe277e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3829abbd0 0x55c3829b5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44800/0 (socket says 172.21.15.12:44800) 2024-03-23T23:08:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.968+0000 7fe277e00640 1 -- 172.21.15.12:0/2309058682 learned_addr learned my addr 172.21.15.12:0/2309058682 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.968+0000 7fe285200640 1 --2- 172.21.15.12:0/2309058682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3829b0d40 0x55c3829bc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.968+0000 7fe277e00640 1 -- 172.21.15.12:0/2309058682 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3829ab1e0 msgr2=0x55c3829b53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.968+0000 7fe277e00640 1 --2- 172.21.15.12:0/2309058682 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3829ab1e0 0x55c3829b53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.968+0000 7fe277e00640 1 -- 172.21.15.12:0/2309058682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3829b0d40 msgr2=0x55c3829bc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.968+0000 7fe277e00640 1 --2- 172.21.15.12:0/2309058682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3829b0d40 0x55c3829bc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.968+0000 7fe277e00640 1 -- 172.21.15.12:0/2309058682 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c382625210 con 0x55c3829abbd0 2024-03-23T23:08:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.969+0000 7fe277e00640 1 --2- 172.21.15.12:0/2309058682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3829abbd0 0x55c3829b5930 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe2680a3ad0 tx=0x7fe268010ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87d0706debcb20a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.969+0000 7fe26e000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:08:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.969+0000 7fe26e000640 1 -- 172.21.15.12:0/2309058682 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe26809b1c0 con 0x55c3829abbd0 2024-03-23T23:08:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.969+0000 7fe26e000640 1 -- 172.21.15.12:0/2309058682 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe2680860b0 con 0x55c3829abbd0 2024-03-23T23:08:07.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.969+0000 7fe289e3e740 1 -- 172.21.15.12:0/2309058682 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe278003d10 con 0x55c3829abbd0 2024-03-23T23:08:07.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.969+0000 7fe26e000640 1 -- 172.21.15.12:0/2309058682 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2680862b0 con 0x55c3829abbd0 2024-03-23T23:08:07.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.969+0000 7fe289e3e740 10 client.27106 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:07.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.969+0000 7fe289e3e740 1 -- 172.21.15.12:0/2309058682 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c3826f1620 con 0x55c3829abbd0 2024-03-23T23:08:07.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.969+0000 7fe26e000640 1 -- 172.21.15.12:0/2309058682 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe278003d10 con 0x55c3829abbd0 2024-03-23T23:08:07.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.970+0000 7fe26e000640 1 -- 172.21.15.12:0/2309058682 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe26809cb90 con 0x55c3829abbd0 2024-03-23T23:08:07.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.970+0000 7fe289e3e740 10 client.27106 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:07.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.970+0000 7fe289e3e740 10 client.27106 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:08:07.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.970+0000 7fe289e3e740 20 client.27106 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:07.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.970+0000 7fe289e3e740 1 --2- 172.21.15.12:0/2309058682 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c3829c11e0 0x55c3829c35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.970+0000 7fe289e3e740 4 client.27106 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:08:07.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.970+0000 7fe289e3e740 1 -- 172.21.15.12:0/2309058682 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c38291dd70 con 0x55c3829c11e0 2024-03-23T23:08:07.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.970+0000 7fe27fe00640 1 --2- 172.21.15.12:0/2309058682 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c3829c11e0 0x55c3829c35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.970+0000 7fe27fe00640 1 --2- 172.21.15.12:0/2309058682 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c3829c11e0 0x55c3829c35d0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c43ac18aca096d0 server_cookie=606dfa2e26d18674 in_seq=0 out_seq=0 2024-03-23T23:08:07.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.970+0000 7fe26e000640 10 client.27106 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:07.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.971+0000 7fe26e000640 1 -- 172.21.15.12:0/2309058682 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c38291dd70 con 0x55c3829c11e0 2024-03-23T23:08:07.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.971+0000 7fe26e000640 10 client.27106 handle_command_reply: tid=0 2024-03-23T23:08:07.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.971+0000 7fe289e3e740 1 client.27106 shutdown 2024-03-23T23:08:07.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.971+0000 7fe289e3e740 1 -- 172.21.15.12:0/2309058682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3829abbd0 msgr2=0x55c3829b5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:07.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.971+0000 7fe289e3e740 1 --2- 172.21.15.12:0/2309058682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3829abbd0 0x55c3829b5930 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe2680a3ad0 tx=0x7fe268010ee0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.972+0000 7fe289e3e740 1 -- 172.21.15.12:0/2309058682 shutdown_connections 2024-03-23T23:08:07.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.972+0000 7fe289e3e740 1 --2- 172.21.15.12:0/2309058682 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c3829c11e0 0x55c3829c35d0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.972+0000 7fe289e3e740 1 --2- 172.21.15.12:0/2309058682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3829b0d40 0x55c3829bc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.972+0000 7fe289e3e740 1 --2- 172.21.15.12:0/2309058682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3829abbd0 0x55c3829b5930 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.972+0000 7fe289e3e740 1 --2- 172.21.15.12:0/2309058682 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3829ab1e0 0x55c3829b53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.972+0000 7fe26e000640 0 client.27106 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:07.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.972+0000 7fe289e3e740 1 -- 172.21.15.12:0/2309058682 >> 172.21.15.12:0/2309058682 conn(0x55c3829a6750 msgr2=0x55c3829a5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:07.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.972+0000 7fe289e3e740 1 -- 172.21.15.12:0/2309058682 shutdown_connections 2024-03-23T23:08:07.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.972+0000 7fe289e3e740 1 -- 172.21.15.12:0/2309058682 wait complete. 2024-03-23T23:08:07.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.972+0000 7fe289e3e740 20 client.27106 trim_cache size 0 max 16384 2024-03-23T23:08:07.989 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.991+0000 7fe289e3e740 1 Processor -- start 2024-03-23T23:08:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.991+0000 7fe289e3e740 1 -- start start 2024-03-23T23:08:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.991+0000 7fe289e3e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3829ab1e0 0x55c3829aa0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.991+0000 7fe289e3e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3829abbd0 0x55c3829bc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.991+0000 7fe289e3e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3829b0d40 0x55c3829c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.991+0000 7fe289e3e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2801b14f0 con 0x55c3829ab1e0 2024-03-23T23:08:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.991+0000 7fe289e3e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2801aeff0 con 0x55c3829b0d40 2024-03-23T23:08:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.991+0000 7fe289e3e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2801ad090 con 0x55c3829abbd0 2024-03-23T23:08:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.991+0000 7fe277e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3829abbd0 0x55c3829bc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.991+0000 7fe285200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3829b0d40 0x55c3829c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.991+0000 7fe277e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3829abbd0 0x55c3829bc8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44812/0 (socket says 172.21.15.12:44812) 2024-03-23T23:08:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.991+0000 7fe277e00640 1 -- 172.21.15.12:0/2796783692 learned_addr learned my addr 172.21.15.12:0/2796783692 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.992+0000 7fe27fe00640 1 --2- 172.21.15.12:0/2796783692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3829ab1e0 0x55c3829aa0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.992+0000 7fe277e00640 1 -- 172.21.15.12:0/2796783692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3829b0d40 msgr2=0x55c3829c7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.992+0000 7fe277e00640 1 --2- 172.21.15.12:0/2796783692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3829b0d40 0x55c3829c7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.992+0000 7fe277e00640 1 -- 172.21.15.12:0/2796783692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3829ab1e0 msgr2=0x55c3829aa0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.992+0000 7fe277e00640 1 --2- 172.21.15.12:0/2796783692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3829ab1e0 0x55c3829aa0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.992+0000 7fe277e00640 1 -- 172.21.15.12:0/2796783692 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3829b3bd0 con 0x55c3829abbd0 2024-03-23T23:08:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.992+0000 7fe27fe00640 1 --2- 172.21.15.12:0/2796783692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3829ab1e0 0x55c3829aa0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:08:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.992+0000 7fe277e00640 1 --2- 172.21.15.12:0/2796783692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3829abbd0 0x55c3829bc8c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe268097cc0 tx=0x7fe268005b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=677e1f29f469328b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.992+0000 7fe274c00640 1 -- 172.21.15.12:0/2796783692 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe26809b630 con 0x55c3829abbd0 2024-03-23T23:08:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.992+0000 7fe274c00640 1 -- 172.21.15.12:0/2796783692 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe2680860e0 con 0x55c3829abbd0 2024-03-23T23:08:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.992+0000 7fe274c00640 1 -- 172.21.15.12:0/2796783692 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe268018b00 con 0x55c3829abbd0 2024-03-23T23:08:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.993+0000 7fe289e3e740 1 -- 172.21.15.12:0/2796783692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3829abbd0 msgr2=0x55c3829bc8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.993+0000 7fe289e3e740 1 --2- 172.21.15.12:0/2796783692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3829abbd0 0x55c3829bc8c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe268097cc0 tx=0x7fe268005b30 comp rx=0 tx=0).stop 2024-03-23T23:08:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.993+0000 7fe289e3e740 1 -- 172.21.15.12:0/2796783692 shutdown_connections 2024-03-23T23:08:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.993+0000 7fe289e3e740 1 --2- 172.21.15.12:0/2796783692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3829b0d40 0x55c3829c7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.993+0000 7fe289e3e740 1 --2- 172.21.15.12:0/2796783692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3829abbd0 0x55c3829bc8c0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.993+0000 7fe289e3e740 1 --2- 172.21.15.12:0/2796783692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3829ab1e0 0x55c3829aa0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.993+0000 7fe289e3e740 1 -- 172.21.15.12:0/2796783692 >> 172.21.15.12:0/2796783692 conn(0x55c3829a6750 msgr2=0x55c3829a99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.993+0000 7fe289e3e740 1 -- 172.21.15.12:0/2796783692 shutdown_connections 2024-03-23T23:08:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.993+0000 7fe289e3e740 1 -- 172.21.15.12:0/2796783692 wait complete. 2024-03-23T23:08:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.994+0000 7fe289e3e740 1 -- start start 2024-03-23T23:08:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.994+0000 7fe289e3e740 1 Processor -- start 2024-03-23T23:08:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.994+0000 7fe289e3e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3829ab1e0 0x55c3829bc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.994+0000 7fe289e3e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3829abbd0 0x55c3829bc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.994+0000 7fe289e3e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3829b0d40 0x55c3829ba900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.994+0000 7fe289e3e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2801ad690 con 0x55c3829ab1e0 2024-03-23T23:08:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.994+0000 7fe289e3e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2801ae990 con 0x55c3829b0d40 2024-03-23T23:08:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.994+0000 7fe289e3e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2801adcd0 con 0x55c3829abbd0 2024-03-23T23:08:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.994+0000 7fe289e3e740 10 client.0 _ll_register_callbacks cb 0x7fe2680d2640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.995+0000 7fe27fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3829ab1e0 0x55c3829bc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.995+0000 7fe27fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3829ab1e0 0x55c3829bc2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59728/0 (socket says 172.21.15.12:59728) 2024-03-23T23:08:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.995+0000 7fe27fe00640 1 -- 172.21.15.12:0/3513735739 learned_addr learned my addr 172.21.15.12:0/3513735739 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.995+0000 7fe27fe00640 1 -- 172.21.15.12:0/3513735739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3829abbd0 msgr2=0x55c3829bc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.995+0000 7fe27fe00640 1 --2- 172.21.15.12:0/3513735739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3829abbd0 0x55c3829bc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.995+0000 7fe27fe00640 1 -- 172.21.15.12:0/3513735739 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3829b0d40 msgr2=0x55c3829ba900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.995+0000 7fe27fe00640 1 --2- 172.21.15.12:0/3513735739 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3829b0d40 0x55c3829ba900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.995+0000 7fe27fe00640 1 -- 172.21.15.12:0/3513735739 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c3829bb130 con 0x55c3829ab1e0 2024-03-23T23:08:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.996+0000 7fe27fe00640 1 --2- 172.21.15.12:0/3513735739 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3829ab1e0 0x55c3829bc2a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe27002c170 tx=0x7fe270042070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ce20b72a5349e87 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.996+0000 7fe26e000640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:08:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.996+0000 7fe26e000640 1 -- 172.21.15.12:0/3513735739 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe270046070 con 0x55c3829ab1e0 2024-03-23T23:08:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.996+0000 7fe289e3e740 1 -- 172.21.15.12:0/3513735739 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe2680860e0 con 0x55c3829ab1e0 2024-03-23T23:08:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.996+0000 7fe26e000640 1 -- 172.21.15.12:0/3513735739 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe2700046c0 con 0x55c3829ab1e0 2024-03-23T23:08:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.997+0000 7fe26e000640 1 -- 172.21.15.12:0/3513735739 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2700048c0 con 0x55c3829ab1e0 2024-03-23T23:08:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.997+0000 7fe289e3e740 10 client.17010 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.997+0000 7fe289e3e740 1 -- 172.21.15.12:0/3513735739 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c382a1d5e0 con 0x55c3829ab1e0 2024-03-23T23:08:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.997+0000 7fe26e000640 1 -- 172.21.15.12:0/3513735739 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe2680860e0 con 0x55c3829ab1e0 2024-03-23T23:08:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.997+0000 7fe26e000640 1 -- 172.21.15.12:0/3513735739 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe270055600 con 0x55c3829ab1e0 2024-03-23T23:08:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.997+0000 7fe289e3e740 10 client.17010 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.997+0000 7fe289e3e740 10 client.17010 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:08:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.997+0000 7fe289e3e740 20 client.17010 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.997+0000 7fe289e3e740 1 --2- 172.21.15.12:0/3513735739 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c382a1df20 0x55c382a20360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.997+0000 7fe289e3e740 4 client.17010 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:08:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.997+0000 7fe289e3e740 1 -- 172.21.15.12:0/3513735739 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c3829180c0 con 0x55c382a1df20 2024-03-23T23:08:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.998+0000 7fe277e00640 1 --2- 172.21.15.12:0/3513735739 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c382a1df20 0x55c382a20360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.999+0000 7fe277e00640 1 --2- 172.21.15.12:0/3513735739 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c382a1df20 0x55c382a20360 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bddcec266eb6c692 server_cookie=f2da042cc32c0740 in_seq=0 out_seq=0 2024-03-23T23:08:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:07.999+0000 7fe26e000640 10 client.17010 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.011+0000 7fe26e000640 1 -- 172.21.15.12:0/3513735739 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c3829180c0 con 0x55c382a1df20 2024-03-23T23:08:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.011+0000 7fe26e000640 10 client.17010 handle_command_reply: tid=0 2024-03-23T23:08:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.011+0000 7fe289e3e740 1 client.17010 shutdown 2024-03-23T23:08:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.011+0000 7fe289e3e740 1 -- 172.21.15.12:0/3513735739 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3829ab1e0 msgr2=0x55c3829bc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.011+0000 7fe289e3e740 1 --2- 172.21.15.12:0/3513735739 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3829ab1e0 0x55c3829bc2a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe27002c170 tx=0x7fe270042070 comp rx=0 tx=0).stop 2024-03-23T23:08:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.011+0000 7fe289e3e740 1 -- 172.21.15.12:0/3513735739 shutdown_connections 2024-03-23T23:08:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.011+0000 7fe289e3e740 1 --2- 172.21.15.12:0/3513735739 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c382a1df20 0x55c382a20360 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.011+0000 7fe289e3e740 1 --2- 172.21.15.12:0/3513735739 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3829b0d40 0x55c3829ba900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.011+0000 7fe289e3e740 1 --2- 172.21.15.12:0/3513735739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c3829abbd0 0x55c3829bc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.011+0000 7fe26e000640 0 client.17010 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.011+0000 7fe289e3e740 1 --2- 172.21.15.12:0/3513735739 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c3829ab1e0 0x55c3829bc2a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.011+0000 7fe289e3e740 1 -- 172.21.15.12:0/3513735739 >> 172.21.15.12:0/3513735739 conn(0x55c3829a6750 msgr2=0x55c3829aa240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.012+0000 7fe289e3e740 1 -- 172.21.15.12:0/3513735739 shutdown_connections 2024-03-23T23:08:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.012+0000 7fe289e3e740 1 -- 172.21.15.12:0/3513735739 wait complete. 2024-03-23T23:08:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.012+0000 7fe289e3e740 20 client.17010 trim_cache size 0 max 16384 2024-03-23T23:08:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:08:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:08:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:08:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:08:08.013 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.013+0000 7fe287200640 1 -- 172.21.15.12:0/3177492258 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe2640a28a0 msgr2=0x7fe2640a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:08.013 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.013+0000 7fe287200640 1 --2- 172.21.15.12:0/3177492258 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe2640a28a0 0x7fe2640a4d60 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe270006010 tx=0x7fe27002df90 comp rx=0 tx=0).stop 2024-03-23T23:08:08.013 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.013+0000 7fe287200640 1 -- 172.21.15.12:0/3177492258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2801028c0 msgr2=0x7fe280107990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:08.013 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.013+0000 7fe287200640 1 --2- 172.21.15.12:0/3177492258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2801028c0 0x7fe280107990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe26800e9f0 tx=0x7fe26800eeb0 comp rx=0 tx=0).stop 2024-03-23T23:08:08.013 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.013+0000 7fe287200640 1 -- 172.21.15.12:0/3177492258 shutdown_connections 2024-03-23T23:08:08.013 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.013+0000 7fe287200640 1 --2- 172.21.15.12:0/3177492258 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe2640a28a0 0x7fe2640a4d60 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:08.013 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.013+0000 7fe287200640 1 --2- 172.21.15.12:0/3177492258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe280103280 0x7fe280103fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:08.014 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.013+0000 7fe287200640 1 --2- 172.21.15.12:0/3177492258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2801028c0 0x7fe280107990 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:08.014 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.013+0000 7fe287200640 1 --2- 172.21.15.12:0/3177492258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2801016d0 0x7fe280107450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:08.014 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.013+0000 7fe287200640 1 -- 172.21.15.12:0/3177492258 >> 172.21.15.12:0/3177492258 conn(0x7fe2800fcc60 msgr2=0x7fe2800738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:08.014 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.013+0000 7fe287200640 1 -- 172.21.15.12:0/3177492258 shutdown_connections 2024-03-23T23:08:08.014 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:08.013+0000 7fe287200640 1 -- 172.21.15.12:0/3177492258 wait complete. 2024-03-23T23:08:08.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:07 smithi012 ceph-mon[31695]: scrub queued for path: / 2024-03-23T23:08:08.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:07 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:08:08.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:07 smithi012 ceph-mon[31695]: scrub summary: active paths [/] 2024-03-23T23:08:08.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:07 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:08:08.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:07 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:08:08.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:07 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:08:08.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:07 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:08:08.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:07 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:08:08.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:07 smithi173 ceph-mon[36116]: scrub queued for path: / 2024-03-23T23:08:08.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:07 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:08:08.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:07 smithi173 ceph-mon[36116]: scrub summary: active paths [/] 2024-03-23T23:08:08.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:07 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:08:08.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:07 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:08:08.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:07 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:08:08.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:07 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:08:08.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:07 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:08:08.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:07 smithi203 ceph-mon[36693]: scrub queued for path: / 2024-03-23T23:08:08.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:07 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:08:08.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:07 smithi203 ceph-mon[36693]: scrub summary: active paths [/] 2024-03-23T23:08:08.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:07 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:08:08.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:07 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:08:08.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:07 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:08:08.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:07 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:08:08.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:07 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:08:09.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:08 smithi012 ceph-mon[31695]: Adjusting osd_memory_target on smithi173 to 174.1M 2024-03-23T23:08:09.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:08 smithi012 ceph-mon[31695]: Unable to set osd_memory_target on smithi173 to 182637772: error parsing value: Value '182637772' is below minimum 939524096 2024-03-23T23:08:09.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:08 smithi012 ceph-mon[31695]: pgmap v580: 97 pgs: 97 active+clean; 4.0 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:08:09.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:08 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:08:09.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:08 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:08:09.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:08 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:08:09.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:08 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:08:09.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:08 smithi012 ceph-mon[31695]: Adjusting osd_memory_target on smithi012 to 177.8M 2024-03-23T23:08:09.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:08 smithi012 ceph-mon[31695]: Unable to set osd_memory_target on smithi012 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-23T23:08:09.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:08 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:08:09.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:08 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:08:09.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:08 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:08:09.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:08 smithi173 ceph-mon[36116]: Adjusting osd_memory_target on smithi173 to 174.1M 2024-03-23T23:08:09.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:08 smithi173 ceph-mon[36116]: Unable to set osd_memory_target on smithi173 to 182637772: error parsing value: Value '182637772' is below minimum 939524096 2024-03-23T23:08:09.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:08 smithi173 ceph-mon[36116]: pgmap v580: 97 pgs: 97 active+clean; 4.0 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:08:09.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:08 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:08:09.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:08 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:08:09.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:08 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:08:09.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:08 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:08:09.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:08 smithi173 ceph-mon[36116]: Adjusting osd_memory_target on smithi012 to 177.8M 2024-03-23T23:08:09.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:08 smithi173 ceph-mon[36116]: Unable to set osd_memory_target on smithi012 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-23T23:08:09.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:08 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:08:09.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:08 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:08:09.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:08 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:08:09.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:08 smithi203 ceph-mon[36693]: Adjusting osd_memory_target on smithi173 to 174.1M 2024-03-23T23:08:09.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:08 smithi203 ceph-mon[36693]: Unable to set osd_memory_target on smithi173 to 182637772: error parsing value: Value '182637772' is below minimum 939524096 2024-03-23T23:08:09.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:08 smithi203 ceph-mon[36693]: pgmap v580: 97 pgs: 97 active+clean; 4.0 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:08:09.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:08 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:08:09.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:08 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:08:09.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:08 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:08:09.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:08 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:08:09.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:08 smithi203 ceph-mon[36693]: Adjusting osd_memory_target on smithi012 to 177.8M 2024-03-23T23:08:09.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:08 smithi203 ceph-mon[36693]: Unable to set osd_memory_target on smithi012 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-23T23:08:09.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:08 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:08:09.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:08 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:08:09.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:08 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:08:11.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:10 smithi012 ceph-mon[31695]: pgmap v581: 97 pgs: 97 active+clean; 4.0 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-03-23T23:08:11.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:10 smithi173 ceph-mon[36116]: pgmap v581: 97 pgs: 97 active+clean; 4.0 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-03-23T23:08:11.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:10 smithi203 ceph-mon[36693]: pgmap v581: 97 pgs: 97 active+clean; 4.0 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-03-23T23:08:12.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:11 smithi012 ceph-mon[31695]: scrub summary: idle 2024-03-23T23:08:12.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:11 smithi173 ceph-mon[36116]: scrub summary: idle 2024-03-23T23:08:12.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:11 smithi203 ceph-mon[36693]: scrub summary: idle 2024-03-23T23:08:13.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:12 smithi012 ceph-mon[31695]: pgmap v582: 97 pgs: 97 active+clean; 4.0 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-23T23:08:13.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:12 smithi173 ceph-mon[36116]: pgmap v582: 97 pgs: 97 active+clean; 4.0 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-23T23:08:13.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:12 smithi203 ceph-mon[36693]: pgmap v582: 97 pgs: 97 active+clean; 4.0 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-23T23:08:15.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:14 smithi173 ceph-mon[36116]: pgmap v583: 97 pgs: 97 active+clean; 4.1 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:08:15.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:14 smithi203 ceph-mon[36693]: pgmap v583: 97 pgs: 97 active+clean; 4.1 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:08:15.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:14 smithi012 ceph-mon[31695]: pgmap v583: 97 pgs: 97 active+clean; 4.1 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:08:17.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:16 smithi173 ceph-mon[36116]: pgmap v584: 97 pgs: 97 active+clean; 4.1 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:08:17.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:16 smithi203 ceph-mon[36693]: pgmap v584: 97 pgs: 97 active+clean; 4.1 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:08:17.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:16 smithi012 ceph-mon[31695]: pgmap v584: 97 pgs: 97 active+clean; 4.1 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:08:19.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:18 smithi173 ceph-mon[36116]: pgmap v585: 97 pgs: 97 active+clean; 4.1 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:08:19.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:18 smithi203 ceph-mon[36693]: pgmap v585: 97 pgs: 97 active+clean; 4.1 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:08:19.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:18 smithi012 ceph-mon[31695]: pgmap v585: 97 pgs: 97 active+clean; 4.1 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:08:21.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:20 smithi173 ceph-mon[36116]: pgmap v586: 97 pgs: 97 active+clean; 4.2 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:08:21.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:20 smithi203 ceph-mon[36693]: pgmap v586: 97 pgs: 97 active+clean; 4.2 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:08:21.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.158+0000 7f8ea5a00640 1 -- 172.21.15.12:0/1111635083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ea0104af0 msgr2=0x7f8ea0104f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:21.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.158+0000 7f8ea5a00640 1 --2- 172.21.15.12:0/1111635083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ea0104af0 0x7f8ea0104f70 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f8e88009900 tx=0x7f8e8802bc30 comp rx=0 tx=0).stop 2024-03-23T23:08:21.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.158+0000 7f8ea5a00640 1 -- 172.21.15.12:0/1111635083 shutdown_connections 2024-03-23T23:08:21.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.158+0000 7f8ea5a00640 1 --2- 172.21.15.12:0/1111635083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ea01054b0 0x7f8ea010bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.158+0000 7f8ea5a00640 1 --2- 172.21.15.12:0/1111635083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ea0104af0 0x7f8ea0104f70 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.158+0000 7f8ea5a00640 1 --2- 172.21.15.12:0/1111635083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ea0103900 0x7f8ea0103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.158+0000 7f8ea5a00640 1 -- 172.21.15.12:0/1111635083 >> 172.21.15.12:0/1111635083 conn(0x7f8ea00fee90 msgr2=0x7f8ea01012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:21.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.159+0000 7f8ea5a00640 1 -- 172.21.15.12:0/1111635083 shutdown_connections 2024-03-23T23:08:21.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.159+0000 7f8ea5a00640 1 -- 172.21.15.12:0/1111635083 wait complete. 2024-03-23T23:08:21.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.160+0000 7f8ea5a00640 1 Processor -- start 2024-03-23T23:08:21.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.160+0000 7f8ea5a00640 1 -- start start 2024-03-23T23:08:21.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.161+0000 7f8ea5a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ea0103900 0x7f8ea0199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:21.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.161+0000 7f8ea5a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ea0104af0 0x7f8ea019a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:21.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.161+0000 7f8ea5a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ea01054b0 0x7f8ea01a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:21.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.161+0000 7f8ea5a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea010f560 con 0x7f8ea0104af0 2024-03-23T23:08:21.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.161+0000 7f8ea5a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea010f3e0 con 0x7f8ea01054b0 2024-03-23T23:08:21.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.161+0000 7f8ea5a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea010f6e0 con 0x7f8ea0103900 2024-03-23T23:08:21.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.161+0000 7f8e9ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ea0103900 0x7f8ea0199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:21.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.161+0000 7f8e9ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ea0103900 0x7f8ea0199b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:46088/0 (socket says 172.21.15.12:46088) 2024-03-23T23:08:21.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.161+0000 7f8e9ea00640 1 -- 172.21.15.12:0/1164595113 learned_addr learned my addr 172.21.15.12:0/1164595113 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:21.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.161+0000 7f8e9f400640 1 --2- 172.21.15.12:0/1164595113 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ea01054b0 0x7f8ea01a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:21.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.161+0000 7f8e97e00640 1 --2- 172.21.15.12:0/1164595113 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ea0104af0 0x7f8ea019a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:21.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.161+0000 7f8e9ea00640 1 -- 172.21.15.12:0/1164595113 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ea01054b0 msgr2=0x7f8ea01a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:21.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.161+0000 7f8e9ea00640 1 --2- 172.21.15.12:0/1164595113 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ea01054b0 0x7f8ea01a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.161+0000 7f8e9ea00640 1 -- 172.21.15.12:0/1164595113 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ea0104af0 msgr2=0x7f8ea019a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:21.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.162+0000 7f8e9ea00640 1 --2- 172.21.15.12:0/1164595113 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ea0104af0 0x7f8ea019a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.162+0000 7f8e9ea00640 1 -- 172.21.15.12:0/1164595113 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea01a1790 con 0x7f8ea0103900 2024-03-23T23:08:21.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.162+0000 7f8e9ea00640 1 --2- 172.21.15.12:0/1164595113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ea0103900 0x7f8ea0199b70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8e9000e9c0 tx=0x7f8e9000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:21.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.162+0000 7f8e97400640 1 -- 172.21.15.12:0/1164595113 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9000cd80 con 0x7f8ea0103900 2024-03-23T23:08:21.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.162+0000 7f8e97400640 1 -- 172.21.15.12:0/1164595113 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e900035f0 con 0x7f8ea0103900 2024-03-23T23:08:21.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.162+0000 7f8e97400640 1 -- 172.21.15.12:0/1164595113 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e90010690 con 0x7f8ea0103900 2024-03-23T23:08:21.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.162+0000 7f8ea5a00640 1 -- 172.21.15.12:0/1164595113 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea01a1a80 con 0x7f8ea0103900 2024-03-23T23:08:21.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.162+0000 7f8ea5a00640 1 -- 172.21.15.12:0/1164595113 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea01a1fa0 con 0x7f8ea0103900 2024-03-23T23:08:21.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.165+0000 7f8ea87dc740 1 Processor -- start 2024-03-23T23:08:21.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.166+0000 7f8e97400640 1 -- 172.21.15.12:0/1164595113 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8e900037a0 con 0x7f8ea0103900 2024-03-23T23:08:21.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.166+0000 7f8ea87dc740 1 -- start start 2024-03-23T23:08:21.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.166+0000 7f8ea87dc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a7fe1e0 0x56444a7fe600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:21.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.166+0000 7f8ea87dc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56444a7febd0 0x56444a8036d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:21.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.166+0000 7f8ea87dc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56444a803d40 0x56444a8061d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:21.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.166+0000 7f8ea87dc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56444a7c13b0 con 0x56444a803d40 2024-03-23T23:08:21.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.166+0000 7f8ea87dc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56444a763830 con 0x56444a7febd0 2024-03-23T23:08:21.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.166+0000 7f8ea87dc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x56444a70c5d0 con 0x56444a7fe1e0 2024-03-23T23:08:21.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.166+0000 7f8e9f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a7fe1e0 0x56444a7fe600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:21.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.166+0000 7f8e9ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56444a7febd0 0x56444a8036d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:21.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.166+0000 7f8e9f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a7fe1e0 0x56444a7fe600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:46096/0 (socket says 172.21.15.12:46096) 2024-03-23T23:08:21.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.166+0000 7f8e9f400640 1 -- 172.21.15.12:0/4045540185 learned_addr learned my addr 172.21.15.12:0/4045540185 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:21.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.167+0000 7f8e97e00640 1 --2- 172.21.15.12:0/4045540185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56444a803d40 0x56444a8061d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:21.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.167+0000 7f8e9f400640 1 -- 172.21.15.12:0/4045540185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56444a7febd0 msgr2=0x56444a8036d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:21.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.167+0000 7f8e9f400640 1 --2- 172.21.15.12:0/4045540185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56444a7febd0 0x56444a8036d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.167+0000 7f8e9f400640 1 -- 172.21.15.12:0/4045540185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56444a803d40 msgr2=0x56444a8061d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:21.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.167+0000 7f8e9f400640 1 --2- 172.21.15.12:0/4045540185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56444a803d40 0x56444a8061d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.167+0000 7f8e9f400640 1 -- 172.21.15.12:0/4045540185 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56444a7242c0 con 0x56444a7fe1e0 2024-03-23T23:08:21.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.167+0000 7f8e9f400640 1 --2- 172.21.15.12:0/4045540185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a7fe1e0 0x56444a7fe600 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8e980039f0 tx=0x7f8e98003eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca96eb3449fef953 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:21.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.167+0000 7f8e97400640 1 --2- 172.21.15.12:0/1164595113 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8e840a28a0 0x7f8e840a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:21.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.167+0000 7f8e8fe00640 1 -- 172.21.15.12:0/4045540185 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9801a470 con 0x56444a7fe1e0 2024-03-23T23:08:21.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.167+0000 7f8e8fe00640 1 -- 172.21.15.12:0/4045540185 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e98015070 con 0x56444a7fe1e0 2024-03-23T23:08:21.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.167+0000 7f8e8fe00640 1 -- 172.21.15.12:0/4045540185 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e98010900 con 0x56444a7fe1e0 2024-03-23T23:08:21.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.167+0000 7f8e9ea00640 1 --2- 172.21.15.12:0/1164595113 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8e840a28a0 0x7f8e840a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:21.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.167+0000 7f8e97400640 1 -- 172.21.15.12:0/1164595113 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8e900cee60 con 0x7f8ea0103900 2024-03-23T23:08:21.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.168+0000 7f8ea87dc740 1 -- 172.21.15.12:0/4045540185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a7fe1e0 msgr2=0x56444a7fe600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.168+0000 7f8ea87dc740 1 --2- 172.21.15.12:0/4045540185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a7fe1e0 0x56444a7fe600 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8e980039f0 tx=0x7f8e98003eb0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.168+0000 7f8e9ea00640 1 --2- 172.21.15.12:0/1164595113 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8e840a28a0 0x7f8e840a4d60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8e90098180 tx=0x7f8e900d28a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.168+0000 7f8ea87dc740 1 -- 172.21.15.12:0/4045540185 shutdown_connections 2024-03-23T23:08:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.168+0000 7f8ea87dc740 1 --2- 172.21.15.12:0/4045540185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56444a803d40 0x56444a8061d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.168+0000 7f8ea87dc740 1 --2- 172.21.15.12:0/4045540185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56444a7febd0 0x56444a8036d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.168+0000 7f8ea87dc740 1 --2- 172.21.15.12:0/4045540185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a7fe1e0 0x56444a7fe600 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.168+0000 7f8ea87dc740 1 -- 172.21.15.12:0/4045540185 >> 172.21.15.12:0/4045540185 conn(0x56444a7f9750 msgr2=0x56444a7f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.168+0000 7f8ea87dc740 1 -- 172.21.15.12:0/4045540185 shutdown_connections 2024-03-23T23:08:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.168+0000 7f8ea87dc740 1 -- 172.21.15.12:0/4045540185 wait complete. 2024-03-23T23:08:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.169+0000 7f8ea87dc740 1 -- start start 2024-03-23T23:08:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.169+0000 7f8ea87dc740 1 Processor -- start 2024-03-23T23:08:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.169+0000 7f8ea87dc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56444a7fe1e0 0x56444a8083d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.169+0000 7f8ea87dc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56444a7febd0 0x56444a808910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:21.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.169+0000 7f8ea87dc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a803d40 0x56444a80f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:21.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.169+0000 7f8ea87dc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56444a71fa20 con 0x56444a7fe1e0 2024-03-23T23:08:21.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.169+0000 7f8ea87dc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56444a75f8d0 con 0x56444a7febd0 2024-03-23T23:08:21.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.169+0000 7f8ea87dc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x56444a806cf0 con 0x56444a803d40 2024-03-23T23:08:21.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.170+0000 7f8ea87dc740 10 client.0 _ll_register_callbacks cb 0x56444a75c1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:21.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.170+0000 7f8e97e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56444a7febd0 0x56444a808910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:21.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.170+0000 7f8e9f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56444a7fe1e0 0x56444a8083d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:21.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.170+0000 7f8e9f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56444a7fe1e0 0x56444a8083d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41786/0 (socket says 172.21.15.12:41786) 2024-03-23T23:08:21.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.170+0000 7f8e97e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56444a7febd0 0x56444a808910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:40742/0 (socket says 172.21.15.12:40742) 2024-03-23T23:08:21.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.170+0000 7f8e9f400640 1 -- 172.21.15.12:0/1349093862 learned_addr learned my addr 172.21.15.12:0/1349093862 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:21.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.170+0000 7f8e97e00640 1 -- 172.21.15.12:0/1349093862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a803d40 msgr2=0x56444a80f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:21.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.170+0000 7f8e9f400640 1 --2- 172.21.15.12:0/1349093862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a803d40 0x56444a80f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:21.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.170+0000 7f8e97e00640 1 --2- 172.21.15.12:0/1349093862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a803d40 0x56444a80f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.170+0000 7f8e97e00640 1 -- 172.21.15.12:0/1349093862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56444a7fe1e0 msgr2=0x56444a8083d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.170+0000 7f8e97e00640 1 --2- 172.21.15.12:0/1349093862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56444a7fe1e0 0x56444a8083d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.170+0000 7f8e97e00640 1 -- 172.21.15.12:0/1349093862 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56444a478210 con 0x56444a7febd0 2024-03-23T23:08:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.170+0000 7f8e9f400640 1 --2- 172.21.15.12:0/1349093862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a803d40 0x56444a80f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:08:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.171+0000 7f8e97e00640 1 --2- 172.21.15.12:0/1349093862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56444a7febd0 0x56444a808910 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f8e88009230 tx=0x7f8e88004250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=640877e5bfcec7b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.171+0000 7f8e8cc00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:08:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.171+0000 7f8e8cc00640 1 -- 172.21.15.12:0/1349093862 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e88005d50 con 0x56444a7febd0 2024-03-23T23:08:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.171+0000 7f8e8cc00640 1 -- 172.21.15.12:0/1349093862 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8e88005230 con 0x56444a7febd0 2024-03-23T23:08:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.171+0000 7f8ea87dc740 1 -- 172.21.15.12:0/1349093862 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8e98015070 con 0x56444a7febd0 2024-03-23T23:08:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.171+0000 7f8e8cc00640 1 -- 172.21.15.12:0/1349093862 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e88005430 con 0x56444a7febd0 2024-03-23T23:08:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.171+0000 7f8ea87dc740 10 client.27146 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.171+0000 7f8ea87dc740 1 -- 172.21.15.12:0/1349093862 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56444a544620 con 0x56444a7febd0 2024-03-23T23:08:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.171+0000 7f8e8cc00640 1 -- 172.21.15.12:0/1349093862 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e98015070 con 0x56444a7febd0 2024-03-23T23:08:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.172+0000 7f8e8cc00640 1 -- 172.21.15.12:0/1349093862 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8e880037b0 con 0x56444a7febd0 2024-03-23T23:08:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.172+0000 7f8ea87dc740 10 client.27146 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.172+0000 7f8ea87dc740 10 client.27146 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:08:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.172+0000 7f8ea87dc740 20 client.27146 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:21.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.172+0000 7f8ea87dc740 1 --2- 172.21.15.12:0/1349093862 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56444a814180 0x56444a816570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:21.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.172+0000 7f8ea87dc740 4 client.27146 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:08:21.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.172+0000 7f8ea87dc740 1 -- 172.21.15.12:0/1349093862 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56444a770d70 con 0x56444a814180 2024-03-23T23:08:21.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.172+0000 7f8e9f400640 1 --2- 172.21.15.12:0/1349093862 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56444a814180 0x56444a816570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:21.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.173+0000 7f8e9f400640 1 --2- 172.21.15.12:0/1349093862 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56444a814180 0x56444a816570 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9874b65bcb376ec server_cookie=7d38fb94db37c0c3 in_seq=0 out_seq=0 2024-03-23T23:08:21.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.173+0000 7f8e8cc00640 10 client.27146 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.173+0000 7f8e8cc00640 1 -- 172.21.15.12:0/1349093862 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56444a770d70 con 0x56444a814180 2024-03-23T23:08:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.173+0000 7f8e8cc00640 10 client.27146 handle_command_reply: tid=0 2024-03-23T23:08:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.173+0000 7f8ea87dc740 1 client.27146 shutdown 2024-03-23T23:08:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.174+0000 7f8ea87dc740 1 -- 172.21.15.12:0/1349093862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56444a7febd0 msgr2=0x56444a808910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.174+0000 7f8ea87dc740 1 --2- 172.21.15.12:0/1349093862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56444a7febd0 0x56444a808910 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f8e88009230 tx=0x7f8e88004250 comp rx=0 tx=0).stop 2024-03-23T23:08:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.174+0000 7f8ea87dc740 1 -- 172.21.15.12:0/1349093862 shutdown_connections 2024-03-23T23:08:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.174+0000 7f8ea87dc740 1 --2- 172.21.15.12:0/1349093862 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56444a814180 0x56444a816570 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.174+0000 7f8ea87dc740 1 --2- 172.21.15.12:0/1349093862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a803d40 0x56444a80f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.174+0000 7f8ea87dc740 1 --2- 172.21.15.12:0/1349093862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56444a7febd0 0x56444a808910 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.174+0000 7f8ea87dc740 1 --2- 172.21.15.12:0/1349093862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56444a7fe1e0 0x56444a8083d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.174+0000 7f8ea87dc740 1 -- 172.21.15.12:0/1349093862 >> 172.21.15.12:0/1349093862 conn(0x56444a7f9750 msgr2=0x56444a7f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.174+0000 7f8e8cc00640 0 client.27146 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.174+0000 7f8ea87dc740 1 -- 172.21.15.12:0/1349093862 shutdown_connections 2024-03-23T23:08:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.174+0000 7f8ea87dc740 1 -- 172.21.15.12:0/1349093862 wait complete. 2024-03-23T23:08:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.174+0000 7f8ea87dc740 20 client.27146 trim_cache size 0 max 16384 2024-03-23T23:08:21.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.189+0000 7f8ea87dc740 1 Processor -- start 2024-03-23T23:08:21.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.189+0000 7f8ea87dc740 1 -- start start 2024-03-23T23:08:21.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.189+0000 7f8ea87dc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56444a7fe1e0 0x56444a816570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:21.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.189+0000 7f8ea87dc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a7febd0 0x56444a80f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.189+0000 7f8ea87dc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56444a803d40 0x56444a7fc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.189+0000 7f8ea87dc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea01ad190 con 0x56444a7fe1e0 2024-03-23T23:08:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.189+0000 7f8ea87dc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea01aac90 con 0x56444a803d40 2024-03-23T23:08:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.189+0000 7f8ea87dc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea01a8d30 con 0x56444a7febd0 2024-03-23T23:08:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.190+0000 7f8e97e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a7febd0 0x56444a80f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.190+0000 7f8e9f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56444a7fe1e0 0x56444a816570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.190+0000 7f8e97e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a7febd0 0x56444a80f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:46122/0 (socket says 172.21.15.12:46122) 2024-03-23T23:08:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.190+0000 7f8e97e00640 1 -- 172.21.15.12:0/1624874295 learned_addr learned my addr 172.21.15.12:0/1624874295 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.190+0000 7f8e9f400640 1 --2- 172.21.15.12:0/1624874295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56444a803d40 0x56444a7fc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.190+0000 7f8e97e00640 1 -- 172.21.15.12:0/1624874295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56444a803d40 msgr2=0x56444a7fc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.190+0000 7f8e97e00640 1 --2- 172.21.15.12:0/1624874295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56444a803d40 0x56444a7fc9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.190+0000 7f8e97e00640 1 -- 172.21.15.12:0/1624874295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56444a7fe1e0 msgr2=0x56444a816570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:21.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.190+0000 7f8e97e00640 1 --2- 172.21.15.12:0/1624874295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56444a7fe1e0 0x56444a816570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.190+0000 7f8e97e00640 1 -- 172.21.15.12:0/1624874295 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56444a816bd0 con 0x56444a7febd0 2024-03-23T23:08:21.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.190+0000 7f8e97e00640 1 --2- 172.21.15.12:0/1624874295 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a7febd0 0x56444a80f910 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f8e880092a0 tx=0x7f8e88005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a21072d643876749 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:21.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.191+0000 7f8e8f400640 1 -- 172.21.15.12:0/1624874295 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e88005d50 con 0x56444a7febd0 2024-03-23T23:08:21.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.191+0000 7f8e8f400640 1 -- 172.21.15.12:0/1624874295 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e8804b030 con 0x56444a7febd0 2024-03-23T23:08:21.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.191+0000 7f8e8f400640 1 -- 172.21.15.12:0/1624874295 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e88034b30 con 0x56444a7febd0 2024-03-23T23:08:21.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.191+0000 7f8ea87dc740 1 -- 172.21.15.12:0/1624874295 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a7febd0 msgr2=0x56444a80f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:21.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.191+0000 7f8ea87dc740 1 --2- 172.21.15.12:0/1624874295 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a7febd0 0x56444a80f910 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f8e880092a0 tx=0x7f8e88005fd0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.191+0000 7f8ea87dc740 1 -- 172.21.15.12:0/1624874295 shutdown_connections 2024-03-23T23:08:21.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.191+0000 7f8ea87dc740 1 --2- 172.21.15.12:0/1624874295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56444a803d40 0x56444a7fc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.191+0000 7f8ea87dc740 1 --2- 172.21.15.12:0/1624874295 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a7febd0 0x56444a80f910 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.191+0000 7f8ea87dc740 1 --2- 172.21.15.12:0/1624874295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56444a7fe1e0 0x56444a816570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.191+0000 7f8ea87dc740 1 -- 172.21.15.12:0/1624874295 >> 172.21.15.12:0/1624874295 conn(0x56444a7f9750 msgr2=0x56444a7f9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:21.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.192+0000 7f8ea87dc740 1 -- 172.21.15.12:0/1624874295 shutdown_connections 2024-03-23T23:08:21.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.192+0000 7f8ea87dc740 1 -- 172.21.15.12:0/1624874295 wait complete. 2024-03-23T23:08:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.192+0000 7f8ea87dc740 1 -- start start 2024-03-23T23:08:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.192+0000 7f8ea87dc740 1 Processor -- start 2024-03-23T23:08:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.192+0000 7f8ea87dc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a7fe1e0 0x56444a80f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.192+0000 7f8ea87dc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56444a7febd0 0x56444a80de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.193+0000 7f8ea87dc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56444a803d40 0x56444a80e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.193+0000 7f8ea87dc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea01a9330 con 0x56444a803d40 2024-03-23T23:08:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.193+0000 7f8ea87dc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea01aa630 con 0x56444a7febd0 2024-03-23T23:08:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.193+0000 7f8ea87dc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea01a9970 con 0x56444a7fe1e0 2024-03-23T23:08:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.193+0000 7f8ea87dc740 10 client.0 _ll_register_callbacks cb 0x7f8e88009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.193+0000 7f8e9f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a7fe1e0 0x56444a80f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.193+0000 7f8e97e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56444a7febd0 0x56444a80de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.193+0000 7f8e97e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56444a7febd0 0x56444a80de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:40754/0 (socket says 172.21.15.12:40754) 2024-03-23T23:08:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.193+0000 7f8e97e00640 1 -- 172.21.15.12:0/4006823600 learned_addr learned my addr 172.21.15.12:0/4006823600 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.193+0000 7f8e9f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a7fe1e0 0x56444a80f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:46128/0 (socket says 172.21.15.12:46128) 2024-03-23T23:08:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.193+0000 7f8e9f400640 1 --2- 172.21.15.12:0/4006823600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56444a803d40 0x56444a80e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.193+0000 7f8e97e00640 1 -- 172.21.15.12:0/4006823600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a7fe1e0 msgr2=0x56444a80f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.193+0000 7f8e97e00640 1 --2- 172.21.15.12:0/4006823600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a7fe1e0 0x56444a80f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.193+0000 7f8e97e00640 1 -- 172.21.15.12:0/4006823600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56444a803d40 msgr2=0x56444a80e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.193+0000 7f8e97e00640 1 --2- 172.21.15.12:0/4006823600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56444a803d40 0x56444a80e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.193+0000 7f8e97e00640 1 -- 172.21.15.12:0/4006823600 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56444a808c80 con 0x56444a7febd0 2024-03-23T23:08:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.194+0000 7f8e97e00640 1 --2- 172.21.15.12:0/4006823600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56444a7febd0 0x56444a80de60 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f8e8802c170 tx=0x7f8e88003a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3fa53651f0b0cc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.194+0000 7f8e8cc00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:08:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.194+0000 7f8e8cc00640 1 -- 172.21.15.12:0/4006823600 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e88005d50 con 0x56444a7febd0 2024-03-23T23:08:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.194+0000 7f8ea87dc740 1 -- 172.21.15.12:0/4006823600 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8e8804b030 con 0x56444a7febd0 2024-03-23T23:08:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.194+0000 7f8e8cc00640 1 -- 172.21.15.12:0/4006823600 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8e88034870 con 0x56444a7febd0 2024-03-23T23:08:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.194+0000 7f8e8cc00640 1 -- 172.21.15.12:0/4006823600 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e88067050 con 0x56444a7febd0 2024-03-23T23:08:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.194+0000 7f8ea87dc740 10 client.27152 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:21.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.194+0000 7f8ea87dc740 1 -- 172.21.15.12:0/4006823600 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56444a8708a0 con 0x56444a7febd0 2024-03-23T23:08:21.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.194+0000 7f8e8cc00640 1 -- 172.21.15.12:0/4006823600 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e8804b030 con 0x56444a7febd0 2024-03-23T23:08:21.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.194+0000 7f8e8cc00640 1 -- 172.21.15.12:0/4006823600 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8e88058820 con 0x56444a7febd0 2024-03-23T23:08:21.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.195+0000 7f8ea87dc740 10 client.27152 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:21.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.195+0000 7f8ea87dc740 10 client.27152 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:08:21.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.195+0000 7f8ea87dc740 20 client.27152 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:21.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.195+0000 7f8ea87dc740 1 --2- 172.21.15.12:0/4006823600 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56444a871190 0x56444a8735b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:21.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.195+0000 7f8ea87dc740 4 client.27152 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:08:21.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.195+0000 7f8ea87dc740 1 -- 172.21.15.12:0/4006823600 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56444a76b0c0 con 0x56444a871190 2024-03-23T23:08:21.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.195+0000 7f8e9f400640 1 --2- 172.21.15.12:0/4006823600 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56444a871190 0x56444a8735b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:21.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.195+0000 7f8e9f400640 1 --2- 172.21.15.12:0/4006823600 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56444a871190 0x56444a8735b0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3523b59ca26a916 server_cookie=b641faa8f259b7fe in_seq=0 out_seq=0 2024-03-23T23:08:21.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.195+0000 7f8e8cc00640 10 client.27152 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.209+0000 7f8e8cc00640 1 -- 172.21.15.12:0/4006823600 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56444a76b0c0 con 0x56444a871190 2024-03-23T23:08:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.209+0000 7f8e8cc00640 10 client.27152 handle_command_reply: tid=0 2024-03-23T23:08:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.209+0000 7f8ea87dc740 1 client.27152 shutdown 2024-03-23T23:08:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.209+0000 7f8ea87dc740 1 -- 172.21.15.12:0/4006823600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56444a7febd0 msgr2=0x56444a80de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.209+0000 7f8ea87dc740 1 --2- 172.21.15.12:0/4006823600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56444a7febd0 0x56444a80de60 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f8e8802c170 tx=0x7f8e88003a80 comp rx=0 tx=0).stop 2024-03-23T23:08:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.209+0000 7f8ea87dc740 1 -- 172.21.15.12:0/4006823600 shutdown_connections 2024-03-23T23:08:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.209+0000 7f8ea87dc740 1 --2- 172.21.15.12:0/4006823600 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56444a871190 0x56444a8735b0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.209+0000 7f8ea87dc740 1 --2- 172.21.15.12:0/4006823600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56444a803d40 0x56444a80e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.209+0000 7f8ea87dc740 1 --2- 172.21.15.12:0/4006823600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56444a7febd0 0x56444a80de60 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.209+0000 7f8ea87dc740 1 --2- 172.21.15.12:0/4006823600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56444a7fe1e0 0x56444a80f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.209+0000 7f8e8cc00640 0 client.27152 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.209+0000 7f8ea87dc740 1 -- 172.21.15.12:0/4006823600 >> 172.21.15.12:0/4006823600 conn(0x56444a7f9750 msgr2=0x56444a7fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.209+0000 7f8ea87dc740 1 -- 172.21.15.12:0/4006823600 shutdown_connections 2024-03-23T23:08:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.209+0000 7f8ea87dc740 1 -- 172.21.15.12:0/4006823600 wait complete. 2024-03-23T23:08:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.210+0000 7f8ea87dc740 20 client.27152 trim_cache size 0 max 16384 2024-03-23T23:08:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:08:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:08:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:08:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:08:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.210+0000 7f8ea5a00640 1 -- 172.21.15.12:0/1164595113 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8e840a28a0 msgr2=0x7f8e840a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.210+0000 7f8ea5a00640 1 --2- 172.21.15.12:0/1164595113 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8e840a28a0 0x7f8e840a4d60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8e90098180 tx=0x7f8e900d28a0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.210+0000 7f8ea5a00640 1 -- 172.21.15.12:0/1164595113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ea0103900 msgr2=0x7f8ea0199b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.210+0000 7f8ea5a00640 1 --2- 172.21.15.12:0/1164595113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ea0103900 0x7f8ea0199b70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8e9000e9c0 tx=0x7f8e9000ee80 comp rx=0 tx=0).stop 2024-03-23T23:08:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.210+0000 7f8ea5a00640 1 -- 172.21.15.12:0/1164595113 shutdown_connections 2024-03-23T23:08:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.210+0000 7f8ea5a00640 1 --2- 172.21.15.12:0/1164595113 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8e840a28a0 0x7f8e840a4d60 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.210+0000 7f8ea5a00640 1 --2- 172.21.15.12:0/1164595113 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ea01054b0 0x7f8ea01a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.210+0000 7f8ea5a00640 1 --2- 172.21.15.12:0/1164595113 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ea0104af0 0x7f8ea019a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.210+0000 7f8ea5a00640 1 --2- 172.21.15.12:0/1164595113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ea0103900 0x7f8ea0199b70 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.210+0000 7f8ea5a00640 1 -- 172.21.15.12:0/1164595113 >> 172.21.15.12:0/1164595113 conn(0x7f8ea00fee90 msgr2=0x7f8ea0107d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.211+0000 7f8ea5a00640 1 -- 172.21.15.12:0/1164595113 shutdown_connections 2024-03-23T23:08:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:21.211+0000 7f8ea5a00640 1 -- 172.21.15.12:0/1164595113 wait complete. 2024-03-23T23:08:21.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:20 smithi012 ceph-mon[31695]: pgmap v586: 97 pgs: 97 active+clean; 4.2 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:08:23.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:22 smithi173 ceph-mon[36116]: pgmap v587: 97 pgs: 97 active+clean; 4.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:08:23.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:22 smithi203 ceph-mon[36693]: pgmap v587: 97 pgs: 97 active+clean; 4.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:08:23.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:22 smithi012 ceph-mon[31695]: pgmap v587: 97 pgs: 97 active+clean; 4.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:08:25.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:24 smithi173 ceph-mon[36116]: pgmap v588: 97 pgs: 97 active+clean; 4.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:08:25.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:24 smithi203 ceph-mon[36693]: pgmap v588: 97 pgs: 97 active+clean; 4.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:08:25.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:24 smithi012 ceph-mon[31695]: pgmap v588: 97 pgs: 97 active+clean; 4.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:08:27.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:26 smithi173 ceph-mon[36116]: pgmap v589: 97 pgs: 97 active+clean; 4.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:08:27.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:26 smithi203 ceph-mon[36693]: pgmap v589: 97 pgs: 97 active+clean; 4.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:08:27.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:26 smithi012 ceph-mon[31695]: pgmap v589: 97 pgs: 97 active+clean; 4.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:08:29.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:28 smithi173 ceph-mon[36116]: pgmap v590: 97 pgs: 97 active+clean; 4.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:08:29.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:28 smithi203 ceph-mon[36693]: pgmap v590: 97 pgs: 97 active+clean; 4.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:08:29.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:28 smithi012 ceph-mon[31695]: pgmap v590: 97 pgs: 97 active+clean; 4.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:08:31.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:30 smithi173 ceph-mon[36116]: pgmap v591: 97 pgs: 97 active+clean; 4.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:08:31.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:30 smithi203 ceph-mon[36693]: pgmap v591: 97 pgs: 97 active+clean; 4.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:08:31.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:30 smithi012 ceph-mon[31695]: pgmap v591: 97 pgs: 97 active+clean; 4.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:08:33.106 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:32 smithi012 ceph-mon[31695]: pgmap v592: 97 pgs: 97 active+clean; 4.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:08:33.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:32 smithi173 ceph-mon[36116]: pgmap v592: 97 pgs: 97 active+clean; 4.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:08:33.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:32 smithi203 ceph-mon[36693]: pgmap v592: 97 pgs: 97 active+clean; 4.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:08:34.359 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.358+0000 7f0228400640 1 -- 172.21.15.12:0/1695616079 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0220101800 msgr2=0x7f0220101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:34.359 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.358+0000 7f0228400640 1 --2- 172.21.15.12:0/1695616079 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0220101800 0x7f0220101c80 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f020c00d910 tx=0x7f020c02dbb0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.359 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.358+0000 7f0228400640 1 -- 172.21.15.12:0/1695616079 shutdown_connections 2024-03-23T23:08:34.359 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.358+0000 7f0228400640 1 --2- 172.21.15.12:0/1695616079 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02201021c0 0x7f0220108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.359 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.358+0000 7f0228400640 1 --2- 172.21.15.12:0/1695616079 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0220101800 0x7f0220101c80 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.359 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.358+0000 7f0228400640 1 --2- 172.21.15.12:0/1695616079 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0220100610 0x7f0220100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.359 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.358+0000 7f0228400640 1 -- 172.21.15.12:0/1695616079 >> 172.21.15.12:0/1695616079 conn(0x7f02200fbb80 msgr2=0x7f02200fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:34.359 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.359+0000 7f0228400640 1 -- 172.21.15.12:0/1695616079 shutdown_connections 2024-03-23T23:08:34.359 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.359+0000 7f0228400640 1 -- 172.21.15.12:0/1695616079 wait complete. 2024-03-23T23:08:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.360+0000 7f0228400640 1 Processor -- start 2024-03-23T23:08:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.360+0000 7f0228400640 1 -- start start 2024-03-23T23:08:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.360+0000 7f0228400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0220100610 0x7f0220199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.360+0000 7f0228400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0220101800 0x7f022019a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.361+0000 7f0228400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02201021c0 0x7f02201a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.361+0000 7f0228400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022010b1e0 con 0x7f02201021c0 2024-03-23T23:08:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.361+0000 7f0228400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022010b060 con 0x7f0220100610 2024-03-23T23:08:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.361+0000 7f0228400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022010b360 con 0x7f0220101800 2024-03-23T23:08:34.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.361+0000 7f0225a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0220100610 0x7f0220199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:34.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.361+0000 7f0225a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0220100610 0x7f0220199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57208/0 (socket says 172.21.15.12:57208) 2024-03-23T23:08:34.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.361+0000 7f0225a00640 1 -- 172.21.15.12:0/2705925758 learned_addr learned my addr 172.21.15.12:0/2705925758 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:34.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.361+0000 7f0225000640 1 --2- 172.21.15.12:0/2705925758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0220101800 0x7f022019a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:34.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.361+0000 7f0226400640 1 --2- 172.21.15.12:0/2705925758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02201021c0 0x7f02201a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:34.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.361+0000 7f0225a00640 1 -- 172.21.15.12:0/2705925758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0220101800 msgr2=0x7f022019a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:34.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.361+0000 7f0225a00640 1 --2- 172.21.15.12:0/2705925758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0220101800 0x7f022019a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.361+0000 7f0225a00640 1 -- 172.21.15.12:0/2705925758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02201021c0 msgr2=0x7f02201a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:34.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.361+0000 7f0225a00640 1 --2- 172.21.15.12:0/2705925758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02201021c0 0x7f02201a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.361+0000 7f0225a00640 1 -- 172.21.15.12:0/2705925758 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02201a17b0 con 0x7f0220100610 2024-03-23T23:08:34.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.361+0000 7f0226400640 1 --2- 172.21.15.12:0/2705925758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02201021c0 0x7f02201a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:08:34.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.362+0000 7f0225a00640 1 --2- 172.21.15.12:0/2705925758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0220100610 0x7f0220199be0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f021400e9c0 tx=0x7f021400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:34.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.362+0000 7f021e000640 1 -- 172.21.15.12:0/2705925758 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021400cd80 con 0x7f0220100610 2024-03-23T23:08:34.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.362+0000 7f021e000640 1 -- 172.21.15.12:0/2705925758 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f02140035f0 con 0x7f0220100610 2024-03-23T23:08:34.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.362+0000 7f021e000640 1 -- 172.21.15.12:0/2705925758 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0214010690 con 0x7f0220100610 2024-03-23T23:08:34.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.362+0000 7f0228400640 1 -- 172.21.15.12:0/2705925758 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02201a1aa0 con 0x7f0220100610 2024-03-23T23:08:34.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.362+0000 7f0228400640 1 -- 172.21.15.12:0/2705925758 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02201a1fc0 con 0x7f0220100610 2024-03-23T23:08:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.365+0000 7f022b195740 1 Processor -- start 2024-03-23T23:08:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.365+0000 7f022b195740 1 -- start start 2024-03-23T23:08:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.366+0000 7f022b195740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557570fab1e0 0x557570fab600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.366+0000 7f022b195740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557570fabbd0 0x557570fb06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.366+0000 7f022b195740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fb0d40 0x557570fb31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.366+0000 7f022b195740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x557570f6e3b0 con 0x557570fabbd0 2024-03-23T23:08:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.366+0000 7f022b195740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x557570f10830 con 0x557570fb0d40 2024-03-23T23:08:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.366+0000 7f022b195740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x557570eb95d0 con 0x557570fab1e0 2024-03-23T23:08:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.366+0000 7f0226400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557570fab1e0 0x557570fab600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.366+0000 7f0226400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557570fab1e0 0x557570fab600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:42158/0 (socket says 172.21.15.12:42158) 2024-03-23T23:08:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.366+0000 7f0226400640 1 -- 172.21.15.12:0/1585304340 learned_addr learned my addr 172.21.15.12:0/1585304340 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.366+0000 7f0225000640 1 --2- 172.21.15.12:0/1585304340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fb0d40 0x557570fb31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.366+0000 7f021e000640 1 -- 172.21.15.12:0/2705925758 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f02140037a0 con 0x7f0220100610 2024-03-23T23:08:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.366+0000 7f0226400640 1 -- 172.21.15.12:0/1585304340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fb0d40 msgr2=0x557570fb31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.366+0000 7f0226400640 1 --2- 172.21.15.12:0/1585304340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fb0d40 0x557570fb31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.366+0000 7f0226400640 1 -- 172.21.15.12:0/1585304340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557570fabbd0 msgr2=0x557570fb06d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T23:08:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.366+0000 7f0226400640 1 --2- 172.21.15.12:0/1585304340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557570fabbd0 0x557570fb06d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.366+0000 7f0226400640 1 -- 172.21.15.12:0/1585304340 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557570ed12c0 con 0x557570fab1e0 2024-03-23T23:08:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.366+0000 7f0225000640 1 --2- 172.21.15.12:0/1585304340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fb0d40 0x557570fb31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:08:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.367+0000 7f0226400640 1 --2- 172.21.15.12:0/1585304340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557570fab1e0 0x557570fab600 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0218010450 tx=0x7f0218010910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edafca53a95cfc94 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.367+0000 7f0212a00640 1 -- 172.21.15.12:0/1585304340 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0218017070 con 0x557570fab1e0 2024-03-23T23:08:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.367+0000 7f0212a00640 1 -- 172.21.15.12:0/1585304340 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0218011c40 con 0x557570fab1e0 2024-03-23T23:08:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.367+0000 7f0212a00640 1 -- 172.21.15.12:0/1585304340 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021801adb0 con 0x557570fab1e0 2024-03-23T23:08:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.368+0000 7f022b195740 1 -- 172.21.15.12:0/1585304340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557570fab1e0 msgr2=0x557570fab600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.368+0000 7f022b195740 1 --2- 172.21.15.12:0/1585304340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557570fab1e0 0x557570fab600 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0218010450 tx=0x7f0218010910 comp rx=0 tx=0).stop 2024-03-23T23:08:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.368+0000 7f022b195740 1 -- 172.21.15.12:0/1585304340 shutdown_connections 2024-03-23T23:08:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.368+0000 7f022b195740 1 --2- 172.21.15.12:0/1585304340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fb0d40 0x557570fb31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.368+0000 7f022b195740 1 --2- 172.21.15.12:0/1585304340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557570fabbd0 0x557570fb06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.368+0000 7f022b195740 1 --2- 172.21.15.12:0/1585304340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557570fab1e0 0x557570fab600 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.368+0000 7f022b195740 1 -- 172.21.15.12:0/1585304340 >> 172.21.15.12:0/1585304340 conn(0x557570fa6750 msgr2=0x557570fa6b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.368+0000 7f022b195740 1 -- 172.21.15.12:0/1585304340 shutdown_connections 2024-03-23T23:08:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.368+0000 7f022b195740 1 -- 172.21.15.12:0/1585304340 wait complete. 2024-03-23T23:08:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.368+0000 7f021e000640 1 --2- 172.21.15.12:0/2705925758 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f02080a27b0 0x7f02080a4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.369+0000 7f0225000640 1 --2- 172.21.15.12:0/2705925758 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f02080a27b0 0x7f02080a4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.369+0000 7f021e000640 1 -- 172.21.15.12:0/2705925758 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f02140ceeb0 con 0x7f0220100610 2024-03-23T23:08:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.369+0000 7f022b195740 1 -- start start 2024-03-23T23:08:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.369+0000 7f022b195740 1 Processor -- start 2024-03-23T23:08:34.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.370+0000 7f0225000640 1 --2- 172.21.15.12:0/2705925758 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f02080a27b0 0x7f02080a4c70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f020c02e1c0 tx=0x7f020c004cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:34.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.370+0000 7f022b195740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fab1e0 0x557570fb53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:34.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.370+0000 7f022b195740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557570fabbd0 0x557570fb5910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:34.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.370+0000 7f022b195740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557570fb0d40 0x557570fbc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:34.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.370+0000 7f022b195740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x557570ecca20 con 0x557570fb0d40 2024-03-23T23:08:34.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.370+0000 7f022b195740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x557570f0c8d0 con 0x557570fab1e0 2024-03-23T23:08:34.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.370+0000 7f022b195740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x557570fb3cf0 con 0x557570fabbd0 2024-03-23T23:08:34.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.370+0000 7f022b195740 10 client.0 _ll_register_callbacks cb 0x557570f091c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:34.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.370+0000 7f0225a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fab1e0 0x557570fb53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:34.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.370+0000 7f0225a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fab1e0 0x557570fb53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57226/0 (socket says 172.21.15.12:57226) 2024-03-23T23:08:34.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.370+0000 7f0225a00640 1 -- 172.21.15.12:0/3918852482 learned_addr learned my addr 172.21.15.12:0/3918852482 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.370+0000 7f0225000640 1 --2- 172.21.15.12:0/3918852482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557570fabbd0 0x557570fb5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.370+0000 7f0226400640 1 --2- 172.21.15.12:0/3918852482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557570fb0d40 0x557570fbc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.370+0000 7f0225a00640 1 -- 172.21.15.12:0/3918852482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557570fabbd0 msgr2=0x557570fb5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.370+0000 7f0225a00640 1 --2- 172.21.15.12:0/3918852482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557570fabbd0 0x557570fb5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.370+0000 7f0225a00640 1 -- 172.21.15.12:0/3918852482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557570fb0d40 msgr2=0x557570fbc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.370+0000 7f0225a00640 1 --2- 172.21.15.12:0/3918852482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557570fb0d40 0x557570fbc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.370+0000 7f0225a00640 1 -- 172.21.15.12:0/3918852482 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557570c25210 con 0x557570fab1e0 2024-03-23T23:08:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.371+0000 7f0225a00640 1 --2- 172.21.15.12:0/3918852482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fab1e0 0x557570fb53d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f0214097cc0 tx=0x7f021409bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=149d0c3cd2a9176e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.371+0000 7f0207400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:08:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.371+0000 7f0207400640 1 -- 172.21.15.12:0/3918852482 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02140a4000 con 0x557570fab1e0 2024-03-23T23:08:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.371+0000 7f0207400640 1 -- 172.21.15.12:0/3918852482 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0214003b30 con 0x557570fab1e0 2024-03-23T23:08:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.371+0000 7f022b195740 1 -- 172.21.15.12:0/3918852482 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0218011c40 con 0x557570fab1e0 2024-03-23T23:08:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.371+0000 7f0207400640 1 -- 172.21.15.12:0/3918852482 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02140a3b70 con 0x557570fab1e0 2024-03-23T23:08:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.371+0000 7f022b195740 10 client.27164 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.371+0000 7f022b195740 1 -- 172.21.15.12:0/3918852482 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x557570cf1620 con 0x557570fab1e0 2024-03-23T23:08:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.371+0000 7f0207400640 1 -- 172.21.15.12:0/3918852482 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0218011c40 con 0x557570fab1e0 2024-03-23T23:08:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.372+0000 7f0207400640 1 -- 172.21.15.12:0/3918852482 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f021409acb0 con 0x557570fab1e0 2024-03-23T23:08:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.372+0000 7f022b195740 10 client.27164 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.372+0000 7f022b195740 10 client.27164 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:08:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.372+0000 7f022b195740 20 client.27164 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.372+0000 7f022b195740 1 --2- 172.21.15.12:0/3918852482 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557570fc1190 0x557570fc35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.372+0000 7f022b195740 4 client.27164 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:08:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.372+0000 7f022b195740 1 -- 172.21.15.12:0/3918852482 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557570f1dd70 con 0x557570fc1190 2024-03-23T23:08:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.372+0000 7f0225000640 1 --2- 172.21.15.12:0/3918852482 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557570fc1190 0x557570fc35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.373+0000 7f0225000640 1 --2- 172.21.15.12:0/3918852482 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557570fc1190 0x557570fc35a0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=919a5c2f13cd137b server_cookie=7e9ce936e4b7ae8 in_seq=0 out_seq=0 2024-03-23T23:08:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.373+0000 7f0207400640 10 client.27164 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.373+0000 7f0207400640 1 -- 172.21.15.12:0/3918852482 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557570f1dd70 con 0x557570fc1190 2024-03-23T23:08:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.373+0000 7f0207400640 10 client.27164 handle_command_reply: tid=0 2024-03-23T23:08:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.373+0000 7f022b195740 1 client.27164 shutdown 2024-03-23T23:08:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.373+0000 7f022b195740 1 -- 172.21.15.12:0/3918852482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fab1e0 msgr2=0x557570fb53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.373+0000 7f022b195740 1 --2- 172.21.15.12:0/3918852482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fab1e0 0x557570fb53d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f0214097cc0 tx=0x7f021409bf60 comp rx=0 tx=0).stop 2024-03-23T23:08:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.374+0000 7f022b195740 1 -- 172.21.15.12:0/3918852482 shutdown_connections 2024-03-23T23:08:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.374+0000 7f022b195740 1 --2- 172.21.15.12:0/3918852482 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557570fc1190 0x557570fc35a0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.374+0000 7f022b195740 1 --2- 172.21.15.12:0/3918852482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557570fb0d40 0x557570fbc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.374+0000 7f022b195740 1 --2- 172.21.15.12:0/3918852482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557570fabbd0 0x557570fb5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.374+0000 7f022b195740 1 --2- 172.21.15.12:0/3918852482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fab1e0 0x557570fb53d0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.374+0000 7f022b195740 1 -- 172.21.15.12:0/3918852482 >> 172.21.15.12:0/3918852482 conn(0x557570fa6750 msgr2=0x557570fa5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.374+0000 7f0207400640 0 client.27164 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.374+0000 7f022b195740 1 -- 172.21.15.12:0/3918852482 shutdown_connections 2024-03-23T23:08:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.374+0000 7f022b195740 1 -- 172.21.15.12:0/3918852482 wait complete. 2024-03-23T23:08:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.374+0000 7f022b195740 20 client.27164 trim_cache size 0 max 16384 2024-03-23T23:08:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.389+0000 7f022b195740 1 Processor -- start 2024-03-23T23:08:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.389+0000 7f022b195740 1 -- start start 2024-03-23T23:08:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.389+0000 7f022b195740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557570fab1e0 0x557570fb5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.389+0000 7f022b195740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557570fabbd0 0x557570fbc910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.389+0000 7f022b195740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fb0d40 0x557570fa15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.389+0000 7f022b195740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02201ad170 con 0x557570fab1e0 2024-03-23T23:08:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.389+0000 7f022b195740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02201aac70 con 0x557570fb0d40 2024-03-23T23:08:34.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.389+0000 7f022b195740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02201a8d10 con 0x557570fabbd0 2024-03-23T23:08:34.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.390+0000 7f0226400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fb0d40 0x557570fa15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:34.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.390+0000 7f0226400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fb0d40 0x557570fa15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57238/0 (socket says 172.21.15.12:57238) 2024-03-23T23:08:34.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.390+0000 7f0226400640 1 -- 172.21.15.12:0/2735882065 learned_addr learned my addr 172.21.15.12:0/2735882065 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:34.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.390+0000 7f0225000640 1 --2- 172.21.15.12:0/2735882065 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557570fabbd0 0x557570fbc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.390+0000 7f0225a00640 1 --2- 172.21.15.12:0/2735882065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557570fab1e0 0x557570fb5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.390+0000 7f0226400640 1 -- 172.21.15.12:0/2735882065 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557570fabbd0 msgr2=0x557570fbc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.390+0000 7f0226400640 1 --2- 172.21.15.12:0/2735882065 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557570fabbd0 0x557570fbc910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.390+0000 7f0226400640 1 -- 172.21.15.12:0/2735882065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557570fab1e0 msgr2=0x557570fb5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.390+0000 7f0226400640 1 --2- 172.21.15.12:0/2735882065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557570fab1e0 0x557570fb5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.390+0000 7f0226400640 1 -- 172.21.15.12:0/2735882065 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557570fa4d40 con 0x557570fb0d40 2024-03-23T23:08:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.390+0000 7f0225000640 1 --2- 172.21.15.12:0/2735882065 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557570fabbd0 0x557570fbc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.390+0000 7f0226400640 1 --2- 172.21.15.12:0/2735882065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fb0d40 0x557570fa15d0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x557570fac430 tx=0x7f0218009be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88523ddbfd6fccf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.391+0000 7f0212000640 1 -- 172.21.15.12:0/2735882065 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0218017040 con 0x557570fb0d40 2024-03-23T23:08:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.391+0000 7f0212000640 1 -- 172.21.15.12:0/2735882065 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f021802a070 con 0x557570fb0d40 2024-03-23T23:08:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.391+0000 7f0212000640 1 -- 172.21.15.12:0/2735882065 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02180043c0 con 0x557570fb0d40 2024-03-23T23:08:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.391+0000 7f022b195740 1 -- 172.21.15.12:0/2735882065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fb0d40 msgr2=0x557570fa15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.391+0000 7f022b195740 1 --2- 172.21.15.12:0/2735882065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fb0d40 0x557570fa15d0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x557570fac430 tx=0x7f0218009be0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.391+0000 7f022b195740 1 -- 172.21.15.12:0/2735882065 shutdown_connections 2024-03-23T23:08:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.391+0000 7f022b195740 1 --2- 172.21.15.12:0/2735882065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fb0d40 0x557570fa15d0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.391+0000 7f022b195740 1 --2- 172.21.15.12:0/2735882065 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557570fabbd0 0x557570fbc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.391+0000 7f022b195740 1 --2- 172.21.15.12:0/2735882065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557570fab1e0 0x557570fb5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.391+0000 7f022b195740 1 -- 172.21.15.12:0/2735882065 >> 172.21.15.12:0/2735882065 conn(0x557570fa6750 msgr2=0x557570fa99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.391+0000 7f022b195740 1 -- 172.21.15.12:0/2735882065 shutdown_connections 2024-03-23T23:08:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.391+0000 7f022b195740 1 -- 172.21.15.12:0/2735882065 wait complete. 2024-03-23T23:08:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.392+0000 7f022b195740 1 -- start start 2024-03-23T23:08:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.392+0000 7f022b195740 1 Processor -- start 2024-03-23T23:08:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.392+0000 7f022b195740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557570fab1e0 0x557570fbc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.392+0000 7f022b195740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557570fabbd0 0x557570fbc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.392+0000 7f022b195740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fb0d40 0x557570fba9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.392+0000 7f022b195740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02201a9310 con 0x557570fabbd0 2024-03-23T23:08:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.392+0000 7f022b195740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02201aa610 con 0x557570fb0d40 2024-03-23T23:08:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.392+0000 7f022b195740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02201a9950 con 0x557570fab1e0 2024-03-23T23:08:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.392+0000 7f022b195740 10 client.0 _ll_register_callbacks cb 0x7f021409b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.393+0000 7f0226400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fb0d40 0x557570fba9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.393+0000 7f0226400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fb0d40 0x557570fba9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57242/0 (socket says 172.21.15.12:57242) 2024-03-23T23:08:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.393+0000 7f0226400640 1 -- 172.21.15.12:0/1981484428 learned_addr learned my addr 172.21.15.12:0/1981484428 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.393+0000 7f0225a00640 1 --2- 172.21.15.12:0/1981484428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557570fab1e0 0x557570fbc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.393+0000 7f0225000640 1 --2- 172.21.15.12:0/1981484428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557570fabbd0 0x557570fbc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.393+0000 7f0226400640 1 -- 172.21.15.12:0/1981484428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557570fab1e0 msgr2=0x557570fbc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.393+0000 7f0226400640 1 --2- 172.21.15.12:0/1981484428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557570fab1e0 0x557570fbc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.393+0000 7f0226400640 1 -- 172.21.15.12:0/1981484428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557570fabbd0 msgr2=0x557570fbc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.393+0000 7f0226400640 1 --2- 172.21.15.12:0/1981484428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557570fabbd0 0x557570fbc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.393+0000 7f0226400640 1 -- 172.21.15.12:0/1981484428 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557570fbb1d0 con 0x557570fb0d40 2024-03-23T23:08:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.393+0000 7f0225a00640 1 --2- 172.21.15.12:0/1981484428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557570fab1e0 0x557570fbc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.393+0000 7f0226400640 1 --2- 172.21.15.12:0/1981484428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fb0d40 0x557570fba9a0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f0218010e10 tx=0x7f0218004370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b03eee95a64a39b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.393+0000 7f0207400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:08:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.394+0000 7f0207400640 1 -- 172.21.15.12:0/1981484428 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0218033060 con 0x557570fb0d40 2024-03-23T23:08:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.394+0000 7f0207400640 1 -- 172.21.15.12:0/1981484428 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f02180047a0 con 0x557570fb0d40 2024-03-23T23:08:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.394+0000 7f022b195740 1 -- 172.21.15.12:0/1981484428 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f021802a070 con 0x557570fb0d40 2024-03-23T23:08:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.394+0000 7f0207400640 1 -- 172.21.15.12:0/1981484428 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0218017040 con 0x557570fb0d40 2024-03-23T23:08:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.394+0000 7f022b195740 10 client.27170 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.394+0000 7f022b195740 1 -- 172.21.15.12:0/1981484428 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55757101d5c0 con 0x557570fb0d40 2024-03-23T23:08:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.394+0000 7f0207400640 1 -- 172.21.15.12:0/1981484428 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f021802a070 con 0x557570fb0d40 2024-03-23T23:08:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.394+0000 7f0207400640 1 -- 172.21.15.12:0/1981484428 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f0218025470 con 0x557570fb0d40 2024-03-23T23:08:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.394+0000 7f022b195740 10 client.27170 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.394+0000 7f022b195740 10 client.27170 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:08:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.394+0000 7f022b195740 20 client.27170 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.394+0000 7f022b195740 1 --2- 172.21.15.12:0/1981484428 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55757101dcd0 0x5575710200f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.395+0000 7f022b195740 4 client.27170 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:08:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.395+0000 7f022b195740 1 -- 172.21.15.12:0/1981484428 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557570f180c0 con 0x55757101dcd0 2024-03-23T23:08:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.395+0000 7f0225a00640 1 --2- 172.21.15.12:0/1981484428 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55757101dcd0 0x5575710200f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.395+0000 7f0225a00640 1 --2- 172.21.15.12:0/1981484428 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55757101dcd0 0x5575710200f0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e342354b126c83ed server_cookie=bdc241524196e92f in_seq=0 out_seq=0 2024-03-23T23:08:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.395+0000 7f0207400640 10 client.27170 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:34.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.409+0000 7f0207400640 1 -- 172.21.15.12:0/1981484428 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557570f180c0 con 0x55757101dcd0 2024-03-23T23:08:34.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.409+0000 7f0207400640 10 client.27170 handle_command_reply: tid=0 2024-03-23T23:08:34.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.410+0000 7f022b195740 1 client.27170 shutdown 2024-03-23T23:08:34.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.410+0000 7f022b195740 1 -- 172.21.15.12:0/1981484428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fb0d40 msgr2=0x557570fba9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:34.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.410+0000 7f022b195740 1 --2- 172.21.15.12:0/1981484428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fb0d40 0x557570fba9a0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f0218010e10 tx=0x7f0218004370 comp rx=0 tx=0).stop 2024-03-23T23:08:34.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.410+0000 7f022b195740 1 -- 172.21.15.12:0/1981484428 shutdown_connections 2024-03-23T23:08:34.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.410+0000 7f022b195740 1 --2- 172.21.15.12:0/1981484428 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55757101dcd0 0x5575710200f0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.410+0000 7f022b195740 1 --2- 172.21.15.12:0/1981484428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557570fb0d40 0x557570fba9a0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.410+0000 7f022b195740 1 --2- 172.21.15.12:0/1981484428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557570fabbd0 0x557570fbc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.411 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.410+0000 7f022b195740 1 --2- 172.21.15.12:0/1981484428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557570fab1e0 0x557570fbc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.411 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.410+0000 7f0207400640 0 client.27170 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:34.411 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.410+0000 7f022b195740 1 -- 172.21.15.12:0/1981484428 >> 172.21.15.12:0/1981484428 conn(0x557570fa6750 msgr2=0x557570faa240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:34.411 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.410+0000 7f022b195740 1 -- 172.21.15.12:0/1981484428 shutdown_connections 2024-03-23T23:08:34.411 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.410+0000 7f022b195740 1 -- 172.21.15.12:0/1981484428 wait complete. 2024-03-23T23:08:34.411 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.410+0000 7f022b195740 20 client.27170 trim_cache size 0 max 16384 2024-03-23T23:08:34.411 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:08:34.411 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:08:34.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:08:34.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:08:34.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.411+0000 7f0228400640 1 -- 172.21.15.12:0/2705925758 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f02080a27b0 msgr2=0x7f02080a4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:34.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.411+0000 7f0228400640 1 --2- 172.21.15.12:0/2705925758 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f02080a27b0 0x7f02080a4c70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f020c02e1c0 tx=0x7f020c004cf0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.411+0000 7f0228400640 1 -- 172.21.15.12:0/2705925758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0220100610 msgr2=0x7f0220199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:34.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.411+0000 7f0228400640 1 --2- 172.21.15.12:0/2705925758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0220100610 0x7f0220199be0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f021400e9c0 tx=0x7f021400ee80 comp rx=0 tx=0).stop 2024-03-23T23:08:34.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.411+0000 7f0228400640 1 -- 172.21.15.12:0/2705925758 shutdown_connections 2024-03-23T23:08:34.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.411+0000 7f0228400640 1 --2- 172.21.15.12:0/2705925758 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f02080a27b0 0x7f02080a4c70 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.411+0000 7f0228400640 1 --2- 172.21.15.12:0/2705925758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02201021c0 0x7f02201a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.411+0000 7f0228400640 1 --2- 172.21.15.12:0/2705925758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0220101800 0x7f022019a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.411+0000 7f0228400640 1 --2- 172.21.15.12:0/2705925758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0220100610 0x7f0220199be0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:34.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.411+0000 7f0228400640 1 -- 172.21.15.12:0/2705925758 >> 172.21.15.12:0/2705925758 conn(0x7f02200fbb80 msgr2=0x7f0220190880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:34.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.411+0000 7f0228400640 1 -- 172.21.15.12:0/2705925758 shutdown_connections 2024-03-23T23:08:34.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:34.411+0000 7f0228400640 1 -- 172.21.15.12:0/2705925758 wait complete. 2024-03-23T23:08:35.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:34 smithi173 ceph-mon[36116]: pgmap v593: 97 pgs: 97 active+clean; 4.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:08:35.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:34 smithi203 ceph-mon[36693]: pgmap v593: 97 pgs: 97 active+clean; 4.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:08:35.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:34 smithi012 ceph-mon[31695]: pgmap v593: 97 pgs: 97 active+clean; 4.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:08:37.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:36 smithi173 ceph-mon[36116]: pgmap v594: 97 pgs: 97 active+clean; 4.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:08:37.145 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:08:37.145 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:08:37.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:36 smithi203 ceph-mon[36693]: pgmap v594: 97 pgs: 97 active+clean; 4.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:08:37.239 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:36 smithi012 ceph-mon[31695]: pgmap v594: 97 pgs: 97 active+clean; 4.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:08:37.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.385+0000 7f31b1a00640 1 -- 172.21.15.12:0/3525458489 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31ac107cc0 msgr2=0x7f31ac1080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:37.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.385+0000 7f31b1a00640 1 --2- 172.21.15.12:0/3525458489 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31ac107cc0 0x7f31ac1080e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f319c009900 tx=0x7f319c02bc40 comp rx=0 tx=0).stop 2024-03-23T23:08:37.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.385+0000 7f31b1a00640 1 -- 172.21.15.12:0/3525458489 shutdown_connections 2024-03-23T23:08:37.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.385+0000 7f31b1a00640 1 --2- 172.21.15.12:0/3525458489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31ac109870 0x7f31ac110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.386+0000 7f31b1a00640 1 --2- 172.21.15.12:0/3525458489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31ac108eb0 0x7f31ac109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.386+0000 7f31b1a00640 1 --2- 172.21.15.12:0/3525458489 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31ac107cc0 0x7f31ac1080e0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.386+0000 7f31b1a00640 1 -- 172.21.15.12:0/3525458489 >> 172.21.15.12:0/3525458489 conn(0x7f31ac076c60 msgr2=0x7f31ac0790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:37.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.386+0000 7f31b1a00640 1 -- 172.21.15.12:0/3525458489 shutdown_connections 2024-03-23T23:08:37.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.386+0000 7f31b1a00640 1 -- 172.21.15.12:0/3525458489 wait complete. 2024-03-23T23:08:37.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.387+0000 7f31b1a00640 1 Processor -- start 2024-03-23T23:08:37.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.387+0000 7f31b1a00640 1 -- start start 2024-03-23T23:08:37.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.387+0000 7f31b1a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31ac107cc0 0x7f31ac19dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:37.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.387+0000 7f31b1a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31ac108eb0 0x7f31ac19e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:37.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.387+0000 7f31b1a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31ac109870 0x7f31ac1a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:37.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.387+0000 7f31b1a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac113960 con 0x7f31ac109870 2024-03-23T23:08:37.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.387+0000 7f31b1a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac1137e0 con 0x7f31ac108eb0 2024-03-23T23:08:37.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.387+0000 7f31b1a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac113ae0 con 0x7f31ac107cc0 2024-03-23T23:08:37.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.387+0000 7f31aaa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31ac107cc0 0x7f31ac19dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:37.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.387+0000 7f31aa000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31ac108eb0 0x7f31ac19e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:37.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.387+0000 7f31aaa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31ac107cc0 0x7f31ac19dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:42196/0 (socket says 172.21.15.12:42196) 2024-03-23T23:08:37.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.387+0000 7f31aaa00640 1 -- 172.21.15.12:0/2479058273 learned_addr learned my addr 172.21.15.12:0/2479058273 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:37.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.387+0000 7f31aa000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31ac108eb0 0x7f31ac19e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57256/0 (socket says 172.21.15.12:57256) 2024-03-23T23:08:37.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.387+0000 7f31ab400640 1 --2- 172.21.15.12:0/2479058273 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31ac109870 0x7f31ac1a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:37.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.388+0000 7f31aaa00640 1 -- 172.21.15.12:0/2479058273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31ac108eb0 msgr2=0x7f31ac19e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:37.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.388+0000 7f31aaa00640 1 --2- 172.21.15.12:0/2479058273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31ac108eb0 0x7f31ac19e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.388+0000 7f31aaa00640 1 -- 172.21.15.12:0/2479058273 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31ac109870 msgr2=0x7f31ac1a5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:37.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.388+0000 7f31aaa00640 1 --2- 172.21.15.12:0/2479058273 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31ac109870 0x7f31ac1a5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.388+0000 7f31aaa00640 1 -- 172.21.15.12:0/2479058273 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ac1a5c00 con 0x7f31ac107cc0 2024-03-23T23:08:37.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.388+0000 7f31ab400640 1 --2- 172.21.15.12:0/2479058273 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31ac109870 0x7f31ac1a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:08:37.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.388+0000 7f31aaa00640 1 --2- 172.21.15.12:0/2479058273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31ac107cc0 0x7f31ac19dfe0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f319c02efd0 tx=0x7f319c005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:37.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.388+0000 7f31a3400640 1 -- 172.21.15.12:0/2479058273 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c041070 con 0x7f31ac107cc0 2024-03-23T23:08:37.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.388+0000 7f31a3400640 1 -- 172.21.15.12:0/2479058273 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f319c0044d0 con 0x7f31ac107cc0 2024-03-23T23:08:37.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.388+0000 7f31a3400640 1 -- 172.21.15.12:0/2479058273 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c034440 con 0x7f31ac107cc0 2024-03-23T23:08:37.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.388+0000 7f31b1a00640 1 -- 172.21.15.12:0/2479058273 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31ac1a5e90 con 0x7f31ac107cc0 2024-03-23T23:08:37.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.388+0000 7f31b1a00640 1 -- 172.21.15.12:0/2479058273 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31ac1a6240 con 0x7f31ac107cc0 2024-03-23T23:08:37.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.392+0000 7f31a3400640 1 -- 172.21.15.12:0/2479058273 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f319c044070 con 0x7f31ac107cc0 2024-03-23T23:08:37.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.392+0000 7f31b47b8740 1 Processor -- start 2024-03-23T23:08:37.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.392+0000 7f31b47b8740 1 -- start start 2024-03-23T23:08:37.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.392+0000 7f31b47b8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55817e267e80 0x55817e2682a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:37.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.392+0000 7f31b47b8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e268870 0x55817e26d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:37.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.392+0000 7f31b47b8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55817e26d9e0 0x55817e26fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:37.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.392+0000 7f31b47b8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55817e1cd250 con 0x55817e267e80 2024-03-23T23:08:37.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.392+0000 7f31b47b8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55817e175a30 con 0x55817e268870 2024-03-23T23:08:37.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.392+0000 7f31b47b8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55817e1c92f0 con 0x55817e26d9e0 2024-03-23T23:08:37.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.393+0000 7f31aaa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e268870 0x55817e26d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:37.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.393+0000 7f31ab400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55817e267e80 0x55817e2682a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:37.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.393+0000 7f31aaa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e268870 0x55817e26d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57268/0 (socket says 172.21.15.12:57268) 2024-03-23T23:08:37.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.393+0000 7f31aaa00640 1 -- 172.21.15.12:0/439379904 learned_addr learned my addr 172.21.15.12:0/439379904 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:37.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.393+0000 7f31aa000640 1 --2- 172.21.15.12:0/439379904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55817e26d9e0 0x55817e26fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:37.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.393+0000 7f31aaa00640 1 -- 172.21.15.12:0/439379904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55817e26d9e0 msgr2=0x55817e26fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:37.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.393+0000 7f31aaa00640 1 --2- 172.21.15.12:0/439379904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55817e26d9e0 0x55817e26fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.393+0000 7f31aaa00640 1 -- 172.21.15.12:0/439379904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55817e267e80 msgr2=0x55817e2682a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:37.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.393+0000 7f31aaa00640 1 --2- 172.21.15.12:0/439379904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55817e267e80 0x55817e2682a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.393+0000 7f31aaa00640 1 -- 172.21.15.12:0/439379904 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55817e160690 con 0x55817e268870 2024-03-23T23:08:37.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.393+0000 7f31a3400640 1 --2- 172.21.15.12:0/2479058273 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f31900a2880 0x7f31900a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:37.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.393+0000 7f31aaa00640 1 --2- 172.21.15.12:0/439379904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e268870 0x55817e26d370 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f319c0f65c0 tx=0x7f319c0f66b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcf9aee424aeeeb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:37.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.394+0000 7f31a3400640 1 -- 172.21.15.12:0/2479058273 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f319c0f0650 con 0x7f31ac107cc0 2024-03-23T23:08:37.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.394+0000 7f3197e00640 1 -- 172.21.15.12:0/439379904 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c0b9240 con 0x55817e268870 2024-03-23T23:08:37.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.394+0000 7f31ab400640 1 --2- 172.21.15.12:0/2479058273 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f31900a2880 0x7f31900a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:37.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.394+0000 7f3197e00640 1 -- 172.21.15.12:0/439379904 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f319c0bd6a0 con 0x55817e268870 2024-03-23T23:08:37.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.394+0000 7f3197e00640 1 -- 172.21.15.12:0/439379904 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c0bc900 con 0x55817e268870 2024-03-23T23:08:37.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.394+0000 7f31b47b8740 1 -- 172.21.15.12:0/439379904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e268870 msgr2=0x55817e26d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.394+0000 7f31b47b8740 1 --2- 172.21.15.12:0/439379904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e268870 0x55817e26d370 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f319c0f65c0 tx=0x7f319c0f66b0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.395+0000 7f31ab400640 1 --2- 172.21.15.12:0/2479058273 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f31900a2880 0x7f31900a4d40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x55817e2690d0 tx=0x7f31a40098a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.395+0000 7f31b47b8740 1 -- 172.21.15.12:0/439379904 shutdown_connections 2024-03-23T23:08:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.395+0000 7f31b47b8740 1 --2- 172.21.15.12:0/439379904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55817e26d9e0 0x55817e26fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.395+0000 7f31b47b8740 1 --2- 172.21.15.12:0/439379904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e268870 0x55817e26d370 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.395+0000 7f31b47b8740 1 --2- 172.21.15.12:0/439379904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55817e267e80 0x55817e2682a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.395+0000 7f31b47b8740 1 -- 172.21.15.12:0/439379904 >> 172.21.15.12:0/439379904 conn(0x55817e230200 msgr2=0x55817e2637f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.395+0000 7f31b47b8740 1 -- 172.21.15.12:0/439379904 shutdown_connections 2024-03-23T23:08:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.395+0000 7f31b47b8740 1 -- 172.21.15.12:0/439379904 wait complete. 2024-03-23T23:08:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.395+0000 7f31b47b8740 1 -- start start 2024-03-23T23:08:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.395+0000 7f31b47b8740 1 Processor -- start 2024-03-23T23:08:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.396+0000 7f31b47b8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55817e267e80 0x55817e271bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:37.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.396+0000 7f31b47b8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55817e268870 0x55817e272110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:37.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.396+0000 7f31b47b8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e26d9e0 0x55817e279150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:37.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.396+0000 7f31b47b8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55817e270750 con 0x55817e268870 2024-03-23T23:08:37.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.396+0000 7f31b47b8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55817e188e80 con 0x55817e26d9e0 2024-03-23T23:08:37.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.396+0000 7f31b47b8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55817e2708d0 con 0x55817e267e80 2024-03-23T23:08:37.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.396+0000 7f31b47b8740 10 client.0 _ll_register_callbacks cb 0x55817e19b130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:37.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.396+0000 7f31aaa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55817e267e80 0x55817e271bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:37.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.396+0000 7f31aa000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55817e268870 0x55817e272110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:37.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.396+0000 7f31ab400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e26d9e0 0x55817e279150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:37.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.396+0000 7f31aaa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55817e267e80 0x55817e271bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:42208/0 (socket says 172.21.15.12:42208) 2024-03-23T23:08:37.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.396+0000 7f31aaa00640 1 -- 172.21.15.12:0/2517575511 learned_addr learned my addr 172.21.15.12:0/2517575511 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:37.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.396+0000 7f31ab400640 1 -- 172.21.15.12:0/2517575511 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55817e267e80 msgr2=0x55817e271bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:37.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.396+0000 7f31ab400640 1 --2- 172.21.15.12:0/2517575511 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55817e267e80 0x55817e271bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.396+0000 7f31ab400640 1 -- 172.21.15.12:0/2517575511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55817e268870 msgr2=0x55817e272110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:37.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.397+0000 7f31ab400640 1 --2- 172.21.15.12:0/2517575511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55817e268870 0x55817e272110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.397+0000 7f31ab400640 1 -- 172.21.15.12:0/2517575511 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55817dfab7d0 con 0x55817e26d9e0 2024-03-23T23:08:37.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.397+0000 7f31aaa00640 1 --2- 172.21.15.12:0/2517575511 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55817e267e80 0x55817e271bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:37.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.397+0000 7f31ab400640 1 --2- 172.21.15.12:0/2517575511 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e26d9e0 0x55817e279150 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f31a4009ed0 tx=0x7f31a4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7b42214278510b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:37.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.397+0000 7f3194c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:08:37.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.397+0000 7f3194c00640 1 -- 172.21.15.12:0/2517575511 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a401e6d0 con 0x55817e26d9e0 2024-03-23T23:08:37.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.397+0000 7f31b47b8740 1 -- 172.21.15.12:0/2517575511 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f319c0bd6a0 con 0x55817e26d9e0 2024-03-23T23:08:37.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.397+0000 7f3194c00640 1 -- 172.21.15.12:0/2517575511 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f31a401e870 con 0x55817e26d9e0 2024-03-23T23:08:37.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.397+0000 7f3194c00640 1 -- 172.21.15.12:0/2517575511 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f31a401ea70 con 0x55817e26d9e0 2024-03-23T23:08:37.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.397+0000 7f31b47b8740 10 client.27182 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:37.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.397+0000 7f31b47b8740 1 -- 172.21.15.12:0/2517575511 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55817dee13f0 con 0x55817e26d9e0 2024-03-23T23:08:37.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.398+0000 7f3194c00640 1 -- 172.21.15.12:0/2517575511 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f319c0bd6a0 con 0x55817e26d9e0 2024-03-23T23:08:37.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.398+0000 7f3194c00640 1 -- 172.21.15.12:0/2517575511 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f31a401d410 con 0x55817e26d9e0 2024-03-23T23:08:37.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.398+0000 7f31b47b8740 10 client.27182 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:37.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.398+0000 7f31b47b8740 10 client.27182 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:08:37.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.398+0000 7f31b47b8740 20 client.27182 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:37.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.398+0000 7f31b47b8740 1 --2- 172.21.15.12:0/2517575511 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55817e27d9d0 0x55817e27fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:37.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.398+0000 7f31b47b8740 4 client.27182 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:08:37.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.398+0000 7f31b47b8740 1 -- 172.21.15.12:0/2517575511 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55817e1da790 con 0x55817e27d9d0 2024-03-23T23:08:37.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.398+0000 7f31aaa00640 1 --2- 172.21.15.12:0/2517575511 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55817e27d9d0 0x55817e27fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:37.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.399+0000 7f31aaa00640 1 --2- 172.21.15.12:0/2517575511 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55817e27d9d0 0x55817e27fe00 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5def10f67d008d9 server_cookie=11c6fd4bf7236fdb in_seq=0 out_seq=0 2024-03-23T23:08:37.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.399+0000 7f3194c00640 10 client.27182 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:37.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.400+0000 7f3194c00640 1 -- 172.21.15.12:0/2517575511 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55817e1da790 con 0x55817e27d9d0 2024-03-23T23:08:37.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.400+0000 7f3194c00640 10 client.27182 handle_command_reply: tid=0 2024-03-23T23:08:37.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.400+0000 7f31b47b8740 1 client.27182 shutdown 2024-03-23T23:08:37.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.400+0000 7f31b47b8740 1 -- 172.21.15.12:0/2517575511 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e26d9e0 msgr2=0x55817e279150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:37.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.400+0000 7f31b47b8740 1 --2- 172.21.15.12:0/2517575511 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e26d9e0 0x55817e279150 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f31a4009ed0 tx=0x7f31a4004a40 comp rx=0 tx=0).stop 2024-03-23T23:08:37.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.400+0000 7f31b47b8740 1 -- 172.21.15.12:0/2517575511 shutdown_connections 2024-03-23T23:08:37.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.400+0000 7f31b47b8740 1 --2- 172.21.15.12:0/2517575511 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55817e27d9d0 0x55817e27fe00 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.400+0000 7f31b47b8740 1 --2- 172.21.15.12:0/2517575511 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e26d9e0 0x55817e279150 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.400+0000 7f31b47b8740 1 --2- 172.21.15.12:0/2517575511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55817e268870 0x55817e272110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.400+0000 7f31b47b8740 1 --2- 172.21.15.12:0/2517575511 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55817e267e80 0x55817e271bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.400+0000 7f31b47b8740 1 -- 172.21.15.12:0/2517575511 >> 172.21.15.12:0/2517575511 conn(0x55817e230200 msgr2=0x55817e262a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:37.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.400+0000 7f3194c00640 0 client.27182 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:37.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.400+0000 7f31b47b8740 1 -- 172.21.15.12:0/2517575511 shutdown_connections 2024-03-23T23:08:37.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.400+0000 7f31b47b8740 1 -- 172.21.15.12:0/2517575511 wait complete. 2024-03-23T23:08:37.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.401+0000 7f31b47b8740 20 client.27182 trim_cache size 0 max 16384 2024-03-23T23:08:37.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.418+0000 7f31b47b8740 1 Processor -- start 2024-03-23T23:08:37.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.418+0000 7f31b47b8740 1 -- start start 2024-03-23T23:08:37.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.419+0000 7f31b47b8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e267e80 0x55817e262200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:37.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.419+0000 7f31b47b8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55817e268870 0x55817e279110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:37.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.419+0000 7f31b47b8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55817e26d9e0 0x55817e262740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:37.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.419+0000 7f31b47b8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac1b1550 con 0x55817e26d9e0 2024-03-23T23:08:37.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.419+0000 7f31b47b8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac1af050 con 0x55817e267e80 2024-03-23T23:08:37.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.419+0000 7f31b47b8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac1ad0f0 con 0x55817e268870 2024-03-23T23:08:37.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.419+0000 7f31aa000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55817e268870 0x55817e279110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:37.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.419+0000 7f31ab400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55817e26d9e0 0x55817e262740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:37.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.419+0000 7f31aaa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e267e80 0x55817e262200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:37.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.419+0000 7f31aa000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55817e268870 0x55817e279110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:42224/0 (socket says 172.21.15.12:42224) 2024-03-23T23:08:37.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.419+0000 7f31aa000640 1 -- 172.21.15.12:0/106934528 learned_addr learned my addr 172.21.15.12:0/106934528 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:37.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.419+0000 7f31aaa00640 1 -- 172.21.15.12:0/106934528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55817e268870 msgr2=0x55817e279110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:37.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.419+0000 7f31aaa00640 1 --2- 172.21.15.12:0/106934528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55817e268870 0x55817e279110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.419+0000 7f31aaa00640 1 -- 172.21.15.12:0/106934528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55817e26d9e0 msgr2=0x55817e262740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:37.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.419+0000 7f31aaa00640 1 --2- 172.21.15.12:0/106934528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55817e26d9e0 0x55817e262740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.419+0000 7f31aaa00640 1 -- 172.21.15.12:0/106934528 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55817dea10f0 con 0x55817e267e80 2024-03-23T23:08:37.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.419+0000 7f31aa000640 1 --2- 172.21.15.12:0/106934528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55817e268870 0x55817e279110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:37.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.420+0000 7f31aaa00640 1 --2- 172.21.15.12:0/106934528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e267e80 0x55817e262200 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f319c0f6bb0 tx=0x7f319c0f6ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c705c1a579f2ef server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:37.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.420+0000 7f3197400640 1 -- 172.21.15.12:0/106934528 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c04b040 con 0x55817e267e80 2024-03-23T23:08:37.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.420+0000 7f3197400640 1 -- 172.21.15.12:0/106934528 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f319c0bd6a0 con 0x55817e267e80 2024-03-23T23:08:37.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.420+0000 7f3197400640 1 -- 172.21.15.12:0/106934528 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c0bcae0 con 0x55817e267e80 2024-03-23T23:08:37.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.421+0000 7f31b47b8740 1 -- 172.21.15.12:0/106934528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e267e80 msgr2=0x55817e262200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:37.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.421+0000 7f31b47b8740 1 --2- 172.21.15.12:0/106934528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e267e80 0x55817e262200 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f319c0f6bb0 tx=0x7f319c0f6ee0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.421+0000 7f31b47b8740 1 -- 172.21.15.12:0/106934528 shutdown_connections 2024-03-23T23:08:37.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.421+0000 7f31b47b8740 1 --2- 172.21.15.12:0/106934528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55817e26d9e0 0x55817e262740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.421+0000 7f31b47b8740 1 --2- 172.21.15.12:0/106934528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55817e268870 0x55817e279110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.421+0000 7f31b47b8740 1 --2- 172.21.15.12:0/106934528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e267e80 0x55817e262200 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.421+0000 7f31b47b8740 1 -- 172.21.15.12:0/106934528 >> 172.21.15.12:0/106934528 conn(0x55817e230200 msgr2=0x55817e263330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:37.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.421+0000 7f31b47b8740 1 -- 172.21.15.12:0/106934528 shutdown_connections 2024-03-23T23:08:37.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.421+0000 7f31b47b8740 1 -- 172.21.15.12:0/106934528 wait complete. 2024-03-23T23:08:37.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.421+0000 7f31b47b8740 1 -- start start 2024-03-23T23:08:37.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.421+0000 7f31b47b8740 1 Processor -- start 2024-03-23T23:08:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.422+0000 7f31b47b8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e267e80 0x55817e276af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.422+0000 7f31b47b8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55817e268870 0x55817e277030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.422+0000 7f31b47b8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55817e26d9e0 0x55817e277f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.422+0000 7f31b47b8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac1ad6f0 con 0x55817e268870 2024-03-23T23:08:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.422+0000 7f31b47b8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac1ae9f0 con 0x55817e267e80 2024-03-23T23:08:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.422+0000 7f31b47b8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac1add30 con 0x55817e26d9e0 2024-03-23T23:08:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.422+0000 7f31b47b8740 10 client.0 _ll_register_callbacks cb 0x7f31a4010620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.422+0000 7f31aaa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e267e80 0x55817e276af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.422+0000 7f31aaa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e267e80 0x55817e276af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57292/0 (socket says 172.21.15.12:57292) 2024-03-23T23:08:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.422+0000 7f31aaa00640 1 -- 172.21.15.12:0/1942912839 learned_addr learned my addr 172.21.15.12:0/1942912839 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.422+0000 7f31ab400640 1 --2- 172.21.15.12:0/1942912839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55817e26d9e0 0x55817e277f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.422+0000 7f31aa000640 1 --2- 172.21.15.12:0/1942912839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55817e268870 0x55817e277030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.422+0000 7f31aaa00640 1 -- 172.21.15.12:0/1942912839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55817e26d9e0 msgr2=0x55817e277f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.422+0000 7f31aaa00640 1 --2- 172.21.15.12:0/1942912839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55817e26d9e0 0x55817e277f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.422+0000 7f31aaa00640 1 -- 172.21.15.12:0/1942912839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55817e268870 msgr2=0x55817e277030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.422+0000 7f31aaa00640 1 --2- 172.21.15.12:0/1942912839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55817e268870 0x55817e277030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.423+0000 7f31aaa00640 1 -- 172.21.15.12:0/1942912839 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55817e278570 con 0x55817e267e80 2024-03-23T23:08:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.423+0000 7f31aa000640 1 --2- 172.21.15.12:0/1942912839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55817e268870 0x55817e277030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:08:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.423+0000 7f31aaa00640 1 --2- 172.21.15.12:0/1942912839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e267e80 0x55817e276af0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f319c02c180 tx=0x7f319c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=418b62e3c5e64a17 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.423+0000 7f3194c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:08:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.423+0000 7f3194c00640 1 -- 172.21.15.12:0/1942912839 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c04b070 con 0x55817e267e80 2024-03-23T23:08:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.423+0000 7f3194c00640 1 -- 172.21.15.12:0/1942912839 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f319c0ee180 con 0x55817e267e80 2024-03-23T23:08:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.423+0000 7f31b47b8740 1 -- 172.21.15.12:0/1942912839 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f319c0bd6a0 con 0x55817e267e80 2024-03-23T23:08:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.423+0000 7f3194c00640 1 -- 172.21.15.12:0/1942912839 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f319c0bcd70 con 0x55817e267e80 2024-03-23T23:08:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.423+0000 7f31b47b8740 10 client.27188 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.423+0000 7f31b47b8740 1 -- 172.21.15.12:0/1942912839 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55817e2d93d0 con 0x55817e267e80 2024-03-23T23:08:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.423+0000 7f3194c00640 1 -- 172.21.15.12:0/1942912839 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f319c0bd6a0 con 0x55817e267e80 2024-03-23T23:08:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.424+0000 7f3194c00640 1 -- 172.21.15.12:0/1942912839 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f319c0c52b0 con 0x55817e267e80 2024-03-23T23:08:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.424+0000 7f31b47b8740 10 client.27188 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.424+0000 7f31b47b8740 10 client.27188 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:08:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.424+0000 7f31b47b8740 20 client.27188 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.424+0000 7f31b47b8740 1 --2- 172.21.15.12:0/1942912839 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55817e2d9990 0x55817e2dbd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.424+0000 7f31b47b8740 4 client.27188 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:08:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.424+0000 7f31b47b8740 1 -- 172.21.15.12:0/1942912839 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55817e1d4ae0 con 0x55817e2d9990 2024-03-23T23:08:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.424+0000 7f31aa000640 1 --2- 172.21.15.12:0/1942912839 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55817e2d9990 0x55817e2dbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.425+0000 7f31aa000640 1 --2- 172.21.15.12:0/1942912839 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55817e2d9990 0x55817e2dbd80 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=631b06f6fa62164d server_cookie=92fa5c4d199c7790 in_seq=0 out_seq=0 2024-03-23T23:08:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.425+0000 7f3194c00640 10 client.27188 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.425+0000 7f3194c00640 1 -- 172.21.15.12:0/1942912839 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55817e1d4ae0 con 0x55817e2d9990 2024-03-23T23:08:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.425+0000 7f3194c00640 10 client.27188 handle_command_reply: tid=0 2024-03-23T23:08:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.425+0000 7f31b47b8740 1 client.27188 shutdown 2024-03-23T23:08:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.425+0000 7f31b47b8740 1 -- 172.21.15.12:0/1942912839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e267e80 msgr2=0x55817e276af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.425+0000 7f31b47b8740 1 --2- 172.21.15.12:0/1942912839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e267e80 0x55817e276af0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f319c02c180 tx=0x7f319c009260 comp rx=0 tx=0).stop 2024-03-23T23:08:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.425+0000 7f31b47b8740 1 -- 172.21.15.12:0/1942912839 shutdown_connections 2024-03-23T23:08:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.425+0000 7f31b47b8740 1 --2- 172.21.15.12:0/1942912839 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55817e2d9990 0x55817e2dbd80 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.425+0000 7f31b47b8740 1 --2- 172.21.15.12:0/1942912839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55817e26d9e0 0x55817e277f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.425+0000 7f31b47b8740 1 --2- 172.21.15.12:0/1942912839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55817e268870 0x55817e277030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.425+0000 7f31b47b8740 1 --2- 172.21.15.12:0/1942912839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55817e267e80 0x55817e276af0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.425+0000 7f31b47b8740 1 -- 172.21.15.12:0/1942912839 >> 172.21.15.12:0/1942912839 conn(0x55817e230200 msgr2=0x55817e269270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.425+0000 7f3194c00640 0 client.27188 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:37.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.425+0000 7f31b47b8740 1 -- 172.21.15.12:0/1942912839 shutdown_connections 2024-03-23T23:08:37.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.425+0000 7f31b47b8740 1 -- 172.21.15.12:0/1942912839 wait complete. 2024-03-23T23:08:37.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.425+0000 7f31b47b8740 20 client.27188 trim_cache size 0 max 16384 2024-03-23T23:08:37.430 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:08:37.430 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-03-23T23:08:37.430 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-03-23T23:08:37.430 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:08:37.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.426+0000 7f31b1a00640 1 -- 172.21.15.12:0/2479058273 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f31900a2880 msgr2=0x7f31900a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:37.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.426+0000 7f31b1a00640 1 --2- 172.21.15.12:0/2479058273 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f31900a2880 0x7f31900a4d40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x55817e2690d0 tx=0x7f31a40098a0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.426+0000 7f31b1a00640 1 -- 172.21.15.12:0/2479058273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31ac107cc0 msgr2=0x7f31ac19dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:37.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.426+0000 7f31b1a00640 1 --2- 172.21.15.12:0/2479058273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31ac107cc0 0x7f31ac19dfe0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f319c02efd0 tx=0x7f319c005d50 comp rx=0 tx=0).stop 2024-03-23T23:08:37.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.426+0000 7f31b1a00640 1 -- 172.21.15.12:0/2479058273 shutdown_connections 2024-03-23T23:08:37.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.426+0000 7f31b1a00640 1 --2- 172.21.15.12:0/2479058273 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f31900a2880 0x7f31900a4d40 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.426+0000 7f31b1a00640 1 --2- 172.21.15.12:0/2479058273 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31ac109870 0x7f31ac1a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.426+0000 7f31b1a00640 1 --2- 172.21.15.12:0/2479058273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31ac108eb0 0x7f31ac19e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.426+0000 7f31b1a00640 1 --2- 172.21.15.12:0/2479058273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31ac107cc0 0x7f31ac19dfe0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.426+0000 7f31b1a00640 1 -- 172.21.15.12:0/2479058273 >> 172.21.15.12:0/2479058273 conn(0x7f31ac076c60 msgr2=0x7f31ac10c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:37.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.426+0000 7f31b1a00640 1 -- 172.21.15.12:0/2479058273 shutdown_connections 2024-03-23T23:08:37.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.427+0000 7f31b1a00640 1 -- 172.21.15.12:0/2479058273 wait complete. 2024-03-23T23:08:37.444 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T23:08:37.444 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:08:37.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.666+0000 7f59fac00640 1 -- 172.21.15.12:0/304305589 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59f4104ad0 msgr2=0x7f59f4104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:37.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.666+0000 7f59fac00640 1 --2- 172.21.15.12:0/304305589 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59f4104ad0 0x7f59f4104f50 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f59e0009900 tx=0x7f59e002bc30 comp rx=0 tx=0).stop 2024-03-23T23:08:37.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.667+0000 7f59fac00640 1 -- 172.21.15.12:0/304305589 shutdown_connections 2024-03-23T23:08:37.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.667+0000 7f59fac00640 1 --2- 172.21.15.12:0/304305589 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59f4105490 0x7f59f410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.667+0000 7f59fac00640 1 --2- 172.21.15.12:0/304305589 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59f4104ad0 0x7f59f4104f50 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.667+0000 7f59fac00640 1 --2- 172.21.15.12:0/304305589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59f41038e0 0x7f59f4103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.667+0000 7f59fac00640 1 -- 172.21.15.12:0/304305589 >> 172.21.15.12:0/304305589 conn(0x7f59f40fee50 msgr2=0x7f59f4101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:37.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.667+0000 7f59fac00640 1 -- 172.21.15.12:0/304305589 shutdown_connections 2024-03-23T23:08:37.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.667+0000 7f59fac00640 1 -- 172.21.15.12:0/304305589 wait complete. 2024-03-23T23:08:37.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.667+0000 7f59fac00640 1 Processor -- start 2024-03-23T23:08:37.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.668+0000 7f59fac00640 1 -- start start 2024-03-23T23:08:37.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.668+0000 7f59fac00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59f41038e0 0x7f59f4199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:37.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.668+0000 7f59fac00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59f4104ad0 0x7f59f419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:37.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.668+0000 7f59fac00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59f4105490 0x7f59f41a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:37.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.668+0000 7f59fac00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f410f590 con 0x7f59f4104ad0 2024-03-23T23:08:37.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.668+0000 7f59fac00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f410f410 con 0x7f59f41038e0 2024-03-23T23:08:37.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.668+0000 7f59fac00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f410f710 con 0x7f59f4105490 2024-03-23T23:08:37.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.668+0000 7f59f8c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59f4105490 0x7f59f41a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:37.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.668+0000 7f59f3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59f41038e0 0x7f59f4199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:37.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.668+0000 7f59f3400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59f4104ad0 0x7f59f419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:37.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.668+0000 7f59f8c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59f4105490 0x7f59f41a1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:42268/0 (socket says 172.21.15.12:42268) 2024-03-23T23:08:37.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.668+0000 7f59f8c00640 1 -- 172.21.15.12:0/398804588 learned_addr learned my addr 172.21.15.12:0/398804588 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:37.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.668+0000 7f59f3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59f41038e0 0x7f59f4199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57308/0 (socket says 172.21.15.12:57308) 2024-03-23T23:08:37.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.669+0000 7f59f8c00640 1 -- 172.21.15.12:0/398804588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59f41038e0 msgr2=0x7f59f4199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:37.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.669+0000 7f59f8c00640 1 --2- 172.21.15.12:0/398804588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59f41038e0 0x7f59f4199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.669+0000 7f59f8c00640 1 -- 172.21.15.12:0/398804588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59f4104ad0 msgr2=0x7f59f419a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:37.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.669+0000 7f59f8c00640 1 --2- 172.21.15.12:0/398804588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59f4104ad0 0x7f59f419a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.669+0000 7f59f8c00640 1 -- 172.21.15.12:0/398804588 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f41a1830 con 0x7f59f4105490 2024-03-23T23:08:37.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.669+0000 7f59f8c00640 1 --2- 172.21.15.12:0/398804588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59f4105490 0x7f59f41a1190 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f59ec00eb10 tx=0x7f59ec00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:37.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.669+0000 7f59f0c00640 1 -- 172.21.15.12:0/398804588 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ec004b20 con 0x7f59f4105490 2024-03-23T23:08:37.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.669+0000 7f59fac00640 1 -- 172.21.15.12:0/398804588 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f41a1b20 con 0x7f59f4105490 2024-03-23T23:08:37.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.669+0000 7f59f0c00640 1 -- 172.21.15.12:0/398804588 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59ec004cc0 con 0x7f59f4105490 2024-03-23T23:08:37.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.669+0000 7f59f0c00640 1 -- 172.21.15.12:0/398804588 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ec010760 con 0x7f59f4105490 2024-03-23T23:08:37.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.670+0000 7f59fac00640 1 -- 172.21.15.12:0/398804588 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f41a2020 con 0x7f59f4105490 2024-03-23T23:08:37.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.671+0000 7f59fac00640 1 -- 172.21.15.12:0/398804588 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f410ac80 con 0x7f59f4105490 2024-03-23T23:08:37.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.673+0000 7f59f0c00640 1 -- 172.21.15.12:0/398804588 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f59ec005020 con 0x7f59f4105490 2024-03-23T23:08:37.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.674+0000 7f59f0c00640 1 --2- 172.21.15.12:0/398804588 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f59d80a2800 0x7f59d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:37.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.674+0000 7f59f0c00640 1 -- 172.21.15.12:0/398804588 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f59ec0cf050 con 0x7f59f4105490 2024-03-23T23:08:37.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.674+0000 7f59f3e00640 1 --2- 172.21.15.12:0/398804588 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f59d80a2800 0x7f59d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:37.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.675+0000 7f59f3e00640 1 --2- 172.21.15.12:0/398804588 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f59d80a2800 0x7f59d80a4cc0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f59f419b0f0 tx=0x7f59e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:37.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.682+0000 7f59f0c00640 1 -- 172.21.15.12:0/398804588 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f59ec09c100 con 0x7f59f4105490 2024-03-23T23:08:37.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.927+0000 7f59fac00640 1 -- 172.21.15.12:0/398804588 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59f402d070 con 0x7f59f4105490 2024-03-23T23:08:37.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.928+0000 7f59f0c00640 1 -- 172.21.15.12:0/398804588 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f59ec09ba00 con 0x7f59f4105490 2024-03-23T23:08:37.928 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:08:37.929 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:08:37.929 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:08:37.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.931+0000 7f59fac00640 1 -- 172.21.15.12:0/398804588 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f59d80a2800 msgr2=0x7f59d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:37.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.931+0000 7f59fac00640 1 --2- 172.21.15.12:0/398804588 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f59d80a2800 0x7f59d80a4cc0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f59f419b0f0 tx=0x7f59e4005f90 comp rx=0 tx=0).stop 2024-03-23T23:08:37.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.931+0000 7f59fac00640 1 -- 172.21.15.12:0/398804588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59f4105490 msgr2=0x7f59f41a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:37.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.931+0000 7f59fac00640 1 --2- 172.21.15.12:0/398804588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59f4105490 0x7f59f41a1190 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f59ec00eb10 tx=0x7f59ec00c2d0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.931+0000 7f59fac00640 1 -- 172.21.15.12:0/398804588 shutdown_connections 2024-03-23T23:08:37.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.931+0000 7f59fac00640 1 --2- 172.21.15.12:0/398804588 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f59d80a2800 0x7f59d80a4cc0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.931+0000 7f59fac00640 1 --2- 172.21.15.12:0/398804588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59f4105490 0x7f59f41a1190 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.931+0000 7f59fac00640 1 --2- 172.21.15.12:0/398804588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59f4104ad0 0x7f59f419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.931+0000 7f59fac00640 1 --2- 172.21.15.12:0/398804588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59f41038e0 0x7f59f4199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:37.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.931+0000 7f59fac00640 1 -- 172.21.15.12:0/398804588 >> 172.21.15.12:0/398804588 conn(0x7f59f40fee50 msgr2=0x7f59f4107e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:37.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.931+0000 7f59fac00640 1 -- 172.21.15.12:0/398804588 shutdown_connections 2024-03-23T23:08:37.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:37.931+0000 7f59fac00640 1 -- 172.21.15.12:0/398804588 wait complete. 2024-03-23T23:08:37.952 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:08:37.952 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-23T23:08:38.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.178+0000 7f7ca0a00640 1 -- 172.21.15.12:0/3266248596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c9c104930 msgr2=0x7f7c9c106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.178+0000 7f7ca0a00640 1 --2- 172.21.15.12:0/3266248596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c9c104930 0x7f7c9c106d20 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7c88009900 tx=0x7f7c8802bc50 comp rx=0 tx=0).stop 2024-03-23T23:08:38.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.179+0000 7f7ca0a00640 1 -- 172.21.15.12:0/3266248596 shutdown_connections 2024-03-23T23:08:38.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.179+0000 7f7ca0a00640 1 --2- 172.21.15.12:0/3266248596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c9c107260 0x7f7c9c1096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.179+0000 7f7ca0a00640 1 --2- 172.21.15.12:0/3266248596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c9c104930 0x7f7c9c106d20 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.179+0000 7f7ca0a00640 1 --2- 172.21.15.12:0/3266248596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c9c102000 0x7f7c9c1043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.179+0000 7f7ca0a00640 1 -- 172.21.15.12:0/3266248596 >> 172.21.15.12:0/3266248596 conn(0x7f7c9c0fba10 msgr2=0x7f7c9c0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:38.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.179+0000 7f7ca0a00640 1 -- 172.21.15.12:0/3266248596 shutdown_connections 2024-03-23T23:08:38.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.179+0000 7f7ca0a00640 1 -- 172.21.15.12:0/3266248596 wait complete. 2024-03-23T23:08:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.180+0000 7f7ca0a00640 1 Processor -- start 2024-03-23T23:08:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.180+0000 7f7ca0a00640 1 -- start start 2024-03-23T23:08:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.180+0000 7f7ca0a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c9c102000 0x7f7c9c199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.180+0000 7f7ca0a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c9c104930 0x7f7c9c19a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.180+0000 7f7ca0a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c9c107260 0x7f7c9c1a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.180+0000 7f7ca0a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9c10d260 con 0x7f7c9c104930 2024-03-23T23:08:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.180+0000 7f7ca0a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9c10d0e0 con 0x7f7c9c102000 2024-03-23T23:08:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.180+0000 7f7ca0a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9c10d3e0 con 0x7f7c9c107260 2024-03-23T23:08:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.181+0000 7f7c9a800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c9c102000 0x7f7c9c199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.181+0000 7f7c9a800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c9c102000 0x7f7c9c199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57330/0 (socket says 172.21.15.12:57330) 2024-03-23T23:08:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.181+0000 7f7c9a800640 1 -- 172.21.15.12:0/1465423623 learned_addr learned my addr 172.21.15.12:0/1465423623 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.181+0000 7f7c9b200640 1 --2- 172.21.15.12:0/1465423623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c9c107260 0x7f7c9c1a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.181+0000 7f7c99e00640 1 --2- 172.21.15.12:0/1465423623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c9c104930 0x7f7c9c19a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.181+0000 7f7c9a800640 1 -- 172.21.15.12:0/1465423623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c9c107260 msgr2=0x7f7c9c1a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.181+0000 7f7c9a800640 1 --2- 172.21.15.12:0/1465423623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c9c107260 0x7f7c9c1a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.181+0000 7f7c9a800640 1 -- 172.21.15.12:0/1465423623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c9c104930 msgr2=0x7f7c9c19a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.181+0000 7f7c9a800640 1 --2- 172.21.15.12:0/1465423623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c9c104930 0x7f7c9c19a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.181+0000 7f7c9a800640 1 -- 172.21.15.12:0/1465423623 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c9c1a1710 con 0x7f7c9c102000 2024-03-23T23:08:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.181+0000 7f7c9b200640 1 --2- 172.21.15.12:0/1465423623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c9c107260 0x7f7c9c1a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.181+0000 7f7c9a800640 1 --2- 172.21.15.12:0/1465423623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c9c102000 0x7f7c9c199af0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c00d950 tx=0x7f7c8c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.182+0000 7f7c93400640 1 -- 172.21.15.12:0/1465423623 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c017030 con 0x7f7c9c102000 2024-03-23T23:08:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.182+0000 7f7c93400640 1 -- 172.21.15.12:0/1465423623 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c8c004020 con 0x7f7c9c102000 2024-03-23T23:08:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.182+0000 7f7ca0a00640 1 -- 172.21.15.12:0/1465423623 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c9c1a1a00 con 0x7f7c9c102000 2024-03-23T23:08:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.182+0000 7f7c93400640 1 -- 172.21.15.12:0/1465423623 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c0047a0 con 0x7f7c9c102000 2024-03-23T23:08:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.182+0000 7f7ca0a00640 1 -- 172.21.15.12:0/1465423623 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c9c1a1f20 con 0x7f7c9c102000 2024-03-23T23:08:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.186+0000 7f7ca41ed740 1 Processor -- start 2024-03-23T23:08:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.186+0000 7f7ca41ed740 1 -- start start 2024-03-23T23:08:38.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.187+0000 7f7ca41ed740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd469979f0 0x55dd46997e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.187+0000 7f7c93400640 1 -- 172.21.15.12:0/1465423623 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7c8c0041d0 con 0x7f7c9c102000 2024-03-23T23:08:38.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.187+0000 7f7ca41ed740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd46998bc0 0x55dd46999040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.187+0000 7f7ca41ed740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46999580 0x55dd4699fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.187+0000 7f7ca41ed740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd468fc2c0 con 0x55dd46998bc0 2024-03-23T23:08:38.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.187+0000 7f7ca41ed740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd468a4c90 con 0x55dd469979f0 2024-03-23T23:08:38.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.187+0000 7f7ca41ed740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd468f8360 con 0x55dd46999580 2024-03-23T23:08:38.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.187+0000 7f7c9b200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd469979f0 0x55dd46997e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.187+0000 7f7c9b200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd469979f0 0x55dd46997e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57338/0 (socket says 172.21.15.12:57338) 2024-03-23T23:08:38.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.187+0000 7f7c9b200640 1 -- 172.21.15.12:0/2094858491 learned_addr learned my addr 172.21.15.12:0/2094858491 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:38.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.187+0000 7f7c9a800640 1 --2- 172.21.15.12:0/2094858491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd46998bc0 0x55dd46999040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.187+0000 7f7c99e00640 1 --2- 172.21.15.12:0/2094858491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46999580 0x55dd4699fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.187+0000 7f7c9b200640 1 -- 172.21.15.12:0/2094858491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46999580 msgr2=0x55dd4699fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.187+0000 7f7c9b200640 1 --2- 172.21.15.12:0/2094858491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46999580 0x55dd4699fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.187+0000 7f7c9b200640 1 -- 172.21.15.12:0/2094858491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd46998bc0 msgr2=0x55dd46999040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.187+0000 7f7c9b200640 1 --2- 172.21.15.12:0/2094858491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd46998bc0 0x55dd46999040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.187+0000 7f7c9b200640 1 -- 172.21.15.12:0/2094858491 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd468c9080 con 0x55dd469979f0 2024-03-23T23:08:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.188+0000 7f7c9b200640 1 --2- 172.21.15.12:0/2094858491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd469979f0 0x55dd46997e10 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f7c94010710 tx=0x7f7c94010bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f3c74e1b74b3a16 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.188+0000 7f7c87400640 1 -- 172.21.15.12:0/2094858491 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9400e8d0 con 0x55dd469979f0 2024-03-23T23:08:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.188+0000 7f7c87400640 1 -- 172.21.15.12:0/2094858491 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c94012450 con 0x55dd469979f0 2024-03-23T23:08:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.188+0000 7f7c87400640 1 -- 172.21.15.12:0/2094858491 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9401c650 con 0x55dd469979f0 2024-03-23T23:08:38.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.188+0000 7f7c93400640 1 --2- 172.21.15.12:0/1465423623 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7c800a2800 0x7f7c800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.188+0000 7f7ca41ed740 1 -- 172.21.15.12:0/2094858491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd469979f0 msgr2=0x55dd46997e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.188+0000 7f7ca41ed740 1 --2- 172.21.15.12:0/2094858491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd469979f0 0x55dd46997e10 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f7c94010710 tx=0x7f7c94010bd0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.188+0000 7f7c93400640 1 -- 172.21.15.12:0/1465423623 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7c8c0ce410 con 0x7f7c9c102000 2024-03-23T23:08:38.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.189+0000 7f7ca41ed740 1 -- 172.21.15.12:0/2094858491 shutdown_connections 2024-03-23T23:08:38.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.189+0000 7f7ca41ed740 1 --2- 172.21.15.12:0/2094858491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46999580 0x55dd4699fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.189+0000 7f7ca41ed740 1 --2- 172.21.15.12:0/2094858491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd46998bc0 0x55dd46999040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.189+0000 7f7ca41ed740 1 --2- 172.21.15.12:0/2094858491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd469979f0 0x55dd46997e10 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.189+0000 7f7ca41ed740 1 -- 172.21.15.12:0/2094858491 >> 172.21.15.12:0/2094858491 conn(0x55dd46992de0 msgr2=0x55dd469931e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:38.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.189+0000 7f7c9a800640 1 --2- 172.21.15.12:0/1465423623 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7c800a2800 0x7f7c800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.189+0000 7f7ca41ed740 1 -- 172.21.15.12:0/2094858491 shutdown_connections 2024-03-23T23:08:38.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.189+0000 7f7ca41ed740 1 -- 172.21.15.12:0/2094858491 wait complete. 2024-03-23T23:08:38.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.189+0000 7f7ca41ed740 1 -- start start 2024-03-23T23:08:38.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.189+0000 7f7c9a800640 1 --2- 172.21.15.12:0/1465423623 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7c800a2800 0x7f7c800a4cc0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c09ad50 tx=0x7f7c8c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:38.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.189+0000 7f7ca41ed740 1 Processor -- start 2024-03-23T23:08:38.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.190+0000 7f7ca41ed740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd469979f0 0x55dd469a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.190+0000 7f7ca41ed740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd46998bc0 0x55dd469a21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.190+0000 7f7ca41ed740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46999580 0x55dd469a91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.190+0000 7f7ca41ed740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd469a0590 con 0x55dd46998bc0 2024-03-23T23:08:38.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.190+0000 7f7ca41ed740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd468b80d0 con 0x55dd469979f0 2024-03-23T23:08:38.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.190+0000 7f7ca41ed740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd469a0710 con 0x55dd46999580 2024-03-23T23:08:38.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.190+0000 7f7ca41ed740 10 client.0 _ll_register_callbacks cb 0x55dd468bce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:38.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.190+0000 7f7c99e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd46998bc0 0x55dd469a21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.190+0000 7f7c9b200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd469979f0 0x55dd469a1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.190+0000 7f7c99e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd46998bc0 0x55dd469a21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32846/0 (socket says 172.21.15.12:32846) 2024-03-23T23:08:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.190+0000 7f7c99e00640 1 -- 172.21.15.12:0/1574959302 learned_addr learned my addr 172.21.15.12:0/1574959302 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.190+0000 7f7c9b200640 1 --2- 172.21.15.12:0/1574959302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46999580 0x55dd469a91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.191+0000 7f7c99e00640 1 -- 172.21.15.12:0/1574959302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46999580 msgr2=0x55dd469a91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.191+0000 7f7c99e00640 1 --2- 172.21.15.12:0/1574959302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46999580 0x55dd469a91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.191+0000 7f7c99e00640 1 -- 172.21.15.12:0/1574959302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd469979f0 msgr2=0x55dd469a1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.191+0000 7f7c99e00640 1 --2- 172.21.15.12:0/1574959302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd469979f0 0x55dd469a1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.191+0000 7f7c99e00640 1 -- 172.21.15.12:0/1574959302 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dd466103f0 con 0x55dd46998bc0 2024-03-23T23:08:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.191+0000 7f7c9b200640 1 --2- 172.21.15.12:0/1574959302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46999580 0x55dd469a91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:08:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.191+0000 7f7c99e00640 1 --2- 172.21.15.12:0/1574959302 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd46998bc0 0x55dd469a21a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7c88005ab0 tx=0x7f7c88004250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=def8ee4355e4df37 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.191+0000 7f7c7be00640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:08:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.191+0000 7f7c7be00640 1 -- 172.21.15.12:0/1574959302 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c88005d50 con 0x55dd46998bc0 2024-03-23T23:08:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.191+0000 7f7c7be00640 1 -- 172.21.15.12:0/1574959302 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7c880046c0 con 0x55dd46998bc0 2024-03-23T23:08:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.191+0000 7f7ca41ed740 1 -- 172.21.15.12:0/1574959302 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7c94012450 con 0x55dd46998bc0 2024-03-23T23:08:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.192+0000 7f7c7be00640 1 -- 172.21.15.12:0/1574959302 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c880048c0 con 0x55dd46998bc0 2024-03-23T23:08:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.192+0000 7f7ca41ed740 10 client.17079 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.192+0000 7f7ca41ed740 1 -- 172.21.15.12:0/1574959302 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55dd466da7d0 con 0x55dd46998bc0 2024-03-23T23:08:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.192+0000 7f7c7be00640 1 -- 172.21.15.12:0/1574959302 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c94012450 con 0x55dd46998bc0 2024-03-23T23:08:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.192+0000 7f7c7be00640 1 -- 172.21.15.12:0/1574959302 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f7c88003680 con 0x55dd46998bc0 2024-03-23T23:08:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.192+0000 7f7ca41ed740 10 client.17079 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.192+0000 7f7ca41ed740 10 client.17079 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:08:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.192+0000 7f7ca41ed740 20 client.17079 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.192+0000 7f7ca41ed740 1 --2- 172.21.15.12:0/1574959302 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55dd469ad6c0 0x55dd469afb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.192+0000 7f7ca41ed740 4 client.17079 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:08:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.192+0000 7f7ca41ed740 1 -- 172.21.15.12:0/1574959302 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dd46909800 con 0x55dd469ad6c0 2024-03-23T23:08:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.193+0000 7f7c9b200640 1 --2- 172.21.15.12:0/1574959302 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55dd469ad6c0 0x55dd469afb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.193+0000 7f7c9b200640 1 --2- 172.21.15.12:0/1574959302 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55dd469ad6c0 0x55dd469afb20 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f9fcf15db2a4b1bc server_cookie=246c120d547a876a in_seq=0 out_seq=0 2024-03-23T23:08:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.193+0000 7f7c7be00640 10 client.17079 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:08:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.194+0000 7f7c7be00640 1 -- 172.21.15.12:0/1574959302 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55dd46909800 con 0x55dd469ad6c0 2024-03-23T23:08:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.194+0000 7f7c7be00640 10 client.17079 handle_command_reply: tid=0 2024-03-23T23:08:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.194+0000 7f7ca41ed740 1 client.17079 shutdown 2024-03-23T23:08:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.194+0000 7f7ca41ed740 1 -- 172.21.15.12:0/1574959302 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd46998bc0 msgr2=0x55dd469a21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.194+0000 7f7ca41ed740 1 --2- 172.21.15.12:0/1574959302 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd46998bc0 0x55dd469a21a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7c88005ab0 tx=0x7f7c88004250 comp rx=0 tx=0).stop 2024-03-23T23:08:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.194+0000 7f7ca41ed740 1 -- 172.21.15.12:0/1574959302 shutdown_connections 2024-03-23T23:08:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.194+0000 7f7ca41ed740 1 --2- 172.21.15.12:0/1574959302 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55dd469ad6c0 0x55dd469afb20 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.194+0000 7f7c7be00640 0 client.17079 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:08:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.194+0000 7f7ca41ed740 1 --2- 172.21.15.12:0/1574959302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46999580 0x55dd469a91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.194+0000 7f7ca41ed740 1 --2- 172.21.15.12:0/1574959302 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd46998bc0 0x55dd469a21a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.194+0000 7f7ca41ed740 1 --2- 172.21.15.12:0/1574959302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd469979f0 0x55dd469a1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.194+0000 7f7ca41ed740 1 -- 172.21.15.12:0/1574959302 >> 172.21.15.12:0/1574959302 conn(0x55dd46992de0 msgr2=0x55dd46992160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.194+0000 7f7ca41ed740 1 -- 172.21.15.12:0/1574959302 shutdown_connections 2024-03-23T23:08:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.195+0000 7f7ca41ed740 1 -- 172.21.15.12:0/1574959302 wait complete. 2024-03-23T23:08:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.195+0000 7f7ca41ed740 20 client.17079 trim_cache size 0 max 16384 2024-03-23T23:08:38.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.209+0000 7f7ca41ed740 1 Processor -- start 2024-03-23T23:08:38.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.209+0000 7f7ca41ed740 1 -- start start 2024-03-23T23:08:38.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.209+0000 7f7ca41ed740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd469979f0 0x55dd4699bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.209+0000 7f7ca41ed740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd46998bc0 0x55dd469a91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.209+0000 7f7ca41ed740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd46999580 0x55dd4699c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.209+0000 7f7ca41ed740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9c1acfd0 con 0x55dd46999580 2024-03-23T23:08:38.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.209+0000 7f7ca41ed740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9c1aaad0 con 0x55dd46998bc0 2024-03-23T23:08:38.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.209+0000 7f7ca41ed740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9c1a8b70 con 0x55dd469979f0 2024-03-23T23:08:38.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.209+0000 7f7c99e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd46998bc0 0x55dd469a91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.209+0000 7f7c99e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd46998bc0 0x55dd469a91a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57364/0 (socket says 172.21.15.12:57364) 2024-03-23T23:08:38.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.209+0000 7f7c99e00640 1 -- 172.21.15.12:0/2691441538 learned_addr learned my addr 172.21.15.12:0/2691441538 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:38.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.209+0000 7f7c9b200640 1 --2- 172.21.15.12:0/2691441538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd469979f0 0x55dd4699bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.210+0000 7f7c9b200640 1 --2- 172.21.15.12:0/2691441538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd46999580 0x55dd4699c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.210+0000 7f7c99e00640 1 -- 172.21.15.12:0/2691441538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd469979f0 msgr2=0x55dd4699bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:38.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.210+0000 7f7c99e00640 1 --2- 172.21.15.12:0/2691441538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd469979f0 0x55dd4699bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.210+0000 7f7c99e00640 1 -- 172.21.15.12:0/2691441538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd46999580 msgr2=0x55dd4699c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:38.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.210+0000 7f7c99e00640 1 --2- 172.21.15.12:0/2691441538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd46999580 0x55dd4699c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.210+0000 7f7c99e00640 1 -- 172.21.15.12:0/2691441538 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd46992b50 con 0x55dd46998bc0 2024-03-23T23:08:38.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.210+0000 7f7c9b200640 1 --2- 172.21.15.12:0/2691441538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd46999580 0x55dd4699c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:08:38.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.210+0000 7f7c99e00640 1 --2- 172.21.15.12:0/2691441538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd46998bc0 0x55dd469a91a0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7c88003ad0 tx=0x7f7c88009830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21e94a8aae95119e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:38.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.210+0000 7f7c86a00640 1 -- 172.21.15.12:0/2691441538 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c88005d50 con 0x55dd46998bc0 2024-03-23T23:08:38.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.210+0000 7f7c86a00640 1 -- 172.21.15.12:0/2691441538 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c8802cd40 con 0x55dd46998bc0 2024-03-23T23:08:38.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.210+0000 7f7c86a00640 1 -- 172.21.15.12:0/2691441538 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c880035f0 con 0x55dd46998bc0 2024-03-23T23:08:38.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.211+0000 7f7ca41ed740 1 -- 172.21.15.12:0/2691441538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd46998bc0 msgr2=0x55dd469a91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.211+0000 7f7ca41ed740 1 --2- 172.21.15.12:0/2691441538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd46998bc0 0x55dd469a91a0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7c88003ad0 tx=0x7f7c88009830 comp rx=0 tx=0).stop 2024-03-23T23:08:38.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.211+0000 7f7ca41ed740 1 -- 172.21.15.12:0/2691441538 shutdown_connections 2024-03-23T23:08:38.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.211+0000 7f7ca41ed740 1 --2- 172.21.15.12:0/2691441538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd46999580 0x55dd4699c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.211+0000 7f7ca41ed740 1 --2- 172.21.15.12:0/2691441538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd46998bc0 0x55dd469a91a0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.211+0000 7f7ca41ed740 1 --2- 172.21.15.12:0/2691441538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd469979f0 0x55dd4699bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.211+0000 7f7ca41ed740 1 -- 172.21.15.12:0/2691441538 >> 172.21.15.12:0/2691441538 conn(0x55dd46992de0 msgr2=0x55dd469968d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:38.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.211+0000 7f7ca41ed740 1 -- 172.21.15.12:0/2691441538 shutdown_connections 2024-03-23T23:08:38.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.211+0000 7f7ca41ed740 1 -- 172.21.15.12:0/2691441538 wait complete. 2024-03-23T23:08:38.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.212+0000 7f7ca41ed740 1 -- start start 2024-03-23T23:08:38.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.212+0000 7f7ca41ed740 1 Processor -- start 2024-03-23T23:08:38.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.212+0000 7f7ca41ed740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd469979f0 0x55dd469a6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.212+0000 7f7ca41ed740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd46998bc0 0x55dd469a74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.212+0000 7f7ca41ed740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46999580 0x55dd469a8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.212+0000 7f7ca41ed740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9c1a9170 con 0x55dd46998bc0 2024-03-23T23:08:38.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.212+0000 7f7ca41ed740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9c1aa470 con 0x55dd469979f0 2024-03-23T23:08:38.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.212+0000 7f7ca41ed740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9c1a97b0 con 0x55dd46999580 2024-03-23T23:08:38.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.212+0000 7f7ca41ed740 10 client.0 _ll_register_callbacks cb 0x7f7c8802c5f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:38.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.212+0000 7f7c99e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd46998bc0 0x55dd469a74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.213+0000 7f7c9b200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd469979f0 0x55dd469a6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.213+0000 7f7c99e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd46998bc0 0x55dd469a74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32864/0 (socket says 172.21.15.12:32864) 2024-03-23T23:08:38.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.213+0000 7f7c99e00640 1 -- 172.21.15.12:0/2420749244 learned_addr learned my addr 172.21.15.12:0/2420749244 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:38.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.213+0000 7f7c9b200640 1 --2- 172.21.15.12:0/2420749244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46999580 0x55dd469a8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.213+0000 7f7c99e00640 1 -- 172.21.15.12:0/2420749244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46999580 msgr2=0x55dd469a8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:38.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.213+0000 7f7c99e00640 1 --2- 172.21.15.12:0/2420749244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46999580 0x55dd469a8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.213+0000 7f7c99e00640 1 -- 172.21.15.12:0/2420749244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd469979f0 msgr2=0x55dd469a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:38.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.213+0000 7f7c99e00640 1 --2- 172.21.15.12:0/2420749244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd469979f0 0x55dd469a6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.213+0000 7f7c99e00640 1 -- 172.21.15.12:0/2420749244 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dd469a8a80 con 0x55dd46998bc0 2024-03-23T23:08:38.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.213+0000 7f7c9b200640 1 --2- 172.21.15.12:0/2420749244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd469979f0 0x55dd469a6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:38.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.213+0000 7f7c99e00640 1 --2- 172.21.15.12:0/2420749244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd46998bc0 0x55dd469a74f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7c88003a50 tx=0x7f7c88003e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b06ee12b3d48d998 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:38.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.213+0000 7f7c7be00640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:08:38.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.214+0000 7f7c7be00640 1 -- 172.21.15.12:0/2420749244 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c880035f0 con 0x55dd46998bc0 2024-03-23T23:08:38.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.214+0000 7f7c7be00640 1 -- 172.21.15.12:0/2420749244 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7c88003790 con 0x55dd46998bc0 2024-03-23T23:08:38.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.214+0000 7f7ca41ed740 1 -- 172.21.15.12:0/2420749244 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dd46903b50 con 0x55dd46998bc0 2024-03-23T23:08:38.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.214+0000 7f7c7be00640 1 -- 172.21.15.12:0/2420749244 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c88004eb0 con 0x55dd46998bc0 2024-03-23T23:08:38.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.214+0000 7f7ca41ed740 10 client.17082 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:38.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.214+0000 7f7ca41ed740 1 -- 172.21.15.12:0/2420749244 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55dd46a095a0 con 0x55dd46998bc0 2024-03-23T23:08:38.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.214+0000 7f7c7be00640 1 -- 172.21.15.12:0/2420749244 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55dd46903b50 con 0x55dd46998bc0 2024-03-23T23:08:38.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.214+0000 7f7c7be00640 1 -- 172.21.15.12:0/2420749244 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f7c8805c810 con 0x55dd46998bc0 2024-03-23T23:08:38.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.214+0000 7f7ca41ed740 10 client.17082 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:38.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.214+0000 7f7ca41ed740 10 client.17082 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:08:38.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.214+0000 7f7ca41ed740 20 client.17082 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:38.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.214+0000 7f7ca41ed740 1 --2- 172.21.15.12:0/2420749244 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55dd46a09e90 0x55dd46a0c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.214+0000 7f7ca41ed740 4 client.17082 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:08:38.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.214+0000 7f7ca41ed740 1 -- 172.21.15.12:0/2420749244 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55dd46881500 con 0x55dd46a09e90 2024-03-23T23:08:38.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.215+0000 7f7c9b200640 1 --2- 172.21.15.12:0/2420749244 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55dd46a09e90 0x55dd46a0c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.215+0000 7f7c9b200640 1 --2- 172.21.15.12:0/2420749244 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55dd46a09e90 0x55dd46a0c2b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d5c7d6340e63f2ce server_cookie=3f440717994c1922 in_seq=0 out_seq=0 2024-03-23T23:08:38.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.215+0000 7f7c7be00640 10 client.17082 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:08:38.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.215+0000 7f7c7be00640 1 -- 172.21.15.12:0/2420749244 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55dd46881500 con 0x55dd46a09e90 2024-03-23T23:08:38.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.215+0000 7f7c7be00640 10 client.17082 handle_command_reply: tid=0 2024-03-23T23:08:38.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.215+0000 7f7ca41ed740 1 client.17082 shutdown 2024-03-23T23:08:38.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.215+0000 7f7ca41ed740 1 -- 172.21.15.12:0/2420749244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd46998bc0 msgr2=0x55dd469a74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.215+0000 7f7ca41ed740 1 --2- 172.21.15.12:0/2420749244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd46998bc0 0x55dd469a74f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7c88003a50 tx=0x7f7c88003e70 comp rx=0 tx=0).stop 2024-03-23T23:08:38.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.215+0000 7f7ca41ed740 1 -- 172.21.15.12:0/2420749244 shutdown_connections 2024-03-23T23:08:38.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.215+0000 7f7ca41ed740 1 --2- 172.21.15.12:0/2420749244 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55dd46a09e90 0x55dd46a0c2b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.216+0000 7f7ca41ed740 1 --2- 172.21.15.12:0/2420749244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46999580 0x55dd469a8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.216+0000 7f7ca41ed740 1 --2- 172.21.15.12:0/2420749244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd46998bc0 0x55dd469a74f0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.216+0000 7f7c7be00640 0 client.17082 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:08:38.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.216+0000 7f7ca41ed740 1 --2- 172.21.15.12:0/2420749244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd469979f0 0x55dd469a6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.216+0000 7f7ca41ed740 1 -- 172.21.15.12:0/2420749244 >> 172.21.15.12:0/2420749244 conn(0x55dd46992de0 msgr2=0x55dd469968d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:38.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.216+0000 7f7ca41ed740 1 -- 172.21.15.12:0/2420749244 shutdown_connections 2024-03-23T23:08:38.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.216+0000 7f7ca41ed740 1 -- 172.21.15.12:0/2420749244 wait complete. 2024-03-23T23:08:38.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.216+0000 7f7ca41ed740 20 client.17082 trim_cache size 0 max 16384 2024-03-23T23:08:38.221 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:08:38.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.216+0000 7f7ca0a00640 1 -- 172.21.15.12:0/1465423623 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7c800a2800 msgr2=0x7f7c800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.216+0000 7f7ca0a00640 1 --2- 172.21.15.12:0/1465423623 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7c800a2800 0x7f7c800a4cc0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c09ad50 tx=0x7f7c8c000bc0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.216+0000 7f7ca0a00640 1 -- 172.21.15.12:0/1465423623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c9c102000 msgr2=0x7f7c9c199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.216+0000 7f7ca0a00640 1 --2- 172.21.15.12:0/1465423623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c9c102000 0x7f7c9c199af0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c00d950 tx=0x7f7c8c00de10 comp rx=0 tx=0).stop 2024-03-23T23:08:38.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.217+0000 7f7ca0a00640 1 -- 172.21.15.12:0/1465423623 shutdown_connections 2024-03-23T23:08:38.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.217+0000 7f7ca0a00640 1 --2- 172.21.15.12:0/1465423623 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7c800a2800 0x7f7c800a4cc0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.217+0000 7f7ca0a00640 1 --2- 172.21.15.12:0/1465423623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c9c107260 0x7f7c9c1a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.217+0000 7f7ca0a00640 1 --2- 172.21.15.12:0/1465423623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c9c104930 0x7f7c9c19a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.217+0000 7f7ca0a00640 1 --2- 172.21.15.12:0/1465423623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c9c102000 0x7f7c9c199af0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.217+0000 7f7ca0a00640 1 -- 172.21.15.12:0/1465423623 >> 172.21.15.12:0/1465423623 conn(0x7f7c9c0fba10 msgr2=0x7f7c9c1017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:38.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.217+0000 7f7ca0a00640 1 -- 172.21.15.12:0/1465423623 shutdown_connections 2024-03-23T23:08:38.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.217+0000 7f7ca0a00640 1 -- 172.21.15.12:0/1465423623 wait complete. 2024-03-23T23:08:38.236 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:08:38.236 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-23T23:08:38.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.460+0000 7f7018e00640 1 -- 172.21.15.12:0/734658971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70140756e0 msgr2=0x7f7014073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.460+0000 7f7018e00640 1 --2- 172.21.15.12:0/734658971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70140756e0 0x7f7014073b30 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f7004009ca0 tx=0x7f700402c100 comp rx=0 tx=0).stop 2024-03-23T23:08:38.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.460+0000 7f7018e00640 1 -- 172.21.15.12:0/734658971 shutdown_connections 2024-03-23T23:08:38.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.460+0000 7f7018e00640 1 --2- 172.21.15.12:0/734658971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7014109590 0x7f701410ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.460+0000 7f7018e00640 1 --2- 172.21.15.12:0/734658971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7014074100 0x7f7014074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.460+0000 7f7018e00640 1 --2- 172.21.15.12:0/734658971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70140756e0 0x7f7014073b30 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.460+0000 7f7018e00640 1 -- 172.21.15.12:0/734658971 >> 172.21.15.12:0/734658971 conn(0x7f70140fcc40 msgr2=0x7f70140ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:38.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.460+0000 7f7018e00640 1 -- 172.21.15.12:0/734658971 shutdown_connections 2024-03-23T23:08:38.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.460+0000 7f7018e00640 1 -- 172.21.15.12:0/734658971 wait complete. 2024-03-23T23:08:38.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.461+0000 7f7018e00640 1 Processor -- start 2024-03-23T23:08:38.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.461+0000 7f7018e00640 1 -- start start 2024-03-23T23:08:38.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.461+0000 7f7018e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7014074100 0x7f7014199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.461+0000 7f7018e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70140756e0 0x7f701419a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.461+0000 7f7018e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7014109590 0x7f70141a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.461+0000 7f7018e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701410f510 con 0x7f7014109590 2024-03-23T23:08:38.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.461+0000 7f7018e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701410f390 con 0x7f7014074100 2024-03-23T23:08:38.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.461+0000 7f7018e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701410f690 con 0x7f70140756e0 2024-03-23T23:08:38.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.461+0000 7f7011e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7014074100 0x7f7014199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.462+0000 7f7011400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70140756e0 0x7f701419a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.462+0000 7f7011e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7014074100 0x7f7014199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57386/0 (socket says 172.21.15.12:57386) 2024-03-23T23:08:38.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.462+0000 7f7011e00640 1 -- 172.21.15.12:0/3667687253 learned_addr learned my addr 172.21.15.12:0/3667687253 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:38.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.462+0000 7f7012800640 1 --2- 172.21.15.12:0/3667687253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7014109590 0x7f70141a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.462+0000 7f7011e00640 1 -- 172.21.15.12:0/3667687253 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70140756e0 msgr2=0x7f701419a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.462+0000 7f7011e00640 1 --2- 172.21.15.12:0/3667687253 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70140756e0 0x7f701419a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.462+0000 7f7011e00640 1 -- 172.21.15.12:0/3667687253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7014109590 msgr2=0x7f70141a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.462+0000 7f7011e00640 1 --2- 172.21.15.12:0/3667687253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7014109590 0x7f70141a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.462+0000 7f7011e00640 1 -- 172.21.15.12:0/3667687253 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70141a1800 con 0x7f7014074100 2024-03-23T23:08:38.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.462+0000 7f7011e00640 1 --2- 172.21.15.12:0/3667687253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7014074100 0x7f7014199be0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f70040325f0 tx=0x7f7004035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:38.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.462+0000 7f700aa00640 1 -- 172.21.15.12:0/3667687253 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7004039040 con 0x7f7014074100 2024-03-23T23:08:38.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.463+0000 7f700aa00640 1 -- 172.21.15.12:0/3667687253 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7004044070 con 0x7f7014074100 2024-03-23T23:08:38.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.463+0000 7f700aa00640 1 -- 172.21.15.12:0/3667687253 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7004004460 con 0x7f7014074100 2024-03-23T23:08:38.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.463+0000 7f7018e00640 1 -- 172.21.15.12:0/3667687253 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70141a1a90 con 0x7f7014074100 2024-03-23T23:08:38.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.463+0000 7f7018e00640 1 -- 172.21.15.12:0/3667687253 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70141a1f50 con 0x7f7014074100 2024-03-23T23:08:38.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.466+0000 7f701bb86740 1 Processor -- start 2024-03-23T23:08:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.466+0000 7f701bb86740 1 -- start start 2024-03-23T23:08:38.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.466+0000 7f700aa00640 1 -- 172.21.15.12:0/3667687253 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7004004600 con 0x7f7014074100 2024-03-23T23:08:38.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.466+0000 7f701bb86740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563808607810 0x563808607c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.466+0000 7f701bb86740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638086089e0 0x563808608e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.466+0000 7f701bb86740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638086093a0 0x56380860fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.466+0000 7f701bb86740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56380856c2c0 con 0x5638086093a0 2024-03-23T23:08:38.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.466+0000 7f701bb86740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x563808514c90 con 0x563808607810 2024-03-23T23:08:38.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.466+0000 7f701bb86740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x563808568360 con 0x5638086089e0 2024-03-23T23:08:38.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.467+0000 7f7012800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563808607810 0x563808607c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.467+0000 7f7012800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563808607810 0x563808607c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57398/0 (socket says 172.21.15.12:57398) 2024-03-23T23:08:38.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.467+0000 7f7012800640 1 -- 172.21.15.12:0/1362214844 learned_addr learned my addr 172.21.15.12:0/1362214844 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:38.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.467+0000 7f7011e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638086089e0 0x563808608e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.467+0000 7f7011400640 1 --2- 172.21.15.12:0/1362214844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638086093a0 0x56380860fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.467+0000 7f7012800640 1 -- 172.21.15.12:0/1362214844 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638086089e0 msgr2=0x563808608e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:38.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.467+0000 7f7012800640 1 --2- 172.21.15.12:0/1362214844 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638086089e0 0x563808608e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.467+0000 7f7012800640 1 -- 172.21.15.12:0/1362214844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638086093a0 msgr2=0x56380860fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:38.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.467+0000 7f7012800640 1 --2- 172.21.15.12:0/1362214844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638086093a0 0x56380860fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.467+0000 7f7012800640 1 -- 172.21.15.12:0/1362214844 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563808539080 con 0x563808607810 2024-03-23T23:08:38.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.467+0000 7f7011400640 1 --2- 172.21.15.12:0/1362214844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638086093a0 0x56380860fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:08:38.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.467+0000 7f7012800640 1 --2- 172.21.15.12:0/1362214844 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563808607810 0x563808607c30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f700c009c90 tx=0x7f700c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4898daafb418564b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:38.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.468+0000 7f7003400640 1 -- 172.21.15.12:0/1362214844 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c004230 con 0x563808607810 2024-03-23T23:08:38.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.468+0000 7f7003400640 1 -- 172.21.15.12:0/1362214844 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f700c009e40 con 0x563808607810 2024-03-23T23:08:38.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.468+0000 7f7003400640 1 -- 172.21.15.12:0/1362214844 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c0197e0 con 0x563808607810 2024-03-23T23:08:38.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.468+0000 7f700aa00640 1 --2- 172.21.15.12:0/3667687253 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6ff80a2800 0x7f6ff80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.468+0000 7f700aa00640 1 -- 172.21.15.12:0/3667687253 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f70040f0f90 con 0x7f7014074100 2024-03-23T23:08:38.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.468+0000 7f7011e00640 1 --2- 172.21.15.12:0/3667687253 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6ff80a2800 0x7f6ff80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.468+0000 7f701bb86740 1 -- 172.21.15.12:0/1362214844 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563808607810 msgr2=0x563808607c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.468+0000 7f701bb86740 1 --2- 172.21.15.12:0/1362214844 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563808607810 0x563808607c30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f700c009c90 tx=0x7f700c011810 comp rx=0 tx=0).stop 2024-03-23T23:08:38.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.469+0000 7f701bb86740 1 -- 172.21.15.12:0/1362214844 shutdown_connections 2024-03-23T23:08:38.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.469+0000 7f701bb86740 1 --2- 172.21.15.12:0/1362214844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638086093a0 0x56380860fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.469+0000 7f701bb86740 1 --2- 172.21.15.12:0/1362214844 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638086089e0 0x563808608e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.469+0000 7f701bb86740 1 --2- 172.21.15.12:0/1362214844 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563808607810 0x563808607c30 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.469+0000 7f701bb86740 1 -- 172.21.15.12:0/1362214844 >> 172.21.15.12:0/1362214844 conn(0x563808602c00 msgr2=0x563808603000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:38.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.469+0000 7f701bb86740 1 -- 172.21.15.12:0/1362214844 shutdown_connections 2024-03-23T23:08:38.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.469+0000 7f7011e00640 1 --2- 172.21.15.12:0/3667687253 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6ff80a2800 0x7f6ff80a4cc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f70040b9d40 tx=0x7f70040f49d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:38.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.469+0000 7f701bb86740 1 -- 172.21.15.12:0/1362214844 wait complete. 2024-03-23T23:08:38.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.470+0000 7f701bb86740 1 -- start start 2024-03-23T23:08:38.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.470+0000 7f701bb86740 1 Processor -- start 2024-03-23T23:08:38.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.470+0000 7f701bb86740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563808607810 0x563808611a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.470+0000 7f701bb86740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638086089e0 0x563808611fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.470+0000 7f701bb86740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638086093a0 0x563808619000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.470+0000 7f701bb86740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638086103b0 con 0x5638086093a0 2024-03-23T23:08:38.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.470+0000 7f701bb86740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638085280d0 con 0x563808607810 2024-03-23T23:08:38.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.470+0000 7f701bb86740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x563808610530 con 0x5638086089e0 2024-03-23T23:08:38.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.470+0000 7f701bb86740 10 client.0 _ll_register_callbacks cb 0x56380852ce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:38.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.471+0000 7f7011400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638086089e0 0x563808611fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.471+0000 7f7012800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563808607810 0x563808611a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.471+0000 7f7011400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638086089e0 0x563808611fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:42348/0 (socket says 172.21.15.12:42348) 2024-03-23T23:08:38.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.471+0000 7f7011400640 1 -- 172.21.15.12:0/3791161393 learned_addr learned my addr 172.21.15.12:0/3791161393 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:38.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.471+0000 7f7012800640 1 --2- 172.21.15.12:0/3791161393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638086093a0 0x563808619000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.471+0000 7f7011400640 1 -- 172.21.15.12:0/3791161393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563808607810 msgr2=0x563808611a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:38.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.471+0000 7f7011400640 1 --2- 172.21.15.12:0/3791161393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563808607810 0x563808611a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.471+0000 7f7011400640 1 -- 172.21.15.12:0/3791161393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638086093a0 msgr2=0x563808619000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:38.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.471+0000 7f7011400640 1 --2- 172.21.15.12:0/3791161393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638086093a0 0x563808619000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.471+0000 7f7011400640 1 -- 172.21.15.12:0/3791161393 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5638082803f0 con 0x5638086089e0 2024-03-23T23:08:38.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.472+0000 7f7011400640 1 --2- 172.21.15.12:0/3791161393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638086089e0 0x563808611fc0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc012700 tx=0x7f6ffc012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74501391992adc1c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:38.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.472+0000 7f6ff7e00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:08:38.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.472+0000 7f6ff7e00640 1 -- 172.21.15.12:0/3791161393 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc004050 con 0x5638086089e0 2024-03-23T23:08:38.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.472+0000 7f6ff7e00640 1 -- 172.21.15.12:0/3791161393 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f6ffc0041f0 con 0x5638086089e0 2024-03-23T23:08:38.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.472+0000 7f701bb86740 1 -- 172.21.15.12:0/3791161393 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563808579800 con 0x5638086089e0 2024-03-23T23:08:38.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.472+0000 7f6ff7e00640 1 -- 172.21.15.12:0/3791161393 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ffc004430 con 0x5638086089e0 2024-03-23T23:08:38.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.472+0000 7f701bb86740 10 client.27196 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:38.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.472+0000 7f701bb86740 1 -- 172.21.15.12:0/3791161393 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56380834a7d0 con 0x5638086089e0 2024-03-23T23:08:38.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.472+0000 7f6ff7e00640 1 -- 172.21.15.12:0/3791161393 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563808579800 con 0x5638086089e0 2024-03-23T23:08:38.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.473+0000 7f6ff7e00640 1 -- 172.21.15.12:0/3791161393 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f6ffc003b80 con 0x5638086089e0 2024-03-23T23:08:38.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.473+0000 7f701bb86740 10 client.27196 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:38.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.473+0000 7f701bb86740 10 client.27196 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:08:38.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.473+0000 7f701bb86740 20 client.27196 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:38.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.473+0000 7f701bb86740 1 --2- 172.21.15.12:0/3791161393 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x56380861d5e0 0x56380861fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.473+0000 7f701bb86740 4 client.27196 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:08:38.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.473+0000 7f701bb86740 1 -- 172.21.15.12:0/3791161393 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563808573b50 con 0x56380861d5e0 2024-03-23T23:08:38.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.473+0000 7f7012800640 1 --2- 172.21.15.12:0/3791161393 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x56380861d5e0 0x56380861fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.473+0000 7f7012800640 1 --2- 172.21.15.12:0/3791161393 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x56380861d5e0 0x56380861fa00 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d83df30535477e64 server_cookie=26918b1d2042b83e in_seq=0 out_seq=0 2024-03-23T23:08:38.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.473+0000 7f6ff7e00640 10 client.27196 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:08:38.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.474+0000 7f6ff7e00640 1 -- 172.21.15.12:0/3791161393 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563808573b50 con 0x56380861d5e0 2024-03-23T23:08:38.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.474+0000 7f6ff7e00640 10 client.27196 handle_command_reply: tid=0 2024-03-23T23:08:38.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.474+0000 7f701bb86740 1 client.27196 shutdown 2024-03-23T23:08:38.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.474+0000 7f701bb86740 1 -- 172.21.15.12:0/3791161393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638086089e0 msgr2=0x563808611fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.474+0000 7f701bb86740 1 --2- 172.21.15.12:0/3791161393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638086089e0 0x563808611fc0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc012700 tx=0x7f6ffc012bc0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.474+0000 7f701bb86740 1 -- 172.21.15.12:0/3791161393 shutdown_connections 2024-03-23T23:08:38.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.474+0000 7f701bb86740 1 --2- 172.21.15.12:0/3791161393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638086093a0 0x563808619000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.474+0000 7f701bb86740 1 --2- 172.21.15.12:0/3791161393 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x56380861d5e0 0x56380861fa00 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.474+0000 7f701bb86740 1 --2- 172.21.15.12:0/3791161393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638086089e0 0x563808611fc0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.475+0000 7f701bb86740 1 --2- 172.21.15.12:0/3791161393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563808607810 0x563808611a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.475+0000 7f6ff7e00640 0 client.27196 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:08:38.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.475+0000 7f701bb86740 1 -- 172.21.15.12:0/3791161393 >> 172.21.15.12:0/3791161393 conn(0x563808602c00 msgr2=0x563808601eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:38.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.475+0000 7f701bb86740 1 -- 172.21.15.12:0/3791161393 shutdown_connections 2024-03-23T23:08:38.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.475+0000 7f701bb86740 1 -- 172.21.15.12:0/3791161393 wait complete. 2024-03-23T23:08:38.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.475+0000 7f701bb86740 20 client.27196 trim_cache size 0 max 16384 2024-03-23T23:08:38.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.489+0000 7f701bb86740 1 Processor -- start 2024-03-23T23:08:38.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.489+0000 7f701bb86740 1 -- start start 2024-03-23T23:08:38.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.489+0000 7f701bb86740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563808607810 0x563808606570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.489+0000 7f701bb86740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638086089e0 0x563808618fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.489+0000 7f701bb86740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638086093a0 0x56380860bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.490+0000 7f701bb86740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70141ad1a0 con 0x5638086089e0 2024-03-23T23:08:38.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.490+0000 7f701bb86740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70141aaca0 con 0x563808607810 2024-03-23T23:08:38.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.490+0000 7f701bb86740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70141a8d40 con 0x5638086093a0 2024-03-23T23:08:38.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.490+0000 7f7012800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563808607810 0x563808606570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.490+0000 7f7012800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563808607810 0x563808606570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57418/0 (socket says 172.21.15.12:57418) 2024-03-23T23:08:38.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.490+0000 7f7012800640 1 -- 172.21.15.12:0/3354151995 learned_addr learned my addr 172.21.15.12:0/3354151995 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:38.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.490+0000 7f7012800640 1 --2- 172.21.15.12:0/3354151995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638086093a0 0x56380860bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.490+0000 7f7011400640 1 --2- 172.21.15.12:0/3354151995 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638086089e0 0x563808618fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.490+0000 7f7012800640 1 -- 172.21.15.12:0/3354151995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638086093a0 msgr2=0x56380860bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:38.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.490+0000 7f7012800640 1 --2- 172.21.15.12:0/3354151995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638086093a0 0x56380860bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.490+0000 7f7012800640 1 -- 172.21.15.12:0/3354151995 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638086089e0 msgr2=0x563808618fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:38.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.490+0000 7f7012800640 1 --2- 172.21.15.12:0/3354151995 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638086089e0 0x563808618fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.490+0000 7f7012800640 1 -- 172.21.15.12:0/3354151995 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563808603650 con 0x563808607810 2024-03-23T23:08:38.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.491+0000 7f7012800640 1 --2- 172.21.15.12:0/3354151995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563808607810 0x563808606570 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x5638086088e0 tx=0x7f700c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a56b6db1ca3240ad server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:38.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.491+0000 7f7002a00640 1 -- 172.21.15.12:0/3354151995 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c02c0b0 con 0x563808607810 2024-03-23T23:08:38.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.491+0000 7f7002a00640 1 -- 172.21.15.12:0/3354151995 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f700c02c250 con 0x563808607810 2024-03-23T23:08:38.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.491+0000 7f7002a00640 1 -- 172.21.15.12:0/3354151995 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c003850 con 0x563808607810 2024-03-23T23:08:38.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.491+0000 7f701bb86740 1 -- 172.21.15.12:0/3354151995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563808607810 msgr2=0x563808606570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.491+0000 7f701bb86740 1 --2- 172.21.15.12:0/3354151995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563808607810 0x563808606570 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x5638086088e0 tx=0x7f700c005f90 comp rx=0 tx=0).stop 2024-03-23T23:08:38.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.492+0000 7f701bb86740 1 -- 172.21.15.12:0/3354151995 shutdown_connections 2024-03-23T23:08:38.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.492+0000 7f701bb86740 1 --2- 172.21.15.12:0/3354151995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638086093a0 0x56380860bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.492+0000 7f701bb86740 1 --2- 172.21.15.12:0/3354151995 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638086089e0 0x563808618fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.492+0000 7f701bb86740 1 --2- 172.21.15.12:0/3354151995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563808607810 0x563808606570 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.492+0000 7f701bb86740 1 -- 172.21.15.12:0/3354151995 >> 172.21.15.12:0/3354151995 conn(0x563808602c00 msgr2=0x563808605e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:38.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.492+0000 7f701bb86740 1 -- 172.21.15.12:0/3354151995 shutdown_connections 2024-03-23T23:08:38.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.492+0000 7f701bb86740 1 -- 172.21.15.12:0/3354151995 wait complete. 2024-03-23T23:08:38.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.492+0000 7f701bb86740 1 -- start start 2024-03-23T23:08:38.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.492+0000 7f701bb86740 1 Processor -- start 2024-03-23T23:08:38.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.493+0000 7f701bb86740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563808607810 0x5638086148f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.493+0000 7f701bb86740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638086089e0 0x563808614e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.493+0000 7f701bb86740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638086093a0 0x563808617db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.493+0000 7f701bb86740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70141a9340 con 0x563808607810 2024-03-23T23:08:38.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.493+0000 7f701bb86740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70141aa640 con 0x5638086093a0 2024-03-23T23:08:38.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.493+0000 7f701bb86740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70141a9980 con 0x5638086089e0 2024-03-23T23:08:38.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.493+0000 7f701bb86740 10 client.0 _ll_register_callbacks cb 0x7f6ffc009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:38.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.493+0000 7f7012800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563808607810 0x5638086148f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.493+0000 7f7011400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638086089e0 0x563808614e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.493+0000 7f7011400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638086089e0 0x563808614e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:42366/0 (socket says 172.21.15.12:42366) 2024-03-23T23:08:38.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.493+0000 7f7011400640 1 -- 172.21.15.12:0/680828037 learned_addr learned my addr 172.21.15.12:0/680828037 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:38.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.493+0000 7f7012800640 1 --2- 172.21.15.12:0/680828037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638086093a0 0x563808617db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.493+0000 7f7011400640 1 -- 172.21.15.12:0/680828037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638086093a0 msgr2=0x563808617db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:38.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.493+0000 7f7011400640 1 --2- 172.21.15.12:0/680828037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638086093a0 0x563808617db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.494+0000 7f7011400640 1 -- 172.21.15.12:0/680828037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563808607810 msgr2=0x5638086148f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:38.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.494+0000 7f7011400640 1 --2- 172.21.15.12:0/680828037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563808607810 0x5638086148f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.494+0000 7f7011400640 1 -- 172.21.15.12:0/680828037 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5638086183a0 con 0x5638086089e0 2024-03-23T23:08:38.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.494+0000 7f7012800640 1 --2- 172.21.15.12:0/680828037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638086093a0 0x563808617db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:38.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.494+0000 7f7011400640 1 --2- 172.21.15.12:0/680828037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638086089e0 0x563808614e30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc000c00 tx=0x7f6ffc005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10c13db7ba71107 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:38.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.494+0000 7f6ff7e00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:08:38.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.494+0000 7f6ff7e00640 1 -- 172.21.15.12:0/680828037 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc009e50 con 0x5638086089e0 2024-03-23T23:08:38.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.494+0000 7f6ff7e00640 1 -- 172.21.15.12:0/680828037 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f6ffc00f790 con 0x5638086089e0 2024-03-23T23:08:38.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.494+0000 7f701bb86740 1 -- 172.21.15.12:0/680828037 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f700c02c250 con 0x5638086089e0 2024-03-23T23:08:38.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.495+0000 7f6ff7e00640 1 -- 172.21.15.12:0/680828037 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ffc00f990 con 0x5638086089e0 2024-03-23T23:08:38.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.495+0000 7f701bb86740 10 client.27202 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:38.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.495+0000 7f701bb86740 1 -- 172.21.15.12:0/680828037 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563808678f70 con 0x5638086089e0 2024-03-23T23:08:38.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.495+0000 7f6ff7e00640 1 -- 172.21.15.12:0/680828037 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f700c02c250 con 0x5638086089e0 2024-03-23T23:08:38.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.495+0000 7f6ff7e00640 1 -- 172.21.15.12:0/680828037 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f6ffc039030 con 0x5638086089e0 2024-03-23T23:08:38.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.495+0000 7f701bb86740 10 client.27202 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:38.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.495+0000 7f701bb86740 10 client.27202 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:08:38.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.495+0000 7f701bb86740 20 client.27202 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:38.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.495+0000 7f701bb86740 1 --2- 172.21.15.12:0/680828037 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x563808679810 0x56380867bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.495+0000 7f701bb86740 4 client.27202 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:08:38.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.495+0000 7f701bb86740 1 -- 172.21.15.12:0/680828037 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5638084f1500 con 0x563808679810 2024-03-23T23:08:38.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.496+0000 7f7012800640 1 --2- 172.21.15.12:0/680828037 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x563808679810 0x56380867bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.496+0000 7f7012800640 1 --2- 172.21.15.12:0/680828037 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x563808679810 0x56380867bc70 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=eb763b679096e614 server_cookie=8da56422140eb4aa in_seq=0 out_seq=0 2024-03-23T23:08:38.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.496+0000 7f6ff7e00640 10 client.27202 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:08:38.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.496+0000 7f6ff7e00640 1 -- 172.21.15.12:0/680828037 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5638084f1500 con 0x563808679810 2024-03-23T23:08:38.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.496+0000 7f6ff7e00640 10 client.27202 handle_command_reply: tid=0 2024-03-23T23:08:38.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.496+0000 7f701bb86740 1 client.27202 shutdown 2024-03-23T23:08:38.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.496+0000 7f701bb86740 1 -- 172.21.15.12:0/680828037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638086089e0 msgr2=0x563808614e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.496+0000 7f701bb86740 1 --2- 172.21.15.12:0/680828037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638086089e0 0x563808614e30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc000c00 tx=0x7f6ffc005270 comp rx=0 tx=0).stop 2024-03-23T23:08:38.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.496+0000 7f701bb86740 1 -- 172.21.15.12:0/680828037 shutdown_connections 2024-03-23T23:08:38.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.496+0000 7f701bb86740 1 --2- 172.21.15.12:0/680828037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638086093a0 0x563808617db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.496+0000 7f701bb86740 1 --2- 172.21.15.12:0/680828037 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x563808679810 0x56380867bc70 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.496+0000 7f701bb86740 1 --2- 172.21.15.12:0/680828037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638086089e0 0x563808614e30 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.496+0000 7f701bb86740 1 --2- 172.21.15.12:0/680828037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563808607810 0x5638086148f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.496+0000 7f701bb86740 1 -- 172.21.15.12:0/680828037 >> 172.21.15.12:0/680828037 conn(0x563808602c00 msgr2=0x5638086066f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:38.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.496+0000 7f6ff7e00640 0 client.27202 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:08:38.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.497+0000 7f701bb86740 1 -- 172.21.15.12:0/680828037 shutdown_connections 2024-03-23T23:08:38.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.497+0000 7f701bb86740 1 -- 172.21.15.12:0/680828037 wait complete. 2024-03-23T23:08:38.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.497+0000 7f701bb86740 20 client.27202 trim_cache size 0 max 16384 2024-03-23T23:08:38.502 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:08:38.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.497+0000 7f7018e00640 1 -- 172.21.15.12:0/3667687253 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6ff80a2800 msgr2=0x7f6ff80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.497+0000 7f7018e00640 1 --2- 172.21.15.12:0/3667687253 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6ff80a2800 0x7f6ff80a4cc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f70040b9d40 tx=0x7f70040f49d0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.497+0000 7f7018e00640 1 -- 172.21.15.12:0/3667687253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7014074100 msgr2=0x7f7014199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.497+0000 7f7018e00640 1 --2- 172.21.15.12:0/3667687253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7014074100 0x7f7014199be0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f70040325f0 tx=0x7f7004035d10 comp rx=0 tx=0).stop 2024-03-23T23:08:38.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.497+0000 7f7018e00640 1 -- 172.21.15.12:0/3667687253 shutdown_connections 2024-03-23T23:08:38.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.497+0000 7f7018e00640 1 --2- 172.21.15.12:0/3667687253 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f6ff80a2800 0x7f6ff80a4cc0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.497+0000 7f7018e00640 1 --2- 172.21.15.12:0/3667687253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7014109590 0x7f70141a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.497+0000 7f7018e00640 1 --2- 172.21.15.12:0/3667687253 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70140756e0 0x7f701419a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.497+0000 7f7018e00640 1 --2- 172.21.15.12:0/3667687253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7014074100 0x7f7014199be0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.497+0000 7f7018e00640 1 -- 172.21.15.12:0/3667687253 >> 172.21.15.12:0/3667687253 conn(0x7f70140fcc40 msgr2=0x7f7014105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:38.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.498+0000 7f7018e00640 1 -- 172.21.15.12:0/3667687253 shutdown_connections 2024-03-23T23:08:38.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.498+0000 7f7018e00640 1 -- 172.21.15.12:0/3667687253 wait complete. 2024-03-23T23:08:38.516 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:08:38.516 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-23T23:08:38.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.728+0000 7fbe50000640 1 -- 172.21.15.12:0/3793216089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe48104ab0 msgr2=0x7fbe48104f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.728+0000 7fbe50000640 1 --2- 172.21.15.12:0/3793216089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe48104ab0 0x7fbe48104f30 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fbe34009900 tx=0x7fbe3402bc30 comp rx=0 tx=0).stop 2024-03-23T23:08:38.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.728+0000 7fbe50000640 1 -- 172.21.15.12:0/3793216089 shutdown_connections 2024-03-23T23:08:38.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.728+0000 7fbe50000640 1 --2- 172.21.15.12:0/3793216089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe48105470 0x7fbe4810bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.728+0000 7fbe50000640 1 --2- 172.21.15.12:0/3793216089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe48104ab0 0x7fbe48104f30 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.728+0000 7fbe50000640 1 --2- 172.21.15.12:0/3793216089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe481038c0 0x7fbe48103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.728+0000 7fbe50000640 1 -- 172.21.15.12:0/3793216089 >> 172.21.15.12:0/3793216089 conn(0x7fbe480fee50 msgr2=0x7fbe48101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:38.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.729+0000 7fbe50000640 1 -- 172.21.15.12:0/3793216089 shutdown_connections 2024-03-23T23:08:38.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.729+0000 7fbe50000640 1 -- 172.21.15.12:0/3793216089 wait complete. 2024-03-23T23:08:38.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.730+0000 7fbe50000640 1 Processor -- start 2024-03-23T23:08:38.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.730+0000 7fbe50000640 1 -- start start 2024-03-23T23:08:38.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.730+0000 7fbe50000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe481038c0 0x7fbe48199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.730+0000 7fbe50000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe48104ab0 0x7fbe4819a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.730+0000 7fbe50000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe48105470 0x7fbe481a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.730+0000 7fbe50000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4810f570 con 0x7fbe48104ab0 2024-03-23T23:08:38.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.730+0000 7fbe50000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4810f3f0 con 0x7fbe48105470 2024-03-23T23:08:38.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.730+0000 7fbe50000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4810f6f0 con 0x7fbe481038c0 2024-03-23T23:08:38.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.730+0000 7fbe4d600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe481038c0 0x7fbe48199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.730+0000 7fbe4d600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe481038c0 0x7fbe48199bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:42382/0 (socket says 172.21.15.12:42382) 2024-03-23T23:08:38.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.730+0000 7fbe4d600640 1 -- 172.21.15.12:0/2183915642 learned_addr learned my addr 172.21.15.12:0/2183915642 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:38.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.730+0000 7fbe4e000640 1 --2- 172.21.15.12:0/2183915642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe48105470 0x7fbe481a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.731+0000 7fbe4cc00640 1 --2- 172.21.15.12:0/2183915642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe48104ab0 0x7fbe4819a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.731+0000 7fbe4d600640 1 -- 172.21.15.12:0/2183915642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe48105470 msgr2=0x7fbe481a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.731+0000 7fbe4d600640 1 --2- 172.21.15.12:0/2183915642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe48105470 0x7fbe481a1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.731+0000 7fbe4d600640 1 -- 172.21.15.12:0/2183915642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe48104ab0 msgr2=0x7fbe4819a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.731+0000 7fbe4d600640 1 --2- 172.21.15.12:0/2183915642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe48104ab0 0x7fbe4819a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.731+0000 7fbe4d600640 1 -- 172.21.15.12:0/2183915642 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe481a17d0 con 0x7fbe481038c0 2024-03-23T23:08:38.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.731+0000 7fbe4d600640 1 --2- 172.21.15.12:0/2183915642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe481038c0 0x7fbe48199bb0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c00ebc0 tx=0x7fbe3c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:38.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.731+0000 7fbe46000640 1 -- 172.21.15.12:0/2183915642 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c003b70 con 0x7fbe481038c0 2024-03-23T23:08:38.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.732+0000 7fbe46000640 1 -- 172.21.15.12:0/2183915642 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe3c003d10 con 0x7fbe481038c0 2024-03-23T23:08:38.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.732+0000 7fbe50000640 1 -- 172.21.15.12:0/2183915642 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe481a1ac0 con 0x7fbe481038c0 2024-03-23T23:08:38.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.732+0000 7fbe46000640 1 -- 172.21.15.12:0/2183915642 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c019980 con 0x7fbe481038c0 2024-03-23T23:08:38.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.732+0000 7fbe50000640 1 -- 172.21.15.12:0/2183915642 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe481a2030 con 0x7fbe481038c0 2024-03-23T23:08:38.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.736+0000 7fbe52d96740 1 Processor -- start 2024-03-23T23:08:38.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.736+0000 7fbe52d96740 1 -- start start 2024-03-23T23:08:38.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.737+0000 7fbe46000640 1 -- 172.21.15.12:0/2183915642 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbe3c019b20 con 0x7fbe481038c0 2024-03-23T23:08:38.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.737+0000 7fbe52d96740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe072a2810 0x55fe072a2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.737+0000 7fbe52d96740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fe072a39e0 0x55fe072a3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.737+0000 7fbe52d96740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a43a0 0x55fe072aac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.737+0000 7fbe52d96740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe072072c0 con 0x55fe072a39e0 2024-03-23T23:08:38.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.737+0000 7fbe52d96740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe071afc90 con 0x55fe072a43a0 2024-03-23T23:08:38.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.737+0000 7fbe52d96740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe07203360 con 0x55fe072a2810 2024-03-23T23:08:38.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.737+0000 7fbe4e000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe072a2810 0x55fe072a2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.737+0000 7fbe4e000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe072a2810 0x55fe072a2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:42392/0 (socket says 172.21.15.12:42392) 2024-03-23T23:08:38.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.737+0000 7fbe4cc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a43a0 0x55fe072aac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.737+0000 7fbe4e000640 1 -- 172.21.15.12:0/2699859836 learned_addr learned my addr 172.21.15.12:0/2699859836 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:38.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.737+0000 7fbe4d600640 1 --2- 172.21.15.12:0/2699859836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fe072a39e0 0x55fe072a3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.737+0000 7fbe4e000640 1 -- 172.21.15.12:0/2699859836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a43a0 msgr2=0x55fe072aac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:38.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.737+0000 7fbe4e000640 1 --2- 172.21.15.12:0/2699859836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a43a0 0x55fe072aac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.737+0000 7fbe4e000640 1 -- 172.21.15.12:0/2699859836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fe072a39e0 msgr2=0x55fe072a3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:38.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.737+0000 7fbe4e000640 1 --2- 172.21.15.12:0/2699859836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fe072a39e0 0x55fe072a3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.737+0000 7fbe4e000640 1 -- 172.21.15.12:0/2699859836 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe071d4080 con 0x55fe072a2810 2024-03-23T23:08:38.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.738+0000 7fbe4d600640 1 --2- 172.21.15.12:0/2699859836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fe072a39e0 0x55fe072a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:08:38.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.738+0000 7fbe4e000640 1 --2- 172.21.15.12:0/2699859836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe072a2810 0x55fe072a2c30 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fbe40009ca0 tx=0x7fbe40011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f38762a4aca390d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:38.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.738+0000 7fbe3a600640 1 -- 172.21.15.12:0/2699859836 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4001e070 con 0x55fe072a2810 2024-03-23T23:08:38.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.738+0000 7fbe3a600640 1 -- 172.21.15.12:0/2699859836 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe40009e50 con 0x55fe072a2810 2024-03-23T23:08:38.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.738+0000 7fbe3a600640 1 -- 172.21.15.12:0/2699859836 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4000eac0 con 0x55fe072a2810 2024-03-23T23:08:38.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.738+0000 7fbe46000640 1 --2- 172.21.15.12:0/2183915642 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbe300a2880 0x7fbe300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.739+0000 7fbe46000640 1 -- 172.21.15.12:0/2183915642 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fbe3c0cf180 con 0x7fbe481038c0 2024-03-23T23:08:38.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.739+0000 7fbe4d600640 1 --2- 172.21.15.12:0/2183915642 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbe300a2880 0x7fbe300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.739+0000 7fbe52d96740 1 -- 172.21.15.12:0/2699859836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe072a2810 msgr2=0x55fe072a2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.739+0000 7fbe52d96740 1 --2- 172.21.15.12:0/2699859836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe072a2810 0x55fe072a2c30 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fbe40009ca0 tx=0x7fbe40011810 comp rx=0 tx=0).stop 2024-03-23T23:08:38.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.739+0000 7fbe52d96740 1 -- 172.21.15.12:0/2699859836 shutdown_connections 2024-03-23T23:08:38.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.739+0000 7fbe52d96740 1 --2- 172.21.15.12:0/2699859836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a43a0 0x55fe072aac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.739+0000 7fbe52d96740 1 --2- 172.21.15.12:0/2699859836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fe072a39e0 0x55fe072a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.739+0000 7fbe52d96740 1 --2- 172.21.15.12:0/2699859836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe072a2810 0x55fe072a2c30 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.739+0000 7fbe52d96740 1 -- 172.21.15.12:0/2699859836 >> 172.21.15.12:0/2699859836 conn(0x55fe0729dc00 msgr2=0x55fe0729e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:38.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.739+0000 7fbe52d96740 1 -- 172.21.15.12:0/2699859836 shutdown_connections 2024-03-23T23:08:38.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.739+0000 7fbe52d96740 1 -- 172.21.15.12:0/2699859836 wait complete. 2024-03-23T23:08:38.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.739+0000 7fbe4d600640 1 --2- 172.21.15.12:0/2183915642 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbe300a2880 0x7fbe300a4d40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c0ccc90 tx=0x7fbe3c00cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:38.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.740+0000 7fbe52d96740 1 -- start start 2024-03-23T23:08:38.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.740+0000 7fbe52d96740 1 Processor -- start 2024-03-23T23:08:38.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.740+0000 7fbe52d96740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe072a2810 0x55fe072acae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.740+0000 7fbe52d96740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a39e0 0x55fe072ad020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.740+0000 7fbe52d96740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fe072a43a0 0x55fe072b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.740+0000 7fbe52d96740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe072ab3b0 con 0x55fe072a43a0 2024-03-23T23:08:38.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.740+0000 7fbe52d96740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe071c30d0 con 0x55fe072a39e0 2024-03-23T23:08:38.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.740+0000 7fbe52d96740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe072ab530 con 0x55fe072a2810 2024-03-23T23:08:38.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.740+0000 7fbe52d96740 10 client.0 _ll_register_callbacks cb 0x55fe071c7e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:38.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.741+0000 7fbe4cc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a39e0 0x55fe072ad020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.741+0000 7fbe4cc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a39e0 0x55fe072ad020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57452/0 (socket says 172.21.15.12:57452) 2024-03-23T23:08:38.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.741+0000 7fbe4cc00640 1 -- 172.21.15.12:0/4140627085 learned_addr learned my addr 172.21.15.12:0/4140627085 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:38.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.741+0000 7fbe4e000640 1 --2- 172.21.15.12:0/4140627085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe072a2810 0x55fe072acae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.741+0000 7fbe4e000640 1 --2- 172.21.15.12:0/4140627085 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fe072a43a0 0x55fe072b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.741+0000 7fbe4cc00640 1 -- 172.21.15.12:0/4140627085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe072a2810 msgr2=0x55fe072acae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:38.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.741+0000 7fbe4cc00640 1 --2- 172.21.15.12:0/4140627085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe072a2810 0x55fe072acae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.741+0000 7fbe4cc00640 1 -- 172.21.15.12:0/4140627085 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fe072a43a0 msgr2=0x55fe072b3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:38.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.741+0000 7fbe4cc00640 1 --2- 172.21.15.12:0/4140627085 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fe072a43a0 0x55fe072b3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.741+0000 7fbe4cc00640 1 -- 172.21.15.12:0/4140627085 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe06f1b3f0 con 0x55fe072a39e0 2024-03-23T23:08:38.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.741+0000 7fbe4e000640 1 --2- 172.21.15.12:0/4140627085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe072a2810 0x55fe072acae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:08:38.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.741+0000 7fbe4cc00640 1 --2- 172.21.15.12:0/4140627085 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a39e0 0x55fe072ad020 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fbe34000c00 tx=0x7fbe34004250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e93c8d3167c4e630 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:38.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.742+0000 7fbe2ea00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:08:38.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.742+0000 7fbe2ea00640 1 -- 172.21.15.12:0/4140627085 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe34005d50 con 0x55fe072a39e0 2024-03-23T23:08:38.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.742+0000 7fbe2ea00640 1 -- 172.21.15.12:0/4140627085 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fbe340046c0 con 0x55fe072a39e0 2024-03-23T23:08:38.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.742+0000 7fbe52d96740 1 -- 172.21.15.12:0/4140627085 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbe40009e50 con 0x55fe072a39e0 2024-03-23T23:08:38.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.742+0000 7fbe2ea00640 1 -- 172.21.15.12:0/4140627085 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe340048e0 con 0x55fe072a39e0 2024-03-23T23:08:38.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.742+0000 7fbe52d96740 10 client.27242 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:38.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.742+0000 7fbe52d96740 1 -- 172.21.15.12:0/4140627085 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55fe06fe57d0 con 0x55fe072a39e0 2024-03-23T23:08:38.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.743+0000 7fbe2ea00640 1 -- 172.21.15.12:0/4140627085 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe40009e50 con 0x55fe072a39e0 2024-03-23T23:08:38.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.743+0000 7fbe2ea00640 1 -- 172.21.15.12:0/4140627085 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fbe340035f0 con 0x55fe072a39e0 2024-03-23T23:08:38.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.743+0000 7fbe52d96740 10 client.27242 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:38.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.743+0000 7fbe52d96740 10 client.27242 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:08:38.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.743+0000 7fbe52d96740 20 client.27242 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:38.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.743+0000 7fbe52d96740 1 --2- 172.21.15.12:0/4140627085 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fe072b8510 0x55fe072ba930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.743+0000 7fbe52d96740 4 client.27242 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:08:38.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.743+0000 7fbe52d96740 1 -- 172.21.15.12:0/4140627085 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fe07214800 con 0x55fe072b8510 2024-03-23T23:08:38.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.743+0000 7fbe4e000640 1 --2- 172.21.15.12:0/4140627085 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fe072b8510 0x55fe072ba930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.744+0000 7fbe4e000640 1 --2- 172.21.15.12:0/4140627085 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fe072b8510 0x55fe072ba930 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5141a13a12b51a9 server_cookie=a8d883de1a13728e in_seq=0 out_seq=0 2024-03-23T23:08:38.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.744+0000 7fbe2ea00640 10 client.27242 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:38.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.744+0000 7fbe2ea00640 1 -- 172.21.15.12:0/4140627085 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55fe07214800 con 0x55fe072b8510 2024-03-23T23:08:38.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.744+0000 7fbe2ea00640 10 client.27242 handle_command_reply: tid=0 2024-03-23T23:08:38.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.744+0000 7fbe52d96740 1 client.27242 shutdown 2024-03-23T23:08:38.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.745+0000 7fbe52d96740 1 -- 172.21.15.12:0/4140627085 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a39e0 msgr2=0x55fe072ad020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.745+0000 7fbe52d96740 1 --2- 172.21.15.12:0/4140627085 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a39e0 0x55fe072ad020 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fbe34000c00 tx=0x7fbe34004250 comp rx=0 tx=0).stop 2024-03-23T23:08:38.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.745+0000 7fbe52d96740 1 -- 172.21.15.12:0/4140627085 shutdown_connections 2024-03-23T23:08:38.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.745+0000 7fbe52d96740 1 --2- 172.21.15.12:0/4140627085 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fe072b8510 0x55fe072ba930 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.745+0000 7fbe52d96740 1 --2- 172.21.15.12:0/4140627085 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fe072a43a0 0x55fe072b3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.745+0000 7fbe52d96740 1 --2- 172.21.15.12:0/4140627085 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a39e0 0x55fe072ad020 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.745+0000 7fbe52d96740 1 --2- 172.21.15.12:0/4140627085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe072a2810 0x55fe072acae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.745+0000 7fbe2ea00640 0 client.27242 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:38.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.745+0000 7fbe52d96740 1 -- 172.21.15.12:0/4140627085 >> 172.21.15.12:0/4140627085 conn(0x55fe0729dc00 msgr2=0x55fe0729ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:38.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.745+0000 7fbe52d96740 1 -- 172.21.15.12:0/4140627085 shutdown_connections 2024-03-23T23:08:38.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.745+0000 7fbe52d96740 1 -- 172.21.15.12:0/4140627085 wait complete. 2024-03-23T23:08:38.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.745+0000 7fbe52d96740 20 client.27242 trim_cache size 0 max 16384 2024-03-23T23:08:38.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.758+0000 7fbe52d96740 1 Processor -- start 2024-03-23T23:08:38.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.759+0000 7fbe52d96740 1 -- start start 2024-03-23T23:08:38.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.759+0000 7fbe52d96740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a2810 0x55fe072a1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.759+0000 7fbe52d96740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe072a39e0 0x55fe072b3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.759+0000 7fbe52d96740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fe072a43a0 0x55fe072a6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.759+0000 7fbe52d96740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe481ad170 con 0x55fe072a43a0 2024-03-23T23:08:38.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.759+0000 7fbe52d96740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe481aac70 con 0x55fe072a2810 2024-03-23T23:08:38.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.759+0000 7fbe52d96740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe481a8d10 con 0x55fe072a39e0 2024-03-23T23:08:38.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.759+0000 7fbe4e000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a2810 0x55fe072a1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.759+0000 7fbe4e000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a2810 0x55fe072a1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57458/0 (socket says 172.21.15.12:57458) 2024-03-23T23:08:38.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.759+0000 7fbe4e000640 1 -- 172.21.15.12:0/3448995451 learned_addr learned my addr 172.21.15.12:0/3448995451 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:38.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.759+0000 7fbe4e000640 1 --2- 172.21.15.12:0/3448995451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fe072a43a0 0x55fe072a6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.759+0000 7fbe4cc00640 1 --2- 172.21.15.12:0/3448995451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe072a39e0 0x55fe072b3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.759+0000 7fbe4e000640 1 -- 172.21.15.12:0/3448995451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe072a39e0 msgr2=0x55fe072b3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:38.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.760+0000 7fbe4e000640 1 --2- 172.21.15.12:0/3448995451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe072a39e0 0x55fe072b3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.760+0000 7fbe4e000640 1 -- 172.21.15.12:0/3448995451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fe072a43a0 msgr2=0x55fe072a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:38.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.760+0000 7fbe4e000640 1 --2- 172.21.15.12:0/3448995451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fe072a43a0 0x55fe072a6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.760+0000 7fbe4e000640 1 -- 172.21.15.12:0/3448995451 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe072b82d0 con 0x55fe072a2810 2024-03-23T23:08:38.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.760+0000 7fbe4e000640 1 --2- 172.21.15.12:0/3448995451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a2810 0x55fe072a1570 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fbe40009ab0 tx=0x7fbe40000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18914c74f42053ac server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:38.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.760+0000 7fbe39c00640 1 -- 172.21.15.12:0/3448995451 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4001e040 con 0x55fe072a2810 2024-03-23T23:08:38.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.760+0000 7fbe39c00640 1 -- 172.21.15.12:0/3448995451 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe4000ec00 con 0x55fe072a2810 2024-03-23T23:08:38.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.760+0000 7fbe39c00640 1 -- 172.21.15.12:0/3448995451 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4002f780 con 0x55fe072a2810 2024-03-23T23:08:38.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.761+0000 7fbe52d96740 1 -- 172.21.15.12:0/3448995451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a2810 msgr2=0x55fe072a1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.761+0000 7fbe52d96740 1 --2- 172.21.15.12:0/3448995451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a2810 0x55fe072a1570 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fbe40009ab0 tx=0x7fbe40000bc0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.761+0000 7fbe52d96740 1 -- 172.21.15.12:0/3448995451 shutdown_connections 2024-03-23T23:08:38.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.761+0000 7fbe52d96740 1 --2- 172.21.15.12:0/3448995451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fe072a43a0 0x55fe072a6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.761+0000 7fbe52d96740 1 --2- 172.21.15.12:0/3448995451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe072a39e0 0x55fe072b3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.761+0000 7fbe52d96740 1 --2- 172.21.15.12:0/3448995451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a2810 0x55fe072a1570 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.761+0000 7fbe52d96740 1 -- 172.21.15.12:0/3448995451 >> 172.21.15.12:0/3448995451 conn(0x55fe0729dc00 msgr2=0x55fe072a0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:38.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.761+0000 7fbe52d96740 1 -- 172.21.15.12:0/3448995451 shutdown_connections 2024-03-23T23:08:38.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.761+0000 7fbe52d96740 1 -- 172.21.15.12:0/3448995451 wait complete. 2024-03-23T23:08:38.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.761+0000 7fbe52d96740 1 -- start start 2024-03-23T23:08:38.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.761+0000 7fbe52d96740 1 Processor -- start 2024-03-23T23:08:38.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.762+0000 7fbe52d96740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe072a2810 0x55fe072b1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.762+0000 7fbe52d96740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a39e0 0x55fe072b22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.762+0000 7fbe52d96740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fe072a43a0 0x55fe072b3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.762+0000 7fbe52d96740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe481a9310 con 0x55fe072a43a0 2024-03-23T23:08:38.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.762+0000 7fbe52d96740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe481aa610 con 0x55fe072a39e0 2024-03-23T23:08:38.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.762+0000 7fbe52d96740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe481a9950 con 0x55fe072a2810 2024-03-23T23:08:38.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.762+0000 7fbe52d96740 10 client.0 _ll_register_callbacks cb 0x7fbe34009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:38.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.762+0000 7fbe4cc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a39e0 0x55fe072b22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.762+0000 7fbe4cc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a39e0 0x55fe072b22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57466/0 (socket says 172.21.15.12:57466) 2024-03-23T23:08:38.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.762+0000 7fbe4cc00640 1 -- 172.21.15.12:0/611818296 learned_addr learned my addr 172.21.15.12:0/611818296 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:38.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.762+0000 7fbe4e000640 1 --2- 172.21.15.12:0/611818296 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe072a2810 0x55fe072b1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.762+0000 7fbe4e000640 1 --2- 172.21.15.12:0/611818296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fe072a43a0 0x55fe072b3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.763+0000 7fbe4cc00640 1 -- 172.21.15.12:0/611818296 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe072a2810 msgr2=0x55fe072b1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:38.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.763+0000 7fbe4cc00640 1 --2- 172.21.15.12:0/611818296 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe072a2810 0x55fe072b1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.763+0000 7fbe4cc00640 1 -- 172.21.15.12:0/611818296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fe072a43a0 msgr2=0x55fe072b3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:38.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.763+0000 7fbe4cc00640 1 --2- 172.21.15.12:0/611818296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fe072a43a0 0x55fe072b3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.763+0000 7fbe4cc00640 1 -- 172.21.15.12:0/611818296 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe072b3870 con 0x55fe072a39e0 2024-03-23T23:08:38.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.763+0000 7fbe4e000640 1 --2- 172.21.15.12:0/611818296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fe072a43a0 0x55fe072b3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:08:38.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.763+0000 7fbe4cc00640 1 --2- 172.21.15.12:0/611818296 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a39e0 0x55fe072b22c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fbe3402bc70 tx=0x7fbe34003fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4deae5583943a9f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:38.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.763+0000 7fbe2ea00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:08:38.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.763+0000 7fbe2ea00640 1 -- 172.21.15.12:0/611818296 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe34005d50 con 0x55fe072a39e0 2024-03-23T23:08:38.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.763+0000 7fbe2ea00640 1 -- 172.21.15.12:0/611818296 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fbe34003c00 con 0x55fe072a39e0 2024-03-23T23:08:38.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.763+0000 7fbe52d96740 1 -- 172.21.15.12:0/611818296 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbe4000ec00 con 0x55fe072a39e0 2024-03-23T23:08:38.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.763+0000 7fbe2ea00640 1 -- 172.21.15.12:0/611818296 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe34004a80 con 0x55fe072a39e0 2024-03-23T23:08:38.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.763+0000 7fbe52d96740 10 client.27248 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:38.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.763+0000 7fbe52d96740 1 -- 172.21.15.12:0/611818296 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55fe07314430 con 0x55fe072a39e0 2024-03-23T23:08:38.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.764+0000 7fbe2ea00640 1 -- 172.21.15.12:0/611818296 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe4000ec00 con 0x55fe072a39e0 2024-03-23T23:08:38.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.764+0000 7fbe2ea00640 1 -- 172.21.15.12:0/611818296 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fbe3405f030 con 0x55fe072a39e0 2024-03-23T23:08:38.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.764+0000 7fbe52d96740 10 client.27248 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:38.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.764+0000 7fbe52d96740 10 client.27248 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:08:38.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.764+0000 7fbe52d96740 20 client.27248 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:38.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.764+0000 7fbe52d96740 1 --2- 172.21.15.12:0/611818296 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fe07314d80 0x55fe073171a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:38.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.764+0000 7fbe52d96740 4 client.27248 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:08:38.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.764+0000 7fbe52d96740 1 -- 172.21.15.12:0/611818296 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55fe0720eb50 con 0x55fe07314d80 2024-03-23T23:08:38.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.764+0000 7fbe4e000640 1 --2- 172.21.15.12:0/611818296 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fe07314d80 0x55fe073171a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:38.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.765+0000 7fbe4e000640 1 --2- 172.21.15.12:0/611818296 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fe07314d80 0x55fe073171a0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6aa3134ec60675d1 server_cookie=a47021c369b3413a in_seq=0 out_seq=0 2024-03-23T23:08:38.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.765+0000 7fbe2ea00640 10 client.27248 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:38.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.765+0000 7fbe2ea00640 1 -- 172.21.15.12:0/611818296 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55fe0720eb50 con 0x55fe07314d80 2024-03-23T23:08:38.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.765+0000 7fbe2ea00640 10 client.27248 handle_command_reply: tid=0 2024-03-23T23:08:38.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.765+0000 7fbe52d96740 1 client.27248 shutdown 2024-03-23T23:08:38.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.765+0000 7fbe52d96740 1 -- 172.21.15.12:0/611818296 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a39e0 msgr2=0x55fe072b22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.765+0000 7fbe52d96740 1 --2- 172.21.15.12:0/611818296 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a39e0 0x55fe072b22c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fbe3402bc70 tx=0x7fbe34003fc0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.765+0000 7fbe52d96740 1 -- 172.21.15.12:0/611818296 shutdown_connections 2024-03-23T23:08:38.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.765+0000 7fbe52d96740 1 --2- 172.21.15.12:0/611818296 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fe07314d80 0x55fe073171a0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.765+0000 7fbe52d96740 1 --2- 172.21.15.12:0/611818296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fe072a43a0 0x55fe072b3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.765+0000 7fbe52d96740 1 --2- 172.21.15.12:0/611818296 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe072a39e0 0x55fe072b22c0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.765+0000 7fbe2ea00640 0 client.27248 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:38.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.765+0000 7fbe52d96740 1 --2- 172.21.15.12:0/611818296 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe072a2810 0x55fe072b1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.765+0000 7fbe52d96740 1 -- 172.21.15.12:0/611818296 >> 172.21.15.12:0/611818296 conn(0x55fe0729dc00 msgr2=0x55fe072a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:38.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.765+0000 7fbe52d96740 1 -- 172.21.15.12:0/611818296 shutdown_connections 2024-03-23T23:08:38.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.765+0000 7fbe52d96740 1 -- 172.21.15.12:0/611818296 wait complete. 2024-03-23T23:08:38.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.765+0000 7fbe52d96740 20 client.27248 trim_cache size 0 max 16384 2024-03-23T23:08:38.770 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:08:38.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.766+0000 7fbe50000640 1 -- 172.21.15.12:0/2183915642 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbe300a2880 msgr2=0x7fbe300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.766+0000 7fbe50000640 1 --2- 172.21.15.12:0/2183915642 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbe300a2880 0x7fbe300a4d40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c0ccc90 tx=0x7fbe3c00cf90 comp rx=0 tx=0).stop 2024-03-23T23:08:38.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.766+0000 7fbe50000640 1 -- 172.21.15.12:0/2183915642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe481038c0 msgr2=0x7fbe48199bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:38.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.766+0000 7fbe50000640 1 --2- 172.21.15.12:0/2183915642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe481038c0 0x7fbe48199bb0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c00ebc0 tx=0x7fbe3c00c6a0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.766+0000 7fbe50000640 1 -- 172.21.15.12:0/2183915642 shutdown_connections 2024-03-23T23:08:38.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.766+0000 7fbe50000640 1 --2- 172.21.15.12:0/2183915642 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbe300a2880 0x7fbe300a4d40 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.766+0000 7fbe50000640 1 --2- 172.21.15.12:0/2183915642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe48105470 0x7fbe481a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.766+0000 7fbe50000640 1 --2- 172.21.15.12:0/2183915642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe48104ab0 0x7fbe4819a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.766+0000 7fbe50000640 1 --2- 172.21.15.12:0/2183915642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe481038c0 0x7fbe48199bb0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:38.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.766+0000 7fbe50000640 1 -- 172.21.15.12:0/2183915642 >> 172.21.15.12:0/2183915642 conn(0x7fbe480fee50 msgr2=0x7fbe48107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:38.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.766+0000 7fbe50000640 1 -- 172.21.15.12:0/2183915642 shutdown_connections 2024-03-23T23:08:38.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:38.766+0000 7fbe50000640 1 -- 172.21.15.12:0/2183915642 wait complete. 2024-03-23T23:08:38.781 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:38 smithi012 ceph-mon[31695]: pgmap v595: 97 pgs: 97 active+clean; 4.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:08:38.781 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:38 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/398804588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:08:38.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:38 smithi173 ceph-mon[36116]: pgmap v595: 97 pgs: 97 active+clean; 4.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:08:38.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:38 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/398804588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:08:38.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:38 smithi203 ceph-mon[36693]: pgmap v595: 97 pgs: 97 active+clean; 4.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:08:38.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:38 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/398804588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:08:39.785 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-23T23:08:39.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:08:39.786 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-23T23:08:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.028+0000 7f0e24000640 1 -- 172.21.15.12:0/1508354788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e1c1006e0 msgr2=0x7f0e1c100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.028+0000 7f0e24000640 1 --2- 172.21.15.12:0/1508354788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e1c1006e0 0x7f0e1c100b60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0e08009900 tx=0x7f0e0802bc50 comp rx=0 tx=0).stop 2024-03-23T23:08:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.028+0000 7f0e24000640 1 -- 172.21.15.12:0/1508354788 shutdown_connections 2024-03-23T23:08:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.028+0000 7f0e24000640 1 --2- 172.21.15.12:0/1508354788 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e1c1010a0 0x7f0e1c101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.028+0000 7f0e24000640 1 --2- 172.21.15.12:0/1508354788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e1c1006e0 0x7f0e1c100b60 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.028+0000 7f0e24000640 1 --2- 172.21.15.12:0/1508354788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e1c0ff980 0x7f0e1c0ffda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.028+0000 7f0e24000640 1 -- 172.21.15.12:0/1508354788 >> 172.21.15.12:0/1508354788 conn(0x7f0e1c0fb290 msgr2=0x7f0e1c0fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.028+0000 7f0e24000640 1 -- 172.21.15.12:0/1508354788 shutdown_connections 2024-03-23T23:08:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.028+0000 7f0e24000640 1 -- 172.21.15.12:0/1508354788 wait complete. 2024-03-23T23:08:40.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.029+0000 7f0e24000640 1 Processor -- start 2024-03-23T23:08:40.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.029+0000 7f0e24000640 1 -- start start 2024-03-23T23:08:40.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.029+0000 7f0e24000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e1c0ff980 0x7f0e1c199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.030+0000 7f0e24000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e1c1006e0 0x7f0e1c19a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.030+0000 7f0e24000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e1c1010a0 0x7f0e1c1a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.030+0000 7f0e24000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c10f540 con 0x7f0e1c1010a0 2024-03-23T23:08:40.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.030+0000 7f0e24000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c10f3c0 con 0x7f0e1c1006e0 2024-03-23T23:08:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.030+0000 7f0e24000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c10f6c0 con 0x7f0e1c0ff980 2024-03-23T23:08:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.030+0000 7f0e21600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e1c0ff980 0x7f0e1c199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.030+0000 7f0e20c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e1c1006e0 0x7f0e1c19a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.030+0000 7f0e21600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e1c0ff980 0x7f0e1c199d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:42430/0 (socket says 172.21.15.12:42430) 2024-03-23T23:08:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.030+0000 7f0e21600640 1 -- 172.21.15.12:0/653576078 learned_addr learned my addr 172.21.15.12:0/653576078 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.030+0000 7f0e22000640 1 --2- 172.21.15.12:0/653576078 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e1c1010a0 0x7f0e1c1a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.030+0000 7f0e20c00640 1 -- 172.21.15.12:0/653576078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e1c0ff980 msgr2=0x7f0e1c199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.030+0000 7f0e20c00640 1 --2- 172.21.15.12:0/653576078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e1c0ff980 0x7f0e1c199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.030+0000 7f0e20c00640 1 -- 172.21.15.12:0/653576078 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e1c1010a0 msgr2=0x7f0e1c1a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.030+0000 7f0e20c00640 1 --2- 172.21.15.12:0/653576078 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e1c1010a0 0x7f0e1c1a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.030+0000 7f0e20c00640 1 -- 172.21.15.12:0/653576078 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1c1a1930 con 0x7f0e1c1006e0 2024-03-23T23:08:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.030+0000 7f0e21600640 1 --2- 172.21.15.12:0/653576078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e1c0ff980 0x7f0e1c199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.031+0000 7f0e20c00640 1 --2- 172.21.15.12:0/653576078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e1c1006e0 0x7f0e1c19a2b0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0e08036040 tx=0x7f0e08005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.031+0000 7f0e1a000640 1 -- 172.21.15.12:0/653576078 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e0804f070 con 0x7f0e1c1006e0 2024-03-23T23:08:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.031+0000 7f0e1a000640 1 -- 172.21.15.12:0/653576078 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e08005000 con 0x7f0e1c1006e0 2024-03-23T23:08:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.031+0000 7f0e24000640 1 -- 172.21.15.12:0/653576078 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e1c102950 con 0x7f0e1c1006e0 2024-03-23T23:08:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.031+0000 7f0e1a000640 1 -- 172.21.15.12:0/653576078 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e08003950 con 0x7f0e1c1006e0 2024-03-23T23:08:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.031+0000 7f0e24000640 1 -- 172.21.15.12:0/653576078 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e1c102ec0 con 0x7f0e1c1006e0 2024-03-23T23:08:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.034+0000 7f0e1a000640 1 -- 172.21.15.12:0/653576078 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0e08052070 con 0x7f0e1c1006e0 2024-03-23T23:08:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.035+0000 7f0e26ded740 1 Processor -- start 2024-03-23T23:08:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.035+0000 7f0e26ded740 1 -- start start 2024-03-23T23:08:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.035+0000 7f0e26ded740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654a84f80a0 0x5654a84f84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.035+0000 7f0e26ded740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84f8a90 0x5654a84fd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.035+0000 7f0e26ded740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654a84fdc00 0x5654a8500090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.035+0000 7f0e26ded740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654a845deb0 con 0x5654a84f80a0 2024-03-23T23:08:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.035+0000 7f0e26ded740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654a8406a60 con 0x5654a84f8a90 2024-03-23T23:08:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.035+0000 7f0e26ded740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654a8459f50 con 0x5654a84fdc00 2024-03-23T23:08:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.035+0000 7f0e22000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654a84f80a0 0x5654a84f84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.036+0000 7f0e22000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654a84f80a0 0x5654a84f84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33002/0 (socket says 172.21.15.12:33002) 2024-03-23T23:08:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.036+0000 7f0e22000640 1 -- 172.21.15.12:0/3780517567 learned_addr learned my addr 172.21.15.12:0/3780517567 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.035+0000 7f0e21600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84f8a90 0x5654a84fd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.036+0000 7f0e20c00640 1 --2- 172.21.15.12:0/3780517567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654a84fdc00 0x5654a8500090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.036+0000 7f0e22000640 1 -- 172.21.15.12:0/3780517567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654a84fdc00 msgr2=0x5654a8500090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.036+0000 7f0e22000640 1 --2- 172.21.15.12:0/3780517567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654a84fdc00 0x5654a8500090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.036+0000 7f0e22000640 1 -- 172.21.15.12:0/3780517567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84f8a90 msgr2=0x5654a84fd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.036+0000 7f0e22000640 1 --2- 172.21.15.12:0/3780517567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84f8a90 0x5654a84fd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.036+0000 7f0e22000640 1 -- 172.21.15.12:0/3780517567 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654a823a7e0 con 0x5654a84f80a0 2024-03-23T23:08:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.036+0000 7f0e21600640 1 --2- 172.21.15.12:0/3780517567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84f8a90 0x5654a84fd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.036+0000 7f0e1a000640 1 --2- 172.21.15.12:0/653576078 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0e040a2880 0x7f0e040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.036+0000 7f0e1a000640 1 -- 172.21.15.12:0/653576078 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0e080f8360 con 0x7f0e1c1006e0 2024-03-23T23:08:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.036+0000 7f0e22000640 1 --2- 172.21.15.12:0/3780517567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654a84f80a0 0x5654a84f84c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0e14010370 tx=0x7f0e14010830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=746b00b9684ee1c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.037+0000 7f0e21600640 1 --2- 172.21.15.12:0/653576078 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0e040a2880 0x7f0e040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.037+0000 7f0e0e600640 1 -- 172.21.15.12:0/3780517567 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e14016070 con 0x5654a84f80a0 2024-03-23T23:08:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.037+0000 7f0e0e600640 1 -- 172.21.15.12:0/3780517567 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e14004d20 con 0x5654a84f80a0 2024-03-23T23:08:40.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.037+0000 7f0e0e600640 1 -- 172.21.15.12:0/3780517567 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e14005480 con 0x5654a84f80a0 2024-03-23T23:08:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.038+0000 7f0e26ded740 1 -- 172.21.15.12:0/3780517567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654a84f80a0 msgr2=0x5654a84f84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.038+0000 7f0e21600640 1 --2- 172.21.15.12:0/653576078 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0e040a2880 0x7f0e040a4d40 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x5654a84f8e90 tx=0x7f0e10005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.038+0000 7f0e26ded740 1 --2- 172.21.15.12:0/3780517567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654a84f80a0 0x5654a84f84c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0e14010370 tx=0x7f0e14010830 comp rx=0 tx=0).stop 2024-03-23T23:08:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.038+0000 7f0e26ded740 1 -- 172.21.15.12:0/3780517567 shutdown_connections 2024-03-23T23:08:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.038+0000 7f0e26ded740 1 --2- 172.21.15.12:0/3780517567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654a84fdc00 0x5654a8500090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.038+0000 7f0e26ded740 1 --2- 172.21.15.12:0/3780517567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84f8a90 0x5654a84fd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.038+0000 7f0e26ded740 1 --2- 172.21.15.12:0/3780517567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654a84f80a0 0x5654a84f84c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.038+0000 7f0e26ded740 1 -- 172.21.15.12:0/3780517567 >> 172.21.15.12:0/3780517567 conn(0x5654a84f3610 msgr2=0x5654a84f3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.038+0000 7f0e26ded740 1 -- 172.21.15.12:0/3780517567 shutdown_connections 2024-03-23T23:08:40.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.038+0000 7f0e26ded740 1 -- 172.21.15.12:0/3780517567 wait complete. 2024-03-23T23:08:40.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.038+0000 7f0e26ded740 1 -- start start 2024-03-23T23:08:40.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.039+0000 7f0e26ded740 1 Processor -- start 2024-03-23T23:08:40.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.039+0000 7f0e26ded740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654a84f80a0 0x5654a85022f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.039+0000 7f0e26ded740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84f8a90 0x5654a8502830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.039+0000 7f0e26ded740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654a84fdc00 0x5654a8509870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.039+0000 7f0e26ded740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654a8500bb0 con 0x5654a84f80a0 2024-03-23T23:08:40.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.039+0000 7f0e26ded740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654a8419eb0 con 0x5654a84f8a90 2024-03-23T23:08:40.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.039+0000 7f0e26ded740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654a8500d30 con 0x5654a84fdc00 2024-03-23T23:08:40.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.039+0000 7f0e20c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84f8a90 0x5654a8502830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.039+0000 7f0e26ded740 10 client.0 _ll_register_callbacks cb 0x5654a840e150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:40.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.039+0000 7f0e20c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84f8a90 0x5654a8502830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57500/0 (socket says 172.21.15.12:57500) 2024-03-23T23:08:40.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.039+0000 7f0e21600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654a84f80a0 0x5654a85022f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.039+0000 7f0e20c00640 1 -- 172.21.15.12:0/2024944553 learned_addr learned my addr 172.21.15.12:0/2024944553 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:40.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.039+0000 7f0e22000640 1 --2- 172.21.15.12:0/2024944553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654a84fdc00 0x5654a8509870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.040+0000 7f0e20c00640 1 -- 172.21.15.12:0/2024944553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654a84fdc00 msgr2=0x5654a8509870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:40.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.040+0000 7f0e20c00640 1 --2- 172.21.15.12:0/2024944553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654a84fdc00 0x5654a8509870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.040+0000 7f0e20c00640 1 -- 172.21.15.12:0/2024944553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654a84f80a0 msgr2=0x5654a85022f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:40.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.040+0000 7f0e20c00640 1 --2- 172.21.15.12:0/2024944553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654a84f80a0 0x5654a85022f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.040+0000 7f0e20c00640 1 -- 172.21.15.12:0/2024944553 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5654a829af70 con 0x5654a84f8a90 2024-03-23T23:08:40.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.040+0000 7f0e20c00640 1 --2- 172.21.15.12:0/2024944553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84f8a90 0x5654a8502830 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f0e080c16a0 tx=0x7f0e080c2d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad4e6b1539797207 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:40.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.040+0000 7f0e02a00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:08:40.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.040+0000 7f0e02a00640 1 -- 172.21.15.12:0/2024944553 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e080c4b30 con 0x5654a84f8a90 2024-03-23T23:08:40.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.040+0000 7f0e26ded740 1 -- 172.21.15.12:0/2024944553 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0e14004d20 con 0x5654a84f8a90 2024-03-23T23:08:40.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.040+0000 7f0e02a00640 1 -- 172.21.15.12:0/2024944553 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0e080ccfc0 con 0x5654a84f8a90 2024-03-23T23:08:40.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.041+0000 7f0e02a00640 1 -- 172.21.15.12:0/2024944553 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e080c2ea0 con 0x5654a84f8a90 2024-03-23T23:08:40.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.041+0000 7f0e26ded740 10 client.27260 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:40.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.041+0000 7f0e26ded740 1 -- 172.21.15.12:0/2024944553 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5654a8132170 con 0x5654a84f8a90 2024-03-23T23:08:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.041+0000 7f0e02a00640 1 -- 172.21.15.12:0/2024944553 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e14004d20 con 0x5654a84f8a90 2024-03-23T23:08:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.041+0000 7f0e02a00640 1 -- 172.21.15.12:0/2024944553 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f0e08049030 con 0x5654a84f8a90 2024-03-23T23:08:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.041+0000 7f0e26ded740 10 client.27260 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.041+0000 7f0e26ded740 10 client.27260 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:08:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.041+0000 7f0e26ded740 20 client.27260 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.041+0000 7f0e26ded740 1 --2- 172.21.15.12:0/2024944553 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5654a850e0a0 0x5654a8510500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.041+0000 7f0e26ded740 4 client.27260 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:08:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.041+0000 7f0e26ded740 1 -- 172.21.15.12:0/2024944553 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5654a846b3f0 con 0x5654a850e0a0 2024-03-23T23:08:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.041+0000 7f0e21600640 1 --2- 172.21.15.12:0/2024944553 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5654a850e0a0 0x5654a8510500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.042+0000 7f0e21600640 1 --2- 172.21.15.12:0/2024944553 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5654a850e0a0 0x5654a8510500 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9cdd4650931dff9 server_cookie=cbf8475a024b0fba in_seq=0 out_seq=0 2024-03-23T23:08:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.042+0000 7f0e02a00640 10 client.27260 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.043+0000 7f0e02a00640 1 -- 172.21.15.12:0/2024944553 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5654a846b3f0 con 0x5654a850e0a0 2024-03-23T23:08:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.043+0000 7f0e02a00640 10 client.27260 handle_command_reply: tid=0 2024-03-23T23:08:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.043+0000 7f0e26ded740 1 client.27260 shutdown 2024-03-23T23:08:40.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.043+0000 7f0e26ded740 1 -- 172.21.15.12:0/2024944553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84f8a90 msgr2=0x5654a8502830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:40.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.043+0000 7f0e26ded740 1 --2- 172.21.15.12:0/2024944553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84f8a90 0x5654a8502830 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f0e080c16a0 tx=0x7f0e080c2d70 comp rx=0 tx=0).stop 2024-03-23T23:08:40.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.043+0000 7f0e26ded740 1 -- 172.21.15.12:0/2024944553 shutdown_connections 2024-03-23T23:08:40.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.043+0000 7f0e26ded740 1 --2- 172.21.15.12:0/2024944553 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5654a850e0a0 0x5654a8510500 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.043+0000 7f0e26ded740 1 --2- 172.21.15.12:0/2024944553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654a84fdc00 0x5654a8509870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.043+0000 7f0e02a00640 0 client.27260 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:40.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.043+0000 7f0e26ded740 1 --2- 172.21.15.12:0/2024944553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84f8a90 0x5654a8502830 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.043+0000 7f0e26ded740 1 --2- 172.21.15.12:0/2024944553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654a84f80a0 0x5654a85022f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.043+0000 7f0e26ded740 1 -- 172.21.15.12:0/2024944553 >> 172.21.15.12:0/2024944553 conn(0x5654a84f3610 msgr2=0x5654a84f2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:40.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.043+0000 7f0e26ded740 1 -- 172.21.15.12:0/2024944553 shutdown_connections 2024-03-23T23:08:40.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.044+0000 7f0e26ded740 1 -- 172.21.15.12:0/2024944553 wait complete. 2024-03-23T23:08:40.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.044+0000 7f0e26ded740 20 client.27260 trim_cache size 0 max 16384 2024-03-23T23:08:40.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.061+0000 7f0e26ded740 1 Processor -- start 2024-03-23T23:08:40.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.061+0000 7f0e26ded740 1 -- start start 2024-03-23T23:08:40.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.062+0000 7f0e26ded740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654a84f80a0 0x5654a84f6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.062+0000 7f0e26ded740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654a84f8a90 0x5654a8509830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.062+0000 7f0e26ded740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84fdc00 0x5654a8514e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.062+0000 7f0e26ded740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c1a8cb0 con 0x5654a84f80a0 2024-03-23T23:08:40.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.062+0000 7f0e26ded740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c1a67b0 con 0x5654a84fdc00 2024-03-23T23:08:40.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.062+0000 7f0e26ded740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c1a4850 con 0x5654a84f8a90 2024-03-23T23:08:40.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.062+0000 7f0e21600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654a84f80a0 0x5654a84f6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.062+0000 7f0e22000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84fdc00 0x5654a8514e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.062+0000 7f0e20c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654a84f8a90 0x5654a8509830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.062+0000 7f0e21600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654a84f80a0 0x5654a84f6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33016/0 (socket says 172.21.15.12:33016) 2024-03-23T23:08:40.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.062+0000 7f0e21600640 1 -- 172.21.15.12:0/182501744 learned_addr learned my addr 172.21.15.12:0/182501744 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:40.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.062+0000 7f0e22000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84fdc00 0x5654a8514e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57506/0 (socket says 172.21.15.12:57506) 2024-03-23T23:08:40.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.062+0000 7f0e21600640 1 -- 172.21.15.12:0/182501744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654a84f8a90 msgr2=0x5654a8509830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.062+0000 7f0e21600640 1 --2- 172.21.15.12:0/182501744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654a84f8a90 0x5654a8509830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.062+0000 7f0e21600640 1 -- 172.21.15.12:0/182501744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84fdc00 msgr2=0x5654a8514e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.063+0000 7f0e21600640 1 --2- 172.21.15.12:0/182501744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84fdc00 0x5654a8514e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.063+0000 7f0e21600640 1 -- 172.21.15.12:0/182501744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654a8510b80 con 0x5654a84f80a0 2024-03-23T23:08:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.063+0000 7f0e20c00640 1 --2- 172.21.15.12:0/182501744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654a84f8a90 0x5654a8509830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.063+0000 7f0e22000640 1 --2- 172.21.15.12:0/182501744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84fdc00 0x5654a8514e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.063+0000 7f0e21600640 1 --2- 172.21.15.12:0/182501744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654a84f80a0 0x5654a84f6ee0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0e10009620 tx=0x7f0e1001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ced455b3713f0508 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:40.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.063+0000 7f0e0dc00640 1 -- 172.21.15.12:0/182501744 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e10030070 con 0x5654a84f80a0 2024-03-23T23:08:40.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.063+0000 7f0e0dc00640 1 -- 172.21.15.12:0/182501744 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e1000fde0 con 0x5654a84f80a0 2024-03-23T23:08:40.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.063+0000 7f0e0dc00640 1 -- 172.21.15.12:0/182501744 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e10003a90 con 0x5654a84f80a0 2024-03-23T23:08:40.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.064+0000 7f0e26ded740 1 -- 172.21.15.12:0/182501744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654a84f80a0 msgr2=0x5654a84f6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:40.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.064+0000 7f0e26ded740 1 --2- 172.21.15.12:0/182501744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654a84f80a0 0x5654a84f6ee0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0e10009620 tx=0x7f0e1001b000 comp rx=0 tx=0).stop 2024-03-23T23:08:40.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.064+0000 7f0e26ded740 1 -- 172.21.15.12:0/182501744 shutdown_connections 2024-03-23T23:08:40.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.064+0000 7f0e26ded740 1 --2- 172.21.15.12:0/182501744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84fdc00 0x5654a8514e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.064+0000 7f0e26ded740 1 --2- 172.21.15.12:0/182501744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654a84f8a90 0x5654a8509830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.064+0000 7f0e26ded740 1 --2- 172.21.15.12:0/182501744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654a84f80a0 0x5654a84f6ee0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.064+0000 7f0e26ded740 1 -- 172.21.15.12:0/182501744 >> 172.21.15.12:0/182501744 conn(0x5654a84f3610 msgr2=0x5654a84f2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:40.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.064+0000 7f0e26ded740 1 -- 172.21.15.12:0/182501744 shutdown_connections 2024-03-23T23:08:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.064+0000 7f0e26ded740 1 -- 172.21.15.12:0/182501744 wait complete. 2024-03-23T23:08:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.064+0000 7f0e26ded740 1 -- start start 2024-03-23T23:08:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.065+0000 7f0e26ded740 1 Processor -- start 2024-03-23T23:08:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.065+0000 7f0e26ded740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654a84f80a0 0x5654a85097c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.065+0000 7f0e26ded740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654a84f8a90 0x5654a8507dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.065+0000 7f0e26ded740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84fdc00 0x5654a85083b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.065+0000 7f0e26ded740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c1a4e50 con 0x5654a84f8a90 2024-03-23T23:08:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.065+0000 7f0e26ded740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c1a6150 con 0x5654a84fdc00 2024-03-23T23:08:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.065+0000 7f0e26ded740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c1a5490 con 0x5654a84f80a0 2024-03-23T23:08:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.065+0000 7f0e26ded740 10 client.0 _ll_register_callbacks cb 0x7f0e08005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.065+0000 7f0e21600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654a84f80a0 0x5654a85097c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.065+0000 7f0e22000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84fdc00 0x5654a85083b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.065+0000 7f0e21600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654a84f80a0 0x5654a85097c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:42466/0 (socket says 172.21.15.12:42466) 2024-03-23T23:08:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.065+0000 7f0e21600640 1 -- 172.21.15.12:0/2739818864 learned_addr learned my addr 172.21.15.12:0/2739818864 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.065+0000 7f0e22000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84fdc00 0x5654a85083b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57520/0 (socket says 172.21.15.12:57520) 2024-03-23T23:08:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.065+0000 7f0e20c00640 1 --2- 172.21.15.12:0/2739818864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654a84f8a90 0x5654a8507dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.066+0000 7f0e21600640 1 -- 172.21.15.12:0/2739818864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84fdc00 msgr2=0x5654a85083b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.066+0000 7f0e21600640 1 --2- 172.21.15.12:0/2739818864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84fdc00 0x5654a85083b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.066+0000 7f0e21600640 1 -- 172.21.15.12:0/2739818864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654a84f8a90 msgr2=0x5654a8507dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.066+0000 7f0e21600640 1 --2- 172.21.15.12:0/2739818864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654a84f8a90 0x5654a8507dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.066+0000 7f0e21600640 1 -- 172.21.15.12:0/2739818864 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5654a85699d0 con 0x5654a84f80a0 2024-03-23T23:08:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.066+0000 7f0e21600640 1 --2- 172.21.15.12:0/2739818864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654a84f80a0 0x5654a85097c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0e1002e040 tx=0x7f0e10005760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c47e9b571619c631 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.066+0000 7f0e02a00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:08:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.066+0000 7f0e02a00640 1 -- 172.21.15.12:0/2739818864 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e10030040 con 0x5654a84f80a0 2024-03-23T23:08:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.067+0000 7f0e02a00640 1 -- 172.21.15.12:0/2739818864 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0e1002b880 con 0x5654a84f80a0 2024-03-23T23:08:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.067+0000 7f0e26ded740 1 -- 172.21.15.12:0/2739818864 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0e1000fde0 con 0x5654a84f80a0 2024-03-23T23:08:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.067+0000 7f0e02a00640 1 -- 172.21.15.12:0/2739818864 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e1002ba80 con 0x5654a84f80a0 2024-03-23T23:08:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.067+0000 7f0e26ded740 10 client.27238 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.067+0000 7f0e26ded740 1 -- 172.21.15.12:0/2739818864 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5654a856ab00 con 0x5654a84f80a0 2024-03-23T23:08:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.067+0000 7f0e02a00640 1 -- 172.21.15.12:0/2739818864 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e1000fde0 con 0x5654a84f80a0 2024-03-23T23:08:40.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.067+0000 7f0e02a00640 1 -- 172.21.15.12:0/2739818864 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f0e10033030 con 0x5654a84f80a0 2024-03-23T23:08:40.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.067+0000 7f0e26ded740 10 client.27238 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:40.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.067+0000 7f0e26ded740 10 client.27238 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:08:40.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.067+0000 7f0e26ded740 20 client.27238 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:40.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.067+0000 7f0e26ded740 1 --2- 172.21.15.12:0/2739818864 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5654a856b3b0 0x5654a856d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.068+0000 7f0e26ded740 4 client.27238 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-23T23:08:40.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.068+0000 7f0e26ded740 1 -- 172.21.15.12:0/2739818864 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5654a8465740 con 0x5654a856b3b0 2024-03-23T23:08:40.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.068+0000 7f0e20c00640 1 --2- 172.21.15.12:0/2739818864 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5654a856b3b0 0x5654a856d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.068+0000 7f0e20c00640 1 --2- 172.21.15.12:0/2739818864 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5654a856b3b0 0x5654a856d7d0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85a4915957f4e35a server_cookie=398668d1687e3f5d in_seq=0 out_seq=0 2024-03-23T23:08:40.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.068+0000 7f0e02a00640 10 client.27238 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:40.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.070+0000 7f0e02a00640 1 -- 172.21.15.12:0/2739818864 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5654a8465740 con 0x5654a856b3b0 2024-03-23T23:08:40.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.070+0000 7f0e02a00640 10 client.27238 handle_command_reply: tid=0 2024-03-23T23:08:40.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.070+0000 7f0e26ded740 1 client.27238 shutdown 2024-03-23T23:08:40.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.070+0000 7f0e26ded740 1 -- 172.21.15.12:0/2739818864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654a84f80a0 msgr2=0x5654a85097c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:40.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.070+0000 7f0e26ded740 1 --2- 172.21.15.12:0/2739818864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654a84f80a0 0x5654a85097c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0e1002e040 tx=0x7f0e10005760 comp rx=0 tx=0).stop 2024-03-23T23:08:40.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.070+0000 7f0e26ded740 1 -- 172.21.15.12:0/2739818864 shutdown_connections 2024-03-23T23:08:40.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.070+0000 7f0e26ded740 1 --2- 172.21.15.12:0/2739818864 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5654a856b3b0 0x5654a856d7d0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.070+0000 7f0e26ded740 1 --2- 172.21.15.12:0/2739818864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654a84fdc00 0x5654a85083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.070+0000 7f0e26ded740 1 --2- 172.21.15.12:0/2739818864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654a84f8a90 0x5654a8507dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.070+0000 7f0e26ded740 1 --2- 172.21.15.12:0/2739818864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654a84f80a0 0x5654a85097c0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.070+0000 7f0e02a00640 0 client.27238 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:40.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.070+0000 7f0e26ded740 1 -- 172.21.15.12:0/2739818864 >> 172.21.15.12:0/2739818864 conn(0x5654a84f3610 msgr2=0x5654a84f9320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:40.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.070+0000 7f0e26ded740 1 -- 172.21.15.12:0/2739818864 shutdown_connections 2024-03-23T23:08:40.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.070+0000 7f0e26ded740 1 -- 172.21.15.12:0/2739818864 wait complete. 2024-03-23T23:08:40.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.070+0000 7f0e26ded740 20 client.27238 trim_cache size 0 max 16384 2024-03-23T23:08:40.073 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:08:40.073 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-03-23T23:08:40.073 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "5ba06596-68c3-4642-bae0-1cce971052f0", 2024-03-23T23:08:40.073 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-03-23T23:08:40.074 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:08:40.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.071+0000 7f0e24000640 1 -- 172.21.15.12:0/653576078 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0e040a2880 msgr2=0x7f0e040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:40.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.071+0000 7f0e24000640 1 --2- 172.21.15.12:0/653576078 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0e040a2880 0x7f0e040a4d40 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x5654a84f8e90 tx=0x7f0e10005f90 comp rx=0 tx=0).stop 2024-03-23T23:08:40.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.071+0000 7f0e24000640 1 -- 172.21.15.12:0/653576078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e1c1006e0 msgr2=0x7f0e1c19a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:40.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.071+0000 7f0e24000640 1 --2- 172.21.15.12:0/653576078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e1c1006e0 0x7f0e1c19a2b0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0e08036040 tx=0x7f0e08005d50 comp rx=0 tx=0).stop 2024-03-23T23:08:40.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.071+0000 7f0e24000640 1 -- 172.21.15.12:0/653576078 shutdown_connections 2024-03-23T23:08:40.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.071+0000 7f0e24000640 1 --2- 172.21.15.12:0/653576078 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0e040a2880 0x7f0e040a4d40 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.071+0000 7f0e24000640 1 --2- 172.21.15.12:0/653576078 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e1c1010a0 0x7f0e1c1a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.071+0000 7f0e24000640 1 --2- 172.21.15.12:0/653576078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e1c1006e0 0x7f0e1c19a2b0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.071+0000 7f0e24000640 1 --2- 172.21.15.12:0/653576078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e1c0ff980 0x7f0e1c199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.071+0000 7f0e24000640 1 -- 172.21.15.12:0/653576078 >> 172.21.15.12:0/653576078 conn(0x7f0e1c0fb290 msgr2=0x7f0e1c0fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:40.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.071+0000 7f0e24000640 1 -- 172.21.15.12:0/653576078 shutdown_connections 2024-03-23T23:08:40.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.071+0000 7f0e24000640 1 -- 172.21.15.12:0/653576078 wait complete. 2024-03-23T23:08:40.091 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:08:40.091 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:08:40.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.317+0000 7fb2eb200640 1 -- 172.21.15.12:0/284942504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2ec101580 msgr2=0x7fb2ec1019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:40.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.317+0000 7fb2eb200640 1 --2- 172.21.15.12:0/284942504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2ec101580 0x7fb2ec1019a0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc009900 tx=0x7fb2dc02bc30 comp rx=0 tx=0).stop 2024-03-23T23:08:40.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.317+0000 7fb2eb200640 1 -- 172.21.15.12:0/284942504 shutdown_connections 2024-03-23T23:08:40.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.317+0000 7fb2eb200640 1 --2- 172.21.15.12:0/284942504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb2ec103130 0x7fb2ec1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.317+0000 7fb2eb200640 1 --2- 172.21.15.12:0/284942504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2ec102770 0x7fb2ec102bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.317+0000 7fb2eb200640 1 --2- 172.21.15.12:0/284942504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2ec101580 0x7fb2ec1019a0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.318+0000 7fb2eb200640 1 -- 172.21.15.12:0/284942504 >> 172.21.15.12:0/284942504 conn(0x7fb2ec0fcb10 msgr2=0x7fb2ec0fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:40.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.318+0000 7fb2eb200640 1 -- 172.21.15.12:0/284942504 shutdown_connections 2024-03-23T23:08:40.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.318+0000 7fb2eb200640 1 -- 172.21.15.12:0/284942504 wait complete. 2024-03-23T23:08:40.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.319+0000 7fb2eb200640 1 Processor -- start 2024-03-23T23:08:40.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.319+0000 7fb2eb200640 1 -- start start 2024-03-23T23:08:40.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.319+0000 7fb2eb200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2ec101580 0x7fb2ec075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.319+0000 7fb2eb200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2ec102770 0x7fb2ec073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.319+0000 7fb2eb200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb2ec103130 0x7fb2ec073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.319+0000 7fb2eb200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec10d220 con 0x7fb2ec103130 2024-03-23T23:08:40.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.319+0000 7fb2eb200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec10d0a0 con 0x7fb2ec101580 2024-03-23T23:08:40.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.319+0000 7fb2eb200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec10d3a0 con 0x7fb2ec102770 2024-03-23T23:08:40.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.320+0000 7fb2e9e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2ec101580 0x7fb2ec075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.320+0000 7fb2e9e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2ec101580 0x7fb2ec075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57526/0 (socket says 172.21.15.12:57526) 2024-03-23T23:08:40.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.320+0000 7fb2e9e00640 1 -- 172.21.15.12:0/183073100 learned_addr learned my addr 172.21.15.12:0/183073100 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:40.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.320+0000 7fb2e9400640 1 --2- 172.21.15.12:0/183073100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2ec102770 0x7fb2ec073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.320+0000 7fb2ea800640 1 --2- 172.21.15.12:0/183073100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb2ec103130 0x7fb2ec073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.320+0000 7fb2e9e00640 1 -- 172.21.15.12:0/183073100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2ec102770 msgr2=0x7fb2ec073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:40.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.320+0000 7fb2e9e00640 1 --2- 172.21.15.12:0/183073100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2ec102770 0x7fb2ec073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.320+0000 7fb2e9e00640 1 -- 172.21.15.12:0/183073100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb2ec103130 msgr2=0x7fb2ec073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:40.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.320+0000 7fb2e9e00640 1 --2- 172.21.15.12:0/183073100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb2ec103130 0x7fb2ec073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.320+0000 7fb2e9e00640 1 -- 172.21.15.12:0/183073100 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ec074860 con 0x7fb2ec101580 2024-03-23T23:08:40.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.320+0000 7fb2e9400640 1 --2- 172.21.15.12:0/183073100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2ec102770 0x7fb2ec073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:40.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.321+0000 7fb2e9e00640 1 --2- 172.21.15.12:0/183073100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2ec101580 0x7fb2ec075450 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc000c00 tx=0x7fb2dc02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:40.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.321+0000 7fb2e2a00640 1 -- 172.21.15.12:0/183073100 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc043020 con 0x7fb2ec101580 2024-03-23T23:08:40.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.321+0000 7fb2eb200640 1 -- 172.21.15.12:0/183073100 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2ec1a19b0 con 0x7fb2ec101580 2024-03-23T23:08:40.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.321+0000 7fb2e2a00640 1 -- 172.21.15.12:0/183073100 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb2dc0054e0 con 0x7fb2ec101580 2024-03-23T23:08:40.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.321+0000 7fb2e2a00640 1 -- 172.21.15.12:0/183073100 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc003bd0 con 0x7fb2ec101580 2024-03-23T23:08:40.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.321+0000 7fb2eb200640 1 -- 172.21.15.12:0/183073100 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2ec1a1f00 con 0x7fb2ec101580 2024-03-23T23:08:40.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.324+0000 7fb2f3d76740 1 Processor -- start 2024-03-23T23:08:40.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.324+0000 7fb2f3d76740 1 -- start start 2024-03-23T23:08:40.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.324+0000 7fb2e2a00640 1 -- 172.21.15.12:0/183073100 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb2dc046020 con 0x7fb2ec101580 2024-03-23T23:08:40.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.325+0000 7fb2f3d76740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c2a1175050 0x55c2a1175470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.325+0000 7fb2f3d76740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175a40 0x55c2a117a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.325+0000 7fb2f3d76740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c2a117abb0 0x55c2a117d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.325+0000 7fb2f3d76740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2a10da250 con 0x55c2a117abb0 2024-03-23T23:08:40.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.325+0000 7fb2f3d76740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2a1082a30 con 0x55c2a1175050 2024-03-23T23:08:40.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.325+0000 7fb2f3d76740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2a10d62f0 con 0x55c2a1175a40 2024-03-23T23:08:40.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.325+0000 7fb2e9e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175a40 0x55c2a117a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.325+0000 7fb2ea800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c2a1175050 0x55c2a1175470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.325+0000 7fb2ea800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c2a1175050 0x55c2a1175470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57534/0 (socket says 172.21.15.12:57534) 2024-03-23T23:08:40.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.325+0000 7fb2ea800640 1 -- 172.21.15.12:0/3119374251 learned_addr learned my addr 172.21.15.12:0/3119374251 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:40.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.325+0000 7fb2e9400640 1 --2- 172.21.15.12:0/3119374251 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c2a117abb0 0x55c2a117d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.325+0000 7fb2ea800640 1 -- 172.21.15.12:0/3119374251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175a40 msgr2=0x55c2a117a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:40.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.325+0000 7fb2ea800640 1 --2- 172.21.15.12:0/3119374251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175a40 0x55c2a117a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.326+0000 7fb2ea800640 1 -- 172.21.15.12:0/3119374251 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c2a117abb0 msgr2=0x55c2a117d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:40.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.326+0000 7fb2ea800640 1 --2- 172.21.15.12:0/3119374251 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c2a117abb0 0x55c2a117d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.326+0000 7fb2ea800640 1 -- 172.21.15.12:0/3119374251 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2a106d690 con 0x55c2a1175050 2024-03-23T23:08:40.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.326+0000 7fb2e9e00640 1 --2- 172.21.15.12:0/3119374251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175a40 0x55c2a117a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:40.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.326+0000 7fb2ea800640 1 --2- 172.21.15.12:0/3119374251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c2a1175050 0x55c2a1175470 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4011440 tx=0x7fb2e4011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc1ebd40bbbe1802 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:40.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.326+0000 7fb2d7400640 1 -- 172.21.15.12:0/3119374251 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e4003c30 con 0x55c2a1175050 2024-03-23T23:08:40.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.326+0000 7fb2d7400640 1 -- 172.21.15.12:0/3119374251 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb2e4003dd0 con 0x55c2a1175050 2024-03-23T23:08:40.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.326+0000 7fb2d7400640 1 -- 172.21.15.12:0/3119374251 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e400faf0 con 0x55c2a1175050 2024-03-23T23:08:40.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.326+0000 7fb2e2a00640 1 --2- 172.21.15.12:0/183073100 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb2d00a2800 0x7fb2d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.326+0000 7fb2e2a00640 1 -- 172.21.15.12:0/183073100 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb2dc0f0ad0 con 0x7fb2ec101580 2024-03-23T23:08:40.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.326+0000 7fb2e9e00640 1 --2- 172.21.15.12:0/183073100 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb2d00a2800 0x7fb2d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.327+0000 7fb2f3d76740 1 -- 172.21.15.12:0/3119374251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c2a1175050 msgr2=0x55c2a1175470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:40.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.327+0000 7fb2f3d76740 1 --2- 172.21.15.12:0/3119374251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c2a1175050 0x55c2a1175470 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4011440 tx=0x7fb2e4011900 comp rx=0 tx=0).stop 2024-03-23T23:08:40.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.327+0000 7fb2f3d76740 1 -- 172.21.15.12:0/3119374251 shutdown_connections 2024-03-23T23:08:40.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.327+0000 7fb2f3d76740 1 --2- 172.21.15.12:0/3119374251 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c2a117abb0 0x55c2a117d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.327+0000 7fb2f3d76740 1 --2- 172.21.15.12:0/3119374251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175a40 0x55c2a117a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.327+0000 7fb2f3d76740 1 --2- 172.21.15.12:0/3119374251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c2a1175050 0x55c2a1175470 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.327+0000 7fb2f3d76740 1 -- 172.21.15.12:0/3119374251 >> 172.21.15.12:0/3119374251 conn(0x55c2a113d200 msgr2=0x55c2a11709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:40.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.327+0000 7fb2f3d76740 1 -- 172.21.15.12:0/3119374251 shutdown_connections 2024-03-23T23:08:40.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.327+0000 7fb2f3d76740 1 -- 172.21.15.12:0/3119374251 wait complete. 2024-03-23T23:08:40.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.327+0000 7fb2e9e00640 1 --2- 172.21.15.12:0/183073100 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb2d00a2800 0x7fb2d00a4cc0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0b9df0 tx=0x7fb2dc0b9040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:40.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.328+0000 7fb2f3d76740 1 -- start start 2024-03-23T23:08:40.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.328+0000 7fb2f3d76740 1 Processor -- start 2024-03-23T23:08:40.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.328+0000 7fb2f3d76740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c2a1175050 0x55c2a117f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.328+0000 7fb2f3d76740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175a40 0x55c2a117f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.328+0000 7fb2f3d76740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c2a117abb0 0x55c2a11865e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.328+0000 7fb2f3d76740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2a117db60 con 0x55c2a117abb0 2024-03-23T23:08:40.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.328+0000 7fb2f3d76740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2a1095e80 con 0x55c2a1175050 2024-03-23T23:08:40.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.328+0000 7fb2f3d76740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2a117dce0 con 0x55c2a1175a40 2024-03-23T23:08:40.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.328+0000 7fb2f3d76740 10 client.0 _ll_register_callbacks cb 0x55c2a10a8130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:40.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.328+0000 7fb2e9400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175a40 0x55c2a117f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.329+0000 7fb2e9400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175a40 0x55c2a117f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:42478/0 (socket says 172.21.15.12:42478) 2024-03-23T23:08:40.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.329+0000 7fb2e9400640 1 -- 172.21.15.12:0/2415595672 learned_addr learned my addr 172.21.15.12:0/2415595672 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:40.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.329+0000 7fb2ea800640 1 --2- 172.21.15.12:0/2415595672 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c2a1175050 0x55c2a117f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.329+0000 7fb2ea800640 1 --2- 172.21.15.12:0/2415595672 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c2a117abb0 0x55c2a11865e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.329+0000 7fb2e9400640 1 -- 172.21.15.12:0/2415595672 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c2a1175050 msgr2=0x55c2a117f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:40.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.329+0000 7fb2e9400640 1 --2- 172.21.15.12:0/2415595672 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c2a1175050 0x55c2a117f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.329+0000 7fb2e9400640 1 -- 172.21.15.12:0/2415595672 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c2a117abb0 msgr2=0x55c2a11865e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:40.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.329+0000 7fb2e9400640 1 --2- 172.21.15.12:0/2415595672 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c2a117abb0 0x55c2a11865e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.329+0000 7fb2e9400640 1 -- 172.21.15.12:0/2415595672 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c2a0eb87d0 con 0x55c2a1175a40 2024-03-23T23:08:40.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.329+0000 7fb2ea800640 1 --2- 172.21.15.12:0/2415595672 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c2a1175050 0x55c2a117f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:08:40.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.329+0000 7fb2e9400640 1 --2- 172.21.15.12:0/2415595672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175a40 0x55c2a117f5a0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8012d00 tx=0x7fb2d800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c2e5b75fe3482ba server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:40.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.329+0000 7fb2cbe00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:08:40.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.329+0000 7fb2cbe00640 1 -- 172.21.15.12:0/2415595672 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8012110 con 0x55c2a1175a40 2024-03-23T23:08:40.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.329+0000 7fb2f3d76740 1 -- 172.21.15.12:0/2415595672 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb2e4003dd0 con 0x55c2a1175a40 2024-03-23T23:08:40.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.329+0000 7fb2cbe00640 1 -- 172.21.15.12:0/2415595672 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb2d8004050 con 0x55c2a1175a40 2024-03-23T23:08:40.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.330+0000 7fb2cbe00640 1 -- 172.21.15.12:0/2415595672 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2d80122b0 con 0x55c2a1175a40 2024-03-23T23:08:40.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.330+0000 7fb2f3d76740 10 client.27250 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:40.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.330+0000 7fb2f3d76740 1 -- 172.21.15.12:0/2415595672 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c2a0dee3f0 con 0x55c2a1175a40 2024-03-23T23:08:40.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.330+0000 7fb2cbe00640 1 -- 172.21.15.12:0/2415595672 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb2e4003dd0 con 0x55c2a1175a40 2024-03-23T23:08:40.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.330+0000 7fb2cbe00640 1 -- 172.21.15.12:0/2415595672 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fb2d8010430 con 0x55c2a1175a40 2024-03-23T23:08:40.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.330+0000 7fb2f3d76740 10 client.27250 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:40.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.330+0000 7fb2f3d76740 10 client.27250 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:08:40.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.330+0000 7fb2f3d76740 20 client.27250 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:40.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.330+0000 7fb2f3d76740 1 --2- 172.21.15.12:0/2415595672 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c2a118afa0 0x55c2a118d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.330+0000 7fb2f3d76740 4 client.27250 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:08:40.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.330+0000 7fb2f3d76740 1 -- 172.21.15.12:0/2415595672 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c2a10e7790 con 0x55c2a118afa0 2024-03-23T23:08:40.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.331+0000 7fb2ea800640 1 --2- 172.21.15.12:0/2415595672 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c2a118afa0 0x55c2a118d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.331+0000 7fb2ea800640 1 --2- 172.21.15.12:0/2415595672 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c2a118afa0 0x55c2a118d3e0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24239773a8ca8a41 server_cookie=db4d22dddd271f9c in_seq=0 out_seq=0 2024-03-23T23:08:40.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.331+0000 7fb2cbe00640 10 client.27250 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:40.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.332+0000 7fb2cbe00640 1 -- 172.21.15.12:0/2415595672 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c2a10e7790 con 0x55c2a118afa0 2024-03-23T23:08:40.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.332+0000 7fb2cbe00640 10 client.27250 handle_command_reply: tid=0 2024-03-23T23:08:40.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.332+0000 7fb2f3d76740 1 client.27250 shutdown 2024-03-23T23:08:40.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.332+0000 7fb2f3d76740 1 -- 172.21.15.12:0/2415595672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175a40 msgr2=0x55c2a117f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:40.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.332+0000 7fb2f3d76740 1 --2- 172.21.15.12:0/2415595672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175a40 0x55c2a117f5a0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8012d00 tx=0x7fb2d800f790 comp rx=0 tx=0).stop 2024-03-23T23:08:40.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.332+0000 7fb2f3d76740 1 -- 172.21.15.12:0/2415595672 shutdown_connections 2024-03-23T23:08:40.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.332+0000 7fb2f3d76740 1 --2- 172.21.15.12:0/2415595672 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c2a118afa0 0x55c2a118d3e0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.332+0000 7fb2f3d76740 1 --2- 172.21.15.12:0/2415595672 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c2a117abb0 0x55c2a11865e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.332+0000 7fb2f3d76740 1 --2- 172.21.15.12:0/2415595672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175a40 0x55c2a117f5a0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.332+0000 7fb2f3d76740 1 --2- 172.21.15.12:0/2415595672 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c2a1175050 0x55c2a117f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.332+0000 7fb2cbe00640 0 client.27250 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:40.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.332+0000 7fb2f3d76740 1 -- 172.21.15.12:0/2415595672 >> 172.21.15.12:0/2415595672 conn(0x55c2a113d200 msgr2=0x55c2a116fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:40.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.333+0000 7fb2f3d76740 1 -- 172.21.15.12:0/2415595672 shutdown_connections 2024-03-23T23:08:40.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.333+0000 7fb2f3d76740 1 -- 172.21.15.12:0/2415595672 wait complete. 2024-03-23T23:08:40.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.333+0000 7fb2f3d76740 20 client.27250 trim_cache size 0 max 16384 2024-03-23T23:08:40.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.351+0000 7fb2f3d76740 1 Processor -- start 2024-03-23T23:08:40.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.351+0000 7fb2f3d76740 1 -- start start 2024-03-23T23:08:40.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.351+0000 7fb2f3d76740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c2a1175050 0x55c2a116fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.351+0000 7fb2f3d76740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175a40 0x55c2a116ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.351+0000 7fb2f3d76740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c2a117abb0 0x55c2a11865a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.351+0000 7fb2f3d76740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec1acf10 con 0x55c2a117abb0 2024-03-23T23:08:40.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.351+0000 7fb2f3d76740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec1aaa10 con 0x55c2a1175050 2024-03-23T23:08:40.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.351+0000 7fb2f3d76740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec1a8ab0 con 0x55c2a1175a40 2024-03-23T23:08:40.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.351+0000 7fb2e9400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175a40 0x55c2a116ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.351+0000 7fb2e9400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175a40 0x55c2a116ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:42490/0 (socket says 172.21.15.12:42490) 2024-03-23T23:08:40.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.351+0000 7fb2e9400640 1 -- 172.21.15.12:0/4224581243 learned_addr learned my addr 172.21.15.12:0/4224581243 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:40.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.351+0000 7fb2ea800640 1 --2- 172.21.15.12:0/4224581243 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c2a1175050 0x55c2a116fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.351+0000 7fb2ea800640 1 --2- 172.21.15.12:0/4224581243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c2a117abb0 0x55c2a11865a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.352+0000 7fb2e9400640 1 -- 172.21.15.12:0/4224581243 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c2a1175050 msgr2=0x55c2a116fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:40.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.352+0000 7fb2e9400640 1 --2- 172.21.15.12:0/4224581243 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c2a1175050 0x55c2a116fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.352+0000 7fb2e9400640 1 -- 172.21.15.12:0/4224581243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c2a117abb0 msgr2=0x55c2a11865a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:40.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.352+0000 7fb2e9400640 1 --2- 172.21.15.12:0/4224581243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c2a117abb0 0x55c2a11865a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.352+0000 7fb2e9400640 1 -- 172.21.15.12:0/4224581243 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2a0dae0f0 con 0x55c2a1175a40 2024-03-23T23:08:40.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.352+0000 7fb2e9400640 1 --2- 172.21.15.12:0/4224581243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175a40 0x55c2a116ff80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8010920 tx=0x7fb2d8005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30914ca8e43b58f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:40.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.352+0000 7fb2d6a00640 1 -- 172.21.15.12:0/4224581243 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8010a90 con 0x55c2a1175a40 2024-03-23T23:08:40.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.352+0000 7fb2d6a00640 1 -- 172.21.15.12:0/4224581243 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb2d8010c30 con 0x55c2a1175a40 2024-03-23T23:08:40.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.352+0000 7fb2d6a00640 1 -- 172.21.15.12:0/4224581243 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8003a50 con 0x55c2a1175a40 2024-03-23T23:08:40.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.353+0000 7fb2f3d76740 1 -- 172.21.15.12:0/4224581243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175a40 msgr2=0x55c2a116ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:40.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.353+0000 7fb2f3d76740 1 --2- 172.21.15.12:0/4224581243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175a40 0x55c2a116ff80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8010920 tx=0x7fb2d8005850 comp rx=0 tx=0).stop 2024-03-23T23:08:40.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.353+0000 7fb2f3d76740 1 -- 172.21.15.12:0/4224581243 shutdown_connections 2024-03-23T23:08:40.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.353+0000 7fb2f3d76740 1 --2- 172.21.15.12:0/4224581243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c2a117abb0 0x55c2a11865a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.353+0000 7fb2f3d76740 1 --2- 172.21.15.12:0/4224581243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175a40 0x55c2a116ff80 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.353+0000 7fb2f3d76740 1 --2- 172.21.15.12:0/4224581243 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c2a1175050 0x55c2a116fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.353+0000 7fb2f3d76740 1 -- 172.21.15.12:0/4224581243 >> 172.21.15.12:0/4224581243 conn(0x55c2a113d200 msgr2=0x55c2a116f330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:40.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.353+0000 7fb2f3d76740 1 -- 172.21.15.12:0/4224581243 shutdown_connections 2024-03-23T23:08:40.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.353+0000 7fb2f3d76740 1 -- 172.21.15.12:0/4224581243 wait complete. 2024-03-23T23:08:40.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.353+0000 7fb2f3d76740 1 -- start start 2024-03-23T23:08:40.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.354+0000 7fb2f3d76740 1 Processor -- start 2024-03-23T23:08:40.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.354+0000 7fb2f3d76740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175050 0x55c2a1184560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.354+0000 7fb2f3d76740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c2a1175a40 0x55c2a1184aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.354+0000 7fb2f3d76740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c2a117abb0 0x55c2a1185ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.354+0000 7fb2f3d76740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec1a90b0 con 0x55c2a1175a40 2024-03-23T23:08:40.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.354+0000 7fb2f3d76740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec1aa3b0 con 0x55c2a117abb0 2024-03-23T23:08:40.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.354+0000 7fb2f3d76740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec1a96f0 con 0x55c2a1175050 2024-03-23T23:08:40.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.354+0000 7fb2f3d76740 10 client.0 _ll_register_callbacks cb 0x7fb2d8009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:40.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.354+0000 7fb2ea800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175050 0x55c2a1184560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.354+0000 7fb2ea800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175050 0x55c2a1184560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:42494/0 (socket says 172.21.15.12:42494) 2024-03-23T23:08:40.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.354+0000 7fb2ea800640 1 -- 172.21.15.12:0/2601174758 learned_addr learned my addr 172.21.15.12:0/2601174758 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:40.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.354+0000 7fb2e9400640 1 --2- 172.21.15.12:0/2601174758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c2a1175a40 0x55c2a1184aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.354+0000 7fb2ea800640 1 --2- 172.21.15.12:0/2601174758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c2a117abb0 0x55c2a1185ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.355+0000 7fb2ea800640 1 -- 172.21.15.12:0/2601174758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c2a117abb0 msgr2=0x55c2a1185ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:40.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.355+0000 7fb2ea800640 1 --2- 172.21.15.12:0/2601174758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c2a117abb0 0x55c2a1185ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.355+0000 7fb2ea800640 1 -- 172.21.15.12:0/2601174758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c2a1175a40 msgr2=0x55c2a1184aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:40.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.355+0000 7fb2ea800640 1 --2- 172.21.15.12:0/2601174758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c2a1175a40 0x55c2a1184aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.355+0000 7fb2ea800640 1 -- 172.21.15.12:0/2601174758 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c2a11860a0 con 0x55c2a1175050 2024-03-23T23:08:40.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.355+0000 7fb2ea800640 1 --2- 172.21.15.12:0/2601174758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175050 0x55c2a1184560 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4009fa0 tx=0x7fb2e4011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6477453812e7597c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:40.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.355+0000 7fb2cbe00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:08:40.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.355+0000 7fb2cbe00640 1 -- 172.21.15.12:0/2601174758 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e4004c60 con 0x55c2a1175050 2024-03-23T23:08:40.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.355+0000 7fb2f3d76740 1 -- 172.21.15.12:0/2601174758 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c2a10e1ae0 con 0x55c2a1175050 2024-03-23T23:08:40.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.355+0000 7fb2cbe00640 1 -- 172.21.15.12:0/2601174758 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb2e4004e00 con 0x55c2a1175050 2024-03-23T23:08:40.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.356+0000 7fb2cbe00640 1 -- 172.21.15.12:0/2601174758 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2e4004050 con 0x55c2a1175050 2024-03-23T23:08:40.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.356+0000 7fb2f3d76740 10 client.27256 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:40.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.356+0000 7fb2f3d76740 1 -- 172.21.15.12:0/2601174758 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c2a11ef8b0 con 0x55c2a1175050 2024-03-23T23:08:40.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.356+0000 7fb2cbe00640 1 -- 172.21.15.12:0/2601174758 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c2a10e1ae0 con 0x55c2a1175050 2024-03-23T23:08:40.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.356+0000 7fb2cbe00640 1 -- 172.21.15.12:0/2601174758 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fb2e4025040 con 0x55c2a1175050 2024-03-23T23:08:40.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.356+0000 7fb2f3d76740 10 client.27256 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:40.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.356+0000 7fb2f3d76740 10 client.27256 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:08:40.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.356+0000 7fb2f3d76740 20 client.27256 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:40.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.356+0000 7fb2f3d76740 1 --2- 172.21.15.12:0/2601174758 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c2a11f00b0 0x55c2a11f24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:40.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.356+0000 7fb2f3d76740 4 client.27256 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:08:40.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.356+0000 7fb2f3d76740 1 -- 172.21.15.12:0/2601174758 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c2a105f3a0 con 0x55c2a11f00b0 2024-03-23T23:08:40.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.356+0000 7fb2e9400640 1 --2- 172.21.15.12:0/2601174758 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c2a11f00b0 0x55c2a11f24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:40.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.357+0000 7fb2e9400640 1 --2- 172.21.15.12:0/2601174758 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c2a11f00b0 0x55c2a11f24d0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ce691e957520fe3 server_cookie=adcca0fc738a7df4 in_seq=0 out_seq=0 2024-03-23T23:08:40.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.357+0000 7fb2cbe00640 10 client.27256 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:40.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.357+0000 7fb2cbe00640 1 -- 172.21.15.12:0/2601174758 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55c2a105f3a0 con 0x55c2a11f00b0 2024-03-23T23:08:40.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.357+0000 7fb2cbe00640 10 client.27256 handle_command_reply: tid=0 2024-03-23T23:08:40.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.357+0000 7fb2f3d76740 1 client.27256 shutdown 2024-03-23T23:08:40.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.357+0000 7fb2f3d76740 1 -- 172.21.15.12:0/2601174758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175050 msgr2=0x55c2a1184560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:40.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.357+0000 7fb2f3d76740 1 --2- 172.21.15.12:0/2601174758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175050 0x55c2a1184560 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4009fa0 tx=0x7fb2e4011900 comp rx=0 tx=0).stop 2024-03-23T23:08:40.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.357+0000 7fb2f3d76740 1 -- 172.21.15.12:0/2601174758 shutdown_connections 2024-03-23T23:08:40.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.357+0000 7fb2f3d76740 1 --2- 172.21.15.12:0/2601174758 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c2a11f00b0 0x55c2a11f24d0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.357+0000 7fb2f3d76740 1 --2- 172.21.15.12:0/2601174758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c2a117abb0 0x55c2a1185ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.357+0000 7fb2f3d76740 1 --2- 172.21.15.12:0/2601174758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c2a1175a40 0x55c2a1184aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.357+0000 7fb2cbe00640 0 client.27256 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:40.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.357+0000 7fb2f3d76740 1 --2- 172.21.15.12:0/2601174758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2a1175050 0x55c2a1184560 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.357+0000 7fb2f3d76740 1 -- 172.21.15.12:0/2601174758 >> 172.21.15.12:0/2601174758 conn(0x55c2a113d200 msgr2=0x55c2a116fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:40.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.358+0000 7fb2f3d76740 1 -- 172.21.15.12:0/2601174758 shutdown_connections 2024-03-23T23:08:40.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.358+0000 7fb2f3d76740 1 -- 172.21.15.12:0/2601174758 wait complete. 2024-03-23T23:08:40.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.358+0000 7fb2f3d76740 20 client.27256 trim_cache size 0 max 16384 2024-03-23T23:08:40.362 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:08:40.362 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T23:08:40.362 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-03-23T23:08:40.362 INFO:teuthology.orchestra.run.smithi012.stdout: "5ba06596-68c3-4642-bae0-1cce971052f0": { 2024-03-23T23:08:40.362 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-03-23T23:08:40.362 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "5ba06596-68c3-4642-bae0-1cce971052f0", 2024-03-23T23:08:40.362 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-03-23T23:08:40.362 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:08:40.362 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:08:40.362 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:08:40.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.358+0000 7fb2eb200640 1 -- 172.21.15.12:0/183073100 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb2d00a2800 msgr2=0x7fb2d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:40.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.358+0000 7fb2eb200640 1 --2- 172.21.15.12:0/183073100 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb2d00a2800 0x7fb2d00a4cc0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0b9df0 tx=0x7fb2dc0b9040 comp rx=0 tx=0).stop 2024-03-23T23:08:40.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.358+0000 7fb2eb200640 1 -- 172.21.15.12:0/183073100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2ec101580 msgr2=0x7fb2ec075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:40.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.358+0000 7fb2eb200640 1 --2- 172.21.15.12:0/183073100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2ec101580 0x7fb2ec075450 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc000c00 tx=0x7fb2dc02c650 comp rx=0 tx=0).stop 2024-03-23T23:08:40.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.359+0000 7fb2eb200640 1 -- 172.21.15.12:0/183073100 shutdown_connections 2024-03-23T23:08:40.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.359+0000 7fb2eb200640 1 --2- 172.21.15.12:0/183073100 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb2d00a2800 0x7fb2d00a4cc0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.359+0000 7fb2eb200640 1 --2- 172.21.15.12:0/183073100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb2ec103130 0x7fb2ec073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.359+0000 7fb2eb200640 1 --2- 172.21.15.12:0/183073100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2ec102770 0x7fb2ec073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.359+0000 7fb2eb200640 1 --2- 172.21.15.12:0/183073100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2ec101580 0x7fb2ec075450 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:40.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.359+0000 7fb2eb200640 1 -- 172.21.15.12:0/183073100 >> 172.21.15.12:0/183073100 conn(0x7fb2ec0fcb10 msgr2=0x7fb2ec105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:40.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.359+0000 7fb2eb200640 1 -- 172.21.15.12:0/183073100 shutdown_connections 2024-03-23T23:08:40.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:08:40.359+0000 7fb2eb200640 1 -- 172.21.15.12:0/183073100 wait complete. 2024-03-23T23:08:40.377 INFO:tasks.cephfs.filesystem:scrub status for tag:5ba06596-68c3-4642-bae0-1cce971052f0 - {'path': '/', 'tag': '5ba06596-68c3-4642-bae0-1cce971052f0', 'options': 'recursive,force'} 2024-03-23T23:08:40.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:40 smithi173 ceph-mon[36116]: pgmap v596: 97 pgs: 97 active+clean; 4.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:08:40.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:40 smithi203 ceph-mon[36693]: pgmap v596: 97 pgs: 97 active+clean; 4.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:08:41.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:40 smithi012 ceph-mon[31695]: pgmap v596: 97 pgs: 97 active+clean; 4.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:08:41.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:41 smithi173 ceph-mon[36116]: scrub queued for path: / 2024-03-23T23:08:41.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:41 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:08:41.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:41 smithi173 ceph-mon[36116]: scrub summary: active paths [/] 2024-03-23T23:08:41.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:41 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:08:41.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:41 smithi173 ceph-mon[36116]: scrub summary: idle 2024-03-23T23:08:41.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:41 smithi203 ceph-mon[36693]: scrub queued for path: / 2024-03-23T23:08:41.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:41 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:08:41.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:41 smithi203 ceph-mon[36693]: scrub summary: active paths [/] 2024-03-23T23:08:41.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:41 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:08:41.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:41 smithi203 ceph-mon[36693]: scrub summary: idle 2024-03-23T23:08:42.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:41 smithi012 ceph-mon[31695]: scrub queued for path: / 2024-03-23T23:08:42.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:41 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:08:42.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:41 smithi012 ceph-mon[31695]: scrub summary: active paths [/] 2024-03-23T23:08:42.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:41 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:08:42.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:41 smithi012 ceph-mon[31695]: scrub summary: idle 2024-03-23T23:08:42.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:42 smithi173 ceph-mon[36116]: pgmap v597: 97 pgs: 97 active+clean; 4.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:08:42.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:42 smithi203 ceph-mon[36693]: pgmap v597: 97 pgs: 97 active+clean; 4.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:08:43.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:42 smithi012 ceph-mon[31695]: pgmap v597: 97 pgs: 97 active+clean; 4.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:08:44.821 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.821 INFO:tasks.workunit.client.0.smithi012.stdout:251/600: open fd = 3 2024-03-23T23:08:44.821 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.822 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.822 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.822 INFO:tasks.workunit.client.0.smithi012.stdout:252/600: open fd = 3 2024-03-23T23:08:44.822 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.822 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.822 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.822 INFO:tasks.workunit.client.0.smithi012.stdout:253/600: open fd = 3 2024-03-23T23:08:44.822 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.822 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.822 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.822 INFO:tasks.workunit.client.0.smithi012.stdout:254/600: open fd = 3 2024-03-23T23:08:44.822 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.822 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.822 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.822 INFO:tasks.workunit.client.0.smithi012.stdout:255/600: open fd = 3 2024-03-23T23:08:44.823 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.823 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.823 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.823 INFO:tasks.workunit.client.0.smithi012.stdout:256/600: open fd = 3 2024-03-23T23:08:44.823 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.823 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.823 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.823 INFO:tasks.workunit.client.0.smithi012.stdout:257/600: open fd = 3 2024-03-23T23:08:44.823 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.823 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.823 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.823 INFO:tasks.workunit.client.0.smithi012.stdout:258/600: open fd = 3 2024-03-23T23:08:44.823 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.823 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.823 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.823 INFO:tasks.workunit.client.0.smithi012.stdout:259/600: open fd = 3 2024-03-23T23:08:44.823 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.824 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.824 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.824 INFO:tasks.workunit.client.0.smithi012.stdout:260/600: open fd = 3 2024-03-23T23:08:44.824 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.824 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.824 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.824 INFO:tasks.workunit.client.0.smithi012.stdout:261/600: open fd = 3 2024-03-23T23:08:44.824 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.824 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.824 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.824 INFO:tasks.workunit.client.0.smithi012.stdout:262/600: open fd = 3 2024-03-23T23:08:44.824 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.824 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.824 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.824 INFO:tasks.workunit.client.0.smithi012.stdout:263/600: open fd = 3 2024-03-23T23:08:44.824 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.824 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.825 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.825 INFO:tasks.workunit.client.0.smithi012.stdout:264/600: open fd = 3 2024-03-23T23:08:44.825 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.825 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.825 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.825 INFO:tasks.workunit.client.0.smithi012.stdout:265/600: open fd = 3 2024-03-23T23:08:44.825 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.825 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.825 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.825 INFO:tasks.workunit.client.0.smithi012.stdout:266/600: open fd = 3 2024-03-23T23:08:44.825 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.825 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.825 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.825 INFO:tasks.workunit.client.0.smithi012.stdout:267/600: open fd = 3 2024-03-23T23:08:44.825 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.825 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.825 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.826 INFO:tasks.workunit.client.0.smithi012.stdout:268/600: open fd = 3 2024-03-23T23:08:44.826 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.826 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.826 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.826 INFO:tasks.workunit.client.0.smithi012.stdout:269/600: open fd = 3 2024-03-23T23:08:44.826 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.826 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.826 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.826 INFO:tasks.workunit.client.0.smithi012.stdout:270/600: open fd = 3 2024-03-23T23:08:44.826 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.826 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.826 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.826 INFO:tasks.workunit.client.0.smithi012.stdout:271/600: open fd = 3 2024-03-23T23:08:44.826 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.826 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.826 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.827 INFO:tasks.workunit.client.0.smithi012.stdout:272/600: open fd = 3 2024-03-23T23:08:44.827 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.827 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.827 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.827 INFO:tasks.workunit.client.0.smithi012.stdout:273/600: open fd = 3 2024-03-23T23:08:44.827 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.827 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.827 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.827 INFO:tasks.workunit.client.0.smithi012.stdout:274/600: open fd = 3 2024-03-23T23:08:44.827 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.827 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.827 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.827 INFO:tasks.workunit.client.0.smithi012.stdout:275/600: open fd = 3 2024-03-23T23:08:44.827 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.827 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.827 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.828 INFO:tasks.workunit.client.0.smithi012.stdout:276/600: open fd = 3 2024-03-23T23:08:44.828 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.828 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.828 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.828 INFO:tasks.workunit.client.0.smithi012.stdout:277/600: open fd = 3 2024-03-23T23:08:44.828 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.828 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.828 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.828 INFO:tasks.workunit.client.0.smithi012.stdout:278/600: open fd = 3 2024-03-23T23:08:44.828 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.828 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.828 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.828 INFO:tasks.workunit.client.0.smithi012.stdout:279/600: open fd = 3 2024-03-23T23:08:44.828 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.828 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.828 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.828 INFO:tasks.workunit.client.0.smithi012.stdout:280/600: open fd = 3 2024-03-23T23:08:44.828 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.829 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.829 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.829 INFO:tasks.workunit.client.0.smithi012.stdout:281/600: open fd = 3 2024-03-23T23:08:44.829 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.829 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.829 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.829 INFO:tasks.workunit.client.0.smithi012.stdout:282/600: open fd = 3 2024-03-23T23:08:44.829 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.829 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.829 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.829 INFO:tasks.workunit.client.0.smithi012.stdout:283/600: open fd = 3 2024-03-23T23:08:44.829 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.829 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.829 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.829 INFO:tasks.workunit.client.0.smithi012.stdout:284/600: open fd = 3 2024-03-23T23:08:44.829 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.829 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.830 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.830 INFO:tasks.workunit.client.0.smithi012.stdout:285/600: open fd = 3 2024-03-23T23:08:44.830 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.830 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.830 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.830 INFO:tasks.workunit.client.0.smithi012.stdout:286/600: open fd = 3 2024-03-23T23:08:44.830 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.830 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.830 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.830 INFO:tasks.workunit.client.0.smithi012.stdout:287/600: open fd = 3 2024-03-23T23:08:44.830 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.830 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.830 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.830 INFO:tasks.workunit.client.0.smithi012.stdout:288/600: open fd = 3 2024-03-23T23:08:44.830 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.831 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.831 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.831 INFO:tasks.workunit.client.0.smithi012.stdout:289/600: open fd = 3 2024-03-23T23:08:44.831 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.831 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.831 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.831 INFO:tasks.workunit.client.0.smithi012.stdout:290/600: open fd = 3 2024-03-23T23:08:44.831 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.831 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.831 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.831 INFO:tasks.workunit.client.0.smithi012.stdout:291/600: open fd = 3 2024-03-23T23:08:44.831 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.831 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.831 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.831 INFO:tasks.workunit.client.0.smithi012.stdout:292/600: open fd = 3 2024-03-23T23:08:44.831 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.831 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.832 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.832 INFO:tasks.workunit.client.0.smithi012.stdout:293/600: open fd = 3 2024-03-23T23:08:44.832 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.832 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.832 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.832 INFO:tasks.workunit.client.0.smithi012.stdout:294/600: open fd = 3 2024-03-23T23:08:44.832 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.832 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.832 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.832 INFO:tasks.workunit.client.0.smithi012.stdout:295/600: open fd = 3 2024-03-23T23:08:44.832 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.832 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.832 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.832 INFO:tasks.workunit.client.0.smithi012.stdout:296/600: open fd = 3 2024-03-23T23:08:44.832 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.832 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.832 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.833 INFO:tasks.workunit.client.0.smithi012.stdout:297/600: open fd = 3 2024-03-23T23:08:44.833 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.833 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.833 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.833 INFO:tasks.workunit.client.0.smithi012.stdout:298/600: open fd = 3 2024-03-23T23:08:44.833 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.833 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.833 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.833 INFO:tasks.workunit.client.0.smithi012.stdout:299/600: open fd = 3 2024-03-23T23:08:44.833 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.833 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.833 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.833 INFO:tasks.workunit.client.0.smithi012.stdout:300/600: open fd = 3 2024-03-23T23:08:44.833 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.833 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.833 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.834 INFO:tasks.workunit.client.0.smithi012.stdout:301/600: open fd = 3 2024-03-23T23:08:44.834 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.834 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.834 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.834 INFO:tasks.workunit.client.0.smithi012.stdout:302/600: open fd = 3 2024-03-23T23:08:44.834 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.834 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.834 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.834 INFO:tasks.workunit.client.0.smithi012.stdout:303/600: open fd = 3 2024-03-23T23:08:44.834 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.834 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.834 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.834 INFO:tasks.workunit.client.0.smithi012.stdout:304/600: open fd = 3 2024-03-23T23:08:44.834 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.834 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.835 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.835 INFO:tasks.workunit.client.0.smithi012.stdout:305/600: open fd = 3 2024-03-23T23:08:44.835 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.835 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.835 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.835 INFO:tasks.workunit.client.0.smithi012.stdout:306/600: open fd = 3 2024-03-23T23:08:44.835 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.835 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.835 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.835 INFO:tasks.workunit.client.0.smithi012.stdout:307/600: open fd = 3 2024-03-23T23:08:44.835 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.835 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.835 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.835 INFO:tasks.workunit.client.0.smithi012.stdout:308/600: open fd = 3 2024-03-23T23:08:44.835 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.835 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.835 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.835 INFO:tasks.workunit.client.0.smithi012.stdout:309/600: open fd = 3 2024-03-23T23:08:44.836 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.836 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.836 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.836 INFO:tasks.workunit.client.0.smithi012.stdout:310/600: open fd = 3 2024-03-23T23:08:44.836 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.836 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.836 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.836 INFO:tasks.workunit.client.0.smithi012.stdout:311/600: open fd = 3 2024-03-23T23:08:44.836 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.836 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.836 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:08:44.836 INFO:tasks.workunit.client.0.smithi012.stdout:312/600: open fd = 3 2024-03-23T23:08:44.836 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.836 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:08:44.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:44 smithi173 ceph-mon[36116]: pgmap v598: 97 pgs: 97 active+clean; 4.5 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:08:44.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:44 smithi203 ceph-mon[36693]: pgmap v598: 97 pgs: 97 active+clean; 4.5 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:08:45.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:44 smithi012 ceph-mon[31695]: pgmap v598: 97 pgs: 97 active+clean; 4.5 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:08:46.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:46 smithi173 ceph-mon[36116]: pgmap v599: 97 pgs: 97 active+clean; 4.5 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-03-23T23:08:46.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:46 smithi203 ceph-mon[36693]: pgmap v599: 97 pgs: 97 active+clean; 4.5 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-03-23T23:08:47.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:46 smithi012 ceph-mon[31695]: pgmap v599: 97 pgs: 97 active+clean; 4.5 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-03-23T23:08:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.558+0000 7ff968400640 1 -- 172.21.15.12:0/922490708 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9601098b0 msgr2=0x7ff960110140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.558+0000 7ff968400640 1 --2- 172.21.15.12:0/922490708 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9601098b0 0x7ff960110140 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ff958009ca0 tx=0x7ff95802c100 comp rx=0 tx=0).stop 2024-03-23T23:08:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.559+0000 7ff968400640 1 -- 172.21.15.12:0/922490708 shutdown_connections 2024-03-23T23:08:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.559+0000 7ff968400640 1 --2- 172.21.15.12:0/922490708 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9601098b0 0x7ff960110140 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.559+0000 7ff968400640 1 --2- 172.21.15.12:0/922490708 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff960108ef0 0x7ff960109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.559+0000 7ff968400640 1 --2- 172.21.15.12:0/922490708 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff960107d00 0x7ff960108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.559+0000 7ff968400640 1 -- 172.21.15.12:0/922490708 >> 172.21.15.12:0/922490708 conn(0x7ff960076cb0 msgr2=0x7ff9600790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.559+0000 7ff968400640 1 -- 172.21.15.12:0/922490708 shutdown_connections 2024-03-23T23:08:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.559+0000 7ff968400640 1 -- 172.21.15.12:0/922490708 wait complete. 2024-03-23T23:08:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.560+0000 7ff968400640 1 Processor -- start 2024-03-23T23:08:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.560+0000 7ff968400640 1 -- start start 2024-03-23T23:08:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.560+0000 7ff968400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff960107d00 0x7ff96019df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.560+0000 7ff968400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff960108ef0 0x7ff96019e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.560+0000 7ff968400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9601098b0 0x7ff9601a54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.560+0000 7ff968400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff960113960 con 0x7ff960108ef0 2024-03-23T23:08:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.560+0000 7ff968400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9601137e0 con 0x7ff960107d00 2024-03-23T23:08:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.560+0000 7ff968400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff960113ae0 con 0x7ff9601098b0 2024-03-23T23:08:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.561+0000 7ff965000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff960108ef0 0x7ff96019e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.561+0000 7ff965a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff960107d00 0x7ff96019df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.561+0000 7ff966400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9601098b0 0x7ff9601a54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.561+0000 7ff965000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff960108ef0 0x7ff96019e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42646/0 (socket says 172.21.15.12:42646) 2024-03-23T23:08:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.561+0000 7ff965000640 1 -- 172.21.15.12:0/3902826320 learned_addr learned my addr 172.21.15.12:0/3902826320 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.561+0000 7ff965000640 1 -- 172.21.15.12:0/3902826320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9601098b0 msgr2=0x7ff9601a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.561+0000 7ff965000640 1 --2- 172.21.15.12:0/3902826320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9601098b0 0x7ff9601a54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.561+0000 7ff965000640 1 -- 172.21.15.12:0/3902826320 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff960107d00 msgr2=0x7ff96019df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.561+0000 7ff965000640 1 --2- 172.21.15.12:0/3902826320 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff960107d00 0x7ff96019df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.561+0000 7ff965000640 1 -- 172.21.15.12:0/3902826320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9601a5b30 con 0x7ff960108ef0 2024-03-23T23:08:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.561+0000 7ff966400640 1 --2- 172.21.15.12:0/3902826320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9601098b0 0x7ff9601a54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.561+0000 7ff965000640 1 --2- 172.21.15.12:0/3902826320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff960108ef0 0x7ff96019e450 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff94c00ea60 tx=0x7ff94c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.562+0000 7ff95e000640 1 -- 172.21.15.12:0/3902826320 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94c00cde0 con 0x7ff960108ef0 2024-03-23T23:08:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.562+0000 7ff95e000640 1 -- 172.21.15.12:0/3902826320 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff94c0035f0 con 0x7ff960108ef0 2024-03-23T23:08:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.562+0000 7ff95e000640 1 -- 172.21.15.12:0/3902826320 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94c010670 con 0x7ff960108ef0 2024-03-23T23:08:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.562+0000 7ff968400640 1 -- 172.21.15.12:0/3902826320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9601a5e20 con 0x7ff960108ef0 2024-03-23T23:08:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.562+0000 7ff968400640 1 -- 172.21.15.12:0/3902826320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9601a6290 con 0x7ff960108ef0 2024-03-23T23:08:47.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.565+0000 7ff96b17e740 1 Processor -- start 2024-03-23T23:08:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.565+0000 7ff96b17e740 1 -- start start 2024-03-23T23:08:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.565+0000 7ff96b17e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0b1e0 0x556ce5f0b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.565+0000 7ff96b17e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556ce5f0bbd0 0x556ce5f106d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.565+0000 7ff96b17e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ce5f10d40 0x556ce5f131d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.565+0000 7ff96b17e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ce5ece3b0 con 0x556ce5f0bbd0 2024-03-23T23:08:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.565+0000 7ff96b17e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ce5e70830 con 0x556ce5f10d40 2024-03-23T23:08:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.565+0000 7ff96b17e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ce5e195d0 con 0x556ce5f0b1e0 2024-03-23T23:08:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.566+0000 7ff95e000640 1 -- 172.21.15.12:0/3902826320 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff94c0037a0 con 0x7ff960108ef0 2024-03-23T23:08:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.566+0000 7ff966400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0b1e0 0x556ce5f0b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.566+0000 7ff966400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0b1e0 0x556ce5f0b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:36628/0 (socket says 172.21.15.12:36628) 2024-03-23T23:08:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.566+0000 7ff966400640 1 -- 172.21.15.12:0/4153078187 learned_addr learned my addr 172.21.15.12:0/4153078187 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.566+0000 7ff965a00640 1 --2- 172.21.15.12:0/4153078187 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556ce5f0bbd0 0x556ce5f106d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.566+0000 7ff965000640 1 --2- 172.21.15.12:0/4153078187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ce5f10d40 0x556ce5f131d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.566+0000 7ff966400640 1 -- 172.21.15.12:0/4153078187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ce5f10d40 msgr2=0x556ce5f131d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.566+0000 7ff966400640 1 --2- 172.21.15.12:0/4153078187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ce5f10d40 0x556ce5f131d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.566+0000 7ff966400640 1 -- 172.21.15.12:0/4153078187 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556ce5f0bbd0 msgr2=0x556ce5f106d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.566+0000 7ff966400640 1 --2- 172.21.15.12:0/4153078187 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556ce5f0bbd0 0x556ce5f106d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.566+0000 7ff966400640 1 -- 172.21.15.12:0/4153078187 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556ce5e312c0 con 0x556ce5f0b1e0 2024-03-23T23:08:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.566+0000 7ff965000640 1 --2- 172.21.15.12:0/4153078187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ce5f10d40 0x556ce5f131d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:08:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.567+0000 7ff966400640 1 --2- 172.21.15.12:0/4153078187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0b1e0 0x556ce5f0b600 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff95802c710 tx=0x7ff9580323d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdaaaf7b4b264fa7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.567+0000 7ff952a00640 1 -- 172.21.15.12:0/4153078187 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff958043070 con 0x556ce5f0b1e0 2024-03-23T23:08:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.567+0000 7ff952a00640 1 -- 172.21.15.12:0/4153078187 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff958032c90 con 0x556ce5f0b1e0 2024-03-23T23:08:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.567+0000 7ff952a00640 1 -- 172.21.15.12:0/4153078187 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff958039040 con 0x556ce5f0b1e0 2024-03-23T23:08:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.567+0000 7ff95e000640 1 --2- 172.21.15.12:0/3902826320 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff9480a28a0 0x7ff9480a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.567+0000 7ff96b17e740 1 -- 172.21.15.12:0/4153078187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0b1e0 msgr2=0x556ce5f0b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.567+0000 7ff96b17e740 1 --2- 172.21.15.12:0/4153078187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0b1e0 0x556ce5f0b600 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff95802c710 tx=0x7ff9580323d0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.568+0000 7ff95e000640 1 -- 172.21.15.12:0/3902826320 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff94c0ced30 con 0x7ff960108ef0 2024-03-23T23:08:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.568+0000 7ff96b17e740 1 -- 172.21.15.12:0/4153078187 shutdown_connections 2024-03-23T23:08:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.568+0000 7ff96b17e740 1 --2- 172.21.15.12:0/4153078187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ce5f10d40 0x556ce5f131d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.568+0000 7ff96b17e740 1 --2- 172.21.15.12:0/4153078187 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556ce5f0bbd0 0x556ce5f106d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.568+0000 7ff96b17e740 1 --2- 172.21.15.12:0/4153078187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0b1e0 0x556ce5f0b600 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.568+0000 7ff96b17e740 1 -- 172.21.15.12:0/4153078187 >> 172.21.15.12:0/4153078187 conn(0x556ce5f06750 msgr2=0x556ce5f06b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.568+0000 7ff965a00640 1 --2- 172.21.15.12:0/3902826320 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff9480a28a0 0x7ff9480a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.568+0000 7ff96b17e740 1 -- 172.21.15.12:0/4153078187 shutdown_connections 2024-03-23T23:08:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.568+0000 7ff96b17e740 1 -- 172.21.15.12:0/4153078187 wait complete. 2024-03-23T23:08:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.568+0000 7ff96b17e740 1 -- start start 2024-03-23T23:08:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.568+0000 7ff965a00640 1 --2- 172.21.15.12:0/3902826320 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff9480a28a0 0x7ff9480a4d60 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x556ce5f076c0 tx=0x7ff954005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.569+0000 7ff96b17e740 1 Processor -- start 2024-03-23T23:08:47.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.569+0000 7ff96b17e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ce5f0b1e0 0x556ce5f15360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:47.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.569+0000 7ff96b17e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0bbd0 0x556ce5f158a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:47.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.569+0000 7ff96b17e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556ce5f10d40 0x556ce5f1c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:47.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.569+0000 7ff96b17e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ce5e2ca20 con 0x556ce5f10d40 2024-03-23T23:08:47.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.569+0000 7ff96b17e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ce5e6c8d0 con 0x556ce5f0b1e0 2024-03-23T23:08:47.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.569+0000 7ff96b17e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ce5f13c90 con 0x556ce5f0bbd0 2024-03-23T23:08:47.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.569+0000 7ff96b17e740 10 client.0 _ll_register_callbacks cb 0x556ce5e691c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:47.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.569+0000 7ff965000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0bbd0 0x556ce5f158a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:47.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.569+0000 7ff965a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ce5f0b1e0 0x556ce5f15360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:47.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.569+0000 7ff965000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0bbd0 0x556ce5f158a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:36636/0 (socket says 172.21.15.12:36636) 2024-03-23T23:08:47.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.569+0000 7ff965000640 1 -- 172.21.15.12:0/3316695893 learned_addr learned my addr 172.21.15.12:0/3316695893 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:47.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.570+0000 7ff966400640 1 --2- 172.21.15.12:0/3316695893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556ce5f10d40 0x556ce5f1c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:47.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.570+0000 7ff965a00640 1 -- 172.21.15.12:0/3316695893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0bbd0 msgr2=0x556ce5f158a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:47.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.570+0000 7ff965a00640 1 --2- 172.21.15.12:0/3316695893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0bbd0 0x556ce5f158a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.570+0000 7ff965a00640 1 -- 172.21.15.12:0/3316695893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556ce5f10d40 msgr2=0x556ce5f1c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:47.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.570+0000 7ff965a00640 1 --2- 172.21.15.12:0/3316695893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556ce5f10d40 0x556ce5f1c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.570+0000 7ff965a00640 1 -- 172.21.15.12:0/3316695893 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556ce5b85210 con 0x556ce5f0b1e0 2024-03-23T23:08:47.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.570+0000 7ff966400640 1 --2- 172.21.15.12:0/3316695893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556ce5f10d40 0x556ce5f1c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:08:47.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.570+0000 7ff965000640 1 --2- 172.21.15.12:0/3316695893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0bbd0 0x556ce5f158a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:47.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.570+0000 7ff965a00640 1 --2- 172.21.15.12:0/3316695893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ce5f0b1e0 0x556ce5f15360 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x556ce5f0c640 tx=0x7ff954004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a8d2760c7f61d0e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:47.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.570+0000 7ff947400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:08:47.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.570+0000 7ff947400640 1 -- 172.21.15.12:0/3316695893 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff954029440 con 0x556ce5f0b1e0 2024-03-23T23:08:47.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.571+0000 7ff947400640 1 -- 172.21.15.12:0/3316695893 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff95401f420 con 0x556ce5f0b1e0 2024-03-23T23:08:47.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.571+0000 7ff96b17e740 1 -- 172.21.15.12:0/3316695893 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff958032c90 con 0x556ce5f0b1e0 2024-03-23T23:08:47.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.571+0000 7ff947400640 1 -- 172.21.15.12:0/3316695893 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff95401f620 con 0x556ce5f0b1e0 2024-03-23T23:08:47.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.571+0000 7ff96b17e740 10 client.27296 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:47.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.571+0000 7ff96b17e740 1 -- 172.21.15.12:0/3316695893 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556ce5c51620 con 0x556ce5f0b1e0 2024-03-23T23:08:47.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.571+0000 7ff947400640 1 -- 172.21.15.12:0/3316695893 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff958032c90 con 0x556ce5f0b1e0 2024-03-23T23:08:47.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.571+0000 7ff947400640 1 -- 172.21.15.12:0/3316695893 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ff954027070 con 0x556ce5f0b1e0 2024-03-23T23:08:47.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.571+0000 7ff96b17e740 10 client.27296 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:47.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.571+0000 7ff96b17e740 10 client.27296 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:08:47.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.571+0000 7ff96b17e740 20 client.27296 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:47.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.571+0000 7ff96b17e740 1 --2- 172.21.15.12:0/3316695893 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x556ce5f211e0 0x556ce5f235f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:47.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.571+0000 7ff96b17e740 4 client.27296 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:08:47.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.571+0000 7ff96b17e740 1 -- 172.21.15.12:0/3316695893 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556ce5e7dd70 con 0x556ce5f211e0 2024-03-23T23:08:47.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.572+0000 7ff965000640 1 --2- 172.21.15.12:0/3316695893 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x556ce5f211e0 0x556ce5f235f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:47.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.572+0000 7ff965000640 1 --2- 172.21.15.12:0/3316695893 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x556ce5f211e0 0x556ce5f235f0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6b6419f944765ce server_cookie=d94f2d66d522442b in_seq=0 out_seq=0 2024-03-23T23:08:47.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.572+0000 7ff947400640 10 client.27296 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:47.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.573+0000 7ff947400640 1 -- 172.21.15.12:0/3316695893 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556ce5e7dd70 con 0x556ce5f211e0 2024-03-23T23:08:47.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.573+0000 7ff947400640 10 client.27296 handle_command_reply: tid=0 2024-03-23T23:08:47.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.573+0000 7ff96b17e740 1 client.27296 shutdown 2024-03-23T23:08:47.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.573+0000 7ff96b17e740 1 -- 172.21.15.12:0/3316695893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ce5f0b1e0 msgr2=0x556ce5f15360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:47.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.573+0000 7ff96b17e740 1 --2- 172.21.15.12:0/3316695893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ce5f0b1e0 0x556ce5f15360 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x556ce5f0c640 tx=0x7ff954004730 comp rx=0 tx=0).stop 2024-03-23T23:08:47.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.573+0000 7ff96b17e740 1 -- 172.21.15.12:0/3316695893 shutdown_connections 2024-03-23T23:08:47.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.573+0000 7ff96b17e740 1 --2- 172.21.15.12:0/3316695893 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x556ce5f211e0 0x556ce5f235f0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.573+0000 7ff96b17e740 1 --2- 172.21.15.12:0/3316695893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556ce5f10d40 0x556ce5f1c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.573+0000 7ff96b17e740 1 --2- 172.21.15.12:0/3316695893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0bbd0 0x556ce5f158a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.573+0000 7ff96b17e740 1 --2- 172.21.15.12:0/3316695893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ce5f0b1e0 0x556ce5f15360 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.573+0000 7ff947400640 0 client.27296 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:47.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.573+0000 7ff96b17e740 1 -- 172.21.15.12:0/3316695893 >> 172.21.15.12:0/3316695893 conn(0x556ce5f06750 msgr2=0x556ce5f05950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:47.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.574+0000 7ff96b17e740 1 -- 172.21.15.12:0/3316695893 shutdown_connections 2024-03-23T23:08:47.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.574+0000 7ff96b17e740 1 -- 172.21.15.12:0/3316695893 wait complete. 2024-03-23T23:08:47.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.574+0000 7ff96b17e740 20 client.27296 trim_cache size 0 max 16384 2024-03-23T23:08:47.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.589+0000 7ff96b17e740 1 Processor -- start 2024-03-23T23:08:47.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.589+0000 7ff96b17e740 1 -- start start 2024-03-23T23:08:47.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.589+0000 7ff96b17e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556ce5f0b1e0 0x556ce5f235d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:47.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.589+0000 7ff96b17e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0bbd0 0x556ce5f1c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:47.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.590+0000 7ff96b17e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ce5f10d40 0x556ce5f15680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:47.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.590+0000 7ff96b17e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9601b1570 con 0x556ce5f0b1e0 2024-03-23T23:08:47.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.590+0000 7ff96b17e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9601af070 con 0x556ce5f10d40 2024-03-23T23:08:47.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.590+0000 7ff96b17e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9601ad110 con 0x556ce5f0bbd0 2024-03-23T23:08:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.590+0000 7ff965a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556ce5f0b1e0 0x556ce5f235d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.590+0000 7ff965a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556ce5f0b1e0 0x556ce5f235d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42686/0 (socket says 172.21.15.12:42686) 2024-03-23T23:08:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.590+0000 7ff965a00640 1 -- 172.21.15.12:0/3931585504 learned_addr learned my addr 172.21.15.12:0/3931585504 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.590+0000 7ff966400640 1 --2- 172.21.15.12:0/3931585504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ce5f10d40 0x556ce5f15680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.590+0000 7ff965000640 1 --2- 172.21.15.12:0/3931585504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0bbd0 0x556ce5f1c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.590+0000 7ff965a00640 1 -- 172.21.15.12:0/3931585504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0bbd0 msgr2=0x556ce5f1c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.590+0000 7ff965a00640 1 --2- 172.21.15.12:0/3931585504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0bbd0 0x556ce5f1c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.590+0000 7ff965a00640 1 -- 172.21.15.12:0/3931585504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ce5f10d40 msgr2=0x556ce5f15680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.590+0000 7ff965a00640 1 --2- 172.21.15.12:0/3931585504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ce5f10d40 0x556ce5f15680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.590+0000 7ff965a00640 1 -- 172.21.15.12:0/3931585504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556ce5f0c460 con 0x556ce5f0b1e0 2024-03-23T23:08:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.590+0000 7ff966400640 1 --2- 172.21.15.12:0/3931585504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ce5f10d40 0x556ce5f15680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.590+0000 7ff965000640 1 --2- 172.21.15.12:0/3931585504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0bbd0 0x556ce5f1c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:08:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.591+0000 7ff965a00640 1 --2- 172.21.15.12:0/3931585504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556ce5f0b1e0 0x556ce5f235d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff954019040 tx=0x7ff95401f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec803a257ad5c411 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.591+0000 7ff952000640 1 -- 172.21.15.12:0/3931585504 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff954038030 con 0x556ce5f0b1e0 2024-03-23T23:08:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.591+0000 7ff952000640 1 -- 172.21.15.12:0/3931585504 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff95402f030 con 0x556ce5f0b1e0 2024-03-23T23:08:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.591+0000 7ff952000640 1 -- 172.21.15.12:0/3931585504 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95401fd00 con 0x556ce5f0b1e0 2024-03-23T23:08:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.592+0000 7ff96b17e740 1 -- 172.21.15.12:0/3931585504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556ce5f0b1e0 msgr2=0x556ce5f235d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.592+0000 7ff96b17e740 1 --2- 172.21.15.12:0/3931585504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556ce5f0b1e0 0x556ce5f235d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff954019040 tx=0x7ff95401f5b0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.592+0000 7ff96b17e740 1 -- 172.21.15.12:0/3931585504 shutdown_connections 2024-03-23T23:08:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.592+0000 7ff96b17e740 1 --2- 172.21.15.12:0/3931585504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ce5f10d40 0x556ce5f15680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.592+0000 7ff96b17e740 1 --2- 172.21.15.12:0/3931585504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0bbd0 0x556ce5f1c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.592+0000 7ff96b17e740 1 --2- 172.21.15.12:0/3931585504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556ce5f0b1e0 0x556ce5f235d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.592+0000 7ff96b17e740 1 -- 172.21.15.12:0/3931585504 >> 172.21.15.12:0/3931585504 conn(0x556ce5f06750 msgr2=0x556ce5f14f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.592+0000 7ff96b17e740 1 -- 172.21.15.12:0/3931585504 shutdown_connections 2024-03-23T23:08:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.592+0000 7ff96b17e740 1 -- 172.21.15.12:0/3931585504 wait complete. 2024-03-23T23:08:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.592+0000 7ff96b17e740 1 -- start start 2024-03-23T23:08:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.592+0000 7ff96b17e740 1 Processor -- start 2024-03-23T23:08:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.593+0000 7ff96b17e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0b1e0 0x556ce5f1a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.593+0000 7ff96b17e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556ce5f0bbd0 0x556ce5f1ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.593+0000 7ff96b17e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ce5f10d40 0x556ce5f1bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.593+0000 7ff96b17e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9601ad710 con 0x556ce5f0bbd0 2024-03-23T23:08:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.593+0000 7ff96b17e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9601aea10 con 0x556ce5f10d40 2024-03-23T23:08:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.593+0000 7ff96b17e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9601add50 con 0x556ce5f0b1e0 2024-03-23T23:08:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.593+0000 7ff96b17e740 10 client.0 _ll_register_callbacks cb 0x7ff954012ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:08:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.593+0000 7ff965000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556ce5f0bbd0 0x556ce5f1ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.593+0000 7ff965a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0b1e0 0x556ce5f1a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.593+0000 7ff965a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0b1e0 0x556ce5f1a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:36654/0 (socket says 172.21.15.12:36654) 2024-03-23T23:08:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.593+0000 7ff965a00640 1 -- 172.21.15.12:0/1340679244 learned_addr learned my addr 172.21.15.12:0/1340679244 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:08:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.593+0000 7ff965000640 1 --2- 172.21.15.12:0/1340679244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556ce5f0bbd0 0x556ce5f1ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42698/0 (socket says 172.21.15.12:42698) 2024-03-23T23:08:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.593+0000 7ff966400640 1 --2- 172.21.15.12:0/1340679244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ce5f10d40 0x556ce5f1bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.594+0000 7ff965a00640 1 -- 172.21.15.12:0/1340679244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ce5f10d40 msgr2=0x556ce5f1bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.594+0000 7ff965a00640 1 --2- 172.21.15.12:0/1340679244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ce5f10d40 0x556ce5f1bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.594+0000 7ff965a00640 1 -- 172.21.15.12:0/1340679244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556ce5f0bbd0 msgr2=0x556ce5f1ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:08:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.594+0000 7ff965a00640 1 --2- 172.21.15.12:0/1340679244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556ce5f0bbd0 0x556ce5f1ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.594+0000 7ff965a00640 1 -- 172.21.15.12:0/1340679244 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556ce5f1c370 con 0x556ce5f0b1e0 2024-03-23T23:08:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.594+0000 7ff965a00640 1 --2- 172.21.15.12:0/1340679244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0b1e0 0x556ce5f1a830 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff954003b10 tx=0x7ff954029930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a80c68cced551a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:08:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.594+0000 7ff947400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:08:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.594+0000 7ff947400640 1 -- 172.21.15.12:0/1340679244 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff954040020 con 0x556ce5f0b1e0 2024-03-23T23:08:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.594+0000 7ff96b17e740 1 -- 172.21.15.12:0/1340679244 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff95402f030 con 0x556ce5f0b1e0 2024-03-23T23:08:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.594+0000 7ff947400640 1 -- 172.21.15.12:0/1340679244 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff95401fc00 con 0x556ce5f0b1e0 2024-03-23T23:08:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.594+0000 7ff947400640 1 -- 172.21.15.12:0/1340679244 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff954023030 con 0x556ce5f0b1e0 2024-03-23T23:08:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.594+0000 7ff96b17e740 10 client.27274 fetch_fsmap learned FSMap version 23 2024-03-23T23:08:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.594+0000 7ff96b17e740 1 -- 172.21.15.12:0/1340679244 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556ce5f7d370 con 0x556ce5f0b1e0 2024-03-23T23:08:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.595+0000 7ff947400640 1 -- 172.21.15.12:0/1340679244 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff95402f030 con 0x556ce5f0b1e0 2024-03-23T23:08:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.595+0000 7ff947400640 1 -- 172.21.15.12:0/1340679244 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ff954027040 con 0x556ce5f0b1e0 2024-03-23T23:08:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.595+0000 7ff96b17e740 10 client.27274 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:08:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.595+0000 7ff96b17e740 10 client.27274 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:08:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.595+0000 7ff96b17e740 20 client.27274 populate_metadata read hostname 'smithi012' 2024-03-23T23:08:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.595+0000 7ff96b17e740 1 --2- 172.21.15.12:0/1340679244 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x556ce5f7dc10 0x556ce5f80030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:08:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.595+0000 7ff96b17e740 4 client.27274 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:08:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.595+0000 7ff96b17e740 1 -- 172.21.15.12:0/1340679244 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x556ce5e780c0 con 0x556ce5f7dc10 2024-03-23T23:08:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.595+0000 7ff965000640 1 --2- 172.21.15.12:0/1340679244 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x556ce5f7dc10 0x556ce5f80030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:08:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.596+0000 7ff965000640 1 --2- 172.21.15.12:0/1340679244 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x556ce5f7dc10 0x556ce5f80030 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c90158a3cacdaf5 server_cookie=43094741a02668cb in_seq=0 out_seq=0 2024-03-23T23:08:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.596+0000 7ff947400640 10 client.27274 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:47.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.609+0000 7ff947400640 1 -- 172.21.15.12:0/1340679244 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x556ce5e780c0 con 0x556ce5f7dc10 2024-03-23T23:08:47.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.609+0000 7ff947400640 10 client.27274 handle_command_reply: tid=0 2024-03-23T23:08:47.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.609+0000 7ff96b17e740 1 client.27274 shutdown 2024-03-23T23:08:47.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.609+0000 7ff96b17e740 1 -- 172.21.15.12:0/1340679244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0b1e0 msgr2=0x556ce5f1a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:47.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.609+0000 7ff96b17e740 1 --2- 172.21.15.12:0/1340679244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0b1e0 0x556ce5f1a830 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff954003b10 tx=0x7ff954029930 comp rx=0 tx=0).stop 2024-03-23T23:08:47.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.609+0000 7ff96b17e740 1 -- 172.21.15.12:0/1340679244 shutdown_connections 2024-03-23T23:08:47.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.609+0000 7ff96b17e740 1 --2- 172.21.15.12:0/1340679244 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x556ce5f7dc10 0x556ce5f80030 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.609+0000 7ff96b17e740 1 --2- 172.21.15.12:0/1340679244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ce5f10d40 0x556ce5f1bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.609+0000 7ff96b17e740 1 --2- 172.21.15.12:0/1340679244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556ce5f0bbd0 0x556ce5f1ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.609+0000 7ff96b17e740 1 --2- 172.21.15.12:0/1340679244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ce5f0b1e0 0x556ce5f1a830 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.609+0000 7ff947400640 0 client.27274 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:08:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.609+0000 7ff96b17e740 1 -- 172.21.15.12:0/1340679244 >> 172.21.15.12:0/1340679244 conn(0x556ce5f06750 msgr2=0x556ce5f14f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.609+0000 7ff96b17e740 1 -- 172.21.15.12:0/1340679244 shutdown_connections 2024-03-23T23:08:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.609+0000 7ff96b17e740 1 -- 172.21.15.12:0/1340679244 wait complete. 2024-03-23T23:08:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.609+0000 7ff96b17e740 20 client.27274 trim_cache size 0 max 16384 2024-03-23T23:08:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:08:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:08:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:08:47.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:08:47.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.610+0000 7ff968400640 1 -- 172.21.15.12:0/3902826320 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff9480a28a0 msgr2=0x7ff9480a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:47.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.610+0000 7ff968400640 1 --2- 172.21.15.12:0/3902826320 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff9480a28a0 0x7ff9480a4d60 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x556ce5f076c0 tx=0x7ff954005f90 comp rx=0 tx=0).stop 2024-03-23T23:08:47.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.610+0000 7ff968400640 1 -- 172.21.15.12:0/3902826320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff960108ef0 msgr2=0x7ff96019e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:08:47.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.610+0000 7ff968400640 1 --2- 172.21.15.12:0/3902826320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff960108ef0 0x7ff96019e450 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff94c00ea60 tx=0x7ff94c00ef20 comp rx=0 tx=0).stop 2024-03-23T23:08:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.610+0000 7ff968400640 1 -- 172.21.15.12:0/3902826320 shutdown_connections 2024-03-23T23:08:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.610+0000 7ff968400640 1 --2- 172.21.15.12:0/3902826320 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff9480a28a0 0x7ff9480a4d60 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.610+0000 7ff968400640 1 --2- 172.21.15.12:0/3902826320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9601098b0 0x7ff9601a54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.610+0000 7ff968400640 1 --2- 172.21.15.12:0/3902826320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff960108ef0 0x7ff96019e450 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.611+0000 7ff968400640 1 --2- 172.21.15.12:0/3902826320 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff960107d00 0x7ff96019df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:08:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.611+0000 7ff968400640 1 -- 172.21.15.12:0/3902826320 >> 172.21.15.12:0/3902826320 conn(0x7ff960076cb0 msgr2=0x7ff96010c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:08:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.611+0000 7ff968400640 1 -- 172.21.15.12:0/3902826320 shutdown_connections 2024-03-23T23:08:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:08:47.611+0000 7ff968400640 1 -- 172.21.15.12:0/3902826320 wait complete. 2024-03-23T23:08:48.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:48 smithi173 ceph-mon[36116]: pgmap v600: 97 pgs: 97 active+clean; 4.5 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:08:48.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:48 smithi203 ceph-mon[36693]: pgmap v600: 97 pgs: 97 active+clean; 4.5 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:08:49.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:48 smithi012 ceph-mon[31695]: pgmap v600: 97 pgs: 97 active+clean; 4.5 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:08:50.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:50 smithi173 ceph-mon[36116]: pgmap v601: 97 pgs: 97 active+clean; 4.5 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 16 op/s 2024-03-23T23:08:50.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:50 smithi203 ceph-mon[36693]: pgmap v601: 97 pgs: 97 active+clean; 4.5 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 16 op/s 2024-03-23T23:08:51.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:50 smithi012 ceph-mon[31695]: pgmap v601: 97 pgs: 97 active+clean; 4.5 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 16 op/s 2024-03-23T23:08:52.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:52 smithi173 ceph-mon[36116]: pgmap v602: 97 pgs: 97 active+clean; 4.5 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:08:52.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:52 smithi203 ceph-mon[36693]: pgmap v602: 97 pgs: 97 active+clean; 4.5 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:08:53.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:52 smithi012 ceph-mon[31695]: pgmap v602: 97 pgs: 97 active+clean; 4.5 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:08:54.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:54 smithi173 ceph-mon[36116]: pgmap v603: 97 pgs: 97 active+clean; 4.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 15 op/s 2024-03-23T23:08:54.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:54 smithi203 ceph-mon[36693]: pgmap v603: 97 pgs: 97 active+clean; 4.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 15 op/s 2024-03-23T23:08:55.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:54 smithi012 ceph-mon[31695]: pgmap v603: 97 pgs: 97 active+clean; 4.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 15 op/s 2024-03-23T23:08:56.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:56 smithi173 ceph-mon[36116]: pgmap v604: 97 pgs: 97 active+clean; 4.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 15 op/s 2024-03-23T23:08:56.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:56 smithi203 ceph-mon[36693]: pgmap v604: 97 pgs: 97 active+clean; 4.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 15 op/s 2024-03-23T23:08:57.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:56 smithi012 ceph-mon[31695]: pgmap v604: 97 pgs: 97 active+clean; 4.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 15 op/s 2024-03-23T23:08:58.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:08:58 smithi173 ceph-mon[36116]: pgmap v605: 97 pgs: 97 active+clean; 4.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 17 KiB/s wr, 14 op/s 2024-03-23T23:08:58.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:08:58 smithi203 ceph-mon[36693]: pgmap v605: 97 pgs: 97 active+clean; 4.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 17 KiB/s wr, 14 op/s 2024-03-23T23:08:59.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:08:58 smithi012 ceph-mon[31695]: pgmap v605: 97 pgs: 97 active+clean; 4.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 17 KiB/s wr, 14 op/s 2024-03-23T23:09:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.757+0000 7faa8ce00640 1 -- 172.21.15.12:0/2313839701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa88102140 msgr2=0x7faa88104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.757+0000 7faa8ce00640 1 --2- 172.21.15.12:0/2313839701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa88102140 0x7faa88104530 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7faa70009900 tx=0x7faa7002bc40 comp rx=0 tx=0).stop 2024-03-23T23:09:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.757+0000 7faa8ce00640 1 -- 172.21.15.12:0/2313839701 shutdown_connections 2024-03-23T23:09:00.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.757+0000 7faa8ce00640 1 --2- 172.21.15.12:0/2313839701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa881073a0 0x7faa881097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.757+0000 7faa8ce00640 1 --2- 172.21.15.12:0/2313839701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa88104a70 0x7faa88106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.757+0000 7faa8ce00640 1 --2- 172.21.15.12:0/2313839701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa88102140 0x7faa88104530 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.757+0000 7faa8ce00640 1 -- 172.21.15.12:0/2313839701 >> 172.21.15.12:0/2313839701 conn(0x7faa880fbb30 msgr2=0x7faa880fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:00.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.757+0000 7faa8ce00640 1 -- 172.21.15.12:0/2313839701 shutdown_connections 2024-03-23T23:09:00.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.757+0000 7faa8ce00640 1 -- 172.21.15.12:0/2313839701 wait complete. 2024-03-23T23:09:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.758+0000 7faa8ce00640 1 Processor -- start 2024-03-23T23:09:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.759+0000 7faa8ce00640 1 -- start start 2024-03-23T23:09:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.759+0000 7faa8ce00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa88102140 0x7faa88197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.759+0000 7faa8ce00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa88104a70 0x7faa88197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.759+0000 7faa8ce00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa881073a0 0x7faa8819ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.759+0000 7faa8ce00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8810d3b0 con 0x7faa88104a70 2024-03-23T23:09:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.759+0000 7faa8ce00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8810d230 con 0x7faa881073a0 2024-03-23T23:09:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.759+0000 7faa8ce00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8810d530 con 0x7faa88102140 2024-03-23T23:09:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.759+0000 7faa85400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa88104a70 0x7faa88197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.759+0000 7faa85e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa88102140 0x7faa88197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.759+0000 7faa85e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa88102140 0x7faa88197960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49656/0 (socket says 172.21.15.12:49656) 2024-03-23T23:09:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.759+0000 7faa86800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa881073a0 0x7faa8819ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.759+0000 7faa85400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa88104a70 0x7faa88197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35806/0 (socket says 172.21.15.12:35806) 2024-03-23T23:09:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.759+0000 7faa85e00640 1 -- 172.21.15.12:0/581009664 learned_addr learned my addr 172.21.15.12:0/581009664 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.760+0000 7faa85e00640 1 -- 172.21.15.12:0/581009664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa881073a0 msgr2=0x7faa8819ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.760+0000 7faa85e00640 1 --2- 172.21.15.12:0/581009664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa881073a0 0x7faa8819ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.760+0000 7faa85e00640 1 -- 172.21.15.12:0/581009664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa88104a70 msgr2=0x7faa88197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.760+0000 7faa85e00640 1 --2- 172.21.15.12:0/581009664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa88104a70 0x7faa88197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.760+0000 7faa85e00640 1 -- 172.21.15.12:0/581009664 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8819f580 con 0x7faa88102140 2024-03-23T23:09:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.760+0000 7faa86800640 1 --2- 172.21.15.12:0/581009664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa881073a0 0x7faa8819ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.760+0000 7faa85e00640 1 --2- 172.21.15.12:0/581009664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa88102140 0x7faa88197960 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7faa7002c8f0 tx=0x7faa7002c5e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.760+0000 7faa7ea00640 1 -- 172.21.15.12:0/581009664 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7004a070 con 0x7faa88102140 2024-03-23T23:09:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.760+0000 7faa7ea00640 1 -- 172.21.15.12:0/581009664 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faa70004450 con 0x7faa88102140 2024-03-23T23:09:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.760+0000 7faa8ce00640 1 -- 172.21.15.12:0/581009664 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa880fff30 con 0x7faa88102140 2024-03-23T23:09:00.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.760+0000 7faa7ea00640 1 -- 172.21.15.12:0/581009664 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa70003e30 con 0x7faa88102140 2024-03-23T23:09:00.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.761+0000 7faa8ce00640 1 -- 172.21.15.12:0/581009664 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa88100420 con 0x7faa88102140 2024-03-23T23:09:00.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.764+0000 7faa7ea00640 1 -- 172.21.15.12:0/581009664 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7faa7004d070 con 0x7faa88102140 2024-03-23T23:09:00.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.764+0000 7faa8fb72740 1 Processor -- start 2024-03-23T23:09:00.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.764+0000 7faa8fb72740 1 -- start start 2024-03-23T23:09:00.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.764+0000 7faa8fb72740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558abcf681e0 0x558abcf68600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:00.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.764+0000 7faa8fb72740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf68bd0 0x558abcf6d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:00.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.764+0000 7faa8fb72740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558abcf6dd40 0x558abcf701d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:00.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.764+0000 7faa8fb72740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x558abcf2b3b0 con 0x558abcf681e0 2024-03-23T23:09:00.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.764+0000 7faa8fb72740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x558abcecd830 con 0x558abcf68bd0 2024-03-23T23:09:00.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.764+0000 7faa8fb72740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x558abce765d0 con 0x558abcf6dd40 2024-03-23T23:09:00.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.765+0000 7faa86800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558abcf681e0 0x558abcf68600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:00.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.765+0000 7faa85e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf68bd0 0x558abcf6d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.765+0000 7faa86800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558abcf681e0 0x558abcf68600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35812/0 (socket says 172.21.15.12:35812) 2024-03-23T23:09:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.765+0000 7faa86800640 1 -- 172.21.15.12:0/4175884971 learned_addr learned my addr 172.21.15.12:0/4175884971 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.765+0000 7faa85e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf68bd0 0x558abcf6d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:39666/0 (socket says 172.21.15.12:39666) 2024-03-23T23:09:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.765+0000 7faa85400640 1 --2- 172.21.15.12:0/4175884971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558abcf6dd40 0x558abcf701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.765+0000 7faa85e00640 1 -- 172.21.15.12:0/4175884971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558abcf6dd40 msgr2=0x558abcf701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.765+0000 7faa85e00640 1 --2- 172.21.15.12:0/4175884971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558abcf6dd40 0x558abcf701d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.765+0000 7faa85e00640 1 -- 172.21.15.12:0/4175884971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558abcf681e0 msgr2=0x558abcf68600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.765+0000 7faa85e00640 1 --2- 172.21.15.12:0/4175884971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558abcf681e0 0x558abcf68600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.765+0000 7faa85e00640 1 -- 172.21.15.12:0/4175884971 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558abce8e2c0 con 0x558abcf68bd0 2024-03-23T23:09:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.765+0000 7faa85e00640 1 --2- 172.21.15.12:0/4175884971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf68bd0 0x558abcf6d6d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7faa700f1ca0 tx=0x7faa700fc2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=218c0512d912948e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.765+0000 7faa7ea00640 1 --2- 172.21.15.12:0/581009664 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7faa6c0a28a0 0x7faa6c0a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.766+0000 7faa77400640 1 -- 172.21.15.12:0/4175884971 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa700bcef0 con 0x558abcf68bd0 2024-03-23T23:09:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.766+0000 7faa86800640 1 --2- 172.21.15.12:0/581009664 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7faa6c0a28a0 0x7faa6c0a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.766+0000 7faa77400640 1 -- 172.21.15.12:0/4175884971 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faa70101030 con 0x558abcf68bd0 2024-03-23T23:09:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.766+0000 7faa77400640 1 -- 172.21.15.12:0/4175884971 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa700c8ca0 con 0x558abcf68bd0 2024-03-23T23:09:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.766+0000 7faa7ea00640 1 -- 172.21.15.12:0/581009664 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7faa700f4250 con 0x7faa88102140 2024-03-23T23:09:00.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.766+0000 7faa8fb72740 1 -- 172.21.15.12:0/4175884971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf68bd0 msgr2=0x558abcf6d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:00.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.766+0000 7faa8fb72740 1 --2- 172.21.15.12:0/4175884971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf68bd0 0x558abcf6d6d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7faa700f1ca0 tx=0x7faa700fc2d0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.766+0000 7faa86800640 1 --2- 172.21.15.12:0/581009664 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7faa6c0a28a0 0x7faa6c0a4d60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x558abcf69430 tx=0x7faa80005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:00.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.767+0000 7faa8fb72740 1 -- 172.21.15.12:0/4175884971 shutdown_connections 2024-03-23T23:09:00.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.767+0000 7faa8fb72740 1 --2- 172.21.15.12:0/4175884971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558abcf6dd40 0x558abcf701d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.767+0000 7faa8fb72740 1 --2- 172.21.15.12:0/4175884971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf68bd0 0x558abcf6d6d0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.767+0000 7faa8fb72740 1 --2- 172.21.15.12:0/4175884971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558abcf681e0 0x558abcf68600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.767+0000 7faa8fb72740 1 -- 172.21.15.12:0/4175884971 >> 172.21.15.12:0/4175884971 conn(0x558abcf63750 msgr2=0x558abcf63b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:00.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.767+0000 7faa8fb72740 1 -- 172.21.15.12:0/4175884971 shutdown_connections 2024-03-23T23:09:00.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.767+0000 7faa8fb72740 1 -- 172.21.15.12:0/4175884971 wait complete. 2024-03-23T23:09:00.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.767+0000 7faa8fb72740 1 -- start start 2024-03-23T23:09:00.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.767+0000 7faa8fb72740 1 Processor -- start 2024-03-23T23:09:00.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.768+0000 7faa8fb72740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf681e0 0x558abcf72050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:00.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.768+0000 7faa8fb72740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558abcf68bd0 0x558abcf72590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:00.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.768+0000 7faa8fb72740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558abcf6dd40 0x558abcf795d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:00.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.768+0000 7faa8fb72740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x558abce89a20 con 0x558abcf68bd0 2024-03-23T23:09:00.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.768+0000 7faa8fb72740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x558abcec98d0 con 0x558abcf681e0 2024-03-23T23:09:00.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.768+0000 7faa8fb72740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x558abcf70ab0 con 0x558abcf6dd40 2024-03-23T23:09:00.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.768+0000 7faa8fb72740 10 client.0 _ll_register_callbacks cb 0x558abcec61c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:00.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.768+0000 7faa86800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558abcf6dd40 0x558abcf795d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:00.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.768+0000 7faa85e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf681e0 0x558abcf72050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:00.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.768+0000 7faa85400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558abcf68bd0 0x558abcf72590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:00.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.768+0000 7faa85e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf681e0 0x558abcf72050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:39668/0 (socket says 172.21.15.12:39668) 2024-03-23T23:09:00.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.768+0000 7faa85e00640 1 -- 172.21.15.12:0/3777762658 learned_addr learned my addr 172.21.15.12:0/3777762658 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:00.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.768+0000 7faa85e00640 1 -- 172.21.15.12:0/3777762658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558abcf6dd40 msgr2=0x558abcf795d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:00.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.768+0000 7faa85e00640 1 --2- 172.21.15.12:0/3777762658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558abcf6dd40 0x558abcf795d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.769+0000 7faa85e00640 1 -- 172.21.15.12:0/3777762658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558abcf68bd0 msgr2=0x558abcf72590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:00.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.769+0000 7faa85e00640 1 --2- 172.21.15.12:0/3777762658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558abcf68bd0 0x558abcf72590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.769+0000 7faa85e00640 1 -- 172.21.15.12:0/3777762658 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558abcbe2210 con 0x558abcf681e0 2024-03-23T23:09:00.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.769+0000 7faa85e00640 1 --2- 172.21.15.12:0/3777762658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf681e0 0x558abcf72050 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7faa700c0950 tx=0x7faa700c9050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7983ff3babc5403c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:00.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.769+0000 7faa6be00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:09:00.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.769+0000 7faa6be00640 1 -- 172.21.15.12:0/3777762658 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa700c8eb0 con 0x558abcf681e0 2024-03-23T23:09:00.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.769+0000 7faa6be00640 1 -- 172.21.15.12:0/3777762658 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7faa700bef50 con 0x558abcf681e0 2024-03-23T23:09:00.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.769+0000 7faa8fb72740 1 -- 172.21.15.12:0/3777762658 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faa70101030 con 0x558abcf681e0 2024-03-23T23:09:00.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.769+0000 7faa6be00640 1 -- 172.21.15.12:0/3777762658 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7faa700c90f0 con 0x558abcf681e0 2024-03-23T23:09:00.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.769+0000 7faa8fb72740 10 client.27314 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.769+0000 7faa8fb72740 1 -- 172.21.15.12:0/3777762658 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x558abccae620 con 0x558abcf681e0 2024-03-23T23:09:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.769+0000 7faa6be00640 1 -- 172.21.15.12:0/3777762658 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faa70101030 con 0x558abcf681e0 2024-03-23T23:09:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.770+0000 7faa6be00640 1 -- 172.21.15.12:0/3777762658 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7faa700bd2c0 con 0x558abcf681e0 2024-03-23T23:09:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.770+0000 7faa8fb72740 10 client.27314 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.770+0000 7faa8fb72740 10 client.27314 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.770+0000 7faa8fb72740 20 client.27314 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.770+0000 7faa8fb72740 1 --2- 172.21.15.12:0/3777762658 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558abcf7dbd0 0x558abcf7ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.770+0000 7faa8fb72740 4 client.27314 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:09:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.770+0000 7faa8fb72740 1 -- 172.21.15.12:0/3777762658 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558abcedad70 con 0x558abcf7dbd0 2024-03-23T23:09:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.770+0000 7faa85400640 1 --2- 172.21.15.12:0/3777762658 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558abcf7dbd0 0x558abcf7ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.770+0000 7faa85400640 1 --2- 172.21.15.12:0/3777762658 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558abcf7dbd0 0x558abcf7ffc0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1afe9641147b50c7 server_cookie=2a669746c111be6 in_seq=0 out_seq=0 2024-03-23T23:09:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.770+0000 7faa6be00640 10 client.27314 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.771+0000 7faa6be00640 1 -- 172.21.15.12:0/3777762658 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558abcedad70 con 0x558abcf7dbd0 2024-03-23T23:09:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.771+0000 7faa6be00640 10 client.27314 handle_command_reply: tid=0 2024-03-23T23:09:00.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.771+0000 7faa8fb72740 1 client.27314 shutdown 2024-03-23T23:09:00.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.771+0000 7faa8fb72740 1 -- 172.21.15.12:0/3777762658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf681e0 msgr2=0x558abcf72050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:00.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.771+0000 7faa8fb72740 1 --2- 172.21.15.12:0/3777762658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf681e0 0x558abcf72050 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7faa700c0950 tx=0x7faa700c9050 comp rx=0 tx=0).stop 2024-03-23T23:09:00.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.771+0000 7faa8fb72740 1 -- 172.21.15.12:0/3777762658 shutdown_connections 2024-03-23T23:09:00.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.771+0000 7faa8fb72740 1 --2- 172.21.15.12:0/3777762658 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558abcf7dbd0 0x558abcf7ffc0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.771+0000 7faa8fb72740 1 --2- 172.21.15.12:0/3777762658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558abcf6dd40 0x558abcf795d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.771+0000 7faa8fb72740 1 --2- 172.21.15.12:0/3777762658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558abcf68bd0 0x558abcf72590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.771+0000 7faa6be00640 0 client.27314 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:00.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.771+0000 7faa8fb72740 1 --2- 172.21.15.12:0/3777762658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf681e0 0x558abcf72050 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.772+0000 7faa8fb72740 1 -- 172.21.15.12:0/3777762658 >> 172.21.15.12:0/3777762658 conn(0x558abcf63750 msgr2=0x558abcf62950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:00.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.772+0000 7faa8fb72740 1 -- 172.21.15.12:0/3777762658 shutdown_connections 2024-03-23T23:09:00.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.772+0000 7faa8fb72740 1 -- 172.21.15.12:0/3777762658 wait complete. 2024-03-23T23:09:00.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.772+0000 7faa8fb72740 20 client.27314 trim_cache size 0 max 16384 2024-03-23T23:09:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.787+0000 7faa8fb72740 1 Processor -- start 2024-03-23T23:09:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.787+0000 7faa8fb72740 1 -- start start 2024-03-23T23:09:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.787+0000 7faa8fb72740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf63750 0x558abcf669b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.787+0000 7faa8fb72740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558abcf681e0 0x558abcf79590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.787+0000 7faa8fb72740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558abcf68bd0 0x558abcf66ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.787+0000 7faa8fb72740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa881a8bf0 con 0x558abcf68bd0 2024-03-23T23:09:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.787+0000 7faa8fb72740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa881a66f0 con 0x558abcf63750 2024-03-23T23:09:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.787+0000 7faa8fb72740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa881a4790 con 0x558abcf681e0 2024-03-23T23:09:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.788+0000 7faa85e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf63750 0x558abcf669b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.788+0000 7faa86800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558abcf68bd0 0x558abcf66ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.788+0000 7faa85400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558abcf681e0 0x558abcf79590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.788+0000 7faa85e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf63750 0x558abcf669b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:39682/0 (socket says 172.21.15.12:39682) 2024-03-23T23:09:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.788+0000 7faa85e00640 1 -- 172.21.15.12:0/2855321199 learned_addr learned my addr 172.21.15.12:0/2855321199 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.788+0000 7faa85400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558abcf681e0 0x558abcf79590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49674/0 (socket says 172.21.15.12:49674) 2024-03-23T23:09:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.788+0000 7faa85e00640 1 -- 172.21.15.12:0/2855321199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558abcf681e0 msgr2=0x558abcf79590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.788+0000 7faa85e00640 1 --2- 172.21.15.12:0/2855321199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558abcf681e0 0x558abcf79590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.788+0000 7faa85e00640 1 -- 172.21.15.12:0/2855321199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558abcf68bd0 msgr2=0x558abcf66ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.788+0000 7faa85e00640 1 --2- 172.21.15.12:0/2855321199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558abcf68bd0 0x558abcf66ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.788+0000 7faa85e00640 1 -- 172.21.15.12:0/2855321199 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558abcf644c0 con 0x558abcf63750 2024-03-23T23:09:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.788+0000 7faa85e00640 1 --2- 172.21.15.12:0/2855321199 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf63750 0x558abcf669b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7faa70053030 tx=0x7faa700fc2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28d038c6593eebc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.788+0000 7faa76a00640 1 -- 172.21.15.12:0/2855321199 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa700bf950 con 0x558abcf63750 2024-03-23T23:09:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.788+0000 7faa76a00640 1 -- 172.21.15.12:0/2855321199 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faa70101070 con 0x558abcf63750 2024-03-23T23:09:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.789+0000 7faa76a00640 1 -- 172.21.15.12:0/2855321199 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa700c05b0 con 0x558abcf63750 2024-03-23T23:09:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.789+0000 7faa8fb72740 1 -- 172.21.15.12:0/2855321199 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf63750 msgr2=0x558abcf669b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.789+0000 7faa8fb72740 1 --2- 172.21.15.12:0/2855321199 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf63750 0x558abcf669b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7faa70053030 tx=0x7faa700fc2d0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.789+0000 7faa8fb72740 1 -- 172.21.15.12:0/2855321199 shutdown_connections 2024-03-23T23:09:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.789+0000 7faa8fb72740 1 --2- 172.21.15.12:0/2855321199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558abcf68bd0 0x558abcf66ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.789+0000 7faa8fb72740 1 --2- 172.21.15.12:0/2855321199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558abcf681e0 0x558abcf79590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.789+0000 7faa8fb72740 1 --2- 172.21.15.12:0/2855321199 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf63750 0x558abcf669b0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.789+0000 7faa8fb72740 1 -- 172.21.15.12:0/2855321199 >> 172.21.15.12:0/2855321199 conn(0x558abcf6d810 msgr2=0x558abcf63b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.789+0000 7faa8fb72740 1 -- 172.21.15.12:0/2855321199 shutdown_connections 2024-03-23T23:09:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.789+0000 7faa8fb72740 1 -- 172.21.15.12:0/2855321199 wait complete. 2024-03-23T23:09:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.790+0000 7faa8fb72740 1 -- start start 2024-03-23T23:09:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.790+0000 7faa8fb72740 1 Processor -- start 2024-03-23T23:09:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.790+0000 7faa8fb72740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558abcf63750 0x558abcf769a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.790+0000 7faa8fb72740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf681e0 0x558abcf76ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.790+0000 7faa8fb72740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558abcf68bd0 0x558abcf77ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.790+0000 7faa8fb72740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa881a4d90 con 0x558abcf68bd0 2024-03-23T23:09:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.790+0000 7faa8fb72740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa881a6090 con 0x558abcf681e0 2024-03-23T23:09:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.790+0000 7faa8fb72740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa881a53d0 con 0x558abcf63750 2024-03-23T23:09:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.790+0000 7faa8fb72740 10 client.0 _ll_register_callbacks cb 0x7faa80009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.790+0000 7faa85e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558abcf63750 0x558abcf769a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.790+0000 7faa85400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf681e0 0x558abcf76ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.790+0000 7faa85e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558abcf63750 0x558abcf769a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49690/0 (socket says 172.21.15.12:49690) 2024-03-23T23:09:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.790+0000 7faa86800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558abcf68bd0 0x558abcf77ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.790+0000 7faa85e00640 1 -- 172.21.15.12:0/2114520569 learned_addr learned my addr 172.21.15.12:0/2114520569 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.791+0000 7faa85400640 1 -- 172.21.15.12:0/2114520569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558abcf63750 msgr2=0x558abcf769a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.791+0000 7faa85400640 1 --2- 172.21.15.12:0/2114520569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558abcf63750 0x558abcf769a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.791+0000 7faa85400640 1 -- 172.21.15.12:0/2114520569 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558abcf68bd0 msgr2=0x558abcf77ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.791+0000 7faa85400640 1 --2- 172.21.15.12:0/2114520569 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558abcf68bd0 0x558abcf77ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.791+0000 7faa85400640 1 -- 172.21.15.12:0/2114520569 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558abcf784e0 con 0x558abcf681e0 2024-03-23T23:09:00.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.791+0000 7faa85e00640 1 --2- 172.21.15.12:0/2114520569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558abcf63750 0x558abcf769a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:00.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.791+0000 7faa85400640 1 --2- 172.21.15.12:0/2114520569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf681e0 0x558abcf76ee0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x558abcf73530 tx=0x7faa78010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad33c77e100fe346 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:00.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.791+0000 7faa6be00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:09:00.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.791+0000 7faa6be00640 1 -- 172.21.15.12:0/2114520569 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa78024070 con 0x558abcf681e0 2024-03-23T23:09:00.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.791+0000 7faa8fb72740 1 -- 172.21.15.12:0/2114520569 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faa70101070 con 0x558abcf681e0 2024-03-23T23:09:00.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.791+0000 7faa6be00640 1 -- 172.21.15.12:0/2114520569 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7faa78020cb0 con 0x558abcf681e0 2024-03-23T23:09:00.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.792+0000 7faa6be00640 1 -- 172.21.15.12:0/2114520569 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7faa78030050 con 0x558abcf681e0 2024-03-23T23:09:00.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.792+0000 7faa8fb72740 10 client.27320 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:00.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.792+0000 7faa8fb72740 1 -- 172.21.15.12:0/2114520569 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x558abcf79860 con 0x558abcf681e0 2024-03-23T23:09:00.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.792+0000 7faa6be00640 1 -- 172.21.15.12:0/2114520569 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faa70101070 con 0x558abcf681e0 2024-03-23T23:09:00.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.792+0000 7faa6be00640 1 -- 172.21.15.12:0/2114520569 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7faa7801d480 con 0x558abcf681e0 2024-03-23T23:09:00.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.792+0000 7faa8fb72740 10 client.27320 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:00.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.792+0000 7faa8fb72740 10 client.27320 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:00.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.792+0000 7faa8fb72740 20 client.27320 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:00.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.792+0000 7faa8fb72740 1 --2- 172.21.15.12:0/2114520569 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558abcf6dd40 0x558abcfdbb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:00.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.792+0000 7faa8fb72740 4 client.27320 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:09:00.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.792+0000 7faa8fb72740 1 -- 172.21.15.12:0/2114520569 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558abced50c0 con 0x558abcf6dd40 2024-03-23T23:09:00.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.792+0000 7faa85e00640 1 --2- 172.21.15.12:0/2114520569 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558abcf6dd40 0x558abcfdbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:00.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.793+0000 7faa85e00640 1 --2- 172.21.15.12:0/2114520569 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558abcf6dd40 0x558abcfdbb80 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=870a667edb76b220 server_cookie=a231c651dae1bb9e in_seq=0 out_seq=0 2024-03-23T23:09:00.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.793+0000 7faa6be00640 10 client.27320 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.805+0000 7faa6be00640 1 -- 172.21.15.12:0/2114520569 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558abced50c0 con 0x558abcf6dd40 2024-03-23T23:09:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.805+0000 7faa6be00640 10 client.27320 handle_command_reply: tid=0 2024-03-23T23:09:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.805+0000 7faa8fb72740 1 client.27320 shutdown 2024-03-23T23:09:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.805+0000 7faa8fb72740 1 -- 172.21.15.12:0/2114520569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf681e0 msgr2=0x558abcf76ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.805+0000 7faa8fb72740 1 --2- 172.21.15.12:0/2114520569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf681e0 0x558abcf76ee0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x558abcf73530 tx=0x7faa78010800 comp rx=0 tx=0).stop 2024-03-23T23:09:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.805+0000 7faa8fb72740 1 -- 172.21.15.12:0/2114520569 shutdown_connections 2024-03-23T23:09:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.805+0000 7faa8fb72740 1 --2- 172.21.15.12:0/2114520569 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558abcf6dd40 0x558abcfdbb80 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.805+0000 7faa8fb72740 1 --2- 172.21.15.12:0/2114520569 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558abcf68bd0 0x558abcf77ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.805+0000 7faa8fb72740 1 --2- 172.21.15.12:0/2114520569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558abcf681e0 0x558abcf76ee0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.805+0000 7faa8fb72740 1 --2- 172.21.15.12:0/2114520569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558abcf63750 0x558abcf769a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.805+0000 7faa6be00640 0 client.27320 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.805+0000 7faa8fb72740 1 -- 172.21.15.12:0/2114520569 >> 172.21.15.12:0/2114520569 conn(0x558abcf6d810 msgr2=0x558abcf63b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.805+0000 7faa8fb72740 1 -- 172.21.15.12:0/2114520569 shutdown_connections 2024-03-23T23:09:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.805+0000 7faa8fb72740 1 -- 172.21.15.12:0/2114520569 wait complete. 2024-03-23T23:09:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.805+0000 7faa8fb72740 20 client.27320 trim_cache size 0 max 16384 2024-03-23T23:09:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:09:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:09:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:09:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:09:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.806+0000 7faa8ce00640 1 -- 172.21.15.12:0/581009664 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7faa6c0a28a0 msgr2=0x7faa6c0a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.806+0000 7faa8ce00640 1 --2- 172.21.15.12:0/581009664 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7faa6c0a28a0 0x7faa6c0a4d60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x558abcf69430 tx=0x7faa80005f90 comp rx=0 tx=0).stop 2024-03-23T23:09:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.806+0000 7faa8ce00640 1 -- 172.21.15.12:0/581009664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa88102140 msgr2=0x7faa88197960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.806+0000 7faa8ce00640 1 --2- 172.21.15.12:0/581009664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa88102140 0x7faa88197960 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7faa7002c8f0 tx=0x7faa7002c5e0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.806+0000 7faa8ce00640 1 -- 172.21.15.12:0/581009664 shutdown_connections 2024-03-23T23:09:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.806+0000 7faa8ce00640 1 --2- 172.21.15.12:0/581009664 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7faa6c0a28a0 0x7faa6c0a4d60 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.806+0000 7faa8ce00640 1 --2- 172.21.15.12:0/581009664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa881073a0 0x7faa8819ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.806+0000 7faa8ce00640 1 --2- 172.21.15.12:0/581009664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa88104a70 0x7faa88197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.806+0000 7faa8ce00640 1 --2- 172.21.15.12:0/581009664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa88102140 0x7faa88197960 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.806+0000 7faa8ce00640 1 -- 172.21.15.12:0/581009664 >> 172.21.15.12:0/581009664 conn(0x7faa880fbb30 msgr2=0x7faa88102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.806+0000 7faa8ce00640 1 -- 172.21.15.12:0/581009664 shutdown_connections 2024-03-23T23:09:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:00.807+0000 7faa8ce00640 1 -- 172.21.15.12:0/581009664 wait complete. 2024-03-23T23:09:00.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:00 smithi173 ceph-mon[36116]: pgmap v606: 97 pgs: 97 active+clean; 4.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 14 op/s 2024-03-23T23:09:00.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:00 smithi203 ceph-mon[36693]: pgmap v606: 97 pgs: 97 active+clean; 4.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 14 op/s 2024-03-23T23:09:01.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:00 smithi012 ceph-mon[31695]: pgmap v606: 97 pgs: 97 active+clean; 4.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 14 op/s 2024-03-23T23:09:02.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:02 smithi173 ceph-mon[36116]: pgmap v607: 97 pgs: 97 active+clean; 4.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 15 op/s 2024-03-23T23:09:02.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:02 smithi203 ceph-mon[36693]: pgmap v607: 97 pgs: 97 active+clean; 4.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 15 op/s 2024-03-23T23:09:03.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:02 smithi012 ceph-mon[31695]: pgmap v607: 97 pgs: 97 active+clean; 4.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 15 op/s 2024-03-23T23:09:04.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:04 smithi203 ceph-mon[36693]: pgmap v608: 97 pgs: 97 active+clean; 691 KiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 15 op/s 2024-03-23T23:09:05.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:04 smithi012 ceph-mon[31695]: pgmap v608: 97 pgs: 97 active+clean; 691 KiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 15 op/s 2024-03-23T23:09:05.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:04 smithi173 ceph-mon[36116]: pgmap v608: 97 pgs: 97 active+clean; 691 KiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 15 op/s 2024-03-23T23:09:06.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:06 smithi203 ceph-mon[36693]: pgmap v609: 97 pgs: 97 active+clean; 761 KiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-03-23T23:09:07.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:06 smithi012 ceph-mon[31695]: pgmap v609: 97 pgs: 97 active+clean; 761 KiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-03-23T23:09:07.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:06 smithi173 ceph-mon[36116]: pgmap v609: 97 pgs: 97 active+clean; 761 KiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-03-23T23:09:08.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:08 smithi203 ceph-mon[36693]: pgmap v610: 97 pgs: 97 active+clean; 761 KiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:09:08.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:08 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:09:09.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:08 smithi012 ceph-mon[31695]: pgmap v610: 97 pgs: 97 active+clean; 761 KiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:09:09.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:08 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:09:09.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:08 smithi173 ceph-mon[36116]: pgmap v610: 97 pgs: 97 active+clean; 761 KiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:09:09.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:08 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:09:09.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:09 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:09:09.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:09 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:09:09.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:09 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:09:09.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:09 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:09:09.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:09 smithi203 ceph-mon[36693]: Adjusting osd_memory_target on smithi203 to 174.1M 2024-03-23T23:09:09.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:09 smithi203 ceph-mon[36693]: Unable to set osd_memory_target on smithi203 to 182606233: error parsing value: Value '182606233' is below minimum 939524096 2024-03-23T23:09:09.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:09 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:09:09.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:09 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:09:09.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:09 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:09:10.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:09 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:09:10.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:09 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:09:10.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:09 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:09:10.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:09 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:09:10.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:09 smithi012 ceph-mon[31695]: Adjusting osd_memory_target on smithi203 to 174.1M 2024-03-23T23:09:10.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:09 smithi012 ceph-mon[31695]: Unable to set osd_memory_target on smithi203 to 182606233: error parsing value: Value '182606233' is below minimum 939524096 2024-03-23T23:09:10.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:09 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:09:10.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:09 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:09:10.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:09 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:09:10.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:09 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:09:10.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:09 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:09:10.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:09 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:09:10.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:09 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-23T23:09:10.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:09 smithi173 ceph-mon[36116]: Adjusting osd_memory_target on smithi203 to 174.1M 2024-03-23T23:09:10.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:09 smithi173 ceph-mon[36116]: Unable to set osd_memory_target on smithi203 to 182606233: error parsing value: Value '182606233' is below minimum 939524096 2024-03-23T23:09:10.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:09 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:09:10.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:09 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:09:10.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:09 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:09:10.378 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:09:10.378 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:09:10.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.618+0000 7f31d7200640 1 -- 172.21.15.12:0/382914819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31d0104ad0 msgr2=0x7f31d0104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:10.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.618+0000 7f31d7200640 1 --2- 172.21.15.12:0/382914819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31d0104ad0 0x7f31d0104f50 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f31b8009900 tx=0x7f31b802bc30 comp rx=0 tx=0).stop 2024-03-23T23:09:10.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.618+0000 7f31d7200640 1 -- 172.21.15.12:0/382914819 shutdown_connections 2024-03-23T23:09:10.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.619+0000 7f31d7200640 1 --2- 172.21.15.12:0/382914819 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31d0105490 0x7f31d010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.619+0000 7f31d7200640 1 --2- 172.21.15.12:0/382914819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31d0104ad0 0x7f31d0104f50 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.619+0000 7f31d7200640 1 --2- 172.21.15.12:0/382914819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31d01038e0 0x7f31d0103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.619+0000 7f31d7200640 1 -- 172.21.15.12:0/382914819 >> 172.21.15.12:0/382914819 conn(0x7f31d00fee50 msgr2=0x7f31d0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:10.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.619+0000 7f31d7200640 1 -- 172.21.15.12:0/382914819 shutdown_connections 2024-03-23T23:09:10.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.619+0000 7f31d7200640 1 -- 172.21.15.12:0/382914819 wait complete. 2024-03-23T23:09:10.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.620+0000 7f31d7200640 1 Processor -- start 2024-03-23T23:09:10.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.620+0000 7f31d7200640 1 -- start start 2024-03-23T23:09:10.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.620+0000 7f31d7200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31d01038e0 0x7f31d0199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:10.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.620+0000 7f31d7200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31d0104ad0 0x7f31d019a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:10.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.620+0000 7f31d7200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31d0105490 0x7f31d01a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:10.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.621+0000 7f31d7200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d010f590 con 0x7f31d01038e0 2024-03-23T23:09:10.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.621+0000 7f31d7200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d010f410 con 0x7f31d0104ad0 2024-03-23T23:09:10.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.621+0000 7f31d7200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d010f710 con 0x7f31d0105490 2024-03-23T23:09:10.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.621+0000 7f31c7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31d0104ad0 0x7f31d019a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:10.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.621+0000 7f31cfe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31d01038e0 0x7f31d0199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:10.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.621+0000 7f31c7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31d0104ad0 0x7f31d019a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:33998/0 (socket says 172.21.15.12:33998) 2024-03-23T23:09:10.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.621+0000 7f31c7e00640 1 -- 172.21.15.12:0/3018763338 learned_addr learned my addr 172.21.15.12:0/3018763338 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:10.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.621+0000 7f31d5200640 1 --2- 172.21.15.12:0/3018763338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31d0105490 0x7f31d01a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:10.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.621+0000 7f31c7e00640 1 -- 172.21.15.12:0/3018763338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31d0105490 msgr2=0x7f31d01a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:10.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.621+0000 7f31c7e00640 1 --2- 172.21.15.12:0/3018763338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31d0105490 0x7f31d01a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.621+0000 7f31c7e00640 1 -- 172.21.15.12:0/3018763338 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31d01038e0 msgr2=0x7f31d0199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:10.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.621+0000 7f31c7e00640 1 --2- 172.21.15.12:0/3018763338 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31d01038e0 0x7f31d0199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.621+0000 7f31c7e00640 1 -- 172.21.15.12:0/3018763338 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d01a1830 con 0x7f31d0104ad0 2024-03-23T23:09:10.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.622+0000 7f31c7e00640 1 --2- 172.21.15.12:0/3018763338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31d0104ad0 0x7f31d019a160 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f31b8000c00 tx=0x7f31b802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:10.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.622+0000 7f31cd600640 1 -- 172.21.15.12:0/3018763338 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b8043020 con 0x7f31d0104ad0 2024-03-23T23:09:10.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.622+0000 7f31cd600640 1 -- 172.21.15.12:0/3018763338 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f31b80054e0 con 0x7f31d0104ad0 2024-03-23T23:09:10.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.622+0000 7f31d7200640 1 -- 172.21.15.12:0/3018763338 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d01a1ac0 con 0x7f31d0104ad0 2024-03-23T23:09:10.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.622+0000 7f31cd600640 1 -- 172.21.15.12:0/3018763338 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b8003c40 con 0x7f31d0104ad0 2024-03-23T23:09:10.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.622+0000 7f31d7200640 1 -- 172.21.15.12:0/3018763338 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d01a2060 con 0x7f31d0104ad0 2024-03-23T23:09:10.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.625+0000 7f31cd600640 1 -- 172.21.15.12:0/3018763338 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f31b8046020 con 0x7f31d0104ad0 2024-03-23T23:09:10.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.626+0000 7f31d9e6d740 1 Processor -- start 2024-03-23T23:09:10.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.626+0000 7f31d9e6d740 1 -- start start 2024-03-23T23:09:10.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.626+0000 7f31d9e6d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5573bfcd2e80 0x5573bfcd32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:10.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.626+0000 7f31d9e6d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5573bfcd3870 0x5573bfcd8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:10.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.626+0000 7f31d9e6d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573bfcd89e0 0x5573bfcdae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:10.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.626+0000 7f31d9e6d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573bfc38250 con 0x5573bfcd89e0 2024-03-23T23:09:10.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.626+0000 7f31d9e6d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573bfbe0a30 con 0x5573bfcd3870 2024-03-23T23:09:10.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.626+0000 7f31d9e6d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573bfc342f0 con 0x5573bfcd2e80 2024-03-23T23:09:10.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.626+0000 7f31d5200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5573bfcd2e80 0x5573bfcd32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:10.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.626+0000 7f31cfe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5573bfcd3870 0x5573bfcd8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:10.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.627+0000 7f31d5200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5573bfcd2e80 0x5573bfcd32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54846/0 (socket says 172.21.15.12:54846) 2024-03-23T23:09:10.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.627+0000 7f31d5200640 1 -- 172.21.15.12:0/308686619 learned_addr learned my addr 172.21.15.12:0/308686619 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:10.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.627+0000 7f31c7e00640 1 --2- 172.21.15.12:0/308686619 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573bfcd89e0 0x5573bfcdae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:10.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.627+0000 7f31cd600640 1 --2- 172.21.15.12:0/3018763338 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f31b40a2880 0x7f31b40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:10.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.627+0000 7f31d5200640 1 -- 172.21.15.12:0/308686619 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5573bfcd3870 msgr2=0x5573bfcd8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:10.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.627+0000 7f31d5200640 1 --2- 172.21.15.12:0/308686619 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5573bfcd3870 0x5573bfcd8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.627+0000 7f31d5200640 1 -- 172.21.15.12:0/308686619 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573bfcd89e0 msgr2=0x5573bfcdae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:10.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.627+0000 7f31d5200640 1 --2- 172.21.15.12:0/308686619 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573bfcd89e0 0x5573bfcdae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.627+0000 7f31d5200640 1 -- 172.21.15.12:0/308686619 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573bfbcb690 con 0x5573bfcd2e80 2024-03-23T23:09:10.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.627+0000 7f31cfe00640 1 --2- 172.21.15.12:0/308686619 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5573bfcd3870 0x5573bfcd8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:10.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.627+0000 7f31c7e00640 1 --2- 172.21.15.12:0/308686619 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573bfcd89e0 0x5573bfcdae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:09:10.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.627+0000 7f31cd600640 1 -- 172.21.15.12:0/3018763338 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f31b80f0b00 con 0x7f31d0104ad0 2024-03-23T23:09:10.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.627+0000 7f31d5200640 1 --2- 172.21.15.12:0/3018763338 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f31b40a2880 0x7f31b40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:10.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.628+0000 7f31d5200640 1 --2- 172.21.15.12:0/308686619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5573bfcd2e80 0x5573bfcd32a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f31c8009fd0 tx=0x7f31c80118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e71036bd96b68a6d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:10.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.628+0000 7f31c5600640 1 -- 172.21.15.12:0/308686619 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c8004250 con 0x5573bfcd2e80 2024-03-23T23:09:10.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.628+0000 7f31c5600640 1 -- 172.21.15.12:0/308686619 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f31c8003db0 con 0x5573bfcd2e80 2024-03-23T23:09:10.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.628+0000 7f31c5600640 1 -- 172.21.15.12:0/308686619 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c801bcc0 con 0x5573bfcd2e80 2024-03-23T23:09:10.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.629+0000 7f31d9e6d740 1 -- 172.21.15.12:0/308686619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5573bfcd2e80 msgr2=0x5573bfcd32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:10.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.629+0000 7f31d9e6d740 1 --2- 172.21.15.12:0/308686619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5573bfcd2e80 0x5573bfcd32a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f31c8009fd0 tx=0x7f31c80118d0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.629+0000 7f31d5200640 1 --2- 172.21.15.12:0/3018763338 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f31b40a2880 0x7f31b40a4d40 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f31c8020000 tx=0x7f31c8021000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:10.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.629+0000 7f31d9e6d740 1 -- 172.21.15.12:0/308686619 shutdown_connections 2024-03-23T23:09:10.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.629+0000 7f31d9e6d740 1 --2- 172.21.15.12:0/308686619 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573bfcd89e0 0x5573bfcdae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.629+0000 7f31d9e6d740 1 --2- 172.21.15.12:0/308686619 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5573bfcd3870 0x5573bfcd8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.629+0000 7f31d9e6d740 1 --2- 172.21.15.12:0/308686619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5573bfcd2e80 0x5573bfcd32a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.629+0000 7f31d9e6d740 1 -- 172.21.15.12:0/308686619 >> 172.21.15.12:0/308686619 conn(0x5573bfc9b200 msgr2=0x5573bfcce7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:10.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.629+0000 7f31d9e6d740 1 -- 172.21.15.12:0/308686619 shutdown_connections 2024-03-23T23:09:10.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.629+0000 7f31d9e6d740 1 -- 172.21.15.12:0/308686619 wait complete. 2024-03-23T23:09:10.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.629+0000 7f31d9e6d740 1 -- start start 2024-03-23T23:09:10.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.630+0000 7f31d9e6d740 1 Processor -- start 2024-03-23T23:09:10.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.630+0000 7f31d9e6d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5573bfcd2e80 0x5573bfcdce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:10.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.630+0000 7f31d9e6d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573bfcd3870 0x5573bfcdd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:10.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.630+0000 7f31d9e6d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5573bfcd89e0 0x5573bfce43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:10.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.630+0000 7f31d9e6d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573bfcdb990 con 0x5573bfcd3870 2024-03-23T23:09:10.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.630+0000 7f31d9e6d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573bfbf3e80 con 0x5573bfcd89e0 2024-03-23T23:09:10.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.630+0000 7f31d9e6d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573bfcdbb10 con 0x5573bfcd2e80 2024-03-23T23:09:10.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.630+0000 7f31d9e6d740 10 client.0 _ll_register_callbacks cb 0x5573bfc06130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:10.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.630+0000 7f31cfe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5573bfcd2e80 0x5573bfcdce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:10.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.630+0000 7f31cfe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5573bfcd2e80 0x5573bfcdce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54850/0 (socket says 172.21.15.12:54850) 2024-03-23T23:09:10.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.630+0000 7f31c7e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573bfcd3870 0x5573bfcdd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:10.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.630+0000 7f31cfe00640 1 -- 172.21.15.12:0/3593080597 learned_addr learned my addr 172.21.15.12:0/3593080597 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:10.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.630+0000 7f31d5200640 1 --2- 172.21.15.12:0/3593080597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5573bfcd89e0 0x5573bfce43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:10.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.631+0000 7f31cfe00640 1 -- 172.21.15.12:0/3593080597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5573bfcd89e0 msgr2=0x5573bfce43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:10.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.631+0000 7f31cfe00640 1 --2- 172.21.15.12:0/3593080597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5573bfcd89e0 0x5573bfce43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.631+0000 7f31cfe00640 1 -- 172.21.15.12:0/3593080597 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573bfcd3870 msgr2=0x5573bfcdd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:10.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.631+0000 7f31cfe00640 1 --2- 172.21.15.12:0/3593080597 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573bfcd3870 0x5573bfcdd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.631+0000 7f31cfe00640 1 -- 172.21.15.12:0/3593080597 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5573bfa167d0 con 0x5573bfcd2e80 2024-03-23T23:09:10.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.631+0000 7f31cfe00640 1 --2- 172.21.15.12:0/3593080597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5573bfcd2e80 0x5573bfcdce30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f31c00126d0 tx=0x7f31c0012ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5656adc6cffc6faf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:10.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.631+0000 7f31be000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:09:10.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.631+0000 7f31be000640 1 -- 172.21.15.12:0/3593080597 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c0004030 con 0x5573bfcd2e80 2024-03-23T23:09:10.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.631+0000 7f31be000640 1 -- 172.21.15.12:0/3593080597 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f31c00041d0 con 0x5573bfcd2e80 2024-03-23T23:09:10.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.631+0000 7f31d9e6d740 1 -- 172.21.15.12:0/3593080597 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f31c8003db0 con 0x5573bfcd2e80 2024-03-23T23:09:10.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.632+0000 7f31be000640 1 -- 172.21.15.12:0/3593080597 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f31c00043d0 con 0x5573bfcd2e80 2024-03-23T23:09:10.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.632+0000 7f31d9e6d740 10 client.27304 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:10.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.632+0000 7f31d9e6d740 1 -- 172.21.15.12:0/3593080597 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5573bf94c3f0 con 0x5573bfcd2e80 2024-03-23T23:09:10.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.632+0000 7f31be000640 1 -- 172.21.15.12:0/3593080597 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f31c8003db0 con 0x5573bfcd2e80 2024-03-23T23:09:10.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.632+0000 7f31be000640 1 -- 172.21.15.12:0/3593080597 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f31c000f870 con 0x5573bfcd2e80 2024-03-23T23:09:10.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.632+0000 7f31d9e6d740 10 client.27304 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:10.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.632+0000 7f31d9e6d740 10 client.27304 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:10.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.632+0000 7f31d9e6d740 20 client.27304 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:10.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.632+0000 7f31d9e6d740 1 --2- 172.21.15.12:0/3593080597 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5573bfce8c90 0x5573bfceb110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:10.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.632+0000 7f31d9e6d740 4 client.27304 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:09:10.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.632+0000 7f31d9e6d740 1 -- 172.21.15.12:0/3593080597 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5573bfc45790 con 0x5573bfce8c90 2024-03-23T23:09:10.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.633+0000 7f31c7e00640 1 --2- 172.21.15.12:0/3593080597 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5573bfce8c90 0x5573bfceb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:10.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.633+0000 7f31c7e00640 1 --2- 172.21.15.12:0/3593080597 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5573bfce8c90 0x5573bfceb110 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f072daac471d4424 server_cookie=ff1f23622e1f79cd in_seq=0 out_seq=0 2024-03-23T23:09:10.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.633+0000 7f31be000640 10 client.27304 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:10.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.634+0000 7f31be000640 1 -- 172.21.15.12:0/3593080597 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5573bfc45790 con 0x5573bfce8c90 2024-03-23T23:09:10.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.634+0000 7f31be000640 10 client.27304 handle_command_reply: tid=0 2024-03-23T23:09:10.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.634+0000 7f31d9e6d740 1 client.27304 shutdown 2024-03-23T23:09:10.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.634+0000 7f31d9e6d740 1 -- 172.21.15.12:0/3593080597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5573bfcd2e80 msgr2=0x5573bfcdce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:10.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.634+0000 7f31d9e6d740 1 --2- 172.21.15.12:0/3593080597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5573bfcd2e80 0x5573bfcdce30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f31c00126d0 tx=0x7f31c0012ad0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.634+0000 7f31d9e6d740 1 -- 172.21.15.12:0/3593080597 shutdown_connections 2024-03-23T23:09:10.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.634+0000 7f31d9e6d740 1 --2- 172.21.15.12:0/3593080597 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5573bfce8c90 0x5573bfceb110 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.634+0000 7f31d9e6d740 1 --2- 172.21.15.12:0/3593080597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5573bfcd89e0 0x5573bfce43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.634+0000 7f31be000640 0 client.27304 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:10.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.634+0000 7f31d9e6d740 1 --2- 172.21.15.12:0/3593080597 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573bfcd3870 0x5573bfcdd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.634+0000 7f31d9e6d740 1 --2- 172.21.15.12:0/3593080597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5573bfcd2e80 0x5573bfcdce30 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.634+0000 7f31d9e6d740 1 -- 172.21.15.12:0/3593080597 >> 172.21.15.12:0/3593080597 conn(0x5573bfc9b200 msgr2=0x5573bfccda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:10.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.634+0000 7f31d9e6d740 1 -- 172.21.15.12:0/3593080597 shutdown_connections 2024-03-23T23:09:10.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.634+0000 7f31d9e6d740 1 -- 172.21.15.12:0/3593080597 wait complete. 2024-03-23T23:09:10.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.634+0000 7f31d9e6d740 20 client.27304 trim_cache size 0 max 16384 2024-03-23T23:09:10.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.653+0000 7f31d9e6d740 1 Processor -- start 2024-03-23T23:09:10.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.653+0000 7f31d9e6d740 1 -- start start 2024-03-23T23:09:10.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.653+0000 7f31d9e6d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5573bfcd2e80 0x5573bfccd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:10.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.653+0000 7f31d9e6d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5573bfcd3870 0x5573bfce4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:10.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.653+0000 7f31d9e6d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573bfcd89e0 0x5573bfcef9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:10.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.653+0000 7f31d9e6d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d01ad130 con 0x5573bfcd89e0 2024-03-23T23:09:10.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.653+0000 7f31d9e6d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d01aac30 con 0x5573bfcd3870 2024-03-23T23:09:10.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.653+0000 7f31d9e6d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d01a8cd0 con 0x5573bfcd2e80 2024-03-23T23:09:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.654+0000 7f31c7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5573bfcd3870 0x5573bfce4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.654+0000 7f31c7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5573bfcd3870 0x5573bfce4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:34028/0 (socket says 172.21.15.12:34028) 2024-03-23T23:09:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.654+0000 7f31c7e00640 1 -- 172.21.15.12:0/3463577855 learned_addr learned my addr 172.21.15.12:0/3463577855 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.654+0000 7f31cfe00640 1 --2- 172.21.15.12:0/3463577855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5573bfcd2e80 0x5573bfccd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.654+0000 7f31d5200640 1 --2- 172.21.15.12:0/3463577855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573bfcd89e0 0x5573bfcef9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.654+0000 7f31c7e00640 1 -- 172.21.15.12:0/3463577855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5573bfcd2e80 msgr2=0x5573bfccd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.654+0000 7f31c7e00640 1 --2- 172.21.15.12:0/3463577855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5573bfcd2e80 0x5573bfccd200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.654+0000 7f31c7e00640 1 -- 172.21.15.12:0/3463577855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573bfcd89e0 msgr2=0x5573bfcef9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.654+0000 7f31c7e00640 1 --2- 172.21.15.12:0/3463577855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573bfcd89e0 0x5573bfcef9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.654+0000 7f31c7e00640 1 -- 172.21.15.12:0/3463577855 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573bf90c0f0 con 0x5573bfcd3870 2024-03-23T23:09:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.654+0000 7f31c7e00640 1 --2- 172.21.15.12:0/3463577855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5573bfcd3870 0x5573bfce4370 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f31b8005e90 tx=0x7f31b80dd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74c86f4152e06d78 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.654+0000 7f31c4c00640 1 -- 172.21.15.12:0/3463577855 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b80035f0 con 0x5573bfcd3870 2024-03-23T23:09:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.654+0000 7f31c4c00640 1 -- 172.21.15.12:0/3463577855 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f31b8005690 con 0x5573bfcd3870 2024-03-23T23:09:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.655+0000 7f31c4c00640 1 -- 172.21.15.12:0/3463577855 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b80c5bf0 con 0x5573bfcd3870 2024-03-23T23:09:10.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.655+0000 7f31d9e6d740 1 -- 172.21.15.12:0/3463577855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5573bfcd3870 msgr2=0x5573bfce4370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:10.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.655+0000 7f31d9e6d740 1 --2- 172.21.15.12:0/3463577855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5573bfcd3870 0x5573bfce4370 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f31b8005e90 tx=0x7f31b80dd040 comp rx=0 tx=0).stop 2024-03-23T23:09:10.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.655+0000 7f31d9e6d740 1 -- 172.21.15.12:0/3463577855 shutdown_connections 2024-03-23T23:09:10.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.655+0000 7f31d9e6d740 1 --2- 172.21.15.12:0/3463577855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573bfcd89e0 0x5573bfcef9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.655+0000 7f31d9e6d740 1 --2- 172.21.15.12:0/3463577855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5573bfcd3870 0x5573bfce4370 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.655+0000 7f31d9e6d740 1 --2- 172.21.15.12:0/3463577855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5573bfcd2e80 0x5573bfccd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.655+0000 7f31d9e6d740 1 -- 172.21.15.12:0/3463577855 >> 172.21.15.12:0/3463577855 conn(0x5573bfc9b200 msgr2=0x5573bfcd22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:10.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.656+0000 7f31d9e6d740 1 -- 172.21.15.12:0/3463577855 shutdown_connections 2024-03-23T23:09:10.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.656+0000 7f31d9e6d740 1 -- 172.21.15.12:0/3463577855 wait complete. 2024-03-23T23:09:10.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.656+0000 7f31d9e6d740 1 -- start start 2024-03-23T23:09:10.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.656+0000 7f31d9e6d740 1 Processor -- start 2024-03-23T23:09:10.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.657+0000 7f31d9e6d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573bfcd2e80 0x5573bfce21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:10.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.657+0000 7f31d9e6d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5573bfcd3870 0x5573bfce26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:10.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.657+0000 7f31d9e6d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5573bfcd89e0 0x5573bfce3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:10.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.657+0000 7f31d9e6d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d01a92d0 con 0x5573bfcd2e80 2024-03-23T23:09:10.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.657+0000 7f31d9e6d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d01aa5d0 con 0x5573bfcd89e0 2024-03-23T23:09:10.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.657+0000 7f31d9e6d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d01a9910 con 0x5573bfcd3870 2024-03-23T23:09:10.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.657+0000 7f31d9e6d740 10 client.0 _ll_register_callbacks cb 0x7f31c0009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:10.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.657+0000 7f31cfe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573bfcd2e80 0x5573bfce21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:10.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.657+0000 7f31cfe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573bfcd2e80 0x5573bfce21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39332/0 (socket says 172.21.15.12:39332) 2024-03-23T23:09:10.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.657+0000 7f31cfe00640 1 -- 172.21.15.12:0/3738290679 learned_addr learned my addr 172.21.15.12:0/3738290679 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:10.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.657+0000 7f31c7e00640 1 --2- 172.21.15.12:0/3738290679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5573bfcd3870 0x5573bfce26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:10.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.657+0000 7f31d5200640 1 --2- 172.21.15.12:0/3738290679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5573bfcd89e0 0x5573bfce3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:10.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.657+0000 7f31c7e00640 1 -- 172.21.15.12:0/3738290679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5573bfcd89e0 msgr2=0x5573bfce3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:10.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.657+0000 7f31c7e00640 1 --2- 172.21.15.12:0/3738290679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5573bfcd89e0 0x5573bfce3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.657+0000 7f31c7e00640 1 -- 172.21.15.12:0/3738290679 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573bfcd2e80 msgr2=0x5573bfce21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:10.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.657+0000 7f31c7e00640 1 --2- 172.21.15.12:0/3738290679 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573bfcd2e80 0x5573bfce21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.657+0000 7f31c7e00640 1 -- 172.21.15.12:0/3738290679 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5573bfce3cf0 con 0x5573bfcd3870 2024-03-23T23:09:10.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.657+0000 7f31d5200640 1 --2- 172.21.15.12:0/3738290679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5573bfcd89e0 0x5573bfce3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:10.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.657+0000 7f31cfe00640 1 --2- 172.21.15.12:0/3738290679 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573bfcd2e80 0x5573bfce21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:10.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.658+0000 7f31c7e00640 1 --2- 172.21.15.12:0/3738290679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5573bfcd3870 0x5573bfce26f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f31b8005e90 tx=0x7f31b802bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d080ee713f4c67ac server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:10.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.658+0000 7f31be000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:09:10.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.658+0000 7f31be000640 1 -- 172.21.15.12:0/3738290679 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b80035f0 con 0x5573bfcd3870 2024-03-23T23:09:10.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.658+0000 7f31be000640 1 -- 172.21.15.12:0/3738290679 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f31b80ee6c0 con 0x5573bfcd3870 2024-03-23T23:09:10.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.658+0000 7f31d9e6d740 1 -- 172.21.15.12:0/3738290679 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5573bfc3fae0 con 0x5573bfcd3870 2024-03-23T23:09:10.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.658+0000 7f31be000640 1 -- 172.21.15.12:0/3738290679 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f31b80bdb50 con 0x5573bfcd3870 2024-03-23T23:09:10.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.658+0000 7f31d9e6d740 10 client.27310 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:10.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.658+0000 7f31d9e6d740 1 -- 172.21.15.12:0/3738290679 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5573bfd44c50 con 0x5573bfcd3870 2024-03-23T23:09:10.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.658+0000 7f31be000640 1 -- 172.21.15.12:0/3738290679 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5573bfc3fae0 con 0x5573bfcd3870 2024-03-23T23:09:10.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.658+0000 7f31be000640 1 -- 172.21.15.12:0/3738290679 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f31b80c5760 con 0x5573bfcd3870 2024-03-23T23:09:10.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.659+0000 7f31d9e6d740 10 client.27310 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:10.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.659+0000 7f31d9e6d740 10 client.27310 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:10.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.659+0000 7f31d9e6d740 20 client.27310 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:10.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.659+0000 7f31d9e6d740 1 --2- 172.21.15.12:0/3738290679 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5573bfd45570 0x5573bfd47990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:10.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.659+0000 7f31d9e6d740 4 client.27310 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:09:10.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.659+0000 7f31d9e6d740 1 -- 172.21.15.12:0/3738290679 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5573bfbbd3a0 con 0x5573bfd45570 2024-03-23T23:09:10.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.659+0000 7f31cfe00640 1 --2- 172.21.15.12:0/3738290679 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5573bfd45570 0x5573bfd47990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:10.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.659+0000 7f31cfe00640 1 --2- 172.21.15.12:0/3738290679 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5573bfd45570 0x5573bfd47990 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a55284a983bfcc44 server_cookie=bef2f54785c18fdf in_seq=0 out_seq=0 2024-03-23T23:09:10.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.659+0000 7f31be000640 10 client.27310 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:10.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.660+0000 7f31be000640 1 -- 172.21.15.12:0/3738290679 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5573bfbbd3a0 con 0x5573bfd45570 2024-03-23T23:09:10.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.660+0000 7f31be000640 10 client.27310 handle_command_reply: tid=0 2024-03-23T23:09:10.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.660+0000 7f31d9e6d740 1 client.27310 shutdown 2024-03-23T23:09:10.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.660+0000 7f31d9e6d740 1 -- 172.21.15.12:0/3738290679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5573bfcd3870 msgr2=0x5573bfce26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:10.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.660+0000 7f31d9e6d740 1 --2- 172.21.15.12:0/3738290679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5573bfcd3870 0x5573bfce26f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f31b8005e90 tx=0x7f31b802bc30 comp rx=0 tx=0).stop 2024-03-23T23:09:10.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.660+0000 7f31d9e6d740 1 -- 172.21.15.12:0/3738290679 shutdown_connections 2024-03-23T23:09:10.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.660+0000 7f31d9e6d740 1 --2- 172.21.15.12:0/3738290679 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5573bfd45570 0x5573bfd47990 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.660+0000 7f31d9e6d740 1 --2- 172.21.15.12:0/3738290679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5573bfcd89e0 0x5573bfce3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.660+0000 7f31d9e6d740 1 --2- 172.21.15.12:0/3738290679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5573bfcd3870 0x5573bfce26f0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.660+0000 7f31d9e6d740 1 --2- 172.21.15.12:0/3738290679 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573bfcd2e80 0x5573bfce21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.660+0000 7f31d9e6d740 1 -- 172.21.15.12:0/3738290679 >> 172.21.15.12:0/3738290679 conn(0x5573bfc9b200 msgr2=0x5573bfcd4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:10.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.660+0000 7f31be000640 0 client.27310 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:10.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.660+0000 7f31d9e6d740 1 -- 172.21.15.12:0/3738290679 shutdown_connections 2024-03-23T23:09:10.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.660+0000 7f31d9e6d740 1 -- 172.21.15.12:0/3738290679 wait complete. 2024-03-23T23:09:10.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.660+0000 7f31d9e6d740 20 client.27310 trim_cache size 0 max 16384 2024-03-23T23:09:10.664 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:09:10.664 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-03-23T23:09:10.664 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-03-23T23:09:10.664 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:09:10.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.661+0000 7f31d7200640 1 -- 172.21.15.12:0/3018763338 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f31b40a2880 msgr2=0x7f31b40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:10.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.661+0000 7f31d7200640 1 --2- 172.21.15.12:0/3018763338 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f31b40a2880 0x7f31b40a4d40 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f31c8020000 tx=0x7f31c8021000 comp rx=0 tx=0).stop 2024-03-23T23:09:10.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.661+0000 7f31d7200640 1 -- 172.21.15.12:0/3018763338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31d0104ad0 msgr2=0x7f31d019a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:10.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.661+0000 7f31d7200640 1 --2- 172.21.15.12:0/3018763338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31d0104ad0 0x7f31d019a160 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f31b8000c00 tx=0x7f31b802c650 comp rx=0 tx=0).stop 2024-03-23T23:09:10.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.661+0000 7f31d7200640 1 -- 172.21.15.12:0/3018763338 shutdown_connections 2024-03-23T23:09:10.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.661+0000 7f31d7200640 1 --2- 172.21.15.12:0/3018763338 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f31b40a2880 0x7f31b40a4d40 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.661+0000 7f31d7200640 1 --2- 172.21.15.12:0/3018763338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31d0105490 0x7f31d01a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.661+0000 7f31d7200640 1 --2- 172.21.15.12:0/3018763338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31d0104ad0 0x7f31d019a160 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.661+0000 7f31d7200640 1 --2- 172.21.15.12:0/3018763338 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31d01038e0 0x7f31d0199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.661+0000 7f31d7200640 1 -- 172.21.15.12:0/3018763338 >> 172.21.15.12:0/3018763338 conn(0x7f31d00fee50 msgr2=0x7f31d0107e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:10.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.661+0000 7f31d7200640 1 -- 172.21.15.12:0/3018763338 shutdown_connections 2024-03-23T23:09:10.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.661+0000 7f31d7200640 1 -- 172.21.15.12:0/3018763338 wait complete. 2024-03-23T23:09:10.679 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T23:09:10.679 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:09:10.756 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:10 smithi012 ceph-mon[31695]: pgmap v611: 97 pgs: 97 active+clean; 761 KiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:09:10.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.898+0000 7fa180a00640 1 -- 172.21.15.12:0/4788004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa17c108cf0 msgr2=0x7fa17c109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:10.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.898+0000 7fa180a00640 1 --2- 172.21.15.12:0/4788004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa17c108cf0 0x7fa17c109170 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fa168009900 tx=0x7fa16802bc40 comp rx=0 tx=0).stop 2024-03-23T23:09:10.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.898+0000 7fa180a00640 1 -- 172.21.15.12:0/4788004 shutdown_connections 2024-03-23T23:09:10.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.898+0000 7fa180a00640 1 --2- 172.21.15.12:0/4788004 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa17c1096b0 0x7fa17c10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.898+0000 7fa180a00640 1 --2- 172.21.15.12:0/4788004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa17c108cf0 0x7fa17c109170 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.898+0000 7fa180a00640 1 --2- 172.21.15.12:0/4788004 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa17c107b00 0x7fa17c107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.898+0000 7fa180a00640 1 -- 172.21.15.12:0/4788004 >> 172.21.15.12:0/4788004 conn(0x7fa17c076ab0 msgr2=0x7fa17c078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:10.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.898+0000 7fa180a00640 1 -- 172.21.15.12:0/4788004 shutdown_connections 2024-03-23T23:09:10.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.898+0000 7fa180a00640 1 -- 172.21.15.12:0/4788004 wait complete. 2024-03-23T23:09:10.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.899+0000 7fa180a00640 1 Processor -- start 2024-03-23T23:09:10.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.899+0000 7fa180a00640 1 -- start start 2024-03-23T23:09:10.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.900+0000 7fa180a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa17c107b00 0x7fa17c19dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:10.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.900+0000 7fa180a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa17c108cf0 0x7fa17c19e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:10.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.900+0000 7fa180a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa17c1096b0 0x7fa17c1a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:10.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.900+0000 7fa180a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17c1137d0 con 0x7fa17c1096b0 2024-03-23T23:09:10.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.900+0000 7fa180a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17c113650 con 0x7fa17c107b00 2024-03-23T23:09:10.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.900+0000 7fa180a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17c113950 con 0x7fa17c108cf0 2024-03-23T23:09:10.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.900+0000 7fa179e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa17c108cf0 0x7fa17c19e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:10.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.900+0000 7fa17b200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa17c1096b0 0x7fa17c1a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:10.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.900+0000 7fa17a800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa17c107b00 0x7fa17c19dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:10.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.900+0000 7fa179e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa17c108cf0 0x7fa17c19e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54880/0 (socket says 172.21.15.12:54880) 2024-03-23T23:09:10.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.900+0000 7fa179e00640 1 -- 172.21.15.12:0/3279346050 learned_addr learned my addr 172.21.15.12:0/3279346050 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:10.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.900+0000 7fa179e00640 1 -- 172.21.15.12:0/3279346050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa17c107b00 msgr2=0x7fa17c19dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:10.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.900+0000 7fa179e00640 1 --2- 172.21.15.12:0/3279346050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa17c107b00 0x7fa17c19dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.900+0000 7fa179e00640 1 -- 172.21.15.12:0/3279346050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa17c1096b0 msgr2=0x7fa17c1a5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:10.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.901+0000 7fa179e00640 1 --2- 172.21.15.12:0/3279346050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa17c1096b0 0x7fa17c1a5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:10.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.901+0000 7fa179e00640 1 -- 172.21.15.12:0/3279346050 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17c1a5a00 con 0x7fa17c108cf0 2024-03-23T23:09:10.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.901+0000 7fa179e00640 1 --2- 172.21.15.12:0/3279346050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa17c108cf0 0x7fa17c19e320 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fa16802c8f0 tx=0x7fa16802c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:10.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.901+0000 7fa173400640 1 -- 172.21.15.12:0/3279346050 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa168041070 con 0x7fa17c108cf0 2024-03-23T23:09:10.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.901+0000 7fa173400640 1 -- 172.21.15.12:0/3279346050 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa168004450 con 0x7fa17c108cf0 2024-03-23T23:09:10.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.901+0000 7fa173400640 1 -- 172.21.15.12:0/3279346050 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa168034440 con 0x7fa17c108cf0 2024-03-23T23:09:10.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.901+0000 7fa180a00640 1 -- 172.21.15.12:0/3279346050 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa17c1a5c90 con 0x7fa17c108cf0 2024-03-23T23:09:10.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.901+0000 7fa180a00640 1 -- 172.21.15.12:0/3279346050 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa17c1a61d0 con 0x7fa17c108cf0 2024-03-23T23:09:10.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.902+0000 7fa180a00640 1 -- 172.21.15.12:0/3279346050 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa17c10edc0 con 0x7fa17c108cf0 2024-03-23T23:09:10.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.904+0000 7fa173400640 1 -- 172.21.15.12:0/3279346050 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa168044070 con 0x7fa17c108cf0 2024-03-23T23:09:10.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.905+0000 7fa173400640 1 --2- 172.21.15.12:0/3279346050 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa1600a2880 0x7fa1600a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:10.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.905+0000 7fa173400640 1 -- 172.21.15.12:0/3279346050 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fa1680f0520 con 0x7fa17c108cf0 2024-03-23T23:09:10.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.906+0000 7fa17a800640 1 --2- 172.21.15.12:0/3279346050 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa1600a2880 0x7fa1600a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:10.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.906+0000 7fa17a800640 1 --2- 172.21.15.12:0/3279346050 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa1600a2880 0x7fa1600a4d40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa17c108b50 tx=0x7fa16c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:10.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:10 smithi203 ceph-mon[36693]: pgmap v611: 97 pgs: 97 active+clean; 761 KiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:09:10.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:10.909+0000 7fa173400640 1 -- 172.21.15.12:0/3279346050 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa1680bcf20 con 0x7fa17c108cf0 2024-03-23T23:09:11.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:10 smithi173 ceph-mon[36116]: pgmap v611: 97 pgs: 97 active+clean; 761 KiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:09:11.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.155+0000 7fa180a00640 1 -- 172.21.15.12:0/3279346050 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa17c02d070 con 0x7fa17c108cf0 2024-03-23T23:09:11.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.156+0000 7fa173400640 1 -- 172.21.15.12:0/3279346050 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7fa1680c0fc0 con 0x7fa17c108cf0 2024-03-23T23:09:11.156 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:09:11.156 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:09:11.157 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:09:11.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.159+0000 7fa180a00640 1 -- 172.21.15.12:0/3279346050 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa1600a2880 msgr2=0x7fa1600a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.159+0000 7fa180a00640 1 --2- 172.21.15.12:0/3279346050 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa1600a2880 0x7fa1600a4d40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa17c108b50 tx=0x7fa16c005f90 comp rx=0 tx=0).stop 2024-03-23T23:09:11.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.159+0000 7fa180a00640 1 -- 172.21.15.12:0/3279346050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa17c108cf0 msgr2=0x7fa17c19e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.159+0000 7fa180a00640 1 --2- 172.21.15.12:0/3279346050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa17c108cf0 0x7fa17c19e320 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fa16802c8f0 tx=0x7fa16802c660 comp rx=0 tx=0).stop 2024-03-23T23:09:11.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.159+0000 7fa180a00640 1 -- 172.21.15.12:0/3279346050 shutdown_connections 2024-03-23T23:09:11.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.159+0000 7fa180a00640 1 --2- 172.21.15.12:0/3279346050 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa1600a2880 0x7fa1600a4d40 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.159+0000 7fa180a00640 1 --2- 172.21.15.12:0/3279346050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa17c1096b0 0x7fa17c1a5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.159+0000 7fa180a00640 1 --2- 172.21.15.12:0/3279346050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa17c108cf0 0x7fa17c19e320 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.159+0000 7fa180a00640 1 --2- 172.21.15.12:0/3279346050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa17c107b00 0x7fa17c19dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.159+0000 7fa180a00640 1 -- 172.21.15.12:0/3279346050 >> 172.21.15.12:0/3279346050 conn(0x7fa17c076ab0 msgr2=0x7fa17c10bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:11.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.160+0000 7fa180a00640 1 -- 172.21.15.12:0/3279346050 shutdown_connections 2024-03-23T23:09:11.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.160+0000 7fa180a00640 1 -- 172.21.15.12:0/3279346050 wait complete. 2024-03-23T23:09:11.181 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:09:11.181 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-23T23:09:11.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.408+0000 7f048ee00640 1 -- 172.21.15.12:0/1458895532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0488104ad0 msgr2=0x7f0488104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.408+0000 7f048ee00640 1 --2- 172.21.15.12:0/1458895532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0488104ad0 0x7f0488104f50 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f0470009ca0 tx=0x7f047002c0f0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.408+0000 7f048ee00640 1 -- 172.21.15.12:0/1458895532 shutdown_connections 2024-03-23T23:09:11.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.408+0000 7f048ee00640 1 --2- 172.21.15.12:0/1458895532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0488105490 0x7f048810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.408+0000 7f048ee00640 1 --2- 172.21.15.12:0/1458895532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0488104ad0 0x7f0488104f50 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.408+0000 7f048ee00640 1 --2- 172.21.15.12:0/1458895532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04881038e0 0x7f0488103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.408+0000 7f048ee00640 1 -- 172.21.15.12:0/1458895532 >> 172.21.15.12:0/1458895532 conn(0x7f04880fee50 msgr2=0x7f0488101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:11.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.408+0000 7f048ee00640 1 -- 172.21.15.12:0/1458895532 shutdown_connections 2024-03-23T23:09:11.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.408+0000 7f048ee00640 1 -- 172.21.15.12:0/1458895532 wait complete. 2024-03-23T23:09:11.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.410+0000 7f048ee00640 1 Processor -- start 2024-03-23T23:09:11.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.410+0000 7f048ee00640 1 -- start start 2024-03-23T23:09:11.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.410+0000 7f048ee00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04881038e0 0x7f0488199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.410+0000 7f048ee00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0488104ad0 0x7f048819a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.410+0000 7f048ee00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0488105490 0x7f04881a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.410+0000 7f048ee00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048810f540 con 0x7f04881038e0 2024-03-23T23:09:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.410+0000 7f048ee00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048810f3c0 con 0x7f0488104ad0 2024-03-23T23:09:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.410+0000 7f048ee00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048810f6c0 con 0x7f0488105490 2024-03-23T23:09:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.410+0000 7f0487e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04881038e0 0x7f0488199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.410+0000 7f0487400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0488104ad0 0x7f048819a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.411+0000 7f0487400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0488104ad0 0x7f048819a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:34062/0 (socket says 172.21.15.12:34062) 2024-03-23T23:09:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.411+0000 7f0487400640 1 -- 172.21.15.12:0/2135766482 learned_addr learned my addr 172.21.15.12:0/2135766482 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.411+0000 7f048ce00640 1 --2- 172.21.15.12:0/2135766482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0488105490 0x7f04881a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.411+0000 7f0487400640 1 -- 172.21.15.12:0/2135766482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0488105490 msgr2=0x7f04881a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.411+0000 7f0487400640 1 --2- 172.21.15.12:0/2135766482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0488105490 0x7f04881a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.411+0000 7f0487400640 1 -- 172.21.15.12:0/2135766482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04881038e0 msgr2=0x7f0488199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.411+0000 7f0487400640 1 --2- 172.21.15.12:0/2135766482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04881038e0 0x7f0488199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.411+0000 7f0487400640 1 -- 172.21.15.12:0/2135766482 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04881a1800 con 0x7f0488104ad0 2024-03-23T23:09:11.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.411+0000 7f0487400640 1 --2- 172.21.15.12:0/2135766482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0488104ad0 0x7f048819a120 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f04700325f0 tx=0x7f0470032ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:11.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.412+0000 7f0484c00640 1 -- 172.21.15.12:0/2135766482 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0470041020 con 0x7f0488104ad0 2024-03-23T23:09:11.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.412+0000 7f0484c00640 1 -- 172.21.15.12:0/2135766482 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0470034cd0 con 0x7f0488104ad0 2024-03-23T23:09:11.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.412+0000 7f0484c00640 1 -- 172.21.15.12:0/2135766482 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0470039040 con 0x7f0488104ad0 2024-03-23T23:09:11.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.412+0000 7f048ee00640 1 -- 172.21.15.12:0/2135766482 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04881a1a90 con 0x7f0488104ad0 2024-03-23T23:09:11.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.412+0000 7f048ee00640 1 -- 172.21.15.12:0/2135766482 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04881a1f00 con 0x7f0488104ad0 2024-03-23T23:09:11.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.415+0000 7f0484c00640 1 -- 172.21.15.12:0/2135766482 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0470044020 con 0x7f0488104ad0 2024-03-23T23:09:11.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.415+0000 7f0491bf5740 1 Processor -- start 2024-03-23T23:09:11.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.415+0000 7f0491bf5740 1 -- start start 2024-03-23T23:09:11.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.415+0000 7f0491bf5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c7467810 0x55e4c7467c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.415+0000 7f0491bf5740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e4c74689e0 0x55e4c7468e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.415+0000 7f0491bf5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c74693a0 0x55e4c746fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.415+0000 7f0491bf5740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4c73cc2c0 con 0x55e4c74689e0 2024-03-23T23:09:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.415+0000 7f0491bf5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4c7374c90 con 0x55e4c7467810 2024-03-23T23:09:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.415+0000 7f0491bf5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4c73c8360 con 0x55e4c74693a0 2024-03-23T23:09:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.416+0000 7f048ce00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c7467810 0x55e4c7467c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.416+0000 7f048ce00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c7467810 0x55e4c7467c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:34070/0 (socket says 172.21.15.12:34070) 2024-03-23T23:09:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.416+0000 7f048ce00640 1 -- 172.21.15.12:0/4280779916 learned_addr learned my addr 172.21.15.12:0/4280779916 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.416+0000 7f0487400640 1 --2- 172.21.15.12:0/4280779916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c74693a0 0x55e4c746fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.416+0000 7f0487e00640 1 --2- 172.21.15.12:0/4280779916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e4c74689e0 0x55e4c7468e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.416+0000 7f048ce00640 1 -- 172.21.15.12:0/4280779916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c74693a0 msgr2=0x55e4c746fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.416+0000 7f048ce00640 1 --2- 172.21.15.12:0/4280779916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c74693a0 0x55e4c746fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.416+0000 7f048ce00640 1 -- 172.21.15.12:0/4280779916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e4c74689e0 msgr2=0x55e4c7468e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.416+0000 7f048ce00640 1 --2- 172.21.15.12:0/4280779916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e4c74689e0 0x55e4c7468e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.416+0000 7f048ce00640 1 -- 172.21.15.12:0/4280779916 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e4c7399080 con 0x55e4c7467810 2024-03-23T23:09:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.416+0000 7f048ce00640 1 --2- 172.21.15.12:0/4280779916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c7467810 0x55e4c7467c30 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f0478009ca0 tx=0x7f0478011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5e821bfde62cfe0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:11.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.417+0000 7f047d600640 1 -- 172.21.15.12:0/4280779916 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047801e070 con 0x55e4c7467810 2024-03-23T23:09:11.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.417+0000 7f047d600640 1 -- 172.21.15.12:0/4280779916 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0478009e50 con 0x55e4c7467810 2024-03-23T23:09:11.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.417+0000 7f047d600640 1 -- 172.21.15.12:0/4280779916 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047800ea00 con 0x55e4c7467810 2024-03-23T23:09:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.417+0000 7f0484c00640 1 --2- 172.21.15.12:0/2135766482 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f046c0a2880 0x7f046c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.417+0000 7f0491bf5740 1 -- 172.21.15.12:0/4280779916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c7467810 msgr2=0x55e4c7467c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.417+0000 7f0491bf5740 1 --2- 172.21.15.12:0/4280779916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c7467810 0x55e4c7467c30 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f0478009ca0 tx=0x7f0478011810 comp rx=0 tx=0).stop 2024-03-23T23:09:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.417+0000 7f0491bf5740 1 -- 172.21.15.12:0/4280779916 shutdown_connections 2024-03-23T23:09:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.417+0000 7f0491bf5740 1 --2- 172.21.15.12:0/4280779916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c74693a0 0x55e4c746fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.417+0000 7f0491bf5740 1 --2- 172.21.15.12:0/4280779916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e4c74689e0 0x55e4c7468e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.417+0000 7f0484c00640 1 -- 172.21.15.12:0/2135766482 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f04700f0850 con 0x7f0488104ad0 2024-03-23T23:09:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.417+0000 7f0491bf5740 1 --2- 172.21.15.12:0/4280779916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c7467810 0x55e4c7467c30 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.418+0000 7f0487e00640 1 --2- 172.21.15.12:0/2135766482 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f046c0a2880 0x7f046c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.418+0000 7f0491bf5740 1 -- 172.21.15.12:0/4280779916 >> 172.21.15.12:0/4280779916 conn(0x55e4c7462c00 msgr2=0x55e4c7463000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.418+0000 7f0491bf5740 1 -- 172.21.15.12:0/4280779916 shutdown_connections 2024-03-23T23:09:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.418+0000 7f0491bf5740 1 -- 172.21.15.12:0/4280779916 wait complete. 2024-03-23T23:09:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.418+0000 7f0491bf5740 1 -- start start 2024-03-23T23:09:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.418+0000 7f0487e00640 1 --2- 172.21.15.12:0/2135766482 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f046c0a2880 0x7f046c0a4d40 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x55e4c7463b70 tx=0x7f0480005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.418+0000 7f0491bf5740 1 Processor -- start 2024-03-23T23:09:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.419+0000 7f0491bf5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c7467810 0x55e4c7471ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.419+0000 7f0491bf5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c74689e0 0x55e4c7472000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.419+0000 7f0491bf5740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e4c74693a0 0x55e4c7479040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.419+0000 7f0491bf5740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4c74703b0 con 0x55e4c74693a0 2024-03-23T23:09:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.419+0000 7f0491bf5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4c73880d0 con 0x55e4c74689e0 2024-03-23T23:09:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.419+0000 7f0491bf5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4c7470530 con 0x55e4c7467810 2024-03-23T23:09:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.419+0000 7f0491bf5740 10 client.0 _ll_register_callbacks cb 0x55e4c738ce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.419+0000 7f0487e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c7467810 0x55e4c7471ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.419+0000 7f0487400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c74689e0 0x55e4c7472000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.419+0000 7f0487400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c74689e0 0x55e4c7472000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:34074/0 (socket says 172.21.15.12:34074) 2024-03-23T23:09:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.419+0000 7f0487e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c7467810 0x55e4c7471ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54900/0 (socket says 172.21.15.12:54900) 2024-03-23T23:09:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.419+0000 7f0487400640 1 -- 172.21.15.12:0/2775712846 learned_addr learned my addr 172.21.15.12:0/2775712846 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.419+0000 7f048ce00640 1 --2- 172.21.15.12:0/2775712846 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e4c74693a0 0x55e4c7479040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.419+0000 7f0487400640 1 -- 172.21.15.12:0/2775712846 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c7467810 msgr2=0x55e4c7471ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.419+0000 7f0487400640 1 --2- 172.21.15.12:0/2775712846 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c7467810 0x55e4c7471ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.420+0000 7f0487400640 1 -- 172.21.15.12:0/2775712846 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e4c74693a0 msgr2=0x55e4c7479040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.420+0000 7f0487400640 1 --2- 172.21.15.12:0/2775712846 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e4c74693a0 0x55e4c7479040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.420+0000 7f0487400640 1 -- 172.21.15.12:0/2775712846 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e4c70e03f0 con 0x55e4c74689e0 2024-03-23T23:09:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.420+0000 7f0487e00640 1 --2- 172.21.15.12:0/2775712846 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c7467810 0x55e4c7471ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.420+0000 7f0487400640 1 --2- 172.21.15.12:0/2775712846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c74689e0 0x55e4c7472000 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f04700bcf30 tx=0x7f04700b94d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=284f347b2ec0c933 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.420+0000 7f0476000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:09:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.420+0000 7f0476000640 1 -- 172.21.15.12:0/2775712846 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04700bcb90 con 0x55e4c74689e0 2024-03-23T23:09:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.420+0000 7f0476000640 1 -- 172.21.15.12:0/2775712846 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0470003a70 con 0x55e4c74689e0 2024-03-23T23:09:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.420+0000 7f0491bf5740 1 -- 172.21.15.12:0/2775712846 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0478009e50 con 0x55e4c74689e0 2024-03-23T23:09:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.420+0000 7f0476000640 1 -- 172.21.15.12:0/2775712846 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0470034920 con 0x55e4c74689e0 2024-03-23T23:09:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.420+0000 7f0491bf5740 10 client.27356 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.420+0000 7f0491bf5740 1 -- 172.21.15.12:0/2775712846 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55e4c71aa7d0 con 0x55e4c74689e0 2024-03-23T23:09:11.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.421+0000 7f0476000640 1 -- 172.21.15.12:0/2775712846 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0478009e50 con 0x55e4c74689e0 2024-03-23T23:09:11.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.421+0000 7f0476000640 1 -- 172.21.15.12:0/2775712846 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f047003c030 con 0x55e4c74689e0 2024-03-23T23:09:11.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.421+0000 7f0491bf5740 10 client.27356 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:11.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.421+0000 7f0491bf5740 10 client.27356 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:09:11.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.421+0000 7f0491bf5740 20 client.27356 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:11.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.421+0000 7f0491bf5740 1 --2- 172.21.15.12:0/2775712846 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55e4c747d470 0x55e4c747f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.421+0000 7f0491bf5740 4 client.27356 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:09:11.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.421+0000 7f0491bf5740 1 -- 172.21.15.12:0/2775712846 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e4c73d9800 con 0x55e4c747d470 2024-03-23T23:09:11.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.421+0000 7f0487e00640 1 --2- 172.21.15.12:0/2775712846 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55e4c747d470 0x55e4c747f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.422+0000 7f0487e00640 1 --2- 172.21.15.12:0/2775712846 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55e4c747d470 0x55e4c747f8b0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c1375cd82e86b51e server_cookie=a31c303092d17405 in_seq=0 out_seq=0 2024-03-23T23:09:11.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.422+0000 7f0476000640 10 client.27356 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:09:11.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.422+0000 7f0476000640 1 -- 172.21.15.12:0/2775712846 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e4c73d9800 con 0x55e4c747d470 2024-03-23T23:09:11.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.422+0000 7f0476000640 10 client.27356 handle_command_reply: tid=0 2024-03-23T23:09:11.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.422+0000 7f0491bf5740 1 client.27356 shutdown 2024-03-23T23:09:11.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.423+0000 7f0491bf5740 1 -- 172.21.15.12:0/2775712846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c74689e0 msgr2=0x55e4c7472000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.423+0000 7f0491bf5740 1 --2- 172.21.15.12:0/2775712846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c74689e0 0x55e4c7472000 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f04700bcf30 tx=0x7f04700b94d0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.423+0000 7f0491bf5740 1 -- 172.21.15.12:0/2775712846 shutdown_connections 2024-03-23T23:09:11.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.423+0000 7f0491bf5740 1 --2- 172.21.15.12:0/2775712846 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55e4c747d470 0x55e4c747f8b0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.423+0000 7f0491bf5740 1 --2- 172.21.15.12:0/2775712846 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e4c74693a0 0x55e4c7479040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.423+0000 7f0491bf5740 1 --2- 172.21.15.12:0/2775712846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c74689e0 0x55e4c7472000 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.423+0000 7f0491bf5740 1 --2- 172.21.15.12:0/2775712846 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c7467810 0x55e4c7471ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.423+0000 7f0491bf5740 1 -- 172.21.15.12:0/2775712846 >> 172.21.15.12:0/2775712846 conn(0x55e4c7462c00 msgr2=0x55e4c7461eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:11.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.423+0000 7f0476000640 0 client.27356 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:09:11.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.423+0000 7f0491bf5740 1 -- 172.21.15.12:0/2775712846 shutdown_connections 2024-03-23T23:09:11.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.423+0000 7f0491bf5740 1 -- 172.21.15.12:0/2775712846 wait complete. 2024-03-23T23:09:11.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.423+0000 7f0491bf5740 20 client.27356 trim_cache size 0 max 16384 2024-03-23T23:09:11.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.437+0000 7f0491bf5740 1 Processor -- start 2024-03-23T23:09:11.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.437+0000 7f0491bf5740 1 -- start start 2024-03-23T23:09:11.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.437+0000 7f0491bf5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c7467810 0x55e4c746c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.437+0000 7f0491bf5740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e4c74689e0 0x55e4c746c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.437+0000 7f0491bf5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c74693a0 0x55e4c7479000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.437+0000 7f0491bf5740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04881ad180 con 0x55e4c74689e0 2024-03-23T23:09:11.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.437+0000 7f0491bf5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04881aac80 con 0x55e4c74693a0 2024-03-23T23:09:11.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.437+0000 7f0491bf5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04881a8d20 con 0x55e4c7467810 2024-03-23T23:09:11.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.438+0000 7f048ce00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c74693a0 0x55e4c7479000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.438+0000 7f0487e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c7467810 0x55e4c746c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.438+0000 7f048ce00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c74693a0 0x55e4c7479000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:34088/0 (socket says 172.21.15.12:34088) 2024-03-23T23:09:11.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.438+0000 7f048ce00640 1 -- 172.21.15.12:0/1694559412 learned_addr learned my addr 172.21.15.12:0/1694559412 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:11.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.438+0000 7f0487e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c7467810 0x55e4c746c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54906/0 (socket says 172.21.15.12:54906) 2024-03-23T23:09:11.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.438+0000 7f0487400640 1 --2- 172.21.15.12:0/1694559412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e4c74689e0 0x55e4c746c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.438+0000 7f048ce00640 1 -- 172.21.15.12:0/1694559412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c7467810 msgr2=0x55e4c746c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:11.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.438+0000 7f048ce00640 1 --2- 172.21.15.12:0/1694559412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c7467810 0x55e4c746c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.438+0000 7f048ce00640 1 -- 172.21.15.12:0/1694559412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e4c74689e0 msgr2=0x55e4c746c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:11.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.438+0000 7f048ce00640 1 --2- 172.21.15.12:0/1694559412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e4c74689e0 0x55e4c746c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.438+0000 7f048ce00640 1 -- 172.21.15.12:0/1694559412 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e4c747ff30 con 0x55e4c74693a0 2024-03-23T23:09:11.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.438+0000 7f0487e00640 1 --2- 172.21.15.12:0/1694559412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c7467810 0x55e4c746c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:11.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.438+0000 7f048ce00640 1 --2- 172.21.15.12:0/1694559412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c74693a0 0x55e4c7479000 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x55e4c74688e0 tx=0x7f04780096d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8a897388621609e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:11.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.439+0000 7f047cc00640 1 -- 172.21.15.12:0/1694559412 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047801e070 con 0x55e4c74693a0 2024-03-23T23:09:11.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.439+0000 7f047cc00640 1 -- 172.21.15.12:0/1694559412 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0478029070 con 0x55e4c74693a0 2024-03-23T23:09:11.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.439+0000 7f047cc00640 1 -- 172.21.15.12:0/1694559412 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0478003e10 con 0x55e4c74693a0 2024-03-23T23:09:11.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.439+0000 7f0491bf5740 1 -- 172.21.15.12:0/1694559412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c74693a0 msgr2=0x55e4c7479000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.439+0000 7f0491bf5740 1 --2- 172.21.15.12:0/1694559412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c74693a0 0x55e4c7479000 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x55e4c74688e0 tx=0x7f04780096d0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.439+0000 7f0491bf5740 1 -- 172.21.15.12:0/1694559412 shutdown_connections 2024-03-23T23:09:11.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.439+0000 7f0491bf5740 1 --2- 172.21.15.12:0/1694559412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c74693a0 0x55e4c7479000 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.439+0000 7f0491bf5740 1 --2- 172.21.15.12:0/1694559412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e4c74689e0 0x55e4c746c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.439+0000 7f0491bf5740 1 --2- 172.21.15.12:0/1694559412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c7467810 0x55e4c746c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.439+0000 7f0491bf5740 1 -- 172.21.15.12:0/1694559412 >> 172.21.15.12:0/1694559412 conn(0x55e4c7462c00 msgr2=0x55e4c746bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:11.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.439+0000 7f0491bf5740 1 -- 172.21.15.12:0/1694559412 shutdown_connections 2024-03-23T23:09:11.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.439+0000 7f0491bf5740 1 -- 172.21.15.12:0/1694559412 wait complete. 2024-03-23T23:09:11.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.440+0000 7f0491bf5740 1 -- start start 2024-03-23T23:09:11.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.440+0000 7f0491bf5740 1 Processor -- start 2024-03-23T23:09:11.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.440+0000 7f0491bf5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c7467810 0x55e4c7478910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.440+0000 7f0491bf5740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e4c74689e0 0x55e4c7478e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.440+0000 7f0491bf5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c74693a0 0x55e4c7476f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.440+0000 7f0491bf5740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04881a9320 con 0x55e4c74689e0 2024-03-23T23:09:11.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.440+0000 7f0491bf5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04881aa620 con 0x55e4c74693a0 2024-03-23T23:09:11.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.440+0000 7f0491bf5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04881a9960 con 0x55e4c7467810 2024-03-23T23:09:11.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.441+0000 7f0491bf5740 10 client.0 _ll_register_callbacks cb 0x7f0470034e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:11.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.441+0000 7f0487e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c7467810 0x55e4c7478910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.441+0000 7f0487e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c7467810 0x55e4c7478910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54916/0 (socket says 172.21.15.12:54916) 2024-03-23T23:09:11.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.441+0000 7f0487e00640 1 -- 172.21.15.12:0/1990358358 learned_addr learned my addr 172.21.15.12:0/1990358358 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:11.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.441+0000 7f048ce00640 1 --2- 172.21.15.12:0/1990358358 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c74693a0 0x55e4c7476f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.441+0000 7f0487400640 1 --2- 172.21.15.12:0/1990358358 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e4c74689e0 0x55e4c7478e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.441+0000 7f0487e00640 1 -- 172.21.15.12:0/1990358358 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c74693a0 msgr2=0x55e4c7476f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:11.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.441+0000 7f0487e00640 1 --2- 172.21.15.12:0/1990358358 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c74693a0 0x55e4c7476f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.441+0000 7f0487e00640 1 -- 172.21.15.12:0/1990358358 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e4c74689e0 msgr2=0x55e4c7478e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:11.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.441+0000 7f0487e00640 1 --2- 172.21.15.12:0/1990358358 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e4c74689e0 0x55e4c7478e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.441+0000 7f0487e00640 1 -- 172.21.15.12:0/1990358358 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e4c74776e0 con 0x55e4c7467810 2024-03-23T23:09:11.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.441+0000 7f0487e00640 1 --2- 172.21.15.12:0/1990358358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c7467810 0x55e4c7478910 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0480009fd0 tx=0x7f04800046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea19a8994cb4c97d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:11.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.441+0000 7f0476000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:09:11.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.442+0000 7f0476000640 1 -- 172.21.15.12:0/1990358358 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048002b680 con 0x55e4c7467810 2024-03-23T23:09:11.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.442+0000 7f0476000640 1 -- 172.21.15.12:0/1990358358 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f048002b820 con 0x55e4c7467810 2024-03-23T23:09:11.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.442+0000 7f0491bf5740 1 -- 172.21.15.12:0/1990358358 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0478029070 con 0x55e4c7467810 2024-03-23T23:09:11.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.442+0000 7f0476000640 1 -- 172.21.15.12:0/1990358358 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f048002ba20 con 0x55e4c7467810 2024-03-23T23:09:11.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.442+0000 7f0491bf5740 10 client.27334 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:11.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.442+0000 7f0491bf5740 1 -- 172.21.15.12:0/1990358358 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55e4c74d98b0 con 0x55e4c7467810 2024-03-23T23:09:11.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.442+0000 7f0476000640 1 -- 172.21.15.12:0/1990358358 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0478029070 con 0x55e4c7467810 2024-03-23T23:09:11.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.442+0000 7f0476000640 1 -- 172.21.15.12:0/1990358358 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f0480033400 con 0x55e4c7467810 2024-03-23T23:09:11.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.442+0000 7f0491bf5740 10 client.27334 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:11.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.442+0000 7f0491bf5740 10 client.27334 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:09:11.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.442+0000 7f0491bf5740 20 client.27334 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:11.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.442+0000 7f0491bf5740 1 --2- 172.21.15.12:0/1990358358 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55e4c74da1f0 0x55e4c74dc610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.442+0000 7f0491bf5740 4 client.27334 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:09:11.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.442+0000 7f0491bf5740 1 -- 172.21.15.12:0/1990358358 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e4c74718b0 con 0x55e4c74da1f0 2024-03-23T23:09:11.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.443+0000 7f0487400640 1 --2- 172.21.15.12:0/1990358358 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55e4c74da1f0 0x55e4c74dc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.443+0000 7f0487400640 1 --2- 172.21.15.12:0/1990358358 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55e4c74da1f0 0x55e4c74dc610 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5800c249d3a7758 server_cookie=940ed53363655ecd in_seq=0 out_seq=0 2024-03-23T23:09:11.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.443+0000 7f0476000640 10 client.27334 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:09:11.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.443+0000 7f0476000640 1 -- 172.21.15.12:0/1990358358 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e4c74718b0 con 0x55e4c74da1f0 2024-03-23T23:09:11.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.443+0000 7f0476000640 10 client.27334 handle_command_reply: tid=0 2024-03-23T23:09:11.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.443+0000 7f0491bf5740 1 client.27334 shutdown 2024-03-23T23:09:11.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.444+0000 7f0491bf5740 1 -- 172.21.15.12:0/1990358358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c7467810 msgr2=0x55e4c7478910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.444+0000 7f0491bf5740 1 --2- 172.21.15.12:0/1990358358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c7467810 0x55e4c7478910 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0480009fd0 tx=0x7f04800046c0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.444+0000 7f0491bf5740 1 -- 172.21.15.12:0/1990358358 shutdown_connections 2024-03-23T23:09:11.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.444+0000 7f0491bf5740 1 --2- 172.21.15.12:0/1990358358 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55e4c74da1f0 0x55e4c74dc610 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.444+0000 7f0491bf5740 1 --2- 172.21.15.12:0/1990358358 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e4c74693a0 0x55e4c7476f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.444+0000 7f0491bf5740 1 --2- 172.21.15.12:0/1990358358 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e4c74689e0 0x55e4c7478e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.444+0000 7f0491bf5740 1 --2- 172.21.15.12:0/1990358358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e4c7467810 0x55e4c7478910 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.444+0000 7f0491bf5740 1 -- 172.21.15.12:0/1990358358 >> 172.21.15.12:0/1990358358 conn(0x55e4c7462c00 msgr2=0x55e4c746c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:11.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.444+0000 7f0476000640 0 client.27334 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:09:11.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.444+0000 7f0491bf5740 1 -- 172.21.15.12:0/1990358358 shutdown_connections 2024-03-23T23:09:11.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.444+0000 7f0491bf5740 1 -- 172.21.15.12:0/1990358358 wait complete. 2024-03-23T23:09:11.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.444+0000 7f0491bf5740 20 client.27334 trim_cache size 0 max 16384 2024-03-23T23:09:11.448 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:09:11.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.445+0000 7f048ee00640 1 -- 172.21.15.12:0/2135766482 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f046c0a2880 msgr2=0x7f046c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.445+0000 7f048ee00640 1 --2- 172.21.15.12:0/2135766482 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f046c0a2880 0x7f046c0a4d40 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x55e4c7463b70 tx=0x7f0480005f90 comp rx=0 tx=0).stop 2024-03-23T23:09:11.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.445+0000 7f048ee00640 1 -- 172.21.15.12:0/2135766482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0488104ad0 msgr2=0x7f048819a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.445+0000 7f048ee00640 1 --2- 172.21.15.12:0/2135766482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0488104ad0 0x7f048819a120 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f04700325f0 tx=0x7f0470032ec0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.445+0000 7f048ee00640 1 -- 172.21.15.12:0/2135766482 shutdown_connections 2024-03-23T23:09:11.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.445+0000 7f048ee00640 1 --2- 172.21.15.12:0/2135766482 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f046c0a2880 0x7f046c0a4d40 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.445+0000 7f048ee00640 1 --2- 172.21.15.12:0/2135766482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0488105490 0x7f04881a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.445+0000 7f048ee00640 1 --2- 172.21.15.12:0/2135766482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0488104ad0 0x7f048819a120 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.445+0000 7f048ee00640 1 --2- 172.21.15.12:0/2135766482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04881038e0 0x7f0488199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.445+0000 7f048ee00640 1 -- 172.21.15.12:0/2135766482 >> 172.21.15.12:0/2135766482 conn(0x7f04880fee50 msgr2=0x7f0488107ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:11.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.445+0000 7f048ee00640 1 -- 172.21.15.12:0/2135766482 shutdown_connections 2024-03-23T23:09:11.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.445+0000 7f048ee00640 1 -- 172.21.15.12:0/2135766482 wait complete. 2024-03-23T23:09:11.462 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:09:11.463 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-23T23:09:11.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.693+0000 7f88ffe00640 1 -- 172.21.15.12:0/247860370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8900105350 msgr2=0x7f890010bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.693+0000 7f88ffe00640 1 --2- 172.21.15.12:0/247860370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8900105350 0x7f890010bbe0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f88f8009ca0 tx=0x7f88f802c100 comp rx=0 tx=0).stop 2024-03-23T23:09:11.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.693+0000 7f88ffe00640 1 -- 172.21.15.12:0/247860370 shutdown_connections 2024-03-23T23:09:11.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.693+0000 7f88ffe00640 1 --2- 172.21.15.12:0/247860370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8900105350 0x7f890010bbe0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.693+0000 7f88ffe00640 1 --2- 172.21.15.12:0/247860370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8900104990 0x7f8900104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.693+0000 7f88ffe00640 1 --2- 172.21.15.12:0/247860370 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89001037a0 0x7f8900103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.693+0000 7f88ffe00640 1 -- 172.21.15.12:0/247860370 >> 172.21.15.12:0/247860370 conn(0x7f89000fed10 msgr2=0x7f8900101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:11.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.693+0000 7f88ffe00640 1 -- 172.21.15.12:0/247860370 shutdown_connections 2024-03-23T23:09:11.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.693+0000 7f88ffe00640 1 -- 172.21.15.12:0/247860370 wait complete. 2024-03-23T23:09:11.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.694+0000 7f88ffe00640 1 Processor -- start 2024-03-23T23:09:11.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.694+0000 7f88ffe00640 1 -- start start 2024-03-23T23:09:11.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.694+0000 7f88ffe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89001037a0 0x7f8900199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.694+0000 7f88ffe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8900104990 0x7f8900199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.694+0000 7f88ffe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8900105350 0x7f89001a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.694+0000 7f88ffe00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890010f400 con 0x7f8900105350 2024-03-23T23:09:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.694+0000 7f88ffe00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890010f280 con 0x7f89001037a0 2024-03-23T23:09:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.694+0000 7f88ffe00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890010f580 con 0x7f8900104990 2024-03-23T23:09:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.695+0000 7f88fea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89001037a0 0x7f8900199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.695+0000 7f88fea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89001037a0 0x7f8900199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:34118/0 (socket says 172.21.15.12:34118) 2024-03-23T23:09:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.695+0000 7f88fea00640 1 -- 172.21.15.12:0/2708375748 learned_addr learned my addr 172.21.15.12:0/2708375748 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.695+0000 7f88fe000640 1 --2- 172.21.15.12:0/2708375748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8900104990 0x7f8900199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.695+0000 7f88ff400640 1 --2- 172.21.15.12:0/2708375748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8900105350 0x7f89001a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.695+0000 7f88fea00640 1 -- 172.21.15.12:0/2708375748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8900104990 msgr2=0x7f8900199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.695+0000 7f88fea00640 1 --2- 172.21.15.12:0/2708375748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8900104990 0x7f8900199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.695+0000 7f88fea00640 1 -- 172.21.15.12:0/2708375748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8900105350 msgr2=0x7f89001a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.695+0000 7f88fea00640 1 --2- 172.21.15.12:0/2708375748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8900105350 0x7f89001a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.695+0000 7f88fea00640 1 -- 172.21.15.12:0/2708375748 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89001a1630 con 0x7f89001037a0 2024-03-23T23:09:11.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.695+0000 7f88fea00640 1 --2- 172.21.15.12:0/2708375748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89001037a0 0x7f8900199a10 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f88f000d770 tx=0x7f88f000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:11.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.696+0000 7f8904a00640 1 -- 172.21.15.12:0/2708375748 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f0017070 con 0x7f89001037a0 2024-03-23T23:09:11.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.696+0000 7f8904a00640 1 -- 172.21.15.12:0/2708375748 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88f0009c30 con 0x7f89001037a0 2024-03-23T23:09:11.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.696+0000 7f8904a00640 1 -- 172.21.15.12:0/2708375748 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f0003dd0 con 0x7f89001037a0 2024-03-23T23:09:11.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.696+0000 7f88ffe00640 1 -- 172.21.15.12:0/2708375748 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89001a1920 con 0x7f89001037a0 2024-03-23T23:09:11.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.696+0000 7f88ffe00640 1 -- 172.21.15.12:0/2708375748 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89001a1e70 con 0x7f89001037a0 2024-03-23T23:09:11.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.698+0000 7f8904a00640 1 -- 172.21.15.12:0/2708375748 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f88f0003f70 con 0x7f89001037a0 2024-03-23T23:09:11.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.699+0000 7f8908ba8740 1 Processor -- start 2024-03-23T23:09:11.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.699+0000 7f8908ba8740 1 -- start start 2024-03-23T23:09:11.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.699+0000 7f8908ba8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d4bd739f0 0x556d4bd73e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.699+0000 7f8908ba8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d4bd74bc0 0x556d4bd75040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.699+0000 7f8908ba8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd75580 0x556d4bd7be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.699+0000 7f8908ba8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d4bcd82c0 con 0x556d4bd739f0 2024-03-23T23:09:11.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.699+0000 7f8908ba8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d4bc80c90 con 0x556d4bd75580 2024-03-23T23:09:11.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.699+0000 7f8908ba8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d4bcd4360 con 0x556d4bd74bc0 2024-03-23T23:09:11.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.700+0000 7f88fe000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd75580 0x556d4bd7be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.700+0000 7f88fea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d4bd74bc0 0x556d4bd75040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.700+0000 7f88ff400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d4bd739f0 0x556d4bd73e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.700+0000 7f88fe000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd75580 0x556d4bd7be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:34128/0 (socket says 172.21.15.12:34128) 2024-03-23T23:09:11.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.700+0000 7f88fe000640 1 -- 172.21.15.12:0/270191151 learned_addr learned my addr 172.21.15.12:0/270191151 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:11.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.700+0000 7f8904a00640 1 --2- 172.21.15.12:0/2708375748 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f88e40a2880 0x7f88e40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.700+0000 7f88fe000640 1 -- 172.21.15.12:0/270191151 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d4bd74bc0 msgr2=0x556d4bd75040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:11.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.700+0000 7f88fe000640 1 --2- 172.21.15.12:0/270191151 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d4bd74bc0 0x556d4bd75040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.700+0000 7f88fe000640 1 -- 172.21.15.12:0/270191151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d4bd739f0 msgr2=0x556d4bd73e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:11.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.700+0000 7f88fe000640 1 --2- 172.21.15.12:0/270191151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d4bd739f0 0x556d4bd73e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.700+0000 7f88fe000640 1 -- 172.21.15.12:0/270191151 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d4bca5080 con 0x556d4bd75580 2024-03-23T23:09:11.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.700+0000 7f8904a00640 1 -- 172.21.15.12:0/2708375748 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f88f00ce300 con 0x7f89001037a0 2024-03-23T23:09:11.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.700+0000 7f88fe000640 1 --2- 172.21.15.12:0/270191151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd75580 0x556d4bd7be10 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00bcb0 tx=0x7f88ec0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92f6caf045321d50 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:11.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.701+0000 7f88ff400640 1 --2- 172.21.15.12:0/2708375748 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f88e40a2880 0x7f88e40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.701+0000 7f88ebe00640 1 -- 172.21.15.12:0/270191151 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec004030 con 0x556d4bd75580 2024-03-23T23:09:11.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.701+0000 7f88ebe00640 1 -- 172.21.15.12:0/270191151 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88ec0041d0 con 0x556d4bd75580 2024-03-23T23:09:11.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.701+0000 7f88ebe00640 1 -- 172.21.15.12:0/270191151 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec0117b0 con 0x556d4bd75580 2024-03-23T23:09:11.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.701+0000 7f8908ba8740 1 -- 172.21.15.12:0/270191151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd75580 msgr2=0x556d4bd7be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.701+0000 7f8908ba8740 1 --2- 172.21.15.12:0/270191151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd75580 0x556d4bd7be10 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00bcb0 tx=0x7f88ec0107a0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.701+0000 7f88ff400640 1 --2- 172.21.15.12:0/2708375748 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f88e40a2880 0x7f88e40a4d40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f88f802c640 tx=0x7f88f802e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:11.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.702+0000 7f8908ba8740 1 -- 172.21.15.12:0/270191151 shutdown_connections 2024-03-23T23:09:11.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.702+0000 7f8908ba8740 1 --2- 172.21.15.12:0/270191151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd75580 0x556d4bd7be10 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.702+0000 7f8908ba8740 1 --2- 172.21.15.12:0/270191151 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d4bd74bc0 0x556d4bd75040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.702+0000 7f8908ba8740 1 --2- 172.21.15.12:0/270191151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d4bd739f0 0x556d4bd73e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.702+0000 7f8908ba8740 1 -- 172.21.15.12:0/270191151 >> 172.21.15.12:0/270191151 conn(0x556d4bd6ede0 msgr2=0x556d4bd6f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:11.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.702+0000 7f8908ba8740 1 -- 172.21.15.12:0/270191151 shutdown_connections 2024-03-23T23:09:11.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.702+0000 7f8908ba8740 1 -- 172.21.15.12:0/270191151 wait complete. 2024-03-23T23:09:11.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.702+0000 7f8908ba8740 1 -- start start 2024-03-23T23:09:11.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.702+0000 7f8908ba8740 1 Processor -- start 2024-03-23T23:09:11.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.703+0000 7f8908ba8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d4bd739f0 0x556d4bd7dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.703+0000 7f8908ba8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d4bd74bc0 0x556d4bd7e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.703+0000 7f8908ba8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd75580 0x556d4bd85180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.703+0000 7f8908ba8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d4bd7c590 con 0x556d4bd739f0 2024-03-23T23:09:11.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.703+0000 7f8908ba8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d4bc940d0 con 0x556d4bd75580 2024-03-23T23:09:11.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.703+0000 7f8908ba8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d4bd7c710 con 0x556d4bd74bc0 2024-03-23T23:09:11.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.703+0000 7f8908ba8740 10 client.0 _ll_register_callbacks cb 0x556d4bc98e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:11.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.703+0000 7f88fea00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d4bd739f0 0x556d4bd7dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.703+0000 7f88fe000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d4bd74bc0 0x556d4bd7e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.703+0000 7f88fea00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d4bd739f0 0x556d4bd7dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39398/0 (socket says 172.21.15.12:39398) 2024-03-23T23:09:11.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.703+0000 7f88fea00640 1 -- 172.21.15.12:0/273003944 learned_addr learned my addr 172.21.15.12:0/273003944 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:11.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.703+0000 7f88ff400640 1 --2- 172.21.15.12:0/273003944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd75580 0x556d4bd85180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.704+0000 7f88fe000640 1 -- 172.21.15.12:0/273003944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd75580 msgr2=0x556d4bd85180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:11.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.704+0000 7f88fe000640 1 --2- 172.21.15.12:0/273003944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd75580 0x556d4bd85180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.704+0000 7f88fe000640 1 -- 172.21.15.12:0/273003944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d4bd739f0 msgr2=0x556d4bd7dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:11.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.704+0000 7f88fe000640 1 --2- 172.21.15.12:0/273003944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d4bd739f0 0x556d4bd7dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.704+0000 7f88fe000640 1 -- 172.21.15.12:0/273003944 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556d4b9ec3f0 con 0x556d4bd74bc0 2024-03-23T23:09:11.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.704+0000 7f88fea00640 1 --2- 172.21.15.12:0/273003944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d4bd739f0 0x556d4bd7dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:11.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.704+0000 7f88fe000640 1 --2- 172.21.15.12:0/273003944 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d4bd74bc0 0x556d4bd7e140 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00bc80 tx=0x7f88ec00bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9149d37ac692382 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:11.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.704+0000 7f88e8c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:09:11.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.704+0000 7f88e8c00640 1 -- 172.21.15.12:0/273003944 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec005040 con 0x556d4bd74bc0 2024-03-23T23:09:11.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.704+0000 7f88e8c00640 1 -- 172.21.15.12:0/273003944 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f88ec0051e0 con 0x556d4bd74bc0 2024-03-23T23:09:11.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.704+0000 7f8908ba8740 1 -- 172.21.15.12:0/273003944 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f88ec0041d0 con 0x556d4bd74bc0 2024-03-23T23:09:11.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.704+0000 7f88e8c00640 1 -- 172.21.15.12:0/273003944 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88ec005420 con 0x556d4bd74bc0 2024-03-23T23:09:11.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.705+0000 7f8908ba8740 10 client.27346 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:11.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.705+0000 7f8908ba8740 1 -- 172.21.15.12:0/273003944 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556d4bab67d0 con 0x556d4bd74bc0 2024-03-23T23:09:11.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.705+0000 7f88e8c00640 1 -- 172.21.15.12:0/273003944 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88ec0041d0 con 0x556d4bd74bc0 2024-03-23T23:09:11.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.705+0000 7f88e8c00640 1 -- 172.21.15.12:0/273003944 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f88ec0048f0 con 0x556d4bd74bc0 2024-03-23T23:09:11.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.705+0000 7f8908ba8740 10 client.27346 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:11.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.705+0000 7f8908ba8740 10 client.27346 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:09:11.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.705+0000 7f8908ba8740 20 client.27346 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:11.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.705+0000 7f8908ba8740 1 --2- 172.21.15.12:0/273003944 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x556d4bd89390 0x556d4bd8b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.705+0000 7f8908ba8740 4 client.27346 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:09:11.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.705+0000 7f8908ba8740 1 -- 172.21.15.12:0/273003944 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556d4bce5800 con 0x556d4bd89390 2024-03-23T23:09:11.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.705+0000 7f88fea00640 1 --2- 172.21.15.12:0/273003944 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x556d4bd89390 0x556d4bd8b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.706+0000 7f88fea00640 1 --2- 172.21.15.12:0/273003944 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x556d4bd89390 0x556d4bd8b780 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=edcd50b3cda93618 server_cookie=15380b214fe5b39a in_seq=0 out_seq=0 2024-03-23T23:09:11.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.706+0000 7f88e8c00640 10 client.27346 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:09:11.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.707+0000 7f88e8c00640 1 -- 172.21.15.12:0/273003944 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556d4bce5800 con 0x556d4bd89390 2024-03-23T23:09:11.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.707+0000 7f88e8c00640 10 client.27346 handle_command_reply: tid=0 2024-03-23T23:09:11.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.707+0000 7f8908ba8740 1 client.27346 shutdown 2024-03-23T23:09:11.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.707+0000 7f8908ba8740 1 -- 172.21.15.12:0/273003944 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d4bd74bc0 msgr2=0x556d4bd7e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.707+0000 7f8908ba8740 1 --2- 172.21.15.12:0/273003944 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d4bd74bc0 0x556d4bd7e140 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00bc80 tx=0x7f88ec00bfd0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.707+0000 7f8908ba8740 1 -- 172.21.15.12:0/273003944 shutdown_connections 2024-03-23T23:09:11.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.707+0000 7f8908ba8740 1 --2- 172.21.15.12:0/273003944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd75580 0x556d4bd85180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.707+0000 7f8908ba8740 1 --2- 172.21.15.12:0/273003944 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x556d4bd89390 0x556d4bd8b780 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.707+0000 7f8908ba8740 1 --2- 172.21.15.12:0/273003944 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d4bd74bc0 0x556d4bd7e140 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.707+0000 7f8908ba8740 1 --2- 172.21.15.12:0/273003944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d4bd739f0 0x556d4bd7dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.707+0000 7f8908ba8740 1 -- 172.21.15.12:0/273003944 >> 172.21.15.12:0/273003944 conn(0x556d4bd6ede0 msgr2=0x556d4bd6e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:11.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.707+0000 7f88e8c00640 0 client.27346 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:09:11.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.707+0000 7f8908ba8740 1 -- 172.21.15.12:0/273003944 shutdown_connections 2024-03-23T23:09:11.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.707+0000 7f8908ba8740 1 -- 172.21.15.12:0/273003944 wait complete. 2024-03-23T23:09:11.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.707+0000 7f8908ba8740 20 client.27346 trim_cache size 0 max 16384 2024-03-23T23:09:11.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.721+0000 7f8908ba8740 1 Processor -- start 2024-03-23T23:09:11.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.721+0000 7f8908ba8740 1 -- start start 2024-03-23T23:09:11.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.721+0000 7f8908ba8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d4bd739f0 0x556d4bd69ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.721+0000 7f8908ba8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d4bd74bc0 0x556d4bd85140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.721+0000 7f8908ba8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd75580 0x556d4bd77df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.721+0000 7f8908ba8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89001acf90 con 0x556d4bd74bc0 2024-03-23T23:09:11.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.721+0000 7f8908ba8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89001aaa90 con 0x556d4bd75580 2024-03-23T23:09:11.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.721+0000 7f8908ba8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89001a8b30 con 0x556d4bd739f0 2024-03-23T23:09:11.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.722+0000 7f88ff400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd75580 0x556d4bd77df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.722+0000 7f88ff400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd75580 0x556d4bd77df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:34142/0 (socket says 172.21.15.12:34142) 2024-03-23T23:09:11.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.722+0000 7f88ff400640 1 -- 172.21.15.12:0/3693315781 learned_addr learned my addr 172.21.15.12:0/3693315781 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:11.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.722+0000 7f88fea00640 1 --2- 172.21.15.12:0/3693315781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d4bd739f0 0x556d4bd69ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.722+0000 7f88fe000640 1 --2- 172.21.15.12:0/3693315781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d4bd74bc0 0x556d4bd85140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.722+0000 7f88ff400640 1 -- 172.21.15.12:0/3693315781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d4bd739f0 msgr2=0x556d4bd69ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:11.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.722+0000 7f88ff400640 1 --2- 172.21.15.12:0/3693315781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d4bd739f0 0x556d4bd69ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.722+0000 7f88ff400640 1 -- 172.21.15.12:0/3693315781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d4bd74bc0 msgr2=0x556d4bd85140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:11.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.722+0000 7f88ff400640 1 --2- 172.21.15.12:0/3693315781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d4bd74bc0 0x556d4bd85140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.722+0000 7f88ff400640 1 -- 172.21.15.12:0/3693315781 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d4bd7ebf0 con 0x556d4bd75580 2024-03-23T23:09:11.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.722+0000 7f88ff400640 1 --2- 172.21.15.12:0/3693315781 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd75580 0x556d4bd77df0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f88f8009ca0 tx=0x7f88f8041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e0d45d1da91fb5d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:11.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.722+0000 7f88eb400640 1 -- 172.21.15.12:0/3693315781 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f8039040 con 0x556d4bd75580 2024-03-23T23:09:11.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.722+0000 7f88eb400640 1 -- 172.21.15.12:0/3693315781 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88f8053070 con 0x556d4bd75580 2024-03-23T23:09:11.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.722+0000 7f88eb400640 1 -- 172.21.15.12:0/3693315781 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f80419d0 con 0x556d4bd75580 2024-03-23T23:09:11.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.723+0000 7f8908ba8740 1 -- 172.21.15.12:0/3693315781 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd75580 msgr2=0x556d4bd77df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.723+0000 7f8908ba8740 1 --2- 172.21.15.12:0/3693315781 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd75580 0x556d4bd77df0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f88f8009ca0 tx=0x7f88f8041000 comp rx=0 tx=0).stop 2024-03-23T23:09:11.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.723+0000 7f8908ba8740 1 -- 172.21.15.12:0/3693315781 shutdown_connections 2024-03-23T23:09:11.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.723+0000 7f8908ba8740 1 --2- 172.21.15.12:0/3693315781 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd75580 0x556d4bd77df0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.723+0000 7f8908ba8740 1 --2- 172.21.15.12:0/3693315781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d4bd74bc0 0x556d4bd85140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.723+0000 7f8908ba8740 1 --2- 172.21.15.12:0/3693315781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d4bd739f0 0x556d4bd69ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.723+0000 7f8908ba8740 1 -- 172.21.15.12:0/3693315781 >> 172.21.15.12:0/3693315781 conn(0x556d4bd6ede0 msgr2=0x556d4bd728d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:11.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.723+0000 7f8908ba8740 1 -- 172.21.15.12:0/3693315781 shutdown_connections 2024-03-23T23:09:11.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.723+0000 7f8908ba8740 1 -- 172.21.15.12:0/3693315781 wait complete. 2024-03-23T23:09:11.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.724+0000 7f8908ba8740 1 -- start start 2024-03-23T23:09:11.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.724+0000 7f8908ba8740 1 Processor -- start 2024-03-23T23:09:11.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.724+0000 7f8908ba8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd739f0 0x556d4bd84a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.724+0000 7f8908ba8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d4bd74bc0 0x556d4bd84f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.724+0000 7f8908ba8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d4bd75580 0x556d4bd83040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.724+0000 7f8908ba8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89001a9130 con 0x556d4bd75580 2024-03-23T23:09:11.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.724+0000 7f8908ba8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89001aa430 con 0x556d4bd739f0 2024-03-23T23:09:11.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.724+0000 7f8908ba8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89001a9770 con 0x556d4bd74bc0 2024-03-23T23:09:11.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.724+0000 7f8908ba8740 10 client.0 _ll_register_callbacks cb 0x7f88f8009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:11.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.724+0000 7f88fe000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d4bd74bc0 0x556d4bd84f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.724+0000 7f88fea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd739f0 0x556d4bd84a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.724+0000 7f88fe000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d4bd74bc0 0x556d4bd84f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54944/0 (socket says 172.21.15.12:54944) 2024-03-23T23:09:11.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.724+0000 7f88fea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd739f0 0x556d4bd84a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:34144/0 (socket says 172.21.15.12:34144) 2024-03-23T23:09:11.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.724+0000 7f88fe000640 1 -- 172.21.15.12:0/770618576 learned_addr learned my addr 172.21.15.12:0/770618576 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:11.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.725+0000 7f88ff400640 1 --2- 172.21.15.12:0/770618576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d4bd75580 0x556d4bd83040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.725+0000 7f88fe000640 1 -- 172.21.15.12:0/770618576 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd739f0 msgr2=0x556d4bd84a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:11.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.725+0000 7f88fe000640 1 --2- 172.21.15.12:0/770618576 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd739f0 0x556d4bd84a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.725+0000 7f88fe000640 1 -- 172.21.15.12:0/770618576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d4bd75580 msgr2=0x556d4bd83040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:11.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.725+0000 7f88fe000640 1 --2- 172.21.15.12:0/770618576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d4bd75580 0x556d4bd83040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.725+0000 7f88fe000640 1 -- 172.21.15.12:0/770618576 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556d4bd83780 con 0x556d4bd74bc0 2024-03-23T23:09:11.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.725+0000 7f88fea00640 1 --2- 172.21.15.12:0/770618576 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd739f0 0x556d4bd84a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:11.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.725+0000 7f88fe000640 1 --2- 172.21.15.12:0/770618576 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d4bd74bc0 0x556d4bd84f70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00b3d0 tx=0x7f88ec010ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7749829d99181ca1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:11.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.725+0000 7f88e8c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:09:11.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.725+0000 7f88e8c00640 1 -- 172.21.15.12:0/770618576 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec005000 con 0x556d4bd74bc0 2024-03-23T23:09:11.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.725+0000 7f88e8c00640 1 -- 172.21.15.12:0/770618576 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f88ec0051a0 con 0x556d4bd74bc0 2024-03-23T23:09:11.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.725+0000 7f8908ba8740 1 -- 172.21.15.12:0/770618576 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f88f8053070 con 0x556d4bd74bc0 2024-03-23T23:09:11.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.726+0000 7f88e8c00640 1 -- 172.21.15.12:0/770618576 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88ec0055c0 con 0x556d4bd74bc0 2024-03-23T23:09:11.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.726+0000 7f8908ba8740 10 client.27352 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:11.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.726+0000 7f8908ba8740 1 -- 172.21.15.12:0/770618576 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556d4bde5490 con 0x556d4bd74bc0 2024-03-23T23:09:11.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.726+0000 7f88e8c00640 1 -- 172.21.15.12:0/770618576 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88f8053070 con 0x556d4bd74bc0 2024-03-23T23:09:11.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.726+0000 7f88e8c00640 1 -- 172.21.15.12:0/770618576 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f88ec011a00 con 0x556d4bd74bc0 2024-03-23T23:09:11.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.726+0000 7f8908ba8740 10 client.27352 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:11.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.726+0000 7f8908ba8740 10 client.27352 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:09:11.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.726+0000 7f8908ba8740 20 client.27352 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:11.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.726+0000 7f8908ba8740 1 --2- 172.21.15.12:0/770618576 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x556d4bde5d90 0x556d4bde81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.726+0000 7f8908ba8740 4 client.27352 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:09:11.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.726+0000 7f8908ba8740 1 -- 172.21.15.12:0/770618576 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556d4bcdfb50 con 0x556d4bde5d90 2024-03-23T23:09:11.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.727+0000 7f88fea00640 1 --2- 172.21.15.12:0/770618576 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x556d4bde5d90 0x556d4bde81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.727+0000 7f88fea00640 1 --2- 172.21.15.12:0/770618576 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x556d4bde5d90 0x556d4bde81b0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7eea298002ee496d server_cookie=a952c2edcbae9c06 in_seq=0 out_seq=0 2024-03-23T23:09:11.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.727+0000 7f88e8c00640 10 client.27352 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:09:11.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.727+0000 7f88e8c00640 1 -- 172.21.15.12:0/770618576 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556d4bcdfb50 con 0x556d4bde5d90 2024-03-23T23:09:11.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.727+0000 7f88e8c00640 10 client.27352 handle_command_reply: tid=0 2024-03-23T23:09:11.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.727+0000 7f8908ba8740 1 client.27352 shutdown 2024-03-23T23:09:11.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.727+0000 7f8908ba8740 1 -- 172.21.15.12:0/770618576 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d4bd74bc0 msgr2=0x556d4bd84f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.727+0000 7f8908ba8740 1 --2- 172.21.15.12:0/770618576 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d4bd74bc0 0x556d4bd84f70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00b3d0 tx=0x7f88ec010ca0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.727+0000 7f8908ba8740 1 -- 172.21.15.12:0/770618576 shutdown_connections 2024-03-23T23:09:11.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.727+0000 7f8908ba8740 1 --2- 172.21.15.12:0/770618576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d4bd75580 0x556d4bd83040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.727+0000 7f8908ba8740 1 --2- 172.21.15.12:0/770618576 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x556d4bde5d90 0x556d4bde81b0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.727+0000 7f8908ba8740 1 --2- 172.21.15.12:0/770618576 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d4bd74bc0 0x556d4bd84f70 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.727+0000 7f8908ba8740 1 --2- 172.21.15.12:0/770618576 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4bd739f0 0x556d4bd84a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.727+0000 7f88e8c00640 0 client.27352 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:09:11.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.727+0000 7f8908ba8740 1 -- 172.21.15.12:0/770618576 >> 172.21.15.12:0/770618576 conn(0x556d4bd6ede0 msgr2=0x556d4bd6d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:11.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.728+0000 7f8908ba8740 1 -- 172.21.15.12:0/770618576 shutdown_connections 2024-03-23T23:09:11.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.728+0000 7f8908ba8740 1 -- 172.21.15.12:0/770618576 wait complete. 2024-03-23T23:09:11.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.728+0000 7f8908ba8740 20 client.27352 trim_cache size 0 max 16384 2024-03-23T23:09:11.732 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:09:11.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.728+0000 7f88ffe00640 1 -- 172.21.15.12:0/2708375748 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f88e40a2880 msgr2=0x7f88e40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.728+0000 7f88ffe00640 1 --2- 172.21.15.12:0/2708375748 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f88e40a2880 0x7f88e40a4d40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f88f802c640 tx=0x7f88f802e040 comp rx=0 tx=0).stop 2024-03-23T23:09:11.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.728+0000 7f88ffe00640 1 -- 172.21.15.12:0/2708375748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89001037a0 msgr2=0x7f8900199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.728+0000 7f88ffe00640 1 --2- 172.21.15.12:0/2708375748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89001037a0 0x7f8900199a10 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f88f000d770 tx=0x7f88f000dc30 comp rx=0 tx=0).stop 2024-03-23T23:09:11.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.728+0000 7f88ffe00640 1 -- 172.21.15.12:0/2708375748 shutdown_connections 2024-03-23T23:09:11.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.729+0000 7f88ffe00640 1 --2- 172.21.15.12:0/2708375748 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f88e40a2880 0x7f88e40a4d40 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.729+0000 7f88ffe00640 1 --2- 172.21.15.12:0/2708375748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8900105350 0x7f89001a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.729+0000 7f88ffe00640 1 --2- 172.21.15.12:0/2708375748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8900104990 0x7f8900199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.729+0000 7f88ffe00640 1 --2- 172.21.15.12:0/2708375748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89001037a0 0x7f8900199a10 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.729+0000 7f88ffe00640 1 -- 172.21.15.12:0/2708375748 >> 172.21.15.12:0/2708375748 conn(0x7f89000fed10 msgr2=0x7f8900107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:11.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.729+0000 7f88ffe00640 1 -- 172.21.15.12:0/2708375748 shutdown_connections 2024-03-23T23:09:11.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.729+0000 7f88ffe00640 1 -- 172.21.15.12:0/2708375748 wait complete. 2024-03-23T23:09:11.744 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:11 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3279346050' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:09:11.746 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:09:11.746 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-23T23:09:11.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:11 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3279346050' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:09:11.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.958+0000 7f5493600640 1 -- 172.21.15.12:0/431284190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f548c102140 msgr2=0x7f548c104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.958+0000 7f5493600640 1 --2- 172.21.15.12:0/431284190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f548c102140 0x7f548c104530 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f547c009ca0 tx=0x7f547c02c100 comp rx=0 tx=0).stop 2024-03-23T23:09:11.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.958+0000 7f5493600640 1 -- 172.21.15.12:0/431284190 shutdown_connections 2024-03-23T23:09:11.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.958+0000 7f5493600640 1 --2- 172.21.15.12:0/431284190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f548c1073a0 0x7f548c1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.958+0000 7f5493600640 1 --2- 172.21.15.12:0/431284190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f548c104a70 0x7f548c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.958+0000 7f5493600640 1 --2- 172.21.15.12:0/431284190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f548c102140 0x7f548c104530 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.958+0000 7f5493600640 1 -- 172.21.15.12:0/431284190 >> 172.21.15.12:0/431284190 conn(0x7f548c0fbb30 msgr2=0x7f548c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:11.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.958+0000 7f5493600640 1 -- 172.21.15.12:0/431284190 shutdown_connections 2024-03-23T23:09:11.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.958+0000 7f5493600640 1 -- 172.21.15.12:0/431284190 wait complete. 2024-03-23T23:09:11.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.960+0000 7f5493600640 1 Processor -- start 2024-03-23T23:09:11.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.960+0000 7f5493600640 1 -- start start 2024-03-23T23:09:11.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.960+0000 7f5493600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f548c102140 0x7f548c101ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.960+0000 7f5493600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f548c104a70 0x7f548c1000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.960+0000 7f5493600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f548c1073a0 0x7f548c100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.960+0000 7f5493600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c10d370 con 0x7f548c104a70 2024-03-23T23:09:11.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.960+0000 7f5493600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c10d1f0 con 0x7f548c102140 2024-03-23T23:09:11.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.960+0000 7f5493600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c10d4f0 con 0x7f548c1073a0 2024-03-23T23:09:11.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.960+0000 7f5490c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f548c102140 0x7f548c101ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.961+0000 7f5490c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f548c102140 0x7f548c101ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:34170/0 (socket says 172.21.15.12:34170) 2024-03-23T23:09:11.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.961+0000 7f5490c00640 1 -- 172.21.15.12:0/324014277 learned_addr learned my addr 172.21.15.12:0/324014277 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:11.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.961+0000 7f5491600640 1 --2- 172.21.15.12:0/324014277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f548c1073a0 0x7f548c100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.961+0000 7f548be00640 1 --2- 172.21.15.12:0/324014277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f548c104a70 0x7f548c1000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.961+0000 7f5490c00640 1 -- 172.21.15.12:0/324014277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f548c1073a0 msgr2=0x7f548c100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.961+0000 7f5490c00640 1 --2- 172.21.15.12:0/324014277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f548c1073a0 0x7f548c100630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.961+0000 7f5490c00640 1 -- 172.21.15.12:0/324014277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f548c104a70 msgr2=0x7f548c1000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.961+0000 7f5490c00640 1 --2- 172.21.15.12:0/324014277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f548c104a70 0x7f548c1000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.961+0000 7f5490c00640 1 -- 172.21.15.12:0/324014277 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548c100ec0 con 0x7f548c102140 2024-03-23T23:09:11.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.961+0000 7f5491600640 1 --2- 172.21.15.12:0/324014277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f548c1073a0 0x7f548c100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:09:11.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.961+0000 7f548be00640 1 --2- 172.21.15.12:0/324014277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f548c104a70 0x7f548c1000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:09:11.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.961+0000 7f5490c00640 1 --2- 172.21.15.12:0/324014277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f548c102140 0x7f548c101ab0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f547c02c710 tx=0x7f547c035d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:11.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.961+0000 7f5489600640 1 -- 172.21.15.12:0/324014277 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c039040 con 0x7f548c102140 2024-03-23T23:09:11.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.961+0000 7f5489600640 1 -- 172.21.15.12:0/324014277 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f547c044070 con 0x7f548c102140 2024-03-23T23:09:11.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.962+0000 7f5489600640 1 -- 172.21.15.12:0/324014277 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c004460 con 0x7f548c102140 2024-03-23T23:09:11.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.962+0000 7f5493600640 1 -- 172.21.15.12:0/324014277 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f548c1a1b10 con 0x7f548c102140 2024-03-23T23:09:11.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.962+0000 7f5493600640 1 -- 172.21.15.12:0/324014277 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f548c1a20b0 con 0x7f548c102140 2024-03-23T23:09:11.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.965+0000 7f54963a8740 1 Processor -- start 2024-03-23T23:09:11.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.965+0000 7f5489600640 1 -- 172.21.15.12:0/324014277 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f547c004600 con 0x7f548c102140 2024-03-23T23:09:11.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.965+0000 7f54963a8740 1 -- start start 2024-03-23T23:09:11.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.965+0000 7f54963a8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55902ee56810 0x55902ee56c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.965+0000 7f54963a8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee579e0 0x55902ee57e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.965+0000 7f54963a8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55902ee583a0 0x55902ee5ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.965+0000 7f54963a8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902edbb2c0 con 0x55902ee56810 2024-03-23T23:09:11.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.965+0000 7f54963a8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902ed63c90 con 0x55902ee583a0 2024-03-23T23:09:11.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.965+0000 7f54963a8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902edb7360 con 0x55902ee579e0 2024-03-23T23:09:11.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.965+0000 7f548be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55902ee583a0 0x55902ee5ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.965+0000 7f5491600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55902ee56810 0x55902ee56c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.965+0000 7f548be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55902ee583a0 0x55902ee5ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:34172/0 (socket says 172.21.15.12:34172) 2024-03-23T23:09:11.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.966+0000 7f548be00640 1 -- 172.21.15.12:0/1406181205 learned_addr learned my addr 172.21.15.12:0/1406181205 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:11.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.966+0000 7f5490c00640 1 --2- 172.21.15.12:0/1406181205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee579e0 0x55902ee57e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.966+0000 7f548be00640 1 -- 172.21.15.12:0/1406181205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee579e0 msgr2=0x55902ee57e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:11.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.966+0000 7f548be00640 1 --2- 172.21.15.12:0/1406181205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee579e0 0x55902ee57e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.966+0000 7f548be00640 1 -- 172.21.15.12:0/1406181205 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55902ee56810 msgr2=0x55902ee56c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:11.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.966+0000 7f548be00640 1 --2- 172.21.15.12:0/1406181205 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55902ee56810 0x55902ee56c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.966+0000 7f548be00640 1 -- 172.21.15.12:0/1406181205 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55902ed88080 con 0x55902ee583a0 2024-03-23T23:09:11.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.966+0000 7f548be00640 1 --2- 172.21.15.12:0/1406181205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55902ee583a0 0x55902ee5ec30 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f547400bce0 tx=0x7f547400f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e75179df8eee5884 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:11.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.966+0000 7f5481c00640 1 -- 172.21.15.12:0/1406181205 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5474015070 con 0x55902ee583a0 2024-03-23T23:09:11.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.966+0000 7f5481c00640 1 -- 172.21.15.12:0/1406181205 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f547401a940 con 0x55902ee583a0 2024-03-23T23:09:11.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.966+0000 7f5481c00640 1 -- 172.21.15.12:0/1406181205 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5474011a30 con 0x55902ee583a0 2024-03-23T23:09:11.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.967+0000 7f5489600640 1 --2- 172.21.15.12:0/324014277 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f54700a2880 0x7f54700a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.967+0000 7f5489600640 1 -- 172.21.15.12:0/324014277 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f547c0f1160 con 0x7f548c102140 2024-03-23T23:09:11.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.967+0000 7f5491600640 1 --2- 172.21.15.12:0/324014277 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f54700a2880 0x7f54700a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.967+0000 7f54963a8740 1 -- 172.21.15.12:0/1406181205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55902ee583a0 msgr2=0x55902ee5ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.967+0000 7f54963a8740 1 --2- 172.21.15.12:0/1406181205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55902ee583a0 0x55902ee5ec30 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f547400bce0 tx=0x7f547400f730 comp rx=0 tx=0).stop 2024-03-23T23:09:11.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.967+0000 7f54963a8740 1 -- 172.21.15.12:0/1406181205 shutdown_connections 2024-03-23T23:09:11.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.967+0000 7f54963a8740 1 --2- 172.21.15.12:0/1406181205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55902ee583a0 0x55902ee5ec30 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.967+0000 7f54963a8740 1 --2- 172.21.15.12:0/1406181205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee579e0 0x55902ee57e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.967+0000 7f54963a8740 1 --2- 172.21.15.12:0/1406181205 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55902ee56810 0x55902ee56c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.967+0000 7f54963a8740 1 -- 172.21.15.12:0/1406181205 >> 172.21.15.12:0/1406181205 conn(0x55902ee51c00 msgr2=0x55902ee52000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:11.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.967+0000 7f54963a8740 1 -- 172.21.15.12:0/1406181205 shutdown_connections 2024-03-23T23:09:11.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.968+0000 7f54963a8740 1 -- 172.21.15.12:0/1406181205 wait complete. 2024-03-23T23:09:11.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.968+0000 7f5491600640 1 --2- 172.21.15.12:0/324014277 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f54700a2880 0x7f54700a4d40 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x55902ee578e0 tx=0x7f5484005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:11.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.968+0000 7f54963a8740 1 -- start start 2024-03-23T23:09:11.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.968+0000 7f54963a8740 1 Processor -- start 2024-03-23T23:09:11.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.969+0000 7f54963a8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee56810 0x55902ee60a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.969+0000 7f54963a8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55902ee579e0 0x55902ee60fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.969+0000 7f54963a8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55902ee583a0 0x55902ee68000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.969+0000 7f54963a8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902ee5f3b0 con 0x55902ee579e0 2024-03-23T23:09:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.969+0000 7f54963a8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902ed770d0 con 0x55902ee583a0 2024-03-23T23:09:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.969+0000 7f54963a8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902ee5f530 con 0x55902ee56810 2024-03-23T23:09:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.969+0000 7f54963a8740 10 client.0 _ll_register_callbacks cb 0x55902ed7be80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.969+0000 7f5490c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee56810 0x55902ee60a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.969+0000 7f5490c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee56810 0x55902ee60a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54980/0 (socket says 172.21.15.12:54980) 2024-03-23T23:09:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.969+0000 7f5490c00640 1 -- 172.21.15.12:0/385232397 learned_addr learned my addr 172.21.15.12:0/385232397 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.969+0000 7f5491600640 1 --2- 172.21.15.12:0/385232397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55902ee583a0 0x55902ee68000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.969+0000 7f548be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55902ee579e0 0x55902ee60fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.969+0000 7f5490c00640 1 -- 172.21.15.12:0/385232397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55902ee583a0 msgr2=0x55902ee68000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.969+0000 7f5490c00640 1 --2- 172.21.15.12:0/385232397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55902ee583a0 0x55902ee68000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.969+0000 7f5490c00640 1 -- 172.21.15.12:0/385232397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55902ee579e0 msgr2=0x55902ee60fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.969+0000 7f5490c00640 1 --2- 172.21.15.12:0/385232397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55902ee579e0 0x55902ee60fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.969+0000 7f5490c00640 1 -- 172.21.15.12:0/385232397 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55902eacf3f0 con 0x55902ee56810 2024-03-23T23:09:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.970+0000 7f5490c00640 1 --2- 172.21.15.12:0/385232397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee56810 0x55902ee60a80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f547c0bd810 tx=0x7f547c009360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb7165a195ccc8d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.970+0000 7f547a000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:09:11.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.970+0000 7f547a000640 1 -- 172.21.15.12:0/385232397 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c0bd8e0 con 0x55902ee56810 2024-03-23T23:09:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.970+0000 7f547a000640 1 -- 172.21.15.12:0/385232397 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f547c0eecf0 con 0x55902ee56810 2024-03-23T23:09:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.970+0000 7f54963a8740 1 -- 172.21.15.12:0/385232397 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f547401a940 con 0x55902ee56810 2024-03-23T23:09:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.970+0000 7f547a000640 1 -- 172.21.15.12:0/385232397 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f547c0bd470 con 0x55902ee56810 2024-03-23T23:09:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.970+0000 7f54963a8740 10 client.27364 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.970+0000 7f54963a8740 1 -- 172.21.15.12:0/385232397 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55902eb997d0 con 0x55902ee56810 2024-03-23T23:09:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.970+0000 7f547a000640 1 -- 172.21.15.12:0/385232397 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f547401a940 con 0x55902ee56810 2024-03-23T23:09:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.970+0000 7f547a000640 1 -- 172.21.15.12:0/385232397 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f547c03c030 con 0x55902ee56810 2024-03-23T23:09:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.970+0000 7f54963a8740 10 client.27364 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.970+0000 7f54963a8740 10 client.27364 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.971+0000 7f54963a8740 20 client.27364 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.971+0000 7f54963a8740 1 --2- 172.21.15.12:0/385232397 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55902ee6c590 0x55902ee6e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.971+0000 7f54963a8740 4 client.27364 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:09:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.971+0000 7f54963a8740 1 -- 172.21.15.12:0/385232397 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55902edc8800 con 0x55902ee6c590 2024-03-23T23:09:11.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.971+0000 7f548be00640 1 --2- 172.21.15.12:0/385232397 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55902ee6c590 0x55902ee6e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.971+0000 7f548be00640 1 --2- 172.21.15.12:0/385232397 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55902ee6c590 0x55902ee6e9e0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=737872c1038bad7a server_cookie=3c86ce337a7cb6c5 in_seq=0 out_seq=0 2024-03-23T23:09:11.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.971+0000 7f547a000640 10 client.27364 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:11.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.972+0000 7f547a000640 1 -- 172.21.15.12:0/385232397 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55902edc8800 con 0x55902ee6c590 2024-03-23T23:09:11.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.972+0000 7f547a000640 10 client.27364 handle_command_reply: tid=0 2024-03-23T23:09:11.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.972+0000 7f54963a8740 1 client.27364 shutdown 2024-03-23T23:09:11.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.972+0000 7f54963a8740 1 -- 172.21.15.12:0/385232397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee56810 msgr2=0x55902ee60a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.972+0000 7f54963a8740 1 --2- 172.21.15.12:0/385232397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee56810 0x55902ee60a80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f547c0bd810 tx=0x7f547c009360 comp rx=0 tx=0).stop 2024-03-23T23:09:11.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.973+0000 7f54963a8740 1 -- 172.21.15.12:0/385232397 shutdown_connections 2024-03-23T23:09:11.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.973+0000 7f54963a8740 1 --2- 172.21.15.12:0/385232397 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55902ee6c590 0x55902ee6e9e0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.973+0000 7f54963a8740 1 --2- 172.21.15.12:0/385232397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55902ee583a0 0x55902ee68000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.973+0000 7f54963a8740 1 --2- 172.21.15.12:0/385232397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55902ee579e0 0x55902ee60fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.973+0000 7f547a000640 0 client.27364 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:11.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.973+0000 7f54963a8740 1 --2- 172.21.15.12:0/385232397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee56810 0x55902ee60a80 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.973+0000 7f54963a8740 1 -- 172.21.15.12:0/385232397 >> 172.21.15.12:0/385232397 conn(0x55902ee51c00 msgr2=0x55902ee50eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:11.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.973+0000 7f54963a8740 1 -- 172.21.15.12:0/385232397 shutdown_connections 2024-03-23T23:09:11.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.973+0000 7f54963a8740 1 -- 172.21.15.12:0/385232397 wait complete. 2024-03-23T23:09:11.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.973+0000 7f54963a8740 20 client.27364 trim_cache size 0 max 16384 2024-03-23T23:09:11.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.987+0000 7f54963a8740 1 Processor -- start 2024-03-23T23:09:11.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.987+0000 7f54963a8740 1 -- start start 2024-03-23T23:09:11.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.987+0000 7f54963a8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55902ee56810 0x55902ee55570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.987+0000 7f54963a8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55902ee579e0 0x55902ee67fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.987+0000 7f54963a8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee583a0 0x55902ee5ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.987+0000 7f54963a8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c1ad140 con 0x55902ee579e0 2024-03-23T23:09:11.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.987+0000 7f54963a8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c1aac40 con 0x55902ee56810 2024-03-23T23:09:11.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.987+0000 7f54963a8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c1a8ce0 con 0x55902ee583a0 2024-03-23T23:09:11.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.987+0000 7f5491600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee583a0 0x55902ee5ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.987+0000 7f548be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55902ee579e0 0x55902ee67fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.987+0000 7f5491600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee583a0 0x55902ee5ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54982/0 (socket says 172.21.15.12:54982) 2024-03-23T23:09:11.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.987+0000 7f5490c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55902ee56810 0x55902ee55570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.987+0000 7f5491600640 1 -- 172.21.15.12:0/4174577885 learned_addr learned my addr 172.21.15.12:0/4174577885 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:11.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.988+0000 7f5491600640 1 -- 172.21.15.12:0/4174577885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55902ee56810 msgr2=0x55902ee55570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:11.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.988+0000 7f5491600640 1 --2- 172.21.15.12:0/4174577885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55902ee56810 0x55902ee55570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.988+0000 7f5491600640 1 -- 172.21.15.12:0/4174577885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55902ee579e0 msgr2=0x55902ee67fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:11.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.988+0000 7f5491600640 1 --2- 172.21.15.12:0/4174577885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55902ee579e0 0x55902ee67fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.988+0000 7f5491600640 1 -- 172.21.15.12:0/4174577885 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55902ee57460 con 0x55902ee583a0 2024-03-23T23:09:11.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.988+0000 7f5491600640 1 --2- 172.21.15.12:0/4174577885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee583a0 0x55902ee5ac10 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f548401f740 tx=0x7f548401a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83cd14560ab23599 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:11.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.988+0000 7f5481200640 1 -- 172.21.15.12:0/4174577885 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5484011600 con 0x55902ee583a0 2024-03-23T23:09:11.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.988+0000 7f5481200640 1 -- 172.21.15.12:0/4174577885 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5484011c80 con 0x55902ee583a0 2024-03-23T23:09:11.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.988+0000 7f5481200640 1 -- 172.21.15.12:0/4174577885 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548401dda0 con 0x55902ee583a0 2024-03-23T23:09:11.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.989+0000 7f54963a8740 1 -- 172.21.15.12:0/4174577885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee583a0 msgr2=0x55902ee5ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.989+0000 7f54963a8740 1 --2- 172.21.15.12:0/4174577885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee583a0 0x55902ee5ac10 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f548401f740 tx=0x7f548401a000 comp rx=0 tx=0).stop 2024-03-23T23:09:11.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.989+0000 7f54963a8740 1 -- 172.21.15.12:0/4174577885 shutdown_connections 2024-03-23T23:09:11.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.989+0000 7f54963a8740 1 --2- 172.21.15.12:0/4174577885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee583a0 0x55902ee5ac10 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.989+0000 7f54963a8740 1 --2- 172.21.15.12:0/4174577885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55902ee579e0 0x55902ee67fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.989+0000 7f54963a8740 1 --2- 172.21.15.12:0/4174577885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55902ee56810 0x55902ee55570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.989+0000 7f54963a8740 1 -- 172.21.15.12:0/4174577885 >> 172.21.15.12:0/4174577885 conn(0x55902ee51c00 msgr2=0x55902ee54e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:11.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.989+0000 7f54963a8740 1 -- 172.21.15.12:0/4174577885 shutdown_connections 2024-03-23T23:09:11.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.989+0000 7f54963a8740 1 -- 172.21.15.12:0/4174577885 wait complete. 2024-03-23T23:09:11.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.989+0000 7f54963a8740 1 -- start start 2024-03-23T23:09:11.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.990+0000 7f54963a8740 1 Processor -- start 2024-03-23T23:09:11.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.990+0000 7f54963a8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55902ee56810 0x55902ee65f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.990+0000 7f54963a8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55902ee579e0 0x55902ee664a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.990+0000 7f54963a8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee583a0 0x55902ee674b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.990+0000 7f54963a8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c1a92e0 con 0x55902ee56810 2024-03-23T23:09:11.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.990+0000 7f54963a8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c1aa5e0 con 0x55902ee579e0 2024-03-23T23:09:11.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.990+0000 7f54963a8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c1a9920 con 0x55902ee583a0 2024-03-23T23:09:11.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.990+0000 7f54963a8740 10 client.0 _ll_register_callbacks cb 0x7f547c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:11.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.990+0000 7f5491600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee583a0 0x55902ee674b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.990+0000 7f548be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55902ee579e0 0x55902ee664a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.991+0000 7f5490c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55902ee56810 0x55902ee65f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.991+0000 7f5491600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee583a0 0x55902ee674b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54984/0 (socket says 172.21.15.12:54984) 2024-03-23T23:09:11.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.991+0000 7f5491600640 1 -- 172.21.15.12:0/3771886464 learned_addr learned my addr 172.21.15.12:0/3771886464 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:11.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.991+0000 7f548be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55902ee579e0 0x55902ee664a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:34194/0 (socket says 172.21.15.12:34194) 2024-03-23T23:09:11.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.991+0000 7f548be00640 1 -- 172.21.15.12:0/3771886464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee583a0 msgr2=0x55902ee674b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:11.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.991+0000 7f548be00640 1 --2- 172.21.15.12:0/3771886464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee583a0 0x55902ee674b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.991+0000 7f548be00640 1 -- 172.21.15.12:0/3771886464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55902ee56810 msgr2=0x55902ee65f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:11.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.991+0000 7f548be00640 1 --2- 172.21.15.12:0/3771886464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55902ee56810 0x55902ee65f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.991+0000 7f548be00640 1 -- 172.21.15.12:0/3771886464 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55902ee67aa0 con 0x55902ee579e0 2024-03-23T23:09:11.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.991+0000 7f5491600640 1 --2- 172.21.15.12:0/3771886464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee583a0 0x55902ee674b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:11.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.991+0000 7f5490c00640 1 --2- 172.21.15.12:0/3771886464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55902ee56810 0x55902ee65f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.991+0000 7f548be00640 1 --2- 172.21.15.12:0/3771886464 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55902ee579e0 0x55902ee664a0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f5474000c00 tx=0x7f5474005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36f94576729a7371 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.991+0000 7f547a000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:09:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.991+0000 7f547a000640 1 -- 172.21.15.12:0/3771886464 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5474015040 con 0x55902ee579e0 2024-03-23T23:09:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.991+0000 7f547a000640 1 -- 172.21.15.12:0/3771886464 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5474004220 con 0x55902ee579e0 2024-03-23T23:09:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.991+0000 7f54963a8740 1 -- 172.21.15.12:0/3771886464 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5484011c80 con 0x55902ee579e0 2024-03-23T23:09:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.992+0000 7f547a000640 1 -- 172.21.15.12:0/3771886464 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f547400b580 con 0x55902ee579e0 2024-03-23T23:09:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.992+0000 7f54963a8740 10 client.27398 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.992+0000 7f54963a8740 1 -- 172.21.15.12:0/3771886464 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55902eec8720 con 0x55902ee579e0 2024-03-23T23:09:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.992+0000 7f547a000640 1 -- 172.21.15.12:0/3771886464 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5484011c80 con 0x55902ee579e0 2024-03-23T23:09:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.992+0000 7f547a000640 1 -- 172.21.15.12:0/3771886464 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f5474004680 con 0x55902ee579e0 2024-03-23T23:09:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.992+0000 7f54963a8740 10 client.27398 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.992+0000 7f54963a8740 10 client.27398 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.992+0000 7f54963a8740 20 client.27398 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.992+0000 7f54963a8740 1 --2- 172.21.15.12:0/3771886464 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55902eec8d60 0x55902eecb180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.992+0000 7f54963a8740 4 client.27398 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:09:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.992+0000 7f54963a8740 1 -- 172.21.15.12:0/3771886464 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55902edc2b50 con 0x55902eec8d60 2024-03-23T23:09:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.993+0000 7f5490c00640 1 --2- 172.21.15.12:0/3771886464 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55902eec8d60 0x55902eecb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.993+0000 7f5490c00640 1 --2- 172.21.15.12:0/3771886464 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55902eec8d60 0x55902eecb180 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed547c36d5ffd27a server_cookie=2427de155a2b9974 in_seq=0 out_seq=0 2024-03-23T23:09:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.993+0000 7f547a000640 10 client.27398 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.993+0000 7f547a000640 1 -- 172.21.15.12:0/3771886464 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55902edc2b50 con 0x55902eec8d60 2024-03-23T23:09:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.993+0000 7f547a000640 10 client.27398 handle_command_reply: tid=0 2024-03-23T23:09:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.993+0000 7f54963a8740 1 client.27398 shutdown 2024-03-23T23:09:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.993+0000 7f54963a8740 1 -- 172.21.15.12:0/3771886464 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55902ee579e0 msgr2=0x55902ee664a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.993+0000 7f54963a8740 1 --2- 172.21.15.12:0/3771886464 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55902ee579e0 0x55902ee664a0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f5474000c00 tx=0x7f5474005270 comp rx=0 tx=0).stop 2024-03-23T23:09:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.993+0000 7f54963a8740 1 -- 172.21.15.12:0/3771886464 shutdown_connections 2024-03-23T23:09:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.993+0000 7f54963a8740 1 --2- 172.21.15.12:0/3771886464 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55902eec8d60 0x55902eecb180 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.993+0000 7f54963a8740 1 --2- 172.21.15.12:0/3771886464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55902ee583a0 0x55902ee674b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.993+0000 7f54963a8740 1 --2- 172.21.15.12:0/3771886464 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55902ee579e0 0x55902ee664a0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.993+0000 7f54963a8740 1 --2- 172.21.15.12:0/3771886464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55902ee56810 0x55902ee65f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.993+0000 7f54963a8740 1 -- 172.21.15.12:0/3771886464 >> 172.21.15.12:0/3771886464 conn(0x55902ee51c00 msgr2=0x55902ee556f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.993+0000 7f547a000640 0 client.27398 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.994+0000 7f54963a8740 1 -- 172.21.15.12:0/3771886464 shutdown_connections 2024-03-23T23:09:11.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.994+0000 7f54963a8740 1 -- 172.21.15.12:0/3771886464 wait complete. 2024-03-23T23:09:11.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.994+0000 7f54963a8740 20 client.27398 trim_cache size 0 max 16384 2024-03-23T23:09:11.998 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:09:11.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.994+0000 7f5493600640 1 -- 172.21.15.12:0/324014277 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f54700a2880 msgr2=0x7f54700a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.994+0000 7f5493600640 1 --2- 172.21.15.12:0/324014277 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f54700a2880 0x7f54700a4d40 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x55902ee578e0 tx=0x7f5484005f90 comp rx=0 tx=0).stop 2024-03-23T23:09:11.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.994+0000 7f5493600640 1 -- 172.21.15.12:0/324014277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f548c102140 msgr2=0x7f548c101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:11.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.994+0000 7f5493600640 1 --2- 172.21.15.12:0/324014277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f548c102140 0x7f548c101ab0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f547c02c710 tx=0x7f547c035d90 comp rx=0 tx=0).stop 2024-03-23T23:09:11.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.994+0000 7f5493600640 1 -- 172.21.15.12:0/324014277 shutdown_connections 2024-03-23T23:09:11.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.994+0000 7f5493600640 1 --2- 172.21.15.12:0/324014277 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f54700a2880 0x7f54700a4d40 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.994+0000 7f5493600640 1 --2- 172.21.15.12:0/324014277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f548c1073a0 0x7f548c100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.994+0000 7f5493600640 1 --2- 172.21.15.12:0/324014277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f548c104a70 0x7f548c1000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.994+0000 7f5493600640 1 --2- 172.21.15.12:0/324014277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f548c102140 0x7f548c101ab0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:11.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.994+0000 7f5493600640 1 -- 172.21.15.12:0/324014277 >> 172.21.15.12:0/324014277 conn(0x7f548c0fbb30 msgr2=0x7f548c102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:11.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.995+0000 7f5493600640 1 -- 172.21.15.12:0/324014277 shutdown_connections 2024-03-23T23:09:11.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:11.995+0000 7f5493600640 1 -- 172.21.15.12:0/324014277 wait complete. 2024-03-23T23:09:12.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:11 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3279346050' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:09:12.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:12 smithi203 ceph-mon[36693]: pgmap v612: 97 pgs: 97 active+clean; 820 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-03-23T23:09:13.012 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-23T23:09:13.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:09:13.013 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-23T23:09:13.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:12 smithi012 ceph-mon[31695]: pgmap v612: 97 pgs: 97 active+clean; 820 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-03-23T23:09:13.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:12 smithi173 ceph-mon[36116]: pgmap v612: 97 pgs: 97 active+clean; 820 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-03-23T23:09:13.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.251+0000 7f34fb800640 1 -- 172.21.15.12:0/2948987883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34f4104ad0 msgr2=0x7f34f4104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:13.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.251+0000 7f34fb800640 1 --2- 172.21.15.12:0/2948987883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34f4104ad0 0x7f34f4104f50 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f34e0009ca0 tx=0x7f34e002c100 comp rx=0 tx=0).stop 2024-03-23T23:09:13.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.251+0000 7f34fb800640 1 -- 172.21.15.12:0/2948987883 shutdown_connections 2024-03-23T23:09:13.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.251+0000 7f34fb800640 1 --2- 172.21.15.12:0/2948987883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34f4105490 0x7f34f410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.251+0000 7f34fb800640 1 --2- 172.21.15.12:0/2948987883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34f4104ad0 0x7f34f4104f50 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.251+0000 7f34fb800640 1 --2- 172.21.15.12:0/2948987883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34f41038e0 0x7f34f4103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.251+0000 7f34fb800640 1 -- 172.21.15.12:0/2948987883 >> 172.21.15.12:0/2948987883 conn(0x7f34f40fee50 msgr2=0x7f34f4101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:13.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.251+0000 7f34fb800640 1 -- 172.21.15.12:0/2948987883 shutdown_connections 2024-03-23T23:09:13.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.251+0000 7f34fb800640 1 -- 172.21.15.12:0/2948987883 wait complete. 2024-03-23T23:09:13.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.253+0000 7f34fb800640 1 Processor -- start 2024-03-23T23:09:13.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.253+0000 7f34fb800640 1 -- start start 2024-03-23T23:09:13.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.253+0000 7f34fb800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34f41038e0 0x7f34f4199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.253+0000 7f34fb800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34f4104ad0 0x7f34f419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.253+0000 7f34fb800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34f4105490 0x7f34f41a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.253+0000 7f34fb800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f410f560 con 0x7f34f4104ad0 2024-03-23T23:09:13.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.253+0000 7f34fb800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f410f3e0 con 0x7f34f41038e0 2024-03-23T23:09:13.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.253+0000 7f34fb800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f410f6e0 con 0x7f34f4105490 2024-03-23T23:09:13.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.253+0000 7f34f8e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34f41038e0 0x7f34f4199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.253+0000 7f34f9800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34f4105490 0x7f34f41a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.253+0000 7f34f8e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34f41038e0 0x7f34f4199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:34210/0 (socket says 172.21.15.12:34210) 2024-03-23T23:09:13.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.253+0000 7f34f8e00640 1 -- 172.21.15.12:0/4235381729 learned_addr learned my addr 172.21.15.12:0/4235381729 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:13.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.253+0000 7f34f9800640 1 --2- 172.21.15.12:0/4235381729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34f4105490 0x7f34f41a1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54992/0 (socket says 172.21.15.12:54992) 2024-03-23T23:09:13.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.254+0000 7f34f3e00640 1 --2- 172.21.15.12:0/4235381729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34f4104ad0 0x7f34f419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.254+0000 7f34f9800640 1 -- 172.21.15.12:0/4235381729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34f41038e0 msgr2=0x7f34f4199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:13.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.254+0000 7f34f9800640 1 --2- 172.21.15.12:0/4235381729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34f41038e0 0x7f34f4199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.254+0000 7f34f9800640 1 -- 172.21.15.12:0/4235381729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34f4104ad0 msgr2=0x7f34f419a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:13.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.254+0000 7f34f9800640 1 --2- 172.21.15.12:0/4235381729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34f4104ad0 0x7f34f419a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.254+0000 7f34f9800640 1 -- 172.21.15.12:0/4235381729 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f41a1830 con 0x7f34f4105490 2024-03-23T23:09:13.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.254+0000 7f34f9800640 1 --2- 172.21.15.12:0/4235381729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34f4105490 0x7f34f41a1190 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f34ec00e970 tx=0x7f34ec00ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:13.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.254+0000 7f34f1600640 1 -- 172.21.15.12:0/4235381729 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec00ccc0 con 0x7f34f4105490 2024-03-23T23:09:13.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.254+0000 7f34f1600640 1 -- 172.21.15.12:0/4235381729 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f34ec005020 con 0x7f34f4105490 2024-03-23T23:09:13.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.254+0000 7f34f1600640 1 -- 172.21.15.12:0/4235381729 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec004a00 con 0x7f34f4105490 2024-03-23T23:09:13.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.254+0000 7f34fb800640 1 -- 172.21.15.12:0/4235381729 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f41a1b20 con 0x7f34f4105490 2024-03-23T23:09:13.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.255+0000 7f34fb800640 1 -- 172.21.15.12:0/4235381729 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f41a2020 con 0x7f34f4105490 2024-03-23T23:09:13.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.258+0000 7f34f1600640 1 -- 172.21.15.12:0/4235381729 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f34ec0051d0 con 0x7f34f4105490 2024-03-23T23:09:13.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.258+0000 7f34fe5fb740 1 Processor -- start 2024-03-23T23:09:13.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.258+0000 7f34fe5fb740 1 -- start start 2024-03-23T23:09:13.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.258+0000 7f34fe5fb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d9a0d60a0 0x561d9a0d64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.258+0000 7f34fe5fb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d9a0d6a90 0x561d9a0db590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.258+0000 7f34fe5fb740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9a0dbc00 0x561d9a0de090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.258+0000 7f34fe5fb740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d9a03beb0 con 0x561d9a0dbc00 2024-03-23T23:09:13.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.258+0000 7f34fe5fb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d99fe4a60 con 0x561d9a0d60a0 2024-03-23T23:09:13.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.258+0000 7f34fe5fb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d9a037f50 con 0x561d9a0d6a90 2024-03-23T23:09:13.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.259+0000 7f34f8e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d9a0d6a90 0x561d9a0db590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.259+0000 7f34f8e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d9a0d6a90 0x561d9a0db590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55006/0 (socket says 172.21.15.12:55006) 2024-03-23T23:09:13.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.259+0000 7f34f8e00640 1 -- 172.21.15.12:0/1830391611 learned_addr learned my addr 172.21.15.12:0/1830391611 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:13.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.259+0000 7f34f3e00640 1 --2- 172.21.15.12:0/1830391611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d9a0d60a0 0x561d9a0d64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.259+0000 7f34f8e00640 1 -- 172.21.15.12:0/1830391611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d9a0d60a0 msgr2=0x561d9a0d64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:13.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.259+0000 7f34f8e00640 1 --2- 172.21.15.12:0/1830391611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d9a0d60a0 0x561d9a0d64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.259+0000 7f34f3e00640 1 --2- 172.21.15.12:0/1830391611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9a0dbc00 0x561d9a0de090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.259+0000 7f34f8e00640 1 -- 172.21.15.12:0/1830391611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9a0dbc00 msgr2=0x561d9a0de090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:13.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.259+0000 7f34f8e00640 1 --2- 172.21.15.12:0/1830391611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9a0dbc00 0x561d9a0de090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.259+0000 7f34f1600640 1 --2- 172.21.15.12:0/4235381729 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f34d80a2800 0x7f34d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.259+0000 7f34f8e00640 1 -- 172.21.15.12:0/1830391611 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d99e187e0 con 0x561d9a0d6a90 2024-03-23T23:09:13.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.259+0000 7f34f3e00640 1 --2- 172.21.15.12:0/1830391611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9a0dbc00 0x561d9a0de090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:09:13.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.259+0000 7f34f1600640 1 -- 172.21.15.12:0/4235381729 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f34ec0cea10 con 0x7f34f4105490 2024-03-23T23:09:13.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.259+0000 7f34f8e00640 1 --2- 172.21.15.12:0/1830391611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d9a0d6a90 0x561d9a0db590 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f34e400be20 tx=0x7f34e40107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=313084e80c09d8d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:13.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.260+0000 7f34f3e00640 1 --2- 172.21.15.12:0/4235381729 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f34d80a2800 0x7f34d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.260+0000 7f34ea000640 1 -- 172.21.15.12:0/1830391611 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e4011640 con 0x561d9a0d6a90 2024-03-23T23:09:13.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.260+0000 7f34ea000640 1 -- 172.21.15.12:0/1830391611 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f34e4011cc0 con 0x561d9a0d6a90 2024-03-23T23:09:13.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.260+0000 7f34ea000640 1 -- 172.21.15.12:0/1830391611 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e4019dc0 con 0x561d9a0d6a90 2024-03-23T23:09:13.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.261+0000 7f34fe5fb740 1 -- 172.21.15.12:0/1830391611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d9a0d6a90 msgr2=0x561d9a0db590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:13.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.261+0000 7f34fe5fb740 1 --2- 172.21.15.12:0/1830391611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d9a0d6a90 0x561d9a0db590 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f34e400be20 tx=0x7f34e40107a0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.261+0000 7f34f3e00640 1 --2- 172.21.15.12:0/4235381729 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f34d80a2800 0x7f34d80a4cc0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f34e002c710 tx=0x7f34e002c100 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:13.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.261+0000 7f34fe5fb740 1 -- 172.21.15.12:0/1830391611 shutdown_connections 2024-03-23T23:09:13.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.261+0000 7f34fe5fb740 1 --2- 172.21.15.12:0/1830391611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9a0dbc00 0x561d9a0de090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.261+0000 7f34fe5fb740 1 --2- 172.21.15.12:0/1830391611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d9a0d6a90 0x561d9a0db590 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.261+0000 7f34fe5fb740 1 --2- 172.21.15.12:0/1830391611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d9a0d60a0 0x561d9a0d64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.261+0000 7f34fe5fb740 1 -- 172.21.15.12:0/1830391611 >> 172.21.15.12:0/1830391611 conn(0x561d9a0d1610 msgr2=0x561d9a0d1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:13.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.261+0000 7f34fe5fb740 1 -- 172.21.15.12:0/1830391611 shutdown_connections 2024-03-23T23:09:13.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.261+0000 7f34fe5fb740 1 -- 172.21.15.12:0/1830391611 wait complete. 2024-03-23T23:09:13.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.261+0000 7f34fe5fb740 1 -- start start 2024-03-23T23:09:13.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.261+0000 7f34fe5fb740 1 Processor -- start 2024-03-23T23:09:13.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.262+0000 7f34fe5fb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d9a0d60a0 0x561d9a0e0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.262+0000 7f34fe5fb740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9a0d6a90 0x561d9a0e07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.262+0000 7f34fe5fb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d9a0dbc00 0x561d9a0e7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.262+0000 7f34fe5fb740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d9a0debb0 con 0x561d9a0d6a90 2024-03-23T23:09:13.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.262+0000 7f34fe5fb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d99ff7eb0 con 0x561d9a0d60a0 2024-03-23T23:09:13.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.262+0000 7f34fe5fb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d9a0ded30 con 0x561d9a0dbc00 2024-03-23T23:09:13.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.262+0000 7f34fe5fb740 10 client.0 _ll_register_callbacks cb 0x561d99fec150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:13.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.262+0000 7f34f3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9a0d6a90 0x561d9a0e07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.262+0000 7f34f8e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d9a0d60a0 0x561d9a0e0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.262+0000 7f34f8e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d9a0d60a0 0x561d9a0e0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:34234/0 (socket says 172.21.15.12:34234) 2024-03-23T23:09:13.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.262+0000 7f34f3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9a0d6a90 0x561d9a0e07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39488/0 (socket says 172.21.15.12:39488) 2024-03-23T23:09:13.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.262+0000 7f34f9800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d9a0dbc00 0x561d9a0e7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.262+0000 7f34f8e00640 1 -- 172.21.15.12:0/3138612388 learned_addr learned my addr 172.21.15.12:0/3138612388 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:13.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.263+0000 7f34f9800640 1 -- 172.21.15.12:0/3138612388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d9a0d60a0 msgr2=0x561d9a0e0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:13.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.263+0000 7f34f9800640 1 --2- 172.21.15.12:0/3138612388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d9a0d60a0 0x561d9a0e0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.263+0000 7f34f9800640 1 -- 172.21.15.12:0/3138612388 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9a0d6a90 msgr2=0x561d9a0e07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.263+0000 7f34f9800640 1 --2- 172.21.15.12:0/3138612388 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9a0d6a90 0x561d9a0e07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.263+0000 7f34f9800640 1 -- 172.21.15.12:0/3138612388 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561d99e78f70 con 0x561d9a0dbc00 2024-03-23T23:09:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.263+0000 7f34f9800640 1 --2- 172.21.15.12:0/3138612388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d9a0dbc00 0x561d9a0e7800 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f34ec0cc4c0 tx=0x7f34ec005560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83ae31c533d4848f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.263+0000 7f34dea00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:09:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.263+0000 7f34dea00640 1 -- 172.21.15.12:0/3138612388 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec0b4060 con 0x561d9a0dbc00 2024-03-23T23:09:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.263+0000 7f34dea00640 1 -- 172.21.15.12:0/3138612388 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f34ec0a3670 con 0x561d9a0dbc00 2024-03-23T23:09:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.263+0000 7f34fe5fb740 1 -- 172.21.15.12:0/3138612388 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f34e4011cc0 con 0x561d9a0dbc00 2024-03-23T23:09:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.263+0000 7f34dea00640 1 -- 172.21.15.12:0/3138612388 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f34ec0b9e60 con 0x561d9a0dbc00 2024-03-23T23:09:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.263+0000 7f34fe5fb740 10 client.27382 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.263+0000 7f34fe5fb740 1 -- 172.21.15.12:0/3138612388 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561d99d10170 con 0x561d9a0dbc00 2024-03-23T23:09:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.264+0000 7f34dea00640 1 -- 172.21.15.12:0/3138612388 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f34e4011cc0 con 0x561d9a0dbc00 2024-03-23T23:09:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.264+0000 7f34dea00640 1 -- 172.21.15.12:0/3138612388 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f34ec004ba0 con 0x561d9a0dbc00 2024-03-23T23:09:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.264+0000 7f34fe5fb740 10 client.27382 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.264+0000 7f34fe5fb740 10 client.27382 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.264+0000 7f34fe5fb740 20 client.27382 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:13.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.264+0000 7f34fe5fb740 1 --2- 172.21.15.12:0/3138612388 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561d9a0ebfc0 0x561d9a0ee410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.264+0000 7f34fe5fb740 4 client.27382 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:09:13.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.264+0000 7f34fe5fb740 1 -- 172.21.15.12:0/3138612388 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561d9a0493f0 con 0x561d9a0ebfc0 2024-03-23T23:09:13.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.264+0000 7f34f8e00640 1 --2- 172.21.15.12:0/3138612388 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561d9a0ebfc0 0x561d9a0ee410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.265+0000 7f34f8e00640 1 --2- 172.21.15.12:0/3138612388 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561d9a0ebfc0 0x561d9a0ee410 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fe9c173f7920292 server_cookie=8c0dd1048df61401 in_seq=0 out_seq=0 2024-03-23T23:09:13.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.265+0000 7f34dea00640 10 client.27382 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:13.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.265+0000 7f34dea00640 1 -- 172.21.15.12:0/3138612388 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561d9a0493f0 con 0x561d9a0ebfc0 2024-03-23T23:09:13.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.265+0000 7f34dea00640 10 client.27382 handle_command_reply: tid=0 2024-03-23T23:09:13.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.266+0000 7f34fe5fb740 1 client.27382 shutdown 2024-03-23T23:09:13.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.266+0000 7f34fe5fb740 1 -- 172.21.15.12:0/3138612388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d9a0dbc00 msgr2=0x561d9a0e7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:13.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.266+0000 7f34fe5fb740 1 --2- 172.21.15.12:0/3138612388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d9a0dbc00 0x561d9a0e7800 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f34ec0cc4c0 tx=0x7f34ec005560 comp rx=0 tx=0).stop 2024-03-23T23:09:13.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.266+0000 7f34fe5fb740 1 -- 172.21.15.12:0/3138612388 shutdown_connections 2024-03-23T23:09:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.266+0000 7f34fe5fb740 1 --2- 172.21.15.12:0/3138612388 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561d9a0ebfc0 0x561d9a0ee410 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.266+0000 7f34fe5fb740 1 --2- 172.21.15.12:0/3138612388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d9a0dbc00 0x561d9a0e7800 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.266+0000 7f34fe5fb740 1 --2- 172.21.15.12:0/3138612388 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9a0d6a90 0x561d9a0e07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.266+0000 7f34dea00640 0 client.27382 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.266+0000 7f34fe5fb740 1 --2- 172.21.15.12:0/3138612388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d9a0d60a0 0x561d9a0e0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.266+0000 7f34fe5fb740 1 -- 172.21.15.12:0/3138612388 >> 172.21.15.12:0/3138612388 conn(0x561d9a0d1610 msgr2=0x561d9a0d0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.266+0000 7f34fe5fb740 1 -- 172.21.15.12:0/3138612388 shutdown_connections 2024-03-23T23:09:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.266+0000 7f34fe5fb740 1 -- 172.21.15.12:0/3138612388 wait complete. 2024-03-23T23:09:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.266+0000 7f34fe5fb740 20 client.27382 trim_cache size 0 max 16384 2024-03-23T23:09:13.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.284+0000 7f34fe5fb740 1 Processor -- start 2024-03-23T23:09:13.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.284+0000 7f34fe5fb740 1 -- start start 2024-03-23T23:09:13.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.285+0000 7f34fe5fb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d9a0d60a0 0x561d9a0d7500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.285+0000 7f34fe5fb740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9a0d6a90 0x561d9a0e77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.285+0000 7f34fe5fb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d9a0dbc00 0x561d9a0d7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.285+0000 7f34fe5fb740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f41ad180 con 0x561d9a0d6a90 2024-03-23T23:09:13.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.285+0000 7f34fe5fb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f41aac80 con 0x561d9a0dbc00 2024-03-23T23:09:13.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.285+0000 7f34fe5fb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f41a8d20 con 0x561d9a0d60a0 2024-03-23T23:09:13.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.285+0000 7f34f8e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d9a0d60a0 0x561d9a0d7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.285+0000 7f34f8e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d9a0d60a0 0x561d9a0d7500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55022/0 (socket says 172.21.15.12:55022) 2024-03-23T23:09:13.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.285+0000 7f34f8e00640 1 -- 172.21.15.12:0/791141783 learned_addr learned my addr 172.21.15.12:0/791141783 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:13.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.285+0000 7f34f9800640 1 --2- 172.21.15.12:0/791141783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d9a0dbc00 0x561d9a0d7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.285+0000 7f34f3e00640 1 --2- 172.21.15.12:0/791141783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9a0d6a90 0x561d9a0e77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.285+0000 7f34f8e00640 1 -- 172.21.15.12:0/791141783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d9a0dbc00 msgr2=0x561d9a0d7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.285+0000 7f34f8e00640 1 --2- 172.21.15.12:0/791141783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d9a0dbc00 0x561d9a0d7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.285+0000 7f34f8e00640 1 -- 172.21.15.12:0/791141783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9a0d6a90 msgr2=0x561d9a0e77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.285+0000 7f34f8e00640 1 --2- 172.21.15.12:0/791141783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9a0d6a90 0x561d9a0e77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.285+0000 7f34f8e00640 1 -- 172.21.15.12:0/791141783 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d9a0ccc20 con 0x561d9a0d60a0 2024-03-23T23:09:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.285+0000 7f34f3e00640 1 --2- 172.21.15.12:0/791141783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9a0d6a90 0x561d9a0e77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:09:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.285+0000 7f34f9800640 1 --2- 172.21.15.12:0/791141783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d9a0dbc00 0x561d9a0d7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.286+0000 7f34f8e00640 1 --2- 172.21.15.12:0/791141783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d9a0d60a0 0x561d9a0d7500 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f34e400bd40 tx=0x7f34e400b6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4a62e1d81a5d7a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.286+0000 7f34e9600640 1 -- 172.21.15.12:0/791141783 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e402b070 con 0x561d9a0d60a0 2024-03-23T23:09:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.286+0000 7f34e9600640 1 -- 172.21.15.12:0/791141783 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f34e402c260 con 0x561d9a0d60a0 2024-03-23T23:09:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.286+0000 7f34e9600640 1 -- 172.21.15.12:0/791141783 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e4019a70 con 0x561d9a0d60a0 2024-03-23T23:09:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.287+0000 7f34fe5fb740 1 -- 172.21.15.12:0/791141783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d9a0d60a0 msgr2=0x561d9a0d7500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.287+0000 7f34fe5fb740 1 --2- 172.21.15.12:0/791141783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d9a0d60a0 0x561d9a0d7500 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f34e400bd40 tx=0x7f34e400b6c0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.287+0000 7f34fe5fb740 1 -- 172.21.15.12:0/791141783 shutdown_connections 2024-03-23T23:09:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.287+0000 7f34fe5fb740 1 --2- 172.21.15.12:0/791141783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d9a0dbc00 0x561d9a0d7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.287+0000 7f34fe5fb740 1 --2- 172.21.15.12:0/791141783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9a0d6a90 0x561d9a0e77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.287+0000 7f34fe5fb740 1 --2- 172.21.15.12:0/791141783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d9a0d60a0 0x561d9a0d7500 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.287+0000 7f34fe5fb740 1 -- 172.21.15.12:0/791141783 >> 172.21.15.12:0/791141783 conn(0x561d9a0d1610 msgr2=0x561d9a0d5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.287+0000 7f34fe5fb740 1 -- 172.21.15.12:0/791141783 shutdown_connections 2024-03-23T23:09:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.287+0000 7f34fe5fb740 1 -- 172.21.15.12:0/791141783 wait complete. 2024-03-23T23:09:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.287+0000 7f34fe5fb740 1 -- start start 2024-03-23T23:09:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.287+0000 7f34fe5fb740 1 Processor -- start 2024-03-23T23:09:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.288+0000 7f34fe5fb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d9a0d60a0 0x561d9a0e7410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.288+0000 7f34fe5fb740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9a0d6a90 0x561d9a0e5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.288+0000 7f34fe5fb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d9a0dbc00 0x561d9a0e6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.288+0000 7f34fe5fb740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f41a9320 con 0x561d9a0d6a90 2024-03-23T23:09:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.288+0000 7f34fe5fb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f41aa620 con 0x561d9a0d60a0 2024-03-23T23:09:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.288+0000 7f34fe5fb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f41a9960 con 0x561d9a0dbc00 2024-03-23T23:09:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.288+0000 7f34fe5fb740 10 client.0 _ll_register_callbacks cb 0x7f34ec0046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.288+0000 7f34f8e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d9a0d60a0 0x561d9a0e7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.288+0000 7f34f3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9a0d6a90 0x561d9a0e5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.288+0000 7f34f9800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d9a0dbc00 0x561d9a0e6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.288+0000 7f34f8e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d9a0d60a0 0x561d9a0e7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:34250/0 (socket says 172.21.15.12:34250) 2024-03-23T23:09:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.288+0000 7f34f8e00640 1 -- 172.21.15.12:0/4067021878 learned_addr learned my addr 172.21.15.12:0/4067021878 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.288+0000 7f34f8e00640 1 -- 172.21.15.12:0/4067021878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d9a0dbc00 msgr2=0x561d9a0e6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.288+0000 7f34f8e00640 1 --2- 172.21.15.12:0/4067021878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d9a0dbc00 0x561d9a0e6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.288+0000 7f34f8e00640 1 -- 172.21.15.12:0/4067021878 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9a0d6a90 msgr2=0x561d9a0e5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.289+0000 7f34f8e00640 1 --2- 172.21.15.12:0/4067021878 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9a0d6a90 0x561d9a0e5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.289+0000 7f34f8e00640 1 -- 172.21.15.12:0/4067021878 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561d9a0e0b30 con 0x561d9a0d60a0 2024-03-23T23:09:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.289+0000 7f34f8e00640 1 --2- 172.21.15.12:0/4067021878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d9a0d60a0 0x561d9a0e7410 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f34e400b990 tx=0x7f34e4019570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40b9a5a1356c8c87 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.289+0000 7f34dea00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:09:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.289+0000 7f34dea00640 1 -- 172.21.15.12:0/4067021878 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e403b060 con 0x561d9a0d60a0 2024-03-23T23:09:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.289+0000 7f34dea00640 1 -- 172.21.15.12:0/4067021878 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f34e402c960 con 0x561d9a0d60a0 2024-03-23T23:09:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.289+0000 7f34fe5fb740 1 -- 172.21.15.12:0/4067021878 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f34e402c260 con 0x561d9a0d60a0 2024-03-23T23:09:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.289+0000 7f34dea00640 1 -- 172.21.15.12:0/4067021878 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f34e402b040 con 0x561d9a0d60a0 2024-03-23T23:09:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.290+0000 7f34fe5fb740 10 client.27416 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.290+0000 7f34fe5fb740 1 -- 172.21.15.12:0/4067021878 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561d9a1484c0 con 0x561d9a0d60a0 2024-03-23T23:09:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.290+0000 7f34dea00640 1 -- 172.21.15.12:0/4067021878 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f34e402c260 con 0x561d9a0d60a0 2024-03-23T23:09:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.290+0000 7f34dea00640 1 -- 172.21.15.12:0/4067021878 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f34e40199c0 con 0x561d9a0d60a0 2024-03-23T23:09:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.290+0000 7f34fe5fb740 10 client.27416 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.290+0000 7f34fe5fb740 10 client.27416 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.290+0000 7f34fe5fb740 20 client.27416 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.290+0000 7f34fe5fb740 1 --2- 172.21.15.12:0/4067021878 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561d9a148c20 0x561d9a14b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.290+0000 7f34fe5fb740 4 client.27416 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-23T23:09:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.290+0000 7f34fe5fb740 1 -- 172.21.15.12:0/4067021878 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x561d9a043740 con 0x561d9a148c20 2024-03-23T23:09:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.291+0000 7f34f3e00640 1 --2- 172.21.15.12:0/4067021878 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561d9a148c20 0x561d9a14b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.291+0000 7f34f3e00640 1 --2- 172.21.15.12:0/4067021878 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561d9a148c20 0x561d9a14b040 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8d50a64f1ed53c server_cookie=16b2e56f1b1e7805 in_seq=0 out_seq=0 2024-03-23T23:09:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.291+0000 7f34dea00640 10 client.27416 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.292+0000 7f34dea00640 1 -- 172.21.15.12:0/4067021878 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x561d9a043740 con 0x561d9a148c20 2024-03-23T23:09:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.292+0000 7f34dea00640 10 client.27416 handle_command_reply: tid=0 2024-03-23T23:09:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.292+0000 7f34fe5fb740 1 client.27416 shutdown 2024-03-23T23:09:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.292+0000 7f34fe5fb740 1 -- 172.21.15.12:0/4067021878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d9a0d60a0 msgr2=0x561d9a0e7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.292+0000 7f34fe5fb740 1 --2- 172.21.15.12:0/4067021878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d9a0d60a0 0x561d9a0e7410 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f34e400b990 tx=0x7f34e4019570 comp rx=0 tx=0).stop 2024-03-23T23:09:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.292+0000 7f34fe5fb740 1 -- 172.21.15.12:0/4067021878 shutdown_connections 2024-03-23T23:09:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.292+0000 7f34fe5fb740 1 --2- 172.21.15.12:0/4067021878 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561d9a148c20 0x561d9a14b040 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.292+0000 7f34fe5fb740 1 --2- 172.21.15.12:0/4067021878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d9a0dbc00 0x561d9a0e6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.292+0000 7f34fe5fb740 1 --2- 172.21.15.12:0/4067021878 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9a0d6a90 0x561d9a0e5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.292+0000 7f34fe5fb740 1 --2- 172.21.15.12:0/4067021878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d9a0d60a0 0x561d9a0e7410 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.292+0000 7f34dea00640 0 client.27416 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.292+0000 7f34fe5fb740 1 -- 172.21.15.12:0/4067021878 >> 172.21.15.12:0/4067021878 conn(0x561d9a0d1610 msgr2=0x561d9a0cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.293+0000 7f34fe5fb740 1 -- 172.21.15.12:0/4067021878 shutdown_connections 2024-03-23T23:09:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.293+0000 7f34fe5fb740 1 -- 172.21.15.12:0/4067021878 wait complete. 2024-03-23T23:09:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.293+0000 7f34fe5fb740 20 client.27416 trim_cache size 0 max 16384 2024-03-23T23:09:13.296 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:09:13.296 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-03-23T23:09:13.296 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "9b8749fd-9c7b-4f36-99f8-e96645293954", 2024-03-23T23:09:13.296 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-03-23T23:09:13.296 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:09:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.293+0000 7f34fb800640 1 -- 172.21.15.12:0/4235381729 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f34d80a2800 msgr2=0x7f34d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.293+0000 7f34fb800640 1 --2- 172.21.15.12:0/4235381729 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f34d80a2800 0x7f34d80a4cc0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f34e002c710 tx=0x7f34e002c100 comp rx=0 tx=0).stop 2024-03-23T23:09:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.293+0000 7f34fb800640 1 -- 172.21.15.12:0/4235381729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34f4105490 msgr2=0x7f34f41a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.293+0000 7f34fb800640 1 --2- 172.21.15.12:0/4235381729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34f4105490 0x7f34f41a1190 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f34ec00e970 tx=0x7f34ec00ee30 comp rx=0 tx=0).stop 2024-03-23T23:09:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.294+0000 7f34fb800640 1 -- 172.21.15.12:0/4235381729 shutdown_connections 2024-03-23T23:09:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.294+0000 7f34fb800640 1 --2- 172.21.15.12:0/4235381729 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f34d80a2800 0x7f34d80a4cc0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.294+0000 7f34fb800640 1 --2- 172.21.15.12:0/4235381729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34f4105490 0x7f34f41a1190 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.294+0000 7f34fb800640 1 --2- 172.21.15.12:0/4235381729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34f4104ad0 0x7f34f419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.294+0000 7f34fb800640 1 --2- 172.21.15.12:0/4235381729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34f41038e0 0x7f34f4199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.294+0000 7f34fb800640 1 -- 172.21.15.12:0/4235381729 >> 172.21.15.12:0/4235381729 conn(0x7f34f40fee50 msgr2=0x7f34f4107e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.294+0000 7f34fb800640 1 -- 172.21.15.12:0/4235381729 shutdown_connections 2024-03-23T23:09:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.294+0000 7f34fb800640 1 -- 172.21.15.12:0/4235381729 wait complete. 2024-03-23T23:09:13.313 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:09:13.313 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:09:13.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.580+0000 7f7fdca00640 1 -- 172.21.15.12:0/1746537732 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fd81038c0 msgr2=0x7f7fd8103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:13.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.580+0000 7f7fdca00640 1 --2- 172.21.15.12:0/1746537732 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fd81038c0 0x7f7fd8103ce0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8009900 tx=0x7f7fc802bc40 comp rx=0 tx=0).stop 2024-03-23T23:09:13.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.580+0000 7f7fdca00640 1 -- 172.21.15.12:0/1746537732 shutdown_connections 2024-03-23T23:09:13.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.580+0000 7f7fdca00640 1 --2- 172.21.15.12:0/1746537732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fd8105470 0x7f7fd810bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.580+0000 7f7fdca00640 1 --2- 172.21.15.12:0/1746537732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fd8104ab0 0x7f7fd8104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.580+0000 7f7fdca00640 1 --2- 172.21.15.12:0/1746537732 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fd81038c0 0x7f7fd8103ce0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.580+0000 7f7fdca00640 1 -- 172.21.15.12:0/1746537732 >> 172.21.15.12:0/1746537732 conn(0x7f7fd80fee50 msgr2=0x7f7fd8101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:13.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.580+0000 7f7fdca00640 1 -- 172.21.15.12:0/1746537732 shutdown_connections 2024-03-23T23:09:13.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.580+0000 7f7fdca00640 1 -- 172.21.15.12:0/1746537732 wait complete. 2024-03-23T23:09:13.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.581+0000 7f7fdca00640 1 Processor -- start 2024-03-23T23:09:13.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.581+0000 7f7fdca00640 1 -- start start 2024-03-23T23:09:13.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.582+0000 7f7fdca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fd81038c0 0x7f7fd8199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.582+0000 7f7fdca00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fd8104ab0 0x7f7fd819a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.582+0000 7f7fdca00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fd8105470 0x7f7fd81a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.582+0000 7f7fdca00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd810f560 con 0x7f7fd8105470 2024-03-23T23:09:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.582+0000 7f7fdca00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd810f3e0 con 0x7f7fd81038c0 2024-03-23T23:09:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.582+0000 7f7fdca00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd810f6e0 con 0x7f7fd8104ab0 2024-03-23T23:09:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.582+0000 7f7fd5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fd81038c0 0x7f7fd8199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.582+0000 7f7fd5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fd81038c0 0x7f7fd8199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:34266/0 (socket says 172.21.15.12:34266) 2024-03-23T23:09:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.582+0000 7f7fd5e00640 1 -- 172.21.15.12:0/3269000587 learned_addr learned my addr 172.21.15.12:0/3269000587 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.582+0000 7f7fd5400640 1 --2- 172.21.15.12:0/3269000587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fd8104ab0 0x7f7fd819a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.582+0000 7f7fd6800640 1 --2- 172.21.15.12:0/3269000587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fd8105470 0x7f7fd81a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.583+0000 7f7fd5e00640 1 -- 172.21.15.12:0/3269000587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fd8104ab0 msgr2=0x7f7fd819a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.583+0000 7f7fd5e00640 1 --2- 172.21.15.12:0/3269000587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fd8104ab0 0x7f7fd819a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.583+0000 7f7fd5e00640 1 -- 172.21.15.12:0/3269000587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fd8105470 msgr2=0x7f7fd81a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.583+0000 7f7fd5e00640 1 --2- 172.21.15.12:0/3269000587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fd8105470 0x7f7fd81a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.583+0000 7f7fd5e00640 1 -- 172.21.15.12:0/3269000587 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd81a17f0 con 0x7f7fd81038c0 2024-03-23T23:09:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.583+0000 7f7fd6800640 1 --2- 172.21.15.12:0/3269000587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fd8105470 0x7f7fd81a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:09:13.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.583+0000 7f7fd5e00640 1 --2- 172.21.15.12:0/3269000587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fd81038c0 0x7f7fd8199bd0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f7fc802efd0 tx=0x7f7fc8005e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:13.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.583+0000 7f7fcea00640 1 -- 172.21.15.12:0/3269000587 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc8041070 con 0x7f7fd81038c0 2024-03-23T23:09:13.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.583+0000 7f7fcea00640 1 -- 172.21.15.12:0/3269000587 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7fc80054e0 con 0x7f7fd81038c0 2024-03-23T23:09:13.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.583+0000 7f7fcea00640 1 -- 172.21.15.12:0/3269000587 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc8034440 con 0x7f7fd81038c0 2024-03-23T23:09:13.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.583+0000 7f7fdca00640 1 -- 172.21.15.12:0/3269000587 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fd81a1a80 con 0x7f7fd81038c0 2024-03-23T23:09:13.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.584+0000 7f7fdca00640 1 -- 172.21.15.12:0/3269000587 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fd81a1ef0 con 0x7f7fd81038c0 2024-03-23T23:09:13.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.588+0000 7f7fdf7d7740 1 Processor -- start 2024-03-23T23:09:13.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.588+0000 7f7fdf7d7740 1 -- start start 2024-03-23T23:09:13.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.588+0000 7f7fdf7d7740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5633da978e80 0x5633da9792a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.588+0000 7f7fdf7d7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633da979870 0x5633da97e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.588+0000 7f7fdf7d7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633da97e9e0 0x5633da980e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.588+0000 7f7fdf7d7740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633da8de250 con 0x5633da978e80 2024-03-23T23:09:13.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.588+0000 7f7fdf7d7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633da886a30 con 0x5633da97e9e0 2024-03-23T23:09:13.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.588+0000 7f7fdf7d7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633da8da2f0 con 0x5633da979870 2024-03-23T23:09:13.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.588+0000 7f7fcea00640 1 -- 172.21.15.12:0/3269000587 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7fc8044070 con 0x7f7fd81038c0 2024-03-23T23:09:13.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.589+0000 7f7fd6800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5633da978e80 0x5633da9792a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.589+0000 7f7fd6800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5633da978e80 0x5633da9792a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39532/0 (socket says 172.21.15.12:39532) 2024-03-23T23:09:13.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.589+0000 7f7fd6800640 1 -- 172.21.15.12:0/3208040486 learned_addr learned my addr 172.21.15.12:0/3208040486 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:13.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.589+0000 7f7fd5e00640 1 --2- 172.21.15.12:0/3208040486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633da979870 0x5633da97e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.589+0000 7f7fd5400640 1 --2- 172.21.15.12:0/3208040486 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633da97e9e0 0x5633da980e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.589+0000 7f7fd6800640 1 -- 172.21.15.12:0/3208040486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633da979870 msgr2=0x5633da97e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:13.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.589+0000 7f7fd6800640 1 --2- 172.21.15.12:0/3208040486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633da979870 0x5633da97e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.589+0000 7f7fd6800640 1 -- 172.21.15.12:0/3208040486 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633da97e9e0 msgr2=0x5633da980e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:13.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.589+0000 7f7fd6800640 1 --2- 172.21.15.12:0/3208040486 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633da97e9e0 0x5633da980e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.589+0000 7f7fd6800640 1 -- 172.21.15.12:0/3208040486 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5633da871690 con 0x5633da978e80 2024-03-23T23:09:13.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.589+0000 7f7fd5400640 1 --2- 172.21.15.12:0/3208040486 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633da97e9e0 0x5633da980e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:13.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.589+0000 7f7fd6800640 1 --2- 172.21.15.12:0/3208040486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5633da978e80 0x5633da9792a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0009fb0 tx=0x7f7fd0010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8055088c9c2e116 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:13.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.590+0000 7f7fc3000640 1 -- 172.21.15.12:0/3208040486 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0016070 con 0x5633da978e80 2024-03-23T23:09:13.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.590+0000 7f7fc3000640 1 -- 172.21.15.12:0/3208040486 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7fd0012bf0 con 0x5633da978e80 2024-03-23T23:09:13.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.590+0000 7f7fc3000640 1 -- 172.21.15.12:0/3208040486 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd001bc70 con 0x5633da978e80 2024-03-23T23:09:13.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.590+0000 7f7fcea00640 1 --2- 172.21.15.12:0/3269000587 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7fbc0a2880 0x7f7fbc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.590+0000 7f7fcea00640 1 -- 172.21.15.12:0/3269000587 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7fc80f06f0 con 0x7f7fd81038c0 2024-03-23T23:09:13.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.590+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/3208040486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5633da978e80 msgr2=0x5633da9792a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:13.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.590+0000 7f7fdf7d7740 1 --2- 172.21.15.12:0/3208040486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5633da978e80 0x5633da9792a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0009fb0 tx=0x7f7fd0010800 comp rx=0 tx=0).stop 2024-03-23T23:09:13.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.591+0000 7f7fd5e00640 1 --2- 172.21.15.12:0/3269000587 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7fbc0a2880 0x7f7fbc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.591+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/3208040486 shutdown_connections 2024-03-23T23:09:13.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.591+0000 7f7fdf7d7740 1 --2- 172.21.15.12:0/3208040486 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633da97e9e0 0x5633da980e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.591+0000 7f7fdf7d7740 1 --2- 172.21.15.12:0/3208040486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633da979870 0x5633da97e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.591+0000 7f7fdf7d7740 1 --2- 172.21.15.12:0/3208040486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5633da978e80 0x5633da9792a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.591+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/3208040486 >> 172.21.15.12:0/3208040486 conn(0x5633da941200 msgr2=0x5633da9747f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:13.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.591+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/3208040486 shutdown_connections 2024-03-23T23:09:13.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.591+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/3208040486 wait complete. 2024-03-23T23:09:13.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.591+0000 7f7fd5e00640 1 --2- 172.21.15.12:0/3269000587 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7fbc0a2880 0x7f7fbc0a4d40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7fc80b94a0 tx=0x7f7fc80f6530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:13.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.591+0000 7f7fdf7d7740 1 -- start start 2024-03-23T23:09:13.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.592+0000 7f7fdf7d7740 1 Processor -- start 2024-03-23T23:09:13.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.592+0000 7f7fdf7d7740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5633da978e80 0x5633da982e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.592+0000 7f7fdf7d7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633da979870 0x5633da9833d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.592+0000 7f7fdf7d7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633da97e9e0 0x5633da98a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.592+0000 7f7fdf7d7740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633da981990 con 0x5633da978e80 2024-03-23T23:09:13.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.592+0000 7f7fdf7d7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633da899e80 con 0x5633da979870 2024-03-23T23:09:13.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.592+0000 7f7fdf7d7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633da981b10 con 0x5633da97e9e0 2024-03-23T23:09:13.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.592+0000 7f7fdf7d7740 10 client.0 _ll_register_callbacks cb 0x5633da8ac130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:13.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.592+0000 7f7fd5400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633da979870 0x5633da9833d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.592+0000 7f7fd6800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5633da978e80 0x5633da982e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.592+0000 7f7fd5400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633da979870 0x5633da9833d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:34288/0 (socket says 172.21.15.12:34288) 2024-03-23T23:09:13.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.592+0000 7f7fd5400640 1 -- 172.21.15.12:0/1657696347 learned_addr learned my addr 172.21.15.12:0/1657696347 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:13.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.593+0000 7f7fd6800640 1 --2- 172.21.15.12:0/1657696347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633da97e9e0 0x5633da98a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.593+0000 7f7fd5400640 1 -- 172.21.15.12:0/1657696347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633da97e9e0 msgr2=0x5633da98a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:13.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.593+0000 7f7fd5400640 1 --2- 172.21.15.12:0/1657696347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633da97e9e0 0x5633da98a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.593+0000 7f7fd5400640 1 -- 172.21.15.12:0/1657696347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5633da978e80 msgr2=0x5633da982e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:13.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.593+0000 7f7fd5400640 1 --2- 172.21.15.12:0/1657696347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5633da978e80 0x5633da982e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.593+0000 7f7fd5400640 1 -- 172.21.15.12:0/1657696347 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5633da6bc7d0 con 0x5633da979870 2024-03-23T23:09:13.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.593+0000 7f7fd5400640 1 --2- 172.21.15.12:0/1657696347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633da979870 0x5633da9833d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f7fc400bad0 tx=0x7f7fc400bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef92eae4eeaedd13 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:13.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.593+0000 7f7fb7400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:09:13.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.593+0000 7f7fb7400640 1 -- 172.21.15.12:0/1657696347 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc4003e20 con 0x5633da979870 2024-03-23T23:09:13.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.593+0000 7f7fb7400640 1 -- 172.21.15.12:0/1657696347 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7fc4011440 con 0x5633da979870 2024-03-23T23:09:13.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.594+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/1657696347 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7fd0012bf0 con 0x5633da979870 2024-03-23T23:09:13.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.594+0000 7f7fb7400640 1 -- 172.21.15.12:0/1657696347 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7fc4011640 con 0x5633da979870 2024-03-23T23:09:13.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.594+0000 7f7fdf7d7740 10 client.27428 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:13.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.594+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/1657696347 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5633da5f23f0 con 0x5633da979870 2024-03-23T23:09:13.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.594+0000 7f7fb7400640 1 -- 172.21.15.12:0/1657696347 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7fd0012bf0 con 0x5633da979870 2024-03-23T23:09:13.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.594+0000 7f7fb7400640 1 -- 172.21.15.12:0/1657696347 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f7fc401ba70 con 0x5633da979870 2024-03-23T23:09:13.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.595+0000 7f7fdf7d7740 10 client.27428 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:13.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.595+0000 7f7fdf7d7740 10 client.27428 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:13.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.595+0000 7f7fdf7d7740 20 client.27428 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:13.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.595+0000 7f7fdf7d7740 1 --2- 172.21.15.12:0/1657696347 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5633da98ebe0 0x5633da991020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.595+0000 7f7fdf7d7740 4 client.27428 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:09:13.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.595+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/1657696347 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5633da8eb790 con 0x5633da98ebe0 2024-03-23T23:09:13.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.595+0000 7f7fd6800640 1 --2- 172.21.15.12:0/1657696347 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5633da98ebe0 0x5633da991020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.595+0000 7f7fd6800640 1 --2- 172.21.15.12:0/1657696347 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5633da98ebe0 0x5633da991020 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e2b22a7cb3563c1 server_cookie=d3a6a3fb741882ca in_seq=0 out_seq=0 2024-03-23T23:09:13.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.595+0000 7f7fb7400640 10 client.27428 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:13.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.596+0000 7f7fb7400640 1 -- 172.21.15.12:0/1657696347 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5633da8eb790 con 0x5633da98ebe0 2024-03-23T23:09:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.596+0000 7f7fb7400640 10 client.27428 handle_command_reply: tid=0 2024-03-23T23:09:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.596+0000 7f7fdf7d7740 1 client.27428 shutdown 2024-03-23T23:09:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.596+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/1657696347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633da979870 msgr2=0x5633da9833d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.596+0000 7f7fdf7d7740 1 --2- 172.21.15.12:0/1657696347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633da979870 0x5633da9833d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f7fc400bad0 tx=0x7f7fc400bf90 comp rx=0 tx=0).stop 2024-03-23T23:09:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.597+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/1657696347 shutdown_connections 2024-03-23T23:09:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.597+0000 7f7fdf7d7740 1 --2- 172.21.15.12:0/1657696347 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5633da98ebe0 0x5633da991020 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.597+0000 7f7fdf7d7740 1 --2- 172.21.15.12:0/1657696347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633da97e9e0 0x5633da98a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.597+0000 7f7fb7400640 0 client.27428 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.597+0000 7f7fdf7d7740 1 --2- 172.21.15.12:0/1657696347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633da979870 0x5633da9833d0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.597+0000 7f7fdf7d7740 1 --2- 172.21.15.12:0/1657696347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5633da978e80 0x5633da982e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.597+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/1657696347 >> 172.21.15.12:0/1657696347 conn(0x5633da941200 msgr2=0x5633da973a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.597+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/1657696347 shutdown_connections 2024-03-23T23:09:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.597+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/1657696347 wait complete. 2024-03-23T23:09:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.597+0000 7f7fdf7d7740 20 client.27428 trim_cache size 0 max 16384 2024-03-23T23:09:13.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.615+0000 7f7fdf7d7740 1 Processor -- start 2024-03-23T23:09:13.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.615+0000 7f7fdf7d7740 1 -- start start 2024-03-23T23:09:13.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.616+0000 7f7fdf7d7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633da978e80 0x5633da982180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.616+0000 7f7fdf7d7740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5633da979870 0x5633da98a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.616+0000 7f7fdf7d7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633da97e9e0 0x5633da97aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.616+0000 7f7fdf7d7740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd81ad130 con 0x5633da979870 2024-03-23T23:09:13.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.616+0000 7f7fdf7d7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd81aac30 con 0x5633da97e9e0 2024-03-23T23:09:13.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.616+0000 7f7fdf7d7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd81a8cd0 con 0x5633da978e80 2024-03-23T23:09:13.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.616+0000 7f7fd6800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633da978e80 0x5633da982180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.616+0000 7f7fd6800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633da978e80 0x5633da982180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:55098/0 (socket says 172.21.15.12:55098) 2024-03-23T23:09:13.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.616+0000 7f7fd6800640 1 -- 172.21.15.12:0/1858118731 learned_addr learned my addr 172.21.15.12:0/1858118731 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:13.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.616+0000 7f7fd5400640 1 --2- 172.21.15.12:0/1858118731 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5633da979870 0x5633da98a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.616+0000 7f7fd6800640 1 --2- 172.21.15.12:0/1858118731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633da97e9e0 0x5633da97aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.617+0000 7f7fd6800640 1 -- 172.21.15.12:0/1858118731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633da97e9e0 msgr2=0x5633da97aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:13.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.617+0000 7f7fd6800640 1 --2- 172.21.15.12:0/1858118731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633da97e9e0 0x5633da97aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.617+0000 7f7fd6800640 1 -- 172.21.15.12:0/1858118731 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5633da979870 msgr2=0x5633da98a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.617+0000 7f7fd6800640 1 --2- 172.21.15.12:0/1858118731 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5633da979870 0x5633da98a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.617+0000 7f7fd6800640 1 -- 172.21.15.12:0/1858118731 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5633da5b20f0 con 0x5633da978e80 2024-03-23T23:09:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.617+0000 7f7fd6800640 1 --2- 172.21.15.12:0/1858118731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633da978e80 0x5633da982180 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0006010 tx=0x7f7fd0000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53fd2c6ed4ed01d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.617+0000 7f7fc2600640 1 -- 172.21.15.12:0/1858118731 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0016040 con 0x5633da978e80 2024-03-23T23:09:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.617+0000 7f7fc2600640 1 -- 172.21.15.12:0/1858118731 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7fd00129f0 con 0x5633da978e80 2024-03-23T23:09:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.617+0000 7f7fc2600640 1 -- 172.21.15.12:0/1858118731 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd001bcd0 con 0x5633da978e80 2024-03-23T23:09:13.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.618+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/1858118731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633da978e80 msgr2=0x5633da982180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:13.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.618+0000 7f7fdf7d7740 1 --2- 172.21.15.12:0/1858118731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633da978e80 0x5633da982180 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0006010 tx=0x7f7fd0000f40 comp rx=0 tx=0).stop 2024-03-23T23:09:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.618+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/1858118731 shutdown_connections 2024-03-23T23:09:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.618+0000 7f7fdf7d7740 1 --2- 172.21.15.12:0/1858118731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633da97e9e0 0x5633da97aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.618+0000 7f7fdf7d7740 1 --2- 172.21.15.12:0/1858118731 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5633da979870 0x5633da98a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.618+0000 7f7fdf7d7740 1 --2- 172.21.15.12:0/1858118731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633da978e80 0x5633da982180 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.618+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/1858118731 >> 172.21.15.12:0/1858118731 conn(0x5633da941200 msgr2=0x5633da97a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.618+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/1858118731 shutdown_connections 2024-03-23T23:09:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.618+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/1858118731 wait complete. 2024-03-23T23:09:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.618+0000 7f7fdf7d7740 1 -- start start 2024-03-23T23:09:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.619+0000 7f7fdf7d7740 1 Processor -- start 2024-03-23T23:09:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.619+0000 7f7fdf7d7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633da978e80 0x5633da987e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.619+0000 7f7fdf7d7740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5633da979870 0x5633da988380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.619+0000 7f7fdf7d7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633da97e9e0 0x5633da989390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.619+0000 7f7fdf7d7740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd81a92d0 con 0x5633da979870 2024-03-23T23:09:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.619+0000 7f7fdf7d7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd81aa5d0 con 0x5633da97e9e0 2024-03-23T23:09:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.619+0000 7f7fdf7d7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd81a9910 con 0x5633da978e80 2024-03-23T23:09:13.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.619+0000 7f7fdf7d7740 10 client.0 _ll_register_callbacks cb 0x7f7fc4004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:13.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.619+0000 7f7fd5400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5633da979870 0x5633da988380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.619+0000 7f7fd6800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633da978e80 0x5633da987e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.619+0000 7f7fd5400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5633da979870 0x5633da988380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39566/0 (socket says 172.21.15.12:39566) 2024-03-23T23:09:13.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.619+0000 7f7fd5400640 1 -- 172.21.15.12:0/1890321068 learned_addr learned my addr 172.21.15.12:0/1890321068 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:13.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.619+0000 7f7fd6800640 1 --2- 172.21.15.12:0/1890321068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633da97e9e0 0x5633da989390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.620+0000 7f7fd5400640 1 -- 172.21.15.12:0/1890321068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633da978e80 msgr2=0x5633da987e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:13.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.620+0000 7f7fd5400640 1 --2- 172.21.15.12:0/1890321068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633da978e80 0x5633da987e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.620+0000 7f7fd5400640 1 -- 172.21.15.12:0/1890321068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633da97e9e0 msgr2=0x5633da989390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:13.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.620+0000 7f7fd5400640 1 --2- 172.21.15.12:0/1890321068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633da97e9e0 0x5633da989390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.620+0000 7f7fd5400640 1 -- 172.21.15.12:0/1890321068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5633da989980 con 0x5633da979870 2024-03-23T23:09:13.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.620+0000 7f7fd5400640 1 --2- 172.21.15.12:0/1890321068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5633da979870 0x5633da988380 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x5633da984370 tx=0x7f7fc4005540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bc2ac9284962e28 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:13.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.620+0000 7f7fb7400640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:09:13.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.620+0000 7f7fb7400640 1 -- 172.21.15.12:0/1890321068 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc4033060 con 0x5633da979870 2024-03-23T23:09:13.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.620+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/1890321068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7fd00129f0 con 0x5633da979870 2024-03-23T23:09:13.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.620+0000 7f7fb7400640 1 -- 172.21.15.12:0/1890321068 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7fc4017040 con 0x5633da979870 2024-03-23T23:09:13.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.621+0000 7f7fb7400640 1 -- 172.21.15.12:0/1890321068 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7fc4011440 con 0x5633da979870 2024-03-23T23:09:13.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.621+0000 7f7fdf7d7740 10 client.17295 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:13.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.621+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/1890321068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5633da9ea8e0 con 0x5633da979870 2024-03-23T23:09:13.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.621+0000 7f7fb7400640 1 -- 172.21.15.12:0/1890321068 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7fd00129f0 con 0x5633da979870 2024-03-23T23:09:13.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.621+0000 7f7fb7400640 1 -- 172.21.15.12:0/1890321068 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f7fc4004940 con 0x5633da979870 2024-03-23T23:09:13.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.621+0000 7f7fdf7d7740 10 client.17295 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:13.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.621+0000 7f7fdf7d7740 10 client.17295 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:13.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.621+0000 7f7fdf7d7740 20 client.17295 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:13.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.621+0000 7f7fdf7d7740 1 --2- 172.21.15.12:0/1890321068 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5633da98ebe0 0x5633da9ed240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.621+0000 7f7fdf7d7740 4 client.17295 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:09:13.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.621+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/1890321068 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5633da8e5ae0 con 0x5633da98ebe0 2024-03-23T23:09:13.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.622+0000 7f7fd6800640 1 --2- 172.21.15.12:0/1890321068 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5633da98ebe0 0x5633da9ed240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.622+0000 7f7fd6800640 1 --2- 172.21.15.12:0/1890321068 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5633da98ebe0 0x5633da9ed240 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4bc0c0db88a10c0 server_cookie=448a27ce024f3873 in_seq=0 out_seq=0 2024-03-23T23:09:13.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.622+0000 7f7fb7400640 10 client.17295 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:13.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.622+0000 7f7fb7400640 1 -- 172.21.15.12:0/1890321068 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5633da8e5ae0 con 0x5633da98ebe0 2024-03-23T23:09:13.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.622+0000 7f7fb7400640 10 client.17295 handle_command_reply: tid=0 2024-03-23T23:09:13.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.622+0000 7f7fdf7d7740 1 client.17295 shutdown 2024-03-23T23:09:13.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.623+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/1890321068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5633da979870 msgr2=0x5633da988380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:13.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.623+0000 7f7fdf7d7740 1 --2- 172.21.15.12:0/1890321068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5633da979870 0x5633da988380 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x5633da984370 tx=0x7f7fc4005540 comp rx=0 tx=0).stop 2024-03-23T23:09:13.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.623+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/1890321068 shutdown_connections 2024-03-23T23:09:13.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.623+0000 7f7fdf7d7740 1 --2- 172.21.15.12:0/1890321068 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5633da98ebe0 0x5633da9ed240 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.623+0000 7f7fdf7d7740 1 --2- 172.21.15.12:0/1890321068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633da97e9e0 0x5633da989390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.623+0000 7f7fdf7d7740 1 --2- 172.21.15.12:0/1890321068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5633da979870 0x5633da988380 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.623+0000 7f7fb7400640 0 client.17295 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:13.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.623+0000 7f7fdf7d7740 1 --2- 172.21.15.12:0/1890321068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633da978e80 0x5633da987e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.623+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/1890321068 >> 172.21.15.12:0/1890321068 conn(0x5633da941200 msgr2=0x5633da97aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:13.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.623+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/1890321068 shutdown_connections 2024-03-23T23:09:13.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.623+0000 7f7fdf7d7740 1 -- 172.21.15.12:0/1890321068 wait complete. 2024-03-23T23:09:13.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.623+0000 7f7fdf7d7740 20 client.17295 trim_cache size 0 max 16384 2024-03-23T23:09:13.626 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:09:13.626 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T23:09:13.627 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-03-23T23:09:13.627 INFO:teuthology.orchestra.run.smithi012.stdout: "9b8749fd-9c7b-4f36-99f8-e96645293954": { 2024-03-23T23:09:13.627 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-03-23T23:09:13.627 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "9b8749fd-9c7b-4f36-99f8-e96645293954", 2024-03-23T23:09:13.627 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-03-23T23:09:13.627 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:09:13.627 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:09:13.627 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:09:13.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.623+0000 7f7fdca00640 1 -- 172.21.15.12:0/3269000587 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7fbc0a2880 msgr2=0x7f7fbc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:13.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.624+0000 7f7fdca00640 1 --2- 172.21.15.12:0/3269000587 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7fbc0a2880 0x7f7fbc0a4d40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7fc80b94a0 tx=0x7f7fc80f6530 comp rx=0 tx=0).stop 2024-03-23T23:09:13.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.624+0000 7f7fdca00640 1 -- 172.21.15.12:0/3269000587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fd81038c0 msgr2=0x7f7fd8199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:13.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.624+0000 7f7fdca00640 1 --2- 172.21.15.12:0/3269000587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fd81038c0 0x7f7fd8199bd0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f7fc802efd0 tx=0x7f7fc8005e40 comp rx=0 tx=0).stop 2024-03-23T23:09:13.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.624+0000 7f7fdca00640 1 -- 172.21.15.12:0/3269000587 shutdown_connections 2024-03-23T23:09:13.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.624+0000 7f7fdca00640 1 --2- 172.21.15.12:0/3269000587 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7fbc0a2880 0x7f7fbc0a4d40 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.624+0000 7f7fdca00640 1 --2- 172.21.15.12:0/3269000587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fd8105470 0x7f7fd81a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.624+0000 7f7fdca00640 1 --2- 172.21.15.12:0/3269000587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fd8104ab0 0x7f7fd819a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.624+0000 7f7fdca00640 1 --2- 172.21.15.12:0/3269000587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fd81038c0 0x7f7fd8199bd0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.624+0000 7f7fdca00640 1 -- 172.21.15.12:0/3269000587 >> 172.21.15.12:0/3269000587 conn(0x7f7fd80fee50 msgr2=0x7f7fd8107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:13.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.624+0000 7f7fdca00640 1 -- 172.21.15.12:0/3269000587 shutdown_connections 2024-03-23T23:09:13.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:13.624+0000 7f7fdca00640 1 -- 172.21.15.12:0/3269000587 wait complete. 2024-03-23T23:09:13.640 INFO:tasks.cephfs.filesystem:scrub status for tag:9b8749fd-9c7b-4f36-99f8-e96645293954 - {'path': '/', 'tag': '9b8749fd-9c7b-4f36-99f8-e96645293954', 'options': 'recursive,force'} 2024-03-23T23:09:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.955+0000 7fdb2ca00640 1 -- 172.21.15.12:0/2392112617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb281098b0 msgr2=0x7fdb28110140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.955+0000 7fdb2ca00640 1 --2- 172.21.15.12:0/2392112617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb281098b0 0x7fdb28110140 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdb20009be0 tx=0x7fdb2002bef0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.956+0000 7fdb2ca00640 1 -- 172.21.15.12:0/2392112617 shutdown_connections 2024-03-23T23:09:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.956+0000 7fdb2ca00640 1 --2- 172.21.15.12:0/2392112617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb281098b0 0x7fdb28110140 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.956+0000 7fdb2ca00640 1 --2- 172.21.15.12:0/2392112617 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb28108ef0 0x7fdb28109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.956+0000 7fdb2ca00640 1 --2- 172.21.15.12:0/2392112617 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb28107d00 0x7fdb28108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.956+0000 7fdb2ca00640 1 -- 172.21.15.12:0/2392112617 >> 172.21.15.12:0/2392112617 conn(0x7fdb28076cb0 msgr2=0x7fdb280790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.956+0000 7fdb2ca00640 1 -- 172.21.15.12:0/2392112617 shutdown_connections 2024-03-23T23:09:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.956+0000 7fdb2ca00640 1 -- 172.21.15.12:0/2392112617 wait complete. 2024-03-23T23:09:13.958 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.957+0000 7fdb2ca00640 1 Processor -- start 2024-03-23T23:09:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.957+0000 7fdb2ca00640 1 -- start start 2024-03-23T23:09:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.957+0000 7fdb2ca00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb28107d00 0x7fdb2819df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.957+0000 7fdb2ca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb28108ef0 0x7fdb2819e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.957+0000 7fdb2ca00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb281098b0 0x7fdb281a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.957+0000 7fdb2ca00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb281139b0 con 0x7fdb281098b0 2024-03-23T23:09:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.957+0000 7fdb2ca00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb28113830 con 0x7fdb28108ef0 2024-03-23T23:09:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.957+0000 7fdb2ca00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb28113b30 con 0x7fdb28107d00 2024-03-23T23:09:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.957+0000 7fdb25e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb28107d00 0x7fdb2819df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.957+0000 7fdb25e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb28107d00 0x7fdb2819df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:36382/0 (socket says 172.21.15.12:36382) 2024-03-23T23:09:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.957+0000 7fdb25e00640 1 -- 172.21.15.12:0/1862858733 learned_addr learned my addr 172.21.15.12:0/1862858733 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.957+0000 7fdb25400640 1 --2- 172.21.15.12:0/1862858733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb28108ef0 0x7fdb2819e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.958+0000 7fdb26800640 1 --2- 172.21.15.12:0/1862858733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb281098b0 0x7fdb281a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.958+0000 7fdb25e00640 1 -- 172.21.15.12:0/1862858733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb28108ef0 msgr2=0x7fdb2819e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.958+0000 7fdb25e00640 1 --2- 172.21.15.12:0/1862858733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb28108ef0 0x7fdb2819e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.958+0000 7fdb25e00640 1 -- 172.21.15.12:0/1862858733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb281098b0 msgr2=0x7fdb281a54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.958+0000 7fdb25e00640 1 --2- 172.21.15.12:0/1862858733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb281098b0 0x7fdb281a54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.958+0000 7fdb25e00640 1 -- 172.21.15.12:0/1862858733 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb281a5b20 con 0x7fdb28107d00 2024-03-23T23:09:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.958+0000 7fdb26800640 1 --2- 172.21.15.12:0/1862858733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb281098b0 0x7fdb281a54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:09:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.958+0000 7fdb25e00640 1 --2- 172.21.15.12:0/1862858733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb28107d00 0x7fdb2819df00 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fdb1800e9c0 tx=0x7fdb1800edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.958+0000 7fdb1ea00640 1 -- 172.21.15.12:0/1862858733 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1800ccc0 con 0x7fdb28107d00 2024-03-23T23:09:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.958+0000 7fdb1ea00640 1 -- 172.21.15.12:0/1862858733 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb180035f0 con 0x7fdb28107d00 2024-03-23T23:09:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.958+0000 7fdb2ca00640 1 -- 172.21.15.12:0/1862858733 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb281a5e10 con 0x7fdb28107d00 2024-03-23T23:09:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.958+0000 7fdb1ea00640 1 -- 172.21.15.12:0/1862858733 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1800ce60 con 0x7fdb28107d00 2024-03-23T23:09:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.959+0000 7fdb2ca00640 1 -- 172.21.15.12:0/1862858733 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb281a62d0 con 0x7fdb28107d00 2024-03-23T23:09:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.962+0000 7fdb1ea00640 1 -- 172.21.15.12:0/1862858733 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fdb180037a0 con 0x7fdb28107d00 2024-03-23T23:09:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.962+0000 7fdb2f322740 1 Processor -- start 2024-03-23T23:09:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.962+0000 7fdb2f322740 1 -- start start 2024-03-23T23:09:13.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.962+0000 7fdb2f322740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560acd2281e0 0x560acd228600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.962+0000 7fdb2f322740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560acd228bd0 0x560acd22d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.962+0000 7fdb2f322740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560acd22dd40 0x560acd2301d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.962+0000 7fdb2f322740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x560acd1eb3b0 con 0x560acd228bd0 2024-03-23T23:09:13.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.962+0000 7fdb2f322740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x560acd18d830 con 0x560acd22dd40 2024-03-23T23:09:13.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.962+0000 7fdb2f322740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x560acd1365d0 con 0x560acd2281e0 2024-03-23T23:09:13.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.963+0000 7fdb26800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560acd2281e0 0x560acd228600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.963+0000 7fdb26800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560acd2281e0 0x560acd228600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:36390/0 (socket says 172.21.15.12:36390) 2024-03-23T23:09:13.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.963+0000 7fdb26800640 1 -- 172.21.15.12:0/504983793 learned_addr learned my addr 172.21.15.12:0/504983793 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:13.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.963+0000 7fdb25e00640 1 --2- 172.21.15.12:0/504983793 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560acd228bd0 0x560acd22d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.963+0000 7fdb25400640 1 --2- 172.21.15.12:0/504983793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560acd22dd40 0x560acd2301d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.963+0000 7fdb26800640 1 -- 172.21.15.12:0/504983793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560acd22dd40 msgr2=0x560acd2301d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:13.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.963+0000 7fdb26800640 1 --2- 172.21.15.12:0/504983793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560acd22dd40 0x560acd2301d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.963+0000 7fdb26800640 1 -- 172.21.15.12:0/504983793 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560acd228bd0 msgr2=0x560acd22d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:13.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.963+0000 7fdb26800640 1 --2- 172.21.15.12:0/504983793 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560acd228bd0 0x560acd22d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.963+0000 7fdb26800640 1 -- 172.21.15.12:0/504983793 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560acd14e2c0 con 0x560acd2281e0 2024-03-23T23:09:13.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.963+0000 7fdb25400640 1 --2- 172.21.15.12:0/504983793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560acd22dd40 0x560acd2301d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:13.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.963+0000 7fdb26800640 1 --2- 172.21.15.12:0/504983793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560acd2281e0 0x560acd228600 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fdb20034fd0 tx=0x7fdb20009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ddc97b3a3666d52 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:13.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.964+0000 7fdb1ea00640 1 --2- 172.21.15.12:0/1862858733 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdb0c0a2920 0x7fdb0c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.964+0000 7fdb17000640 1 -- 172.21.15.12:0/504983793 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb20043070 con 0x560acd2281e0 2024-03-23T23:09:13.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.964+0000 7fdb17000640 1 -- 172.21.15.12:0/504983793 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb20033bc0 con 0x560acd2281e0 2024-03-23T23:09:13.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.964+0000 7fdb17000640 1 -- 172.21.15.12:0/504983793 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2002cb80 con 0x560acd2281e0 2024-03-23T23:09:13.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.964+0000 7fdb25e00640 1 --2- 172.21.15.12:0/1862858733 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdb0c0a2920 0x7fdb0c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.964+0000 7fdb1ea00640 1 -- 172.21.15.12:0/1862858733 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fdb180cee50 con 0x7fdb28107d00 2024-03-23T23:09:13.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.965+0000 7fdb2f322740 1 -- 172.21.15.12:0/504983793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560acd2281e0 msgr2=0x560acd228600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:13.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.965+0000 7fdb2f322740 1 --2- 172.21.15.12:0/504983793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560acd2281e0 0x560acd228600 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fdb20034fd0 tx=0x7fdb20009840 comp rx=0 tx=0).stop 2024-03-23T23:09:13.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.965+0000 7fdb25e00640 1 --2- 172.21.15.12:0/1862858733 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdb0c0a2920 0x7fdb0c0a4de0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdb18009830 tx=0x7fdb180d2890 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:13.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.965+0000 7fdb2f322740 1 -- 172.21.15.12:0/504983793 shutdown_connections 2024-03-23T23:09:13.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.965+0000 7fdb2f322740 1 --2- 172.21.15.12:0/504983793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560acd22dd40 0x560acd2301d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.965+0000 7fdb2f322740 1 --2- 172.21.15.12:0/504983793 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560acd228bd0 0x560acd22d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.965+0000 7fdb2f322740 1 --2- 172.21.15.12:0/504983793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560acd2281e0 0x560acd228600 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.965+0000 7fdb2f322740 1 -- 172.21.15.12:0/504983793 >> 172.21.15.12:0/504983793 conn(0x560acd223750 msgr2=0x560acd223b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.965+0000 7fdb2f322740 1 -- 172.21.15.12:0/504983793 shutdown_connections 2024-03-23T23:09:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.965+0000 7fdb2f322740 1 -- 172.21.15.12:0/504983793 wait complete. 2024-03-23T23:09:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.965+0000 7fdb2f322740 1 -- start start 2024-03-23T23:09:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.966+0000 7fdb2f322740 1 Processor -- start 2024-03-23T23:09:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.966+0000 7fdb2f322740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560acd2281e0 0x560acd232370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.966+0000 7fdb2f322740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560acd228bd0 0x560acd2328b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.966+0000 7fdb2f322740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560acd22dd40 0x560acd2398f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.966+0000 7fdb2f322740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x560acd149a20 con 0x560acd22dd40 2024-03-23T23:09:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.966+0000 7fdb2f322740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x560acd1898d0 con 0x560acd228bd0 2024-03-23T23:09:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.966+0000 7fdb2f322740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x560acd230c90 con 0x560acd2281e0 2024-03-23T23:09:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.966+0000 7fdb2f322740 10 client.0 _ll_register_callbacks cb 0x560acd1861c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.966+0000 7fdb26800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560acd2281e0 0x560acd232370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.966+0000 7fdb26800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560acd22dd40 0x560acd2398f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.966+0000 7fdb25400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560acd228bd0 0x560acd2328b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.966+0000 7fdb26800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560acd2281e0 0x560acd232370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:36406/0 (socket says 172.21.15.12:36406) 2024-03-23T23:09:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.966+0000 7fdb26800640 1 -- 172.21.15.12:0/170516726 learned_addr learned my addr 172.21.15.12:0/170516726 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.967+0000 7fdb25400640 1 -- 172.21.15.12:0/170516726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560acd2281e0 msgr2=0x560acd232370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.967+0000 7fdb25400640 1 --2- 172.21.15.12:0/170516726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560acd2281e0 0x560acd232370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.967+0000 7fdb25400640 1 -- 172.21.15.12:0/170516726 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560acd22dd40 msgr2=0x560acd2398f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.967+0000 7fdb25400640 1 --2- 172.21.15.12:0/170516726 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560acd22dd40 0x560acd2398f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.967+0000 7fdb25400640 1 -- 172.21.15.12:0/170516726 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560accea2210 con 0x560acd228bd0 2024-03-23T23:09:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.967+0000 7fdb25400640 1 --2- 172.21.15.12:0/170516726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560acd228bd0 0x560acd2328b0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fdb2819f410 tx=0x7fdb10012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b446eb86e708f022 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.967+0000 7fdb0b400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:09:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.967+0000 7fdb0b400640 1 -- 172.21.15.12:0/170516726 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb10004030 con 0x560acd228bd0 2024-03-23T23:09:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.967+0000 7fdb0b400640 1 -- 172.21.15.12:0/170516726 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fdb100041d0 con 0x560acd228bd0 2024-03-23T23:09:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.967+0000 7fdb2f322740 1 -- 172.21.15.12:0/170516726 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdb20033bc0 con 0x560acd228bd0 2024-03-23T23:09:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.968+0000 7fdb0b400640 1 -- 172.21.15.12:0/170516726 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb100043d0 con 0x560acd228bd0 2024-03-23T23:09:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.968+0000 7fdb2f322740 10 client.27446 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.968+0000 7fdb2f322740 1 -- 172.21.15.12:0/170516726 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x560accf6e620 con 0x560acd228bd0 2024-03-23T23:09:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.968+0000 7fdb0b400640 1 -- 172.21.15.12:0/170516726 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb20033bc0 con 0x560acd228bd0 2024-03-23T23:09:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.968+0000 7fdb0b400640 1 -- 172.21.15.12:0/170516726 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fdb10003c40 con 0x560acd228bd0 2024-03-23T23:09:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.968+0000 7fdb2f322740 10 client.27446 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.968+0000 7fdb2f322740 10 client.27446 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.968+0000 7fdb2f322740 20 client.27446 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.968+0000 7fdb2f322740 1 --2- 172.21.15.12:0/170516726 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x560acd23e2f0 0x560acd240710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.968+0000 7fdb2f322740 4 client.27446 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:09:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.968+0000 7fdb2f322740 1 -- 172.21.15.12:0/170516726 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560acd19ad70 con 0x560acd23e2f0 2024-03-23T23:09:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.968+0000 7fdb26800640 1 --2- 172.21.15.12:0/170516726 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x560acd23e2f0 0x560acd240710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.969+0000 7fdb26800640 1 --2- 172.21.15.12:0/170516726 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x560acd23e2f0 0x560acd240710 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0f90de21a1c72a4 server_cookie=9555773a66b9cb22 in_seq=0 out_seq=0 2024-03-23T23:09:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.969+0000 7fdb0b400640 10 client.27446 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.970+0000 7fdb0b400640 1 -- 172.21.15.12:0/170516726 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560acd19ad70 con 0x560acd23e2f0 2024-03-23T23:09:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.970+0000 7fdb0b400640 10 client.27446 handle_command_reply: tid=0 2024-03-23T23:09:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.970+0000 7fdb2f322740 1 client.27446 shutdown 2024-03-23T23:09:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.970+0000 7fdb2f322740 1 -- 172.21.15.12:0/170516726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560acd228bd0 msgr2=0x560acd2328b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.970+0000 7fdb2f322740 1 --2- 172.21.15.12:0/170516726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560acd228bd0 0x560acd2328b0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fdb2819f410 tx=0x7fdb10012b60 comp rx=0 tx=0).stop 2024-03-23T23:09:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.970+0000 7fdb2f322740 1 -- 172.21.15.12:0/170516726 shutdown_connections 2024-03-23T23:09:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.970+0000 7fdb2f322740 1 --2- 172.21.15.12:0/170516726 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x560acd23e2f0 0x560acd240710 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.970+0000 7fdb2f322740 1 --2- 172.21.15.12:0/170516726 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560acd22dd40 0x560acd2398f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.970+0000 7fdb2f322740 1 --2- 172.21.15.12:0/170516726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560acd228bd0 0x560acd2328b0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.970+0000 7fdb2f322740 1 --2- 172.21.15.12:0/170516726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560acd2281e0 0x560acd232370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.970+0000 7fdb0b400640 0 client.27446 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.970+0000 7fdb2f322740 1 -- 172.21.15.12:0/170516726 >> 172.21.15.12:0/170516726 conn(0x560acd223750 msgr2=0x560acd222950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.970+0000 7fdb2f322740 1 -- 172.21.15.12:0/170516726 shutdown_connections 2024-03-23T23:09:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.970+0000 7fdb2f322740 1 -- 172.21.15.12:0/170516726 wait complete. 2024-03-23T23:09:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.971+0000 7fdb2f322740 20 client.27446 trim_cache size 0 max 16384 2024-03-23T23:09:13.986 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.986+0000 7fdb2f322740 1 Processor -- start 2024-03-23T23:09:13.986 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.986+0000 7fdb2f322740 1 -- start start 2024-03-23T23:09:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.986+0000 7fdb2f322740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560acd2281e0 0x560acd223b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.986+0000 7fdb2f322740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560acd228bd0 0x560acd2398b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.986+0000 7fdb2f322740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560acd22dd40 0x560acd2220c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.986+0000 7fdb2f322740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb281b14e0 con 0x560acd2281e0 2024-03-23T23:09:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.986+0000 7fdb2f322740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb281aefe0 con 0x560acd228bd0 2024-03-23T23:09:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.986+0000 7fdb2f322740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb281ad080 con 0x560acd22dd40 2024-03-23T23:09:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.987+0000 7fdb25400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560acd228bd0 0x560acd2398b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.987+0000 7fdb26800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560acd2281e0 0x560acd223b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.987+0000 7fdb25400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560acd228bd0 0x560acd2398b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48262/0 (socket says 172.21.15.12:48262) 2024-03-23T23:09:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.987+0000 7fdb25400640 1 -- 172.21.15.12:0/2384560881 learned_addr learned my addr 172.21.15.12:0/2384560881 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.987+0000 7fdb25400640 1 -- 172.21.15.12:0/2384560881 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560acd22dd40 msgr2=0x560acd2220c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-23T23:09:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.987+0000 7fdb25400640 1 --2- 172.21.15.12:0/2384560881 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560acd22dd40 0x560acd2220c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.987+0000 7fdb25400640 1 -- 172.21.15.12:0/2384560881 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560acd2281e0 msgr2=0x560acd223b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.987+0000 7fdb25400640 1 --2- 172.21.15.12:0/2384560881 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560acd2281e0 0x560acd223b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.987+0000 7fdb25400640 1 -- 172.21.15.12:0/2384560881 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560acd22d1a0 con 0x560acd228bd0 2024-03-23T23:09:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.987+0000 7fdb26800640 1 --2- 172.21.15.12:0/2384560881 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560acd2281e0 0x560acd223b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:09:13.991 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.987+0000 7fdb25400640 1 --2- 172.21.15.12:0/2384560881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560acd228bd0 0x560acd2398b0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fdb10009340 tx=0x7fdb1000f7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c5ed21cf8ef5b33 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:13.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.987+0000 7fdb16600640 1 -- 172.21.15.12:0/2384560881 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1002c070 con 0x560acd228bd0 2024-03-23T23:09:13.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.988+0000 7fdb16600640 1 -- 172.21.15.12:0/2384560881 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb1000fdd0 con 0x560acd228bd0 2024-03-23T23:09:13.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.988+0000 7fdb16600640 1 -- 172.21.15.12:0/2384560881 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb10010d60 con 0x560acd228bd0 2024-03-23T23:09:13.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.988+0000 7fdb2f322740 1 -- 172.21.15.12:0/2384560881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560acd228bd0 msgr2=0x560acd2398b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:13.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.988+0000 7fdb2f322740 1 --2- 172.21.15.12:0/2384560881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560acd228bd0 0x560acd2398b0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fdb10009340 tx=0x7fdb1000f7b0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.988+0000 7fdb2f322740 1 -- 172.21.15.12:0/2384560881 shutdown_connections 2024-03-23T23:09:13.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.988+0000 7fdb2f322740 1 --2- 172.21.15.12:0/2384560881 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560acd22dd40 0x560acd2220c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.988+0000 7fdb2f322740 1 --2- 172.21.15.12:0/2384560881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560acd228bd0 0x560acd2398b0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.988+0000 7fdb2f322740 1 --2- 172.21.15.12:0/2384560881 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560acd2281e0 0x560acd223b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.988+0000 7fdb2f322740 1 -- 172.21.15.12:0/2384560881 >> 172.21.15.12:0/2384560881 conn(0x560acd223750 msgr2=0x560acd229eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:13.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.988+0000 7fdb2f322740 1 -- 172.21.15.12:0/2384560881 shutdown_connections 2024-03-23T23:09:13.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.988+0000 7fdb2f322740 1 -- 172.21.15.12:0/2384560881 wait complete. 2024-03-23T23:09:13.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.989+0000 7fdb2f322740 1 -- start start 2024-03-23T23:09:13.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.989+0000 7fdb2f322740 1 Processor -- start 2024-03-23T23:09:13.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.989+0000 7fdb2f322740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560acd2281e0 0x560acd239330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.989+0000 7fdb2f322740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560acd228bd0 0x560acd239870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.989+0000 7fdb2f322740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560acd22dd40 0x560acd2379a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.989+0000 7fdb2f322740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb281ad680 con 0x560acd2281e0 2024-03-23T23:09:13.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.989+0000 7fdb2f322740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb281ae980 con 0x560acd22dd40 2024-03-23T23:09:13.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.989+0000 7fdb2f322740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb281adcc0 con 0x560acd228bd0 2024-03-23T23:09:13.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.989+0000 7fdb2f322740 10 client.0 _ll_register_callbacks cb 0x7fdb10009eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:13.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.989+0000 7fdb26800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560acd2281e0 0x560acd239330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.989+0000 7fdb25400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560acd228bd0 0x560acd239870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.990+0000 7fdb26800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560acd2281e0 0x560acd239330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53616/0 (socket says 172.21.15.12:53616) 2024-03-23T23:09:13.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.990+0000 7fdb25400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560acd228bd0 0x560acd239870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:36428/0 (socket says 172.21.15.12:36428) 2024-03-23T23:09:13.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.990+0000 7fdb26800640 1 -- 172.21.15.12:0/1105656277 learned_addr learned my addr 172.21.15.12:0/1105656277 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:13.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.990+0000 7fdb26800640 1 --2- 172.21.15.12:0/1105656277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560acd22dd40 0x560acd2379a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.990+0000 7fdb25400640 1 -- 172.21.15.12:0/1105656277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560acd22dd40 msgr2=0x560acd2379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:13.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.990+0000 7fdb25400640 1 --2- 172.21.15.12:0/1105656277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560acd22dd40 0x560acd2379a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.990+0000 7fdb25400640 1 -- 172.21.15.12:0/1105656277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560acd2281e0 msgr2=0x560acd239330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:13.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.990+0000 7fdb25400640 1 --2- 172.21.15.12:0/1105656277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560acd2281e0 0x560acd239330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:13.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.990+0000 7fdb25400640 1 -- 172.21.15.12:0/1105656277 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560acd2381d0 con 0x560acd228bd0 2024-03-23T23:09:13.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.990+0000 7fdb25400640 1 --2- 172.21.15.12:0/1105656277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560acd228bd0 0x560acd239870 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fdb10009df0 tx=0x7fdb10005840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b13574339412ea0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:13.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.990+0000 7fdb0b400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:09:13.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.991+0000 7fdb0b400640 1 -- 172.21.15.12:0/1105656277 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb10036020 con 0x560acd228bd0 2024-03-23T23:09:13.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.991+0000 7fdb0b400640 1 -- 172.21.15.12:0/1105656277 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fdb1000f9b0 con 0x560acd228bd0 2024-03-23T23:09:13.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.991+0000 7fdb2f322740 1 -- 172.21.15.12:0/1105656277 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdb1000fdd0 con 0x560acd228bd0 2024-03-23T23:09:13.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.991+0000 7fdb0b400640 1 -- 172.21.15.12:0/1105656277 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb1002c040 con 0x560acd228bd0 2024-03-23T23:09:13.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.991+0000 7fdb2f322740 10 client.27421 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:13.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.991+0000 7fdb2f322740 1 -- 172.21.15.12:0/1105656277 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x560acd29a6f0 con 0x560acd228bd0 2024-03-23T23:09:13.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.991+0000 7fdb0b400640 1 -- 172.21.15.12:0/1105656277 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb1000fdd0 con 0x560acd228bd0 2024-03-23T23:09:13.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.991+0000 7fdb0b400640 1 -- 172.21.15.12:0/1105656277 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fdb10003c30 con 0x560acd228bd0 2024-03-23T23:09:13.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.991+0000 7fdb2f322740 10 client.27421 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:13.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.992+0000 7fdb2f322740 10 client.27421 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:13.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.992+0000 7fdb2f322740 20 client.27421 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:13.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.992+0000 7fdb2f322740 1 --2- 172.21.15.12:0/1105656277 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x560acd29af60 0x560acd29d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:13.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.992+0000 7fdb2f322740 4 client.27421 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:09:13.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.992+0000 7fdb2f322740 1 -- 172.21.15.12:0/1105656277 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560acd21ea80 con 0x560acd29af60 2024-03-23T23:09:13.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.992+0000 7fdb26800640 1 --2- 172.21.15.12:0/1105656277 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x560acd29af60 0x560acd29d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:13.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.992+0000 7fdb26800640 1 --2- 172.21.15.12:0/1105656277 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x560acd29af60 0x560acd29d380 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b84d8fc2f70aed7 server_cookie=6fb81ed283005b30 in_seq=0 out_seq=0 2024-03-23T23:09:13.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:13.992+0000 7fdb0b400640 10 client.27421 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:14.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.003+0000 7fdb0b400640 1 -- 172.21.15.12:0/1105656277 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560acd21ea80 con 0x560acd29af60 2024-03-23T23:09:14.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.003+0000 7fdb0b400640 10 client.27421 handle_command_reply: tid=0 2024-03-23T23:09:14.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.003+0000 7fdb2f322740 1 client.27421 shutdown 2024-03-23T23:09:14.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.004+0000 7fdb2f322740 1 -- 172.21.15.12:0/1105656277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560acd228bd0 msgr2=0x560acd239870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:14.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.004+0000 7fdb2f322740 1 --2- 172.21.15.12:0/1105656277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560acd228bd0 0x560acd239870 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fdb10009df0 tx=0x7fdb10005840 comp rx=0 tx=0).stop 2024-03-23T23:09:14.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.004+0000 7fdb2f322740 1 -- 172.21.15.12:0/1105656277 shutdown_connections 2024-03-23T23:09:14.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.004+0000 7fdb2f322740 1 --2- 172.21.15.12:0/1105656277 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x560acd29af60 0x560acd29d380 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:14.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.004+0000 7fdb2f322740 1 --2- 172.21.15.12:0/1105656277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560acd22dd40 0x560acd2379a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:14.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.004+0000 7fdb2f322740 1 --2- 172.21.15.12:0/1105656277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560acd228bd0 0x560acd239870 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:14.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.004+0000 7fdb2f322740 1 --2- 172.21.15.12:0/1105656277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560acd2281e0 0x560acd239330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:14.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.004+0000 7fdb0b400640 0 client.27421 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:14.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.004+0000 7fdb2f322740 1 -- 172.21.15.12:0/1105656277 >> 172.21.15.12:0/1105656277 conn(0x560acd223750 msgr2=0x560acd2269b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:14.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.004+0000 7fdb2f322740 1 -- 172.21.15.12:0/1105656277 shutdown_connections 2024-03-23T23:09:14.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.004+0000 7fdb2f322740 1 -- 172.21.15.12:0/1105656277 wait complete. 2024-03-23T23:09:14.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.004+0000 7fdb2f322740 20 client.27421 trim_cache size 0 max 16384 2024-03-23T23:09:14.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:09:14.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:09:14.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:09:14.006 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:09:14.006 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.005+0000 7fdb2ca00640 1 -- 172.21.15.12:0/1862858733 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdb0c0a2920 msgr2=0x7fdb0c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:14.006 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.005+0000 7fdb2ca00640 1 --2- 172.21.15.12:0/1862858733 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdb0c0a2920 0x7fdb0c0a4de0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdb18009830 tx=0x7fdb180d2890 comp rx=0 tx=0).stop 2024-03-23T23:09:14.006 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.005+0000 7fdb2ca00640 1 -- 172.21.15.12:0/1862858733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb28107d00 msgr2=0x7fdb2819df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:14.006 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.005+0000 7fdb2ca00640 1 --2- 172.21.15.12:0/1862858733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb28107d00 0x7fdb2819df00 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fdb1800e9c0 tx=0x7fdb1800edc0 comp rx=0 tx=0).stop 2024-03-23T23:09:14.006 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.005+0000 7fdb2ca00640 1 -- 172.21.15.12:0/1862858733 shutdown_connections 2024-03-23T23:09:14.006 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.005+0000 7fdb2ca00640 1 --2- 172.21.15.12:0/1862858733 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdb0c0a2920 0x7fdb0c0a4de0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:14.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.005+0000 7fdb2ca00640 1 --2- 172.21.15.12:0/1862858733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb281098b0 0x7fdb281a54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:14.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.005+0000 7fdb2ca00640 1 --2- 172.21.15.12:0/1862858733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb28108ef0 0x7fdb2819e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:14.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.005+0000 7fdb2ca00640 1 --2- 172.21.15.12:0/1862858733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb28107d00 0x7fdb2819df00 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:14.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.005+0000 7fdb2ca00640 1 -- 172.21.15.12:0/1862858733 >> 172.21.15.12:0/1862858733 conn(0x7fdb28076cb0 msgr2=0x7fdb2810c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:14.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.005+0000 7fdb2ca00640 1 -- 172.21.15.12:0/1862858733 shutdown_connections 2024-03-23T23:09:14.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:14.005+0000 7fdb2ca00640 1 -- 172.21.15.12:0/1862858733 wait complete. 2024-03-23T23:09:14.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:14 smithi203 ceph-mon[36693]: scrub queued for path: / 2024-03-23T23:09:14.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:14 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:09:14.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:14 smithi203 ceph-mon[36693]: scrub summary: active paths [/] 2024-03-23T23:09:14.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:14 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:09:14.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:14 smithi203 ceph-mon[36693]: pgmap v613: 97 pgs: 97 active+clean; 820 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:09:15.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:14 smithi012 ceph-mon[31695]: scrub queued for path: / 2024-03-23T23:09:15.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:14 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:09:15.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:14 smithi012 ceph-mon[31695]: scrub summary: active paths [/] 2024-03-23T23:09:15.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:14 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:09:15.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:14 smithi012 ceph-mon[31695]: pgmap v613: 97 pgs: 97 active+clean; 820 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:09:15.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:14 smithi173 ceph-mon[36116]: scrub queued for path: / 2024-03-23T23:09:15.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:14 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:09:15.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:14 smithi173 ceph-mon[36116]: scrub summary: active paths [/] 2024-03-23T23:09:15.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:14 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:09:15.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:14 smithi173 ceph-mon[36116]: pgmap v613: 97 pgs: 97 active+clean; 820 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:09:17.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:16 smithi012 ceph-mon[31695]: pgmap v614: 97 pgs: 97 active+clean; 888 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:09:17.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:16 smithi012 ceph-mon[31695]: scrub summary: idle 2024-03-23T23:09:17.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:16 smithi173 ceph-mon[36116]: pgmap v614: 97 pgs: 97 active+clean; 888 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:09:17.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:16 smithi173 ceph-mon[36116]: scrub summary: idle 2024-03-23T23:09:17.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:16 smithi203 ceph-mon[36693]: pgmap v614: 97 pgs: 97 active+clean; 888 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:09:17.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:16 smithi203 ceph-mon[36693]: scrub summary: idle 2024-03-23T23:09:19.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:18 smithi012 ceph-mon[31695]: pgmap v615: 97 pgs: 97 active+clean; 888 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:09:19.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:18 smithi173 ceph-mon[36116]: pgmap v615: 97 pgs: 97 active+clean; 888 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:09:19.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:18 smithi203 ceph-mon[36693]: pgmap v615: 97 pgs: 97 active+clean; 888 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:09:21.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:20 smithi012 ceph-mon[31695]: pgmap v616: 97 pgs: 97 active+clean; 888 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:09:21.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:20 smithi173 ceph-mon[36116]: pgmap v616: 97 pgs: 97 active+clean; 888 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:09:21.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:20 smithi203 ceph-mon[36693]: pgmap v616: 97 pgs: 97 active+clean; 888 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:09:23.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:22 smithi012 ceph-mon[31695]: pgmap v617: 97 pgs: 97 active+clean; 945 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:09:23.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:22 smithi173 ceph-mon[36116]: pgmap v617: 97 pgs: 97 active+clean; 945 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:09:23.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:22 smithi203 ceph-mon[36693]: pgmap v617: 97 pgs: 97 active+clean; 945 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:09:25.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:24 smithi012 ceph-mon[31695]: pgmap v618: 97 pgs: 97 active+clean; 945 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:09:25.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:24 smithi173 ceph-mon[36116]: pgmap v618: 97 pgs: 97 active+clean; 945 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:09:25.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:24 smithi203 ceph-mon[36693]: pgmap v618: 97 pgs: 97 active+clean; 945 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:09:27.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:26 smithi012 ceph-mon[31695]: pgmap v619: 97 pgs: 97 active+clean; 1007 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:09:27.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:26 smithi173 ceph-mon[36116]: pgmap v619: 97 pgs: 97 active+clean; 1007 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:09:27.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.151+0000 7fee03e00640 1 -- 172.21.15.12:0/2091922149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee041052d0 msgr2=0x7fee0410bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:27.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.151+0000 7fee03e00640 1 --2- 172.21.15.12:0/2091922149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee041052d0 0x7fee0410bb60 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fedfc009be0 tx=0x7fedfc02bef0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.151+0000 7fee03e00640 1 -- 172.21.15.12:0/2091922149 shutdown_connections 2024-03-23T23:09:27.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.151+0000 7fee03e00640 1 --2- 172.21.15.12:0/2091922149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee041052d0 0x7fee0410bb60 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.151+0000 7fee03e00640 1 --2- 172.21.15.12:0/2091922149 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee04104910 0x7fee04104d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.151+0000 7fee03e00640 1 --2- 172.21.15.12:0/2091922149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee04103720 0x7fee04103b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.151+0000 7fee03e00640 1 -- 172.21.15.12:0/2091922149 >> 172.21.15.12:0/2091922149 conn(0x7fee040fecb0 msgr2=0x7fee041010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:27.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.152+0000 7fee03e00640 1 -- 172.21.15.12:0/2091922149 shutdown_connections 2024-03-23T23:09:27.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.152+0000 7fee03e00640 1 -- 172.21.15.12:0/2091922149 wait complete. 2024-03-23T23:09:27.153 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.153+0000 7fee03e00640 1 Processor -- start 2024-03-23T23:09:27.153 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.153+0000 7fee03e00640 1 -- start start 2024-03-23T23:09:27.154 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.153+0000 7fee03e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee04103720 0x7fee04199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:27.154 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.154+0000 7fee03e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee04104910 0x7fee04199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:27.154 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.154+0000 7fee03e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee041052d0 0x7fee041a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.154+0000 7fee03e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0410f3f0 con 0x7fee04104910 2024-03-23T23:09:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.154+0000 7fee03e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0410f270 con 0x7fee04103720 2024-03-23T23:09:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.154+0000 7fee03e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0410f570 con 0x7fee041052d0 2024-03-23T23:09:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.154+0000 7fee02a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee04103720 0x7fee04199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.154+0000 7fee02a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee04103720 0x7fee04199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:39132/0 (socket says 172.21.15.12:39132) 2024-03-23T23:09:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.154+0000 7fee02a00640 1 -- 172.21.15.12:0/2389792928 learned_addr learned my addr 172.21.15.12:0/2389792928 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.154+0000 7fee03400640 1 --2- 172.21.15.12:0/2389792928 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee041052d0 0x7fee041a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.154+0000 7fee02000640 1 --2- 172.21.15.12:0/2389792928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee04104910 0x7fee04199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.154+0000 7fee02a00640 1 -- 172.21.15.12:0/2389792928 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee041052d0 msgr2=0x7fee041a1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.154+0000 7fee02a00640 1 --2- 172.21.15.12:0/2389792928 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee041052d0 0x7fee041a1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.154+0000 7fee02a00640 1 -- 172.21.15.12:0/2389792928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee04104910 msgr2=0x7fee04199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.154+0000 7fee02a00640 1 --2- 172.21.15.12:0/2389792928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee04104910 0x7fee04199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.154+0000 7fee02a00640 1 -- 172.21.15.12:0/2389792928 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee041a1690 con 0x7fee04103720 2024-03-23T23:09:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.154+0000 7fee02000640 1 --2- 172.21.15.12:0/2389792928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee04104910 0x7fee04199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:09:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.155+0000 7fee02a00640 1 --2- 172.21.15.12:0/2389792928 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee04103720 0x7fee04199a70 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fedf400e9f0 tx=0x7fedf400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:27.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.155+0000 7fee08a00640 1 -- 172.21.15.12:0/2389792928 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf400ccc0 con 0x7fee04103720 2024-03-23T23:09:27.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.155+0000 7fee08a00640 1 -- 172.21.15.12:0/2389792928 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedf40035f0 con 0x7fee04103720 2024-03-23T23:09:27.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.155+0000 7fee08a00640 1 -- 172.21.15.12:0/2389792928 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf400ce60 con 0x7fee04103720 2024-03-23T23:09:27.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.155+0000 7fee03e00640 1 -- 172.21.15.12:0/2389792928 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee041a1980 con 0x7fee04103720 2024-03-23T23:09:27.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.155+0000 7fee03e00640 1 -- 172.21.15.12:0/2389792928 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee041a1e80 con 0x7fee04103720 2024-03-23T23:09:27.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:26 smithi203 ceph-mon[36693]: pgmap v619: 97 pgs: 97 active+clean; 1007 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:09:27.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.159+0000 7fee0ca2f740 1 Processor -- start 2024-03-23T23:09:27.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.159+0000 7fee0ca2f740 1 -- start start 2024-03-23T23:09:27.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.159+0000 7fee0ca2f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b96af4b3f0 0x55b96af4b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:27.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.159+0000 7fee08a00640 1 -- 172.21.15.12:0/2389792928 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fedf4003b10 con 0x7fee04103720 2024-03-23T23:09:27.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.159+0000 7fee0ca2f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b96af4bde0 0x55b96af508e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:27.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.159+0000 7fee0ca2f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af50f50 0x55b96af533e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:27.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.159+0000 7fee0ca2f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b96af0e3b0 con 0x55b96af4bde0 2024-03-23T23:09:27.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.159+0000 7fee0ca2f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b96aeb0830 con 0x55b96af4b3f0 2024-03-23T23:09:27.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.159+0000 7fee0ca2f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b96ae595d0 con 0x55b96af50f50 2024-03-23T23:09:27.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.159+0000 7fee03400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b96af4b3f0 0x55b96af4b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:27.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.160+0000 7fee03400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b96af4b3f0 0x55b96af4b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:39146/0 (socket says 172.21.15.12:39146) 2024-03-23T23:09:27.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.160+0000 7fee03400640 1 -- 172.21.15.12:0/566445477 learned_addr learned my addr 172.21.15.12:0/566445477 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:27.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.160+0000 7fee02a00640 1 --2- 172.21.15.12:0/566445477 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b96af4bde0 0x55b96af508e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:27.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.160+0000 7fee02000640 1 --2- 172.21.15.12:0/566445477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af50f50 0x55b96af533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:27.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.160+0000 7fee03400640 1 -- 172.21.15.12:0/566445477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af50f50 msgr2=0x55b96af533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:27.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.160+0000 7fee03400640 1 --2- 172.21.15.12:0/566445477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af50f50 0x55b96af533e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.160+0000 7fee03400640 1 -- 172.21.15.12:0/566445477 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b96af4bde0 msgr2=0x55b96af508e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:27.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.160+0000 7fee03400640 1 --2- 172.21.15.12:0/566445477 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b96af4bde0 0x55b96af508e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.160+0000 7fee03400640 1 -- 172.21.15.12:0/566445477 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b96ae712c0 con 0x55b96af4b3f0 2024-03-23T23:09:27.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.160+0000 7fee02000640 1 --2- 172.21.15.12:0/566445477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af50f50 0x55b96af533e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:09:27.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.160+0000 7fee03400640 1 --2- 172.21.15.12:0/566445477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b96af4b3f0 0x55b96af4b810 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fedfc02c500 tx=0x7fedfc009330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28a4d50298a68736 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:27.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.161+0000 7fee08a00640 1 --2- 172.21.15.12:0/2389792928 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fede80a2870 0x7fede80a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:27.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.161+0000 7fedefe00640 1 -- 172.21.15.12:0/566445477 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc043030 con 0x55b96af4b3f0 2024-03-23T23:09:27.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.161+0000 7fedefe00640 1 -- 172.21.15.12:0/566445477 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedfc033bc0 con 0x55b96af4b3f0 2024-03-23T23:09:27.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.161+0000 7fedefe00640 1 -- 172.21.15.12:0/566445477 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc02c980 con 0x55b96af4b3f0 2024-03-23T23:09:27.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.161+0000 7fee08a00640 1 -- 172.21.15.12:0/2389792928 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fedf40ced20 con 0x7fee04103720 2024-03-23T23:09:27.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.161+0000 7fee02a00640 1 --2- 172.21.15.12:0/2389792928 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fede80a2870 0x7fede80a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.162+0000 7fee0ca2f740 1 -- 172.21.15.12:0/566445477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b96af4b3f0 msgr2=0x55b96af4b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.162+0000 7fee0ca2f740 1 --2- 172.21.15.12:0/566445477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b96af4b3f0 0x55b96af4b810 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fedfc02c500 tx=0x7fedfc009330 comp rx=0 tx=0).stop 2024-03-23T23:09:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.162+0000 7fee0ca2f740 1 -- 172.21.15.12:0/566445477 shutdown_connections 2024-03-23T23:09:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.162+0000 7fee0ca2f740 1 --2- 172.21.15.12:0/566445477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af50f50 0x55b96af533e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.162+0000 7fee0ca2f740 1 --2- 172.21.15.12:0/566445477 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b96af4bde0 0x55b96af508e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.162+0000 7fee0ca2f740 1 --2- 172.21.15.12:0/566445477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b96af4b3f0 0x55b96af4b810 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.162+0000 7fee0ca2f740 1 -- 172.21.15.12:0/566445477 >> 172.21.15.12:0/566445477 conn(0x55b96af46960 msgr2=0x55b96af46d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.162+0000 7fee02a00640 1 --2- 172.21.15.12:0/2389792928 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fede80a2870 0x7fede80a4d30 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x55b96af4c1e0 tx=0x7fedf40d27f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.162+0000 7fee0ca2f740 1 -- 172.21.15.12:0/566445477 shutdown_connections 2024-03-23T23:09:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.162+0000 7fee0ca2f740 1 -- 172.21.15.12:0/566445477 wait complete. 2024-03-23T23:09:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.162+0000 7fee0ca2f740 1 -- start start 2024-03-23T23:09:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.162+0000 7fee0ca2f740 1 Processor -- start 2024-03-23T23:09:27.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.163+0000 7fee0ca2f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af4b3f0 0x55b96af55570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:27.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.163+0000 7fee0ca2f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b96af4bde0 0x55b96af55ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:27.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.163+0000 7fee0ca2f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b96af50f50 0x55b96af5caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:27.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.163+0000 7fee0ca2f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b96ae6ca20 con 0x55b96af4bde0 2024-03-23T23:09:27.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.163+0000 7fee0ca2f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b96aeac8d0 con 0x55b96af50f50 2024-03-23T23:09:27.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.163+0000 7fee0ca2f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b96af53ea0 con 0x55b96af4b3f0 2024-03-23T23:09:27.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.163+0000 7fee0ca2f740 10 client.0 _ll_register_callbacks cb 0x55b96aea91c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:27.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.163+0000 7fee03400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af4b3f0 0x55b96af55570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:27.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.163+0000 7fee02000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b96af4bde0 0x55b96af55ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:27.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.163+0000 7fee03400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af4b3f0 0x55b96af55570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:48628/0 (socket says 172.21.15.12:48628) 2024-03-23T23:09:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.163+0000 7fee03400640 1 -- 172.21.15.12:0/808479647 learned_addr learned my addr 172.21.15.12:0/808479647 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.163+0000 7fee03400640 1 --2- 172.21.15.12:0/808479647 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b96af50f50 0x55b96af5caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.164+0000 7fee03400640 1 -- 172.21.15.12:0/808479647 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b96af50f50 msgr2=0x55b96af5caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.164+0000 7fee03400640 1 --2- 172.21.15.12:0/808479647 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b96af50f50 0x55b96af5caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.164+0000 7fee03400640 1 -- 172.21.15.12:0/808479647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b96af4bde0 msgr2=0x55b96af55ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.164+0000 7fee03400640 1 --2- 172.21.15.12:0/808479647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b96af4bde0 0x55b96af55ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.164+0000 7fee03400640 1 -- 172.21.15.12:0/808479647 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b96abc5210 con 0x55b96af4b3f0 2024-03-23T23:09:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.164+0000 7fee03400640 1 --2- 172.21.15.12:0/808479647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af4b3f0 0x55b96af55570 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fedfc034fd0 tx=0x7fedfc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bbd4676dfca80e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.164+0000 7fedecc00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:09:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.164+0000 7fedecc00640 1 -- 172.21.15.12:0/808479647 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc043040 con 0x55b96af4b3f0 2024-03-23T23:09:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.164+0000 7fedecc00640 1 -- 172.21.15.12:0/808479647 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fedfc004b40 con 0x55b96af4b3f0 2024-03-23T23:09:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.164+0000 7fee0ca2f740 1 -- 172.21.15.12:0/808479647 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fedfc033bc0 con 0x55b96af4b3f0 2024-03-23T23:09:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.164+0000 7fedecc00640 1 -- 172.21.15.12:0/808479647 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedfc004d40 con 0x55b96af4b3f0 2024-03-23T23:09:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.164+0000 7fee0ca2f740 10 client.27433 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.164+0000 7fee0ca2f740 1 -- 172.21.15.12:0/808479647 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55b96ac91620 con 0x55b96af4b3f0 2024-03-23T23:09:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.165+0000 7fedecc00640 1 -- 172.21.15.12:0/808479647 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedfc033bc0 con 0x55b96af4b3f0 2024-03-23T23:09:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.165+0000 7fedecc00640 1 -- 172.21.15.12:0/808479647 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fedfc004350 con 0x55b96af4b3f0 2024-03-23T23:09:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.165+0000 7fee0ca2f740 10 client.27433 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.165+0000 7fee0ca2f740 10 client.27433 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.165+0000 7fee0ca2f740 20 client.27433 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.165+0000 7fee0ca2f740 1 --2- 172.21.15.12:0/808479647 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b96af60fe0 0x55b96af633d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.165+0000 7fee0ca2f740 4 client.27433 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:09:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.165+0000 7fee0ca2f740 1 -- 172.21.15.12:0/808479647 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b96aebdd70 con 0x55b96af60fe0 2024-03-23T23:09:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.165+0000 7fee02000640 1 --2- 172.21.15.12:0/808479647 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b96af60fe0 0x55b96af633d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.166+0000 7fee02000640 1 --2- 172.21.15.12:0/808479647 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b96af60fe0 0x55b96af633d0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1490e0b414df326 server_cookie=2676a5c5552064b2 in_seq=0 out_seq=0 2024-03-23T23:09:27.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.166+0000 7fedecc00640 10 client.27433 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.167+0000 7fedecc00640 1 -- 172.21.15.12:0/808479647 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b96aebdd70 con 0x55b96af60fe0 2024-03-23T23:09:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.167+0000 7fedecc00640 10 client.27433 handle_command_reply: tid=0 2024-03-23T23:09:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.167+0000 7fee0ca2f740 1 client.27433 shutdown 2024-03-23T23:09:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.167+0000 7fee0ca2f740 1 -- 172.21.15.12:0/808479647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af4b3f0 msgr2=0x55b96af55570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.167+0000 7fee0ca2f740 1 --2- 172.21.15.12:0/808479647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af4b3f0 0x55b96af55570 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fedfc034fd0 tx=0x7fedfc004a40 comp rx=0 tx=0).stop 2024-03-23T23:09:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.167+0000 7fee0ca2f740 1 -- 172.21.15.12:0/808479647 shutdown_connections 2024-03-23T23:09:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.167+0000 7fee0ca2f740 1 --2- 172.21.15.12:0/808479647 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b96af60fe0 0x55b96af633d0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.167+0000 7fee0ca2f740 1 --2- 172.21.15.12:0/808479647 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b96af50f50 0x55b96af5caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.167+0000 7fee0ca2f740 1 --2- 172.21.15.12:0/808479647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b96af4bde0 0x55b96af55ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.167+0000 7fedecc00640 0 client.27433 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.167+0000 7fee0ca2f740 1 --2- 172.21.15.12:0/808479647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af4b3f0 0x55b96af55570 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.167+0000 7fee0ca2f740 1 -- 172.21.15.12:0/808479647 >> 172.21.15.12:0/808479647 conn(0x55b96af46960 msgr2=0x55b96af45b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.167+0000 7fee0ca2f740 1 -- 172.21.15.12:0/808479647 shutdown_connections 2024-03-23T23:09:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.168+0000 7fee0ca2f740 1 -- 172.21.15.12:0/808479647 wait complete. 2024-03-23T23:09:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.168+0000 7fee0ca2f740 20 client.27433 trim_cache size 0 max 16384 2024-03-23T23:09:27.183 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.183+0000 7fee0ca2f740 1 Processor -- start 2024-03-23T23:09:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.183+0000 7fee0ca2f740 1 -- start start 2024-03-23T23:09:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.183+0000 7fee0ca2f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b96af4b3f0 0x55b96af55a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.183+0000 7fee0ca2f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af4bde0 0x55b96af5ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.184+0000 7fee0ca2f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b96af50f50 0x55b96af416a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.184+0000 7fee0ca2f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee041acf80 con 0x55b96af50f50 2024-03-23T23:09:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.184+0000 7fee0ca2f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee041aaa80 con 0x55b96af4b3f0 2024-03-23T23:09:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.184+0000 7fee0ca2f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee041a8b20 con 0x55b96af4bde0 2024-03-23T23:09:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.184+0000 7fee02000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af4bde0 0x55b96af5ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.184+0000 7fee02000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af4bde0 0x55b96af5ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:48640/0 (socket says 172.21.15.12:48640) 2024-03-23T23:09:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.184+0000 7fee02000640 1 -- 172.21.15.12:0/1392323365 learned_addr learned my addr 172.21.15.12:0/1392323365 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.184+0000 7fee03400640 1 --2- 172.21.15.12:0/1392323365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b96af4b3f0 0x55b96af55a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.184+0000 7fee03400640 1 --2- 172.21.15.12:0/1392323365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b96af50f50 0x55b96af416a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:27.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.184+0000 7fee02000640 1 -- 172.21.15.12:0/1392323365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b96af4b3f0 msgr2=0x55b96af55a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:27.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.184+0000 7fee02000640 1 --2- 172.21.15.12:0/1392323365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b96af4b3f0 0x55b96af55a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.184+0000 7fee02000640 1 -- 172.21.15.12:0/1392323365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b96af50f50 msgr2=0x55b96af416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:27.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.184+0000 7fee02000640 1 --2- 172.21.15.12:0/1392323365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b96af50f50 0x55b96af416a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.184+0000 7fee02000640 1 -- 172.21.15.12:0/1392323365 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b96af54600 con 0x55b96af4bde0 2024-03-23T23:09:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.184+0000 7fee03400640 1 --2- 172.21.15.12:0/1392323365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b96af50f50 0x55b96af416a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:09:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.185+0000 7fee02000640 1 --2- 172.21.15.12:0/1392323365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af4bde0 0x55b96af5ca80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x55b96af478d0 tx=0x7fedf000de70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c623391f4c6cbbb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.185+0000 7fedef400640 1 -- 172.21.15.12:0/1392323365 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf001cdd0 con 0x55b96af4bde0 2024-03-23T23:09:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.185+0000 7fedef400640 1 -- 172.21.15.12:0/1392323365 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedf0005500 con 0x55b96af4bde0 2024-03-23T23:09:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.185+0000 7fedef400640 1 -- 172.21.15.12:0/1392323365 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf0003850 con 0x55b96af4bde0 2024-03-23T23:09:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.185+0000 7fee0ca2f740 1 -- 172.21.15.12:0/1392323365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af4bde0 msgr2=0x55b96af5ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.185+0000 7fee0ca2f740 1 --2- 172.21.15.12:0/1392323365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af4bde0 0x55b96af5ca80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x55b96af478d0 tx=0x7fedf000de70 comp rx=0 tx=0).stop 2024-03-23T23:09:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.185+0000 7fee0ca2f740 1 -- 172.21.15.12:0/1392323365 shutdown_connections 2024-03-23T23:09:27.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.185+0000 7fee0ca2f740 1 --2- 172.21.15.12:0/1392323365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b96af50f50 0x55b96af416a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.185+0000 7fee0ca2f740 1 --2- 172.21.15.12:0/1392323365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af4bde0 0x55b96af5ca80 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.185+0000 7fee0ca2f740 1 --2- 172.21.15.12:0/1392323365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b96af4b3f0 0x55b96af55a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.185+0000 7fee0ca2f740 1 -- 172.21.15.12:0/1392323365 >> 172.21.15.12:0/1392323365 conn(0x55b96af46960 msgr2=0x55b96af46d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:27.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.186+0000 7fee0ca2f740 1 -- 172.21.15.12:0/1392323365 shutdown_connections 2024-03-23T23:09:27.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.186+0000 7fee0ca2f740 1 -- 172.21.15.12:0/1392323365 wait complete. 2024-03-23T23:09:27.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.186+0000 7fee0ca2f740 1 -- start start 2024-03-23T23:09:27.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.186+0000 7fee0ca2f740 1 Processor -- start 2024-03-23T23:09:27.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.187+0000 7fee0ca2f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b96af4b3f0 0x55b96af5a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:27.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.187+0000 7fee0ca2f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b96af4bde0 0x55b96af5ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:27.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.187+0000 7fee0ca2f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af50f50 0x55b96af5bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:27.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.187+0000 7fee0ca2f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee041a9120 con 0x55b96af4b3f0 2024-03-23T23:09:27.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.187+0000 7fee0ca2f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee041aa420 con 0x55b96af4bde0 2024-03-23T23:09:27.191 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.187+0000 7fee0ca2f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee041a9760 con 0x55b96af50f50 2024-03-23T23:09:27.192 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.187+0000 7fee0ca2f740 10 client.0 _ll_register_callbacks cb 0x7fedf40d26c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:27.192 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.187+0000 7fee02000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b96af4bde0 0x55b96af5ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:27.192 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.187+0000 7fee02000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b96af4bde0 0x55b96af5ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:39180/0 (socket says 172.21.15.12:39180) 2024-03-23T23:09:27.192 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.187+0000 7fee02000640 1 -- 172.21.15.12:0/2645749982 learned_addr learned my addr 172.21.15.12:0/2645749982 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:27.192 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.187+0000 7fee03400640 1 --2- 172.21.15.12:0/2645749982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b96af4b3f0 0x55b96af5a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:27.192 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.187+0000 7fee03400640 1 --2- 172.21.15.12:0/2645749982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af50f50 0x55b96af5bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:27.192 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.187+0000 7fee02000640 1 -- 172.21.15.12:0/2645749982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af50f50 msgr2=0x55b96af5bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:27.192 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.187+0000 7fee02000640 1 --2- 172.21.15.12:0/2645749982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af50f50 0x55b96af5bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.192 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.187+0000 7fee02000640 1 -- 172.21.15.12:0/2645749982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b96af4b3f0 msgr2=0x55b96af5a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:27.192 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.187+0000 7fee02000640 1 --2- 172.21.15.12:0/2645749982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b96af4b3f0 0x55b96af5a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.192 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.187+0000 7fee02000640 1 -- 172.21.15.12:0/2645749982 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b96af5c1f0 con 0x55b96af4bde0 2024-03-23T23:09:27.192 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.187+0000 7fee03400640 1 --2- 172.21.15.12:0/2645749982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af50f50 0x55b96af5bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:09:27.192 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.188+0000 7fee02000640 1 --2- 172.21.15.12:0/2645749982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b96af4bde0 0x55b96af5ac40 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fedf0000c00 tx=0x7fedf001cbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5bf62b6f8d9fc1c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:27.192 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.188+0000 7fedecc00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:09:27.193 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.188+0000 7fedecc00640 1 -- 172.21.15.12:0/2645749982 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf0005000 con 0x55b96af4bde0 2024-03-23T23:09:27.193 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.188+0000 7fedecc00640 1 -- 172.21.15.12:0/2645749982 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fedf00051a0 con 0x55b96af4bde0 2024-03-23T23:09:27.193 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.188+0000 7fee0ca2f740 1 -- 172.21.15.12:0/2645749982 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fedf0005500 con 0x55b96af4bde0 2024-03-23T23:09:27.193 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.188+0000 7fedecc00640 1 -- 172.21.15.12:0/2645749982 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedf00046c0 con 0x55b96af4bde0 2024-03-23T23:09:27.193 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.188+0000 7fee0ca2f740 10 client.27470 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:27.193 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.188+0000 7fee0ca2f740 1 -- 172.21.15.12:0/2645749982 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55b96afbcd30 con 0x55b96af4bde0 2024-03-23T23:09:27.193 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.188+0000 7fedecc00640 1 -- 172.21.15.12:0/2645749982 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedf0005500 con 0x55b96af4bde0 2024-03-23T23:09:27.193 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.189+0000 7fedecc00640 1 -- 172.21.15.12:0/2645749982 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fedf0025440 con 0x55b96af4bde0 2024-03-23T23:09:27.193 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.189+0000 7fee0ca2f740 10 client.27470 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:27.193 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.189+0000 7fee0ca2f740 10 client.27470 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:27.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.189+0000 7fee0ca2f740 20 client.27470 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:27.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.189+0000 7fee0ca2f740 1 --2- 172.21.15.12:0/2645749982 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b96afbd560 0x55b96afbf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:27.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.189+0000 7fee0ca2f740 4 client.27470 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:09:27.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.189+0000 7fee0ca2f740 1 -- 172.21.15.12:0/2645749982 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b96aeb80c0 con 0x55b96afbd560 2024-03-23T23:09:27.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.189+0000 7fee03400640 1 --2- 172.21.15.12:0/2645749982 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b96afbd560 0x55b96afbf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:27.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.189+0000 7fee03400640 1 --2- 172.21.15.12:0/2645749982 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b96afbd560 0x55b96afbf980 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a301d26d0d3cf33 server_cookie=6f8dbab8c5865f26 in_seq=0 out_seq=0 2024-03-23T23:09:27.194 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.189+0000 7fedecc00640 10 client.27470 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.203+0000 7fedecc00640 1 -- 172.21.15.12:0/2645749982 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b96aeb80c0 con 0x55b96afbd560 2024-03-23T23:09:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.203+0000 7fedecc00640 10 client.27470 handle_command_reply: tid=0 2024-03-23T23:09:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.203+0000 7fee0ca2f740 1 client.27470 shutdown 2024-03-23T23:09:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.203+0000 7fee0ca2f740 1 -- 172.21.15.12:0/2645749982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b96af4bde0 msgr2=0x55b96af5ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.203+0000 7fee0ca2f740 1 --2- 172.21.15.12:0/2645749982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b96af4bde0 0x55b96af5ac40 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fedf0000c00 tx=0x7fedf001cbd0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.203+0000 7fee0ca2f740 1 -- 172.21.15.12:0/2645749982 shutdown_connections 2024-03-23T23:09:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.203+0000 7fee0ca2f740 1 --2- 172.21.15.12:0/2645749982 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b96afbd560 0x55b96afbf980 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.203+0000 7fee0ca2f740 1 --2- 172.21.15.12:0/2645749982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b96af50f50 0x55b96af5bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.203+0000 7fee0ca2f740 1 --2- 172.21.15.12:0/2645749982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b96af4bde0 0x55b96af5ac40 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.203+0000 7fee0ca2f740 1 --2- 172.21.15.12:0/2645749982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b96af4b3f0 0x55b96af5a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.203+0000 7fedecc00640 0 client.27470 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.203+0000 7fee0ca2f740 1 -- 172.21.15.12:0/2645749982 >> 172.21.15.12:0/2645749982 conn(0x55b96af46960 msgr2=0x55b96af452e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.203+0000 7fee0ca2f740 1 -- 172.21.15.12:0/2645749982 shutdown_connections 2024-03-23T23:09:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.203+0000 7fee0ca2f740 1 -- 172.21.15.12:0/2645749982 wait complete. 2024-03-23T23:09:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.203+0000 7fee0ca2f740 20 client.27470 trim_cache size 0 max 16384 2024-03-23T23:09:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:09:27.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:09:27.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:09:27.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:09:27.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.203+0000 7fee03e00640 1 -- 172.21.15.12:0/2389792928 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fede80a2870 msgr2=0x7fede80a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:27.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.203+0000 7fee03e00640 1 --2- 172.21.15.12:0/2389792928 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fede80a2870 0x7fede80a4d30 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x55b96af4c1e0 tx=0x7fedf40d27f0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.203+0000 7fee03e00640 1 -- 172.21.15.12:0/2389792928 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee04103720 msgr2=0x7fee04199a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:27.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.203+0000 7fee03e00640 1 --2- 172.21.15.12:0/2389792928 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee04103720 0x7fee04199a70 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fedf400e9f0 tx=0x7fedf400eeb0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.203+0000 7fee03e00640 1 -- 172.21.15.12:0/2389792928 shutdown_connections 2024-03-23T23:09:27.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.203+0000 7fee03e00640 1 --2- 172.21.15.12:0/2389792928 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fede80a2870 0x7fede80a4d30 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.203+0000 7fee03e00640 1 --2- 172.21.15.12:0/2389792928 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee041052d0 0x7fee041a1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.203+0000 7fee03e00640 1 --2- 172.21.15.12:0/2389792928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee04104910 0x7fee04199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.203+0000 7fee03e00640 1 --2- 172.21.15.12:0/2389792928 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee04103720 0x7fee04199a70 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:27.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.203+0000 7fee03e00640 1 -- 172.21.15.12:0/2389792928 >> 172.21.15.12:0/2389792928 conn(0x7fee040fecb0 msgr2=0x7fee04107b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:27.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.204+0000 7fee03e00640 1 -- 172.21.15.12:0/2389792928 shutdown_connections 2024-03-23T23:09:27.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:27.204+0000 7fee03e00640 1 -- 172.21.15.12:0/2389792928 wait complete. 2024-03-23T23:09:29.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:28 smithi012 ceph-mon[31695]: pgmap v620: 97 pgs: 97 active+clean; 1007 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:09:29.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:28 smithi173 ceph-mon[36116]: pgmap v620: 97 pgs: 97 active+clean; 1007 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:09:29.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:28 smithi203 ceph-mon[36693]: pgmap v620: 97 pgs: 97 active+clean; 1007 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:09:31.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:30 smithi012 ceph-mon[31695]: pgmap v621: 97 pgs: 97 active+clean; 1007 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:09:31.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:30 smithi173 ceph-mon[36116]: pgmap v621: 97 pgs: 97 active+clean; 1007 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:09:31.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:30 smithi203 ceph-mon[36693]: pgmap v621: 97 pgs: 97 active+clean; 1007 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:09:33.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:32 smithi012 ceph-mon[31695]: pgmap v622: 97 pgs: 97 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:09:33.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:32 smithi173 ceph-mon[36116]: pgmap v622: 97 pgs: 97 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:09:33.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:32 smithi203 ceph-mon[36693]: pgmap v622: 97 pgs: 97 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:09:34.959 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:34 smithi203 ceph-mon[36693]: pgmap v623: 97 pgs: 97 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:09:35.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:34 smithi012 ceph-mon[31695]: pgmap v623: 97 pgs: 97 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:09:35.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:34 smithi173 ceph-mon[36116]: pgmap v623: 97 pgs: 97 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:09:37.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:36 smithi012 ceph-mon[31695]: pgmap v624: 97 pgs: 97 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:09:37.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:36 smithi173 ceph-mon[36116]: pgmap v624: 97 pgs: 97 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:09:37.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:36 smithi203 ceph-mon[36693]: pgmap v624: 97 pgs: 97 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:09:39.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:38 smithi012 ceph-mon[31695]: pgmap v625: 97 pgs: 97 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:09:39.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:38 smithi173 ceph-mon[36116]: pgmap v625: 97 pgs: 97 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:09:39.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:38 smithi203 ceph-mon[36693]: pgmap v625: 97 pgs: 97 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:09:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.355+0000 7fe6b3200640 1 -- 172.21.15.12:0/3018076958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6b410dae0 msgr2=0x7fe6b410fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.355+0000 7fe6b3200640 1 --2- 172.21.15.12:0/3018076958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6b410dae0 0x7fe6b410fed0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac009be0 tx=0x7fe6ac02bef0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.355+0000 7fe6b3200640 1 -- 172.21.15.12:0/3018076958 shutdown_connections 2024-03-23T23:09:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.355+0000 7fe6b3200640 1 --2- 172.21.15.12:0/3018076958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6b410dae0 0x7fe6b410fed0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.355+0000 7fe6b3200640 1 --2- 172.21.15.12:0/3018076958 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6b4074100 0x7fe6b4074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.355+0000 7fe6b3200640 1 --2- 172.21.15.12:0/3018076958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6b40756e0 0x7fe6b4073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.355+0000 7fe6b3200640 1 -- 172.21.15.12:0/3018076958 >> 172.21.15.12:0/3018076958 conn(0x7fe6b40fca80 msgr2=0x7fe6b40feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.355+0000 7fe6b3200640 1 -- 172.21.15.12:0/3018076958 shutdown_connections 2024-03-23T23:09:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.355+0000 7fe6b3200640 1 -- 172.21.15.12:0/3018076958 wait complete. 2024-03-23T23:09:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.356+0000 7fe6b3200640 1 Processor -- start 2024-03-23T23:09:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.356+0000 7fe6b3200640 1 -- start start 2024-03-23T23:09:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.356+0000 7fe6b3200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6b4074100 0x7fe6b419ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.356+0000 7fe6b3200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6b40756e0 0x7fe6b419e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.356+0000 7fe6b3200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6b410dae0 0x7fe6b41a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.356+0000 7fe6b3200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b41137d0 con 0x7fe6b40756e0 2024-03-23T23:09:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.356+0000 7fe6b3200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b4113650 con 0x7fe6b410dae0 2024-03-23T23:09:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.356+0000 7fe6b3200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b4113950 con 0x7fe6b4074100 2024-03-23T23:09:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.357+0000 7fe6b1400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6b40756e0 0x7fe6b419e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.357+0000 7fe6b1400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6b40756e0 0x7fe6b419e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40518/0 (socket says 172.21.15.12:40518) 2024-03-23T23:09:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.357+0000 7fe6b1400640 1 -- 172.21.15.12:0/508039833 learned_addr learned my addr 172.21.15.12:0/508039833 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.357+0000 7fe6b2800640 1 --2- 172.21.15.12:0/508039833 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6b410dae0 0x7fe6b41a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.357+0000 7fe6b1e00640 1 --2- 172.21.15.12:0/508039833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6b4074100 0x7fe6b419ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.357+0000 7fe6b1400640 1 -- 172.21.15.12:0/508039833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6b4074100 msgr2=0x7fe6b419ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.357+0000 7fe6b1400640 1 --2- 172.21.15.12:0/508039833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6b4074100 0x7fe6b419ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.357+0000 7fe6b1400640 1 -- 172.21.15.12:0/508039833 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6b410dae0 msgr2=0x7fe6b41a5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.357+0000 7fe6b1400640 1 --2- 172.21.15.12:0/508039833 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6b410dae0 0x7fe6b41a5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.357+0000 7fe6b1400640 1 -- 172.21.15.12:0/508039833 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b41a5980 con 0x7fe6b40756e0 2024-03-23T23:09:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.357+0000 7fe6b1400640 1 --2- 172.21.15.12:0/508039833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6b40756e0 0x7fe6b419e2f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe6a000e9c0 tx=0x7fe6a000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.358+0000 7fe6aaa00640 1 -- 172.21.15.12:0/508039833 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a000cde0 con 0x7fe6b40756e0 2024-03-23T23:09:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.358+0000 7fe6b3200640 1 -- 172.21.15.12:0/508039833 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b41a5c70 con 0x7fe6b40756e0 2024-03-23T23:09:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.358+0000 7fe6aaa00640 1 -- 172.21.15.12:0/508039833 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6a00035f0 con 0x7fe6b40756e0 2024-03-23T23:09:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.358+0000 7fe6aaa00640 1 -- 172.21.15.12:0/508039833 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a0018630 con 0x7fe6b40756e0 2024-03-23T23:09:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.358+0000 7fe6b3200640 1 -- 172.21.15.12:0/508039833 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b41a61c0 con 0x7fe6b40756e0 2024-03-23T23:09:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.361+0000 7fe6bbfd8740 1 Processor -- start 2024-03-23T23:09:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.361+0000 7fe6bbfd8740 1 -- start start 2024-03-23T23:09:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.362+0000 7fe6bbfd8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b1ee53f0 0x55f7b1ee5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.362+0000 7fe6bbfd8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f7b1ee5de0 0x55f7b1eea8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.362+0000 7fe6bbfd8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1eeaf50 0x55f7b1eed3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.362+0000 7fe6bbfd8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b1ea83b0 con 0x55f7b1ee5de0 2024-03-23T23:09:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.362+0000 7fe6bbfd8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b1e4a830 con 0x55f7b1eeaf50 2024-03-23T23:09:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.362+0000 7fe6bbfd8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b1df35d0 con 0x55f7b1ee53f0 2024-03-23T23:09:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.362+0000 7fe6aaa00640 1 -- 172.21.15.12:0/508039833 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe6a00037a0 con 0x7fe6b40756e0 2024-03-23T23:09:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.362+0000 7fe6b2800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b1ee53f0 0x55f7b1ee5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.362+0000 7fe6b2800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b1ee53f0 0x55f7b1ee5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35248/0 (socket says 172.21.15.12:35248) 2024-03-23T23:09:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.362+0000 7fe6b2800640 1 -- 172.21.15.12:0/755397564 learned_addr learned my addr 172.21.15.12:0/755397564 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.362+0000 7fe6b1e00640 1 --2- 172.21.15.12:0/755397564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f7b1ee5de0 0x55f7b1eea8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.362+0000 7fe6b1400640 1 --2- 172.21.15.12:0/755397564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1eeaf50 0x55f7b1eed3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.362+0000 7fe6b2800640 1 -- 172.21.15.12:0/755397564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1eeaf50 msgr2=0x55f7b1eed3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.362+0000 7fe6b2800640 1 --2- 172.21.15.12:0/755397564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1eeaf50 0x55f7b1eed3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.362+0000 7fe6b2800640 1 -- 172.21.15.12:0/755397564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f7b1ee5de0 msgr2=0x55f7b1eea8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.362+0000 7fe6b2800640 1 --2- 172.21.15.12:0/755397564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f7b1ee5de0 0x55f7b1eea8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.362+0000 7fe6b2800640 1 -- 172.21.15.12:0/755397564 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f7b1e0b2c0 con 0x55f7b1ee53f0 2024-03-23T23:09:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.362+0000 7fe6b1400640 1 --2- 172.21.15.12:0/755397564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1eeaf50 0x55f7b1eed3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:09:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.363+0000 7fe6b2800640 1 --2- 172.21.15.12:0/755397564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b1ee53f0 0x55f7b1ee5810 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac009be0 tx=0x7fe6ac009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c444c2126b32a903 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.363+0000 7fe69f400640 1 -- 172.21.15.12:0/755397564 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac043030 con 0x55f7b1ee53f0 2024-03-23T23:09:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.363+0000 7fe69f400640 1 -- 172.21.15.12:0/755397564 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6ac033c40 con 0x55f7b1ee53f0 2024-03-23T23:09:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.363+0000 7fe69f400640 1 -- 172.21.15.12:0/755397564 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac02c890 con 0x55f7b1ee53f0 2024-03-23T23:09:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.364+0000 7fe6aaa00640 1 --2- 172.21.15.12:0/508039833 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe6980a27f0 0x7fe6980a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.364+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/755397564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b1ee53f0 msgr2=0x55f7b1ee5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.364+0000 7fe6bbfd8740 1 --2- 172.21.15.12:0/755397564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b1ee53f0 0x55f7b1ee5810 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac009be0 tx=0x7fe6ac009310 comp rx=0 tx=0).stop 2024-03-23T23:09:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.364+0000 7fe6aaa00640 1 -- 172.21.15.12:0/508039833 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe6a00cece0 con 0x7fe6b40756e0 2024-03-23T23:09:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.364+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/755397564 shutdown_connections 2024-03-23T23:09:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.364+0000 7fe6bbfd8740 1 --2- 172.21.15.12:0/755397564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1eeaf50 0x55f7b1eed3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.364+0000 7fe6b1e00640 1 --2- 172.21.15.12:0/508039833 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe6980a27f0 0x7fe6980a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.364+0000 7fe6bbfd8740 1 --2- 172.21.15.12:0/755397564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f7b1ee5de0 0x55f7b1eea8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.364+0000 7fe6bbfd8740 1 --2- 172.21.15.12:0/755397564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b1ee53f0 0x55f7b1ee5810 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.364+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/755397564 >> 172.21.15.12:0/755397564 conn(0x55f7b1ee0960 msgr2=0x55f7b1ee0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.364+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/755397564 shutdown_connections 2024-03-23T23:09:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.364+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/755397564 wait complete. 2024-03-23T23:09:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.365+0000 7fe6bbfd8740 1 -- start start 2024-03-23T23:09:40.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.365+0000 7fe6b1e00640 1 --2- 172.21.15.12:0/508039833 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe6980a27f0 0x7fe6980a4cb0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x55f7b1ee18d0 tx=0x7fe6a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:40.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.365+0000 7fe6bbfd8740 1 Processor -- start 2024-03-23T23:09:40.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.365+0000 7fe6bbfd8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b1ee53f0 0x55f7b1eef5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:40.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.365+0000 7fe6bbfd8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f7b1ee5de0 0x55f7b1eefb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:40.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.365+0000 7fe6bbfd8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1eeaf50 0x55f7b1ef6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:40.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.365+0000 7fe6bbfd8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b1e06a20 con 0x55f7b1ee5de0 2024-03-23T23:09:40.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.365+0000 7fe6bbfd8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b1e468d0 con 0x55f7b1eeaf50 2024-03-23T23:09:40.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.365+0000 7fe6bbfd8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b1eedea0 con 0x55f7b1ee53f0 2024-03-23T23:09:40.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.365+0000 7fe6bbfd8740 10 client.0 _ll_register_callbacks cb 0x55f7b1e431c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:40.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.366+0000 7fe6b1e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b1ee53f0 0x55f7b1eef5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:40.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.366+0000 7fe6b1e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b1ee53f0 0x55f7b1eef5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35264/0 (socket says 172.21.15.12:35264) 2024-03-23T23:09:40.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.366+0000 7fe6b2800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1eeaf50 0x55f7b1ef6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:40.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.366+0000 7fe6b1e00640 1 -- 172.21.15.12:0/328755528 learned_addr learned my addr 172.21.15.12:0/328755528 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:40.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.366+0000 7fe6b1400640 1 --2- 172.21.15.12:0/328755528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f7b1ee5de0 0x55f7b1eefb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:40.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.366+0000 7fe6b1e00640 1 -- 172.21.15.12:0/328755528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1eeaf50 msgr2=0x55f7b1ef6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:40.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.366+0000 7fe6b1e00640 1 --2- 172.21.15.12:0/328755528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1eeaf50 0x55f7b1ef6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.366+0000 7fe6b1e00640 1 -- 172.21.15.12:0/328755528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f7b1ee5de0 msgr2=0x55f7b1eefb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:40.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.366+0000 7fe6b1e00640 1 --2- 172.21.15.12:0/328755528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f7b1ee5de0 0x55f7b1eefb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.366+0000 7fe6b1e00640 1 -- 172.21.15.12:0/328755528 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f7b1b5f210 con 0x55f7b1ee53f0 2024-03-23T23:09:40.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.366+0000 7fe6b1400640 1 --2- 172.21.15.12:0/328755528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f7b1ee5de0 0x55f7b1eefb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:40.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.366+0000 7fe6b2800640 1 --2- 172.21.15.12:0/328755528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1eeaf50 0x55f7b1ef6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:40.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.367+0000 7fe6b1e00640 1 --2- 172.21.15.12:0/328755528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b1ee53f0 0x55f7b1eef5e0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe6a401e9b0 tx=0x7fe6a4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eedfeba6a25418ca server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:40.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.367+0000 7fe693e00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:09:40.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.367+0000 7fe693e00640 1 -- 172.21.15.12:0/328755528 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a402e060 con 0x55f7b1ee53f0 2024-03-23T23:09:40.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.367+0000 7fe693e00640 1 -- 172.21.15.12:0/328755528 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe6a4018030 con 0x55f7b1ee53f0 2024-03-23T23:09:40.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.367+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/328755528 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe6ac033c40 con 0x55f7b1ee53f0 2024-03-23T23:09:40.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.367+0000 7fe693e00640 1 -- 172.21.15.12:0/328755528 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6a4029420 con 0x55f7b1ee53f0 2024-03-23T23:09:40.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.367+0000 7fe6bbfd8740 10 client.27448 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.367+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/328755528 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f7b1c2b620 con 0x55f7b1ee53f0 2024-03-23T23:09:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.368+0000 7fe693e00640 1 -- 172.21.15.12:0/328755528 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6ac033c40 con 0x55f7b1ee53f0 2024-03-23T23:09:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.368+0000 7fe693e00640 1 -- 172.21.15.12:0/328755528 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe6a400f450 con 0x55f7b1ee53f0 2024-03-23T23:09:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.368+0000 7fe6bbfd8740 10 client.27448 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.368+0000 7fe6bbfd8740 10 client.27448 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.368+0000 7fe6bbfd8740 20 client.27448 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.368+0000 7fe6bbfd8740 1 --2- 172.21.15.12:0/328755528 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f7b1efb480 0x55f7b1efd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.368+0000 7fe6bbfd8740 4 client.27448 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:09:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.368+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/328755528 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f7b1e57d70 con 0x55f7b1efb480 2024-03-23T23:09:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.368+0000 7fe6b1400640 1 --2- 172.21.15.12:0/328755528 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f7b1efb480 0x55f7b1efd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.369+0000 7fe6b1400640 1 --2- 172.21.15.12:0/328755528 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f7b1efb480 0x55f7b1efd890 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13aed4e9277d1c31 server_cookie=28ec224029c953c6 in_seq=0 out_seq=0 2024-03-23T23:09:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.369+0000 7fe693e00640 10 client.27448 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.369+0000 7fe693e00640 1 -- 172.21.15.12:0/328755528 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f7b1e57d70 con 0x55f7b1efb480 2024-03-23T23:09:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.369+0000 7fe693e00640 10 client.27448 handle_command_reply: tid=0 2024-03-23T23:09:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.369+0000 7fe6bbfd8740 1 client.27448 shutdown 2024-03-23T23:09:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.370+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/328755528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b1ee53f0 msgr2=0x55f7b1eef5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.370+0000 7fe6bbfd8740 1 --2- 172.21.15.12:0/328755528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b1ee53f0 0x55f7b1eef5e0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe6a401e9b0 tx=0x7fe6a4004730 comp rx=0 tx=0).stop 2024-03-23T23:09:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.370+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/328755528 shutdown_connections 2024-03-23T23:09:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.370+0000 7fe6bbfd8740 1 --2- 172.21.15.12:0/328755528 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f7b1efb480 0x55f7b1efd890 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.370+0000 7fe6bbfd8740 1 --2- 172.21.15.12:0/328755528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1eeaf50 0x55f7b1ef6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.370+0000 7fe6bbfd8740 1 --2- 172.21.15.12:0/328755528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f7b1ee5de0 0x55f7b1eefb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.370+0000 7fe6bbfd8740 1 --2- 172.21.15.12:0/328755528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b1ee53f0 0x55f7b1eef5e0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.370+0000 7fe693e00640 0 client.27448 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.370+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/328755528 >> 172.21.15.12:0/328755528 conn(0x55f7b1ee0960 msgr2=0x55f7b1edfb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.370+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/328755528 shutdown_connections 2024-03-23T23:09:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.370+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/328755528 wait complete. 2024-03-23T23:09:40.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.370+0000 7fe6bbfd8740 20 client.27448 trim_cache size 0 max 16384 2024-03-23T23:09:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.385+0000 7fe6bbfd8740 1 Processor -- start 2024-03-23T23:09:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.385+0000 7fe6bbfd8740 1 -- start start 2024-03-23T23:09:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.386+0000 7fe6bbfd8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b1ee53f0 0x55f7b1eefa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.386+0000 7fe6bbfd8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f7b1ee5de0 0x55f7b1ef6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.386+0000 7fe6bbfd8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1eeaf50 0x55f7b1ee0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.386+0000 7fe6bbfd8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b41b1260 con 0x55f7b1ee5de0 2024-03-23T23:09:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.386+0000 7fe6bbfd8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b41aed60 con 0x55f7b1eeaf50 2024-03-23T23:09:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.386+0000 7fe6bbfd8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b41ace00 con 0x55f7b1ee53f0 2024-03-23T23:09:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.386+0000 7fe6b2800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1eeaf50 0x55f7b1ee0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.386+0000 7fe6b1400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f7b1ee5de0 0x55f7b1ef6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.386+0000 7fe6b2800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1eeaf50 0x55f7b1ee0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60938/0 (socket says 172.21.15.12:60938) 2024-03-23T23:09:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.386+0000 7fe6b2800640 1 -- 172.21.15.12:0/4085014155 learned_addr learned my addr 172.21.15.12:0/4085014155 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.386+0000 7fe6b1e00640 1 --2- 172.21.15.12:0/4085014155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b1ee53f0 0x55f7b1eefa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:40.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.386+0000 7fe6b2800640 1 -- 172.21.15.12:0/4085014155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b1ee53f0 msgr2=0x55f7b1eefa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.386+0000 7fe6b2800640 1 --2- 172.21.15.12:0/4085014155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b1ee53f0 0x55f7b1eefa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.386+0000 7fe6b2800640 1 -- 172.21.15.12:0/4085014155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f7b1ee5de0 msgr2=0x55f7b1ef6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.386+0000 7fe6b2800640 1 --2- 172.21.15.12:0/4085014155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f7b1ee5de0 0x55f7b1ef6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.386+0000 7fe6b2800640 1 -- 172.21.15.12:0/4085014155 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f7b1ee57d0 con 0x55f7b1eeaf50 2024-03-23T23:09:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.386+0000 7fe6b1400640 1 --2- 172.21.15.12:0/4085014155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f7b1ee5de0 0x55f7b1ef6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.387+0000 7fe6b2800640 1 --2- 172.21.15.12:0/4085014155 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1eeaf50 0x55f7b1ee0d40 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac009be0 tx=0x7fe6ac009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc4fddbbcd901ab2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.387+0000 7fe69ea00640 1 -- 172.21.15.12:0/4085014155 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac04c020 con 0x55f7b1eeaf50 2024-03-23T23:09:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.387+0000 7fe69ea00640 1 -- 172.21.15.12:0/4085014155 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6ac003e50 con 0x55f7b1eeaf50 2024-03-23T23:09:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.387+0000 7fe69ea00640 1 -- 172.21.15.12:0/4085014155 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac043040 con 0x55f7b1eeaf50 2024-03-23T23:09:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.387+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/4085014155 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1eeaf50 msgr2=0x55f7b1ee0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.387+0000 7fe6bbfd8740 1 --2- 172.21.15.12:0/4085014155 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1eeaf50 0x55f7b1ee0d40 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac009be0 tx=0x7fe6ac009500 comp rx=0 tx=0).stop 2024-03-23T23:09:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.388+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/4085014155 shutdown_connections 2024-03-23T23:09:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.388+0000 7fe6bbfd8740 1 --2- 172.21.15.12:0/4085014155 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1eeaf50 0x55f7b1ee0d40 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.388+0000 7fe6bbfd8740 1 --2- 172.21.15.12:0/4085014155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f7b1ee5de0 0x55f7b1ef6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.388+0000 7fe6bbfd8740 1 --2- 172.21.15.12:0/4085014155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b1ee53f0 0x55f7b1eefa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.388+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/4085014155 >> 172.21.15.12:0/4085014155 conn(0x55f7b1ee0960 msgr2=0x55f7b1ee6670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.388+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/4085014155 shutdown_connections 2024-03-23T23:09:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.388+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/4085014155 wait complete. 2024-03-23T23:09:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.388+0000 7fe6bbfd8740 1 -- start start 2024-03-23T23:09:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.388+0000 7fe6bbfd8740 1 Processor -- start 2024-03-23T23:09:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.389+0000 7fe6bbfd8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1ee53f0 0x55f7b1ef4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.389+0000 7fe6bbfd8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f7b1ee5de0 0x55f7b1ef4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.389+0000 7fe6bbfd8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b1eeaf50 0x55f7b1ef5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.389+0000 7fe6bbfd8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b41ad400 con 0x55f7b1ee5de0 2024-03-23T23:09:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.389+0000 7fe6bbfd8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b41ae700 con 0x55f7b1ee53f0 2024-03-23T23:09:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.389+0000 7fe6bbfd8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b41ada40 con 0x55f7b1eeaf50 2024-03-23T23:09:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.389+0000 7fe6bbfd8740 10 client.0 _ll_register_callbacks cb 0x7fe6a4009a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.389+0000 7fe6b1e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1ee53f0 0x55f7b1ef4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.389+0000 7fe6b1400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f7b1ee5de0 0x55f7b1ef4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.389+0000 7fe6b1e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1ee53f0 0x55f7b1ef4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60952/0 (socket says 172.21.15.12:60952) 2024-03-23T23:09:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.389+0000 7fe6b1e00640 1 -- 172.21.15.12:0/1487739173 learned_addr learned my addr 172.21.15.12:0/1487739173 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.389+0000 7fe6b2800640 1 --2- 172.21.15.12:0/1487739173 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b1eeaf50 0x55f7b1ef5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.389+0000 7fe6b1e00640 1 -- 172.21.15.12:0/1487739173 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b1eeaf50 msgr2=0x55f7b1ef5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.389+0000 7fe6b1e00640 1 --2- 172.21.15.12:0/1487739173 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b1eeaf50 0x55f7b1ef5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.389+0000 7fe6b1e00640 1 -- 172.21.15.12:0/1487739173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f7b1ee5de0 msgr2=0x55f7b1ef4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.389+0000 7fe6b1e00640 1 --2- 172.21.15.12:0/1487739173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f7b1ee5de0 0x55f7b1ef4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.389+0000 7fe6b1e00640 1 -- 172.21.15.12:0/1487739173 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f7b1ef5cf0 con 0x55f7b1ee53f0 2024-03-23T23:09:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.390+0000 7fe6b1e00640 1 --2- 172.21.15.12:0/1487739173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1ee53f0 0x55f7b1ef4200 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4005eb0 tx=0x7fe6a4009ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77f5815b4e2afa3e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.390+0000 7fe693e00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:09:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.390+0000 7fe693e00640 1 -- 172.21.15.12:0/1487739173 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a4039070 con 0x55f7b1ee53f0 2024-03-23T23:09:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.390+0000 7fe693e00640 1 -- 172.21.15.12:0/1487739173 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe6a400f3c0 con 0x55f7b1ee53f0 2024-03-23T23:09:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.390+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/1487739173 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe6ac003e50 con 0x55f7b1ee53f0 2024-03-23T23:09:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.390+0000 7fe693e00640 1 -- 172.21.15.12:0/1487739173 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6a400f5e0 con 0x55f7b1ee53f0 2024-03-23T23:09:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.390+0000 7fe6bbfd8740 10 client.27488 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.390+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/1487739173 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f7b1f56d60 con 0x55f7b1ee53f0 2024-03-23T23:09:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.390+0000 7fe693e00640 1 -- 172.21.15.12:0/1487739173 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6ac003e50 con 0x55f7b1ee53f0 2024-03-23T23:09:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.391+0000 7fe693e00640 1 -- 172.21.15.12:0/1487739173 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe6a4035450 con 0x55f7b1ee53f0 2024-03-23T23:09:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.391+0000 7fe6bbfd8740 10 client.27488 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.391+0000 7fe6bbfd8740 10 client.27488 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.391+0000 7fe6bbfd8740 20 client.27488 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.391+0000 7fe6bbfd8740 1 --2- 172.21.15.12:0/1487739173 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f7b1f575c0 0x55f7b1f599e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.391+0000 7fe6bbfd8740 4 client.27488 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:09:40.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.391+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/1487739173 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f7b1e520c0 con 0x55f7b1f575c0 2024-03-23T23:09:40.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.391+0000 7fe6b1400640 1 --2- 172.21.15.12:0/1487739173 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f7b1f575c0 0x55f7b1f599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:40.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.391+0000 7fe6b1400640 1 --2- 172.21.15.12:0/1487739173 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f7b1f575c0 0x55f7b1f599e0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6c372df384a09b5 server_cookie=7ca433410db31d88 in_seq=0 out_seq=0 2024-03-23T23:09:40.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.391+0000 7fe693e00640 10 client.27488 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.404+0000 7fe693e00640 1 -- 172.21.15.12:0/1487739173 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f7b1e520c0 con 0x55f7b1f575c0 2024-03-23T23:09:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.404+0000 7fe693e00640 10 client.27488 handle_command_reply: tid=0 2024-03-23T23:09:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.404+0000 7fe6bbfd8740 1 client.27488 shutdown 2024-03-23T23:09:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.404+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/1487739173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1ee53f0 msgr2=0x55f7b1ef4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.404+0000 7fe6bbfd8740 1 --2- 172.21.15.12:0/1487739173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1ee53f0 0x55f7b1ef4200 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4005eb0 tx=0x7fe6a4009ea0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.404+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/1487739173 shutdown_connections 2024-03-23T23:09:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.404+0000 7fe6bbfd8740 1 --2- 172.21.15.12:0/1487739173 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f7b1f575c0 0x55f7b1f599e0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.404+0000 7fe6bbfd8740 1 --2- 172.21.15.12:0/1487739173 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b1eeaf50 0x55f7b1ef5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.404+0000 7fe6bbfd8740 1 --2- 172.21.15.12:0/1487739173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f7b1ee5de0 0x55f7b1ef4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.404+0000 7fe6bbfd8740 1 --2- 172.21.15.12:0/1487739173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f7b1ee53f0 0x55f7b1ef4200 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.404+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/1487739173 >> 172.21.15.12:0/1487739173 conn(0x55f7b1ee0960 msgr2=0x55f7b1ee6640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.404+0000 7fe693e00640 0 client.27488 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.404+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/1487739173 shutdown_connections 2024-03-23T23:09:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.404+0000 7fe6bbfd8740 1 -- 172.21.15.12:0/1487739173 wait complete. 2024-03-23T23:09:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.404+0000 7fe6bbfd8740 20 client.27488 trim_cache size 0 max 16384 2024-03-23T23:09:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:09:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:09:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:09:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:09:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.405+0000 7fe6b3200640 1 -- 172.21.15.12:0/508039833 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe6980a27f0 msgr2=0x7fe6980a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.405+0000 7fe6b3200640 1 --2- 172.21.15.12:0/508039833 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe6980a27f0 0x7fe6980a4cb0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x55f7b1ee18d0 tx=0x7fe6a4005f90 comp rx=0 tx=0).stop 2024-03-23T23:09:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.405+0000 7fe6b3200640 1 -- 172.21.15.12:0/508039833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6b40756e0 msgr2=0x7fe6b419e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.405+0000 7fe6b3200640 1 --2- 172.21.15.12:0/508039833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6b40756e0 0x7fe6b419e2f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe6a000e9c0 tx=0x7fe6a000ee80 comp rx=0 tx=0).stop 2024-03-23T23:09:40.406 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.405+0000 7fe6b3200640 1 -- 172.21.15.12:0/508039833 shutdown_connections 2024-03-23T23:09:40.406 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.405+0000 7fe6b3200640 1 --2- 172.21.15.12:0/508039833 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe6980a27f0 0x7fe6980a4cb0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.406 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.405+0000 7fe6b3200640 1 --2- 172.21.15.12:0/508039833 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6b410dae0 0x7fe6b41a5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.406 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.405+0000 7fe6b3200640 1 --2- 172.21.15.12:0/508039833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6b40756e0 0x7fe6b419e2f0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.406 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.405+0000 7fe6b3200640 1 --2- 172.21.15.12:0/508039833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6b4074100 0x7fe6b419ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:40.406 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.405+0000 7fe6b3200640 1 -- 172.21.15.12:0/508039833 >> 172.21.15.12:0/508039833 conn(0x7fe6b40fca80 msgr2=0x7fe6b4100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:40.406 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.406+0000 7fe6b3200640 1 -- 172.21.15.12:0/508039833 shutdown_connections 2024-03-23T23:09:40.406 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:40.406+0000 7fe6b3200640 1 -- 172.21.15.12:0/508039833 wait complete. 2024-03-23T23:09:41.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:40 smithi012 ceph-mon[31695]: pgmap v626: 97 pgs: 97 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:09:41.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:40 smithi173 ceph-mon[36116]: pgmap v626: 97 pgs: 97 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:09:41.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:40 smithi203 ceph-mon[36693]: pgmap v626: 97 pgs: 97 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:09:43.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:42 smithi012 ceph-mon[31695]: pgmap v627: 97 pgs: 97 active+clean; 1.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:09:43.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:42 smithi173 ceph-mon[36116]: pgmap v627: 97 pgs: 97 active+clean; 1.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:09:43.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:42 smithi203 ceph-mon[36693]: pgmap v627: 97 pgs: 97 active+clean; 1.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:09:43.641 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:09:43.641 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:09:43.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.876+0000 7f1a3c800640 1 -- 172.21.15.12:0/2141357115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a3410dc80 msgr2=0x7f1a34110070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:43.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.876+0000 7f1a3c800640 1 --2- 172.21.15.12:0/2141357115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a3410dc80 0x7f1a34110070 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f1a30009cd0 tx=0x7f1a3002c0e0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.876+0000 7f1a3c800640 1 -- 172.21.15.12:0/2141357115 shutdown_connections 2024-03-23T23:09:43.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.876+0000 7f1a3c800640 1 --2- 172.21.15.12:0/2141357115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a3410dc80 0x7f1a34110070 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.876+0000 7f1a3c800640 1 --2- 172.21.15.12:0/2141357115 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a34074100 0x7f1a34074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.876+0000 7f1a3c800640 1 --2- 172.21.15.12:0/2141357115 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a340756e0 0x7f1a34073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.876+0000 7f1a3c800640 1 -- 172.21.15.12:0/2141357115 >> 172.21.15.12:0/2141357115 conn(0x7f1a340fcc40 msgr2=0x7f1a340ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:43.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.876+0000 7f1a3c800640 1 -- 172.21.15.12:0/2141357115 shutdown_connections 2024-03-23T23:09:43.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.876+0000 7f1a3c800640 1 -- 172.21.15.12:0/2141357115 wait complete. 2024-03-23T23:09:43.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.877+0000 7f1a3c800640 1 Processor -- start 2024-03-23T23:09:43.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.877+0000 7f1a3c800640 1 -- start start 2024-03-23T23:09:43.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.877+0000 7f1a3c800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a34074100 0x7f1a3419dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:43.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.877+0000 7f1a3c800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a340756e0 0x7f1a3419e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:43.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.878+0000 7f1a3c800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a3410dc80 0x7f1a341a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:43.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.878+0000 7f1a3c800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a34113900 con 0x7f1a340756e0 2024-03-23T23:09:43.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.878+0000 7f1a3c800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a34113780 con 0x7f1a34074100 2024-03-23T23:09:43.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.878+0000 7f1a3c800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a34113a80 con 0x7f1a3410dc80 2024-03-23T23:09:43.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.878+0000 7f1a39e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a34074100 0x7f1a3419dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:43.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.878+0000 7f1a39e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a34074100 0x7f1a3419dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60966/0 (socket says 172.21.15.12:60966) 2024-03-23T23:09:43.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.878+0000 7f1a39e00640 1 -- 172.21.15.12:0/291721609 learned_addr learned my addr 172.21.15.12:0/291721609 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:43.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.878+0000 7f1a3a800640 1 --2- 172.21.15.12:0/291721609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a3410dc80 0x7f1a341a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:43.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.878+0000 7f1a39400640 1 --2- 172.21.15.12:0/291721609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a340756e0 0x7f1a3419e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:43.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.878+0000 7f1a39e00640 1 -- 172.21.15.12:0/291721609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a3410dc80 msgr2=0x7f1a341a5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:43.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.878+0000 7f1a39e00640 1 --2- 172.21.15.12:0/291721609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a3410dc80 0x7f1a341a5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.878+0000 7f1a39e00640 1 -- 172.21.15.12:0/291721609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a340756e0 msgr2=0x7f1a3419e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:43.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.878+0000 7f1a39e00640 1 --2- 172.21.15.12:0/291721609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a340756e0 0x7f1a3419e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.878+0000 7f1a39e00640 1 -- 172.21.15.12:0/291721609 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a341a5bc0 con 0x7f1a34074100 2024-03-23T23:09:43.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.878+0000 7f1a3a800640 1 --2- 172.21.15.12:0/291721609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a3410dc80 0x7f1a341a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:43.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.879+0000 7f1a39e00640 1 --2- 172.21.15.12:0/291721609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a34074100 0x7f1a3419dfa0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1a2800d950 tx=0x7f1a2800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:43.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.879+0000 7f1a2ea00640 1 -- 172.21.15.12:0/291721609 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a28017030 con 0x7f1a34074100 2024-03-23T23:09:43.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.879+0000 7f1a2ea00640 1 -- 172.21.15.12:0/291721609 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a28004020 con 0x7f1a34074100 2024-03-23T23:09:43.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.879+0000 7f1a2ea00640 1 -- 172.21.15.12:0/291721609 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a280047a0 con 0x7f1a34074100 2024-03-23T23:09:43.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.879+0000 7f1a3c800640 1 -- 172.21.15.12:0/291721609 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a341a5eb0 con 0x7f1a34074100 2024-03-23T23:09:43.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.879+0000 7f1a3c800640 1 -- 172.21.15.12:0/291721609 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a341a62c0 con 0x7f1a34074100 2024-03-23T23:09:43.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.883+0000 7f1a3f446740 1 Processor -- start 2024-03-23T23:09:43.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.884+0000 7f1a3f446740 1 -- start start 2024-03-23T23:09:43.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.884+0000 7f1a3f446740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557055488e80 0x5570554892a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:43.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.884+0000 7f1a3f446740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557055489870 0x55705548e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:43.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.884+0000 7f1a3f446740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55705548e9e0 0x557055490e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:43.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.884+0000 7f1a3f446740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570553ee250 con 0x557055488e80 2024-03-23T23:09:43.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.884+0000 7f1a3f446740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x557055396a30 con 0x55705548e9e0 2024-03-23T23:09:43.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.884+0000 7f1a3f446740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570553ea2f0 con 0x557055489870 2024-03-23T23:09:43.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.884+0000 7f1a2ea00640 1 -- 172.21.15.12:0/291721609 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1a280041d0 con 0x7f1a34074100 2024-03-23T23:09:43.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.884+0000 7f1a3a800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557055488e80 0x5570554892a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:43.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.884+0000 7f1a39400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55705548e9e0 0x557055490e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:43.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.884+0000 7f1a39400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55705548e9e0 0x557055490e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60978/0 (socket says 172.21.15.12:60978) 2024-03-23T23:09:43.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.884+0000 7f1a39400640 1 -- 172.21.15.12:0/1949017752 learned_addr learned my addr 172.21.15.12:0/1949017752 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:43.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.885+0000 7f1a39e00640 1 --2- 172.21.15.12:0/1949017752 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557055489870 0x55705548e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:43.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.885+0000 7f1a39400640 1 -- 172.21.15.12:0/1949017752 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557055489870 msgr2=0x55705548e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:43.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.885+0000 7f1a39400640 1 --2- 172.21.15.12:0/1949017752 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557055489870 0x55705548e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.885+0000 7f1a39400640 1 -- 172.21.15.12:0/1949017752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557055488e80 msgr2=0x5570554892a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:43.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.885+0000 7f1a39400640 1 --2- 172.21.15.12:0/1949017752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557055488e80 0x5570554892a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.885+0000 7f1a39400640 1 -- 172.21.15.12:0/1949017752 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557055381690 con 0x55705548e9e0 2024-03-23T23:09:43.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.885+0000 7f1a39e00640 1 --2- 172.21.15.12:0/1949017752 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557055489870 0x55705548e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:09:43.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.885+0000 7f1a39400640 1 --2- 172.21.15.12:0/1949017752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55705548e9e0 0x557055490e70 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009c90 tx=0x7f1a24011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71beb72ef8dfbd81 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:43.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.885+0000 7f1a23000640 1 -- 172.21.15.12:0/1949017752 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2401e070 con 0x55705548e9e0 2024-03-23T23:09:43.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.885+0000 7f1a23000640 1 -- 172.21.15.12:0/1949017752 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a24009e40 con 0x55705548e9e0 2024-03-23T23:09:43.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.885+0000 7f1a23000640 1 -- 172.21.15.12:0/1949017752 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a24018820 con 0x55705548e9e0 2024-03-23T23:09:43.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.886+0000 7f1a2ea00640 1 --2- 172.21.15.12:0/291721609 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1a1c0a2800 0x7f1a1c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:43.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.886+0000 7f1a3f446740 1 -- 172.21.15.12:0/1949017752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55705548e9e0 msgr2=0x557055490e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:43.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.886+0000 7f1a3f446740 1 --2- 172.21.15.12:0/1949017752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55705548e9e0 0x557055490e70 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009c90 tx=0x7f1a24011810 comp rx=0 tx=0).stop 2024-03-23T23:09:43.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.886+0000 7f1a2ea00640 1 -- 172.21.15.12:0/291721609 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1a280ce400 con 0x7f1a34074100 2024-03-23T23:09:43.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.886+0000 7f1a3f446740 1 -- 172.21.15.12:0/1949017752 shutdown_connections 2024-03-23T23:09:43.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.886+0000 7f1a3f446740 1 --2- 172.21.15.12:0/1949017752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55705548e9e0 0x557055490e70 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.886+0000 7f1a3f446740 1 --2- 172.21.15.12:0/1949017752 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557055489870 0x55705548e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.886+0000 7f1a3f446740 1 --2- 172.21.15.12:0/1949017752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557055488e80 0x5570554892a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.886+0000 7f1a3f446740 1 -- 172.21.15.12:0/1949017752 >> 172.21.15.12:0/1949017752 conn(0x557055451200 msgr2=0x5570554847f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:43.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.886+0000 7f1a3a800640 1 --2- 172.21.15.12:0/291721609 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1a1c0a2800 0x7f1a1c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:43.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.886+0000 7f1a3f446740 1 -- 172.21.15.12:0/1949017752 shutdown_connections 2024-03-23T23:09:43.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.886+0000 7f1a3f446740 1 -- 172.21.15.12:0/1949017752 wait complete. 2024-03-23T23:09:43.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.887+0000 7f1a3f446740 1 -- start start 2024-03-23T23:09:43.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.887+0000 7f1a3a800640 1 --2- 172.21.15.12:0/291721609 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1a1c0a2800 0x7f1a1c0a4cc0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1a300324c0 tx=0x7f1a30005ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:43.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.887+0000 7f1a3f446740 1 Processor -- start 2024-03-23T23:09:43.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.888+0000 7f1a3f446740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557055488e80 0x557055492e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:43.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.888+0000 7f1a3f446740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557055489870 0x557055493360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:43.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.888+0000 7f1a3f446740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55705548e9e0 0x55705549a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:43.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.888+0000 7f1a3f446740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x557055491990 con 0x55705548e9e0 2024-03-23T23:09:43.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.888+0000 7f1a3f446740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570553a9e80 con 0x557055489870 2024-03-23T23:09:43.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.888+0000 7f1a3f446740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x557055491b10 con 0x557055488e80 2024-03-23T23:09:43.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.888+0000 7f1a3f446740 10 client.0 _ll_register_callbacks cb 0x5570553bc130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:43.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.888+0000 7f1a39400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557055489870 0x557055493360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:43.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.888+0000 7f1a3a800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55705548e9e0 0x55705549a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:43.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.888+0000 7f1a39400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557055489870 0x557055493360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:33494/0 (socket says 172.21.15.12:33494) 2024-03-23T23:09:43.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.888+0000 7f1a39e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557055488e80 0x557055492e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:43.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.888+0000 7f1a39400640 1 -- 172.21.15.12:0/326719937 learned_addr learned my addr 172.21.15.12:0/326719937 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:43.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.888+0000 7f1a39400640 1 -- 172.21.15.12:0/326719937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557055488e80 msgr2=0x557055492e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:43.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.888+0000 7f1a39400640 1 --2- 172.21.15.12:0/326719937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557055488e80 0x557055492e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.888+0000 7f1a39400640 1 -- 172.21.15.12:0/326719937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55705548e9e0 msgr2=0x55705549a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:43.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.888+0000 7f1a39400640 1 --2- 172.21.15.12:0/326719937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55705548e9e0 0x55705549a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.888+0000 7f1a39400640 1 -- 172.21.15.12:0/326719937 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5570551cc7d0 con 0x557055489870 2024-03-23T23:09:43.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.889+0000 7f1a39400640 1 --2- 172.21.15.12:0/326719937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557055489870 0x557055493360 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f1a24012fd0 tx=0x7f1a24005710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=131015193d727b14 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:43.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.889+0000 7f1a17400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:09:43.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.889+0000 7f1a17400640 1 -- 172.21.15.12:0/326719937 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2401e050 con 0x557055489870 2024-03-23T23:09:43.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.889+0000 7f1a3f446740 1 -- 172.21.15.12:0/326719937 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5570553fb790 con 0x557055489870 2024-03-23T23:09:43.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.889+0000 7f1a17400640 1 -- 172.21.15.12:0/326719937 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1a240048d0 con 0x557055489870 2024-03-23T23:09:43.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.889+0000 7f1a17400640 1 -- 172.21.15.12:0/326719937 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a240098d0 con 0x557055489870 2024-03-23T23:09:43.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.889+0000 7f1a3f446740 10 client.27500 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:43.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.889+0000 7f1a3f446740 1 -- 172.21.15.12:0/326719937 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5570551023f0 con 0x557055489870 2024-03-23T23:09:43.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.890+0000 7f1a17400640 1 -- 172.21.15.12:0/326719937 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5570553fb790 con 0x557055489870 2024-03-23T23:09:43.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.890+0000 7f1a17400640 1 -- 172.21.15.12:0/326719937 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f1a24018520 con 0x557055489870 2024-03-23T23:09:43.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.890+0000 7f1a3f446740 10 client.27500 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:43.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.890+0000 7f1a3f446740 10 client.27500 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:43.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.890+0000 7f1a3f446740 20 client.27500 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:43.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.890+0000 7f1a3f446740 1 --2- 172.21.15.12:0/326719937 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55705549ea10 0x5570554a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:43.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.890+0000 7f1a3f446740 4 client.27500 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:09:43.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.890+0000 7f1a3f446740 1 -- 172.21.15.12:0/326719937 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5570553f5ae0 con 0x55705549ea10 2024-03-23T23:09:43.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.890+0000 7f1a39e00640 1 --2- 172.21.15.12:0/326719937 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55705549ea10 0x5570554a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:43.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.891+0000 7f1a39e00640 1 --2- 172.21.15.12:0/326719937 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55705549ea10 0x5570554a0e50 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb47a3a94b78df62 server_cookie=e7349512bd7f4e3 in_seq=0 out_seq=0 2024-03-23T23:09:43.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.891+0000 7f1a17400640 10 client.27500 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:43.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.891+0000 7f1a17400640 1 -- 172.21.15.12:0/326719937 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5570553f5ae0 con 0x55705549ea10 2024-03-23T23:09:43.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.891+0000 7f1a17400640 10 client.27500 handle_command_reply: tid=0 2024-03-23T23:09:43.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.891+0000 7f1a3f446740 1 client.27500 shutdown 2024-03-23T23:09:43.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.891+0000 7f1a3f446740 1 -- 172.21.15.12:0/326719937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557055489870 msgr2=0x557055493360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:43.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.891+0000 7f1a3f446740 1 --2- 172.21.15.12:0/326719937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557055489870 0x557055493360 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f1a24012fd0 tx=0x7f1a24005710 comp rx=0 tx=0).stop 2024-03-23T23:09:43.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.892+0000 7f1a3f446740 1 -- 172.21.15.12:0/326719937 shutdown_connections 2024-03-23T23:09:43.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.892+0000 7f1a3f446740 1 --2- 172.21.15.12:0/326719937 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55705549ea10 0x5570554a0e50 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.892+0000 7f1a3f446740 1 --2- 172.21.15.12:0/326719937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55705548e9e0 0x55705549a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.892+0000 7f1a3f446740 1 --2- 172.21.15.12:0/326719937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557055489870 0x557055493360 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.892+0000 7f1a3f446740 1 --2- 172.21.15.12:0/326719937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557055488e80 0x557055492e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.892+0000 7f1a3f446740 1 -- 172.21.15.12:0/326719937 >> 172.21.15.12:0/326719937 conn(0x557055451200 msgr2=0x557055483a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:43.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.892+0000 7f1a17400640 0 client.27500 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:43.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.892+0000 7f1a3f446740 1 -- 172.21.15.12:0/326719937 shutdown_connections 2024-03-23T23:09:43.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.892+0000 7f1a3f446740 1 -- 172.21.15.12:0/326719937 wait complete. 2024-03-23T23:09:43.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.892+0000 7f1a3f446740 20 client.27500 trim_cache size 0 max 16384 2024-03-23T23:09:43.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.909+0000 7f1a3f446740 1 Processor -- start 2024-03-23T23:09:43.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.910+0000 7f1a3f446740 1 -- start start 2024-03-23T23:09:43.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.910+0000 7f1a3f446740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557055488e80 0x557055484a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:43.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.910+0000 7f1a3f446740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557055489870 0x55705549a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:43.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.910+0000 7f1a3f446740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705548e9e0 0x557055487a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:43.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.910+0000 7f1a3f446740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a341b1610 con 0x557055489870 2024-03-23T23:09:43.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.910+0000 7f1a3f446740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a341af110 con 0x557055488e80 2024-03-23T23:09:43.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.910+0000 7f1a3f446740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a341ad1b0 con 0x55705548e9e0 2024-03-23T23:09:43.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.910+0000 7f1a39e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557055488e80 0x557055484a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:43.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.910+0000 7f1a39e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557055488e80 0x557055484a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:33510/0 (socket says 172.21.15.12:33510) 2024-03-23T23:09:43.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.910+0000 7f1a39e00640 1 -- 172.21.15.12:0/3668702519 learned_addr learned my addr 172.21.15.12:0/3668702519 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:43.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.910+0000 7f1a3a800640 1 --2- 172.21.15.12:0/3668702519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705548e9e0 0x557055487a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:43.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.910+0000 7f1a39400640 1 --2- 172.21.15.12:0/3668702519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557055489870 0x55705549a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:43.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.910+0000 7f1a39e00640 1 -- 172.21.15.12:0/3668702519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705548e9e0 msgr2=0x557055487a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:43.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.910+0000 7f1a39e00640 1 --2- 172.21.15.12:0/3668702519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705548e9e0 0x557055487a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.910+0000 7f1a39e00640 1 -- 172.21.15.12:0/3668702519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557055489870 msgr2=0x55705549a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:43.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.910+0000 7f1a39e00640 1 --2- 172.21.15.12:0/3668702519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557055489870 0x55705549a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.911+0000 7f1a39e00640 1 -- 172.21.15.12:0/3668702519 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570550c20f0 con 0x557055488e80 2024-03-23T23:09:43.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.911+0000 7f1a39e00640 1 --2- 172.21.15.12:0/3668702519 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557055488e80 0x557055484a20 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f1a280971b0 tx=0x7f1a28000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c27a075aa90cc7a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:43.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.911+0000 7f1a22600640 1 -- 172.21.15.12:0/3668702519 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2809a6b0 con 0x557055488e80 2024-03-23T23:09:43.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.911+0000 7f1a22600640 1 -- 172.21.15.12:0/3668702519 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a2809b450 con 0x557055488e80 2024-03-23T23:09:43.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.911+0000 7f1a22600640 1 -- 172.21.15.12:0/3668702519 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a280990e0 con 0x557055488e80 2024-03-23T23:09:43.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.912+0000 7f1a3f446740 1 -- 172.21.15.12:0/3668702519 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557055488e80 msgr2=0x557055484a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:43.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.912+0000 7f1a3f446740 1 --2- 172.21.15.12:0/3668702519 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557055488e80 0x557055484a20 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f1a280971b0 tx=0x7f1a28000bc0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.912+0000 7f1a3f446740 1 -- 172.21.15.12:0/3668702519 shutdown_connections 2024-03-23T23:09:43.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.912+0000 7f1a3f446740 1 --2- 172.21.15.12:0/3668702519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705548e9e0 0x557055487a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.912+0000 7f1a3f446740 1 --2- 172.21.15.12:0/3668702519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557055489870 0x55705549a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.912+0000 7f1a3f446740 1 --2- 172.21.15.12:0/3668702519 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557055488e80 0x557055484a20 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.912+0000 7f1a3f446740 1 -- 172.21.15.12:0/3668702519 >> 172.21.15.12:0/3668702519 conn(0x557055451200 msgr2=0x557055484310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:43.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.912+0000 7f1a3f446740 1 -- 172.21.15.12:0/3668702519 shutdown_connections 2024-03-23T23:09:43.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.912+0000 7f1a3f446740 1 -- 172.21.15.12:0/3668702519 wait complete. 2024-03-23T23:09:43.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.912+0000 7f1a3f446740 1 -- start start 2024-03-23T23:09:43.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.913+0000 7f1a3f446740 1 Processor -- start 2024-03-23T23:09:43.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.913+0000 7f1a3f446740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557055488e80 0x5570554973d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:43.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.913+0000 7f1a3f446740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557055489870 0x557055497910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:43.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.913+0000 7f1a3f446740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705548e9e0 0x557055498920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.913+0000 7f1a3f446740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a341ad7b0 con 0x557055489870 2024-03-23T23:09:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.913+0000 7f1a3f446740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a341aeab0 con 0x557055488e80 2024-03-23T23:09:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.913+0000 7f1a3f446740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a341addf0 con 0x55705548e9e0 2024-03-23T23:09:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.913+0000 7f1a3f446740 10 client.0 _ll_register_callbacks cb 0x7f1a2401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.913+0000 7f1a39e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557055488e80 0x5570554973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.913+0000 7f1a39e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557055488e80 0x5570554973d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:33526/0 (socket says 172.21.15.12:33526) 2024-03-23T23:09:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.913+0000 7f1a39e00640 1 -- 172.21.15.12:0/2948694938 learned_addr learned my addr 172.21.15.12:0/2948694938 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.913+0000 7f1a39400640 1 --2- 172.21.15.12:0/2948694938 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557055489870 0x557055497910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.913+0000 7f1a3a800640 1 --2- 172.21.15.12:0/2948694938 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705548e9e0 0x557055498920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.914+0000 7f1a39e00640 1 -- 172.21.15.12:0/2948694938 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705548e9e0 msgr2=0x557055498920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.914+0000 7f1a39e00640 1 --2- 172.21.15.12:0/2948694938 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705548e9e0 0x557055498920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.914+0000 7f1a39e00640 1 -- 172.21.15.12:0/2948694938 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557055489870 msgr2=0x557055497910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.914+0000 7f1a39e00640 1 --2- 172.21.15.12:0/2948694938 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557055489870 0x557055497910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.914+0000 7f1a39e00640 1 -- 172.21.15.12:0/2948694938 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557055498f10 con 0x557055488e80 2024-03-23T23:09:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.914+0000 7f1a39e00640 1 --2- 172.21.15.12:0/2948694938 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557055488e80 0x5570554973d0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f1a2809ace0 tx=0x7f1a28098df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0c926ae88f44310 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.914+0000 7f1a17400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:09:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.914+0000 7f1a17400640 1 -- 172.21.15.12:0/2948694938 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a280a34f0 con 0x557055488e80 2024-03-23T23:09:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.914+0000 7f1a17400640 1 -- 172.21.15.12:0/2948694938 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1a280973c0 con 0x557055488e80 2024-03-23T23:09:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.914+0000 7f1a3f446740 1 -- 172.21.15.12:0/2948694938 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1a2809b450 con 0x557055488e80 2024-03-23T23:09:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.915+0000 7f1a17400640 1 -- 172.21.15.12:0/2948694938 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a2809ad50 con 0x557055488e80 2024-03-23T23:09:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.915+0000 7f1a3f446740 10 client.27506 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.915+0000 7f1a3f446740 1 -- 172.21.15.12:0/2948694938 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55705549a290 con 0x557055488e80 2024-03-23T23:09:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.915+0000 7f1a17400640 1 -- 172.21.15.12:0/2948694938 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a2809b450 con 0x557055488e80 2024-03-23T23:09:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.915+0000 7f1a17400640 1 -- 172.21.15.12:0/2948694938 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f1a2809a1a0 con 0x557055488e80 2024-03-23T23:09:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.915+0000 7f1a3f446740 10 client.27506 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.915+0000 7f1a3f446740 10 client.27506 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.915+0000 7f1a3f446740 20 client.27506 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.915+0000 7f1a3f446740 1 --2- 172.21.15.12:0/2948694938 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5570554fa560 0x5570554fc980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.915+0000 7f1a3f446740 4 client.27506 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:09:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.915+0000 7f1a3f446740 1 -- 172.21.15.12:0/2948694938 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5570553733a0 con 0x5570554fa560 2024-03-23T23:09:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.915+0000 7f1a39400640 1 --2- 172.21.15.12:0/2948694938 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5570554fa560 0x5570554fc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.916+0000 7f1a39400640 1 --2- 172.21.15.12:0/2948694938 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5570554fa560 0x5570554fc980 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1512a3c4a16f4a8 server_cookie=b9299b63508da062 in_seq=0 out_seq=0 2024-03-23T23:09:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.916+0000 7f1a17400640 10 client.27506 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.916+0000 7f1a17400640 1 -- 172.21.15.12:0/2948694938 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5570553733a0 con 0x5570554fa560 2024-03-23T23:09:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.916+0000 7f1a17400640 10 client.27506 handle_command_reply: tid=0 2024-03-23T23:09:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.916+0000 7f1a3f446740 1 client.27506 shutdown 2024-03-23T23:09:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.916+0000 7f1a3f446740 1 -- 172.21.15.12:0/2948694938 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557055488e80 msgr2=0x5570554973d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.916+0000 7f1a3f446740 1 --2- 172.21.15.12:0/2948694938 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557055488e80 0x5570554973d0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f1a2809ace0 tx=0x7f1a28098df0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.916+0000 7f1a3f446740 1 -- 172.21.15.12:0/2948694938 shutdown_connections 2024-03-23T23:09:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.916+0000 7f1a3f446740 1 --2- 172.21.15.12:0/2948694938 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5570554fa560 0x5570554fc980 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.916+0000 7f1a3f446740 1 --2- 172.21.15.12:0/2948694938 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705548e9e0 0x557055498920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.916+0000 7f1a3f446740 1 --2- 172.21.15.12:0/2948694938 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557055489870 0x557055497910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.916+0000 7f1a3f446740 1 --2- 172.21.15.12:0/2948694938 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557055488e80 0x5570554973d0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.916+0000 7f1a17400640 0 client.27506 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.916+0000 7f1a3f446740 1 -- 172.21.15.12:0/2948694938 >> 172.21.15.12:0/2948694938 conn(0x557055451200 msgr2=0x557055487a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.916+0000 7f1a3f446740 1 -- 172.21.15.12:0/2948694938 shutdown_connections 2024-03-23T23:09:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.916+0000 7f1a3f446740 1 -- 172.21.15.12:0/2948694938 wait complete. 2024-03-23T23:09:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.917+0000 7f1a3f446740 20 client.27506 trim_cache size 0 max 16384 2024-03-23T23:09:43.920 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:09:43.920 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-03-23T23:09:43.920 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-03-23T23:09:43.920 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:09:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.917+0000 7f1a3c800640 1 -- 172.21.15.12:0/291721609 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1a1c0a2800 msgr2=0x7f1a1c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.917+0000 7f1a3c800640 1 --2- 172.21.15.12:0/291721609 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1a1c0a2800 0x7f1a1c0a4cc0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1a300324c0 tx=0x7f1a30005ef0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.917+0000 7f1a3c800640 1 -- 172.21.15.12:0/291721609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a34074100 msgr2=0x7f1a3419dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.917+0000 7f1a3c800640 1 --2- 172.21.15.12:0/291721609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a34074100 0x7f1a3419dfa0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1a2800d950 tx=0x7f1a2800de10 comp rx=0 tx=0).stop 2024-03-23T23:09:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.917+0000 7f1a3c800640 1 -- 172.21.15.12:0/291721609 shutdown_connections 2024-03-23T23:09:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.917+0000 7f1a3c800640 1 --2- 172.21.15.12:0/291721609 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1a1c0a2800 0x7f1a1c0a4cc0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.917+0000 7f1a3c800640 1 --2- 172.21.15.12:0/291721609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a3410dc80 0x7f1a341a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.917+0000 7f1a3c800640 1 --2- 172.21.15.12:0/291721609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a340756e0 0x7f1a3419e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.917+0000 7f1a3c800640 1 --2- 172.21.15.12:0/291721609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a34074100 0x7f1a3419dfa0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.917+0000 7f1a3c800640 1 -- 172.21.15.12:0/291721609 >> 172.21.15.12:0/291721609 conn(0x7f1a340fcc40 msgr2=0x7f1a34101020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.918+0000 7f1a3c800640 1 -- 172.21.15.12:0/291721609 shutdown_connections 2024-03-23T23:09:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:43.918+0000 7f1a3c800640 1 -- 172.21.15.12:0/291721609 wait complete. 2024-03-23T23:09:43.934 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T23:09:43.934 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:09:44.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.161+0000 7ff71aa00640 1 -- 172.21.15.12:0/4179457998 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7141028b0 msgr2=0x7ff714102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:44.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.161+0000 7ff71aa00640 1 --2- 172.21.15.12:0/4179457998 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7141028b0 0x7ff714102d30 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc009900 tx=0x7ff6fc02bc40 comp rx=0 tx=0).stop 2024-03-23T23:09:44.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.161+0000 7ff71aa00640 1 -- 172.21.15.12:0/4179457998 shutdown_connections 2024-03-23T23:09:44.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.161+0000 7ff71aa00640 1 --2- 172.21.15.12:0/4179457998 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff714103270 0x7ff714109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.161+0000 7ff71aa00640 1 --2- 172.21.15.12:0/4179457998 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7141028b0 0x7ff714102d30 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.161+0000 7ff71aa00640 1 --2- 172.21.15.12:0/4179457998 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7141016c0 0x7ff714101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.161+0000 7ff71aa00640 1 -- 172.21.15.12:0/4179457998 >> 172.21.15.12:0/4179457998 conn(0x7ff7140fcc50 msgr2=0x7ff7140ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:44.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.161+0000 7ff71aa00640 1 -- 172.21.15.12:0/4179457998 shutdown_connections 2024-03-23T23:09:44.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.161+0000 7ff71aa00640 1 -- 172.21.15.12:0/4179457998 wait complete. 2024-03-23T23:09:44.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.162+0000 7ff71aa00640 1 Processor -- start 2024-03-23T23:09:44.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.163+0000 7ff71aa00640 1 -- start start 2024-03-23T23:09:44.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.163+0000 7ff71aa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7141016c0 0x7ff714075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.163+0000 7ff71aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7141028b0 0x7ff714073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.163+0000 7ff71aa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff714103270 0x7ff714073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.163+0000 7ff71aa00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71410d370 con 0x7ff714103270 2024-03-23T23:09:44.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.163+0000 7ff71aa00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71410d1f0 con 0x7ff7141028b0 2024-03-23T23:09:44.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.163+0000 7ff71aa00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71410d4f0 con 0x7ff7141016c0 2024-03-23T23:09:44.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.164+0000 7ff713e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7141016c0 0x7ff714075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.164+0000 7ff713e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7141016c0 0x7ff714075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:57552/0 (socket says 172.21.15.12:57552) 2024-03-23T23:09:44.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.164+0000 7ff713e00640 1 -- 172.21.15.12:0/1319018529 learned_addr learned my addr 172.21.15.12:0/1319018529 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:44.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.164+0000 7ff70b400640 1 --2- 172.21.15.12:0/1319018529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7141028b0 0x7ff714073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.164+0000 7ff718a00640 1 --2- 172.21.15.12:0/1319018529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff714103270 0x7ff714073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.164+0000 7ff713e00640 1 -- 172.21.15.12:0/1319018529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7141028b0 msgr2=0x7ff714073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:44.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.164+0000 7ff713e00640 1 --2- 172.21.15.12:0/1319018529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7141028b0 0x7ff714073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.164+0000 7ff713e00640 1 -- 172.21.15.12:0/1319018529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff714103270 msgr2=0x7ff714073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:44.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.164+0000 7ff713e00640 1 --2- 172.21.15.12:0/1319018529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff714103270 0x7ff714073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.164+0000 7ff713e00640 1 -- 172.21.15.12:0/1319018529 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff714074860 con 0x7ff7141016c0 2024-03-23T23:09:44.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.164+0000 7ff718a00640 1 --2- 172.21.15.12:0/1319018529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff714103270 0x7ff714073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:09:44.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.164+0000 7ff713e00640 1 --2- 172.21.15.12:0/1319018529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7141016c0 0x7ff714075450 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff70400e9c0 tx=0x7ff70400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:44.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.164+0000 7ff711600640 1 -- 172.21.15.12:0/1319018529 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70400cd80 con 0x7ff7141016c0 2024-03-23T23:09:44.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.165+0000 7ff711600640 1 -- 172.21.15.12:0/1319018529 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff7040035f0 con 0x7ff7141016c0 2024-03-23T23:09:44.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.165+0000 7ff71aa00640 1 -- 172.21.15.12:0/1319018529 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7141a1ad0 con 0x7ff7141016c0 2024-03-23T23:09:44.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.165+0000 7ff711600640 1 -- 172.21.15.12:0/1319018529 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff704010690 con 0x7ff7141016c0 2024-03-23T23:09:44.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.165+0000 7ff71aa00640 1 -- 172.21.15.12:0/1319018529 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7141a2040 con 0x7ff7141016c0 2024-03-23T23:09:44.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.166+0000 7ff71aa00640 1 -- 172.21.15.12:0/1319018529 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7140688e0 con 0x7ff7141016c0 2024-03-23T23:09:44.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.168+0000 7ff711600640 1 -- 172.21.15.12:0/1319018529 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff7040037a0 con 0x7ff7141016c0 2024-03-23T23:09:44.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.169+0000 7ff711600640 1 --2- 172.21.15.12:0/1319018529 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff6f80a2880 0x7ff6f80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.169+0000 7ff711600640 1 -- 172.21.15.12:0/1319018529 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff7040ceed0 con 0x7ff7141016c0 2024-03-23T23:09:44.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.169+0000 7ff70b400640 1 --2- 172.21.15.12:0/1319018529 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff6f80a2880 0x7ff6f80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.170+0000 7ff70b400640 1 --2- 172.21.15.12:0/1319018529 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff6f80a2880 0x7ff6f80a4d40 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc02c250 tx=0x7ff6fc02cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:44.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.173+0000 7ff711600640 1 -- 172.21.15.12:0/1319018529 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff70409bf20 con 0x7ff7141016c0 2024-03-23T23:09:44.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.422+0000 7ff71aa00640 1 -- 172.21.15.12:0/1319018529 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff714108980 con 0x7ff7141016c0 2024-03-23T23:09:44.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.424+0000 7ff711600640 1 -- 172.21.15.12:0/1319018529 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7ff714108980 con 0x7ff7141016c0 2024-03-23T23:09:44.423 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:09:44.423 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:09:44.423 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:09:44.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.427+0000 7ff71aa00640 1 -- 172.21.15.12:0/1319018529 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff6f80a2880 msgr2=0x7ff6f80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:44.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.427+0000 7ff71aa00640 1 --2- 172.21.15.12:0/1319018529 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff6f80a2880 0x7ff6f80a4d40 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc02c250 tx=0x7ff6fc02cf90 comp rx=0 tx=0).stop 2024-03-23T23:09:44.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.427+0000 7ff71aa00640 1 -- 172.21.15.12:0/1319018529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7141016c0 msgr2=0x7ff714075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:44.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.427+0000 7ff71aa00640 1 --2- 172.21.15.12:0/1319018529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7141016c0 0x7ff714075450 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff70400e9c0 tx=0x7ff70400ee80 comp rx=0 tx=0).stop 2024-03-23T23:09:44.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.427+0000 7ff71aa00640 1 -- 172.21.15.12:0/1319018529 shutdown_connections 2024-03-23T23:09:44.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.427+0000 7ff71aa00640 1 --2- 172.21.15.12:0/1319018529 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff6f80a2880 0x7ff6f80a4d40 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.427+0000 7ff71aa00640 1 --2- 172.21.15.12:0/1319018529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff714103270 0x7ff714073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.427+0000 7ff71aa00640 1 --2- 172.21.15.12:0/1319018529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7141028b0 0x7ff714073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.427+0000 7ff71aa00640 1 --2- 172.21.15.12:0/1319018529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7141016c0 0x7ff714075450 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.427+0000 7ff71aa00640 1 -- 172.21.15.12:0/1319018529 >> 172.21.15.12:0/1319018529 conn(0x7ff7140fcc50 msgr2=0x7ff714105b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:44.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.427+0000 7ff71aa00640 1 -- 172.21.15.12:0/1319018529 shutdown_connections 2024-03-23T23:09:44.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.427+0000 7ff71aa00640 1 -- 172.21.15.12:0/1319018529 wait complete. 2024-03-23T23:09:44.447 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:09:44.447 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-23T23:09:44.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.678+0000 7fc996600640 1 -- 172.21.15.12:0/794262180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9901038c0 msgr2=0x7fc990103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:44.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.678+0000 7fc996600640 1 --2- 172.21.15.12:0/794262180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9901038c0 0x7fc990103ce0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fc980009900 tx=0x7fc98002bc30 comp rx=0 tx=0).stop 2024-03-23T23:09:44.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.678+0000 7fc996600640 1 -- 172.21.15.12:0/794262180 shutdown_connections 2024-03-23T23:09:44.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.678+0000 7fc996600640 1 --2- 172.21.15.12:0/794262180 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc990105470 0x7fc99010bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.678+0000 7fc996600640 1 --2- 172.21.15.12:0/794262180 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc990104ab0 0x7fc990104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.678+0000 7fc996600640 1 --2- 172.21.15.12:0/794262180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9901038c0 0x7fc990103ce0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.678+0000 7fc996600640 1 -- 172.21.15.12:0/794262180 >> 172.21.15.12:0/794262180 conn(0x7fc9900fee50 msgr2=0x7fc990101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:44.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.678+0000 7fc996600640 1 -- 172.21.15.12:0/794262180 shutdown_connections 2024-03-23T23:09:44.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.678+0000 7fc996600640 1 -- 172.21.15.12:0/794262180 wait complete. 2024-03-23T23:09:44.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.679+0000 7fc996600640 1 Processor -- start 2024-03-23T23:09:44.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.680+0000 7fc996600640 1 -- start start 2024-03-23T23:09:44.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.680+0000 7fc996600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9901038c0 0x7fc990199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.680+0000 7fc996600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc990104ab0 0x7fc99019a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.680+0000 7fc996600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc990105470 0x7fc9901a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.680+0000 7fc996600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99010f580 con 0x7fc9901038c0 2024-03-23T23:09:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.680+0000 7fc996600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99010f400 con 0x7fc990104ab0 2024-03-23T23:09:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.680+0000 7fc996600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99010f700 con 0x7fc990105470 2024-03-23T23:09:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.680+0000 7fc98f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9901038c0 0x7fc990199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.680+0000 7fc98fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc990105470 0x7fc9901a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.680+0000 7fc98ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc990104ab0 0x7fc99019a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.680+0000 7fc98f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9901038c0 0x7fc990199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38920/0 (socket says 172.21.15.12:38920) 2024-03-23T23:09:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.680+0000 7fc98fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc990105470 0x7fc9901a1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:57584/0 (socket says 172.21.15.12:57584) 2024-03-23T23:09:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.680+0000 7fc98f400640 1 -- 172.21.15.12:0/2063357617 learned_addr learned my addr 172.21.15.12:0/2063357617 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.681+0000 7fc98fe00640 1 -- 172.21.15.12:0/2063357617 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc990104ab0 msgr2=0x7fc99019a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.681+0000 7fc98fe00640 1 --2- 172.21.15.12:0/2063357617 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc990104ab0 0x7fc99019a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.681+0000 7fc98fe00640 1 -- 172.21.15.12:0/2063357617 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9901038c0 msgr2=0x7fc990199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.681+0000 7fc98fe00640 1 --2- 172.21.15.12:0/2063357617 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9901038c0 0x7fc990199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.681+0000 7fc98fe00640 1 -- 172.21.15.12:0/2063357617 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9901a1830 con 0x7fc990105470 2024-03-23T23:09:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.681+0000 7fc98fe00640 1 --2- 172.21.15.12:0/2063357617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc990105470 0x7fc9901a1190 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fc988009fd0 tx=0x7fc98800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.681+0000 7fc987e00640 1 -- 172.21.15.12:0/2063357617 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98800ce20 con 0x7fc990105470 2024-03-23T23:09:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.681+0000 7fc987e00640 1 -- 172.21.15.12:0/2063357617 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc988005020 con 0x7fc990105470 2024-03-23T23:09:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.681+0000 7fc996600640 1 -- 172.21.15.12:0/2063357617 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9901a1b20 con 0x7fc990105470 2024-03-23T23:09:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.681+0000 7fc987e00640 1 -- 172.21.15.12:0/2063357617 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc988010720 con 0x7fc990105470 2024-03-23T23:09:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.682+0000 7fc996600640 1 -- 172.21.15.12:0/2063357617 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9901a2070 con 0x7fc990105470 2024-03-23T23:09:44.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.686+0000 7fc9993ef740 1 Processor -- start 2024-03-23T23:09:44.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.686+0000 7fc987e00640 1 -- 172.21.15.12:0/2063357617 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc988005540 con 0x7fc990105470 2024-03-23T23:09:44.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.686+0000 7fc9993ef740 1 -- start start 2024-03-23T23:09:44.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.687+0000 7fc9993ef740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576771e5810 0x5576771e5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.687+0000 7fc9993ef740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576771e69e0 0x5576771e6e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.687+0000 7fc9993ef740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e73a0 0x5576771edc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.687+0000 7fc9993ef740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55767714a2c0 con 0x5576771e69e0 2024-03-23T23:09:44.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.687+0000 7fc9993ef740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576770f2c90 con 0x5576771e5810 2024-03-23T23:09:44.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.687+0000 7fc9993ef740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x557677146360 con 0x5576771e73a0 2024-03-23T23:09:44.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.687+0000 7fc98ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576771e5810 0x5576771e5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.687+0000 7fc98ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576771e5810 0x5576771e5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:33598/0 (socket says 172.21.15.12:33598) 2024-03-23T23:09:44.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.687+0000 7fc98ea00640 1 -- 172.21.15.12:0/2100125227 learned_addr learned my addr 172.21.15.12:0/2100125227 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:44.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.687+0000 7fc98ea00640 1 --2- 172.21.15.12:0/2100125227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e73a0 0x5576771edc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.687+0000 7fc98f400640 1 --2- 172.21.15.12:0/2100125227 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576771e69e0 0x5576771e6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.687+0000 7fc98ea00640 1 -- 172.21.15.12:0/2100125227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e73a0 msgr2=0x5576771edc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:44.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.687+0000 7fc98ea00640 1 --2- 172.21.15.12:0/2100125227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e73a0 0x5576771edc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.687+0000 7fc98ea00640 1 -- 172.21.15.12:0/2100125227 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576771e69e0 msgr2=0x5576771e6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:44.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.687+0000 7fc98ea00640 1 --2- 172.21.15.12:0/2100125227 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576771e69e0 0x5576771e6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.687+0000 7fc98ea00640 1 -- 172.21.15.12:0/2100125227 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557677117080 con 0x5576771e5810 2024-03-23T23:09:44.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.688+0000 7fc98ea00640 1 --2- 172.21.15.12:0/2100125227 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576771e5810 0x5576771e5c30 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fc97c011870 tx=0x7fc97c0097d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4aeb58cb967920c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:44.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.688+0000 7fc987e00640 1 --2- 172.21.15.12:0/2063357617 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc9740a2880 0x7fc9740a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.688+0000 7fc984c00640 1 -- 172.21.15.12:0/2100125227 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c00f7c0 con 0x5576771e5810 2024-03-23T23:09:44.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.688+0000 7fc984c00640 1 -- 172.21.15.12:0/2100125227 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc97c00fe40 con 0x5576771e5810 2024-03-23T23:09:44.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.688+0000 7fc984c00640 1 -- 172.21.15.12:0/2100125227 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c01fe60 con 0x5576771e5810 2024-03-23T23:09:44.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.688+0000 7fc987e00640 1 -- 172.21.15.12:0/2063357617 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fc9880cf1d0 con 0x7fc990105470 2024-03-23T23:09:44.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.688+0000 7fc98f400640 1 --2- 172.21.15.12:0/2063357617 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc9740a2880 0x7fc9740a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.689+0000 7fc9993ef740 1 -- 172.21.15.12:0/2100125227 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576771e5810 msgr2=0x5576771e5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:44.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.689+0000 7fc9993ef740 1 --2- 172.21.15.12:0/2100125227 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576771e5810 0x5576771e5c30 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fc97c011870 tx=0x7fc97c0097d0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.689+0000 7fc9993ef740 1 -- 172.21.15.12:0/2100125227 shutdown_connections 2024-03-23T23:09:44.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.689+0000 7fc9993ef740 1 --2- 172.21.15.12:0/2100125227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e73a0 0x5576771edc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.689+0000 7fc9993ef740 1 --2- 172.21.15.12:0/2100125227 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576771e69e0 0x5576771e6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.689+0000 7fc9993ef740 1 --2- 172.21.15.12:0/2100125227 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576771e5810 0x5576771e5c30 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.689+0000 7fc9993ef740 1 -- 172.21.15.12:0/2100125227 >> 172.21.15.12:0/2100125227 conn(0x5576771e0c00 msgr2=0x5576771e1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:44.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.689+0000 7fc9993ef740 1 -- 172.21.15.12:0/2100125227 shutdown_connections 2024-03-23T23:09:44.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.689+0000 7fc9993ef740 1 -- 172.21.15.12:0/2100125227 wait complete. 2024-03-23T23:09:44.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.689+0000 7fc98f400640 1 --2- 172.21.15.12:0/2063357617 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc9740a2880 0x7fc9740a4d40 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc99019b0f0 tx=0x7fc98002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:44.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.689+0000 7fc9993ef740 1 -- start start 2024-03-23T23:09:44.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.690+0000 7fc9993ef740 1 Processor -- start 2024-03-23T23:09:44.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.690+0000 7fc9993ef740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576771e5810 0x5576771efa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.690+0000 7fc9993ef740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e69e0 0x5576771effd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.690+0000 7fc9993ef740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576771e73a0 0x5576771f7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.690+0000 7fc9993ef740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576771ee3b0 con 0x5576771e5810 2024-03-23T23:09:44.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.690+0000 7fc9993ef740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576771060d0 con 0x5576771e73a0 2024-03-23T23:09:44.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.690+0000 7fc9993ef740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576771ee530 con 0x5576771e69e0 2024-03-23T23:09:44.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.690+0000 7fc9993ef740 10 client.0 _ll_register_callbacks cb 0x55767710ae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:44.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.690+0000 7fc98ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e69e0 0x5576771effd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.691+0000 7fc98f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576771e5810 0x5576771efa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.691+0000 7fc98fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576771e73a0 0x5576771f7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.691+0000 7fc98ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e69e0 0x5576771effd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:57604/0 (socket says 172.21.15.12:57604) 2024-03-23T23:09:44.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.691+0000 7fc98ea00640 1 -- 172.21.15.12:0/2761753415 learned_addr learned my addr 172.21.15.12:0/2761753415 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:44.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.691+0000 7fc98ea00640 1 -- 172.21.15.12:0/2761753415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576771e73a0 msgr2=0x5576771f7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:44.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.691+0000 7fc98ea00640 1 --2- 172.21.15.12:0/2761753415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576771e73a0 0x5576771f7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.691+0000 7fc98ea00640 1 -- 172.21.15.12:0/2761753415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576771e5810 msgr2=0x5576771efa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:44.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.691+0000 7fc98ea00640 1 --2- 172.21.15.12:0/2761753415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576771e5810 0x5576771efa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.691+0000 7fc98ea00640 1 -- 172.21.15.12:0/2761753415 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557676e5e3f0 con 0x5576771e69e0 2024-03-23T23:09:44.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.691+0000 7fc98fe00640 1 --2- 172.21.15.12:0/2761753415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576771e73a0 0x5576771f7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:44.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.691+0000 7fc98ea00640 1 --2- 172.21.15.12:0/2761753415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e69e0 0x5576771effd0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc97c0097a0 tx=0x7fc97c005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=475103f31271ccfe server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:44.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.691+0000 7fc979600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:09:44.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.691+0000 7fc979600640 1 -- 172.21.15.12:0/2761753415 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c02c060 con 0x5576771e69e0 2024-03-23T23:09:44.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.692+0000 7fc979600640 1 -- 172.21.15.12:0/2761753415 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fc97c019070 con 0x5576771e69e0 2024-03-23T23:09:44.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.692+0000 7fc9993ef740 1 -- 172.21.15.12:0/2761753415 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc97c00fe40 con 0x5576771e69e0 2024-03-23T23:09:44.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.692+0000 7fc979600640 1 -- 172.21.15.12:0/2761753415 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc97c01f580 con 0x5576771e69e0 2024-03-23T23:09:44.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.692+0000 7fc9993ef740 10 client.27490 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:44.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.692+0000 7fc9993ef740 1 -- 172.21.15.12:0/2761753415 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x557676f287d0 con 0x5576771e69e0 2024-03-23T23:09:44.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.692+0000 7fc979600640 1 -- 172.21.15.12:0/2761753415 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc97c00fe40 con 0x5576771e69e0 2024-03-23T23:09:44.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.692+0000 7fc979600640 1 -- 172.21.15.12:0/2761753415 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fc97c01d020 con 0x5576771e69e0 2024-03-23T23:09:44.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.692+0000 7fc9993ef740 10 client.27490 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:44.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.692+0000 7fc9993ef740 10 client.27490 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:09:44.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.692+0000 7fc9993ef740 20 client.27490 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:44.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.692+0000 7fc9993ef740 1 --2- 172.21.15.12:0/2761753415 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5576771fb1c0 0x5576771fd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.692+0000 7fc9993ef740 4 client.27490 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:09:44.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.692+0000 7fc9993ef740 1 -- 172.21.15.12:0/2761753415 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557677157800 con 0x5576771fb1c0 2024-03-23T23:09:44.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.693+0000 7fc98f400640 1 --2- 172.21.15.12:0/2761753415 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5576771fb1c0 0x5576771fd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.693+0000 7fc98f400640 1 --2- 172.21.15.12:0/2761753415 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5576771fb1c0 0x5576771fd5d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e7f47d46a8cb09f1 server_cookie=e1d2882e028ea112 in_seq=0 out_seq=0 2024-03-23T23:09:44.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.693+0000 7fc979600640 10 client.27490 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:09:44.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.694+0000 7fc979600640 1 -- 172.21.15.12:0/2761753415 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557677157800 con 0x5576771fb1c0 2024-03-23T23:09:44.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.694+0000 7fc979600640 10 client.27490 handle_command_reply: tid=0 2024-03-23T23:09:44.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.694+0000 7fc9993ef740 1 client.27490 shutdown 2024-03-23T23:09:44.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.694+0000 7fc9993ef740 1 -- 172.21.15.12:0/2761753415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e69e0 msgr2=0x5576771effd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:44.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.694+0000 7fc9993ef740 1 --2- 172.21.15.12:0/2761753415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e69e0 0x5576771effd0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc97c0097a0 tx=0x7fc97c005fd0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.694+0000 7fc9993ef740 1 -- 172.21.15.12:0/2761753415 shutdown_connections 2024-03-23T23:09:44.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.694+0000 7fc9993ef740 1 --2- 172.21.15.12:0/2761753415 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5576771fb1c0 0x5576771fd5d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.694+0000 7fc9993ef740 1 --2- 172.21.15.12:0/2761753415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576771e73a0 0x5576771f7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.694+0000 7fc9993ef740 1 --2- 172.21.15.12:0/2761753415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e69e0 0x5576771effd0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.694+0000 7fc9993ef740 1 --2- 172.21.15.12:0/2761753415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576771e5810 0x5576771efa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.694+0000 7fc9993ef740 1 -- 172.21.15.12:0/2761753415 >> 172.21.15.12:0/2761753415 conn(0x5576771e0c00 msgr2=0x5576771dfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:44.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.694+0000 7fc979600640 0 client.27490 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:09:44.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.695+0000 7fc9993ef740 1 -- 172.21.15.12:0/2761753415 shutdown_connections 2024-03-23T23:09:44.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.695+0000 7fc9993ef740 1 -- 172.21.15.12:0/2761753415 wait complete. 2024-03-23T23:09:44.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.695+0000 7fc9993ef740 20 client.27490 trim_cache size 0 max 16384 2024-03-23T23:09:44.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.708+0000 7fc9993ef740 1 Processor -- start 2024-03-23T23:09:44.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.708+0000 7fc9993ef740 1 -- start start 2024-03-23T23:09:44.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.709+0000 7fc9993ef740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576771e5810 0x5576771fd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.709+0000 7fc9993ef740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e69e0 0x5576771f6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.709+0000 7fc9993ef740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576771e73a0 0x5576771e4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.709+0000 7fc9993ef740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9901ad190 con 0x5576771e73a0 2024-03-23T23:09:44.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.709+0000 7fc9993ef740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9901aac90 con 0x5576771e5810 2024-03-23T23:09:44.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.709+0000 7fc9993ef740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9901a8d30 con 0x5576771e69e0 2024-03-23T23:09:44.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.709+0000 7fc98ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e69e0 0x5576771f6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.709+0000 7fc98ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e69e0 0x5576771f6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:57618/0 (socket says 172.21.15.12:57618) 2024-03-23T23:09:44.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.709+0000 7fc98ea00640 1 -- 172.21.15.12:0/973019836 learned_addr learned my addr 172.21.15.12:0/973019836 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:44.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.709+0000 7fc98f400640 1 --2- 172.21.15.12:0/973019836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576771e5810 0x5576771fd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.709+0000 7fc98fe00640 1 --2- 172.21.15.12:0/973019836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576771e73a0 0x5576771e4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.709+0000 7fc98ea00640 1 -- 172.21.15.12:0/973019836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576771e5810 msgr2=0x5576771fd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:44.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.709+0000 7fc98ea00640 1 --2- 172.21.15.12:0/973019836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576771e5810 0x5576771fd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.709+0000 7fc98ea00640 1 -- 172.21.15.12:0/973019836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576771e73a0 msgr2=0x5576771e4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:44.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.709+0000 7fc98ea00640 1 --2- 172.21.15.12:0/973019836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576771e73a0 0x5576771e4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.709+0000 7fc98ea00640 1 -- 172.21.15.12:0/973019836 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576771eeea0 con 0x5576771e69e0 2024-03-23T23:09:44.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.709+0000 7fc98fe00640 1 --2- 172.21.15.12:0/973019836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576771e73a0 0x5576771e4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:09:44.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.709+0000 7fc98f400640 1 --2- 172.21.15.12:0/973019836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576771e5810 0x5576771fd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:44.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.710+0000 7fc98ea00640 1 --2- 172.21.15.12:0/973019836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e69e0 0x5576771f6fd0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc97c00a010 tx=0x7fc97c0097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a80543f2434d337d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:44.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.710+0000 7fc97be00640 1 -- 172.21.15.12:0/973019836 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c037070 con 0x5576771e69e0 2024-03-23T23:09:44.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.710+0000 7fc97be00640 1 -- 172.21.15.12:0/973019836 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc97c005550 con 0x5576771e69e0 2024-03-23T23:09:44.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.710+0000 7fc97be00640 1 -- 172.21.15.12:0/973019836 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c02c040 con 0x5576771e69e0 2024-03-23T23:09:44.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.710+0000 7fc9993ef740 1 -- 172.21.15.12:0/973019836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e69e0 msgr2=0x5576771f6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:44.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.710+0000 7fc9993ef740 1 --2- 172.21.15.12:0/973019836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e69e0 0x5576771f6fd0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc97c00a010 tx=0x7fc97c0097d0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.711+0000 7fc9993ef740 1 -- 172.21.15.12:0/973019836 shutdown_connections 2024-03-23T23:09:44.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.711+0000 7fc9993ef740 1 --2- 172.21.15.12:0/973019836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576771e73a0 0x5576771e4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.711+0000 7fc9993ef740 1 --2- 172.21.15.12:0/973019836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e69e0 0x5576771f6fd0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.711+0000 7fc9993ef740 1 --2- 172.21.15.12:0/973019836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576771e5810 0x5576771fd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.711+0000 7fc9993ef740 1 -- 172.21.15.12:0/973019836 >> 172.21.15.12:0/973019836 conn(0x5576771e0c00 msgr2=0x5576771e3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:44.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.711+0000 7fc9993ef740 1 -- 172.21.15.12:0/973019836 shutdown_connections 2024-03-23T23:09:44.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.711+0000 7fc9993ef740 1 -- 172.21.15.12:0/973019836 wait complete. 2024-03-23T23:09:44.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.711+0000 7fc9993ef740 1 -- start start 2024-03-23T23:09:44.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.711+0000 7fc9993ef740 1 Processor -- start 2024-03-23T23:09:44.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.712+0000 7fc9993ef740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576771e5810 0x5576771f2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.712+0000 7fc9993ef740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e69e0 0x5576771f2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.712+0000 7fc9993ef740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576771e73a0 0x5576771f5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.712+0000 7fc9993ef740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9901a9330 con 0x5576771e5810 2024-03-23T23:09:44.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.712+0000 7fc9993ef740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9901aa630 con 0x5576771e73a0 2024-03-23T23:09:44.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.712+0000 7fc9993ef740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9901a9970 con 0x5576771e69e0 2024-03-23T23:09:44.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.712+0000 7fc9993ef740 10 client.0 _ll_register_callbacks cb 0x7fc97c009ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:44.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.712+0000 7fc98f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576771e5810 0x5576771f2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.712+0000 7fc98ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e69e0 0x5576771f2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.712+0000 7fc98f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576771e5810 0x5576771f2550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38946/0 (socket says 172.21.15.12:38946) 2024-03-23T23:09:44.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.712+0000 7fc98ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e69e0 0x5576771f2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:57620/0 (socket says 172.21.15.12:57620) 2024-03-23T23:09:44.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.712+0000 7fc98fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576771e73a0 0x5576771f5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.712+0000 7fc98f400640 1 -- 172.21.15.12:0/320633178 learned_addr learned my addr 172.21.15.12:0/320633178 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:44.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.712+0000 7fc98f400640 1 -- 172.21.15.12:0/320633178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e69e0 msgr2=0x5576771f2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:44.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.712+0000 7fc98f400640 1 --2- 172.21.15.12:0/320633178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e69e0 0x5576771f2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.712+0000 7fc98f400640 1 -- 172.21.15.12:0/320633178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576771e73a0 msgr2=0x5576771f5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:44.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.712+0000 7fc98f400640 1 --2- 172.21.15.12:0/320633178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576771e73a0 0x5576771f5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.713+0000 7fc98f400640 1 -- 172.21.15.12:0/320633178 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5576771f60a0 con 0x5576771e5810 2024-03-23T23:09:44.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.713+0000 7fc98fe00640 1 --2- 172.21.15.12:0/320633178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576771e73a0 0x5576771f5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:44.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.713+0000 7fc98ea00640 1 --2- 172.21.15.12:0/320633178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e69e0 0x5576771f2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:44.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.713+0000 7fc98f400640 1 --2- 172.21.15.12:0/320633178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576771e5810 0x5576771f2550 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc980006010 tx=0x7fc980009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0c0fa2717cfb82 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:44.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.713+0000 7fc979600640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:09:44.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.713+0000 7fc979600640 1 -- 172.21.15.12:0/320633178 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc980045070 con 0x5576771e5810 2024-03-23T23:09:44.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.713+0000 7fc979600640 1 -- 172.21.15.12:0/320633178 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fc980057050 con 0x5576771e5810 2024-03-23T23:09:44.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.713+0000 7fc9993ef740 1 -- 172.21.15.12:0/320633178 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc97c005550 con 0x5576771e5810 2024-03-23T23:09:44.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.713+0000 7fc979600640 1 -- 172.21.15.12:0/320633178 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc980057250 con 0x5576771e5810 2024-03-23T23:09:44.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.713+0000 7fc9993ef740 10 client.17391 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:44.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.713+0000 7fc9993ef740 1 -- 172.21.15.12:0/320633178 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x557677256a60 con 0x5576771e5810 2024-03-23T23:09:44.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.714+0000 7fc979600640 1 -- 172.21.15.12:0/320633178 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc97c005550 con 0x5576771e5810 2024-03-23T23:09:44.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.714+0000 7fc979600640 1 -- 172.21.15.12:0/320633178 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fc980005360 con 0x5576771e5810 2024-03-23T23:09:44.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.714+0000 7fc9993ef740 10 client.17391 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:44.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.714+0000 7fc9993ef740 10 client.17391 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:09:44.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.714+0000 7fc9993ef740 20 client.17391 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:44.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.714+0000 7fc9993ef740 1 --2- 172.21.15.12:0/320633178 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x557677257260 0x557677259670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.714+0000 7fc9993ef740 4 client.17391 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:09:44.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.714+0000 7fc9993ef740 1 -- 172.21.15.12:0/320633178 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557677151b50 con 0x557677257260 2024-03-23T23:09:44.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.714+0000 7fc98ea00640 1 --2- 172.21.15.12:0/320633178 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x557677257260 0x557677259670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.715+0000 7fc98ea00640 1 --2- 172.21.15.12:0/320633178 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x557677257260 0x557677259670 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=248aba7b8d87c2fb server_cookie=2471095ae076ac56 in_seq=0 out_seq=0 2024-03-23T23:09:44.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.715+0000 7fc979600640 10 client.17391 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:09:44.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.715+0000 7fc979600640 1 -- 172.21.15.12:0/320633178 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557677151b50 con 0x557677257260 2024-03-23T23:09:44.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.715+0000 7fc979600640 10 client.17391 handle_command_reply: tid=0 2024-03-23T23:09:44.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.715+0000 7fc9993ef740 1 client.17391 shutdown 2024-03-23T23:09:44.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.715+0000 7fc9993ef740 1 -- 172.21.15.12:0/320633178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576771e5810 msgr2=0x5576771f2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:44.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.715+0000 7fc9993ef740 1 --2- 172.21.15.12:0/320633178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576771e5810 0x5576771f2550 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc980006010 tx=0x7fc980009260 comp rx=0 tx=0).stop 2024-03-23T23:09:44.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.715+0000 7fc9993ef740 1 -- 172.21.15.12:0/320633178 shutdown_connections 2024-03-23T23:09:44.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.715+0000 7fc9993ef740 1 --2- 172.21.15.12:0/320633178 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x557677257260 0x557677259670 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.715+0000 7fc9993ef740 1 --2- 172.21.15.12:0/320633178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576771e73a0 0x5576771f5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.715+0000 7fc9993ef740 1 --2- 172.21.15.12:0/320633178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576771e69e0 0x5576771f2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.715+0000 7fc9993ef740 1 --2- 172.21.15.12:0/320633178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5576771e5810 0x5576771f2550 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.715+0000 7fc979600640 0 client.17391 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:09:44.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.715+0000 7fc9993ef740 1 -- 172.21.15.12:0/320633178 >> 172.21.15.12:0/320633178 conn(0x5576771e0c00 msgr2=0x5576771e46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:44.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.715+0000 7fc9993ef740 1 -- 172.21.15.12:0/320633178 shutdown_connections 2024-03-23T23:09:44.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.715+0000 7fc9993ef740 1 -- 172.21.15.12:0/320633178 wait complete. 2024-03-23T23:09:44.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.716+0000 7fc9993ef740 20 client.17391 trim_cache size 0 max 16384 2024-03-23T23:09:44.719 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:09:44.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.716+0000 7fc996600640 1 -- 172.21.15.12:0/2063357617 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc9740a2880 msgr2=0x7fc9740a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:44.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.716+0000 7fc996600640 1 --2- 172.21.15.12:0/2063357617 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc9740a2880 0x7fc9740a4d40 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc99019b0f0 tx=0x7fc98002df90 comp rx=0 tx=0).stop 2024-03-23T23:09:44.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.716+0000 7fc996600640 1 -- 172.21.15.12:0/2063357617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc990105470 msgr2=0x7fc9901a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:44.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.716+0000 7fc996600640 1 --2- 172.21.15.12:0/2063357617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc990105470 0x7fc9901a1190 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fc988009fd0 tx=0x7fc98800ef10 comp rx=0 tx=0).stop 2024-03-23T23:09:44.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.716+0000 7fc996600640 1 -- 172.21.15.12:0/2063357617 shutdown_connections 2024-03-23T23:09:44.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.716+0000 7fc996600640 1 --2- 172.21.15.12:0/2063357617 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc9740a2880 0x7fc9740a4d40 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.716+0000 7fc996600640 1 --2- 172.21.15.12:0/2063357617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc990105470 0x7fc9901a1190 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.716+0000 7fc996600640 1 --2- 172.21.15.12:0/2063357617 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc990104ab0 0x7fc99019a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.716+0000 7fc996600640 1 --2- 172.21.15.12:0/2063357617 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9901038c0 0x7fc990199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.716+0000 7fc996600640 1 -- 172.21.15.12:0/2063357617 >> 172.21.15.12:0/2063357617 conn(0x7fc9900fee50 msgr2=0x7fc990107e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:44.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.716+0000 7fc996600640 1 -- 172.21.15.12:0/2063357617 shutdown_connections 2024-03-23T23:09:44.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.716+0000 7fc996600640 1 -- 172.21.15.12:0/2063357617 wait complete. 2024-03-23T23:09:44.735 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:09:44.735 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-23T23:09:44.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.964+0000 7f8e6be00640 1 -- 172.21.15.12:0/4158649590 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8e640756a0 msgr2=0x7f8e64073af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:44.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.964+0000 7f8e6be00640 1 --2- 172.21.15.12:0/4158649590 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8e640756a0 0x7f8e64073af0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f8e54009900 tx=0x7f8e5402bc30 comp rx=0 tx=0).stop 2024-03-23T23:09:44.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.964+0000 7f8e6be00640 1 -- 172.21.15.12:0/4158649590 shutdown_connections 2024-03-23T23:09:44.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.964+0000 7f8e6be00640 1 --2- 172.21.15.12:0/4158649590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e64109580 0x7f8e6410b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.964+0000 7f8e6be00640 1 --2- 172.21.15.12:0/4158649590 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e640740c0 0x7f8e64074540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.964+0000 7f8e6be00640 1 --2- 172.21.15.12:0/4158649590 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8e640756a0 0x7f8e64073af0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.964+0000 7f8e6be00640 1 -- 172.21.15.12:0/4158649590 >> 172.21.15.12:0/4158649590 conn(0x7f8e640fcc40 msgr2=0x7f8e640ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:44.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.964+0000 7f8e6be00640 1 -- 172.21.15.12:0/4158649590 shutdown_connections 2024-03-23T23:09:44.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.964+0000 7f8e6be00640 1 -- 172.21.15.12:0/4158649590 wait complete. 2024-03-23T23:09:44.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.965+0000 7f8e6be00640 1 Processor -- start 2024-03-23T23:09:44.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.965+0000 7f8e6be00640 1 -- start start 2024-03-23T23:09:44.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.966+0000 7f8e6be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e640740c0 0x7f8e64199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.966+0000 7f8e6be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8e640756a0 0x7f8e6419a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.966+0000 7f8e6be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e64109580 0x7f8e641a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.966+0000 7f8e6be00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6410f580 con 0x7f8e640740c0 2024-03-23T23:09:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.966+0000 7f8e6be00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6410f400 con 0x7f8e64109580 2024-03-23T23:09:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.966+0000 7f8e6be00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6410f700 con 0x7f8e640756a0 2024-03-23T23:09:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.966+0000 7f8e69400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e640740c0 0x7f8e64199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.966+0000 7f8e68a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8e640756a0 0x7f8e6419a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.966+0000 7f8e69e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e64109580 0x7f8e641a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.966+0000 7f8e69400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e640740c0 0x7f8e64199bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38966/0 (socket says 172.21.15.12:38966) 2024-03-23T23:09:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.966+0000 7f8e68a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8e640756a0 0x7f8e6419a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:57638/0 (socket says 172.21.15.12:57638) 2024-03-23T23:09:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.966+0000 7f8e69400640 1 -- 172.21.15.12:0/3369639316 learned_addr learned my addr 172.21.15.12:0/3369639316 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.966+0000 7f8e69e00640 1 -- 172.21.15.12:0/3369639316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8e640756a0 msgr2=0x7f8e6419a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.966+0000 7f8e69e00640 1 --2- 172.21.15.12:0/3369639316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8e640756a0 0x7f8e6419a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.967+0000 7f8e69e00640 1 -- 172.21.15.12:0/3369639316 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e640740c0 msgr2=0x7f8e64199bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.967+0000 7f8e69e00640 1 --2- 172.21.15.12:0/3369639316 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e640740c0 0x7f8e64199bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.967+0000 7f8e69e00640 1 -- 172.21.15.12:0/3369639316 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e641a17d0 con 0x7f8e64109580 2024-03-23T23:09:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.967+0000 7f8e69400640 1 --2- 172.21.15.12:0/3369639316 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e640740c0 0x7f8e64199bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.967+0000 7f8e69e00640 1 --2- 172.21.15.12:0/3369639316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e64109580 0x7f8e641a1160 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c00e970 tx=0x7f8e5c00ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:44.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.967+0000 7f8e62000640 1 -- 172.21.15.12:0/3369639316 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c00ccc0 con 0x7f8e64109580 2024-03-23T23:09:44.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.967+0000 7f8e62000640 1 -- 172.21.15.12:0/3369639316 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e5c005020 con 0x7f8e64109580 2024-03-23T23:09:44.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.967+0000 7f8e6be00640 1 -- 172.21.15.12:0/3369639316 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e641a1ac0 con 0x7f8e64109580 2024-03-23T23:09:44.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.967+0000 7f8e62000640 1 -- 172.21.15.12:0/3369639316 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c004a00 con 0x7f8e64109580 2024-03-23T23:09:44.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.967+0000 7f8e6be00640 1 -- 172.21.15.12:0/3369639316 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e641a2060 con 0x7f8e64109580 2024-03-23T23:09:44.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.970+0000 7f8e62000640 1 -- 172.21.15.12:0/3369639316 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8e5c0051d0 con 0x7f8e64109580 2024-03-23T23:09:44.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.970+0000 7f8e6ea3b740 1 Processor -- start 2024-03-23T23:09:44.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.971+0000 7f8e6ea3b740 1 -- start start 2024-03-23T23:09:44.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.971+0000 7f8e6ea3b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d3f43f8810 0x55d3f43f8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.971+0000 7f8e6ea3b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f99e0 0x55d3f43f9e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.971+0000 7f8e6ea3b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3f43fa3a0 0x55d3f4400c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.971+0000 7f8e6ea3b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f435d2c0 con 0x55d3f43f8810 2024-03-23T23:09:44.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.971+0000 7f8e6ea3b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f4305c90 con 0x55d3f43f99e0 2024-03-23T23:09:44.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.971+0000 7f8e6ea3b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f4359360 con 0x55d3f43fa3a0 2024-03-23T23:09:44.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.971+0000 7f8e69400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f99e0 0x55d3f43f9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.971+0000 7f8e68a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d3f43f8810 0x55d3f43f8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.971+0000 7f8e69400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f99e0 0x55d3f43f9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:33666/0 (socket says 172.21.15.12:33666) 2024-03-23T23:09:44.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.971+0000 7f8e69400640 1 -- 172.21.15.12:0/4069664685 learned_addr learned my addr 172.21.15.12:0/4069664685 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:44.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.971+0000 7f8e68a00640 1 --2- 172.21.15.12:0/4069664685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3f43fa3a0 0x55d3f4400c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.972+0000 7f8e69400640 1 -- 172.21.15.12:0/4069664685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3f43fa3a0 msgr2=0x55d3f4400c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:44.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.972+0000 7f8e69400640 1 --2- 172.21.15.12:0/4069664685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3f43fa3a0 0x55d3f4400c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.972+0000 7f8e69400640 1 -- 172.21.15.12:0/4069664685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d3f43f8810 msgr2=0x55d3f43f8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:44.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.972+0000 7f8e69400640 1 --2- 172.21.15.12:0/4069664685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d3f43f8810 0x55d3f43f8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.972+0000 7f8e69400640 1 -- 172.21.15.12:0/4069664685 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3f432a080 con 0x55d3f43f99e0 2024-03-23T23:09:44.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.972+0000 7f8e68a00640 1 --2- 172.21.15.12:0/4069664685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3f43fa3a0 0x55d3f4400c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:09:44.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.972+0000 7f8e69400640 1 --2- 172.21.15.12:0/4069664685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f99e0 0x55d3f43f9e60 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f8e5402c170 tx=0x7f8e54009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2d7c01959adfca2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:44.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.972+0000 7f8e62000640 1 --2- 172.21.15.12:0/3369639316 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8e480a2880 0x7f8e480a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.972+0000 7f8e5a600640 1 -- 172.21.15.12:0/4069664685 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e54045070 con 0x55d3f43f99e0 2024-03-23T23:09:44.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.972+0000 7f8e5a600640 1 -- 172.21.15.12:0/4069664685 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e5402cd30 con 0x55d3f43f99e0 2024-03-23T23:09:44.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.972+0000 7f8e5a600640 1 -- 172.21.15.12:0/4069664685 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e540050d0 con 0x55d3f43f99e0 2024-03-23T23:09:44.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.973+0000 7f8e62000640 1 -- 172.21.15.12:0/3369639316 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8e5c0ce970 con 0x7f8e64109580 2024-03-23T23:09:44.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.973+0000 7f8e68a00640 1 --2- 172.21.15.12:0/3369639316 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8e480a2880 0x7f8e480a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.973+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/4069664685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f99e0 msgr2=0x55d3f43f9e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:44.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.973+0000 7f8e6ea3b740 1 --2- 172.21.15.12:0/4069664685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f99e0 0x55d3f43f9e60 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f8e5402c170 tx=0x7f8e54009560 comp rx=0 tx=0).stop 2024-03-23T23:09:44.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.973+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/4069664685 shutdown_connections 2024-03-23T23:09:44.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.973+0000 7f8e6ea3b740 1 --2- 172.21.15.12:0/4069664685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3f43fa3a0 0x55d3f4400c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.973+0000 7f8e6ea3b740 1 --2- 172.21.15.12:0/4069664685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f99e0 0x55d3f43f9e60 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.973+0000 7f8e6ea3b740 1 --2- 172.21.15.12:0/4069664685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d3f43f8810 0x55d3f43f8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.973+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/4069664685 >> 172.21.15.12:0/4069664685 conn(0x55d3f43f3c00 msgr2=0x55d3f43f4000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:44.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.973+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/4069664685 shutdown_connections 2024-03-23T23:09:44.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.973+0000 7f8e68a00640 1 --2- 172.21.15.12:0/3369639316 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8e480a2880 0x7f8e480a4d40 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x55d3f43f98e0 tx=0x7f8e50015040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:44.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.973+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/4069664685 wait complete. 2024-03-23T23:09:44.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.974+0000 7f8e6ea3b740 1 -- start start 2024-03-23T23:09:44.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.974+0000 7f8e6ea3b740 1 Processor -- start 2024-03-23T23:09:44.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.974+0000 7f8e6ea3b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f8810 0x55d3f44029a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.974+0000 7f8e6ea3b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d3f43f99e0 0x55d3f4402ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.974+0000 7f8e6ea3b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3f43fa3a0 0x55d3f4409f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.974+0000 7f8e6ea3b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f4401350 con 0x55d3f43f99e0 2024-03-23T23:09:44.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.974+0000 7f8e6ea3b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f43190d0 con 0x55d3f43f8810 2024-03-23T23:09:44.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.974+0000 7f8e6ea3b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f44014d0 con 0x55d3f43fa3a0 2024-03-23T23:09:44.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.975+0000 7f8e6ea3b740 10 client.0 _ll_register_callbacks cb 0x55d3f431de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:44.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.975+0000 7f8e69400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f8810 0x55d3f44029a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.975+0000 7f8e69e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3f43fa3a0 0x55d3f4409f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.975+0000 7f8e69400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f8810 0x55d3f44029a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:33674/0 (socket says 172.21.15.12:33674) 2024-03-23T23:09:44.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.975+0000 7f8e69400640 1 -- 172.21.15.12:0/923004162 learned_addr learned my addr 172.21.15.12:0/923004162 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:44.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.975+0000 7f8e68a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d3f43f99e0 0x55d3f4402ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.975+0000 7f8e69e00640 1 -- 172.21.15.12:0/923004162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f8810 msgr2=0x55d3f44029a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:44.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.975+0000 7f8e69e00640 1 --2- 172.21.15.12:0/923004162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f8810 0x55d3f44029a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.975+0000 7f8e69e00640 1 -- 172.21.15.12:0/923004162 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d3f43f99e0 msgr2=0x55d3f4402ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:44.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.975+0000 7f8e69e00640 1 --2- 172.21.15.12:0/923004162 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d3f43f99e0 0x55d3f4402ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.975+0000 7f8e69e00640 1 -- 172.21.15.12:0/923004162 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d3f40713f0 con 0x55d3f43fa3a0 2024-03-23T23:09:44.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.975+0000 7f8e69400640 1 --2- 172.21.15.12:0/923004162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f8810 0x55d3f44029a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:44.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.975+0000 7f8e69e00640 1 --2- 172.21.15.12:0/923004162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3f43fa3a0 0x55d3f4409f20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c0cc420 tx=0x7f8e5c0056a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c776607f3ba4739 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:44.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.975+0000 7f8e4ea00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:09:44.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.976+0000 7f8e4ea00640 1 -- 172.21.15.12:0/923004162 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c0b4060 con 0x55d3f43fa3a0 2024-03-23T23:09:44.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.976+0000 7f8e4ea00640 1 -- 172.21.15.12:0/923004162 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8e5c0bb070 con 0x55d3f43fa3a0 2024-03-23T23:09:44.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.976+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/923004162 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8e5402cd30 con 0x55d3f43fa3a0 2024-03-23T23:09:44.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.976+0000 7f8e4ea00640 1 -- 172.21.15.12:0/923004162 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e5c01c040 con 0x55d3f43fa3a0 2024-03-23T23:09:44.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.976+0000 7f8e6ea3b740 10 client.27508 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:44.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.976+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/923004162 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55d3f413b7d0 con 0x55d3f43fa3a0 2024-03-23T23:09:44.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.976+0000 7f8e4ea00640 1 -- 172.21.15.12:0/923004162 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e5402cd30 con 0x55d3f43fa3a0 2024-03-23T23:09:44.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.976+0000 7f8e4ea00640 1 -- 172.21.15.12:0/923004162 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8e5c004ba0 con 0x55d3f43fa3a0 2024-03-23T23:09:44.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.976+0000 7f8e6ea3b740 10 client.27508 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:44.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.976+0000 7f8e6ea3b740 10 client.27508 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:09:44.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.976+0000 7f8e6ea3b740 20 client.27508 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:44.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.976+0000 7f8e6ea3b740 1 --2- 172.21.15.12:0/923004162 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55d3f440e480 0x55d3f4410920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.977+0000 7f8e6ea3b740 4 client.27508 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:09:44.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.977+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/923004162 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d3f436a800 con 0x55d3f440e480 2024-03-23T23:09:44.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.977+0000 7f8e69400640 1 --2- 172.21.15.12:0/923004162 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55d3f440e480 0x55d3f4410920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.977+0000 7f8e69400640 1 --2- 172.21.15.12:0/923004162 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55d3f440e480 0x55d3f4410920 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=713c6c48a6e222da server_cookie=cb50be3d16b6a98 in_seq=0 out_seq=0 2024-03-23T23:09:44.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.977+0000 7f8e4ea00640 10 client.27508 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:09:44.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.978+0000 7f8e4ea00640 1 -- 172.21.15.12:0/923004162 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d3f436a800 con 0x55d3f440e480 2024-03-23T23:09:44.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.978+0000 7f8e4ea00640 10 client.27508 handle_command_reply: tid=0 2024-03-23T23:09:44.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.978+0000 7f8e6ea3b740 1 client.27508 shutdown 2024-03-23T23:09:44.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.978+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/923004162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3f43fa3a0 msgr2=0x55d3f4409f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:44.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.978+0000 7f8e6ea3b740 1 --2- 172.21.15.12:0/923004162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3f43fa3a0 0x55d3f4409f20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c0cc420 tx=0x7f8e5c0056a0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.978+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/923004162 shutdown_connections 2024-03-23T23:09:44.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.978+0000 7f8e6ea3b740 1 --2- 172.21.15.12:0/923004162 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55d3f440e480 0x55d3f4410920 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.978+0000 7f8e6ea3b740 1 --2- 172.21.15.12:0/923004162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3f43fa3a0 0x55d3f4409f20 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.978+0000 7f8e6ea3b740 1 --2- 172.21.15.12:0/923004162 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d3f43f99e0 0x55d3f4402ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.978+0000 7f8e6ea3b740 1 --2- 172.21.15.12:0/923004162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f8810 0x55d3f44029a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.978+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/923004162 >> 172.21.15.12:0/923004162 conn(0x55d3f43f3c00 msgr2=0x55d3f43f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:44.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.978+0000 7f8e4ea00640 0 client.27508 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:09:44.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.979+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/923004162 shutdown_connections 2024-03-23T23:09:44.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.979+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/923004162 wait complete. 2024-03-23T23:09:44.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.979+0000 7f8e6ea3b740 20 client.27508 trim_cache size 0 max 16384 2024-03-23T23:09:44.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.992+0000 7f8e6ea3b740 1 Processor -- start 2024-03-23T23:09:44.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.993+0000 7f8e6ea3b740 1 -- start start 2024-03-23T23:09:44.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.993+0000 7f8e6ea3b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3f43f8810 0x55d3f43f2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.993+0000 7f8e6ea3b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f99e0 0x55d3f4409ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.993+0000 7f8e6ea3b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d3f43fa3a0 0x55d3f43f2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.993+0000 7f8e6ea3b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e641ad130 con 0x55d3f43fa3a0 2024-03-23T23:09:44.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.993+0000 7f8e6ea3b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e641aac30 con 0x55d3f43f99e0 2024-03-23T23:09:44.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.993+0000 7f8e6ea3b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e641a8cd0 con 0x55d3f43f8810 2024-03-23T23:09:44.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.993+0000 7f8e68a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f99e0 0x55d3f4409ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.993+0000 7f8e68a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f99e0 0x55d3f4409ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:33684/0 (socket says 172.21.15.12:33684) 2024-03-23T23:09:44.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.993+0000 7f8e68a00640 1 -- 172.21.15.12:0/4184480727 learned_addr learned my addr 172.21.15.12:0/4184480727 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:44.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.993+0000 7f8e69e00640 1 --2- 172.21.15.12:0/4184480727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d3f43fa3a0 0x55d3f43f2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.993+0000 7f8e69400640 1 --2- 172.21.15.12:0/4184480727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3f43f8810 0x55d3f43f2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.994+0000 7f8e68a00640 1 -- 172.21.15.12:0/4184480727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3f43f8810 msgr2=0x55d3f43f2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:44.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.994+0000 7f8e68a00640 1 --2- 172.21.15.12:0/4184480727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3f43f8810 0x55d3f43f2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.994+0000 7f8e68a00640 1 -- 172.21.15.12:0/4184480727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d3f43fa3a0 msgr2=0x55d3f43f2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:44.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.994+0000 7f8e68a00640 1 --2- 172.21.15.12:0/4184480727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d3f43fa3a0 0x55d3f43f2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.994+0000 7f8e68a00640 1 -- 172.21.15.12:0/4184480727 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3f43f9460 con 0x55d3f43f99e0 2024-03-23T23:09:44.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.994+0000 7f8e68a00640 1 --2- 172.21.15.12:0/4184480727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f99e0 0x55d3f4409ee0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f8e50005fd0 tx=0x7f8e50004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf0205f2dfa47612 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:44.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.994+0000 7f8e59c00640 1 -- 172.21.15.12:0/4184480727 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e50004240 con 0x55d3f43f99e0 2024-03-23T23:09:44.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.994+0000 7f8e59c00640 1 -- 172.21.15.12:0/4184480727 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e50028030 con 0x55d3f43f99e0 2024-03-23T23:09:44.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.994+0000 7f8e59c00640 1 -- 172.21.15.12:0/4184480727 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e50024d20 con 0x55d3f43f99e0 2024-03-23T23:09:44.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.995+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/4184480727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f99e0 msgr2=0x55d3f4409ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:44.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.995+0000 7f8e6ea3b740 1 --2- 172.21.15.12:0/4184480727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f99e0 0x55d3f4409ee0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f8e50005fd0 tx=0x7f8e50004ab0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.995+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/4184480727 shutdown_connections 2024-03-23T23:09:44.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.995+0000 7f8e6ea3b740 1 --2- 172.21.15.12:0/4184480727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d3f43fa3a0 0x55d3f43f2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.995+0000 7f8e6ea3b740 1 --2- 172.21.15.12:0/4184480727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f99e0 0x55d3f4409ee0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.995+0000 7f8e6ea3b740 1 --2- 172.21.15.12:0/4184480727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3f43f8810 0x55d3f43f2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.995+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/4184480727 >> 172.21.15.12:0/4184480727 conn(0x55d3f43f3c00 msgr2=0x55d3f44029a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:44.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.995+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/4184480727 shutdown_connections 2024-03-23T23:09:44.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.995+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/4184480727 wait complete. 2024-03-23T23:09:44.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.995+0000 7f8e6ea3b740 1 -- start start 2024-03-23T23:09:44.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.995+0000 7f8e6ea3b740 1 Processor -- start 2024-03-23T23:09:44.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.996+0000 7f8e6ea3b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3f43f8810 0x55d3f4405550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.996+0000 7f8e6ea3b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f99e0 0x55d3f4405a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.996+0000 7f8e6ea3b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d3f43fa3a0 0x55d3f4408a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:44.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.996+0000 7f8e6ea3b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e641a92d0 con 0x55d3f43fa3a0 2024-03-23T23:09:44.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.996+0000 7f8e6ea3b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e641aa5d0 con 0x55d3f43f99e0 2024-03-23T23:09:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.996+0000 7f8e6ea3b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e641a9910 con 0x55d3f43f8810 2024-03-23T23:09:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.996+0000 7f8e6ea3b740 10 client.0 _ll_register_callbacks cb 0x7f8e5c09ac20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.996+0000 7f8e68a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f99e0 0x55d3f4405a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.996+0000 7f8e68a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f99e0 0x55d3f4405a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:33688/0 (socket says 172.21.15.12:33688) 2024-03-23T23:09:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.996+0000 7f8e69400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3f43f8810 0x55d3f4405550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.996+0000 7f8e68a00640 1 -- 172.21.15.12:0/3504872277 learned_addr learned my addr 172.21.15.12:0/3504872277 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.996+0000 7f8e69e00640 1 --2- 172.21.15.12:0/3504872277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d3f43fa3a0 0x55d3f4408a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.997+0000 7f8e68a00640 1 -- 172.21.15.12:0/3504872277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3f43f8810 msgr2=0x55d3f4405550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.997+0000 7f8e68a00640 1 --2- 172.21.15.12:0/3504872277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3f43f8810 0x55d3f4405550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.997+0000 7f8e68a00640 1 -- 172.21.15.12:0/3504872277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d3f43fa3a0 msgr2=0x55d3f4408a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.997+0000 7f8e68a00640 1 --2- 172.21.15.12:0/3504872277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d3f43fa3a0 0x55d3f4408a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.997+0000 7f8e68a00640 1 -- 172.21.15.12:0/3504872277 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d3f4409050 con 0x55d3f43f99e0 2024-03-23T23:09:44.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.997+0000 7f8e69400640 1 --2- 172.21.15.12:0/3504872277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3f43f8810 0x55d3f4405550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:44.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.997+0000 7f8e68a00640 1 --2- 172.21.15.12:0/3504872277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f99e0 0x55d3f4405a90 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f8e50005ee0 tx=0x7f8e50024ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23cd60f871fdd95b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:44.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.997+0000 7f8e4ea00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:09:44.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.997+0000 7f8e4ea00640 1 -- 172.21.15.12:0/3504872277 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e500041b0 con 0x55d3f43f99e0 2024-03-23T23:09:44.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.997+0000 7f8e4ea00640 1 -- 172.21.15.12:0/3504872277 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8e50004350 con 0x55d3f43f99e0 2024-03-23T23:09:44.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.997+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/3504872277 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8e50028030 con 0x55d3f43f99e0 2024-03-23T23:09:44.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.998+0000 7f8e4ea00640 1 -- 172.21.15.12:0/3504872277 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e50004550 con 0x55d3f43f99e0 2024-03-23T23:09:44.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.998+0000 7f8e6ea3b740 10 client.27548 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:44.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.998+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/3504872277 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55d3f4469db0 con 0x55d3f43f99e0 2024-03-23T23:09:44.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.998+0000 7f8e4ea00640 1 -- 172.21.15.12:0/3504872277 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e50028030 con 0x55d3f43f99e0 2024-03-23T23:09:44.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.998+0000 7f8e4ea00640 1 -- 172.21.15.12:0/3504872277 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8e50035020 con 0x55d3f43f99e0 2024-03-23T23:09:45.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.998+0000 7f8e6ea3b740 10 client.27548 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:45.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.998+0000 7f8e6ea3b740 10 client.27548 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:09:45.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.998+0000 7f8e6ea3b740 20 client.27548 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:45.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.998+0000 7f8e6ea3b740 1 --2- 172.21.15.12:0/3504872277 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55d3f446a530 0x55d3f446c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:45.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.998+0000 7f8e6ea3b740 4 client.27548 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:09:45.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.998+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/3504872277 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d3f4364b50 con 0x55d3f446a530 2024-03-23T23:09:45.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.998+0000 7f8e69400640 1 --2- 172.21.15.12:0/3504872277 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55d3f446a530 0x55d3f446c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:45.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.999+0000 7f8e69400640 1 --2- 172.21.15.12:0/3504872277 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55d3f446a530 0x55d3f446c950 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=44ced7e6bc46a94e server_cookie=1ef1ff5fc7a25325 in_seq=0 out_seq=0 2024-03-23T23:09:45.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.999+0000 7f8e4ea00640 10 client.27548 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:09:45.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.999+0000 7f8e4ea00640 1 -- 172.21.15.12:0/3504872277 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d3f4364b50 con 0x55d3f446a530 2024-03-23T23:09:45.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.999+0000 7f8e4ea00640 10 client.27548 handle_command_reply: tid=0 2024-03-23T23:09:45.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.999+0000 7f8e6ea3b740 1 client.27548 shutdown 2024-03-23T23:09:45.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.999+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/3504872277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f99e0 msgr2=0x55d3f4405a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:45.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.999+0000 7f8e6ea3b740 1 --2- 172.21.15.12:0/3504872277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f99e0 0x55d3f4405a90 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f8e50005ee0 tx=0x7f8e50024ee0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.999+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/3504872277 shutdown_connections 2024-03-23T23:09:45.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.999+0000 7f8e6ea3b740 1 --2- 172.21.15.12:0/3504872277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d3f43fa3a0 0x55d3f4408a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.999+0000 7f8e6ea3b740 1 --2- 172.21.15.12:0/3504872277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f43f99e0 0x55d3f4405a90 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.999+0000 7f8e6ea3b740 1 --2- 172.21.15.12:0/3504872277 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x55d3f446a530 0x55d3f446c950 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.999+0000 7f8e6ea3b740 1 --2- 172.21.15.12:0/3504872277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3f43f8810 0x55d3f4405550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.999+0000 7f8e4ea00640 0 client.27548 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:09:45.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:44.999+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/3504872277 >> 172.21.15.12:0/3504872277 conn(0x55d3f43f3c00 msgr2=0x55d3f43f6e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:45.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.000+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/3504872277 shutdown_connections 2024-03-23T23:09:45.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.000+0000 7f8e6ea3b740 1 -- 172.21.15.12:0/3504872277 wait complete. 2024-03-23T23:09:45.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.000+0000 7f8e6ea3b740 20 client.27548 trim_cache size 0 max 16384 2024-03-23T23:09:45.003 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:09:45.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.000+0000 7f8e6be00640 1 -- 172.21.15.12:0/3369639316 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8e480a2880 msgr2=0x7f8e480a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:45.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.000+0000 7f8e6be00640 1 --2- 172.21.15.12:0/3369639316 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8e480a2880 0x7f8e480a4d40 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x55d3f43f98e0 tx=0x7f8e50015040 comp rx=0 tx=0).stop 2024-03-23T23:09:45.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.000+0000 7f8e6be00640 1 -- 172.21.15.12:0/3369639316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e64109580 msgr2=0x7f8e641a1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:45.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.000+0000 7f8e6be00640 1 --2- 172.21.15.12:0/3369639316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e64109580 0x7f8e641a1160 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c00e970 tx=0x7f8e5c00ee30 comp rx=0 tx=0).stop 2024-03-23T23:09:45.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.001+0000 7f8e6be00640 1 -- 172.21.15.12:0/3369639316 shutdown_connections 2024-03-23T23:09:45.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.001+0000 7f8e6be00640 1 --2- 172.21.15.12:0/3369639316 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8e480a2880 0x7f8e480a4d40 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.001+0000 7f8e6be00640 1 --2- 172.21.15.12:0/3369639316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e64109580 0x7f8e641a1160 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.001+0000 7f8e6be00640 1 --2- 172.21.15.12:0/3369639316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8e640756a0 0x7f8e6419a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.001+0000 7f8e6be00640 1 --2- 172.21.15.12:0/3369639316 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e640740c0 0x7f8e64199bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.001+0000 7f8e6be00640 1 -- 172.21.15.12:0/3369639316 >> 172.21.15.12:0/3369639316 conn(0x7f8e640fcc40 msgr2=0x7f8e641054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:45.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.001+0000 7f8e6be00640 1 -- 172.21.15.12:0/3369639316 shutdown_connections 2024-03-23T23:09:45.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.001+0000 7f8e6be00640 1 -- 172.21.15.12:0/3369639316 wait complete. 2024-03-23T23:09:45.015 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:44 smithi012 ceph-mon[31695]: pgmap v628: 97 pgs: 97 active+clean; 1.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:09:45.015 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:44 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1319018529' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:09:45.017 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:09:45.017 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-23T23:09:45.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:44 smithi173 ceph-mon[36116]: pgmap v628: 97 pgs: 97 active+clean; 1.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:09:45.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:44 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1319018529' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:09:45.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:44 smithi203 ceph-mon[36693]: pgmap v628: 97 pgs: 97 active+clean; 1.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:09:45.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:44 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1319018529' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:09:45.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.233+0000 7f4c39a00640 1 -- 172.21.15.12:0/2443020877 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c341038c0 msgr2=0x7f4c34103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:45.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.233+0000 7f4c39a00640 1 --2- 172.21.15.12:0/2443020877 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c341038c0 0x7f4c34103ce0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4c24009900 tx=0x7f4c2402bc30 comp rx=0 tx=0).stop 2024-03-23T23:09:45.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.233+0000 7f4c39a00640 1 -- 172.21.15.12:0/2443020877 shutdown_connections 2024-03-23T23:09:45.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.233+0000 7f4c39a00640 1 --2- 172.21.15.12:0/2443020877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c34105470 0x7f4c3410bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.233+0000 7f4c39a00640 1 --2- 172.21.15.12:0/2443020877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c34104ab0 0x7f4c34104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.233+0000 7f4c39a00640 1 --2- 172.21.15.12:0/2443020877 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c341038c0 0x7f4c34103ce0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.233+0000 7f4c39a00640 1 -- 172.21.15.12:0/2443020877 >> 172.21.15.12:0/2443020877 conn(0x7f4c340fee50 msgr2=0x7f4c34101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:45.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.233+0000 7f4c39a00640 1 -- 172.21.15.12:0/2443020877 shutdown_connections 2024-03-23T23:09:45.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.233+0000 7f4c39a00640 1 -- 172.21.15.12:0/2443020877 wait complete. 2024-03-23T23:09:45.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.234+0000 7f4c39a00640 1 Processor -- start 2024-03-23T23:09:45.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.234+0000 7f4c39a00640 1 -- start start 2024-03-23T23:09:45.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.235+0000 7f4c39a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c341038c0 0x7f4c34199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:45.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.235+0000 7f4c39a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c34104ab0 0x7f4c3419a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:45.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.235+0000 7f4c39a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c34105470 0x7f4c341a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:45.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.235+0000 7f4c39a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3410f560 con 0x7f4c34105470 2024-03-23T23:09:45.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.235+0000 7f4c39a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3410f3e0 con 0x7f4c34104ab0 2024-03-23T23:09:45.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.235+0000 7f4c39a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3410f6e0 con 0x7f4c341038c0 2024-03-23T23:09:45.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.235+0000 7f4c32a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c341038c0 0x7f4c34199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:45.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.235+0000 7f4c32000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c34104ab0 0x7f4c3419a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:45.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.235+0000 7f4c32000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c34104ab0 0x7f4c3419a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:33700/0 (socket says 172.21.15.12:33700) 2024-03-23T23:09:45.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.235+0000 7f4c32000640 1 -- 172.21.15.12:0/3410281993 learned_addr learned my addr 172.21.15.12:0/3410281993 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:45.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.235+0000 7f4c32a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c341038c0 0x7f4c34199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:57684/0 (socket says 172.21.15.12:57684) 2024-03-23T23:09:45.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.235+0000 7f4c33400640 1 --2- 172.21.15.12:0/3410281993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c34105470 0x7f4c341a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:45.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.236+0000 7f4c32000640 1 -- 172.21.15.12:0/3410281993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c341038c0 msgr2=0x7f4c34199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:45.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.236+0000 7f4c32000640 1 --2- 172.21.15.12:0/3410281993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c341038c0 0x7f4c34199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.236+0000 7f4c32000640 1 -- 172.21.15.12:0/3410281993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c34105470 msgr2=0x7f4c341a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:45.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.236+0000 7f4c32000640 1 --2- 172.21.15.12:0/3410281993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c34105470 0x7f4c341a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.236+0000 7f4c32000640 1 -- 172.21.15.12:0/3410281993 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c341a1800 con 0x7f4c34104ab0 2024-03-23T23:09:45.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.236+0000 7f4c32a00640 1 --2- 172.21.15.12:0/3410281993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c341038c0 0x7f4c34199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:45.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.236+0000 7f4c32000640 1 --2- 172.21.15.12:0/3410281993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c34104ab0 0x7f4c3419a120 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f4c2000e9f0 tx=0x7f4c2000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:45.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.236+0000 7f4c2b400640 1 -- 172.21.15.12:0/3410281993 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2000cd80 con 0x7f4c34104ab0 2024-03-23T23:09:45.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.236+0000 7f4c2b400640 1 -- 172.21.15.12:0/3410281993 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c20010910 con 0x7f4c34104ab0 2024-03-23T23:09:45.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.236+0000 7f4c39a00640 1 -- 172.21.15.12:0/3410281993 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c341a1af0 con 0x7f4c34104ab0 2024-03-23T23:09:45.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.236+0000 7f4c2b400640 1 -- 172.21.15.12:0/3410281993 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c20018990 con 0x7f4c34104ab0 2024-03-23T23:09:45.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.236+0000 7f4c39a00640 1 -- 172.21.15.12:0/3410281993 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c341a2090 con 0x7f4c34104ab0 2024-03-23T23:09:45.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.241+0000 7f4c3c7b0740 1 Processor -- start 2024-03-23T23:09:45.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.241+0000 7f4c3c7b0740 1 -- start start 2024-03-23T23:09:45.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.242+0000 7f4c3c7b0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f7097810 0x55b5f7097c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:45.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.242+0000 7f4c3c7b0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5f70989e0 0x55b5f7098e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:45.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.242+0000 7f4c3c7b0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5f70993a0 0x55b5f709fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:45.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.242+0000 7f4c3c7b0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5f6ffc2c0 con 0x55b5f70993a0 2024-03-23T23:09:45.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.242+0000 7f4c3c7b0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5f6fa4c90 con 0x55b5f7097810 2024-03-23T23:09:45.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.242+0000 7f4c3c7b0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5f6ff8360 con 0x55b5f70989e0 2024-03-23T23:09:45.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.242+0000 7f4c2b400640 1 -- 172.21.15.12:0/3410281993 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4c20010ac0 con 0x7f4c34104ab0 2024-03-23T23:09:45.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.242+0000 7f4c33400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f7097810 0x55b5f7097c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:45.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.242+0000 7f4c32a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5f70989e0 0x55b5f7098e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:45.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.242+0000 7f4c33400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f7097810 0x55b5f7097c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:33714/0 (socket says 172.21.15.12:33714) 2024-03-23T23:09:45.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.242+0000 7f4c33400640 1 -- 172.21.15.12:0/3228954585 learned_addr learned my addr 172.21.15.12:0/3228954585 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:45.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.242+0000 7f4c32000640 1 --2- 172.21.15.12:0/3228954585 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5f70993a0 0x55b5f709fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:45.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.242+0000 7f4c33400640 1 -- 172.21.15.12:0/3228954585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5f70989e0 msgr2=0x55b5f7098e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:45.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.242+0000 7f4c33400640 1 --2- 172.21.15.12:0/3228954585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5f70989e0 0x55b5f7098e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.242+0000 7f4c33400640 1 -- 172.21.15.12:0/3228954585 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5f70993a0 msgr2=0x55b5f709fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:45.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.242+0000 7f4c33400640 1 --2- 172.21.15.12:0/3228954585 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5f70993a0 0x55b5f709fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.242+0000 7f4c33400640 1 -- 172.21.15.12:0/3228954585 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5f6fc9080 con 0x55b5f7097810 2024-03-23T23:09:45.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.243+0000 7f4c33400640 1 --2- 172.21.15.12:0/3228954585 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f7097810 0x55b5f7097c30 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c0039f0 tx=0x7f4c2c003eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=211793dd217b7a20 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:45.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.243+0000 7f4c1fe00640 1 -- 172.21.15.12:0/3228954585 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c01a470 con 0x55b5f7097810 2024-03-23T23:09:45.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.243+0000 7f4c1fe00640 1 -- 172.21.15.12:0/3228954585 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c2c015070 con 0x55b5f7097810 2024-03-23T23:09:45.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.243+0000 7f4c1fe00640 1 -- 172.21.15.12:0/3228954585 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c0108a0 con 0x55b5f7097810 2024-03-23T23:09:45.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.243+0000 7f4c2b400640 1 --2- 172.21.15.12:0/3410281993 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4c180a2880 0x7f4c180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:45.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.244+0000 7f4c2b400640 1 -- 172.21.15.12:0/3410281993 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4c200ced40 con 0x7f4c34104ab0 2024-03-23T23:09:45.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.244+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/3228954585 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f7097810 msgr2=0x55b5f7097c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:45.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.244+0000 7f4c32a00640 1 --2- 172.21.15.12:0/3410281993 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4c180a2880 0x7f4c180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:45.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.244+0000 7f4c3c7b0740 1 --2- 172.21.15.12:0/3228954585 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f7097810 0x55b5f7097c30 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c0039f0 tx=0x7f4c2c003eb0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.244+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/3228954585 shutdown_connections 2024-03-23T23:09:45.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.244+0000 7f4c3c7b0740 1 --2- 172.21.15.12:0/3228954585 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5f70993a0 0x55b5f709fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.244+0000 7f4c3c7b0740 1 --2- 172.21.15.12:0/3228954585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5f70989e0 0x55b5f7098e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.244+0000 7f4c3c7b0740 1 --2- 172.21.15.12:0/3228954585 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f7097810 0x55b5f7097c30 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.244+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/3228954585 >> 172.21.15.12:0/3228954585 conn(0x55b5f7092c00 msgr2=0x55b5f7093000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:45.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.244+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/3228954585 shutdown_connections 2024-03-23T23:09:45.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.244+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/3228954585 wait complete. 2024-03-23T23:09:45.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.245+0000 7f4c32a00640 1 --2- 172.21.15.12:0/3410281993 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4c180a2880 0x7f4c180a4d40 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4c2402c170 tx=0x7f4c2402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:45.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.245+0000 7f4c3c7b0740 1 -- start start 2024-03-23T23:09:45.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.245+0000 7f4c3c7b0740 1 Processor -- start 2024-03-23T23:09:45.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.245+0000 7f4c3c7b0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f7097810 0x55b5f70a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:45.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.245+0000 7f4c3c7b0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5f70989e0 0x55b5f70a1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:45.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.245+0000 7f4c3c7b0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5f70993a0 0x55b5f70a8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:45.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.246+0000 7f4c3c7b0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5f70a03b0 con 0x55b5f70989e0 2024-03-23T23:09:45.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.246+0000 7f4c3c7b0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5f6fb80d0 con 0x55b5f7097810 2024-03-23T23:09:45.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.246+0000 7f4c3c7b0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5f70a0530 con 0x55b5f70993a0 2024-03-23T23:09:45.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.246+0000 7f4c3c7b0740 10 client.0 _ll_register_callbacks cb 0x55b5f6fbce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:45.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.246+0000 7f4c32a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f7097810 0x55b5f70a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:45.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.246+0000 7f4c32000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5f70989e0 0x55b5f70a1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:45.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.246+0000 7f4c32a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f7097810 0x55b5f70a1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:33728/0 (socket says 172.21.15.12:33728) 2024-03-23T23:09:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.246+0000 7f4c32a00640 1 -- 172.21.15.12:0/468097690 learned_addr learned my addr 172.21.15.12:0/468097690 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.246+0000 7f4c33400640 1 --2- 172.21.15.12:0/468097690 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5f70993a0 0x55b5f70a8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.246+0000 7f4c32a00640 1 -- 172.21.15.12:0/468097690 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5f70993a0 msgr2=0x55b5f70a8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.246+0000 7f4c32a00640 1 --2- 172.21.15.12:0/468097690 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5f70993a0 0x55b5f70a8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.246+0000 7f4c32a00640 1 -- 172.21.15.12:0/468097690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5f70989e0 msgr2=0x55b5f70a1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.246+0000 7f4c32a00640 1 --2- 172.21.15.12:0/468097690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5f70989e0 0x55b5f70a1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.246+0000 7f4c32a00640 1 -- 172.21.15.12:0/468097690 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b5f6d103f0 con 0x55b5f7097810 2024-03-23T23:09:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.246+0000 7f4c32000640 1 --2- 172.21.15.12:0/468097690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5f70989e0 0x55b5f70a1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.247+0000 7f4c32a00640 1 --2- 172.21.15.12:0/468097690 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f7097810 0x55b5f70a1a50 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4c2402c240 tx=0x7f4c24009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5eabdfdad13d2e03 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.247+0000 7f4c1cc00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:09:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.247+0000 7f4c1cc00640 1 -- 172.21.15.12:0/468097690 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c24046070 con 0x55b5f7097810 2024-03-23T23:09:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.247+0000 7f4c1cc00640 1 -- 172.21.15.12:0/468097690 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4c24004ff0 con 0x55b5f7097810 2024-03-23T23:09:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.247+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/468097690 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4c2c015070 con 0x55b5f7097810 2024-03-23T23:09:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.247+0000 7f4c1cc00640 1 -- 172.21.15.12:0/468097690 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c240051f0 con 0x55b5f7097810 2024-03-23T23:09:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.247+0000 7f4c3c7b0740 10 client.27560 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:45.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.247+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/468097690 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55b5f6dda7d0 con 0x55b5f7097810 2024-03-23T23:09:45.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.248+0000 7f4c1cc00640 1 -- 172.21.15.12:0/468097690 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c2c015070 con 0x55b5f7097810 2024-03-23T23:09:45.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.248+0000 7f4c1cc00640 1 -- 172.21.15.12:0/468097690 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f4c24034440 con 0x55b5f7097810 2024-03-23T23:09:45.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.248+0000 7f4c3c7b0740 10 client.27560 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:45.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.248+0000 7f4c3c7b0740 10 client.27560 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:45.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.248+0000 7f4c3c7b0740 20 client.27560 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:45.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.248+0000 7f4c3c7b0740 1 --2- 172.21.15.12:0/468097690 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b5f70ad410 0x55b5f70af860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:45.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.248+0000 7f4c3c7b0740 4 client.27560 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:09:45.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.248+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/468097690 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b5f7009800 con 0x55b5f70ad410 2024-03-23T23:09:45.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.248+0000 7f4c32000640 1 --2- 172.21.15.12:0/468097690 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b5f70ad410 0x55b5f70af860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:45.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.249+0000 7f4c32000640 1 --2- 172.21.15.12:0/468097690 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b5f70ad410 0x55b5f70af860 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c220ff87005156e server_cookie=d66156cf4792b2df in_seq=0 out_seq=0 2024-03-23T23:09:45.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.249+0000 7f4c1cc00640 10 client.27560 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:45.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.250+0000 7f4c1cc00640 1 -- 172.21.15.12:0/468097690 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b5f7009800 con 0x55b5f70ad410 2024-03-23T23:09:45.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.250+0000 7f4c1cc00640 10 client.27560 handle_command_reply: tid=0 2024-03-23T23:09:45.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.250+0000 7f4c3c7b0740 1 client.27560 shutdown 2024-03-23T23:09:45.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.250+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/468097690 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f7097810 msgr2=0x55b5f70a1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:45.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.250+0000 7f4c3c7b0740 1 --2- 172.21.15.12:0/468097690 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f7097810 0x55b5f70a1a50 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4c2402c240 tx=0x7f4c24009260 comp rx=0 tx=0).stop 2024-03-23T23:09:45.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.250+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/468097690 shutdown_connections 2024-03-23T23:09:45.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.250+0000 7f4c3c7b0740 1 --2- 172.21.15.12:0/468097690 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b5f70ad410 0x55b5f70af860 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.250+0000 7f4c3c7b0740 1 --2- 172.21.15.12:0/468097690 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5f70993a0 0x55b5f70a8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.250+0000 7f4c3c7b0740 1 --2- 172.21.15.12:0/468097690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5f70989e0 0x55b5f70a1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.250+0000 7f4c3c7b0740 1 --2- 172.21.15.12:0/468097690 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f7097810 0x55b5f70a1a50 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.250+0000 7f4c1cc00640 0 client.27560 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:45.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.250+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/468097690 >> 172.21.15.12:0/468097690 conn(0x55b5f7092c00 msgr2=0x55b5f7091eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:45.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.250+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/468097690 shutdown_connections 2024-03-23T23:09:45.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.250+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/468097690 wait complete. 2024-03-23T23:09:45.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.250+0000 7f4c3c7b0740 20 client.27560 trim_cache size 0 max 16384 2024-03-23T23:09:45.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.264+0000 7f4c3c7b0740 1 Processor -- start 2024-03-23T23:09:45.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.264+0000 7f4c3c7b0740 1 -- start start 2024-03-23T23:09:45.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.265+0000 7f4c3c7b0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5f7097810 0x55b5f70a1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:45.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.265+0000 7f4c3c7b0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5f70989e0 0x55b5f70a8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:45.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.265+0000 7f4c3c7b0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f70993a0 0x55b5f7096570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:45.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.265+0000 7f4c3c7b0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c341ad200 con 0x55b5f70989e0 2024-03-23T23:09:45.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.265+0000 7f4c3c7b0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c341aad00 con 0x55b5f70993a0 2024-03-23T23:09:45.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.265+0000 7f4c3c7b0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c341a8da0 con 0x55b5f7097810 2024-03-23T23:09:45.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.265+0000 7f4c32a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5f7097810 0x55b5f70a1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:45.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.265+0000 7f4c32a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5f7097810 0x55b5f70a1ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:57728/0 (socket says 172.21.15.12:57728) 2024-03-23T23:09:45.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.265+0000 7f4c32a00640 1 -- 172.21.15.12:0/724923594 learned_addr learned my addr 172.21.15.12:0/724923594 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:45.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.265+0000 7f4c33400640 1 --2- 172.21.15.12:0/724923594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f70993a0 0x55b5f7096570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:45.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.265+0000 7f4c32000640 1 --2- 172.21.15.12:0/724923594 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5f70989e0 0x55b5f70a8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:45.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.265+0000 7f4c32a00640 1 -- 172.21.15.12:0/724923594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f70993a0 msgr2=0x55b5f7096570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:45.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.265+0000 7f4c32a00640 1 --2- 172.21.15.12:0/724923594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f70993a0 0x55b5f7096570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.265+0000 7f4c32a00640 1 -- 172.21.15.12:0/724923594 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5f70989e0 msgr2=0x55b5f70a8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:45.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.265+0000 7f4c32a00640 1 --2- 172.21.15.12:0/724923594 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5f70989e0 0x55b5f70a8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.265+0000 7f4c32a00640 1 -- 172.21.15.12:0/724923594 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5f7092970 con 0x55b5f7097810 2024-03-23T23:09:45.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.265+0000 7f4c32000640 1 --2- 172.21.15.12:0/724923594 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5f70989e0 0x55b5f70a8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:09:45.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.265+0000 7f4c33400640 1 --2- 172.21.15.12:0/724923594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f70993a0 0x55b5f7096570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:09:45.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.266+0000 7f4c32a00640 1 --2- 172.21.15.12:0/724923594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5f7097810 0x55b5f70a1ef0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f4c240347e0 tx=0x7f4c2402cdd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2d1c178c6a640b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:45.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.266+0000 7f4c1f400640 1 -- 172.21.15.12:0/724923594 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c24034440 con 0x55b5f7097810 2024-03-23T23:09:45.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.266+0000 7f4c1f400640 1 -- 172.21.15.12:0/724923594 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c2405b030 con 0x55b5f7097810 2024-03-23T23:09:45.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.266+0000 7f4c1f400640 1 -- 172.21.15.12:0/724923594 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c24003850 con 0x55b5f7097810 2024-03-23T23:09:45.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.266+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/724923594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5f7097810 msgr2=0x55b5f70a1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:45.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.266+0000 7f4c3c7b0740 1 --2- 172.21.15.12:0/724923594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5f7097810 0x55b5f70a1ef0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f4c240347e0 tx=0x7f4c2402cdd0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.267+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/724923594 shutdown_connections 2024-03-23T23:09:45.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.267+0000 7f4c3c7b0740 1 --2- 172.21.15.12:0/724923594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f70993a0 0x55b5f7096570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.267+0000 7f4c3c7b0740 1 --2- 172.21.15.12:0/724923594 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5f70989e0 0x55b5f70a8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.267+0000 7f4c3c7b0740 1 --2- 172.21.15.12:0/724923594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5f7097810 0x55b5f70a1ef0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.267+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/724923594 >> 172.21.15.12:0/724923594 conn(0x55b5f7092c00 msgr2=0x55b5f7095e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:45.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.267+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/724923594 shutdown_connections 2024-03-23T23:09:45.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.267+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/724923594 wait complete. 2024-03-23T23:09:45.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.267+0000 7f4c3c7b0740 1 -- start start 2024-03-23T23:09:45.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.267+0000 7f4c3c7b0740 1 Processor -- start 2024-03-23T23:09:45.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.268+0000 7f4c3c7b0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5f7097810 0x55b5f70a6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:45.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.268+0000 7f4c3c7b0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f70989e0 0x55b5f70a7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:45.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.268+0000 7f4c3c7b0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5f70993a0 0x55b5f70a8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:45.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.268+0000 7f4c3c7b0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c341a93a0 con 0x55b5f70993a0 2024-03-23T23:09:45.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.268+0000 7f4c3c7b0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c341aa6a0 con 0x55b5f70989e0 2024-03-23T23:09:45.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.268+0000 7f4c3c7b0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c341a99e0 con 0x55b5f7097810 2024-03-23T23:09:45.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.268+0000 7f4c3c7b0740 10 client.0 _ll_register_callbacks cb 0x7f4c24005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:45.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.268+0000 7f4c32a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5f7097810 0x55b5f70a6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:45.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.268+0000 7f4c32000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f70989e0 0x55b5f70a7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:45.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.268+0000 7f4c33400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5f70993a0 0x55b5f70a8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:45.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.268+0000 7f4c32a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5f7097810 0x55b5f70a6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:57732/0 (socket says 172.21.15.12:57732) 2024-03-23T23:09:45.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.268+0000 7f4c32a00640 1 -- 172.21.15.12:0/22985722 learned_addr learned my addr 172.21.15.12:0/22985722 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:45.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.268+0000 7f4c32a00640 1 -- 172.21.15.12:0/22985722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f70989e0 msgr2=0x55b5f70a7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:45.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.268+0000 7f4c32a00640 1 --2- 172.21.15.12:0/22985722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f70989e0 0x55b5f70a7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.268+0000 7f4c32a00640 1 -- 172.21.15.12:0/22985722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5f70993a0 msgr2=0x55b5f70a8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:45.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.268+0000 7f4c32a00640 1 --2- 172.21.15.12:0/22985722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5f70993a0 0x55b5f70a8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.268+0000 7f4c32a00640 1 -- 172.21.15.12:0/22985722 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b5f70a8820 con 0x55b5f7097810 2024-03-23T23:09:45.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.268+0000 7f4c32000640 1 --2- 172.21.15.12:0/22985722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f70989e0 0x55b5f70a7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:45.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.269+0000 7f4c32a00640 1 --2- 172.21.15.12:0/22985722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5f7097810 0x55b5f70a6ce0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4c24009900 tx=0x7f4c24005180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b878d2d326d237c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:45.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.269+0000 7f4c1cc00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:09:45.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.269+0000 7f4c1cc00640 1 -- 172.21.15.12:0/22985722 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2406d070 con 0x55b5f7097810 2024-03-23T23:09:45.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.269+0000 7f4c1cc00640 1 -- 172.21.15.12:0/22985722 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4c24005390 con 0x55b5f7097810 2024-03-23T23:09:45.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.269+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/22985722 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4c2405b030 con 0x55b5f7097810 2024-03-23T23:09:45.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.269+0000 7f4c1cc00640 1 -- 172.21.15.12:0/22985722 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c24079050 con 0x55b5f7097810 2024-03-23T23:09:45.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.269+0000 7f4c3c7b0740 10 client.27532 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:45.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.269+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/22985722 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55b5f7109260 con 0x55b5f7097810 2024-03-23T23:09:45.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.269+0000 7f4c1cc00640 1 -- 172.21.15.12:0/22985722 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c2405b030 con 0x55b5f7097810 2024-03-23T23:09:45.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.269+0000 7f4c1cc00640 1 -- 172.21.15.12:0/22985722 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f4c24069470 con 0x55b5f7097810 2024-03-23T23:09:45.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.269+0000 7f4c3c7b0740 10 client.27532 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:45.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.270+0000 7f4c3c7b0740 10 client.27532 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:45.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.270+0000 7f4c3c7b0740 20 client.27532 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:45.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.270+0000 7f4c3c7b0740 1 --2- 172.21.15.12:0/22985722 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b5f7109aa0 0x55b5f710bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:45.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.270+0000 7f4c3c7b0740 4 client.27532 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:09:45.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.270+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/22985722 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b5f7003b50 con 0x55b5f7109aa0 2024-03-23T23:09:45.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.270+0000 7f4c32000640 1 --2- 172.21.15.12:0/22985722 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b5f7109aa0 0x55b5f710bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:45.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.270+0000 7f4c32000640 1 --2- 172.21.15.12:0/22985722 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b5f7109aa0 0x55b5f710bec0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=345532aab98edf69 server_cookie=42bd81e744420cd4 in_seq=0 out_seq=0 2024-03-23T23:09:45.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.270+0000 7f4c1cc00640 10 client.27532 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:45.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.270+0000 7f4c1cc00640 1 -- 172.21.15.12:0/22985722 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b5f7003b50 con 0x55b5f7109aa0 2024-03-23T23:09:45.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.270+0000 7f4c1cc00640 10 client.27532 handle_command_reply: tid=0 2024-03-23T23:09:45.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.271+0000 7f4c3c7b0740 1 client.27532 shutdown 2024-03-23T23:09:45.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.271+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/22985722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5f7097810 msgr2=0x55b5f70a6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:45.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.271+0000 7f4c3c7b0740 1 --2- 172.21.15.12:0/22985722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5f7097810 0x55b5f70a6ce0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4c24009900 tx=0x7f4c24005180 comp rx=0 tx=0).stop 2024-03-23T23:09:45.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.271+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/22985722 shutdown_connections 2024-03-23T23:09:45.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.271+0000 7f4c3c7b0740 1 --2- 172.21.15.12:0/22985722 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b5f7109aa0 0x55b5f710bec0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.271+0000 7f4c3c7b0740 1 --2- 172.21.15.12:0/22985722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5f70993a0 0x55b5f70a8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.271+0000 7f4c3c7b0740 1 --2- 172.21.15.12:0/22985722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5f70989e0 0x55b5f70a7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.271+0000 7f4c3c7b0740 1 --2- 172.21.15.12:0/22985722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5f7097810 0x55b5f70a6ce0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.271+0000 7f4c1cc00640 0 client.27532 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:45.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.271+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/22985722 >> 172.21.15.12:0/22985722 conn(0x55b5f7092c00 msgr2=0x55b5f70966f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:45.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.271+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/22985722 shutdown_connections 2024-03-23T23:09:45.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.271+0000 7f4c3c7b0740 1 -- 172.21.15.12:0/22985722 wait complete. 2024-03-23T23:09:45.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.271+0000 7f4c3c7b0740 20 client.27532 trim_cache size 0 max 16384 2024-03-23T23:09:45.274 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:09:45.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.272+0000 7f4c39a00640 1 -- 172.21.15.12:0/3410281993 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4c180a2880 msgr2=0x7f4c180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:45.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.272+0000 7f4c39a00640 1 --2- 172.21.15.12:0/3410281993 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4c180a2880 0x7f4c180a4d40 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4c2402c170 tx=0x7f4c2402df90 comp rx=0 tx=0).stop 2024-03-23T23:09:45.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.272+0000 7f4c39a00640 1 -- 172.21.15.12:0/3410281993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c34104ab0 msgr2=0x7f4c3419a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:45.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.272+0000 7f4c39a00640 1 --2- 172.21.15.12:0/3410281993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c34104ab0 0x7f4c3419a120 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f4c2000e9f0 tx=0x7f4c2000eeb0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.272+0000 7f4c39a00640 1 -- 172.21.15.12:0/3410281993 shutdown_connections 2024-03-23T23:09:45.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.272+0000 7f4c39a00640 1 --2- 172.21.15.12:0/3410281993 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4c180a2880 0x7f4c180a4d40 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.272+0000 7f4c39a00640 1 --2- 172.21.15.12:0/3410281993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c34105470 0x7f4c341a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.272+0000 7f4c39a00640 1 --2- 172.21.15.12:0/3410281993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c34104ab0 0x7f4c3419a120 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.272+0000 7f4c39a00640 1 --2- 172.21.15.12:0/3410281993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c341038c0 0x7f4c34199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:45.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.272+0000 7f4c39a00640 1 -- 172.21.15.12:0/3410281993 >> 172.21.15.12:0/3410281993 conn(0x7f4c340fee50 msgr2=0x7f4c34107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:45.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.272+0000 7f4c39a00640 1 -- 172.21.15.12:0/3410281993 shutdown_connections 2024-03-23T23:09:45.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:45.272+0000 7f4c39a00640 1 -- 172.21.15.12:0/3410281993 wait complete. 2024-03-23T23:09:46.289 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-23T23:09:46.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:09:46.289 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-23T23:09:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.541+0000 7fd8aae00640 1 -- 172.21.15.12:0/2032702998 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8a4103260 msgr2=0x7fd8a4109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.541+0000 7fd8aae00640 1 --2- 172.21.15.12:0/2032702998 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8a4103260 0x7fd8a4109af0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fd89c009ca0 tx=0x7fd89c02c0f0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.541+0000 7fd8aae00640 1 -- 172.21.15.12:0/2032702998 shutdown_connections 2024-03-23T23:09:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.541+0000 7fd8aae00640 1 --2- 172.21.15.12:0/2032702998 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8a4103260 0x7fd8a4109af0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.541+0000 7fd8aae00640 1 --2- 172.21.15.12:0/2032702998 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8a41028a0 0x7fd8a4102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.541+0000 7fd8aae00640 1 --2- 172.21.15.12:0/2032702998 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8a41016b0 0x7fd8a4101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.541+0000 7fd8aae00640 1 -- 172.21.15.12:0/2032702998 >> 172.21.15.12:0/2032702998 conn(0x7fd8a40fcc40 msgr2=0x7fd8a40ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.541+0000 7fd8aae00640 1 -- 172.21.15.12:0/2032702998 shutdown_connections 2024-03-23T23:09:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.541+0000 7fd8aae00640 1 -- 172.21.15.12:0/2032702998 wait complete. 2024-03-23T23:09:46.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.542+0000 7fd8aae00640 1 Processor -- start 2024-03-23T23:09:46.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.543+0000 7fd8aae00640 1 -- start start 2024-03-23T23:09:46.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.543+0000 7fd8aae00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8a41016b0 0x7fd8a4199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.543+0000 7fd8aae00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8a41028a0 0x7fd8a419a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.543+0000 7fd8aae00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8a4103260 0x7fd8a41a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.543+0000 7fd8aae00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a410d330 con 0x7fd8a41028a0 2024-03-23T23:09:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.543+0000 7fd8aae00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a410d1b0 con 0x7fd8a41016b0 2024-03-23T23:09:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.543+0000 7fd8aae00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a410d4b0 con 0x7fd8a4103260 2024-03-23T23:09:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.543+0000 7fd8a3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8a41016b0 0x7fd8a4199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.543+0000 7fd8a3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8a41016b0 0x7fd8a4199b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:33760/0 (socket says 172.21.15.12:33760) 2024-03-23T23:09:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.543+0000 7fd8a3e00640 1 -- 172.21.15.12:0/3047153545 learned_addr learned my addr 172.21.15.12:0/3047153545 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.543+0000 7fd8a8e00640 1 --2- 172.21.15.12:0/3047153545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8a4103260 0x7fd8a41a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.543+0000 7fd8a3400640 1 --2- 172.21.15.12:0/3047153545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8a41028a0 0x7fd8a419a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.544+0000 7fd8a3e00640 1 -- 172.21.15.12:0/3047153545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8a4103260 msgr2=0x7fd8a41a1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.544+0000 7fd8a3e00640 1 --2- 172.21.15.12:0/3047153545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8a4103260 0x7fd8a41a1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.544+0000 7fd8a3e00640 1 -- 172.21.15.12:0/3047153545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8a41028a0 msgr2=0x7fd8a419a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.544+0000 7fd8a3e00640 1 --2- 172.21.15.12:0/3047153545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8a41028a0 0x7fd8a419a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.544+0000 7fd8a3e00640 1 -- 172.21.15.12:0/3047153545 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a41a1780 con 0x7fd8a41016b0 2024-03-23T23:09:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.544+0000 7fd8a8e00640 1 --2- 172.21.15.12:0/3047153545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8a4103260 0x7fd8a41a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.544+0000 7fd8a3e00640 1 --2- 172.21.15.12:0/3047153545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8a41016b0 0x7fd8a4199b60 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fd89400e9c0 tx=0x7fd89400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.544+0000 7fd8a0c00640 1 -- 172.21.15.12:0/3047153545 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89400cd80 con 0x7fd8a41016b0 2024-03-23T23:09:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.544+0000 7fd8a0c00640 1 -- 172.21.15.12:0/3047153545 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8940035f0 con 0x7fd8a41016b0 2024-03-23T23:09:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.544+0000 7fd8aae00640 1 -- 172.21.15.12:0/3047153545 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8a41a1a70 con 0x7fd8a41016b0 2024-03-23T23:09:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.544+0000 7fd8a0c00640 1 -- 172.21.15.12:0/3047153545 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd894010690 con 0x7fd8a41016b0 2024-03-23T23:09:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.545+0000 7fd8aae00640 1 -- 172.21.15.12:0/3047153545 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8a41a1fc0 con 0x7fd8a41016b0 2024-03-23T23:09:46.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.547+0000 7fd8a0c00640 1 -- 172.21.15.12:0/3047153545 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd8940037a0 con 0x7fd8a41016b0 2024-03-23T23:09:46.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.548+0000 7fd8ada42740 1 Processor -- start 2024-03-23T23:09:46.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.548+0000 7fd8ada42740 1 -- start start 2024-03-23T23:09:46.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.548+0000 7fd8ada42740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb30a0 0x55fd0acb34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.548+0000 7fd8ada42740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd0acb3a90 0x55fd0acb8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.548+0000 7fd8ada42740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd0acb8c00 0x55fd0acbb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.548+0000 7fd8ada42740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd0ac18eb0 con 0x55fd0acb3a90 2024-03-23T23:09:46.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.548+0000 7fd8ada42740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd0abc1a60 con 0x55fd0acb30a0 2024-03-23T23:09:46.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.548+0000 7fd8ada42740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd0ac14f50 con 0x55fd0acb8c00 2024-03-23T23:09:46.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.548+0000 7fd8a8e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb30a0 0x55fd0acb34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.548+0000 7fd8a3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd0acb3a90 0x55fd0acb8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.548+0000 7fd8a3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd0acb8c00 0x55fd0acbb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.548+0000 7fd8a8e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb30a0 0x55fd0acb34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:33776/0 (socket says 172.21.15.12:33776) 2024-03-23T23:09:46.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.548+0000 7fd8a8e00640 1 -- 172.21.15.12:0/3854040898 learned_addr learned my addr 172.21.15.12:0/3854040898 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:46.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.549+0000 7fd8a8e00640 1 -- 172.21.15.12:0/3854040898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd0acb8c00 msgr2=0x55fd0acbb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:46.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.549+0000 7fd8a8e00640 1 --2- 172.21.15.12:0/3854040898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd0acb8c00 0x55fd0acbb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.549+0000 7fd8a8e00640 1 -- 172.21.15.12:0/3854040898 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd0acb3a90 msgr2=0x55fd0acb8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:46.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.549+0000 7fd8a8e00640 1 --2- 172.21.15.12:0/3854040898 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd0acb3a90 0x55fd0acb8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.549+0000 7fd8a8e00640 1 -- 172.21.15.12:0/3854040898 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fd0a9f57e0 con 0x55fd0acb30a0 2024-03-23T23:09:46.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.549+0000 7fd8a8e00640 1 --2- 172.21.15.12:0/3854040898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb30a0 0x55fd0acb34c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd89c02c700 tx=0x7fd89c0323d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b97a9f075cf8bc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.549+0000 7fd8a0c00640 1 --2- 172.21.15.12:0/3047153545 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd8880a2880 0x7fd8880a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.549+0000 7fd899600640 1 -- 172.21.15.12:0/3854040898 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c043070 con 0x55fd0acb30a0 2024-03-23T23:09:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.549+0000 7fd899600640 1 -- 172.21.15.12:0/3854040898 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd89c02cca0 con 0x55fd0acb30a0 2024-03-23T23:09:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.549+0000 7fd899600640 1 -- 172.21.15.12:0/3854040898 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c039040 con 0x55fd0acb30a0 2024-03-23T23:09:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.550+0000 7fd8a0c00640 1 -- 172.21.15.12:0/3047153545 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd8940cee30 con 0x7fd8a41016b0 2024-03-23T23:09:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.550+0000 7fd8a3e00640 1 --2- 172.21.15.12:0/3047153545 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd8880a2880 0x7fd8880a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.550+0000 7fd8ada42740 1 -- 172.21.15.12:0/3854040898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb30a0 msgr2=0x55fd0acb34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.550+0000 7fd8ada42740 1 --2- 172.21.15.12:0/3854040898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb30a0 0x55fd0acb34c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd89c02c700 tx=0x7fd89c0323d0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.550+0000 7fd8a3e00640 1 --2- 172.21.15.12:0/3047153545 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd8880a2880 0x7fd8880a4d40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd8940a39f0 tx=0x7fd8940d2870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.550+0000 7fd8ada42740 1 -- 172.21.15.12:0/3854040898 shutdown_connections 2024-03-23T23:09:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.550+0000 7fd8ada42740 1 --2- 172.21.15.12:0/3854040898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd0acb8c00 0x55fd0acbb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.550+0000 7fd8ada42740 1 --2- 172.21.15.12:0/3854040898 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd0acb3a90 0x55fd0acb8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.550+0000 7fd8ada42740 1 --2- 172.21.15.12:0/3854040898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb30a0 0x55fd0acb34c0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.550+0000 7fd8ada42740 1 -- 172.21.15.12:0/3854040898 >> 172.21.15.12:0/3854040898 conn(0x55fd0acae610 msgr2=0x55fd0acaea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:46.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.551+0000 7fd8ada42740 1 -- 172.21.15.12:0/3854040898 shutdown_connections 2024-03-23T23:09:46.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.551+0000 7fd8ada42740 1 -- 172.21.15.12:0/3854040898 wait complete. 2024-03-23T23:09:46.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.551+0000 7fd8ada42740 1 -- start start 2024-03-23T23:09:46.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.551+0000 7fd8ada42740 1 Processor -- start 2024-03-23T23:09:46.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.552+0000 7fd8ada42740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd0acb30a0 0x55fd0acbd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.552+0000 7fd8ada42740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb3a90 0x55fd0acbd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.552+0000 7fd8ada42740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd0acb8c00 0x55fd0acc47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.552+0000 7fd8ada42740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd0acbbb50 con 0x55fd0acb8c00 2024-03-23T23:09:46.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.552+0000 7fd8ada42740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd0abd4eb0 con 0x55fd0acb3a90 2024-03-23T23:09:46.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.552+0000 7fd8ada42740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd0acbbcd0 con 0x55fd0acb30a0 2024-03-23T23:09:46.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.552+0000 7fd8ada42740 10 client.0 _ll_register_callbacks cb 0x55fd0abc9150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:46.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.552+0000 7fd8a3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb3a90 0x55fd0acbd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.552+0000 7fd8a8e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd0acb30a0 0x55fd0acbd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.552+0000 7fd8a3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb3a90 0x55fd0acbd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:33782/0 (socket says 172.21.15.12:33782) 2024-03-23T23:09:46.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.552+0000 7fd8a3400640 1 -- 172.21.15.12:0/2664031312 learned_addr learned my addr 172.21.15.12:0/2664031312 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:46.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.552+0000 7fd8a8e00640 1 --2- 172.21.15.12:0/2664031312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd0acb8c00 0x55fd0acc47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.552+0000 7fd8a3400640 1 -- 172.21.15.12:0/2664031312 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd0acb30a0 msgr2=0x55fd0acbd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:46.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.552+0000 7fd8a3400640 1 --2- 172.21.15.12:0/2664031312 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd0acb30a0 0x55fd0acbd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.552+0000 7fd8a3400640 1 -- 172.21.15.12:0/2664031312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd0acb8c00 msgr2=0x55fd0acc47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:46.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.552+0000 7fd8a3400640 1 --2- 172.21.15.12:0/2664031312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd0acb8c00 0x55fd0acc47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.552+0000 7fd8a3400640 1 -- 172.21.15.12:0/2664031312 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fd0aa55f70 con 0x55fd0acb3a90 2024-03-23T23:09:46.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.553+0000 7fd8a3400640 1 --2- 172.21.15.12:0/2664031312 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb3a90 0x55fd0acbd7a0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fd89000da10 tx=0x7fd89000ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a8ea087e961c545 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:46.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.553+0000 7fd88e000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:09:46.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.553+0000 7fd88e000640 1 -- 172.21.15.12:0/2664031312 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd890011640 con 0x55fd0acb3a90 2024-03-23T23:09:46.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.553+0000 7fd88e000640 1 -- 172.21.15.12:0/2664031312 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd8900117e0 con 0x55fd0acb3a90 2024-03-23T23:09:46.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.553+0000 7fd8ada42740 1 -- 172.21.15.12:0/2664031312 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd89c02cca0 con 0x55fd0acb3a90 2024-03-23T23:09:46.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.553+0000 7fd88e000640 1 -- 172.21.15.12:0/2664031312 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8900119e0 con 0x55fd0acb3a90 2024-03-23T23:09:46.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.553+0000 7fd8ada42740 10 client.27578 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:46.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.553+0000 7fd8ada42740 1 -- 172.21.15.12:0/2664031312 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55fd0a8ed170 con 0x55fd0acb3a90 2024-03-23T23:09:46.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.554+0000 7fd88e000640 1 -- 172.21.15.12:0/2664031312 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd89c02cca0 con 0x55fd0acb3a90 2024-03-23T23:09:46.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.554+0000 7fd88e000640 1 -- 172.21.15.12:0/2664031312 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fd890019030 con 0x55fd0acb3a90 2024-03-23T23:09:46.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.554+0000 7fd8ada42740 10 client.27578 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:46.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.554+0000 7fd8ada42740 10 client.27578 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:46.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.554+0000 7fd8ada42740 20 client.27578 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:46.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.554+0000 7fd8ada42740 1 --2- 172.21.15.12:0/2664031312 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fd0acc90e0 0x55fd0accb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.554+0000 7fd8ada42740 4 client.27578 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:09:46.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.554+0000 7fd8ada42740 1 -- 172.21.15.12:0/2664031312 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fd0ac263f0 con 0x55fd0acc90e0 2024-03-23T23:09:46.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.554+0000 7fd8a8e00640 1 --2- 172.21.15.12:0/2664031312 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fd0acc90e0 0x55fd0accb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.555+0000 7fd8a8e00640 1 --2- 172.21.15.12:0/2664031312 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fd0acc90e0 0x55fd0accb4d0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff7f357a7326fd1f server_cookie=4141208e307d5c94 in_seq=0 out_seq=0 2024-03-23T23:09:46.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.555+0000 7fd88e000640 10 client.27578 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:46.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.555+0000 7fd88e000640 1 -- 172.21.15.12:0/2664031312 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55fd0ac263f0 con 0x55fd0acc90e0 2024-03-23T23:09:46.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.555+0000 7fd88e000640 10 client.27578 handle_command_reply: tid=0 2024-03-23T23:09:46.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.555+0000 7fd8ada42740 1 client.27578 shutdown 2024-03-23T23:09:46.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.556+0000 7fd8ada42740 1 -- 172.21.15.12:0/2664031312 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb3a90 msgr2=0x55fd0acbd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:46.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.556+0000 7fd8ada42740 1 --2- 172.21.15.12:0/2664031312 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb3a90 0x55fd0acbd7a0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fd89000da10 tx=0x7fd89000ded0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.556+0000 7fd8ada42740 1 -- 172.21.15.12:0/2664031312 shutdown_connections 2024-03-23T23:09:46.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.556+0000 7fd8ada42740 1 --2- 172.21.15.12:0/2664031312 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fd0acc90e0 0x55fd0accb4d0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.556+0000 7fd8ada42740 1 --2- 172.21.15.12:0/2664031312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd0acb8c00 0x55fd0acc47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.556+0000 7fd8ada42740 1 --2- 172.21.15.12:0/2664031312 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb3a90 0x55fd0acbd7a0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.556+0000 7fd8ada42740 1 --2- 172.21.15.12:0/2664031312 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd0acb30a0 0x55fd0acbd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.556+0000 7fd8ada42740 1 -- 172.21.15.12:0/2664031312 >> 172.21.15.12:0/2664031312 conn(0x55fd0acae610 msgr2=0x55fd0acad830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:46.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.556+0000 7fd88e000640 0 client.27578 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:46.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.556+0000 7fd8ada42740 1 -- 172.21.15.12:0/2664031312 shutdown_connections 2024-03-23T23:09:46.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.556+0000 7fd8ada42740 1 -- 172.21.15.12:0/2664031312 wait complete. 2024-03-23T23:09:46.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.556+0000 7fd8ada42740 20 client.27578 trim_cache size 0 max 16384 2024-03-23T23:09:46.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.574+0000 7fd8ada42740 1 Processor -- start 2024-03-23T23:09:46.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.574+0000 7fd8ada42740 1 -- start start 2024-03-23T23:09:46.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.574+0000 7fd8ada42740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb30a0 0x55fd0acb1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.574+0000 7fd8ada42740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd0acb3a90 0x55fd0acc47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.574+0000 7fd8ada42740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd0acb8c00 0x55fd0acb4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.574+0000 7fd8ada42740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a41ad080 con 0x55fd0acb3a90 2024-03-23T23:09:46.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.574+0000 7fd8ada42740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a41aab80 con 0x55fd0acb30a0 2024-03-23T23:09:46.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.574+0000 7fd8ada42740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a41a8c20 con 0x55fd0acb8c00 2024-03-23T23:09:46.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.574+0000 7fd8a8e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb30a0 0x55fd0acb1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.575+0000 7fd8a8e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb30a0 0x55fd0acb1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:33792/0 (socket says 172.21.15.12:33792) 2024-03-23T23:09:46.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.575+0000 7fd8a8e00640 1 -- 172.21.15.12:0/1185797170 learned_addr learned my addr 172.21.15.12:0/1185797170 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:46.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.575+0000 7fd8a8e00640 1 --2- 172.21.15.12:0/1185797170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd0acb8c00 0x55fd0acb4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.575+0000 7fd8a3400640 1 --2- 172.21.15.12:0/1185797170 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd0acb3a90 0x55fd0acc47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.575+0000 7fd8a8e00640 1 -- 172.21.15.12:0/1185797170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd0acb8c00 msgr2=0x55fd0acb4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:46.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.575+0000 7fd8a8e00640 1 --2- 172.21.15.12:0/1185797170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd0acb8c00 0x55fd0acb4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.575+0000 7fd8a8e00640 1 -- 172.21.15.12:0/1185797170 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd0acb3a90 msgr2=0x55fd0acc47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:46.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.575+0000 7fd8a8e00640 1 --2- 172.21.15.12:0/1185797170 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd0acb3a90 0x55fd0acc47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.575+0000 7fd8a8e00640 1 -- 172.21.15.12:0/1185797170 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fd0accb840 con 0x55fd0acb30a0 2024-03-23T23:09:46.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.575+0000 7fd8a8e00640 1 --2- 172.21.15.12:0/1185797170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb30a0 0x55fd0acb1f80 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fd89c006010 tx=0x7fd89c02e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f2361bebb4076cf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:46.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.576+0000 7fd898c00640 1 -- 172.21.15.12:0/1185797170 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c039040 con 0x55fd0acb30a0 2024-03-23T23:09:46.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.576+0000 7fd898c00640 1 -- 172.21.15.12:0/1185797170 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd89c032c90 con 0x55fd0acb30a0 2024-03-23T23:09:46.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.576+0000 7fd898c00640 1 -- 172.21.15.12:0/1185797170 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c043040 con 0x55fd0acb30a0 2024-03-23T23:09:46.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.576+0000 7fd8ada42740 1 -- 172.21.15.12:0/1185797170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb30a0 msgr2=0x55fd0acb1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:46.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.576+0000 7fd8ada42740 1 --2- 172.21.15.12:0/1185797170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb30a0 0x55fd0acb1f80 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fd89c006010 tx=0x7fd89c02e040 comp rx=0 tx=0).stop 2024-03-23T23:09:46.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.576+0000 7fd8ada42740 1 -- 172.21.15.12:0/1185797170 shutdown_connections 2024-03-23T23:09:46.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.576+0000 7fd8ada42740 1 --2- 172.21.15.12:0/1185797170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd0acb8c00 0x55fd0acb4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.576+0000 7fd8ada42740 1 --2- 172.21.15.12:0/1185797170 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd0acb3a90 0x55fd0acc47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.576+0000 7fd8ada42740 1 --2- 172.21.15.12:0/1185797170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb30a0 0x55fd0acb1f80 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.576+0000 7fd8ada42740 1 -- 172.21.15.12:0/1185797170 >> 172.21.15.12:0/1185797170 conn(0x55fd0acae610 msgr2=0x55fd0acb1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:46.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.576+0000 7fd8ada42740 1 -- 172.21.15.12:0/1185797170 shutdown_connections 2024-03-23T23:09:46.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.576+0000 7fd8ada42740 1 -- 172.21.15.12:0/1185797170 wait complete. 2024-03-23T23:09:46.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.577+0000 7fd8ada42740 1 -- start start 2024-03-23T23:09:46.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.577+0000 7fd8ada42740 1 Processor -- start 2024-03-23T23:09:46.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.577+0000 7fd8ada42740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd0acb30a0 0x55fd0acc25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.577+0000 7fd8ada42740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb3a90 0x55fd0acc2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.577+0000 7fd8ada42740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd0acb8c00 0x55fd0acc3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.577+0000 7fd8ada42740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a41a9220 con 0x55fd0acb8c00 2024-03-23T23:09:46.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.577+0000 7fd8ada42740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a41aa520 con 0x55fd0acb3a90 2024-03-23T23:09:46.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.577+0000 7fd8ada42740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a41a9860 con 0x55fd0acb30a0 2024-03-23T23:09:46.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.577+0000 7fd8ada42740 10 client.0 _ll_register_callbacks cb 0x7fd890004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:46.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.578+0000 7fd8a3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb3a90 0x55fd0acc2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.578+0000 7fd8a3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb3a90 0x55fd0acc2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:33802/0 (socket says 172.21.15.12:33802) 2024-03-23T23:09:46.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.578+0000 7fd8a3400640 1 -- 172.21.15.12:0/1922772449 learned_addr learned my addr 172.21.15.12:0/1922772449 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:46.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.578+0000 7fd8a8e00640 1 --2- 172.21.15.12:0/1922772449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd0acb30a0 0x55fd0acc25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.578+0000 7fd8a8e00640 1 --2- 172.21.15.12:0/1922772449 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd0acb8c00 0x55fd0acc3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.578+0000 7fd8a3400640 1 -- 172.21.15.12:0/1922772449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd0acb30a0 msgr2=0x55fd0acc25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:46.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.578+0000 7fd8a3400640 1 --2- 172.21.15.12:0/1922772449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd0acb30a0 0x55fd0acc25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.578+0000 7fd8a3400640 1 -- 172.21.15.12:0/1922772449 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd0acb8c00 msgr2=0x55fd0acc3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:46.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.578+0000 7fd8a3400640 1 --2- 172.21.15.12:0/1922772449 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd0acb8c00 0x55fd0acc3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.578+0000 7fd8a3400640 1 -- 172.21.15.12:0/1922772449 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fd0acc40b0 con 0x55fd0acb3a90 2024-03-23T23:09:46.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.578+0000 7fd8a3400640 1 --2- 172.21.15.12:0/1922772449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb3a90 0x55fd0acc2b20 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fd890025fd0 tx=0x7fd89000d5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3168ca4e67d15c2c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:46.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.578+0000 7fd88e000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:09:46.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.579+0000 7fd88e000640 1 -- 172.21.15.12:0/1922772449 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89002a040 con 0x55fd0acb3a90 2024-03-23T23:09:46.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.579+0000 7fd88e000640 1 -- 172.21.15.12:0/1922772449 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd890011640 con 0x55fd0acb3a90 2024-03-23T23:09:46.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.579+0000 7fd8ada42740 1 -- 172.21.15.12:0/1922772449 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd89c032c90 con 0x55fd0acb3a90 2024-03-23T23:09:46.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.579+0000 7fd88e000640 1 -- 172.21.15.12:0/1922772449 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd890011840 con 0x55fd0acb3a90 2024-03-23T23:09:46.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.579+0000 7fd8ada42740 10 client.27584 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:46.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.579+0000 7fd8ada42740 1 -- 172.21.15.12:0/1922772449 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55fd0ad250e0 con 0x55fd0acb3a90 2024-03-23T23:09:46.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.579+0000 7fd88e000640 1 -- 172.21.15.12:0/1922772449 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd89c032c90 con 0x55fd0acb3a90 2024-03-23T23:09:46.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.579+0000 7fd88e000640 1 -- 172.21.15.12:0/1922772449 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fd890019070 con 0x55fd0acb3a90 2024-03-23T23:09:46.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.579+0000 7fd8ada42740 10 client.27584 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:46.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.579+0000 7fd8ada42740 10 client.27584 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:46.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.579+0000 7fd8ada42740 20 client.27584 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:46.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.579+0000 7fd8ada42740 1 --2- 172.21.15.12:0/1922772449 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fd0ad25a00 0x55fd0ad27e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.579+0000 7fd8ada42740 4 client.27584 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-23T23:09:46.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.579+0000 7fd8ada42740 1 -- 172.21.15.12:0/1922772449 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55fd0ac20740 con 0x55fd0ad25a00 2024-03-23T23:09:46.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.580+0000 7fd8a8e00640 1 --2- 172.21.15.12:0/1922772449 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fd0ad25a00 0x55fd0ad27e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.580+0000 7fd8a8e00640 1 --2- 172.21.15.12:0/1922772449 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fd0ad25a00 0x55fd0ad27e20 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96a26a212744383e server_cookie=ca53d5b06298768c in_seq=0 out_seq=0 2024-03-23T23:09:46.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.580+0000 7fd88e000640 10 client.27584 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:46.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.581+0000 7fd88e000640 1 -- 172.21.15.12:0/1922772449 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55fd0ac20740 con 0x55fd0ad25a00 2024-03-23T23:09:46.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.581+0000 7fd88e000640 10 client.27584 handle_command_reply: tid=0 2024-03-23T23:09:46.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.581+0000 7fd8ada42740 1 client.27584 shutdown 2024-03-23T23:09:46.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.581+0000 7fd8ada42740 1 -- 172.21.15.12:0/1922772449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb3a90 msgr2=0x55fd0acc2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:46.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.581+0000 7fd8ada42740 1 --2- 172.21.15.12:0/1922772449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb3a90 0x55fd0acc2b20 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fd890025fd0 tx=0x7fd89000d5b0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.581+0000 7fd8ada42740 1 -- 172.21.15.12:0/1922772449 shutdown_connections 2024-03-23T23:09:46.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.581+0000 7fd8ada42740 1 --2- 172.21.15.12:0/1922772449 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fd0ad25a00 0x55fd0ad27e20 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.581+0000 7fd8ada42740 1 --2- 172.21.15.12:0/1922772449 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd0acb8c00 0x55fd0acc3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.581+0000 7fd8ada42740 1 --2- 172.21.15.12:0/1922772449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd0acb3a90 0x55fd0acc2b20 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.581+0000 7fd8ada42740 1 --2- 172.21.15.12:0/1922772449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd0acb30a0 0x55fd0acc25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.581+0000 7fd88e000640 0 client.27584 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:46.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.581+0000 7fd8ada42740 1 -- 172.21.15.12:0/1922772449 >> 172.21.15.12:0/1922772449 conn(0x55fd0acae610 msgr2=0x55fd0acb2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:46.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.582+0000 7fd8ada42740 1 -- 172.21.15.12:0/1922772449 shutdown_connections 2024-03-23T23:09:46.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.582+0000 7fd8ada42740 1 -- 172.21.15.12:0/1922772449 wait complete. 2024-03-23T23:09:46.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.582+0000 7fd8ada42740 20 client.27584 trim_cache size 0 max 16384 2024-03-23T23:09:46.585 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:09:46.586 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-03-23T23:09:46.586 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "12a2d79e-52dd-45df-aa9b-2379ff872728", 2024-03-23T23:09:46.586 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-03-23T23:09:46.586 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:09:46.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.582+0000 7fd8aae00640 1 -- 172.21.15.12:0/3047153545 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd8880a2880 msgr2=0x7fd8880a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:46.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.582+0000 7fd8aae00640 1 --2- 172.21.15.12:0/3047153545 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd8880a2880 0x7fd8880a4d40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd8940a39f0 tx=0x7fd8940d2870 comp rx=0 tx=0).stop 2024-03-23T23:09:46.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.582+0000 7fd8aae00640 1 -- 172.21.15.12:0/3047153545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8a41016b0 msgr2=0x7fd8a4199b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:46.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.582+0000 7fd8aae00640 1 --2- 172.21.15.12:0/3047153545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8a41016b0 0x7fd8a4199b60 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fd89400e9c0 tx=0x7fd89400ee80 comp rx=0 tx=0).stop 2024-03-23T23:09:46.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.583+0000 7fd8aae00640 1 -- 172.21.15.12:0/3047153545 shutdown_connections 2024-03-23T23:09:46.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.583+0000 7fd8aae00640 1 --2- 172.21.15.12:0/3047153545 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd8880a2880 0x7fd8880a4d40 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.583+0000 7fd8aae00640 1 --2- 172.21.15.12:0/3047153545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8a4103260 0x7fd8a41a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.583+0000 7fd8aae00640 1 --2- 172.21.15.12:0/3047153545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8a41028a0 0x7fd8a419a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.583+0000 7fd8aae00640 1 --2- 172.21.15.12:0/3047153545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8a41016b0 0x7fd8a4199b60 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.583+0000 7fd8aae00640 1 -- 172.21.15.12:0/3047153545 >> 172.21.15.12:0/3047153545 conn(0x7fd8a40fcc40 msgr2=0x7fd8a40738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:46.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.583+0000 7fd8aae00640 1 -- 172.21.15.12:0/3047153545 shutdown_connections 2024-03-23T23:09:46.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.583+0000 7fd8aae00640 1 -- 172.21.15.12:0/3047153545 wait complete. 2024-03-23T23:09:46.599 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:09:46.600 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:09:46.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.822+0000 7f2dd1a00640 1 -- 172.21.15.12:0/880324878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dcc074100 msgr2=0x7f2dcc074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:46.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.822+0000 7f2dd1a00640 1 --2- 172.21.15.12:0/880324878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dcc074100 0x7f2dcc074580 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f2db8009900 tx=0x7f2db802bc30 comp rx=0 tx=0).stop 2024-03-23T23:09:46.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.823+0000 7f2dd1a00640 1 -- 172.21.15.12:0/880324878 shutdown_connections 2024-03-23T23:09:46.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.823+0000 7f2dd1a00640 1 --2- 172.21.15.12:0/880324878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dcc10db50 0x7f2dcc10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.823+0000 7f2dd1a00640 1 --2- 172.21.15.12:0/880324878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dcc074100 0x7f2dcc074580 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.823+0000 7f2dd1a00640 1 --2- 172.21.15.12:0/880324878 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dcc0756e0 0x7f2dcc073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.823+0000 7f2dd1a00640 1 -- 172.21.15.12:0/880324878 >> 172.21.15.12:0/880324878 conn(0x7f2dcc0fcb10 msgr2=0x7f2dcc0fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:46.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.823+0000 7f2dd1a00640 1 -- 172.21.15.12:0/880324878 shutdown_connections 2024-03-23T23:09:46.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.823+0000 7f2dd1a00640 1 -- 172.21.15.12:0/880324878 wait complete. 2024-03-23T23:09:46.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.824+0000 7f2dd1a00640 1 Processor -- start 2024-03-23T23:09:46.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.824+0000 7f2dd1a00640 1 -- start start 2024-03-23T23:09:46.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.824+0000 7f2dd1a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dcc074100 0x7f2dcc19de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.824+0000 7f2dd1a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dcc0756e0 0x7f2dcc19e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.824+0000 7f2dd1a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dcc10db50 0x7f2dcc1a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.824+0000 7f2dd1a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc113840 con 0x7f2dcc074100 2024-03-23T23:09:46.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.824+0000 7f2dd1a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc1136c0 con 0x7f2dcc0756e0 2024-03-23T23:09:46.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.824+0000 7f2dd1a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc1139c0 con 0x7f2dcc10db50 2024-03-23T23:09:46.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.824+0000 7f2dd1000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dcc10db50 0x7f2dcc1a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.824+0000 7f2dcb400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dcc0756e0 0x7f2dcc19e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.824+0000 7f2dcbe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dcc074100 0x7f2dcc19de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.824+0000 7f2dd1000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dcc10db50 0x7f2dcc1a5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:57806/0 (socket says 172.21.15.12:57806) 2024-03-23T23:09:46.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.824+0000 7f2dd1000640 1 -- 172.21.15.12:0/1692767007 learned_addr learned my addr 172.21.15.12:0/1692767007 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:46.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.825+0000 7f2dd1000640 1 -- 172.21.15.12:0/1692767007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dcc0756e0 msgr2=0x7f2dcc19e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:46.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.825+0000 7f2dd1000640 1 --2- 172.21.15.12:0/1692767007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dcc0756e0 0x7f2dcc19e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.825+0000 7f2dd1000640 1 -- 172.21.15.12:0/1692767007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dcc074100 msgr2=0x7f2dcc19de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:46.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.825+0000 7f2dd1000640 1 --2- 172.21.15.12:0/1692767007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dcc074100 0x7f2dcc19de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.825+0000 7f2dd1000640 1 -- 172.21.15.12:0/1692767007 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dcc1a5a90 con 0x7f2dcc10db50 2024-03-23T23:09:46.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.825+0000 7f2dcbe00640 1 --2- 172.21.15.12:0/1692767007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dcc074100 0x7f2dcc19de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:09:46.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.825+0000 7f2dd1000640 1 --2- 172.21.15.12:0/1692767007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dcc10db50 0x7f2dcc1a5420 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f2dc400eb10 tx=0x7f2dc400c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:46.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.825+0000 7f2dc8c00640 1 -- 172.21.15.12:0/1692767007 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc4004b20 con 0x7f2dcc10db50 2024-03-23T23:09:46.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.825+0000 7f2dc8c00640 1 -- 172.21.15.12:0/1692767007 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2dc4004cc0 con 0x7f2dcc10db50 2024-03-23T23:09:46.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.825+0000 7f2dc8c00640 1 -- 172.21.15.12:0/1692767007 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc4010760 con 0x7f2dcc10db50 2024-03-23T23:09:46.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.825+0000 7f2dd1a00640 1 -- 172.21.15.12:0/1692767007 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dcc1a5d80 con 0x7f2dcc10db50 2024-03-23T23:09:46.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.826+0000 7f2dd1a00640 1 -- 172.21.15.12:0/1692767007 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dcc1a62a0 con 0x7f2dcc10db50 2024-03-23T23:09:46.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.828+0000 7f2dc8c00640 1 -- 172.21.15.12:0/1692767007 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2dc4005020 con 0x7f2dcc10db50 2024-03-23T23:09:46.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.829+0000 7f2dd5deb740 1 Processor -- start 2024-03-23T23:09:46.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.829+0000 7f2dd5deb740 1 -- start start 2024-03-23T23:09:46.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.829+0000 7f2dd5deb740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ba0a3f050 0x558ba0a3f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.829+0000 7f2dd5deb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ba0a3fa40 0x558ba0a44540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.829+0000 7f2dd5deb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ba0a44bb0 0x558ba0a47040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.829+0000 7f2dd5deb740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ba09a4250 con 0x558ba0a3f050 2024-03-23T23:09:46.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.829+0000 7f2dd5deb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ba094ca30 con 0x558ba0a3fa40 2024-03-23T23:09:46.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.829+0000 7f2dd5deb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ba09a02f0 con 0x558ba0a44bb0 2024-03-23T23:09:46.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.829+0000 7f2dcbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ba0a3fa40 0x558ba0a44540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.830+0000 7f2dcbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ba0a3fa40 0x558ba0a44540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:33838/0 (socket says 172.21.15.12:33838) 2024-03-23T23:09:46.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.830+0000 7f2dcbe00640 1 -- 172.21.15.12:0/2263182437 learned_addr learned my addr 172.21.15.12:0/2263182437 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:46.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.830+0000 7f2dcb400640 1 --2- 172.21.15.12:0/2263182437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ba0a3f050 0x558ba0a3f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.830+0000 7f2dcbe00640 1 -- 172.21.15.12:0/2263182437 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ba0a44bb0 msgr2=0x558ba0a47040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T23:09:46.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.830+0000 7f2dcbe00640 1 --2- 172.21.15.12:0/2263182437 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ba0a44bb0 0x558ba0a47040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.830+0000 7f2dcbe00640 1 -- 172.21.15.12:0/2263182437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ba0a3f050 msgr2=0x558ba0a3f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:46.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.830+0000 7f2dcbe00640 1 --2- 172.21.15.12:0/2263182437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ba0a3f050 0x558ba0a3f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.830+0000 7f2dcbe00640 1 -- 172.21.15.12:0/2263182437 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558ba0937690 con 0x558ba0a3fa40 2024-03-23T23:09:46.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.830+0000 7f2dcb400640 1 --2- 172.21.15.12:0/2263182437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ba0a3f050 0x558ba0a3f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:09:46.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.830+0000 7f2dc8c00640 1 --2- 172.21.15.12:0/1692767007 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2db00a2880 0x7f2db00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.830+0000 7f2dcbe00640 1 --2- 172.21.15.12:0/2263182437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ba0a3fa40 0x558ba0a44540 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc009d50 tx=0x7f2dbc0117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96bbf0f53ea3ada0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:46.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.830+0000 7f2dc8c00640 1 -- 172.21.15.12:0/1692767007 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2dc40cf1b0 con 0x7f2dcc10db50 2024-03-23T23:09:46.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.831+0000 7f2dc1600640 1 -- 172.21.15.12:0/2263182437 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc012650 con 0x558ba0a3fa40 2024-03-23T23:09:46.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.831+0000 7f2dc1600640 1 -- 172.21.15.12:0/2263182437 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2dbc009e40 con 0x558ba0a3fa40 2024-03-23T23:09:46.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.831+0000 7f2dcb400640 1 --2- 172.21.15.12:0/1692767007 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2db00a2880 0x7f2db00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.831+0000 7f2dc1600640 1 -- 172.21.15.12:0/2263182437 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc00fc60 con 0x558ba0a3fa40 2024-03-23T23:09:46.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.831+0000 7f2dd5deb740 1 -- 172.21.15.12:0/2263182437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ba0a3fa40 msgr2=0x558ba0a44540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:46.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.831+0000 7f2dd5deb740 1 --2- 172.21.15.12:0/2263182437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ba0a3fa40 0x558ba0a44540 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc009d50 tx=0x7f2dbc0117b0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.831+0000 7f2dd5deb740 1 -- 172.21.15.12:0/2263182437 shutdown_connections 2024-03-23T23:09:46.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.831+0000 7f2dd5deb740 1 --2- 172.21.15.12:0/2263182437 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ba0a44bb0 0x558ba0a47040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.831+0000 7f2dd5deb740 1 --2- 172.21.15.12:0/2263182437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ba0a3fa40 0x558ba0a44540 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.831+0000 7f2dd5deb740 1 --2- 172.21.15.12:0/2263182437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ba0a3f050 0x558ba0a3f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.831+0000 7f2dd5deb740 1 -- 172.21.15.12:0/2263182437 >> 172.21.15.12:0/2263182437 conn(0x558ba0a07200 msgr2=0x558ba0a3a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:46.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.831+0000 7f2dcb400640 1 --2- 172.21.15.12:0/1692767007 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2db00a2880 0x7f2db00a4d40 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2db802c240 tx=0x7f2db802e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:46.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.832+0000 7f2dd5deb740 1 -- 172.21.15.12:0/2263182437 shutdown_connections 2024-03-23T23:09:46.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.832+0000 7f2dd5deb740 1 -- 172.21.15.12:0/2263182437 wait complete. 2024-03-23T23:09:46.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.832+0000 7f2dd5deb740 1 -- start start 2024-03-23T23:09:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.832+0000 7f2dd5deb740 1 Processor -- start 2024-03-23T23:09:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.833+0000 7f2dd5deb740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ba0a3f050 0x558ba0a49030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.833+0000 7f2dd5deb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ba0a3fa40 0x558ba0a49570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.833+0000 7f2dd5deb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ba0a44bb0 0x558ba0a505b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.833+0000 7f2dd5deb740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ba0a47b60 con 0x558ba0a3f050 2024-03-23T23:09:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.833+0000 7f2dd5deb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ba095fe80 con 0x558ba0a3fa40 2024-03-23T23:09:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.833+0000 7f2dd5deb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ba0a47ce0 con 0x558ba0a44bb0 2024-03-23T23:09:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.833+0000 7f2dd5deb740 10 client.0 _ll_register_callbacks cb 0x558ba0972130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.833+0000 7f2dcbe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ba0a3f050 0x558ba0a49030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.833+0000 7f2dcb400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ba0a3fa40 0x558ba0a49570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.833+0000 7f2dd1000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ba0a44bb0 0x558ba0a505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.833+0000 7f2dcbe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ba0a3f050 0x558ba0a49030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39140/0 (socket says 172.21.15.12:39140) 2024-03-23T23:09:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.833+0000 7f2dcbe00640 1 -- 172.21.15.12:0/346701914 learned_addr learned my addr 172.21.15.12:0/346701914 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.833+0000 7f2dd1000640 1 -- 172.21.15.12:0/346701914 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ba0a3fa40 msgr2=0x558ba0a49570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:46.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.833+0000 7f2dd1000640 1 --2- 172.21.15.12:0/346701914 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ba0a3fa40 0x558ba0a49570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.833+0000 7f2dd1000640 1 -- 172.21.15.12:0/346701914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ba0a3f050 msgr2=0x558ba0a49030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:46.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.833+0000 7f2dd1000640 1 --2- 172.21.15.12:0/346701914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ba0a3f050 0x558ba0a49030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.833+0000 7f2dd1000640 1 -- 172.21.15.12:0/346701914 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558ba07827d0 con 0x558ba0a44bb0 2024-03-23T23:09:46.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.833+0000 7f2dcbe00640 1 --2- 172.21.15.12:0/346701914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ba0a3f050 0x558ba0a49030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:46.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.834+0000 7f2dd1000640 1 --2- 172.21.15.12:0/346701914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ba0a44bb0 0x558ba0a505b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f2dc40049d0 tx=0x7f2dc409b460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61f407063023c674 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:46.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.834+0000 7f2db6000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:09:46.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.834+0000 7f2db6000640 1 -- 172.21.15.12:0/346701914 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc40b4dc0 con 0x558ba0a44bb0 2024-03-23T23:09:46.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.834+0000 7f2db6000640 1 -- 172.21.15.12:0/346701914 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2dc4005310 con 0x558ba0a44bb0 2024-03-23T23:09:46.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.834+0000 7f2dd5deb740 1 -- 172.21.15.12:0/346701914 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558ba09b1790 con 0x558ba0a44bb0 2024-03-23T23:09:46.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.834+0000 7f2db6000640 1 -- 172.21.15.12:0/346701914 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2dc409fc50 con 0x558ba0a44bb0 2024-03-23T23:09:46.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.834+0000 7f2dd5deb740 10 client.27559 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:46.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.834+0000 7f2dd5deb740 1 -- 172.21.15.12:0/346701914 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x558ba06b83f0 con 0x558ba0a44bb0 2024-03-23T23:09:46.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.834+0000 7f2db6000640 1 -- 172.21.15.12:0/346701914 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x558ba09b1790 con 0x558ba0a44bb0 2024-03-23T23:09:46.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.835+0000 7f2db6000640 1 -- 172.21.15.12:0/346701914 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f2dc409af50 con 0x558ba0a44bb0 2024-03-23T23:09:46.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.835+0000 7f2dd5deb740 10 client.27559 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:46.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.835+0000 7f2dd5deb740 10 client.27559 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:46.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.835+0000 7f2dd5deb740 20 client.27559 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:46.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.835+0000 7f2dd5deb740 1 --2- 172.21.15.12:0/346701914 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558ba0a54e50 0x558ba0a572c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.835+0000 7f2dd5deb740 4 client.27559 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:09:46.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.835+0000 7f2dd5deb740 1 -- 172.21.15.12:0/346701914 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558ba09abae0 con 0x558ba0a54e50 2024-03-23T23:09:46.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.835+0000 7f2dcbe00640 1 --2- 172.21.15.12:0/346701914 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558ba0a54e50 0x558ba0a572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.836+0000 7f2dcbe00640 1 --2- 172.21.15.12:0/346701914 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558ba0a54e50 0x558ba0a572c0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=502d5f7664f8530 server_cookie=4245ce33e0b33ff6 in_seq=0 out_seq=0 2024-03-23T23:09:46.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.836+0000 7f2db6000640 10 client.27559 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:46.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.836+0000 7f2db6000640 1 -- 172.21.15.12:0/346701914 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558ba09abae0 con 0x558ba0a54e50 2024-03-23T23:09:46.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.836+0000 7f2db6000640 10 client.27559 handle_command_reply: tid=0 2024-03-23T23:09:46.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.836+0000 7f2dd5deb740 1 client.27559 shutdown 2024-03-23T23:09:46.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.837+0000 7f2dd5deb740 1 -- 172.21.15.12:0/346701914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ba0a44bb0 msgr2=0x558ba0a505b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:46.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.837+0000 7f2dd5deb740 1 --2- 172.21.15.12:0/346701914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ba0a44bb0 0x558ba0a505b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f2dc40049d0 tx=0x7f2dc409b460 comp rx=0 tx=0).stop 2024-03-23T23:09:46.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.837+0000 7f2dd5deb740 1 -- 172.21.15.12:0/346701914 shutdown_connections 2024-03-23T23:09:46.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.837+0000 7f2dd5deb740 1 --2- 172.21.15.12:0/346701914 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558ba0a54e50 0x558ba0a572c0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.837+0000 7f2dd5deb740 1 --2- 172.21.15.12:0/346701914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ba0a44bb0 0x558ba0a505b0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.837+0000 7f2dd5deb740 1 --2- 172.21.15.12:0/346701914 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ba0a3fa40 0x558ba0a49570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.837+0000 7f2dd5deb740 1 --2- 172.21.15.12:0/346701914 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ba0a3f050 0x558ba0a49030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.837+0000 7f2dd5deb740 1 -- 172.21.15.12:0/346701914 >> 172.21.15.12:0/346701914 conn(0x558ba0a07200 msgr2=0x558ba0a39bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:46.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.837+0000 7f2db6000640 0 client.27559 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:46.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.837+0000 7f2dd5deb740 1 -- 172.21.15.12:0/346701914 shutdown_connections 2024-03-23T23:09:46.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.837+0000 7f2dd5deb740 1 -- 172.21.15.12:0/346701914 wait complete. 2024-03-23T23:09:46.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.837+0000 7f2dd5deb740 20 client.27559 trim_cache size 0 max 16384 2024-03-23T23:09:46.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.855+0000 7f2dd5deb740 1 Processor -- start 2024-03-23T23:09:46.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.855+0000 7f2dd5deb740 1 -- start start 2024-03-23T23:09:46.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.855+0000 7f2dd5deb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ba0a3f050 0x558ba0a40bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.855+0000 7f2dd5deb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ba0a3fa40 0x558ba0a50570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.855+0000 7f2dd5deb740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ba0a44bb0 0x558ba0a39330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.855+0000 7f2dd5deb740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc1b13e0 con 0x558ba0a44bb0 2024-03-23T23:09:46.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.855+0000 7f2dd5deb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc1aeee0 con 0x558ba0a3fa40 2024-03-23T23:09:46.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.855+0000 7f2dd5deb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc1acf80 con 0x558ba0a3f050 2024-03-23T23:09:46.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.855+0000 7f2dcb400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ba0a3fa40 0x558ba0a50570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.855+0000 7f2dcb400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ba0a3fa40 0x558ba0a50570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:33856/0 (socket says 172.21.15.12:33856) 2024-03-23T23:09:46.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.855+0000 7f2dcb400640 1 -- 172.21.15.12:0/284297507 learned_addr learned my addr 172.21.15.12:0/284297507 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:46.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.856+0000 7f2dcbe00640 1 --2- 172.21.15.12:0/284297507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ba0a3f050 0x558ba0a40bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.856+0000 7f2dd1000640 1 --2- 172.21.15.12:0/284297507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ba0a44bb0 0x558ba0a39330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.856+0000 7f2dcb400640 1 -- 172.21.15.12:0/284297507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ba0a3f050 msgr2=0x558ba0a40bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:46.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.856+0000 7f2dcb400640 1 --2- 172.21.15.12:0/284297507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ba0a3f050 0x558ba0a40bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.856+0000 7f2dcb400640 1 -- 172.21.15.12:0/284297507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ba0a44bb0 msgr2=0x558ba0a39330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:46.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.856+0000 7f2dcb400640 1 --2- 172.21.15.12:0/284297507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ba0a44bb0 0x558ba0a39330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.856+0000 7f2dcb400640 1 -- 172.21.15.12:0/284297507 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558ba06780f0 con 0x558ba0a3fa40 2024-03-23T23:09:46.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.856+0000 7f2dd1000640 1 --2- 172.21.15.12:0/284297507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ba0a44bb0 0x558ba0a39330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:09:46.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.856+0000 7f2dcb400640 1 --2- 172.21.15.12:0/284297507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ba0a3fa40 0x558ba0a50570 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f2db802e010 tx=0x7f2db8041070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76a3d34020769270 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:46.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.857+0000 7f2dc0c00640 1 -- 172.21.15.12:0/284297507 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db80043c0 con 0x558ba0a3fa40 2024-03-23T23:09:46.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.857+0000 7f2dc0c00640 1 -- 172.21.15.12:0/284297507 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2db8003ad0 con 0x558ba0a3fa40 2024-03-23T23:09:46.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.857+0000 7f2dc0c00640 1 -- 172.21.15.12:0/284297507 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db80349c0 con 0x558ba0a3fa40 2024-03-23T23:09:46.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.857+0000 7f2dd5deb740 1 -- 172.21.15.12:0/284297507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ba0a3fa40 msgr2=0x558ba0a50570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:46.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.857+0000 7f2dd5deb740 1 --2- 172.21.15.12:0/284297507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ba0a3fa40 0x558ba0a50570 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f2db802e010 tx=0x7f2db8041070 comp rx=0 tx=0).stop 2024-03-23T23:09:46.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.857+0000 7f2dd5deb740 1 -- 172.21.15.12:0/284297507 shutdown_connections 2024-03-23T23:09:46.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.857+0000 7f2dd5deb740 1 --2- 172.21.15.12:0/284297507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ba0a44bb0 0x558ba0a39330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.857+0000 7f2dd5deb740 1 --2- 172.21.15.12:0/284297507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ba0a3fa40 0x558ba0a50570 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.857+0000 7f2dd5deb740 1 --2- 172.21.15.12:0/284297507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ba0a3f050 0x558ba0a40bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.857+0000 7f2dd5deb740 1 -- 172.21.15.12:0/284297507 >> 172.21.15.12:0/284297507 conn(0x558ba0a07200 msgr2=0x558ba0a404b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:46.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.857+0000 7f2dd5deb740 1 -- 172.21.15.12:0/284297507 shutdown_connections 2024-03-23T23:09:46.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.857+0000 7f2dd5deb740 1 -- 172.21.15.12:0/284297507 wait complete. 2024-03-23T23:09:46.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.858+0000 7f2dd5deb740 1 -- start start 2024-03-23T23:09:46.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.858+0000 7f2dd5deb740 1 Processor -- start 2024-03-23T23:09:46.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.858+0000 7f2dd5deb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ba0a3f050 0x558ba0a4e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.858+0000 7f2dd5deb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ba0a3fa40 0x558ba0a4e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.858+0000 7f2dd5deb740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ba0a44bb0 0x558ba0a4f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.858+0000 7f2dd5deb740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc1ad580 con 0x558ba0a44bb0 2024-03-23T23:09:46.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.858+0000 7f2dd5deb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc1ae880 con 0x558ba0a3fa40 2024-03-23T23:09:46.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.858+0000 7f2dd5deb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc1adbc0 con 0x558ba0a3f050 2024-03-23T23:09:46.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.858+0000 7f2dd5deb740 10 client.0 _ll_register_callbacks cb 0x7f2dc4004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:46.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.859+0000 7f2dcbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ba0a3f050 0x558ba0a4e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.859+0000 7f2dcb400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ba0a3fa40 0x558ba0a4e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.859+0000 7f2dcbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ba0a3f050 0x558ba0a4e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:57842/0 (socket says 172.21.15.12:57842) 2024-03-23T23:09:46.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.859+0000 7f2dcbe00640 1 -- 172.21.15.12:0/3383428822 learned_addr learned my addr 172.21.15.12:0/3383428822 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:46.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.859+0000 7f2dd1000640 1 --2- 172.21.15.12:0/3383428822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ba0a44bb0 0x558ba0a4f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.859+0000 7f2dcb400640 1 -- 172.21.15.12:0/3383428822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ba0a3f050 msgr2=0x558ba0a4e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:46.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.859+0000 7f2dcb400640 1 --2- 172.21.15.12:0/3383428822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ba0a3f050 0x558ba0a4e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.859+0000 7f2dcb400640 1 -- 172.21.15.12:0/3383428822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ba0a44bb0 msgr2=0x558ba0a4f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:46.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.859+0000 7f2dcb400640 1 --2- 172.21.15.12:0/3383428822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ba0a44bb0 0x558ba0a4f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.859+0000 7f2dcb400640 1 -- 172.21.15.12:0/3383428822 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558ba0a4fbc0 con 0x558ba0a3fa40 2024-03-23T23:09:46.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.859+0000 7f2dcbe00640 1 --2- 172.21.15.12:0/3383428822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ba0a3f050 0x558ba0a4e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:46.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.859+0000 7f2dcb400640 1 --2- 172.21.15.12:0/3383428822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ba0a3fa40 0x558ba0a4e5c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2db8006010 tx=0x7f2db8004450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=daffcec034c1b646 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:46.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.859+0000 7f2db6000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:09:46.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.859+0000 7f2db6000640 1 -- 172.21.15.12:0/3383428822 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db805e070 con 0x558ba0a3fa40 2024-03-23T23:09:46.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.860+0000 7f2dd5deb740 1 -- 172.21.15.12:0/3383428822 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2db8003ad0 con 0x558ba0a3fa40 2024-03-23T23:09:46.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.860+0000 7f2db6000640 1 -- 172.21.15.12:0/3383428822 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2db8034740 con 0x558ba0a3fa40 2024-03-23T23:09:46.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.860+0000 7f2db6000640 1 -- 172.21.15.12:0/3383428822 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2db806a050 con 0x558ba0a3fa40 2024-03-23T23:09:46.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.860+0000 7f2dd5deb740 10 client.27602 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:46.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.860+0000 7f2dd5deb740 1 -- 172.21.15.12:0/3383428822 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x558ba0ab0bc0 con 0x558ba0a3fa40 2024-03-23T23:09:46.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.860+0000 7f2db6000640 1 -- 172.21.15.12:0/3383428822 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2db8003ad0 con 0x558ba0a3fa40 2024-03-23T23:09:46.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.860+0000 7f2db6000640 1 -- 172.21.15.12:0/3383428822 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f2db8056860 con 0x558ba0a3fa40 2024-03-23T23:09:46.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.860+0000 7f2dd5deb740 10 client.27602 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:46.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.860+0000 7f2dd5deb740 10 client.27602 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:46.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.860+0000 7f2dd5deb740 20 client.27602 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:46.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.860+0000 7f2dd5deb740 1 --2- 172.21.15.12:0/3383428822 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558ba0ab14a0 0x558ba0ab38c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:46.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.860+0000 7f2dd5deb740 4 client.27602 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:09:46.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.860+0000 7f2dd5deb740 1 -- 172.21.15.12:0/3383428822 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558ba09293a0 con 0x558ba0ab14a0 2024-03-23T23:09:46.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.861+0000 7f2dcbe00640 1 --2- 172.21.15.12:0/3383428822 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558ba0ab14a0 0x558ba0ab38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:46.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.861+0000 7f2dcbe00640 1 --2- 172.21.15.12:0/3383428822 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558ba0ab14a0 0x558ba0ab38c0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=975c5d40046d1b4d server_cookie=df6748ecee2fd127 in_seq=0 out_seq=0 2024-03-23T23:09:46.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.861+0000 7f2db6000640 10 client.27602 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:46.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.861+0000 7f2db6000640 1 -- 172.21.15.12:0/3383428822 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x558ba09293a0 con 0x558ba0ab14a0 2024-03-23T23:09:46.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.861+0000 7f2db6000640 10 client.27602 handle_command_reply: tid=0 2024-03-23T23:09:46.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.861+0000 7f2dd5deb740 1 client.27602 shutdown 2024-03-23T23:09:46.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.862+0000 7f2dd5deb740 1 -- 172.21.15.12:0/3383428822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ba0a3fa40 msgr2=0x558ba0a4e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:46.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.862+0000 7f2dd5deb740 1 --2- 172.21.15.12:0/3383428822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ba0a3fa40 0x558ba0a4e5c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2db8006010 tx=0x7f2db8004450 comp rx=0 tx=0).stop 2024-03-23T23:09:46.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.862+0000 7f2dd5deb740 1 -- 172.21.15.12:0/3383428822 shutdown_connections 2024-03-23T23:09:46.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.862+0000 7f2dd5deb740 1 --2- 172.21.15.12:0/3383428822 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558ba0ab14a0 0x558ba0ab38c0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.862+0000 7f2dd5deb740 1 --2- 172.21.15.12:0/3383428822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ba0a44bb0 0x558ba0a4f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.862+0000 7f2dd5deb740 1 --2- 172.21.15.12:0/3383428822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ba0a3fa40 0x558ba0a4e5c0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.862+0000 7f2dd5deb740 1 --2- 172.21.15.12:0/3383428822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ba0a3f050 0x558ba0a4e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.862+0000 7f2dd5deb740 1 -- 172.21.15.12:0/3383428822 >> 172.21.15.12:0/3383428822 conn(0x558ba0a07200 msgr2=0x558ba0a3e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:46.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.862+0000 7f2db6000640 0 client.27602 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:46.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.862+0000 7f2dd5deb740 1 -- 172.21.15.12:0/3383428822 shutdown_connections 2024-03-23T23:09:46.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.862+0000 7f2dd5deb740 1 -- 172.21.15.12:0/3383428822 wait complete. 2024-03-23T23:09:46.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.862+0000 7f2dd5deb740 20 client.27602 trim_cache size 0 max 16384 2024-03-23T23:09:46.868 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:09:46.868 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T23:09:46.868 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-03-23T23:09:46.868 INFO:teuthology.orchestra.run.smithi012.stdout: "12a2d79e-52dd-45df-aa9b-2379ff872728": { 2024-03-23T23:09:46.868 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-03-23T23:09:46.868 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "12a2d79e-52dd-45df-aa9b-2379ff872728", 2024-03-23T23:09:46.868 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-03-23T23:09:46.868 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:09:46.868 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:09:46.868 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:09:46.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.863+0000 7f2dd1a00640 1 -- 172.21.15.12:0/1692767007 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2db00a2880 msgr2=0x7f2db00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:46.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.863+0000 7f2dd1a00640 1 --2- 172.21.15.12:0/1692767007 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2db00a2880 0x7f2db00a4d40 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2db802c240 tx=0x7f2db802e040 comp rx=0 tx=0).stop 2024-03-23T23:09:46.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.863+0000 7f2dd1a00640 1 -- 172.21.15.12:0/1692767007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dcc10db50 msgr2=0x7f2dcc1a5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:46.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.863+0000 7f2dd1a00640 1 --2- 172.21.15.12:0/1692767007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dcc10db50 0x7f2dcc1a5420 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f2dc400eb10 tx=0x7f2dc400c2d0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.863+0000 7f2dd1a00640 1 -- 172.21.15.12:0/1692767007 shutdown_connections 2024-03-23T23:09:46.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.863+0000 7f2dd1a00640 1 --2- 172.21.15.12:0/1692767007 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2db00a2880 0x7f2db00a4d40 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.863+0000 7f2dd1a00640 1 --2- 172.21.15.12:0/1692767007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dcc10db50 0x7f2dcc1a5420 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.863+0000 7f2dd1a00640 1 --2- 172.21.15.12:0/1692767007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dcc0756e0 0x7f2dcc19e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.863+0000 7f2dd1a00640 1 --2- 172.21.15.12:0/1692767007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2dcc074100 0x7f2dcc19de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:46.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.863+0000 7f2dd1a00640 1 -- 172.21.15.12:0/1692767007 >> 172.21.15.12:0/1692767007 conn(0x7f2dcc0fcb10 msgr2=0x7f2dcc100ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:46.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.863+0000 7f2dd1a00640 1 -- 172.21.15.12:0/1692767007 shutdown_connections 2024-03-23T23:09:46.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:09:46.863+0000 7f2dd1a00640 1 -- 172.21.15.12:0/1692767007 wait complete. 2024-03-23T23:09:46.880 INFO:tasks.cephfs.filesystem:scrub status for tag:12a2d79e-52dd-45df-aa9b-2379ff872728 - {'path': '/', 'tag': '12a2d79e-52dd-45df-aa9b-2379ff872728', 'options': 'recursive,force'} 2024-03-23T23:09:47.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:46 smithi012 ceph-mon[31695]: pgmap v629: 97 pgs: 97 active+clean; 1.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:09:47.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:46 smithi173 ceph-mon[36116]: pgmap v629: 97 pgs: 97 active+clean; 1.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:09:47.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:46 smithi203 ceph-mon[36693]: pgmap v629: 97 pgs: 97 active+clean; 1.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:09:47.260 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.260 INFO:tasks.workunit.client.0.smithi012.stdout:313/600: open fd = 3 2024-03-23T23:09:47.260 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.261 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.261 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.261 INFO:tasks.workunit.client.0.smithi012.stdout:314/600: open fd = 3 2024-03-23T23:09:47.261 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.261 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.261 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.261 INFO:tasks.workunit.client.0.smithi012.stdout:315/600: open fd = 3 2024-03-23T23:09:47.261 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.261 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.261 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.261 INFO:tasks.workunit.client.0.smithi012.stdout:316/600: open fd = 3 2024-03-23T23:09:47.261 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.261 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.262 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.262 INFO:tasks.workunit.client.0.smithi012.stdout:317/600: open fd = 3 2024-03-23T23:09:47.262 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.262 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.262 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.262 INFO:tasks.workunit.client.0.smithi012.stdout:318/600: open fd = 3 2024-03-23T23:09:47.262 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.262 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.262 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.262 INFO:tasks.workunit.client.0.smithi012.stdout:319/600: open fd = 3 2024-03-23T23:09:47.262 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.262 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.262 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.262 INFO:tasks.workunit.client.0.smithi012.stdout:320/600: open fd = 3 2024-03-23T23:09:47.263 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.263 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.263 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.263 INFO:tasks.workunit.client.0.smithi012.stdout:321/600: open fd = 3 2024-03-23T23:09:47.263 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.263 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.263 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.263 INFO:tasks.workunit.client.0.smithi012.stdout:322/600: open fd = 3 2024-03-23T23:09:47.263 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.263 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.263 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.263 INFO:tasks.workunit.client.0.smithi012.stdout:323/600: open fd = 3 2024-03-23T23:09:47.263 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.263 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.264 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.264 INFO:tasks.workunit.client.0.smithi012.stdout:324/600: open fd = 3 2024-03-23T23:09:47.264 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.264 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.264 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.264 INFO:tasks.workunit.client.0.smithi012.stdout:325/600: open fd = 3 2024-03-23T23:09:47.264 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.264 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.264 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.264 INFO:tasks.workunit.client.0.smithi012.stdout:326/600: open fd = 3 2024-03-23T23:09:47.264 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.264 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.264 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.265 INFO:tasks.workunit.client.0.smithi012.stdout:327/600: open fd = 3 2024-03-23T23:09:47.265 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.265 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.265 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.265 INFO:tasks.workunit.client.0.smithi012.stdout:328/600: open fd = 3 2024-03-23T23:09:47.265 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.265 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.265 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.265 INFO:tasks.workunit.client.0.smithi012.stdout:329/600: open fd = 3 2024-03-23T23:09:47.265 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.265 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.266 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.266 INFO:tasks.workunit.client.0.smithi012.stdout:330/600: open fd = 3 2024-03-23T23:09:47.266 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.266 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.266 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.266 INFO:tasks.workunit.client.0.smithi012.stdout:331/600: open fd = 3 2024-03-23T23:09:47.266 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.266 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.266 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.266 INFO:tasks.workunit.client.0.smithi012.stdout:332/600: open fd = 3 2024-03-23T23:09:47.266 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.266 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.266 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.267 INFO:tasks.workunit.client.0.smithi012.stdout:333/600: open fd = 3 2024-03-23T23:09:47.267 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.267 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.267 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.267 INFO:tasks.workunit.client.0.smithi012.stdout:334/600: open fd = 3 2024-03-23T23:09:47.267 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.267 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.267 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.267 INFO:tasks.workunit.client.0.smithi012.stdout:335/600: open fd = 3 2024-03-23T23:09:47.267 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.267 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.267 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.267 INFO:tasks.workunit.client.0.smithi012.stdout:336/600: open fd = 3 2024-03-23T23:09:47.268 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.268 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.268 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.268 INFO:tasks.workunit.client.0.smithi012.stdout:337/600: open fd = 3 2024-03-23T23:09:47.268 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.268 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.268 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.268 INFO:tasks.workunit.client.0.smithi012.stdout:338/600: open fd = 3 2024-03-23T23:09:47.268 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.268 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.268 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.268 INFO:tasks.workunit.client.0.smithi012.stdout:339/600: open fd = 3 2024-03-23T23:09:47.268 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.269 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.269 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.269 INFO:tasks.workunit.client.0.smithi012.stdout:340/600: open fd = 3 2024-03-23T23:09:47.269 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.269 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.269 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.269 INFO:tasks.workunit.client.0.smithi012.stdout:341/600: open fd = 3 2024-03-23T23:09:47.269 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.269 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.269 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.269 INFO:tasks.workunit.client.0.smithi012.stdout:342/600: open fd = 3 2024-03-23T23:09:47.269 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.269 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.269 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.270 INFO:tasks.workunit.client.0.smithi012.stdout:343/600: open fd = 3 2024-03-23T23:09:47.270 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.270 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.270 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.270 INFO:tasks.workunit.client.0.smithi012.stdout:344/600: open fd = 3 2024-03-23T23:09:47.270 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.270 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.270 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.270 INFO:tasks.workunit.client.0.smithi012.stdout:345/600: open fd = 3 2024-03-23T23:09:47.270 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.270 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.270 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.270 INFO:tasks.workunit.client.0.smithi012.stdout:346/600: open fd = 3 2024-03-23T23:09:47.270 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.271 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.271 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.271 INFO:tasks.workunit.client.0.smithi012.stdout:347/600: open fd = 3 2024-03-23T23:09:47.271 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.271 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.271 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.271 INFO:tasks.workunit.client.0.smithi012.stdout:348/600: open fd = 3 2024-03-23T23:09:47.271 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.271 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.271 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.271 INFO:tasks.workunit.client.0.smithi012.stdout:349/600: open fd = 3 2024-03-23T23:09:47.271 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.271 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.271 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.272 INFO:tasks.workunit.client.0.smithi012.stdout:350/600: open fd = 3 2024-03-23T23:09:47.272 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.272 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.272 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.272 INFO:tasks.workunit.client.0.smithi012.stdout:351/600: open fd = 3 2024-03-23T23:09:47.272 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.272 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.272 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.272 INFO:tasks.workunit.client.0.smithi012.stdout:352/600: open fd = 3 2024-03-23T23:09:47.272 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.272 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.272 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.272 INFO:tasks.workunit.client.0.smithi012.stdout:353/600: open fd = 3 2024-03-23T23:09:47.272 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.272 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.273 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.273 INFO:tasks.workunit.client.0.smithi012.stdout:354/600: open fd = 3 2024-03-23T23:09:47.273 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.273 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.273 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.273 INFO:tasks.workunit.client.0.smithi012.stdout:355/600: open fd = 3 2024-03-23T23:09:47.273 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.273 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.273 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.273 INFO:tasks.workunit.client.0.smithi012.stdout:356/600: open fd = 3 2024-03-23T23:09:47.273 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.273 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.273 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.274 INFO:tasks.workunit.client.0.smithi012.stdout:357/600: open fd = 3 2024-03-23T23:09:47.274 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.274 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.274 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.274 INFO:tasks.workunit.client.0.smithi012.stdout:358/600: open fd = 3 2024-03-23T23:09:47.274 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.274 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.274 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.274 INFO:tasks.workunit.client.0.smithi012.stdout:359/600: open fd = 3 2024-03-23T23:09:47.274 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.274 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.274 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.274 INFO:tasks.workunit.client.0.smithi012.stdout:360/600: open fd = 3 2024-03-23T23:09:47.275 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.275 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.275 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.275 INFO:tasks.workunit.client.0.smithi012.stdout:361/600: open fd = 3 2024-03-23T23:09:47.275 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.275 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.275 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.275 INFO:tasks.workunit.client.0.smithi012.stdout:362/600: open fd = 3 2024-03-23T23:09:47.275 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.275 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.275 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.275 INFO:tasks.workunit.client.0.smithi012.stdout:363/600: open fd = 3 2024-03-23T23:09:47.275 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.275 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.275 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.276 INFO:tasks.workunit.client.0.smithi012.stdout:364/600: open fd = 3 2024-03-23T23:09:47.276 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.276 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.276 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.276 INFO:tasks.workunit.client.0.smithi012.stdout:365/600: open fd = 3 2024-03-23T23:09:47.276 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.276 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.276 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.276 INFO:tasks.workunit.client.0.smithi012.stdout:366/600: open fd = 3 2024-03-23T23:09:47.276 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.276 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.276 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.276 INFO:tasks.workunit.client.0.smithi012.stdout:367/600: open fd = 3 2024-03-23T23:09:47.276 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.276 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.277 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.277 INFO:tasks.workunit.client.0.smithi012.stdout:368/600: open fd = 3 2024-03-23T23:09:47.277 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.277 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.277 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.277 INFO:tasks.workunit.client.0.smithi012.stdout:369/600: open fd = 3 2024-03-23T23:09:47.277 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.277 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.277 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.277 INFO:tasks.workunit.client.0.smithi012.stdout:370/600: open fd = 3 2024-03-23T23:09:47.277 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.277 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.277 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.277 INFO:tasks.workunit.client.0.smithi012.stdout:371/600: open fd = 3 2024-03-23T23:09:47.277 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.278 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.278 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.278 INFO:tasks.workunit.client.0.smithi012.stdout:372/600: open fd = 3 2024-03-23T23:09:47.278 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.278 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.278 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.278 INFO:tasks.workunit.client.0.smithi012.stdout:373/600: open fd = 3 2024-03-23T23:09:47.278 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.278 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.278 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:47.278 INFO:tasks.workunit.client.0.smithi012.stdout:374/600: open fd = 3 2024-03-23T23:09:47.278 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.278 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:09:47.278 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:09:48.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:47 smithi012 ceph-mon[31695]: scrub queued for path: / 2024-03-23T23:09:48.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:47 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:09:48.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:47 smithi012 ceph-mon[31695]: scrub summary: active paths [/] 2024-03-23T23:09:48.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:47 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:09:48.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:47 smithi173 ceph-mon[36116]: scrub queued for path: / 2024-03-23T23:09:48.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:47 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:09:48.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:47 smithi173 ceph-mon[36116]: scrub summary: active paths [/] 2024-03-23T23:09:48.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:47 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:09:48.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:47 smithi203 ceph-mon[36693]: scrub queued for path: / 2024-03-23T23:09:48.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:47 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:09:48.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:47 smithi203 ceph-mon[36693]: scrub summary: active paths [/] 2024-03-23T23:09:48.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:47 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:09:49.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:48 smithi012 ceph-mon[31695]: pgmap v630: 97 pgs: 97 active+clean; 1.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:09:49.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:48 smithi173 ceph-mon[36116]: pgmap v630: 97 pgs: 97 active+clean; 1.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:09:49.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:48 smithi203 ceph-mon[36693]: pgmap v630: 97 pgs: 97 active+clean; 1.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:09:51.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:50 smithi012 ceph-mon[31695]: pgmap v631: 97 pgs: 97 active+clean; 1.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:09:51.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:50 smithi173 ceph-mon[36116]: pgmap v631: 97 pgs: 97 active+clean; 1.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:09:51.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:50 smithi203 ceph-mon[36693]: pgmap v631: 97 pgs: 97 active+clean; 1.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:09:52.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:51 smithi012 ceph-mon[31695]: scrub summary: idle 2024-03-23T23:09:52.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:51 smithi173 ceph-mon[36116]: scrub summary: idle 2024-03-23T23:09:52.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:51 smithi203 ceph-mon[36693]: scrub summary: idle 2024-03-23T23:09:53.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:52 smithi012 ceph-mon[31695]: pgmap v632: 97 pgs: 97 active+clean; 1.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 18 op/s 2024-03-23T23:09:53.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:52 smithi173 ceph-mon[36116]: pgmap v632: 97 pgs: 97 active+clean; 1.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 18 op/s 2024-03-23T23:09:53.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:52 smithi203 ceph-mon[36693]: pgmap v632: 97 pgs: 97 active+clean; 1.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 18 op/s 2024-03-23T23:09:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.553+0000 7f5270000640 1 -- 172.21.15.12:0/761889078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52680772a0 msgr2=0x7f5268077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.553+0000 7f5270000640 1 --2- 172.21.15.12:0/761889078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52680772a0 0x7f5268077700 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f5254009900 tx=0x7f525402bc30 comp rx=0 tx=0).stop 2024-03-23T23:09:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.554+0000 7f5270000640 1 -- 172.21.15.12:0/761889078 shutdown_connections 2024-03-23T23:09:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.554+0000 7f5270000640 1 --2- 172.21.15.12:0/761889078 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5268077c40 0x7f52680784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.554+0000 7f5270000640 1 --2- 172.21.15.12:0/761889078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52680772a0 0x7f5268077700 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.554+0000 7f5270000640 1 --2- 172.21.15.12:0/761889078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5268076060 0x7f5268076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.554+0000 7f5270000640 1 -- 172.21.15.12:0/761889078 >> 172.21.15.12:0/761889078 conn(0x7f52680fee90 msgr2=0x7f52681012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.554+0000 7f5270000640 1 -- 172.21.15.12:0/761889078 shutdown_connections 2024-03-23T23:09:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.554+0000 7f5270000640 1 -- 172.21.15.12:0/761889078 wait complete. 2024-03-23T23:09:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.555+0000 7f5270000640 1 Processor -- start 2024-03-23T23:09:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.555+0000 7f5270000640 1 -- start start 2024-03-23T23:09:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.555+0000 7f5270000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5268076060 0x7f526819def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.555+0000 7f5270000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52680772a0 0x7f526819e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.555+0000 7f5270000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5268077c40 0x7f52681a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.555+0000 7f5270000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52681139d0 con 0x7f5268076060 2024-03-23T23:09:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.555+0000 7f5270000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5268113850 con 0x7f52680772a0 2024-03-23T23:09:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.555+0000 7f5270000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5268113b50 con 0x7f5268077c40 2024-03-23T23:09:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.556+0000 7f526d600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5268076060 0x7f526819def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.556+0000 7f526e000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5268077c40 0x7f52681a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.556+0000 7f526cc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52680772a0 0x7f526819e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.556+0000 7f526d600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5268076060 0x7f526819def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39184/0 (socket says 172.21.15.12:39184) 2024-03-23T23:09:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.556+0000 7f526d600640 1 -- 172.21.15.12:0/1557859527 learned_addr learned my addr 172.21.15.12:0/1557859527 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.556+0000 7f526e000640 1 -- 172.21.15.12:0/1557859527 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52680772a0 msgr2=0x7f526819e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.556+0000 7f526e000640 1 --2- 172.21.15.12:0/1557859527 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52680772a0 0x7f526819e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.556+0000 7f526e000640 1 -- 172.21.15.12:0/1557859527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5268076060 msgr2=0x7f526819def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.556+0000 7f526e000640 1 --2- 172.21.15.12:0/1557859527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5268076060 0x7f526819def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.556+0000 7f526e000640 1 -- 172.21.15.12:0/1557859527 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52681a5b10 con 0x7f5268077c40 2024-03-23T23:09:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.556+0000 7f526e000640 1 --2- 172.21.15.12:0/1557859527 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5268077c40 0x7f52681a54a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f5260009fd0 tx=0x7f526000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.557+0000 7f5266000640 1 -- 172.21.15.12:0/1557859527 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526000ce20 con 0x7f5268077c40 2024-03-23T23:09:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.557+0000 7f5270000640 1 -- 172.21.15.12:0/1557859527 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52681a5e00 con 0x7f5268077c40 2024-03-23T23:09:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.557+0000 7f5266000640 1 -- 172.21.15.12:0/1557859527 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5260005020 con 0x7f5268077c40 2024-03-23T23:09:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.557+0000 7f5266000640 1 -- 172.21.15.12:0/1557859527 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5260010720 con 0x7f5268077c40 2024-03-23T23:09:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.557+0000 7f5270000640 1 -- 172.21.15.12:0/1557859527 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52681a6270 con 0x7f5268077c40 2024-03-23T23:09:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.560+0000 7f5266000640 1 -- 172.21.15.12:0/1557859527 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5260005540 con 0x7f5268077c40 2024-03-23T23:09:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.560+0000 7f5272df4740 1 Processor -- start 2024-03-23T23:09:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.560+0000 7f5272df4740 1 -- start start 2024-03-23T23:09:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.561+0000 7f5272df4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d81e0 0x5588715d8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.561+0000 7f5272df4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5588715d8bd0 0x5588715dd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.561+0000 7f5272df4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5588715ddd40 0x5588715e01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.561+0000 7f5272df4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55887159b3b0 con 0x5588715ddd40 2024-03-23T23:09:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.561+0000 7f5272df4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55887153d830 con 0x5588715d81e0 2024-03-23T23:09:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.561+0000 7f5272df4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588714e65d0 con 0x5588715d8bd0 2024-03-23T23:09:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.561+0000 7f526d600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5588715d8bd0 0x5588715dd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.561+0000 7f526cc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d81e0 0x5588715d8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.561+0000 7f526d600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5588715d8bd0 0x5588715dd6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:57860/0 (socket says 172.21.15.12:57860) 2024-03-23T23:09:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.561+0000 7f526d600640 1 -- 172.21.15.12:0/4263352458 learned_addr learned my addr 172.21.15.12:0/4263352458 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.561+0000 7f526cc00640 1 --2- 172.21.15.12:0/4263352458 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5588715ddd40 0x5588715e01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.562+0000 7f526d600640 1 -- 172.21.15.12:0/4263352458 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d81e0 msgr2=0x5588715d8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.562+0000 7f526d600640 1 --2- 172.21.15.12:0/4263352458 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d81e0 0x5588715d8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.562+0000 7f526d600640 1 -- 172.21.15.12:0/4263352458 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5588715ddd40 msgr2=0x5588715e01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.562+0000 7f526d600640 1 --2- 172.21.15.12:0/4263352458 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5588715ddd40 0x5588715e01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.562+0000 7f526d600640 1 -- 172.21.15.12:0/4263352458 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588714fe2c0 con 0x5588715d8bd0 2024-03-23T23:09:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.562+0000 7f526d600640 1 --2- 172.21.15.12:0/4263352458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5588715d8bd0 0x5588715dd6d0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f5258009dc0 tx=0x7f5258011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa7b546103d7da1b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.562+0000 7f5266000640 1 --2- 172.21.15.12:0/1557859527 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f524c0a28a0 0x7f524c0a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.562+0000 7f525e600640 1 -- 172.21.15.12:0/4263352458 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5258005220 con 0x5588715d8bd0 2024-03-23T23:09:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.562+0000 7f525e600640 1 -- 172.21.15.12:0/4263352458 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5258004c80 con 0x5588715d8bd0 2024-03-23T23:09:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.562+0000 7f525e600640 1 -- 172.21.15.12:0/4263352458 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5258019970 con 0x5588715d8bd0 2024-03-23T23:09:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.562+0000 7f5266000640 1 -- 172.21.15.12:0/1557859527 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f52600cf190 con 0x7f5268077c40 2024-03-23T23:09:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.562+0000 7f526cc00640 1 --2- 172.21.15.12:0/1557859527 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f524c0a28a0 0x7f524c0a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.563+0000 7f5272df4740 1 -- 172.21.15.12:0/4263352458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5588715d8bd0 msgr2=0x5588715dd6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.563+0000 7f5272df4740 1 --2- 172.21.15.12:0/4263352458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5588715d8bd0 0x5588715dd6d0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f5258009dc0 tx=0x7f5258011810 comp rx=0 tx=0).stop 2024-03-23T23:09:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.563+0000 7f526cc00640 1 --2- 172.21.15.12:0/1557859527 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f524c0a28a0 0x7f524c0a4d60 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f525402c240 tx=0x7f525402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.563+0000 7f5272df4740 1 -- 172.21.15.12:0/4263352458 shutdown_connections 2024-03-23T23:09:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.563+0000 7f5272df4740 1 --2- 172.21.15.12:0/4263352458 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5588715ddd40 0x5588715e01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.563+0000 7f5272df4740 1 --2- 172.21.15.12:0/4263352458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5588715d8bd0 0x5588715dd6d0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.563+0000 7f5272df4740 1 --2- 172.21.15.12:0/4263352458 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d81e0 0x5588715d8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.563+0000 7f5272df4740 1 -- 172.21.15.12:0/4263352458 >> 172.21.15.12:0/4263352458 conn(0x5588715d3750 msgr2=0x5588715d3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.563+0000 7f5272df4740 1 -- 172.21.15.12:0/4263352458 shutdown_connections 2024-03-23T23:09:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.563+0000 7f5272df4740 1 -- 172.21.15.12:0/4263352458 wait complete. 2024-03-23T23:09:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.564+0000 7f5272df4740 1 -- start start 2024-03-23T23:09:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.564+0000 7f5272df4740 1 Processor -- start 2024-03-23T23:09:53.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.564+0000 7f5272df4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d81e0 0x5588715e2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:53.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.565+0000 7f5272df4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5588715d8bd0 0x5588715e28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:53.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.565+0000 7f5272df4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5588715ddd40 0x5588715e9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:53.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.565+0000 7f5272df4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588714f9a20 con 0x5588715d8bd0 2024-03-23T23:09:53.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.565+0000 7f5272df4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588715398d0 con 0x5588715d81e0 2024-03-23T23:09:53.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.565+0000 7f5272df4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588715e0cf0 con 0x5588715ddd40 2024-03-23T23:09:53.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.565+0000 7f526d600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d81e0 0x5588715e2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:53.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.565+0000 7f526d600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d81e0 0x5588715e2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:33902/0 (socket says 172.21.15.12:33902) 2024-03-23T23:09:53.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.565+0000 7f526d600640 1 -- 172.21.15.12:0/1365059960 learned_addr learned my addr 172.21.15.12:0/1365059960 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:53.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.565+0000 7f526e000640 1 --2- 172.21.15.12:0/1365059960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5588715ddd40 0x5588715e9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:53.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.565+0000 7f526cc00640 1 --2- 172.21.15.12:0/1365059960 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5588715d8bd0 0x5588715e28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:53.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.565+0000 7f5272df4740 10 client.0 _ll_register_callbacks cb 0x5588715361c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:53.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.565+0000 7f526d600640 1 -- 172.21.15.12:0/1365059960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5588715ddd40 msgr2=0x5588715e9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:53.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.565+0000 7f526d600640 1 --2- 172.21.15.12:0/1365059960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5588715ddd40 0x5588715e9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.565+0000 7f526d600640 1 -- 172.21.15.12:0/1365059960 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5588715d8bd0 msgr2=0x5588715e28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:53.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.565+0000 7f526d600640 1 --2- 172.21.15.12:0/1365059960 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5588715d8bd0 0x5588715e28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.565+0000 7f526d600640 1 -- 172.21.15.12:0/1365059960 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558871252210 con 0x5588715d81e0 2024-03-23T23:09:53.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.565+0000 7f526e000640 1 --2- 172.21.15.12:0/1365059960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5588715ddd40 0x5588715e9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:53.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.566+0000 7f526d600640 1 --2- 172.21.15.12:0/1365059960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d81e0 0x5588715e2390 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f525801b040 tx=0x7f5258004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d789d733d6ffdc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:53.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.566+0000 7f5252a00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:09:53.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.566+0000 7f5252a00640 1 -- 172.21.15.12:0/1365059960 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5258027070 con 0x5588715d81e0 2024-03-23T23:09:53.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.566+0000 7f5252a00640 1 -- 172.21.15.12:0/1365059960 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5258005030 con 0x5588715d81e0 2024-03-23T23:09:53.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.566+0000 7f5272df4740 1 -- 172.21.15.12:0/1365059960 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5258004c80 con 0x5588715d81e0 2024-03-23T23:09:53.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.566+0000 7f5252a00640 1 -- 172.21.15.12:0/1365059960 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5258033050 con 0x5588715d81e0 2024-03-23T23:09:53.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.566+0000 7f5272df4740 10 client.27614 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:53.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.566+0000 7f5272df4740 1 -- 172.21.15.12:0/1365059960 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55887131e620 con 0x5588715d81e0 2024-03-23T23:09:53.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.566+0000 7f5252a00640 1 -- 172.21.15.12:0/1365059960 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5258004c80 con 0x5588715d81e0 2024-03-23T23:09:53.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.567+0000 7f5252a00640 1 -- 172.21.15.12:0/1365059960 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f5258003610 con 0x5588715d81e0 2024-03-23T23:09:53.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.567+0000 7f5272df4740 10 client.27614 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:53.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.567+0000 7f5272df4740 10 client.27614 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:53.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.567+0000 7f5272df4740 20 client.27614 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:53.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.567+0000 7f5272df4740 1 --2- 172.21.15.12:0/1365059960 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5588715edf60 0x5588715f0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:53.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.567+0000 7f5272df4740 4 client.27614 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:09:53.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.567+0000 7f5272df4740 1 -- 172.21.15.12:0/1365059960 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55887154ad70 con 0x5588715edf60 2024-03-23T23:09:53.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.567+0000 7f526cc00640 1 --2- 172.21.15.12:0/1365059960 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5588715edf60 0x5588715f0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:53.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.568+0000 7f526cc00640 1 --2- 172.21.15.12:0/1365059960 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5588715edf60 0x5588715f0350 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d53b50c4fd92171 server_cookie=d15b7729d26e5761 in_seq=0 out_seq=0 2024-03-23T23:09:53.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.568+0000 7f5252a00640 10 client.27614 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:53.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.568+0000 7f5252a00640 1 -- 172.21.15.12:0/1365059960 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55887154ad70 con 0x5588715edf60 2024-03-23T23:09:53.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.568+0000 7f5252a00640 10 client.27614 handle_command_reply: tid=0 2024-03-23T23:09:53.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.568+0000 7f5272df4740 1 client.27614 shutdown 2024-03-23T23:09:53.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.569+0000 7f5272df4740 1 -- 172.21.15.12:0/1365059960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d81e0 msgr2=0x5588715e2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:53.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.569+0000 7f5272df4740 1 --2- 172.21.15.12:0/1365059960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d81e0 0x5588715e2390 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f525801b040 tx=0x7f5258004270 comp rx=0 tx=0).stop 2024-03-23T23:09:53.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.569+0000 7f5272df4740 1 -- 172.21.15.12:0/1365059960 shutdown_connections 2024-03-23T23:09:53.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.569+0000 7f5272df4740 1 --2- 172.21.15.12:0/1365059960 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5588715edf60 0x5588715f0350 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.571 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.569+0000 7f5272df4740 1 --2- 172.21.15.12:0/1365059960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5588715ddd40 0x5588715e9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.571 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.569+0000 7f5252a00640 0 client.27614 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:53.571 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.569+0000 7f5272df4740 1 --2- 172.21.15.12:0/1365059960 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5588715d8bd0 0x5588715e28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.571 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.569+0000 7f5272df4740 1 --2- 172.21.15.12:0/1365059960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d81e0 0x5588715e2390 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.571 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.569+0000 7f5272df4740 1 -- 172.21.15.12:0/1365059960 >> 172.21.15.12:0/1365059960 conn(0x5588715d3750 msgr2=0x5588715d2950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:53.571 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.569+0000 7f5272df4740 1 -- 172.21.15.12:0/1365059960 shutdown_connections 2024-03-23T23:09:53.571 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.569+0000 7f5272df4740 1 -- 172.21.15.12:0/1365059960 wait complete. 2024-03-23T23:09:53.571 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.569+0000 7f5272df4740 20 client.27614 trim_cache size 0 max 16384 2024-03-23T23:09:53.587 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.587+0000 7f5272df4740 1 Processor -- start 2024-03-23T23:09:53.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.587+0000 7f5272df4740 1 -- start start 2024-03-23T23:09:53.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.588+0000 7f5272df4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d81e0 0x5588715d69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:53.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.588+0000 7f5272df4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5588715d8bd0 0x5588715e98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:53.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.588+0000 7f5272df4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5588715ddd40 0x5588715d6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:53.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.588+0000 7f5272df4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52681b1410 con 0x5588715d8bd0 2024-03-23T23:09:53.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.588+0000 7f5272df4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52681aef10 con 0x5588715d81e0 2024-03-23T23:09:53.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.588+0000 7f5272df4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52681acfb0 con 0x5588715ddd40 2024-03-23T23:09:53.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.588+0000 7f526d600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d81e0 0x5588715d69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:53.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.588+0000 7f526d600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d81e0 0x5588715d69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:33910/0 (socket says 172.21.15.12:33910) 2024-03-23T23:09:53.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.588+0000 7f526d600640 1 -- 172.21.15.12:0/2915654094 learned_addr learned my addr 172.21.15.12:0/2915654094 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:53.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.588+0000 7f526cc00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5588715d8bd0 0x5588715e98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:53.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.588+0000 7f526e000640 1 --2- 172.21.15.12:0/2915654094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5588715ddd40 0x5588715d6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:53.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.588+0000 7f526d600640 1 -- 172.21.15.12:0/2915654094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5588715ddd40 msgr2=0x5588715d6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:53.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.588+0000 7f526d600640 1 --2- 172.21.15.12:0/2915654094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5588715ddd40 0x5588715d6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.588+0000 7f526d600640 1 -- 172.21.15.12:0/2915654094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5588715d8bd0 msgr2=0x5588715e98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:53.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.588+0000 7f526d600640 1 --2- 172.21.15.12:0/2915654094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5588715d8bd0 0x5588715e98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.588+0000 7f526d600640 1 -- 172.21.15.12:0/2915654094 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588715f09b0 con 0x5588715d81e0 2024-03-23T23:09:53.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.589+0000 7f526d600640 1 --2- 172.21.15.12:0/2915654094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d81e0 0x5588715d69b0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f5258003fd0 tx=0x7f5258011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ffb12f5b948c629 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:53.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.589+0000 7f525dc00640 1 -- 172.21.15.12:0/2915654094 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5258030070 con 0x5588715d81e0 2024-03-23T23:09:53.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.589+0000 7f525dc00640 1 -- 172.21.15.12:0/2915654094 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52580035f0 con 0x5588715d81e0 2024-03-23T23:09:53.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.589+0000 7f525dc00640 1 -- 172.21.15.12:0/2915654094 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5258027030 con 0x5588715d81e0 2024-03-23T23:09:53.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.590+0000 7f5272df4740 1 -- 172.21.15.12:0/2915654094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d81e0 msgr2=0x5588715d69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:53.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.590+0000 7f5272df4740 1 --2- 172.21.15.12:0/2915654094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d81e0 0x5588715d69b0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f5258003fd0 tx=0x7f5258011810 comp rx=0 tx=0).stop 2024-03-23T23:09:53.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.590+0000 7f5272df4740 1 -- 172.21.15.12:0/2915654094 shutdown_connections 2024-03-23T23:09:53.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.590+0000 7f5272df4740 1 --2- 172.21.15.12:0/2915654094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5588715ddd40 0x5588715d6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.590+0000 7f5272df4740 1 --2- 172.21.15.12:0/2915654094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5588715d8bd0 0x5588715e98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.590+0000 7f5272df4740 1 --2- 172.21.15.12:0/2915654094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d81e0 0x5588715d69b0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.590+0000 7f5272df4740 1 -- 172.21.15.12:0/2915654094 >> 172.21.15.12:0/2915654094 conn(0x5588715d3750 msgr2=0x5588715d9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.590+0000 7f5272df4740 1 -- 172.21.15.12:0/2915654094 shutdown_connections 2024-03-23T23:09:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.590+0000 7f5272df4740 1 -- 172.21.15.12:0/2915654094 wait complete. 2024-03-23T23:09:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.590+0000 7f5272df4740 1 -- start start 2024-03-23T23:09:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.591+0000 7f5272df4740 1 Processor -- start 2024-03-23T23:09:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.591+0000 7f5272df4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5588715d81e0 0x5588715e68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.591+0000 7f5272df4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d8bd0 0x5588715e6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.591+0000 7f5272df4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5588715ddd40 0x5588715e7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.591+0000 7f5272df4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52681ad5b0 con 0x5588715d81e0 2024-03-23T23:09:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.591+0000 7f5272df4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52681ae8b0 con 0x5588715d8bd0 2024-03-23T23:09:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.591+0000 7f5272df4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52681adbf0 con 0x5588715ddd40 2024-03-23T23:09:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.591+0000 7f5272df4740 10 client.0 _ll_register_callbacks cb 0x7f5258009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:09:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.591+0000 7f526cc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d8bd0 0x5588715e6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.591+0000 7f526e000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5588715ddd40 0x5588715e7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.591+0000 7f526cc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d8bd0 0x5588715e6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:33914/0 (socket says 172.21.15.12:33914) 2024-03-23T23:09:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.591+0000 7f526cc00640 1 -- 172.21.15.12:0/387600389 learned_addr learned my addr 172.21.15.12:0/387600389 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:09:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.591+0000 7f526d600640 1 --2- 172.21.15.12:0/387600389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5588715d81e0 0x5588715e68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.592+0000 7f526cc00640 1 -- 172.21.15.12:0/387600389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5588715ddd40 msgr2=0x5588715e7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.592+0000 7f526cc00640 1 --2- 172.21.15.12:0/387600389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5588715ddd40 0x5588715e7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.592+0000 7f526cc00640 1 -- 172.21.15.12:0/387600389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5588715d81e0 msgr2=0x5588715e68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:09:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.592+0000 7f526cc00640 1 --2- 172.21.15.12:0/387600389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5588715d81e0 0x5588715e68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.592+0000 7f526cc00640 1 -- 172.21.15.12:0/387600389 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5588715e8420 con 0x5588715d8bd0 2024-03-23T23:09:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.592+0000 7f526e000640 1 --2- 172.21.15.12:0/387600389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5588715ddd40 0x5588715e7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:09:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.592+0000 7f526cc00640 1 --2- 172.21.15.12:0/387600389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d8bd0 0x5588715e6e20 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f525402ceb0 tx=0x7f5254009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=864f194cccd1caec server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:09:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.592+0000 7f5252a00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:09:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.592+0000 7f5252a00640 1 -- 172.21.15.12:0/387600389 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5254004b90 con 0x5588715d8bd0 2024-03-23T23:09:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.593+0000 7f5252a00640 1 -- 172.21.15.12:0/387600389 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5254004eb0 con 0x5588715d8bd0 2024-03-23T23:09:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.593+0000 7f5272df4740 1 -- 172.21.15.12:0/387600389 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f52580035f0 con 0x5588715d8bd0 2024-03-23T23:09:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.593+0000 7f5252a00640 1 -- 172.21.15.12:0/387600389 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52540050b0 con 0x5588715d8bd0 2024-03-23T23:09:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.593+0000 7f5272df4740 10 client.27620 fetch_fsmap learned FSMap version 23 2024-03-23T23:09:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.593+0000 7f5272df4740 1 -- 172.21.15.12:0/387600389 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5588715e97a0 con 0x5588715d8bd0 2024-03-23T23:09:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.593+0000 7f5252a00640 1 -- 172.21.15.12:0/387600389 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52580035f0 con 0x5588715d8bd0 2024-03-23T23:09:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.593+0000 7f5252a00640 1 -- 172.21.15.12:0/387600389 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f5254003c30 con 0x5588715d8bd0 2024-03-23T23:09:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.593+0000 7f5272df4740 10 client.27620 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:09:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.593+0000 7f5272df4740 10 client.27620 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:09:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.593+0000 7f5272df4740 20 client.27620 populate_metadata read hostname 'smithi012' 2024-03-23T23:09:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.593+0000 7f5272df4740 1 --2- 172.21.15.12:0/387600389 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558871649ad0 0x55887164bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:09:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.593+0000 7f5272df4740 4 client.27620 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:09:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.593+0000 7f5272df4740 1 -- 172.21.15.12:0/387600389 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5588715450c0 con 0x558871649ad0 2024-03-23T23:09:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.594+0000 7f526d600640 1 --2- 172.21.15.12:0/387600389 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558871649ad0 0x55887164bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:09:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.594+0000 7f526d600640 1 --2- 172.21.15.12:0/387600389 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558871649ad0 0x55887164bec0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f12211b1a2eabb49 server_cookie=ee2e075167109882 in_seq=0 out_seq=0 2024-03-23T23:09:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.594+0000 7f5252a00640 10 client.27620 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:53.605 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.605+0000 7f5252a00640 1 -- 172.21.15.12:0/387600389 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5588715450c0 con 0x558871649ad0 2024-03-23T23:09:53.605 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.605+0000 7f5252a00640 10 client.27620 handle_command_reply: tid=0 2024-03-23T23:09:53.605 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.605+0000 7f5272df4740 1 client.27620 shutdown 2024-03-23T23:09:53.605 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.606+0000 7f5272df4740 1 -- 172.21.15.12:0/387600389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d8bd0 msgr2=0x5588715e6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:53.606 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.606+0000 7f5272df4740 1 --2- 172.21.15.12:0/387600389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d8bd0 0x5588715e6e20 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f525402ceb0 tx=0x7f5254009260 comp rx=0 tx=0).stop 2024-03-23T23:09:53.606 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.606+0000 7f5272df4740 1 -- 172.21.15.12:0/387600389 shutdown_connections 2024-03-23T23:09:53.606 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.606+0000 7f5272df4740 1 --2- 172.21.15.12:0/387600389 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x558871649ad0 0x55887164bec0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.606 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.606+0000 7f5272df4740 1 --2- 172.21.15.12:0/387600389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5588715ddd40 0x5588715e7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.606 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.606+0000 7f5272df4740 1 --2- 172.21.15.12:0/387600389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588715d8bd0 0x5588715e6e20 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.606 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.606+0000 7f5272df4740 1 --2- 172.21.15.12:0/387600389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5588715d81e0 0x5588715e68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.606 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.606+0000 7f5252a00640 0 client.27620 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:09:53.606 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.606+0000 7f5272df4740 1 -- 172.21.15.12:0/387600389 >> 172.21.15.12:0/387600389 conn(0x5588715d3750 msgr2=0x5588715d69b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:53.606 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.606+0000 7f5272df4740 1 -- 172.21.15.12:0/387600389 shutdown_connections 2024-03-23T23:09:53.606 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.606+0000 7f5272df4740 1 -- 172.21.15.12:0/387600389 wait complete. 2024-03-23T23:09:53.606 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.606+0000 7f5272df4740 20 client.27620 trim_cache size 0 max 16384 2024-03-23T23:09:53.607 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:09:53.607 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:09:53.607 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:09:53.607 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:09:53.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.607+0000 7f5270000640 1 -- 172.21.15.12:0/1557859527 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f524c0a28a0 msgr2=0x7f524c0a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:53.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.607+0000 7f5270000640 1 --2- 172.21.15.12:0/1557859527 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f524c0a28a0 0x7f524c0a4d60 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f525402c240 tx=0x7f525402df90 comp rx=0 tx=0).stop 2024-03-23T23:09:53.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.607+0000 7f5270000640 1 -- 172.21.15.12:0/1557859527 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5268077c40 msgr2=0x7f52681a54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:09:53.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.607+0000 7f5270000640 1 --2- 172.21.15.12:0/1557859527 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5268077c40 0x7f52681a54a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f5260009fd0 tx=0x7f526000ef10 comp rx=0 tx=0).stop 2024-03-23T23:09:53.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.607+0000 7f5270000640 1 -- 172.21.15.12:0/1557859527 shutdown_connections 2024-03-23T23:09:53.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.607+0000 7f5270000640 1 --2- 172.21.15.12:0/1557859527 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f524c0a28a0 0x7f524c0a4d60 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.607+0000 7f5270000640 1 --2- 172.21.15.12:0/1557859527 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5268077c40 0x7f52681a54a0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.607+0000 7f5270000640 1 --2- 172.21.15.12:0/1557859527 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52680772a0 0x7f526819e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.607+0000 7f5270000640 1 --2- 172.21.15.12:0/1557859527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5268076060 0x7f526819def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:09:53.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.607+0000 7f5270000640 1 -- 172.21.15.12:0/1557859527 >> 172.21.15.12:0/1557859527 conn(0x7f52680fee90 msgr2=0x7f5268100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:09:53.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.608+0000 7f5270000640 1 -- 172.21.15.12:0/1557859527 shutdown_connections 2024-03-23T23:09:53.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:09:53.608+0000 7f5270000640 1 -- 172.21.15.12:0/1557859527 wait complete. 2024-03-23T23:09:55.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:54 smithi012 ceph-mon[31695]: pgmap v633: 97 pgs: 97 active+clean; 1.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:09:55.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:54 smithi173 ceph-mon[36116]: pgmap v633: 97 pgs: 97 active+clean; 1.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:09:55.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:54 smithi203 ceph-mon[36693]: pgmap v633: 97 pgs: 97 active+clean; 1.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:09:57.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:56 smithi012 ceph-mon[31695]: pgmap v634: 97 pgs: 97 active+clean; 1.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 16 op/s 2024-03-23T23:09:57.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:56 smithi173 ceph-mon[36116]: pgmap v634: 97 pgs: 97 active+clean; 1.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 16 op/s 2024-03-23T23:09:57.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:56 smithi203 ceph-mon[36693]: pgmap v634: 97 pgs: 97 active+clean; 1.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 16 op/s 2024-03-23T23:09:59.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:09:58 smithi012 ceph-mon[31695]: pgmap v635: 97 pgs: 97 active+clean; 1.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:09:59.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:09:58 smithi173 ceph-mon[36116]: pgmap v635: 97 pgs: 97 active+clean; 1.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:09:59.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:09:58 smithi203 ceph-mon[36693]: pgmap v635: 97 pgs: 97 active+clean; 1.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:10:01.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:00 smithi012 ceph-mon[31695]: pgmap v636: 97 pgs: 97 active+clean; 1.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:10:01.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:00 smithi012 ceph-mon[31695]: overall HEALTH_OK 2024-03-23T23:10:01.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:00 smithi173 ceph-mon[36116]: pgmap v636: 97 pgs: 97 active+clean; 1.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:10:01.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:00 smithi173 ceph-mon[36116]: overall HEALTH_OK 2024-03-23T23:10:01.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:00 smithi203 ceph-mon[36693]: pgmap v636: 97 pgs: 97 active+clean; 1.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:10:01.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:00 smithi203 ceph-mon[36693]: overall HEALTH_OK 2024-03-23T23:10:03.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:02 smithi012 ceph-mon[31695]: pgmap v637: 97 pgs: 97 active+clean; 1.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:10:03.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:02 smithi173 ceph-mon[36116]: pgmap v637: 97 pgs: 97 active+clean; 1.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:10:03.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:02 smithi203 ceph-mon[36693]: pgmap v637: 97 pgs: 97 active+clean; 1.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:10:05.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:04 smithi012 ceph-mon[31695]: pgmap v638: 97 pgs: 97 active+clean; 1.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:10:05.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:04 smithi173 ceph-mon[36116]: pgmap v638: 97 pgs: 97 active+clean; 1.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:10:05.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:04 smithi203 ceph-mon[36693]: pgmap v638: 97 pgs: 97 active+clean; 1.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:10:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.755+0000 7f5a0d800640 1 -- 172.21.15.12:0/2695022352 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a080fff40 msgr2=0x7f5a08102370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.755+0000 7f5a0d800640 1 --2- 172.21.15.12:0/2695022352 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a080fff40 0x7f5a08102370 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f5a00009ca0 tx=0x7f5a0002c0f0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.755+0000 7f5a0d800640 1 -- 172.21.15.12:0/2695022352 shutdown_connections 2024-03-23T23:10:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.755+0000 7f5a0d800640 1 --2- 172.21.15.12:0/2695022352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a08105260 0x7f5a081056e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.755+0000 7f5a0d800640 1 --2- 172.21.15.12:0/2695022352 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a081028b0 0x7f5a08104d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.755+0000 7f5a0d800640 1 --2- 172.21.15.12:0/2695022352 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a080fff40 0x7f5a08102370 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.755+0000 7f5a0d800640 1 -- 172.21.15.12:0/2695022352 >> 172.21.15.12:0/2695022352 conn(0x7f5a080fbb60 msgr2=0x7f5a080fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.755+0000 7f5a0d800640 1 -- 172.21.15.12:0/2695022352 shutdown_connections 2024-03-23T23:10:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.755+0000 7f5a0d800640 1 -- 172.21.15.12:0/2695022352 wait complete. 2024-03-23T23:10:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.756+0000 7f5a0d800640 1 Processor -- start 2024-03-23T23:10:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.756+0000 7f5a0d800640 1 -- start start 2024-03-23T23:10:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.757+0000 7f5a0d800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a080fff40 0x7f5a0819ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.757+0000 7f5a0d800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a081028b0 0x7f5a0819e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.757+0000 7f5a0d800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a08105260 0x7f5a081a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.757+0000 7f5a0d800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a08113920 con 0x7f5a081028b0 2024-03-23T23:10:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.757+0000 7f5a0d800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a081137a0 con 0x7f5a08105260 2024-03-23T23:10:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.757+0000 7f5a0d800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a08113aa0 con 0x7f5a080fff40 2024-03-23T23:10:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.757+0000 7f5a06a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a080fff40 0x7f5a0819ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.757+0000 7f5a06a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a080fff40 0x7f5a0819ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:39076/0 (socket says 172.21.15.12:39076) 2024-03-23T23:10:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.757+0000 7f5a06a00640 1 -- 172.21.15.12:0/3352079552 learned_addr learned my addr 172.21.15.12:0/3352079552 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.757+0000 7f5a06000640 1 --2- 172.21.15.12:0/3352079552 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a081028b0 0x7f5a0819e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.757+0000 7f5a07400640 1 --2- 172.21.15.12:0/3352079552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a08105260 0x7f5a081a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.757+0000 7f5a06a00640 1 -- 172.21.15.12:0/3352079552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a08105260 msgr2=0x7f5a081a5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.757+0000 7f5a06a00640 1 --2- 172.21.15.12:0/3352079552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a08105260 0x7f5a081a5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.757+0000 7f5a06a00640 1 -- 172.21.15.12:0/3352079552 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a081028b0 msgr2=0x7f5a0819e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.757+0000 7f5a06a00640 1 --2- 172.21.15.12:0/3352079552 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a081028b0 0x7f5a0819e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.757+0000 7f5a06a00640 1 -- 172.21.15.12:0/3352079552 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a081a5af0 con 0x7f5a080fff40 2024-03-23T23:10:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.757+0000 7f5a06000640 1 --2- 172.21.15.12:0/3352079552 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a081028b0 0x7f5a0819e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:10:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.758+0000 7f5a06a00640 1 --2- 172.21.15.12:0/3352079552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a080fff40 0x7f5a0819ded0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5a000325f0 tx=0x7f5a00032ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.758+0000 7f59ff400640 1 -- 172.21.15.12:0/3352079552 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a00041020 con 0x7f5a080fff40 2024-03-23T23:10:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.758+0000 7f59ff400640 1 -- 172.21.15.12:0/3352079552 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a00034cd0 con 0x7f5a080fff40 2024-03-23T23:10:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.758+0000 7f5a0d800640 1 -- 172.21.15.12:0/3352079552 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a081a5d80 con 0x7f5a080fff40 2024-03-23T23:10:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.758+0000 7f59ff400640 1 -- 172.21.15.12:0/3352079552 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a00039040 con 0x7f5a080fff40 2024-03-23T23:10:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.758+0000 7f5a0d800640 1 -- 172.21.15.12:0/3352079552 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a081a61d0 con 0x7f5a080fff40 2024-03-23T23:10:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.762+0000 7f5a1061a740 1 Processor -- start 2024-03-23T23:10:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.763+0000 7f5a1061a740 1 -- start start 2024-03-23T23:10:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.763+0000 7f59ff400640 1 -- 172.21.15.12:0/3352079552 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5a00044020 con 0x7f5a080fff40 2024-03-23T23:10:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.763+0000 7f5a1061a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c81422e1e0 0x55c81422e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.763+0000 7f5a1061a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c81422ebd0 0x55c8142336d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.763+0000 7f5a1061a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c814233d40 0x55c8142361d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.763+0000 7f5a1061a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8141f13b0 con 0x55c81422ebd0 2024-03-23T23:10:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.763+0000 7f5a1061a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c814193830 con 0x55c81422e1e0 2024-03-23T23:10:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.763+0000 7f5a1061a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c81413c5d0 con 0x55c814233d40 2024-03-23T23:10:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.763+0000 7f5a06a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c81422ebd0 0x55c8142336d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.763+0000 7f5a07400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c81422e1e0 0x55c81422e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.763+0000 7f5a07400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c81422e1e0 0x55c81422e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:47202/0 (socket says 172.21.15.12:47202) 2024-03-23T23:10:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.763+0000 7f5a07400640 1 -- 172.21.15.12:0/550363658 learned_addr learned my addr 172.21.15.12:0/550363658 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.763+0000 7f5a06000640 1 --2- 172.21.15.12:0/550363658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c814233d40 0x55c8142361d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.764+0000 7f5a07400640 1 -- 172.21.15.12:0/550363658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c814233d40 msgr2=0x55c8142361d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.764+0000 7f5a07400640 1 --2- 172.21.15.12:0/550363658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c814233d40 0x55c8142361d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.764+0000 7f5a07400640 1 -- 172.21.15.12:0/550363658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c81422ebd0 msgr2=0x55c8142336d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.764+0000 7f5a07400640 1 --2- 172.21.15.12:0/550363658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c81422ebd0 0x55c8142336d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.764+0000 7f5a07400640 1 -- 172.21.15.12:0/550363658 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c8141542c0 con 0x55c81422e1e0 2024-03-23T23:10:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.764+0000 7f5a07400640 1 --2- 172.21.15.12:0/550363658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c81422e1e0 0x55c81422e600 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f59f8009ca0 tx=0x7f59f8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3365a3ebde76a4b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.764+0000 7f59f7e00640 1 -- 172.21.15.12:0/550363658 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f801e070 con 0x55c81422e1e0 2024-03-23T23:10:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.764+0000 7f59f7e00640 1 -- 172.21.15.12:0/550363658 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59f8009e50 con 0x55c81422e1e0 2024-03-23T23:10:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.764+0000 7f59f7e00640 1 -- 172.21.15.12:0/550363658 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f800eac0 con 0x55c81422e1e0 2024-03-23T23:10:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.765+0000 7f5a1061a740 1 -- 172.21.15.12:0/550363658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c81422e1e0 msgr2=0x55c81422e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.765+0000 7f5a1061a740 1 --2- 172.21.15.12:0/550363658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c81422e1e0 0x55c81422e600 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f59f8009ca0 tx=0x7f59f8011810 comp rx=0 tx=0).stop 2024-03-23T23:10:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.765+0000 7f5a1061a740 1 -- 172.21.15.12:0/550363658 shutdown_connections 2024-03-23T23:10:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.765+0000 7f5a1061a740 1 --2- 172.21.15.12:0/550363658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c814233d40 0x55c8142361d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.765+0000 7f5a1061a740 1 --2- 172.21.15.12:0/550363658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c81422ebd0 0x55c8142336d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.765+0000 7f5a1061a740 1 --2- 172.21.15.12:0/550363658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c81422e1e0 0x55c81422e600 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.765+0000 7f5a1061a740 1 -- 172.21.15.12:0/550363658 >> 172.21.15.12:0/550363658 conn(0x55c814229750 msgr2=0x55c814229b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.765+0000 7f5a1061a740 1 -- 172.21.15.12:0/550363658 shutdown_connections 2024-03-23T23:10:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.765+0000 7f5a1061a740 1 -- 172.21.15.12:0/550363658 wait complete. 2024-03-23T23:10:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.765+0000 7f59ff400640 1 --2- 172.21.15.12:0/3352079552 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f59ec0a2830 0x7f59ec0a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.766+0000 7f5a1061a740 1 -- start start 2024-03-23T23:10:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.766+0000 7f59ff400640 1 -- 172.21.15.12:0/3352079552 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5a000f0830 con 0x7f5a080fff40 2024-03-23T23:10:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.766+0000 7f5a07400640 1 --2- 172.21.15.12:0/3352079552 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f59ec0a2830 0x7f59ec0a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.766+0000 7f5a1061a740 1 Processor -- start 2024-03-23T23:10:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.766+0000 7f5a1061a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c81422e1e0 0x55c814238370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.766+0000 7f5a1061a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c81422ebd0 0x55c8142388b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.766+0000 7f5a1061a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c814233d40 0x55c81423f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.767+0000 7f5a1061a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c81414fa20 con 0x55c81422ebd0 2024-03-23T23:10:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.767+0000 7f5a1061a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c81418f8d0 con 0x55c814233d40 2024-03-23T23:10:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.767+0000 7f5a1061a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c814236cf0 con 0x55c81422e1e0 2024-03-23T23:10:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.767+0000 7f5a1061a740 10 client.0 _ll_register_callbacks cb 0x55c81418c1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.767+0000 7f5a07400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c814233d40 0x55c81423f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.767+0000 7f5a06a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c81422e1e0 0x55c814238370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.767+0000 7f5a06000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c81422ebd0 0x55c8142388b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.767+0000 7f5a07400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c814233d40 0x55c81423f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:47210/0 (socket says 172.21.15.12:47210) 2024-03-23T23:10:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.767+0000 7f5a07400640 1 -- 172.21.15.12:0/680845806 learned_addr learned my addr 172.21.15.12:0/680845806 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.767+0000 7f5a07400640 1 --2- 172.21.15.12:0/3352079552 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f59ec0a2830 0x7f59ec0a4cf0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f59f8009ab0 tx=0x7f59f8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.767+0000 7f5a06a00640 1 -- 172.21.15.12:0/680845806 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c814233d40 msgr2=0x55c81423f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.767+0000 7f5a06a00640 1 --2- 172.21.15.12:0/680845806 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c814233d40 0x55c81423f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.767+0000 7f5a06a00640 1 -- 172.21.15.12:0/680845806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c81422ebd0 msgr2=0x55c8142388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.767+0000 7f5a06a00640 1 --2- 172.21.15.12:0/680845806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c81422ebd0 0x55c8142388b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.767+0000 7f5a06a00640 1 -- 172.21.15.12:0/680845806 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c813ea8210 con 0x55c81422e1e0 2024-03-23T23:10:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.768+0000 7f5a06a00640 1 --2- 172.21.15.12:0/680845806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c81422e1e0 0x55c814238370 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5a000ee390 tx=0x7f5a00005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1de93701fc904ce9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.768+0000 7f59f4c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:10:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.768+0000 7f59f4c00640 1 -- 172.21.15.12:0/680845806 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a000bd040 con 0x55c81422e1e0 2024-03-23T23:10:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.768+0000 7f59f4c00640 1 -- 172.21.15.12:0/680845806 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5a00003a70 con 0x55c81422e1e0 2024-03-23T23:10:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.768+0000 7f5a1061a740 1 -- 172.21.15.12:0/680845806 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f59f8009e50 con 0x55c81422e1e0 2024-03-23T23:10:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.768+0000 7f59f4c00640 1 -- 172.21.15.12:0/680845806 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a000bcbd0 con 0x55c81422e1e0 2024-03-23T23:10:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.768+0000 7f5a1061a740 10 client.27595 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.768+0000 7f5a1061a740 1 -- 172.21.15.12:0/680845806 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c813f74620 con 0x55c81422e1e0 2024-03-23T23:10:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.768+0000 7f59f4c00640 1 -- 172.21.15.12:0/680845806 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59f8009e50 con 0x55c81422e1e0 2024-03-23T23:10:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.769+0000 7f59f4c00640 1 -- 172.21.15.12:0/680845806 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f5a000f0ce0 con 0x55c81422e1e0 2024-03-23T23:10:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.769+0000 7f5a1061a740 10 client.27595 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.769+0000 7f5a1061a740 10 client.27595 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.769+0000 7f5a1061a740 20 client.27595 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.769+0000 7f5a1061a740 1 --2- 172.21.15.12:0/680845806 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c814244140 0x55c8142465b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.769+0000 7f5a1061a740 4 client.27595 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:10:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.769+0000 7f5a1061a740 1 -- 172.21.15.12:0/680845806 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c8141a0d70 con 0x55c814244140 2024-03-23T23:10:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.769+0000 7f5a06000640 1 --2- 172.21.15.12:0/680845806 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c814244140 0x55c8142465b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.770+0000 7f59f4c00640 10 client.27595 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.769+0000 7f5a06000640 1 --2- 172.21.15.12:0/680845806 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c814244140 0x55c8142465b0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f120e98c8b0a3196 server_cookie=651bf9ae8adc27e6 in_seq=0 out_seq=0 2024-03-23T23:10:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.770+0000 7f59f4c00640 1 -- 172.21.15.12:0/680845806 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c8141a0d70 con 0x55c814244140 2024-03-23T23:10:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.770+0000 7f59f4c00640 10 client.27595 handle_command_reply: tid=0 2024-03-23T23:10:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.771+0000 7f5a1061a740 1 client.27595 shutdown 2024-03-23T23:10:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.771+0000 7f5a1061a740 1 -- 172.21.15.12:0/680845806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c81422e1e0 msgr2=0x55c814238370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:06.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.771+0000 7f5a1061a740 1 --2- 172.21.15.12:0/680845806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c81422e1e0 0x55c814238370 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5a000ee390 tx=0x7f5a00005ef0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.771+0000 7f5a1061a740 1 -- 172.21.15.12:0/680845806 shutdown_connections 2024-03-23T23:10:06.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.771+0000 7f5a1061a740 1 --2- 172.21.15.12:0/680845806 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c814244140 0x55c8142465b0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.771+0000 7f5a1061a740 1 --2- 172.21.15.12:0/680845806 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c814233d40 0x55c81423f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.771+0000 7f5a1061a740 1 --2- 172.21.15.12:0/680845806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c81422ebd0 0x55c8142388b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.771+0000 7f59f4c00640 0 client.27595 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:06.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.771+0000 7f5a1061a740 1 --2- 172.21.15.12:0/680845806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c81422e1e0 0x55c814238370 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.771+0000 7f5a1061a740 1 -- 172.21.15.12:0/680845806 >> 172.21.15.12:0/680845806 conn(0x55c814229750 msgr2=0x55c814228950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:06.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.771+0000 7f5a1061a740 1 -- 172.21.15.12:0/680845806 shutdown_connections 2024-03-23T23:10:06.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.771+0000 7f5a1061a740 1 -- 172.21.15.12:0/680845806 wait complete. 2024-03-23T23:10:06.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.771+0000 7f5a1061a740 20 client.27595 trim_cache size 0 max 16384 2024-03-23T23:10:06.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.786+0000 7f5a1061a740 1 Processor -- start 2024-03-23T23:10:06.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.786+0000 7f5a1061a740 1 -- start start 2024-03-23T23:10:06.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.787+0000 7f5a1061a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c814229750 0x55c81422c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:06.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.787+0000 7f5a1061a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c81422e1e0 0x55c81423f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:06.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.787+0000 7f5a1061a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c81422ebd0 0x55c814237f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:06.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.787+0000 7f5a1061a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a081b12d0 con 0x55c81422ebd0 2024-03-23T23:10:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.787+0000 7f5a1061a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a081aedd0 con 0x55c81422e1e0 2024-03-23T23:10:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.787+0000 7f5a1061a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a081ace70 con 0x55c814229750 2024-03-23T23:10:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.787+0000 7f5a06a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c814229750 0x55c81422c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.787+0000 7f5a06000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c81422e1e0 0x55c81423f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.787+0000 7f5a06000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c81422e1e0 0x55c81423f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:47216/0 (socket says 172.21.15.12:47216) 2024-03-23T23:10:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.787+0000 7f5a06a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c814229750 0x55c81422c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:39106/0 (socket says 172.21.15.12:39106) 2024-03-23T23:10:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.787+0000 7f5a06000640 1 -- 172.21.15.12:0/159204035 learned_addr learned my addr 172.21.15.12:0/159204035 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.787+0000 7f5a07400640 1 --2- 172.21.15.12:0/159204035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c81422ebd0 0x55c814237f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.787+0000 7f5a06000640 1 -- 172.21.15.12:0/159204035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c814229750 msgr2=0x55c81422c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.787+0000 7f5a06000640 1 --2- 172.21.15.12:0/159204035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c814229750 0x55c81422c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.787+0000 7f5a06000640 1 -- 172.21.15.12:0/159204035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c81422ebd0 msgr2=0x55c814237f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.787+0000 7f5a06000640 1 --2- 172.21.15.12:0/159204035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c81422ebd0 0x55c814237f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.787+0000 7f5a06000640 1 -- 172.21.15.12:0/159204035 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c81422f460 con 0x55c81422e1e0 2024-03-23T23:10:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.788+0000 7f5a06000640 1 --2- 172.21.15.12:0/159204035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c81422e1e0 0x55c81423f880 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f59f0009df0 tx=0x7f59f0021d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b761c5580a4a118 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.788+0000 7f59f7400640 1 -- 172.21.15.12:0/159204035 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f00128a0 con 0x55c81422e1e0 2024-03-23T23:10:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.788+0000 7f59f7400640 1 -- 172.21.15.12:0/159204035 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59f0025030 con 0x55c81422e1e0 2024-03-23T23:10:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.788+0000 7f59f7400640 1 -- 172.21.15.12:0/159204035 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f0004a00 con 0x55c81422e1e0 2024-03-23T23:10:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.788+0000 7f5a1061a740 1 -- 172.21.15.12:0/159204035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c81422e1e0 msgr2=0x55c81423f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.788+0000 7f5a1061a740 1 --2- 172.21.15.12:0/159204035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c81422e1e0 0x55c81423f880 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f59f0009df0 tx=0x7f59f0021d40 comp rx=0 tx=0).stop 2024-03-23T23:10:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.789+0000 7f5a1061a740 1 -- 172.21.15.12:0/159204035 shutdown_connections 2024-03-23T23:10:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.789+0000 7f5a1061a740 1 --2- 172.21.15.12:0/159204035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c81422ebd0 0x55c814237f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.789+0000 7f5a1061a740 1 --2- 172.21.15.12:0/159204035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c81422e1e0 0x55c81423f880 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.789+0000 7f5a1061a740 1 --2- 172.21.15.12:0/159204035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c814229750 0x55c81422c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.789+0000 7f5a1061a740 1 -- 172.21.15.12:0/159204035 >> 172.21.15.12:0/159204035 conn(0x55c814233880 msgr2=0x55c814229b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.789+0000 7f5a1061a740 1 -- 172.21.15.12:0/159204035 shutdown_connections 2024-03-23T23:10:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.789+0000 7f5a1061a740 1 -- 172.21.15.12:0/159204035 wait complete. 2024-03-23T23:10:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.789+0000 7f5a1061a740 1 -- start start 2024-03-23T23:10:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.789+0000 7f5a1061a740 1 Processor -- start 2024-03-23T23:10:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.790+0000 7f5a1061a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c814229750 0x55c81423d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.790+0000 7f5a1061a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c81422e1e0 0x55c81423dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.790+0000 7f5a1061a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c81422ebd0 0x55c81423ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.790+0000 7f5a1061a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a081ad470 con 0x55c81422e1e0 2024-03-23T23:10:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.790+0000 7f5a1061a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a081ae770 con 0x55c814229750 2024-03-23T23:10:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.790+0000 7f5a1061a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a081adab0 con 0x55c81422ebd0 2024-03-23T23:10:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.790+0000 7f5a1061a740 10 client.0 _ll_register_callbacks cb 0x7f5a00003970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.790+0000 7f5a06a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c814229750 0x55c81423d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.790+0000 7f5a07400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c81422ebd0 0x55c81423ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.790+0000 7f5a06a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c814229750 0x55c81423d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:47218/0 (socket says 172.21.15.12:47218) 2024-03-23T23:10:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.790+0000 7f5a06a00640 1 -- 172.21.15.12:0/822266632 learned_addr learned my addr 172.21.15.12:0/822266632 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.790+0000 7f5a06000640 1 --2- 172.21.15.12:0/822266632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c81422e1e0 0x55c81423dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.790+0000 7f5a06a00640 1 -- 172.21.15.12:0/822266632 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c81422ebd0 msgr2=0x55c81423ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.790+0000 7f5a06a00640 1 --2- 172.21.15.12:0/822266632 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c81422ebd0 0x55c81423ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.790+0000 7f5a06a00640 1 -- 172.21.15.12:0/822266632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c81422e1e0 msgr2=0x55c81423dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.790+0000 7f5a06a00640 1 --2- 172.21.15.12:0/822266632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c81422e1e0 0x55c81423dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.790+0000 7f5a06a00640 1 -- 172.21.15.12:0/822266632 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c81423f1e0 con 0x55c814229750 2024-03-23T23:10:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.791+0000 7f5a06a00640 1 --2- 172.21.15.12:0/822266632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c814229750 0x55c81423d6a0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f5a00005e30 tx=0x7f5a00009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d3dfe6aed1f5cf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.791+0000 7f59f4c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:10:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.791+0000 7f59f4c00640 1 -- 172.21.15.12:0/822266632 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a000bcdb0 con 0x55c814229750 2024-03-23T23:10:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.791+0000 7f5a1061a740 1 -- 172.21.15.12:0/822266632 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f59f0025030 con 0x55c814229750 2024-03-23T23:10:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.791+0000 7f59f4c00640 1 -- 172.21.15.12:0/822266632 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5a000c56f0 con 0x55c814229750 2024-03-23T23:10:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.791+0000 7f59f4c00640 1 -- 172.21.15.12:0/822266632 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a000bd040 con 0x55c814229750 2024-03-23T23:10:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.791+0000 7f5a1061a740 10 client.27638 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.791+0000 7f5a1061a740 1 -- 172.21.15.12:0/822266632 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c8142a0150 con 0x55c814229750 2024-03-23T23:10:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.791+0000 7f59f4c00640 1 -- 172.21.15.12:0/822266632 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59f0025030 con 0x55c814229750 2024-03-23T23:10:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.791+0000 7f59f4c00640 1 -- 172.21.15.12:0/822266632 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f5a000d8050 con 0x55c814229750 2024-03-23T23:10:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.791+0000 7f5a1061a740 10 client.27638 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.791+0000 7f5a1061a740 10 client.27638 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.792+0000 7f5a1061a740 20 client.27638 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.792+0000 7f5a1061a740 1 --2- 172.21.15.12:0/822266632 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c814233d40 0x55c8142a2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.792+0000 7f5a1061a740 4 client.27638 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:10:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.792+0000 7f5a1061a740 1 -- 172.21.15.12:0/822266632 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c81419b0c0 con 0x55c814233d40 2024-03-23T23:10:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.792+0000 7f5a06000640 1 --2- 172.21.15.12:0/822266632 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c814233d40 0x55c8142a2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.792+0000 7f5a06000640 1 --2- 172.21.15.12:0/822266632 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c814233d40 0x55c8142a2770 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c10c810133a52a7 server_cookie=aa66a4d148055364 in_seq=0 out_seq=0 2024-03-23T23:10:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.792+0000 7f59f4c00640 10 client.27638 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:06.804 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.805+0000 7f59f4c00640 1 -- 172.21.15.12:0/822266632 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c81419b0c0 con 0x55c814233d40 2024-03-23T23:10:06.804 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.805+0000 7f59f4c00640 10 client.27638 handle_command_reply: tid=0 2024-03-23T23:10:06.804 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.805+0000 7f5a1061a740 1 client.27638 shutdown 2024-03-23T23:10:06.804 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.805+0000 7f5a1061a740 1 -- 172.21.15.12:0/822266632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c814229750 msgr2=0x55c81423d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:06.804 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.805+0000 7f5a1061a740 1 --2- 172.21.15.12:0/822266632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c814229750 0x55c81423d6a0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f5a00005e30 tx=0x7f5a00009220 comp rx=0 tx=0).stop 2024-03-23T23:10:06.804 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.805+0000 7f5a1061a740 1 -- 172.21.15.12:0/822266632 shutdown_connections 2024-03-23T23:10:06.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.805+0000 7f5a1061a740 1 --2- 172.21.15.12:0/822266632 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c814233d40 0x55c8142a2770 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.805+0000 7f5a1061a740 1 --2- 172.21.15.12:0/822266632 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c81422ebd0 0x55c81423ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.805+0000 7f5a1061a740 1 --2- 172.21.15.12:0/822266632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c81422e1e0 0x55c81423dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.805+0000 7f5a1061a740 1 --2- 172.21.15.12:0/822266632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c814229750 0x55c81423d6a0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.805+0000 7f59f4c00640 0 client.27638 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:06.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.805+0000 7f5a1061a740 1 -- 172.21.15.12:0/822266632 >> 172.21.15.12:0/822266632 conn(0x55c814233880 msgr2=0x55c814229b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:06.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.805+0000 7f5a1061a740 1 -- 172.21.15.12:0/822266632 shutdown_connections 2024-03-23T23:10:06.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.805+0000 7f5a1061a740 1 -- 172.21.15.12:0/822266632 wait complete. 2024-03-23T23:10:06.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.806+0000 7f5a1061a740 20 client.27638 trim_cache size 0 max 16384 2024-03-23T23:10:06.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:10:06.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:10:06.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:10:06.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:10:06.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.806+0000 7f5a0d800640 1 -- 172.21.15.12:0/3352079552 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f59ec0a2830 msgr2=0x7f59ec0a4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:06.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.806+0000 7f5a0d800640 1 --2- 172.21.15.12:0/3352079552 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f59ec0a2830 0x7f59ec0a4cf0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f59f8009ab0 tx=0x7f59f8005850 comp rx=0 tx=0).stop 2024-03-23T23:10:06.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.806+0000 7f5a0d800640 1 -- 172.21.15.12:0/3352079552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a080fff40 msgr2=0x7f5a0819ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:06.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.806+0000 7f5a0d800640 1 --2- 172.21.15.12:0/3352079552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a080fff40 0x7f5a0819ded0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5a000325f0 tx=0x7f5a00032ec0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.806+0000 7f5a0d800640 1 -- 172.21.15.12:0/3352079552 shutdown_connections 2024-03-23T23:10:06.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.806+0000 7f5a0d800640 1 --2- 172.21.15.12:0/3352079552 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f59ec0a2830 0x7f59ec0a4cf0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.806+0000 7f5a0d800640 1 --2- 172.21.15.12:0/3352079552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a08105260 0x7f5a081a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.806+0000 7f5a0d800640 1 --2- 172.21.15.12:0/3352079552 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a081028b0 0x7f5a0819e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.806+0000 7f5a0d800640 1 --2- 172.21.15.12:0/3352079552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a080fff40 0x7f5a0819ded0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:06.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.806+0000 7f5a0d800640 1 -- 172.21.15.12:0/3352079552 >> 172.21.15.12:0/3352079552 conn(0x7f5a080fbb60 msgr2=0x7f5a08103240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:06.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.807+0000 7f5a0d800640 1 -- 172.21.15.12:0/3352079552 shutdown_connections 2024-03-23T23:10:06.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:06.807+0000 7f5a0d800640 1 -- 172.21.15.12:0/3352079552 wait complete. 2024-03-23T23:10:07.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:06 smithi012 ceph-mon[31695]: pgmap v639: 97 pgs: 97 active+clean; 1.5 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:10:07.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:06 smithi173 ceph-mon[36116]: pgmap v639: 97 pgs: 97 active+clean; 1.5 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:10:07.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:06 smithi203 ceph-mon[36693]: pgmap v639: 97 pgs: 97 active+clean; 1.5 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:10:09.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:08 smithi012 ceph-mon[31695]: pgmap v640: 97 pgs: 97 active+clean; 1.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:10:09.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:08 smithi173 ceph-mon[36116]: pgmap v640: 97 pgs: 97 active+clean; 1.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:10:09.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:08 smithi203 ceph-mon[36693]: pgmap v640: 97 pgs: 97 active+clean; 1.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:10:11.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:10 smithi012 ceph-mon[31695]: pgmap v641: 97 pgs: 97 active+clean; 1.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:10:11.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:10 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:10:11.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:10 smithi173 ceph-mon[36116]: pgmap v641: 97 pgs: 97 active+clean; 1.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:10:11.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:10 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:10:11.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:10 smithi203 ceph-mon[36693]: pgmap v641: 97 pgs: 97 active+clean; 1.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:10:11.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:10 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:10:12.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:11 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:10:12.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:11 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:10:12.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:11 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:10:12.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:11 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:10:12.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:11 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:10:12.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:11 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:10:12.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:11 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:10:12.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:11 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:10:12.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:11 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:10:13.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:13 smithi012 ceph-mon[31695]: pgmap v642: 97 pgs: 97 active+clean; 1.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:10:13.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:13 smithi173 ceph-mon[36116]: pgmap v642: 97 pgs: 97 active+clean; 1.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:10:13.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:13 smithi203 ceph-mon[36693]: pgmap v642: 97 pgs: 97 active+clean; 1.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:10:15.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:15 smithi012 ceph-mon[31695]: pgmap v643: 97 pgs: 97 active+clean; 1.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:10:15.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:15 smithi173 ceph-mon[36116]: pgmap v643: 97 pgs: 97 active+clean; 1.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:10:15.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:15 smithi203 ceph-mon[36693]: pgmap v643: 97 pgs: 97 active+clean; 1.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:10:16.881 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:10:16.881 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:10:17.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.118+0000 7f3c5cc00640 1 -- 172.21.15.12:0/676806552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c5810c0b0 msgr2=0x7f3c5810c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:17.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.118+0000 7f3c5cc00640 1 --2- 172.21.15.12:0/676806552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c5810c0b0 0x7f3c5810c4d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f3c48009900 tx=0x7f3c4802bc50 comp rx=0 tx=0).stop 2024-03-23T23:10:17.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.119+0000 7f3c5cc00640 1 -- 172.21.15.12:0/676806552 shutdown_connections 2024-03-23T23:10:17.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.119+0000 7f3c5cc00640 1 --2- 172.21.15.12:0/676806552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c5810dc60 0x7f3c581144f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.119+0000 7f3c5cc00640 1 --2- 172.21.15.12:0/676806552 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c5810d2a0 0x7f3c5810d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.119+0000 7f3c5cc00640 1 --2- 172.21.15.12:0/676806552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c5810c0b0 0x7f3c5810c4d0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.119+0000 7f3c5cc00640 1 -- 172.21.15.12:0/676806552 >> 172.21.15.12:0/676806552 conn(0x7f3c580729f0 msgr2=0x7f3c58072e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:17.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.119+0000 7f3c5cc00640 1 -- 172.21.15.12:0/676806552 shutdown_connections 2024-03-23T23:10:17.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.119+0000 7f3c5cc00640 1 -- 172.21.15.12:0/676806552 wait complete. 2024-03-23T23:10:17.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.120+0000 7f3c5cc00640 1 Processor -- start 2024-03-23T23:10:17.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.120+0000 7f3c5cc00640 1 -- start start 2024-03-23T23:10:17.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.120+0000 7f3c5cc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c5810c0b0 0x7f3c581a2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.120+0000 7f3c5cc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c5810d2a0 0x7f3c581a29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.120+0000 7f3c5cc00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c5810dc60 0x7f3c581a9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.120+0000 7f3c5cc00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c58117d90 con 0x7f3c5810dc60 2024-03-23T23:10:17.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.120+0000 7f3c5cc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c58117c10 con 0x7f3c5810c0b0 2024-03-23T23:10:17.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.120+0000 7f3c5cc00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c58117f10 con 0x7f3c5810d2a0 2024-03-23T23:10:17.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.121+0000 7f3c55e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c5810c0b0 0x7f3c581a2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.121+0000 7f3c55e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c5810c0b0 0x7f3c581a2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48392/0 (socket says 172.21.15.12:48392) 2024-03-23T23:10:17.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.121+0000 7f3c4fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c5810d2a0 0x7f3c581a29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.121+0000 7f3c55e00640 1 -- 172.21.15.12:0/2741174551 learned_addr learned my addr 172.21.15.12:0/2741174551 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:17.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.121+0000 7f3c56800640 1 --2- 172.21.15.12:0/2741174551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c5810dc60 0x7f3c581a9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.121+0000 7f3c55e00640 1 -- 172.21.15.12:0/2741174551 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c5810d2a0 msgr2=0x7f3c581a29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:17.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.121+0000 7f3c55e00640 1 --2- 172.21.15.12:0/2741174551 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c5810d2a0 0x7f3c581a29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.121+0000 7f3c55e00640 1 -- 172.21.15.12:0/2741174551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c5810dc60 msgr2=0x7f3c581a9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:17.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.121+0000 7f3c55e00640 1 --2- 172.21.15.12:0/2741174551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c5810dc60 0x7f3c581a9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.121+0000 7f3c55e00640 1 -- 172.21.15.12:0/2741174551 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c581aa0a0 con 0x7f3c5810c0b0 2024-03-23T23:10:17.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.121+0000 7f3c4fe00640 1 --2- 172.21.15.12:0/2741174551 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c5810d2a0 0x7f3c581a29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:17.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.121+0000 7f3c55e00640 1 --2- 172.21.15.12:0/2741174551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c5810c0b0 0x7f3c581a2480 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f3c48005ab0 tx=0x7f3c4802c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:17.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.122+0000 7f3c4ea00640 1 -- 172.21.15.12:0/2741174551 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c48041070 con 0x7f3c5810c0b0 2024-03-23T23:10:17.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.122+0000 7f3c4ea00640 1 -- 172.21.15.12:0/2741174551 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c48004450 con 0x7f3c5810c0b0 2024-03-23T23:10:17.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.122+0000 7f3c4ea00640 1 -- 172.21.15.12:0/2741174551 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c48034440 con 0x7f3c5810c0b0 2024-03-23T23:10:17.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.122+0000 7f3c5cc00640 1 -- 172.21.15.12:0/2741174551 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c581aa330 con 0x7f3c5810c0b0 2024-03-23T23:10:17.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.122+0000 7f3c5cc00640 1 -- 172.21.15.12:0/2741174551 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c581aa830 con 0x7f3c5810c0b0 2024-03-23T23:10:17.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.125+0000 7f3c4ea00640 1 -- 172.21.15.12:0/2741174551 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3c48044070 con 0x7f3c5810c0b0 2024-03-23T23:10:17.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.125+0000 7f3c5fa16740 1 Processor -- start 2024-03-23T23:10:17.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.125+0000 7f3c5fa16740 1 -- start start 2024-03-23T23:10:17.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.126+0000 7f3c5fa16740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d41b0de80 0x561d41b0e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.126+0000 7f3c5fa16740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d41b0e870 0x561d41b13370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.126+0000 7f3c5fa16740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d41b139e0 0x561d41b15e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.126+0000 7f3c5fa16740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d41a73250 con 0x561d41b0e870 2024-03-23T23:10:17.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.126+0000 7f3c5fa16740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d41a1ba30 con 0x561d41b139e0 2024-03-23T23:10:17.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.126+0000 7f3c5fa16740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d41a6f2f0 con 0x561d41b0de80 2024-03-23T23:10:17.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.126+0000 7f3c56800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d41b0de80 0x561d41b0e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.126+0000 7f3c4fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d41b139e0 0x561d41b15e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.126+0000 7f3c55e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d41b0e870 0x561d41b13370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.126+0000 7f3c56800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d41b0de80 0x561d41b0e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:41146/0 (socket says 172.21.15.12:41146) 2024-03-23T23:10:17.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.126+0000 7f3c56800640 1 -- 172.21.15.12:0/3492497824 learned_addr learned my addr 172.21.15.12:0/3492497824 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:17.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.126+0000 7f3c56800640 1 -- 172.21.15.12:0/3492497824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d41b139e0 msgr2=0x561d41b15e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:17.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.126+0000 7f3c56800640 1 --2- 172.21.15.12:0/3492497824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d41b139e0 0x561d41b15e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.126+0000 7f3c56800640 1 -- 172.21.15.12:0/3492497824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d41b0e870 msgr2=0x561d41b13370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:17.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.126+0000 7f3c56800640 1 --2- 172.21.15.12:0/3492497824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d41b0e870 0x561d41b13370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.126+0000 7f3c56800640 1 -- 172.21.15.12:0/3492497824 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d41a06690 con 0x561d41b0de80 2024-03-23T23:10:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.126+0000 7f3c4fe00640 1 --2- 172.21.15.12:0/3492497824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d41b139e0 0x561d41b15e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.127+0000 7f3c56800640 1 --2- 172.21.15.12:0/3492497824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d41b0de80 0x561d41b0e2a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3c50009f60 tx=0x7f3c50009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ee064e5807f5ea2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.127+0000 7f3c4ea00640 1 --2- 172.21.15.12:0/2741174551 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3c3c0a2800 0x7f3c3c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.127+0000 7f3c47000640 1 -- 172.21.15.12:0/3492497824 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5001e070 con 0x561d41b0de80 2024-03-23T23:10:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.127+0000 7f3c47000640 1 -- 172.21.15.12:0/3492497824 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c50004030 con 0x561d41b0de80 2024-03-23T23:10:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.127+0000 7f3c47000640 1 -- 172.21.15.12:0/3492497824 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5000f720 con 0x561d41b0de80 2024-03-23T23:10:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.127+0000 7f3c55e00640 1 --2- 172.21.15.12:0/2741174551 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3c3c0a2800 0x7f3c3c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.127+0000 7f3c4ea00640 1 -- 172.21.15.12:0/2741174551 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f3c480f05b0 con 0x7f3c5810c0b0 2024-03-23T23:10:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.128+0000 7f3c5fa16740 1 -- 172.21.15.12:0/3492497824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d41b0de80 msgr2=0x561d41b0e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.128+0000 7f3c5fa16740 1 --2- 172.21.15.12:0/3492497824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d41b0de80 0x561d41b0e2a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3c50009f60 tx=0x7f3c50009f90 comp rx=0 tx=0).stop 2024-03-23T23:10:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.128+0000 7f3c55e00640 1 --2- 172.21.15.12:0/2741174551 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3c3c0a2800 0x7f3c3c0a4cc0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f3c480b98d0 tx=0x7f3c480f6060 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.128+0000 7f3c5fa16740 1 -- 172.21.15.12:0/3492497824 shutdown_connections 2024-03-23T23:10:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.128+0000 7f3c5fa16740 1 --2- 172.21.15.12:0/3492497824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d41b139e0 0x561d41b15e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.128+0000 7f3c5fa16740 1 --2- 172.21.15.12:0/3492497824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d41b0e870 0x561d41b13370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.128+0000 7f3c5fa16740 1 --2- 172.21.15.12:0/3492497824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d41b0de80 0x561d41b0e2a0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.128+0000 7f3c5fa16740 1 -- 172.21.15.12:0/3492497824 >> 172.21.15.12:0/3492497824 conn(0x561d41ad6200 msgr2=0x561d41b097f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:17.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.128+0000 7f3c5fa16740 1 -- 172.21.15.12:0/3492497824 shutdown_connections 2024-03-23T23:10:17.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.128+0000 7f3c5fa16740 1 -- 172.21.15.12:0/3492497824 wait complete. 2024-03-23T23:10:17.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.128+0000 7f3c5fa16740 1 -- start start 2024-03-23T23:10:17.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.129+0000 7f3c5fa16740 1 Processor -- start 2024-03-23T23:10:17.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.129+0000 7f3c5fa16740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d41b0de80 0x561d41b17e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.129+0000 7f3c5fa16740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d41b0e870 0x561d41b183d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.129+0000 7f3c5fa16740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d41b139e0 0x561d41b1f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.129+0000 7f3c5fa16740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d41b16990 con 0x561d41b0e870 2024-03-23T23:10:17.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.129+0000 7f3c5fa16740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d41a2ee80 con 0x561d41b0de80 2024-03-23T23:10:17.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.129+0000 7f3c5fa16740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d41b16b10 con 0x561d41b139e0 2024-03-23T23:10:17.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.129+0000 7f3c5fa16740 10 client.0 _ll_register_callbacks cb 0x561d41a41130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:17.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.129+0000 7f3c56800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d41b0de80 0x561d41b17e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.129+0000 7f3c56800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d41b0de80 0x561d41b17e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48408/0 (socket says 172.21.15.12:48408) 2024-03-23T23:10:17.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.129+0000 7f3c56800640 1 -- 172.21.15.12:0/3784275687 learned_addr learned my addr 172.21.15.12:0/3784275687 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:17.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.129+0000 7f3c4fe00640 1 --2- 172.21.15.12:0/3784275687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d41b0e870 0x561d41b183d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.129+0000 7f3c56800640 1 --2- 172.21.15.12:0/3784275687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d41b139e0 0x561d41b1f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.130+0000 7f3c56800640 1 -- 172.21.15.12:0/3784275687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d41b139e0 msgr2=0x561d41b1f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:17.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.130+0000 7f3c56800640 1 --2- 172.21.15.12:0/3784275687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d41b139e0 0x561d41b1f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.130+0000 7f3c56800640 1 -- 172.21.15.12:0/3784275687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d41b0e870 msgr2=0x561d41b183d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.130+0000 7f3c56800640 1 --2- 172.21.15.12:0/3784275687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d41b0e870 0x561d41b183d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.130+0000 7f3c56800640 1 -- 172.21.15.12:0/3784275687 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561d418517d0 con 0x561d41b0de80 2024-03-23T23:10:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.130+0000 7f3c56800640 1 --2- 172.21.15.12:0/3784275687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d41b0de80 0x561d41b17e90 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f3c50009f30 tx=0x7f3c50000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87c272d13ef87a36 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.130+0000 7f3c3b400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:10:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.130+0000 7f3c3b400640 1 -- 172.21.15.12:0/3784275687 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5002d060 con 0x561d41b0de80 2024-03-23T23:10:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.130+0000 7f3c3b400640 1 -- 172.21.15.12:0/3784275687 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f3c5000f6d0 con 0x561d41b0de80 2024-03-23T23:10:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.130+0000 7f3c5fa16740 1 -- 172.21.15.12:0/3784275687 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3c50004030 con 0x561d41b0de80 2024-03-23T23:10:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.130+0000 7f3c3b400640 1 -- 172.21.15.12:0/3784275687 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c5001e070 con 0x561d41b0de80 2024-03-23T23:10:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.131+0000 7f3c5fa16740 10 client.27650 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.131+0000 7f3c5fa16740 1 -- 172.21.15.12:0/3784275687 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561d417873f0 con 0x561d41b0de80 2024-03-23T23:10:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.131+0000 7f3c3b400640 1 -- 172.21.15.12:0/3784275687 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c50004030 con 0x561d41b0de80 2024-03-23T23:10:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.131+0000 7f3c3b400640 1 -- 172.21.15.12:0/3784275687 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f3c50019940 con 0x561d41b0de80 2024-03-23T23:10:17.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.131+0000 7f3c5fa16740 10 client.27650 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.131+0000 7f3c5fa16740 10 client.27650 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.131+0000 7f3c5fa16740 20 client.27650 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.131+0000 7f3c5fa16740 1 --2- 172.21.15.12:0/3784275687 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561d41b239b0 0x561d41b25da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.131+0000 7f3c5fa16740 4 client.27650 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:10:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.131+0000 7f3c5fa16740 1 -- 172.21.15.12:0/3784275687 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561d41a80790 con 0x561d41b239b0 2024-03-23T23:10:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.131+0000 7f3c4fe00640 1 --2- 172.21.15.12:0/3784275687 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561d41b239b0 0x561d41b25da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.132+0000 7f3c4fe00640 1 --2- 172.21.15.12:0/3784275687 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561d41b239b0 0x561d41b25da0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df2a29811570b956 server_cookie=9cc555034b8dc9dd in_seq=0 out_seq=0 2024-03-23T23:10:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.132+0000 7f3c3b400640 10 client.27650 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.133+0000 7f3c3b400640 1 -- 172.21.15.12:0/3784275687 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561d41a80790 con 0x561d41b239b0 2024-03-23T23:10:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.133+0000 7f3c3b400640 10 client.27650 handle_command_reply: tid=0 2024-03-23T23:10:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.133+0000 7f3c5fa16740 1 client.27650 shutdown 2024-03-23T23:10:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.134+0000 7f3c5fa16740 1 -- 172.21.15.12:0/3784275687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d41b0de80 msgr2=0x561d41b17e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.134+0000 7f3c5fa16740 1 --2- 172.21.15.12:0/3784275687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d41b0de80 0x561d41b17e90 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f3c50009f30 tx=0x7f3c50000f40 comp rx=0 tx=0).stop 2024-03-23T23:10:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.134+0000 7f3c5fa16740 1 -- 172.21.15.12:0/3784275687 shutdown_connections 2024-03-23T23:10:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.134+0000 7f3c5fa16740 1 --2- 172.21.15.12:0/3784275687 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561d41b239b0 0x561d41b25da0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.134+0000 7f3c5fa16740 1 --2- 172.21.15.12:0/3784275687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d41b139e0 0x561d41b1f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.134+0000 7f3c5fa16740 1 --2- 172.21.15.12:0/3784275687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d41b0e870 0x561d41b183d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.134+0000 7f3c3b400640 0 client.27650 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:17.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.134+0000 7f3c5fa16740 1 --2- 172.21.15.12:0/3784275687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d41b0de80 0x561d41b17e90 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.134+0000 7f3c5fa16740 1 -- 172.21.15.12:0/3784275687 >> 172.21.15.12:0/3784275687 conn(0x561d41ad6200 msgr2=0x561d41b08a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:17.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.134+0000 7f3c5fa16740 1 -- 172.21.15.12:0/3784275687 shutdown_connections 2024-03-23T23:10:17.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.134+0000 7f3c5fa16740 1 -- 172.21.15.12:0/3784275687 wait complete. 2024-03-23T23:10:17.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.134+0000 7f3c5fa16740 20 client.27650 trim_cache size 0 max 16384 2024-03-23T23:10:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.152+0000 7f3c5fa16740 1 Processor -- start 2024-03-23T23:10:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.152+0000 7f3c5fa16740 1 -- start start 2024-03-23T23:10:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.152+0000 7f3c5fa16740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d41b0de80 0x561d41b2a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.152+0000 7f3c5fa16740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d41b0e870 0x561d41b1f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.152+0000 7f3c5fa16740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d41b139e0 0x561d41b2ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.152+0000 7f3c5fa16740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c581b5a40 con 0x561d41b0de80 2024-03-23T23:10:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.152+0000 7f3c5fa16740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c581b3540 con 0x561d41b0e870 2024-03-23T23:10:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.152+0000 7f3c5fa16740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c581b15e0 con 0x561d41b139e0 2024-03-23T23:10:17.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.153+0000 7f3c4fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d41b0e870 0x561d41b1f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.153+0000 7f3c4fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d41b0e870 0x561d41b1f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48414/0 (socket says 172.21.15.12:48414) 2024-03-23T23:10:17.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.153+0000 7f3c4fe00640 1 -- 172.21.15.12:0/2958202150 learned_addr learned my addr 172.21.15.12:0/2958202150 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:17.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.153+0000 7f3c56800640 1 --2- 172.21.15.12:0/2958202150 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d41b0de80 0x561d41b2a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.153+0000 7f3c4fe00640 1 -- 172.21.15.12:0/2958202150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d41b139e0 msgr2=0x561d41b2ab80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-23T23:10:17.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.153+0000 7f3c4fe00640 1 --2- 172.21.15.12:0/2958202150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d41b139e0 0x561d41b2ab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.153+0000 7f3c4fe00640 1 -- 172.21.15.12:0/2958202150 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d41b0de80 msgr2=0x561d41b2a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:17.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.153+0000 7f3c4fe00640 1 --2- 172.21.15.12:0/2958202150 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d41b0de80 0x561d41b2a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.153+0000 7f3c4fe00640 1 -- 172.21.15.12:0/2958202150 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d417470f0 con 0x561d41b0e870 2024-03-23T23:10:17.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.153+0000 7f3c56800640 1 --2- 172.21.15.12:0/2958202150 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d41b0de80 0x561d41b2a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:10:17.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.153+0000 7f3c4fe00640 1 --2- 172.21.15.12:0/2958202150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d41b0e870 0x561d41b1f3a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3c40005d00 tx=0x7f3c4000b410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a051268f5e3303f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:17.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.154+0000 7f3c46600640 1 -- 172.21.15.12:0/2958202150 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c40005020 con 0x561d41b0e870 2024-03-23T23:10:17.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.154+0000 7f3c46600640 1 -- 172.21.15.12:0/2958202150 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c40024030 con 0x561d41b0e870 2024-03-23T23:10:17.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.154+0000 7f3c46600640 1 -- 172.21.15.12:0/2958202150 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4001d650 con 0x561d41b0e870 2024-03-23T23:10:17.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.154+0000 7f3c5fa16740 1 -- 172.21.15.12:0/2958202150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d41b0e870 msgr2=0x561d41b1f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:17.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.154+0000 7f3c5fa16740 1 --2- 172.21.15.12:0/2958202150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d41b0e870 0x561d41b1f3a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3c40005d00 tx=0x7f3c4000b410 comp rx=0 tx=0).stop 2024-03-23T23:10:17.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.154+0000 7f3c5fa16740 1 -- 172.21.15.12:0/2958202150 shutdown_connections 2024-03-23T23:10:17.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.154+0000 7f3c5fa16740 1 --2- 172.21.15.12:0/2958202150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d41b139e0 0x561d41b2ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.154+0000 7f3c5fa16740 1 --2- 172.21.15.12:0/2958202150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d41b0e870 0x561d41b1f3a0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.154+0000 7f3c5fa16740 1 --2- 172.21.15.12:0/2958202150 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d41b0de80 0x561d41b2a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.154+0000 7f3c5fa16740 1 -- 172.21.15.12:0/2958202150 >> 172.21.15.12:0/2958202150 conn(0x561d41ad6200 msgr2=0x561d41b0fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:17.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.154+0000 7f3c5fa16740 1 -- 172.21.15.12:0/2958202150 shutdown_connections 2024-03-23T23:10:17.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.155+0000 7f3c5fa16740 1 -- 172.21.15.12:0/2958202150 wait complete. 2024-03-23T23:10:17.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.155+0000 7f3c5fa16740 1 -- start start 2024-03-23T23:10:17.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.155+0000 7f3c5fa16740 1 Processor -- start 2024-03-23T23:10:17.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.155+0000 7f3c5fa16740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d41b0de80 0x561d41b1c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.156+0000 7f3c5fa16740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d41b0e870 0x561d41b1cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.156+0000 7f3c5fa16740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d41b139e0 0x561d41b1dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.156+0000 7f3c5fa16740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c581b1be0 con 0x561d41b0de80 2024-03-23T23:10:17.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.156+0000 7f3c5fa16740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c581b2ee0 con 0x561d41b139e0 2024-03-23T23:10:17.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.156+0000 7f3c5fa16740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c581b2220 con 0x561d41b0e870 2024-03-23T23:10:17.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.156+0000 7f3c5fa16740 10 client.0 _ll_register_callbacks cb 0x7f3c5001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:17.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.156+0000 7f3c56800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d41b0de80 0x561d41b1c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.156+0000 7f3c56800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d41b0de80 0x561d41b1c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49138/0 (socket says 172.21.15.12:49138) 2024-03-23T23:10:17.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.156+0000 7f3c56800640 1 -- 172.21.15.12:0/1339916913 learned_addr learned my addr 172.21.15.12:0/1339916913 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:17.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.156+0000 7f3c4fe00640 1 --2- 172.21.15.12:0/1339916913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d41b0e870 0x561d41b1cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.156+0000 7f3c56800640 1 --2- 172.21.15.12:0/1339916913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d41b139e0 0x561d41b1dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.156+0000 7f3c56800640 1 -- 172.21.15.12:0/1339916913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d41b0e870 msgr2=0x561d41b1cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:17.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.156+0000 7f3c56800640 1 --2- 172.21.15.12:0/1339916913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d41b0e870 0x561d41b1cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.156+0000 7f3c56800640 1 -- 172.21.15.12:0/1339916913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d41b139e0 msgr2=0x561d41b1dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:17.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.156+0000 7f3c56800640 1 --2- 172.21.15.12:0/1339916913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d41b139e0 0x561d41b1dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.156+0000 7f3c56800640 1 -- 172.21.15.12:0/1339916913 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561d41b1e310 con 0x561d41b0de80 2024-03-23T23:10:17.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.156+0000 7f3c4fe00640 1 --2- 172.21.15.12:0/1339916913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d41b0e870 0x561d41b1cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:17.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.157+0000 7f3c56800640 1 --2- 172.21.15.12:0/1339916913 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d41b0de80 0x561d41b1c7d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3c50003d80 tx=0x7f3c50009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=383b0ed4f9aa058c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:17.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.157+0000 7f3c3b400640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:10:17.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.157+0000 7f3c3b400640 1 -- 172.21.15.12:0/1339916913 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c50037070 con 0x561d41b0de80 2024-03-23T23:10:17.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.157+0000 7f3c3b400640 1 -- 172.21.15.12:0/1339916913 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f3c50004ab0 con 0x561d41b0de80 2024-03-23T23:10:17.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.157+0000 7f3c5fa16740 1 -- 172.21.15.12:0/1339916913 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3c40024030 con 0x561d41b0de80 2024-03-23T23:10:17.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.157+0000 7f3c3b400640 1 -- 172.21.15.12:0/1339916913 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c50044050 con 0x561d41b0de80 2024-03-23T23:10:17.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.157+0000 7f3c5fa16740 10 client.17517 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:17.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.157+0000 7f3c5fa16740 1 -- 172.21.15.12:0/1339916913 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561d41b1f620 con 0x561d41b0de80 2024-03-23T23:10:17.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.157+0000 7f3c3b400640 1 -- 172.21.15.12:0/1339916913 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c40024030 con 0x561d41b0de80 2024-03-23T23:10:17.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.158+0000 7f3c3b400640 1 -- 172.21.15.12:0/1339916913 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f3c5000f6d0 con 0x561d41b0de80 2024-03-23T23:10:17.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.158+0000 7f3c5fa16740 10 client.17517 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:17.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.158+0000 7f3c5fa16740 10 client.17517 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:17.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.158+0000 7f3c5fa16740 20 client.17517 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:17.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.158+0000 7f3c5fa16740 1 --2- 172.21.15.12:0/1339916913 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561d41b7f7e0 0x561d41b81c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.158+0000 7f3c5fa16740 4 client.17517 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:10:17.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.158+0000 7f3c5fa16740 1 -- 172.21.15.12:0/1339916913 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561d41a7aae0 con 0x561d41b7f7e0 2024-03-23T23:10:17.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.158+0000 7f3c4fe00640 1 --2- 172.21.15.12:0/1339916913 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561d41b7f7e0 0x561d41b81c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.158+0000 7f3c4fe00640 1 --2- 172.21.15.12:0/1339916913 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561d41b7f7e0 0x561d41b81c00 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8cba7132161177a server_cookie=96dbbc830cfded27 in_seq=0 out_seq=0 2024-03-23T23:10:17.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.158+0000 7f3c3b400640 10 client.17517 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:17.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.159+0000 7f3c3b400640 1 -- 172.21.15.12:0/1339916913 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x561d41a7aae0 con 0x561d41b7f7e0 2024-03-23T23:10:17.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.159+0000 7f3c3b400640 10 client.17517 handle_command_reply: tid=0 2024-03-23T23:10:17.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.159+0000 7f3c5fa16740 1 client.17517 shutdown 2024-03-23T23:10:17.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.159+0000 7f3c5fa16740 1 -- 172.21.15.12:0/1339916913 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d41b0de80 msgr2=0x561d41b1c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:17.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.159+0000 7f3c5fa16740 1 --2- 172.21.15.12:0/1339916913 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d41b0de80 0x561d41b1c7d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3c50003d80 tx=0x7f3c50009900 comp rx=0 tx=0).stop 2024-03-23T23:10:17.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.159+0000 7f3c5fa16740 1 -- 172.21.15.12:0/1339916913 shutdown_connections 2024-03-23T23:10:17.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.159+0000 7f3c5fa16740 1 --2- 172.21.15.12:0/1339916913 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561d41b7f7e0 0x561d41b81c00 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.159+0000 7f3c5fa16740 1 --2- 172.21.15.12:0/1339916913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561d41b139e0 0x561d41b1dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.159+0000 7f3c5fa16740 1 --2- 172.21.15.12:0/1339916913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561d41b0e870 0x561d41b1cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.159+0000 7f3c5fa16740 1 --2- 172.21.15.12:0/1339916913 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d41b0de80 0x561d41b1c7d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.159+0000 7f3c3b400640 0 client.17517 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:17.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.159+0000 7f3c5fa16740 1 -- 172.21.15.12:0/1339916913 >> 172.21.15.12:0/1339916913 conn(0x561d41ad6200 msgr2=0x561d41b0ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:17.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.159+0000 7f3c5fa16740 1 -- 172.21.15.12:0/1339916913 shutdown_connections 2024-03-23T23:10:17.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.159+0000 7f3c5fa16740 1 -- 172.21.15.12:0/1339916913 wait complete. 2024-03-23T23:10:17.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.159+0000 7f3c5fa16740 20 client.17517 trim_cache size 0 max 16384 2024-03-23T23:10:17.163 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:10:17.163 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-03-23T23:10:17.163 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-03-23T23:10:17.163 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:10:17.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.160+0000 7f3c5cc00640 1 -- 172.21.15.12:0/2741174551 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3c3c0a2800 msgr2=0x7f3c3c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:17.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.160+0000 7f3c5cc00640 1 --2- 172.21.15.12:0/2741174551 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3c3c0a2800 0x7f3c3c0a4cc0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f3c480b98d0 tx=0x7f3c480f6060 comp rx=0 tx=0).stop 2024-03-23T23:10:17.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.160+0000 7f3c5cc00640 1 -- 172.21.15.12:0/2741174551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c5810c0b0 msgr2=0x7f3c581a2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:17.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.160+0000 7f3c5cc00640 1 --2- 172.21.15.12:0/2741174551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c5810c0b0 0x7f3c581a2480 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f3c48005ab0 tx=0x7f3c4802c6e0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.160+0000 7f3c5cc00640 1 -- 172.21.15.12:0/2741174551 shutdown_connections 2024-03-23T23:10:17.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.160+0000 7f3c5cc00640 1 --2- 172.21.15.12:0/2741174551 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3c3c0a2800 0x7f3c3c0a4cc0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.160+0000 7f3c5cc00640 1 --2- 172.21.15.12:0/2741174551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c5810dc60 0x7f3c581a9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.160+0000 7f3c5cc00640 1 --2- 172.21.15.12:0/2741174551 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c5810d2a0 0x7f3c581a29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.160+0000 7f3c5cc00640 1 --2- 172.21.15.12:0/2741174551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c5810c0b0 0x7f3c581a2480 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.160+0000 7f3c5cc00640 1 -- 172.21.15.12:0/2741174551 >> 172.21.15.12:0/2741174551 conn(0x7f3c580729f0 msgr2=0x7f3c581105c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:17.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.160+0000 7f3c5cc00640 1 -- 172.21.15.12:0/2741174551 shutdown_connections 2024-03-23T23:10:17.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.160+0000 7f3c5cc00640 1 -- 172.21.15.12:0/2741174551 wait complete. 2024-03-23T23:10:17.178 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T23:10:17.178 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:10:17.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.396+0000 7f543fe00640 1 -- 172.21.15.12:0/540197169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54401048e0 msgr2=0x7f5440104d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:17.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.396+0000 7f543fe00640 1 --2- 172.21.15.12:0/540197169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54401048e0 0x7f5440104d60 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f542c009860 tx=0x7f542c02bb70 comp rx=0 tx=0).stop 2024-03-23T23:10:17.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.396+0000 7f543fe00640 1 -- 172.21.15.12:0/540197169 shutdown_connections 2024-03-23T23:10:17.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.396+0000 7f543fe00640 1 --2- 172.21.15.12:0/540197169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f54401052a0 0x7f544010bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.396+0000 7f543fe00640 1 --2- 172.21.15.12:0/540197169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54401048e0 0x7f5440104d60 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.396+0000 7f543fe00640 1 --2- 172.21.15.12:0/540197169 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54401036f0 0x7f5440103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.396+0000 7f543fe00640 1 -- 172.21.15.12:0/540197169 >> 172.21.15.12:0/540197169 conn(0x7f54400fec80 msgr2=0x7f54401010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:17.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.396+0000 7f543fe00640 1 -- 172.21.15.12:0/540197169 shutdown_connections 2024-03-23T23:10:17.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.396+0000 7f543fe00640 1 -- 172.21.15.12:0/540197169 wait complete. 2024-03-23T23:10:17.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.397+0000 7f543fe00640 1 Processor -- start 2024-03-23T23:10:17.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.397+0000 7f543fe00640 1 -- start start 2024-03-23T23:10:17.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.398+0000 7f543fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54401036f0 0x7f5440199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.398+0000 7f543fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f54401048e0 0x7f5440199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.398+0000 7f543fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54401052a0 0x7f54401a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.398+0000 7f543fe00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544010f350 con 0x7f54401036f0 2024-03-23T23:10:17.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.398+0000 7f543fe00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544010f1d0 con 0x7f54401048e0 2024-03-23T23:10:17.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.398+0000 7f543fe00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544010f4d0 con 0x7f54401052a0 2024-03-23T23:10:17.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.398+0000 7f543e000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f54401048e0 0x7f5440199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.398+0000 7f543e000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f54401048e0 0x7f5440199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48432/0 (socket says 172.21.15.12:48432) 2024-03-23T23:10:17.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.398+0000 7f543ea00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54401036f0 0x7f5440199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.398+0000 7f543e000640 1 -- 172.21.15.12:0/2428035030 learned_addr learned my addr 172.21.15.12:0/2428035030 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:17.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.398+0000 7f543f400640 1 --2- 172.21.15.12:0/2428035030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54401052a0 0x7f54401a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.398+0000 7f543e000640 1 -- 172.21.15.12:0/2428035030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54401052a0 msgr2=0x7f54401a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:17.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.398+0000 7f543e000640 1 --2- 172.21.15.12:0/2428035030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54401052a0 0x7f54401a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.398+0000 7f543e000640 1 -- 172.21.15.12:0/2428035030 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54401036f0 msgr2=0x7f5440199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:17.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.398+0000 7f543e000640 1 --2- 172.21.15.12:0/2428035030 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54401036f0 0x7f5440199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.398+0000 7f543e000640 1 -- 172.21.15.12:0/2428035030 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54401a15f0 con 0x7f54401048e0 2024-03-23T23:10:17.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.399+0000 7f543e000640 1 --2- 172.21.15.12:0/2428035030 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f54401048e0 0x7f5440199f60 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f542c005940 tx=0x7f542c032470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:17.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.399+0000 7f5444a00640 1 -- 172.21.15.12:0/2428035030 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542c041020 con 0x7f54401048e0 2024-03-23T23:10:17.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.399+0000 7f5444a00640 1 -- 172.21.15.12:0/2428035030 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f542c02c9f0 con 0x7f54401048e0 2024-03-23T23:10:17.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.399+0000 7f5444a00640 1 -- 172.21.15.12:0/2428035030 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542c039040 con 0x7f54401048e0 2024-03-23T23:10:17.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.399+0000 7f543fe00640 1 -- 172.21.15.12:0/2428035030 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54401a1880 con 0x7f54401048e0 2024-03-23T23:10:17.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.399+0000 7f543fe00640 1 -- 172.21.15.12:0/2428035030 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54401a1d20 con 0x7f54401048e0 2024-03-23T23:10:17.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.400+0000 7f543fe00640 1 -- 172.21.15.12:0/2428035030 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f544010aa10 con 0x7f54401048e0 2024-03-23T23:10:17.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.402+0000 7f5444a00640 1 -- 172.21.15.12:0/2428035030 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f542c044020 con 0x7f54401048e0 2024-03-23T23:10:17.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.403+0000 7f5444a00640 1 --2- 172.21.15.12:0/2428035030 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f54240a2880 0x7f54240a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.403+0000 7f5444a00640 1 -- 172.21.15.12:0/2428035030 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f542c0f0380 con 0x7f54401048e0 2024-03-23T23:10:17.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.403+0000 7f543ea00640 1 --2- 172.21.15.12:0/2428035030 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f54240a2880 0x7f54240a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.404+0000 7f543ea00640 1 --2- 172.21.15.12:0/2428035030 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f54240a2880 0x7f54240a4d40 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5440104740 tx=0x7f5430005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:17.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.407+0000 7f5444a00640 1 -- 172.21.15.12:0/2428035030 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f542c04b020 con 0x7f54401048e0 2024-03-23T23:10:17.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:17 smithi012 ceph-mon[31695]: pgmap v644: 97 pgs: 97 active+clean; 1.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:10:17.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:17 smithi173 ceph-mon[36116]: pgmap v644: 97 pgs: 97 active+clean; 1.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:10:17.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:17 smithi203 ceph-mon[36693]: pgmap v644: 97 pgs: 97 active+clean; 1.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:10:17.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.659+0000 7f543fe00640 1 -- 172.21.15.12:0/2428035030 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54400622a0 con 0x7f54401048e0 2024-03-23T23:10:17.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.660+0000 7f5444a00640 1 -- 172.21.15.12:0/2428035030 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f542c0bd3d0 con 0x7f54401048e0 2024-03-23T23:10:17.660 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:10:17.660 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:10:17.660 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:10:17.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.662+0000 7f543fe00640 1 -- 172.21.15.12:0/2428035030 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f54240a2880 msgr2=0x7f54240a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:17.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.662+0000 7f543fe00640 1 --2- 172.21.15.12:0/2428035030 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f54240a2880 0x7f54240a4d40 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5440104740 tx=0x7f5430005f90 comp rx=0 tx=0).stop 2024-03-23T23:10:17.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.662+0000 7f543fe00640 1 -- 172.21.15.12:0/2428035030 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f54401048e0 msgr2=0x7f5440199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:17.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.662+0000 7f543fe00640 1 --2- 172.21.15.12:0/2428035030 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f54401048e0 0x7f5440199f60 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f542c005940 tx=0x7f542c032470 comp rx=0 tx=0).stop 2024-03-23T23:10:17.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.663+0000 7f543fe00640 1 -- 172.21.15.12:0/2428035030 shutdown_connections 2024-03-23T23:10:17.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.663+0000 7f543fe00640 1 --2- 172.21.15.12:0/2428035030 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f54240a2880 0x7f54240a4d40 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.663+0000 7f543fe00640 1 --2- 172.21.15.12:0/2428035030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54401052a0 0x7f54401a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.663+0000 7f543fe00640 1 --2- 172.21.15.12:0/2428035030 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f54401048e0 0x7f5440199f60 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.663+0000 7f543fe00640 1 --2- 172.21.15.12:0/2428035030 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54401036f0 0x7f5440199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.663+0000 7f543fe00640 1 -- 172.21.15.12:0/2428035030 >> 172.21.15.12:0/2428035030 conn(0x7f54400fec80 msgr2=0x7f5440107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:17.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.663+0000 7f543fe00640 1 -- 172.21.15.12:0/2428035030 shutdown_connections 2024-03-23T23:10:17.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.663+0000 7f543fe00640 1 -- 172.21.15.12:0/2428035030 wait complete. 2024-03-23T23:10:17.684 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:10:17.684 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-23T23:10:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.926+0000 7f4641c00640 1 -- 172.21.15.12:0/2050188105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f463c104ab0 msgr2=0x7f463c104f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.926+0000 7f4641c00640 1 --2- 172.21.15.12:0/2050188105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f463c104ab0 0x7f463c104f30 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f4624009900 tx=0x7f462402bc30 comp rx=0 tx=0).stop 2024-03-23T23:10:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.926+0000 7f4641c00640 1 -- 172.21.15.12:0/2050188105 shutdown_connections 2024-03-23T23:10:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.926+0000 7f4641c00640 1 --2- 172.21.15.12:0/2050188105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f463c105470 0x7f463c10bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.926+0000 7f4641c00640 1 --2- 172.21.15.12:0/2050188105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f463c104ab0 0x7f463c104f30 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.926+0000 7f4641c00640 1 --2- 172.21.15.12:0/2050188105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f463c1038c0 0x7f463c103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.926+0000 7f4641c00640 1 -- 172.21.15.12:0/2050188105 >> 172.21.15.12:0/2050188105 conn(0x7f463c0fee50 msgr2=0x7f463c101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.926+0000 7f4641c00640 1 -- 172.21.15.12:0/2050188105 shutdown_connections 2024-03-23T23:10:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.926+0000 7f4641c00640 1 -- 172.21.15.12:0/2050188105 wait complete. 2024-03-23T23:10:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.927+0000 7f4641c00640 1 Processor -- start 2024-03-23T23:10:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.927+0000 7f4641c00640 1 -- start start 2024-03-23T23:10:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.928+0000 7f4641c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f463c1038c0 0x7f463c199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.928+0000 7f4641c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f463c104ab0 0x7f463c19a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.928+0000 7f4641c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f463c105470 0x7f463c1a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.928+0000 7f4641c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c10f590 con 0x7f463c104ab0 2024-03-23T23:10:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.928+0000 7f4641c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c10f410 con 0x7f463c1038c0 2024-03-23T23:10:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.928+0000 7f4641c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c10f710 con 0x7f463c105470 2024-03-23T23:10:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.928+0000 7f4632000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f463c104ab0 0x7f463c19a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.928+0000 7f463aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f463c1038c0 0x7f463c199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.928+0000 7f463aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f463c1038c0 0x7f463c199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48440/0 (socket says 172.21.15.12:48440) 2024-03-23T23:10:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.928+0000 7f463aa00640 1 -- 172.21.15.12:0/2325502305 learned_addr learned my addr 172.21.15.12:0/2325502305 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.928+0000 7f463b400640 1 --2- 172.21.15.12:0/2325502305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f463c105470 0x7f463c1a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.929+0000 7f463aa00640 1 -- 172.21.15.12:0/2325502305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f463c105470 msgr2=0x7f463c1a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.929+0000 7f463aa00640 1 --2- 172.21.15.12:0/2325502305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f463c105470 0x7f463c1a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.929+0000 7f463aa00640 1 -- 172.21.15.12:0/2325502305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f463c104ab0 msgr2=0x7f463c19a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.929+0000 7f463aa00640 1 --2- 172.21.15.12:0/2325502305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f463c104ab0 0x7f463c19a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.929+0000 7f463aa00640 1 -- 172.21.15.12:0/2325502305 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463c1a17c0 con 0x7f463c1038c0 2024-03-23T23:10:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.929+0000 7f463b400640 1 --2- 172.21.15.12:0/2325502305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f463c105470 0x7f463c1a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.929+0000 7f4632000640 1 --2- 172.21.15.12:0/2325502305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f463c104ab0 0x7f463c19a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.929+0000 7f463aa00640 1 --2- 172.21.15.12:0/2325502305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f463c1038c0 0x7f463c199ba0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f462c00e9c0 tx=0x7f462c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.929+0000 7f4633e00640 1 -- 172.21.15.12:0/2325502305 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462c00cde0 con 0x7f463c1038c0 2024-03-23T23:10:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.929+0000 7f4633e00640 1 -- 172.21.15.12:0/2325502305 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f462c010910 con 0x7f463c1038c0 2024-03-23T23:10:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.929+0000 7f4641c00640 1 -- 172.21.15.12:0/2325502305 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f463c1a1ab0 con 0x7f463c1038c0 2024-03-23T23:10:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.929+0000 7f4633e00640 1 -- 172.21.15.12:0/2325502305 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462c018a00 con 0x7f463c1038c0 2024-03-23T23:10:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.929+0000 7f4641c00640 1 -- 172.21.15.12:0/2325502305 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f463c1a1fd0 con 0x7f463c1038c0 2024-03-23T23:10:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.932+0000 7f46449cc740 1 Processor -- start 2024-03-23T23:10:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.933+0000 7f46449cc740 1 -- start start 2024-03-23T23:10:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.933+0000 7f4633e00640 1 -- 172.21.15.12:0/2325502305 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f462c01f020 con 0x7f463c1038c0 2024-03-23T23:10:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.933+0000 7f46449cc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f3119b1810 0x55f3119b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.933+0000 7f46449cc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b29e0 0x55f3119b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.933+0000 7f46449cc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3119b33a0 0x55f3119b9c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.933+0000 7f46449cc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3119162c0 con 0x55f3119b1810 2024-03-23T23:10:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.933+0000 7f46449cc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3118bec90 con 0x55f3119b29e0 2024-03-23T23:10:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.933+0000 7f46449cc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f311912360 con 0x55f3119b33a0 2024-03-23T23:10:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.933+0000 7f463aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b29e0 0x55f3119b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.933+0000 7f463aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b29e0 0x55f3119b2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48452/0 (socket says 172.21.15.12:48452) 2024-03-23T23:10:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.933+0000 7f463aa00640 1 -- 172.21.15.12:0/4246924551 learned_addr learned my addr 172.21.15.12:0/4246924551 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.933+0000 7f4632000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3119b33a0 0x55f3119b9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.933+0000 7f463b400640 1 --2- 172.21.15.12:0/4246924551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f3119b1810 0x55f3119b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.934+0000 7f463aa00640 1 -- 172.21.15.12:0/4246924551 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3119b33a0 msgr2=0x55f3119b9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.934+0000 7f463aa00640 1 --2- 172.21.15.12:0/4246924551 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3119b33a0 0x55f3119b9c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.934+0000 7f463aa00640 1 -- 172.21.15.12:0/4246924551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f3119b1810 msgr2=0x55f3119b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.934+0000 7f463aa00640 1 --2- 172.21.15.12:0/4246924551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f3119b1810 0x55f3119b1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.934+0000 7f463aa00640 1 -- 172.21.15.12:0/4246924551 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f3118e3080 con 0x55f3119b29e0 2024-03-23T23:10:17.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.934+0000 7f463aa00640 1 --2- 172.21.15.12:0/4246924551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b29e0 0x55f3119b2e60 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f462c0d2dc0 tx=0x7f462c0d2eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ece285bc6401607 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:17.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.934+0000 7f462be00640 1 -- 172.21.15.12:0/4246924551 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462c0d7e20 con 0x55f3119b29e0 2024-03-23T23:10:17.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.934+0000 7f462be00640 1 -- 172.21.15.12:0/4246924551 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f462c0d98d0 con 0x55f3119b29e0 2024-03-23T23:10:17.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.934+0000 7f462be00640 1 -- 172.21.15.12:0/4246924551 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462c0d59a0 con 0x55f3119b29e0 2024-03-23T23:10:17.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.935+0000 7f4633e00640 1 --2- 172.21.15.12:0/2325502305 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f46200a2800 0x7f46200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.935+0000 7f4633e00640 1 -- 172.21.15.12:0/2325502305 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f462c0cec90 con 0x7f463c1038c0 2024-03-23T23:10:17.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.935+0000 7f463b400640 1 --2- 172.21.15.12:0/2325502305 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f46200a2800 0x7f46200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.935+0000 7f46449cc740 1 -- 172.21.15.12:0/4246924551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b29e0 msgr2=0x55f3119b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:17.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.935+0000 7f46449cc740 1 --2- 172.21.15.12:0/4246924551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b29e0 0x55f3119b2e60 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f462c0d2dc0 tx=0x7f462c0d2eb0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.936+0000 7f46449cc740 1 -- 172.21.15.12:0/4246924551 shutdown_connections 2024-03-23T23:10:17.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.936+0000 7f46449cc740 1 --2- 172.21.15.12:0/4246924551 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3119b33a0 0x55f3119b9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.936+0000 7f46449cc740 1 --2- 172.21.15.12:0/4246924551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b29e0 0x55f3119b2e60 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.936+0000 7f46449cc740 1 --2- 172.21.15.12:0/4246924551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f3119b1810 0x55f3119b1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.936+0000 7f46449cc740 1 -- 172.21.15.12:0/4246924551 >> 172.21.15.12:0/4246924551 conn(0x55f3119acc00 msgr2=0x55f3119ad000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:17.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.936+0000 7f46449cc740 1 -- 172.21.15.12:0/4246924551 shutdown_connections 2024-03-23T23:10:17.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.936+0000 7f463b400640 1 --2- 172.21.15.12:0/2325502305 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f46200a2800 0x7f46200a4cc0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x55f3119b28e0 tx=0x7f4634009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:17.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.936+0000 7f46449cc740 1 -- 172.21.15.12:0/4246924551 wait complete. 2024-03-23T23:10:17.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.936+0000 7f46449cc740 1 -- start start 2024-03-23T23:10:17.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.936+0000 7f46449cc740 1 Processor -- start 2024-03-23T23:10:17.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.937+0000 7f46449cc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b1810 0x55f3119bba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.937+0000 7f46449cc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f3119b29e0 0x55f3119bbf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.937+0000 7f46449cc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3119b33a0 0x55f3119c2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.937+0000 7f46449cc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3119ba3b0 con 0x55f3119b29e0 2024-03-23T23:10:17.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.937+0000 7f46449cc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3118d20d0 con 0x55f3119b1810 2024-03-23T23:10:17.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.937+0000 7f46449cc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3119ba530 con 0x55f3119b33a0 2024-03-23T23:10:17.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.937+0000 7f46449cc740 10 client.0 _ll_register_callbacks cb 0x55f3118d6e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:17.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.937+0000 7f463aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b1810 0x55f3119bba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.937+0000 7f4632000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f3119b29e0 0x55f3119bbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.937+0000 7f463aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b1810 0x55f3119bba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48458/0 (socket says 172.21.15.12:48458) 2024-03-23T23:10:17.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.937+0000 7f463aa00640 1 -- 172.21.15.12:0/1447254803 learned_addr learned my addr 172.21.15.12:0/1447254803 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:17.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.937+0000 7f463b400640 1 --2- 172.21.15.12:0/1447254803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3119b33a0 0x55f3119c2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.938+0000 7f463aa00640 1 -- 172.21.15.12:0/1447254803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3119b33a0 msgr2=0x55f3119c2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:17.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.938+0000 7f463aa00640 1 --2- 172.21.15.12:0/1447254803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3119b33a0 0x55f3119c2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.938+0000 7f463aa00640 1 -- 172.21.15.12:0/1447254803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f3119b29e0 msgr2=0x55f3119bbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:17.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.938+0000 7f463aa00640 1 --2- 172.21.15.12:0/1447254803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f3119b29e0 0x55f3119bbf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.938+0000 7f463aa00640 1 -- 172.21.15.12:0/1447254803 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f31162a3f0 con 0x55f3119b1810 2024-03-23T23:10:17.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.938+0000 7f463aa00640 1 --2- 172.21.15.12:0/1447254803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b1810 0x55f3119bba30 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f462c097fb0 tx=0x7f462c005b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=958ce159d20c96f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:17.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.938+0000 7f4628c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:10:17.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.938+0000 7f4628c00640 1 -- 172.21.15.12:0/1447254803 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462c097880 con 0x55f3119b1810 2024-03-23T23:10:17.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.938+0000 7f4628c00640 1 -- 172.21.15.12:0/1447254803 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f462c0d28d0 con 0x55f3119b1810 2024-03-23T23:10:17.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.938+0000 7f46449cc740 1 -- 172.21.15.12:0/1447254803 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f462c0d98d0 con 0x55f3119b1810 2024-03-23T23:10:17.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.938+0000 7f4628c00640 1 -- 172.21.15.12:0/1447254803 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f462c09af40 con 0x55f3119b1810 2024-03-23T23:10:17.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.938+0000 7f46449cc740 10 client.27674 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:17.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.938+0000 7f46449cc740 1 -- 172.21.15.12:0/1447254803 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f3116f47d0 con 0x55f3119b1810 2024-03-23T23:10:17.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.939+0000 7f4628c00640 1 -- 172.21.15.12:0/1447254803 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f462c0d98d0 con 0x55f3119b1810 2024-03-23T23:10:17.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.939+0000 7f4628c00640 1 -- 172.21.15.12:0/1447254803 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f462c0d5070 con 0x55f3119b1810 2024-03-23T23:10:17.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.939+0000 7f46449cc740 10 client.27674 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:17.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.939+0000 7f46449cc740 10 client.27674 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:10:17.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.939+0000 7f46449cc740 20 client.27674 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:17.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.939+0000 7f46449cc740 1 --2- 172.21.15.12:0/1447254803 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55f3119c7110 0x55f3119c9500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.939+0000 7f46449cc740 4 client.27674 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:10:17.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.939+0000 7f46449cc740 1 -- 172.21.15.12:0/1447254803 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f311923800 con 0x55f3119c7110 2024-03-23T23:10:17.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.939+0000 7f4632000640 1 --2- 172.21.15.12:0/1447254803 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55f3119c7110 0x55f3119c9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.940+0000 7f4632000640 1 --2- 172.21.15.12:0/1447254803 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55f3119c7110 0x55f3119c9500 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7e7c8cb1cc018b71 server_cookie=fb04dd6c8cba5a7b in_seq=0 out_seq=0 2024-03-23T23:10:17.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.940+0000 7f4628c00640 10 client.27674 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:10:17.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.940+0000 7f4628c00640 1 -- 172.21.15.12:0/1447254803 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f311923800 con 0x55f3119c7110 2024-03-23T23:10:17.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.941+0000 7f4628c00640 10 client.27674 handle_command_reply: tid=0 2024-03-23T23:10:17.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.941+0000 7f46449cc740 1 client.27674 shutdown 2024-03-23T23:10:17.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.941+0000 7f46449cc740 1 -- 172.21.15.12:0/1447254803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b1810 msgr2=0x55f3119bba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:17.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.941+0000 7f46449cc740 1 --2- 172.21.15.12:0/1447254803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b1810 0x55f3119bba30 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f462c097fb0 tx=0x7f462c005b20 comp rx=0 tx=0).stop 2024-03-23T23:10:17.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.941+0000 7f46449cc740 1 -- 172.21.15.12:0/1447254803 shutdown_connections 2024-03-23T23:10:17.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.941+0000 7f46449cc740 1 --2- 172.21.15.12:0/1447254803 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55f3119c7110 0x55f3119c9500 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.941+0000 7f46449cc740 1 --2- 172.21.15.12:0/1447254803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3119b33a0 0x55f3119c2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.941+0000 7f46449cc740 1 --2- 172.21.15.12:0/1447254803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f3119b29e0 0x55f3119bbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.941+0000 7f46449cc740 1 --2- 172.21.15.12:0/1447254803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b1810 0x55f3119bba30 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.941+0000 7f4628c00640 0 client.27674 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:10:17.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.941+0000 7f46449cc740 1 -- 172.21.15.12:0/1447254803 >> 172.21.15.12:0/1447254803 conn(0x55f3119acc00 msgr2=0x55f3119abeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:17.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.941+0000 7f46449cc740 1 -- 172.21.15.12:0/1447254803 shutdown_connections 2024-03-23T23:10:17.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.941+0000 7f46449cc740 1 -- 172.21.15.12:0/1447254803 wait complete. 2024-03-23T23:10:17.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.941+0000 7f46449cc740 20 client.27674 trim_cache size 0 max 16384 2024-03-23T23:10:17.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.955+0000 7f46449cc740 1 Processor -- start 2024-03-23T23:10:17.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.955+0000 7f46449cc740 1 -- start start 2024-03-23T23:10:17.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.955+0000 7f46449cc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b1810 0x55f3119b59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.955+0000 7f46449cc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3119b29e0 0x55f3119c2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.955+0000 7f46449cc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f3119b33a0 0x55f3119b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.955+0000 7f46449cc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c1ad210 con 0x55f3119b33a0 2024-03-23T23:10:17.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.955+0000 7f46449cc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c1aad10 con 0x55f3119b1810 2024-03-23T23:10:17.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.955+0000 7f46449cc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c1a8db0 con 0x55f3119b29e0 2024-03-23T23:10:17.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.956+0000 7f463aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b1810 0x55f3119b59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.956+0000 7f463aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b1810 0x55f3119b59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48466/0 (socket says 172.21.15.12:48466) 2024-03-23T23:10:17.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.956+0000 7f463aa00640 1 -- 172.21.15.12:0/1455172492 learned_addr learned my addr 172.21.15.12:0/1455172492 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:17.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.956+0000 7f4632000640 1 --2- 172.21.15.12:0/1455172492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3119b29e0 0x55f3119c2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.956+0000 7f463b400640 1 --2- 172.21.15.12:0/1455172492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f3119b33a0 0x55f3119b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.956+0000 7f463aa00640 1 -- 172.21.15.12:0/1455172492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3119b29e0 msgr2=0x55f3119c2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:17.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.956+0000 7f463aa00640 1 --2- 172.21.15.12:0/1455172492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3119b29e0 0x55f3119c2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.956+0000 7f463aa00640 1 -- 172.21.15.12:0/1455172492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f3119b33a0 msgr2=0x55f3119b5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:17.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.956+0000 7f463aa00640 1 --2- 172.21.15.12:0/1455172492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f3119b33a0 0x55f3119b5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.956+0000 7f463aa00640 1 -- 172.21.15.12:0/1455172492 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f3119a7d40 con 0x55f3119b1810 2024-03-23T23:10:17.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.956+0000 7f463b400640 1 --2- 172.21.15.12:0/1455172492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f3119b33a0 0x55f3119b5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:10:17.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.956+0000 7f463aa00640 1 --2- 172.21.15.12:0/1455172492 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b1810 0x55f3119b59f0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f462c000c00 tx=0x7f462c099af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0e98389a1d32b7f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:17.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.956+0000 7f462b400640 1 -- 172.21.15.12:0/1455172492 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462c09b120 con 0x55f3119b1810 2024-03-23T23:10:17.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.956+0000 7f462b400640 1 -- 172.21.15.12:0/1455172492 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f462c09bce0 con 0x55f3119b1810 2024-03-23T23:10:17.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.956+0000 7f462b400640 1 -- 172.21.15.12:0/1455172492 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462c09b3b0 con 0x55f3119b1810 2024-03-23T23:10:17.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.957+0000 7f46449cc740 1 -- 172.21.15.12:0/1455172492 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b1810 msgr2=0x55f3119b59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:17.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.957+0000 7f46449cc740 1 --2- 172.21.15.12:0/1455172492 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b1810 0x55f3119b59f0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f462c000c00 tx=0x7f462c099af0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.957+0000 7f46449cc740 1 -- 172.21.15.12:0/1455172492 shutdown_connections 2024-03-23T23:10:17.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.957+0000 7f46449cc740 1 --2- 172.21.15.12:0/1455172492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f3119b33a0 0x55f3119b5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.957+0000 7f46449cc740 1 --2- 172.21.15.12:0/1455172492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3119b29e0 0x55f3119c2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.957+0000 7f46449cc740 1 --2- 172.21.15.12:0/1455172492 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b1810 0x55f3119b59f0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.957+0000 7f46449cc740 1 -- 172.21.15.12:0/1455172492 >> 172.21.15.12:0/1455172492 conn(0x55f3119acc00 msgr2=0x55f3119b06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:17.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.957+0000 7f46449cc740 1 -- 172.21.15.12:0/1455172492 shutdown_connections 2024-03-23T23:10:17.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.957+0000 7f46449cc740 1 -- 172.21.15.12:0/1455172492 wait complete. 2024-03-23T23:10:17.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.957+0000 7f46449cc740 1 -- start start 2024-03-23T23:10:17.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.958+0000 7f46449cc740 1 Processor -- start 2024-03-23T23:10:17.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.958+0000 7f46449cc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f3119b1810 0x55f3119c2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.958+0000 7f46449cc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b29e0 0x55f3119c2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.958+0000 7f46449cc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3119b33a0 0x55f3119bf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.958+0000 7f46449cc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c1a93b0 con 0x55f3119b1810 2024-03-23T23:10:17.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.958+0000 7f46449cc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c1aa6b0 con 0x55f3119b29e0 2024-03-23T23:10:17.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.958+0000 7f46449cc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c1a99f0 con 0x55f3119b33a0 2024-03-23T23:10:17.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.958+0000 7f46449cc740 10 client.0 _ll_register_callbacks cb 0x7f462c0d0030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:17.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.958+0000 7f463aa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f3119b1810 0x55f3119c2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.958+0000 7f4632000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b29e0 0x55f3119c2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.958+0000 7f463b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3119b33a0 0x55f3119bf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.958+0000 7f463b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3119b33a0 0x55f3119bf670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:41244/0 (socket says 172.21.15.12:41244) 2024-03-23T23:10:17.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.958+0000 7f4632000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b29e0 0x55f3119c2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48478/0 (socket says 172.21.15.12:48478) 2024-03-23T23:10:17.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.958+0000 7f463b400640 1 -- 172.21.15.12:0/2562698630 learned_addr learned my addr 172.21.15.12:0/2562698630 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:17.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.959+0000 7f463b400640 1 -- 172.21.15.12:0/2562698630 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b29e0 msgr2=0x55f3119c2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:17.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.959+0000 7f463b400640 1 --2- 172.21.15.12:0/2562698630 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b29e0 0x55f3119c2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.959+0000 7f463b400640 1 -- 172.21.15.12:0/2562698630 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f3119b1810 msgr2=0x55f3119c2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:17.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.959+0000 7f463b400640 1 --2- 172.21.15.12:0/2562698630 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f3119b1810 0x55f3119c2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.959+0000 7f463b400640 1 -- 172.21.15.12:0/2562698630 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f3119bfea0 con 0x55f3119b33a0 2024-03-23T23:10:17.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.959+0000 7f463aa00640 1 --2- 172.21.15.12:0/2562698630 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f3119b1810 0x55f3119c2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:17.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.959+0000 7f463b400640 1 --2- 172.21.15.12:0/2562698630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3119b33a0 0x55f3119bf670 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f463400fb50 tx=0x7f4634004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=457a9ad4d2acccca server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:17.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.959+0000 7f4628c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:10:17.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.959+0000 7f4628c00640 1 -- 172.21.15.12:0/2562698630 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463402e060 con 0x55f3119b33a0 2024-03-23T23:10:17.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.959+0000 7f46449cc740 1 -- 172.21.15.12:0/2562698630 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f462c09bce0 con 0x55f3119b33a0 2024-03-23T23:10:17.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.959+0000 7f4628c00640 1 -- 172.21.15.12:0/2562698630 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f46340119c0 con 0x55f3119b33a0 2024-03-23T23:10:17.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.960+0000 7f4628c00640 1 -- 172.21.15.12:0/2562698630 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4634011bc0 con 0x55f3119b33a0 2024-03-23T23:10:17.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.960+0000 7f46449cc740 10 client.27640 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:17.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.960+0000 7f46449cc740 1 -- 172.21.15.12:0/2562698630 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f311a2be80 con 0x55f3119b33a0 2024-03-23T23:10:17.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.960+0000 7f4628c00640 1 -- 172.21.15.12:0/2562698630 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f462c09bce0 con 0x55f3119b33a0 2024-03-23T23:10:17.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.960+0000 7f4628c00640 1 -- 172.21.15.12:0/2562698630 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f4634005640 con 0x55f3119b33a0 2024-03-23T23:10:17.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.960+0000 7f46449cc740 10 client.27640 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:17.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.960+0000 7f46449cc740 10 client.27640 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:10:17.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.960+0000 7f46449cc740 20 client.27640 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:17.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.960+0000 7f46449cc740 1 --2- 172.21.15.12:0/2562698630 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55f311a2c7c0 0x55f311a2ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:17.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.960+0000 7f46449cc740 4 client.27640 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:10:17.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.960+0000 7f46449cc740 1 -- 172.21.15.12:0/2562698630 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f31191db50 con 0x55f311a2c7c0 2024-03-23T23:10:17.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.960+0000 7f463aa00640 1 --2- 172.21.15.12:0/2562698630 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55f311a2c7c0 0x55f311a2ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:17.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.961+0000 7f463aa00640 1 --2- 172.21.15.12:0/2562698630 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55f311a2c7c0 0x55f311a2ebe0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=19c787da8e3270c1 server_cookie=161227a4ab893cfd in_seq=0 out_seq=0 2024-03-23T23:10:17.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.961+0000 7f4628c00640 10 client.27640 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:10:17.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.961+0000 7f4628c00640 1 -- 172.21.15.12:0/2562698630 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f31191db50 con 0x55f311a2c7c0 2024-03-23T23:10:17.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.961+0000 7f4628c00640 10 client.27640 handle_command_reply: tid=0 2024-03-23T23:10:17.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.961+0000 7f46449cc740 1 client.27640 shutdown 2024-03-23T23:10:17.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.961+0000 7f46449cc740 1 -- 172.21.15.12:0/2562698630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3119b33a0 msgr2=0x55f3119bf670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:17.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.961+0000 7f46449cc740 1 --2- 172.21.15.12:0/2562698630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3119b33a0 0x55f3119bf670 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f463400fb50 tx=0x7f4634004a40 comp rx=0 tx=0).stop 2024-03-23T23:10:17.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.961+0000 7f46449cc740 1 -- 172.21.15.12:0/2562698630 shutdown_connections 2024-03-23T23:10:17.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.961+0000 7f46449cc740 1 --2- 172.21.15.12:0/2562698630 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55f311a2c7c0 0x55f311a2ebe0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.961+0000 7f46449cc740 1 --2- 172.21.15.12:0/2562698630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3119b33a0 0x55f3119bf670 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.961+0000 7f46449cc740 1 --2- 172.21.15.12:0/2562698630 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f3119b29e0 0x55f3119c2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.961+0000 7f46449cc740 1 --2- 172.21.15.12:0/2562698630 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f3119b1810 0x55f3119c2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.961+0000 7f4628c00640 0 client.27640 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:10:17.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.961+0000 7f46449cc740 1 -- 172.21.15.12:0/2562698630 >> 172.21.15.12:0/2562698630 conn(0x55f3119acc00 msgr2=0x55f3119ab620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:17.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.961+0000 7f46449cc740 1 -- 172.21.15.12:0/2562698630 shutdown_connections 2024-03-23T23:10:17.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.961+0000 7f46449cc740 1 -- 172.21.15.12:0/2562698630 wait complete. 2024-03-23T23:10:17.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.962+0000 7f46449cc740 20 client.27640 trim_cache size 0 max 16384 2024-03-23T23:10:17.966 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:10:17.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.962+0000 7f4641c00640 1 -- 172.21.15.12:0/2325502305 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f46200a2800 msgr2=0x7f46200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:17.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.962+0000 7f4641c00640 1 --2- 172.21.15.12:0/2325502305 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f46200a2800 0x7f46200a4cc0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x55f3119b28e0 tx=0x7f4634009500 comp rx=0 tx=0).stop 2024-03-23T23:10:17.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.962+0000 7f4641c00640 1 -- 172.21.15.12:0/2325502305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f463c1038c0 msgr2=0x7f463c199ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:17.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.962+0000 7f4641c00640 1 --2- 172.21.15.12:0/2325502305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f463c1038c0 0x7f463c199ba0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f462c00e9c0 tx=0x7f462c00ee80 comp rx=0 tx=0).stop 2024-03-23T23:10:17.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.962+0000 7f4641c00640 1 -- 172.21.15.12:0/2325502305 shutdown_connections 2024-03-23T23:10:17.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.962+0000 7f4641c00640 1 --2- 172.21.15.12:0/2325502305 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f46200a2800 0x7f46200a4cc0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.962+0000 7f4641c00640 1 --2- 172.21.15.12:0/2325502305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f463c105470 0x7f463c1a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.962+0000 7f4641c00640 1 --2- 172.21.15.12:0/2325502305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f463c104ab0 0x7f463c19a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.962+0000 7f4641c00640 1 --2- 172.21.15.12:0/2325502305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f463c1038c0 0x7f463c199ba0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:17.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.962+0000 7f4641c00640 1 -- 172.21.15.12:0/2325502305 >> 172.21.15.12:0/2325502305 conn(0x7f463c0fee50 msgr2=0x7f463c107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:17.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.962+0000 7f4641c00640 1 -- 172.21.15.12:0/2325502305 shutdown_connections 2024-03-23T23:10:17.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:17.962+0000 7f4641c00640 1 -- 172.21.15.12:0/2325502305 wait complete. 2024-03-23T23:10:17.979 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:10:17.979 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-23T23:10:18.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.201+0000 7f8c00c00640 1 -- 172.21.15.12:0/4054785844 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bfc102790 msgr2=0x7f8bfc102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:18.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.201+0000 7f8c00c00640 1 --2- 172.21.15.12:0/4054785844 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bfc102790 0x7f8bfc102c10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f8be4009900 tx=0x7f8be402bc40 comp rx=0 tx=0).stop 2024-03-23T23:10:18.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.201+0000 7f8c00c00640 1 -- 172.21.15.12:0/4054785844 shutdown_connections 2024-03-23T23:10:18.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.201+0000 7f8c00c00640 1 --2- 172.21.15.12:0/4054785844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bfc103150 0x7f8bfc1099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.201+0000 7f8c00c00640 1 --2- 172.21.15.12:0/4054785844 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bfc102790 0x7f8bfc102c10 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.201+0000 7f8c00c00640 1 --2- 172.21.15.12:0/4054785844 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bfc1015a0 0x7f8bfc1019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.201+0000 7f8c00c00640 1 -- 172.21.15.12:0/4054785844 >> 172.21.15.12:0/4054785844 conn(0x7f8bfc0fcb10 msgr2=0x7f8bfc0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:18.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.201+0000 7f8c00c00640 1 -- 172.21.15.12:0/4054785844 shutdown_connections 2024-03-23T23:10:18.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.201+0000 7f8c00c00640 1 -- 172.21.15.12:0/4054785844 wait complete. 2024-03-23T23:10:18.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.202+0000 7f8c00c00640 1 Processor -- start 2024-03-23T23:10:18.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.202+0000 7f8c00c00640 1 -- start start 2024-03-23T23:10:18.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.202+0000 7f8c00c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bfc1015a0 0x7f8bfc199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.202+0000 7f8c00c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bfc102790 0x7f8bfc199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.202+0000 7f8c00c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bfc103150 0x7f8bfc1a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.202+0000 7f8c00c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc10d250 con 0x7f8bfc1015a0 2024-03-23T23:10:18.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.202+0000 7f8c00c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc10d0d0 con 0x7f8bfc102790 2024-03-23T23:10:18.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.202+0000 7f8c00c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc10d3d0 con 0x7f8bfc103150 2024-03-23T23:10:18.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.203+0000 7f8bfaa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bfc102790 0x7f8bfc199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.203+0000 7f8bfb400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bfc1015a0 0x7f8bfc199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.203+0000 7f8bfaa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bfc102790 0x7f8bfc199fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48490/0 (socket says 172.21.15.12:48490) 2024-03-23T23:10:18.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.203+0000 7f8bfaa00640 1 -- 172.21.15.12:0/1189379103 learned_addr learned my addr 172.21.15.12:0/1189379103 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:18.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.203+0000 7f8bfbe00640 1 --2- 172.21.15.12:0/1189379103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bfc103150 0x7f8bfc1a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.203+0000 7f8bfaa00640 1 -- 172.21.15.12:0/1189379103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bfc103150 msgr2=0x7f8bfc1a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:18.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.203+0000 7f8bfaa00640 1 --2- 172.21.15.12:0/1189379103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bfc103150 0x7f8bfc1a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.203+0000 7f8bfaa00640 1 -- 172.21.15.12:0/1189379103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bfc1015a0 msgr2=0x7f8bfc199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:18.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.203+0000 7f8bfaa00640 1 --2- 172.21.15.12:0/1189379103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bfc1015a0 0x7f8bfc199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.203+0000 7f8bfaa00640 1 -- 172.21.15.12:0/1189379103 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc1a16a0 con 0x7f8bfc102790 2024-03-23T23:10:18.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.203+0000 7f8bfbe00640 1 --2- 172.21.15.12:0/1189379103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bfc103150 0x7f8bfc1a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:18.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.204+0000 7f8bfaa00640 1 --2- 172.21.15.12:0/1189379103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bfc102790 0x7f8bfc199fc0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8be402c8f0 tx=0x7f8be402c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:18.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.204+0000 7f8bf3e00640 1 -- 172.21.15.12:0/1189379103 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be4041070 con 0x7f8bfc102790 2024-03-23T23:10:18.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.204+0000 7f8bf3e00640 1 -- 172.21.15.12:0/1189379103 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8be4004450 con 0x7f8bfc102790 2024-03-23T23:10:18.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.204+0000 7f8c00c00640 1 -- 172.21.15.12:0/1189379103 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bfc1a1930 con 0x7f8bfc102790 2024-03-23T23:10:18.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.204+0000 7f8bf3e00640 1 -- 172.21.15.12:0/1189379103 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be4034440 con 0x7f8bfc102790 2024-03-23T23:10:18.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.204+0000 7f8c00c00640 1 -- 172.21.15.12:0/1189379103 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bfc1a1e70 con 0x7f8bfc102790 2024-03-23T23:10:18.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.208+0000 7f8c04f8e740 1 Processor -- start 2024-03-23T23:10:18.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.208+0000 7f8c04f8e740 1 -- start start 2024-03-23T23:10:18.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.209+0000 7f8bf3e00640 1 -- 172.21.15.12:0/1189379103 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8be4044070 con 0x7f8bfc102790 2024-03-23T23:10:18.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.209+0000 7f8c04f8e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5647c39099f0 0x5647c3909e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.209+0000 7f8c04f8e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5647c390abc0 0x5647c390b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.209+0000 7f8c04f8e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5647c390b580 0x5647c3911e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.209+0000 7f8c04f8e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647c386e2c0 con 0x5647c39099f0 2024-03-23T23:10:18.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.209+0000 7f8c04f8e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647c3816c90 con 0x5647c390abc0 2024-03-23T23:10:18.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.209+0000 7f8c04f8e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647c386a360 con 0x5647c390b580 2024-03-23T23:10:18.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.209+0000 7f8bfb400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5647c390abc0 0x5647c390b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.209+0000 7f8bfbe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5647c39099f0 0x5647c3909e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.209+0000 7f8bfb400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5647c390abc0 0x5647c390b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48494/0 (socket says 172.21.15.12:48494) 2024-03-23T23:10:18.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.209+0000 7f8bfb400640 1 -- 172.21.15.12:0/995412453 learned_addr learned my addr 172.21.15.12:0/995412453 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:18.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.209+0000 7f8bfaa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5647c390b580 0x5647c3911e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.209+0000 7f8bfb400640 1 -- 172.21.15.12:0/995412453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5647c390b580 msgr2=0x5647c3911e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:18.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.209+0000 7f8bfb400640 1 --2- 172.21.15.12:0/995412453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5647c390b580 0x5647c3911e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.210+0000 7f8bfb400640 1 -- 172.21.15.12:0/995412453 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5647c39099f0 msgr2=0x5647c3909e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:18.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.210+0000 7f8bfb400640 1 --2- 172.21.15.12:0/995412453 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5647c39099f0 0x5647c3909e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.210+0000 7f8bfb400640 1 -- 172.21.15.12:0/995412453 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5647c383b080 con 0x5647c390abc0 2024-03-23T23:10:18.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.210+0000 7f8bfb400640 1 --2- 172.21.15.12:0/995412453 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5647c390abc0 0x5647c390b040 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f8bec011470 tx=0x7f8bec011930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5bb906cc326061 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:18.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.210+0000 7f8bebe00640 1 -- 172.21.15.12:0/995412453 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bec003970 con 0x5647c390abc0 2024-03-23T23:10:18.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.210+0000 7f8bebe00640 1 -- 172.21.15.12:0/995412453 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8bec004020 con 0x5647c390abc0 2024-03-23T23:10:18.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.210+0000 7f8bebe00640 1 -- 172.21.15.12:0/995412453 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bec00f9c0 con 0x5647c390abc0 2024-03-23T23:10:18.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.210+0000 7f8bf3e00640 1 --2- 172.21.15.12:0/1189379103 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8be00a2880 0x7f8be00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.211+0000 7f8c04f8e740 1 -- 172.21.15.12:0/995412453 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5647c390abc0 msgr2=0x5647c390b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:18.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.211+0000 7f8c04f8e740 1 --2- 172.21.15.12:0/995412453 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5647c390abc0 0x5647c390b040 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f8bec011470 tx=0x7f8bec011930 comp rx=0 tx=0).stop 2024-03-23T23:10:18.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.211+0000 7f8bf3e00640 1 -- 172.21.15.12:0/1189379103 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8be40f0530 con 0x7f8bfc102790 2024-03-23T23:10:18.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.211+0000 7f8bfbe00640 1 --2- 172.21.15.12:0/1189379103 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8be00a2880 0x7f8be00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.211+0000 7f8c04f8e740 1 -- 172.21.15.12:0/995412453 shutdown_connections 2024-03-23T23:10:18.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.211+0000 7f8c04f8e740 1 --2- 172.21.15.12:0/995412453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5647c390b580 0x5647c3911e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.211+0000 7f8c04f8e740 1 --2- 172.21.15.12:0/995412453 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5647c390abc0 0x5647c390b040 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.211+0000 7f8c04f8e740 1 --2- 172.21.15.12:0/995412453 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5647c39099f0 0x5647c3909e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.211+0000 7f8c04f8e740 1 -- 172.21.15.12:0/995412453 >> 172.21.15.12:0/995412453 conn(0x5647c3904de0 msgr2=0x5647c39051e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:18.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.211+0000 7f8c04f8e740 1 -- 172.21.15.12:0/995412453 shutdown_connections 2024-03-23T23:10:18.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.211+0000 7f8c04f8e740 1 -- 172.21.15.12:0/995412453 wait complete. 2024-03-23T23:10:18.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.211+0000 7f8c04f8e740 1 -- start start 2024-03-23T23:10:18.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.211+0000 7f8bfbe00640 1 --2- 172.21.15.12:0/1189379103 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8be00a2880 0x7f8be00a4d40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x5647c390aac0 tx=0x7f8bf4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:18.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.212+0000 7f8c04f8e740 1 Processor -- start 2024-03-23T23:10:18.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.212+0000 7f8c04f8e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5647c39099f0 0x5647c3913c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.212+0000 7f8c04f8e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5647c390abc0 0x5647c3914170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.212+0000 7f8c04f8e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5647c390b580 0x5647c391b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.212+0000 7f8c04f8e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647c3912590 con 0x5647c390b580 2024-03-23T23:10:18.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.212+0000 7f8c04f8e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647c382a0d0 con 0x5647c390abc0 2024-03-23T23:10:18.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.212+0000 7f8c04f8e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647c3912710 con 0x5647c39099f0 2024-03-23T23:10:18.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.212+0000 7f8c04f8e740 10 client.0 _ll_register_callbacks cb 0x5647c382ee80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:18.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.212+0000 7f8bfb400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5647c39099f0 0x5647c3913c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.212+0000 7f8bfb400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5647c39099f0 0x5647c3913c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:41284/0 (socket says 172.21.15.12:41284) 2024-03-23T23:10:18.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.212+0000 7f8bfb400640 1 -- 172.21.15.12:0/3672111461 learned_addr learned my addr 172.21.15.12:0/3672111461 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:18.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.212+0000 7f8bfaa00640 1 --2- 172.21.15.12:0/3672111461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5647c390abc0 0x5647c3914170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.213+0000 7f8bfbe00640 1 --2- 172.21.15.12:0/3672111461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5647c390b580 0x5647c391b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.213+0000 7f8bfb400640 1 -- 172.21.15.12:0/3672111461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5647c390abc0 msgr2=0x5647c3914170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.213+0000 7f8bfb400640 1 --2- 172.21.15.12:0/3672111461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5647c390abc0 0x5647c3914170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.213+0000 7f8bfb400640 1 -- 172.21.15.12:0/3672111461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5647c390b580 msgr2=0x5647c391b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.213+0000 7f8bfb400640 1 --2- 172.21.15.12:0/3672111461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5647c390b580 0x5647c391b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.213+0000 7f8bfb400640 1 -- 172.21.15.12:0/3672111461 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5647c35823f0 con 0x5647c39099f0 2024-03-23T23:10:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.213+0000 7f8bfb400640 1 --2- 172.21.15.12:0/3672111461 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5647c39099f0 0x5647c3913c30 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f8bec01b040 tx=0x7f8bec004c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3652b876084e6f21 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.213+0000 7f8be8c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:10:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.213+0000 7f8be8c00640 1 -- 172.21.15.12:0/3672111461 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bec027020 con 0x5647c39099f0 2024-03-23T23:10:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.213+0000 7f8c04f8e740 1 -- 172.21.15.12:0/3672111461 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8bec004020 con 0x5647c39099f0 2024-03-23T23:10:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.213+0000 7f8be8c00640 1 -- 172.21.15.12:0/3672111461 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8bec003b80 con 0x5647c39099f0 2024-03-23T23:10:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.214+0000 7f8be8c00640 1 -- 172.21.15.12:0/3672111461 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8bec004e60 con 0x5647c39099f0 2024-03-23T23:10:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.214+0000 7f8c04f8e740 10 client.27652 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.214+0000 7f8c04f8e740 1 -- 172.21.15.12:0/3672111461 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5647c364c7d0 con 0x5647c39099f0 2024-03-23T23:10:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.214+0000 7f8be8c00640 1 -- 172.21.15.12:0/3672111461 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8bec004020 con 0x5647c39099f0 2024-03-23T23:10:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.214+0000 7f8be8c00640 1 -- 172.21.15.12:0/3672111461 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8bec005040 con 0x5647c39099f0 2024-03-23T23:10:18.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.214+0000 7f8c04f8e740 10 client.27652 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:18.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.214+0000 7f8c04f8e740 10 client.27652 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:10:18.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.214+0000 7f8c04f8e740 20 client.27652 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:18.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.214+0000 7f8c04f8e740 1 --2- 172.21.15.12:0/3672111461 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5647c391f3c0 0x5647c39217d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.214+0000 7f8c04f8e740 4 client.27652 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:10:18.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.214+0000 7f8c04f8e740 1 -- 172.21.15.12:0/3672111461 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5647c387b800 con 0x5647c391f3c0 2024-03-23T23:10:18.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.215+0000 7f8bfaa00640 1 --2- 172.21.15.12:0/3672111461 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5647c391f3c0 0x5647c39217d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.215+0000 7f8bfaa00640 1 --2- 172.21.15.12:0/3672111461 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5647c391f3c0 0x5647c39217d0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fba04fcc45308adb server_cookie=8866bf290187944f in_seq=0 out_seq=0 2024-03-23T23:10:18.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.215+0000 7f8be8c00640 10 client.27652 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:10:18.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.216+0000 7f8be8c00640 1 -- 172.21.15.12:0/3672111461 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5647c387b800 con 0x5647c391f3c0 2024-03-23T23:10:18.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.216+0000 7f8be8c00640 10 client.27652 handle_command_reply: tid=0 2024-03-23T23:10:18.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.216+0000 7f8c04f8e740 1 client.27652 shutdown 2024-03-23T23:10:18.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.216+0000 7f8c04f8e740 1 -- 172.21.15.12:0/3672111461 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5647c39099f0 msgr2=0x5647c3913c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:18.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.216+0000 7f8c04f8e740 1 --2- 172.21.15.12:0/3672111461 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5647c39099f0 0x5647c3913c30 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f8bec01b040 tx=0x7f8bec004c70 comp rx=0 tx=0).stop 2024-03-23T23:10:18.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.216+0000 7f8c04f8e740 1 -- 172.21.15.12:0/3672111461 shutdown_connections 2024-03-23T23:10:18.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.216+0000 7f8c04f8e740 1 --2- 172.21.15.12:0/3672111461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5647c390b580 0x5647c391b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.216+0000 7f8c04f8e740 1 --2- 172.21.15.12:0/3672111461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5647c390abc0 0x5647c3914170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.216+0000 7f8c04f8e740 1 --2- 172.21.15.12:0/3672111461 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5647c391f3c0 0x5647c39217d0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.216+0000 7f8c04f8e740 1 --2- 172.21.15.12:0/3672111461 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5647c39099f0 0x5647c3913c30 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.216+0000 7f8c04f8e740 1 -- 172.21.15.12:0/3672111461 >> 172.21.15.12:0/3672111461 conn(0x5647c3904de0 msgr2=0x5647c3904160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:18.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.216+0000 7f8be8c00640 0 client.27652 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:10:18.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.216+0000 7f8c04f8e740 1 -- 172.21.15.12:0/3672111461 shutdown_connections 2024-03-23T23:10:18.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.216+0000 7f8c04f8e740 1 -- 172.21.15.12:0/3672111461 wait complete. 2024-03-23T23:10:18.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.216+0000 7f8c04f8e740 20 client.27652 trim_cache size 0 max 16384 2024-03-23T23:10:18.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.230+0000 7f8c04f8e740 1 Processor -- start 2024-03-23T23:10:18.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.230+0000 7f8c04f8e740 1 -- start start 2024-03-23T23:10:18.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.231+0000 7f8c04f8e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5647c39099f0 0x5647c3908b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.231+0000 7f8c04f8e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5647c390abc0 0x5647c391b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.231+0000 7f8c04f8e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5647c390b580 0x5647c39260d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.231+0000 7f8c04f8e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc1acf40 con 0x5647c390b580 2024-03-23T23:10:18.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.231+0000 7f8c04f8e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc1aaa40 con 0x5647c39099f0 2024-03-23T23:10:18.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.231+0000 7f8c04f8e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc1a8ae0 con 0x5647c390abc0 2024-03-23T23:10:18.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.231+0000 7f8bfb400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5647c39099f0 0x5647c3908b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.231+0000 7f8bfaa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5647c390abc0 0x5647c391b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.231+0000 7f8bfb400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5647c39099f0 0x5647c3908b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48514/0 (socket says 172.21.15.12:48514) 2024-03-23T23:10:18.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.231+0000 7f8bfaa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5647c390abc0 0x5647c391b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:41300/0 (socket says 172.21.15.12:41300) 2024-03-23T23:10:18.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.231+0000 7f8bfb400640 1 -- 172.21.15.12:0/3393317553 learned_addr learned my addr 172.21.15.12:0/3393317553 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:18.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.231+0000 7f8bfbe00640 1 --2- 172.21.15.12:0/3393317553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5647c390b580 0x5647c39260d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.231+0000 7f8bfb400640 1 -- 172.21.15.12:0/3393317553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5647c390abc0 msgr2=0x5647c391b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:18.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.231+0000 7f8bfb400640 1 --2- 172.21.15.12:0/3393317553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5647c390abc0 0x5647c391b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.231+0000 7f8bfb400640 1 -- 172.21.15.12:0/3393317553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5647c390b580 msgr2=0x5647c39260d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:18.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.231+0000 7f8bfb400640 1 --2- 172.21.15.12:0/3393317553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5647c390b580 0x5647c39260d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.231+0000 7f8bfb400640 1 -- 172.21.15.12:0/3393317553 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5647c390a640 con 0x5647c39099f0 2024-03-23T23:10:18.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.231+0000 7f8bfbe00640 1 --2- 172.21.15.12:0/3393317553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5647c390b580 0x5647c39260d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:10:18.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.232+0000 7f8bfb400640 1 --2- 172.21.15.12:0/3393317553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5647c39099f0 0x5647c3908b10 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f8bec0053e0 tx=0x7f8bec0021d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afe2fd387f33dd70 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:18.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.232+0000 7f8beb400640 1 -- 172.21.15.12:0/3393317553 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bec027070 con 0x5647c39099f0 2024-03-23T23:10:18.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.232+0000 7f8beb400640 1 -- 172.21.15.12:0/3393317553 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8bec005040 con 0x5647c39099f0 2024-03-23T23:10:18.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.232+0000 7f8beb400640 1 -- 172.21.15.12:0/3393317553 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bec011030 con 0x5647c39099f0 2024-03-23T23:10:18.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.233+0000 7f8c04f8e740 1 -- 172.21.15.12:0/3393317553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5647c39099f0 msgr2=0x5647c3908b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:18.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.233+0000 7f8c04f8e740 1 --2- 172.21.15.12:0/3393317553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5647c39099f0 0x5647c3908b10 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f8bec0053e0 tx=0x7f8bec0021d0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.233+0000 7f8c04f8e740 1 -- 172.21.15.12:0/3393317553 shutdown_connections 2024-03-23T23:10:18.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.233+0000 7f8c04f8e740 1 --2- 172.21.15.12:0/3393317553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5647c390b580 0x5647c39260d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.233+0000 7f8c04f8e740 1 --2- 172.21.15.12:0/3393317553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5647c390abc0 0x5647c391b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.233+0000 7f8c04f8e740 1 --2- 172.21.15.12:0/3393317553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5647c39099f0 0x5647c3908b10 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.233+0000 7f8c04f8e740 1 -- 172.21.15.12:0/3393317553 >> 172.21.15.12:0/3393317553 conn(0x5647c3904de0 msgr2=0x5647c3904160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:18.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.233+0000 7f8c04f8e740 1 -- 172.21.15.12:0/3393317553 shutdown_connections 2024-03-23T23:10:18.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.233+0000 7f8c04f8e740 1 -- 172.21.15.12:0/3393317553 wait complete. 2024-03-23T23:10:18.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.233+0000 7f8c04f8e740 1 -- start start 2024-03-23T23:10:18.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.233+0000 7f8c04f8e740 1 Processor -- start 2024-03-23T23:10:18.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.234+0000 7f8c04f8e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5647c39099f0 0x5647c3918a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.234+0000 7f8c04f8e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5647c390abc0 0x5647c3918f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.234+0000 7f8c04f8e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5647c390b580 0x5647c3919f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.234+0000 7f8c04f8e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc1a90e0 con 0x5647c390b580 2024-03-23T23:10:18.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.234+0000 7f8c04f8e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc1aa3e0 con 0x5647c390abc0 2024-03-23T23:10:18.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.234+0000 7f8c04f8e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc1a9720 con 0x5647c39099f0 2024-03-23T23:10:18.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.234+0000 7f8c04f8e740 10 client.0 _ll_register_callbacks cb 0x7f8bf4009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:18.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.234+0000 7f8bfb400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5647c39099f0 0x5647c3918a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.234+0000 7f8bfaa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5647c390abc0 0x5647c3918f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.234+0000 7f8bfb400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5647c39099f0 0x5647c3918a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:41302/0 (socket says 172.21.15.12:41302) 2024-03-23T23:10:18.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.234+0000 7f8bfb400640 1 -- 172.21.15.12:0/1835973086 learned_addr learned my addr 172.21.15.12:0/1835973086 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:18.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.234+0000 7f8bfbe00640 1 --2- 172.21.15.12:0/1835973086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5647c390b580 0x5647c3919f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.234+0000 7f8bfb400640 1 -- 172.21.15.12:0/1835973086 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5647c390abc0 msgr2=0x5647c3918f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:18.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.234+0000 7f8bfb400640 1 --2- 172.21.15.12:0/1835973086 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5647c390abc0 0x5647c3918f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.234+0000 7f8bfb400640 1 -- 172.21.15.12:0/1835973086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5647c390b580 msgr2=0x5647c3919f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:18.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.234+0000 7f8bfb400640 1 --2- 172.21.15.12:0/1835973086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5647c390b580 0x5647c3919f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.235+0000 7f8bfb400640 1 -- 172.21.15.12:0/1835973086 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5647c391a550 con 0x5647c39099f0 2024-03-23T23:10:18.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.235+0000 7f8bfaa00640 1 --2- 172.21.15.12:0/1835973086 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5647c390abc0 0x5647c3918f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:18.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.235+0000 7f8bfb400640 1 --2- 172.21.15.12:0/1835973086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5647c39099f0 0x5647c3918a10 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8bec005480 tx=0x7f8bec0094a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1e38559d9045dad server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:18.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.235+0000 7f8be8c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:10:18.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.235+0000 7f8be8c00640 1 -- 172.21.15.12:0/1835973086 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bec027070 con 0x5647c39099f0 2024-03-23T23:10:18.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.235+0000 7f8be8c00640 1 -- 172.21.15.12:0/1835973086 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8bec011230 con 0x5647c39099f0 2024-03-23T23:10:18.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.235+0000 7f8c04f8e740 1 -- 172.21.15.12:0/1835973086 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8bec005040 con 0x5647c39099f0 2024-03-23T23:10:18.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.235+0000 7f8be8c00640 1 -- 172.21.15.12:0/1835973086 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8bec003b80 con 0x5647c39099f0 2024-03-23T23:10:18.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.235+0000 7f8c04f8e740 10 client.27658 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:18.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.235+0000 7f8c04f8e740 1 -- 172.21.15.12:0/1835973086 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5647c397ae40 con 0x5647c39099f0 2024-03-23T23:10:18.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.236+0000 7f8be8c00640 1 -- 172.21.15.12:0/1835973086 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8bec005040 con 0x5647c39099f0 2024-03-23T23:10:18.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.236+0000 7f8be8c00640 1 -- 172.21.15.12:0/1835973086 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8bec043440 con 0x5647c39099f0 2024-03-23T23:10:18.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.236+0000 7f8c04f8e740 10 client.27658 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:18.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.236+0000 7f8c04f8e740 10 client.27658 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:10:18.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.236+0000 7f8c04f8e740 20 client.27658 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:18.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.236+0000 7f8c04f8e740 1 --2- 172.21.15.12:0/1835973086 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5647c397b630 0x5647c397daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.236+0000 7f8c04f8e740 4 client.27658 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:10:18.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.236+0000 7f8c04f8e740 1 -- 172.21.15.12:0/1835973086 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5647c3875b50 con 0x5647c397b630 2024-03-23T23:10:18.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.236+0000 7f8bfaa00640 1 --2- 172.21.15.12:0/1835973086 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5647c397b630 0x5647c397daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.237+0000 7f8bfaa00640 1 --2- 172.21.15.12:0/1835973086 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5647c397b630 0x5647c397daf0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=851fdec1ca35bc87 server_cookie=676a3d2e4205eb9e in_seq=0 out_seq=0 2024-03-23T23:10:18.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.237+0000 7f8be8c00640 10 client.27658 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:10:18.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.237+0000 7f8be8c00640 1 -- 172.21.15.12:0/1835973086 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5647c3875b50 con 0x5647c397b630 2024-03-23T23:10:18.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.237+0000 7f8be8c00640 10 client.27658 handle_command_reply: tid=0 2024-03-23T23:10:18.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.237+0000 7f8c04f8e740 1 client.27658 shutdown 2024-03-23T23:10:18.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.237+0000 7f8c04f8e740 1 -- 172.21.15.12:0/1835973086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5647c39099f0 msgr2=0x5647c3918a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:18.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.237+0000 7f8c04f8e740 1 --2- 172.21.15.12:0/1835973086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5647c39099f0 0x5647c3918a10 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8bec005480 tx=0x7f8bec0094a0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.237+0000 7f8c04f8e740 1 -- 172.21.15.12:0/1835973086 shutdown_connections 2024-03-23T23:10:18.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.237+0000 7f8c04f8e740 1 --2- 172.21.15.12:0/1835973086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5647c390b580 0x5647c3919f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.237+0000 7f8c04f8e740 1 --2- 172.21.15.12:0/1835973086 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5647c390abc0 0x5647c3918f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.237+0000 7f8c04f8e740 1 --2- 172.21.15.12:0/1835973086 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x5647c397b630 0x5647c397daf0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.237+0000 7f8c04f8e740 1 --2- 172.21.15.12:0/1835973086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5647c39099f0 0x5647c3918a10 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.237+0000 7f8be8c00640 0 client.27658 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:10:18.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.237+0000 7f8c04f8e740 1 -- 172.21.15.12:0/1835973086 >> 172.21.15.12:0/1835973086 conn(0x5647c3904de0 msgr2=0x5647c39038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:18.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.237+0000 7f8c04f8e740 1 -- 172.21.15.12:0/1835973086 shutdown_connections 2024-03-23T23:10:18.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.238+0000 7f8c04f8e740 1 -- 172.21.15.12:0/1835973086 wait complete. 2024-03-23T23:10:18.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.238+0000 7f8c04f8e740 20 client.27658 trim_cache size 0 max 16384 2024-03-23T23:10:18.242 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:10:18.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.238+0000 7f8c00c00640 1 -- 172.21.15.12:0/1189379103 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8be00a2880 msgr2=0x7f8be00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:18.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.238+0000 7f8c00c00640 1 --2- 172.21.15.12:0/1189379103 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8be00a2880 0x7f8be00a4d40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x5647c390aac0 tx=0x7f8bf4005f90 comp rx=0 tx=0).stop 2024-03-23T23:10:18.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.238+0000 7f8c00c00640 1 -- 172.21.15.12:0/1189379103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bfc102790 msgr2=0x7f8bfc199fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:18.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.238+0000 7f8c00c00640 1 --2- 172.21.15.12:0/1189379103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bfc102790 0x7f8bfc199fc0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8be402c8f0 tx=0x7f8be402c660 comp rx=0 tx=0).stop 2024-03-23T23:10:18.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.238+0000 7f8c00c00640 1 -- 172.21.15.12:0/1189379103 shutdown_connections 2024-03-23T23:10:18.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.238+0000 7f8c00c00640 1 --2- 172.21.15.12:0/1189379103 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8be00a2880 0x7f8be00a4d40 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.238+0000 7f8c00c00640 1 --2- 172.21.15.12:0/1189379103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bfc103150 0x7f8bfc1a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.238+0000 7f8c00c00640 1 --2- 172.21.15.12:0/1189379103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bfc102790 0x7f8bfc199fc0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.238+0000 7f8c00c00640 1 --2- 172.21.15.12:0/1189379103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bfc1015a0 0x7f8bfc199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.238+0000 7f8c00c00640 1 -- 172.21.15.12:0/1189379103 >> 172.21.15.12:0/1189379103 conn(0x7f8bfc0fcb10 msgr2=0x7f8bfc0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:18.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.239+0000 7f8c00c00640 1 -- 172.21.15.12:0/1189379103 shutdown_connections 2024-03-23T23:10:18.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.239+0000 7f8c00c00640 1 -- 172.21.15.12:0/1189379103 wait complete. 2024-03-23T23:10:18.256 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:10:18.256 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-23T23:10:18.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.468+0000 7f52c5800640 1 -- 172.21.15.12:0/68710574 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52c01049b0 msgr2=0x7f52c0104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:18.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.468+0000 7f52c5800640 1 --2- 172.21.15.12:0/68710574 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52c01049b0 0x7f52c0104e30 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f52ac009900 tx=0x7f52ac02bc30 comp rx=0 tx=0).stop 2024-03-23T23:10:18.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.468+0000 7f52c5800640 1 -- 172.21.15.12:0/68710574 shutdown_connections 2024-03-23T23:10:18.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.468+0000 7f52c5800640 1 --2- 172.21.15.12:0/68710574 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52c0105370 0x7f52c010bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.468+0000 7f52c5800640 1 --2- 172.21.15.12:0/68710574 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52c01049b0 0x7f52c0104e30 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.468+0000 7f52c5800640 1 --2- 172.21.15.12:0/68710574 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52c01037c0 0x7f52c0103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.468+0000 7f52c5800640 1 -- 172.21.15.12:0/68710574 >> 172.21.15.12:0/68710574 conn(0x7f52c00fed50 msgr2=0x7f52c0101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:18.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.468+0000 7f52c5800640 1 -- 172.21.15.12:0/68710574 shutdown_connections 2024-03-23T23:10:18.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.468+0000 7f52c5800640 1 -- 172.21.15.12:0/68710574 wait complete. 2024-03-23T23:10:18.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.469+0000 7f52c5800640 1 Processor -- start 2024-03-23T23:10:18.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.469+0000 7f52c5800640 1 -- start start 2024-03-23T23:10:18.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.470+0000 7f52c5800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52c01037c0 0x7f52c0199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.470+0000 7f52c5800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52c01049b0 0x7f52c0199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.470+0000 7f52c5800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52c0105370 0x7f52c01a1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.470+0000 7f52c5800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c010f470 con 0x7f52c01037c0 2024-03-23T23:10:18.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.470+0000 7f52c5800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c010f2f0 con 0x7f52c01049b0 2024-03-23T23:10:18.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.470+0000 7f52c5800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c010f5f0 con 0x7f52c0105370 2024-03-23T23:10:18.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.470+0000 7f52bfe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52c01037c0 0x7f52c0199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.470+0000 7f52bfe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52c01037c0 0x7f52c0199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49282/0 (socket says 172.21.15.12:49282) 2024-03-23T23:10:18.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.470+0000 7f52bfe00640 1 -- 172.21.15.12:0/498738262 learned_addr learned my addr 172.21.15.12:0/498738262 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:18.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.470+0000 7f52c4e00640 1 --2- 172.21.15.12:0/498738262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52c0105370 0x7f52c01a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.470+0000 7f52bf400640 1 --2- 172.21.15.12:0/498738262 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52c01049b0 0x7f52c0199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.471+0000 7f52bfe00640 1 -- 172.21.15.12:0/498738262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52c0105370 msgr2=0x7f52c01a1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.471+0000 7f52bfe00640 1 --2- 172.21.15.12:0/498738262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52c0105370 0x7f52c01a1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.471+0000 7f52bfe00640 1 -- 172.21.15.12:0/498738262 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52c01049b0 msgr2=0x7f52c0199fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.471+0000 7f52bfe00640 1 --2- 172.21.15.12:0/498738262 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52c01049b0 0x7f52c0199fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.471+0000 7f52bfe00640 1 -- 172.21.15.12:0/498738262 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c01a16b0 con 0x7f52c01037c0 2024-03-23T23:10:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.471+0000 7f52c4e00640 1 --2- 172.21.15.12:0/498738262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52c0105370 0x7f52c01a1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.471+0000 7f52bfe00640 1 --2- 172.21.15.12:0/498738262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52c01037c0 0x7f52c0199a90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f52b000d770 tx=0x7f52b000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.471+0000 7f52bcc00640 1 -- 172.21.15.12:0/498738262 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b0017070 con 0x7f52c01037c0 2024-03-23T23:10:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.471+0000 7f52bcc00640 1 -- 172.21.15.12:0/498738262 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52b0009c30 con 0x7f52c01037c0 2024-03-23T23:10:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.471+0000 7f52c5800640 1 -- 172.21.15.12:0/498738262 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c01a19a0 con 0x7f52c01037c0 2024-03-23T23:10:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.471+0000 7f52bcc00640 1 -- 172.21.15.12:0/498738262 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b0003dd0 con 0x7f52c01037c0 2024-03-23T23:10:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.472+0000 7f52c5800640 1 -- 172.21.15.12:0/498738262 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c01a1ec0 con 0x7f52c01037c0 2024-03-23T23:10:18.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.475+0000 7f52bcc00640 1 -- 172.21.15.12:0/498738262 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f52b0003f70 con 0x7f52c01037c0 2024-03-23T23:10:18.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.475+0000 7f52c9b9f740 1 Processor -- start 2024-03-23T23:10:18.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.475+0000 7f52c9b9f740 1 -- start start 2024-03-23T23:10:18.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.475+0000 7f52c9b9f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563749d219f0 0x563749d21e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.475+0000 7f52c9b9f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d22bc0 0x563749d23040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.475+0000 7f52c9b9f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563749d23580 0x563749d29e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.476+0000 7f52c9b9f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563749c862c0 con 0x563749d23580 2024-03-23T23:10:18.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.476+0000 7f52c9b9f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x563749c2ec90 con 0x563749d219f0 2024-03-23T23:10:18.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.476+0000 7f52c9b9f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x563749c82360 con 0x563749d22bc0 2024-03-23T23:10:18.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.476+0000 7f52bfe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d22bc0 0x563749d23040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.476+0000 7f52c4e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563749d219f0 0x563749d21e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.476+0000 7f52c4e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563749d219f0 0x563749d21e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48558/0 (socket says 172.21.15.12:48558) 2024-03-23T23:10:18.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.476+0000 7f52bfe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d22bc0 0x563749d23040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:41324/0 (socket says 172.21.15.12:41324) 2024-03-23T23:10:18.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.476+0000 7f52bf400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563749d23580 0x563749d29e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.476+0000 7f52c4e00640 1 -- 172.21.15.12:0/3762370939 learned_addr learned my addr 172.21.15.12:0/3762370939 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:18.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.476+0000 7f52bfe00640 1 -- 172.21.15.12:0/3762370939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563749d219f0 msgr2=0x563749d21e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:18.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.476+0000 7f52bfe00640 1 --2- 172.21.15.12:0/3762370939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563749d219f0 0x563749d21e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.476+0000 7f52bfe00640 1 -- 172.21.15.12:0/3762370939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563749d23580 msgr2=0x563749d29e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:18.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.476+0000 7f52bfe00640 1 --2- 172.21.15.12:0/3762370939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563749d23580 0x563749d29e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.476+0000 7f52bfe00640 1 -- 172.21.15.12:0/3762370939 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563749c53080 con 0x563749d22bc0 2024-03-23T23:10:18.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.476+0000 7f52c4e00640 1 --2- 172.21.15.12:0/3762370939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563749d219f0 0x563749d21e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:18.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.476+0000 7f52bcc00640 1 --2- 172.21.15.12:0/498738262 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f52a40a2800 0x7f52a40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.477+0000 7f52c4e00640 1 --2- 172.21.15.12:0/498738262 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f52a40a2800 0x7f52a40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.477+0000 7f52bcc00640 1 -- 172.21.15.12:0/498738262 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f52b00ce390 con 0x7f52c01037c0 2024-03-23T23:10:18.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.477+0000 7f52bfe00640 1 --2- 172.21.15.12:0/3762370939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d22bc0 0x563749d23040 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f52b00cbde0 tx=0x7f52b00d6970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fd64b63fc86ab46 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:18.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.477+0000 7f52b5600640 1 -- 172.21.15.12:0/3762370939 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b0020030 con 0x563749d22bc0 2024-03-23T23:10:18.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.477+0000 7f52b5600640 1 -- 172.21.15.12:0/3762370939 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52b0009de0 con 0x563749d22bc0 2024-03-23T23:10:18.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.477+0000 7f52b5600640 1 -- 172.21.15.12:0/3762370939 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b009a8d0 con 0x563749d22bc0 2024-03-23T23:10:18.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.478+0000 7f52c9b9f740 1 -- 172.21.15.12:0/3762370939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d22bc0 msgr2=0x563749d23040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:18.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.478+0000 7f52c9b9f740 1 --2- 172.21.15.12:0/3762370939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d22bc0 0x563749d23040 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f52b00cbde0 tx=0x7f52b00d6970 comp rx=0 tx=0).stop 2024-03-23T23:10:18.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.478+0000 7f52c4e00640 1 --2- 172.21.15.12:0/498738262 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f52a40a2800 0x7f52a40a4cc0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x563749d22ac0 tx=0x7f52b8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:18.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.478+0000 7f52c9b9f740 1 -- 172.21.15.12:0/3762370939 shutdown_connections 2024-03-23T23:10:18.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.478+0000 7f52c9b9f740 1 --2- 172.21.15.12:0/3762370939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563749d23580 0x563749d29e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.478+0000 7f52c9b9f740 1 --2- 172.21.15.12:0/3762370939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d22bc0 0x563749d23040 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.478+0000 7f52c9b9f740 1 --2- 172.21.15.12:0/3762370939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563749d219f0 0x563749d21e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.478+0000 7f52c9b9f740 1 -- 172.21.15.12:0/3762370939 >> 172.21.15.12:0/3762370939 conn(0x563749d1cde0 msgr2=0x563749d1d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:18.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.478+0000 7f52c9b9f740 1 -- 172.21.15.12:0/3762370939 shutdown_connections 2024-03-23T23:10:18.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.478+0000 7f52c9b9f740 1 -- 172.21.15.12:0/3762370939 wait complete. 2024-03-23T23:10:18.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.479+0000 7f52c9b9f740 1 -- start start 2024-03-23T23:10:18.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.479+0000 7f52c9b9f740 1 Processor -- start 2024-03-23T23:10:18.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.479+0000 7f52c9b9f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d219f0 0x563749d2bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.479+0000 7f52c9b9f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563749d22bc0 0x563749d2c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.479+0000 7f52c9b9f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563749d23580 0x563749d33180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.479+0000 7f52c9b9f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563749d2a590 con 0x563749d22bc0 2024-03-23T23:10:18.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.479+0000 7f52c9b9f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x563749c420d0 con 0x563749d23580 2024-03-23T23:10:18.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.479+0000 7f52c9b9f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x563749d2a710 con 0x563749d219f0 2024-03-23T23:10:18.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.480+0000 7f52c9b9f740 10 client.0 _ll_register_callbacks cb 0x563749c46e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:18.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.480+0000 7f52bf400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563749d22bc0 0x563749d2c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.480+0000 7f52bfe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d219f0 0x563749d2bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.480+0000 7f52bfe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d219f0 0x563749d2bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:41328/0 (socket says 172.21.15.12:41328) 2024-03-23T23:10:18.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.480+0000 7f52bfe00640 1 -- 172.21.15.12:0/2820738541 learned_addr learned my addr 172.21.15.12:0/2820738541 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:18.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.480+0000 7f52c4e00640 1 --2- 172.21.15.12:0/2820738541 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563749d23580 0x563749d33180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.480+0000 7f52bfe00640 1 -- 172.21.15.12:0/2820738541 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563749d23580 msgr2=0x563749d33180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:18.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.480+0000 7f52bfe00640 1 --2- 172.21.15.12:0/2820738541 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563749d23580 0x563749d33180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.480+0000 7f52bfe00640 1 -- 172.21.15.12:0/2820738541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563749d22bc0 msgr2=0x563749d2c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:18.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.480+0000 7f52bfe00640 1 --2- 172.21.15.12:0/2820738541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563749d22bc0 0x563749d2c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.480+0000 7f52bfe00640 1 -- 172.21.15.12:0/2820738541 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56374999a3f0 con 0x563749d219f0 2024-03-23T23:10:18.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.480+0000 7f52bfe00640 1 --2- 172.21.15.12:0/2820738541 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d219f0 0x563749d2bc00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f52b0012040 tx=0x7f52b0003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76765931cddc2036 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:18.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.480+0000 7f52aa000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:10:18.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.481+0000 7f52aa000640 1 -- 172.21.15.12:0/2820738541 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b0020040 con 0x563749d219f0 2024-03-23T23:10:18.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.481+0000 7f52aa000640 1 -- 172.21.15.12:0/2820738541 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f52b0097400 con 0x563749d219f0 2024-03-23T23:10:18.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.481+0000 7f52c9b9f740 1 -- 172.21.15.12:0/2820738541 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f52b0009de0 con 0x563749d219f0 2024-03-23T23:10:18.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.481+0000 7f52aa000640 1 -- 172.21.15.12:0/2820738541 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52b009a6f0 con 0x563749d219f0 2024-03-23T23:10:18.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.481+0000 7f52c9b9f740 10 client.27670 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:18.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.481+0000 7f52c9b9f740 1 -- 172.21.15.12:0/2820738541 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563749a647d0 con 0x563749d219f0 2024-03-23T23:10:18.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.481+0000 7f52aa000640 1 -- 172.21.15.12:0/2820738541 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52b0009de0 con 0x563749d219f0 2024-03-23T23:10:18.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.481+0000 7f52aa000640 1 -- 172.21.15.12:0/2820738541 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f52b009aff0 con 0x563749d219f0 2024-03-23T23:10:18.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.481+0000 7f52c9b9f740 10 client.27670 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:18.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.481+0000 7f52c9b9f740 10 client.27670 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:18.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.481+0000 7f52c9b9f740 20 client.27670 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:18.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.481+0000 7f52c9b9f740 1 --2- 172.21.15.12:0/2820738541 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563749d372e0 0x563749d396d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.481+0000 7f52c9b9f740 4 client.27670 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:10:18.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.481+0000 7f52c9b9f740 1 -- 172.21.15.12:0/2820738541 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563749c93800 con 0x563749d372e0 2024-03-23T23:10:18.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.482+0000 7f52bf400640 1 --2- 172.21.15.12:0/2820738541 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563749d372e0 0x563749d396d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.482+0000 7f52bf400640 1 --2- 172.21.15.12:0/2820738541 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563749d372e0 0x563749d396d0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f2a21bc1173a4cc server_cookie=a90ff3418819900a in_seq=0 out_seq=0 2024-03-23T23:10:18.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.482+0000 7f52aa000640 10 client.27670 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:18.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.483+0000 7f52aa000640 1 -- 172.21.15.12:0/2820738541 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563749c93800 con 0x563749d372e0 2024-03-23T23:10:18.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.483+0000 7f52aa000640 10 client.27670 handle_command_reply: tid=0 2024-03-23T23:10:18.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.483+0000 7f52c9b9f740 1 client.27670 shutdown 2024-03-23T23:10:18.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.483+0000 7f52c9b9f740 1 -- 172.21.15.12:0/2820738541 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d219f0 msgr2=0x563749d2bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:18.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.483+0000 7f52c9b9f740 1 --2- 172.21.15.12:0/2820738541 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d219f0 0x563749d2bc00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f52b0012040 tx=0x7f52b0003770 comp rx=0 tx=0).stop 2024-03-23T23:10:18.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.483+0000 7f52c9b9f740 1 -- 172.21.15.12:0/2820738541 shutdown_connections 2024-03-23T23:10:18.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.483+0000 7f52c9b9f740 1 --2- 172.21.15.12:0/2820738541 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563749d372e0 0x563749d396d0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.483+0000 7f52c9b9f740 1 --2- 172.21.15.12:0/2820738541 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563749d23580 0x563749d33180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.483+0000 7f52c9b9f740 1 --2- 172.21.15.12:0/2820738541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563749d22bc0 0x563749d2c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.483+0000 7f52c9b9f740 1 --2- 172.21.15.12:0/2820738541 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d219f0 0x563749d2bc00 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.483+0000 7f52c9b9f740 1 -- 172.21.15.12:0/2820738541 >> 172.21.15.12:0/2820738541 conn(0x563749d1cde0 msgr2=0x563749d1c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:18.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.483+0000 7f52aa000640 0 client.27670 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:18.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.484+0000 7f52c9b9f740 1 -- 172.21.15.12:0/2820738541 shutdown_connections 2024-03-23T23:10:18.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.484+0000 7f52c9b9f740 1 -- 172.21.15.12:0/2820738541 wait complete. 2024-03-23T23:10:18.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.484+0000 7f52c9b9f740 20 client.27670 trim_cache size 0 max 16384 2024-03-23T23:10:18.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.498+0000 7f52c9b9f740 1 Processor -- start 2024-03-23T23:10:18.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.498+0000 7f52c9b9f740 1 -- start start 2024-03-23T23:10:18.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.498+0000 7f52c9b9f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563749d219f0 0x563749d2c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.498+0000 7f52c9b9f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d22bc0 0x563749d33140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.498+0000 7f52c9b9f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563749d23580 0x563749d20750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.498+0000 7f52c9b9f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c01acfc0 con 0x563749d23580 2024-03-23T23:10:18.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.498+0000 7f52c9b9f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c01aaac0 con 0x563749d219f0 2024-03-23T23:10:18.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.498+0000 7f52c9b9f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c01a8b60 con 0x563749d22bc0 2024-03-23T23:10:18.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.498+0000 7f52bfe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563749d219f0 0x563749d2c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.498+0000 7f52bf400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d22bc0 0x563749d33140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.498+0000 7f52bfe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563749d219f0 0x563749d2c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48572/0 (socket says 172.21.15.12:48572) 2024-03-23T23:10:18.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.498+0000 7f52bfe00640 1 -- 172.21.15.12:0/1741642932 learned_addr learned my addr 172.21.15.12:0/1741642932 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:18.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.498+0000 7f52c4e00640 1 --2- 172.21.15.12:0/1741642932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563749d23580 0x563749d20750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.499+0000 7f52bf400640 1 -- 172.21.15.12:0/1741642932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563749d219f0 msgr2=0x563749d2c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:18.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.499+0000 7f52bf400640 1 --2- 172.21.15.12:0/1741642932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563749d219f0 0x563749d2c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.499+0000 7f52bf400640 1 -- 172.21.15.12:0/1741642932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563749d23580 msgr2=0x563749d20750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:18.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.499+0000 7f52bf400640 1 --2- 172.21.15.12:0/1741642932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563749d23580 0x563749d20750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.499+0000 7f52bf400640 1 -- 172.21.15.12:0/1741642932 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563749d22640 con 0x563749d22bc0 2024-03-23T23:10:18.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.499+0000 7f52bfe00640 1 --2- 172.21.15.12:0/1741642932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563749d219f0 0x563749d2c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:18.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.499+0000 7f52bf400640 1 --2- 172.21.15.12:0/1741642932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d22bc0 0x563749d33140 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f52ac036040 tx=0x7f52ac02df90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83ca58f6a4062d7b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:18.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.499+0000 7f52b4c00640 1 -- 172.21.15.12:0/1741642932 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac02cb00 con 0x563749d22bc0 2024-03-23T23:10:18.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.499+0000 7f52b4c00640 1 -- 172.21.15.12:0/1741642932 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52ac051030 con 0x563749d22bc0 2024-03-23T23:10:18.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.499+0000 7f52b4c00640 1 -- 172.21.15.12:0/1741642932 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac004eb0 con 0x563749d22bc0 2024-03-23T23:10:18.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.500+0000 7f52c9b9f740 1 -- 172.21.15.12:0/1741642932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d22bc0 msgr2=0x563749d33140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:18.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.500+0000 7f52c9b9f740 1 --2- 172.21.15.12:0/1741642932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d22bc0 0x563749d33140 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f52ac036040 tx=0x7f52ac02df90 comp rx=0 tx=0).stop 2024-03-23T23:10:18.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.500+0000 7f52c9b9f740 1 -- 172.21.15.12:0/1741642932 shutdown_connections 2024-03-23T23:10:18.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.500+0000 7f52c9b9f740 1 --2- 172.21.15.12:0/1741642932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563749d23580 0x563749d20750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.500+0000 7f52c9b9f740 1 --2- 172.21.15.12:0/1741642932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d22bc0 0x563749d33140 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.500+0000 7f52c9b9f740 1 --2- 172.21.15.12:0/1741642932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563749d219f0 0x563749d2c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.500+0000 7f52c9b9f740 1 -- 172.21.15.12:0/1741642932 >> 172.21.15.12:0/1741642932 conn(0x563749d1cde0 msgr2=0x563749d20040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:18.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.500+0000 7f52c9b9f740 1 -- 172.21.15.12:0/1741642932 shutdown_connections 2024-03-23T23:10:18.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.500+0000 7f52c9b9f740 1 -- 172.21.15.12:0/1741642932 wait complete. 2024-03-23T23:10:18.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.501+0000 7f52c9b9f740 1 -- start start 2024-03-23T23:10:18.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.501+0000 7f52c9b9f740 1 Processor -- start 2024-03-23T23:10:18.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.501+0000 7f52c9b9f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563749d219f0 0x563749d30b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.501+0000 7f52c9b9f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d22bc0 0x563749d31090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.501+0000 7f52c9b9f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563749d23580 0x563749d32030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.501+0000 7f52c9b9f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c01a9160 con 0x563749d219f0 2024-03-23T23:10:18.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.501+0000 7f52c9b9f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c01aa460 con 0x563749d23580 2024-03-23T23:10:18.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.501+0000 7f52c9b9f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c01a97a0 con 0x563749d22bc0 2024-03-23T23:10:18.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.501+0000 7f52c9b9f740 10 client.0 _ll_register_callbacks cb 0x7f52ac009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:18.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.502+0000 7f52bf400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d22bc0 0x563749d31090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.502+0000 7f52c4e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563749d23580 0x563749d32030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.502+0000 7f52bf400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d22bc0 0x563749d31090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:41352/0 (socket says 172.21.15.12:41352) 2024-03-23T23:10:18.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.502+0000 7f52bf400640 1 -- 172.21.15.12:0/534573999 learned_addr learned my addr 172.21.15.12:0/534573999 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:18.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.502+0000 7f52bfe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563749d219f0 0x563749d30b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.502+0000 7f52bf400640 1 -- 172.21.15.12:0/534573999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563749d23580 msgr2=0x563749d32030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:18.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.502+0000 7f52bf400640 1 --2- 172.21.15.12:0/534573999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563749d23580 0x563749d32030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.502+0000 7f52bf400640 1 -- 172.21.15.12:0/534573999 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563749d219f0 msgr2=0x563749d30b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:18.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.502+0000 7f52bf400640 1 --2- 172.21.15.12:0/534573999 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563749d219f0 0x563749d30b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.502+0000 7f52bf400640 1 -- 172.21.15.12:0/534573999 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563749d32620 con 0x563749d22bc0 2024-03-23T23:10:18.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.502+0000 7f52bf400640 1 --2- 172.21.15.12:0/534573999 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d22bc0 0x563749d31090 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f52ac04d480 tx=0x7f52ac004e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cfed81f2a43ab89 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:18.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.502+0000 7f52aa000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:10:18.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.502+0000 7f52aa000640 1 -- 172.21.15.12:0/534573999 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac0048c0 con 0x563749d22bc0 2024-03-23T23:10:18.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.503+0000 7f52c9b9f740 1 -- 172.21.15.12:0/534573999 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f52ac051030 con 0x563749d22bc0 2024-03-23T23:10:18.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.503+0000 7f52aa000640 1 -- 172.21.15.12:0/534573999 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f52ac004030 con 0x563749d22bc0 2024-03-23T23:10:18.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.503+0000 7f52aa000640 1 -- 172.21.15.12:0/534573999 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52ac067050 con 0x563749d22bc0 2024-03-23T23:10:18.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.503+0000 7f52c9b9f740 10 client.27676 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:18.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.503+0000 7f52c9b9f740 1 -- 172.21.15.12:0/534573999 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563749d92d90 con 0x563749d22bc0 2024-03-23T23:10:18.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.503+0000 7f52aa000640 1 -- 172.21.15.12:0/534573999 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52ac051030 con 0x563749d22bc0 2024-03-23T23:10:18.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.503+0000 7f52aa000640 1 -- 172.21.15.12:0/534573999 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f52ac04d690 con 0x563749d22bc0 2024-03-23T23:10:18.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.503+0000 7f52c9b9f740 10 client.27676 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:18.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.503+0000 7f52c9b9f740 10 client.27676 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:18.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.503+0000 7f52c9b9f740 20 client.27676 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:18.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.503+0000 7f52c9b9f740 1 --2- 172.21.15.12:0/534573999 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563749d93610 0x563749d95a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:18.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.503+0000 7f52c9b9f740 4 client.27676 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:10:18.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.503+0000 7f52c9b9f740 1 -- 172.21.15.12:0/534573999 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563749c8db50 con 0x563749d93610 2024-03-23T23:10:18.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.504+0000 7f52bfe00640 1 --2- 172.21.15.12:0/534573999 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563749d93610 0x563749d95a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:18.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.504+0000 7f52bfe00640 1 --2- 172.21.15.12:0/534573999 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563749d93610 0x563749d95a50 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62811eba0c0dbd0d server_cookie=7c64f7749fd22eaa in_seq=0 out_seq=0 2024-03-23T23:10:18.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.504+0000 7f52aa000640 10 client.27676 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:18.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.504+0000 7f52aa000640 1 -- 172.21.15.12:0/534573999 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563749c8db50 con 0x563749d93610 2024-03-23T23:10:18.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.504+0000 7f52aa000640 10 client.27676 handle_command_reply: tid=0 2024-03-23T23:10:18.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.504+0000 7f52c9b9f740 1 client.27676 shutdown 2024-03-23T23:10:18.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.504+0000 7f52c9b9f740 1 -- 172.21.15.12:0/534573999 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d22bc0 msgr2=0x563749d31090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:18.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.504+0000 7f52c9b9f740 1 --2- 172.21.15.12:0/534573999 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d22bc0 0x563749d31090 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f52ac04d480 tx=0x7f52ac004e20 comp rx=0 tx=0).stop 2024-03-23T23:10:18.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.504+0000 7f52c9b9f740 1 -- 172.21.15.12:0/534573999 shutdown_connections 2024-03-23T23:10:18.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.504+0000 7f52c9b9f740 1 --2- 172.21.15.12:0/534573999 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563749d93610 0x563749d95a50 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.504+0000 7f52c9b9f740 1 --2- 172.21.15.12:0/534573999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563749d23580 0x563749d32030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.504+0000 7f52c9b9f740 1 --2- 172.21.15.12:0/534573999 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563749d22bc0 0x563749d31090 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.504+0000 7f52c9b9f740 1 --2- 172.21.15.12:0/534573999 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563749d219f0 0x563749d30b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.504+0000 7f52c9b9f740 1 -- 172.21.15.12:0/534573999 >> 172.21.15.12:0/534573999 conn(0x563749d1cde0 msgr2=0x563749d208d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:18.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.504+0000 7f52aa000640 0 client.27676 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:18.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.505+0000 7f52c9b9f740 1 -- 172.21.15.12:0/534573999 shutdown_connections 2024-03-23T23:10:18.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.505+0000 7f52c9b9f740 1 -- 172.21.15.12:0/534573999 wait complete. 2024-03-23T23:10:18.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.505+0000 7f52c9b9f740 20 client.27676 trim_cache size 0 max 16384 2024-03-23T23:10:18.510 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:10:18.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.505+0000 7f52c5800640 1 -- 172.21.15.12:0/498738262 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f52a40a2800 msgr2=0x7f52a40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:18.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.505+0000 7f52c5800640 1 --2- 172.21.15.12:0/498738262 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f52a40a2800 0x7f52a40a4cc0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x563749d22ac0 tx=0x7f52b8009500 comp rx=0 tx=0).stop 2024-03-23T23:10:18.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.505+0000 7f52c5800640 1 -- 172.21.15.12:0/498738262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52c01037c0 msgr2=0x7f52c0199a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:18.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.505+0000 7f52c5800640 1 --2- 172.21.15.12:0/498738262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52c01037c0 0x7f52c0199a90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f52b000d770 tx=0x7f52b000dc30 comp rx=0 tx=0).stop 2024-03-23T23:10:18.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.505+0000 7f52c5800640 1 -- 172.21.15.12:0/498738262 shutdown_connections 2024-03-23T23:10:18.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.505+0000 7f52c5800640 1 --2- 172.21.15.12:0/498738262 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f52a40a2800 0x7f52a40a4cc0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.505+0000 7f52c5800640 1 --2- 172.21.15.12:0/498738262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52c0105370 0x7f52c01a1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.505+0000 7f52c5800640 1 --2- 172.21.15.12:0/498738262 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52c01049b0 0x7f52c0199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.505+0000 7f52c5800640 1 --2- 172.21.15.12:0/498738262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52c01037c0 0x7f52c0199a90 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:18.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.505+0000 7f52c5800640 1 -- 172.21.15.12:0/498738262 >> 172.21.15.12:0/498738262 conn(0x7f52c00fed50 msgr2=0x7f52c0107c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:18.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.506+0000 7f52c5800640 1 -- 172.21.15.12:0/498738262 shutdown_connections 2024-03-23T23:10:18.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:18.506+0000 7f52c5800640 1 -- 172.21.15.12:0/498738262 wait complete. 2024-03-23T23:10:18.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:18 smithi012 ceph-mon[31695]: pgmap v645: 97 pgs: 97 active+clean; 1.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:10:18.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:18 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2428035030' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:10:18.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:18 smithi173 ceph-mon[36116]: pgmap v645: 97 pgs: 97 active+clean; 1.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:10:18.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:18 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2428035030' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:10:18.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:18 smithi203 ceph-mon[36693]: pgmap v645: 97 pgs: 97 active+clean; 1.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:10:18.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:18 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2428035030' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:10:19.524 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-23T23:10:19.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:10:19.525 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-23T23:10:19.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.763+0000 7ff0db200640 1 -- 172.21.15.12:0/1469649557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0dc109770 msgr2=0x7ff0dc110000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:19.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.763+0000 7ff0db200640 1 --2- 172.21.15.12:0/1469649557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0dc109770 0x7ff0dc110000 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4009ca0 tx=0x7ff0d402c100 comp rx=0 tx=0).stop 2024-03-23T23:10:19.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.763+0000 7ff0db200640 1 -- 172.21.15.12:0/1469649557 shutdown_connections 2024-03-23T23:10:19.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.763+0000 7ff0db200640 1 --2- 172.21.15.12:0/1469649557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0dc109770 0x7ff0dc110000 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.763+0000 7ff0db200640 1 --2- 172.21.15.12:0/1469649557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0dc108db0 0x7ff0dc109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.763+0000 7ff0db200640 1 --2- 172.21.15.12:0/1469649557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0dc107bc0 0x7ff0dc107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.763+0000 7ff0db200640 1 -- 172.21.15.12:0/1469649557 >> 172.21.15.12:0/1469649557 conn(0x7ff0dc076b70 msgr2=0x7ff0dc078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:19.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.763+0000 7ff0db200640 1 -- 172.21.15.12:0/1469649557 shutdown_connections 2024-03-23T23:10:19.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.763+0000 7ff0db200640 1 -- 172.21.15.12:0/1469649557 wait complete. 2024-03-23T23:10:19.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.764+0000 7ff0db200640 1 Processor -- start 2024-03-23T23:10:19.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.764+0000 7ff0db200640 1 -- start start 2024-03-23T23:10:19.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.764+0000 7ff0db200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0dc107bc0 0x7ff0dc19de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.764+0000 7ff0db200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0dc108db0 0x7ff0dc19e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.764+0000 7ff0db200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0dc109770 0x7ff0dc1a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.764+0000 7ff0db200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc113840 con 0x7ff0dc109770 2024-03-23T23:10:19.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.764+0000 7ff0db200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc1136c0 con 0x7ff0dc108db0 2024-03-23T23:10:19.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.764+0000 7ff0db200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc1139c0 con 0x7ff0dc107bc0 2024-03-23T23:10:19.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.765+0000 7ff0d3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0dc108db0 0x7ff0dc19e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.765+0000 7ff0d3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0dc108db0 0x7ff0dc19e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48600/0 (socket says 172.21.15.12:48600) 2024-03-23T23:10:19.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.765+0000 7ff0d3e00640 1 -- 172.21.15.12:0/971303294 learned_addr learned my addr 172.21.15.12:0/971303294 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:19.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.765+0000 7ff0da800640 1 --2- 172.21.15.12:0/971303294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0dc109770 0x7ff0dc1a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.765+0000 7ff0d9e00640 1 --2- 172.21.15.12:0/971303294 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0dc107bc0 0x7ff0dc19de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.765+0000 7ff0d3e00640 1 -- 172.21.15.12:0/971303294 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0dc107bc0 msgr2=0x7ff0dc19de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:19.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.765+0000 7ff0d3e00640 1 --2- 172.21.15.12:0/971303294 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0dc107bc0 0x7ff0dc19de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.765+0000 7ff0d3e00640 1 -- 172.21.15.12:0/971303294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0dc109770 msgr2=0x7ff0dc1a5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:19.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.765+0000 7ff0d3e00640 1 --2- 172.21.15.12:0/971303294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0dc109770 0x7ff0dc1a5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.765+0000 7ff0d3e00640 1 -- 172.21.15.12:0/971303294 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0dc1a5a70 con 0x7ff0dc108db0 2024-03-23T23:10:19.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.765+0000 7ff0d3e00640 1 --2- 172.21.15.12:0/971303294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0dc108db0 0x7ff0dc19e390 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7ff0c400e9f0 tx=0x7ff0c400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:19.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.766+0000 7ff0d2a00640 1 -- 172.21.15.12:0/971303294 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c400cd80 con 0x7ff0dc108db0 2024-03-23T23:10:19.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.766+0000 7ff0d2a00640 1 -- 172.21.15.12:0/971303294 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0c4010910 con 0x7ff0dc108db0 2024-03-23T23:10:19.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.766+0000 7ff0db200640 1 -- 172.21.15.12:0/971303294 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0dc1a5d60 con 0x7ff0dc108db0 2024-03-23T23:10:19.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.766+0000 7ff0d2a00640 1 -- 172.21.15.12:0/971303294 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c4018990 con 0x7ff0dc108db0 2024-03-23T23:10:19.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.766+0000 7ff0db200640 1 -- 172.21.15.12:0/971303294 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0dc1a6270 con 0x7ff0dc108db0 2024-03-23T23:10:19.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.769+0000 7ff0d2a00640 1 -- 172.21.15.12:0/971303294 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff0c4010ac0 con 0x7ff0dc108db0 2024-03-23T23:10:19.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.769+0000 7ff0e3603740 1 Processor -- start 2024-03-23T23:10:19.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.769+0000 7ff0e3603740 1 -- start start 2024-03-23T23:10:19.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.770+0000 7ff0e3603740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487d72b0 0x55f8487d76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.770+0000 7ff0e3603740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f8487d7ca0 0x55f8487dc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.770+0000 7ff0e3603740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487dce10 0x55f8487df2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.770+0000 7ff0e3603740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f84873ceb0 con 0x55f8487d7ca0 2024-03-23T23:10:19.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.770+0000 7ff0e3603740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8486e5a60 con 0x55f8487dce10 2024-03-23T23:10:19.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.770+0000 7ff0e3603740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f848738f50 con 0x55f8487d72b0 2024-03-23T23:10:19.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.770+0000 7ff0d3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487dce10 0x55f8487df2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.770+0000 7ff0da800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487d72b0 0x55f8487d76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.770+0000 7ff0da800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487d72b0 0x55f8487d76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:41390/0 (socket says 172.21.15.12:41390) 2024-03-23T23:10:19.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.770+0000 7ff0d3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487dce10 0x55f8487df2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48602/0 (socket says 172.21.15.12:48602) 2024-03-23T23:10:19.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.770+0000 7ff0da800640 1 -- 172.21.15.12:0/740036735 learned_addr learned my addr 172.21.15.12:0/740036735 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:19.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.770+0000 7ff0d9e00640 1 --2- 172.21.15.12:0/740036735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f8487d7ca0 0x55f8487dc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.770+0000 7ff0da800640 1 -- 172.21.15.12:0/740036735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487dce10 msgr2=0x55f8487df2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:19.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.770+0000 7ff0da800640 1 --2- 172.21.15.12:0/740036735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487dce10 0x55f8487df2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.770+0000 7ff0da800640 1 -- 172.21.15.12:0/740036735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f8487d7ca0 msgr2=0x55f8487dc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:19.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.770+0000 7ff0da800640 1 --2- 172.21.15.12:0/740036735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f8487d7ca0 0x55f8487dc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.770+0000 7ff0da800640 1 -- 172.21.15.12:0/740036735 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f8485197e0 con 0x55f8487d72b0 2024-03-23T23:10:19.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.771+0000 7ff0d3e00640 1 --2- 172.21.15.12:0/740036735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487dce10 0x55f8487df2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:19.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.771+0000 7ff0da800640 1 --2- 172.21.15.12:0/740036735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487d72b0 0x55f8487d76d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff0d402c710 tx=0x7ff0d40323d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5f7249af12fa3d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:19.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.771+0000 7ff0d2a00640 1 --2- 172.21.15.12:0/971303294 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff0c00a2880 0x7ff0c00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.771+0000 7ff0d2a00640 1 -- 172.21.15.12:0/971303294 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff0c40ced30 con 0x7ff0dc108db0 2024-03-23T23:10:19.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.771+0000 7ff0cb000640 1 -- 172.21.15.12:0/740036735 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d4043070 con 0x55f8487d72b0 2024-03-23T23:10:19.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.771+0000 7ff0cb000640 1 -- 172.21.15.12:0/740036735 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0d4032c90 con 0x55f8487d72b0 2024-03-23T23:10:19.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.771+0000 7ff0cb000640 1 -- 172.21.15.12:0/740036735 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d4039040 con 0x55f8487d72b0 2024-03-23T23:10:19.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.771+0000 7ff0d9e00640 1 --2- 172.21.15.12:0/971303294 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff0c00a2880 0x7ff0c00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.772+0000 7ff0e3603740 1 -- 172.21.15.12:0/740036735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487d72b0 msgr2=0x55f8487d76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:19.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.772+0000 7ff0e3603740 1 --2- 172.21.15.12:0/740036735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487d72b0 0x55f8487d76d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff0d402c710 tx=0x7ff0d40323d0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.772+0000 7ff0d9e00640 1 --2- 172.21.15.12:0/971303294 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff0c00a2880 0x7ff0c00a4d40 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x55f8487d80a0 tx=0x7ff0cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:19.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.772+0000 7ff0e3603740 1 -- 172.21.15.12:0/740036735 shutdown_connections 2024-03-23T23:10:19.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.772+0000 7ff0e3603740 1 --2- 172.21.15.12:0/740036735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487dce10 0x55f8487df2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.772+0000 7ff0e3603740 1 --2- 172.21.15.12:0/740036735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f8487d7ca0 0x55f8487dc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.772+0000 7ff0e3603740 1 --2- 172.21.15.12:0/740036735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487d72b0 0x55f8487d76d0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.772+0000 7ff0e3603740 1 -- 172.21.15.12:0/740036735 >> 172.21.15.12:0/740036735 conn(0x55f8487d2820 msgr2=0x55f8487d2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:19.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.772+0000 7ff0e3603740 1 -- 172.21.15.12:0/740036735 shutdown_connections 2024-03-23T23:10:19.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.772+0000 7ff0e3603740 1 -- 172.21.15.12:0/740036735 wait complete. 2024-03-23T23:10:19.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.773+0000 7ff0e3603740 1 -- start start 2024-03-23T23:10:19.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.773+0000 7ff0e3603740 1 Processor -- start 2024-03-23T23:10:19.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.773+0000 7ff0e3603740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487d72b0 0x55f8487e1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.773+0000 7ff0e3603740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f8487d7ca0 0x55f8487e19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.773+0000 7ff0e3603740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487dce10 0x55f8487e8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.773+0000 7ff0e3603740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8487dfd60 con 0x55f8487d7ca0 2024-03-23T23:10:19.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.773+0000 7ff0e3603740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8486f8eb0 con 0x55f8487d72b0 2024-03-23T23:10:19.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.773+0000 7ff0e3603740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8487dfee0 con 0x55f8487dce10 2024-03-23T23:10:19.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.773+0000 7ff0e3603740 10 client.0 _ll_register_callbacks cb 0x55f8486ed150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:19.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.774+0000 7ff0d3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f8487d7ca0 0x55f8487e19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.774+0000 7ff0d9e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487d72b0 0x55f8487e1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.774+0000 7ff0da800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487dce10 0x55f8487e8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.774+0000 7ff0d9e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487d72b0 0x55f8487e1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48614/0 (socket says 172.21.15.12:48614) 2024-03-23T23:10:19.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.774+0000 7ff0da800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487dce10 0x55f8487e8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:41406/0 (socket says 172.21.15.12:41406) 2024-03-23T23:10:19.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.774+0000 7ff0d9e00640 1 -- 172.21.15.12:0/112106150 learned_addr learned my addr 172.21.15.12:0/112106150 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:19.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.774+0000 7ff0d3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f8487d7ca0 0x55f8487e19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49342/0 (socket says 172.21.15.12:49342) 2024-03-23T23:10:19.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.774+0000 7ff0d9e00640 1 -- 172.21.15.12:0/112106150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487dce10 msgr2=0x55f8487e8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:19.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.774+0000 7ff0d9e00640 1 --2- 172.21.15.12:0/112106150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487dce10 0x55f8487e8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.774+0000 7ff0d9e00640 1 -- 172.21.15.12:0/112106150 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f8487d7ca0 msgr2=0x55f8487e19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:19.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.774+0000 7ff0d9e00640 1 --2- 172.21.15.12:0/112106150 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f8487d7ca0 0x55f8487e19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.774+0000 7ff0d9e00640 1 -- 172.21.15.12:0/112106150 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f848579f70 con 0x55f8487d72b0 2024-03-23T23:10:19.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.774+0000 7ff0da800640 1 --2- 172.21.15.12:0/112106150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487dce10 0x55f8487e8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:19.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.774+0000 7ff0d9e00640 1 --2- 172.21.15.12:0/112106150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487d72b0 0x55f8487e1490 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc004620 tx=0x7ff0cc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42ee28e6b7c71161 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:19.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.774+0000 7ff0bf400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:10:19.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.775+0000 7ff0bf400640 1 -- 172.21.15.12:0/112106150 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc01f630 con 0x55f8487d72b0 2024-03-23T23:10:19.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.775+0000 7ff0bf400640 1 -- 172.21.15.12:0/112106150 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff0cc01f7d0 con 0x55f8487d72b0 2024-03-23T23:10:19.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.775+0000 7ff0e3603740 1 -- 172.21.15.12:0/112106150 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff0d4032c90 con 0x55f8487d72b0 2024-03-23T23:10:19.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.775+0000 7ff0bf400640 1 -- 172.21.15.12:0/112106150 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0cc01f9d0 con 0x55f8487d72b0 2024-03-23T23:10:19.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.775+0000 7ff0e3603740 10 client.27728 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:19.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.775+0000 7ff0e3603740 1 -- 172.21.15.12:0/112106150 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f848411170 con 0x55f8487d72b0 2024-03-23T23:10:19.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.775+0000 7ff0bf400640 1 -- 172.21.15.12:0/112106150 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0d4032c90 con 0x55f8487d72b0 2024-03-23T23:10:19.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.775+0000 7ff0bf400640 1 -- 172.21.15.12:0/112106150 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ff0cc027070 con 0x55f8487d72b0 2024-03-23T23:10:19.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.775+0000 7ff0e3603740 10 client.27728 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:19.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.775+0000 7ff0e3603740 10 client.27728 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:19.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.775+0000 7ff0e3603740 20 client.27728 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:19.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.775+0000 7ff0e3603740 1 --2- 172.21.15.12:0/112106150 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f8487ed290 0x55f8487ef6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.775+0000 7ff0e3603740 4 client.27728 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:10:19.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.775+0000 7ff0e3603740 1 -- 172.21.15.12:0/112106150 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f84874a3f0 con 0x55f8487ed290 2024-03-23T23:10:19.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.776+0000 7ff0d3e00640 1 --2- 172.21.15.12:0/112106150 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f8487ed290 0x55f8487ef6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.776+0000 7ff0d3e00640 1 --2- 172.21.15.12:0/112106150 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f8487ed290 0x55f8487ef6f0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a71220f892430f1 server_cookie=9c44c8355e57abcd in_seq=0 out_seq=0 2024-03-23T23:10:19.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.776+0000 7ff0bf400640 10 client.27728 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:19.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.777+0000 7ff0bf400640 1 -- 172.21.15.12:0/112106150 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f84874a3f0 con 0x55f8487ed290 2024-03-23T23:10:19.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.777+0000 7ff0bf400640 10 client.27728 handle_command_reply: tid=0 2024-03-23T23:10:19.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.777+0000 7ff0e3603740 1 client.27728 shutdown 2024-03-23T23:10:19.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.778+0000 7ff0e3603740 1 -- 172.21.15.12:0/112106150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487d72b0 msgr2=0x55f8487e1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:19.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.778+0000 7ff0e3603740 1 --2- 172.21.15.12:0/112106150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487d72b0 0x55f8487e1490 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc004620 tx=0x7ff0cc004730 comp rx=0 tx=0).stop 2024-03-23T23:10:19.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.778+0000 7ff0e3603740 1 -- 172.21.15.12:0/112106150 shutdown_connections 2024-03-23T23:10:19.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.778+0000 7ff0e3603740 1 --2- 172.21.15.12:0/112106150 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f8487ed290 0x55f8487ef6f0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.778+0000 7ff0e3603740 1 --2- 172.21.15.12:0/112106150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487dce10 0x55f8487e8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.778+0000 7ff0e3603740 1 --2- 172.21.15.12:0/112106150 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f8487d7ca0 0x55f8487e19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.778+0000 7ff0bf400640 0 client.27728 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:19.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.778+0000 7ff0e3603740 1 --2- 172.21.15.12:0/112106150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487d72b0 0x55f8487e1490 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.778+0000 7ff0e3603740 1 -- 172.21.15.12:0/112106150 >> 172.21.15.12:0/112106150 conn(0x55f8487d2820 msgr2=0x55f8487d1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:19.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.778+0000 7ff0e3603740 1 -- 172.21.15.12:0/112106150 shutdown_connections 2024-03-23T23:10:19.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.778+0000 7ff0e3603740 1 -- 172.21.15.12:0/112106150 wait complete. 2024-03-23T23:10:19.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.778+0000 7ff0e3603740 20 client.27728 trim_cache size 0 max 16384 2024-03-23T23:10:19.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.796+0000 7ff0e3603740 1 Processor -- start 2024-03-23T23:10:19.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.796+0000 7ff0e3603740 1 -- start start 2024-03-23T23:10:19.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.796+0000 7ff0e3603740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f8487d72b0 0x55f8487f3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.796+0000 7ff0e3603740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487d7ca0 0x55f8487e89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.796+0000 7ff0e3603740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487dce10 0x55f8487f44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.796+0000 7ff0e3603740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc1b1300 con 0x55f8487d72b0 2024-03-23T23:10:19.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.796+0000 7ff0e3603740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc1aee00 con 0x55f8487dce10 2024-03-23T23:10:19.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.796+0000 7ff0e3603740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc1acea0 con 0x55f8487d7ca0 2024-03-23T23:10:19.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.797+0000 7ff0d3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487d7ca0 0x55f8487e89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.797+0000 7ff0d3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487d7ca0 0x55f8487e89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:41422/0 (socket says 172.21.15.12:41422) 2024-03-23T23:10:19.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.797+0000 7ff0d3e00640 1 -- 172.21.15.12:0/3231408596 learned_addr learned my addr 172.21.15.12:0/3231408596 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:19.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.797+0000 7ff0d9e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f8487d72b0 0x55f8487f3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.797+0000 7ff0da800640 1 --2- 172.21.15.12:0/3231408596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487dce10 0x55f8487f44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.797+0000 7ff0d3e00640 1 -- 172.21.15.12:0/3231408596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487dce10 msgr2=0x55f8487f44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:19.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.797+0000 7ff0d3e00640 1 --2- 172.21.15.12:0/3231408596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487dce10 0x55f8487f44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.797+0000 7ff0d3e00640 1 -- 172.21.15.12:0/3231408596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f8487d72b0 msgr2=0x55f8487f3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:19.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.797+0000 7ff0d3e00640 1 --2- 172.21.15.12:0/3231408596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f8487d72b0 0x55f8487f3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.797+0000 7ff0d3e00640 1 -- 172.21.15.12:0/3231408596 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f8487e1db0 con 0x55f8487d7ca0 2024-03-23T23:10:19.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.797+0000 7ff0d3e00640 1 --2- 172.21.15.12:0/3231408596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487d7ca0 0x55f8487e89d0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff0c409b430 tx=0x7ff0c4086040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1a9d18de155528a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:19.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.798+0000 7ff0ca600640 1 -- 172.21.15.12:0/3231408596 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c409b090 con 0x55f8487d7ca0 2024-03-23T23:10:19.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.798+0000 7ff0ca600640 1 -- 172.21.15.12:0/3231408596 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0c40bb040 con 0x55f8487d7ca0 2024-03-23T23:10:19.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.798+0000 7ff0ca600640 1 -- 172.21.15.12:0/3231408596 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c4097da0 con 0x55f8487d7ca0 2024-03-23T23:10:19.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.798+0000 7ff0e3603740 1 -- 172.21.15.12:0/3231408596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487d7ca0 msgr2=0x55f8487e89d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:19.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.798+0000 7ff0e3603740 1 --2- 172.21.15.12:0/3231408596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487d7ca0 0x55f8487e89d0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff0c409b430 tx=0x7ff0c4086040 comp rx=0 tx=0).stop 2024-03-23T23:10:19.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.798+0000 7ff0e3603740 1 -- 172.21.15.12:0/3231408596 shutdown_connections 2024-03-23T23:10:19.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.798+0000 7ff0e3603740 1 --2- 172.21.15.12:0/3231408596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487dce10 0x55f8487f44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.798+0000 7ff0e3603740 1 --2- 172.21.15.12:0/3231408596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487d7ca0 0x55f8487e89d0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.798+0000 7ff0e3603740 1 --2- 172.21.15.12:0/3231408596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f8487d72b0 0x55f8487f3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.798+0000 7ff0e3603740 1 -- 172.21.15.12:0/3231408596 >> 172.21.15.12:0/3231408596 conn(0x55f8487d2820 msgr2=0x55f8487d6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:19.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.799+0000 7ff0e3603740 1 -- 172.21.15.12:0/3231408596 shutdown_connections 2024-03-23T23:10:19.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.799+0000 7ff0e3603740 1 -- 172.21.15.12:0/3231408596 wait complete. 2024-03-23T23:10:19.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.799+0000 7ff0e3603740 1 -- start start 2024-03-23T23:10:19.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.799+0000 7ff0e3603740 1 Processor -- start 2024-03-23T23:10:19.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.800+0000 7ff0e3603740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f8487d72b0 0x55f8487e68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.800+0000 7ff0e3603740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487d7ca0 0x55f8487e6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.800+0000 7ff0e3603740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487dce10 0x55f8487e7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.800+0000 7ff0e3603740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc1ad4a0 con 0x55f8487d72b0 2024-03-23T23:10:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.800+0000 7ff0e3603740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc1ae7a0 con 0x55f8487d7ca0 2024-03-23T23:10:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.800+0000 7ff0e3603740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc1adae0 con 0x55f8487dce10 2024-03-23T23:10:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.800+0000 7ff0e3603740 10 client.0 _ll_register_callbacks cb 0x7ff0cc009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.800+0000 7ff0d3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487d7ca0 0x55f8487e6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.800+0000 7ff0d9e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f8487d72b0 0x55f8487e68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.800+0000 7ff0d3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487d7ca0 0x55f8487e6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48632/0 (socket says 172.21.15.12:48632) 2024-03-23T23:10:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.800+0000 7ff0d3e00640 1 -- 172.21.15.12:0/2126653341 learned_addr learned my addr 172.21.15.12:0/2126653341 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.800+0000 7ff0da800640 1 --2- 172.21.15.12:0/2126653341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487dce10 0x55f8487e7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.800+0000 7ff0d3e00640 1 -- 172.21.15.12:0/2126653341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487dce10 msgr2=0x55f8487e7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.800+0000 7ff0d3e00640 1 --2- 172.21.15.12:0/2126653341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487dce10 0x55f8487e7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.800+0000 7ff0d3e00640 1 -- 172.21.15.12:0/2126653341 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f8487d72b0 msgr2=0x55f8487e68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:19.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.800+0000 7ff0d3e00640 1 --2- 172.21.15.12:0/2126653341 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f8487d72b0 0x55f8487e68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.800+0000 7ff0d3e00640 1 -- 172.21.15.12:0/2126653341 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f8487e8430 con 0x55f8487d7ca0 2024-03-23T23:10:19.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.801+0000 7ff0d3e00640 1 --2- 172.21.15.12:0/2126653341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487d7ca0 0x55f8487e6e30 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff0c400c6a0 tx=0x7ff0c4018420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1613792c58db0c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:19.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.801+0000 7ff0bf400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:10:19.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.801+0000 7ff0bf400640 1 -- 172.21.15.12:0/2126653341 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c409b090 con 0x55f8487d7ca0 2024-03-23T23:10:19.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.801+0000 7ff0bf400640 1 -- 172.21.15.12:0/2126653341 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff0c40a3990 con 0x55f8487d7ca0 2024-03-23T23:10:19.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.801+0000 7ff0e3603740 1 -- 172.21.15.12:0/2126653341 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff0c40bb040 con 0x55f8487d7ca0 2024-03-23T23:10:19.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.801+0000 7ff0bf400640 1 -- 172.21.15.12:0/2126653341 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0c409b730 con 0x55f8487d7ca0 2024-03-23T23:10:19.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.801+0000 7ff0e3603740 10 client.27734 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:19.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.801+0000 7ff0e3603740 1 -- 172.21.15.12:0/2126653341 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f8488493d0 con 0x55f8487d7ca0 2024-03-23T23:10:19.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.801+0000 7ff0bf400640 1 -- 172.21.15.12:0/2126653341 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0c40bb040 con 0x55f8487d7ca0 2024-03-23T23:10:19.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.801+0000 7ff0bf400640 1 -- 172.21.15.12:0/2126653341 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ff0c409ab80 con 0x55f8487d7ca0 2024-03-23T23:10:19.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.801+0000 7ff0e3603740 10 client.27734 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:19.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.801+0000 7ff0e3603740 10 client.27734 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:19.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.801+0000 7ff0e3603740 20 client.27734 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:19.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.801+0000 7ff0e3603740 1 --2- 172.21.15.12:0/2126653341 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f848849d50 0x55f84884c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.801+0000 7ff0e3603740 4 client.27734 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-23T23:10:19.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.801+0000 7ff0e3603740 1 -- 172.21.15.12:0/2126653341 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f848744740 con 0x55f848849d50 2024-03-23T23:10:19.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.802+0000 7ff0d9e00640 1 --2- 172.21.15.12:0/2126653341 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f848849d50 0x55f84884c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.802+0000 7ff0d9e00640 1 --2- 172.21.15.12:0/2126653341 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f848849d50 0x55f84884c170 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83d8526a4eb6636c server_cookie=9038a8aa2eb21e9e in_seq=0 out_seq=0 2024-03-23T23:10:19.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.802+0000 7ff0bf400640 10 client.27734 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:19.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.803+0000 7ff0bf400640 1 -- 172.21.15.12:0/2126653341 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f848744740 con 0x55f848849d50 2024-03-23T23:10:19.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.803+0000 7ff0bf400640 10 client.27734 handle_command_reply: tid=0 2024-03-23T23:10:19.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.803+0000 7ff0e3603740 1 client.27734 shutdown 2024-03-23T23:10:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.803+0000 7ff0e3603740 1 -- 172.21.15.12:0/2126653341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487d7ca0 msgr2=0x55f8487e6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.803+0000 7ff0e3603740 1 --2- 172.21.15.12:0/2126653341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487d7ca0 0x55f8487e6e30 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff0c400c6a0 tx=0x7ff0c4018420 comp rx=0 tx=0).stop 2024-03-23T23:10:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.803+0000 7ff0e3603740 1 -- 172.21.15.12:0/2126653341 shutdown_connections 2024-03-23T23:10:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.803+0000 7ff0e3603740 1 --2- 172.21.15.12:0/2126653341 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f848849d50 0x55f84884c170 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.803+0000 7ff0e3603740 1 --2- 172.21.15.12:0/2126653341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f8487dce10 0x55f8487e7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.803+0000 7ff0e3603740 1 --2- 172.21.15.12:0/2126653341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f8487d7ca0 0x55f8487e6e30 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.803+0000 7ff0e3603740 1 --2- 172.21.15.12:0/2126653341 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f8487d72b0 0x55f8487e68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.803+0000 7ff0bf400640 0 client.27734 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.803+0000 7ff0e3603740 1 -- 172.21.15.12:0/2126653341 >> 172.21.15.12:0/2126653341 conn(0x55f8487d2820 msgr2=0x55f8487d11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.804+0000 7ff0e3603740 1 -- 172.21.15.12:0/2126653341 shutdown_connections 2024-03-23T23:10:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.804+0000 7ff0e3603740 1 -- 172.21.15.12:0/2126653341 wait complete. 2024-03-23T23:10:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.804+0000 7ff0e3603740 20 client.27734 trim_cache size 0 max 16384 2024-03-23T23:10:19.808 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:10:19.808 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-03-23T23:10:19.808 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "98c681b7-90f9-4d43-b9a9-e0e1f0eb0995", 2024-03-23T23:10:19.808 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-03-23T23:10:19.809 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:10:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.804+0000 7ff0db200640 1 -- 172.21.15.12:0/971303294 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff0c00a2880 msgr2=0x7ff0c00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.804+0000 7ff0db200640 1 --2- 172.21.15.12:0/971303294 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff0c00a2880 0x7ff0c00a4d40 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x55f8487d80a0 tx=0x7ff0cc005f90 comp rx=0 tx=0).stop 2024-03-23T23:10:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.804+0000 7ff0db200640 1 -- 172.21.15.12:0/971303294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0dc108db0 msgr2=0x7ff0dc19e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.804+0000 7ff0db200640 1 --2- 172.21.15.12:0/971303294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0dc108db0 0x7ff0dc19e390 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7ff0c400e9f0 tx=0x7ff0c400eeb0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.804+0000 7ff0db200640 1 -- 172.21.15.12:0/971303294 shutdown_connections 2024-03-23T23:10:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.805+0000 7ff0db200640 1 --2- 172.21.15.12:0/971303294 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff0c00a2880 0x7ff0c00a4d40 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.805+0000 7ff0db200640 1 --2- 172.21.15.12:0/971303294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0dc109770 0x7ff0dc1a5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.805+0000 7ff0db200640 1 --2- 172.21.15.12:0/971303294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0dc108db0 0x7ff0dc19e390 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.805+0000 7ff0db200640 1 --2- 172.21.15.12:0/971303294 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0dc107bc0 0x7ff0dc19de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.805+0000 7ff0db200640 1 -- 172.21.15.12:0/971303294 >> 172.21.15.12:0/971303294 conn(0x7ff0dc076b70 msgr2=0x7ff0dc10c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.805+0000 7ff0db200640 1 -- 172.21.15.12:0/971303294 shutdown_connections 2024-03-23T23:10:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:19.805+0000 7ff0db200640 1 -- 172.21.15.12:0/971303294 wait complete. 2024-03-23T23:10:19.821 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:10:19.822 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:10:19.938 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.938+0000 7fe1f3200640 1 -- 172.21.15.12:0/986545819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1ec102140 msgr2=0x7fe1ec104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.938+0000 7fe1f3200640 1 --2- 172.21.15.12:0/986545819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1ec102140 0x7fe1ec104530 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc009900 tx=0x7fe1dc02bc30 comp rx=0 tx=0).stop 2024-03-23T23:10:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.938+0000 7fe1f3200640 1 -- 172.21.15.12:0/986545819 shutdown_connections 2024-03-23T23:10:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.938+0000 7fe1f3200640 1 --2- 172.21.15.12:0/986545819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1ec1073a0 0x7fe1ec1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.938+0000 7fe1f3200640 1 --2- 172.21.15.12:0/986545819 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1ec104a70 0x7fe1ec106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.938+0000 7fe1f3200640 1 --2- 172.21.15.12:0/986545819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1ec102140 0x7fe1ec104530 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.938+0000 7fe1f3200640 1 -- 172.21.15.12:0/986545819 >> 172.21.15.12:0/986545819 conn(0x7fe1ec0fbb30 msgr2=0x7fe1ec0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.939+0000 7fe1f3200640 1 -- 172.21.15.12:0/986545819 shutdown_connections 2024-03-23T23:10:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.939+0000 7fe1f3200640 1 -- 172.21.15.12:0/986545819 wait complete. 2024-03-23T23:10:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.940+0000 7fe1f3200640 1 Processor -- start 2024-03-23T23:10:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.940+0000 7fe1f3200640 1 -- start start 2024-03-23T23:10:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.940+0000 7fe1f3200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1ec102140 0x7fe1ec197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.940+0000 7fe1f3200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1ec104a70 0x7fe1ec197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.940+0000 7fe1f3200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1ec1073a0 0x7fe1ec19eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.940+0000 7fe1f3200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec10d390 con 0x7fe1ec102140 2024-03-23T23:10:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.940+0000 7fe1f3200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec10d210 con 0x7fe1ec104a70 2024-03-23T23:10:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.940+0000 7fe1f3200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec10d510 con 0x7fe1ec1073a0 2024-03-23T23:10:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.940+0000 7fe1ebe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1ec102140 0x7fe1ec197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.940+0000 7fe1f1200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1ec1073a0 0x7fe1ec19eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.940+0000 7fe1e3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1ec104a70 0x7fe1ec197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.940+0000 7fe1f1200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1ec1073a0 0x7fe1ec19eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:41454/0 (socket says 172.21.15.12:41454) 2024-03-23T23:10:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.940+0000 7fe1ebe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1ec102140 0x7fe1ec197940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49384/0 (socket says 172.21.15.12:49384) 2024-03-23T23:10:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.940+0000 7fe1f1200640 1 -- 172.21.15.12:0/3586410678 learned_addr learned my addr 172.21.15.12:0/3586410678 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.941+0000 7fe1f1200640 1 -- 172.21.15.12:0/3586410678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1ec104a70 msgr2=0x7fe1ec197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.941+0000 7fe1f1200640 1 --2- 172.21.15.12:0/3586410678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1ec104a70 0x7fe1ec197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.941+0000 7fe1f1200640 1 -- 172.21.15.12:0/3586410678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1ec102140 msgr2=0x7fe1ec197940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.941+0000 7fe1f1200640 1 --2- 172.21.15.12:0/3586410678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1ec102140 0x7fe1ec197940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.941+0000 7fe1f1200640 1 -- 172.21.15.12:0/3586410678 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1ec19f560 con 0x7fe1ec1073a0 2024-03-23T23:10:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.941+0000 7fe1e3400640 1 --2- 172.21.15.12:0/3586410678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1ec104a70 0x7fe1ec197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.941+0000 7fe1f1200640 1 --2- 172.21.15.12:0/3586410678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1ec1073a0 0x7fe1ec19eef0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fe1e400e970 tx=0x7fe1e400ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.941+0000 7fe1e9600640 1 -- 172.21.15.12:0/3586410678 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e400ccc0 con 0x7fe1ec1073a0 2024-03-23T23:10:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.942+0000 7fe1e9600640 1 -- 172.21.15.12:0/3586410678 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe1e400ce60 con 0x7fe1ec1073a0 2024-03-23T23:10:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.942+0000 7fe1f3200640 1 -- 172.21.15.12:0/3586410678 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1ec19f850 con 0x7fe1ec1073a0 2024-03-23T23:10:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.942+0000 7fe1e9600640 1 -- 172.21.15.12:0/3586410678 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e4010490 con 0x7fe1ec1073a0 2024-03-23T23:10:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.942+0000 7fe1f3200640 1 -- 172.21.15.12:0/3586410678 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1ec19fda0 con 0x7fe1ec1073a0 2024-03-23T23:10:19.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.945+0000 7fe1e9600640 1 -- 172.21.15.12:0/3586410678 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe1e4005020 con 0x7fe1ec1073a0 2024-03-23T23:10:19.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.945+0000 7fe1f6026740 1 Processor -- start 2024-03-23T23:10:19.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.945+0000 7fe1f6026740 1 -- start start 2024-03-23T23:10:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.946+0000 7fe1f6026740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56207d9065f0 0x56207d906a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.946+0000 7fe1f6026740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d906fe0 0x56207d90bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.946+0000 7fe1f6026740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56207d90c150 0x56207d90e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.946+0000 7fe1f6026740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56207d8c53b0 con 0x56207d9065f0 2024-03-23T23:10:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.946+0000 7fe1f6026740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56207d867830 con 0x56207d906fe0 2024-03-23T23:10:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.946+0000 7fe1f6026740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x56207d8105d0 con 0x56207d90c150 2024-03-23T23:10:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.946+0000 7fe1ebe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d906fe0 0x56207d90bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.946+0000 7fe1ebe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d906fe0 0x56207d90bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48650/0 (socket says 172.21.15.12:48650) 2024-03-23T23:10:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.946+0000 7fe1ebe00640 1 -- 172.21.15.12:0/3214704351 learned_addr learned my addr 172.21.15.12:0/3214704351 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.946+0000 7fe1e3400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56207d9065f0 0x56207d906a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.946+0000 7fe1e3400640 1 --2- 172.21.15.12:0/3214704351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56207d90c150 0x56207d90e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.946+0000 7fe1ebe00640 1 -- 172.21.15.12:0/3214704351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56207d90c150 msgr2=0x56207d90e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.946+0000 7fe1ebe00640 1 --2- 172.21.15.12:0/3214704351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56207d90c150 0x56207d90e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.946+0000 7fe1ebe00640 1 -- 172.21.15.12:0/3214704351 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56207d9065f0 msgr2=0x56207d906a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.946+0000 7fe1ebe00640 1 --2- 172.21.15.12:0/3214704351 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56207d9065f0 0x56207d906a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.946+0000 7fe1ebe00640 1 -- 172.21.15.12:0/3214704351 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56207d8282c0 con 0x56207d906fe0 2024-03-23T23:10:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.946+0000 7fe1e3400640 1 --2- 172.21.15.12:0/3214704351 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56207d9065f0 0x56207d906a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:10:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.947+0000 7fe1ebe00640 1 --2- 172.21.15.12:0/3214704351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d906fe0 0x56207d90bae0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc000c00 tx=0x7fe1dc009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3ccd8d94c31c796 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.947+0000 7fe1e9600640 1 --2- 172.21.15.12:0/3586410678 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe1d00a2920 0x7fe1d00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.947+0000 7fe1e1600640 1 -- 172.21.15.12:0/3214704351 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc045070 con 0x56207d906fe0 2024-03-23T23:10:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.947+0000 7fe1e1600640 1 -- 172.21.15.12:0/3214704351 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe1dc004040 con 0x56207d906fe0 2024-03-23T23:10:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.947+0000 7fe1e1600640 1 -- 172.21.15.12:0/3214704351 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc004eb0 con 0x56207d906fe0 2024-03-23T23:10:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.947+0000 7fe1e9600640 1 -- 172.21.15.12:0/3586410678 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe1e40ceea0 con 0x7fe1ec1073a0 2024-03-23T23:10:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.947+0000 7fe1e3400640 1 --2- 172.21.15.12:0/3586410678 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe1d00a2920 0x7fe1d00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.948+0000 7fe1f6026740 1 -- 172.21.15.12:0/3214704351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d906fe0 msgr2=0x56207d90bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.948+0000 7fe1f6026740 1 --2- 172.21.15.12:0/3214704351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d906fe0 0x56207d90bae0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc000c00 tx=0x7fe1dc009560 comp rx=0 tx=0).stop 2024-03-23T23:10:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.948+0000 7fe1e3400640 1 --2- 172.21.15.12:0/3586410678 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe1d00a2920 0x7fe1d00a4de0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x56207d907840 tx=0x7fe1d4005bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.948+0000 7fe1f6026740 1 -- 172.21.15.12:0/3214704351 shutdown_connections 2024-03-23T23:10:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.948+0000 7fe1f6026740 1 --2- 172.21.15.12:0/3214704351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56207d90c150 0x56207d90e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.948+0000 7fe1f6026740 1 --2- 172.21.15.12:0/3214704351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d906fe0 0x56207d90bae0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.948+0000 7fe1f6026740 1 --2- 172.21.15.12:0/3214704351 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56207d9065f0 0x56207d906a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.948+0000 7fe1f6026740 1 -- 172.21.15.12:0/3214704351 >> 172.21.15.12:0/3214704351 conn(0x56207d8fd750 msgr2=0x56207d8fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.948+0000 7fe1f6026740 1 -- 172.21.15.12:0/3214704351 shutdown_connections 2024-03-23T23:10:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.948+0000 7fe1f6026740 1 -- 172.21.15.12:0/3214704351 wait complete. 2024-03-23T23:10:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.949+0000 7fe1f6026740 1 -- start start 2024-03-23T23:10:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.949+0000 7fe1f6026740 1 Processor -- start 2024-03-23T23:10:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.949+0000 7fe1f6026740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56207d9065f0 0x56207d910700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.949+0000 7fe1f6026740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56207d906fe0 0x56207d910c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.949+0000 7fe1f6026740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d90c150 0x56207d917c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.949+0000 7fe1f6026740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56207d823a20 con 0x56207d906fe0 2024-03-23T23:10:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.949+0000 7fe1f6026740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56207d8638d0 con 0x56207d90c150 2024-03-23T23:10:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.949+0000 7fe1f6026740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x56207d90f040 con 0x56207d9065f0 2024-03-23T23:10:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.949+0000 7fe1f6026740 10 client.0 _ll_register_callbacks cb 0x56207d8601c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.949+0000 7fe1ebe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56207d9065f0 0x56207d910700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.950+0000 7fe1ebe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56207d9065f0 0x56207d910700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:41482/0 (socket says 172.21.15.12:41482) 2024-03-23T23:10:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.950+0000 7fe1ebe00640 1 -- 172.21.15.12:0/3360920055 learned_addr learned my addr 172.21.15.12:0/3360920055 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.950+0000 7fe1e3400640 1 --2- 172.21.15.12:0/3360920055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56207d906fe0 0x56207d910c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.950+0000 7fe1f1200640 1 --2- 172.21.15.12:0/3360920055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d90c150 0x56207d917c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.950+0000 7fe1ebe00640 1 -- 172.21.15.12:0/3360920055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d90c150 msgr2=0x56207d917c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.950+0000 7fe1ebe00640 1 --2- 172.21.15.12:0/3360920055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d90c150 0x56207d917c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.950+0000 7fe1ebe00640 1 -- 172.21.15.12:0/3360920055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56207d906fe0 msgr2=0x56207d910c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.950+0000 7fe1ebe00640 1 --2- 172.21.15.12:0/3360920055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56207d906fe0 0x56207d910c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.950+0000 7fe1ebe00640 1 -- 172.21.15.12:0/3360920055 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56207d57c210 con 0x56207d9065f0 2024-03-23T23:10:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.950+0000 7fe1e3400640 1 --2- 172.21.15.12:0/3360920055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56207d906fe0 0x56207d910c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:10:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.950+0000 7fe1ebe00640 1 --2- 172.21.15.12:0/3360920055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56207d9065f0 0x56207d910700 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc036040 tx=0x7fe1dc004420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67dbe57a0bff760b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.950+0000 7fe1da000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:10:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.951+0000 7fe1da000640 1 -- 172.21.15.12:0/3360920055 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc045040 con 0x56207d9065f0 2024-03-23T23:10:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.951+0000 7fe1f6026740 1 -- 172.21.15.12:0/3360920055 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe1dc004040 con 0x56207d9065f0 2024-03-23T23:10:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.951+0000 7fe1da000640 1 -- 172.21.15.12:0/3360920055 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe1dc004eb0 con 0x56207d9065f0 2024-03-23T23:10:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.951+0000 7fe1da000640 1 -- 172.21.15.12:0/3360920055 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe1dc0050b0 con 0x56207d9065f0 2024-03-23T23:10:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.951+0000 7fe1f6026740 10 client.27706 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.951+0000 7fe1f6026740 1 -- 172.21.15.12:0/3360920055 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56207d648620 con 0x56207d9065f0 2024-03-23T23:10:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.951+0000 7fe1da000640 1 -- 172.21.15.12:0/3360920055 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe1dc004040 con 0x56207d9065f0 2024-03-23T23:10:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.951+0000 7fe1da000640 1 -- 172.21.15.12:0/3360920055 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe1dc04a450 con 0x56207d9065f0 2024-03-23T23:10:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.951+0000 7fe1f6026740 10 client.27706 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.951+0000 7fe1f6026740 10 client.27706 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.951+0000 7fe1f6026740 20 client.27706 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.951+0000 7fe1f6026740 1 --2- 172.21.15.12:0/3360920055 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56207d91c170 0x56207d91e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.952+0000 7fe1f6026740 4 client.27706 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:10:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.952+0000 7fe1f6026740 1 -- 172.21.15.12:0/3360920055 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56207d874d70 con 0x56207d91c170 2024-03-23T23:10:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.952+0000 7fe1e3400640 1 --2- 172.21.15.12:0/3360920055 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56207d91c170 0x56207d91e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.952+0000 7fe1e3400640 1 --2- 172.21.15.12:0/3360920055 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56207d91c170 0x56207d91e560 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84970e8ba01dea5f server_cookie=225b7f55409bf41c in_seq=0 out_seq=0 2024-03-23T23:10:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.952+0000 7fe1da000640 10 client.27706 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:19.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.953+0000 7fe1da000640 1 -- 172.21.15.12:0/3360920055 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56207d874d70 con 0x56207d91c170 2024-03-23T23:10:19.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.953+0000 7fe1da000640 10 client.27706 handle_command_reply: tid=0 2024-03-23T23:10:19.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.953+0000 7fe1f6026740 1 client.27706 shutdown 2024-03-23T23:10:19.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.954+0000 7fe1f6026740 1 -- 172.21.15.12:0/3360920055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56207d9065f0 msgr2=0x56207d910700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:19.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.954+0000 7fe1f6026740 1 --2- 172.21.15.12:0/3360920055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56207d9065f0 0x56207d910700 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc036040 tx=0x7fe1dc004420 comp rx=0 tx=0).stop 2024-03-23T23:10:19.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.954+0000 7fe1f6026740 1 -- 172.21.15.12:0/3360920055 shutdown_connections 2024-03-23T23:10:19.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.954+0000 7fe1f6026740 1 --2- 172.21.15.12:0/3360920055 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56207d91c170 0x56207d91e560 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.954+0000 7fe1f6026740 1 --2- 172.21.15.12:0/3360920055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d90c150 0x56207d917c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.954+0000 7fe1f6026740 1 --2- 172.21.15.12:0/3360920055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56207d906fe0 0x56207d910c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.958 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.954+0000 7fe1f6026740 1 --2- 172.21.15.12:0/3360920055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56207d9065f0 0x56207d910700 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.958 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.954+0000 7fe1da000640 0 client.27706 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:19.958 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.954+0000 7fe1f6026740 1 -- 172.21.15.12:0/3360920055 >> 172.21.15.12:0/3360920055 conn(0x56207d8fd750 msgr2=0x56207d8fc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:19.958 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.954+0000 7fe1f6026740 1 -- 172.21.15.12:0/3360920055 shutdown_connections 2024-03-23T23:10:19.958 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.954+0000 7fe1f6026740 1 -- 172.21.15.12:0/3360920055 wait complete. 2024-03-23T23:10:19.958 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.954+0000 7fe1f6026740 20 client.27706 trim_cache size 0 max 16384 2024-03-23T23:10:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.969+0000 7fe1f6026740 1 Processor -- start 2024-03-23T23:10:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.969+0000 7fe1f6026740 1 -- start start 2024-03-23T23:10:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.969+0000 7fe1f6026740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56207d9065f0 0x56207d922d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.969+0000 7fe1f6026740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d906fe0 0x56207d917c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.969+0000 7fe1f6026740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56207d90c150 0x56207d923270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.969+0000 7fe1f6026740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec1a8bf0 con 0x56207d9065f0 2024-03-23T23:10:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.969+0000 7fe1f6026740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec1a66f0 con 0x56207d906fe0 2024-03-23T23:10:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.969+0000 7fe1f6026740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec1a4790 con 0x56207d90c150 2024-03-23T23:10:19.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.970+0000 7fe1e3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d906fe0 0x56207d917c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.970+0000 7fe1ebe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56207d9065f0 0x56207d922d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.970+0000 7fe1f1200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56207d90c150 0x56207d923270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.970+0000 7fe1e3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d906fe0 0x56207d917c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48664/0 (socket says 172.21.15.12:48664) 2024-03-23T23:10:19.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.970+0000 7fe1e3400640 1 -- 172.21.15.12:0/1494528283 learned_addr learned my addr 172.21.15.12:0/1494528283 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:19.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.970+0000 7fe1e3400640 1 -- 172.21.15.12:0/1494528283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56207d90c150 msgr2=0x56207d923270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:19.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.970+0000 7fe1e3400640 1 --2- 172.21.15.12:0/1494528283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56207d90c150 0x56207d923270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.970+0000 7fe1e3400640 1 -- 172.21.15.12:0/1494528283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56207d9065f0 msgr2=0x56207d922d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:19.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.970+0000 7fe1e3400640 1 --2- 172.21.15.12:0/1494528283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56207d9065f0 0x56207d922d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.970+0000 7fe1e3400640 1 -- 172.21.15.12:0/1494528283 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56207d90b670 con 0x56207d906fe0 2024-03-23T23:10:19.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.970+0000 7fe1e3400640 1 --2- 172.21.15.12:0/1494528283 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d906fe0 0x56207d917c40 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec069880 tx=0x7fe1d4019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=475137422cc20ed4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.971+0000 7fe1e0c00640 1 -- 172.21.15.12:0/1494528283 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4003b70 con 0x56207d906fe0 2024-03-23T23:10:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.971+0000 7fe1e0c00640 1 -- 172.21.15.12:0/1494528283 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe1d4003d10 con 0x56207d906fe0 2024-03-23T23:10:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.971+0000 7fe1e0c00640 1 -- 172.21.15.12:0/1494528283 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4004710 con 0x56207d906fe0 2024-03-23T23:10:19.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.972+0000 7fe1f6026740 1 -- 172.21.15.12:0/1494528283 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d906fe0 msgr2=0x56207d917c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:19.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.972+0000 7fe1f6026740 1 --2- 172.21.15.12:0/1494528283 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d906fe0 0x56207d917c40 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec069880 tx=0x7fe1d4019000 comp rx=0 tx=0).stop 2024-03-23T23:10:19.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.972+0000 7fe1f6026740 1 -- 172.21.15.12:0/1494528283 shutdown_connections 2024-03-23T23:10:19.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.972+0000 7fe1f6026740 1 --2- 172.21.15.12:0/1494528283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56207d90c150 0x56207d923270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.975 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.972+0000 7fe1f6026740 1 --2- 172.21.15.12:0/1494528283 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d906fe0 0x56207d917c40 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.972+0000 7fe1f6026740 1 --2- 172.21.15.12:0/1494528283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56207d9065f0 0x56207d922d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.972+0000 7fe1f6026740 1 -- 172.21.15.12:0/1494528283 >> 172.21.15.12:0/1494528283 conn(0x56207d8fd750 msgr2=0x56207d901240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.972+0000 7fe1f6026740 1 -- 172.21.15.12:0/1494528283 shutdown_connections 2024-03-23T23:10:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.972+0000 7fe1f6026740 1 -- 172.21.15.12:0/1494528283 wait complete. 2024-03-23T23:10:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.972+0000 7fe1f6026740 1 -- start start 2024-03-23T23:10:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.972+0000 7fe1f6026740 1 Processor -- start 2024-03-23T23:10:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.973+0000 7fe1f6026740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56207d9065f0 0x56207d915520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.973+0000 7fe1f6026740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d906fe0 0x56207d915a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.973+0000 7fe1f6026740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56207d90c150 0x56207d916a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.973+0000 7fe1f6026740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec1a4d90 con 0x56207d9065f0 2024-03-23T23:10:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.973+0000 7fe1f6026740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec1a6090 con 0x56207d906fe0 2024-03-23T23:10:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.973+0000 7fe1f6026740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec1a53d0 con 0x56207d90c150 2024-03-23T23:10:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.973+0000 7fe1f6026740 10 client.0 _ll_register_callbacks cb 0x7fe1dc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.973+0000 7fe1e3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d906fe0 0x56207d915a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.973+0000 7fe1ebe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56207d9065f0 0x56207d915520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.973+0000 7fe1e3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d906fe0 0x56207d915a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48674/0 (socket says 172.21.15.12:48674) 2024-03-23T23:10:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.973+0000 7fe1e3400640 1 -- 172.21.15.12:0/584486792 learned_addr learned my addr 172.21.15.12:0/584486792 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.973+0000 7fe1f1200640 1 --2- 172.21.15.12:0/584486792 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56207d90c150 0x56207d916a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.974+0000 7fe1e3400640 1 -- 172.21.15.12:0/584486792 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56207d90c150 msgr2=0x56207d916a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.974+0000 7fe1e3400640 1 --2- 172.21.15.12:0/584486792 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56207d90c150 0x56207d916a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.974+0000 7fe1e3400640 1 -- 172.21.15.12:0/584486792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56207d9065f0 msgr2=0x56207d915520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.974+0000 7fe1e3400640 1 --2- 172.21.15.12:0/584486792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56207d9065f0 0x56207d915520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.974+0000 7fe1e3400640 1 -- 172.21.15.12:0/584486792 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56207d917060 con 0x56207d906fe0 2024-03-23T23:10:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.974+0000 7fe1e3400640 1 --2- 172.21.15.12:0/584486792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d906fe0 0x56207d915a60 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4005c40 tx=0x7fe1d4013f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bb8aae822d7d5cd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.974+0000 7fe1da000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:10:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.974+0000 7fe1da000640 1 -- 172.21.15.12:0/584486792 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d403d060 con 0x56207d906fe0 2024-03-23T23:10:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.974+0000 7fe1da000640 1 -- 172.21.15.12:0/584486792 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe1d4003a70 con 0x56207d906fe0 2024-03-23T23:10:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.974+0000 7fe1f6026740 1 -- 172.21.15.12:0/584486792 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe1d4003d10 con 0x56207d906fe0 2024-03-23T23:10:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.974+0000 7fe1da000640 1 -- 172.21.15.12:0/584486792 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe1d4003ec0 con 0x56207d906fe0 2024-03-23T23:10:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.974+0000 7fe1f6026740 10 client.27752 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.975+0000 7fe1f6026740 1 -- 172.21.15.12:0/584486792 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56207d977b40 con 0x56207d906fe0 2024-03-23T23:10:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.975+0000 7fe1da000640 1 -- 172.21.15.12:0/584486792 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe1d4003d10 con 0x56207d906fe0 2024-03-23T23:10:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.975+0000 7fe1da000640 1 -- 172.21.15.12:0/584486792 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe1d4021430 con 0x56207d906fe0 2024-03-23T23:10:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.975+0000 7fe1f6026740 10 client.27752 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.975+0000 7fe1f6026740 10 client.27752 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.975+0000 7fe1f6026740 20 client.27752 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.975+0000 7fe1f6026740 1 --2- 172.21.15.12:0/584486792 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56207d978460 0x56207d97a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.975+0000 7fe1f6026740 4 client.27752 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:10:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.975+0000 7fe1f6026740 1 -- 172.21.15.12:0/584486792 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56207d86f0c0 con 0x56207d978460 2024-03-23T23:10:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.975+0000 7fe1ebe00640 1 --2- 172.21.15.12:0/584486792 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56207d978460 0x56207d97a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.976+0000 7fe1ebe00640 1 --2- 172.21.15.12:0/584486792 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56207d978460 0x56207d97a880 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc697b0cf0790dce server_cookie=f98ca4941ae3628a in_seq=0 out_seq=0 2024-03-23T23:10:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.976+0000 7fe1da000640 10 client.27752 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:19.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.987+0000 7fe1da000640 1 -- 172.21.15.12:0/584486792 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56207d86f0c0 con 0x56207d978460 2024-03-23T23:10:19.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.987+0000 7fe1da000640 10 client.27752 handle_command_reply: tid=0 2024-03-23T23:10:19.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.987+0000 7fe1f6026740 1 client.27752 shutdown 2024-03-23T23:10:19.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.987+0000 7fe1f6026740 1 -- 172.21.15.12:0/584486792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d906fe0 msgr2=0x56207d915a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:19.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.987+0000 7fe1f6026740 1 --2- 172.21.15.12:0/584486792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d906fe0 0x56207d915a60 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4005c40 tx=0x7fe1d4013f60 comp rx=0 tx=0).stop 2024-03-23T23:10:19.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.987+0000 7fe1f6026740 1 -- 172.21.15.12:0/584486792 shutdown_connections 2024-03-23T23:10:19.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.987+0000 7fe1f6026740 1 --2- 172.21.15.12:0/584486792 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56207d978460 0x56207d97a880 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.987+0000 7fe1f6026740 1 --2- 172.21.15.12:0/584486792 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56207d90c150 0x56207d916a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.987+0000 7fe1f6026740 1 --2- 172.21.15.12:0/584486792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56207d906fe0 0x56207d915a60 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.988+0000 7fe1f6026740 1 --2- 172.21.15.12:0/584486792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56207d9065f0 0x56207d915520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.988+0000 7fe1f6026740 1 -- 172.21.15.12:0/584486792 >> 172.21.15.12:0/584486792 conn(0x56207d8fd750 msgr2=0x56207d8fc0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:19.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.988+0000 7fe1da000640 0 client.27752 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:19.989 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.988+0000 7fe1f6026740 1 -- 172.21.15.12:0/584486792 shutdown_connections 2024-03-23T23:10:19.989 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.988+0000 7fe1f6026740 1 -- 172.21.15.12:0/584486792 wait complete. 2024-03-23T23:10:19.989 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.988+0000 7fe1f6026740 20 client.27752 trim_cache size 0 max 16384 2024-03-23T23:10:19.989 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:10:19.989 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:10:19.989 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:10:19.989 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:10:19.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.989+0000 7fe1f3200640 1 -- 172.21.15.12:0/3586410678 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe1d00a2920 msgr2=0x7fe1d00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:19.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.989+0000 7fe1f3200640 1 --2- 172.21.15.12:0/3586410678 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe1d00a2920 0x7fe1d00a4de0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x56207d907840 tx=0x7fe1d4005bd0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.989+0000 7fe1f3200640 1 -- 172.21.15.12:0/3586410678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1ec1073a0 msgr2=0x7fe1ec19eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:19.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.989+0000 7fe1f3200640 1 --2- 172.21.15.12:0/3586410678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1ec1073a0 0x7fe1ec19eef0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fe1e400e970 tx=0x7fe1e400ed70 comp rx=0 tx=0).stop 2024-03-23T23:10:19.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.989+0000 7fe1f3200640 1 -- 172.21.15.12:0/3586410678 shutdown_connections 2024-03-23T23:10:19.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.989+0000 7fe1f3200640 1 --2- 172.21.15.12:0/3586410678 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe1d00a2920 0x7fe1d00a4de0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.989+0000 7fe1f3200640 1 --2- 172.21.15.12:0/3586410678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1ec1073a0 0x7fe1ec19eef0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.989+0000 7fe1f3200640 1 --2- 172.21.15.12:0/3586410678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1ec104a70 0x7fe1ec197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.990 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.989+0000 7fe1f3200640 1 --2- 172.21.15.12:0/3586410678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1ec102140 0x7fe1ec197940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:19.991 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.989+0000 7fe1f3200640 1 -- 172.21.15.12:0/3586410678 >> 172.21.15.12:0/3586410678 conn(0x7fe1ec0fbb30 msgr2=0x7fe1ec102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:19.991 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.989+0000 7fe1f3200640 1 -- 172.21.15.12:0/3586410678 shutdown_connections 2024-03-23T23:10:19.991 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:19.989+0000 7fe1f3200640 1 -- 172.21.15.12:0/3586410678 wait complete. 2024-03-23T23:10:20.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.031+0000 7f9e35600640 1 -- 172.21.15.12:0/1877862641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e301038e0 msgr2=0x7f9e30103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:20.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.031+0000 7f9e35600640 1 --2- 172.21.15.12:0/1877862641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e301038e0 0x7f9e30103d00 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9e20009900 tx=0x7f9e2002bc40 comp rx=0 tx=0).stop 2024-03-23T23:10:20.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.031+0000 7f9e35600640 1 -- 172.21.15.12:0/1877862641 shutdown_connections 2024-03-23T23:10:20.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.031+0000 7f9e35600640 1 --2- 172.21.15.12:0/1877862641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e30105490 0x7f9e3010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.031+0000 7f9e35600640 1 --2- 172.21.15.12:0/1877862641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e30104ad0 0x7f9e30104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.031+0000 7f9e35600640 1 --2- 172.21.15.12:0/1877862641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e301038e0 0x7f9e30103d00 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.031+0000 7f9e35600640 1 -- 172.21.15.12:0/1877862641 >> 172.21.15.12:0/1877862641 conn(0x7f9e300fee50 msgr2=0x7f9e30101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:20.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.031+0000 7f9e35600640 1 -- 172.21.15.12:0/1877862641 shutdown_connections 2024-03-23T23:10:20.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.031+0000 7f9e35600640 1 -- 172.21.15.12:0/1877862641 wait complete. 2024-03-23T23:10:20.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.032+0000 7f9e35600640 1 Processor -- start 2024-03-23T23:10:20.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.032+0000 7f9e35600640 1 -- start start 2024-03-23T23:10:20.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.033+0000 7f9e35600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e301038e0 0x7f9e30199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:20.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.033+0000 7f9e35600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e30104ad0 0x7f9e3019a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:20.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.033+0000 7f9e35600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e30105490 0x7f9e301a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:20.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.033+0000 7f9e35600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3010f5a0 con 0x7f9e301038e0 2024-03-23T23:10:20.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.033+0000 7f9e35600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3010f420 con 0x7f9e30104ad0 2024-03-23T23:10:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.033+0000 7f9e35600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3010f720 con 0x7f9e30105490 2024-03-23T23:10:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.033+0000 7f9e26000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e30104ad0 0x7f9e3019a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.033+0000 7f9e2ea00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e301038e0 0x7f9e30199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.033+0000 7f9e26000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e30104ad0 0x7f9e3019a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48690/0 (socket says 172.21.15.12:48690) 2024-03-23T23:10:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.033+0000 7f9e26000640 1 -- 172.21.15.12:0/2818242663 learned_addr learned my addr 172.21.15.12:0/2818242663 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.033+0000 7f9e2f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e30105490 0x7f9e301a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.033+0000 7f9e26000640 1 -- 172.21.15.12:0/2818242663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e30105490 msgr2=0x7f9e301a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.033+0000 7f9e26000640 1 --2- 172.21.15.12:0/2818242663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e30105490 0x7f9e301a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.033+0000 7f9e26000640 1 -- 172.21.15.12:0/2818242663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e301038e0 msgr2=0x7f9e30199c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.033+0000 7f9e26000640 1 --2- 172.21.15.12:0/2818242663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e301038e0 0x7f9e30199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.033+0000 7f9e26000640 1 -- 172.21.15.12:0/2818242663 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e301a1870 con 0x7f9e30104ad0 2024-03-23T23:10:20.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.034+0000 7f9e26000640 1 --2- 172.21.15.12:0/2818242663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e30104ad0 0x7f9e3019a1a0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f9e1800e9f0 tx=0x7f9e1800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:20.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.034+0000 7f9e27e00640 1 -- 172.21.15.12:0/2818242663 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1800cd80 con 0x7f9e30104ad0 2024-03-23T23:10:20.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.034+0000 7f9e27e00640 1 -- 172.21.15.12:0/2818242663 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e18005500 con 0x7f9e30104ad0 2024-03-23T23:10:20.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.034+0000 7f9e35600640 1 -- 172.21.15.12:0/2818242663 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e301a1b60 con 0x7f9e30104ad0 2024-03-23T23:10:20.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.034+0000 7f9e27e00640 1 -- 172.21.15.12:0/2818242663 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e18010690 con 0x7f9e30104ad0 2024-03-23T23:10:20.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.034+0000 7f9e35600640 1 -- 172.21.15.12:0/2818242663 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e301a2030 con 0x7f9e30104ad0 2024-03-23T23:10:20.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.037+0000 7f9e27e00640 1 -- 172.21.15.12:0/2818242663 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9e180046c0 con 0x7f9e30104ad0 2024-03-23T23:10:20.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.038+0000 7f9e383e4740 1 Processor -- start 2024-03-23T23:10:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.038+0000 7f9e383e4740 1 -- start start 2024-03-23T23:10:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.038+0000 7f9e383e4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe159e80 0x55eefe15a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.038+0000 7f9e383e4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eefe15a870 0x55eefe15f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.038+0000 7f9e383e4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eefe15f9e0 0x55eefe161e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.038+0000 7f9e383e4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eefe0bf250 con 0x55eefe15f9e0 2024-03-23T23:10:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.038+0000 7f9e383e4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eefe067a30 con 0x55eefe15a870 2024-03-23T23:10:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.038+0000 7f9e383e4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eefe0bb2f0 con 0x55eefe159e80 2024-03-23T23:10:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.038+0000 7f9e2f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe159e80 0x55eefe15a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.038+0000 7f9e2ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eefe15a870 0x55eefe15f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.038+0000 7f9e26000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eefe15f9e0 0x55eefe161e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.038+0000 7f9e2ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eefe15a870 0x55eefe15f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48696/0 (socket says 172.21.15.12:48696) 2024-03-23T23:10:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.038+0000 7f9e2f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe159e80 0x55eefe15a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:41532/0 (socket says 172.21.15.12:41532) 2024-03-23T23:10:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.038+0000 7f9e2ea00640 1 -- 172.21.15.12:0/438263769 learned_addr learned my addr 172.21.15.12:0/438263769 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.039+0000 7f9e2f400640 1 -- 172.21.15.12:0/438263769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eefe15a870 msgr2=0x55eefe15f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.039+0000 7f9e2f400640 1 --2- 172.21.15.12:0/438263769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eefe15a870 0x55eefe15f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.039+0000 7f9e2f400640 1 -- 172.21.15.12:0/438263769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eefe15f9e0 msgr2=0x55eefe161e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.039+0000 7f9e2f400640 1 --2- 172.21.15.12:0/438263769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eefe15f9e0 0x55eefe161e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.039+0000 7f9e2f400640 1 -- 172.21.15.12:0/438263769 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eefe052690 con 0x55eefe159e80 2024-03-23T23:10:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.039+0000 7f9e2f400640 1 --2- 172.21.15.12:0/438263769 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe159e80 0x55eefe15a2a0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f9e28009f30 tx=0x7f9e28010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5aa12b6eabd1bea6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.039+0000 7f9e27e00640 1 --2- 172.21.15.12:0/2818242663 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9e140a2800 0x7f9e140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:20.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.039+0000 7f9e1f400640 1 -- 172.21.15.12:0/438263769 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e28016070 con 0x55eefe159e80 2024-03-23T23:10:20.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.039+0000 7f9e1f400640 1 -- 172.21.15.12:0/438263769 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e28004d20 con 0x55eefe159e80 2024-03-23T23:10:20.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.039+0000 7f9e1f400640 1 -- 172.21.15.12:0/438263769 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e28012870 con 0x55eefe159e80 2024-03-23T23:10:20.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.039+0000 7f9e27e00640 1 -- 172.21.15.12:0/2818242663 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9e180cef30 con 0x7f9e30104ad0 2024-03-23T23:10:20.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.040+0000 7f9e2ea00640 1 --2- 172.21.15.12:0/2818242663 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9e140a2800 0x7f9e140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:20.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.040+0000 7f9e383e4740 1 -- 172.21.15.12:0/438263769 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe159e80 msgr2=0x55eefe15a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:20.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.040+0000 7f9e383e4740 1 --2- 172.21.15.12:0/438263769 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe159e80 0x55eefe15a2a0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f9e28009f30 tx=0x7f9e28010800 comp rx=0 tx=0).stop 2024-03-23T23:10:20.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.040+0000 7f9e383e4740 1 -- 172.21.15.12:0/438263769 shutdown_connections 2024-03-23T23:10:20.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.040+0000 7f9e383e4740 1 --2- 172.21.15.12:0/438263769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eefe15f9e0 0x55eefe161e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.040+0000 7f9e383e4740 1 --2- 172.21.15.12:0/438263769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eefe15a870 0x55eefe15f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.040+0000 7f9e383e4740 1 --2- 172.21.15.12:0/438263769 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe159e80 0x55eefe15a2a0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.040+0000 7f9e383e4740 1 -- 172.21.15.12:0/438263769 >> 172.21.15.12:0/438263769 conn(0x55eefe122200 msgr2=0x55eefe1557f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:20.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.040+0000 7f9e383e4740 1 -- 172.21.15.12:0/438263769 shutdown_connections 2024-03-23T23:10:20.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.040+0000 7f9e383e4740 1 -- 172.21.15.12:0/438263769 wait complete. 2024-03-23T23:10:20.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.040+0000 7f9e2ea00640 1 --2- 172.21.15.12:0/2818242663 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9e140a2800 0x7f9e140a4cc0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9e20005ab0 tx=0x7f9e2002cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:20.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.041+0000 7f9e383e4740 1 -- start start 2024-03-23T23:10:20.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.041+0000 7f9e383e4740 1 Processor -- start 2024-03-23T23:10:20.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.041+0000 7f9e383e4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe159e80 0x55eefe163df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:20.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.041+0000 7f9e383e4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eefe15a870 0x55eefe164330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:20.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.041+0000 7f9e383e4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eefe15f9e0 0x55eefe16b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:20.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.041+0000 7f9e383e4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eefe162990 con 0x55eefe15a870 2024-03-23T23:10:20.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.041+0000 7f9e383e4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eefe07ae80 con 0x55eefe15f9e0 2024-03-23T23:10:20.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.041+0000 7f9e383e4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eefe162b10 con 0x55eefe159e80 2024-03-23T23:10:20.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.042+0000 7f9e383e4740 10 client.0 _ll_register_callbacks cb 0x55eefe08d130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:20.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.042+0000 7f9e2ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe159e80 0x55eefe163df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:20.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.042+0000 7f9e2ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe159e80 0x55eefe163df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:41546/0 (socket says 172.21.15.12:41546) 2024-03-23T23:10:20.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.042+0000 7f9e2ea00640 1 -- 172.21.15.12:0/3872299798 learned_addr learned my addr 172.21.15.12:0/3872299798 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:20.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.042+0000 7f9e2f400640 1 --2- 172.21.15.12:0/3872299798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eefe15f9e0 0x55eefe16b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:20.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.042+0000 7f9e26000640 1 --2- 172.21.15.12:0/3872299798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eefe15a870 0x55eefe164330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:20.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.042+0000 7f9e2ea00640 1 -- 172.21.15.12:0/3872299798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eefe15f9e0 msgr2=0x55eefe16b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:20.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.042+0000 7f9e2ea00640 1 --2- 172.21.15.12:0/3872299798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eefe15f9e0 0x55eefe16b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.042+0000 7f9e2ea00640 1 -- 172.21.15.12:0/3872299798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eefe15a870 msgr2=0x55eefe164330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:20.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.042+0000 7f9e2ea00640 1 --2- 172.21.15.12:0/3872299798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eefe15a870 0x55eefe164330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.042+0000 7f9e2ea00640 1 -- 172.21.15.12:0/3872299798 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eefde9d7d0 con 0x55eefe159e80 2024-03-23T23:10:20.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.042+0000 7f9e2ea00640 1 --2- 172.21.15.12:0/3872299798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe159e80 0x55eefe163df0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f9e20009230 tx=0x7f9e2002bc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bffa7cbcac5fe78 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:20.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.042+0000 7f9e13e00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:10:20.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.043+0000 7f9e13e00640 1 -- 172.21.15.12:0/3872299798 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e20047460 con 0x55eefe159e80 2024-03-23T23:10:20.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.043+0000 7f9e13e00640 1 -- 172.21.15.12:0/3872299798 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9e20047600 con 0x55eefe159e80 2024-03-23T23:10:20.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.043+0000 7f9e383e4740 1 -- 172.21.15.12:0/3872299798 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9e28004d20 con 0x55eefe159e80 2024-03-23T23:10:20.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.043+0000 7f9e13e00640 1 -- 172.21.15.12:0/3872299798 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e20047800 con 0x55eefe159e80 2024-03-23T23:10:20.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.043+0000 7f9e383e4740 10 client.27724 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:20.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.043+0000 7f9e383e4740 1 -- 172.21.15.12:0/3872299798 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55eefddd33f0 con 0x55eefe159e80 2024-03-23T23:10:20.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.043+0000 7f9e13e00640 1 -- 172.21.15.12:0/3872299798 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e28004d20 con 0x55eefe159e80 2024-03-23T23:10:20.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.044+0000 7f9e13e00640 1 -- 172.21.15.12:0/3872299798 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f9e20050070 con 0x55eefe159e80 2024-03-23T23:10:20.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.044+0000 7f9e383e4740 10 client.27724 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:20.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.044+0000 7f9e383e4740 10 client.27724 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:20.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.044+0000 7f9e383e4740 20 client.27724 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:20.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.044+0000 7f9e383e4740 1 --2- 172.21.15.12:0/3872299798 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55eefe16fbf0 0x55eefe172060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:20.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.044+0000 7f9e383e4740 4 client.27724 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:10:20.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.044+0000 7f9e383e4740 1 -- 172.21.15.12:0/3872299798 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eefe0cc790 con 0x55eefe16fbf0 2024-03-23T23:10:20.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.044+0000 7f9e26000640 1 --2- 172.21.15.12:0/3872299798 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55eefe16fbf0 0x55eefe172060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:20.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.044+0000 7f9e26000640 1 --2- 172.21.15.12:0/3872299798 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55eefe16fbf0 0x55eefe172060 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=292a417f85bbdae1 server_cookie=ad749208ca6f90b7 in_seq=0 out_seq=0 2024-03-23T23:10:20.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.044+0000 7f9e13e00640 10 client.27724 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:20.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.045+0000 7f9e13e00640 1 -- 172.21.15.12:0/3872299798 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55eefe0cc790 con 0x55eefe16fbf0 2024-03-23T23:10:20.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.045+0000 7f9e13e00640 10 client.27724 handle_command_reply: tid=0 2024-03-23T23:10:20.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.045+0000 7f9e383e4740 1 client.27724 shutdown 2024-03-23T23:10:20.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.045+0000 7f9e383e4740 1 -- 172.21.15.12:0/3872299798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe159e80 msgr2=0x55eefe163df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:20.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.045+0000 7f9e383e4740 1 --2- 172.21.15.12:0/3872299798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe159e80 0x55eefe163df0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f9e20009230 tx=0x7f9e2002bc40 comp rx=0 tx=0).stop 2024-03-23T23:10:20.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.046+0000 7f9e383e4740 1 -- 172.21.15.12:0/3872299798 shutdown_connections 2024-03-23T23:10:20.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.046+0000 7f9e383e4740 1 --2- 172.21.15.12:0/3872299798 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55eefe16fbf0 0x55eefe172060 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.046+0000 7f9e383e4740 1 --2- 172.21.15.12:0/3872299798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eefe15f9e0 0x55eefe16b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.046+0000 7f9e383e4740 1 --2- 172.21.15.12:0/3872299798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eefe15a870 0x55eefe164330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.046+0000 7f9e383e4740 1 --2- 172.21.15.12:0/3872299798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe159e80 0x55eefe163df0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.046+0000 7f9e383e4740 1 -- 172.21.15.12:0/3872299798 >> 172.21.15.12:0/3872299798 conn(0x55eefe122200 msgr2=0x55eefe154a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:20.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.046+0000 7f9e13e00640 0 client.27724 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:20.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.046+0000 7f9e383e4740 1 -- 172.21.15.12:0/3872299798 shutdown_connections 2024-03-23T23:10:20.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.046+0000 7f9e383e4740 1 -- 172.21.15.12:0/3872299798 wait complete. 2024-03-23T23:10:20.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.046+0000 7f9e383e4740 20 client.27724 trim_cache size 0 max 16384 2024-03-23T23:10:20.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.063+0000 7f9e383e4740 1 Processor -- start 2024-03-23T23:10:20.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.063+0000 7f9e383e4740 1 -- start start 2024-03-23T23:10:20.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.064+0000 7f9e383e4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eefe159e80 0x55eefe158a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:20.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.064+0000 7f9e383e4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eefe15a870 0x55eefe16b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:20.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.064+0000 7f9e383e4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe15f9e0 0x55eefe158f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:20.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.064+0000 7f9e383e4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e301ad0c0 con 0x55eefe159e80 2024-03-23T23:10:20.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.064+0000 7f9e383e4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e301aabc0 con 0x55eefe15a870 2024-03-23T23:10:20.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.064+0000 7f9e383e4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e301a8c60 con 0x55eefe15f9e0 2024-03-23T23:10:20.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.064+0000 7f9e26000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eefe15a870 0x55eefe16b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:20.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.064+0000 7f9e2ea00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eefe159e80 0x55eefe158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:20.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.064+0000 7f9e2f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe15f9e0 0x55eefe158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:20.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.064+0000 7f9e26000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eefe15a870 0x55eefe16b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48726/0 (socket says 172.21.15.12:48726) 2024-03-23T23:10:20.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.064+0000 7f9e26000640 1 -- 172.21.15.12:0/2559571884 learned_addr learned my addr 172.21.15.12:0/2559571884 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:20.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.064+0000 7f9e26000640 1 -- 172.21.15.12:0/2559571884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe15f9e0 msgr2=0x55eefe158f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:20.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.064+0000 7f9e26000640 1 --2- 172.21.15.12:0/2559571884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe15f9e0 0x55eefe158f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.064+0000 7f9e26000640 1 -- 172.21.15.12:0/2559571884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eefe159e80 msgr2=0x55eefe158a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:20.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.064+0000 7f9e26000640 1 --2- 172.21.15.12:0/2559571884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eefe159e80 0x55eefe158a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.064+0000 7f9e26000640 1 -- 172.21.15.12:0/2559571884 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eefdd930f0 con 0x55eefe15a870 2024-03-23T23:10:20.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.065+0000 7f9e26000640 1 --2- 172.21.15.12:0/2559571884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eefe15a870 0x55eefe16b300 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f9e180049d0 tx=0x7f9e18086040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84d10c0b5d996ff5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:20.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.065+0000 7f9e1ea00640 1 -- 172.21.15.12:0/2559571884 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1809b650 con 0x55eefe15a870 2024-03-23T23:10:20.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.065+0000 7f9e1ea00640 1 -- 172.21.15.12:0/2559571884 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e1809bf80 con 0x55eefe15a870 2024-03-23T23:10:20.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.065+0000 7f9e1ea00640 1 -- 172.21.15.12:0/2559571884 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e180999c0 con 0x55eefe15a870 2024-03-23T23:10:20.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.065+0000 7f9e383e4740 1 -- 172.21.15.12:0/2559571884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eefe15a870 msgr2=0x55eefe16b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:20.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.065+0000 7f9e383e4740 1 --2- 172.21.15.12:0/2559571884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eefe15a870 0x55eefe16b300 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f9e180049d0 tx=0x7f9e18086040 comp rx=0 tx=0).stop 2024-03-23T23:10:20.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.065+0000 7f9e383e4740 1 -- 172.21.15.12:0/2559571884 shutdown_connections 2024-03-23T23:10:20.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.065+0000 7f9e383e4740 1 --2- 172.21.15.12:0/2559571884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe15f9e0 0x55eefe158f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.065+0000 7f9e383e4740 1 --2- 172.21.15.12:0/2559571884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eefe15a870 0x55eefe16b300 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.065+0000 7f9e383e4740 1 --2- 172.21.15.12:0/2559571884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eefe159e80 0x55eefe158a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.065+0000 7f9e383e4740 1 -- 172.21.15.12:0/2559571884 >> 172.21.15.12:0/2559571884 conn(0x55eefe122200 msgr2=0x55eefe15bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:20.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.066+0000 7f9e383e4740 1 -- 172.21.15.12:0/2559571884 shutdown_connections 2024-03-23T23:10:20.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.066+0000 7f9e383e4740 1 -- 172.21.15.12:0/2559571884 wait complete. 2024-03-23T23:10:20.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.066+0000 7f9e383e4740 1 -- start start 2024-03-23T23:10:20.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.066+0000 7f9e383e4740 1 Processor -- start 2024-03-23T23:10:20.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.066+0000 7f9e383e4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe159e80 0x55eefe168ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:20.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.066+0000 7f9e383e4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eefe15a870 0x55eefe169530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:20.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.066+0000 7f9e383e4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eefe15f9e0 0x55eefe16a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:20.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.067+0000 7f9e383e4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e301a9260 con 0x55eefe15f9e0 2024-03-23T23:10:20.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.067+0000 7f9e383e4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e301aa560 con 0x55eefe15a870 2024-03-23T23:10:20.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.067+0000 7f9e383e4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e301a98a0 con 0x55eefe159e80 2024-03-23T23:10:20.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.067+0000 7f9e383e4740 10 client.0 _ll_register_callbacks cb 0x7f9e2002c6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:20.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.067+0000 7f9e26000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eefe15a870 0x55eefe169530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:20.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.067+0000 7f9e2ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe159e80 0x55eefe168ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:20.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.067+0000 7f9e26000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eefe15a870 0x55eefe169530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:48728/0 (socket says 172.21.15.12:48728) 2024-03-23T23:10:20.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.067+0000 7f9e26000640 1 -- 172.21.15.12:0/1245449400 learned_addr learned my addr 172.21.15.12:0/1245449400 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:20.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.067+0000 7f9e2ea00640 1 --2- 172.21.15.12:0/1245449400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe159e80 0x55eefe168ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:41554/0 (socket says 172.21.15.12:41554) 2024-03-23T23:10:20.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.067+0000 7f9e2f400640 1 --2- 172.21.15.12:0/1245449400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eefe15f9e0 0x55eefe16a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:20.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.067+0000 7f9e26000640 1 -- 172.21.15.12:0/1245449400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe159e80 msgr2=0x55eefe168ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:20.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.067+0000 7f9e26000640 1 --2- 172.21.15.12:0/1245449400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe159e80 0x55eefe168ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.067+0000 7f9e26000640 1 -- 172.21.15.12:0/1245449400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eefe15f9e0 msgr2=0x55eefe16a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:20.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.067+0000 7f9e26000640 1 --2- 172.21.15.12:0/1245449400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eefe15f9e0 0x55eefe16a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.067+0000 7f9e26000640 1 -- 172.21.15.12:0/1245449400 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eefe16aa20 con 0x55eefe15a870 2024-03-23T23:10:20.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.067+0000 7f9e2ea00640 1 --2- 172.21.15.12:0/1245449400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe159e80 0x55eefe168ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:20.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.067+0000 7f9e26000640 1 --2- 172.21.15.12:0/1245449400 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eefe15a870 0x55eefe169530 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f9e180a3af0 tx=0x7f9e1800c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43019bc5312e2146 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:20.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.067+0000 7f9e13e00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:10:20.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.068+0000 7f9e13e00640 1 -- 172.21.15.12:0/1245449400 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1809b1e0 con 0x55eefe15a870 2024-03-23T23:10:20.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.068+0000 7f9e13e00640 1 -- 172.21.15.12:0/1245449400 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9e18097e60 con 0x55eefe15a870 2024-03-23T23:10:20.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.068+0000 7f9e383e4740 1 -- 172.21.15.12:0/1245449400 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9e1809bf80 con 0x55eefe15a870 2024-03-23T23:10:20.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.068+0000 7f9e13e00640 1 -- 172.21.15.12:0/1245449400 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e1809b3c0 con 0x55eefe15a870 2024-03-23T23:10:20.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.068+0000 7f9e383e4740 10 client.27770 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:20.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.068+0000 7f9e383e4740 1 -- 172.21.15.12:0/1245449400 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55eefe1cba70 con 0x55eefe15a870 2024-03-23T23:10:20.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.068+0000 7f9e13e00640 1 -- 172.21.15.12:0/1245449400 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e1809bf80 con 0x55eefe15a870 2024-03-23T23:10:20.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.068+0000 7f9e13e00640 1 -- 172.21.15.12:0/1245449400 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f9e180a6080 con 0x55eefe15a870 2024-03-23T23:10:20.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.068+0000 7f9e383e4740 10 client.27770 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:20.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.068+0000 7f9e383e4740 10 client.27770 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:20.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.068+0000 7f9e383e4740 20 client.27770 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:20.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.068+0000 7f9e383e4740 1 --2- 172.21.15.12:0/1245449400 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55eefe1cc320 0x55eefe1ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:20.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.068+0000 7f9e383e4740 4 client.27770 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:10:20.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.068+0000 7f9e383e4740 1 -- 172.21.15.12:0/1245449400 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55eefe0c6ae0 con 0x55eefe1cc320 2024-03-23T23:10:20.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.069+0000 7f9e2ea00640 1 --2- 172.21.15.12:0/1245449400 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55eefe1cc320 0x55eefe1ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:20.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.069+0000 7f9e2ea00640 1 --2- 172.21.15.12:0/1245449400 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55eefe1cc320 0x55eefe1ce740 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6be91ff8fcc15fe9 server_cookie=e44bdd6c50400187 in_seq=0 out_seq=0 2024-03-23T23:10:20.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.069+0000 7f9e13e00640 10 client.27770 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:20.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.069+0000 7f9e13e00640 1 -- 172.21.15.12:0/1245449400 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55eefe0c6ae0 con 0x55eefe1cc320 2024-03-23T23:10:20.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.069+0000 7f9e13e00640 10 client.27770 handle_command_reply: tid=0 2024-03-23T23:10:20.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.069+0000 7f9e383e4740 1 client.27770 shutdown 2024-03-23T23:10:20.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.069+0000 7f9e383e4740 1 -- 172.21.15.12:0/1245449400 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eefe15a870 msgr2=0x55eefe169530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:20.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.069+0000 7f9e383e4740 1 --2- 172.21.15.12:0/1245449400 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eefe15a870 0x55eefe169530 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f9e180a3af0 tx=0x7f9e1800c7d0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.069+0000 7f9e383e4740 1 -- 172.21.15.12:0/1245449400 shutdown_connections 2024-03-23T23:10:20.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.069+0000 7f9e383e4740 1 --2- 172.21.15.12:0/1245449400 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55eefe1cc320 0x55eefe1ce740 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.069+0000 7f9e383e4740 1 --2- 172.21.15.12:0/1245449400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eefe15f9e0 0x55eefe16a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.069+0000 7f9e383e4740 1 --2- 172.21.15.12:0/1245449400 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eefe15a870 0x55eefe169530 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.069+0000 7f9e383e4740 1 --2- 172.21.15.12:0/1245449400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eefe159e80 0x55eefe168ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.069+0000 7f9e383e4740 1 -- 172.21.15.12:0/1245449400 >> 172.21.15.12:0/1245449400 conn(0x55eefe122200 msgr2=0x55eefe154200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:20.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.069+0000 7f9e13e00640 0 client.27770 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:20.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.070+0000 7f9e383e4740 1 -- 172.21.15.12:0/1245449400 shutdown_connections 2024-03-23T23:10:20.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.070+0000 7f9e383e4740 1 -- 172.21.15.12:0/1245449400 wait complete. 2024-03-23T23:10:20.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.070+0000 7f9e383e4740 20 client.27770 trim_cache size 0 max 16384 2024-03-23T23:10:20.077 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:10:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T23:10:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-03-23T23:10:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: "98c681b7-90f9-4d43-b9a9-e0e1f0eb0995": { 2024-03-23T23:10:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-03-23T23:10:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "98c681b7-90f9-4d43-b9a9-e0e1f0eb0995", 2024-03-23T23:10:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-03-23T23:10:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:10:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:10:20.077 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:10:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.070+0000 7f9e35600640 1 -- 172.21.15.12:0/2818242663 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9e140a2800 msgr2=0x7f9e140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.070+0000 7f9e35600640 1 --2- 172.21.15.12:0/2818242663 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9e140a2800 0x7f9e140a4cc0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9e20005ab0 tx=0x7f9e2002cf90 comp rx=0 tx=0).stop 2024-03-23T23:10:20.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.070+0000 7f9e35600640 1 -- 172.21.15.12:0/2818242663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e30104ad0 msgr2=0x7f9e3019a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:20.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.070+0000 7f9e35600640 1 --2- 172.21.15.12:0/2818242663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e30104ad0 0x7f9e3019a1a0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f9e1800e9f0 tx=0x7f9e1800edf0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.070+0000 7f9e35600640 1 -- 172.21.15.12:0/2818242663 shutdown_connections 2024-03-23T23:10:20.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.070+0000 7f9e35600640 1 --2- 172.21.15.12:0/2818242663 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f9e140a2800 0x7f9e140a4cc0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.070+0000 7f9e35600640 1 --2- 172.21.15.12:0/2818242663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e30105490 0x7f9e301a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.070+0000 7f9e35600640 1 --2- 172.21.15.12:0/2818242663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e30104ad0 0x7f9e3019a1a0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.070+0000 7f9e35600640 1 --2- 172.21.15.12:0/2818242663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e301038e0 0x7f9e30199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:20.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.070+0000 7f9e35600640 1 -- 172.21.15.12:0/2818242663 >> 172.21.15.12:0/2818242663 conn(0x7f9e300fee50 msgr2=0x7f9e30107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:20.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.071+0000 7f9e35600640 1 -- 172.21.15.12:0/2818242663 shutdown_connections 2024-03-23T23:10:20.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:20.071+0000 7f9e35600640 1 -- 172.21.15.12:0/2818242663 wait complete. 2024-03-23T23:10:20.089 INFO:tasks.cephfs.filesystem:scrub status for tag:98c681b7-90f9-4d43-b9a9-e0e1f0eb0995 - {'path': '/', 'tag': '98c681b7-90f9-4d43-b9a9-e0e1f0eb0995', 'options': 'recursive,force'} 2024-03-23T23:10:20.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:20 smithi173 ceph-mon[36116]: pgmap v646: 97 pgs: 97 active+clean; 1.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:10:20.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:20 smithi173 ceph-mon[36116]: scrub queued for path: / 2024-03-23T23:10:20.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:20 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:10:20.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:20 smithi173 ceph-mon[36116]: scrub summary: active paths [/] 2024-03-23T23:10:20.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:20 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:10:20.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:20 smithi203 ceph-mon[36693]: pgmap v646: 97 pgs: 97 active+clean; 1.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:10:20.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:20 smithi203 ceph-mon[36693]: scrub queued for path: / 2024-03-23T23:10:20.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:20 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:10:20.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:20 smithi203 ceph-mon[36693]: scrub summary: active paths [/] 2024-03-23T23:10:20.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:20 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:10:21.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:20 smithi012 ceph-mon[31695]: pgmap v646: 97 pgs: 97 active+clean; 1.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:10:21.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:20 smithi012 ceph-mon[31695]: scrub queued for path: / 2024-03-23T23:10:21.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:20 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:10:21.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:20 smithi012 ceph-mon[31695]: scrub summary: active paths [/] 2024-03-23T23:10:21.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:20 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:10:21.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:21 smithi173 ceph-mon[36116]: scrub summary: idle 2024-03-23T23:10:21.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:21 smithi203 ceph-mon[36693]: scrub summary: idle 2024-03-23T23:10:22.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:21 smithi012 ceph-mon[31695]: scrub summary: idle 2024-03-23T23:10:22.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:22 smithi173 ceph-mon[36116]: pgmap v647: 97 pgs: 97 active+clean; 1.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:10:22.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:22 smithi203 ceph-mon[36693]: pgmap v647: 97 pgs: 97 active+clean; 1.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:10:23.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:22 smithi012 ceph-mon[31695]: pgmap v647: 97 pgs: 97 active+clean; 1.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:10:24.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:24 smithi173 ceph-mon[36116]: pgmap v648: 97 pgs: 97 active+clean; 1.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-03-23T23:10:24.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:24 smithi203 ceph-mon[36693]: pgmap v648: 97 pgs: 97 active+clean; 1.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-03-23T23:10:25.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:24 smithi012 ceph-mon[31695]: pgmap v648: 97 pgs: 97 active+clean; 1.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-03-23T23:10:26.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:26 smithi173 ceph-mon[36116]: pgmap v649: 97 pgs: 97 active+clean; 1.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:10:26.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:26 smithi203 ceph-mon[36693]: pgmap v649: 97 pgs: 97 active+clean; 1.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:10:27.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:26 smithi012 ceph-mon[31695]: pgmap v649: 97 pgs: 97 active+clean; 1.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:10:28.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:28 smithi173 ceph-mon[36116]: pgmap v650: 97 pgs: 97 active+clean; 1.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:10:28.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:28 smithi203 ceph-mon[36693]: pgmap v650: 97 pgs: 97 active+clean; 1.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:10:29.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:28 smithi012 ceph-mon[31695]: pgmap v650: 97 pgs: 97 active+clean; 1.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:10:30.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:30 smithi173 ceph-mon[36116]: pgmap v651: 97 pgs: 97 active+clean; 1.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:10:30.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:30 smithi203 ceph-mon[36693]: pgmap v651: 97 pgs: 97 active+clean; 1.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:10:31.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:30 smithi012 ceph-mon[31695]: pgmap v651: 97 pgs: 97 active+clean; 1.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:10:32.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:32 smithi173 ceph-mon[36116]: pgmap v652: 97 pgs: 97 active+clean; 1.8 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:10:32.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:32 smithi203 ceph-mon[36693]: pgmap v652: 97 pgs: 97 active+clean; 1.8 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:10:33.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:32 smithi012 ceph-mon[31695]: pgmap v652: 97 pgs: 97 active+clean; 1.8 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:10:33.137 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.134+0000 7f00b5400640 1 -- 172.21.15.12:0/1665813660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00b01071d0 msgr2=0x7f00b0109620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:33.137 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.134+0000 7f00b5400640 1 --2- 172.21.15.12:0/1665813660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00b01071d0 0x7f00b0109620 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f00a8009be0 tx=0x7f00a802bef0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.137 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.134+0000 7f00b5400640 1 -- 172.21.15.12:0/1665813660 shutdown_connections 2024-03-23T23:10:33.137 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.134+0000 7f00b5400640 1 --2- 172.21.15.12:0/1665813660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00b01071d0 0x7f00b0109620 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.137 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.134+0000 7f00b5400640 1 --2- 172.21.15.12:0/1665813660 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00b01048a0 0x7f00b0106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.137 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.134+0000 7f00b5400640 1 --2- 172.21.15.12:0/1665813660 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00b0101f70 0x7f00b0104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.138 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.134+0000 7f00b5400640 1 -- 172.21.15.12:0/1665813660 >> 172.21.15.12:0/1665813660 conn(0x7f00b00fb980 msgr2=0x7f00b00fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:33.138 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.134+0000 7f00b5400640 1 -- 172.21.15.12:0/1665813660 shutdown_connections 2024-03-23T23:10:33.138 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.135+0000 7f00b5400640 1 -- 172.21.15.12:0/1665813660 wait complete. 2024-03-23T23:10:33.141 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.135+0000 7f00b5400640 1 Processor -- start 2024-03-23T23:10:33.141 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.136+0000 7f00b5400640 1 -- start start 2024-03-23T23:10:33.141 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.136+0000 7f00b5400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00b0101f70 0x7f00b0199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:33.141 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.136+0000 7f00b5400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00b01048a0 0x7f00b0199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:33.141 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.136+0000 7f00b5400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00b01071d0 0x7f00b01a0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:33.141 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.136+0000 7f00b5400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b010d1d0 con 0x7f00b01071d0 2024-03-23T23:10:33.141 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.136+0000 7f00b5400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b010d050 con 0x7f00b0101f70 2024-03-23T23:10:33.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.136+0000 7f00b5400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b010d350 con 0x7f00b01048a0 2024-03-23T23:10:33.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.136+0000 7f00afe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00b0101f70 0x7f00b0199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:33.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.136+0000 7f00afe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00b0101f70 0x7f00b0199980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:46858/0 (socket says 172.21.15.12:46858) 2024-03-23T23:10:33.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.136+0000 7f00afe00640 1 -- 172.21.15.12:0/2082047653 learned_addr learned my addr 172.21.15.12:0/2082047653 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:33.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.136+0000 7f00af400640 1 --2- 172.21.15.12:0/2082047653 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00b01048a0 0x7f00b0199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:33.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.136+0000 7f00b4a00640 1 --2- 172.21.15.12:0/2082047653 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00b01071d0 0x7f00b01a0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:33.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.137+0000 7f00afe00640 1 -- 172.21.15.12:0/2082047653 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00b01048a0 msgr2=0x7f00b0199ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:33.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.137+0000 7f00afe00640 1 --2- 172.21.15.12:0/2082047653 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00b01048a0 0x7f00b0199ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.137+0000 7f00afe00640 1 -- 172.21.15.12:0/2082047653 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00b01071d0 msgr2=0x7f00b01a0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:33.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.137+0000 7f00afe00640 1 --2- 172.21.15.12:0/2082047653 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00b01071d0 0x7f00b01a0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.137+0000 7f00afe00640 1 -- 172.21.15.12:0/2082047653 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b01a15a0 con 0x7f00b0101f70 2024-03-23T23:10:33.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.137+0000 7f00af400640 1 --2- 172.21.15.12:0/2082047653 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00b01048a0 0x7f00b0199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:33.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.137+0000 7f00afe00640 1 --2- 172.21.15.12:0/2082047653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00b0101f70 0x7f00b0199980 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f009800e9f0 tx=0x7f009800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:33.143 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.137+0000 7f00acc00640 1 -- 172.21.15.12:0/2082047653 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009800cd80 con 0x7f00b0101f70 2024-03-23T23:10:33.143 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.137+0000 7f00acc00640 1 -- 172.21.15.12:0/2082047653 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0098010910 con 0x7f00b0101f70 2024-03-23T23:10:33.143 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.137+0000 7f00b5400640 1 -- 172.21.15.12:0/2082047653 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00b01a1890 con 0x7f00b0101f70 2024-03-23T23:10:33.143 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.137+0000 7f00acc00640 1 -- 172.21.15.12:0/2082047653 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0098018990 con 0x7f00b0101f70 2024-03-23T23:10:33.143 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.137+0000 7f00b5400640 1 -- 172.21.15.12:0/2082047653 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00b01a1de0 con 0x7f00b0101f70 2024-03-23T23:10:33.143 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.140+0000 7f00b965f740 1 Processor -- start 2024-03-23T23:10:33.143 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.140+0000 7f00acc00640 1 -- 172.21.15.12:0/2082047653 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0098010ac0 con 0x7f00b0101f70 2024-03-23T23:10:33.143 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.141+0000 7f00b965f740 1 -- start start 2024-03-23T23:10:33.143 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.141+0000 7f00b965f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5568d8add3f0 0x5568d8add810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:33.143 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.141+0000 7f00b965f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5568d8addde0 0x5568d8ae28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:33.143 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.141+0000 7f00b965f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8ae2f50 0x5568d8ae53e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:33.143 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.141+0000 7f00b965f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568d8aa03b0 con 0x5568d8addde0 2024-03-23T23:10:33.143 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.141+0000 7f00b965f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568d8a42830 con 0x5568d8ae2f50 2024-03-23T23:10:33.143 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.141+0000 7f00b965f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568d89eb5d0 con 0x5568d8add3f0 2024-03-23T23:10:33.144 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.141+0000 7f00b4a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5568d8add3f0 0x5568d8add810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:33.144 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.141+0000 7f00afe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5568d8addde0 0x5568d8ae28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:33.144 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.141+0000 7f00af400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8ae2f50 0x5568d8ae53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:33.144 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.141+0000 7f00af400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8ae2f50 0x5568d8ae53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:46870/0 (socket says 172.21.15.12:46870) 2024-03-23T23:10:33.144 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.141+0000 7f00b4a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5568d8add3f0 0x5568d8add810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:59042/0 (socket says 172.21.15.12:59042) 2024-03-23T23:10:33.144 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.141+0000 7f00af400640 1 -- 172.21.15.12:0/491505110 learned_addr learned my addr 172.21.15.12:0/491505110 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:33.150 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.142+0000 7f00b4a00640 1 -- 172.21.15.12:0/491505110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8ae2f50 msgr2=0x5568d8ae53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:33.150 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.142+0000 7f00b4a00640 1 --2- 172.21.15.12:0/491505110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8ae2f50 0x5568d8ae53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.150 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.142+0000 7f00b4a00640 1 -- 172.21.15.12:0/491505110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5568d8addde0 msgr2=0x5568d8ae28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:33.150 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.142+0000 7f00b4a00640 1 --2- 172.21.15.12:0/491505110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5568d8addde0 0x5568d8ae28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.150 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.142+0000 7f00b4a00640 1 -- 172.21.15.12:0/491505110 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5568d8a032c0 con 0x5568d8add3f0 2024-03-23T23:10:33.150 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.142+0000 7f00af400640 1 --2- 172.21.15.12:0/491505110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8ae2f50 0x5568d8ae53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:33.150 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.142+0000 7f00b4a00640 1 --2- 172.21.15.12:0/491505110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5568d8add3f0 0x5568d8add810 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f00a802c430 tx=0x7f00a8009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72a7ca88812622cf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:33.150 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.142+0000 7f00acc00640 1 --2- 172.21.15.12:0/2082047653 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f00940a27f0 0x7f00940a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:33.150 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.142+0000 7f00a5200640 1 -- 172.21.15.12:0/491505110 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a8043030 con 0x5568d8add3f0 2024-03-23T23:10:33.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.142+0000 7f00a5200640 1 -- 172.21.15.12:0/491505110 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00a8033580 con 0x5568d8add3f0 2024-03-23T23:10:33.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.142+0000 7f00a5200640 1 -- 172.21.15.12:0/491505110 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a802c890 con 0x5568d8add3f0 2024-03-23T23:10:33.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.143+0000 7f00acc00640 1 -- 172.21.15.12:0/2082047653 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f00980cee00 con 0x7f00b0101f70 2024-03-23T23:10:33.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.143+0000 7f00afe00640 1 --2- 172.21.15.12:0/2082047653 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f00940a27f0 0x7f00940a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:33.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.143+0000 7f00b965f740 1 -- 172.21.15.12:0/491505110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5568d8add3f0 msgr2=0x5568d8add810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:33.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.143+0000 7f00b965f740 1 --2- 172.21.15.12:0/491505110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5568d8add3f0 0x5568d8add810 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f00a802c430 tx=0x7f00a8009310 comp rx=0 tx=0).stop 2024-03-23T23:10:33.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.143+0000 7f00b965f740 1 -- 172.21.15.12:0/491505110 shutdown_connections 2024-03-23T23:10:33.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.143+0000 7f00b965f740 1 --2- 172.21.15.12:0/491505110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8ae2f50 0x5568d8ae53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.143+0000 7f00b965f740 1 --2- 172.21.15.12:0/491505110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5568d8addde0 0x5568d8ae28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.143+0000 7f00b965f740 1 --2- 172.21.15.12:0/491505110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5568d8add3f0 0x5568d8add810 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.143+0000 7f00b965f740 1 -- 172.21.15.12:0/491505110 >> 172.21.15.12:0/491505110 conn(0x5568d8ad8960 msgr2=0x5568d8ad8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:33.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.143+0000 7f00afe00640 1 --2- 172.21.15.12:0/2082047653 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f00940a27f0 0x7f00940a4cb0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f009809b500 tx=0x7f00980d2840 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:33.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.143+0000 7f00b965f740 1 -- 172.21.15.12:0/491505110 shutdown_connections 2024-03-23T23:10:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.144+0000 7f00b965f740 1 -- 172.21.15.12:0/491505110 wait complete. 2024-03-23T23:10:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.144+0000 7f00b965f740 1 -- start start 2024-03-23T23:10:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.144+0000 7f00b965f740 1 Processor -- start 2024-03-23T23:10:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.144+0000 7f00b965f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5568d8add3f0 0x5568d8ae7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.144+0000 7f00b965f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8addde0 0x5568d8ae7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.145+0000 7f00b965f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5568d8ae2f50 0x5568d8aeeac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.145+0000 7f00b965f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568d89fea20 con 0x5568d8ae2f50 2024-03-23T23:10:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.145+0000 7f00b965f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568d8a3e8d0 con 0x5568d8addde0 2024-03-23T23:10:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.145+0000 7f00b965f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568d8ae5ea0 con 0x5568d8add3f0 2024-03-23T23:10:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.145+0000 7f00b965f740 10 client.0 _ll_register_callbacks cb 0x5568d8a3b1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.145+0000 7f00b4a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5568d8add3f0 0x5568d8ae7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.145+0000 7f00af400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8addde0 0x5568d8ae7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.145+0000 7f00af400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8addde0 0x5568d8ae7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:46878/0 (socket says 172.21.15.12:46878) 2024-03-23T23:10:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.145+0000 7f00af400640 1 -- 172.21.15.12:0/3201564417 learned_addr learned my addr 172.21.15.12:0/3201564417 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.145+0000 7f00b4a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5568d8add3f0 0x5568d8ae7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:59050/0 (socket says 172.21.15.12:59050) 2024-03-23T23:10:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.145+0000 7f00b4a00640 1 --2- 172.21.15.12:0/3201564417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5568d8ae2f50 0x5568d8aeeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.145+0000 7f00af400640 1 -- 172.21.15.12:0/3201564417 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5568d8add3f0 msgr2=0x5568d8ae7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.145+0000 7f00af400640 1 --2- 172.21.15.12:0/3201564417 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5568d8add3f0 0x5568d8ae7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.145+0000 7f00af400640 1 -- 172.21.15.12:0/3201564417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5568d8ae2f50 msgr2=0x5568d8aeeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.145+0000 7f00af400640 1 --2- 172.21.15.12:0/3201564417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5568d8ae2f50 0x5568d8aeeac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.145+0000 7f00af400640 1 -- 172.21.15.12:0/3201564417 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5568d8757210 con 0x5568d8addde0 2024-03-23T23:10:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.146+0000 7f00af400640 1 --2- 172.21.15.12:0/3201564417 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8addde0 0x5568d8ae7a80 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f00a0012d20 tx=0x7f00a000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78e37f155bcc80f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.146+0000 7f009d600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:10:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.146+0000 7f009d600640 1 -- 172.21.15.12:0/3201564417 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a000fe20 con 0x5568d8addde0 2024-03-23T23:10:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.146+0000 7f009d600640 1 -- 172.21.15.12:0/3201564417 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f00a0010430 con 0x5568d8addde0 2024-03-23T23:10:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.146+0000 7f00b965f740 1 -- 172.21.15.12:0/3201564417 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f00a8033580 con 0x5568d8addde0 2024-03-23T23:10:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.146+0000 7f009d600640 1 -- 172.21.15.12:0/3201564417 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00a0004020 con 0x5568d8addde0 2024-03-23T23:10:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.146+0000 7f00b965f740 10 client.27782 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.146+0000 7f00b965f740 1 -- 172.21.15.12:0/3201564417 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5568d8823620 con 0x5568d8addde0 2024-03-23T23:10:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.146+0000 7f009d600640 1 -- 172.21.15.12:0/3201564417 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00a8033580 con 0x5568d8addde0 2024-03-23T23:10:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.147+0000 7f009d600640 1 -- 172.21.15.12:0/3201564417 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f00a0025020 con 0x5568d8addde0 2024-03-23T23:10:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.147+0000 7f00b965f740 10 client.27782 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.147+0000 7f00b965f740 10 client.27782 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.147+0000 7f00b965f740 20 client.27782 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.147+0000 7f00b965f740 1 --2- 172.21.15.12:0/3201564417 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5568d8af3400 0x5568d8af57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.147+0000 7f00b965f740 4 client.27782 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:10:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.147+0000 7f00b965f740 1 -- 172.21.15.12:0/3201564417 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5568d8a4fd70 con 0x5568d8af3400 2024-03-23T23:10:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.147+0000 7f00b4a00640 1 --2- 172.21.15.12:0/3201564417 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5568d8af3400 0x5568d8af57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.148+0000 7f00b4a00640 1 --2- 172.21.15.12:0/3201564417 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5568d8af3400 0x5568d8af57f0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51ed185481fda08f server_cookie=c5b9f60550a907c0 in_seq=0 out_seq=0 2024-03-23T23:10:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.148+0000 7f009d600640 10 client.27782 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.148+0000 7f009d600640 1 -- 172.21.15.12:0/3201564417 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5568d8a4fd70 con 0x5568d8af3400 2024-03-23T23:10:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.148+0000 7f009d600640 10 client.27782 handle_command_reply: tid=0 2024-03-23T23:10:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.148+0000 7f00b965f740 1 client.27782 shutdown 2024-03-23T23:10:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.149+0000 7f00b965f740 1 -- 172.21.15.12:0/3201564417 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8addde0 msgr2=0x5568d8ae7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.149+0000 7f00b965f740 1 --2- 172.21.15.12:0/3201564417 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8addde0 0x5568d8ae7a80 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f00a0012d20 tx=0x7f00a000f790 comp rx=0 tx=0).stop 2024-03-23T23:10:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.149+0000 7f00b965f740 1 -- 172.21.15.12:0/3201564417 shutdown_connections 2024-03-23T23:10:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.149+0000 7f00b965f740 1 --2- 172.21.15.12:0/3201564417 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5568d8af3400 0x5568d8af57f0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.149+0000 7f00b965f740 1 --2- 172.21.15.12:0/3201564417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5568d8ae2f50 0x5568d8aeeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.149+0000 7f00b965f740 1 --2- 172.21.15.12:0/3201564417 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8addde0 0x5568d8ae7a80 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.149+0000 7f00b965f740 1 --2- 172.21.15.12:0/3201564417 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5568d8add3f0 0x5568d8ae7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.149+0000 7f009d600640 0 client.27782 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.149+0000 7f00b965f740 1 -- 172.21.15.12:0/3201564417 >> 172.21.15.12:0/3201564417 conn(0x5568d8ad8960 msgr2=0x5568d8ad7b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.149+0000 7f00b965f740 1 -- 172.21.15.12:0/3201564417 shutdown_connections 2024-03-23T23:10:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.149+0000 7f00b965f740 1 -- 172.21.15.12:0/3201564417 wait complete. 2024-03-23T23:10:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.149+0000 7f00b965f740 20 client.27782 trim_cache size 0 max 16384 2024-03-23T23:10:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.164+0000 7f00b965f740 1 Processor -- start 2024-03-23T23:10:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.165+0000 7f00b965f740 1 -- start start 2024-03-23T23:10:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.165+0000 7f00b965f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5568d8add3f0 0x5568d8adf100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.165+0000 7f00b965f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8addde0 0x5568d8adf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.165+0000 7f00b965f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5568d8ae2f50 0x5568d8aeea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.165+0000 7f00b965f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b01ace60 con 0x5568d8ae2f50 2024-03-23T23:10:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.165+0000 7f00b965f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b01aa960 con 0x5568d8addde0 2024-03-23T23:10:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.165+0000 7f00b965f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b01a8a00 con 0x5568d8add3f0 2024-03-23T23:10:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.165+0000 7f00af400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8addde0 0x5568d8adf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.165+0000 7f00af400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8addde0 0x5568d8adf640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:46882/0 (socket says 172.21.15.12:46882) 2024-03-23T23:10:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.165+0000 7f00af400640 1 -- 172.21.15.12:0/3272293001 learned_addr learned my addr 172.21.15.12:0/3272293001 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.165+0000 7f00b4a00640 1 --2- 172.21.15.12:0/3272293001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5568d8add3f0 0x5568d8adf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.165+0000 7f00b4a00640 1 --2- 172.21.15.12:0/3272293001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5568d8ae2f50 0x5568d8aeea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.166+0000 7f00af400640 1 -- 172.21.15.12:0/3272293001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5568d8add3f0 msgr2=0x5568d8adf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.166+0000 7f00af400640 1 --2- 172.21.15.12:0/3272293001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5568d8add3f0 0x5568d8adf100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.166+0000 7f00af400640 1 -- 172.21.15.12:0/3272293001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5568d8ae2f50 msgr2=0x5568d8aeea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.166+0000 7f00af400640 1 --2- 172.21.15.12:0/3272293001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5568d8ae2f50 0x5568d8aeea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.166+0000 7f00af400640 1 -- 172.21.15.12:0/3272293001 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5568d8add7d0 con 0x5568d8addde0 2024-03-23T23:10:33.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.166+0000 7f00b4a00640 1 --2- 172.21.15.12:0/3272293001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5568d8add3f0 0x5568d8adf100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:10:33.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.166+0000 7f00af400640 1 --2- 172.21.15.12:0/3272293001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8addde0 0x5568d8adf640 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f00a00035f0 tx=0x7f00a0012990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b91ba5cca7f3cf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.166+0000 7f009fe00640 1 -- 172.21.15.12:0/3272293001 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a00041c0 con 0x5568d8addde0 2024-03-23T23:10:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.166+0000 7f009fe00640 1 -- 172.21.15.12:0/3272293001 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00a002b020 con 0x5568d8addde0 2024-03-23T23:10:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.166+0000 7f009fe00640 1 -- 172.21.15.12:0/3272293001 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a0003ac0 con 0x5568d8addde0 2024-03-23T23:10:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.167+0000 7f00b965f740 1 -- 172.21.15.12:0/3272293001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8addde0 msgr2=0x5568d8adf640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.167+0000 7f00b965f740 1 --2- 172.21.15.12:0/3272293001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8addde0 0x5568d8adf640 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f00a00035f0 tx=0x7f00a0012990 comp rx=0 tx=0).stop 2024-03-23T23:10:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.167+0000 7f00b965f740 1 -- 172.21.15.12:0/3272293001 shutdown_connections 2024-03-23T23:10:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.167+0000 7f00b965f740 1 --2- 172.21.15.12:0/3272293001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5568d8ae2f50 0x5568d8aeea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.167+0000 7f00b965f740 1 --2- 172.21.15.12:0/3272293001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8addde0 0x5568d8adf640 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.167+0000 7f00b965f740 1 --2- 172.21.15.12:0/3272293001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5568d8add3f0 0x5568d8adf100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.167+0000 7f00b965f740 1 -- 172.21.15.12:0/3272293001 >> 172.21.15.12:0/3272293001 conn(0x5568d8ad8960 msgr2=0x5568d8ad8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:33.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.167+0000 7f00b965f740 1 -- 172.21.15.12:0/3272293001 shutdown_connections 2024-03-23T23:10:33.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.167+0000 7f00b965f740 1 -- 172.21.15.12:0/3272293001 wait complete. 2024-03-23T23:10:33.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.167+0000 7f00b965f740 1 -- start start 2024-03-23T23:10:33.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.168+0000 7f00b965f740 1 Processor -- start 2024-03-23T23:10:33.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.168+0000 7f00b965f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8add3f0 0x5568d8aec840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:33.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.168+0000 7f00b965f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5568d8addde0 0x5568d8aecd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:33.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.168+0000 7f00b965f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5568d8ae2f50 0x5568d8aedd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:33.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.168+0000 7f00b965f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b01a9000 con 0x5568d8addde0 2024-03-23T23:10:33.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.168+0000 7f00b965f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b01aa300 con 0x5568d8add3f0 2024-03-23T23:10:33.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.168+0000 7f00b965f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b01a9640 con 0x5568d8ae2f50 2024-03-23T23:10:33.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.168+0000 7f00b965f740 10 client.0 _ll_register_callbacks cb 0x7f00a0009c40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:33.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.168+0000 7f00b4a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8add3f0 0x5568d8aec840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:33.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.168+0000 7f00b4a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8add3f0 0x5568d8aec840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:46898/0 (socket says 172.21.15.12:46898) 2024-03-23T23:10:33.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.168+0000 7f00b4a00640 1 -- 172.21.15.12:0/2823373855 learned_addr learned my addr 172.21.15.12:0/2823373855 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:33.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.168+0000 7f00b4a00640 1 --2- 172.21.15.12:0/2823373855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5568d8ae2f50 0x5568d8aedd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:33.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.168+0000 7f00af400640 1 --2- 172.21.15.12:0/2823373855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5568d8addde0 0x5568d8aecd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:33.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.169+0000 7f00b4a00640 1 -- 172.21.15.12:0/2823373855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5568d8ae2f50 msgr2=0x5568d8aedd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:33.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.169+0000 7f00b4a00640 1 --2- 172.21.15.12:0/2823373855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5568d8ae2f50 0x5568d8aedd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.169+0000 7f00b4a00640 1 -- 172.21.15.12:0/2823373855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5568d8addde0 msgr2=0x5568d8aecd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:33.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.169+0000 7f00b4a00640 1 --2- 172.21.15.12:0/2823373855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5568d8addde0 0x5568d8aecd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.169+0000 7f00b4a00640 1 -- 172.21.15.12:0/2823373855 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5568d8aee310 con 0x5568d8add3f0 2024-03-23T23:10:33.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.169+0000 7f00b4a00640 1 --2- 172.21.15.12:0/2823373855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8add3f0 0x5568d8aec840 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f00a8033a70 tx=0x7f00a8041040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d947b2d37f3f5b00 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:33.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.169+0000 7f009d600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:10:33.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.169+0000 7f009d600640 1 -- 172.21.15.12:0/2823373855 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a8043040 con 0x5568d8add3f0 2024-03-23T23:10:33.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.169+0000 7f009d600640 1 -- 172.21.15.12:0/2823373855 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f00a80564b0 con 0x5568d8add3f0 2024-03-23T23:10:33.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.169+0000 7f00b965f740 1 -- 172.21.15.12:0/2823373855 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f00a002b020 con 0x5568d8add3f0 2024-03-23T23:10:33.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.170+0000 7f009d600640 1 -- 172.21.15.12:0/2823373855 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00a802daf0 con 0x5568d8add3f0 2024-03-23T23:10:33.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.170+0000 7f00b965f740 10 client.27788 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:33.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.170+0000 7f00b965f740 1 -- 172.21.15.12:0/2823373855 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5568d8b4f220 con 0x5568d8add3f0 2024-03-23T23:10:33.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.170+0000 7f009d600640 1 -- 172.21.15.12:0/2823373855 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00a002b020 con 0x5568d8add3f0 2024-03-23T23:10:33.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.170+0000 7f009d600640 1 -- 172.21.15.12:0/2823373855 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f00a8062030 con 0x5568d8add3f0 2024-03-23T23:10:33.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.170+0000 7f00b965f740 10 client.27788 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:33.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.170+0000 7f00b965f740 10 client.27788 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:33.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.170+0000 7f00b965f740 20 client.27788 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:33.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.170+0000 7f00b965f740 1 --2- 172.21.15.12:0/2823373855 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5568d8b4f780 0x5568d8b51b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:33.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.170+0000 7f00b965f740 4 client.27788 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:10:33.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.170+0000 7f00b965f740 1 -- 172.21.15.12:0/2823373855 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5568d8a4a0c0 con 0x5568d8b4f780 2024-03-23T23:10:33.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.170+0000 7f00af400640 1 --2- 172.21.15.12:0/2823373855 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5568d8b4f780 0x5568d8b51b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:33.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.171+0000 7f00af400640 1 --2- 172.21.15.12:0/2823373855 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5568d8b4f780 0x5568d8b51b70 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98dda6c21c69e26c server_cookie=c27d79db6da3a24a in_seq=0 out_seq=0 2024-03-23T23:10:33.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.171+0000 7f009d600640 10 client.27788 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:33.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.184+0000 7f009d600640 1 -- 172.21.15.12:0/2823373855 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5568d8a4a0c0 con 0x5568d8b4f780 2024-03-23T23:10:33.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.184+0000 7f009d600640 10 client.27788 handle_command_reply: tid=0 2024-03-23T23:10:33.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.184+0000 7f00b965f740 1 client.27788 shutdown 2024-03-23T23:10:33.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.184+0000 7f00b965f740 1 -- 172.21.15.12:0/2823373855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8add3f0 msgr2=0x5568d8aec840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:33.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.184+0000 7f00b965f740 1 --2- 172.21.15.12:0/2823373855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8add3f0 0x5568d8aec840 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f00a8033a70 tx=0x7f00a8041040 comp rx=0 tx=0).stop 2024-03-23T23:10:33.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.184+0000 7f00b965f740 1 -- 172.21.15.12:0/2823373855 shutdown_connections 2024-03-23T23:10:33.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.184+0000 7f00b965f740 1 --2- 172.21.15.12:0/2823373855 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5568d8b4f780 0x5568d8b51b70 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.184+0000 7f00b965f740 1 --2- 172.21.15.12:0/2823373855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5568d8ae2f50 0x5568d8aedd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.184+0000 7f00b965f740 1 --2- 172.21.15.12:0/2823373855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5568d8addde0 0x5568d8aecd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.184+0000 7f00b965f740 1 --2- 172.21.15.12:0/2823373855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568d8add3f0 0x5568d8aec840 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.184+0000 7f009d600640 0 client.27788 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:33.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.184+0000 7f00b965f740 1 -- 172.21.15.12:0/2823373855 >> 172.21.15.12:0/2823373855 conn(0x5568d8ad8960 msgr2=0x5568d8ae74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:33.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.184+0000 7f00b965f740 1 -- 172.21.15.12:0/2823373855 shutdown_connections 2024-03-23T23:10:33.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.184+0000 7f00b965f740 1 -- 172.21.15.12:0/2823373855 wait complete. 2024-03-23T23:10:33.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.184+0000 7f00b965f740 20 client.27788 trim_cache size 0 max 16384 2024-03-23T23:10:33.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:10:33.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:10:33.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:10:33.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:10:33.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.185+0000 7f00b5400640 1 -- 172.21.15.12:0/2082047653 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f00940a27f0 msgr2=0x7f00940a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:33.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.185+0000 7f00b5400640 1 --2- 172.21.15.12:0/2082047653 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f00940a27f0 0x7f00940a4cb0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f009809b500 tx=0x7f00980d2840 comp rx=0 tx=0).stop 2024-03-23T23:10:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.185+0000 7f00b5400640 1 -- 172.21.15.12:0/2082047653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00b0101f70 msgr2=0x7f00b0199980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.185+0000 7f00b5400640 1 --2- 172.21.15.12:0/2082047653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00b0101f70 0x7f00b0199980 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f009800e9f0 tx=0x7f009800eeb0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.185+0000 7f00b5400640 1 -- 172.21.15.12:0/2082047653 shutdown_connections 2024-03-23T23:10:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.185+0000 7f00b5400640 1 --2- 172.21.15.12:0/2082047653 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f00940a27f0 0x7f00940a4cb0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.185+0000 7f00b5400640 1 --2- 172.21.15.12:0/2082047653 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00b01071d0 0x7f00b01a0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.185+0000 7f00b5400640 1 --2- 172.21.15.12:0/2082047653 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00b01048a0 0x7f00b0199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.185+0000 7f00b5400640 1 --2- 172.21.15.12:0/2082047653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00b0101f70 0x7f00b0199980 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.185+0000 7f00b5400640 1 -- 172.21.15.12:0/2082047653 >> 172.21.15.12:0/2082047653 conn(0x7f00b00fb980 msgr2=0x7f00b01015e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.186+0000 7f00b5400640 1 -- 172.21.15.12:0/2082047653 shutdown_connections 2024-03-23T23:10:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:33.186+0000 7f00b5400640 1 -- 172.21.15.12:0/2082047653 wait complete. 2024-03-23T23:10:34.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:34 smithi173 ceph-mon[36116]: pgmap v653: 97 pgs: 97 active+clean; 1.8 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:10:34.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:34 smithi203 ceph-mon[36693]: pgmap v653: 97 pgs: 97 active+clean; 1.8 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:10:35.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:34 smithi012 ceph-mon[31695]: pgmap v653: 97 pgs: 97 active+clean; 1.8 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:10:36.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:36 smithi203 ceph-mon[36693]: pgmap v654: 97 pgs: 97 active+clean; 1.9 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-03-23T23:10:37.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:36 smithi012 ceph-mon[31695]: pgmap v654: 97 pgs: 97 active+clean; 1.9 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-03-23T23:10:37.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:36 smithi173 ceph-mon[36116]: pgmap v654: 97 pgs: 97 active+clean; 1.9 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-03-23T23:10:38.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:38 smithi203 ceph-mon[36693]: pgmap v655: 97 pgs: 97 active+clean; 1.9 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:10:39.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:38 smithi012 ceph-mon[31695]: pgmap v655: 97 pgs: 97 active+clean; 1.9 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:10:39.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:38 smithi173 ceph-mon[36116]: pgmap v655: 97 pgs: 97 active+clean; 1.9 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:10:40.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:40 smithi203 ceph-mon[36693]: pgmap v656: 97 pgs: 97 active+clean; 1.9 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:10:41.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:40 smithi012 ceph-mon[31695]: pgmap v656: 97 pgs: 97 active+clean; 1.9 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:10:41.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:40 smithi173 ceph-mon[36116]: pgmap v656: 97 pgs: 97 active+clean; 1.9 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:10:42.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:42 smithi203 ceph-mon[36693]: pgmap v657: 97 pgs: 97 active+clean; 1.9 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:10:43.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:42 smithi012 ceph-mon[31695]: pgmap v657: 97 pgs: 97 active+clean; 1.9 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:10:43.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:42 smithi173 ceph-mon[36116]: pgmap v657: 97 pgs: 97 active+clean; 1.9 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:10:44.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:44 smithi203 ceph-mon[36693]: pgmap v658: 97 pgs: 97 active+clean; 1.9 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:10:45.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:44 smithi012 ceph-mon[31695]: pgmap v658: 97 pgs: 97 active+clean; 1.9 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:10:45.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:44 smithi173 ceph-mon[36116]: pgmap v658: 97 pgs: 97 active+clean; 1.9 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:10:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.331+0000 7fc6eaa00640 1 -- 172.21.15.12:0/2335406060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6e41054b0 msgr2=0x7fc6e410bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.331+0000 7fc6eaa00640 1 --2- 172.21.15.12:0/2335406060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6e41054b0 0x7fc6e410bd40 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc009be0 tx=0x7fc6dc02bef0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.331+0000 7fc6eaa00640 1 -- 172.21.15.12:0/2335406060 shutdown_connections 2024-03-23T23:10:46.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.331+0000 7fc6eaa00640 1 --2- 172.21.15.12:0/2335406060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6e41054b0 0x7fc6e410bd40 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.331+0000 7fc6eaa00640 1 --2- 172.21.15.12:0/2335406060 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6e4104af0 0x7fc6e4104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.331+0000 7fc6eaa00640 1 --2- 172.21.15.12:0/2335406060 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6e4103900 0x7fc6e4103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.331+0000 7fc6eaa00640 1 -- 172.21.15.12:0/2335406060 >> 172.21.15.12:0/2335406060 conn(0x7fc6e40fee90 msgr2=0x7fc6e41012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:46.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.331+0000 7fc6eaa00640 1 -- 172.21.15.12:0/2335406060 shutdown_connections 2024-03-23T23:10:46.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.331+0000 7fc6eaa00640 1 -- 172.21.15.12:0/2335406060 wait complete. 2024-03-23T23:10:46.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.332+0000 7fc6eaa00640 1 Processor -- start 2024-03-23T23:10:46.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.333+0000 7fc6eaa00640 1 -- start start 2024-03-23T23:10:46.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.333+0000 7fc6eaa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6e4103900 0x7fc6e4199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:46.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.333+0000 7fc6eaa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6e4104af0 0x7fc6e419a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:46.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.333+0000 7fc6eaa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6e41054b0 0x7fc6e41a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:46.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.333+0000 7fc6eaa00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e410f580 con 0x7fc6e41054b0 2024-03-23T23:10:46.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.333+0000 7fc6eaa00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e410f400 con 0x7fc6e4104af0 2024-03-23T23:10:46.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.333+0000 7fc6eaa00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e410f700 con 0x7fc6e4103900 2024-03-23T23:10:46.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.333+0000 7fc6e3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6e4103900 0x7fc6e4199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:46.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.333+0000 7fc6dbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6e4104af0 0x7fc6e419a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:46.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.333+0000 7fc6dbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6e4104af0 0x7fc6e419a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37552/0 (socket says 172.21.15.12:37552) 2024-03-23T23:10:46.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.333+0000 7fc6dbe00640 1 -- 172.21.15.12:0/3960085678 learned_addr learned my addr 172.21.15.12:0/3960085678 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:46.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.333+0000 7fc6e3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6e4103900 0x7fc6e4199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54010/0 (socket says 172.21.15.12:54010) 2024-03-23T23:10:46.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.333+0000 7fc6e8a00640 1 --2- 172.21.15.12:0/3960085678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6e41054b0 0x7fc6e41a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:46.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.334+0000 7fc6e3e00640 1 -- 172.21.15.12:0/3960085678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6e4104af0 msgr2=0x7fc6e419a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:46.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.334+0000 7fc6e3e00640 1 --2- 172.21.15.12:0/3960085678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6e4104af0 0x7fc6e419a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.334+0000 7fc6e3e00640 1 -- 172.21.15.12:0/3960085678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6e41054b0 msgr2=0x7fc6e41a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:46.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.334+0000 7fc6e3e00640 1 --2- 172.21.15.12:0/3960085678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6e41054b0 0x7fc6e41a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.334+0000 7fc6e3e00640 1 -- 172.21.15.12:0/3960085678 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e41a1730 con 0x7fc6e4103900 2024-03-23T23:10:46.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.334+0000 7fc6e8a00640 1 --2- 172.21.15.12:0/3960085678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6e41054b0 0x7fc6e41a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:10:46.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.334+0000 7fc6e3e00640 1 --2- 172.21.15.12:0/3960085678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6e4103900 0x7fc6e4199b10 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc6d400e9c0 tx=0x7fc6d400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:46.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.334+0000 7fc6e1600640 1 -- 172.21.15.12:0/3960085678 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d400ccc0 con 0x7fc6e4103900 2024-03-23T23:10:46.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.334+0000 7fc6e1600640 1 -- 172.21.15.12:0/3960085678 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc6d40035f0 con 0x7fc6e4103900 2024-03-23T23:10:46.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.334+0000 7fc6e1600640 1 -- 172.21.15.12:0/3960085678 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d400ce60 con 0x7fc6e4103900 2024-03-23T23:10:46.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.334+0000 7fc6eaa00640 1 -- 172.21.15.12:0/3960085678 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e41a1a20 con 0x7fc6e4103900 2024-03-23T23:10:46.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.335+0000 7fc6eaa00640 1 -- 172.21.15.12:0/3960085678 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e41a1f20 con 0x7fc6e4103900 2024-03-23T23:10:46.340 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.338+0000 7fc6ed837740 1 Processor -- start 2024-03-23T23:10:46.340 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.338+0000 7fc6e1600640 1 -- 172.21.15.12:0/3960085678 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc6d40037a0 con 0x7fc6e4103900 2024-03-23T23:10:46.340 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.338+0000 7fc6ed837740 1 -- start start 2024-03-23T23:10:46.340 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.339+0000 7fc6ed837740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c79e1a91e0 0x55c79e1a9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:46.340 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.339+0000 7fc6ed837740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c79e1a9bd0 0x55c79e1ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:46.340 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.339+0000 7fc6ed837740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c79e1aed40 0x55c79e1b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:46.340 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.339+0000 7fc6ed837740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c79e16c3b0 con 0x55c79e1a9bd0 2024-03-23T23:10:46.340 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.339+0000 7fc6ed837740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c79e10e830 con 0x55c79e1aed40 2024-03-23T23:10:46.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.339+0000 7fc6ed837740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c79e0b75d0 con 0x55c79e1a91e0 2024-03-23T23:10:46.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.339+0000 7fc6e8a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c79e1a91e0 0x55c79e1a9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:46.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.339+0000 7fc6e8a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c79e1a91e0 0x55c79e1a9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54020/0 (socket says 172.21.15.12:54020) 2024-03-23T23:10:46.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.339+0000 7fc6e8a00640 1 -- 172.21.15.12:0/3884677448 learned_addr learned my addr 172.21.15.12:0/3884677448 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:46.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.339+0000 7fc6e3e00640 1 --2- 172.21.15.12:0/3884677448 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c79e1a9bd0 0x55c79e1ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:46.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.339+0000 7fc6dbe00640 1 --2- 172.21.15.12:0/3884677448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c79e1aed40 0x55c79e1b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:46.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.339+0000 7fc6e8a00640 1 -- 172.21.15.12:0/3884677448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c79e1aed40 msgr2=0x55c79e1b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:46.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.339+0000 7fc6e8a00640 1 --2- 172.21.15.12:0/3884677448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c79e1aed40 0x55c79e1b11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.339+0000 7fc6e8a00640 1 -- 172.21.15.12:0/3884677448 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c79e1a9bd0 msgr2=0x55c79e1ae6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:46.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.339+0000 7fc6e8a00640 1 --2- 172.21.15.12:0/3884677448 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c79e1a9bd0 0x55c79e1ae6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.339+0000 7fc6e8a00640 1 -- 172.21.15.12:0/3884677448 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c79e0cf2c0 con 0x55c79e1a91e0 2024-03-23T23:10:46.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.339+0000 7fc6dbe00640 1 --2- 172.21.15.12:0/3884677448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c79e1aed40 0x55c79e1b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:10:46.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.340+0000 7fc6e8a00640 1 --2- 172.21.15.12:0/3884677448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c79e1a91e0 0x55c79e1a9600 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc034fd0 tx=0x7fc6dc009300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6e67c1650222ace server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:46.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.340+0000 7fc6d9200640 1 -- 172.21.15.12:0/3884677448 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc043070 con 0x55c79e1a91e0 2024-03-23T23:10:46.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.340+0000 7fc6d9200640 1 -- 172.21.15.12:0/3884677448 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc6dc033cd0 con 0x55c79e1a91e0 2024-03-23T23:10:46.342 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.340+0000 7fc6d9200640 1 -- 172.21.15.12:0/3884677448 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc02cc70 con 0x55c79e1a91e0 2024-03-23T23:10:46.342 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.340+0000 7fc6e1600640 1 --2- 172.21.15.12:0/3960085678 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc6c80a2920 0x7fc6c80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:46.342 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.340+0000 7fc6e1600640 1 -- 172.21.15.12:0/3960085678 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fc6d40cee40 con 0x7fc6e4103900 2024-03-23T23:10:46.342 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.340+0000 7fc6e3e00640 1 --2- 172.21.15.12:0/3960085678 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc6c80a2920 0x7fc6c80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:46.347 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.341+0000 7fc6ed837740 1 -- 172.21.15.12:0/3884677448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c79e1a91e0 msgr2=0x55c79e1a9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:46.347 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.341+0000 7fc6ed837740 1 --2- 172.21.15.12:0/3884677448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c79e1a91e0 0x55c79e1a9600 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc034fd0 tx=0x7fc6dc009300 comp rx=0 tx=0).stop 2024-03-23T23:10:46.347 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.341+0000 7fc6ed837740 1 -- 172.21.15.12:0/3884677448 shutdown_connections 2024-03-23T23:10:46.347 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.341+0000 7fc6ed837740 1 --2- 172.21.15.12:0/3884677448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c79e1aed40 0x55c79e1b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.347 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.341+0000 7fc6ed837740 1 --2- 172.21.15.12:0/3884677448 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c79e1a9bd0 0x55c79e1ae6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.347 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.341+0000 7fc6ed837740 1 --2- 172.21.15.12:0/3884677448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c79e1a91e0 0x55c79e1a9600 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.347 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.341+0000 7fc6ed837740 1 -- 172.21.15.12:0/3884677448 >> 172.21.15.12:0/3884677448 conn(0x55c79e1a4750 msgr2=0x55c79e1a4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:46.347 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.341+0000 7fc6ed837740 1 -- 172.21.15.12:0/3884677448 shutdown_connections 2024-03-23T23:10:46.347 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.341+0000 7fc6ed837740 1 -- 172.21.15.12:0/3884677448 wait complete. 2024-03-23T23:10:46.347 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.341+0000 7fc6e3e00640 1 --2- 172.21.15.12:0/3960085678 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc6c80a2920 0x7fc6c80a4de0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fc6d40a3a00 tx=0x7fc6d4003e10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:46.347 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.342+0000 7fc6ed837740 1 -- start start 2024-03-23T23:10:46.347 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.342+0000 7fc6ed837740 1 Processor -- start 2024-03-23T23:10:46.348 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.342+0000 7fc6ed837740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c79e1a91e0 0x55c79e1b3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:46.348 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.342+0000 7fc6ed837740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c79e1a9bd0 0x55c79e1b3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:46.348 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.342+0000 7fc6ed837740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c79e1aed40 0x55c79e1ba890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:46.348 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.342+0000 7fc6ed837740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c79e0caa20 con 0x55c79e1a91e0 2024-03-23T23:10:46.348 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.342+0000 7fc6ed837740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c79e10a8d0 con 0x55c79e1aed40 2024-03-23T23:10:46.348 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.342+0000 7fc6ed837740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c79e1b1c90 con 0x55c79e1a9bd0 2024-03-23T23:10:46.348 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.342+0000 7fc6ed837740 10 client.0 _ll_register_callbacks cb 0x55c79e1071c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:46.348 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.342+0000 7fc6e8a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c79e1a91e0 0x55c79e1b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:46.348 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.343+0000 7fc6dbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c79e1a9bd0 0x55c79e1b3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:46.348 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.343+0000 7fc6e8a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c79e1a91e0 0x55c79e1b3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54494/0 (socket says 172.21.15.12:54494) 2024-03-23T23:10:46.348 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.343+0000 7fc6e8a00640 1 -- 172.21.15.12:0/3072151840 learned_addr learned my addr 172.21.15.12:0/3072151840 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:46.348 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.343+0000 7fc6e8a00640 1 --2- 172.21.15.12:0/3072151840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c79e1aed40 0x55c79e1ba890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:46.348 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.343+0000 7fc6dbe00640 1 -- 172.21.15.12:0/3072151840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c79e1aed40 msgr2=0x55c79e1ba890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:46.348 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.343+0000 7fc6dbe00640 1 --2- 172.21.15.12:0/3072151840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c79e1aed40 0x55c79e1ba890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.343+0000 7fc6dbe00640 1 -- 172.21.15.12:0/3072151840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c79e1a91e0 msgr2=0x55c79e1b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.343+0000 7fc6dbe00640 1 --2- 172.21.15.12:0/3072151840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c79e1a91e0 0x55c79e1b3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.343+0000 7fc6dbe00640 1 -- 172.21.15.12:0/3072151840 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c79de23210 con 0x55c79e1a9bd0 2024-03-23T23:10:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.343+0000 7fc6e8a00640 1 --2- 172.21.15.12:0/3072151840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c79e1a91e0 0x55c79e1b3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.343+0000 7fc6dbe00640 1 --2- 172.21.15.12:0/3072151840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c79e1a9bd0 0x55c79e1b3850 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fc6e419b020 tx=0x7fc6cc012aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9b5a3490112495f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.343+0000 7fc6d1600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:10:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.344+0000 7fc6d1600640 1 -- 172.21.15.12:0/3072151840 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc00f660 con 0x55c79e1a9bd0 2024-03-23T23:10:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.344+0000 7fc6d1600640 1 -- 172.21.15.12:0/3072151840 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fc6cc00f800 con 0x55c79e1a9bd0 2024-03-23T23:10:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.344+0000 7fc6ed837740 1 -- 172.21.15.12:0/3072151840 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc6dc033cd0 con 0x55c79e1a9bd0 2024-03-23T23:10:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.344+0000 7fc6d1600640 1 -- 172.21.15.12:0/3072151840 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6cc00fa00 con 0x55c79e1a9bd0 2024-03-23T23:10:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.344+0000 7fc6ed837740 10 client.27760 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.344+0000 7fc6ed837740 1 -- 172.21.15.12:0/3072151840 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c79deef620 con 0x55c79e1a9bd0 2024-03-23T23:10:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.344+0000 7fc6d1600640 1 -- 172.21.15.12:0/3072151840 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc6dc033cd0 con 0x55c79e1a9bd0 2024-03-23T23:10:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.344+0000 7fc6d1600640 1 -- 172.21.15.12:0/3072151840 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fc6cc010ce0 con 0x55c79e1a9bd0 2024-03-23T23:10:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.345+0000 7fc6ed837740 10 client.27760 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.345+0000 7fc6ed837740 10 client.27760 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.345+0000 7fc6ed837740 20 client.27760 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.345+0000 7fc6ed837740 1 --2- 172.21.15.12:0/3072151840 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c79e1bf180 0x55c79e1c15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.345+0000 7fc6ed837740 4 client.27760 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:10:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.345+0000 7fc6ed837740 1 -- 172.21.15.12:0/3072151840 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c79e11bd70 con 0x55c79e1bf180 2024-03-23T23:10:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.345+0000 7fc6e8a00640 1 --2- 172.21.15.12:0/3072151840 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c79e1bf180 0x55c79e1c15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.345+0000 7fc6e8a00640 1 --2- 172.21.15.12:0/3072151840 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c79e1bf180 0x55c79e1c15a0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b574f0e14b13cd01 server_cookie=1c54004eac70e28c in_seq=0 out_seq=0 2024-03-23T23:10:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.345+0000 7fc6d1600640 10 client.27760 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.346+0000 7fc6d1600640 1 -- 172.21.15.12:0/3072151840 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c79e11bd70 con 0x55c79e1bf180 2024-03-23T23:10:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.346+0000 7fc6d1600640 10 client.27760 handle_command_reply: tid=0 2024-03-23T23:10:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.346+0000 7fc6ed837740 1 client.27760 shutdown 2024-03-23T23:10:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.346+0000 7fc6ed837740 1 -- 172.21.15.12:0/3072151840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c79e1a9bd0 msgr2=0x55c79e1b3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:46.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.346+0000 7fc6ed837740 1 --2- 172.21.15.12:0/3072151840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c79e1a9bd0 0x55c79e1b3850 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fc6e419b020 tx=0x7fc6cc012aa0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.346+0000 7fc6ed837740 1 -- 172.21.15.12:0/3072151840 shutdown_connections 2024-03-23T23:10:46.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.346+0000 7fc6ed837740 1 --2- 172.21.15.12:0/3072151840 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c79e1bf180 0x55c79e1c15a0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.346+0000 7fc6ed837740 1 --2- 172.21.15.12:0/3072151840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c79e1aed40 0x55c79e1ba890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.346+0000 7fc6ed837740 1 --2- 172.21.15.12:0/3072151840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c79e1a9bd0 0x55c79e1b3850 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.346+0000 7fc6ed837740 1 --2- 172.21.15.12:0/3072151840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c79e1a91e0 0x55c79e1b3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.346+0000 7fc6d1600640 0 client.27760 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:46.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.346+0000 7fc6ed837740 1 -- 172.21.15.12:0/3072151840 >> 172.21.15.12:0/3072151840 conn(0x55c79e1a4750 msgr2=0x55c79e1a3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:46.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.347+0000 7fc6ed837740 1 -- 172.21.15.12:0/3072151840 shutdown_connections 2024-03-23T23:10:46.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.347+0000 7fc6ed837740 1 -- 172.21.15.12:0/3072151840 wait complete. 2024-03-23T23:10:46.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.347+0000 7fc6ed837740 20 client.27760 trim_cache size 0 max 16384 2024-03-23T23:10:46.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.362+0000 7fc6ed837740 1 Processor -- start 2024-03-23T23:10:46.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.362+0000 7fc6ed837740 1 -- start start 2024-03-23T23:10:46.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.362+0000 7fc6ed837740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c79e1a91e0 0x55c79e1a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:46.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.362+0000 7fc6ed837740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c79e1a9bd0 0x55c79e1ba850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:46.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.362+0000 7fc6ed837740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c79e1aed40 0x55c79e1a3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:46.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.362+0000 7fc6ed837740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e41ad130 con 0x55c79e1a91e0 2024-03-23T23:10:46.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.362+0000 7fc6ed837740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e41aac30 con 0x55c79e1aed40 2024-03-23T23:10:46.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.362+0000 7fc6ed837740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e41a8cd0 con 0x55c79e1a9bd0 2024-03-23T23:10:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.363+0000 7fc6dbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c79e1a9bd0 0x55c79e1ba850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.363+0000 7fc6e8a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c79e1a91e0 0x55c79e1a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.363+0000 7fc6dbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c79e1a9bd0 0x55c79e1ba850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54044/0 (socket says 172.21.15.12:54044) 2024-03-23T23:10:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.363+0000 7fc6dbe00640 1 -- 172.21.15.12:0/3497065889 learned_addr learned my addr 172.21.15.12:0/3497065889 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.363+0000 7fc6e8a00640 1 --2- 172.21.15.12:0/3497065889 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c79e1aed40 0x55c79e1a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.363+0000 7fc6dbe00640 1 -- 172.21.15.12:0/3497065889 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c79e1aed40 msgr2=0x55c79e1a3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.363+0000 7fc6dbe00640 1 --2- 172.21.15.12:0/3497065889 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c79e1aed40 0x55c79e1a3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.363+0000 7fc6dbe00640 1 -- 172.21.15.12:0/3497065889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c79e1a91e0 msgr2=0x55c79e1a30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.363+0000 7fc6dbe00640 1 --2- 172.21.15.12:0/3497065889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c79e1a91e0 0x55c79e1a30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.363+0000 7fc6dbe00640 1 -- 172.21.15.12:0/3497065889 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c79e19f3a0 con 0x55c79e1a9bd0 2024-03-23T23:10:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.363+0000 7fc6e8a00640 1 --2- 172.21.15.12:0/3497065889 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c79e1aed40 0x55c79e1a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.363+0000 7fc6dbe00640 1 --2- 172.21.15.12:0/3497065889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c79e1a9bd0 0x55c79e1ba850 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc001eb0 tx=0x7fc6cc005df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67edf87450eee061 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.364+0000 7fc6d3e00640 1 -- 172.21.15.12:0/3497065889 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc02b030 con 0x55c79e1a9bd0 2024-03-23T23:10:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.364+0000 7fc6d3e00640 1 -- 172.21.15.12:0/3497065889 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc6cc022030 con 0x55c79e1a9bd0 2024-03-23T23:10:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.364+0000 7fc6d3e00640 1 -- 172.21.15.12:0/3497065889 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc00fba0 con 0x55c79e1a9bd0 2024-03-23T23:10:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.364+0000 7fc6ed837740 1 -- 172.21.15.12:0/3497065889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c79e1a9bd0 msgr2=0x55c79e1ba850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.364+0000 7fc6ed837740 1 --2- 172.21.15.12:0/3497065889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c79e1a9bd0 0x55c79e1ba850 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc001eb0 tx=0x7fc6cc005df0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.364+0000 7fc6ed837740 1 -- 172.21.15.12:0/3497065889 shutdown_connections 2024-03-23T23:10:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.364+0000 7fc6ed837740 1 --2- 172.21.15.12:0/3497065889 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c79e1aed40 0x55c79e1a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.364+0000 7fc6ed837740 1 --2- 172.21.15.12:0/3497065889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c79e1a9bd0 0x55c79e1ba850 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.364+0000 7fc6ed837740 1 --2- 172.21.15.12:0/3497065889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c79e1a91e0 0x55c79e1a30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.364+0000 7fc6ed837740 1 -- 172.21.15.12:0/3497065889 >> 172.21.15.12:0/3497065889 conn(0x55c79e1a4750 msgr2=0x55c79e1aaeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.364+0000 7fc6ed837740 1 -- 172.21.15.12:0/3497065889 shutdown_connections 2024-03-23T23:10:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.364+0000 7fc6ed837740 1 -- 172.21.15.12:0/3497065889 wait complete. 2024-03-23T23:10:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.365+0000 7fc6ed837740 1 -- start start 2024-03-23T23:10:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.365+0000 7fc6ed837740 1 Processor -- start 2024-03-23T23:10:46.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.365+0000 7fc6ed837740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c79e1a91e0 0x55c79e1b83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.365+0000 7fc6ed837740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c79e1a9bd0 0x55c79e1b88f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.366+0000 7fc6ed837740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c79e1aed40 0x55c79e1b9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.366+0000 7fc6ed837740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e41a92d0 con 0x55c79e1a9bd0 2024-03-23T23:10:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.366+0000 7fc6ed837740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e41aa5d0 con 0x55c79e1a91e0 2024-03-23T23:10:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.366+0000 7fc6ed837740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e41a9910 con 0x55c79e1aed40 2024-03-23T23:10:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.366+0000 7fc6ed837740 10 client.0 _ll_register_callbacks cb 0x7fc6cc009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.366+0000 7fc6dbe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c79e1a9bd0 0x55c79e1b88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.366+0000 7fc6e8a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c79e1a91e0 0x55c79e1b83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.366+0000 7fc6e8a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c79e1a91e0 0x55c79e1b83b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37582/0 (socket says 172.21.15.12:37582) 2024-03-23T23:10:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.366+0000 7fc6e8a00640 1 -- 172.21.15.12:0/1086343376 learned_addr learned my addr 172.21.15.12:0/1086343376 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.366+0000 7fc6e8a00640 1 --2- 172.21.15.12:0/1086343376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c79e1aed40 0x55c79e1b9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.366+0000 7fc6e8a00640 1 -- 172.21.15.12:0/1086343376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c79e1aed40 msgr2=0x55c79e1b9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.366+0000 7fc6e8a00640 1 --2- 172.21.15.12:0/1086343376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c79e1aed40 0x55c79e1b9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.366+0000 7fc6e8a00640 1 -- 172.21.15.12:0/1086343376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c79e1a9bd0 msgr2=0x55c79e1b88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.366+0000 7fc6e8a00640 1 --2- 172.21.15.12:0/1086343376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c79e1a9bd0 0x55c79e1b88f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.366+0000 7fc6e8a00640 1 -- 172.21.15.12:0/1086343376 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c79e1b9ef0 con 0x55c79e1a91e0 2024-03-23T23:10:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.366+0000 7fc6dbe00640 1 --2- 172.21.15.12:0/1086343376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c79e1a9bd0 0x55c79e1b88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.367+0000 7fc6e8a00640 1 --2- 172.21.15.12:0/1086343376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c79e1a91e0 0x55c79e1b83b0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc033f70 tx=0x7fc6dc009300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14699afa70d55280 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.367+0000 7fc6d1600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:10:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.367+0000 7fc6d1600640 1 -- 172.21.15.12:0/1086343376 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc052040 con 0x55c79e1a91e0 2024-03-23T23:10:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.367+0000 7fc6ed837740 1 -- 172.21.15.12:0/1086343376 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc6cc022030 con 0x55c79e1a91e0 2024-03-23T23:10:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.367+0000 7fc6d1600640 1 -- 172.21.15.12:0/1086343376 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fc6dc033860 con 0x55c79e1a91e0 2024-03-23T23:10:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.367+0000 7fc6d1600640 1 -- 172.21.15.12:0/1086343376 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6dc043040 con 0x55c79e1a91e0 2024-03-23T23:10:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.367+0000 7fc6ed837740 10 client.27806 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.367+0000 7fc6ed837740 1 -- 172.21.15.12:0/1086343376 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c79e21aea0 con 0x55c79e1a91e0 2024-03-23T23:10:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.367+0000 7fc6d1600640 1 -- 172.21.15.12:0/1086343376 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc6cc022030 con 0x55c79e1a91e0 2024-03-23T23:10:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.368+0000 7fc6d1600640 1 -- 172.21.15.12:0/1086343376 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fc6dc060080 con 0x55c79e1a91e0 2024-03-23T23:10:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.368+0000 7fc6ed837740 10 client.27806 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.368+0000 7fc6ed837740 10 client.27806 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.368+0000 7fc6ed837740 20 client.27806 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.368+0000 7fc6ed837740 1 --2- 172.21.15.12:0/1086343376 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c79e21b230 0x55c79e21d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.368+0000 7fc6ed837740 4 client.27806 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:10:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.368+0000 7fc6ed837740 1 -- 172.21.15.12:0/1086343376 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c79e1160c0 con 0x55c79e21b230 2024-03-23T23:10:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.368+0000 7fc6dbe00640 1 --2- 172.21.15.12:0/1086343376 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c79e21b230 0x55c79e21d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.368+0000 7fc6dbe00640 1 --2- 172.21.15.12:0/1086343376 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c79e21b230 0x55c79e21d620 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b707a4b3d18080f7 server_cookie=d1a5d3f1f567817d in_seq=0 out_seq=0 2024-03-23T23:10:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.368+0000 7fc6d1600640 10 client.27806 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.380+0000 7fc6d1600640 1 -- 172.21.15.12:0/1086343376 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c79e1160c0 con 0x55c79e21b230 2024-03-23T23:10:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.380+0000 7fc6d1600640 10 client.27806 handle_command_reply: tid=0 2024-03-23T23:10:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.380+0000 7fc6ed837740 1 client.27806 shutdown 2024-03-23T23:10:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.381+0000 7fc6ed837740 1 -- 172.21.15.12:0/1086343376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c79e1a91e0 msgr2=0x55c79e1b83b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.381+0000 7fc6ed837740 1 --2- 172.21.15.12:0/1086343376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c79e1a91e0 0x55c79e1b83b0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc033f70 tx=0x7fc6dc009300 comp rx=0 tx=0).stop 2024-03-23T23:10:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.381+0000 7fc6ed837740 1 -- 172.21.15.12:0/1086343376 shutdown_connections 2024-03-23T23:10:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.381+0000 7fc6ed837740 1 --2- 172.21.15.12:0/1086343376 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c79e21b230 0x55c79e21d620 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.381+0000 7fc6ed837740 1 --2- 172.21.15.12:0/1086343376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c79e1aed40 0x55c79e1b9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.381+0000 7fc6ed837740 1 --2- 172.21.15.12:0/1086343376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c79e1a9bd0 0x55c79e1b88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.381+0000 7fc6ed837740 1 --2- 172.21.15.12:0/1086343376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c79e1a91e0 0x55c79e1b83b0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.381+0000 7fc6ed837740 1 -- 172.21.15.12:0/1086343376 >> 172.21.15.12:0/1086343376 conn(0x55c79e1a4750 msgr2=0x55c79e1a79b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.381+0000 7fc6d1600640 0 client.27806 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.381+0000 7fc6ed837740 1 -- 172.21.15.12:0/1086343376 shutdown_connections 2024-03-23T23:10:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.381+0000 7fc6ed837740 1 -- 172.21.15.12:0/1086343376 wait complete. 2024-03-23T23:10:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.381+0000 7fc6ed837740 20 client.27806 trim_cache size 0 max 16384 2024-03-23T23:10:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:10:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:10:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:10:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:10:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.382+0000 7fc6eaa00640 1 -- 172.21.15.12:0/3960085678 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc6c80a2920 msgr2=0x7fc6c80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.382+0000 7fc6eaa00640 1 --2- 172.21.15.12:0/3960085678 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc6c80a2920 0x7fc6c80a4de0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fc6d40a3a00 tx=0x7fc6d4003e10 comp rx=0 tx=0).stop 2024-03-23T23:10:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.382+0000 7fc6eaa00640 1 -- 172.21.15.12:0/3960085678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6e4103900 msgr2=0x7fc6e4199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.382+0000 7fc6eaa00640 1 --2- 172.21.15.12:0/3960085678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6e4103900 0x7fc6e4199b10 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc6d400e9c0 tx=0x7fc6d400ee80 comp rx=0 tx=0).stop 2024-03-23T23:10:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.382+0000 7fc6eaa00640 1 -- 172.21.15.12:0/3960085678 shutdown_connections 2024-03-23T23:10:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.382+0000 7fc6eaa00640 1 --2- 172.21.15.12:0/3960085678 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc6c80a2920 0x7fc6c80a4de0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.382+0000 7fc6eaa00640 1 --2- 172.21.15.12:0/3960085678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6e41054b0 0x7fc6e41a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.382+0000 7fc6eaa00640 1 --2- 172.21.15.12:0/3960085678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6e4104af0 0x7fc6e419a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.382+0000 7fc6eaa00640 1 --2- 172.21.15.12:0/3960085678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6e4103900 0x7fc6e4199b10 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.382+0000 7fc6eaa00640 1 -- 172.21.15.12:0/3960085678 >> 172.21.15.12:0/3960085678 conn(0x7fc6e40fee90 msgr2=0x7fc6e4107d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.382+0000 7fc6eaa00640 1 -- 172.21.15.12:0/3960085678 shutdown_connections 2024-03-23T23:10:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:46.382+0000 7fc6eaa00640 1 -- 172.21.15.12:0/3960085678 wait complete. 2024-03-23T23:10:46.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:46 smithi203 ceph-mon[36693]: pgmap v659: 97 pgs: 97 active+clean; 2.0 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:10:47.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:46 smithi012 ceph-mon[31695]: pgmap v659: 97 pgs: 97 active+clean; 2.0 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:10:47.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:46 smithi173 ceph-mon[36116]: pgmap v659: 97 pgs: 97 active+clean; 2.0 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:10:48.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:48 smithi203 ceph-mon[36693]: pgmap v660: 97 pgs: 97 active+clean; 2.0 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:10:49.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:48 smithi012 ceph-mon[31695]: pgmap v660: 97 pgs: 97 active+clean; 2.0 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:10:49.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:48 smithi173 ceph-mon[36116]: pgmap v660: 97 pgs: 97 active+clean; 2.0 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:10:49.692 INFO:tasks.workunit.client.0.smithi012.stdout:375/600: open fd = 3 2024-03-23T23:10:49.692 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.692 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.692 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.692 INFO:tasks.workunit.client.0.smithi012.stdout:376/600: open fd = 3 2024-03-23T23:10:49.692 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.692 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.692 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.692 INFO:tasks.workunit.client.0.smithi012.stdout:377/600: open fd = 3 2024-03-23T23:10:49.692 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.692 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.693 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.693 INFO:tasks.workunit.client.0.smithi012.stdout:378/600: open fd = 3 2024-03-23T23:10:49.693 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.693 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.693 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.693 INFO:tasks.workunit.client.0.smithi012.stdout:379/600: open fd = 3 2024-03-23T23:10:49.693 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.693 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.693 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.693 INFO:tasks.workunit.client.0.smithi012.stdout:380/600: open fd = 3 2024-03-23T23:10:49.693 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.693 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.693 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.693 INFO:tasks.workunit.client.0.smithi012.stdout:381/600: open fd = 3 2024-03-23T23:10:49.693 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.693 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.694 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.694 INFO:tasks.workunit.client.0.smithi012.stdout:382/600: open fd = 3 2024-03-23T23:10:49.694 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.694 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.694 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.694 INFO:tasks.workunit.client.0.smithi012.stdout:383/600: open fd = 3 2024-03-23T23:10:49.694 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.694 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.694 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.694 INFO:tasks.workunit.client.0.smithi012.stdout:384/600: open fd = 3 2024-03-23T23:10:49.694 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.694 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.694 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.694 INFO:tasks.workunit.client.0.smithi012.stdout:385/600: open fd = 3 2024-03-23T23:10:49.695 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.695 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.695 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.695 INFO:tasks.workunit.client.0.smithi012.stdout:386/600: open fd = 3 2024-03-23T23:10:49.695 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.695 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.695 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.695 INFO:tasks.workunit.client.0.smithi012.stdout:387/600: open fd = 3 2024-03-23T23:10:49.695 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.695 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.695 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.695 INFO:tasks.workunit.client.0.smithi012.stdout:388/600: open fd = 3 2024-03-23T23:10:49.695 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.695 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.695 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.695 INFO:tasks.workunit.client.0.smithi012.stdout:389/600: open fd = 3 2024-03-23T23:10:49.696 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.696 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.696 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.696 INFO:tasks.workunit.client.0.smithi012.stdout:390/600: open fd = 3 2024-03-23T23:10:49.696 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.696 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.696 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.696 INFO:tasks.workunit.client.0.smithi012.stdout:391/600: open fd = 3 2024-03-23T23:10:49.696 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.696 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.696 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.696 INFO:tasks.workunit.client.0.smithi012.stdout:392/600: open fd = 3 2024-03-23T23:10:49.696 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.696 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.696 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.696 INFO:tasks.workunit.client.0.smithi012.stdout:393/600: open fd = 3 2024-03-23T23:10:49.697 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.697 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.697 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.697 INFO:tasks.workunit.client.0.smithi012.stdout:394/600: open fd = 3 2024-03-23T23:10:49.697 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.697 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.697 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.697 INFO:tasks.workunit.client.0.smithi012.stdout:395/600: open fd = 3 2024-03-23T23:10:49.697 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.697 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.697 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.697 INFO:tasks.workunit.client.0.smithi012.stdout:396/600: open fd = 3 2024-03-23T23:10:49.697 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.697 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.697 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.697 INFO:tasks.workunit.client.0.smithi012.stdout:397/600: open fd = 3 2024-03-23T23:10:49.697 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.698 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.698 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.698 INFO:tasks.workunit.client.0.smithi012.stdout:398/600: open fd = 3 2024-03-23T23:10:49.698 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.698 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.698 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.698 INFO:tasks.workunit.client.0.smithi012.stdout:399/600: open fd = 3 2024-03-23T23:10:49.698 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.698 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.698 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.698 INFO:tasks.workunit.client.0.smithi012.stdout:400/600: open fd = 3 2024-03-23T23:10:49.698 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.698 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.698 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.699 INFO:tasks.workunit.client.0.smithi012.stdout:401/600: open fd = 3 2024-03-23T23:10:49.699 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.699 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.699 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.699 INFO:tasks.workunit.client.0.smithi012.stdout:402/600: open fd = 3 2024-03-23T23:10:49.699 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.699 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.699 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.699 INFO:tasks.workunit.client.0.smithi012.stdout:403/600: open fd = 3 2024-03-23T23:10:49.699 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.699 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.699 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.699 INFO:tasks.workunit.client.0.smithi012.stdout:404/600: open fd = 3 2024-03-23T23:10:49.699 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.699 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.699 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.700 INFO:tasks.workunit.client.0.smithi012.stdout:405/600: open fd = 3 2024-03-23T23:10:49.700 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.700 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.700 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.700 INFO:tasks.workunit.client.0.smithi012.stdout:406/600: open fd = 3 2024-03-23T23:10:49.700 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.700 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.700 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.700 INFO:tasks.workunit.client.0.smithi012.stdout:407/600: open fd = 3 2024-03-23T23:10:49.700 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.700 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.700 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.700 INFO:tasks.workunit.client.0.smithi012.stdout:408/600: open fd = 3 2024-03-23T23:10:49.700 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.700 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.700 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.701 INFO:tasks.workunit.client.0.smithi012.stdout:409/600: open fd = 3 2024-03-23T23:10:49.701 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.701 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.701 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.701 INFO:tasks.workunit.client.0.smithi012.stdout:410/600: open fd = 3 2024-03-23T23:10:49.701 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.701 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.701 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.701 INFO:tasks.workunit.client.0.smithi012.stdout:411/600: open fd = 3 2024-03-23T23:10:49.701 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.701 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.701 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.701 INFO:tasks.workunit.client.0.smithi012.stdout:412/600: open fd = 3 2024-03-23T23:10:49.701 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.701 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.701 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.701 INFO:tasks.workunit.client.0.smithi012.stdout:413/600: open fd = 3 2024-03-23T23:10:49.702 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.702 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.702 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.702 INFO:tasks.workunit.client.0.smithi012.stdout:414/600: open fd = 3 2024-03-23T23:10:49.702 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.702 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.702 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.702 INFO:tasks.workunit.client.0.smithi012.stdout:415/600: open fd = 3 2024-03-23T23:10:49.702 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.702 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.702 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.702 INFO:tasks.workunit.client.0.smithi012.stdout:416/600: open fd = 3 2024-03-23T23:10:49.702 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.702 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.703 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.703 INFO:tasks.workunit.client.0.smithi012.stdout:417/600: open fd = 3 2024-03-23T23:10:49.703 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.703 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.703 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.703 INFO:tasks.workunit.client.0.smithi012.stdout:418/600: open fd = 3 2024-03-23T23:10:49.703 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.703 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.703 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.703 INFO:tasks.workunit.client.0.smithi012.stdout:419/600: open fd = 3 2024-03-23T23:10:49.703 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.703 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.703 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.703 INFO:tasks.workunit.client.0.smithi012.stdout:420/600: open fd = 3 2024-03-23T23:10:49.703 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.703 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.704 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.704 INFO:tasks.workunit.client.0.smithi012.stdout:421/600: open fd = 3 2024-03-23T23:10:49.704 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.704 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.704 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.704 INFO:tasks.workunit.client.0.smithi012.stdout:422/600: open fd = 3 2024-03-23T23:10:49.704 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.704 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.704 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.704 INFO:tasks.workunit.client.0.smithi012.stdout:423/600: open fd = 3 2024-03-23T23:10:49.704 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.704 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.704 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.704 INFO:tasks.workunit.client.0.smithi012.stdout:424/600: open fd = 3 2024-03-23T23:10:49.704 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.704 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.704 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.705 INFO:tasks.workunit.client.0.smithi012.stdout:425/600: open fd = 3 2024-03-23T23:10:49.705 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.705 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.705 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.705 INFO:tasks.workunit.client.0.smithi012.stdout:426/600: open fd = 3 2024-03-23T23:10:49.705 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.705 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.705 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.705 INFO:tasks.workunit.client.0.smithi012.stdout:427/600: open fd = 3 2024-03-23T23:10:49.705 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.705 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.705 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.705 INFO:tasks.workunit.client.0.smithi012.stdout:428/600: open fd = 3 2024-03-23T23:10:49.705 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.705 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.705 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.705 INFO:tasks.workunit.client.0.smithi012.stdout:429/600: open fd = 3 2024-03-23T23:10:49.706 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.706 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.706 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.706 INFO:tasks.workunit.client.0.smithi012.stdout:430/600: open fd = 3 2024-03-23T23:10:49.706 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.706 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.706 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.706 INFO:tasks.workunit.client.0.smithi012.stdout:431/600: open fd = 3 2024-03-23T23:10:49.706 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.706 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.706 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.706 INFO:tasks.workunit.client.0.smithi012.stdout:432/600: open fd = 3 2024-03-23T23:10:49.706 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.706 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.707 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.707 INFO:tasks.workunit.client.0.smithi012.stdout:433/600: open fd = 3 2024-03-23T23:10:49.707 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.707 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.707 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.707 INFO:tasks.workunit.client.0.smithi012.stdout:434/600: open fd = 3 2024-03-23T23:10:49.707 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.707 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.707 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.707 INFO:tasks.workunit.client.0.smithi012.stdout:435/600: open fd = 3 2024-03-23T23:10:49.707 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.707 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.707 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:49.707 INFO:tasks.workunit.client.0.smithi012.stdout:436/600: open fd = 3 2024-03-23T23:10:49.707 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.707 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:10:49.708 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:10:50.090 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:10:50.090 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:10:50.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.322+0000 7f3de1400640 1 -- 172.21.15.12:0/3621598224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ddc1037c0 msgr2=0x7f3ddc103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:50.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.322+0000 7f3de1400640 1 --2- 172.21.15.12:0/3621598224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ddc1037c0 0x7f3ddc103be0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc009900 tx=0x7f3dcc02bc50 comp rx=0 tx=0).stop 2024-03-23T23:10:50.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.322+0000 7f3de1400640 1 -- 172.21.15.12:0/3621598224 shutdown_connections 2024-03-23T23:10:50.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.322+0000 7f3de1400640 1 --2- 172.21.15.12:0/3621598224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ddc105370 0x7f3ddc10bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.322+0000 7f3de1400640 1 --2- 172.21.15.12:0/3621598224 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ddc1049b0 0x7f3ddc104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.322+0000 7f3de1400640 1 --2- 172.21.15.12:0/3621598224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ddc1037c0 0x7f3ddc103be0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.322+0000 7f3de1400640 1 -- 172.21.15.12:0/3621598224 >> 172.21.15.12:0/3621598224 conn(0x7f3ddc0fed50 msgr2=0x7f3ddc101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:50.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.322+0000 7f3de1400640 1 -- 172.21.15.12:0/3621598224 shutdown_connections 2024-03-23T23:10:50.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.322+0000 7f3de1400640 1 -- 172.21.15.12:0/3621598224 wait complete. 2024-03-23T23:10:50.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.323+0000 7f3de1400640 1 Processor -- start 2024-03-23T23:10:50.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.323+0000 7f3de1400640 1 -- start start 2024-03-23T23:10:50.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.323+0000 7f3de1400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ddc1037c0 0x7f3ddc199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:50.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.324+0000 7f3de1400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ddc1049b0 0x7f3ddc19a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:50.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.324+0000 7f3de1400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ddc105370 0x7f3ddc1a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:50.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.324+0000 7f3de1400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc10f460 con 0x7f3ddc1049b0 2024-03-23T23:10:50.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.324+0000 7f3de1400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc10f2e0 con 0x7f3ddc1037c0 2024-03-23T23:10:50.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.324+0000 7f3de1400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc10f5e0 con 0x7f3ddc105370 2024-03-23T23:10:50.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.324+0000 7f3ddbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ddc1037c0 0x7f3ddc199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:50.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.324+0000 7f3de0a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ddc105370 0x7f3ddc1a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:50.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.324+0000 7f3ddbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ddc1037c0 0x7f3ddc199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37598/0 (socket says 172.21.15.12:37598) 2024-03-23T23:10:50.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.324+0000 7f3ddbe00640 1 -- 172.21.15.12:0/3901537486 learned_addr learned my addr 172.21.15.12:0/3901537486 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:50.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.324+0000 7f3dd3e00640 1 --2- 172.21.15.12:0/3901537486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ddc1049b0 0x7f3ddc19a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:50.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.324+0000 7f3ddbe00640 1 -- 172.21.15.12:0/3901537486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ddc105370 msgr2=0x7f3ddc1a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:50.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.324+0000 7f3ddbe00640 1 --2- 172.21.15.12:0/3901537486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ddc105370 0x7f3ddc1a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.324+0000 7f3ddbe00640 1 -- 172.21.15.12:0/3901537486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ddc1049b0 msgr2=0x7f3ddc19a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:50.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.324+0000 7f3ddbe00640 1 --2- 172.21.15.12:0/3901537486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ddc1049b0 0x7f3ddc19a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.324+0000 7f3ddbe00640 1 -- 172.21.15.12:0/3901537486 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ddc1a1720 con 0x7f3ddc1037c0 2024-03-23T23:10:50.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.324+0000 7f3dd3e00640 1 --2- 172.21.15.12:0/3901537486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ddc1049b0 0x7f3ddc19a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:10:50.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.325+0000 7f3ddbe00640 1 --2- 172.21.15.12:0/3901537486 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ddc1037c0 0x7f3ddc199b00 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc02c260 tx=0x7f3dcc02c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:50.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.325+0000 7f3dd9600640 1 -- 172.21.15.12:0/3901537486 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc041070 con 0x7f3ddc1037c0 2024-03-23T23:10:50.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.325+0000 7f3de1400640 1 -- 172.21.15.12:0/3901537486 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ddc1a19b0 con 0x7f3ddc1037c0 2024-03-23T23:10:50.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.325+0000 7f3dd9600640 1 -- 172.21.15.12:0/3901537486 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3dcc004450 con 0x7f3ddc1037c0 2024-03-23T23:10:50.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.325+0000 7f3dd9600640 1 -- 172.21.15.12:0/3901537486 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc034440 con 0x7f3ddc1037c0 2024-03-23T23:10:50.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.325+0000 7f3de1400640 1 -- 172.21.15.12:0/3901537486 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ddc1a1f00 con 0x7f3ddc1037c0 2024-03-23T23:10:50.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.330+0000 7f3de57f5740 1 Processor -- start 2024-03-23T23:10:50.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.330+0000 7f3dd9600640 1 -- 172.21.15.12:0/3901537486 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3dcc044070 con 0x7f3ddc1037c0 2024-03-23T23:10:50.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.330+0000 7f3de57f5740 1 -- start start 2024-03-23T23:10:50.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.330+0000 7f3de57f5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561cfaf94050 0x561cfaf94470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:50.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.330+0000 7f3de57f5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94a40 0x561cfaf99540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:50.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.330+0000 7f3de57f5740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561cfaf99bb0 0x561cfaf9c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:50.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.330+0000 7f3de57f5740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cfaef9250 con 0x561cfaf99bb0 2024-03-23T23:10:50.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.330+0000 7f3de57f5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cfaea1a30 con 0x561cfaf94a40 2024-03-23T23:10:50.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.330+0000 7f3de57f5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cfaef52f0 con 0x561cfaf94050 2024-03-23T23:10:50.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.331+0000 7f3ddbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94a40 0x561cfaf99540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:50.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.331+0000 7f3de0a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561cfaf94050 0x561cfaf94470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:50.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.331+0000 7f3ddbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94a40 0x561cfaf99540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37606/0 (socket says 172.21.15.12:37606) 2024-03-23T23:10:50.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.331+0000 7f3de0a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561cfaf94050 0x561cfaf94470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54086/0 (socket says 172.21.15.12:54086) 2024-03-23T23:10:50.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.331+0000 7f3ddbe00640 1 -- 172.21.15.12:0/3239498242 learned_addr learned my addr 172.21.15.12:0/3239498242 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:50.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.331+0000 7f3dd3e00640 1 --2- 172.21.15.12:0/3239498242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561cfaf99bb0 0x561cfaf9c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:50.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.331+0000 7f3ddbe00640 1 -- 172.21.15.12:0/3239498242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561cfaf94050 msgr2=0x561cfaf94470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:50.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.331+0000 7f3ddbe00640 1 --2- 172.21.15.12:0/3239498242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561cfaf94050 0x561cfaf94470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.331+0000 7f3ddbe00640 1 -- 172.21.15.12:0/3239498242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561cfaf99bb0 msgr2=0x561cfaf9c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:50.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.331+0000 7f3ddbe00640 1 --2- 172.21.15.12:0/3239498242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561cfaf99bb0 0x561cfaf9c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.331+0000 7f3ddbe00640 1 -- 172.21.15.12:0/3239498242 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561cfae8c690 con 0x561cfaf94a40 2024-03-23T23:10:50.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.331+0000 7f3de0a00640 1 --2- 172.21.15.12:0/3239498242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561cfaf94050 0x561cfaf94470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:50.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.331+0000 7f3ddbe00640 1 --2- 172.21.15.12:0/3239498242 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94a40 0x561cfaf99540 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc0f6560 tx=0x7f3dcc0f6650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf903da72f415a9b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:50.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.332+0000 7f3dd9600640 1 --2- 172.21.15.12:0/3901537486 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3dc00a2800 0x7f3dc00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:50.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.332+0000 7f3dd1200640 1 -- 172.21.15.12:0/3239498242 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc0b91d0 con 0x561cfaf94a40 2024-03-23T23:10:50.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.332+0000 7f3dd1200640 1 -- 172.21.15.12:0/3239498242 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3dcc0fd030 con 0x561cfaf94a40 2024-03-23T23:10:50.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.332+0000 7f3dd1200640 1 -- 172.21.15.12:0/3239498242 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc0bc380 con 0x561cfaf94a40 2024-03-23T23:10:50.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.332+0000 7f3dd9600640 1 -- 172.21.15.12:0/3901537486 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f3dcc0f0580 con 0x7f3ddc1037c0 2024-03-23T23:10:50.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.332+0000 7f3de0a00640 1 --2- 172.21.15.12:0/3901537486 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3dc00a2800 0x7f3dc00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:50.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.333+0000 7f3de57f5740 1 -- 172.21.15.12:0/3239498242 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94a40 msgr2=0x561cfaf99540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:50.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.333+0000 7f3de57f5740 1 --2- 172.21.15.12:0/3239498242 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94a40 0x561cfaf99540 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc0f6560 tx=0x7f3dcc0f6650 comp rx=0 tx=0).stop 2024-03-23T23:10:50.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.333+0000 7f3de0a00640 1 --2- 172.21.15.12:0/3901537486 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3dc00a2800 0x7f3dc00a4cc0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x561cfaf952a0 tx=0x7f3dd4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:50.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.333+0000 7f3de57f5740 1 -- 172.21.15.12:0/3239498242 shutdown_connections 2024-03-23T23:10:50.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.333+0000 7f3de57f5740 1 --2- 172.21.15.12:0/3239498242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561cfaf99bb0 0x561cfaf9c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.333+0000 7f3de57f5740 1 --2- 172.21.15.12:0/3239498242 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94a40 0x561cfaf99540 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.333+0000 7f3de57f5740 1 --2- 172.21.15.12:0/3239498242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561cfaf94050 0x561cfaf94470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.333+0000 7f3de57f5740 1 -- 172.21.15.12:0/3239498242 >> 172.21.15.12:0/3239498242 conn(0x561cfaf5c200 msgr2=0x561cfaf8f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:50.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.333+0000 7f3de57f5740 1 -- 172.21.15.12:0/3239498242 shutdown_connections 2024-03-23T23:10:50.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.333+0000 7f3de57f5740 1 -- 172.21.15.12:0/3239498242 wait complete. 2024-03-23T23:10:50.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.333+0000 7f3de57f5740 1 -- start start 2024-03-23T23:10:50.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.334+0000 7f3de57f5740 1 Processor -- start 2024-03-23T23:10:50.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.334+0000 7f3de57f5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94050 0x561cfaf9dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:50.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.334+0000 7f3de57f5740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561cfaf94a40 0x561cfaf9e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:50.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.334+0000 7f3de57f5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561cfaf99bb0 0x561cfafa5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:50.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.334+0000 7f3de57f5740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cfaf9c920 con 0x561cfaf94a40 2024-03-23T23:10:50.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.334+0000 7f3de57f5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cfaeb4e80 con 0x561cfaf94050 2024-03-23T23:10:50.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.334+0000 7f3de57f5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cfaf9caa0 con 0x561cfaf99bb0 2024-03-23T23:10:50.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.334+0000 7f3de57f5740 10 client.0 _ll_register_callbacks cb 0x561cfaec7130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:50.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.334+0000 7f3ddbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94050 0x561cfaf9dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:50.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.335+0000 7f3dd3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561cfaf94a40 0x561cfaf9e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:50.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.335+0000 7f3de0a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561cfaf99bb0 0x561cfafa5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:50.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.335+0000 7f3ddbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94050 0x561cfaf9dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37620/0 (socket says 172.21.15.12:37620) 2024-03-23T23:10:50.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.335+0000 7f3ddbe00640 1 -- 172.21.15.12:0/1732951808 learned_addr learned my addr 172.21.15.12:0/1732951808 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:50.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.335+0000 7f3de0a00640 1 -- 172.21.15.12:0/1732951808 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94050 msgr2=0x561cfaf9dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:50.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.335+0000 7f3de0a00640 1 --2- 172.21.15.12:0/1732951808 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94050 0x561cfaf9dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.335+0000 7f3de0a00640 1 -- 172.21.15.12:0/1732951808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561cfaf94a40 msgr2=0x561cfaf9e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:50.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.335+0000 7f3de0a00640 1 --2- 172.21.15.12:0/1732951808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561cfaf94a40 0x561cfaf9e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.335+0000 7f3de0a00640 1 -- 172.21.15.12:0/1732951808 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561cfacd77d0 con 0x561cfaf99bb0 2024-03-23T23:10:50.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.335+0000 7f3ddbe00640 1 --2- 172.21.15.12:0/1732951808 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94050 0x561cfaf9dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:50.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.335+0000 7f3de0a00640 1 --2- 172.21.15.12:0/1732951808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561cfaf99bb0 0x561cfafa5240 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4003b80 tx=0x7f3dd4004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e507d58f36b34a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:50.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.335+0000 7f3dc9600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:10:50.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.336+0000 7f3dc9600640 1 -- 172.21.15.12:0/1732951808 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd4004250 con 0x561cfaf99bb0 2024-03-23T23:10:50.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.336+0000 7f3dc9600640 1 -- 172.21.15.12:0/1732951808 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f3dd40043f0 con 0x561cfaf99bb0 2024-03-23T23:10:50.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.336+0000 7f3de57f5740 1 -- 172.21.15.12:0/1732951808 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3dcc0fd030 con 0x561cfaf99bb0 2024-03-23T23:10:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.336+0000 7f3dc9600640 1 -- 172.21.15.12:0/1732951808 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3dd40045f0 con 0x561cfaf99bb0 2024-03-23T23:10:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.336+0000 7f3de57f5740 10 client.27778 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.336+0000 7f3de57f5740 1 -- 172.21.15.12:0/1732951808 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561cfac0d3f0 con 0x561cfaf99bb0 2024-03-23T23:10:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.336+0000 7f3dc9600640 1 -- 172.21.15.12:0/1732951808 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3dcc0fd030 con 0x561cfaf99bb0 2024-03-23T23:10:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.336+0000 7f3dc9600640 1 -- 172.21.15.12:0/1732951808 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f3dd4026070 con 0x561cfaf99bb0 2024-03-23T23:10:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.336+0000 7f3de57f5740 10 client.27778 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.336+0000 7f3de57f5740 10 client.27778 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.337+0000 7f3de57f5740 20 client.27778 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.337+0000 7f3de57f5740 1 --2- 172.21.15.12:0/1732951808 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561cfafa9ad0 0x561cfafabf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.337+0000 7f3de57f5740 4 client.27778 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:10:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.337+0000 7f3de57f5740 1 -- 172.21.15.12:0/1732951808 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561cfaf06790 con 0x561cfafa9ad0 2024-03-23T23:10:50.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.337+0000 7f3ddbe00640 1 --2- 172.21.15.12:0/1732951808 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561cfafa9ad0 0x561cfafabf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:50.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.337+0000 7f3dc9600640 10 client.27778 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:50.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.337+0000 7f3ddbe00640 1 --2- 172.21.15.12:0/1732951808 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561cfafa9ad0 0x561cfafabf70 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b2b08a4dfdf5967 server_cookie=e3f4d581ae0ab7e5 in_seq=0 out_seq=0 2024-03-23T23:10:50.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.338+0000 7f3dc9600640 1 -- 172.21.15.12:0/1732951808 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561cfaf06790 con 0x561cfafa9ad0 2024-03-23T23:10:50.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.338+0000 7f3dc9600640 10 client.27778 handle_command_reply: tid=0 2024-03-23T23:10:50.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.338+0000 7f3de57f5740 1 client.27778 shutdown 2024-03-23T23:10:50.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.338+0000 7f3de57f5740 1 -- 172.21.15.12:0/1732951808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561cfaf99bb0 msgr2=0x561cfafa5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:50.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.338+0000 7f3de57f5740 1 --2- 172.21.15.12:0/1732951808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561cfaf99bb0 0x561cfafa5240 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4003b80 tx=0x7f3dd4004ab0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.338+0000 7f3de57f5740 1 -- 172.21.15.12:0/1732951808 shutdown_connections 2024-03-23T23:10:50.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.338+0000 7f3de57f5740 1 --2- 172.21.15.12:0/1732951808 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561cfafa9ad0 0x561cfafabf70 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.339+0000 7f3de57f5740 1 --2- 172.21.15.12:0/1732951808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561cfaf99bb0 0x561cfafa5240 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.339+0000 7f3de57f5740 1 --2- 172.21.15.12:0/1732951808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561cfaf94a40 0x561cfaf9e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.339+0000 7f3de57f5740 1 --2- 172.21.15.12:0/1732951808 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94050 0x561cfaf9dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.339+0000 7f3de57f5740 1 -- 172.21.15.12:0/1732951808 >> 172.21.15.12:0/1732951808 conn(0x561cfaf5c200 msgr2=0x561cfaf8ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:50.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.339+0000 7f3dc9600640 0 client.27778 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:50.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.339+0000 7f3de57f5740 1 -- 172.21.15.12:0/1732951808 shutdown_connections 2024-03-23T23:10:50.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.339+0000 7f3de57f5740 1 -- 172.21.15.12:0/1732951808 wait complete. 2024-03-23T23:10:50.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.339+0000 7f3de57f5740 20 client.27778 trim_cache size 0 max 16384 2024-03-23T23:10:50.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.356+0000 7f3de57f5740 1 Processor -- start 2024-03-23T23:10:50.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.357+0000 7f3de57f5740 1 -- start start 2024-03-23T23:10:50.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.357+0000 7f3de57f5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94050 0x561cfaf8fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:50.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.357+0000 7f3de57f5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561cfaf94a40 0x561cfafa5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:50.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.357+0000 7f3de57f5740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561cfaf99bb0 0x561cfaf8e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:50.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.357+0000 7f3de57f5740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc1acf90 con 0x561cfaf99bb0 2024-03-23T23:10:50.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.357+0000 7f3de57f5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc1aaa90 con 0x561cfaf94050 2024-03-23T23:10:50.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.357+0000 7f3de57f5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc1a8b30 con 0x561cfaf94a40 2024-03-23T23:10:50.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.357+0000 7f3dd3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561cfaf94a40 0x561cfafa5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:50.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.357+0000 7f3dd3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561cfaf94a40 0x561cfafa5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54110/0 (socket says 172.21.15.12:54110) 2024-03-23T23:10:50.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.357+0000 7f3dd3e00640 1 -- 172.21.15.12:0/1452240422 learned_addr learned my addr 172.21.15.12:0/1452240422 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:50.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.357+0000 7f3ddbe00640 1 --2- 172.21.15.12:0/1452240422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94050 0x561cfaf8fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:50.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.357+0000 7f3de0a00640 1 --2- 172.21.15.12:0/1452240422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561cfaf99bb0 0x561cfaf8e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:50.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.357+0000 7f3dd3e00640 1 -- 172.21.15.12:0/1452240422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94050 msgr2=0x561cfaf8fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:50.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.357+0000 7f3dd3e00640 1 --2- 172.21.15.12:0/1452240422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94050 0x561cfaf8fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.357+0000 7f3dd3e00640 1 -- 172.21.15.12:0/1452240422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561cfaf99bb0 msgr2=0x561cfaf8e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:50.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.357+0000 7f3dd3e00640 1 --2- 172.21.15.12:0/1452240422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561cfaf99bb0 0x561cfaf8e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.357+0000 7f3dd3e00640 1 -- 172.21.15.12:0/1452240422 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561cfabcd0f0 con 0x561cfaf94a40 2024-03-23T23:10:50.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.358+0000 7f3de0a00640 1 --2- 172.21.15.12:0/1452240422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561cfaf99bb0 0x561cfaf8e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:10:50.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.358+0000 7f3dd3e00640 1 --2- 172.21.15.12:0/1452240422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561cfaf94a40 0x561cfafa5200 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4009cc0 tx=0x7f3dc4013830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db59305d0b8b035e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:50.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.358+0000 7f3dcbe00640 1 -- 172.21.15.12:0/1452240422 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4009db0 con 0x561cfaf94a40 2024-03-23T23:10:50.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.358+0000 7f3dcbe00640 1 -- 172.21.15.12:0/1452240422 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3dc4005020 con 0x561cfaf94a40 2024-03-23T23:10:50.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.358+0000 7f3dcbe00640 1 -- 172.21.15.12:0/1452240422 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4004bf0 con 0x561cfaf94a40 2024-03-23T23:10:50.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.359+0000 7f3de57f5740 1 -- 172.21.15.12:0/1452240422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561cfaf94a40 msgr2=0x561cfafa5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:50.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.359+0000 7f3de57f5740 1 --2- 172.21.15.12:0/1452240422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561cfaf94a40 0x561cfafa5200 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4009cc0 tx=0x7f3dc4013830 comp rx=0 tx=0).stop 2024-03-23T23:10:50.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.359+0000 7f3de57f5740 1 -- 172.21.15.12:0/1452240422 shutdown_connections 2024-03-23T23:10:50.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.359+0000 7f3de57f5740 1 --2- 172.21.15.12:0/1452240422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561cfaf99bb0 0x561cfaf8e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.359+0000 7f3de57f5740 1 --2- 172.21.15.12:0/1452240422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561cfaf94a40 0x561cfafa5200 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.359+0000 7f3de57f5740 1 --2- 172.21.15.12:0/1452240422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94050 0x561cfaf8fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.359+0000 7f3de57f5740 1 -- 172.21.15.12:0/1452240422 >> 172.21.15.12:0/1452240422 conn(0x561cfaf5c200 msgr2=0x561cfaf8f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:50.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.359+0000 7f3de57f5740 1 -- 172.21.15.12:0/1452240422 shutdown_connections 2024-03-23T23:10:50.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.359+0000 7f3de57f5740 1 -- 172.21.15.12:0/1452240422 wait complete. 2024-03-23T23:10:50.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.359+0000 7f3de57f5740 1 -- start start 2024-03-23T23:10:50.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.359+0000 7f3de57f5740 1 Processor -- start 2024-03-23T23:10:50.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.360+0000 7f3de57f5740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561cfaf94050 0x561cfafa3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:50.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.360+0000 7f3de57f5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94a40 0x561cfafa3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:50.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.360+0000 7f3de57f5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561cfaf99bb0 0x561cfafa4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:50.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.360+0000 7f3de57f5740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc1a9130 con 0x561cfaf94050 2024-03-23T23:10:50.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.360+0000 7f3de57f5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc1aa430 con 0x561cfaf94a40 2024-03-23T23:10:50.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.360+0000 7f3de57f5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc1a9770 con 0x561cfaf99bb0 2024-03-23T23:10:50.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.360+0000 7f3de57f5740 10 client.0 _ll_register_callbacks cb 0x7f3dd40114a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:50.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.360+0000 7f3dd3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94a40 0x561cfafa3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:50.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.360+0000 7f3ddbe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561cfaf94050 0x561cfafa3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:50.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.360+0000 7f3dd3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94a40 0x561cfafa3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37626/0 (socket says 172.21.15.12:37626) 2024-03-23T23:10:50.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.360+0000 7f3dd3e00640 1 -- 172.21.15.12:0/3641518642 learned_addr learned my addr 172.21.15.12:0/3641518642 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:50.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.360+0000 7f3de0a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561cfaf99bb0 0x561cfafa4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:50.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.360+0000 7f3dd3e00640 1 -- 172.21.15.12:0/3641518642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561cfaf99bb0 msgr2=0x561cfafa4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:50.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.360+0000 7f3dd3e00640 1 --2- 172.21.15.12:0/3641518642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561cfaf99bb0 0x561cfafa4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.360+0000 7f3dd3e00640 1 -- 172.21.15.12:0/3641518642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561cfaf94050 msgr2=0x561cfafa3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:50.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.360+0000 7f3dd3e00640 1 --2- 172.21.15.12:0/3641518642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561cfaf94050 0x561cfafa3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.361+0000 7f3dd3e00640 1 -- 172.21.15.12:0/3641518642 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561cfafa4b40 con 0x561cfaf94a40 2024-03-23T23:10:50.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.361+0000 7f3dd3e00640 1 --2- 172.21.15.12:0/3641518642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94a40 0x561cfafa3540 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f3dc401e040 tx=0x7f3dc40099c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b348239bbd3bd83 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:50.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.361+0000 7f3dc9600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:10:50.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.361+0000 7f3dc9600640 1 -- 172.21.15.12:0/3641518642 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4029020 con 0x561cfaf94a40 2024-03-23T23:10:50.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.361+0000 7f3dc9600640 1 -- 172.21.15.12:0/3641518642 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f3dc4004230 con 0x561cfaf94a40 2024-03-23T23:10:50.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.361+0000 7f3de57f5740 1 -- 172.21.15.12:0/3641518642 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3dc4005020 con 0x561cfaf94a40 2024-03-23T23:10:50.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.361+0000 7f3dc9600640 1 -- 172.21.15.12:0/3641518642 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3dc4020050 con 0x561cfaf94a40 2024-03-23T23:10:50.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.361+0000 7f3de57f5740 10 client.27824 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:50.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.361+0000 7f3de57f5740 1 -- 172.21.15.12:0/3641518642 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561cfb005a10 con 0x561cfaf94a40 2024-03-23T23:10:50.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.362+0000 7f3dc9600640 1 -- 172.21.15.12:0/3641518642 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3dc4005020 con 0x561cfaf94a40 2024-03-23T23:10:50.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.362+0000 7f3dc9600640 1 -- 172.21.15.12:0/3641518642 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f3dc4003b60 con 0x561cfaf94a40 2024-03-23T23:10:50.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.362+0000 7f3de57f5740 10 client.27824 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:50.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.362+0000 7f3de57f5740 10 client.27824 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:50.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.362+0000 7f3de57f5740 20 client.27824 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:50.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.362+0000 7f3de57f5740 1 --2- 172.21.15.12:0/3641518642 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561cfb006340 0x561cfb008760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.362+0000 7f3de57f5740 4 client.27824 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:10:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.362+0000 7f3de57f5740 1 -- 172.21.15.12:0/3641518642 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561cfaf00ae0 con 0x561cfb006340 2024-03-23T23:10:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.362+0000 7f3ddbe00640 1 --2- 172.21.15.12:0/3641518642 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561cfb006340 0x561cfb008760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.363+0000 7f3ddbe00640 1 --2- 172.21.15.12:0/3641518642 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561cfb006340 0x561cfb008760 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=292e0dbc78581b87 server_cookie=a9bf6f6d075db7d1 in_seq=0 out_seq=0 2024-03-23T23:10:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.363+0000 7f3dc9600640 10 client.27824 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.363+0000 7f3dc9600640 1 -- 172.21.15.12:0/3641518642 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x561cfaf00ae0 con 0x561cfb006340 2024-03-23T23:10:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.363+0000 7f3dc9600640 10 client.27824 handle_command_reply: tid=0 2024-03-23T23:10:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.363+0000 7f3de57f5740 1 client.27824 shutdown 2024-03-23T23:10:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.363+0000 7f3de57f5740 1 -- 172.21.15.12:0/3641518642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94a40 msgr2=0x561cfafa3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.363+0000 7f3de57f5740 1 --2- 172.21.15.12:0/3641518642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94a40 0x561cfafa3540 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f3dc401e040 tx=0x7f3dc40099c0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.363+0000 7f3de57f5740 1 -- 172.21.15.12:0/3641518642 shutdown_connections 2024-03-23T23:10:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.363+0000 7f3de57f5740 1 --2- 172.21.15.12:0/3641518642 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561cfb006340 0x561cfb008760 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.363+0000 7f3de57f5740 1 --2- 172.21.15.12:0/3641518642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561cfaf99bb0 0x561cfafa4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.363+0000 7f3de57f5740 1 --2- 172.21.15.12:0/3641518642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cfaf94a40 0x561cfafa3540 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.363+0000 7f3dc9600640 0 client.27824 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.363+0000 7f3de57f5740 1 --2- 172.21.15.12:0/3641518642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561cfaf94050 0x561cfafa3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.363+0000 7f3de57f5740 1 -- 172.21.15.12:0/3641518642 >> 172.21.15.12:0/3641518642 conn(0x561cfaf5c200 msgr2=0x561cfaf8e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:50.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.363+0000 7f3de57f5740 1 -- 172.21.15.12:0/3641518642 shutdown_connections 2024-03-23T23:10:50.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.363+0000 7f3de57f5740 1 -- 172.21.15.12:0/3641518642 wait complete. 2024-03-23T23:10:50.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.363+0000 7f3de57f5740 20 client.27824 trim_cache size 0 max 16384 2024-03-23T23:10:50.368 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:10:50.368 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-03-23T23:10:50.368 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-03-23T23:10:50.368 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:10:50.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.364+0000 7f3de1400640 1 -- 172.21.15.12:0/3901537486 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3dc00a2800 msgr2=0x7f3dc00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:50.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.364+0000 7f3de1400640 1 --2- 172.21.15.12:0/3901537486 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3dc00a2800 0x7f3dc00a4cc0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x561cfaf952a0 tx=0x7f3dd4009500 comp rx=0 tx=0).stop 2024-03-23T23:10:50.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.364+0000 7f3de1400640 1 -- 172.21.15.12:0/3901537486 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ddc1037c0 msgr2=0x7f3ddc199b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:50.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.364+0000 7f3de1400640 1 --2- 172.21.15.12:0/3901537486 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ddc1037c0 0x7f3ddc199b00 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc02c260 tx=0x7f3dcc02c670 comp rx=0 tx=0).stop 2024-03-23T23:10:50.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.364+0000 7f3de1400640 1 -- 172.21.15.12:0/3901537486 shutdown_connections 2024-03-23T23:10:50.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.364+0000 7f3de1400640 1 --2- 172.21.15.12:0/3901537486 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f3dc00a2800 0x7f3dc00a4cc0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.364+0000 7f3de1400640 1 --2- 172.21.15.12:0/3901537486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ddc105370 0x7f3ddc1a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.364+0000 7f3de1400640 1 --2- 172.21.15.12:0/3901537486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ddc1049b0 0x7f3ddc19a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.364+0000 7f3de1400640 1 --2- 172.21.15.12:0/3901537486 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ddc1037c0 0x7f3ddc199b00 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.364+0000 7f3de1400640 1 -- 172.21.15.12:0/3901537486 >> 172.21.15.12:0/3901537486 conn(0x7f3ddc0fed50 msgr2=0x7f3ddc107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:50.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.364+0000 7f3de1400640 1 -- 172.21.15.12:0/3901537486 shutdown_connections 2024-03-23T23:10:50.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.364+0000 7f3de1400640 1 -- 172.21.15.12:0/3901537486 wait complete. 2024-03-23T23:10:50.382 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T23:10:50.382 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:10:50.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.598+0000 7fb231000640 1 -- 172.21.15.12:0/998367727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb22c077c40 msgr2=0x7fb22c0784e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:50.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.598+0000 7fb231000640 1 --2- 172.21.15.12:0/998367727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb22c077c40 0x7fb22c0784e0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb224009be0 tx=0x7fb22402bef0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.599+0000 7fb231000640 1 -- 172.21.15.12:0/998367727 shutdown_connections 2024-03-23T23:10:50.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.599+0000 7fb231000640 1 --2- 172.21.15.12:0/998367727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb22c077c40 0x7fb22c0784e0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.599+0000 7fb231000640 1 --2- 172.21.15.12:0/998367727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb22c0772a0 0x7fb22c077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.599+0000 7fb231000640 1 --2- 172.21.15.12:0/998367727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb22c076060 0x7fb22c076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.599+0000 7fb231000640 1 -- 172.21.15.12:0/998367727 >> 172.21.15.12:0/998367727 conn(0x7fb22c0fee50 msgr2=0x7fb22c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:50.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.599+0000 7fb231000640 1 -- 172.21.15.12:0/998367727 shutdown_connections 2024-03-23T23:10:50.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.599+0000 7fb231000640 1 -- 172.21.15.12:0/998367727 wait complete. 2024-03-23T23:10:50.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.600+0000 7fb231000640 1 Processor -- start 2024-03-23T23:10:50.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.600+0000 7fb231000640 1 -- start start 2024-03-23T23:10:50.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.600+0000 7fb231000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb22c076060 0x7fb22c19df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:50.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.600+0000 7fb231000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb22c0772a0 0x7fb22c19e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:50.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.600+0000 7fb231000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb22c077c40 0x7fb22c1a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:50.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.600+0000 7fb231000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c1139b0 con 0x7fb22c077c40 2024-03-23T23:10:50.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.600+0000 7fb231000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c113830 con 0x7fb22c076060 2024-03-23T23:10:50.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.600+0000 7fb231000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c113b30 con 0x7fb22c0772a0 2024-03-23T23:10:50.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.600+0000 7fb223e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb22c0772a0 0x7fb22c19e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:50.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.600+0000 7fb229e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb22c076060 0x7fb22c19df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:50.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.600+0000 7fb223e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb22c0772a0 0x7fb22c19e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54150/0 (socket says 172.21.15.12:54150) 2024-03-23T23:10:50.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.600+0000 7fb223e00640 1 -- 172.21.15.12:0/3114889532 learned_addr learned my addr 172.21.15.12:0/3114889532 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:50.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.600+0000 7fb22a800640 1 --2- 172.21.15.12:0/3114889532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb22c077c40 0x7fb22c1a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:50.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.601+0000 7fb229e00640 1 -- 172.21.15.12:0/3114889532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb22c0772a0 msgr2=0x7fb22c19e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:50.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.601+0000 7fb229e00640 1 --2- 172.21.15.12:0/3114889532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb22c0772a0 0x7fb22c19e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.601+0000 7fb229e00640 1 -- 172.21.15.12:0/3114889532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb22c077c40 msgr2=0x7fb22c1a5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:50.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.601+0000 7fb229e00640 1 --2- 172.21.15.12:0/3114889532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb22c077c40 0x7fb22c1a5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.601+0000 7fb229e00640 1 -- 172.21.15.12:0/3114889532 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c1a5b70 con 0x7fb22c076060 2024-03-23T23:10:50.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.601+0000 7fb223e00640 1 --2- 172.21.15.12:0/3114889532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb22c0772a0 0x7fb22c19e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:50.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.601+0000 7fb229e00640 1 --2- 172.21.15.12:0/3114889532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb22c076060 0x7fb22c19df50 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb21c00e790 tx=0x7fb21c00ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:50.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.601+0000 7fb222a00640 1 -- 172.21.15.12:0/3114889532 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21c00eea0 con 0x7fb22c076060 2024-03-23T23:10:50.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.601+0000 7fb222a00640 1 -- 172.21.15.12:0/3114889532 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb21c00ccc0 con 0x7fb22c076060 2024-03-23T23:10:50.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.601+0000 7fb222a00640 1 -- 172.21.15.12:0/3114889532 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21c018780 con 0x7fb22c076060 2024-03-23T23:10:50.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.602+0000 7fb231000640 1 -- 172.21.15.12:0/3114889532 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb22c1a5e60 con 0x7fb22c076060 2024-03-23T23:10:50.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.602+0000 7fb231000640 1 -- 172.21.15.12:0/3114889532 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb22c1a6270 con 0x7fb22c076060 2024-03-23T23:10:50.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.603+0000 7fb231000640 1 -- 172.21.15.12:0/3114889532 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb22c110350 con 0x7fb22c076060 2024-03-23T23:10:50.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.605+0000 7fb222a00640 1 -- 172.21.15.12:0/3114889532 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb21c010430 con 0x7fb22c076060 2024-03-23T23:10:50.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.606+0000 7fb222a00640 1 --2- 172.21.15.12:0/3114889532 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb2100a2880 0x7fb2100a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:50.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.606+0000 7fb222a00640 1 -- 172.21.15.12:0/3114889532 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb21c0cebe0 con 0x7fb22c076060 2024-03-23T23:10:50.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.607+0000 7fb223e00640 1 --2- 172.21.15.12:0/3114889532 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb2100a2880 0x7fb2100a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:50.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.607+0000 7fb223e00640 1 --2- 172.21.15.12:0/3114889532 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb2100a2880 0x7fb2100a4d40 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb22c19f460 tx=0x7fb214005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:50.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.615+0000 7fb222a00640 1 -- 172.21.15.12:0/3114889532 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb21c09b5e0 con 0x7fb22c076060 2024-03-23T23:10:50.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:50 smithi012 ceph-mon[31695]: pgmap v661: 97 pgs: 97 active+clean; 2.0 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:10:50.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.863+0000 7fb231000640 1 -- 172.21.15.12:0/3114889532 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb22c02d320 con 0x7fb22c076060 2024-03-23T23:10:50.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.865+0000 7fb222a00640 1 -- 172.21.15.12:0/3114889532 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7fb21c09f680 con 0x7fb22c076060 2024-03-23T23:10:50.865 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:10:50.865 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:10:50.865 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:10:50.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.868+0000 7fb231000640 1 -- 172.21.15.12:0/3114889532 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb2100a2880 msgr2=0x7fb2100a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:50.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.868+0000 7fb231000640 1 --2- 172.21.15.12:0/3114889532 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb2100a2880 0x7fb2100a4d40 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb22c19f460 tx=0x7fb214005f90 comp rx=0 tx=0).stop 2024-03-23T23:10:50.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.868+0000 7fb231000640 1 -- 172.21.15.12:0/3114889532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb22c076060 msgr2=0x7fb22c19df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:50.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.868+0000 7fb231000640 1 --2- 172.21.15.12:0/3114889532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb22c076060 0x7fb22c19df50 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb21c00e790 tx=0x7fb21c00ec50 comp rx=0 tx=0).stop 2024-03-23T23:10:50.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.868+0000 7fb231000640 1 -- 172.21.15.12:0/3114889532 shutdown_connections 2024-03-23T23:10:50.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.868+0000 7fb231000640 1 --2- 172.21.15.12:0/3114889532 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb2100a2880 0x7fb2100a4d40 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.868+0000 7fb231000640 1 --2- 172.21.15.12:0/3114889532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb22c077c40 0x7fb22c1a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.868+0000 7fb231000640 1 --2- 172.21.15.12:0/3114889532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb22c0772a0 0x7fb22c19e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.868+0000 7fb231000640 1 --2- 172.21.15.12:0/3114889532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb22c076060 0x7fb22c19df50 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:50.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.868+0000 7fb231000640 1 -- 172.21.15.12:0/3114889532 >> 172.21.15.12:0/3114889532 conn(0x7fb22c0fee50 msgr2=0x7fb22c1004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:50.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.868+0000 7fb231000640 1 -- 172.21.15.12:0/3114889532 shutdown_connections 2024-03-23T23:10:50.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:50.868+0000 7fb231000640 1 -- 172.21.15.12:0/3114889532 wait complete. 2024-03-23T23:10:50.887 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:10:50.887 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-23T23:10:50.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:50 smithi203 ceph-mon[36693]: pgmap v661: 97 pgs: 97 active+clean; 2.0 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:10:51.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.115+0000 7febf5000640 1 -- 172.21.15.12:0/4001921165 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7febf01037c0 msgr2=0x7febf0103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.115+0000 7febf5000640 1 --2- 172.21.15.12:0/4001921165 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7febf01037c0 0x7febf0103be0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7febe0009900 tx=0x7febe002bc30 comp rx=0 tx=0).stop 2024-03-23T23:10:51.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.115+0000 7febf5000640 1 -- 172.21.15.12:0/4001921165 shutdown_connections 2024-03-23T23:10:51.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.115+0000 7febf5000640 1 --2- 172.21.15.12:0/4001921165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf0105370 0x7febf010bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.115+0000 7febf5000640 1 --2- 172.21.15.12:0/4001921165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febf01049b0 0x7febf0104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.115+0000 7febf5000640 1 --2- 172.21.15.12:0/4001921165 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7febf01037c0 0x7febf0103be0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.115+0000 7febf5000640 1 -- 172.21.15.12:0/4001921165 >> 172.21.15.12:0/4001921165 conn(0x7febf00fed50 msgr2=0x7febf0101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:51.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.115+0000 7febf5000640 1 -- 172.21.15.12:0/4001921165 shutdown_connections 2024-03-23T23:10:51.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.115+0000 7febf5000640 1 -- 172.21.15.12:0/4001921165 wait complete. 2024-03-23T23:10:51.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.116+0000 7febf5000640 1 Processor -- start 2024-03-23T23:10:51.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.116+0000 7febf5000640 1 -- start start 2024-03-23T23:10:51.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.117+0000 7febf5000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf01037c0 0x7febf0199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.117+0000 7febf5000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7febf01049b0 0x7febf019a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.117+0000 7febf5000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febf0105370 0x7febf01a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.117+0000 7febf5000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf010f480 con 0x7febf01037c0 2024-03-23T23:10:51.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.117+0000 7febf5000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf010f300 con 0x7febf01049b0 2024-03-23T23:10:51.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.117+0000 7febf5000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf010f600 con 0x7febf0105370 2024-03-23T23:10:51.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.117+0000 7febef400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf01037c0 0x7febf0199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.117+0000 7febef400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf01037c0 0x7febf0199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54592/0 (socket says 172.21.15.12:54592) 2024-03-23T23:10:51.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.117+0000 7febef400640 1 -- 172.21.15.12:0/1919987899 learned_addr learned my addr 172.21.15.12:0/1919987899 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:51.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.117+0000 7febeea00640 1 --2- 172.21.15.12:0/1919987899 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7febf01049b0 0x7febf019a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.117+0000 7febefe00640 1 --2- 172.21.15.12:0/1919987899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febf0105370 0x7febf01a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:50 smithi173 ceph-mon[36116]: pgmap v661: 97 pgs: 97 active+clean; 2.0 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:10:51.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.117+0000 7febef400640 1 -- 172.21.15.12:0/1919987899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febf0105370 msgr2=0x7febf01a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.117+0000 7febef400640 1 --2- 172.21.15.12:0/1919987899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febf0105370 0x7febf01a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.117+0000 7febef400640 1 -- 172.21.15.12:0/1919987899 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7febf01049b0 msgr2=0x7febf019a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.117+0000 7febef400640 1 --2- 172.21.15.12:0/1919987899 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7febf01049b0 0x7febf019a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.117+0000 7febef400640 1 -- 172.21.15.12:0/1919987899 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf01a1710 con 0x7febf01037c0 2024-03-23T23:10:51.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.117+0000 7febeea00640 1 --2- 172.21.15.12:0/1919987899 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7febf01049b0 0x7febf019a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:51.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.118+0000 7febef400640 1 --2- 172.21.15.12:0/1919987899 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf01037c0 0x7febf0199af0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7febe0000c00 tx=0x7febe002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:51.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.118+0000 7febe7e00640 1 -- 172.21.15.12:0/1919987899 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe0043020 con 0x7febf01037c0 2024-03-23T23:10:51.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.118+0000 7febe7e00640 1 -- 172.21.15.12:0/1919987899 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7febe00054e0 con 0x7febf01037c0 2024-03-23T23:10:51.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.118+0000 7febf5000640 1 -- 172.21.15.12:0/1919987899 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf01a19a0 con 0x7febf01037c0 2024-03-23T23:10:51.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.118+0000 7febe7e00640 1 -- 172.21.15.12:0/1919987899 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe0003bd0 con 0x7febf01037c0 2024-03-23T23:10:51.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.118+0000 7febf5000640 1 -- 172.21.15.12:0/1919987899 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf01a1ef0 con 0x7febf01037c0 2024-03-23T23:10:51.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.122+0000 7febe7e00640 1 -- 172.21.15.12:0/1919987899 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7febe0046020 con 0x7febf01037c0 2024-03-23T23:10:51.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.122+0000 7febf9281740 1 Processor -- start 2024-03-23T23:10:51.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.122+0000 7febf9281740 1 -- start start 2024-03-23T23:10:51.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.122+0000 7febf9281740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558101d299f0 0x558101d29e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.122+0000 7febf9281740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d2abc0 0x558101d2b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.122+0000 7febf9281740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558101d2b580 0x558101d31e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.122+0000 7febf9281740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x558101c8e2c0 con 0x558101d299f0 2024-03-23T23:10:51.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.122+0000 7febf9281740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x558101c36c90 con 0x558101d2abc0 2024-03-23T23:10:51.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.122+0000 7febf9281740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x558101c8a360 con 0x558101d2b580 2024-03-23T23:10:51.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.123+0000 7febef400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d2abc0 0x558101d2b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.123+0000 7febefe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558101d299f0 0x558101d29e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.123+0000 7febef400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d2abc0 0x558101d2b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37664/0 (socket says 172.21.15.12:37664) 2024-03-23T23:10:51.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.123+0000 7febeea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558101d2b580 0x558101d31e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.123+0000 7febef400640 1 -- 172.21.15.12:0/3021123626 learned_addr learned my addr 172.21.15.12:0/3021123626 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:51.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.123+0000 7febef400640 1 -- 172.21.15.12:0/3021123626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558101d2b580 msgr2=0x558101d31e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:51.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.123+0000 7febef400640 1 --2- 172.21.15.12:0/3021123626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558101d2b580 0x558101d31e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.123+0000 7febef400640 1 -- 172.21.15.12:0/3021123626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558101d299f0 msgr2=0x558101d29e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:51.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.123+0000 7febef400640 1 --2- 172.21.15.12:0/3021123626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558101d299f0 0x558101d29e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.123+0000 7febef400640 1 -- 172.21.15.12:0/3021123626 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558101c5b080 con 0x558101d2abc0 2024-03-23T23:10:51.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.123+0000 7febeea00640 1 --2- 172.21.15.12:0/3021123626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558101d2b580 0x558101d31e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:51.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.124+0000 7febef400640 1 --2- 172.21.15.12:0/3021123626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d2abc0 0x558101d2b040 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7febe00f49f0 tx=0x7febe00f4ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5ad09a0aff68d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:51.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.124+0000 7febe7e00640 1 --2- 172.21.15.12:0/1919987899 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7febd40a2800 0x7febd40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.124+0000 7febe4c00640 1 -- 172.21.15.12:0/3021123626 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe0104070 con 0x558101d2abc0 2024-03-23T23:10:51.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.124+0000 7febefe00640 1 --2- 172.21.15.12:0/1919987899 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7febd40a2800 0x7febd40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.124+0000 7febe4c00640 1 -- 172.21.15.12:0/3021123626 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7febe00bdb00 con 0x558101d2abc0 2024-03-23T23:10:51.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.124+0000 7febe4c00640 1 -- 172.21.15.12:0/3021123626 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe00bcd60 con 0x558101d2abc0 2024-03-23T23:10:51.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.124+0000 7febe7e00640 1 -- 172.21.15.12:0/1919987899 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7febe00f0a50 con 0x7febf01037c0 2024-03-23T23:10:51.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.125+0000 7febf9281740 1 -- 172.21.15.12:0/3021123626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d2abc0 msgr2=0x558101d2b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.125+0000 7febf9281740 1 --2- 172.21.15.12:0/3021123626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d2abc0 0x558101d2b040 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7febe00f49f0 tx=0x7febe00f4ae0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.125+0000 7febefe00640 1 --2- 172.21.15.12:0/1919987899 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7febd40a2800 0x7febd40a4cc0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x558101d2aac0 tx=0x7febe8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:51.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.125+0000 7febf9281740 1 -- 172.21.15.12:0/3021123626 shutdown_connections 2024-03-23T23:10:51.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.125+0000 7febf9281740 1 --2- 172.21.15.12:0/3021123626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558101d2b580 0x558101d31e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.125+0000 7febf9281740 1 --2- 172.21.15.12:0/3021123626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d2abc0 0x558101d2b040 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.125+0000 7febf9281740 1 --2- 172.21.15.12:0/3021123626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558101d299f0 0x558101d29e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.125+0000 7febf9281740 1 -- 172.21.15.12:0/3021123626 >> 172.21.15.12:0/3021123626 conn(0x558101d24de0 msgr2=0x558101d251e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.125+0000 7febf9281740 1 -- 172.21.15.12:0/3021123626 shutdown_connections 2024-03-23T23:10:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.125+0000 7febf9281740 1 -- 172.21.15.12:0/3021123626 wait complete. 2024-03-23T23:10:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.125+0000 7febf9281740 1 -- start start 2024-03-23T23:10:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.125+0000 7febf9281740 1 Processor -- start 2024-03-23T23:10:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.126+0000 7febf9281740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558101d299f0 0x558101d33830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.126+0000 7febf9281740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d2abc0 0x558101d33d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.126+0000 7febf9281740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558101d2b580 0x558101d3ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.126+0000 7febf9281740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x558101c36c90 con 0x558101d2b580 2024-03-23T23:10:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.126+0000 7febf9281740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x558101c4a0d0 con 0x558101d2abc0 2024-03-23T23:10:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.126+0000 7febf9281740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x558101d32350 con 0x558101d299f0 2024-03-23T23:10:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.126+0000 7febf9281740 10 client.0 _ll_register_callbacks cb 0x558101c4ee80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.126+0000 7febeea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d2abc0 0x558101d33d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.126+0000 7febeea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d2abc0 0x558101d33d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37672/0 (socket says 172.21.15.12:37672) 2024-03-23T23:10:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.126+0000 7febef400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558101d299f0 0x558101d33830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.126+0000 7febeea00640 1 -- 172.21.15.12:0/507412922 learned_addr learned my addr 172.21.15.12:0/507412922 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.126+0000 7febefe00640 1 --2- 172.21.15.12:0/507412922 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558101d2b580 0x558101d3ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.127+0000 7febeea00640 1 -- 172.21.15.12:0/507412922 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558101d299f0 msgr2=0x558101d33830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:51.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.127+0000 7febeea00640 1 --2- 172.21.15.12:0/507412922 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558101d299f0 0x558101d33830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.127+0000 7febeea00640 1 -- 172.21.15.12:0/507412922 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558101d2b580 msgr2=0x558101d3ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:51.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.127+0000 7febeea00640 1 --2- 172.21.15.12:0/507412922 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558101d2b580 0x558101d3ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.127+0000 7febeea00640 1 -- 172.21.15.12:0/507412922 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5581019a23f0 con 0x558101d2abc0 2024-03-23T23:10:51.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.127+0000 7febef400640 1 --2- 172.21.15.12:0/507412922 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558101d299f0 0x558101d33830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:51.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.127+0000 7febeea00640 1 --2- 172.21.15.12:0/507412922 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d2abc0 0x558101d33d70 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7febdc012d20 tx=0x7febdc00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bed66354ded00683 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:51.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.127+0000 7febd9600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:10:51.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.127+0000 7febd9600640 1 -- 172.21.15.12:0/507412922 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febdc0120d0 con 0x558101d2abc0 2024-03-23T23:10:51.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.127+0000 7febd9600640 1 -- 172.21.15.12:0/507412922 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7febdc010430 con 0x558101d2abc0 2024-03-23T23:10:51.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.127+0000 7febf9281740 1 -- 172.21.15.12:0/507412922 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7febe00bdb00 con 0x558101d2abc0 2024-03-23T23:10:51.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.127+0000 7febd9600640 1 -- 172.21.15.12:0/507412922 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7febdc012270 con 0x558101d2abc0 2024-03-23T23:10:51.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.127+0000 7febf9281740 10 client.27842 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:51.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.127+0000 7febf9281740 1 -- 172.21.15.12:0/507412922 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x558101a6c7d0 con 0x558101d2abc0 2024-03-23T23:10:51.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.128+0000 7febd9600640 1 -- 172.21.15.12:0/507412922 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7febe00bdb00 con 0x558101d2abc0 2024-03-23T23:10:51.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.128+0000 7febd9600640 1 -- 172.21.15.12:0/507412922 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7febdc01c780 con 0x558101d2abc0 2024-03-23T23:10:51.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.128+0000 7febf9281740 10 client.27842 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:51.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.128+0000 7febf9281740 10 client.27842 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:10:51.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.128+0000 7febf9281740 20 client.27842 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:51.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.128+0000 7febf9281740 1 --2- 172.21.15.12:0/507412922 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x558101d3f290 0x558101d416d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.128+0000 7febf9281740 4 client.27842 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:10:51.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.128+0000 7febf9281740 1 -- 172.21.15.12:0/507412922 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558101c9b800 con 0x558101d3f290 2024-03-23T23:10:51.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.128+0000 7febef400640 1 --2- 172.21.15.12:0/507412922 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x558101d3f290 0x558101d416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.129+0000 7febef400640 1 --2- 172.21.15.12:0/507412922 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x558101d3f290 0x558101d416d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a2996b288231672f server_cookie=a00296c313e80eb0 in_seq=0 out_seq=0 2024-03-23T23:10:51.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.129+0000 7febd9600640 10 client.27842 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:10:51.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.129+0000 7febd9600640 1 -- 172.21.15.12:0/507412922 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558101c9b800 con 0x558101d3f290 2024-03-23T23:10:51.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.130+0000 7febd9600640 10 client.27842 handle_command_reply: tid=0 2024-03-23T23:10:51.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.130+0000 7febf9281740 1 client.27842 shutdown 2024-03-23T23:10:51.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.130+0000 7febf9281740 1 -- 172.21.15.12:0/507412922 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d2abc0 msgr2=0x558101d33d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.130+0000 7febf9281740 1 --2- 172.21.15.12:0/507412922 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d2abc0 0x558101d33d70 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7febdc012d20 tx=0x7febdc00f790 comp rx=0 tx=0).stop 2024-03-23T23:10:51.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.130+0000 7febf9281740 1 -- 172.21.15.12:0/507412922 shutdown_connections 2024-03-23T23:10:51.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.130+0000 7febf9281740 1 --2- 172.21.15.12:0/507412922 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x558101d3f290 0x558101d416d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.130+0000 7febf9281740 1 --2- 172.21.15.12:0/507412922 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558101d2b580 0x558101d3ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.130+0000 7febf9281740 1 --2- 172.21.15.12:0/507412922 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d2abc0 0x558101d33d70 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.130+0000 7febf9281740 1 --2- 172.21.15.12:0/507412922 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558101d299f0 0x558101d33830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.130+0000 7febd9600640 0 client.27842 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:10:51.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.130+0000 7febf9281740 1 -- 172.21.15.12:0/507412922 >> 172.21.15.12:0/507412922 conn(0x558101d24de0 msgr2=0x558101d24160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:51.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.130+0000 7febf9281740 1 -- 172.21.15.12:0/507412922 shutdown_connections 2024-03-23T23:10:51.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.130+0000 7febf9281740 1 -- 172.21.15.12:0/507412922 wait complete. 2024-03-23T23:10:51.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.130+0000 7febf9281740 20 client.27842 trim_cache size 0 max 16384 2024-03-23T23:10:51.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.144+0000 7febf9281740 1 Processor -- start 2024-03-23T23:10:51.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.144+0000 7febf9281740 1 -- start start 2024-03-23T23:10:51.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.144+0000 7febf9281740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d299f0 0x558101d31ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.144+0000 7febf9281740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558101d2abc0 0x558101d3ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.144+0000 7febf9281740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558101d2b580 0x558101d23fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.144+0000 7febf9281740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf01acfc0 con 0x558101d2abc0 2024-03-23T23:10:51.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.144+0000 7febf9281740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf01aaac0 con 0x558101d299f0 2024-03-23T23:10:51.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.144+0000 7febf9281740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf01a8b60 con 0x558101d2b580 2024-03-23T23:10:51.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.144+0000 7febef400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d299f0 0x558101d31ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.144+0000 7febef400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d299f0 0x558101d31ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37682/0 (socket says 172.21.15.12:37682) 2024-03-23T23:10:51.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.144+0000 7febef400640 1 -- 172.21.15.12:0/4037486869 learned_addr learned my addr 172.21.15.12:0/4037486869 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:51.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.144+0000 7febeea00640 1 --2- 172.21.15.12:0/4037486869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558101d2abc0 0x558101d3ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.145+0000 7febefe00640 1 --2- 172.21.15.12:0/4037486869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558101d2b580 0x558101d23fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.145+0000 7febef400640 1 -- 172.21.15.12:0/4037486869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558101d2b580 msgr2=0x558101d23fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:51.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.145+0000 7febef400640 1 --2- 172.21.15.12:0/4037486869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558101d2b580 0x558101d23fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.145+0000 7febef400640 1 -- 172.21.15.12:0/4037486869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558101d2abc0 msgr2=0x558101d3ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:51.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.145+0000 7febef400640 1 --2- 172.21.15.12:0/4037486869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558101d2abc0 0x558101d3ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.145+0000 7febef400640 1 -- 172.21.15.12:0/4037486869 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558101d2a640 con 0x558101d299f0 2024-03-23T23:10:51.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.145+0000 7febef400640 1 --2- 172.21.15.12:0/4037486869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d299f0 0x558101d31ad0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7febe00b9d80 tx=0x7febe00b9040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1c4e44b23c028cf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:51.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.145+0000 7febdbe00640 1 -- 172.21.15.12:0/4037486869 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe0045700 con 0x558101d299f0 2024-03-23T23:10:51.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.145+0000 7febdbe00640 1 -- 172.21.15.12:0/4037486869 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7febe0005690 con 0x558101d299f0 2024-03-23T23:10:51.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.145+0000 7febdbe00640 1 -- 172.21.15.12:0/4037486869 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe0045920 con 0x558101d299f0 2024-03-23T23:10:51.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.146+0000 7febf9281740 1 -- 172.21.15.12:0/4037486869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d299f0 msgr2=0x558101d31ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.146+0000 7febf9281740 1 --2- 172.21.15.12:0/4037486869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d299f0 0x558101d31ad0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7febe00b9d80 tx=0x7febe00b9040 comp rx=0 tx=0).stop 2024-03-23T23:10:51.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.146+0000 7febf9281740 1 -- 172.21.15.12:0/4037486869 shutdown_connections 2024-03-23T23:10:51.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.146+0000 7febf9281740 1 --2- 172.21.15.12:0/4037486869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558101d2b580 0x558101d23fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.146+0000 7febf9281740 1 --2- 172.21.15.12:0/4037486869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558101d2abc0 0x558101d3ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.146+0000 7febf9281740 1 --2- 172.21.15.12:0/4037486869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d299f0 0x558101d31ad0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.146+0000 7febf9281740 1 -- 172.21.15.12:0/4037486869 >> 172.21.15.12:0/4037486869 conn(0x558101d24de0 msgr2=0x558101d238d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:51.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.146+0000 7febf9281740 1 -- 172.21.15.12:0/4037486869 shutdown_connections 2024-03-23T23:10:51.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.146+0000 7febf9281740 1 -- 172.21.15.12:0/4037486869 wait complete. 2024-03-23T23:10:51.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.146+0000 7febf9281740 1 -- start start 2024-03-23T23:10:51.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.147+0000 7febf9281740 1 Processor -- start 2024-03-23T23:10:51.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.147+0000 7febf9281740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558101d299f0 0x558101d36cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.147+0000 7febf9281740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d2abc0 0x558101d37230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.147+0000 7febf9281740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558101d2b580 0x558101d3a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.147+0000 7febf9281740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf01a9160 con 0x558101d2b580 2024-03-23T23:10:51.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.147+0000 7febf9281740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf01aa460 con 0x558101d2abc0 2024-03-23T23:10:51.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.147+0000 7febf9281740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf01a97a0 con 0x558101d299f0 2024-03-23T23:10:51.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.147+0000 7febf9281740 10 client.0 _ll_register_callbacks cb 0x7febdc009be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:51.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.147+0000 7febefe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558101d2b580 0x558101d3a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.147+0000 7febeea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d2abc0 0x558101d37230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.148+0000 7febeea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d2abc0 0x558101d37230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37690/0 (socket says 172.21.15.12:37690) 2024-03-23T23:10:51.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.148+0000 7febeea00640 1 -- 172.21.15.12:0/471852973 learned_addr learned my addr 172.21.15.12:0/471852973 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:51.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.148+0000 7febefe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558101d2b580 0x558101d3a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54630/0 (socket says 172.21.15.12:54630) 2024-03-23T23:10:51.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.148+0000 7febef400640 1 --2- 172.21.15.12:0/471852973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558101d299f0 0x558101d36cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.148+0000 7febeea00640 1 -- 172.21.15.12:0/471852973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558101d299f0 msgr2=0x558101d36cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:51.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.148+0000 7febeea00640 1 --2- 172.21.15.12:0/471852973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558101d299f0 0x558101d36cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.148+0000 7febeea00640 1 -- 172.21.15.12:0/471852973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558101d2b580 msgr2=0x558101d3a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:51.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.148+0000 7febeea00640 1 --2- 172.21.15.12:0/471852973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558101d2b580 0x558101d3a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.148+0000 7febeea00640 1 -- 172.21.15.12:0/471852973 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558101d3a7d0 con 0x558101d2abc0 2024-03-23T23:10:51.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.148+0000 7febeea00640 1 --2- 172.21.15.12:0/471852973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d2abc0 0x558101d37230 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7febdc01cc70 tx=0x7febdc01c9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4023e45be83c1d56 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:51.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.148+0000 7febd9600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:10:51.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.148+0000 7febd9600640 1 -- 172.21.15.12:0/471852973 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febdc0120d0 con 0x558101d2abc0 2024-03-23T23:10:51.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.149+0000 7febf9281740 1 -- 172.21.15.12:0/471852973 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558101c95b50 con 0x558101d2abc0 2024-03-23T23:10:51.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.149+0000 7febd9600640 1 -- 172.21.15.12:0/471852973 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7febdc005040 con 0x558101d2abc0 2024-03-23T23:10:51.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.149+0000 7febd9600640 1 -- 172.21.15.12:0/471852973 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7febdc009ea0 con 0x558101d2abc0 2024-03-23T23:10:51.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.149+0000 7febf9281740 10 client.27848 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:51.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.149+0000 7febf9281740 1 -- 172.21.15.12:0/471852973 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x558101d9b430 con 0x558101d2abc0 2024-03-23T23:10:51.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.149+0000 7febd9600640 1 -- 172.21.15.12:0/471852973 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x558101c95b50 con 0x558101d2abc0 2024-03-23T23:10:51.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.149+0000 7febd9600640 1 -- 172.21.15.12:0/471852973 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7febdc004c20 con 0x558101d2abc0 2024-03-23T23:10:51.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.149+0000 7febf9281740 10 client.27848 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:51.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.149+0000 7febf9281740 10 client.27848 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:10:51.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.149+0000 7febf9281740 20 client.27848 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:51.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.149+0000 7febf9281740 1 --2- 172.21.15.12:0/471852973 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x558101d9bd50 0x558101d9e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.149+0000 7febf9281740 4 client.27848 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:10:51.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.149+0000 7febf9281740 1 -- 172.21.15.12:0/471852973 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558101c13500 con 0x558101d9bd50 2024-03-23T23:10:51.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.150+0000 7febef400640 1 --2- 172.21.15.12:0/471852973 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x558101d9bd50 0x558101d9e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.150+0000 7febef400640 1 --2- 172.21.15.12:0/471852973 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x558101d9bd50 0x558101d9e170 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5cd6c6dc3dd9b711 server_cookie=a80f8b3088e181 in_seq=0 out_seq=0 2024-03-23T23:10:51.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.150+0000 7febd9600640 10 client.27848 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:10:51.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.150+0000 7febd9600640 1 -- 172.21.15.12:0/471852973 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558101c13500 con 0x558101d9bd50 2024-03-23T23:10:51.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.150+0000 7febd9600640 10 client.27848 handle_command_reply: tid=0 2024-03-23T23:10:51.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.150+0000 7febf9281740 1 client.27848 shutdown 2024-03-23T23:10:51.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.151+0000 7febf9281740 1 -- 172.21.15.12:0/471852973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d2abc0 msgr2=0x558101d37230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.151+0000 7febf9281740 1 --2- 172.21.15.12:0/471852973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d2abc0 0x558101d37230 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7febdc01cc70 tx=0x7febdc01c9a0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.151+0000 7febf9281740 1 -- 172.21.15.12:0/471852973 shutdown_connections 2024-03-23T23:10:51.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.151+0000 7febf9281740 1 --2- 172.21.15.12:0/471852973 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x558101d9bd50 0x558101d9e170 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.151+0000 7febf9281740 1 --2- 172.21.15.12:0/471852973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558101d2b580 0x558101d3a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.151+0000 7febd9600640 0 client.27848 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:10:51.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.151+0000 7febf9281740 1 --2- 172.21.15.12:0/471852973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558101d2abc0 0x558101d37230 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.151+0000 7febf9281740 1 --2- 172.21.15.12:0/471852973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558101d299f0 0x558101d36cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.151+0000 7febf9281740 1 -- 172.21.15.12:0/471852973 >> 172.21.15.12:0/471852973 conn(0x558101d24de0 msgr2=0x558101d28040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:51.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.151+0000 7febf9281740 1 -- 172.21.15.12:0/471852973 shutdown_connections 2024-03-23T23:10:51.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.151+0000 7febf9281740 1 -- 172.21.15.12:0/471852973 wait complete. 2024-03-23T23:10:51.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.151+0000 7febf9281740 20 client.27848 trim_cache size 0 max 16384 2024-03-23T23:10:51.154 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:10:51.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.151+0000 7febf5000640 1 -- 172.21.15.12:0/1919987899 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7febd40a2800 msgr2=0x7febd40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.151+0000 7febf5000640 1 --2- 172.21.15.12:0/1919987899 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7febd40a2800 0x7febd40a4cc0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x558101d2aac0 tx=0x7febe8009500 comp rx=0 tx=0).stop 2024-03-23T23:10:51.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.151+0000 7febf5000640 1 -- 172.21.15.12:0/1919987899 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf01037c0 msgr2=0x7febf0199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.152+0000 7febf5000640 1 --2- 172.21.15.12:0/1919987899 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf01037c0 0x7febf0199af0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7febe0000c00 tx=0x7febe002c650 comp rx=0 tx=0).stop 2024-03-23T23:10:51.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.152+0000 7febf5000640 1 -- 172.21.15.12:0/1919987899 shutdown_connections 2024-03-23T23:10:51.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.152+0000 7febf5000640 1 --2- 172.21.15.12:0/1919987899 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7febd40a2800 0x7febd40a4cc0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.152+0000 7febf5000640 1 --2- 172.21.15.12:0/1919987899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febf0105370 0x7febf01a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.152+0000 7febf5000640 1 --2- 172.21.15.12:0/1919987899 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7febf01049b0 0x7febf019a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.152+0000 7febf5000640 1 --2- 172.21.15.12:0/1919987899 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf01037c0 0x7febf0199af0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.152+0000 7febf5000640 1 -- 172.21.15.12:0/1919987899 >> 172.21.15.12:0/1919987899 conn(0x7febf00fed50 msgr2=0x7febf0107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:51.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.152+0000 7febf5000640 1 -- 172.21.15.12:0/1919987899 shutdown_connections 2024-03-23T23:10:51.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.152+0000 7febf5000640 1 -- 172.21.15.12:0/1919987899 wait complete. 2024-03-23T23:10:51.169 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:10:51.169 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-23T23:10:51.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.384+0000 7f49dda00640 1 -- 172.21.15.12:0/3528759831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49d8100690 msgr2=0x7f49d8100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.384+0000 7f49dda00640 1 --2- 172.21.15.12:0/3528759831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49d8100690 0x7f49d8100af0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f49c0009900 tx=0x7f49c002bc30 comp rx=0 tx=0).stop 2024-03-23T23:10:51.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.384+0000 7f49dda00640 1 -- 172.21.15.12:0/3528759831 shutdown_connections 2024-03-23T23:10:51.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.384+0000 7f49dda00640 1 --2- 172.21.15.12:0/3528759831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49d8101030 0x7f49d81018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.384+0000 7f49dda00640 1 --2- 172.21.15.12:0/3528759831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49d8100690 0x7f49d8100af0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.384+0000 7f49dda00640 1 --2- 172.21.15.12:0/3528759831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49d80ff620 0x7f49d80ffa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.384+0000 7f49dda00640 1 -- 172.21.15.12:0/3528759831 >> 172.21.15.12:0/3528759831 conn(0x7f49d80fae60 msgr2=0x7f49d80fd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:51.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.384+0000 7f49dda00640 1 -- 172.21.15.12:0/3528759831 shutdown_connections 2024-03-23T23:10:51.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.384+0000 7f49dda00640 1 -- 172.21.15.12:0/3528759831 wait complete. 2024-03-23T23:10:51.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.385+0000 7f49dda00640 1 Processor -- start 2024-03-23T23:10:51.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.385+0000 7f49dda00640 1 -- start start 2024-03-23T23:10:51.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.386+0000 7f49dda00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49d80ff620 0x7f49d8199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.386+0000 7f49dda00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49d8100690 0x7f49d819a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.386+0000 7f49dda00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49d8101030 0x7f49d81a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.386+0000 7f49dda00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d810f590 con 0x7f49d8101030 2024-03-23T23:10:51.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.386+0000 7f49dda00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d810f410 con 0x7f49d80ff620 2024-03-23T23:10:51.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.386+0000 7f49dda00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d810f710 con 0x7f49d8100690 2024-03-23T23:10:51.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.386+0000 7f49d6a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49d80ff620 0x7f49d8199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.386+0000 7f49d6000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49d8100690 0x7f49d819a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.386+0000 7f49d7400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49d8101030 0x7f49d81a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.386+0000 7f49d6a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49d80ff620 0x7f49d8199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37706/0 (socket says 172.21.15.12:37706) 2024-03-23T23:10:51.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.386+0000 7f49d6000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49d8100690 0x7f49d819a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54230/0 (socket says 172.21.15.12:54230) 2024-03-23T23:10:51.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.386+0000 7f49d6a00640 1 -- 172.21.15.12:0/1540961619 learned_addr learned my addr 172.21.15.12:0/1540961619 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:51.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.386+0000 7f49d6000640 1 -- 172.21.15.12:0/1540961619 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49d80ff620 msgr2=0x7f49d8199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.386+0000 7f49d6000640 1 --2- 172.21.15.12:0/1540961619 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49d80ff620 0x7f49d8199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.386+0000 7f49d6000640 1 -- 172.21.15.12:0/1540961619 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49d8101030 msgr2=0x7f49d81a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.386+0000 7f49d6000640 1 --2- 172.21.15.12:0/1540961619 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49d8101030 0x7f49d81a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.386+0000 7f49d6000640 1 -- 172.21.15.12:0/1540961619 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d81a1800 con 0x7f49d8100690 2024-03-23T23:10:51.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.387+0000 7f49d6000640 1 --2- 172.21.15.12:0/1540961619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49d8100690 0x7f49d819a120 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f49c0036040 tx=0x7f49c002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:51.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.387+0000 7f49cf400640 1 -- 172.21.15.12:0/1540961619 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0043020 con 0x7f49d8100690 2024-03-23T23:10:51.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.387+0000 7f49cf400640 1 -- 172.21.15.12:0/1540961619 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49c00054e0 con 0x7f49d8100690 2024-03-23T23:10:51.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.387+0000 7f49cf400640 1 -- 172.21.15.12:0/1540961619 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0003c30 con 0x7f49d8100690 2024-03-23T23:10:51.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.387+0000 7f49dda00640 1 -- 172.21.15.12:0/1540961619 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d81a1a90 con 0x7f49d8100690 2024-03-23T23:10:51.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.387+0000 7f49dda00640 1 -- 172.21.15.12:0/1540961619 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d81a1f60 con 0x7f49d8100690 2024-03-23T23:10:51.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.390+0000 7f49cf400640 1 -- 172.21.15.12:0/1540961619 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f49c0046020 con 0x7f49d8100690 2024-03-23T23:10:51.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.391+0000 7f49e082f740 1 Processor -- start 2024-03-23T23:10:51.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.391+0000 7f49e082f740 1 -- start start 2024-03-23T23:10:51.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.391+0000 7f49e082f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058ffe810 0x559058ffec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.391+0000 7f49e082f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559058fff9e0 0x559058fffe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.391+0000 7f49e082f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590590003a0 0x559059006c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.391+0000 7f49e082f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x559058f632c0 con 0x5590590003a0 2024-03-23T23:10:51.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.391+0000 7f49e082f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x559058f0bc90 con 0x559058fff9e0 2024-03-23T23:10:51.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.391+0000 7f49e082f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x559058f5f360 con 0x559058ffe810 2024-03-23T23:10:51.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.392+0000 7f49d7400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058ffe810 0x559058ffec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.392+0000 7f49d7400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058ffe810 0x559058ffec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54234/0 (socket says 172.21.15.12:54234) 2024-03-23T23:10:51.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.392+0000 7f49d7400640 1 -- 172.21.15.12:0/4282163039 learned_addr learned my addr 172.21.15.12:0/4282163039 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:51.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.392+0000 7f49d6000640 1 --2- 172.21.15.12:0/4282163039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590590003a0 0x559059006c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.392+0000 7f49d6a00640 1 --2- 172.21.15.12:0/4282163039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559058fff9e0 0x559058fffe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.392+0000 7f49d7400640 1 -- 172.21.15.12:0/4282163039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559058fff9e0 msgr2=0x559058fffe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:51.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.392+0000 7f49d7400640 1 --2- 172.21.15.12:0/4282163039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559058fff9e0 0x559058fffe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.392+0000 7f49d7400640 1 -- 172.21.15.12:0/4282163039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590590003a0 msgr2=0x559059006c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:51.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.392+0000 7f49d7400640 1 --2- 172.21.15.12:0/4282163039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590590003a0 0x559059006c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.392+0000 7f49d7400640 1 -- 172.21.15.12:0/4282163039 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559058f30080 con 0x559058ffe810 2024-03-23T23:10:51.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.392+0000 7f49d6a00640 1 --2- 172.21.15.12:0/4282163039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559058fff9e0 0x559058fffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:10:51.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.392+0000 7f49d7400640 1 --2- 172.21.15.12:0/4282163039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058ffe810 0x559058ffec30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f49d0009fd0 tx=0x7f49d00118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e83a139ed90de6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:51.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.392+0000 7f49cf400640 1 --2- 172.21.15.12:0/1540961619 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f49bc0a2880 0x7f49bc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.393+0000 7f49c7e00640 1 -- 172.21.15.12:0/4282163039 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d0004d20 con 0x559058ffe810 2024-03-23T23:10:51.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.393+0000 7f49c7e00640 1 -- 172.21.15.12:0/4282163039 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49d0004050 con 0x559058ffe810 2024-03-23T23:10:51.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.393+0000 7f49cf400640 1 -- 172.21.15.12:0/1540961619 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f49c00f0970 con 0x7f49d8100690 2024-03-23T23:10:51.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.393+0000 7f49c7e00640 1 -- 172.21.15.12:0/4282163039 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d0005260 con 0x559058ffe810 2024-03-23T23:10:51.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.393+0000 7f49d6a00640 1 --2- 172.21.15.12:0/1540961619 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f49bc0a2880 0x7f49bc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.393+0000 7f49e082f740 1 -- 172.21.15.12:0/4282163039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058ffe810 msgr2=0x559058ffec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.393+0000 7f49e082f740 1 --2- 172.21.15.12:0/4282163039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058ffe810 0x559058ffec30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f49d0009fd0 tx=0x7f49d00118d0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.394+0000 7f49e082f740 1 -- 172.21.15.12:0/4282163039 shutdown_connections 2024-03-23T23:10:51.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.394+0000 7f49d6a00640 1 --2- 172.21.15.12:0/1540961619 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f49bc0a2880 0x7f49bc0a4d40 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f49c8009eb0 tx=0x7f49c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:51.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.394+0000 7f49e082f740 1 --2- 172.21.15.12:0/4282163039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590590003a0 0x559059006c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.394+0000 7f49e082f740 1 --2- 172.21.15.12:0/4282163039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559058fff9e0 0x559058fffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.394+0000 7f49e082f740 1 --2- 172.21.15.12:0/4282163039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058ffe810 0x559058ffec30 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.394+0000 7f49e082f740 1 -- 172.21.15.12:0/4282163039 >> 172.21.15.12:0/4282163039 conn(0x559058ff9c00 msgr2=0x559058ffa000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:51.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.394+0000 7f49e082f740 1 -- 172.21.15.12:0/4282163039 shutdown_connections 2024-03-23T23:10:51.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.394+0000 7f49e082f740 1 -- 172.21.15.12:0/4282163039 wait complete. 2024-03-23T23:10:51.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.394+0000 7f49e082f740 1 -- start start 2024-03-23T23:10:51.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.394+0000 7f49e082f740 1 Processor -- start 2024-03-23T23:10:51.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.395+0000 7f49e082f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058ffe810 0x559059008a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.395+0000 7f49e082f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559058fff9e0 0x559059008fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.395+0000 7f49e082f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590590003a0 0x559059010010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.395+0000 7f49e082f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590590073b0 con 0x559058fff9e0 2024-03-23T23:10:51.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.395+0000 7f49e082f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x559058f1f0d0 con 0x5590590003a0 2024-03-23T23:10:51.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.395+0000 7f49e082f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x559059007530 con 0x559058ffe810 2024-03-23T23:10:51.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.395+0000 7f49e082f740 10 client.0 _ll_register_callbacks cb 0x559058f23e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:51.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.395+0000 7f49d6000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559058fff9e0 0x559059008fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.395+0000 7f49d6a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058ffe810 0x559059008a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.395+0000 7f49d7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590590003a0 0x559059010010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.395+0000 7f49d6000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559058fff9e0 0x559059008fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54650/0 (socket says 172.21.15.12:54650) 2024-03-23T23:10:51.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.395+0000 7f49d6a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058ffe810 0x559059008a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54248/0 (socket says 172.21.15.12:54248) 2024-03-23T23:10:51.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.395+0000 7f49d6000640 1 -- 172.21.15.12:0/3094905776 learned_addr learned my addr 172.21.15.12:0/3094905776 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:51.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.395+0000 7f49d7400640 1 -- 172.21.15.12:0/3094905776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058ffe810 msgr2=0x559059008a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:51.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.395+0000 7f49d7400640 1 --2- 172.21.15.12:0/3094905776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058ffe810 0x559059008a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.395+0000 7f49d7400640 1 -- 172.21.15.12:0/3094905776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559058fff9e0 msgr2=0x559059008fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:51.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.395+0000 7f49d7400640 1 --2- 172.21.15.12:0/3094905776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559058fff9e0 0x559059008fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.395+0000 7f49d7400640 1 -- 172.21.15.12:0/3094905776 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559058c773f0 con 0x5590590003a0 2024-03-23T23:10:51.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.396+0000 7f49d6a00640 1 --2- 172.21.15.12:0/3094905776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058ffe810 0x559059008a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:51.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.396+0000 7f49d7400640 1 --2- 172.21.15.12:0/3094905776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590590003a0 0x559059010010 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f49d0011dd0 tx=0x7f49d0004cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73b680efd6bdd450 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:51.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.396+0000 7f49c4c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:10:51.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.396+0000 7f49c4c00640 1 -- 172.21.15.12:0/3094905776 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d0028020 con 0x5590590003a0 2024-03-23T23:10:51.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.396+0000 7f49e082f740 1 -- 172.21.15.12:0/3094905776 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f49d0004050 con 0x5590590003a0 2024-03-23T23:10:51.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.396+0000 7f49c4c00640 1 -- 172.21.15.12:0/3094905776 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f49d0004410 con 0x5590590003a0 2024-03-23T23:10:51.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.396+0000 7f49c4c00640 1 -- 172.21.15.12:0/3094905776 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49d0004610 con 0x5590590003a0 2024-03-23T23:10:51.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.397+0000 7f49e082f740 10 client.27860 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:51.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.397+0000 7f49e082f740 1 -- 172.21.15.12:0/3094905776 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x559058d417d0 con 0x5590590003a0 2024-03-23T23:10:51.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.397+0000 7f49c4c00640 1 -- 172.21.15.12:0/3094905776 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49d0004050 con 0x5590590003a0 2024-03-23T23:10:51.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.397+0000 7f49c4c00640 1 -- 172.21.15.12:0/3094905776 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f49d000f570 con 0x5590590003a0 2024-03-23T23:10:51.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.397+0000 7f49e082f740 10 client.27860 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:51.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.397+0000 7f49e082f740 10 client.27860 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:10:51.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.397+0000 7f49e082f740 20 client.27860 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:51.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.397+0000 7f49e082f740 1 --2- 172.21.15.12:0/3094905776 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x559059014310 0x559059016700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.397+0000 7f49e082f740 4 client.27860 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:10:51.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.397+0000 7f49e082f740 1 -- 172.21.15.12:0/3094905776 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559058f70800 con 0x559059014310 2024-03-23T23:10:51.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.397+0000 7f49d6a00640 1 --2- 172.21.15.12:0/3094905776 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x559059014310 0x559059016700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.398+0000 7f49d6a00640 1 --2- 172.21.15.12:0/3094905776 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x559059014310 0x559059016700 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f3e7fefe388fbd64 server_cookie=c5fa0784ac22555d in_seq=0 out_seq=0 2024-03-23T23:10:51.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.398+0000 7f49c4c00640 10 client.27860 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:10:51.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.399+0000 7f49c4c00640 1 -- 172.21.15.12:0/3094905776 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559058f70800 con 0x559059014310 2024-03-23T23:10:51.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.399+0000 7f49c4c00640 10 client.27860 handle_command_reply: tid=0 2024-03-23T23:10:51.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.399+0000 7f49e082f740 1 client.27860 shutdown 2024-03-23T23:10:51.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.399+0000 7f49e082f740 1 -- 172.21.15.12:0/3094905776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590590003a0 msgr2=0x559059010010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.399+0000 7f49e082f740 1 --2- 172.21.15.12:0/3094905776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590590003a0 0x559059010010 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f49d0011dd0 tx=0x7f49d0004cf0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.399+0000 7f49e082f740 1 -- 172.21.15.12:0/3094905776 shutdown_connections 2024-03-23T23:10:51.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.399+0000 7f49e082f740 1 --2- 172.21.15.12:0/3094905776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590590003a0 0x559059010010 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.399+0000 7f49e082f740 1 --2- 172.21.15.12:0/3094905776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559058fff9e0 0x559059008fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.399+0000 7f49e082f740 1 --2- 172.21.15.12:0/3094905776 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x559059014310 0x559059016700 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.399+0000 7f49e082f740 1 --2- 172.21.15.12:0/3094905776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058ffe810 0x559059008a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.399+0000 7f49e082f740 1 -- 172.21.15.12:0/3094905776 >> 172.21.15.12:0/3094905776 conn(0x559058ff9c00 msgr2=0x559058ff8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:51.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.399+0000 7f49c4c00640 0 client.27860 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:10:51.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.400+0000 7f49e082f740 1 -- 172.21.15.12:0/3094905776 shutdown_connections 2024-03-23T23:10:51.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.400+0000 7f49e082f740 1 -- 172.21.15.12:0/3094905776 wait complete. 2024-03-23T23:10:51.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.400+0000 7f49e082f740 20 client.27860 trim_cache size 0 max 16384 2024-03-23T23:10:51.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.413+0000 7f49e082f740 1 Processor -- start 2024-03-23T23:10:51.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.413+0000 7f49e082f740 1 -- start start 2024-03-23T23:10:51.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.414+0000 7f49e082f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058ffe810 0x559059002a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.414+0000 7f49e082f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559058fff9e0 0x55905900ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.414+0000 7f49e082f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590590003a0 0x559059002fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.414+0000 7f49e082f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d81ad070 con 0x559058fff9e0 2024-03-23T23:10:51.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.414+0000 7f49e082f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d81aab70 con 0x5590590003a0 2024-03-23T23:10:51.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.414+0000 7f49e082f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d81a8c10 con 0x559058ffe810 2024-03-23T23:10:51.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.414+0000 7f49d7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590590003a0 0x559059002fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.414+0000 7f49d7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590590003a0 0x559059002fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37742/0 (socket says 172.21.15.12:37742) 2024-03-23T23:10:51.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.414+0000 7f49d7400640 1 -- 172.21.15.12:0/3133768829 learned_addr learned my addr 172.21.15.12:0/3133768829 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:51.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.414+0000 7f49d6000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559058fff9e0 0x55905900ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.414+0000 7f49d6a00640 1 --2- 172.21.15.12:0/3133768829 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058ffe810 0x559059002a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.414+0000 7f49d7400640 1 -- 172.21.15.12:0/3133768829 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058ffe810 msgr2=0x559059002a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:51.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.414+0000 7f49d7400640 1 --2- 172.21.15.12:0/3133768829 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058ffe810 0x559059002a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.414+0000 7f49d7400640 1 -- 172.21.15.12:0/3133768829 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559058fff9e0 msgr2=0x55905900ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:51.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.414+0000 7f49d7400640 1 --2- 172.21.15.12:0/3133768829 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559058fff9e0 0x55905900ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.414+0000 7f49d7400640 1 -- 172.21.15.12:0/3133768829 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559058ff54e0 con 0x5590590003a0 2024-03-23T23:10:51.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.415+0000 7f49d6a00640 1 --2- 172.21.15.12:0/3133768829 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058ffe810 0x559059002a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:10:51.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.415+0000 7f49d7400640 1 --2- 172.21.15.12:0/3133768829 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590590003a0 0x559059002fc0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f49d0009720 tx=0x7f49d0005f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75685cfb57c153e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:51.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.415+0000 7f49c7400640 1 -- 172.21.15.12:0/3133768829 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d0028040 con 0x5590590003a0 2024-03-23T23:10:51.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.415+0000 7f49c7400640 1 -- 172.21.15.12:0/3133768829 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49d0004000 con 0x5590590003a0 2024-03-23T23:10:51.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.415+0000 7f49c7400640 1 -- 172.21.15.12:0/3133768829 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d0019910 con 0x5590590003a0 2024-03-23T23:10:51.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.416+0000 7f49e082f740 1 -- 172.21.15.12:0/3133768829 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590590003a0 msgr2=0x559059002fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.416+0000 7f49e082f740 1 --2- 172.21.15.12:0/3133768829 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590590003a0 0x559059002fc0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f49d0009720 tx=0x7f49d0005f30 comp rx=0 tx=0).stop 2024-03-23T23:10:51.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.416+0000 7f49e082f740 1 -- 172.21.15.12:0/3133768829 shutdown_connections 2024-03-23T23:10:51.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.416+0000 7f49e082f740 1 --2- 172.21.15.12:0/3133768829 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590590003a0 0x559059002fc0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.416+0000 7f49e082f740 1 --2- 172.21.15.12:0/3133768829 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559058fff9e0 0x55905900ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.416+0000 7f49e082f740 1 --2- 172.21.15.12:0/3133768829 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058ffe810 0x559059002a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.416+0000 7f49e082f740 1 -- 172.21.15.12:0/3133768829 >> 172.21.15.12:0/3133768829 conn(0x559058ff9c00 msgr2=0x559058ffd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:51.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.416+0000 7f49e082f740 1 -- 172.21.15.12:0/3133768829 shutdown_connections 2024-03-23T23:10:51.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.416+0000 7f49e082f740 1 -- 172.21.15.12:0/3133768829 wait complete. 2024-03-23T23:10:51.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.416+0000 7f49e082f740 1 -- start start 2024-03-23T23:10:51.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.416+0000 7f49e082f740 1 Processor -- start 2024-03-23T23:10:51.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.417+0000 7f49e082f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559058ffe810 0x55905900b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.417+0000 7f49e082f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058fff9e0 0x55905900bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.417+0000 7f49e082f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590590003a0 0x55905900eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.417+0000 7f49e082f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d81a9210 con 0x559058ffe810 2024-03-23T23:10:51.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.417+0000 7f49e082f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d81aa510 con 0x5590590003a0 2024-03-23T23:10:51.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.417+0000 7f49e082f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d81a9850 con 0x559058fff9e0 2024-03-23T23:10:51.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.417+0000 7f49e082f740 10 client.0 _ll_register_callbacks cb 0x7f49d0009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:51.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.417+0000 7f49d7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590590003a0 0x55905900eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.417+0000 7f49d6000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058fff9e0 0x55905900bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.417+0000 7f49d6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559058ffe810 0x55905900b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.417+0000 7f49d7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590590003a0 0x55905900eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37756/0 (socket says 172.21.15.12:37756) 2024-03-23T23:10:51.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.417+0000 7f49d7400640 1 -- 172.21.15.12:0/746752710 learned_addr learned my addr 172.21.15.12:0/746752710 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:51.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.417+0000 7f49d6000640 1 -- 172.21.15.12:0/746752710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590590003a0 msgr2=0x55905900eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:51.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.417+0000 7f49d6000640 1 --2- 172.21.15.12:0/746752710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590590003a0 0x55905900eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.417+0000 7f49d6000640 1 -- 172.21.15.12:0/746752710 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559058ffe810 msgr2=0x55905900b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:51.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.417+0000 7f49d6000640 1 --2- 172.21.15.12:0/746752710 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559058ffe810 0x55905900b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.417+0000 7f49d6000640 1 -- 172.21.15.12:0/746752710 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55905900f290 con 0x559058fff9e0 2024-03-23T23:10:51.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.417+0000 7f49d7400640 1 --2- 172.21.15.12:0/746752710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590590003a0 0x55905900eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:51.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.417+0000 7f49d6a00640 1 --2- 172.21.15.12:0/746752710 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559058ffe810 0x55905900b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:51.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.418+0000 7f49d6000640 1 --2- 172.21.15.12:0/746752710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058fff9e0 0x55905900bcf0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f49c00bd250 tx=0x7f49c0000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=684ecdb67a0d8d7e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:51.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.418+0000 7f49c4c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:10:51.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.418+0000 7f49c4c00640 1 -- 172.21.15.12:0/746752710 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c00bcc20 con 0x559058fff9e0 2024-03-23T23:10:51.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.418+0000 7f49c4c00640 1 -- 172.21.15.12:0/746752710 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f49c00bb5c0 con 0x559058fff9e0 2024-03-23T23:10:51.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.418+0000 7f49e082f740 1 -- 172.21.15.12:0/746752710 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f49d0004000 con 0x559058fff9e0 2024-03-23T23:10:51.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.418+0000 7f49c4c00640 1 -- 172.21.15.12:0/746752710 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49c0005690 con 0x559058fff9e0 2024-03-23T23:10:51.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.418+0000 7f49e082f740 10 client.27826 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:51.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.418+0000 7f49e082f740 1 -- 172.21.15.12:0/746752710 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x559059078430 con 0x559058fff9e0 2024-03-23T23:10:51.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.418+0000 7f49c4c00640 1 -- 172.21.15.12:0/746752710 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49d0004000 con 0x559058fff9e0 2024-03-23T23:10:51.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.419+0000 7f49c4c00640 1 -- 172.21.15.12:0/746752710 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f49c00dd0b0 con 0x559058fff9e0 2024-03-23T23:10:51.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.419+0000 7f49e082f740 10 client.27826 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:51.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.419+0000 7f49e082f740 10 client.27826 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:10:51.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.419+0000 7f49e082f740 20 client.27826 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:51.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.419+0000 7f49e082f740 1 --2- 172.21.15.12:0/746752710 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x559059078d70 0x55905907b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.419+0000 7f49e082f740 4 client.27826 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:10:51.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.419+0000 7f49e082f740 1 -- 172.21.15.12:0/746752710 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559058f6ab50 con 0x559059078d70 2024-03-23T23:10:51.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.419+0000 7f49d6a00640 1 --2- 172.21.15.12:0/746752710 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x559059078d70 0x55905907b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.419+0000 7f49d6a00640 1 --2- 172.21.15.12:0/746752710 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x559059078d70 0x55905907b190 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=844e796466cf9629 server_cookie=16e2e203373d78a4 in_seq=0 out_seq=0 2024-03-23T23:10:51.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.419+0000 7f49c4c00640 10 client.27826 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:10:51.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.420+0000 7f49c4c00640 1 -- 172.21.15.12:0/746752710 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559058f6ab50 con 0x559059078d70 2024-03-23T23:10:51.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.420+0000 7f49c4c00640 10 client.27826 handle_command_reply: tid=0 2024-03-23T23:10:51.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.420+0000 7f49e082f740 1 client.27826 shutdown 2024-03-23T23:10:51.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.420+0000 7f49e082f740 1 -- 172.21.15.12:0/746752710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058fff9e0 msgr2=0x55905900bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.420+0000 7f49e082f740 1 --2- 172.21.15.12:0/746752710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058fff9e0 0x55905900bcf0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f49c00bd250 tx=0x7f49c0000bc0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.420+0000 7f49e082f740 1 -- 172.21.15.12:0/746752710 shutdown_connections 2024-03-23T23:10:51.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.420+0000 7f49e082f740 1 --2- 172.21.15.12:0/746752710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590590003a0 0x55905900eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.420+0000 7f49e082f740 1 --2- 172.21.15.12:0/746752710 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x559059078d70 0x55905907b190 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.420+0000 7f49e082f740 1 --2- 172.21.15.12:0/746752710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559058fff9e0 0x55905900bcf0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.420+0000 7f49e082f740 1 --2- 172.21.15.12:0/746752710 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559058ffe810 0x55905900b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.420+0000 7f49e082f740 1 -- 172.21.15.12:0/746752710 >> 172.21.15.12:0/746752710 conn(0x559058ff9c00 msgr2=0x559058ff8620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:51.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.420+0000 7f49c4c00640 0 client.27826 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:10:51.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.420+0000 7f49e082f740 1 -- 172.21.15.12:0/746752710 shutdown_connections 2024-03-23T23:10:51.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.420+0000 7f49e082f740 1 -- 172.21.15.12:0/746752710 wait complete. 2024-03-23T23:10:51.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.420+0000 7f49e082f740 20 client.27826 trim_cache size 0 max 16384 2024-03-23T23:10:51.425 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:10:51.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.421+0000 7f49dda00640 1 -- 172.21.15.12:0/1540961619 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f49bc0a2880 msgr2=0x7f49bc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.421+0000 7f49dda00640 1 --2- 172.21.15.12:0/1540961619 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f49bc0a2880 0x7f49bc0a4d40 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f49c8009eb0 tx=0x7f49c8005f90 comp rx=0 tx=0).stop 2024-03-23T23:10:51.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.421+0000 7f49dda00640 1 -- 172.21.15.12:0/1540961619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49d8100690 msgr2=0x7f49d819a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.421+0000 7f49dda00640 1 --2- 172.21.15.12:0/1540961619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49d8100690 0x7f49d819a120 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f49c0036040 tx=0x7f49c002c650 comp rx=0 tx=0).stop 2024-03-23T23:10:51.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.421+0000 7f49dda00640 1 -- 172.21.15.12:0/1540961619 shutdown_connections 2024-03-23T23:10:51.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.421+0000 7f49dda00640 1 --2- 172.21.15.12:0/1540961619 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f49bc0a2880 0x7f49bc0a4d40 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.421+0000 7f49dda00640 1 --2- 172.21.15.12:0/1540961619 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49d8101030 0x7f49d81a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.421+0000 7f49dda00640 1 --2- 172.21.15.12:0/1540961619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49d8100690 0x7f49d819a120 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.421+0000 7f49dda00640 1 --2- 172.21.15.12:0/1540961619 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49d80ff620 0x7f49d8199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.421+0000 7f49dda00640 1 -- 172.21.15.12:0/1540961619 >> 172.21.15.12:0/1540961619 conn(0x7f49d80fae60 msgr2=0x7f49d80fc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:51.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.421+0000 7f49dda00640 1 -- 172.21.15.12:0/1540961619 shutdown_connections 2024-03-23T23:10:51.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.421+0000 7f49dda00640 1 -- 172.21.15.12:0/1540961619 wait complete. 2024-03-23T23:10:51.439 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:10:51.440 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-23T23:10:51.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.667+0000 7f936b000640 1 -- 172.21.15.12:0/3442824769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9364104930 msgr2=0x7f9364106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.667+0000 7f936b000640 1 --2- 172.21.15.12:0/3442824769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9364104930 0x7f9364106d20 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f9350009900 tx=0x7f935002bc30 comp rx=0 tx=0).stop 2024-03-23T23:10:51.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.667+0000 7f936b000640 1 -- 172.21.15.12:0/3442824769 shutdown_connections 2024-03-23T23:10:51.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.667+0000 7f936b000640 1 --2- 172.21.15.12:0/3442824769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9364107260 0x7f93641096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.667+0000 7f936b000640 1 --2- 172.21.15.12:0/3442824769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9364104930 0x7f9364106d20 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.667+0000 7f936b000640 1 --2- 172.21.15.12:0/3442824769 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9364102000 0x7f93641043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.667+0000 7f936b000640 1 -- 172.21.15.12:0/3442824769 >> 172.21.15.12:0/3442824769 conn(0x7f93640fba10 msgr2=0x7f93640fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:51.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.667+0000 7f936b000640 1 -- 172.21.15.12:0/3442824769 shutdown_connections 2024-03-23T23:10:51.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.667+0000 7f936b000640 1 -- 172.21.15.12:0/3442824769 wait complete. 2024-03-23T23:10:51.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.668+0000 7f936b000640 1 Processor -- start 2024-03-23T23:10:51.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.669+0000 7f936b000640 1 -- start start 2024-03-23T23:10:51.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.669+0000 7f936b000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9364102000 0x7f9364101970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.669+0000 7f936b000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9364104930 0x7f93640fffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.669+0000 7f936b000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9364107260 0x7f93641004f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.669+0000 7f936b000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936410d260 con 0x7f9364102000 2024-03-23T23:10:51.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.669+0000 7f936b000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936410d0e0 con 0x7f9364107260 2024-03-23T23:10:51.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.669+0000 7f936b000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936410d3e0 con 0x7f9364104930 2024-03-23T23:10:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.669+0000 7f9369c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9364102000 0x7f9364101970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.669+0000 7f936a600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9364107260 0x7f93641004f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.669+0000 7f9369200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9364104930 0x7f93640fffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.670+0000 7f936a600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9364107260 0x7f93641004f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37776/0 (socket says 172.21.15.12:37776) 2024-03-23T23:10:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.670+0000 7f9369200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9364104930 0x7f93640fffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54284/0 (socket says 172.21.15.12:54284) 2024-03-23T23:10:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.670+0000 7f936a600640 1 -- 172.21.15.12:0/1662199549 learned_addr learned my addr 172.21.15.12:0/1662199549 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.670+0000 7f936a600640 1 -- 172.21.15.12:0/1662199549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9364104930 msgr2=0x7f93640fffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.670+0000 7f936a600640 1 --2- 172.21.15.12:0/1662199549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9364104930 0x7f93640fffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.670+0000 7f936a600640 1 -- 172.21.15.12:0/1662199549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9364102000 msgr2=0x7f9364101970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.670+0000 7f936a600640 1 --2- 172.21.15.12:0/1662199549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9364102000 0x7f9364101970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.670+0000 7f936a600640 1 -- 172.21.15.12:0/1662199549 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9364100d80 con 0x7f9364107260 2024-03-23T23:10:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.670+0000 7f9369200640 1 --2- 172.21.15.12:0/1662199549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9364104930 0x7f93640fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.671+0000 7f936a600640 1 --2- 172.21.15.12:0/1662199549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9364107260 0x7f93641004f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f935c00eaa0 tx=0x7f935c00ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:51.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.671+0000 7f9362000640 1 -- 172.21.15.12:0/1662199549 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f935c00cdb0 con 0x7f9364107260 2024-03-23T23:10:51.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.671+0000 7f9362000640 1 -- 172.21.15.12:0/1662199549 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f935c005020 con 0x7f9364107260 2024-03-23T23:10:51.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.671+0000 7f9362000640 1 -- 172.21.15.12:0/1662199549 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f935c0116b0 con 0x7f9364107260 2024-03-23T23:10:51.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.671+0000 7f936b000640 1 -- 172.21.15.12:0/1662199549 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93641a19e0 con 0x7f9364107260 2024-03-23T23:10:51.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.671+0000 7f936b000640 1 -- 172.21.15.12:0/1662199549 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93641a1eb0 con 0x7f9364107260 2024-03-23T23:10:51.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.675+0000 7f936f41e740 1 Processor -- start 2024-03-23T23:10:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.675+0000 7f9362000640 1 -- 172.21.15.12:0/1662199549 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f935c005560 con 0x7f9364107260 2024-03-23T23:10:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.676+0000 7f936f41e740 1 -- start start 2024-03-23T23:10:51.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.676+0000 7f936f41e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd0a7a79f0 0x55cd0a7a7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.676+0000 7f936f41e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd0a7a8bc0 0x55cd0a7a9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.676+0000 7f936f41e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd0a7a9580 0x55cd0a7afe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.676+0000 7f936f41e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd0a70c2c0 con 0x55cd0a7a79f0 2024-03-23T23:10:51.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.676+0000 7f936f41e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd0a6b4c90 con 0x55cd0a7a9580 2024-03-23T23:10:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.676+0000 7f936f41e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd0a708360 con 0x55cd0a7a8bc0 2024-03-23T23:10:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.676+0000 7f9369c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd0a7a8bc0 0x55cd0a7a9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.676+0000 7f9369c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd0a7a8bc0 0x55cd0a7a9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54292/0 (socket says 172.21.15.12:54292) 2024-03-23T23:10:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.676+0000 7f9369200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd0a7a79f0 0x55cd0a7a7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.676+0000 7f9369c00640 1 -- 172.21.15.12:0/765417566 learned_addr learned my addr 172.21.15.12:0/765417566 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.677+0000 7f9369200640 1 --2- 172.21.15.12:0/765417566 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd0a7a9580 0x55cd0a7afe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.677+0000 7f9369c00640 1 -- 172.21.15.12:0/765417566 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd0a7a9580 msgr2=0x55cd0a7afe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.677+0000 7f9369c00640 1 --2- 172.21.15.12:0/765417566 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd0a7a9580 0x55cd0a7afe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.677+0000 7f9369c00640 1 -- 172.21.15.12:0/765417566 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd0a7a79f0 msgr2=0x55cd0a7a7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.677+0000 7f9369c00640 1 --2- 172.21.15.12:0/765417566 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd0a7a79f0 0x55cd0a7a7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.677+0000 7f9369c00640 1 -- 172.21.15.12:0/765417566 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd0a6d9080 con 0x55cd0a7a8bc0 2024-03-23T23:10:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.677+0000 7f9369200640 1 --2- 172.21.15.12:0/765417566 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd0a7a9580 0x55cd0a7afe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:10:51.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.677+0000 7f9369c00640 1 --2- 172.21.15.12:0/765417566 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd0a7a8bc0 0x55cd0a7a9040 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f9358009cf0 tx=0x7f9358011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=faa6489e8b3dec8e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:51.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.677+0000 7f9362000640 1 --2- 172.21.15.12:0/1662199549 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f934c0a2800 0x7f934c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.677+0000 7f9356a00640 1 -- 172.21.15.12:0/765417566 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9358004230 con 0x55cd0a7a8bc0 2024-03-23T23:10:51.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.677+0000 7f9362000640 1 -- 172.21.15.12:0/1662199549 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f935c0cec70 con 0x7f9364107260 2024-03-23T23:10:51.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.677+0000 7f9356a00640 1 -- 172.21.15.12:0/765417566 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93580043d0 con 0x55cd0a7a8bc0 2024-03-23T23:10:51.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.677+0000 7f9356a00640 1 -- 172.21.15.12:0/765417566 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9358019960 con 0x55cd0a7a8bc0 2024-03-23T23:10:51.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.677+0000 7f9369200640 1 --2- 172.21.15.12:0/1662199549 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f934c0a2800 0x7f934c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.678+0000 7f936f41e740 1 -- 172.21.15.12:0/765417566 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd0a7a8bc0 msgr2=0x55cd0a7a9040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.678+0000 7f936f41e740 1 --2- 172.21.15.12:0/765417566 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd0a7a8bc0 0x55cd0a7a9040 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f9358009cf0 tx=0x7f9358011810 comp rx=0 tx=0).stop 2024-03-23T23:10:51.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.678+0000 7f9369200640 1 --2- 172.21.15.12:0/1662199549 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f934c0a2800 0x7f934c0a4cc0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f935002c170 tx=0x7f935002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:51.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.678+0000 7f936f41e740 1 -- 172.21.15.12:0/765417566 shutdown_connections 2024-03-23T23:10:51.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.678+0000 7f936f41e740 1 --2- 172.21.15.12:0/765417566 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd0a7a9580 0x55cd0a7afe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.678+0000 7f936f41e740 1 --2- 172.21.15.12:0/765417566 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd0a7a8bc0 0x55cd0a7a9040 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.678+0000 7f936f41e740 1 --2- 172.21.15.12:0/765417566 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd0a7a79f0 0x55cd0a7a7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.678+0000 7f936f41e740 1 -- 172.21.15.12:0/765417566 >> 172.21.15.12:0/765417566 conn(0x55cd0a7a2de0 msgr2=0x55cd0a7a31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:51.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.678+0000 7f936f41e740 1 -- 172.21.15.12:0/765417566 shutdown_connections 2024-03-23T23:10:51.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.679+0000 7f936f41e740 1 -- 172.21.15.12:0/765417566 wait complete. 2024-03-23T23:10:51.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.679+0000 7f936f41e740 1 -- start start 2024-03-23T23:10:51.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.679+0000 7f936f41e740 1 Processor -- start 2024-03-23T23:10:51.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.680+0000 7f936f41e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd0a7a79f0 0x55cd0a7b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.680+0000 7f936f41e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd0a7a8bc0 0x55cd0a7b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.680+0000 7f936f41e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd0a7a9580 0x55cd0a7b91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.680+0000 7f936f41e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd0a7b0590 con 0x55cd0a7a8bc0 2024-03-23T23:10:51.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.680+0000 7f936f41e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd0a6c80d0 con 0x55cd0a7a79f0 2024-03-23T23:10:51.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.680+0000 7f936f41e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd0a7b0710 con 0x55cd0a7a9580 2024-03-23T23:10:51.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.680+0000 7f936f41e740 10 client.0 _ll_register_callbacks cb 0x55cd0a6cce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:51.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.680+0000 7f936a600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd0a7a9580 0x55cd0a7b91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.680+0000 7f9369c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd0a7a79f0 0x55cd0a7b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.680+0000 7f9369200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd0a7a8bc0 0x55cd0a7b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.680+0000 7f936a600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd0a7a9580 0x55cd0a7b91f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54298/0 (socket says 172.21.15.12:54298) 2024-03-23T23:10:51.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.680+0000 7f936a600640 1 -- 172.21.15.12:0/3754443351 learned_addr learned my addr 172.21.15.12:0/3754443351 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.680+0000 7f936a600640 1 -- 172.21.15.12:0/3754443351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd0a7a79f0 msgr2=0x55cd0a7b1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.680+0000 7f936a600640 1 --2- 172.21.15.12:0/3754443351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd0a7a79f0 0x55cd0a7b1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.680+0000 7f936a600640 1 -- 172.21.15.12:0/3754443351 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd0a7a8bc0 msgr2=0x55cd0a7b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.680+0000 7f936a600640 1 --2- 172.21.15.12:0/3754443351 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd0a7a8bc0 0x55cd0a7b21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.680+0000 7f936a600640 1 -- 172.21.15.12:0/3754443351 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cd0a4203f0 con 0x55cd0a7a9580 2024-03-23T23:10:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.680+0000 7f936a600640 1 --2- 172.21.15.12:0/3754443351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd0a7a9580 0x55cd0a7b91f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f935c09b3d0 tx=0x7f935c09f8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=946bac81806885b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.680+0000 7f934b400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:10:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.681+0000 7f934b400640 1 -- 172.21.15.12:0/3754443351 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f935c0b4060 con 0x55cd0a7a9580 2024-03-23T23:10:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.681+0000 7f934b400640 1 -- 172.21.15.12:0/3754443351 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f935c0b4200 con 0x55cd0a7a9580 2024-03-23T23:10:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.681+0000 7f936f41e740 1 -- 172.21.15.12:0/3754443351 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f93580043d0 con 0x55cd0a7a9580 2024-03-23T23:10:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.681+0000 7f934b400640 1 -- 172.21.15.12:0/3754443351 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f935c01f040 con 0x55cd0a7a9580 2024-03-23T23:10:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.681+0000 7f936f41e740 10 client.27838 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.681+0000 7f936f41e740 1 -- 172.21.15.12:0/3754443351 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55cd0a4ea7d0 con 0x55cd0a7a9580 2024-03-23T23:10:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.681+0000 7f934b400640 1 -- 172.21.15.12:0/3754443351 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93580043d0 con 0x55cd0a7a9580 2024-03-23T23:10:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.681+0000 7f934b400640 1 -- 172.21.15.12:0/3754443351 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f935c00f040 con 0x55cd0a7a9580 2024-03-23T23:10:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.681+0000 7f936f41e740 10 client.27838 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:51.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.681+0000 7f936f41e740 10 client.27838 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:51.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.681+0000 7f936f41e740 20 client.27838 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:51.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.682+0000 7f936f41e740 1 --2- 172.21.15.12:0/3754443351 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cd0a7bd630 0x55cd0a7bfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.682+0000 7f936f41e740 4 client.27838 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:10:51.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.682+0000 7f936f41e740 1 -- 172.21.15.12:0/3754443351 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cd0a719800 con 0x55cd0a7bd630 2024-03-23T23:10:51.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.682+0000 7f9369c00640 1 --2- 172.21.15.12:0/3754443351 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cd0a7bd630 0x55cd0a7bfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.682+0000 7f9369c00640 1 --2- 172.21.15.12:0/3754443351 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cd0a7bd630 0x55cd0a7bfad0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=666109613fdd593 server_cookie=3ef1003984c882a1 in_seq=0 out_seq=0 2024-03-23T23:10:51.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.682+0000 7f934b400640 10 client.27838 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:51.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.683+0000 7f934b400640 1 -- 172.21.15.12:0/3754443351 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cd0a719800 con 0x55cd0a7bd630 2024-03-23T23:10:51.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.683+0000 7f934b400640 10 client.27838 handle_command_reply: tid=0 2024-03-23T23:10:51.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.683+0000 7f936f41e740 1 client.27838 shutdown 2024-03-23T23:10:51.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.684+0000 7f936f41e740 1 -- 172.21.15.12:0/3754443351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd0a7a9580 msgr2=0x55cd0a7b91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.684+0000 7f936f41e740 1 --2- 172.21.15.12:0/3754443351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd0a7a9580 0x55cd0a7b91f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f935c09b3d0 tx=0x7f935c09f8b0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.684+0000 7f936f41e740 1 -- 172.21.15.12:0/3754443351 shutdown_connections 2024-03-23T23:10:51.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.684+0000 7f936f41e740 1 --2- 172.21.15.12:0/3754443351 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cd0a7bd630 0x55cd0a7bfad0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.684+0000 7f936f41e740 1 --2- 172.21.15.12:0/3754443351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd0a7a9580 0x55cd0a7b91f0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.684+0000 7f936f41e740 1 --2- 172.21.15.12:0/3754443351 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd0a7a8bc0 0x55cd0a7b21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.684+0000 7f936f41e740 1 --2- 172.21.15.12:0/3754443351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd0a7a79f0 0x55cd0a7b1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.684+0000 7f936f41e740 1 -- 172.21.15.12:0/3754443351 >> 172.21.15.12:0/3754443351 conn(0x55cd0a7a2de0 msgr2=0x55cd0a7a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:51.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.684+0000 7f934b400640 0 client.27838 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:51.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.684+0000 7f936f41e740 1 -- 172.21.15.12:0/3754443351 shutdown_connections 2024-03-23T23:10:51.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.684+0000 7f936f41e740 1 -- 172.21.15.12:0/3754443351 wait complete. 2024-03-23T23:10:51.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.684+0000 7f936f41e740 20 client.27838 trim_cache size 0 max 16384 2024-03-23T23:10:51.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.698+0000 7f936f41e740 1 Processor -- start 2024-03-23T23:10:51.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.698+0000 7f936f41e740 1 -- start start 2024-03-23T23:10:51.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.698+0000 7f936f41e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd0a7a79f0 0x55cd0a7a1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.698+0000 7f936f41e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd0a7a8bc0 0x55cd0a7b91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.698+0000 7f936f41e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd0a7a9580 0x55cd0a7c4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.698+0000 7f936f41e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93641acfe0 con 0x55cd0a7a9580 2024-03-23T23:10:51.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.698+0000 7f936f41e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93641aaae0 con 0x55cd0a7a79f0 2024-03-23T23:10:51.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.698+0000 7f936f41e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93641a8b80 con 0x55cd0a7a8bc0 2024-03-23T23:10:51.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.698+0000 7f9369c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd0a7a79f0 0x55cd0a7a1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.698+0000 7f9369c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd0a7a79f0 0x55cd0a7a1fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37802/0 (socket says 172.21.15.12:37802) 2024-03-23T23:10:51.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.698+0000 7f9369c00640 1 -- 172.21.15.12:0/1301628435 learned_addr learned my addr 172.21.15.12:0/1301628435 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:51.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.698+0000 7f9369200640 1 --2- 172.21.15.12:0/1301628435 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd0a7a8bc0 0x55cd0a7b91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.699+0000 7f936a600640 1 --2- 172.21.15.12:0/1301628435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd0a7a9580 0x55cd0a7c4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.699+0000 7f9369c00640 1 -- 172.21.15.12:0/1301628435 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd0a7a8bc0 msgr2=0x55cd0a7b91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:51.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.699+0000 7f9369c00640 1 --2- 172.21.15.12:0/1301628435 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd0a7a8bc0 0x55cd0a7b91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.699+0000 7f9369c00640 1 -- 172.21.15.12:0/1301628435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd0a7a9580 msgr2=0x55cd0a7c4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:51.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.699+0000 7f9369c00640 1 --2- 172.21.15.12:0/1301628435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd0a7a9580 0x55cd0a7c4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.699+0000 7f9369c00640 1 -- 172.21.15.12:0/1301628435 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd0a7b2520 con 0x55cd0a7a79f0 2024-03-23T23:10:51.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.699+0000 7f936a600640 1 --2- 172.21.15.12:0/1301628435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd0a7a9580 0x55cd0a7c4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:10:51.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.699+0000 7f9369c00640 1 --2- 172.21.15.12:0/1301628435 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd0a7a79f0 0x55cd0a7a1fe0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f9358009c40 tx=0x7f9358005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84cac9e927f2ec1b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:51.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.699+0000 7f9356000640 1 -- 172.21.15.12:0/1301628435 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9358004180 con 0x55cd0a7a79f0 2024-03-23T23:10:51.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.700+0000 7f9356000640 1 -- 172.21.15.12:0/1301628435 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9358031070 con 0x55cd0a7a79f0 2024-03-23T23:10:51.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.700+0000 7f9356000640 1 -- 172.21.15.12:0/1301628435 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9358003850 con 0x55cd0a7a79f0 2024-03-23T23:10:51.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.700+0000 7f936f41e740 1 -- 172.21.15.12:0/1301628435 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd0a7a79f0 msgr2=0x55cd0a7a1fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.700+0000 7f936f41e740 1 --2- 172.21.15.12:0/1301628435 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd0a7a79f0 0x55cd0a7a1fe0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f9358009c40 tx=0x7f9358005f90 comp rx=0 tx=0).stop 2024-03-23T23:10:51.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.700+0000 7f936f41e740 1 -- 172.21.15.12:0/1301628435 shutdown_connections 2024-03-23T23:10:51.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.700+0000 7f936f41e740 1 --2- 172.21.15.12:0/1301628435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd0a7a9580 0x55cd0a7c4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.700+0000 7f936f41e740 1 --2- 172.21.15.12:0/1301628435 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd0a7a8bc0 0x55cd0a7b91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.700+0000 7f936f41e740 1 --2- 172.21.15.12:0/1301628435 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd0a7a79f0 0x55cd0a7a1fe0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.700+0000 7f936f41e740 1 -- 172.21.15.12:0/1301628435 >> 172.21.15.12:0/1301628435 conn(0x55cd0a7a2de0 msgr2=0x55cd0a7a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:51.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.700+0000 7f936f41e740 1 -- 172.21.15.12:0/1301628435 shutdown_connections 2024-03-23T23:10:51.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.700+0000 7f936f41e740 1 -- 172.21.15.12:0/1301628435 wait complete. 2024-03-23T23:10:51.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.701+0000 7f936f41e740 1 -- start start 2024-03-23T23:10:51.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.701+0000 7f936f41e740 1 Processor -- start 2024-03-23T23:10:51.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.701+0000 7f936f41e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd0a7a79f0 0x55cd0a7b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.701+0000 7f936f41e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd0a7a8bc0 0x55cd0a7b57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.701+0000 7f936f41e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd0a7a9580 0x55cd0a7b87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.701+0000 7f936f41e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93641a9180 con 0x55cd0a7a9580 2024-03-23T23:10:51.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.701+0000 7f936f41e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93641aa480 con 0x55cd0a7a8bc0 2024-03-23T23:10:51.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.701+0000 7f936f41e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93641a97c0 con 0x55cd0a7a79f0 2024-03-23T23:10:51.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.701+0000 7f936f41e740 10 client.0 _ll_register_callbacks cb 0x7f935c097970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:51.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.702+0000 7f9369200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd0a7a8bc0 0x55cd0a7b57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.702+0000 7f9369200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd0a7a8bc0 0x55cd0a7b57b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37812/0 (socket says 172.21.15.12:37812) 2024-03-23T23:10:51.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.702+0000 7f9369200640 1 -- 172.21.15.12:0/1551996342 learned_addr learned my addr 172.21.15.12:0/1551996342 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:51.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.702+0000 7f9369c00640 1 --2- 172.21.15.12:0/1551996342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd0a7a79f0 0x55cd0a7b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.702+0000 7f936a600640 1 --2- 172.21.15.12:0/1551996342 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd0a7a9580 0x55cd0a7b87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.702+0000 7f9369c00640 1 -- 172.21.15.12:0/1551996342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd0a7a8bc0 msgr2=0x55cd0a7b57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:51.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.702+0000 7f9369c00640 1 --2- 172.21.15.12:0/1551996342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd0a7a8bc0 0x55cd0a7b57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.702+0000 7f9369c00640 1 -- 172.21.15.12:0/1551996342 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd0a7a9580 msgr2=0x55cd0a7b87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:51.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.702+0000 7f9369c00640 1 --2- 172.21.15.12:0/1551996342 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd0a7a9580 0x55cd0a7b87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.702+0000 7f9369c00640 1 -- 172.21.15.12:0/1551996342 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cd0a7b8dc0 con 0x55cd0a7a79f0 2024-03-23T23:10:51.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.702+0000 7f9369200640 1 --2- 172.21.15.12:0/1551996342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd0a7a8bc0 0x55cd0a7b57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:51.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.702+0000 7f936a600640 1 --2- 172.21.15.12:0/1551996342 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd0a7a9580 0x55cd0a7b87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:51.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.702+0000 7f9369c00640 1 --2- 172.21.15.12:0/1551996342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd0a7a79f0 0x55cd0a7b5270 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9358009cf0 tx=0x7f9358004030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4ff9e64fff42ea2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:51.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.702+0000 7f934b400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:10:51.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.703+0000 7f934b400640 1 -- 172.21.15.12:0/1551996342 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9358005240 con 0x55cd0a7a79f0 2024-03-23T23:10:51.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.703+0000 7f934b400640 1 -- 172.21.15.12:0/1551996342 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f93580053e0 con 0x55cd0a7a79f0 2024-03-23T23:10:51.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.703+0000 7f936f41e740 1 -- 172.21.15.12:0/1551996342 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9358031070 con 0x55cd0a7a79f0 2024-03-23T23:10:51.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.703+0000 7f934b400640 1 -- 172.21.15.12:0/1551996342 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93580055e0 con 0x55cd0a7a79f0 2024-03-23T23:10:51.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.703+0000 7f936f41e740 10 client.27844 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:51.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.703+0000 7f936f41e740 1 -- 172.21.15.12:0/1551996342 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55cd0a819940 con 0x55cd0a7a79f0 2024-03-23T23:10:51.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.704+0000 7f934b400640 1 -- 172.21.15.12:0/1551996342 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9358031070 con 0x55cd0a7a79f0 2024-03-23T23:10:51.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.704+0000 7f934b400640 1 -- 172.21.15.12:0/1551996342 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f9358004c50 con 0x55cd0a7a79f0 2024-03-23T23:10:51.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.704+0000 7f936f41e740 10 client.27844 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:51.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.704+0000 7f936f41e740 10 client.27844 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:51.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.704+0000 7f936f41e740 20 client.27844 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:51.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.704+0000 7f936f41e740 1 --2- 172.21.15.12:0/1551996342 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cd0a81a070 0x55cd0a81c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:51.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.704+0000 7f936f41e740 4 client.27844 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:10:51.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.704+0000 7f936f41e740 1 -- 172.21.15.12:0/1551996342 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55cd0a713b50 con 0x55cd0a81a070 2024-03-23T23:10:51.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.704+0000 7f9369200640 1 --2- 172.21.15.12:0/1551996342 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cd0a81a070 0x55cd0a81c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:51.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.705+0000 7f9369200640 1 --2- 172.21.15.12:0/1551996342 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cd0a81a070 0x55cd0a81c490 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f65bf042c2898b35 server_cookie=2ce7bc0790722d in_seq=0 out_seq=0 2024-03-23T23:10:51.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.705+0000 7f934b400640 10 client.27844 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:51.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.706+0000 7f934b400640 1 -- 172.21.15.12:0/1551996342 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55cd0a713b50 con 0x55cd0a81a070 2024-03-23T23:10:51.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.706+0000 7f934b400640 10 client.27844 handle_command_reply: tid=0 2024-03-23T23:10:51.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.706+0000 7f936f41e740 1 client.27844 shutdown 2024-03-23T23:10:51.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.706+0000 7f936f41e740 1 -- 172.21.15.12:0/1551996342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd0a7a79f0 msgr2=0x55cd0a7b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.707+0000 7f936f41e740 1 --2- 172.21.15.12:0/1551996342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd0a7a79f0 0x55cd0a7b5270 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9358009cf0 tx=0x7f9358004030 comp rx=0 tx=0).stop 2024-03-23T23:10:51.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.707+0000 7f936f41e740 1 -- 172.21.15.12:0/1551996342 shutdown_connections 2024-03-23T23:10:51.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.707+0000 7f936f41e740 1 --2- 172.21.15.12:0/1551996342 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cd0a81a070 0x55cd0a81c490 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.707+0000 7f936f41e740 1 --2- 172.21.15.12:0/1551996342 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd0a7a9580 0x55cd0a7b87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.707+0000 7f934b400640 0 client.27844 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:51.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.707+0000 7f936f41e740 1 --2- 172.21.15.12:0/1551996342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd0a7a8bc0 0x55cd0a7b57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.707+0000 7f936f41e740 1 --2- 172.21.15.12:0/1551996342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd0a7a79f0 0x55cd0a7b5270 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.707+0000 7f936f41e740 1 -- 172.21.15.12:0/1551996342 >> 172.21.15.12:0/1551996342 conn(0x55cd0a7a2de0 msgr2=0x55cd0a7a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:51.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.707+0000 7f936f41e740 1 -- 172.21.15.12:0/1551996342 shutdown_connections 2024-03-23T23:10:51.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.707+0000 7f936f41e740 1 -- 172.21.15.12:0/1551996342 wait complete. 2024-03-23T23:10:51.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.707+0000 7f936f41e740 20 client.27844 trim_cache size 0 max 16384 2024-03-23T23:10:51.709 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:10:51.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.708+0000 7f936b000640 1 -- 172.21.15.12:0/1662199549 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f934c0a2800 msgr2=0x7f934c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.708+0000 7f936b000640 1 --2- 172.21.15.12:0/1662199549 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f934c0a2800 0x7f934c0a4cc0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f935002c170 tx=0x7f935002df90 comp rx=0 tx=0).stop 2024-03-23T23:10:51.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.708+0000 7f936b000640 1 -- 172.21.15.12:0/1662199549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9364107260 msgr2=0x7f93641004f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:51.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.708+0000 7f936b000640 1 --2- 172.21.15.12:0/1662199549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9364107260 0x7f93641004f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f935c00eaa0 tx=0x7f935c00ef60 comp rx=0 tx=0).stop 2024-03-23T23:10:51.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.708+0000 7f936b000640 1 -- 172.21.15.12:0/1662199549 shutdown_connections 2024-03-23T23:10:51.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.708+0000 7f936b000640 1 --2- 172.21.15.12:0/1662199549 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f934c0a2800 0x7f934c0a4cc0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.708+0000 7f936b000640 1 --2- 172.21.15.12:0/1662199549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9364107260 0x7f93641004f0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.708+0000 7f936b000640 1 --2- 172.21.15.12:0/1662199549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9364104930 0x7f93640fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.708+0000 7f936b000640 1 --2- 172.21.15.12:0/1662199549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9364102000 0x7f9364101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:51.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.708+0000 7f936b000640 1 -- 172.21.15.12:0/1662199549 >> 172.21.15.12:0/1662199549 conn(0x7f93640fba10 msgr2=0x7f93641028a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:51.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.708+0000 7f936b000640 1 -- 172.21.15.12:0/1662199549 shutdown_connections 2024-03-23T23:10:51.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:51.708+0000 7f936b000640 1 -- 172.21.15.12:0/1662199549 wait complete. 2024-03-23T23:10:51.723 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:51 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3114889532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:10:52.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:51 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3114889532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:10:52.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:51 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3114889532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:10:52.725 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-23T23:10:52.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:10:52.726 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-23T23:10:52.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.962+0000 7feda2c00640 1 -- 172.21.15.12:0/3291292970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed9c0fff10 msgr2=0x7fed9c102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:52.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.962+0000 7feda2c00640 1 --2- 172.21.15.12:0/3291292970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed9c0fff10 0x7fed9c102340 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fed8c009be0 tx=0x7fed8c02bef0 comp rx=0 tx=0).stop 2024-03-23T23:10:52.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.962+0000 7feda2c00640 1 -- 172.21.15.12:0/3291292970 shutdown_connections 2024-03-23T23:10:52.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.962+0000 7feda2c00640 1 --2- 172.21.15.12:0/3291292970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed9c105230 0x7fed9c1056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:52.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.962+0000 7feda2c00640 1 --2- 172.21.15.12:0/3291292970 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed9c102880 0x7fed9c104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:52.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.962+0000 7feda2c00640 1 --2- 172.21.15.12:0/3291292970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed9c0fff10 0x7fed9c102340 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:52.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.962+0000 7feda2c00640 1 -- 172.21.15.12:0/3291292970 >> 172.21.15.12:0/3291292970 conn(0x7fed9c0fbb30 msgr2=0x7fed9c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:52.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.962+0000 7feda2c00640 1 -- 172.21.15.12:0/3291292970 shutdown_connections 2024-03-23T23:10:52.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.962+0000 7feda2c00640 1 -- 172.21.15.12:0/3291292970 wait complete. 2024-03-23T23:10:52.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.963+0000 7feda2c00640 1 Processor -- start 2024-03-23T23:10:52.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.963+0000 7feda2c00640 1 -- start start 2024-03-23T23:10:52.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.964+0000 7feda2c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed9c0fff10 0x7fed9c19dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:52.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.964+0000 7feda2c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed9c102880 0x7fed9c19e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:52.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.964+0000 7feda2c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed9c105230 0x7fed9c1a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:52.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.964+0000 7feda2c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c113940 con 0x7fed9c0fff10 2024-03-23T23:10:52.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.964+0000 7feda2c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c1137c0 con 0x7fed9c102880 2024-03-23T23:10:52.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.964+0000 7feda2c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c113ac0 con 0x7fed9c105230 2024-03-23T23:10:52.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.964+0000 7fed9b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed9c102880 0x7fed9c19e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:52.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.964+0000 7fed9be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed9c0fff10 0x7fed9c19dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:52.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.964+0000 7fed9b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed9c102880 0x7fed9c19e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37830/0 (socket says 172.21.15.12:37830) 2024-03-23T23:10:52.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.964+0000 7fed9b400640 1 -- 172.21.15.12:0/1553574036 learned_addr learned my addr 172.21.15.12:0/1553574036 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:52.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.964+0000 7feda0c00640 1 --2- 172.21.15.12:0/1553574036 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed9c105230 0x7fed9c1a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:52.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.964+0000 7fed9b400640 1 -- 172.21.15.12:0/1553574036 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed9c105230 msgr2=0x7fed9c1a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:52.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.964+0000 7fed9b400640 1 --2- 172.21.15.12:0/1553574036 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed9c105230 0x7fed9c1a5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:52.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.964+0000 7fed9b400640 1 -- 172.21.15.12:0/1553574036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed9c0fff10 msgr2=0x7fed9c19dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:52.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.964+0000 7fed9b400640 1 --2- 172.21.15.12:0/1553574036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed9c0fff10 0x7fed9c19dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:52.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.964+0000 7fed9b400640 1 -- 172.21.15.12:0/1553574036 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed9c1a5be0 con 0x7fed9c102880 2024-03-23T23:10:52.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.965+0000 7fed9b400640 1 --2- 172.21.15.12:0/1553574036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed9c102880 0x7fed9c19e500 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fed8800e9f0 tx=0x7fed8800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:52.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.965+0000 7fed98c00640 1 -- 172.21.15.12:0/1553574036 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8800ccc0 con 0x7fed9c102880 2024-03-23T23:10:52.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.965+0000 7fed98c00640 1 -- 172.21.15.12:0/1553574036 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed88010910 con 0x7fed9c102880 2024-03-23T23:10:52.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.965+0000 7fed98c00640 1 -- 172.21.15.12:0/1553574036 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed880188a0 con 0x7fed9c102880 2024-03-23T23:10:52.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.965+0000 7feda2c00640 1 -- 172.21.15.12:0/1553574036 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed9c1a5ed0 con 0x7fed9c102880 2024-03-23T23:10:52.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.965+0000 7feda2c00640 1 -- 172.21.15.12:0/1553574036 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed9c1a6340 con 0x7fed9c102880 2024-03-23T23:10:52.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.968+0000 7feda59ce740 1 Processor -- start 2024-03-23T23:10:52.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.969+0000 7feda59ce740 1 -- start start 2024-03-23T23:10:52.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.969+0000 7feda59ce740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d070e40a0 0x557d070e44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:52.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.969+0000 7feda59ce740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d070e4a90 0x557d070e9590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:52.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.969+0000 7feda59ce740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557d070e9c00 0x557d070ec090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:52.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.969+0000 7feda59ce740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d07049eb0 con 0x557d070e9c00 2024-03-23T23:10:52.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.969+0000 7feda59ce740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d06ff2a60 con 0x557d070e40a0 2024-03-23T23:10:52.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.969+0000 7feda59ce740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d07045f50 con 0x557d070e4a90 2024-03-23T23:10:52.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.969+0000 7fed98c00640 1 -- 172.21.15.12:0/1553574036 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fed88010ac0 con 0x7fed9c102880 2024-03-23T23:10:52.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.969+0000 7feda0c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d070e40a0 0x557d070e44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:52.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.969+0000 7feda0c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d070e40a0 0x557d070e44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37836/0 (socket says 172.21.15.12:37836) 2024-03-23T23:10:52.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.969+0000 7feda0c00640 1 -- 172.21.15.12:0/977194046 learned_addr learned my addr 172.21.15.12:0/977194046 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:52.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.969+0000 7fed9be00640 1 --2- 172.21.15.12:0/977194046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d070e4a90 0x557d070e9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:52.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.969+0000 7fed9b400640 1 --2- 172.21.15.12:0/977194046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557d070e9c00 0x557d070ec090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:52.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.970+0000 7feda0c00640 1 -- 172.21.15.12:0/977194046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d070e4a90 msgr2=0x557d070e9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:52.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.970+0000 7feda0c00640 1 --2- 172.21.15.12:0/977194046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d070e4a90 0x557d070e9590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:52.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.970+0000 7feda0c00640 1 -- 172.21.15.12:0/977194046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557d070e9c00 msgr2=0x557d070ec090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:52.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.970+0000 7feda0c00640 1 --2- 172.21.15.12:0/977194046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557d070e9c00 0x557d070ec090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:52.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.970+0000 7feda0c00640 1 -- 172.21.15.12:0/977194046 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d06e267e0 con 0x557d070e40a0 2024-03-23T23:10:52.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.970+0000 7feda0c00640 1 --2- 172.21.15.12:0/977194046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d070e40a0 0x557d070e44c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fed94009c60 tx=0x7fed94011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdf550e31150ee5e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:52.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.970+0000 7fed91200640 1 -- 172.21.15.12:0/977194046 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9401e070 con 0x557d070e40a0 2024-03-23T23:10:52.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.970+0000 7fed91200640 1 -- 172.21.15.12:0/977194046 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed94019b10 con 0x557d070e40a0 2024-03-23T23:10:52.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.970+0000 7fed91200640 1 -- 172.21.15.12:0/977194046 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9400ea70 con 0x557d070e40a0 2024-03-23T23:10:52.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.971+0000 7fed98c00640 1 --2- 172.21.15.12:0/1553574036 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fed800a2880 0x7fed800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:52.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.971+0000 7fed98c00640 1 -- 172.21.15.12:0/1553574036 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fed880cee20 con 0x7fed9c102880 2024-03-23T23:10:52.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.971+0000 7feda59ce740 1 -- 172.21.15.12:0/977194046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d070e40a0 msgr2=0x557d070e44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:52.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.971+0000 7feda59ce740 1 --2- 172.21.15.12:0/977194046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d070e40a0 0x557d070e44c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fed94009c60 tx=0x7fed94011810 comp rx=0 tx=0).stop 2024-03-23T23:10:52.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.971+0000 7fed9be00640 1 --2- 172.21.15.12:0/1553574036 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fed800a2880 0x7fed800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:52.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.971+0000 7feda59ce740 1 -- 172.21.15.12:0/977194046 shutdown_connections 2024-03-23T23:10:52.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.971+0000 7feda59ce740 1 --2- 172.21.15.12:0/977194046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557d070e9c00 0x557d070ec090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:52.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.971+0000 7feda59ce740 1 --2- 172.21.15.12:0/977194046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d070e4a90 0x557d070e9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:52.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.971+0000 7feda59ce740 1 --2- 172.21.15.12:0/977194046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d070e40a0 0x557d070e44c0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:52.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.971+0000 7feda59ce740 1 -- 172.21.15.12:0/977194046 >> 172.21.15.12:0/977194046 conn(0x557d070df610 msgr2=0x557d070dfa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:52.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.971+0000 7feda59ce740 1 -- 172.21.15.12:0/977194046 shutdown_connections 2024-03-23T23:10:52.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.971+0000 7feda59ce740 1 -- 172.21.15.12:0/977194046 wait complete. 2024-03-23T23:10:52.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.972+0000 7feda59ce740 1 -- start start 2024-03-23T23:10:52.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.972+0000 7fed9be00640 1 --2- 172.21.15.12:0/1553574036 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fed800a2880 0x7fed800a4d40 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fed8c006010 tx=0x7fed8c000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:52.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.972+0000 7feda59ce740 1 Processor -- start 2024-03-23T23:10:52.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.972+0000 7feda59ce740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557d070e40a0 0x557d070ee240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:52.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.972+0000 7feda59ce740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d070e4a90 0x557d070ee780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:52.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.972+0000 7feda59ce740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d070e9c00 0x557d070f57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:52.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.972+0000 7feda59ce740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d070ecbb0 con 0x557d070e40a0 2024-03-23T23:10:52.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.973+0000 7feda59ce740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d07005eb0 con 0x557d070e9c00 2024-03-23T23:10:52.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.973+0000 7feda59ce740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d070ecd30 con 0x557d070e4a90 2024-03-23T23:10:52.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.973+0000 7feda59ce740 10 client.0 _ll_register_callbacks cb 0x557d06ffa150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:52.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.973+0000 7fed9b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d070e4a90 0x557d070ee780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:52.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.973+0000 7feda0c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d070e9c00 0x557d070f57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:52.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.973+0000 7feda0c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d070e9c00 0x557d070f57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37842/0 (socket says 172.21.15.12:37842) 2024-03-23T23:10:52.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.973+0000 7fed9be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557d070e40a0 0x557d070ee240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:52.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.973+0000 7feda0c00640 1 -- 172.21.15.12:0/2657523340 learned_addr learned my addr 172.21.15.12:0/2657523340 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:52.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.973+0000 7fed9b400640 1 -- 172.21.15.12:0/2657523340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d070e9c00 msgr2=0x557d070f57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:52.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.973+0000 7fed9b400640 1 --2- 172.21.15.12:0/2657523340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d070e9c00 0x557d070f57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:52.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.973+0000 7fed9b400640 1 -- 172.21.15.12:0/2657523340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557d070e40a0 msgr2=0x557d070ee240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:52.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.973+0000 7fed9b400640 1 --2- 172.21.15.12:0/2657523340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557d070e40a0 0x557d070ee240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:52.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.973+0000 7fed9b400640 1 -- 172.21.15.12:0/2657523340 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d06e86f70 con 0x557d070e4a90 2024-03-23T23:10:52.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.973+0000 7feda0c00640 1 --2- 172.21.15.12:0/2657523340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d070e9c00 0x557d070f57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:52.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.973+0000 7fed9b400640 1 --2- 172.21.15.12:0/2657523340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d070e4a90 0x557d070ee780 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fed8809b700 tx=0x7fed8800cef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab6ee3b1cc9462d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:52.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.973+0000 7fed85600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:10:52.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.974+0000 7fed85600640 1 -- 172.21.15.12:0/2657523340 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8809b770 con 0x557d070e4a90 2024-03-23T23:10:52.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.974+0000 7fed85600640 1 -- 172.21.15.12:0/2657523340 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fed88097de0 con 0x557d070e4a90 2024-03-23T23:10:52.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.974+0000 7feda59ce740 1 -- 172.21.15.12:0/2657523340 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fed94019b10 con 0x557d070e4a90 2024-03-23T23:10:52.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.974+0000 7fed85600640 1 -- 172.21.15.12:0/2657523340 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed8809f8c0 con 0x557d070e4a90 2024-03-23T23:10:52.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.974+0000 7feda59ce740 10 client.27856 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:52.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.974+0000 7feda59ce740 1 -- 172.21.15.12:0/2657523340 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x557d06d1e170 con 0x557d070e4a90 2024-03-23T23:10:52.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.974+0000 7fed85600640 1 -- 172.21.15.12:0/2657523340 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed94019b10 con 0x557d070e4a90 2024-03-23T23:10:52.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.974+0000 7fed85600640 1 -- 172.21.15.12:0/2657523340 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fed880974c0 con 0x557d070e4a90 2024-03-23T23:10:52.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.974+0000 7feda59ce740 10 client.27856 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:52.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.974+0000 7feda59ce740 10 client.27856 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:52.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.974+0000 7feda59ce740 20 client.27856 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:52.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.975+0000 7feda59ce740 1 --2- 172.21.15.12:0/2657523340 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557d070fa020 0x557d070fc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:52.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.975+0000 7feda59ce740 4 client.27856 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:10:52.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.975+0000 7feda59ce740 1 -- 172.21.15.12:0/2657523340 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557d070573f0 con 0x557d070fa020 2024-03-23T23:10:52.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.975+0000 7fed9be00640 1 --2- 172.21.15.12:0/2657523340 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557d070fa020 0x557d070fc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:52.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.975+0000 7fed9be00640 1 --2- 172.21.15.12:0/2657523340 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557d070fa020 0x557d070fc460 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d08b4c9d909066b server_cookie=69a48302b899dad4 in_seq=0 out_seq=0 2024-03-23T23:10:52.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.975+0000 7fed85600640 10 client.27856 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:52.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.976+0000 7fed85600640 1 -- 172.21.15.12:0/2657523340 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557d070573f0 con 0x557d070fa020 2024-03-23T23:10:52.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.976+0000 7fed85600640 10 client.27856 handle_command_reply: tid=0 2024-03-23T23:10:52.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.976+0000 7feda59ce740 1 client.27856 shutdown 2024-03-23T23:10:52.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.976+0000 7feda59ce740 1 -- 172.21.15.12:0/2657523340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d070e4a90 msgr2=0x557d070ee780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:52.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.976+0000 7feda59ce740 1 --2- 172.21.15.12:0/2657523340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d070e4a90 0x557d070ee780 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fed8809b700 tx=0x7fed8800cef0 comp rx=0 tx=0).stop 2024-03-23T23:10:52.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.976+0000 7feda59ce740 1 -- 172.21.15.12:0/2657523340 shutdown_connections 2024-03-23T23:10:52.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.976+0000 7feda59ce740 1 --2- 172.21.15.12:0/2657523340 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557d070fa020 0x557d070fc460 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:52.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.976+0000 7feda59ce740 1 --2- 172.21.15.12:0/2657523340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d070e9c00 0x557d070f57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:52.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.976+0000 7feda59ce740 1 --2- 172.21.15.12:0/2657523340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d070e4a90 0x557d070ee780 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:52.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.976+0000 7fed85600640 0 client.27856 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:52.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.976+0000 7feda59ce740 1 --2- 172.21.15.12:0/2657523340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557d070e40a0 0x557d070ee240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:52.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.976+0000 7feda59ce740 1 -- 172.21.15.12:0/2657523340 >> 172.21.15.12:0/2657523340 conn(0x557d070df610 msgr2=0x557d070de830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:52.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.977+0000 7feda59ce740 1 -- 172.21.15.12:0/2657523340 shutdown_connections 2024-03-23T23:10:52.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.977+0000 7feda59ce740 1 -- 172.21.15.12:0/2657523340 wait complete. 2024-03-23T23:10:52.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.977+0000 7feda59ce740 20 client.27856 trim_cache size 0 max 16384 2024-03-23T23:10:52.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.994+0000 7feda59ce740 1 Processor -- start 2024-03-23T23:10:52.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.994+0000 7feda59ce740 1 -- start start 2024-03-23T23:10:52.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.995+0000 7feda59ce740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d070e40a0 0x557d070e2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:52.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.995+0000 7feda59ce740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d070e4a90 0x557d070f5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:52.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.995+0000 7feda59ce740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557d070e9c00 0x557d070ddfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:52.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.995+0000 7feda59ce740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c1b1640 con 0x557d070e9c00 2024-03-23T23:10:52.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.995+0000 7feda59ce740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c1af140 con 0x557d070e40a0 2024-03-23T23:10:52.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.995+0000 7feda59ce740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c1ad1e0 con 0x557d070e4a90 2024-03-23T23:10:52.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.995+0000 7fed9b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d070e4a90 0x557d070f5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:52.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.995+0000 7feda0c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557d070e9c00 0x557d070ddfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:52.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.995+0000 7fed9b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d070e4a90 0x557d070f5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54374/0 (socket says 172.21.15.12:54374) 2024-03-23T23:10:52.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.995+0000 7fed9b400640 1 -- 172.21.15.12:0/2159225267 learned_addr learned my addr 172.21.15.12:0/2159225267 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:52.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.995+0000 7fed9be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d070e40a0 0x557d070e2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:52.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.995+0000 7fed9b400640 1 -- 172.21.15.12:0/2159225267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d070e40a0 msgr2=0x557d070e2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:52.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.995+0000 7fed9b400640 1 --2- 172.21.15.12:0/2159225267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d070e40a0 0x557d070e2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:52.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.995+0000 7fed9b400640 1 -- 172.21.15.12:0/2159225267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557d070e9c00 msgr2=0x557d070ddfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:52.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.995+0000 7fed9b400640 1 --2- 172.21.15.12:0/2159225267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557d070e9c00 0x557d070ddfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:52.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.995+0000 7fed9b400640 1 -- 172.21.15.12:0/2159225267 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d070e0080 con 0x557d070e4a90 2024-03-23T23:10:52.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.996+0000 7fed9b400640 1 --2- 172.21.15.12:0/2159225267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d070e4a90 0x557d070f5780 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fed88097bd0 tx=0x7fed8800e940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cc412542610f646 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:52.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.996+0000 7fed87e00640 1 -- 172.21.15.12:0/2159225267 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8809b0d0 con 0x557d070e4a90 2024-03-23T23:10:52.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.996+0000 7fed87e00640 1 -- 172.21.15.12:0/2159225267 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed880a3f10 con 0x557d070e4a90 2024-03-23T23:10:52.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.996+0000 7fed87e00640 1 -- 172.21.15.12:0/2159225267 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed880afe40 con 0x557d070e4a90 2024-03-23T23:10:52.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.996+0000 7feda59ce740 1 -- 172.21.15.12:0/2159225267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d070e4a90 msgr2=0x557d070f5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:52.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.996+0000 7feda59ce740 1 --2- 172.21.15.12:0/2159225267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d070e4a90 0x557d070f5780 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fed88097bd0 tx=0x7fed8800e940 comp rx=0 tx=0).stop 2024-03-23T23:10:52.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.997+0000 7feda59ce740 1 -- 172.21.15.12:0/2159225267 shutdown_connections 2024-03-23T23:10:52.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.997+0000 7feda59ce740 1 --2- 172.21.15.12:0/2159225267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557d070e9c00 0x557d070ddfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:52.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.997+0000 7feda59ce740 1 --2- 172.21.15.12:0/2159225267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d070e4a90 0x557d070f5780 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:52.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.997+0000 7feda59ce740 1 --2- 172.21.15.12:0/2159225267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d070e40a0 0x557d070e2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:52.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.997+0000 7feda59ce740 1 -- 172.21.15.12:0/2159225267 >> 172.21.15.12:0/2159225267 conn(0x557d070df610 msgr2=0x557d070e5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:52.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.997+0000 7feda59ce740 1 -- 172.21.15.12:0/2159225267 shutdown_connections 2024-03-23T23:10:52.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.997+0000 7feda59ce740 1 -- 172.21.15.12:0/2159225267 wait complete. 2024-03-23T23:10:53.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.997+0000 7feda59ce740 1 -- start start 2024-03-23T23:10:53.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.997+0000 7feda59ce740 1 Processor -- start 2024-03-23T23:10:53.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.998+0000 7feda59ce740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557d070e40a0 0x557d070f52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:53.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.998+0000 7feda59ce740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d070e4a90 0x557d070f3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:53.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.998+0000 7feda59ce740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d070e9c00 0x557d070f3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:53.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.998+0000 7feda59ce740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c1ad7e0 con 0x557d070e40a0 2024-03-23T23:10:53.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.998+0000 7feda59ce740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c1aeae0 con 0x557d070e4a90 2024-03-23T23:10:53.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.998+0000 7feda59ce740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c1ade20 con 0x557d070e9c00 2024-03-23T23:10:53.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.998+0000 7feda59ce740 10 client.0 _ll_register_callbacks cb 0x7fed880d2390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:53.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.998+0000 7fed9be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557d070e40a0 0x557d070f52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:53.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.998+0000 7feda0c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d070e9c00 0x557d070f3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:53.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.998+0000 7fed9b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d070e4a90 0x557d070f3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:53.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.998+0000 7feda0c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d070e9c00 0x557d070f3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54388/0 (socket says 172.21.15.12:54388) 2024-03-23T23:10:53.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.998+0000 7feda0c00640 1 -- 172.21.15.12:0/2129019916 learned_addr learned my addr 172.21.15.12:0/2129019916 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:53.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.998+0000 7fed9b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d070e4a90 0x557d070f3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37862/0 (socket says 172.21.15.12:37862) 2024-03-23T23:10:53.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.998+0000 7feda0c00640 1 -- 172.21.15.12:0/2129019916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d070e4a90 msgr2=0x557d070f3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:53.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.998+0000 7feda0c00640 1 --2- 172.21.15.12:0/2129019916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d070e4a90 0x557d070f3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.998+0000 7feda0c00640 1 -- 172.21.15.12:0/2129019916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557d070e40a0 msgr2=0x557d070f52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:53.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.998+0000 7feda0c00640 1 --2- 172.21.15.12:0/2129019916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557d070e40a0 0x557d070f52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.998+0000 7feda0c00640 1 -- 172.21.15.12:0/2129019916 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d070f43b0 con 0x557d070e9c00 2024-03-23T23:10:53.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.999+0000 7feda0c00640 1 --2- 172.21.15.12:0/2129019916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d070e9c00 0x557d070f3e70 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fed94009670 tx=0x7fed940097c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=424be484ffe1680c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:53.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.999+0000 7fed85600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:10:53.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.999+0000 7fed85600640 1 -- 172.21.15.12:0/2129019916 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9401e040 con 0x557d070e9c00 2024-03-23T23:10:53.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.999+0000 7feda59ce740 1 -- 172.21.15.12:0/2129019916 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fed880a3f10 con 0x557d070e9c00 2024-03-23T23:10:53.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.999+0000 7fed85600640 1 -- 172.21.15.12:0/2129019916 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fed940054c0 con 0x557d070e9c00 2024-03-23T23:10:53.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.999+0000 7fed85600640 1 -- 172.21.15.12:0/2129019916 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed940035f0 con 0x557d070e9c00 2024-03-23T23:10:53.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.999+0000 7feda59ce740 10 client.27862 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:53.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:52.999+0000 7feda59ce740 1 -- 172.21.15.12:0/2129019916 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x557d07156620 con 0x557d070e9c00 2024-03-23T23:10:53.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.000+0000 7fed85600640 1 -- 172.21.15.12:0/2129019916 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed880a3f10 con 0x557d070e9c00 2024-03-23T23:10:53.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.000+0000 7fed85600640 1 -- 172.21.15.12:0/2129019916 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fed940308c0 con 0x557d070e9c00 2024-03-23T23:10:53.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.000+0000 7feda59ce740 10 client.27862 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:53.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.000+0000 7feda59ce740 10 client.27862 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:53.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.000+0000 7feda59ce740 20 client.27862 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:53.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.000+0000 7feda59ce740 1 --2- 172.21.15.12:0/2129019916 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557d07156e30 0x557d07159250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:53.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.000+0000 7feda59ce740 4 client.27862 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-23T23:10:53.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.000+0000 7feda59ce740 1 -- 172.21.15.12:0/2129019916 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x557d07051740 con 0x557d07156e30 2024-03-23T23:10:53.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.000+0000 7fed9be00640 1 --2- 172.21.15.12:0/2129019916 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557d07156e30 0x557d07159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:53.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.001+0000 7fed9be00640 1 --2- 172.21.15.12:0/2129019916 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557d07156e30 0x557d07159250 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1ba3ab90be4578f server_cookie=ace065574cbbd050 in_seq=0 out_seq=0 2024-03-23T23:10:53.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.001+0000 7fed85600640 10 client.27862 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:53.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.002+0000 7fed85600640 1 -- 172.21.15.12:0/2129019916 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x557d07051740 con 0x557d07156e30 2024-03-23T23:10:53.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.002+0000 7fed85600640 10 client.27862 handle_command_reply: tid=0 2024-03-23T23:10:53.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.002+0000 7feda59ce740 1 client.27862 shutdown 2024-03-23T23:10:53.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.002+0000 7feda59ce740 1 -- 172.21.15.12:0/2129019916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d070e9c00 msgr2=0x557d070f3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:53.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.002+0000 7feda59ce740 1 --2- 172.21.15.12:0/2129019916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d070e9c00 0x557d070f3e70 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fed94009670 tx=0x7fed940097c0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.002+0000 7feda59ce740 1 -- 172.21.15.12:0/2129019916 shutdown_connections 2024-03-23T23:10:53.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.002+0000 7feda59ce740 1 --2- 172.21.15.12:0/2129019916 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x557d07156e30 0x557d07159250 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.002+0000 7feda59ce740 1 --2- 172.21.15.12:0/2129019916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d070e9c00 0x557d070f3e70 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.002+0000 7feda59ce740 1 --2- 172.21.15.12:0/2129019916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d070e4a90 0x557d070f3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.002+0000 7feda59ce740 1 --2- 172.21.15.12:0/2129019916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557d070e40a0 0x557d070f52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.002+0000 7fed85600640 0 client.27862 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:53.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.002+0000 7feda59ce740 1 -- 172.21.15.12:0/2129019916 >> 172.21.15.12:0/2129019916 conn(0x557d070df610 msgr2=0x557d07100d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:53.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.002+0000 7feda59ce740 1 -- 172.21.15.12:0/2129019916 shutdown_connections 2024-03-23T23:10:53.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.002+0000 7feda59ce740 1 -- 172.21.15.12:0/2129019916 wait complete. 2024-03-23T23:10:53.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.002+0000 7feda59ce740 20 client.27862 trim_cache size 0 max 16384 2024-03-23T23:10:53.005 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:10:53.005 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-03-23T23:10:53.005 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "030dcd17-08a2-4ba6-a094-91ebd5061e5b", 2024-03-23T23:10:53.005 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-03-23T23:10:53.005 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:10:53.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.003+0000 7feda2c00640 1 -- 172.21.15.12:0/1553574036 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fed800a2880 msgr2=0x7fed800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:53.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.003+0000 7feda2c00640 1 --2- 172.21.15.12:0/1553574036 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fed800a2880 0x7fed800a4d40 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fed8c006010 tx=0x7fed8c000f40 comp rx=0 tx=0).stop 2024-03-23T23:10:53.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.003+0000 7feda2c00640 1 -- 172.21.15.12:0/1553574036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed9c102880 msgr2=0x7fed9c19e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:53.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.003+0000 7feda2c00640 1 --2- 172.21.15.12:0/1553574036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed9c102880 0x7fed9c19e500 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fed8800e9f0 tx=0x7fed8800edf0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.003+0000 7feda2c00640 1 -- 172.21.15.12:0/1553574036 shutdown_connections 2024-03-23T23:10:53.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.003+0000 7feda2c00640 1 --2- 172.21.15.12:0/1553574036 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fed800a2880 0x7fed800a4d40 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.003+0000 7feda2c00640 1 --2- 172.21.15.12:0/1553574036 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed9c105230 0x7fed9c1a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.003+0000 7feda2c00640 1 --2- 172.21.15.12:0/1553574036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed9c102880 0x7fed9c19e500 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.003+0000 7feda2c00640 1 --2- 172.21.15.12:0/1553574036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed9c0fff10 0x7fed9c19dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.003+0000 7feda2c00640 1 -- 172.21.15.12:0/1553574036 >> 172.21.15.12:0/1553574036 conn(0x7fed9c0fbb30 msgr2=0x7fed9c103210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:53.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.003+0000 7feda2c00640 1 -- 172.21.15.12:0/1553574036 shutdown_connections 2024-03-23T23:10:53.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.003+0000 7feda2c00640 1 -- 172.21.15.12:0/1553574036 wait complete. 2024-03-23T23:10:53.019 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:52 smithi012 ceph-mon[31695]: pgmap v662: 97 pgs: 97 active+clean; 2.0 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:10:53.023 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:10:53.023 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:10:53.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:52 smithi173 ceph-mon[36116]: pgmap v662: 97 pgs: 97 active+clean; 2.0 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:10:53.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:52 smithi203 ceph-mon[36693]: pgmap v662: 97 pgs: 97 active+clean; 2.0 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:10:53.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.247+0000 7f3277e00640 1 -- 172.21.15.12:0/1299814833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3278107bc0 msgr2=0x7f3278107fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:53.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.247+0000 7f3277e00640 1 --2- 172.21.15.12:0/1299814833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3278107bc0 0x7f3278107fe0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f3268009900 tx=0x7f326802bc30 comp rx=0 tx=0).stop 2024-03-23T23:10:53.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.248+0000 7f3277e00640 1 -- 172.21.15.12:0/1299814833 shutdown_connections 2024-03-23T23:10:53.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.248+0000 7f3277e00640 1 --2- 172.21.15.12:0/1299814833 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3278109770 0x7f3278110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.248+0000 7f3277e00640 1 --2- 172.21.15.12:0/1299814833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3278108db0 0x7f3278109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.248+0000 7f3277e00640 1 --2- 172.21.15.12:0/1299814833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3278107bc0 0x7f3278107fe0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.248+0000 7f3277e00640 1 -- 172.21.15.12:0/1299814833 >> 172.21.15.12:0/1299814833 conn(0x7f3278076b70 msgr2=0x7f3278078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:53.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.248+0000 7f3277e00640 1 -- 172.21.15.12:0/1299814833 shutdown_connections 2024-03-23T23:10:53.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.248+0000 7f3277e00640 1 -- 172.21.15.12:0/1299814833 wait complete. 2024-03-23T23:10:53.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.249+0000 7f3277e00640 1 Processor -- start 2024-03-23T23:10:53.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.249+0000 7f3277e00640 1 -- start start 2024-03-23T23:10:53.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.249+0000 7f3277e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3278107bc0 0x7f327819df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:53.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.250+0000 7f3277e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3278108db0 0x7f327819e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:53.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.250+0000 7f3277e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3278109770 0x7f32781a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:53.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.250+0000 7f3277e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3278113860 con 0x7f3278108db0 2024-03-23T23:10:53.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.250+0000 7f3277e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32781136e0 con 0x7f3278107bc0 2024-03-23T23:10:53.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.250+0000 7f3277e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32781139e0 con 0x7f3278109770 2024-03-23T23:10:53.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.250+0000 7f3276a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3278107bc0 0x7f327819df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:53.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.250+0000 7f3276a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3278107bc0 0x7f327819df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37876/0 (socket says 172.21.15.12:37876) 2024-03-23T23:10:53.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.250+0000 7f3276a00640 1 -- 172.21.15.12:0/224141029 learned_addr learned my addr 172.21.15.12:0/224141029 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:53.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.250+0000 7f326fe00640 1 --2- 172.21.15.12:0/224141029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3278108db0 0x7f327819e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:53.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.250+0000 7f3277400640 1 --2- 172.21.15.12:0/224141029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3278109770 0x7f32781a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:53.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.250+0000 7f3276a00640 1 -- 172.21.15.12:0/224141029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3278109770 msgr2=0x7f32781a5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:53.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.250+0000 7f3276a00640 1 --2- 172.21.15.12:0/224141029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3278109770 0x7f32781a5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.250+0000 7f3276a00640 1 -- 172.21.15.12:0/224141029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3278108db0 msgr2=0x7f327819e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:53.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.250+0000 7f3276a00640 1 --2- 172.21.15.12:0/224141029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3278108db0 0x7f327819e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.250+0000 7f3276a00640 1 -- 172.21.15.12:0/224141029 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32781a5ab0 con 0x7f3278107bc0 2024-03-23T23:10:53.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.250+0000 7f3277400640 1 --2- 172.21.15.12:0/224141029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3278109770 0x7f32781a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:53.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.251+0000 7f3276a00640 1 --2- 172.21.15.12:0/224141029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3278107bc0 0x7f327819df30 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3268000c00 tx=0x7f326802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:53.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.251+0000 7f326f400640 1 -- 172.21.15.12:0/224141029 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3268043020 con 0x7f3278107bc0 2024-03-23T23:10:53.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.251+0000 7f326f400640 1 -- 172.21.15.12:0/224141029 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32680054e0 con 0x7f3278107bc0 2024-03-23T23:10:53.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.251+0000 7f326f400640 1 -- 172.21.15.12:0/224141029 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3268003c40 con 0x7f3278107bc0 2024-03-23T23:10:53.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.251+0000 7f3277e00640 1 -- 172.21.15.12:0/224141029 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32781a5d40 con 0x7f3278107bc0 2024-03-23T23:10:53.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.251+0000 7f3277e00640 1 -- 172.21.15.12:0/224141029 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32781a6220 con 0x7f3278107bc0 2024-03-23T23:10:53.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.254+0000 7f328067a740 1 Processor -- start 2024-03-23T23:10:53.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.255+0000 7f326f400640 1 -- 172.21.15.12:0/224141029 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3268046020 con 0x7f3278107bc0 2024-03-23T23:10:53.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.255+0000 7f328067a740 1 -- start start 2024-03-23T23:10:53.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.255+0000 7f328067a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2e7050 0x55aaad2e7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:53.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.255+0000 7f328067a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aaad2e7a40 0x55aaad2ec540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:53.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.255+0000 7f328067a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55aaad2ecbb0 0x55aaad2ef040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:53.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.255+0000 7f328067a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaad24c250 con 0x55aaad2ecbb0 2024-03-23T23:10:53.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.255+0000 7f328067a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaad1f4a30 con 0x55aaad2e7050 2024-03-23T23:10:53.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.255+0000 7f328067a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaad2482f0 con 0x55aaad2e7a40 2024-03-23T23:10:53.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.255+0000 7f3276a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aaad2e7a40 0x55aaad2ec540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:53.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.255+0000 7f3277400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2e7050 0x55aaad2e7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:53.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.255+0000 7f3276a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aaad2e7a40 0x55aaad2ec540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54412/0 (socket says 172.21.15.12:54412) 2024-03-23T23:10:53.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.255+0000 7f3276a00640 1 -- 172.21.15.12:0/1174917901 learned_addr learned my addr 172.21.15.12:0/1174917901 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:53.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.255+0000 7f3277400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2e7050 0x55aaad2e7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37878/0 (socket says 172.21.15.12:37878) 2024-03-23T23:10:53.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.256+0000 7f326fe00640 1 --2- 172.21.15.12:0/1174917901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55aaad2ecbb0 0x55aaad2ef040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:53.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.256+0000 7f3277400640 1 -- 172.21.15.12:0/1174917901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aaad2e7a40 msgr2=0x55aaad2ec540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:53.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.256+0000 7f3277400640 1 --2- 172.21.15.12:0/1174917901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aaad2e7a40 0x55aaad2ec540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.256+0000 7f3277400640 1 -- 172.21.15.12:0/1174917901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55aaad2ecbb0 msgr2=0x55aaad2ef040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:53.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.256+0000 7f3277400640 1 --2- 172.21.15.12:0/1174917901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55aaad2ecbb0 0x55aaad2ef040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.256+0000 7f3277400640 1 -- 172.21.15.12:0/1174917901 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aaad1df690 con 0x55aaad2e7050 2024-03-23T23:10:53.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.256+0000 7f3277400640 1 --2- 172.21.15.12:0/1174917901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2e7050 0x55aaad2e7470 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f3270005fd0 tx=0x7f3270011bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b331cc3648138b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:53.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.256+0000 7f3267e00640 1 -- 172.21.15.12:0/1174917901 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3270004d20 con 0x55aaad2e7050 2024-03-23T23:10:53.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.256+0000 7f326f400640 1 --2- 172.21.15.12:0/224141029 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f325c0a2880 0x7f325c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:53.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.256+0000 7f3267e00640 1 -- 172.21.15.12:0/1174917901 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f327000f9a0 con 0x55aaad2e7050 2024-03-23T23:10:53.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.256+0000 7f3267e00640 1 -- 172.21.15.12:0/1174917901 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3270004510 con 0x55aaad2e7050 2024-03-23T23:10:53.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.257+0000 7f326f400640 1 -- 172.21.15.12:0/224141029 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f32680f09d0 con 0x7f3278107bc0 2024-03-23T23:10:53.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.257+0000 7f3276a00640 1 --2- 172.21.15.12:0/224141029 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f325c0a2880 0x7f325c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:53.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.257+0000 7f328067a740 1 -- 172.21.15.12:0/1174917901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2e7050 msgr2=0x55aaad2e7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:53.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.257+0000 7f328067a740 1 --2- 172.21.15.12:0/1174917901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2e7050 0x55aaad2e7470 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f3270005fd0 tx=0x7f3270011bf0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.257+0000 7f328067a740 1 -- 172.21.15.12:0/1174917901 shutdown_connections 2024-03-23T23:10:53.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.257+0000 7f328067a740 1 --2- 172.21.15.12:0/1174917901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55aaad2ecbb0 0x55aaad2ef040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.257+0000 7f328067a740 1 --2- 172.21.15.12:0/1174917901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aaad2e7a40 0x55aaad2ec540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.257+0000 7f328067a740 1 --2- 172.21.15.12:0/1174917901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2e7050 0x55aaad2e7470 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.257+0000 7f3276a00640 1 --2- 172.21.15.12:0/224141029 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f325c0a2880 0x7f325c0a4d40 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f3268005e90 tx=0x7f32680dd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:53.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.257+0000 7f328067a740 1 -- 172.21.15.12:0/1174917901 >> 172.21.15.12:0/1174917901 conn(0x55aaad2af200 msgr2=0x55aaad2e29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:53.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.257+0000 7f328067a740 1 -- 172.21.15.12:0/1174917901 shutdown_connections 2024-03-23T23:10:53.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.257+0000 7f328067a740 1 -- 172.21.15.12:0/1174917901 wait complete. 2024-03-23T23:10:53.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.258+0000 7f328067a740 1 -- start start 2024-03-23T23:10:53.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.258+0000 7f328067a740 1 Processor -- start 2024-03-23T23:10:53.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.259+0000 7f328067a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2e7050 0x55aaad2f1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:53.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.259+0000 7f328067a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55aaad2e7a40 0x55aaad2f15c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:53.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.259+0000 7f328067a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aaad2ecbb0 0x55aaad2f8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:53.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.259+0000 7f328067a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaad2efb60 con 0x55aaad2e7a40 2024-03-23T23:10:53.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.259+0000 7f328067a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaad207e80 con 0x55aaad2e7050 2024-03-23T23:10:53.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.259+0000 7f328067a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaad2efce0 con 0x55aaad2ecbb0 2024-03-23T23:10:53.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.259+0000 7f328067a740 10 client.0 _ll_register_callbacks cb 0x55aaad21a130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:53.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.259+0000 7f326fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55aaad2e7a40 0x55aaad2f15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:53.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.259+0000 7f3277400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2e7050 0x55aaad2f1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:53.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.259+0000 7f3277400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2e7050 0x55aaad2f1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37880/0 (socket says 172.21.15.12:37880) 2024-03-23T23:10:53.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.259+0000 7f326fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55aaad2e7a40 0x55aaad2f15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54810/0 (socket says 172.21.15.12:54810) 2024-03-23T23:10:53.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.259+0000 7f3277400640 1 -- 172.21.15.12:0/1297477879 learned_addr learned my addr 172.21.15.12:0/1297477879 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:53.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.259+0000 7f3277400640 1 --2- 172.21.15.12:0/1297477879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aaad2ecbb0 0x55aaad2f8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:53.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.259+0000 7f3277400640 1 -- 172.21.15.12:0/1297477879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aaad2ecbb0 msgr2=0x55aaad2f8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:53.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.259+0000 7f3277400640 1 --2- 172.21.15.12:0/1297477879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aaad2ecbb0 0x55aaad2f8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.259+0000 7f3277400640 1 -- 172.21.15.12:0/1297477879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55aaad2e7a40 msgr2=0x55aaad2f15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:53.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.259+0000 7f3277400640 1 --2- 172.21.15.12:0/1297477879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55aaad2e7a40 0x55aaad2f15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.259+0000 7f3277400640 1 -- 172.21.15.12:0/1297477879 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aaad02a7d0 con 0x55aaad2e7050 2024-03-23T23:10:53.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.259+0000 7f326fe00640 1 --2- 172.21.15.12:0/1297477879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55aaad2e7a40 0x55aaad2f15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:53.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.260+0000 7f3277400640 1 --2- 172.21.15.12:0/1297477879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2e7050 0x55aaad2f1080 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f32700115a0 tx=0x7f3270005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff601fa081fc1dfb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:53.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.260+0000 7f3264c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:10:53.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.260+0000 7f3264c00640 1 -- 172.21.15.12:0/1297477879 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327001e040 con 0x55aaad2e7050 2024-03-23T23:10:53.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.260+0000 7f3264c00640 1 -- 172.21.15.12:0/1297477879 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f3270011230 con 0x55aaad2e7050 2024-03-23T23:10:53.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.260+0000 7f328067a740 1 -- 172.21.15.12:0/1297477879 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f327000f9a0 con 0x55aaad2e7050 2024-03-23T23:10:53.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.260+0000 7f3264c00640 1 -- 172.21.15.12:0/1297477879 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32700044a0 con 0x55aaad2e7050 2024-03-23T23:10:53.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.260+0000 7f328067a740 10 client.27911 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:53.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.260+0000 7f328067a740 1 -- 172.21.15.12:0/1297477879 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55aaacf603f0 con 0x55aaad2e7050 2024-03-23T23:10:53.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.260+0000 7f3264c00640 1 -- 172.21.15.12:0/1297477879 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f327000f9a0 con 0x55aaad2e7050 2024-03-23T23:10:53.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.261+0000 7f3264c00640 1 -- 172.21.15.12:0/1297477879 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f3270024070 con 0x55aaad2e7050 2024-03-23T23:10:53.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.261+0000 7f328067a740 10 client.27911 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:53.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.261+0000 7f328067a740 10 client.27911 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:53.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.261+0000 7f328067a740 20 client.27911 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:53.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.261+0000 7f328067a740 1 --2- 172.21.15.12:0/1297477879 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55aaad2fcd40 0x55aaad2ff130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:53.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.261+0000 7f328067a740 4 client.27911 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:10:53.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.261+0000 7f328067a740 1 -- 172.21.15.12:0/1297477879 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aaad259790 con 0x55aaad2fcd40 2024-03-23T23:10:53.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.261+0000 7f326fe00640 1 --2- 172.21.15.12:0/1297477879 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55aaad2fcd40 0x55aaad2ff130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:53.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.261+0000 7f326fe00640 1 --2- 172.21.15.12:0/1297477879 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55aaad2fcd40 0x55aaad2ff130 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f07150b6fd12c2ab server_cookie=955842d3d5b83204 in_seq=0 out_seq=0 2024-03-23T23:10:53.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.261+0000 7f3264c00640 10 client.27911 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:53.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.262+0000 7f3264c00640 1 -- 172.21.15.12:0/1297477879 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55aaad259790 con 0x55aaad2fcd40 2024-03-23T23:10:53.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.262+0000 7f3264c00640 10 client.27911 handle_command_reply: tid=0 2024-03-23T23:10:53.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.262+0000 7f328067a740 1 client.27911 shutdown 2024-03-23T23:10:53.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.262+0000 7f328067a740 1 -- 172.21.15.12:0/1297477879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2e7050 msgr2=0x55aaad2f1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:53.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.262+0000 7f328067a740 1 --2- 172.21.15.12:0/1297477879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2e7050 0x55aaad2f1080 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f32700115a0 tx=0x7f3270005bd0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.262+0000 7f328067a740 1 -- 172.21.15.12:0/1297477879 shutdown_connections 2024-03-23T23:10:53.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.262+0000 7f328067a740 1 --2- 172.21.15.12:0/1297477879 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55aaad2fcd40 0x55aaad2ff130 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.262+0000 7f328067a740 1 --2- 172.21.15.12:0/1297477879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aaad2ecbb0 0x55aaad2f8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.262+0000 7f328067a740 1 --2- 172.21.15.12:0/1297477879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55aaad2e7a40 0x55aaad2f15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.262+0000 7f328067a740 1 --2- 172.21.15.12:0/1297477879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2e7050 0x55aaad2f1080 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.262+0000 7f3264c00640 0 client.27911 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:53.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.262+0000 7f328067a740 1 -- 172.21.15.12:0/1297477879 >> 172.21.15.12:0/1297477879 conn(0x55aaad2af200 msgr2=0x55aaad2e1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:53.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.263+0000 7f328067a740 1 -- 172.21.15.12:0/1297477879 shutdown_connections 2024-03-23T23:10:53.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.263+0000 7f328067a740 1 -- 172.21.15.12:0/1297477879 wait complete. 2024-03-23T23:10:53.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.263+0000 7f328067a740 20 client.27911 trim_cache size 0 max 16384 2024-03-23T23:10:53.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.281+0000 7f328067a740 1 Processor -- start 2024-03-23T23:10:53.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.281+0000 7f328067a740 1 -- start start 2024-03-23T23:10:53.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.281+0000 7f328067a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aaad2e7050 0x55aaad2f13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:53.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.281+0000 7f328067a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55aaad2e7a40 0x55aaad2f85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:53.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.281+0000 7f328067a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2ecbb0 0x55aaad2e1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:53.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.281+0000 7f328067a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32781b12c0 con 0x55aaad2e7a40 2024-03-23T23:10:53.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.281+0000 7f328067a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32781aedc0 con 0x55aaad2ecbb0 2024-03-23T23:10:53.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.281+0000 7f328067a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32781ace60 con 0x55aaad2e7050 2024-03-23T23:10:53.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.282+0000 7f3277400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aaad2e7050 0x55aaad2f13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:53.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.282+0000 7f3277400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aaad2e7050 0x55aaad2f13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:54440/0 (socket says 172.21.15.12:54440) 2024-03-23T23:10:53.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.282+0000 7f3277400640 1 -- 172.21.15.12:0/626018752 learned_addr learned my addr 172.21.15.12:0/626018752 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:53.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.282+0000 7f3277400640 1 --2- 172.21.15.12:0/626018752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2ecbb0 0x55aaad2e1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:53.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.282+0000 7f3277400640 1 -- 172.21.15.12:0/626018752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2ecbb0 msgr2=0x55aaad2e1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:53.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.282+0000 7f3277400640 1 --2- 172.21.15.12:0/626018752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2ecbb0 0x55aaad2e1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.282+0000 7f3277400640 1 -- 172.21.15.12:0/626018752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55aaad2e7a40 msgr2=0x55aaad2f85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:53.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.282+0000 7f3277400640 1 --2- 172.21.15.12:0/626018752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55aaad2e7a40 0x55aaad2f85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.282+0000 7f3277400640 1 -- 172.21.15.12:0/626018752 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aaacf200f0 con 0x55aaad2e7050 2024-03-23T23:10:53.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.282+0000 7f3277400640 1 --2- 172.21.15.12:0/626018752 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aaad2e7050 0x55aaad2f13a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3270005d70 tx=0x7f3270011bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3803867c7ed7523e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:53.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.283+0000 7f3267400640 1 -- 172.21.15.12:0/626018752 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3270037040 con 0x55aaad2e7050 2024-03-23T23:10:53.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.283+0000 7f3267400640 1 -- 172.21.15.12:0/626018752 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3270005990 con 0x55aaad2e7050 2024-03-23T23:10:53.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.283+0000 7f3267400640 1 -- 172.21.15.12:0/626018752 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327000fa70 con 0x55aaad2e7050 2024-03-23T23:10:53.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.283+0000 7f328067a740 1 -- 172.21.15.12:0/626018752 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aaad2e7050 msgr2=0x55aaad2f13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:53.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.283+0000 7f328067a740 1 --2- 172.21.15.12:0/626018752 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aaad2e7050 0x55aaad2f13a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3270005d70 tx=0x7f3270011bf0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.283+0000 7f328067a740 1 -- 172.21.15.12:0/626018752 shutdown_connections 2024-03-23T23:10:53.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.283+0000 7f328067a740 1 --2- 172.21.15.12:0/626018752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2ecbb0 0x55aaad2e1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.283+0000 7f328067a740 1 --2- 172.21.15.12:0/626018752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55aaad2e7a40 0x55aaad2f85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.283+0000 7f328067a740 1 --2- 172.21.15.12:0/626018752 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aaad2e7050 0x55aaad2f13a0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.283+0000 7f328067a740 1 -- 172.21.15.12:0/626018752 >> 172.21.15.12:0/626018752 conn(0x55aaad2af200 msgr2=0x55aaad2f0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.283+0000 7f328067a740 1 -- 172.21.15.12:0/626018752 shutdown_connections 2024-03-23T23:10:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.283+0000 7f328067a740 1 -- 172.21.15.12:0/626018752 wait complete. 2024-03-23T23:10:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.284+0000 7f328067a740 1 -- start start 2024-03-23T23:10:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.284+0000 7f328067a740 1 Processor -- start 2024-03-23T23:10:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.285+0000 7f328067a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55aaad2e7050 0x55aaad2f5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.285+0000 7f328067a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2e7a40 0x55aaad2f5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.285+0000 7f328067a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aaad2ecbb0 0x55aaad3565d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.285+0000 7f328067a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32781ad460 con 0x55aaad2e7050 2024-03-23T23:10:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.285+0000 7f328067a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32781ae760 con 0x55aaad2e7a40 2024-03-23T23:10:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.285+0000 7f328067a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32781adaa0 con 0x55aaad2ecbb0 2024-03-23T23:10:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.285+0000 7f328067a740 10 client.0 _ll_register_callbacks cb 0x7f32680f6380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.285+0000 7f326fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2e7a40 0x55aaad2f5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.285+0000 7f326fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2e7a40 0x55aaad2f5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37890/0 (socket says 172.21.15.12:37890) 2024-03-23T23:10:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.285+0000 7f3277400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55aaad2e7050 0x55aaad2f5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.285+0000 7f326fe00640 1 -- 172.21.15.12:0/2634142303 learned_addr learned my addr 172.21.15.12:0/2634142303 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.285+0000 7f3277400640 1 --2- 172.21.15.12:0/2634142303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aaad2ecbb0 0x55aaad3565d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:53.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.285+0000 7f326fe00640 1 -- 172.21.15.12:0/2634142303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aaad2ecbb0 msgr2=0x55aaad3565d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:53.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.285+0000 7f326fe00640 1 --2- 172.21.15.12:0/2634142303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aaad2ecbb0 0x55aaad3565d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.285+0000 7f326fe00640 1 -- 172.21.15.12:0/2634142303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55aaad2e7050 msgr2=0x55aaad2f5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:53.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.285+0000 7f326fe00640 1 --2- 172.21.15.12:0/2634142303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55aaad2e7050 0x55aaad2f5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.285+0000 7f326fe00640 1 -- 172.21.15.12:0/2634142303 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aaad2f8780 con 0x55aaad2e7a40 2024-03-23T23:10:53.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.285+0000 7f3277400640 1 --2- 172.21.15.12:0/2634142303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55aaad2e7050 0x55aaad2f5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:10:53.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.286+0000 7f326fe00640 1 --2- 172.21.15.12:0/2634142303 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2e7a40 0x55aaad2f5750 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x55aaad2ff9c0 tx=0x7f3260022d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e5842314f64d45d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:53.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.286+0000 7f3264c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:10:53.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.286+0000 7f3264c00640 1 -- 172.21.15.12:0/2634142303 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3260012110 con 0x55aaad2e7a40 2024-03-23T23:10:53.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.286+0000 7f3264c00640 1 -- 172.21.15.12:0/2634142303 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f3260012a80 con 0x55aaad2e7a40 2024-03-23T23:10:53.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.286+0000 7f328067a740 1 -- 172.21.15.12:0/2634142303 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3270005990 con 0x55aaad2e7a40 2024-03-23T23:10:53.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.286+0000 7f3264c00640 1 -- 172.21.15.12:0/2634142303 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32600122b0 con 0x55aaad2e7a40 2024-03-23T23:10:53.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.286+0000 7f328067a740 10 client.27917 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:53.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.286+0000 7f328067a740 1 -- 172.21.15.12:0/2634142303 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55aaad357b40 con 0x55aaad2e7a40 2024-03-23T23:10:53.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.286+0000 7f3264c00640 1 -- 172.21.15.12:0/2634142303 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3270005990 con 0x55aaad2e7a40 2024-03-23T23:10:53.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.287+0000 7f3264c00640 1 -- 172.21.15.12:0/2634142303 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f326001f3f0 con 0x55aaad2e7a40 2024-03-23T23:10:53.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.287+0000 7f328067a740 10 client.27917 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:53.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.287+0000 7f328067a740 10 client.27917 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:53.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.287+0000 7f328067a740 20 client.27917 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:53.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.287+0000 7f328067a740 1 --2- 172.21.15.12:0/2634142303 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55aaad358520 0x55aaad35a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:53.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.287+0000 7f328067a740 4 client.27917 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:10:53.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.287+0000 7f328067a740 1 -- 172.21.15.12:0/2634142303 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55aaad253ae0 con 0x55aaad358520 2024-03-23T23:10:53.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.287+0000 7f3277400640 1 --2- 172.21.15.12:0/2634142303 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55aaad358520 0x55aaad35a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:53.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.287+0000 7f3277400640 1 --2- 172.21.15.12:0/2634142303 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55aaad358520 0x55aaad35a940 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7762d7ea97112bfd server_cookie=2d9e72a84479cea7 in_seq=0 out_seq=0 2024-03-23T23:10:53.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.287+0000 7f3264c00640 10 client.27917 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:53.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.288+0000 7f3264c00640 1 -- 172.21.15.12:0/2634142303 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55aaad253ae0 con 0x55aaad358520 2024-03-23T23:10:53.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.288+0000 7f3264c00640 10 client.27917 handle_command_reply: tid=0 2024-03-23T23:10:53.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.288+0000 7f328067a740 1 client.27917 shutdown 2024-03-23T23:10:53.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.288+0000 7f328067a740 1 -- 172.21.15.12:0/2634142303 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2e7a40 msgr2=0x55aaad2f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:53.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.288+0000 7f328067a740 1 --2- 172.21.15.12:0/2634142303 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2e7a40 0x55aaad2f5750 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x55aaad2ff9c0 tx=0x7f3260022d70 comp rx=0 tx=0).stop 2024-03-23T23:10:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.288+0000 7f328067a740 1 -- 172.21.15.12:0/2634142303 shutdown_connections 2024-03-23T23:10:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.288+0000 7f328067a740 1 --2- 172.21.15.12:0/2634142303 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55aaad358520 0x55aaad35a940 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.288+0000 7f328067a740 1 --2- 172.21.15.12:0/2634142303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aaad2ecbb0 0x55aaad3565d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.288+0000 7f328067a740 1 --2- 172.21.15.12:0/2634142303 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aaad2e7a40 0x55aaad2f5750 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.288+0000 7f328067a740 1 --2- 172.21.15.12:0/2634142303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55aaad2e7050 0x55aaad2f5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.288+0000 7f3264c00640 0 client.27917 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.288+0000 7f328067a740 1 -- 172.21.15.12:0/2634142303 >> 172.21.15.12:0/2634142303 conn(0x55aaad2af200 msgr2=0x55aaad2e84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.288+0000 7f328067a740 1 -- 172.21.15.12:0/2634142303 shutdown_connections 2024-03-23T23:10:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.288+0000 7f328067a740 1 -- 172.21.15.12:0/2634142303 wait complete. 2024-03-23T23:10:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.288+0000 7f328067a740 20 client.27917 trim_cache size 0 max 16384 2024-03-23T23:10:53.291 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:10:53.291 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T23:10:53.291 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-03-23T23:10:53.292 INFO:teuthology.orchestra.run.smithi012.stdout: "030dcd17-08a2-4ba6-a094-91ebd5061e5b": { 2024-03-23T23:10:53.292 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-03-23T23:10:53.292 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "030dcd17-08a2-4ba6-a094-91ebd5061e5b", 2024-03-23T23:10:53.292 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-03-23T23:10:53.292 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:10:53.292 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:10:53.292 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:10:53.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.289+0000 7f3277e00640 1 -- 172.21.15.12:0/224141029 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f325c0a2880 msgr2=0x7f325c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:53.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.289+0000 7f3277e00640 1 --2- 172.21.15.12:0/224141029 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f325c0a2880 0x7f325c0a4d40 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f3268005e90 tx=0x7f32680dd040 comp rx=0 tx=0).stop 2024-03-23T23:10:53.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.289+0000 7f3277e00640 1 -- 172.21.15.12:0/224141029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3278107bc0 msgr2=0x7f327819df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:53.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.289+0000 7f3277e00640 1 --2- 172.21.15.12:0/224141029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3278107bc0 0x7f327819df30 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3268000c00 tx=0x7f326802c650 comp rx=0 tx=0).stop 2024-03-23T23:10:53.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.289+0000 7f3277e00640 1 -- 172.21.15.12:0/224141029 shutdown_connections 2024-03-23T23:10:53.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.289+0000 7f3277e00640 1 --2- 172.21.15.12:0/224141029 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f325c0a2880 0x7f325c0a4d40 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.289+0000 7f3277e00640 1 --2- 172.21.15.12:0/224141029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3278109770 0x7f32781a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.289+0000 7f3277e00640 1 --2- 172.21.15.12:0/224141029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3278108db0 0x7f327819e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.289+0000 7f3277e00640 1 --2- 172.21.15.12:0/224141029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3278107bc0 0x7f327819df30 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:53.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.289+0000 7f3277e00640 1 -- 172.21.15.12:0/224141029 >> 172.21.15.12:0/224141029 conn(0x7f3278076b70 msgr2=0x7f327810c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:53.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.289+0000 7f3277e00640 1 -- 172.21.15.12:0/224141029 shutdown_connections 2024-03-23T23:10:53.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:10:53.289+0000 7f3277e00640 1 -- 172.21.15.12:0/224141029 wait complete. 2024-03-23T23:10:53.309 INFO:tasks.cephfs.filesystem:scrub status for tag:030dcd17-08a2-4ba6-a094-91ebd5061e5b - {'path': '/', 'tag': '030dcd17-08a2-4ba6-a094-91ebd5061e5b', 'options': 'recursive,force'} 2024-03-23T23:10:55.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:54 smithi012 ceph-mon[31695]: scrub queued for path: / 2024-03-23T23:10:55.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:54 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:10:55.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:54 smithi012 ceph-mon[31695]: scrub summary: active paths [/] 2024-03-23T23:10:55.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:54 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:10:55.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:54 smithi012 ceph-mon[31695]: pgmap v663: 97 pgs: 97 active+clean; 2.0 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:10:55.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:54 smithi173 ceph-mon[36116]: scrub queued for path: / 2024-03-23T23:10:55.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:54 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:10:55.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:54 smithi173 ceph-mon[36116]: scrub summary: active paths [/] 2024-03-23T23:10:55.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:54 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:10:55.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:54 smithi173 ceph-mon[36116]: pgmap v663: 97 pgs: 97 active+clean; 2.0 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:10:55.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:54 smithi203 ceph-mon[36693]: scrub queued for path: / 2024-03-23T23:10:55.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:54 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:10:55.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:54 smithi203 ceph-mon[36693]: scrub summary: active paths [/] 2024-03-23T23:10:55.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:54 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:10:55.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:54 smithi203 ceph-mon[36693]: pgmap v663: 97 pgs: 97 active+clean; 2.0 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:10:57.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:56 smithi012 ceph-mon[31695]: pgmap v664: 97 pgs: 97 active+clean; 2.1 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:10:57.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:56 smithi012 ceph-mon[31695]: scrub summary: idle 2024-03-23T23:10:57.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:56 smithi173 ceph-mon[36116]: pgmap v664: 97 pgs: 97 active+clean; 2.1 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:10:57.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:56 smithi173 ceph-mon[36116]: scrub summary: idle 2024-03-23T23:10:57.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:56 smithi203 ceph-mon[36693]: pgmap v664: 97 pgs: 97 active+clean; 2.1 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:10:57.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:56 smithi203 ceph-mon[36693]: scrub summary: idle 2024-03-23T23:10:59.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:10:58 smithi012 ceph-mon[31695]: pgmap v665: 97 pgs: 97 active+clean; 2.1 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:10:59.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:10:58 smithi173 ceph-mon[36116]: pgmap v665: 97 pgs: 97 active+clean; 2.1 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:10:59.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:10:58 smithi203 ceph-mon[36693]: pgmap v665: 97 pgs: 97 active+clean; 2.1 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:10:59.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.529+0000 7f1f17e00640 1 -- 172.21.15.12:0/3241110376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f10103900 msgr2=0x7f1f10103d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:59.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.529+0000 7f1f17e00640 1 --2- 172.21.15.12:0/3241110376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f10103900 0x7f1f10103d20 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f1f04009900 tx=0x7f1f0402bc30 comp rx=0 tx=0).stop 2024-03-23T23:10:59.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.529+0000 7f1f17e00640 1 -- 172.21.15.12:0/3241110376 shutdown_connections 2024-03-23T23:10:59.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.529+0000 7f1f17e00640 1 --2- 172.21.15.12:0/3241110376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f101054b0 0x7f1f1010bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.529+0000 7f1f17e00640 1 --2- 172.21.15.12:0/3241110376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f10104af0 0x7f1f10104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.529+0000 7f1f17e00640 1 --2- 172.21.15.12:0/3241110376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f10103900 0x7f1f10103d20 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.529+0000 7f1f17e00640 1 -- 172.21.15.12:0/3241110376 >> 172.21.15.12:0/3241110376 conn(0x7f1f100fee90 msgr2=0x7f1f101012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:59.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.529+0000 7f1f17e00640 1 -- 172.21.15.12:0/3241110376 shutdown_connections 2024-03-23T23:10:59.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.529+0000 7f1f17e00640 1 -- 172.21.15.12:0/3241110376 wait complete. 2024-03-23T23:10:59.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.530+0000 7f1f17e00640 1 Processor -- start 2024-03-23T23:10:59.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.531+0000 7f1f17e00640 1 -- start start 2024-03-23T23:10:59.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.531+0000 7f1f17e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f10103900 0x7f1f10199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:59.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.531+0000 7f1f17e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f10104af0 0x7f1f1019a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:59.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.531+0000 7f1f17e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f101054b0 0x7f1f101a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:59.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.531+0000 7f1f17e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1010f5a0 con 0x7f1f10103900 2024-03-23T23:10:59.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.531+0000 7f1f17e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1010f420 con 0x7f1f10104af0 2024-03-23T23:10:59.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.531+0000 7f1f17e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1010f720 con 0x7f1f101054b0 2024-03-23T23:10:59.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.531+0000 7f1f15400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f10103900 0x7f1f10199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:59.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.531+0000 7f1f15e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f101054b0 0x7f1f101a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:59.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.531+0000 7f1f14a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f10104af0 0x7f1f1019a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:59.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.531+0000 7f1f15400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f10103900 0x7f1f10199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48594/0 (socket says 172.21.15.12:48594) 2024-03-23T23:10:59.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.531+0000 7f1f15400640 1 -- 172.21.15.12:0/4090363653 learned_addr learned my addr 172.21.15.12:0/4090363653 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:59.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.531+0000 7f1f15e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f101054b0 0x7f1f101a1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:37526/0 (socket says 172.21.15.12:37526) 2024-03-23T23:10:59.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.532+0000 7f1f15e00640 1 -- 172.21.15.12:0/4090363653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f10104af0 msgr2=0x7f1f1019a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:59.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.532+0000 7f1f15e00640 1 --2- 172.21.15.12:0/4090363653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f10104af0 0x7f1f1019a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.532+0000 7f1f15e00640 1 -- 172.21.15.12:0/4090363653 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f10103900 msgr2=0x7f1f10199b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:59.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.532+0000 7f1f15e00640 1 --2- 172.21.15.12:0/4090363653 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f10103900 0x7f1f10199b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.532+0000 7f1f15e00640 1 -- 172.21.15.12:0/4090363653 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f101a1770 con 0x7f1f101054b0 2024-03-23T23:10:59.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.532+0000 7f1f15e00640 1 --2- 172.21.15.12:0/4090363653 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f101054b0 0x7f1f101a1100 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f1f08009fd0 tx=0x7f1f0800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:59.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.532+0000 7f1f0e000640 1 -- 172.21.15.12:0/4090363653 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0800ce20 con 0x7f1f101054b0 2024-03-23T23:10:59.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.532+0000 7f1f0e000640 1 -- 172.21.15.12:0/4090363653 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f08005020 con 0x7f1f101054b0 2024-03-23T23:10:59.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.532+0000 7f1f0e000640 1 -- 172.21.15.12:0/4090363653 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f08010720 con 0x7f1f101054b0 2024-03-23T23:10:59.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.532+0000 7f1f17e00640 1 -- 172.21.15.12:0/4090363653 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f101a1a60 con 0x7f1f101054b0 2024-03-23T23:10:59.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.533+0000 7f1f17e00640 1 -- 172.21.15.12:0/4090363653 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f101a1f60 con 0x7f1f101054b0 2024-03-23T23:10:59.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.536+0000 7f1f0e000640 1 -- 172.21.15.12:0/4090363653 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1f08005540 con 0x7f1f101054b0 2024-03-23T23:10:59.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.536+0000 7f1f1aa6a740 1 Processor -- start 2024-03-23T23:10:59.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.536+0000 7f1f1aa6a740 1 -- start start 2024-03-23T23:10:59.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.536+0000 7f1f1aa6a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5cf4171e0 0x55d5cf417600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:59.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.536+0000 7f1f1aa6a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf417bd0 0x55d5cf41c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:59.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.536+0000 7f1f1aa6a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5cf41cd40 0x55d5cf41f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:59.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.536+0000 7f1f1aa6a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5cf3da3b0 con 0x55d5cf4171e0 2024-03-23T23:10:59.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.536+0000 7f1f1aa6a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5cf37c830 con 0x55d5cf417bd0 2024-03-23T23:10:59.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.536+0000 7f1f1aa6a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5cf3255d0 con 0x55d5cf41cd40 2024-03-23T23:10:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.537+0000 7f1f15400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf417bd0 0x55d5cf41c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.537+0000 7f1f14a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5cf4171e0 0x55d5cf417600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.537+0000 7f1f15400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf417bd0 0x55d5cf41c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:39318/0 (socket says 172.21.15.12:39318) 2024-03-23T23:10:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.537+0000 7f1f15400640 1 -- 172.21.15.12:0/1237084073 learned_addr learned my addr 172.21.15.12:0/1237084073 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.537+0000 7f1f14a00640 1 --2- 172.21.15.12:0/1237084073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5cf41cd40 0x55d5cf41f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.537+0000 7f1f15400640 1 -- 172.21.15.12:0/1237084073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5cf41cd40 msgr2=0x55d5cf41f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.537+0000 7f1f15400640 1 --2- 172.21.15.12:0/1237084073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5cf41cd40 0x55d5cf41f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.537+0000 7f1f15400640 1 -- 172.21.15.12:0/1237084073 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5cf4171e0 msgr2=0x55d5cf417600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.537+0000 7f1f15400640 1 --2- 172.21.15.12:0/1237084073 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5cf4171e0 0x55d5cf417600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.537+0000 7f1f15400640 1 -- 172.21.15.12:0/1237084073 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d5cf33d2c0 con 0x55d5cf417bd0 2024-03-23T23:10:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.537+0000 7f1f0e000640 1 --2- 172.21.15.12:0/4090363653 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1ef80a2920 0x7f1ef80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.537+0000 7f1f15400640 1 --2- 172.21.15.12:0/1237084073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf417bd0 0x55d5cf41c6d0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f1f0402c170 tx=0x7f1f040095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba13d535efd90fab server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.538+0000 7f1f0e000640 1 -- 172.21.15.12:0/4090363653 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1f080cf140 con 0x7f1f101054b0 2024-03-23T23:10:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.538+0000 7f1f14a00640 1 --2- 172.21.15.12:0/4090363653 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1ef80a2920 0x7f1ef80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:59.539 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.538+0000 7f1f02600640 1 -- 172.21.15.12:0/1237084073 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f04045070 con 0x55d5cf417bd0 2024-03-23T23:10:59.539 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.538+0000 7f1f02600640 1 -- 172.21.15.12:0/1237084073 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f04004040 con 0x55d5cf417bd0 2024-03-23T23:10:59.539 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.538+0000 7f1f02600640 1 -- 172.21.15.12:0/1237084073 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f04004ef0 con 0x55d5cf417bd0 2024-03-23T23:10:59.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.538+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/1237084073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf417bd0 msgr2=0x55d5cf41c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:59.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.538+0000 7f1f14a00640 1 --2- 172.21.15.12:0/4090363653 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1ef80a2920 0x7f1ef80a4de0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x55d5cf418430 tx=0x7f1efc005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:59.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.538+0000 7f1f1aa6a740 1 --2- 172.21.15.12:0/1237084073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf417bd0 0x55d5cf41c6d0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f1f0402c170 tx=0x7f1f040095e0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.539+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/1237084073 shutdown_connections 2024-03-23T23:10:59.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.539+0000 7f1f1aa6a740 1 --2- 172.21.15.12:0/1237084073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5cf41cd40 0x55d5cf41f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.539+0000 7f1f1aa6a740 1 --2- 172.21.15.12:0/1237084073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf417bd0 0x55d5cf41c6d0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.539+0000 7f1f1aa6a740 1 --2- 172.21.15.12:0/1237084073 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5cf4171e0 0x55d5cf417600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.539+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/1237084073 >> 172.21.15.12:0/1237084073 conn(0x55d5cf412750 msgr2=0x55d5cf412b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:59.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.539+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/1237084073 shutdown_connections 2024-03-23T23:10:59.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.539+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/1237084073 wait complete. 2024-03-23T23:10:59.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.539+0000 7f1f1aa6a740 1 -- start start 2024-03-23T23:10:59.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.539+0000 7f1f1aa6a740 1 Processor -- start 2024-03-23T23:10:59.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.540+0000 7f1f1aa6a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf4171e0 0x55d5cf421300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:59.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.540+0000 7f1f1aa6a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5cf417bd0 0x55d5cf421840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:59.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.540+0000 7f1f1aa6a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5cf41cd40 0x55d5cf428880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:59.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.540+0000 7f1f1aa6a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5cf338a20 con 0x55d5cf41cd40 2024-03-23T23:10:59.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.540+0000 7f1f1aa6a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5cf3788d0 con 0x55d5cf4171e0 2024-03-23T23:10:59.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.540+0000 7f1f1aa6a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5cf41fc90 con 0x55d5cf417bd0 2024-03-23T23:10:59.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.540+0000 7f1f1aa6a740 10 client.0 _ll_register_callbacks cb 0x55d5cf3751c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:59.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.540+0000 7f1f15400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf4171e0 0x55d5cf421300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:59.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.540+0000 7f1f14a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5cf417bd0 0x55d5cf421840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:59.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.540+0000 7f1f15400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf4171e0 0x55d5cf421300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:39328/0 (socket says 172.21.15.12:39328) 2024-03-23T23:10:59.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.540+0000 7f1f15400640 1 -- 172.21.15.12:0/1586216941 learned_addr learned my addr 172.21.15.12:0/1586216941 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:59.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.540+0000 7f1f15e00640 1 --2- 172.21.15.12:0/1586216941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5cf41cd40 0x55d5cf428880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:59.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.541+0000 7f1f15400640 1 -- 172.21.15.12:0/1586216941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5cf417bd0 msgr2=0x55d5cf421840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:59.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.541+0000 7f1f15400640 1 --2- 172.21.15.12:0/1586216941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5cf417bd0 0x55d5cf421840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.541+0000 7f1f15400640 1 -- 172.21.15.12:0/1586216941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5cf41cd40 msgr2=0x55d5cf428880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:59.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.541+0000 7f1f15400640 1 --2- 172.21.15.12:0/1586216941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5cf41cd40 0x55d5cf428880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.541+0000 7f1f15400640 1 -- 172.21.15.12:0/1586216941 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d5cf091210 con 0x55d5cf4171e0 2024-03-23T23:10:59.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.541+0000 7f1f15e00640 1 --2- 172.21.15.12:0/1586216941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5cf41cd40 0x55d5cf428880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:10:59.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.541+0000 7f1f15400640 1 --2- 172.21.15.12:0/1586216941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf4171e0 0x55d5cf421300 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f1f04005ab0 tx=0x7f1f0402bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32c46ef131912fe server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:59.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.541+0000 7f1ef6a00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:10:59.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.541+0000 7f1ef6a00640 1 -- 172.21.15.12:0/1586216941 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f04051070 con 0x55d5cf4171e0 2024-03-23T23:10:59.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.541+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/1586216941 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1f04004040 con 0x55d5cf4171e0 2024-03-23T23:10:59.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.541+0000 7f1ef6a00640 1 -- 172.21.15.12:0/1586216941 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1f04004ef0 con 0x55d5cf4171e0 2024-03-23T23:10:59.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.541+0000 7f1ef6a00640 1 -- 172.21.15.12:0/1586216941 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f04045040 con 0x55d5cf4171e0 2024-03-23T23:10:59.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.542+0000 7f1f1aa6a740 10 client.27929 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:59.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.542+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/1586216941 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55d5cf15d620 con 0x55d5cf4171e0 2024-03-23T23:10:59.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.542+0000 7f1ef6a00640 1 -- 172.21.15.12:0/1586216941 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f04004040 con 0x55d5cf4171e0 2024-03-23T23:10:59.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.542+0000 7f1ef6a00640 1 -- 172.21.15.12:0/1586216941 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f1f04054030 con 0x55d5cf4171e0 2024-03-23T23:10:59.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.542+0000 7f1f1aa6a740 10 client.27929 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:59.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.542+0000 7f1f1aa6a740 10 client.27929 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:59.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.542+0000 7f1f1aa6a740 20 client.27929 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:59.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.542+0000 7f1f1aa6a740 1 --2- 172.21.15.12:0/1586216941 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d5cf42cd70 0x55d5cf42f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:59.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.542+0000 7f1f1aa6a740 4 client.27929 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:10:59.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.542+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/1586216941 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d5cf389d70 con 0x55d5cf42cd70 2024-03-23T23:10:59.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.542+0000 7f1f14a00640 1 --2- 172.21.15.12:0/1586216941 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d5cf42cd70 0x55d5cf42f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:59.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.543+0000 7f1f14a00640 1 --2- 172.21.15.12:0/1586216941 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d5cf42cd70 0x55d5cf42f160 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d841e56a3b753e9a server_cookie=19a009614d0ff69f in_seq=0 out_seq=0 2024-03-23T23:10:59.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.543+0000 7f1ef6a00640 10 client.27929 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:59.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.544+0000 7f1ef6a00640 1 -- 172.21.15.12:0/1586216941 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d5cf389d70 con 0x55d5cf42cd70 2024-03-23T23:10:59.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.544+0000 7f1ef6a00640 10 client.27929 handle_command_reply: tid=0 2024-03-23T23:10:59.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.544+0000 7f1f1aa6a740 1 client.27929 shutdown 2024-03-23T23:10:59.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.544+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/1586216941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf4171e0 msgr2=0x55d5cf421300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:59.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.544+0000 7f1f1aa6a740 1 --2- 172.21.15.12:0/1586216941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf4171e0 0x55d5cf421300 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f1f04005ab0 tx=0x7f1f0402bc30 comp rx=0 tx=0).stop 2024-03-23T23:10:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.544+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/1586216941 shutdown_connections 2024-03-23T23:10:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.544+0000 7f1f1aa6a740 1 --2- 172.21.15.12:0/1586216941 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d5cf42cd70 0x55d5cf42f160 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.544+0000 7f1f1aa6a740 1 --2- 172.21.15.12:0/1586216941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5cf41cd40 0x55d5cf428880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.544+0000 7f1f1aa6a740 1 --2- 172.21.15.12:0/1586216941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5cf417bd0 0x55d5cf421840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.544+0000 7f1f1aa6a740 1 --2- 172.21.15.12:0/1586216941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf4171e0 0x55d5cf421300 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.544+0000 7f1ef6a00640 0 client.27929 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.544+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/1586216941 >> 172.21.15.12:0/1586216941 conn(0x55d5cf412750 msgr2=0x55d5cf411950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.544+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/1586216941 shutdown_connections 2024-03-23T23:10:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.544+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/1586216941 wait complete. 2024-03-23T23:10:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.544+0000 7f1f1aa6a740 20 client.27929 trim_cache size 0 max 16384 2024-03-23T23:10:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.560+0000 7f1f1aa6a740 1 Processor -- start 2024-03-23T23:10:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.560+0000 7f1f1aa6a740 1 -- start start 2024-03-23T23:10:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.560+0000 7f1f1aa6a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5cf4171e0 0x55d5cf40d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.560+0000 7f1f1aa6a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5cf417bd0 0x55d5cf4117d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.560+0000 7f1f1aa6a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf41cd40 0x55d5cf411d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.560+0000 7f1f1aa6a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f101acff0 con 0x55d5cf4171e0 2024-03-23T23:10:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.560+0000 7f1f1aa6a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f101aaaf0 con 0x55d5cf41cd40 2024-03-23T23:10:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.560+0000 7f1f1aa6a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f101a8b90 con 0x55d5cf417bd0 2024-03-23T23:10:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.560+0000 7f1f15400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5cf4171e0 0x55d5cf40d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:59.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.560+0000 7f1f15e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf41cd40 0x55d5cf411d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:59.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.560+0000 7f1f14a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5cf417bd0 0x55d5cf4117d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:59.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.560+0000 7f1f15e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf41cd40 0x55d5cf411d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:39342/0 (socket says 172.21.15.12:39342) 2024-03-23T23:10:59.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.560+0000 7f1f15e00640 1 -- 172.21.15.12:0/2570646377 learned_addr learned my addr 172.21.15.12:0/2570646377 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:59.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.561+0000 7f1f15e00640 1 -- 172.21.15.12:0/2570646377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5cf417bd0 msgr2=0x55d5cf4117d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:59.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.561+0000 7f1f15e00640 1 --2- 172.21.15.12:0/2570646377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5cf417bd0 0x55d5cf4117d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.561+0000 7f1f15e00640 1 -- 172.21.15.12:0/2570646377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5cf4171e0 msgr2=0x55d5cf40d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:59.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.561+0000 7f1f15e00640 1 --2- 172.21.15.12:0/2570646377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5cf4171e0 0x55d5cf40d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.561+0000 7f1f15e00640 1 -- 172.21.15.12:0/2570646377 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d5cf42f490 con 0x55d5cf41cd40 2024-03-23T23:10:59.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.561+0000 7f1f15e00640 1 --2- 172.21.15.12:0/2570646377 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf41cd40 0x55d5cf411d10 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f1f0809b790 tx=0x7f1f080054b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f9daaa4b1e79e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:59.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.561+0000 7f1f01c00640 1 -- 172.21.15.12:0/2570646377 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f080b9070 con 0x55d5cf41cd40 2024-03-23T23:10:59.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.561+0000 7f1f01c00640 1 -- 172.21.15.12:0/2570646377 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f080a4230 con 0x55d5cf41cd40 2024-03-23T23:10:59.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.561+0000 7f1f01c00640 1 -- 172.21.15.12:0/2570646377 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f080b4b50 con 0x55d5cf41cd40 2024-03-23T23:10:59.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.562+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/2570646377 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf41cd40 msgr2=0x55d5cf411d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:59.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.562+0000 7f1f1aa6a740 1 --2- 172.21.15.12:0/2570646377 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf41cd40 0x55d5cf411d10 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f1f0809b790 tx=0x7f1f080054b0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.562+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/2570646377 shutdown_connections 2024-03-23T23:10:59.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.562+0000 7f1f1aa6a740 1 --2- 172.21.15.12:0/2570646377 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf41cd40 0x55d5cf411d10 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.562+0000 7f1f1aa6a740 1 --2- 172.21.15.12:0/2570646377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5cf417bd0 0x55d5cf4117d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.562+0000 7f1f1aa6a740 1 --2- 172.21.15.12:0/2570646377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5cf4171e0 0x55d5cf40d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.562+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/2570646377 >> 172.21.15.12:0/2570646377 conn(0x55d5cf412750 msgr2=0x55d5cf4110c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:59.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.562+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/2570646377 shutdown_connections 2024-03-23T23:10:59.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.562+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/2570646377 wait complete. 2024-03-23T23:10:59.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.562+0000 7f1f1aa6a740 1 -- start start 2024-03-23T23:10:59.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.563+0000 7f1f1aa6a740 1 Processor -- start 2024-03-23T23:10:59.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.563+0000 7f1f1aa6a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5cf4171e0 0x55d5cf427b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:59.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.563+0000 7f1f1aa6a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf417bd0 0x55d5cf4261f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:59.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.563+0000 7f1f1aa6a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5cf41cd40 0x55d5cf426730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:59.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.563+0000 7f1f1aa6a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f101a9190 con 0x55d5cf4171e0 2024-03-23T23:10:59.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.563+0000 7f1f1aa6a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f101aa490 con 0x55d5cf417bd0 2024-03-23T23:10:59.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.563+0000 7f1f1aa6a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f101a97d0 con 0x55d5cf41cd40 2024-03-23T23:10:59.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.563+0000 7f1f1aa6a740 10 client.0 _ll_register_callbacks cb 0x7f1efc00ba70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:10:59.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.563+0000 7f1f14a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf417bd0 0x55d5cf4261f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:59.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.563+0000 7f1f15e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5cf41cd40 0x55d5cf426730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:59.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.563+0000 7f1f14a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf417bd0 0x55d5cf4261f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:39346/0 (socket says 172.21.15.12:39346) 2024-03-23T23:10:59.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.563+0000 7f1f14a00640 1 -- 172.21.15.12:0/1451191897 learned_addr learned my addr 172.21.15.12:0/1451191897 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:10:59.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.564+0000 7f1f15400640 1 --2- 172.21.15.12:0/1451191897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5cf4171e0 0x55d5cf427b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:59.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.564+0000 7f1f14a00640 1 -- 172.21.15.12:0/1451191897 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5cf41cd40 msgr2=0x55d5cf426730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:59.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.564+0000 7f1f14a00640 1 --2- 172.21.15.12:0/1451191897 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5cf41cd40 0x55d5cf426730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.564+0000 7f1f14a00640 1 -- 172.21.15.12:0/1451191897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5cf4171e0 msgr2=0x55d5cf427b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:10:59.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.564+0000 7f1f14a00640 1 --2- 172.21.15.12:0/1451191897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5cf4171e0 0x55d5cf427b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.564+0000 7f1f14a00640 1 -- 172.21.15.12:0/1451191897 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d5cf426c70 con 0x55d5cf417bd0 2024-03-23T23:10:59.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.564+0000 7f1f15400640 1 --2- 172.21.15.12:0/1451191897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5cf4171e0 0x55d5cf427b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:10:59.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.564+0000 7f1f15e00640 1 --2- 172.21.15.12:0/1451191897 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5cf41cd40 0x55d5cf426730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:10:59.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.564+0000 7f1f14a00640 1 --2- 172.21.15.12:0/1451191897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf417bd0 0x55d5cf4261f0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f1efc0123c0 tx=0x7f1efc00b390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c31c27c34e7f9ffd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:10:59.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.564+0000 7f1ef6a00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:10:59.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.564+0000 7f1ef6a00640 1 -- 172.21.15.12:0/1451191897 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1efc02e300 con 0x55d5cf417bd0 2024-03-23T23:10:59.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.564+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/1451191897 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1f080a4230 con 0x55d5cf417bd0 2024-03-23T23:10:59.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.564+0000 7f1ef6a00640 1 -- 172.21.15.12:0/1451191897 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1efc01b030 con 0x55d5cf417bd0 2024-03-23T23:10:59.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.565+0000 7f1ef6a00640 1 -- 172.21.15.12:0/1451191897 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1efc005020 con 0x55d5cf417bd0 2024-03-23T23:10:59.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.565+0000 7f1f1aa6a740 10 client.27935 fetch_fsmap learned FSMap version 23 2024-03-23T23:10:59.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.565+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/1451191897 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55d5cf4892e0 con 0x55d5cf417bd0 2024-03-23T23:10:59.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.565+0000 7f1ef6a00640 1 -- 172.21.15.12:0/1451191897 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f080a4230 con 0x55d5cf417bd0 2024-03-23T23:10:59.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.565+0000 7f1ef6a00640 1 -- 172.21.15.12:0/1451191897 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f1efc0124e0 con 0x55d5cf417bd0 2024-03-23T23:10:59.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.565+0000 7f1f1aa6a740 10 client.27935 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:10:59.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.565+0000 7f1f1aa6a740 10 client.27935 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:10:59.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.565+0000 7f1f1aa6a740 20 client.27935 populate_metadata read hostname 'smithi012' 2024-03-23T23:10:59.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.565+0000 7f1f1aa6a740 1 --2- 172.21.15.12:0/1451191897 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d5cf489c70 0x55d5cf48c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:10:59.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.565+0000 7f1f1aa6a740 4 client.27935 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:10:59.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.565+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/1451191897 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d5cf3840c0 con 0x55d5cf489c70 2024-03-23T23:10:59.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.565+0000 7f1f15400640 1 --2- 172.21.15.12:0/1451191897 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d5cf489c70 0x55d5cf48c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:10:59.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.566+0000 7f1f15400640 1 --2- 172.21.15.12:0/1451191897 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d5cf489c70 0x55d5cf48c090 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e524ed102206e3a server_cookie=8760987f0ff7681e in_seq=0 out_seq=0 2024-03-23T23:10:59.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.566+0000 7f1ef6a00640 10 client.27935 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:59.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.578+0000 7f1ef6a00640 1 -- 172.21.15.12:0/1451191897 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d5cf3840c0 con 0x55d5cf489c70 2024-03-23T23:10:59.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.578+0000 7f1ef6a00640 10 client.27935 handle_command_reply: tid=0 2024-03-23T23:10:59.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.578+0000 7f1f1aa6a740 1 client.27935 shutdown 2024-03-23T23:10:59.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.578+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/1451191897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf417bd0 msgr2=0x55d5cf4261f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:59.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.578+0000 7f1f1aa6a740 1 --2- 172.21.15.12:0/1451191897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf417bd0 0x55d5cf4261f0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f1efc0123c0 tx=0x7f1efc00b390 comp rx=0 tx=0).stop 2024-03-23T23:10:59.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.579+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/1451191897 shutdown_connections 2024-03-23T23:10:59.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.579+0000 7f1f1aa6a740 1 --2- 172.21.15.12:0/1451191897 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55d5cf489c70 0x55d5cf48c090 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.579+0000 7f1f1aa6a740 1 --2- 172.21.15.12:0/1451191897 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5cf41cd40 0x55d5cf426730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.579+0000 7f1f1aa6a740 1 --2- 172.21.15.12:0/1451191897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cf417bd0 0x55d5cf4261f0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.579+0000 7f1f1aa6a740 1 --2- 172.21.15.12:0/1451191897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d5cf4171e0 0x55d5cf427b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.579+0000 7f1ef6a00640 0 client.27935 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:10:59.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.579+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/1451191897 >> 172.21.15.12:0/1451191897 conn(0x55d5cf412750 msgr2=0x55d5cf411950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:59.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.579+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/1451191897 shutdown_connections 2024-03-23T23:10:59.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.579+0000 7f1f1aa6a740 1 -- 172.21.15.12:0/1451191897 wait complete. 2024-03-23T23:10:59.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.579+0000 7f1f1aa6a740 20 client.27935 trim_cache size 0 max 16384 2024-03-23T23:10:59.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:10:59.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:10:59.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:10:59.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:10:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.580+0000 7f1f17e00640 1 -- 172.21.15.12:0/4090363653 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1ef80a2920 msgr2=0x7f1ef80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.580+0000 7f1f17e00640 1 --2- 172.21.15.12:0/4090363653 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1ef80a2920 0x7f1ef80a4de0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x55d5cf418430 tx=0x7f1efc005c90 comp rx=0 tx=0).stop 2024-03-23T23:10:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.580+0000 7f1f17e00640 1 -- 172.21.15.12:0/4090363653 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f101054b0 msgr2=0x7f1f101a1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:10:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.580+0000 7f1f17e00640 1 --2- 172.21.15.12:0/4090363653 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f101054b0 0x7f1f101a1100 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f1f08009fd0 tx=0x7f1f0800ef10 comp rx=0 tx=0).stop 2024-03-23T23:10:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.580+0000 7f1f17e00640 1 -- 172.21.15.12:0/4090363653 shutdown_connections 2024-03-23T23:10:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.580+0000 7f1f17e00640 1 --2- 172.21.15.12:0/4090363653 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1ef80a2920 0x7f1ef80a4de0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.580+0000 7f1f17e00640 1 --2- 172.21.15.12:0/4090363653 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f101054b0 0x7f1f101a1100 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.580+0000 7f1f17e00640 1 --2- 172.21.15.12:0/4090363653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f10104af0 0x7f1f1019a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.580+0000 7f1f17e00640 1 --2- 172.21.15.12:0/4090363653 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f10103900 0x7f1f10199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:10:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.580+0000 7f1f17e00640 1 -- 172.21.15.12:0/4090363653 >> 172.21.15.12:0/4090363653 conn(0x7f1f100fee90 msgr2=0x7f1f10107d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:10:59.582 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.580+0000 7f1f17e00640 1 -- 172.21.15.12:0/4090363653 shutdown_connections 2024-03-23T23:10:59.582 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:10:59.580+0000 7f1f17e00640 1 -- 172.21.15.12:0/4090363653 wait complete. 2024-03-23T23:11:01.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:00 smithi012 ceph-mon[31695]: pgmap v666: 97 pgs: 97 active+clean; 2.1 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:11:01.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:00 smithi173 ceph-mon[36116]: pgmap v666: 97 pgs: 97 active+clean; 2.1 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:11:01.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:00 smithi203 ceph-mon[36693]: pgmap v666: 97 pgs: 97 active+clean; 2.1 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:11:03.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:02 smithi012 ceph-mon[31695]: pgmap v667: 97 pgs: 97 active+clean; 2.2 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:11:03.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:02 smithi173 ceph-mon[36116]: pgmap v667: 97 pgs: 97 active+clean; 2.2 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:11:03.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:02 smithi203 ceph-mon[36693]: pgmap v667: 97 pgs: 97 active+clean; 2.2 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:11:04.959 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:04 smithi203 ceph-mon[36693]: pgmap v668: 97 pgs: 97 active+clean; 2.2 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:11:05.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:04 smithi012 ceph-mon[31695]: pgmap v668: 97 pgs: 97 active+clean; 2.2 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:11:05.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:04 smithi173 ceph-mon[36116]: pgmap v668: 97 pgs: 97 active+clean; 2.2 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:11:07.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:06 smithi012 ceph-mon[31695]: pgmap v669: 97 pgs: 97 active+clean; 2.2 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:11:07.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:06 smithi173 ceph-mon[36116]: pgmap v669: 97 pgs: 97 active+clean; 2.2 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:11:07.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:06 smithi203 ceph-mon[36693]: pgmap v669: 97 pgs: 97 active+clean; 2.2 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:11:09.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:08 smithi012 ceph-mon[31695]: pgmap v670: 97 pgs: 97 active+clean; 2.2 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:11:09.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:08 smithi173 ceph-mon[36116]: pgmap v670: 97 pgs: 97 active+clean; 2.2 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:11:09.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:08 smithi203 ceph-mon[36693]: pgmap v670: 97 pgs: 97 active+clean; 2.2 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:11:11.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:10 smithi012 ceph-mon[31695]: pgmap v671: 97 pgs: 97 active+clean; 2.2 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:11:11.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:10 smithi173 ceph-mon[36116]: pgmap v671: 97 pgs: 97 active+clean; 2.2 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:11:11.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:10 smithi203 ceph-mon[36693]: pgmap v671: 97 pgs: 97 active+clean; 2.2 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:11:12.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:11 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:11:12.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:11 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:11:12.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:11 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:11:12.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.725+0000 7fb095600640 1 -- 172.21.15.12:0/724332758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb090108ef0 msgr2=0x7fb090109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.725+0000 7fb095600640 1 --2- 172.21.15.12:0/724332758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb090108ef0 0x7fb090109370 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb078009900 tx=0x7fb07802bc30 comp rx=0 tx=0).stop 2024-03-23T23:11:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.725+0000 7fb095600640 1 -- 172.21.15.12:0/724332758 shutdown_connections 2024-03-23T23:11:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.725+0000 7fb095600640 1 --2- 172.21.15.12:0/724332758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0901098b0 0x7fb090110140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.725+0000 7fb095600640 1 --2- 172.21.15.12:0/724332758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb090108ef0 0x7fb090109370 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.725+0000 7fb095600640 1 --2- 172.21.15.12:0/724332758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb090107d00 0x7fb090108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.725+0000 7fb095600640 1 -- 172.21.15.12:0/724332758 >> 172.21.15.12:0/724332758 conn(0x7fb090076cb0 msgr2=0x7fb0900790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.725+0000 7fb095600640 1 -- 172.21.15.12:0/724332758 shutdown_connections 2024-03-23T23:11:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.725+0000 7fb095600640 1 -- 172.21.15.12:0/724332758 wait complete. 2024-03-23T23:11:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.726+0000 7fb095600640 1 Processor -- start 2024-03-23T23:11:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.727+0000 7fb095600640 1 -- start start 2024-03-23T23:11:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.727+0000 7fb095600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb090107d00 0x7fb09019df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.727+0000 7fb095600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb090108ef0 0x7fb09019e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.727+0000 7fb095600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0901098b0 0x7fb0901a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.727+0000 7fb095600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0901139b0 con 0x7fb090107d00 2024-03-23T23:11:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.727+0000 7fb095600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb090113830 con 0x7fb090108ef0 2024-03-23T23:11:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.727+0000 7fb095600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb090113b30 con 0x7fb0901098b0 2024-03-23T23:11:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.727+0000 7fb08ea00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb090107d00 0x7fb09019df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.727+0000 7fb08ea00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb090107d00 0x7fb09019df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52380/0 (socket says 172.21.15.12:52380) 2024-03-23T23:11:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.727+0000 7fb08ea00640 1 -- 172.21.15.12:0/3068654954 learned_addr learned my addr 172.21.15.12:0/3068654954 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.727+0000 7fb08f400640 1 --2- 172.21.15.12:0/3068654954 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0901098b0 0x7fb0901a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.727+0000 7fb08e000640 1 --2- 172.21.15.12:0/3068654954 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb090108ef0 0x7fb09019e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.728+0000 7fb08ea00640 1 -- 172.21.15.12:0/3068654954 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0901098b0 msgr2=0x7fb0901a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.728+0000 7fb08ea00640 1 --2- 172.21.15.12:0/3068654954 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0901098b0 0x7fb0901a54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.728+0000 7fb08ea00640 1 -- 172.21.15.12:0/3068654954 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb090108ef0 msgr2=0x7fb09019e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.728+0000 7fb08ea00640 1 --2- 172.21.15.12:0/3068654954 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb090108ef0 0x7fb09019e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.728+0000 7fb08ea00640 1 -- 172.21.15.12:0/3068654954 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0901a5b50 con 0x7fb090107d00 2024-03-23T23:11:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.728+0000 7fb08ea00640 1 --2- 172.21.15.12:0/3068654954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb090107d00 0x7fb09019df30 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb080009fd0 tx=0x7fb08000c270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.728+0000 7fb087400640 1 -- 172.21.15.12:0/3068654954 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb080011470 con 0x7fb090107d00 2024-03-23T23:11:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.728+0000 7fb087400640 1 -- 172.21.15.12:0/3068654954 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb080011af0 con 0x7fb090107d00 2024-03-23T23:11:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.728+0000 7fb095600640 1 -- 172.21.15.12:0/3068654954 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0901a5e40 con 0x7fb090107d00 2024-03-23T23:11:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.728+0000 7fb087400640 1 -- 172.21.15.12:0/3068654954 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08001ab90 con 0x7fb090107d00 2024-03-23T23:11:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.729+0000 7fb095600640 1 -- 172.21.15.12:0/3068654954 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0901a62a0 con 0x7fb090107d00 2024-03-23T23:11:12.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.733+0000 7fb098371740 1 Processor -- start 2024-03-23T23:11:12.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.733+0000 7fb098371740 1 -- start start 2024-03-23T23:11:12.734 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.733+0000 7fb098371740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e9f611e0 0x5635e9f61600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:12.734 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.733+0000 7fb098371740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635e9f61bd0 0x5635e9f666d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:12.734 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.733+0000 7fb098371740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e9f66d40 0x5635e9f691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:12.734 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.733+0000 7fb098371740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635e9f243b0 con 0x5635e9f611e0 2024-03-23T23:11:12.734 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.733+0000 7fb098371740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635e9ec6830 con 0x5635e9f66d40 2024-03-23T23:11:12.734 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.733+0000 7fb098371740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635e9e6f5d0 con 0x5635e9f61bd0 2024-03-23T23:11:12.734 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.733+0000 7fb087400640 1 -- 172.21.15.12:0/3068654954 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb080011610 con 0x7fb090107d00 2024-03-23T23:11:12.734 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.734+0000 7fb08e000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e9f66d40 0x5635e9f691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:12.734 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.734+0000 7fb08f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e9f611e0 0x5635e9f61600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:12.734 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.734+0000 7fb08e000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e9f66d40 0x5635e9f691d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:43442/0 (socket says 172.21.15.12:43442) 2024-03-23T23:11:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.734+0000 7fb08e000640 1 -- 172.21.15.12:0/2609757251 learned_addr learned my addr 172.21.15.12:0/2609757251 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.734+0000 7fb08ea00640 1 --2- 172.21.15.12:0/2609757251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635e9f61bd0 0x5635e9f666d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.734+0000 7fb08e000640 1 -- 172.21.15.12:0/2609757251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635e9f61bd0 msgr2=0x5635e9f666d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.734+0000 7fb08e000640 1 --2- 172.21.15.12:0/2609757251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635e9f61bd0 0x5635e9f666d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.734+0000 7fb08e000640 1 -- 172.21.15.12:0/2609757251 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e9f611e0 msgr2=0x5635e9f61600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.734+0000 7fb08e000640 1 --2- 172.21.15.12:0/2609757251 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e9f611e0 0x5635e9f61600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.734+0000 7fb08e000640 1 -- 172.21.15.12:0/2609757251 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635e9e872c0 con 0x5635e9f66d40 2024-03-23T23:11:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.734+0000 7fb08ea00640 1 --2- 172.21.15.12:0/2609757251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635e9f61bd0 0x5635e9f666d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:11:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.734+0000 7fb08e000640 1 --2- 172.21.15.12:0/2609757251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e9f66d40 0x5635e9f691d0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb078000c00 tx=0x7fb0780095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe23de2a5111ce9b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.735+0000 7fb07f400640 1 -- 172.21.15.12:0/2609757251 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb078045070 con 0x5635e9f66d40 2024-03-23T23:11:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.735+0000 7fb07f400640 1 -- 172.21.15.12:0/2609757251 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb078004040 con 0x5635e9f66d40 2024-03-23T23:11:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.735+0000 7fb07f400640 1 -- 172.21.15.12:0/2609757251 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb078004eb0 con 0x5635e9f66d40 2024-03-23T23:11:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.735+0000 7fb087400640 1 --2- 172.21.15.12:0/3068654954 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb0740a28a0 0x7fb0740a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.735+0000 7fb087400640 1 -- 172.21.15.12:0/3068654954 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb0800cefe0 con 0x7fb090107d00 2024-03-23T23:11:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.735+0000 7fb08f400640 1 --2- 172.21.15.12:0/3068654954 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb0740a28a0 0x7fb0740a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.735+0000 7fb098371740 1 -- 172.21.15.12:0/2609757251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e9f66d40 msgr2=0x5635e9f691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.735+0000 7fb098371740 1 --2- 172.21.15.12:0/2609757251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e9f66d40 0x5635e9f691d0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb078000c00 tx=0x7fb0780095e0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.736+0000 7fb098371740 1 -- 172.21.15.12:0/2609757251 shutdown_connections 2024-03-23T23:11:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.736+0000 7fb098371740 1 --2- 172.21.15.12:0/2609757251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e9f66d40 0x5635e9f691d0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.736+0000 7fb098371740 1 --2- 172.21.15.12:0/2609757251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635e9f61bd0 0x5635e9f666d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.736+0000 7fb098371740 1 --2- 172.21.15.12:0/2609757251 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e9f611e0 0x5635e9f61600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.736+0000 7fb098371740 1 -- 172.21.15.12:0/2609757251 >> 172.21.15.12:0/2609757251 conn(0x5635e9f5c750 msgr2=0x5635e9f5cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.736+0000 7fb098371740 1 -- 172.21.15.12:0/2609757251 shutdown_connections 2024-03-23T23:11:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.736+0000 7fb098371740 1 -- 172.21.15.12:0/2609757251 wait complete. 2024-03-23T23:11:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.736+0000 7fb08f400640 1 --2- 172.21.15.12:0/3068654954 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb0740a28a0 0x7fb0740a4d60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x5635e9f62430 tx=0x7fb088005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.736+0000 7fb098371740 1 -- start start 2024-03-23T23:11:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.737+0000 7fb098371740 1 Processor -- start 2024-03-23T23:11:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.737+0000 7fb098371740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635e9f611e0 0x5635e9f6b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.737+0000 7fb098371740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e9f61bd0 0x5635e9f6b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.737+0000 7fb098371740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e9f66d40 0x5635e9f72840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.737+0000 7fb098371740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635e9e82a20 con 0x5635e9f66d40 2024-03-23T23:11:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.737+0000 7fb098371740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635e9ec28d0 con 0x5635e9f61bd0 2024-03-23T23:11:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.737+0000 7fb098371740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635e9f69c90 con 0x5635e9f611e0 2024-03-23T23:11:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.737+0000 7fb098371740 10 client.0 _ll_register_callbacks cb 0x5635e9ebf1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.737+0000 7fb08ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635e9f611e0 0x5635e9f6b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.737+0000 7fb08ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635e9f611e0 0x5635e9f6b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:34300/0 (socket says 172.21.15.12:34300) 2024-03-23T23:11:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.737+0000 7fb08ea00640 1 -- 172.21.15.12:0/485037608 learned_addr learned my addr 172.21.15.12:0/485037608 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.737+0000 7fb08f400640 1 --2- 172.21.15.12:0/485037608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e9f66d40 0x5635e9f72840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.738+0000 7fb08ea00640 1 -- 172.21.15.12:0/485037608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e9f61bd0 msgr2=0x5635e9f6b870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-23T23:11:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.738+0000 7fb08ea00640 1 --2- 172.21.15.12:0/485037608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e9f61bd0 0x5635e9f6b870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.738+0000 7fb08ea00640 1 -- 172.21.15.12:0/485037608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e9f66d40 msgr2=0x5635e9f72840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.738+0000 7fb08ea00640 1 --2- 172.21.15.12:0/485037608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e9f66d40 0x5635e9f72840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.738+0000 7fb08ea00640 1 -- 172.21.15.12:0/485037608 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5635e9bdb210 con 0x5635e9f611e0 2024-03-23T23:11:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.738+0000 7fb08ea00640 1 --2- 172.21.15.12:0/485037608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635e9f611e0 0x5635e9f6b330 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb080098320 tx=0x7fb08009b580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65b056a42150a7f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.738+0000 7fb073e00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:11:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.738+0000 7fb073e00640 1 -- 172.21.15.12:0/485037608 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08009faa0 con 0x5635e9f611e0 2024-03-23T23:11:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.738+0000 7fb073e00640 1 -- 172.21.15.12:0/485037608 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb080011ca0 con 0x5635e9f611e0 2024-03-23T23:11:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.738+0000 7fb098371740 1 -- 172.21.15.12:0/485037608 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb078004040 con 0x5635e9f611e0 2024-03-23T23:11:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.738+0000 7fb073e00640 1 -- 172.21.15.12:0/485037608 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0800b9050 con 0x5635e9f611e0 2024-03-23T23:11:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.738+0000 7fb098371740 10 client.27907 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.738+0000 7fb098371740 1 -- 172.21.15.12:0/485037608 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5635e9ca7620 con 0x5635e9f611e0 2024-03-23T23:11:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.739+0000 7fb073e00640 1 -- 172.21.15.12:0/485037608 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb078004040 con 0x5635e9f611e0 2024-03-23T23:11:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.739+0000 7fb073e00640 1 -- 172.21.15.12:0/485037608 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fb080022030 con 0x5635e9f611e0 2024-03-23T23:11:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.739+0000 7fb098371740 10 client.27907 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.739+0000 7fb098371740 10 client.27907 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:11:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.739+0000 7fb098371740 20 client.27907 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.739+0000 7fb098371740 1 --2- 172.21.15.12:0/485037608 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5635e9f77100 0x5635e9f794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.739+0000 7fb098371740 4 client.27907 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:11:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.739+0000 7fb098371740 1 -- 172.21.15.12:0/485037608 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5635e9ed3d70 con 0x5635e9f77100 2024-03-23T23:11:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.739+0000 7fb08e000640 1 --2- 172.21.15.12:0/485037608 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5635e9f77100 0x5635e9f794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.740+0000 7fb08e000640 1 --2- 172.21.15.12:0/485037608 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5635e9f77100 0x5635e9f794f0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0e87e857c325d server_cookie=2ed29ea3501204b in_seq=0 out_seq=0 2024-03-23T23:11:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.740+0000 7fb073e00640 10 client.27907 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.741+0000 7fb073e00640 1 -- 172.21.15.12:0/485037608 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5635e9ed3d70 con 0x5635e9f77100 2024-03-23T23:11:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.741+0000 7fb073e00640 10 client.27907 handle_command_reply: tid=0 2024-03-23T23:11:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.741+0000 7fb098371740 1 client.27907 shutdown 2024-03-23T23:11:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.741+0000 7fb098371740 1 -- 172.21.15.12:0/485037608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635e9f611e0 msgr2=0x5635e9f6b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.741+0000 7fb098371740 1 --2- 172.21.15.12:0/485037608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635e9f611e0 0x5635e9f6b330 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb080098320 tx=0x7fb08009b580 comp rx=0 tx=0).stop 2024-03-23T23:11:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.741+0000 7fb098371740 1 -- 172.21.15.12:0/485037608 shutdown_connections 2024-03-23T23:11:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.741+0000 7fb098371740 1 --2- 172.21.15.12:0/485037608 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5635e9f77100 0x5635e9f794f0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.741+0000 7fb098371740 1 --2- 172.21.15.12:0/485037608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e9f66d40 0x5635e9f72840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.741+0000 7fb098371740 1 --2- 172.21.15.12:0/485037608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e9f61bd0 0x5635e9f6b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.741+0000 7fb098371740 1 --2- 172.21.15.12:0/485037608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635e9f611e0 0x5635e9f6b330 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.741+0000 7fb073e00640 0 client.27907 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.741+0000 7fb098371740 1 -- 172.21.15.12:0/485037608 >> 172.21.15.12:0/485037608 conn(0x5635e9f5c750 msgr2=0x5635e9f5b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.741+0000 7fb098371740 1 -- 172.21.15.12:0/485037608 shutdown_connections 2024-03-23T23:11:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.741+0000 7fb098371740 1 -- 172.21.15.12:0/485037608 wait complete. 2024-03-23T23:11:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.741+0000 7fb098371740 20 client.27907 trim_cache size 0 max 16384 2024-03-23T23:11:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.756+0000 7fb098371740 1 Processor -- start 2024-03-23T23:11:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.756+0000 7fb098371740 1 -- start start 2024-03-23T23:11:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.757+0000 7fb098371740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e9f5c750 0x5635e9f5f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.757+0000 7fb098371740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e9f611e0 0x5635e9f72800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.757+0000 7fb098371740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635e9f61bd0 0x5635e9f5fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.757+0000 7fb098371740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0901b1360 con 0x5635e9f5c750 2024-03-23T23:11:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.757+0000 7fb098371740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0901aee60 con 0x5635e9f611e0 2024-03-23T23:11:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.757+0000 7fb098371740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0901acf00 con 0x5635e9f61bd0 2024-03-23T23:11:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.757+0000 7fb08ea00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e9f5c750 0x5635e9f5f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.757+0000 7fb08e000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e9f611e0 0x5635e9f72800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:12.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.757+0000 7fb08f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635e9f61bd0 0x5635e9f5fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:12.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.757+0000 7fb08e000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e9f611e0 0x5635e9f72800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:43464/0 (socket says 172.21.15.12:43464) 2024-03-23T23:11:12.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.757+0000 7fb08f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635e9f61bd0 0x5635e9f5fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:34308/0 (socket says 172.21.15.12:34308) 2024-03-23T23:11:12.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.757+0000 7fb08e000640 1 -- 172.21.15.12:0/3298355138 learned_addr learned my addr 172.21.15.12:0/3298355138 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:12.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.757+0000 7fb08e000640 1 -- 172.21.15.12:0/3298355138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635e9f61bd0 msgr2=0x5635e9f5fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:12.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.757+0000 7fb08e000640 1 --2- 172.21.15.12:0/3298355138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635e9f61bd0 0x5635e9f5fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.757+0000 7fb08e000640 1 -- 172.21.15.12:0/3298355138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e9f5c750 msgr2=0x5635e9f5f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:12.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.757+0000 7fb08e000640 1 --2- 172.21.15.12:0/3298355138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e9f5c750 0x5635e9f5f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.757+0000 7fb08e000640 1 -- 172.21.15.12:0/3298355138 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635e9f69bd0 con 0x5635e9f611e0 2024-03-23T23:11:12.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.758+0000 7fb08e000640 1 --2- 172.21.15.12:0/3298355138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e9f611e0 0x5635e9f72800 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fb07802c710 tx=0x7fb0780095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5acf04aeaf3e4b55 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:12.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.758+0000 7fb07ea00640 1 -- 172.21.15.12:0/3298355138 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb078045040 con 0x5635e9f611e0 2024-03-23T23:11:12.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.758+0000 7fb07ea00640 1 -- 172.21.15.12:0/3298355138 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb07804e030 con 0x5635e9f611e0 2024-03-23T23:11:12.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.758+0000 7fb07ea00640 1 -- 172.21.15.12:0/3298355138 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb078004eb0 con 0x5635e9f611e0 2024-03-23T23:11:12.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.759+0000 7fb098371740 1 -- 172.21.15.12:0/3298355138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e9f611e0 msgr2=0x5635e9f72800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:12.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.759+0000 7fb098371740 1 --2- 172.21.15.12:0/3298355138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e9f611e0 0x5635e9f72800 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fb07802c710 tx=0x7fb0780095e0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.759+0000 7fb098371740 1 -- 172.21.15.12:0/3298355138 shutdown_connections 2024-03-23T23:11:12.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.759+0000 7fb098371740 1 --2- 172.21.15.12:0/3298355138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635e9f61bd0 0x5635e9f5fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.759+0000 7fb098371740 1 --2- 172.21.15.12:0/3298355138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e9f611e0 0x5635e9f72800 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.759+0000 7fb098371740 1 --2- 172.21.15.12:0/3298355138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e9f5c750 0x5635e9f5f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.759+0000 7fb098371740 1 -- 172.21.15.12:0/3298355138 >> 172.21.15.12:0/3298355138 conn(0x5635e9f66840 msgr2=0x5635e9f5cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:12.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.759+0000 7fb098371740 1 -- 172.21.15.12:0/3298355138 shutdown_connections 2024-03-23T23:11:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.759+0000 7fb098371740 1 -- 172.21.15.12:0/3298355138 wait complete. 2024-03-23T23:11:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.759+0000 7fb098371740 1 -- start start 2024-03-23T23:11:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.759+0000 7fb098371740 1 Processor -- start 2024-03-23T23:11:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.760+0000 7fb098371740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e9f5c750 0x5635e9f70490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.760+0000 7fb098371740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e9f611e0 0x5635e9f709d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.760+0000 7fb098371740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635e9f61bd0 0x5635e9f719e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.760+0000 7fb098371740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0901ad500 con 0x5635e9f611e0 2024-03-23T23:11:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.760+0000 7fb098371740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0901ae800 con 0x5635e9f5c750 2024-03-23T23:11:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.760+0000 7fb098371740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0901adb40 con 0x5635e9f61bd0 2024-03-23T23:11:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.760+0000 7fb098371740 10 client.0 _ll_register_callbacks cb 0x7fb08009b790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.760+0000 7fb08ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e9f5c750 0x5635e9f70490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.760+0000 7fb08ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e9f5c750 0x5635e9f70490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:43466/0 (socket says 172.21.15.12:43466) 2024-03-23T23:11:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.760+0000 7fb08ea00640 1 -- 172.21.15.12:0/3591555008 learned_addr learned my addr 172.21.15.12:0/3591555008 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.760+0000 7fb08f400640 1 --2- 172.21.15.12:0/3591555008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635e9f61bd0 0x5635e9f719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.760+0000 7fb08e000640 1 --2- 172.21.15.12:0/3591555008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e9f611e0 0x5635e9f709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.760+0000 7fb08ea00640 1 -- 172.21.15.12:0/3591555008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635e9f61bd0 msgr2=0x5635e9f719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.761+0000 7fb08ea00640 1 --2- 172.21.15.12:0/3591555008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635e9f61bd0 0x5635e9f719e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.761+0000 7fb08ea00640 1 -- 172.21.15.12:0/3591555008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e9f611e0 msgr2=0x5635e9f709d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.761+0000 7fb08ea00640 1 --2- 172.21.15.12:0/3591555008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e9f611e0 0x5635e9f709d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.761+0000 7fb08ea00640 1 -- 172.21.15.12:0/3591555008 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5635e9f71fd0 con 0x5635e9f5c750 2024-03-23T23:11:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.761+0000 7fb08ea00640 1 --2- 172.21.15.12:0/3591555008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e9f5c750 0x5635e9f70490 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fb0800982d0 tx=0x7fb08001af60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c28390a50d0f96b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.761+0000 7fb073e00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:11:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.761+0000 7fb073e00640 1 -- 172.21.15.12:0/3591555008 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0800978a0 con 0x5635e9f5c750 2024-03-23T23:11:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.761+0000 7fb098371740 1 -- 172.21.15.12:0/3591555008 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb07804e030 con 0x5635e9f5c750 2024-03-23T23:11:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.761+0000 7fb073e00640 1 -- 172.21.15.12:0/3591555008 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb08009a810 con 0x5635e9f5c750 2024-03-23T23:11:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.761+0000 7fb073e00640 1 -- 172.21.15.12:0/3591555008 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0800d2360 con 0x5635e9f5c750 2024-03-23T23:11:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.761+0000 7fb098371740 10 client.27950 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.761+0000 7fb098371740 1 -- 172.21.15.12:0/3591555008 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5635e9fd2f40 con 0x5635e9f5c750 2024-03-23T23:11:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.762+0000 7fb073e00640 1 -- 172.21.15.12:0/3591555008 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb07804e030 con 0x5635e9f5c750 2024-03-23T23:11:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.762+0000 7fb073e00640 1 -- 172.21.15.12:0/3591555008 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fb080022030 con 0x5635e9f5c750 2024-03-23T23:11:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.762+0000 7fb098371740 10 client.27950 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.762+0000 7fb098371740 10 client.27950 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:11:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.762+0000 7fb098371740 20 client.27950 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.762+0000 7fb098371740 1 --2- 172.21.15.12:0/3591555008 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5635e9f66d40 0x5635e9fd5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.762+0000 7fb098371740 4 client.27950 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:11:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.762+0000 7fb098371740 1 -- 172.21.15.12:0/3591555008 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5635e9ece0c0 con 0x5635e9f66d40 2024-03-23T23:11:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.762+0000 7fb08e000640 1 --2- 172.21.15.12:0/3591555008 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5635e9f66d40 0x5635e9fd5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.762+0000 7fb08e000640 1 --2- 172.21.15.12:0/3591555008 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5635e9f66d40 0x5635e9fd5810 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e99bffdddb0a51ef server_cookie=fc09a8344be15157 in_seq=0 out_seq=0 2024-03-23T23:11:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.762+0000 7fb073e00640 10 client.27950 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.774+0000 7fb073e00640 1 -- 172.21.15.12:0/3591555008 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5635e9ece0c0 con 0x5635e9f66d40 2024-03-23T23:11:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.774+0000 7fb073e00640 10 client.27950 handle_command_reply: tid=0 2024-03-23T23:11:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.775+0000 7fb098371740 1 client.27950 shutdown 2024-03-23T23:11:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.775+0000 7fb098371740 1 -- 172.21.15.12:0/3591555008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e9f5c750 msgr2=0x5635e9f70490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.775+0000 7fb098371740 1 --2- 172.21.15.12:0/3591555008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e9f5c750 0x5635e9f70490 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fb0800982d0 tx=0x7fb08001af60 comp rx=0 tx=0).stop 2024-03-23T23:11:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.775+0000 7fb098371740 1 -- 172.21.15.12:0/3591555008 shutdown_connections 2024-03-23T23:11:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.775+0000 7fb098371740 1 --2- 172.21.15.12:0/3591555008 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5635e9f66d40 0x5635e9fd5810 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.775+0000 7fb098371740 1 --2- 172.21.15.12:0/3591555008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635e9f61bd0 0x5635e9f719e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.775+0000 7fb098371740 1 --2- 172.21.15.12:0/3591555008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e9f611e0 0x5635e9f709d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.775+0000 7fb098371740 1 --2- 172.21.15.12:0/3591555008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e9f5c750 0x5635e9f70490 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.775+0000 7fb073e00640 0 client.27950 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.775+0000 7fb098371740 1 -- 172.21.15.12:0/3591555008 >> 172.21.15.12:0/3591555008 conn(0x5635e9f66840 msgr2=0x5635e9f5f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:12.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.775+0000 7fb098371740 1 -- 172.21.15.12:0/3591555008 shutdown_connections 2024-03-23T23:11:12.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.775+0000 7fb098371740 1 -- 172.21.15.12:0/3591555008 wait complete. 2024-03-23T23:11:12.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.775+0000 7fb098371740 20 client.27950 trim_cache size 0 max 16384 2024-03-23T23:11:12.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:11:12.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:11:12.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:11:12.776 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:11:12.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.776+0000 7fb095600640 1 -- 172.21.15.12:0/3068654954 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb0740a28a0 msgr2=0x7fb0740a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:12.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.776+0000 7fb095600640 1 --2- 172.21.15.12:0/3068654954 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb0740a28a0 0x7fb0740a4d60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x5635e9f62430 tx=0x7fb088005f90 comp rx=0 tx=0).stop 2024-03-23T23:11:12.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.776+0000 7fb095600640 1 -- 172.21.15.12:0/3068654954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb090107d00 msgr2=0x7fb09019df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:12.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.776+0000 7fb095600640 1 --2- 172.21.15.12:0/3068654954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb090107d00 0x7fb09019df30 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb080009fd0 tx=0x7fb08000c270 comp rx=0 tx=0).stop 2024-03-23T23:11:12.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.776+0000 7fb095600640 1 -- 172.21.15.12:0/3068654954 shutdown_connections 2024-03-23T23:11:12.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.776+0000 7fb095600640 1 --2- 172.21.15.12:0/3068654954 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb0740a28a0 0x7fb0740a4d60 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.776+0000 7fb095600640 1 --2- 172.21.15.12:0/3068654954 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0901098b0 0x7fb0901a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.776+0000 7fb095600640 1 --2- 172.21.15.12:0/3068654954 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb090108ef0 0x7fb09019e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.776+0000 7fb095600640 1 --2- 172.21.15.12:0/3068654954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb090107d00 0x7fb09019df30 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:12.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.776+0000 7fb095600640 1 -- 172.21.15.12:0/3068654954 >> 172.21.15.12:0/3068654954 conn(0x7fb090076cb0 msgr2=0x7fb09010c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:12.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.776+0000 7fb095600640 1 -- 172.21.15.12:0/3068654954 shutdown_connections 2024-03-23T23:11:12.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:12.776+0000 7fb095600640 1 -- 172.21.15.12:0/3068654954 wait complete. 2024-03-23T23:11:13.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:12 smithi012 ceph-mon[31695]: pgmap v672: 97 pgs: 97 active+clean; 2.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:11:13.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:12 smithi173 ceph-mon[36116]: pgmap v672: 97 pgs: 97 active+clean; 2.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:11:13.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:12 smithi203 ceph-mon[36693]: pgmap v672: 97 pgs: 97 active+clean; 2.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:11:14.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:13 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:11:14.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:13 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:11:14.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:13 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:11:14.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:13 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:11:14.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:13 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:11:14.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:13 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:11:14.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:13 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:11:14.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:13 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:11:14.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:13 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:11:15.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:14 smithi203 ceph-mon[36693]: pgmap v673: 97 pgs: 97 active+clean; 2.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:11:15.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:14 smithi012 ceph-mon[31695]: pgmap v673: 97 pgs: 97 active+clean; 2.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:11:15.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:14 smithi173 ceph-mon[36116]: pgmap v673: 97 pgs: 97 active+clean; 2.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:11:17.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:16 smithi203 ceph-mon[36693]: pgmap v674: 97 pgs: 97 active+clean; 2.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 13 op/s 2024-03-23T23:11:17.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:16 smithi012 ceph-mon[31695]: pgmap v674: 97 pgs: 97 active+clean; 2.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 13 op/s 2024-03-23T23:11:17.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:16 smithi173 ceph-mon[36116]: pgmap v674: 97 pgs: 97 active+clean; 2.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 13 op/s 2024-03-23T23:11:19.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:18 smithi203 ceph-mon[36693]: pgmap v675: 97 pgs: 97 active+clean; 2.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:11:19.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:18 smithi012 ceph-mon[31695]: pgmap v675: 97 pgs: 97 active+clean; 2.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:11:19.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:18 smithi173 ceph-mon[36116]: pgmap v675: 97 pgs: 97 active+clean; 2.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:11:21.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:20 smithi012 ceph-mon[31695]: pgmap v676: 97 pgs: 97 active+clean; 2.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:11:21.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:20 smithi173 ceph-mon[36116]: pgmap v676: 97 pgs: 97 active+clean; 2.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:11:21.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:20 smithi203 ceph-mon[36693]: pgmap v676: 97 pgs: 97 active+clean; 2.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:11:23.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:22 smithi012 ceph-mon[31695]: pgmap v677: 97 pgs: 97 active+clean; 2.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-03-23T23:11:23.310 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:11:23.310 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:11:23.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:22 smithi173 ceph-mon[36116]: pgmap v677: 97 pgs: 97 active+clean; 2.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-03-23T23:11:23.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:22 smithi203 ceph-mon[36693]: pgmap v677: 97 pgs: 97 active+clean; 2.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-03-23T23:11:23.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.549+0000 7f7decc00640 1 -- 172.21.15.12:0/2468707732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7de8109770 msgr2=0x7f7de8110000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:23.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.549+0000 7f7decc00640 1 --2- 172.21.15.12:0/2468707732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7de8109770 0x7f7de8110000 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f7de0009be0 tx=0x7f7de002bef0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.549+0000 7f7decc00640 1 -- 172.21.15.12:0/2468707732 shutdown_connections 2024-03-23T23:11:23.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.549+0000 7f7decc00640 1 --2- 172.21.15.12:0/2468707732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7de8109770 0x7f7de8110000 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.549+0000 7f7decc00640 1 --2- 172.21.15.12:0/2468707732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7de8108db0 0x7f7de8109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.549+0000 7f7decc00640 1 --2- 172.21.15.12:0/2468707732 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7de8107bc0 0x7f7de8107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.549+0000 7f7decc00640 1 -- 172.21.15.12:0/2468707732 >> 172.21.15.12:0/2468707732 conn(0x7f7de8076b70 msgr2=0x7f7de8078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:23.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.549+0000 7f7decc00640 1 -- 172.21.15.12:0/2468707732 shutdown_connections 2024-03-23T23:11:23.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.549+0000 7f7decc00640 1 -- 172.21.15.12:0/2468707732 wait complete. 2024-03-23T23:11:23.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.550+0000 7f7decc00640 1 Processor -- start 2024-03-23T23:11:23.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.551+0000 7f7decc00640 1 -- start start 2024-03-23T23:11:23.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.551+0000 7f7decc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7de8107bc0 0x7f7de819ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:23.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.551+0000 7f7decc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7de8108db0 0x7f7de819e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:23.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.551+0000 7f7decc00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7de8109770 0x7f7de81a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:23.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.551+0000 7f7decc00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de8113870 con 0x7f7de8109770 2024-03-23T23:11:23.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.551+0000 7f7decc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de81136f0 con 0x7f7de8107bc0 2024-03-23T23:11:23.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.551+0000 7f7decc00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de81139f0 con 0x7f7de8108db0 2024-03-23T23:11:23.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.551+0000 7f7de7e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7de8109770 0x7f7de81a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:23.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.551+0000 7f7de6a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7de8108db0 0x7f7de819e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:23.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.551+0000 7f7de7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7de8107bc0 0x7f7de819ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:23.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.551+0000 7f7de6a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7de8108db0 0x7f7de819e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35888/0 (socket says 172.21.15.12:35888) 2024-03-23T23:11:23.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.552+0000 7f7de6a00640 1 -- 172.21.15.12:0/2096832621 learned_addr learned my addr 172.21.15.12:0/2096832621 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:23.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.552+0000 7f7de6a00640 1 -- 172.21.15.12:0/2096832621 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7de8107bc0 msgr2=0x7f7de819ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:23.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.552+0000 7f7de6a00640 1 --2- 172.21.15.12:0/2096832621 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7de8107bc0 0x7f7de819ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.552+0000 7f7de6a00640 1 -- 172.21.15.12:0/2096832621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7de8109770 msgr2=0x7f7de81a5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:23.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.552+0000 7f7de6a00640 1 --2- 172.21.15.12:0/2096832621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7de8109770 0x7f7de81a5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.552+0000 7f7de6a00640 1 -- 172.21.15.12:0/2096832621 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de81a59f0 con 0x7f7de8108db0 2024-03-23T23:11:23.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.552+0000 7f7de6a00640 1 --2- 172.21.15.12:0/2096832621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7de8108db0 0x7f7de819e310 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400e9f0 tx=0x7f7dd400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:23.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.553+0000 7f7ddfe00640 1 -- 172.21.15.12:0/2096832621 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd400ccc0 con 0x7f7de8108db0 2024-03-23T23:11:23.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.553+0000 7f7ddfe00640 1 -- 172.21.15.12:0/2096832621 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7dd4010910 con 0x7f7de8108db0 2024-03-23T23:11:23.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.553+0000 7f7ddfe00640 1 -- 172.21.15.12:0/2096832621 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd40188a0 con 0x7f7de8108db0 2024-03-23T23:11:23.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.553+0000 7f7decc00640 1 -- 172.21.15.12:0/2096832621 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7de81a5ce0 con 0x7f7de8108db0 2024-03-23T23:11:23.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.553+0000 7f7decc00640 1 -- 172.21.15.12:0/2096832621 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7de81a6220 con 0x7f7de8108db0 2024-03-23T23:11:23.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.556+0000 7f7df0e63740 1 Processor -- start 2024-03-23T23:11:23.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.556+0000 7f7df0e63740 1 -- start start 2024-03-23T23:11:23.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.557+0000 7f7ddfe00640 1 -- 172.21.15.12:0/2096832621 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7dd4010ac0 con 0x7f7de8108db0 2024-03-23T23:11:23.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.557+0000 7f7df0e63740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642539a9050 0x5642539a9470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:23.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.557+0000 7f7df0e63740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5642539a9a40 0x5642539ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:23.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.557+0000 7f7df0e63740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642539aebb0 0x5642539b1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:23.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.557+0000 7f7df0e63740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56425390e250 con 0x5642539a9a40 2024-03-23T23:11:23.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.557+0000 7f7df0e63740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642538b6a30 con 0x5642539a9050 2024-03-23T23:11:23.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.557+0000 7f7df0e63740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x56425390a2f0 con 0x5642539aebb0 2024-03-23T23:11:23.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.557+0000 7f7de6a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642539aebb0 0x5642539b1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:23.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.557+0000 7f7de6a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642539aebb0 0x5642539b1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:35892/0 (socket says 172.21.15.12:35892) 2024-03-23T23:11:23.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.557+0000 7f7de6a00640 1 -- 172.21.15.12:0/3330088658 learned_addr learned my addr 172.21.15.12:0/3330088658 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:23.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.557+0000 7f7de7e00640 1 --2- 172.21.15.12:0/3330088658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642539a9050 0x5642539a9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:23.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.557+0000 7f7de7400640 1 --2- 172.21.15.12:0/3330088658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5642539a9a40 0x5642539ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:23.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.558+0000 7f7de6a00640 1 -- 172.21.15.12:0/3330088658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642539a9050 msgr2=0x5642539a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:23.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.558+0000 7f7de6a00640 1 --2- 172.21.15.12:0/3330088658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642539a9050 0x5642539a9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.558+0000 7f7de6a00640 1 -- 172.21.15.12:0/3330088658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5642539a9a40 msgr2=0x5642539ae540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:23.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.558+0000 7f7de6a00640 1 --2- 172.21.15.12:0/3330088658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5642539a9a40 0x5642539ae540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.558+0000 7f7de6a00640 1 -- 172.21.15.12:0/3330088658 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5642538a1690 con 0x5642539aebb0 2024-03-23T23:11:23.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.558+0000 7f7de7400640 1 --2- 172.21.15.12:0/3330088658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5642539a9a40 0x5642539ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:11:23.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.558+0000 7f7de6a00640 1 --2- 172.21.15.12:0/3330088658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642539aebb0 0x5642539b1040 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f7dd40d2ed0 tx=0x7f7dd40d57f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e45b59f0c171cd64 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.558+0000 7f7dd3e00640 1 -- 172.21.15.12:0/3330088658 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd40e0430 con 0x5642539aebb0 2024-03-23T23:11:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.558+0000 7f7dd3e00640 1 -- 172.21.15.12:0/3330088658 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7dd40d38e0 con 0x5642539aebb0 2024-03-23T23:11:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.558+0000 7f7dd3e00640 1 -- 172.21.15.12:0/3330088658 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd40df7d0 con 0x5642539aebb0 2024-03-23T23:11:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.559+0000 7f7ddfe00640 1 --2- 172.21.15.12:0/2096832621 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7dcc0a2880 0x7f7dcc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.559+0000 7f7df0e63740 1 -- 172.21.15.12:0/3330088658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642539aebb0 msgr2=0x5642539b1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.559+0000 7f7df0e63740 1 --2- 172.21.15.12:0/3330088658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642539aebb0 0x5642539b1040 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f7dd40d2ed0 tx=0x7f7dd40d57f0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.559+0000 7f7ddfe00640 1 -- 172.21.15.12:0/2096832621 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7dd40cee20 con 0x7f7de8108db0 2024-03-23T23:11:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.559+0000 7f7de7e00640 1 --2- 172.21.15.12:0/2096832621 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7dcc0a2880 0x7f7dcc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.559+0000 7f7df0e63740 1 -- 172.21.15.12:0/3330088658 shutdown_connections 2024-03-23T23:11:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.559+0000 7f7df0e63740 1 --2- 172.21.15.12:0/3330088658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642539aebb0 0x5642539b1040 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.559+0000 7f7df0e63740 1 --2- 172.21.15.12:0/3330088658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5642539a9a40 0x5642539ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.559+0000 7f7df0e63740 1 --2- 172.21.15.12:0/3330088658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642539a9050 0x5642539a9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.559+0000 7f7df0e63740 1 -- 172.21.15.12:0/3330088658 >> 172.21.15.12:0/3330088658 conn(0x564253971200 msgr2=0x5642539a49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.559+0000 7f7df0e63740 1 -- 172.21.15.12:0/3330088658 shutdown_connections 2024-03-23T23:11:23.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.559+0000 7f7df0e63740 1 -- 172.21.15.12:0/3330088658 wait complete. 2024-03-23T23:11:23.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.560+0000 7f7de7e00640 1 --2- 172.21.15.12:0/2096832621 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7dcc0a2880 0x7f7dcc0a4d40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f7de0006010 tx=0x7f7de0004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:23.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.560+0000 7f7df0e63740 1 -- start start 2024-03-23T23:11:23.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.560+0000 7f7df0e63740 1 Processor -- start 2024-03-23T23:11:23.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.560+0000 7f7df0e63740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642539a9050 0x5642539b3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:23.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.561+0000 7f7df0e63740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5642539a9a40 0x5642539b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:23.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.561+0000 7f7df0e63740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642539aebb0 0x5642539ba5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:23.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.561+0000 7f7df0e63740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642539b1b60 con 0x5642539a9a40 2024-03-23T23:11:23.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.561+0000 7f7df0e63740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642538c9e80 con 0x5642539aebb0 2024-03-23T23:11:23.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.561+0000 7f7df0e63740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642539b1ce0 con 0x5642539a9050 2024-03-23T23:11:23.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.561+0000 7f7df0e63740 10 client.0 _ll_register_callbacks cb 0x5642538dc130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:23.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.561+0000 7f7de6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5642539a9a40 0x5642539b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:23.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.561+0000 7f7de7400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642539a9050 0x5642539b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:23.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.561+0000 7f7de6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5642539a9a40 0x5642539b35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41794/0 (socket says 172.21.15.12:41794) 2024-03-23T23:11:23.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.561+0000 7f7de6a00640 1 -- 172.21.15.12:0/3838930469 learned_addr learned my addr 172.21.15.12:0/3838930469 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:23.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.561+0000 7f7de7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642539aebb0 0x5642539ba5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:23.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.561+0000 7f7de7400640 1 -- 172.21.15.12:0/3838930469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642539aebb0 msgr2=0x5642539ba5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:23.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.561+0000 7f7de7400640 1 --2- 172.21.15.12:0/3838930469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642539aebb0 0x5642539ba5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.561+0000 7f7de7400640 1 -- 172.21.15.12:0/3838930469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5642539a9a40 msgr2=0x5642539b35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.561+0000 7f7de7400640 1 --2- 172.21.15.12:0/3838930469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5642539a9a40 0x5642539b35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.561+0000 7f7de7400640 1 -- 172.21.15.12:0/3838930469 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5642536ec7d0 con 0x5642539a9050 2024-03-23T23:11:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.561+0000 7f7de6a00640 1 --2- 172.21.15.12:0/3838930469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5642539a9a40 0x5642539b35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.561+0000 7f7de7e00640 1 --2- 172.21.15.12:0/3838930469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642539aebb0 0x5642539ba5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.562+0000 7f7de7400640 1 --2- 172.21.15.12:0/3838930469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642539a9050 0x5642539b3060 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x5642539a9e40 tx=0x7f7dd8012aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d72f3bd75f51f32 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.562+0000 7f7dd0c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:11:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.562+0000 7f7dd0c00640 1 -- 172.21.15.12:0/3838930469 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd8004030 con 0x5642539a9050 2024-03-23T23:11:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.562+0000 7f7dd0c00640 1 -- 172.21.15.12:0/3838930469 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7dd80041d0 con 0x5642539a9050 2024-03-23T23:11:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.562+0000 7f7df0e63740 1 -- 172.21.15.12:0/3838930469 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7dd40d38e0 con 0x5642539a9050 2024-03-23T23:11:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.562+0000 7f7dd0c00640 1 -- 172.21.15.12:0/3838930469 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7dd80043d0 con 0x5642539a9050 2024-03-23T23:11:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.562+0000 7f7df0e63740 10 client.27925 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.562+0000 7f7df0e63740 1 -- 172.21.15.12:0/3838930469 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5642536223f0 con 0x5642539a9050 2024-03-23T23:11:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.562+0000 7f7dd0c00640 1 -- 172.21.15.12:0/3838930469 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7dd40d38e0 con 0x5642539a9050 2024-03-23T23:11:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.563+0000 7f7dd0c00640 1 -- 172.21.15.12:0/3838930469 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f7dd80038f0 con 0x5642539a9050 2024-03-23T23:11:23.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.563+0000 7f7df0e63740 10 client.27925 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:23.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.563+0000 7f7df0e63740 10 client.27925 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:11:23.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.563+0000 7f7df0e63740 20 client.27925 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:23.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.563+0000 7f7df0e63740 1 --2- 172.21.15.12:0/3838930469 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5642539bef50 0x5642539c13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:23.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.563+0000 7f7df0e63740 4 client.27925 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:11:23.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.563+0000 7f7df0e63740 1 -- 172.21.15.12:0/3838930469 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56425391b790 con 0x5642539bef50 2024-03-23T23:11:23.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.563+0000 7f7de6a00640 1 --2- 172.21.15.12:0/3838930469 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5642539bef50 0x5642539c13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:23.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.563+0000 7f7de6a00640 1 --2- 172.21.15.12:0/3838930469 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5642539bef50 0x5642539c13a0 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67bdfa73b415328b server_cookie=611d797aa7b078c5 in_seq=0 out_seq=0 2024-03-23T23:11:23.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.563+0000 7f7dd0c00640 10 client.27925 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:23.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.564+0000 7f7dd0c00640 1 -- 172.21.15.12:0/3838930469 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56425391b790 con 0x5642539bef50 2024-03-23T23:11:23.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.564+0000 7f7dd0c00640 10 client.27925 handle_command_reply: tid=0 2024-03-23T23:11:23.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.564+0000 7f7df0e63740 1 client.27925 shutdown 2024-03-23T23:11:23.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.564+0000 7f7df0e63740 1 -- 172.21.15.12:0/3838930469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642539a9050 msgr2=0x5642539b3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:23.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.564+0000 7f7df0e63740 1 --2- 172.21.15.12:0/3838930469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642539a9050 0x5642539b3060 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x5642539a9e40 tx=0x7f7dd8012aa0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.565+0000 7f7df0e63740 1 -- 172.21.15.12:0/3838930469 shutdown_connections 2024-03-23T23:11:23.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.565+0000 7f7df0e63740 1 --2- 172.21.15.12:0/3838930469 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5642539bef50 0x5642539c13a0 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.565+0000 7f7df0e63740 1 --2- 172.21.15.12:0/3838930469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642539aebb0 0x5642539ba5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.565+0000 7f7df0e63740 1 --2- 172.21.15.12:0/3838930469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5642539a9a40 0x5642539b35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.565+0000 7f7df0e63740 1 --2- 172.21.15.12:0/3838930469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642539a9050 0x5642539b3060 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.565+0000 7f7dd0c00640 0 client.27925 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:23.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.565+0000 7f7df0e63740 1 -- 172.21.15.12:0/3838930469 >> 172.21.15.12:0/3838930469 conn(0x564253971200 msgr2=0x5642539a3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:23.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.565+0000 7f7df0e63740 1 -- 172.21.15.12:0/3838930469 shutdown_connections 2024-03-23T23:11:23.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.565+0000 7f7df0e63740 1 -- 172.21.15.12:0/3838930469 wait complete. 2024-03-23T23:11:23.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.565+0000 7f7df0e63740 20 client.27925 trim_cache size 0 max 16384 2024-03-23T23:11:23.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.582+0000 7f7df0e63740 1 Processor -- start 2024-03-23T23:11:23.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.582+0000 7f7df0e63740 1 -- start start 2024-03-23T23:11:23.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.583+0000 7f7df0e63740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5642539a9050 0x5642539a8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:23.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.583+0000 7f7df0e63740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642539a9a40 0x5642539ba5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:23.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.583+0000 7f7df0e63740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642539aebb0 0x5642539a3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:23.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.583+0000 7f7df0e63740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de81b1330 con 0x5642539a9050 2024-03-23T23:11:23.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.583+0000 7f7df0e63740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de81aee30 con 0x5642539a9a40 2024-03-23T23:11:23.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.583+0000 7f7df0e63740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de81aced0 con 0x5642539aebb0 2024-03-23T23:11:23.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.583+0000 7f7de7400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5642539a9050 0x5642539a8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:23.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.583+0000 7f7de7e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642539aebb0 0x5642539a3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:23.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.583+0000 7f7de7400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5642539a9050 0x5642539a8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41802/0 (socket says 172.21.15.12:41802) 2024-03-23T23:11:23.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.583+0000 7f7de6a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642539a9a40 0x5642539ba5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:23.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.583+0000 7f7de7400640 1 -- 172.21.15.12:0/1912664329 learned_addr learned my addr 172.21.15.12:0/1912664329 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:23.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.583+0000 7f7de7e00640 1 -- 172.21.15.12:0/1912664329 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642539a9a40 msgr2=0x5642539ba5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:23.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.583+0000 7f7de7e00640 1 --2- 172.21.15.12:0/1912664329 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642539a9a40 0x5642539ba5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.583+0000 7f7de7e00640 1 -- 172.21.15.12:0/1912664329 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5642539a9050 msgr2=0x5642539a8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:23.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.583+0000 7f7de7e00640 1 --2- 172.21.15.12:0/1912664329 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5642539a9050 0x5642539a8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.584+0000 7f7de7e00640 1 -- 172.21.15.12:0/1912664329 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5642535e20f0 con 0x5642539aebb0 2024-03-23T23:11:23.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.584+0000 7f7de7e00640 1 --2- 172.21.15.12:0/1912664329 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642539aebb0 0x5642539a3a40 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7de002c430 tx=0x7f7de0042000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3e00c61851c2df8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:23.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.584+0000 7f7dd3400640 1 -- 172.21.15.12:0/1912664329 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de002cd60 con 0x5642539aebb0 2024-03-23T23:11:23.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.584+0000 7f7dd3400640 1 -- 172.21.15.12:0/1912664329 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7de0004520 con 0x5642539aebb0 2024-03-23T23:11:23.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.584+0000 7f7dd3400640 1 -- 172.21.15.12:0/1912664329 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de0034a90 con 0x5642539aebb0 2024-03-23T23:11:23.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.585+0000 7f7df0e63740 1 -- 172.21.15.12:0/1912664329 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642539aebb0 msgr2=0x5642539a3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:23.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.585+0000 7f7df0e63740 1 --2- 172.21.15.12:0/1912664329 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642539aebb0 0x5642539a3a40 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7de002c430 tx=0x7f7de0042000 comp rx=0 tx=0).stop 2024-03-23T23:11:23.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.585+0000 7f7df0e63740 1 -- 172.21.15.12:0/1912664329 shutdown_connections 2024-03-23T23:11:23.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.585+0000 7f7df0e63740 1 --2- 172.21.15.12:0/1912664329 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642539aebb0 0x5642539a3a40 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.585+0000 7f7df0e63740 1 --2- 172.21.15.12:0/1912664329 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642539a9a40 0x5642539ba5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.585+0000 7f7df0e63740 1 --2- 172.21.15.12:0/1912664329 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5642539a9050 0x5642539a8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.585+0000 7f7df0e63740 1 -- 172.21.15.12:0/1912664329 >> 172.21.15.12:0/1912664329 conn(0x564253971200 msgr2=0x5642539a3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:23.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.585+0000 7f7df0e63740 1 -- 172.21.15.12:0/1912664329 shutdown_connections 2024-03-23T23:11:23.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.585+0000 7f7df0e63740 1 -- 172.21.15.12:0/1912664329 wait complete. 2024-03-23T23:11:23.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.586+0000 7f7df0e63740 1 -- start start 2024-03-23T23:11:23.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.586+0000 7f7df0e63740 1 Processor -- start 2024-03-23T23:11:23.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.586+0000 7f7df0e63740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642539a9050 0x5642539b75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:23.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.586+0000 7f7df0e63740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5642539a9a40 0x5642539b7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:23.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.586+0000 7f7df0e63740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642539aebb0 0x5642539b8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.586+0000 7f7df0e63740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de81ad4d0 con 0x5642539a9a40 2024-03-23T23:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.586+0000 7f7df0e63740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de81ae7d0 con 0x5642539a9050 2024-03-23T23:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.586+0000 7f7df0e63740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de81adb10 con 0x5642539aebb0 2024-03-23T23:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.586+0000 7f7df0e63740 10 client.0 _ll_register_callbacks cb 0x7f7dd8009e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.587+0000 7f7de7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642539a9050 0x5642539b75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.587+0000 7f7de7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642539a9050 0x5642539b75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37398/0 (socket says 172.21.15.12:37398) 2024-03-23T23:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.587+0000 7f7de6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5642539a9a40 0x5642539b7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.587+0000 7f7de7400640 1 -- 172.21.15.12:0/4266013901 learned_addr learned my addr 172.21.15.12:0/4266013901 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.587+0000 7f7de7e00640 1 --2- 172.21.15.12:0/4266013901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642539aebb0 0x5642539b8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.587+0000 7f7de7400640 1 -- 172.21.15.12:0/4266013901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642539aebb0 msgr2=0x5642539b8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.587+0000 7f7de7400640 1 --2- 172.21.15.12:0/4266013901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642539aebb0 0x5642539b8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.587+0000 7f7de7400640 1 -- 172.21.15.12:0/4266013901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5642539a9a40 msgr2=0x5642539b7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.587+0000 7f7de7400640 1 --2- 172.21.15.12:0/4266013901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5642539a9a40 0x5642539b7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.587+0000 7f7de7400640 1 -- 172.21.15.12:0/4266013901 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5642539b9130 con 0x5642539a9050 2024-03-23T23:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.587+0000 7f7de7400640 1 --2- 172.21.15.12:0/4266013901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642539a9050 0x5642539b75f0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8003ce0 tx=0x7f7dd8009cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b88d920bcab66e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.587+0000 7f7dd0c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:11:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.587+0000 7f7dd0c00640 1 -- 172.21.15.12:0/4266013901 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd8003800 con 0x5642539a9050 2024-03-23T23:11:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.588+0000 7f7dd0c00640 1 -- 172.21.15.12:0/4266013901 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7dd8018070 con 0x5642539a9050 2024-03-23T23:11:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.588+0000 7f7df0e63740 1 -- 172.21.15.12:0/4266013901 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7de0004520 con 0x5642539a9050 2024-03-23T23:11:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.588+0000 7f7dd0c00640 1 -- 172.21.15.12:0/4266013901 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7dd80039a0 con 0x5642539a9050 2024-03-23T23:11:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.588+0000 7f7df0e63740 10 client.27968 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.588+0000 7f7df0e63740 1 -- 172.21.15.12:0/4266013901 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5642539ba440 con 0x5642539a9050 2024-03-23T23:11:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.588+0000 7f7dd0c00640 1 -- 172.21.15.12:0/4266013901 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7de0004520 con 0x5642539a9050 2024-03-23T23:11:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.588+0000 7f7dd0c00640 1 -- 172.21.15.12:0/4266013901 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f7dd800faf0 con 0x5642539a9050 2024-03-23T23:11:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.588+0000 7f7df0e63740 10 client.27968 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.588+0000 7f7df0e63740 10 client.27968 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:11:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.588+0000 7f7df0e63740 20 client.27968 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.588+0000 7f7df0e63740 1 --2- 172.21.15.12:0/4266013901 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x564253a1aab0 0x564253a1cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.588+0000 7f7df0e63740 4 client.27968 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:11:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.588+0000 7f7df0e63740 1 -- 172.21.15.12:0/4266013901 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564253915ae0 con 0x564253a1aab0 2024-03-23T23:11:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.589+0000 7f7de6a00640 1 --2- 172.21.15.12:0/4266013901 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x564253a1aab0 0x564253a1cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.589+0000 7f7de6a00640 1 --2- 172.21.15.12:0/4266013901 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x564253a1aab0 0x564253a1cea0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a96fd04031e5ad7d server_cookie=21586d5a63854f84 in_seq=0 out_seq=0 2024-03-23T23:11:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.589+0000 7f7dd0c00640 10 client.27968 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.589+0000 7f7dd0c00640 1 -- 172.21.15.12:0/4266013901 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x564253915ae0 con 0x564253a1aab0 2024-03-23T23:11:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.589+0000 7f7dd0c00640 10 client.27968 handle_command_reply: tid=0 2024-03-23T23:11:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.589+0000 7f7df0e63740 1 client.27968 shutdown 2024-03-23T23:11:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.589+0000 7f7df0e63740 1 -- 172.21.15.12:0/4266013901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642539a9050 msgr2=0x5642539b75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.589+0000 7f7df0e63740 1 --2- 172.21.15.12:0/4266013901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642539a9050 0x5642539b75f0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8003ce0 tx=0x7f7dd8009cc0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.589+0000 7f7df0e63740 1 -- 172.21.15.12:0/4266013901 shutdown_connections 2024-03-23T23:11:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.589+0000 7f7df0e63740 1 --2- 172.21.15.12:0/4266013901 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x564253a1aab0 0x564253a1cea0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.590+0000 7f7df0e63740 1 --2- 172.21.15.12:0/4266013901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642539aebb0 0x5642539b8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.590+0000 7f7df0e63740 1 --2- 172.21.15.12:0/4266013901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5642539a9a40 0x5642539b7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.590+0000 7f7df0e63740 1 --2- 172.21.15.12:0/4266013901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642539a9050 0x5642539b75f0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.590+0000 7f7df0e63740 1 -- 172.21.15.12:0/4266013901 >> 172.21.15.12:0/4266013901 conn(0x564253971200 msgr2=0x5642539a8310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.590+0000 7f7dd0c00640 0 client.27968 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.590+0000 7f7df0e63740 1 -- 172.21.15.12:0/4266013901 shutdown_connections 2024-03-23T23:11:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.590+0000 7f7df0e63740 1 -- 172.21.15.12:0/4266013901 wait complete. 2024-03-23T23:11:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.590+0000 7f7df0e63740 20 client.27968 trim_cache size 0 max 16384 2024-03-23T23:11:23.592 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:11:23.592 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-03-23T23:11:23.592 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-03-23T23:11:23.592 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:11:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.590+0000 7f7decc00640 1 -- 172.21.15.12:0/2096832621 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7dcc0a2880 msgr2=0x7f7dcc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.590+0000 7f7decc00640 1 --2- 172.21.15.12:0/2096832621 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7dcc0a2880 0x7f7dcc0a4d40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f7de0006010 tx=0x7f7de0004a40 comp rx=0 tx=0).stop 2024-03-23T23:11:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.590+0000 7f7decc00640 1 -- 172.21.15.12:0/2096832621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7de8108db0 msgr2=0x7f7de819e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.590+0000 7f7decc00640 1 --2- 172.21.15.12:0/2096832621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7de8108db0 0x7f7de819e310 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400e9f0 tx=0x7f7dd400edf0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.591+0000 7f7decc00640 1 -- 172.21.15.12:0/2096832621 shutdown_connections 2024-03-23T23:11:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.591+0000 7f7decc00640 1 --2- 172.21.15.12:0/2096832621 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7dcc0a2880 0x7f7dcc0a4d40 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.591+0000 7f7decc00640 1 --2- 172.21.15.12:0/2096832621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7de8109770 0x7f7de81a5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.591+0000 7f7decc00640 1 --2- 172.21.15.12:0/2096832621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7de8108db0 0x7f7de819e310 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.591+0000 7f7decc00640 1 --2- 172.21.15.12:0/2096832621 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7de8107bc0 0x7f7de819ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.591+0000 7f7decc00640 1 -- 172.21.15.12:0/2096832621 >> 172.21.15.12:0/2096832621 conn(0x7f7de8076b70 msgr2=0x7f7de810c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.591+0000 7f7decc00640 1 -- 172.21.15.12:0/2096832621 shutdown_connections 2024-03-23T23:11:23.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.591+0000 7f7decc00640 1 -- 172.21.15.12:0/2096832621 wait complete. 2024-03-23T23:11:23.607 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T23:11:23.608 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:11:23.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.816+0000 7fdf22400640 1 -- 172.21.15.12:0/4061532668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf1c1036f0 msgr2=0x7fdf1c103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:23.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.816+0000 7fdf22400640 1 --2- 172.21.15.12:0/4061532668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf1c1036f0 0x7fdf1c103b10 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fdf10009900 tx=0x7fdf1002bc50 comp rx=0 tx=0).stop 2024-03-23T23:11:23.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.816+0000 7fdf22400640 1 -- 172.21.15.12:0/4061532668 shutdown_connections 2024-03-23T23:11:23.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.816+0000 7fdf22400640 1 --2- 172.21.15.12:0/4061532668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf1c1052a0 0x7fdf1c10bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.816+0000 7fdf22400640 1 --2- 172.21.15.12:0/4061532668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf1c1048e0 0x7fdf1c104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.816+0000 7fdf22400640 1 --2- 172.21.15.12:0/4061532668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf1c1036f0 0x7fdf1c103b10 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.816+0000 7fdf22400640 1 -- 172.21.15.12:0/4061532668 >> 172.21.15.12:0/4061532668 conn(0x7fdf1c0fec80 msgr2=0x7fdf1c1010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:23.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.816+0000 7fdf22400640 1 -- 172.21.15.12:0/4061532668 shutdown_connections 2024-03-23T23:11:23.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.816+0000 7fdf22400640 1 -- 172.21.15.12:0/4061532668 wait complete. 2024-03-23T23:11:23.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.818+0000 7fdf22400640 1 Processor -- start 2024-03-23T23:11:23.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.818+0000 7fdf22400640 1 -- start start 2024-03-23T23:11:23.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.818+0000 7fdf22400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf1c1036f0 0x7fdf1c1999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:23.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.818+0000 7fdf22400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf1c1048e0 0x7fdf1c199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:23.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.819+0000 7fdf22400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf1c1052a0 0x7fdf1c1a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:23.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.819+0000 7fdf22400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c10f390 con 0x7fdf1c1036f0 2024-03-23T23:11:23.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.819+0000 7fdf22400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c10f210 con 0x7fdf1c1052a0 2024-03-23T23:11:23.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.819+0000 7fdf22400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c10f510 con 0x7fdf1c1048e0 2024-03-23T23:11:23.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.819+0000 7fdf21000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf1c1036f0 0x7fdf1c1999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:23.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.819+0000 7fdf1be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf1c1048e0 0x7fdf1c199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:23.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.819+0000 7fdf21000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf1c1036f0 0x7fdf1c1999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41842/0 (socket says 172.21.15.12:41842) 2024-03-23T23:11:23.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.819+0000 7fdf21000640 1 -- 172.21.15.12:0/2025027588 learned_addr learned my addr 172.21.15.12:0/2025027588 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:23.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.819+0000 7fdf21a00640 1 --2- 172.21.15.12:0/2025027588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf1c1052a0 0x7fdf1c1a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.819+0000 7fdf21000640 1 -- 172.21.15.12:0/2025027588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf1c1048e0 msgr2=0x7fdf1c199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.819+0000 7fdf21000640 1 --2- 172.21.15.12:0/2025027588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf1c1048e0 0x7fdf1c199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.819+0000 7fdf21000640 1 -- 172.21.15.12:0/2025027588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf1c1052a0 msgr2=0x7fdf1c1a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.819+0000 7fdf21000640 1 --2- 172.21.15.12:0/2025027588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf1c1052a0 0x7fdf1c1a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:23.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.819+0000 7fdf21000640 1 -- 172.21.15.12:0/2025027588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1c1a15f0 con 0x7fdf1c1036f0 2024-03-23T23:11:23.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.820+0000 7fdf21000640 1 --2- 172.21.15.12:0/2025027588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf1c1036f0 0x7fdf1c1999d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fdf1002c260 tx=0x7fdf1002c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:23.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.820+0000 7fdf19600640 1 -- 172.21.15.12:0/2025027588 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf10041070 con 0x7fdf1c1036f0 2024-03-23T23:11:23.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.820+0000 7fdf22400640 1 -- 172.21.15.12:0/2025027588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf1c1a1880 con 0x7fdf1c1036f0 2024-03-23T23:11:23.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.820+0000 7fdf19600640 1 -- 172.21.15.12:0/2025027588 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdf10004450 con 0x7fdf1c1036f0 2024-03-23T23:11:23.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.820+0000 7fdf19600640 1 -- 172.21.15.12:0/2025027588 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf10034440 con 0x7fdf1c1036f0 2024-03-23T23:11:23.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.820+0000 7fdf22400640 1 -- 172.21.15.12:0/2025027588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf1c1a1e20 con 0x7fdf1c1036f0 2024-03-23T23:11:23.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.821+0000 7fdf22400640 1 -- 172.21.15.12:0/2025027588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf1c10aa10 con 0x7fdf1c1036f0 2024-03-23T23:11:23.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.823+0000 7fdf19600640 1 -- 172.21.15.12:0/2025027588 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fdf10044070 con 0x7fdf1c1036f0 2024-03-23T23:11:23.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.825+0000 7fdf19600640 1 --2- 172.21.15.12:0/2025027588 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdf040a2880 0x7fdf040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:23.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.825+0000 7fdf19600640 1 -- 172.21.15.12:0/2025027588 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fdf100f04d0 con 0x7fdf1c1036f0 2024-03-23T23:11:23.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.825+0000 7fdf1be00640 1 --2- 172.21.15.12:0/2025027588 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdf040a2880 0x7fdf040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:23.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.826+0000 7fdf1be00640 1 --2- 172.21.15.12:0/2025027588 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdf040a2880 0x7fdf040a4d40 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fdf080095f0 tx=0x7fdf08005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:23.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:23.829+0000 7fdf19600640 1 -- 172.21.15.12:0/2025027588 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdf100bced0 con 0x7fdf1c1036f0 2024-03-23T23:11:24.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.071+0000 7fdf22400640 1 -- 172.21.15.12:0/2025027588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf1c0622a0 con 0x7fdf1c1036f0 2024-03-23T23:11:24.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.073+0000 7fdf19600640 1 -- 172.21.15.12:0/2025027588 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7fdf100c0f70 con 0x7fdf1c1036f0 2024-03-23T23:11:24.072 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:11:24.072 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:11:24.072 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:11:24.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.075+0000 7fdf22400640 1 -- 172.21.15.12:0/2025027588 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdf040a2880 msgr2=0x7fdf040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.075+0000 7fdf22400640 1 --2- 172.21.15.12:0/2025027588 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdf040a2880 0x7fdf040a4d40 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fdf080095f0 tx=0x7fdf08005f90 comp rx=0 tx=0).stop 2024-03-23T23:11:24.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.075+0000 7fdf22400640 1 -- 172.21.15.12:0/2025027588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf1c1036f0 msgr2=0x7fdf1c1999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.075+0000 7fdf22400640 1 --2- 172.21.15.12:0/2025027588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf1c1036f0 0x7fdf1c1999d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fdf1002c260 tx=0x7fdf1002c670 comp rx=0 tx=0).stop 2024-03-23T23:11:24.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.075+0000 7fdf22400640 1 -- 172.21.15.12:0/2025027588 shutdown_connections 2024-03-23T23:11:24.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.075+0000 7fdf22400640 1 --2- 172.21.15.12:0/2025027588 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fdf040a2880 0x7fdf040a4d40 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.075+0000 7fdf22400640 1 --2- 172.21.15.12:0/2025027588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf1c1052a0 0x7fdf1c1a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.075+0000 7fdf22400640 1 --2- 172.21.15.12:0/2025027588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf1c1048e0 0x7fdf1c199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.075+0000 7fdf22400640 1 --2- 172.21.15.12:0/2025027588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf1c1036f0 0x7fdf1c1999d0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.075+0000 7fdf22400640 1 -- 172.21.15.12:0/2025027588 >> 172.21.15.12:0/2025027588 conn(0x7fdf1c0fec80 msgr2=0x7fdf1c107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:24.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.075+0000 7fdf22400640 1 -- 172.21.15.12:0/2025027588 shutdown_connections 2024-03-23T23:11:24.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.075+0000 7fdf22400640 1 -- 172.21.15.12:0/2025027588 wait complete. 2024-03-23T23:11:24.093 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:11:24.093 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-23T23:11:24.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.322+0000 7f209a200640 1 -- 172.21.15.12:0/3218582910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20941037c0 msgr2=0x7f2094103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.322+0000 7f209a200640 1 --2- 172.21.15.12:0/3218582910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20941037c0 0x7f2094103be0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2088009900 tx=0x7f208802bc40 comp rx=0 tx=0).stop 2024-03-23T23:11:24.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.322+0000 7f209a200640 1 -- 172.21.15.12:0/3218582910 shutdown_connections 2024-03-23T23:11:24.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.322+0000 7f209a200640 1 --2- 172.21.15.12:0/3218582910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2094105370 0x7f209410bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.322+0000 7f209a200640 1 --2- 172.21.15.12:0/3218582910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20941049b0 0x7f2094104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.322+0000 7f209a200640 1 --2- 172.21.15.12:0/3218582910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20941037c0 0x7f2094103be0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.322+0000 7f209a200640 1 -- 172.21.15.12:0/3218582910 >> 172.21.15.12:0/3218582910 conn(0x7f20940fed50 msgr2=0x7f2094101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:24.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.323+0000 7f209a200640 1 -- 172.21.15.12:0/3218582910 shutdown_connections 2024-03-23T23:11:24.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.323+0000 7f209a200640 1 -- 172.21.15.12:0/3218582910 wait complete. 2024-03-23T23:11:24.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.324+0000 7f209a200640 1 Processor -- start 2024-03-23T23:11:24.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.324+0000 7f209a200640 1 -- start start 2024-03-23T23:11:24.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.324+0000 7f209a200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20941037c0 0x7f2094199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.325+0000 7f209a200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f20941049b0 0x7f209419a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.325+0000 7f209a200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2094105370 0x7f20941a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.325+0000 7f209a200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f209410f460 con 0x7f2094105370 2024-03-23T23:11:24.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.325+0000 7f209a200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f209410f2e0 con 0x7f20941049b0 2024-03-23T23:11:24.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.325+0000 7f209a200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f209410f5e0 con 0x7f20941037c0 2024-03-23T23:11:24.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.325+0000 7f2098e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20941037c0 0x7f2094199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.325+0000 7f2093e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f20941049b0 0x7f209419a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.325+0000 7f2098e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20941037c0 0x7f2094199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:56328/0 (socket says 172.21.15.12:56328) 2024-03-23T23:11:24.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.325+0000 7f2098e00640 1 -- 172.21.15.12:0/2901705501 learned_addr learned my addr 172.21.15.12:0/2901705501 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:24.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.325+0000 7f2099800640 1 --2- 172.21.15.12:0/2901705501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2094105370 0x7f20941a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.325+0000 7f2093e00640 1 -- 172.21.15.12:0/2901705501 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20941037c0 msgr2=0x7f2094199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.325+0000 7f2093e00640 1 --2- 172.21.15.12:0/2901705501 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20941037c0 0x7f2094199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.325+0000 7f2093e00640 1 -- 172.21.15.12:0/2901705501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2094105370 msgr2=0x7f20941a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.325+0000 7f2093e00640 1 --2- 172.21.15.12:0/2901705501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2094105370 0x7f20941a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.325+0000 7f2093e00640 1 -- 172.21.15.12:0/2901705501 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20941a1730 con 0x7f20941049b0 2024-03-23T23:11:24.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.326+0000 7f2093e00640 1 --2- 172.21.15.12:0/2901705501 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f20941049b0 0x7f209419a050 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f208000e9f0 tx=0x7f208000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:24.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.326+0000 7f2091600640 1 -- 172.21.15.12:0/2901705501 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208000ccc0 con 0x7f20941049b0 2024-03-23T23:11:24.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.326+0000 7f2091600640 1 -- 172.21.15.12:0/2901705501 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f20800035f0 con 0x7f20941049b0 2024-03-23T23:11:24.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.326+0000 7f209a200640 1 -- 172.21.15.12:0/2901705501 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20941a1a20 con 0x7f20941049b0 2024-03-23T23:11:24.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.326+0000 7f2091600640 1 -- 172.21.15.12:0/2901705501 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208000ce60 con 0x7f20941049b0 2024-03-23T23:11:24.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.326+0000 7f209a200640 1 -- 172.21.15.12:0/2901705501 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20941a1f70 con 0x7f20941049b0 2024-03-23T23:11:24.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.329+0000 7f209e62f740 1 Processor -- start 2024-03-23T23:11:24.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.329+0000 7f209e62f740 1 -- start start 2024-03-23T23:11:24.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.329+0000 7f209e62f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620098659f0 0x562009865e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.329+0000 7f209e62f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562009866bc0 0x562009867040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.329+0000 7f209e62f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562009867580 0x56200986de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.330+0000 7f209e62f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620097ca2c0 con 0x562009866bc0 2024-03-23T23:11:24.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.330+0000 7f209e62f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x562009772c90 con 0x562009867580 2024-03-23T23:11:24.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.330+0000 7f209e62f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620097c6360 con 0x5620098659f0 2024-03-23T23:11:24.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.330+0000 7f2091600640 1 -- 172.21.15.12:0/2901705501 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f20800037a0 con 0x7f20941049b0 2024-03-23T23:11:24.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.330+0000 7f2099800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620098659f0 0x562009865e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.330+0000 7f2098e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562009866bc0 0x562009867040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.330+0000 7f2099800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620098659f0 0x562009865e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:56336/0 (socket says 172.21.15.12:56336) 2024-03-23T23:11:24.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.330+0000 7f2099800640 1 -- 172.21.15.12:0/326501059 learned_addr learned my addr 172.21.15.12:0/326501059 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:24.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.330+0000 7f2099800640 1 -- 172.21.15.12:0/326501059 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562009867580 msgr2=0x56200986de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:24.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.330+0000 7f2093e00640 1 --2- 172.21.15.12:0/326501059 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562009867580 0x56200986de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.330+0000 7f2099800640 1 --2- 172.21.15.12:0/326501059 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562009867580 0x56200986de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.330+0000 7f2099800640 1 -- 172.21.15.12:0/326501059 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562009866bc0 msgr2=0x562009867040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.330+0000 7f2099800640 1 --2- 172.21.15.12:0/326501059 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562009866bc0 0x562009867040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.330+0000 7f2099800640 1 -- 172.21.15.12:0/326501059 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562009797080 con 0x5620098659f0 2024-03-23T23:11:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.330+0000 7f2098e00640 1 --2- 172.21.15.12:0/326501059 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562009866bc0 0x562009867040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.330+0000 7f2093e00640 1 --2- 172.21.15.12:0/326501059 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562009867580 0x56200986de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:11:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.331+0000 7f2099800640 1 --2- 172.21.15.12:0/326501059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620098659f0 0x562009865e10 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f208c011440 tx=0x7f208c011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56271ef26c266891 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.331+0000 7f2086000640 1 -- 172.21.15.12:0/326501059 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208c003c30 con 0x5620098659f0 2024-03-23T23:11:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.331+0000 7f2086000640 1 -- 172.21.15.12:0/326501059 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f208c003dd0 con 0x5620098659f0 2024-03-23T23:11:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.331+0000 7f2086000640 1 -- 172.21.15.12:0/326501059 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208c00faf0 con 0x5620098659f0 2024-03-23T23:11:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.331+0000 7f2091600640 1 --2- 172.21.15.12:0/2901705501 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f207c0a2800 0x7f207c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.332+0000 7f209e62f740 1 -- 172.21.15.12:0/326501059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620098659f0 msgr2=0x562009865e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.332+0000 7f209e62f740 1 --2- 172.21.15.12:0/326501059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620098659f0 0x562009865e10 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f208c011440 tx=0x7f208c011900 comp rx=0 tx=0).stop 2024-03-23T23:11:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.332+0000 7f209e62f740 1 -- 172.21.15.12:0/326501059 shutdown_connections 2024-03-23T23:11:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.332+0000 7f209e62f740 1 --2- 172.21.15.12:0/326501059 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562009867580 0x56200986de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.332+0000 7f209e62f740 1 --2- 172.21.15.12:0/326501059 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562009866bc0 0x562009867040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.332+0000 7f209e62f740 1 --2- 172.21.15.12:0/326501059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620098659f0 0x562009865e10 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.332+0000 7f209e62f740 1 -- 172.21.15.12:0/326501059 >> 172.21.15.12:0/326501059 conn(0x562009860de0 msgr2=0x5620098611e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.332+0000 7f2091600640 1 -- 172.21.15.12:0/2901705501 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f20800cec80 con 0x7f20941049b0 2024-03-23T23:11:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.332+0000 7f209e62f740 1 -- 172.21.15.12:0/326501059 shutdown_connections 2024-03-23T23:11:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.332+0000 7f2098e00640 1 --2- 172.21.15.12:0/2901705501 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f207c0a2800 0x7f207c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.332+0000 7f209e62f740 1 -- 172.21.15.12:0/326501059 wait complete. 2024-03-23T23:11:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.332+0000 7f209e62f740 1 -- start start 2024-03-23T23:11:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.333+0000 7f2098e00640 1 --2- 172.21.15.12:0/2901705501 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f207c0a2800 0x7f207c0a4cc0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f2088005ab0 tx=0x7f208802e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.333+0000 7f209e62f740 1 Processor -- start 2024-03-23T23:11:24.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.333+0000 7f209e62f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5620098659f0 0x56200986fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.333+0000 7f209e62f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562009866bc0 0x562009870180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.333+0000 7f209e62f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562009867580 0x5620098771c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.333+0000 7f209e62f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56200986e590 con 0x562009866bc0 2024-03-23T23:11:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.333+0000 7f209e62f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620097860d0 con 0x5620098659f0 2024-03-23T23:11:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.333+0000 7f209e62f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x56200986e710 con 0x562009867580 2024-03-23T23:11:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.333+0000 7f209e62f740 10 client.0 _ll_register_callbacks cb 0x56200978ae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.333+0000 7f2098e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5620098659f0 0x56200986fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.333+0000 7f2093e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562009866bc0 0x562009870180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.333+0000 7f2099800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562009867580 0x5620098771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.333+0000 7f2098e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5620098659f0 0x56200986fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:41510/0 (socket says 172.21.15.12:41510) 2024-03-23T23:11:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.334+0000 7f2098e00640 1 -- 172.21.15.12:0/272964996 learned_addr learned my addr 172.21.15.12:0/272964996 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.334+0000 7f2099800640 1 -- 172.21.15.12:0/272964996 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5620098659f0 msgr2=0x56200986fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.334+0000 7f2099800640 1 --2- 172.21.15.12:0/272964996 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5620098659f0 0x56200986fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.334+0000 7f2099800640 1 -- 172.21.15.12:0/272964996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562009866bc0 msgr2=0x562009870180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.334+0000 7f2099800640 1 --2- 172.21.15.12:0/272964996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562009866bc0 0x562009870180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.334+0000 7f2099800640 1 -- 172.21.15.12:0/272964996 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5620094de3f0 con 0x562009867580 2024-03-23T23:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.334+0000 7f2099800640 1 --2- 172.21.15.12:0/272964996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562009867580 0x5620098771c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f208c000f80 tx=0x7f208c00f5b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29848fdf55f16158 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.334+0000 7f207aa00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.334+0000 7f207aa00640 1 -- 172.21.15.12:0/272964996 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208c029020 con 0x562009867580 2024-03-23T23:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.335+0000 7f209e62f740 1 -- 172.21.15.12:0/272964996 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f208c003dd0 con 0x562009867580 2024-03-23T23:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.335+0000 7f207aa00640 1 -- 172.21.15.12:0/272964996 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f208c00fc80 con 0x562009867580 2024-03-23T23:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.335+0000 7f207aa00640 1 -- 172.21.15.12:0/272964996 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f208c00fe80 con 0x562009867580 2024-03-23T23:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.335+0000 7f209e62f740 10 client.27949 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.335+0000 7f209e62f740 1 -- 172.21.15.12:0/272964996 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5620095a87d0 con 0x562009867580 2024-03-23T23:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.335+0000 7f207aa00640 1 -- 172.21.15.12:0/272964996 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f208c003dd0 con 0x562009867580 2024-03-23T23:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.335+0000 7f207aa00640 1 -- 172.21.15.12:0/272964996 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f208c005340 con 0x562009867580 2024-03-23T23:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.335+0000 7f209e62f740 10 client.27949 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.336+0000 7f209e62f740 10 client.27949 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.336+0000 7f209e62f740 20 client.27949 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.336+0000 7f209e62f740 1 --2- 172.21.15.12:0/272964996 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x56200987b3e0 0x56200987d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.336+0000 7f209e62f740 4 client.27949 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:11:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.336+0000 7f209e62f740 1 -- 172.21.15.12:0/272964996 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5620097d7800 con 0x56200987b3e0 2024-03-23T23:11:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.336+0000 7f2098e00640 1 --2- 172.21.15.12:0/272964996 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x56200987b3e0 0x56200987d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.336+0000 7f2098e00640 1 --2- 172.21.15.12:0/272964996 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x56200987b3e0 0x56200987d7d0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=288e0a1a683b280f server_cookie=e61e640a11fdc712 in_seq=0 out_seq=0 2024-03-23T23:11:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.336+0000 7f207aa00640 10 client.27949 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:11:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.337+0000 7f207aa00640 1 -- 172.21.15.12:0/272964996 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5620097d7800 con 0x56200987b3e0 2024-03-23T23:11:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.337+0000 7f207aa00640 10 client.27949 handle_command_reply: tid=0 2024-03-23T23:11:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.337+0000 7f209e62f740 1 client.27949 shutdown 2024-03-23T23:11:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.337+0000 7f209e62f740 1 -- 172.21.15.12:0/272964996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562009867580 msgr2=0x5620098771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.337+0000 7f209e62f740 1 --2- 172.21.15.12:0/272964996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562009867580 0x5620098771c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f208c000f80 tx=0x7f208c00f5b0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.337+0000 7f209e62f740 1 -- 172.21.15.12:0/272964996 shutdown_connections 2024-03-23T23:11:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.337+0000 7f209e62f740 1 --2- 172.21.15.12:0/272964996 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x56200987b3e0 0x56200987d7d0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.337+0000 7f209e62f740 1 --2- 172.21.15.12:0/272964996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562009867580 0x5620098771c0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.337+0000 7f209e62f740 1 --2- 172.21.15.12:0/272964996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562009866bc0 0x562009870180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.337+0000 7f207aa00640 0 client.27949 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:11:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.337+0000 7f209e62f740 1 --2- 172.21.15.12:0/272964996 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5620098659f0 0x56200986fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.337+0000 7f209e62f740 1 -- 172.21.15.12:0/272964996 >> 172.21.15.12:0/272964996 conn(0x562009860de0 msgr2=0x562009860160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.338+0000 7f209e62f740 1 -- 172.21.15.12:0/272964996 shutdown_connections 2024-03-23T23:11:24.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.338+0000 7f209e62f740 1 -- 172.21.15.12:0/272964996 wait complete. 2024-03-23T23:11:24.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.338+0000 7f209e62f740 20 client.27949 trim_cache size 0 max 16384 2024-03-23T23:11:24.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.351+0000 7f209e62f740 1 Processor -- start 2024-03-23T23:11:24.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.351+0000 7f209e62f740 1 -- start start 2024-03-23T23:11:24.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.351+0000 7f209e62f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620098659f0 0x562009864750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.351+0000 7f209e62f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562009866bc0 0x562009877180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.351+0000 7f209e62f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562009867580 0x562009882040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.351+0000 7f209e62f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20941acfd0 con 0x562009867580 2024-03-23T23:11:24.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.351+0000 7f209e62f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20941aaad0 con 0x562009866bc0 2024-03-23T23:11:24.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.351+0000 7f209e62f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20941a8b70 con 0x5620098659f0 2024-03-23T23:11:24.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.351+0000 7f2098e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620098659f0 0x562009864750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.351+0000 7f2098e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620098659f0 0x562009864750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:56360/0 (socket says 172.21.15.12:56360) 2024-03-23T23:11:24.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.351+0000 7f2098e00640 1 -- 172.21.15.12:0/4284102392 learned_addr learned my addr 172.21.15.12:0/4284102392 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:24.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.352+0000 7f2093e00640 1 --2- 172.21.15.12:0/4284102392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562009866bc0 0x562009877180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.352+0000 7f2099800640 1 --2- 172.21.15.12:0/4284102392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562009867580 0x562009882040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.352+0000 7f2098e00640 1 -- 172.21.15.12:0/4284102392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562009866bc0 msgr2=0x562009877180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:24.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.352+0000 7f2098e00640 1 --2- 172.21.15.12:0/4284102392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562009866bc0 0x562009877180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.352+0000 7f2098e00640 1 -- 172.21.15.12:0/4284102392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562009867580 msgr2=0x562009882040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:24.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.352+0000 7f2098e00640 1 --2- 172.21.15.12:0/4284102392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562009867580 0x562009882040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.352+0000 7f2098e00640 1 -- 172.21.15.12:0/4284102392 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562009867960 con 0x5620098659f0 2024-03-23T23:11:24.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.352+0000 7f2098e00640 1 --2- 172.21.15.12:0/4284102392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620098659f0 0x562009864750 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f208802c8f0 tx=0x7f2088041070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e6a7ed749b35ed5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:24.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.352+0000 7f2085600640 1 -- 172.21.15.12:0/4284102392 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20880046c0 con 0x5620098659f0 2024-03-23T23:11:24.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.353+0000 7f2085600640 1 -- 172.21.15.12:0/4284102392 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f208805a070 con 0x5620098659f0 2024-03-23T23:11:24.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.353+0000 7f2085600640 1 -- 172.21.15.12:0/4284102392 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20880035f0 con 0x5620098659f0 2024-03-23T23:11:24.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.353+0000 7f209e62f740 1 -- 172.21.15.12:0/4284102392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620098659f0 msgr2=0x562009864750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.353+0000 7f209e62f740 1 --2- 172.21.15.12:0/4284102392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620098659f0 0x562009864750 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f208802c8f0 tx=0x7f2088041070 comp rx=0 tx=0).stop 2024-03-23T23:11:24.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.353+0000 7f209e62f740 1 -- 172.21.15.12:0/4284102392 shutdown_connections 2024-03-23T23:11:24.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.353+0000 7f209e62f740 1 --2- 172.21.15.12:0/4284102392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562009867580 0x562009882040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.353+0000 7f209e62f740 1 --2- 172.21.15.12:0/4284102392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562009866bc0 0x562009877180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.353+0000 7f209e62f740 1 --2- 172.21.15.12:0/4284102392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620098659f0 0x562009864750 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.353+0000 7f209e62f740 1 -- 172.21.15.12:0/4284102392 >> 172.21.15.12:0/4284102392 conn(0x562009860de0 msgr2=0x562009864040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:24.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.353+0000 7f209e62f740 1 -- 172.21.15.12:0/4284102392 shutdown_connections 2024-03-23T23:11:24.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.353+0000 7f209e62f740 1 -- 172.21.15.12:0/4284102392 wait complete. 2024-03-23T23:11:24.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.354+0000 7f209e62f740 1 -- start start 2024-03-23T23:11:24.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.354+0000 7f209e62f740 1 Processor -- start 2024-03-23T23:11:24.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.354+0000 7f209e62f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620098659f0 0x562009872ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.354+0000 7f209e62f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562009866bc0 0x562009873020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.354+0000 7f209e62f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562009867580 0x562009876040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.354+0000 7f209e62f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20941a9170 con 0x562009867580 2024-03-23T23:11:24.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.354+0000 7f209e62f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20941aa470 con 0x562009866bc0 2024-03-23T23:11:24.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.354+0000 7f209e62f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20941a97b0 con 0x5620098659f0 2024-03-23T23:11:24.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.354+0000 7f209e62f740 10 client.0 _ll_register_callbacks cb 0x7f2080005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:24.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.354+0000 7f2098e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620098659f0 0x562009872ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.354+0000 7f2099800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562009867580 0x562009876040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.354+0000 7f2098e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620098659f0 0x562009872ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:56362/0 (socket says 172.21.15.12:56362) 2024-03-23T23:11:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.354+0000 7f2098e00640 1 -- 172.21.15.12:0/432183778 learned_addr learned my addr 172.21.15.12:0/432183778 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.354+0000 7f2093e00640 1 --2- 172.21.15.12:0/432183778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562009866bc0 0x562009873020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.355+0000 7f2098e00640 1 -- 172.21.15.12:0/432183778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562009866bc0 msgr2=0x562009873020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.355+0000 7f2098e00640 1 --2- 172.21.15.12:0/432183778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562009866bc0 0x562009873020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.355+0000 7f2098e00640 1 -- 172.21.15.12:0/432183778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562009867580 msgr2=0x562009876040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.355+0000 7f2098e00640 1 --2- 172.21.15.12:0/432183778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562009867580 0x562009876040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.355+0000 7f2098e00640 1 -- 172.21.15.12:0/432183778 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562009876630 con 0x5620098659f0 2024-03-23T23:11:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.355+0000 7f2093e00640 1 --2- 172.21.15.12:0/432183778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562009866bc0 0x562009873020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.355+0000 7f2098e00640 1 --2- 172.21.15.12:0/432183778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620098659f0 0x562009872ae0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f20880092a0 tx=0x7f2088004440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=830b6096b3b940f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.355+0000 7f207aa00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:11:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.355+0000 7f207aa00640 1 -- 172.21.15.12:0/432183778 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2088005700 con 0x5620098659f0 2024-03-23T23:11:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.355+0000 7f207aa00640 1 -- 172.21.15.12:0/432183778 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f20880046c0 con 0x5620098659f0 2024-03-23T23:11:24.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.355+0000 7f209e62f740 1 -- 172.21.15.12:0/432183778 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f208805a070 con 0x5620098659f0 2024-03-23T23:11:24.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.356+0000 7f207aa00640 1 -- 172.21.15.12:0/432183778 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f20880048c0 con 0x5620098659f0 2024-03-23T23:11:24.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.356+0000 7f209e62f740 10 client.27955 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:24.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.356+0000 7f209e62f740 1 -- 172.21.15.12:0/432183778 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5620098d6e20 con 0x5620098659f0 2024-03-23T23:11:24.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.356+0000 7f207aa00640 1 -- 172.21.15.12:0/432183778 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f208805a070 con 0x5620098659f0 2024-03-23T23:11:24.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.356+0000 7f207aa00640 1 -- 172.21.15.12:0/432183778 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f2088034440 con 0x5620098659f0 2024-03-23T23:11:24.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.356+0000 7f209e62f740 10 client.27955 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:24.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.356+0000 7f209e62f740 10 client.27955 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:11:24.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.356+0000 7f209e62f740 20 client.27955 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:24.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.356+0000 7f209e62f740 1 --2- 172.21.15.12:0/432183778 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5620098d7630 0x5620098d9a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.356+0000 7f209e62f740 4 client.27955 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:11:24.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.356+0000 7f209e62f740 1 -- 172.21.15.12:0/432183778 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56200986fa70 con 0x5620098d7630 2024-03-23T23:11:24.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.356+0000 7f2093e00640 1 --2- 172.21.15.12:0/432183778 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5620098d7630 0x5620098d9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.357+0000 7f2093e00640 1 --2- 172.21.15.12:0/432183778 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5620098d7630 0x5620098d9a90 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=27415c17cb0550db server_cookie=3ff2bc60b290dbef in_seq=0 out_seq=0 2024-03-23T23:11:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.357+0000 7f207aa00640 10 client.27955 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:11:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.357+0000 7f207aa00640 1 -- 172.21.15.12:0/432183778 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56200986fa70 con 0x5620098d7630 2024-03-23T23:11:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.357+0000 7f207aa00640 10 client.27955 handle_command_reply: tid=0 2024-03-23T23:11:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.357+0000 7f209e62f740 1 client.27955 shutdown 2024-03-23T23:11:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.357+0000 7f209e62f740 1 -- 172.21.15.12:0/432183778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620098659f0 msgr2=0x562009872ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.357+0000 7f209e62f740 1 --2- 172.21.15.12:0/432183778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620098659f0 0x562009872ae0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f20880092a0 tx=0x7f2088004440 comp rx=0 tx=0).stop 2024-03-23T23:11:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.357+0000 7f209e62f740 1 -- 172.21.15.12:0/432183778 shutdown_connections 2024-03-23T23:11:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.357+0000 7f209e62f740 1 --2- 172.21.15.12:0/432183778 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x5620098d7630 0x5620098d9a90 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.357+0000 7f209e62f740 1 --2- 172.21.15.12:0/432183778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562009867580 0x562009876040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.357+0000 7f209e62f740 1 --2- 172.21.15.12:0/432183778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562009866bc0 0x562009873020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.357+0000 7f209e62f740 1 --2- 172.21.15.12:0/432183778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620098659f0 0x562009872ae0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.357+0000 7f207aa00640 0 client.27955 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:11:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.357+0000 7f209e62f740 1 -- 172.21.15.12:0/432183778 >> 172.21.15.12:0/432183778 conn(0x562009860de0 msgr2=0x5620098648d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.357+0000 7f209e62f740 1 -- 172.21.15.12:0/432183778 shutdown_connections 2024-03-23T23:11:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.358+0000 7f209e62f740 1 -- 172.21.15.12:0/432183778 wait complete. 2024-03-23T23:11:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.358+0000 7f209e62f740 20 client.27955 trim_cache size 0 max 16384 2024-03-23T23:11:24.362 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:11:24.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.358+0000 7f209a200640 1 -- 172.21.15.12:0/2901705501 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f207c0a2800 msgr2=0x7f207c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.358+0000 7f209a200640 1 --2- 172.21.15.12:0/2901705501 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f207c0a2800 0x7f207c0a4cc0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f2088005ab0 tx=0x7f208802e040 comp rx=0 tx=0).stop 2024-03-23T23:11:24.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.358+0000 7f209a200640 1 -- 172.21.15.12:0/2901705501 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f20941049b0 msgr2=0x7f209419a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.358+0000 7f209a200640 1 --2- 172.21.15.12:0/2901705501 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f20941049b0 0x7f209419a050 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f208000e9f0 tx=0x7f208000edf0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.358+0000 7f209a200640 1 -- 172.21.15.12:0/2901705501 shutdown_connections 2024-03-23T23:11:24.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.358+0000 7f209a200640 1 --2- 172.21.15.12:0/2901705501 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f207c0a2800 0x7f207c0a4cc0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.358+0000 7f209a200640 1 --2- 172.21.15.12:0/2901705501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2094105370 0x7f20941a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.358+0000 7f209a200640 1 --2- 172.21.15.12:0/2901705501 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f20941049b0 0x7f209419a050 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.358+0000 7f209a200640 1 --2- 172.21.15.12:0/2901705501 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20941037c0 0x7f2094199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.358+0000 7f209a200640 1 -- 172.21.15.12:0/2901705501 >> 172.21.15.12:0/2901705501 conn(0x7f20940fed50 msgr2=0x7f2094107d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:24.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.358+0000 7f209a200640 1 -- 172.21.15.12:0/2901705501 shutdown_connections 2024-03-23T23:11:24.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.358+0000 7f209a200640 1 -- 172.21.15.12:0/2901705501 wait complete. 2024-03-23T23:11:24.376 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:11:24.376 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-23T23:11:24.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.586+0000 7fe9ae200640 1 -- 172.21.15.12:0/823320001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9a8101210 msgr2=0x7fe9a8101690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.586+0000 7fe9ae200640 1 --2- 172.21.15.12:0/823320001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9a8101210 0x7fe9a8101690 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe9a0009ca0 tx=0x7fe9a002c100 comp rx=0 tx=0).stop 2024-03-23T23:11:24.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.586+0000 7fe9ae200640 1 -- 172.21.15.12:0/823320001 shutdown_connections 2024-03-23T23:11:24.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.586+0000 7fe9ae200640 1 --2- 172.21.15.12:0/823320001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9a8101210 0x7fe9a8101690 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.586+0000 7fe9ae200640 1 --2- 172.21.15.12:0/823320001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9a8100870 0x7fe9a8100cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.586+0000 7fe9ae200640 1 --2- 172.21.15.12:0/823320001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9a80fff10 0x7fe9a8100330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.586+0000 7fe9ae200640 1 -- 172.21.15.12:0/823320001 >> 172.21.15.12:0/823320001 conn(0x7fe9a80fbb30 msgr2=0x7fe9a80fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:24.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.586+0000 7fe9ae200640 1 -- 172.21.15.12:0/823320001 shutdown_connections 2024-03-23T23:11:24.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.586+0000 7fe9ae200640 1 -- 172.21.15.12:0/823320001 wait complete. 2024-03-23T23:11:24.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.587+0000 7fe9ae200640 1 Processor -- start 2024-03-23T23:11:24.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.587+0000 7fe9ae200640 1 -- start start 2024-03-23T23:11:24.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.588+0000 7fe9ae200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9a80fff10 0x7fe9a8199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.588+0000 7fe9ae200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9a8100870 0x7fe9a819a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.588+0000 7fe9ae200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9a8101210 0x7fe9a81a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.588+0000 7fe9ae200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a810f510 con 0x7fe9a8100870 2024-03-23T23:11:24.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.588+0000 7fe9ae200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a810f390 con 0x7fe9a80fff10 2024-03-23T23:11:24.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.588+0000 7fe9ae200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a810f690 con 0x7fe9a8101210 2024-03-23T23:11:24.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.588+0000 7fe9a6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9a8100870 0x7fe9a819a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.588+0000 7fe9a7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9a80fff10 0x7fe9a8199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.588+0000 7fe9a7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9a80fff10 0x7fe9a8199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:41560/0 (socket says 172.21.15.12:41560) 2024-03-23T23:11:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.588+0000 7fe9a7400640 1 -- 172.21.15.12:0/4190308730 learned_addr learned my addr 172.21.15.12:0/4190308730 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.588+0000 7fe9a7e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9a8101210 0x7fe9a81a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.588+0000 7fe9a6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9a8100870 0x7fe9a819a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44982/0 (socket says 172.21.15.12:44982) 2024-03-23T23:11:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.588+0000 7fe9a7400640 1 -- 172.21.15.12:0/4190308730 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9a8101210 msgr2=0x7fe9a81a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.588+0000 7fe9a7400640 1 --2- 172.21.15.12:0/4190308730 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9a8101210 0x7fe9a81a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.588+0000 7fe9a7400640 1 -- 172.21.15.12:0/4190308730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9a8100870 msgr2=0x7fe9a819a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.588+0000 7fe9a7400640 1 --2- 172.21.15.12:0/4190308730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9a8100870 0x7fe9a819a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.588+0000 7fe9a7400640 1 -- 172.21.15.12:0/4190308730 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a81a17a0 con 0x7fe9a80fff10 2024-03-23T23:11:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.589+0000 7fe9a7400640 1 --2- 172.21.15.12:0/4190308730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9a80fff10 0x7fe9a8199b80 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fe99800e9f0 tx=0x7fe99800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.589+0000 7fe99fe00640 1 -- 172.21.15.12:0/4190308730 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99800cd80 con 0x7fe9a80fff10 2024-03-23T23:11:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.589+0000 7fe99fe00640 1 -- 172.21.15.12:0/4190308730 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe998005500 con 0x7fe9a80fff10 2024-03-23T23:11:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.589+0000 7fe99fe00640 1 -- 172.21.15.12:0/4190308730 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe998010690 con 0x7fe9a80fff10 2024-03-23T23:11:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.589+0000 7fe9ae200640 1 -- 172.21.15.12:0/4190308730 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a81a1a90 con 0x7fe9a80fff10 2024-03-23T23:11:24.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.589+0000 7fe9ae200640 1 -- 172.21.15.12:0/4190308730 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a81a1f90 con 0x7fe9a80fff10 2024-03-23T23:11:24.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.592+0000 7fe9b0f71740 1 Processor -- start 2024-03-23T23:11:24.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.592+0000 7fe99fe00640 1 -- 172.21.15.12:0/4190308730 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe9980046c0 con 0x7fe9a80fff10 2024-03-23T23:11:24.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.592+0000 7fe9b0f71740 1 -- start start 2024-03-23T23:11:24.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.593+0000 7fe9b0f71740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e7038d810 0x555e7038dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.593+0000 7fe9b0f71740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e7038e9e0 0x555e7038ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.593+0000 7fe9b0f71740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555e7038f3a0 0x555e70395c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.593+0000 7fe9b0f71740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e702f22c0 con 0x555e7038d810 2024-03-23T23:11:24.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.593+0000 7fe9b0f71740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e7029ac90 con 0x555e7038e9e0 2024-03-23T23:11:24.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.593+0000 7fe9b0f71740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e702ee360 con 0x555e7038f3a0 2024-03-23T23:11:24.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.593+0000 7fe9a7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e7038e9e0 0x555e7038ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.593+0000 7fe9a7e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e7038d810 0x555e7038dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.593+0000 7fe9a7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e7038e9e0 0x555e7038ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:41574/0 (socket says 172.21.15.12:41574) 2024-03-23T23:11:24.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.593+0000 7fe9a7400640 1 -- 172.21.15.12:0/93693300 learned_addr learned my addr 172.21.15.12:0/93693300 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:24.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.593+0000 7fe9a6a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555e7038f3a0 0x555e70395c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.593+0000 7fe9a7400640 1 -- 172.21.15.12:0/93693300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555e7038f3a0 msgr2=0x555e70395c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:24.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.593+0000 7fe9a7400640 1 --2- 172.21.15.12:0/93693300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555e7038f3a0 0x555e70395c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.593+0000 7fe9a7400640 1 -- 172.21.15.12:0/93693300 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e7038d810 msgr2=0x555e7038dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:24.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.593+0000 7fe9a7400640 1 --2- 172.21.15.12:0/93693300 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e7038d810 0x555e7038dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.593+0000 7fe9a7400640 1 -- 172.21.15.12:0/93693300 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e702bf080 con 0x555e7038e9e0 2024-03-23T23:11:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.594+0000 7fe9a7400640 1 --2- 172.21.15.12:0/93693300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e7038e9e0 0x555e7038ee60 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fe9980d2db0 tx=0x7fe9980d2ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9353379a2b84432d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.594+0000 7fe997e00640 1 -- 172.21.15.12:0/93693300 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9980e2070 con 0x555e7038e9e0 2024-03-23T23:11:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.594+0000 7fe997e00640 1 -- 172.21.15.12:0/93693300 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe9980d7bc0 con 0x555e7038e9e0 2024-03-23T23:11:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.594+0000 7fe997e00640 1 -- 172.21.15.12:0/93693300 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9980dc5e0 con 0x555e7038e9e0 2024-03-23T23:11:24.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.594+0000 7fe99fe00640 1 --2- 172.21.15.12:0/4190308730 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe98c0a2800 0x7fe98c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.594+0000 7fe99fe00640 1 -- 172.21.15.12:0/4190308730 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe9980ceea0 con 0x7fe9a80fff10 2024-03-23T23:11:24.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.594+0000 7fe9a7e00640 1 --2- 172.21.15.12:0/4190308730 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe98c0a2800 0x7fe98c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.594+0000 7fe9b0f71740 1 -- 172.21.15.12:0/93693300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e7038e9e0 msgr2=0x555e7038ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.594+0000 7fe9b0f71740 1 --2- 172.21.15.12:0/93693300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e7038e9e0 0x555e7038ee60 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fe9980d2db0 tx=0x7fe9980d2ea0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.595+0000 7fe9b0f71740 1 -- 172.21.15.12:0/93693300 shutdown_connections 2024-03-23T23:11:24.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.595+0000 7fe9b0f71740 1 --2- 172.21.15.12:0/93693300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555e7038f3a0 0x555e70395c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.595+0000 7fe9b0f71740 1 --2- 172.21.15.12:0/93693300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e7038e9e0 0x555e7038ee60 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.595+0000 7fe9b0f71740 1 --2- 172.21.15.12:0/93693300 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e7038d810 0x555e7038dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.595+0000 7fe9b0f71740 1 -- 172.21.15.12:0/93693300 >> 172.21.15.12:0/93693300 conn(0x555e70388c00 msgr2=0x555e70389000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:24.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.595+0000 7fe9b0f71740 1 -- 172.21.15.12:0/93693300 shutdown_connections 2024-03-23T23:11:24.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.595+0000 7fe9b0f71740 1 -- 172.21.15.12:0/93693300 wait complete. 2024-03-23T23:11:24.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.595+0000 7fe9a7e00640 1 --2- 172.21.15.12:0/4190308730 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe98c0a2800 0x7fe98c0a4cc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe9a002c640 tx=0x7fe9a0032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:24.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.595+0000 7fe9b0f71740 1 -- start start 2024-03-23T23:11:24.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.595+0000 7fe9b0f71740 1 Processor -- start 2024-03-23T23:11:24.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.596+0000 7fe9b0f71740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555e7038d810 0x555e70397a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.596+0000 7fe9b0f71740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e7038e9e0 0x555e70397fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.596+0000 7fe9b0f71740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e7038f3a0 0x555e7039efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.596+0000 7fe9b0f71740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e703963b0 con 0x555e7038e9e0 2024-03-23T23:11:24.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.596+0000 7fe9b0f71740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e702ae0d0 con 0x555e7038f3a0 2024-03-23T23:11:24.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.596+0000 7fe9b0f71740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e70396530 con 0x555e7038d810 2024-03-23T23:11:24.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.596+0000 7fe9b0f71740 10 client.0 _ll_register_callbacks cb 0x555e702b2e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:24.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.596+0000 7fe9a7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e7038f3a0 0x555e7039efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.596+0000 7fe9a6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e7038e9e0 0x555e70397fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.596+0000 7fe9a7400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555e7038d810 0x555e70397a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.596+0000 7fe9a7400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555e7038d810 0x555e70397a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:56394/0 (socket says 172.21.15.12:56394) 2024-03-23T23:11:24.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.596+0000 7fe9a7400640 1 -- 172.21.15.12:0/764481194 learned_addr learned my addr 172.21.15.12:0/764481194 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:24.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.596+0000 7fe9a7400640 1 -- 172.21.15.12:0/764481194 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e7038f3a0 msgr2=0x555e7039efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:24.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.596+0000 7fe9a7400640 1 --2- 172.21.15.12:0/764481194 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e7038f3a0 0x555e7039efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.597+0000 7fe9a7400640 1 -- 172.21.15.12:0/764481194 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e7038e9e0 msgr2=0x555e70397fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.597+0000 7fe9a7400640 1 --2- 172.21.15.12:0/764481194 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e7038e9e0 0x555e70397fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.597+0000 7fe9a7400640 1 -- 172.21.15.12:0/764481194 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555e700063f0 con 0x555e7038d810 2024-03-23T23:11:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.597+0000 7fe9a7400640 1 --2- 172.21.15.12:0/764481194 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555e7038d810 0x555e70397a60 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fe9980981c0 tx=0x7fe998009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=266cbf84417092de server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.597+0000 7fe994c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:11:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.597+0000 7fe994c00640 1 -- 172.21.15.12:0/764481194 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe998097a90 con 0x555e7038d810 2024-03-23T23:11:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.597+0000 7fe994c00640 1 -- 172.21.15.12:0/764481194 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe9980cca60 con 0x555e7038d810 2024-03-23T23:11:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.597+0000 7fe9b0f71740 1 -- 172.21.15.12:0/764481194 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe9980d7bc0 con 0x555e7038d810 2024-03-23T23:11:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.597+0000 7fe994c00640 1 -- 172.21.15.12:0/764481194 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe99809b150 con 0x555e7038d810 2024-03-23T23:11:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.597+0000 7fe9b0f71740 10 client.27967 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.597+0000 7fe9b0f71740 1 -- 172.21.15.12:0/764481194 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x555e700d07d0 con 0x555e7038d810 2024-03-23T23:11:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.597+0000 7fe994c00640 1 -- 172.21.15.12:0/764481194 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe9980d7bc0 con 0x555e7038d810 2024-03-23T23:11:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.598+0000 7fe994c00640 1 -- 172.21.15.12:0/764481194 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe9980d1030 con 0x555e7038d810 2024-03-23T23:11:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.598+0000 7fe9b0f71740 10 client.27967 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.598+0000 7fe9b0f71740 10 client.27967 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:11:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.598+0000 7fe9b0f71740 20 client.27967 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.598+0000 7fe9b0f71740 1 --2- 172.21.15.12:0/764481194 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x555e703a3140 0x555e703a5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.598+0000 7fe9b0f71740 4 client.27967 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:11:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.598+0000 7fe9b0f71740 1 -- 172.21.15.12:0/764481194 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555e702ff800 con 0x555e703a3140 2024-03-23T23:11:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.598+0000 7fe9a6a00640 1 --2- 172.21.15.12:0/764481194 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x555e703a3140 0x555e703a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.599+0000 7fe9a6a00640 1 --2- 172.21.15.12:0/764481194 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x555e703a3140 0x555e703a5530 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a8059f2bf2164042 server_cookie=b853462864fbcb78 in_seq=0 out_seq=0 2024-03-23T23:11:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.599+0000 7fe994c00640 10 client.27967 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:11:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.599+0000 7fe994c00640 1 -- 172.21.15.12:0/764481194 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555e702ff800 con 0x555e703a3140 2024-03-23T23:11:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.599+0000 7fe994c00640 10 client.27967 handle_command_reply: tid=0 2024-03-23T23:11:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.599+0000 7fe9b0f71740 1 client.27967 shutdown 2024-03-23T23:11:24.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.599+0000 7fe9b0f71740 1 -- 172.21.15.12:0/764481194 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555e7038d810 msgr2=0x555e70397a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.599+0000 7fe9b0f71740 1 --2- 172.21.15.12:0/764481194 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555e7038d810 0x555e70397a60 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fe9980981c0 tx=0x7fe998009310 comp rx=0 tx=0).stop 2024-03-23T23:11:24.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.600+0000 7fe9b0f71740 1 -- 172.21.15.12:0/764481194 shutdown_connections 2024-03-23T23:11:24.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.600+0000 7fe9b0f71740 1 --2- 172.21.15.12:0/764481194 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e7038f3a0 0x555e7039efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.600+0000 7fe9b0f71740 1 --2- 172.21.15.12:0/764481194 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e7038e9e0 0x555e70397fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.600+0000 7fe9b0f71740 1 --2- 172.21.15.12:0/764481194 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x555e703a3140 0x555e703a5530 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.600+0000 7fe9b0f71740 1 --2- 172.21.15.12:0/764481194 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555e7038d810 0x555e70397a60 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.600+0000 7fe9b0f71740 1 -- 172.21.15.12:0/764481194 >> 172.21.15.12:0/764481194 conn(0x555e70388c00 msgr2=0x555e70387eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:24.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.600+0000 7fe994c00640 0 client.27967 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:11:24.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.600+0000 7fe9b0f71740 1 -- 172.21.15.12:0/764481194 shutdown_connections 2024-03-23T23:11:24.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.600+0000 7fe9b0f71740 1 -- 172.21.15.12:0/764481194 wait complete. 2024-03-23T23:11:24.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.600+0000 7fe9b0f71740 20 client.27967 trim_cache size 0 max 16384 2024-03-23T23:11:24.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.613+0000 7fe9b0f71740 1 Processor -- start 2024-03-23T23:11:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.613+0000 7fe9b0f71740 1 -- start start 2024-03-23T23:11:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.614+0000 7fe9b0f71740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555e7038d810 0x555e70391a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.614+0000 7fe9b0f71740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e7038e9e0 0x555e7039efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.614+0000 7fe9b0f71740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e7038f3a0 0x555e70391fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.614+0000 7fe9b0f71740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a81ad1a0 con 0x555e7038e9e0 2024-03-23T23:11:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.614+0000 7fe9b0f71740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a81aaca0 con 0x555e7038f3a0 2024-03-23T23:11:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.614+0000 7fe9b0f71740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a81a8d40 con 0x555e7038d810 2024-03-23T23:11:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.614+0000 7fe9a6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e7038e9e0 0x555e7039efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.614+0000 7fe9a7400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555e7038d810 0x555e70391a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.614+0000 7fe9a7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e7038f3a0 0x555e70391fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.614+0000 7fe9a6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e7038e9e0 0x555e7039efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45014/0 (socket says 172.21.15.12:45014) 2024-03-23T23:11:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.614+0000 7fe9a6a00640 1 -- 172.21.15.12:0/3516714370 learned_addr learned my addr 172.21.15.12:0/3516714370 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.614+0000 7fe9a6a00640 1 -- 172.21.15.12:0/3516714370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555e7038d810 msgr2=0x555e70391a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.614+0000 7fe9a6a00640 1 --2- 172.21.15.12:0/3516714370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555e7038d810 0x555e70391a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.614+0000 7fe9a6a00640 1 -- 172.21.15.12:0/3516714370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e7038f3a0 msgr2=0x555e70391fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.614+0000 7fe9a6a00640 1 --2- 172.21.15.12:0/3516714370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e7038f3a0 0x555e70391fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.614+0000 7fe9a6a00640 1 -- 172.21.15.12:0/3516714370 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e70395f70 con 0x555e7038e9e0 2024-03-23T23:11:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.614+0000 7fe9a7400640 1 --2- 172.21.15.12:0/3516714370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555e7038d810 0x555e70391a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.614+0000 7fe9a7e00640 1 --2- 172.21.15.12:0/3516714370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e7038f3a0 0x555e70391fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.615+0000 7fe9a6a00640 1 --2- 172.21.15.12:0/3516714370 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e7038e9e0 0x555e7039efa0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe990009d70 tx=0x7fe990005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eddd77bf164f2056 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.615+0000 7fe997400640 1 -- 172.21.15.12:0/3516714370 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99001f230 con 0x555e7038e9e0 2024-03-23T23:11:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.615+0000 7fe997400640 1 -- 172.21.15.12:0/3516714370 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe990024070 con 0x555e7038e9e0 2024-03-23T23:11:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.615+0000 7fe997400640 1 -- 172.21.15.12:0/3516714370 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9900035f0 con 0x555e7038e9e0 2024-03-23T23:11:24.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.615+0000 7fe9b0f71740 1 -- 172.21.15.12:0/3516714370 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e7038e9e0 msgr2=0x555e7039efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.615+0000 7fe9b0f71740 1 --2- 172.21.15.12:0/3516714370 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e7038e9e0 0x555e7039efa0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe990009d70 tx=0x7fe990005f90 comp rx=0 tx=0).stop 2024-03-23T23:11:24.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.616+0000 7fe9b0f71740 1 -- 172.21.15.12:0/3516714370 shutdown_connections 2024-03-23T23:11:24.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.616+0000 7fe9b0f71740 1 --2- 172.21.15.12:0/3516714370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e7038f3a0 0x555e70391fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.616+0000 7fe9b0f71740 1 --2- 172.21.15.12:0/3516714370 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e7038e9e0 0x555e7039efa0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.616+0000 7fe9b0f71740 1 --2- 172.21.15.12:0/3516714370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555e7038d810 0x555e70391a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.616+0000 7fe9b0f71740 1 -- 172.21.15.12:0/3516714370 >> 172.21.15.12:0/3516714370 conn(0x555e70388c00 msgr2=0x555e7038c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.616+0000 7fe9b0f71740 1 -- 172.21.15.12:0/3516714370 shutdown_connections 2024-03-23T23:11:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.616+0000 7fe9b0f71740 1 -- 172.21.15.12:0/3516714370 wait complete. 2024-03-23T23:11:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.616+0000 7fe9b0f71740 1 -- start start 2024-03-23T23:11:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.616+0000 7fe9b0f71740 1 Processor -- start 2024-03-23T23:11:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.617+0000 7fe9b0f71740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e7038d810 0x555e7039b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.617+0000 7fe9b0f71740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e7038e9e0 0x555e7039b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.617+0000 7fe9b0f71740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555e7038f3a0 0x555e7039e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.617+0000 7fe9b0f71740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a81a9340 con 0x555e7038e9e0 2024-03-23T23:11:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.617+0000 7fe9b0f71740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a81aa640 con 0x555e7038d810 2024-03-23T23:11:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.617+0000 7fe9b0f71740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a81a9980 con 0x555e7038f3a0 2024-03-23T23:11:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.617+0000 7fe9b0f71740 10 client.0 _ll_register_callbacks cb 0x7fe9980e0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.617+0000 7fe9a6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e7038e9e0 0x555e7039b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.617+0000 7fe9a7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e7038d810 0x555e7039b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.617+0000 7fe9a7e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555e7038f3a0 0x555e7039e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.617+0000 7fe9a7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e7038d810 0x555e7039b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:41608/0 (socket says 172.21.15.12:41608) 2024-03-23T23:11:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.617+0000 7fe9a7400640 1 -- 172.21.15.12:0/1169282141 learned_addr learned my addr 172.21.15.12:0/1169282141 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.617+0000 7fe9a6a00640 1 -- 172.21.15.12:0/1169282141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555e7038f3a0 msgr2=0x555e7039e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.617+0000 7fe9a6a00640 1 --2- 172.21.15.12:0/1169282141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555e7038f3a0 0x555e7039e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.617+0000 7fe9a6a00640 1 -- 172.21.15.12:0/1169282141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e7038d810 msgr2=0x555e7039b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.617+0000 7fe9a6a00640 1 --2- 172.21.15.12:0/1169282141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e7038d810 0x555e7039b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.617+0000 7fe9a6a00640 1 -- 172.21.15.12:0/1169282141 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555e7039ecd0 con 0x555e7038e9e0 2024-03-23T23:11:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.617+0000 7fe9a7400640 1 --2- 172.21.15.12:0/1169282141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e7038d810 0x555e7039b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.618+0000 7fe9a6a00640 1 --2- 172.21.15.12:0/1169282141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e7038e9e0 0x555e7039b7a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x555e702f2350 tx=0x7fe990009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1233aab9b34c9a71 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.618+0000 7fe994c00640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:11:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.618+0000 7fe994c00640 1 -- 172.21.15.12:0/1169282141 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99001f220 con 0x555e7038e9e0 2024-03-23T23:11:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.618+0000 7fe994c00640 1 -- 172.21.15.12:0/1169282141 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe990004050 con 0x555e7038e9e0 2024-03-23T23:11:24.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.618+0000 7fe9b0f71740 1 -- 172.21.15.12:0/1169282141 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe990024070 con 0x555e7038e9e0 2024-03-23T23:11:24.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.618+0000 7fe994c00640 1 -- 172.21.15.12:0/1169282141 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe990004250 con 0x555e7038e9e0 2024-03-23T23:11:24.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.618+0000 7fe9b0f71740 10 client.17871 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:24.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.618+0000 7fe9b0f71740 1 -- 172.21.15.12:0/1169282141 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x555e703ff250 con 0x555e7038e9e0 2024-03-23T23:11:24.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.619+0000 7fe994c00640 1 -- 172.21.15.12:0/1169282141 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe990024070 con 0x555e7038e9e0 2024-03-23T23:11:24.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.619+0000 7fe994c00640 1 -- 172.21.15.12:0/1169282141 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe99001e420 con 0x555e7038e9e0 2024-03-23T23:11:24.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.619+0000 7fe9b0f71740 10 client.17871 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:24.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.619+0000 7fe9b0f71740 10 client.17871 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:11:24.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.619+0000 7fe9b0f71740 20 client.17871 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:24.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.619+0000 7fe9b0f71740 1 --2- 172.21.15.12:0/1169282141 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x555e703ffbd0 0x555e70401ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.619+0000 7fe9b0f71740 4 client.17871 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:11:24.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.619+0000 7fe9b0f71740 1 -- 172.21.15.12:0/1169282141 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555e702f9b50 con 0x555e703ffbd0 2024-03-23T23:11:24.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.619+0000 7fe9a7400640 1 --2- 172.21.15.12:0/1169282141 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x555e703ffbd0 0x555e70401ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.620+0000 7fe9a7400640 1 --2- 172.21.15.12:0/1169282141 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x555e703ffbd0 0x555e70401ff0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=db7318c1d4f3dfab server_cookie=d6d2d64afc9dbf86 in_seq=0 out_seq=0 2024-03-23T23:11:24.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.620+0000 7fe994c00640 10 client.17871 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:11:24.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.620+0000 7fe994c00640 1 -- 172.21.15.12:0/1169282141 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555e702f9b50 con 0x555e703ffbd0 2024-03-23T23:11:24.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.620+0000 7fe994c00640 10 client.17871 handle_command_reply: tid=0 2024-03-23T23:11:24.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.620+0000 7fe9b0f71740 1 client.17871 shutdown 2024-03-23T23:11:24.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.620+0000 7fe9b0f71740 1 -- 172.21.15.12:0/1169282141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e7038e9e0 msgr2=0x555e7039b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.620+0000 7fe9b0f71740 1 --2- 172.21.15.12:0/1169282141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e7038e9e0 0x555e7039b7a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x555e702f2350 tx=0x7fe990009b30 comp rx=0 tx=0).stop 2024-03-23T23:11:24.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.620+0000 7fe9b0f71740 1 -- 172.21.15.12:0/1169282141 shutdown_connections 2024-03-23T23:11:24.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.620+0000 7fe9b0f71740 1 --2- 172.21.15.12:0/1169282141 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x555e703ffbd0 0x555e70401ff0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.620+0000 7fe9b0f71740 1 --2- 172.21.15.12:0/1169282141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555e7038f3a0 0x555e7039e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.620+0000 7fe9b0f71740 1 --2- 172.21.15.12:0/1169282141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e7038e9e0 0x555e7039b7a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.620+0000 7fe994c00640 0 client.17871 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:11:24.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.620+0000 7fe9b0f71740 1 --2- 172.21.15.12:0/1169282141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e7038d810 0x555e7039b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.620+0000 7fe9b0f71740 1 -- 172.21.15.12:0/1169282141 >> 172.21.15.12:0/1169282141 conn(0x555e70388c00 msgr2=0x555e70387620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:24.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.620+0000 7fe9b0f71740 1 -- 172.21.15.12:0/1169282141 shutdown_connections 2024-03-23T23:11:24.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.620+0000 7fe9b0f71740 1 -- 172.21.15.12:0/1169282141 wait complete. 2024-03-23T23:11:24.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.620+0000 7fe9b0f71740 20 client.17871 trim_cache size 0 max 16384 2024-03-23T23:11:24.624 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:11:24.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.621+0000 7fe9ae200640 1 -- 172.21.15.12:0/4190308730 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe98c0a2800 msgr2=0x7fe98c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.621+0000 7fe9ae200640 1 --2- 172.21.15.12:0/4190308730 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe98c0a2800 0x7fe98c0a4cc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe9a002c640 tx=0x7fe9a0032590 comp rx=0 tx=0).stop 2024-03-23T23:11:24.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.621+0000 7fe9ae200640 1 -- 172.21.15.12:0/4190308730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9a80fff10 msgr2=0x7fe9a8199b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.621+0000 7fe9ae200640 1 --2- 172.21.15.12:0/4190308730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9a80fff10 0x7fe9a8199b80 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fe99800e9f0 tx=0x7fe99800edf0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.621+0000 7fe9ae200640 1 -- 172.21.15.12:0/4190308730 shutdown_connections 2024-03-23T23:11:24.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.621+0000 7fe9ae200640 1 --2- 172.21.15.12:0/4190308730 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe98c0a2800 0x7fe98c0a4cc0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.621+0000 7fe9ae200640 1 --2- 172.21.15.12:0/4190308730 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9a8101210 0x7fe9a81a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.621+0000 7fe9ae200640 1 --2- 172.21.15.12:0/4190308730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9a8100870 0x7fe9a819a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.621+0000 7fe9ae200640 1 --2- 172.21.15.12:0/4190308730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9a80fff10 0x7fe9a8199b80 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.621+0000 7fe9ae200640 1 -- 172.21.15.12:0/4190308730 >> 172.21.15.12:0/4190308730 conn(0x7fe9a80fbb30 msgr2=0x7fe9a80fd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:24.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.621+0000 7fe9ae200640 1 -- 172.21.15.12:0/4190308730 shutdown_connections 2024-03-23T23:11:24.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.621+0000 7fe9ae200640 1 -- 172.21.15.12:0/4190308730 wait complete. 2024-03-23T23:11:24.636 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:11:24.637 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-23T23:11:24.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.858+0000 7f31e8800640 1 -- 172.21.15.12:0/870245127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31e0101790 msgr2=0x7f31e0101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.858+0000 7f31e8800640 1 --2- 172.21.15.12:0/870245127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31e0101790 0x7f31e0101c10 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f31d0009900 tx=0x7f31d002bc40 comp rx=0 tx=0).stop 2024-03-23T23:11:24.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.858+0000 7f31e8800640 1 -- 172.21.15.12:0/870245127 shutdown_connections 2024-03-23T23:11:24.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.858+0000 7f31e8800640 1 --2- 172.21.15.12:0/870245127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31e0102150 0x7f31e01089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.858+0000 7f31e8800640 1 --2- 172.21.15.12:0/870245127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31e0101790 0x7f31e0101c10 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.858+0000 7f31e8800640 1 --2- 172.21.15.12:0/870245127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31e01005a0 0x7f31e01009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.858+0000 7f31e8800640 1 -- 172.21.15.12:0/870245127 >> 172.21.15.12:0/870245127 conn(0x7f31e00fbb30 msgr2=0x7f31e00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:24.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.858+0000 7f31e8800640 1 -- 172.21.15.12:0/870245127 shutdown_connections 2024-03-23T23:11:24.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.858+0000 7f31e8800640 1 -- 172.21.15.12:0/870245127 wait complete. 2024-03-23T23:11:24.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.859+0000 7f31e8800640 1 Processor -- start 2024-03-23T23:11:24.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.859+0000 7f31e8800640 1 -- start start 2024-03-23T23:11:24.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.860+0000 7f31e8800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31e01005a0 0x7f31e01044e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.860+0000 7f31e8800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31e0101790 0x7f31e0102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.860+0000 7f31e8800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31e0102150 0x7f31e01030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.860+0000 7f31e8800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e010c270 con 0x7f31e01005a0 2024-03-23T23:11:24.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.860+0000 7f31e8800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e010c0f0 con 0x7f31e0101790 2024-03-23T23:11:24.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.860+0000 7f31e8800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e010c3f0 con 0x7f31e0102150 2024-03-23T23:11:24.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.860+0000 7f31e5e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31e01005a0 0x7f31e01044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.860+0000 7f31e5e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31e01005a0 0x7f31e01044e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45048/0 (socket says 172.21.15.12:45048) 2024-03-23T23:11:24.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.860+0000 7f31e5e00640 1 -- 172.21.15.12:0/3763730139 learned_addr learned my addr 172.21.15.12:0/3763730139 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:24.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.860+0000 7f31e5400640 1 --2- 172.21.15.12:0/3763730139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31e0101790 0x7f31e0102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.860+0000 7f31e6800640 1 --2- 172.21.15.12:0/3763730139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31e0102150 0x7f31e01030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.860+0000 7f31e5400640 1 -- 172.21.15.12:0/3763730139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31e0102150 msgr2=0x7f31e01030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.860+0000 7f31e5400640 1 --2- 172.21.15.12:0/3763730139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31e0102150 0x7f31e01030b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.860+0000 7f31e5400640 1 -- 172.21.15.12:0/3763730139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31e01005a0 msgr2=0x7f31e01044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.860+0000 7f31e5400640 1 --2- 172.21.15.12:0/3763730139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31e01005a0 0x7f31e01044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.860+0000 7f31e5400640 1 -- 172.21.15.12:0/3763730139 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e0103970 con 0x7f31e0101790 2024-03-23T23:11:24.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.861+0000 7f31e5400640 1 --2- 172.21.15.12:0/3763730139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31e0101790 0x7f31e0102b70 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f31d0036040 tx=0x7f31d002c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:24.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.861+0000 7f31daa00640 1 -- 172.21.15.12:0/3763730139 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d0041070 con 0x7f31e0101790 2024-03-23T23:11:24.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.861+0000 7f31daa00640 1 -- 172.21.15.12:0/3763730139 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f31d0004450 con 0x7f31e0101790 2024-03-23T23:11:24.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.861+0000 7f31daa00640 1 -- 172.21.15.12:0/3763730139 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d0034440 con 0x7f31e0101790 2024-03-23T23:11:24.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.861+0000 7f31e8800640 1 -- 172.21.15.12:0/3763730139 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31e01a1b30 con 0x7f31e0101790 2024-03-23T23:11:24.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.861+0000 7f31e8800640 1 -- 172.21.15.12:0/3763730139 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31e01a1ff0 con 0x7f31e0101790 2024-03-23T23:11:24.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.864+0000 7f31eb45b740 1 Processor -- start 2024-03-23T23:11:24.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.864+0000 7f31eb45b740 1 -- start start 2024-03-23T23:11:24.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.864+0000 7f31daa00640 1 -- 172.21.15.12:0/3763730139 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f31d0044070 con 0x7f31e0101790 2024-03-23T23:11:24.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.865+0000 7f31eb45b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559487a9c810 0x559487a9cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.865+0000 7f31eb45b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9d9e0 0x559487a9de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.865+0000 7f31eb45b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559487a9e3a0 0x559487aa4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.865+0000 7f31eb45b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x559487a012c0 con 0x559487a9e3a0 2024-03-23T23:11:24.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.865+0000 7f31eb45b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594879a9c90 con 0x559487a9d9e0 2024-03-23T23:11:24.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.865+0000 7f31eb45b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594879fd360 con 0x559487a9c810 2024-03-23T23:11:24.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.865+0000 7f31e5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9d9e0 0x559487a9de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.865+0000 7f31e6800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559487a9c810 0x559487a9cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.865+0000 7f31e5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9d9e0 0x559487a9de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:41624/0 (socket says 172.21.15.12:41624) 2024-03-23T23:11:24.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.865+0000 7f31e5e00640 1 -- 172.21.15.12:0/2561940761 learned_addr learned my addr 172.21.15.12:0/2561940761 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:24.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.865+0000 7f31e5400640 1 --2- 172.21.15.12:0/2561940761 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559487a9e3a0 0x559487aa4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.865+0000 7f31e5e00640 1 -- 172.21.15.12:0/2561940761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559487a9c810 msgr2=0x559487a9cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:24.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.865+0000 7f31e5e00640 1 --2- 172.21.15.12:0/2561940761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559487a9c810 0x559487a9cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.865+0000 7f31e5e00640 1 -- 172.21.15.12:0/2561940761 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559487a9e3a0 msgr2=0x559487aa4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:24.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.865+0000 7f31e5e00640 1 --2- 172.21.15.12:0/2561940761 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559487a9e3a0 0x559487aa4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.865+0000 7f31e5e00640 1 -- 172.21.15.12:0/2561940761 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594879ce080 con 0x559487a9d9e0 2024-03-23T23:11:24.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.866+0000 7f31e5e00640 1 --2- 172.21.15.12:0/2561940761 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9d9e0 0x559487a9de60 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009d60 tx=0x7f31d4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc6cbebabd0bf396 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:24.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.866+0000 7f31cf000640 1 -- 172.21.15.12:0/2561940761 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d4004230 con 0x559487a9d9e0 2024-03-23T23:11:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.866+0000 7f31cf000640 1 -- 172.21.15.12:0/2561940761 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f31d40043d0 con 0x559487a9d9e0 2024-03-23T23:11:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.866+0000 7f31cf000640 1 -- 172.21.15.12:0/2561940761 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d40198f0 con 0x559487a9d9e0 2024-03-23T23:11:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.866+0000 7f31daa00640 1 --2- 172.21.15.12:0/3763730139 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f31c80a2800 0x7f31c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.866+0000 7f31daa00640 1 -- 172.21.15.12:0/3763730139 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f31d00f0510 con 0x7f31e0101790 2024-03-23T23:11:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.866+0000 7f31e6800640 1 --2- 172.21.15.12:0/3763730139 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f31c80a2800 0x7f31c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.866+0000 7f31eb45b740 1 -- 172.21.15.12:0/2561940761 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9d9e0 msgr2=0x559487a9de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.866+0000 7f31eb45b740 1 --2- 172.21.15.12:0/2561940761 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9d9e0 0x559487a9de60 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009d60 tx=0x7f31d4011810 comp rx=0 tx=0).stop 2024-03-23T23:11:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.867+0000 7f31eb45b740 1 -- 172.21.15.12:0/2561940761 shutdown_connections 2024-03-23T23:11:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.867+0000 7f31eb45b740 1 --2- 172.21.15.12:0/2561940761 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559487a9e3a0 0x559487aa4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.867+0000 7f31eb45b740 1 --2- 172.21.15.12:0/2561940761 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9d9e0 0x559487a9de60 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.867+0000 7f31eb45b740 1 --2- 172.21.15.12:0/2561940761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559487a9c810 0x559487a9cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.867+0000 7f31eb45b740 1 -- 172.21.15.12:0/2561940761 >> 172.21.15.12:0/2561940761 conn(0x559487a97c00 msgr2=0x559487a98000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.867+0000 7f31eb45b740 1 -- 172.21.15.12:0/2561940761 shutdown_connections 2024-03-23T23:11:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.867+0000 7f31eb45b740 1 -- 172.21.15.12:0/2561940761 wait complete. 2024-03-23T23:11:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.867+0000 7f31e6800640 1 --2- 172.21.15.12:0/3763730139 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f31c80a2800 0x7f31c80a4cc0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x559487a9d8e0 tx=0x7f31dc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.867+0000 7f31eb45b740 1 -- start start 2024-03-23T23:11:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.868+0000 7f31eb45b740 1 Processor -- start 2024-03-23T23:11:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.868+0000 7f31eb45b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9c810 0x559487aa6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.868+0000 7f31eb45b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559487a9d9e0 0x559487aa6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.868+0000 7f31eb45b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559487a9e3a0 0x559487aae010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.868+0000 7f31eb45b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x559487aa53b0 con 0x559487a9d9e0 2024-03-23T23:11:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.868+0000 7f31eb45b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594879bd0d0 con 0x559487a9c810 2024-03-23T23:11:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.868+0000 7f31eb45b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x559487aa5530 con 0x559487a9e3a0 2024-03-23T23:11:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.868+0000 7f31eb45b740 10 client.0 _ll_register_callbacks cb 0x5594879c1e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.868+0000 7f31e5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9c810 0x559487aa6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.868+0000 7f31e5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9c810 0x559487aa6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:41626/0 (socket says 172.21.15.12:41626) 2024-03-23T23:11:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.868+0000 7f31e5e00640 1 -- 172.21.15.12:0/3396286432 learned_addr learned my addr 172.21.15.12:0/3396286432 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.868+0000 7f31e6800640 1 --2- 172.21.15.12:0/3396286432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559487a9e3a0 0x559487aae010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.868+0000 7f31e5400640 1 --2- 172.21.15.12:0/3396286432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559487a9d9e0 0x559487aa6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.869+0000 7f31e5e00640 1 -- 172.21.15.12:0/3396286432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559487a9e3a0 msgr2=0x559487aae010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.869+0000 7f31e5e00640 1 --2- 172.21.15.12:0/3396286432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559487a9e3a0 0x559487aae010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.869+0000 7f31e5e00640 1 -- 172.21.15.12:0/3396286432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559487a9d9e0 msgr2=0x559487aa6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.869+0000 7f31e5e00640 1 --2- 172.21.15.12:0/3396286432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559487a9d9e0 0x559487aa6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.869+0000 7f31e5e00640 1 -- 172.21.15.12:0/3396286432 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5594877153f0 con 0x559487a9c810 2024-03-23T23:11:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.869+0000 7f31e5400640 1 --2- 172.21.15.12:0/3396286432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559487a9d9e0 0x559487aa6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:11:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.869+0000 7f31e5e00640 1 --2- 172.21.15.12:0/3396286432 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9c810 0x559487aa6a90 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009620 tx=0x7f31d4009a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66a3d4d48145f474 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.869+0000 7f31c3400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:11:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.869+0000 7f31c3400640 1 -- 172.21.15.12:0/3396286432 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d4028070 con 0x559487a9c810 2024-03-23T23:11:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.869+0000 7f31c3400640 1 -- 172.21.15.12:0/3396286432 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f31d40052d0 con 0x559487a9c810 2024-03-23T23:11:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.869+0000 7f31eb45b740 1 -- 172.21.15.12:0/3396286432 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f31d40043d0 con 0x559487a9c810 2024-03-23T23:11:24.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.870+0000 7f31c3400640 1 -- 172.21.15.12:0/3396286432 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f31d4034050 con 0x559487a9c810 2024-03-23T23:11:24.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.870+0000 7f31eb45b740 10 client.28019 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:24.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.870+0000 7f31eb45b740 1 -- 172.21.15.12:0/3396286432 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5594877df7d0 con 0x559487a9c810 2024-03-23T23:11:24.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.870+0000 7f31c3400640 1 -- 172.21.15.12:0/3396286432 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f31d40043d0 con 0x559487a9c810 2024-03-23T23:11:24.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.870+0000 7f31c3400640 1 -- 172.21.15.12:0/3396286432 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f31d4004a30 con 0x559487a9c810 2024-03-23T23:11:24.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.870+0000 7f31eb45b740 10 client.28019 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:24.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.870+0000 7f31eb45b740 10 client.28019 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:11:24.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.870+0000 7f31eb45b740 20 client.28019 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:24.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.870+0000 7f31eb45b740 1 --2- 172.21.15.12:0/3396286432 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x559487ab2290 0x559487ab46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.870+0000 7f31eb45b740 4 client.28019 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:11:24.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.870+0000 7f31eb45b740 1 -- 172.21.15.12:0/3396286432 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559487a0e800 con 0x559487ab2290 2024-03-23T23:11:24.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.870+0000 7f31e5400640 1 --2- 172.21.15.12:0/3396286432 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x559487ab2290 0x559487ab46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.871+0000 7f31e5400640 1 --2- 172.21.15.12:0/3396286432 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x559487ab2290 0x559487ab46e0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=903863cc0b609c72 server_cookie=12ad019a6dd4daf8 in_seq=0 out_seq=0 2024-03-23T23:11:24.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.871+0000 7f31c3400640 10 client.28019 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:24.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.872+0000 7f31c3400640 1 -- 172.21.15.12:0/3396286432 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559487a0e800 con 0x559487ab2290 2024-03-23T23:11:24.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.872+0000 7f31c3400640 10 client.28019 handle_command_reply: tid=0 2024-03-23T23:11:24.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.872+0000 7f31eb45b740 1 client.28019 shutdown 2024-03-23T23:11:24.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.872+0000 7f31eb45b740 1 -- 172.21.15.12:0/3396286432 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9c810 msgr2=0x559487aa6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.872+0000 7f31eb45b740 1 --2- 172.21.15.12:0/3396286432 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9c810 0x559487aa6a90 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009620 tx=0x7f31d4009a90 comp rx=0 tx=0).stop 2024-03-23T23:11:24.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.872+0000 7f31eb45b740 1 -- 172.21.15.12:0/3396286432 shutdown_connections 2024-03-23T23:11:24.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.872+0000 7f31eb45b740 1 --2- 172.21.15.12:0/3396286432 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x559487ab2290 0x559487ab46e0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.872+0000 7f31eb45b740 1 --2- 172.21.15.12:0/3396286432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559487a9e3a0 0x559487aae010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.872+0000 7f31eb45b740 1 --2- 172.21.15.12:0/3396286432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559487a9d9e0 0x559487aa6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.872+0000 7f31c3400640 0 client.28019 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:24.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.872+0000 7f31eb45b740 1 --2- 172.21.15.12:0/3396286432 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9c810 0x559487aa6a90 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.872+0000 7f31eb45b740 1 -- 172.21.15.12:0/3396286432 >> 172.21.15.12:0/3396286432 conn(0x559487a97c00 msgr2=0x559487a96eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:24.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.872+0000 7f31eb45b740 1 -- 172.21.15.12:0/3396286432 shutdown_connections 2024-03-23T23:11:24.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.872+0000 7f31eb45b740 1 -- 172.21.15.12:0/3396286432 wait complete. 2024-03-23T23:11:24.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.873+0000 7f31eb45b740 20 client.28019 trim_cache size 0 max 16384 2024-03-23T23:11:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.886+0000 7f31eb45b740 1 Processor -- start 2024-03-23T23:11:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.886+0000 7f31eb45b740 1 -- start start 2024-03-23T23:11:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.886+0000 7f31eb45b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559487a9c810 0x559487a9ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.886+0000 7f31eb45b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9d9e0 0x559487aadfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.886+0000 7f31eb45b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559487a9e3a0 0x559487ab8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.886+0000 7f31eb45b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e01ad120 con 0x559487a9c810 2024-03-23T23:11:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.886+0000 7f31eb45b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e01aac20 con 0x559487a9d9e0 2024-03-23T23:11:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.886+0000 7f31eb45b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e01a8cc0 con 0x559487a9e3a0 2024-03-23T23:11:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.887+0000 7f31e5e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559487a9c810 0x559487a9ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.887+0000 7f31e5400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9d9e0 0x559487aadfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.887+0000 7f31e6800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559487a9e3a0 0x559487ab8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.887+0000 7f31e5e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559487a9c810 0x559487a9ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45080/0 (socket says 172.21.15.12:45080) 2024-03-23T23:11:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.887+0000 7f31e5400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9d9e0 0x559487aadfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:41628/0 (socket says 172.21.15.12:41628) 2024-03-23T23:11:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.887+0000 7f31e5e00640 1 -- 172.21.15.12:0/824418919 learned_addr learned my addr 172.21.15.12:0/824418919 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.887+0000 7f31e5400640 1 -- 172.21.15.12:0/824418919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559487a9e3a0 msgr2=0x559487ab8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.887+0000 7f31e5400640 1 --2- 172.21.15.12:0/824418919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559487a9e3a0 0x559487ab8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.887+0000 7f31e5400640 1 -- 172.21.15.12:0/824418919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559487a9c810 msgr2=0x559487a9ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.887+0000 7f31e5400640 1 --2- 172.21.15.12:0/824418919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559487a9c810 0x559487a9ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.887+0000 7f31e5400640 1 -- 172.21.15.12:0/824418919 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559487aa5290 con 0x559487a9d9e0 2024-03-23T23:11:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.887+0000 7f31e5e00640 1 --2- 172.21.15.12:0/824418919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559487a9c810 0x559487a9ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.888+0000 7f31e5400640 1 --2- 172.21.15.12:0/824418919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9d9e0 0x559487aadfd0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x559487aa7f70 tx=0x7f31d0005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=597dbc14b5987f77 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.888+0000 7f31ce600640 1 -- 172.21.15.12:0/824418919 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d004b040 con 0x559487a9d9e0 2024-03-23T23:11:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.888+0000 7f31ce600640 1 -- 172.21.15.12:0/824418919 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f31d0004c80 con 0x559487a9d9e0 2024-03-23T23:11:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.888+0000 7f31ce600640 1 -- 172.21.15.12:0/824418919 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d00b9580 con 0x559487a9d9e0 2024-03-23T23:11:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.888+0000 7f31eb45b740 1 -- 172.21.15.12:0/824418919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9d9e0 msgr2=0x559487aadfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.888+0000 7f31eb45b740 1 --2- 172.21.15.12:0/824418919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9d9e0 0x559487aadfd0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x559487aa7f70 tx=0x7f31d0005e50 comp rx=0 tx=0).stop 2024-03-23T23:11:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.888+0000 7f31eb45b740 1 -- 172.21.15.12:0/824418919 shutdown_connections 2024-03-23T23:11:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.888+0000 7f31eb45b740 1 --2- 172.21.15.12:0/824418919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559487a9e3a0 0x559487ab8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.888+0000 7f31eb45b740 1 --2- 172.21.15.12:0/824418919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9d9e0 0x559487aadfd0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.888+0000 7f31eb45b740 1 --2- 172.21.15.12:0/824418919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559487a9c810 0x559487a9ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.888+0000 7f31eb45b740 1 -- 172.21.15.12:0/824418919 >> 172.21.15.12:0/824418919 conn(0x559487a97c00 msgr2=0x559487a96eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.888+0000 7f31eb45b740 1 -- 172.21.15.12:0/824418919 shutdown_connections 2024-03-23T23:11:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.889+0000 7f31eb45b740 1 -- 172.21.15.12:0/824418919 wait complete. 2024-03-23T23:11:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.889+0000 7f31eb45b740 1 -- start start 2024-03-23T23:11:24.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.889+0000 7f31eb45b740 1 Processor -- start 2024-03-23T23:11:24.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.889+0000 7f31eb45b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9c810 0x559487aa9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.889+0000 7f31eb45b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559487a9d9e0 0x559487aaa220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.889+0000 7f31eb45b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559487a9e3a0 0x559487aad240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.889+0000 7f31eb45b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e01a92c0 con 0x559487a9d9e0 2024-03-23T23:11:24.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.889+0000 7f31eb45b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e01aa5c0 con 0x559487a9c810 2024-03-23T23:11:24.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.889+0000 7f31eb45b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e01a9900 con 0x559487a9e3a0 2024-03-23T23:11:24.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.890+0000 7f31eb45b740 10 client.0 _ll_register_callbacks cb 0x7f31dc009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:24.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.890+0000 7f31e5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9c810 0x559487aa9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.890+0000 7f31e5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9c810 0x559487aa9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:41636/0 (socket says 172.21.15.12:41636) 2024-03-23T23:11:24.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.890+0000 7f31e5e00640 1 -- 172.21.15.12:0/866435264 learned_addr learned my addr 172.21.15.12:0/866435264 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:24.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.890+0000 7f31e5400640 1 --2- 172.21.15.12:0/866435264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559487a9d9e0 0x559487aaa220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.890+0000 7f31e6800640 1 --2- 172.21.15.12:0/866435264 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559487a9e3a0 0x559487aad240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.890+0000 7f31e5e00640 1 -- 172.21.15.12:0/866435264 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559487a9e3a0 msgr2=0x559487aad240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:24.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.890+0000 7f31e5e00640 1 --2- 172.21.15.12:0/866435264 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559487a9e3a0 0x559487aad240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.890+0000 7f31e5e00640 1 -- 172.21.15.12:0/866435264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559487a9d9e0 msgr2=0x559487aaa220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:24.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.890+0000 7f31e5e00640 1 --2- 172.21.15.12:0/866435264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559487a9d9e0 0x559487aaa220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.890+0000 7f31e5e00640 1 -- 172.21.15.12:0/866435264 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559487aad830 con 0x559487a9c810 2024-03-23T23:11:24.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.890+0000 7f31e5e00640 1 --2- 172.21.15.12:0/866435264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9c810 0x559487aa9ce0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f31d40058c0 tx=0x7f31d40095a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=507cdda52984d9da server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:24.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.890+0000 7f31c3400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:11:24.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.891+0000 7f31c3400640 1 -- 172.21.15.12:0/866435264 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d403d060 con 0x559487a9c810 2024-03-23T23:11:24.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.891+0000 7f31c3400640 1 -- 172.21.15.12:0/866435264 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f31d40052d0 con 0x559487a9c810 2024-03-23T23:11:24.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.891+0000 7f31eb45b740 1 -- 172.21.15.12:0/866435264 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f31d0004c80 con 0x559487a9c810 2024-03-23T23:11:24.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.891+0000 7f31c3400640 1 -- 172.21.15.12:0/866435264 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f31d4028040 con 0x559487a9c810 2024-03-23T23:11:24.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.891+0000 7f31eb45b740 10 client.28025 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:24.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.891+0000 7f31eb45b740 1 -- 172.21.15.12:0/866435264 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x559487b0e140 con 0x559487a9c810 2024-03-23T23:11:24.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.891+0000 7f31c3400640 1 -- 172.21.15.12:0/866435264 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f31d0004c80 con 0x559487a9c810 2024-03-23T23:11:24.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.892+0000 7f31c3400640 1 -- 172.21.15.12:0/866435264 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f31d4004630 con 0x559487a9c810 2024-03-23T23:11:24.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.892+0000 7f31eb45b740 10 client.28025 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.892+0000 7f31eb45b740 10 client.28025 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:11:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.892+0000 7f31eb45b740 20 client.28025 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.892+0000 7f31eb45b740 1 --2- 172.21.15.12:0/866435264 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x559487b0ea60 0x559487b10e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.892+0000 7f31eb45b740 4 client.28025 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:11:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.892+0000 7f31eb45b740 1 -- 172.21.15.12:0/866435264 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559487a08b50 con 0x559487b0ea60 2024-03-23T23:11:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.892+0000 7f31e5400640 1 --2- 172.21.15.12:0/866435264 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x559487b0ea60 0x559487b10e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.892+0000 7f31e5400640 1 --2- 172.21.15.12:0/866435264 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x559487b0ea60 0x559487b10e80 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc88d3f284052a2d server_cookie=65a40e24d4aa748d in_seq=0 out_seq=0 2024-03-23T23:11:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.892+0000 7f31c3400640 10 client.28025 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.893+0000 7f31c3400640 1 -- 172.21.15.12:0/866435264 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559487a08b50 con 0x559487b0ea60 2024-03-23T23:11:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.893+0000 7f31c3400640 10 client.28025 handle_command_reply: tid=0 2024-03-23T23:11:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.893+0000 7f31eb45b740 1 client.28025 shutdown 2024-03-23T23:11:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.893+0000 7f31eb45b740 1 -- 172.21.15.12:0/866435264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9c810 msgr2=0x559487aa9ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.893+0000 7f31eb45b740 1 --2- 172.21.15.12:0/866435264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9c810 0x559487aa9ce0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f31d40058c0 tx=0x7f31d40095a0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.893+0000 7f31eb45b740 1 -- 172.21.15.12:0/866435264 shutdown_connections 2024-03-23T23:11:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.893+0000 7f31eb45b740 1 --2- 172.21.15.12:0/866435264 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x559487b0ea60 0x559487b10e80 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.893+0000 7f31eb45b740 1 --2- 172.21.15.12:0/866435264 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559487a9e3a0 0x559487aad240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.893+0000 7f31c3400640 0 client.28025 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:24.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.893+0000 7f31eb45b740 1 --2- 172.21.15.12:0/866435264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559487a9d9e0 0x559487aaa220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.893+0000 7f31eb45b740 1 --2- 172.21.15.12:0/866435264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559487a9c810 0x559487aa9ce0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.893+0000 7f31eb45b740 1 -- 172.21.15.12:0/866435264 >> 172.21.15.12:0/866435264 conn(0x559487a97c00 msgr2=0x559487aa0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:24.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.893+0000 7f31eb45b740 1 -- 172.21.15.12:0/866435264 shutdown_connections 2024-03-23T23:11:24.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.893+0000 7f31eb45b740 1 -- 172.21.15.12:0/866435264 wait complete. 2024-03-23T23:11:24.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.893+0000 7f31eb45b740 20 client.28025 trim_cache size 0 max 16384 2024-03-23T23:11:24.896 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:11:24.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.894+0000 7f31e8800640 1 -- 172.21.15.12:0/3763730139 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f31c80a2800 msgr2=0x7f31c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.894+0000 7f31e8800640 1 --2- 172.21.15.12:0/3763730139 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f31c80a2800 0x7f31c80a4cc0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x559487a9d8e0 tx=0x7f31dc009500 comp rx=0 tx=0).stop 2024-03-23T23:11:24.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.894+0000 7f31e8800640 1 -- 172.21.15.12:0/3763730139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31e0101790 msgr2=0x7f31e0102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:24.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.894+0000 7f31e8800640 1 --2- 172.21.15.12:0/3763730139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31e0101790 0x7f31e0102b70 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f31d0036040 tx=0x7f31d002c660 comp rx=0 tx=0).stop 2024-03-23T23:11:24.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.894+0000 7f31e8800640 1 -- 172.21.15.12:0/3763730139 shutdown_connections 2024-03-23T23:11:24.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.894+0000 7f31e8800640 1 --2- 172.21.15.12:0/3763730139 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f31c80a2800 0x7f31c80a4cc0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.894+0000 7f31e8800640 1 --2- 172.21.15.12:0/3763730139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31e0102150 0x7f31e01030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.894+0000 7f31e8800640 1 --2- 172.21.15.12:0/3763730139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31e0101790 0x7f31e0102b70 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.894+0000 7f31e8800640 1 --2- 172.21.15.12:0/3763730139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31e01005a0 0x7f31e01044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:24.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.894+0000 7f31e8800640 1 -- 172.21.15.12:0/3763730139 >> 172.21.15.12:0/3763730139 conn(0x7f31e00fbb30 msgr2=0x7f31e00fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:24.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.894+0000 7f31e8800640 1 -- 172.21.15.12:0/3763730139 shutdown_connections 2024-03-23T23:11:24.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:24.894+0000 7f31e8800640 1 -- 172.21.15.12:0/3763730139 wait complete. 2024-03-23T23:11:25.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:24 smithi012 ceph-mon[31695]: pgmap v678: 97 pgs: 97 active+clean; 2.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:11:25.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:24 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2025027588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:11:25.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:24 smithi173 ceph-mon[36116]: pgmap v678: 97 pgs: 97 active+clean; 2.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:11:25.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:24 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2025027588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:11:25.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:24 smithi203 ceph-mon[36693]: pgmap v678: 97 pgs: 97 active+clean; 2.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:11:25.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:24 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2025027588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:11:25.910 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-23T23:11:25.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:11:25.910 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-23T23:11:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.925+0000 7f33e3e00640 1 -- 172.21.15.12:0/3420187588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33e4101520 msgr2=0x7f33e4101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.925+0000 7f33e3e00640 1 --2- 172.21.15.12:0/3420187588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33e4101520 0x7f33e4101940 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f33d4009900 tx=0x7f33d402bc30 comp rx=0 tx=0).stop 2024-03-23T23:11:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.925+0000 7f33e3e00640 1 -- 172.21.15.12:0/3420187588 shutdown_connections 2024-03-23T23:11:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.925+0000 7f33e3e00640 1 --2- 172.21.15.12:0/3420187588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33e41030d0 0x7f33e4109960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.925+0000 7f33e3e00640 1 --2- 172.21.15.12:0/3420187588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33e4102710 0x7f33e4102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.925+0000 7f33e3e00640 1 --2- 172.21.15.12:0/3420187588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33e4101520 0x7f33e4101940 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.925+0000 7f33e3e00640 1 -- 172.21.15.12:0/3420187588 >> 172.21.15.12:0/3420187588 conn(0x7f33e40fcab0 msgr2=0x7f33e40feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.926+0000 7f33e3e00640 1 -- 172.21.15.12:0/3420187588 shutdown_connections 2024-03-23T23:11:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.926+0000 7f33e3e00640 1 -- 172.21.15.12:0/3420187588 wait complete. 2024-03-23T23:11:25.926 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.927+0000 7f33e3e00640 1 Processor -- start 2024-03-23T23:11:25.926 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.927+0000 7f33e3e00640 1 -- start start 2024-03-23T23:11:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.927+0000 7f33e3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33e4101520 0x7f33e4199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.927+0000 7f33e3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33e4102710 0x7f33e4199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.927+0000 7f33e3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33e41030d0 0x7f33e41a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.927+0000 7f33e3e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e410d200 con 0x7f33e4102710 2024-03-23T23:11:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.927+0000 7f33e3e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e410d080 con 0x7f33e4101520 2024-03-23T23:11:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.927+0000 7f33e3e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e410d380 con 0x7f33e41030d0 2024-03-23T23:11:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.927+0000 7f33e2a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33e4101520 0x7f33e4199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.927+0000 7f33e2a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33e4101520 0x7f33e4199a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:41664/0 (socket says 172.21.15.12:41664) 2024-03-23T23:11:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.927+0000 7f33e2a00640 1 -- 172.21.15.12:0/1760333266 learned_addr learned my addr 172.21.15.12:0/1760333266 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.927+0000 7f33e3400640 1 --2- 172.21.15.12:0/1760333266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33e41030d0 0x7f33e41a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.927+0000 7f33da000640 1 --2- 172.21.15.12:0/1760333266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33e4102710 0x7f33e4199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.928+0000 7f33e2a00640 1 -- 172.21.15.12:0/1760333266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33e41030d0 msgr2=0x7f33e41a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.928+0000 7f33e2a00640 1 --2- 172.21.15.12:0/1760333266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33e41030d0 0x7f33e41a0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.928+0000 7f33e2a00640 1 -- 172.21.15.12:0/1760333266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33e4102710 msgr2=0x7f33e4199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.928+0000 7f33e2a00640 1 --2- 172.21.15.12:0/1760333266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33e4102710 0x7f33e4199f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.928+0000 7f33e2a00640 1 -- 172.21.15.12:0/1760333266 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e41a1660 con 0x7f33e4101520 2024-03-23T23:11:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.928+0000 7f33da000640 1 --2- 172.21.15.12:0/1760333266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33e4102710 0x7f33e4199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:11:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.928+0000 7f33e2a00640 1 --2- 172.21.15.12:0/1760333266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33e4101520 0x7f33e4199a40 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f33d4000c00 tx=0x7f33d402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:25.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.928+0000 7f33e8a00640 1 -- 172.21.15.12:0/1760333266 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d4043020 con 0x7f33e4101520 2024-03-23T23:11:25.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.928+0000 7f33e8a00640 1 -- 172.21.15.12:0/1760333266 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33d40054e0 con 0x7f33e4101520 2024-03-23T23:11:25.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.928+0000 7f33e8a00640 1 -- 172.21.15.12:0/1760333266 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d4003c40 con 0x7f33e4101520 2024-03-23T23:11:25.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.929+0000 7f33e3e00640 1 -- 172.21.15.12:0/1760333266 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33e41a18f0 con 0x7f33e4101520 2024-03-23T23:11:25.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.929+0000 7f33e3e00640 1 -- 172.21.15.12:0/1760333266 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33e41a1df0 con 0x7f33e4101520 2024-03-23T23:11:25.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.933+0000 7f33e8a00640 1 -- 172.21.15.12:0/1760333266 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f33d4046020 con 0x7f33e4101520 2024-03-23T23:11:25.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.933+0000 7f33ecb66740 1 Processor -- start 2024-03-23T23:11:25.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.933+0000 7f33ecb66740 1 -- start start 2024-03-23T23:11:25.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.933+0000 7f33ecb66740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55835d7553f0 0x55835d755810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:25.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.933+0000 7f33ecb66740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55835d755de0 0x55835d75a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:25.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.934+0000 7f33ecb66740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d75af50 0x55835d75d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:25.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.934+0000 7f33ecb66740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55835d7183b0 con 0x55835d7553f0 2024-03-23T23:11:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.934+0000 7f33ecb66740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55835d6ba830 con 0x55835d755de0 2024-03-23T23:11:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.934+0000 7f33ecb66740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55835d6635d0 con 0x55835d75af50 2024-03-23T23:11:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.934+0000 7f33e2a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55835d755de0 0x55835d75a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.934+0000 7f33da000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d75af50 0x55835d75d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.934+0000 7f33e3400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55835d7553f0 0x55835d755810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.934+0000 7f33da000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d75af50 0x55835d75d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:56486/0 (socket says 172.21.15.12:56486) 2024-03-23T23:11:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.934+0000 7f33e2a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55835d755de0 0x55835d75a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:41670/0 (socket says 172.21.15.12:41670) 2024-03-23T23:11:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.934+0000 7f33da000640 1 -- 172.21.15.12:0/1133924479 learned_addr learned my addr 172.21.15.12:0/1133924479 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.934+0000 7f33da000640 1 -- 172.21.15.12:0/1133924479 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55835d755de0 msgr2=0x55835d75a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.934+0000 7f33da000640 1 --2- 172.21.15.12:0/1133924479 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55835d755de0 0x55835d75a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.934+0000 7f33da000640 1 -- 172.21.15.12:0/1133924479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55835d7553f0 msgr2=0x55835d755810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.934+0000 7f33da000640 1 --2- 172.21.15.12:0/1133924479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55835d7553f0 0x55835d755810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.934+0000 7f33da000640 1 -- 172.21.15.12:0/1133924479 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55835d67b2c0 con 0x55835d75af50 2024-03-23T23:11:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.935+0000 7f33da000640 1 --2- 172.21.15.12:0/1133924479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d75af50 0x55835d75d3e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009cf0 tx=0x7f33cc0117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2a64ef83e7ec1c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.935+0000 7f33e8a00640 1 --2- 172.21.15.12:0/1760333266 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f33c80a2870 0x7f33c80a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.935+0000 7f33d3e00640 1 -- 172.21.15.12:0/1133924479 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33cc012450 con 0x55835d75af50 2024-03-23T23:11:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.935+0000 7f33d3e00640 1 -- 172.21.15.12:0/1133924479 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33cc012ad0 con 0x55835d75af50 2024-03-23T23:11:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.935+0000 7f33d3e00640 1 -- 172.21.15.12:0/1133924479 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33cc00fbf0 con 0x55835d75af50 2024-03-23T23:11:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.935+0000 7f33e8a00640 1 -- 172.21.15.12:0/1760333266 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f33d40f0ac0 con 0x7f33e4101520 2024-03-23T23:11:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.935+0000 7f33e3400640 1 --2- 172.21.15.12:0/1760333266 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f33c80a2870 0x7f33c80a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.936+0000 7f33ecb66740 1 -- 172.21.15.12:0/1133924479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d75af50 msgr2=0x55835d75d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.936+0000 7f33ecb66740 1 --2- 172.21.15.12:0/1133924479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d75af50 0x55835d75d3e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009cf0 tx=0x7f33cc0117b0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.936+0000 7f33ecb66740 1 -- 172.21.15.12:0/1133924479 shutdown_connections 2024-03-23T23:11:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.936+0000 7f33ecb66740 1 --2- 172.21.15.12:0/1133924479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d75af50 0x55835d75d3e0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.936+0000 7f33ecb66740 1 --2- 172.21.15.12:0/1133924479 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55835d755de0 0x55835d75a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.936+0000 7f33ecb66740 1 --2- 172.21.15.12:0/1133924479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55835d7553f0 0x55835d755810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.936+0000 7f33ecb66740 1 -- 172.21.15.12:0/1133924479 >> 172.21.15.12:0/1133924479 conn(0x55835d750960 msgr2=0x55835d750d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.936+0000 7f33e3400640 1 --2- 172.21.15.12:0/1760333266 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f33c80a2870 0x7f33c80a4d30 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x55835d756640 tx=0x7f33dc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.936+0000 7f33ecb66740 1 -- 172.21.15.12:0/1133924479 shutdown_connections 2024-03-23T23:11:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.937+0000 7f33ecb66740 1 -- 172.21.15.12:0/1133924479 wait complete. 2024-03-23T23:11:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.937+0000 7f33ecb66740 1 -- start start 2024-03-23T23:11:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.937+0000 7f33ecb66740 1 Processor -- start 2024-03-23T23:11:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.938+0000 7f33ecb66740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d7553f0 0x55835d75f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.938+0000 7f33ecb66740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55835d755de0 0x55835d75fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.938+0000 7f33ecb66740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55835d75af50 0x55835d766b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.938+0000 7f33ecb66740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55835d676a20 con 0x55835d755de0 2024-03-23T23:11:25.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.938+0000 7f33ecb66740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55835d6b68d0 con 0x55835d75af50 2024-03-23T23:11:25.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.938+0000 7f33ecb66740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55835d75df00 con 0x55835d7553f0 2024-03-23T23:11:25.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.938+0000 7f33ecb66740 10 client.0 _ll_register_callbacks cb 0x55835d6b31c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:25.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.938+0000 7f33e2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d7553f0 0x55835d75f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:25.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.938+0000 7f33e2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d7553f0 0x55835d75f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:56498/0 (socket says 172.21.15.12:56498) 2024-03-23T23:11:25.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.938+0000 7f33e2a00640 1 -- 172.21.15.12:0/1596819386 learned_addr learned my addr 172.21.15.12:0/1596819386 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:25.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.938+0000 7f33e3400640 1 --2- 172.21.15.12:0/1596819386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55835d75af50 0x55835d766b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:25.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.938+0000 7f33da000640 1 --2- 172.21.15.12:0/1596819386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55835d755de0 0x55835d75fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:25.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.938+0000 7f33e2a00640 1 -- 172.21.15.12:0/1596819386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55835d75af50 msgr2=0x55835d766b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:25.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.938+0000 7f33e2a00640 1 --2- 172.21.15.12:0/1596819386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55835d75af50 0x55835d766b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.938+0000 7f33e2a00640 1 -- 172.21.15.12:0/1596819386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55835d755de0 msgr2=0x55835d75fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:25.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.938+0000 7f33e2a00640 1 --2- 172.21.15.12:0/1596819386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55835d755de0 0x55835d75fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.938+0000 7f33e2a00640 1 -- 172.21.15.12:0/1596819386 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55835d3cf210 con 0x55835d7553f0 2024-03-23T23:11:25.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.938+0000 7f33da000640 1 --2- 172.21.15.12:0/1596819386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55835d755de0 0x55835d75fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:11:25.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.939+0000 7f33e2a00640 1 --2- 172.21.15.12:0/1596819386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d7553f0 0x55835d75f5a0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f33d40b9870 tx=0x7f33d4009620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e94fd8434cc28dd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:25.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.939+0000 7f33d0c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:11:25.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.939+0000 7f33d0c00640 1 -- 172.21.15.12:0/1596819386 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d40bcf50 con 0x55835d7553f0 2024-03-23T23:11:25.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.939+0000 7f33d0c00640 1 -- 172.21.15.12:0/1596819386 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f33d40ee680 con 0x55835d7553f0 2024-03-23T23:11:25.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.939+0000 7f33ecb66740 1 -- 172.21.15.12:0/1596819386 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f33cc012ad0 con 0x55835d7553f0 2024-03-23T23:11:25.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.939+0000 7f33d0c00640 1 -- 172.21.15.12:0/1596819386 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33d40bcd70 con 0x55835d7553f0 2024-03-23T23:11:25.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.939+0000 7f33ecb66740 10 client.28003 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:25.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.939+0000 7f33ecb66740 1 -- 172.21.15.12:0/1596819386 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55835d49b620 con 0x55835d7553f0 2024-03-23T23:11:25.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.939+0000 7f33d0c00640 1 -- 172.21.15.12:0/1596819386 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33cc012ad0 con 0x55835d7553f0 2024-03-23T23:11:25.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.940+0000 7f33d0c00640 1 -- 172.21.15.12:0/1596819386 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f33d40c1150 con 0x55835d7553f0 2024-03-23T23:11:25.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.940+0000 7f33ecb66740 10 client.28003 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:25.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.940+0000 7f33ecb66740 10 client.28003 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:11:25.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.940+0000 7f33ecb66740 20 client.28003 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:25.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.940+0000 7f33ecb66740 1 --2- 172.21.15.12:0/1596819386 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55835d76b3f0 0x55835d76d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:25.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.940+0000 7f33ecb66740 4 client.28003 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:11:25.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.940+0000 7f33ecb66740 1 -- 172.21.15.12:0/1596819386 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55835d6c7d70 con 0x55835d76b3f0 2024-03-23T23:11:25.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.940+0000 7f33da000640 1 --2- 172.21.15.12:0/1596819386 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55835d76b3f0 0x55835d76d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:25.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.941+0000 7f33da000640 1 --2- 172.21.15.12:0/1596819386 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55835d76b3f0 0x55835d76d860 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=198730498ec9ed95 server_cookie=ec401772968a2311 in_seq=0 out_seq=0 2024-03-23T23:11:25.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.941+0000 7f33d0c00640 10 client.28003 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:25.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.941+0000 7f33d0c00640 1 -- 172.21.15.12:0/1596819386 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55835d6c7d70 con 0x55835d76b3f0 2024-03-23T23:11:25.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.941+0000 7f33d0c00640 10 client.28003 handle_command_reply: tid=0 2024-03-23T23:11:25.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.941+0000 7f33ecb66740 1 client.28003 shutdown 2024-03-23T23:11:25.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.942+0000 7f33ecb66740 1 -- 172.21.15.12:0/1596819386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d7553f0 msgr2=0x55835d75f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:25.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.942+0000 7f33ecb66740 1 --2- 172.21.15.12:0/1596819386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d7553f0 0x55835d75f5a0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f33d40b9870 tx=0x7f33d4009620 comp rx=0 tx=0).stop 2024-03-23T23:11:25.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.942+0000 7f33ecb66740 1 -- 172.21.15.12:0/1596819386 shutdown_connections 2024-03-23T23:11:25.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.942+0000 7f33ecb66740 1 --2- 172.21.15.12:0/1596819386 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55835d76b3f0 0x55835d76d860 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.942+0000 7f33ecb66740 1 --2- 172.21.15.12:0/1596819386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55835d75af50 0x55835d766b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.942+0000 7f33ecb66740 1 --2- 172.21.15.12:0/1596819386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55835d755de0 0x55835d75fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.942+0000 7f33ecb66740 1 --2- 172.21.15.12:0/1596819386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d7553f0 0x55835d75f5a0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.942+0000 7f33d0c00640 0 client.28003 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:25.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.942+0000 7f33ecb66740 1 -- 172.21.15.12:0/1596819386 >> 172.21.15.12:0/1596819386 conn(0x55835d750960 msgr2=0x55835d74fb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:25.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.942+0000 7f33ecb66740 1 -- 172.21.15.12:0/1596819386 shutdown_connections 2024-03-23T23:11:25.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.942+0000 7f33ecb66740 1 -- 172.21.15.12:0/1596819386 wait complete. 2024-03-23T23:11:25.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.942+0000 7f33ecb66740 20 client.28003 trim_cache size 0 max 16384 2024-03-23T23:11:25.958 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.958+0000 7f33ecb66740 1 Processor -- start 2024-03-23T23:11:25.958 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.958+0000 7f33ecb66740 1 -- start start 2024-03-23T23:11:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.958+0000 7f33ecb66740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55835d750960 0x55835d75f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.958+0000 7f33ecb66740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55835d7553f0 0x55835d766ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.958+0000 7f33ecb66740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d755de0 0x55835d75f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.958+0000 7f33ecb66740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e41acff0 con 0x55835d750960 2024-03-23T23:11:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.958+0000 7f33ecb66740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e41aaaf0 con 0x55835d7553f0 2024-03-23T23:11:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.958+0000 7f33ecb66740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e41a8b90 con 0x55835d755de0 2024-03-23T23:11:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.959+0000 7f33da000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55835d7553f0 0x55835d766ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.959+0000 7f33e2a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55835d750960 0x55835d75f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.959+0000 7f33e3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d755de0 0x55835d75f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.959+0000 7f33da000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55835d7553f0 0x55835d766ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:41682/0 (socket says 172.21.15.12:41682) 2024-03-23T23:11:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.959+0000 7f33da000640 1 -- 172.21.15.12:0/1283645281 learned_addr learned my addr 172.21.15.12:0/1283645281 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.959+0000 7f33e2a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55835d750960 0x55835d75f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45124/0 (socket says 172.21.15.12:45124) 2024-03-23T23:11:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.959+0000 7f33e3400640 1 -- 172.21.15.12:0/1283645281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55835d7553f0 msgr2=0x55835d766ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.959+0000 7f33e3400640 1 --2- 172.21.15.12:0/1283645281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55835d7553f0 0x55835d766ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.959+0000 7f33e3400640 1 -- 172.21.15.12:0/1283645281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55835d750960 msgr2=0x55835d75f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:25.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.959+0000 7f33e3400640 1 --2- 172.21.15.12:0/1283645281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55835d750960 0x55835d75f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.959+0000 7f33e3400640 1 -- 172.21.15.12:0/1283645281 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55835d74bd10 con 0x55835d755de0 2024-03-23T23:11:25.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.959+0000 7f33e3400640 1 --2- 172.21.15.12:0/1283645281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d755de0 0x55835d75f6f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f33dc009f50 tx=0x7f33dc01a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c1d7b99d6c73fa9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:25.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.960+0000 7f33d3400640 1 -- 172.21.15.12:0/1283645281 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc00f4e0 con 0x55835d755de0 2024-03-23T23:11:25.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.960+0000 7f33d3400640 1 -- 172.21.15.12:0/1283645281 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33dc00fb60 con 0x55835d755de0 2024-03-23T23:11:25.960 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.960+0000 7f33d3400640 1 -- 172.21.15.12:0/1283645281 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc029db0 con 0x55835d755de0 2024-03-23T23:11:25.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.961+0000 7f33ecb66740 1 -- 172.21.15.12:0/1283645281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d755de0 msgr2=0x55835d75f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:25.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.961+0000 7f33ecb66740 1 --2- 172.21.15.12:0/1283645281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d755de0 0x55835d75f6f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f33dc009f50 tx=0x7f33dc01a000 comp rx=0 tx=0).stop 2024-03-23T23:11:25.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.961+0000 7f33ecb66740 1 -- 172.21.15.12:0/1283645281 shutdown_connections 2024-03-23T23:11:25.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.961+0000 7f33ecb66740 1 --2- 172.21.15.12:0/1283645281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d755de0 0x55835d75f6f0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.961+0000 7f33ecb66740 1 --2- 172.21.15.12:0/1283645281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55835d7553f0 0x55835d766ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.961+0000 7f33ecb66740 1 --2- 172.21.15.12:0/1283645281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55835d750960 0x55835d75f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.961+0000 7f33ecb66740 1 -- 172.21.15.12:0/1283645281 >> 172.21.15.12:0/1283645281 conn(0x55835d75aa90 msgr2=0x55835d750d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:25.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.961+0000 7f33ecb66740 1 -- 172.21.15.12:0/1283645281 shutdown_connections 2024-03-23T23:11:25.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.961+0000 7f33ecb66740 1 -- 172.21.15.12:0/1283645281 wait complete. 2024-03-23T23:11:25.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.961+0000 7f33ecb66740 1 -- start start 2024-03-23T23:11:25.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.962+0000 7f33ecb66740 1 Processor -- start 2024-03-23T23:11:25.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.962+0000 7f33ecb66740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55835d750960 0x55835d766800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:25.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.962+0000 7f33ecb66740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55835d7553f0 0x55835d764e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:25.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.962+0000 7f33ecb66740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d755de0 0x55835d765350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:25.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.962+0000 7f33ecb66740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e41a9190 con 0x55835d750960 2024-03-23T23:11:25.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.962+0000 7f33ecb66740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e41aa490 con 0x55835d7553f0 2024-03-23T23:11:25.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.962+0000 7f33ecb66740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e41a97d0 con 0x55835d755de0 2024-03-23T23:11:25.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.962+0000 7f33ecb66740 10 client.0 _ll_register_callbacks cb 0x7f33d40f40f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:25.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.962+0000 7f33da000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55835d7553f0 0x55835d764e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:25.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.962+0000 7f33e3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d755de0 0x55835d765350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:25.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.962+0000 7f33da000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55835d7553f0 0x55835d764e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:41688/0 (socket says 172.21.15.12:41688) 2024-03-23T23:11:25.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.962+0000 7f33da000640 1 -- 172.21.15.12:0/1241429688 learned_addr learned my addr 172.21.15.12:0/1241429688 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:25.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.962+0000 7f33e2a00640 1 --2- 172.21.15.12:0/1241429688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55835d750960 0x55835d766800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:25.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.963+0000 7f33e3400640 1 -- 172.21.15.12:0/1241429688 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55835d7553f0 msgr2=0x55835d764e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:25.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.963+0000 7f33e3400640 1 --2- 172.21.15.12:0/1241429688 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55835d7553f0 0x55835d764e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.963+0000 7f33e3400640 1 -- 172.21.15.12:0/1241429688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55835d750960 msgr2=0x55835d766800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:25.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.963+0000 7f33e3400640 1 --2- 172.21.15.12:0/1241429688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55835d750960 0x55835d766800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.963+0000 7f33e3400640 1 -- 172.21.15.12:0/1241429688 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55835d765990 con 0x55835d755de0 2024-03-23T23:11:25.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.963+0000 7f33da000640 1 --2- 172.21.15.12:0/1241429688 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55835d7553f0 0x55835d764e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:25.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.963+0000 7f33e3400640 1 --2- 172.21.15.12:0/1241429688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d755de0 0x55835d765350 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f33dc012010 tx=0x7f33dc009760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eda31474249f0b6c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:25.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.963+0000 7f33d0c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:11:25.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.963+0000 7f33d0c00640 1 -- 172.21.15.12:0/1241429688 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc029af0 con 0x55835d755de0 2024-03-23T23:11:25.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.963+0000 7f33ecb66740 1 -- 172.21.15.12:0/1241429688 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f33dc00fb60 con 0x55835d755de0 2024-03-23T23:11:25.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.963+0000 7f33d0c00640 1 -- 172.21.15.12:0/1241429688 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f33dc029c90 con 0x55835d755de0 2024-03-23T23:11:25.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.964+0000 7f33d0c00640 1 -- 172.21.15.12:0/1241429688 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33dc004a40 con 0x55835d755de0 2024-03-23T23:11:25.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.964+0000 7f33ecb66740 10 client.28009 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:25.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.964+0000 7f33ecb66740 1 -- 172.21.15.12:0/1241429688 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55835d7c7c40 con 0x55835d755de0 2024-03-23T23:11:25.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.964+0000 7f33d0c00640 1 -- 172.21.15.12:0/1241429688 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33dc00fb60 con 0x55835d755de0 2024-03-23T23:11:25.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.964+0000 7f33d0c00640 1 -- 172.21.15.12:0/1241429688 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f33dc005040 con 0x55835d755de0 2024-03-23T23:11:25.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.964+0000 7f33ecb66740 10 client.28009 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:25.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.964+0000 7f33ecb66740 10 client.28009 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:11:25.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.964+0000 7f33ecb66740 20 client.28009 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:25.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.964+0000 7f33ecb66740 1 --2- 172.21.15.12:0/1241429688 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55835d75af50 0x55835d7ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:25.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.964+0000 7f33ecb66740 4 client.28009 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:11:25.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.964+0000 7f33ecb66740 1 -- 172.21.15.12:0/1241429688 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55835d6c20c0 con 0x55835d75af50 2024-03-23T23:11:25.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.964+0000 7f33e2a00640 1 --2- 172.21.15.12:0/1241429688 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55835d75af50 0x55835d7ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:25.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.965+0000 7f33e2a00640 1 --2- 172.21.15.12:0/1241429688 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55835d75af50 0x55835d7ca5c0 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=837b536e1fb01 server_cookie=cb6398f54212f6ef in_seq=0 out_seq=0 2024-03-23T23:11:25.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.965+0000 7f33d0c00640 10 client.28009 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.977+0000 7f33d0c00640 1 -- 172.21.15.12:0/1241429688 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55835d6c20c0 con 0x55835d75af50 2024-03-23T23:11:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.977+0000 7f33d0c00640 10 client.28009 handle_command_reply: tid=0 2024-03-23T23:11:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.977+0000 7f33ecb66740 1 client.28009 shutdown 2024-03-23T23:11:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.977+0000 7f33ecb66740 1 -- 172.21.15.12:0/1241429688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d755de0 msgr2=0x55835d765350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.977+0000 7f33ecb66740 1 --2- 172.21.15.12:0/1241429688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d755de0 0x55835d765350 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f33dc012010 tx=0x7f33dc009760 comp rx=0 tx=0).stop 2024-03-23T23:11:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.977+0000 7f33ecb66740 1 -- 172.21.15.12:0/1241429688 shutdown_connections 2024-03-23T23:11:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.977+0000 7f33ecb66740 1 --2- 172.21.15.12:0/1241429688 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55835d75af50 0x55835d7ca5c0 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.978+0000 7f33ecb66740 1 --2- 172.21.15.12:0/1241429688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55835d755de0 0x55835d765350 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.978+0000 7f33ecb66740 1 --2- 172.21.15.12:0/1241429688 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55835d7553f0 0x55835d764e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.978+0000 7f33ecb66740 1 --2- 172.21.15.12:0/1241429688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55835d750960 0x55835d766800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.978+0000 7f33ecb66740 1 -- 172.21.15.12:0/1241429688 >> 172.21.15.12:0/1241429688 conn(0x55835d75aa90 msgr2=0x55835d75f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:25.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.978+0000 7f33d0c00640 0 client.28009 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:25.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.978+0000 7f33ecb66740 1 -- 172.21.15.12:0/1241429688 shutdown_connections 2024-03-23T23:11:25.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.978+0000 7f33ecb66740 1 -- 172.21.15.12:0/1241429688 wait complete. 2024-03-23T23:11:25.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.978+0000 7f33ecb66740 20 client.28009 trim_cache size 0 max 16384 2024-03-23T23:11:25.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:11:25.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:11:25.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:11:25.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:11:25.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.979+0000 7f33e3e00640 1 -- 172.21.15.12:0/1760333266 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f33c80a2870 msgr2=0x7f33c80a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:25.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.979+0000 7f33e3e00640 1 --2- 172.21.15.12:0/1760333266 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f33c80a2870 0x7f33c80a4d30 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x55835d756640 tx=0x7f33dc009500 comp rx=0 tx=0).stop 2024-03-23T23:11:25.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.979+0000 7f33e3e00640 1 -- 172.21.15.12:0/1760333266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33e4101520 msgr2=0x7f33e4199a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:25.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.979+0000 7f33e3e00640 1 --2- 172.21.15.12:0/1760333266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33e4101520 0x7f33e4199a40 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f33d4000c00 tx=0x7f33d402c650 comp rx=0 tx=0).stop 2024-03-23T23:11:25.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.979+0000 7f33e3e00640 1 -- 172.21.15.12:0/1760333266 shutdown_connections 2024-03-23T23:11:25.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.979+0000 7f33e3e00640 1 --2- 172.21.15.12:0/1760333266 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f33c80a2870 0x7f33c80a4d30 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.979+0000 7f33e3e00640 1 --2- 172.21.15.12:0/1760333266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33e41030d0 0x7f33e41a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.979+0000 7f33e3e00640 1 --2- 172.21.15.12:0/1760333266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33e4102710 0x7f33e4199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.979+0000 7f33e3e00640 1 --2- 172.21.15.12:0/1760333266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33e4101520 0x7f33e4199a40 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:25.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.979+0000 7f33e3e00640 1 -- 172.21.15.12:0/1760333266 >> 172.21.15.12:0/1760333266 conn(0x7f33e40fcab0 msgr2=0x7f33e40738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:25.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.979+0000 7f33e3e00640 1 -- 172.21.15.12:0/1760333266 shutdown_connections 2024-03-23T23:11:25.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:25.979+0000 7f33e3e00640 1 -- 172.21.15.12:0/1760333266 wait complete. 2024-03-23T23:11:26.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.118+0000 7f1034a00640 1 -- 172.21.15.12:0/333908298 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1030102000 msgr2=0x7f10301043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:26.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.118+0000 7f1034a00640 1 --2- 172.21.15.12:0/333908298 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1030102000 0x7f10301043f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f1020009ca0 tx=0x7f102002c100 comp rx=0 tx=0).stop 2024-03-23T23:11:26.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.118+0000 7f1034a00640 1 -- 172.21.15.12:0/333908298 shutdown_connections 2024-03-23T23:11:26.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.118+0000 7f1034a00640 1 --2- 172.21.15.12:0/333908298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1030107260 0x7f10301096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.118+0000 7f1034a00640 1 --2- 172.21.15.12:0/333908298 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1030104930 0x7f1030106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.118+0000 7f1034a00640 1 --2- 172.21.15.12:0/333908298 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1030102000 0x7f10301043f0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.118+0000 7f1034a00640 1 -- 172.21.15.12:0/333908298 >> 172.21.15.12:0/333908298 conn(0x7f10300fba10 msgr2=0x7f10300fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:26.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.118+0000 7f1034a00640 1 -- 172.21.15.12:0/333908298 shutdown_connections 2024-03-23T23:11:26.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.118+0000 7f1034a00640 1 -- 172.21.15.12:0/333908298 wait complete. 2024-03-23T23:11:26.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.119+0000 7f1034a00640 1 Processor -- start 2024-03-23T23:11:26.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.119+0000 7f1034a00640 1 -- start start 2024-03-23T23:11:26.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.120+0000 7f1034a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1030102000 0x7f1030197890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.120+0000 7f1034a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1030104930 0x7f1030197dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.120+0000 7f1034a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1030107260 0x7f103019ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.120+0000 7f1034a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103010d210 con 0x7f1030102000 2024-03-23T23:11:26.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.120+0000 7f1034a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103010d090 con 0x7f1030104930 2024-03-23T23:11:26.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.120+0000 7f1034a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103010d390 con 0x7f1030107260 2024-03-23T23:11:26.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.120+0000 7f102f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1030102000 0x7f1030197890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.120+0000 7f102ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1030104930 0x7f1030197dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.120+0000 7f102f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1030102000 0x7f1030197890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45156/0 (socket says 172.21.15.12:45156) 2024-03-23T23:11:26.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.120+0000 7f102f400640 1 -- 172.21.15.12:0/678301102 learned_addr learned my addr 172.21.15.12:0/678301102 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:26.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.120+0000 7f102fe00640 1 --2- 172.21.15.12:0/678301102 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1030107260 0x7f103019ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.121+0000 7f102ea00640 1 -- 172.21.15.12:0/678301102 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1030107260 msgr2=0x7f103019ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:26.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.121+0000 7f102ea00640 1 --2- 172.21.15.12:0/678301102 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1030107260 0x7f103019ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.121+0000 7f102ea00640 1 -- 172.21.15.12:0/678301102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1030102000 msgr2=0x7f1030197890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:26.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.121+0000 7f102ea00640 1 --2- 172.21.15.12:0/678301102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1030102000 0x7f1030197890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.121+0000 7f102ea00640 1 -- 172.21.15.12:0/678301102 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103019f4b0 con 0x7f1030104930 2024-03-23T23:11:26.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.121+0000 7f102f400640 1 --2- 172.21.15.12:0/678301102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1030102000 0x7f1030197890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:26.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.121+0000 7f102ea00640 1 --2- 172.21.15.12:0/678301102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1030104930 0x7f1030197dd0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f101800e9c0 tx=0x7f101800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:26.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.121+0000 7f1027e00640 1 -- 172.21.15.12:0/678301102 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101800ccc0 con 0x7f1030104930 2024-03-23T23:11:26.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.121+0000 7f1034a00640 1 -- 172.21.15.12:0/678301102 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10300ffe50 con 0x7f1030104930 2024-03-23T23:11:26.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.121+0000 7f1027e00640 1 -- 172.21.15.12:0/678301102 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1018005020 con 0x7f1030104930 2024-03-23T23:11:26.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.121+0000 7f1027e00640 1 -- 172.21.15.12:0/678301102 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1018004a00 con 0x7f1030104930 2024-03-23T23:11:26.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.121+0000 7f1034a00640 1 -- 172.21.15.12:0/678301102 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10301003a0 con 0x7f1030104930 2024-03-23T23:11:26.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.125+0000 7f1027e00640 1 -- 172.21.15.12:0/678301102 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f10180051d0 con 0x7f1030104930 2024-03-23T23:11:26.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.125+0000 7f1038dfe740 1 Processor -- start 2024-03-23T23:11:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.125+0000 7f1038dfe740 1 -- start start 2024-03-23T23:11:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.125+0000 7f1038dfe740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565011d112b0 0x565011d116d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.125+0000 7f1038dfe740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d11ca0 0x565011d167a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.125+0000 7f1038dfe740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565011d16e10 0x565011d192a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.125+0000 7f1038dfe740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x565011c76eb0 con 0x565011d112b0 2024-03-23T23:11:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.125+0000 7f1038dfe740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x565011c1fa60 con 0x565011d11ca0 2024-03-23T23:11:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.125+0000 7f1038dfe740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x565011c72f50 con 0x565011d16e10 2024-03-23T23:11:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.125+0000 7f102f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d11ca0 0x565011d167a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.125+0000 7f102ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565011d16e10 0x565011d192a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.125+0000 7f102f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d11ca0 0x565011d167a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:41708/0 (socket says 172.21.15.12:41708) 2024-03-23T23:11:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.125+0000 7f102f400640 1 -- 172.21.15.12:0/3360596983 learned_addr learned my addr 172.21.15.12:0/3360596983 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.125+0000 7f102fe00640 1 --2- 172.21.15.12:0/3360596983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565011d112b0 0x565011d116d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.126+0000 7f102f400640 1 -- 172.21.15.12:0/3360596983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565011d16e10 msgr2=0x565011d192a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.126+0000 7f102f400640 1 --2- 172.21.15.12:0/3360596983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565011d16e10 0x565011d192a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.126+0000 7f102f400640 1 -- 172.21.15.12:0/3360596983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565011d112b0 msgr2=0x565011d116d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:26.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.126+0000 7f102f400640 1 --2- 172.21.15.12:0/3360596983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565011d112b0 0x565011d116d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.126+0000 7f102f400640 1 -- 172.21.15.12:0/3360596983 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565011a537e0 con 0x565011d11ca0 2024-03-23T23:11:26.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.126+0000 7f102fe00640 1 --2- 172.21.15.12:0/3360596983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565011d112b0 0x565011d116d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:11:26.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.126+0000 7f102f400640 1 --2- 172.21.15.12:0/3360596983 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d11ca0 0x565011d167a0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f1020037040 tx=0x7f10200323e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3430fe542850c31 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:26.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.126+0000 7f1027e00640 1 --2- 172.21.15.12:0/678301102 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f10140a2800 0x7f10140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.126+0000 7f101fe00640 1 -- 172.21.15.12:0/3360596983 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1020043070 con 0x565011d11ca0 2024-03-23T23:11:26.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.126+0000 7f1027e00640 1 -- 172.21.15.12:0/678301102 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f10180cea00 con 0x7f1030104930 2024-03-23T23:11:26.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.127+0000 7f101fe00640 1 -- 172.21.15.12:0/3360596983 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1020003970 con 0x565011d11ca0 2024-03-23T23:11:26.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.127+0000 7f102fe00640 1 --2- 172.21.15.12:0/678301102 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f10140a2800 0x7f10140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.127+0000 7f101fe00640 1 -- 172.21.15.12:0/3360596983 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1020039040 con 0x565011d11ca0 2024-03-23T23:11:26.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.127+0000 7f1038dfe740 1 -- 172.21.15.12:0/3360596983 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d11ca0 msgr2=0x565011d167a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:26.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.127+0000 7f1038dfe740 1 --2- 172.21.15.12:0/3360596983 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d11ca0 0x565011d167a0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f1020037040 tx=0x7f10200323e0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.127+0000 7f102fe00640 1 --2- 172.21.15.12:0/678301102 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f10140a2800 0x7f10140a4cc0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x565011d12500 tx=0x7f1028005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:26.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.127+0000 7f1038dfe740 1 -- 172.21.15.12:0/3360596983 shutdown_connections 2024-03-23T23:11:26.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.127+0000 7f1038dfe740 1 --2- 172.21.15.12:0/3360596983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565011d16e10 0x565011d192a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.127+0000 7f1038dfe740 1 --2- 172.21.15.12:0/3360596983 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d11ca0 0x565011d167a0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.127+0000 7f1038dfe740 1 --2- 172.21.15.12:0/3360596983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565011d112b0 0x565011d116d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.127+0000 7f1038dfe740 1 -- 172.21.15.12:0/3360596983 >> 172.21.15.12:0/3360596983 conn(0x565011d0c820 msgr2=0x565011d0cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:26.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.127+0000 7f1038dfe740 1 -- 172.21.15.12:0/3360596983 shutdown_connections 2024-03-23T23:11:26.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.127+0000 7f1038dfe740 1 -- 172.21.15.12:0/3360596983 wait complete. 2024-03-23T23:11:26.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.128+0000 7f1038dfe740 1 -- start start 2024-03-23T23:11:26.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.128+0000 7f1038dfe740 1 Processor -- start 2024-03-23T23:11:26.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.129+0000 7f1038dfe740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565011d112b0 0x565011d1b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.129+0000 7f1038dfe740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d11ca0 0x565011d1b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.129+0000 7f1038dfe740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565011d16e10 0x565011d22960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.129+0000 7f1038dfe740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x565011d19d60 con 0x565011d16e10 2024-03-23T23:11:26.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.129+0000 7f1038dfe740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x565011c32eb0 con 0x565011d11ca0 2024-03-23T23:11:26.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.129+0000 7f1038dfe740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x565011d19ee0 con 0x565011d112b0 2024-03-23T23:11:26.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.129+0000 7f1038dfe740 10 client.0 _ll_register_callbacks cb 0x565011c27150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:26.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.129+0000 7f102ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d11ca0 0x565011d1b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.129+0000 7f102f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565011d112b0 0x565011d1b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.129+0000 7f102ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d11ca0 0x565011d1b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:41722/0 (socket says 172.21.15.12:41722) 2024-03-23T23:11:26.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.129+0000 7f102ea00640 1 -- 172.21.15.12:0/3808115088 learned_addr learned my addr 172.21.15.12:0/3808115088 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:26.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.129+0000 7f102fe00640 1 --2- 172.21.15.12:0/3808115088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565011d16e10 0x565011d22960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.129+0000 7f102ea00640 1 -- 172.21.15.12:0/3808115088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565011d112b0 msgr2=0x565011d1b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:26.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.129+0000 7f102ea00640 1 --2- 172.21.15.12:0/3808115088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565011d112b0 0x565011d1b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.129+0000 7f102ea00640 1 -- 172.21.15.12:0/3808115088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565011d16e10 msgr2=0x565011d22960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:26.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.129+0000 7f102ea00640 1 --2- 172.21.15.12:0/3808115088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565011d16e10 0x565011d22960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.129+0000 7f102ea00640 1 -- 172.21.15.12:0/3808115088 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565011ab3f70 con 0x565011d11ca0 2024-03-23T23:11:26.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.129+0000 7f102fe00640 1 --2- 172.21.15.12:0/3808115088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565011d16e10 0x565011d22960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:11:26.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.129+0000 7f102f400640 1 --2- 172.21.15.12:0/3808115088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565011d112b0 0x565011d1b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:26.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.130+0000 7f102ea00640 1 --2- 172.21.15.12:0/3808115088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d11ca0 0x565011d1b920 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f10180a35c0 tx=0x7f101809b1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=185f670cdd4fcb69 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:26.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.130+0000 7f101cc00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:11:26.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.130+0000 7f101cc00640 1 -- 172.21.15.12:0/3808115088 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101801c070 con 0x565011d11ca0 2024-03-23T23:11:26.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.130+0000 7f101cc00640 1 -- 172.21.15.12:0/3808115088 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f10180cc5c0 con 0x565011d11ca0 2024-03-23T23:11:26.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.130+0000 7f1038dfe740 1 -- 172.21.15.12:0/3808115088 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1020003970 con 0x565011d11ca0 2024-03-23T23:11:26.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.130+0000 7f101cc00640 1 -- 172.21.15.12:0/3808115088 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f101809b350 con 0x565011d11ca0 2024-03-23T23:11:26.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.130+0000 7f1038dfe740 10 client.28055 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:26.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.130+0000 7f1038dfe740 1 -- 172.21.15.12:0/3808115088 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56501194b170 con 0x565011d11ca0 2024-03-23T23:11:26.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.130+0000 7f101cc00640 1 -- 172.21.15.12:0/3808115088 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1020003970 con 0x565011d11ca0 2024-03-23T23:11:26.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.130+0000 7f101cc00640 1 -- 172.21.15.12:0/3808115088 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f10180a3660 con 0x565011d11ca0 2024-03-23T23:11:26.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.131+0000 7f1038dfe740 10 client.28055 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:26.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.131+0000 7f1038dfe740 10 client.28055 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:11:26.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.131+0000 7f1038dfe740 20 client.28055 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:26.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.131+0000 7f1038dfe740 1 --2- 172.21.15.12:0/3808115088 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x565011d27180 0x565011d295c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.131+0000 7f1038dfe740 4 client.28055 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:11:26.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.131+0000 7f1038dfe740 1 -- 172.21.15.12:0/3808115088 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565011c843f0 con 0x565011d27180 2024-03-23T23:11:26.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.131+0000 7f102f400640 1 --2- 172.21.15.12:0/3808115088 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x565011d27180 0x565011d295c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.131+0000 7f102f400640 1 --2- 172.21.15.12:0/3808115088 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x565011d27180 0x565011d295c0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a205b2793ffc355c server_cookie=869931adccb67a42 in_seq=0 out_seq=0 2024-03-23T23:11:26.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.131+0000 7f101cc00640 10 client.28055 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:26.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.132+0000 7f101cc00640 1 -- 172.21.15.12:0/3808115088 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x565011c843f0 con 0x565011d27180 2024-03-23T23:11:26.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.132+0000 7f101cc00640 10 client.28055 handle_command_reply: tid=0 2024-03-23T23:11:26.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.132+0000 7f1038dfe740 1 client.28055 shutdown 2024-03-23T23:11:26.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.133+0000 7f1038dfe740 1 -- 172.21.15.12:0/3808115088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d11ca0 msgr2=0x565011d1b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:26.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.133+0000 7f1038dfe740 1 --2- 172.21.15.12:0/3808115088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d11ca0 0x565011d1b920 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f10180a35c0 tx=0x7f101809b1e0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.133+0000 7f1038dfe740 1 -- 172.21.15.12:0/3808115088 shutdown_connections 2024-03-23T23:11:26.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.133+0000 7f1038dfe740 1 --2- 172.21.15.12:0/3808115088 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x565011d27180 0x565011d295c0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.133+0000 7f1038dfe740 1 --2- 172.21.15.12:0/3808115088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565011d16e10 0x565011d22960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.133+0000 7f1038dfe740 1 --2- 172.21.15.12:0/3808115088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d11ca0 0x565011d1b920 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.133+0000 7f101cc00640 0 client.28055 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:26.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.133+0000 7f1038dfe740 1 --2- 172.21.15.12:0/3808115088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565011d112b0 0x565011d1b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.133+0000 7f1038dfe740 1 -- 172.21.15.12:0/3808115088 >> 172.21.15.12:0/3808115088 conn(0x565011d0c820 msgr2=0x565011d0ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:26.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.133+0000 7f1038dfe740 1 -- 172.21.15.12:0/3808115088 shutdown_connections 2024-03-23T23:11:26.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.133+0000 7f1038dfe740 1 -- 172.21.15.12:0/3808115088 wait complete. 2024-03-23T23:11:26.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.133+0000 7f1038dfe740 20 client.28055 trim_cache size 0 max 16384 2024-03-23T23:11:26.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.151+0000 7f1038dfe740 1 Processor -- start 2024-03-23T23:11:26.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.151+0000 7f1038dfe740 1 -- start start 2024-03-23T23:11:26.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.151+0000 7f1038dfe740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d1a5f0 0x565011d2de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.151+0000 7f1038dfe740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565011d0c820 0x565011d22920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.151+0000 7f1038dfe740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565011d112b0 0x565011d2e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.151+0000 7f1038dfe740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10301acf30 con 0x565011d112b0 2024-03-23T23:11:26.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.151+0000 7f1038dfe740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10301aaa30 con 0x565011d1a5f0 2024-03-23T23:11:26.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.151+0000 7f1038dfe740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10301a8ad0 con 0x565011d0c820 2024-03-23T23:11:26.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.152+0000 7f102f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d1a5f0 0x565011d2de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.152+0000 7f102ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565011d0c820 0x565011d22920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.152+0000 7f102f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d1a5f0 0x565011d2de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:41736/0 (socket says 172.21.15.12:41736) 2024-03-23T23:11:26.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.152+0000 7f102f400640 1 -- 172.21.15.12:0/3629091631 learned_addr learned my addr 172.21.15.12:0/3629091631 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:26.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.152+0000 7f102fe00640 1 --2- 172.21.15.12:0/3629091631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565011d112b0 0x565011d2e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.152+0000 7f102f400640 1 -- 172.21.15.12:0/3629091631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565011d0c820 msgr2=0x565011d22920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:26.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.152+0000 7f102f400640 1 --2- 172.21.15.12:0/3629091631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565011d0c820 0x565011d22920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.152+0000 7f102f400640 1 -- 172.21.15.12:0/3629091631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565011d112b0 msgr2=0x565011d2e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:26.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.152+0000 7f102f400640 1 --2- 172.21.15.12:0/3629091631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565011d112b0 0x565011d2e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.152+0000 7f102f400640 1 -- 172.21.15.12:0/3629091631 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565011d0d590 con 0x565011d1a5f0 2024-03-23T23:11:26.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.152+0000 7f102ea00640 1 --2- 172.21.15.12:0/3629091631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565011d0c820 0x565011d22920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:26.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.152+0000 7f102f400640 1 --2- 172.21.15.12:0/3629091631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d1a5f0 0x565011d2de80 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f10200325f0 tx=0x7f102002c100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cd9cfe36c593a29 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:26.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.153+0000 7f101f400640 1 -- 172.21.15.12:0/3629091631 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1020057070 con 0x565011d1a5f0 2024-03-23T23:11:26.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.153+0000 7f101f400640 1 -- 172.21.15.12:0/3629091631 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1020032c90 con 0x565011d1a5f0 2024-03-23T23:11:26.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.153+0000 7f101f400640 1 -- 172.21.15.12:0/3629091631 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1020039040 con 0x565011d1a5f0 2024-03-23T23:11:26.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.153+0000 7f1038dfe740 1 -- 172.21.15.12:0/3629091631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d1a5f0 msgr2=0x565011d2de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:26.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.153+0000 7f1038dfe740 1 --2- 172.21.15.12:0/3629091631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d1a5f0 0x565011d2de80 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f10200325f0 tx=0x7f102002c100 comp rx=0 tx=0).stop 2024-03-23T23:11:26.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.153+0000 7f1038dfe740 1 -- 172.21.15.12:0/3629091631 shutdown_connections 2024-03-23T23:11:26.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.153+0000 7f1038dfe740 1 --2- 172.21.15.12:0/3629091631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565011d112b0 0x565011d2e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.153+0000 7f1038dfe740 1 --2- 172.21.15.12:0/3629091631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565011d0c820 0x565011d22920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.153+0000 7f1038dfe740 1 --2- 172.21.15.12:0/3629091631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d1a5f0 0x565011d2de80 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.153+0000 7f1038dfe740 1 -- 172.21.15.12:0/3629091631 >> 172.21.15.12:0/3629091631 conn(0x565011d106b0 msgr2=0x565011d0ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:26.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.153+0000 7f1038dfe740 1 -- 172.21.15.12:0/3629091631 shutdown_connections 2024-03-23T23:11:26.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.153+0000 7f1038dfe740 1 -- 172.21.15.12:0/3629091631 wait complete. 2024-03-23T23:11:26.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.154+0000 7f1038dfe740 1 -- start start 2024-03-23T23:11:26.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.154+0000 7f1038dfe740 1 Processor -- start 2024-03-23T23:11:26.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.154+0000 7f1038dfe740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d0c820 0x565011d213a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.154+0000 7f1038dfe740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565011d112b0 0x565011d218e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.154+0000 7f1038dfe740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565011d1a5f0 0x565011d21e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.154+0000 7f1038dfe740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10301a90d0 con 0x565011d112b0 2024-03-23T23:11:26.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.154+0000 7f1038dfe740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10301aa3d0 con 0x565011d0c820 2024-03-23T23:11:26.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.154+0000 7f1038dfe740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10301a9710 con 0x565011d1a5f0 2024-03-23T23:11:26.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.154+0000 7f1038dfe740 10 client.0 _ll_register_callbacks cb 0x7f10180d4220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:26.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.155+0000 7f102f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d0c820 0x565011d213a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.155+0000 7f102f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d0c820 0x565011d213a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:41744/0 (socket says 172.21.15.12:41744) 2024-03-23T23:11:26.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.155+0000 7f102f400640 1 -- 172.21.15.12:0/3925205800 learned_addr learned my addr 172.21.15.12:0/3925205800 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:26.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.155+0000 7f102fe00640 1 --2- 172.21.15.12:0/3925205800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565011d1a5f0 0x565011d21e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.155+0000 7f102ea00640 1 --2- 172.21.15.12:0/3925205800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565011d112b0 0x565011d218e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.155+0000 7f102f400640 1 -- 172.21.15.12:0/3925205800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565011d1a5f0 msgr2=0x565011d21e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:26.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.155+0000 7f102f400640 1 --2- 172.21.15.12:0/3925205800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565011d1a5f0 0x565011d21e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.155+0000 7f102f400640 1 -- 172.21.15.12:0/3925205800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565011d112b0 msgr2=0x565011d218e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:26.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.155+0000 7f102f400640 1 --2- 172.21.15.12:0/3925205800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565011d112b0 0x565011d218e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.155+0000 7f102f400640 1 -- 172.21.15.12:0/3925205800 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565011d82c70 con 0x565011d0c820 2024-03-23T23:11:26.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.155+0000 7f102f400640 1 --2- 172.21.15.12:0/3925205800 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d0c820 0x565011d213a0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f1020051480 tx=0x7f1020005f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=646dc16a31b4d5e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:26.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.155+0000 7f101cc00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:11:26.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.155+0000 7f101cc00640 1 -- 172.21.15.12:0/3925205800 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1020057070 con 0x565011d0c820 2024-03-23T23:11:26.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.156+0000 7f101cc00640 1 -- 172.21.15.12:0/3925205800 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f10200347f0 con 0x565011d0c820 2024-03-23T23:11:26.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.156+0000 7f1038dfe740 1 -- 172.21.15.12:0/3925205800 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1020032c90 con 0x565011d0c820 2024-03-23T23:11:26.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.156+0000 7f101cc00640 1 -- 172.21.15.12:0/3925205800 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1020039050 con 0x565011d0c820 2024-03-23T23:11:26.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.156+0000 7f1038dfe740 10 client.28061 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:26.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.156+0000 7f1038dfe740 1 -- 172.21.15.12:0/3925205800 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x565011d83fb0 con 0x565011d0c820 2024-03-23T23:11:26.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.156+0000 7f101cc00640 1 -- 172.21.15.12:0/3925205800 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1020032c90 con 0x565011d0c820 2024-03-23T23:11:26.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.156+0000 7f101cc00640 1 -- 172.21.15.12:0/3925205800 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f1020003f50 con 0x565011d0c820 2024-03-23T23:11:26.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.156+0000 7f1038dfe740 10 client.28061 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:26.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.156+0000 7f1038dfe740 10 client.28061 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:11:26.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.156+0000 7f1038dfe740 20 client.28061 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:26.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.156+0000 7f1038dfe740 1 --2- 172.21.15.12:0/3925205800 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x565011d11ca0 0x565011d863e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.156+0000 7f1038dfe740 4 client.28061 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-23T23:11:26.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.156+0000 7f1038dfe740 1 -- 172.21.15.12:0/3925205800 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x565011c7e740 con 0x565011d11ca0 2024-03-23T23:11:26.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.157+0000 7f102ea00640 1 --2- 172.21.15.12:0/3925205800 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x565011d11ca0 0x565011d863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.157+0000 7f102ea00640 1 --2- 172.21.15.12:0/3925205800 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x565011d11ca0 0x565011d863e0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1f12e30da5e3779 server_cookie=5079151338ca29a2 in_seq=0 out_seq=0 2024-03-23T23:11:26.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.157+0000 7f101cc00640 10 client.28061 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:26.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.158+0000 7f101cc00640 1 -- 172.21.15.12:0/3925205800 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x565011c7e740 con 0x565011d11ca0 2024-03-23T23:11:26.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.158+0000 7f101cc00640 10 client.28061 handle_command_reply: tid=0 2024-03-23T23:11:26.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.158+0000 7f1038dfe740 1 client.28061 shutdown 2024-03-23T23:11:26.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.158+0000 7f1038dfe740 1 -- 172.21.15.12:0/3925205800 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d0c820 msgr2=0x565011d213a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:26.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.158+0000 7f1038dfe740 1 --2- 172.21.15.12:0/3925205800 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d0c820 0x565011d213a0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f1020051480 tx=0x7f1020005f80 comp rx=0 tx=0).stop 2024-03-23T23:11:26.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.158+0000 7f1038dfe740 1 -- 172.21.15.12:0/3925205800 shutdown_connections 2024-03-23T23:11:26.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.158+0000 7f1038dfe740 1 --2- 172.21.15.12:0/3925205800 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x565011d11ca0 0x565011d863e0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.158+0000 7f1038dfe740 1 --2- 172.21.15.12:0/3925205800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565011d1a5f0 0x565011d21e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.158+0000 7f1038dfe740 1 --2- 172.21.15.12:0/3925205800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565011d112b0 0x565011d218e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.158+0000 7f1038dfe740 1 --2- 172.21.15.12:0/3925205800 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565011d0c820 0x565011d213a0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.158+0000 7f1038dfe740 1 -- 172.21.15.12:0/3925205800 >> 172.21.15.12:0/3925205800 conn(0x565011d106b0 msgr2=0x565011d12740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:26.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.158+0000 7f101cc00640 0 client.28061 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:26.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.158+0000 7f1038dfe740 1 -- 172.21.15.12:0/3925205800 shutdown_connections 2024-03-23T23:11:26.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.159+0000 7f1038dfe740 1 -- 172.21.15.12:0/3925205800 wait complete. 2024-03-23T23:11:26.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.159+0000 7f1038dfe740 20 client.28061 trim_cache size 0 max 16384 2024-03-23T23:11:26.161 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:11:26.161 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-03-23T23:11:26.161 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "c30b556f-b8bf-4f8c-88e3-03e8edd63702", 2024-03-23T23:11:26.161 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-03-23T23:11:26.161 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:11:26.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.159+0000 7f1034a00640 1 -- 172.21.15.12:0/678301102 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f10140a2800 msgr2=0x7f10140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:26.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.159+0000 7f1034a00640 1 --2- 172.21.15.12:0/678301102 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f10140a2800 0x7f10140a4cc0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x565011d12500 tx=0x7f1028005f90 comp rx=0 tx=0).stop 2024-03-23T23:11:26.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.159+0000 7f1034a00640 1 -- 172.21.15.12:0/678301102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1030104930 msgr2=0x7f1030197dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:26.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.159+0000 7f1034a00640 1 --2- 172.21.15.12:0/678301102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1030104930 0x7f1030197dd0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f101800e9c0 tx=0x7f101800ee80 comp rx=0 tx=0).stop 2024-03-23T23:11:26.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.159+0000 7f1034a00640 1 -- 172.21.15.12:0/678301102 shutdown_connections 2024-03-23T23:11:26.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.159+0000 7f1034a00640 1 --2- 172.21.15.12:0/678301102 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f10140a2800 0x7f10140a4cc0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.159+0000 7f1034a00640 1 --2- 172.21.15.12:0/678301102 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1030107260 0x7f103019ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.159+0000 7f1034a00640 1 --2- 172.21.15.12:0/678301102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1030104930 0x7f1030197dd0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.159+0000 7f1034a00640 1 --2- 172.21.15.12:0/678301102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1030102000 0x7f1030197890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.159+0000 7f1034a00640 1 -- 172.21.15.12:0/678301102 >> 172.21.15.12:0/678301102 conn(0x7f10300fba10 msgr2=0x7f1030102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:26.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.160+0000 7f1034a00640 1 -- 172.21.15.12:0/678301102 shutdown_connections 2024-03-23T23:11:26.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.160+0000 7f1034a00640 1 -- 172.21.15.12:0/678301102 wait complete. 2024-03-23T23:11:26.176 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:11:26.176 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:11:26.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.449+0000 7f8863800640 1 -- 172.21.15.12:0/2511653005 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f885c105470 msgr2=0x7f885c10bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:26.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.449+0000 7f8863800640 1 --2- 172.21.15.12:0/2511653005 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f885c105470 0x7f885c10bd00 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f8854009860 tx=0x7f885402bb70 comp rx=0 tx=0).stop 2024-03-23T23:11:26.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.449+0000 7f8863800640 1 -- 172.21.15.12:0/2511653005 shutdown_connections 2024-03-23T23:11:26.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.449+0000 7f8863800640 1 --2- 172.21.15.12:0/2511653005 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f885c105470 0x7f885c10bd00 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.449+0000 7f8863800640 1 --2- 172.21.15.12:0/2511653005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f885c104ab0 0x7f885c104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.449+0000 7f8863800640 1 --2- 172.21.15.12:0/2511653005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f885c1038c0 0x7f885c103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.449+0000 7f8863800640 1 -- 172.21.15.12:0/2511653005 >> 172.21.15.12:0/2511653005 conn(0x7f885c0fee50 msgr2=0x7f885c101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:26.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.449+0000 7f8863800640 1 -- 172.21.15.12:0/2511653005 shutdown_connections 2024-03-23T23:11:26.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.449+0000 7f8863800640 1 -- 172.21.15.12:0/2511653005 wait complete. 2024-03-23T23:11:26.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.450+0000 7f8863800640 1 Processor -- start 2024-03-23T23:11:26.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.451+0000 7f8863800640 1 -- start start 2024-03-23T23:11:26.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.451+0000 7f8863800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f885c1038c0 0x7f885c199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.451+0000 7f8863800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f885c104ab0 0x7f885c19a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.451+0000 7f8863800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f885c105470 0x7f885c1a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.451+0000 7f8863800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885c10f520 con 0x7f885c104ab0 2024-03-23T23:11:26.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.451+0000 7f8863800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885c10f3a0 con 0x7f885c105470 2024-03-23T23:11:26.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.451+0000 7f8863800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885c10f6a0 con 0x7f885c1038c0 2024-03-23T23:11:26.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.451+0000 7f8860e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f885c1038c0 0x7f885c199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.451+0000 7f8860e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f885c1038c0 0x7f885c199b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:56574/0 (socket says 172.21.15.12:56574) 2024-03-23T23:11:26.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.451+0000 7f8860e00640 1 -- 172.21.15.12:0/3380005600 learned_addr learned my addr 172.21.15.12:0/3380005600 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:26.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.451+0000 7f8861800640 1 --2- 172.21.15.12:0/3380005600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f885c105470 0x7f885c1a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.451+0000 7f885be00640 1 --2- 172.21.15.12:0/3380005600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f885c104ab0 0x7f885c19a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.452+0000 7f8860e00640 1 -- 172.21.15.12:0/3380005600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f885c105470 msgr2=0x7f885c1a1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:26.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.452+0000 7f8860e00640 1 --2- 172.21.15.12:0/3380005600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f885c105470 0x7f885c1a1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.452+0000 7f8860e00640 1 -- 172.21.15.12:0/3380005600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f885c104ab0 msgr2=0x7f885c19a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:26.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.452+0000 7f8860e00640 1 --2- 172.21.15.12:0/3380005600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f885c104ab0 0x7f885c19a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.452+0000 7f8860e00640 1 -- 172.21.15.12:0/3380005600 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f885c1a1780 con 0x7f885c1038c0 2024-03-23T23:11:26.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.452+0000 7f885be00640 1 --2- 172.21.15.12:0/3380005600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f885c104ab0 0x7f885c19a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:11:26.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.452+0000 7f8860e00640 1 --2- 172.21.15.12:0/3380005600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f885c1038c0 0x7f885c199b60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f884c00c3f0 tx=0x7f884c009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:26.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.452+0000 7f8859600640 1 -- 172.21.15.12:0/3380005600 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f884c01b070 con 0x7f885c1038c0 2024-03-23T23:11:26.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.452+0000 7f8859600640 1 -- 172.21.15.12:0/3380005600 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f884c00bb10 con 0x7f885c1038c0 2024-03-23T23:11:26.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.452+0000 7f8859600640 1 -- 172.21.15.12:0/3380005600 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f884c016d80 con 0x7f885c1038c0 2024-03-23T23:11:26.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.452+0000 7f8863800640 1 -- 172.21.15.12:0/3380005600 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f885c1a1a70 con 0x7f885c1038c0 2024-03-23T23:11:26.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.452+0000 7f8863800640 1 -- 172.21.15.12:0/3380005600 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f885c1a1f70 con 0x7f885c1038c0 2024-03-23T23:11:26.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.456+0000 7f8859600640 1 -- 172.21.15.12:0/3380005600 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f884c01e070 con 0x7f885c1038c0 2024-03-23T23:11:26.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.456+0000 7f88665d3740 1 Processor -- start 2024-03-23T23:11:26.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.456+0000 7f88665d3740 1 -- start start 2024-03-23T23:11:26.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.457+0000 7f88665d3740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621614bfe80 0x5621614c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.457+0000 7f88665d3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614c0870 0x5621614c5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.457+0000 7f88665d3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621614c59e0 0x5621614c7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.457+0000 7f88665d3740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x562161425250 con 0x5621614bfe80 2024-03-23T23:11:26.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.457+0000 7f88665d3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621613cda30 con 0x5621614c59e0 2024-03-23T23:11:26.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.457+0000 7f88665d3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621614212f0 con 0x5621614c0870 2024-03-23T23:11:26.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.457+0000 7f8860e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614c0870 0x5621614c5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.457+0000 7f8861800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621614bfe80 0x5621614c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.457+0000 7f885be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621614c59e0 0x5621614c7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.457+0000 7f8860e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614c0870 0x5621614c5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:56580/0 (socket says 172.21.15.12:56580) 2024-03-23T23:11:26.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.457+0000 7f8861800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621614bfe80 0x5621614c02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45202/0 (socket says 172.21.15.12:45202) 2024-03-23T23:11:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.457+0000 7f8860e00640 1 -- 172.21.15.12:0/3268702629 learned_addr learned my addr 172.21.15.12:0/3268702629 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.457+0000 7f8860e00640 1 -- 172.21.15.12:0/3268702629 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621614c59e0 msgr2=0x5621614c7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.457+0000 7f8860e00640 1 --2- 172.21.15.12:0/3268702629 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621614c59e0 0x5621614c7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.457+0000 7f8860e00640 1 -- 172.21.15.12:0/3268702629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621614bfe80 msgr2=0x5621614c02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.457+0000 7f8860e00640 1 --2- 172.21.15.12:0/3268702629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621614bfe80 0x5621614c02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.458+0000 7f8860e00640 1 -- 172.21.15.12:0/3268702629 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5621613b8690 con 0x5621614c0870 2024-03-23T23:11:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.458+0000 7f8861800640 1 --2- 172.21.15.12:0/3268702629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621614bfe80 0x5621614c02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.458+0000 7f8859600640 1 --2- 172.21.15.12:0/3380005600 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f88400a2880 0x7f88400a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.458+0000 7f8860e00640 1 --2- 172.21.15.12:0/3268702629 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614c0870 0x5621614c5370 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f884c0d4c20 tx=0x7f884c0d6670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=760a2d135403c376 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.458+0000 7f8861800640 1 --2- 172.21.15.12:0/3380005600 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f88400a2880 0x7f88400a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.458+0000 7f8859600640 1 -- 172.21.15.12:0/3380005600 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f884c0ce9a0 con 0x7f885c1038c0 2024-03-23T23:11:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.458+0000 7f8852000640 1 -- 172.21.15.12:0/3268702629 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f884c095e60 con 0x5621614c0870 2024-03-23T23:11:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.458+0000 7f8852000640 1 -- 172.21.15.12:0/3268702629 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f884c09b9f0 con 0x5621614c0870 2024-03-23T23:11:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.458+0000 7f8852000640 1 -- 172.21.15.12:0/3268702629 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f884c09ac50 con 0x5621614c0870 2024-03-23T23:11:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.459+0000 7f88665d3740 1 -- 172.21.15.12:0/3268702629 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614c0870 msgr2=0x5621614c5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.459+0000 7f8861800640 1 --2- 172.21.15.12:0/3380005600 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f88400a2880 0x7f88400a4d40 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f885402c180 tx=0x7f88540056d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.459+0000 7f88665d3740 1 --2- 172.21.15.12:0/3268702629 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614c0870 0x5621614c5370 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f884c0d4c20 tx=0x7f884c0d6670 comp rx=0 tx=0).stop 2024-03-23T23:11:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.459+0000 7f88665d3740 1 -- 172.21.15.12:0/3268702629 shutdown_connections 2024-03-23T23:11:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.459+0000 7f88665d3740 1 --2- 172.21.15.12:0/3268702629 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621614c59e0 0x5621614c7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.459+0000 7f88665d3740 1 --2- 172.21.15.12:0/3268702629 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614c0870 0x5621614c5370 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.459+0000 7f88665d3740 1 --2- 172.21.15.12:0/3268702629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621614bfe80 0x5621614c02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.459+0000 7f88665d3740 1 -- 172.21.15.12:0/3268702629 >> 172.21.15.12:0/3268702629 conn(0x562161488200 msgr2=0x5621614bb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:26.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.459+0000 7f88665d3740 1 -- 172.21.15.12:0/3268702629 shutdown_connections 2024-03-23T23:11:26.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.459+0000 7f88665d3740 1 -- 172.21.15.12:0/3268702629 wait complete. 2024-03-23T23:11:26.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.460+0000 7f88665d3740 1 -- start start 2024-03-23T23:11:26.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.460+0000 7f88665d3740 1 Processor -- start 2024-03-23T23:11:26.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.460+0000 7f88665d3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614bfe80 0x5621614c9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.460+0000 7f88665d3740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621614c0870 0x5621614ca390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.460+0000 7f88665d3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621614c59e0 0x5621614d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.460+0000 7f88665d3740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621614c8990 con 0x5621614c0870 2024-03-23T23:11:26.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.460+0000 7f88665d3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621613e0e80 con 0x5621614c59e0 2024-03-23T23:11:26.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.460+0000 7f88665d3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621614c8b10 con 0x5621614bfe80 2024-03-23T23:11:26.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.460+0000 7f88665d3740 10 client.0 _ll_register_callbacks cb 0x5621613f3130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:26.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.461+0000 7f8861800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621614c59e0 0x5621614d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.461+0000 7f8860e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614bfe80 0x5621614c9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.461+0000 7f8861800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621614c59e0 0x5621614d13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:41786/0 (socket says 172.21.15.12:41786) 2024-03-23T23:11:26.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.461+0000 7f8861800640 1 -- 172.21.15.12:0/1742075947 learned_addr learned my addr 172.21.15.12:0/1742075947 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:26.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.461+0000 7f8860e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614bfe80 0x5621614c9e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:56586/0 (socket says 172.21.15.12:56586) 2024-03-23T23:11:26.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.461+0000 7f885be00640 1 --2- 172.21.15.12:0/1742075947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621614c0870 0x5621614ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.461+0000 7f8861800640 1 -- 172.21.15.12:0/1742075947 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614bfe80 msgr2=0x5621614c9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:26.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.461+0000 7f8861800640 1 --2- 172.21.15.12:0/1742075947 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614bfe80 0x5621614c9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.461+0000 7f8861800640 1 -- 172.21.15.12:0/1742075947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621614c0870 msgr2=0x5621614ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:26.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.461+0000 7f8861800640 1 --2- 172.21.15.12:0/1742075947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621614c0870 0x5621614ca390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.461+0000 7f8861800640 1 -- 172.21.15.12:0/1742075947 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5621612037d0 con 0x5621614c59e0 2024-03-23T23:11:26.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.461+0000 7f885be00640 1 --2- 172.21.15.12:0/1742075947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621614c0870 0x5621614ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:11:26.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.461+0000 7f8860e00640 1 --2- 172.21.15.12:0/1742075947 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614bfe80 0x5621614c9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:26.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.461+0000 7f8861800640 1 --2- 172.21.15.12:0/1742075947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621614c59e0 0x5621614d13d0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f8854032570 tx=0x7f885402bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7687e9f0500172d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:26.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.461+0000 7f884aa00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:11:26.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.462+0000 7f884aa00640 1 -- 172.21.15.12:0/1742075947 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8854039040 con 0x5621614c59e0 2024-03-23T23:11:26.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.462+0000 7f88665d3740 1 -- 172.21.15.12:0/1742075947 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f884c09b9f0 con 0x5621614c59e0 2024-03-23T23:11:26.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.462+0000 7f884aa00640 1 -- 172.21.15.12:0/1742075947 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f885402c9e0 con 0x5621614c59e0 2024-03-23T23:11:26.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.462+0000 7f884aa00640 1 -- 172.21.15.12:0/1742075947 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f885402cbe0 con 0x5621614c59e0 2024-03-23T23:11:26.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.462+0000 7f88665d3740 10 client.28073 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:26.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.462+0000 7f88665d3740 1 -- 172.21.15.12:0/1742075947 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5621611393f0 con 0x5621614c59e0 2024-03-23T23:11:26.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.462+0000 7f884aa00640 1 -- 172.21.15.12:0/1742075947 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f884c09b9f0 con 0x5621614c59e0 2024-03-23T23:11:26.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.462+0000 7f884aa00640 1 -- 172.21.15.12:0/1742075947 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8854052030 con 0x5621614c59e0 2024-03-23T23:11:26.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.462+0000 7f88665d3740 10 client.28073 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:26.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.462+0000 7f88665d3740 10 client.28073 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:11:26.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.462+0000 7f88665d3740 20 client.28073 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:26.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.463+0000 7f88665d3740 1 --2- 172.21.15.12:0/1742075947 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5621614d5cc0 0x5621614d8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.463+0000 7f88665d3740 4 client.28073 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:11:26.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.463+0000 7f88665d3740 1 -- 172.21.15.12:0/1742075947 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562161432790 con 0x5621614d5cc0 2024-03-23T23:11:26.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.463+0000 7f8860e00640 1 --2- 172.21.15.12:0/1742075947 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5621614d5cc0 0x5621614d8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.463+0000 7f8860e00640 1 --2- 172.21.15.12:0/1742075947 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5621614d5cc0 0x5621614d8120 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2b481418655ac86 server_cookie=ff9e445ce0bf3238 in_seq=0 out_seq=0 2024-03-23T23:11:26.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.463+0000 7f884aa00640 10 client.28073 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:26.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.464+0000 7f884aa00640 1 -- 172.21.15.12:0/1742075947 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562161432790 con 0x5621614d5cc0 2024-03-23T23:11:26.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.464+0000 7f884aa00640 10 client.28073 handle_command_reply: tid=0 2024-03-23T23:11:26.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.465+0000 7f88665d3740 1 client.28073 shutdown 2024-03-23T23:11:26.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.465+0000 7f88665d3740 1 -- 172.21.15.12:0/1742075947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621614c59e0 msgr2=0x5621614d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:26.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.465+0000 7f88665d3740 1 --2- 172.21.15.12:0/1742075947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621614c59e0 0x5621614d13d0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f8854032570 tx=0x7f885402bb70 comp rx=0 tx=0).stop 2024-03-23T23:11:26.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.465+0000 7f88665d3740 1 -- 172.21.15.12:0/1742075947 shutdown_connections 2024-03-23T23:11:26.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.465+0000 7f88665d3740 1 --2- 172.21.15.12:0/1742075947 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5621614d5cc0 0x5621614d8120 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.465+0000 7f88665d3740 1 --2- 172.21.15.12:0/1742075947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621614c59e0 0x5621614d13d0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.465+0000 7f88665d3740 1 --2- 172.21.15.12:0/1742075947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621614c0870 0x5621614ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.465+0000 7f88665d3740 1 --2- 172.21.15.12:0/1742075947 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614bfe80 0x5621614c9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.465+0000 7f884aa00640 0 client.28073 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:26.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.465+0000 7f88665d3740 1 -- 172.21.15.12:0/1742075947 >> 172.21.15.12:0/1742075947 conn(0x562161488200 msgr2=0x5621614baa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:26.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.465+0000 7f88665d3740 1 -- 172.21.15.12:0/1742075947 shutdown_connections 2024-03-23T23:11:26.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.465+0000 7f88665d3740 1 -- 172.21.15.12:0/1742075947 wait complete. 2024-03-23T23:11:26.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.465+0000 7f88665d3740 20 client.28073 trim_cache size 0 max 16384 2024-03-23T23:11:26.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.484+0000 7f88665d3740 1 Processor -- start 2024-03-23T23:11:26.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.484+0000 7f88665d3740 1 -- start start 2024-03-23T23:11:26.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.484+0000 7f88665d3740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621614bfe80 0x5621614ba200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.484+0000 7f88665d3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621614c0870 0x5621614d1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.484+0000 7f88665d3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614c59e0 0x5621614ba740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.484+0000 7f88665d3740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885c1ad190 con 0x5621614bfe80 2024-03-23T23:11:26.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.484+0000 7f88665d3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885c1aac90 con 0x5621614c0870 2024-03-23T23:11:26.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.484+0000 7f88665d3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885c1a8d30 con 0x5621614c59e0 2024-03-23T23:11:26.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.484+0000 7f8861800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614c59e0 0x5621614ba740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.484+0000 7f8861800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614c59e0 0x5621614ba740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:56602/0 (socket says 172.21.15.12:56602) 2024-03-23T23:11:26.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.484+0000 7f8861800640 1 -- 172.21.15.12:0/1524588803 learned_addr learned my addr 172.21.15.12:0/1524588803 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:26.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.485+0000 7f885be00640 1 --2- 172.21.15.12:0/1524588803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621614c0870 0x5621614d1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.485+0000 7f8861800640 1 -- 172.21.15.12:0/1524588803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621614c0870 msgr2=0x5621614d1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:26.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.485+0000 7f8861800640 1 --2- 172.21.15.12:0/1524588803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621614c0870 0x5621614d1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.485+0000 7f8860e00640 1 --2- 172.21.15.12:0/1524588803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621614bfe80 0x5621614ba200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.485+0000 7f8861800640 1 -- 172.21.15.12:0/1524588803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621614bfe80 msgr2=0x5621614ba200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:26.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.485+0000 7f8861800640 1 --2- 172.21.15.12:0/1524588803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621614bfe80 0x5621614ba200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.485+0000 7f8861800640 1 -- 172.21.15.12:0/1524588803 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5621610f90f0 con 0x5621614c59e0 2024-03-23T23:11:26.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.485+0000 7f8860e00640 1 --2- 172.21.15.12:0/1524588803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621614bfe80 0x5621614ba200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:11:26.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.485+0000 7f8861800640 1 --2- 172.21.15.12:0/1524588803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614c59e0 0x5621614ba740 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8854005c40 tx=0x7f8854005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c10dc7f6db524744 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:26.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.486+0000 7f8851600640 1 -- 172.21.15.12:0/1524588803 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8854039070 con 0x5621614c59e0 2024-03-23T23:11:26.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.486+0000 7f8851600640 1 -- 172.21.15.12:0/1524588803 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88540037f0 con 0x5621614c59e0 2024-03-23T23:11:26.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.486+0000 7f8851600640 1 -- 172.21.15.12:0/1524588803 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8854032d80 con 0x5621614c59e0 2024-03-23T23:11:26.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.486+0000 7f88665d3740 1 -- 172.21.15.12:0/1524588803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614c59e0 msgr2=0x5621614ba740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:26.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.486+0000 7f88665d3740 1 --2- 172.21.15.12:0/1524588803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614c59e0 0x5621614ba740 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8854005c40 tx=0x7f8854005bd0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.486+0000 7f88665d3740 1 -- 172.21.15.12:0/1524588803 shutdown_connections 2024-03-23T23:11:26.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.486+0000 7f88665d3740 1 --2- 172.21.15.12:0/1524588803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614c59e0 0x5621614ba740 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.486+0000 7f88665d3740 1 --2- 172.21.15.12:0/1524588803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621614c0870 0x5621614d1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.486+0000 7f88665d3740 1 --2- 172.21.15.12:0/1524588803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621614bfe80 0x5621614ba200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.486+0000 7f88665d3740 1 -- 172.21.15.12:0/1524588803 >> 172.21.15.12:0/1524588803 conn(0x562161488200 msgr2=0x5621614c9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:26.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.487+0000 7f88665d3740 1 -- 172.21.15.12:0/1524588803 shutdown_connections 2024-03-23T23:11:26.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.487+0000 7f88665d3740 1 -- 172.21.15.12:0/1524588803 wait complete. 2024-03-23T23:11:26.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.487+0000 7f88665d3740 1 -- start start 2024-03-23T23:11:26.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.487+0000 7f88665d3740 1 Processor -- start 2024-03-23T23:11:26.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.488+0000 7f88665d3740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621614bfe80 0x5621614ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.488+0000 7f88665d3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614c0870 0x5621614ceb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.488+0000 7f88665d3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621614c59e0 0x5621614cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.488+0000 7f88665d3740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885c1a9330 con 0x5621614bfe80 2024-03-23T23:11:26.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.488+0000 7f88665d3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885c1aa630 con 0x5621614c59e0 2024-03-23T23:11:26.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.488+0000 7f88665d3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885c1a9970 con 0x5621614c0870 2024-03-23T23:11:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.488+0000 7f88665d3740 10 client.0 _ll_register_callbacks cb 0x7f884c009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.488+0000 7f8860e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621614bfe80 0x5621614ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.488+0000 7f885be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614c0870 0x5621614ceb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.488+0000 7f885be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614c0870 0x5621614ceb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:56616/0 (socket says 172.21.15.12:56616) 2024-03-23T23:11:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.488+0000 7f8860e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621614bfe80 0x5621614ce5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45240/0 (socket says 172.21.15.12:45240) 2024-03-23T23:11:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.488+0000 7f885be00640 1 -- 172.21.15.12:0/3090098141 learned_addr learned my addr 172.21.15.12:0/3090098141 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.488+0000 7f8861800640 1 --2- 172.21.15.12:0/3090098141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621614c59e0 0x5621614cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.488+0000 7f885be00640 1 -- 172.21.15.12:0/3090098141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621614c59e0 msgr2=0x5621614cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.488+0000 7f885be00640 1 --2- 172.21.15.12:0/3090098141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621614c59e0 0x5621614cfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.488+0000 7f885be00640 1 -- 172.21.15.12:0/3090098141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621614bfe80 msgr2=0x5621614ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.488+0000 7f885be00640 1 --2- 172.21.15.12:0/3090098141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621614bfe80 0x5621614ce5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.488+0000 7f885be00640 1 -- 172.21.15.12:0/3090098141 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5621614d0120 con 0x5621614c0870 2024-03-23T23:11:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.489+0000 7f885be00640 1 --2- 172.21.15.12:0/3090098141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614c0870 0x5621614ceb20 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f885c19b070 tx=0x7f88440039c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81ae102e8e9ac981 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.489+0000 7f884aa00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:11:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.489+0000 7f884aa00640 1 -- 172.21.15.12:0/3090098141 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8844012860 con 0x5621614c0870 2024-03-23T23:11:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.489+0000 7f884aa00640 1 -- 172.21.15.12:0/3090098141 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8844012a00 con 0x5621614c0870 2024-03-23T23:11:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.489+0000 7f88665d3740 1 -- 172.21.15.12:0/3090098141 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f88540037f0 con 0x5621614c0870 2024-03-23T23:11:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.489+0000 7f884aa00640 1 -- 172.21.15.12:0/3090098141 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8844012c00 con 0x5621614c0870 2024-03-23T23:11:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.489+0000 7f88665d3740 10 client.28045 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.489+0000 7f88665d3740 1 -- 172.21.15.12:0/3090098141 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5621614d14a0 con 0x5621614c0870 2024-03-23T23:11:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.489+0000 7f884aa00640 1 -- 172.21.15.12:0/3090098141 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88540037f0 con 0x5621614c0870 2024-03-23T23:11:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.490+0000 7f884aa00640 1 -- 172.21.15.12:0/3090098141 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f884401c070 con 0x5621614c0870 2024-03-23T23:11:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.490+0000 7f88665d3740 10 client.28045 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.490+0000 7f88665d3740 10 client.28045 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:11:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.490+0000 7f88665d3740 20 client.28045 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.490+0000 7f88665d3740 1 --2- 172.21.15.12:0/3090098141 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562161531bd0 0x562161533ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:26.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.490+0000 7f88665d3740 4 client.28045 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:11:26.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.490+0000 7f88665d3740 1 -- 172.21.15.12:0/3090098141 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56216142cae0 con 0x562161531bd0 2024-03-23T23:11:26.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.490+0000 7f8860e00640 1 --2- 172.21.15.12:0/3090098141 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562161531bd0 0x562161533ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:26.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.490+0000 7f8860e00640 1 --2- 172.21.15.12:0/3090098141 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562161531bd0 0x562161533ff0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93e9f107f7f92794 server_cookie=686b5abf1e79ab90 in_seq=0 out_seq=0 2024-03-23T23:11:26.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.490+0000 7f884aa00640 10 client.28045 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:26.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.491+0000 7f884aa00640 1 -- 172.21.15.12:0/3090098141 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56216142cae0 con 0x562161531bd0 2024-03-23T23:11:26.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.491+0000 7f884aa00640 10 client.28045 handle_command_reply: tid=0 2024-03-23T23:11:26.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.491+0000 7f88665d3740 1 client.28045 shutdown 2024-03-23T23:11:26.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.491+0000 7f88665d3740 1 -- 172.21.15.12:0/3090098141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614c0870 msgr2=0x5621614ceb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:26.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.491+0000 7f88665d3740 1 --2- 172.21.15.12:0/3090098141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614c0870 0x5621614ceb20 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f885c19b070 tx=0x7f88440039c0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.491+0000 7f88665d3740 1 -- 172.21.15.12:0/3090098141 shutdown_connections 2024-03-23T23:11:26.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.491+0000 7f88665d3740 1 --2- 172.21.15.12:0/3090098141 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562161531bd0 0x562161533ff0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.491+0000 7f88665d3740 1 --2- 172.21.15.12:0/3090098141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621614c59e0 0x5621614cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.491+0000 7f88665d3740 1 --2- 172.21.15.12:0/3090098141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621614c0870 0x5621614ceb20 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.491+0000 7f884aa00640 0 client.28045 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:26.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.491+0000 7f88665d3740 1 --2- 172.21.15.12:0/3090098141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621614bfe80 0x5621614ce5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.491+0000 7f88665d3740 1 -- 172.21.15.12:0/3090098141 >> 172.21.15.12:0/3090098141 conn(0x562161488200 msgr2=0x5621614ba200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:26.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.491+0000 7f88665d3740 1 -- 172.21.15.12:0/3090098141 shutdown_connections 2024-03-23T23:11:26.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.491+0000 7f88665d3740 1 -- 172.21.15.12:0/3090098141 wait complete. 2024-03-23T23:11:26.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.491+0000 7f88665d3740 20 client.28045 trim_cache size 0 max 16384 2024-03-23T23:11:26.496 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:11:26.497 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T23:11:26.497 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-03-23T23:11:26.497 INFO:teuthology.orchestra.run.smithi012.stdout: "c30b556f-b8bf-4f8c-88e3-03e8edd63702": { 2024-03-23T23:11:26.497 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-03-23T23:11:26.497 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "c30b556f-b8bf-4f8c-88e3-03e8edd63702", 2024-03-23T23:11:26.497 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-03-23T23:11:26.497 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:11:26.497 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:11:26.497 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:11:26.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.492+0000 7f8863800640 1 -- 172.21.15.12:0/3380005600 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f88400a2880 msgr2=0x7f88400a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:26.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.492+0000 7f8863800640 1 --2- 172.21.15.12:0/3380005600 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f88400a2880 0x7f88400a4d40 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f885402c180 tx=0x7f88540056d0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.492+0000 7f8863800640 1 -- 172.21.15.12:0/3380005600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f885c1038c0 msgr2=0x7f885c199b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:26.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.492+0000 7f8863800640 1 --2- 172.21.15.12:0/3380005600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f885c1038c0 0x7f885c199b60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f884c00c3f0 tx=0x7f884c009f90 comp rx=0 tx=0).stop 2024-03-23T23:11:26.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.492+0000 7f8863800640 1 -- 172.21.15.12:0/3380005600 shutdown_connections 2024-03-23T23:11:26.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.492+0000 7f8863800640 1 --2- 172.21.15.12:0/3380005600 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f88400a2880 0x7f88400a4d40 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.492+0000 7f8863800640 1 --2- 172.21.15.12:0/3380005600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f885c105470 0x7f885c1a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.492+0000 7f8863800640 1 --2- 172.21.15.12:0/3380005600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f885c104ab0 0x7f885c19a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.492+0000 7f8863800640 1 --2- 172.21.15.12:0/3380005600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f885c1038c0 0x7f885c199b60 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:26.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.492+0000 7f8863800640 1 -- 172.21.15.12:0/3380005600 >> 172.21.15.12:0/3380005600 conn(0x7f885c0fee50 msgr2=0x7f885c107d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:26.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.492+0000 7f8863800640 1 -- 172.21.15.12:0/3380005600 shutdown_connections 2024-03-23T23:11:26.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:26.492+0000 7f8863800640 1 -- 172.21.15.12:0/3380005600 wait complete. 2024-03-23T23:11:26.508 INFO:tasks.cephfs.filesystem:scrub status for tag:c30b556f-b8bf-4f8c-88e3-03e8edd63702 - {'path': '/', 'tag': 'c30b556f-b8bf-4f8c-88e3-03e8edd63702', 'options': 'recursive,force'} 2024-03-23T23:11:27.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:26 smithi012 ceph-mon[31695]: pgmap v679: 97 pgs: 97 active+clean; 2.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:11:27.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:26 smithi012 ceph-mon[31695]: scrub queued for path: / 2024-03-23T23:11:27.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:26 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:11:27.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:26 smithi012 ceph-mon[31695]: scrub summary: active paths [/] 2024-03-23T23:11:27.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:26 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:11:27.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:26 smithi173 ceph-mon[36116]: pgmap v679: 97 pgs: 97 active+clean; 2.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:11:27.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:26 smithi173 ceph-mon[36116]: scrub queued for path: / 2024-03-23T23:11:27.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:26 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:11:27.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:26 smithi173 ceph-mon[36116]: scrub summary: active paths [/] 2024-03-23T23:11:27.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:26 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:11:27.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:26 smithi203 ceph-mon[36693]: pgmap v679: 97 pgs: 97 active+clean; 2.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:11:27.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:26 smithi203 ceph-mon[36693]: scrub queued for path: / 2024-03-23T23:11:27.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:26 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:11:27.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:26 smithi203 ceph-mon[36693]: scrub summary: active paths [/] 2024-03-23T23:11:27.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:26 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:11:29.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:28 smithi012 ceph-mon[31695]: pgmap v680: 97 pgs: 97 active+clean; 2.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:11:29.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:28 smithi173 ceph-mon[36116]: pgmap v680: 97 pgs: 97 active+clean; 2.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:11:29.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:28 smithi203 ceph-mon[36693]: pgmap v680: 97 pgs: 97 active+clean; 2.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:11:31.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:30 smithi012 ceph-mon[31695]: pgmap v681: 97 pgs: 97 active+clean; 2.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:11:31.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:30 smithi012 ceph-mon[31695]: scrub summary: idle 2024-03-23T23:11:31.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:30 smithi173 ceph-mon[36116]: pgmap v681: 97 pgs: 97 active+clean; 2.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:11:31.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:30 smithi173 ceph-mon[36116]: scrub summary: idle 2024-03-23T23:11:31.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:30 smithi203 ceph-mon[36693]: pgmap v681: 97 pgs: 97 active+clean; 2.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:11:31.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:30 smithi203 ceph-mon[36693]: scrub summary: idle 2024-03-23T23:11:33.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:32 smithi012 ceph-mon[31695]: pgmap v682: 97 pgs: 97 active+clean; 2.5 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:11:33.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:32 smithi173 ceph-mon[36116]: pgmap v682: 97 pgs: 97 active+clean; 2.5 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:11:33.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:32 smithi203 ceph-mon[36693]: pgmap v682: 97 pgs: 97 active+clean; 2.5 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:11:35.209 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:34 smithi203 ceph-mon[36693]: pgmap v683: 97 pgs: 97 active+clean; 2.5 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:11:35.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:34 smithi012 ceph-mon[31695]: pgmap v683: 97 pgs: 97 active+clean; 2.5 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:11:35.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:34 smithi173 ceph-mon[36116]: pgmap v683: 97 pgs: 97 active+clean; 2.5 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:11:37.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:36 smithi012 ceph-mon[31695]: pgmap v684: 97 pgs: 97 active+clean; 2.6 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:11:37.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:36 smithi173 ceph-mon[36116]: pgmap v684: 97 pgs: 97 active+clean; 2.6 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:11:37.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:36 smithi203 ceph-mon[36693]: pgmap v684: 97 pgs: 97 active+clean; 2.6 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:11:39.124 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.125+0000 7f106ea00640 1 -- 172.21.15.12:0/2264132669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1068103720 msgr2=0x7f1068103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:39.124 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.125+0000 7f106ea00640 1 --2- 172.21.15.12:0/2264132669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1068103720 0x7f1068103b40 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f105c009900 tx=0x7f105c02bc30 comp rx=0 tx=0).stop 2024-03-23T23:11:39.124 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.125+0000 7f106ea00640 1 -- 172.21.15.12:0/2264132669 shutdown_connections 2024-03-23T23:11:39.125 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.125+0000 7f106ea00640 1 --2- 172.21.15.12:0/2264132669 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10681052d0 0x7f106810bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.125 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.125+0000 7f106ea00640 1 --2- 172.21.15.12:0/2264132669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1068104910 0x7f1068104d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.125 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.125+0000 7f106ea00640 1 --2- 172.21.15.12:0/2264132669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1068103720 0x7f1068103b40 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.125 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.125+0000 7f106ea00640 1 -- 172.21.15.12:0/2264132669 >> 172.21.15.12:0/2264132669 conn(0x7f10680fecb0 msgr2=0x7f10681010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:39.125 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.125+0000 7f106ea00640 1 -- 172.21.15.12:0/2264132669 shutdown_connections 2024-03-23T23:11:39.125 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.125+0000 7f106ea00640 1 -- 172.21.15.12:0/2264132669 wait complete. 2024-03-23T23:11:39.127 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.126+0000 7f106ea00640 1 Processor -- start 2024-03-23T23:11:39.127 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.126+0000 7f106ea00640 1 -- start start 2024-03-23T23:11:39.127 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.127+0000 7f106ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1068103720 0x7f1068199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:39.127 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.127+0000 7f106ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1068104910 0x7f1068199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:39.127 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.127+0000 7f106ea00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10681052d0 0x7f10681a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:39.127 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.127+0000 7f106ea00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106810f3e0 con 0x7f10681052d0 2024-03-23T23:11:39.127 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.127+0000 7f106ea00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106810f260 con 0x7f1068103720 2024-03-23T23:11:39.127 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.127+0000 7f106ea00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106810f560 con 0x7f1068104910 2024-03-23T23:11:39.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.127+0000 7f106e000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10681052d0 0x7f10681a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:39.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.127+0000 7f106cc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1068104910 0x7f1068199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:39.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.127+0000 7f106d600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1068103720 0x7f1068199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:39.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.127+0000 7f106e000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10681052d0 0x7f10681a1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51170/0 (socket says 172.21.15.12:51170) 2024-03-23T23:11:39.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.127+0000 7f106cc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1068104910 0x7f1068199f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49524/0 (socket says 172.21.15.12:49524) 2024-03-23T23:11:39.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.127+0000 7f106d600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1068103720 0x7f1068199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:46974/0 (socket says 172.21.15.12:46974) 2024-03-23T23:11:39.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.127+0000 7f106cc00640 1 -- 172.21.15.12:0/2508104 learned_addr learned my addr 172.21.15.12:0/2508104 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:39.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.128+0000 7f106d600640 1 -- 172.21.15.12:0/2508104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1068104910 msgr2=0x7f1068199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:39.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.128+0000 7f106d600640 1 --2- 172.21.15.12:0/2508104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1068104910 0x7f1068199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.128+0000 7f106d600640 1 -- 172.21.15.12:0/2508104 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10681052d0 msgr2=0x7f10681a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:39.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.128+0000 7f106d600640 1 --2- 172.21.15.12:0/2508104 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10681052d0 0x7f10681a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.128+0000 7f106d600640 1 -- 172.21.15.12:0/2508104 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10681a1670 con 0x7f1068103720 2024-03-23T23:11:39.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.128+0000 7f106cc00640 1 --2- 172.21.15.12:0/2508104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1068104910 0x7f1068199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:39.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.128+0000 7f106d600640 1 --2- 172.21.15.12:0/2508104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1068103720 0x7f1068199a50 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f105c009230 tx=0x7f105c02c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:39.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.128+0000 7f1066000640 1 -- 172.21.15.12:0/2508104 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c043020 con 0x7f1068103720 2024-03-23T23:11:39.129 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.128+0000 7f106ea00640 1 -- 172.21.15.12:0/2508104 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10681a1900 con 0x7f1068103720 2024-03-23T23:11:39.129 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.128+0000 7f1066000640 1 -- 172.21.15.12:0/2508104 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f105c0054e0 con 0x7f1068103720 2024-03-23T23:11:39.129 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.128+0000 7f1066000640 1 -- 172.21.15.12:0/2508104 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c003c60 con 0x7f1068103720 2024-03-23T23:11:39.129 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.128+0000 7f106ea00640 1 -- 172.21.15.12:0/2508104 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10681a1e70 con 0x7f1068103720 2024-03-23T23:11:39.131 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.131+0000 7f1072dd6740 1 Processor -- start 2024-03-23T23:11:39.131 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.132+0000 7f1066000640 1 -- 172.21.15.12:0/2508104 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f105c046020 con 0x7f1068103720 2024-03-23T23:11:39.131 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.132+0000 7f1072dd6740 1 -- start start 2024-03-23T23:11:39.131 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.132+0000 7f1072dd6740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd6d40f3f0 0x55fd6d40f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:39.131 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.132+0000 7f1072dd6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd6d40fde0 0x55fd6d4148e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:39.131 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.132+0000 7f1072dd6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d414f50 0x55fd6d4173e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:39.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.132+0000 7f1072dd6740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd6d3d23b0 con 0x55fd6d40f3f0 2024-03-23T23:11:39.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.132+0000 7f1072dd6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd6d374830 con 0x55fd6d414f50 2024-03-23T23:11:39.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.132+0000 7f1072dd6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd6d31d5d0 con 0x55fd6d40fde0 2024-03-23T23:11:39.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.132+0000 7f106d600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd6d40fde0 0x55fd6d4148e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:39.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.132+0000 7f106cc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d414f50 0x55fd6d4173e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:39.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.132+0000 7f106e000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd6d40f3f0 0x55fd6d40f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:39.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.132+0000 7f106d600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd6d40fde0 0x55fd6d4148e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49534/0 (socket says 172.21.15.12:49534) 2024-03-23T23:11:39.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.132+0000 7f106d600640 1 -- 172.21.15.12:0/3217827838 learned_addr learned my addr 172.21.15.12:0/3217827838 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:39.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.132+0000 7f106cc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d414f50 0x55fd6d4173e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:46990/0 (socket says 172.21.15.12:46990) 2024-03-23T23:11:39.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.133+0000 7f106d600640 1 -- 172.21.15.12:0/3217827838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d414f50 msgr2=0x55fd6d4173e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:39.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.133+0000 7f106d600640 1 --2- 172.21.15.12:0/3217827838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d414f50 0x55fd6d4173e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.133+0000 7f106d600640 1 -- 172.21.15.12:0/3217827838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd6d40f3f0 msgr2=0x55fd6d40f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:39.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.133+0000 7f106d600640 1 --2- 172.21.15.12:0/3217827838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd6d40f3f0 0x55fd6d40f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.133+0000 7f106d600640 1 -- 172.21.15.12:0/3217827838 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fd6d3352c0 con 0x55fd6d40fde0 2024-03-23T23:11:39.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.133+0000 7f106d600640 1 --2- 172.21.15.12:0/3217827838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd6d40fde0 0x55fd6d4148e0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f105c0f49f0 tx=0x7f105c0f4ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a4d551bace95152 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:39.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.133+0000 7f105a600640 1 -- 172.21.15.12:0/3217827838 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c0f7070 con 0x55fd6d40fde0 2024-03-23T23:11:39.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.133+0000 7f1066000640 1 --2- 172.21.15.12:0/2508104 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f10500a27f0 0x7f10500a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:39.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.134+0000 7f105a600640 1 -- 172.21.15.12:0/3217827838 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f105c0bdb50 con 0x55fd6d40fde0 2024-03-23T23:11:39.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.134+0000 7f105a600640 1 -- 172.21.15.12:0/3217827838 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c0c5bf0 con 0x55fd6d40fde0 2024-03-23T23:11:39.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.134+0000 7f1066000640 1 -- 172.21.15.12:0/2508104 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f105c0f0aa0 con 0x7f1068103720 2024-03-23T23:11:39.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.134+0000 7f106e000640 1 --2- 172.21.15.12:0/2508104 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f10500a27f0 0x7f10500a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:39.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.134+0000 7f1072dd6740 1 -- 172.21.15.12:0/3217827838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd6d40fde0 msgr2=0x55fd6d4148e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:39.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.134+0000 7f1072dd6740 1 --2- 172.21.15.12:0/3217827838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd6d40fde0 0x55fd6d4148e0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f105c0f49f0 tx=0x7f105c0f4ae0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.134+0000 7f1072dd6740 1 -- 172.21.15.12:0/3217827838 shutdown_connections 2024-03-23T23:11:39.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.134+0000 7f1072dd6740 1 --2- 172.21.15.12:0/3217827838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d414f50 0x55fd6d4173e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.135+0000 7f1072dd6740 1 --2- 172.21.15.12:0/3217827838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd6d40fde0 0x55fd6d4148e0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.135+0000 7f1072dd6740 1 --2- 172.21.15.12:0/3217827838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd6d40f3f0 0x55fd6d40f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.135+0000 7f1072dd6740 1 -- 172.21.15.12:0/3217827838 >> 172.21.15.12:0/3217827838 conn(0x55fd6d40a960 msgr2=0x55fd6d40ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:39.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.135+0000 7f106e000640 1 --2- 172.21.15.12:0/2508104 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f10500a27f0 0x7f10500a4cb0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x55fd6d410640 tx=0x7f10600096a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:39.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.135+0000 7f1072dd6740 1 -- 172.21.15.12:0/3217827838 shutdown_connections 2024-03-23T23:11:39.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.135+0000 7f1072dd6740 1 -- 172.21.15.12:0/3217827838 wait complete. 2024-03-23T23:11:39.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.135+0000 7f1072dd6740 1 -- start start 2024-03-23T23:11:39.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.135+0000 7f1072dd6740 1 Processor -- start 2024-03-23T23:11:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.136+0000 7f1072dd6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d40f3f0 0x55fd6d419260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.136+0000 7f1072dd6740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd6d40fde0 0x55fd6d4197a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.136+0000 7f1072dd6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd6d414f50 0x55fd6d4207e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.136+0000 7f1072dd6740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd6d330a20 con 0x55fd6d40fde0 2024-03-23T23:11:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.136+0000 7f1072dd6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd6d3708d0 con 0x55fd6d40f3f0 2024-03-23T23:11:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.136+0000 7f1072dd6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd6d417cc0 con 0x55fd6d414f50 2024-03-23T23:11:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.136+0000 7f1072dd6740 10 client.0 _ll_register_callbacks cb 0x55fd6d36d1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.136+0000 7f106d600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d40f3f0 0x55fd6d419260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.136+0000 7f106e000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd6d414f50 0x55fd6d4207e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.136+0000 7f106cc00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd6d40fde0 0x55fd6d4197a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.136+0000 7f106d600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d40f3f0 0x55fd6d419260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:46996/0 (socket says 172.21.15.12:46996) 2024-03-23T23:11:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.136+0000 7f106d600640 1 -- 172.21.15.12:0/981604694 learned_addr learned my addr 172.21.15.12:0/981604694 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.136+0000 7f106d600640 1 -- 172.21.15.12:0/981604694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd6d414f50 msgr2=0x55fd6d4207e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.136+0000 7f106d600640 1 --2- 172.21.15.12:0/981604694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd6d414f50 0x55fd6d4207e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.136+0000 7f106d600640 1 -- 172.21.15.12:0/981604694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd6d40fde0 msgr2=0x55fd6d4197a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.136+0000 7f106d600640 1 --2- 172.21.15.12:0/981604694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd6d40fde0 0x55fd6d4197a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.136+0000 7f106d600640 1 -- 172.21.15.12:0/981604694 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fd6d089210 con 0x55fd6d40f3f0 2024-03-23T23:11:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.137+0000 7f106d600640 1 --2- 172.21.15.12:0/981604694 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d40f3f0 0x55fd6d419260 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f105c0ee600 tx=0x7f105c0f40e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=415fcc056b18d219 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.137+0000 7f104ea00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:11:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.137+0000 7f104ea00640 1 -- 172.21.15.12:0/981604694 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c0f7070 con 0x55fd6d40f3f0 2024-03-23T23:11:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.137+0000 7f104ea00640 1 -- 172.21.15.12:0/981604694 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f105c0bb750 con 0x55fd6d40f3f0 2024-03-23T23:11:39.141 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.137+0000 7f1072dd6740 1 -- 172.21.15.12:0/981604694 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f105c0bdb50 con 0x55fd6d40f3f0 2024-03-23T23:11:39.141 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.137+0000 7f104ea00640 1 -- 172.21.15.12:0/981604694 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f105c0b96f0 con 0x55fd6d40f3f0 2024-03-23T23:11:39.141 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.137+0000 7f1072dd6740 10 client.28091 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:39.141 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.137+0000 7f1072dd6740 1 -- 172.21.15.12:0/981604694 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55fd6d155620 con 0x55fd6d40f3f0 2024-03-23T23:11:39.141 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.137+0000 7f104ea00640 1 -- 172.21.15.12:0/981604694 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f105c0bdb50 con 0x55fd6d40f3f0 2024-03-23T23:11:39.141 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.138+0000 7f104ea00640 1 -- 172.21.15.12:0/981604694 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f105c0ee630 con 0x55fd6d40f3f0 2024-03-23T23:11:39.141 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.138+0000 7f1072dd6740 10 client.28091 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:39.141 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.138+0000 7f1072dd6740 10 client.28091 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:11:39.141 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.138+0000 7f1072dd6740 20 client.28091 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:39.141 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.138+0000 7f1072dd6740 1 --2- 172.21.15.12:0/981604694 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fd6d424cd0 0x55fd6d4270c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:39.141 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.138+0000 7f1072dd6740 4 client.28091 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:11:39.141 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.138+0000 7f1072dd6740 1 -- 172.21.15.12:0/981604694 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fd6d381d70 con 0x55fd6d424cd0 2024-03-23T23:11:39.141 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.138+0000 7f106cc00640 1 --2- 172.21.15.12:0/981604694 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fd6d424cd0 0x55fd6d4270c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:39.141 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.138+0000 7f106cc00640 1 --2- 172.21.15.12:0/981604694 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fd6d424cd0 0x55fd6d4270c0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5014ad65badda4d server_cookie=a332ca8f2a586eed in_seq=0 out_seq=0 2024-03-23T23:11:39.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.139+0000 7f104ea00640 10 client.28091 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:39.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.139+0000 7f104ea00640 1 -- 172.21.15.12:0/981604694 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55fd6d381d70 con 0x55fd6d424cd0 2024-03-23T23:11:39.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.139+0000 7f104ea00640 10 client.28091 handle_command_reply: tid=0 2024-03-23T23:11:39.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.139+0000 7f1072dd6740 1 client.28091 shutdown 2024-03-23T23:11:39.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.140+0000 7f1072dd6740 1 -- 172.21.15.12:0/981604694 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d40f3f0 msgr2=0x55fd6d419260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:39.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.140+0000 7f1072dd6740 1 --2- 172.21.15.12:0/981604694 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d40f3f0 0x55fd6d419260 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f105c0ee600 tx=0x7f105c0f40e0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.140+0000 7f1072dd6740 1 -- 172.21.15.12:0/981604694 shutdown_connections 2024-03-23T23:11:39.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.140+0000 7f1072dd6740 1 --2- 172.21.15.12:0/981604694 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fd6d424cd0 0x55fd6d4270c0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.140+0000 7f1072dd6740 1 --2- 172.21.15.12:0/981604694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd6d414f50 0x55fd6d4207e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.140+0000 7f1072dd6740 1 --2- 172.21.15.12:0/981604694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd6d40fde0 0x55fd6d4197a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.140+0000 7f1072dd6740 1 --2- 172.21.15.12:0/981604694 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d40f3f0 0x55fd6d419260 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.140+0000 7f104ea00640 0 client.28091 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:39.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.140+0000 7f1072dd6740 1 -- 172.21.15.12:0/981604694 >> 172.21.15.12:0/981604694 conn(0x55fd6d40a960 msgr2=0x55fd6d409b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:39.142 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.140+0000 7f1072dd6740 1 -- 172.21.15.12:0/981604694 shutdown_connections 2024-03-23T23:11:39.143 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.140+0000 7f1072dd6740 1 -- 172.21.15.12:0/981604694 wait complete. 2024-03-23T23:11:39.143 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.140+0000 7f1072dd6740 20 client.28091 trim_cache size 0 max 16384 2024-03-23T23:11:39.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.155+0000 7f1072dd6740 1 Processor -- start 2024-03-23T23:11:39.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.155+0000 7f1072dd6740 1 -- start start 2024-03-23T23:11:39.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.155+0000 7f1072dd6740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd6d40f3f0 0x55fd6d40ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:39.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.155+0000 7f1072dd6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd6d40fde0 0x55fd6d4207a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:39.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.156+0000 7f1072dd6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d414f50 0x55fd6d418e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:39.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.156+0000 7f1072dd6740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10681aceb0 con 0x55fd6d40f3f0 2024-03-23T23:11:39.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.156+0000 7f1072dd6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10681aa9b0 con 0x55fd6d414f50 2024-03-23T23:11:39.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.156+0000 7f1072dd6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10681a8a50 con 0x55fd6d40fde0 2024-03-23T23:11:39.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.156+0000 7f106cc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd6d40fde0 0x55fd6d4207a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:39.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.156+0000 7f106d600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd6d40f3f0 0x55fd6d40ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:39.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.156+0000 7f106cc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd6d40fde0 0x55fd6d4207a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49554/0 (socket says 172.21.15.12:49554) 2024-03-23T23:11:39.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.156+0000 7f106e000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d414f50 0x55fd6d418e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:39.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.156+0000 7f106cc00640 1 -- 172.21.15.12:0/1597614381 learned_addr learned my addr 172.21.15.12:0/1597614381 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:39.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.156+0000 7f106cc00640 1 -- 172.21.15.12:0/1597614381 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d414f50 msgr2=0x55fd6d418e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:39.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.156+0000 7f106cc00640 1 --2- 172.21.15.12:0/1597614381 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d414f50 0x55fd6d418e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.156+0000 7f106cc00640 1 -- 172.21.15.12:0/1597614381 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd6d40f3f0 msgr2=0x55fd6d40ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:39.157 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.156+0000 7f106cc00640 1 --2- 172.21.15.12:0/1597614381 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd6d40f3f0 0x55fd6d40ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.157 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.156+0000 7f106cc00640 1 -- 172.21.15.12:0/1597614381 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fd6d427650 con 0x55fd6d40fde0 2024-03-23T23:11:39.157 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.156+0000 7f106e000640 1 --2- 172.21.15.12:0/1597614381 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d414f50 0x55fd6d418e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:39.157 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.157+0000 7f106cc00640 1 --2- 172.21.15.12:0/1597614381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd6d40fde0 0x55fd6d4207a0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f1054009640 tx=0x7f1054005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ffd3cc5954da82c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:39.157 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.157+0000 7f1059c00640 1 -- 172.21.15.12:0/1597614381 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1054022e30 con 0x55fd6d40fde0 2024-03-23T23:11:39.157 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.157+0000 7f1059c00640 1 -- 172.21.15.12:0/1597614381 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1054005500 con 0x55fd6d40fde0 2024-03-23T23:11:39.157 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.157+0000 7f1059c00640 1 -- 172.21.15.12:0/1597614381 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1054004b20 con 0x55fd6d40fde0 2024-03-23T23:11:39.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.157+0000 7f1072dd6740 1 -- 172.21.15.12:0/1597614381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd6d40fde0 msgr2=0x55fd6d4207a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:39.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.157+0000 7f1072dd6740 1 --2- 172.21.15.12:0/1597614381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd6d40fde0 0x55fd6d4207a0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f1054009640 tx=0x7f1054005f90 comp rx=0 tx=0).stop 2024-03-23T23:11:39.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.158+0000 7f1072dd6740 1 -- 172.21.15.12:0/1597614381 shutdown_connections 2024-03-23T23:11:39.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.158+0000 7f1072dd6740 1 --2- 172.21.15.12:0/1597614381 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d414f50 0x55fd6d418e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.158+0000 7f1072dd6740 1 --2- 172.21.15.12:0/1597614381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd6d40fde0 0x55fd6d4207a0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.158+0000 7f1072dd6740 1 --2- 172.21.15.12:0/1597614381 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd6d40f3f0 0x55fd6d40ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.158+0000 7f1072dd6740 1 -- 172.21.15.12:0/1597614381 >> 172.21.15.12:0/1597614381 conn(0x55fd6d40a960 msgr2=0x55fd6d4056a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:39.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.158+0000 7f1072dd6740 1 -- 172.21.15.12:0/1597614381 shutdown_connections 2024-03-23T23:11:39.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.158+0000 7f1072dd6740 1 -- 172.21.15.12:0/1597614381 wait complete. 2024-03-23T23:11:39.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.158+0000 7f1072dd6740 1 -- start start 2024-03-23T23:11:39.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.158+0000 7f1072dd6740 1 Processor -- start 2024-03-23T23:11:39.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.159+0000 7f1072dd6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d40f3f0 0x55fd6d41e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:39.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.159+0000 7f1072dd6740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd6d40fde0 0x55fd6d41e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:39.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.159+0000 7f1072dd6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd6d414f50 0x55fd6d41f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:39.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.159+0000 7f1072dd6740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10681a9050 con 0x55fd6d40fde0 2024-03-23T23:11:39.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.159+0000 7f1072dd6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10681aa350 con 0x55fd6d40f3f0 2024-03-23T23:11:39.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.159+0000 7f1072dd6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10681a9690 con 0x55fd6d414f50 2024-03-23T23:11:39.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.159+0000 7f1072dd6740 10 client.0 _ll_register_callbacks cb 0x7f1060009ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:39.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.159+0000 7f106d600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d40f3f0 0x55fd6d41e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:39.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.159+0000 7f106cc00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd6d40fde0 0x55fd6d41e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:39.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.159+0000 7f106d600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d40f3f0 0x55fd6d41e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:47012/0 (socket says 172.21.15.12:47012) 2024-03-23T23:11:39.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.159+0000 7f106d600640 1 -- 172.21.15.12:0/3886271771 learned_addr learned my addr 172.21.15.12:0/3886271771 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:39.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.159+0000 7f106e000640 1 --2- 172.21.15.12:0/3886271771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd6d414f50 0x55fd6d41f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:39.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.159+0000 7f106d600640 1 -- 172.21.15.12:0/3886271771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd6d414f50 msgr2=0x55fd6d41f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:39.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.159+0000 7f106d600640 1 --2- 172.21.15.12:0/3886271771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd6d414f50 0x55fd6d41f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.159+0000 7f106d600640 1 -- 172.21.15.12:0/3886271771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd6d40fde0 msgr2=0x55fd6d41e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:39.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.159+0000 7f106d600640 1 --2- 172.21.15.12:0/3886271771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd6d40fde0 0x55fd6d41e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.159+0000 7f106d600640 1 -- 172.21.15.12:0/3886271771 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fd6d41fc60 con 0x55fd6d40f3f0 2024-03-23T23:11:39.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.159+0000 7f106cc00640 1 --2- 172.21.15.12:0/3886271771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd6d40fde0 0x55fd6d41e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:39.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.160+0000 7f106d600640 1 --2- 172.21.15.12:0/3886271771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d40f3f0 0x55fd6d41e120 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f105c0c5db0 tx=0x7f105c02bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f799efd5d83da5f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:39.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.160+0000 7f104ea00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:11:39.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.160+0000 7f104ea00640 1 -- 172.21.15.12:0/3886271771 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c0f7070 con 0x55fd6d40f3f0 2024-03-23T23:11:39.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.160+0000 7f104ea00640 1 -- 172.21.15.12:0/3886271771 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f105c0c5760 con 0x55fd6d40f3f0 2024-03-23T23:11:39.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.160+0000 7f1072dd6740 1 -- 172.21.15.12:0/3886271771 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1054005500 con 0x55fd6d40f3f0 2024-03-23T23:11:39.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.160+0000 7f104ea00640 1 -- 172.21.15.12:0/3886271771 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f105c0bcf90 con 0x55fd6d40f3f0 2024-03-23T23:11:39.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.160+0000 7f1072dd6740 10 client.28097 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:39.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.160+0000 7f1072dd6740 1 -- 172.21.15.12:0/3886271771 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55fd6d41fc60 con 0x55fd6d40f3f0 2024-03-23T23:11:39.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.160+0000 7f104ea00640 1 -- 172.21.15.12:0/3886271771 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1054005500 con 0x55fd6d40f3f0 2024-03-23T23:11:39.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.160+0000 7f104ea00640 1 -- 172.21.15.12:0/3886271771 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f105c004250 con 0x55fd6d40f3f0 2024-03-23T23:11:39.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.161+0000 7f1072dd6740 10 client.28097 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:39.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.161+0000 7f1072dd6740 10 client.28097 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:11:39.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.161+0000 7f1072dd6740 20 client.28097 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:39.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.161+0000 7f1072dd6740 1 --2- 172.21.15.12:0/3886271771 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fd6d480bc0 0x55fd6d482fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:39.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.161+0000 7f1072dd6740 4 client.28097 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:11:39.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.161+0000 7f1072dd6740 1 -- 172.21.15.12:0/3886271771 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55fd6d37c0c0 con 0x55fd6d480bc0 2024-03-23T23:11:39.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.161+0000 7f106cc00640 1 --2- 172.21.15.12:0/3886271771 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fd6d480bc0 0x55fd6d482fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:39.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.161+0000 7f106cc00640 1 --2- 172.21.15.12:0/3886271771 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fd6d480bc0 0x55fd6d482fe0 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a245c3fc9b779021 server_cookie=208dc187f302ec5b in_seq=0 out_seq=0 2024-03-23T23:11:39.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.161+0000 7f104ea00640 10 client.28097 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:39.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.172+0000 7f104ea00640 1 -- 172.21.15.12:0/3886271771 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55fd6d37c0c0 con 0x55fd6d480bc0 2024-03-23T23:11:39.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.172+0000 7f104ea00640 10 client.28097 handle_command_reply: tid=0 2024-03-23T23:11:39.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.173+0000 7f1072dd6740 1 client.28097 shutdown 2024-03-23T23:11:39.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.173+0000 7f1072dd6740 1 -- 172.21.15.12:0/3886271771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d40f3f0 msgr2=0x55fd6d41e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:39.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.173+0000 7f1072dd6740 1 --2- 172.21.15.12:0/3886271771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d40f3f0 0x55fd6d41e120 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f105c0c5db0 tx=0x7f105c02bc30 comp rx=0 tx=0).stop 2024-03-23T23:11:39.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.173+0000 7f1072dd6740 1 -- 172.21.15.12:0/3886271771 shutdown_connections 2024-03-23T23:11:39.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.173+0000 7f1072dd6740 1 --2- 172.21.15.12:0/3886271771 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55fd6d480bc0 0x55fd6d482fe0 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.173+0000 7f1072dd6740 1 --2- 172.21.15.12:0/3886271771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd6d414f50 0x55fd6d41f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.173+0000 7f1072dd6740 1 --2- 172.21.15.12:0/3886271771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fd6d40fde0 0x55fd6d41e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.173+0000 7f104ea00640 0 client.28097 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:39.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.173+0000 7f1072dd6740 1 --2- 172.21.15.12:0/3886271771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd6d40f3f0 0x55fd6d41e120 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.173+0000 7f1072dd6740 1 -- 172.21.15.12:0/3886271771 >> 172.21.15.12:0/3886271771 conn(0x55fd6d40a960 msgr2=0x55fd6d40ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:39.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.173+0000 7f1072dd6740 1 -- 172.21.15.12:0/3886271771 shutdown_connections 2024-03-23T23:11:39.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.173+0000 7f1072dd6740 1 -- 172.21.15.12:0/3886271771 wait complete. 2024-03-23T23:11:39.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.173+0000 7f1072dd6740 20 client.28097 trim_cache size 0 max 16384 2024-03-23T23:11:39.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:11:39.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:11:39.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:11:39.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:11:39.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.174+0000 7f106ea00640 1 -- 172.21.15.12:0/2508104 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f10500a27f0 msgr2=0x7f10500a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:39.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.174+0000 7f106ea00640 1 --2- 172.21.15.12:0/2508104 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f10500a27f0 0x7f10500a4cb0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x55fd6d410640 tx=0x7f10600096a0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.174+0000 7f106ea00640 1 -- 172.21.15.12:0/2508104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1068103720 msgr2=0x7f1068199a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:39.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.174+0000 7f106ea00640 1 --2- 172.21.15.12:0/2508104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1068103720 0x7f1068199a50 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f105c009230 tx=0x7f105c02c6c0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.174 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.174+0000 7f106ea00640 1 -- 172.21.15.12:0/2508104 shutdown_connections 2024-03-23T23:11:39.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.174+0000 7f106ea00640 1 --2- 172.21.15.12:0/2508104 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f10500a27f0 0x7f10500a4cb0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.174+0000 7f106ea00640 1 --2- 172.21.15.12:0/2508104 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10681052d0 0x7f10681a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.174+0000 7f106ea00640 1 --2- 172.21.15.12:0/2508104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1068104910 0x7f1068199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.174+0000 7f106ea00640 1 --2- 172.21.15.12:0/2508104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1068103720 0x7f1068199a50 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:39.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.174+0000 7f106ea00640 1 -- 172.21.15.12:0/2508104 >> 172.21.15.12:0/2508104 conn(0x7f10680fecb0 msgr2=0x7f1068107c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:39.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.174+0000 7f106ea00640 1 -- 172.21.15.12:0/2508104 shutdown_connections 2024-03-23T23:11:39.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:39.174+0000 7f106ea00640 1 -- 172.21.15.12:0/2508104 wait complete. 2024-03-23T23:11:39.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:38 smithi012 ceph-mon[31695]: pgmap v685: 97 pgs: 97 active+clean; 2.6 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:11:39.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:38 smithi173 ceph-mon[36116]: pgmap v685: 97 pgs: 97 active+clean; 2.6 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:11:39.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:38 smithi203 ceph-mon[36693]: pgmap v685: 97 pgs: 97 active+clean; 2.6 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:11:41.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:40 smithi012 ceph-mon[31695]: pgmap v686: 97 pgs: 97 active+clean; 2.6 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:11:41.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:40 smithi173 ceph-mon[36116]: pgmap v686: 97 pgs: 97 active+clean; 2.6 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:11:41.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:40 smithi203 ceph-mon[36693]: pgmap v686: 97 pgs: 97 active+clean; 2.6 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:11:43.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:42 smithi012 ceph-mon[31695]: pgmap v687: 97 pgs: 97 active+clean; 2.6 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:11:43.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:42 smithi173 ceph-mon[36116]: pgmap v687: 97 pgs: 97 active+clean; 2.6 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:11:43.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:42 smithi203 ceph-mon[36693]: pgmap v687: 97 pgs: 97 active+clean; 2.6 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:11:45.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:44 smithi012 ceph-mon[31695]: pgmap v688: 97 pgs: 97 active+clean; 2.6 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:11:45.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:44 smithi173 ceph-mon[36116]: pgmap v688: 97 pgs: 97 active+clean; 2.6 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:11:45.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:44 smithi203 ceph-mon[36693]: pgmap v688: 97 pgs: 97 active+clean; 2.6 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:11:47.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:46 smithi012 ceph-mon[31695]: pgmap v689: 97 pgs: 97 active+clean; 2.7 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:11:47.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:46 smithi173 ceph-mon[36116]: pgmap v689: 97 pgs: 97 active+clean; 2.7 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:11:47.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:46 smithi203 ceph-mon[36693]: pgmap v689: 97 pgs: 97 active+clean; 2.7 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:11:49.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:48 smithi012 ceph-mon[31695]: pgmap v690: 97 pgs: 97 active+clean; 2.7 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:11:49.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:48 smithi173 ceph-mon[36116]: pgmap v690: 97 pgs: 97 active+clean; 2.7 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:11:49.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:48 smithi203 ceph-mon[36693]: pgmap v690: 97 pgs: 97 active+clean; 2.7 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:11:51.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:50 smithi012 ceph-mon[31695]: pgmap v691: 97 pgs: 97 active+clean; 2.7 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:11:51.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:50 smithi173 ceph-mon[36116]: pgmap v691: 97 pgs: 97 active+clean; 2.7 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:11:51.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:50 smithi203 ceph-mon[36693]: pgmap v691: 97 pgs: 97 active+clean; 2.7 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:11:52.136 INFO:tasks.workunit.client.0.smithi012.stdout:437/600: open fd = 3 2024-03-23T23:11:52.137 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.137 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.137 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.137 INFO:tasks.workunit.client.0.smithi012.stdout:438/600: open fd = 3 2024-03-23T23:11:52.137 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.137 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.137 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.137 INFO:tasks.workunit.client.0.smithi012.stdout:439/600: open fd = 3 2024-03-23T23:11:52.137 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.137 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.137 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.137 INFO:tasks.workunit.client.0.smithi012.stdout:440/600: open fd = 3 2024-03-23T23:11:52.137 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.137 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.138 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.138 INFO:tasks.workunit.client.0.smithi012.stdout:441/600: open fd = 3 2024-03-23T23:11:52.138 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.138 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.138 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.138 INFO:tasks.workunit.client.0.smithi012.stdout:442/600: open fd = 3 2024-03-23T23:11:52.138 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.138 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.138 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.138 INFO:tasks.workunit.client.0.smithi012.stdout:443/600: open fd = 3 2024-03-23T23:11:52.138 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.138 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.138 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.139 INFO:tasks.workunit.client.0.smithi012.stdout:444/600: open fd = 3 2024-03-23T23:11:52.139 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.139 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.139 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.139 INFO:tasks.workunit.client.0.smithi012.stdout:445/600: open fd = 3 2024-03-23T23:11:52.139 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.139 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.139 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.139 INFO:tasks.workunit.client.0.smithi012.stdout:446/600: open fd = 3 2024-03-23T23:11:52.139 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.139 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.139 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.139 INFO:tasks.workunit.client.0.smithi012.stdout:447/600: open fd = 3 2024-03-23T23:11:52.139 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.139 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.139 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.139 INFO:tasks.workunit.client.0.smithi012.stdout:448/600: open fd = 3 2024-03-23T23:11:52.140 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.140 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.140 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.140 INFO:tasks.workunit.client.0.smithi012.stdout:449/600: open fd = 3 2024-03-23T23:11:52.140 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.140 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.140 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.140 INFO:tasks.workunit.client.0.smithi012.stdout:450/600: open fd = 3 2024-03-23T23:11:52.140 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.140 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.140 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.140 INFO:tasks.workunit.client.0.smithi012.stdout:451/600: open fd = 3 2024-03-23T23:11:52.140 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.140 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.140 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.140 INFO:tasks.workunit.client.0.smithi012.stdout:452/600: open fd = 3 2024-03-23T23:11:52.141 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.141 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.141 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.141 INFO:tasks.workunit.client.0.smithi012.stdout:453/600: open fd = 3 2024-03-23T23:11:52.141 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.141 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.141 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.141 INFO:tasks.workunit.client.0.smithi012.stdout:454/600: open fd = 3 2024-03-23T23:11:52.141 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.141 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.141 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.141 INFO:tasks.workunit.client.0.smithi012.stdout:455/600: open fd = 3 2024-03-23T23:11:52.141 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.141 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.141 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.142 INFO:tasks.workunit.client.0.smithi012.stdout:456/600: open fd = 3 2024-03-23T23:11:52.142 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.142 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.142 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.142 INFO:tasks.workunit.client.0.smithi012.stdout:457/600: open fd = 3 2024-03-23T23:11:52.142 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.142 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.142 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.142 INFO:tasks.workunit.client.0.smithi012.stdout:458/600: open fd = 3 2024-03-23T23:11:52.142 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.142 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.142 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.142 INFO:tasks.workunit.client.0.smithi012.stdout:459/600: open fd = 3 2024-03-23T23:11:52.142 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.142 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.142 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.143 INFO:tasks.workunit.client.0.smithi012.stdout:460/600: open fd = 3 2024-03-23T23:11:52.143 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.143 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.143 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.143 INFO:tasks.workunit.client.0.smithi012.stdout:461/600: open fd = 3 2024-03-23T23:11:52.143 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.143 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.143 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.143 INFO:tasks.workunit.client.0.smithi012.stdout:462/600: open fd = 3 2024-03-23T23:11:52.143 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.143 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.143 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.143 INFO:tasks.workunit.client.0.smithi012.stdout:463/600: open fd = 3 2024-03-23T23:11:52.143 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.143 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.144 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.144 INFO:tasks.workunit.client.0.smithi012.stdout:464/600: open fd = 3 2024-03-23T23:11:52.144 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.144 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.144 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.144 INFO:tasks.workunit.client.0.smithi012.stdout:465/600: open fd = 3 2024-03-23T23:11:52.144 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.144 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.144 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.144 INFO:tasks.workunit.client.0.smithi012.stdout:466/600: open fd = 3 2024-03-23T23:11:52.144 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.144 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.144 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.144 INFO:tasks.workunit.client.0.smithi012.stdout:467/600: open fd = 3 2024-03-23T23:11:52.145 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.145 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.145 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.145 INFO:tasks.workunit.client.0.smithi012.stdout:468/600: open fd = 3 2024-03-23T23:11:52.145 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.145 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.145 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.145 INFO:tasks.workunit.client.0.smithi012.stdout:469/600: open fd = 3 2024-03-23T23:11:52.145 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.145 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.145 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.145 INFO:tasks.workunit.client.0.smithi012.stdout:470/600: open fd = 3 2024-03-23T23:11:52.145 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.145 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.145 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.145 INFO:tasks.workunit.client.0.smithi012.stdout:471/600: open fd = 3 2024-03-23T23:11:52.146 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.146 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.146 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.146 INFO:tasks.workunit.client.0.smithi012.stdout:472/600: open fd = 3 2024-03-23T23:11:52.146 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.146 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.146 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.146 INFO:tasks.workunit.client.0.smithi012.stdout:473/600: open fd = 3 2024-03-23T23:11:52.146 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.146 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.146 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.146 INFO:tasks.workunit.client.0.smithi012.stdout:474/600: open fd = 3 2024-03-23T23:11:52.146 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.146 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.146 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.147 INFO:tasks.workunit.client.0.smithi012.stdout:475/600: open fd = 3 2024-03-23T23:11:52.147 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.147 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.147 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.147 INFO:tasks.workunit.client.0.smithi012.stdout:476/600: open fd = 3 2024-03-23T23:11:52.147 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.147 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.147 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.147 INFO:tasks.workunit.client.0.smithi012.stdout:477/600: open fd = 3 2024-03-23T23:11:52.147 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.147 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.147 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.147 INFO:tasks.workunit.client.0.smithi012.stdout:478/600: open fd = 3 2024-03-23T23:11:52.147 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.147 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.148 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.148 INFO:tasks.workunit.client.0.smithi012.stdout:479/600: open fd = 3 2024-03-23T23:11:52.148 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.148 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.148 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.148 INFO:tasks.workunit.client.0.smithi012.stdout:480/600: open fd = 3 2024-03-23T23:11:52.148 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.148 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.148 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.148 INFO:tasks.workunit.client.0.smithi012.stdout:481/600: open fd = 3 2024-03-23T23:11:52.148 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.148 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.148 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.148 INFO:tasks.workunit.client.0.smithi012.stdout:482/600: open fd = 3 2024-03-23T23:11:52.148 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.148 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.148 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.149 INFO:tasks.workunit.client.0.smithi012.stdout:483/600: open fd = 3 2024-03-23T23:11:52.149 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.149 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.149 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.149 INFO:tasks.workunit.client.0.smithi012.stdout:484/600: open fd = 3 2024-03-23T23:11:52.149 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.149 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.149 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.149 INFO:tasks.workunit.client.0.smithi012.stdout:485/600: open fd = 3 2024-03-23T23:11:52.149 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.149 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.149 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.149 INFO:tasks.workunit.client.0.smithi012.stdout:486/600: open fd = 3 2024-03-23T23:11:52.149 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.149 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.150 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.150 INFO:tasks.workunit.client.0.smithi012.stdout:487/600: open fd = 3 2024-03-23T23:11:52.150 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.150 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.150 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.150 INFO:tasks.workunit.client.0.smithi012.stdout:488/600: open fd = 3 2024-03-23T23:11:52.150 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.150 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.150 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.150 INFO:tasks.workunit.client.0.smithi012.stdout:489/600: open fd = 3 2024-03-23T23:11:52.150 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.150 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.150 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.150 INFO:tasks.workunit.client.0.smithi012.stdout:490/600: open fd = 3 2024-03-23T23:11:52.151 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.151 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.151 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.151 INFO:tasks.workunit.client.0.smithi012.stdout:491/600: open fd = 3 2024-03-23T23:11:52.151 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.151 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.151 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.151 INFO:tasks.workunit.client.0.smithi012.stdout:492/600: open fd = 3 2024-03-23T23:11:52.151 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.151 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.151 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.151 INFO:tasks.workunit.client.0.smithi012.stdout:493/600: open fd = 3 2024-03-23T23:11:52.151 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.152 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.152 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.152 INFO:tasks.workunit.client.0.smithi012.stdout:494/600: open fd = 3 2024-03-23T23:11:52.152 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.152 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.152 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.152 INFO:tasks.workunit.client.0.smithi012.stdout:495/600: open fd = 3 2024-03-23T23:11:52.152 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.152 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.152 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.152 INFO:tasks.workunit.client.0.smithi012.stdout:496/600: open fd = 3 2024-03-23T23:11:52.152 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.152 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.152 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.152 INFO:tasks.workunit.client.0.smithi012.stdout:497/600: open fd = 3 2024-03-23T23:11:52.152 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.152 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.153 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.153 INFO:tasks.workunit.client.0.smithi012.stdout:498/600: open fd = 3 2024-03-23T23:11:52.153 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.153 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:11:52.153 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:11:52.318 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.319+0000 7f8eda400640 1 -- 172.21.15.12:0/772535142 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ed41073a0 msgr2=0x7f8ed41097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:52.318 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.319+0000 7f8eda400640 1 --2- 172.21.15.12:0/772535142 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ed41073a0 0x7f8ed41097f0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc009be0 tx=0x7f8ecc02bef0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.318 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.319+0000 7f8eda400640 1 -- 172.21.15.12:0/772535142 shutdown_connections 2024-03-23T23:11:52.318 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.319+0000 7f8eda400640 1 --2- 172.21.15.12:0/772535142 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ed41073a0 0x7f8ed41097f0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.318 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.319+0000 7f8eda400640 1 --2- 172.21.15.12:0/772535142 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ed4104a70 0x7f8ed4106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.318 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.319+0000 7f8eda400640 1 --2- 172.21.15.12:0/772535142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ed4102140 0x7f8ed4104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.318 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.319+0000 7f8eda400640 1 -- 172.21.15.12:0/772535142 >> 172.21.15.12:0/772535142 conn(0x7f8ed40fbb30 msgr2=0x7f8ed40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:52.319 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.319+0000 7f8eda400640 1 -- 172.21.15.12:0/772535142 shutdown_connections 2024-03-23T23:11:52.319 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.319+0000 7f8eda400640 1 -- 172.21.15.12:0/772535142 wait complete. 2024-03-23T23:11:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.320+0000 7f8eda400640 1 Processor -- start 2024-03-23T23:11:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.320+0000 7f8eda400640 1 -- start start 2024-03-23T23:11:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.320+0000 7f8eda400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ed4102140 0x7f8ed4197890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.320+0000 7f8eda400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ed4104a70 0x7f8ed4197dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.321+0000 7f8eda400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ed41073a0 0x7f8ed419ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.321+0000 7f8eda400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed410d390 con 0x7f8ed4104a70 2024-03-23T23:11:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.321+0000 7f8eda400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed410d210 con 0x7f8ed41073a0 2024-03-23T23:11:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.321+0000 7f8eda400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed410d510 con 0x7f8ed4102140 2024-03-23T23:11:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.321+0000 7f8ed3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ed4102140 0x7f8ed4197890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.321+0000 7f8ed2a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ed4104a70 0x7f8ed4197dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.321+0000 7f8ed3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ed4102140 0x7f8ed4197890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:41912/0 (socket says 172.21.15.12:41912) 2024-03-23T23:11:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.321+0000 7f8ed3400640 1 -- 172.21.15.12:0/2265652293 learned_addr learned my addr 172.21.15.12:0/2265652293 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.321+0000 7f8ed3e00640 1 --2- 172.21.15.12:0/2265652293 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ed41073a0 0x7f8ed419ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.321+0000 7f8ed3400640 1 -- 172.21.15.12:0/2265652293 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ed41073a0 msgr2=0x7f8ed419ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.321+0000 7f8ed3400640 1 --2- 172.21.15.12:0/2265652293 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ed41073a0 0x7f8ed419ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.321+0000 7f8ed3400640 1 -- 172.21.15.12:0/2265652293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ed4104a70 msgr2=0x7f8ed4197dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.321+0000 7f8ed3400640 1 --2- 172.21.15.12:0/2265652293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ed4104a70 0x7f8ed4197dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.321+0000 7f8ed3400640 1 -- 172.21.15.12:0/2265652293 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed419f4b0 con 0x7f8ed4102140 2024-03-23T23:11:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.322+0000 7f8ed3400640 1 --2- 172.21.15.12:0/2265652293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ed4102140 0x7f8ed4197890 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8ec400e9c0 tx=0x7f8ec400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.322+0000 7f8ecbe00640 1 -- 172.21.15.12:0/2265652293 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec400cde0 con 0x7f8ed4102140 2024-03-23T23:11:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.322+0000 7f8ecbe00640 1 -- 172.21.15.12:0/2265652293 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ec40035f0 con 0x7f8ed4102140 2024-03-23T23:11:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.322+0000 7f8ecbe00640 1 -- 172.21.15.12:0/2265652293 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4010690 con 0x7f8ed4102140 2024-03-23T23:11:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.322+0000 7f8eda400640 1 -- 172.21.15.12:0/2265652293 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed419f7a0 con 0x7f8ed4102140 2024-03-23T23:11:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.322+0000 7f8eda400640 1 -- 172.21.15.12:0/2265652293 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed419fd40 con 0x7f8ed4102140 2024-03-23T23:11:52.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.325+0000 7f8edd021740 1 Processor -- start 2024-03-23T23:11:52.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.325+0000 7f8ecbe00640 1 -- 172.21.15.12:0/2265652293 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8ec40037a0 con 0x7f8ed4102140 2024-03-23T23:11:52.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.325+0000 7f8edd021740 1 -- start start 2024-03-23T23:11:52.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.325+0000 7f8edd021740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625eae595f0 0x5625eae59a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:52.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.326+0000 7f8edd021740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625eae59fe0 0x5625eae5eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:52.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.326+0000 7f8edd021740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625eae5f150 0x5625eae615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:52.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.326+0000 7f8edd021740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625eae183b0 con 0x5625eae59fe0 2024-03-23T23:11:52.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.326+0000 7f8edd021740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625eadba830 con 0x5625eae5f150 2024-03-23T23:11:52.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.326+0000 7f8edd021740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625ead635d0 con 0x5625eae595f0 2024-03-23T23:11:52.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.326+0000 7f8ed3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625eae595f0 0x5625eae59a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:52.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.326+0000 7f8ed3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625eae595f0 0x5625eae59a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:41920/0 (socket says 172.21.15.12:41920) 2024-03-23T23:11:52.326 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.326+0000 7f8ed3e00640 1 -- 172.21.15.12:0/2643133012 learned_addr learned my addr 172.21.15.12:0/2643133012 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:52.326 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.326+0000 7f8ed2a00640 1 --2- 172.21.15.12:0/2643133012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625eae5f150 0x5625eae615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:52.326 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.326+0000 7f8ed3400640 1 --2- 172.21.15.12:0/2643133012 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625eae59fe0 0x5625eae5eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:52.326 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.326+0000 7f8ed3e00640 1 -- 172.21.15.12:0/2643133012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625eae5f150 msgr2=0x5625eae615e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:52.326 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.326+0000 7f8ed3e00640 1 --2- 172.21.15.12:0/2643133012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625eae5f150 0x5625eae615e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.326 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.326+0000 7f8ed3e00640 1 -- 172.21.15.12:0/2643133012 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625eae59fe0 msgr2=0x5625eae5eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:52.326 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.326+0000 7f8ed3e00640 1 --2- 172.21.15.12:0/2643133012 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625eae59fe0 0x5625eae5eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.326 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.326+0000 7f8ed3e00640 1 -- 172.21.15.12:0/2643133012 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5625ead7b2c0 con 0x5625eae595f0 2024-03-23T23:11:52.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.327+0000 7f8ed3e00640 1 --2- 172.21.15.12:0/2643133012 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625eae595f0 0x5625eae59a10 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc02c500 tx=0x7f8ecc009330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9af7e11223d5f15a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:52.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.327+0000 7f8ec8c00640 1 -- 172.21.15.12:0/2643133012 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc043030 con 0x5625eae595f0 2024-03-23T23:11:52.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.327+0000 7f8ec8c00640 1 -- 172.21.15.12:0/2643133012 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ecc033580 con 0x5625eae595f0 2024-03-23T23:11:52.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.327+0000 7f8ec8c00640 1 -- 172.21.15.12:0/2643133012 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc02c890 con 0x5625eae595f0 2024-03-23T23:11:52.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.327+0000 7f8ecbe00640 1 --2- 172.21.15.12:0/2265652293 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8eb80a2920 0x7f8eb80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:52.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.327+0000 7f8ecbe00640 1 -- 172.21.15.12:0/2265652293 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8ec40ced00 con 0x7f8ed4102140 2024-03-23T23:11:52.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.327+0000 7f8ed3400640 1 --2- 172.21.15.12:0/2265652293 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8eb80a2920 0x7f8eb80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:52.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.328+0000 7f8edd021740 1 -- 172.21.15.12:0/2643133012 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625eae595f0 msgr2=0x5625eae59a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:52.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.328+0000 7f8edd021740 1 --2- 172.21.15.12:0/2643133012 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625eae595f0 0x5625eae59a10 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc02c500 tx=0x7f8ecc009330 comp rx=0 tx=0).stop 2024-03-23T23:11:52.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.328+0000 7f8edd021740 1 -- 172.21.15.12:0/2643133012 shutdown_connections 2024-03-23T23:11:52.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.328+0000 7f8edd021740 1 --2- 172.21.15.12:0/2643133012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625eae5f150 0x5625eae615e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.328+0000 7f8edd021740 1 --2- 172.21.15.12:0/2643133012 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625eae59fe0 0x5625eae5eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.328+0000 7f8edd021740 1 --2- 172.21.15.12:0/2643133012 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625eae595f0 0x5625eae59a10 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.328+0000 7f8edd021740 1 -- 172.21.15.12:0/2643133012 >> 172.21.15.12:0/2643133012 conn(0x5625eae54b60 msgr2=0x5625eae54f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:52.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.328+0000 7f8edd021740 1 -- 172.21.15.12:0/2643133012 shutdown_connections 2024-03-23T23:11:52.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.328+0000 7f8edd021740 1 -- 172.21.15.12:0/2643133012 wait complete. 2024-03-23T23:11:52.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.328+0000 7f8ed3400640 1 --2- 172.21.15.12:0/2265652293 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8eb80a2920 0x7f8eb80a4de0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f8ec40ccb80 tx=0x7f8ec4004570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:52.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.328+0000 7f8edd021740 1 -- start start 2024-03-23T23:11:52.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.329+0000 7f8edd021740 1 Processor -- start 2024-03-23T23:11:52.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.329+0000 7f8edd021740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625eae595f0 0x5625eae636d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:52.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.329+0000 7f8edd021740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625eae59fe0 0x5625eae63c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:52.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.329+0000 7f8edd021740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625eae5f150 0x5625eae6ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:52.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.329+0000 7f8edd021740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625ead76a20 con 0x5625eae595f0 2024-03-23T23:11:52.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.329+0000 7f8edd021740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625eadb68d0 con 0x5625eae5f150 2024-03-23T23:11:52.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.329+0000 7f8edd021740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625eae62040 con 0x5625eae59fe0 2024-03-23T23:11:52.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.329+0000 7f8edd021740 10 client.0 _ll_register_callbacks cb 0x5625eadb31c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:52.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.329+0000 7f8ed3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625eae595f0 0x5625eae636d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:52.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.329+0000 7f8ed3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625eae595f0 0x5625eae636d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35852/0 (socket says 172.21.15.12:35852) 2024-03-23T23:11:52.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.329+0000 7f8ed3e00640 1 -- 172.21.15.12:0/1771523862 learned_addr learned my addr 172.21.15.12:0/1771523862 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:52.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.329+0000 7f8ed2a00640 1 --2- 172.21.15.12:0/1771523862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625eae59fe0 0x5625eae63c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:52.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.329+0000 7f8ed3e00640 1 --2- 172.21.15.12:0/1771523862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625eae5f150 0x5625eae6ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:52.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.330+0000 7f8ed2a00640 1 -- 172.21.15.12:0/1771523862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625eae5f150 msgr2=0x5625eae6ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:52.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.330+0000 7f8ed2a00640 1 --2- 172.21.15.12:0/1771523862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625eae5f150 0x5625eae6ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.330+0000 7f8ed2a00640 1 -- 172.21.15.12:0/1771523862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625eae595f0 msgr2=0x5625eae636d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:52.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.330+0000 7f8ed2a00640 1 --2- 172.21.15.12:0/1771523862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625eae595f0 0x5625eae636d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.330+0000 7f8ed2a00640 1 -- 172.21.15.12:0/1771523862 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5625eaacf210 con 0x5625eae59fe0 2024-03-23T23:11:52.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.330+0000 7f8ed3e00640 1 --2- 172.21.15.12:0/1771523862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625eae595f0 0x5625eae636d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:52.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.330+0000 7f8ed2a00640 1 --2- 172.21.15.12:0/1771523862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625eae59fe0 0x5625eae63c10 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8ed4198da0 tx=0x7f8ec0012b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91eff2e1f8a83c91 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:52.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.330+0000 7f8ebd600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:11:52.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.330+0000 7f8ebd600640 1 -- 172.21.15.12:0/1771523862 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec000f890 con 0x5625eae59fe0 2024-03-23T23:11:52.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.330+0000 7f8ebd600640 1 -- 172.21.15.12:0/1771523862 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8ec000fa30 con 0x5625eae59fe0 2024-03-23T23:11:52.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.330+0000 7f8edd021740 1 -- 172.21.15.12:0/1771523862 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ecc033580 con 0x5625eae59fe0 2024-03-23T23:11:52.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.331+0000 7f8ebd600640 1 -- 172.21.15.12:0/1771523862 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ec000fc30 con 0x5625eae59fe0 2024-03-23T23:11:52.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.331+0000 7f8edd021740 10 client.28075 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:52.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.331+0000 7f8edd021740 1 -- 172.21.15.12:0/1771523862 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5625eab9b620 con 0x5625eae59fe0 2024-03-23T23:11:52.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.331+0000 7f8ebd600640 1 -- 172.21.15.12:0/1771523862 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ecc033580 con 0x5625eae59fe0 2024-03-23T23:11:52.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.331+0000 7f8ebd600640 1 -- 172.21.15.12:0/1771523862 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8ec001c420 con 0x5625eae59fe0 2024-03-23T23:11:52.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.331+0000 7f8edd021740 10 client.28075 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:52.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.331+0000 7f8edd021740 10 client.28075 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:11:52.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.331+0000 7f8edd021740 20 client.28075 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:52.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.331+0000 7f8edd021740 1 --2- 172.21.15.12:0/1771523862 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5625eae6f610 0x5625eae71a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:52.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.331+0000 7f8edd021740 4 client.28075 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:11:52.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.331+0000 7f8edd021740 1 -- 172.21.15.12:0/1771523862 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5625eadc7d70 con 0x5625eae6f610 2024-03-23T23:11:52.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.332+0000 7f8ed3e00640 1 --2- 172.21.15.12:0/1771523862 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5625eae6f610 0x5625eae71a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:52.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.332+0000 7f8ed3e00640 1 --2- 172.21.15.12:0/1771523862 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5625eae6f610 0x5625eae71a30 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8659e9c6d8a07c3b server_cookie=775d4a77be14b15f in_seq=0 out_seq=0 2024-03-23T23:11:52.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.332+0000 7f8ebd600640 10 client.28075 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:52.337 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.333+0000 7f8ebd600640 1 -- 172.21.15.12:0/1771523862 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5625eadc7d70 con 0x5625eae6f610 2024-03-23T23:11:52.337 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.333+0000 7f8ebd600640 10 client.28075 handle_command_reply: tid=0 2024-03-23T23:11:52.337 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.333+0000 7f8edd021740 1 client.28075 shutdown 2024-03-23T23:11:52.337 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.333+0000 7f8edd021740 1 -- 172.21.15.12:0/1771523862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625eae59fe0 msgr2=0x5625eae63c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:52.337 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.333+0000 7f8edd021740 1 --2- 172.21.15.12:0/1771523862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625eae59fe0 0x5625eae63c10 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8ed4198da0 tx=0x7f8ec0012b60 comp rx=0 tx=0).stop 2024-03-23T23:11:52.337 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.333+0000 7f8edd021740 1 -- 172.21.15.12:0/1771523862 shutdown_connections 2024-03-23T23:11:52.337 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.333+0000 7f8edd021740 1 --2- 172.21.15.12:0/1771523862 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5625eae6f610 0x5625eae71a30 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.337 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.333+0000 7f8edd021740 1 --2- 172.21.15.12:0/1771523862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625eae5f150 0x5625eae6ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.337 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.333+0000 7f8edd021740 1 --2- 172.21.15.12:0/1771523862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625eae59fe0 0x5625eae63c10 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.338 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.333+0000 7f8edd021740 1 --2- 172.21.15.12:0/1771523862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625eae595f0 0x5625eae636d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.338 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.333+0000 7f8ebd600640 0 client.28075 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:52.338 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.333+0000 7f8edd021740 1 -- 172.21.15.12:0/1771523862 >> 172.21.15.12:0/1771523862 conn(0x5625eae54b60 msgr2=0x5625eae4f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:52.338 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.334+0000 7f8edd021740 1 -- 172.21.15.12:0/1771523862 shutdown_connections 2024-03-23T23:11:52.338 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.334+0000 7f8edd021740 1 -- 172.21.15.12:0/1771523862 wait complete. 2024-03-23T23:11:52.338 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.334+0000 7f8edd021740 20 client.28075 trim_cache size 0 max 16384 2024-03-23T23:11:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.349+0000 7f8edd021740 1 Processor -- start 2024-03-23T23:11:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.349+0000 7f8edd021740 1 -- start start 2024-03-23T23:11:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.349+0000 7f8edd021740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625eae595f0 0x5625eae636d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.349+0000 7f8edd021740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625eae59fe0 0x5625eae6ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.349+0000 7f8edd021740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625eae5f150 0x5625eae63c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.349+0000 7f8edd021740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed41a8bd0 con 0x5625eae595f0 2024-03-23T23:11:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.349+0000 7f8edd021740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed41a66d0 con 0x5625eae59fe0 2024-03-23T23:11:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.349+0000 7f8edd021740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed41a4770 con 0x5625eae5f150 2024-03-23T23:11:52.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.349+0000 7f8ed3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625eae595f0 0x5625eae636d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:52.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.349+0000 7f8ed2a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625eae59fe0 0x5625eae6ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:52.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.349+0000 7f8ed3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625eae595f0 0x5625eae636d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35854/0 (socket says 172.21.15.12:35854) 2024-03-23T23:11:52.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.349+0000 7f8ed2a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625eae59fe0 0x5625eae6ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:42192/0 (socket says 172.21.15.12:42192) 2024-03-23T23:11:52.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.349+0000 7f8ed3e00640 1 -- 172.21.15.12:0/2940281339 learned_addr learned my addr 172.21.15.12:0/2940281339 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:52.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.349+0000 7f8ed3e00640 1 --2- 172.21.15.12:0/2940281339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625eae5f150 0x5625eae63c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.350+0000 7f8ed2a00640 1 -- 172.21.15.12:0/2940281339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625eae5f150 msgr2=0x5625eae63c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.350+0000 7f8ed2a00640 1 --2- 172.21.15.12:0/2940281339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625eae5f150 0x5625eae63c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.350+0000 7f8ed2a00640 1 -- 172.21.15.12:0/2940281339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625eae595f0 msgr2=0x5625eae636d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.350+0000 7f8ed2a00640 1 --2- 172.21.15.12:0/2940281339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625eae595f0 0x5625eae636d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.350+0000 7f8ed2a00640 1 -- 172.21.15.12:0/2940281339 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5625eae62130 con 0x5625eae59fe0 2024-03-23T23:11:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.350+0000 7f8ed2a00640 1 --2- 172.21.15.12:0/2940281339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625eae59fe0 0x5625eae6ac10 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0000c00 tx=0x7f8ec0009850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=befaeb0f1ad3b212 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.350+0000 7f8ebfe00640 1 -- 172.21.15.12:0/2940281339 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec001cb40 con 0x5625eae59fe0 2024-03-23T23:11:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.350+0000 7f8ebfe00640 1 -- 172.21.15.12:0/2940281339 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ec0022030 con 0x5625eae59fe0 2024-03-23T23:11:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.350+0000 7f8ebfe00640 1 -- 172.21.15.12:0/2940281339 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec00035f0 con 0x5625eae59fe0 2024-03-23T23:11:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.351+0000 7f8edd021740 1 -- 172.21.15.12:0/2940281339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625eae59fe0 msgr2=0x5625eae6ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.351+0000 7f8edd021740 1 --2- 172.21.15.12:0/2940281339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625eae59fe0 0x5625eae6ac10 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0000c00 tx=0x7f8ec0009850 comp rx=0 tx=0).stop 2024-03-23T23:11:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.351+0000 7f8edd021740 1 -- 172.21.15.12:0/2940281339 shutdown_connections 2024-03-23T23:11:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.351+0000 7f8edd021740 1 --2- 172.21.15.12:0/2940281339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625eae5f150 0x5625eae63c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.351+0000 7f8edd021740 1 --2- 172.21.15.12:0/2940281339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625eae59fe0 0x5625eae6ac10 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.351+0000 7f8edd021740 1 --2- 172.21.15.12:0/2940281339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625eae595f0 0x5625eae636d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.351+0000 7f8edd021740 1 -- 172.21.15.12:0/2940281339 >> 172.21.15.12:0/2940281339 conn(0x5625eae54b60 msgr2=0x5625eae5b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.351+0000 7f8edd021740 1 -- 172.21.15.12:0/2940281339 shutdown_connections 2024-03-23T23:11:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.351+0000 7f8edd021740 1 -- 172.21.15.12:0/2940281339 wait complete. 2024-03-23T23:11:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.351+0000 7f8edd021740 1 -- start start 2024-03-23T23:11:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.352+0000 7f8edd021740 1 Processor -- start 2024-03-23T23:11:52.354 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.352+0000 7f8edd021740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625eae595f0 0x5625eae686d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:52.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.352+0000 7f8edd021740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625eae59fe0 0x5625eae68c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:52.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.352+0000 7f8edd021740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625eae5f150 0x5625eae69bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:52.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.352+0000 7f8edd021740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed41a4d70 con 0x5625eae59fe0 2024-03-23T23:11:52.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.352+0000 7f8edd021740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed41a6070 con 0x5625eae5f150 2024-03-23T23:11:52.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.352+0000 7f8edd021740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed41a53b0 con 0x5625eae595f0 2024-03-23T23:11:52.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.352+0000 7f8edd021740 10 client.0 _ll_register_callbacks cb 0x7f8ec0009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:52.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.352+0000 7f8ed3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625eae595f0 0x5625eae686d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:52.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.352+0000 7f8ed3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625eae595f0 0x5625eae686d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:41950/0 (socket says 172.21.15.12:41950) 2024-03-23T23:11:52.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.352+0000 7f8ed3e00640 1 -- 172.21.15.12:0/2073410034 learned_addr learned my addr 172.21.15.12:0/2073410034 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:52.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.352+0000 7f8ed2a00640 1 --2- 172.21.15.12:0/2073410034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625eae59fe0 0x5625eae68c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:52.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.352+0000 7f8ed3e00640 1 --2- 172.21.15.12:0/2073410034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625eae5f150 0x5625eae69bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:52.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.353+0000 7f8ed3e00640 1 -- 172.21.15.12:0/2073410034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625eae5f150 msgr2=0x5625eae69bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:52.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.353+0000 7f8ed3e00640 1 --2- 172.21.15.12:0/2073410034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625eae5f150 0x5625eae69bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.353+0000 7f8ed3e00640 1 -- 172.21.15.12:0/2073410034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625eae59fe0 msgr2=0x5625eae68c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.353+0000 7f8ed3e00640 1 --2- 172.21.15.12:0/2073410034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625eae59fe0 0x5625eae68c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.353+0000 7f8ed3e00640 1 -- 172.21.15.12:0/2073410034 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5625eae6a1a0 con 0x5625eae595f0 2024-03-23T23:11:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.353+0000 7f8ed2a00640 1 --2- 172.21.15.12:0/2073410034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625eae59fe0 0x5625eae68c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.353+0000 7f8ed3e00640 1 --2- 172.21.15.12:0/2073410034 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625eae595f0 0x5625eae686d0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc009b30 tx=0x7f8ecc04b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3016b36a846010e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.353+0000 7f8ebd600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:11:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.353+0000 7f8ebd600640 1 -- 172.21.15.12:0/2073410034 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc043040 con 0x5625eae595f0 2024-03-23T23:11:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.353+0000 7f8ebd600640 1 -- 172.21.15.12:0/2073410034 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8ecc02c890 con 0x5625eae595f0 2024-03-23T23:11:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.353+0000 7f8edd021740 1 -- 172.21.15.12:0/2073410034 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ec0022030 con 0x5625eae595f0 2024-03-23T23:11:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.354+0000 7f8ebd600640 1 -- 172.21.15.12:0/2073410034 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ecc02ca90 con 0x5625eae595f0 2024-03-23T23:11:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.354+0000 7f8edd021740 10 client.28081 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.354+0000 7f8edd021740 1 -- 172.21.15.12:0/2073410034 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5625eaecb160 con 0x5625eae595f0 2024-03-23T23:11:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.354+0000 7f8ebd600640 1 -- 172.21.15.12:0/2073410034 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ec0022030 con 0x5625eae595f0 2024-03-23T23:11:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.354+0000 7f8ebd600640 1 -- 172.21.15.12:0/2073410034 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8ecc034430 con 0x5625eae595f0 2024-03-23T23:11:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.354+0000 7f8edd021740 10 client.28081 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.354+0000 7f8edd021740 10 client.28081 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:11:52.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.354+0000 7f8edd021740 20 client.28081 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:52.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.354+0000 7f8edd021740 1 --2- 172.21.15.12:0/2073410034 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5625eaecb7c0 0x5625eaecdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:52.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.354+0000 7f8edd021740 4 client.28081 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:11:52.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.354+0000 7f8edd021740 1 -- 172.21.15.12:0/2073410034 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5625eadc20c0 con 0x5625eaecb7c0 2024-03-23T23:11:52.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.355+0000 7f8ed2a00640 1 --2- 172.21.15.12:0/2073410034 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5625eaecb7c0 0x5625eaecdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:52.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.355+0000 7f8ed2a00640 1 --2- 172.21.15.12:0/2073410034 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5625eaecb7c0 0x5625eaecdbb0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e650be9564e11473 server_cookie=58a7d01b00015a57 in_seq=0 out_seq=0 2024-03-23T23:11:52.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.355+0000 7f8ebd600640 10 client.28081 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:52.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.368+0000 7f8ebd600640 1 -- 172.21.15.12:0/2073410034 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5625eadc20c0 con 0x5625eaecb7c0 2024-03-23T23:11:52.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.368+0000 7f8ebd600640 10 client.28081 handle_command_reply: tid=0 2024-03-23T23:11:52.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.368+0000 7f8edd021740 1 client.28081 shutdown 2024-03-23T23:11:52.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.368+0000 7f8edd021740 1 -- 172.21.15.12:0/2073410034 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625eae595f0 msgr2=0x5625eae686d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:52.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.368+0000 7f8edd021740 1 --2- 172.21.15.12:0/2073410034 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625eae595f0 0x5625eae686d0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc009b30 tx=0x7f8ecc04b040 comp rx=0 tx=0).stop 2024-03-23T23:11:52.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.368+0000 7f8edd021740 1 -- 172.21.15.12:0/2073410034 shutdown_connections 2024-03-23T23:11:52.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.368+0000 7f8edd021740 1 --2- 172.21.15.12:0/2073410034 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5625eaecb7c0 0x5625eaecdbb0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.368+0000 7f8edd021740 1 --2- 172.21.15.12:0/2073410034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625eae5f150 0x5625eae69bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.368+0000 7f8edd021740 1 --2- 172.21.15.12:0/2073410034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625eae59fe0 0x5625eae68c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.368+0000 7f8edd021740 1 --2- 172.21.15.12:0/2073410034 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625eae595f0 0x5625eae686d0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.368+0000 7f8edd021740 1 -- 172.21.15.12:0/2073410034 >> 172.21.15.12:0/2073410034 conn(0x5625eae54b60 msgr2=0x5625eae5b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:52.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.368+0000 7f8ebd600640 0 client.28081 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:52.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.368+0000 7f8edd021740 1 -- 172.21.15.12:0/2073410034 shutdown_connections 2024-03-23T23:11:52.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.368+0000 7f8edd021740 1 -- 172.21.15.12:0/2073410034 wait complete. 2024-03-23T23:11:52.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.369+0000 7f8edd021740 20 client.28081 trim_cache size 0 max 16384 2024-03-23T23:11:52.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:11:52.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:11:52.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:11:52.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:11:52.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.369+0000 7f8eda400640 1 -- 172.21.15.12:0/2265652293 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8eb80a2920 msgr2=0x7f8eb80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:52.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.369+0000 7f8eda400640 1 --2- 172.21.15.12:0/2265652293 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8eb80a2920 0x7f8eb80a4de0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f8ec40ccb80 tx=0x7f8ec4004570 comp rx=0 tx=0).stop 2024-03-23T23:11:52.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.369+0000 7f8eda400640 1 -- 172.21.15.12:0/2265652293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ed4102140 msgr2=0x7f8ed4197890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:52.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.369+0000 7f8eda400640 1 --2- 172.21.15.12:0/2265652293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ed4102140 0x7f8ed4197890 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8ec400e9c0 tx=0x7f8ec400ee80 comp rx=0 tx=0).stop 2024-03-23T23:11:52.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.370+0000 7f8eda400640 1 -- 172.21.15.12:0/2265652293 shutdown_connections 2024-03-23T23:11:52.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.370+0000 7f8eda400640 1 --2- 172.21.15.12:0/2265652293 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8eb80a2920 0x7f8eb80a4de0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.370+0000 7f8eda400640 1 --2- 172.21.15.12:0/2265652293 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ed41073a0 0x7f8ed419ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.370+0000 7f8eda400640 1 --2- 172.21.15.12:0/2265652293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ed4104a70 0x7f8ed4197dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.370+0000 7f8eda400640 1 --2- 172.21.15.12:0/2265652293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ed4102140 0x7f8ed4197890 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:52.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.370+0000 7f8eda400640 1 -- 172.21.15.12:0/2265652293 >> 172.21.15.12:0/2265652293 conn(0x7f8ed40fbb30 msgr2=0x7f8ed4102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:52.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.370+0000 7f8eda400640 1 -- 172.21.15.12:0/2265652293 shutdown_connections 2024-03-23T23:11:52.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:11:52.370+0000 7f8eda400640 1 -- 172.21.15.12:0/2265652293 wait complete. 2024-03-23T23:11:53.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:52 smithi012 ceph-mon[31695]: pgmap v692: 97 pgs: 97 active+clean; 2.8 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:11:53.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:52 smithi173 ceph-mon[36116]: pgmap v692: 97 pgs: 97 active+clean; 2.8 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:11:53.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:52 smithi203 ceph-mon[36693]: pgmap v692: 97 pgs: 97 active+clean; 2.8 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:11:55.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:54 smithi012 ceph-mon[31695]: pgmap v693: 97 pgs: 97 active+clean; 2.8 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:11:55.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:54 smithi173 ceph-mon[36116]: pgmap v693: 97 pgs: 97 active+clean; 2.8 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:11:55.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:54 smithi203 ceph-mon[36693]: pgmap v693: 97 pgs: 97 active+clean; 2.8 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:11:56.509 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:11:56.510 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:11:56.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.747+0000 7fd703400640 1 -- 172.21.15.12:0/1228775806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6fc074100 msgr2=0x7fd6fc074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:56.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.747+0000 7fd703400640 1 --2- 172.21.15.12:0/1228775806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6fc074100 0x7fd6fc074580 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8009900 tx=0x7fd6e802bc30 comp rx=0 tx=0).stop 2024-03-23T23:11:56.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.747+0000 7fd703400640 1 -- 172.21.15.12:0/1228775806 shutdown_connections 2024-03-23T23:11:56.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.747+0000 7fd703400640 1 --2- 172.21.15.12:0/1228775806 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6fc10dc80 0x7fd6fc110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.747+0000 7fd703400640 1 --2- 172.21.15.12:0/1228775806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6fc074100 0x7fd6fc074580 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.747+0000 7fd703400640 1 --2- 172.21.15.12:0/1228775806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6fc0756e0 0x7fd6fc073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.747+0000 7fd703400640 1 -- 172.21.15.12:0/1228775806 >> 172.21.15.12:0/1228775806 conn(0x7fd6fc0fcc40 msgr2=0x7fd6fc0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:56.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.747+0000 7fd703400640 1 -- 172.21.15.12:0/1228775806 shutdown_connections 2024-03-23T23:11:56.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.747+0000 7fd703400640 1 -- 172.21.15.12:0/1228775806 wait complete. 2024-03-23T23:11:56.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.748+0000 7fd703400640 1 Processor -- start 2024-03-23T23:11:56.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.748+0000 7fd703400640 1 -- start start 2024-03-23T23:11:56.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.748+0000 7fd703400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6fc074100 0x7fd6fc19e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:56.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.748+0000 7fd703400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6fc0756e0 0x7fd6fc19e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:56.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.748+0000 7fd703400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6fc10dc80 0x7fd6fc1a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:56.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.748+0000 7fd703400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc113950 con 0x7fd6fc074100 2024-03-23T23:11:56.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.748+0000 7fd703400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc1137d0 con 0x7fd6fc0756e0 2024-03-23T23:11:56.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.748+0000 7fd703400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc113ad0 con 0x7fd6fc10dc80 2024-03-23T23:11:56.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.749+0000 7fd6fbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6fc0756e0 0x7fd6fc19e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:56.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.749+0000 7fd701400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6fc10dc80 0x7fd6fc1a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:56.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.749+0000 7fd700a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6fc074100 0x7fd6fc19e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:56.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.749+0000 7fd6fbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6fc0756e0 0x7fd6fc19e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:51466/0 (socket says 172.21.15.12:51466) 2024-03-23T23:11:56.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.749+0000 7fd6fbe00640 1 -- 172.21.15.12:0/2209254630 learned_addr learned my addr 172.21.15.12:0/2209254630 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:56.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.749+0000 7fd701400640 1 -- 172.21.15.12:0/2209254630 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6fc0756e0 msgr2=0x7fd6fc19e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:56.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.749+0000 7fd701400640 1 --2- 172.21.15.12:0/2209254630 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6fc0756e0 0x7fd6fc19e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.749+0000 7fd701400640 1 -- 172.21.15.12:0/2209254630 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6fc074100 msgr2=0x7fd6fc19e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:56.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.749+0000 7fd701400640 1 --2- 172.21.15.12:0/2209254630 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6fc074100 0x7fd6fc19e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.749+0000 7fd701400640 1 -- 172.21.15.12:0/2209254630 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6fc1a5c20 con 0x7fd6fc10dc80 2024-03-23T23:11:56.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.749+0000 7fd701400640 1 --2- 172.21.15.12:0/2209254630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6fc10dc80 0x7fd6fc1a55b0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4009fd0 tx=0x7fd6f400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:56.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.750+0000 7fd6f9600640 1 -- 172.21.15.12:0/2209254630 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f400ce20 con 0x7fd6fc10dc80 2024-03-23T23:11:56.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.750+0000 7fd6f9600640 1 -- 172.21.15.12:0/2209254630 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6f4005020 con 0x7fd6fc10dc80 2024-03-23T23:11:56.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.750+0000 7fd6f9600640 1 -- 172.21.15.12:0/2209254630 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f4010720 con 0x7fd6fc10dc80 2024-03-23T23:11:56.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.750+0000 7fd703400640 1 -- 172.21.15.12:0/2209254630 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6fc1a5f10 con 0x7fd6fc10dc80 2024-03-23T23:11:56.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.750+0000 7fd703400640 1 -- 172.21.15.12:0/2209254630 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6fc1a6380 con 0x7fd6fc10dc80 2024-03-23T23:11:56.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.753+0000 7fd6f9600640 1 -- 172.21.15.12:0/2209254630 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd6f4005540 con 0x7fd6fc10dc80 2024-03-23T23:11:56.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.753+0000 7fd706189740 1 Processor -- start 2024-03-23T23:11:56.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.753+0000 7fd706189740 1 -- start start 2024-03-23T23:11:56.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.754+0000 7fd706189740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9195e80 0x5608e91962a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:56.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.754+0000 7fd706189740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608e9196870 0x5608e919b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:56.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.754+0000 7fd706189740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608e919b9e0 0x5608e919de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:56.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.754+0000 7fd706189740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608e90fb250 con 0x5608e9196870 2024-03-23T23:11:56.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.754+0000 7fd706189740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608e90a3a30 con 0x5608e9195e80 2024-03-23T23:11:56.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.754+0000 7fd706189740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608e90f72f0 con 0x5608e919b9e0 2024-03-23T23:11:56.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.754+0000 7fd6fbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9195e80 0x5608e91962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:56.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.754+0000 7fd700a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608e9196870 0x5608e919b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:56.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.754+0000 7fd6fbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9195e80 0x5608e91962a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:51476/0 (socket says 172.21.15.12:51476) 2024-03-23T23:11:56.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.754+0000 7fd6fbe00640 1 -- 172.21.15.12:0/1871200148 learned_addr learned my addr 172.21.15.12:0/1871200148 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:56.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.754+0000 7fd6fbe00640 1 --2- 172.21.15.12:0/1871200148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608e919b9e0 0x5608e919de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:56.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.754+0000 7fd6fbe00640 1 -- 172.21.15.12:0/1871200148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608e919b9e0 msgr2=0x5608e919de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:56.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.754+0000 7fd6fbe00640 1 --2- 172.21.15.12:0/1871200148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608e919b9e0 0x5608e919de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.754+0000 7fd6fbe00640 1 -- 172.21.15.12:0/1871200148 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608e9196870 msgr2=0x5608e919b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:56.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.754+0000 7fd6fbe00640 1 --2- 172.21.15.12:0/1871200148 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608e9196870 0x5608e919b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.754+0000 7fd6fbe00640 1 -- 172.21.15.12:0/1871200148 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5608e908e690 con 0x5608e9195e80 2024-03-23T23:11:56.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.755+0000 7fd6fbe00640 1 --2- 172.21.15.12:0/1871200148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9195e80 0x5608e91962a0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fd6e80096c0 tx=0x7fd6e802bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95d916976752738b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:56.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.755+0000 7fd6f9600640 1 --2- 172.21.15.12:0/2209254630 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd6e40a2880 0x7fd6e40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:56.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.755+0000 7fd6edc00640 1 -- 172.21.15.12:0/1871200148 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8004230 con 0x5608e9195e80 2024-03-23T23:11:56.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.755+0000 7fd6edc00640 1 -- 172.21.15.12:0/1871200148 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6e802cd20 con 0x5608e9195e80 2024-03-23T23:11:56.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.755+0000 7fd6edc00640 1 -- 172.21.15.12:0/1871200148 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8005440 con 0x5608e9195e80 2024-03-23T23:11:56.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.755+0000 7fd6f9600640 1 -- 172.21.15.12:0/2209254630 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd6f40cf090 con 0x7fd6fc10dc80 2024-03-23T23:11:56.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.755+0000 7fd700a00640 1 --2- 172.21.15.12:0/2209254630 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd6e40a2880 0x7fd6e40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:56.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.756+0000 7fd706189740 1 -- 172.21.15.12:0/1871200148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9195e80 msgr2=0x5608e91962a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:56.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.756+0000 7fd706189740 1 --2- 172.21.15.12:0/1871200148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9195e80 0x5608e91962a0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fd6e80096c0 tx=0x7fd6e802bc30 comp rx=0 tx=0).stop 2024-03-23T23:11:56.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.756+0000 7fd706189740 1 -- 172.21.15.12:0/1871200148 shutdown_connections 2024-03-23T23:11:56.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.756+0000 7fd706189740 1 --2- 172.21.15.12:0/1871200148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608e919b9e0 0x5608e919de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.756+0000 7fd706189740 1 --2- 172.21.15.12:0/1871200148 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608e9196870 0x5608e919b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.756+0000 7fd706189740 1 --2- 172.21.15.12:0/1871200148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9195e80 0x5608e91962a0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.756+0000 7fd706189740 1 -- 172.21.15.12:0/1871200148 >> 172.21.15.12:0/1871200148 conn(0x5608e915e200 msgr2=0x5608e91917f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:56.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.756+0000 7fd706189740 1 -- 172.21.15.12:0/1871200148 shutdown_connections 2024-03-23T23:11:56.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.756+0000 7fd706189740 1 -- 172.21.15.12:0/1871200148 wait complete. 2024-03-23T23:11:56.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.756+0000 7fd700a00640 1 --2- 172.21.15.12:0/2209254630 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd6e40a2880 0x7fd6e40a4d40 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x5608e9196c70 tx=0x7fd6f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:56.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.756+0000 7fd706189740 1 -- start start 2024-03-23T23:11:56.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.757+0000 7fd706189740 1 Processor -- start 2024-03-23T23:11:56.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.757+0000 7fd706189740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608e9195e80 0x5608e919fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:56.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.757+0000 7fd706189740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9196870 0x5608e91a02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:56.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.757+0000 7fd706189740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608e919b9e0 0x5608e91a7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:56.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.757+0000 7fd706189740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608e919e930 con 0x5608e9195e80 2024-03-23T23:11:56.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.757+0000 7fd706189740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608e90b6e80 con 0x5608e9196870 2024-03-23T23:11:56.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.757+0000 7fd706189740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608e919eab0 con 0x5608e919b9e0 2024-03-23T23:11:56.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.757+0000 7fd706189740 10 client.0 _ll_register_callbacks cb 0x5608e90c9130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:56.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.757+0000 7fd700a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608e9195e80 0x5608e919fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:56.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.757+0000 7fd6fbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9196870 0x5608e91a02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:56.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.757+0000 7fd700a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608e9195e80 0x5608e919fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49894/0 (socket says 172.21.15.12:49894) 2024-03-23T23:11:56.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.757+0000 7fd700a00640 1 -- 172.21.15.12:0/4220510907 learned_addr learned my addr 172.21.15.12:0/4220510907 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:56.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.757+0000 7fd701400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608e919b9e0 0x5608e91a7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:56.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.758+0000 7fd700a00640 1 -- 172.21.15.12:0/4220510907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608e919b9e0 msgr2=0x5608e91a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:56.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.758+0000 7fd700a00640 1 --2- 172.21.15.12:0/4220510907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608e919b9e0 0x5608e91a7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.758+0000 7fd700a00640 1 -- 172.21.15.12:0/4220510907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9196870 msgr2=0x5608e91a02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:56.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.758+0000 7fd700a00640 1 --2- 172.21.15.12:0/4220510907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9196870 0x5608e91a02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.758+0000 7fd700a00640 1 -- 172.21.15.12:0/4220510907 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5608e8ed97d0 con 0x5608e9195e80 2024-03-23T23:11:56.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.758+0000 7fd6fbe00640 1 --2- 172.21.15.12:0/4220510907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9196870 0x5608e91a02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:56.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.758+0000 7fd701400640 1 --2- 172.21.15.12:0/4220510907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608e919b9e0 0x5608e91a7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:56.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.758+0000 7fd700a00640 1 --2- 172.21.15.12:0/4220510907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608e9195e80 0x5608e919fda0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd6f00045f0 tx=0x7fd6f0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32745e2ba1d43220 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:56.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.758+0000 7fd6e2000640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:11:56.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.758+0000 7fd6e2000640 1 -- 172.21.15.12:0/4220510907 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f001f640 con 0x5608e9195e80 2024-03-23T23:11:56.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.758+0000 7fd6e2000640 1 -- 172.21.15.12:0/4220510907 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd6f001f7e0 con 0x5608e9195e80 2024-03-23T23:11:56.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.759+0000 7fd706189740 1 -- 172.21.15.12:0/4220510907 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd6e802cd20 con 0x5608e9195e80 2024-03-23T23:11:56.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.759+0000 7fd6e2000640 1 -- 172.21.15.12:0/4220510907 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6f001f9e0 con 0x5608e9195e80 2024-03-23T23:11:56.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.759+0000 7fd706189740 10 client.17982 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:56.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.759+0000 7fd706189740 1 -- 172.21.15.12:0/4220510907 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5608e8e0f3f0 con 0x5608e9195e80 2024-03-23T23:11:56.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.759+0000 7fd6e2000640 1 -- 172.21.15.12:0/4220510907 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6e802cd20 con 0x5608e9195e80 2024-03-23T23:11:56.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.759+0000 7fd6e2000640 1 -- 172.21.15.12:0/4220510907 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fd6f0027070 con 0x5608e9195e80 2024-03-23T23:11:56.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.759+0000 7fd706189740 10 client.17982 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:56.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.759+0000 7fd706189740 10 client.17982 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:11:56.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.759+0000 7fd706189740 20 client.17982 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:56.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.759+0000 7fd706189740 1 --2- 172.21.15.12:0/4220510907 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5608e91abbb0 0x5608e91ae020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:56.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.759+0000 7fd706189740 4 client.17982 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:11:56.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.759+0000 7fd706189740 1 -- 172.21.15.12:0/4220510907 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5608e9108790 con 0x5608e91abbb0 2024-03-23T23:11:56.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.760+0000 7fd6fbe00640 1 --2- 172.21.15.12:0/4220510907 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5608e91abbb0 0x5608e91ae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:56.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.760+0000 7fd6fbe00640 1 --2- 172.21.15.12:0/4220510907 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5608e91abbb0 0x5608e91ae020 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b4b2a54ba64cdc1 server_cookie=4dcaf84bd9bc005d in_seq=0 out_seq=0 2024-03-23T23:11:56.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.760+0000 7fd6e2000640 10 client.17982 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:56.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.761+0000 7fd6e2000640 1 -- 172.21.15.12:0/4220510907 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5608e9108790 con 0x5608e91abbb0 2024-03-23T23:11:56.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.761+0000 7fd6e2000640 10 client.17982 handle_command_reply: tid=0 2024-03-23T23:11:56.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.761+0000 7fd706189740 1 client.17982 shutdown 2024-03-23T23:11:56.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.761+0000 7fd706189740 1 -- 172.21.15.12:0/4220510907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608e9195e80 msgr2=0x5608e919fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:56.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.761+0000 7fd706189740 1 --2- 172.21.15.12:0/4220510907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608e9195e80 0x5608e919fda0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd6f00045f0 tx=0x7fd6f0004730 comp rx=0 tx=0).stop 2024-03-23T23:11:56.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.761+0000 7fd706189740 1 -- 172.21.15.12:0/4220510907 shutdown_connections 2024-03-23T23:11:56.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.761+0000 7fd706189740 1 --2- 172.21.15.12:0/4220510907 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5608e91abbb0 0x5608e91ae020 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.761+0000 7fd706189740 1 --2- 172.21.15.12:0/4220510907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608e919b9e0 0x5608e91a7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.761+0000 7fd706189740 1 --2- 172.21.15.12:0/4220510907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9196870 0x5608e91a02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.761+0000 7fd6e2000640 0 client.17982 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:56.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.761+0000 7fd706189740 1 --2- 172.21.15.12:0/4220510907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608e9195e80 0x5608e919fda0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.761+0000 7fd706189740 1 -- 172.21.15.12:0/4220510907 >> 172.21.15.12:0/4220510907 conn(0x5608e915e200 msgr2=0x5608e9190a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:56.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.761+0000 7fd706189740 1 -- 172.21.15.12:0/4220510907 shutdown_connections 2024-03-23T23:11:56.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.761+0000 7fd706189740 1 -- 172.21.15.12:0/4220510907 wait complete. 2024-03-23T23:11:56.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.762+0000 7fd706189740 20 client.17982 trim_cache size 0 max 16384 2024-03-23T23:11:56.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.779+0000 7fd706189740 1 Processor -- start 2024-03-23T23:11:56.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.779+0000 7fd706189740 1 -- start start 2024-03-23T23:11:56.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.780+0000 7fd706189740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9195e80 0x5608e91a0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:56.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.780+0000 7fd706189740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608e9196870 0x5608e91a72b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:56.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.780+0000 7fd706189740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608e919b9e0 0x5608e91979d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:56.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.780+0000 7fd706189740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc1b1570 con 0x5608e9196870 2024-03-23T23:11:56.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.780+0000 7fd706189740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc1af070 con 0x5608e9195e80 2024-03-23T23:11:56.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.780+0000 7fd706189740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc1ad110 con 0x5608e919b9e0 2024-03-23T23:11:56.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.780+0000 7fd700a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9195e80 0x5608e91a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:56.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.780+0000 7fd701400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608e919b9e0 0x5608e91979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:56.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.780+0000 7fd700a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9195e80 0x5608e91a0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:51482/0 (socket says 172.21.15.12:51482) 2024-03-23T23:11:56.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.780+0000 7fd700a00640 1 -- 172.21.15.12:0/4288764941 learned_addr learned my addr 172.21.15.12:0/4288764941 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:56.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.780+0000 7fd701400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608e919b9e0 0x5608e91979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:52538/0 (socket says 172.21.15.12:52538) 2024-03-23T23:11:56.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.780+0000 7fd6fbe00640 1 --2- 172.21.15.12:0/4288764941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608e9196870 0x5608e91a72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:56.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.780+0000 7fd700a00640 1 -- 172.21.15.12:0/4288764941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608e919b9e0 msgr2=0x5608e91979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:56.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.780+0000 7fd700a00640 1 --2- 172.21.15.12:0/4288764941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608e919b9e0 0x5608e91979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.780+0000 7fd700a00640 1 -- 172.21.15.12:0/4288764941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608e9196870 msgr2=0x5608e91a72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:56.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.780+0000 7fd700a00640 1 --2- 172.21.15.12:0/4288764941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608e9196870 0x5608e91a72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.780+0000 7fd700a00640 1 -- 172.21.15.12:0/4288764941 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5608e8dcf0f0 con 0x5608e9195e80 2024-03-23T23:11:56.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.780+0000 7fd6fbe00640 1 --2- 172.21.15.12:0/4288764941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608e9196870 0x5608e91a72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:11:56.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.781+0000 7fd700a00640 1 --2- 172.21.15.12:0/4288764941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9195e80 0x5608e91a0240 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0004570 tx=0x7fd6f0009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2132a00e008a1366 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:56.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.781+0000 7fd6ed200640 1 -- 172.21.15.12:0/4288764941 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f001f840 con 0x5608e9195e80 2024-03-23T23:11:56.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.781+0000 7fd6ed200640 1 -- 172.21.15.12:0/4288764941 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6f0037070 con 0x5608e9195e80 2024-03-23T23:11:56.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.781+0000 7fd6ed200640 1 -- 172.21.15.12:0/4288764941 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f0004ba0 con 0x5608e9195e80 2024-03-23T23:11:56.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.781+0000 7fd706189740 1 -- 172.21.15.12:0/4288764941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9195e80 msgr2=0x5608e91a0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:56.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.781+0000 7fd706189740 1 --2- 172.21.15.12:0/4288764941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9195e80 0x5608e91a0240 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0004570 tx=0x7fd6f0009c00 comp rx=0 tx=0).stop 2024-03-23T23:11:56.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.782+0000 7fd706189740 1 -- 172.21.15.12:0/4288764941 shutdown_connections 2024-03-23T23:11:56.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.782+0000 7fd706189740 1 --2- 172.21.15.12:0/4288764941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608e919b9e0 0x5608e91979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.782+0000 7fd706189740 1 --2- 172.21.15.12:0/4288764941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608e9196870 0x5608e91a72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.782+0000 7fd706189740 1 --2- 172.21.15.12:0/4288764941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9195e80 0x5608e91a0240 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.782+0000 7fd706189740 1 -- 172.21.15.12:0/4288764941 >> 172.21.15.12:0/4288764941 conn(0x5608e915e200 msgr2=0x5608e91972c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:56.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.782+0000 7fd706189740 1 -- 172.21.15.12:0/4288764941 shutdown_connections 2024-03-23T23:11:56.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.782+0000 7fd706189740 1 -- 172.21.15.12:0/4288764941 wait complete. 2024-03-23T23:11:56.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.782+0000 7fd706189740 1 -- start start 2024-03-23T23:11:56.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.782+0000 7fd706189740 1 Processor -- start 2024-03-23T23:11:56.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.783+0000 7fd706189740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608e9195e80 0x5608e91a4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:56.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.783+0000 7fd706189740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9196870 0x5608e91a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:56.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.783+0000 7fd706189740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608e919b9e0 0x5608e91a5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:56.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.783+0000 7fd706189740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc1ad710 con 0x5608e919b9e0 2024-03-23T23:11:56.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.783+0000 7fd706189740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc1aea10 con 0x5608e9196870 2024-03-23T23:11:56.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.783+0000 7fd706189740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc1add50 con 0x5608e9195e80 2024-03-23T23:11:56.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.783+0000 7fd706189740 10 client.0 _ll_register_callbacks cb 0x7fd6f0009ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:56.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.783+0000 7fd700a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608e9195e80 0x5608e91a4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:56.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.783+0000 7fd700a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608e9195e80 0x5608e91a4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:52552/0 (socket says 172.21.15.12:52552) 2024-03-23T23:11:56.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.783+0000 7fd700a00640 1 -- 172.21.15.12:0/3809271822 learned_addr learned my addr 172.21.15.12:0/3809271822 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:56.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.783+0000 7fd6fbe00640 1 --2- 172.21.15.12:0/3809271822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9196870 0x5608e91a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:56.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.783+0000 7fd701400640 1 --2- 172.21.15.12:0/3809271822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608e919b9e0 0x5608e91a5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:56.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.783+0000 7fd700a00640 1 -- 172.21.15.12:0/3809271822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9196870 msgr2=0x5608e91a4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:56.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.783+0000 7fd700a00640 1 --2- 172.21.15.12:0/3809271822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9196870 0x5608e91a4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.783+0000 7fd700a00640 1 -- 172.21.15.12:0/3809271822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608e919b9e0 msgr2=0x5608e91a5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:56.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.783+0000 7fd700a00640 1 --2- 172.21.15.12:0/3809271822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608e919b9e0 0x5608e91a5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.783+0000 7fd700a00640 1 -- 172.21.15.12:0/3809271822 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5608e91a6450 con 0x5608e9195e80 2024-03-23T23:11:56.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.783+0000 7fd6fbe00640 1 --2- 172.21.15.12:0/3809271822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9196870 0x5608e91a4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:56.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.784+0000 7fd701400640 1 --2- 172.21.15.12:0/3809271822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608e919b9e0 0x5608e91a5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:11:56.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.784+0000 7fd700a00640 1 --2- 172.21.15.12:0/3809271822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608e9195e80 0x5608e91a4910 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0012d20 tx=0x7fd6f0004470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45cfc0937d9a7358 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:56.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.784+0000 7fd6e2000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:11:56.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.784+0000 7fd6e2000640 1 -- 172.21.15.12:0/3809271822 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f001d600 con 0x5608e9195e80 2024-03-23T23:11:56.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.784+0000 7fd6e2000640 1 -- 172.21.15.12:0/3809271822 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd6f001d7a0 con 0x5608e9195e80 2024-03-23T23:11:56.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.784+0000 7fd706189740 1 -- 172.21.15.12:0/3809271822 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd6f0037070 con 0x5608e9195e80 2024-03-23T23:11:56.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.784+0000 7fd6e2000640 1 -- 172.21.15.12:0/3809271822 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6f001d9a0 con 0x5608e9195e80 2024-03-23T23:11:56.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.784+0000 7fd706189740 10 client.28099 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:56.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.784+0000 7fd706189740 1 -- 172.21.15.12:0/3809271822 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5608e92074b0 con 0x5608e9195e80 2024-03-23T23:11:56.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.785+0000 7fd6e2000640 1 -- 172.21.15.12:0/3809271822 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6f0037070 con 0x5608e9195e80 2024-03-23T23:11:56.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.785+0000 7fd6e2000640 1 -- 172.21.15.12:0/3809271822 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fd6f0027050 con 0x5608e9195e80 2024-03-23T23:11:56.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.785+0000 7fd706189740 10 client.28099 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:56.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.785+0000 7fd706189740 10 client.28099 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:11:56.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.785+0000 7fd706189740 20 client.28099 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:56.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.785+0000 7fd706189740 1 --2- 172.21.15.12:0/3809271822 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5608e9207c50 0x5608e920a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:56.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.785+0000 7fd706189740 4 client.28099 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:11:56.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.785+0000 7fd706189740 1 -- 172.21.15.12:0/3809271822 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5608e9102ae0 con 0x5608e9207c50 2024-03-23T23:11:56.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.785+0000 7fd6fbe00640 1 --2- 172.21.15.12:0/3809271822 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5608e9207c50 0x5608e920a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:56.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.786+0000 7fd6fbe00640 1 --2- 172.21.15.12:0/3809271822 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5608e9207c50 0x5608e920a070 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a8ec71c7d2ae9b2 server_cookie=80f1e4bb8dfa6082 in_seq=0 out_seq=0 2024-03-23T23:11:56.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.786+0000 7fd6e2000640 10 client.28099 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:56.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.786+0000 7fd6e2000640 1 -- 172.21.15.12:0/3809271822 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5608e9102ae0 con 0x5608e9207c50 2024-03-23T23:11:56.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.786+0000 7fd6e2000640 10 client.28099 handle_command_reply: tid=0 2024-03-23T23:11:56.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.786+0000 7fd706189740 1 client.28099 shutdown 2024-03-23T23:11:56.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.786+0000 7fd706189740 1 -- 172.21.15.12:0/3809271822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608e9195e80 msgr2=0x5608e91a4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:56.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.786+0000 7fd706189740 1 --2- 172.21.15.12:0/3809271822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608e9195e80 0x5608e91a4910 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0012d20 tx=0x7fd6f0004470 comp rx=0 tx=0).stop 2024-03-23T23:11:56.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.786+0000 7fd706189740 1 -- 172.21.15.12:0/3809271822 shutdown_connections 2024-03-23T23:11:56.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.786+0000 7fd706189740 1 --2- 172.21.15.12:0/3809271822 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5608e9207c50 0x5608e920a070 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.786+0000 7fd706189740 1 --2- 172.21.15.12:0/3809271822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608e919b9e0 0x5608e91a5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.786+0000 7fd706189740 1 --2- 172.21.15.12:0/3809271822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608e9196870 0x5608e91a4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.786+0000 7fd706189740 1 --2- 172.21.15.12:0/3809271822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608e9195e80 0x5608e91a4910 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.786+0000 7fd6e2000640 0 client.28099 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:56.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.786+0000 7fd706189740 1 -- 172.21.15.12:0/3809271822 >> 172.21.15.12:0/3809271822 conn(0x5608e915e200 msgr2=0x5608e9190a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:56.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.786+0000 7fd706189740 1 -- 172.21.15.12:0/3809271822 shutdown_connections 2024-03-23T23:11:56.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.786+0000 7fd706189740 1 -- 172.21.15.12:0/3809271822 wait complete. 2024-03-23T23:11:56.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.786+0000 7fd706189740 20 client.28099 trim_cache size 0 max 16384 2024-03-23T23:11:56.790 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:11:56.790 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-03-23T23:11:56.790 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-03-23T23:11:56.790 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:11:56.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.787+0000 7fd703400640 1 -- 172.21.15.12:0/2209254630 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd6e40a2880 msgr2=0x7fd6e40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:56.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.787+0000 7fd703400640 1 --2- 172.21.15.12:0/2209254630 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd6e40a2880 0x7fd6e40a4d40 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x5608e9196c70 tx=0x7fd6f0005f90 comp rx=0 tx=0).stop 2024-03-23T23:11:56.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.787+0000 7fd703400640 1 -- 172.21.15.12:0/2209254630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6fc10dc80 msgr2=0x7fd6fc1a55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:56.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.787+0000 7fd703400640 1 --2- 172.21.15.12:0/2209254630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6fc10dc80 0x7fd6fc1a55b0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4009fd0 tx=0x7fd6f400ef10 comp rx=0 tx=0).stop 2024-03-23T23:11:56.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.787+0000 7fd703400640 1 -- 172.21.15.12:0/2209254630 shutdown_connections 2024-03-23T23:11:56.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.787+0000 7fd703400640 1 --2- 172.21.15.12:0/2209254630 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd6e40a2880 0x7fd6e40a4d40 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.787+0000 7fd703400640 1 --2- 172.21.15.12:0/2209254630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6fc10dc80 0x7fd6fc1a55b0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.787+0000 7fd703400640 1 --2- 172.21.15.12:0/2209254630 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6fc0756e0 0x7fd6fc19e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.787+0000 7fd703400640 1 --2- 172.21.15.12:0/2209254630 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6fc074100 0x7fd6fc19e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:56.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.787+0000 7fd703400640 1 -- 172.21.15.12:0/2209254630 >> 172.21.15.12:0/2209254630 conn(0x7fd6fc0fcc40 msgr2=0x7fd6fc101020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:56.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.787+0000 7fd703400640 1 -- 172.21.15.12:0/2209254630 shutdown_connections 2024-03-23T23:11:56.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:56.787+0000 7fd703400640 1 -- 172.21.15.12:0/2209254630 wait complete. 2024-03-23T23:11:56.804 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T23:11:56.804 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:11:57.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.027+0000 7f97f5000640 1 -- 172.21.15.12:0/1191169012 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97f01052a0 msgr2=0x7f97f010bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:57.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.027+0000 7f97f5000640 1 --2- 172.21.15.12:0/1191169012 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97f01052a0 0x7f97f010bb30 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f97e8009900 tx=0x7f97e802bc40 comp rx=0 tx=0).stop 2024-03-23T23:11:57.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.028+0000 7f97f5000640 1 -- 172.21.15.12:0/1191169012 shutdown_connections 2024-03-23T23:11:57.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.028+0000 7f97f5000640 1 --2- 172.21.15.12:0/1191169012 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97f01052a0 0x7f97f010bb30 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.028+0000 7f97f5000640 1 --2- 172.21.15.12:0/1191169012 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97f01048e0 0x7f97f0104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.028+0000 7f97f5000640 1 --2- 172.21.15.12:0/1191169012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97f01036f0 0x7f97f0103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.028+0000 7f97f5000640 1 -- 172.21.15.12:0/1191169012 >> 172.21.15.12:0/1191169012 conn(0x7f97f00fec80 msgr2=0x7f97f01010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:57.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.028+0000 7f97f5000640 1 -- 172.21.15.12:0/1191169012 shutdown_connections 2024-03-23T23:11:57.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.028+0000 7f97f5000640 1 -- 172.21.15.12:0/1191169012 wait complete. 2024-03-23T23:11:57.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.029+0000 7f97f5000640 1 Processor -- start 2024-03-23T23:11:57.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.029+0000 7f97f5000640 1 -- start start 2024-03-23T23:11:57.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.029+0000 7f97f5000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97f01036f0 0x7f97f01999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.029+0000 7f97f5000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97f01048e0 0x7f97f0199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.029+0000 7f97f5000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97f01052a0 0x7f97f01a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.029+0000 7f97f5000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f010f3c0 con 0x7f97f01036f0 2024-03-23T23:11:57.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.029+0000 7f97f5000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f010f240 con 0x7f97f01048e0 2024-03-23T23:11:57.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.029+0000 7f97f5000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f010f540 con 0x7f97f01052a0 2024-03-23T23:11:57.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.029+0000 7f97efe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97f01052a0 0x7f97f01a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.029+0000 7f97e6a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97f01048e0 0x7f97f0199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.029+0000 7f97ef400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97f01036f0 0x7f97f01999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.029+0000 7f97efe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97f01052a0 0x7f97f01a0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:52564/0 (socket says 172.21.15.12:52564) 2024-03-23T23:11:57.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.029+0000 7f97efe00640 1 -- 172.21.15.12:0/1766765918 learned_addr learned my addr 172.21.15.12:0/1766765918 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:57.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.030+0000 7f97e6a00640 1 -- 172.21.15.12:0/1766765918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97f01052a0 msgr2=0x7f97f01a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:57.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.030+0000 7f97e6a00640 1 --2- 172.21.15.12:0/1766765918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97f01052a0 0x7f97f01a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.030+0000 7f97e6a00640 1 -- 172.21.15.12:0/1766765918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97f01036f0 msgr2=0x7f97f01999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:57.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.030+0000 7f97e6a00640 1 --2- 172.21.15.12:0/1766765918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97f01036f0 0x7f97f01999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.030+0000 7f97e6a00640 1 -- 172.21.15.12:0/1766765918 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f01a1600 con 0x7f97f01048e0 2024-03-23T23:11:57.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.030+0000 7f97efe00640 1 --2- 172.21.15.12:0/1766765918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97f01052a0 0x7f97f01a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:57.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.030+0000 7f97ef400640 1 --2- 172.21.15.12:0/1766765918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97f01036f0 0x7f97f01999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:57.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.030+0000 7f97e6a00640 1 --2- 172.21.15.12:0/1766765918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97f01048e0 0x7f97f0199f20 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f97d800e9f0 tx=0x7f97d800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:57.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.030+0000 7f97ecc00640 1 -- 172.21.15.12:0/1766765918 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d800ccc0 con 0x7f97f01048e0 2024-03-23T23:11:57.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.030+0000 7f97ecc00640 1 -- 172.21.15.12:0/1766765918 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f97d8010910 con 0x7f97f01048e0 2024-03-23T23:11:57.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.030+0000 7f97ecc00640 1 -- 172.21.15.12:0/1766765918 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d8018960 con 0x7f97f01048e0 2024-03-23T23:11:57.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.030+0000 7f97f5000640 1 -- 172.21.15.12:0/1766765918 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f01a18f0 con 0x7f97f01048e0 2024-03-23T23:11:57.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.031+0000 7f97f5000640 1 -- 172.21.15.12:0/1766765918 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f01a1df0 con 0x7f97f01048e0 2024-03-23T23:11:57.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.032+0000 7f97f5000640 1 -- 172.21.15.12:0/1766765918 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f010a9b0 con 0x7f97f01048e0 2024-03-23T23:11:57.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.034+0000 7f97ecc00640 1 -- 172.21.15.12:0/1766765918 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f97d8010ac0 con 0x7f97f01048e0 2024-03-23T23:11:57.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.035+0000 7f97ecc00640 1 --2- 172.21.15.12:0/1766765918 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f97d40a2800 0x7f97d40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.035+0000 7f97ecc00640 1 -- 172.21.15.12:0/1766765918 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f97d80cee40 con 0x7f97f01048e0 2024-03-23T23:11:57.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.035+0000 7f97ef400640 1 --2- 172.21.15.12:0/1766765918 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f97d40a2800 0x7f97d40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.036+0000 7f97ef400640 1 --2- 172.21.15.12:0/1766765918 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f97d40a2800 0x7f97d40a4cc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f97f0104740 tx=0x7f97e0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:57.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.037+0000 7f97ecc00640 1 -- 172.21.15.12:0/1766765918 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f97d809b840 con 0x7f97f01048e0 2024-03-23T23:11:57.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.282+0000 7f97f5000640 1 -- 172.21.15.12:0/1766765918 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97f002d230 con 0x7f97f01048e0 2024-03-23T23:11:57.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.283+0000 7f97ecc00640 1 -- 172.21.15.12:0/1766765918 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f97d809f8e0 con 0x7f97f01048e0 2024-03-23T23:11:57.283 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:11:57.283 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:11:57.283 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:11:57.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.286+0000 7f97f5000640 1 -- 172.21.15.12:0/1766765918 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f97d40a2800 msgr2=0x7f97d40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:57.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.286+0000 7f97f5000640 1 --2- 172.21.15.12:0/1766765918 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f97d40a2800 0x7f97d40a4cc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f97f0104740 tx=0x7f97e0009500 comp rx=0 tx=0).stop 2024-03-23T23:11:57.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.286+0000 7f97f5000640 1 -- 172.21.15.12:0/1766765918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97f01048e0 msgr2=0x7f97f0199f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:57.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.286+0000 7f97f5000640 1 --2- 172.21.15.12:0/1766765918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97f01048e0 0x7f97f0199f20 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f97d800e9f0 tx=0x7f97d800eeb0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.286+0000 7f97f5000640 1 -- 172.21.15.12:0/1766765918 shutdown_connections 2024-03-23T23:11:57.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.286+0000 7f97f5000640 1 --2- 172.21.15.12:0/1766765918 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f97d40a2800 0x7f97d40a4cc0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.287+0000 7f97f5000640 1 --2- 172.21.15.12:0/1766765918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97f01052a0 0x7f97f01a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.287+0000 7f97f5000640 1 --2- 172.21.15.12:0/1766765918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97f01048e0 0x7f97f0199f20 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.287+0000 7f97f5000640 1 --2- 172.21.15.12:0/1766765918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97f01036f0 0x7f97f01999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.287+0000 7f97f5000640 1 -- 172.21.15.12:0/1766765918 >> 172.21.15.12:0/1766765918 conn(0x7f97f00fec80 msgr2=0x7f97f0107b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:57.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.287+0000 7f97f5000640 1 -- 172.21.15.12:0/1766765918 shutdown_connections 2024-03-23T23:11:57.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.287+0000 7f97f5000640 1 -- 172.21.15.12:0/1766765918 wait complete. 2024-03-23T23:11:57.302 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:56 smithi012 ceph-mon[31695]: pgmap v694: 97 pgs: 97 active+clean; 2.8 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:11:57.305 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:11:57.305 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-23T23:11:57.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:56 smithi173 ceph-mon[36116]: pgmap v694: 97 pgs: 97 active+clean; 2.8 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:11:57.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:56 smithi203 ceph-mon[36693]: pgmap v694: 97 pgs: 97 active+clean; 2.8 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:11:57.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.525+0000 7ff8d8800640 1 -- 172.21.15.12:0/2233542154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8d0107cc0 msgr2=0x7ff8d01080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:57.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.525+0000 7ff8d8800640 1 --2- 172.21.15.12:0/2233542154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8d0107cc0 0x7ff8d01080e0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4009900 tx=0x7ff8c402bc40 comp rx=0 tx=0).stop 2024-03-23T23:11:57.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.525+0000 7ff8d8800640 1 -- 172.21.15.12:0/2233542154 shutdown_connections 2024-03-23T23:11:57.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.525+0000 7ff8d8800640 1 --2- 172.21.15.12:0/2233542154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8d0109870 0x7ff8d0110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.525+0000 7ff8d8800640 1 --2- 172.21.15.12:0/2233542154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8d0108eb0 0x7ff8d0109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.525+0000 7ff8d8800640 1 --2- 172.21.15.12:0/2233542154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8d0107cc0 0x7ff8d01080e0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.525+0000 7ff8d8800640 1 -- 172.21.15.12:0/2233542154 >> 172.21.15.12:0/2233542154 conn(0x7ff8d0076c60 msgr2=0x7ff8d00790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:57.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.526+0000 7ff8d8800640 1 -- 172.21.15.12:0/2233542154 shutdown_connections 2024-03-23T23:11:57.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.526+0000 7ff8d8800640 1 -- 172.21.15.12:0/2233542154 wait complete. 2024-03-23T23:11:57.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.527+0000 7ff8d8800640 1 Processor -- start 2024-03-23T23:11:57.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.527+0000 7ff8d8800640 1 -- start start 2024-03-23T23:11:57.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.527+0000 7ff8d8800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8d0107cc0 0x7ff8d019dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.527+0000 7ff8d8800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8d0108eb0 0x7ff8d019e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.527+0000 7ff8d8800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8d0109870 0x7ff8d01a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.527+0000 7ff8d8800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d0113990 con 0x7ff8d0109870 2024-03-23T23:11:57.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.527+0000 7ff8d8800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d0113810 con 0x7ff8d0107cc0 2024-03-23T23:11:57.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.527+0000 7ff8d8800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d0113b10 con 0x7ff8d0108eb0 2024-03-23T23:11:57.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.527+0000 7ff8d5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8d0107cc0 0x7ff8d019dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.527+0000 7ff8d5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8d0107cc0 0x7ff8d019dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:51522/0 (socket says 172.21.15.12:51522) 2024-03-23T23:11:57.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.527+0000 7ff8d5e00640 1 -- 172.21.15.12:0/2858051655 learned_addr learned my addr 172.21.15.12:0/2858051655 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:57.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.527+0000 7ff8d5400640 1 --2- 172.21.15.12:0/2858051655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8d0108eb0 0x7ff8d019e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.527+0000 7ff8d6800640 1 --2- 172.21.15.12:0/2858051655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8d0109870 0x7ff8d01a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.528+0000 7ff8d5e00640 1 -- 172.21.15.12:0/2858051655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8d0108eb0 msgr2=0x7ff8d019e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:57.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.528+0000 7ff8d5e00640 1 --2- 172.21.15.12:0/2858051655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8d0108eb0 0x7ff8d019e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.528+0000 7ff8d5e00640 1 -- 172.21.15.12:0/2858051655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8d0109870 msgr2=0x7ff8d01a5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:57.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.528+0000 7ff8d5e00640 1 --2- 172.21.15.12:0/2858051655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8d0109870 0x7ff8d01a5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.528+0000 7ff8d5e00640 1 -- 172.21.15.12:0/2858051655 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d01a5bf0 con 0x7ff8d0107cc0 2024-03-23T23:11:57.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.528+0000 7ff8d5400640 1 --2- 172.21.15.12:0/2858051655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8d0108eb0 0x7ff8d019e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:57.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.528+0000 7ff8d5e00640 1 --2- 172.21.15.12:0/2858051655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8d0107cc0 0x7ff8d019dfd0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7ff8c402efd0 tx=0x7ff8c4005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:57.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.528+0000 7ff8caa00640 1 -- 172.21.15.12:0/2858051655 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c4041070 con 0x7ff8d0107cc0 2024-03-23T23:11:57.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.528+0000 7ff8caa00640 1 -- 172.21.15.12:0/2858051655 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8c40044d0 con 0x7ff8d0107cc0 2024-03-23T23:11:57.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.528+0000 7ff8caa00640 1 -- 172.21.15.12:0/2858051655 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c4034440 con 0x7ff8d0107cc0 2024-03-23T23:11:57.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.529+0000 7ff8d8800640 1 -- 172.21.15.12:0/2858051655 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d01a5e80 con 0x7ff8d0107cc0 2024-03-23T23:11:57.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.529+0000 7ff8d8800640 1 -- 172.21.15.12:0/2858051655 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d01a6230 con 0x7ff8d0107cc0 2024-03-23T23:11:57.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.532+0000 7ff8caa00640 1 -- 172.21.15.12:0/2858051655 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff8c4044070 con 0x7ff8d0107cc0 2024-03-23T23:11:57.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.532+0000 7ff8db427740 1 Processor -- start 2024-03-23T23:11:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.532+0000 7ff8db427740 1 -- start start 2024-03-23T23:11:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.532+0000 7ff8db427740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c8f4fa810 0x564c8f4fac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.532+0000 7ff8db427740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c8f4fb9e0 0x564c8f4fbe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.532+0000 7ff8db427740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564c8f4fc3a0 0x564c8f502c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.532+0000 7ff8db427740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c8f45f2c0 con 0x564c8f4fc3a0 2024-03-23T23:11:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.532+0000 7ff8db427740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c8f407c90 con 0x564c8f4fa810 2024-03-23T23:11:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.532+0000 7ff8db427740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c8f45b360 con 0x564c8f4fb9e0 2024-03-23T23:11:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.532+0000 7ff8d5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c8f4fb9e0 0x564c8f4fbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.532+0000 7ff8d6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c8f4fa810 0x564c8f4fac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.532+0000 7ff8d6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c8f4fa810 0x564c8f4fac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:51528/0 (socket says 172.21.15.12:51528) 2024-03-23T23:11:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.533+0000 7ff8d6800640 1 -- 172.21.15.12:0/1677181331 learned_addr learned my addr 172.21.15.12:0/1677181331 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.533+0000 7ff8d5400640 1 --2- 172.21.15.12:0/1677181331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564c8f4fc3a0 0x564c8f502c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.533+0000 7ff8d6800640 1 -- 172.21.15.12:0/1677181331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c8f4fb9e0 msgr2=0x564c8f4fbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:57.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.533+0000 7ff8d6800640 1 --2- 172.21.15.12:0/1677181331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c8f4fb9e0 0x564c8f4fbe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.533+0000 7ff8d6800640 1 -- 172.21.15.12:0/1677181331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564c8f4fc3a0 msgr2=0x564c8f502c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:57.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.533+0000 7ff8d6800640 1 --2- 172.21.15.12:0/1677181331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564c8f4fc3a0 0x564c8f502c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.533+0000 7ff8d6800640 1 -- 172.21.15.12:0/1677181331 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c8f42c080 con 0x564c8f4fa810 2024-03-23T23:11:57.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.533+0000 7ff8d6800640 1 --2- 172.21.15.12:0/1677181331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c8f4fa810 0x564c8f4fac30 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc009f50 tx=0x7ff8cc0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc0c2c24161f44ba server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:57.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.533+0000 7ff8bf000640 1 -- 172.21.15.12:0/1677181331 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc004d20 con 0x564c8f4fa810 2024-03-23T23:11:57.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.533+0000 7ff8bf000640 1 -- 172.21.15.12:0/1677181331 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8cc004050 con 0x564c8f4fa810 2024-03-23T23:11:57.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.533+0000 7ff8bf000640 1 -- 172.21.15.12:0/1677181331 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc005280 con 0x564c8f4fa810 2024-03-23T23:11:57.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.533+0000 7ff8caa00640 1 --2- 172.21.15.12:0/2858051655 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff8b80a2880 0x7ff8b80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.534+0000 7ff8caa00640 1 -- 172.21.15.12:0/2858051655 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff8c40f0550 con 0x7ff8d0107cc0 2024-03-23T23:11:57.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.534+0000 7ff8d5e00640 1 --2- 172.21.15.12:0/2858051655 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff8b80a2880 0x7ff8b80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.534+0000 7ff8db427740 1 -- 172.21.15.12:0/1677181331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c8f4fa810 msgr2=0x564c8f4fac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:57.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.534+0000 7ff8db427740 1 --2- 172.21.15.12:0/1677181331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c8f4fa810 0x564c8f4fac30 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc009f50 tx=0x7ff8cc0118d0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.534+0000 7ff8db427740 1 -- 172.21.15.12:0/1677181331 shutdown_connections 2024-03-23T23:11:57.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.534+0000 7ff8db427740 1 --2- 172.21.15.12:0/1677181331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564c8f4fc3a0 0x564c8f502c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.534+0000 7ff8db427740 1 --2- 172.21.15.12:0/1677181331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c8f4fb9e0 0x564c8f4fbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.534+0000 7ff8db427740 1 --2- 172.21.15.12:0/1677181331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c8f4fa810 0x564c8f4fac30 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.534+0000 7ff8db427740 1 -- 172.21.15.12:0/1677181331 >> 172.21.15.12:0/1677181331 conn(0x564c8f4f5c00 msgr2=0x564c8f4f6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:57.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.534+0000 7ff8db427740 1 -- 172.21.15.12:0/1677181331 shutdown_connections 2024-03-23T23:11:57.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.534+0000 7ff8db427740 1 -- 172.21.15.12:0/1677181331 wait complete. 2024-03-23T23:11:57.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.534+0000 7ff8d5e00640 1 --2- 172.21.15.12:0/2858051655 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff8b80a2880 0x7ff8b80a4d40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40b93b0 tx=0x7ff8c40f3f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:57.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.535+0000 7ff8db427740 1 -- start start 2024-03-23T23:11:57.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.535+0000 7ff8db427740 1 Processor -- start 2024-03-23T23:11:57.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.535+0000 7ff8db427740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c8f4fa810 0x564c8f504ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.535+0000 7ff8db427740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564c8f4fb9e0 0x564c8f504ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.535+0000 7ff8db427740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c8f4fc3a0 0x564c8f50c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.535+0000 7ff8db427740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c8f5033b0 con 0x564c8f4fb9e0 2024-03-23T23:11:57.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.535+0000 7ff8db427740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c8f41b0d0 con 0x564c8f4fa810 2024-03-23T23:11:57.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.535+0000 7ff8db427740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c8f503530 con 0x564c8f4fc3a0 2024-03-23T23:11:57.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.535+0000 7ff8db427740 10 client.0 _ll_register_callbacks cb 0x564c8f41fe80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:57.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.536+0000 7ff8d6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c8f4fa810 0x564c8f504ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.536+0000 7ff8d5400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564c8f4fb9e0 0x564c8f504ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.536+0000 7ff8d5400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564c8f4fb9e0 0x564c8f504ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49968/0 (socket says 172.21.15.12:49968) 2024-03-23T23:11:57.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.536+0000 7ff8d5400640 1 -- 172.21.15.12:0/2930512897 learned_addr learned my addr 172.21.15.12:0/2930512897 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:57.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.536+0000 7ff8d6800640 1 --2- 172.21.15.12:0/2930512897 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c8f4fc3a0 0x564c8f50c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.536+0000 7ff8d5400640 1 -- 172.21.15.12:0/2930512897 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c8f4fc3a0 msgr2=0x564c8f50c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:57.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.536+0000 7ff8d5400640 1 --2- 172.21.15.12:0/2930512897 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c8f4fc3a0 0x564c8f50c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.536+0000 7ff8d5400640 1 -- 172.21.15.12:0/2930512897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c8f4fa810 msgr2=0x564c8f504ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:57.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.536+0000 7ff8d5400640 1 --2- 172.21.15.12:0/2930512897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c8f4fa810 0x564c8f504ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.536+0000 7ff8d5400640 1 -- 172.21.15.12:0/2930512897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564c8f1733f0 con 0x564c8f4fb9e0 2024-03-23T23:11:57.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.536+0000 7ff8d5400640 1 --2- 172.21.15.12:0/2930512897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564c8f4fb9e0 0x564c8f504ff0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0011830 tx=0x7ff8c0011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1348ae55c46fdec server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:57.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.536+0000 7ff8b3400640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:11:57.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.537+0000 7ff8b3400640 1 -- 172.21.15.12:0/2930512897 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c0010c00 con 0x564c8f4fb9e0 2024-03-23T23:11:57.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.537+0000 7ff8b3400640 1 -- 172.21.15.12:0/2930512897 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff8c0010da0 con 0x564c8f4fb9e0 2024-03-23T23:11:57.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.537+0000 7ff8db427740 1 -- 172.21.15.12:0/2930512897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff8cc004050 con 0x564c8f4fb9e0 2024-03-23T23:11:57.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.537+0000 7ff8b3400640 1 -- 172.21.15.12:0/2930512897 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8c0004020 con 0x564c8f4fb9e0 2024-03-23T23:11:57.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.537+0000 7ff8db427740 10 client.18006 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:57.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.537+0000 7ff8db427740 1 -- 172.21.15.12:0/2930512897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x564c8f23d7d0 con 0x564c8f4fb9e0 2024-03-23T23:11:57.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.537+0000 7ff8b3400640 1 -- 172.21.15.12:0/2930512897 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8cc004050 con 0x564c8f4fb9e0 2024-03-23T23:11:57.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.538+0000 7ff8b3400640 1 -- 172.21.15.12:0/2930512897 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ff8c001a410 con 0x564c8f4fb9e0 2024-03-23T23:11:57.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.538+0000 7ff8db427740 10 client.18006 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:57.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.538+0000 7ff8db427740 10 client.18006 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:11:57.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.538+0000 7ff8db427740 20 client.18006 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:57.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.538+0000 7ff8db427740 1 --2- 172.21.15.12:0/2930512897 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x564c8f5105d0 0x564c8f512a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.538+0000 7ff8db427740 4 client.18006 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:11:57.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.538+0000 7ff8db427740 1 -- 172.21.15.12:0/2930512897 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564c8f46c800 con 0x564c8f5105d0 2024-03-23T23:11:57.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.538+0000 7ff8d6800640 1 --2- 172.21.15.12:0/2930512897 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x564c8f5105d0 0x564c8f512a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.538+0000 7ff8d6800640 1 --2- 172.21.15.12:0/2930512897 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x564c8f5105d0 0x564c8f512a00 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4a4e7acb2386b27f server_cookie=65b357bcabbe828d in_seq=0 out_seq=0 2024-03-23T23:11:57.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.538+0000 7ff8b3400640 10 client.18006 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:11:57.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.539+0000 7ff8b3400640 1 -- 172.21.15.12:0/2930512897 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564c8f46c800 con 0x564c8f5105d0 2024-03-23T23:11:57.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.539+0000 7ff8b3400640 10 client.18006 handle_command_reply: tid=0 2024-03-23T23:11:57.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.539+0000 7ff8db427740 1 client.18006 shutdown 2024-03-23T23:11:57.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.539+0000 7ff8db427740 1 -- 172.21.15.12:0/2930512897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564c8f4fb9e0 msgr2=0x564c8f504ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:57.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.539+0000 7ff8db427740 1 --2- 172.21.15.12:0/2930512897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564c8f4fb9e0 0x564c8f504ff0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0011830 tx=0x7ff8c0011cf0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.539+0000 7ff8db427740 1 -- 172.21.15.12:0/2930512897 shutdown_connections 2024-03-23T23:11:57.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.539+0000 7ff8db427740 1 --2- 172.21.15.12:0/2930512897 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x564c8f5105d0 0x564c8f512a00 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.539+0000 7ff8db427740 1 --2- 172.21.15.12:0/2930512897 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c8f4fc3a0 0x564c8f50c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.539+0000 7ff8db427740 1 --2- 172.21.15.12:0/2930512897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564c8f4fb9e0 0x564c8f504ff0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.539+0000 7ff8db427740 1 --2- 172.21.15.12:0/2930512897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c8f4fa810 0x564c8f504ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.539+0000 7ff8db427740 1 -- 172.21.15.12:0/2930512897 >> 172.21.15.12:0/2930512897 conn(0x564c8f4f5c00 msgr2=0x564c8f4f4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:57.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.539+0000 7ff8b3400640 0 client.18006 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:11:57.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.540+0000 7ff8db427740 1 -- 172.21.15.12:0/2930512897 shutdown_connections 2024-03-23T23:11:57.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.540+0000 7ff8db427740 1 -- 172.21.15.12:0/2930512897 wait complete. 2024-03-23T23:11:57.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.540+0000 7ff8db427740 20 client.18006 trim_cache size 0 max 16384 2024-03-23T23:11:57.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.553+0000 7ff8db427740 1 Processor -- start 2024-03-23T23:11:57.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.553+0000 7ff8db427740 1 -- start start 2024-03-23T23:11:57.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.554+0000 7ff8db427740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c8f4fa810 0x564c8f4fec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.554+0000 7ff8db427740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564c8f4fb9e0 0x564c8f50bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.554+0000 7ff8db427740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c8f4fc3a0 0x564c8f4ff150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.554+0000 7ff8db427740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d01b14e0 con 0x564c8f4fb9e0 2024-03-23T23:11:57.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.554+0000 7ff8db427740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d01aefe0 con 0x564c8f4fc3a0 2024-03-23T23:11:57.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.554+0000 7ff8db427740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d01ad080 con 0x564c8f4fa810 2024-03-23T23:11:57.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.554+0000 7ff8d6800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c8f4fa810 0x564c8f4fec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.554+0000 7ff8d6800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c8f4fa810 0x564c8f4fec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:52594/0 (socket says 172.21.15.12:52594) 2024-03-23T23:11:57.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.554+0000 7ff8d6800640 1 -- 172.21.15.12:0/479263765 learned_addr learned my addr 172.21.15.12:0/479263765 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:57.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.554+0000 7ff8d5400640 1 --2- 172.21.15.12:0/479263765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564c8f4fb9e0 0x564c8f50bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.554+0000 7ff8d6800640 1 --2- 172.21.15.12:0/479263765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c8f4fc3a0 0x564c8f4ff150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.554+0000 7ff8d6800640 1 -- 172.21.15.12:0/479263765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c8f4fc3a0 msgr2=0x564c8f4ff150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:57.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.554+0000 7ff8d6800640 1 --2- 172.21.15.12:0/479263765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c8f4fc3a0 0x564c8f4ff150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.554+0000 7ff8d6800640 1 -- 172.21.15.12:0/479263765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564c8f4fb9e0 msgr2=0x564c8f50bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.554+0000 7ff8d6800640 1 --2- 172.21.15.12:0/479263765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564c8f4fb9e0 0x564c8f50bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.554+0000 7ff8d6800640 1 -- 172.21.15.12:0/479263765 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c8f4fb460 con 0x564c8f4fa810 2024-03-23T23:11:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.555+0000 7ff8d6800640 1 --2- 172.21.15.12:0/479263765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c8f4fa810 0x564c8f4fec10 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc012fd0 tx=0x7ff8cc009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e9fcaca1ef7a5b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.555+0000 7ff8be600640 1 -- 172.21.15.12:0/479263765 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc003970 con 0x564c8f4fa810 2024-03-23T23:11:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.555+0000 7ff8be600640 1 -- 172.21.15.12:0/479263765 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8cc004050 con 0x564c8f4fa810 2024-03-23T23:11:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.555+0000 7ff8be600640 1 -- 172.21.15.12:0/479263765 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc027840 con 0x564c8f4fa810 2024-03-23T23:11:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.556+0000 7ff8db427740 1 -- 172.21.15.12:0/479263765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c8f4fa810 msgr2=0x564c8f4fec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.556+0000 7ff8db427740 1 --2- 172.21.15.12:0/479263765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c8f4fa810 0x564c8f4fec10 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc012fd0 tx=0x7ff8cc009500 comp rx=0 tx=0).stop 2024-03-23T23:11:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.556+0000 7ff8db427740 1 -- 172.21.15.12:0/479263765 shutdown_connections 2024-03-23T23:11:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.556+0000 7ff8db427740 1 --2- 172.21.15.12:0/479263765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c8f4fc3a0 0x564c8f4ff150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.556+0000 7ff8db427740 1 --2- 172.21.15.12:0/479263765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564c8f4fb9e0 0x564c8f50bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.556+0000 7ff8db427740 1 --2- 172.21.15.12:0/479263765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c8f4fa810 0x564c8f4fec10 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.556+0000 7ff8db427740 1 -- 172.21.15.12:0/479263765 >> 172.21.15.12:0/479263765 conn(0x564c8f4f5c00 msgr2=0x564c8f4f96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.556+0000 7ff8db427740 1 -- 172.21.15.12:0/479263765 shutdown_connections 2024-03-23T23:11:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.556+0000 7ff8db427740 1 -- 172.21.15.12:0/479263765 wait complete. 2024-03-23T23:11:57.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.556+0000 7ff8db427740 1 -- start start 2024-03-23T23:11:57.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.556+0000 7ff8db427740 1 Processor -- start 2024-03-23T23:11:57.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.557+0000 7ff8db427740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564c8f4fa810 0x564c8f509a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.557+0000 7ff8db427740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c8f4fb9e0 0x564c8f509f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.557+0000 7ff8db427740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c8f4fc3a0 0x564c8f50af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.557+0000 7ff8db427740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d01ad680 con 0x564c8f4fa810 2024-03-23T23:11:57.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.557+0000 7ff8db427740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d01ae980 con 0x564c8f4fc3a0 2024-03-23T23:11:57.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.557+0000 7ff8db427740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d01adcc0 con 0x564c8f4fb9e0 2024-03-23T23:11:57.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.557+0000 7ff8db427740 10 client.0 _ll_register_callbacks cb 0x7ff8c000bcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:57.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.557+0000 7ff8d5400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c8f4fb9e0 0x564c8f509f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.557+0000 7ff8d5400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c8f4fb9e0 0x564c8f509f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:52604/0 (socket says 172.21.15.12:52604) 2024-03-23T23:11:57.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.557+0000 7ff8d5400640 1 -- 172.21.15.12:0/2175779901 learned_addr learned my addr 172.21.15.12:0/2175779901 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:57.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.557+0000 7ff8d6800640 1 --2- 172.21.15.12:0/2175779901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564c8f4fa810 0x564c8f509a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.557+0000 7ff8d6800640 1 --2- 172.21.15.12:0/2175779901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c8f4fc3a0 0x564c8f50af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.558+0000 7ff8d5400640 1 -- 172.21.15.12:0/2175779901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c8f4fc3a0 msgr2=0x564c8f50af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:57.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.558+0000 7ff8d5400640 1 --2- 172.21.15.12:0/2175779901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c8f4fc3a0 0x564c8f50af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.558+0000 7ff8d5400640 1 -- 172.21.15.12:0/2175779901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564c8f4fa810 msgr2=0x564c8f509a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:57.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.558+0000 7ff8d5400640 1 --2- 172.21.15.12:0/2175779901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564c8f4fa810 0x564c8f509a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.558+0000 7ff8d5400640 1 -- 172.21.15.12:0/2175779901 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564c8f50b550 con 0x564c8f4fb9e0 2024-03-23T23:11:57.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.558+0000 7ff8d6800640 1 --2- 172.21.15.12:0/2175779901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c8f4fc3a0 0x564c8f50af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:11:57.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.558+0000 7ff8d5400640 1 --2- 172.21.15.12:0/2175779901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c8f4fb9e0 0x564c8f509f50 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ff8c001a8d0 tx=0x7ff8c00114c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8fc2d8b2436a9e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:57.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.558+0000 7ff8b3400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:11:57.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.558+0000 7ff8b3400640 1 -- 172.21.15.12:0/2175779901 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c0029040 con 0x564c8f4fb9e0 2024-03-23T23:11:57.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.558+0000 7ff8db427740 1 -- 172.21.15.12:0/2175779901 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff8cc004050 con 0x564c8f4fb9e0 2024-03-23T23:11:57.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.558+0000 7ff8b3400640 1 -- 172.21.15.12:0/2175779901 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff8c0017070 con 0x564c8f4fb9e0 2024-03-23T23:11:57.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.558+0000 7ff8b3400640 1 -- 172.21.15.12:0/2175779901 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8c0010c00 con 0x564c8f4fb9e0 2024-03-23T23:11:57.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.558+0000 7ff8db427740 10 client.28123 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:57.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.558+0000 7ff8db427740 1 -- 172.21.15.12:0/2175779901 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x564c8f56c0f0 con 0x564c8f4fb9e0 2024-03-23T23:11:57.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.559+0000 7ff8b3400640 1 -- 172.21.15.12:0/2175779901 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8cc004050 con 0x564c8f4fb9e0 2024-03-23T23:11:57.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.559+0000 7ff8b3400640 1 -- 172.21.15.12:0/2175779901 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ff8c0005160 con 0x564c8f4fb9e0 2024-03-23T23:11:57.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.559+0000 7ff8db427740 10 client.28123 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:57.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.559+0000 7ff8db427740 10 client.28123 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:11:57.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.559+0000 7ff8db427740 20 client.28123 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:57.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.559+0000 7ff8db427740 1 --2- 172.21.15.12:0/2175779901 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x564c8f56ca60 0x564c8f56ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.559+0000 7ff8db427740 4 client.28123 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:11:57.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.559+0000 7ff8db427740 1 -- 172.21.15.12:0/2175779901 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564c8f466b50 con 0x564c8f56ca60 2024-03-23T23:11:57.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.559+0000 7ff8d6800640 1 --2- 172.21.15.12:0/2175779901 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x564c8f56ca60 0x564c8f56ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.560+0000 7ff8d6800640 1 --2- 172.21.15.12:0/2175779901 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x564c8f56ca60 0x564c8f56ee80 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=81636da947dd7e1d server_cookie=5689df0e9100450e in_seq=0 out_seq=0 2024-03-23T23:11:57.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.560+0000 7ff8b3400640 10 client.28123 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:11:57.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.560+0000 7ff8b3400640 1 -- 172.21.15.12:0/2175779901 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564c8f466b50 con 0x564c8f56ca60 2024-03-23T23:11:57.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.560+0000 7ff8b3400640 10 client.28123 handle_command_reply: tid=0 2024-03-23T23:11:57.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.560+0000 7ff8db427740 1 client.28123 shutdown 2024-03-23T23:11:57.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.560+0000 7ff8db427740 1 -- 172.21.15.12:0/2175779901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c8f4fb9e0 msgr2=0x564c8f509f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:57.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.560+0000 7ff8db427740 1 --2- 172.21.15.12:0/2175779901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c8f4fb9e0 0x564c8f509f50 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ff8c001a8d0 tx=0x7ff8c00114c0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.560+0000 7ff8db427740 1 -- 172.21.15.12:0/2175779901 shutdown_connections 2024-03-23T23:11:57.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.560+0000 7ff8db427740 1 --2- 172.21.15.12:0/2175779901 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x564c8f56ca60 0x564c8f56ee80 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.560+0000 7ff8db427740 1 --2- 172.21.15.12:0/2175779901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c8f4fc3a0 0x564c8f50af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.560+0000 7ff8db427740 1 --2- 172.21.15.12:0/2175779901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c8f4fb9e0 0x564c8f509f50 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.560+0000 7ff8b3400640 0 client.28123 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:11:57.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.560+0000 7ff8db427740 1 --2- 172.21.15.12:0/2175779901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564c8f4fa810 0x564c8f509a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.560+0000 7ff8db427740 1 -- 172.21.15.12:0/2175779901 >> 172.21.15.12:0/2175779901 conn(0x564c8f4f5c00 msgr2=0x564c8f4f4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:57.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.560+0000 7ff8db427740 1 -- 172.21.15.12:0/2175779901 shutdown_connections 2024-03-23T23:11:57.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.560+0000 7ff8db427740 1 -- 172.21.15.12:0/2175779901 wait complete. 2024-03-23T23:11:57.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.560+0000 7ff8db427740 20 client.28123 trim_cache size 0 max 16384 2024-03-23T23:11:57.564 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:11:57.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.561+0000 7ff8d8800640 1 -- 172.21.15.12:0/2858051655 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff8b80a2880 msgr2=0x7ff8b80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:57.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.561+0000 7ff8d8800640 1 --2- 172.21.15.12:0/2858051655 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff8b80a2880 0x7ff8b80a4d40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40b93b0 tx=0x7ff8c40f3f90 comp rx=0 tx=0).stop 2024-03-23T23:11:57.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.561+0000 7ff8d8800640 1 -- 172.21.15.12:0/2858051655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8d0107cc0 msgr2=0x7ff8d019dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:57.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.561+0000 7ff8d8800640 1 --2- 172.21.15.12:0/2858051655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8d0107cc0 0x7ff8d019dfd0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7ff8c402efd0 tx=0x7ff8c4005d50 comp rx=0 tx=0).stop 2024-03-23T23:11:57.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.561+0000 7ff8d8800640 1 -- 172.21.15.12:0/2858051655 shutdown_connections 2024-03-23T23:11:57.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.561+0000 7ff8d8800640 1 --2- 172.21.15.12:0/2858051655 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff8b80a2880 0x7ff8b80a4d40 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.561+0000 7ff8d8800640 1 --2- 172.21.15.12:0/2858051655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8d0109870 0x7ff8d01a5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.561+0000 7ff8d8800640 1 --2- 172.21.15.12:0/2858051655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8d0108eb0 0x7ff8d019e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.561+0000 7ff8d8800640 1 --2- 172.21.15.12:0/2858051655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8d0107cc0 0x7ff8d019dfd0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.561+0000 7ff8d8800640 1 -- 172.21.15.12:0/2858051655 >> 172.21.15.12:0/2858051655 conn(0x7ff8d0076c60 msgr2=0x7ff8d010c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:57.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.561+0000 7ff8d8800640 1 -- 172.21.15.12:0/2858051655 shutdown_connections 2024-03-23T23:11:57.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.561+0000 7ff8d8800640 1 -- 172.21.15.12:0/2858051655 wait complete. 2024-03-23T23:11:57.579 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:11:57.579 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-23T23:11:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.804+0000 7f7a31600640 1 -- 172.21.15.12:0/1678869481 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a2c105370 msgr2=0x7f7a2c10bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.804+0000 7f7a31600640 1 --2- 172.21.15.12:0/1678869481 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a2c105370 0x7f7a2c10bc00 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f7a24009ca0 tx=0x7f7a2402c0f0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.804+0000 7f7a31600640 1 -- 172.21.15.12:0/1678869481 shutdown_connections 2024-03-23T23:11:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.804+0000 7f7a31600640 1 --2- 172.21.15.12:0/1678869481 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a2c105370 0x7f7a2c10bc00 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.804+0000 7f7a31600640 1 --2- 172.21.15.12:0/1678869481 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7a2c1049b0 0x7f7a2c104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.804+0000 7f7a31600640 1 --2- 172.21.15.12:0/1678869481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a2c1037c0 0x7f7a2c103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.804+0000 7f7a31600640 1 -- 172.21.15.12:0/1678869481 >> 172.21.15.12:0/1678869481 conn(0x7f7a2c0fed50 msgr2=0x7f7a2c101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.804+0000 7f7a31600640 1 -- 172.21.15.12:0/1678869481 shutdown_connections 2024-03-23T23:11:57.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.804+0000 7f7a31600640 1 -- 172.21.15.12:0/1678869481 wait complete. 2024-03-23T23:11:57.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.805+0000 7f7a31600640 1 Processor -- start 2024-03-23T23:11:57.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.805+0000 7f7a31600640 1 -- start start 2024-03-23T23:11:57.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.806+0000 7f7a31600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a2c1037c0 0x7f7a2c199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.806+0000 7f7a31600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a2c1049b0 0x7f7a2c199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.806+0000 7f7a31600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7a2c105370 0x7f7a2c1a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.806+0000 7f7a31600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c10f440 con 0x7f7a2c1049b0 2024-03-23T23:11:57.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.806+0000 7f7a31600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c10f2c0 con 0x7f7a2c105370 2024-03-23T23:11:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.806+0000 7f7a31600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c10f5c0 con 0x7f7a2c1037c0 2024-03-23T23:11:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.806+0000 7f7a2be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a2c1037c0 0x7f7a2c199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.806+0000 7f7a2be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a2c1037c0 0x7f7a2c199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:52620/0 (socket says 172.21.15.12:52620) 2024-03-23T23:11:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.806+0000 7f7a2be00640 1 -- 172.21.15.12:0/1772051889 learned_addr learned my addr 172.21.15.12:0/1772051889 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.806+0000 7f7a23e00640 1 --2- 172.21.15.12:0/1772051889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a2c1049b0 0x7f7a2c199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.806+0000 7f7a30c00640 1 --2- 172.21.15.12:0/1772051889 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7a2c105370 0x7f7a2c1a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.806+0000 7f7a2be00640 1 -- 172.21.15.12:0/1772051889 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7a2c105370 msgr2=0x7f7a2c1a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.806+0000 7f7a2be00640 1 --2- 172.21.15.12:0/1772051889 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7a2c105370 0x7f7a2c1a0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.806+0000 7f7a2be00640 1 -- 172.21.15.12:0/1772051889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a2c1049b0 msgr2=0x7f7a2c199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.806+0000 7f7a2be00640 1 --2- 172.21.15.12:0/1772051889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a2c1049b0 0x7f7a2c199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.806+0000 7f7a2be00640 1 -- 172.21.15.12:0/1772051889 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2c1a1640 con 0x7f7a2c1037c0 2024-03-23T23:11:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.806+0000 7f7a23e00640 1 --2- 172.21.15.12:0/1772051889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a2c1049b0 0x7f7a2c199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:11:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.807+0000 7f7a2be00640 1 --2- 172.21.15.12:0/1772051889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a2c1037c0 0x7f7a2c199a20 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c00d950 tx=0x7f7a1c00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.807+0000 7f7a29600640 1 -- 172.21.15.12:0/1772051889 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c017020 con 0x7f7a2c1037c0 2024-03-23T23:11:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.807+0000 7f7a29600640 1 -- 172.21.15.12:0/1772051889 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a1c009c30 con 0x7f7a2c1037c0 2024-03-23T23:11:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.807+0000 7f7a31600640 1 -- 172.21.15.12:0/1772051889 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a2c1a1930 con 0x7f7a2c1037c0 2024-03-23T23:11:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.807+0000 7f7a29600640 1 -- 172.21.15.12:0/1772051889 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c00f720 con 0x7f7a2c1037c0 2024-03-23T23:11:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.807+0000 7f7a31600640 1 -- 172.21.15.12:0/1772051889 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a2c1a1ed0 con 0x7f7a2c1037c0 2024-03-23T23:11:57.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.810+0000 7f7a29600640 1 -- 172.21.15.12:0/1772051889 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7a1c01a020 con 0x7f7a2c1037c0 2024-03-23T23:11:57.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.810+0000 7f7a35972740 1 Processor -- start 2024-03-23T23:11:57.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.810+0000 7f7a35972740 1 -- start start 2024-03-23T23:11:57.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.811+0000 7f7a35972740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565493d679f0 0x565493d67e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.811+0000 7f7a35972740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565493d68bc0 0x565493d69040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.811+0000 7f7a35972740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565493d69580 0x565493d6fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.811+0000 7f7a35972740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x565493ccc2c0 con 0x565493d68bc0 2024-03-23T23:11:57.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.811+0000 7f7a35972740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x565493c74c90 con 0x565493d69580 2024-03-23T23:11:57.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.811+0000 7f7a35972740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x565493cc8360 con 0x565493d679f0 2024-03-23T23:11:57.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.811+0000 7f7a30c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565493d679f0 0x565493d67e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.811+0000 7f7a30c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565493d679f0 0x565493d67e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:52632/0 (socket says 172.21.15.12:52632) 2024-03-23T23:11:57.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.811+0000 7f7a30c00640 1 -- 172.21.15.12:0/728046191 learned_addr learned my addr 172.21.15.12:0/728046191 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:57.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.811+0000 7f7a2be00640 1 --2- 172.21.15.12:0/728046191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565493d68bc0 0x565493d69040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.811+0000 7f7a23e00640 1 --2- 172.21.15.12:0/728046191 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565493d69580 0x565493d6fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.811+0000 7f7a30c00640 1 -- 172.21.15.12:0/728046191 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565493d69580 msgr2=0x565493d6fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:57.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.811+0000 7f7a30c00640 1 --2- 172.21.15.12:0/728046191 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565493d69580 0x565493d6fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.811+0000 7f7a30c00640 1 -- 172.21.15.12:0/728046191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565493d68bc0 msgr2=0x565493d69040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:57.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.811+0000 7f7a30c00640 1 --2- 172.21.15.12:0/728046191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565493d68bc0 0x565493d69040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.811+0000 7f7a30c00640 1 -- 172.21.15.12:0/728046191 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565493c99080 con 0x565493d679f0 2024-03-23T23:11:57.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.811+0000 7f7a2be00640 1 --2- 172.21.15.12:0/728046191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565493d68bc0 0x565493d69040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:11:57.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.811+0000 7f7a23e00640 1 --2- 172.21.15.12:0/728046191 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565493d69580 0x565493d6fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:11:57.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.812+0000 7f7a30c00640 1 --2- 172.21.15.12:0/728046191 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565493d679f0 0x565493d67e10 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f7a2402c700 tx=0x7f7a240323d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfc84c952f1a19c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:57.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.812+0000 7f7a21200640 1 -- 172.21.15.12:0/728046191 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a24043070 con 0x565493d679f0 2024-03-23T23:11:57.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.812+0000 7f7a29600640 1 --2- 172.21.15.12:0/1772051889 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7a100a2880 0x7f7a100a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.812+0000 7f7a21200640 1 -- 172.21.15.12:0/728046191 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a2402cca0 con 0x565493d679f0 2024-03-23T23:11:57.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.812+0000 7f7a21200640 1 -- 172.21.15.12:0/728046191 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a24039040 con 0x565493d679f0 2024-03-23T23:11:57.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.812+0000 7f7a29600640 1 -- 172.21.15.12:0/1772051889 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7a1c0ce9a0 con 0x7f7a2c1037c0 2024-03-23T23:11:57.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.812+0000 7f7a2be00640 1 --2- 172.21.15.12:0/1772051889 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7a100a2880 0x7f7a100a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.812+0000 7f7a35972740 1 -- 172.21.15.12:0/728046191 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565493d679f0 msgr2=0x565493d67e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:57.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.812+0000 7f7a35972740 1 --2- 172.21.15.12:0/728046191 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565493d679f0 0x565493d67e10 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f7a2402c700 tx=0x7f7a240323d0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.813+0000 7f7a35972740 1 -- 172.21.15.12:0/728046191 shutdown_connections 2024-03-23T23:11:57.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.813+0000 7f7a35972740 1 --2- 172.21.15.12:0/728046191 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565493d69580 0x565493d6fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.813+0000 7f7a35972740 1 --2- 172.21.15.12:0/728046191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565493d68bc0 0x565493d69040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.813+0000 7f7a35972740 1 --2- 172.21.15.12:0/728046191 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565493d679f0 0x565493d67e10 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.813+0000 7f7a35972740 1 -- 172.21.15.12:0/728046191 >> 172.21.15.12:0/728046191 conn(0x565493d62de0 msgr2=0x565493d631e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:57.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.813+0000 7f7a35972740 1 -- 172.21.15.12:0/728046191 shutdown_connections 2024-03-23T23:11:57.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.813+0000 7f7a35972740 1 -- 172.21.15.12:0/728046191 wait complete. 2024-03-23T23:11:57.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.813+0000 7f7a2be00640 1 --2- 172.21.15.12:0/1772051889 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7a100a2880 0x7f7a100a4d40 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c009610 tx=0x7f7a1c010e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:57.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.813+0000 7f7a35972740 1 -- start start 2024-03-23T23:11:57.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.813+0000 7f7a35972740 1 Processor -- start 2024-03-23T23:11:57.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.814+0000 7f7a35972740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565493d679f0 0x565493d71c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.814+0000 7f7a35972740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565493d68bc0 0x565493d721a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.814+0000 7f7a35972740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565493d69580 0x565493d791e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.814+0000 7f7a35972740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x565493d70530 con 0x565493d679f0 2024-03-23T23:11:57.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.814+0000 7f7a35972740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x565493c880d0 con 0x565493d69580 2024-03-23T23:11:57.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.814+0000 7f7a35972740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x565493d706b0 con 0x565493d68bc0 2024-03-23T23:11:57.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.814+0000 7f7a35972740 10 client.0 _ll_register_callbacks cb 0x565493c8ce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:57.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.814+0000 7f7a30c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565493d679f0 0x565493d71c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.814+0000 7f7a23e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565493d68bc0 0x565493d721a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.814+0000 7f7a23e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565493d68bc0 0x565493d721a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:52638/0 (socket says 172.21.15.12:52638) 2024-03-23T23:11:57.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.814+0000 7f7a23e00640 1 -- 172.21.15.12:0/4212393149 learned_addr learned my addr 172.21.15.12:0/4212393149 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:57.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.814+0000 7f7a30c00640 1 --2- 172.21.15.12:0/4212393149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565493d69580 0x565493d791e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.814+0000 7f7a23e00640 1 -- 172.21.15.12:0/4212393149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565493d69580 msgr2=0x565493d791e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:57.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.814+0000 7f7a23e00640 1 --2- 172.21.15.12:0/4212393149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565493d69580 0x565493d791e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.815+0000 7f7a23e00640 1 -- 172.21.15.12:0/4212393149 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565493d679f0 msgr2=0x565493d71c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:57.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.815+0000 7f7a23e00640 1 --2- 172.21.15.12:0/4212393149 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565493d679f0 0x565493d71c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.815+0000 7f7a23e00640 1 -- 172.21.15.12:0/4212393149 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5654939e03f0 con 0x565493d68bc0 2024-03-23T23:11:57.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.815+0000 7f7a23e00640 1 --2- 172.21.15.12:0/4212393149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565493d68bc0 0x565493d721a0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7a14013790 tx=0x7f7a14013c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=357ac726c967406e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:57.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.815+0000 7f7a19600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:11:57.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.815+0000 7f7a19600640 1 -- 172.21.15.12:0/4212393149 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a14020060 con 0x565493d68bc0 2024-03-23T23:11:57.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.815+0000 7f7a19600640 1 -- 172.21.15.12:0/4212393149 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7a14011d70 con 0x565493d68bc0 2024-03-23T23:11:57.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.815+0000 7f7a35972740 1 -- 172.21.15.12:0/4212393149 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7a2402cca0 con 0x565493d68bc0 2024-03-23T23:11:57.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.815+0000 7f7a19600640 1 -- 172.21.15.12:0/4212393149 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a1400f450 con 0x565493d68bc0 2024-03-23T23:11:57.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.815+0000 7f7a35972740 10 client.28135 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:57.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.815+0000 7f7a35972740 1 -- 172.21.15.12:0/4212393149 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x565493aaa7d0 con 0x565493d68bc0 2024-03-23T23:11:57.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.816+0000 7f7a19600640 1 -- 172.21.15.12:0/4212393149 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a2402cca0 con 0x565493d68bc0 2024-03-23T23:11:57.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.816+0000 7f7a19600640 1 -- 172.21.15.12:0/4212393149 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f7a1401a030 con 0x565493d68bc0 2024-03-23T23:11:57.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.816+0000 7f7a35972740 10 client.28135 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:57.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.816+0000 7f7a35972740 10 client.28135 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:11:57.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.816+0000 7f7a35972740 20 client.28135 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:57.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.816+0000 7f7a35972740 1 --2- 172.21.15.12:0/4212393149 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x565493d7d7a0 0x565493d7fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.816+0000 7f7a35972740 4 client.28135 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:11:57.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.816+0000 7f7a35972740 1 -- 172.21.15.12:0/4212393149 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565493cd9800 con 0x565493d7d7a0 2024-03-23T23:11:57.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.816+0000 7f7a30c00640 1 --2- 172.21.15.12:0/4212393149 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x565493d7d7a0 0x565493d7fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.817+0000 7f7a30c00640 1 --2- 172.21.15.12:0/4212393149 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x565493d7d7a0 0x565493d7fc00 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=588cf112381b8847 server_cookie=1637ce127a95614a in_seq=0 out_seq=0 2024-03-23T23:11:57.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.817+0000 7f7a19600640 10 client.28135 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:11:57.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.817+0000 7f7a19600640 1 -- 172.21.15.12:0/4212393149 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x565493cd9800 con 0x565493d7d7a0 2024-03-23T23:11:57.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.817+0000 7f7a19600640 10 client.28135 handle_command_reply: tid=0 2024-03-23T23:11:57.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.817+0000 7f7a35972740 1 client.28135 shutdown 2024-03-23T23:11:57.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.818+0000 7f7a35972740 1 -- 172.21.15.12:0/4212393149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565493d68bc0 msgr2=0x565493d721a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:57.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.818+0000 7f7a35972740 1 --2- 172.21.15.12:0/4212393149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565493d68bc0 0x565493d721a0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7a14013790 tx=0x7f7a14013c50 comp rx=0 tx=0).stop 2024-03-23T23:11:57.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.818+0000 7f7a35972740 1 -- 172.21.15.12:0/4212393149 shutdown_connections 2024-03-23T23:11:57.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.818+0000 7f7a35972740 1 --2- 172.21.15.12:0/4212393149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565493d69580 0x565493d791e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.818+0000 7f7a35972740 1 --2- 172.21.15.12:0/4212393149 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x565493d7d7a0 0x565493d7fc00 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.818+0000 7f7a35972740 1 --2- 172.21.15.12:0/4212393149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565493d68bc0 0x565493d721a0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.818+0000 7f7a35972740 1 --2- 172.21.15.12:0/4212393149 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565493d679f0 0x565493d71c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.818+0000 7f7a19600640 0 client.28135 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:11:57.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.818+0000 7f7a35972740 1 -- 172.21.15.12:0/4212393149 >> 172.21.15.12:0/4212393149 conn(0x565493d62de0 msgr2=0x565493d62160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:57.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.818+0000 7f7a35972740 1 -- 172.21.15.12:0/4212393149 shutdown_connections 2024-03-23T23:11:57.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.818+0000 7f7a35972740 1 -- 172.21.15.12:0/4212393149 wait complete. 2024-03-23T23:11:57.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.818+0000 7f7a35972740 20 client.28135 trim_cache size 0 max 16384 2024-03-23T23:11:57.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.831+0000 7f7a35972740 1 Processor -- start 2024-03-23T23:11:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.832+0000 7f7a35972740 1 -- start start 2024-03-23T23:11:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.832+0000 7f7a35972740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565493d679f0 0x565493d5dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.832+0000 7f7a35972740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565493d68bc0 0x565493d791a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.832+0000 7f7a35972740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565493d69580 0x565493d6bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.832+0000 7f7a35972740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c1acf50 con 0x565493d69580 2024-03-23T23:11:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.832+0000 7f7a35972740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c1aaa50 con 0x565493d68bc0 2024-03-23T23:11:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.832+0000 7f7a35972740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c1a8af0 con 0x565493d679f0 2024-03-23T23:11:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.832+0000 7f7a23e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565493d68bc0 0x565493d791a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.832+0000 7f7a23e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565493d68bc0 0x565493d791a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:51610/0 (socket says 172.21.15.12:51610) 2024-03-23T23:11:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.832+0000 7f7a23e00640 1 -- 172.21.15.12:0/1063072581 learned_addr learned my addr 172.21.15.12:0/1063072581 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.832+0000 7f7a30c00640 1 --2- 172.21.15.12:0/1063072581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565493d679f0 0x565493d5dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.832+0000 7f7a30c00640 1 --2- 172.21.15.12:0/1063072581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565493d69580 0x565493d6bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.832+0000 7f7a23e00640 1 -- 172.21.15.12:0/1063072581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565493d679f0 msgr2=0x565493d5dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:57.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.832+0000 7f7a23e00640 1 --2- 172.21.15.12:0/1063072581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565493d679f0 0x565493d5dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.832+0000 7f7a23e00640 1 -- 172.21.15.12:0/1063072581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565493d69580 msgr2=0x565493d6bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:57.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.832+0000 7f7a23e00640 1 --2- 172.21.15.12:0/1063072581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565493d69580 0x565493d6bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.832+0000 7f7a23e00640 1 -- 172.21.15.12:0/1063072581 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565493d7d580 con 0x565493d68bc0 2024-03-23T23:11:57.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.832+0000 7f7a30c00640 1 --2- 172.21.15.12:0/1063072581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565493d69580 0x565493d6bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:11:57.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.833+0000 7f7a23e00640 1 --2- 172.21.15.12:0/1063072581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565493d68bc0 0x565493d791a0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f7a14009c20 tx=0x7f7a14009d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c295ff970131b38 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:57.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.833+0000 7f7a1be00640 1 -- 172.21.15.12:0/1063072581 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1402b070 con 0x565493d68bc0 2024-03-23T23:11:57.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.833+0000 7f7a1be00640 1 -- 172.21.15.12:0/1063072581 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a14005510 con 0x565493d68bc0 2024-03-23T23:11:57.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.833+0000 7f7a1be00640 1 -- 172.21.15.12:0/1063072581 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a14011e60 con 0x565493d68bc0 2024-03-23T23:11:57.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.834+0000 7f7a35972740 1 -- 172.21.15.12:0/1063072581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565493d68bc0 msgr2=0x565493d791a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:57.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.834+0000 7f7a35972740 1 --2- 172.21.15.12:0/1063072581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565493d68bc0 0x565493d791a0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f7a14009c20 tx=0x7f7a14009d40 comp rx=0 tx=0).stop 2024-03-23T23:11:57.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.834+0000 7f7a35972740 1 -- 172.21.15.12:0/1063072581 shutdown_connections 2024-03-23T23:11:57.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.834+0000 7f7a35972740 1 --2- 172.21.15.12:0/1063072581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565493d69580 0x565493d6bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.834+0000 7f7a35972740 1 --2- 172.21.15.12:0/1063072581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565493d68bc0 0x565493d791a0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.834+0000 7f7a35972740 1 --2- 172.21.15.12:0/1063072581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565493d679f0 0x565493d5dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.834+0000 7f7a35972740 1 -- 172.21.15.12:0/1063072581 >> 172.21.15.12:0/1063072581 conn(0x565493d62de0 msgr2=0x565493d668d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:57.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.834+0000 7f7a35972740 1 -- 172.21.15.12:0/1063072581 shutdown_connections 2024-03-23T23:11:57.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.834+0000 7f7a35972740 1 -- 172.21.15.12:0/1063072581 wait complete. 2024-03-23T23:11:57.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.834+0000 7f7a35972740 1 -- start start 2024-03-23T23:11:57.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.834+0000 7f7a35972740 1 Processor -- start 2024-03-23T23:11:57.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.835+0000 7f7a35972740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565493d679f0 0x565493d753f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.835+0000 7f7a35972740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565493d68bc0 0x565493d75930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.835+0000 7f7a35972740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565493d69580 0x565493d78860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.835+0000 7f7a35972740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c1a90f0 con 0x565493d69580 2024-03-23T23:11:57.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.835+0000 7f7a35972740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c1aa3f0 con 0x565493d679f0 2024-03-23T23:11:57.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.835+0000 7f7a35972740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c1a9730 con 0x565493d68bc0 2024-03-23T23:11:57.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.835+0000 7f7a35972740 10 client.0 _ll_register_callbacks cb 0x7f7a14009550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:57.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.835+0000 7f7a30c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565493d679f0 0x565493d753f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.835+0000 7f7a23e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565493d68bc0 0x565493d75930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.835+0000 7f7a30c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565493d679f0 0x565493d753f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:51612/0 (socket says 172.21.15.12:51612) 2024-03-23T23:11:57.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.835+0000 7f7a30c00640 1 -- 172.21.15.12:0/3804378991 learned_addr learned my addr 172.21.15.12:0/3804378991 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:57.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.835+0000 7f7a30c00640 1 --2- 172.21.15.12:0/3804378991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565493d69580 0x565493d78860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.835+0000 7f7a30c00640 1 -- 172.21.15.12:0/3804378991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565493d68bc0 msgr2=0x565493d75930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:57.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.835+0000 7f7a30c00640 1 --2- 172.21.15.12:0/3804378991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565493d68bc0 0x565493d75930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.835+0000 7f7a30c00640 1 -- 172.21.15.12:0/3804378991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565493d69580 msgr2=0x565493d78860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:57.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.835+0000 7f7a30c00640 1 --2- 172.21.15.12:0/3804378991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565493d69580 0x565493d78860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.835+0000 7f7a30c00640 1 -- 172.21.15.12:0/3804378991 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565493d63850 con 0x565493d679f0 2024-03-23T23:11:57.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.835+0000 7f7a23e00640 1 --2- 172.21.15.12:0/3804378991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565493d68bc0 0x565493d75930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:57.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.836+0000 7f7a30c00640 1 --2- 172.21.15.12:0/3804378991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565493d679f0 0x565493d753f0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f7a2402c130 tx=0x7f7a24036040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=890be63bb44357d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:57.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.836+0000 7f7a19600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:11:57.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.836+0000 7f7a19600640 1 -- 172.21.15.12:0/3804378991 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a24039040 con 0x565493d679f0 2024-03-23T23:11:57.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.836+0000 7f7a19600640 1 -- 172.21.15.12:0/3804378991 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7a24032690 con 0x565493d679f0 2024-03-23T23:11:57.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.836+0000 7f7a35972740 1 -- 172.21.15.12:0/3804378991 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7a14005510 con 0x565493d679f0 2024-03-23T23:11:57.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.836+0000 7f7a19600640 1 -- 172.21.15.12:0/3804378991 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a2402de60 con 0x565493d679f0 2024-03-23T23:11:57.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.836+0000 7f7a35972740 10 client.28172 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:57.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.836+0000 7f7a35972740 1 -- 172.21.15.12:0/3804378991 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x565493de2140 con 0x565493d679f0 2024-03-23T23:11:57.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.836+0000 7f7a19600640 1 -- 172.21.15.12:0/3804378991 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a14005510 con 0x565493d679f0 2024-03-23T23:11:57.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.837+0000 7f7a19600640 1 -- 172.21.15.12:0/3804378991 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f7a24048450 con 0x565493d679f0 2024-03-23T23:11:57.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.837+0000 7f7a35972740 10 client.28172 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:57.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.837+0000 7f7a35972740 10 client.28172 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:11:57.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.837+0000 7f7a35972740 20 client.28172 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:57.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.837+0000 7f7a35972740 1 --2- 172.21.15.12:0/3804378991 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x565493de2680 0x565493de4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:57.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.837+0000 7f7a35972740 4 client.28172 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:11:57.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.837+0000 7f7a35972740 1 -- 172.21.15.12:0/3804378991 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x565493cd3b50 con 0x565493de2680 2024-03-23T23:11:57.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.837+0000 7f7a23e00640 1 --2- 172.21.15.12:0/3804378991 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x565493de2680 0x565493de4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:57.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.837+0000 7f7a23e00640 1 --2- 172.21.15.12:0/3804378991 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x565493de2680 0x565493de4aa0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2b98b71dbfd9b70b server_cookie=14212e71cc5c1941 in_seq=0 out_seq=0 2024-03-23T23:11:57.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.837+0000 7f7a19600640 10 client.28172 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:11:57.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.838+0000 7f7a19600640 1 -- 172.21.15.12:0/3804378991 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x565493cd3b50 con 0x565493de2680 2024-03-23T23:11:57.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.838+0000 7f7a19600640 10 client.28172 handle_command_reply: tid=0 2024-03-23T23:11:57.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.838+0000 7f7a35972740 1 client.28172 shutdown 2024-03-23T23:11:57.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.838+0000 7f7a35972740 1 -- 172.21.15.12:0/3804378991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565493d679f0 msgr2=0x565493d753f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:57.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.838+0000 7f7a35972740 1 --2- 172.21.15.12:0/3804378991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565493d679f0 0x565493d753f0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f7a2402c130 tx=0x7f7a24036040 comp rx=0 tx=0).stop 2024-03-23T23:11:57.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.838+0000 7f7a35972740 1 -- 172.21.15.12:0/3804378991 shutdown_connections 2024-03-23T23:11:57.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.838+0000 7f7a35972740 1 --2- 172.21.15.12:0/3804378991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565493d69580 0x565493d78860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.838+0000 7f7a35972740 1 --2- 172.21.15.12:0/3804378991 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x565493de2680 0x565493de4aa0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.838+0000 7f7a35972740 1 --2- 172.21.15.12:0/3804378991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565493d68bc0 0x565493d75930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.838+0000 7f7a35972740 1 --2- 172.21.15.12:0/3804378991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565493d679f0 0x565493d753f0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.838+0000 7f7a35972740 1 -- 172.21.15.12:0/3804378991 >> 172.21.15.12:0/3804378991 conn(0x565493d62de0 msgr2=0x565493d618d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:57.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.838+0000 7f7a19600640 0 client.28172 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:11:57.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.838+0000 7f7a35972740 1 -- 172.21.15.12:0/3804378991 shutdown_connections 2024-03-23T23:11:57.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.838+0000 7f7a35972740 1 -- 172.21.15.12:0/3804378991 wait complete. 2024-03-23T23:11:57.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.838+0000 7f7a35972740 20 client.28172 trim_cache size 0 max 16384 2024-03-23T23:11:57.841 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:11:57.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.839+0000 7f7a31600640 1 -- 172.21.15.12:0/1772051889 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7a100a2880 msgr2=0x7f7a100a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:57.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.839+0000 7f7a31600640 1 --2- 172.21.15.12:0/1772051889 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7a100a2880 0x7f7a100a4d40 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c009610 tx=0x7f7a1c010e30 comp rx=0 tx=0).stop 2024-03-23T23:11:57.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.839+0000 7f7a31600640 1 -- 172.21.15.12:0/1772051889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a2c1037c0 msgr2=0x7f7a2c199a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:57.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.839+0000 7f7a31600640 1 --2- 172.21.15.12:0/1772051889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a2c1037c0 0x7f7a2c199a20 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c00d950 tx=0x7f7a1c00de10 comp rx=0 tx=0).stop 2024-03-23T23:11:57.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.839+0000 7f7a31600640 1 -- 172.21.15.12:0/1772051889 shutdown_connections 2024-03-23T23:11:57.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.839+0000 7f7a31600640 1 --2- 172.21.15.12:0/1772051889 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7a100a2880 0x7f7a100a4d40 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.839+0000 7f7a31600640 1 --2- 172.21.15.12:0/1772051889 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7a2c105370 0x7f7a2c1a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.839+0000 7f7a31600640 1 --2- 172.21.15.12:0/1772051889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a2c1049b0 0x7f7a2c199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.839+0000 7f7a31600640 1 --2- 172.21.15.12:0/1772051889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a2c1037c0 0x7f7a2c199a20 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:57.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.839+0000 7f7a31600640 1 -- 172.21.15.12:0/1772051889 >> 172.21.15.12:0/1772051889 conn(0x7f7a2c0fed50 msgr2=0x7f7a2c107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:57.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.839+0000 7f7a31600640 1 -- 172.21.15.12:0/1772051889 shutdown_connections 2024-03-23T23:11:57.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:57.839+0000 7f7a31600640 1 -- 172.21.15.12:0/1772051889 wait complete. 2024-03-23T23:11:57.856 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:11:57.856 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-23T23:11:58.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.085+0000 7fe4c1400640 1 -- 172.21.15.12:0/2416987466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe4bc1049b0 msgr2=0x7fe4bc104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:58.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.085+0000 7fe4c1400640 1 --2- 172.21.15.12:0/2416987466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe4bc1049b0 0x7fe4bc104e30 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4009900 tx=0x7fe4a402bc30 comp rx=0 tx=0).stop 2024-03-23T23:11:58.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.086+0000 7fe4c1400640 1 -- 172.21.15.12:0/2416987466 shutdown_connections 2024-03-23T23:11:58.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.086+0000 7fe4c1400640 1 --2- 172.21.15.12:0/2416987466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4bc105370 0x7fe4bc10bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.086+0000 7fe4c1400640 1 --2- 172.21.15.12:0/2416987466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe4bc1049b0 0x7fe4bc104e30 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.086+0000 7fe4c1400640 1 --2- 172.21.15.12:0/2416987466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4bc1037c0 0x7fe4bc103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.086+0000 7fe4c1400640 1 -- 172.21.15.12:0/2416987466 >> 172.21.15.12:0/2416987466 conn(0x7fe4bc0fed50 msgr2=0x7fe4bc101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:58.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.086+0000 7fe4c1400640 1 -- 172.21.15.12:0/2416987466 shutdown_connections 2024-03-23T23:11:58.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.086+0000 7fe4c1400640 1 -- 172.21.15.12:0/2416987466 wait complete. 2024-03-23T23:11:58.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.087+0000 7fe4c1400640 1 Processor -- start 2024-03-23T23:11:58.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.087+0000 7fe4c1400640 1 -- start start 2024-03-23T23:11:58.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.087+0000 7fe4c1400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4bc1037c0 0x7fe4bc199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.087+0000 7fe4c1400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4bc1049b0 0x7fe4bc19a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.087+0000 7fe4c1400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe4bc105370 0x7fe4bc1a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.087+0000 7fe4c1400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4bc10f470 con 0x7fe4bc1049b0 2024-03-23T23:11:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.087+0000 7fe4c1400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4bc10f2f0 con 0x7fe4bc105370 2024-03-23T23:11:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.087+0000 7fe4c1400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4bc10f5f0 con 0x7fe4bc1037c0 2024-03-23T23:11:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.087+0000 7fe4c0a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe4bc105370 0x7fe4bc1a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.087+0000 7fe4bbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4bc1037c0 0x7fe4bc199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.087+0000 7fe4c0a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe4bc105370 0x7fe4bc1a1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:51624/0 (socket says 172.21.15.12:51624) 2024-03-23T23:11:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.087+0000 7fe4c0a00640 1 -- 172.21.15.12:0/3027549092 learned_addr learned my addr 172.21.15.12:0/3027549092 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.087+0000 7fe4b3400640 1 --2- 172.21.15.12:0/3027549092 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4bc1049b0 0x7fe4bc19a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.088+0000 7fe4c0a00640 1 -- 172.21.15.12:0/3027549092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4bc1037c0 msgr2=0x7fe4bc199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.088+0000 7fe4c0a00640 1 --2- 172.21.15.12:0/3027549092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4bc1037c0 0x7fe4bc199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.088+0000 7fe4c0a00640 1 -- 172.21.15.12:0/3027549092 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4bc1049b0 msgr2=0x7fe4bc19a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.088+0000 7fe4c0a00640 1 --2- 172.21.15.12:0/3027549092 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4bc1049b0 0x7fe4bc19a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.088+0000 7fe4c0a00640 1 -- 172.21.15.12:0/3027549092 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4bc1a16e0 con 0x7fe4bc105370 2024-03-23T23:11:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.088+0000 7fe4bbe00640 1 --2- 172.21.15.12:0/3027549092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4bc1037c0 0x7fe4bc199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.088+0000 7fe4c0a00640 1 --2- 172.21.15.12:0/3027549092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe4bc105370 0x7fe4bc1a1070 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fe4b400efd0 tx=0x7fe4b400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.088+0000 7fe4b9600640 1 -- 172.21.15.12:0/3027549092 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b4003b70 con 0x7fe4bc105370 2024-03-23T23:11:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.088+0000 7fe4b9600640 1 -- 172.21.15.12:0/3027549092 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4b4003d10 con 0x7fe4bc105370 2024-03-23T23:11:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.088+0000 7fe4b9600640 1 -- 172.21.15.12:0/3027549092 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b40109f0 con 0x7fe4bc105370 2024-03-23T23:11:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.088+0000 7fe4c1400640 1 -- 172.21.15.12:0/3027549092 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4bc1a19d0 con 0x7fe4bc105370 2024-03-23T23:11:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.089+0000 7fe4c1400640 1 -- 172.21.15.12:0/3027549092 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4bc1a1ea0 con 0x7fe4bc105370 2024-03-23T23:11:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.091+0000 7fe4b9600640 1 -- 172.21.15.12:0/3027549092 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe4b4004020 con 0x7fe4bc105370 2024-03-23T23:11:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.092+0000 7fe4c5651740 1 Processor -- start 2024-03-23T23:11:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.092+0000 7fe4c5651740 1 -- start start 2024-03-23T23:11:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.092+0000 7fe4c5651740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5619e62009f0 0x5619e6200e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.092+0000 7fe4c5651740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e6201bc0 0x5619e6202040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.092+0000 7fe4c5651740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5619e6202580 0x5619e6208e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.092+0000 7fe4c5651740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619e61652c0 con 0x5619e6202580 2024-03-23T23:11:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.092+0000 7fe4c5651740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619e610dc90 con 0x5619e6201bc0 2024-03-23T23:11:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.092+0000 7fe4c5651740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619e6161360 con 0x5619e62009f0 2024-03-23T23:11:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.092+0000 7fe4bbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e6201bc0 0x5619e6202040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.092+0000 7fe4bbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e6201bc0 0x5619e6202040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:51634/0 (socket says 172.21.15.12:51634) 2024-03-23T23:11:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.092+0000 7fe4bbe00640 1 -- 172.21.15.12:0/1130049097 learned_addr learned my addr 172.21.15.12:0/1130049097 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.092+0000 7fe4b3400640 1 --2- 172.21.15.12:0/1130049097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5619e62009f0 0x5619e6200e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.092+0000 7fe4b3400640 1 --2- 172.21.15.12:0/1130049097 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5619e6202580 0x5619e6208e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.093+0000 7fe4bbe00640 1 -- 172.21.15.12:0/1130049097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5619e62009f0 msgr2=0x5619e6200e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.093+0000 7fe4bbe00640 1 --2- 172.21.15.12:0/1130049097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5619e62009f0 0x5619e6200e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.093+0000 7fe4bbe00640 1 -- 172.21.15.12:0/1130049097 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5619e6202580 msgr2=0x5619e6208e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.093+0000 7fe4bbe00640 1 --2- 172.21.15.12:0/1130049097 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5619e6202580 0x5619e6208e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.093+0000 7fe4bbe00640 1 -- 172.21.15.12:0/1130049097 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619e6132080 con 0x5619e6201bc0 2024-03-23T23:11:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.093+0000 7fe4b3400640 1 --2- 172.21.15.12:0/1130049097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5619e62009f0 0x5619e6200e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:11:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.093+0000 7fe4bbe00640 1 --2- 172.21.15.12:0/1130049097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e6201bc0 0x5619e6202040 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac0114c0 tx=0x7fe4ac011980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c322a7de23ee78d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.093+0000 7fe4b9600640 1 --2- 172.21.15.12:0/3027549092 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe4a00a2880 0x7fe4a00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.093+0000 7fe4b1200640 1 -- 172.21.15.12:0/1130049097 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac004020 con 0x5619e6201bc0 2024-03-23T23:11:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.093+0000 7fe4b1200640 1 -- 172.21.15.12:0/1130049097 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4ac005500 con 0x5619e6201bc0 2024-03-23T23:11:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.093+0000 7fe4b1200640 1 -- 172.21.15.12:0/1130049097 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac004e10 con 0x5619e6201bc0 2024-03-23T23:11:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.093+0000 7fe4b9600640 1 -- 172.21.15.12:0/3027549092 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe4b40ceea0 con 0x7fe4bc105370 2024-03-23T23:11:58.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.093+0000 7fe4b3400640 1 --2- 172.21.15.12:0/3027549092 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe4a00a2880 0x7fe4a00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.094+0000 7fe4c5651740 1 -- 172.21.15.12:0/1130049097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e6201bc0 msgr2=0x5619e6202040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.094+0000 7fe4c5651740 1 --2- 172.21.15.12:0/1130049097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e6201bc0 0x5619e6202040 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac0114c0 tx=0x7fe4ac011980 comp rx=0 tx=0).stop 2024-03-23T23:11:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.094+0000 7fe4c5651740 1 -- 172.21.15.12:0/1130049097 shutdown_connections 2024-03-23T23:11:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.094+0000 7fe4c5651740 1 --2- 172.21.15.12:0/1130049097 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5619e6202580 0x5619e6208e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.094+0000 7fe4c5651740 1 --2- 172.21.15.12:0/1130049097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e6201bc0 0x5619e6202040 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.094+0000 7fe4c5651740 1 --2- 172.21.15.12:0/1130049097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5619e62009f0 0x5619e6200e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.094+0000 7fe4c5651740 1 -- 172.21.15.12:0/1130049097 >> 172.21.15.12:0/1130049097 conn(0x5619e61fbde0 msgr2=0x5619e61fc1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.094+0000 7fe4c5651740 1 -- 172.21.15.12:0/1130049097 shutdown_connections 2024-03-23T23:11:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.094+0000 7fe4b3400640 1 --2- 172.21.15.12:0/3027549092 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe4a00a2880 0x7fe4a00a4d40 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe4a402c240 tx=0x7fe4a402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.094+0000 7fe4c5651740 1 -- 172.21.15.12:0/1130049097 wait complete. 2024-03-23T23:11:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.095+0000 7fe4c5651740 1 -- start start 2024-03-23T23:11:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.095+0000 7fe4c5651740 1 Processor -- start 2024-03-23T23:11:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.095+0000 7fe4c5651740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5619e62009f0 0x5619e620abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.095+0000 7fe4c5651740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5619e6201bc0 0x5619e620b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.095+0000 7fe4c5651740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e6202580 0x5619e6212160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.095+0000 7fe4c5651740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619e6209590 con 0x5619e62009f0 2024-03-23T23:11:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.095+0000 7fe4c5651740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619e61210d0 con 0x5619e6202580 2024-03-23T23:11:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.095+0000 7fe4c5651740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619e6209710 con 0x5619e6201bc0 2024-03-23T23:11:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.096+0000 7fe4c5651740 10 client.0 _ll_register_callbacks cb 0x5619e6125e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.096+0000 7fe4b3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5619e6201bc0 0x5619e620b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.096+0000 7fe4bbe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5619e62009f0 0x5619e620abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.096+0000 7fe4c0a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e6202580 0x5619e6212160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.096+0000 7fe4b3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5619e6201bc0 0x5619e620b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:52704/0 (socket says 172.21.15.12:52704) 2024-03-23T23:11:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.096+0000 7fe4b3400640 1 -- 172.21.15.12:0/2602797792 learned_addr learned my addr 172.21.15.12:0/2602797792 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.096+0000 7fe4c0a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e6202580 0x5619e6212160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:51636/0 (socket says 172.21.15.12:51636) 2024-03-23T23:11:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.096+0000 7fe4c0a00640 1 -- 172.21.15.12:0/2602797792 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5619e6201bc0 msgr2=0x5619e620b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.096+0000 7fe4c0a00640 1 --2- 172.21.15.12:0/2602797792 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5619e6201bc0 0x5619e620b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.096+0000 7fe4c0a00640 1 -- 172.21.15.12:0/2602797792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5619e62009f0 msgr2=0x5619e620abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.096+0000 7fe4c0a00640 1 --2- 172.21.15.12:0/2602797792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5619e62009f0 0x5619e620abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.096+0000 7fe4c0a00640 1 -- 172.21.15.12:0/2602797792 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5619e5e793f0 con 0x5619e6202580 2024-03-23T23:11:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.096+0000 7fe4c0a00640 1 --2- 172.21.15.12:0/2602797792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e6202580 0x5619e6212160 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fe4b400cba0 tx=0x7fe4b409b210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ea16cb1b54944ff server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.096+0000 7fe4a9600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:11:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.096+0000 7fe4a9600640 1 -- 172.21.15.12:0/2602797792 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b40b4060 con 0x5619e6202580 2024-03-23T23:11:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.096+0000 7fe4a9600640 1 -- 172.21.15.12:0/2602797792 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe4b4004310 con 0x5619e6202580 2024-03-23T23:11:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.097+0000 7fe4c5651740 1 -- 172.21.15.12:0/2602797792 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe4ac005500 con 0x5619e6202580 2024-03-23T23:11:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.097+0000 7fe4a9600640 1 -- 172.21.15.12:0/2602797792 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4b409b7f0 con 0x5619e6202580 2024-03-23T23:11:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.097+0000 7fe4c5651740 10 client.28184 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.097+0000 7fe4c5651740 1 -- 172.21.15.12:0/2602797792 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5619e5f437d0 con 0x5619e6202580 2024-03-23T23:11:58.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.097+0000 7fe4a9600640 1 -- 172.21.15.12:0/2602797792 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4ac005500 con 0x5619e6202580 2024-03-23T23:11:58.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.097+0000 7fe4a9600640 1 -- 172.21.15.12:0/2602797792 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe4b40bb040 con 0x5619e6202580 2024-03-23T23:11:58.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.097+0000 7fe4c5651740 10 client.28184 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:58.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.097+0000 7fe4c5651740 10 client.28184 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:11:58.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.097+0000 7fe4c5651740 20 client.28184 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:58.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.097+0000 7fe4c5651740 1 --2- 172.21.15.12:0/2602797792 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5619e6216580 0x5619e6218a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:58.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.097+0000 7fe4c5651740 4 client.28184 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:11:58.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.097+0000 7fe4c5651740 1 -- 172.21.15.12:0/2602797792 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5619e6172800 con 0x5619e6216580 2024-03-23T23:11:58.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.098+0000 7fe4bbe00640 1 --2- 172.21.15.12:0/2602797792 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5619e6216580 0x5619e6218a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:58.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.098+0000 7fe4bbe00640 1 --2- 172.21.15.12:0/2602797792 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5619e6216580 0x5619e6218a20 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d1024300933b10 server_cookie=d8b2ad6512643cf6 in_seq=0 out_seq=0 2024-03-23T23:11:58.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.098+0000 7fe4a9600640 10 client.28184 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:58.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.099+0000 7fe4a9600640 1 -- 172.21.15.12:0/2602797792 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5619e6172800 con 0x5619e6216580 2024-03-23T23:11:58.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.099+0000 7fe4a9600640 10 client.28184 handle_command_reply: tid=0 2024-03-23T23:11:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.099+0000 7fe4c5651740 1 client.28184 shutdown 2024-03-23T23:11:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.099+0000 7fe4c5651740 1 -- 172.21.15.12:0/2602797792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e6202580 msgr2=0x5619e6212160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.099+0000 7fe4c5651740 1 --2- 172.21.15.12:0/2602797792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e6202580 0x5619e6212160 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fe4b400cba0 tx=0x7fe4b409b210 comp rx=0 tx=0).stop 2024-03-23T23:11:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.099+0000 7fe4c5651740 1 -- 172.21.15.12:0/2602797792 shutdown_connections 2024-03-23T23:11:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.099+0000 7fe4c5651740 1 --2- 172.21.15.12:0/2602797792 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5619e6216580 0x5619e6218a20 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.099+0000 7fe4c5651740 1 --2- 172.21.15.12:0/2602797792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e6202580 0x5619e6212160 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.099+0000 7fe4c5651740 1 --2- 172.21.15.12:0/2602797792 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5619e6201bc0 0x5619e620b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.099+0000 7fe4a9600640 0 client.28184 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.099+0000 7fe4c5651740 1 --2- 172.21.15.12:0/2602797792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5619e62009f0 0x5619e620abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.099+0000 7fe4c5651740 1 -- 172.21.15.12:0/2602797792 >> 172.21.15.12:0/2602797792 conn(0x5619e61fbde0 msgr2=0x5619e61fb160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.100+0000 7fe4c5651740 1 -- 172.21.15.12:0/2602797792 shutdown_connections 2024-03-23T23:11:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.100+0000 7fe4c5651740 1 -- 172.21.15.12:0/2602797792 wait complete. 2024-03-23T23:11:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.100+0000 7fe4c5651740 20 client.28184 trim_cache size 0 max 16384 2024-03-23T23:11:58.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.113+0000 7fe4c5651740 1 Processor -- start 2024-03-23T23:11:58.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.113+0000 7fe4c5651740 1 -- start start 2024-03-23T23:11:58.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.114+0000 7fe4c5651740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e62009f0 0x5619e61ff750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:58.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.114+0000 7fe4c5651740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5619e6201bc0 0x5619e6212120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:58.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.114+0000 7fe4c5651740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5619e6202580 0x5619e6204bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:58.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.114+0000 7fe4c5651740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4bc1ace50 con 0x5619e6202580 2024-03-23T23:11:58.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.114+0000 7fe4c5651740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4bc1aa950 con 0x5619e62009f0 2024-03-23T23:11:58.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.114+0000 7fe4c5651740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4bc1a89f0 con 0x5619e6201bc0 2024-03-23T23:11:58.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.114+0000 7fe4bbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e62009f0 0x5619e61ff750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:58.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.114+0000 7fe4b3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5619e6201bc0 0x5619e6212120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:58.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.114+0000 7fe4bbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e62009f0 0x5619e61ff750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:51652/0 (socket says 172.21.15.12:51652) 2024-03-23T23:11:58.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.114+0000 7fe4bbe00640 1 -- 172.21.15.12:0/4173513830 learned_addr learned my addr 172.21.15.12:0/4173513830 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:58.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.114+0000 7fe4c0a00640 1 --2- 172.21.15.12:0/4173513830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5619e6202580 0x5619e6204bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:58.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.114+0000 7fe4bbe00640 1 -- 172.21.15.12:0/4173513830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5619e6201bc0 msgr2=0x5619e6212120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:58.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.114+0000 7fe4bbe00640 1 --2- 172.21.15.12:0/4173513830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5619e6201bc0 0x5619e6212120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.114+0000 7fe4bbe00640 1 -- 172.21.15.12:0/4173513830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5619e6202580 msgr2=0x5619e6204bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:58.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.114+0000 7fe4bbe00640 1 --2- 172.21.15.12:0/4173513830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5619e6202580 0x5619e6204bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.114+0000 7fe4bbe00640 1 -- 172.21.15.12:0/4173513830 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619e6201640 con 0x5619e62009f0 2024-03-23T23:11:58.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.114+0000 7fe4bbe00640 1 --2- 172.21.15.12:0/4173513830 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e62009f0 0x5619e61ff750 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac009fb0 tx=0x7fe4ac004550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10ba7117bdc44bc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:58.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.115+0000 7fe4abe00640 1 -- 172.21.15.12:0/4173513830 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac030070 con 0x5619e62009f0 2024-03-23T23:11:58.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.115+0000 7fe4abe00640 1 -- 172.21.15.12:0/4173513830 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4ac004640 con 0x5619e62009f0 2024-03-23T23:11:58.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.115+0000 7fe4abe00640 1 -- 172.21.15.12:0/4173513830 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac01d040 con 0x5619e62009f0 2024-03-23T23:11:58.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.115+0000 7fe4c5651740 1 -- 172.21.15.12:0/4173513830 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e62009f0 msgr2=0x5619e61ff750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:58.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.115+0000 7fe4c5651740 1 --2- 172.21.15.12:0/4173513830 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e62009f0 0x5619e61ff750 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac009fb0 tx=0x7fe4ac004550 comp rx=0 tx=0).stop 2024-03-23T23:11:58.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.115+0000 7fe4c5651740 1 -- 172.21.15.12:0/4173513830 shutdown_connections 2024-03-23T23:11:58.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.115+0000 7fe4c5651740 1 --2- 172.21.15.12:0/4173513830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5619e6202580 0x5619e6204bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.115+0000 7fe4c5651740 1 --2- 172.21.15.12:0/4173513830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5619e6201bc0 0x5619e6212120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.115+0000 7fe4c5651740 1 --2- 172.21.15.12:0/4173513830 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e62009f0 0x5619e61ff750 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.115+0000 7fe4c5651740 1 -- 172.21.15.12:0/4173513830 >> 172.21.15.12:0/4173513830 conn(0x5619e61fbde0 msgr2=0x5619e61ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:58.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.115+0000 7fe4c5651740 1 -- 172.21.15.12:0/4173513830 shutdown_connections 2024-03-23T23:11:58.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.115+0000 7fe4c5651740 1 -- 172.21.15.12:0/4173513830 wait complete. 2024-03-23T23:11:58.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.116+0000 7fe4c5651740 1 -- start start 2024-03-23T23:11:58.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.116+0000 7fe4c5651740 1 Processor -- start 2024-03-23T23:11:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.116+0000 7fe4c5651740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e62009f0 0x5619e620dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.116+0000 7fe4c5651740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5619e6201bc0 0x5619e620e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.116+0000 7fe4c5651740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5619e6202580 0x5619e62114e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.116+0000 7fe4c5651740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4bc1a8ff0 con 0x5619e6201bc0 2024-03-23T23:11:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.116+0000 7fe4c5651740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4bc1aa2f0 con 0x5619e62009f0 2024-03-23T23:11:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.116+0000 7fe4c5651740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4bc1a9630 con 0x5619e6202580 2024-03-23T23:11:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.117+0000 7fe4c5651740 10 client.0 _ll_register_callbacks cb 0x7fe4b4009b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.117+0000 7fe4b3400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5619e6201bc0 0x5619e620e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.117+0000 7fe4bbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e62009f0 0x5619e620dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.117+0000 7fe4bbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e62009f0 0x5619e620dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:51662/0 (socket says 172.21.15.12:51662) 2024-03-23T23:11:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.117+0000 7fe4c0a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5619e6202580 0x5619e62114e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.117+0000 7fe4bbe00640 1 -- 172.21.15.12:0/2534365375 learned_addr learned my addr 172.21.15.12:0/2534365375 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.117+0000 7fe4b3400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5619e6201bc0 0x5619e620e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50114/0 (socket says 172.21.15.12:50114) 2024-03-23T23:11:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.117+0000 7fe4bbe00640 1 -- 172.21.15.12:0/2534365375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5619e6202580 msgr2=0x5619e62114e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.117+0000 7fe4bbe00640 1 --2- 172.21.15.12:0/2534365375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5619e6202580 0x5619e62114e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.117+0000 7fe4bbe00640 1 -- 172.21.15.12:0/2534365375 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5619e6201bc0 msgr2=0x5619e620e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:58.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.117+0000 7fe4bbe00640 1 --2- 172.21.15.12:0/2534365375 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5619e6201bc0 0x5619e620e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.117+0000 7fe4bbe00640 1 -- 172.21.15.12:0/2534365375 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5619e6211ad0 con 0x5619e62009f0 2024-03-23T23:11:58.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.117+0000 7fe4c0a00640 1 --2- 172.21.15.12:0/2534365375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5619e6202580 0x5619e62114e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:58.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.117+0000 7fe4bbe00640 1 --2- 172.21.15.12:0/2534365375 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e62009f0 0x5619e620dff0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac011e80 tx=0x7fe4ac005380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=583bc5a5976c3371 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:58.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.117+0000 7fe4a9600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:11:58.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.118+0000 7fe4a9600640 1 -- 172.21.15.12:0/2534365375 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac038060 con 0x5619e62009f0 2024-03-23T23:11:58.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.118+0000 7fe4a9600640 1 -- 172.21.15.12:0/2534365375 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe4ac0047f0 con 0x5619e62009f0 2024-03-23T23:11:58.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.118+0000 7fe4c5651740 1 -- 172.21.15.12:0/2534365375 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe4ac004640 con 0x5619e62009f0 2024-03-23T23:11:58.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.118+0000 7fe4a9600640 1 -- 172.21.15.12:0/2534365375 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4ac030070 con 0x5619e62009f0 2024-03-23T23:11:58.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.118+0000 7fe4c5651740 10 client.28190 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:58.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.118+0000 7fe4c5651740 1 -- 172.21.15.12:0/2534365375 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5619e6272630 con 0x5619e62009f0 2024-03-23T23:11:58.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.118+0000 7fe4a9600640 1 -- 172.21.15.12:0/2534365375 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4ac004640 con 0x5619e62009f0 2024-03-23T23:11:58.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.118+0000 7fe4a9600640 1 -- 172.21.15.12:0/2534365375 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe4ac02b050 con 0x5619e62009f0 2024-03-23T23:11:58.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.118+0000 7fe4c5651740 10 client.28190 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:58.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.118+0000 7fe4c5651740 10 client.28190 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:11:58.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.118+0000 7fe4c5651740 20 client.28190 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:58.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.118+0000 7fe4c5651740 1 --2- 172.21.15.12:0/2534365375 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5619e6272d80 0x5619e6275170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:58.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.118+0000 7fe4c5651740 4 client.28190 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:11:58.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.118+0000 7fe4c5651740 1 -- 172.21.15.12:0/2534365375 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5619e616cb50 con 0x5619e6272d80 2024-03-23T23:11:58.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.119+0000 7fe4b3400640 1 --2- 172.21.15.12:0/2534365375 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5619e6272d80 0x5619e6275170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:58.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.119+0000 7fe4b3400640 1 --2- 172.21.15.12:0/2534365375 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5619e6272d80 0x5619e6275170 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=416dfd413264c180 server_cookie=d67b2cf787a11289 in_seq=0 out_seq=0 2024-03-23T23:11:58.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.119+0000 7fe4a9600640 10 client.28190 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:58.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.119+0000 7fe4a9600640 1 -- 172.21.15.12:0/2534365375 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5619e616cb50 con 0x5619e6272d80 2024-03-23T23:11:58.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.119+0000 7fe4a9600640 10 client.28190 handle_command_reply: tid=0 2024-03-23T23:11:58.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.119+0000 7fe4c5651740 1 client.28190 shutdown 2024-03-23T23:11:58.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.119+0000 7fe4c5651740 1 -- 172.21.15.12:0/2534365375 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e62009f0 msgr2=0x5619e620dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:58.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.119+0000 7fe4c5651740 1 --2- 172.21.15.12:0/2534365375 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e62009f0 0x5619e620dff0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac011e80 tx=0x7fe4ac005380 comp rx=0 tx=0).stop 2024-03-23T23:11:58.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.119+0000 7fe4c5651740 1 -- 172.21.15.12:0/2534365375 shutdown_connections 2024-03-23T23:11:58.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.119+0000 7fe4c5651740 1 --2- 172.21.15.12:0/2534365375 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5619e6272d80 0x5619e6275170 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.119+0000 7fe4c5651740 1 --2- 172.21.15.12:0/2534365375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5619e6202580 0x5619e62114e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.119+0000 7fe4a9600640 0 client.28190 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:58.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.119+0000 7fe4c5651740 1 --2- 172.21.15.12:0/2534365375 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5619e6201bc0 0x5619e620e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.119+0000 7fe4c5651740 1 --2- 172.21.15.12:0/2534365375 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619e62009f0 0x5619e620dff0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.119+0000 7fe4c5651740 1 -- 172.21.15.12:0/2534365375 >> 172.21.15.12:0/2534365375 conn(0x5619e61fbde0 msgr2=0x5619e61ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:58.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.120+0000 7fe4c5651740 1 -- 172.21.15.12:0/2534365375 shutdown_connections 2024-03-23T23:11:58.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.120+0000 7fe4c5651740 1 -- 172.21.15.12:0/2534365375 wait complete. 2024-03-23T23:11:58.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.120+0000 7fe4c5651740 20 client.28190 trim_cache size 0 max 16384 2024-03-23T23:11:58.123 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:11:58.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.120+0000 7fe4c1400640 1 -- 172.21.15.12:0/3027549092 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe4a00a2880 msgr2=0x7fe4a00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:58.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.120+0000 7fe4c1400640 1 --2- 172.21.15.12:0/3027549092 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe4a00a2880 0x7fe4a00a4d40 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe4a402c240 tx=0x7fe4a402df90 comp rx=0 tx=0).stop 2024-03-23T23:11:58.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.120+0000 7fe4c1400640 1 -- 172.21.15.12:0/3027549092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe4bc105370 msgr2=0x7fe4bc1a1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:58.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.120+0000 7fe4c1400640 1 --2- 172.21.15.12:0/3027549092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe4bc105370 0x7fe4bc1a1070 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fe4b400efd0 tx=0x7fe4b400c6a0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.120+0000 7fe4c1400640 1 -- 172.21.15.12:0/3027549092 shutdown_connections 2024-03-23T23:11:58.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.120+0000 7fe4c1400640 1 --2- 172.21.15.12:0/3027549092 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe4a00a2880 0x7fe4a00a4d40 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.120+0000 7fe4c1400640 1 --2- 172.21.15.12:0/3027549092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe4bc105370 0x7fe4bc1a1070 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.120+0000 7fe4c1400640 1 --2- 172.21.15.12:0/3027549092 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4bc1049b0 0x7fe4bc19a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.120+0000 7fe4c1400640 1 --2- 172.21.15.12:0/3027549092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4bc1037c0 0x7fe4bc199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:58.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.120+0000 7fe4c1400640 1 -- 172.21.15.12:0/3027549092 >> 172.21.15.12:0/3027549092 conn(0x7fe4bc0fed50 msgr2=0x7fe4bc107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:58.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.121+0000 7fe4c1400640 1 -- 172.21.15.12:0/3027549092 shutdown_connections 2024-03-23T23:11:58.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:58.121+0000 7fe4c1400640 1 -- 172.21.15.12:0/3027549092 wait complete. 2024-03-23T23:11:58.133 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:57 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1766765918' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:11:58.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:57 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1766765918' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:11:58.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:58 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1766765918' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:11:59.136 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-23T23:11:59.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:11:59.136 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-23T23:11:59.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:11:59 smithi012 ceph-mon[31695]: pgmap v695: 97 pgs: 97 active+clean; 2.8 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:11:59.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:11:59 smithi173 ceph-mon[36116]: pgmap v695: 97 pgs: 97 active+clean; 2.8 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:11:59.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.389+0000 7fe507200640 1 -- 172.21.15.12:0/853455393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5080772a0 msgr2=0x7fe508077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:59.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.389+0000 7fe507200640 1 --2- 172.21.15.12:0/853455393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5080772a0 0x7fe508077700 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4009900 tx=0x7fe4f402bc30 comp rx=0 tx=0).stop 2024-03-23T23:11:59.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.389+0000 7fe507200640 1 -- 172.21.15.12:0/853455393 shutdown_connections 2024-03-23T23:11:59.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.389+0000 7fe507200640 1 --2- 172.21.15.12:0/853455393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe508077c40 0x7fe5080784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.389+0000 7fe507200640 1 --2- 172.21.15.12:0/853455393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5080772a0 0x7fe508077700 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.389+0000 7fe507200640 1 --2- 172.21.15.12:0/853455393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe508076060 0x7fe508076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.389+0000 7fe507200640 1 -- 172.21.15.12:0/853455393 >> 172.21.15.12:0/853455393 conn(0x7fe5080fed10 msgr2=0x7fe508101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:59.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.389+0000 7fe507200640 1 -- 172.21.15.12:0/853455393 shutdown_connections 2024-03-23T23:11:59.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.389+0000 7fe507200640 1 -- 172.21.15.12:0/853455393 wait complete. 2024-03-23T23:11:59.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.390+0000 7fe507200640 1 Processor -- start 2024-03-23T23:11:59.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.390+0000 7fe507200640 1 -- start start 2024-03-23T23:11:59.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.391+0000 7fe507200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe508076060 0x7fe50819df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.391+0000 7fe507200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5080772a0 0x7fe50819e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.391+0000 7fe507200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe508077c40 0x7fe5081a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.391+0000 7fe507200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe508113870 con 0x7fe5080772a0 2024-03-23T23:11:59.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.391+0000 7fe507200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5081136f0 con 0x7fe508076060 2024-03-23T23:11:59.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.391+0000 7fe507200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5081139f0 con 0x7fe508077c40 2024-03-23T23:11:59.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.391+0000 7fe505e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe508076060 0x7fe50819df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.391+0000 7fe505400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5080772a0 0x7fe50819e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.391+0000 7fe506800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe508077c40 0x7fe5081a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.391+0000 7fe505e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe508076060 0x7fe50819df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:51678/0 (socket says 172.21.15.12:51678) 2024-03-23T23:11:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.391+0000 7fe505e00640 1 -- 172.21.15.12:0/1397583398 learned_addr learned my addr 172.21.15.12:0/1397583398 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.391+0000 7fe506800640 1 -- 172.21.15.12:0/1397583398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe508076060 msgr2=0x7fe50819df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.391+0000 7fe506800640 1 --2- 172.21.15.12:0/1397583398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe508076060 0x7fe50819df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.391+0000 7fe506800640 1 -- 172.21.15.12:0/1397583398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5080772a0 msgr2=0x7fe50819e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.391+0000 7fe506800640 1 --2- 172.21.15.12:0/1397583398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5080772a0 0x7fe50819e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.391+0000 7fe506800640 1 -- 172.21.15.12:0/1397583398 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5081a5b20 con 0x7fe508077c40 2024-03-23T23:11:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.391+0000 7fe505e00640 1 --2- 172.21.15.12:0/1397583398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe508076060 0x7fe50819df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.392+0000 7fe506800640 1 --2- 172.21.15.12:0/1397583398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe508077c40 0x7fe5081a54b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fe50000e940 tx=0x7fe50000ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.392+0000 7fe4fea00640 1 -- 172.21.15.12:0/1397583398 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe500004d20 con 0x7fe508077c40 2024-03-23T23:11:59.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.392+0000 7fe4fea00640 1 -- 172.21.15.12:0/1397583398 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe500005020 con 0x7fe508077c40 2024-03-23T23:11:59.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.392+0000 7fe507200640 1 -- 172.21.15.12:0/1397583398 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5081a5e10 con 0x7fe508077c40 2024-03-23T23:11:59.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.392+0000 7fe4fea00640 1 -- 172.21.15.12:0/1397583398 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5000116b0 con 0x7fe508077c40 2024-03-23T23:11:59.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.392+0000 7fe507200640 1 -- 172.21.15.12:0/1397583398 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5081a6280 con 0x7fe508077c40 2024-03-23T23:11:59.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.395+0000 7fe4fea00640 1 -- 172.21.15.12:0/1397583398 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe5000054d0 con 0x7fe508077c40 2024-03-23T23:11:59.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.396+0000 7fe50f82c740 1 Processor -- start 2024-03-23T23:11:59.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.396+0000 7fe50f82c740 1 -- start start 2024-03-23T23:11:59.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.396+0000 7fe50f82c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd8f2122b0 0x55cd8f2126d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.396+0000 7fe50f82c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f212ca0 0x55cd8f2177a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.396+0000 7fe50f82c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd8f217e10 0x55cd8f21a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.396+0000 7fe50f82c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd8f177eb0 con 0x55cd8f217e10 2024-03-23T23:11:59.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.396+0000 7fe50f82c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd8f120a60 con 0x55cd8f2122b0 2024-03-23T23:11:59.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.396+0000 7fe50f82c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd8f173f50 con 0x55cd8f212ca0 2024-03-23T23:11:59.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.396+0000 7fe505400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd8f2122b0 0x55cd8f2126d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.396+0000 7fe505400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd8f217e10 0x55cd8f21a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.396+0000 7fe505e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f212ca0 0x55cd8f2177a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.396+0000 7fe505e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f212ca0 0x55cd8f2177a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:52754/0 (socket says 172.21.15.12:52754) 2024-03-23T23:11:59.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.396+0000 7fe505e00640 1 -- 172.21.15.12:0/1944109669 learned_addr learned my addr 172.21.15.12:0/1944109669 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:59.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.397+0000 7fe505e00640 1 -- 172.21.15.12:0/1944109669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd8f2122b0 msgr2=0x55cd8f2126d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:59.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.397+0000 7fe505e00640 1 --2- 172.21.15.12:0/1944109669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd8f2122b0 0x55cd8f2126d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.397+0000 7fe505e00640 1 -- 172.21.15.12:0/1944109669 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd8f217e10 msgr2=0x55cd8f21a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:59.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.397+0000 7fe505e00640 1 --2- 172.21.15.12:0/1944109669 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd8f217e10 0x55cd8f21a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.397+0000 7fe505e00640 1 -- 172.21.15.12:0/1944109669 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd8ef547e0 con 0x55cd8f212ca0 2024-03-23T23:11:59.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.397+0000 7fe4fea00640 1 --2- 172.21.15.12:0/1397583398 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe4ec0a2800 0x7fe4ec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.397+0000 7fe505e00640 1 --2- 172.21.15.12:0/1944109669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f212ca0 0x55cd8f2177a0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8011650 tx=0x7fe4f8009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15effe4fa68289a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:59.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.398+0000 7fe4fea00640 1 -- 172.21.15.12:0/1397583398 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe5000cea30 con 0x7fe508077c40 2024-03-23T23:11:59.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.398+0000 7fe505400640 1 --2- 172.21.15.12:0/1397583398 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe4ec0a2800 0x7fe4ec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.398+0000 7fe4f3000640 1 -- 172.21.15.12:0/1944109669 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f801e070 con 0x55cd8f212ca0 2024-03-23T23:11:59.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.398+0000 7fe4f3000640 1 -- 172.21.15.12:0/1944109669 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4f800fb40 con 0x55cd8f212ca0 2024-03-23T23:11:59.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.398+0000 7fe4f3000640 1 -- 172.21.15.12:0/1944109669 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f8019cb0 con 0x55cd8f212ca0 2024-03-23T23:11:59.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.398+0000 7fe50f82c740 1 -- 172.21.15.12:0/1944109669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f212ca0 msgr2=0x55cd8f2177a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:59.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.398+0000 7fe505400640 1 --2- 172.21.15.12:0/1397583398 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe4ec0a2800 0x7fe4ec0a4cc0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fe4f402c240 tx=0x7fe4f402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:59.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.398+0000 7fe50f82c740 1 --2- 172.21.15.12:0/1944109669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f212ca0 0x55cd8f2177a0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8011650 tx=0x7fe4f8009f90 comp rx=0 tx=0).stop 2024-03-23T23:11:59.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.399+0000 7fe50f82c740 1 -- 172.21.15.12:0/1944109669 shutdown_connections 2024-03-23T23:11:59.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.399+0000 7fe50f82c740 1 --2- 172.21.15.12:0/1944109669 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd8f217e10 0x55cd8f21a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.399+0000 7fe50f82c740 1 --2- 172.21.15.12:0/1944109669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f212ca0 0x55cd8f2177a0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.399+0000 7fe50f82c740 1 --2- 172.21.15.12:0/1944109669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd8f2122b0 0x55cd8f2126d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.399+0000 7fe50f82c740 1 -- 172.21.15.12:0/1944109669 >> 172.21.15.12:0/1944109669 conn(0x55cd8f20d820 msgr2=0x55cd8f20dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:59.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.399+0000 7fe50f82c740 1 -- 172.21.15.12:0/1944109669 shutdown_connections 2024-03-23T23:11:59.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.399+0000 7fe50f82c740 1 -- 172.21.15.12:0/1944109669 wait complete. 2024-03-23T23:11:59.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.399+0000 7fe50f82c740 1 -- start start 2024-03-23T23:11:59.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.399+0000 7fe50f82c740 1 Processor -- start 2024-03-23T23:11:59.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.400+0000 7fe50f82c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f2122b0 0x55cd8f21c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.400+0000 7fe50f82c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd8f212ca0 0x55cd8f21ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.400+0000 7fe50f82c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd8f217e10 0x55cd8f223a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.400+0000 7fe50f82c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd8f21adc0 con 0x55cd8f212ca0 2024-03-23T23:11:59.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.400+0000 7fe50f82c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd8f133eb0 con 0x55cd8f217e10 2024-03-23T23:11:59.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.400+0000 7fe50f82c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd8f21af40 con 0x55cd8f2122b0 2024-03-23T23:11:59.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.400+0000 7fe50f82c740 10 client.0 _ll_register_callbacks cb 0x55cd8f128150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:59.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.400+0000 7fe505e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f2122b0 0x55cd8f21c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.400+0000 7fe505e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f2122b0 0x55cd8f21c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:52770/0 (socket says 172.21.15.12:52770) 2024-03-23T23:11:59.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.400+0000 7fe505e00640 1 -- 172.21.15.12:0/1059019908 learned_addr learned my addr 172.21.15.12:0/1059019908 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:59.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.400+0000 7fe505400640 1 --2- 172.21.15.12:0/1059019908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd8f212ca0 0x55cd8f21ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.400+0000 7fe506800640 1 --2- 172.21.15.12:0/1059019908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd8f217e10 0x55cd8f223a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.401+0000 7fe505e00640 1 -- 172.21.15.12:0/1059019908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd8f217e10 msgr2=0x55cd8f223a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:59.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.401+0000 7fe505e00640 1 --2- 172.21.15.12:0/1059019908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd8f217e10 0x55cd8f223a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.401+0000 7fe505e00640 1 -- 172.21.15.12:0/1059019908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd8f212ca0 msgr2=0x55cd8f21ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:59.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.401+0000 7fe505e00640 1 --2- 172.21.15.12:0/1059019908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd8f212ca0 0x55cd8f21ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.401+0000 7fe505e00640 1 -- 172.21.15.12:0/1059019908 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cd8efb4f70 con 0x55cd8f2122b0 2024-03-23T23:11:59.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.401+0000 7fe505e00640 1 --2- 172.21.15.12:0/1059019908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f2122b0 0x55cd8f21c4c0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8005e00 tx=0x7fe4f8009ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcbc7ad9015c631b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:59.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.401+0000 7fe4e7400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:11:59.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.401+0000 7fe4e7400640 1 -- 172.21.15.12:0/1059019908 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f8027060 con 0x55cd8f2122b0 2024-03-23T23:11:59.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.401+0000 7fe4e7400640 1 -- 172.21.15.12:0/1059019908 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe4f800f760 con 0x55cd8f2122b0 2024-03-23T23:11:59.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.401+0000 7fe50f82c740 1 -- 172.21.15.12:0/1059019908 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe4f800fb40 con 0x55cd8f2122b0 2024-03-23T23:11:59.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.401+0000 7fe4e7400640 1 -- 172.21.15.12:0/1059019908 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4f801e040 con 0x55cd8f2122b0 2024-03-23T23:11:59.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.402+0000 7fe50f82c740 10 client.28171 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:59.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.402+0000 7fe50f82c740 1 -- 172.21.15.12:0/1059019908 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55cd8ee4c170 con 0x55cd8f2122b0 2024-03-23T23:11:59.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.402+0000 7fe4e7400640 1 -- 172.21.15.12:0/1059019908 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4f800fb40 con 0x55cd8f2122b0 2024-03-23T23:11:59.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.402+0000 7fe4e7400640 1 -- 172.21.15.12:0/1059019908 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe4f8021070 con 0x55cd8f2122b0 2024-03-23T23:11:59.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.402+0000 7fe50f82c740 10 client.28171 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:59.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.402+0000 7fe50f82c740 10 client.28171 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:11:59.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.402+0000 7fe50f82c740 20 client.28171 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:59.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.402+0000 7fe50f82c740 1 --2- 172.21.15.12:0/1059019908 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cd8f228040 0x55cd8f22a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.402+0000 7fe50f82c740 4 client.28171 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:11:59.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.402+0000 7fe50f82c740 1 -- 172.21.15.12:0/1059019908 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cd8f1853f0 con 0x55cd8f228040 2024-03-23T23:11:59.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.402+0000 7fe505400640 1 --2- 172.21.15.12:0/1059019908 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cd8f228040 0x55cd8f22a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.403+0000 7fe505400640 1 --2- 172.21.15.12:0/1059019908 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cd8f228040 0x55cd8f22a470 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82383b5d6aa410ff server_cookie=4b326f800ba40ebb in_seq=0 out_seq=0 2024-03-23T23:11:59.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.403+0000 7fe4e7400640 10 client.28171 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:59.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.404+0000 7fe4e7400640 1 -- 172.21.15.12:0/1059019908 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cd8f1853f0 con 0x55cd8f228040 2024-03-23T23:11:59.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.404+0000 7fe4e7400640 10 client.28171 handle_command_reply: tid=0 2024-03-23T23:11:59.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.404+0000 7fe50f82c740 1 client.28171 shutdown 2024-03-23T23:11:59.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.404+0000 7fe50f82c740 1 -- 172.21.15.12:0/1059019908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f2122b0 msgr2=0x55cd8f21c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:59.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.404+0000 7fe50f82c740 1 --2- 172.21.15.12:0/1059019908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f2122b0 0x55cd8f21c4c0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8005e00 tx=0x7fe4f8009ef0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.404+0000 7fe50f82c740 1 -- 172.21.15.12:0/1059019908 shutdown_connections 2024-03-23T23:11:59.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.404+0000 7fe50f82c740 1 --2- 172.21.15.12:0/1059019908 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cd8f228040 0x55cd8f22a470 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.404+0000 7fe50f82c740 1 --2- 172.21.15.12:0/1059019908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd8f217e10 0x55cd8f223a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.404+0000 7fe50f82c740 1 --2- 172.21.15.12:0/1059019908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd8f212ca0 0x55cd8f21ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.404+0000 7fe50f82c740 1 --2- 172.21.15.12:0/1059019908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f2122b0 0x55cd8f21c4c0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.404+0000 7fe4e7400640 0 client.28171 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:59.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.404+0000 7fe50f82c740 1 -- 172.21.15.12:0/1059019908 >> 172.21.15.12:0/1059019908 conn(0x55cd8f20d820 msgr2=0x55cd8f20ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:59.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.404+0000 7fe50f82c740 1 -- 172.21.15.12:0/1059019908 shutdown_connections 2024-03-23T23:11:59.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.404+0000 7fe50f82c740 1 -- 172.21.15.12:0/1059019908 wait complete. 2024-03-23T23:11:59.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.404+0000 7fe50f82c740 20 client.28171 trim_cache size 0 max 16384 2024-03-23T23:11:59.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:11:59 smithi203 ceph-mon[36693]: pgmap v695: 97 pgs: 97 active+clean; 2.8 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:11:59.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.422+0000 7fe50f82c740 1 Processor -- start 2024-03-23T23:11:59.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.422+0000 7fe50f82c740 1 -- start start 2024-03-23T23:11:59.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.423+0000 7fe50f82c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd8f2122b0 0x55cd8f213e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.423+0000 7fe50f82c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd8f212ca0 0x55cd8f223a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.423+0000 7fe50f82c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f217e10 0x55cd8f214360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.423+0000 7fe50f82c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5081b1420 con 0x55cd8f2122b0 2024-03-23T23:11:59.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.423+0000 7fe50f82c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5081aef20 con 0x55cd8f212ca0 2024-03-23T23:11:59.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.423+0000 7fe50f82c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5081acfc0 con 0x55cd8f217e10 2024-03-23T23:11:59.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.423+0000 7fe505400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd8f212ca0 0x55cd8f223a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.423+0000 7fe506800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f217e10 0x55cd8f214360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.423+0000 7fe505400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd8f212ca0 0x55cd8f223a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:51698/0 (socket says 172.21.15.12:51698) 2024-03-23T23:11:59.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.423+0000 7fe505400640 1 -- 172.21.15.12:0/4235751001 learned_addr learned my addr 172.21.15.12:0/4235751001 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:59.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.423+0000 7fe505e00640 1 --2- 172.21.15.12:0/4235751001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd8f2122b0 0x55cd8f213e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.423+0000 7fe506800640 1 -- 172.21.15.12:0/4235751001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd8f212ca0 msgr2=0x55cd8f223a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:59.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.423+0000 7fe506800640 1 --2- 172.21.15.12:0/4235751001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd8f212ca0 0x55cd8f223a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.423+0000 7fe506800640 1 -- 172.21.15.12:0/4235751001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd8f2122b0 msgr2=0x55cd8f213e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:59.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.423+0000 7fe506800640 1 --2- 172.21.15.12:0/4235751001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd8f2122b0 0x55cd8f213e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.423+0000 7fe506800640 1 -- 172.21.15.12:0/4235751001 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd8f20e590 con 0x55cd8f217e10 2024-03-23T23:11:59.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.423+0000 7fe505e00640 1 --2- 172.21.15.12:0/4235751001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd8f2122b0 0x55cd8f213e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:11:59.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.423+0000 7fe505400640 1 --2- 172.21.15.12:0/4235751001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd8f212ca0 0x55cd8f223a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:59.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.424+0000 7fe506800640 1 --2- 172.21.15.12:0/4235751001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f217e10 0x55cd8f214360 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fe5000cc520 tx=0x7fe500005840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cace46fa2df6562 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:59.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.424+0000 7fe4f2600640 1 -- 172.21.15.12:0/4235751001 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50001f040 con 0x55cd8f217e10 2024-03-23T23:11:59.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.424+0000 7fe4f2600640 1 -- 172.21.15.12:0/4235751001 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe5000b4270 con 0x55cd8f217e10 2024-03-23T23:11:59.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.424+0000 7fe4f2600640 1 -- 172.21.15.12:0/4235751001 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50009f0d0 con 0x55cd8f217e10 2024-03-23T23:11:59.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.424+0000 7fe50f82c740 1 -- 172.21.15.12:0/4235751001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f217e10 msgr2=0x55cd8f214360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:59.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.424+0000 7fe50f82c740 1 --2- 172.21.15.12:0/4235751001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f217e10 0x55cd8f214360 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fe5000cc520 tx=0x7fe500005840 comp rx=0 tx=0).stop 2024-03-23T23:11:59.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.424+0000 7fe50f82c740 1 -- 172.21.15.12:0/4235751001 shutdown_connections 2024-03-23T23:11:59.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.424+0000 7fe50f82c740 1 --2- 172.21.15.12:0/4235751001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f217e10 0x55cd8f214360 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.424+0000 7fe50f82c740 1 --2- 172.21.15.12:0/4235751001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd8f212ca0 0x55cd8f223a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.424+0000 7fe50f82c740 1 --2- 172.21.15.12:0/4235751001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd8f2122b0 0x55cd8f213e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.424+0000 7fe50f82c740 1 -- 172.21.15.12:0/4235751001 >> 172.21.15.12:0/4235751001 conn(0x55cd8f20d820 msgr2=0x55cd8f213710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:59.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.425+0000 7fe50f82c740 1 -- 172.21.15.12:0/4235751001 shutdown_connections 2024-03-23T23:11:59.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.425+0000 7fe50f82c740 1 -- 172.21.15.12:0/4235751001 wait complete. 2024-03-23T23:11:59.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.425+0000 7fe50f82c740 1 -- start start 2024-03-23T23:11:59.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.425+0000 7fe50f82c740 1 Processor -- start 2024-03-23T23:11:59.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.425+0000 7fe50f82c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd8f2122b0 0x55cd8f221940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.425+0000 7fe50f82c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f212ca0 0x55cd8f221e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.426+0000 7fe50f82c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd8f217e10 0x55cd8f222e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.426+0000 7fe50f82c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5081ad5c0 con 0x55cd8f2122b0 2024-03-23T23:11:59.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.426+0000 7fe50f82c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5081ae8c0 con 0x55cd8f217e10 2024-03-23T23:11:59.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.426+0000 7fe50f82c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5081adc00 con 0x55cd8f212ca0 2024-03-23T23:11:59.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.426+0000 7fe50f82c740 10 client.0 _ll_register_callbacks cb 0x7fe4f801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:59.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.426+0000 7fe505400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f212ca0 0x55cd8f221e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.426+0000 7fe505e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd8f2122b0 0x55cd8f221940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.426+0000 7fe505400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f212ca0 0x55cd8f221e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:52782/0 (socket says 172.21.15.12:52782) 2024-03-23T23:11:59.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.426+0000 7fe506800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd8f217e10 0x55cd8f222e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.426+0000 7fe505400640 1 -- 172.21.15.12:0/3252277404 learned_addr learned my addr 172.21.15.12:0/3252277404 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:59.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.426+0000 7fe505400640 1 -- 172.21.15.12:0/3252277404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd8f217e10 msgr2=0x55cd8f222e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:59.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.426+0000 7fe505400640 1 --2- 172.21.15.12:0/3252277404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd8f217e10 0x55cd8f222e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.426+0000 7fe505400640 1 -- 172.21.15.12:0/3252277404 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd8f2122b0 msgr2=0x55cd8f221940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:59.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.426+0000 7fe505400640 1 --2- 172.21.15.12:0/3252277404 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd8f2122b0 0x55cd8f221940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.426+0000 7fe505400640 1 -- 172.21.15.12:0/3252277404 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cd8f223480 con 0x55cd8f212ca0 2024-03-23T23:11:59.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.426+0000 7fe506800640 1 --2- 172.21.15.12:0/3252277404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd8f217e10 0x55cd8f222e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:59.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.426+0000 7fe505400640 1 --2- 172.21.15.12:0/3252277404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f212ca0 0x55cd8f221e80 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4036040 tx=0x7fe4f402bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47acf07a370d4162 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:59.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.426+0000 7fe4e7400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:11:59.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.427+0000 7fe4e7400640 1 -- 172.21.15.12:0/3252277404 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f40572c0 con 0x55cd8f212ca0 2024-03-23T23:11:59.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.427+0000 7fe4e7400640 1 -- 172.21.15.12:0/3252277404 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe4f403a040 con 0x55cd8f212ca0 2024-03-23T23:11:59.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.427+0000 7fe50f82c740 1 -- 172.21.15.12:0/3252277404 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe5000b4270 con 0x55cd8f212ca0 2024-03-23T23:11:59.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.427+0000 7fe4e7400640 1 -- 172.21.15.12:0/3252277404 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4f40050c0 con 0x55cd8f212ca0 2024-03-23T23:11:59.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.427+0000 7fe50f82c740 10 client.28177 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:59.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.427+0000 7fe50f82c740 1 -- 172.21.15.12:0/3252277404 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55cd8f284220 con 0x55cd8f212ca0 2024-03-23T23:11:59.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.427+0000 7fe4e7400640 1 -- 172.21.15.12:0/3252277404 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe5000b4270 con 0x55cd8f212ca0 2024-03-23T23:11:59.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.427+0000 7fe4e7400640 1 -- 172.21.15.12:0/3252277404 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe4f4043440 con 0x55cd8f212ca0 2024-03-23T23:11:59.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.427+0000 7fe50f82c740 10 client.28177 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:59.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.427+0000 7fe50f82c740 10 client.28177 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:11:59.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.427+0000 7fe50f82c740 20 client.28177 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:59.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.427+0000 7fe50f82c740 1 --2- 172.21.15.12:0/3252277404 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cd8f284bb0 0x55cd8f286fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.427+0000 7fe50f82c740 4 client.28177 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-23T23:11:59.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.427+0000 7fe50f82c740 1 -- 172.21.15.12:0/3252277404 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55cd8f17f740 con 0x55cd8f284bb0 2024-03-23T23:11:59.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.428+0000 7fe505e00640 1 --2- 172.21.15.12:0/3252277404 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cd8f284bb0 0x55cd8f286fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.428+0000 7fe505e00640 1 --2- 172.21.15.12:0/3252277404 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cd8f284bb0 0x55cd8f286fd0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e18324634b9d1d server_cookie=4f6b98780420d2c0 in_seq=0 out_seq=0 2024-03-23T23:11:59.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.428+0000 7fe4e7400640 10 client.28177 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:59.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.429+0000 7fe4e7400640 1 -- 172.21.15.12:0/3252277404 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55cd8f17f740 con 0x55cd8f284bb0 2024-03-23T23:11:59.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.429+0000 7fe4e7400640 10 client.28177 handle_command_reply: tid=0 2024-03-23T23:11:59.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.429+0000 7fe50f82c740 1 client.28177 shutdown 2024-03-23T23:11:59.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.430+0000 7fe50f82c740 1 -- 172.21.15.12:0/3252277404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f212ca0 msgr2=0x55cd8f221e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:59.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.430+0000 7fe50f82c740 1 --2- 172.21.15.12:0/3252277404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f212ca0 0x55cd8f221e80 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4036040 tx=0x7fe4f402bc30 comp rx=0 tx=0).stop 2024-03-23T23:11:59.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.430+0000 7fe50f82c740 1 -- 172.21.15.12:0/3252277404 shutdown_connections 2024-03-23T23:11:59.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.430+0000 7fe50f82c740 1 --2- 172.21.15.12:0/3252277404 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cd8f284bb0 0x55cd8f286fd0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.430+0000 7fe50f82c740 1 --2- 172.21.15.12:0/3252277404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd8f217e10 0x55cd8f222e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.430+0000 7fe50f82c740 1 --2- 172.21.15.12:0/3252277404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd8f212ca0 0x55cd8f221e80 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.430+0000 7fe50f82c740 1 --2- 172.21.15.12:0/3252277404 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cd8f2122b0 0x55cd8f221940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.430+0000 7fe4e7400640 0 client.28177 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:59.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.430+0000 7fe50f82c740 1 -- 172.21.15.12:0/3252277404 >> 172.21.15.12:0/3252277404 conn(0x55cd8f20d820 msgr2=0x55cd8f213ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:59.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.430+0000 7fe50f82c740 1 -- 172.21.15.12:0/3252277404 shutdown_connections 2024-03-23T23:11:59.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.430+0000 7fe50f82c740 1 -- 172.21.15.12:0/3252277404 wait complete. 2024-03-23T23:11:59.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.430+0000 7fe50f82c740 20 client.28177 trim_cache size 0 max 16384 2024-03-23T23:11:59.434 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:11:59.434 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-03-23T23:11:59.434 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "3ceefe24-798f-48b2-b64e-17d4162abd30", 2024-03-23T23:11:59.434 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-03-23T23:11:59.434 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:11:59.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.431+0000 7fe507200640 1 -- 172.21.15.12:0/1397583398 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe4ec0a2800 msgr2=0x7fe4ec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:59.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.431+0000 7fe507200640 1 --2- 172.21.15.12:0/1397583398 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe4ec0a2800 0x7fe4ec0a4cc0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fe4f402c240 tx=0x7fe4f402df90 comp rx=0 tx=0).stop 2024-03-23T23:11:59.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.431+0000 7fe507200640 1 -- 172.21.15.12:0/1397583398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe508077c40 msgr2=0x7fe5081a54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:59.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.431+0000 7fe507200640 1 --2- 172.21.15.12:0/1397583398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe508077c40 0x7fe5081a54b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fe50000e940 tx=0x7fe50000ee00 comp rx=0 tx=0).stop 2024-03-23T23:11:59.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.431+0000 7fe507200640 1 -- 172.21.15.12:0/1397583398 shutdown_connections 2024-03-23T23:11:59.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.431+0000 7fe507200640 1 --2- 172.21.15.12:0/1397583398 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe4ec0a2800 0x7fe4ec0a4cc0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.431+0000 7fe507200640 1 --2- 172.21.15.12:0/1397583398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe508077c40 0x7fe5081a54b0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.431+0000 7fe507200640 1 --2- 172.21.15.12:0/1397583398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5080772a0 0x7fe50819e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.431+0000 7fe507200640 1 --2- 172.21.15.12:0/1397583398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe508076060 0x7fe50819df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.431+0000 7fe507200640 1 -- 172.21.15.12:0/1397583398 >> 172.21.15.12:0/1397583398 conn(0x7fe5080fed10 msgr2=0x7fe508100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:59.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.431+0000 7fe507200640 1 -- 172.21.15.12:0/1397583398 shutdown_connections 2024-03-23T23:11:59.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.431+0000 7fe507200640 1 -- 172.21.15.12:0/1397583398 wait complete. 2024-03-23T23:11:59.447 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:11:59.447 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:11:59.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.666+0000 7f3203e00640 1 -- 172.21.15.12:0/1349477734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32041007b0 msgr2=0x7f3204100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:59.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.666+0000 7f3203e00640 1 --2- 172.21.15.12:0/1349477734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32041007b0 0x7f3204100c10 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f31ec009900 tx=0x7f31ec02bc50 comp rx=0 tx=0).stop 2024-03-23T23:11:59.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.666+0000 7f3203e00640 1 -- 172.21.15.12:0/1349477734 shutdown_connections 2024-03-23T23:11:59.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.666+0000 7f3203e00640 1 --2- 172.21.15.12:0/1349477734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3204101150 0x7f32041015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.666+0000 7f3203e00640 1 --2- 172.21.15.12:0/1349477734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32041007b0 0x7f3204100c10 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.666+0000 7f3203e00640 1 --2- 172.21.15.12:0/1349477734 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32040ffe50 0x7f3204100270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.666+0000 7f3203e00640 1 -- 172.21.15.12:0/1349477734 >> 172.21.15.12:0/1349477734 conn(0x7f32040fba10 msgr2=0x7f32040fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:59.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.666+0000 7f3203e00640 1 -- 172.21.15.12:0/1349477734 shutdown_connections 2024-03-23T23:11:59.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.666+0000 7f3203e00640 1 -- 172.21.15.12:0/1349477734 wait complete. 2024-03-23T23:11:59.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.667+0000 7f3203e00640 1 Processor -- start 2024-03-23T23:11:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.667+0000 7f3203e00640 1 -- start start 2024-03-23T23:11:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.667+0000 7f3203e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32040ffe50 0x7f3204199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.668+0000 7f3203e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32041007b0 0x7f3204199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.668+0000 7f3203e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3204101150 0x7f32041a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.668+0000 7f3203e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320410f3c0 con 0x7f32040ffe50 2024-03-23T23:11:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.668+0000 7f3203e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320410f240 con 0x7f32041007b0 2024-03-23T23:11:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.668+0000 7f3203e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320410f540 con 0x7f3204101150 2024-03-23T23:11:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.668+0000 7f3202a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32040ffe50 0x7f3204199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.668+0000 7f3202000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32041007b0 0x7f3204199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.668+0000 7f3202a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32040ffe50 0x7f3204199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50192/0 (socket says 172.21.15.12:50192) 2024-03-23T23:11:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.668+0000 7f3202a00640 1 -- 172.21.15.12:0/988833356 learned_addr learned my addr 172.21.15.12:0/988833356 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.668+0000 7f3203400640 1 --2- 172.21.15.12:0/988833356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3204101150 0x7f32041a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.668+0000 7f3202a00640 1 -- 172.21.15.12:0/988833356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3204101150 msgr2=0x7f32041a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.668+0000 7f3202a00640 1 --2- 172.21.15.12:0/988833356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3204101150 0x7f32041a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.668+0000 7f3202a00640 1 -- 172.21.15.12:0/988833356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32041007b0 msgr2=0x7f3204199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:59.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.668+0000 7f3202a00640 1 --2- 172.21.15.12:0/988833356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32041007b0 0x7f3204199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.668+0000 7f3202a00640 1 -- 172.21.15.12:0/988833356 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32041a16d0 con 0x7f32040ffe50 2024-03-23T23:11:59.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.668+0000 7f3202000640 1 --2- 172.21.15.12:0/988833356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32041007b0 0x7f3204199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:11:59.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.669+0000 7f3202a00640 1 --2- 172.21.15.12:0/988833356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32040ffe50 0x7f3204199ab0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f31f400ebc0 tx=0x7f31f400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:59.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.669+0000 7f31fb400640 1 -- 172.21.15.12:0/988833356 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f4010630 con 0x7f32040ffe50 2024-03-23T23:11:59.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.669+0000 7f31fb400640 1 -- 172.21.15.12:0/988833356 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f31f4010cb0 con 0x7f32040ffe50 2024-03-23T23:11:59.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.669+0000 7f3203e00640 1 -- 172.21.15.12:0/988833356 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32041a19c0 con 0x7f32040ffe50 2024-03-23T23:11:59.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.669+0000 7f31fb400640 1 -- 172.21.15.12:0/988833356 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f4018bd0 con 0x7f32040ffe50 2024-03-23T23:11:59.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.669+0000 7f3203e00640 1 -- 172.21.15.12:0/988833356 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32041a1ee0 con 0x7f32040ffe50 2024-03-23T23:11:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.672+0000 7f31fb400640 1 -- 172.21.15.12:0/988833356 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f31f40107d0 con 0x7f32040ffe50 2024-03-23T23:11:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.672+0000 7f320c5c4740 1 Processor -- start 2024-03-23T23:11:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.672+0000 7f320c5c4740 1 -- start start 2024-03-23T23:11:59.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.673+0000 7f320c5c4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684ae5050 0x563684ae5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.673+0000 7f320c5c4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563684ae5a40 0x563684aea540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.673+0000 7f320c5c4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563684aeabb0 0x563684aed040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.673+0000 7f320c5c4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563684a4a250 con 0x563684aeabb0 2024-03-23T23:11:59.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.673+0000 7f320c5c4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636849f2a30 con 0x563684ae5050 2024-03-23T23:11:59.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.673+0000 7f320c5c4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x563684a462f0 con 0x563684ae5a40 2024-03-23T23:11:59.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.673+0000 7f3203400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684ae5050 0x563684ae5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.673+0000 7f3203400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684ae5050 0x563684ae5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:51736/0 (socket says 172.21.15.12:51736) 2024-03-23T23:11:59.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.673+0000 7f3203400640 1 -- 172.21.15.12:0/1737200508 learned_addr learned my addr 172.21.15.12:0/1737200508 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:59.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.673+0000 7f3202a00640 1 --2- 172.21.15.12:0/1737200508 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563684ae5a40 0x563684aea540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.673+0000 7f3202000640 1 --2- 172.21.15.12:0/1737200508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563684aeabb0 0x563684aed040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.673+0000 7f3203400640 1 -- 172.21.15.12:0/1737200508 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563684ae5a40 msgr2=0x563684aea540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.673+0000 7f3203400640 1 --2- 172.21.15.12:0/1737200508 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563684ae5a40 0x563684aea540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.673+0000 7f3203400640 1 -- 172.21.15.12:0/1737200508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563684aeabb0 msgr2=0x563684aed040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.673+0000 7f3203400640 1 --2- 172.21.15.12:0/1737200508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563684aeabb0 0x563684aed040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.673+0000 7f3203400640 1 -- 172.21.15.12:0/1737200508 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636849dd690 con 0x563684ae5050 2024-03-23T23:11:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.673+0000 7f3202000640 1 --2- 172.21.15.12:0/1737200508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563684aeabb0 0x563684aed040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:11:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.674+0000 7f3203400640 1 --2- 172.21.15.12:0/1737200508 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684ae5050 0x563684ae5470 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f31fc009ca0 tx=0x7f31fc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45b3caaca6b02975 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.674+0000 7f31fb400640 1 --2- 172.21.15.12:0/988833356 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f31e80a2800 0x7f31e80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.674+0000 7f31f3e00640 1 -- 172.21.15.12:0/1737200508 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc01e070 con 0x563684ae5050 2024-03-23T23:11:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.674+0000 7f31f3e00640 1 -- 172.21.15.12:0/1737200508 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f31fc009e50 con 0x563684ae5050 2024-03-23T23:11:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.674+0000 7f31f3e00640 1 -- 172.21.15.12:0/1737200508 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc00eac0 con 0x563684ae5050 2024-03-23T23:11:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.674+0000 7f31fb400640 1 -- 172.21.15.12:0/988833356 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f31f40cf030 con 0x7f32040ffe50 2024-03-23T23:11:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.674+0000 7f3202a00640 1 --2- 172.21.15.12:0/988833356 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f31e80a2800 0x7f31e80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.675+0000 7f320c5c4740 1 -- 172.21.15.12:0/1737200508 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684ae5050 msgr2=0x563684ae5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:59.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.675+0000 7f320c5c4740 1 --2- 172.21.15.12:0/1737200508 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684ae5050 0x563684ae5470 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f31fc009ca0 tx=0x7f31fc011810 comp rx=0 tx=0).stop 2024-03-23T23:11:59.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.675+0000 7f3202a00640 1 --2- 172.21.15.12:0/988833356 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f31e80a2800 0x7f31e80a4cc0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f31f40cceb0 tx=0x7f31f400cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:59.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.675+0000 7f320c5c4740 1 -- 172.21.15.12:0/1737200508 shutdown_connections 2024-03-23T23:11:59.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.675+0000 7f320c5c4740 1 --2- 172.21.15.12:0/1737200508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563684aeabb0 0x563684aed040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.675+0000 7f320c5c4740 1 --2- 172.21.15.12:0/1737200508 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563684ae5a40 0x563684aea540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.675+0000 7f320c5c4740 1 --2- 172.21.15.12:0/1737200508 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684ae5050 0x563684ae5470 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.675+0000 7f320c5c4740 1 -- 172.21.15.12:0/1737200508 >> 172.21.15.12:0/1737200508 conn(0x563684aad200 msgr2=0x563684ae09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:59.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.675+0000 7f320c5c4740 1 -- 172.21.15.12:0/1737200508 shutdown_connections 2024-03-23T23:11:59.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.675+0000 7f320c5c4740 1 -- 172.21.15.12:0/1737200508 wait complete. 2024-03-23T23:11:59.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.676+0000 7f320c5c4740 1 -- start start 2024-03-23T23:11:59.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.676+0000 7f320c5c4740 1 Processor -- start 2024-03-23T23:11:59.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.676+0000 7f320c5c4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684ae5050 0x563684aef090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.676+0000 7f320c5c4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563684ae5a40 0x563684aef5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.676+0000 7f320c5c4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563684aeabb0 0x563684af6610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.676+0000 7f320c5c4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563684aedb60 con 0x563684aeabb0 2024-03-23T23:11:59.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.676+0000 7f320c5c4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x563684a05e80 con 0x563684ae5050 2024-03-23T23:11:59.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.676+0000 7f320c5c4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x563684aedce0 con 0x563684ae5a40 2024-03-23T23:11:59.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.677+0000 7f320c5c4740 10 client.0 _ll_register_callbacks cb 0x563684a18130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:59.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.677+0000 7f3203400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684ae5050 0x563684aef090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.677+0000 7f3203400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684ae5050 0x563684aef090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:51748/0 (socket says 172.21.15.12:51748) 2024-03-23T23:11:59.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.677+0000 7f3203400640 1 -- 172.21.15.12:0/2739068467 learned_addr learned my addr 172.21.15.12:0/2739068467 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:59.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.677+0000 7f3202000640 1 --2- 172.21.15.12:0/2739068467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563684ae5a40 0x563684aef5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.677+0000 7f3203400640 1 --2- 172.21.15.12:0/2739068467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563684aeabb0 0x563684af6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.677+0000 7f3202000640 1 -- 172.21.15.12:0/2739068467 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684ae5050 msgr2=0x563684aef090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:59.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.677+0000 7f3202000640 1 --2- 172.21.15.12:0/2739068467 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684ae5050 0x563684aef090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.677+0000 7f3202000640 1 -- 172.21.15.12:0/2739068467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563684aeabb0 msgr2=0x563684af6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:59.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.677+0000 7f3202000640 1 --2- 172.21.15.12:0/2739068467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563684aeabb0 0x563684af6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.677+0000 7f3202000640 1 -- 172.21.15.12:0/2739068467 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5636848287d0 con 0x563684ae5a40 2024-03-23T23:11:59.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.677+0000 7f3203400640 1 --2- 172.21.15.12:0/2739068467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563684aeabb0 0x563684af6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:11:59.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.677+0000 7f3202000640 1 --2- 172.21.15.12:0/2739068467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563684ae5a40 0x563684aef5d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f31ec009230 tx=0x7f31ec02c5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3387d62e67de6d19 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:59.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.677+0000 7f31f0c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:11:59.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.678+0000 7f31f0c00640 1 -- 172.21.15.12:0/2739068467 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec043030 con 0x563684ae5a40 2024-03-23T23:11:59.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.678+0000 7f31f0c00640 1 -- 172.21.15.12:0/2739068467 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f31ec005250 con 0x563684ae5a40 2024-03-23T23:11:59.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.678+0000 7f320c5c4740 1 -- 172.21.15.12:0/2739068467 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f31fc009e50 con 0x563684ae5a40 2024-03-23T23:11:59.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.678+0000 7f31f0c00640 1 -- 172.21.15.12:0/2739068467 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f31ec005450 con 0x563684ae5a40 2024-03-23T23:11:59.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.678+0000 7f320c5c4740 10 client.28189 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:59.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.678+0000 7f320c5c4740 1 -- 172.21.15.12:0/2739068467 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56368475e3f0 con 0x563684ae5a40 2024-03-23T23:11:59.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.678+0000 7f31f0c00640 1 -- 172.21.15.12:0/2739068467 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f31fc009e50 con 0x563684ae5a40 2024-03-23T23:11:59.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.679+0000 7f31f0c00640 1 -- 172.21.15.12:0/2739068467 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f31ec003680 con 0x563684ae5a40 2024-03-23T23:11:59.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.679+0000 7f320c5c4740 10 client.28189 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:59.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.679+0000 7f320c5c4740 10 client.28189 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:11:59.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.679+0000 7f320c5c4740 20 client.28189 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:59.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.679+0000 7f320c5c4740 1 --2- 172.21.15.12:0/2739068467 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563684afaf40 0x563684afd360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.679+0000 7f320c5c4740 4 client.28189 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:11:59.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.679+0000 7f320c5c4740 1 -- 172.21.15.12:0/2739068467 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563684a57790 con 0x563684afaf40 2024-03-23T23:11:59.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.679+0000 7f3203400640 1 --2- 172.21.15.12:0/2739068467 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563684afaf40 0x563684afd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.679+0000 7f3203400640 1 --2- 172.21.15.12:0/2739068467 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563684afaf40 0x563684afd360 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96ed40e588cc9932 server_cookie=17dee0d8737c2238 in_seq=0 out_seq=0 2024-03-23T23:11:59.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.679+0000 7f31f0c00640 10 client.28189 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:59.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.680+0000 7f31f0c00640 1 -- 172.21.15.12:0/2739068467 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563684a57790 con 0x563684afaf40 2024-03-23T23:11:59.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.680+0000 7f31f0c00640 10 client.28189 handle_command_reply: tid=0 2024-03-23T23:11:59.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.680+0000 7f320c5c4740 1 client.28189 shutdown 2024-03-23T23:11:59.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.680+0000 7f320c5c4740 1 -- 172.21.15.12:0/2739068467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563684ae5a40 msgr2=0x563684aef5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:59.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.680+0000 7f320c5c4740 1 --2- 172.21.15.12:0/2739068467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563684ae5a40 0x563684aef5d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f31ec009230 tx=0x7f31ec02c5f0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.680+0000 7f320c5c4740 1 -- 172.21.15.12:0/2739068467 shutdown_connections 2024-03-23T23:11:59.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.680+0000 7f320c5c4740 1 --2- 172.21.15.12:0/2739068467 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563684afaf40 0x563684afd360 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.680+0000 7f320c5c4740 1 --2- 172.21.15.12:0/2739068467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563684aeabb0 0x563684af6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.680+0000 7f320c5c4740 1 --2- 172.21.15.12:0/2739068467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563684ae5a40 0x563684aef5d0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.680+0000 7f31f0c00640 0 client.28189 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:59.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.680+0000 7f320c5c4740 1 --2- 172.21.15.12:0/2739068467 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684ae5050 0x563684aef090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.680+0000 7f320c5c4740 1 -- 172.21.15.12:0/2739068467 >> 172.21.15.12:0/2739068467 conn(0x563684aad200 msgr2=0x563684adfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:59.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.681+0000 7f320c5c4740 1 -- 172.21.15.12:0/2739068467 shutdown_connections 2024-03-23T23:11:59.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.681+0000 7f320c5c4740 1 -- 172.21.15.12:0/2739068467 wait complete. 2024-03-23T23:11:59.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.681+0000 7f320c5c4740 20 client.28189 trim_cache size 0 max 16384 2024-03-23T23:11:59.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.698+0000 7f320c5c4740 1 Processor -- start 2024-03-23T23:11:59.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.698+0000 7f320c5c4740 1 -- start start 2024-03-23T23:11:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.699+0000 7f320c5c4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563684ae5050 0x563684adb920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.699+0000 7f320c5c4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684ae5a40 0x563684af65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.699+0000 7f320c5c4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563684aeabb0 0x563684adf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.699+0000 7f320c5c4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32041acf70 con 0x563684ae5050 2024-03-23T23:11:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.699+0000 7f320c5c4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32041aaa70 con 0x563684ae5a40 2024-03-23T23:11:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.699+0000 7f320c5c4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32041a8b10 con 0x563684aeabb0 2024-03-23T23:11:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.699+0000 7f3202000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684ae5a40 0x563684af65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.699+0000 7f3202000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684ae5a40 0x563684af65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:51758/0 (socket says 172.21.15.12:51758) 2024-03-23T23:11:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.699+0000 7f3202000640 1 -- 172.21.15.12:0/2694806010 learned_addr learned my addr 172.21.15.12:0/2694806010 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.699+0000 7f3203400640 1 --2- 172.21.15.12:0/2694806010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563684ae5050 0x563684adb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.699+0000 7f3202000640 1 -- 172.21.15.12:0/2694806010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563684aeabb0 msgr2=0x563684adf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.699+0000 7f3203400640 1 --2- 172.21.15.12:0/2694806010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563684aeabb0 0x563684adf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.699+0000 7f3202000640 1 --2- 172.21.15.12:0/2694806010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563684aeabb0 0x563684adf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.699+0000 7f3202000640 1 -- 172.21.15.12:0/2694806010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563684ae5050 msgr2=0x563684adb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.699+0000 7f3202000640 1 --2- 172.21.15.12:0/2694806010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563684ae5050 0x563684adb920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.699+0000 7f3202000640 1 -- 172.21.15.12:0/2694806010 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56368471e0f0 con 0x563684ae5a40 2024-03-23T23:11:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.699+0000 7f3203400640 1 --2- 172.21.15.12:0/2694806010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563684aeabb0 0x563684adf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:11:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.700+0000 7f3202000640 1 --2- 172.21.15.12:0/2694806010 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684ae5a40 0x563684af65d0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f31ec02c190 tx=0x7f31ec005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97c7278f48d24536 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.700+0000 7f31f3400640 1 -- 172.21.15.12:0/2694806010 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec058070 con 0x563684ae5a40 2024-03-23T23:11:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.700+0000 7f31f3400640 1 -- 172.21.15.12:0/2694806010 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f31ec009780 con 0x563684ae5a40 2024-03-23T23:11:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.700+0000 7f31f3400640 1 -- 172.21.15.12:0/2694806010 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec043070 con 0x563684ae5a40 2024-03-23T23:11:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.700+0000 7f320c5c4740 1 -- 172.21.15.12:0/2694806010 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684ae5a40 msgr2=0x563684af65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.700+0000 7f320c5c4740 1 --2- 172.21.15.12:0/2694806010 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684ae5a40 0x563684af65d0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f31ec02c190 tx=0x7f31ec005fd0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.701+0000 7f320c5c4740 1 -- 172.21.15.12:0/2694806010 shutdown_connections 2024-03-23T23:11:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.701+0000 7f320c5c4740 1 --2- 172.21.15.12:0/2694806010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563684aeabb0 0x563684adf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.701+0000 7f320c5c4740 1 --2- 172.21.15.12:0/2694806010 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684ae5a40 0x563684af65d0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.701+0000 7f320c5c4740 1 --2- 172.21.15.12:0/2694806010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563684ae5050 0x563684adb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.701+0000 7f320c5c4740 1 -- 172.21.15.12:0/2694806010 >> 172.21.15.12:0/2694806010 conn(0x563684aad200 msgr2=0x563684ae6d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.701+0000 7f320c5c4740 1 -- 172.21.15.12:0/2694806010 shutdown_connections 2024-03-23T23:11:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.701+0000 7f320c5c4740 1 -- 172.21.15.12:0/2694806010 wait complete. 2024-03-23T23:11:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.701+0000 7f320c5c4740 1 -- start start 2024-03-23T23:11:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.701+0000 7f320c5c4740 1 Processor -- start 2024-03-23T23:11:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.702+0000 7f320c5c4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563684ae5050 0x563684af63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.702+0000 7f320c5c4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563684ae5a40 0x563684af49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.702+0000 7f320c5c4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684aeabb0 0x563684af4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.702+0000 7f320c5c4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32041a9110 con 0x563684ae5a40 2024-03-23T23:11:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.702+0000 7f320c5c4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32041aa410 con 0x563684aeabb0 2024-03-23T23:11:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.702+0000 7f320c5c4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32041a9750 con 0x563684ae5050 2024-03-23T23:11:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.702+0000 7f320c5c4740 10 client.0 _ll_register_callbacks cb 0x7f31f40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:11:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.702+0000 7f3202000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563684ae5a40 0x563684af49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.702+0000 7f3202000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563684ae5a40 0x563684af49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50226/0 (socket says 172.21.15.12:50226) 2024-03-23T23:11:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.702+0000 7f3202000640 1 -- 172.21.15.12:0/568711074 learned_addr learned my addr 172.21.15.12:0/568711074 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:11:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.702+0000 7f3203400640 1 --2- 172.21.15.12:0/568711074 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563684ae5050 0x563684af63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.702+0000 7f3203400640 1 --2- 172.21.15.12:0/568711074 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684aeabb0 0x563684af4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.702+0000 7f3202000640 1 -- 172.21.15.12:0/568711074 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563684ae5050 msgr2=0x563684af63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.702+0000 7f3202000640 1 --2- 172.21.15.12:0/568711074 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563684ae5050 0x563684af63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.702+0000 7f3202000640 1 -- 172.21.15.12:0/568711074 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684aeabb0 msgr2=0x563684af4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:11:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.702+0000 7f3202000640 1 --2- 172.21.15.12:0/568711074 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684aeabb0 0x563684af4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.702+0000 7f3202000640 1 -- 172.21.15.12:0/568711074 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563684af5530 con 0x563684ae5a40 2024-03-23T23:11:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.702+0000 7f3203400640 1 --2- 172.21.15.12:0/568711074 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684aeabb0 0x563684af4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:11:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.703+0000 7f3202000640 1 --2- 172.21.15.12:0/568711074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563684ae5a40 0x563684af49b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f31ec02f020 tx=0x7f31ec02c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df7ca3dd6208d17 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:11:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.703+0000 7f31f0c00640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:11:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.703+0000 7f31f0c00640 1 -- 172.21.15.12:0/568711074 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec058070 con 0x563684ae5a40 2024-03-23T23:11:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.703+0000 7f31f0c00640 1 -- 172.21.15.12:0/568711074 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f31ec0046c0 con 0x563684ae5a40 2024-03-23T23:11:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.703+0000 7f320c5c4740 1 -- 172.21.15.12:0/568711074 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f31ec009780 con 0x563684ae5a40 2024-03-23T23:11:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.703+0000 7f31f0c00640 1 -- 172.21.15.12:0/568711074 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f31ec043070 con 0x563684ae5a40 2024-03-23T23:11:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.703+0000 7f320c5c4740 10 client.18084 fetch_fsmap learned FSMap version 23 2024-03-23T23:11:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.703+0000 7f320c5c4740 1 -- 172.21.15.12:0/568711074 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563684b57580 con 0x563684ae5a40 2024-03-23T23:11:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.704+0000 7f31f0c00640 1 -- 172.21.15.12:0/568711074 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f31ec009780 con 0x563684ae5a40 2024-03-23T23:11:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.704+0000 7f31f0c00640 1 -- 172.21.15.12:0/568711074 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f31ec05b020 con 0x563684ae5a40 2024-03-23T23:11:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.704+0000 7f320c5c4740 10 client.18084 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:11:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.704+0000 7f320c5c4740 10 client.18084 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:11:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.704+0000 7f320c5c4740 20 client.18084 populate_metadata read hostname 'smithi012' 2024-03-23T23:11:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.704+0000 7f320c5c4740 1 --2- 172.21.15.12:0/568711074 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563684b57e50 0x563684b5a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:11:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.704+0000 7f320c5c4740 4 client.18084 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:11:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.704+0000 7f320c5c4740 1 -- 172.21.15.12:0/568711074 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563684a51ae0 con 0x563684b57e50 2024-03-23T23:11:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.704+0000 7f3203400640 1 --2- 172.21.15.12:0/568711074 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563684b57e50 0x563684b5a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:11:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.705+0000 7f3203400640 1 --2- 172.21.15.12:0/568711074 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563684b57e50 0x563684b5a270 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45f196d6d8066a4 server_cookie=2826cd73f65a6c04 in_seq=0 out_seq=0 2024-03-23T23:11:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.705+0000 7f31f0c00640 10 client.18084 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.705+0000 7f31f0c00640 1 -- 172.21.15.12:0/568711074 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x563684a51ae0 con 0x563684b57e50 2024-03-23T23:11:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.705+0000 7f31f0c00640 10 client.18084 handle_command_reply: tid=0 2024-03-23T23:11:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.705+0000 7f320c5c4740 1 client.18084 shutdown 2024-03-23T23:11:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.705+0000 7f320c5c4740 1 -- 172.21.15.12:0/568711074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563684ae5a40 msgr2=0x563684af49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.705+0000 7f320c5c4740 1 --2- 172.21.15.12:0/568711074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563684ae5a40 0x563684af49b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f31ec02f020 tx=0x7f31ec02c5f0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.705+0000 7f320c5c4740 1 -- 172.21.15.12:0/568711074 shutdown_connections 2024-03-23T23:11:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.705+0000 7f320c5c4740 1 --2- 172.21.15.12:0/568711074 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563684b57e50 0x563684b5a270 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.705+0000 7f320c5c4740 1 --2- 172.21.15.12:0/568711074 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563684aeabb0 0x563684af4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.705+0000 7f320c5c4740 1 --2- 172.21.15.12:0/568711074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563684ae5a40 0x563684af49b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.705+0000 7f320c5c4740 1 --2- 172.21.15.12:0/568711074 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563684ae5050 0x563684af63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.705+0000 7f31f0c00640 0 client.18084 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:11:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.705+0000 7f320c5c4740 1 -- 172.21.15.12:0/568711074 >> 172.21.15.12:0/568711074 conn(0x563684aad200 msgr2=0x563684ae3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:59.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.705+0000 7f320c5c4740 1 -- 172.21.15.12:0/568711074 shutdown_connections 2024-03-23T23:11:59.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.705+0000 7f320c5c4740 1 -- 172.21.15.12:0/568711074 wait complete. 2024-03-23T23:11:59.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.706+0000 7f320c5c4740 20 client.18084 trim_cache size 0 max 16384 2024-03-23T23:11:59.708 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:11:59.708 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T23:11:59.708 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-03-23T23:11:59.708 INFO:teuthology.orchestra.run.smithi012.stdout: "3ceefe24-798f-48b2-b64e-17d4162abd30": { 2024-03-23T23:11:59.708 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-03-23T23:11:59.708 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "3ceefe24-798f-48b2-b64e-17d4162abd30", 2024-03-23T23:11:59.708 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-03-23T23:11:59.708 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:11:59.708 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:11:59.709 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:11:59.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.706+0000 7f3203e00640 1 -- 172.21.15.12:0/988833356 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f31e80a2800 msgr2=0x7f31e80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:59.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.706+0000 7f3203e00640 1 --2- 172.21.15.12:0/988833356 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f31e80a2800 0x7f31e80a4cc0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f31f40cceb0 tx=0x7f31f400cf90 comp rx=0 tx=0).stop 2024-03-23T23:11:59.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.706+0000 7f3203e00640 1 -- 172.21.15.12:0/988833356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32040ffe50 msgr2=0x7f3204199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:11:59.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.706+0000 7f3203e00640 1 --2- 172.21.15.12:0/988833356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32040ffe50 0x7f3204199ab0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f31f400ebc0 tx=0x7f31f400c6a0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.706+0000 7f3203e00640 1 -- 172.21.15.12:0/988833356 shutdown_connections 2024-03-23T23:11:59.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.706+0000 7f3203e00640 1 --2- 172.21.15.12:0/988833356 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f31e80a2800 0x7f31e80a4cc0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.706+0000 7f3203e00640 1 --2- 172.21.15.12:0/988833356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3204101150 0x7f32041a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.706+0000 7f3203e00640 1 --2- 172.21.15.12:0/988833356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32041007b0 0x7f3204199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.706+0000 7f3203e00640 1 --2- 172.21.15.12:0/988833356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32040ffe50 0x7f3204199ab0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:11:59.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.706+0000 7f3203e00640 1 -- 172.21.15.12:0/988833356 >> 172.21.15.12:0/988833356 conn(0x7f32040fba10 msgr2=0x7f32040fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:11:59.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.706+0000 7f3203e00640 1 -- 172.21.15.12:0/988833356 shutdown_connections 2024-03-23T23:11:59.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:11:59.707+0000 7f3203e00640 1 -- 172.21.15.12:0/988833356 wait complete. 2024-03-23T23:11:59.723 INFO:tasks.cephfs.filesystem:scrub status for tag:3ceefe24-798f-48b2-b64e-17d4162abd30 - {'path': '/', 'tag': '3ceefe24-798f-48b2-b64e-17d4162abd30', 'options': 'recursive,force'} 2024-03-23T23:12:00.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:00 smithi012 ceph-mon[31695]: scrub queued for path: / 2024-03-23T23:12:00.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:00 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:12:00.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:00 smithi012 ceph-mon[31695]: scrub summary: active paths [/] 2024-03-23T23:12:00.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:00 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:12:00.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:00 smithi173 ceph-mon[36116]: scrub queued for path: / 2024-03-23T23:12:00.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:00 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:12:00.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:00 smithi173 ceph-mon[36116]: scrub summary: active paths [/] 2024-03-23T23:12:00.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:00 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:12:00.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:00 smithi203 ceph-mon[36693]: scrub queued for path: / 2024-03-23T23:12:00.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:00 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:12:00.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:00 smithi203 ceph-mon[36693]: scrub summary: active paths [/] 2024-03-23T23:12:00.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:00 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:12:01.269 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:01 smithi173 ceph-mon[36116]: pgmap v696: 97 pgs: 97 active+clean; 2.8 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:12:01.269 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:01 smithi173 ceph-mon[36116]: scrub summary: idle 2024-03-23T23:12:01.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:01 smithi012 ceph-mon[31695]: pgmap v696: 97 pgs: 97 active+clean; 2.8 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:12:01.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:01 smithi012 ceph-mon[31695]: scrub summary: idle 2024-03-23T23:12:01.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:01 smithi203 ceph-mon[36693]: pgmap v696: 97 pgs: 97 active+clean; 2.8 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:12:01.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:01 smithi203 ceph-mon[36693]: scrub summary: idle 2024-03-23T23:12:03.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:03 smithi012 ceph-mon[31695]: pgmap v697: 97 pgs: 97 active+clean; 2.9 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:12:03.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:03 smithi173 ceph-mon[36116]: pgmap v697: 97 pgs: 97 active+clean; 2.9 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:12:03.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:03 smithi203 ceph-mon[36693]: pgmap v697: 97 pgs: 97 active+clean; 2.9 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:12:05.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:05 smithi012 ceph-mon[31695]: pgmap v698: 97 pgs: 97 active+clean; 2.9 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:12:05.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:05 smithi173 ceph-mon[36116]: pgmap v698: 97 pgs: 97 active+clean; 2.9 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:12:05.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:05 smithi203 ceph-mon[36693]: pgmap v698: 97 pgs: 97 active+clean; 2.9 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:12:05.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.517+0000 7f8c93000640 1 -- 172.21.15.12:0/2432727452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c8c1052d0 msgr2=0x7f8c8c10bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:05.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.517+0000 7f8c93000640 1 --2- 172.21.15.12:0/2432727452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c8c1052d0 0x7f8c8c10bb60 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f8c84009be0 tx=0x7f8c8402bef0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.518+0000 7f8c93000640 1 -- 172.21.15.12:0/2432727452 shutdown_connections 2024-03-23T23:12:05.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.518+0000 7f8c93000640 1 --2- 172.21.15.12:0/2432727452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c8c1052d0 0x7f8c8c10bb60 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.518+0000 7f8c93000640 1 --2- 172.21.15.12:0/2432727452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c8c104910 0x7f8c8c104d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.518+0000 7f8c93000640 1 --2- 172.21.15.12:0/2432727452 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c8c103720 0x7f8c8c103b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.518+0000 7f8c93000640 1 -- 172.21.15.12:0/2432727452 >> 172.21.15.12:0/2432727452 conn(0x7f8c8c0fecb0 msgr2=0x7f8c8c1010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:05.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.518+0000 7f8c93000640 1 -- 172.21.15.12:0/2432727452 shutdown_connections 2024-03-23T23:12:05.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.518+0000 7f8c93000640 1 -- 172.21.15.12:0/2432727452 wait complete. 2024-03-23T23:12:05.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.519+0000 7f8c93000640 1 Processor -- start 2024-03-23T23:12:05.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.519+0000 7f8c93000640 1 -- start start 2024-03-23T23:12:05.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.519+0000 7f8c93000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c8c103720 0x7f8c8c199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:05.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.519+0000 7f8c93000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c8c104910 0x7f8c8c199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:05.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.519+0000 7f8c93000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c8c1052d0 0x7f8c8c1a0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.519+0000 7f8c93000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c10f3d0 con 0x7f8c8c104910 2024-03-23T23:12:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.519+0000 7f8c93000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c10f250 con 0x7f8c8c1052d0 2024-03-23T23:12:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.519+0000 7f8c93000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c10f550 con 0x7f8c8c103720 2024-03-23T23:12:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.519+0000 7f8c91c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c8c103720 0x7f8c8c199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.519+0000 7f8c91c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c8c103720 0x7f8c8c199990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:50318/0 (socket says 172.21.15.12:50318) 2024-03-23T23:12:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.519+0000 7f8c91c00640 1 -- 172.21.15.12:0/1438710076 learned_addr learned my addr 172.21.15.12:0/1438710076 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.519+0000 7f8c91200640 1 --2- 172.21.15.12:0/1438710076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c8c104910 0x7f8c8c199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.519+0000 7f8c92600640 1 --2- 172.21.15.12:0/1438710076 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c8c1052d0 0x7f8c8c1a0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.520+0000 7f8c91c00640 1 -- 172.21.15.12:0/1438710076 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c8c1052d0 msgr2=0x7f8c8c1a0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.520+0000 7f8c91c00640 1 --2- 172.21.15.12:0/1438710076 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c8c1052d0 0x7f8c8c1a0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.520+0000 7f8c91c00640 1 -- 172.21.15.12:0/1438710076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c8c104910 msgr2=0x7f8c8c199ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.520+0000 7f8c91c00640 1 --2- 172.21.15.12:0/1438710076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c8c104910 0x7f8c8c199ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.520+0000 7f8c91c00640 1 -- 172.21.15.12:0/1438710076 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8c1a15b0 con 0x7f8c8c103720 2024-03-23T23:12:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.520+0000 7f8c91c00640 1 --2- 172.21.15.12:0/1438710076 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c8c103720 0x7f8c8c199990 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8c8000e9c0 tx=0x7f8c8000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.520+0000 7f8c8a000640 1 -- 172.21.15.12:0/1438710076 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8000cd80 con 0x7f8c8c103720 2024-03-23T23:12:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.520+0000 7f8c8a000640 1 -- 172.21.15.12:0/1438710076 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8c800035f0 con 0x7f8c8c103720 2024-03-23T23:12:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.520+0000 7f8c93000640 1 -- 172.21.15.12:0/1438710076 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c8c1a18a0 con 0x7f8c8c103720 2024-03-23T23:12:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.520+0000 7f8c8a000640 1 -- 172.21.15.12:0/1438710076 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c80010690 con 0x7f8c8c103720 2024-03-23T23:12:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.521+0000 7f8c93000640 1 -- 172.21.15.12:0/1438710076 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c8c1a1df0 con 0x7f8c8c103720 2024-03-23T23:12:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.524+0000 7f8c8a000640 1 -- 172.21.15.12:0/1438710076 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8c800037a0 con 0x7f8c8c103720 2024-03-23T23:12:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.524+0000 7f8c97216740 1 Processor -- start 2024-03-23T23:12:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.524+0000 7f8c97216740 1 -- start start 2024-03-23T23:12:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.524+0000 7f8c97216740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56319b5b43f0 0x56319b5b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.524+0000 7f8c97216740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56319b5b4de0 0x56319b5b98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.524+0000 7f8c97216740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56319b5b9f50 0x56319b5bc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.524+0000 7f8c97216740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56319b5773b0 con 0x56319b5b9f50 2024-03-23T23:12:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.524+0000 7f8c97216740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56319b519830 con 0x56319b5b43f0 2024-03-23T23:12:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.524+0000 7f8c97216740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x56319b4c25d0 con 0x56319b5b4de0 2024-03-23T23:12:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.525+0000 7f8c92600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56319b5b43f0 0x56319b5b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.525+0000 7f8c91c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56319b5b4de0 0x56319b5b98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.525+0000 7f8c92600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56319b5b43f0 0x56319b5b4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:45996/0 (socket says 172.21.15.12:45996) 2024-03-23T23:12:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.525+0000 7f8c92600640 1 -- 172.21.15.12:0/2322219052 learned_addr learned my addr 172.21.15.12:0/2322219052 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.525+0000 7f8c91200640 1 --2- 172.21.15.12:0/2322219052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56319b5b9f50 0x56319b5bc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.525+0000 7f8c92600640 1 -- 172.21.15.12:0/2322219052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56319b5b4de0 msgr2=0x56319b5b98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.525+0000 7f8c92600640 1 --2- 172.21.15.12:0/2322219052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56319b5b4de0 0x56319b5b98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.525+0000 7f8c92600640 1 -- 172.21.15.12:0/2322219052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56319b5b9f50 msgr2=0x56319b5bc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.525+0000 7f8c92600640 1 --2- 172.21.15.12:0/2322219052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56319b5b9f50 0x56319b5bc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.525+0000 7f8c92600640 1 -- 172.21.15.12:0/2322219052 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56319b4da2c0 con 0x56319b5b43f0 2024-03-23T23:12:05.525 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.525+0000 7f8c91200640 1 --2- 172.21.15.12:0/2322219052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56319b5b9f50 0x56319b5bc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:12:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.525+0000 7f8c92600640 1 --2- 172.21.15.12:0/2322219052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56319b5b43f0 0x56319b5b4810 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f8c8402c500 tx=0x7f8c84009330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=501c974449ad2013 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.525+0000 7f8c8a000640 1 --2- 172.21.15.12:0/1438710076 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8c740a27f0 0x7f8c740a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.526+0000 7f8c7ea00640 1 -- 172.21.15.12:0/2322219052 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c84043030 con 0x56319b5b43f0 2024-03-23T23:12:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.526+0000 7f8c91c00640 1 --2- 172.21.15.12:0/1438710076 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8c740a27f0 0x7f8c740a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.526+0000 7f8c7ea00640 1 -- 172.21.15.12:0/2322219052 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8c84033580 con 0x56319b5b43f0 2024-03-23T23:12:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.526+0000 7f8c7ea00640 1 -- 172.21.15.12:0/2322219052 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8402c890 con 0x56319b5b43f0 2024-03-23T23:12:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.526+0000 7f8c8a000640 1 -- 172.21.15.12:0/1438710076 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8c800cee60 con 0x7f8c8c103720 2024-03-23T23:12:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.526+0000 7f8c97216740 1 -- 172.21.15.12:0/2322219052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56319b5b43f0 msgr2=0x56319b5b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.526+0000 7f8c97216740 1 --2- 172.21.15.12:0/2322219052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56319b5b43f0 0x56319b5b4810 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f8c8402c500 tx=0x7f8c84009330 comp rx=0 tx=0).stop 2024-03-23T23:12:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.526+0000 7f8c91c00640 1 --2- 172.21.15.12:0/1438710076 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8c740a27f0 0x7f8c740a4cb0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f8c800a3a20 tx=0x7f8c800d28a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:05.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.526+0000 7f8c97216740 1 -- 172.21.15.12:0/2322219052 shutdown_connections 2024-03-23T23:12:05.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.526+0000 7f8c97216740 1 --2- 172.21.15.12:0/2322219052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56319b5b9f50 0x56319b5bc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.526+0000 7f8c97216740 1 --2- 172.21.15.12:0/2322219052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56319b5b4de0 0x56319b5b98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.526+0000 7f8c97216740 1 --2- 172.21.15.12:0/2322219052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56319b5b43f0 0x56319b5b4810 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.526+0000 7f8c97216740 1 -- 172.21.15.12:0/2322219052 >> 172.21.15.12:0/2322219052 conn(0x56319b5af960 msgr2=0x56319b5afd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:05.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.527+0000 7f8c97216740 1 -- 172.21.15.12:0/2322219052 shutdown_connections 2024-03-23T23:12:05.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.527+0000 7f8c97216740 1 -- 172.21.15.12:0/2322219052 wait complete. 2024-03-23T23:12:05.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.527+0000 7f8c97216740 1 -- start start 2024-03-23T23:12:05.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.527+0000 7f8c97216740 1 Processor -- start 2024-03-23T23:12:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.528+0000 7f8c97216740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56319b5b43f0 0x56319b5be580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.528+0000 7f8c97216740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56319b5b4de0 0x56319b5beac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.528+0000 7f8c97216740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56319b5b9f50 0x56319b5c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.528+0000 7f8c97216740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56319b4d5a20 con 0x56319b5b43f0 2024-03-23T23:12:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.528+0000 7f8c97216740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56319b5158d0 con 0x56319b5b4de0 2024-03-23T23:12:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.528+0000 7f8c97216740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x56319b5bcea0 con 0x56319b5b9f50 2024-03-23T23:12:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.528+0000 7f8c97216740 10 client.0 _ll_register_callbacks cb 0x56319b5121c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:12:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.528+0000 7f8c92600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56319b5b43f0 0x56319b5be580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.528+0000 7f8c92600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56319b5b43f0 0x56319b5be580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59314/0 (socket says 172.21.15.12:59314) 2024-03-23T23:12:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.528+0000 7f8c92600640 1 -- 172.21.15.12:0/3135911722 learned_addr learned my addr 172.21.15.12:0/3135911722 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.528+0000 7f8c91200640 1 --2- 172.21.15.12:0/3135911722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56319b5b4de0 0x56319b5beac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.528+0000 7f8c92600640 1 --2- 172.21.15.12:0/3135911722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56319b5b9f50 0x56319b5c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.528+0000 7f8c91200640 1 -- 172.21.15.12:0/3135911722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56319b5b9f50 msgr2=0x56319b5c5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.528+0000 7f8c91200640 1 --2- 172.21.15.12:0/3135911722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56319b5b9f50 0x56319b5c5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.528+0000 7f8c91200640 1 -- 172.21.15.12:0/3135911722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56319b5b43f0 msgr2=0x56319b5be580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.528+0000 7f8c91200640 1 --2- 172.21.15.12:0/3135911722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56319b5b43f0 0x56319b5be580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.528+0000 7f8c91200640 1 -- 172.21.15.12:0/3135911722 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56319b22e210 con 0x56319b5b4de0 2024-03-23T23:12:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.529+0000 7f8c91200640 1 --2- 172.21.15.12:0/3135911722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56319b5b4de0 0x56319b5beac0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c19aea0 tx=0x7f8c78012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d92d9ea09f5faccd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.529+0000 7f8c73400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:12:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.529+0000 7f8c73400640 1 -- 172.21.15.12:0/3135911722 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7800f970 con 0x56319b5b4de0 2024-03-23T23:12:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.529+0000 7f8c73400640 1 -- 172.21.15.12:0/3135911722 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8c7800fb10 con 0x56319b5b4de0 2024-03-23T23:12:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.529+0000 7f8c97216740 1 -- 172.21.15.12:0/3135911722 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8c84033580 con 0x56319b5b4de0 2024-03-23T23:12:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.529+0000 7f8c73400640 1 -- 172.21.15.12:0/3135911722 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c7800fd50 con 0x56319b5b4de0 2024-03-23T23:12:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.529+0000 7f8c97216740 10 client.28238 fetch_fsmap learned FSMap version 23 2024-03-23T23:12:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.529+0000 7f8c97216740 1 -- 172.21.15.12:0/3135911722 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56319b2fa620 con 0x56319b5b4de0 2024-03-23T23:12:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.530+0000 7f8c73400640 1 -- 172.21.15.12:0/3135911722 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8c84033580 con 0x56319b5b4de0 2024-03-23T23:12:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.530+0000 7f8c73400640 1 -- 172.21.15.12:0/3135911722 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8c7801c440 con 0x56319b5b4de0 2024-03-23T23:12:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.530+0000 7f8c97216740 10 client.28238 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:12:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.530+0000 7f8c97216740 10 client.28238 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:12:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.530+0000 7f8c97216740 20 client.28238 populate_metadata read hostname 'smithi012' 2024-03-23T23:12:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.530+0000 7f8c97216740 1 --2- 172.21.15.12:0/3135911722 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56319b5ca330 0x56319b5cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.530+0000 7f8c97216740 4 client.28238 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:12:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.530+0000 7f8c97216740 1 -- 172.21.15.12:0/3135911722 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56319b526d70 con 0x56319b5ca330 2024-03-23T23:12:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.530+0000 7f8c92600640 1 --2- 172.21.15.12:0/3135911722 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56319b5ca330 0x56319b5cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.531+0000 7f8c92600640 1 --2- 172.21.15.12:0/3135911722 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56319b5ca330 0x56319b5cc750 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2813c42f331b6efe server_cookie=f7e87bef20a33229 in_seq=0 out_seq=0 2024-03-23T23:12:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.531+0000 7f8c73400640 10 client.28238 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.532+0000 7f8c73400640 1 -- 172.21.15.12:0/3135911722 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56319b526d70 con 0x56319b5ca330 2024-03-23T23:12:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.532+0000 7f8c73400640 10 client.28238 handle_command_reply: tid=0 2024-03-23T23:12:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.532+0000 7f8c97216740 1 client.28238 shutdown 2024-03-23T23:12:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.532+0000 7f8c97216740 1 -- 172.21.15.12:0/3135911722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56319b5b4de0 msgr2=0x56319b5beac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.532+0000 7f8c97216740 1 --2- 172.21.15.12:0/3135911722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56319b5b4de0 0x56319b5beac0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c19aea0 tx=0x7f8c78012b60 comp rx=0 tx=0).stop 2024-03-23T23:12:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.532+0000 7f8c97216740 1 -- 172.21.15.12:0/3135911722 shutdown_connections 2024-03-23T23:12:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.532+0000 7f8c97216740 1 --2- 172.21.15.12:0/3135911722 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56319b5ca330 0x56319b5cc750 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.532+0000 7f8c97216740 1 --2- 172.21.15.12:0/3135911722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56319b5b9f50 0x56319b5c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.532+0000 7f8c97216740 1 --2- 172.21.15.12:0/3135911722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56319b5b4de0 0x56319b5beac0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.532+0000 7f8c97216740 1 --2- 172.21.15.12:0/3135911722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56319b5b43f0 0x56319b5be580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.532+0000 7f8c73400640 0 client.28238 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.532+0000 7f8c97216740 1 -- 172.21.15.12:0/3135911722 >> 172.21.15.12:0/3135911722 conn(0x56319b5af960 msgr2=0x56319b5aeb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.532+0000 7f8c97216740 1 -- 172.21.15.12:0/3135911722 shutdown_connections 2024-03-23T23:12:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.532+0000 7f8c97216740 1 -- 172.21.15.12:0/3135911722 wait complete. 2024-03-23T23:12:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.532+0000 7f8c97216740 20 client.28238 trim_cache size 0 max 16384 2024-03-23T23:12:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.548+0000 7f8c97216740 1 Processor -- start 2024-03-23T23:12:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.548+0000 7f8c97216740 1 -- start start 2024-03-23T23:12:05.548 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.549+0000 7f8c97216740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56319b5b43f0 0x56319b5b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:05.548 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.549+0000 7f8c97216740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56319b5b4de0 0x56319b5b6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:05.548 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.549+0000 7f8c97216740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56319b5b9f50 0x56319b5c5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:05.548 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.549+0000 7f8c97216740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c1ace80 con 0x56319b5b43f0 2024-03-23T23:12:05.548 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.549+0000 7f8c97216740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c1aa980 con 0x56319b5b9f50 2024-03-23T23:12:05.548 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.549+0000 7f8c97216740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c1a8a20 con 0x56319b5b4de0 2024-03-23T23:12:05.548 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.549+0000 7f8c92600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56319b5b43f0 0x56319b5b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:05.548 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.549+0000 7f8c91200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56319b5b4de0 0x56319b5b6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:05.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.549+0000 7f8c91200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56319b5b4de0 0x56319b5b6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:50342/0 (socket says 172.21.15.12:50342) 2024-03-23T23:12:05.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.549+0000 7f8c91200640 1 -- 172.21.15.12:0/2510425016 learned_addr learned my addr 172.21.15.12:0/2510425016 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:05.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.549+0000 7f8c92600640 1 --2- 172.21.15.12:0/2510425016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56319b5b9f50 0x56319b5c5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:05.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.549+0000 7f8c91200640 1 -- 172.21.15.12:0/2510425016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56319b5b9f50 msgr2=0x56319b5c5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:05.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.549+0000 7f8c91200640 1 --2- 172.21.15.12:0/2510425016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56319b5b9f50 0x56319b5c5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.549+0000 7f8c91200640 1 -- 172.21.15.12:0/2510425016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56319b5b43f0 msgr2=0x56319b5b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:05.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.549+0000 7f8c91200640 1 --2- 172.21.15.12:0/2510425016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56319b5b43f0 0x56319b5b6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.549+0000 7f8c91200640 1 -- 172.21.15.12:0/2510425016 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56319b5bcfe0 con 0x56319b5b4de0 2024-03-23T23:12:05.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.550+0000 7f8c91200640 1 --2- 172.21.15.12:0/2510425016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56319b5b4de0 0x56319b5b6680 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8c78001eb0 tx=0x7f8c7801cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=542ee9ba572b5c5c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:05.551 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.550+0000 7f8c7e000640 1 -- 172.21.15.12:0/2510425016 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7801c420 con 0x56319b5b4de0 2024-03-23T23:12:05.551 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.550+0000 7f8c7e000640 1 -- 172.21.15.12:0/2510425016 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8c78022030 con 0x56319b5b4de0 2024-03-23T23:12:05.551 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.550+0000 7f8c7e000640 1 -- 172.21.15.12:0/2510425016 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7800f970 con 0x56319b5b4de0 2024-03-23T23:12:05.551 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.550+0000 7f8c97216740 1 -- 172.21.15.12:0/2510425016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56319b5b4de0 msgr2=0x56319b5b6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:05.551 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.550+0000 7f8c97216740 1 --2- 172.21.15.12:0/2510425016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56319b5b4de0 0x56319b5b6680 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8c78001eb0 tx=0x7f8c7801cc80 comp rx=0 tx=0).stop 2024-03-23T23:12:05.551 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.551+0000 7f8c97216740 1 -- 172.21.15.12:0/2510425016 shutdown_connections 2024-03-23T23:12:05.551 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.551+0000 7f8c97216740 1 --2- 172.21.15.12:0/2510425016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56319b5b9f50 0x56319b5c5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.551 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.551+0000 7f8c97216740 1 --2- 172.21.15.12:0/2510425016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56319b5b4de0 0x56319b5b6680 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.551 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.551+0000 7f8c97216740 1 --2- 172.21.15.12:0/2510425016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56319b5b43f0 0x56319b5b6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.551 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.551+0000 7f8c97216740 1 -- 172.21.15.12:0/2510425016 >> 172.21.15.12:0/2510425016 conn(0x56319b5af960 msgr2=0x56319b5afd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:05.551 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.551+0000 7f8c97216740 1 -- 172.21.15.12:0/2510425016 shutdown_connections 2024-03-23T23:12:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.551+0000 7f8c97216740 1 -- 172.21.15.12:0/2510425016 wait complete. 2024-03-23T23:12:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.551+0000 7f8c97216740 1 -- start start 2024-03-23T23:12:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.551+0000 7f8c97216740 1 Processor -- start 2024-03-23T23:12:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.552+0000 7f8c97216740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56319b5b43f0 0x56319b5c3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.552+0000 7f8c97216740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56319b5b4de0 0x56319b5c3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.552+0000 7f8c97216740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56319b5b9f50 0x56319b5c4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.552+0000 7f8c97216740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c1a9020 con 0x56319b5b4de0 2024-03-23T23:12:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.552+0000 7f8c97216740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c1aa320 con 0x56319b5b9f50 2024-03-23T23:12:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.552+0000 7f8c97216740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c1a9660 con 0x56319b5b43f0 2024-03-23T23:12:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.552+0000 7f8c97216740 10 client.0 _ll_register_callbacks cb 0x7f8c78009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:12:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.552+0000 7f8c92600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56319b5b43f0 0x56319b5c3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.552+0000 7f8c91200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56319b5b4de0 0x56319b5c3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.552+0000 7f8c92600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56319b5b43f0 0x56319b5c3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:50346/0 (socket says 172.21.15.12:50346) 2024-03-23T23:12:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.552+0000 7f8c92600640 1 -- 172.21.15.12:0/3280323174 learned_addr learned my addr 172.21.15.12:0/3280323174 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.552+0000 7f8c92600640 1 --2- 172.21.15.12:0/3280323174 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56319b5b9f50 0x56319b5c4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.552+0000 7f8c92600640 1 -- 172.21.15.12:0/3280323174 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56319b5b9f50 msgr2=0x56319b5c4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:05.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.552+0000 7f8c92600640 1 --2- 172.21.15.12:0/3280323174 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56319b5b9f50 0x56319b5c4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.552+0000 7f8c92600640 1 -- 172.21.15.12:0/3280323174 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56319b5b4de0 msgr2=0x56319b5c3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:05.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.552+0000 7f8c92600640 1 --2- 172.21.15.12:0/3280323174 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56319b5b4de0 0x56319b5c3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.552+0000 7f8c92600640 1 -- 172.21.15.12:0/3280323174 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56319b5c5130 con 0x56319b5b43f0 2024-03-23T23:12:05.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.553+0000 7f8c92600640 1 --2- 172.21.15.12:0/3280323174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56319b5b43f0 0x56319b5c3640 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f8c8402c500 tx=0x7f8c84009330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c4c52f73c8500b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:05.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.553+0000 7f8c73400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:12:05.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.553+0000 7f8c73400640 1 -- 172.21.15.12:0/3280323174 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8405c070 con 0x56319b5b43f0 2024-03-23T23:12:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.553+0000 7f8c73400640 1 -- 172.21.15.12:0/3280323174 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8c84033d40 con 0x56319b5b43f0 2024-03-23T23:12:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.553+0000 7f8c97216740 1 -- 172.21.15.12:0/3280323174 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8c78022030 con 0x56319b5b43f0 2024-03-23T23:12:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.553+0000 7f8c73400640 1 -- 172.21.15.12:0/3280323174 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c84043040 con 0x56319b5b43f0 2024-03-23T23:12:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.553+0000 7f8c97216740 10 client.28210 fetch_fsmap learned FSMap version 23 2024-03-23T23:12:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.553+0000 7f8c97216740 1 -- 172.21.15.12:0/3280323174 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56319b625f10 con 0x56319b5b43f0 2024-03-23T23:12:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.554+0000 7f8c73400640 1 -- 172.21.15.12:0/3280323174 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8c78022030 con 0x56319b5b43f0 2024-03-23T23:12:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.554+0000 7f8c73400640 1 -- 172.21.15.12:0/3280323174 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8c8405f030 con 0x56319b5b43f0 2024-03-23T23:12:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.554+0000 7f8c97216740 10 client.28210 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:12:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.554+0000 7f8c97216740 10 client.28210 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:12:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.554+0000 7f8c97216740 20 client.28210 populate_metadata read hostname 'smithi012' 2024-03-23T23:12:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.554+0000 7f8c97216740 1 --2- 172.21.15.12:0/3280323174 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56319b626460 0x56319b628890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.554+0000 7f8c97216740 4 client.28210 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:12:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.554+0000 7f8c97216740 1 -- 172.21.15.12:0/3280323174 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56319b5210c0 con 0x56319b626460 2024-03-23T23:12:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.554+0000 7f8c91200640 1 --2- 172.21.15.12:0/3280323174 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56319b626460 0x56319b628890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.555+0000 7f8c91200640 1 --2- 172.21.15.12:0/3280323174 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56319b626460 0x56319b628890 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=228ae57a8f94aba4 server_cookie=8ecec8c798867e32 in_seq=0 out_seq=0 2024-03-23T23:12:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.555+0000 7f8c73400640 10 client.28210 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.566+0000 7f8c73400640 1 -- 172.21.15.12:0/3280323174 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56319b5210c0 con 0x56319b626460 2024-03-23T23:12:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.566+0000 7f8c73400640 10 client.28210 handle_command_reply: tid=0 2024-03-23T23:12:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.566+0000 7f8c97216740 1 client.28210 shutdown 2024-03-23T23:12:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.567+0000 7f8c97216740 1 -- 172.21.15.12:0/3280323174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56319b5b43f0 msgr2=0x56319b5c3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.567+0000 7f8c97216740 1 --2- 172.21.15.12:0/3280323174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56319b5b43f0 0x56319b5c3640 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f8c8402c500 tx=0x7f8c84009330 comp rx=0 tx=0).stop 2024-03-23T23:12:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.567+0000 7f8c97216740 1 -- 172.21.15.12:0/3280323174 shutdown_connections 2024-03-23T23:12:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.567+0000 7f8c97216740 1 --2- 172.21.15.12:0/3280323174 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56319b626460 0x56319b628890 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.567+0000 7f8c97216740 1 --2- 172.21.15.12:0/3280323174 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56319b5b9f50 0x56319b5c4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.567+0000 7f8c97216740 1 --2- 172.21.15.12:0/3280323174 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56319b5b4de0 0x56319b5c3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.567+0000 7f8c73400640 0 client.28210 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.567+0000 7f8c97216740 1 --2- 172.21.15.12:0/3280323174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56319b5b43f0 0x56319b5c3640 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.567+0000 7f8c97216740 1 -- 172.21.15.12:0/3280323174 >> 172.21.15.12:0/3280323174 conn(0x56319b5af960 msgr2=0x56319b5be560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.567+0000 7f8c97216740 1 -- 172.21.15.12:0/3280323174 shutdown_connections 2024-03-23T23:12:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.567+0000 7f8c97216740 1 -- 172.21.15.12:0/3280323174 wait complete. 2024-03-23T23:12:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.567+0000 7f8c97216740 20 client.28210 trim_cache size 0 max 16384 2024-03-23T23:12:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:12:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:12:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:12:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:12:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.568+0000 7f8c93000640 1 -- 172.21.15.12:0/1438710076 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8c740a27f0 msgr2=0x7f8c740a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.568+0000 7f8c93000640 1 --2- 172.21.15.12:0/1438710076 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8c740a27f0 0x7f8c740a4cb0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f8c800a3a20 tx=0x7f8c800d28a0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.568+0000 7f8c93000640 1 -- 172.21.15.12:0/1438710076 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c8c103720 msgr2=0x7f8c8c199990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.568+0000 7f8c93000640 1 --2- 172.21.15.12:0/1438710076 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c8c103720 0x7f8c8c199990 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8c8000e9c0 tx=0x7f8c8000ee80 comp rx=0 tx=0).stop 2024-03-23T23:12:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.568+0000 7f8c93000640 1 -- 172.21.15.12:0/1438710076 shutdown_connections 2024-03-23T23:12:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.568+0000 7f8c93000640 1 --2- 172.21.15.12:0/1438710076 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8c740a27f0 0x7f8c740a4cb0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.568+0000 7f8c93000640 1 --2- 172.21.15.12:0/1438710076 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c8c1052d0 0x7f8c8c1a0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.568+0000 7f8c93000640 1 --2- 172.21.15.12:0/1438710076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c8c104910 0x7f8c8c199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.568+0000 7f8c93000640 1 --2- 172.21.15.12:0/1438710076 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c8c103720 0x7f8c8c199990 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.568+0000 7f8c93000640 1 -- 172.21.15.12:0/1438710076 >> 172.21.15.12:0/1438710076 conn(0x7f8c8c0fecb0 msgr2=0x7f8c8c107bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:05.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.568+0000 7f8c93000640 1 -- 172.21.15.12:0/1438710076 shutdown_connections 2024-03-23T23:12:05.569 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:05.568+0000 7f8c93000640 1 -- 172.21.15.12:0/1438710076 wait complete. 2024-03-23T23:12:07.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:07 smithi012 ceph-mon[31695]: pgmap v699: 97 pgs: 97 active+clean; 2.9 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-03-23T23:12:07.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:07 smithi173 ceph-mon[36116]: pgmap v699: 97 pgs: 97 active+clean; 2.9 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-03-23T23:12:07.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:07 smithi203 ceph-mon[36693]: pgmap v699: 97 pgs: 97 active+clean; 2.9 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-03-23T23:12:09.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:09 smithi012 ceph-mon[31695]: pgmap v700: 97 pgs: 97 active+clean; 2.9 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-23T23:12:09.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:09 smithi173 ceph-mon[36116]: pgmap v700: 97 pgs: 97 active+clean; 2.9 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-23T23:12:09.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:09 smithi203 ceph-mon[36693]: pgmap v700: 97 pgs: 97 active+clean; 2.9 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-23T23:12:11.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:11 smithi012 ceph-mon[31695]: pgmap v701: 97 pgs: 97 active+clean; 2.9 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-23T23:12:11.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:11 smithi173 ceph-mon[36116]: pgmap v701: 97 pgs: 97 active+clean; 2.9 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-23T23:12:11.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:11 smithi203 ceph-mon[36693]: pgmap v701: 97 pgs: 97 active+clean; 2.9 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-23T23:12:13.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:13 smithi012 ceph-mon[31695]: pgmap v702: 97 pgs: 97 active+clean; 3.0 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:12:13.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:13 smithi173 ceph-mon[36116]: pgmap v702: 97 pgs: 97 active+clean; 3.0 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:12:13.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:13 smithi203 ceph-mon[36693]: pgmap v702: 97 pgs: 97 active+clean; 3.0 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:12:14.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:12:14.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:14.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:14.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:14 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:14.350 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:14 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:12:14.350 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:14 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:14.350 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:14 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:14.350 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:14 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:14.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:14 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:12:14.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:14 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:14.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:14 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:14.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:14 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:15.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:15 smithi012 ceph-mon[31695]: pgmap v703: 97 pgs: 97 active+clean; 3.0 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:12:15.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:15 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:15.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:15 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:15.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:15 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:15.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:15 smithi173 ceph-mon[36116]: pgmap v703: 97 pgs: 97 active+clean; 3.0 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:12:15.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:15 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:15.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:15 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:15.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:15 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:15.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:15 smithi203 ceph-mon[36693]: pgmap v703: 97 pgs: 97 active+clean; 3.0 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:12:15.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:15 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:15.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:15 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:15.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:15 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:16.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:16 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:16.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:16 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:16.804 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:16 smithi012 ceph-mon[31695]: pgmap v704: 97 pgs: 97 active+clean; 3.1 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:12:16.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:16 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:16.867 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:16 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:16.868 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:16 smithi173 ceph-mon[36116]: pgmap v704: 97 pgs: 97 active+clean; 3.1 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:12:16.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:16 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:16.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:16 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:16.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:16 smithi203 ceph-mon[36693]: pgmap v704: 97 pgs: 97 active+clean; 3.1 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:12:18.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:18 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:18.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:18 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:18.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:18 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:18.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:18 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:18.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:18 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:18.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:18 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:18.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.713+0000 7f173ac00640 1 -- 172.21.15.12:0/1634887571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1734104ae0 msgr2=0x7f1734104f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:18.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.713+0000 7f173ac00640 1 --2- 172.21.15.12:0/1634887571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1734104ae0 0x7f1734104f60 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f171c009860 tx=0x7f171c02bb70 comp rx=0 tx=0).stop 2024-03-23T23:12:18.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.714+0000 7f173ac00640 1 -- 172.21.15.12:0/1634887571 shutdown_connections 2024-03-23T23:12:18.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.714+0000 7f173ac00640 1 --2- 172.21.15.12:0/1634887571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17341054a0 0x7f173410bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.714+0000 7f173ac00640 1 --2- 172.21.15.12:0/1634887571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1734104ae0 0x7f1734104f60 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.714+0000 7f173ac00640 1 --2- 172.21.15.12:0/1634887571 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17341038f0 0x7f1734103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.714+0000 7f173ac00640 1 -- 172.21.15.12:0/1634887571 >> 172.21.15.12:0/1634887571 conn(0x7f17340fee60 msgr2=0x7f17341012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:18.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.714+0000 7f173ac00640 1 -- 172.21.15.12:0/1634887571 shutdown_connections 2024-03-23T23:12:18.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.714+0000 7f173ac00640 1 -- 172.21.15.12:0/1634887571 wait complete. 2024-03-23T23:12:18.715 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.715+0000 7f173ac00640 1 Processor -- start 2024-03-23T23:12:18.716 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.715+0000 7f173ac00640 1 -- start start 2024-03-23T23:12:18.716 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.716+0000 7f173ac00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17341038f0 0x7f1734199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:18.716 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.716+0000 7f173ac00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1734104ae0 0x7f173419a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:18.716 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.716+0000 7f173ac00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17341054a0 0x7f17341a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:18.716 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.716+0000 7f173ac00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173410f550 con 0x7f1734104ae0 2024-03-23T23:12:18.716 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.716+0000 7f173ac00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173410f3d0 con 0x7f17341054a0 2024-03-23T23:12:18.716 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.716+0000 7f173ac00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173410f6d0 con 0x7f17341038f0 2024-03-23T23:12:18.716 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.716+0000 7f1733e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17341038f0 0x7f1734199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:18.716 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.716+0000 7f1733e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17341038f0 0x7f1734199b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:57780/0 (socket says 172.21.15.12:57780) 2024-03-23T23:12:18.716 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.716+0000 7f1733e00640 1 -- 172.21.15.12:0/3374328816 learned_addr learned my addr 172.21.15.12:0/3374328816 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:18.716 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.716+0000 7f1738c00640 1 --2- 172.21.15.12:0/3374328816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17341054a0 0x7f17341a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:18.716 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.716+0000 7f172be00640 1 --2- 172.21.15.12:0/3374328816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1734104ae0 0x7f173419a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:18.717 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.716+0000 7f1733e00640 1 -- 172.21.15.12:0/3374328816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17341054a0 msgr2=0x7f17341a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:18.717 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.716+0000 7f1733e00640 1 --2- 172.21.15.12:0/3374328816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17341054a0 0x7f17341a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.717 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.716+0000 7f1733e00640 1 -- 172.21.15.12:0/3374328816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1734104ae0 msgr2=0x7f173419a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:18.718 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.716+0000 7f1733e00640 1 --2- 172.21.15.12:0/3374328816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1734104ae0 0x7f173419a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.718 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.716+0000 7f1733e00640 1 -- 172.21.15.12:0/3374328816 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17341a17b0 con 0x7f17341038f0 2024-03-23T23:12:18.718 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.717+0000 7f1733e00640 1 --2- 172.21.15.12:0/3374328816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17341038f0 0x7f1734199b90 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f172400e9c0 tx=0x7f172400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:18.718 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.717+0000 7f1731600640 1 -- 172.21.15.12:0/3374328816 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172400cd80 con 0x7f17341038f0 2024-03-23T23:12:18.718 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.717+0000 7f1731600640 1 -- 172.21.15.12:0/3374328816 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17240035f0 con 0x7f17341038f0 2024-03-23T23:12:18.718 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.717+0000 7f1731600640 1 -- 172.21.15.12:0/3374328816 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1724010690 con 0x7f17341038f0 2024-03-23T23:12:18.718 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.717+0000 7f173ac00640 1 -- 172.21.15.12:0/3374328816 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17341a1aa0 con 0x7f17341038f0 2024-03-23T23:12:18.718 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.717+0000 7f173ac00640 1 -- 172.21.15.12:0/3374328816 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17341a1f70 con 0x7f17341038f0 2024-03-23T23:12:18.720 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.720+0000 7f1731600640 1 -- 172.21.15.12:0/3374328816 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f17240037a0 con 0x7f17341038f0 2024-03-23T23:12:18.721 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.722+0000 7f173d80a740 1 Processor -- start 2024-03-23T23:12:18.723 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.722+0000 7f1731600640 1 --2- 172.21.15.12:0/3374328816 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f17180a28a0 0x7f17180a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:18.723 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.722+0000 7f173d80a740 1 -- start start 2024-03-23T23:12:18.723 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.722+0000 7f1731600640 1 -- 172.21.15.12:0/3374328816 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f17240cee60 con 0x7f17341038f0 2024-03-23T23:12:18.723 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.722+0000 7f1738c00640 1 --2- 172.21.15.12:0/3374328816 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f17180a28a0 0x7f17180a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:18.723 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.722+0000 7f173d80a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5574e9b699f0 0x5574e9b69e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:18.723 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.722+0000 7f173d80a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b6a3e0 0x5574e9b6eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:18.723 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.722+0000 7f173d80a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574e9b6f550 0x5574e9b719e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:18.723 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.722+0000 7f173d80a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574e9b243b0 con 0x5574e9b699f0 2024-03-23T23:12:18.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.722+0000 7f173d80a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574e9ac6830 con 0x5574e9b6a3e0 2024-03-23T23:12:18.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.722+0000 7f173d80a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574e9a6f5d0 con 0x5574e9b6f550 2024-03-23T23:12:18.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.723+0000 7f172be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b6a3e0 0x5574e9b6eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:18.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.723+0000 7f1733e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5574e9b699f0 0x5574e9b69e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:18.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.723+0000 7f172be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b6a3e0 0x5574e9b6eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60142/0 (socket says 172.21.15.12:60142) 2024-03-23T23:12:18.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.723+0000 7f172be00640 1 -- 172.21.15.12:0/2597819766 learned_addr learned my addr 172.21.15.12:0/2597819766 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:18.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.723+0000 7f1738c00640 1 --2- 172.21.15.12:0/2597819766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574e9b6f550 0x5574e9b719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:18.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.723+0000 7f172be00640 1 -- 172.21.15.12:0/2597819766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574e9b6f550 msgr2=0x5574e9b719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:18.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.723+0000 7f172be00640 1 --2- 172.21.15.12:0/2597819766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574e9b6f550 0x5574e9b719e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.723+0000 7f172be00640 1 -- 172.21.15.12:0/2597819766 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5574e9b699f0 msgr2=0x5574e9b69e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:18.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.723+0000 7f172be00640 1 --2- 172.21.15.12:0/2597819766 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5574e9b699f0 0x5574e9b69e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.723+0000 7f172be00640 1 -- 172.21.15.12:0/2597819766 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574e9a872c0 con 0x5574e9b6a3e0 2024-03-23T23:12:18.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.723+0000 7f1738c00640 1 --2- 172.21.15.12:0/2597819766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574e9b6f550 0x5574e9b719e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:12:18.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.723+0000 7f1733e00640 1 --2- 172.21.15.12:0/2597819766 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5574e9b699f0 0x5574e9b69e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:12:18.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.723+0000 7f1738c00640 1 --2- 172.21.15.12:0/3374328816 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f17180a28a0 0x7f17180a4d60 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f172c009fd0 tx=0x7f172c00eb70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:18.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.723+0000 7f172be00640 1 --2- 172.21.15.12:0/2597819766 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b6a3e0 0x5574e9b6eee0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f171c02c180 tx=0x7f171c005b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de0494d0057ce859 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:18.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.724+0000 7f1729200640 1 -- 172.21.15.12:0/2597819766 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171c043020 con 0x5574e9b6a3e0 2024-03-23T23:12:18.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.724+0000 7f1729200640 1 -- 172.21.15.12:0/2597819766 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f171c0323c0 con 0x5574e9b6a3e0 2024-03-23T23:12:18.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.724+0000 7f1729200640 1 -- 172.21.15.12:0/2597819766 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171c039040 con 0x5574e9b6a3e0 2024-03-23T23:12:18.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.725+0000 7f173d80a740 1 -- 172.21.15.12:0/2597819766 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b6a3e0 msgr2=0x5574e9b6eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.725+0000 7f173d80a740 1 --2- 172.21.15.12:0/2597819766 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b6a3e0 0x5574e9b6eee0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f171c02c180 tx=0x7f171c005b20 comp rx=0 tx=0).stop 2024-03-23T23:12:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.725+0000 7f173d80a740 1 -- 172.21.15.12:0/2597819766 shutdown_connections 2024-03-23T23:12:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.725+0000 7f173d80a740 1 --2- 172.21.15.12:0/2597819766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574e9b6f550 0x5574e9b719e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.725+0000 7f173d80a740 1 --2- 172.21.15.12:0/2597819766 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b6a3e0 0x5574e9b6eee0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.725+0000 7f173d80a740 1 --2- 172.21.15.12:0/2597819766 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5574e9b699f0 0x5574e9b69e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.725+0000 7f173d80a740 1 -- 172.21.15.12:0/2597819766 >> 172.21.15.12:0/2597819766 conn(0x5574e9b5c750 msgr2=0x5574e9b5cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.725+0000 7f173d80a740 1 -- 172.21.15.12:0/2597819766 shutdown_connections 2024-03-23T23:12:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.725+0000 7f173d80a740 1 -- 172.21.15.12:0/2597819766 wait complete. 2024-03-23T23:12:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.725+0000 7f173d80a740 1 -- start start 2024-03-23T23:12:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.726+0000 7f173d80a740 1 Processor -- start 2024-03-23T23:12:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.726+0000 7f173d80a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b699f0 0x5574e9b73bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.726+0000 7f173d80a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5574e9b6a3e0 0x5574e9b74110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.726+0000 7f173d80a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574e9b6f550 0x5574e9b7b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.726+0000 7f173d80a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574e9a82a20 con 0x5574e9b6a3e0 2024-03-23T23:12:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.726+0000 7f173d80a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574e9ac28d0 con 0x5574e9b699f0 2024-03-23T23:12:18.728 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.726+0000 7f173d80a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574e9b724a0 con 0x5574e9b6f550 2024-03-23T23:12:18.728 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.726+0000 7f173d80a740 10 client.0 _ll_register_callbacks cb 0x5574e9abf1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:12:18.728 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.726+0000 7f1738c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574e9b6f550 0x5574e9b7b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:18.728 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.726+0000 7f172be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5574e9b6a3e0 0x5574e9b74110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:18.728 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.726+0000 7f1733e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b699f0 0x5574e9b73bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:18.728 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.726+0000 7f1738c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574e9b6f550 0x5574e9b7b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:57808/0 (socket says 172.21.15.12:57808) 2024-03-23T23:12:18.728 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.726+0000 7f1738c00640 1 -- 172.21.15.12:0/4206868923 learned_addr learned my addr 172.21.15.12:0/4206868923 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.727+0000 7f1738c00640 1 -- 172.21.15.12:0/4206868923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b699f0 msgr2=0x5574e9b73bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.727+0000 7f1738c00640 1 --2- 172.21.15.12:0/4206868923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b699f0 0x5574e9b73bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.727+0000 7f1738c00640 1 -- 172.21.15.12:0/4206868923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5574e9b6a3e0 msgr2=0x5574e9b74110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.727+0000 7f1738c00640 1 --2- 172.21.15.12:0/4206868923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5574e9b6a3e0 0x5574e9b74110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.727+0000 7f1738c00640 1 -- 172.21.15.12:0/4206868923 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5574e97db210 con 0x5574e9b6f550 2024-03-23T23:12:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.727+0000 7f1733e00640 1 --2- 172.21.15.12:0/4206868923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b699f0 0x5574e9b73bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:12:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.727+0000 7f1738c00640 1 --2- 172.21.15.12:0/4206868923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574e9b6f550 0x5574e9b7b150 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f172c01d6a0 tx=0x7f172c004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36c80c20340c33e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.727+0000 7f1721600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:12:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.727+0000 7f1721600640 1 -- 172.21.15.12:0/4206868923 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172c01e7d0 con 0x5574e9b6f550 2024-03-23T23:12:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.727+0000 7f1721600640 1 -- 172.21.15.12:0/4206868923 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f172c017070 con 0x5574e9b6f550 2024-03-23T23:12:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.727+0000 7f173d80a740 1 -- 172.21.15.12:0/4206868923 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f171c0323c0 con 0x5574e9b6f550 2024-03-23T23:12:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.728+0000 7f1721600640 1 -- 172.21.15.12:0/4206868923 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f172c003970 con 0x5574e9b6f550 2024-03-23T23:12:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.728+0000 7f173d80a740 10 client.28222 fetch_fsmap learned FSMap version 23 2024-03-23T23:12:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.728+0000 7f173d80a740 1 -- 172.21.15.12:0/4206868923 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5574e98a7620 con 0x5574e9b6f550 2024-03-23T23:12:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.728+0000 7f1721600640 1 -- 172.21.15.12:0/4206868923 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f171c0323c0 con 0x5574e9b6f550 2024-03-23T23:12:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.728+0000 7f1721600640 1 -- 172.21.15.12:0/4206868923 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f172c025070 con 0x5574e9b6f550 2024-03-23T23:12:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.728+0000 7f173d80a740 10 client.28222 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:12:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.728+0000 7f173d80a740 10 client.28222 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:12:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.728+0000 7f173d80a740 20 client.28222 populate_metadata read hostname 'smithi012' 2024-03-23T23:12:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.728+0000 7f173d80a740 1 --2- 172.21.15.12:0/4206868923 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5574e9b7fa10 0x5574e9b81e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.728+0000 7f173d80a740 4 client.28222 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:12:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.728+0000 7f173d80a740 1 -- 172.21.15.12:0/4206868923 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5574e9ad3d70 con 0x5574e9b7fa10 2024-03-23T23:12:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.729+0000 7f1733e00640 1 --2- 172.21.15.12:0/4206868923 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5574e9b7fa10 0x5574e9b81e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.729+0000 7f1733e00640 1 --2- 172.21.15.12:0/4206868923 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5574e9b7fa10 0x5574e9b81e20 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=118231402ed6501c server_cookie=d1359b2067e3dffb in_seq=0 out_seq=0 2024-03-23T23:12:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.729+0000 7f1721600640 10 client.28222 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.730+0000 7f1721600640 1 -- 172.21.15.12:0/4206868923 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5574e9ad3d70 con 0x5574e9b7fa10 2024-03-23T23:12:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.730+0000 7f1721600640 10 client.28222 handle_command_reply: tid=0 2024-03-23T23:12:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.730+0000 7f173d80a740 1 client.28222 shutdown 2024-03-23T23:12:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.730+0000 7f173d80a740 1 -- 172.21.15.12:0/4206868923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574e9b6f550 msgr2=0x5574e9b7b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.730+0000 7f173d80a740 1 --2- 172.21.15.12:0/4206868923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574e9b6f550 0x5574e9b7b150 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f172c01d6a0 tx=0x7f172c004ab0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.730+0000 7f173d80a740 1 -- 172.21.15.12:0/4206868923 shutdown_connections 2024-03-23T23:12:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.730+0000 7f173d80a740 1 --2- 172.21.15.12:0/4206868923 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5574e9b7fa10 0x5574e9b81e20 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.730+0000 7f173d80a740 1 --2- 172.21.15.12:0/4206868923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574e9b6f550 0x5574e9b7b150 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.730+0000 7f173d80a740 1 --2- 172.21.15.12:0/4206868923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5574e9b6a3e0 0x5574e9b74110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.730+0000 7f173d80a740 1 --2- 172.21.15.12:0/4206868923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b699f0 0x5574e9b73bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.730+0000 7f173d80a740 1 -- 172.21.15.12:0/4206868923 >> 172.21.15.12:0/4206868923 conn(0x5574e9b5c750 msgr2=0x5574e9b5b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.730+0000 7f1721600640 0 client.28222 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.730+0000 7f173d80a740 1 -- 172.21.15.12:0/4206868923 shutdown_connections 2024-03-23T23:12:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.730+0000 7f173d80a740 1 -- 172.21.15.12:0/4206868923 wait complete. 2024-03-23T23:12:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.730+0000 7f173d80a740 20 client.28222 trim_cache size 0 max 16384 2024-03-23T23:12:18.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.745+0000 7f173d80a740 1 Processor -- start 2024-03-23T23:12:18.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.745+0000 7f173d80a740 1 -- start start 2024-03-23T23:12:18.746 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.746+0000 7f173d80a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b699f0 0x5574e9b5b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:18.746 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.746+0000 7f173d80a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5574e9b6a3e0 0x5574e9b7b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:18.746 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.746+0000 7f173d80a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574e9b6f550 0x5574e9b5b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:18.746 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.746+0000 7f173d80a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17341a8cd0 con 0x5574e9b6a3e0 2024-03-23T23:12:18.746 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.746+0000 7f173d80a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17341a67d0 con 0x5574e9b699f0 2024-03-23T23:12:18.746 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.746+0000 7f173d80a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17341a4890 con 0x5574e9b6f550 2024-03-23T23:12:18.747 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.746+0000 7f1733e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b699f0 0x5574e9b5b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:18.747 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.746+0000 7f1733e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b699f0 0x5574e9b5b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60168/0 (socket says 172.21.15.12:60168) 2024-03-23T23:12:18.747 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.746+0000 7f1733e00640 1 -- 172.21.15.12:0/4093108724 learned_addr learned my addr 172.21.15.12:0/4093108724 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:18.747 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.746+0000 7f1738c00640 1 --2- 172.21.15.12:0/4093108724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574e9b6f550 0x5574e9b5b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:18.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.746+0000 7f172be00640 1 --2- 172.21.15.12:0/4093108724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5574e9b6a3e0 0x5574e9b7b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:18.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.746+0000 7f1733e00640 1 -- 172.21.15.12:0/4093108724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574e9b6f550 msgr2=0x5574e9b5b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:18.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.746+0000 7f1733e00640 1 --2- 172.21.15.12:0/4093108724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574e9b6f550 0x5574e9b5b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.746+0000 7f1733e00640 1 -- 172.21.15.12:0/4093108724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5574e9b6a3e0 msgr2=0x5574e9b7b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:18.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.746+0000 7f1733e00640 1 --2- 172.21.15.12:0/4093108724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5574e9b6a3e0 0x5574e9b7b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.746+0000 7f1733e00640 1 -- 172.21.15.12:0/4093108724 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574e9b573a0 con 0x5574e9b699f0 2024-03-23T23:12:18.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.746+0000 7f172be00640 1 --2- 172.21.15.12:0/4093108724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5574e9b6a3e0 0x5574e9b7b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:12:18.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.747+0000 7f1733e00640 1 --2- 172.21.15.12:0/4093108724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b699f0 0x5574e9b5b0c0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f172409b4b0 tx=0x7f1724097ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44ffd22f5a821bf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:18.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.747+0000 7f1723e00640 1 -- 172.21.15.12:0/4093108724 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17240a3f50 con 0x5574e9b699f0 2024-03-23T23:12:18.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.747+0000 7f1723e00640 1 -- 172.21.15.12:0/4093108724 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1724099ab0 con 0x5574e9b699f0 2024-03-23T23:12:18.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.747+0000 7f1723e00640 1 -- 172.21.15.12:0/4093108724 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17240afdf0 con 0x5574e9b699f0 2024-03-23T23:12:18.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.748+0000 7f173d80a740 1 -- 172.21.15.12:0/4093108724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b699f0 msgr2=0x5574e9b5b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:18.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.748+0000 7f173d80a740 1 --2- 172.21.15.12:0/4093108724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b699f0 0x5574e9b5b0c0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f172409b4b0 tx=0x7f1724097ad0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.751 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.748+0000 7f173d80a740 1 -- 172.21.15.12:0/4093108724 shutdown_connections 2024-03-23T23:12:18.751 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.748+0000 7f173d80a740 1 --2- 172.21.15.12:0/4093108724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574e9b6f550 0x5574e9b5b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.751 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.748+0000 7f173d80a740 1 --2- 172.21.15.12:0/4093108724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5574e9b6a3e0 0x5574e9b7b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.751 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.748+0000 7f173d80a740 1 --2- 172.21.15.12:0/4093108724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b699f0 0x5574e9b5b0c0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.751 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.748+0000 7f173d80a740 1 -- 172.21.15.12:0/4093108724 >> 172.21.15.12:0/4093108724 conn(0x5574e9b5c750 msgr2=0x5574e9b6b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:18.751 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.748+0000 7f173d80a740 1 -- 172.21.15.12:0/4093108724 shutdown_connections 2024-03-23T23:12:18.751 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.748+0000 7f173d80a740 1 -- 172.21.15.12:0/4093108724 wait complete. 2024-03-23T23:12:18.751 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.749+0000 7f173d80a740 1 -- start start 2024-03-23T23:12:18.751 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.749+0000 7f173d80a740 1 Processor -- start 2024-03-23T23:12:18.751 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.750+0000 7f173d80a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5574e9b699f0 0x5574e9b78f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:18.751 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.750+0000 7f173d80a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b6a3e0 0x5574e9b794a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:18.751 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.750+0000 7f173d80a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574e9b6f550 0x5574e9b7a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:18.751 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.750+0000 7f173d80a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17341a4e90 con 0x5574e9b699f0 2024-03-23T23:12:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.750+0000 7f173d80a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17341a6170 con 0x5574e9b6a3e0 2024-03-23T23:12:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.750+0000 7f173d80a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17341a54b0 con 0x5574e9b6f550 2024-03-23T23:12:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.750+0000 7f173d80a740 10 client.0 _ll_register_callbacks cb 0x7f172c00e450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:12:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.750+0000 7f172be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b6a3e0 0x5574e9b794a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.750+0000 7f172be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b6a3e0 0x5574e9b794a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:60174/0 (socket says 172.21.15.12:60174) 2024-03-23T23:12:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.750+0000 7f1733e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5574e9b699f0 0x5574e9b78f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.750+0000 7f172be00640 1 -- 172.21.15.12:0/972143117 learned_addr learned my addr 172.21.15.12:0/972143117 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.750+0000 7f1738c00640 1 --2- 172.21.15.12:0/972143117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574e9b6f550 0x5574e9b7a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.750+0000 7f172be00640 1 -- 172.21.15.12:0/972143117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574e9b6f550 msgr2=0x5574e9b7a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.750+0000 7f172be00640 1 --2- 172.21.15.12:0/972143117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574e9b6f550 0x5574e9b7a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.750+0000 7f172be00640 1 -- 172.21.15.12:0/972143117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5574e9b699f0 msgr2=0x5574e9b78f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.750+0000 7f172be00640 1 --2- 172.21.15.12:0/972143117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5574e9b699f0 0x5574e9b78f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.750+0000 7f172be00640 1 -- 172.21.15.12:0/972143117 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5574e9b7aaa0 con 0x5574e9b6a3e0 2024-03-23T23:12:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.751+0000 7f172be00640 1 --2- 172.21.15.12:0/972143117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b6a3e0 0x5574e9b794a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f171c02c260 tx=0x7f171c02bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f89c8e7bfa33d7a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:18.753 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.751+0000 7f1721600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:12:18.753 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.751+0000 7f1721600640 1 -- 172.21.15.12:0/972143117 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171c043040 con 0x5574e9b6a3e0 2024-03-23T23:12:18.753 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.751+0000 7f1721600640 1 -- 172.21.15.12:0/972143117 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f171c004020 con 0x5574e9b6a3e0 2024-03-23T23:12:18.753 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.751+0000 7f173d80a740 1 -- 172.21.15.12:0/972143117 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1724099ab0 con 0x5574e9b6a3e0 2024-03-23T23:12:18.754 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.751+0000 7f1721600640 1 -- 172.21.15.12:0/972143117 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f171c057050 con 0x5574e9b6a3e0 2024-03-23T23:12:18.754 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.751+0000 7f173d80a740 10 client.28262 fetch_fsmap learned FSMap version 23 2024-03-23T23:12:18.754 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.751+0000 7f173d80a740 1 -- 172.21.15.12:0/972143117 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5574e9bdbb20 con 0x5574e9b6a3e0 2024-03-23T23:12:18.754 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.752+0000 7f1721600640 1 -- 172.21.15.12:0/972143117 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1724099ab0 con 0x5574e9b6a3e0 2024-03-23T23:12:18.754 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.752+0000 7f1721600640 1 -- 172.21.15.12:0/972143117 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f171c02c510 con 0x5574e9b6a3e0 2024-03-23T23:12:18.754 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.752+0000 7f173d80a740 10 client.28262 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:12:18.754 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.752+0000 7f173d80a740 10 client.28262 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:12:18.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.752+0000 7f173d80a740 20 client.28262 populate_metadata read hostname 'smithi012' 2024-03-23T23:12:18.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.752+0000 7f173d80a740 1 --2- 172.21.15.12:0/972143117 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5574e9bdc060 0x5574e9bde480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:18.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.752+0000 7f173d80a740 4 client.28262 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:12:18.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.752+0000 7f173d80a740 1 -- 172.21.15.12:0/972143117 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5574e9ace0c0 con 0x5574e9bdc060 2024-03-23T23:12:18.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.752+0000 7f1733e00640 1 --2- 172.21.15.12:0/972143117 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5574e9bdc060 0x5574e9bde480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:18.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.753+0000 7f1733e00640 1 --2- 172.21.15.12:0/972143117 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5574e9bdc060 0x5574e9bde480 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ee29de3644f7cc8 server_cookie=e6c3d464ea4ed30a in_seq=0 out_seq=0 2024-03-23T23:12:18.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.753+0000 7f1721600640 10 client.28262 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:18.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.766+0000 7f1721600640 1 -- 172.21.15.12:0/972143117 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5574e9ace0c0 con 0x5574e9bdc060 2024-03-23T23:12:18.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.766+0000 7f1721600640 10 client.28262 handle_command_reply: tid=0 2024-03-23T23:12:18.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.766+0000 7f173d80a740 1 client.28262 shutdown 2024-03-23T23:12:18.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.766+0000 7f173d80a740 1 -- 172.21.15.12:0/972143117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b6a3e0 msgr2=0x5574e9b794a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:18.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.766+0000 7f173d80a740 1 --2- 172.21.15.12:0/972143117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b6a3e0 0x5574e9b794a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f171c02c260 tx=0x7f171c02bb70 comp rx=0 tx=0).stop 2024-03-23T23:12:18.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.766+0000 7f173d80a740 1 -- 172.21.15.12:0/972143117 shutdown_connections 2024-03-23T23:12:18.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.766+0000 7f173d80a740 1 --2- 172.21.15.12:0/972143117 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5574e9bdc060 0x5574e9bde480 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.766+0000 7f173d80a740 1 --2- 172.21.15.12:0/972143117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574e9b6f550 0x5574e9b7a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.766+0000 7f173d80a740 1 --2- 172.21.15.12:0/972143117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574e9b6a3e0 0x5574e9b794a0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.766+0000 7f173d80a740 1 --2- 172.21.15.12:0/972143117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5574e9b699f0 0x5574e9b78f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.766+0000 7f1721600640 0 client.28262 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:18.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.766+0000 7f173d80a740 1 -- 172.21.15.12:0/972143117 >> 172.21.15.12:0/972143117 conn(0x5574e9b5c750 msgr2=0x5574e9b73bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:18.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.766+0000 7f173d80a740 1 -- 172.21.15.12:0/972143117 shutdown_connections 2024-03-23T23:12:18.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.766+0000 7f173d80a740 1 -- 172.21.15.12:0/972143117 wait complete. 2024-03-23T23:12:18.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.767+0000 7f173d80a740 20 client.28262 trim_cache size 0 max 16384 2024-03-23T23:12:18.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:12:18.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:12:18.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:12:18.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:12:18.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.767+0000 7f173ac00640 1 -- 172.21.15.12:0/3374328816 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f17180a28a0 msgr2=0x7f17180a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:18.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.767+0000 7f173ac00640 1 --2- 172.21.15.12:0/3374328816 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f17180a28a0 0x7f17180a4d60 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f172c009fd0 tx=0x7f172c00eb70 comp rx=0 tx=0).stop 2024-03-23T23:12:18.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.767+0000 7f173ac00640 1 -- 172.21.15.12:0/3374328816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17341038f0 msgr2=0x7f1734199b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:18.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.767+0000 7f173ac00640 1 --2- 172.21.15.12:0/3374328816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17341038f0 0x7f1734199b90 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f172400e9c0 tx=0x7f172400ee80 comp rx=0 tx=0).stop 2024-03-23T23:12:18.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.768+0000 7f173ac00640 1 -- 172.21.15.12:0/3374328816 shutdown_connections 2024-03-23T23:12:18.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.768+0000 7f173ac00640 1 --2- 172.21.15.12:0/3374328816 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f17180a28a0 0x7f17180a4d60 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.768+0000 7f173ac00640 1 --2- 172.21.15.12:0/3374328816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17341054a0 0x7f17341a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.768+0000 7f173ac00640 1 --2- 172.21.15.12:0/3374328816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1734104ae0 0x7f173419a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.768+0000 7f173ac00640 1 --2- 172.21.15.12:0/3374328816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17341038f0 0x7f1734199b90 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:18.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.768+0000 7f173ac00640 1 -- 172.21.15.12:0/3374328816 >> 172.21.15.12:0/3374328816 conn(0x7f17340fee60 msgr2=0x7f1734107de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:18.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.768+0000 7f173ac00640 1 -- 172.21.15.12:0/3374328816 shutdown_connections 2024-03-23T23:12:18.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:18.768+0000 7f173ac00640 1 -- 172.21.15.12:0/3374328816 wait complete. 2024-03-23T23:12:19.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:19 smithi012 ceph-mon[31695]: pgmap v705: 97 pgs: 97 active+clean; 3.1 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:12:19.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:19 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:12:19.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:19 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:12:19.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:19 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:19.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:19 smithi173 ceph-mon[36116]: pgmap v705: 97 pgs: 97 active+clean; 3.1 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:12:19.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:19 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:12:19.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:19 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:12:19.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:19 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:19.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:19 smithi203 ceph-mon[36693]: pgmap v705: 97 pgs: 97 active+clean; 3.1 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:12:19.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:12:19.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:12:19.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:12:21.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:21 smithi012 ceph-mon[31695]: pgmap v706: 97 pgs: 97 active+clean; 3.1 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-03-23T23:12:21.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:21 smithi173 ceph-mon[36116]: pgmap v706: 97 pgs: 97 active+clean; 3.1 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-03-23T23:12:21.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:21 smithi203 ceph-mon[36693]: pgmap v706: 97 pgs: 97 active+clean; 3.1 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-03-23T23:12:23.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:23 smithi012 ceph-mon[31695]: pgmap v707: 97 pgs: 97 active+clean; 3.1 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:12:23.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:23 smithi173 ceph-mon[36116]: pgmap v707: 97 pgs: 97 active+clean; 3.1 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:12:23.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:23 smithi203 ceph-mon[36693]: pgmap v707: 97 pgs: 97 active+clean; 3.1 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:12:25.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:25 smithi012 ceph-mon[31695]: pgmap v708: 97 pgs: 97 active+clean; 3.1 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:12:25.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:25 smithi173 ceph-mon[36116]: pgmap v708: 97 pgs: 97 active+clean; 3.1 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:12:25.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:25 smithi203 ceph-mon[36693]: pgmap v708: 97 pgs: 97 active+clean; 3.1 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:12:27.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:27 smithi012 ceph-mon[31695]: pgmap v709: 97 pgs: 97 active+clean; 3.2 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:12:27.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:27 smithi173 ceph-mon[36116]: pgmap v709: 97 pgs: 97 active+clean; 3.2 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:12:27.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:27 smithi203 ceph-mon[36693]: pgmap v709: 97 pgs: 97 active+clean; 3.2 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:12:29.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:29 smithi012 ceph-mon[31695]: pgmap v710: 97 pgs: 97 active+clean; 3.2 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:12:29.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:29 smithi173 ceph-mon[36116]: pgmap v710: 97 pgs: 97 active+clean; 3.2 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:12:29.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:29 smithi203 ceph-mon[36693]: pgmap v710: 97 pgs: 97 active+clean; 3.2 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:12:29.723 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:12:29.724 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:12:29.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.965+0000 7f93dbe00640 1 -- 172.21.15.12:0/4200720645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93dc1037c0 msgr2=0x7f93dc103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:29.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.965+0000 7f93dbe00640 1 --2- 172.21.15.12:0/4200720645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93dc1037c0 0x7f93dc103be0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f93cc009900 tx=0x7f93cc02bc30 comp rx=0 tx=0).stop 2024-03-23T23:12:29.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.965+0000 7f93dbe00640 1 -- 172.21.15.12:0/4200720645 shutdown_connections 2024-03-23T23:12:29.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.965+0000 7f93dbe00640 1 --2- 172.21.15.12:0/4200720645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93dc105370 0x7f93dc10bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:29.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.965+0000 7f93dbe00640 1 --2- 172.21.15.12:0/4200720645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93dc1049b0 0x7f93dc104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:29.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.965+0000 7f93dbe00640 1 --2- 172.21.15.12:0/4200720645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93dc1037c0 0x7f93dc103be0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:29.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.965+0000 7f93dbe00640 1 -- 172.21.15.12:0/4200720645 >> 172.21.15.12:0/4200720645 conn(0x7f93dc0fed50 msgr2=0x7f93dc101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:29.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.965+0000 7f93dbe00640 1 -- 172.21.15.12:0/4200720645 shutdown_connections 2024-03-23T23:12:29.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.965+0000 7f93dbe00640 1 -- 172.21.15.12:0/4200720645 wait complete. 2024-03-23T23:12:29.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.967+0000 7f93dbe00640 1 Processor -- start 2024-03-23T23:12:29.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.967+0000 7f93dbe00640 1 -- start start 2024-03-23T23:12:29.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.967+0000 7f93dbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93dc1037c0 0x7f93dc199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:29.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.967+0000 7f93dbe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93dc1049b0 0x7f93dc19a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:29.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.967+0000 7f93dbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93dc105370 0x7f93dc1a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:29.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.967+0000 7f93dbe00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc10f460 con 0x7f93dc1049b0 2024-03-23T23:12:29.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.967+0000 7f93dbe00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc10f2e0 con 0x7f93dc1037c0 2024-03-23T23:12:29.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.967+0000 7f93dbe00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc10f5e0 con 0x7f93dc105370 2024-03-23T23:12:29.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.968+0000 7f93daa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93dc1037c0 0x7f93dc199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:29.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.968+0000 7f93daa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93dc1037c0 0x7f93dc199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37850/0 (socket says 172.21.15.12:37850) 2024-03-23T23:12:29.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.968+0000 7f93daa00640 1 -- 172.21.15.12:0/2469041037 learned_addr learned my addr 172.21.15.12:0/2469041037 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:29.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.968+0000 7f93db400640 1 --2- 172.21.15.12:0/2469041037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93dc105370 0x7f93dc1a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:29.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.968+0000 7f93d3e00640 1 --2- 172.21.15.12:0/2469041037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93dc1049b0 0x7f93dc19a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:29.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.968+0000 7f93daa00640 1 -- 172.21.15.12:0/2469041037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93dc105370 msgr2=0x7f93dc1a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:29.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.968+0000 7f93daa00640 1 --2- 172.21.15.12:0/2469041037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93dc105370 0x7f93dc1a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:29.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.968+0000 7f93daa00640 1 -- 172.21.15.12:0/2469041037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93dc1049b0 msgr2=0x7f93dc19a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:29.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.968+0000 7f93daa00640 1 --2- 172.21.15.12:0/2469041037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93dc1049b0 0x7f93dc19a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:29.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.968+0000 7f93daa00640 1 -- 172.21.15.12:0/2469041037 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93dc1a1720 con 0x7f93dc1037c0 2024-03-23T23:12:29.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.968+0000 7f93daa00640 1 --2- 172.21.15.12:0/2469041037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93dc1037c0 0x7f93dc199b00 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f93cc000c00 tx=0x7f93cc02c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:29.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.969+0000 7f93d3400640 1 -- 172.21.15.12:0/2469041037 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93cc043020 con 0x7f93dc1037c0 2024-03-23T23:12:29.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.969+0000 7f93d3400640 1 -- 172.21.15.12:0/2469041037 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93cc0054e0 con 0x7f93dc1037c0 2024-03-23T23:12:29.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.969+0000 7f93d3400640 1 -- 172.21.15.12:0/2469041037 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93cc003ce0 con 0x7f93dc1037c0 2024-03-23T23:12:29.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.969+0000 7f93dbe00640 1 -- 172.21.15.12:0/2469041037 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93dc1a19b0 con 0x7f93dc1037c0 2024-03-23T23:12:29.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.969+0000 7f93dbe00640 1 -- 172.21.15.12:0/2469041037 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93dc1a1ed0 con 0x7f93dc1037c0 2024-03-23T23:12:29.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.972+0000 7f93d3400640 1 -- 172.21.15.12:0/2469041037 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f93cc046020 con 0x7f93dc1037c0 2024-03-23T23:12:29.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.972+0000 7f93e43d2740 1 Processor -- start 2024-03-23T23:12:29.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.973+0000 7f93e43d2740 1 -- start start 2024-03-23T23:12:29.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.973+0000 7f93e43d2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829741050 0x55a829741470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:29.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.973+0000 7f93e43d2740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a829741a40 0x55a829746540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:29.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.973+0000 7f93e43d2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a829746bb0 0x55a829749040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:29.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.973+0000 7f93e43d2740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8296a6250 con 0x55a829741a40 2024-03-23T23:12:29.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.973+0000 7f93e43d2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a82964ea30 con 0x55a829741050 2024-03-23T23:12:29.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.973+0000 7f93e43d2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8296a22f0 con 0x55a829746bb0 2024-03-23T23:12:29.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.973+0000 7f93db400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829741050 0x55a829741470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:29.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.973+0000 7f93db400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829741050 0x55a829741470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37866/0 (socket says 172.21.15.12:37866) 2024-03-23T23:12:29.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.973+0000 7f93db400640 1 -- 172.21.15.12:0/1234863476 learned_addr learned my addr 172.21.15.12:0/1234863476 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:29.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.973+0000 7f93daa00640 1 --2- 172.21.15.12:0/1234863476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a829741a40 0x55a829746540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:29.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.973+0000 7f93d3e00640 1 --2- 172.21.15.12:0/1234863476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a829746bb0 0x55a829749040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:29.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.973+0000 7f93db400640 1 -- 172.21.15.12:0/1234863476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a829746bb0 msgr2=0x55a829749040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:29.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.973+0000 7f93db400640 1 --2- 172.21.15.12:0/1234863476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a829746bb0 0x55a829749040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:29.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.973+0000 7f93db400640 1 -- 172.21.15.12:0/1234863476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a829741a40 msgr2=0x55a829746540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:29.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.973+0000 7f93db400640 1 --2- 172.21.15.12:0/1234863476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a829741a40 0x55a829746540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:29.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.973+0000 7f93db400640 1 -- 172.21.15.12:0/1234863476 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a829639690 con 0x55a829741050 2024-03-23T23:12:29.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.973+0000 7f93d3e00640 1 --2- 172.21.15.12:0/1234863476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a829746bb0 0x55a829749040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:12:29.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.974+0000 7f93d3400640 1 --2- 172.21.15.12:0/2469041037 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f93c00a2800 0x7f93c00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:29.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.974+0000 7f93db400640 1 --2- 172.21.15.12:0/1234863476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829741050 0x55a829741470 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f93d4010370 tx=0x7f93d4010830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e50d98ed2b333898 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:29.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.974+0000 7f93d3400640 1 -- 172.21.15.12:0/2469041037 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f93cc0f0c40 con 0x7f93dc1037c0 2024-03-23T23:12:29.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.974+0000 7f93cb400640 1 -- 172.21.15.12:0/1234863476 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d4016070 con 0x55a829741050 2024-03-23T23:12:29.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.974+0000 7f93daa00640 1 --2- 172.21.15.12:0/2469041037 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f93c00a2800 0x7f93c00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:29.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.974+0000 7f93cb400640 1 -- 172.21.15.12:0/1234863476 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93d4004d20 con 0x55a829741050 2024-03-23T23:12:29.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.974+0000 7f93cb400640 1 -- 172.21.15.12:0/1234863476 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d4005480 con 0x55a829741050 2024-03-23T23:12:29.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.975+0000 7f93e43d2740 1 -- 172.21.15.12:0/1234863476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829741050 msgr2=0x55a829741470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:29.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.975+0000 7f93e43d2740 1 --2- 172.21.15.12:0/1234863476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829741050 0x55a829741470 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f93d4010370 tx=0x7f93d4010830 comp rx=0 tx=0).stop 2024-03-23T23:12:29.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.975+0000 7f93daa00640 1 --2- 172.21.15.12:0/2469041037 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f93c00a2800 0x7f93c00a4cc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f93cc0bd520 tx=0x7f93cc02bc30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:29.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.975+0000 7f93e43d2740 1 -- 172.21.15.12:0/1234863476 shutdown_connections 2024-03-23T23:12:29.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.975+0000 7f93e43d2740 1 --2- 172.21.15.12:0/1234863476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a829746bb0 0x55a829749040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:29.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.975+0000 7f93e43d2740 1 --2- 172.21.15.12:0/1234863476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a829741a40 0x55a829746540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:29.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.975+0000 7f93e43d2740 1 --2- 172.21.15.12:0/1234863476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829741050 0x55a829741470 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:29.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.975+0000 7f93e43d2740 1 -- 172.21.15.12:0/1234863476 >> 172.21.15.12:0/1234863476 conn(0x55a829709200 msgr2=0x55a82973c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.975+0000 7f93e43d2740 1 -- 172.21.15.12:0/1234863476 shutdown_connections 2024-03-23T23:12:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.975+0000 7f93e43d2740 1 -- 172.21.15.12:0/1234863476 wait complete. 2024-03-23T23:12:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.975+0000 7f93e43d2740 1 -- start start 2024-03-23T23:12:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.976+0000 7f93e43d2740 1 Processor -- start 2024-03-23T23:12:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.976+0000 7f93e43d2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a829741050 0x55a82974b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.976+0000 7f93e43d2740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a829741a40 0x55a82974b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.976+0000 7f93e43d2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829746bb0 0x55a829752610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.976+0000 7f93e43d2740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a829749b60 con 0x55a829741a40 2024-03-23T23:12:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.976+0000 7f93e43d2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a829661e80 con 0x55a829746bb0 2024-03-23T23:12:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.976+0000 7f93e43d2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a829749ce0 con 0x55a829741050 2024-03-23T23:12:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.976+0000 7f93e43d2740 10 client.0 _ll_register_callbacks cb 0x55a829674130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:12:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.976+0000 7f93db400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a829741050 0x55a82974b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.976+0000 7f93d3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a829741a40 0x55a82974b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.976+0000 7f93db400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a829741050 0x55a82974b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:57890/0 (socket says 172.21.15.12:57890) 2024-03-23T23:12:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.976+0000 7f93db400640 1 -- 172.21.15.12:0/2445228040 learned_addr learned my addr 172.21.15.12:0/2445228040 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.977+0000 7f93db400640 1 --2- 172.21.15.12:0/2445228040 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829746bb0 0x55a829752610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.977+0000 7f93db400640 1 -- 172.21.15.12:0/2445228040 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829746bb0 msgr2=0x55a829752610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.977+0000 7f93db400640 1 --2- 172.21.15.12:0/2445228040 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829746bb0 0x55a829752610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.977+0000 7f93db400640 1 -- 172.21.15.12:0/2445228040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a829741a40 msgr2=0x55a82974b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.977+0000 7f93db400640 1 --2- 172.21.15.12:0/2445228040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a829741a40 0x55a82974b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.977+0000 7f93db400640 1 -- 172.21.15.12:0/2445228040 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a8294847d0 con 0x55a829741050 2024-03-23T23:12:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.977+0000 7f93db400640 1 --2- 172.21.15.12:0/2445228040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a829741050 0x55a82974b090 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f93d4004ff0 tx=0x7f93d4009bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa116fe83552255a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.977+0000 7f93bfe00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:12:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.977+0000 7f93bfe00640 1 -- 172.21.15.12:0/2445228040 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d4016070 con 0x55a829741050 2024-03-23T23:12:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.977+0000 7f93bfe00640 1 -- 172.21.15.12:0/2445228040 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f93d4003970 con 0x55a829741050 2024-03-23T23:12:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.977+0000 7f93e43d2740 1 -- 172.21.15.12:0/2445228040 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f93d4004d20 con 0x55a829741050 2024-03-23T23:12:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.978+0000 7f93bfe00640 1 -- 172.21.15.12:0/2445228040 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93d4003b70 con 0x55a829741050 2024-03-23T23:12:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.978+0000 7f93e43d2740 10 client.28240 fetch_fsmap learned FSMap version 23 2024-03-23T23:12:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.978+0000 7f93e43d2740 1 -- 172.21.15.12:0/2445228040 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a8293ba3f0 con 0x55a829741050 2024-03-23T23:12:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.978+0000 7f93bfe00640 1 -- 172.21.15.12:0/2445228040 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93d4004d20 con 0x55a829741050 2024-03-23T23:12:29.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.978+0000 7f93bfe00640 1 -- 172.21.15.12:0/2445228040 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f93d4028040 con 0x55a829741050 2024-03-23T23:12:29.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.978+0000 7f93e43d2740 10 client.28240 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:12:29.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.978+0000 7f93e43d2740 10 client.28240 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:12:29.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.978+0000 7f93e43d2740 20 client.28240 populate_metadata read hostname 'smithi012' 2024-03-23T23:12:29.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.978+0000 7f93e43d2740 1 --2- 172.21.15.12:0/2445228040 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a829756c80 0x55a829759070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:29.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.978+0000 7f93e43d2740 4 client.28240 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:12:29.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.978+0000 7f93e43d2740 1 -- 172.21.15.12:0/2445228040 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a8296b3790 con 0x55a829756c80 2024-03-23T23:12:29.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.979+0000 7f93d3e00640 1 --2- 172.21.15.12:0/2445228040 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a829756c80 0x55a829759070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:29.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.979+0000 7f93d3e00640 1 --2- 172.21.15.12:0/2445228040 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a829756c80 0x55a829759070 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba1dc507887096c4 server_cookie=199e2a3b575503a1 in_seq=0 out_seq=0 2024-03-23T23:12:29.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.979+0000 7f93bfe00640 10 client.28240 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:29.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.980+0000 7f93bfe00640 1 -- 172.21.15.12:0/2445228040 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a8296b3790 con 0x55a829756c80 2024-03-23T23:12:29.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.980+0000 7f93bfe00640 10 client.28240 handle_command_reply: tid=0 2024-03-23T23:12:29.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.980+0000 7f93e43d2740 1 client.28240 shutdown 2024-03-23T23:12:29.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.980+0000 7f93e43d2740 1 -- 172.21.15.12:0/2445228040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a829741050 msgr2=0x55a82974b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:29.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.980+0000 7f93e43d2740 1 --2- 172.21.15.12:0/2445228040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a829741050 0x55a82974b090 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f93d4004ff0 tx=0x7f93d4009bb0 comp rx=0 tx=0).stop 2024-03-23T23:12:29.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.980+0000 7f93e43d2740 1 -- 172.21.15.12:0/2445228040 shutdown_connections 2024-03-23T23:12:29.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.980+0000 7f93e43d2740 1 --2- 172.21.15.12:0/2445228040 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a829756c80 0x55a829759070 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:29.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.980+0000 7f93e43d2740 1 --2- 172.21.15.12:0/2445228040 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829746bb0 0x55a829752610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:29.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.980+0000 7f93e43d2740 1 --2- 172.21.15.12:0/2445228040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a829741a40 0x55a82974b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:29.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.980+0000 7f93e43d2740 1 --2- 172.21.15.12:0/2445228040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a829741050 0x55a82974b090 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:29.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.980+0000 7f93bfe00640 0 client.28240 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:29.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.980+0000 7f93e43d2740 1 -- 172.21.15.12:0/2445228040 >> 172.21.15.12:0/2445228040 conn(0x55a829709200 msgr2=0x55a82973bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:29.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.981+0000 7f93e43d2740 1 -- 172.21.15.12:0/2445228040 shutdown_connections 2024-03-23T23:12:29.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.981+0000 7f93e43d2740 1 -- 172.21.15.12:0/2445228040 wait complete. 2024-03-23T23:12:29.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.981+0000 7f93e43d2740 20 client.28240 trim_cache size 0 max 16384 2024-03-23T23:12:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.999+0000 7f93e43d2740 1 Processor -- start 2024-03-23T23:12:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.999+0000 7f93e43d2740 1 -- start start 2024-03-23T23:12:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.999+0000 7f93e43d2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a829741050 0x55a829740310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.999+0000 7f93e43d2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829741a40 0x55a8297525a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.999+0000 7f93e43d2740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a829746bb0 0x55a82973b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.999+0000 7f93e43d2740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc1acfd0 con 0x55a829746bb0 2024-03-23T23:12:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.999+0000 7f93e43d2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc1aaad0 con 0x55a829741a40 2024-03-23T23:12:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:29.999+0000 7f93e43d2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc1a8b70 con 0x55a829741050 2024-03-23T23:12:30.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.000+0000 7f93d3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829741a40 0x55a8297525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.000+0000 7f93d3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829741a40 0x55a8297525a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37890/0 (socket says 172.21.15.12:37890) 2024-03-23T23:12:30.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.000+0000 7f93d3e00640 1 -- 172.21.15.12:0/1059394387 learned_addr learned my addr 172.21.15.12:0/1059394387 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:30.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.000+0000 7f93db400640 1 --2- 172.21.15.12:0/1059394387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a829741050 0x55a829740310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.000+0000 7f93db400640 1 --2- 172.21.15.12:0/1059394387 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a829746bb0 0x55a82973b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.000+0000 7f93d3e00640 1 -- 172.21.15.12:0/1059394387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a829741050 msgr2=0x55a829740310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:30.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.000+0000 7f93d3e00640 1 --2- 172.21.15.12:0/1059394387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a829741050 0x55a829740310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.000+0000 7f93d3e00640 1 -- 172.21.15.12:0/1059394387 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a829746bb0 msgr2=0x55a82973b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:30.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.000+0000 7f93d3e00640 1 --2- 172.21.15.12:0/1059394387 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a829746bb0 0x55a82973b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.000+0000 7f93d3e00640 1 -- 172.21.15.12:0/1059394387 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a82937a0f0 con 0x55a829741a40 2024-03-23T23:12:30.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.000+0000 7f93d3e00640 1 --2- 172.21.15.12:0/1059394387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829741a40 0x55a8297525a0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x55a8297598e0 tx=0x7f93c4005900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73dc9c09619fb656 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:30.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.000+0000 7f93caa00640 1 -- 172.21.15.12:0/1059394387 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c4024070 con 0x55a829741a40 2024-03-23T23:12:30.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.001+0000 7f93caa00640 1 -- 172.21.15.12:0/1059394387 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93c40035f0 con 0x55a829741a40 2024-03-23T23:12:30.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.001+0000 7f93caa00640 1 -- 172.21.15.12:0/1059394387 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c40147f0 con 0x55a829741a40 2024-03-23T23:12:30.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.001+0000 7f93e43d2740 1 -- 172.21.15.12:0/1059394387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829741a40 msgr2=0x55a8297525a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:30.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.001+0000 7f93e43d2740 1 --2- 172.21.15.12:0/1059394387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829741a40 0x55a8297525a0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x55a8297598e0 tx=0x7f93c4005900 comp rx=0 tx=0).stop 2024-03-23T23:12:30.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.001+0000 7f93e43d2740 1 -- 172.21.15.12:0/1059394387 shutdown_connections 2024-03-23T23:12:30.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.001+0000 7f93e43d2740 1 --2- 172.21.15.12:0/1059394387 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a829746bb0 0x55a82973b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.001+0000 7f93e43d2740 1 --2- 172.21.15.12:0/1059394387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829741a40 0x55a8297525a0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.001+0000 7f93e43d2740 1 --2- 172.21.15.12:0/1059394387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a829741050 0x55a829740310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.001+0000 7f93e43d2740 1 -- 172.21.15.12:0/1059394387 >> 172.21.15.12:0/1059394387 conn(0x55a829709200 msgr2=0x55a82973fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:30.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.001+0000 7f93e43d2740 1 -- 172.21.15.12:0/1059394387 shutdown_connections 2024-03-23T23:12:30.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.001+0000 7f93e43d2740 1 -- 172.21.15.12:0/1059394387 wait complete. 2024-03-23T23:12:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.002+0000 7f93e43d2740 1 -- start start 2024-03-23T23:12:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.002+0000 7f93e43d2740 1 Processor -- start 2024-03-23T23:12:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.002+0000 7f93e43d2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829741050 0x55a8297503a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.003+0000 7f93e43d2740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a829741a40 0x55a8297508e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.003+0000 7f93e43d2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a829746bb0 0x55a8297518a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.003+0000 7f93e43d2740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc1a9170 con 0x55a829741a40 2024-03-23T23:12:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.003+0000 7f93e43d2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc1aa470 con 0x55a829741050 2024-03-23T23:12:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.003+0000 7f93e43d2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc1a97b0 con 0x55a829746bb0 2024-03-23T23:12:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.003+0000 7f93e43d2740 10 client.0 _ll_register_callbacks cb 0x7f93d4009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:12:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.003+0000 7f93db400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829741050 0x55a8297503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.003+0000 7f93db400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829741050 0x55a8297503a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37902/0 (socket says 172.21.15.12:37902) 2024-03-23T23:12:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.003+0000 7f93db400640 1 -- 172.21.15.12:0/3408298763 learned_addr learned my addr 172.21.15.12:0/3408298763 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.003+0000 7f93db400640 1 --2- 172.21.15.12:0/3408298763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a829746bb0 0x55a8297518a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.003+0000 7f93d3e00640 1 --2- 172.21.15.12:0/3408298763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a829741a40 0x55a8297508e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.003+0000 7f93db400640 1 -- 172.21.15.12:0/3408298763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a829746bb0 msgr2=0x55a8297518a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.003+0000 7f93db400640 1 --2- 172.21.15.12:0/3408298763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a829746bb0 0x55a8297518a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.003+0000 7f93db400640 1 -- 172.21.15.12:0/3408298763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a829741a40 msgr2=0x55a8297508e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.003+0000 7f93db400640 1 --2- 172.21.15.12:0/3408298763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a829741a40 0x55a8297508e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.003+0000 7f93db400640 1 -- 172.21.15.12:0/3408298763 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a829751e90 con 0x55a829741050 2024-03-23T23:12:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.004+0000 7f93db400640 1 --2- 172.21.15.12:0/3408298763 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829741050 0x55a8297503a0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f93d4009880 tx=0x7f93d4010fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0aa37bbcbbc64d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.004+0000 7f93bfe00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:12:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.004+0000 7f93bfe00640 1 -- 172.21.15.12:0/3408298763 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d4016040 con 0x55a829741050 2024-03-23T23:12:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.004+0000 7f93e43d2740 1 -- 172.21.15.12:0/3408298763 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f93c40035f0 con 0x55a829741050 2024-03-23T23:12:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.004+0000 7f93bfe00640 1 -- 172.21.15.12:0/3408298763 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f93d4002030 con 0x55a829741050 2024-03-23T23:12:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.004+0000 7f93bfe00640 1 -- 172.21.15.12:0/3408298763 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93d4047050 con 0x55a829741050 2024-03-23T23:12:30.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.004+0000 7f93e43d2740 10 client.28280 fetch_fsmap learned FSMap version 23 2024-03-23T23:12:30.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.004+0000 7f93e43d2740 1 -- 172.21.15.12:0/3408298763 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a8297b2bd0 con 0x55a829741050 2024-03-23T23:12:30.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.004+0000 7f93bfe00640 1 -- 172.21.15.12:0/3408298763 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93c40035f0 con 0x55a829741050 2024-03-23T23:12:30.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.005+0000 7f93bfe00640 1 -- 172.21.15.12:0/3408298763 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f93d400f040 con 0x55a829741050 2024-03-23T23:12:30.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.005+0000 7f93e43d2740 10 client.28280 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:12:30.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.005+0000 7f93e43d2740 10 client.28280 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:12:30.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.005+0000 7f93e43d2740 20 client.28280 populate_metadata read hostname 'smithi012' 2024-03-23T23:12:30.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.005+0000 7f93e43d2740 1 --2- 172.21.15.12:0/3408298763 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a8297b3340 0x55a8297b5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.005+0000 7f93e43d2740 4 client.28280 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:12:30.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.005+0000 7f93e43d2740 1 -- 172.21.15.12:0/3408298763 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a8296adae0 con 0x55a8297b3340 2024-03-23T23:12:30.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.005+0000 7f93d3e00640 1 --2- 172.21.15.12:0/3408298763 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a8297b3340 0x55a8297b5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.005+0000 7f93d3e00640 1 --2- 172.21.15.12:0/3408298763 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a8297b3340 0x55a8297b5760 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72e297dfa9390331 server_cookie=3fb17f3fbd6d4d22 in_seq=0 out_seq=0 2024-03-23T23:12:30.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.005+0000 7f93bfe00640 10 client.28280 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:30.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.006+0000 7f93bfe00640 1 -- 172.21.15.12:0/3408298763 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55a8296adae0 con 0x55a8297b3340 2024-03-23T23:12:30.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.006+0000 7f93bfe00640 10 client.28280 handle_command_reply: tid=0 2024-03-23T23:12:30.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.006+0000 7f93e43d2740 1 client.28280 shutdown 2024-03-23T23:12:30.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.006+0000 7f93e43d2740 1 -- 172.21.15.12:0/3408298763 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829741050 msgr2=0x55a8297503a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:30.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.006+0000 7f93e43d2740 1 --2- 172.21.15.12:0/3408298763 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829741050 0x55a8297503a0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f93d4009880 tx=0x7f93d4010fa0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.006+0000 7f93e43d2740 1 -- 172.21.15.12:0/3408298763 shutdown_connections 2024-03-23T23:12:30.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.006+0000 7f93e43d2740 1 --2- 172.21.15.12:0/3408298763 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55a8297b3340 0x55a8297b5760 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.006+0000 7f93e43d2740 1 --2- 172.21.15.12:0/3408298763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a829746bb0 0x55a8297518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.006+0000 7f93e43d2740 1 --2- 172.21.15.12:0/3408298763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a829741a40 0x55a8297508e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.006+0000 7f93bfe00640 0 client.28280 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:30.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.006+0000 7f93e43d2740 1 --2- 172.21.15.12:0/3408298763 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a829741050 0x55a8297503a0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.006+0000 7f93e43d2740 1 -- 172.21.15.12:0/3408298763 >> 172.21.15.12:0/3408298763 conn(0x55a829709200 msgr2=0x55a829740490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:30.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.006+0000 7f93e43d2740 1 -- 172.21.15.12:0/3408298763 shutdown_connections 2024-03-23T23:12:30.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.006+0000 7f93e43d2740 1 -- 172.21.15.12:0/3408298763 wait complete. 2024-03-23T23:12:30.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.006+0000 7f93e43d2740 20 client.28280 trim_cache size 0 max 16384 2024-03-23T23:12:30.010 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:12:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-03-23T23:12:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-03-23T23:12:30.010 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:12:30.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.007+0000 7f93dbe00640 1 -- 172.21.15.12:0/2469041037 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f93c00a2800 msgr2=0x7f93c00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:30.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.007+0000 7f93dbe00640 1 --2- 172.21.15.12:0/2469041037 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f93c00a2800 0x7f93c00a4cc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f93cc0bd520 tx=0x7f93cc02bc30 comp rx=0 tx=0).stop 2024-03-23T23:12:30.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.007+0000 7f93dbe00640 1 -- 172.21.15.12:0/2469041037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93dc1037c0 msgr2=0x7f93dc199b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:30.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.007+0000 7f93dbe00640 1 --2- 172.21.15.12:0/2469041037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93dc1037c0 0x7f93dc199b00 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f93cc000c00 tx=0x7f93cc02c6c0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.007+0000 7f93dbe00640 1 -- 172.21.15.12:0/2469041037 shutdown_connections 2024-03-23T23:12:30.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.007+0000 7f93dbe00640 1 --2- 172.21.15.12:0/2469041037 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f93c00a2800 0x7f93c00a4cc0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.007+0000 7f93dbe00640 1 --2- 172.21.15.12:0/2469041037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93dc105370 0x7f93dc1a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.007+0000 7f93dbe00640 1 --2- 172.21.15.12:0/2469041037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93dc1049b0 0x7f93dc19a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.007+0000 7f93dbe00640 1 --2- 172.21.15.12:0/2469041037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93dc1037c0 0x7f93dc199b00 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.007+0000 7f93dbe00640 1 -- 172.21.15.12:0/2469041037 >> 172.21.15.12:0/2469041037 conn(0x7f93dc0fed50 msgr2=0x7f93dc107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:30.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.007+0000 7f93dbe00640 1 -- 172.21.15.12:0/2469041037 shutdown_connections 2024-03-23T23:12:30.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.007+0000 7f93dbe00640 1 -- 172.21.15.12:0/2469041037 wait complete. 2024-03-23T23:12:30.025 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T23:12:30.025 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:12:30.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.242+0000 7f8f22c00640 1 -- 172.21.15.12:0/2757939556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f1c108ed0 msgr2=0x7f8f1c109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:30.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.242+0000 7f8f22c00640 1 --2- 172.21.15.12:0/2757939556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f1c108ed0 0x7f8f1c109350 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f8f04009900 tx=0x7f8f0402bc30 comp rx=0 tx=0).stop 2024-03-23T23:12:30.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.242+0000 7f8f22c00640 1 -- 172.21.15.12:0/2757939556 shutdown_connections 2024-03-23T23:12:30.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.242+0000 7f8f22c00640 1 --2- 172.21.15.12:0/2757939556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f1c109890 0x7f8f1c110120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.242+0000 7f8f22c00640 1 --2- 172.21.15.12:0/2757939556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f1c108ed0 0x7f8f1c109350 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.242+0000 7f8f22c00640 1 --2- 172.21.15.12:0/2757939556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f1c107ce0 0x7f8f1c108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.242+0000 7f8f22c00640 1 -- 172.21.15.12:0/2757939556 >> 172.21.15.12:0/2757939556 conn(0x7f8f1c076cb0 msgr2=0x7f8f1c0790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:30.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.243+0000 7f8f22c00640 1 -- 172.21.15.12:0/2757939556 shutdown_connections 2024-03-23T23:12:30.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.243+0000 7f8f22c00640 1 -- 172.21.15.12:0/2757939556 wait complete. 2024-03-23T23:12:30.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.244+0000 7f8f22c00640 1 Processor -- start 2024-03-23T23:12:30.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.244+0000 7f8f22c00640 1 -- start start 2024-03-23T23:12:30.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.245+0000 7f8f22c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f1c107ce0 0x7f8f1c19dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.245+0000 7f8f22c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f1c108ed0 0x7f8f1c19e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.245+0000 7f8f22c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f1c109890 0x7f8f1c1a55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.245+0000 7f8f22c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c1139b0 con 0x7f8f1c108ed0 2024-03-23T23:12:30.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.245+0000 7f8f22c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c113830 con 0x7f8f1c109890 2024-03-23T23:12:30.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.245+0000 7f8f22c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c113b30 con 0x7f8f1c107ce0 2024-03-23T23:12:30.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.245+0000 7f8f1be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f1c107ce0 0x7f8f1c19dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.245+0000 7f8f1be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f1c107ce0 0x7f8f1c19dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:57930/0 (socket says 172.21.15.12:57930) 2024-03-23T23:12:30.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.245+0000 7f8f1be00640 1 -- 172.21.15.12:0/3208659140 learned_addr learned my addr 172.21.15.12:0/3208659140 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:30.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.245+0000 7f8f13400640 1 --2- 172.21.15.12:0/3208659140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f1c108ed0 0x7f8f1c19e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.245+0000 7f8f20c00640 1 --2- 172.21.15.12:0/3208659140 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f1c109890 0x7f8f1c1a55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.245+0000 7f8f1be00640 1 -- 172.21.15.12:0/3208659140 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f1c109890 msgr2=0x7f8f1c1a55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:30.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.246+0000 7f8f1be00640 1 --2- 172.21.15.12:0/3208659140 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f1c109890 0x7f8f1c1a55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.246+0000 7f8f1be00640 1 -- 172.21.15.12:0/3208659140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f1c108ed0 msgr2=0x7f8f1c19e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:30.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.246+0000 7f8f1be00640 1 --2- 172.21.15.12:0/3208659140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f1c108ed0 0x7f8f1c19e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.246+0000 7f8f1be00640 1 -- 172.21.15.12:0/3208659140 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1c1a5c10 con 0x7f8f1c107ce0 2024-03-23T23:12:30.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.246+0000 7f8f13400640 1 --2- 172.21.15.12:0/3208659140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f1c108ed0 0x7f8f1c19e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:12:30.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.246+0000 7f8f20c00640 1 --2- 172.21.15.12:0/3208659140 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f1c109890 0x7f8f1c1a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:12:30.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.246+0000 7f8f1be00640 1 --2- 172.21.15.12:0/3208659140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f1c107ce0 0x7f8f1c19dff0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c00eb70 tx=0x7f8f0c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:30.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.247+0000 7f8f19600640 1 -- 172.21.15.12:0/3208659140 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c003b70 con 0x7f8f1c107ce0 2024-03-23T23:12:30.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.247+0000 7f8f19600640 1 -- 172.21.15.12:0/3208659140 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f0c00ee70 con 0x7f8f1c107ce0 2024-03-23T23:12:30.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.247+0000 7f8f19600640 1 -- 172.21.15.12:0/3208659140 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c010830 con 0x7f8f1c107ce0 2024-03-23T23:12:30.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.247+0000 7f8f22c00640 1 -- 172.21.15.12:0/3208659140 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f1c1a5f00 con 0x7f8f1c107ce0 2024-03-23T23:12:30.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.247+0000 7f8f22c00640 1 -- 172.21.15.12:0/3208659140 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f1c1a6310 con 0x7f8f1c107ce0 2024-03-23T23:12:30.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.249+0000 7f8f22c00640 1 -- 172.21.15.12:0/3208659140 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f1c10f000 con 0x7f8f1c107ce0 2024-03-23T23:12:30.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.251+0000 7f8f19600640 1 -- 172.21.15.12:0/3208659140 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8f0c004a40 con 0x7f8f1c107ce0 2024-03-23T23:12:30.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.252+0000 7f8f19600640 1 --2- 172.21.15.12:0/3208659140 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8f000a2880 0x7f8f000a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.252+0000 7f8f19600640 1 -- 172.21.15.12:0/3208659140 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8f0c0cf1c0 con 0x7f8f1c107ce0 2024-03-23T23:12:30.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.253+0000 7f8f13400640 1 --2- 172.21.15.12:0/3208659140 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8f000a2880 0x7f8f000a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.253+0000 7f8f13400640 1 --2- 172.21.15.12:0/3208659140 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8f000a2880 0x7f8f000a4d40 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f8f0402c240 tx=0x7f8f0402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:30.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.254+0000 7f8f19600640 1 -- 172.21.15.12:0/3208659140 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8f0c09c210 con 0x7f8f1c107ce0 2024-03-23T23:12:30.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.501+0000 7f8f22c00640 1 -- 172.21.15.12:0/3208659140 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f1c02d070 con 0x7f8f1c107ce0 2024-03-23T23:12:30.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.502+0000 7f8f19600640 1 -- 172.21.15.12:0/3208659140 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f8f0c09bb10 con 0x7f8f1c107ce0 2024-03-23T23:12:30.502 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:12:30.502 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:12:30.502 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:12:30.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.504+0000 7f8f22c00640 1 -- 172.21.15.12:0/3208659140 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8f000a2880 msgr2=0x7f8f000a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:30.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.504+0000 7f8f22c00640 1 --2- 172.21.15.12:0/3208659140 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8f000a2880 0x7f8f000a4d40 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f8f0402c240 tx=0x7f8f0402df90 comp rx=0 tx=0).stop 2024-03-23T23:12:30.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.504+0000 7f8f22c00640 1 -- 172.21.15.12:0/3208659140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f1c107ce0 msgr2=0x7f8f1c19dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:30.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.504+0000 7f8f22c00640 1 --2- 172.21.15.12:0/3208659140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f1c107ce0 0x7f8f1c19dff0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c00eb70 tx=0x7f8f0c00c6a0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.505+0000 7f8f22c00640 1 -- 172.21.15.12:0/3208659140 shutdown_connections 2024-03-23T23:12:30.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.505+0000 7f8f22c00640 1 --2- 172.21.15.12:0/3208659140 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f8f000a2880 0x7f8f000a4d40 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.505+0000 7f8f22c00640 1 --2- 172.21.15.12:0/3208659140 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f1c109890 0x7f8f1c1a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.505+0000 7f8f22c00640 1 --2- 172.21.15.12:0/3208659140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f1c108ed0 0x7f8f1c19e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.505+0000 7f8f22c00640 1 --2- 172.21.15.12:0/3208659140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f1c107ce0 0x7f8f1c19dff0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.505+0000 7f8f22c00640 1 -- 172.21.15.12:0/3208659140 >> 172.21.15.12:0/3208659140 conn(0x7f8f1c076cb0 msgr2=0x7f8f1c10c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:30.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.505+0000 7f8f22c00640 1 -- 172.21.15.12:0/3208659140 shutdown_connections 2024-03-23T23:12:30.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.505+0000 7f8f22c00640 1 -- 172.21.15.12:0/3208659140 wait complete. 2024-03-23T23:12:30.525 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:12:30.526 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-23T23:12:30.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.748+0000 7f0910600640 1 -- 172.21.15.12:0/1860680532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f090810dca0 msgr2=0x7f0908110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:30.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.748+0000 7f0910600640 1 --2- 172.21.15.12:0/1860680532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f090810dca0 0x7f0908110090 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0900009cd0 tx=0x7f090002c0e0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.748+0000 7f0910600640 1 -- 172.21.15.12:0/1860680532 shutdown_connections 2024-03-23T23:12:30.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.748+0000 7f0910600640 1 --2- 172.21.15.12:0/1860680532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f090810dca0 0x7f0908110090 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.748+0000 7f0910600640 1 --2- 172.21.15.12:0/1860680532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0908074100 0x7f0908074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.748+0000 7f0910600640 1 --2- 172.21.15.12:0/1860680532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09080756e0 0x7f0908073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.748+0000 7f0910600640 1 -- 172.21.15.12:0/1860680532 >> 172.21.15.12:0/1860680532 conn(0x7f09080fcc40 msgr2=0x7f09080ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:30.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.748+0000 7f0910600640 1 -- 172.21.15.12:0/1860680532 shutdown_connections 2024-03-23T23:12:30.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.748+0000 7f0910600640 1 -- 172.21.15.12:0/1860680532 wait complete. 2024-03-23T23:12:30.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.749+0000 7f0910600640 1 Processor -- start 2024-03-23T23:12:30.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.749+0000 7f0910600640 1 -- start start 2024-03-23T23:12:30.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.749+0000 7f0910600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0908074100 0x7f090819df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.749+0000 7f0910600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09080756e0 0x7f090819e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.749+0000 7f0910600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f090810dca0 0x7f09081a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.749+0000 7f0910600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0908113920 con 0x7f090810dca0 2024-03-23T23:12:30.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.749+0000 7f0910600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09081137a0 con 0x7f09080756e0 2024-03-23T23:12:30.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.749+0000 7f0910600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0908113aa0 con 0x7f0908074100 2024-03-23T23:12:30.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.750+0000 7f090d200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09080756e0 0x7f090819e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.750+0000 7f090e600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f090810dca0 0x7f09081a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.750+0000 7f090dc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0908074100 0x7f090819df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.750+0000 7f090d200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09080756e0 0x7f090819e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37928/0 (socket says 172.21.15.12:37928) 2024-03-23T23:12:30.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.750+0000 7f090d200640 1 -- 172.21.15.12:0/1276998102 learned_addr learned my addr 172.21.15.12:0/1276998102 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:30.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.750+0000 7f090dc00640 1 -- 172.21.15.12:0/1276998102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09080756e0 msgr2=0x7f090819e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:30.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.750+0000 7f090dc00640 1 --2- 172.21.15.12:0/1276998102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09080756e0 0x7f090819e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.750+0000 7f090dc00640 1 -- 172.21.15.12:0/1276998102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f090810dca0 msgr2=0x7f09081a5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:30.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.750+0000 7f090dc00640 1 --2- 172.21.15.12:0/1276998102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f090810dca0 0x7f09081a5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.750+0000 7f090dc00640 1 -- 172.21.15.12:0/1276998102 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09081a5bb0 con 0x7f0908074100 2024-03-23T23:12:30.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.750+0000 7f090d200640 1 --2- 172.21.15.12:0/1276998102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09080756e0 0x7f090819e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:12:30.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.750+0000 7f090e600640 1 --2- 172.21.15.12:0/1276998102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f090810dca0 0x7f09081a5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:12:30.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.751+0000 7f090dc00640 1 --2- 172.21.15.12:0/1276998102 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0908074100 0x7f090819df90 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f08fc00e9f0 tx=0x7f08fc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:30.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.751+0000 7f0906000640 1 -- 172.21.15.12:0/1276998102 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08fc00ccc0 con 0x7f0908074100 2024-03-23T23:12:30.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.751+0000 7f0906000640 1 -- 172.21.15.12:0/1276998102 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f08fc010910 con 0x7f0908074100 2024-03-23T23:12:30.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.751+0000 7f0910600640 1 -- 172.21.15.12:0/1276998102 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09081a5ea0 con 0x7f0908074100 2024-03-23T23:12:30.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.751+0000 7f0906000640 1 -- 172.21.15.12:0/1276998102 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08fc018960 con 0x7f0908074100 2024-03-23T23:12:30.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.751+0000 7f0910600640 1 -- 172.21.15.12:0/1276998102 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09081a6300 con 0x7f0908074100 2024-03-23T23:12:30.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.754+0000 7f09133dc740 1 Processor -- start 2024-03-23T23:12:30.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.754+0000 7f0906000640 1 -- 172.21.15.12:0/1276998102 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f08fc010ac0 con 0x7f0908074100 2024-03-23T23:12:30.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.754+0000 7f09133dc740 1 -- start start 2024-03-23T23:12:30.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.755+0000 7f09133dc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec6810 0x558f1bec6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.755+0000 7f09133dc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f1bec79e0 0x558f1bec7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.755+0000 7f09133dc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f1bec83a0 0x558f1becec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.755+0000 7f09133dc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f1be2b2c0 con 0x558f1bec83a0 2024-03-23T23:12:30.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.755+0000 7f09133dc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f1bdd3c90 con 0x558f1bec6810 2024-03-23T23:12:30.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.755+0000 7f09133dc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f1be27360 con 0x558f1bec79e0 2024-03-23T23:12:30.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.755+0000 7f090e600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec6810 0x558f1bec6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.755+0000 7f090d200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f1bec83a0 0x558f1becec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.755+0000 7f090e600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec6810 0x558f1bec6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37938/0 (socket says 172.21.15.12:37938) 2024-03-23T23:12:30.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.755+0000 7f090dc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f1bec79e0 0x558f1bec7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.755+0000 7f090e600640 1 -- 172.21.15.12:0/3769887557 learned_addr learned my addr 172.21.15.12:0/3769887557 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:30.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.755+0000 7f090d200640 1 -- 172.21.15.12:0/3769887557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f1bec79e0 msgr2=0x558f1bec7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:30.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.755+0000 7f090d200640 1 --2- 172.21.15.12:0/3769887557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f1bec79e0 0x558f1bec7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.755+0000 7f090d200640 1 -- 172.21.15.12:0/3769887557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec6810 msgr2=0x558f1bec6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:30.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.755+0000 7f090d200640 1 --2- 172.21.15.12:0/3769887557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec6810 0x558f1bec6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.755+0000 7f090d200640 1 -- 172.21.15.12:0/3769887557 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f1bdf8080 con 0x558f1bec83a0 2024-03-23T23:12:30.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.755+0000 7f090e600640 1 --2- 172.21.15.12:0/3769887557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec6810 0x558f1bec6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:12:30.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.756+0000 7f090d200640 1 --2- 172.21.15.12:0/3769887557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f1bec83a0 0x558f1becec30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f08f40114c0 tx=0x7f08f4011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e957ab415019fc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:30.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.756+0000 7f0906000640 1 --2- 172.21.15.12:0/1276998102 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f08f00a2800 0x7f08f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.756+0000 7f08faa00640 1 -- 172.21.15.12:0/3769887557 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f4004030 con 0x558f1bec83a0 2024-03-23T23:12:30.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.756+0000 7f08faa00640 1 -- 172.21.15.12:0/3769887557 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f08f40041d0 con 0x558f1bec83a0 2024-03-23T23:12:30.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.756+0000 7f08faa00640 1 -- 172.21.15.12:0/3769887557 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f400f8f0 con 0x558f1bec83a0 2024-03-23T23:12:30.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.756+0000 7f0906000640 1 -- 172.21.15.12:0/1276998102 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f08fc0cee60 con 0x7f0908074100 2024-03-23T23:12:30.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.756+0000 7f090e600640 1 --2- 172.21.15.12:0/1276998102 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f08f00a2800 0x7f08f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.757+0000 7f09133dc740 1 -- 172.21.15.12:0/3769887557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f1bec83a0 msgr2=0x558f1becec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:30.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.757+0000 7f09133dc740 1 --2- 172.21.15.12:0/3769887557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f1bec83a0 0x558f1becec30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f08f40114c0 tx=0x7f08f4011980 comp rx=0 tx=0).stop 2024-03-23T23:12:30.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.757+0000 7f09133dc740 1 -- 172.21.15.12:0/3769887557 shutdown_connections 2024-03-23T23:12:30.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.757+0000 7f09133dc740 1 --2- 172.21.15.12:0/3769887557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f1bec83a0 0x558f1becec30 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.757+0000 7f09133dc740 1 --2- 172.21.15.12:0/3769887557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f1bec79e0 0x558f1bec7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.757+0000 7f09133dc740 1 --2- 172.21.15.12:0/3769887557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec6810 0x558f1bec6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.757+0000 7f09133dc740 1 -- 172.21.15.12:0/3769887557 >> 172.21.15.12:0/3769887557 conn(0x558f1bec1c00 msgr2=0x558f1bec2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:30.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.757+0000 7f09133dc740 1 -- 172.21.15.12:0/3769887557 shutdown_connections 2024-03-23T23:12:30.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.757+0000 7f09133dc740 1 -- 172.21.15.12:0/3769887557 wait complete. 2024-03-23T23:12:30.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.757+0000 7f090e600640 1 --2- 172.21.15.12:0/1276998102 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f08f00a2800 0x7f08f00a4cc0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f0900009430 tx=0x7f0900009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:30.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.758+0000 7f09133dc740 1 -- start start 2024-03-23T23:12:30.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.758+0000 7f09133dc740 1 Processor -- start 2024-03-23T23:12:30.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.758+0000 7f09133dc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f1bec6810 0x558f1bed09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.758+0000 7f09133dc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec79e0 0x558f1bed0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.758+0000 7f09133dc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f1bec83a0 0x558f1bed7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.758+0000 7f09133dc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f1becf3b0 con 0x558f1bec83a0 2024-03-23T23:12:30.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.758+0000 7f09133dc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f1bde70d0 con 0x558f1bec79e0 2024-03-23T23:12:30.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.758+0000 7f09133dc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f1becf530 con 0x558f1bec6810 2024-03-23T23:12:30.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.758+0000 7f09133dc740 10 client.0 _ll_register_callbacks cb 0x558f1bdebe80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:12:30.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.759+0000 7f090d200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec79e0 0x558f1bed0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.759+0000 7f090dc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f1bec6810 0x558f1bed09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.759+0000 7f090d200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec79e0 0x558f1bed0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37952/0 (socket says 172.21.15.12:37952) 2024-03-23T23:12:30.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.759+0000 7f090d200640 1 -- 172.21.15.12:0/4035585047 learned_addr learned my addr 172.21.15.12:0/4035585047 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:30.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.759+0000 7f090e600640 1 --2- 172.21.15.12:0/4035585047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f1bec83a0 0x558f1bed7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.759+0000 7f090d200640 1 -- 172.21.15.12:0/4035585047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f1bec6810 msgr2=0x558f1bed09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:30.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.759+0000 7f090d200640 1 --2- 172.21.15.12:0/4035585047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f1bec6810 0x558f1bed09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.759+0000 7f090d200640 1 -- 172.21.15.12:0/4035585047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f1bec83a0 msgr2=0x558f1bed7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:30.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.759+0000 7f090d200640 1 --2- 172.21.15.12:0/4035585047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f1bec83a0 0x558f1bed7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.759+0000 7f090d200640 1 -- 172.21.15.12:0/4035585047 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558f1bb3f3f0 con 0x558f1bec79e0 2024-03-23T23:12:30.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.759+0000 7f090e600640 1 --2- 172.21.15.12:0/4035585047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f1bec83a0 0x558f1bed7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:12:30.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.759+0000 7f090dc00640 1 --2- 172.21.15.12:0/4035585047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f1bec6810 0x558f1bed09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:12:30.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.759+0000 7f090d200640 1 --2- 172.21.15.12:0/4035585047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec79e0 0x558f1bed0f30 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f08f40097c0 tx=0x7f08f4005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef42204d7fa8c675 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:30.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.759+0000 7f08ef400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:12:30.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.760+0000 7f08ef400640 1 -- 172.21.15.12:0/4035585047 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f4027070 con 0x558f1bec79e0 2024-03-23T23:12:30.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.760+0000 7f08ef400640 1 -- 172.21.15.12:0/4035585047 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f08f40035f0 con 0x558f1bec79e0 2024-03-23T23:12:30.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.760+0000 7f09133dc740 1 -- 172.21.15.12:0/4035585047 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f08f40041d0 con 0x558f1bec79e0 2024-03-23T23:12:30.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.760+0000 7f08ef400640 1 -- 172.21.15.12:0/4035585047 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f08f4033050 con 0x558f1bec79e0 2024-03-23T23:12:30.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.760+0000 7f09133dc740 10 client.28298 fetch_fsmap learned FSMap version 23 2024-03-23T23:12:30.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.760+0000 7f09133dc740 1 -- 172.21.15.12:0/4035585047 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x558f1bc097d0 con 0x558f1bec79e0 2024-03-23T23:12:30.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.760+0000 7f08ef400640 1 -- 172.21.15.12:0/4035585047 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f08f40041d0 con 0x558f1bec79e0 2024-03-23T23:12:30.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.760+0000 7f08ef400640 1 -- 172.21.15.12:0/4035585047 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f08f400f630 con 0x558f1bec79e0 2024-03-23T23:12:30.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.760+0000 7f09133dc740 10 client.28298 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:12:30.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.760+0000 7f09133dc740 10 client.28298 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:12:30.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.761+0000 7f09133dc740 20 client.28298 populate_metadata read hostname 'smithi012' 2024-03-23T23:12:30.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.761+0000 7f09133dc740 1 --2- 172.21.15.12:0/4035585047 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x558f1bedc1f0 0x558f1bede5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.761+0000 7f09133dc740 4 client.28298 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:12:30.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.761+0000 7f09133dc740 1 -- 172.21.15.12:0/4035585047 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558f1be38800 con 0x558f1bedc1f0 2024-03-23T23:12:30.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.761+0000 7f090dc00640 1 --2- 172.21.15.12:0/4035585047 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x558f1bedc1f0 0x558f1bede5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.761+0000 7f090dc00640 1 --2- 172.21.15.12:0/4035585047 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x558f1bedc1f0 0x558f1bede5e0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=24e3b55a2fa02327 server_cookie=2bb6d3a33128538 in_seq=0 out_seq=0 2024-03-23T23:12:30.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.761+0000 7f08ef400640 10 client.28298 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:12:30.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.762+0000 7f08ef400640 1 -- 172.21.15.12:0/4035585047 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558f1be38800 con 0x558f1bedc1f0 2024-03-23T23:12:30.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.762+0000 7f08ef400640 10 client.28298 handle_command_reply: tid=0 2024-03-23T23:12:30.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.762+0000 7f09133dc740 1 client.28298 shutdown 2024-03-23T23:12:30.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.763+0000 7f09133dc740 1 -- 172.21.15.12:0/4035585047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec79e0 msgr2=0x558f1bed0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:30.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.763+0000 7f09133dc740 1 --2- 172.21.15.12:0/4035585047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec79e0 0x558f1bed0f30 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f08f40097c0 tx=0x7f08f4005740 comp rx=0 tx=0).stop 2024-03-23T23:12:30.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.763+0000 7f09133dc740 1 -- 172.21.15.12:0/4035585047 shutdown_connections 2024-03-23T23:12:30.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.763+0000 7f09133dc740 1 --2- 172.21.15.12:0/4035585047 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x558f1bedc1f0 0x558f1bede5e0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.763+0000 7f09133dc740 1 --2- 172.21.15.12:0/4035585047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f1bec83a0 0x558f1bed7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.763+0000 7f09133dc740 1 --2- 172.21.15.12:0/4035585047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec79e0 0x558f1bed0f30 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.763+0000 7f09133dc740 1 --2- 172.21.15.12:0/4035585047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f1bec6810 0x558f1bed09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.763+0000 7f08ef400640 0 client.28298 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:12:30.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.763+0000 7f09133dc740 1 -- 172.21.15.12:0/4035585047 >> 172.21.15.12:0/4035585047 conn(0x558f1bec1c00 msgr2=0x558f1bec0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:30.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.763+0000 7f09133dc740 1 -- 172.21.15.12:0/4035585047 shutdown_connections 2024-03-23T23:12:30.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.763+0000 7f09133dc740 1 -- 172.21.15.12:0/4035585047 wait complete. 2024-03-23T23:12:30.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.763+0000 7f09133dc740 20 client.28298 trim_cache size 0 max 16384 2024-03-23T23:12:30.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.777+0000 7f09133dc740 1 Processor -- start 2024-03-23T23:12:30.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.777+0000 7f09133dc740 1 -- start start 2024-03-23T23:12:30.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.777+0000 7f09133dc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f1bec6810 0x558f1bed0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.777+0000 7f09133dc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f1bec79e0 0x558f1bed7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.777+0000 7f09133dc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec83a0 0x558f1becac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.777+0000 7f09133dc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09081b15a0 con 0x558f1bec6810 2024-03-23T23:12:30.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.777+0000 7f09133dc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09081af0a0 con 0x558f1bec83a0 2024-03-23T23:12:30.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.777+0000 7f09133dc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09081ad140 con 0x558f1bec79e0 2024-03-23T23:12:30.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.777+0000 7f090d200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f1bec79e0 0x558f1bed7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.777+0000 7f090e600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec83a0 0x558f1becac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.778+0000 7f090d200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f1bec79e0 0x558f1bed7f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:57952/0 (socket says 172.21.15.12:57952) 2024-03-23T23:12:30.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.778+0000 7f090e600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec83a0 0x558f1becac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37964/0 (socket says 172.21.15.12:37964) 2024-03-23T23:12:30.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.778+0000 7f090dc00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f1bec6810 0x558f1bed0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.778+0000 7f090d200640 1 -- 172.21.15.12:0/2946851721 learned_addr learned my addr 172.21.15.12:0/2946851721 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:30.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.778+0000 7f090d200640 1 -- 172.21.15.12:0/2946851721 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec83a0 msgr2=0x558f1becac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:30.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.778+0000 7f090d200640 1 --2- 172.21.15.12:0/2946851721 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec83a0 0x558f1becac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.778+0000 7f090d200640 1 -- 172.21.15.12:0/2946851721 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f1bec6810 msgr2=0x558f1bed0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:30.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.778+0000 7f090d200640 1 --2- 172.21.15.12:0/2946851721 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f1bec6810 0x558f1bed0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.778+0000 7f090d200640 1 -- 172.21.15.12:0/2946851721 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f1becf290 con 0x558f1bec79e0 2024-03-23T23:12:30.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.778+0000 7f090d200640 1 --2- 172.21.15.12:0/2946851721 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f1bec79e0 0x558f1bed7f30 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f08f40119c0 tx=0x7f08f4011ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58535f6431ec26c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:30.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.778+0000 7f08fa000640 1 -- 172.21.15.12:0/2946851721 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f4027040 con 0x558f1bec79e0 2024-03-23T23:12:30.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.779+0000 7f08fa000640 1 -- 172.21.15.12:0/2946851721 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f08f400f7c0 con 0x558f1bec79e0 2024-03-23T23:12:30.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.779+0000 7f08fa000640 1 -- 172.21.15.12:0/2946851721 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f40196b0 con 0x558f1bec79e0 2024-03-23T23:12:30.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.779+0000 7f09133dc740 1 -- 172.21.15.12:0/2946851721 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f1bec79e0 msgr2=0x558f1bed7f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:30.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.779+0000 7f09133dc740 1 --2- 172.21.15.12:0/2946851721 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f1bec79e0 0x558f1bed7f30 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f08f40119c0 tx=0x7f08f4011ea0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.779+0000 7f09133dc740 1 -- 172.21.15.12:0/2946851721 shutdown_connections 2024-03-23T23:12:30.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.779+0000 7f09133dc740 1 --2- 172.21.15.12:0/2946851721 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec83a0 0x558f1becac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.779+0000 7f09133dc740 1 --2- 172.21.15.12:0/2946851721 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f1bec79e0 0x558f1bed7f30 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.779+0000 7f09133dc740 1 --2- 172.21.15.12:0/2946851721 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f1bec6810 0x558f1bed0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.779+0000 7f09133dc740 1 -- 172.21.15.12:0/2946851721 >> 172.21.15.12:0/2946851721 conn(0x558f1bec1c00 msgr2=0x558f1bec56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:30.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.779+0000 7f09133dc740 1 -- 172.21.15.12:0/2946851721 shutdown_connections 2024-03-23T23:12:30.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.779+0000 7f09133dc740 1 -- 172.21.15.12:0/2946851721 wait complete. 2024-03-23T23:12:30.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.780+0000 7f09133dc740 1 -- start start 2024-03-23T23:12:30.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.780+0000 7f09133dc740 1 Processor -- start 2024-03-23T23:12:30.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.780+0000 7f09133dc740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f1bec6810 0x558f1bed7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.780+0000 7f09133dc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec79e0 0x558f1bed6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.780+0000 7f09133dc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f1bec83a0 0x558f1bed65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.780+0000 7f09133dc740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09081ad740 con 0x558f1bec6810 2024-03-23T23:12:30.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.780+0000 7f09133dc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09081aea40 con 0x558f1bec79e0 2024-03-23T23:12:30.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.780+0000 7f09133dc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09081add80 con 0x558f1bec83a0 2024-03-23T23:12:30.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.781+0000 7f09133dc740 10 client.0 _ll_register_callbacks cb 0x7f08fc005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:12:30.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.781+0000 7f090d200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec79e0 0x558f1bed6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.781+0000 7f090dc00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f1bec6810 0x558f1bed7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.781+0000 7f090e600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f1bec83a0 0x558f1bed65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.781+0000 7f090d200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec79e0 0x558f1bed6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37972/0 (socket says 172.21.15.12:37972) 2024-03-23T23:12:30.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.781+0000 7f090d200640 1 -- 172.21.15.12:0/1804696298 learned_addr learned my addr 172.21.15.12:0/1804696298 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:30.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.781+0000 7f090e600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f1bec83a0 0x558f1bed65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:57958/0 (socket says 172.21.15.12:57958) 2024-03-23T23:12:30.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.781+0000 7f090d200640 1 -- 172.21.15.12:0/1804696298 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f1bec83a0 msgr2=0x558f1bed65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:30.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.781+0000 7f090d200640 1 --2- 172.21.15.12:0/1804696298 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f1bec83a0 0x558f1bed65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.781+0000 7f090d200640 1 -- 172.21.15.12:0/1804696298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f1bec6810 msgr2=0x558f1bed7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:30.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.781+0000 7f090d200640 1 --2- 172.21.15.12:0/1804696298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f1bec6810 0x558f1bed7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.781+0000 7f090d200640 1 -- 172.21.15.12:0/1804696298 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558f1bed6b10 con 0x558f1bec79e0 2024-03-23T23:12:30.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.781+0000 7f090d200640 1 --2- 172.21.15.12:0/1804696298 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec79e0 0x558f1bed6090 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f08f40114c0 tx=0x7f08f40047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b02810c56c47989 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:30.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.781+0000 7f08ef400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:12:30.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.782+0000 7f08ef400640 1 -- 172.21.15.12:0/1804696298 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f403b070 con 0x558f1bec79e0 2024-03-23T23:12:30.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.782+0000 7f08ef400640 1 -- 172.21.15.12:0/1804696298 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f08f4019450 con 0x558f1bec79e0 2024-03-23T23:12:30.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.782+0000 7f09133dc740 1 -- 172.21.15.12:0/1804696298 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558f1bed07e0 con 0x558f1bec79e0 2024-03-23T23:12:30.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.782+0000 7f08ef400640 1 -- 172.21.15.12:0/1804696298 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f08f4047050 con 0x558f1bec79e0 2024-03-23T23:12:30.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.782+0000 7f09133dc740 10 client.28304 fetch_fsmap learned FSMap version 23 2024-03-23T23:12:30.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.782+0000 7f09133dc740 1 -- 172.21.15.12:0/1804696298 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x558f1bf386f0 con 0x558f1bec79e0 2024-03-23T23:12:30.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.782+0000 7f08ef400640 1 -- 172.21.15.12:0/1804696298 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x558f1bed07e0 con 0x558f1bec79e0 2024-03-23T23:12:30.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.782+0000 7f08ef400640 1 -- 172.21.15.12:0/1804696298 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f08f402c440 con 0x558f1bec79e0 2024-03-23T23:12:30.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.782+0000 7f09133dc740 10 client.28304 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:12:30.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.782+0000 7f09133dc740 10 client.28304 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:12:30.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.782+0000 7f09133dc740 20 client.28304 populate_metadata read hostname 'smithi012' 2024-03-23T23:12:30.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.782+0000 7f09133dc740 1 --2- 172.21.15.12:0/1804696298 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x558f1bf38fc0 0x558f1bf3b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:30.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.782+0000 7f09133dc740 4 client.28304 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:12:30.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.782+0000 7f09133dc740 1 -- 172.21.15.12:0/1804696298 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558f1be32b50 con 0x558f1bf38fc0 2024-03-23T23:12:30.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.783+0000 7f090dc00640 1 --2- 172.21.15.12:0/1804696298 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x558f1bf38fc0 0x558f1bf3b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:30.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.783+0000 7f090dc00640 1 --2- 172.21.15.12:0/1804696298 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x558f1bf38fc0 0x558f1bf3b3e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=29d96491574df323 server_cookie=524e228dc646022 in_seq=0 out_seq=0 2024-03-23T23:12:30.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.783+0000 7f08ef400640 10 client.28304 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:12:30.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.783+0000 7f08ef400640 1 -- 172.21.15.12:0/1804696298 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558f1be32b50 con 0x558f1bf38fc0 2024-03-23T23:12:30.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.783+0000 7f08ef400640 10 client.28304 handle_command_reply: tid=0 2024-03-23T23:12:30.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.783+0000 7f09133dc740 1 client.28304 shutdown 2024-03-23T23:12:30.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.784+0000 7f09133dc740 1 -- 172.21.15.12:0/1804696298 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec79e0 msgr2=0x558f1bed6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:30.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.784+0000 7f09133dc740 1 --2- 172.21.15.12:0/1804696298 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec79e0 0x558f1bed6090 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f08f40114c0 tx=0x7f08f40047c0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.784+0000 7f09133dc740 1 -- 172.21.15.12:0/1804696298 shutdown_connections 2024-03-23T23:12:30.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.784+0000 7f09133dc740 1 --2- 172.21.15.12:0/1804696298 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x558f1bf38fc0 0x558f1bf3b3e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.784+0000 7f09133dc740 1 --2- 172.21.15.12:0/1804696298 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f1bec83a0 0x558f1bed65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.784+0000 7f09133dc740 1 --2- 172.21.15.12:0/1804696298 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f1bec79e0 0x558f1bed6090 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.784+0000 7f09133dc740 1 --2- 172.21.15.12:0/1804696298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f1bec6810 0x558f1bed7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.784+0000 7f08ef400640 0 client.28304 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:12:30.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.784+0000 7f09133dc740 1 -- 172.21.15.12:0/1804696298 >> 172.21.15.12:0/1804696298 conn(0x558f1bec1c00 msgr2=0x558f1bec56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:30.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.784+0000 7f09133dc740 1 -- 172.21.15.12:0/1804696298 shutdown_connections 2024-03-23T23:12:30.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.784+0000 7f09133dc740 1 -- 172.21.15.12:0/1804696298 wait complete. 2024-03-23T23:12:30.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.784+0000 7f09133dc740 20 client.28304 trim_cache size 0 max 16384 2024-03-23T23:12:30.788 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:12:30.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.784+0000 7f0910600640 1 -- 172.21.15.12:0/1276998102 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f08f00a2800 msgr2=0x7f08f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:30.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.784+0000 7f0910600640 1 --2- 172.21.15.12:0/1276998102 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f08f00a2800 0x7f08f00a4cc0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f0900009430 tx=0x7f0900009360 comp rx=0 tx=0).stop 2024-03-23T23:12:30.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.784+0000 7f0910600640 1 -- 172.21.15.12:0/1276998102 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0908074100 msgr2=0x7f090819df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:30.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.784+0000 7f0910600640 1 --2- 172.21.15.12:0/1276998102 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0908074100 0x7f090819df90 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f08fc00e9f0 tx=0x7f08fc00eeb0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.785+0000 7f0910600640 1 -- 172.21.15.12:0/1276998102 shutdown_connections 2024-03-23T23:12:30.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.785+0000 7f0910600640 1 --2- 172.21.15.12:0/1276998102 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f08f00a2800 0x7f08f00a4cc0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.785+0000 7f0910600640 1 --2- 172.21.15.12:0/1276998102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f090810dca0 0x7f09081a5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.785+0000 7f0910600640 1 --2- 172.21.15.12:0/1276998102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09080756e0 0x7f090819e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.785+0000 7f0910600640 1 --2- 172.21.15.12:0/1276998102 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0908074100 0x7f090819df90 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:30.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.785+0000 7f0910600640 1 -- 172.21.15.12:0/1276998102 >> 172.21.15.12:0/1276998102 conn(0x7f09080fcc40 msgr2=0x7f0908101040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:30.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.785+0000 7f0910600640 1 -- 172.21.15.12:0/1276998102 shutdown_connections 2024-03-23T23:12:30.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:30.785+0000 7f0910600640 1 -- 172.21.15.12:0/1276998102 wait complete. 2024-03-23T23:12:30.803 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:12:30.804 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-23T23:12:31.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.028+0000 7f3426400640 1 -- 172.21.15.12:0/2563318980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3420109870 msgr2=0x7f3420110100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.028+0000 7f3426400640 1 --2- 172.21.15.12:0/2563318980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3420109870 0x7f3420110100 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f3418009900 tx=0x7f341802bc30 comp rx=0 tx=0).stop 2024-03-23T23:12:31.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.028+0000 7f3426400640 1 -- 172.21.15.12:0/2563318980 shutdown_connections 2024-03-23T23:12:31.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.028+0000 7f3426400640 1 --2- 172.21.15.12:0/2563318980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3420109870 0x7f3420110100 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.028+0000 7f3426400640 1 --2- 172.21.15.12:0/2563318980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3420108eb0 0x7f3420109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.028+0000 7f3426400640 1 --2- 172.21.15.12:0/2563318980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3420107cc0 0x7f34201080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.028+0000 7f3426400640 1 -- 172.21.15.12:0/2563318980 >> 172.21.15.12:0/2563318980 conn(0x7f3420076c60 msgr2=0x7f34200790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:31.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.028+0000 7f3426400640 1 -- 172.21.15.12:0/2563318980 shutdown_connections 2024-03-23T23:12:31.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.028+0000 7f3426400640 1 -- 172.21.15.12:0/2563318980 wait complete. 2024-03-23T23:12:31.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.030+0000 7f3426400640 1 Processor -- start 2024-03-23T23:12:31.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.030+0000 7f3426400640 1 -- start start 2024-03-23T23:12:31.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.030+0000 7f3426400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3420107cc0 0x7f342019df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.030+0000 7f3426400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3420108eb0 0x7f342019e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.030+0000 7f3426400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3420109870 0x7f34201a5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.030+0000 7f3426400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3420113990 con 0x7f3420109870 2024-03-23T23:12:31.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.030+0000 7f3426400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3420113810 con 0x7f3420108eb0 2024-03-23T23:12:31.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.030+0000 7f3426400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3420113b10 con 0x7f3420107cc0 2024-03-23T23:12:31.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.030+0000 7f341f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3420107cc0 0x7f342019df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.031+0000 7f341f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3420107cc0 0x7f342019df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:57972/0 (socket says 172.21.15.12:57972) 2024-03-23T23:12:31.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.031+0000 7f341f400640 1 -- 172.21.15.12:0/2706545134 learned_addr learned my addr 172.21.15.12:0/2706545134 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:31.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.031+0000 7f3416a00640 1 --2- 172.21.15.12:0/2706545134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3420108eb0 0x7f342019e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.031+0000 7f341fe00640 1 --2- 172.21.15.12:0/2706545134 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3420109870 0x7f34201a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.031+0000 7f341f400640 1 -- 172.21.15.12:0/2706545134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3420108eb0 msgr2=0x7f342019e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.031+0000 7f341f400640 1 --2- 172.21.15.12:0/2706545134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3420108eb0 0x7f342019e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.031+0000 7f341f400640 1 -- 172.21.15.12:0/2706545134 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3420109870 msgr2=0x7f34201a5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.031+0000 7f341f400640 1 --2- 172.21.15.12:0/2706545134 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3420109870 0x7f34201a5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.031+0000 7f341f400640 1 -- 172.21.15.12:0/2706545134 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34201a5b90 con 0x7f3420107cc0 2024-03-23T23:12:31.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.031+0000 7f341fe00640 1 --2- 172.21.15.12:0/2706545134 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3420109870 0x7f34201a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:12:31.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.031+0000 7f341f400640 1 --2- 172.21.15.12:0/2706545134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3420107cc0 0x7f342019df70 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f341000eb70 tx=0x7f341000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:31.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.031+0000 7f341cc00640 1 -- 172.21.15.12:0/2706545134 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3410003b70 con 0x7f3420107cc0 2024-03-23T23:12:31.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.032+0000 7f341cc00640 1 -- 172.21.15.12:0/2706545134 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f341000ee70 con 0x7f3420107cc0 2024-03-23T23:12:31.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.032+0000 7f341cc00640 1 -- 172.21.15.12:0/2706545134 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3410010830 con 0x7f3420107cc0 2024-03-23T23:12:31.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.032+0000 7f3426400640 1 -- 172.21.15.12:0/2706545134 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34201a5e80 con 0x7f3420107cc0 2024-03-23T23:12:31.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.032+0000 7f3426400640 1 -- 172.21.15.12:0/2706545134 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34201a62e0 con 0x7f3420107cc0 2024-03-23T23:12:31.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.035+0000 7f34291a3740 1 Processor -- start 2024-03-23T23:12:31.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.035+0000 7f341cc00640 1 -- 172.21.15.12:0/2706545134 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3410004020 con 0x7f3420107cc0 2024-03-23T23:12:31.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.035+0000 7f34291a3740 1 -- start start 2024-03-23T23:12:31.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.035+0000 7f34291a3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559db31d3810 0x559db31d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.035+0000 7f34291a3740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559db31d49e0 0x559db31d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.035+0000 7f34291a3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559db31d53a0 0x559db31dbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.035+0000 7f34291a3740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x559db31382c0 con 0x559db31d49e0 2024-03-23T23:12:31.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.035+0000 7f34291a3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x559db30e0c90 con 0x559db31d3810 2024-03-23T23:12:31.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.035+0000 7f34291a3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x559db3134360 con 0x559db31d53a0 2024-03-23T23:12:31.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.036+0000 7f341fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559db31d3810 0x559db31d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.036+0000 7f341fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559db31d3810 0x559db31d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:37992/0 (socket says 172.21.15.12:37992) 2024-03-23T23:12:31.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.036+0000 7f341fe00640 1 -- 172.21.15.12:0/2061493326 learned_addr learned my addr 172.21.15.12:0/2061493326 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:31.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.036+0000 7f3416a00640 1 --2- 172.21.15.12:0/2061493326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559db31d53a0 0x559db31dbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.036+0000 7f341f400640 1 --2- 172.21.15.12:0/2061493326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559db31d49e0 0x559db31d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.036+0000 7f3416a00640 1 -- 172.21.15.12:0/2061493326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559db31d3810 msgr2=0x559db31d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:31.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.036+0000 7f3416a00640 1 --2- 172.21.15.12:0/2061493326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559db31d3810 0x559db31d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.036+0000 7f3416a00640 1 -- 172.21.15.12:0/2061493326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559db31d49e0 msgr2=0x559db31d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:31.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.036+0000 7f3416a00640 1 --2- 172.21.15.12:0/2061493326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559db31d49e0 0x559db31d4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.036+0000 7f3416a00640 1 -- 172.21.15.12:0/2061493326 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559db3105080 con 0x559db31d53a0 2024-03-23T23:12:31.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.036+0000 7f341fe00640 1 --2- 172.21.15.12:0/2061493326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559db31d3810 0x559db31d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:12:31.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.037+0000 7f3416a00640 1 --2- 172.21.15.12:0/2061493326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559db31d53a0 0x559db31dbc30 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f3408005d00 tx=0x7f34080107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f08b8174ebb9b8e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:31.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.037+0000 7f3414c00640 1 -- 172.21.15.12:0/2061493326 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340801c480 con 0x559db31d53a0 2024-03-23T23:12:31.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.037+0000 7f3414c00640 1 -- 172.21.15.12:0/2061493326 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f340801cb00 con 0x559db31d53a0 2024-03-23T23:12:31.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.037+0000 7f3414c00640 1 -- 172.21.15.12:0/2061493326 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3408011c30 con 0x559db31d53a0 2024-03-23T23:12:31.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.037+0000 7f341cc00640 1 --2- 172.21.15.12:0/2706545134 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f34040a2880 0x7f34040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.037+0000 7f341cc00640 1 -- 172.21.15.12:0/2706545134 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f34100cf150 con 0x7f3420107cc0 2024-03-23T23:12:31.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.037+0000 7f341fe00640 1 --2- 172.21.15.12:0/2706545134 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f34040a2880 0x7f34040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.038+0000 7f34291a3740 1 -- 172.21.15.12:0/2061493326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559db31d53a0 msgr2=0x559db31dbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.038+0000 7f34291a3740 1 --2- 172.21.15.12:0/2061493326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559db31d53a0 0x559db31dbc30 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f3408005d00 tx=0x7f34080107a0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.038+0000 7f34291a3740 1 -- 172.21.15.12:0/2061493326 shutdown_connections 2024-03-23T23:12:31.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.038+0000 7f34291a3740 1 --2- 172.21.15.12:0/2061493326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559db31d53a0 0x559db31dbc30 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.038+0000 7f34291a3740 1 --2- 172.21.15.12:0/2061493326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559db31d49e0 0x559db31d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.038+0000 7f34291a3740 1 --2- 172.21.15.12:0/2061493326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559db31d3810 0x559db31d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.038+0000 7f34291a3740 1 -- 172.21.15.12:0/2061493326 >> 172.21.15.12:0/2061493326 conn(0x559db31cec00 msgr2=0x559db31cf000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:31.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.038+0000 7f34291a3740 1 -- 172.21.15.12:0/2061493326 shutdown_connections 2024-03-23T23:12:31.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.038+0000 7f341fe00640 1 --2- 172.21.15.12:0/2706545134 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f34040a2880 0x7f34040a4d40 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3418005c40 tx=0x7f341802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:31.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.038+0000 7f34291a3740 1 -- 172.21.15.12:0/2061493326 wait complete. 2024-03-23T23:12:31.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.038+0000 7f34291a3740 1 -- start start 2024-03-23T23:12:31.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.039+0000 7f34291a3740 1 Processor -- start 2024-03-23T23:12:31.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.039+0000 7f34291a3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559db31d3810 0x559db31dda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.039+0000 7f34291a3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559db31d49e0 0x559db31ddfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.039+0000 7f34291a3740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559db31d53a0 0x559db31e5000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.039+0000 7f34291a3740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x559db31dc3b0 con 0x559db31d53a0 2024-03-23T23:12:31.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.039+0000 7f34291a3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x559db30f40d0 con 0x559db31d3810 2024-03-23T23:12:31.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.039+0000 7f34291a3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x559db31dc530 con 0x559db31d49e0 2024-03-23T23:12:31.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.039+0000 7f341f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559db31d3810 0x559db31dda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.039+0000 7f341f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559db31d3810 0x559db31dda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:38000/0 (socket says 172.21.15.12:38000) 2024-03-23T23:12:31.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.039+0000 7f341f400640 1 -- 172.21.15.12:0/2660920031 learned_addr learned my addr 172.21.15.12:0/2660920031 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:31.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.039+0000 7f3416a00640 1 --2- 172.21.15.12:0/2660920031 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559db31d49e0 0x559db31ddfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.039+0000 7f34291a3740 10 client.0 _ll_register_callbacks cb 0x559db30f8e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:12:31.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.040+0000 7f341fe00640 1 --2- 172.21.15.12:0/2660920031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559db31d53a0 0x559db31e5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.040+0000 7f341f400640 1 -- 172.21.15.12:0/2660920031 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559db31d49e0 msgr2=0x559db31ddfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:31.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.040+0000 7f341f400640 1 --2- 172.21.15.12:0/2660920031 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559db31d49e0 0x559db31ddfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.040+0000 7f341f400640 1 -- 172.21.15.12:0/2660920031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559db31d53a0 msgr2=0x559db31e5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:31.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.040+0000 7f341f400640 1 --2- 172.21.15.12:0/2660920031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559db31d53a0 0x559db31e5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.040+0000 7f341f400640 1 -- 172.21.15.12:0/2660920031 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559db2e4c3f0 con 0x559db31d3810 2024-03-23T23:12:31.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.040+0000 7f341f400640 1 --2- 172.21.15.12:0/2660920031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559db31d3810 0x559db31dda80 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f341009ba30 tx=0x7f34100099b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8707cf460790d26 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:31.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.040+0000 7f340d600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:12:31.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.040+0000 7f340d600640 1 -- 172.21.15.12:0/2660920031 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341009b870 con 0x559db31d3810 2024-03-23T23:12:31.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.040+0000 7f340d600640 1 -- 172.21.15.12:0/2660920031 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f3410004310 con 0x559db31d3810 2024-03-23T23:12:31.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.040+0000 7f34291a3740 1 -- 172.21.15.12:0/2660920031 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f340801cb00 con 0x559db31d3810 2024-03-23T23:12:31.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.041+0000 7f340d600640 1 -- 172.21.15.12:0/2660920031 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f341009b400 con 0x559db31d3810 2024-03-23T23:12:31.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.041+0000 7f34291a3740 10 client.28316 fetch_fsmap learned FSMap version 23 2024-03-23T23:12:31.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.041+0000 7f34291a3740 1 -- 172.21.15.12:0/2660920031 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x559db2f167d0 con 0x559db31d3810 2024-03-23T23:12:31.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.041+0000 7f340d600640 1 -- 172.21.15.12:0/2660920031 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f340801cb00 con 0x559db31d3810 2024-03-23T23:12:31.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.041+0000 7f340d600640 1 -- 172.21.15.12:0/2660920031 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f34100a3db0 con 0x559db31d3810 2024-03-23T23:12:31.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.041+0000 7f34291a3740 10 client.28316 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:12:31.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.041+0000 7f34291a3740 10 client.28316 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:12:31.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.041+0000 7f34291a3740 20 client.28316 populate_metadata read hostname 'smithi012' 2024-03-23T23:12:31.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.041+0000 7f34291a3740 1 --2- 172.21.15.12:0/2660920031 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x559db31e9500 0x559db31eb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.041+0000 7f34291a3740 4 client.28316 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:12:31.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.041+0000 7f34291a3740 1 -- 172.21.15.12:0/2660920031 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559db3145800 con 0x559db31e9500 2024-03-23T23:12:31.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.042+0000 7f3416a00640 1 --2- 172.21.15.12:0/2660920031 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x559db31e9500 0x559db31eb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.042+0000 7f3416a00640 1 --2- 172.21.15.12:0/2660920031 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x559db31e9500 0x559db31eb9a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4f8a643e4f4e4dc1 server_cookie=c9b161db125cb9ae in_seq=0 out_seq=0 2024-03-23T23:12:31.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.042+0000 7f340d600640 10 client.28316 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:12:31.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.043+0000 7f340d600640 1 -- 172.21.15.12:0/2660920031 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559db3145800 con 0x559db31e9500 2024-03-23T23:12:31.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.043+0000 7f340d600640 10 client.28316 handle_command_reply: tid=0 2024-03-23T23:12:31.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.043+0000 7f34291a3740 1 client.28316 shutdown 2024-03-23T23:12:31.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.043+0000 7f34291a3740 1 -- 172.21.15.12:0/2660920031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559db31d3810 msgr2=0x559db31dda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.043+0000 7f34291a3740 1 --2- 172.21.15.12:0/2660920031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559db31d3810 0x559db31dda80 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f341009ba30 tx=0x7f34100099b0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.043+0000 7f34291a3740 1 -- 172.21.15.12:0/2660920031 shutdown_connections 2024-03-23T23:12:31.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.043+0000 7f34291a3740 1 --2- 172.21.15.12:0/2660920031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559db31d53a0 0x559db31e5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.043+0000 7f34291a3740 1 --2- 172.21.15.12:0/2660920031 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x559db31e9500 0x559db31eb9a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.043+0000 7f34291a3740 1 --2- 172.21.15.12:0/2660920031 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559db31d49e0 0x559db31ddfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.043+0000 7f34291a3740 1 --2- 172.21.15.12:0/2660920031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559db31d3810 0x559db31dda80 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.043+0000 7f340d600640 0 client.28316 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:12:31.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.043+0000 7f34291a3740 1 -- 172.21.15.12:0/2660920031 >> 172.21.15.12:0/2660920031 conn(0x559db31cec00 msgr2=0x559db31cdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:31.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.044+0000 7f34291a3740 1 -- 172.21.15.12:0/2660920031 shutdown_connections 2024-03-23T23:12:31.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.044+0000 7f34291a3740 1 -- 172.21.15.12:0/2660920031 wait complete. 2024-03-23T23:12:31.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.044+0000 7f34291a3740 20 client.28316 trim_cache size 0 max 16384 2024-03-23T23:12:31.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.058+0000 7f34291a3740 1 Processor -- start 2024-03-23T23:12:31.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.058+0000 7f34291a3740 1 -- start start 2024-03-23T23:12:31.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.058+0000 7f34291a3740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559db31d3810 0x559db31cdd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.058+0000 7f34291a3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559db31d49e0 0x559db31e4f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.058+0000 7f34291a3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559db31d53a0 0x559db31f0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.058+0000 7f34291a3740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34201b14d0 con 0x559db31d3810 2024-03-23T23:12:31.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.058+0000 7f34291a3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34201aefd0 con 0x559db31d53a0 2024-03-23T23:12:31.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.058+0000 7f34291a3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34201ad070 con 0x559db31d49e0 2024-03-23T23:12:31.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.058+0000 7f3416a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559db31d49e0 0x559db31e4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.058+0000 7f341f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559db31d3810 0x559db31cdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.058+0000 7f3416a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559db31d49e0 0x559db31e4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:57998/0 (socket says 172.21.15.12:57998) 2024-03-23T23:12:31.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.058+0000 7f3416a00640 1 -- 172.21.15.12:0/2345388957 learned_addr learned my addr 172.21.15.12:0/2345388957 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:31.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.058+0000 7f341fe00640 1 --2- 172.21.15.12:0/2345388957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559db31d53a0 0x559db31f0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.059+0000 7f3416a00640 1 -- 172.21.15.12:0/2345388957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559db31d53a0 msgr2=0x559db31f0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:31.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.059+0000 7f3416a00640 1 --2- 172.21.15.12:0/2345388957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559db31d53a0 0x559db31f0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.059+0000 7f3416a00640 1 -- 172.21.15.12:0/2345388957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559db31d3810 msgr2=0x559db31cdd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:31.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.059+0000 7f3416a00640 1 --2- 172.21.15.12:0/2345388957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559db31d3810 0x559db31cdd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.059+0000 7f3416a00640 1 -- 172.21.15.12:0/2345388957 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559db31e92c0 con 0x559db31d49e0 2024-03-23T23:12:31.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.059+0000 7f341f400640 1 --2- 172.21.15.12:0/2345388957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559db31d3810 0x559db31cdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:12:31.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.059+0000 7f3416a00640 1 --2- 172.21.15.12:0/2345388957 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559db31d49e0 0x559db31e4f90 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x559db31cfb70 tx=0x7f340800b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=766ac7bf5f167d91 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:31.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.059+0000 7f340fe00640 1 -- 172.21.15.12:0/2345388957 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340802e020 con 0x559db31d49e0 2024-03-23T23:12:31.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.059+0000 7f340fe00640 1 -- 172.21.15.12:0/2345388957 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f340802a0d0 con 0x559db31d49e0 2024-03-23T23:12:31.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.059+0000 7f340fe00640 1 -- 172.21.15.12:0/2345388957 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340802a780 con 0x559db31d49e0 2024-03-23T23:12:31.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.060+0000 7f34291a3740 1 -- 172.21.15.12:0/2345388957 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559db31d49e0 msgr2=0x559db31e4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.060+0000 7f34291a3740 1 --2- 172.21.15.12:0/2345388957 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559db31d49e0 0x559db31e4f90 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x559db31cfb70 tx=0x7f340800b710 comp rx=0 tx=0).stop 2024-03-23T23:12:31.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.060+0000 7f34291a3740 1 -- 172.21.15.12:0/2345388957 shutdown_connections 2024-03-23T23:12:31.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.060+0000 7f34291a3740 1 --2- 172.21.15.12:0/2345388957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559db31d53a0 0x559db31f0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.060+0000 7f34291a3740 1 --2- 172.21.15.12:0/2345388957 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559db31d49e0 0x559db31e4f90 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.060+0000 7f34291a3740 1 --2- 172.21.15.12:0/2345388957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559db31d3810 0x559db31cdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.060+0000 7f34291a3740 1 -- 172.21.15.12:0/2345388957 >> 172.21.15.12:0/2345388957 conn(0x559db31cec00 msgr2=0x559db31cd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:31.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.060+0000 7f34291a3740 1 -- 172.21.15.12:0/2345388957 shutdown_connections 2024-03-23T23:12:31.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.060+0000 7f34291a3740 1 -- 172.21.15.12:0/2345388957 wait complete. 2024-03-23T23:12:31.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.061+0000 7f34291a3740 1 -- start start 2024-03-23T23:12:31.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.061+0000 7f34291a3740 1 Processor -- start 2024-03-23T23:12:31.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.061+0000 7f34291a3740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559db31d3810 0x559db31e10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.062+0000 7f34291a3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559db31d49e0 0x559db31e1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.062+0000 7f34291a3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559db31d53a0 0x559db31e4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.062+0000 7f34291a3740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34201ad670 con 0x559db31d3810 2024-03-23T23:12:31.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.062+0000 7f34291a3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34201ae970 con 0x559db31d53a0 2024-03-23T23:12:31.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.062+0000 7f34291a3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34201adcb0 con 0x559db31d49e0 2024-03-23T23:12:31.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.062+0000 7f34291a3740 10 client.0 _ll_register_callbacks cb 0x7f3410003ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:12:31.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.062+0000 7f3416a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559db31d49e0 0x559db31e1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.062+0000 7f341fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559db31d53a0 0x559db31e4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.062+0000 7f341f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559db31d3810 0x559db31e10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.062+0000 7f3416a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559db31d49e0 0x559db31e1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58008/0 (socket says 172.21.15.12:58008) 2024-03-23T23:12:31.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.062+0000 7f3416a00640 1 -- 172.21.15.12:0/2609191330 learned_addr learned my addr 172.21.15.12:0/2609191330 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:31.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.062+0000 7f341fe00640 1 -- 172.21.15.12:0/2609191330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559db31d49e0 msgr2=0x559db31e1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:31.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.062+0000 7f341fe00640 1 --2- 172.21.15.12:0/2609191330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559db31d49e0 0x559db31e1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.062+0000 7f341fe00640 1 -- 172.21.15.12:0/2609191330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559db31d3810 msgr2=0x559db31e10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:31.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.062+0000 7f341fe00640 1 --2- 172.21.15.12:0/2609191330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559db31d3810 0x559db31e10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.062+0000 7f341fe00640 1 -- 172.21.15.12:0/2609191330 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559db31e4c40 con 0x559db31d53a0 2024-03-23T23:12:31.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.062+0000 7f341f400640 1 --2- 172.21.15.12:0/2609191330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559db31d3810 0x559db31e10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:12:31.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.063+0000 7f341fe00640 1 --2- 172.21.15.12:0/2609191330 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559db31d53a0 0x559db31e4650 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x559db31d48e0 tx=0x7f341802bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3f0d666a1582913 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:31.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.063+0000 7f340d600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:12:31.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.063+0000 7f340d600640 1 -- 172.21.15.12:0/2609191330 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3418051070 con 0x559db31d53a0 2024-03-23T23:12:31.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.063+0000 7f340d600640 1 -- 172.21.15.12:0/2609191330 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f34180046c0 con 0x559db31d53a0 2024-03-23T23:12:31.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.063+0000 7f34291a3740 1 -- 172.21.15.12:0/2609191330 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f340802a0d0 con 0x559db31d53a0 2024-03-23T23:12:31.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.063+0000 7f340d600640 1 -- 172.21.15.12:0/2609191330 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f34180048c0 con 0x559db31d53a0 2024-03-23T23:12:31.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.063+0000 7f34291a3740 10 client.28322 fetch_fsmap learned FSMap version 23 2024-03-23T23:12:31.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.063+0000 7f34291a3740 1 -- 172.21.15.12:0/2609191330 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x559db3245880 con 0x559db31d53a0 2024-03-23T23:12:31.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.063+0000 7f340d600640 1 -- 172.21.15.12:0/2609191330 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f340802a0d0 con 0x559db31d53a0 2024-03-23T23:12:31.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.064+0000 7f340d600640 1 -- 172.21.15.12:0/2609191330 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f3418056460 con 0x559db31d53a0 2024-03-23T23:12:31.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.064+0000 7f34291a3740 10 client.28322 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:12:31.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.064+0000 7f34291a3740 10 client.28322 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:12:31.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.064+0000 7f34291a3740 20 client.28322 populate_metadata read hostname 'smithi012' 2024-03-23T23:12:31.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.064+0000 7f34291a3740 1 --2- 172.21.15.12:0/2609191330 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x559db3246200 0x559db3248620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.064+0000 7f34291a3740 4 client.28322 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:12:31.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.064+0000 7f34291a3740 1 -- 172.21.15.12:0/2609191330 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559db313fb50 con 0x559db3246200 2024-03-23T23:12:31.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.064+0000 7f341f400640 1 --2- 172.21.15.12:0/2609191330 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x559db3246200 0x559db3248620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.064+0000 7f341f400640 1 --2- 172.21.15.12:0/2609191330 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x559db3246200 0x559db3248620 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b27430aa8a801506 server_cookie=44d1e7974e3083d6 in_seq=0 out_seq=0 2024-03-23T23:12:31.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.064+0000 7f340d600640 10 client.28322 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:12:31.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.064+0000 7f340d600640 1 -- 172.21.15.12:0/2609191330 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559db313fb50 con 0x559db3246200 2024-03-23T23:12:31.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.064+0000 7f340d600640 10 client.28322 handle_command_reply: tid=0 2024-03-23T23:12:31.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.065+0000 7f34291a3740 1 client.28322 shutdown 2024-03-23T23:12:31.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.065+0000 7f34291a3740 1 -- 172.21.15.12:0/2609191330 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559db31d53a0 msgr2=0x559db31e4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.065+0000 7f34291a3740 1 --2- 172.21.15.12:0/2609191330 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559db31d53a0 0x559db31e4650 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x559db31d48e0 tx=0x7f341802bc30 comp rx=0 tx=0).stop 2024-03-23T23:12:31.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.065+0000 7f34291a3740 1 -- 172.21.15.12:0/2609191330 shutdown_connections 2024-03-23T23:12:31.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.065+0000 7f34291a3740 1 --2- 172.21.15.12:0/2609191330 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559db31d53a0 0x559db31e4650 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.065+0000 7f34291a3740 1 --2- 172.21.15.12:0/2609191330 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x559db3246200 0x559db3248620 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.065+0000 7f34291a3740 1 --2- 172.21.15.12:0/2609191330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559db31d49e0 0x559db31e1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.065+0000 7f34291a3740 1 --2- 172.21.15.12:0/2609191330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559db31d3810 0x559db31e10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.065+0000 7f340d600640 0 client.28322 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:12:31.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.065+0000 7f34291a3740 1 -- 172.21.15.12:0/2609191330 >> 172.21.15.12:0/2609191330 conn(0x559db31cec00 msgr2=0x559db31cdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:31.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.065+0000 7f34291a3740 1 -- 172.21.15.12:0/2609191330 shutdown_connections 2024-03-23T23:12:31.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.065+0000 7f34291a3740 1 -- 172.21.15.12:0/2609191330 wait complete. 2024-03-23T23:12:31.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.065+0000 7f34291a3740 20 client.28322 trim_cache size 0 max 16384 2024-03-23T23:12:31.069 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:12:31.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.066+0000 7f3426400640 1 -- 172.21.15.12:0/2706545134 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f34040a2880 msgr2=0x7f34040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.066+0000 7f3426400640 1 --2- 172.21.15.12:0/2706545134 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f34040a2880 0x7f34040a4d40 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3418005c40 tx=0x7f341802df90 comp rx=0 tx=0).stop 2024-03-23T23:12:31.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.066+0000 7f3426400640 1 -- 172.21.15.12:0/2706545134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3420107cc0 msgr2=0x7f342019df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.066+0000 7f3426400640 1 --2- 172.21.15.12:0/2706545134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3420107cc0 0x7f342019df70 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f341000eb70 tx=0x7f341000c6a0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.066+0000 7f3426400640 1 -- 172.21.15.12:0/2706545134 shutdown_connections 2024-03-23T23:12:31.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.066+0000 7f3426400640 1 --2- 172.21.15.12:0/2706545134 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f34040a2880 0x7f34040a4d40 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.066+0000 7f3426400640 1 --2- 172.21.15.12:0/2706545134 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3420109870 0x7f34201a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.066+0000 7f3426400640 1 --2- 172.21.15.12:0/2706545134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3420108eb0 0x7f342019e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.066+0000 7f3426400640 1 --2- 172.21.15.12:0/2706545134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3420107cc0 0x7f342019df70 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.066+0000 7f3426400640 1 -- 172.21.15.12:0/2706545134 >> 172.21.15.12:0/2706545134 conn(0x7f3420076c60 msgr2=0x7f342010c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:31.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.066+0000 7f3426400640 1 -- 172.21.15.12:0/2706545134 shutdown_connections 2024-03-23T23:12:31.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.066+0000 7f3426400640 1 -- 172.21.15.12:0/2706545134 wait complete. 2024-03-23T23:12:31.084 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:12:31.084 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-23T23:12:31.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.316+0000 7f2c8bc00640 1 -- 172.21.15.12:0/2352814541 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c841016b0 msgr2=0x7f2c84101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.316+0000 7f2c8bc00640 1 --2- 172.21.15.12:0/2352814541 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c841016b0 0x7f2c84101ad0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f2c78009900 tx=0x7f2c7802bc30 comp rx=0 tx=0).stop 2024-03-23T23:12:31.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.316+0000 7f2c8bc00640 1 -- 172.21.15.12:0/2352814541 shutdown_connections 2024-03-23T23:12:31.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.316+0000 7f2c8bc00640 1 --2- 172.21.15.12:0/2352814541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c84103260 0x7f2c84109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.316+0000 7f2c8bc00640 1 --2- 172.21.15.12:0/2352814541 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c841028a0 0x7f2c84102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.316+0000 7f2c8bc00640 1 --2- 172.21.15.12:0/2352814541 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c841016b0 0x7f2c84101ad0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.316+0000 7f2c8bc00640 1 -- 172.21.15.12:0/2352814541 >> 172.21.15.12:0/2352814541 conn(0x7f2c840fcc40 msgr2=0x7f2c840ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:31.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.316+0000 7f2c8bc00640 1 -- 172.21.15.12:0/2352814541 shutdown_connections 2024-03-23T23:12:31.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.316+0000 7f2c8bc00640 1 -- 172.21.15.12:0/2352814541 wait complete. 2024-03-23T23:12:31.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.317+0000 7f2c8bc00640 1 Processor -- start 2024-03-23T23:12:31.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.317+0000 7f2c8bc00640 1 -- start start 2024-03-23T23:12:31.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.318+0000 7f2c8bc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c841016b0 0x7f2c841979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.318+0000 7f2c8bc00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c841028a0 0x7f2c84197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.318+0000 7f2c8bc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c84103260 0x7f2c8419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.318+0000 7f2c8bc00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8410d350 con 0x7f2c841028a0 2024-03-23T23:12:31.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.318+0000 7f2c8bc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8410d1d0 con 0x7f2c84103260 2024-03-23T23:12:31.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.318+0000 7f2c8bc00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8410d4d0 con 0x7f2c841016b0 2024-03-23T23:12:31.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.318+0000 7f2c83e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c841028a0 0x7f2c84197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.318+0000 7f2c89200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c841016b0 0x7f2c841979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.318+0000 7f2c89c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c84103260 0x7f2c8419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.318+0000 7f2c89200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c841016b0 0x7f2c841979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58026/0 (socket says 172.21.15.12:58026) 2024-03-23T23:12:31.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.318+0000 7f2c89200640 1 -- 172.21.15.12:0/1731416365 learned_addr learned my addr 172.21.15.12:0/1731416365 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:31.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.318+0000 7f2c89200640 1 -- 172.21.15.12:0/1731416365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c84103260 msgr2=0x7f2c8419efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.318+0000 7f2c89200640 1 --2- 172.21.15.12:0/1731416365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c84103260 0x7f2c8419efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.318+0000 7f2c89200640 1 -- 172.21.15.12:0/1731416365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c841028a0 msgr2=0x7f2c84197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.318+0000 7f2c89200640 1 --2- 172.21.15.12:0/1731416365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c841028a0 0x7f2c84197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.318+0000 7f2c89200640 1 -- 172.21.15.12:0/1731416365 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c8419f610 con 0x7f2c841016b0 2024-03-23T23:12:31.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.319+0000 7f2c89200640 1 --2- 172.21.15.12:0/1731416365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c841016b0 0x7f2c841979f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f2c78000c00 tx=0x7f2c7802c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:31.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.319+0000 7f2c81600640 1 -- 172.21.15.12:0/1731416365 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c78043020 con 0x7f2c841016b0 2024-03-23T23:12:31.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.319+0000 7f2c81600640 1 -- 172.21.15.12:0/1731416365 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c780054e0 con 0x7f2c841016b0 2024-03-23T23:12:31.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.319+0000 7f2c8bc00640 1 -- 172.21.15.12:0/1731416365 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c8419f8a0 con 0x7f2c841016b0 2024-03-23T23:12:31.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.319+0000 7f2c81600640 1 -- 172.21.15.12:0/1731416365 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c78003c40 con 0x7f2c841016b0 2024-03-23T23:12:31.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.319+0000 7f2c8bc00640 1 -- 172.21.15.12:0/1731416365 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c8419fe40 con 0x7f2c841016b0 2024-03-23T23:12:31.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.322+0000 7f2c81600640 1 -- 172.21.15.12:0/1731416365 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2c78046020 con 0x7f2c841016b0 2024-03-23T23:12:31.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.323+0000 7f2c8ea06740 1 Processor -- start 2024-03-23T23:12:31.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.323+0000 7f2c8ea06740 1 -- start start 2024-03-23T23:12:31.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.324+0000 7f2c8ea06740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cab3867810 0x55cab3867c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.324+0000 7f2c8ea06740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cab38689e0 0x55cab3868e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.324+0000 7f2c8ea06740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cab38693a0 0x55cab386fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.324+0000 7f2c8ea06740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cab37cc2c0 con 0x55cab38693a0 2024-03-23T23:12:31.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.324+0000 7f2c8ea06740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cab3774c90 con 0x55cab38689e0 2024-03-23T23:12:31.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.324+0000 7f2c8ea06740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cab37c8360 con 0x55cab3867810 2024-03-23T23:12:31.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.324+0000 7f2c89c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cab3867810 0x55cab3867c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.324+0000 7f2c89c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cab3867810 0x55cab3867c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58040/0 (socket says 172.21.15.12:58040) 2024-03-23T23:12:31.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.324+0000 7f2c89c00640 1 -- 172.21.15.12:0/3846915074 learned_addr learned my addr 172.21.15.12:0/3846915074 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:31.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.324+0000 7f2c89200640 1 --2- 172.21.15.12:0/3846915074 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cab38689e0 0x55cab3868e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.324+0000 7f2c83e00640 1 --2- 172.21.15.12:0/3846915074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cab38693a0 0x55cab386fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.324+0000 7f2c81600640 1 --2- 172.21.15.12:0/1731416365 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2c6c0a2880 0x7f2c6c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.324+0000 7f2c89c00640 1 -- 172.21.15.12:0/3846915074 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cab38689e0 msgr2=0x55cab3868e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:31.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.324+0000 7f2c89c00640 1 --2- 172.21.15.12:0/3846915074 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cab38689e0 0x55cab3868e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.325+0000 7f2c89c00640 1 -- 172.21.15.12:0/3846915074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cab38693a0 msgr2=0x55cab386fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:31.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.325+0000 7f2c89c00640 1 --2- 172.21.15.12:0/3846915074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cab38693a0 0x55cab386fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.325+0000 7f2c89c00640 1 -- 172.21.15.12:0/3846915074 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cab3799080 con 0x55cab3867810 2024-03-23T23:12:31.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.325+0000 7f2c83e00640 1 --2- 172.21.15.12:0/3846915074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cab38693a0 0x55cab386fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:12:31.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.325+0000 7f2c81600640 1 -- 172.21.15.12:0/1731416365 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2c780f0a10 con 0x7f2c841016b0 2024-03-23T23:12:31.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.325+0000 7f2c89c00640 1 --2- 172.21.15.12:0/1731416365 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2c6c0a2880 0x7f2c6c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.325+0000 7f2c89c00640 1 --2- 172.21.15.12:0/3846915074 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cab3867810 0x55cab3867c30 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c009fd0 tx=0x7f2c7c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b59606054275558b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:31.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.326+0000 7f2c72000640 1 -- 172.21.15.12:0/3846915074 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c00f6e0 con 0x55cab3867810 2024-03-23T23:12:31.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.326+0000 7f2c72000640 1 -- 172.21.15.12:0/3846915074 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c7c00fd60 con 0x55cab3867810 2024-03-23T23:12:31.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.326+0000 7f2c72000640 1 -- 172.21.15.12:0/3846915074 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c01bcc0 con 0x55cab3867810 2024-03-23T23:12:31.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.326+0000 7f2c8ea06740 1 -- 172.21.15.12:0/3846915074 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cab3867810 msgr2=0x55cab3867c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.326+0000 7f2c8ea06740 1 --2- 172.21.15.12:0/3846915074 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cab3867810 0x55cab3867c30 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c009fd0 tx=0x7f2c7c0118d0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.326+0000 7f2c89c00640 1 --2- 172.21.15.12:0/1731416365 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2c6c0a2880 0x7f2c6c0a4d40 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c020000 tx=0x7f2c7c021000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:31.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.326+0000 7f2c8ea06740 1 -- 172.21.15.12:0/3846915074 shutdown_connections 2024-03-23T23:12:31.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.326+0000 7f2c8ea06740 1 --2- 172.21.15.12:0/3846915074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cab38693a0 0x55cab386fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.326+0000 7f2c8ea06740 1 --2- 172.21.15.12:0/3846915074 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cab38689e0 0x55cab3868e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.326+0000 7f2c8ea06740 1 --2- 172.21.15.12:0/3846915074 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cab3867810 0x55cab3867c30 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.326+0000 7f2c8ea06740 1 -- 172.21.15.12:0/3846915074 >> 172.21.15.12:0/3846915074 conn(0x55cab3862c00 msgr2=0x55cab3863000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:31.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.327+0000 7f2c8ea06740 1 -- 172.21.15.12:0/3846915074 shutdown_connections 2024-03-23T23:12:31.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.327+0000 7f2c8ea06740 1 -- 172.21.15.12:0/3846915074 wait complete. 2024-03-23T23:12:31.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.327+0000 7f2c8ea06740 1 -- start start 2024-03-23T23:12:31.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.327+0000 7f2c8ea06740 1 Processor -- start 2024-03-23T23:12:31.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.328+0000 7f2c8ea06740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cab3867810 0x55cab3875e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.328+0000 7f2c8ea06740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cab38689e0 0x55cab38763d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.328+0000 7f2c8ea06740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cab38693a0 0x55cab387d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.328+0000 7f2c8ea06740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cab38703b0 con 0x55cab38689e0 2024-03-23T23:12:31.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.328+0000 7f2c8ea06740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cab37880d0 con 0x55cab3867810 2024-03-23T23:12:31.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.328+0000 7f2c8ea06740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cab3870530 con 0x55cab38693a0 2024-03-23T23:12:31.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.328+0000 7f2c8ea06740 10 client.0 _ll_register_callbacks cb 0x55cab378ce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:12:31.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.328+0000 7f2c89200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cab3867810 0x55cab3875e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.328+0000 7f2c89200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cab3867810 0x55cab3875e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:38042/0 (socket says 172.21.15.12:38042) 2024-03-23T23:12:31.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.328+0000 7f2c89c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cab38693a0 0x55cab387d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.328+0000 7f2c89200640 1 -- 172.21.15.12:0/1460040041 learned_addr learned my addr 172.21.15.12:0/1460040041 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:31.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.328+0000 7f2c83e00640 1 --2- 172.21.15.12:0/1460040041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cab38689e0 0x55cab38763d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.328+0000 7f2c89200640 1 -- 172.21.15.12:0/1460040041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cab38693a0 msgr2=0x55cab387d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:31.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.328+0000 7f2c89200640 1 --2- 172.21.15.12:0/1460040041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cab38693a0 0x55cab387d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.328+0000 7f2c89200640 1 -- 172.21.15.12:0/1460040041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cab38689e0 msgr2=0x55cab38763d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:31.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.328+0000 7f2c89200640 1 --2- 172.21.15.12:0/1460040041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cab38689e0 0x55cab38763d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.328+0000 7f2c89200640 1 -- 172.21.15.12:0/1460040041 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cab34e03f0 con 0x55cab3867810 2024-03-23T23:12:31.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.328+0000 7f2c83e00640 1 --2- 172.21.15.12:0/1460040041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cab38689e0 0x55cab38763d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:12:31.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.329+0000 7f2c89200640 1 --2- 172.21.15.12:0/1460040041 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cab3867810 0x55cab3875e90 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f2c780b9d30 tx=0x7f2c780f4040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5531def8a584fb51 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:31.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.329+0000 7f2c66a00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:12:31.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.329+0000 7f2c66a00640 1 -- 172.21.15.12:0/1460040041 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c780dbe60 con 0x55cab3867810 2024-03-23T23:12:31.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.329+0000 7f2c66a00640 1 -- 172.21.15.12:0/1460040041 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2c780bb660 con 0x55cab3867810 2024-03-23T23:12:31.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.329+0000 7f2c8ea06740 1 -- 172.21.15.12:0/1460040041 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2c7c00fd60 con 0x55cab3867810 2024-03-23T23:12:31.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.329+0000 7f2c66a00640 1 -- 172.21.15.12:0/1460040041 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c78004250 con 0x55cab3867810 2024-03-23T23:12:31.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.329+0000 7f2c8ea06740 10 client.28334 fetch_fsmap learned FSMap version 23 2024-03-23T23:12:31.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.329+0000 7f2c8ea06740 1 -- 172.21.15.12:0/1460040041 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55cab35aa7d0 con 0x55cab3867810 2024-03-23T23:12:31.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.329+0000 7f2c66a00640 1 -- 172.21.15.12:0/1460040041 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c7c00fd60 con 0x55cab3867810 2024-03-23T23:12:31.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.329+0000 7f2c66a00640 1 -- 172.21.15.12:0/1460040041 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f2c780dd0b0 con 0x55cab3867810 2024-03-23T23:12:31.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.330+0000 7f2c8ea06740 10 client.28334 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:12:31.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.330+0000 7f2c8ea06740 10 client.28334 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:12:31.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.330+0000 7f2c8ea06740 20 client.28334 populate_metadata read hostname 'smithi012' 2024-03-23T23:12:31.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.330+0000 7f2c8ea06740 1 --2- 172.21.15.12:0/1460040041 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cab3881900 0x55cab3883d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.330+0000 7f2c8ea06740 4 client.28334 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:12:31.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.330+0000 7f2c8ea06740 1 -- 172.21.15.12:0/1460040041 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cab37d9800 con 0x55cab3881900 2024-03-23T23:12:31.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.330+0000 7f2c83e00640 1 --2- 172.21.15.12:0/1460040041 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cab3881900 0x55cab3883d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.330+0000 7f2c83e00640 1 --2- 172.21.15.12:0/1460040041 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cab3881900 0x55cab3883d30 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dca941b8e5921d4f server_cookie=868c0f4c1052a586 in_seq=0 out_seq=0 2024-03-23T23:12:31.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.330+0000 7f2c66a00640 10 client.28334 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:31.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.331+0000 7f2c66a00640 1 -- 172.21.15.12:0/1460040041 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cab37d9800 con 0x55cab3881900 2024-03-23T23:12:31.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.331+0000 7f2c66a00640 10 client.28334 handle_command_reply: tid=0 2024-03-23T23:12:31.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.331+0000 7f2c8ea06740 1 client.28334 shutdown 2024-03-23T23:12:31.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.331+0000 7f2c8ea06740 1 -- 172.21.15.12:0/1460040041 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cab3867810 msgr2=0x55cab3875e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.331+0000 7f2c8ea06740 1 --2- 172.21.15.12:0/1460040041 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cab3867810 0x55cab3875e90 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f2c780b9d30 tx=0x7f2c780f4040 comp rx=0 tx=0).stop 2024-03-23T23:12:31.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.332+0000 7f2c8ea06740 1 -- 172.21.15.12:0/1460040041 shutdown_connections 2024-03-23T23:12:31.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.332+0000 7f2c8ea06740 1 --2- 172.21.15.12:0/1460040041 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cab3881900 0x55cab3883d30 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.332+0000 7f2c8ea06740 1 --2- 172.21.15.12:0/1460040041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cab38693a0 0x55cab387d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.332+0000 7f2c8ea06740 1 --2- 172.21.15.12:0/1460040041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cab38689e0 0x55cab38763d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.332+0000 7f2c8ea06740 1 --2- 172.21.15.12:0/1460040041 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cab3867810 0x55cab3875e90 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.332+0000 7f2c8ea06740 1 -- 172.21.15.12:0/1460040041 >> 172.21.15.12:0/1460040041 conn(0x55cab3862c00 msgr2=0x55cab3861eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:31.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.332+0000 7f2c66a00640 0 client.28334 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:31.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.332+0000 7f2c8ea06740 1 -- 172.21.15.12:0/1460040041 shutdown_connections 2024-03-23T23:12:31.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.332+0000 7f2c8ea06740 1 -- 172.21.15.12:0/1460040041 wait complete. 2024-03-23T23:12:31.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.332+0000 7f2c8ea06740 20 client.28334 trim_cache size 0 max 16384 2024-03-23T23:12:31.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.346+0000 7f2c8ea06740 1 Processor -- start 2024-03-23T23:12:31.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.346+0000 7f2c8ea06740 1 -- start start 2024-03-23T23:12:31.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.346+0000 7f2c8ea06740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cab3867810 0x55cab3862fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.346+0000 7f2c8ea06740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cab38689e0 0x55cab387d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.346+0000 7f2c8ea06740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cab38693a0 0x55cab386ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.346+0000 7f2c8ea06740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c841a8d60 con 0x55cab38693a0 2024-03-23T23:12:31.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.346+0000 7f2c8ea06740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c841a6860 con 0x55cab38689e0 2024-03-23T23:12:31.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.346+0000 7f2c8ea06740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c841a4900 con 0x55cab3867810 2024-03-23T23:12:31.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.346+0000 7f2c89200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cab3867810 0x55cab3862fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.346+0000 7f2c83e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cab38689e0 0x55cab387d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.346+0000 7f2c89200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cab3867810 0x55cab3862fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58062/0 (socket says 172.21.15.12:58062) 2024-03-23T23:12:31.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.346+0000 7f2c89200640 1 -- 172.21.15.12:0/3907191695 learned_addr learned my addr 172.21.15.12:0/3907191695 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:31.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.346+0000 7f2c89c00640 1 --2- 172.21.15.12:0/3907191695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cab38693a0 0x55cab386ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.347+0000 7f2c89200640 1 -- 172.21.15.12:0/3907191695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cab38689e0 msgr2=0x55cab387d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.347+0000 7f2c89200640 1 --2- 172.21.15.12:0/3907191695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cab38689e0 0x55cab387d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.347+0000 7f2c89200640 1 -- 172.21.15.12:0/3907191695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cab38693a0 msgr2=0x55cab386ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.347+0000 7f2c89200640 1 --2- 172.21.15.12:0/3907191695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cab38693a0 0x55cab386ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.347+0000 7f2c89200640 1 -- 172.21.15.12:0/3907191695 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cab3870670 con 0x55cab3867810 2024-03-23T23:12:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.347+0000 7f2c83e00640 1 --2- 172.21.15.12:0/3907191695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cab38689e0 0x55cab387d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:12:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.347+0000 7f2c89200640 1 --2- 172.21.15.12:0/3907191695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cab3867810 0x55cab3862fe0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f2c7802ceb0 tx=0x7f2c7802bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=795dfd6baddb49f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.347+0000 7f2c71600640 1 -- 172.21.15.12:0/3907191695 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c780da070 con 0x55cab3867810 2024-03-23T23:12:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.347+0000 7f2c71600640 1 -- 172.21.15.12:0/3907191695 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c78005690 con 0x55cab3867810 2024-03-23T23:12:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.347+0000 7f2c71600640 1 -- 172.21.15.12:0/3907191695 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c780dbe60 con 0x55cab3867810 2024-03-23T23:12:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.348+0000 7f2c8ea06740 1 -- 172.21.15.12:0/3907191695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cab3867810 msgr2=0x55cab3862fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.348+0000 7f2c8ea06740 1 --2- 172.21.15.12:0/3907191695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cab3867810 0x55cab3862fe0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f2c7802ceb0 tx=0x7f2c7802bc30 comp rx=0 tx=0).stop 2024-03-23T23:12:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.348+0000 7f2c8ea06740 1 -- 172.21.15.12:0/3907191695 shutdown_connections 2024-03-23T23:12:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.348+0000 7f2c8ea06740 1 --2- 172.21.15.12:0/3907191695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cab38693a0 0x55cab386ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.348+0000 7f2c8ea06740 1 --2- 172.21.15.12:0/3907191695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cab38689e0 0x55cab387d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.348+0000 7f2c8ea06740 1 --2- 172.21.15.12:0/3907191695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cab3867810 0x55cab3862fe0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.348+0000 7f2c8ea06740 1 -- 172.21.15.12:0/3907191695 >> 172.21.15.12:0/3907191695 conn(0x55cab3862c00 msgr2=0x55cab38666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.348+0000 7f2c8ea06740 1 -- 172.21.15.12:0/3907191695 shutdown_connections 2024-03-23T23:12:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.348+0000 7f2c8ea06740 1 -- 172.21.15.12:0/3907191695 wait complete. 2024-03-23T23:12:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.348+0000 7f2c8ea06740 1 -- start start 2024-03-23T23:12:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.349+0000 7f2c8ea06740 1 Processor -- start 2024-03-23T23:12:31.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.349+0000 7f2c8ea06740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cab3867810 0x55cab387cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.349+0000 7f2c8ea06740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cab38689e0 0x55cab387d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.349+0000 7f2c8ea06740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cab38693a0 0x55cab3879750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.349+0000 7f2c8ea06740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c841a4f00 con 0x55cab3867810 2024-03-23T23:12:31.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.349+0000 7f2c8ea06740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c841a6200 con 0x55cab38689e0 2024-03-23T23:12:31.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.349+0000 7f2c8ea06740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c841a5540 con 0x55cab38693a0 2024-03-23T23:12:31.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.349+0000 7f2c8ea06740 10 client.0 _ll_register_callbacks cb 0x7f2c78005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:12:31.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.349+0000 7f2c89200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cab3867810 0x55cab387cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.349+0000 7f2c83e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cab38689e0 0x55cab387d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.349+0000 7f2c89c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cab38693a0 0x55cab3879750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.349+0000 7f2c83e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cab38689e0 0x55cab387d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:38060/0 (socket says 172.21.15.12:38060) 2024-03-23T23:12:31.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.349+0000 7f2c89200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cab3867810 0x55cab387cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59496/0 (socket says 172.21.15.12:59496) 2024-03-23T23:12:31.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.349+0000 7f2c83e00640 1 -- 172.21.15.12:0/2768097284 learned_addr learned my addr 172.21.15.12:0/2768097284 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:31.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.350+0000 7f2c83e00640 1 -- 172.21.15.12:0/2768097284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cab38693a0 msgr2=0x55cab3879750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:31.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.350+0000 7f2c83e00640 1 --2- 172.21.15.12:0/2768097284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cab38693a0 0x55cab3879750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.350+0000 7f2c83e00640 1 -- 172.21.15.12:0/2768097284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cab3867810 msgr2=0x55cab387cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:31.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.350+0000 7f2c83e00640 1 --2- 172.21.15.12:0/2768097284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cab3867810 0x55cab387cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.350+0000 7f2c83e00640 1 -- 172.21.15.12:0/2768097284 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cab3879df0 con 0x55cab38689e0 2024-03-23T23:12:31.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.350+0000 7f2c83e00640 1 --2- 172.21.15.12:0/2768097284 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cab38689e0 0x55cab387d130 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x55cab3863b70 tx=0x7f2c7401edd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a49d09adf15b8c57 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:31.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.350+0000 7f2c66a00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:12:31.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.350+0000 7f2c66a00640 1 -- 172.21.15.12:0/2768097284 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c74024060 con 0x55cab38689e0 2024-03-23T23:12:31.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.350+0000 7f2c66a00640 1 -- 172.21.15.12:0/2768097284 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2c74010880 con 0x55cab38689e0 2024-03-23T23:12:31.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.350+0000 7f2c8ea06740 1 -- 172.21.15.12:0/2768097284 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cab37d3b50 con 0x55cab38689e0 2024-03-23T23:12:31.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.351+0000 7f2c66a00640 1 -- 172.21.15.12:0/2768097284 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c74010a80 con 0x55cab38689e0 2024-03-23T23:12:31.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.351+0000 7f2c8ea06740 10 client.28340 fetch_fsmap learned FSMap version 23 2024-03-23T23:12:31.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.351+0000 7f2c8ea06740 1 -- 172.21.15.12:0/2768097284 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55cab38dde50 con 0x55cab38689e0 2024-03-23T23:12:31.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.351+0000 7f2c66a00640 1 -- 172.21.15.12:0/2768097284 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55cab37d3b50 con 0x55cab38689e0 2024-03-23T23:12:31.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.351+0000 7f2c66a00640 1 -- 172.21.15.12:0/2768097284 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f2c7402a460 con 0x55cab38689e0 2024-03-23T23:12:31.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.351+0000 7f2c8ea06740 10 client.28340 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:12:31.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.351+0000 7f2c8ea06740 10 client.28340 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:12:31.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.351+0000 7f2c8ea06740 20 client.28340 populate_metadata read hostname 'smithi012' 2024-03-23T23:12:31.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.351+0000 7f2c8ea06740 1 --2- 172.21.15.12:0/2768097284 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cab38de790 0x55cab38e0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.351+0000 7f2c8ea06740 4 client.28340 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:12:31.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.351+0000 7f2c8ea06740 1 -- 172.21.15.12:0/2768097284 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55cab3751500 con 0x55cab38de790 2024-03-23T23:12:31.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.351+0000 7f2c89200640 1 --2- 172.21.15.12:0/2768097284 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cab38de790 0x55cab38e0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.352+0000 7f2c89200640 1 --2- 172.21.15.12:0/2768097284 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cab38de790 0x55cab38e0bb0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc68269641de000c server_cookie=668cec67c1b6fdda in_seq=0 out_seq=0 2024-03-23T23:12:31.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.352+0000 7f2c66a00640 10 client.28340 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:31.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.352+0000 7f2c66a00640 1 -- 172.21.15.12:0/2768097284 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55cab3751500 con 0x55cab38de790 2024-03-23T23:12:31.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.352+0000 7f2c66a00640 10 client.28340 handle_command_reply: tid=0 2024-03-23T23:12:31.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.352+0000 7f2c8ea06740 1 client.28340 shutdown 2024-03-23T23:12:31.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.352+0000 7f2c8ea06740 1 -- 172.21.15.12:0/2768097284 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cab38689e0 msgr2=0x55cab387d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.352+0000 7f2c8ea06740 1 --2- 172.21.15.12:0/2768097284 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cab38689e0 0x55cab387d130 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x55cab3863b70 tx=0x7f2c7401edd0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.352+0000 7f2c8ea06740 1 -- 172.21.15.12:0/2768097284 shutdown_connections 2024-03-23T23:12:31.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.352+0000 7f2c8ea06740 1 --2- 172.21.15.12:0/2768097284 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55cab38de790 0x55cab38e0bb0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.352+0000 7f2c8ea06740 1 --2- 172.21.15.12:0/2768097284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cab38693a0 0x55cab3879750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.352+0000 7f2c8ea06740 1 --2- 172.21.15.12:0/2768097284 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cab38689e0 0x55cab387d130 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.352+0000 7f2c66a00640 0 client.28340 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:31.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.352+0000 7f2c8ea06740 1 --2- 172.21.15.12:0/2768097284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cab3867810 0x55cab387cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.352+0000 7f2c8ea06740 1 -- 172.21.15.12:0/2768097284 >> 172.21.15.12:0/2768097284 conn(0x55cab3862c00 msgr2=0x55cab38666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:31.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.352+0000 7f2c8ea06740 1 -- 172.21.15.12:0/2768097284 shutdown_connections 2024-03-23T23:12:31.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.352+0000 7f2c8ea06740 1 -- 172.21.15.12:0/2768097284 wait complete. 2024-03-23T23:12:31.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.353+0000 7f2c8ea06740 20 client.28340 trim_cache size 0 max 16384 2024-03-23T23:12:31.357 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:12:31.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.353+0000 7f2c8bc00640 1 -- 172.21.15.12:0/1731416365 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2c6c0a2880 msgr2=0x7f2c6c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.353+0000 7f2c8bc00640 1 --2- 172.21.15.12:0/1731416365 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2c6c0a2880 0x7f2c6c0a4d40 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c020000 tx=0x7f2c7c021000 comp rx=0 tx=0).stop 2024-03-23T23:12:31.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.353+0000 7f2c8bc00640 1 -- 172.21.15.12:0/1731416365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c841016b0 msgr2=0x7f2c841979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.353+0000 7f2c8bc00640 1 --2- 172.21.15.12:0/1731416365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c841016b0 0x7f2c841979f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f2c78000c00 tx=0x7f2c7802c650 comp rx=0 tx=0).stop 2024-03-23T23:12:31.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.353+0000 7f2c8bc00640 1 -- 172.21.15.12:0/1731416365 shutdown_connections 2024-03-23T23:12:31.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.353+0000 7f2c8bc00640 1 --2- 172.21.15.12:0/1731416365 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2c6c0a2880 0x7f2c6c0a4d40 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.353+0000 7f2c8bc00640 1 --2- 172.21.15.12:0/1731416365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c84103260 0x7f2c8419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.353+0000 7f2c8bc00640 1 --2- 172.21.15.12:0/1731416365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c841028a0 0x7f2c84197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.353+0000 7f2c8bc00640 1 --2- 172.21.15.12:0/1731416365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c841016b0 0x7f2c841979f0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.353+0000 7f2c8bc00640 1 -- 172.21.15.12:0/1731416365 >> 172.21.15.12:0/1731416365 conn(0x7f2c840fcc40 msgr2=0x7f2c84105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:31.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.353+0000 7f2c8bc00640 1 -- 172.21.15.12:0/1731416365 shutdown_connections 2024-03-23T23:12:31.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:31.353+0000 7f2c8bc00640 1 -- 172.21.15.12:0/1731416365 wait complete. 2024-03-23T23:12:31.368 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:31 smithi012 ceph-mon[31695]: pgmap v711: 97 pgs: 97 active+clean; 3.2 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:12:31.368 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:31 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3208659140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:12:31.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:31 smithi173 ceph-mon[36116]: pgmap v711: 97 pgs: 97 active+clean; 3.2 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:12:31.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:31 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3208659140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:12:31.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:31 smithi203 ceph-mon[36693]: pgmap v711: 97 pgs: 97 active+clean; 3.2 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:12:31.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:31 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3208659140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:12:31.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.917+0000 7f3405c00640 1 -- 172.21.15.12:0/3626515277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3400102170 msgr2=0x7f3400108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.917+0000 7f3405c00640 1 --2- 172.21.15.12:0/3626515277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3400102170 0x7f3400108a00 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f33f8009ca0 tx=0x7f33f802c100 comp rx=0 tx=0).stop 2024-03-23T23:12:31.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.917+0000 7f3405c00640 1 -- 172.21.15.12:0/3626515277 shutdown_connections 2024-03-23T23:12:31.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.917+0000 7f3405c00640 1 --2- 172.21.15.12:0/3626515277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3400102170 0x7f3400108a00 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.917+0000 7f3405c00640 1 --2- 172.21.15.12:0/3626515277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34001017b0 0x7f3400101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.918+0000 7f3405c00640 1 --2- 172.21.15.12:0/3626515277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34001005c0 0x7f34001009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.918+0000 7f3405c00640 1 -- 172.21.15.12:0/3626515277 >> 172.21.15.12:0/3626515277 conn(0x7f34000fbb30 msgr2=0x7f34000fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:31.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.918+0000 7f3405c00640 1 -- 172.21.15.12:0/3626515277 shutdown_connections 2024-03-23T23:12:31.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.918+0000 7f3405c00640 1 -- 172.21.15.12:0/3626515277 wait complete. 2024-03-23T23:12:31.920 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.919+0000 7f3405c00640 1 Processor -- start 2024-03-23T23:12:31.920 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.919+0000 7f3405c00640 1 -- start start 2024-03-23T23:12:31.920 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.919+0000 7f3405c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34001005c0 0x7f3400104550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.921 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.919+0000 7f3405c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34001017b0 0x7f3400102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.921 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.919+0000 7f3405c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3400102170 0x7f34001030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.921 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.919+0000 7f3405c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340010c220 con 0x7f3400102170 2024-03-23T23:12:31.921 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.919+0000 7f3405c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340010c0a0 con 0x7f34001017b0 2024-03-23T23:12:31.921 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.919+0000 7f3405c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340010c3a0 con 0x7f34001005c0 2024-03-23T23:12:31.921 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.919+0000 7f33fea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34001005c0 0x7f3400104550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.921 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.919+0000 7f33fea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34001005c0 0x7f3400104550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58086/0 (socket says 172.21.15.12:58086) 2024-03-23T23:12:31.921 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.919+0000 7f33fea00640 1 -- 172.21.15.12:0/2696232950 learned_addr learned my addr 172.21.15.12:0/2696232950 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:31.921 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.919+0000 7f33fe000640 1 --2- 172.21.15.12:0/2696232950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34001017b0 0x7f3400102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.921 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.919+0000 7f33ff400640 1 --2- 172.21.15.12:0/2696232950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3400102170 0x7f34001030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.921 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.920+0000 7f33fea00640 1 -- 172.21.15.12:0/2696232950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34001017b0 msgr2=0x7f3400102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.921 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.920+0000 7f33fea00640 1 --2- 172.21.15.12:0/2696232950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34001017b0 0x7f3400102b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.921 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.920+0000 7f33fea00640 1 -- 172.21.15.12:0/2696232950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3400102170 msgr2=0x7f34001030d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.921 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.920+0000 7f33fea00640 1 --2- 172.21.15.12:0/2696232950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3400102170 0x7f34001030d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.922 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.920+0000 7f33fea00640 1 -- 172.21.15.12:0/2696232950 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3400103960 con 0x7f34001005c0 2024-03-23T23:12:31.922 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.920+0000 7f33ff400640 1 --2- 172.21.15.12:0/2696232950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3400102170 0x7f34001030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:12:31.922 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.920+0000 7f33fe000640 1 --2- 172.21.15.12:0/2696232950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34001017b0 0x7f3400102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:12:31.922 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.920+0000 7f33fea00640 1 --2- 172.21.15.12:0/2696232950 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34001005c0 0x7f3400104550 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f33f000e9c0 tx=0x7f33f000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:31.922 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.920+0000 7f33f7400640 1 -- 172.21.15.12:0/2696232950 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f000cd80 con 0x7f34001005c0 2024-03-23T23:12:31.922 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.920+0000 7f33f7400640 1 -- 172.21.15.12:0/2696232950 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33f00035f0 con 0x7f34001005c0 2024-03-23T23:12:31.922 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.920+0000 7f3405c00640 1 -- 172.21.15.12:0/2696232950 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34001a19f0 con 0x7f34001005c0 2024-03-23T23:12:31.922 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.920+0000 7f33f7400640 1 -- 172.21.15.12:0/2696232950 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f0010690 con 0x7f34001005c0 2024-03-23T23:12:31.922 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.921+0000 7f3405c00640 1 -- 172.21.15.12:0/2696232950 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34001a1f40 con 0x7f34001005c0 2024-03-23T23:12:31.923 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.924+0000 7f33f7400640 1 -- 172.21.15.12:0/2696232950 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f33f00037a0 con 0x7f34001005c0 2024-03-23T23:12:31.924 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.924+0000 7f3408802740 1 Processor -- start 2024-03-23T23:12:31.924 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.924+0000 7f3408802740 1 -- start start 2024-03-23T23:12:31.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.924+0000 7f3408802740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7b1e0 0x555d89c7b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.924+0000 7f3408802740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d89c7bbd0 0x555d89c806d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.924+0000 7f3408802740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555d89c80d40 0x555d89c831d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.924+0000 7f3408802740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d89c3e3b0 con 0x555d89c80d40 2024-03-23T23:12:31.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.924+0000 7f3408802740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d89be0830 con 0x555d89c7bbd0 2024-03-23T23:12:31.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.924+0000 7f3408802740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d89b895d0 con 0x555d89c7b1e0 2024-03-23T23:12:31.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.925+0000 7f33ff400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7b1e0 0x555d89c7b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.925+0000 7f33ff400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7b1e0 0x555d89c7b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58088/0 (socket says 172.21.15.12:58088) 2024-03-23T23:12:31.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.925+0000 7f33ff400640 1 -- 172.21.15.12:0/336836180 learned_addr learned my addr 172.21.15.12:0/336836180 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:31.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.925+0000 7f33fea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d89c7bbd0 0x555d89c806d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.925+0000 7f33fe000640 1 --2- 172.21.15.12:0/336836180 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555d89c80d40 0x555d89c831d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.926 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.925+0000 7f33ff400640 1 -- 172.21.15.12:0/336836180 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d89c7bbd0 msgr2=0x555d89c806d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:31.926 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.925+0000 7f33ff400640 1 --2- 172.21.15.12:0/336836180 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d89c7bbd0 0x555d89c806d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.926 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.925+0000 7f33ff400640 1 -- 172.21.15.12:0/336836180 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555d89c80d40 msgr2=0x555d89c831d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:31.926 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.925+0000 7f33ff400640 1 --2- 172.21.15.12:0/336836180 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555d89c80d40 0x555d89c831d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.926 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.925+0000 7f33ff400640 1 -- 172.21.15.12:0/336836180 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d89ba12c0 con 0x555d89c7b1e0 2024-03-23T23:12:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.925+0000 7f33ff400640 1 --2- 172.21.15.12:0/336836180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7b1e0 0x555d89c7b600 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f33f8009ca0 tx=0x7f33f8009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=764d591ec6f8618e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.926+0000 7f33ebe00640 1 -- 172.21.15.12:0/336836180 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f8043020 con 0x555d89c7b1e0 2024-03-23T23:12:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.926+0000 7f33f7400640 1 --2- 172.21.15.12:0/2696232950 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f33e40a28a0 0x7f33e40a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.926+0000 7f33ebe00640 1 -- 172.21.15.12:0/336836180 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33f8003970 con 0x555d89c7b1e0 2024-03-23T23:12:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.926+0000 7f33ebe00640 1 -- 172.21.15.12:0/336836180 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f8039040 con 0x555d89c7b1e0 2024-03-23T23:12:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.926+0000 7f33f7400640 1 -- 172.21.15.12:0/2696232950 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f33f00cef70 con 0x7f34001005c0 2024-03-23T23:12:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.926+0000 7f33fea00640 1 --2- 172.21.15.12:0/2696232950 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f33e40a28a0 0x7f33e40a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.927+0000 7f3408802740 1 -- 172.21.15.12:0/336836180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7b1e0 msgr2=0x555d89c7b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.927+0000 7f3408802740 1 --2- 172.21.15.12:0/336836180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7b1e0 0x555d89c7b600 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f33f8009ca0 tx=0x7f33f8009800 comp rx=0 tx=0).stop 2024-03-23T23:12:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.927+0000 7f3408802740 1 -- 172.21.15.12:0/336836180 shutdown_connections 2024-03-23T23:12:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.927+0000 7f3408802740 1 --2- 172.21.15.12:0/336836180 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555d89c80d40 0x555d89c831d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.927+0000 7f3408802740 1 --2- 172.21.15.12:0/336836180 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d89c7bbd0 0x555d89c806d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.927+0000 7f3408802740 1 --2- 172.21.15.12:0/336836180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7b1e0 0x555d89c7b600 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.927+0000 7f3408802740 1 -- 172.21.15.12:0/336836180 >> 172.21.15.12:0/336836180 conn(0x555d89c76750 msgr2=0x555d89c76b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.927+0000 7f3408802740 1 -- 172.21.15.12:0/336836180 shutdown_connections 2024-03-23T23:12:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.927+0000 7f33fea00640 1 --2- 172.21.15.12:0/2696232950 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f33e40a28a0 0x7f33e40a4d60 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f33f0098290 tx=0x7f33f00d29b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.927+0000 7f3408802740 1 -- 172.21.15.12:0/336836180 wait complete. 2024-03-23T23:12:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.927+0000 7f3408802740 1 -- start start 2024-03-23T23:12:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.928+0000 7f3408802740 1 Processor -- start 2024-03-23T23:12:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.928+0000 7f3408802740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7b1e0 0x555d89c85330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.928+0000 7f3408802740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555d89c7bbd0 0x555d89c85870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.928+0000 7f3408802740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d89c80d40 0x555d89c8c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.928+0000 7f3408802740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d89b9ca20 con 0x555d89c7bbd0 2024-03-23T23:12:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.928+0000 7f3408802740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d89bdc8d0 con 0x555d89c80d40 2024-03-23T23:12:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.928+0000 7f3408802740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d89c83c90 con 0x555d89c7b1e0 2024-03-23T23:12:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.928+0000 7f3408802740 10 client.0 _ll_register_callbacks cb 0x555d89bd91c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:12:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.928+0000 7f33ff400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7b1e0 0x555d89c85330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.928+0000 7f33ff400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7b1e0 0x555d89c85330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58092/0 (socket says 172.21.15.12:58092) 2024-03-23T23:12:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.928+0000 7f33ff400640 1 -- 172.21.15.12:0/1058655798 learned_addr learned my addr 172.21.15.12:0/1058655798 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.928+0000 7f33fe000640 1 --2- 172.21.15.12:0/1058655798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555d89c7bbd0 0x555d89c85870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.928+0000 7f33ff400640 1 --2- 172.21.15.12:0/1058655798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d89c80d40 0x555d89c8c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.929+0000 7f33ff400640 1 -- 172.21.15.12:0/1058655798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d89c80d40 msgr2=0x555d89c8c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.929+0000 7f33ff400640 1 --2- 172.21.15.12:0/1058655798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d89c80d40 0x555d89c8c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.929+0000 7f33ff400640 1 -- 172.21.15.12:0/1058655798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555d89c7bbd0 msgr2=0x555d89c85870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.929+0000 7f33ff400640 1 --2- 172.21.15.12:0/1058655798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555d89c7bbd0 0x555d89c85870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.929+0000 7f33ff400640 1 -- 172.21.15.12:0/1058655798 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555d898f5210 con 0x555d89c7b1e0 2024-03-23T23:12:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.929+0000 7f33ff400640 1 --2- 172.21.15.12:0/1058655798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7b1e0 0x555d89c85330 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f33f80324c0 tx=0x7f33f802e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95c4b4a87350d760 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.929+0000 7f33e8c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:12:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.929+0000 7f33e8c00640 1 -- 172.21.15.12:0/1058655798 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f804d020 con 0x555d89c7b1e0 2024-03-23T23:12:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.929+0000 7f33e8c00640 1 -- 172.21.15.12:0/1058655798 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f33f80042c0 con 0x555d89c7b1e0 2024-03-23T23:12:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.929+0000 7f3408802740 1 -- 172.21.15.12:0/1058655798 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f33f8003970 con 0x555d89c7b1e0 2024-03-23T23:12:31.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.930+0000 7f33e8c00640 1 -- 172.21.15.12:0/1058655798 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33f8039070 con 0x555d89c7b1e0 2024-03-23T23:12:31.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.930+0000 7f3408802740 10 client.28318 fetch_fsmap learned FSMap version 23 2024-03-23T23:12:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.930+0000 7f3408802740 1 -- 172.21.15.12:0/1058655798 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x555d899c1620 con 0x555d89c7b1e0 2024-03-23T23:12:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.930+0000 7f33e8c00640 1 -- 172.21.15.12:0/1058655798 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33f8003970 con 0x555d89c7b1e0 2024-03-23T23:12:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.930+0000 7f33e8c00640 1 -- 172.21.15.12:0/1058655798 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f33f803fbc0 con 0x555d89c7b1e0 2024-03-23T23:12:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.930+0000 7f3408802740 10 client.28318 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:12:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.930+0000 7f3408802740 10 client.28318 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:12:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.930+0000 7f3408802740 20 client.28318 populate_metadata read hostname 'smithi012' 2024-03-23T23:12:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.930+0000 7f3408802740 1 --2- 172.21.15.12:0/1058655798 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x555d89c90da0 0x555d89c93190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.930+0000 7f3408802740 4 client.28318 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:12:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.930+0000 7f3408802740 1 -- 172.21.15.12:0/1058655798 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555d89bedd70 con 0x555d89c90da0 2024-03-23T23:12:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.930+0000 7f33fe000640 1 --2- 172.21.15.12:0/1058655798 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x555d89c90da0 0x555d89c93190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.931+0000 7f33fe000640 1 --2- 172.21.15.12:0/1058655798 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x555d89c90da0 0x555d89c93190 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=853b3e89d422c473 server_cookie=298e771dea237699 in_seq=0 out_seq=0 2024-03-23T23:12:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.931+0000 7f33e8c00640 10 client.28318 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.932+0000 7f33e8c00640 1 -- 172.21.15.12:0/1058655798 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555d89bedd70 con 0x555d89c90da0 2024-03-23T23:12:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.932+0000 7f33e8c00640 10 client.28318 handle_command_reply: tid=0 2024-03-23T23:12:31.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.932+0000 7f3408802740 1 client.28318 shutdown 2024-03-23T23:12:31.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.932+0000 7f3408802740 1 -- 172.21.15.12:0/1058655798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7b1e0 msgr2=0x555d89c85330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.932+0000 7f3408802740 1 --2- 172.21.15.12:0/1058655798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7b1e0 0x555d89c85330 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f33f80324c0 tx=0x7f33f802e040 comp rx=0 tx=0).stop 2024-03-23T23:12:31.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.932+0000 7f3408802740 1 -- 172.21.15.12:0/1058655798 shutdown_connections 2024-03-23T23:12:31.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.932+0000 7f3408802740 1 --2- 172.21.15.12:0/1058655798 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x555d89c90da0 0x555d89c93190 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.932+0000 7f3408802740 1 --2- 172.21.15.12:0/1058655798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d89c80d40 0x555d89c8c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.932+0000 7f3408802740 1 --2- 172.21.15.12:0/1058655798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555d89c7bbd0 0x555d89c85870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.932+0000 7f3408802740 1 --2- 172.21.15.12:0/1058655798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7b1e0 0x555d89c85330 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.932+0000 7f33e8c00640 0 client.28318 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:31.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.932+0000 7f3408802740 1 -- 172.21.15.12:0/1058655798 >> 172.21.15.12:0/1058655798 conn(0x555d89c76750 msgr2=0x555d89c75950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:31.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.932+0000 7f3408802740 1 -- 172.21.15.12:0/1058655798 shutdown_connections 2024-03-23T23:12:31.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.932+0000 7f3408802740 1 -- 172.21.15.12:0/1058655798 wait complete. 2024-03-23T23:12:31.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.932+0000 7f3408802740 20 client.28318 trim_cache size 0 max 16384 2024-03-23T23:12:31.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.948+0000 7f3408802740 1 Processor -- start 2024-03-23T23:12:31.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.948+0000 7f3408802740 1 -- start start 2024-03-23T23:12:31.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.948+0000 7f3408802740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d89c7b1e0 0x555d89c85650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.948+0000 7f3408802740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7bbd0 0x555d89c8c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.948+0000 7f3408802740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555d89c80d40 0x555d89c979b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.948+0000 7f3408802740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34001ad060 con 0x555d89c80d40 2024-03-23T23:12:31.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.948+0000 7f3408802740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34001aab60 con 0x555d89c7b1e0 2024-03-23T23:12:31.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.948+0000 7f3408802740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34001a8c00 con 0x555d89c7bbd0 2024-03-23T23:12:31.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.948+0000 7f33fe000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7bbd0 0x555d89c8c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.948+0000 7f33fe000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7bbd0 0x555d89c8c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58102/0 (socket says 172.21.15.12:58102) 2024-03-23T23:12:31.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.948+0000 7f33fe000640 1 -- 172.21.15.12:0/303793345 learned_addr learned my addr 172.21.15.12:0/303793345 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:31.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.948+0000 7f33ff400640 1 --2- 172.21.15.12:0/303793345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d89c7b1e0 0x555d89c85650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.949+0000 7f33ff400640 1 --2- 172.21.15.12:0/303793345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555d89c80d40 0x555d89c979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.949+0000 7f33fe000640 1 -- 172.21.15.12:0/303793345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d89c7b1e0 msgr2=0x555d89c85650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:31.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.949+0000 7f33fe000640 1 --2- 172.21.15.12:0/303793345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d89c7b1e0 0x555d89c85650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.949+0000 7f33fe000640 1 -- 172.21.15.12:0/303793345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555d89c80d40 msgr2=0x555d89c979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:31.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.949+0000 7f33fe000640 1 --2- 172.21.15.12:0/303793345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555d89c80d40 0x555d89c979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.949+0000 7f33fe000640 1 -- 172.21.15.12:0/303793345 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d89c93500 con 0x555d89c7bbd0 2024-03-23T23:12:31.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.949+0000 7f33fe000640 1 --2- 172.21.15.12:0/303793345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7bbd0 0x555d89c8c840 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f34001042c0 tx=0x7f33ec00b390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=728e394e647179f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:31.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.949+0000 7f33eb400640 1 -- 172.21.15.12:0/303793345 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec011bc0 con 0x555d89c7bbd0 2024-03-23T23:12:31.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.949+0000 7f33eb400640 1 -- 172.21.15.12:0/303793345 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33ec024030 con 0x555d89c7bbd0 2024-03-23T23:12:31.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.949+0000 7f33eb400640 1 -- 172.21.15.12:0/303793345 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec003d70 con 0x555d89c7bbd0 2024-03-23T23:12:31.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.950+0000 7f3408802740 1 -- 172.21.15.12:0/303793345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7bbd0 msgr2=0x555d89c8c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.950+0000 7f3408802740 1 --2- 172.21.15.12:0/303793345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7bbd0 0x555d89c8c840 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f34001042c0 tx=0x7f33ec00b390 comp rx=0 tx=0).stop 2024-03-23T23:12:31.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.950+0000 7f3408802740 1 -- 172.21.15.12:0/303793345 shutdown_connections 2024-03-23T23:12:31.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.950+0000 7f3408802740 1 --2- 172.21.15.12:0/303793345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555d89c80d40 0x555d89c979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.950+0000 7f3408802740 1 --2- 172.21.15.12:0/303793345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7bbd0 0x555d89c8c840 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.950+0000 7f3408802740 1 --2- 172.21.15.12:0/303793345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d89c7b1e0 0x555d89c85650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.950+0000 7f3408802740 1 -- 172.21.15.12:0/303793345 >> 172.21.15.12:0/303793345 conn(0x555d89c76750 msgr2=0x555d89c84f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:31.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.950+0000 7f3408802740 1 -- 172.21.15.12:0/303793345 shutdown_connections 2024-03-23T23:12:31.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.950+0000 7f3408802740 1 -- 172.21.15.12:0/303793345 wait complete. 2024-03-23T23:12:31.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.950+0000 7f3408802740 1 -- start start 2024-03-23T23:12:31.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.950+0000 7f3408802740 1 Processor -- start 2024-03-23T23:12:31.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.951+0000 7f3408802740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7b1e0 0x555d89c8a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.951+0000 7f3408802740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555d89c7bbd0 0x555d89c8aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.951+0000 7f3408802740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d89c80d40 0x555d89c8ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.951+0000 7f3408802740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34001a9200 con 0x555d89c7bbd0 2024-03-23T23:12:31.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.951+0000 7f3408802740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34001aa500 con 0x555d89c80d40 2024-03-23T23:12:31.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.951+0000 7f3408802740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34001a9840 con 0x555d89c7b1e0 2024-03-23T23:12:31.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.951+0000 7f3408802740 10 client.0 _ll_register_callbacks cb 0x7f33ec00bb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:12:31.954 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.951+0000 7f33ff400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7b1e0 0x555d89c8a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.951+0000 7f33ff400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7b1e0 0x555d89c8a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58116/0 (socket says 172.21.15.12:58116) 2024-03-23T23:12:31.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.951+0000 7f33fe000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555d89c7bbd0 0x555d89c8aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.951+0000 7f33ff400640 1 -- 172.21.15.12:0/202545411 learned_addr learned my addr 172.21.15.12:0/202545411 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:31.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.951+0000 7f33ff400640 1 --2- 172.21.15.12:0/202545411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d89c80d40 0x555d89c8ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.952+0000 7f33ff400640 1 -- 172.21.15.12:0/202545411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d89c80d40 msgr2=0x555d89c8ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:31.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.952+0000 7f33ff400640 1 --2- 172.21.15.12:0/202545411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d89c80d40 0x555d89c8ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.952+0000 7f33ff400640 1 -- 172.21.15.12:0/202545411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555d89c7bbd0 msgr2=0x555d89c8aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:31.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.952+0000 7f33ff400640 1 --2- 172.21.15.12:0/202545411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555d89c7bbd0 0x555d89c8aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.952+0000 7f33ff400640 1 -- 172.21.15.12:0/202545411 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555d89c8c010 con 0x555d89c7b1e0 2024-03-23T23:12:31.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.952+0000 7f33ff400640 1 --2- 172.21.15.12:0/202545411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7b1e0 0x555d89c8a4d0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f33f80325f0 tx=0x7f33f8004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a449239fea61a04f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:31.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.952+0000 7f33e8c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:12:31.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.952+0000 7f33e8c00640 1 -- 172.21.15.12:0/202545411 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f804d040 con 0x555d89c7b1e0 2024-03-23T23:12:31.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.952+0000 7f33e8c00640 1 -- 172.21.15.12:0/202545411 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f33f803fa30 con 0x555d89c7b1e0 2024-03-23T23:12:31.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.952+0000 7f3408802740 1 -- 172.21.15.12:0/202545411 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f33ec024030 con 0x555d89c7b1e0 2024-03-23T23:12:31.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.953+0000 7f33e8c00640 1 -- 172.21.15.12:0/202545411 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33f8043040 con 0x555d89c7b1e0 2024-03-23T23:12:31.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.953+0000 7f3408802740 10 client.28324 fetch_fsmap learned FSMap version 23 2024-03-23T23:12:31.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.953+0000 7f3408802740 1 -- 172.21.15.12:0/202545411 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x555d89cecad0 con 0x555d89c7b1e0 2024-03-23T23:12:31.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.953+0000 7f33e8c00640 1 -- 172.21.15.12:0/202545411 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33ec024030 con 0x555d89c7b1e0 2024-03-23T23:12:31.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.953+0000 7f33e8c00640 1 -- 172.21.15.12:0/202545411 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f33f80040b0 con 0x555d89c7b1e0 2024-03-23T23:12:31.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.953+0000 7f3408802740 10 client.28324 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:12:31.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.953+0000 7f3408802740 10 client.28324 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:12:31.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.953+0000 7f3408802740 20 client.28324 populate_metadata read hostname 'smithi012' 2024-03-23T23:12:31.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.953+0000 7f3408802740 1 --2- 172.21.15.12:0/202545411 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x555d89ced2a0 0x555d89cef690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:31.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.953+0000 7f3408802740 4 client.28324 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:12:31.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.953+0000 7f3408802740 1 -- 172.21.15.12:0/202545411 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555d89be80c0 con 0x555d89ced2a0 2024-03-23T23:12:31.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.953+0000 7f33fe000640 1 --2- 172.21.15.12:0/202545411 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x555d89ced2a0 0x555d89cef690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:31.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.954+0000 7f33fe000640 1 --2- 172.21.15.12:0/202545411 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x555d89ced2a0 0x555d89cef690 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68e7004687ebdbba server_cookie=d85b4863c0718753 in_seq=0 out_seq=0 2024-03-23T23:12:31.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.954+0000 7f33e8c00640 10 client.28324 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:31.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.967+0000 7f33e8c00640 1 -- 172.21.15.12:0/202545411 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555d89be80c0 con 0x555d89ced2a0 2024-03-23T23:12:31.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.967+0000 7f33e8c00640 10 client.28324 handle_command_reply: tid=0 2024-03-23T23:12:31.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.967+0000 7f3408802740 1 client.28324 shutdown 2024-03-23T23:12:31.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.967+0000 7f3408802740 1 -- 172.21.15.12:0/202545411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7b1e0 msgr2=0x555d89c8a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.967+0000 7f3408802740 1 --2- 172.21.15.12:0/202545411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7b1e0 0x555d89c8a4d0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f33f80325f0 tx=0x7f33f8004040 comp rx=0 tx=0).stop 2024-03-23T23:12:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.967+0000 7f3408802740 1 -- 172.21.15.12:0/202545411 shutdown_connections 2024-03-23T23:12:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.967+0000 7f3408802740 1 --2- 172.21.15.12:0/202545411 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x555d89ced2a0 0x555d89cef690 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.967+0000 7f3408802740 1 --2- 172.21.15.12:0/202545411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d89c80d40 0x555d89c8ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.967+0000 7f3408802740 1 --2- 172.21.15.12:0/202545411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555d89c7bbd0 0x555d89c8aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.967+0000 7f3408802740 1 --2- 172.21.15.12:0/202545411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d89c7b1e0 0x555d89c8a4d0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.967+0000 7f3408802740 1 -- 172.21.15.12:0/202545411 >> 172.21.15.12:0/202545411 conn(0x555d89c76750 msgr2=0x555d89c750c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.968+0000 7f33e8c00640 0 client.28324 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.968+0000 7f3408802740 1 -- 172.21.15.12:0/202545411 shutdown_connections 2024-03-23T23:12:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.968+0000 7f3408802740 1 -- 172.21.15.12:0/202545411 wait complete. 2024-03-23T23:12:31.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.968+0000 7f3408802740 20 client.28324 trim_cache size 0 max 16384 2024-03-23T23:12:31.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:12:31.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:12:31.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:12:31.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:12:31.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.969+0000 7f3405c00640 1 -- 172.21.15.12:0/2696232950 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f33e40a28a0 msgr2=0x7f33e40a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.969+0000 7f3405c00640 1 --2- 172.21.15.12:0/2696232950 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f33e40a28a0 0x7f33e40a4d60 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f33f0098290 tx=0x7f33f00d29b0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.969+0000 7f3405c00640 1 -- 172.21.15.12:0/2696232950 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34001005c0 msgr2=0x7f3400104550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:31.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.969+0000 7f3405c00640 1 --2- 172.21.15.12:0/2696232950 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34001005c0 0x7f3400104550 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f33f000e9c0 tx=0x7f33f000ee80 comp rx=0 tx=0).stop 2024-03-23T23:12:31.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.969+0000 7f3405c00640 1 -- 172.21.15.12:0/2696232950 shutdown_connections 2024-03-23T23:12:31.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.969+0000 7f3405c00640 1 --2- 172.21.15.12:0/2696232950 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f33e40a28a0 0x7f33e40a4d60 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.969+0000 7f3405c00640 1 --2- 172.21.15.12:0/2696232950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3400102170 0x7f34001030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.969+0000 7f3405c00640 1 --2- 172.21.15.12:0/2696232950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34001017b0 0x7f3400102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.969+0000 7f3405c00640 1 --2- 172.21.15.12:0/2696232950 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34001005c0 0x7f3400104550 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:31.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.969+0000 7f3405c00640 1 -- 172.21.15.12:0/2696232950 >> 172.21.15.12:0/2696232950 conn(0x7f34000fbb30 msgr2=0x7f34000fd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:31.970 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.969+0000 7f3405c00640 1 -- 172.21.15.12:0/2696232950 shutdown_connections 2024-03-23T23:12:31.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:31.969+0000 7f3405c00640 1 -- 172.21.15.12:0/2696232950 wait complete. 2024-03-23T23:12:32.373 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-23T23:12:32.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:12:32.374 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-23T23:12:32.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.612+0000 7fd442800640 1 -- 172.21.15.12:0/934975281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd43c1049b0 msgr2=0x7fd43c104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:32.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.612+0000 7fd442800640 1 --2- 172.21.15.12:0/934975281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd43c1049b0 0x7fd43c104e30 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fd42c009900 tx=0x7fd42c02bc30 comp rx=0 tx=0).stop 2024-03-23T23:12:32.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.612+0000 7fd442800640 1 -- 172.21.15.12:0/934975281 shutdown_connections 2024-03-23T23:12:32.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.612+0000 7fd442800640 1 --2- 172.21.15.12:0/934975281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd43c105370 0x7fd43c10bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.612+0000 7fd442800640 1 --2- 172.21.15.12:0/934975281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd43c1049b0 0x7fd43c104e30 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.612+0000 7fd442800640 1 --2- 172.21.15.12:0/934975281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd43c1037c0 0x7fd43c103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.613+0000 7fd442800640 1 -- 172.21.15.12:0/934975281 >> 172.21.15.12:0/934975281 conn(0x7fd43c0fed50 msgr2=0x7fd43c101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:32.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.613+0000 7fd442800640 1 -- 172.21.15.12:0/934975281 shutdown_connections 2024-03-23T23:12:32.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.613+0000 7fd442800640 1 -- 172.21.15.12:0/934975281 wait complete. 2024-03-23T23:12:32.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.614+0000 7fd442800640 1 Processor -- start 2024-03-23T23:12:32.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.614+0000 7fd442800640 1 -- start start 2024-03-23T23:12:32.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.615+0000 7fd442800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd43c1037c0 0x7fd43c199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.615+0000 7fd442800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd43c1049b0 0x7fd43c199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.615+0000 7fd442800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd43c105370 0x7fd43c1a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.615+0000 7fd442800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c10f490 con 0x7fd43c105370 2024-03-23T23:12:32.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.615+0000 7fd442800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c10f310 con 0x7fd43c1049b0 2024-03-23T23:12:32.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.615+0000 7fd442800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c10f610 con 0x7fd43c1037c0 2024-03-23T23:12:32.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.615+0000 7fd441400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd43c1037c0 0x7fd43c199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.615+0000 7fd440a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd43c1049b0 0x7fd43c199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.615+0000 7fd441400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd43c1037c0 0x7fd43c199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58136/0 (socket says 172.21.15.12:58136) 2024-03-23T23:12:32.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.615+0000 7fd441400640 1 -- 172.21.15.12:0/4007037461 learned_addr learned my addr 172.21.15.12:0/4007037461 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:32.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.615+0000 7fd441e00640 1 --2- 172.21.15.12:0/4007037461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd43c105370 0x7fd43c1a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.616+0000 7fd441400640 1 -- 172.21.15.12:0/4007037461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd43c1049b0 msgr2=0x7fd43c199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:32.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.616+0000 7fd441400640 1 --2- 172.21.15.12:0/4007037461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd43c1049b0 0x7fd43c199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.616+0000 7fd441400640 1 -- 172.21.15.12:0/4007037461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd43c105370 msgr2=0x7fd43c1a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:32.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.616+0000 7fd441400640 1 --2- 172.21.15.12:0/4007037461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd43c105370 0x7fd43c1a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.616+0000 7fd441400640 1 -- 172.21.15.12:0/4007037461 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd43c1a16d0 con 0x7fd43c1037c0 2024-03-23T23:12:32.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.616+0000 7fd441400640 1 --2- 172.21.15.12:0/4007037461 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd43c1037c0 0x7fd43c199ab0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fd43000e9c0 tx=0x7fd43000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:32.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.616+0000 7fd43a000640 1 -- 172.21.15.12:0/4007037461 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43000cd80 con 0x7fd43c1037c0 2024-03-23T23:12:32.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.616+0000 7fd43a000640 1 -- 172.21.15.12:0/4007037461 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd4300035f0 con 0x7fd43c1037c0 2024-03-23T23:12:32.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.616+0000 7fd442800640 1 -- 172.21.15.12:0/4007037461 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd43c1a19c0 con 0x7fd43c1037c0 2024-03-23T23:12:32.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.616+0000 7fd43a000640 1 -- 172.21.15.12:0/4007037461 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd430010690 con 0x7fd43c1037c0 2024-03-23T23:12:32.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.616+0000 7fd442800640 1 -- 172.21.15.12:0/4007037461 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd43c1a1f30 con 0x7fd43c1037c0 2024-03-23T23:12:32.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.619+0000 7fd43a000640 1 -- 172.21.15.12:0/4007037461 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd4300037a0 con 0x7fd43c1037c0 2024-03-23T23:12:32.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.619+0000 7fd446a1c740 1 Processor -- start 2024-03-23T23:12:32.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.620+0000 7fd446a1c740 1 -- start start 2024-03-23T23:12:32.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.620+0000 7fd446a1c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ce084f2b0 0x562ce084f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.620+0000 7fd446a1c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce084fca0 0x562ce08547a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.620+0000 7fd446a1c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ce0854e10 0x562ce08572a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.620+0000 7fd446a1c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ce07b4eb0 con 0x562ce084f2b0 2024-03-23T23:12:32.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.620+0000 7fd446a1c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ce075da60 con 0x562ce0854e10 2024-03-23T23:12:32.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.620+0000 7fd446a1c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ce07b0f50 con 0x562ce084fca0 2024-03-23T23:12:32.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.620+0000 7fd441400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce084fca0 0x562ce08547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.620+0000 7fd441e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ce084f2b0 0x562ce084f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.620+0000 7fd441400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce084fca0 0x562ce08547a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58146/0 (socket says 172.21.15.12:58146) 2024-03-23T23:12:32.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.620+0000 7fd441400640 1 -- 172.21.15.12:0/1443807990 learned_addr learned my addr 172.21.15.12:0/1443807990 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:32.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.620+0000 7fd440a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ce0854e10 0x562ce08572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.621+0000 7fd441400640 1 -- 172.21.15.12:0/1443807990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ce0854e10 msgr2=0x562ce08572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:32.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.621+0000 7fd441400640 1 --2- 172.21.15.12:0/1443807990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ce0854e10 0x562ce08572a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.621+0000 7fd441400640 1 -- 172.21.15.12:0/1443807990 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ce084f2b0 msgr2=0x562ce084f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:32.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.621+0000 7fd441400640 1 --2- 172.21.15.12:0/1443807990 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ce084f2b0 0x562ce084f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.621+0000 7fd441400640 1 -- 172.21.15.12:0/1443807990 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ce05917e0 con 0x562ce084fca0 2024-03-23T23:12:32.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.621+0000 7fd441400640 1 --2- 172.21.15.12:0/1443807990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce084fca0 0x562ce08547a0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd4300d2da0 tx=0x7fd4300d74b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ad9e04840c1523f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:32.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.621+0000 7fd43a000640 1 --2- 172.21.15.12:0/4007037461 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd4240a2880 0x7fd4240a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.621+0000 7fd42a600640 1 -- 172.21.15.12:0/1443807990 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4300e2070 con 0x562ce084fca0 2024-03-23T23:12:32.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.621+0000 7fd441e00640 1 --2- 172.21.15.12:0/4007037461 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd4240a2880 0x7fd4240a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.622+0000 7fd43a000640 1 -- 172.21.15.12:0/4007037461 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd4300cee90 con 0x7fd43c1037c0 2024-03-23T23:12:32.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.622+0000 7fd42a600640 1 -- 172.21.15.12:0/1443807990 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd43009bee0 con 0x562ce084fca0 2024-03-23T23:12:32.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.622+0000 7fd42a600640 1 -- 172.21.15.12:0/1443807990 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4300a3f80 con 0x562ce084fca0 2024-03-23T23:12:32.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.622+0000 7fd441e00640 1 --2- 172.21.15.12:0/4007037461 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd4240a2880 0x7fd4240a4d40 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x562ce0850500 tx=0x7fd434009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:32.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.622+0000 7fd446a1c740 1 -- 172.21.15.12:0/1443807990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce084fca0 msgr2=0x562ce08547a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:32.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.622+0000 7fd446a1c740 1 --2- 172.21.15.12:0/1443807990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce084fca0 0x562ce08547a0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd4300d2da0 tx=0x7fd4300d74b0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.622+0000 7fd446a1c740 1 -- 172.21.15.12:0/1443807990 shutdown_connections 2024-03-23T23:12:32.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.622+0000 7fd446a1c740 1 --2- 172.21.15.12:0/1443807990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ce0854e10 0x562ce08572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.622+0000 7fd446a1c740 1 --2- 172.21.15.12:0/1443807990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce084fca0 0x562ce08547a0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.622+0000 7fd446a1c740 1 --2- 172.21.15.12:0/1443807990 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ce084f2b0 0x562ce084f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.622+0000 7fd446a1c740 1 -- 172.21.15.12:0/1443807990 >> 172.21.15.12:0/1443807990 conn(0x562ce084a820 msgr2=0x562ce084ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:32.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.622+0000 7fd446a1c740 1 -- 172.21.15.12:0/1443807990 shutdown_connections 2024-03-23T23:12:32.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.622+0000 7fd446a1c740 1 -- 172.21.15.12:0/1443807990 wait complete. 2024-03-23T23:12:32.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.623+0000 7fd446a1c740 1 -- start start 2024-03-23T23:12:32.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.623+0000 7fd446a1c740 1 Processor -- start 2024-03-23T23:12:32.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.624+0000 7fd446a1c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce084f2b0 0x562ce0859470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.624+0000 7fd446a1c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ce084fca0 0x562ce08599b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.624+0000 7fd446a1c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ce0854e10 0x562ce08609f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.624+0000 7fd446a1c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ce0857dc0 con 0x562ce084fca0 2024-03-23T23:12:32.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.624+0000 7fd446a1c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ce0770eb0 con 0x562ce0854e10 2024-03-23T23:12:32.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.624+0000 7fd446a1c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ce0857f40 con 0x562ce084f2b0 2024-03-23T23:12:32.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.624+0000 7fd446a1c740 10 client.0 _ll_register_callbacks cb 0x562ce0765150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:12:32.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.624+0000 7fd441400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce084f2b0 0x562ce0859470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.624+0000 7fd441400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce084f2b0 0x562ce0859470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58152/0 (socket says 172.21.15.12:58152) 2024-03-23T23:12:32.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.624+0000 7fd441e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ce0854e10 0x562ce08609f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.624+0000 7fd441400640 1 -- 172.21.15.12:0/4272018044 learned_addr learned my addr 172.21.15.12:0/4272018044 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:32.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.624+0000 7fd440a00640 1 --2- 172.21.15.12:0/4272018044 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ce084fca0 0x562ce08599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.624+0000 7fd441400640 1 -- 172.21.15.12:0/4272018044 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ce0854e10 msgr2=0x562ce08609f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:32.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.624+0000 7fd441400640 1 --2- 172.21.15.12:0/4272018044 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ce0854e10 0x562ce08609f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.624+0000 7fd441400640 1 -- 172.21.15.12:0/4272018044 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ce084fca0 msgr2=0x562ce08599b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:32.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.624+0000 7fd441400640 1 --2- 172.21.15.12:0/4272018044 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ce084fca0 0x562ce08599b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.624+0000 7fd441400640 1 -- 172.21.15.12:0/4272018044 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562ce05f1f70 con 0x562ce084f2b0 2024-03-23T23:12:32.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.624+0000 7fd441e00640 1 --2- 172.21.15.12:0/4272018044 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ce0854e10 0x562ce08609f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:12:32.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.624+0000 7fd440a00640 1 --2- 172.21.15.12:0/4272018044 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ce084fca0 0x562ce08599b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:12:32.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.625+0000 7fd441400640 1 --2- 172.21.15.12:0/4272018044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce084f2b0 0x562ce0859470 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd43009b490 tx=0x7fd4300d26a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3eda31c25289ad7e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:32.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.625+0000 7fd41ea00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:12:32.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.625+0000 7fd41ea00640 1 -- 172.21.15.12:0/4272018044 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd430097a80 con 0x562ce084f2b0 2024-03-23T23:12:32.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.625+0000 7fd41ea00640 1 -- 172.21.15.12:0/4272018044 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd43005f4f0 con 0x562ce084f2b0 2024-03-23T23:12:32.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.625+0000 7fd446a1c740 1 -- 172.21.15.12:0/4272018044 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd43009bee0 con 0x562ce084f2b0 2024-03-23T23:12:32.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.625+0000 7fd41ea00640 1 -- 172.21.15.12:0/4272018044 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd43009b5b0 con 0x562ce084f2b0 2024-03-23T23:12:32.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.625+0000 7fd446a1c740 10 client.28336 fetch_fsmap learned FSMap version 23 2024-03-23T23:12:32.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.625+0000 7fd446a1c740 1 -- 172.21.15.12:0/4272018044 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x562ce0489170 con 0x562ce084f2b0 2024-03-23T23:12:32.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.625+0000 7fd41ea00640 1 -- 172.21.15.12:0/4272018044 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd43009bee0 con 0x562ce084f2b0 2024-03-23T23:12:32.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.625+0000 7fd41ea00640 1 -- 172.21.15.12:0/4272018044 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fd4300cc9c0 con 0x562ce084f2b0 2024-03-23T23:12:32.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.626+0000 7fd446a1c740 10 client.28336 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:12:32.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.626+0000 7fd446a1c740 10 client.28336 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:12:32.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.626+0000 7fd446a1c740 20 client.28336 populate_metadata read hostname 'smithi012' 2024-03-23T23:12:32.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.626+0000 7fd446a1c740 1 --2- 172.21.15.12:0/4272018044 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562ce0864ee0 0x562ce08672d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.626+0000 7fd446a1c740 4 client.28336 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:12:32.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.626+0000 7fd446a1c740 1 -- 172.21.15.12:0/4272018044 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562ce07c23f0 con 0x562ce0864ee0 2024-03-23T23:12:32.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.626+0000 7fd440a00640 1 --2- 172.21.15.12:0/4272018044 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562ce0864ee0 0x562ce08672d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.626+0000 7fd440a00640 1 --2- 172.21.15.12:0/4272018044 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562ce0864ee0 0x562ce08672d0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbd86a160adaba3b server_cookie=4090d9f9c96af772 in_seq=0 out_seq=0 2024-03-23T23:12:32.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.626+0000 7fd41ea00640 10 client.28336 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:32.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.627+0000 7fd41ea00640 1 -- 172.21.15.12:0/4272018044 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562ce07c23f0 con 0x562ce0864ee0 2024-03-23T23:12:32.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.627+0000 7fd41ea00640 10 client.28336 handle_command_reply: tid=0 2024-03-23T23:12:32.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.627+0000 7fd446a1c740 1 client.28336 shutdown 2024-03-23T23:12:32.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.628+0000 7fd446a1c740 1 -- 172.21.15.12:0/4272018044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce084f2b0 msgr2=0x562ce0859470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:32.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.628+0000 7fd446a1c740 1 --2- 172.21.15.12:0/4272018044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce084f2b0 0x562ce0859470 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd43009b490 tx=0x7fd4300d26a0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.628+0000 7fd446a1c740 1 -- 172.21.15.12:0/4272018044 shutdown_connections 2024-03-23T23:12:32.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.628+0000 7fd446a1c740 1 --2- 172.21.15.12:0/4272018044 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562ce0864ee0 0x562ce08672d0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.628+0000 7fd446a1c740 1 --2- 172.21.15.12:0/4272018044 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ce0854e10 0x562ce08609f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.628+0000 7fd446a1c740 1 --2- 172.21.15.12:0/4272018044 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ce084fca0 0x562ce08599b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.628+0000 7fd41ea00640 0 client.28336 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:32.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.628+0000 7fd446a1c740 1 --2- 172.21.15.12:0/4272018044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce084f2b0 0x562ce0859470 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.628+0000 7fd446a1c740 1 -- 172.21.15.12:0/4272018044 >> 172.21.15.12:0/4272018044 conn(0x562ce084a820 msgr2=0x562ce0849a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:32.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.628+0000 7fd446a1c740 1 -- 172.21.15.12:0/4272018044 shutdown_connections 2024-03-23T23:12:32.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.628+0000 7fd446a1c740 1 -- 172.21.15.12:0/4272018044 wait complete. 2024-03-23T23:12:32.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.628+0000 7fd446a1c740 20 client.28336 trim_cache size 0 max 16384 2024-03-23T23:12:32.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.646+0000 7fd446a1c740 1 Processor -- start 2024-03-23T23:12:32.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.646+0000 7fd446a1c740 1 -- start start 2024-03-23T23:12:32.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.647+0000 7fd446a1c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce084f2b0 0x562ce0849900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.647+0000 7fd446a1c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ce084fca0 0x562ce086b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.647+0000 7fd446a1c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ce0854e10 0x562ce086be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.647+0000 7fd446a1c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c1ad060 con 0x562ce084fca0 2024-03-23T23:12:32.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.647+0000 7fd446a1c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c1aab60 con 0x562ce0854e10 2024-03-23T23:12:32.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.647+0000 7fd446a1c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c1a8c00 con 0x562ce084f2b0 2024-03-23T23:12:32.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.647+0000 7fd441400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce084f2b0 0x562ce0849900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.647+0000 7fd441e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ce0854e10 0x562ce086be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.647+0000 7fd441e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ce0854e10 0x562ce086be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:38168/0 (socket says 172.21.15.12:38168) 2024-03-23T23:12:32.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.647+0000 7fd441400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce084f2b0 0x562ce0849900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58160/0 (socket says 172.21.15.12:58160) 2024-03-23T23:12:32.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.647+0000 7fd441e00640 1 -- 172.21.15.12:0/2728116804 learned_addr learned my addr 172.21.15.12:0/2728116804 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:32.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.647+0000 7fd440a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ce084fca0 0x562ce086b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.647+0000 7fd441400640 1 -- 172.21.15.12:0/2728116804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ce0854e10 msgr2=0x562ce086be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:32.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.647+0000 7fd441400640 1 --2- 172.21.15.12:0/2728116804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ce0854e10 0x562ce086be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.647+0000 7fd441400640 1 -- 172.21.15.12:0/2728116804 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ce084fca0 msgr2=0x562ce086b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:32.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.647+0000 7fd441400640 1 --2- 172.21.15.12:0/2728116804 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ce084fca0 0x562ce086b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.647+0000 7fd441400640 1 -- 172.21.15.12:0/2728116804 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ce0851720 con 0x562ce084f2b0 2024-03-23T23:12:32.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.647+0000 7fd441e00640 1 --2- 172.21.15.12:0/2728116804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ce0854e10 0x562ce086be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:12:32.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.648+0000 7fd441400640 1 --2- 172.21.15.12:0/2728116804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce084f2b0 0x562ce0849900 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd4300d7e20 tx=0x7fd430099cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3c9d0289ddd9a67 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:32.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.648+0000 7fd429c00640 1 -- 172.21.15.12:0/2728116804 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd430097a80 con 0x562ce084f2b0 2024-03-23T23:12:32.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.648+0000 7fd429c00640 1 -- 172.21.15.12:0/2728116804 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd430086220 con 0x562ce084f2b0 2024-03-23T23:12:32.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.648+0000 7fd429c00640 1 -- 172.21.15.12:0/2728116804 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43009b140 con 0x562ce084f2b0 2024-03-23T23:12:32.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.649+0000 7fd446a1c740 1 -- 172.21.15.12:0/2728116804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce084f2b0 msgr2=0x562ce0849900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:32.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.649+0000 7fd446a1c740 1 --2- 172.21.15.12:0/2728116804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce084f2b0 0x562ce0849900 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd4300d7e20 tx=0x7fd430099cf0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.649+0000 7fd446a1c740 1 -- 172.21.15.12:0/2728116804 shutdown_connections 2024-03-23T23:12:32.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.649+0000 7fd446a1c740 1 --2- 172.21.15.12:0/2728116804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ce0854e10 0x562ce086be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.649+0000 7fd446a1c740 1 --2- 172.21.15.12:0/2728116804 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ce084fca0 0x562ce086b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.649+0000 7fd446a1c740 1 --2- 172.21.15.12:0/2728116804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce084f2b0 0x562ce0849900 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.649+0000 7fd446a1c740 1 -- 172.21.15.12:0/2728116804 >> 172.21.15.12:0/2728116804 conn(0x562ce084a820 msgr2=0x562ce08491f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:32.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.649+0000 7fd446a1c740 1 -- 172.21.15.12:0/2728116804 shutdown_connections 2024-03-23T23:12:32.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.649+0000 7fd446a1c740 1 -- 172.21.15.12:0/2728116804 wait complete. 2024-03-23T23:12:32.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.649+0000 7fd446a1c740 1 -- start start 2024-03-23T23:12:32.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.649+0000 7fd446a1c740 1 Processor -- start 2024-03-23T23:12:32.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.650+0000 7fd446a1c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ce084f2b0 0x562ce085fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.650+0000 7fd446a1c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ce084fca0 0x562ce08601b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.650+0000 7fd446a1c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce0854e10 0x562ce085e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.650+0000 7fd446a1c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c1a9200 con 0x562ce084f2b0 2024-03-23T23:12:32.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.650+0000 7fd446a1c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c1aa500 con 0x562ce084fca0 2024-03-23T23:12:32.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.650+0000 7fd446a1c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c1a9840 con 0x562ce0854e10 2024-03-23T23:12:32.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.650+0000 7fd446a1c740 10 client.0 _ll_register_callbacks cb 0x7fd4300e0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:12:32.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.650+0000 7fd441e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce0854e10 0x562ce085e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.650+0000 7fd441400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ce084f2b0 0x562ce085fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.650+0000 7fd440a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ce084fca0 0x562ce08601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.650+0000 7fd441e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce0854e10 0x562ce085e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58172/0 (socket says 172.21.15.12:58172) 2024-03-23T23:12:32.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.650+0000 7fd440a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ce084fca0 0x562ce08601b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:38172/0 (socket says 172.21.15.12:38172) 2024-03-23T23:12:32.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.650+0000 7fd441e00640 1 -- 172.21.15.12:0/774051403 learned_addr learned my addr 172.21.15.12:0/774051403 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:32.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.650+0000 7fd441400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ce084f2b0 0x562ce085fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59602/0 (socket says 172.21.15.12:59602) 2024-03-23T23:12:32.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.651+0000 7fd441e00640 1 -- 172.21.15.12:0/774051403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ce084fca0 msgr2=0x562ce08601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:32.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.651+0000 7fd441e00640 1 --2- 172.21.15.12:0/774051403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ce084fca0 0x562ce08601b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.651+0000 7fd441e00640 1 -- 172.21.15.12:0/774051403 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ce084f2b0 msgr2=0x562ce085fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:32.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.651+0000 7fd441e00640 1 --2- 172.21.15.12:0/774051403 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ce084f2b0 0x562ce085fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.651+0000 7fd441e00640 1 -- 172.21.15.12:0/774051403 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562ce085eb80 con 0x562ce0854e10 2024-03-23T23:12:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.651+0000 7fd441400640 1 --2- 172.21.15.12:0/774051403 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ce084f2b0 0x562ce085fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:12:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.651+0000 7fd441e00640 1 --2- 172.21.15.12:0/774051403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce0854e10 0x562ce085e350 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd434009fd0 tx=0x7fd434004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=734b9a5083d38b47 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.651+0000 7fd41ea00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:12:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.651+0000 7fd41ea00640 1 -- 172.21.15.12:0/774051403 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd434020320 con 0x562ce0854e10 2024-03-23T23:12:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.651+0000 7fd446a1c740 1 -- 172.21.15.12:0/774051403 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd430086220 con 0x562ce0854e10 2024-03-23T23:12:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.651+0000 7fd41ea00640 1 -- 172.21.15.12:0/774051403 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd43401d960 con 0x562ce0854e10 2024-03-23T23:12:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.652+0000 7fd41ea00640 1 -- 172.21.15.12:0/774051403 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd43401db60 con 0x562ce0854e10 2024-03-23T23:12:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.652+0000 7fd446a1c740 10 client.28342 fetch_fsmap learned FSMap version 23 2024-03-23T23:12:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.652+0000 7fd446a1c740 1 -- 172.21.15.12:0/774051403 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x562ce08c9a60 con 0x562ce0854e10 2024-03-23T23:12:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.652+0000 7fd41ea00640 1 -- 172.21.15.12:0/774051403 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd430086220 con 0x562ce0854e10 2024-03-23T23:12:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.652+0000 7fd41ea00640 1 -- 172.21.15.12:0/774051403 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fd43402a460 con 0x562ce0854e10 2024-03-23T23:12:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.652+0000 7fd446a1c740 10 client.28342 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:12:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.652+0000 7fd446a1c740 10 client.28342 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:12:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.652+0000 7fd446a1c740 20 client.28342 populate_metadata read hostname 'smithi012' 2024-03-23T23:12:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.652+0000 7fd446a1c740 1 --2- 172.21.15.12:0/774051403 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562ce08ca420 0x562ce08cc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.652+0000 7fd446a1c740 4 client.28342 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-23T23:12:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.652+0000 7fd446a1c740 1 -- 172.21.15.12:0/774051403 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x562ce07bc740 con 0x562ce08ca420 2024-03-23T23:12:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.652+0000 7fd441400640 1 --2- 172.21.15.12:0/774051403 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562ce08ca420 0x562ce08cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.653+0000 7fd441400640 1 --2- 172.21.15.12:0/774051403 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562ce08ca420 0x562ce08cc840 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3d872afbf450323 server_cookie=60cf89f12f95de36 in_seq=0 out_seq=0 2024-03-23T23:12:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.653+0000 7fd41ea00640 10 client.28342 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.654+0000 7fd41ea00640 1 -- 172.21.15.12:0/774051403 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x562ce07bc740 con 0x562ce08ca420 2024-03-23T23:12:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.654+0000 7fd41ea00640 10 client.28342 handle_command_reply: tid=0 2024-03-23T23:12:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.654+0000 7fd446a1c740 1 client.28342 shutdown 2024-03-23T23:12:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.654+0000 7fd446a1c740 1 -- 172.21.15.12:0/774051403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce0854e10 msgr2=0x562ce085e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.654+0000 7fd446a1c740 1 --2- 172.21.15.12:0/774051403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce0854e10 0x562ce085e350 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd434009fd0 tx=0x7fd434004a40 comp rx=0 tx=0).stop 2024-03-23T23:12:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.655+0000 7fd446a1c740 1 -- 172.21.15.12:0/774051403 shutdown_connections 2024-03-23T23:12:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.655+0000 7fd446a1c740 1 --2- 172.21.15.12:0/774051403 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562ce08ca420 0x562ce08cc840 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.655+0000 7fd446a1c740 1 --2- 172.21.15.12:0/774051403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ce0854e10 0x562ce085e350 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.655+0000 7fd446a1c740 1 --2- 172.21.15.12:0/774051403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ce084fca0 0x562ce08601b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.655+0000 7fd41ea00640 0 client.28342 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.655+0000 7fd446a1c740 1 --2- 172.21.15.12:0/774051403 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562ce084f2b0 0x562ce085fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.655+0000 7fd446a1c740 1 -- 172.21.15.12:0/774051403 >> 172.21.15.12:0/774051403 conn(0x562ce084a820 msgr2=0x562ce0849a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.655+0000 7fd446a1c740 1 -- 172.21.15.12:0/774051403 shutdown_connections 2024-03-23T23:12:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.655+0000 7fd446a1c740 1 -- 172.21.15.12:0/774051403 wait complete. 2024-03-23T23:12:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.655+0000 7fd446a1c740 20 client.28342 trim_cache size 0 max 16384 2024-03-23T23:12:32.658 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:12:32.658 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-03-23T23:12:32.658 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "95658e9f-7d38-4862-b1e8-5c1a8880ff32", 2024-03-23T23:12:32.658 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-03-23T23:12:32.659 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:12:32.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.655+0000 7fd442800640 1 -- 172.21.15.12:0/4007037461 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd4240a2880 msgr2=0x7fd4240a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:32.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.656+0000 7fd442800640 1 --2- 172.21.15.12:0/4007037461 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd4240a2880 0x7fd4240a4d40 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x562ce0850500 tx=0x7fd434009500 comp rx=0 tx=0).stop 2024-03-23T23:12:32.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.656+0000 7fd442800640 1 -- 172.21.15.12:0/4007037461 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd43c1037c0 msgr2=0x7fd43c199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:32.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.656+0000 7fd442800640 1 --2- 172.21.15.12:0/4007037461 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd43c1037c0 0x7fd43c199ab0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fd43000e9c0 tx=0x7fd43000ee80 comp rx=0 tx=0).stop 2024-03-23T23:12:32.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.656+0000 7fd442800640 1 -- 172.21.15.12:0/4007037461 shutdown_connections 2024-03-23T23:12:32.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.656+0000 7fd442800640 1 --2- 172.21.15.12:0/4007037461 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd4240a2880 0x7fd4240a4d40 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.656+0000 7fd442800640 1 --2- 172.21.15.12:0/4007037461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd43c105370 0x7fd43c1a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.656+0000 7fd442800640 1 --2- 172.21.15.12:0/4007037461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd43c1049b0 0x7fd43c199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.656+0000 7fd442800640 1 --2- 172.21.15.12:0/4007037461 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd43c1037c0 0x7fd43c199ab0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.656+0000 7fd442800640 1 -- 172.21.15.12:0/4007037461 >> 172.21.15.12:0/4007037461 conn(0x7fd43c0fed50 msgr2=0x7fd43c107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:32.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.656+0000 7fd442800640 1 -- 172.21.15.12:0/4007037461 shutdown_connections 2024-03-23T23:12:32.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.656+0000 7fd442800640 1 -- 172.21.15.12:0/4007037461 wait complete. 2024-03-23T23:12:32.676 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:12:32.676 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:12:32.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.902+0000 7fd8bca00640 1 -- 172.21.15.12:0/736795810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8b8109570 msgr2=0x7fd8b810ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:32.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.902+0000 7fd8bca00640 1 --2- 172.21.15.12:0/736795810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8b8109570 0x7fd8b810ba00 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0009be0 tx=0x7fd8b002bef0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.902+0000 7fd8bca00640 1 -- 172.21.15.12:0/736795810 shutdown_connections 2024-03-23T23:12:32.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.902+0000 7fd8bca00640 1 --2- 172.21.15.12:0/736795810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8b8109570 0x7fd8b810ba00 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.902+0000 7fd8bca00640 1 --2- 172.21.15.12:0/736795810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8b8074100 0x7fd8b8074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.902+0000 7fd8bca00640 1 --2- 172.21.15.12:0/736795810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8b80756e0 0x7fd8b8073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.902+0000 7fd8bca00640 1 -- 172.21.15.12:0/736795810 >> 172.21.15.12:0/736795810 conn(0x7fd8b80fcc40 msgr2=0x7fd8b80ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:32.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.903+0000 7fd8bca00640 1 -- 172.21.15.12:0/736795810 shutdown_connections 2024-03-23T23:12:32.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.903+0000 7fd8bca00640 1 -- 172.21.15.12:0/736795810 wait complete. 2024-03-23T23:12:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.903+0000 7fd8bca00640 1 Processor -- start 2024-03-23T23:12:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.904+0000 7fd8bca00640 1 -- start start 2024-03-23T23:12:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.904+0000 7fd8bca00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8b8074100 0x7fd8b8199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.904+0000 7fd8bca00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8b80756e0 0x7fd8b819a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.904+0000 7fd8bca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8b8109570 0x7fd8b81a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.904+0000 7fd8bca00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b810f540 con 0x7fd8b80756e0 2024-03-23T23:12:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.904+0000 7fd8bca00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b810f3c0 con 0x7fd8b8109570 2024-03-23T23:12:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.904+0000 7fd8bca00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b810f6c0 con 0x7fd8b8074100 2024-03-23T23:12:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.904+0000 7fd8b6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8b8109570 0x7fd8b81a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.904+0000 7fd8b5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8b8074100 0x7fd8b8199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.904+0000 7fd8b6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8b8109570 0x7fd8b81a1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:38178/0 (socket says 172.21.15.12:38178) 2024-03-23T23:12:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.904+0000 7fd8b6800640 1 -- 172.21.15.12:0/3546344118 learned_addr learned my addr 172.21.15.12:0/3546344118 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:32.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.904+0000 7fd8afe00640 1 --2- 172.21.15.12:0/3546344118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8b80756e0 0x7fd8b819a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.905+0000 7fd8b6800640 1 -- 172.21.15.12:0/3546344118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8b8074100 msgr2=0x7fd8b8199b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:32.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.905+0000 7fd8b6800640 1 --2- 172.21.15.12:0/3546344118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8b8074100 0x7fd8b8199b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.905+0000 7fd8b6800640 1 -- 172.21.15.12:0/3546344118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8b80756e0 msgr2=0x7fd8b819a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:32.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.905+0000 7fd8b6800640 1 --2- 172.21.15.12:0/3546344118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8b80756e0 0x7fd8b819a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.905+0000 7fd8b6800640 1 -- 172.21.15.12:0/3546344118 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b81a1770 con 0x7fd8b8109570 2024-03-23T23:12:32.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.905+0000 7fd8afe00640 1 --2- 172.21.15.12:0/3546344118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8b80756e0 0x7fd8b819a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:12:32.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.905+0000 7fd8b6800640 1 --2- 172.21.15.12:0/3546344118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8b8109570 0x7fd8b81a1100 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0036040 tx=0x7fd8b00337b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:32.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.905+0000 7fd8aea00640 1 -- 172.21.15.12:0/3546344118 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0043020 con 0x7fd8b8109570 2024-03-23T23:12:32.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.905+0000 7fd8bca00640 1 -- 172.21.15.12:0/3546344118 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b81a1a00 con 0x7fd8b8109570 2024-03-23T23:12:32.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.906+0000 7fd8aea00640 1 -- 172.21.15.12:0/3546344118 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8b002ca60 con 0x7fd8b8109570 2024-03-23T23:12:32.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.906+0000 7fd8aea00640 1 -- 172.21.15.12:0/3546344118 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0034690 con 0x7fd8b8109570 2024-03-23T23:12:32.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.906+0000 7fd8bca00640 1 -- 172.21.15.12:0/3546344118 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b81a1f50 con 0x7fd8b8109570 2024-03-23T23:12:32.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.909+0000 7fd8aea00640 1 -- 172.21.15.12:0/3546344118 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd8b0046020 con 0x7fd8b8109570 2024-03-23T23:12:32.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.909+0000 7fd8bf771740 1 Processor -- start 2024-03-23T23:12:32.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.909+0000 7fd8bf771740 1 -- start start 2024-03-23T23:12:32.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.909+0000 7fd8bf771740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56080711ae80 0x56080711b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.909+0000 7fd8bf771740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56080711b870 0x560807120370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.909+0000 7fd8bf771740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608071209e0 0x560807122e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.910+0000 7fd8bf771740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x560807080250 con 0x5608071209e0 2024-03-23T23:12:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.910+0000 7fd8bf771740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x560807028a30 con 0x56080711b870 2024-03-23T23:12:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.910+0000 7fd8bf771740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x56080707c2f0 con 0x56080711ae80 2024-03-23T23:12:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.910+0000 7fd8b5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56080711b870 0x560807120370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.910+0000 7fd8b5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56080711b870 0x560807120370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:38180/0 (socket says 172.21.15.12:38180) 2024-03-23T23:12:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.910+0000 7fd8b5e00640 1 -- 172.21.15.12:0/2645027726 learned_addr learned my addr 172.21.15.12:0/2645027726 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.910+0000 7fd8afe00640 1 --2- 172.21.15.12:0/2645027726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56080711ae80 0x56080711b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.910+0000 7fd8afe00640 1 --2- 172.21.15.12:0/2645027726 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608071209e0 0x560807122e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.910+0000 7fd8b5e00640 1 -- 172.21.15.12:0/2645027726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56080711ae80 msgr2=0x56080711b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.910+0000 7fd8b5e00640 1 --2- 172.21.15.12:0/2645027726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56080711ae80 0x56080711b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.910+0000 7fd8b5e00640 1 -- 172.21.15.12:0/2645027726 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608071209e0 msgr2=0x560807122e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:32.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.910+0000 7fd8b5e00640 1 --2- 172.21.15.12:0/2645027726 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608071209e0 0x560807122e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.910+0000 7fd8b5e00640 1 -- 172.21.15.12:0/2645027726 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560807013690 con 0x56080711b870 2024-03-23T23:12:32.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.910+0000 7fd8aea00640 1 --2- 172.21.15.12:0/3546344118 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd89c0a2800 0x7fd89c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.911+0000 7fd8aea00640 1 -- 172.21.15.12:0/3546344118 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd8b00f1110 con 0x7fd8b8109570 2024-03-23T23:12:32.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.911+0000 7fd8b5e00640 1 --2- 172.21.15.12:0/2645027726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56080711b870 0x560807120370 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009cc0 tx=0x7fd8a8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e85075ce538e37fa server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:32.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.911+0000 7fd8afe00640 1 --2- 172.21.15.12:0/3546344118 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd89c0a2800 0x7fd89c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.911+0000 7fd8a7000640 1 -- 172.21.15.12:0/2645027726 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8005220 con 0x56080711b870 2024-03-23T23:12:32.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.911+0000 7fd8a7000640 1 -- 172.21.15.12:0/2645027726 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8a8004c80 con 0x56080711b870 2024-03-23T23:12:32.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.911+0000 7fd8a7000640 1 -- 172.21.15.12:0/2645027726 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8019870 con 0x56080711b870 2024-03-23T23:12:32.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.912+0000 7fd8bf771740 1 -- 172.21.15.12:0/2645027726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56080711b870 msgr2=0x560807120370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:32.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.912+0000 7fd8afe00640 1 --2- 172.21.15.12:0/3546344118 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd89c0a2800 0x7fd89c0a4cc0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x56080711c0d0 tx=0x7fd8a0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:32.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.912+0000 7fd8bf771740 1 --2- 172.21.15.12:0/2645027726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56080711b870 0x560807120370 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009cc0 tx=0x7fd8a8011810 comp rx=0 tx=0).stop 2024-03-23T23:12:32.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.912+0000 7fd8bf771740 1 -- 172.21.15.12:0/2645027726 shutdown_connections 2024-03-23T23:12:32.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.912+0000 7fd8bf771740 1 --2- 172.21.15.12:0/2645027726 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5608071209e0 0x560807122e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.912+0000 7fd8bf771740 1 --2- 172.21.15.12:0/2645027726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56080711b870 0x560807120370 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.912+0000 7fd8bf771740 1 --2- 172.21.15.12:0/2645027726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56080711ae80 0x56080711b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.912+0000 7fd8bf771740 1 -- 172.21.15.12:0/2645027726 >> 172.21.15.12:0/2645027726 conn(0x5608070e3200 msgr2=0x5608071167f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:32.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.912+0000 7fd8bf771740 1 -- 172.21.15.12:0/2645027726 shutdown_connections 2024-03-23T23:12:32.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.912+0000 7fd8bf771740 1 -- 172.21.15.12:0/2645027726 wait complete. 2024-03-23T23:12:32.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.912+0000 7fd8bf771740 1 -- start start 2024-03-23T23:12:32.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.912+0000 7fd8bf771740 1 Processor -- start 2024-03-23T23:12:32.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.913+0000 7fd8bf771740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56080711ae80 0x560807124dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.913+0000 7fd8bf771740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56080711b870 0x560807125310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.913+0000 7fd8bf771740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608071209e0 0x56080712c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.913+0000 7fd8bf771740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x560807123990 con 0x56080711b870 2024-03-23T23:12:32.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.913+0000 7fd8bf771740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56080703be80 con 0x5608071209e0 2024-03-23T23:12:32.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.913+0000 7fd8bf771740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x560807123b10 con 0x56080711ae80 2024-03-23T23:12:32.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.913+0000 7fd8bf771740 10 client.0 _ll_register_callbacks cb 0x56080704e130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:12:32.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.913+0000 7fd8b5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56080711ae80 0x560807124dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.913+0000 7fd8b6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608071209e0 0x56080712c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.913+0000 7fd8afe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56080711b870 0x560807125310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.913+0000 7fd8b5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56080711ae80 0x560807124dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58198/0 (socket says 172.21.15.12:58198) 2024-03-23T23:12:32.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.913+0000 7fd8b5e00640 1 -- 172.21.15.12:0/1685659204 learned_addr learned my addr 172.21.15.12:0/1685659204 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:32.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.914+0000 7fd8b5e00640 1 -- 172.21.15.12:0/1685659204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608071209e0 msgr2=0x56080712c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:32.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.914+0000 7fd8b5e00640 1 --2- 172.21.15.12:0/1685659204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608071209e0 0x56080712c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.914+0000 7fd8b5e00640 1 -- 172.21.15.12:0/1685659204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56080711b870 msgr2=0x560807125310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:32.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.914+0000 7fd8b5e00640 1 --2- 172.21.15.12:0/1685659204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56080711b870 0x560807125310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.914+0000 7fd8b5e00640 1 -- 172.21.15.12:0/1685659204 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560806e5e7d0 con 0x56080711ae80 2024-03-23T23:12:32.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.914+0000 7fd8b5e00640 1 --2- 172.21.15.12:0/1685659204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56080711ae80 0x560807124dd0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8011d10 tx=0x7fd8a80050e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4e098929702cec1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:32.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.914+0000 7fd89b400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:12:32.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.914+0000 7fd89b400640 1 -- 172.21.15.12:0/1685659204 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8029070 con 0x56080711ae80 2024-03-23T23:12:32.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.914+0000 7fd89b400640 1 -- 172.21.15.12:0/1685659204 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd8a80199c0 con 0x56080711ae80 2024-03-23T23:12:32.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.914+0000 7fd8bf771740 1 -- 172.21.15.12:0/1685659204 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd8a8004c80 con 0x56080711ae80 2024-03-23T23:12:32.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.914+0000 7fd89b400640 1 -- 172.21.15.12:0/1685659204 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8a8035050 con 0x56080711ae80 2024-03-23T23:12:32.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.914+0000 7fd8bf771740 10 client.28354 fetch_fsmap learned FSMap version 23 2024-03-23T23:12:32.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.914+0000 7fd8bf771740 1 -- 172.21.15.12:0/1685659204 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x560806d943f0 con 0x56080711ae80 2024-03-23T23:12:32.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.915+0000 7fd89b400640 1 -- 172.21.15.12:0/1685659204 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8a8004c80 con 0x56080711ae80 2024-03-23T23:12:32.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.915+0000 7fd89b400640 1 -- 172.21.15.12:0/1685659204 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fd8a8023910 con 0x56080711ae80 2024-03-23T23:12:32.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.915+0000 7fd8bf771740 10 client.28354 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:12:32.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.915+0000 7fd8bf771740 10 client.28354 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:12:32.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.915+0000 7fd8bf771740 20 client.28354 populate_metadata read hostname 'smithi012' 2024-03-23T23:12:32.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.915+0000 7fd8bf771740 1 --2- 172.21.15.12:0/1685659204 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x560807130960 0x560807132de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.915+0000 7fd8bf771740 4 client.28354 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:12:32.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.915+0000 7fd8bf771740 1 -- 172.21.15.12:0/1685659204 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56080708d790 con 0x560807130960 2024-03-23T23:12:32.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.915+0000 7fd8afe00640 1 --2- 172.21.15.12:0/1685659204 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x560807130960 0x560807132de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.916+0000 7fd8afe00640 1 --2- 172.21.15.12:0/1685659204 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x560807130960 0x560807132de0 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7557e80be755d89 server_cookie=78f0e50cbad9a10f in_seq=0 out_seq=0 2024-03-23T23:12:32.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.916+0000 7fd89b400640 10 client.28354 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:32.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.917+0000 7fd89b400640 1 -- 172.21.15.12:0/1685659204 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56080708d790 con 0x560807130960 2024-03-23T23:12:32.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.917+0000 7fd89b400640 10 client.28354 handle_command_reply: tid=0 2024-03-23T23:12:32.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.917+0000 7fd8bf771740 1 client.28354 shutdown 2024-03-23T23:12:32.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.917+0000 7fd8bf771740 1 -- 172.21.15.12:0/1685659204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56080711ae80 msgr2=0x560807124dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:32.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.917+0000 7fd8bf771740 1 --2- 172.21.15.12:0/1685659204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56080711ae80 0x560807124dd0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8011d10 tx=0x7fd8a80050e0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.917+0000 7fd8bf771740 1 -- 172.21.15.12:0/1685659204 shutdown_connections 2024-03-23T23:12:32.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.917+0000 7fd8bf771740 1 --2- 172.21.15.12:0/1685659204 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x560807130960 0x560807132de0 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.917+0000 7fd8bf771740 1 --2- 172.21.15.12:0/1685659204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608071209e0 0x56080712c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.917+0000 7fd8bf771740 1 --2- 172.21.15.12:0/1685659204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56080711b870 0x560807125310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.917+0000 7fd89b400640 0 client.28354 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:32.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.917+0000 7fd8bf771740 1 --2- 172.21.15.12:0/1685659204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56080711ae80 0x560807124dd0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.917+0000 7fd8bf771740 1 -- 172.21.15.12:0/1685659204 >> 172.21.15.12:0/1685659204 conn(0x5608070e3200 msgr2=0x560807115a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:32.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.917+0000 7fd8bf771740 1 -- 172.21.15.12:0/1685659204 shutdown_connections 2024-03-23T23:12:32.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.918+0000 7fd8bf771740 1 -- 172.21.15.12:0/1685659204 wait complete. 2024-03-23T23:12:32.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.918+0000 7fd8bf771740 20 client.28354 trim_cache size 0 max 16384 2024-03-23T23:12:32.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.935+0000 7fd8bf771740 1 Processor -- start 2024-03-23T23:12:32.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.936+0000 7fd8bf771740 1 -- start start 2024-03-23T23:12:32.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.936+0000 7fd8bf771740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56080711ae80 0x56080711c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.936+0000 7fd8bf771740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56080711b870 0x56080712c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.936+0000 7fd8bf771740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608071209e0 0x5608071243f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.936+0000 7fd8bf771740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b81ad160 con 0x56080711ae80 2024-03-23T23:12:32.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.936+0000 7fd8bf771740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b81aac60 con 0x56080711b870 2024-03-23T23:12:32.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.936+0000 7fd8bf771740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b81a8d00 con 0x5608071209e0 2024-03-23T23:12:32.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.936+0000 7fd8b5e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56080711ae80 0x56080711c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.936+0000 7fd8b6800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608071209e0 0x5608071243f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.936+0000 7fd8b5e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56080711ae80 0x56080711c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59660/0 (socket says 172.21.15.12:59660) 2024-03-23T23:12:32.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.936+0000 7fd8b5e00640 1 -- 172.21.15.12:0/3040431645 learned_addr learned my addr 172.21.15.12:0/3040431645 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:32.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.936+0000 7fd8b6800640 1 --2- 172.21.15.12:0/3040431645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608071209e0 0x5608071243f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58206/0 (socket says 172.21.15.12:58206) 2024-03-23T23:12:32.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.936+0000 7fd8afe00640 1 --2- 172.21.15.12:0/3040431645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56080711b870 0x56080712c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.937+0000 7fd8b6800640 1 -- 172.21.15.12:0/3040431645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56080711b870 msgr2=0x56080712c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:32.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.937+0000 7fd8b6800640 1 --2- 172.21.15.12:0/3040431645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56080711b870 0x56080712c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.937+0000 7fd8b6800640 1 -- 172.21.15.12:0/3040431645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56080711ae80 msgr2=0x56080711c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:32.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.937+0000 7fd8b6800640 1 --2- 172.21.15.12:0/3040431645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56080711ae80 0x56080711c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.937+0000 7fd8b6800640 1 -- 172.21.15.12:0/3040431645 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560806d540f0 con 0x5608071209e0 2024-03-23T23:12:32.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.937+0000 7fd8b6800640 1 --2- 172.21.15.12:0/3040431645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608071209e0 0x5608071243f0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd8b00eec20 tx=0x7fd8b0004990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e98aa47774536cc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:32.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.937+0000 7fd8a6600640 1 -- 172.21.15.12:0/3040431645 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b00d5020 con 0x5608071209e0 2024-03-23T23:12:32.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.937+0000 7fd8a6600640 1 -- 172.21.15.12:0/3040431645 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8b00be1c0 con 0x5608071209e0 2024-03-23T23:12:32.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.937+0000 7fd8a6600640 1 -- 172.21.15.12:0/3040431645 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0003d80 con 0x5608071209e0 2024-03-23T23:12:32.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.938+0000 7fd8bf771740 1 -- 172.21.15.12:0/3040431645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608071209e0 msgr2=0x5608071243f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:32.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.938+0000 7fd8bf771740 1 --2- 172.21.15.12:0/3040431645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608071209e0 0x5608071243f0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd8b00eec20 tx=0x7fd8b0004990 comp rx=0 tx=0).stop 2024-03-23T23:12:32.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.938+0000 7fd8bf771740 1 -- 172.21.15.12:0/3040431645 shutdown_connections 2024-03-23T23:12:32.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.938+0000 7fd8bf771740 1 --2- 172.21.15.12:0/3040431645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608071209e0 0x5608071243f0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.938+0000 7fd8bf771740 1 --2- 172.21.15.12:0/3040431645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56080711b870 0x56080712c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.938+0000 7fd8bf771740 1 --2- 172.21.15.12:0/3040431645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56080711ae80 0x56080711c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.938+0000 7fd8bf771740 1 -- 172.21.15.12:0/3040431645 >> 172.21.15.12:0/3040431645 conn(0x5608070e3200 msgr2=0x56080711c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:32.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.938+0000 7fd8bf771740 1 -- 172.21.15.12:0/3040431645 shutdown_connections 2024-03-23T23:12:32.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.938+0000 7fd8bf771740 1 -- 172.21.15.12:0/3040431645 wait complete. 2024-03-23T23:12:32.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.938+0000 7fd8bf771740 1 -- start start 2024-03-23T23:12:32.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.938+0000 7fd8bf771740 1 Processor -- start 2024-03-23T23:12:32.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.939+0000 7fd8bf771740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56080711ae80 0x560807129c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.939+0000 7fd8bf771740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56080711b870 0x56080712a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.939+0000 7fd8bf771740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608071209e0 0x56080712b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.939+0000 7fd8bf771740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b81a9300 con 0x56080711b870 2024-03-23T23:12:32.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.939+0000 7fd8bf771740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b81aa600 con 0x5608071209e0 2024-03-23T23:12:32.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.939+0000 7fd8bf771740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b81a9940 con 0x56080711ae80 2024-03-23T23:12:32.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.939+0000 7fd8bf771740 10 client.0 _ll_register_callbacks cb 0x7fd8a0009710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:12:32.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.939+0000 7fd8b5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56080711ae80 0x560807129c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.939+0000 7fd8afe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56080711b870 0x56080712a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.939+0000 7fd8b6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608071209e0 0x56080712b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.939+0000 7fd8b5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56080711ae80 0x560807129c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:58220/0 (socket says 172.21.15.12:58220) 2024-03-23T23:12:32.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.939+0000 7fd8b5e00640 1 -- 172.21.15.12:0/903105663 learned_addr learned my addr 172.21.15.12:0/903105663 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:32.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.939+0000 7fd8b5e00640 1 -- 172.21.15.12:0/903105663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608071209e0 msgr2=0x56080712b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:32.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.939+0000 7fd8b5e00640 1 --2- 172.21.15.12:0/903105663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608071209e0 0x56080712b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.939+0000 7fd8b5e00640 1 -- 172.21.15.12:0/903105663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56080711b870 msgr2=0x56080712a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:32.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.939+0000 7fd8b5e00640 1 --2- 172.21.15.12:0/903105663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56080711b870 0x56080712a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.939+0000 7fd8b5e00640 1 -- 172.21.15.12:0/903105663 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56080712b780 con 0x56080711ae80 2024-03-23T23:12:32.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.940+0000 7fd8b5e00640 1 --2- 172.21.15.12:0/903105663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56080711ae80 0x560807129c40 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8006010 tx=0x7fd8a8005970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e68d952c18466158 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:32.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.940+0000 7fd89b400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:12:32.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.940+0000 7fd89b400640 1 -- 172.21.15.12:0/903105663 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8029040 con 0x56080711ae80 2024-03-23T23:12:32.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.940+0000 7fd8bf771740 1 -- 172.21.15.12:0/903105663 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd8b00be1c0 con 0x56080711ae80 2024-03-23T23:12:32.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.940+0000 7fd89b400640 1 -- 172.21.15.12:0/903105663 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd8a80197c0 con 0x56080711ae80 2024-03-23T23:12:32.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.940+0000 7fd89b400640 1 -- 172.21.15.12:0/903105663 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8a8004c80 con 0x56080711ae80 2024-03-23T23:12:32.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.940+0000 7fd8bf771740 10 client.28360 fetch_fsmap learned FSMap version 23 2024-03-23T23:12:32.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.940+0000 7fd8bf771740 1 -- 172.21.15.12:0/903105663 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56080718c3c0 con 0x56080711ae80 2024-03-23T23:12:32.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.941+0000 7fd89b400640 1 -- 172.21.15.12:0/903105663 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8b00be1c0 con 0x56080711ae80 2024-03-23T23:12:32.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.941+0000 7fd89b400640 1 -- 172.21.15.12:0/903105663 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fd8a8023640 con 0x56080711ae80 2024-03-23T23:12:32.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.941+0000 7fd8bf771740 10 client.28360 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:12:32.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.941+0000 7fd8bf771740 10 client.28360 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:12:32.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.941+0000 7fd8bf771740 20 client.28360 populate_metadata read hostname 'smithi012' 2024-03-23T23:12:32.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.941+0000 7fd8bf771740 1 --2- 172.21.15.12:0/903105663 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56080718cb60 0x56080718ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:32.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.941+0000 7fd8bf771740 4 client.28360 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:12:32.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.941+0000 7fd8bf771740 1 -- 172.21.15.12:0/903105663 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560807087ae0 con 0x56080718cb60 2024-03-23T23:12:32.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.941+0000 7fd8afe00640 1 --2- 172.21.15.12:0/903105663 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56080718cb60 0x56080718ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:32.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.942+0000 7fd8afe00640 1 --2- 172.21.15.12:0/903105663 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56080718cb60 0x56080718ef80 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=389f2ea6d2e3aea7 server_cookie=c78f4ba18322de5f in_seq=0 out_seq=0 2024-03-23T23:12:32.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.942+0000 7fd89b400640 10 client.28360 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:32.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.942+0000 7fd89b400640 1 -- 172.21.15.12:0/903105663 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x560807087ae0 con 0x56080718cb60 2024-03-23T23:12:32.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.942+0000 7fd89b400640 10 client.28360 handle_command_reply: tid=0 2024-03-23T23:12:32.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.942+0000 7fd8bf771740 1 client.28360 shutdown 2024-03-23T23:12:32.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.942+0000 7fd8bf771740 1 -- 172.21.15.12:0/903105663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56080711ae80 msgr2=0x560807129c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:32.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.942+0000 7fd8bf771740 1 --2- 172.21.15.12:0/903105663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56080711ae80 0x560807129c40 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8006010 tx=0x7fd8a8005970 comp rx=0 tx=0).stop 2024-03-23T23:12:32.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.942+0000 7fd8bf771740 1 -- 172.21.15.12:0/903105663 shutdown_connections 2024-03-23T23:12:32.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.942+0000 7fd8bf771740 1 --2- 172.21.15.12:0/903105663 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56080718cb60 0x56080718ef80 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.942+0000 7fd8bf771740 1 --2- 172.21.15.12:0/903105663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5608071209e0 0x56080712b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.942+0000 7fd8bf771740 1 --2- 172.21.15.12:0/903105663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56080711b870 0x56080712a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.942+0000 7fd8bf771740 1 --2- 172.21.15.12:0/903105663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56080711ae80 0x560807129c40 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.942+0000 7fd89b400640 0 client.28360 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:32.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.942+0000 7fd8bf771740 1 -- 172.21.15.12:0/903105663 >> 172.21.15.12:0/903105663 conn(0x5608070e3200 msgr2=0x560807115a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:32.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.942+0000 7fd8bf771740 1 -- 172.21.15.12:0/903105663 shutdown_connections 2024-03-23T23:12:32.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.942+0000 7fd8bf771740 1 -- 172.21.15.12:0/903105663 wait complete. 2024-03-23T23:12:32.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.942+0000 7fd8bf771740 20 client.28360 trim_cache size 0 max 16384 2024-03-23T23:12:32.946 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:12:32.946 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T23:12:32.946 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-03-23T23:12:32.947 INFO:teuthology.orchestra.run.smithi012.stdout: "95658e9f-7d38-4862-b1e8-5c1a8880ff32": { 2024-03-23T23:12:32.947 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-03-23T23:12:32.947 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "95658e9f-7d38-4862-b1e8-5c1a8880ff32", 2024-03-23T23:12:32.947 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-03-23T23:12:32.947 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:12:32.947 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:12:32.947 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:12:32.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.943+0000 7fd8bca00640 1 -- 172.21.15.12:0/3546344118 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd89c0a2800 msgr2=0x7fd89c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:32.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.943+0000 7fd8bca00640 1 --2- 172.21.15.12:0/3546344118 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd89c0a2800 0x7fd89c0a4cc0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x56080711c0d0 tx=0x7fd8a0005f90 comp rx=0 tx=0).stop 2024-03-23T23:12:32.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.943+0000 7fd8bca00640 1 -- 172.21.15.12:0/3546344118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8b8109570 msgr2=0x7fd8b81a1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:32.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.943+0000 7fd8bca00640 1 --2- 172.21.15.12:0/3546344118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8b8109570 0x7fd8b81a1100 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0036040 tx=0x7fd8b00337b0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.943+0000 7fd8bca00640 1 -- 172.21.15.12:0/3546344118 shutdown_connections 2024-03-23T23:12:32.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.943+0000 7fd8bca00640 1 --2- 172.21.15.12:0/3546344118 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fd89c0a2800 0x7fd89c0a4cc0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.943+0000 7fd8bca00640 1 --2- 172.21.15.12:0/3546344118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8b8109570 0x7fd8b81a1100 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.943+0000 7fd8bca00640 1 --2- 172.21.15.12:0/3546344118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd8b80756e0 0x7fd8b819a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.943+0000 7fd8bca00640 1 --2- 172.21.15.12:0/3546344118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8b8074100 0x7fd8b8199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:32.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.943+0000 7fd8bca00640 1 -- 172.21.15.12:0/3546344118 >> 172.21.15.12:0/3546344118 conn(0x7fd8b80fcc40 msgr2=0x7fd8b8105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:32.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.943+0000 7fd8bca00640 1 -- 172.21.15.12:0/3546344118 shutdown_connections 2024-03-23T23:12:32.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:12:32.943+0000 7fd8bca00640 1 -- 172.21.15.12:0/3546344118 wait complete. 2024-03-23T23:12:32.962 INFO:tasks.cephfs.filesystem:scrub status for tag:95658e9f-7d38-4862-b1e8-5c1a8880ff32 - {'path': '/', 'tag': '95658e9f-7d38-4862-b1e8-5c1a8880ff32', 'options': 'recursive,force'} 2024-03-23T23:12:33.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:33 smithi012 ceph-mon[31695]: pgmap v712: 97 pgs: 97 active+clean; 3.3 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:12:33.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:33 smithi012 ceph-mon[31695]: scrub queued for path: / 2024-03-23T23:12:33.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:33 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:12:33.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:33 smithi012 ceph-mon[31695]: scrub summary: active paths [/] 2024-03-23T23:12:33.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:33 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:12:33.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:33 smithi173 ceph-mon[36116]: pgmap v712: 97 pgs: 97 active+clean; 3.3 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:12:33.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:33 smithi173 ceph-mon[36116]: scrub queued for path: / 2024-03-23T23:12:33.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:33 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:12:33.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:33 smithi173 ceph-mon[36116]: scrub summary: active paths [/] 2024-03-23T23:12:33.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:33 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:12:33.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:33 smithi203 ceph-mon[36693]: pgmap v712: 97 pgs: 97 active+clean; 3.3 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:12:33.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:33 smithi203 ceph-mon[36693]: scrub queued for path: / 2024-03-23T23:12:33.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:33 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:12:33.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:33 smithi203 ceph-mon[36693]: scrub summary: active paths [/] 2024-03-23T23:12:33.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:33 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:12:35.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:35 smithi012 ceph-mon[31695]: pgmap v713: 97 pgs: 97 active+clean; 3.3 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:12:35.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:35 smithi173 ceph-mon[36116]: pgmap v713: 97 pgs: 97 active+clean; 3.3 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:12:35.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:35 smithi203 ceph-mon[36693]: pgmap v713: 97 pgs: 97 active+clean; 3.3 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:12:36.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:36 smithi012 ceph-mon[31695]: scrub summary: idle 2024-03-23T23:12:36.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:36 smithi173 ceph-mon[36116]: scrub summary: idle 2024-03-23T23:12:36.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:36 smithi203 ceph-mon[36693]: scrub summary: idle 2024-03-23T23:12:37.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:37 smithi012 ceph-mon[31695]: pgmap v714: 97 pgs: 97 active+clean; 3.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:12:37.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:37 smithi173 ceph-mon[36116]: pgmap v714: 97 pgs: 97 active+clean; 3.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:12:37.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:37 smithi203 ceph-mon[36693]: pgmap v714: 97 pgs: 97 active+clean; 3.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:12:39.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:39 smithi012 ceph-mon[31695]: pgmap v715: 97 pgs: 97 active+clean; 3.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:12:39.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:39 smithi173 ceph-mon[36116]: pgmap v715: 97 pgs: 97 active+clean; 3.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:12:39.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:39 smithi203 ceph-mon[36693]: pgmap v715: 97 pgs: 97 active+clean; 3.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:12:41.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:41 smithi012 ceph-mon[31695]: pgmap v716: 97 pgs: 97 active+clean; 3.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:12:41.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:41 smithi173 ceph-mon[36116]: pgmap v716: 97 pgs: 97 active+clean; 3.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:12:41.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:41 smithi203 ceph-mon[36693]: pgmap v716: 97 pgs: 97 active+clean; 3.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:12:43.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:43 smithi012 ceph-mon[31695]: pgmap v717: 97 pgs: 97 active+clean; 3.4 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:12:43.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:43 smithi173 ceph-mon[36116]: pgmap v717: 97 pgs: 97 active+clean; 3.4 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:12:43.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:43 smithi203 ceph-mon[36693]: pgmap v717: 97 pgs: 97 active+clean; 3.4 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:12:45.117 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.117+0000 7fae54c00640 1 -- 172.21.15.12:0/3744911095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae50102710 msgr2=0x7fae50102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:45.118 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.117+0000 7fae54c00640 1 --2- 172.21.15.12:0/3744911095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae50102710 0x7fae50102b90 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fae38009be0 tx=0x7fae3802bef0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.118 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.117+0000 7fae54c00640 1 -- 172.21.15.12:0/3744911095 shutdown_connections 2024-03-23T23:12:45.118 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.117+0000 7fae54c00640 1 --2- 172.21.15.12:0/3744911095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae501030d0 0x7fae50109960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.118 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.117+0000 7fae54c00640 1 --2- 172.21.15.12:0/3744911095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae50102710 0x7fae50102b90 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.118 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.117+0000 7fae54c00640 1 --2- 172.21.15.12:0/3744911095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae50101520 0x7fae50101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.118 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.117+0000 7fae54c00640 1 -- 172.21.15.12:0/3744911095 >> 172.21.15.12:0/3744911095 conn(0x7fae500fcab0 msgr2=0x7fae500feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:45.118 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.118+0000 7fae54c00640 1 -- 172.21.15.12:0/3744911095 shutdown_connections 2024-03-23T23:12:45.118 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.118+0000 7fae54c00640 1 -- 172.21.15.12:0/3744911095 wait complete. 2024-03-23T23:12:45.119 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.119+0000 7fae54c00640 1 Processor -- start 2024-03-23T23:12:45.119 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.119+0000 7fae54c00640 1 -- start start 2024-03-23T23:12:45.119 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.119+0000 7fae54c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae50101520 0x7fae50199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.119+0000 7fae54c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae50102710 0x7fae50199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.119+0000 7fae54c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae501030d0 0x7fae501a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.120+0000 7fae54c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5010d200 con 0x7fae50101520 2024-03-23T23:12:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.120+0000 7fae54c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5010d080 con 0x7fae50102710 2024-03-23T23:12:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.120+0000 7fae54c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5010d380 con 0x7fae501030d0 2024-03-23T23:12:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.120+0000 7fae4f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae50101520 0x7fae50199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.120+0000 7fae4fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae501030d0 0x7fae501a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.120+0000 7fae4ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae50102710 0x7fae50199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.120+0000 7fae4f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae50101520 0x7fae50199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40068/0 (socket says 172.21.15.12:40068) 2024-03-23T23:12:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.120+0000 7fae4f400640 1 -- 172.21.15.12:0/3235846407 learned_addr learned my addr 172.21.15.12:0/3235846407 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:45.121 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.120+0000 7fae4f400640 1 -- 172.21.15.12:0/3235846407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae501030d0 msgr2=0x7fae501a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:45.121 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.120+0000 7fae4f400640 1 --2- 172.21.15.12:0/3235846407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae501030d0 0x7fae501a0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.121 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.120+0000 7fae4f400640 1 -- 172.21.15.12:0/3235846407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae50102710 msgr2=0x7fae50199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:45.121 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.120+0000 7fae4f400640 1 --2- 172.21.15.12:0/3235846407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae50102710 0x7fae50199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.122 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.120+0000 7fae4f400640 1 -- 172.21.15.12:0/3235846407 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae501a1640 con 0x7fae50101520 2024-03-23T23:12:45.122 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.120+0000 7fae4fe00640 1 --2- 172.21.15.12:0/3235846407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae501030d0 0x7fae501a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:12:45.122 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.121+0000 7fae4f400640 1 --2- 172.21.15.12:0/3235846407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae50101520 0x7fae50199a20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fae4000e9c0 tx=0x7fae4000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:45.122 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.121+0000 7fae47e00640 1 -- 172.21.15.12:0/3235846407 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4000cd80 con 0x7fae50101520 2024-03-23T23:12:45.122 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.121+0000 7fae47e00640 1 -- 172.21.15.12:0/3235846407 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae400035f0 con 0x7fae50101520 2024-03-23T23:12:45.122 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.121+0000 7fae54c00640 1 -- 172.21.15.12:0/3235846407 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae501a1930 con 0x7fae50101520 2024-03-23T23:12:45.122 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.121+0000 7fae47e00640 1 -- 172.21.15.12:0/3235846407 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae40010690 con 0x7fae50101520 2024-03-23T23:12:45.122 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.121+0000 7fae54c00640 1 -- 172.21.15.12:0/3235846407 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae501a1e50 con 0x7fae50101520 2024-03-23T23:12:45.124 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.124+0000 7fae47e00640 1 -- 172.21.15.12:0/3235846407 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fae400037a0 con 0x7fae50101520 2024-03-23T23:12:45.124 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.124+0000 7fae58f9d740 1 Processor -- start 2024-03-23T23:12:45.124 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.125+0000 7fae58f9d740 1 -- start start 2024-03-23T23:12:45.125 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.125+0000 7fae58f9d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf93f0 0x561e73bf9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:45.125 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.125+0000 7fae58f9d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e73bf9de0 0x561e73bfe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:45.125 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.125+0000 7fae58f9d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e73bfef50 0x561e73c013e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:45.125 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.125+0000 7fae58f9d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e73bbc3b0 con 0x561e73bfef50 2024-03-23T23:12:45.126 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.125+0000 7fae58f9d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e73b5e830 con 0x561e73bf93f0 2024-03-23T23:12:45.126 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.125+0000 7fae58f9d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e73b075d0 con 0x561e73bf9de0 2024-03-23T23:12:45.126 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.125+0000 7fae4f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e73bf9de0 0x561e73bfe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:45.126 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.125+0000 7fae4fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf93f0 0x561e73bf9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:45.126 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.125+0000 7fae4fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf93f0 0x561e73bf9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:49562/0 (socket says 172.21.15.12:49562) 2024-03-23T23:12:45.126 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.125+0000 7fae4fe00640 1 -- 172.21.15.12:0/86692822 learned_addr learned my addr 172.21.15.12:0/86692822 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:45.126 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.125+0000 7fae4ea00640 1 --2- 172.21.15.12:0/86692822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e73bfef50 0x561e73c013e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:45.126 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.126+0000 7fae4fe00640 1 -- 172.21.15.12:0/86692822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e73bf9de0 msgr2=0x561e73bfe8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:45.126 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.126+0000 7fae4fe00640 1 --2- 172.21.15.12:0/86692822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e73bf9de0 0x561e73bfe8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.126 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.126+0000 7fae4fe00640 1 -- 172.21.15.12:0/86692822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e73bfef50 msgr2=0x561e73c013e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:45.126 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.126+0000 7fae4fe00640 1 --2- 172.21.15.12:0/86692822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e73bfef50 0x561e73c013e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.126 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.126+0000 7fae4fe00640 1 -- 172.21.15.12:0/86692822 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e73b1f2c0 con 0x561e73bf93f0 2024-03-23T23:12:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.126+0000 7fae4fe00640 1 --2- 172.21.15.12:0/86692822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf93f0 0x561e73bf9810 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fae48011470 tx=0x7fae48011930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a027fa8bc0c6804b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.126+0000 7fae47e00640 1 --2- 172.21.15.12:0/3235846407 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fae300a27f0 0x7fae300a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.126+0000 7fae3fe00640 1 -- 172.21.15.12:0/86692822 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4801e070 con 0x561e73bf93f0 2024-03-23T23:12:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.126+0000 7fae3fe00640 1 -- 172.21.15.12:0/86692822 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae480035f0 con 0x561e73bf93f0 2024-03-23T23:12:45.130 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.126+0000 7fae3fe00640 1 -- 172.21.15.12:0/86692822 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4800f860 con 0x561e73bf93f0 2024-03-23T23:12:45.130 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.127+0000 7fae47e00640 1 -- 172.21.15.12:0/3235846407 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fae400cee60 con 0x7fae50101520 2024-03-23T23:12:45.130 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.127+0000 7fae4f400640 1 --2- 172.21.15.12:0/3235846407 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fae300a27f0 0x7fae300a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:45.130 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.127+0000 7fae4f400640 1 --2- 172.21.15.12:0/3235846407 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fae300a27f0 0x7fae300a4cb0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fae40097c10 tx=0x7fae400d28a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:45.130 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.127+0000 7fae58f9d740 1 -- 172.21.15.12:0/86692822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf93f0 msgr2=0x561e73bf9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:45.130 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.127+0000 7fae58f9d740 1 --2- 172.21.15.12:0/86692822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf93f0 0x561e73bf9810 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fae48011470 tx=0x7fae48011930 comp rx=0 tx=0).stop 2024-03-23T23:12:45.130 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.127+0000 7fae58f9d740 1 -- 172.21.15.12:0/86692822 shutdown_connections 2024-03-23T23:12:45.130 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.127+0000 7fae58f9d740 1 --2- 172.21.15.12:0/86692822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e73bfef50 0x561e73c013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.130 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.127+0000 7fae58f9d740 1 --2- 172.21.15.12:0/86692822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e73bf9de0 0x561e73bfe8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.130 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.127+0000 7fae58f9d740 1 --2- 172.21.15.12:0/86692822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf93f0 0x561e73bf9810 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.130 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.127+0000 7fae58f9d740 1 -- 172.21.15.12:0/86692822 >> 172.21.15.12:0/86692822 conn(0x561e73bf4960 msgr2=0x561e73bf4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:45.130 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.127+0000 7fae58f9d740 1 -- 172.21.15.12:0/86692822 shutdown_connections 2024-03-23T23:12:45.130 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.127+0000 7fae58f9d740 1 -- 172.21.15.12:0/86692822 wait complete. 2024-03-23T23:12:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.128+0000 7fae58f9d740 1 -- start start 2024-03-23T23:12:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.128+0000 7fae58f9d740 1 Processor -- start 2024-03-23T23:12:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.129+0000 7fae58f9d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e73bf93f0 0x561e73c035e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.129+0000 7fae58f9d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf9de0 0x561e73c03b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.129+0000 7fae58f9d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e73bfef50 0x561e73c0aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.129+0000 7fae58f9d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e73b1aa20 con 0x561e73bfef50 2024-03-23T23:12:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.129+0000 7fae58f9d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e73b5a8d0 con 0x561e73bf9de0 2024-03-23T23:12:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.129+0000 7fae58f9d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e73c01f00 con 0x561e73bf93f0 2024-03-23T23:12:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.129+0000 7fae58f9d740 10 client.0 _ll_register_callbacks cb 0x561e73b571c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:12:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.129+0000 7fae4ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf9de0 0x561e73c03b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.129+0000 7fae4ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf9de0 0x561e73c03b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:49576/0 (socket says 172.21.15.12:49576) 2024-03-23T23:12:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.129+0000 7fae4ea00640 1 -- 172.21.15.12:0/403933138 learned_addr learned my addr 172.21.15.12:0/403933138 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.129+0000 7fae4fe00640 1 --2- 172.21.15.12:0/403933138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e73bf93f0 0x561e73c035e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.129+0000 7fae4fe00640 1 --2- 172.21.15.12:0/403933138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e73bfef50 0x561e73c0aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.129+0000 7fae4ea00640 1 -- 172.21.15.12:0/403933138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e73bf93f0 msgr2=0x561e73c035e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.129+0000 7fae4ea00640 1 --2- 172.21.15.12:0/403933138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e73bf93f0 0x561e73c035e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.129+0000 7fae4ea00640 1 -- 172.21.15.12:0/403933138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e73bfef50 msgr2=0x561e73c0aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.129+0000 7fae4ea00640 1 --2- 172.21.15.12:0/403933138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e73bfef50 0x561e73c0aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.129+0000 7fae4ea00640 1 -- 172.21.15.12:0/403933138 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561e73873210 con 0x561e73bf9de0 2024-03-23T23:12:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.129+0000 7fae4fe00640 1 --2- 172.21.15.12:0/403933138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e73bf93f0 0x561e73c035e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:12:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.130+0000 7fae4ea00640 1 --2- 172.21.15.12:0/403933138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf9de0 0x561e73c03b20 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fae38009be0 tx=0x7fae38009280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ee6b6895e1a77c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.130+0000 7fae3cc00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:12:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.130+0000 7fae3cc00640 1 -- 172.21.15.12:0/403933138 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae38044070 con 0x561e73bf9de0 2024-03-23T23:12:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.130+0000 7fae3cc00640 1 -- 172.21.15.12:0/403933138 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fae38033850 con 0x561e73bf9de0 2024-03-23T23:12:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.130+0000 7fae58f9d740 1 -- 172.21.15.12:0/403933138 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fae480035f0 con 0x561e73bf9de0 2024-03-23T23:12:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.130+0000 7fae3cc00640 1 -- 172.21.15.12:0/403933138 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae38003970 con 0x561e73bf9de0 2024-03-23T23:12:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.130+0000 7fae58f9d740 10 client.28403 fetch_fsmap learned FSMap version 23 2024-03-23T23:12:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.130+0000 7fae58f9d740 1 -- 172.21.15.12:0/403933138 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561e7393f620 con 0x561e73bf9de0 2024-03-23T23:12:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.130+0000 7fae3cc00640 1 -- 172.21.15.12:0/403933138 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae480035f0 con 0x561e73bf9de0 2024-03-23T23:12:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.131+0000 7fae3cc00640 1 -- 172.21.15.12:0/403933138 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fae38004200 con 0x561e73bf9de0 2024-03-23T23:12:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.131+0000 7fae58f9d740 10 client.28403 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:12:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.131+0000 7fae58f9d740 10 client.28403 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:12:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.131+0000 7fae58f9d740 20 client.28403 populate_metadata read hostname 'smithi012' 2024-03-23T23:12:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.131+0000 7fae58f9d740 1 --2- 172.21.15.12:0/403933138 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561e73c0f440 0x561e73c11860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.131+0000 7fae58f9d740 4 client.28403 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:12:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.131+0000 7fae58f9d740 1 -- 172.21.15.12:0/403933138 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561e73b6bd70 con 0x561e73c0f440 2024-03-23T23:12:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.131+0000 7fae4fe00640 1 --2- 172.21.15.12:0/403933138 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561e73c0f440 0x561e73c11860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.131+0000 7fae4fe00640 1 --2- 172.21.15.12:0/403933138 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561e73c0f440 0x561e73c11860 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43d41afb251f89ba server_cookie=4a1a5336e4328582 in_seq=0 out_seq=0 2024-03-23T23:12:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.131+0000 7fae3cc00640 10 client.28403 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.132+0000 7fae3cc00640 1 -- 172.21.15.12:0/403933138 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561e73b6bd70 con 0x561e73c0f440 2024-03-23T23:12:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.132+0000 7fae3cc00640 10 client.28403 handle_command_reply: tid=0 2024-03-23T23:12:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.133+0000 7fae58f9d740 1 client.28403 shutdown 2024-03-23T23:12:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.133+0000 7fae58f9d740 1 -- 172.21.15.12:0/403933138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf9de0 msgr2=0x561e73c03b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.133+0000 7fae58f9d740 1 --2- 172.21.15.12:0/403933138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf9de0 0x561e73c03b20 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fae38009be0 tx=0x7fae38009280 comp rx=0 tx=0).stop 2024-03-23T23:12:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.133+0000 7fae58f9d740 1 -- 172.21.15.12:0/403933138 shutdown_connections 2024-03-23T23:12:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.133+0000 7fae58f9d740 1 --2- 172.21.15.12:0/403933138 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561e73c0f440 0x561e73c11860 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.133+0000 7fae58f9d740 1 --2- 172.21.15.12:0/403933138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e73bfef50 0x561e73c0aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.133+0000 7fae58f9d740 1 --2- 172.21.15.12:0/403933138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf9de0 0x561e73c03b20 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.133+0000 7fae58f9d740 1 --2- 172.21.15.12:0/403933138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e73bf93f0 0x561e73c035e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.133+0000 7fae3cc00640 0 client.28403 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.133+0000 7fae58f9d740 1 -- 172.21.15.12:0/403933138 >> 172.21.15.12:0/403933138 conn(0x561e73bf4960 msgr2=0x561e73bf3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.133+0000 7fae58f9d740 1 -- 172.21.15.12:0/403933138 shutdown_connections 2024-03-23T23:12:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.133+0000 7fae58f9d740 1 -- 172.21.15.12:0/403933138 wait complete. 2024-03-23T23:12:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.133+0000 7fae58f9d740 20 client.28403 trim_cache size 0 max 16384 2024-03-23T23:12:45.148 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.148+0000 7fae58f9d740 1 Processor -- start 2024-03-23T23:12:45.148 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.149+0000 7fae58f9d740 1 -- start start 2024-03-23T23:12:45.148 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.149+0000 7fae58f9d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf93f0 0x561e73bfb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:45.148 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.149+0000 7fae58f9d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e73bf9de0 0x561e73bfb680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:45.149 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.149+0000 7fae58f9d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e73bfef50 0x561e73c0aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:45.149 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.149+0000 7fae58f9d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae501ad000 con 0x561e73bfef50 2024-03-23T23:12:45.149 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.149+0000 7fae58f9d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae501aab00 con 0x561e73bf93f0 2024-03-23T23:12:45.149 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.149+0000 7fae58f9d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae501a8ba0 con 0x561e73bf9de0 2024-03-23T23:12:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.149+0000 7fae4ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e73bf9de0 0x561e73bfb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.149+0000 7fae4ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e73bf9de0 0x561e73bfb680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:41460/0 (socket says 172.21.15.12:41460) 2024-03-23T23:12:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.149+0000 7fae4ea00640 1 -- 172.21.15.12:0/2851187419 learned_addr learned my addr 172.21.15.12:0/2851187419 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.149+0000 7fae4fe00640 1 --2- 172.21.15.12:0/2851187419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf93f0 0x561e73bfb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.149+0000 7fae4fe00640 1 --2- 172.21.15.12:0/2851187419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e73bfef50 0x561e73c0aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.149+0000 7fae4ea00640 1 -- 172.21.15.12:0/2851187419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf93f0 msgr2=0x561e73bfb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.149+0000 7fae4ea00640 1 --2- 172.21.15.12:0/2851187419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf93f0 0x561e73bfb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.149+0000 7fae4ea00640 1 -- 172.21.15.12:0/2851187419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e73bfef50 msgr2=0x561e73c0aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.149+0000 7fae4ea00640 1 --2- 172.21.15.12:0/2851187419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e73bfef50 0x561e73c0aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.149+0000 7fae4ea00640 1 -- 172.21.15.12:0/2851187419 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e73bfa670 con 0x561e73bf9de0 2024-03-23T23:12:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.150+0000 7fae4ea00640 1 --2- 172.21.15.12:0/2851187419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e73bf9de0 0x561e73bfb680 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fae38033ff0 tx=0x7fae38005e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b63d1991f1c1e253 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.150+0000 7fae3f400640 1 -- 172.21.15.12:0/2851187419 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae38057070 con 0x561e73bf9de0 2024-03-23T23:12:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.150+0000 7fae3f400640 1 -- 172.21.15.12:0/2851187419 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae38004810 con 0x561e73bf9de0 2024-03-23T23:12:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.150+0000 7fae3f400640 1 -- 172.21.15.12:0/2851187419 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae38044040 con 0x561e73bf9de0 2024-03-23T23:12:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.151+0000 7fae58f9d740 1 -- 172.21.15.12:0/2851187419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e73bf9de0 msgr2=0x561e73bfb680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.151+0000 7fae58f9d740 1 --2- 172.21.15.12:0/2851187419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e73bf9de0 0x561e73bfb680 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fae38033ff0 tx=0x7fae38005e30 comp rx=0 tx=0).stop 2024-03-23T23:12:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.151+0000 7fae58f9d740 1 -- 172.21.15.12:0/2851187419 shutdown_connections 2024-03-23T23:12:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.151+0000 7fae58f9d740 1 --2- 172.21.15.12:0/2851187419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e73bfef50 0x561e73c0aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.151+0000 7fae58f9d740 1 --2- 172.21.15.12:0/2851187419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e73bf9de0 0x561e73bfb680 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.151+0000 7fae58f9d740 1 --2- 172.21.15.12:0/2851187419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf93f0 0x561e73bfb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.151+0000 7fae58f9d740 1 -- 172.21.15.12:0/2851187419 >> 172.21.15.12:0/2851187419 conn(0x561e73bf4960 msgr2=0x561e73c035e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.151+0000 7fae58f9d740 1 -- 172.21.15.12:0/2851187419 shutdown_connections 2024-03-23T23:12:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.151+0000 7fae58f9d740 1 -- 172.21.15.12:0/2851187419 wait complete. 2024-03-23T23:12:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.151+0000 7fae58f9d740 1 -- start start 2024-03-23T23:12:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.151+0000 7fae58f9d740 1 Processor -- start 2024-03-23T23:12:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.152+0000 7fae58f9d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf93f0 0x561e73c08440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.152+0000 7fae58f9d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e73bf9de0 0x561e73c08980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.152+0000 7fae58f9d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e73bfef50 0x561e73c09990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.152+0000 7fae58f9d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae501a91a0 con 0x561e73bf9de0 2024-03-23T23:12:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.152+0000 7fae58f9d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae501aa4a0 con 0x561e73bf93f0 2024-03-23T23:12:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.152+0000 7fae58f9d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae501a97e0 con 0x561e73bfef50 2024-03-23T23:12:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.152+0000 7fae58f9d740 10 client.0 _ll_register_callbacks cb 0x7fae40005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:12:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.152+0000 7fae4fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf93f0 0x561e73c08440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.152+0000 7fae4ea00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e73bf9de0 0x561e73c08980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.152+0000 7fae4fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf93f0 0x561e73c08440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:49594/0 (socket says 172.21.15.12:49594) 2024-03-23T23:12:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.152+0000 7fae4fe00640 1 -- 172.21.15.12:0/81261575 learned_addr learned my addr 172.21.15.12:0/81261575 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.152+0000 7fae4fe00640 1 --2- 172.21.15.12:0/81261575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e73bfef50 0x561e73c09990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.152+0000 7fae4fe00640 1 -- 172.21.15.12:0/81261575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e73bfef50 msgr2=0x561e73c09990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.152+0000 7fae4fe00640 1 --2- 172.21.15.12:0/81261575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e73bfef50 0x561e73c09990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.152+0000 7fae4fe00640 1 -- 172.21.15.12:0/81261575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e73bf9de0 msgr2=0x561e73c08980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.152+0000 7fae4fe00640 1 --2- 172.21.15.12:0/81261575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e73bf9de0 0x561e73c08980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.152+0000 7fae4fe00640 1 -- 172.21.15.12:0/81261575 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561e73c09f80 con 0x561e73bf93f0 2024-03-23T23:12:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.153+0000 7fae4ea00640 1 --2- 172.21.15.12:0/81261575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e73bf9de0 0x561e73c08980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:12:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.153+0000 7fae4fe00640 1 --2- 172.21.15.12:0/81261575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf93f0 0x561e73c08440 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fae48009e40 tx=0x7fae4802a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bb1e30c489cce3d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.153+0000 7fae3cc00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:12:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.153+0000 7fae3cc00640 1 -- 172.21.15.12:0/81261575 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4801e040 con 0x561e73bf93f0 2024-03-23T23:12:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.153+0000 7fae3cc00640 1 -- 172.21.15.12:0/81261575 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fae48016040 con 0x561e73bf93f0 2024-03-23T23:12:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.153+0000 7fae58f9d740 1 -- 172.21.15.12:0/81261575 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fae38004810 con 0x561e73bf93f0 2024-03-23T23:12:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.153+0000 7fae3cc00640 1 -- 172.21.15.12:0/81261575 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae4802c980 con 0x561e73bf93f0 2024-03-23T23:12:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.153+0000 7fae58f9d740 10 client.28409 fetch_fsmap learned FSMap version 23 2024-03-23T23:12:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.153+0000 7fae58f9d740 1 -- 172.21.15.12:0/81261575 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561e73c6af70 con 0x561e73bf93f0 2024-03-23T23:12:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.154+0000 7fae3cc00640 1 -- 172.21.15.12:0/81261575 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae38004810 con 0x561e73bf93f0 2024-03-23T23:12:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.154+0000 7fae3cc00640 1 -- 172.21.15.12:0/81261575 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fae4800f540 con 0x561e73bf93f0 2024-03-23T23:12:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.154+0000 7fae58f9d740 10 client.28409 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:12:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.154+0000 7fae58f9d740 10 client.28409 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:12:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.154+0000 7fae58f9d740 20 client.28409 populate_metadata read hostname 'smithi012' 2024-03-23T23:12:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.154+0000 7fae58f9d740 1 --2- 172.21.15.12:0/81261575 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561e73c6b620 0x561e73c6dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.154+0000 7fae58f9d740 4 client.28409 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:12:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.154+0000 7fae58f9d740 1 -- 172.21.15.12:0/81261575 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x561e73b660c0 con 0x561e73c6b620 2024-03-23T23:12:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.154+0000 7fae4ea00640 1 --2- 172.21.15.12:0/81261575 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561e73c6b620 0x561e73c6dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.154+0000 7fae4ea00640 1 --2- 172.21.15.12:0/81261575 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561e73c6b620 0x561e73c6dac0 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50f0d920a4bbbe84 server_cookie=95f7eb6b372474f3 in_seq=0 out_seq=0 2024-03-23T23:12:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.154+0000 7fae3cc00640 10 client.28409 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.168+0000 7fae3cc00640 1 -- 172.21.15.12:0/81261575 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x561e73b660c0 con 0x561e73c6b620 2024-03-23T23:12:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.168+0000 7fae3cc00640 10 client.28409 handle_command_reply: tid=0 2024-03-23T23:12:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.168+0000 7fae58f9d740 1 client.28409 shutdown 2024-03-23T23:12:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.168+0000 7fae58f9d740 1 -- 172.21.15.12:0/81261575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf93f0 msgr2=0x561e73c08440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.168+0000 7fae58f9d740 1 --2- 172.21.15.12:0/81261575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf93f0 0x561e73c08440 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fae48009e40 tx=0x7fae4802a040 comp rx=0 tx=0).stop 2024-03-23T23:12:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.168+0000 7fae58f9d740 1 -- 172.21.15.12:0/81261575 shutdown_connections 2024-03-23T23:12:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.168+0000 7fae58f9d740 1 --2- 172.21.15.12:0/81261575 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x561e73c6b620 0x561e73c6dac0 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.168+0000 7fae58f9d740 1 --2- 172.21.15.12:0/81261575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e73bfef50 0x561e73c09990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.168+0000 7fae58f9d740 1 --2- 172.21.15.12:0/81261575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561e73bf9de0 0x561e73c08980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.168+0000 7fae58f9d740 1 --2- 172.21.15.12:0/81261575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561e73bf93f0 0x561e73c08440 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.168+0000 7fae58f9d740 1 -- 172.21.15.12:0/81261575 >> 172.21.15.12:0/81261575 conn(0x561e73bf4960 msgr2=0x561e73bf32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.168+0000 7fae3cc00640 0 client.28409 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.168+0000 7fae58f9d740 1 -- 172.21.15.12:0/81261575 shutdown_connections 2024-03-23T23:12:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.168+0000 7fae58f9d740 1 -- 172.21.15.12:0/81261575 wait complete. 2024-03-23T23:12:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.169+0000 7fae58f9d740 20 client.28409 trim_cache size 0 max 16384 2024-03-23T23:12:45.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:12:45.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:12:45.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:12:45.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:12:45.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.169+0000 7fae54c00640 1 -- 172.21.15.12:0/3235846407 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fae300a27f0 msgr2=0x7fae300a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:45.170 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.169+0000 7fae54c00640 1 --2- 172.21.15.12:0/3235846407 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fae300a27f0 0x7fae300a4cb0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fae40097c10 tx=0x7fae400d28a0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.169+0000 7fae54c00640 1 -- 172.21.15.12:0/3235846407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae50101520 msgr2=0x7fae50199a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:45.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.169+0000 7fae54c00640 1 --2- 172.21.15.12:0/3235846407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae50101520 0x7fae50199a20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fae4000e9c0 tx=0x7fae4000ee80 comp rx=0 tx=0).stop 2024-03-23T23:12:45.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.169+0000 7fae54c00640 1 -- 172.21.15.12:0/3235846407 shutdown_connections 2024-03-23T23:12:45.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.169+0000 7fae54c00640 1 --2- 172.21.15.12:0/3235846407 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fae300a27f0 0x7fae300a4cb0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.169+0000 7fae54c00640 1 --2- 172.21.15.12:0/3235846407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae501030d0 0x7fae501a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.169+0000 7fae54c00640 1 --2- 172.21.15.12:0/3235846407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae50102710 0x7fae50199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:45.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.169+0000 7fae54c00640 1 --2- 172.21.15.12:0/3235846407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae50101520 0x7fae50199a20 secure :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fae4000e9c0 tx=0x7fae4000ee80 comp rx=0 tx=0).stop 2024-03-23T23:12:45.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.170+0000 7fae54c00640 1 -- 172.21.15.12:0/3235846407 >> 172.21.15.12:0/3235846407 conn(0x7fae500fcab0 msgr2=0x7fae500738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:45.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.170+0000 7fae54c00640 1 -- 172.21.15.12:0/3235846407 shutdown_connections 2024-03-23T23:12:45.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:45.170+0000 7fae54c00640 1 -- 172.21.15.12:0/3235846407 wait complete. 2024-03-23T23:12:45.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:45 smithi012 ceph-mon[31695]: pgmap v718: 97 pgs: 97 active+clean; 3.4 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:12:45.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:45 smithi173 ceph-mon[36116]: pgmap v718: 97 pgs: 97 active+clean; 3.4 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:12:45.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:45 smithi203 ceph-mon[36693]: pgmap v718: 97 pgs: 97 active+clean; 3.4 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:12:47.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:47 smithi012 ceph-mon[31695]: pgmap v719: 97 pgs: 97 active+clean; 3.4 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-03-23T23:12:47.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:47 smithi173 ceph-mon[36116]: pgmap v719: 97 pgs: 97 active+clean; 3.4 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-03-23T23:12:47.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:47 smithi203 ceph-mon[36693]: pgmap v719: 97 pgs: 97 active+clean; 3.4 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-03-23T23:12:49.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:49 smithi173 ceph-mon[36116]: pgmap v720: 97 pgs: 97 active+clean; 3.4 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:12:49.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:49 smithi203 ceph-mon[36693]: pgmap v720: 97 pgs: 97 active+clean; 3.4 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:12:49.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:49 smithi012 ceph-mon[31695]: pgmap v720: 97 pgs: 97 active+clean; 3.4 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-03-23T23:12:51.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:51 smithi173 ceph-mon[36116]: pgmap v721: 97 pgs: 97 active+clean; 3.4 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:12:51.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:51 smithi203 ceph-mon[36693]: pgmap v721: 97 pgs: 97 active+clean; 3.4 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:12:51.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:51 smithi012 ceph-mon[31695]: pgmap v721: 97 pgs: 97 active+clean; 3.4 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-03-23T23:12:53.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:53 smithi173 ceph-mon[36116]: pgmap v722: 97 pgs: 97 active+clean; 3.5 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:12:53.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:53 smithi203 ceph-mon[36693]: pgmap v722: 97 pgs: 97 active+clean; 3.5 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:12:53.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:53 smithi012 ceph-mon[31695]: pgmap v722: 97 pgs: 97 active+clean; 3.5 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:12:54.589 INFO:tasks.workunit.client.0.smithi012.stdout:499/600: open fd = 3 2024-03-23T23:12:54.589 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.589 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.589 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.589 INFO:tasks.workunit.client.0.smithi012.stdout:500/600: open fd = 3 2024-03-23T23:12:54.589 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.589 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.589 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.590 INFO:tasks.workunit.client.0.smithi012.stdout:501/600: open fd = 3 2024-03-23T23:12:54.590 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.590 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.590 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.590 INFO:tasks.workunit.client.0.smithi012.stdout:502/600: open fd = 3 2024-03-23T23:12:54.590 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.590 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.590 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.590 INFO:tasks.workunit.client.0.smithi012.stdout:503/600: open fd = 3 2024-03-23T23:12:54.590 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.590 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.590 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.590 INFO:tasks.workunit.client.0.smithi012.stdout:504/600: open fd = 3 2024-03-23T23:12:54.591 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.591 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.591 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.593 INFO:tasks.workunit.client.0.smithi012.stdout:505/600: open fd = 3 2024-03-23T23:12:54.593 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.593 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.593 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.594 INFO:tasks.workunit.client.0.smithi012.stdout:506/600: open fd = 3 2024-03-23T23:12:54.594 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.594 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.594 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.594 INFO:tasks.workunit.client.0.smithi012.stdout:507/600: open fd = 3 2024-03-23T23:12:54.594 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.595 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.595 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.595 INFO:tasks.workunit.client.0.smithi012.stdout:508/600: open fd = 3 2024-03-23T23:12:54.595 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.595 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.595 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.595 INFO:tasks.workunit.client.0.smithi012.stdout:509/600: open fd = 3 2024-03-23T23:12:54.596 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.596 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.596 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.596 INFO:tasks.workunit.client.0.smithi012.stdout:510/600: open fd = 3 2024-03-23T23:12:54.596 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.596 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.596 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.597 INFO:tasks.workunit.client.0.smithi012.stdout:511/600: open fd = 3 2024-03-23T23:12:54.597 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.597 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.597 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.597 INFO:tasks.workunit.client.0.smithi012.stdout:512/600: open fd = 3 2024-03-23T23:12:54.597 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.598 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.598 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.598 INFO:tasks.workunit.client.0.smithi012.stdout:513/600: open fd = 3 2024-03-23T23:12:54.598 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.598 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.598 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.599 INFO:tasks.workunit.client.0.smithi012.stdout:514/600: open fd = 3 2024-03-23T23:12:54.599 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.599 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.599 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.599 INFO:tasks.workunit.client.0.smithi012.stdout:515/600: open fd = 3 2024-03-23T23:12:54.599 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.599 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.600 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.600 INFO:tasks.workunit.client.0.smithi012.stdout:516/600: open fd = 3 2024-03-23T23:12:54.600 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.600 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.600 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.600 INFO:tasks.workunit.client.0.smithi012.stdout:517/600: open fd = 3 2024-03-23T23:12:54.601 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.601 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.601 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.601 INFO:tasks.workunit.client.0.smithi012.stdout:518/600: open fd = 3 2024-03-23T23:12:54.601 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.601 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.601 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.601 INFO:tasks.workunit.client.0.smithi012.stdout:519/600: open fd = 3 2024-03-23T23:12:54.602 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.602 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.602 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.602 INFO:tasks.workunit.client.0.smithi012.stdout:520/600: open fd = 3 2024-03-23T23:12:54.602 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.602 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.603 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.603 INFO:tasks.workunit.client.0.smithi012.stdout:521/600: open fd = 3 2024-03-23T23:12:54.603 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.603 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.603 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.603 INFO:tasks.workunit.client.0.smithi012.stdout:522/600: open fd = 3 2024-03-23T23:12:54.603 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.604 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.604 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.604 INFO:tasks.workunit.client.0.smithi012.stdout:523/600: open fd = 3 2024-03-23T23:12:54.604 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.604 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.604 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.604 INFO:tasks.workunit.client.0.smithi012.stdout:524/600: open fd = 3 2024-03-23T23:12:54.605 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.605 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.605 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.605 INFO:tasks.workunit.client.0.smithi012.stdout:525/600: open fd = 3 2024-03-23T23:12:54.605 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.605 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.606 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.606 INFO:tasks.workunit.client.0.smithi012.stdout:526/600: open fd = 3 2024-03-23T23:12:54.606 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.606 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.606 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.606 INFO:tasks.workunit.client.0.smithi012.stdout:527/600: open fd = 3 2024-03-23T23:12:54.606 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.607 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.607 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.607 INFO:tasks.workunit.client.0.smithi012.stdout:528/600: open fd = 3 2024-03-23T23:12:54.607 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.607 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.607 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.607 INFO:tasks.workunit.client.0.smithi012.stdout:529/600: open fd = 3 2024-03-23T23:12:54.608 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.608 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.608 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.608 INFO:tasks.workunit.client.0.smithi012.stdout:530/600: open fd = 3 2024-03-23T23:12:54.608 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.608 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.608 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.609 INFO:tasks.workunit.client.0.smithi012.stdout:531/600: open fd = 3 2024-03-23T23:12:54.609 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.609 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.609 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.609 INFO:tasks.workunit.client.0.smithi012.stdout:532/600: open fd = 3 2024-03-23T23:12:54.609 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.609 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.610 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.610 INFO:tasks.workunit.client.0.smithi012.stdout:533/600: open fd = 3 2024-03-23T23:12:54.610 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.610 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.610 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.610 INFO:tasks.workunit.client.0.smithi012.stdout:534/600: open fd = 3 2024-03-23T23:12:54.611 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.611 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.611 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.611 INFO:tasks.workunit.client.0.smithi012.stdout:535/600: open fd = 3 2024-03-23T23:12:54.611 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.611 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.612 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.612 INFO:tasks.workunit.client.0.smithi012.stdout:536/600: open fd = 3 2024-03-23T23:12:54.612 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.612 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.612 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.612 INFO:tasks.workunit.client.0.smithi012.stdout:537/600: open fd = 3 2024-03-23T23:12:54.612 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.613 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.613 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.613 INFO:tasks.workunit.client.0.smithi012.stdout:538/600: open fd = 3 2024-03-23T23:12:54.613 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.613 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.613 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.613 INFO:tasks.workunit.client.0.smithi012.stdout:539/600: open fd = 3 2024-03-23T23:12:54.614 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.614 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.614 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.614 INFO:tasks.workunit.client.0.smithi012.stdout:540/600: open fd = 3 2024-03-23T23:12:54.614 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.614 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.615 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.615 INFO:tasks.workunit.client.0.smithi012.stdout:541/600: open fd = 3 2024-03-23T23:12:54.615 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.615 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.615 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.615 INFO:tasks.workunit.client.0.smithi012.stdout:542/600: open fd = 3 2024-03-23T23:12:54.615 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.615 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.616 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.616 INFO:tasks.workunit.client.0.smithi012.stdout:543/600: open fd = 3 2024-03-23T23:12:54.616 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.616 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.616 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.616 INFO:tasks.workunit.client.0.smithi012.stdout:544/600: open fd = 3 2024-03-23T23:12:54.616 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.617 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.617 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.617 INFO:tasks.workunit.client.0.smithi012.stdout:545/600: open fd = 3 2024-03-23T23:12:54.617 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.617 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.617 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.617 INFO:tasks.workunit.client.0.smithi012.stdout:546/600: open fd = 3 2024-03-23T23:12:54.617 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.618 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.618 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.618 INFO:tasks.workunit.client.0.smithi012.stdout:547/600: open fd = 3 2024-03-23T23:12:54.618 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.618 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.618 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.619 INFO:tasks.workunit.client.0.smithi012.stdout:548/600: open fd = 3 2024-03-23T23:12:54.619 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.619 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.619 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.619 INFO:tasks.workunit.client.0.smithi012.stdout:549/600: open fd = 3 2024-03-23T23:12:54.619 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.620 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.620 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.620 INFO:tasks.workunit.client.0.smithi012.stdout:550/600: open fd = 3 2024-03-23T23:12:54.620 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.620 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.620 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.620 INFO:tasks.workunit.client.0.smithi012.stdout:551/600: open fd = 3 2024-03-23T23:12:54.621 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.621 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.621 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.621 INFO:tasks.workunit.client.0.smithi012.stdout:552/600: open fd = 3 2024-03-23T23:12:54.621 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.621 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.621 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.621 INFO:tasks.workunit.client.0.smithi012.stdout:553/600: open fd = 3 2024-03-23T23:12:54.622 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.622 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.622 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.622 INFO:tasks.workunit.client.0.smithi012.stdout:554/600: open fd = 3 2024-03-23T23:12:54.622 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.622 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.623 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.623 INFO:tasks.workunit.client.0.smithi012.stdout:555/600: open fd = 3 2024-03-23T23:12:54.623 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.623 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.623 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.623 INFO:tasks.workunit.client.0.smithi012.stdout:556/600: open fd = 3 2024-03-23T23:12:54.623 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.624 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.624 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.624 INFO:tasks.workunit.client.0.smithi012.stdout:557/600: open fd = 3 2024-03-23T23:12:54.624 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.624 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.624 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.624 INFO:tasks.workunit.client.0.smithi012.stdout:558/600: open fd = 3 2024-03-23T23:12:54.624 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.625 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.625 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.625 INFO:tasks.workunit.client.0.smithi012.stdout:559/600: open fd = 3 2024-03-23T23:12:54.625 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.625 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.625 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:54.626 INFO:tasks.workunit.client.0.smithi012.stdout:560/600: open fd = 3 2024-03-23T23:12:54.626 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.626 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:12:54.626 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:12:55.617 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:55 smithi173 ceph-mon[36116]: pgmap v723: 97 pgs: 97 active+clean; 3.5 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:12:55.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:55 smithi203 ceph-mon[36693]: pgmap v723: 97 pgs: 97 active+clean; 3.5 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:12:55.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:55 smithi012 ceph-mon[31695]: pgmap v723: 97 pgs: 97 active+clean; 3.5 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:12:57.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:57 smithi173 ceph-mon[36116]: pgmap v724: 97 pgs: 97 active+clean; 3.6 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:12:57.658 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:57 smithi203 ceph-mon[36693]: pgmap v724: 97 pgs: 97 active+clean; 3.6 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:12:57.866 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:57 smithi012 ceph-mon[31695]: pgmap v724: 97 pgs: 97 active+clean; 3.6 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:12:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.319+0000 7fe67d000640 1 -- 172.21.15.12:0/571023446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe678107d00 msgr2=0x7fe678108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.319+0000 7fe67d000640 1 --2- 172.21.15.12:0/571023446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe678107d00 0x7fe678108120 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe668009900 tx=0x7fe66802bc30 comp rx=0 tx=0).stop 2024-03-23T23:12:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.319+0000 7fe67d000640 1 -- 172.21.15.12:0/571023446 shutdown_connections 2024-03-23T23:12:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.319+0000 7fe67d000640 1 --2- 172.21.15.12:0/571023446 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6781098b0 0x7fe678110140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.319+0000 7fe67d000640 1 --2- 172.21.15.12:0/571023446 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe678108ef0 0x7fe678109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.319+0000 7fe67d000640 1 --2- 172.21.15.12:0/571023446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe678107d00 0x7fe678108120 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.319+0000 7fe67d000640 1 -- 172.21.15.12:0/571023446 >> 172.21.15.12:0/571023446 conn(0x7fe678076cb0 msgr2=0x7fe6780790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.319+0000 7fe67d000640 1 -- 172.21.15.12:0/571023446 shutdown_connections 2024-03-23T23:12:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.319+0000 7fe67d000640 1 -- 172.21.15.12:0/571023446 wait complete. 2024-03-23T23:12:58.321 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.320+0000 7fe67d000640 1 Processor -- start 2024-03-23T23:12:58.321 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.320+0000 7fe67d000640 1 -- start start 2024-03-23T23:12:58.321 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.321+0000 7fe67d000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe678107d00 0x7fe67819df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:58.321 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.321+0000 7fe67d000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe678108ef0 0x7fe67819e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:58.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.321+0000 7fe67d000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6781098b0 0x7fe6781a5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:58.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.321+0000 7fe67d000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6781139c0 con 0x7fe678107d00 2024-03-23T23:12:58.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.321+0000 7fe67d000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe678113840 con 0x7fe6781098b0 2024-03-23T23:12:58.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.321+0000 7fe67d000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe678113b40 con 0x7fe678108ef0 2024-03-23T23:12:58.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.321+0000 7fe675e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe678107d00 0x7fe67819df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:58.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.321+0000 7fe675e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe678107d00 0x7fe67819df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43150/0 (socket says 172.21.15.12:43150) 2024-03-23T23:12:58.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.321+0000 7fe676800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6781098b0 0x7fe6781a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:58.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.321+0000 7fe675e00640 1 -- 172.21.15.12:0/3498105254 learned_addr learned my addr 172.21.15.12:0/3498105254 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:58.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.321+0000 7fe66d400640 1 --2- 172.21.15.12:0/3498105254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe678108ef0 0x7fe67819e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:58.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.321+0000 7fe676800640 1 -- 172.21.15.12:0/3498105254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe678108ef0 msgr2=0x7fe67819e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:58.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.321+0000 7fe676800640 1 --2- 172.21.15.12:0/3498105254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe678108ef0 0x7fe67819e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.321+0000 7fe676800640 1 -- 172.21.15.12:0/3498105254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe678107d00 msgr2=0x7fe67819df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:58.322 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.321+0000 7fe676800640 1 --2- 172.21.15.12:0/3498105254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe678107d00 0x7fe67819df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.323 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.321+0000 7fe676800640 1 -- 172.21.15.12:0/3498105254 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6781a5b90 con 0x7fe6781098b0 2024-03-23T23:12:58.323 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.322+0000 7fe676800640 1 --2- 172.21.15.12:0/3498105254 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6781098b0 0x7fe6781a5520 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fe670009fd0 tx=0x7fe67000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:58.323 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.322+0000 7fe66f400640 1 -- 172.21.15.12:0/3498105254 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67000ce20 con 0x7fe6781098b0 2024-03-23T23:12:58.323 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.322+0000 7fe66f400640 1 -- 172.21.15.12:0/3498105254 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe670005020 con 0x7fe6781098b0 2024-03-23T23:12:58.324 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.322+0000 7fe67d000640 1 -- 172.21.15.12:0/3498105254 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6781a5e80 con 0x7fe6781098b0 2024-03-23T23:12:58.324 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.322+0000 7fe66f400640 1 -- 172.21.15.12:0/3498105254 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe670010720 con 0x7fe6781098b0 2024-03-23T23:12:58.324 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.322+0000 7fe67d000640 1 -- 172.21.15.12:0/3498105254 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6781a62f0 con 0x7fe6781098b0 2024-03-23T23:12:58.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.326+0000 7fe67fd9a740 1 Processor -- start 2024-03-23T23:12:58.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.326+0000 7fe66f400640 1 -- 172.21.15.12:0/3498105254 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe670005540 con 0x7fe6781098b0 2024-03-23T23:12:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.326+0000 7fe67fd9a740 1 -- start start 2024-03-23T23:12:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.326+0000 7fe67fd9a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562f5d7621e0 0x562f5d762600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.326+0000 7fe67fd9a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f5d762bd0 0x562f5d7676d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.326+0000 7fe67fd9a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f5d767d40 0x562f5d76a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.326+0000 7fe67fd9a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f5d7253b0 con 0x562f5d7621e0 2024-03-23T23:12:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.326+0000 7fe67fd9a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f5d6c7830 con 0x562f5d762bd0 2024-03-23T23:12:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.326+0000 7fe67fd9a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f5d6705d0 con 0x562f5d767d40 2024-03-23T23:12:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.327+0000 7fe675e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f5d762bd0 0x562f5d7676d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.327+0000 7fe66d400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562f5d7621e0 0x562f5d762600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.327+0000 7fe675e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f5d762bd0 0x562f5d7676d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57928/0 (socket says 172.21.15.12:57928) 2024-03-23T23:12:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.327+0000 7fe675e00640 1 -- 172.21.15.12:0/2406006925 learned_addr learned my addr 172.21.15.12:0/2406006925 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.327+0000 7fe66d400640 1 --2- 172.21.15.12:0/2406006925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f5d767d40 0x562f5d76a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.327+0000 7fe675e00640 1 -- 172.21.15.12:0/2406006925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f5d767d40 msgr2=0x562f5d76a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.327+0000 7fe675e00640 1 --2- 172.21.15.12:0/2406006925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f5d767d40 0x562f5d76a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.327+0000 7fe675e00640 1 -- 172.21.15.12:0/2406006925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562f5d7621e0 msgr2=0x562f5d762600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.327+0000 7fe675e00640 1 --2- 172.21.15.12:0/2406006925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562f5d7621e0 0x562f5d762600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.327+0000 7fe675e00640 1 -- 172.21.15.12:0/2406006925 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562f5d6882c0 con 0x562f5d762bd0 2024-03-23T23:12:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.328+0000 7fe675e00640 1 --2- 172.21.15.12:0/2406006925 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f5d762bd0 0x562f5d7676d0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe66802c170 tx=0x7fe668009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d345dab2511df84d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.328+0000 7fe66f400640 1 --2- 172.21.15.12:0/3498105254 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe65c0a28a0 0x7fe65c0a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.328+0000 7fe667400640 1 -- 172.21.15.12:0/2406006925 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe668045070 con 0x562f5d762bd0 2024-03-23T23:12:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.328+0000 7fe667400640 1 -- 172.21.15.12:0/2406006925 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe668004040 con 0x562f5d762bd0 2024-03-23T23:12:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.328+0000 7fe667400640 1 -- 172.21.15.12:0/2406006925 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe668004c20 con 0x562f5d762bd0 2024-03-23T23:12:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.328+0000 7fe66f400640 1 -- 172.21.15.12:0/3498105254 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe6700cf180 con 0x7fe6781098b0 2024-03-23T23:12:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.328+0000 7fe66d400640 1 --2- 172.21.15.12:0/3498105254 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe65c0a28a0 0x7fe65c0a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.329+0000 7fe67fd9a740 1 -- 172.21.15.12:0/2406006925 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f5d762bd0 msgr2=0x562f5d7676d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.329+0000 7fe67fd9a740 1 --2- 172.21.15.12:0/2406006925 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f5d762bd0 0x562f5d7676d0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe66802c170 tx=0x7fe668009560 comp rx=0 tx=0).stop 2024-03-23T23:12:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.329+0000 7fe66d400640 1 --2- 172.21.15.12:0/3498105254 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe65c0a28a0 0x7fe65c0a4d60 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fe660009710 tx=0x7fe660015040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.329+0000 7fe67fd9a740 1 -- 172.21.15.12:0/2406006925 shutdown_connections 2024-03-23T23:12:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.329+0000 7fe67fd9a740 1 --2- 172.21.15.12:0/2406006925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f5d767d40 0x562f5d76a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.329+0000 7fe67fd9a740 1 --2- 172.21.15.12:0/2406006925 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f5d762bd0 0x562f5d7676d0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.329+0000 7fe67fd9a740 1 --2- 172.21.15.12:0/2406006925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562f5d7621e0 0x562f5d762600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.329+0000 7fe67fd9a740 1 -- 172.21.15.12:0/2406006925 >> 172.21.15.12:0/2406006925 conn(0x562f5d75d750 msgr2=0x562f5d75db50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.329+0000 7fe67fd9a740 1 -- 172.21.15.12:0/2406006925 shutdown_connections 2024-03-23T23:12:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.329+0000 7fe67fd9a740 1 -- 172.21.15.12:0/2406006925 wait complete. 2024-03-23T23:12:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.329+0000 7fe67fd9a740 1 -- start start 2024-03-23T23:12:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.330+0000 7fe67fd9a740 1 Processor -- start 2024-03-23T23:12:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.330+0000 7fe67fd9a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562f5d7621e0 0x562f5d76c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.330+0000 7fe67fd9a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f5d762bd0 0x562f5d76c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.330+0000 7fe67fd9a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f5d767d40 0x562f5d773880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.330+0000 7fe67fd9a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f5d683a20 con 0x562f5d7621e0 2024-03-23T23:12:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.330+0000 7fe67fd9a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f5d6c38d0 con 0x562f5d762bd0 2024-03-23T23:12:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.330+0000 7fe67fd9a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f5d76ac90 con 0x562f5d767d40 2024-03-23T23:12:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.330+0000 7fe67fd9a740 10 client.0 _ll_register_callbacks cb 0x562f5d6c01c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:12:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.330+0000 7fe676800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f5d767d40 0x562f5d773880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.330+0000 7fe66d400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f5d762bd0 0x562f5d76c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.330+0000 7fe676800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f5d767d40 0x562f5d773880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44706/0 (socket says 172.21.15.12:44706) 2024-03-23T23:12:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.330+0000 7fe675e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562f5d7621e0 0x562f5d76c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.330+0000 7fe676800640 1 -- 172.21.15.12:0/2602863839 learned_addr learned my addr 172.21.15.12:0/2602863839 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.331+0000 7fe676800640 1 -- 172.21.15.12:0/2602863839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f5d762bd0 msgr2=0x562f5d76c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.331+0000 7fe676800640 1 --2- 172.21.15.12:0/2602863839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f5d762bd0 0x562f5d76c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.331+0000 7fe676800640 1 -- 172.21.15.12:0/2602863839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562f5d7621e0 msgr2=0x562f5d76c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.331+0000 7fe676800640 1 --2- 172.21.15.12:0/2602863839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562f5d7621e0 0x562f5d76c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.331+0000 7fe676800640 1 -- 172.21.15.12:0/2602863839 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562f5d3dc210 con 0x562f5d767d40 2024-03-23T23:12:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.331+0000 7fe676800640 1 --2- 172.21.15.12:0/2602863839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f5d767d40 0x562f5d773880 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fe670097f40 tx=0x7fe67009bc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9158c349ef99b206 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.331+0000 7fe65be00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:12:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.331+0000 7fe65be00640 1 -- 172.21.15.12:0/2602863839 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6700051d0 con 0x562f5d767d40 2024-03-23T23:12:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.331+0000 7fe65be00640 1 -- 172.21.15.12:0/2602863839 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe6700a3df0 con 0x562f5d767d40 2024-03-23T23:12:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.331+0000 7fe67fd9a740 1 -- 172.21.15.12:0/2602863839 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe668004040 con 0x562f5d767d40 2024-03-23T23:12:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.332+0000 7fe65be00640 1 -- 172.21.15.12:0/2602863839 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6700a3ff0 con 0x562f5d767d40 2024-03-23T23:12:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.332+0000 7fe67fd9a740 10 client.28390 fetch_fsmap learned FSMap version 23 2024-03-23T23:12:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.332+0000 7fe67fd9a740 1 -- 172.21.15.12:0/2602863839 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x562f5d4a8620 con 0x562f5d767d40 2024-03-23T23:12:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.332+0000 7fe65be00640 1 -- 172.21.15.12:0/2602863839 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe668004040 con 0x562f5d767d40 2024-03-23T23:12:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.332+0000 7fe65be00640 1 -- 172.21.15.12:0/2602863839 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe67009af30 con 0x562f5d767d40 2024-03-23T23:12:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.332+0000 7fe67fd9a740 10 client.28390 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:12:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.332+0000 7fe67fd9a740 10 client.28390 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:12:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.332+0000 7fe67fd9a740 20 client.28390 populate_metadata read hostname 'smithi012' 2024-03-23T23:12:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.332+0000 7fe67fd9a740 1 --2- 172.21.15.12:0/2602863839 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562f5d7781f0 0x562f5d77a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.332+0000 7fe67fd9a740 4 client.28390 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:12:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.332+0000 7fe67fd9a740 1 -- 172.21.15.12:0/2602863839 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562f5d6d4d70 con 0x562f5d7781f0 2024-03-23T23:12:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.332+0000 7fe675e00640 1 --2- 172.21.15.12:0/2602863839 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562f5d7781f0 0x562f5d77a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.333+0000 7fe675e00640 1 --2- 172.21.15.12:0/2602863839 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562f5d7781f0 0x562f5d77a660 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b62ba969005a2ce server_cookie=ceff489a77f570fa in_seq=0 out_seq=0 2024-03-23T23:12:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.333+0000 7fe65be00640 10 client.28390 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.334+0000 7fe65be00640 1 -- 172.21.15.12:0/2602863839 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562f5d6d4d70 con 0x562f5d7781f0 2024-03-23T23:12:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.334+0000 7fe65be00640 10 client.28390 handle_command_reply: tid=0 2024-03-23T23:12:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.334+0000 7fe67fd9a740 1 client.28390 shutdown 2024-03-23T23:12:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.334+0000 7fe67fd9a740 1 -- 172.21.15.12:0/2602863839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f5d767d40 msgr2=0x562f5d773880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.334+0000 7fe67fd9a740 1 --2- 172.21.15.12:0/2602863839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f5d767d40 0x562f5d773880 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fe670097f40 tx=0x7fe67009bc20 comp rx=0 tx=0).stop 2024-03-23T23:12:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.334+0000 7fe67fd9a740 1 -- 172.21.15.12:0/2602863839 shutdown_connections 2024-03-23T23:12:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.334+0000 7fe67fd9a740 1 --2- 172.21.15.12:0/2602863839 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562f5d7781f0 0x562f5d77a660 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.334+0000 7fe67fd9a740 1 --2- 172.21.15.12:0/2602863839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f5d767d40 0x562f5d773880 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.334+0000 7fe67fd9a740 1 --2- 172.21.15.12:0/2602863839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f5d762bd0 0x562f5d76c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.334+0000 7fe67fd9a740 1 --2- 172.21.15.12:0/2602863839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562f5d7621e0 0x562f5d76c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.334+0000 7fe65be00640 0 client.28390 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.334+0000 7fe67fd9a740 1 -- 172.21.15.12:0/2602863839 >> 172.21.15.12:0/2602863839 conn(0x562f5d75d750 msgr2=0x562f5d75c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.334+0000 7fe67fd9a740 1 -- 172.21.15.12:0/2602863839 shutdown_connections 2024-03-23T23:12:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.334+0000 7fe67fd9a740 1 -- 172.21.15.12:0/2602863839 wait complete. 2024-03-23T23:12:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.334+0000 7fe67fd9a740 20 client.28390 trim_cache size 0 max 16384 2024-03-23T23:12:58.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.350+0000 7fe67fd9a740 1 Processor -- start 2024-03-23T23:12:58.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.350+0000 7fe67fd9a740 1 -- start start 2024-03-23T23:12:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.350+0000 7fe67fd9a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562f5d7621e0 0x562f5d7609b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.350+0000 7fe67fd9a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f5d762bd0 0x562f5d773840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.350+0000 7fe67fd9a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f5d767d40 0x562f5d760ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.350+0000 7fe67fd9a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6781b1400 con 0x562f5d7621e0 2024-03-23T23:12:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.350+0000 7fe67fd9a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6781aef00 con 0x562f5d767d40 2024-03-23T23:12:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.350+0000 7fe67fd9a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6781acfa0 con 0x562f5d762bd0 2024-03-23T23:12:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.350+0000 7fe66d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f5d762bd0 0x562f5d773840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.350+0000 7fe66d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f5d762bd0 0x562f5d773840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44720/0 (socket says 172.21.15.12:44720) 2024-03-23T23:12:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.350+0000 7fe66d400640 1 -- 172.21.15.12:0/876514442 learned_addr learned my addr 172.21.15.12:0/876514442 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.350+0000 7fe676800640 1 --2- 172.21.15.12:0/876514442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f5d767d40 0x562f5d760ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.350+0000 7fe66d400640 1 -- 172.21.15.12:0/876514442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f5d767d40 msgr2=0x562f5d760ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.350+0000 7fe66d400640 1 --2- 172.21.15.12:0/876514442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f5d767d40 0x562f5d760ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.351+0000 7fe66d400640 1 -- 172.21.15.12:0/876514442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562f5d7621e0 msgr2=0x562f5d7609b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.351+0000 7fe675e00640 1 --2- 172.21.15.12:0/876514442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562f5d7621e0 0x562f5d7609b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.351+0000 7fe66d400640 1 --2- 172.21.15.12:0/876514442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562f5d7621e0 0x562f5d7609b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.351+0000 7fe66d400640 1 -- 172.21.15.12:0/876514442 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562f5d76cbb0 con 0x562f5d762bd0 2024-03-23T23:12:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.351+0000 7fe676800640 1 --2- 172.21.15.12:0/876514442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f5d767d40 0x562f5d760ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:12:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.351+0000 7fe675e00640 1 --2- 172.21.15.12:0/876514442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562f5d7621e0 0x562f5d7609b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:12:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.351+0000 7fe66d400640 1 --2- 172.21.15.12:0/876514442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f5d762bd0 0x562f5d773840 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fe66000f5b0 tx=0x7fe6600046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aecc498390ae4b00 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:58.354 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.351+0000 7fe666a00640 1 -- 172.21.15.12:0/876514442 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe660005020 con 0x562f5d762bd0 2024-03-23T23:12:58.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.351+0000 7fe666a00640 1 -- 172.21.15.12:0/876514442 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6600047a0 con 0x562f5d762bd0 2024-03-23T23:12:58.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.351+0000 7fe666a00640 1 -- 172.21.15.12:0/876514442 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe660022650 con 0x562f5d762bd0 2024-03-23T23:12:58.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.352+0000 7fe67fd9a740 1 -- 172.21.15.12:0/876514442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f5d762bd0 msgr2=0x562f5d773840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:58.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.352+0000 7fe67fd9a740 1 --2- 172.21.15.12:0/876514442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f5d762bd0 0x562f5d773840 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fe66000f5b0 tx=0x7fe6600046c0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.352+0000 7fe67fd9a740 1 -- 172.21.15.12:0/876514442 shutdown_connections 2024-03-23T23:12:58.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.352+0000 7fe67fd9a740 1 --2- 172.21.15.12:0/876514442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f5d767d40 0x562f5d760ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.352+0000 7fe67fd9a740 1 --2- 172.21.15.12:0/876514442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f5d762bd0 0x562f5d773840 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.352+0000 7fe67fd9a740 1 --2- 172.21.15.12:0/876514442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562f5d7621e0 0x562f5d7609b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.352+0000 7fe67fd9a740 1 -- 172.21.15.12:0/876514442 >> 172.21.15.12:0/876514442 conn(0x562f5d75d750 msgr2=0x562f5d763e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:58.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.352+0000 7fe67fd9a740 1 -- 172.21.15.12:0/876514442 shutdown_connections 2024-03-23T23:12:58.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.352+0000 7fe67fd9a740 1 -- 172.21.15.12:0/876514442 wait complete. 2024-03-23T23:12:58.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.353+0000 7fe67fd9a740 1 -- start start 2024-03-23T23:12:58.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.353+0000 7fe67fd9a740 1 Processor -- start 2024-03-23T23:12:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.353+0000 7fe67fd9a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562f5d7621e0 0x562f5d771540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.353+0000 7fe67fd9a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f5d762bd0 0x562f5d771a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.353+0000 7fe67fd9a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f5d767d40 0x562f5d772a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.353+0000 7fe67fd9a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6781ad5a0 con 0x562f5d7621e0 2024-03-23T23:12:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.353+0000 7fe67fd9a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6781ae8a0 con 0x562f5d767d40 2024-03-23T23:12:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.353+0000 7fe67fd9a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6781adbe0 con 0x562f5d762bd0 2024-03-23T23:12:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.353+0000 7fe67fd9a740 10 client.0 _ll_register_callbacks cb 0x7fe670097d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:12:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.354+0000 7fe675e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562f5d7621e0 0x562f5d771540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.354+0000 7fe676800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f5d767d40 0x562f5d772a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.354+0000 7fe66d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f5d762bd0 0x562f5d771a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.354+0000 7fe676800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f5d767d40 0x562f5d772a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57966/0 (socket says 172.21.15.12:57966) 2024-03-23T23:12:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.354+0000 7fe66d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f5d762bd0 0x562f5d771a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44730/0 (socket says 172.21.15.12:44730) 2024-03-23T23:12:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.354+0000 7fe676800640 1 -- 172.21.15.12:0/2915139712 learned_addr learned my addr 172.21.15.12:0/2915139712 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:12:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.354+0000 7fe676800640 1 -- 172.21.15.12:0/2915139712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f5d762bd0 msgr2=0x562f5d771a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:58.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.354+0000 7fe676800640 1 --2- 172.21.15.12:0/2915139712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f5d762bd0 0x562f5d771a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.354+0000 7fe676800640 1 -- 172.21.15.12:0/2915139712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562f5d7621e0 msgr2=0x562f5d771540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:12:58.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.354+0000 7fe676800640 1 --2- 172.21.15.12:0/2915139712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562f5d7621e0 0x562f5d771540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.354+0000 7fe676800640 1 -- 172.21.15.12:0/2915139712 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562f5d773080 con 0x562f5d767d40 2024-03-23T23:12:58.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.355+0000 7fe676800640 1 --2- 172.21.15.12:0/2915139712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f5d767d40 0x562f5d772a90 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fe67000c760 tx=0x7fe670005b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=114cc6dbc9f462fe server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:12:58.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.355+0000 7fe65be00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:12:58.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.355+0000 7fe65be00640 1 -- 172.21.15.12:0/2915139712 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67009a9a0 con 0x562f5d767d40 2024-03-23T23:12:58.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.355+0000 7fe65be00640 1 -- 172.21.15.12:0/2915139712 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe6700ba170 con 0x562f5d767d40 2024-03-23T23:12:58.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.355+0000 7fe67fd9a740 1 -- 172.21.15.12:0/2915139712 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe6600047a0 con 0x562f5d767d40 2024-03-23T23:12:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.355+0000 7fe65be00640 1 -- 172.21.15.12:0/2915139712 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6700b4ba0 con 0x562f5d767d40 2024-03-23T23:12:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.355+0000 7fe67fd9a740 10 client.28427 fetch_fsmap learned FSMap version 23 2024-03-23T23:12:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.355+0000 7fe67fd9a740 1 -- 172.21.15.12:0/2915139712 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x562f5d7d4110 con 0x562f5d767d40 2024-03-23T23:12:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.356+0000 7fe65be00640 1 -- 172.21.15.12:0/2915139712 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6600047a0 con 0x562f5d767d40 2024-03-23T23:12:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.356+0000 7fe65be00640 1 -- 172.21.15.12:0/2915139712 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fe6700a0d50 con 0x562f5d767d40 2024-03-23T23:12:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.356+0000 7fe67fd9a740 10 client.28427 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:12:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.356+0000 7fe67fd9a740 10 client.28427 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:12:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.356+0000 7fe67fd9a740 20 client.28427 populate_metadata read hostname 'smithi012' 2024-03-23T23:12:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.356+0000 7fe67fd9a740 1 --2- 172.21.15.12:0/2915139712 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562f5d7d4a60 0x562f5d7d6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:12:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.356+0000 7fe67fd9a740 4 client.28427 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:12:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.356+0000 7fe67fd9a740 1 -- 172.21.15.12:0/2915139712 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562f5d6cf0c0 con 0x562f5d7d4a60 2024-03-23T23:12:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.358+0000 7fe675e00640 1 --2- 172.21.15.12:0/2915139712 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562f5d7d4a60 0x562f5d7d6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:12:58.360 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.360+0000 7fe675e00640 1 --2- 172.21.15.12:0/2915139712 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562f5d7d4a60 0x562f5d7d6e80 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f313b4ee594c23ea server_cookie=d5c6a91273f5045 in_seq=0 out_seq=0 2024-03-23T23:12:58.360 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.360+0000 7fe65be00640 10 client.28427 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:58.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.372+0000 7fe65be00640 1 -- 172.21.15.12:0/2915139712 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562f5d6cf0c0 con 0x562f5d7d4a60 2024-03-23T23:12:58.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.372+0000 7fe65be00640 10 client.28427 handle_command_reply: tid=0 2024-03-23T23:12:58.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.372+0000 7fe67fd9a740 1 client.28427 shutdown 2024-03-23T23:12:58.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.373+0000 7fe67fd9a740 1 -- 172.21.15.12:0/2915139712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f5d767d40 msgr2=0x562f5d772a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:58.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.373+0000 7fe67fd9a740 1 --2- 172.21.15.12:0/2915139712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f5d767d40 0x562f5d772a90 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fe67000c760 tx=0x7fe670005b20 comp rx=0 tx=0).stop 2024-03-23T23:12:58.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.373+0000 7fe67fd9a740 1 -- 172.21.15.12:0/2915139712 shutdown_connections 2024-03-23T23:12:58.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.373+0000 7fe67fd9a740 1 --2- 172.21.15.12:0/2915139712 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x562f5d7d4a60 0x562f5d7d6e80 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.373+0000 7fe67fd9a740 1 --2- 172.21.15.12:0/2915139712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f5d767d40 0x562f5d772a90 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.373+0000 7fe67fd9a740 1 --2- 172.21.15.12:0/2915139712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f5d762bd0 0x562f5d771a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.373+0000 7fe67fd9a740 1 --2- 172.21.15.12:0/2915139712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562f5d7621e0 0x562f5d771540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.373+0000 7fe67fd9a740 1 -- 172.21.15.12:0/2915139712 >> 172.21.15.12:0/2915139712 conn(0x562f5d75d750 msgr2=0x562f5d7742d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.373+0000 7fe65be00640 0 client.28427 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:12:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.373+0000 7fe67fd9a740 1 -- 172.21.15.12:0/2915139712 shutdown_connections 2024-03-23T23:12:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.373+0000 7fe67fd9a740 1 -- 172.21.15.12:0/2915139712 wait complete. 2024-03-23T23:12:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.373+0000 7fe67fd9a740 20 client.28427 trim_cache size 0 max 16384 2024-03-23T23:12:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:12:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:12:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:12:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:12:58.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.374+0000 7fe67d000640 1 -- 172.21.15.12:0/3498105254 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe65c0a28a0 msgr2=0x7fe65c0a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:58.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.374+0000 7fe67d000640 1 --2- 172.21.15.12:0/3498105254 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe65c0a28a0 0x7fe65c0a4d60 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fe660009710 tx=0x7fe660015040 comp rx=0 tx=0).stop 2024-03-23T23:12:58.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.374+0000 7fe67d000640 1 -- 172.21.15.12:0/3498105254 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6781098b0 msgr2=0x7fe6781a5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:12:58.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.374+0000 7fe67d000640 1 --2- 172.21.15.12:0/3498105254 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6781098b0 0x7fe6781a5520 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fe670009fd0 tx=0x7fe67000ef10 comp rx=0 tx=0).stop 2024-03-23T23:12:58.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.374+0000 7fe67d000640 1 -- 172.21.15.12:0/3498105254 shutdown_connections 2024-03-23T23:12:58.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.374+0000 7fe67d000640 1 --2- 172.21.15.12:0/3498105254 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fe65c0a28a0 0x7fe65c0a4d60 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.374+0000 7fe67d000640 1 --2- 172.21.15.12:0/3498105254 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6781098b0 0x7fe6781a5520 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.374+0000 7fe67d000640 1 --2- 172.21.15.12:0/3498105254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe678108ef0 0x7fe67819e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.374+0000 7fe67d000640 1 --2- 172.21.15.12:0/3498105254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe678107d00 0x7fe67819df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:12:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.374+0000 7fe67d000640 1 -- 172.21.15.12:0/3498105254 >> 172.21.15.12:0/3498105254 conn(0x7fe678076cb0 msgr2=0x7fe67810c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:12:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.375+0000 7fe67d000640 1 -- 172.21.15.12:0/3498105254 shutdown_connections 2024-03-23T23:12:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:12:58.375+0000 7fe67d000640 1 -- 172.21.15.12:0/3498105254 wait complete. 2024-03-23T23:12:59.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:12:59 smithi173 ceph-mon[36116]: pgmap v725: 97 pgs: 97 active+clean; 3.6 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:12:59.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:12:59 smithi203 ceph-mon[36693]: pgmap v725: 97 pgs: 97 active+clean; 3.6 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:12:59.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:12:59 smithi012 ceph-mon[31695]: pgmap v725: 97 pgs: 97 active+clean; 3.6 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:13:01.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:01 smithi173 ceph-mon[36116]: pgmap v726: 97 pgs: 97 active+clean; 3.6 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-03-23T23:13:01.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:01 smithi203 ceph-mon[36693]: pgmap v726: 97 pgs: 97 active+clean; 3.6 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-03-23T23:13:01.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:01 smithi012 ceph-mon[31695]: pgmap v726: 97 pgs: 97 active+clean; 3.6 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-03-23T23:13:02.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:02 smithi173 ceph-mon[36116]: pgmap v727: 97 pgs: 97 active+clean; 3.6 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:13:02.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:02 smithi203 ceph-mon[36693]: pgmap v727: 97 pgs: 97 active+clean; 3.6 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:13:02.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:02 smithi012 ceph-mon[31695]: pgmap v727: 97 pgs: 97 active+clean; 3.6 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:13:02.963 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:13:02.963 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:13:03.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.204+0000 7f2987200640 1 -- 172.21.15.12:0/239948564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2988100750 msgr2=0x7f2988100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:03.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.204+0000 7f2987200640 1 --2- 172.21.15.12:0/239948564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2988100750 0x7f2988100bb0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f2970009900 tx=0x7f297002bc30 comp rx=0 tx=0).stop 2024-03-23T23:13:03.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.204+0000 7f2987200640 1 -- 172.21.15.12:0/239948564 shutdown_connections 2024-03-23T23:13:03.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.204+0000 7f2987200640 1 --2- 172.21.15.12:0/239948564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29881010f0 0x7f2988101570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.204+0000 7f2987200640 1 --2- 172.21.15.12:0/239948564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2988100750 0x7f2988100bb0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.204+0000 7f2987200640 1 --2- 172.21.15.12:0/239948564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29880ffdf0 0x7f2988100210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.204+0000 7f2987200640 1 -- 172.21.15.12:0/239948564 >> 172.21.15.12:0/239948564 conn(0x7f29880fba10 msgr2=0x7f29880fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:03.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.204+0000 7f2987200640 1 -- 172.21.15.12:0/239948564 shutdown_connections 2024-03-23T23:13:03.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.205+0000 7f2987200640 1 -- 172.21.15.12:0/239948564 wait complete. 2024-03-23T23:13:03.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.205+0000 7f2987200640 1 Processor -- start 2024-03-23T23:13:03.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.206+0000 7f2987200640 1 -- start start 2024-03-23T23:13:03.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.206+0000 7f2987200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29880ffdf0 0x7f2988199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:03.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.206+0000 7f2987200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2988100750 0x7f2988199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:03.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.206+0000 7f2987200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29881010f0 0x7f29881a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:03.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.206+0000 7f2987200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298810f3e0 con 0x7f2988100750 2024-03-23T23:13:03.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.206+0000 7f2987200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298810f260 con 0x7f29881010f0 2024-03-23T23:13:03.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.206+0000 7f2987200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298810f560 con 0x7f29880ffdf0 2024-03-23T23:13:03.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.206+0000 7f2985e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29880ffdf0 0x7f2988199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:03.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.206+0000 7f2985e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29880ffdf0 0x7f2988199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44752/0 (socket says 172.21.15.12:44752) 2024-03-23T23:13:03.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.206+0000 7f2985e00640 1 -- 172.21.15.12:0/4014450057 learned_addr learned my addr 172.21.15.12:0/4014450057 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:03.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.206+0000 7f297d400640 1 --2- 172.21.15.12:0/4014450057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2988100750 0x7f2988199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:03.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.206+0000 7f2986800640 1 --2- 172.21.15.12:0/4014450057 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29881010f0 0x7f29881a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:03.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.207+0000 7f2985e00640 1 -- 172.21.15.12:0/4014450057 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29881010f0 msgr2=0x7f29881a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:03.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.207+0000 7f2985e00640 1 --2- 172.21.15.12:0/4014450057 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29881010f0 0x7f29881a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.207+0000 7f2985e00640 1 -- 172.21.15.12:0/4014450057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2988100750 msgr2=0x7f2988199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:03.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.207+0000 7f2985e00640 1 --2- 172.21.15.12:0/4014450057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2988100750 0x7f2988199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.207+0000 7f2985e00640 1 -- 172.21.15.12:0/4014450057 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29881a1670 con 0x7f29880ffdf0 2024-03-23T23:13:03.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.207+0000 7f2985e00640 1 --2- 172.21.15.12:0/4014450057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29880ffdf0 0x7f2988199a50 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f297800e9f0 tx=0x7f297800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:03.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.207+0000 7f297f400640 1 -- 172.21.15.12:0/4014450057 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297800cd80 con 0x7f29880ffdf0 2024-03-23T23:13:03.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.207+0000 7f297f400640 1 -- 172.21.15.12:0/4014450057 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2978005500 con 0x7f29880ffdf0 2024-03-23T23:13:03.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.207+0000 7f2987200640 1 -- 172.21.15.12:0/4014450057 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29881a1960 con 0x7f29880ffdf0 2024-03-23T23:13:03.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.207+0000 7f297f400640 1 -- 172.21.15.12:0/4014450057 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2978010690 con 0x7f29880ffdf0 2024-03-23T23:13:03.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.207+0000 7f2987200640 1 -- 172.21.15.12:0/4014450057 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29881a1e30 con 0x7f29880ffdf0 2024-03-23T23:13:03.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.210+0000 7f297f400640 1 -- 172.21.15.12:0/4014450057 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f29780046c0 con 0x7f29880ffdf0 2024-03-23T23:13:03.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.211+0000 7f298f9e4740 1 Processor -- start 2024-03-23T23:13:03.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.211+0000 7f298f9e4740 1 -- start start 2024-03-23T23:13:03.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.211+0000 7f298f9e4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581e905050 0x55581e905470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:03.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.211+0000 7f298f9e4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55581e905a40 0x55581e90a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:03.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.211+0000 7f298f9e4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55581e90abb0 0x55581e90d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:03.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.211+0000 7f298f9e4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55581e86a250 con 0x55581e905050 2024-03-23T23:13:03.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.211+0000 7f298f9e4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55581e812a30 con 0x55581e90abb0 2024-03-23T23:13:03.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.211+0000 7f298f9e4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55581e8662f0 con 0x55581e905a40 2024-03-23T23:13:03.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.211+0000 7f2986800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581e905050 0x55581e905470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:03.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.211+0000 7f2985e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55581e905a40 0x55581e90a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:03.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.211+0000 7f297d400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55581e90abb0 0x55581e90d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:03.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.212+0000 7f297d400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55581e90abb0 0x55581e90d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57978/0 (socket says 172.21.15.12:57978) 2024-03-23T23:13:03.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.212+0000 7f297d400640 1 -- 172.21.15.12:0/1054715567 learned_addr learned my addr 172.21.15.12:0/1054715567 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:03.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.212+0000 7f297d400640 1 -- 172.21.15.12:0/1054715567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55581e905a40 msgr2=0x55581e90a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:03.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.212+0000 7f297d400640 1 --2- 172.21.15.12:0/1054715567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55581e905a40 0x55581e90a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.212+0000 7f297d400640 1 -- 172.21.15.12:0/1054715567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581e905050 msgr2=0x55581e905470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:03.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.212+0000 7f297d400640 1 --2- 172.21.15.12:0/1054715567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581e905050 0x55581e905470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.212+0000 7f297d400640 1 -- 172.21.15.12:0/1054715567 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55581e7fd690 con 0x55581e90abb0 2024-03-23T23:13:03.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.212+0000 7f2986800640 1 --2- 172.21.15.12:0/1054715567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581e905050 0x55581e905470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:03.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.212+0000 7f297d400640 1 --2- 172.21.15.12:0/1054715567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55581e90abb0 0x55581e90d040 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f2970009900 tx=0x7f29700095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14469dc6ff26c6f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:03.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.212+0000 7f2977000640 1 -- 172.21.15.12:0/1054715567 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297002cb80 con 0x55581e90abb0 2024-03-23T23:13:03.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.213+0000 7f2977000640 1 -- 172.21.15.12:0/1054715567 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2970005040 con 0x55581e90abb0 2024-03-23T23:13:03.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.213+0000 7f2977000640 1 -- 172.21.15.12:0/1054715567 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2970004920 con 0x55581e90abb0 2024-03-23T23:13:03.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.213+0000 7f297f400640 1 --2- 172.21.15.12:0/4014450057 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f296c0a2800 0x7f296c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:03.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.213+0000 7f2986800640 1 --2- 172.21.15.12:0/4014450057 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f296c0a2800 0x7f296c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:03.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.213+0000 7f297f400640 1 -- 172.21.15.12:0/4014450057 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f29780cee90 con 0x7f29880ffdf0 2024-03-23T23:13:03.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.213+0000 7f298f9e4740 1 -- 172.21.15.12:0/1054715567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55581e90abb0 msgr2=0x55581e90d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:03.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.213+0000 7f298f9e4740 1 --2- 172.21.15.12:0/1054715567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55581e90abb0 0x55581e90d040 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f2970009900 tx=0x7f29700095e0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.213+0000 7f298f9e4740 1 -- 172.21.15.12:0/1054715567 shutdown_connections 2024-03-23T23:13:03.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.213+0000 7f298f9e4740 1 --2- 172.21.15.12:0/1054715567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55581e90abb0 0x55581e90d040 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.213+0000 7f298f9e4740 1 --2- 172.21.15.12:0/1054715567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55581e905a40 0x55581e90a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.213+0000 7f298f9e4740 1 --2- 172.21.15.12:0/1054715567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581e905050 0x55581e905470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.213+0000 7f298f9e4740 1 -- 172.21.15.12:0/1054715567 >> 172.21.15.12:0/1054715567 conn(0x55581e8cd200 msgr2=0x55581e9009c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:03.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.214+0000 7f298f9e4740 1 -- 172.21.15.12:0/1054715567 shutdown_connections 2024-03-23T23:13:03.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.214+0000 7f2986800640 1 --2- 172.21.15.12:0/4014450057 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f296c0a2800 0x7f296c0a4cc0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x55581e9062a0 tx=0x7f2980009600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:03.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.214+0000 7f298f9e4740 1 -- 172.21.15.12:0/1054715567 wait complete. 2024-03-23T23:13:03.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.214+0000 7f298f9e4740 1 -- start start 2024-03-23T23:13:03.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.214+0000 7f298f9e4740 1 Processor -- start 2024-03-23T23:13:03.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.215+0000 7f298f9e4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55581e905050 0x55581e90efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:03.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.215+0000 7f298f9e4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581e905a40 0x55581e90f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:03.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.215+0000 7f298f9e4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55581e90abb0 0x55581e916550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:03.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.215+0000 7f298f9e4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55581e90db00 con 0x55581e905a40 2024-03-23T23:13:03.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.215+0000 7f298f9e4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55581e825e80 con 0x55581e90abb0 2024-03-23T23:13:03.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.215+0000 7f298f9e4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55581e90dc80 con 0x55581e905050 2024-03-23T23:13:03.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.215+0000 7f2985e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55581e905050 0x55581e90efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:03.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.215+0000 7f2985e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55581e905050 0x55581e90efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44768/0 (socket says 172.21.15.12:44768) 2024-03-23T23:13:03.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.215+0000 7f2985e00640 1 -- 172.21.15.12:0/582180534 learned_addr learned my addr 172.21.15.12:0/582180534 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:03.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.215+0000 7f298f9e4740 10 client.0 _ll_register_callbacks cb 0x55581e838130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:13:03.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.215+0000 7f2986800640 1 --2- 172.21.15.12:0/582180534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55581e90abb0 0x55581e916550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:03.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.215+0000 7f297d400640 1 --2- 172.21.15.12:0/582180534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581e905a40 0x55581e90f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:03.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.216+0000 7f2985e00640 1 -- 172.21.15.12:0/582180534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55581e90abb0 msgr2=0x55581e916550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:03.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.216+0000 7f2985e00640 1 --2- 172.21.15.12:0/582180534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55581e90abb0 0x55581e916550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.216+0000 7f2985e00640 1 -- 172.21.15.12:0/582180534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581e905a40 msgr2=0x55581e90f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:03.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.216+0000 7f2985e00640 1 --2- 172.21.15.12:0/582180534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581e905a40 0x55581e90f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.216+0000 7f2985e00640 1 -- 172.21.15.12:0/582180534 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55581e6487d0 con 0x55581e905050 2024-03-23T23:13:03.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.216+0000 7f297d400640 1 --2- 172.21.15.12:0/582180534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581e905a40 0x55581e90f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:13:03.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.216+0000 7f2985e00640 1 --2- 172.21.15.12:0/582180534 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55581e905050 0x55581e90efd0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f29780a3a50 tx=0x7f29780973c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e5b535ba20e9dcc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:03.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.216+0000 7f296b400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:13:03.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.216+0000 7f296b400640 1 -- 172.21.15.12:0/582180534 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297809b320 con 0x55581e905050 2024-03-23T23:13:03.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.216+0000 7f296b400640 1 -- 172.21.15.12:0/582180534 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2978097e50 con 0x55581e905050 2024-03-23T23:13:03.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.216+0000 7f298f9e4740 1 -- 172.21.15.12:0/582180534 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2970005040 con 0x55581e905050 2024-03-23T23:13:03.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.217+0000 7f296b400640 1 -- 172.21.15.12:0/582180534 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f297809b140 con 0x55581e905050 2024-03-23T23:13:03.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.217+0000 7f298f9e4740 10 client.28408 fetch_fsmap learned FSMap version 23 2024-03-23T23:13:03.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.217+0000 7f298f9e4740 1 -- 172.21.15.12:0/582180534 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55581e57e3f0 con 0x55581e905050 2024-03-23T23:13:03.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.217+0000 7f296b400640 1 -- 172.21.15.12:0/582180534 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2970005040 con 0x55581e905050 2024-03-23T23:13:03.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.217+0000 7f296b400640 1 -- 172.21.15.12:0/582180534 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f2978086070 con 0x55581e905050 2024-03-23T23:13:03.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.217+0000 7f298f9e4740 10 client.28408 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:13:03.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.217+0000 7f298f9e4740 10 client.28408 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:13:03.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.217+0000 7f298f9e4740 20 client.28408 populate_metadata read hostname 'smithi012' 2024-03-23T23:13:03.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.217+0000 7f298f9e4740 1 --2- 172.21.15.12:0/582180534 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55581e91ae80 0x55581e91d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:03.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.217+0000 7f298f9e4740 4 client.28408 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:13:03.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.217+0000 7f298f9e4740 1 -- 172.21.15.12:0/582180534 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55581e877790 con 0x55581e91ae80 2024-03-23T23:13:03.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.217+0000 7f297d400640 1 --2- 172.21.15.12:0/582180534 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55581e91ae80 0x55581e91d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:03.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.218+0000 7f297d400640 1 --2- 172.21.15.12:0/582180534 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55581e91ae80 0x55581e91d310 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2a29fb47abedfd0 server_cookie=ac13e9836e428836 in_seq=0 out_seq=0 2024-03-23T23:13:03.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.218+0000 7f296b400640 10 client.28408 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:03.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.219+0000 7f296b400640 1 -- 172.21.15.12:0/582180534 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55581e877790 con 0x55581e91ae80 2024-03-23T23:13:03.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.219+0000 7f296b400640 10 client.28408 handle_command_reply: tid=0 2024-03-23T23:13:03.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.219+0000 7f298f9e4740 1 client.28408 shutdown 2024-03-23T23:13:03.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.219+0000 7f298f9e4740 1 -- 172.21.15.12:0/582180534 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55581e905050 msgr2=0x55581e90efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:03.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.219+0000 7f298f9e4740 1 --2- 172.21.15.12:0/582180534 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55581e905050 0x55581e90efd0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f29780a3a50 tx=0x7f29780973c0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.219+0000 7f298f9e4740 1 -- 172.21.15.12:0/582180534 shutdown_connections 2024-03-23T23:13:03.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.219+0000 7f298f9e4740 1 --2- 172.21.15.12:0/582180534 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55581e91ae80 0x55581e91d310 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.219+0000 7f296b400640 0 client.28408 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:03.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.219+0000 7f298f9e4740 1 --2- 172.21.15.12:0/582180534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55581e90abb0 0x55581e916550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.219+0000 7f298f9e4740 1 --2- 172.21.15.12:0/582180534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581e905a40 0x55581e90f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.219+0000 7f298f9e4740 1 --2- 172.21.15.12:0/582180534 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55581e905050 0x55581e90efd0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.219+0000 7f298f9e4740 1 -- 172.21.15.12:0/582180534 >> 172.21.15.12:0/582180534 conn(0x55581e8cd200 msgr2=0x55581e8ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:03.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.219+0000 7f298f9e4740 1 -- 172.21.15.12:0/582180534 shutdown_connections 2024-03-23T23:13:03.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.220+0000 7f298f9e4740 1 -- 172.21.15.12:0/582180534 wait complete. 2024-03-23T23:13:03.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.220+0000 7f298f9e4740 20 client.28408 trim_cache size 0 max 16384 2024-03-23T23:13:03.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.239+0000 7f298f9e4740 1 Processor -- start 2024-03-23T23:13:03.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.239+0000 7f298f9e4740 1 -- start start 2024-03-23T23:13:03.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.239+0000 7f298f9e4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55581e905050 0x55581e9062d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:03.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.240+0000 7f298f9e4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581e905a40 0x55581e9164e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:03.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.240+0000 7f298f9e4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55581e90abb0 0x55581e906810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:03.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.240+0000 7f298f9e4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29881acd60 con 0x55581e905a40 2024-03-23T23:13:03.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.240+0000 7f298f9e4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29881aa860 con 0x55581e905050 2024-03-23T23:13:03.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.240+0000 7f298f9e4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29881a8900 con 0x55581e90abb0 2024-03-23T23:13:03.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.240+0000 7f2985e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55581e905050 0x55581e9062d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:03.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.240+0000 7f2985e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55581e905050 0x55581e9062d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:57990/0 (socket says 172.21.15.12:57990) 2024-03-23T23:13:03.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.240+0000 7f2986800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55581e90abb0 0x55581e906810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:03.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.240+0000 7f2985e00640 1 -- 172.21.15.12:0/2508902137 learned_addr learned my addr 172.21.15.12:0/2508902137 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:03.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.240+0000 7f297d400640 1 --2- 172.21.15.12:0/2508902137 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581e905a40 0x55581e9164e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:03.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.240+0000 7f2985e00640 1 -- 172.21.15.12:0/2508902137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55581e90abb0 msgr2=0x55581e906810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:03.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.240+0000 7f2985e00640 1 --2- 172.21.15.12:0/2508902137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55581e90abb0 0x55581e906810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.240+0000 7f2985e00640 1 -- 172.21.15.12:0/2508902137 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581e905a40 msgr2=0x55581e9164e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:03.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.240+0000 7f2985e00640 1 --2- 172.21.15.12:0/2508902137 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581e905a40 0x55581e9164e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.240+0000 7f2985e00640 1 -- 172.21.15.12:0/2508902137 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55581e53e0f0 con 0x55581e905050 2024-03-23T23:13:03.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.240+0000 7f2986800640 1 --2- 172.21.15.12:0/2508902137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55581e90abb0 0x55581e906810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:03.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.240+0000 7f297d400640 1 --2- 172.21.15.12:0/2508902137 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581e905a40 0x55581e9164e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:13:03.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.241+0000 7f2985e00640 1 --2- 172.21.15.12:0/2508902137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55581e905050 0x55581e9062d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f29780981b0 tx=0x7f29780cf110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8a75afd43f37480 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:03.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.241+0000 7f2976600640 1 -- 172.21.15.12:0/2508902137 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297809b320 con 0x55581e905050 2024-03-23T23:13:03.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.241+0000 7f2976600640 1 -- 172.21.15.12:0/2508902137 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f297809bee0 con 0x55581e905050 2024-03-23T23:13:03.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.241+0000 7f2976600640 1 -- 172.21.15.12:0/2508902137 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29780afc60 con 0x55581e905050 2024-03-23T23:13:03.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.242+0000 7f298f9e4740 1 -- 172.21.15.12:0/2508902137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55581e905050 msgr2=0x55581e9062d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:03.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.242+0000 7f298f9e4740 1 --2- 172.21.15.12:0/2508902137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55581e905050 0x55581e9062d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f29780981b0 tx=0x7f29780cf110 comp rx=0 tx=0).stop 2024-03-23T23:13:03.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.242+0000 7f298f9e4740 1 -- 172.21.15.12:0/2508902137 shutdown_connections 2024-03-23T23:13:03.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.242+0000 7f298f9e4740 1 --2- 172.21.15.12:0/2508902137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55581e90abb0 0x55581e906810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.242+0000 7f298f9e4740 1 --2- 172.21.15.12:0/2508902137 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581e905a40 0x55581e9164e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.242+0000 7f298f9e4740 1 --2- 172.21.15.12:0/2508902137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55581e905050 0x55581e9062d0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.242+0000 7f298f9e4740 1 -- 172.21.15.12:0/2508902137 >> 172.21.15.12:0/2508902137 conn(0x55581e8cd200 msgr2=0x55581e8ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:03.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.242+0000 7f298f9e4740 1 -- 172.21.15.12:0/2508902137 shutdown_connections 2024-03-23T23:13:03.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.242+0000 7f298f9e4740 1 -- 172.21.15.12:0/2508902137 wait complete. 2024-03-23T23:13:03.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.243+0000 7f298f9e4740 1 -- start start 2024-03-23T23:13:03.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.243+0000 7f298f9e4740 1 Processor -- start 2024-03-23T23:13:03.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.243+0000 7f298f9e4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55581e905050 0x55581e913f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:03.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.243+0000 7f298f9e4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581e905a40 0x55581e914450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:03.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.243+0000 7f298f9e4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55581e90abb0 0x55581e9153f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:03.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.243+0000 7f298f9e4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29881a8f00 con 0x55581e905a40 2024-03-23T23:13:03.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.243+0000 7f298f9e4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29881aa200 con 0x55581e90abb0 2024-03-23T23:13:03.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.243+0000 7f298f9e4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29881a9540 con 0x55581e905050 2024-03-23T23:13:03.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.243+0000 7f298f9e4740 10 client.0 _ll_register_callbacks cb 0x7f297800e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:13:03.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.244+0000 7f2985e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55581e905050 0x55581e913f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:03.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.244+0000 7f2986800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55581e90abb0 0x55581e9153f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:03.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.244+0000 7f2985e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55581e905050 0x55581e913f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44788/0 (socket says 172.21.15.12:44788) 2024-03-23T23:13:03.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.244+0000 7f2985e00640 1 -- 172.21.15.12:0/2254987671 learned_addr learned my addr 172.21.15.12:0/2254987671 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:03.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.244+0000 7f297d400640 1 --2- 172.21.15.12:0/2254987671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581e905a40 0x55581e914450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:03.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.244+0000 7f2985e00640 1 -- 172.21.15.12:0/2254987671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55581e90abb0 msgr2=0x55581e9153f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:03.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.244+0000 7f2985e00640 1 --2- 172.21.15.12:0/2254987671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55581e90abb0 0x55581e9153f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.244+0000 7f2985e00640 1 -- 172.21.15.12:0/2254987671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581e905a40 msgr2=0x55581e914450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:03.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.244+0000 7f2985e00640 1 --2- 172.21.15.12:0/2254987671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581e905a40 0x55581e914450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.244+0000 7f2985e00640 1 -- 172.21.15.12:0/2254987671 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55581e9159e0 con 0x55581e905050 2024-03-23T23:13:03.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.244+0000 7f2985e00640 1 --2- 172.21.15.12:0/2254987671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55581e905050 0x55581e913f10 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f29780049d0 tx=0x7f2978004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b90007347dbaa39 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:03.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.244+0000 7f296b400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:13:03.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.244+0000 7f296b400640 1 -- 172.21.15.12:0/2254987671 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297809b320 con 0x55581e905050 2024-03-23T23:13:03.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.244+0000 7f296b400640 1 -- 172.21.15.12:0/2254987671 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f29780afc60 con 0x55581e905050 2024-03-23T23:13:03.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.245+0000 7f298f9e4740 1 -- 172.21.15.12:0/2254987671 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f297809bee0 con 0x55581e905050 2024-03-23T23:13:03.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.245+0000 7f296b400640 1 -- 172.21.15.12:0/2254987671 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f297809b7e0 con 0x55581e905050 2024-03-23T23:13:03.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.245+0000 7f298f9e4740 10 client.28414 fetch_fsmap learned FSMap version 23 2024-03-23T23:13:03.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.245+0000 7f298f9e4740 1 -- 172.21.15.12:0/2254987671 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55581e9769e0 con 0x55581e905050 2024-03-23T23:13:03.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.245+0000 7f296b400640 1 -- 172.21.15.12:0/2254987671 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f297809bee0 con 0x55581e905050 2024-03-23T23:13:03.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.245+0000 7f296b400640 1 -- 172.21.15.12:0/2254987671 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f29780976c0 con 0x55581e905050 2024-03-23T23:13:03.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.245+0000 7f298f9e4740 10 client.28414 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:13:03.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.245+0000 7f298f9e4740 10 client.28414 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:13:03.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.245+0000 7f298f9e4740 20 client.28414 populate_metadata read hostname 'smithi012' 2024-03-23T23:13:03.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.245+0000 7f298f9e4740 1 --2- 172.21.15.12:0/2254987671 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55581e91ae80 0x55581e9790b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:03.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.245+0000 7f298f9e4740 4 client.28414 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:13:03.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.245+0000 7f298f9e4740 1 -- 172.21.15.12:0/2254987671 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55581e871ae0 con 0x55581e91ae80 2024-03-23T23:13:03.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.245+0000 7f297d400640 1 --2- 172.21.15.12:0/2254987671 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55581e91ae80 0x55581e9790b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:03.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.246+0000 7f297d400640 1 --2- 172.21.15.12:0/2254987671 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55581e91ae80 0x55581e9790b0 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e07bf25e50f7c81 server_cookie=3faa59d69337c1c1 in_seq=0 out_seq=0 2024-03-23T23:13:03.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.246+0000 7f296b400640 10 client.28414 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:03.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.246+0000 7f296b400640 1 -- 172.21.15.12:0/2254987671 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55581e871ae0 con 0x55581e91ae80 2024-03-23T23:13:03.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.246+0000 7f296b400640 10 client.28414 handle_command_reply: tid=0 2024-03-23T23:13:03.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.246+0000 7f298f9e4740 1 client.28414 shutdown 2024-03-23T23:13:03.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.246+0000 7f298f9e4740 1 -- 172.21.15.12:0/2254987671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55581e905050 msgr2=0x55581e913f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:03.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.246+0000 7f298f9e4740 1 --2- 172.21.15.12:0/2254987671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55581e905050 0x55581e913f10 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f29780049d0 tx=0x7f2978004a40 comp rx=0 tx=0).stop 2024-03-23T23:13:03.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.246+0000 7f298f9e4740 1 -- 172.21.15.12:0/2254987671 shutdown_connections 2024-03-23T23:13:03.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.246+0000 7f298f9e4740 1 --2- 172.21.15.12:0/2254987671 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55581e91ae80 0x55581e9790b0 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.246+0000 7f298f9e4740 1 --2- 172.21.15.12:0/2254987671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55581e90abb0 0x55581e9153f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.246+0000 7f298f9e4740 1 --2- 172.21.15.12:0/2254987671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581e905a40 0x55581e914450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.246+0000 7f298f9e4740 1 --2- 172.21.15.12:0/2254987671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55581e905050 0x55581e913f10 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.246+0000 7f296b400640 0 client.28414 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:03.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.246+0000 7f298f9e4740 1 -- 172.21.15.12:0/2254987671 >> 172.21.15.12:0/2254987671 conn(0x55581e8cd200 msgr2=0x55581e903c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:03.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.247+0000 7f298f9e4740 1 -- 172.21.15.12:0/2254987671 shutdown_connections 2024-03-23T23:13:03.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.247+0000 7f298f9e4740 1 -- 172.21.15.12:0/2254987671 wait complete. 2024-03-23T23:13:03.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.247+0000 7f298f9e4740 20 client.28414 trim_cache size 0 max 16384 2024-03-23T23:13:03.251 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:13:03.251 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-03-23T23:13:03.251 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-03-23T23:13:03.251 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:13:03.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.247+0000 7f2987200640 1 -- 172.21.15.12:0/4014450057 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f296c0a2800 msgr2=0x7f296c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:03.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.247+0000 7f2987200640 1 --2- 172.21.15.12:0/4014450057 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f296c0a2800 0x7f296c0a4cc0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x55581e9062a0 tx=0x7f2980009600 comp rx=0 tx=0).stop 2024-03-23T23:13:03.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.247+0000 7f2987200640 1 -- 172.21.15.12:0/4014450057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29880ffdf0 msgr2=0x7f2988199a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:03.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.247+0000 7f2987200640 1 --2- 172.21.15.12:0/4014450057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29880ffdf0 0x7f2988199a50 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f297800e9f0 tx=0x7f297800edf0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.247+0000 7f2987200640 1 -- 172.21.15.12:0/4014450057 shutdown_connections 2024-03-23T23:13:03.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.247+0000 7f2987200640 1 --2- 172.21.15.12:0/4014450057 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f296c0a2800 0x7f296c0a4cc0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.247+0000 7f2987200640 1 --2- 172.21.15.12:0/4014450057 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29881010f0 0x7f29881a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.247+0000 7f2987200640 1 --2- 172.21.15.12:0/4014450057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2988100750 0x7f2988199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.247+0000 7f2987200640 1 --2- 172.21.15.12:0/4014450057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29880ffdf0 0x7f2988199a50 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.248+0000 7f2987200640 1 -- 172.21.15.12:0/4014450057 >> 172.21.15.12:0/4014450057 conn(0x7f29880fba10 msgr2=0x7f29880fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:03.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.248+0000 7f2987200640 1 -- 172.21.15.12:0/4014450057 shutdown_connections 2024-03-23T23:13:03.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.248+0000 7f2987200640 1 -- 172.21.15.12:0/4014450057 wait complete. 2024-03-23T23:13:03.268 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T23:13:03.268 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:13:03.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.493+0000 7fce20800640 1 -- 172.21.15.12:0/42417542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce18105490 msgr2=0x7fce1810bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:03.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.493+0000 7fce20800640 1 --2- 172.21.15.12:0/42417542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce18105490 0x7fce1810bd20 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fce14009be0 tx=0x7fce1402bef0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.493+0000 7fce20800640 1 -- 172.21.15.12:0/42417542 shutdown_connections 2024-03-23T23:13:03.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.493+0000 7fce20800640 1 --2- 172.21.15.12:0/42417542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce18105490 0x7fce1810bd20 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.493+0000 7fce20800640 1 --2- 172.21.15.12:0/42417542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce18104ad0 0x7fce18104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.493+0000 7fce20800640 1 --2- 172.21.15.12:0/42417542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce181038e0 0x7fce18103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.493+0000 7fce20800640 1 -- 172.21.15.12:0/42417542 >> 172.21.15.12:0/42417542 conn(0x7fce180fee90 msgr2=0x7fce181012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:03.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.493+0000 7fce20800640 1 -- 172.21.15.12:0/42417542 shutdown_connections 2024-03-23T23:13:03.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.493+0000 7fce20800640 1 -- 172.21.15.12:0/42417542 wait complete. 2024-03-23T23:13:03.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.494+0000 7fce20800640 1 Processor -- start 2024-03-23T23:13:03.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.494+0000 7fce20800640 1 -- start start 2024-03-23T23:13:03.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.495+0000 7fce20800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce181038e0 0x7fce18199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:03.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.495+0000 7fce20800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce18104ad0 0x7fce1819a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:03.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.495+0000 7fce20800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce18105490 0x7fce181a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:03.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.495+0000 7fce20800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1810f590 con 0x7fce18105490 2024-03-23T23:13:03.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.495+0000 7fce20800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1810f410 con 0x7fce181038e0 2024-03-23T23:13:03.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.495+0000 7fce20800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1810f710 con 0x7fce18104ad0 2024-03-23T23:13:03.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.495+0000 7fce1d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce18104ad0 0x7fce1819a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:03.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.495+0000 7fce1de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce181038e0 0x7fce18199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:03.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.495+0000 7fce1d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce18104ad0 0x7fce1819a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44796/0 (socket says 172.21.15.12:44796) 2024-03-23T23:13:03.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.495+0000 7fce1d400640 1 -- 172.21.15.12:0/2052993134 learned_addr learned my addr 172.21.15.12:0/2052993134 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:03.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.495+0000 7fce1e800640 1 --2- 172.21.15.12:0/2052993134 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce18105490 0x7fce181a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:03.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.496+0000 7fce1d400640 1 -- 172.21.15.12:0/2052993134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce181038e0 msgr2=0x7fce18199b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:03.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.496+0000 7fce1d400640 1 --2- 172.21.15.12:0/2052993134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce181038e0 0x7fce18199b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.496+0000 7fce1d400640 1 -- 172.21.15.12:0/2052993134 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce18105490 msgr2=0x7fce181a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:03.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.496+0000 7fce1d400640 1 --2- 172.21.15.12:0/2052993134 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce18105490 0x7fce181a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.496+0000 7fce1d400640 1 -- 172.21.15.12:0/2052993134 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce181a1770 con 0x7fce18104ad0 2024-03-23T23:13:03.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.496+0000 7fce1de00640 1 --2- 172.21.15.12:0/2052993134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce181038e0 0x7fce18199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:03.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.496+0000 7fce1d400640 1 --2- 172.21.15.12:0/2052993134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce18104ad0 0x7fce1819a090 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fce0800e9c0 tx=0x7fce0800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:03.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.496+0000 7fce12a00640 1 -- 172.21.15.12:0/2052993134 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0800ccc0 con 0x7fce18104ad0 2024-03-23T23:13:03.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.496+0000 7fce12a00640 1 -- 172.21.15.12:0/2052993134 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce08005500 con 0x7fce18104ad0 2024-03-23T23:13:03.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.496+0000 7fce12a00640 1 -- 172.21.15.12:0/2052993134 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce08010430 con 0x7fce18104ad0 2024-03-23T23:13:03.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.496+0000 7fce20800640 1 -- 172.21.15.12:0/2052993134 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce181a1a60 con 0x7fce18104ad0 2024-03-23T23:13:03.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.496+0000 7fce20800640 1 -- 172.21.15.12:0/2052993134 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce181a2000 con 0x7fce18104ad0 2024-03-23T23:13:03.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.497+0000 7fce20800640 1 -- 172.21.15.12:0/2052993134 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce1810aba0 con 0x7fce18104ad0 2024-03-23T23:13:03.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.499+0000 7fce12a00640 1 -- 172.21.15.12:0/2052993134 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fce080046c0 con 0x7fce18104ad0 2024-03-23T23:13:03.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.501+0000 7fce12a00640 1 --2- 172.21.15.12:0/2052993134 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fce000a2880 0x7fce000a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:03.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.501+0000 7fce12a00640 1 -- 172.21.15.12:0/2052993134 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fce080ced20 con 0x7fce18104ad0 2024-03-23T23:13:03.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.501+0000 7fce1de00640 1 --2- 172.21.15.12:0/2052993134 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fce000a2880 0x7fce000a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:03.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.502+0000 7fce1de00640 1 --2- 172.21.15.12:0/2052993134 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fce000a2880 0x7fce000a4d40 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fce18104930 tx=0x7fce0c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:03.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.505+0000 7fce12a00640 1 -- 172.21.15.12:0/2052993134 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fce0809bd70 con 0x7fce18104ad0 2024-03-23T23:13:03.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.753+0000 7fce20800640 1 -- 172.21.15.12:0/2052993134 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce1802d070 con 0x7fce18104ad0 2024-03-23T23:13:03.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.755+0000 7fce12a00640 1 -- 172.21.15.12:0/2052993134 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7fce0809b670 con 0x7fce18104ad0 2024-03-23T23:13:03.755 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:13:03.755 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:13:03.755 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:13:03.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.758+0000 7fce20800640 1 -- 172.21.15.12:0/2052993134 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fce000a2880 msgr2=0x7fce000a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:03.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.758+0000 7fce20800640 1 --2- 172.21.15.12:0/2052993134 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fce000a2880 0x7fce000a4d40 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fce18104930 tx=0x7fce0c005f90 comp rx=0 tx=0).stop 2024-03-23T23:13:03.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.758+0000 7fce20800640 1 -- 172.21.15.12:0/2052993134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce18104ad0 msgr2=0x7fce1819a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:03.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.758+0000 7fce20800640 1 --2- 172.21.15.12:0/2052993134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce18104ad0 0x7fce1819a090 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fce0800e9c0 tx=0x7fce0800ee80 comp rx=0 tx=0).stop 2024-03-23T23:13:03.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.758+0000 7fce20800640 1 -- 172.21.15.12:0/2052993134 shutdown_connections 2024-03-23T23:13:03.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.758+0000 7fce20800640 1 --2- 172.21.15.12:0/2052993134 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fce000a2880 0x7fce000a4d40 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.758+0000 7fce20800640 1 --2- 172.21.15.12:0/2052993134 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce18105490 0x7fce181a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.758+0000 7fce20800640 1 --2- 172.21.15.12:0/2052993134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce18104ad0 0x7fce1819a090 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.758+0000 7fce20800640 1 --2- 172.21.15.12:0/2052993134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce181038e0 0x7fce18199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:03.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.758+0000 7fce20800640 1 -- 172.21.15.12:0/2052993134 >> 172.21.15.12:0/2052993134 conn(0x7fce180fee90 msgr2=0x7fce18107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:03.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.758+0000 7fce20800640 1 -- 172.21.15.12:0/2052993134 shutdown_connections 2024-03-23T23:13:03.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:03.758+0000 7fce20800640 1 -- 172.21.15.12:0/2052993134 wait complete. 2024-03-23T23:13:03.778 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:13:03.779 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-23T23:13:04.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.003+0000 7f2318800640 1 -- 172.21.15.12:0/3441778149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23100772a0 msgr2=0x7f2310077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.003+0000 7f2318800640 1 --2- 172.21.15.12:0/3441778149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23100772a0 0x7f2310077700 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f2300009900 tx=0x7f230002bc40 comp rx=0 tx=0).stop 2024-03-23T23:13:04.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.003+0000 7f2318800640 1 -- 172.21.15.12:0/3441778149 shutdown_connections 2024-03-23T23:13:04.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.003+0000 7f2318800640 1 --2- 172.21.15.12:0/3441778149 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2310077c40 0x7f23100784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.003+0000 7f2318800640 1 --2- 172.21.15.12:0/3441778149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23100772a0 0x7f2310077700 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.003+0000 7f2318800640 1 --2- 172.21.15.12:0/3441778149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2310076060 0x7f2310076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.003+0000 7f2318800640 1 -- 172.21.15.12:0/3441778149 >> 172.21.15.12:0/3441778149 conn(0x7f23100fee50 msgr2=0x7f2310101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:04.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.004+0000 7f2318800640 1 -- 172.21.15.12:0/3441778149 shutdown_connections 2024-03-23T23:13:04.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.004+0000 7f2318800640 1 -- 172.21.15.12:0/3441778149 wait complete. 2024-03-23T23:13:04.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.005+0000 7f2318800640 1 Processor -- start 2024-03-23T23:13:04.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.005+0000 7f2318800640 1 -- start start 2024-03-23T23:13:04.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.005+0000 7f2318800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2310076060 0x7f231019dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.005+0000 7f2318800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23100772a0 0x7f231019e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.005+0000 7f2318800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2310077c40 0x7f23101a55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.005+0000 7f2318800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23101139b0 con 0x7f23100772a0 2024-03-23T23:13:04.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.005+0000 7f2318800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2310113830 con 0x7f2310076060 2024-03-23T23:13:04.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.005+0000 7f2318800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2310113b30 con 0x7f2310077c40 2024-03-23T23:13:04.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.005+0000 7f2315e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2310076060 0x7f231019dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.005+0000 7f2315e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2310076060 0x7f231019dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:58022/0 (socket says 172.21.15.12:58022) 2024-03-23T23:13:04.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.005+0000 7f2315e00640 1 -- 172.21.15.12:0/3940831101 learned_addr learned my addr 172.21.15.12:0/3940831101 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:04.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.005+0000 7f2316800640 1 --2- 172.21.15.12:0/3940831101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2310077c40 0x7f23101a55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.005+0000 7f2315400640 1 --2- 172.21.15.12:0/3940831101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23100772a0 0x7f231019e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.006+0000 7f2315e00640 1 -- 172.21.15.12:0/3940831101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2310077c40 msgr2=0x7f23101a55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.006+0000 7f2315e00640 1 --2- 172.21.15.12:0/3940831101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2310077c40 0x7f23101a55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.006+0000 7f2315e00640 1 -- 172.21.15.12:0/3940831101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23100772a0 msgr2=0x7f231019e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.006+0000 7f2315e00640 1 --2- 172.21.15.12:0/3940831101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23100772a0 0x7f231019e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.006+0000 7f2315e00640 1 -- 172.21.15.12:0/3940831101 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23101a5c10 con 0x7f2310076060 2024-03-23T23:13:04.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.006+0000 7f2316800640 1 --2- 172.21.15.12:0/3940831101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2310077c40 0x7f23101a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:04.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.006+0000 7f2315e00640 1 --2- 172.21.15.12:0/3940831101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2310076060 0x7f231019dff0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f230c00ebc0 tx=0x7f230c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:04.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.006+0000 7f2306a00640 1 -- 172.21.15.12:0/3940831101 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230c010630 con 0x7f2310076060 2024-03-23T23:13:04.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.006+0000 7f2306a00640 1 -- 172.21.15.12:0/3940831101 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f230c010cb0 con 0x7f2310076060 2024-03-23T23:13:04.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.006+0000 7f2306a00640 1 -- 172.21.15.12:0/3940831101 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230c018bd0 con 0x7f2310076060 2024-03-23T23:13:04.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.007+0000 7f2318800640 1 -- 172.21.15.12:0/3940831101 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23101a5f00 con 0x7f2310076060 2024-03-23T23:13:04.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.007+0000 7f2318800640 1 -- 172.21.15.12:0/3940831101 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23101a6310 con 0x7f2310076060 2024-03-23T23:13:04.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.010+0000 7f2306a00640 1 -- 172.21.15.12:0/3940831101 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f230c0107d0 con 0x7f2310076060 2024-03-23T23:13:04.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.010+0000 7f231b5e0740 1 Processor -- start 2024-03-23T23:13:04.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.010+0000 7f231b5e0740 1 -- start start 2024-03-23T23:13:04.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.010+0000 7f231b5e0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dbc5014810 0x55dbc5014c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.010+0000 7f231b5e0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbc50159e0 0x55dbc5015e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.010+0000 7f231b5e0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc50163a0 0x55dbc501cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.010+0000 7f231b5e0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbc4f792c0 con 0x55dbc50159e0 2024-03-23T23:13:04.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.010+0000 7f231b5e0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbc4f21c90 con 0x55dbc50163a0 2024-03-23T23:13:04.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.010+0000 7f231b5e0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbc4f75360 con 0x55dbc5014810 2024-03-23T23:13:04.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.011+0000 7f2315400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc50163a0 0x55dbc501cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.011+0000 7f2315400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc50163a0 0x55dbc501cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:58032/0 (socket says 172.21.15.12:58032) 2024-03-23T23:13:04.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.011+0000 7f2315400640 1 -- 172.21.15.12:0/3301267974 learned_addr learned my addr 172.21.15.12:0/3301267974 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:04.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.011+0000 7f2316800640 1 --2- 172.21.15.12:0/3301267974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dbc5014810 0x55dbc5014c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.011+0000 7f2315e00640 1 --2- 172.21.15.12:0/3301267974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbc50159e0 0x55dbc5015e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.011+0000 7f2315400640 1 -- 172.21.15.12:0/3301267974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dbc5014810 msgr2=0x55dbc5014c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:04.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.011+0000 7f2315400640 1 --2- 172.21.15.12:0/3301267974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dbc5014810 0x55dbc5014c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.011+0000 7f2315400640 1 -- 172.21.15.12:0/3301267974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbc50159e0 msgr2=0x55dbc5015e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:04.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.011+0000 7f2315400640 1 --2- 172.21.15.12:0/3301267974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbc50159e0 0x55dbc5015e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.011+0000 7f2315400640 1 -- 172.21.15.12:0/3301267974 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dbc4f46080 con 0x55dbc50163a0 2024-03-23T23:13:04.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.011+0000 7f2306a00640 1 --2- 172.21.15.12:0/3940831101 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f22fc0a2880 0x7f22fc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.011+0000 7f2315400640 1 --2- 172.21.15.12:0/3301267974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc50163a0 0x55dbc501cc30 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f2300005ab0 tx=0x7f2300000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3aafc32c67aeefdc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:04.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.012+0000 7f2306a00640 1 -- 172.21.15.12:0/3940831101 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f230c0cefd0 con 0x7f2310076060 2024-03-23T23:13:04.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.012+0000 7f22fb000640 1 -- 172.21.15.12:0/3301267974 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2300042070 con 0x55dbc50163a0 2024-03-23T23:13:04.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.012+0000 7f2316800640 1 --2- 172.21.15.12:0/3940831101 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f22fc0a2880 0x7f22fc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.012+0000 7f22fb000640 1 -- 172.21.15.12:0/3301267974 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f230002cb90 con 0x55dbc50163a0 2024-03-23T23:13:04.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.012+0000 7f22fb000640 1 -- 172.21.15.12:0/3301267974 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2300004ed0 con 0x55dbc50163a0 2024-03-23T23:13:04.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.012+0000 7f231b5e0740 1 -- 172.21.15.12:0/3301267974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc50163a0 msgr2=0x55dbc501cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.012+0000 7f231b5e0740 1 --2- 172.21.15.12:0/3301267974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc50163a0 0x55dbc501cc30 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f2300005ab0 tx=0x7f2300000bc0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.012+0000 7f2316800640 1 --2- 172.21.15.12:0/3940831101 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f22fc0a2880 0x7f22fc0a4d40 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x55dbc50158e0 tx=0x7f230800b2e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:04.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.013+0000 7f231b5e0740 1 -- 172.21.15.12:0/3301267974 shutdown_connections 2024-03-23T23:13:04.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.013+0000 7f231b5e0740 1 --2- 172.21.15.12:0/3301267974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc50163a0 0x55dbc501cc30 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.013+0000 7f231b5e0740 1 --2- 172.21.15.12:0/3301267974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbc50159e0 0x55dbc5015e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.013+0000 7f231b5e0740 1 --2- 172.21.15.12:0/3301267974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dbc5014810 0x55dbc5014c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.013+0000 7f231b5e0740 1 -- 172.21.15.12:0/3301267974 >> 172.21.15.12:0/3301267974 conn(0x55dbc500fc00 msgr2=0x55dbc5010000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:04.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.013+0000 7f231b5e0740 1 -- 172.21.15.12:0/3301267974 shutdown_connections 2024-03-23T23:13:04.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.013+0000 7f231b5e0740 1 -- 172.21.15.12:0/3301267974 wait complete. 2024-03-23T23:13:04.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.013+0000 7f231b5e0740 1 -- start start 2024-03-23T23:13:04.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.013+0000 7f231b5e0740 1 Processor -- start 2024-03-23T23:13:04.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.014+0000 7f231b5e0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc5014810 0x55dbc501e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.014+0000 7f231b5e0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbc50159e0 0x55dbc501ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.014+0000 7f231b5e0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dbc50163a0 0x55dbc5025f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.014+0000 7f231b5e0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbc501d350 con 0x55dbc50159e0 2024-03-23T23:13:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.014+0000 7f231b5e0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbc4f350d0 con 0x55dbc5014810 2024-03-23T23:13:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.014+0000 7f231b5e0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbc501d4d0 con 0x55dbc50163a0 2024-03-23T23:13:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.014+0000 7f2315e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc5014810 0x55dbc501e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.014+0000 7f2315e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc5014810 0x55dbc501e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:58042/0 (socket says 172.21.15.12:58042) 2024-03-23T23:13:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.014+0000 7f2315e00640 1 -- 172.21.15.12:0/1792624939 learned_addr learned my addr 172.21.15.12:0/1792624939 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.014+0000 7f231b5e0740 10 client.0 _ll_register_callbacks cb 0x55dbc4f39e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:13:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.014+0000 7f2316800640 1 --2- 172.21.15.12:0/1792624939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dbc50163a0 0x55dbc5025f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.014+0000 7f2315400640 1 --2- 172.21.15.12:0/1792624939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbc50159e0 0x55dbc501ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.014+0000 7f2315e00640 1 -- 172.21.15.12:0/1792624939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dbc50163a0 msgr2=0x55dbc5025f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.014+0000 7f2315e00640 1 --2- 172.21.15.12:0/1792624939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dbc50163a0 0x55dbc5025f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.015+0000 7f2315e00640 1 -- 172.21.15.12:0/1792624939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbc50159e0 msgr2=0x55dbc501ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.015+0000 7f2315e00640 1 --2- 172.21.15.12:0/1792624939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbc50159e0 0x55dbc501ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.015+0000 7f2315e00640 1 -- 172.21.15.12:0/1792624939 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dbc4c8d3f0 con 0x55dbc5014810 2024-03-23T23:13:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.015+0000 7f2315400640 1 --2- 172.21.15.12:0/1792624939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbc50159e0 0x55dbc501ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:13:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.015+0000 7f2315e00640 1 --2- 172.21.15.12:0/1792624939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc5014810 0x55dbc501e9f0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f230c00eb90 tx=0x7f230c097d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6abec56b7a2cc950 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.015+0000 7f22ef400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:13:04.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.015+0000 7f22ef400640 1 -- 172.21.15.12:0/1792624939 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230c010e60 con 0x55dbc5014810 2024-03-23T23:13:04.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.015+0000 7f22ef400640 1 -- 172.21.15.12:0/1792624939 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f230c0d1070 con 0x55dbc5014810 2024-03-23T23:13:04.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.015+0000 7f231b5e0740 1 -- 172.21.15.12:0/1792624939 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f230002cb90 con 0x55dbc5014810 2024-03-23T23:13:04.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.015+0000 7f22ef400640 1 -- 172.21.15.12:0/1792624939 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f230c0d26e0 con 0x55dbc5014810 2024-03-23T23:13:04.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.015+0000 7f231b5e0740 10 client.28463 fetch_fsmap learned FSMap version 23 2024-03-23T23:13:04.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.015+0000 7f231b5e0740 1 -- 172.21.15.12:0/1792624939 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55dbc4d577d0 con 0x55dbc5014810 2024-03-23T23:13:04.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.016+0000 7f22ef400640 1 -- 172.21.15.12:0/1792624939 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f230002cb90 con 0x55dbc5014810 2024-03-23T23:13:04.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.016+0000 7f22ef400640 1 -- 172.21.15.12:0/1792624939 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f230c09ae80 con 0x55dbc5014810 2024-03-23T23:13:04.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.016+0000 7f231b5e0740 10 client.28463 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:13:04.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.016+0000 7f231b5e0740 10 client.28463 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:13:04.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.016+0000 7f231b5e0740 20 client.28463 populate_metadata read hostname 'smithi012' 2024-03-23T23:13:04.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.016+0000 7f231b5e0740 1 --2- 172.21.15.12:0/1792624939 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55dbc502a360 0x55dbc502c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.016+0000 7f231b5e0740 4 client.28463 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:13:04.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.016+0000 7f231b5e0740 1 -- 172.21.15.12:0/1792624939 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dbc4f86800 con 0x55dbc502a360 2024-03-23T23:13:04.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.016+0000 7f2315400640 1 --2- 172.21.15.12:0/1792624939 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55dbc502a360 0x55dbc502c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.017+0000 7f2315400640 1 --2- 172.21.15.12:0/1792624939 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55dbc502a360 0x55dbc502c7c0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=57fd8c79ae2e6a1f server_cookie=73694c09be5ad785 in_seq=0 out_seq=0 2024-03-23T23:13:04.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.017+0000 7f22ef400640 10 client.28463 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:13:04.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.017+0000 7f22ef400640 1 -- 172.21.15.12:0/1792624939 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55dbc4f86800 con 0x55dbc502a360 2024-03-23T23:13:04.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.017+0000 7f22ef400640 10 client.28463 handle_command_reply: tid=0 2024-03-23T23:13:04.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.017+0000 7f231b5e0740 1 client.28463 shutdown 2024-03-23T23:13:04.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.017+0000 7f231b5e0740 1 -- 172.21.15.12:0/1792624939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc5014810 msgr2=0x55dbc501e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.017+0000 7f231b5e0740 1 --2- 172.21.15.12:0/1792624939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc5014810 0x55dbc501e9f0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f230c00eb90 tx=0x7f230c097d90 comp rx=0 tx=0).stop 2024-03-23T23:13:04.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.018+0000 7f231b5e0740 1 -- 172.21.15.12:0/1792624939 shutdown_connections 2024-03-23T23:13:04.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.018+0000 7f231b5e0740 1 --2- 172.21.15.12:0/1792624939 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55dbc502a360 0x55dbc502c7c0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.018+0000 7f231b5e0740 1 --2- 172.21.15.12:0/1792624939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dbc50163a0 0x55dbc5025f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.018+0000 7f231b5e0740 1 --2- 172.21.15.12:0/1792624939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbc50159e0 0x55dbc501ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.018+0000 7f22ef400640 0 client.28463 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:13:04.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.018+0000 7f231b5e0740 1 --2- 172.21.15.12:0/1792624939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc5014810 0x55dbc501e9f0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.018+0000 7f231b5e0740 1 -- 172.21.15.12:0/1792624939 >> 172.21.15.12:0/1792624939 conn(0x55dbc500fc00 msgr2=0x55dbc500eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:04.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.018+0000 7f231b5e0740 1 -- 172.21.15.12:0/1792624939 shutdown_connections 2024-03-23T23:13:04.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.018+0000 7f231b5e0740 1 -- 172.21.15.12:0/1792624939 wait complete. 2024-03-23T23:13:04.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.018+0000 7f231b5e0740 20 client.28463 trim_cache size 0 max 16384 2024-03-23T23:13:04.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.032+0000 7f231b5e0740 1 Processor -- start 2024-03-23T23:13:04.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.032+0000 7f231b5e0740 1 -- start start 2024-03-23T23:13:04.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.032+0000 7f231b5e0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbc5014810 0x55dbc500ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.032+0000 7f231b5e0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dbc50159e0 0x55dbc5025f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.032+0000 7f231b5e0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc50163a0 0x55dbc5031080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.032+0000 7f231b5e0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23101b1520 con 0x55dbc5014810 2024-03-23T23:13:04.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.032+0000 7f231b5e0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23101af020 con 0x55dbc50163a0 2024-03-23T23:13:04.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.032+0000 7f231b5e0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23101ad0c0 con 0x55dbc50159e0 2024-03-23T23:13:04.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.032+0000 7f2316800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc50163a0 0x55dbc5031080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.032+0000 7f2315400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dbc50159e0 0x55dbc5025f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.032+0000 7f2316800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc50163a0 0x55dbc5031080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:58050/0 (socket says 172.21.15.12:58050) 2024-03-23T23:13:04.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.033+0000 7f2316800640 1 -- 172.21.15.12:0/2210774885 learned_addr learned my addr 172.21.15.12:0/2210774885 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:04.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.033+0000 7f2315e00640 1 --2- 172.21.15.12:0/2210774885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbc5014810 0x55dbc500ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.033+0000 7f2315400640 1 -- 172.21.15.12:0/2210774885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc50163a0 msgr2=0x55dbc5031080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:04.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.033+0000 7f2315400640 1 --2- 172.21.15.12:0/2210774885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc50163a0 0x55dbc5031080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.033+0000 7f2315400640 1 -- 172.21.15.12:0/2210774885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbc5014810 msgr2=0x55dbc500ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:04.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.033+0000 7f2315400640 1 --2- 172.21.15.12:0/2210774885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbc5014810 0x55dbc500ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.033+0000 7f2315400640 1 -- 172.21.15.12:0/2210774885 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dbc502a140 con 0x55dbc50159e0 2024-03-23T23:13:04.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.033+0000 7f2316800640 1 --2- 172.21.15.12:0/2210774885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc50163a0 0x55dbc5031080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:04.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.033+0000 7f2315400640 1 --2- 172.21.15.12:0/2210774885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dbc50159e0 0x55dbc5025f30 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f230002c7c0 tx=0x7f2300005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5e385dc4147f320 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:04.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.033+0000 7f22fa600640 1 -- 172.21.15.12:0/2210774885 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2300042040 con 0x55dbc50159e0 2024-03-23T23:13:04.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.033+0000 7f22fa600640 1 -- 172.21.15.12:0/2210774885 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f230002cb90 con 0x55dbc50159e0 2024-03-23T23:13:04.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.033+0000 7f22fa600640 1 -- 172.21.15.12:0/2210774885 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23000046c0 con 0x55dbc50159e0 2024-03-23T23:13:04.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.034+0000 7f231b5e0740 1 -- 172.21.15.12:0/2210774885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dbc50159e0 msgr2=0x55dbc5025f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.034+0000 7f231b5e0740 1 --2- 172.21.15.12:0/2210774885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dbc50159e0 0x55dbc5025f30 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f230002c7c0 tx=0x7f2300005fd0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.034+0000 7f231b5e0740 1 -- 172.21.15.12:0/2210774885 shutdown_connections 2024-03-23T23:13:04.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.034+0000 7f231b5e0740 1 --2- 172.21.15.12:0/2210774885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc50163a0 0x55dbc5031080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.034+0000 7f231b5e0740 1 --2- 172.21.15.12:0/2210774885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dbc50159e0 0x55dbc5025f30 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.034+0000 7f231b5e0740 1 --2- 172.21.15.12:0/2210774885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbc5014810 0x55dbc500ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.034+0000 7f231b5e0740 1 -- 172.21.15.12:0/2210774885 >> 172.21.15.12:0/2210774885 conn(0x55dbc500fc00 msgr2=0x55dbc500e620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:04.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.034+0000 7f231b5e0740 1 -- 172.21.15.12:0/2210774885 shutdown_connections 2024-03-23T23:13:04.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.034+0000 7f231b5e0740 1 -- 172.21.15.12:0/2210774885 wait complete. 2024-03-23T23:13:04.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.035+0000 7f231b5e0740 1 -- start start 2024-03-23T23:13:04.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.035+0000 7f231b5e0740 1 Processor -- start 2024-03-23T23:13:04.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.035+0000 7f231b5e0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc5014810 0x55dbc5025db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.035+0000 7f231b5e0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dbc50159e0 0x55dbc5024410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.035+0000 7f231b5e0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbc50163a0 0x55dbc5024950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.035+0000 7f231b5e0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23101ad6c0 con 0x55dbc50163a0 2024-03-23T23:13:04.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.035+0000 7f231b5e0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23101ae9c0 con 0x55dbc5014810 2024-03-23T23:13:04.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.035+0000 7f231b5e0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23101add00 con 0x55dbc50159e0 2024-03-23T23:13:04.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.035+0000 7f231b5e0740 10 client.0 _ll_register_callbacks cb 0x7f230c0d25c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:13:04.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.035+0000 7f2315e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc5014810 0x55dbc5025db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.035+0000 7f2315400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dbc50159e0 0x55dbc5024410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.035+0000 7f2315e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc5014810 0x55dbc5025db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:58060/0 (socket says 172.21.15.12:58060) 2024-03-23T23:13:04.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.035+0000 7f2315e00640 1 -- 172.21.15.12:0/2081886592 learned_addr learned my addr 172.21.15.12:0/2081886592 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:04.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.036+0000 7f2316800640 1 --2- 172.21.15.12:0/2081886592 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbc50163a0 0x55dbc5024950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.036+0000 7f2315e00640 1 -- 172.21.15.12:0/2081886592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dbc50159e0 msgr2=0x55dbc5024410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:04.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.036+0000 7f2315e00640 1 --2- 172.21.15.12:0/2081886592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dbc50159e0 0x55dbc5024410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.036+0000 7f2315e00640 1 -- 172.21.15.12:0/2081886592 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbc50163a0 msgr2=0x55dbc5024950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:04.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.036+0000 7f2315e00640 1 --2- 172.21.15.12:0/2081886592 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbc50163a0 0x55dbc5024950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.036+0000 7f2315e00640 1 -- 172.21.15.12:0/2081886592 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dbc5025180 con 0x55dbc5014810 2024-03-23T23:13:04.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.036+0000 7f2315400640 1 --2- 172.21.15.12:0/2081886592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dbc50159e0 0x55dbc5024410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:04.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.036+0000 7f2315e00640 1 --2- 172.21.15.12:0/2081886592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc5014810 0x55dbc5025db0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f230c09e040 tx=0x7f230c0861a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54e268ece524f92d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:04.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.036+0000 7f22ef400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:13:04.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.036+0000 7f22ef400640 1 -- 172.21.15.12:0/2081886592 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230c010e60 con 0x55dbc5014810 2024-03-23T23:13:04.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.036+0000 7f22ef400640 1 -- 172.21.15.12:0/2081886592 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f230c097980 con 0x55dbc5014810 2024-03-23T23:13:04.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.036+0000 7f231b5e0740 1 -- 172.21.15.12:0/2081886592 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f230002cb90 con 0x55dbc5014810 2024-03-23T23:13:04.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.037+0000 7f22ef400640 1 -- 172.21.15.12:0/2081886592 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f230c09fad0 con 0x55dbc5014810 2024-03-23T23:13:04.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.037+0000 7f231b5e0740 10 client.28469 fetch_fsmap learned FSMap version 23 2024-03-23T23:13:04.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.037+0000 7f231b5e0740 1 -- 172.21.15.12:0/2081886592 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55dbc5086c90 con 0x55dbc5014810 2024-03-23T23:13:04.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.037+0000 7f22ef400640 1 -- 172.21.15.12:0/2081886592 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f230002cb90 con 0x55dbc5014810 2024-03-23T23:13:04.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.037+0000 7f22ef400640 1 -- 172.21.15.12:0/2081886592 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f230c09a9f0 con 0x55dbc5014810 2024-03-23T23:13:04.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.037+0000 7f231b5e0740 10 client.28469 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:13:04.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.037+0000 7f231b5e0740 10 client.28469 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:13:04.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.037+0000 7f231b5e0740 20 client.28469 populate_metadata read hostname 'smithi012' 2024-03-23T23:13:04.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.037+0000 7f231b5e0740 1 --2- 172.21.15.12:0/2081886592 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55dbc5087430 0x55dbc5089850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.037+0000 7f231b5e0740 4 client.28469 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:13:04.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.037+0000 7f231b5e0740 1 -- 172.21.15.12:0/2081886592 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55dbc4f80b50 con 0x55dbc5087430 2024-03-23T23:13:04.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.037+0000 7f2315400640 1 --2- 172.21.15.12:0/2081886592 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55dbc5087430 0x55dbc5089850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.038+0000 7f2315400640 1 --2- 172.21.15.12:0/2081886592 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55dbc5087430 0x55dbc5089850 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8c5edaed1d74eb09 server_cookie=f77df1fefd4140b6 in_seq=0 out_seq=0 2024-03-23T23:13:04.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.038+0000 7f22ef400640 10 client.28469 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:13:04.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.038+0000 7f22ef400640 1 -- 172.21.15.12:0/2081886592 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55dbc4f80b50 con 0x55dbc5087430 2024-03-23T23:13:04.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.038+0000 7f22ef400640 10 client.28469 handle_command_reply: tid=0 2024-03-23T23:13:04.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.038+0000 7f231b5e0740 1 client.28469 shutdown 2024-03-23T23:13:04.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.038+0000 7f231b5e0740 1 -- 172.21.15.12:0/2081886592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc5014810 msgr2=0x55dbc5025db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.038+0000 7f231b5e0740 1 --2- 172.21.15.12:0/2081886592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc5014810 0x55dbc5025db0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f230c09e040 tx=0x7f230c0861a0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.038+0000 7f231b5e0740 1 -- 172.21.15.12:0/2081886592 shutdown_connections 2024-03-23T23:13:04.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.038+0000 7f231b5e0740 1 --2- 172.21.15.12:0/2081886592 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55dbc5087430 0x55dbc5089850 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.038+0000 7f231b5e0740 1 --2- 172.21.15.12:0/2081886592 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbc50163a0 0x55dbc5024950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.038+0000 7f231b5e0740 1 --2- 172.21.15.12:0/2081886592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dbc50159e0 0x55dbc5024410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.038+0000 7f231b5e0740 1 --2- 172.21.15.12:0/2081886592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbc5014810 0x55dbc5025db0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.038+0000 7f22ef400640 0 client.28469 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:13:04.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.038+0000 7f231b5e0740 1 -- 172.21.15.12:0/2081886592 >> 172.21.15.12:0/2081886592 conn(0x55dbc500fc00 msgr2=0x55dbc5018c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:04.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.038+0000 7f231b5e0740 1 -- 172.21.15.12:0/2081886592 shutdown_connections 2024-03-23T23:13:04.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.039+0000 7f231b5e0740 1 -- 172.21.15.12:0/2081886592 wait complete. 2024-03-23T23:13:04.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.039+0000 7f231b5e0740 20 client.28469 trim_cache size 0 max 16384 2024-03-23T23:13:04.043 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:13:04.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.039+0000 7f2318800640 1 -- 172.21.15.12:0/3940831101 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f22fc0a2880 msgr2=0x7f22fc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.039+0000 7f2318800640 1 --2- 172.21.15.12:0/3940831101 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f22fc0a2880 0x7f22fc0a4d40 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x55dbc50158e0 tx=0x7f230800b2e0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.039+0000 7f2318800640 1 -- 172.21.15.12:0/3940831101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2310076060 msgr2=0x7f231019dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.039+0000 7f2318800640 1 --2- 172.21.15.12:0/3940831101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2310076060 0x7f231019dff0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f230c00ebc0 tx=0x7f230c00c6a0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.039+0000 7f2318800640 1 -- 172.21.15.12:0/3940831101 shutdown_connections 2024-03-23T23:13:04.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.039+0000 7f2318800640 1 --2- 172.21.15.12:0/3940831101 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f22fc0a2880 0x7f22fc0a4d40 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.039+0000 7f2318800640 1 --2- 172.21.15.12:0/3940831101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2310077c40 0x7f23101a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.039+0000 7f2318800640 1 --2- 172.21.15.12:0/3940831101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23100772a0 0x7f231019e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.039+0000 7f2318800640 1 --2- 172.21.15.12:0/3940831101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2310076060 0x7f231019dff0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.039+0000 7f2318800640 1 -- 172.21.15.12:0/3940831101 >> 172.21.15.12:0/3940831101 conn(0x7f23100fee50 msgr2=0x7f2310100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:04.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.040+0000 7f2318800640 1 -- 172.21.15.12:0/3940831101 shutdown_connections 2024-03-23T23:13:04.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.040+0000 7f2318800640 1 -- 172.21.15.12:0/3940831101 wait complete. 2024-03-23T23:13:04.058 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:13:04.059 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-23T23:13:04.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.288+0000 7effa2200640 1 -- 172.21.15.12:0/1294472253 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff9c107cc0 msgr2=0x7eff9c1080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.288+0000 7effa2200640 1 --2- 172.21.15.12:0/1294472253 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff9c107cc0 0x7eff9c1080e0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7eff8c009900 tx=0x7eff8c02bc50 comp rx=0 tx=0).stop 2024-03-23T23:13:04.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.288+0000 7effa2200640 1 -- 172.21.15.12:0/1294472253 shutdown_connections 2024-03-23T23:13:04.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.288+0000 7effa2200640 1 --2- 172.21.15.12:0/1294472253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff9c109870 0x7eff9c110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.288+0000 7effa2200640 1 --2- 172.21.15.12:0/1294472253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff9c108eb0 0x7eff9c109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.288+0000 7effa2200640 1 --2- 172.21.15.12:0/1294472253 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff9c107cc0 0x7eff9c1080e0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.288+0000 7effa2200640 1 -- 172.21.15.12:0/1294472253 >> 172.21.15.12:0/1294472253 conn(0x7eff9c076c60 msgr2=0x7eff9c0790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:04.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.288+0000 7effa2200640 1 -- 172.21.15.12:0/1294472253 shutdown_connections 2024-03-23T23:13:04.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.288+0000 7effa2200640 1 -- 172.21.15.12:0/1294472253 wait complete. 2024-03-23T23:13:04.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.289+0000 7effa2200640 1 Processor -- start 2024-03-23T23:13:04.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.289+0000 7effa2200640 1 -- start start 2024-03-23T23:13:04.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.290+0000 7effa2200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff9c107cc0 0x7eff9c19e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.290+0000 7effa2200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff9c108eb0 0x7eff9c19e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.290+0000 7effa2200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff9c109870 0x7eff9c1a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.290+0000 7effa2200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9c113980 con 0x7eff9c107cc0 2024-03-23T23:13:04.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.290+0000 7effa2200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9c113800 con 0x7eff9c109870 2024-03-23T23:13:04.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.290+0000 7effa2200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9c113b00 con 0x7eff9c108eb0 2024-03-23T23:13:04.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.290+0000 7eff9b400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff9c107cc0 0x7eff9c19e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.290+0000 7eff9be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff9c109870 0x7eff9c1a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.290+0000 7eff93e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff9c108eb0 0x7eff9c19e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.290+0000 7eff9be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff9c109870 0x7eff9c1a55d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:58068/0 (socket says 172.21.15.12:58068) 2024-03-23T23:13:04.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.290+0000 7eff9be00640 1 -- 172.21.15.12:0/2768933042 learned_addr learned my addr 172.21.15.12:0/2768933042 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:04.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.291+0000 7eff9be00640 1 -- 172.21.15.12:0/2768933042 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff9c108eb0 msgr2=0x7eff9c19e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.291+0000 7eff9be00640 1 --2- 172.21.15.12:0/2768933042 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff9c108eb0 0x7eff9c19e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.291+0000 7eff9be00640 1 -- 172.21.15.12:0/2768933042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff9c107cc0 msgr2=0x7eff9c19e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.291+0000 7eff9be00640 1 --2- 172.21.15.12:0/2768933042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff9c107cc0 0x7eff9c19e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.291+0000 7eff9be00640 1 -- 172.21.15.12:0/2768933042 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff9c1a5c40 con 0x7eff9c109870 2024-03-23T23:13:04.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.291+0000 7eff93e00640 1 --2- 172.21.15.12:0/2768933042 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff9c108eb0 0x7eff9c19e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:04.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.291+0000 7eff9be00640 1 --2- 172.21.15.12:0/2768933042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff9c109870 0x7eff9c1a55d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7eff9400eb10 tx=0x7eff9400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:04.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.291+0000 7eff98c00640 1 -- 172.21.15.12:0/2768933042 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff94004b20 con 0x7eff9c109870 2024-03-23T23:13:04.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.291+0000 7eff98c00640 1 -- 172.21.15.12:0/2768933042 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff94004cc0 con 0x7eff9c109870 2024-03-23T23:13:04.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.291+0000 7effa2200640 1 -- 172.21.15.12:0/2768933042 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff9c1a5f30 con 0x7eff9c109870 2024-03-23T23:13:04.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.291+0000 7eff98c00640 1 -- 172.21.15.12:0/2768933042 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff94010760 con 0x7eff9c109870 2024-03-23T23:13:04.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.291+0000 7effa2200640 1 -- 172.21.15.12:0/2768933042 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff9c1a63f0 con 0x7eff9c109870 2024-03-23T23:13:04.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.294+0000 7effa4dfa740 1 Processor -- start 2024-03-23T23:13:04.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.295+0000 7eff98c00640 1 -- 172.21.15.12:0/2768933042 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7eff94005020 con 0x7eff9c109870 2024-03-23T23:13:04.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.295+0000 7effa4dfa740 1 -- start start 2024-03-23T23:13:04.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.295+0000 7effa4dfa740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556811dd6810 0x556811dd6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.295+0000 7effa4dfa740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556811dd79e0 0x556811dd7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.295+0000 7effa4dfa740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd83a0 0x556811ddec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.295+0000 7effa4dfa740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x556811d3b2c0 con 0x556811dd6810 2024-03-23T23:13:04.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.295+0000 7effa4dfa740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x556811ce3c90 con 0x556811dd79e0 2024-03-23T23:13:04.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.295+0000 7effa4dfa740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x556811d37360 con 0x556811dd83a0 2024-03-23T23:13:04.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.295+0000 7eff9b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556811dd79e0 0x556811dd7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.295+0000 7eff93e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556811dd6810 0x556811dd6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.296+0000 7eff9b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556811dd79e0 0x556811dd7e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:58072/0 (socket says 172.21.15.12:58072) 2024-03-23T23:13:04.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.296+0000 7eff9b400640 1 -- 172.21.15.12:0/3928997558 learned_addr learned my addr 172.21.15.12:0/3928997558 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:04.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.296+0000 7eff93e00640 1 --2- 172.21.15.12:0/3928997558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd83a0 0x556811ddec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.296+0000 7eff9b400640 1 -- 172.21.15.12:0/3928997558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd83a0 msgr2=0x556811ddec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:04.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.296+0000 7eff9b400640 1 --2- 172.21.15.12:0/3928997558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd83a0 0x556811ddec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.296+0000 7eff9b400640 1 -- 172.21.15.12:0/3928997558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556811dd6810 msgr2=0x556811dd6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:04.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.296+0000 7eff9b400640 1 --2- 172.21.15.12:0/3928997558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556811dd6810 0x556811dd6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.296+0000 7eff9b400640 1 -- 172.21.15.12:0/3928997558 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556811d08080 con 0x556811dd79e0 2024-03-23T23:13:04.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.296+0000 7eff98c00640 1 --2- 172.21.15.12:0/2768933042 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7eff7c0a2880 0x7eff7c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.296+0000 7eff9b400640 1 --2- 172.21.15.12:0/3928997558 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556811dd79e0 0x556811dd7e60 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7eff8c02c260 tx=0x7eff8c009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=daa83e5ecc90153d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:04.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.297+0000 7eff98c00640 1 -- 172.21.15.12:0/2768933042 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7eff940cef90 con 0x7eff9c109870 2024-03-23T23:13:04.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.297+0000 7eff93e00640 1 --2- 172.21.15.12:0/2768933042 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7eff7c0a2880 0x7eff7c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.297+0000 7eff8be00640 1 -- 172.21.15.12:0/3928997558 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8c042070 con 0x556811dd79e0 2024-03-23T23:13:04.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.297+0000 7eff8be00640 1 -- 172.21.15.12:0/3928997558 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff8c02cba0 con 0x556811dd79e0 2024-03-23T23:13:04.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.297+0000 7eff8be00640 1 -- 172.21.15.12:0/3928997558 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8c004eb0 con 0x556811dd79e0 2024-03-23T23:13:04.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.297+0000 7effa4dfa740 1 -- 172.21.15.12:0/3928997558 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556811dd79e0 msgr2=0x556811dd7e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.297+0000 7effa4dfa740 1 --2- 172.21.15.12:0/3928997558 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556811dd79e0 0x556811dd7e60 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7eff8c02c260 tx=0x7eff8c009560 comp rx=0 tx=0).stop 2024-03-23T23:13:04.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.297+0000 7eff93e00640 1 --2- 172.21.15.12:0/2768933042 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7eff7c0a2880 0x7eff7c0a4d40 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x556811dd78e0 tx=0x7eff84005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:04.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.298+0000 7effa4dfa740 1 -- 172.21.15.12:0/3928997558 shutdown_connections 2024-03-23T23:13:04.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.298+0000 7effa4dfa740 1 --2- 172.21.15.12:0/3928997558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd83a0 0x556811ddec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.298+0000 7effa4dfa740 1 --2- 172.21.15.12:0/3928997558 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556811dd79e0 0x556811dd7e60 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.298+0000 7effa4dfa740 1 --2- 172.21.15.12:0/3928997558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556811dd6810 0x556811dd6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.298+0000 7effa4dfa740 1 -- 172.21.15.12:0/3928997558 >> 172.21.15.12:0/3928997558 conn(0x556811dd1c00 msgr2=0x556811dd2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:04.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.298+0000 7effa4dfa740 1 -- 172.21.15.12:0/3928997558 shutdown_connections 2024-03-23T23:13:04.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.298+0000 7effa4dfa740 1 -- 172.21.15.12:0/3928997558 wait complete. 2024-03-23T23:13:04.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.298+0000 7effa4dfa740 1 -- start start 2024-03-23T23:13:04.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.298+0000 7effa4dfa740 1 Processor -- start 2024-03-23T23:13:04.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.299+0000 7effa4dfa740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd6810 0x556811de0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.299+0000 7effa4dfa740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556811dd79e0 0x556811de0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.299+0000 7effa4dfa740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556811dd83a0 0x556811de7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.299+0000 7effa4dfa740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x556811ddf350 con 0x556811dd79e0 2024-03-23T23:13:04.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.299+0000 7effa4dfa740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x556811cf70d0 con 0x556811dd83a0 2024-03-23T23:13:04.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.299+0000 7effa4dfa740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x556811ddf4d0 con 0x556811dd6810 2024-03-23T23:13:04.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.299+0000 7effa4dfa740 10 client.0 _ll_register_callbacks cb 0x556811cfbe80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:13:04.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.299+0000 7eff9b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd6810 0x556811de0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.299+0000 7eff93e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556811dd79e0 0x556811de0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.299+0000 7eff9b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd6810 0x556811de0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49344/0 (socket says 172.21.15.12:49344) 2024-03-23T23:13:04.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.299+0000 7eff9be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556811dd83a0 0x556811de7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.299+0000 7eff9b400640 1 -- 172.21.15.12:0/868507261 learned_addr learned my addr 172.21.15.12:0/868507261 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:04.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.299+0000 7eff9b400640 1 -- 172.21.15.12:0/868507261 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556811dd83a0 msgr2=0x556811de7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:04.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.299+0000 7eff9b400640 1 --2- 172.21.15.12:0/868507261 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556811dd83a0 0x556811de7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.299+0000 7eff9b400640 1 -- 172.21.15.12:0/868507261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556811dd79e0 msgr2=0x556811de0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:04.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.299+0000 7eff9b400640 1 --2- 172.21.15.12:0/868507261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556811dd79e0 0x556811de0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.299+0000 7eff9b400640 1 -- 172.21.15.12:0/868507261 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556811a4f3f0 con 0x556811dd6810 2024-03-23T23:13:04.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.299+0000 7eff9be00640 1 --2- 172.21.15.12:0/868507261 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556811dd83a0 0x556811de7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:04.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.300+0000 7eff9b400640 1 --2- 172.21.15.12:0/868507261 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd6810 0x556811de0a00 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7eff8c006010 tx=0x7eff8c0058e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d903b72ea1d36fae server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:04.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.300+0000 7eff88c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:13:04.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.300+0000 7eff88c00640 1 -- 172.21.15.12:0/868507261 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8c042070 con 0x556811dd6810 2024-03-23T23:13:04.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.300+0000 7eff88c00640 1 -- 172.21.15.12:0/868507261 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7eff8c0050c0 con 0x556811dd6810 2024-03-23T23:13:04.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.300+0000 7effa4dfa740 1 -- 172.21.15.12:0/868507261 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7eff8c02cba0 con 0x556811dd6810 2024-03-23T23:13:04.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.300+0000 7eff88c00640 1 -- 172.21.15.12:0/868507261 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff8c0052c0 con 0x556811dd6810 2024-03-23T23:13:04.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.300+0000 7effa4dfa740 10 client.28450 fetch_fsmap learned FSMap version 23 2024-03-23T23:13:04.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.300+0000 7effa4dfa740 1 -- 172.21.15.12:0/868507261 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556811b197d0 con 0x556811dd6810 2024-03-23T23:13:04.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.301+0000 7eff88c00640 1 -- 172.21.15.12:0/868507261 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff8c02cba0 con 0x556811dd6810 2024-03-23T23:13:04.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.301+0000 7eff88c00640 1 -- 172.21.15.12:0/868507261 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7eff8c034440 con 0x556811dd6810 2024-03-23T23:13:04.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.301+0000 7effa4dfa740 10 client.28450 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:13:04.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.301+0000 7effa4dfa740 10 client.28450 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:13:04.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.301+0000 7effa4dfa740 20 client.28450 populate_metadata read hostname 'smithi012' 2024-03-23T23:13:04.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.301+0000 7effa4dfa740 1 --2- 172.21.15.12:0/868507261 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x556811dec0e0 0x556811dee4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.301+0000 7effa4dfa740 4 client.28450 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:13:04.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.301+0000 7effa4dfa740 1 -- 172.21.15.12:0/868507261 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556811d48800 con 0x556811dec0e0 2024-03-23T23:13:04.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.301+0000 7eff93e00640 1 --2- 172.21.15.12:0/868507261 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x556811dec0e0 0x556811dee4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.302+0000 7eff93e00640 1 --2- 172.21.15.12:0/868507261 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x556811dec0e0 0x556811dee4d0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5e9df5232e1b6e6a server_cookie=3fe74b6a54741122 in_seq=0 out_seq=0 2024-03-23T23:13:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.302+0000 7eff88c00640 10 client.28450 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:13:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.303+0000 7eff88c00640 1 -- 172.21.15.12:0/868507261 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556811d48800 con 0x556811dec0e0 2024-03-23T23:13:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.303+0000 7eff88c00640 10 client.28450 handle_command_reply: tid=0 2024-03-23T23:13:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.303+0000 7effa4dfa740 1 client.28450 shutdown 2024-03-23T23:13:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.303+0000 7effa4dfa740 1 -- 172.21.15.12:0/868507261 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd6810 msgr2=0x556811de0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.303+0000 7effa4dfa740 1 --2- 172.21.15.12:0/868507261 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd6810 0x556811de0a00 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7eff8c006010 tx=0x7eff8c0058e0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.303+0000 7effa4dfa740 1 -- 172.21.15.12:0/868507261 shutdown_connections 2024-03-23T23:13:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.303+0000 7effa4dfa740 1 --2- 172.21.15.12:0/868507261 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556811dd83a0 0x556811de7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.303+0000 7effa4dfa740 1 --2- 172.21.15.12:0/868507261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556811dd79e0 0x556811de0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.303+0000 7effa4dfa740 1 --2- 172.21.15.12:0/868507261 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x556811dec0e0 0x556811dee4d0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.303+0000 7effa4dfa740 1 --2- 172.21.15.12:0/868507261 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd6810 0x556811de0a00 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.303+0000 7effa4dfa740 1 -- 172.21.15.12:0/868507261 >> 172.21.15.12:0/868507261 conn(0x556811dd1c00 msgr2=0x556811dd0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.303+0000 7eff88c00640 0 client.28450 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:13:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.303+0000 7effa4dfa740 1 -- 172.21.15.12:0/868507261 shutdown_connections 2024-03-23T23:13:04.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.303+0000 7effa4dfa740 1 -- 172.21.15.12:0/868507261 wait complete. 2024-03-23T23:13:04.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.303+0000 7effa4dfa740 20 client.28450 trim_cache size 0 max 16384 2024-03-23T23:13:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.317+0000 7effa4dfa740 1 Processor -- start 2024-03-23T23:13:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.317+0000 7effa4dfa740 1 -- start start 2024-03-23T23:13:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.317+0000 7effa4dfa740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556811dd6810 0x556811dd0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.317+0000 7effa4dfa740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd79e0 0x556811df2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.317+0000 7effa4dfa740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556811dd83a0 0x556811df3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.317+0000 7effa4dfa740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9c1b1510 con 0x556811dd6810 2024-03-23T23:13:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.317+0000 7effa4dfa740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9c1af010 con 0x556811dd83a0 2024-03-23T23:13:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.317+0000 7effa4dfa740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9c1ad0b0 con 0x556811dd79e0 2024-03-23T23:13:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.317+0000 7eff93e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd79e0 0x556811df2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.317+0000 7eff9b400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556811dd6810 0x556811dd0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.318+0000 7eff9be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556811dd83a0 0x556811df3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.318+0000 7eff93e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd79e0 0x556811df2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49360/0 (socket says 172.21.15.12:49360) 2024-03-23T23:13:04.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.318+0000 7eff93e00640 1 -- 172.21.15.12:0/2096336856 learned_addr learned my addr 172.21.15.12:0/2096336856 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:04.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.318+0000 7eff93e00640 1 -- 172.21.15.12:0/2096336856 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556811dd83a0 msgr2=0x556811df3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:04.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.318+0000 7eff93e00640 1 --2- 172.21.15.12:0/2096336856 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556811dd83a0 0x556811df3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.318+0000 7eff93e00640 1 -- 172.21.15.12:0/2096336856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556811dd6810 msgr2=0x556811dd0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:04.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.318+0000 7eff93e00640 1 --2- 172.21.15.12:0/2096336856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556811dd6810 0x556811dd0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.318+0000 7eff93e00640 1 -- 172.21.15.12:0/2096336856 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556811dde570 con 0x556811dd79e0 2024-03-23T23:13:04.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.318+0000 7eff93e00640 1 --2- 172.21.15.12:0/2096336856 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd79e0 0x556811df2c30 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7eff84009fd0 tx=0x7eff84015070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bc2130bd5d90398 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:04.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.318+0000 7eff8b400640 1 -- 172.21.15.12:0/2096336856 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff84005d10 con 0x556811dd79e0 2024-03-23T23:13:04.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.318+0000 7eff8b400640 1 -- 172.21.15.12:0/2096336856 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff84003da0 con 0x556811dd79e0 2024-03-23T23:13:04.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.318+0000 7eff8b400640 1 -- 172.21.15.12:0/2096336856 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff84004b20 con 0x556811dd79e0 2024-03-23T23:13:04.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.319+0000 7effa4dfa740 1 -- 172.21.15.12:0/2096336856 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd79e0 msgr2=0x556811df2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.319+0000 7effa4dfa740 1 --2- 172.21.15.12:0/2096336856 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd79e0 0x556811df2c30 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7eff84009fd0 tx=0x7eff84015070 comp rx=0 tx=0).stop 2024-03-23T23:13:04.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.319+0000 7effa4dfa740 1 -- 172.21.15.12:0/2096336856 shutdown_connections 2024-03-23T23:13:04.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.319+0000 7effa4dfa740 1 --2- 172.21.15.12:0/2096336856 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556811dd83a0 0x556811df3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.319+0000 7effa4dfa740 1 --2- 172.21.15.12:0/2096336856 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd79e0 0x556811df2c30 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.319+0000 7effa4dfa740 1 --2- 172.21.15.12:0/2096336856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556811dd6810 0x556811dd0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.319+0000 7effa4dfa740 1 -- 172.21.15.12:0/2096336856 >> 172.21.15.12:0/2096336856 conn(0x556811dd1c00 msgr2=0x556811dd0620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:04.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.319+0000 7effa4dfa740 1 -- 172.21.15.12:0/2096336856 shutdown_connections 2024-03-23T23:13:04.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.319+0000 7effa4dfa740 1 -- 172.21.15.12:0/2096336856 wait complete. 2024-03-23T23:13:04.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.320+0000 7effa4dfa740 1 -- start start 2024-03-23T23:13:04.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.320+0000 7effa4dfa740 1 Processor -- start 2024-03-23T23:13:04.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.320+0000 7effa4dfa740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd6810 0x556811de32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.320+0000 7effa4dfa740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556811dd79e0 0x556811de3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.320+0000 7effa4dfa740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556811dd83a0 0x556811de6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.320+0000 7effa4dfa740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9c1ad6b0 con 0x556811dd79e0 2024-03-23T23:13:04.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.320+0000 7effa4dfa740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9c1ae9b0 con 0x556811dd83a0 2024-03-23T23:13:04.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.320+0000 7effa4dfa740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9c1adcf0 con 0x556811dd6810 2024-03-23T23:13:04.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.320+0000 7effa4dfa740 10 client.0 _ll_register_callbacks cb 0x7eff84009e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:13:04.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.320+0000 7eff9b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd6810 0x556811de32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.321+0000 7eff9b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd6810 0x556811de32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49362/0 (socket says 172.21.15.12:49362) 2024-03-23T23:13:04.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.321+0000 7eff9b400640 1 -- 172.21.15.12:0/468077824 learned_addr learned my addr 172.21.15.12:0/468077824 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:04.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.321+0000 7eff9be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556811dd83a0 0x556811de6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.321+0000 7eff93e00640 1 --2- 172.21.15.12:0/468077824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556811dd79e0 0x556811de3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.321+0000 7eff9b400640 1 -- 172.21.15.12:0/468077824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556811dd83a0 msgr2=0x556811de6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:04.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.321+0000 7eff9b400640 1 --2- 172.21.15.12:0/468077824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556811dd83a0 0x556811de6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.321+0000 7eff9b400640 1 -- 172.21.15.12:0/468077824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556811dd79e0 msgr2=0x556811de3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:04.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.321+0000 7eff9b400640 1 --2- 172.21.15.12:0/468077824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556811dd79e0 0x556811de3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.321+0000 7eff9b400640 1 -- 172.21.15.12:0/468077824 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556811de6e40 con 0x556811dd6810 2024-03-23T23:13:04.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.321+0000 7eff9be00640 1 --2- 172.21.15.12:0/468077824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556811dd83a0 0x556811de6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:04.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.321+0000 7eff9b400640 1 --2- 172.21.15.12:0/468077824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd6810 0x556811de32f0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7eff8c0347e0 tx=0x7eff8c0051b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac624adda0bbba8a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:04.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.321+0000 7eff88c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:13:04.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.321+0000 7eff88c00640 1 -- 172.21.15.12:0/468077824 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8c042070 con 0x556811dd6810 2024-03-23T23:13:04.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.321+0000 7effa4dfa740 1 -- 172.21.15.12:0/468077824 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7eff84003da0 con 0x556811dd6810 2024-03-23T23:13:04.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.322+0000 7eff88c00640 1 -- 172.21.15.12:0/468077824 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7eff8c005660 con 0x556811dd6810 2024-03-23T23:13:04.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.322+0000 7eff88c00640 1 -- 172.21.15.12:0/468077824 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff8c02c9a0 con 0x556811dd6810 2024-03-23T23:13:04.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.322+0000 7effa4dfa740 10 client.28456 fetch_fsmap learned FSMap version 23 2024-03-23T23:13:04.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.322+0000 7effa4dfa740 1 -- 172.21.15.12:0/468077824 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556811e47d40 con 0x556811dd6810 2024-03-23T23:13:04.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.322+0000 7eff88c00640 1 -- 172.21.15.12:0/468077824 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff84003da0 con 0x556811dd6810 2024-03-23T23:13:04.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.322+0000 7eff88c00640 1 -- 172.21.15.12:0/468077824 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7eff8c0039f0 con 0x556811dd6810 2024-03-23T23:13:04.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.322+0000 7effa4dfa740 10 client.28456 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:13:04.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.322+0000 7effa4dfa740 10 client.28456 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:13:04.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.322+0000 7effa4dfa740 20 client.28456 populate_metadata read hostname 'smithi012' 2024-03-23T23:13:04.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.322+0000 7effa4dfa740 1 --2- 172.21.15.12:0/468077824 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x556811e484d0 0x556811e4a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.322+0000 7effa4dfa740 4 client.28456 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:13:04.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.322+0000 7effa4dfa740 1 -- 172.21.15.12:0/468077824 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556811d42b50 con 0x556811e484d0 2024-03-23T23:13:04.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.323+0000 7eff93e00640 1 --2- 172.21.15.12:0/468077824 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x556811e484d0 0x556811e4a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.323+0000 7eff93e00640 1 --2- 172.21.15.12:0/468077824 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x556811e484d0 0x556811e4a8c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8a36777552e1c022 server_cookie=bbdcd3a3ce24e74c in_seq=0 out_seq=0 2024-03-23T23:13:04.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.323+0000 7eff88c00640 10 client.28456 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:13:04.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.323+0000 7eff88c00640 1 -- 172.21.15.12:0/468077824 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556811d42b50 con 0x556811e484d0 2024-03-23T23:13:04.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.323+0000 7eff88c00640 10 client.28456 handle_command_reply: tid=0 2024-03-23T23:13:04.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.323+0000 7effa4dfa740 1 client.28456 shutdown 2024-03-23T23:13:04.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.323+0000 7effa4dfa740 1 -- 172.21.15.12:0/468077824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd6810 msgr2=0x556811de32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.323+0000 7effa4dfa740 1 --2- 172.21.15.12:0/468077824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd6810 0x556811de32f0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7eff8c0347e0 tx=0x7eff8c0051b0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.324+0000 7effa4dfa740 1 -- 172.21.15.12:0/468077824 shutdown_connections 2024-03-23T23:13:04.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.324+0000 7effa4dfa740 1 --2- 172.21.15.12:0/468077824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556811dd83a0 0x556811de6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.324+0000 7effa4dfa740 1 --2- 172.21.15.12:0/468077824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556811dd79e0 0x556811de3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.324+0000 7effa4dfa740 1 --2- 172.21.15.12:0/468077824 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x556811e484d0 0x556811e4a8c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.324+0000 7effa4dfa740 1 --2- 172.21.15.12:0/468077824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556811dd6810 0x556811de32f0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.324+0000 7effa4dfa740 1 -- 172.21.15.12:0/468077824 >> 172.21.15.12:0/468077824 conn(0x556811dd1c00 msgr2=0x556811dd0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:04.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.324+0000 7eff88c00640 0 client.28456 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:13:04.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.324+0000 7effa4dfa740 1 -- 172.21.15.12:0/468077824 shutdown_connections 2024-03-23T23:13:04.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.324+0000 7effa4dfa740 1 -- 172.21.15.12:0/468077824 wait complete. 2024-03-23T23:13:04.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.324+0000 7effa4dfa740 20 client.28456 trim_cache size 0 max 16384 2024-03-23T23:13:04.328 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:13:04.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.324+0000 7effa2200640 1 -- 172.21.15.12:0/2768933042 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7eff7c0a2880 msgr2=0x7eff7c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.324+0000 7effa2200640 1 --2- 172.21.15.12:0/2768933042 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7eff7c0a2880 0x7eff7c0a4d40 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x556811dd78e0 tx=0x7eff84005f90 comp rx=0 tx=0).stop 2024-03-23T23:13:04.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.324+0000 7effa2200640 1 -- 172.21.15.12:0/2768933042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff9c109870 msgr2=0x7eff9c1a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.324+0000 7effa2200640 1 --2- 172.21.15.12:0/2768933042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff9c109870 0x7eff9c1a55d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7eff9400eb10 tx=0x7eff9400c2d0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.325+0000 7effa2200640 1 -- 172.21.15.12:0/2768933042 shutdown_connections 2024-03-23T23:13:04.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.325+0000 7effa2200640 1 --2- 172.21.15.12:0/2768933042 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7eff7c0a2880 0x7eff7c0a4d40 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.325+0000 7effa2200640 1 --2- 172.21.15.12:0/2768933042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff9c109870 0x7eff9c1a55d0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.325+0000 7effa2200640 1 --2- 172.21.15.12:0/2768933042 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff9c108eb0 0x7eff9c19e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.325+0000 7effa2200640 1 --2- 172.21.15.12:0/2768933042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff9c107cc0 0x7eff9c19e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.325+0000 7effa2200640 1 -- 172.21.15.12:0/2768933042 >> 172.21.15.12:0/2768933042 conn(0x7eff9c076c60 msgr2=0x7eff9c10c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:04.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.325+0000 7effa2200640 1 -- 172.21.15.12:0/2768933042 shutdown_connections 2024-03-23T23:13:04.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.325+0000 7effa2200640 1 -- 172.21.15.12:0/2768933042 wait complete. 2024-03-23T23:13:04.345 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:13:04.345 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-23T23:13:04.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.578+0000 7f33dda00640 1 -- 172.21.15.12:0/556104540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33d8077c40 msgr2=0x7f33d80784e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.578+0000 7f33dda00640 1 --2- 172.21.15.12:0/556104540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33d8077c40 0x7f33d80784e0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f33d0009be0 tx=0x7f33d002bef0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.579+0000 7f33dda00640 1 -- 172.21.15.12:0/556104540 shutdown_connections 2024-03-23T23:13:04.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.579+0000 7f33dda00640 1 --2- 172.21.15.12:0/556104540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33d8077c40 0x7f33d80784e0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.579+0000 7f33dda00640 1 --2- 172.21.15.12:0/556104540 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33d80772a0 0x7f33d8077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.579+0000 7f33dda00640 1 --2- 172.21.15.12:0/556104540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33d8076060 0x7f33d8076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.579+0000 7f33dda00640 1 -- 172.21.15.12:0/556104540 >> 172.21.15.12:0/556104540 conn(0x7f33d80fee50 msgr2=0x7f33d8101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:04.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.579+0000 7f33dda00640 1 -- 172.21.15.12:0/556104540 shutdown_connections 2024-03-23T23:13:04.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.579+0000 7f33dda00640 1 -- 172.21.15.12:0/556104540 wait complete. 2024-03-23T23:13:04.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.580+0000 7f33dda00640 1 Processor -- start 2024-03-23T23:13:04.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.580+0000 7f33dda00640 1 -- start start 2024-03-23T23:13:04.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.580+0000 7f33dda00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33d8076060 0x7f33d819df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.580+0000 7f33dda00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33d80772a0 0x7f33d819e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.580+0000 7f33dda00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33d8077c40 0x7f33d81a5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.580+0000 7f33dda00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d81139b0 con 0x7f33d8077c40 2024-03-23T23:13:04.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.580+0000 7f33dda00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d8113830 con 0x7f33d8076060 2024-03-23T23:13:04.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.580+0000 7f33dda00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d8113b30 con 0x7f33d80772a0 2024-03-23T23:13:04.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.580+0000 7f33d6a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33d8076060 0x7f33d819df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.580+0000 7f33d6a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33d8076060 0x7f33d819df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:58108/0 (socket says 172.21.15.12:58108) 2024-03-23T23:13:04.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.580+0000 7f33d7400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33d8077c40 0x7f33d81a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.580+0000 7f33d6a00640 1 -- 172.21.15.12:0/3312163322 learned_addr learned my addr 172.21.15.12:0/3312163322 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:04.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.581+0000 7f33cfe00640 1 --2- 172.21.15.12:0/3312163322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33d80772a0 0x7f33d819e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.581+0000 7f33d6a00640 1 -- 172.21.15.12:0/3312163322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33d80772a0 msgr2=0x7f33d819e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.581+0000 7f33d6a00640 1 --2- 172.21.15.12:0/3312163322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33d80772a0 0x7f33d819e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.581+0000 7f33d6a00640 1 -- 172.21.15.12:0/3312163322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33d8077c40 msgr2=0x7f33d81a5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.581+0000 7f33d6a00640 1 --2- 172.21.15.12:0/3312163322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33d8077c40 0x7f33d81a5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.581+0000 7f33d6a00640 1 -- 172.21.15.12:0/3312163322 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d81a5b90 con 0x7f33d8076060 2024-03-23T23:13:04.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.581+0000 7f33d6a00640 1 --2- 172.21.15.12:0/3312163322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33d8076060 0x7f33d819df70 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f33c800e9f0 tx=0x7f33c800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:04.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.581+0000 7f33cf400640 1 -- 172.21.15.12:0/3312163322 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c800ccc0 con 0x7f33d8076060 2024-03-23T23:13:04.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.581+0000 7f33cf400640 1 -- 172.21.15.12:0/3312163322 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33c8005500 con 0x7f33d8076060 2024-03-23T23:13:04.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.581+0000 7f33dda00640 1 -- 172.21.15.12:0/3312163322 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33d81a5e80 con 0x7f33d8076060 2024-03-23T23:13:04.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.581+0000 7f33cf400640 1 -- 172.21.15.12:0/3312163322 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c8010690 con 0x7f33d8076060 2024-03-23T23:13:04.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.582+0000 7f33dda00640 1 -- 172.21.15.12:0/3312163322 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d81a62e0 con 0x7f33d8076060 2024-03-23T23:13:04.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.585+0000 7f33e0640740 1 Processor -- start 2024-03-23T23:13:04.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.585+0000 7f33cf400640 1 -- 172.21.15.12:0/3312163322 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f33c80046c0 con 0x7f33d8076060 2024-03-23T23:13:04.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.585+0000 7f33e0640740 1 -- start start 2024-03-23T23:13:04.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.585+0000 7f33e0640740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a0810 0x55670b4a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.585+0000 7f33e0640740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55670b4a19e0 0x55670b4a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.585+0000 7f33e0640740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55670b4a23a0 0x55670b4a8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.585+0000 7f33e0640740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55670b4052c0 con 0x55670b4a23a0 2024-03-23T23:13:04.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.585+0000 7f33e0640740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55670b3adc90 con 0x55670b4a19e0 2024-03-23T23:13:04.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.585+0000 7f33e0640740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55670b401360 con 0x55670b4a0810 2024-03-23T23:13:04.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.585+0000 7f33d7400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a0810 0x55670b4a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.586+0000 7f33d6a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55670b4a19e0 0x55670b4a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.586+0000 7f33d7400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a0810 0x55670b4a0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49378/0 (socket says 172.21.15.12:49378) 2024-03-23T23:13:04.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.586+0000 7f33d7400640 1 -- 172.21.15.12:0/896181176 learned_addr learned my addr 172.21.15.12:0/896181176 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:04.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.586+0000 7f33cfe00640 1 --2- 172.21.15.12:0/896181176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55670b4a23a0 0x55670b4a8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.586+0000 7f33d7400640 1 -- 172.21.15.12:0/896181176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55670b4a19e0 msgr2=0x55670b4a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:04.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.586+0000 7f33d7400640 1 --2- 172.21.15.12:0/896181176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55670b4a19e0 0x55670b4a1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.586+0000 7f33d7400640 1 -- 172.21.15.12:0/896181176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55670b4a23a0 msgr2=0x55670b4a8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:04.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.586+0000 7f33d7400640 1 --2- 172.21.15.12:0/896181176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55670b4a23a0 0x55670b4a8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.586+0000 7f33d7400640 1 -- 172.21.15.12:0/896181176 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55670b3d2080 con 0x55670b4a0810 2024-03-23T23:13:04.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.586+0000 7f33d7400640 1 --2- 172.21.15.12:0/896181176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a0810 0x55670b4a0c30 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f33d002c500 tx=0x7f33d0009330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92b2f08ef8d5185b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:04.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.586+0000 7f33c7e00640 1 -- 172.21.15.12:0/896181176 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d0043030 con 0x55670b4a0810 2024-03-23T23:13:04.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.587+0000 7f33c7e00640 1 -- 172.21.15.12:0/896181176 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33d0033580 con 0x55670b4a0810 2024-03-23T23:13:04.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.587+0000 7f33c7e00640 1 -- 172.21.15.12:0/896181176 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d002c890 con 0x55670b4a0810 2024-03-23T23:13:04.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.587+0000 7f33cf400640 1 --2- 172.21.15.12:0/3312163322 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f33bc0a2800 0x7f33bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.587+0000 7f33cf400640 1 -- 172.21.15.12:0/3312163322 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f33c80ced30 con 0x7f33d8076060 2024-03-23T23:13:04.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.587+0000 7f33d6a00640 1 --2- 172.21.15.12:0/3312163322 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f33bc0a2800 0x7f33bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.587+0000 7f33e0640740 1 -- 172.21.15.12:0/896181176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a0810 msgr2=0x55670b4a0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.587+0000 7f33e0640740 1 --2- 172.21.15.12:0/896181176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a0810 0x55670b4a0c30 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f33d002c500 tx=0x7f33d0009330 comp rx=0 tx=0).stop 2024-03-23T23:13:04.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.587+0000 7f33e0640740 1 -- 172.21.15.12:0/896181176 shutdown_connections 2024-03-23T23:13:04.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.587+0000 7f33e0640740 1 --2- 172.21.15.12:0/896181176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55670b4a23a0 0x55670b4a8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.587+0000 7f33e0640740 1 --2- 172.21.15.12:0/896181176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55670b4a19e0 0x55670b4a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.587+0000 7f33e0640740 1 --2- 172.21.15.12:0/896181176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a0810 0x55670b4a0c30 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.587+0000 7f33e0640740 1 -- 172.21.15.12:0/896181176 >> 172.21.15.12:0/896181176 conn(0x55670b49bc00 msgr2=0x55670b49c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:04.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.588+0000 7f33e0640740 1 -- 172.21.15.12:0/896181176 shutdown_connections 2024-03-23T23:13:04.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.588+0000 7f33e0640740 1 -- 172.21.15.12:0/896181176 wait complete. 2024-03-23T23:13:04.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.588+0000 7f33d6a00640 1 --2- 172.21.15.12:0/3312163322 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f33bc0a2800 0x7f33bc0a4cc0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f33c80cc7e0 tx=0x7f33c80d2770 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:04.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.588+0000 7f33e0640740 1 -- start start 2024-03-23T23:13:04.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.588+0000 7f33e0640740 1 Processor -- start 2024-03-23T23:13:04.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.589+0000 7f33e0640740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55670b4a0810 0x55670b4aaab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.589+0000 7f33e0640740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55670b4a19e0 0x55670b4aaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.589+0000 7f33e0640740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a23a0 0x55670b4b1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.589+0000 7f33e0640740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55670b4a9350 con 0x55670b4a19e0 2024-03-23T23:13:04.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.589+0000 7f33e0640740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55670b3c10d0 con 0x55670b4a0810 2024-03-23T23:13:04.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.589+0000 7f33e0640740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55670b4a94d0 con 0x55670b4a23a0 2024-03-23T23:13:04.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.589+0000 7f33e0640740 10 client.0 _ll_register_callbacks cb 0x55670b3c5e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:13:04.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.589+0000 7f33d7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55670b4a0810 0x55670b4aaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.589+0000 7f33d7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55670b4a0810 0x55670b4aaab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:58120/0 (socket says 172.21.15.12:58120) 2024-03-23T23:13:04.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.589+0000 7f33d7400640 1 -- 172.21.15.12:0/1986304776 learned_addr learned my addr 172.21.15.12:0/1986304776 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:04.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.589+0000 7f33d7400640 1 --2- 172.21.15.12:0/1986304776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a23a0 0x55670b4b1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.589+0000 7f33cfe00640 1 --2- 172.21.15.12:0/1986304776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55670b4a19e0 0x55670b4aaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.589+0000 7f33d7400640 1 -- 172.21.15.12:0/1986304776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a23a0 msgr2=0x55670b4b1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:04.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.589+0000 7f33d7400640 1 --2- 172.21.15.12:0/1986304776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a23a0 0x55670b4b1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.590+0000 7f33d7400640 1 -- 172.21.15.12:0/1986304776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55670b4a19e0 msgr2=0x55670b4aaff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:04.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.590+0000 7f33d7400640 1 --2- 172.21.15.12:0/1986304776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55670b4a19e0 0x55670b4aaff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.590+0000 7f33d7400640 1 -- 172.21.15.12:0/1986304776 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55670b1193f0 con 0x55670b4a0810 2024-03-23T23:13:04.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.590+0000 7f33d7400640 1 --2- 172.21.15.12:0/1986304776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55670b4a0810 0x55670b4aaab0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f33d00099a0 tx=0x7f33d0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c86f38b25a2cd789 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:04.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.590+0000 7f33c4c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:13:04.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.590+0000 7f33c4c00640 1 -- 172.21.15.12:0/1986304776 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d0043040 con 0x55670b4a0810 2024-03-23T23:13:04.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.590+0000 7f33c4c00640 1 -- 172.21.15.12:0/1986304776 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f33d002c890 con 0x55670b4a0810 2024-03-23T23:13:04.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.590+0000 7f33e0640740 1 -- 172.21.15.12:0/1986304776 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f33d0033580 con 0x55670b4a0810 2024-03-23T23:13:04.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.590+0000 7f33c4c00640 1 -- 172.21.15.12:0/1986304776 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33d0004b40 con 0x55670b4a0810 2024-03-23T23:13:04.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.590+0000 7f33e0640740 10 client.28499 fetch_fsmap learned FSMap version 23 2024-03-23T23:13:04.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.590+0000 7f33e0640740 1 -- 172.21.15.12:0/1986304776 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55670b1e37d0 con 0x55670b4a0810 2024-03-23T23:13:04.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.591+0000 7f33c4c00640 1 -- 172.21.15.12:0/1986304776 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33d0033580 con 0x55670b4a0810 2024-03-23T23:13:04.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.591+0000 7f33c4c00640 1 -- 172.21.15.12:0/1986304776 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f33d0004680 con 0x55670b4a0810 2024-03-23T23:13:04.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.591+0000 7f33e0640740 10 client.28499 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:13:04.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.591+0000 7f33e0640740 10 client.28499 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:13:04.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.591+0000 7f33e0640740 20 client.28499 populate_metadata read hostname 'smithi012' 2024-03-23T23:13:04.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.591+0000 7f33e0640740 1 --2- 172.21.15.12:0/1986304776 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55670b4b6120 0x55670b4b8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.591+0000 7f33e0640740 4 client.28499 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:13:04.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.591+0000 7f33e0640740 1 -- 172.21.15.12:0/1986304776 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55670b412800 con 0x55670b4b6120 2024-03-23T23:13:04.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.591+0000 7f33cfe00640 1 --2- 172.21.15.12:0/1986304776 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55670b4b6120 0x55670b4b8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.592+0000 7f33cfe00640 1 --2- 172.21.15.12:0/1986304776 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55670b4b6120 0x55670b4b8510 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7cc2206766de2b0 server_cookie=d54666f8ad0e9a3c in_seq=0 out_seq=0 2024-03-23T23:13:04.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.592+0000 7f33c4c00640 10 client.28499 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:04.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.592+0000 7f33c4c00640 1 -- 172.21.15.12:0/1986304776 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55670b412800 con 0x55670b4b6120 2024-03-23T23:13:04.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.592+0000 7f33c4c00640 10 client.28499 handle_command_reply: tid=0 2024-03-23T23:13:04.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.592+0000 7f33e0640740 1 client.28499 shutdown 2024-03-23T23:13:04.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.593+0000 7f33e0640740 1 -- 172.21.15.12:0/1986304776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55670b4a0810 msgr2=0x55670b4aaab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.593+0000 7f33e0640740 1 --2- 172.21.15.12:0/1986304776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55670b4a0810 0x55670b4aaab0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f33d00099a0 tx=0x7f33d0004a40 comp rx=0 tx=0).stop 2024-03-23T23:13:04.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.593+0000 7f33e0640740 1 -- 172.21.15.12:0/1986304776 shutdown_connections 2024-03-23T23:13:04.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.593+0000 7f33e0640740 1 --2- 172.21.15.12:0/1986304776 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55670b4b6120 0x55670b4b8510 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.593+0000 7f33e0640740 1 --2- 172.21.15.12:0/1986304776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a23a0 0x55670b4b1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.593+0000 7f33e0640740 1 --2- 172.21.15.12:0/1986304776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55670b4a19e0 0x55670b4aaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.593+0000 7f33c4c00640 0 client.28499 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:04.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.593+0000 7f33e0640740 1 --2- 172.21.15.12:0/1986304776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55670b4a0810 0x55670b4aaab0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.593+0000 7f33e0640740 1 -- 172.21.15.12:0/1986304776 >> 172.21.15.12:0/1986304776 conn(0x55670b49bc00 msgr2=0x55670b49aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:04.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.593+0000 7f33e0640740 1 -- 172.21.15.12:0/1986304776 shutdown_connections 2024-03-23T23:13:04.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.593+0000 7f33e0640740 1 -- 172.21.15.12:0/1986304776 wait complete. 2024-03-23T23:13:04.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.593+0000 7f33e0640740 20 client.28499 trim_cache size 0 max 16384 2024-03-23T23:13:04.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.607+0000 7f33e0640740 1 Processor -- start 2024-03-23T23:13:04.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.607+0000 7f33e0640740 1 -- start start 2024-03-23T23:13:04.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.607+0000 7f33e0640740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55670b4a0810 0x55670b4bccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.607+0000 7f33e0640740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a19e0 0x55670b4b1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.607+0000 7f33e0640740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55670b4a23a0 0x55670b4bd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.607+0000 7f33e0640740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d81b14a0 con 0x55670b4a23a0 2024-03-23T23:13:04.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.607+0000 7f33e0640740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d81aefa0 con 0x55670b4a0810 2024-03-23T23:13:04.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.607+0000 7f33e0640740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d81ad040 con 0x55670b4a19e0 2024-03-23T23:13:04.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.607+0000 7f33cfe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a19e0 0x55670b4b1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.607+0000 7f33d7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55670b4a0810 0x55670b4bccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.607+0000 7f33cfe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a19e0 0x55670b4b1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49404/0 (socket says 172.21.15.12:49404) 2024-03-23T23:13:04.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.607+0000 7f33cfe00640 1 -- 172.21.15.12:0/3952326043 learned_addr learned my addr 172.21.15.12:0/3952326043 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:04.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.608+0000 7f33d7400640 1 --2- 172.21.15.12:0/3952326043 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55670b4a23a0 0x55670b4bd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.608+0000 7f33cfe00640 1 -- 172.21.15.12:0/3952326043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55670b4a0810 msgr2=0x55670b4bccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:04.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.608+0000 7f33cfe00640 1 --2- 172.21.15.12:0/3952326043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55670b4a0810 0x55670b4bccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.608+0000 7f33cfe00640 1 -- 172.21.15.12:0/3952326043 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55670b4a23a0 msgr2=0x55670b4bd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:04.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.608+0000 7f33cfe00640 1 --2- 172.21.15.12:0/3952326043 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55670b4a23a0 0x55670b4bd200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.608+0000 7f33cfe00640 1 -- 172.21.15.12:0/3952326043 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55670b4b8880 con 0x55670b4a19e0 2024-03-23T23:13:04.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.608+0000 7f33cfe00640 1 --2- 172.21.15.12:0/3952326043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a19e0 0x55670b4b1f50 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x55670b4b8c70 tx=0x7f33c0005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3742d05729f1949 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:04.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.609+0000 7f33c7400640 1 -- 172.21.15.12:0/3952326043 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c0003ac0 con 0x55670b4a19e0 2024-03-23T23:13:04.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.609+0000 7f33c7400640 1 -- 172.21.15.12:0/3952326043 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33c0003c60 con 0x55670b4a19e0 2024-03-23T23:13:04.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.609+0000 7f33c7400640 1 -- 172.21.15.12:0/3952326043 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c0010a10 con 0x55670b4a19e0 2024-03-23T23:13:04.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.609+0000 7f33e0640740 1 -- 172.21.15.12:0/3952326043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a19e0 msgr2=0x55670b4b1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.609+0000 7f33e0640740 1 --2- 172.21.15.12:0/3952326043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a19e0 0x55670b4b1f50 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x55670b4b8c70 tx=0x7f33c0005f90 comp rx=0 tx=0).stop 2024-03-23T23:13:04.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.609+0000 7f33e0640740 1 -- 172.21.15.12:0/3952326043 shutdown_connections 2024-03-23T23:13:04.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.609+0000 7f33e0640740 1 --2- 172.21.15.12:0/3952326043 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55670b4a23a0 0x55670b4bd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.609+0000 7f33e0640740 1 --2- 172.21.15.12:0/3952326043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a19e0 0x55670b4b1f50 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.609+0000 7f33e0640740 1 --2- 172.21.15.12:0/3952326043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55670b4a0810 0x55670b4bccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.609+0000 7f33e0640740 1 -- 172.21.15.12:0/3952326043 >> 172.21.15.12:0/3952326043 conn(0x55670b49bc00 msgr2=0x55670b49f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:04.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.609+0000 7f33e0640740 1 -- 172.21.15.12:0/3952326043 shutdown_connections 2024-03-23T23:13:04.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.609+0000 7f33e0640740 1 -- 172.21.15.12:0/3952326043 wait complete. 2024-03-23T23:13:04.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.610+0000 7f33e0640740 1 -- start start 2024-03-23T23:13:04.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.610+0000 7f33e0640740 1 Processor -- start 2024-03-23T23:13:04.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.610+0000 7f33e0640740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55670b4a0810 0x55670b4adcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.610+0000 7f33e0640740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a19e0 0x55670b4ae1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.610+0000 7f33e0640740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55670b4a23a0 0x55670b4b11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.610+0000 7f33e0640740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d81ad640 con 0x55670b4a0810 2024-03-23T23:13:04.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.610+0000 7f33e0640740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d81ae940 con 0x55670b4a23a0 2024-03-23T23:13:04.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.610+0000 7f33e0640740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d81adc80 con 0x55670b4a19e0 2024-03-23T23:13:04.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.610+0000 7f33e0640740 10 client.0 _ll_register_callbacks cb 0x7f33c80d2640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:13:04.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.611+0000 7f33cfe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a19e0 0x55670b4ae1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.611+0000 7f33cfe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a19e0 0x55670b4ae1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49406/0 (socket says 172.21.15.12:49406) 2024-03-23T23:13:04.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.611+0000 7f33cfe00640 1 -- 172.21.15.12:0/3403294567 learned_addr learned my addr 172.21.15.12:0/3403294567 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:04.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.611+0000 7f33d7400640 1 --2- 172.21.15.12:0/3403294567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55670b4a0810 0x55670b4adcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.611+0000 7f33d7400640 1 --2- 172.21.15.12:0/3403294567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55670b4a23a0 0x55670b4b11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.611+0000 7f33cfe00640 1 -- 172.21.15.12:0/3403294567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55670b4a23a0 msgr2=0x55670b4b11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:04.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.611+0000 7f33cfe00640 1 --2- 172.21.15.12:0/3403294567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55670b4a23a0 0x55670b4b11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.611+0000 7f33cfe00640 1 -- 172.21.15.12:0/3403294567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55670b4a0810 msgr2=0x55670b4adcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:04.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.611+0000 7f33cfe00640 1 --2- 172.21.15.12:0/3403294567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55670b4a0810 0x55670b4adcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.611+0000 7f33cfe00640 1 -- 172.21.15.12:0/3403294567 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55670b4b1790 con 0x55670b4a19e0 2024-03-23T23:13:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.611+0000 7f33d7400640 1 --2- 172.21.15.12:0/3403294567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55670b4a23a0 0x55670b4b11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:13:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.611+0000 7f33cfe00640 1 --2- 172.21.15.12:0/3403294567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a19e0 0x55670b4ae1f0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f33c0009c30 tx=0x7f33c0010500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17ffe1e1e08e4b99 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.611+0000 7f33c4c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:13:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.612+0000 7f33c4c00640 1 -- 172.21.15.12:0/3403294567 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c002e070 con 0x55670b4a19e0 2024-03-23T23:13:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.612+0000 7f33c4c00640 1 -- 172.21.15.12:0/3403294567 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f33c0020770 con 0x55670b4a19e0 2024-03-23T23:13:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.612+0000 7f33e0640740 1 -- 172.21.15.12:0/3403294567 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f33c0003c60 con 0x55670b4a19e0 2024-03-23T23:13:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.612+0000 7f33c4c00640 1 -- 172.21.15.12:0/3403294567 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33c0020970 con 0x55670b4a19e0 2024-03-23T23:13:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.612+0000 7f33e0640740 10 client.28474 fetch_fsmap learned FSMap version 23 2024-03-23T23:13:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.612+0000 7f33e0640740 1 -- 172.21.15.12:0/3403294567 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55670b511df0 con 0x55670b4a19e0 2024-03-23T23:13:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.612+0000 7f33c4c00640 1 -- 172.21.15.12:0/3403294567 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33c0003c60 con 0x55670b4a19e0 2024-03-23T23:13:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.612+0000 7f33c4c00640 1 -- 172.21.15.12:0/3403294567 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f33c00334f0 con 0x55670b4a19e0 2024-03-23T23:13:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.612+0000 7f33e0640740 10 client.28474 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:13:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.612+0000 7f33e0640740 10 client.28474 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:13:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.612+0000 7f33e0640740 20 client.28474 populate_metadata read hostname 'smithi012' 2024-03-23T23:13:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.612+0000 7f33e0640740 1 --2- 172.21.15.12:0/3403294567 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55670b512570 0x55670b5149d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.612+0000 7f33e0640740 4 client.28474 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:13:04.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.612+0000 7f33e0640740 1 -- 172.21.15.12:0/3403294567 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55670b40cb50 con 0x55670b512570 2024-03-23T23:13:04.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.613+0000 7f33d7400640 1 --2- 172.21.15.12:0/3403294567 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55670b512570 0x55670b5149d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.613+0000 7f33d7400640 1 --2- 172.21.15.12:0/3403294567 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55670b512570 0x55670b5149d0 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3861f826ee4e6d68 server_cookie=5698cd32392d8786 in_seq=0 out_seq=0 2024-03-23T23:13:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.613+0000 7f33c4c00640 10 client.28474 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.613+0000 7f33c4c00640 1 -- 172.21.15.12:0/3403294567 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55670b40cb50 con 0x55670b512570 2024-03-23T23:13:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.613+0000 7f33c4c00640 10 client.28474 handle_command_reply: tid=0 2024-03-23T23:13:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.613+0000 7f33e0640740 1 client.28474 shutdown 2024-03-23T23:13:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.614+0000 7f33e0640740 1 -- 172.21.15.12:0/3403294567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a19e0 msgr2=0x55670b4ae1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.614+0000 7f33e0640740 1 --2- 172.21.15.12:0/3403294567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a19e0 0x55670b4ae1f0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f33c0009c30 tx=0x7f33c0010500 comp rx=0 tx=0).stop 2024-03-23T23:13:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.614+0000 7f33e0640740 1 -- 172.21.15.12:0/3403294567 shutdown_connections 2024-03-23T23:13:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.614+0000 7f33e0640740 1 --2- 172.21.15.12:0/3403294567 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55670b512570 0x55670b5149d0 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.614+0000 7f33e0640740 1 --2- 172.21.15.12:0/3403294567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55670b4a23a0 0x55670b4b11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.614+0000 7f33e0640740 1 --2- 172.21.15.12:0/3403294567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55670b4a19e0 0x55670b4ae1f0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.614+0000 7f33e0640740 1 --2- 172.21.15.12:0/3403294567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55670b4a0810 0x55670b4adcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.614+0000 7f33c4c00640 0 client.28474 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:04.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.614+0000 7f33e0640740 1 -- 172.21.15.12:0/3403294567 >> 172.21.15.12:0/3403294567 conn(0x55670b49bc00 msgr2=0x55670b4a4c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:04.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.614+0000 7f33e0640740 1 -- 172.21.15.12:0/3403294567 shutdown_connections 2024-03-23T23:13:04.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.614+0000 7f33e0640740 1 -- 172.21.15.12:0/3403294567 wait complete. 2024-03-23T23:13:04.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.614+0000 7f33e0640740 20 client.28474 trim_cache size 0 max 16384 2024-03-23T23:13:04.619 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:13:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.615+0000 7f33dda00640 1 -- 172.21.15.12:0/3312163322 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f33bc0a2800 msgr2=0x7f33bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.615+0000 7f33dda00640 1 --2- 172.21.15.12:0/3312163322 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f33bc0a2800 0x7f33bc0a4cc0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f33c80cc7e0 tx=0x7f33c80d2770 comp rx=0 tx=0).stop 2024-03-23T23:13:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.615+0000 7f33dda00640 1 -- 172.21.15.12:0/3312163322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33d8076060 msgr2=0x7f33d819df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.615+0000 7f33dda00640 1 --2- 172.21.15.12:0/3312163322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33d8076060 0x7f33d819df70 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f33c800e9f0 tx=0x7f33c800eeb0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.615+0000 7f33dda00640 1 -- 172.21.15.12:0/3312163322 shutdown_connections 2024-03-23T23:13:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.615+0000 7f33dda00640 1 --2- 172.21.15.12:0/3312163322 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f33bc0a2800 0x7f33bc0a4cc0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.615+0000 7f33dda00640 1 --2- 172.21.15.12:0/3312163322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33d8077c40 0x7f33d81a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.615+0000 7f33dda00640 1 --2- 172.21.15.12:0/3312163322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33d80772a0 0x7f33d819e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.615+0000 7f33dda00640 1 --2- 172.21.15.12:0/3312163322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33d8076060 0x7f33d819df70 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.615+0000 7f33dda00640 1 -- 172.21.15.12:0/3312163322 >> 172.21.15.12:0/3312163322 conn(0x7f33d80fee50 msgr2=0x7f33d81004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:04.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.615+0000 7f33dda00640 1 -- 172.21.15.12:0/3312163322 shutdown_connections 2024-03-23T23:13:04.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:04.615+0000 7f33dda00640 1 -- 172.21.15.12:0/3312163322 wait complete. 2024-03-23T23:13:05.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:04 smithi012 ceph-mon[31695]: pgmap v728: 97 pgs: 97 active+clean; 3.6 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:13:05.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:04 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2052993134' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:13:05.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:04 smithi173 ceph-mon[36116]: pgmap v728: 97 pgs: 97 active+clean; 3.6 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:13:05.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:04 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2052993134' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:13:05.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:04 smithi203 ceph-mon[36693]: pgmap v728: 97 pgs: 97 active+clean; 3.6 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:13:05.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:04 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2052993134' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:13:05.633 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-23T23:13:05.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:13:05.634 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-23T23:13:05.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.871+0000 7f6280a00640 1 -- 172.21.15.12:0/377569638 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f627c0756e0 msgr2=0x7f627c073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:05.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.871+0000 7f6280a00640 1 --2- 172.21.15.12:0/377569638 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f627c0756e0 0x7f627c073b30 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f626c009900 tx=0x7f626c02bc40 comp rx=0 tx=0).stop 2024-03-23T23:13:05.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.871+0000 7f6280a00640 1 -- 172.21.15.12:0/377569638 shutdown_connections 2024-03-23T23:13:05.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.871+0000 7f6280a00640 1 --2- 172.21.15.12:0/377569638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f627c109460 0x7f627c10b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.871+0000 7f6280a00640 1 --2- 172.21.15.12:0/377569638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f627c074100 0x7f627c074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.871+0000 7f6280a00640 1 --2- 172.21.15.12:0/377569638 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f627c0756e0 0x7f627c073b30 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.871+0000 7f6280a00640 1 -- 172.21.15.12:0/377569638 >> 172.21.15.12:0/377569638 conn(0x7f627c0fcb10 msgr2=0x7f627c0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:05.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.871+0000 7f6280a00640 1 -- 172.21.15.12:0/377569638 shutdown_connections 2024-03-23T23:13:05.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.871+0000 7f6280a00640 1 -- 172.21.15.12:0/377569638 wait complete. 2024-03-23T23:13:05.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.872+0000 7f6280a00640 1 Processor -- start 2024-03-23T23:13:05.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.872+0000 7f6280a00640 1 -- start start 2024-03-23T23:13:05.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.872+0000 7f6280a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f627c074100 0x7f627c199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:05.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.872+0000 7f6280a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f627c109460 0x7f627c19a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:05.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.872+0000 7f6280a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f627c19f240 0x7f627c1a1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:05.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.872+0000 7f6280a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627c10f430 con 0x7f627c074100 2024-03-23T23:13:05.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.872+0000 7f6280a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627c10f2b0 con 0x7f627c109460 2024-03-23T23:13:05.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.872+0000 7f6280a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627c10f5b0 con 0x7f627c19f240 2024-03-23T23:13:05.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.873+0000 7f627b400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f627c074100 0x7f627c199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:05.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.873+0000 7f627b400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f627c074100 0x7f627c199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43610/0 (socket says 172.21.15.12:43610) 2024-03-23T23:13:05.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.873+0000 7f627b400640 1 -- 172.21.15.12:0/3137133067 learned_addr learned my addr 172.21.15.12:0/3137133067 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:05.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.873+0000 7f627aa00640 1 --2- 172.21.15.12:0/3137133067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f627c109460 0x7f627c19a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:05.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.873+0000 7f627be00640 1 --2- 172.21.15.12:0/3137133067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f627c19f240 0x7f627c1a1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:05.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.873+0000 7f627b400640 1 -- 172.21.15.12:0/3137133067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f627c19f240 msgr2=0x7f627c1a1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:05.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.873+0000 7f627b400640 1 --2- 172.21.15.12:0/3137133067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f627c19f240 0x7f627c1a1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.873+0000 7f627b400640 1 -- 172.21.15.12:0/3137133067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f627c109460 msgr2=0x7f627c19a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:05.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.873+0000 7f627b400640 1 --2- 172.21.15.12:0/3137133067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f627c109460 0x7f627c19a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.873+0000 7f627b400640 1 -- 172.21.15.12:0/3137133067 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f627c1a1cd0 con 0x7f627c074100 2024-03-23T23:13:05.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.873+0000 7f627aa00640 1 --2- 172.21.15.12:0/3137133067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f627c109460 0x7f627c19a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:05.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.873+0000 7f627b400640 1 --2- 172.21.15.12:0/3137133067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f627c074100 0x7f627c199cb0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f626c02c250 tx=0x7f626c02c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:05.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.874+0000 7f6273e00640 1 -- 172.21.15.12:0/3137133067 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626c041070 con 0x7f627c074100 2024-03-23T23:13:05.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.874+0000 7f6280a00640 1 -- 172.21.15.12:0/3137133067 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f627c1a1f60 con 0x7f627c074100 2024-03-23T23:13:05.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.874+0000 7f6273e00640 1 -- 172.21.15.12:0/3137133067 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f626c004450 con 0x7f627c074100 2024-03-23T23:13:05.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.874+0000 7f6273e00640 1 -- 172.21.15.12:0/3137133067 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626c003e50 con 0x7f627c074100 2024-03-23T23:13:05.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.874+0000 7f6280a00640 1 -- 172.21.15.12:0/3137133067 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f627c1a2310 con 0x7f627c074100 2024-03-23T23:13:05.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.878+0000 7f6284db1740 1 Processor -- start 2024-03-23T23:13:05.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.878+0000 7f6284db1740 1 -- start start 2024-03-23T23:13:05.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.879+0000 7f6273e00640 1 -- 172.21.15.12:0/3137133067 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f626c044070 con 0x7f627c074100 2024-03-23T23:13:05.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.879+0000 7f6284db1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f742b0 0x5595f3f746d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:05.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.879+0000 7f6284db1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595f3f74ca0 0x5595f3f797a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:05.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.879+0000 7f6284db1740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595f3f79e10 0x5595f3f7c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:05.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.879+0000 7f6284db1740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595f3ed9eb0 con 0x5595f3f79e10 2024-03-23T23:13:05.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.879+0000 7f6284db1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595f3e82a60 con 0x5595f3f742b0 2024-03-23T23:13:05.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.879+0000 7f6284db1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595f3ed5f50 con 0x5595f3f74ca0 2024-03-23T23:13:05.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.879+0000 7f627be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f742b0 0x5595f3f746d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:05.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.879+0000 7f627be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f742b0 0x5595f3f746d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:58166/0 (socket says 172.21.15.12:58166) 2024-03-23T23:13:05.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.879+0000 7f627be00640 1 -- 172.21.15.12:0/2891607029 learned_addr learned my addr 172.21.15.12:0/2891607029 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:05.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.879+0000 7f627b400640 1 --2- 172.21.15.12:0/2891607029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595f3f74ca0 0x5595f3f797a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:05.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.879+0000 7f627aa00640 1 --2- 172.21.15.12:0/2891607029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595f3f79e10 0x5595f3f7c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:05.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.880+0000 7f627be00640 1 -- 172.21.15.12:0/2891607029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595f3f74ca0 msgr2=0x5595f3f797a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:05.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.880+0000 7f627be00640 1 --2- 172.21.15.12:0/2891607029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595f3f74ca0 0x5595f3f797a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.880+0000 7f627be00640 1 -- 172.21.15.12:0/2891607029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595f3f79e10 msgr2=0x5595f3f7c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:05.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.880+0000 7f627be00640 1 --2- 172.21.15.12:0/2891607029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595f3f79e10 0x5595f3f7c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.880+0000 7f627be00640 1 -- 172.21.15.12:0/2891607029 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595f3cb67e0 con 0x5595f3f742b0 2024-03-23T23:13:05.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.880+0000 7f627aa00640 1 --2- 172.21.15.12:0/2891607029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595f3f79e10 0x5595f3f7c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:13:05.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.880+0000 7f627be00640 1 --2- 172.21.15.12:0/2891607029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f742b0 0x5595f3f746d0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f6274009fd0 tx=0x7f6274010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7df0c42e9c01e18 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:05.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.880+0000 7f6267e00640 1 -- 172.21.15.12:0/2891607029 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6274011700 con 0x5595f3f742b0 2024-03-23T23:13:05.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.880+0000 7f6267e00640 1 -- 172.21.15.12:0/2891607029 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6274011d80 con 0x5595f3f742b0 2024-03-23T23:13:05.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.880+0000 7f6267e00640 1 -- 172.21.15.12:0/2891607029 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627401a380 con 0x5595f3f742b0 2024-03-23T23:13:05.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.880+0000 7f6273e00640 1 --2- 172.21.15.12:0/3137133067 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f62600a29a0 0x7f62600a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:05.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.881+0000 7f6273e00640 1 -- 172.21.15.12:0/3137133067 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f626c0f06f0 con 0x7f627c074100 2024-03-23T23:13:05.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.881+0000 7f627b400640 1 --2- 172.21.15.12:0/3137133067 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f62600a29a0 0x7f62600a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:05.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.881+0000 7f6284db1740 1 -- 172.21.15.12:0/2891607029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f742b0 msgr2=0x5595f3f746d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:05.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.881+0000 7f6284db1740 1 --2- 172.21.15.12:0/2891607029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f742b0 0x5595f3f746d0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f6274009fd0 tx=0x7f6274010860 comp rx=0 tx=0).stop 2024-03-23T23:13:05.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.881+0000 7f6284db1740 1 -- 172.21.15.12:0/2891607029 shutdown_connections 2024-03-23T23:13:05.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.881+0000 7f6284db1740 1 --2- 172.21.15.12:0/2891607029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595f3f79e10 0x5595f3f7c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.881+0000 7f6284db1740 1 --2- 172.21.15.12:0/2891607029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595f3f74ca0 0x5595f3f797a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.881+0000 7f6284db1740 1 --2- 172.21.15.12:0/2891607029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f742b0 0x5595f3f746d0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.881+0000 7f6284db1740 1 -- 172.21.15.12:0/2891607029 >> 172.21.15.12:0/2891607029 conn(0x5595f3f6f820 msgr2=0x5595f3f6fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:05.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.881+0000 7f6284db1740 1 -- 172.21.15.12:0/2891607029 shutdown_connections 2024-03-23T23:13:05.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.881+0000 7f6284db1740 1 -- 172.21.15.12:0/2891607029 wait complete. 2024-03-23T23:13:05.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.881+0000 7f627b400640 1 --2- 172.21.15.12:0/3137133067 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f62600a29a0 0x7f62600a4e60 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f626c0bcfd0 tx=0x7f626c0f6530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:05.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.882+0000 7f6284db1740 1 -- start start 2024-03-23T23:13:05.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.882+0000 7f6284db1740 1 Processor -- start 2024-03-23T23:13:05.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.883+0000 7f6284db1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595f3f742b0 0x5595f3f7e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:05.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.883+0000 7f6284db1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f74ca0 0x5595f3f7ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:05.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.883+0000 7f6284db1740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595f3f79e10 0x5595f3f85a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:05.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.883+0000 7f6284db1740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595f3f7cdc0 con 0x5595f3f79e10 2024-03-23T23:13:05.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.883+0000 7f6284db1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595f3e95eb0 con 0x5595f3f74ca0 2024-03-23T23:13:05.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.883+0000 7f6284db1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595f3f7cf40 con 0x5595f3f742b0 2024-03-23T23:13:05.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.883+0000 7f6284db1740 10 client.0 _ll_register_callbacks cb 0x5595f3e8a150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:13:05.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.883+0000 7f627aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f74ca0 0x5595f3f7ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:05.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.883+0000 7f627aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f74ca0 0x5595f3f7ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:58180/0 (socket says 172.21.15.12:58180) 2024-03-23T23:13:05.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.883+0000 7f627aa00640 1 -- 172.21.15.12:0/3167297780 learned_addr learned my addr 172.21.15.12:0/3167297780 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:05.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.883+0000 7f627be00640 1 --2- 172.21.15.12:0/3167297780 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595f3f79e10 0x5595f3f85a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:05.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.883+0000 7f627aa00640 1 -- 172.21.15.12:0/3167297780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595f3f742b0 msgr2=0x5595f3f7e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:05.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.883+0000 7f627be00640 1 --2- 172.21.15.12:0/3167297780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595f3f742b0 0x5595f3f7e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:05.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.883+0000 7f627aa00640 1 --2- 172.21.15.12:0/3167297780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595f3f742b0 0x5595f3f7e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.883+0000 7f627aa00640 1 -- 172.21.15.12:0/3167297780 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595f3f79e10 msgr2=0x5595f3f85a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:05.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.883+0000 7f627aa00640 1 --2- 172.21.15.12:0/3167297780 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595f3f79e10 0x5595f3f85a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.883+0000 7f627aa00640 1 -- 172.21.15.12:0/3167297780 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5595f3d16f70 con 0x5595f3f74ca0 2024-03-23T23:13:05.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.883+0000 7f627be00640 1 --2- 172.21.15.12:0/3167297780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595f3f742b0 0x5595f3f7e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:13:05.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.884+0000 7f627aa00640 1 --2- 172.21.15.12:0/3167297780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f74ca0 0x5595f3f7ea10 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f62680129f0 tx=0x7f6268012eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9af0201429d62de6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:05.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.884+0000 7f6264c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:13:05.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.884+0000 7f6264c00640 1 -- 172.21.15.12:0/3167297780 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626800fac0 con 0x5595f3f74ca0 2024-03-23T23:13:05.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.884+0000 7f6284db1740 1 -- 172.21.15.12:0/3167297780 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6274011d80 con 0x5595f3f74ca0 2024-03-23T23:13:05.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.884+0000 7f6264c00640 1 -- 172.21.15.12:0/3167297780 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f626800fc60 con 0x5595f3f74ca0 2024-03-23T23:13:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.884+0000 7f6264c00640 1 -- 172.21.15.12:0/3167297780 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f626800fe60 con 0x5595f3f74ca0 2024-03-23T23:13:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.884+0000 7f6284db1740 10 client.28517 fetch_fsmap learned FSMap version 23 2024-03-23T23:13:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.884+0000 7f6284db1740 1 -- 172.21.15.12:0/3167297780 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5595f3bae170 con 0x5595f3f74ca0 2024-03-23T23:13:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.885+0000 7f6264c00640 1 -- 172.21.15.12:0/3167297780 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6274011d80 con 0x5595f3f74ca0 2024-03-23T23:13:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.885+0000 7f6264c00640 1 -- 172.21.15.12:0/3167297780 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f626801c470 con 0x5595f3f74ca0 2024-03-23T23:13:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.885+0000 7f6284db1740 10 client.28517 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:13:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.885+0000 7f6284db1740 10 client.28517 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:13:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.885+0000 7f6284db1740 20 client.28517 populate_metadata read hostname 'smithi012' 2024-03-23T23:13:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.885+0000 7f6284db1740 1 --2- 172.21.15.12:0/3167297780 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5595f3f8a320 0x5595f3f8c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.885+0000 7f6284db1740 4 client.28517 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:13:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.885+0000 7f6284db1740 1 -- 172.21.15.12:0/3167297780 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5595f3ee73f0 con 0x5595f3f8a320 2024-03-23T23:13:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.885+0000 7f627be00640 1 --2- 172.21.15.12:0/3167297780 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5595f3f8a320 0x5595f3f8c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.886+0000 7f627be00640 1 --2- 172.21.15.12:0/3167297780 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5595f3f8a320 0x5595f3f8c740 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d175fc58ec3fe412 server_cookie=335b94cc359e6098 in_seq=0 out_seq=0 2024-03-23T23:13:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.886+0000 7f6264c00640 10 client.28517 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:05.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.886+0000 7f6264c00640 1 -- 172.21.15.12:0/3167297780 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5595f3ee73f0 con 0x5595f3f8a320 2024-03-23T23:13:05.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.887+0000 7f6264c00640 10 client.28517 handle_command_reply: tid=0 2024-03-23T23:13:05.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.887+0000 7f6284db1740 1 client.28517 shutdown 2024-03-23T23:13:05.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.887+0000 7f6284db1740 1 -- 172.21.15.12:0/3167297780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f74ca0 msgr2=0x5595f3f7ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:05.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.887+0000 7f6284db1740 1 --2- 172.21.15.12:0/3167297780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f74ca0 0x5595f3f7ea10 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f62680129f0 tx=0x7f6268012eb0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.887+0000 7f6284db1740 1 -- 172.21.15.12:0/3167297780 shutdown_connections 2024-03-23T23:13:05.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.887+0000 7f6284db1740 1 --2- 172.21.15.12:0/3167297780 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5595f3f8a320 0x5595f3f8c740 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.887+0000 7f6284db1740 1 --2- 172.21.15.12:0/3167297780 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595f3f79e10 0x5595f3f85a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.887+0000 7f6284db1740 1 --2- 172.21.15.12:0/3167297780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f74ca0 0x5595f3f7ea10 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.887+0000 7f6284db1740 1 --2- 172.21.15.12:0/3167297780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595f3f742b0 0x5595f3f7e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.887+0000 7f6264c00640 0 client.28517 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:05.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.887+0000 7f6284db1740 1 -- 172.21.15.12:0/3167297780 >> 172.21.15.12:0/3167297780 conn(0x5595f3f6f820 msgr2=0x5595f3f6ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:05.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.887+0000 7f6284db1740 1 -- 172.21.15.12:0/3167297780 shutdown_connections 2024-03-23T23:13:05.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.887+0000 7f6284db1740 1 -- 172.21.15.12:0/3167297780 wait complete. 2024-03-23T23:13:05.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.887+0000 7f6284db1740 20 client.28517 trim_cache size 0 max 16384 2024-03-23T23:13:05.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.905+0000 7f6284db1740 1 Processor -- start 2024-03-23T23:13:05.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.905+0000 7f6284db1740 1 -- start start 2024-03-23T23:13:05.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.905+0000 7f6284db1740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595f3f742b0 0x5595f3f75740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:05.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.905+0000 7f6284db1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595f3f74ca0 0x5595f3f85a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:05.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.905+0000 7f6284db1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f79e10 0x5595f3f75c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:05.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.905+0000 7f6284db1740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627c1ad500 con 0x5595f3f742b0 2024-03-23T23:13:05.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.905+0000 7f6284db1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627c1ab000 con 0x5595f3f79e10 2024-03-23T23:13:05.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.905+0000 7f6284db1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627c1a90a0 con 0x5595f3f74ca0 2024-03-23T23:13:05.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.906+0000 7f627aa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595f3f74ca0 0x5595f3f85a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.906+0000 7f627aa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595f3f74ca0 0x5595f3f85a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49434/0 (socket says 172.21.15.12:49434) 2024-03-23T23:13:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.906+0000 7f627aa00640 1 -- 172.21.15.12:0/4222036516 learned_addr learned my addr 172.21.15.12:0/4222036516 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.906+0000 7f627be00640 1 --2- 172.21.15.12:0/4222036516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595f3f742b0 0x5595f3f75740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.906+0000 7f627be00640 1 --2- 172.21.15.12:0/4222036516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f79e10 0x5595f3f75c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.906+0000 7f627aa00640 1 -- 172.21.15.12:0/4222036516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f79e10 msgr2=0x5595f3f75c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.906+0000 7f627aa00640 1 --2- 172.21.15.12:0/4222036516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f79e10 0x5595f3f75c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.906+0000 7f627aa00640 1 -- 172.21.15.12:0/4222036516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595f3f742b0 msgr2=0x5595f3f75740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.906+0000 7f627aa00640 1 --2- 172.21.15.12:0/4222036516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595f3f742b0 0x5595f3f75740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.906+0000 7f627aa00640 1 -- 172.21.15.12:0/4222036516 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595f3f7f4c0 con 0x5595f3f74ca0 2024-03-23T23:13:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.906+0000 7f627be00640 1 --2- 172.21.15.12:0/4222036516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f79e10 0x5595f3f75c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:13:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.907+0000 7f627aa00640 1 --2- 172.21.15.12:0/4222036516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595f3f74ca0 0x5595f3f85a10 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f6268000c00 tx=0x7f6268009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c96a55d3a2ab68ba server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.907+0000 7f6267400640 1 -- 172.21.15.12:0/4222036516 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626801cc90 con 0x5595f3f74ca0 2024-03-23T23:13:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.907+0000 7f6267400640 1 -- 172.21.15.12:0/4222036516 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f626800fac0 con 0x5595f3f74ca0 2024-03-23T23:13:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.907+0000 7f6267400640 1 -- 172.21.15.12:0/4222036516 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6268003830 con 0x5595f3f74ca0 2024-03-23T23:13:05.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.907+0000 7f6284db1740 1 -- 172.21.15.12:0/4222036516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595f3f74ca0 msgr2=0x5595f3f85a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:05.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.907+0000 7f6284db1740 1 --2- 172.21.15.12:0/4222036516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595f3f74ca0 0x5595f3f85a10 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f6268000c00 tx=0x7f6268009240 comp rx=0 tx=0).stop 2024-03-23T23:13:05.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.908+0000 7f6284db1740 1 -- 172.21.15.12:0/4222036516 shutdown_connections 2024-03-23T23:13:05.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.908+0000 7f6284db1740 1 --2- 172.21.15.12:0/4222036516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f79e10 0x5595f3f75c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.908+0000 7f6284db1740 1 --2- 172.21.15.12:0/4222036516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595f3f74ca0 0x5595f3f85a10 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.908+0000 7f6284db1740 1 --2- 172.21.15.12:0/4222036516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595f3f742b0 0x5595f3f75740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.908+0000 7f6284db1740 1 -- 172.21.15.12:0/4222036516 >> 172.21.15.12:0/4222036516 conn(0x5595f3f6f820 msgr2=0x5595f3f73310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:05.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.908+0000 7f6284db1740 1 -- 172.21.15.12:0/4222036516 shutdown_connections 2024-03-23T23:13:05.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.908+0000 7f6284db1740 1 -- 172.21.15.12:0/4222036516 wait complete. 2024-03-23T23:13:05.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.908+0000 7f6284db1740 1 -- start start 2024-03-23T23:13:05.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.908+0000 7f6284db1740 1 Processor -- start 2024-03-23T23:13:05.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.909+0000 7f6284db1740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595f3f742b0 0x5595f3f84970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:05.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.909+0000 7f6284db1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f74ca0 0x5595f3f84eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:05.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.909+0000 7f6284db1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595f3f79e10 0x5595f3fe5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:05.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.909+0000 7f6284db1740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627c1a96a0 con 0x5595f3f742b0 2024-03-23T23:13:05.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.909+0000 7f6284db1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627c1aa9a0 con 0x5595f3f74ca0 2024-03-23T23:13:05.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.909+0000 7f6284db1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627c1a9ce0 con 0x5595f3f79e10 2024-03-23T23:13:05.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.909+0000 7f6284db1740 10 client.0 _ll_register_callbacks cb 0x7f6268009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:13:05.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.909+0000 7f627aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f74ca0 0x5595f3f84eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:05.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.909+0000 7f627aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f74ca0 0x5595f3f84eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:58186/0 (socket says 172.21.15.12:58186) 2024-03-23T23:13:05.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.909+0000 7f627aa00640 1 -- 172.21.15.12:0/329100007 learned_addr learned my addr 172.21.15.12:0/329100007 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:05.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.909+0000 7f627be00640 1 --2- 172.21.15.12:0/329100007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595f3f742b0 0x5595f3f84970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:05.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.909+0000 7f627be00640 1 --2- 172.21.15.12:0/329100007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595f3f79e10 0x5595f3fe5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:05.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.909+0000 7f627aa00640 1 -- 172.21.15.12:0/329100007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595f3f79e10 msgr2=0x5595f3fe5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:05.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.909+0000 7f627aa00640 1 --2- 172.21.15.12:0/329100007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595f3f79e10 0x5595f3fe5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.909+0000 7f627aa00640 1 -- 172.21.15.12:0/329100007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595f3f742b0 msgr2=0x5595f3f84970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:05.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.909+0000 7f627aa00640 1 --2- 172.21.15.12:0/329100007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595f3f742b0 0x5595f3f84970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.909+0000 7f627aa00640 1 -- 172.21.15.12:0/329100007 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5595f3fe6120 con 0x5595f3f74ca0 2024-03-23T23:13:05.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.909+0000 7f627be00640 1 --2- 172.21.15.12:0/329100007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595f3f742b0 0x5595f3f84970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:13:05.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.910+0000 7f627aa00640 1 --2- 172.21.15.12:0/329100007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f74ca0 0x5595f3f84eb0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f626801c9b0 tx=0x7f6268005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c55a7a2558ab989e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:05.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.910+0000 7f6264c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:13:05.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.910+0000 7f6264c00640 1 -- 172.21.15.12:0/329100007 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626800fc70 con 0x5595f3f74ca0 2024-03-23T23:13:05.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.910+0000 7f6284db1740 1 -- 172.21.15.12:0/329100007 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f626800fac0 con 0x5595f3f74ca0 2024-03-23T23:13:05.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.910+0000 7f6264c00640 1 -- 172.21.15.12:0/329100007 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f6268018040 con 0x5595f3f74ca0 2024-03-23T23:13:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.910+0000 7f6264c00640 1 -- 172.21.15.12:0/329100007 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6268005230 con 0x5595f3f74ca0 2024-03-23T23:13:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.910+0000 7f6284db1740 10 client.28523 fetch_fsmap learned FSMap version 23 2024-03-23T23:13:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.910+0000 7f6284db1740 1 -- 172.21.15.12:0/329100007 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5595f3fe7460 con 0x5595f3f74ca0 2024-03-23T23:13:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.911+0000 7f6264c00640 1 -- 172.21.15.12:0/329100007 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f626800fac0 con 0x5595f3f74ca0 2024-03-23T23:13:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.911+0000 7f6264c00640 1 -- 172.21.15.12:0/329100007 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f6268030580 con 0x5595f3f74ca0 2024-03-23T23:13:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.911+0000 7f6284db1740 10 client.28523 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:13:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.911+0000 7f6284db1740 10 client.28523 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:13:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.911+0000 7f6284db1740 20 client.28523 populate_metadata read hostname 'smithi012' 2024-03-23T23:13:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.911+0000 7f6284db1740 1 --2- 172.21.15.12:0/329100007 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5595f3fe7ca0 0x5595f3fea0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.911+0000 7f6284db1740 4 client.28523 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-23T23:13:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.911+0000 7f6284db1740 1 -- 172.21.15.12:0/329100007 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5595f3ee1740 con 0x5595f3fe7ca0 2024-03-23T23:13:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.911+0000 7f627be00640 1 --2- 172.21.15.12:0/329100007 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5595f3fe7ca0 0x5595f3fea0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.911+0000 7f627be00640 1 --2- 172.21.15.12:0/329100007 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5595f3fe7ca0 0x5595f3fea0c0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd263c293dda764d server_cookie=a650160f8e7306c4 in_seq=0 out_seq=0 2024-03-23T23:13:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.911+0000 7f6264c00640 10 client.28523 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.912+0000 7f6264c00640 1 -- 172.21.15.12:0/329100007 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5595f3ee1740 con 0x5595f3fe7ca0 2024-03-23T23:13:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.912+0000 7f6264c00640 10 client.28523 handle_command_reply: tid=0 2024-03-23T23:13:05.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.912+0000 7f6284db1740 1 client.28523 shutdown 2024-03-23T23:13:05.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.913+0000 7f6284db1740 1 -- 172.21.15.12:0/329100007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f74ca0 msgr2=0x5595f3f84eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:05.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.913+0000 7f6284db1740 1 --2- 172.21.15.12:0/329100007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f74ca0 0x5595f3f84eb0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f626801c9b0 tx=0x7f6268005dc0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.913+0000 7f6284db1740 1 -- 172.21.15.12:0/329100007 shutdown_connections 2024-03-23T23:13:05.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.913+0000 7f6284db1740 1 --2- 172.21.15.12:0/329100007 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5595f3fe7ca0 0x5595f3fea0c0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.913+0000 7f6284db1740 1 --2- 172.21.15.12:0/329100007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595f3f79e10 0x5595f3fe5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.913+0000 7f6284db1740 1 --2- 172.21.15.12:0/329100007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595f3f74ca0 0x5595f3f84eb0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.913+0000 7f6284db1740 1 --2- 172.21.15.12:0/329100007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595f3f742b0 0x5595f3f84970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.913+0000 7f6284db1740 1 -- 172.21.15.12:0/329100007 >> 172.21.15.12:0/329100007 conn(0x5595f3f6f820 msgr2=0x5595f3f6e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:05.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.913+0000 7f6264c00640 0 client.28523 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:05.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.913+0000 7f6284db1740 1 -- 172.21.15.12:0/329100007 shutdown_connections 2024-03-23T23:13:05.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.913+0000 7f6284db1740 1 -- 172.21.15.12:0/329100007 wait complete. 2024-03-23T23:13:05.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.913+0000 7f6284db1740 20 client.28523 trim_cache size 0 max 16384 2024-03-23T23:13:05.917 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:13:05.917 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-03-23T23:13:05.917 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "f5b3c141-92ce-44a7-9616-f5830479a338", 2024-03-23T23:13:05.917 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-03-23T23:13:05.917 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:13:05.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.914+0000 7f6280a00640 1 -- 172.21.15.12:0/3137133067 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f62600a29a0 msgr2=0x7f62600a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:05.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.914+0000 7f6280a00640 1 --2- 172.21.15.12:0/3137133067 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f62600a29a0 0x7f62600a4e60 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f626c0bcfd0 tx=0x7f626c0f6530 comp rx=0 tx=0).stop 2024-03-23T23:13:05.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.914+0000 7f6280a00640 1 -- 172.21.15.12:0/3137133067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f627c074100 msgr2=0x7f627c199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:05.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.914+0000 7f6280a00640 1 --2- 172.21.15.12:0/3137133067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f627c074100 0x7f627c199cb0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f626c02c250 tx=0x7f626c02c6e0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.914+0000 7f6280a00640 1 -- 172.21.15.12:0/3137133067 shutdown_connections 2024-03-23T23:13:05.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.914+0000 7f6280a00640 1 --2- 172.21.15.12:0/3137133067 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f62600a29a0 0x7f62600a4e60 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.914+0000 7f6280a00640 1 --2- 172.21.15.12:0/3137133067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f627c19f240 0x7f627c1a1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.914+0000 7f6280a00640 1 --2- 172.21.15.12:0/3137133067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f627c109460 0x7f627c19a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.914+0000 7f6280a00640 1 --2- 172.21.15.12:0/3137133067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f627c074100 0x7f627c199cb0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:05.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.914+0000 7f6280a00640 1 -- 172.21.15.12:0/3137133067 >> 172.21.15.12:0/3137133067 conn(0x7f627c0fcb10 msgr2=0x7f627c105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:05.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.914+0000 7f6280a00640 1 -- 172.21.15.12:0/3137133067 shutdown_connections 2024-03-23T23:13:05.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:05.914+0000 7f6280a00640 1 -- 172.21.15.12:0/3137133067 wait complete. 2024-03-23T23:13:05.931 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:13:05.931 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:13:06.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.146+0000 7fbdf3800640 1 -- 172.21.15.12:0/3201050309 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdec074100 msgr2=0x7fbdec074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:06.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.146+0000 7fbdf3800640 1 --2- 172.21.15.12:0/3201050309 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdec074100 0x7fbdec074580 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8009900 tx=0x7fbdd802bc40 comp rx=0 tx=0).stop 2024-03-23T23:13:06.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.147+0000 7fbdf3800640 1 -- 172.21.15.12:0/3201050309 shutdown_connections 2024-03-23T23:13:06.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.147+0000 7fbdf3800640 1 --2- 172.21.15.12:0/3201050309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbdec109570 0x7fbdec10ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.147+0000 7fbdf3800640 1 --2- 172.21.15.12:0/3201050309 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdec074100 0x7fbdec074580 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.147+0000 7fbdf3800640 1 --2- 172.21.15.12:0/3201050309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdec0756e0 0x7fbdec073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.147+0000 7fbdf3800640 1 -- 172.21.15.12:0/3201050309 >> 172.21.15.12:0/3201050309 conn(0x7fbdec0fcc40 msgr2=0x7fbdec0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:06.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.147+0000 7fbdf3800640 1 -- 172.21.15.12:0/3201050309 shutdown_connections 2024-03-23T23:13:06.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.147+0000 7fbdf3800640 1 -- 172.21.15.12:0/3201050309 wait complete. 2024-03-23T23:13:06.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.148+0000 7fbdf3800640 1 Processor -- start 2024-03-23T23:13:06.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.148+0000 7fbdf3800640 1 -- start start 2024-03-23T23:13:06.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.148+0000 7fbdf3800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbdec074100 0x7fbdec199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:06.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.148+0000 7fbdf3800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdec0756e0 0x7fbdec19a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:06.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.148+0000 7fbdf3800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdec109570 0x7fbdec1a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:06.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.148+0000 7fbdf3800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec10f5c0 con 0x7fbdec109570 2024-03-23T23:13:06.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.148+0000 7fbdf3800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec10f440 con 0x7fbdec074100 2024-03-23T23:13:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.148+0000 7fbdf3800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec10f740 con 0x7fbdec0756e0 2024-03-23T23:13:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.149+0000 7fbdebe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdec0756e0 0x7fbdec19a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.149+0000 7fbdebe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdec0756e0 0x7fbdec19a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49470/0 (socket says 172.21.15.12:49470) 2024-03-23T23:13:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.149+0000 7fbdebe00640 1 -- 172.21.15.12:0/1156210454 learned_addr learned my addr 172.21.15.12:0/1156210454 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.149+0000 7fbdf1800640 1 --2- 172.21.15.12:0/1156210454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdec109570 0x7fbdec1a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.149+0000 7fbdf0e00640 1 --2- 172.21.15.12:0/1156210454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbdec074100 0x7fbdec199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.149+0000 7fbdebe00640 1 -- 172.21.15.12:0/1156210454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbdec074100 msgr2=0x7fbdec199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.149+0000 7fbdebe00640 1 --2- 172.21.15.12:0/1156210454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbdec074100 0x7fbdec199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.149+0000 7fbdebe00640 1 -- 172.21.15.12:0/1156210454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdec109570 msgr2=0x7fbdec1a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.149+0000 7fbdebe00640 1 --2- 172.21.15.12:0/1156210454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdec109570 0x7fbdec1a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.149+0000 7fbdebe00640 1 -- 172.21.15.12:0/1156210454 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdec1a1840 con 0x7fbdec0756e0 2024-03-23T23:13:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.149+0000 7fbdf0e00640 1 --2- 172.21.15.12:0/1156210454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbdec074100 0x7fbdec199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:13:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.149+0000 7fbdebe00640 1 --2- 172.21.15.12:0/1156210454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdec0756e0 0x7fbdec19a170 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fbdd802c8f0 tx=0x7fbdd8005e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:06.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.150+0000 7fbde9600640 1 -- 172.21.15.12:0/1156210454 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd8041070 con 0x7fbdec0756e0 2024-03-23T23:13:06.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.150+0000 7fbde9600640 1 -- 172.21.15.12:0/1156210454 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbdd80054e0 con 0x7fbdec0756e0 2024-03-23T23:13:06.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.150+0000 7fbde9600640 1 -- 172.21.15.12:0/1156210454 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd8034440 con 0x7fbdec0756e0 2024-03-23T23:13:06.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.150+0000 7fbdf3800640 1 -- 172.21.15.12:0/1156210454 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdec1a1ad0 con 0x7fbdec0756e0 2024-03-23T23:13:06.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.150+0000 7fbdf3800640 1 -- 172.21.15.12:0/1156210454 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdec1a1f40 con 0x7fbdec0756e0 2024-03-23T23:13:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.153+0000 7fbde9600640 1 -- 172.21.15.12:0/1156210454 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbdd8044070 con 0x7fbdec0756e0 2024-03-23T23:13:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.153+0000 7fbdf6445740 1 Processor -- start 2024-03-23T23:13:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.153+0000 7fbdf6445740 1 -- start start 2024-03-23T23:13:06.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.154+0000 7fbdf6445740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563b9e967e80 0x563b9e9682a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:06.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.154+0000 7fbdf6445740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b9e968870 0x563b9e96d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:06.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.154+0000 7fbdf6445740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9e96d9e0 0x563b9e96fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:06.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.154+0000 7fbdf6445740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b9e8cd250 con 0x563b9e968870 2024-03-23T23:13:06.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.154+0000 7fbdf6445740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b9e875a30 con 0x563b9e96d9e0 2024-03-23T23:13:06.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.154+0000 7fbdf6445740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b9e8c92f0 con 0x563b9e967e80 2024-03-23T23:13:06.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.154+0000 7fbdf1800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563b9e967e80 0x563b9e9682a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:06.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.154+0000 7fbdebe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9e96d9e0 0x563b9e96fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:06.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.154+0000 7fbdf0e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b9e968870 0x563b9e96d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:06.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.154+0000 7fbdf1800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563b9e967e80 0x563b9e9682a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49478/0 (socket says 172.21.15.12:49478) 2024-03-23T23:13:06.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.154+0000 7fbdf1800640 1 -- 172.21.15.12:0/1864667912 learned_addr learned my addr 172.21.15.12:0/1864667912 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:06.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.154+0000 7fbdf1800640 1 -- 172.21.15.12:0/1864667912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9e96d9e0 msgr2=0x563b9e96fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:06.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.154+0000 7fbdf1800640 1 --2- 172.21.15.12:0/1864667912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9e96d9e0 0x563b9e96fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.154+0000 7fbdf1800640 1 -- 172.21.15.12:0/1864667912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b9e968870 msgr2=0x563b9e96d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:06.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.154+0000 7fbdf1800640 1 --2- 172.21.15.12:0/1864667912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b9e968870 0x563b9e96d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.154+0000 7fbdf1800640 1 -- 172.21.15.12:0/1864667912 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563b9e860690 con 0x563b9e967e80 2024-03-23T23:13:06.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.154+0000 7fbdf0e00640 1 --2- 172.21.15.12:0/1864667912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b9e968870 0x563b9e96d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:06.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.155+0000 7fbdf1800640 1 --2- 172.21.15.12:0/1864667912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563b9e967e80 0x563b9e9682a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fbde400a430 tx=0x7fbde4009650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfd1fc9306f5a127 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:06.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.155+0000 7fbde9600640 1 --2- 172.21.15.12:0/1156210454 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbdd40a2800 0x7fbdd40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:06.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.155+0000 7fbdde000640 1 -- 172.21.15.12:0/1864667912 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde4013790 con 0x563b9e967e80 2024-03-23T23:13:06.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.155+0000 7fbdde000640 1 -- 172.21.15.12:0/1864667912 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbde4019070 con 0x563b9e967e80 2024-03-23T23:13:06.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.155+0000 7fbdde000640 1 -- 172.21.15.12:0/1864667912 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde4014de0 con 0x563b9e967e80 2024-03-23T23:13:06.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.155+0000 7fbdf0e00640 1 --2- 172.21.15.12:0/1156210454 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbdd40a2800 0x7fbdd40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:06.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.155+0000 7fbde9600640 1 -- 172.21.15.12:0/1156210454 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fbdd80f0890 con 0x7fbdec0756e0 2024-03-23T23:13:06.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.156+0000 7fbdf6445740 1 -- 172.21.15.12:0/1864667912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563b9e967e80 msgr2=0x563b9e9682a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:06.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.156+0000 7fbdf6445740 1 --2- 172.21.15.12:0/1864667912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563b9e967e80 0x563b9e9682a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fbde400a430 tx=0x7fbde4009650 comp rx=0 tx=0).stop 2024-03-23T23:13:06.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.156+0000 7fbdf0e00640 1 --2- 172.21.15.12:0/1156210454 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbdd40a2800 0x7fbdd40a4cc0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x563b9e968c70 tx=0x7fbde00094c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:06.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.156+0000 7fbdf6445740 1 -- 172.21.15.12:0/1864667912 shutdown_connections 2024-03-23T23:13:06.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.156+0000 7fbdf6445740 1 --2- 172.21.15.12:0/1864667912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9e96d9e0 0x563b9e96fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.156+0000 7fbdf6445740 1 --2- 172.21.15.12:0/1864667912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b9e968870 0x563b9e96d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.156+0000 7fbdf6445740 1 --2- 172.21.15.12:0/1864667912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563b9e967e80 0x563b9e9682a0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.156+0000 7fbdf6445740 1 -- 172.21.15.12:0/1864667912 >> 172.21.15.12:0/1864667912 conn(0x563b9e930200 msgr2=0x563b9e9637f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:06.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.156+0000 7fbdf6445740 1 -- 172.21.15.12:0/1864667912 shutdown_connections 2024-03-23T23:13:06.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.156+0000 7fbdf6445740 1 -- 172.21.15.12:0/1864667912 wait complete. 2024-03-23T23:13:06.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.157+0000 7fbdf6445740 1 -- start start 2024-03-23T23:13:06.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.157+0000 7fbdf6445740 1 Processor -- start 2024-03-23T23:13:06.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.157+0000 7fbdf6445740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563b9e967e80 0x563b9e971e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:06.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.157+0000 7fbdf6445740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9e968870 0x563b9e9723c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:06.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.157+0000 7fbdf6445740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b9e96d9e0 0x563b9e979400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:06.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.157+0000 7fbdf6445740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b9e970990 con 0x563b9e96d9e0 2024-03-23T23:13:06.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.157+0000 7fbdf6445740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b9e888e80 con 0x563b9e968870 2024-03-23T23:13:06.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.157+0000 7fbdf6445740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b9e970b10 con 0x563b9e967e80 2024-03-23T23:13:06.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.157+0000 7fbdf6445740 10 client.0 _ll_register_callbacks cb 0x563b9e89b130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:13:06.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.157+0000 7fbdf0e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563b9e967e80 0x563b9e971e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:06.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.157+0000 7fbdebe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9e968870 0x563b9e9723c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:06.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.157+0000 7fbdf0e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563b9e967e80 0x563b9e971e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49480/0 (socket says 172.21.15.12:49480) 2024-03-23T23:13:06.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.158+0000 7fbdf0e00640 1 -- 172.21.15.12:0/1836152972 learned_addr learned my addr 172.21.15.12:0/1836152972 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:06.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.158+0000 7fbdf1800640 1 --2- 172.21.15.12:0/1836152972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b9e96d9e0 0x563b9e979400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:06.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.158+0000 7fbdf0e00640 1 -- 172.21.15.12:0/1836152972 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9e968870 msgr2=0x563b9e9723c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:06.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.158+0000 7fbdf0e00640 1 --2- 172.21.15.12:0/1836152972 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9e968870 0x563b9e9723c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.158+0000 7fbdf0e00640 1 -- 172.21.15.12:0/1836152972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b9e96d9e0 msgr2=0x563b9e979400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:06.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.158+0000 7fbdf0e00640 1 --2- 172.21.15.12:0/1836152972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b9e96d9e0 0x563b9e979400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.158+0000 7fbdf0e00640 1 -- 172.21.15.12:0/1836152972 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563b9e6ab7d0 con 0x563b9e967e80 2024-03-23T23:13:06.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.158+0000 7fbdebe00640 1 --2- 172.21.15.12:0/1836152972 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9e968870 0x563b9e9723c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:06.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.158+0000 7fbdf1800640 1 --2- 172.21.15.12:0/1836152972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b9e96d9e0 0x563b9e979400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:13:06.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.158+0000 7fbdf0e00640 1 --2- 172.21.15.12:0/1836152972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563b9e967e80 0x563b9e971e80 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fbde00126d0 tx=0x7fbde00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d0efd1fb1de01b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:06.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.158+0000 7fbdd2a00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:13:06.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.158+0000 7fbdd2a00640 1 -- 172.21.15.12:0/1836152972 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde0004220 con 0x563b9e967e80 2024-03-23T23:13:06.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.159+0000 7fbdd2a00640 1 -- 172.21.15.12:0/1836152972 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fbde00043c0 con 0x563b9e967e80 2024-03-23T23:13:06.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.159+0000 7fbdf6445740 1 -- 172.21.15.12:0/1836152972 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbde4019070 con 0x563b9e967e80 2024-03-23T23:13:06.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.159+0000 7fbdd2a00640 1 -- 172.21.15.12:0/1836152972 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbde001f630 con 0x563b9e967e80 2024-03-23T23:13:06.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.159+0000 7fbdf6445740 10 client.28498 fetch_fsmap learned FSMap version 23 2024-03-23T23:13:06.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.159+0000 7fbdf6445740 1 -- 172.21.15.12:0/1836152972 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563b9e5e13f0 con 0x563b9e967e80 2024-03-23T23:13:06.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.159+0000 7fbdd2a00640 1 -- 172.21.15.12:0/1836152972 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbde4019070 con 0x563b9e967e80 2024-03-23T23:13:06.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.159+0000 7fbdd2a00640 1 -- 172.21.15.12:0/1836152972 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fbde000f040 con 0x563b9e967e80 2024-03-23T23:13:06.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.159+0000 7fbdf6445740 10 client.28498 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:13:06.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.159+0000 7fbdf6445740 10 client.28498 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:13:06.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.159+0000 7fbdf6445740 20 client.28498 populate_metadata read hostname 'smithi012' 2024-03-23T23:13:06.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.159+0000 7fbdf6445740 1 --2- 172.21.15.12:0/1836152972 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563b9e97dd70 0x563b9e9801c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:06.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.160+0000 7fbdf6445740 4 client.28498 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:13:06.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.160+0000 7fbdf6445740 1 -- 172.21.15.12:0/1836152972 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563b9e8da790 con 0x563b9e97dd70 2024-03-23T23:13:06.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.160+0000 7fbdebe00640 1 --2- 172.21.15.12:0/1836152972 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563b9e97dd70 0x563b9e9801c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:06.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.160+0000 7fbdebe00640 1 --2- 172.21.15.12:0/1836152972 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563b9e97dd70 0x563b9e9801c0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=481a270539882700 server_cookie=b4c9cb348553614f in_seq=0 out_seq=0 2024-03-23T23:13:06.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.160+0000 7fbdd2a00640 10 client.28498 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:06.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.161+0000 7fbdd2a00640 1 -- 172.21.15.12:0/1836152972 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563b9e8da790 con 0x563b9e97dd70 2024-03-23T23:13:06.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.161+0000 7fbdd2a00640 10 client.28498 handle_command_reply: tid=0 2024-03-23T23:13:06.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.161+0000 7fbdf6445740 1 client.28498 shutdown 2024-03-23T23:13:06.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.161+0000 7fbdf6445740 1 -- 172.21.15.12:0/1836152972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563b9e967e80 msgr2=0x563b9e971e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:06.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.161+0000 7fbdf6445740 1 --2- 172.21.15.12:0/1836152972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563b9e967e80 0x563b9e971e80 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fbde00126d0 tx=0x7fbde00046c0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.161+0000 7fbdf6445740 1 -- 172.21.15.12:0/1836152972 shutdown_connections 2024-03-23T23:13:06.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.161+0000 7fbdf6445740 1 --2- 172.21.15.12:0/1836152972 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563b9e97dd70 0x563b9e9801c0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.161+0000 7fbdf6445740 1 --2- 172.21.15.12:0/1836152972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b9e96d9e0 0x563b9e979400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.161+0000 7fbdf6445740 1 --2- 172.21.15.12:0/1836152972 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9e968870 0x563b9e9723c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.161+0000 7fbdf6445740 1 --2- 172.21.15.12:0/1836152972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563b9e967e80 0x563b9e971e80 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.161+0000 7fbdd2a00640 0 client.28498 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:06.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.161+0000 7fbdf6445740 1 -- 172.21.15.12:0/1836152972 >> 172.21.15.12:0/1836152972 conn(0x563b9e930200 msgr2=0x563b9e962a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:06.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.161+0000 7fbdf6445740 1 -- 172.21.15.12:0/1836152972 shutdown_connections 2024-03-23T23:13:06.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.161+0000 7fbdf6445740 1 -- 172.21.15.12:0/1836152972 wait complete. 2024-03-23T23:13:06.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.162+0000 7fbdf6445740 20 client.28498 trim_cache size 0 max 16384 2024-03-23T23:13:06.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.179+0000 7fbdf6445740 1 Processor -- start 2024-03-23T23:13:06.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.179+0000 7fbdf6445740 1 -- start start 2024-03-23T23:13:06.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.179+0000 7fbdf6445740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9e967e80 0x563b9e972100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:06.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.179+0000 7fbdf6445740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b9e968870 0x563b9e979390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:06.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.179+0000 7fbdf6445740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563b9e96d9e0 0x563b9e966a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:06.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.179+0000 7fbdf6445740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec1ad1a0 con 0x563b9e968870 2024-03-23T23:13:06.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.179+0000 7fbdf6445740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec1aaca0 con 0x563b9e967e80 2024-03-23T23:13:06.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.179+0000 7fbdf6445740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec1a8d40 con 0x563b9e96d9e0 2024-03-23T23:13:06.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.180+0000 7fbdf0e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9e967e80 0x563b9e972100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:06.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.180+0000 7fbdf0e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9e967e80 0x563b9e972100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:58238/0 (socket says 172.21.15.12:58238) 2024-03-23T23:13:06.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.180+0000 7fbdf0e00640 1 -- 172.21.15.12:0/1351225582 learned_addr learned my addr 172.21.15.12:0/1351225582 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:06.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.180+0000 7fbdf1800640 1 --2- 172.21.15.12:0/1351225582 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563b9e96d9e0 0x563b9e966a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:06.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.180+0000 7fbdebe00640 1 --2- 172.21.15.12:0/1351225582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b9e968870 0x563b9e979390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:06.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.180+0000 7fbdf0e00640 1 -- 172.21.15.12:0/1351225582 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563b9e96d9e0 msgr2=0x563b9e966a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:06.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.180+0000 7fbdf0e00640 1 --2- 172.21.15.12:0/1351225582 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563b9e96d9e0 0x563b9e966a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.180+0000 7fbdf0e00640 1 -- 172.21.15.12:0/1351225582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b9e968870 msgr2=0x563b9e979390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:06.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.180+0000 7fbdf0e00640 1 --2- 172.21.15.12:0/1351225582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b9e968870 0x563b9e979390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.180+0000 7fbdf0e00640 1 -- 172.21.15.12:0/1351225582 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563b9e5a10f0 con 0x563b9e967e80 2024-03-23T23:13:06.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.180+0000 7fbdf1800640 1 --2- 172.21.15.12:0/1351225582 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563b9e96d9e0 0x563b9e966a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:06.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.181+0000 7fbdf0e00640 1 --2- 172.21.15.12:0/1351225582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9e967e80 0x563b9e972100 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fbde0003ce0 tx=0x7fbde0005f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75f1d507d7ec6d8a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:06.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.181+0000 7fbddd600640 1 -- 172.21.15.12:0/1351225582 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde00035f0 con 0x563b9e967e80 2024-03-23T23:13:06.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.181+0000 7fbddd600640 1 -- 172.21.15.12:0/1351225582 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbde0005500 con 0x563b9e967e80 2024-03-23T23:13:06.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.181+0000 7fbddd600640 1 -- 172.21.15.12:0/1351225582 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde0004320 con 0x563b9e967e80 2024-03-23T23:13:06.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.181+0000 7fbdf6445740 1 -- 172.21.15.12:0/1351225582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9e967e80 msgr2=0x563b9e972100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:06.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.181+0000 7fbdf6445740 1 --2- 172.21.15.12:0/1351225582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9e967e80 0x563b9e972100 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fbde0003ce0 tx=0x7fbde0005f30 comp rx=0 tx=0).stop 2024-03-23T23:13:06.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.181+0000 7fbdf6445740 1 -- 172.21.15.12:0/1351225582 shutdown_connections 2024-03-23T23:13:06.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.181+0000 7fbdf6445740 1 --2- 172.21.15.12:0/1351225582 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563b9e96d9e0 0x563b9e966a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.181+0000 7fbdf6445740 1 --2- 172.21.15.12:0/1351225582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b9e968870 0x563b9e979390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.181+0000 7fbdf6445740 1 --2- 172.21.15.12:0/1351225582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9e967e80 0x563b9e972100 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.181+0000 7fbdf6445740 1 -- 172.21.15.12:0/1351225582 >> 172.21.15.12:0/1351225582 conn(0x563b9e930200 msgr2=0x563b9e969b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:06.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.182+0000 7fbdf6445740 1 -- 172.21.15.12:0/1351225582 shutdown_connections 2024-03-23T23:13:06.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.182+0000 7fbdf6445740 1 -- 172.21.15.12:0/1351225582 wait complete. 2024-03-23T23:13:06.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.182+0000 7fbdf6445740 1 -- start start 2024-03-23T23:13:06.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.182+0000 7fbdf6445740 1 Processor -- start 2024-03-23T23:13:06.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.183+0000 7fbdf6445740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9e967e80 0x563b9e976be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:06.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.183+0000 7fbdf6445740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563b9e968870 0x563b9e977120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:06.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.183+0000 7fbdf6445740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b9e96d9e0 0x563b9e978130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:06.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.183+0000 7fbdf6445740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec1a9340 con 0x563b9e96d9e0 2024-03-23T23:13:06.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.183+0000 7fbdf6445740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec1aa640 con 0x563b9e967e80 2024-03-23T23:13:06.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.183+0000 7fbdf6445740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec1a9980 con 0x563b9e968870 2024-03-23T23:13:06.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.183+0000 7fbdf6445740 10 client.0 _ll_register_callbacks cb 0x7fbde00096e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:13:06.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.183+0000 7fbdf1800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b9e96d9e0 0x563b9e978130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:06.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.183+0000 7fbdf1800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b9e96d9e0 0x563b9e978130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43702/0 (socket says 172.21.15.12:43702) 2024-03-23T23:13:06.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.183+0000 7fbdf1800640 1 -- 172.21.15.12:0/1528255359 learned_addr learned my addr 172.21.15.12:0/1528255359 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:06.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.183+0000 7fbdf1800640 1 -- 172.21.15.12:0/1528255359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563b9e968870 msgr2=0x563b9e977120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:06.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.183+0000 7fbdf1800640 1 --2- 172.21.15.12:0/1528255359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563b9e968870 0x563b9e977120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.183+0000 7fbdf1800640 1 -- 172.21.15.12:0/1528255359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9e967e80 msgr2=0x563b9e976be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:06.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.183+0000 7fbdf1800640 1 --2- 172.21.15.12:0/1528255359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9e967e80 0x563b9e976be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.183+0000 7fbdf1800640 1 -- 172.21.15.12:0/1528255359 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563b9e978720 con 0x563b9e96d9e0 2024-03-23T23:13:06.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.184+0000 7fbdf1800640 1 --2- 172.21.15.12:0/1528255359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b9e96d9e0 0x563b9e978130 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x563b9e9690d0 tx=0x7fbde4007070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f6b4cc91d4fc10a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:06.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.184+0000 7fbdd2a00640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:13:06.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.184+0000 7fbdd2a00640 1 -- 172.21.15.12:0/1528255359 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde402c060 con 0x563b9e96d9e0 2024-03-23T23:13:06.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.184+0000 7fbdd2a00640 1 -- 172.21.15.12:0/1528255359 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fbde40137a0 con 0x563b9e96d9e0 2024-03-23T23:13:06.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.184+0000 7fbdf6445740 1 -- 172.21.15.12:0/1528255359 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbde0005500 con 0x563b9e96d9e0 2024-03-23T23:13:06.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.185+0000 7fbdd2a00640 1 -- 172.21.15.12:0/1528255359 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbde40139a0 con 0x563b9e96d9e0 2024-03-23T23:13:06.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.185+0000 7fbdf6445740 10 client.18393 fetch_fsmap learned FSMap version 23 2024-03-23T23:13:06.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.185+0000 7fbdf6445740 1 -- 172.21.15.12:0/1528255359 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563b9e9d97e0 con 0x563b9e96d9e0 2024-03-23T23:13:06.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.185+0000 7fbdd2a00640 1 -- 172.21.15.12:0/1528255359 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbde0005500 con 0x563b9e96d9e0 2024-03-23T23:13:06.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.185+0000 7fbdd2a00640 1 -- 172.21.15.12:0/1528255359 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fbde4027410 con 0x563b9e96d9e0 2024-03-23T23:13:06.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.185+0000 7fbdf6445740 10 client.18393 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:13:06.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.185+0000 7fbdf6445740 10 client.18393 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:13:06.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.185+0000 7fbdf6445740 20 client.18393 populate_metadata read hostname 'smithi012' 2024-03-23T23:13:06.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.185+0000 7fbdf6445740 1 --2- 172.21.15.12:0/1528255359 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563b9e9d9f90 0x563b9e9dc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:06.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.185+0000 7fbdf6445740 4 client.18393 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:13:06.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.185+0000 7fbdf6445740 1 -- 172.21.15.12:0/1528255359 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563b9e8d4ae0 con 0x563b9e9d9f90 2024-03-23T23:13:06.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.186+0000 7fbdf0e00640 1 --2- 172.21.15.12:0/1528255359 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563b9e9d9f90 0x563b9e9dc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:06.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.186+0000 7fbdf0e00640 1 --2- 172.21.15.12:0/1528255359 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563b9e9d9f90 0x563b9e9dc3b0 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88c0c2ae2b59a81d server_cookie=d1957deb94a9500b in_seq=0 out_seq=0 2024-03-23T23:13:06.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.186+0000 7fbdd2a00640 10 client.18393 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:06.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.187+0000 7fbdd2a00640 1 -- 172.21.15.12:0/1528255359 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x563b9e8d4ae0 con 0x563b9e9d9f90 2024-03-23T23:13:06.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.187+0000 7fbdd2a00640 10 client.18393 handle_command_reply: tid=0 2024-03-23T23:13:06.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.187+0000 7fbdf6445740 1 client.18393 shutdown 2024-03-23T23:13:06.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.187+0000 7fbdf6445740 1 -- 172.21.15.12:0/1528255359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b9e96d9e0 msgr2=0x563b9e978130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:06.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.187+0000 7fbdf6445740 1 --2- 172.21.15.12:0/1528255359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b9e96d9e0 0x563b9e978130 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x563b9e9690d0 tx=0x7fbde4007070 comp rx=0 tx=0).stop 2024-03-23T23:13:06.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.187+0000 7fbdf6445740 1 -- 172.21.15.12:0/1528255359 shutdown_connections 2024-03-23T23:13:06.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.187+0000 7fbdf6445740 1 --2- 172.21.15.12:0/1528255359 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x563b9e9d9f90 0x563b9e9dc3b0 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.187+0000 7fbdf6445740 1 --2- 172.21.15.12:0/1528255359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b9e96d9e0 0x563b9e978130 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.187+0000 7fbdf6445740 1 --2- 172.21.15.12:0/1528255359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563b9e968870 0x563b9e977120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.187+0000 7fbdf6445740 1 --2- 172.21.15.12:0/1528255359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9e967e80 0x563b9e976be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.187+0000 7fbdd2a00640 0 client.18393 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:06.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.187+0000 7fbdf6445740 1 -- 172.21.15.12:0/1528255359 >> 172.21.15.12:0/1528255359 conn(0x563b9e930200 msgr2=0x563b9e969b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:06.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.187+0000 7fbdf6445740 1 -- 172.21.15.12:0/1528255359 shutdown_connections 2024-03-23T23:13:06.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.187+0000 7fbdf6445740 1 -- 172.21.15.12:0/1528255359 wait complete. 2024-03-23T23:13:06.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.187+0000 7fbdf6445740 20 client.18393 trim_cache size 0 max 16384 2024-03-23T23:13:06.190 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:13:06.190 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T23:13:06.190 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-03-23T23:13:06.191 INFO:teuthology.orchestra.run.smithi012.stdout: "f5b3c141-92ce-44a7-9616-f5830479a338": { 2024-03-23T23:13:06.191 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-03-23T23:13:06.191 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "f5b3c141-92ce-44a7-9616-f5830479a338", 2024-03-23T23:13:06.191 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-03-23T23:13:06.191 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:06.191 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:06.191 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:13:06.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.188+0000 7fbdf3800640 1 -- 172.21.15.12:0/1156210454 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbdd40a2800 msgr2=0x7fbdd40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:06.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.188+0000 7fbdf3800640 1 --2- 172.21.15.12:0/1156210454 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbdd40a2800 0x7fbdd40a4cc0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x563b9e968c70 tx=0x7fbde00094c0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.188+0000 7fbdf3800640 1 -- 172.21.15.12:0/1156210454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdec0756e0 msgr2=0x7fbdec19a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:06.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.188+0000 7fbdf3800640 1 --2- 172.21.15.12:0/1156210454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdec0756e0 0x7fbdec19a170 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fbdd802c8f0 tx=0x7fbdd8005e40 comp rx=0 tx=0).stop 2024-03-23T23:13:06.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.188+0000 7fbdf3800640 1 -- 172.21.15.12:0/1156210454 shutdown_connections 2024-03-23T23:13:06.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.188+0000 7fbdf3800640 1 --2- 172.21.15.12:0/1156210454 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbdd40a2800 0x7fbdd40a4cc0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.188+0000 7fbdf3800640 1 --2- 172.21.15.12:0/1156210454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdec109570 0x7fbdec1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.188+0000 7fbdf3800640 1 --2- 172.21.15.12:0/1156210454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdec0756e0 0x7fbdec19a170 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.188+0000 7fbdf3800640 1 --2- 172.21.15.12:0/1156210454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbdec074100 0x7fbdec199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:06.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.188+0000 7fbdf3800640 1 -- 172.21.15.12:0/1156210454 >> 172.21.15.12:0/1156210454 conn(0x7fbdec0fcc40 msgr2=0x7fbdec105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:06.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.188+0000 7fbdf3800640 1 -- 172.21.15.12:0/1156210454 shutdown_connections 2024-03-23T23:13:06.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:06.188+0000 7fbdf3800640 1 -- 172.21.15.12:0/1156210454 wait complete. 2024-03-23T23:13:06.205 INFO:tasks.cephfs.filesystem:scrub status for tag:f5b3c141-92ce-44a7-9616-f5830479a338 - {'path': '/', 'tag': 'f5b3c141-92ce-44a7-9616-f5830479a338', 'options': 'recursive,force'} 2024-03-23T23:13:07.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:06 smithi012 ceph-mon[31695]: pgmap v729: 97 pgs: 97 active+clean; 3.7 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:13:07.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:06 smithi012 ceph-mon[31695]: scrub queued for path: / 2024-03-23T23:13:07.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:06 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:13:07.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:06 smithi012 ceph-mon[31695]: scrub summary: active paths [/] 2024-03-23T23:13:07.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:06 smithi012 ceph-mon[31695]: scrub summary: idle+waiting paths [/] 2024-03-23T23:13:07.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:06 smithi173 ceph-mon[36116]: pgmap v729: 97 pgs: 97 active+clean; 3.7 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:13:07.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:06 smithi173 ceph-mon[36116]: scrub queued for path: / 2024-03-23T23:13:07.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:06 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:13:07.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:06 smithi173 ceph-mon[36116]: scrub summary: active paths [/] 2024-03-23T23:13:07.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:06 smithi173 ceph-mon[36116]: scrub summary: idle+waiting paths [/] 2024-03-23T23:13:07.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:06 smithi203 ceph-mon[36693]: pgmap v729: 97 pgs: 97 active+clean; 3.7 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-03-23T23:13:07.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:06 smithi203 ceph-mon[36693]: scrub queued for path: / 2024-03-23T23:13:07.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:06 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:13:07.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:06 smithi203 ceph-mon[36693]: scrub summary: active paths [/] 2024-03-23T23:13:07.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:06 smithi203 ceph-mon[36693]: scrub summary: idle+waiting paths [/] 2024-03-23T23:13:09.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:08 smithi012 ceph-mon[31695]: pgmap v730: 97 pgs: 97 active+clean; 3.7 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:13:09.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:08 smithi173 ceph-mon[36116]: pgmap v730: 97 pgs: 97 active+clean; 3.7 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:13:09.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:08 smithi203 ceph-mon[36693]: pgmap v730: 97 pgs: 97 active+clean; 3.7 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:13:11.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:10 smithi012 ceph-mon[31695]: pgmap v731: 97 pgs: 97 active+clean; 3.7 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:13:11.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:10 smithi173 ceph-mon[36116]: pgmap v731: 97 pgs: 97 active+clean; 3.7 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:13:11.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:10 smithi203 ceph-mon[36693]: pgmap v731: 97 pgs: 97 active+clean; 3.7 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:13:11.522 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.522+0000 7f0c5a600640 1 -- 172.21.15.12:0/1990238975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c54103900 msgr2=0x7f0c54103d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:11.522 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.522+0000 7f0c5a600640 1 --2- 172.21.15.12:0/1990238975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c54103900 0x7f0c54103d20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0c44009ca0 tx=0x7f0c4402c0f0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.522 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.523+0000 7f0c5a600640 1 -- 172.21.15.12:0/1990238975 shutdown_connections 2024-03-23T23:13:11.522 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.523+0000 7f0c5a600640 1 --2- 172.21.15.12:0/1990238975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0c541054b0 0x7f0c5410bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.522 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.523+0000 7f0c5a600640 1 --2- 172.21.15.12:0/1990238975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c54104af0 0x7f0c54104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.522 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.523+0000 7f0c5a600640 1 --2- 172.21.15.12:0/1990238975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c54103900 0x7f0c54103d20 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.523 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.523+0000 7f0c5a600640 1 -- 172.21.15.12:0/1990238975 >> 172.21.15.12:0/1990238975 conn(0x7f0c540fee90 msgr2=0x7f0c541012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:11.523 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.523+0000 7f0c5a600640 1 -- 172.21.15.12:0/1990238975 shutdown_connections 2024-03-23T23:13:11.523 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.523+0000 7f0c5a600640 1 -- 172.21.15.12:0/1990238975 wait complete. 2024-03-23T23:13:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.524+0000 7f0c5a600640 1 Processor -- start 2024-03-23T23:13:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.524+0000 7f0c5a600640 1 -- start start 2024-03-23T23:13:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.524+0000 7f0c5a600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0c54103900 0x7f0c54199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.524+0000 7f0c5a600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c54104af0 0x7f0c5419a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.524+0000 7f0c5a600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c541054b0 0x7f0c541a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.524+0000 7f0c5a600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5410f650 con 0x7f0c54104af0 2024-03-23T23:13:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.524+0000 7f0c5a600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5410f4d0 con 0x7f0c54103900 2024-03-23T23:13:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.524+0000 7f0c5a600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5410f7d0 con 0x7f0c541054b0 2024-03-23T23:13:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.525+0000 7f0c52a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c54104af0 0x7f0c5419a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.525+0000 7f0c53400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0c54103900 0x7f0c54199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.525+0000 7f0c52a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c54104af0 0x7f0c5419a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43712/0 (socket says 172.21.15.12:43712) 2024-03-23T23:13:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.525+0000 7f0c52a00640 1 -- 172.21.15.12:0/1370801391 learned_addr learned my addr 172.21.15.12:0/1370801391 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.525+0000 7f0c53e00640 1 --2- 172.21.15.12:0/1370801391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c541054b0 0x7f0c541a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.525+0000 7f0c52a00640 1 -- 172.21.15.12:0/1370801391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c541054b0 msgr2=0x7f0c541a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.525+0000 7f0c52a00640 1 --2- 172.21.15.12:0/1370801391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c541054b0 0x7f0c541a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.525+0000 7f0c52a00640 1 -- 172.21.15.12:0/1370801391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0c54103900 msgr2=0x7f0c54199c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.525+0000 7f0c52a00640 1 --2- 172.21.15.12:0/1370801391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0c54103900 0x7f0c54199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.525+0000 7f0c52a00640 1 -- 172.21.15.12:0/1370801391 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c541a1870 con 0x7f0c54104af0 2024-03-23T23:13:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.525+0000 7f0c52a00640 1 --2- 172.21.15.12:0/1370801391 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c54104af0 0x7f0c5419a1a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c00b4a0 tx=0x7f0c3c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.526+0000 7f0c4be00640 1 -- 172.21.15.12:0/1370801391 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c014070 con 0x7f0c54104af0 2024-03-23T23:13:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.526+0000 7f0c4be00640 1 -- 172.21.15.12:0/1370801391 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c3c010af0 con 0x7f0c54104af0 2024-03-23T23:13:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.526+0000 7f0c5a600640 1 -- 172.21.15.12:0/1370801391 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c541a1b60 con 0x7f0c54104af0 2024-03-23T23:13:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.526+0000 7f0c4be00640 1 -- 172.21.15.12:0/1370801391 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c017d10 con 0x7f0c54104af0 2024-03-23T23:13:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.526+0000 7f0c5a600640 1 -- 172.21.15.12:0/1370801391 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c541a2060 con 0x7f0c54104af0 2024-03-23T23:13:11.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.530+0000 7f0c5d39f740 1 Processor -- start 2024-03-23T23:13:11.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.530+0000 7f0c5d39f740 1 -- start start 2024-03-23T23:13:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.531+0000 7f0c5d39f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f2464771e0 0x55f246477600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.531+0000 7f0c5d39f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f246477bd0 0x55f24647c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.531+0000 7f0c4be00640 1 -- 172.21.15.12:0/1370801391 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0c3c017070 con 0x7f0c54104af0 2024-03-23T23:13:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.531+0000 7f0c5d39f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f24647cd40 0x55f24647f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.531+0000 7f0c5d39f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f24643a3b0 con 0x55f2464771e0 2024-03-23T23:13:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.531+0000 7f0c5d39f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2463dc830 con 0x55f24647cd40 2024-03-23T23:13:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.531+0000 7f0c5d39f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2463855d0 con 0x55f246477bd0 2024-03-23T23:13:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.531+0000 7f0c53e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f2464771e0 0x55f246477600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.531+0000 7f0c52a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f24647cd40 0x55f24647f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.531+0000 7f0c53e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f2464771e0 0x55f246477600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43718/0 (socket says 172.21.15.12:43718) 2024-03-23T23:13:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.531+0000 7f0c53e00640 1 -- 172.21.15.12:0/3136893058 learned_addr learned my addr 172.21.15.12:0/3136893058 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.531+0000 7f0c52a00640 1 --2- 172.21.15.12:0/3136893058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f24647cd40 0x55f24647f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:58278/0 (socket says 172.21.15.12:58278) 2024-03-23T23:13:11.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.532+0000 7f0c52a00640 1 -- 172.21.15.12:0/3136893058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f246477bd0 msgr2=0x55f24647c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:11.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.532+0000 7f0c52a00640 1 --2- 172.21.15.12:0/3136893058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f246477bd0 0x55f24647c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.532+0000 7f0c52a00640 1 -- 172.21.15.12:0/3136893058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f2464771e0 msgr2=0x55f246477600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.532+0000 7f0c52a00640 1 --2- 172.21.15.12:0/3136893058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f2464771e0 0x55f246477600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.532+0000 7f0c52a00640 1 -- 172.21.15.12:0/3136893058 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f24639d2c0 con 0x55f24647cd40 2024-03-23T23:13:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.532+0000 7f0c53e00640 1 --2- 172.21.15.12:0/3136893058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f2464771e0 0x55f246477600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.532+0000 7f0c52a00640 1 --2- 172.21.15.12:0/3136893058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f24647cd40 0x55f24647f1d0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c0cb870 tx=0x7f0c3c0d47f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a3aa493572bcc2a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.532+0000 7f0c48c00640 1 -- 172.21.15.12:0/3136893058 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c0df430 con 0x55f24647cd40 2024-03-23T23:13:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.532+0000 7f0c48c00640 1 -- 172.21.15.12:0/3136893058 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c3c0d28e0 con 0x55f24647cd40 2024-03-23T23:13:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.532+0000 7f0c48c00640 1 -- 172.21.15.12:0/3136893058 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c0de7f0 con 0x55f24647cd40 2024-03-23T23:13:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.532+0000 7f0c4be00640 1 --2- 172.21.15.12:0/1370801391 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0c380a28a0 0x7f0c380a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.533+0000 7f0c4be00640 1 -- 172.21.15.12:0/1370801391 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0c3c0cde20 con 0x7f0c54104af0 2024-03-23T23:13:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.533+0000 7f0c53e00640 1 --2- 172.21.15.12:0/1370801391 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0c380a28a0 0x7f0c380a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.533+0000 7f0c5d39f740 1 -- 172.21.15.12:0/3136893058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f24647cd40 msgr2=0x55f24647f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.533+0000 7f0c5d39f740 1 --2- 172.21.15.12:0/3136893058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f24647cd40 0x55f24647f1d0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c0cb870 tx=0x7f0c3c0d47f0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.533+0000 7f0c5d39f740 1 -- 172.21.15.12:0/3136893058 shutdown_connections 2024-03-23T23:13:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.533+0000 7f0c5d39f740 1 --2- 172.21.15.12:0/3136893058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f24647cd40 0x55f24647f1d0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.533+0000 7f0c5d39f740 1 --2- 172.21.15.12:0/3136893058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f246477bd0 0x55f24647c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.533+0000 7f0c5d39f740 1 --2- 172.21.15.12:0/3136893058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f2464771e0 0x55f246477600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.533+0000 7f0c5d39f740 1 -- 172.21.15.12:0/3136893058 >> 172.21.15.12:0/3136893058 conn(0x55f246472750 msgr2=0x55f246472b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.533+0000 7f0c5d39f740 1 -- 172.21.15.12:0/3136893058 shutdown_connections 2024-03-23T23:13:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.533+0000 7f0c5d39f740 1 -- 172.21.15.12:0/3136893058 wait complete. 2024-03-23T23:13:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.533+0000 7f0c53e00640 1 --2- 172.21.15.12:0/1370801391 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0c380a28a0 0x7f0c380a4d60 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x55f246478430 tx=0x7f0c4c0090e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.534+0000 7f0c5d39f740 1 -- start start 2024-03-23T23:13:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.534+0000 7f0c5d39f740 1 Processor -- start 2024-03-23T23:13:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.534+0000 7f0c5d39f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f2464771e0 0x55f246481390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.534+0000 7f0c5d39f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f246477bd0 0x55f2464818d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.534+0000 7f0c5d39f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f24647cd40 0x55f246488910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.534+0000 7f0c5d39f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f246398a20 con 0x55f24647cd40 2024-03-23T23:13:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.534+0000 7f0c5d39f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2463d88d0 con 0x55f246477bd0 2024-03-23T23:13:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.534+0000 7f0c5d39f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f24647fcf0 con 0x55f2464771e0 2024-03-23T23:13:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.535+0000 7f0c5d39f740 10 client.0 _ll_register_callbacks cb 0x55f2463d51c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:13:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.535+0000 7f0c53e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f24647cd40 0x55f246488910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:11.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.535+0000 7f0c53e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f24647cd40 0x55f246488910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43724/0 (socket says 172.21.15.12:43724) 2024-03-23T23:13:11.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.535+0000 7f0c53e00640 1 -- 172.21.15.12:0/1057025380 learned_addr learned my addr 172.21.15.12:0/1057025380 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:11.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.535+0000 7f0c53e00640 1 -- 172.21.15.12:0/1057025380 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f2464771e0 msgr2=0x55f246481390 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-23T23:13:11.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.535+0000 7f0c53e00640 1 --2- 172.21.15.12:0/1057025380 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f2464771e0 0x55f246481390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.535+0000 7f0c53e00640 1 -- 172.21.15.12:0/1057025380 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f246477bd0 msgr2=0x55f2464818d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-23T23:13:11.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.535+0000 7f0c53e00640 1 --2- 172.21.15.12:0/1057025380 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f246477bd0 0x55f2464818d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.535+0000 7f0c53e00640 1 -- 172.21.15.12:0/1057025380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f2460f1210 con 0x55f24647cd40 2024-03-23T23:13:11.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.535+0000 7f0c53e00640 1 --2- 172.21.15.12:0/1057025380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f24647cd40 0x55f246488910 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c0149c0 tx=0x7f0c4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54579b7135e1af5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:11.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.535+0000 7f0c41600640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:13:11.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.536+0000 7f0c41600640 1 -- 172.21.15.12:0/1057025380 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c025070 con 0x55f24647cd40 2024-03-23T23:13:11.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.536+0000 7f0c41600640 1 -- 172.21.15.12:0/1057025380 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0c4c021600 con 0x55f24647cd40 2024-03-23T23:13:11.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.536+0000 7f0c5d39f740 1 -- 172.21.15.12:0/1057025380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0c3c0d28e0 con 0x55f24647cd40 2024-03-23T23:13:11.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.536+0000 7f0c41600640 1 -- 172.21.15.12:0/1057025380 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c4c021840 con 0x55f24647cd40 2024-03-23T23:13:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.536+0000 7f0c5d39f740 10 client.18405 fetch_fsmap learned FSMap version 23 2024-03-23T23:13:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.536+0000 7f0c5d39f740 1 -- 172.21.15.12:0/1057025380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f2461bd620 con 0x55f24647cd40 2024-03-23T23:13:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.536+0000 7f0c41600640 1 -- 172.21.15.12:0/1057025380 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c3c0d28e0 con 0x55f24647cd40 2024-03-23T23:13:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.537+0000 7f0c41600640 1 -- 172.21.15.12:0/1057025380 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f0c4c020070 con 0x55f24647cd40 2024-03-23T23:13:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.537+0000 7f0c5d39f740 10 client.18405 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:13:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.537+0000 7f0c5d39f740 10 client.18405 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:13:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.537+0000 7f0c5d39f740 20 client.18405 populate_metadata read hostname 'smithi012' 2024-03-23T23:13:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.537+0000 7f0c5d39f740 1 --2- 172.21.15.12:0/1057025380 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f24648d280 0x55f24648f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.537+0000 7f0c5d39f740 4 client.18405 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:13:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.537+0000 7f0c5d39f740 1 -- 172.21.15.12:0/1057025380 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f2463e9d70 con 0x55f24648d280 2024-03-23T23:13:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.537+0000 7f0c53400640 1 --2- 172.21.15.12:0/1057025380 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f24648d280 0x55f24648f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.539+0000 7f0c53400640 1 --2- 172.21.15.12:0/1057025380 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f24648d280 0x55f24648f6d0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d55c74fbfc99acf server_cookie=6d9db40ce2ba6b71 in_seq=0 out_seq=0 2024-03-23T23:13:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.539+0000 7f0c41600640 10 client.18405 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.541+0000 7f0c41600640 1 -- 172.21.15.12:0/1057025380 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f2463e9d70 con 0x55f24648d280 2024-03-23T23:13:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.541+0000 7f0c41600640 10 client.18405 handle_command_reply: tid=0 2024-03-23T23:13:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.541+0000 7f0c5d39f740 1 client.18405 shutdown 2024-03-23T23:13:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.541+0000 7f0c5d39f740 1 -- 172.21.15.12:0/1057025380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f24647cd40 msgr2=0x55f246488910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.541+0000 7f0c5d39f740 1 --2- 172.21.15.12:0/1057025380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f24647cd40 0x55f246488910 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c0149c0 tx=0x7f0c4c0046c0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.541+0000 7f0c5d39f740 1 -- 172.21.15.12:0/1057025380 shutdown_connections 2024-03-23T23:13:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.541+0000 7f0c5d39f740 1 --2- 172.21.15.12:0/1057025380 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f24648d280 0x55f24648f6d0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.541+0000 7f0c5d39f740 1 --2- 172.21.15.12:0/1057025380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f24647cd40 0x55f246488910 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.541+0000 7f0c5d39f740 1 --2- 172.21.15.12:0/1057025380 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f246477bd0 0x55f2464818d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.541+0000 7f0c5d39f740 1 --2- 172.21.15.12:0/1057025380 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f2464771e0 0x55f246481390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.541+0000 7f0c41600640 0 client.18405 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.541+0000 7f0c5d39f740 1 -- 172.21.15.12:0/1057025380 >> 172.21.15.12:0/1057025380 conn(0x55f246472750 msgr2=0x55f246471950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.541+0000 7f0c5d39f740 1 -- 172.21.15.12:0/1057025380 shutdown_connections 2024-03-23T23:13:11.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.541+0000 7f0c5d39f740 1 -- 172.21.15.12:0/1057025380 wait complete. 2024-03-23T23:13:11.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.542+0000 7f0c5d39f740 20 client.18405 trim_cache size 0 max 16384 2024-03-23T23:13:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.557+0000 7f0c5d39f740 1 Processor -- start 2024-03-23T23:13:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.557+0000 7f0c5d39f740 1 -- start start 2024-03-23T23:13:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.557+0000 7f0c5d39f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f2464771e0 0x55f246478ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.557+0000 7f0c5d39f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f246477bd0 0x55f246479430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.557+0000 7f0c5d39f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f24647cd40 0x55f2464888d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.557+0000 7f0c5d39f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c541ad2b0 con 0x55f246477bd0 2024-03-23T23:13:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.557+0000 7f0c5d39f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c541aadb0 con 0x55f24647cd40 2024-03-23T23:13:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.557+0000 7f0c5d39f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c541a8e50 con 0x55f2464771e0 2024-03-23T23:13:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.557+0000 7f0c52a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f246477bd0 0x55f246479430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.557+0000 7f0c53400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f2464771e0 0x55f246478ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.557+0000 7f0c53400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f2464771e0 0x55f246478ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49534/0 (socket says 172.21.15.12:49534) 2024-03-23T23:13:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.557+0000 7f0c53400640 1 -- 172.21.15.12:0/2244551680 learned_addr learned my addr 172.21.15.12:0/2244551680 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.557+0000 7f0c52a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f246477bd0 0x55f246479430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43728/0 (socket says 172.21.15.12:43728) 2024-03-23T23:13:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.557+0000 7f0c53e00640 1 --2- 172.21.15.12:0/2244551680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f24647cd40 0x55f2464888d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.558+0000 7f0c53400640 1 -- 172.21.15.12:0/2244551680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f24647cd40 msgr2=0x55f2464888d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.558+0000 7f0c53400640 1 --2- 172.21.15.12:0/2244551680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f24647cd40 0x55f2464888d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.558+0000 7f0c53400640 1 -- 172.21.15.12:0/2244551680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f246477bd0 msgr2=0x55f246479430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.558+0000 7f0c53400640 1 --2- 172.21.15.12:0/2244551680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f246477bd0 0x55f246479430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.558+0000 7f0c53400640 1 -- 172.21.15.12:0/2244551680 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f246477fb0 con 0x55f2464771e0 2024-03-23T23:13:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.558+0000 7f0c52a00640 1 --2- 172.21.15.12:0/2244551680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f246477bd0 0x55f246479430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.558+0000 7f0c53400640 1 --2- 172.21.15.12:0/2244551680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f2464771e0 0x55f246478ef0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f0c44009ca0 tx=0x7f0c44000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=258950e89e190682 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.558+0000 7f0c43e00640 1 -- 172.21.15.12:0/2244551680 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c44039040 con 0x55f2464771e0 2024-03-23T23:13:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.558+0000 7f0c43e00640 1 -- 172.21.15.12:0/2244551680 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c44049070 con 0x55f2464771e0 2024-03-23T23:13:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.558+0000 7f0c43e00640 1 -- 172.21.15.12:0/2244551680 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c44043820 con 0x55f2464771e0 2024-03-23T23:13:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.559+0000 7f0c5d39f740 1 -- 172.21.15.12:0/2244551680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f2464771e0 msgr2=0x55f246478ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.559+0000 7f0c5d39f740 1 --2- 172.21.15.12:0/2244551680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f2464771e0 0x55f246478ef0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f0c44009ca0 tx=0x7f0c44000f40 comp rx=0 tx=0).stop 2024-03-23T23:13:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.559+0000 7f0c5d39f740 1 -- 172.21.15.12:0/2244551680 shutdown_connections 2024-03-23T23:13:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.559+0000 7f0c5d39f740 1 --2- 172.21.15.12:0/2244551680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f24647cd40 0x55f2464888d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.559+0000 7f0c5d39f740 1 --2- 172.21.15.12:0/2244551680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f246477bd0 0x55f246479430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.559+0000 7f0c5d39f740 1 --2- 172.21.15.12:0/2244551680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f2464771e0 0x55f246478ef0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.559+0000 7f0c5d39f740 1 -- 172.21.15.12:0/2244551680 >> 172.21.15.12:0/2244551680 conn(0x55f246472750 msgr2=0x55f246481390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.559+0000 7f0c5d39f740 1 -- 172.21.15.12:0/2244551680 shutdown_connections 2024-03-23T23:13:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.559+0000 7f0c5d39f740 1 -- 172.21.15.12:0/2244551680 wait complete. 2024-03-23T23:13:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.560+0000 7f0c5d39f740 1 -- start start 2024-03-23T23:13:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.560+0000 7f0c5d39f740 1 Processor -- start 2024-03-23T23:13:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.560+0000 7f0c5d39f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f2464771e0 0x55f2464862b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.560+0000 7f0c5d39f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f246477bd0 0x55f2464867f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.560+0000 7f0c5d39f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f24647cd40 0x55f2464876d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.560+0000 7f0c5d39f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c541a9450 con 0x55f2464771e0 2024-03-23T23:13:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.560+0000 7f0c5d39f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c541aa750 con 0x55f24647cd40 2024-03-23T23:13:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.560+0000 7f0c5d39f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c541a9a90 con 0x55f246477bd0 2024-03-23T23:13:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.560+0000 7f0c5d39f740 10 client.0 _ll_register_callbacks cb 0x7f0c4c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:13:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.561+0000 7f0c53400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f2464771e0 0x55f2464862b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.561+0000 7f0c52a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f246477bd0 0x55f2464867f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.561+0000 7f0c52a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f246477bd0 0x55f2464867f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:49540/0 (socket says 172.21.15.12:49540) 2024-03-23T23:13:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.561+0000 7f0c52a00640 1 -- 172.21.15.12:0/1979987804 learned_addr learned my addr 172.21.15.12:0/1979987804 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.561+0000 7f0c53400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f2464771e0 0x55f2464862b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43734/0 (socket says 172.21.15.12:43734) 2024-03-23T23:13:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.561+0000 7f0c52a00640 1 -- 172.21.15.12:0/1979987804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f24647cd40 msgr2=0x55f2464876d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.561+0000 7f0c52a00640 1 --2- 172.21.15.12:0/1979987804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f24647cd40 0x55f2464876d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.561+0000 7f0c52a00640 1 -- 172.21.15.12:0/1979987804 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f2464771e0 msgr2=0x55f2464862b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.561+0000 7f0c52a00640 1 --2- 172.21.15.12:0/1979987804 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f2464771e0 0x55f2464862b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.561+0000 7f0c52a00640 1 -- 172.21.15.12:0/1979987804 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f246487cc0 con 0x55f246477bd0 2024-03-23T23:13:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.561+0000 7f0c52a00640 1 --2- 172.21.15.12:0/1979987804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f246477bd0 0x55f2464867f0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c0d1e70 tx=0x7f0c3c0ce0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c219ea7b5e9ee33 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.561+0000 7f0c41600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:13:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.562+0000 7f0c41600640 1 -- 172.21.15.12:0/1979987804 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c01e070 con 0x55f246477bd0 2024-03-23T23:13:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.562+0000 7f0c41600640 1 -- 172.21.15.12:0/1979987804 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0c3c0850b0 con 0x55f246477bd0 2024-03-23T23:13:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.562+0000 7f0c5d39f740 1 -- 172.21.15.12:0/1979987804 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0c44049070 con 0x55f246477bd0 2024-03-23T23:13:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.562+0000 7f0c41600640 1 -- 172.21.15.12:0/1979987804 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c3c09a5f0 con 0x55f246477bd0 2024-03-23T23:13:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.562+0000 7f0c5d39f740 10 client.28510 fetch_fsmap learned FSMap version 23 2024-03-23T23:13:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.562+0000 7f0c5d39f740 1 -- 172.21.15.12:0/1979987804 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f2464e8da0 con 0x55f246477bd0 2024-03-23T23:13:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.562+0000 7f0c41600640 1 -- 172.21.15.12:0/1979987804 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c44049070 con 0x55f246477bd0 2024-03-23T23:13:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.562+0000 7f0c41600640 1 -- 172.21.15.12:0/1979987804 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f0c3c0cb950 con 0x55f246477bd0 2024-03-23T23:13:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.562+0000 7f0c5d39f740 10 client.28510 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:13:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.562+0000 7f0c5d39f740 10 client.28510 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:13:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.562+0000 7f0c5d39f740 20 client.28510 populate_metadata read hostname 'smithi012' 2024-03-23T23:13:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.562+0000 7f0c5d39f740 1 --2- 172.21.15.12:0/1979987804 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f2464e91d0 0x55f2464eb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.562+0000 7f0c5d39f740 4 client.28510 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:13:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.562+0000 7f0c5d39f740 1 -- 172.21.15.12:0/1979987804 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f2463e40c0 con 0x55f2464e91d0 2024-03-23T23:13:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.563+0000 7f0c53400640 1 --2- 172.21.15.12:0/1979987804 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f2464e91d0 0x55f2464eb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.563+0000 7f0c53400640 1 --2- 172.21.15.12:0/1979987804 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f2464e91d0 0x55f2464eb5e0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79df19b3e4a64f49 server_cookie=b52a5f880d24fc55 in_seq=0 out_seq=0 2024-03-23T23:13:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.563+0000 7f0c41600640 10 client.28510 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:11.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.576+0000 7f0c41600640 1 -- 172.21.15.12:0/1979987804 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f2463e40c0 con 0x55f2464e91d0 2024-03-23T23:13:11.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.576+0000 7f0c41600640 10 client.28510 handle_command_reply: tid=0 2024-03-23T23:13:11.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.576+0000 7f0c5d39f740 1 client.28510 shutdown 2024-03-23T23:13:11.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.576+0000 7f0c5d39f740 1 -- 172.21.15.12:0/1979987804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f246477bd0 msgr2=0x55f2464867f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:11.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.576+0000 7f0c5d39f740 1 --2- 172.21.15.12:0/1979987804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f246477bd0 0x55f2464867f0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c0d1e70 tx=0x7f0c3c0ce0a0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.576+0000 7f0c5d39f740 1 -- 172.21.15.12:0/1979987804 shutdown_connections 2024-03-23T23:13:11.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.576+0000 7f0c5d39f740 1 --2- 172.21.15.12:0/1979987804 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55f2464e91d0 0x55f2464eb5e0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.576+0000 7f0c5d39f740 1 --2- 172.21.15.12:0/1979987804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f24647cd40 0x55f2464876d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.576+0000 7f0c5d39f740 1 --2- 172.21.15.12:0/1979987804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f246477bd0 0x55f2464867f0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.576+0000 7f0c5d39f740 1 --2- 172.21.15.12:0/1979987804 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f2464771e0 0x55f2464862b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.576+0000 7f0c5d39f740 1 -- 172.21.15.12:0/1979987804 >> 172.21.15.12:0/1979987804 conn(0x55f246472750 msgr2=0x55f246481390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:11.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.576+0000 7f0c41600640 0 client.28510 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:11.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.576+0000 7f0c5d39f740 1 -- 172.21.15.12:0/1979987804 shutdown_connections 2024-03-23T23:13:11.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.576+0000 7f0c5d39f740 1 -- 172.21.15.12:0/1979987804 wait complete. 2024-03-23T23:13:11.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.576+0000 7f0c5d39f740 20 client.28510 trim_cache size 0 max 16384 2024-03-23T23:13:11.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:13:11.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:13:11.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:13:11.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:13:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.577+0000 7f0c5a600640 1 -- 172.21.15.12:0/1370801391 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0c380a28a0 msgr2=0x7f0c380a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.577+0000 7f0c5a600640 1 --2- 172.21.15.12:0/1370801391 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0c380a28a0 0x7f0c380a4d60 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x55f246478430 tx=0x7f0c4c0090e0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.577+0000 7f0c5a600640 1 -- 172.21.15.12:0/1370801391 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c54104af0 msgr2=0x7f0c5419a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.577+0000 7f0c5a600640 1 --2- 172.21.15.12:0/1370801391 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c54104af0 0x7f0c5419a1a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c00b4a0 tx=0x7f0c3c00b960 comp rx=0 tx=0).stop 2024-03-23T23:13:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.577+0000 7f0c5a600640 1 -- 172.21.15.12:0/1370801391 shutdown_connections 2024-03-23T23:13:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.577+0000 7f0c5a600640 1 --2- 172.21.15.12:0/1370801391 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f0c380a28a0 0x7f0c380a4d60 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.577+0000 7f0c5a600640 1 --2- 172.21.15.12:0/1370801391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c541054b0 0x7f0c541a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.577+0000 7f0c5a600640 1 --2- 172.21.15.12:0/1370801391 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c54104af0 0x7f0c5419a1a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.577+0000 7f0c5a600640 1 --2- 172.21.15.12:0/1370801391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0c54103900 0x7f0c54199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.577+0000 7f0c5a600640 1 -- 172.21.15.12:0/1370801391 >> 172.21.15.12:0/1370801391 conn(0x7f0c540fee90 msgr2=0x7f0c54107d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.578+0000 7f0c5a600640 1 -- 172.21.15.12:0/1370801391 shutdown_connections 2024-03-23T23:13:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:11.578+0000 7f0c5a600640 1 -- 172.21.15.12:0/1370801391 wait complete. 2024-03-23T23:13:12.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:11 smithi012 ceph-mon[31695]: scrub summary: idle 2024-03-23T23:13:12.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:11 smithi173 ceph-mon[36116]: scrub summary: idle 2024-03-23T23:13:12.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:11 smithi203 ceph-mon[36693]: scrub summary: idle 2024-03-23T23:13:13.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:12 smithi012 ceph-mon[31695]: pgmap v732: 97 pgs: 97 active+clean; 3.8 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:13:13.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:12 smithi173 ceph-mon[36116]: pgmap v732: 97 pgs: 97 active+clean; 3.8 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:13:13.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:12 smithi203 ceph-mon[36693]: pgmap v732: 97 pgs: 97 active+clean; 3.8 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-03-23T23:13:15.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:14 smithi012 ceph-mon[31695]: pgmap v733: 97 pgs: 97 active+clean; 3.8 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-03-23T23:13:15.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:14 smithi173 ceph-mon[36116]: pgmap v733: 97 pgs: 97 active+clean; 3.8 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-03-23T23:13:15.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:14 smithi203 ceph-mon[36693]: pgmap v733: 97 pgs: 97 active+clean; 3.8 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-03-23T23:13:17.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:16 smithi012 ceph-mon[31695]: pgmap v734: 97 pgs: 97 active+clean; 3.8 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:13:17.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:16 smithi173 ceph-mon[36116]: pgmap v734: 97 pgs: 97 active+clean; 3.8 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:13:17.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:16 smithi203 ceph-mon[36693]: pgmap v734: 97 pgs: 97 active+clean; 3.8 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:13:19.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:18 smithi012 ceph-mon[31695]: pgmap v735: 97 pgs: 97 active+clean; 3.8 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:13:19.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:18 smithi173 ceph-mon[36116]: pgmap v735: 97 pgs: 97 active+clean; 3.8 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:13:19.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:18 smithi203 ceph-mon[36693]: pgmap v735: 97 pgs: 97 active+clean; 3.8 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:13:20.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:19 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:13:20.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:19 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:13:20.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:19 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:13:20.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:19 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:13:20.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:19 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:13:20.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:19 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:13:20.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:13:20.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:13:20.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:19 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:13:21.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:20 smithi012 ceph-mon[31695]: pgmap v736: 97 pgs: 97 active+clean; 3.8 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:13:21.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:20 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:13:21.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:20 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:13:21.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:20 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:13:21.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:20 smithi173 ceph-mon[36116]: pgmap v736: 97 pgs: 97 active+clean; 3.8 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:13:21.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:20 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:13:21.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:20 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:13:21.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:20 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:13:21.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:20 smithi203 ceph-mon[36693]: pgmap v736: 97 pgs: 97 active+clean; 3.8 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-03-23T23:13:21.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:20 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:13:21.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:20 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:13:21.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:20 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:13:23.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:22 smithi012 ceph-mon[31695]: pgmap v737: 97 pgs: 97 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:13:23.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:22 smithi173 ceph-mon[36116]: pgmap v737: 97 pgs: 97 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:13:23.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:22 smithi203 ceph-mon[36693]: pgmap v737: 97 pgs: 97 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:13:24.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.727+0000 7efcae000640 1 -- 172.21.15.12:0/1077097356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efca8103280 msgr2=0x7efca8109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:24.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.727+0000 7efcae000640 1 --2- 172.21.15.12:0/1077097356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efca8103280 0x7efca8109b10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7efca0009be0 tx=0x7efca002bef0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.727+0000 7efcae000640 1 -- 172.21.15.12:0/1077097356 shutdown_connections 2024-03-23T23:13:24.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.727+0000 7efcae000640 1 --2- 172.21.15.12:0/1077097356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efca8103280 0x7efca8109b10 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.727+0000 7efcae000640 1 --2- 172.21.15.12:0/1077097356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efca81028c0 0x7efca8102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.727+0000 7efcae000640 1 --2- 172.21.15.12:0/1077097356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efca81016d0 0x7efca8101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.727+0000 7efcae000640 1 -- 172.21.15.12:0/1077097356 >> 172.21.15.12:0/1077097356 conn(0x7efca80fcc60 msgr2=0x7efca80ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.727+0000 7efcae000640 1 -- 172.21.15.12:0/1077097356 shutdown_connections 2024-03-23T23:13:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.727+0000 7efcae000640 1 -- 172.21.15.12:0/1077097356 wait complete. 2024-03-23T23:13:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.728+0000 7efcae000640 1 Processor -- start 2024-03-23T23:13:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.728+0000 7efcae000640 1 -- start start 2024-03-23T23:13:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.728+0000 7efcae000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efca81016d0 0x7efca8075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.728+0000 7efcae000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efca81028c0 0x7efca8073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.728+0000 7efcae000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efca8103280 0x7efca8073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.728+0000 7efcae000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca810d380 con 0x7efca81016d0 2024-03-23T23:13:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.728+0000 7efcae000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca810d200 con 0x7efca81028c0 2024-03-23T23:13:24.734 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.728+0000 7efcae000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca810d500 con 0x7efca8103280 2024-03-23T23:13:24.734 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.729+0000 7efca7400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efca81016d0 0x7efca8075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:24.734 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.729+0000 7efca7e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efca8103280 0x7efca8073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:24.734 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.729+0000 7efca7400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efca81016d0 0x7efca8075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42166/0 (socket says 172.21.15.12:42166) 2024-03-23T23:13:24.734 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.729+0000 7efca7400640 1 -- 172.21.15.12:0/343050171 learned_addr learned my addr 172.21.15.12:0/343050171 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:24.734 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.729+0000 7efca7e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efca8103280 0x7efca8073fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:57662/0 (socket says 172.21.15.12:57662) 2024-03-23T23:13:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.729+0000 7efca6a00640 1 --2- 172.21.15.12:0/343050171 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efca81028c0 0x7efca8073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.729+0000 7efca7400640 1 -- 172.21.15.12:0/343050171 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efca8103280 msgr2=0x7efca8073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.729+0000 7efca7400640 1 --2- 172.21.15.12:0/343050171 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efca8103280 0x7efca8073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.729+0000 7efca7400640 1 -- 172.21.15.12:0/343050171 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efca81028c0 msgr2=0x7efca8073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.729+0000 7efca7400640 1 --2- 172.21.15.12:0/343050171 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efca81028c0 0x7efca8073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.729+0000 7efca7400640 1 -- 172.21.15.12:0/343050171 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca8074860 con 0x7efca81016d0 2024-03-23T23:13:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.730+0000 7efca7400640 1 --2- 172.21.15.12:0/343050171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efca81016d0 0x7efca8075450 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7efc9800e9c0 tx=0x7efc9800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.730+0000 7efc9fe00640 1 -- 172.21.15.12:0/343050171 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9800cd80 con 0x7efca81016d0 2024-03-23T23:13:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.730+0000 7efc9fe00640 1 -- 172.21.15.12:0/343050171 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efc980035f0 con 0x7efca81016d0 2024-03-23T23:13:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.730+0000 7efcae000640 1 -- 172.21.15.12:0/343050171 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca81a19c0 con 0x7efca81016d0 2024-03-23T23:13:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.730+0000 7efc9fe00640 1 -- 172.21.15.12:0/343050171 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc98010690 con 0x7efca81016d0 2024-03-23T23:13:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.730+0000 7efcae000640 1 -- 172.21.15.12:0/343050171 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efca81a1f60 con 0x7efca81016d0 2024-03-23T23:13:24.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.733+0000 7efcb0d7a740 1 Processor -- start 2024-03-23T23:13:24.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.733+0000 7efcb0d7a740 1 -- start start 2024-03-23T23:13:24.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.733+0000 7efc9fe00640 1 -- 172.21.15.12:0/343050171 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7efc980037a0 con 0x7efca81016d0 2024-03-23T23:13:24.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.734+0000 7efcb0d7a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623033c91e0 0x5623033c9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.734+0000 7efcb0d7a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623033c9bd0 0x5623033ce6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.734+0000 7efcb0d7a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623033ced40 0x5623033d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.734+0000 7efcb0d7a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x56230338c3b0 con 0x5623033ced40 2024-03-23T23:13:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.734+0000 7efcb0d7a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56230332e830 con 0x5623033c9bd0 2024-03-23T23:13:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.734+0000 7efcb0d7a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623032d75d0 con 0x5623033c91e0 2024-03-23T23:13:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.734+0000 7efca7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623033c9bd0 0x5623033ce6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.734+0000 7efca7e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623033c91e0 0x5623033c9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.734+0000 7efca7e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623033c91e0 0x5623033c9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:57674/0 (socket says 172.21.15.12:57674) 2024-03-23T23:13:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.734+0000 7efca6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623033ced40 0x5623033d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.734+0000 7efca7e00640 1 -- 172.21.15.12:0/1776724467 learned_addr learned my addr 172.21.15.12:0/1776724467 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.734+0000 7efca7e00640 1 -- 172.21.15.12:0/1776724467 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623033c9bd0 msgr2=0x5623033ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.734+0000 7efca7e00640 1 --2- 172.21.15.12:0/1776724467 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623033c9bd0 0x5623033ce6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.734+0000 7efca7e00640 1 -- 172.21.15.12:0/1776724467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623033ced40 msgr2=0x5623033d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.734+0000 7efca7e00640 1 --2- 172.21.15.12:0/1776724467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623033ced40 0x5623033d11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.735+0000 7efca7e00640 1 -- 172.21.15.12:0/1776724467 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5623032ef2c0 con 0x5623033c91e0 2024-03-23T23:13:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.735+0000 7efca7e00640 1 --2- 172.21.15.12:0/1776724467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623033c91e0 0x5623033c9600 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7efca002c500 tx=0x7efca0009330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f921cb544ee852f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.735+0000 7efc93e00640 1 -- 172.21.15.12:0/1776724467 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca0043030 con 0x5623033c91e0 2024-03-23T23:13:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.735+0000 7efc93e00640 1 -- 172.21.15.12:0/1776724467 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efca0033580 con 0x5623033c91e0 2024-03-23T23:13:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.735+0000 7efc93e00640 1 -- 172.21.15.12:0/1776724467 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca002c890 con 0x5623033c91e0 2024-03-23T23:13:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.736+0000 7efc9fe00640 1 --2- 172.21.15.12:0/343050171 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7efc8c0a2920 0x7efc8c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.736+0000 7efc9fe00640 1 -- 172.21.15.12:0/343050171 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7efc980cee40 con 0x7efca81016d0 2024-03-23T23:13:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.736+0000 7efca7400640 1 --2- 172.21.15.12:0/343050171 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7efc8c0a2920 0x7efc8c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.736+0000 7efcb0d7a740 1 -- 172.21.15.12:0/1776724467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623033c91e0 msgr2=0x5623033c9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.736+0000 7efcb0d7a740 1 --2- 172.21.15.12:0/1776724467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623033c91e0 0x5623033c9600 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7efca002c500 tx=0x7efca0009330 comp rx=0 tx=0).stop 2024-03-23T23:13:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.736+0000 7efcb0d7a740 1 -- 172.21.15.12:0/1776724467 shutdown_connections 2024-03-23T23:13:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.736+0000 7efcb0d7a740 1 --2- 172.21.15.12:0/1776724467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623033ced40 0x5623033d11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.736+0000 7efcb0d7a740 1 --2- 172.21.15.12:0/1776724467 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623033c9bd0 0x5623033ce6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.736+0000 7efcb0d7a740 1 --2- 172.21.15.12:0/1776724467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623033c91e0 0x5623033c9600 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.736+0000 7efcb0d7a740 1 -- 172.21.15.12:0/1776724467 >> 172.21.15.12:0/1776724467 conn(0x5623033c4750 msgr2=0x5623033c4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.736+0000 7efcb0d7a740 1 -- 172.21.15.12:0/1776724467 shutdown_connections 2024-03-23T23:13:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.737+0000 7efcb0d7a740 1 -- 172.21.15.12:0/1776724467 wait complete. 2024-03-23T23:13:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.737+0000 7efca7400640 1 --2- 172.21.15.12:0/343050171 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7efc8c0a2920 0x7efc8c0a4de0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7efc98098160 tx=0x7efc980d2880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.737+0000 7efcb0d7a740 1 -- start start 2024-03-23T23:13:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.737+0000 7efcb0d7a740 1 Processor -- start 2024-03-23T23:13:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.738+0000 7efcb0d7a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623033c91e0 0x5623033d3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.738+0000 7efcb0d7a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623033c9bd0 0x5623033d3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.738+0000 7efcb0d7a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623033ced40 0x5623033da880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.738+0000 7efcb0d7a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623032eaa20 con 0x5623033c9bd0 2024-03-23T23:13:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.738+0000 7efcb0d7a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56230332a8d0 con 0x5623033c91e0 2024-03-23T23:13:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.738+0000 7efcb0d7a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623033d1c90 con 0x5623033ced40 2024-03-23T23:13:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.738+0000 7efcb0d7a740 10 client.0 _ll_register_callbacks cb 0x5623033271c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:13:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.738+0000 7efca7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623033c91e0 0x5623033d3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.738+0000 7efca7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623033c91e0 0x5623033d3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44376/0 (socket says 172.21.15.12:44376) 2024-03-23T23:13:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.738+0000 7efca7e00640 1 -- 172.21.15.12:0/2206915561 learned_addr learned my addr 172.21.15.12:0/2206915561 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.738+0000 7efca6a00640 1 --2- 172.21.15.12:0/2206915561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623033c9bd0 0x5623033d3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.738+0000 7efca7e00640 1 --2- 172.21.15.12:0/2206915561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623033ced40 0x5623033da880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.738+0000 7efca7e00640 1 -- 172.21.15.12:0/2206915561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623033ced40 msgr2=0x5623033da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.738+0000 7efca7e00640 1 --2- 172.21.15.12:0/2206915561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623033ced40 0x5623033da880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.738+0000 7efca7e00640 1 -- 172.21.15.12:0/2206915561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623033c9bd0 msgr2=0x5623033d3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.738+0000 7efca7e00640 1 --2- 172.21.15.12:0/2206915561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623033c9bd0 0x5623033d3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.738+0000 7efca7e00640 1 -- 172.21.15.12:0/2206915561 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562303043210 con 0x5623033c91e0 2024-03-23T23:13:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.739+0000 7efca7e00640 1 --2- 172.21.15.12:0/2206915561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623033c91e0 0x5623033d3300 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7efca0004ab0 tx=0x7efca0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83bf0c91d8aa693d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.739+0000 7efc90c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:13:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.739+0000 7efc90c00640 1 -- 172.21.15.12:0/2206915561 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca0043040 con 0x5623033c91e0 2024-03-23T23:13:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.739+0000 7efc90c00640 1 -- 172.21.15.12:0/2206915561 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7efca0033840 con 0x5623033c91e0 2024-03-23T23:13:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.739+0000 7efcb0d7a740 1 -- 172.21.15.12:0/2206915561 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efca0033580 con 0x5623033c91e0 2024-03-23T23:13:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.745+0000 7efc90c00640 1 -- 172.21.15.12:0/2206915561 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7efca0004b40 con 0x5623033c91e0 2024-03-23T23:13:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.745+0000 7efcb0d7a740 10 client.28553 fetch_fsmap learned FSMap version 23 2024-03-23T23:13:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.745+0000 7efcb0d7a740 1 -- 172.21.15.12:0/2206915561 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56230310f620 con 0x5623033c91e0 2024-03-23T23:13:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.745+0000 7efc90c00640 1 -- 172.21.15.12:0/2206915561 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efca0033580 con 0x5623033c91e0 2024-03-23T23:13:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.745+0000 7efc90c00640 1 -- 172.21.15.12:0/2206915561 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7efca0004420 con 0x5623033c91e0 2024-03-23T23:13:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.745+0000 7efcb0d7a740 10 client.28553 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:13:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.745+0000 7efcb0d7a740 10 client.28553 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:13:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.745+0000 7efcb0d7a740 20 client.28553 populate_metadata read hostname 'smithi012' 2024-03-23T23:13:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.745+0000 7efcb0d7a740 1 --2- 172.21.15.12:0/2206915561 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5623033ded70 0x5623033e1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.745+0000 7efcb0d7a740 4 client.28553 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:13:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.745+0000 7efcb0d7a740 1 -- 172.21.15.12:0/2206915561 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56230333bd70 con 0x5623033ded70 2024-03-23T23:13:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.746+0000 7efca6a00640 1 --2- 172.21.15.12:0/2206915561 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5623033ded70 0x5623033e1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.746+0000 7efca6a00640 1 --2- 172.21.15.12:0/2206915561 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5623033ded70 0x5623033e1160 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32b20465bfb62ec9 server_cookie=9f14cc89a0b02c9d in_seq=0 out_seq=0 2024-03-23T23:13:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.746+0000 7efc90c00640 10 client.28553 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.747+0000 7efc90c00640 1 -- 172.21.15.12:0/2206915561 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56230333bd70 con 0x5623033ded70 2024-03-23T23:13:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.747+0000 7efc90c00640 10 client.28553 handle_command_reply: tid=0 2024-03-23T23:13:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.747+0000 7efcb0d7a740 1 client.28553 shutdown 2024-03-23T23:13:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.747+0000 7efcb0d7a740 1 -- 172.21.15.12:0/2206915561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623033c91e0 msgr2=0x5623033d3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.747+0000 7efcb0d7a740 1 --2- 172.21.15.12:0/2206915561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623033c91e0 0x5623033d3300 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7efca0004ab0 tx=0x7efca0004a40 comp rx=0 tx=0).stop 2024-03-23T23:13:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.748+0000 7efcb0d7a740 1 -- 172.21.15.12:0/2206915561 shutdown_connections 2024-03-23T23:13:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.748+0000 7efcb0d7a740 1 --2- 172.21.15.12:0/2206915561 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x5623033ded70 0x5623033e1160 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.748+0000 7efcb0d7a740 1 --2- 172.21.15.12:0/2206915561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623033ced40 0x5623033da880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.748+0000 7efcb0d7a740 1 --2- 172.21.15.12:0/2206915561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623033c9bd0 0x5623033d3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.748+0000 7efc90c00640 0 client.28553 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.748+0000 7efcb0d7a740 1 --2- 172.21.15.12:0/2206915561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623033c91e0 0x5623033d3300 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.748+0000 7efcb0d7a740 1 -- 172.21.15.12:0/2206915561 >> 172.21.15.12:0/2206915561 conn(0x5623033c4750 msgr2=0x5623033c3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.748+0000 7efcb0d7a740 1 -- 172.21.15.12:0/2206915561 shutdown_connections 2024-03-23T23:13:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.748+0000 7efcb0d7a740 1 -- 172.21.15.12:0/2206915561 wait complete. 2024-03-23T23:13:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.748+0000 7efcb0d7a740 20 client.28553 trim_cache size 0 max 16384 2024-03-23T23:13:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.763+0000 7efcb0d7a740 1 Processor -- start 2024-03-23T23:13:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.763+0000 7efcb0d7a740 1 -- start start 2024-03-23T23:13:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.763+0000 7efcb0d7a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623033c91e0 0x5623033c79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.763+0000 7efcb0d7a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623033c9bd0 0x5623033da810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.763+0000 7efcb0d7a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623033ced40 0x5623033d2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.763+0000 7efcb0d7a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca81ad050 con 0x5623033ced40 2024-03-23T23:13:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.763+0000 7efcb0d7a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca81aab50 con 0x5623033c91e0 2024-03-23T23:13:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.763+0000 7efcb0d7a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca81a8bf0 con 0x5623033c9bd0 2024-03-23T23:13:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.764+0000 7efca7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623033c91e0 0x5623033c79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.764+0000 7efca6a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623033c9bd0 0x5623033da810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.764+0000 7efca6a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623033c9bd0 0x5623033da810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:57690/0 (socket says 172.21.15.12:57690) 2024-03-23T23:13:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.764+0000 7efca7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623033c91e0 0x5623033c79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:44384/0 (socket says 172.21.15.12:44384) 2024-03-23T23:13:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.764+0000 7efca6a00640 1 -- 172.21.15.12:0/2609147454 learned_addr learned my addr 172.21.15.12:0/2609147454 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.764+0000 7efca7e00640 1 --2- 172.21.15.12:0/2609147454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623033ced40 0x5623033d2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.764+0000 7efca6a00640 1 -- 172.21.15.12:0/2609147454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623033c91e0 msgr2=0x5623033c79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.764+0000 7efca6a00640 1 --2- 172.21.15.12:0/2609147454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623033c91e0 0x5623033c79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.764+0000 7efca6a00640 1 -- 172.21.15.12:0/2609147454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623033ced40 msgr2=0x5623033d2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.764+0000 7efca6a00640 1 --2- 172.21.15.12:0/2609147454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623033ced40 0x5623033d2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.764+0000 7efca6a00640 1 -- 172.21.15.12:0/2609147454 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5623033c54c0 con 0x5623033c9bd0 2024-03-23T23:13:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.764+0000 7efca7e00640 1 --2- 172.21.15.12:0/2609147454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623033ced40 0x5623033d2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:13:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.764+0000 7efca6a00640 1 --2- 172.21.15.12:0/2609147454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623033c9bd0 0x5623033da810 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7efc94009610 tx=0x7efc94009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9fb64d7c7482857 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.765+0000 7efc93400640 1 -- 172.21.15.12:0/2609147454 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc940202d0 con 0x5623033c9bd0 2024-03-23T23:13:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.765+0000 7efc93400640 1 -- 172.21.15.12:0/2609147454 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efc94024030 con 0x5623033c9bd0 2024-03-23T23:13:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.765+0000 7efc93400640 1 -- 172.21.15.12:0/2609147454 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc940047a0 con 0x5623033c9bd0 2024-03-23T23:13:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.765+0000 7efcb0d7a740 1 -- 172.21.15.12:0/2609147454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623033c9bd0 msgr2=0x5623033da810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.765+0000 7efcb0d7a740 1 --2- 172.21.15.12:0/2609147454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623033c9bd0 0x5623033da810 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7efc94009610 tx=0x7efc94009260 comp rx=0 tx=0).stop 2024-03-23T23:13:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.765+0000 7efcb0d7a740 1 -- 172.21.15.12:0/2609147454 shutdown_connections 2024-03-23T23:13:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.765+0000 7efcb0d7a740 1 --2- 172.21.15.12:0/2609147454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623033ced40 0x5623033d2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.765+0000 7efcb0d7a740 1 --2- 172.21.15.12:0/2609147454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623033c9bd0 0x5623033da810 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.765+0000 7efcb0d7a740 1 --2- 172.21.15.12:0/2609147454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623033c91e0 0x5623033c79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.765+0000 7efcb0d7a740 1 -- 172.21.15.12:0/2609147454 >> 172.21.15.12:0/2609147454 conn(0x5623033c4750 msgr2=0x5623033d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.765+0000 7efcb0d7a740 1 -- 172.21.15.12:0/2609147454 shutdown_connections 2024-03-23T23:13:24.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.765+0000 7efcb0d7a740 1 -- 172.21.15.12:0/2609147454 wait complete. 2024-03-23T23:13:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.766+0000 7efcb0d7a740 1 -- start start 2024-03-23T23:13:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.766+0000 7efcb0d7a740 1 Processor -- start 2024-03-23T23:13:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.766+0000 7efcb0d7a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623033c91e0 0x5623033d83e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.766+0000 7efcb0d7a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623033c9bd0 0x5623033d8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.766+0000 7efcb0d7a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623033ced40 0x5623033d9930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.766+0000 7efcb0d7a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca81a91f0 con 0x5623033c9bd0 2024-03-23T23:13:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.766+0000 7efcb0d7a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca81aa4f0 con 0x5623033ced40 2024-03-23T23:13:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.766+0000 7efcb0d7a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca81a9830 con 0x5623033c91e0 2024-03-23T23:13:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.766+0000 7efcb0d7a740 10 client.0 _ll_register_callbacks cb 0x7efc94009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:13:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.767+0000 7efca6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623033c9bd0 0x5623033d8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.767+0000 7efca6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623033c9bd0 0x5623033d8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42210/0 (socket says 172.21.15.12:42210) 2024-03-23T23:13:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.767+0000 7efca6a00640 1 -- 172.21.15.12:0/263732454 learned_addr learned my addr 172.21.15.12:0/263732454 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.767+0000 7efca7e00640 1 --2- 172.21.15.12:0/263732454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623033ced40 0x5623033d9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.767+0000 7efca6a00640 1 -- 172.21.15.12:0/263732454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623033c91e0 msgr2=0x5623033d83e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-23T23:13:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.767+0000 7efca6a00640 1 --2- 172.21.15.12:0/263732454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623033c91e0 0x5623033d83e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.767+0000 7efca6a00640 1 -- 172.21.15.12:0/263732454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623033ced40 msgr2=0x5623033d9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.767+0000 7efca6a00640 1 --2- 172.21.15.12:0/263732454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623033ced40 0x5623033d9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.767+0000 7efca6a00640 1 -- 172.21.15.12:0/263732454 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5623033d9f20 con 0x5623033c9bd0 2024-03-23T23:13:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.767+0000 7efca7e00640 1 --2- 172.21.15.12:0/263732454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623033ced40 0x5623033d9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.767+0000 7efca6a00640 1 --2- 172.21.15.12:0/263732454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623033c9bd0 0x5623033d8920 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7efc94009eb0 tx=0x7efc94009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26b85a138762af27 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.767+0000 7efc90c00640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:13:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.768+0000 7efc90c00640 1 -- 172.21.15.12:0/263732454 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9402e070 con 0x5623033c9bd0 2024-03-23T23:13:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.768+0000 7efc90c00640 1 -- 172.21.15.12:0/263732454 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7efc940118f0 con 0x5623033c9bd0 2024-03-23T23:13:24.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.768+0000 7efcb0d7a740 1 -- 172.21.15.12:0/263732454 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efc94024030 con 0x5623033c9bd0 2024-03-23T23:13:24.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.768+0000 7efc90c00640 1 -- 172.21.15.12:0/263732454 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc9403a050 con 0x5623033c9bd0 2024-03-23T23:13:24.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.768+0000 7efcb0d7a740 10 client.18429 fetch_fsmap learned FSMap version 23 2024-03-23T23:13:24.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.768+0000 7efcb0d7a740 1 -- 172.21.15.12:0/263732454 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56230343a9c0 con 0x5623033c9bd0 2024-03-23T23:13:24.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.768+0000 7efc90c00640 1 -- 172.21.15.12:0/263732454 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efc94024030 con 0x5623033c9bd0 2024-03-23T23:13:24.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.768+0000 7efc90c00640 1 -- 172.21.15.12:0/263732454 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7efc94029ba0 con 0x5623033c9bd0 2024-03-23T23:13:24.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.768+0000 7efcb0d7a740 10 client.18429 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:13:24.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.768+0000 7efcb0d7a740 10 client.18429 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-23T23:13:24.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.768+0000 7efcb0d7a740 20 client.18429 populate_metadata read hostname 'smithi012' 2024-03-23T23:13:24.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.768+0000 7efcb0d7a740 1 --2- 172.21.15.12:0/263732454 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56230343b380 0x56230343d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:24.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.768+0000 7efcb0d7a740 4 client.18429 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-23T23:13:24.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.768+0000 7efcb0d7a740 1 -- 172.21.15.12:0/263732454 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5623033360c0 con 0x56230343b380 2024-03-23T23:13:24.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.769+0000 7efca7e00640 1 --2- 172.21.15.12:0/263732454 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56230343b380 0x56230343d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:24.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.769+0000 7efca7e00640 1 --2- 172.21.15.12:0/263732454 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56230343b380 0x56230343d7a0 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cf85dcd17cbb5af server_cookie=936ad5e5c858e93e in_seq=0 out_seq=0 2024-03-23T23:13:24.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.769+0000 7efc90c00640 10 client.18429 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.783+0000 7efc90c00640 1 -- 172.21.15.12:0/263732454 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5623033360c0 con 0x56230343b380 2024-03-23T23:13:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.783+0000 7efc90c00640 10 client.18429 handle_command_reply: tid=0 2024-03-23T23:13:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.783+0000 7efcb0d7a740 1 client.18429 shutdown 2024-03-23T23:13:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.783+0000 7efcb0d7a740 1 -- 172.21.15.12:0/263732454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623033c9bd0 msgr2=0x5623033d8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.783+0000 7efcb0d7a740 1 --2- 172.21.15.12:0/263732454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623033c9bd0 0x5623033d8920 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7efc94009eb0 tx=0x7efc94009c20 comp rx=0 tx=0).stop 2024-03-23T23:13:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.783+0000 7efcb0d7a740 1 -- 172.21.15.12:0/263732454 shutdown_connections 2024-03-23T23:13:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.783+0000 7efcb0d7a740 1 --2- 172.21.15.12:0/263732454 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x56230343b380 0x56230343d7a0 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.783+0000 7efcb0d7a740 1 --2- 172.21.15.12:0/263732454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623033ced40 0x5623033d9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.783+0000 7efcb0d7a740 1 --2- 172.21.15.12:0/263732454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5623033c9bd0 0x5623033d8920 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.783+0000 7efc90c00640 0 client.18429 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.783+0000 7efcb0d7a740 1 --2- 172.21.15.12:0/263732454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623033c91e0 0x5623033d83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.783+0000 7efcb0d7a740 1 -- 172.21.15.12:0/263732454 >> 172.21.15.12:0/263732454 conn(0x5623033c4750 msgr2=0x5623033c79b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.784+0000 7efcb0d7a740 1 -- 172.21.15.12:0/263732454 shutdown_connections 2024-03-23T23:13:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.784+0000 7efcb0d7a740 1 -- 172.21.15.12:0/263732454 wait complete. 2024-03-23T23:13:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.784+0000 7efcb0d7a740 20 client.18429 trim_cache size 0 max 16384 2024-03-23T23:13:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-03-23T23:13:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-03-23T23:13:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-03-23T23:13:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-03-23T23:13:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.785+0000 7efcae000640 1 -- 172.21.15.12:0/343050171 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7efc8c0a2920 msgr2=0x7efc8c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.785+0000 7efcae000640 1 --2- 172.21.15.12:0/343050171 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7efc8c0a2920 0x7efc8c0a4de0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7efc98098160 tx=0x7efc980d2880 comp rx=0 tx=0).stop 2024-03-23T23:13:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.785+0000 7efcae000640 1 -- 172.21.15.12:0/343050171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efca81016d0 msgr2=0x7efca8075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.785+0000 7efcae000640 1 --2- 172.21.15.12:0/343050171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efca81016d0 0x7efca8075450 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7efc9800e9c0 tx=0x7efc9800ee80 comp rx=0 tx=0).stop 2024-03-23T23:13:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.785+0000 7efcae000640 1 -- 172.21.15.12:0/343050171 shutdown_connections 2024-03-23T23:13:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.785+0000 7efcae000640 1 --2- 172.21.15.12:0/343050171 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7efc8c0a2920 0x7efc8c0a4de0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.785+0000 7efcae000640 1 --2- 172.21.15.12:0/343050171 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efca8103280 0x7efca8073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.785+0000 7efcae000640 1 --2- 172.21.15.12:0/343050171 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efca81028c0 0x7efca8073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.785+0000 7efcae000640 1 --2- 172.21.15.12:0/343050171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efca81016d0 0x7efca8075450 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.785+0000 7efcae000640 1 -- 172.21.15.12:0/343050171 >> 172.21.15.12:0/343050171 conn(0x7efca80fcc60 msgr2=0x7efca8105af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.785+0000 7efcae000640 1 -- 172.21.15.12:0/343050171 shutdown_connections 2024-03-23T23:13:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-03-23T23:13:24.785+0000 7efcae000640 1 -- 172.21.15.12:0/343050171 wait complete. 2024-03-23T23:13:25.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:24 smithi012 ceph-mon[31695]: pgmap v738: 97 pgs: 97 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:13:25.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:24 smithi173 ceph-mon[36116]: pgmap v738: 97 pgs: 97 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:13:25.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:24 smithi203 ceph-mon[36693]: pgmap v738: 97 pgs: 97 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:13:27.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:26 smithi012 ceph-mon[31695]: pgmap v739: 97 pgs: 97 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-03-23T23:13:27.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:26 smithi173 ceph-mon[36116]: pgmap v739: 97 pgs: 97 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-03-23T23:13:27.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:26 smithi203 ceph-mon[36693]: pgmap v739: 97 pgs: 97 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-03-23T23:13:29.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:28 smithi012 ceph-mon[31695]: pgmap v740: 97 pgs: 97 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:13:29.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:28 smithi173 ceph-mon[36116]: pgmap v740: 97 pgs: 97 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:13:29.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:28 smithi203 ceph-mon[36693]: pgmap v740: 97 pgs: 97 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-23T23:13:31.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:30 smithi012 ceph-mon[31695]: pgmap v741: 97 pgs: 97 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:13:31.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:30 smithi173 ceph-mon[36116]: pgmap v741: 97 pgs: 97 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:13:31.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:30 smithi203 ceph-mon[36693]: pgmap v741: 97 pgs: 97 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:13:33.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:32 smithi012 ceph-mon[31695]: pgmap v742: 97 pgs: 97 active+clean; 4.0 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:13:33.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:32 smithi173 ceph-mon[36116]: pgmap v742: 97 pgs: 97 active+clean; 4.0 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:13:33.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:32 smithi203 ceph-mon[36693]: pgmap v742: 97 pgs: 97 active+clean; 4.0 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-23T23:13:34.890 INFO:tasks.workunit.client.0.smithi012.stdout:561/600: open fd = 3 2024-03-23T23:13:34.890 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.890 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.890 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.890 INFO:tasks.workunit.client.0.smithi012.stdout:562/600: open fd = 3 2024-03-23T23:13:34.890 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.890 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.891 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.891 INFO:tasks.workunit.client.0.smithi012.stdout:563/600: open fd = 3 2024-03-23T23:13:34.891 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.891 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.891 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.891 INFO:tasks.workunit.client.0.smithi012.stdout:564/600: open fd = 3 2024-03-23T23:13:34.891 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.891 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.891 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.891 INFO:tasks.workunit.client.0.smithi012.stdout:565/600: open fd = 3 2024-03-23T23:13:34.891 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.892 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.892 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.892 INFO:tasks.workunit.client.0.smithi012.stdout:566/600: open fd = 3 2024-03-23T23:13:34.892 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.892 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.892 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.892 INFO:tasks.workunit.client.0.smithi012.stdout:567/600: open fd = 3 2024-03-23T23:13:34.892 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.892 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.892 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.892 INFO:tasks.workunit.client.0.smithi012.stdout:568/600: open fd = 3 2024-03-23T23:13:34.892 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.892 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.892 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.892 INFO:tasks.workunit.client.0.smithi012.stdout:569/600: open fd = 3 2024-03-23T23:13:34.893 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.893 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.893 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.893 INFO:tasks.workunit.client.0.smithi012.stdout:570/600: open fd = 3 2024-03-23T23:13:34.893 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.893 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.893 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.893 INFO:tasks.workunit.client.0.smithi012.stdout:571/600: open fd = 3 2024-03-23T23:13:34.893 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.893 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.893 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.893 INFO:tasks.workunit.client.0.smithi012.stdout:572/600: open fd = 3 2024-03-23T23:13:34.893 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.893 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.893 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.894 INFO:tasks.workunit.client.0.smithi012.stdout:573/600: open fd = 3 2024-03-23T23:13:34.894 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.894 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.894 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.894 INFO:tasks.workunit.client.0.smithi012.stdout:574/600: open fd = 3 2024-03-23T23:13:34.894 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.894 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.894 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.894 INFO:tasks.workunit.client.0.smithi012.stdout:575/600: open fd = 3 2024-03-23T23:13:34.894 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.894 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.894 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.894 INFO:tasks.workunit.client.0.smithi012.stdout:576/600: open fd = 3 2024-03-23T23:13:34.894 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.895 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.895 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.895 INFO:tasks.workunit.client.0.smithi012.stdout:577/600: open fd = 3 2024-03-23T23:13:34.895 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.895 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.895 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.895 INFO:tasks.workunit.client.0.smithi012.stdout:578/600: open fd = 3 2024-03-23T23:13:34.895 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.895 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.895 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.895 INFO:tasks.workunit.client.0.smithi012.stdout:579/600: open fd = 3 2024-03-23T23:13:34.895 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.895 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.895 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.896 INFO:tasks.workunit.client.0.smithi012.stdout:580/600: open fd = 3 2024-03-23T23:13:34.896 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.896 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.896 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.896 INFO:tasks.workunit.client.0.smithi012.stdout:581/600: open fd = 3 2024-03-23T23:13:34.896 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.896 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.896 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.896 INFO:tasks.workunit.client.0.smithi012.stdout:582/600: open fd = 3 2024-03-23T23:13:34.896 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.896 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.896 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.896 INFO:tasks.workunit.client.0.smithi012.stdout:583/600: open fd = 3 2024-03-23T23:13:34.896 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.896 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.896 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.897 INFO:tasks.workunit.client.0.smithi012.stdout:584/600: open fd = 3 2024-03-23T23:13:34.897 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.897 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.897 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.897 INFO:tasks.workunit.client.0.smithi012.stdout:585/600: open fd = 3 2024-03-23T23:13:34.897 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.897 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.897 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.897 INFO:tasks.workunit.client.0.smithi012.stdout:586/600: open fd = 3 2024-03-23T23:13:34.897 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.897 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.897 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.897 INFO:tasks.workunit.client.0.smithi012.stdout:587/600: open fd = 3 2024-03-23T23:13:34.897 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.898 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.898 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.898 INFO:tasks.workunit.client.0.smithi012.stdout:588/600: open fd = 3 2024-03-23T23:13:34.898 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.898 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.898 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.898 INFO:tasks.workunit.client.0.smithi012.stdout:589/600: open fd = 3 2024-03-23T23:13:34.898 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.898 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.898 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.898 INFO:tasks.workunit.client.0.smithi012.stdout:590/600: open fd = 3 2024-03-23T23:13:34.898 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.898 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.898 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.898 INFO:tasks.workunit.client.0.smithi012.stdout:591/600: open fd = 3 2024-03-23T23:13:34.899 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.899 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.899 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.899 INFO:tasks.workunit.client.0.smithi012.stdout:592/600: open fd = 3 2024-03-23T23:13:34.899 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.899 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.899 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.899 INFO:tasks.workunit.client.0.smithi012.stdout:593/600: open fd = 3 2024-03-23T23:13:34.899 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.899 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.899 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.899 INFO:tasks.workunit.client.0.smithi012.stdout:594/600: open fd = 3 2024-03-23T23:13:34.899 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.899 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.899 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.899 INFO:tasks.workunit.client.0.smithi012.stdout:595/600: open fd = 3 2024-03-23T23:13:34.899 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.899 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.900 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.900 INFO:tasks.workunit.client.0.smithi012.stdout:596/600: open fd = 3 2024-03-23T23:13:34.900 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.900 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.900 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.900 INFO:tasks.workunit.client.0.smithi012.stdout:597/600: open fd = 3 2024-03-23T23:13:34.900 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.900 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.900 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.900 INFO:tasks.workunit.client.0.smithi012.stdout:598/600: open fd = 3 2024-03-23T23:13:34.900 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.900 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.900 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.900 INFO:tasks.workunit.client.0.smithi012.stdout:599/600: open fd = 3 2024-03-23T23:13:34.900 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.900 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.901 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.901 INFO:tasks.workunit.client.0.smithi012.stdout:600/600: open fd = 3 2024-03-23T23:13:34.901 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.901 INFO:tasks.workunit.client.0.smithi012.stdout:write ret = 32 2024-03-23T23:13:34.901 INFO:tasks.workunit.client.0.smithi012.stdout:pread ret = 64 2024-03-23T23:13:34.901 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo OK 2024-03-23T23:13:34.902 INFO:tasks.workunit.client.0.smithi012.stdout:OK 2024-03-23T23:13:34.902 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-23T23:13:34.902 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-23T23:13:35.035 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:34 smithi012 ceph-mon[31695]: pgmap v743: 97 pgs: 97 active+clean; 4.0 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:13:35.068 INFO:tasks.workunit:Stopping ['fs/test_o_trunc.sh'] on client.0... 2024-03-23T23:13:35.069 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-23T23:13:35.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:34 smithi173 ceph-mon[36116]: pgmap v743: 97 pgs: 97 active+clean; 4.0 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:13:35.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:34 smithi203 ceph-mon[36693]: pgmap v743: 97 pgs: 97 active+clean; 4.0 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-03-23T23:13:36.148 DEBUG:teuthology.parallel:result is None 2024-03-23T23:13:36.149 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-23T23:13:36.205 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-23T23:13:36.206 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-23T23:13:36.234 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-23T23:13:36.234 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-23T23:13:36.245 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi012.front.sepia.ceph.com 2024-03-23T23:13:36.289 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-23T23:13:36.299 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-23T23:13:36.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.409+0000 7f4f5f200640 1 -- 172.21.15.12:0/3571373150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f60109770 msgr2=0x7f4f60110000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:36.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.409+0000 7f4f5f200640 1 --2- 172.21.15.12:0/3571373150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f60109770 0x7f4f60110000 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f4f58009be0 tx=0x7f4f5802bef0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.409+0000 7f4f5f200640 1 -- 172.21.15.12:0/3571373150 shutdown_connections 2024-03-23T23:13:36.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.409+0000 7f4f5f200640 1 --2- 172.21.15.12:0/3571373150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f60109770 0x7f4f60110000 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.409+0000 7f4f5f200640 1 --2- 172.21.15.12:0/3571373150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f60108db0 0x7f4f60109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.409+0000 7f4f5f200640 1 --2- 172.21.15.12:0/3571373150 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f60107bc0 0x7f4f60107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.409+0000 7f4f5f200640 1 -- 172.21.15.12:0/3571373150 >> 172.21.15.12:0/3571373150 conn(0x7f4f60076b70 msgr2=0x7f4f60078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:36.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.409+0000 7f4f5f200640 1 -- 172.21.15.12:0/3571373150 shutdown_connections 2024-03-23T23:13:36.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.409+0000 7f4f5f200640 1 -- 172.21.15.12:0/3571373150 wait complete. 2024-03-23T23:13:36.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.411+0000 7f4f5f200640 1 Processor -- start 2024-03-23T23:13:36.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.411+0000 7f4f5f200640 1 -- start start 2024-03-23T23:13:36.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.411+0000 7f4f5f200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f60107bc0 0x7f4f6019de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:36.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.411+0000 7f4f5f200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f60108db0 0x7f4f6019e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.411+0000 7f4f5f200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f60109770 0x7f4f601a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.411+0000 7f4f5f200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f60113890 con 0x7f4f60107bc0 2024-03-23T23:13:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.411+0000 7f4f5f200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f60113710 con 0x7f4f60108db0 2024-03-23T23:13:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.411+0000 7f4f5f200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f60113a10 con 0x7f4f60109770 2024-03-23T23:13:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.411+0000 7f4f5de00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f60107bc0 0x7f4f6019de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.412+0000 7f4f5e800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f60109770 0x7f4f601a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.412+0000 7f4f57e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f60108db0 0x7f4f6019e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.412+0000 7f4f5de00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f60107bc0 0x7f4f6019de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57110/0 (socket says 172.21.15.12:57110) 2024-03-23T23:13:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.412+0000 7f4f5de00640 1 -- 172.21.15.12:0/2996176655 learned_addr learned my addr 172.21.15.12:0/2996176655 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.412+0000 7f4f5e800640 1 -- 172.21.15.12:0/2996176655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f60108db0 msgr2=0x7f4f6019e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.412+0000 7f4f5e800640 1 --2- 172.21.15.12:0/2996176655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f60108db0 0x7f4f6019e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.412+0000 7f4f5e800640 1 -- 172.21.15.12:0/2996176655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f60107bc0 msgr2=0x7f4f6019de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.412+0000 7f4f5e800640 1 --2- 172.21.15.12:0/2996176655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f60107bc0 0x7f4f6019de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.412+0000 7f4f5e800640 1 -- 172.21.15.12:0/2996176655 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f601a5a40 con 0x7f4f60109770 2024-03-23T23:13:36.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.412+0000 7f4f5e800640 1 --2- 172.21.15.12:0/2996176655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f60109770 0x7f4f601a53d0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f4f5802c430 tx=0x7f4f580337b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:36.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.412+0000 7f4f56a00640 1 -- 172.21.15.12:0/2996176655 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f58043020 con 0x7f4f60109770 2024-03-23T23:13:36.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.412+0000 7f4f56a00640 1 -- 172.21.15.12:0/2996176655 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f5802ca60 con 0x7f4f60109770 2024-03-23T23:13:36.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.413+0000 7f4f56a00640 1 -- 172.21.15.12:0/2996176655 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f58034690 con 0x7f4f60109770 2024-03-23T23:13:36.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.413+0000 7f4f5f200640 1 -- 172.21.15.12:0/2996176655 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f601a5cd0 con 0x7f4f60109770 2024-03-23T23:13:36.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.413+0000 7f4f5f200640 1 -- 172.21.15.12:0/2996176655 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f601a6210 con 0x7f4f60109770 2024-03-23T23:13:36.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.416+0000 7f4f67b95740 1 Processor -- start 2024-03-23T23:13:36.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.416+0000 7f4f56a00640 1 -- 172.21.15.12:0/2996176655 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4f58046020 con 0x7f4f60109770 2024-03-23T23:13:36.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.416+0000 7f4f67b95740 1 -- start start 2024-03-23T23:13:36.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.416+0000 7f4f67b95740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4d9fa6050 0x55b4d9fa6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:36.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.416+0000 7f4f67b95740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d9fa6a40 0x55b4d9fab540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:36.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.416+0000 7f4f67b95740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4d9fabbb0 0x55b4d9fae040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:36.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.416+0000 7f4f67b95740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4d9f0b250 con 0x55b4d9fa6a40 2024-03-23T23:13:36.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.416+0000 7f4f67b95740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4d9eb3a30 con 0x55b4d9fa6050 2024-03-23T23:13:36.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.416+0000 7f4f67b95740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4d9f072f0 con 0x55b4d9fabbb0 2024-03-23T23:13:36.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.416+0000 7f4f57e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4d9fa6050 0x55b4d9fa6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:36.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.416+0000 7f4f57e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4d9fa6050 0x55b4d9fa6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:38134/0 (socket says 172.21.15.12:38134) 2024-03-23T23:13:36.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.416+0000 7f4f57e00640 1 -- 172.21.15.12:0/1479125858 learned_addr learned my addr 172.21.15.12:0/1479125858 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:36.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.417+0000 7f4f5de00640 1 --2- 172.21.15.12:0/1479125858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d9fa6a40 0x55b4d9fab540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:36.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.417+0000 7f4f57e00640 1 --2- 172.21.15.12:0/1479125858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4d9fabbb0 0x55b4d9fae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:36.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.417+0000 7f4f57e00640 1 -- 172.21.15.12:0/1479125858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4d9fabbb0 msgr2=0x55b4d9fae040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:36.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.417+0000 7f4f57e00640 1 --2- 172.21.15.12:0/1479125858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4d9fabbb0 0x55b4d9fae040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.417+0000 7f4f57e00640 1 -- 172.21.15.12:0/1479125858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d9fa6a40 msgr2=0x55b4d9fab540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:36.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.417+0000 7f4f57e00640 1 --2- 172.21.15.12:0/1479125858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d9fa6a40 0x55b4d9fab540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.417+0000 7f4f57e00640 1 -- 172.21.15.12:0/1479125858 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b4d9e9e690 con 0x55b4d9fa6050 2024-03-23T23:13:36.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.417+0000 7f4f57e00640 1 --2- 172.21.15.12:0/1479125858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4d9fa6050 0x55b4d9fa6470 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f4f48013800 tx=0x7f4f48005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2941c58e9548118 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:36.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.417+0000 7f4f56a00640 1 --2- 172.21.15.12:0/2996176655 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4f440a2880 0x7f4f440a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:36.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.417+0000 7f4f4f400640 1 -- 172.21.15.12:0/1479125858 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f48013e40 con 0x55b4d9fa6050 2024-03-23T23:13:36.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.417+0000 7f4f4f400640 1 -- 172.21.15.12:0/1479125858 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f48013120 con 0x55b4d9fa6050 2024-03-23T23:13:36.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.417+0000 7f4f4f400640 1 -- 172.21.15.12:0/1479125858 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f480035f0 con 0x55b4d9fa6050 2024-03-23T23:13:36.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.418+0000 7f4f56a00640 1 -- 172.21.15.12:0/2996176655 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4f580f0f30 con 0x7f4f60109770 2024-03-23T23:13:36.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.418+0000 7f4f5de00640 1 --2- 172.21.15.12:0/2996176655 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4f440a2880 0x7f4f440a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:36.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.418+0000 7f4f67b95740 1 -- 172.21.15.12:0/1479125858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4d9fa6050 msgr2=0x55b4d9fa6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:36.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.418+0000 7f4f67b95740 1 --2- 172.21.15.12:0/1479125858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4d9fa6050 0x55b4d9fa6470 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f4f48013800 tx=0x7f4f48005850 comp rx=0 tx=0).stop 2024-03-23T23:13:36.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.418+0000 7f4f67b95740 1 -- 172.21.15.12:0/1479125858 shutdown_connections 2024-03-23T23:13:36.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.418+0000 7f4f67b95740 1 --2- 172.21.15.12:0/1479125858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4d9fabbb0 0x55b4d9fae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.418+0000 7f4f67b95740 1 --2- 172.21.15.12:0/1479125858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d9fa6a40 0x55b4d9fab540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.418+0000 7f4f67b95740 1 --2- 172.21.15.12:0/1479125858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4d9fa6050 0x55b4d9fa6470 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.418+0000 7f4f67b95740 1 -- 172.21.15.12:0/1479125858 >> 172.21.15.12:0/1479125858 conn(0x55b4d9f6e200 msgr2=0x55b4d9fa19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:36.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.418+0000 7f4f67b95740 1 -- 172.21.15.12:0/1479125858 shutdown_connections 2024-03-23T23:13:36.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.418+0000 7f4f5de00640 1 --2- 172.21.15.12:0/2996176655 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4f440a2880 0x7f4f440a4d40 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x55b4d9fa6e40 tx=0x7f4f50009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:36.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.419+0000 7f4f67b95740 1 -- 172.21.15.12:0/1479125858 wait complete. 2024-03-23T23:13:36.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.419+0000 7f4f67b95740 1 -- start start 2024-03-23T23:13:36.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.419+0000 7f4f67b95740 1 Processor -- start 2024-03-23T23:13:36.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.419+0000 7f4f67b95740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4d9fa6050 0x55b4d9fb0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:36.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.419+0000 7f4f67b95740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d9fa6a40 0x55b4d9fb0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:36.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.419+0000 7f4f67b95740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4d9fabbb0 0x55b4d9fb75b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:36.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.420+0000 7f4f67b95740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4d9faeb00 con 0x55b4d9fa6a40 2024-03-23T23:13:36.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.420+0000 7f4f67b95740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4d9ec6e80 con 0x55b4d9fa6050 2024-03-23T23:13:36.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.420+0000 7f4f67b95740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4d9faec80 con 0x55b4d9fabbb0 2024-03-23T23:13:36.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.420+0000 7f4f67b95740 10 client.0 _ll_register_callbacks cb 0x55b4d9ed9130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:13:36.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.420+0000 7f4f5de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4d9fa6050 0x55b4d9fb0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:36.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.420+0000 7f4f5de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4d9fa6050 0x55b4d9fb0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:38136/0 (socket says 172.21.15.12:38136) 2024-03-23T23:13:36.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.420+0000 7f4f5de00640 1 -- 172.21.15.12:0/2554295267 learned_addr learned my addr 172.21.15.12:0/2554295267 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:36.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.420+0000 7f4f57e00640 1 --2- 172.21.15.12:0/2554295267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d9fa6a40 0x55b4d9fb0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:36.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.420+0000 7f4f5de00640 1 -- 172.21.15.12:0/2554295267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4d9fabbb0 msgr2=0x55b4d9fb75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:36.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.420+0000 7f4f5de00640 1 --2- 172.21.15.12:0/2554295267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4d9fabbb0 0x55b4d9fb75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.420+0000 7f4f5de00640 1 -- 172.21.15.12:0/2554295267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d9fa6a40 msgr2=0x55b4d9fb0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:36.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.420+0000 7f4f5de00640 1 --2- 172.21.15.12:0/2554295267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d9fa6a40 0x55b4d9fb0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.420+0000 7f4f5de00640 1 -- 172.21.15.12:0/2554295267 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b4d9ce97d0 con 0x55b4d9fa6050 2024-03-23T23:13:36.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.420+0000 7f4f57e00640 1 --2- 172.21.15.12:0/2554295267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d9fa6a40 0x55b4d9fb0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:36.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.421+0000 7f4f5de00640 1 --2- 172.21.15.12:0/2554295267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4d9fa6050 0x55b4d9fb0030 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f4f500045e0 tx=0x7f4f50004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2617175cd3da6cf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:36.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.421+0000 7f4f43e00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:13:36.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.421+0000 7f4f43e00640 1 -- 172.21.15.12:0/2554295267 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5001e680 con 0x55b4d9fa6050 2024-03-23T23:13:36.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.421+0000 7f4f43e00640 1 -- 172.21.15.12:0/2554295267 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4f5001e820 con 0x55b4d9fa6050 2024-03-23T23:13:36.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.421+0000 7f4f67b95740 1 -- 172.21.15.12:0/2554295267 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f48013120 con 0x55b4d9fa6050 2024-03-23T23:13:36.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.421+0000 7f4f43e00640 1 -- 172.21.15.12:0/2554295267 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f5001ea20 con 0x55b4d9fa6050 2024-03-23T23:13:36.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.421+0000 7f4f67b95740 10 client.28571 fetch_fsmap learned FSMap version 23 2024-03-23T23:13:36.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.421+0000 7f4f67b95740 1 -- 172.21.15.12:0/2554295267 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55b4d9c1f3f0 con 0x55b4d9fa6050 2024-03-23T23:13:36.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.422+0000 7f4f43e00640 1 -- 172.21.15.12:0/2554295267 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f48013120 con 0x55b4d9fa6050 2024-03-23T23:13:36.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.422+0000 7f4f43e00640 1 -- 172.21.15.12:0/2554295267 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f4f50027070 con 0x55b4d9fa6050 2024-03-23T23:13:36.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.422+0000 7f4f67b95740 10 client.28571 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:13:36.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.422+0000 7f4f67b95740 10 client.28571 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:13:36.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.422+0000 7f4f67b95740 20 client.28571 populate_metadata read hostname 'smithi012' 2024-03-23T23:13:36.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.422+0000 7f4f67b95740 1 --2- 172.21.15.12:0/2554295267 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b4d9fbbe50 0x55b4d9fbe2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:36.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.422+0000 7f4f67b95740 4 client.28571 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:13:36.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.422+0000 7f4f67b95740 1 -- 172.21.15.12:0/2554295267 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b4d9f18790 con 0x55b4d9fbbe50 2024-03-23T23:13:36.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.422+0000 7f4f57e00640 1 --2- 172.21.15.12:0/2554295267 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b4d9fbbe50 0x55b4d9fbe2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:36.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.423+0000 7f4f43e00640 10 client.28571 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:36.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.423+0000 7f4f57e00640 1 --2- 172.21.15.12:0/2554295267 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b4d9fbbe50 0x55b4d9fbe2d0 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=612551062d4129a0 server_cookie=81660ef1a1283695 in_seq=0 out_seq=0 2024-03-23T23:13:36.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.424+0000 7f4f43e00640 1 -- 172.21.15.12:0/2554295267 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b4d9f18790 con 0x55b4d9fbbe50 2024-03-23T23:13:36.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.424+0000 7f4f43e00640 10 client.28571 handle_command_reply: tid=0 2024-03-23T23:13:36.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.424+0000 7f4f67b95740 1 client.28571 shutdown 2024-03-23T23:13:36.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.424+0000 7f4f67b95740 1 -- 172.21.15.12:0/2554295267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4d9fa6050 msgr2=0x55b4d9fb0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:36.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.424+0000 7f4f67b95740 1 --2- 172.21.15.12:0/2554295267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4d9fa6050 0x55b4d9fb0030 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f4f500045e0 tx=0x7f4f50004730 comp rx=0 tx=0).stop 2024-03-23T23:13:36.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.424+0000 7f4f67b95740 1 -- 172.21.15.12:0/2554295267 shutdown_connections 2024-03-23T23:13:36.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.424+0000 7f4f67b95740 1 --2- 172.21.15.12:0/2554295267 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b4d9fbbe50 0x55b4d9fbe2d0 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.424+0000 7f4f67b95740 1 --2- 172.21.15.12:0/2554295267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4d9fabbb0 0x55b4d9fb75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.424+0000 7f4f67b95740 1 --2- 172.21.15.12:0/2554295267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d9fa6a40 0x55b4d9fb0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.424+0000 7f4f67b95740 1 --2- 172.21.15.12:0/2554295267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4d9fa6050 0x55b4d9fb0030 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.424+0000 7f4f67b95740 1 -- 172.21.15.12:0/2554295267 >> 172.21.15.12:0/2554295267 conn(0x55b4d9f6e200 msgr2=0x55b4d9fa0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:36.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.424+0000 7f4f43e00640 0 client.28571 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:36.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.424+0000 7f4f67b95740 1 -- 172.21.15.12:0/2554295267 shutdown_connections 2024-03-23T23:13:36.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.424+0000 7f4f67b95740 1 -- 172.21.15.12:0/2554295267 wait complete. 2024-03-23T23:13:36.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.425+0000 7f4f67b95740 20 client.28571 trim_cache size 0 max 16384 2024-03-23T23:13:36.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.441+0000 7f4f67b95740 1 Processor -- start 2024-03-23T23:13:36.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.441+0000 7f4f67b95740 1 -- start start 2024-03-23T23:13:36.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.442+0000 7f4f67b95740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4d9fa6050 0x55b4d9fa73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:36.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.442+0000 7f4f67b95740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4d9fa6a40 0x55b4d9fb7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:36.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.442+0000 7f4f67b95740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d9fabbb0 0x55b4d9fa7920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:36.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.442+0000 7f4f67b95740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f601b12c0 con 0x55b4d9fabbb0 2024-03-23T23:13:36.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.442+0000 7f4f67b95740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f601aedc0 con 0x55b4d9fa6050 2024-03-23T23:13:36.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.442+0000 7f4f67b95740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f601ace60 con 0x55b4d9fa6a40 2024-03-23T23:13:36.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.442+0000 7f4f57e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4d9fa6a40 0x55b4d9fb7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:36.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.442+0000 7f4f5e800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d9fabbb0 0x55b4d9fa7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:36.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.442+0000 7f4f57e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4d9fa6a40 0x55b4d9fb7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44302/0 (socket says 172.21.15.12:44302) 2024-03-23T23:13:36.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.442+0000 7f4f57e00640 1 -- 172.21.15.12:0/722128873 learned_addr learned my addr 172.21.15.12:0/722128873 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:36.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.442+0000 7f4f57e00640 1 -- 172.21.15.12:0/722128873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4d9fa6050 msgr2=0x55b4d9fa73e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-23T23:13:36.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.442+0000 7f4f57e00640 1 --2- 172.21.15.12:0/722128873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4d9fa6050 0x55b4d9fa73e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.442+0000 7f4f57e00640 1 -- 172.21.15.12:0/722128873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d9fabbb0 msgr2=0x55b4d9fa7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:36.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.442+0000 7f4f57e00640 1 --2- 172.21.15.12:0/722128873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d9fabbb0 0x55b4d9fa7920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.443+0000 7f4f57e00640 1 -- 172.21.15.12:0/722128873 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b4d9bdf0f0 con 0x55b4d9fa6a40 2024-03-23T23:13:36.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.443+0000 7f4f57e00640 1 --2- 172.21.15.12:0/722128873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4d9fa6a40 0x55b4d9fb7570 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f4f48016020 tx=0x7f4f48005040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=453cadf6109078ba server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:36.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.443+0000 7f4f4ea00640 1 -- 172.21.15.12:0/722128873 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4802d070 con 0x55b4d9fa6a40 2024-03-23T23:13:36.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.443+0000 7f4f4ea00640 1 -- 172.21.15.12:0/722128873 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f4800f790 con 0x55b4d9fa6a40 2024-03-23T23:13:36.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.443+0000 7f4f4ea00640 1 -- 172.21.15.12:0/722128873 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f480035f0 con 0x55b4d9fa6a40 2024-03-23T23:13:36.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.444+0000 7f4f67b95740 1 -- 172.21.15.12:0/722128873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4d9fa6a40 msgr2=0x55b4d9fb7570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:36.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.444+0000 7f4f67b95740 1 --2- 172.21.15.12:0/722128873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4d9fa6a40 0x55b4d9fb7570 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f4f48016020 tx=0x7f4f48005040 comp rx=0 tx=0).stop 2024-03-23T23:13:36.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.444+0000 7f4f67b95740 1 -- 172.21.15.12:0/722128873 shutdown_connections 2024-03-23T23:13:36.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.444+0000 7f4f67b95740 1 --2- 172.21.15.12:0/722128873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d9fabbb0 0x55b4d9fa7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.444+0000 7f4f67b95740 1 --2- 172.21.15.12:0/722128873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4d9fa6a40 0x55b4d9fb7570 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.444+0000 7f4f67b95740 1 --2- 172.21.15.12:0/722128873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4d9fa6050 0x55b4d9fa73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.444+0000 7f4f67b95740 1 -- 172.21.15.12:0/722128873 >> 172.21.15.12:0/722128873 conn(0x55b4d9f6e200 msgr2=0x55b4d9fa5490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:36.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.444+0000 7f4f67b95740 1 -- 172.21.15.12:0/722128873 shutdown_connections 2024-03-23T23:13:36.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.444+0000 7f4f67b95740 1 -- 172.21.15.12:0/722128873 wait complete. 2024-03-23T23:13:36.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.444+0000 7f4f67b95740 1 -- start start 2024-03-23T23:13:36.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.444+0000 7f4f67b95740 1 Processor -- start 2024-03-23T23:13:36.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.445+0000 7f4f67b95740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d9fa6050 0x55b4d9fb54d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:36.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.445+0000 7f4f67b95740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4d9fa6a40 0x55b4d9fb5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:36.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.445+0000 7f4f67b95740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4d9fabbb0 0x55b4d9fb6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:36.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.445+0000 7f4f67b95740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f601ad460 con 0x55b4d9fa6050 2024-03-23T23:13:36.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.445+0000 7f4f67b95740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f601ae760 con 0x55b4d9fa6a40 2024-03-23T23:13:36.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.445+0000 7f4f67b95740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f601adaa0 con 0x55b4d9fabbb0 2024-03-23T23:13:36.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.445+0000 7f4f67b95740 10 client.0 _ll_register_callbacks cb 0x7f4f50009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:13:36.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.445+0000 7f4f5de00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d9fa6050 0x55b4d9fb54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:36.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.445+0000 7f4f5de00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d9fa6050 0x55b4d9fb54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57148/0 (socket says 172.21.15.12:57148) 2024-03-23T23:13:36.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.445+0000 7f4f5de00640 1 -- 172.21.15.12:0/2998103695 learned_addr learned my addr 172.21.15.12:0/2998103695 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:36.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.445+0000 7f4f57e00640 1 --2- 172.21.15.12:0/2998103695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4d9fa6a40 0x55b4d9fb5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:36.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.445+0000 7f4f5e800640 1 --2- 172.21.15.12:0/2998103695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4d9fabbb0 0x55b4d9fb6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:36.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.446+0000 7f4f5de00640 1 -- 172.21.15.12:0/2998103695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4d9fabbb0 msgr2=0x55b4d9fb6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:36.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.446+0000 7f4f5de00640 1 --2- 172.21.15.12:0/2998103695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4d9fabbb0 0x55b4d9fb6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.446+0000 7f4f5de00640 1 -- 172.21.15.12:0/2998103695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4d9fa6a40 msgr2=0x55b4d9fb5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:36.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.446+0000 7f4f5de00640 1 --2- 172.21.15.12:0/2998103695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4d9fa6a40 0x55b4d9fb5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.446+0000 7f4f5de00640 1 -- 172.21.15.12:0/2998103695 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b4d9fb7010 con 0x55b4d9fa6050 2024-03-23T23:13:36.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.446+0000 7f4f57e00640 1 --2- 172.21.15.12:0/2998103695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4d9fa6a40 0x55b4d9fb5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:36.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.446+0000 7f4f5de00640 1 --2- 172.21.15.12:0/2998103695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d9fa6050 0x55b4d9fb54d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4f50012d20 tx=0x7f4f5001e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1079b9f50d58725f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:36.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.446+0000 7f4f43e00640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:13:36.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.446+0000 7f4f43e00640 1 -- 172.21.15.12:0/2998103695 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5001edd0 con 0x55b4d9fa6050 2024-03-23T23:13:36.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.446+0000 7f4f43e00640 1 -- 172.21.15.12:0/2998103695 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4f50005020 con 0x55b4d9fa6050 2024-03-23T23:13:36.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.446+0000 7f4f67b95740 1 -- 172.21.15.12:0/2998103695 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f4800f790 con 0x55b4d9fa6050 2024-03-23T23:13:36.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.447+0000 7f4f43e00640 1 -- 172.21.15.12:0/2998103695 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f50005220 con 0x55b4d9fa6050 2024-03-23T23:13:36.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.447+0000 7f4f67b95740 10 client.18447 fetch_fsmap learned FSMap version 23 2024-03-23T23:13:36.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.447+0000 7f4f67b95740 1 -- 172.21.15.12:0/2998103695 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55b4da017ff0 con 0x55b4d9fa6050 2024-03-23T23:13:36.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.447+0000 7f4f43e00640 1 -- 172.21.15.12:0/2998103695 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f4800f790 con 0x55b4d9fa6050 2024-03-23T23:13:36.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.447+0000 7f4f43e00640 1 -- 172.21.15.12:0/2998103695 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f4f50027050 con 0x55b4d9fa6050 2024-03-23T23:13:36.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.447+0000 7f4f67b95740 10 client.18447 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:13:36.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.447+0000 7f4f67b95740 10 client.18447 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:13:36.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.447+0000 7f4f67b95740 20 client.18447 populate_metadata read hostname 'smithi012' 2024-03-23T23:13:36.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.447+0000 7f4f67b95740 1 --2- 172.21.15.12:0/2998103695 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b4da018970 0x55b4da01ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:36.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.447+0000 7f4f67b95740 4 client.18447 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-23T23:13:36.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.447+0000 7f4f67b95740 1 -- 172.21.15.12:0/2998103695 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b4d9f12ae0 con 0x55b4da018970 2024-03-23T23:13:36.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.448+0000 7f4f57e00640 1 --2- 172.21.15.12:0/2998103695 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b4da018970 0x55b4da01ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:36.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.448+0000 7f4f57e00640 1 --2- 172.21.15.12:0/2998103695 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b4da018970 0x55b4da01ad90 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=868614f97f801227 server_cookie=74788387d6265d07 in_seq=0 out_seq=0 2024-03-23T23:13:36.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.448+0000 7f4f43e00640 10 client.18447 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:36.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.448+0000 7f4f43e00640 1 -- 172.21.15.12:0/2998103695 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55b4d9f12ae0 con 0x55b4da018970 2024-03-23T23:13:36.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.448+0000 7f4f43e00640 10 client.18447 handle_command_reply: tid=0 2024-03-23T23:13:36.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.449+0000 7f4f67b95740 1 client.18447 shutdown 2024-03-23T23:13:36.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.449+0000 7f4f67b95740 1 -- 172.21.15.12:0/2998103695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d9fa6050 msgr2=0x55b4d9fb54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:36.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.449+0000 7f4f67b95740 1 --2- 172.21.15.12:0/2998103695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d9fa6050 0x55b4d9fb54d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4f50012d20 tx=0x7f4f5001e990 comp rx=0 tx=0).stop 2024-03-23T23:13:36.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.449+0000 7f4f67b95740 1 -- 172.21.15.12:0/2998103695 shutdown_connections 2024-03-23T23:13:36.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.449+0000 7f4f67b95740 1 --2- 172.21.15.12:0/2998103695 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55b4da018970 0x55b4da01ad90 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.449+0000 7f4f67b95740 1 --2- 172.21.15.12:0/2998103695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4d9fabbb0 0x55b4d9fb6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.449+0000 7f4f67b95740 1 --2- 172.21.15.12:0/2998103695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4d9fa6a40 0x55b4d9fb5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.449+0000 7f4f43e00640 0 client.18447 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:36.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.449+0000 7f4f67b95740 1 --2- 172.21.15.12:0/2998103695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d9fa6050 0x55b4d9fb54d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.449+0000 7f4f67b95740 1 -- 172.21.15.12:0/2998103695 >> 172.21.15.12:0/2998103695 conn(0x55b4d9f6e200 msgr2=0x55b4d9fa0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:36.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.449+0000 7f4f67b95740 1 -- 172.21.15.12:0/2998103695 shutdown_connections 2024-03-23T23:13:36.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.449+0000 7f4f67b95740 1 -- 172.21.15.12:0/2998103695 wait complete. 2024-03-23T23:13:36.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.449+0000 7f4f67b95740 20 client.18447 trim_cache size 0 max 16384 2024-03-23T23:13:36.454 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:13:36.454 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-03-23T23:13:36.454 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-03-23T23:13:36.454 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:13:36.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.450+0000 7f4f5f200640 1 -- 172.21.15.12:0/2996176655 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4f440a2880 msgr2=0x7f4f440a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:36.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.450+0000 7f4f5f200640 1 --2- 172.21.15.12:0/2996176655 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4f440a2880 0x7f4f440a4d40 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x55b4d9fa6e40 tx=0x7f4f50009260 comp rx=0 tx=0).stop 2024-03-23T23:13:36.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.450+0000 7f4f5f200640 1 -- 172.21.15.12:0/2996176655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f60109770 msgr2=0x7f4f601a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:36.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.450+0000 7f4f5f200640 1 --2- 172.21.15.12:0/2996176655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f60109770 0x7f4f601a53d0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f4f5802c430 tx=0x7f4f580337b0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.450+0000 7f4f5f200640 1 -- 172.21.15.12:0/2996176655 shutdown_connections 2024-03-23T23:13:36.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.450+0000 7f4f5f200640 1 --2- 172.21.15.12:0/2996176655 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f4f440a2880 0x7f4f440a4d40 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.450+0000 7f4f5f200640 1 --2- 172.21.15.12:0/2996176655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f60109770 0x7f4f601a53d0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.450+0000 7f4f5f200640 1 --2- 172.21.15.12:0/2996176655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f60108db0 0x7f4f6019e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.450+0000 7f4f5f200640 1 --2- 172.21.15.12:0/2996176655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f60107bc0 0x7f4f6019de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.450+0000 7f4f5f200640 1 -- 172.21.15.12:0/2996176655 >> 172.21.15.12:0/2996176655 conn(0x7f4f60076b70 msgr2=0x7f4f6010c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:36.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.450+0000 7f4f5f200640 1 -- 172.21.15.12:0/2996176655 shutdown_connections 2024-03-23T23:13:36.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.450+0000 7f4f5f200640 1 -- 172.21.15.12:0/2996176655 wait complete. 2024-03-23T23:13:36.464 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T23:13:36.464 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:13:36.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.679+0000 7fa445a00640 1 -- 172.21.15.12:0/1239862996 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa440104ad0 msgr2=0x7fa440104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:36.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.679+0000 7fa445a00640 1 --2- 172.21.15.12:0/1239862996 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa440104ad0 0x7fa440104f50 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fa428009900 tx=0x7fa42802bc50 comp rx=0 tx=0).stop 2024-03-23T23:13:36.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.679+0000 7fa445a00640 1 -- 172.21.15.12:0/1239862996 shutdown_connections 2024-03-23T23:13:36.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.679+0000 7fa445a00640 1 --2- 172.21.15.12:0/1239862996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa440105490 0x7fa44010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.679+0000 7fa445a00640 1 --2- 172.21.15.12:0/1239862996 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa440104ad0 0x7fa440104f50 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.679+0000 7fa445a00640 1 --2- 172.21.15.12:0/1239862996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4401038e0 0x7fa440103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.679+0000 7fa445a00640 1 -- 172.21.15.12:0/1239862996 >> 172.21.15.12:0/1239862996 conn(0x7fa4400fee90 msgr2=0x7fa4401012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:36.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.679+0000 7fa445a00640 1 -- 172.21.15.12:0/1239862996 shutdown_connections 2024-03-23T23:13:36.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.679+0000 7fa445a00640 1 -- 172.21.15.12:0/1239862996 wait complete. 2024-03-23T23:13:36.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.680+0000 7fa445a00640 1 Processor -- start 2024-03-23T23:13:36.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.680+0000 7fa445a00640 1 -- start start 2024-03-23T23:13:36.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.681+0000 7fa445a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4401038e0 0x7fa440199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:36.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.681+0000 7fa445a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa440104ad0 0x7fa44019a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:36.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.681+0000 7fa445a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa440105490 0x7fa4401a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:36.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.681+0000 7fa445a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44010f5b0 con 0x7fa440104ad0 2024-03-23T23:13:36.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.681+0000 7fa445a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44010f430 con 0x7fa440105490 2024-03-23T23:13:36.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.681+0000 7fa445a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44010f730 con 0x7fa4401038e0 2024-03-23T23:13:36.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.681+0000 7fa437e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa440104ad0 0x7fa44019a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:36.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.681+0000 7fa43f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa440105490 0x7fa4401a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:36.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.681+0000 7fa437e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa440104ad0 0x7fa44019a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57166/0 (socket says 172.21.15.12:57166) 2024-03-23T23:13:36.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.681+0000 7fa437e00640 1 -- 172.21.15.12:0/4277367992 learned_addr learned my addr 172.21.15.12:0/4277367992 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:36.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.681+0000 7fa43ea00640 1 --2- 172.21.15.12:0/4277367992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4401038e0 0x7fa440199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:36.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.681+0000 7fa43f400640 1 -- 172.21.15.12:0/4277367992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4401038e0 msgr2=0x7fa440199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:36.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.681+0000 7fa43f400640 1 --2- 172.21.15.12:0/4277367992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4401038e0 0x7fa440199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.681+0000 7fa43f400640 1 -- 172.21.15.12:0/4277367992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa440104ad0 msgr2=0x7fa44019a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:36.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.681+0000 7fa43f400640 1 --2- 172.21.15.12:0/4277367992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa440104ad0 0x7fa44019a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.681+0000 7fa43f400640 1 -- 172.21.15.12:0/4277367992 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4401a17e0 con 0x7fa440105490 2024-03-23T23:13:36.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.682+0000 7fa43f400640 1 --2- 172.21.15.12:0/4277367992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa440105490 0x7fa4401a1170 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fa438009fd0 tx=0x7fa43800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:36.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.682+0000 7fa437400640 1 -- 172.21.15.12:0/4277367992 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43800ce20 con 0x7fa440105490 2024-03-23T23:13:36.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.682+0000 7fa437400640 1 -- 172.21.15.12:0/4277367992 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa438005020 con 0x7fa440105490 2024-03-23T23:13:36.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.682+0000 7fa445a00640 1 -- 172.21.15.12:0/4277367992 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4401a1ad0 con 0x7fa440105490 2024-03-23T23:13:36.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.682+0000 7fa437400640 1 -- 172.21.15.12:0/4277367992 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa438010720 con 0x7fa440105490 2024-03-23T23:13:36.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.682+0000 7fa445a00640 1 -- 172.21.15.12:0/4277367992 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4401a2070 con 0x7fa440105490 2024-03-23T23:13:36.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.684+0000 7fa445a00640 1 -- 172.21.15.12:0/4277367992 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa44010ac00 con 0x7fa440105490 2024-03-23T23:13:36.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.685+0000 7fa437400640 1 -- 172.21.15.12:0/4277367992 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa438005540 con 0x7fa440105490 2024-03-23T23:13:36.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.686+0000 7fa437400640 1 --2- 172.21.15.12:0/4277367992 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa4200a2880 0x7fa4200a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:36.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.686+0000 7fa437400640 1 -- 172.21.15.12:0/4277367992 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fa4380ceff0 con 0x7fa440105490 2024-03-23T23:13:36.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.687+0000 7fa43ea00640 1 --2- 172.21.15.12:0/4277367992 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa4200a2880 0x7fa4200a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:36.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.687+0000 7fa43ea00640 1 --2- 172.21.15.12:0/4277367992 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa4200a2880 0x7fa4200a4d40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fa440104930 tx=0x7fa430005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:36.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.690+0000 7fa437400640 1 -- 172.21.15.12:0/4277367992 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa43809c080 con 0x7fa440105490 2024-03-23T23:13:36.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.926+0000 7fa445a00640 1 -- 172.21.15.12:0/4277367992 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa44002d070 con 0x7fa440105490 2024-03-23T23:13:36.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.927+0000 7fa437400640 1 -- 172.21.15.12:0/4277367992 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7fa43809b980 con 0x7fa440105490 2024-03-23T23:13:36.929 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:13:36.930 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:13:36.930 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:13:36.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.930+0000 7fa445a00640 1 -- 172.21.15.12:0/4277367992 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa4200a2880 msgr2=0x7fa4200a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:36.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.930+0000 7fa445a00640 1 --2- 172.21.15.12:0/4277367992 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa4200a2880 0x7fa4200a4d40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fa440104930 tx=0x7fa430005f90 comp rx=0 tx=0).stop 2024-03-23T23:13:36.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.930+0000 7fa445a00640 1 -- 172.21.15.12:0/4277367992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa440105490 msgr2=0x7fa4401a1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:36.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.930+0000 7fa445a00640 1 --2- 172.21.15.12:0/4277367992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa440105490 0x7fa4401a1170 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fa438009fd0 tx=0x7fa43800ef10 comp rx=0 tx=0).stop 2024-03-23T23:13:36.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.930+0000 7fa445a00640 1 -- 172.21.15.12:0/4277367992 shutdown_connections 2024-03-23T23:13:36.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.930+0000 7fa445a00640 1 --2- 172.21.15.12:0/4277367992 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fa4200a2880 0x7fa4200a4d40 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.930+0000 7fa445a00640 1 --2- 172.21.15.12:0/4277367992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa440105490 0x7fa4401a1170 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.930+0000 7fa445a00640 1 --2- 172.21.15.12:0/4277367992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa440104ad0 0x7fa44019a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.930+0000 7fa445a00640 1 --2- 172.21.15.12:0/4277367992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4401038e0 0x7fa440199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:36.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.930+0000 7fa445a00640 1 -- 172.21.15.12:0/4277367992 >> 172.21.15.12:0/4277367992 conn(0x7fa4400fee90 msgr2=0x7fa440107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:36.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.930+0000 7fa445a00640 1 -- 172.21.15.12:0/4277367992 shutdown_connections 2024-03-23T23:13:36.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:36.930+0000 7fa445a00640 1 -- 172.21.15.12:0/4277367992 wait complete. 2024-03-23T23:13:36.948 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:13:36.948 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-23T23:13:37.024 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:36 smithi012 ceph-mon[31695]: pgmap v744: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:13:37.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:36 smithi173 ceph-mon[36116]: pgmap v744: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:13:37.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:36 smithi203 ceph-mon[36693]: pgmap v744: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-03-23T23:13:37.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.166+0000 7fc518c00640 1 -- 172.21.15.12:0/1533134268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5141049b0 msgr2=0x7fc514104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.166+0000 7fc518c00640 1 --2- 172.21.15.12:0/1533134268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5141049b0 0x7fc514104e30 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fc504009ca0 tx=0x7fc50402c100 comp rx=0 tx=0).stop 2024-03-23T23:13:37.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.166+0000 7fc518c00640 1 -- 172.21.15.12:0/1533134268 shutdown_connections 2024-03-23T23:13:37.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.166+0000 7fc518c00640 1 --2- 172.21.15.12:0/1533134268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc514105370 0x7fc51410bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.166+0000 7fc518c00640 1 --2- 172.21.15.12:0/1533134268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5141049b0 0x7fc514104e30 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.166+0000 7fc518c00640 1 --2- 172.21.15.12:0/1533134268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5141037c0 0x7fc514103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.166+0000 7fc518c00640 1 -- 172.21.15.12:0/1533134268 >> 172.21.15.12:0/1533134268 conn(0x7fc5140fed50 msgr2=0x7fc514101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:37.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.166+0000 7fc518c00640 1 -- 172.21.15.12:0/1533134268 shutdown_connections 2024-03-23T23:13:37.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.167+0000 7fc518c00640 1 -- 172.21.15.12:0/1533134268 wait complete. 2024-03-23T23:13:37.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.167+0000 7fc518c00640 1 Processor -- start 2024-03-23T23:13:37.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.168+0000 7fc518c00640 1 -- start start 2024-03-23T23:13:37.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.168+0000 7fc518c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5141037c0 0x7fc514199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.168+0000 7fc518c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5141049b0 0x7fc514199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.168+0000 7fc518c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc514105370 0x7fc5141a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.168+0000 7fc518c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51410f420 con 0x7fc5141037c0 2024-03-23T23:13:37.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.168+0000 7fc518c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51410f2a0 con 0x7fc5141049b0 2024-03-23T23:13:37.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.168+0000 7fc518c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51410f5a0 con 0x7fc514105370 2024-03-23T23:13:37.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.168+0000 7fc513400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5141037c0 0x7fc514199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.168+0000 7fc512a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5141049b0 0x7fc514199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.168+0000 7fc513e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc514105370 0x7fc5141a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.168+0000 7fc512a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5141049b0 0x7fc514199f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:38198/0 (socket says 172.21.15.12:38198) 2024-03-23T23:13:37.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.168+0000 7fc513400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5141037c0 0x7fc514199a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57184/0 (socket says 172.21.15.12:57184) 2024-03-23T23:13:37.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.168+0000 7fc512a00640 1 -- 172.21.15.12:0/539338729 learned_addr learned my addr 172.21.15.12:0/539338729 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:37.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.169+0000 7fc512a00640 1 -- 172.21.15.12:0/539338729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc514105370 msgr2=0x7fc5141a0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.169+0000 7fc512a00640 1 --2- 172.21.15.12:0/539338729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc514105370 0x7fc5141a0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.169+0000 7fc512a00640 1 -- 172.21.15.12:0/539338729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5141037c0 msgr2=0x7fc514199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.169+0000 7fc512a00640 1 --2- 172.21.15.12:0/539338729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5141037c0 0x7fc514199a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.169+0000 7fc512a00640 1 -- 172.21.15.12:0/539338729 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5141a1650 con 0x7fc5141049b0 2024-03-23T23:13:37.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.169+0000 7fc512a00640 1 --2- 172.21.15.12:0/539338729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5141049b0 0x7fc514199f70 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fc5040325f0 tx=0x7fc504035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:37.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.169+0000 7fc50be00640 1 -- 172.21.15.12:0/539338729 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc504039040 con 0x7fc5141049b0 2024-03-23T23:13:37.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.169+0000 7fc50be00640 1 -- 172.21.15.12:0/539338729 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc504044070 con 0x7fc5141049b0 2024-03-23T23:13:37.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.169+0000 7fc50be00640 1 -- 172.21.15.12:0/539338729 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc504004440 con 0x7fc5141049b0 2024-03-23T23:13:37.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.169+0000 7fc518c00640 1 -- 172.21.15.12:0/539338729 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5141a18e0 con 0x7fc5141049b0 2024-03-23T23:13:37.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.170+0000 7fc518c00640 1 -- 172.21.15.12:0/539338729 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5141a1d50 con 0x7fc5141049b0 2024-03-23T23:13:37.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.173+0000 7fc50be00640 1 -- 172.21.15.12:0/539338729 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc5040045e0 con 0x7fc5141049b0 2024-03-23T23:13:37.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.173+0000 7fc51ce4a740 1 Processor -- start 2024-03-23T23:13:37.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.173+0000 7fc51ce4a740 1 -- start start 2024-03-23T23:13:37.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.173+0000 7fc51ce4a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a249f0 0x55b4e2a24e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.173+0000 7fc51ce4a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4e2a25bc0 0x55b4e2a26040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.173+0000 7fc51ce4a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4e2a26580 0x55b4e2a2ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.173+0000 7fc51ce4a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4e29892c0 con 0x55b4e2a25bc0 2024-03-23T23:13:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.173+0000 7fc51ce4a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4e2931c90 con 0x55b4e2a249f0 2024-03-23T23:13:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.173+0000 7fc51ce4a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4e2985360 con 0x55b4e2a26580 2024-03-23T23:13:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.173+0000 7fc513400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4e2a25bc0 0x55b4e2a26040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.173+0000 7fc513e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a249f0 0x55b4e2a24e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.173+0000 7fc513e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a249f0 0x55b4e2a24e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:38202/0 (socket says 172.21.15.12:38202) 2024-03-23T23:13:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.173+0000 7fc513e00640 1 -- 172.21.15.12:0/2170858637 learned_addr learned my addr 172.21.15.12:0/2170858637 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.173+0000 7fc512a00640 1 --2- 172.21.15.12:0/2170858637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4e2a26580 0x55b4e2a2ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.174+0000 7fc513e00640 1 -- 172.21.15.12:0/2170858637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4e2a26580 msgr2=0x55b4e2a2ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.174+0000 7fc513e00640 1 --2- 172.21.15.12:0/2170858637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4e2a26580 0x55b4e2a2ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.174+0000 7fc513e00640 1 -- 172.21.15.12:0/2170858637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4e2a25bc0 msgr2=0x55b4e2a26040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.174+0000 7fc513e00640 1 --2- 172.21.15.12:0/2170858637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4e2a25bc0 0x55b4e2a26040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.174+0000 7fc513e00640 1 -- 172.21.15.12:0/2170858637 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b4e2956080 con 0x55b4e2a249f0 2024-03-23T23:13:37.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.174+0000 7fc513400640 1 --2- 172.21.15.12:0/2170858637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4e2a25bc0 0x55b4e2a26040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:37.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.174+0000 7fc513e00640 1 --2- 172.21.15.12:0/2170858637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a249f0 0x55b4e2a24e10 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fc50c009c90 tx=0x7fc50c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93cdbc5100d84591 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:37.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.174+0000 7fc50be00640 1 --2- 172.21.15.12:0/539338729 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc4f80a2800 0x7fc4f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.174+0000 7fc503e00640 1 -- 172.21.15.12:0/2170858637 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c004230 con 0x55b4e2a249f0 2024-03-23T23:13:37.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.174+0000 7fc503e00640 1 -- 172.21.15.12:0/2170858637 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc50c009e40 con 0x55b4e2a249f0 2024-03-23T23:13:37.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.174+0000 7fc503e00640 1 -- 172.21.15.12:0/2170858637 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c0197e0 con 0x55b4e2a249f0 2024-03-23T23:13:37.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.175+0000 7fc50be00640 1 -- 172.21.15.12:0/539338729 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fc5040f1110 con 0x7fc5141049b0 2024-03-23T23:13:37.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.175+0000 7fc513400640 1 --2- 172.21.15.12:0/539338729 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc4f80a2800 0x7fc4f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.175+0000 7fc51ce4a740 1 -- 172.21.15.12:0/2170858637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a249f0 msgr2=0x55b4e2a24e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.175+0000 7fc51ce4a740 1 --2- 172.21.15.12:0/2170858637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a249f0 0x55b4e2a24e10 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fc50c009c90 tx=0x7fc50c011810 comp rx=0 tx=0).stop 2024-03-23T23:13:37.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.175+0000 7fc51ce4a740 1 -- 172.21.15.12:0/2170858637 shutdown_connections 2024-03-23T23:13:37.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.175+0000 7fc51ce4a740 1 --2- 172.21.15.12:0/2170858637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4e2a26580 0x55b4e2a2ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.175+0000 7fc51ce4a740 1 --2- 172.21.15.12:0/2170858637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4e2a25bc0 0x55b4e2a26040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.175+0000 7fc51ce4a740 1 --2- 172.21.15.12:0/2170858637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a249f0 0x55b4e2a24e10 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.175+0000 7fc51ce4a740 1 -- 172.21.15.12:0/2170858637 >> 172.21.15.12:0/2170858637 conn(0x55b4e2a1fde0 msgr2=0x55b4e2a201e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:37.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.175+0000 7fc51ce4a740 1 -- 172.21.15.12:0/2170858637 shutdown_connections 2024-03-23T23:13:37.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.175+0000 7fc51ce4a740 1 -- 172.21.15.12:0/2170858637 wait complete. 2024-03-23T23:13:37.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.175+0000 7fc513400640 1 --2- 172.21.15.12:0/539338729 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc4f80a2800 0x7fc4f80a4cc0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fc514104810 tx=0x7fc4fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:37.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.176+0000 7fc51ce4a740 1 -- start start 2024-03-23T23:13:37.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.176+0000 7fc51ce4a740 1 Processor -- start 2024-03-23T23:13:37.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.176+0000 7fc51ce4a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4e2a249f0 0x55b4e2a2eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.176+0000 7fc51ce4a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4e2a25bc0 0x55b4e2a2f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.176+0000 7fc51ce4a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a26580 0x55b4e2a36220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.176+0000 7fc51ce4a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4e2a2d590 con 0x55b4e2a25bc0 2024-03-23T23:13:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.176+0000 7fc51ce4a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4e29450d0 con 0x55b4e2a26580 2024-03-23T23:13:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.176+0000 7fc51ce4a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4e2a2d710 con 0x55b4e2a249f0 2024-03-23T23:13:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.176+0000 7fc51ce4a740 10 client.0 _ll_register_callbacks cb 0x55b4e2949e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:13:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.177+0000 7fc513e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a26580 0x55b4e2a36220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.177+0000 7fc513e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a26580 0x55b4e2a36220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:38206/0 (socket says 172.21.15.12:38206) 2024-03-23T23:13:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.177+0000 7fc513400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4e2a249f0 0x55b4e2a2eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.177+0000 7fc513e00640 1 -- 172.21.15.12:0/3304112203 learned_addr learned my addr 172.21.15.12:0/3304112203 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.177+0000 7fc512a00640 1 --2- 172.21.15.12:0/3304112203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4e2a25bc0 0x55b4e2a2f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.177+0000 7fc513e00640 1 -- 172.21.15.12:0/3304112203 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4e2a249f0 msgr2=0x55b4e2a2eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.177+0000 7fc513e00640 1 --2- 172.21.15.12:0/3304112203 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4e2a249f0 0x55b4e2a2eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.177+0000 7fc513e00640 1 -- 172.21.15.12:0/3304112203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4e2a25bc0 msgr2=0x55b4e2a2f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.177+0000 7fc513e00640 1 --2- 172.21.15.12:0/3304112203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4e2a25bc0 0x55b4e2a2f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.177+0000 7fc513e00640 1 -- 172.21.15.12:0/3304112203 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b4e269d3f0 con 0x55b4e2a26580 2024-03-23T23:13:37.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.177+0000 7fc513e00640 1 --2- 172.21.15.12:0/3304112203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a26580 0x55b4e2a36220 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fc50c0096f0 tx=0x7fc50c0044a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f37302753984a252 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:37.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.177+0000 7fc500c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:13:37.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.178+0000 7fc500c00640 1 -- 172.21.15.12:0/3304112203 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c027020 con 0x55b4e2a26580 2024-03-23T23:13:37.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.178+0000 7fc500c00640 1 -- 172.21.15.12:0/3304112203 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fc50c005260 con 0x55b4e2a26580 2024-03-23T23:13:37.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.178+0000 7fc51ce4a740 1 -- 172.21.15.12:0/3304112203 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b4e2996800 con 0x55b4e2a26580 2024-03-23T23:13:37.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.178+0000 7fc500c00640 1 -- 172.21.15.12:0/3304112203 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc50c009900 con 0x55b4e2a26580 2024-03-23T23:13:37.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.178+0000 7fc51ce4a740 10 client.28592 fetch_fsmap learned FSMap version 23 2024-03-23T23:13:37.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.178+0000 7fc51ce4a740 1 -- 172.21.15.12:0/3304112203 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55b4e27677d0 con 0x55b4e2a26580 2024-03-23T23:13:37.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.178+0000 7fc500c00640 1 -- 172.21.15.12:0/3304112203 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b4e2996800 con 0x55b4e2a26580 2024-03-23T23:13:37.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.178+0000 7fc500c00640 1 -- 172.21.15.12:0/3304112203 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fc50c018070 con 0x55b4e2a26580 2024-03-23T23:13:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.178+0000 7fc51ce4a740 10 client.28592 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:13:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.178+0000 7fc51ce4a740 10 client.28592 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:13:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.179+0000 7fc51ce4a740 20 client.28592 populate_metadata read hostname 'smithi012' 2024-03-23T23:13:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.179+0000 7fc51ce4a740 1 --2- 172.21.15.12:0/3304112203 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55b4e2a3a470 0x55b4e2a3c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.179+0000 7fc51ce4a740 4 client.28592 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:13:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.179+0000 7fc51ce4a740 1 -- 172.21.15.12:0/3304112203 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b4e2990b50 con 0x55b4e2a3a470 2024-03-23T23:13:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.179+0000 7fc513400640 1 --2- 172.21.15.12:0/3304112203 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55b4e2a3a470 0x55b4e2a3c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.179+0000 7fc513400640 1 --2- 172.21.15.12:0/3304112203 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55b4e2a3a470 0x55b4e2a3c880 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b11a6f18388027ca server_cookie=7eada760fadacf39 in_seq=0 out_seq=0 2024-03-23T23:13:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.179+0000 7fc500c00640 10 client.28592 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:13:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.180+0000 7fc500c00640 1 -- 172.21.15.12:0/3304112203 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b4e2990b50 con 0x55b4e2a3a470 2024-03-23T23:13:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.180+0000 7fc500c00640 10 client.28592 handle_command_reply: tid=0 2024-03-23T23:13:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.180+0000 7fc51ce4a740 1 client.28592 shutdown 2024-03-23T23:13:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.180+0000 7fc51ce4a740 1 -- 172.21.15.12:0/3304112203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a26580 msgr2=0x55b4e2a36220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.180+0000 7fc51ce4a740 1 --2- 172.21.15.12:0/3304112203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a26580 0x55b4e2a36220 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fc50c0096f0 tx=0x7fc50c0044a0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.180+0000 7fc51ce4a740 1 -- 172.21.15.12:0/3304112203 shutdown_connections 2024-03-23T23:13:37.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.180+0000 7fc51ce4a740 1 --2- 172.21.15.12:0/3304112203 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55b4e2a3a470 0x55b4e2a3c880 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.181+0000 7fc51ce4a740 1 --2- 172.21.15.12:0/3304112203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a26580 0x55b4e2a36220 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.181+0000 7fc51ce4a740 1 --2- 172.21.15.12:0/3304112203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4e2a25bc0 0x55b4e2a2f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.181+0000 7fc51ce4a740 1 --2- 172.21.15.12:0/3304112203 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4e2a249f0 0x55b4e2a2eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.181+0000 7fc500c00640 0 client.28592 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:13:37.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.181+0000 7fc51ce4a740 1 -- 172.21.15.12:0/3304112203 >> 172.21.15.12:0/3304112203 conn(0x55b4e2a1fde0 msgr2=0x55b4e2a1f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:37.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.181+0000 7fc51ce4a740 1 -- 172.21.15.12:0/3304112203 shutdown_connections 2024-03-23T23:13:37.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.181+0000 7fc51ce4a740 1 -- 172.21.15.12:0/3304112203 wait complete. 2024-03-23T23:13:37.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.181+0000 7fc51ce4a740 20 client.28592 trim_cache size 0 max 16384 2024-03-23T23:13:37.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.194+0000 7fc51ce4a740 1 Processor -- start 2024-03-23T23:13:37.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.194+0000 7fc51ce4a740 1 -- start start 2024-03-23T23:13:37.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.194+0000 7fc51ce4a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4e2a249f0 0x55b4e2a1efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.194+0000 7fc51ce4a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4e2a25bc0 0x55b4e2a361e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.194+0000 7fc51ce4a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a26580 0x55b4e2a41140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.194+0000 7fc51ce4a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5141acec0 con 0x55b4e2a25bc0 2024-03-23T23:13:37.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.194+0000 7fc51ce4a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5141aa9c0 con 0x55b4e2a26580 2024-03-23T23:13:37.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.194+0000 7fc51ce4a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5141a8a60 con 0x55b4e2a249f0 2024-03-23T23:13:37.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.195+0000 7fc513400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4e2a249f0 0x55b4e2a1efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.195+0000 7fc513400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4e2a249f0 0x55b4e2a1efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44344/0 (socket says 172.21.15.12:44344) 2024-03-23T23:13:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.195+0000 7fc513400640 1 -- 172.21.15.12:0/2968415540 learned_addr learned my addr 172.21.15.12:0/2968415540 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.195+0000 7fc513e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a26580 0x55b4e2a41140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.195+0000 7fc512a00640 1 --2- 172.21.15.12:0/2968415540 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4e2a25bc0 0x55b4e2a361e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.195+0000 7fc513400640 1 -- 172.21.15.12:0/2968415540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a26580 msgr2=0x55b4e2a41140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.195+0000 7fc513400640 1 --2- 172.21.15.12:0/2968415540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a26580 0x55b4e2a41140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.195+0000 7fc513400640 1 -- 172.21.15.12:0/2968415540 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4e2a25bc0 msgr2=0x55b4e2a361e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.195+0000 7fc513400640 1 --2- 172.21.15.12:0/2968415540 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4e2a25bc0 0x55b4e2a361e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.195+0000 7fc513400640 1 -- 172.21.15.12:0/2968415540 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b4e2a2d180 con 0x55b4e2a249f0 2024-03-23T23:13:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.195+0000 7fc513400640 1 --2- 172.21.15.12:0/2968415540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4e2a249f0 0x55b4e2a1efe0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x55b4e2a3d0f0 tx=0x7fc4fc019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0b3ea1a24d0c28d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.196+0000 7fc503400640 1 -- 172.21.15.12:0/2968415540 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc0044a0 con 0x55b4e2a249f0 2024-03-23T23:13:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.196+0000 7fc503400640 1 -- 172.21.15.12:0/2968415540 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc4fc02e070 con 0x55b4e2a249f0 2024-03-23T23:13:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.196+0000 7fc503400640 1 -- 172.21.15.12:0/2968415540 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc01dab0 con 0x55b4e2a249f0 2024-03-23T23:13:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.196+0000 7fc51ce4a740 1 -- 172.21.15.12:0/2968415540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4e2a249f0 msgr2=0x55b4e2a1efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.196+0000 7fc51ce4a740 1 --2- 172.21.15.12:0/2968415540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4e2a249f0 0x55b4e2a1efe0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x55b4e2a3d0f0 tx=0x7fc4fc019000 comp rx=0 tx=0).stop 2024-03-23T23:13:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.196+0000 7fc51ce4a740 1 -- 172.21.15.12:0/2968415540 shutdown_connections 2024-03-23T23:13:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.196+0000 7fc51ce4a740 1 --2- 172.21.15.12:0/2968415540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a26580 0x55b4e2a41140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.196+0000 7fc51ce4a740 1 --2- 172.21.15.12:0/2968415540 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4e2a25bc0 0x55b4e2a361e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.196+0000 7fc51ce4a740 1 --2- 172.21.15.12:0/2968415540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4e2a249f0 0x55b4e2a1efe0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.196+0000 7fc51ce4a740 1 -- 172.21.15.12:0/2968415540 >> 172.21.15.12:0/2968415540 conn(0x55b4e2a1fde0 msgr2=0x55b4e2a1e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.196+0000 7fc51ce4a740 1 -- 172.21.15.12:0/2968415540 shutdown_connections 2024-03-23T23:13:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.196+0000 7fc51ce4a740 1 -- 172.21.15.12:0/2968415540 wait complete. 2024-03-23T23:13:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.197+0000 7fc51ce4a740 1 -- start start 2024-03-23T23:13:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.197+0000 7fc51ce4a740 1 Processor -- start 2024-03-23T23:13:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.197+0000 7fc51ce4a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4e2a249f0 0x55b4e2a32070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.197+0000 7fc51ce4a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a25bc0 0x55b4e2a325b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.197+0000 7fc51ce4a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4e2a26580 0x55b4e2a355d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.197+0000 7fc51ce4a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5141a9060 con 0x55b4e2a26580 2024-03-23T23:13:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.197+0000 7fc51ce4a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5141aa360 con 0x55b4e2a25bc0 2024-03-23T23:13:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.197+0000 7fc51ce4a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5141a96a0 con 0x55b4e2a249f0 2024-03-23T23:13:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.197+0000 7fc51ce4a740 10 client.0 _ll_register_callbacks cb 0x7fc4fc00a630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:13:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.197+0000 7fc512a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a25bc0 0x55b4e2a325b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.197+0000 7fc513400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4e2a249f0 0x55b4e2a32070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.197+0000 7fc512a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a25bc0 0x55b4e2a325b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:38232/0 (socket says 172.21.15.12:38232) 2024-03-23T23:13:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.197+0000 7fc512a00640 1 -- 172.21.15.12:0/2622520545 learned_addr learned my addr 172.21.15.12:0/2622520545 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:37.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.198+0000 7fc513e00640 1 --2- 172.21.15.12:0/2622520545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4e2a26580 0x55b4e2a355d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.198+0000 7fc512a00640 1 -- 172.21.15.12:0/2622520545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4e2a249f0 msgr2=0x55b4e2a32070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:37.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.198+0000 7fc512a00640 1 --2- 172.21.15.12:0/2622520545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4e2a249f0 0x55b4e2a32070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.198+0000 7fc512a00640 1 -- 172.21.15.12:0/2622520545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4e2a26580 msgr2=0x55b4e2a355d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:37.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.198+0000 7fc512a00640 1 --2- 172.21.15.12:0/2622520545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4e2a26580 0x55b4e2a355d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.198+0000 7fc512a00640 1 -- 172.21.15.12:0/2622520545 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b4e2a35bc0 con 0x55b4e2a25bc0 2024-03-23T23:13:37.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.198+0000 7fc513e00640 1 --2- 172.21.15.12:0/2622520545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4e2a26580 0x55b4e2a355d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:13:37.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.198+0000 7fc513400640 1 --2- 172.21.15.12:0/2622520545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4e2a249f0 0x55b4e2a32070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:37.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.198+0000 7fc512a00640 1 --2- 172.21.15.12:0/2622520545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a25bc0 0x55b4e2a325b0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fc5040b9ec0 tx=0x7fc504009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87ca7caade559f9d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:37.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.198+0000 7fc500c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:13:37.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.198+0000 7fc500c00640 1 -- 172.21.15.12:0/2622520545 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5040bd830 con 0x55b4e2a25bc0 2024-03-23T23:13:37.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.198+0000 7fc500c00640 1 -- 172.21.15.12:0/2622520545 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fc5040f4cc0 con 0x55b4e2a25bc0 2024-03-23T23:13:37.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.198+0000 7fc51ce4a740 1 -- 172.21.15.12:0/2622520545 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc4fc02e070 con 0x55b4e2a25bc0 2024-03-23T23:13:37.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.199+0000 7fc500c00640 1 -- 172.21.15.12:0/2622520545 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc5040bd3c0 con 0x55b4e2a25bc0 2024-03-23T23:13:37.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.199+0000 7fc51ce4a740 10 client.28598 fetch_fsmap learned FSMap version 23 2024-03-23T23:13:37.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.199+0000 7fc51ce4a740 1 -- 172.21.15.12:0/2622520545 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55b4e2a96520 con 0x55b4e2a25bc0 2024-03-23T23:13:37.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.199+0000 7fc500c00640 1 -- 172.21.15.12:0/2622520545 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc4fc02e070 con 0x55b4e2a25bc0 2024-03-23T23:13:37.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.199+0000 7fc500c00640 1 -- 172.21.15.12:0/2622520545 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fc50403c030 con 0x55b4e2a25bc0 2024-03-23T23:13:37.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.199+0000 7fc51ce4a740 10 client.28598 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:13:37.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.199+0000 7fc51ce4a740 10 client.28598 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-23T23:13:37.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.199+0000 7fc51ce4a740 20 client.28598 populate_metadata read hostname 'smithi012' 2024-03-23T23:13:37.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.199+0000 7fc51ce4a740 1 --2- 172.21.15.12:0/2622520545 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55b4e2a96e70 0x55b4e2a99290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.199+0000 7fc51ce4a740 4 client.28598 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:13:37.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.199+0000 7fc51ce4a740 1 -- 172.21.15.12:0/2622520545 --> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b4e290e500 con 0x55b4e2a96e70 2024-03-23T23:13:37.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.199+0000 7fc513400640 1 --2- 172.21.15.12:0/2622520545 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55b4e2a96e70 0x55b4e2a99290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.200+0000 7fc513400640 1 --2- 172.21.15.12:0/2622520545 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55b4e2a96e70 0x55b4e2a99290 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=255efaabe7bae1b4 server_cookie=65824234ff89f2b0 in_seq=0 out_seq=0 2024-03-23T23:13:37.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.200+0000 7fc500c00640 10 client.28598 ms_handle_connect on v2:172.21.15.203:6838/852477167 2024-03-23T23:13:37.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.200+0000 7fc500c00640 1 -- 172.21.15.12:0/2622520545 <== mds.2 v2:172.21.15.203:6838/852477167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b4e290e500 con 0x55b4e2a96e70 2024-03-23T23:13:37.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.200+0000 7fc500c00640 10 client.28598 handle_command_reply: tid=0 2024-03-23T23:13:37.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.200+0000 7fc51ce4a740 1 client.28598 shutdown 2024-03-23T23:13:37.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.200+0000 7fc51ce4a740 1 -- 172.21.15.12:0/2622520545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a25bc0 msgr2=0x55b4e2a325b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.200+0000 7fc51ce4a740 1 --2- 172.21.15.12:0/2622520545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a25bc0 0x55b4e2a325b0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fc5040b9ec0 tx=0x7fc504009680 comp rx=0 tx=0).stop 2024-03-23T23:13:37.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.200+0000 7fc51ce4a740 1 -- 172.21.15.12:0/2622520545 shutdown_connections 2024-03-23T23:13:37.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.200+0000 7fc51ce4a740 1 --2- 172.21.15.12:0/2622520545 >> [v2:172.21.15.203:6838/852477167,v1:172.21.15.203:6839/852477167] conn(0x55b4e2a96e70 0x55b4e2a99290 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.200+0000 7fc51ce4a740 1 --2- 172.21.15.12:0/2622520545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4e2a26580 0x55b4e2a355d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.200+0000 7fc500c00640 0 client.28598 ms_handle_reset on v2:172.21.15.203:6838/852477167 2024-03-23T23:13:37.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.200+0000 7fc51ce4a740 1 --2- 172.21.15.12:0/2622520545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4e2a25bc0 0x55b4e2a325b0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.200+0000 7fc51ce4a740 1 --2- 172.21.15.12:0/2622520545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4e2a249f0 0x55b4e2a32070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.200+0000 7fc51ce4a740 1 -- 172.21.15.12:0/2622520545 >> 172.21.15.12:0/2622520545 conn(0x55b4e2a1fde0 msgr2=0x55b4e2a1f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:37.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.200+0000 7fc51ce4a740 1 -- 172.21.15.12:0/2622520545 shutdown_connections 2024-03-23T23:13:37.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.200+0000 7fc51ce4a740 1 -- 172.21.15.12:0/2622520545 wait complete. 2024-03-23T23:13:37.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.200+0000 7fc51ce4a740 20 client.28598 trim_cache size 0 max 16384 2024-03-23T23:13:37.205 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:13:37.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.201+0000 7fc518c00640 1 -- 172.21.15.12:0/539338729 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc4f80a2800 msgr2=0x7fc4f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.201+0000 7fc518c00640 1 --2- 172.21.15.12:0/539338729 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc4f80a2800 0x7fc4f80a4cc0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fc514104810 tx=0x7fc4fc005f90 comp rx=0 tx=0).stop 2024-03-23T23:13:37.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.201+0000 7fc518c00640 1 -- 172.21.15.12:0/539338729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5141049b0 msgr2=0x7fc514199f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.201+0000 7fc518c00640 1 --2- 172.21.15.12:0/539338729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5141049b0 0x7fc514199f70 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fc5040325f0 tx=0x7fc504035d10 comp rx=0 tx=0).stop 2024-03-23T23:13:37.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.201+0000 7fc518c00640 1 -- 172.21.15.12:0/539338729 shutdown_connections 2024-03-23T23:13:37.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.201+0000 7fc518c00640 1 --2- 172.21.15.12:0/539338729 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fc4f80a2800 0x7fc4f80a4cc0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.201+0000 7fc518c00640 1 --2- 172.21.15.12:0/539338729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc514105370 0x7fc5141a0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.201+0000 7fc518c00640 1 --2- 172.21.15.12:0/539338729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5141049b0 0x7fc514199f70 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.201+0000 7fc518c00640 1 --2- 172.21.15.12:0/539338729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5141037c0 0x7fc514199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.201+0000 7fc518c00640 1 -- 172.21.15.12:0/539338729 >> 172.21.15.12:0/539338729 conn(0x7fc5140fed50 msgr2=0x7fc514107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:37.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.201+0000 7fc518c00640 1 -- 172.21.15.12:0/539338729 shutdown_connections 2024-03-23T23:13:37.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.201+0000 7fc518c00640 1 -- 172.21.15.12:0/539338729 wait complete. 2024-03-23T23:13:37.216 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:13:37.216 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-23T23:13:37.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.436+0000 7f7d23200640 1 -- 172.21.15.12:0/3742669976 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d24102770 msgr2=0x7f7d24102bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.436+0000 7f7d23200640 1 --2- 172.21.15.12:0/3742669976 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d24102770 0x7f7d24102bf0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7d10009900 tx=0x7f7d1002bc50 comp rx=0 tx=0).stop 2024-03-23T23:13:37.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.436+0000 7f7d23200640 1 -- 172.21.15.12:0/3742669976 shutdown_connections 2024-03-23T23:13:37.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.436+0000 7f7d23200640 1 --2- 172.21.15.12:0/3742669976 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d24103130 0x7f7d241099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.436+0000 7f7d23200640 1 --2- 172.21.15.12:0/3742669976 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d24102770 0x7f7d24102bf0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.436+0000 7f7d23200640 1 --2- 172.21.15.12:0/3742669976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d24101580 0x7f7d241019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.436+0000 7f7d23200640 1 -- 172.21.15.12:0/3742669976 >> 172.21.15.12:0/3742669976 conn(0x7f7d240fcb10 msgr2=0x7f7d240fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:37.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.436+0000 7f7d23200640 1 -- 172.21.15.12:0/3742669976 shutdown_connections 2024-03-23T23:13:37.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.436+0000 7f7d23200640 1 -- 172.21.15.12:0/3742669976 wait complete. 2024-03-23T23:13:37.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.437+0000 7f7d23200640 1 Processor -- start 2024-03-23T23:13:37.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.437+0000 7f7d23200640 1 -- start start 2024-03-23T23:13:37.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.437+0000 7f7d23200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d24101580 0x7f7d24197880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.437+0000 7f7d23200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d24102770 0x7f7d24197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.437+0000 7f7d23200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d24103130 0x7f7d2419ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.437+0000 7f7d23200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2410d230 con 0x7f7d24102770 2024-03-23T23:13:37.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.437+0000 7f7d23200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2410d0b0 con 0x7f7d24101580 2024-03-23T23:13:37.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.437+0000 7f7d23200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2410d3b0 con 0x7f7d24103130 2024-03-23T23:13:37.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.438+0000 7f7d22800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d24103130 0x7f7d2419ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.438+0000 7f7d21e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d24101580 0x7f7d24197880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.438+0000 7f7d22800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d24103130 0x7f7d2419ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44368/0 (socket says 172.21.15.12:44368) 2024-03-23T23:13:37.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.438+0000 7f7d22800640 1 -- 172.21.15.12:0/2436779509 learned_addr learned my addr 172.21.15.12:0/2436779509 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:37.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.438+0000 7f7d21400640 1 --2- 172.21.15.12:0/2436779509 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d24102770 0x7f7d24197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.438+0000 7f7d22800640 1 -- 172.21.15.12:0/2436779509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d24101580 msgr2=0x7f7d24197880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.438+0000 7f7d22800640 1 --2- 172.21.15.12:0/2436779509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d24101580 0x7f7d24197880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.438+0000 7f7d22800640 1 -- 172.21.15.12:0/2436779509 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d24102770 msgr2=0x7f7d24197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.438+0000 7f7d22800640 1 --2- 172.21.15.12:0/2436779509 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d24102770 0x7f7d24197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.438+0000 7f7d22800640 1 -- 172.21.15.12:0/2436779509 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2419f4a0 con 0x7f7d24103130 2024-03-23T23:13:37.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.438+0000 7f7d22800640 1 --2- 172.21.15.12:0/2436779509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d24103130 0x7f7d2419ee30 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c009fd0 tx=0x7f7d1c00b260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:37.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.439+0000 7f7d1aa00640 1 -- 172.21.15.12:0/2436779509 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c017070 con 0x7f7d24103130 2024-03-23T23:13:37.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.439+0000 7f7d1aa00640 1 -- 172.21.15.12:0/2436779509 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d1c003e50 con 0x7f7d24103130 2024-03-23T23:13:37.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.439+0000 7f7d1aa00640 1 -- 172.21.15.12:0/2436779509 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c0106c0 con 0x7f7d24103130 2024-03-23T23:13:37.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.439+0000 7f7d23200640 1 -- 172.21.15.12:0/2436779509 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d2419f790 con 0x7f7d24103130 2024-03-23T23:13:37.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.439+0000 7f7d23200640 1 -- 172.21.15.12:0/2436779509 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d2419fc60 con 0x7f7d24103130 2024-03-23T23:13:37.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.442+0000 7f7d2bfe2740 1 Processor -- start 2024-03-23T23:13:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.442+0000 7f7d1aa00640 1 -- 172.21.15.12:0/2436779509 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7d1c003970 con 0x7f7d24103130 2024-03-23T23:13:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.442+0000 7f7d2bfe2740 1 -- start start 2024-03-23T23:13:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.442+0000 7f7d2bfe2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ec4810e00 0x561ec4811220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.442+0000 7f7d2bfe2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ec4811fd0 0x561ec4812450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.442+0000 7f7d2bfe2740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4812990 0x561ec4819220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.442+0000 7f7d2bfe2740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ec47712c0 con 0x561ec4812990 2024-03-23T23:13:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.442+0000 7f7d2bfe2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ec4719c90 con 0x561ec4811fd0 2024-03-23T23:13:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.442+0000 7f7d2bfe2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ec476d360 con 0x561ec4810e00 2024-03-23T23:13:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.443+0000 7f7d21e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ec4811fd0 0x561ec4812450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.443+0000 7f7d21400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ec4810e00 0x561ec4811220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.443+0000 7f7d21e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ec4811fd0 0x561ec4812450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:38262/0 (socket says 172.21.15.12:38262) 2024-03-23T23:13:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.443+0000 7f7d21e00640 1 -- 172.21.15.12:0/517752866 learned_addr learned my addr 172.21.15.12:0/517752866 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.443+0000 7f7d21400640 1 --2- 172.21.15.12:0/517752866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4812990 0x561ec4819220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.443+0000 7f7d21e00640 1 -- 172.21.15.12:0/517752866 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ec4810e00 msgr2=0x561ec4811220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.443+0000 7f7d21e00640 1 --2- 172.21.15.12:0/517752866 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ec4810e00 0x561ec4811220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.443+0000 7f7d21e00640 1 -- 172.21.15.12:0/517752866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4812990 msgr2=0x561ec4819220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.443+0000 7f7d21e00640 1 --2- 172.21.15.12:0/517752866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4812990 0x561ec4819220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.443+0000 7f7d21e00640 1 -- 172.21.15.12:0/517752866 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ec473e080 con 0x561ec4811fd0 2024-03-23T23:13:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.443+0000 7f7d21400640 1 --2- 172.21.15.12:0/517752866 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ec4810e00 0x561ec4811220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:13:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.443+0000 7f7d21e00640 1 --2- 172.21.15.12:0/517752866 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ec4811fd0 0x561ec4812450 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f7d14009c80 tx=0x7f7d14011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdaca67aad1be04e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.443+0000 7f7d1aa00640 1 --2- 172.21.15.12:0/2436779509 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7d080a2880 0x7f7d080a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.444+0000 7f7d0f400640 1 -- 172.21.15.12:0/517752866 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1401e070 con 0x561ec4811fd0 2024-03-23T23:13:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.444+0000 7f7d0f400640 1 -- 172.21.15.12:0/517752866 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d14009e30 con 0x561ec4811fd0 2024-03-23T23:13:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.444+0000 7f7d0f400640 1 -- 172.21.15.12:0/517752866 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1400ea20 con 0x561ec4811fd0 2024-03-23T23:13:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.444+0000 7f7d1aa00640 1 -- 172.21.15.12:0/2436779509 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7d1c0ce730 con 0x7f7d24103130 2024-03-23T23:13:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.444+0000 7f7d21400640 1 --2- 172.21.15.12:0/2436779509 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7d080a2880 0x7f7d080a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.444+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/517752866 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ec4811fd0 msgr2=0x561ec4812450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.444+0000 7f7d2bfe2740 1 --2- 172.21.15.12:0/517752866 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ec4811fd0 0x561ec4812450 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f7d14009c80 tx=0x7f7d14011810 comp rx=0 tx=0).stop 2024-03-23T23:13:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.444+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/517752866 shutdown_connections 2024-03-23T23:13:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.444+0000 7f7d2bfe2740 1 --2- 172.21.15.12:0/517752866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4812990 0x561ec4819220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.444+0000 7f7d2bfe2740 1 --2- 172.21.15.12:0/517752866 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ec4811fd0 0x561ec4812450 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.444+0000 7f7d2bfe2740 1 --2- 172.21.15.12:0/517752866 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ec4810e00 0x561ec4811220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.444+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/517752866 >> 172.21.15.12:0/517752866 conn(0x561ec4807de0 msgr2=0x561ec48081e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.444+0000 7f7d21400640 1 --2- 172.21.15.12:0/2436779509 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7d080a2880 0x7f7d080a4d40 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7d1002c260 tx=0x7f7d10000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.445+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/517752866 shutdown_connections 2024-03-23T23:13:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.445+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/517752866 wait complete. 2024-03-23T23:13:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.445+0000 7f7d2bfe2740 1 -- start start 2024-03-23T23:13:37.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.445+0000 7f7d2bfe2740 1 Processor -- start 2024-03-23T23:13:37.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.446+0000 7f7d2bfe2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ec4810e00 0x561ec481b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.446+0000 7f7d2bfe2740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4811fd0 0x561ec481b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.446+0000 7f7d2bfe2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ec4812990 0x561ec48225c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.446+0000 7f7d2bfe2740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ec48199a0 con 0x561ec4811fd0 2024-03-23T23:13:37.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.446+0000 7f7d2bfe2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ec472d0d0 con 0x561ec4812990 2024-03-23T23:13:37.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.446+0000 7f7d2bfe2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ec4819b20 con 0x561ec4810e00 2024-03-23T23:13:37.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.446+0000 7f7d2bfe2740 10 client.0 _ll_register_callbacks cb 0x561ec4731e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:13:37.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.446+0000 7f7d21400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4811fd0 0x561ec481b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.446+0000 7f7d21400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4811fd0 0x561ec481b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57240/0 (socket says 172.21.15.12:57240) 2024-03-23T23:13:37.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.446+0000 7f7d21400640 1 -- 172.21.15.12:0/1760546683 learned_addr learned my addr 172.21.15.12:0/1760546683 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:37.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.446+0000 7f7d21400640 1 -- 172.21.15.12:0/1760546683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ec4810e00 msgr2=0x561ec481b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:37.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.446+0000 7f7d21400640 1 --2- 172.21.15.12:0/1760546683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ec4810e00 0x561ec481b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.446+0000 7f7d21400640 1 -- 172.21.15.12:0/1760546683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ec4812990 msgr2=0x561ec48225c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:37.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.446+0000 7f7d22800640 1 --2- 172.21.15.12:0/1760546683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ec4812990 0x561ec48225c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.446+0000 7f7d21400640 1 --2- 172.21.15.12:0/1760546683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ec4812990 0x561ec48225c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.446+0000 7f7d21400640 1 -- 172.21.15.12:0/1760546683 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561ec44853f0 con 0x561ec4811fd0 2024-03-23T23:13:37.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.446+0000 7f7d22800640 1 --2- 172.21.15.12:0/1760546683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ec4812990 0x561ec48225c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T23:13:37.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.447+0000 7f7d21400640 1 --2- 172.21.15.12:0/1760546683 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4811fd0 0x561ec481b580 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7d100096c0 tx=0x7f7d10005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2508272de06909f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:37.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.447+0000 7f7d03e00640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:13:37.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.447+0000 7f7d03e00640 1 -- 172.21.15.12:0/1760546683 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1002cd50 con 0x561ec4811fd0 2024-03-23T23:13:37.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.447+0000 7f7d03e00640 1 -- 172.21.15.12:0/1760546683 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7d10004ed0 con 0x561ec4811fd0 2024-03-23T23:13:37.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.447+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/1760546683 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7d14009e30 con 0x561ec4811fd0 2024-03-23T23:13:37.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.447+0000 7f7d03e00640 1 -- 172.21.15.12:0/1760546683 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d100050d0 con 0x561ec4811fd0 2024-03-23T23:13:37.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.447+0000 7f7d2bfe2740 10 client.18483 fetch_fsmap learned FSMap version 23 2024-03-23T23:13:37.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.447+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/1760546683 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561ec454f7d0 con 0x561ec4811fd0 2024-03-23T23:13:37.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.448+0000 7f7d03e00640 1 -- 172.21.15.12:0/1760546683 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d14009e30 con 0x561ec4811fd0 2024-03-23T23:13:37.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.448+0000 7f7d03e00640 1 -- 172.21.15.12:0/1760546683 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f7d10046070 con 0x561ec4811fd0 2024-03-23T23:13:37.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.448+0000 7f7d2bfe2740 10 client.18483 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:13:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.448+0000 7f7d2bfe2740 10 client.18483 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:13:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.448+0000 7f7d2bfe2740 20 client.18483 populate_metadata read hostname 'smithi012' 2024-03-23T23:13:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.448+0000 7f7d2bfe2740 1 --2- 172.21.15.12:0/1760546683 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x561ec4826ad0 0x561ec4828f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.448+0000 7f7d2bfe2740 4 client.18483 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:13:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.448+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/1760546683 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561ec477e800 con 0x561ec4826ad0 2024-03-23T23:13:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.449+0000 7f7d21e00640 1 --2- 172.21.15.12:0/1760546683 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x561ec4826ad0 0x561ec4828f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.450+0000 7f7d21e00640 1 --2- 172.21.15.12:0/1760546683 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x561ec4826ad0 0x561ec4828f90 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bb511f79289112f4 server_cookie=f76f7b859272d508 in_seq=0 out_seq=0 2024-03-23T23:13:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.450+0000 7f7d03e00640 10 client.18483 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:13:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.451+0000 7f7d03e00640 1 -- 172.21.15.12:0/1760546683 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561ec477e800 con 0x561ec4826ad0 2024-03-23T23:13:37.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.451+0000 7f7d03e00640 10 client.18483 handle_command_reply: tid=0 2024-03-23T23:13:37.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.451+0000 7f7d2bfe2740 1 client.18483 shutdown 2024-03-23T23:13:37.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.451+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/1760546683 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4811fd0 msgr2=0x561ec481b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.451+0000 7f7d2bfe2740 1 --2- 172.21.15.12:0/1760546683 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4811fd0 0x561ec481b580 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7d100096c0 tx=0x7f7d10005fd0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.451+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/1760546683 shutdown_connections 2024-03-23T23:13:37.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.451+0000 7f7d2bfe2740 1 --2- 172.21.15.12:0/1760546683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ec4812990 0x561ec48225c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.451+0000 7f7d2bfe2740 1 --2- 172.21.15.12:0/1760546683 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4811fd0 0x561ec481b580 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.451+0000 7f7d2bfe2740 1 --2- 172.21.15.12:0/1760546683 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x561ec4826ad0 0x561ec4828f90 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.451+0000 7f7d2bfe2740 1 --2- 172.21.15.12:0/1760546683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ec4810e00 0x561ec481b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.451+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/1760546683 >> 172.21.15.12:0/1760546683 conn(0x561ec4807de0 msgr2=0x561ec4807160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:37.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.451+0000 7f7d03e00640 0 client.18483 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:13:37.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.451+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/1760546683 shutdown_connections 2024-03-23T23:13:37.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.451+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/1760546683 wait complete. 2024-03-23T23:13:37.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.452+0000 7f7d2bfe2740 20 client.18483 trim_cache size 0 max 16384 2024-03-23T23:13:37.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.465+0000 7f7d2bfe2740 1 Processor -- start 2024-03-23T23:13:37.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.465+0000 7f7d2bfe2740 1 -- start start 2024-03-23T23:13:37.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.466+0000 7f7d2bfe2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ec4810e00 0x561ec4815070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.466+0000 7f7d2bfe2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ec4811fd0 0x561ec4822580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.466+0000 7f7d2bfe2740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4812990 0x561ec48155b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.466+0000 7f7d2bfe2740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d241a8aa0 con 0x561ec4812990 2024-03-23T23:13:37.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.466+0000 7f7d2bfe2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d241a65a0 con 0x561ec4810e00 2024-03-23T23:13:37.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.466+0000 7f7d2bfe2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d241a4640 con 0x561ec4811fd0 2024-03-23T23:13:37.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.466+0000 7f7d22800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4812990 0x561ec48155b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.466+0000 7f7d22800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4812990 0x561ec48155b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57256/0 (socket says 172.21.15.12:57256) 2024-03-23T23:13:37.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.466+0000 7f7d22800640 1 -- 172.21.15.12:0/2440403392 learned_addr learned my addr 172.21.15.12:0/2440403392 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:37.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.466+0000 7f7d22800640 1 -- 172.21.15.12:0/2440403392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ec4811fd0 msgr2=0x561ec4822580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:37.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.466+0000 7f7d22800640 1 --2- 172.21.15.12:0/2440403392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ec4811fd0 0x561ec4822580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.466+0000 7f7d22800640 1 -- 172.21.15.12:0/2440403392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ec4810e00 msgr2=0x561ec4815070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:37.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.466+0000 7f7d22800640 1 --2- 172.21.15.12:0/2440403392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ec4810e00 0x561ec4815070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.466+0000 7f7d22800640 1 -- 172.21.15.12:0/2440403392 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ec4818b30 con 0x561ec4812990 2024-03-23T23:13:37.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.467+0000 7f7d22800640 1 --2- 172.21.15.12:0/2440403392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4812990 0x561ec48155b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c00b760 tx=0x7f7d1c09ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a293b05b11f23f42 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:37.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.467+0000 7f7d0ea00640 1 -- 172.21.15.12:0/2440403392 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c020070 con 0x561ec4812990 2024-03-23T23:13:37.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.467+0000 7f7d0ea00640 1 -- 172.21.15.12:0/2440403392 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d1c0a3820 con 0x561ec4812990 2024-03-23T23:13:37.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.467+0000 7f7d0ea00640 1 -- 172.21.15.12:0/2440403392 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c0041b0 con 0x561ec4812990 2024-03-23T23:13:37.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.468+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/2440403392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4812990 msgr2=0x561ec48155b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.468+0000 7f7d2bfe2740 1 --2- 172.21.15.12:0/2440403392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4812990 0x561ec48155b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c00b760 tx=0x7f7d1c09ad30 comp rx=0 tx=0).stop 2024-03-23T23:13:37.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.468+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/2440403392 shutdown_connections 2024-03-23T23:13:37.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.468+0000 7f7d2bfe2740 1 --2- 172.21.15.12:0/2440403392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4812990 0x561ec48155b0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.468+0000 7f7d2bfe2740 1 --2- 172.21.15.12:0/2440403392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ec4811fd0 0x561ec4822580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.468+0000 7f7d2bfe2740 1 --2- 172.21.15.12:0/2440403392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ec4810e00 0x561ec4815070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.468+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/2440403392 >> 172.21.15.12:0/2440403392 conn(0x561ec4807de0 msgr2=0x561ec480b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:37.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.468+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/2440403392 shutdown_connections 2024-03-23T23:13:37.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.468+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/2440403392 wait complete. 2024-03-23T23:13:37.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.468+0000 7f7d2bfe2740 1 -- start start 2024-03-23T23:13:37.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.468+0000 7f7d2bfe2740 1 Processor -- start 2024-03-23T23:13:37.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.469+0000 7f7d2bfe2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ec4810e00 0x561ec48220d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.469+0000 7f7d2bfe2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ec4811fd0 0x561ec481eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.469+0000 7f7d2bfe2740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4812990 0x561ec481f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.469+0000 7f7d2bfe2740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d241a4c40 con 0x561ec4812990 2024-03-23T23:13:37.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.469+0000 7f7d2bfe2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d241a5f40 con 0x561ec4810e00 2024-03-23T23:13:37.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.469+0000 7f7d2bfe2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d241a5280 con 0x561ec4811fd0 2024-03-23T23:13:37.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.469+0000 7f7d2bfe2740 10 client.0 _ll_register_callbacks cb 0x7f7d10009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:13:37.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.469+0000 7f7d22800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4812990 0x561ec481f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.469+0000 7f7d22800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4812990 0x561ec481f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57272/0 (socket says 172.21.15.12:57272) 2024-03-23T23:13:37.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.469+0000 7f7d22800640 1 -- 172.21.15.12:0/3788977289 learned_addr learned my addr 172.21.15.12:0/3788977289 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:37.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.469+0000 7f7d21e00640 1 --2- 172.21.15.12:0/3788977289 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ec4810e00 0x561ec48220d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.469+0000 7f7d22800640 1 -- 172.21.15.12:0/3788977289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ec4811fd0 msgr2=0x561ec481eb70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-23T23:13:37.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.469+0000 7f7d22800640 1 --2- 172.21.15.12:0/3788977289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ec4811fd0 0x561ec481eb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.469+0000 7f7d22800640 1 -- 172.21.15.12:0/3788977289 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ec4810e00 msgr2=0x561ec48220d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:37.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.469+0000 7f7d22800640 1 --2- 172.21.15.12:0/3788977289 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ec4810e00 0x561ec48220d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.469+0000 7f7d22800640 1 -- 172.21.15.12:0/3788977289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561ec4822700 con 0x561ec4812990 2024-03-23T23:13:37.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.470+0000 7f7d22800640 1 --2- 172.21.15.12:0/3788977289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4812990 0x561ec481f0b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c09ade0 tx=0x7f7d1c09f310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d1578ac64f586d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:37.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.470+0000 7f7d03e00640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-23T23:13:37.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.470+0000 7f7d03e00640 1 -- 172.21.15.12:0/3788977289 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c020070 con 0x561ec4812990 2024-03-23T23:13:37.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.470+0000 7f7d03e00640 1 -- 172.21.15.12:0/3788977289 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7d1c0b9d80 con 0x561ec4812990 2024-03-23T23:13:37.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.470+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/3788977289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7d1c0a3820 con 0x561ec4812990 2024-03-23T23:13:37.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.470+0000 7f7d03e00640 1 -- 172.21.15.12:0/3788977289 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d1c099430 con 0x561ec4812990 2024-03-23T23:13:37.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.470+0000 7f7d2bfe2740 10 client.18489 fetch_fsmap learned FSMap version 23 2024-03-23T23:13:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.470+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/3788977289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561ec4883390 con 0x561ec4812990 2024-03-23T23:13:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.470+0000 7f7d03e00640 1 -- 172.21.15.12:0/3788977289 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d1c0a3820 con 0x561ec4812990 2024-03-23T23:13:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.470+0000 7f7d03e00640 1 -- 172.21.15.12:0/3788977289 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f7d1c099ff0 con 0x561ec4812990 2024-03-23T23:13:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.471+0000 7f7d2bfe2740 10 client.18489 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:13:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.471+0000 7f7d2bfe2740 10 client.18489 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-23T23:13:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.471+0000 7f7d2bfe2740 20 client.18489 populate_metadata read hostname 'smithi012' 2024-03-23T23:13:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.471+0000 7f7d2bfe2740 1 --2- 172.21.15.12:0/3788977289 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x561ec4883d40 0x561ec4886160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.471+0000 7f7d2bfe2740 4 client.18489 mds_command: new command op to 24554 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:13:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.471+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/3788977289 --> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561ec4778b50 con 0x561ec4883d40 2024-03-23T23:13:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.471+0000 7f7d21e00640 1 --2- 172.21.15.12:0/3788977289 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x561ec4883d40 0x561ec4886160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.471+0000 7f7d21e00640 1 --2- 172.21.15.12:0/3788977289 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x561ec4883d40 0x561ec4886160 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5d860e8da2344594 server_cookie=fd4c8fff65d6be4 in_seq=0 out_seq=0 2024-03-23T23:13:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.471+0000 7f7d03e00640 10 client.18489 ms_handle_connect on v2:172.21.15.173:6836/2648089141 2024-03-23T23:13:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.472+0000 7f7d03e00640 1 -- 172.21.15.12:0/3788977289 <== mds.1 v2:172.21.15.173:6836/2648089141 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561ec4778b50 con 0x561ec4883d40 2024-03-23T23:13:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.472+0000 7f7d03e00640 10 client.18489 handle_command_reply: tid=0 2024-03-23T23:13:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.472+0000 7f7d2bfe2740 1 client.18489 shutdown 2024-03-23T23:13:37.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.472+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/3788977289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4812990 msgr2=0x561ec481f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.472+0000 7f7d2bfe2740 1 --2- 172.21.15.12:0/3788977289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4812990 0x561ec481f0b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c09ade0 tx=0x7f7d1c09f310 comp rx=0 tx=0).stop 2024-03-23T23:13:37.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.472+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/3788977289 shutdown_connections 2024-03-23T23:13:37.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.472+0000 7f7d2bfe2740 1 --2- 172.21.15.12:0/3788977289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561ec4812990 0x561ec481f0b0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.472+0000 7f7d2bfe2740 1 --2- 172.21.15.12:0/3788977289 >> [v2:172.21.15.173:6836/2648089141,v1:172.21.15.173:6837/2648089141] conn(0x561ec4883d40 0x561ec4886160 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.472+0000 7f7d2bfe2740 1 --2- 172.21.15.12:0/3788977289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ec4811fd0 0x561ec481eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.472+0000 7f7d2bfe2740 1 --2- 172.21.15.12:0/3788977289 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ec4810e00 0x561ec48220d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.472+0000 7f7d03e00640 0 client.18489 ms_handle_reset on v2:172.21.15.173:6836/2648089141 2024-03-23T23:13:37.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.472+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/3788977289 >> 172.21.15.12:0/3788977289 conn(0x561ec4807de0 msgr2=0x561ec48068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:37.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.472+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/3788977289 shutdown_connections 2024-03-23T23:13:37.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.472+0000 7f7d2bfe2740 1 -- 172.21.15.12:0/3788977289 wait complete. 2024-03-23T23:13:37.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.472+0000 7f7d2bfe2740 20 client.18489 trim_cache size 0 max 16384 2024-03-23T23:13:37.474 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:13:37.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.473+0000 7f7d23200640 1 -- 172.21.15.12:0/2436779509 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7d080a2880 msgr2=0x7f7d080a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.473+0000 7f7d23200640 1 --2- 172.21.15.12:0/2436779509 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7d080a2880 0x7f7d080a4d40 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7d1002c260 tx=0x7f7d10000bc0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.473+0000 7f7d23200640 1 -- 172.21.15.12:0/2436779509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d24103130 msgr2=0x7f7d2419ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.473+0000 7f7d23200640 1 --2- 172.21.15.12:0/2436779509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d24103130 0x7f7d2419ee30 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c009fd0 tx=0x7f7d1c00b260 comp rx=0 tx=0).stop 2024-03-23T23:13:37.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.473+0000 7f7d23200640 1 -- 172.21.15.12:0/2436779509 shutdown_connections 2024-03-23T23:13:37.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.473+0000 7f7d23200640 1 --2- 172.21.15.12:0/2436779509 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7d080a2880 0x7f7d080a4d40 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.473+0000 7f7d23200640 1 --2- 172.21.15.12:0/2436779509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d24103130 0x7f7d2419ee30 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.473+0000 7f7d23200640 1 --2- 172.21.15.12:0/2436779509 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d24102770 0x7f7d24197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.473+0000 7f7d23200640 1 --2- 172.21.15.12:0/2436779509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d24101580 0x7f7d24197880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.473+0000 7f7d23200640 1 -- 172.21.15.12:0/2436779509 >> 172.21.15.12:0/2436779509 conn(0x7f7d240fcb10 msgr2=0x7f7d24105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:37.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.473+0000 7f7d23200640 1 -- 172.21.15.12:0/2436779509 shutdown_connections 2024-03-23T23:13:37.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.473+0000 7f7d23200640 1 -- 172.21.15.12:0/2436779509 wait complete. 2024-03-23T23:13:37.488 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-23T23:13:37.488 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-23T23:13:37.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.705+0000 7ff977200640 1 -- 172.21.15.12:0/862945146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff970107bc0 msgr2=0x7ff970107fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.705+0000 7ff977200640 1 --2- 172.21.15.12:0/862945146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff970107bc0 0x7ff970107fe0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff964009900 tx=0x7ff96402bc40 comp rx=0 tx=0).stop 2024-03-23T23:13:37.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.705+0000 7ff977200640 1 -- 172.21.15.12:0/862945146 shutdown_connections 2024-03-23T23:13:37.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.705+0000 7ff977200640 1 --2- 172.21.15.12:0/862945146 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff970109770 0x7ff970110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.705+0000 7ff977200640 1 --2- 172.21.15.12:0/862945146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff970108db0 0x7ff970109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.705+0000 7ff977200640 1 --2- 172.21.15.12:0/862945146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff970107bc0 0x7ff970107fe0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.705+0000 7ff977200640 1 -- 172.21.15.12:0/862945146 >> 172.21.15.12:0/862945146 conn(0x7ff970076b70 msgr2=0x7ff970078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:37.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.705+0000 7ff977200640 1 -- 172.21.15.12:0/862945146 shutdown_connections 2024-03-23T23:13:37.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.705+0000 7ff977200640 1 -- 172.21.15.12:0/862945146 wait complete. 2024-03-23T23:13:37.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.706+0000 7ff977200640 1 Processor -- start 2024-03-23T23:13:37.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.706+0000 7ff977200640 1 -- start start 2024-03-23T23:13:37.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.707+0000 7ff977200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff970107bc0 0x7ff97019ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.707+0000 7ff977200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff970108db0 0x7ff97019e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.707+0000 7ff977200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff970109770 0x7ff9701a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.707+0000 7ff977200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff970113860 con 0x7ff970109770 2024-03-23T23:13:37.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.707+0000 7ff977200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9701136e0 con 0x7ff970108db0 2024-03-23T23:13:37.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.707+0000 7ff977200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9701139e0 con 0x7ff970107bc0 2024-03-23T23:13:37.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.707+0000 7ff975e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff970107bc0 0x7ff97019ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.707+0000 7ff975400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff970108db0 0x7ff97019e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.707+0000 7ff975e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff970107bc0 0x7ff97019ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44420/0 (socket says 172.21.15.12:44420) 2024-03-23T23:13:37.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.707+0000 7ff975e00640 1 -- 172.21.15.12:0/2844051478 learned_addr learned my addr 172.21.15.12:0/2844051478 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:37.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.707+0000 7ff976800640 1 --2- 172.21.15.12:0/2844051478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff970109770 0x7ff9701a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.707+0000 7ff975e00640 1 -- 172.21.15.12:0/2844051478 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff970108db0 msgr2=0x7ff97019e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.707+0000 7ff975e00640 1 --2- 172.21.15.12:0/2844051478 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff970108db0 0x7ff97019e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.707+0000 7ff975e00640 1 -- 172.21.15.12:0/2844051478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff970109770 msgr2=0x7ff9701a5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.707+0000 7ff975e00640 1 --2- 172.21.15.12:0/2844051478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff970109770 0x7ff9701a5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.707+0000 7ff975e00640 1 -- 172.21.15.12:0/2844051478 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9701a5af0 con 0x7ff970107bc0 2024-03-23T23:13:37.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.707+0000 7ff975400640 1 --2- 172.21.15.12:0/2844051478 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff970108db0 0x7ff97019e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:37.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.707+0000 7ff976800640 1 --2- 172.21.15.12:0/2844051478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff970109770 0x7ff9701a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:13:37.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.708+0000 7ff975e00640 1 --2- 172.21.15.12:0/2844051478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff970107bc0 0x7ff97019ded0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7ff96402efd0 tx=0x7ff96402c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:37.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.708+0000 7ff96aa00640 1 -- 172.21.15.12:0/2844051478 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff964041070 con 0x7ff970107bc0 2024-03-23T23:13:37.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.708+0000 7ff96aa00640 1 -- 172.21.15.12:0/2844051478 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff964004450 con 0x7ff970107bc0 2024-03-23T23:13:37.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.708+0000 7ff977200640 1 -- 172.21.15.12:0/2844051478 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9701a5d80 con 0x7ff970107bc0 2024-03-23T23:13:37.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.708+0000 7ff96aa00640 1 -- 172.21.15.12:0/2844051478 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff964003e50 con 0x7ff970107bc0 2024-03-23T23:13:37.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.708+0000 7ff977200640 1 -- 172.21.15.12:0/2844051478 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9701a6190 con 0x7ff970107bc0 2024-03-23T23:13:37.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.711+0000 7ff97b440740 1 Processor -- start 2024-03-23T23:13:37.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.711+0000 7ff97b440740 1 -- start start 2024-03-23T23:13:37.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.711+0000 7ff96aa00640 1 -- 172.21.15.12:0/2844051478 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff964044070 con 0x7ff970107bc0 2024-03-23T23:13:37.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.712+0000 7ff97b440740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a09f0 0x55c14b7a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.712+0000 7ff97b440740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c14b7a1bc0 0x55c14b7a2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.712+0000 7ff97b440740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c14b7a2580 0x55c14b7a8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.712+0000 7ff97b440740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c14b7052c0 con 0x55c14b7a2580 2024-03-23T23:13:37.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.712+0000 7ff97b440740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c14b6adc90 con 0x55c14b7a09f0 2024-03-23T23:13:37.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.712+0000 7ff97b440740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c14b701360 con 0x55c14b7a1bc0 2024-03-23T23:13:37.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.712+0000 7ff976800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a09f0 0x55c14b7a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.712+0000 7ff976800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a09f0 0x55c14b7a0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:38328/0 (socket says 172.21.15.12:38328) 2024-03-23T23:13:37.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.712+0000 7ff976800640 1 -- 172.21.15.12:0/2689863568 learned_addr learned my addr 172.21.15.12:0/2689863568 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:37.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.712+0000 7ff975e00640 1 --2- 172.21.15.12:0/2689863568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c14b7a1bc0 0x55c14b7a2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.712+0000 7ff975400640 1 --2- 172.21.15.12:0/2689863568 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c14b7a2580 0x55c14b7a8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.712+0000 7ff976800640 1 -- 172.21.15.12:0/2689863568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c14b7a1bc0 msgr2=0x55c14b7a2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:37.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.712+0000 7ff976800640 1 --2- 172.21.15.12:0/2689863568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c14b7a1bc0 0x55c14b7a2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.712+0000 7ff976800640 1 -- 172.21.15.12:0/2689863568 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c14b7a2580 msgr2=0x55c14b7a8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:37.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.712+0000 7ff976800640 1 --2- 172.21.15.12:0/2689863568 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c14b7a2580 0x55c14b7a8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.712+0000 7ff976800640 1 -- 172.21.15.12:0/2689863568 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c14b6d2080 con 0x55c14b7a09f0 2024-03-23T23:13:37.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.712+0000 7ff975400640 1 --2- 172.21.15.12:0/2689863568 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c14b7a2580 0x55c14b7a8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:13:37.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.713+0000 7ff976800640 1 --2- 172.21.15.12:0/2689863568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a09f0 0x55c14b7a0e10 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7ff96c010450 tx=0x7ff96c010910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfdea0a822942055 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.713+0000 7ff95f000640 1 -- 172.21.15.12:0/2689863568 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96c017070 con 0x55c14b7a09f0 2024-03-23T23:13:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.713+0000 7ff95f000640 1 -- 172.21.15.12:0/2689863568 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff96c011c40 con 0x55c14b7a09f0 2024-03-23T23:13:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.713+0000 7ff95f000640 1 -- 172.21.15.12:0/2689863568 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96c01adb0 con 0x55c14b7a09f0 2024-03-23T23:13:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.713+0000 7ff96aa00640 1 --2- 172.21.15.12:0/2844051478 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff9580a29a0 0x7ff9580a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.713+0000 7ff96aa00640 1 -- 172.21.15.12:0/2844051478 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff9640f0670 con 0x7ff970107bc0 2024-03-23T23:13:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.713+0000 7ff975e00640 1 --2- 172.21.15.12:0/2844051478 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff9580a29a0 0x7ff9580a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.713+0000 7ff97b440740 1 -- 172.21.15.12:0/2689863568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a09f0 msgr2=0x55c14b7a0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.713+0000 7ff97b440740 1 --2- 172.21.15.12:0/2689863568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a09f0 0x55c14b7a0e10 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7ff96c010450 tx=0x7ff96c010910 comp rx=0 tx=0).stop 2024-03-23T23:13:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.714+0000 7ff97b440740 1 -- 172.21.15.12:0/2689863568 shutdown_connections 2024-03-23T23:13:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.714+0000 7ff97b440740 1 --2- 172.21.15.12:0/2689863568 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c14b7a2580 0x55c14b7a8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.714+0000 7ff97b440740 1 --2- 172.21.15.12:0/2689863568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c14b7a1bc0 0x55c14b7a2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.714+0000 7ff97b440740 1 --2- 172.21.15.12:0/2689863568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a09f0 0x55c14b7a0e10 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.714+0000 7ff97b440740 1 -- 172.21.15.12:0/2689863568 >> 172.21.15.12:0/2689863568 conn(0x55c14b79bde0 msgr2=0x55c14b79c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.714+0000 7ff97b440740 1 -- 172.21.15.12:0/2689863568 shutdown_connections 2024-03-23T23:13:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.714+0000 7ff97b440740 1 -- 172.21.15.12:0/2689863568 wait complete. 2024-03-23T23:13:37.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.714+0000 7ff975e00640 1 --2- 172.21.15.12:0/2844051478 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff9580a29a0 0x7ff9580a4e60 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff9640b9480 tx=0x7ff9640f6550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:37.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.714+0000 7ff97b440740 1 -- start start 2024-03-23T23:13:37.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.714+0000 7ff97b440740 1 Processor -- start 2024-03-23T23:13:37.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.715+0000 7ff97b440740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a09f0 0x55c14b7aaca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.715+0000 7ff97b440740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c14b7a1bc0 0x55c14b7ab1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.715+0000 7ff97b440740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c14b7a2580 0x55c14b7b2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.715+0000 7ff97b440740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c14b7a9590 con 0x55c14b7a1bc0 2024-03-23T23:13:37.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.715+0000 7ff97b440740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c14b6c10d0 con 0x55c14b7a09f0 2024-03-23T23:13:37.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.715+0000 7ff97b440740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c14b7a9710 con 0x55c14b7a2580 2024-03-23T23:13:37.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.715+0000 7ff97b440740 10 client.0 _ll_register_callbacks cb 0x55c14b6c5e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:13:37.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.715+0000 7ff976800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a09f0 0x55c14b7aaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.715+0000 7ff975400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c14b7a1bc0 0x55c14b7ab1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.715+0000 7ff976800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a09f0 0x55c14b7aaca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:38344/0 (socket says 172.21.15.12:38344) 2024-03-23T23:13:37.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.715+0000 7ff976800640 1 -- 172.21.15.12:0/962311102 learned_addr learned my addr 172.21.15.12:0/962311102 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.715+0000 7ff976800640 1 --2- 172.21.15.12:0/962311102 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c14b7a2580 0x55c14b7b2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.716+0000 7ff976800640 1 -- 172.21.15.12:0/962311102 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c14b7a2580 msgr2=0x55c14b7b2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.716+0000 7ff976800640 1 --2- 172.21.15.12:0/962311102 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c14b7a2580 0x55c14b7b2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.716+0000 7ff976800640 1 -- 172.21.15.12:0/962311102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c14b7a1bc0 msgr2=0x55c14b7ab1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.716+0000 7ff976800640 1 --2- 172.21.15.12:0/962311102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c14b7a1bc0 0x55c14b7ab1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.716+0000 7ff976800640 1 -- 172.21.15.12:0/962311102 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c14b4193f0 con 0x55c14b7a09f0 2024-03-23T23:13:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.716+0000 7ff975400640 1 --2- 172.21.15.12:0/962311102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c14b7a1bc0 0x55c14b7ab1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.716+0000 7ff976800640 1 --2- 172.21.15.12:0/962311102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a09f0 0x55c14b7aaca0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7ff96c011fd0 tx=0x7ff96c009280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51ab723a57061a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.716+0000 7ff953400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-23T23:13:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.716+0000 7ff953400640 1 -- 172.21.15.12:0/962311102 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96c017070 con 0x55c14b7a09f0 2024-03-23T23:13:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.716+0000 7ff953400640 1 -- 172.21.15.12:0/962311102 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff96c004a40 con 0x55c14b7a09f0 2024-03-23T23:13:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.716+0000 7ff97b440740 1 -- 172.21.15.12:0/962311102 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff96c011c40 con 0x55c14b7a09f0 2024-03-23T23:13:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.716+0000 7ff953400640 1 -- 172.21.15.12:0/962311102 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff96c004c40 con 0x55c14b7a09f0 2024-03-23T23:13:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.717+0000 7ff97b440740 10 client.28622 fetch_fsmap learned FSMap version 23 2024-03-23T23:13:37.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.717+0000 7ff97b440740 1 -- 172.21.15.12:0/962311102 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c14b4e37d0 con 0x55c14b7a09f0 2024-03-23T23:13:37.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.717+0000 7ff953400640 1 -- 172.21.15.12:0/962311102 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff96c011c40 con 0x55c14b7a09f0 2024-03-23T23:13:37.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.717+0000 7ff953400640 1 -- 172.21.15.12:0/962311102 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ff96c023020 con 0x55c14b7a09f0 2024-03-23T23:13:37.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.717+0000 7ff97b440740 10 client.28622 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:13:37.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.717+0000 7ff97b440740 10 client.28622 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:13:37.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.717+0000 7ff97b440740 20 client.28622 populate_metadata read hostname 'smithi012' 2024-03-23T23:13:37.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.717+0000 7ff97b440740 1 --2- 172.21.15.12:0/962311102 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c14b7b64a0 0x55c14b7b8890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.717+0000 7ff97b440740 4 client.28622 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T23:13:37.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.717+0000 7ff97b440740 1 -- 172.21.15.12:0/962311102 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c14b712800 con 0x55c14b7b64a0 2024-03-23T23:13:37.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.717+0000 7ff975400640 1 --2- 172.21.15.12:0/962311102 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c14b7b64a0 0x55c14b7b8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.718+0000 7ff975400640 1 --2- 172.21.15.12:0/962311102 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c14b7b64a0 0x55c14b7b8890 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b36f5541d6e9c80 server_cookie=828b929e28618d1f in_seq=0 out_seq=0 2024-03-23T23:13:37.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.718+0000 7ff953400640 10 client.28622 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:37.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.719+0000 7ff953400640 1 -- 172.21.15.12:0/962311102 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c14b712800 con 0x55c14b7b64a0 2024-03-23T23:13:37.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.719+0000 7ff953400640 10 client.28622 handle_command_reply: tid=0 2024-03-23T23:13:37.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.719+0000 7ff97b440740 1 client.28622 shutdown 2024-03-23T23:13:37.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.719+0000 7ff97b440740 1 -- 172.21.15.12:0/962311102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a09f0 msgr2=0x55c14b7aaca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.719+0000 7ff97b440740 1 --2- 172.21.15.12:0/962311102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a09f0 0x55c14b7aaca0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7ff96c011fd0 tx=0x7ff96c009280 comp rx=0 tx=0).stop 2024-03-23T23:13:37.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.719+0000 7ff97b440740 1 -- 172.21.15.12:0/962311102 shutdown_connections 2024-03-23T23:13:37.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.719+0000 7ff97b440740 1 --2- 172.21.15.12:0/962311102 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c14b7b64a0 0x55c14b7b8890 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.719+0000 7ff97b440740 1 --2- 172.21.15.12:0/962311102 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c14b7a2580 0x55c14b7b2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.719+0000 7ff97b440740 1 --2- 172.21.15.12:0/962311102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c14b7a1bc0 0x55c14b7ab1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.719+0000 7ff953400640 0 client.28622 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:37.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.719+0000 7ff97b440740 1 --2- 172.21.15.12:0/962311102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a09f0 0x55c14b7aaca0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.719+0000 7ff97b440740 1 -- 172.21.15.12:0/962311102 >> 172.21.15.12:0/962311102 conn(0x55c14b79bde0 msgr2=0x55c14b79b160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:37.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.720+0000 7ff97b440740 1 -- 172.21.15.12:0/962311102 shutdown_connections 2024-03-23T23:13:37.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.720+0000 7ff97b440740 1 -- 172.21.15.12:0/962311102 wait complete. 2024-03-23T23:13:37.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.720+0000 7ff97b440740 20 client.28622 trim_cache size 0 max 16384 2024-03-23T23:13:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.733+0000 7ff97b440740 1 Processor -- start 2024-03-23T23:13:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.733+0000 7ff97b440740 1 -- start start 2024-03-23T23:13:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.734+0000 7ff97b440740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c14b7a09f0 0x55c14b79f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.734+0000 7ff97b440740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a1bc0 0x55c14b7b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.734+0000 7ff97b440740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c14b7a2580 0x55c14b7bd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.734+0000 7ff97b440740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9701b12f0 con 0x55c14b7a2580 2024-03-23T23:13:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.734+0000 7ff97b440740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9701aedf0 con 0x55c14b7a1bc0 2024-03-23T23:13:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.734+0000 7ff97b440740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9701ace90 con 0x55c14b7a09f0 2024-03-23T23:13:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.734+0000 7ff975400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a1bc0 0x55c14b7b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.734+0000 7ff976800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c14b7a09f0 0x55c14b79f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.734+0000 7ff975400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a1bc0 0x55c14b7b21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:38348/0 (socket says 172.21.15.12:38348) 2024-03-23T23:13:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.734+0000 7ff975400640 1 -- 172.21.15.12:0/1599736920 learned_addr learned my addr 172.21.15.12:0/1599736920 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:37.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.734+0000 7ff976800640 1 --2- 172.21.15.12:0/1599736920 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c14b7a2580 0x55c14b7bd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.734+0000 7ff975400640 1 -- 172.21.15.12:0/1599736920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c14b7a09f0 msgr2=0x55c14b79f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:37.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.734+0000 7ff975400640 1 --2- 172.21.15.12:0/1599736920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c14b7a09f0 0x55c14b79f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.734+0000 7ff975400640 1 -- 172.21.15.12:0/1599736920 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c14b7a2580 msgr2=0x55c14b7bd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:37.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.734+0000 7ff975400640 1 --2- 172.21.15.12:0/1599736920 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c14b7a2580 0x55c14b7bd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.734+0000 7ff975400640 1 -- 172.21.15.12:0/1599736920 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c14b7978b0 con 0x55c14b7a1bc0 2024-03-23T23:13:37.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.734+0000 7ff976800640 1 --2- 172.21.15.12:0/1599736920 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c14b7a2580 0x55c14b7bd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:13:37.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.735+0000 7ff975400640 1 --2- 172.21.15.12:0/1599736920 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a1bc0 0x55c14b7b21b0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x55c14b7b90e0 tx=0x7ff960005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d5c180e9cf10e4b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.735+0000 7ff95e600640 1 -- 172.21.15.12:0/1599736920 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff960004020 con 0x55c14b7a1bc0 2024-03-23T23:13:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.735+0000 7ff95e600640 1 -- 172.21.15.12:0/1599736920 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff960005500 con 0x55c14b7a1bc0 2024-03-23T23:13:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.735+0000 7ff95e600640 1 -- 172.21.15.12:0/1599736920 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff960020d70 con 0x55c14b7a1bc0 2024-03-23T23:13:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.735+0000 7ff97b440740 1 -- 172.21.15.12:0/1599736920 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a1bc0 msgr2=0x55c14b7b21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.735+0000 7ff97b440740 1 --2- 172.21.15.12:0/1599736920 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a1bc0 0x55c14b7b21b0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x55c14b7b90e0 tx=0x7ff960005f90 comp rx=0 tx=0).stop 2024-03-23T23:13:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.735+0000 7ff97b440740 1 -- 172.21.15.12:0/1599736920 shutdown_connections 2024-03-23T23:13:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.735+0000 7ff97b440740 1 --2- 172.21.15.12:0/1599736920 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c14b7a2580 0x55c14b7bd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.735+0000 7ff97b440740 1 --2- 172.21.15.12:0/1599736920 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a1bc0 0x55c14b7b21b0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.735+0000 7ff97b440740 1 --2- 172.21.15.12:0/1599736920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c14b7a09f0 0x55c14b79f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.735+0000 7ff97b440740 1 -- 172.21.15.12:0/1599736920 >> 172.21.15.12:0/1599736920 conn(0x55c14b79bde0 msgr2=0x55c14b79f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.736+0000 7ff97b440740 1 -- 172.21.15.12:0/1599736920 shutdown_connections 2024-03-23T23:13:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.736+0000 7ff97b440740 1 -- 172.21.15.12:0/1599736920 wait complete. 2024-03-23T23:13:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.736+0000 7ff97b440740 1 -- start start 2024-03-23T23:13:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.736+0000 7ff97b440740 1 Processor -- start 2024-03-23T23:13:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.736+0000 7ff97b440740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a09f0 0x55c14b7addd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.736+0000 7ff97b440740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c14b7a1bc0 0x55c14b7ae310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.736+0000 7ff97b440740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c14b7a2580 0x55c14b7b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.736+0000 7ff97b440740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9701ad490 con 0x55c14b7a2580 2024-03-23T23:13:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.736+0000 7ff97b440740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9701ae790 con 0x55c14b7a09f0 2024-03-23T23:13:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.736+0000 7ff97b440740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9701adad0 con 0x55c14b7a1bc0 2024-03-23T23:13:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.737+0000 7ff97b440740 10 client.0 _ll_register_callbacks cb 0x7ff9640f6420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T23:13:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.737+0000 7ff975400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c14b7a1bc0 0x55c14b7ae310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.737+0000 7ff976800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a09f0 0x55c14b7addd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.737+0000 7ff975400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c14b7a1bc0 0x55c14b7ae310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:44480/0 (socket says 172.21.15.12:44480) 2024-03-23T23:13:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.737+0000 7ff975400640 1 -- 172.21.15.12:0/3381541236 learned_addr learned my addr 172.21.15.12:0/3381541236 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.737+0000 7ff976800640 1 --2- 172.21.15.12:0/3381541236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c14b7a2580 0x55c14b7b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.737+0000 7ff975400640 1 -- 172.21.15.12:0/3381541236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a09f0 msgr2=0x55c14b7addd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.737+0000 7ff975400640 1 --2- 172.21.15.12:0/3381541236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a09f0 0x55c14b7addd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.737+0000 7ff975400640 1 -- 172.21.15.12:0/3381541236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c14b7a2580 msgr2=0x55c14b7b12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T23:13:37.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.737+0000 7ff975400640 1 --2- 172.21.15.12:0/3381541236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c14b7a2580 0x55c14b7b12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.737+0000 7ff975400640 1 -- 172.21.15.12:0/3381541236 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c14b7b18d0 con 0x55c14b7a1bc0 2024-03-23T23:13:37.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.737+0000 7ff976800640 1 --2- 172.21.15.12:0/3381541236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a09f0 0x55c14b7addd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:37.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.738+0000 7ff975400640 1 --2- 172.21.15.12:0/3381541236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c14b7a1bc0 0x55c14b7ae310 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ff960012780 tx=0x7ff960009ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=555621fd3e4be9e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:37.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.738+0000 7ff953400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-23T23:13:37.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.738+0000 7ff953400640 1 -- 172.21.15.12:0/3381541236 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96002f020 con 0x55c14b7a1bc0 2024-03-23T23:13:37.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.738+0000 7ff953400640 1 -- 172.21.15.12:0/3381541236 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff960020ab0 con 0x55c14b7a1bc0 2024-03-23T23:13:37.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.738+0000 7ff97b440740 1 -- 172.21.15.12:0/3381541236 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff960005500 con 0x55c14b7a1bc0 2024-03-23T23:13:37.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.738+0000 7ff953400640 1 -- 172.21.15.12:0/3381541236 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff960025040 con 0x55c14b7a1bc0 2024-03-23T23:13:37.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.738+0000 7ff97b440740 10 client.28588 fetch_fsmap learned FSMap version 23 2024-03-23T23:13:37.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.738+0000 7ff97b440740 1 -- 172.21.15.12:0/3381541236 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c14b8121e0 con 0x55c14b7a1bc0 2024-03-23T23:13:37.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.738+0000 7ff953400640 1 -- 172.21.15.12:0/3381541236 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff960005500 con 0x55c14b7a1bc0 2024-03-23T23:13:37.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.738+0000 7ff953400640 1 -- 172.21.15.12:0/3381541236 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ff960010430 con 0x55c14b7a1bc0 2024-03-23T23:13:37.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.738+0000 7ff97b440740 10 client.28588 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T23:13:37.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.738+0000 7ff97b440740 10 client.28588 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-23T23:13:37.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.738+0000 7ff97b440740 20 client.28588 populate_metadata read hostname 'smithi012' 2024-03-23T23:13:37.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.738+0000 7ff97b440740 1 --2- 172.21.15.12:0/3381541236 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c14b8129e0 0x55c14b814e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:37.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.739+0000 7ff97b440740 4 client.28588 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-23T23:13:37.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.739+0000 7ff97b440740 1 -- 172.21.15.12:0/3381541236 --> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c14b70cb50 con 0x55c14b8129e0 2024-03-23T23:13:37.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.739+0000 7ff976800640 1 --2- 172.21.15.12:0/3381541236 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c14b8129e0 0x55c14b814e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.739+0000 7ff976800640 1 --2- 172.21.15.12:0/3381541236 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c14b8129e0 0x55c14b814e00 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8d9ba16192c7b48 server_cookie=32ea4eb29e00ce9f in_seq=0 out_seq=0 2024-03-23T23:13:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.739+0000 7ff953400640 10 client.28588 ms_handle_connect on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.739+0000 7ff953400640 1 -- 172.21.15.12:0/3381541236 <== mds.0 v2:172.21.15.173:6838/1645433214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c14b70cb50 con 0x55c14b8129e0 2024-03-23T23:13:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.739+0000 7ff953400640 10 client.28588 handle_command_reply: tid=0 2024-03-23T23:13:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.739+0000 7ff97b440740 1 client.28588 shutdown 2024-03-23T23:13:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.740+0000 7ff97b440740 1 -- 172.21.15.12:0/3381541236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c14b7a1bc0 msgr2=0x55c14b7ae310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.740+0000 7ff97b440740 1 --2- 172.21.15.12:0/3381541236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c14b7a1bc0 0x55c14b7ae310 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ff960012780 tx=0x7ff960009ce0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.740+0000 7ff97b440740 1 -- 172.21.15.12:0/3381541236 shutdown_connections 2024-03-23T23:13:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.740+0000 7ff97b440740 1 --2- 172.21.15.12:0/3381541236 >> [v2:172.21.15.173:6838/1645433214,v1:172.21.15.173:6839/1645433214] conn(0x55c14b8129e0 0x55c14b814e00 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.740+0000 7ff97b440740 1 --2- 172.21.15.12:0/3381541236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c14b7a2580 0x55c14b7b12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.740+0000 7ff97b440740 1 --2- 172.21.15.12:0/3381541236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c14b7a1bc0 0x55c14b7ae310 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.740+0000 7ff953400640 0 client.28588 ms_handle_reset on v2:172.21.15.173:6838/1645433214 2024-03-23T23:13:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.740+0000 7ff97b440740 1 --2- 172.21.15.12:0/3381541236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c14b7a09f0 0x55c14b7addd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.740+0000 7ff97b440740 1 -- 172.21.15.12:0/3381541236 >> 172.21.15.12:0/3381541236 conn(0x55c14b79bde0 msgr2=0x55c14b79f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.740+0000 7ff97b440740 1 -- 172.21.15.12:0/3381541236 shutdown_connections 2024-03-23T23:13:37.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.740+0000 7ff97b440740 1 -- 172.21.15.12:0/3381541236 wait complete. 2024-03-23T23:13:37.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.740+0000 7ff97b440740 20 client.28588 trim_cache size 0 max 16384 2024-03-23T23:13:37.743 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-03-23T23:13:37.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.741+0000 7ff977200640 1 -- 172.21.15.12:0/2844051478 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff9580a29a0 msgr2=0x7ff9580a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.741+0000 7ff977200640 1 --2- 172.21.15.12:0/2844051478 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff9580a29a0 0x7ff9580a4e60 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff9640b9480 tx=0x7ff9640f6550 comp rx=0 tx=0).stop 2024-03-23T23:13:37.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.741+0000 7ff977200640 1 -- 172.21.15.12:0/2844051478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff970107bc0 msgr2=0x7ff97019ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:37.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.741+0000 7ff977200640 1 --2- 172.21.15.12:0/2844051478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff970107bc0 0x7ff97019ded0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7ff96402efd0 tx=0x7ff96402c660 comp rx=0 tx=0).stop 2024-03-23T23:13:37.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.741+0000 7ff977200640 1 -- 172.21.15.12:0/2844051478 shutdown_connections 2024-03-23T23:13:37.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.741+0000 7ff977200640 1 --2- 172.21.15.12:0/2844051478 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff9580a29a0 0x7ff9580a4e60 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.741+0000 7ff977200640 1 --2- 172.21.15.12:0/2844051478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff970109770 0x7ff9701a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.741+0000 7ff977200640 1 --2- 172.21.15.12:0/2844051478 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff970108db0 0x7ff97019e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.741+0000 7ff977200640 1 --2- 172.21.15.12:0/2844051478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff970107bc0 0x7ff97019ded0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:37.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.741+0000 7ff977200640 1 -- 172.21.15.12:0/2844051478 >> 172.21.15.12:0/2844051478 conn(0x7ff970076b70 msgr2=0x7ff97010c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:37.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.741+0000 7ff977200640 1 -- 172.21.15.12:0/2844051478 shutdown_connections 2024-03-23T23:13:37.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:37.741+0000 7ff977200640 1 -- 172.21.15.12:0/2844051478 wait complete. 2024-03-23T23:13:38.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:37 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/4277367992' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:13:38.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:37 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/4277367992' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:13:38.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:37 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/4277367992' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:13:38.758 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-23T23:13:38.758 INFO:tasks.fwd_scrub:done joining 2024-03-23T23:13:38.758 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-23T23:13:38.768 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T23:13:39.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.003+0000 7f6247e00640 1 -- 172.21.15.12:0/3184243184 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62401028b0 msgr2=0x7f6240102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:39.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.003+0000 7f6247e00640 1 --2- 172.21.15.12:0/3184243184 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62401028b0 0x7f6240102d30 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f622c009860 tx=0x7f622c02bb70 comp rx=0 tx=0).stop 2024-03-23T23:13:39.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.003+0000 7f6247e00640 1 -- 172.21.15.12:0/3184243184 shutdown_connections 2024-03-23T23:13:39.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.003+0000 7f6247e00640 1 --2- 172.21.15.12:0/3184243184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6240103270 0x7f6240109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:39.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.003+0000 7f6247e00640 1 --2- 172.21.15.12:0/3184243184 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62401028b0 0x7f6240102d30 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:39.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.003+0000 7f6247e00640 1 --2- 172.21.15.12:0/3184243184 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62401016c0 0x7f6240101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:39.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.003+0000 7f6247e00640 1 -- 172.21.15.12:0/3184243184 >> 172.21.15.12:0/3184243184 conn(0x7f62400fcc50 msgr2=0x7f62400ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:39.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.004+0000 7f6247e00640 1 -- 172.21.15.12:0/3184243184 shutdown_connections 2024-03-23T23:13:39.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.004+0000 7f6247e00640 1 -- 172.21.15.12:0/3184243184 wait complete. 2024-03-23T23:13:39.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.005+0000 7f6247e00640 1 Processor -- start 2024-03-23T23:13:39.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.005+0000 7f6247e00640 1 -- start start 2024-03-23T23:13:39.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.006+0000 7f6247e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62401016c0 0x7f6240199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:39.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.006+0000 7f6247e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62401028b0 0x7f624019a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:39.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.006+0000 7f6247e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6240103270 0x7f62401a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:39.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.006+0000 7f6247e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624010d320 con 0x7f62401028b0 2024-03-23T23:13:39.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.006+0000 7f6247e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624010d1a0 con 0x7f62401016c0 2024-03-23T23:13:39.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.006+0000 7f6247e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624010d4a0 con 0x7f6240103270 2024-03-23T23:13:39.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.006+0000 7f6245400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62401016c0 0x7f6240199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:39.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.006+0000 7f6245400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62401016c0 0x7f6240199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:38368/0 (socket says 172.21.15.12:38368) 2024-03-23T23:13:39.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.006+0000 7f6245400640 1 -- 172.21.15.12:0/2136752235 learned_addr learned my addr 172.21.15.12:0/2136752235 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:39.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.006+0000 7f6245e00640 1 --2- 172.21.15.12:0/2136752235 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6240103270 0x7f62401a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:39.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.006+0000 7f6244a00640 1 --2- 172.21.15.12:0/2136752235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62401028b0 0x7f624019a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:39.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.006+0000 7f6245400640 1 -- 172.21.15.12:0/2136752235 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6240103270 msgr2=0x7f62401a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:39.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.006+0000 7f6245400640 1 --2- 172.21.15.12:0/2136752235 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6240103270 0x7f62401a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:39.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.006+0000 7f6245400640 1 -- 172.21.15.12:0/2136752235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62401028b0 msgr2=0x7f624019a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:39.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.006+0000 7f6245400640 1 --2- 172.21.15.12:0/2136752235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62401028b0 0x7f624019a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:39.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.006+0000 7f6245400640 1 -- 172.21.15.12:0/2136752235 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62401a1830 con 0x7f62401016c0 2024-03-23T23:13:39.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.007+0000 7f6245400640 1 --2- 172.21.15.12:0/2136752235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62401016c0 0x7f6240199c20 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f623400e9c0 tx=0x7f623400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:39.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.007+0000 7f623e000640 1 -- 172.21.15.12:0/2136752235 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623400cd80 con 0x7f62401016c0 2024-03-23T23:13:39.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.007+0000 7f623e000640 1 -- 172.21.15.12:0/2136752235 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62340035f0 con 0x7f62401016c0 2024-03-23T23:13:39.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.007+0000 7f623e000640 1 -- 172.21.15.12:0/2136752235 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6234010690 con 0x7f62401016c0 2024-03-23T23:13:39.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.007+0000 7f6247e00640 1 -- 172.21.15.12:0/2136752235 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62401a1b20 con 0x7f62401016c0 2024-03-23T23:13:39.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.007+0000 7f6247e00640 1 -- 172.21.15.12:0/2136752235 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62401a20c0 con 0x7f62401016c0 2024-03-23T23:13:39.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.008+0000 7f6247e00640 1 -- 172.21.15.12:0/2136752235 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62401a2290 con 0x7f62401016c0 2024-03-23T23:13:39.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.010+0000 7f623e000640 1 -- 172.21.15.12:0/2136752235 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f62340037a0 con 0x7f62401016c0 2024-03-23T23:13:39.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.012+0000 7f623e000640 1 --2- 172.21.15.12:0/2136752235 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f62280a2880 0x7f62280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:39.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.012+0000 7f623e000640 1 -- 172.21.15.12:0/2136752235 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f62340cef20 con 0x7f62401016c0 2024-03-23T23:13:39.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.012+0000 7f6244a00640 1 --2- 172.21.15.12:0/2136752235 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f62280a2880 0x7f62280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:39.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.013+0000 7f6244a00640 1 --2- 172.21.15.12:0/2136752235 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f62280a2880 0x7f62280a4d40 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f622c009830 tx=0x7f622c034ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:39.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.016+0000 7f623e000640 1 -- 172.21.15.12:0/2136752235 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f623409bf70 con 0x7f62401016c0 2024-03-23T23:13:39.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:38 smithi012 ceph-mon[31695]: pgmap v745: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-23T23:13:39.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:38 smithi173 ceph-mon[36116]: pgmap v745: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-23T23:13:39.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:38 smithi203 ceph-mon[36693]: pgmap v745: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-23T23:13:39.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.226+0000 7f6247e00640 1 -- 172.21.15.12:0/2136752235 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62401a2460 con 0x7f62401016c0 2024-03-23T23:13:39.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.228+0000 7f623e000640 1 -- 172.21.15.12:0/2136752235 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22695 (secure 0 0 0) 0x7f623409b870 con 0x7f62401016c0 2024-03-23T23:13:39.228 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:13:39.228 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":88,"fsid":"90f679ae-e967-11ee-95cd-87774f69a715","created":"2024-03-23T22:50:14.906952+0000","modified":"2024-03-23T23:00:24.349052+0000","last_up_change":"2024-03-23T22:58:28.907824+0000","last_in_change":"2024-03-23T22:58:08.342487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T22:54:11.831010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T22:59:25.350974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T22:59:26.390323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b70875d7-a23d-4251-9e8b-1a16d4edc2a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6803","nonce":1190079611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6805","nonce":1190079611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6809","nonce":1190079611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6807","nonce":1190079611}]},"public_addr":"172.21.15.12:6803/1190079611","cluster_addr":"172.21.15.12:6805/1190079611","heartbeat_back_addr":"172.21.15.12:6809/1190079611","heartbeat_front_addr":"172.21.15.12:6807/1190079611","state":["exists","up"]},{"osd":1,"uuid":"416e87af-db5d-41ce-84d9-731e3e07e3c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6801","nonce":1983639525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6803","nonce":1983639525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6807","nonce":1983639525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6805","nonce":1983639525}]},"public_addr":"172.21.15.173:6801/1983639525","cluster_addr":"172.21.15.173:6803/1983639525","heartbeat_back_addr":"172.21.15.173:6807/1983639525","heartbeat_front_addr":"172.21.15.173:6805/1983639525","state":["exists","up"]},{"osd":2,"uuid":"7a559c70-4c54-431b-8baf-4136b6e76493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6801","nonce":2253997387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6803","nonce":2253997387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6807","nonce":2253997387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6805","nonce":2253997387}]},"public_addr":"172.21.15.203:6801/2253997387","cluster_addr":"172.21.15.203:6803/2253997387","heartbeat_back_addr":"172.21.15.203:6807/2253997387","heartbeat_front_addr":"172.21.15.203:6805/2253997387","state":["exists","up"]},{"osd":3,"uuid":"c41cd287-a674-486c-8025-328cd36e5ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6811","nonce":1761446905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6813","nonce":1761446905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6817","nonce":1761446905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6815","nonce":1761446905}]},"public_addr":"172.21.15.12:6811/1761446905","cluster_addr":"172.21.15.12:6813/1761446905","heartbeat_back_addr":"172.21.15.12:6817/1761446905","heartbeat_front_addr":"172.21.15.12:6815/1761446905","state":["exists","up"]},{"osd":4,"uuid":"3c48f886-0b42-4a09-b31e-abccc200bb1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6809","nonce":242958272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6811","nonce":242958272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6815","nonce":242958272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6813","nonce":242958272}]},"public_addr":"172.21.15.173:6809/242958272","cluster_addr":"172.21.15.173:6811/242958272","heartbeat_back_addr":"172.21.15.173:6815/242958272","heartbeat_front_addr":"172.21.15.173:6813/242958272","state":["exists","up"]},{"osd":5,"uuid":"4a34f06f-8805-40c0-b1ab-23407b0276d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6809","nonce":4270966092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6811","nonce":4270966092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6815","nonce":4270966092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6813","nonce":4270966092}]},"public_addr":"172.21.15.203:6809/4270966092","cluster_addr":"172.21.15.203:6811/4270966092","heartbeat_back_addr":"172.21.15.203:6815/4270966092","heartbeat_front_addr":"172.21.15.203:6813/4270966092","state":["exists","up"]},{"osd":6,"uuid":"26ddc663-5623-48c9-80d1-f8b968765d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6819","nonce":3606839295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6821","nonce":3606839295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6825","nonce":3606839295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6823","nonce":3606839295}]},"public_addr":"172.21.15.12:6819/3606839295","cluster_addr":"172.21.15.12:6821/3606839295","heartbeat_back_addr":"172.21.15.12:6825/3606839295","heartbeat_front_addr":"172.21.15.12:6823/3606839295","state":["exists","up"]},{"osd":7,"uuid":"17dc9417-71cd-4576-aa08-933212ec55e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6817","nonce":1241504346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6819","nonce":1241504346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6823","nonce":1241504346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6821","nonce":1241504346}]},"public_addr":"172.21.15.173:6817/1241504346","cluster_addr":"172.21.15.173:6819/1241504346","heartbeat_back_addr":"172.21.15.173:6823/1241504346","heartbeat_front_addr":"172.21.15.173:6821/1241504346","state":["exists","up"]},{"osd":8,"uuid":"90a2b63a-471d-43b8-ac18-bb1f16778d18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6817","nonce":1516996192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6819","nonce":1516996192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6823","nonce":1516996192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6821","nonce":1516996192}]},"public_addr":"172.21.15.203:6817/1516996192","cluster_addr":"172.21.15.203:6819/1516996192","heartbeat_back_addr":"172.21.15.203:6823/1516996192","heartbeat_front_addr":"172.21.15.203:6821/1516996192","state":["exists","up"]},{"osd":9,"uuid":"6e1ca504-40b4-49d0-b245-7946181abfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6827","nonce":1175260774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6829","nonce":1175260774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6833","nonce":1175260774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6831","nonce":1175260774}]},"public_addr":"172.21.15.12:6827/1175260774","cluster_addr":"172.21.15.12:6829/1175260774","heartbeat_back_addr":"172.21.15.12:6833/1175260774","heartbeat_front_addr":"172.21.15.12:6831/1175260774","state":["exists","up"]},{"osd":10,"uuid":"09a595ed-5614-4386-8e61-0170af66e01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6825","nonce":3303506808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6827","nonce":3303506808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6831","nonce":3303506808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6829","nonce":3303506808}]},"public_addr":"172.21.15.173:6825/3303506808","cluster_addr":"172.21.15.173:6827/3303506808","heartbeat_back_addr":"172.21.15.173:6831/3303506808","heartbeat_front_addr":"172.21.15.173:6829/3303506808","state":["exists","up"]},{"osd":11,"uuid":"9ef1d595-a977-4ba7-9c6b-d2b3295f20b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6825","nonce":2323604506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6827","nonce":2323604506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6831","nonce":2323604506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6829","nonce":2323604506}]},"public_addr":"172.21.15.203:6825/2323604506","cluster_addr":"172.21.15.203:6827/2323604506","heartbeat_back_addr":"172.21.15.203:6831/2323604506","heartbeat_front_addr":"172.21.15.203:6829/2323604506","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:18.469038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:42.752628+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:07.069493+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:41.962478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:08.363104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:33.341336+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:07.286541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:34.243302+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:59.749699+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:57:33.822913+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:00.737269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:26.599854+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/4182783120":"2024-03-24T22:51:23.113201+0000","172.21.15.12:6800/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1900938637":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/3414861652":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6801/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/2580541477":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6800/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/501058839":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1192554716":"2024-03-24T22:50:48.401223+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T23:13:39.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.231+0000 7f6247e00640 1 -- 172.21.15.12:0/2136752235 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f62280a2880 msgr2=0x7f62280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:39.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.231+0000 7f6247e00640 1 --2- 172.21.15.12:0/2136752235 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f62280a2880 0x7f62280a4d40 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f622c009830 tx=0x7f622c034ad0 comp rx=0 tx=0).stop 2024-03-23T23:13:39.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.231+0000 7f6247e00640 1 -- 172.21.15.12:0/2136752235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62401016c0 msgr2=0x7f6240199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:39.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.231+0000 7f6247e00640 1 --2- 172.21.15.12:0/2136752235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62401016c0 0x7f6240199c20 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f623400e9c0 tx=0x7f623400ee80 comp rx=0 tx=0).stop 2024-03-23T23:13:39.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.232+0000 7f6247e00640 1 -- 172.21.15.12:0/2136752235 shutdown_connections 2024-03-23T23:13:39.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.232+0000 7f6247e00640 1 --2- 172.21.15.12:0/2136752235 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f62280a2880 0x7f62280a4d40 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:39.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.232+0000 7f6247e00640 1 --2- 172.21.15.12:0/2136752235 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6240103270 0x7f62401a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:39.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.232+0000 7f6247e00640 1 --2- 172.21.15.12:0/2136752235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62401028b0 0x7f624019a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:39.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.232+0000 7f6247e00640 1 --2- 172.21.15.12:0/2136752235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62401016c0 0x7f6240199c20 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:39.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.232+0000 7f6247e00640 1 -- 172.21.15.12:0/2136752235 >> 172.21.15.12:0/2136752235 conn(0x7f62400fcc50 msgr2=0x7f62400738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:39.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.232+0000 7f6247e00640 1 -- 172.21.15.12:0/2136752235 shutdown_connections 2024-03-23T23:13:39.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.232+0000 7f6247e00640 1 -- 172.21.15.12:0/2136752235 wait complete. 2024-03-23T23:13:39.250 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T22:54:11.831010+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T22:59:25.350974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T22:59:26.390323+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T23:13:39.250 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T23:13:39.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.483+0000 7f6875200640 1 -- 172.21.15.12:0/787224997 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f68700772a0 msgr2=0x7f6870077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:39.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.483+0000 7f6875200640 1 --2- 172.21.15.12:0/787224997 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f68700772a0 0x7f6870077700 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f685c009900 tx=0x7f685c02bc30 comp rx=0 tx=0).stop 2024-03-23T23:13:39.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.484+0000 7f6875200640 1 -- 172.21.15.12:0/787224997 shutdown_connections 2024-03-23T23:13:39.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.484+0000 7f6875200640 1 --2- 172.21.15.12:0/787224997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6870077c40 0x7f68700784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:39.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.484+0000 7f6875200640 1 --2- 172.21.15.12:0/787224997 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f68700772a0 0x7f6870077700 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:39.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.484+0000 7f6875200640 1 --2- 172.21.15.12:0/787224997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6870076060 0x7f6870076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:39.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.484+0000 7f6875200640 1 -- 172.21.15.12:0/787224997 >> 172.21.15.12:0/787224997 conn(0x7f68700fee50 msgr2=0x7f6870101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:39.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.484+0000 7f6875200640 1 -- 172.21.15.12:0/787224997 shutdown_connections 2024-03-23T23:13:39.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.484+0000 7f6875200640 1 -- 172.21.15.12:0/787224997 wait complete. 2024-03-23T23:13:39.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.485+0000 7f6875200640 1 Processor -- start 2024-03-23T23:13:39.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.485+0000 7f6875200640 1 -- start start 2024-03-23T23:13:39.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.486+0000 7f6875200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6870076060 0x7f687019e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:39.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.486+0000 7f6875200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68700772a0 0x7f687019e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:39.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.486+0000 7f6875200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6870077c40 0x7f68701a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:39.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.486+0000 7f6875200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68701139b0 con 0x7f68700772a0 2024-03-23T23:13:39.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.486+0000 7f6875200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6870113830 con 0x7f6870076060 2024-03-23T23:13:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.486+0000 7f6875200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6870113b30 con 0x7f6870077c40 2024-03-23T23:13:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.486+0000 7f686de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6870076060 0x7f687019e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.486+0000 7f686de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6870076060 0x7f687019e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:38386/0 (socket says 172.21.15.12:38386) 2024-03-23T23:13:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.486+0000 7f686de00640 1 -- 172.21.15.12:0/3645828456 learned_addr learned my addr 172.21.15.12:0/3645828456 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.486+0000 7f686d400640 1 --2- 172.21.15.12:0/3645828456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68700772a0 0x7f687019e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.486+0000 7f686e800640 1 --2- 172.21.15.12:0/3645828456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6870077c40 0x7f68701a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.486+0000 7f686de00640 1 -- 172.21.15.12:0/3645828456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6870077c40 msgr2=0x7f68701a55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.486+0000 7f686de00640 1 --2- 172.21.15.12:0/3645828456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6870077c40 0x7f68701a55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.486+0000 7f686de00640 1 -- 172.21.15.12:0/3645828456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68700772a0 msgr2=0x7f687019e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.486+0000 7f686de00640 1 --2- 172.21.15.12:0/3645828456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68700772a0 0x7f687019e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.486+0000 7f686de00640 1 -- 172.21.15.12:0/3645828456 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68701a5c30 con 0x7f6870076060 2024-03-23T23:13:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.486+0000 7f686d400640 1 --2- 172.21.15.12:0/3645828456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68700772a0 0x7f687019e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.487+0000 7f686de00640 1 --2- 172.21.15.12:0/3645828456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6870076060 0x7f687019e010 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f686000e9f0 tx=0x7f686000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:39.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.487+0000 7f6866a00640 1 -- 172.21.15.12:0/3645828456 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686000cd80 con 0x7f6870076060 2024-03-23T23:13:39.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.487+0000 7f6866a00640 1 -- 172.21.15.12:0/3645828456 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6860010910 con 0x7f6870076060 2024-03-23T23:13:39.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.487+0000 7f6875200640 1 -- 172.21.15.12:0/3645828456 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68701a5f20 con 0x7f6870076060 2024-03-23T23:13:39.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.487+0000 7f6866a00640 1 -- 172.21.15.12:0/3645828456 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6860018990 con 0x7f6870076060 2024-03-23T23:13:39.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.487+0000 7f6875200640 1 -- 172.21.15.12:0/3645828456 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68701a6390 con 0x7f6870076060 2024-03-23T23:13:39.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.489+0000 7f6875200640 1 -- 172.21.15.12:0/3645828456 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6870110250 con 0x7f6870076060 2024-03-23T23:13:39.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.491+0000 7f6866a00640 1 -- 172.21.15.12:0/3645828456 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6860010ac0 con 0x7f6870076060 2024-03-23T23:13:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.492+0000 7f6866a00640 1 --2- 172.21.15.12:0/3645828456 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f68540a2880 0x7f68540a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:39.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.492+0000 7f6866a00640 1 -- 172.21.15.12:0/3645828456 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f68600ceeb0 con 0x7f6870076060 2024-03-23T23:13:39.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.493+0000 7f686d400640 1 --2- 172.21.15.12:0/3645828456 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f68540a2880 0x7f68540a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:39.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.493+0000 7f686d400640 1 --2- 172.21.15.12:0/3645828456 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f68540a2880 0x7f68540a4d40 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f685c02c240 tx=0x7f685c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:39.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.494+0000 7f6866a00640 1 -- 172.21.15.12:0/3645828456 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f686009b8b0 con 0x7f6870076060 2024-03-23T23:13:39.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.697+0000 7f6875200640 1 -- 172.21.15.12:0/3645828456 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f687002d300 con 0x7f6870076060 2024-03-23T23:13:39.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.698+0000 7f6866a00640 1 -- 172.21.15.12:0/3645828456 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f68600cce20 con 0x7f6870076060 2024-03-23T23:13:39.696 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-23T23:13:39.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.701+0000 7f6875200640 1 -- 172.21.15.12:0/3645828456 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f68540a2880 msgr2=0x7f68540a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:39.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.701+0000 7f6875200640 1 --2- 172.21.15.12:0/3645828456 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f68540a2880 0x7f68540a4d40 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f685c02c240 tx=0x7f685c02df90 comp rx=0 tx=0).stop 2024-03-23T23:13:39.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.701+0000 7f6875200640 1 -- 172.21.15.12:0/3645828456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6870076060 msgr2=0x7f687019e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:39.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.701+0000 7f6875200640 1 --2- 172.21.15.12:0/3645828456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6870076060 0x7f687019e010 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f686000e9f0 tx=0x7f686000eeb0 comp rx=0 tx=0).stop 2024-03-23T23:13:39.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.701+0000 7f6875200640 1 -- 172.21.15.12:0/3645828456 shutdown_connections 2024-03-23T23:13:39.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.701+0000 7f6875200640 1 --2- 172.21.15.12:0/3645828456 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f68540a2880 0x7f68540a4d40 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:39.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.701+0000 7f6875200640 1 --2- 172.21.15.12:0/3645828456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6870077c40 0x7f68701a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:39.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.701+0000 7f6875200640 1 --2- 172.21.15.12:0/3645828456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68700772a0 0x7f687019e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:39.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.701+0000 7f6875200640 1 --2- 172.21.15.12:0/3645828456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6870076060 0x7f687019e010 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:39.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.701+0000 7f6875200640 1 -- 172.21.15.12:0/3645828456 >> 172.21.15.12:0/3645828456 conn(0x7f68700fee50 msgr2=0x7f6870100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:39.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.701+0000 7f6875200640 1 -- 172.21.15.12:0/3645828456 shutdown_connections 2024-03-23T23:13:39.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.701+0000 7f6875200640 1 -- 172.21.15.12:0/3645828456 wait complete. 2024-03-23T23:13:39.718 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T23:13:39.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.954+0000 7fda6ac00640 1 -- 172.21.15.12:0/1476799502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda641038e0 msgr2=0x7fda64103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:39.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.954+0000 7fda6ac00640 1 --2- 172.21.15.12:0/1476799502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda641038e0 0x7fda64103d00 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fda54009900 tx=0x7fda5402bc30 comp rx=0 tx=0).stop 2024-03-23T23:13:39.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.954+0000 7fda6ac00640 1 -- 172.21.15.12:0/1476799502 shutdown_connections 2024-03-23T23:13:39.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.954+0000 7fda6ac00640 1 --2- 172.21.15.12:0/1476799502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda64105490 0x7fda6410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:39.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.954+0000 7fda6ac00640 1 --2- 172.21.15.12:0/1476799502 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda64104ad0 0x7fda64104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:39.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.954+0000 7fda6ac00640 1 --2- 172.21.15.12:0/1476799502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda641038e0 0x7fda64103d00 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:39.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.954+0000 7fda6ac00640 1 -- 172.21.15.12:0/1476799502 >> 172.21.15.12:0/1476799502 conn(0x7fda640fee50 msgr2=0x7fda64101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:39.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.954+0000 7fda6ac00640 1 -- 172.21.15.12:0/1476799502 shutdown_connections 2024-03-23T23:13:39.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.954+0000 7fda6ac00640 1 -- 172.21.15.12:0/1476799502 wait complete. 2024-03-23T23:13:39.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.956+0000 7fda6ac00640 1 Processor -- start 2024-03-23T23:13:39.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.956+0000 7fda6ac00640 1 -- start start 2024-03-23T23:13:39.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.956+0000 7fda6ac00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda641038e0 0x7fda64199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:39.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.956+0000 7fda6ac00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda64104ad0 0x7fda6419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:39.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.956+0000 7fda6ac00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda64105490 0x7fda641a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:39.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.956+0000 7fda6ac00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6410f580 con 0x7fda64104ad0 2024-03-23T23:13:39.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.956+0000 7fda6ac00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6410f400 con 0x7fda641038e0 2024-03-23T23:13:39.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.956+0000 7fda6ac00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6410f700 con 0x7fda64105490 2024-03-23T23:13:39.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.956+0000 7fda63e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda641038e0 0x7fda64199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:39.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.957+0000 7fda5be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda64104ad0 0x7fda6419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:39.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.957+0000 7fda68c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda64105490 0x7fda641a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:39.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.957+0000 7fda5be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda64104ad0 0x7fda6419a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57388/0 (socket says 172.21.15.12:57388) 2024-03-23T23:13:39.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.957+0000 7fda5be00640 1 -- 172.21.15.12:0/412520408 learned_addr learned my addr 172.21.15.12:0/412520408 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:39.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.957+0000 7fda63e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda641038e0 0x7fda64199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:38404/0 (socket says 172.21.15.12:38404) 2024-03-23T23:13:39.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.957+0000 7fda63e00640 1 -- 172.21.15.12:0/412520408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda64105490 msgr2=0x7fda641a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:39.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.957+0000 7fda63e00640 1 --2- 172.21.15.12:0/412520408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda64105490 0x7fda641a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:39.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.957+0000 7fda63e00640 1 -- 172.21.15.12:0/412520408 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda64104ad0 msgr2=0x7fda6419a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:39.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.957+0000 7fda63e00640 1 --2- 172.21.15.12:0/412520408 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda64104ad0 0x7fda6419a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:39.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.957+0000 7fda63e00640 1 -- 172.21.15.12:0/412520408 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda641a17f0 con 0x7fda641038e0 2024-03-23T23:13:39.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.957+0000 7fda5be00640 1 --2- 172.21.15.12:0/412520408 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda64104ad0 0x7fda6419a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:39.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.957+0000 7fda63e00640 1 --2- 172.21.15.12:0/412520408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda641038e0 0x7fda64199bd0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fda54009230 tx=0x7fda5402c5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:39.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.958+0000 7fda61600640 1 -- 172.21.15.12:0/412520408 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda54043020 con 0x7fda641038e0 2024-03-23T23:13:39.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.958+0000 7fda61600640 1 -- 172.21.15.12:0/412520408 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda540054e0 con 0x7fda641038e0 2024-03-23T23:13:39.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.958+0000 7fda6ac00640 1 -- 172.21.15.12:0/412520408 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda641a1a80 con 0x7fda641038e0 2024-03-23T23:13:39.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.958+0000 7fda61600640 1 -- 172.21.15.12:0/412520408 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda54003b60 con 0x7fda641038e0 2024-03-23T23:13:39.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.958+0000 7fda6ac00640 1 -- 172.21.15.12:0/412520408 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda641a1ff0 con 0x7fda641038e0 2024-03-23T23:13:39.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.959+0000 7fda6ac00640 1 -- 172.21.15.12:0/412520408 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda6410ac00 con 0x7fda641038e0 2024-03-23T23:13:39.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.960+0000 7fda61600640 1 -- 172.21.15.12:0/412520408 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fda54046020 con 0x7fda641038e0 2024-03-23T23:13:39.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.962+0000 7fda61600640 1 --2- 172.21.15.12:0/412520408 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fda480a2880 0x7fda480a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:39.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.962+0000 7fda61600640 1 -- 172.21.15.12:0/412520408 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fda540f09d0 con 0x7fda641038e0 2024-03-23T23:13:39.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.962+0000 7fda5be00640 1 --2- 172.21.15.12:0/412520408 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fda480a2880 0x7fda480a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:39.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.963+0000 7fda5be00640 1 --2- 172.21.15.12:0/412520408 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fda480a2880 0x7fda480a4d40 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fda6419b0e0 tx=0x7fda4c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:39.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:39.966+0000 7fda61600640 1 -- 172.21.15.12:0/412520408 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fda5404d020 con 0x7fda641038e0 2024-03-23T23:13:40.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:39 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2136752235' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:13:40.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:39 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3645828456' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:13:40.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:39 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2136752235' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:13:40.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:39 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3645828456' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:13:40.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:39 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2136752235' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:13:40.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:39 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3645828456' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:13:40.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.172+0000 7fda6ac00640 1 -- 172.21.15.12:0/412520408 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fda640622a0 con 0x7fda641038e0 2024-03-23T23:13:40.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.173+0000 7fda61600640 1 -- 172.21.15.12:0/412520408 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7fda540dbe20 con 0x7fda641038e0 2024-03-23T23:13:40.172 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-23T23:13:40.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.175+0000 7fda6ac00640 1 -- 172.21.15.12:0/412520408 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fda480a2880 msgr2=0x7fda480a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:40.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.175+0000 7fda6ac00640 1 --2- 172.21.15.12:0/412520408 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fda480a2880 0x7fda480a4d40 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fda6419b0e0 tx=0x7fda4c005f90 comp rx=0 tx=0).stop 2024-03-23T23:13:40.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.175+0000 7fda6ac00640 1 -- 172.21.15.12:0/412520408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda641038e0 msgr2=0x7fda64199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:40.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.175+0000 7fda6ac00640 1 --2- 172.21.15.12:0/412520408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda641038e0 0x7fda64199bd0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fda54009230 tx=0x7fda5402c5d0 comp rx=0 tx=0).stop 2024-03-23T23:13:40.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.176+0000 7fda6ac00640 1 -- 172.21.15.12:0/412520408 shutdown_connections 2024-03-23T23:13:40.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.176+0000 7fda6ac00640 1 --2- 172.21.15.12:0/412520408 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fda480a2880 0x7fda480a4d40 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:40.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.176+0000 7fda6ac00640 1 --2- 172.21.15.12:0/412520408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda64105490 0x7fda641a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:40.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.176+0000 7fda6ac00640 1 --2- 172.21.15.12:0/412520408 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda64104ad0 0x7fda6419a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:40.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.176+0000 7fda6ac00640 1 --2- 172.21.15.12:0/412520408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda641038e0 0x7fda64199bd0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:40.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.176+0000 7fda6ac00640 1 -- 172.21.15.12:0/412520408 >> 172.21.15.12:0/412520408 conn(0x7fda640fee50 msgr2=0x7fda64107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:40.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.176+0000 7fda6ac00640 1 -- 172.21.15.12:0/412520408 shutdown_connections 2024-03-23T23:13:40.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.176+0000 7fda6ac00640 1 -- 172.21.15.12:0/412520408 wait complete. 2024-03-23T23:13:40.193 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T23:13:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.420+0000 7f742f000640 1 -- 172.21.15.12:0/3229581117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7428077c40 msgr2=0x7f74280784e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.420+0000 7f742f000640 1 --2- 172.21.15.12:0/3229581117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7428077c40 0x7f74280784e0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f7420009860 tx=0x7f742002bb70 comp rx=0 tx=0).stop 2024-03-23T23:13:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.421+0000 7f742f000640 1 -- 172.21.15.12:0/3229581117 shutdown_connections 2024-03-23T23:13:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.421+0000 7f742f000640 1 --2- 172.21.15.12:0/3229581117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7428077c40 0x7f74280784e0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.421+0000 7f742f000640 1 --2- 172.21.15.12:0/3229581117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74280772a0 0x7f7428077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.421+0000 7f742f000640 1 --2- 172.21.15.12:0/3229581117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7428076060 0x7f7428076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.421+0000 7f742f000640 1 -- 172.21.15.12:0/3229581117 >> 172.21.15.12:0/3229581117 conn(0x7f74280fee50 msgr2=0x7f7428101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.421+0000 7f742f000640 1 -- 172.21.15.12:0/3229581117 shutdown_connections 2024-03-23T23:13:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.421+0000 7f742f000640 1 -- 172.21.15.12:0/3229581117 wait complete. 2024-03-23T23:13:40.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.422+0000 7f742f000640 1 Processor -- start 2024-03-23T23:13:40.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.422+0000 7f742f000640 1 -- start start 2024-03-23T23:13:40.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.422+0000 7f742f000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7428076060 0x7f742819df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:40.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.422+0000 7f742f000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74280772a0 0x7f742819e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:40.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.422+0000 7f742f000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7428077c40 0x7f74281a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:40.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.422+0000 7f742f000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7428113950 con 0x7f7428077c40 2024-03-23T23:13:40.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.422+0000 7f742f000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74281137d0 con 0x7f7428076060 2024-03-23T23:13:40.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.422+0000 7f742f000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7428113ad0 con 0x7f74280772a0 2024-03-23T23:13:40.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.423+0000 7f742d000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7428077c40 0x7f74281a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:40.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.423+0000 7f742d000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7428077c40 0x7f74281a54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57396/0 (socket says 172.21.15.12:57396) 2024-03-23T23:13:40.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.423+0000 7f742d000640 1 -- 172.21.15.12:0/330973968 learned_addr learned my addr 172.21.15.12:0/330973968 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:40.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.423+0000 7f7427400640 1 --2- 172.21.15.12:0/330973968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74280772a0 0x7f742819e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:40.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.423+0000 7f7427e00640 1 --2- 172.21.15.12:0/330973968 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7428076060 0x7f742819df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:40.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.423+0000 7f742d000640 1 -- 172.21.15.12:0/330973968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74280772a0 msgr2=0x7f742819e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:40.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.423+0000 7f742d000640 1 --2- 172.21.15.12:0/330973968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74280772a0 0x7f742819e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:40.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.423+0000 7f742d000640 1 -- 172.21.15.12:0/330973968 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7428076060 msgr2=0x7f742819df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:40.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.423+0000 7f742d000640 1 --2- 172.21.15.12:0/330973968 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7428076060 0x7f742819df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:40.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.423+0000 7f742d000640 1 -- 172.21.15.12:0/330973968 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74281a5b50 con 0x7f7428077c40 2024-03-23T23:13:40.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.424+0000 7f742d000640 1 --2- 172.21.15.12:0/330973968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7428077c40 0x7f74281a54e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f742002c180 tx=0x7f7420032460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:40.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.424+0000 7f7424c00640 1 -- 172.21.15.12:0/330973968 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7420041020 con 0x7f7428077c40 2024-03-23T23:13:40.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.424+0000 7f7424c00640 1 -- 172.21.15.12:0/330973968 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f742002c9f0 con 0x7f7428077c40 2024-03-23T23:13:40.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.424+0000 7f7424c00640 1 -- 172.21.15.12:0/330973968 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7420039040 con 0x7f7428077c40 2024-03-23T23:13:40.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.424+0000 7f742f000640 1 -- 172.21.15.12:0/330973968 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74281a5de0 con 0x7f7428077c40 2024-03-23T23:13:40.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.424+0000 7f742f000640 1 -- 172.21.15.12:0/330973968 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74281a6220 con 0x7f7428077c40 2024-03-23T23:13:40.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.425+0000 7f742f000640 1 -- 172.21.15.12:0/330973968 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7428110230 con 0x7f7428077c40 2024-03-23T23:13:40.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.427+0000 7f7424c00640 1 -- 172.21.15.12:0/330973968 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7420044020 con 0x7f7428077c40 2024-03-23T23:13:40.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.429+0000 7f7424c00640 1 --2- 172.21.15.12:0/330973968 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f740c0a2880 0x7f740c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:40.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.429+0000 7f7424c00640 1 -- 172.21.15.12:0/330973968 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f74200f0520 con 0x7f7428077c40 2024-03-23T23:13:40.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.429+0000 7f7427e00640 1 --2- 172.21.15.12:0/330973968 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f740c0a2880 0x7f740c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:40.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.430+0000 7f7427e00640 1 --2- 172.21.15.12:0/330973968 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f740c0a2880 0x7f740c0a4d40 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f7418005fd0 tx=0x7f7418009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:40.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.434+0000 7f7424c00640 1 -- 172.21.15.12:0/330973968 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f742004b020 con 0x7f7428077c40 2024-03-23T23:13:40.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.638+0000 7f742f000640 1 -- 172.21.15.12:0/330973968 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f74280622a0 con 0x7f7428077c40 2024-03-23T23:13:40.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.640+0000 7f7424c00640 1 -- 172.21.15.12:0/330973968 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f74200bd570 con 0x7f7428077c40 2024-03-23T23:13:40.638 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-23T23:13:40.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.642+0000 7f742f000640 1 -- 172.21.15.12:0/330973968 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f740c0a2880 msgr2=0x7f740c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:40.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.642+0000 7f742f000640 1 --2- 172.21.15.12:0/330973968 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f740c0a2880 0x7f740c0a4d40 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f7418005fd0 tx=0x7f7418009500 comp rx=0 tx=0).stop 2024-03-23T23:13:40.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.642+0000 7f742f000640 1 -- 172.21.15.12:0/330973968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7428077c40 msgr2=0x7f74281a54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:40.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.642+0000 7f742f000640 1 --2- 172.21.15.12:0/330973968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7428077c40 0x7f74281a54e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f742002c180 tx=0x7f7420032460 comp rx=0 tx=0).stop 2024-03-23T23:13:40.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.643+0000 7f742f000640 1 -- 172.21.15.12:0/330973968 shutdown_connections 2024-03-23T23:13:40.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.643+0000 7f742f000640 1 --2- 172.21.15.12:0/330973968 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f740c0a2880 0x7f740c0a4d40 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:40.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.643+0000 7f742f000640 1 --2- 172.21.15.12:0/330973968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7428077c40 0x7f74281a54e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:40.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.643+0000 7f742f000640 1 --2- 172.21.15.12:0/330973968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74280772a0 0x7f742819e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:40.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.643+0000 7f742f000640 1 --2- 172.21.15.12:0/330973968 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7428076060 0x7f742819df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:40.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.643+0000 7f742f000640 1 -- 172.21.15.12:0/330973968 >> 172.21.15.12:0/330973968 conn(0x7f74280fee50 msgr2=0x7f7428100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:40.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.643+0000 7f742f000640 1 -- 172.21.15.12:0/330973968 shutdown_connections 2024-03-23T23:13:40.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.643+0000 7f742f000640 1 -- 172.21.15.12:0/330973968 wait complete. 2024-03-23T23:13:40.663 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-23T23:13:40.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.900+0000 7fec8fe00640 1 -- 172.21.15.12:0/3627105843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec901014e0 msgr2=0x7fec90101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:40.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.900+0000 7fec8fe00640 1 --2- 172.21.15.12:0/3627105843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec901014e0 0x7fec90101900 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fec80009900 tx=0x7fec8002bc60 comp rx=0 tx=0).stop 2024-03-23T23:13:40.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.900+0000 7fec8fe00640 1 -- 172.21.15.12:0/3627105843 shutdown_connections 2024-03-23T23:13:40.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.900+0000 7fec8fe00640 1 --2- 172.21.15.12:0/3627105843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec90103090 0x7fec90109920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:40.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.900+0000 7fec8fe00640 1 --2- 172.21.15.12:0/3627105843 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec901026d0 0x7fec90102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:40.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.900+0000 7fec8fe00640 1 --2- 172.21.15.12:0/3627105843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec901014e0 0x7fec90101900 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:40.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.900+0000 7fec8fe00640 1 -- 172.21.15.12:0/3627105843 >> 172.21.15.12:0/3627105843 conn(0x7fec900fca50 msgr2=0x7fec900fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:40.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.900+0000 7fec8fe00640 1 -- 172.21.15.12:0/3627105843 shutdown_connections 2024-03-23T23:13:40.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.900+0000 7fec8fe00640 1 -- 172.21.15.12:0/3627105843 wait complete. 2024-03-23T23:13:40.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.901+0000 7fec8fe00640 1 Processor -- start 2024-03-23T23:13:40.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.901+0000 7fec8fe00640 1 -- start start 2024-03-23T23:13:40.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.902+0000 7fec8fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec901014e0 0x7fec90199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:40.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.902+0000 7fec8fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec901026d0 0x7fec90199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:40.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.902+0000 7fec8fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec90103090 0x7fec901a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:40.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.902+0000 7fec8fe00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9010d220 con 0x7fec90103090 2024-03-23T23:13:40.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.902+0000 7fec8fe00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9010d0a0 con 0x7fec901014e0 2024-03-23T23:13:40.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.902+0000 7fec8fe00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9010d3a0 con 0x7fec901026d0 2024-03-23T23:13:40.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.902+0000 7fec8ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec901014e0 0x7fec90199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:40.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.902+0000 7fec8f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec90103090 0x7fec901a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:40.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.902+0000 7fec8ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec901014e0 0x7fec90199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:38452/0 (socket says 172.21.15.12:38452) 2024-03-23T23:13:40.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.902+0000 7fec87e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec901026d0 0x7fec90199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:40.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.902+0000 7fec8ea00640 1 -- 172.21.15.12:0/2245875883 learned_addr learned my addr 172.21.15.12:0/2245875883 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:13:40.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.903+0000 7fec8ea00640 1 -- 172.21.15.12:0/2245875883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec901026d0 msgr2=0x7fec90199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:40.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.903+0000 7fec8ea00640 1 --2- 172.21.15.12:0/2245875883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec901026d0 0x7fec90199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:40.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.903+0000 7fec8ea00640 1 -- 172.21.15.12:0/2245875883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec90103090 msgr2=0x7fec901a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:40.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.903+0000 7fec8ea00640 1 --2- 172.21.15.12:0/2245875883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec90103090 0x7fec901a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:40.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.903+0000 7fec8ea00640 1 -- 172.21.15.12:0/2245875883 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec901a16c0 con 0x7fec901014e0 2024-03-23T23:13:40.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.903+0000 7fec87e00640 1 --2- 172.21.15.12:0/2245875883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec901026d0 0x7fec90199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:13:40.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.903+0000 7fec8ea00640 1 --2- 172.21.15.12:0/2245875883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec901014e0 0x7fec90199aa0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fec8002efd0 tx=0x7fec80005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:40.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.903+0000 7fec87400640 1 -- 172.21.15.12:0/2245875883 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec80041070 con 0x7fec901014e0 2024-03-23T23:13:40.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.903+0000 7fec87400640 1 -- 172.21.15.12:0/2245875883 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec800054e0 con 0x7fec901014e0 2024-03-23T23:13:40.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.903+0000 7fec8fe00640 1 -- 172.21.15.12:0/2245875883 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec901a1950 con 0x7fec901014e0 2024-03-23T23:13:40.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.903+0000 7fec87400640 1 -- 172.21.15.12:0/2245875883 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec80034440 con 0x7fec901014e0 2024-03-23T23:13:40.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.903+0000 7fec8fe00640 1 -- 172.21.15.12:0/2245875883 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec901a1e10 con 0x7fec901014e0 2024-03-23T23:13:40.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.905+0000 7fec8fe00640 1 -- 172.21.15.12:0/2245875883 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec901a1fe0 con 0x7fec901014e0 2024-03-23T23:13:40.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.906+0000 7fec87400640 1 -- 172.21.15.12:0/2245875883 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fec80044070 con 0x7fec901014e0 2024-03-23T23:13:40.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.908+0000 7fec87400640 1 --2- 172.21.15.12:0/2245875883 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fec740a2800 0x7fec740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:13:40.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.908+0000 7fec87400640 1 -- 172.21.15.12:0/2245875883 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fec800f0a20 con 0x7fec901014e0 2024-03-23T23:13:40.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.908+0000 7fec87e00640 1 --2- 172.21.15.12:0/2245875883 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fec740a2800 0x7fec740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:13:40.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.909+0000 7fec87e00640 1 --2- 172.21.15.12:0/2245875883 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fec740a2800 0x7fec740a4cc0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fec9019afb0 tx=0x7fec780090e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:13:40.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:40.912+0000 7fec87400640 1 -- 172.21.15.12:0/2245875883 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fec800bd420 con 0x7fec901014e0 2024-03-23T23:13:41.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:40 smithi012 ceph-mon[31695]: pgmap v746: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-03-23T23:13:41.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:40 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/412520408' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:13:41.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:40 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/330973968' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:13:41.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:40 smithi173 ceph-mon[36116]: pgmap v746: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-03-23T23:13:41.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:40 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/412520408' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:13:41.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:40 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/330973968' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:13:41.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:40 smithi203 ceph-mon[36693]: pgmap v746: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-03-23T23:13:41.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:40 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/412520408' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:13:41.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:40 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/330973968' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:13:41.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:41.165+0000 7fec8fe00640 1 -- 172.21.15.12:0/2245875883 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7fec901a2760 con 0x7fec901014e0 2024-03-23T23:13:41.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:41.175+0000 7fec87400640 1 -- 172.21.15.12:0/2245875883 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v18) v1 ==== 81+0+739831 (secure 0 0 0) 0x7fec800c14c0 con 0x7fec901014e0 2024-03-23T23:13:41.187 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:13:41.187 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:13:41.187 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 18, 2024-03-23T23:13:41.187 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.187 INFO:teuthology.orchestra.run.smithi012.stdout: "active_gid": 14152, 2024-03-23T23:13:41.187 INFO:teuthology.orchestra.run.smithi012.stdout: "active_name": "x", 2024-03-23T23:13:41.187 INFO:teuthology.orchestra.run.smithi012.stdout: "active_addrs": { 2024-03-23T23:13:41.187 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-03-23T23:13:41.187 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.187 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-03-23T23:13:41.187 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6800", 2024-03-23T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 1345467734 2024-03-23T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-03-23T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6801", 2024-03-23T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 1345467734 2024-03-23T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-23T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stdout: "active_addr": "172.21.15.12:6801/1345467734", 2024-03-23T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stdout: "active_change": "2024-03-23T22:51:23.113484+0000", 2024-03-23T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stdout: "active_mgr_features": 4540701547738038271, 2024-03-23T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stdout: "available": true, 2024-03-23T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stdout: "standbys": [ 2024-03-23T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stdout: "gid": 24112, 2024-03-23T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "z", 2024-03-23T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr_features": 4540701547738038271, 2024-03-23T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stdout: "available_modules": [ 2024-03-23T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "alerts", 2024-03-23T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-23T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-23T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-23T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How frequently to reexamine health status", 2024-03-23T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.192 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.192 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.192 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.192 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.192 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.192 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.192 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.192 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.192 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.192 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.192 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.192 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.192 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.192 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.194 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.194 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.194 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.194 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.194 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.194 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.194 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.194 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.194 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.194 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.194 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.194 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.194 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.195 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.195 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.195 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.195 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.195 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.195 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.195 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.195 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.195 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_destination": { 2024-03-23T23:13:41.195 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_destination", 2024-03-23T23:13:41.195 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.195 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.195 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.195 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.195 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Email address to send alerts to", 2024-03-23T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_from_name": { 2024-03-23T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_from_name", 2024-03-23T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "Ceph", 2024-03-23T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Email From: name", 2024-03-23T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_host": { 2024-03-23T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_host", 2024-03-23T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP server", 2024-03-23T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_password": { 2024-03-23T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_password", 2024-03-23T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Password to authenticate with", 2024-03-23T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_port": { 2024-03-23T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_port", 2024-03-23T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "465", 2024-03-23T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP port", 2024-03-23T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_sender": { 2024-03-23T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_sender", 2024-03-23T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP envelope sender", 2024-03-23T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_ssl": { 2024-03-23T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_ssl", 2024-03-23T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-23T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.202 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.202 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_user": { 2024-03-23T23:13:41.202 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_user", 2024-03-23T23:13:41.202 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.202 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.202 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.202 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.202 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.202 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.202 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.202 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "User to authenticate as", 2024-03-23T23:13:41.202 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.202 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.203 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.203 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.203 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.203 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.203 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.203 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "balancer", 2024-03-23T23:13:41.203 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.203 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.203 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.203 INFO:teuthology.orchestra.run.smithi012.stdout: "active": { 2024-03-23T23:13:41.203 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "active", 2024-03-23T23:13:41.203 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.203 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically balance PGs across cluster", 2024-03-23T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stdout: "begin_time": { 2024-03-23T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "begin_time", 2024-03-23T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.205 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.205 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0000", 2024-03-23T23:13:41.205 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.205 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.205 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.205 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "beginning time of day to automatically balance", 2024-03-23T23:13:41.205 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-23T23:13:41.205 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.205 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.205 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.205 INFO:teuthology.orchestra.run.smithi012.stdout: "begin_weekday": { 2024-03-23T23:13:41.205 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "begin_weekday", 2024-03-23T23:13:41.205 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-23T23:13:41.205 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.205 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.206 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-23T23:13:41.206 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0", 2024-03-23T23:13:41.206 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "6", 2024-03-23T23:13:41.206 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.206 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-23T23:13:41.206 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-23T23:13:41.206 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.206 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.206 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.206 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_max_iterations": { 2024-03-23T23:13:41.206 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_max_iterations", 2024-03-23T23:13:41.206 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-23T23:13:41.206 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.206 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.207 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "25", 2024-03-23T23:13:41.207 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-03-23T23:13:41.207 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "250", 2024-03-23T23:13:41.207 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.207 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-23T23:13:41.207 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.207 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.207 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.207 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.207 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_metrics": { 2024-03-23T23:13:41.207 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_metrics", 2024-03-23T23:13:41.207 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.207 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.207 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.207 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "pgs,objects,bytes", 2024-03-23T23:13:41.207 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.208 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.208 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.208 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-23T23:13:41.208 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-23T23:13:41.208 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.208 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.208 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.208 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_step": { 2024-03-23T23:13:41.208 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_step", 2024-03-23T23:13:41.208 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-23T23:13:41.208 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.208 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.208 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0.5", 2024-03-23T23:13:41.208 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0.001", 2024-03-23T23:13:41.208 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "0.999", 2024-03-23T23:13:41.209 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.209 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "aggressiveness of optimization", 2024-03-23T23:13:41.209 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-23T23:13:41.209 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.209 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.209 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.209 INFO:teuthology.orchestra.run.smithi012.stdout: "end_time": { 2024-03-23T23:13:41.209 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "end_time", 2024-03-23T23:13:41.209 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.209 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.209 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.209 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "2359", 2024-03-23T23:13:41.209 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.209 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.210 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.210 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "ending time of day to automatically balance", 2024-03-23T23:13:41.210 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-23T23:13:41.210 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.210 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.210 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.210 INFO:teuthology.orchestra.run.smithi012.stdout: "end_weekday": { 2024-03-23T23:13:41.210 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "end_weekday", 2024-03-23T23:13:41.210 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-23T23:13:41.210 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.210 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.210 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-23T23:13:41.210 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0", 2024-03-23T23:13:41.210 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "6", 2024-03-23T23:13:41.211 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.211 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-23T23:13:41.211 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-23T23:13:41.211 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.211 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.211 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.211 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.211 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.211 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.211 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.211 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.211 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.211 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.211 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.211 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.212 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.212 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.212 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.212 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.212 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.212 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.212 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.212 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.212 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.212 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.212 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.212 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.212 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.212 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.212 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.213 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.213 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.213 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.213 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.213 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.213 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.213 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.213 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.213 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.213 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.213 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.213 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.213 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.213 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.214 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.214 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.214 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.214 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.214 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.214 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.214 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.214 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.214 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.214 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.214 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.214 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.214 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.214 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.215 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.215 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.215 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.215 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.215 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.215 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.215 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.215 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.215 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.215 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.215 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.215 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.215 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.215 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.216 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.216 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.216 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.216 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.216 INFO:teuthology.orchestra.run.smithi012.stdout: "min_score": { 2024-03-23T23:13:41.216 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "min_score", 2024-03-23T23:13:41.216 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-23T23:13:41.216 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.216 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.216 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-23T23:13:41.216 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.216 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.216 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.216 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-23T23:13:41.216 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.217 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.217 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.217 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.217 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": { 2024-03-23T23:13:41.217 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mode", 2024-03-23T23:13:41.217 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.217 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.217 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.217 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "upmap", 2024-03-23T23:13:41.217 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.217 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.217 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.217 INFO:teuthology.orchestra.run.smithi012.stdout: "crush-compat", 2024-03-23T23:13:41.217 INFO:teuthology.orchestra.run.smithi012.stdout: "none", 2024-03-23T23:13:41.217 INFO:teuthology.orchestra.run.smithi012.stdout: "read", 2024-03-23T23:13:41.218 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap", 2024-03-23T23:13:41.218 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap-read" 2024-03-23T23:13:41.218 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.218 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Balancer mode", 2024-03-23T23:13:41.218 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.218 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.218 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.218 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.218 INFO:teuthology.orchestra.run.smithi012.stdout: "pool_ids": { 2024-03-23T23:13:41.218 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pool_ids", 2024-03-23T23:13:41.218 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.218 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.218 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.218 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.219 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.219 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.219 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.219 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-23T23:13:41.219 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.219 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.219 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.219 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.219 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-23T23:13:41.219 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-23T23:13:41.219 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.219 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.219 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.219 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-23T23:13:41.219 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.220 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.220 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.220 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-23T23:13:41.220 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.220 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.220 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.220 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.220 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap_max_deviation": { 2024-03-23T23:13:41.220 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "upmap_max_deviation", 2024-03-23T23:13:41.220 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.220 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.220 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.220 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-03-23T23:13:41.220 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-03-23T23:13:41.220 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.221 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.221 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-23T23:13:41.221 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-23T23:13:41.221 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.221 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.221 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.221 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap_max_optimizations": { 2024-03-23T23:13:41.221 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "upmap_max_optimizations", 2024-03-23T23:13:41.221 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-23T23:13:41.221 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.221 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.221 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-23T23:13:41.221 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.221 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.221 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.222 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-23T23:13:41.222 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.222 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.222 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.222 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.222 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.222 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.222 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.222 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephadm", 2024-03-23T23:13:41.222 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.222 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.222 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.225 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_down_multiplier": { 2024-03-23T23:13:41.225 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_down_multiplier", 2024-03-23T23:13:41.225 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-23T23:13:41.225 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.225 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.225 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3.0", 2024-03-23T23:13:41.225 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.225 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.226 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.226 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-23T23:13:41.226 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.226 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.226 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.226 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.226 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_refresh_rate": { 2024-03-23T23:13:41.226 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_refresh_rate", 2024-03-23T23:13:41.226 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.226 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.226 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.226 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "20", 2024-03-23T23:13:41.226 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.226 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.227 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.227 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-23T23:13:41.227 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.227 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.227 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.227 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.227 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_starting_port": { 2024-03-23T23:13:41.227 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_starting_port", 2024-03-23T23:13:41.227 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.227 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.227 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.227 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "4721", 2024-03-23T23:13:41.227 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.227 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.227 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.228 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-23T23:13:41.228 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.228 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.228 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.228 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.228 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_ptrace": { 2024-03-23T23:13:41.228 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_ptrace", 2024-03-23T23:13:41.228 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.228 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.228 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.228 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.228 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.228 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.229 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.229 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-23T23:13:41.229 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-23T23:13:41.229 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.229 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.229 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.229 INFO:teuthology.orchestra.run.smithi012.stdout: "autotune_interval": { 2024-03-23T23:13:41.229 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "autotune_interval", 2024-03-23T23:13:41.229 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.229 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.229 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.229 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-23T23:13:41.229 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.229 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.229 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-23T23:13:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: "autotune_memory_target_ratio": { 2024-03-23T23:13:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "autotune_memory_target_ratio", 2024-03-23T23:13:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-23T23:13:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0.7", 2024-03-23T23:13:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-23T23:13:41.231 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.231 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.231 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.231 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.231 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm_log_destination": { 2024-03-23T23:13:41.231 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephadm_log_destination", 2024-03-23T23:13:41.231 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.231 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.231 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.231 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.231 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.231 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.231 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.231 INFO:teuthology.orchestra.run.smithi012.stdout: "file", 2024-03-23T23:13:41.231 INFO:teuthology.orchestra.run.smithi012.stdout: "file,syslog", 2024-03-23T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stdout: "syslog" 2024-03-23T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-23T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stdout: "cgroups_split": { 2024-03-23T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cgroups_split", 2024-03-23T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.233 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-23T23:13:41.233 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.233 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.233 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.233 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.233 INFO:teuthology.orchestra.run.smithi012.stdout: "config_checks_enabled": { 2024-03-23T23:13:41.233 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_checks_enabled", 2024-03-23T23:13:41.233 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.233 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.233 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.233 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.233 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.233 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.233 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.233 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-23T23:13:41.234 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.234 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.234 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.234 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.234 INFO:teuthology.orchestra.run.smithi012.stdout: "config_dashboard": { 2024-03-23T23:13:41.234 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_dashboard", 2024-03-23T23:13:41.234 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.234 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.234 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.234 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.234 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.234 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.234 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.234 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-23T23:13:41.235 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.235 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.235 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.235 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.235 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_alertmanager": { 2024-03-23T23:13:41.235 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_alertmanager", 2024-03-23T23:13:41.235 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.235 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.235 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.235 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-23T23:13:41.235 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.235 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.235 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.235 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-03-23T23:13:41.235 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.235 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.236 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.236 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.236 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_base": { 2024-03-23T23:13:41.236 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_base", 2024-03-23T23:13:41.236 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.236 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.236 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.236 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-23T23:13:41.236 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.236 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.236 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.236 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Container image name, without the tag", 2024-03-23T23:13:41.236 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.236 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.236 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.237 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.237 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_elasticsearch": { 2024-03-23T23:13:41.237 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_elasticsearch", 2024-03-23T23:13:41.237 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.237 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.237 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.237 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-23T23:13:41.237 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.237 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.237 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.237 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "elasticsearch container image", 2024-03-23T23:13:41.237 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.237 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.237 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.237 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.238 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_grafana": { 2024-03-23T23:13:41.238 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_grafana", 2024-03-23T23:13:41.238 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.238 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.238 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.238 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-23T23:13:41.238 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.238 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.238 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.238 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-03-23T23:13:41.238 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.238 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.238 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.238 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.239 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_haproxy": { 2024-03-23T23:13:41.239 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_haproxy", 2024-03-23T23:13:41.239 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.239 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.239 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.239 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-23T23:13:41.239 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.239 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.239 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.239 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "HAproxy container image", 2024-03-23T23:13:41.239 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.239 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.239 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.239 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.239 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_agent": { 2024-03-23T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_agent", 2024-03-23T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-23T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger agent container image", 2024-03-23T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_collector": { 2024-03-23T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_collector", 2024-03-23T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-23T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger collector container image", 2024-03-23T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_query": { 2024-03-23T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_query", 2024-03-23T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.242 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.242 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-23T23:13:41.242 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.242 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.242 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.242 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger query container image", 2024-03-23T23:13:41.242 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.242 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.242 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.242 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.242 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_keepalived": { 2024-03-23T23:13:41.242 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_keepalived", 2024-03-23T23:13:41.242 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.242 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-23T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Keepalived container image", 2024-03-23T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_loki": { 2024-03-23T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_loki", 2024-03-23T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-23T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Loki container image", 2024-03-23T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_node_exporter": { 2024-03-23T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_node_exporter", 2024-03-23T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-23T23:13:41.245 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.245 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.245 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.245 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-03-23T23:13:41.245 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.245 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.245 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.245 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.245 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_nvmeof": { 2024-03-23T23:13:41.245 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_nvmeof", 2024-03-23T23:13:41.245 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.245 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.245 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.245 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-23T23:13:41.246 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.246 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.246 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.246 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Nvme-of container image", 2024-03-23T23:13:41.246 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.246 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.246 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.246 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.246 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_prometheus": { 2024-03-23T23:13:41.246 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_prometheus", 2024-03-23T23:13:41.246 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.246 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.246 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.246 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-23T23:13:41.247 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.247 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.247 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.247 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-03-23T23:13:41.247 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.247 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.247 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.247 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.247 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_promtail": { 2024-03-23T23:13:41.247 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_promtail", 2024-03-23T23:13:41.247 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.247 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.247 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.247 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-23T23:13:41.248 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.248 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.248 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.248 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Promtail container image", 2024-03-23T23:13:41.248 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.248 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.248 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.248 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.248 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_snmp_gateway": { 2024-03-23T23:13:41.248 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_snmp_gateway", 2024-03-23T23:13:41.248 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.248 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.248 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.248 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-23T23:13:41.249 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.249 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.249 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.249 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SNMP Gateway container image", 2024-03-23T23:13:41.249 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.249 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.249 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.249 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.249 INFO:teuthology.orchestra.run.smithi012.stdout: "container_init": { 2024-03-23T23:13:41.249 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_init", 2024-03-23T23:13:41.249 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.249 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.249 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.249 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.249 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Run podman/docker with `--init`", 2024-03-23T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stdout: "daemon_cache_timeout": { 2024-03-23T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "daemon_cache_timeout", 2024-03-23T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-23T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.251 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.251 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.251 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-23T23:13:41.251 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.251 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.251 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.251 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.251 INFO:teuthology.orchestra.run.smithi012.stdout: "default_cephadm_command_timeout": { 2024-03-23T23:13:41.251 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default_cephadm_command_timeout", 2024-03-23T23:13:41.251 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.251 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.251 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.251 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "900", 2024-03-23T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-23T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stdout: "default_registry": { 2024-03-23T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default_registry", 2024-03-23T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io", 2024-03-23T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-23T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stdout: "device_cache_timeout": { 2024-03-23T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_cache_timeout", 2024-03-23T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1800", 2024-03-23T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache device inventory", 2024-03-23T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stdout: "device_enhanced_scan": { 2024-03-23T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_enhanced_scan", 2024-03-23T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-23T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stdout: "facts_cache_timeout": { 2024-03-23T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "facts_cache_timeout", 2024-03-23T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-23T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache host facts data", 2024-03-23T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stdout: "host_check_interval": { 2024-03-23T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "host_check_interval", 2024-03-23T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-23T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to perform a host check", 2024-03-23T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stdout: "hw_monitoring": { 2024-03-23T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "hw_monitoring", 2024-03-23T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-23T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.258 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.258 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.258 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.258 INFO:teuthology.orchestra.run.smithi012.stdout: "inventory_list_all": { 2024-03-23T23:13:41.258 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "inventory_list_all", 2024-03-23T23:13:41.258 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.258 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.258 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.258 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.258 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.258 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.258 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.258 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-23T23:13:41.258 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stdout: "log_refresh_metadata": { 2024-03-23T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_refresh_metadata", 2024-03-23T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.261 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.261 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.261 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-23T23:13:41.261 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.261 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.261 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.261 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.261 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.261 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.261 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.261 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.261 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.261 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.262 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.262 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.262 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-23T23:13:41.262 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.262 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.262 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.262 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.262 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.262 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.262 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.262 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.262 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.262 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.262 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.263 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.263 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.263 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.263 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.263 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.263 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.263 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.263 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.263 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.263 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.263 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.263 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.263 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.263 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.264 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.264 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.264 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.264 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.264 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.264 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.264 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.264 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.264 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.264 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.264 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.264 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.264 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.264 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.265 INFO:teuthology.orchestra.run.smithi012.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-23T23:13:41.265 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-23T23:13:41.265 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.265 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.265 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.265 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.265 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.265 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.265 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.265 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-23T23:13:41.265 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.265 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.265 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.265 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.265 INFO:teuthology.orchestra.run.smithi012.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-23T23:13:41.266 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-23T23:13:41.266 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.266 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.266 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.266 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "*", 2024-03-23T23:13:41.266 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.266 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.266 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.266 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-23T23:13:41.266 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.266 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.266 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.266 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.267 INFO:teuthology.orchestra.run.smithi012.stdout: "max_count_per_host": { 2024-03-23T23:13:41.267 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_count_per_host", 2024-03-23T23:13:41.267 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.267 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.267 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.267 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-23T23:13:41.267 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.267 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.267 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.267 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "max number of daemons per service per host", 2024-03-23T23:13:41.267 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.267 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.267 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.267 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.268 INFO:teuthology.orchestra.run.smithi012.stdout: "max_osd_draining_count": { 2024-03-23T23:13:41.268 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_osd_draining_count", 2024-03-23T23:13:41.268 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.268 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.268 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.268 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-23T23:13:41.268 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.268 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.268 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.268 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-23T23:13:41.268 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.268 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.268 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.268 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.268 INFO:teuthology.orchestra.run.smithi012.stdout: "migration_current": { 2024-03-23T23:13:41.269 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "migration_current", 2024-03-23T23:13:41.269 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.269 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.269 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.269 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.269 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.269 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.269 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.269 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "internal - do not modify", 2024-03-23T23:13:41.269 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.269 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.269 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.269 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.269 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": { 2024-03-23T23:13:41.269 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mode", 2024-03-23T23:13:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "root", 2024-03-23T23:13:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm-package", 2024-03-23T23:13:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: "root" 2024-03-23T23:13:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "mode for remote execution of cephadm", 2024-03-23T23:13:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: "oob_default_addr": { 2024-03-23T23:13:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "oob_default_addr", 2024-03-23T23:13:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "169.254.1.1", 2024-03-23T23:13:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-23T23:13:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "prometheus_alerts_path": { 2024-03-23T23:13:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prometheus_alerts_path", 2024-03-23T23:13:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-23T23:13:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-23T23:13:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_insecure": { 2024-03-23T23:13:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_insecure", 2024-03-23T23:13:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-23T23:13:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_password": { 2024-03-23T23:13:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_password", 2024-03-23T23:13:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-23T23:13:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_url": { 2024-03-23T23:13:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_url", 2024-03-23T23:13:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-23T23:13:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_username": { 2024-03-23T23:13:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_username", 2024-03-23T23:13:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-23T23:13:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: "secure_monitoring_stack": { 2024-03-23T23:13:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "secure_monitoring_stack", 2024-03-23T23:13:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-23T23:13:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: "service_discovery_port": { 2024-03-23T23:13:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "service_discovery_port", 2024-03-23T23:13:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.278 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.278 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.278 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8765", 2024-03-23T23:13:41.278 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.278 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.278 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.278 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "cephadm service discovery port", 2024-03-23T23:13:41.278 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.278 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.278 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.278 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.278 INFO:teuthology.orchestra.run.smithi012.stdout: "ssh_config_file": { 2024-03-23T23:13:41.278 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssh_config_file", 2024-03-23T23:13:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-23T23:13:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: "use_agent": { 2024-03-23T23:13:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "use_agent", 2024-03-23T23:13:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-23T23:13:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: "use_repo_digest": { 2024-03-23T23:13:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "use_repo_digest", 2024-03-23T23:13:41.281 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.281 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.281 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.281 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.281 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.281 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.281 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.281 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-23T23:13:41.281 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.281 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.281 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.281 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.281 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_failed_host_check": { 2024-03-23T23:13:41.281 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_failed_host_check", 2024-03-23T23:13:41.281 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.282 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.282 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.282 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.282 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.282 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.282 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.282 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if the host check fails", 2024-03-23T23:13:41.282 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.282 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.282 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.282 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.282 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_stray_daemons": { 2024-03-23T23:13:41.282 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_stray_daemons", 2024-03-23T23:13:41.282 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.282 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.283 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.283 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.283 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.283 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.283 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.283 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-23T23:13:41.283 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.283 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.283 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.283 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.283 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_stray_hosts": { 2024-03-23T23:13:41.283 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_stray_hosts", 2024-03-23T23:13:41.283 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.283 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.283 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.284 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.284 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.284 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.284 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.284 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-23T23:13:41.284 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.284 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.284 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.284 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.284 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.284 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.284 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.284 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crash", 2024-03-23T23:13:41.284 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.284 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.284 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.285 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.285 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.285 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.285 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.285 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.285 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.285 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.285 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.285 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.285 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.285 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.285 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.285 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.285 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.285 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.285 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.286 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.286 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.286 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.286 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.286 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.286 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.286 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.286 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.286 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.286 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.286 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.286 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.286 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.286 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.287 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.287 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.287 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.287 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.287 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.287 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.287 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.287 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.287 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.287 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.287 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.287 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.287 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.287 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.287 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.287 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.288 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.288 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.288 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.288 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.288 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.288 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.288 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.288 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.288 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.288 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.288 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.288 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.288 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.288 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.288 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.288 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.289 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.289 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.289 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.289 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.289 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.289 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.289 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.289 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.289 INFO:teuthology.orchestra.run.smithi012.stdout: "retain_interval": { 2024-03-23T23:13:41.289 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "retain_interval", 2024-03-23T23:13:41.289 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.289 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.289 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.289 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "31536000", 2024-03-23T23:13:41.289 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.289 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.289 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.290 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-23T23:13:41.290 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.290 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.290 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.290 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.290 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_recent_interval": { 2024-03-23T23:13:41.290 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_recent_interval", 2024-03-23T23:13:41.290 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.290 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.290 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.290 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1209600", 2024-03-23T23:13:41.290 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.290 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.290 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.290 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-23T23:13:41.291 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.291 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.291 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.291 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.291 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.291 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.291 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.291 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "dashboard", 2024-03-23T23:13:41.291 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.291 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.291 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.291 INFO:teuthology.orchestra.run.smithi012.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-23T23:13:41.291 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-23T23:13:41.291 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.292 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.292 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.292 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-23T23:13:41.292 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.292 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.292 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.292 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.292 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.292 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.292 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.292 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.292 INFO:teuthology.orchestra.run.smithi012.stdout: "ALERTMANAGER_API_HOST": { 2024-03-23T23:13:41.292 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-23T23:13:41.292 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.292 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.292 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.293 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.293 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.293 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.293 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.293 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.293 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.293 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.293 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.293 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.293 INFO:teuthology.orchestra.run.smithi012.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-23T23:13:41.293 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-23T23:13:41.293 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.293 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.293 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.293 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.293 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.294 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.294 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.294 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.294 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.294 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.294 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.294 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.294 INFO:teuthology.orchestra.run.smithi012.stdout: "AUDIT_API_ENABLED": { 2024-03-23T23:13:41.294 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "AUDIT_API_ENABLED", 2024-03-23T23:13:41.294 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.294 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.294 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.294 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.294 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.295 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.295 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.295 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.295 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.295 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.295 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.295 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.295 INFO:teuthology.orchestra.run.smithi012.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-23T23:13:41.295 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-23T23:13:41.295 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.295 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.295 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.295 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.295 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.295 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.295 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.295 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.296 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.296 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.296 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.296 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.296 INFO:teuthology.orchestra.run.smithi012.stdout: "ENABLE_BROWSABLE_API": { 2024-03-23T23:13:41.296 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-23T23:13:41.296 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.296 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.296 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.296 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.296 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.296 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.296 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.296 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.296 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.296 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.297 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.297 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.297 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-23T23:13:41.297 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-23T23:13:41.297 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.297 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.297 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.297 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.297 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.297 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.297 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.297 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.297 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.297 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.297 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.297 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.297 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-23T23:13:41.298 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-23T23:13:41.298 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.298 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.298 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.298 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.298 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.298 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.298 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.298 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.298 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.298 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.298 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.298 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.298 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-23T23:13:41.299 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-23T23:13:41.299 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.299 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.299 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.299 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.299 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.299 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.299 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.299 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.299 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.299 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.299 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.299 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.299 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-23T23:13:41.299 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-23T23:13:41.299 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.300 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.300 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.300 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.300 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.300 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.300 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.300 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.300 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.300 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.300 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.300 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.300 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-23T23:13:41.300 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-23T23:13:41.300 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.300 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.300 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.301 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.301 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.301 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.301 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.301 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.301 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.301 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.301 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.301 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.301 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-23T23:13:41.301 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-23T23:13:41.301 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.301 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.301 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.301 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.301 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.302 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.302 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.302 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.302 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.302 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.302 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.302 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.302 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-23T23:13:41.302 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-23T23:13:41.302 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.302 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.302 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.302 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.302 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.302 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.302 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.303 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.303 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.303 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.303 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.303 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.303 INFO:teuthology.orchestra.run.smithi012.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-23T23:13:41.303 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-23T23:13:41.303 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.303 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.303 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.303 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.303 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.303 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.303 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.304 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.304 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.304 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.304 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.304 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.304 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_PASSWORD": { 2024-03-23T23:13:41.304 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-23T23:13:41.304 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.304 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.304 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.304 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-03-23T23:13:41.304 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.304 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.304 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.305 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.305 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.305 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.305 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.305 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.305 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-23T23:13:41.305 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-23T23:13:41.305 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.305 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.305 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.305 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.305 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.305 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.305 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.305 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.306 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.306 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.306 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.306 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.306 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_URL": { 2024-03-23T23:13:41.306 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_URL", 2024-03-23T23:13:41.306 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.306 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.306 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.306 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.306 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.306 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.306 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.306 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.306 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.307 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.307 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.307 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.307 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_USERNAME": { 2024-03-23T23:13:41.307 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-23T23:13:41.307 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.307 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.307 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.307 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-03-23T23:13:41.307 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.307 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.307 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.307 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.307 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.307 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.307 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.308 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.308 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-23T23:13:41.308 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-23T23:13:41.308 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.308 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.308 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.308 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.308 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.308 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.308 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.308 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.308 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.308 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.308 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.308 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.309 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-23T23:13:41.309 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-23T23:13:41.309 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.309 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.309 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.309 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.309 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.309 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.309 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.309 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.309 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.309 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.309 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.309 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.309 INFO:teuthology.orchestra.run.smithi012.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-23T23:13:41.310 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-23T23:13:41.310 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.310 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.310 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.310 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.310 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.310 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.310 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.310 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.310 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.310 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.310 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.310 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-23T23:13:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-23T23:13:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "MULTICLUSTER_CONFIG": { 2024-03-23T23:13:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-23T23:13:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "{}", 2024-03-23T23:13:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "PROMETHEUS_API_HOST": { 2024-03-23T23:13:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-23T23:13:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-23T23:13:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-23T23:13:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-23T23:13:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-23T23:13:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-23T23:13:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-23T23:13:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-23T23:13:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-23T23:13:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-23T23:13:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-23T23:13:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-23T23:13:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-23T23:13:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-23T23:13:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-23T23:13:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-23T23:13:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-23T23:13:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_ENABLED": { 2024-03-23T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-23T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-23T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-23T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-23T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-23T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-23T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-23T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-23T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-23T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.324 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8", 2024-03-23T23:13:41.324 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.324 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.324 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.324 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.324 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.324 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.324 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.324 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.324 INFO:teuthology.orchestra.run.smithi012.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-23T23:13:41.324 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-23T23:13:41.324 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.324 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "45", 2024-03-23T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_ACCESS_KEY": { 2024-03-23T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-23T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-23T23:13:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-23T23:13:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-03-23T23:13:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_SECRET_KEY": { 2024-03-23T23:13:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-23T23:13:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_SSL_VERIFY": { 2024-03-23T23:13:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-23T23:13:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "UNSAFE_TLS_v1_2": { 2024-03-23T23:13:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-23T23:13:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-23T23:13:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-23T23:13:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-23T23:13:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-23T23:13:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-23T23:13:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-23T23:13:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-23T23:13:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-23T23:13:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-03-23T23:13:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: "cross_origin_url": { 2024-03-23T23:13:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cross_origin_url", 2024-03-23T23:13:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: "crt_file": { 2024-03-23T23:13:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crt_file", 2024-03-23T23:13:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: "debug": { 2024-03-23T23:13:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "debug", 2024-03-23T23:13:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable/disable debug options", 2024-03-23T23:13:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: "jwt_token_ttl": { 2024-03-23T23:13:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "jwt_token_ttl", 2024-03-23T23:13:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "28800", 2024-03-23T23:13:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: "key_file": { 2024-03-23T23:13:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "key_file", 2024-03-23T23:13:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.338 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.338 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.338 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.338 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.338 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.338 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.338 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.338 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.338 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.338 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.338 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.338 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.338 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.338 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.338 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.339 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.339 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.339 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.339 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.339 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.339 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.339 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.339 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.339 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.339 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.339 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.339 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.339 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.339 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.341 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.341 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.341 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.341 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.341 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.341 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.341 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.341 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.341 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.341 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.341 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.341 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.341 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.341 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.341 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.342 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.342 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.342 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.342 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.342 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.342 INFO:teuthology.orchestra.run.smithi012.stdout: "motd": { 2024-03-23T23:13:41.342 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "motd", 2024-03-23T23:13:41.342 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.342 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.342 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.342 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.342 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.342 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.343 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.343 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "The message of the day", 2024-03-23T23:13:41.343 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.343 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.343 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.343 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.343 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect_resolve_ip_addr": { 2024-03-23T23:13:41.343 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "redirect_resolve_ip_addr", 2024-03-23T23:13:41.343 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.343 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.343 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.343 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.343 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.343 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.344 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.344 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.344 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.344 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.344 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.344 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.344 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-03-23T23:13:41.344 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-03-23T23:13:41.344 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.344 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.344 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.344 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "::", 2024-03-23T23:13:41.344 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.344 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.344 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.345 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.345 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.345 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.345 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.345 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.345 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-03-23T23:13:41.345 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-03-23T23:13:41.345 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.345 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.345 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.345 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8080", 2024-03-23T23:13:41.345 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.345 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.345 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.345 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.346 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.346 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.346 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.346 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.346 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl": { 2024-03-23T23:13:41.346 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl", 2024-03-23T23:13:41.346 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.346 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.346 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.346 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.346 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.346 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.346 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.346 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl_server_port": { 2024-03-23T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl_server_port", 2024-03-23T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8443", 2024-03-23T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.348 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.348 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.348 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.348 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_behaviour": { 2024-03-23T23:13:41.348 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_behaviour", 2024-03-23T23:13:41.348 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.348 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.348 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.348 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "redirect", 2024-03-23T23:13:41.348 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.348 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.348 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.348 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.348 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect" 2024-03-23T23:13:41.349 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.349 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.349 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.349 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.349 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.349 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.349 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_error_status_code": { 2024-03-23T23:13:41.349 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_error_status_code", 2024-03-23T23:13:41.349 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.349 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.349 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.349 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "500", 2024-03-23T23:13:41.349 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "400", 2024-03-23T23:13:41.349 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "599", 2024-03-23T23:13:41.350 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.350 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.350 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.350 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.350 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.350 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.350 INFO:teuthology.orchestra.run.smithi012.stdout: "url_prefix": { 2024-03-23T23:13:41.350 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "url_prefix", 2024-03-23T23:13:41.350 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.350 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.350 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.350 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.350 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.351 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.351 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.351 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.351 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.351 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.351 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.351 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.351 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.351 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.351 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.351 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "devicehealth", 2024-03-23T23:13:41.351 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.351 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.351 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.351 INFO:teuthology.orchestra.run.smithi012.stdout: "enable_monitoring": { 2024-03-23T23:13:41.351 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enable_monitoring", 2024-03-23T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "monitor device health metrics", 2024-03-23T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.356 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.356 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.356 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.356 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.356 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.356 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.356 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.356 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.356 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.356 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.356 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.356 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.356 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.356 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.357 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.357 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.357 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.357 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.357 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.357 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.357 INFO:teuthology.orchestra.run.smithi012.stdout: "mark_out_threshold": { 2024-03-23T23:13:41.357 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mark_out_threshold", 2024-03-23T23:13:41.357 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.357 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.357 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.357 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "2419200", 2024-03-23T23:13:41.357 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.357 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.357 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.357 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-23T23:13:41.358 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.358 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.358 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.358 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.358 INFO:teuthology.orchestra.run.smithi012.stdout: "pool_name": { 2024-03-23T23:13:41.358 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pool_name", 2024-03-23T23:13:41.358 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.358 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.358 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.358 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "device_health_metrics", 2024-03-23T23:13:41.358 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.358 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.358 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.358 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-23T23:13:41.358 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.358 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.359 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.359 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.359 INFO:teuthology.orchestra.run.smithi012.stdout: "retention_period": { 2024-03-23T23:13:41.359 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "retention_period", 2024-03-23T23:13:41.359 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.359 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.359 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.359 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15552000", 2024-03-23T23:13:41.359 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.359 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.359 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.359 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long to retain device health metrics", 2024-03-23T23:13:41.359 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.359 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.359 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.359 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.359 INFO:teuthology.orchestra.run.smithi012.stdout: "scrape_frequency": { 2024-03-23T23:13:41.359 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "scrape_frequency", 2024-03-23T23:13:41.360 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.360 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.360 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.360 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "86400", 2024-03-23T23:13:41.360 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.360 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.360 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.360 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-23T23:13:41.360 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.360 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.360 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.360 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.360 INFO:teuthology.orchestra.run.smithi012.stdout: "self_heal": { 2024-03-23T23:13:41.360 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "self_heal", 2024-03-23T23:13:41.361 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.361 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.361 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.361 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.361 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.361 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.361 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.361 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-23T23:13:41.361 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.361 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.361 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.361 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.361 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-23T23:13:41.361 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-23T23:13:41.361 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.361 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.362 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.362 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-23T23:13:41.362 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.362 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.362 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.362 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to wake up and check device health", 2024-03-23T23:13:41.362 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.362 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.362 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.362 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.362 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_threshold": { 2024-03-23T23:13:41.362 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_threshold", 2024-03-23T23:13:41.362 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.363 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.363 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.363 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "7257600", 2024-03-23T23:13:41.363 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.363 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.363 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.363 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-23T23:13:41.363 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.363 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.363 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.363 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.363 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.363 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.363 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.363 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "diskprediction_local", 2024-03-23T23:13:41.364 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.364 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.364 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.364 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.364 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.364 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.364 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.364 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.364 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.364 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.364 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.364 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.364 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.364 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.364 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.365 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.365 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.365 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.365 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.365 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.365 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.365 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.365 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.365 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.365 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.365 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.365 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.365 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.365 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.365 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.366 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.366 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.366 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.366 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.366 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.366 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.366 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.366 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.366 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.366 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.366 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.366 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.366 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.366 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.367 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.367 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.367 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.367 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.367 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.367 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.367 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.367 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.367 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.367 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.367 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.367 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.367 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.367 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.368 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.368 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.368 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.368 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.368 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.368 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.368 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.368 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.368 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.368 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.368 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.368 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.368 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.368 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.368 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.368 INFO:teuthology.orchestra.run.smithi012.stdout: "predict_interval": { 2024-03-23T23:13:41.369 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "predict_interval", 2024-03-23T23:13:41.369 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.369 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.369 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.369 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "86400", 2024-03-23T23:13:41.369 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.369 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.369 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.369 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.369 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.369 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.369 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.369 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.369 INFO:teuthology.orchestra.run.smithi012.stdout: "predictor_model": { 2024-03-23T23:13:41.369 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "predictor_model", 2024-03-23T23:13:41.370 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.370 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.370 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.370 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "prophetstor", 2024-03-23T23:13:41.370 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.370 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.370 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.370 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.370 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.370 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.370 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.370 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.370 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-23T23:13:41.370 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-23T23:13:41.370 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.370 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-23T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "influx", 2024-03-23T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": false, 2024-03-23T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "influxdb python module not found", 2024-03-23T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stdout: "batch_size": { 2024-03-23T23:13:41.372 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "batch_size", 2024-03-23T23:13:41.372 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.372 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.372 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.372 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5000", 2024-03-23T23:13:41.372 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.372 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.372 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.372 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-23T23:13:41.372 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.372 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.372 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.372 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.372 INFO:teuthology.orchestra.run.smithi012.stdout: "database": { 2024-03-23T23:13:41.372 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "database", 2024-03-23T23:13:41.373 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.373 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.373 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.373 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "ceph", 2024-03-23T23:13:41.373 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.373 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.373 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.373 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-23T23:13:41.373 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.373 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.373 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.373 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.373 INFO:teuthology.orchestra.run.smithi012.stdout: "hostname": { 2024-03-23T23:13:41.373 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "hostname", 2024-03-23T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB server hostname", 2024-03-23T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-23T23:13:41.375 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-23T23:13:41.375 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.375 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.375 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.375 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "30", 2024-03-23T23:13:41.375 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "5", 2024-03-23T23:13:41.375 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.375 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.375 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-23T23:13:41.375 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.375 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.375 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.375 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.375 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.375 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.376 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.376 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.376 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.376 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.376 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.376 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.376 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.376 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.376 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.376 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.376 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.376 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.376 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.376 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.377 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.377 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.377 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.377 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.377 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.377 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.377 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.377 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.377 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.377 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.377 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.377 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.377 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.377 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.377 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.378 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.378 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.378 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.378 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.378 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.378 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.378 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.378 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.378 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.378 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.378 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.378 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.378 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.379 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.379 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.379 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.379 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.379 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.379 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.379 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.379 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.379 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.379 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.379 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.379 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.379 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.379 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.380 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.380 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.380 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.380 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.380 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.380 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.380 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.380 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.380 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.380 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.380 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.380 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.380 INFO:teuthology.orchestra.run.smithi012.stdout: "password": { 2024-03-23T23:13:41.380 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "password", 2024-03-23T23:13:41.380 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.381 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.381 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.381 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.381 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.381 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.381 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.381 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "password of InfluxDB server user", 2024-03-23T23:13:41.381 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.381 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.381 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.381 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.381 INFO:teuthology.orchestra.run.smithi012.stdout: "port": { 2024-03-23T23:13:41.381 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "port", 2024-03-23T23:13:41.381 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.381 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.382 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.382 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8086", 2024-03-23T23:13:41.382 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.382 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.382 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.382 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB server port", 2024-03-23T23:13:41.382 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.382 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.382 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.382 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.382 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl": { 2024-03-23T23:13:41.382 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl", 2024-03-23T23:13:41.382 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.382 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.382 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.383 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "false", 2024-03-23T23:13:41.383 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.383 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.383 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.383 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-23T23:13:41.383 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.383 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.383 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.383 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.383 INFO:teuthology.orchestra.run.smithi012.stdout: "threads": { 2024-03-23T23:13:41.383 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "threads", 2024-03-23T23:13:41.383 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.383 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.383 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.383 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-03-23T23:13:41.383 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-03-23T23:13:41.383 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "32", 2024-03-23T23:13:41.384 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.384 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-23T23:13:41.384 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.384 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.384 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.384 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.384 INFO:teuthology.orchestra.run.smithi012.stdout: "username": { 2024-03-23T23:13:41.384 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "username", 2024-03-23T23:13:41.384 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.384 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.384 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.384 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.384 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.384 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.385 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.385 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "username of InfluxDB server user", 2024-03-23T23:13:41.385 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.385 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.385 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.385 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.385 INFO:teuthology.orchestra.run.smithi012.stdout: "verify_ssl": { 2024-03-23T23:13:41.385 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "verify_ssl", 2024-03-23T23:13:41.385 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.385 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.385 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.385 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "true", 2024-03-23T23:13:41.385 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.385 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.386 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.386 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-23T23:13:41.386 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.386 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.386 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.386 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.386 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.386 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.386 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.386 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "insights", 2024-03-23T23:13:41.386 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.386 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.386 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.386 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.386 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.387 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.387 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.387 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.387 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.387 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.387 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.387 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.387 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.387 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.387 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.387 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.387 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.387 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.387 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.387 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.387 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.388 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.388 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.388 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.388 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.388 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.388 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.388 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.388 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.388 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.388 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.388 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.388 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.388 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.388 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.388 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.388 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.388 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.388 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.389 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.389 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.389 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.389 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.389 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.389 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.389 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.389 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.389 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.389 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.389 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.389 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.389 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.389 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.389 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.389 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.389 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.390 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.390 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.390 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.390 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.390 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.390 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.390 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.390 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.390 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.390 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.390 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.390 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.390 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.391 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.391 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.391 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.391 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.391 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.391 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.391 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.391 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "iostat", 2024-03-23T23:13:41.391 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.391 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.391 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.391 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.391 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.391 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.392 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.392 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.392 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.392 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.392 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.392 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.392 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.392 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.392 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.392 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.392 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.392 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.392 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.392 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.393 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.393 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.393 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.393 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.393 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.393 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.393 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.393 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.393 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.393 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.393 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.393 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.393 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.393 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.393 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.393 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.394 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.394 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.394 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.394 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.394 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.394 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.394 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.394 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.394 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.394 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.394 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.394 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.394 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.394 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.394 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.394 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "k8sevents", 2024-03-23T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_event_retention_days": { 2024-03-23T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ceph_event_retention_days", 2024-03-23T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "7", 2024-03-23T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-23T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stdout: "config_check_secs": { 2024-03-23T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_check_secs", 2024-03-23T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-23T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "10", 2024-03-23T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-23T23:13:41.398 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.398 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.398 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.398 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.398 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.398 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.398 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.398 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.398 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.398 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.398 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.398 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.398 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.398 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.398 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.398 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.399 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.399 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.399 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.399 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.399 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.399 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.399 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.399 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.399 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.399 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.399 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.399 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.399 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.399 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.399 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.400 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.400 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.400 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.400 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.400 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.400 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.400 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.400 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.400 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.400 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.400 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.400 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.400 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.401 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.401 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.401 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.401 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.401 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.401 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.401 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.401 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.401 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.401 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.401 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.401 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.401 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.401 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.402 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.402 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.402 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.402 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.402 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.402 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.402 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.402 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.402 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.402 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.402 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.402 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.402 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.402 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.402 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.403 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.403 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.403 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.403 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.403 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "localpool", 2024-03-23T23:13:41.403 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.403 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.403 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.403 INFO:teuthology.orchestra.run.smithi012.stdout: "failure_domain": { 2024-03-23T23:13:41.403 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "failure_domain", 2024-03-23T23:13:41.403 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.403 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.403 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.403 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "host", 2024-03-23T23:13:41.403 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.404 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.404 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.404 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "failure domain for any created local pool", 2024-03-23T23:13:41.404 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-23T23:13:41.404 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.404 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.404 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.404 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.404 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.404 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.404 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.404 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.404 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.404 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.404 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.406 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.406 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.406 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.406 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.406 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.406 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.406 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.406 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.406 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.406 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.406 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.406 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.406 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.407 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.407 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.407 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.407 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.407 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.407 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.407 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.407 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.407 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.407 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.407 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.407 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.407 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.407 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.407 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.407 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.407 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.408 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.408 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.408 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.408 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.408 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.408 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.408 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.408 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.408 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.408 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.408 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.408 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.408 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.408 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.408 INFO:teuthology.orchestra.run.smithi012.stdout: "min_size": { 2024-03-23T23:13:41.409 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "min_size", 2024-03-23T23:13:41.409 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.409 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.409 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.409 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.409 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.409 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.409 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.409 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default min_size for any created local pool", 2024-03-23T23:13:41.409 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-23T23:13:41.409 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.409 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.409 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.409 INFO:teuthology.orchestra.run.smithi012.stdout: "num_rep": { 2024-03-23T23:13:41.409 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "num_rep", 2024-03-23T23:13:41.409 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.410 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.410 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.410 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3", 2024-03-23T23:13:41.410 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.410 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.410 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.410 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default replica count for any created local pool", 2024-03-23T23:13:41.410 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.410 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.410 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.410 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.410 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_num": { 2024-03-23T23:13:41.410 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pg_num", 2024-03-23T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "128", 2024-03-23T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default pg_num for any created local pool", 2024-03-23T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stdout: "prefix": { 2024-03-23T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prefix", 2024-03-23T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.412 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.412 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.412 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.412 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.412 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.412 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "name prefix for any created local pool", 2024-03-23T23:13:41.412 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.412 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.412 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.412 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.412 INFO:teuthology.orchestra.run.smithi012.stdout: "subtree": { 2024-03-23T23:13:41.412 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "subtree", 2024-03-23T23:13:41.412 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.412 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.412 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.413 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "rack", 2024-03-23T23:13:41.413 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.413 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.413 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.413 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-23T23:13:41.413 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-23T23:13:41.413 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.413 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.413 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.413 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.413 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.413 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.413 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mds_autoscaler", 2024-03-23T23:13:41.413 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.413 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.414 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.414 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.414 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.414 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.414 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.414 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.414 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.414 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.414 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.414 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.414 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.414 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.414 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.414 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.414 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.415 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.415 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.415 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.415 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.415 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.415 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.415 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.415 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.415 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.415 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.415 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.415 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.415 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.415 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.415 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.416 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.416 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.416 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.416 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.416 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.416 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.416 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.416 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.416 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.416 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.416 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.416 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.416 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.416 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.416 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.417 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.417 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.417 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.417 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.417 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.417 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.417 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.417 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.417 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.417 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.417 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.417 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.417 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.417 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.417 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.418 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.418 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.418 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.418 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.418 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.418 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.418 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.418 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.418 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.418 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.418 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.418 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.418 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.418 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.418 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mirroring", 2024-03-23T23:13:41.418 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.422 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.422 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.422 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.422 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.422 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.422 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.422 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.422 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.422 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.422 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.422 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.422 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.422 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.422 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.423 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.423 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.423 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.423 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.423 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.423 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.423 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.423 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.423 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.423 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.423 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.423 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.423 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.423 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "nfs", 2024-03-23T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.425 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.425 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.425 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.425 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.425 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.425 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.425 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.425 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.425 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.425 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.425 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.425 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.425 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.425 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.425 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.429 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.429 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.429 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.429 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.429 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.429 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.429 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.429 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.429 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.429 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.429 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.429 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.430 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.430 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.430 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.430 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "orchestrator", 2024-03-23T23:13:41.430 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.430 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.430 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.430 INFO:teuthology.orchestra.run.smithi012.stdout: "fail_fs": { 2024-03-23T23:13:41.430 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "fail_fs", 2024-03-23T23:13:41.430 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.430 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.430 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.430 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.430 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-23T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.432 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.432 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.432 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.432 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.432 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.432 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.432 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.432 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.432 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.432 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.432 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.432 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.432 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.432 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.433 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.433 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.433 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.433 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.433 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.433 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.433 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.433 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.433 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.433 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.433 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.433 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.433 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.433 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.433 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.433 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.434 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.434 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.434 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.434 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.434 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.434 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.434 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.434 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.434 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.434 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.434 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.434 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.434 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.434 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.435 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.435 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.435 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.435 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.435 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.435 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.435 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.435 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.435 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.435 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.435 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.435 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.435 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.435 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.435 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.436 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.436 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.436 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator": { 2024-03-23T23:13:41.436 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "orchestrator", 2024-03-23T23:13:41.436 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.436 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.436 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.436 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.436 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.436 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.436 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.436 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm", 2024-03-23T23:13:41.436 INFO:teuthology.orchestra.run.smithi012.stdout: "rook", 2024-03-23T23:13:41.436 INFO:teuthology.orchestra.run.smithi012.stdout: "test_orchestrator" 2024-03-23T23:13:41.436 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.437 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Orchestrator backend", 2024-03-23T23:13:41.437 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.437 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.437 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.437 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.437 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.437 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.437 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.437 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "osd_perf_query", 2024-03-23T23:13:41.437 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.437 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.437 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.437 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.437 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.438 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.438 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.438 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.438 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.438 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.438 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.438 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.438 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.438 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.438 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.438 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.438 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.438 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.438 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.442 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.442 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.442 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.442 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.442 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.442 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.442 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.442 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.442 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.442 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.442 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "osd_support", 2024-03-23T23:13:41.442 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.442 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.445 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.445 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.445 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.445 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.445 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.445 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.445 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.445 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.445 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.445 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.445 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.445 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.445 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.445 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.445 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.446 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.446 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.446 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.446 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.446 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.446 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.446 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.446 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.446 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.446 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.446 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.446 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.446 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.447 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.447 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.447 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.447 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.447 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.447 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.447 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.447 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.447 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.447 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.447 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.447 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.447 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.447 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.447 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.447 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.448 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pg_autoscaler", 2024-03-23T23:13:41.448 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.448 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.448 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.448 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.448 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.448 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.448 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.448 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.448 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.448 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.448 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.448 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.448 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.448 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.448 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.449 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.449 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.449 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.449 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.449 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.449 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.449 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.449 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.449 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.449 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.449 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.449 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.449 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.449 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.449 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.449 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.450 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.450 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.450 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.450 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.450 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.450 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.450 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.450 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.450 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.450 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.450 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.450 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.450 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.450 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.451 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.451 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.451 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.451 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.451 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.451 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.451 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.451 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.451 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.451 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.451 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.451 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.451 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.452 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.452 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.452 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.452 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.452 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.452 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.452 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.452 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.452 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.452 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.452 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.452 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.452 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.452 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.452 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.453 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-23T23:13:41.453 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-23T23:13:41.453 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.453 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.453 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.453 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-23T23:13:41.453 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.453 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.453 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.453 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.453 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.453 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.453 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.453 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.453 INFO:teuthology.orchestra.run.smithi012.stdout: "threshold": { 2024-03-23T23:13:41.454 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "threshold", 2024-03-23T23:13:41.454 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-23T23:13:41.454 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.454 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.454 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3.0", 2024-03-23T23:13:41.454 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1.0", 2024-03-23T23:13:41.454 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.454 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.454 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "scaling threshold", 2024-03-23T23:13:41.454 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-23T23:13:41.454 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.454 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.454 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.454 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.455 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.455 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.455 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "progress", 2024-03-23T23:13:41.455 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.455 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.455 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.455 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_pg_recovery_event": { 2024-03-23T23:13:41.455 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_pg_recovery_event", 2024-03-23T23:13:41.455 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.455 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.455 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.455 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.455 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.455 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.456 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.456 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-23T23:13:41.456 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.456 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.456 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.456 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.456 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": { 2024-03-23T23:13:41.456 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enabled", 2024-03-23T23:13:41.456 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.456 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.456 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.456 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.456 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.456 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.456 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.457 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.457 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.457 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.457 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.457 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.457 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.457 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.457 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.457 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.457 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.457 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.457 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.457 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.457 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.457 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.458 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.458 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.458 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.458 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.458 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.458 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.458 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.458 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.458 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.458 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.458 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.458 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.458 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.458 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.458 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.459 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.459 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.459 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.459 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.459 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.459 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.459 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.459 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.459 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.459 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.459 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.459 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.459 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.459 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.459 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "max_completed_events": { 2024-03-23T23:13:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_completed_events", 2024-03-23T23:13:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "50", 2024-03-23T23:13:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "number of past completed events to remember", 2024-03-23T23:13:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-23T23:13:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-23T23:13:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-03-23T23:13:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long the module is going to sleep", 2024-03-23T23:13:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prometheus", 2024-03-23T23:13:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: "cache": { 2024-03-23T23:13:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cache", 2024-03-23T23:13:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: "exclude_perf_counters": { 2024-03-23T23:13:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "exclude_perf_counters", 2024-03-23T23:13:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-23T23:13:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-23T23:13:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_stats_pools": { 2024-03-23T23:13:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_stats_pools", 2024-03-23T23:13:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-23T23:13:41.472 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-23T23:13:41.472 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.472 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.472 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.472 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "300", 2024-03-23T23:13:41.472 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.472 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.472 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.472 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.472 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.472 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.472 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.472 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.472 INFO:teuthology.orchestra.run.smithi012.stdout: "scrape_interval": { 2024-03-23T23:13:41.473 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "scrape_interval", 2024-03-23T23:13:41.473 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-23T23:13:41.473 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.473 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.473 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15.0", 2024-03-23T23:13:41.473 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.473 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.473 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.473 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.473 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.473 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.473 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.473 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.473 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-03-23T23:13:41.473 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-03-23T23:13:41.473 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.474 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.474 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.474 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "::", 2024-03-23T23:13:41.474 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.474 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.474 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.474 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-23T23:13:41.474 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.474 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.474 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.474 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.474 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-03-23T23:13:41.474 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-03-23T23:13:41.474 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "9283", 2024-03-23T23:13:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-23T23:13:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: "stale_cache_strategy": { 2024-03-23T23:13:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "stale_cache_strategy", 2024-03-23T23:13:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "log", 2024-03-23T23:13:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_behaviour": { 2024-03-23T23:13:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_behaviour", 2024-03-23T23:13:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "default", 2024-03-23T23:13:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: "default", 2024-03-23T23:13:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: "error" 2024-03-23T23:13:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_error_status_code": { 2024-03-23T23:13:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_error_status_code", 2024-03-23T23:13:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "500", 2024-03-23T23:13:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "400", 2024-03-23T23:13:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "599", 2024-03-23T23:13:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_support", 2024-03-23T23:13:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: "max_concurrent_snap_create": { 2024-03-23T23:13:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_concurrent_snap_create", 2024-03-23T23:13:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-23T23:13:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: "mirror_snapshot_schedule": { 2024-03-23T23:13:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mirror_snapshot_schedule", 2024-03-23T23:13:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: "trash_purge_schedule": { 2024-03-23T23:13:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "trash_purge_schedule", 2024-03-23T23:13:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.487 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.487 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "restful", 2024-03-23T23:13:41.487 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.487 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.487 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.487 INFO:teuthology.orchestra.run.smithi012.stdout: "enable_auth": { 2024-03-23T23:13:41.487 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enable_auth", 2024-03-23T23:13:41.487 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.487 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.487 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.487 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.487 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.487 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.487 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.487 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.488 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.488 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.488 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.488 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.488 INFO:teuthology.orchestra.run.smithi012.stdout: "key_file": { 2024-03-23T23:13:41.488 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "key_file", 2024-03-23T23:13:41.488 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.488 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.488 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.488 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.488 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.488 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.488 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.488 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.488 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.488 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.489 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.489 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.489 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.489 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.489 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.489 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.489 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.489 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.489 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.489 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.490 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.490 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.490 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.490 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.490 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.490 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.490 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.490 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.490 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.490 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.490 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.490 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.490 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.490 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.491 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.491 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.491 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.491 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.491 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.491 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.491 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.491 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.491 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.491 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.491 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.491 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.491 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.491 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.492 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.492 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.492 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.492 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.492 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.492 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.492 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.492 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.492 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.492 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.492 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.492 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.492 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.492 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.492 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.492 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.492 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.493 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.493 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.493 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.493 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.493 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.493 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.493 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.493 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.493 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.493 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.493 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.493 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.493 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.493 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.493 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.494 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.494 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.494 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-03-23T23:13:41.494 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-03-23T23:13:41.494 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.494 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.494 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.494 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.494 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.494 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.494 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.494 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.494 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.494 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.495 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.495 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.495 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-03-23T23:13:41.495 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-03-23T23:13:41.495 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.495 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.495 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.495 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.495 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.495 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.495 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.495 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.495 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.495 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.495 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.496 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.496 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.496 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.496 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.496 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rgw", 2024-03-23T23:13:41.496 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.496 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.496 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.496 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.496 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.496 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.496 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.496 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.496 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.496 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.497 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.497 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.497 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.497 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.497 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.497 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.497 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.497 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.497 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.497 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.497 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.497 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.497 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.497 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.497 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.497 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.498 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.498 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.498 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.498 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.498 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.498 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.498 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.498 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.498 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.498 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.498 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.498 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.498 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.499 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.499 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.499 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.499 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.499 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.499 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.499 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.499 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.499 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.499 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.499 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.499 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.499 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.499 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.499 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.499 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.500 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.500 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.500 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.500 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.500 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.500 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.500 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.500 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.500 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.500 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.500 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.500 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.500 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.500 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.500 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rook", 2024-03-23T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.502 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.502 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.502 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.502 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.502 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.502 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.502 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.502 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.502 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.502 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.502 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.502 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.502 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.502 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.503 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.503 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.503 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.503 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.503 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.503 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.503 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.503 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.503 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.503 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.503 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.503 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.503 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.503 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.503 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.503 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.504 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.504 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.504 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.504 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.504 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.504 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.504 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.504 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.504 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.504 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.504 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.504 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.504 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.504 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.504 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.504 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.504 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.505 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.505 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.505 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.505 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.505 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.505 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.505 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.505 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.505 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.505 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.505 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.505 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.505 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.505 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.505 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.506 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.506 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.506 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.506 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_class": { 2024-03-23T23:13:41.506 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "storage_class", 2024-03-23T23:13:41.506 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.506 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.506 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.506 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "local", 2024-03-23T23:13:41.506 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.506 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.506 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.506 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-23T23:13:41.506 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.507 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.507 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.507 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.507 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.507 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.507 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.507 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "selftest", 2024-03-23T23:13:41.507 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.507 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.507 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.507 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.507 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.507 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.507 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.507 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.507 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.507 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.508 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.508 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.508 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.508 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.508 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.508 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.508 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.508 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.508 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.508 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.508 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.508 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.508 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.508 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.508 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.508 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.509 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.509 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.509 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.509 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.509 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.509 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.509 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.509 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.509 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.509 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.509 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.509 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.509 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.509 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.509 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.510 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.510 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.510 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.510 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.510 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.510 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.510 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.510 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.510 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.510 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.510 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.510 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.510 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.510 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.510 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.511 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.511 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.511 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.511 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.511 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.511 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.511 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.511 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.511 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.511 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.511 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.511 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.511 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.511 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.512 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.512 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.512 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.512 INFO:teuthology.orchestra.run.smithi012.stdout: "roption1": { 2024-03-23T23:13:41.512 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "roption1", 2024-03-23T23:13:41.512 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.512 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.512 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.512 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.512 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.512 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.512 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.512 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.512 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.512 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.512 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.512 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.513 INFO:teuthology.orchestra.run.smithi012.stdout: "roption2": { 2024-03-23T23:13:41.513 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "roption2", 2024-03-23T23:13:41.513 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.513 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.513 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.513 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "xyz", 2024-03-23T23:13:41.513 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.513 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.513 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.513 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.513 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.513 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.513 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.513 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.513 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption1": { 2024-03-23T23:13:41.513 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption1", 2024-03-23T23:13:41.514 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.514 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.514 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.514 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.514 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.514 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.514 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.514 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.514 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.514 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.514 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.514 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.514 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption2": { 2024-03-23T23:13:41.514 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption2", 2024-03-23T23:13:41.514 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.515 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.515 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.515 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.515 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.515 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.515 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.515 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.515 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.515 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.515 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.515 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.515 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption3": { 2024-03-23T23:13:41.515 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption3", 2024-03-23T23:13:41.515 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-23T23:13:41.515 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.516 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.516 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.516 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.516 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.516 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.516 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.516 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.516 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.516 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.516 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.516 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption4": { 2024-03-23T23:13:41.516 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption4", 2024-03-23T23:13:41.516 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.516 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.516 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.516 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.517 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.517 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.517 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.517 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.517 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.517 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.517 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.517 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.517 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption5": { 2024-03-23T23:13:41.517 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption5", 2024-03-23T23:13:41.517 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.517 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.517 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.517 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.517 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.517 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.517 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.518 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.518 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.518 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.518 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.518 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.518 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption6": { 2024-03-23T23:13:41.518 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption6", 2024-03-23T23:13:41.518 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.518 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.518 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.518 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.518 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.518 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.518 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.518 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.518 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.519 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.519 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.519 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.519 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption7": { 2024-03-23T23:13:41.519 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption7", 2024-03-23T23:13:41.519 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.519 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.519 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.519 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.519 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-03-23T23:13:41.519 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "42", 2024-03-23T23:13:41.519 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.519 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.519 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.519 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.519 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.520 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.520 INFO:teuthology.orchestra.run.smithi012.stdout: "testkey": { 2024-03-23T23:13:41.520 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testkey", 2024-03-23T23:13:41.520 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.520 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.520 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.520 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.520 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.520 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.520 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.520 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.520 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.520 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.520 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.520 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.521 INFO:teuthology.orchestra.run.smithi012.stdout: "testlkey": { 2024-03-23T23:13:41.521 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testlkey", 2024-03-23T23:13:41.521 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.521 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.521 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.521 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.521 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.521 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.521 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.521 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.521 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.521 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.521 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.521 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.521 INFO:teuthology.orchestra.run.smithi012.stdout: "testnewline": { 2024-03-23T23:13:41.522 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testnewline", 2024-03-23T23:13:41.522 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.522 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.522 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.522 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.522 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.522 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.522 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.522 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.522 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.522 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.522 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.522 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.522 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.522 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.523 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.523 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snap_schedule", 2024-03-23T23:13:41.523 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.523 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.523 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.523 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_m_granularity": { 2024-03-23T23:13:41.523 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_m_granularity", 2024-03-23T23:13:41.523 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.523 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.523 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.523 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.523 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.523 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.523 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.523 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow minute scheduled snapshots", 2024-03-23T23:13:41.524 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.524 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.524 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.524 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.524 INFO:teuthology.orchestra.run.smithi012.stdout: "dump_on_update": { 2024-03-23T23:13:41.524 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "dump_on_update", 2024-03-23T23:13:41.524 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.524 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.524 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.524 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.524 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.524 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.524 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.524 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "dump database to debug log on update", 2024-03-23T23:13:41.524 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.524 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.525 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.525 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.525 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.525 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.525 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.525 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.525 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.525 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.525 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.525 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.525 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.525 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.525 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.525 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.525 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.525 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.525 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.526 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.526 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.526 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.526 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.526 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.526 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.526 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.526 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.526 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.526 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.526 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.526 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.526 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.526 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.526 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.526 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.527 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.527 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.527 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.527 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.527 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.527 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.527 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.527 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.527 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.527 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.527 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.527 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.527 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.527 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.527 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.527 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.528 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.528 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.528 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.528 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.528 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.528 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.528 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.528 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.528 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.528 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.528 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.528 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.528 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.528 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.528 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.528 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.529 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.529 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.529 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.529 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.529 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.529 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.529 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.529 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.529 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.529 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.529 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "stats", 2024-03-23T23:13:41.529 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.529 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.529 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.529 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.529 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.530 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.530 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.530 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.530 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.530 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.530 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.530 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.530 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.530 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.530 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.530 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.530 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.530 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.530 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.530 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.531 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.531 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.531 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.531 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.531 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.531 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.531 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.531 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.531 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.531 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.531 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.531 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.531 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.531 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.532 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.532 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.532 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.532 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.532 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.532 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.532 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.532 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.532 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.532 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.532 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.532 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.532 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.532 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.532 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.532 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.533 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.533 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.533 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.533 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.533 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.533 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.533 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.533 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.533 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.533 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.533 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.533 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.533 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.533 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.533 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.534 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.534 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.534 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.534 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.534 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.534 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.534 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.534 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.534 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.534 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.534 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.534 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "status", 2024-03-23T23:13:41.534 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.534 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.534 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.535 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.535 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.535 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.535 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.535 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.535 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.535 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.535 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.535 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.535 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.535 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.535 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.535 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.535 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.535 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.535 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.536 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.536 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.536 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.536 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.536 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.536 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.536 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.536 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.536 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.536 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.536 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.536 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.536 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.536 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.536 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.537 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.537 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.537 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.537 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.537 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.537 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.537 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.537 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.537 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.537 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.537 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.537 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.537 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.537 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.537 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.538 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.538 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.538 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.538 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.538 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.538 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.538 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.538 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.538 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.538 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.538 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.538 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.538 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.538 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.538 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.538 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.539 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.539 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.539 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.539 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.539 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.539 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.539 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.539 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.539 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.539 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.539 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.539 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "telegraf", 2024-03-23T23:13:41.539 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.539 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.539 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.539 INFO:teuthology.orchestra.run.smithi012.stdout: "address": { 2024-03-23T23:13:41.540 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "address", 2024-03-23T23:13:41.540 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.540 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.540 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.540 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-23T23:13:41.540 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.540 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.540 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.540 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.540 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.540 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.540 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.540 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.540 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-23T23:13:41.541 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-23T23:13:41.541 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.541 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.541 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.541 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15", 2024-03-23T23:13:41.541 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.541 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.541 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.541 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.541 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.541 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.541 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.541 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.541 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.541 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.541 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.542 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.542 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.542 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.542 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.542 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.542 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.542 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.542 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.542 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.542 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.542 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.542 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.542 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.542 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.543 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.543 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.543 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.543 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.543 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.543 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.543 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.543 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.543 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.543 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.543 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.543 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.543 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.543 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.543 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.544 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.544 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.544 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.544 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.544 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.544 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.544 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.544 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.544 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.544 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.544 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.544 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.544 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.544 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.544 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.545 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.545 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.545 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.545 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.545 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.545 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.545 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.545 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.545 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.545 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.545 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.545 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.545 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.545 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.545 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.546 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.546 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.546 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.546 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.546 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.546 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.546 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.546 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.546 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.546 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.546 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.546 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "telemetry", 2024-03-23T23:13:41.546 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.546 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.546 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.547 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_basic": { 2024-03-23T23:13:41.547 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_basic", 2024-03-23T23:13:41.547 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.547 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.547 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.547 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.547 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.547 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.547 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.547 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-23T23:13:41.547 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.547 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.547 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.547 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.547 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_crash": { 2024-03-23T23:13:41.548 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_crash", 2024-03-23T23:13:41.548 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.548 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.548 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.548 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.548 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.548 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.548 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.548 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-23T23:13:41.548 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.548 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.548 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.548 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.548 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_device": { 2024-03-23T23:13:41.548 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_device", 2024-03-23T23:13:41.548 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.549 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.549 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.549 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.549 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.549 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.549 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.549 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-23T23:13:41.549 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.549 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.549 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.549 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.549 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_ident": { 2024-03-23T23:13:41.549 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_ident", 2024-03-23T23:13:41.549 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.549 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.549 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.550 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.550 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.550 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.550 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.550 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-23T23:13:41.550 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.550 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.550 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.550 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.550 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_perf": { 2024-03-23T23:13:41.550 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_perf", 2024-03-23T23:13:41.550 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.550 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.550 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.551 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.551 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.551 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.551 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.551 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-23T23:13:41.551 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.551 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.551 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.551 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.551 INFO:teuthology.orchestra.run.smithi012.stdout: "contact": { 2024-03-23T23:13:41.551 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "contact", 2024-03-23T23:13:41.551 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.551 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.551 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.551 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.552 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.552 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.552 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.552 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.552 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.552 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.552 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.552 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.552 INFO:teuthology.orchestra.run.smithi012.stdout: "description": { 2024-03-23T23:13:41.552 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "description", 2024-03-23T23:13:41.552 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.552 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.552 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.552 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.552 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.552 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.553 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.553 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.553 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.553 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.553 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.553 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.553 INFO:teuthology.orchestra.run.smithi012.stdout: "device_url": { 2024-03-23T23:13:41.553 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_url", 2024-03-23T23:13:41.553 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.553 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.553 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.553 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-23T23:13:41.553 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.553 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.554 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.554 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.554 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.554 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.554 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.554 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.554 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": { 2024-03-23T23:13:41.554 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enabled", 2024-03-23T23:13:41.554 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.554 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.554 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.554 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.554 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.554 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.554 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.555 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.555 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.555 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.555 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.555 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.555 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-23T23:13:41.555 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-23T23:13:41.555 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.555 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.555 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.555 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "24", 2024-03-23T23:13:41.555 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "8", 2024-03-23T23:13:41.555 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.555 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.555 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.556 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.556 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.556 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.556 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.556 INFO:teuthology.orchestra.run.smithi012.stdout: "last_opt_revision": { 2024-03-23T23:13:41.556 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "last_opt_revision", 2024-03-23T23:13:41.556 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.556 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.556 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.556 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1", 2024-03-23T23:13:41.556 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.556 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.556 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.556 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.556 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.557 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.557 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.557 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.557 INFO:teuthology.orchestra.run.smithi012.stdout: "leaderboard": { 2024-03-23T23:13:41.557 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "leaderboard", 2024-03-23T23:13:41.557 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.557 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.557 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.557 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.557 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.557 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.557 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.557 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.557 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.557 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.558 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.558 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.558 INFO:teuthology.orchestra.run.smithi012.stdout: "leaderboard_description": { 2024-03-23T23:13:41.558 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "leaderboard_description", 2024-03-23T23:13:41.558 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.558 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.558 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.558 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.558 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.558 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.558 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.558 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.558 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.558 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.559 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.559 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.559 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.559 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.559 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.559 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.559 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.559 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.559 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.559 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.559 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.559 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.559 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.559 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.559 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.559 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.560 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.560 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.560 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.560 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.560 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.560 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.560 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.560 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.560 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.560 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.560 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.560 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.560 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.560 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.560 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.561 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.561 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.561 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.561 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.561 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.561 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.561 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.561 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.561 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.561 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.561 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.561 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.561 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.561 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.561 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.561 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.562 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.562 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.562 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.562 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.562 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.562 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.562 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.562 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.562 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.562 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.562 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.562 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.562 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.562 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.563 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.563 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.563 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.563 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.563 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.563 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.563 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.563 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.563 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.563 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.563 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.563 INFO:teuthology.orchestra.run.smithi012.stdout: "organization": { 2024-03-23T23:13:41.563 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "organization", 2024-03-23T23:13:41.563 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.563 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.563 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.564 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.564 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.564 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.564 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.564 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.564 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.564 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.564 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.564 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.564 INFO:teuthology.orchestra.run.smithi012.stdout: "proxy": { 2024-03-23T23:13:41.564 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "proxy", 2024-03-23T23:13:41.564 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.564 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.564 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.564 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.565 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.565 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.565 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.565 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.565 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.565 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.565 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.565 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.565 INFO:teuthology.orchestra.run.smithi012.stdout: "url": { 2024-03-23T23:13:41.565 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "url", 2024-03-23T23:13:41.565 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.565 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.565 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.566 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-23T23:13:41.566 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.566 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.566 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.566 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.566 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.566 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.566 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.566 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.566 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.566 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.566 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.566 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test_orchestrator", 2024-03-23T23:13:41.566 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.567 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.567 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.567 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.567 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.567 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.567 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.567 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.567 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.567 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.567 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.567 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.567 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.567 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.567 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.568 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.568 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.568 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.568 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.568 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.568 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.568 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.568 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.568 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.568 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.568 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.568 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.568 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.568 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.568 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.568 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.569 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.569 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.569 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.569 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.569 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.569 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.569 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.569 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.569 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.569 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.569 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.569 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.569 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.569 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.569 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.569 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.570 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.570 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.570 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.570 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.570 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.570 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.570 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.570 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.570 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.570 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.570 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.570 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.570 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.570 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.570 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.571 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.571 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.571 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.571 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.571 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.571 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.571 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.571 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.571 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.571 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.571 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.571 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.571 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.571 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.571 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "volumes", 2024-03-23T23:13:41.571 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.572 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.572 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.572 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.572 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.572 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.572 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.572 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.572 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.572 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.572 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.572 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.572 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.572 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.572 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.572 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.572 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.574 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.574 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.574 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.574 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.574 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.574 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.574 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.574 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.574 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.574 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.574 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.574 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.574 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.574 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.575 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.575 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.575 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.575 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.575 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.575 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.575 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.575 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.575 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.575 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.575 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.575 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.575 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.575 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.575 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.575 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.576 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.576 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.576 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.576 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.576 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.576 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.576 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.576 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.576 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.576 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.576 INFO:teuthology.orchestra.run.smithi012.stdout: "max_concurrent_clones": { 2024-03-23T23:13:41.576 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_concurrent_clones", 2024-03-23T23:13:41.576 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.576 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.577 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.577 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "4", 2024-03-23T23:13:41.577 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.577 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.577 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.577 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-23T23:13:41.577 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.577 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.577 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.577 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.577 INFO:teuthology.orchestra.run.smithi012.stdout: "periodic_async_work": { 2024-03-23T23:13:41.577 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "periodic_async_work", 2024-03-23T23:13:41.577 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.577 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.578 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.578 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.578 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.578 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.578 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.578 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Periodically check for async work", 2024-03-23T23:13:41.578 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.578 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.578 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.578 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.578 INFO:teuthology.orchestra.run.smithi012.stdout: "snapshot_clone_delay": { 2024-03-23T23:13:41.578 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snapshot_clone_delay", 2024-03-23T23:13:41.578 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.579 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.579 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.579 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-23T23:13:41.579 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.579 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.579 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.579 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-23T23:13:41.579 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.579 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.579 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.579 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.579 INFO:teuthology.orchestra.run.smithi012.stdout: "snapshot_clone_no_wait": { 2024-03-23T23:13:41.579 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snapshot_clone_no_wait", 2024-03-23T23:13:41.579 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.579 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.580 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.580 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.580 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.580 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.580 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.580 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-23T23:13:41.580 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.580 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.580 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.580 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.580 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.580 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.580 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.580 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix", 2024-03-23T23:13:41.580 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.581 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.581 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.581 INFO:teuthology.orchestra.run.smithi012.stdout: "discovery_interval": { 2024-03-23T23:13:41.581 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "discovery_interval", 2024-03-23T23:13:41.581 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-23T23:13:41.581 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.581 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.581 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "100", 2024-03-23T23:13:41.581 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.581 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.581 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.581 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.581 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.581 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.581 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.582 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.582 INFO:teuthology.orchestra.run.smithi012.stdout: "identifier": { 2024-03-23T23:13:41.582 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "identifier", 2024-03-23T23:13:41.582 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.582 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.582 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.582 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.582 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.582 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.582 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.582 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.582 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.582 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.583 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.583 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.583 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-23T23:13:41.583 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-23T23:13:41.583 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.583 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.583 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.583 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-23T23:13:41.583 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.583 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.583 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.583 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.583 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.584 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.584 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.584 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.584 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.584 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.584 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.584 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.584 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.584 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.584 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.584 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.584 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.584 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.584 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.584 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.584 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.585 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.585 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.585 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.585 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.585 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.585 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.585 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.585 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.585 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.585 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.585 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.585 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.585 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.585 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.585 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.586 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.586 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.586 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.586 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.586 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.586 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.586 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.586 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.586 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.586 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.586 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.586 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.586 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.586 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.587 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.587 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.587 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.587 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.587 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.587 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.587 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.587 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.587 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.587 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.587 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.587 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.587 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.587 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.588 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.588 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.588 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.588 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.588 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.588 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.588 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.588 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.588 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.588 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.588 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.588 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.588 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.588 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.588 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_host": { 2024-03-23T23:13:41.588 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_host", 2024-03-23T23:13:41.589 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.589 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.589 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.589 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.589 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.589 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.589 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.589 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.589 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.589 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.589 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.589 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.589 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_port": { 2024-03-23T23:13:41.589 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_port", 2024-03-23T23:13:41.590 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.590 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.590 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.590 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10051", 2024-03-23T23:13:41.590 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.590 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.590 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.590 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.590 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.590 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.590 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.590 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.590 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_sender": { 2024-03-23T23:13:41.590 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_sender", 2024-03-23T23:13:41.590 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.591 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.591 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.591 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-23T23:13:41.591 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.591 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.591 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.591 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.591 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.591 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.591 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.591 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.591 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.591 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.591 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-23T23:13:41.591 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.591 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.592 INFO:teuthology.orchestra.run.smithi012.stdout: "gid": 24113, 2024-03-23T23:13:41.592 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "y", 2024-03-23T23:13:41.592 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr_features": 4540701547738038271, 2024-03-23T23:13:41.592 INFO:teuthology.orchestra.run.smithi012.stdout: "available_modules": [ 2024-03-23T23:13:41.592 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.592 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "alerts", 2024-03-23T23:13:41.592 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.592 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.592 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.592 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-23T23:13:41.592 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-23T23:13:41.592 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.592 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.592 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.593 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-23T23:13:41.593 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.593 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.593 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.593 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How frequently to reexamine health status", 2024-03-23T23:13:41.593 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.593 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.593 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.593 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.593 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.593 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.593 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.593 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.593 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.593 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.594 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.594 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.594 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.594 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.594 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.594 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.594 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.594 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.594 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.594 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.594 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.594 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.594 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.594 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.594 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.595 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.595 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.595 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.595 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.595 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.595 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.595 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.595 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.595 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.595 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.595 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.595 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.595 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.595 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.595 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.596 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.596 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.596 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.596 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.596 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.596 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.596 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.596 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.596 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.596 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.596 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.596 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.596 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.596 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.597 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.597 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.597 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.597 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.597 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.597 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.597 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.597 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.597 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.597 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.597 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.597 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.597 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.597 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.597 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.598 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.598 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.598 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.598 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.598 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.598 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_destination": { 2024-03-23T23:13:41.598 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_destination", 2024-03-23T23:13:41.598 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.598 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.598 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.598 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.598 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.598 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.598 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.598 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Email address to send alerts to", 2024-03-23T23:13:41.599 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.599 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.599 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.599 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.599 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_from_name": { 2024-03-23T23:13:41.599 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_from_name", 2024-03-23T23:13:41.599 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.599 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.599 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.599 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "Ceph", 2024-03-23T23:13:41.599 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.599 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.599 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.599 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Email From: name", 2024-03-23T23:13:41.600 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.600 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.600 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.600 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.600 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_host": { 2024-03-23T23:13:41.600 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_host", 2024-03-23T23:13:41.600 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.600 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.600 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.600 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.600 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.600 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.600 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.600 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP server", 2024-03-23T23:13:41.600 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.600 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.600 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.601 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.601 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_password": { 2024-03-23T23:13:41.601 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_password", 2024-03-23T23:13:41.601 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.601 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.601 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.601 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.601 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.601 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.601 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.601 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Password to authenticate with", 2024-03-23T23:13:41.601 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.601 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.601 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.601 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.601 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_port": { 2024-03-23T23:13:41.602 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_port", 2024-03-23T23:13:41.602 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.602 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.602 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.602 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "465", 2024-03-23T23:13:41.602 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.602 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.602 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.602 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP port", 2024-03-23T23:13:41.602 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.602 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.602 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.602 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.602 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_sender": { 2024-03-23T23:13:41.602 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_sender", 2024-03-23T23:13:41.603 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.603 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.603 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.603 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.603 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.603 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.603 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.603 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP envelope sender", 2024-03-23T23:13:41.603 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.603 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.603 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.603 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.603 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_ssl": { 2024-03-23T23:13:41.603 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_ssl", 2024-03-23T23:13:41.603 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.604 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.604 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.604 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.604 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.604 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.604 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.604 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-23T23:13:41.604 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.604 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.604 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.604 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.604 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_user": { 2024-03-23T23:13:41.604 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_user", 2024-03-23T23:13:41.604 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.604 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.604 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.605 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.605 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.605 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.605 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.605 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "User to authenticate as", 2024-03-23T23:13:41.605 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.605 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.605 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.605 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.605 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.605 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.605 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.605 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "balancer", 2024-03-23T23:13:41.605 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.605 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.605 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.605 INFO:teuthology.orchestra.run.smithi012.stdout: "active": { 2024-03-23T23:13:41.606 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "active", 2024-03-23T23:13:41.606 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.606 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.606 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.606 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.606 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.606 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.606 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.606 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically balance PGs across cluster", 2024-03-23T23:13:41.606 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.606 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.606 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.606 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.606 INFO:teuthology.orchestra.run.smithi012.stdout: "begin_time": { 2024-03-23T23:13:41.606 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "begin_time", 2024-03-23T23:13:41.607 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.607 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.607 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.607 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0000", 2024-03-23T23:13:41.607 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.607 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.607 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.607 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "beginning time of day to automatically balance", 2024-03-23T23:13:41.607 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-23T23:13:41.607 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.607 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.607 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.607 INFO:teuthology.orchestra.run.smithi012.stdout: "begin_weekday": { 2024-03-23T23:13:41.607 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "begin_weekday", 2024-03-23T23:13:41.607 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-23T23:13:41.607 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.607 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.608 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-23T23:13:41.608 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0", 2024-03-23T23:13:41.608 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "6", 2024-03-23T23:13:41.608 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.608 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-23T23:13:41.608 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-23T23:13:41.608 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.608 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.608 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.608 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_max_iterations": { 2024-03-23T23:13:41.608 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_max_iterations", 2024-03-23T23:13:41.608 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-23T23:13:41.608 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.608 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.608 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "25", 2024-03-23T23:13:41.608 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-03-23T23:13:41.608 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "250", 2024-03-23T23:13:41.609 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.609 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-23T23:13:41.609 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.609 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.609 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.609 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.609 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_metrics": { 2024-03-23T23:13:41.609 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_metrics", 2024-03-23T23:13:41.609 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.609 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.609 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.609 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "pgs,objects,bytes", 2024-03-23T23:13:41.609 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.609 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.609 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.610 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-23T23:13:41.610 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-23T23:13:41.610 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.610 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.610 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.610 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_step": { 2024-03-23T23:13:41.610 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_step", 2024-03-23T23:13:41.610 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-23T23:13:41.610 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.610 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.610 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0.5", 2024-03-23T23:13:41.610 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0.001", 2024-03-23T23:13:41.610 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "0.999", 2024-03-23T23:13:41.610 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.611 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "aggressiveness of optimization", 2024-03-23T23:13:41.611 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-23T23:13:41.611 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.611 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.611 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.611 INFO:teuthology.orchestra.run.smithi012.stdout: "end_time": { 2024-03-23T23:13:41.611 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "end_time", 2024-03-23T23:13:41.611 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.611 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.611 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.611 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "2359", 2024-03-23T23:13:41.611 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.611 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.611 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.611 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "ending time of day to automatically balance", 2024-03-23T23:13:41.611 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-23T23:13:41.611 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.612 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.612 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.612 INFO:teuthology.orchestra.run.smithi012.stdout: "end_weekday": { 2024-03-23T23:13:41.612 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "end_weekday", 2024-03-23T23:13:41.612 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-23T23:13:41.612 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.612 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.612 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-23T23:13:41.612 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0", 2024-03-23T23:13:41.612 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "6", 2024-03-23T23:13:41.612 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.612 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-23T23:13:41.612 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-23T23:13:41.612 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.612 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.613 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.613 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.613 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.613 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.613 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.613 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.613 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.613 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.613 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.613 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.613 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.613 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.613 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.613 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.613 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.613 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.614 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.614 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.614 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.614 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.614 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.614 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.614 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.614 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.614 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.614 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.614 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.614 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.614 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.614 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.614 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.614 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.615 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.615 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.615 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.615 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.615 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.615 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.615 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.615 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.615 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.615 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.615 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.615 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.615 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.615 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.615 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.615 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.616 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.616 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.616 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.616 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.616 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.616 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.616 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.616 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.616 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.616 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.616 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.616 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.616 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.616 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.616 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.616 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.617 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.617 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.617 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.617 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.617 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.617 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.617 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.617 INFO:teuthology.orchestra.run.smithi012.stdout: "min_score": { 2024-03-23T23:13:41.617 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "min_score", 2024-03-23T23:13:41.617 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-23T23:13:41.617 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.617 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.617 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-23T23:13:41.617 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.617 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.617 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.618 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-23T23:13:41.618 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.618 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.618 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.618 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.618 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": { 2024-03-23T23:13:41.618 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mode", 2024-03-23T23:13:41.618 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.618 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.618 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.618 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "upmap", 2024-03-23T23:13:41.618 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.618 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.618 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.618 INFO:teuthology.orchestra.run.smithi012.stdout: "crush-compat", 2024-03-23T23:13:41.619 INFO:teuthology.orchestra.run.smithi012.stdout: "none", 2024-03-23T23:13:41.619 INFO:teuthology.orchestra.run.smithi012.stdout: "read", 2024-03-23T23:13:41.619 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap", 2024-03-23T23:13:41.619 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap-read" 2024-03-23T23:13:41.619 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.619 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Balancer mode", 2024-03-23T23:13:41.619 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.619 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.619 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.619 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.619 INFO:teuthology.orchestra.run.smithi012.stdout: "pool_ids": { 2024-03-23T23:13:41.619 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pool_ids", 2024-03-23T23:13:41.619 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.619 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.619 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.620 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.620 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.620 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.620 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.620 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-23T23:13:41.620 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.620 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.620 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.620 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.620 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-23T23:13:41.620 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-23T23:13:41.620 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.620 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.620 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.621 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-23T23:13:41.621 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.621 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.621 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.621 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-23T23:13:41.621 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.621 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.621 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.621 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.621 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap_max_deviation": { 2024-03-23T23:13:41.621 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "upmap_max_deviation", 2024-03-23T23:13:41.621 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.621 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.621 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.621 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-03-23T23:13:41.621 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-03-23T23:13:41.622 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.622 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.622 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-23T23:13:41.622 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-23T23:13:41.622 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.622 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.622 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.622 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap_max_optimizations": { 2024-03-23T23:13:41.622 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "upmap_max_optimizations", 2024-03-23T23:13:41.622 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-23T23:13:41.622 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.622 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.622 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-23T23:13:41.622 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.622 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.622 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.623 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-23T23:13:41.623 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.623 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.623 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.623 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.623 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.623 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.623 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.623 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephadm", 2024-03-23T23:13:41.623 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.623 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.623 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.623 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_down_multiplier": { 2024-03-23T23:13:41.623 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_down_multiplier", 2024-03-23T23:13:41.623 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-23T23:13:41.623 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.623 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.624 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3.0", 2024-03-23T23:13:41.624 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.624 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.624 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.624 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-23T23:13:41.624 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.624 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.624 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.624 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.624 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_refresh_rate": { 2024-03-23T23:13:41.624 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_refresh_rate", 2024-03-23T23:13:41.624 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.624 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.624 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.624 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "20", 2024-03-23T23:13:41.624 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.625 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.625 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.625 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-23T23:13:41.625 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.625 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.625 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.625 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.625 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_starting_port": { 2024-03-23T23:13:41.625 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_starting_port", 2024-03-23T23:13:41.625 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.625 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.625 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.625 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "4721", 2024-03-23T23:13:41.625 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.625 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.626 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.626 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-23T23:13:41.626 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.626 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.626 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.626 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.626 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_ptrace": { 2024-03-23T23:13:41.626 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_ptrace", 2024-03-23T23:13:41.626 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.626 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.626 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.626 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.626 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.626 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.626 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.626 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-23T23:13:41.627 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-23T23:13:41.627 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.627 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.627 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.627 INFO:teuthology.orchestra.run.smithi012.stdout: "autotune_interval": { 2024-03-23T23:13:41.627 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "autotune_interval", 2024-03-23T23:13:41.627 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.627 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.627 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.627 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-23T23:13:41.627 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.627 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.627 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.627 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-23T23:13:41.628 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.628 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.628 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.628 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.628 INFO:teuthology.orchestra.run.smithi012.stdout: "autotune_memory_target_ratio": { 2024-03-23T23:13:41.628 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "autotune_memory_target_ratio", 2024-03-23T23:13:41.628 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-23T23:13:41.628 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.628 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.628 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0.7", 2024-03-23T23:13:41.628 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.628 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.628 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.628 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-23T23:13:41.628 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.628 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.628 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.629 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.629 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm_log_destination": { 2024-03-23T23:13:41.629 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephadm_log_destination", 2024-03-23T23:13:41.629 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.629 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.629 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.629 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.629 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.629 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.629 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.629 INFO:teuthology.orchestra.run.smithi012.stdout: "file", 2024-03-23T23:13:41.629 INFO:teuthology.orchestra.run.smithi012.stdout: "file,syslog", 2024-03-23T23:13:41.629 INFO:teuthology.orchestra.run.smithi012.stdout: "syslog" 2024-03-23T23:13:41.629 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.629 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-23T23:13:41.629 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.629 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.630 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.630 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.630 INFO:teuthology.orchestra.run.smithi012.stdout: "cgroups_split": { 2024-03-23T23:13:41.630 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cgroups_split", 2024-03-23T23:13:41.630 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.630 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.630 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.630 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.630 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.630 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.630 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.630 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-23T23:13:41.630 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.630 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.630 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.631 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.631 INFO:teuthology.orchestra.run.smithi012.stdout: "config_checks_enabled": { 2024-03-23T23:13:41.631 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_checks_enabled", 2024-03-23T23:13:41.631 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.631 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.631 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.631 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.631 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.631 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.631 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.631 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-23T23:13:41.631 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.631 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.631 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.631 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.631 INFO:teuthology.orchestra.run.smithi012.stdout: "config_dashboard": { 2024-03-23T23:13:41.632 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_dashboard", 2024-03-23T23:13:41.632 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.632 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.632 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.632 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.632 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.632 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.632 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.632 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-23T23:13:41.632 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.632 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.632 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.632 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.632 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_alertmanager": { 2024-03-23T23:13:41.632 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_alertmanager", 2024-03-23T23:13:41.632 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.633 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.633 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.633 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-23T23:13:41.633 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.633 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.633 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.633 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-03-23T23:13:41.633 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.633 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.633 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.633 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.633 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_base": { 2024-03-23T23:13:41.633 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_base", 2024-03-23T23:13:41.633 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.633 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.634 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.634 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-23T23:13:41.634 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.634 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.634 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.634 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Container image name, without the tag", 2024-03-23T23:13:41.634 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.634 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.634 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.634 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.634 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_elasticsearch": { 2024-03-23T23:13:41.634 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_elasticsearch", 2024-03-23T23:13:41.634 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.635 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.635 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.635 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-23T23:13:41.635 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.635 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.635 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.635 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "elasticsearch container image", 2024-03-23T23:13:41.635 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.635 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.635 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.635 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.635 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_grafana": { 2024-03-23T23:13:41.635 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_grafana", 2024-03-23T23:13:41.635 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.635 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.635 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.636 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-23T23:13:41.636 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.636 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.636 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.636 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-03-23T23:13:41.636 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.636 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.636 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.636 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.636 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_haproxy": { 2024-03-23T23:13:41.636 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_haproxy", 2024-03-23T23:13:41.636 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.636 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.636 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.637 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-23T23:13:41.637 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.637 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.637 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.637 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "HAproxy container image", 2024-03-23T23:13:41.637 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.637 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.637 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.637 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.637 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_agent": { 2024-03-23T23:13:41.637 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_agent", 2024-03-23T23:13:41.637 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.637 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.637 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.638 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-23T23:13:41.638 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.638 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.638 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.638 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger agent container image", 2024-03-23T23:13:41.638 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.638 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.638 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.638 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.638 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_collector": { 2024-03-23T23:13:41.638 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_collector", 2024-03-23T23:13:41.638 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.638 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.639 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.639 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-23T23:13:41.639 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.639 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.639 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.639 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger collector container image", 2024-03-23T23:13:41.639 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.639 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.639 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.639 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.639 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_query": { 2024-03-23T23:13:41.639 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_query", 2024-03-23T23:13:41.639 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.639 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.639 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.639 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-23T23:13:41.640 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.640 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.640 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.640 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger query container image", 2024-03-23T23:13:41.640 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.640 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.640 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.640 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.640 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_keepalived": { 2024-03-23T23:13:41.640 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_keepalived", 2024-03-23T23:13:41.640 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.640 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.640 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.640 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-23T23:13:41.640 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.641 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.641 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.641 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Keepalived container image", 2024-03-23T23:13:41.641 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.641 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.641 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.641 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.641 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_loki": { 2024-03-23T23:13:41.641 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_loki", 2024-03-23T23:13:41.641 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.641 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.641 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.641 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-23T23:13:41.641 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.641 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.641 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.642 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Loki container image", 2024-03-23T23:13:41.642 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.642 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.642 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.642 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.642 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_node_exporter": { 2024-03-23T23:13:41.642 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_node_exporter", 2024-03-23T23:13:41.642 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.642 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.642 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.642 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-23T23:13:41.642 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.642 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.642 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.642 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-03-23T23:13:41.643 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.643 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.643 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.643 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.643 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_nvmeof": { 2024-03-23T23:13:41.643 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_nvmeof", 2024-03-23T23:13:41.643 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.643 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.643 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.643 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-23T23:13:41.643 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.643 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.643 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.643 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Nvme-of container image", 2024-03-23T23:13:41.643 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.644 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.644 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.644 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.644 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_prometheus": { 2024-03-23T23:13:41.644 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_prometheus", 2024-03-23T23:13:41.644 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.644 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.644 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.644 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-23T23:13:41.644 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.644 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.644 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.644 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-03-23T23:13:41.644 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.644 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.645 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.645 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.645 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_promtail": { 2024-03-23T23:13:41.645 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_promtail", 2024-03-23T23:13:41.645 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.645 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.645 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.645 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-23T23:13:41.645 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.645 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.645 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.645 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Promtail container image", 2024-03-23T23:13:41.646 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.646 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.646 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.646 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.646 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_snmp_gateway": { 2024-03-23T23:13:41.646 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_snmp_gateway", 2024-03-23T23:13:41.646 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.646 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.646 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.646 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-23T23:13:41.646 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.646 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.646 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.647 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SNMP Gateway container image", 2024-03-23T23:13:41.647 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.647 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.647 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.647 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.647 INFO:teuthology.orchestra.run.smithi012.stdout: "container_init": { 2024-03-23T23:13:41.647 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_init", 2024-03-23T23:13:41.647 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.647 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.647 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.647 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.647 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.647 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.647 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.647 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Run podman/docker with `--init`", 2024-03-23T23:13:41.648 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.648 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.648 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.648 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.648 INFO:teuthology.orchestra.run.smithi012.stdout: "daemon_cache_timeout": { 2024-03-23T23:13:41.648 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "daemon_cache_timeout", 2024-03-23T23:13:41.648 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.648 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.648 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.648 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-23T23:13:41.648 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.648 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.648 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.648 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-23T23:13:41.648 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.648 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.649 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.649 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.649 INFO:teuthology.orchestra.run.smithi012.stdout: "default_cephadm_command_timeout": { 2024-03-23T23:13:41.649 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default_cephadm_command_timeout", 2024-03-23T23:13:41.649 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.649 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.649 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.649 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "900", 2024-03-23T23:13:41.649 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.649 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.649 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.649 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-23T23:13:41.649 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.649 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.650 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.650 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.650 INFO:teuthology.orchestra.run.smithi012.stdout: "default_registry": { 2024-03-23T23:13:41.650 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default_registry", 2024-03-23T23:13:41.650 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.650 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.650 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.650 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io", 2024-03-23T23:13:41.650 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.650 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.650 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.650 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-23T23:13:41.650 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.650 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.650 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.651 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.651 INFO:teuthology.orchestra.run.smithi012.stdout: "device_cache_timeout": { 2024-03-23T23:13:41.651 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_cache_timeout", 2024-03-23T23:13:41.651 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.651 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.651 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.651 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1800", 2024-03-23T23:13:41.651 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.651 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.651 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.651 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache device inventory", 2024-03-23T23:13:41.651 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.651 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.651 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.651 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.652 INFO:teuthology.orchestra.run.smithi012.stdout: "device_enhanced_scan": { 2024-03-23T23:13:41.652 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_enhanced_scan", 2024-03-23T23:13:41.652 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.652 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.652 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.652 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.652 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.652 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.652 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.652 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-23T23:13:41.652 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.652 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.652 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.652 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.652 INFO:teuthology.orchestra.run.smithi012.stdout: "facts_cache_timeout": { 2024-03-23T23:13:41.652 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "facts_cache_timeout", 2024-03-23T23:13:41.653 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.653 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.653 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.653 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-23T23:13:41.653 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.653 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.653 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.653 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache host facts data", 2024-03-23T23:13:41.653 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.653 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.653 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.653 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.653 INFO:teuthology.orchestra.run.smithi012.stdout: "host_check_interval": { 2024-03-23T23:13:41.653 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "host_check_interval", 2024-03-23T23:13:41.653 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.654 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.654 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.654 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-23T23:13:41.654 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.654 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.654 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.654 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to perform a host check", 2024-03-23T23:13:41.654 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.654 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.654 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.654 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.654 INFO:teuthology.orchestra.run.smithi012.stdout: "hw_monitoring": { 2024-03-23T23:13:41.654 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "hw_monitoring", 2024-03-23T23:13:41.654 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.654 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.655 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.655 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.655 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.655 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.655 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.655 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-23T23:13:41.655 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.655 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.655 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.655 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.655 INFO:teuthology.orchestra.run.smithi012.stdout: "inventory_list_all": { 2024-03-23T23:13:41.655 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "inventory_list_all", 2024-03-23T23:13:41.655 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.655 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.655 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.655 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.656 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.656 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.656 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.656 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-23T23:13:41.656 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.656 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.656 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.656 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.656 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.656 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.656 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.656 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.656 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.656 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.656 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.657 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.657 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.657 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.657 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.657 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.657 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.657 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.657 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.657 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.657 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.657 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.657 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.657 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.657 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.657 INFO:teuthology.orchestra.run.smithi012.stdout: "log_refresh_metadata": { 2024-03-23T23:13:41.657 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_refresh_metadata", 2024-03-23T23:13:41.658 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.658 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.658 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.658 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.658 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.658 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.658 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.658 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-23T23:13:41.658 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.658 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.658 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.658 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.658 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.658 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.658 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.659 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.659 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.659 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.659 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.659 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.659 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.659 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-23T23:13:41.659 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.659 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.659 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.659 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.659 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.659 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.659 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.659 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.659 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.660 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.660 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.660 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.660 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.660 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.660 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.660 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.660 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.660 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.660 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.660 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.660 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.660 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.660 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.660 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.660 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.661 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.661 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.661 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.661 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.661 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.661 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.661 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.661 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.661 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.661 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.661 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.661 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.661 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.661 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.661 INFO:teuthology.orchestra.run.smithi012.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-23T23:13:41.662 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-23T23:13:41.662 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.662 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.662 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.662 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.662 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.662 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.662 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.662 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-23T23:13:41.662 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.662 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.662 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.662 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.662 INFO:teuthology.orchestra.run.smithi012.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-23T23:13:41.663 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-23T23:13:41.663 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.663 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.663 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.663 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "*", 2024-03-23T23:13:41.663 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.663 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.663 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.663 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-23T23:13:41.663 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.663 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.663 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.663 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.663 INFO:teuthology.orchestra.run.smithi012.stdout: "max_count_per_host": { 2024-03-23T23:13:41.663 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_count_per_host", 2024-03-23T23:13:41.664 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.664 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.664 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.664 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-23T23:13:41.664 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.664 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.664 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.664 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "max number of daemons per service per host", 2024-03-23T23:13:41.664 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.664 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.664 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.664 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.664 INFO:teuthology.orchestra.run.smithi012.stdout: "max_osd_draining_count": { 2024-03-23T23:13:41.664 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_osd_draining_count", 2024-03-23T23:13:41.664 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.664 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.665 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.665 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-23T23:13:41.665 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.665 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.665 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.665 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-23T23:13:41.665 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.665 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.665 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.665 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.665 INFO:teuthology.orchestra.run.smithi012.stdout: "migration_current": { 2024-03-23T23:13:41.665 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "migration_current", 2024-03-23T23:13:41.665 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.665 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.665 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.665 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.666 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.666 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.666 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.666 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "internal - do not modify", 2024-03-23T23:13:41.666 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.666 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.666 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.666 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.666 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": { 2024-03-23T23:13:41.666 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mode", 2024-03-23T23:13:41.666 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.666 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.666 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.666 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "root", 2024-03-23T23:13:41.666 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.667 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.667 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.667 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm-package", 2024-03-23T23:13:41.667 INFO:teuthology.orchestra.run.smithi012.stdout: "root" 2024-03-23T23:13:41.667 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.667 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "mode for remote execution of cephadm", 2024-03-23T23:13:41.667 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.667 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.667 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.667 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.667 INFO:teuthology.orchestra.run.smithi012.stdout: "oob_default_addr": { 2024-03-23T23:13:41.667 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "oob_default_addr", 2024-03-23T23:13:41.667 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.667 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.667 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.668 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "169.254.1.1", 2024-03-23T23:13:41.668 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.668 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.668 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.668 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-23T23:13:41.668 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.668 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.668 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.668 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.668 INFO:teuthology.orchestra.run.smithi012.stdout: "prometheus_alerts_path": { 2024-03-23T23:13:41.668 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prometheus_alerts_path", 2024-03-23T23:13:41.668 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.668 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.668 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.668 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-23T23:13:41.668 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.669 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.669 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.669 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-23T23:13:41.669 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.669 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.669 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.669 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.669 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_insecure": { 2024-03-23T23:13:41.669 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_insecure", 2024-03-23T23:13:41.669 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.669 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.669 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.669 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.669 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.669 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.670 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.670 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-23T23:13:41.670 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.670 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.670 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.670 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.670 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_password": { 2024-03-23T23:13:41.670 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_password", 2024-03-23T23:13:41.670 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.670 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.670 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.670 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.670 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.670 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.670 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.671 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-23T23:13:41.671 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.671 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.671 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.671 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.671 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_url": { 2024-03-23T23:13:41.671 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_url", 2024-03-23T23:13:41.671 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.671 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.671 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.671 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.671 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.671 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.671 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.671 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-23T23:13:41.672 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.672 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.672 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.672 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.672 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_username": { 2024-03-23T23:13:41.672 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_username", 2024-03-23T23:13:41.672 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.672 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.672 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.672 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.672 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.672 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.672 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.672 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-23T23:13:41.673 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.673 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.673 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.673 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.673 INFO:teuthology.orchestra.run.smithi012.stdout: "secure_monitoring_stack": { 2024-03-23T23:13:41.673 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "secure_monitoring_stack", 2024-03-23T23:13:41.673 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.673 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.673 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.673 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.673 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.673 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.673 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.673 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-23T23:13:41.673 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.674 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.674 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.674 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.674 INFO:teuthology.orchestra.run.smithi012.stdout: "service_discovery_port": { 2024-03-23T23:13:41.674 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "service_discovery_port", 2024-03-23T23:13:41.674 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.674 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.674 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.674 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8765", 2024-03-23T23:13:41.674 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.674 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.674 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.674 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "cephadm service discovery port", 2024-03-23T23:13:41.674 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stdout: "ssh_config_file": { 2024-03-23T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssh_config_file", 2024-03-23T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-23T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.676 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.676 INFO:teuthology.orchestra.run.smithi012.stdout: "use_agent": { 2024-03-23T23:13:41.676 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "use_agent", 2024-03-23T23:13:41.676 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.676 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.676 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.676 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.676 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.676 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.676 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.676 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-23T23:13:41.676 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.676 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.676 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.676 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.677 INFO:teuthology.orchestra.run.smithi012.stdout: "use_repo_digest": { 2024-03-23T23:13:41.677 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "use_repo_digest", 2024-03-23T23:13:41.677 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.677 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.677 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.677 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.677 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.677 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.677 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.677 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-23T23:13:41.677 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.677 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.677 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.677 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.677 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_failed_host_check": { 2024-03-23T23:13:41.678 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_failed_host_check", 2024-03-23T23:13:41.678 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.678 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.678 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.678 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.678 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.678 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.678 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.678 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if the host check fails", 2024-03-23T23:13:41.678 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.678 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.678 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.678 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.678 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_stray_daemons": { 2024-03-23T23:13:41.678 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_stray_daemons", 2024-03-23T23:13:41.679 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.679 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.679 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.679 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.679 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.679 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.679 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.679 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-23T23:13:41.679 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.679 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.679 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.679 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.679 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_stray_hosts": { 2024-03-23T23:13:41.679 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_stray_hosts", 2024-03-23T23:13:41.679 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.680 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.680 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.680 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.680 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.680 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.680 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.680 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-23T23:13:41.680 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.680 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.680 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.680 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.680 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.680 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.680 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.680 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crash", 2024-03-23T23:13:41.681 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.681 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.681 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.681 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.681 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.681 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.681 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.681 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.681 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.681 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.681 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.681 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.681 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.681 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.682 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.682 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.682 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.682 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.682 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.682 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.682 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.682 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.682 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.682 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.682 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.682 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.682 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.682 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.682 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.683 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.683 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.683 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.683 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.683 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.683 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.683 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.683 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.683 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.683 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.683 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.683 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.683 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.683 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.683 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.684 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.684 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.684 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.684 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.684 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.684 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.684 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.684 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.684 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.684 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.684 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.684 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.684 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.684 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.685 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.685 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.685 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.685 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.685 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.685 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.685 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.685 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.685 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.685 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.685 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.685 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.685 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.685 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.685 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.686 INFO:teuthology.orchestra.run.smithi012.stdout: "retain_interval": { 2024-03-23T23:13:41.686 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "retain_interval", 2024-03-23T23:13:41.686 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.686 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.686 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.686 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "31536000", 2024-03-23T23:13:41.686 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.686 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.686 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.686 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-23T23:13:41.686 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.686 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.686 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.686 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.687 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_recent_interval": { 2024-03-23T23:13:41.687 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_recent_interval", 2024-03-23T23:13:41.687 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.687 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.687 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.687 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1209600", 2024-03-23T23:13:41.687 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.687 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.687 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.687 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-23T23:13:41.687 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.687 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.687 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.687 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.687 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.688 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.688 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.688 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "dashboard", 2024-03-23T23:13:41.688 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.688 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.688 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.688 INFO:teuthology.orchestra.run.smithi012.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-23T23:13:41.688 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-23T23:13:41.688 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.688 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.688 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.688 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-23T23:13:41.688 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.688 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.689 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.689 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.689 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.689 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.689 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.689 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.689 INFO:teuthology.orchestra.run.smithi012.stdout: "ALERTMANAGER_API_HOST": { 2024-03-23T23:13:41.689 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-23T23:13:41.689 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.689 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.689 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.689 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.689 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.689 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.689 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.689 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.690 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.690 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.690 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.690 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.690 INFO:teuthology.orchestra.run.smithi012.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-23T23:13:41.690 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-23T23:13:41.690 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.690 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.690 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.690 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.690 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.690 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.690 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.690 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.690 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.691 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.691 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.691 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.691 INFO:teuthology.orchestra.run.smithi012.stdout: "AUDIT_API_ENABLED": { 2024-03-23T23:13:41.691 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "AUDIT_API_ENABLED", 2024-03-23T23:13:41.691 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.691 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.691 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.691 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.691 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.691 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.691 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.691 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.691 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.691 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.692 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.692 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.692 INFO:teuthology.orchestra.run.smithi012.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-23T23:13:41.692 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-23T23:13:41.692 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.692 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.692 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.692 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.692 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.692 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.692 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.692 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.692 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.692 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.692 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.692 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.692 INFO:teuthology.orchestra.run.smithi012.stdout: "ENABLE_BROWSABLE_API": { 2024-03-23T23:13:41.693 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-23T23:13:41.693 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.693 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.693 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.693 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.693 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.693 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.693 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.693 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.693 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.693 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.693 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.693 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.693 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-23T23:13:41.693 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-23T23:13:41.693 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.693 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.694 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.694 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.694 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.694 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.694 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.694 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.694 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.694 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.694 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.694 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.694 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-23T23:13:41.694 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-23T23:13:41.694 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.694 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.694 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.694 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.695 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.695 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.695 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.695 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.695 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.695 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.695 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.695 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.695 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-23T23:13:41.695 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-23T23:13:41.695 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.695 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.695 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.695 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.695 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.695 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.695 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.696 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.696 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.696 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.696 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.696 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.696 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-23T23:13:41.696 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-23T23:13:41.696 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.696 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.696 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.696 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.696 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.696 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.696 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.697 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.697 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.697 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.697 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.697 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.697 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-23T23:13:41.697 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-23T23:13:41.697 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.697 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.697 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.697 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.697 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.697 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.697 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.698 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.698 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.698 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.698 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.698 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.698 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-23T23:13:41.698 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-23T23:13:41.698 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.698 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.698 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.698 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.698 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.698 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.698 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.698 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.699 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.699 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.699 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.699 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.699 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-23T23:13:41.699 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-23T23:13:41.699 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.699 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.699 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.699 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.699 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.699 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.699 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.699 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.699 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.699 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.700 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.700 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.700 INFO:teuthology.orchestra.run.smithi012.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-23T23:13:41.700 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-23T23:13:41.700 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.700 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.700 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.700 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.700 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.700 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.700 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.700 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.700 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.700 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.700 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.700 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.701 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_PASSWORD": { 2024-03-23T23:13:41.701 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-23T23:13:41.701 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.701 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.701 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.701 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-03-23T23:13:41.701 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.701 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.701 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.701 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.701 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.701 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.701 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.701 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.702 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-23T23:13:41.702 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-23T23:13:41.702 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.702 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.702 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.702 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.702 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.702 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.702 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.702 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.702 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.702 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.702 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.702 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.703 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_URL": { 2024-03-23T23:13:41.703 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_URL", 2024-03-23T23:13:41.703 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.703 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.703 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.703 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.703 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.703 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.703 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.703 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.703 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.703 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.703 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.703 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.703 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_USERNAME": { 2024-03-23T23:13:41.704 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-23T23:13:41.704 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.704 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.704 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.704 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-03-23T23:13:41.704 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.704 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.704 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.704 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.704 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.704 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.704 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.704 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.704 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-23T23:13:41.705 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-23T23:13:41.705 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.705 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.705 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.705 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.705 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.705 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.705 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.705 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.705 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.705 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.705 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.705 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.705 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-23T23:13:41.705 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-23T23:13:41.706 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.706 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.706 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.706 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.706 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.706 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.706 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.706 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.706 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.706 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.706 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.706 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.706 INFO:teuthology.orchestra.run.smithi012.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-23T23:13:41.706 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-23T23:13:41.707 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.707 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.707 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.707 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.707 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.707 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.707 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.707 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.707 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.707 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.707 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.707 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.707 INFO:teuthology.orchestra.run.smithi012.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-23T23:13:41.707 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-23T23:13:41.708 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.708 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.708 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.708 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.708 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.708 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.708 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.708 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.708 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.708 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.708 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.708 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.708 INFO:teuthology.orchestra.run.smithi012.stdout: "MULTICLUSTER_CONFIG": { 2024-03-23T23:13:41.708 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-23T23:13:41.708 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.709 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.709 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.709 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "{}", 2024-03-23T23:13:41.709 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.709 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.709 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.709 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.709 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.709 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.709 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.709 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.709 INFO:teuthology.orchestra.run.smithi012.stdout: "PROMETHEUS_API_HOST": { 2024-03-23T23:13:41.709 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-23T23:13:41.709 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.710 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.710 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.710 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.710 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.710 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.710 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.710 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.710 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.710 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.710 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.710 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.710 INFO:teuthology.orchestra.run.smithi012.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-23T23:13:41.710 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-23T23:13:41.711 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.711 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.711 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.711 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.711 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.711 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.711 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.711 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.711 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.711 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.711 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.711 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.711 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-23T23:13:41.711 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-23T23:13:41.712 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.712 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.712 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.712 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.712 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.712 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.712 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.712 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.712 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.712 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.712 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.712 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.712 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-23T23:13:41.712 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-23T23:13:41.712 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.713 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.713 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.713 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.713 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.713 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.713 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.713 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.713 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.713 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.713 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.713 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.713 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-23T23:13:41.713 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-23T23:13:41.713 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.713 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.713 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.714 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.714 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.714 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.714 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.714 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.714 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.714 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.714 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.714 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.714 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-23T23:13:41.714 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-23T23:13:41.714 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.714 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.714 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.715 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.715 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.715 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.715 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.715 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.715 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.715 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.715 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.715 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.715 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-23T23:13:41.715 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-23T23:13:41.715 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.715 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.715 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.715 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.715 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.716 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.716 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.716 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.716 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.716 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.716 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.716 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.716 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-23T23:13:41.716 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-23T23:13:41.716 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.716 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.716 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.716 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.716 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.716 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.716 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.717 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.717 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.717 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.717 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.717 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.717 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-23T23:13:41.717 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-23T23:13:41.717 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.717 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.717 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.717 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.717 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.717 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.717 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.717 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.717 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.717 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.718 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.718 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.718 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_ENABLED": { 2024-03-23T23:13:41.718 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-23T23:13:41.718 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.718 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.718 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.718 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.718 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.718 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.718 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.718 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.718 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.718 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.718 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.718 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.718 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-23T23:13:41.719 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-23T23:13:41.719 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.719 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.719 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.719 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-23T23:13:41.719 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.719 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.719 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.719 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.719 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.719 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.719 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.719 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.719 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-23T23:13:41.719 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-23T23:13:41.719 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.719 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.720 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.720 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-23T23:13:41.720 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.720 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.720 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.720 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.720 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.720 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.720 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.720 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.720 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-23T23:13:41.720 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-23T23:13:41.720 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.720 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.720 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.720 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8", 2024-03-23T23:13:41.721 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.721 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.721 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.721 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.721 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.721 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.721 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.721 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.721 INFO:teuthology.orchestra.run.smithi012.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-23T23:13:41.721 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-23T23:13:41.721 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.721 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.721 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.721 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "45", 2024-03-23T23:13:41.721 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.721 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.722 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.722 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.722 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.722 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.722 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.722 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.722 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_ACCESS_KEY": { 2024-03-23T23:13:41.722 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-23T23:13:41.722 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.722 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.722 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.722 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.722 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.723 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.723 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.723 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.723 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.723 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.723 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.723 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.723 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-23T23:13:41.723 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-23T23:13:41.723 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.723 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.723 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.723 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-03-23T23:13:41.723 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.723 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.723 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.724 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.724 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.724 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.724 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.724 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.724 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_SECRET_KEY": { 2024-03-23T23:13:41.724 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-23T23:13:41.724 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.724 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.724 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.724 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.724 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.724 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.724 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.724 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.724 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.725 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.725 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.725 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.725 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_SSL_VERIFY": { 2024-03-23T23:13:41.725 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-23T23:13:41.725 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.725 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.725 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.725 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.725 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.725 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.725 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.725 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.725 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.725 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.725 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.726 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.726 INFO:teuthology.orchestra.run.smithi012.stdout: "UNSAFE_TLS_v1_2": { 2024-03-23T23:13:41.726 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-23T23:13:41.726 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.726 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.726 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.726 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.726 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.726 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.726 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.726 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.726 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.726 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.726 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.726 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.726 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-23T23:13:41.727 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-23T23:13:41.727 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.727 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.727 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.727 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-23T23:13:41.727 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.727 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.727 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.727 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.727 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.727 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.727 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.727 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.727 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-23T23:13:41.728 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-23T23:13:41.728 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.728 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.728 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.728 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-23T23:13:41.728 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.728 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.728 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.728 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.728 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.728 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.728 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.728 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.729 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-23T23:13:41.729 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-23T23:13:41.729 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.729 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.729 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.729 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-03-23T23:13:41.729 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.729 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.729 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.729 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.729 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.729 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.729 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.729 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.729 INFO:teuthology.orchestra.run.smithi012.stdout: "cross_origin_url": { 2024-03-23T23:13:41.729 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cross_origin_url", 2024-03-23T23:13:41.729 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.730 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.730 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.730 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.730 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.730 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.730 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.730 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.730 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.730 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.730 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.730 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.730 INFO:teuthology.orchestra.run.smithi012.stdout: "crt_file": { 2024-03-23T23:13:41.730 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crt_file", 2024-03-23T23:13:41.730 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.730 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.731 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.731 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.731 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.731 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.731 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.731 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.731 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.731 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.731 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.731 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.731 INFO:teuthology.orchestra.run.smithi012.stdout: "debug": { 2024-03-23T23:13:41.731 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "debug", 2024-03-23T23:13:41.731 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.731 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.732 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.732 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.732 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.732 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.732 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.732 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable/disable debug options", 2024-03-23T23:13:41.732 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.732 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.732 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.732 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.732 INFO:teuthology.orchestra.run.smithi012.stdout: "jwt_token_ttl": { 2024-03-23T23:13:41.732 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "jwt_token_ttl", 2024-03-23T23:13:41.732 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.732 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.732 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.733 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "28800", 2024-03-23T23:13:41.733 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.733 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.733 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.733 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.733 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.733 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.733 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.733 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.733 INFO:teuthology.orchestra.run.smithi012.stdout: "key_file": { 2024-03-23T23:13:41.733 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "key_file", 2024-03-23T23:13:41.733 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.733 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.733 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.733 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.733 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.734 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.734 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.734 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.734 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.734 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.734 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.734 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.734 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.734 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.734 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.734 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.734 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.734 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.734 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.734 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.734 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.734 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.735 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.735 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.735 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.735 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.735 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.735 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.735 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.735 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.735 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.735 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.735 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.735 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.735 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.735 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.735 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.735 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.735 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.736 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.736 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.736 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.736 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.736 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.736 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.736 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.736 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.736 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.736 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.736 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.736 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.736 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.736 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.736 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.736 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.736 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.737 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.737 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.737 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.737 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.737 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.737 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.737 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.737 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.737 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.737 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.737 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.737 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.737 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.737 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.737 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.737 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.738 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.738 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.738 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.738 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.738 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.738 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.738 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.738 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.738 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.738 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.738 INFO:teuthology.orchestra.run.smithi012.stdout: "motd": { 2024-03-23T23:13:41.738 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "motd", 2024-03-23T23:13:41.738 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.738 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.738 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.739 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.739 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.739 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.739 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.739 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "The message of the day", 2024-03-23T23:13:41.739 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.739 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.739 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.739 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.739 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect_resolve_ip_addr": { 2024-03-23T23:13:41.739 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "redirect_resolve_ip_addr", 2024-03-23T23:13:41.739 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.739 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.739 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.739 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.739 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.739 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.740 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.740 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.740 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.740 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.740 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.740 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.740 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-03-23T23:13:41.740 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-03-23T23:13:41.740 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.740 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.740 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.740 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "::", 2024-03-23T23:13:41.740 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.740 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.740 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.741 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.741 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.741 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.741 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.741 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.741 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-03-23T23:13:41.741 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-03-23T23:13:41.741 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.741 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.741 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.741 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8080", 2024-03-23T23:13:41.741 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.741 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.741 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.741 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.741 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.741 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.742 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.742 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.742 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl": { 2024-03-23T23:13:41.742 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl", 2024-03-23T23:13:41.742 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.742 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.742 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.742 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.742 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.742 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.742 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.742 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.742 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.742 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.742 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.742 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.742 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl_server_port": { 2024-03-23T23:13:41.743 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl_server_port", 2024-03-23T23:13:41.743 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.743 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.743 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.743 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8443", 2024-03-23T23:13:41.743 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.743 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.743 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.743 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.743 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.743 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.743 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.743 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.743 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_behaviour": { 2024-03-23T23:13:41.743 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_behaviour", 2024-03-23T23:13:41.743 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.743 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.744 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.744 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "redirect", 2024-03-23T23:13:41.744 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.744 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.744 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.744 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.744 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect" 2024-03-23T23:13:41.744 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.744 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.744 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.744 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.744 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.744 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.744 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_error_status_code": { 2024-03-23T23:13:41.744 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_error_status_code", 2024-03-23T23:13:41.745 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.745 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.745 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.745 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "500", 2024-03-23T23:13:41.745 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "400", 2024-03-23T23:13:41.745 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "599", 2024-03-23T23:13:41.745 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.745 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.745 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.745 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.745 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.745 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.745 INFO:teuthology.orchestra.run.smithi012.stdout: "url_prefix": { 2024-03-23T23:13:41.745 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "url_prefix", 2024-03-23T23:13:41.745 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.746 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.746 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.746 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.746 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.746 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.746 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.746 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.746 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.746 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.746 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.746 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.746 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.746 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.746 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.747 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "devicehealth", 2024-03-23T23:13:41.747 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.747 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.747 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.747 INFO:teuthology.orchestra.run.smithi012.stdout: "enable_monitoring": { 2024-03-23T23:13:41.747 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enable_monitoring", 2024-03-23T23:13:41.747 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.747 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.747 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.747 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.747 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.747 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.747 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.747 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "monitor device health metrics", 2024-03-23T23:13:41.747 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.747 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.748 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.748 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.748 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.748 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.748 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.748 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.748 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.748 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.748 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.748 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.748 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.748 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.748 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.748 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.749 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.749 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.749 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.749 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.749 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.749 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.749 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.749 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.749 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.749 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.749 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.749 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.749 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.749 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.749 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.750 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.750 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.750 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.750 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.750 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.750 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.750 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.750 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.750 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.750 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.750 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.750 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.750 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.750 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.751 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.751 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.751 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.751 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.751 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.751 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.751 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.751 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.751 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.751 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.751 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.751 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.751 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.751 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.751 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.752 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.752 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.752 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.752 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.752 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.752 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.752 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.752 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.752 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.752 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.752 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.752 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.752 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.752 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.752 INFO:teuthology.orchestra.run.smithi012.stdout: "mark_out_threshold": { 2024-03-23T23:13:41.752 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mark_out_threshold", 2024-03-23T23:13:41.753 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.753 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.753 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.753 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "2419200", 2024-03-23T23:13:41.753 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.753 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.753 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.753 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-23T23:13:41.753 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.753 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.753 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.753 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.753 INFO:teuthology.orchestra.run.smithi012.stdout: "pool_name": { 2024-03-23T23:13:41.753 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pool_name", 2024-03-23T23:13:41.753 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.753 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.754 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.754 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "device_health_metrics", 2024-03-23T23:13:41.754 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.754 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.754 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.754 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-23T23:13:41.754 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.754 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.754 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.754 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.754 INFO:teuthology.orchestra.run.smithi012.stdout: "retention_period": { 2024-03-23T23:13:41.754 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "retention_period", 2024-03-23T23:13:41.754 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.754 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.755 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.755 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15552000", 2024-03-23T23:13:41.755 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.755 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.755 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.755 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long to retain device health metrics", 2024-03-23T23:13:41.755 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.755 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.755 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.755 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.755 INFO:teuthology.orchestra.run.smithi012.stdout: "scrape_frequency": { 2024-03-23T23:13:41.755 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "scrape_frequency", 2024-03-23T23:13:41.755 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.755 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.755 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.755 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "86400", 2024-03-23T23:13:41.756 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.756 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.756 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.756 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-23T23:13:41.756 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.756 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.756 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.756 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.756 INFO:teuthology.orchestra.run.smithi012.stdout: "self_heal": { 2024-03-23T23:13:41.756 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "self_heal", 2024-03-23T23:13:41.756 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.756 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.756 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.756 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.756 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.757 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.757 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.757 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-23T23:13:41.757 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.757 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.757 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.757 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.757 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-23T23:13:41.757 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-23T23:13:41.757 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.757 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.757 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.757 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-23T23:13:41.757 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.757 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.758 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.758 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to wake up and check device health", 2024-03-23T23:13:41.758 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.758 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.758 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.758 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.758 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_threshold": { 2024-03-23T23:13:41.758 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_threshold", 2024-03-23T23:13:41.758 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.758 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.758 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.758 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "7257600", 2024-03-23T23:13:41.758 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.758 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.758 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.759 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-23T23:13:41.759 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.759 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.759 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.759 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.759 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.759 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.759 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.759 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "diskprediction_local", 2024-03-23T23:13:41.759 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.759 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.759 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.759 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.759 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.759 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.759 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.759 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.760 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.760 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.760 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.760 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.760 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.760 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.760 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.760 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.760 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.760 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.760 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.760 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.760 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.760 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.760 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.760 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.760 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.761 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.761 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.761 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.761 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.761 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.761 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.761 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.761 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.761 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.761 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.761 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.761 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.761 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.761 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.761 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.762 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.762 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.762 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.762 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.762 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.762 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.762 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.762 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.762 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.762 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.762 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.762 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.762 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.762 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.763 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.763 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.763 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.763 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.763 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.763 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.763 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.763 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.763 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.763 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.763 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.763 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.763 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.763 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.763 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.763 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.764 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.764 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.764 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.764 INFO:teuthology.orchestra.run.smithi012.stdout: "predict_interval": { 2024-03-23T23:13:41.764 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "predict_interval", 2024-03-23T23:13:41.764 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.764 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.764 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.764 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "86400", 2024-03-23T23:13:41.764 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.764 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.764 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.764 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.764 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.764 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.764 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.764 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.765 INFO:teuthology.orchestra.run.smithi012.stdout: "predictor_model": { 2024-03-23T23:13:41.765 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "predictor_model", 2024-03-23T23:13:41.765 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.765 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.765 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.765 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "prophetstor", 2024-03-23T23:13:41.765 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.765 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.765 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.765 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.765 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.765 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.765 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.765 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.765 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-23T23:13:41.766 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-23T23:13:41.766 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.766 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.766 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.766 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-23T23:13:41.766 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.766 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.766 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.766 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.766 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.766 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.766 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.766 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.766 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.766 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.766 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.766 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "influx", 2024-03-23T23:13:41.767 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": false, 2024-03-23T23:13:41.767 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "influxdb python module not found", 2024-03-23T23:13:41.767 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.767 INFO:teuthology.orchestra.run.smithi012.stdout: "batch_size": { 2024-03-23T23:13:41.767 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "batch_size", 2024-03-23T23:13:41.767 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.767 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.767 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.767 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5000", 2024-03-23T23:13:41.767 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.767 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.767 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.767 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-23T23:13:41.767 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.767 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.767 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.768 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.768 INFO:teuthology.orchestra.run.smithi012.stdout: "database": { 2024-03-23T23:13:41.768 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "database", 2024-03-23T23:13:41.768 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.768 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.768 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.768 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "ceph", 2024-03-23T23:13:41.768 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.768 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.768 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.768 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-23T23:13:41.768 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.768 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.768 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.769 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.769 INFO:teuthology.orchestra.run.smithi012.stdout: "hostname": { 2024-03-23T23:13:41.769 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "hostname", 2024-03-23T23:13:41.769 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.769 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.769 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.769 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.769 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.769 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.769 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.769 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB server hostname", 2024-03-23T23:13:41.769 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.769 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.769 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.769 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.770 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-23T23:13:41.770 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-23T23:13:41.770 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.770 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.770 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.770 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "30", 2024-03-23T23:13:41.770 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "5", 2024-03-23T23:13:41.770 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.770 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.770 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-23T23:13:41.770 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.770 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.770 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.770 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.771 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.771 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.771 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.771 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.771 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.771 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.771 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.771 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.771 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.771 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.771 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.771 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.771 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.771 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.771 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.771 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.771 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.772 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.772 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.772 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.772 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.772 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.772 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.772 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.772 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.772 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.772 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.772 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.772 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.772 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.772 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.772 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.772 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.772 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.773 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.773 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.773 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.773 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.773 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.773 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.773 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.773 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.773 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.773 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.773 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.773 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.773 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.773 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.773 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.773 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.773 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.774 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.774 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.774 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.774 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.774 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.774 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.774 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.774 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.774 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.774 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.774 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.774 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.774 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.774 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.775 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.775 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.775 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.775 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.775 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.775 INFO:teuthology.orchestra.run.smithi012.stdout: "password": { 2024-03-23T23:13:41.775 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "password", 2024-03-23T23:13:41.775 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.775 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.775 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.775 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.775 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.775 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.775 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.775 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "password of InfluxDB server user", 2024-03-23T23:13:41.775 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.776 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.776 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.776 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.776 INFO:teuthology.orchestra.run.smithi012.stdout: "port": { 2024-03-23T23:13:41.776 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "port", 2024-03-23T23:13:41.776 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.776 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.776 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.776 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8086", 2024-03-23T23:13:41.776 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.776 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.776 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.776 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB server port", 2024-03-23T23:13:41.776 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.776 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.776 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.776 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.777 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl": { 2024-03-23T23:13:41.777 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl", 2024-03-23T23:13:41.777 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.777 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.777 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.777 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "false", 2024-03-23T23:13:41.777 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.777 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.777 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.777 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-23T23:13:41.777 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.777 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.777 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.777 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.777 INFO:teuthology.orchestra.run.smithi012.stdout: "threads": { 2024-03-23T23:13:41.778 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "threads", 2024-03-23T23:13:41.778 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.778 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.778 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.778 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-03-23T23:13:41.778 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-03-23T23:13:41.778 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "32", 2024-03-23T23:13:41.778 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.778 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-23T23:13:41.778 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.778 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.778 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.778 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.778 INFO:teuthology.orchestra.run.smithi012.stdout: "username": { 2024-03-23T23:13:41.778 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "username", 2024-03-23T23:13:41.779 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.779 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.779 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.779 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.779 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.779 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.779 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.779 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "username of InfluxDB server user", 2024-03-23T23:13:41.779 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.779 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.779 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.779 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.779 INFO:teuthology.orchestra.run.smithi012.stdout: "verify_ssl": { 2024-03-23T23:13:41.779 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "verify_ssl", 2024-03-23T23:13:41.779 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.779 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.780 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.780 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "true", 2024-03-23T23:13:41.780 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.780 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.780 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.780 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-23T23:13:41.780 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.780 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.780 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.780 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.780 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.780 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.780 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.780 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "insights", 2024-03-23T23:13:41.780 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.780 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.780 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.781 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.781 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.781 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.781 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.781 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.781 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.781 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.781 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.781 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.781 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.781 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.781 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.781 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.781 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.781 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.781 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.782 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.782 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.782 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.782 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.782 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.782 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.782 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.782 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.782 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.782 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.782 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.782 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.782 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.782 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.782 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.782 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.783 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.783 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.783 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.783 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.783 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.783 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.783 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.783 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.783 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.783 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.783 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.783 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.783 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.783 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.783 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.783 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.783 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.784 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.784 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.784 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.784 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.784 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.784 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.784 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.784 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.784 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.784 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.784 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.784 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.784 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.784 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.784 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.785 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.785 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.785 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.785 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.785 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.785 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.785 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.785 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.785 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.785 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "iostat", 2024-03-23T23:13:41.785 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.785 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.785 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.785 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.785 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.785 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.786 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.786 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.786 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.786 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.786 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.786 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.786 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.786 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.786 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.786 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.786 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.786 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.786 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.786 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.786 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.786 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.787 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.787 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.787 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.787 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.787 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.787 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.787 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.787 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.787 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.787 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.787 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.787 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.787 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.787 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.787 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.787 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "k8sevents", 2024-03-23T23:13:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_event_retention_days": { 2024-03-23T23:13:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ceph_event_retention_days", 2024-03-23T23:13:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "7", 2024-03-23T23:13:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-23T23:13:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "config_check_secs": { 2024-03-23T23:13:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_check_secs", 2024-03-23T23:13:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-23T23:13:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "10", 2024-03-23T23:13:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-23T23:13:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "localpool", 2024-03-23T23:13:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: "failure_domain": { 2024-03-23T23:13:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "failure_domain", 2024-03-23T23:13:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "host", 2024-03-23T23:13:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "failure domain for any created local pool", 2024-03-23T23:13:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-23T23:13:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.800 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.800 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.800 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.800 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.800 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.800 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.800 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.800 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.800 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.800 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.800 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.800 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.800 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.800 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.800 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.800 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.800 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.801 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.801 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.801 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.801 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.801 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.801 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.801 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.801 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.801 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.801 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.801 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.801 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.801 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.801 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.802 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.802 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.802 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.802 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.802 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.802 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.802 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.802 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.802 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.802 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.802 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.802 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.802 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.802 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.802 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.802 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.803 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.803 INFO:teuthology.orchestra.run.smithi012.stdout: "min_size": { 2024-03-23T23:13:41.803 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "min_size", 2024-03-23T23:13:41.803 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.803 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.803 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.803 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.803 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.803 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.803 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.803 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default min_size for any created local pool", 2024-03-23T23:13:41.803 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-23T23:13:41.803 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.803 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.803 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.804 INFO:teuthology.orchestra.run.smithi012.stdout: "num_rep": { 2024-03-23T23:13:41.804 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "num_rep", 2024-03-23T23:13:41.804 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.804 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.804 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.804 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3", 2024-03-23T23:13:41.804 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.804 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.804 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.804 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default replica count for any created local pool", 2024-03-23T23:13:41.804 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.804 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.804 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.804 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.804 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_num": { 2024-03-23T23:13:41.805 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pg_num", 2024-03-23T23:13:41.805 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.805 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.805 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.805 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "128", 2024-03-23T23:13:41.805 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.805 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.805 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.805 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default pg_num for any created local pool", 2024-03-23T23:13:41.805 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.805 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.805 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.806 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.806 INFO:teuthology.orchestra.run.smithi012.stdout: "prefix": { 2024-03-23T23:13:41.806 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prefix", 2024-03-23T23:13:41.806 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.806 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.806 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.806 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.806 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.806 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.806 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.806 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "name prefix for any created local pool", 2024-03-23T23:13:41.806 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.806 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.807 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.807 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.807 INFO:teuthology.orchestra.run.smithi012.stdout: "subtree": { 2024-03-23T23:13:41.807 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "subtree", 2024-03-23T23:13:41.807 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.807 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.807 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.807 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "rack", 2024-03-23T23:13:41.807 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.807 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.807 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.807 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-23T23:13:41.807 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-23T23:13:41.807 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.807 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.807 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.808 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.808 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.808 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.808 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mds_autoscaler", 2024-03-23T23:13:41.808 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.808 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.808 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.808 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.808 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.808 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.808 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.808 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.808 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.808 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.809 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.809 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.809 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.809 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.809 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.809 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.809 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.809 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.809 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.809 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.809 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.809 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.809 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.809 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.809 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.810 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.810 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.810 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.810 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.810 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.810 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.810 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.810 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.810 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.810 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.810 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.810 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.810 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.810 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.810 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.811 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.811 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.811 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.811 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.811 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.811 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.811 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.811 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.811 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.811 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.811 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.811 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.811 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.811 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.811 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.812 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.812 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.812 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.812 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.812 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.812 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.812 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.812 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.812 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.812 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.812 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.812 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.812 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.812 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.812 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.812 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.812 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.813 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.813 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.813 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.813 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.813 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mirroring", 2024-03-23T23:13:41.813 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.813 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.813 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.813 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.813 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.813 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.813 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.813 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.813 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.813 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.813 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.814 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.814 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.814 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.814 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.814 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.814 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.814 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.814 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.814 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.814 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.814 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.814 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.814 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.814 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.815 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.815 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.815 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.815 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.815 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.815 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.815 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.815 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.815 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.815 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.815 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.815 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.815 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.815 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.815 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.815 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.816 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.816 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.816 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.816 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.816 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.816 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.816 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.816 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.816 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.816 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.816 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.816 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.816 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.816 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.816 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.817 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.817 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.817 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.817 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.817 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.817 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.817 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.817 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.817 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.817 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.817 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.817 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.817 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.817 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.817 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.817 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.818 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.818 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.818 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.818 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.818 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "nfs", 2024-03-23T23:13:41.818 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.818 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.818 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.818 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.818 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.818 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.818 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.818 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.818 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.819 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.819 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.819 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.819 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.819 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.819 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.819 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.819 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.819 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.819 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.819 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.819 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.819 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.819 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.819 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.820 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.820 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.820 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.820 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.820 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.820 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.820 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.820 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.820 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.820 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.820 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.820 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.820 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.820 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.821 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.821 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.821 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.821 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.821 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.821 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.821 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.821 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.821 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.821 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.821 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.821 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.821 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.821 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.821 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.821 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.822 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.822 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.822 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.822 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.822 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.822 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.822 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.822 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.822 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.822 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.822 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.822 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.822 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.822 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.822 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.823 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.823 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.823 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.823 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.823 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.823 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.823 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.823 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "orchestrator", 2024-03-23T23:13:41.823 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.823 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.823 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.823 INFO:teuthology.orchestra.run.smithi012.stdout: "fail_fs": { 2024-03-23T23:13:41.823 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "fail_fs", 2024-03-23T23:13:41.823 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.823 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.823 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.824 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.824 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.824 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.824 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.824 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-23T23:13:41.824 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.824 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.824 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.824 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.824 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.824 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.824 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.824 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.824 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.824 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.824 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.825 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.825 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.825 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.825 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.825 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.825 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.825 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.825 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.825 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.825 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.825 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.825 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.825 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.825 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.825 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.826 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.826 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.826 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.826 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.826 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.826 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.826 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.826 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.826 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.826 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.826 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.826 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.826 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.826 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.827 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.827 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.827 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.827 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.827 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.827 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.827 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.827 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.827 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.827 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.827 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.827 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.827 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.827 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.827 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.827 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.827 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.827 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.828 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.828 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.828 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.828 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.828 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.828 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.828 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.828 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.828 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.828 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.828 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.828 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.828 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.828 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.828 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.828 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.828 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator": { 2024-03-23T23:13:41.829 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "orchestrator", 2024-03-23T23:13:41.829 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.829 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.829 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.829 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.829 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.829 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.829 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.829 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm", 2024-03-23T23:13:41.829 INFO:teuthology.orchestra.run.smithi012.stdout: "rook", 2024-03-23T23:13:41.829 INFO:teuthology.orchestra.run.smithi012.stdout: "test_orchestrator" 2024-03-23T23:13:41.829 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.829 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Orchestrator backend", 2024-03-23T23:13:41.829 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.829 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.829 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.830 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.830 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.830 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.830 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.830 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "osd_perf_query", 2024-03-23T23:13:41.830 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.830 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.830 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.830 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.830 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.830 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.830 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.830 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.830 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.831 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.831 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.831 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.831 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.831 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.831 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.831 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.831 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.831 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.831 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.831 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.831 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.831 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.831 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.831 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.832 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.832 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.832 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.832 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.832 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.832 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.832 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.832 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.832 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.832 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.832 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.832 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.832 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.832 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.832 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.833 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.833 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.833 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.833 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.833 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.833 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.833 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.833 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.833 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.833 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.833 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.833 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.833 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.833 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.834 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.834 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.834 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.834 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.834 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.834 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.834 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.834 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.834 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.834 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.834 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.834 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.834 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.834 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.834 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.835 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.835 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.835 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.835 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.835 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.835 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.835 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.835 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.835 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "osd_support", 2024-03-23T23:13:41.835 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.835 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.835 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.835 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.835 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.835 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.836 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.836 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.836 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.836 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.836 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.836 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.836 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.836 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.836 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.836 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.836 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.836 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.836 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.836 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.836 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.837 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.837 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.837 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.837 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.837 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.837 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.837 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.837 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.837 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.837 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.837 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.837 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.837 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.837 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.837 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.838 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.838 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.838 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.838 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.838 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.838 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.838 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.838 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.838 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.838 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.838 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.838 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.838 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.838 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.838 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.839 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.839 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.839 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.839 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.839 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.839 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.839 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.839 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.839 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.839 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.839 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.839 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.839 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.839 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.839 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.840 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.840 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.840 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.840 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.840 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.840 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.840 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.840 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.840 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.840 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.840 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pg_autoscaler", 2024-03-23T23:13:41.840 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.840 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.840 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.841 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.841 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.841 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.841 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.841 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.841 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.841 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.841 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.841 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.841 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.841 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.841 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.841 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.841 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.841 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.842 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.842 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.842 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.842 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.842 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.842 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.842 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.842 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.842 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.842 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.842 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.842 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.842 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.842 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.842 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.843 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.843 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.843 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.843 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.843 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.843 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.843 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.843 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.843 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.843 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.843 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.843 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.843 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.843 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.843 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.844 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.844 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.844 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.844 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.844 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.844 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.844 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.844 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.844 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.844 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.844 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.844 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.844 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.844 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.844 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.845 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.845 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.845 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.845 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.845 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.845 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.845 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.845 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.845 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.845 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.845 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-23T23:13:41.845 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-23T23:13:41.845 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.845 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.845 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.845 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-23T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stdout: "threshold": { 2024-03-23T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "threshold", 2024-03-23T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-23T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3.0", 2024-03-23T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1.0", 2024-03-23T23:13:41.847 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.847 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.847 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "scaling threshold", 2024-03-23T23:13:41.847 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-23T23:13:41.847 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.847 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.847 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.847 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.847 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.847 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.847 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "progress", 2024-03-23T23:13:41.847 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.847 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.847 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.847 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_pg_recovery_event": { 2024-03-23T23:13:41.848 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_pg_recovery_event", 2024-03-23T23:13:41.848 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.848 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.848 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.848 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.848 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.848 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.848 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.848 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-23T23:13:41.848 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.848 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.848 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.848 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.848 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": { 2024-03-23T23:13:41.848 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enabled", 2024-03-23T23:13:41.848 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.849 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.849 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.849 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.849 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.849 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.849 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.849 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.849 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.849 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.849 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.849 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.849 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.849 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.849 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.849 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.850 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.850 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.850 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.850 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.850 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.850 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.850 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.850 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.850 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.850 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.850 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.850 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.850 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.850 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.850 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.851 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.851 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.851 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.851 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.851 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.851 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.851 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.851 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.851 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.851 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.851 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.851 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.851 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.851 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.851 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.852 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.852 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.852 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.852 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.852 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.852 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.852 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.852 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.852 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.852 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.852 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.852 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.852 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.852 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.852 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.852 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.853 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.853 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.853 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.853 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.853 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.853 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.853 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.853 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.853 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.853 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.853 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.853 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.853 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.853 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.853 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.854 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.854 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.854 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.854 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.854 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.854 INFO:teuthology.orchestra.run.smithi012.stdout: "max_completed_events": { 2024-03-23T23:13:41.854 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_completed_events", 2024-03-23T23:13:41.854 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.854 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.854 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.854 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "50", 2024-03-23T23:13:41.854 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.854 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.854 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.855 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "number of past completed events to remember", 2024-03-23T23:13:41.855 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.855 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.855 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.855 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.855 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-23T23:13:41.855 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-23T23:13:41.855 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.855 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.855 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.855 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-03-23T23:13:41.855 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.855 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.855 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.855 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long the module is going to sleep", 2024-03-23T23:13:41.855 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.856 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.856 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.856 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.856 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.856 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.856 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.856 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prometheus", 2024-03-23T23:13:41.856 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.856 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.856 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.856 INFO:teuthology.orchestra.run.smithi012.stdout: "cache": { 2024-03-23T23:13:41.856 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cache", 2024-03-23T23:13:41.856 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.856 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.856 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.856 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.856 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.857 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.857 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.857 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.857 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.857 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.857 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.857 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.857 INFO:teuthology.orchestra.run.smithi012.stdout: "exclude_perf_counters": { 2024-03-23T23:13:41.857 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "exclude_perf_counters", 2024-03-23T23:13:41.857 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.857 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.857 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.857 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.857 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.858 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.858 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.858 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-23T23:13:41.858 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-23T23:13:41.858 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.858 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.858 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.858 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.858 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.858 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.858 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.858 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.858 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.858 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.859 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.859 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.859 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.859 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.859 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.859 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.859 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.859 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.859 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.859 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.859 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.859 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.859 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.859 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.859 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.860 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.860 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.860 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.860 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.860 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.860 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.860 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.860 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.860 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.860 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.860 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.860 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.860 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.860 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.860 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.861 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.861 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.861 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.861 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.861 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.861 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.861 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.861 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.861 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.861 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.861 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.861 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.861 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.861 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.862 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.862 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.862 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.862 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.862 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.862 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.862 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.862 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.862 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.862 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.862 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.862 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.862 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.862 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.863 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.863 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.863 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.863 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.863 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.863 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_stats_pools": { 2024-03-23T23:13:41.863 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_stats_pools", 2024-03-23T23:13:41.863 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.863 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.863 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.863 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.863 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.863 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.863 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.863 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-23T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-23T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "300", 2024-03-23T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.865 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.865 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.865 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.865 INFO:teuthology.orchestra.run.smithi012.stdout: "scrape_interval": { 2024-03-23T23:13:41.865 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "scrape_interval", 2024-03-23T23:13:41.865 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-23T23:13:41.865 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.865 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.865 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15.0", 2024-03-23T23:13:41.865 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.865 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.865 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.865 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.866 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.866 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.866 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.866 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.866 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-03-23T23:13:41.866 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-03-23T23:13:41.866 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.866 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.866 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.866 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "::", 2024-03-23T23:13:41.866 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.866 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.866 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.866 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-23T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-03-23T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-03-23T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "9283", 2024-03-23T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-23T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stdout: "stale_cache_strategy": { 2024-03-23T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "stale_cache_strategy", 2024-03-23T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "log", 2024-03-23T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_behaviour": { 2024-03-23T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_behaviour", 2024-03-23T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "default", 2024-03-23T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stdout: "default", 2024-03-23T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stdout: "error" 2024-03-23T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_error_status_code": { 2024-03-23T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_error_status_code", 2024-03-23T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "500", 2024-03-23T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "400", 2024-03-23T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "599", 2024-03-23T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.871 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.871 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.871 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.871 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.871 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.871 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.871 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_support", 2024-03-23T23:13:41.871 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.871 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.871 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.871 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.871 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.871 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.871 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.871 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.872 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.872 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.872 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.872 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.872 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.872 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.872 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.872 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.872 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.872 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.872 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.872 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.872 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.872 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.872 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.874 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.874 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.874 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.874 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.874 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.874 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.874 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.874 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.874 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.874 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.874 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.874 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.874 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.874 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.874 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.875 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.875 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.875 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.875 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.875 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.875 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.875 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.875 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.875 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.875 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.875 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.875 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.875 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.875 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.875 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stdout: "max_concurrent_snap_create": { 2024-03-23T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_concurrent_snap_create", 2024-03-23T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-23T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.877 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.877 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.877 INFO:teuthology.orchestra.run.smithi012.stdout: "mirror_snapshot_schedule": { 2024-03-23T23:13:41.877 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mirror_snapshot_schedule", 2024-03-23T23:13:41.877 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.877 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.877 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.877 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.877 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.877 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.877 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.877 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.877 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.877 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.877 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.878 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.878 INFO:teuthology.orchestra.run.smithi012.stdout: "trash_purge_schedule": { 2024-03-23T23:13:41.878 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "trash_purge_schedule", 2024-03-23T23:13:41.878 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.878 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.878 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.878 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.878 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.878 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.878 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.878 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.878 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.878 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.878 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.878 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.879 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.879 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.879 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.879 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "restful", 2024-03-23T23:13:41.879 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.879 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.879 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.879 INFO:teuthology.orchestra.run.smithi012.stdout: "enable_auth": { 2024-03-23T23:13:41.879 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enable_auth", 2024-03-23T23:13:41.879 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.879 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.879 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.879 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.879 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stdout: "key_file": { 2024-03-23T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "key_file", 2024-03-23T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.881 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.881 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.881 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.881 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.881 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.881 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.881 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.881 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.881 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.881 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.881 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.881 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.881 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.881 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.881 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.881 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.881 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.882 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.882 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.882 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.882 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.882 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.882 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.882 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.882 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.882 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.882 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.882 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.882 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.883 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.883 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.883 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.883 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.883 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.883 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.883 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.883 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.883 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.883 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.883 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.883 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.883 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.883 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.883 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.884 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.884 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.884 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.884 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.884 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.884 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.884 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.884 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.884 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.884 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.884 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.884 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.884 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.884 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.885 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.885 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.885 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.885 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.885 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.885 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.885 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.885 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.885 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.885 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.885 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.885 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.885 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.885 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.885 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.885 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.886 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.886 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.886 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-03-23T23:13:41.886 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-03-23T23:13:41.886 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.886 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.886 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.886 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.886 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.886 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.886 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.886 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.886 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.886 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.886 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.887 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.887 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-03-23T23:13:41.887 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-03-23T23:13:41.887 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.887 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.887 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.887 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.887 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.887 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.887 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.887 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.887 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.887 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.887 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.887 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.888 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.888 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.888 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.888 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rgw", 2024-03-23T23:13:41.888 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.888 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.888 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.888 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.888 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.888 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.888 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.888 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.888 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.889 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.889 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.889 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.889 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.889 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.889 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.889 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.889 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.889 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.889 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.889 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.889 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.889 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.889 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.889 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.890 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.890 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.890 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.890 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.890 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.890 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.890 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.890 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.890 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.890 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.890 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.890 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.890 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.890 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.890 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.891 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.891 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.891 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.891 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.891 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.891 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.891 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.891 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.891 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.891 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.891 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.891 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.891 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.891 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.891 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.892 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.892 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.892 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.892 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.892 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.892 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.892 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.892 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.892 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.892 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.892 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.892 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.892 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.892 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.893 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.893 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.893 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.893 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.893 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.893 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.893 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.893 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.893 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rook", 2024-03-23T23:13:41.893 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.893 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.893 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.893 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.894 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.894 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.894 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.894 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.894 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.894 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.894 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.894 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.894 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.894 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.894 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.894 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.894 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.894 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.895 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.895 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.895 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.895 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.895 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.895 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.895 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.895 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.895 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.895 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.895 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.895 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.895 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.895 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.895 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.895 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.896 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.896 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.896 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.896 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.896 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.896 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.896 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.896 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.896 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.896 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.896 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.896 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.896 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.896 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.897 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.897 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.897 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.897 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.897 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.897 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.897 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.897 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.897 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.897 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.897 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.897 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.897 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.897 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.898 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.898 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.898 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.898 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.898 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.898 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.898 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.898 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.898 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.898 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.898 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.898 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_class": { 2024-03-23T23:13:41.898 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "storage_class", 2024-03-23T23:13:41.898 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.898 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.899 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.899 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "local", 2024-03-23T23:13:41.899 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.899 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.899 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.899 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-23T23:13:41.899 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.899 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.899 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.899 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.899 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.899 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.899 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.899 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "selftest", 2024-03-23T23:13:41.899 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.899 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.900 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.900 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.900 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.900 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.900 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.900 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.900 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.900 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.900 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.900 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.900 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.900 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.900 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.900 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.900 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.901 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.901 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.901 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.901 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.901 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.901 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.901 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.901 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.901 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.901 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.901 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.901 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.901 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.901 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.901 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.902 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.902 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.902 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.902 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.902 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.902 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.902 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.902 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.902 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.902 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.902 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.902 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.902 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.902 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.902 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.902 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.903 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.903 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.903 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.903 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.903 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.903 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.903 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.903 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.903 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.903 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.903 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.903 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.903 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.903 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.903 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.903 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.904 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.904 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.904 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.904 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.904 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.904 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.904 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.904 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.904 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.904 INFO:teuthology.orchestra.run.smithi012.stdout: "roption1": { 2024-03-23T23:13:41.904 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "roption1", 2024-03-23T23:13:41.904 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.904 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.904 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.904 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.905 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.905 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.905 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.905 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.905 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.905 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.905 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.905 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.905 INFO:teuthology.orchestra.run.smithi012.stdout: "roption2": { 2024-03-23T23:13:41.905 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "roption2", 2024-03-23T23:13:41.905 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.905 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.905 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.905 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "xyz", 2024-03-23T23:13:41.905 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.906 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.906 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.906 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.906 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.906 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.906 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.906 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.906 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption1": { 2024-03-23T23:13:41.906 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption1", 2024-03-23T23:13:41.906 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.906 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.906 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.906 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.906 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.907 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.907 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.907 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.907 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.907 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.907 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.907 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.907 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption2": { 2024-03-23T23:13:41.907 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption2", 2024-03-23T23:13:41.907 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.907 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.907 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.907 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.907 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.907 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.907 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.908 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.908 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.908 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.908 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.908 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.908 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption3": { 2024-03-23T23:13:41.908 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption3", 2024-03-23T23:13:41.908 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-23T23:13:41.908 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.908 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.908 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.908 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.908 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.908 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.908 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.909 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.909 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.909 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.909 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.909 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption4": { 2024-03-23T23:13:41.909 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption4", 2024-03-23T23:13:41.909 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.909 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.909 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.909 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.909 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.909 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.909 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.910 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.910 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.910 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.910 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.910 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.910 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption5": { 2024-03-23T23:13:41.910 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption5", 2024-03-23T23:13:41.910 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.910 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.910 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.910 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.910 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.910 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.910 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.911 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.911 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.911 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.911 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.911 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.911 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption6": { 2024-03-23T23:13:41.911 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption6", 2024-03-23T23:13:41.911 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.911 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.911 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.911 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.911 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.911 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.911 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.911 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.912 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.912 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.912 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.912 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.912 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption7": { 2024-03-23T23:13:41.912 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption7", 2024-03-23T23:13:41.912 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.912 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.912 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.912 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.912 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-03-23T23:13:41.912 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "42", 2024-03-23T23:13:41.912 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.912 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.912 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.912 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.913 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.913 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.913 INFO:teuthology.orchestra.run.smithi012.stdout: "testkey": { 2024-03-23T23:13:41.913 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testkey", 2024-03-23T23:13:41.913 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.913 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.913 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.913 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.913 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.913 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.913 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.913 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.913 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.913 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.914 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.914 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.914 INFO:teuthology.orchestra.run.smithi012.stdout: "testlkey": { 2024-03-23T23:13:41.914 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testlkey", 2024-03-23T23:13:41.914 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.914 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.914 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.914 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.914 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.914 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.914 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.914 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.914 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.915 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.915 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.915 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.915 INFO:teuthology.orchestra.run.smithi012.stdout: "testnewline": { 2024-03-23T23:13:41.915 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testnewline", 2024-03-23T23:13:41.915 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.915 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.915 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.915 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.915 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.915 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.915 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.915 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.915 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.915 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.915 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.915 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.916 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.916 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.916 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.916 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snap_schedule", 2024-03-23T23:13:41.916 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.916 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.916 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.916 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_m_granularity": { 2024-03-23T23:13:41.916 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_m_granularity", 2024-03-23T23:13:41.916 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.916 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.916 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.916 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.916 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.916 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.917 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.917 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow minute scheduled snapshots", 2024-03-23T23:13:41.917 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.917 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.917 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.917 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.917 INFO:teuthology.orchestra.run.smithi012.stdout: "dump_on_update": { 2024-03-23T23:13:41.917 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "dump_on_update", 2024-03-23T23:13:41.917 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.917 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.917 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.917 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.917 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.917 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.918 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.918 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "dump database to debug log on update", 2024-03-23T23:13:41.918 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.918 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.918 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.918 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.918 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.918 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.918 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.918 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.918 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.918 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.918 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.918 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.919 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.919 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.919 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.919 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.919 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.919 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.919 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.919 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.919 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.919 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.919 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.919 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.919 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.919 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.919 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.920 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.920 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.920 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.920 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.920 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.920 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.920 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.920 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.920 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.920 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.920 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.920 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.920 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.920 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.920 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.920 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.921 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.921 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.921 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.921 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.921 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.921 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.921 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.921 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.921 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.921 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.921 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.921 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.921 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.921 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.921 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.922 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.922 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.922 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.922 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.922 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.922 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.922 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.922 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.922 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.922 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.922 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.922 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.923 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.923 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.923 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.923 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.923 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.923 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.923 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.923 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "stats", 2024-03-23T23:13:41.923 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.923 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.923 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.923 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.923 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.923 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.923 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.924 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.924 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.924 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.924 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.924 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.924 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.924 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.924 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.924 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.924 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.924 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.924 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.924 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.924 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.924 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.925 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.925 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.925 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.925 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.925 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.925 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.925 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.925 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.925 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.925 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.925 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.925 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.925 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.925 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.925 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.925 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.925 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.926 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.926 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.926 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.926 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.926 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.926 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.926 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.926 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.926 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.926 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.926 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.926 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.926 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.926 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.926 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.927 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.927 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.927 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.927 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.927 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.927 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.927 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.927 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.927 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.927 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.927 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.927 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.927 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.927 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.927 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.927 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.927 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.928 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.928 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.928 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.928 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.928 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.928 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "status", 2024-03-23T23:13:41.928 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.928 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.928 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.928 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.928 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.928 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.928 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.928 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.928 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.928 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.928 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.929 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.929 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.929 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.929 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.929 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.929 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.929 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.929 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.929 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.929 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.929 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.929 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.929 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.929 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.929 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.930 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.930 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.930 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.930 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.930 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.930 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.930 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.930 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.930 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.930 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.930 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.930 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.930 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.930 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.931 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.931 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.931 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.931 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.931 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.931 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.931 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.931 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.931 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.931 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.931 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.931 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.931 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.931 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.931 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.931 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.932 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.932 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.932 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.932 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.932 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.932 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.932 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.932 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.932 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.932 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.932 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.932 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.932 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.932 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.932 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.933 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.933 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.933 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.933 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.933 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.933 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "telegraf", 2024-03-23T23:13:41.933 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.933 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.933 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.933 INFO:teuthology.orchestra.run.smithi012.stdout: "address": { 2024-03-23T23:13:41.933 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "address", 2024-03-23T23:13:41.933 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.933 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.933 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.933 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-23T23:13:41.933 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.933 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.934 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.934 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.934 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.934 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.934 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.934 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.934 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-23T23:13:41.934 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-23T23:13:41.934 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.934 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.934 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.934 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15", 2024-03-23T23:13:41.934 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.934 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.935 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.935 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.935 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.935 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.935 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.935 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.935 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.935 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.935 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.935 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.935 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.935 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.935 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.935 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.935 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.936 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.936 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.936 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.936 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.936 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.936 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.936 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.936 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.936 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.936 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.936 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.936 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.936 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.936 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.936 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.937 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.937 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.937 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.937 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.937 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.937 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.937 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.937 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.937 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.937 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.937 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.937 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.937 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.937 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.937 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.938 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.938 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.938 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.938 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.938 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.938 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.938 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.938 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.938 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.938 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.938 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.938 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.938 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.938 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "telemetry", 2024-03-23T23:13:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_basic": { 2024-03-23T23:13:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_basic", 2024-03-23T23:13:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-23T23:13:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_crash": { 2024-03-23T23:13:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_crash", 2024-03-23T23:13:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-23T23:13:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_device": { 2024-03-23T23:13:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_device", 2024-03-23T23:13:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-23T23:13:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_ident": { 2024-03-23T23:13:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_ident", 2024-03-23T23:13:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-23T23:13:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_perf": { 2024-03-23T23:13:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_perf", 2024-03-23T23:13:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-23T23:13:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: "contact": { 2024-03-23T23:13:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "contact", 2024-03-23T23:13:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: "description": { 2024-03-23T23:13:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "description", 2024-03-23T23:13:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: "device_url": { 2024-03-23T23:13:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_url", 2024-03-23T23:13:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-23T23:13:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": { 2024-03-23T23:13:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enabled", 2024-03-23T23:13:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-23T23:13:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-23T23:13:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "24", 2024-03-23T23:13:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "8", 2024-03-23T23:13:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: "last_opt_revision": { 2024-03-23T23:13:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "last_opt_revision", 2024-03-23T23:13:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1", 2024-03-23T23:13:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: "leaderboard": { 2024-03-23T23:13:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "leaderboard", 2024-03-23T23:13:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.952 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.952 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.952 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.952 INFO:teuthology.orchestra.run.smithi012.stdout: "leaderboard_description": { 2024-03-23T23:13:41.952 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "leaderboard_description", 2024-03-23T23:13:41.952 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.952 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.952 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.952 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.952 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.952 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.952 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.952 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.952 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.952 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.953 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.953 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.953 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.953 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.953 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.953 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.953 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.953 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.953 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.953 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.953 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.953 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.953 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.954 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.954 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.954 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.954 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.954 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.954 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.954 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.954 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.954 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.954 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.954 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.954 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.954 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.954 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.954 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.955 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.955 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.955 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.955 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.955 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.955 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.955 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.955 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.955 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.955 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.955 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.955 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.955 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.955 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.955 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.956 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.956 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.956 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.956 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.956 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.956 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.956 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.956 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.956 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.956 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.956 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.956 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.956 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.956 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.957 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.957 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.957 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.957 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.957 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.957 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.957 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.957 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.957 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.957 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.957 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.957 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.957 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.957 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.957 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.958 INFO:teuthology.orchestra.run.smithi012.stdout: "organization": { 2024-03-23T23:13:41.958 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "organization", 2024-03-23T23:13:41.958 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.958 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.958 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.958 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.958 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.958 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.958 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.958 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.958 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.958 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.958 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.958 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.958 INFO:teuthology.orchestra.run.smithi012.stdout: "proxy": { 2024-03-23T23:13:41.959 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "proxy", 2024-03-23T23:13:41.959 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.959 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.959 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.959 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.959 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.959 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.959 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.959 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.959 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.959 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.959 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.959 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.959 INFO:teuthology.orchestra.run.smithi012.stdout: "url": { 2024-03-23T23:13:41.959 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "url", 2024-03-23T23:13:41.959 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.960 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.960 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.960 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-23T23:13:41.960 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.960 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.960 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.960 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.960 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.960 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.960 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.960 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.960 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.960 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.960 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.960 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test_orchestrator", 2024-03-23T23:13:41.960 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.961 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.961 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.961 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.961 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.961 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.961 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.961 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.961 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.961 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.961 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.961 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.961 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.961 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.961 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.962 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.962 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.962 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.962 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.962 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.962 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.962 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.962 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.962 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.962 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.962 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.962 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.962 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.962 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.962 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.963 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.963 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.963 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.963 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.963 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.963 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.963 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.963 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.963 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.963 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.963 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.963 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.963 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.963 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.963 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.964 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.964 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.964 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.964 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.964 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.964 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.964 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.964 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.964 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.964 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.964 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.964 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.964 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.964 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.964 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.965 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.965 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.965 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.965 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.965 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.965 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.965 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.965 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.965 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.965 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.965 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.965 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.965 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.965 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.965 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.966 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.966 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "volumes", 2024-03-23T23:13:41.966 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.966 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.966 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.966 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.966 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.966 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.966 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.966 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.966 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.966 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.966 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.966 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.967 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.967 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.967 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.967 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.967 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.967 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.967 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.967 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.967 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.967 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.967 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.967 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.967 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.967 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.967 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.967 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.968 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.968 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.968 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.968 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.968 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.968 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.968 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.968 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.968 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.968 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.968 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.968 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.968 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.968 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.968 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.968 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.968 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.969 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.969 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.969 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.969 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.969 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.969 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.969 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.969 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.969 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.969 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.969 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.969 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.969 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.969 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.969 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.969 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.970 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.970 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.970 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.970 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.970 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.970 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.970 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.970 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.970 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.970 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.970 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.970 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.970 INFO:teuthology.orchestra.run.smithi012.stdout: "max_concurrent_clones": { 2024-03-23T23:13:41.970 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_concurrent_clones", 2024-03-23T23:13:41.971 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.971 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.971 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.971 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "4", 2024-03-23T23:13:41.971 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.971 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.971 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.971 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-23T23:13:41.971 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.971 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.971 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.971 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.971 INFO:teuthology.orchestra.run.smithi012.stdout: "periodic_async_work": { 2024-03-23T23:13:41.971 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "periodic_async_work", 2024-03-23T23:13:41.972 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.972 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.972 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.972 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.972 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.972 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.972 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.972 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Periodically check for async work", 2024-03-23T23:13:41.972 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.972 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.972 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.972 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.972 INFO:teuthology.orchestra.run.smithi012.stdout: "snapshot_clone_delay": { 2024-03-23T23:13:41.972 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snapshot_clone_delay", 2024-03-23T23:13:41.972 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.972 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.973 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.973 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-23T23:13:41.973 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.973 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.973 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.973 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-23T23:13:41.973 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.973 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.973 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.973 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.973 INFO:teuthology.orchestra.run.smithi012.stdout: "snapshot_clone_no_wait": { 2024-03-23T23:13:41.973 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snapshot_clone_no_wait", 2024-03-23T23:13:41.973 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.973 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.973 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.974 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.974 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.974 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.974 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.974 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-23T23:13:41.974 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.974 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.974 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.974 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.974 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.974 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.974 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.974 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix", 2024-03-23T23:13:41.974 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.974 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.974 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.975 INFO:teuthology.orchestra.run.smithi012.stdout: "discovery_interval": { 2024-03-23T23:13:41.975 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "discovery_interval", 2024-03-23T23:13:41.975 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-23T23:13:41.975 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.975 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.975 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "100", 2024-03-23T23:13:41.975 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.975 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.975 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.975 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.975 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.975 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.975 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.975 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.975 INFO:teuthology.orchestra.run.smithi012.stdout: "identifier": { 2024-03-23T23:13:41.975 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "identifier", 2024-03-23T23:13:41.975 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.976 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.976 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.976 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.976 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.976 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.976 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.976 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.976 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.976 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.976 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.976 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.976 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-23T23:13:41.976 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-23T23:13:41.976 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.976 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.977 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.977 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-23T23:13:41.977 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.977 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.977 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.977 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.977 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.977 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.977 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.977 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.977 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.977 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.977 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.977 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.977 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.977 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.978 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.978 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.978 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.978 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.978 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.978 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.978 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.978 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.978 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.978 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.978 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.978 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.978 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.978 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.979 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.979 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.979 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.979 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.979 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.979 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.979 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.979 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.979 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.979 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.979 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.979 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.979 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.979 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.979 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.980 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.980 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.980 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.980 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.980 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.980 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.980 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.980 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.980 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.980 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.980 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.980 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.980 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.980 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.980 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.981 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.981 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.981 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.981 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.981 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.981 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.981 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.981 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.981 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.981 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.981 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.981 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.981 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.981 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.981 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.982 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.982 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.982 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.982 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.982 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.982 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_host": { 2024-03-23T23:13:41.982 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_host", 2024-03-23T23:13:41.982 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.982 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.982 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.982 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.982 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.982 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.982 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.982 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.983 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.983 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.983 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.983 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.983 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_port": { 2024-03-23T23:13:41.983 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_port", 2024-03-23T23:13:41.983 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.983 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.983 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.983 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10051", 2024-03-23T23:13:41.983 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.983 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.983 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.983 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.984 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.984 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.984 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.984 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.984 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_sender": { 2024-03-23T23:13:41.984 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_sender", 2024-03-23T23:13:41.984 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.984 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.984 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:41.984 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-23T23:13:41.984 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.984 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.984 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.984 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.984 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.985 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.985 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.985 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.985 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.985 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.985 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-23T23:13:41.985 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.985 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.985 INFO:teuthology.orchestra.run.smithi012.stdout: "modules": [ 2024-03-23T23:13:41.985 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm", 2024-03-23T23:13:41.985 INFO:teuthology.orchestra.run.smithi012.stdout: "dashboard", 2024-03-23T23:13:41.985 INFO:teuthology.orchestra.run.smithi012.stdout: "iostat", 2024-03-23T23:13:41.985 INFO:teuthology.orchestra.run.smithi012.stdout: "nfs", 2024-03-23T23:13:41.985 INFO:teuthology.orchestra.run.smithi012.stdout: "restful" 2024-03-23T23:13:41.985 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.986 INFO:teuthology.orchestra.run.smithi012.stdout: "available_modules": [ 2024-03-23T23:13:41.986 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:41.986 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "alerts", 2024-03-23T23:13:41.986 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:41.986 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:41.986 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:41.986 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-23T23:13:41.986 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-23T23:13:41.986 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:41.986 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.986 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.986 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-23T23:13:41.986 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.986 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.986 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.987 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How frequently to reexamine health status", 2024-03-23T23:13:41.987 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.987 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.987 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.987 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.987 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:41.987 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:41.987 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.987 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.987 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.987 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.987 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.987 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.987 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.988 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.988 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.988 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.988 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.988 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.988 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.988 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.988 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.988 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.988 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.988 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.988 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.988 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:41.988 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:41.988 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.989 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.989 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.989 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.989 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.989 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.989 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.989 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.989 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.989 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.989 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.989 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.989 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:41.989 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:41.989 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.989 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.990 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.990 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:41.990 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.990 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.990 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:41.990 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:41.990 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:41.990 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:41.990 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:41.990 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:41.990 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:41.990 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:41.990 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.990 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.991 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.991 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.991 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.991 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:41.991 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:41.991 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.991 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.991 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.991 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:41.991 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.991 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.991 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.991 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:41.991 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.991 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.992 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.992 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.992 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_destination": { 2024-03-23T23:13:41.992 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_destination", 2024-03-23T23:13:41.992 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.992 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.992 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.992 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.992 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.992 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.992 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.992 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Email address to send alerts to", 2024-03-23T23:13:41.992 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.992 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.992 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.993 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.993 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_from_name": { 2024-03-23T23:13:41.993 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_from_name", 2024-03-23T23:13:41.993 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.993 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.993 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.993 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "Ceph", 2024-03-23T23:13:41.993 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.993 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.993 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.993 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Email From: name", 2024-03-23T23:13:41.993 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.993 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.993 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.993 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.994 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_host": { 2024-03-23T23:13:41.994 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_host", 2024-03-23T23:13:41.994 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.994 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.994 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.994 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.994 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.994 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.994 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.994 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP server", 2024-03-23T23:13:41.994 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.994 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.994 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.994 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.994 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_password": { 2024-03-23T23:13:41.995 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_password", 2024-03-23T23:13:41.995 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.995 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.995 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.995 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.995 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.995 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.995 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.995 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Password to authenticate with", 2024-03-23T23:13:41.995 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.995 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.995 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.995 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.995 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_port": { 2024-03-23T23:13:41.996 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_port", 2024-03-23T23:13:41.996 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:41.996 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.996 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.996 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "465", 2024-03-23T23:13:41.996 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.996 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.996 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.996 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP port", 2024-03-23T23:13:41.996 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.996 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.996 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.996 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.996 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_sender": { 2024-03-23T23:13:41.996 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_sender", 2024-03-23T23:13:41.997 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.997 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.997 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.997 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.997 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.997 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.997 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.997 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP envelope sender", 2024-03-23T23:13:41.997 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.997 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.997 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.997 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.997 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_ssl": { 2024-03-23T23:13:41.997 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_ssl", 2024-03-23T23:13:41.998 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:41.998 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.998 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.998 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:41.998 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.998 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.998 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.998 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-23T23:13:41.998 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.998 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.998 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.998 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:41.998 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_user": { 2024-03-23T23:13:41.999 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_user", 2024-03-23T23:13:41.999 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:41.999 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:41.999 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:41.999 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:41.999 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:41.999 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:41.999 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:41.999 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "User to authenticate as", 2024-03-23T23:13:41.999 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:41.999 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:41.999 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:41.999 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:41.999 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.000 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.000 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.000 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "balancer", 2024-03-23T23:13:42.000 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.000 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.000 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.000 INFO:teuthology.orchestra.run.smithi012.stdout: "active": { 2024-03-23T23:13:42.000 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "active", 2024-03-23T23:13:42.000 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.000 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.000 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.000 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.000 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.000 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.000 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.001 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically balance PGs across cluster", 2024-03-23T23:13:42.001 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.001 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.001 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.001 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.001 INFO:teuthology.orchestra.run.smithi012.stdout: "begin_time": { 2024-03-23T23:13:42.001 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "begin_time", 2024-03-23T23:13:42.001 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.001 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.001 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.001 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0000", 2024-03-23T23:13:42.001 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.001 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.001 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.002 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "beginning time of day to automatically balance", 2024-03-23T23:13:42.002 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-23T23:13:42.002 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.002 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.002 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.002 INFO:teuthology.orchestra.run.smithi012.stdout: "begin_weekday": { 2024-03-23T23:13:42.002 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "begin_weekday", 2024-03-23T23:13:42.002 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-23T23:13:42.002 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.002 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.002 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-23T23:13:42.002 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0", 2024-03-23T23:13:42.002 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "6", 2024-03-23T23:13:42.002 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.003 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-23T23:13:42.003 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-23T23:13:42.003 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.003 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.003 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.003 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_max_iterations": { 2024-03-23T23:13:42.003 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_max_iterations", 2024-03-23T23:13:42.003 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-23T23:13:42.003 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.003 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.003 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "25", 2024-03-23T23:13:42.003 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-03-23T23:13:42.003 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "250", 2024-03-23T23:13:42.003 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.004 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-23T23:13:42.004 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.004 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.004 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.004 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.004 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_metrics": { 2024-03-23T23:13:42.004 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_metrics", 2024-03-23T23:13:42.004 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.004 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.004 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.004 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "pgs,objects,bytes", 2024-03-23T23:13:42.004 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.004 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.004 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.005 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-23T23:13:42.005 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-23T23:13:42.005 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.005 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.005 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.005 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_step": { 2024-03-23T23:13:42.005 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_step", 2024-03-23T23:13:42.005 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-23T23:13:42.005 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.005 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.005 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0.5", 2024-03-23T23:13:42.005 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0.001", 2024-03-23T23:13:42.005 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "0.999", 2024-03-23T23:13:42.005 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.005 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "aggressiveness of optimization", 2024-03-23T23:13:42.006 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-23T23:13:42.006 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.006 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.006 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.006 INFO:teuthology.orchestra.run.smithi012.stdout: "end_time": { 2024-03-23T23:13:42.006 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "end_time", 2024-03-23T23:13:42.006 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.006 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.006 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.006 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "2359", 2024-03-23T23:13:42.006 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.006 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.006 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.006 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "ending time of day to automatically balance", 2024-03-23T23:13:42.006 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-23T23:13:42.007 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.007 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.007 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.007 INFO:teuthology.orchestra.run.smithi012.stdout: "end_weekday": { 2024-03-23T23:13:42.007 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "end_weekday", 2024-03-23T23:13:42.007 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-23T23:13:42.007 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.007 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.007 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-23T23:13:42.007 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0", 2024-03-23T23:13:42.007 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "6", 2024-03-23T23:13:42.007 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.007 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-23T23:13:42.007 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-23T23:13:42.007 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.008 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.008 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.008 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.008 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.008 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.008 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.008 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.008 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.008 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.008 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.008 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.008 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.008 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.008 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.008 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.008 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.009 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.009 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.009 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.009 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.009 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.009 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.009 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.009 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.009 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.009 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.009 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.009 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.009 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.009 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.009 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.009 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.010 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.010 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.010 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.010 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.010 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.010 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.010 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.010 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.010 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.010 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.010 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.010 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.010 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.010 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.011 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.011 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.011 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.011 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.011 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.011 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.011 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.011 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.011 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.011 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.011 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.011 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.011 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.011 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.011 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.011 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.011 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.012 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.012 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.012 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.012 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.012 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.012 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.012 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.012 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.012 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.012 INFO:teuthology.orchestra.run.smithi012.stdout: "min_score": { 2024-03-23T23:13:42.012 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "min_score", 2024-03-23T23:13:42.012 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-23T23:13:42.012 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.012 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.012 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-23T23:13:42.013 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.013 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.013 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.013 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-23T23:13:42.013 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.013 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.013 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.013 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.013 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": { 2024-03-23T23:13:42.013 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mode", 2024-03-23T23:13:42.013 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.013 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.013 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.013 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "upmap", 2024-03-23T23:13:42.014 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.014 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.014 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.014 INFO:teuthology.orchestra.run.smithi012.stdout: "crush-compat", 2024-03-23T23:13:42.014 INFO:teuthology.orchestra.run.smithi012.stdout: "none", 2024-03-23T23:13:42.014 INFO:teuthology.orchestra.run.smithi012.stdout: "read", 2024-03-23T23:13:42.014 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap", 2024-03-23T23:13:42.014 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap-read" 2024-03-23T23:13:42.014 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.014 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Balancer mode", 2024-03-23T23:13:42.014 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.014 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.014 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.014 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.014 INFO:teuthology.orchestra.run.smithi012.stdout: "pool_ids": { 2024-03-23T23:13:42.015 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pool_ids", 2024-03-23T23:13:42.015 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.015 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.015 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.015 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.015 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.015 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.015 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.015 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-23T23:13:42.015 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.015 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.015 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.015 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.015 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-23T23:13:42.016 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-23T23:13:42.016 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:42.016 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.016 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.016 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-23T23:13:42.016 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.016 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.016 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.016 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-23T23:13:42.016 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.016 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.016 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.016 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.016 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap_max_deviation": { 2024-03-23T23:13:42.017 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "upmap_max_deviation", 2024-03-23T23:13:42.017 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.017 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.017 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.017 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-03-23T23:13:42.017 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-03-23T23:13:42.017 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.017 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.017 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-23T23:13:42.017 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-23T23:13:42.017 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.017 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.017 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.017 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap_max_optimizations": { 2024-03-23T23:13:42.017 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "upmap_max_optimizations", 2024-03-23T23:13:42.018 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-23T23:13:42.018 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.018 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.018 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-23T23:13:42.018 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.018 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.018 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.018 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-23T23:13:42.018 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.018 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.018 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.018 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.018 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.018 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.019 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.019 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephadm", 2024-03-23T23:13:42.019 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.019 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.019 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.019 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_down_multiplier": { 2024-03-23T23:13:42.019 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_down_multiplier", 2024-03-23T23:13:42.019 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-23T23:13:42.019 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.019 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.019 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3.0", 2024-03-23T23:13:42.019 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.019 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.019 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.019 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-23T23:13:42.020 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.020 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.020 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.020 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.020 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_refresh_rate": { 2024-03-23T23:13:42.020 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_refresh_rate", 2024-03-23T23:13:42.020 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:42.020 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.020 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.020 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "20", 2024-03-23T23:13:42.020 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.020 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.020 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.020 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-23T23:13:42.021 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.021 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.021 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.021 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.021 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_starting_port": { 2024-03-23T23:13:42.021 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_starting_port", 2024-03-23T23:13:42.021 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.021 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.021 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.021 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "4721", 2024-03-23T23:13:42.021 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.021 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.021 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.021 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-23T23:13:42.021 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.022 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.022 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.022 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.022 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_ptrace": { 2024-03-23T23:13:42.022 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_ptrace", 2024-03-23T23:13:42.022 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.022 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.022 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.022 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.022 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.022 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.022 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.022 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-23T23:13:42.023 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-23T23:13:42.023 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.023 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.023 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.023 INFO:teuthology.orchestra.run.smithi012.stdout: "autotune_interval": { 2024-03-23T23:13:42.023 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "autotune_interval", 2024-03-23T23:13:42.023 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:42.023 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.023 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.023 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-23T23:13:42.023 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.023 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.023 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.023 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-23T23:13:42.024 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.024 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.024 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.024 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.024 INFO:teuthology.orchestra.run.smithi012.stdout: "autotune_memory_target_ratio": { 2024-03-23T23:13:42.024 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "autotune_memory_target_ratio", 2024-03-23T23:13:42.024 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-23T23:13:42.024 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.024 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.024 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0.7", 2024-03-23T23:13:42.024 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.024 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.024 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.024 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-23T23:13:42.024 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.025 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.025 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.025 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.025 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm_log_destination": { 2024-03-23T23:13:42.025 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephadm_log_destination", 2024-03-23T23:13:42.025 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.025 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.025 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.025 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.025 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.025 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.025 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.025 INFO:teuthology.orchestra.run.smithi012.stdout: "file", 2024-03-23T23:13:42.025 INFO:teuthology.orchestra.run.smithi012.stdout: "file,syslog", 2024-03-23T23:13:42.025 INFO:teuthology.orchestra.run.smithi012.stdout: "syslog" 2024-03-23T23:13:42.025 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.026 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-23T23:13:42.026 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.026 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.026 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.026 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.026 INFO:teuthology.orchestra.run.smithi012.stdout: "cgroups_split": { 2024-03-23T23:13:42.026 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cgroups_split", 2024-03-23T23:13:42.026 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.026 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.026 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.026 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.026 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.026 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.026 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.027 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-23T23:13:42.027 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.027 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.027 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.027 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.027 INFO:teuthology.orchestra.run.smithi012.stdout: "config_checks_enabled": { 2024-03-23T23:13:42.027 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_checks_enabled", 2024-03-23T23:13:42.027 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.027 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.027 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.027 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.027 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.027 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.027 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.028 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-23T23:13:42.028 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.028 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.028 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.028 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.028 INFO:teuthology.orchestra.run.smithi012.stdout: "config_dashboard": { 2024-03-23T23:13:42.028 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_dashboard", 2024-03-23T23:13:42.028 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.028 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.028 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.028 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.028 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.028 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.028 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.028 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-23T23:13:42.028 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.029 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.029 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.029 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.029 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_alertmanager": { 2024-03-23T23:13:42.029 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_alertmanager", 2024-03-23T23:13:42.029 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.029 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.029 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.029 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-23T23:13:42.029 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.029 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.029 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.029 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-03-23T23:13:42.029 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.030 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.030 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.030 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.030 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_base": { 2024-03-23T23:13:42.030 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_base", 2024-03-23T23:13:42.030 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.030 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.030 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.030 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-23T23:13:42.030 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.030 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.030 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.030 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Container image name, without the tag", 2024-03-23T23:13:42.031 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.031 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.031 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.031 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.031 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_elasticsearch": { 2024-03-23T23:13:42.031 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_elasticsearch", 2024-03-23T23:13:42.031 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.031 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.031 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.031 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-23T23:13:42.031 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.031 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.031 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.031 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "elasticsearch container image", 2024-03-23T23:13:42.032 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.032 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.032 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.032 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.032 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_grafana": { 2024-03-23T23:13:42.032 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_grafana", 2024-03-23T23:13:42.032 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.032 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.032 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.032 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-23T23:13:42.032 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.032 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.032 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.032 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-03-23T23:13:42.033 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.033 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.033 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.033 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.033 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_haproxy": { 2024-03-23T23:13:42.033 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_haproxy", 2024-03-23T23:13:42.033 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.033 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.033 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.033 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-23T23:13:42.033 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.033 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.033 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.033 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "HAproxy container image", 2024-03-23T23:13:42.034 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.034 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.034 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.034 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.034 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_agent": { 2024-03-23T23:13:42.034 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_agent", 2024-03-23T23:13:42.034 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.034 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.034 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.034 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-23T23:13:42.034 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.034 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.034 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.035 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger agent container image", 2024-03-23T23:13:42.035 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.035 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.035 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.035 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.035 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_collector": { 2024-03-23T23:13:42.035 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_collector", 2024-03-23T23:13:42.035 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.035 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.035 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.035 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-23T23:13:42.035 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.035 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.035 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.036 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger collector container image", 2024-03-23T23:13:42.036 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.036 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.036 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.036 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.036 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_query": { 2024-03-23T23:13:42.036 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_query", 2024-03-23T23:13:42.036 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.036 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.036 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.036 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-23T23:13:42.036 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.036 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.036 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.036 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger query container image", 2024-03-23T23:13:42.037 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.037 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.037 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.037 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.037 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_keepalived": { 2024-03-23T23:13:42.037 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_keepalived", 2024-03-23T23:13:42.037 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.037 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.037 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.037 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-23T23:13:42.037 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.037 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.037 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.037 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Keepalived container image", 2024-03-23T23:13:42.037 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.037 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.038 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.038 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.038 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_loki": { 2024-03-23T23:13:42.038 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_loki", 2024-03-23T23:13:42.038 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.038 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.038 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.038 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-23T23:13:42.038 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.038 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.038 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.038 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Loki container image", 2024-03-23T23:13:42.038 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.039 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.039 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.039 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.039 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_node_exporter": { 2024-03-23T23:13:42.039 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_node_exporter", 2024-03-23T23:13:42.039 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.039 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.039 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.039 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-23T23:13:42.039 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.039 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.039 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.039 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-03-23T23:13:42.039 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.039 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.039 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.040 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.040 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_nvmeof": { 2024-03-23T23:13:42.040 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_nvmeof", 2024-03-23T23:13:42.040 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.040 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.040 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.040 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-23T23:13:42.040 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.040 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.040 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.040 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Nvme-of container image", 2024-03-23T23:13:42.040 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.040 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.040 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.040 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.040 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_prometheus": { 2024-03-23T23:13:42.041 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_prometheus", 2024-03-23T23:13:42.041 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.041 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.041 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.041 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-23T23:13:42.041 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.041 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.041 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.041 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-03-23T23:13:42.041 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.041 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.041 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.041 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.041 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_promtail": { 2024-03-23T23:13:42.041 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_promtail", 2024-03-23T23:13:42.042 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.042 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.042 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.042 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-23T23:13:42.042 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.042 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.042 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.042 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Promtail container image", 2024-03-23T23:13:42.042 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.042 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.042 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.042 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.042 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_snmp_gateway": { 2024-03-23T23:13:42.042 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_snmp_gateway", 2024-03-23T23:13:42.043 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.043 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.043 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.043 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-23T23:13:42.043 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.043 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.043 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.043 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SNMP Gateway container image", 2024-03-23T23:13:42.043 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.043 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.043 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.043 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.043 INFO:teuthology.orchestra.run.smithi012.stdout: "container_init": { 2024-03-23T23:13:42.043 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_init", 2024-03-23T23:13:42.043 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.044 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.044 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.044 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.044 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.044 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.044 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.044 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Run podman/docker with `--init`", 2024-03-23T23:13:42.044 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.044 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.044 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.044 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.044 INFO:teuthology.orchestra.run.smithi012.stdout: "daemon_cache_timeout": { 2024-03-23T23:13:42.044 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "daemon_cache_timeout", 2024-03-23T23:13:42.044 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:42.045 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.045 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.045 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-23T23:13:42.045 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.045 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.045 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.045 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-23T23:13:42.045 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.045 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.045 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.045 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.045 INFO:teuthology.orchestra.run.smithi012.stdout: "default_cephadm_command_timeout": { 2024-03-23T23:13:42.045 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default_cephadm_command_timeout", 2024-03-23T23:13:42.046 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.046 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.046 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.046 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "900", 2024-03-23T23:13:42.046 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.046 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.046 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.046 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-23T23:13:42.046 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.046 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.046 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.046 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.047 INFO:teuthology.orchestra.run.smithi012.stdout: "default_registry": { 2024-03-23T23:13:42.047 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default_registry", 2024-03-23T23:13:42.047 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.047 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.047 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.047 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io", 2024-03-23T23:13:42.047 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.047 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.047 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.047 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-23T23:13:42.047 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.047 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.047 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.047 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.048 INFO:teuthology.orchestra.run.smithi012.stdout: "device_cache_timeout": { 2024-03-23T23:13:42.048 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_cache_timeout", 2024-03-23T23:13:42.048 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:42.048 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.048 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.048 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1800", 2024-03-23T23:13:42.048 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.048 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.048 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.048 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache device inventory", 2024-03-23T23:13:42.048 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.048 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.048 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.048 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.049 INFO:teuthology.orchestra.run.smithi012.stdout: "device_enhanced_scan": { 2024-03-23T23:13:42.049 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_enhanced_scan", 2024-03-23T23:13:42.049 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.049 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.049 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.049 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.049 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.049 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.049 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.049 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-23T23:13:42.049 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.049 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.049 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.049 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.050 INFO:teuthology.orchestra.run.smithi012.stdout: "facts_cache_timeout": { 2024-03-23T23:13:42.050 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "facts_cache_timeout", 2024-03-23T23:13:42.050 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:42.050 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.050 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.050 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-23T23:13:42.050 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.050 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.050 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.050 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache host facts data", 2024-03-23T23:13:42.050 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.050 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.050 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.050 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.051 INFO:teuthology.orchestra.run.smithi012.stdout: "host_check_interval": { 2024-03-23T23:13:42.051 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "host_check_interval", 2024-03-23T23:13:42.051 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:42.051 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.051 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.051 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-23T23:13:42.051 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.051 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.051 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.051 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to perform a host check", 2024-03-23T23:13:42.051 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.051 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.051 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.052 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.052 INFO:teuthology.orchestra.run.smithi012.stdout: "hw_monitoring": { 2024-03-23T23:13:42.052 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "hw_monitoring", 2024-03-23T23:13:42.052 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.052 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.052 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.052 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.052 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.052 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.052 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.052 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-23T23:13:42.052 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.052 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.052 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.052 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.053 INFO:teuthology.orchestra.run.smithi012.stdout: "inventory_list_all": { 2024-03-23T23:13:42.053 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "inventory_list_all", 2024-03-23T23:13:42.053 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.053 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.053 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.053 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.053 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.053 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.053 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.053 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-23T23:13:42.053 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.053 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.053 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.053 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.053 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.053 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.054 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.054 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.054 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.054 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.054 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.054 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.054 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.054 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.054 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.054 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.054 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.054 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.054 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.054 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.055 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.055 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.055 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.055 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.055 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.055 INFO:teuthology.orchestra.run.smithi012.stdout: "log_refresh_metadata": { 2024-03-23T23:13:42.055 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_refresh_metadata", 2024-03-23T23:13:42.055 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.055 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.055 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.055 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.055 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.055 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.055 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.055 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-23T23:13:42.056 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.056 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.056 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.056 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.056 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.056 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.056 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.056 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.056 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.056 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.056 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.056 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.056 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.056 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-23T23:13:42.056 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.056 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.056 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.057 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.057 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.057 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.057 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.057 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.057 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.057 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.057 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.057 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.057 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.057 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.057 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.057 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.057 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.057 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.057 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.057 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.058 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.058 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.058 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.058 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.058 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.058 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.058 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.058 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.058 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.058 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.058 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.058 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.058 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.058 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.058 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.058 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.059 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.059 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.059 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.059 INFO:teuthology.orchestra.run.smithi012.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-23T23:13:42.059 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-23T23:13:42.059 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.059 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.059 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.059 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.059 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.059 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.059 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.059 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-23T23:13:42.059 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.059 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.060 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.060 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.060 INFO:teuthology.orchestra.run.smithi012.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-23T23:13:42.060 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-23T23:13:42.060 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.060 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.060 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.060 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "*", 2024-03-23T23:13:42.060 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.060 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.060 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.060 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-23T23:13:42.060 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.060 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.060 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.060 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.060 INFO:teuthology.orchestra.run.smithi012.stdout: "max_count_per_host": { 2024-03-23T23:13:42.061 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_count_per_host", 2024-03-23T23:13:42.061 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.061 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.061 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.061 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-23T23:13:42.061 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.061 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.061 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.061 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "max number of daemons per service per host", 2024-03-23T23:13:42.061 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.061 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.061 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.061 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.061 INFO:teuthology.orchestra.run.smithi012.stdout: "max_osd_draining_count": { 2024-03-23T23:13:42.061 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_osd_draining_count", 2024-03-23T23:13:42.061 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.062 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.062 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.062 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-23T23:13:42.062 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.062 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.062 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.062 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-23T23:13:42.062 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.062 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.062 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.062 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.062 INFO:teuthology.orchestra.run.smithi012.stdout: "migration_current": { 2024-03-23T23:13:42.062 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "migration_current", 2024-03-23T23:13:42.062 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.063 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.063 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.063 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.063 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.063 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.063 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.063 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "internal - do not modify", 2024-03-23T23:13:42.063 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.063 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.063 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.063 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.063 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": { 2024-03-23T23:13:42.063 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mode", 2024-03-23T23:13:42.063 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.063 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.063 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.064 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "root", 2024-03-23T23:13:42.064 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.064 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.064 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.064 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm-package", 2024-03-23T23:13:42.064 INFO:teuthology.orchestra.run.smithi012.stdout: "root" 2024-03-23T23:13:42.064 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.064 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "mode for remote execution of cephadm", 2024-03-23T23:13:42.064 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.064 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.064 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.064 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.064 INFO:teuthology.orchestra.run.smithi012.stdout: "oob_default_addr": { 2024-03-23T23:13:42.064 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "oob_default_addr", 2024-03-23T23:13:42.064 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.065 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.065 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.065 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "169.254.1.1", 2024-03-23T23:13:42.065 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.065 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.065 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.065 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-23T23:13:42.065 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.065 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.065 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.065 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.065 INFO:teuthology.orchestra.run.smithi012.stdout: "prometheus_alerts_path": { 2024-03-23T23:13:42.065 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prometheus_alerts_path", 2024-03-23T23:13:42.065 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.065 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.066 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.066 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-23T23:13:42.066 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.066 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.066 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.066 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-23T23:13:42.066 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.066 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.066 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.066 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.066 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_insecure": { 2024-03-23T23:13:42.066 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_insecure", 2024-03-23T23:13:42.066 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.066 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.067 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.067 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.067 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.067 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.067 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.067 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-23T23:13:42.067 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.067 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.067 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.067 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.067 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_password": { 2024-03-23T23:13:42.067 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_password", 2024-03-23T23:13:42.067 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.067 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.067 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.067 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.068 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.068 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.068 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.068 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-23T23:13:42.068 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.068 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.068 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.068 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.068 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_url": { 2024-03-23T23:13:42.068 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_url", 2024-03-23T23:13:42.068 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.068 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.068 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.068 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.068 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.069 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.069 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.069 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-23T23:13:42.069 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.069 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.069 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.069 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.069 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_username": { 2024-03-23T23:13:42.069 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_username", 2024-03-23T23:13:42.069 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.069 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.069 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.069 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.069 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.069 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.069 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.070 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-23T23:13:42.070 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.070 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.070 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.070 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.070 INFO:teuthology.orchestra.run.smithi012.stdout: "secure_monitoring_stack": { 2024-03-23T23:13:42.070 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "secure_monitoring_stack", 2024-03-23T23:13:42.070 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.070 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.070 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.070 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.070 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.070 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.070 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.071 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-23T23:13:42.071 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.071 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.071 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.071 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.071 INFO:teuthology.orchestra.run.smithi012.stdout: "service_discovery_port": { 2024-03-23T23:13:42.071 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "service_discovery_port", 2024-03-23T23:13:42.071 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.071 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.071 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.071 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8765", 2024-03-23T23:13:42.071 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.071 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.071 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.072 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "cephadm service discovery port", 2024-03-23T23:13:42.072 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.072 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.072 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.072 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.072 INFO:teuthology.orchestra.run.smithi012.stdout: "ssh_config_file": { 2024-03-23T23:13:42.072 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssh_config_file", 2024-03-23T23:13:42.072 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.072 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.072 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.072 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.072 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.072 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.072 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.072 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-23T23:13:42.072 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.073 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.073 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.073 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.073 INFO:teuthology.orchestra.run.smithi012.stdout: "use_agent": { 2024-03-23T23:13:42.073 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "use_agent", 2024-03-23T23:13:42.073 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.073 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.073 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.073 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.073 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.073 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.073 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.073 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-23T23:13:42.073 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.073 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.074 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.074 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.074 INFO:teuthology.orchestra.run.smithi012.stdout: "use_repo_digest": { 2024-03-23T23:13:42.074 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "use_repo_digest", 2024-03-23T23:13:42.074 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.074 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.074 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.074 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.074 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.074 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.074 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.074 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-23T23:13:42.074 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.074 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.074 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.075 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.075 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_failed_host_check": { 2024-03-23T23:13:42.075 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_failed_host_check", 2024-03-23T23:13:42.075 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.075 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.075 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.075 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.075 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.075 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.075 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.075 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if the host check fails", 2024-03-23T23:13:42.075 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.075 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.075 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.075 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.075 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_stray_daemons": { 2024-03-23T23:13:42.075 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_stray_daemons", 2024-03-23T23:13:42.076 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.076 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.076 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.076 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.076 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.076 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.076 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.076 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-23T23:13:42.076 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.076 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.076 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.076 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.076 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_stray_hosts": { 2024-03-23T23:13:42.076 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_stray_hosts", 2024-03-23T23:13:42.076 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.076 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.077 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.077 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.077 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.077 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.077 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.077 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-23T23:13:42.077 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.077 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.077 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.077 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.077 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.077 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.077 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.077 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crash", 2024-03-23T23:13:42.077 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.077 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.077 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.078 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.078 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.078 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.078 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.078 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.078 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.078 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.078 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.078 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.078 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.078 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.078 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.078 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.078 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.079 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.079 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.079 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.079 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.079 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.079 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.079 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.079 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.079 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.079 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.079 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.079 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.079 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.079 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.080 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.080 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.080 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.080 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.080 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.080 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.080 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.080 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.080 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.080 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.080 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.080 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.080 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.080 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.080 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.080 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.081 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.081 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.081 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.081 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.081 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.081 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.081 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.081 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.081 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.081 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.081 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.081 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.081 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.081 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.081 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.082 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.082 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.082 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.082 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.082 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.082 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.082 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.082 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.082 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.082 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.082 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.082 INFO:teuthology.orchestra.run.smithi012.stdout: "retain_interval": { 2024-03-23T23:13:42.082 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "retain_interval", 2024-03-23T23:13:42.082 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:42.082 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.083 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.083 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "31536000", 2024-03-23T23:13:42.083 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.083 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.083 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.083 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-23T23:13:42.083 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.083 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.083 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.083 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.083 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_recent_interval": { 2024-03-23T23:13:42.083 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_recent_interval", 2024-03-23T23:13:42.083 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:42.083 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.083 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.083 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1209600", 2024-03-23T23:13:42.083 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.084 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.084 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.084 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-23T23:13:42.084 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.084 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.084 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.084 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.084 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.084 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.084 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.084 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "dashboard", 2024-03-23T23:13:42.084 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.084 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.084 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.084 INFO:teuthology.orchestra.run.smithi012.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-23T23:13:42.084 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-23T23:13:42.084 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.085 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.085 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.085 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-23T23:13:42.085 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.085 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.085 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.085 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.085 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.085 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.085 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.085 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.085 INFO:teuthology.orchestra.run.smithi012.stdout: "ALERTMANAGER_API_HOST": { 2024-03-23T23:13:42.085 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-23T23:13:42.085 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.085 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.085 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.085 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.086 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.086 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.086 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.086 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.086 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.086 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.086 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.086 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.086 INFO:teuthology.orchestra.run.smithi012.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-23T23:13:42.086 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-23T23:13:42.086 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.086 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.086 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.086 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.086 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.087 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.087 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.087 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.087 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.087 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.087 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.087 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.087 INFO:teuthology.orchestra.run.smithi012.stdout: "AUDIT_API_ENABLED": { 2024-03-23T23:13:42.087 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "AUDIT_API_ENABLED", 2024-03-23T23:13:42.087 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.087 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.087 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.087 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.087 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.087 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.087 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.088 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.088 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.088 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.088 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.088 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.088 INFO:teuthology.orchestra.run.smithi012.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-23T23:13:42.088 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-23T23:13:42.088 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.088 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.088 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.088 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.088 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.089 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.089 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.089 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.089 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.089 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.089 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.089 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.089 INFO:teuthology.orchestra.run.smithi012.stdout: "ENABLE_BROWSABLE_API": { 2024-03-23T23:13:42.089 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-23T23:13:42.089 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.089 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.089 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.089 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.090 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.090 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.090 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.090 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.090 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.090 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.090 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.090 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.090 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-23T23:13:42.090 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-23T23:13:42.090 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.090 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.090 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.091 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.091 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.091 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.091 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.091 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.091 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.091 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.091 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.091 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.091 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-23T23:13:42.091 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-23T23:13:42.091 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.091 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.091 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.091 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.091 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.091 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.092 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.092 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.092 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.092 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.092 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.092 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.092 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-23T23:13:42.092 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-23T23:13:42.092 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.092 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.092 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.092 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.092 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.092 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.092 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.093 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.093 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.093 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.093 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.093 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.093 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-23T23:13:42.093 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-23T23:13:42.093 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.093 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.093 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.093 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.093 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.093 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.094 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.094 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.094 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.094 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.094 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.094 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.094 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-23T23:13:42.094 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-23T23:13:42.094 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.094 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.095 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.095 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.095 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.095 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.095 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.095 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.095 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.095 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.095 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.095 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.095 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-23T23:13:42.095 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-23T23:13:42.095 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.095 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.095 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.096 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.096 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.096 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.096 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.096 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.096 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.096 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.096 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.096 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.096 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-23T23:13:42.096 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-23T23:13:42.096 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.096 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.096 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.097 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.097 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.097 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.097 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.097 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.097 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.097 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.097 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.097 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.097 INFO:teuthology.orchestra.run.smithi012.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-23T23:13:42.097 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-23T23:13:42.097 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.097 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.097 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.098 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.098 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.098 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.098 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.098 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.098 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.098 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.098 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.098 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.098 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_PASSWORD": { 2024-03-23T23:13:42.098 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-23T23:13:42.098 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.098 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.098 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.098 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-03-23T23:13:42.098 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.098 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.099 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.099 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.099 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.099 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.099 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.099 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.099 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-23T23:13:42.099 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-23T23:13:42.099 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.099 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.099 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.099 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.099 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.099 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.099 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.099 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.100 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.100 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.100 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.100 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.100 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_URL": { 2024-03-23T23:13:42.100 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_URL", 2024-03-23T23:13:42.100 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.100 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.100 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.100 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.100 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.100 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.100 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.100 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.100 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.100 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.100 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.101 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.101 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_USERNAME": { 2024-03-23T23:13:42.101 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-23T23:13:42.101 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.101 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.101 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.101 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-03-23T23:13:42.101 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.101 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.101 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.101 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.101 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.101 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.101 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.101 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.102 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-23T23:13:42.102 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-23T23:13:42.102 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.102 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.102 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.102 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.102 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.102 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.102 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.102 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.102 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.102 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.102 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.102 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.102 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-23T23:13:42.102 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-23T23:13:42.102 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.103 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.103 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.103 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.103 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.103 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.103 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.103 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.103 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.103 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.103 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.103 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.103 INFO:teuthology.orchestra.run.smithi012.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-23T23:13:42.103 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-23T23:13:42.103 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.103 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.104 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.104 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.104 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.104 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.104 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.104 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.104 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.104 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.104 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.104 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.104 INFO:teuthology.orchestra.run.smithi012.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-23T23:13:42.104 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-23T23:13:42.104 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.104 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.104 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.104 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.105 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.105 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.105 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.105 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.105 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.105 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.105 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.105 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.105 INFO:teuthology.orchestra.run.smithi012.stdout: "MULTICLUSTER_CONFIG": { 2024-03-23T23:13:42.105 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-23T23:13:42.105 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.105 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.105 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.105 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "{}", 2024-03-23T23:13:42.105 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.106 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.106 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.106 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.106 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.106 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.106 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.106 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.106 INFO:teuthology.orchestra.run.smithi012.stdout: "PROMETHEUS_API_HOST": { 2024-03-23T23:13:42.106 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-23T23:13:42.106 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.106 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.106 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.106 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.106 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.106 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.106 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.106 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.107 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.107 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.107 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.107 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.107 INFO:teuthology.orchestra.run.smithi012.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-23T23:13:42.107 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-23T23:13:42.107 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.107 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.107 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.107 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.107 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.107 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.107 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.107 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.107 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.107 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.108 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.108 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.108 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-23T23:13:42.108 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-23T23:13:42.108 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.108 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.108 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.108 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.108 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.108 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.108 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.108 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.108 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.108 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.108 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.108 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.109 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-23T23:13:42.109 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-23T23:13:42.109 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.109 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.109 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.109 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.109 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.109 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.109 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.109 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.109 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.109 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.109 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.109 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.109 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-23T23:13:42.109 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-23T23:13:42.110 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.110 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.110 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.110 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.110 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.110 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.110 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.110 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.110 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.110 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.110 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.110 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.110 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-23T23:13:42.110 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-23T23:13:42.110 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.111 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.111 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.111 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.111 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.111 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.111 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.111 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.111 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.111 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.111 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.111 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.111 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-23T23:13:42.111 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-23T23:13:42.111 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.111 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.111 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.112 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.112 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.112 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.112 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.112 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.112 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.112 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.112 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.112 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.112 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-23T23:13:42.112 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-23T23:13:42.112 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.112 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.112 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.112 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.112 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.113 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.113 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.113 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.113 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.113 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.113 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.113 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.113 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-23T23:13:42.113 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-23T23:13:42.113 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.113 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.113 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.113 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.113 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.113 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.114 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.114 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.114 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.114 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.114 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.114 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.114 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_ENABLED": { 2024-03-23T23:13:42.114 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-23T23:13:42.114 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.114 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.114 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.114 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.114 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.114 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.115 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.115 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.115 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.115 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.115 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.115 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.115 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-23T23:13:42.115 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-23T23:13:42.115 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.115 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.115 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.115 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-23T23:13:42.115 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.115 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.115 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.115 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.115 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.116 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.116 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.116 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.116 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-23T23:13:42.116 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-23T23:13:42.116 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.116 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.116 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.116 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-23T23:13:42.116 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.116 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.116 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.116 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.116 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.116 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.116 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.116 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.117 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-23T23:13:42.117 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-23T23:13:42.117 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.117 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.117 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.117 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8", 2024-03-23T23:13:42.117 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.117 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.117 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.117 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.117 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.117 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.117 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.117 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.117 INFO:teuthology.orchestra.run.smithi012.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-23T23:13:42.118 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-23T23:13:42.118 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.118 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.118 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.118 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "45", 2024-03-23T23:13:42.118 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.118 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.118 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.118 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.118 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.118 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.118 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.118 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.118 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_ACCESS_KEY": { 2024-03-23T23:13:42.118 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-23T23:13:42.118 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.119 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.119 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.119 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.119 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.119 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.119 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.119 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.119 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.119 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.119 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.119 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.119 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-23T23:13:42.119 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-23T23:13:42.119 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.119 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.119 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.120 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-03-23T23:13:42.120 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.120 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.120 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.120 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.120 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.120 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.120 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.120 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.120 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_SECRET_KEY": { 2024-03-23T23:13:42.120 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-23T23:13:42.120 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.120 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.120 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.120 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.120 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.120 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.121 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.121 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.121 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.121 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.121 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.121 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.121 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_SSL_VERIFY": { 2024-03-23T23:13:42.121 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-23T23:13:42.121 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.121 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.121 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.121 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.121 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.121 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.122 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.122 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.122 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.122 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.122 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.122 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.122 INFO:teuthology.orchestra.run.smithi012.stdout: "UNSAFE_TLS_v1_2": { 2024-03-23T23:13:42.122 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-23T23:13:42.122 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.122 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.122 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.122 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.122 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.122 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.122 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.123 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.123 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.123 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.123 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.123 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.123 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-23T23:13:42.123 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-23T23:13:42.123 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.123 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.123 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.123 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-23T23:13:42.123 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.123 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.123 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.123 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.123 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.123 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.124 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.124 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.124 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-23T23:13:42.124 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-23T23:13:42.124 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.124 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.124 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.124 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-23T23:13:42.124 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.124 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.124 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.124 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.124 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.124 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.124 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.124 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.124 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-23T23:13:42.125 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-23T23:13:42.125 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.125 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.125 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.125 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-03-23T23:13:42.125 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.125 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.125 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.125 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.125 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.125 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.125 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.125 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.125 INFO:teuthology.orchestra.run.smithi012.stdout: "cross_origin_url": { 2024-03-23T23:13:42.125 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cross_origin_url", 2024-03-23T23:13:42.125 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.126 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.126 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.126 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.126 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.126 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.126 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.126 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.126 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.126 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.126 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.126 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.126 INFO:teuthology.orchestra.run.smithi012.stdout: "crt_file": { 2024-03-23T23:13:42.126 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crt_file", 2024-03-23T23:13:42.126 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.126 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.126 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.126 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.127 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.127 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.127 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.127 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.127 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.127 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.127 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.127 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.127 INFO:teuthology.orchestra.run.smithi012.stdout: "debug": { 2024-03-23T23:13:42.127 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "debug", 2024-03-23T23:13:42.127 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.127 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.127 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.127 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.127 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.127 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.128 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.128 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable/disable debug options", 2024-03-23T23:13:42.128 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.128 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.128 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.128 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.128 INFO:teuthology.orchestra.run.smithi012.stdout: "jwt_token_ttl": { 2024-03-23T23:13:42.128 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "jwt_token_ttl", 2024-03-23T23:13:42.128 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.128 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.128 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.128 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "28800", 2024-03-23T23:13:42.128 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.128 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.128 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.128 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.129 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.129 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.129 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.129 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.129 INFO:teuthology.orchestra.run.smithi012.stdout: "key_file": { 2024-03-23T23:13:42.129 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "key_file", 2024-03-23T23:13:42.129 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.129 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.129 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.129 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.129 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.129 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.129 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.129 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.129 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.129 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.130 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.130 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.130 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.130 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.130 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.130 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.130 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.130 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.130 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.130 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.130 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.130 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.130 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.130 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.130 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.130 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.131 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.131 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.131 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.131 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.131 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.131 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.131 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.131 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.131 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.131 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.131 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.131 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.131 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.131 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.131 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.131 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.132 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.132 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.132 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.132 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.132 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.132 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.132 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.132 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.132 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.132 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.132 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.132 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.132 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.132 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.132 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.132 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.133 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.133 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.133 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.133 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.133 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.133 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.133 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.133 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.133 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.133 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.133 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.133 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.133 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.133 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.133 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.133 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.134 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.134 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.134 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.134 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.134 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.134 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.134 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.134 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.134 INFO:teuthology.orchestra.run.smithi012.stdout: "motd": { 2024-03-23T23:13:42.134 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "motd", 2024-03-23T23:13:42.134 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.134 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.134 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.134 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.134 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.135 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.135 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.135 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "The message of the day", 2024-03-23T23:13:42.135 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.135 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.135 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.135 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.135 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect_resolve_ip_addr": { 2024-03-23T23:13:42.135 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "redirect_resolve_ip_addr", 2024-03-23T23:13:42.135 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.135 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.135 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.135 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.135 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.135 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.136 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.136 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.136 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.136 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.136 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.136 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.136 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-03-23T23:13:42.136 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-03-23T23:13:42.136 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.136 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.136 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.136 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "::", 2024-03-23T23:13:42.136 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.136 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.136 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.136 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.136 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.137 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.137 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.137 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.137 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-03-23T23:13:42.137 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-03-23T23:13:42.137 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.137 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.137 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.137 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8080", 2024-03-23T23:13:42.137 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.137 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.137 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.137 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.137 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.137 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.138 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.138 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.138 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl": { 2024-03-23T23:13:42.138 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl", 2024-03-23T23:13:42.138 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.138 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.138 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.138 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.138 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.138 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.138 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.138 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.138 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.138 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.139 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.139 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.139 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl_server_port": { 2024-03-23T23:13:42.139 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl_server_port", 2024-03-23T23:13:42.139 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.139 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.139 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.139 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8443", 2024-03-23T23:13:42.139 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.139 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.139 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.139 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.139 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.139 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.139 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.139 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.139 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_behaviour": { 2024-03-23T23:13:42.140 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_behaviour", 2024-03-23T23:13:42.140 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.140 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.140 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.140 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "redirect", 2024-03-23T23:13:42.140 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.140 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.140 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.140 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.140 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect" 2024-03-23T23:13:42.140 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.140 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.140 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.140 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.140 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.141 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.141 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_error_status_code": { 2024-03-23T23:13:42.141 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_error_status_code", 2024-03-23T23:13:42.141 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.141 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.141 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.141 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "500", 2024-03-23T23:13:42.141 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "400", 2024-03-23T23:13:42.141 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "599", 2024-03-23T23:13:42.141 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.141 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.141 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.141 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.141 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.141 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.141 INFO:teuthology.orchestra.run.smithi012.stdout: "url_prefix": { 2024-03-23T23:13:42.141 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "url_prefix", 2024-03-23T23:13:42.142 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.142 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.142 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.142 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.142 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.142 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.142 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.142 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.142 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.142 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.142 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.142 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.142 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.142 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.142 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.143 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "devicehealth", 2024-03-23T23:13:42.143 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.143 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.143 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.143 INFO:teuthology.orchestra.run.smithi012.stdout: "enable_monitoring": { 2024-03-23T23:13:42.143 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enable_monitoring", 2024-03-23T23:13:42.143 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.143 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.143 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.143 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.143 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.143 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.143 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.143 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "monitor device health metrics", 2024-03-23T23:13:42.143 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.144 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.144 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.144 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.144 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.144 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.144 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.144 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.144 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.144 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.144 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.144 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.144 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.144 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.144 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.144 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.145 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.145 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.145 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.145 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.145 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.145 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.145 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.145 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.145 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.145 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.145 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.145 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.145 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.145 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.145 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.146 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.146 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.146 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.146 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.146 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.146 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.146 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.146 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.146 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.146 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.146 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.146 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.146 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.146 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.147 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.147 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.147 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.147 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.147 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.147 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.147 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.147 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.147 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.147 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.147 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.147 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.147 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.147 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.147 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.148 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.148 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.148 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.148 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.148 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.148 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.148 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.148 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.148 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.148 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.148 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.148 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.148 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.148 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.149 INFO:teuthology.orchestra.run.smithi012.stdout: "mark_out_threshold": { 2024-03-23T23:13:42.149 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mark_out_threshold", 2024-03-23T23:13:42.149 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:42.149 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.149 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.149 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "2419200", 2024-03-23T23:13:42.149 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.149 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.149 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.149 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-23T23:13:42.149 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.149 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.149 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.149 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.149 INFO:teuthology.orchestra.run.smithi012.stdout: "pool_name": { 2024-03-23T23:13:42.150 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pool_name", 2024-03-23T23:13:42.150 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.150 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.150 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.150 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "device_health_metrics", 2024-03-23T23:13:42.150 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.150 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.150 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.150 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-23T23:13:42.150 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.150 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.150 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.150 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.150 INFO:teuthology.orchestra.run.smithi012.stdout: "retention_period": { 2024-03-23T23:13:42.150 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "retention_period", 2024-03-23T23:13:42.151 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:42.151 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.151 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.151 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15552000", 2024-03-23T23:13:42.151 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.151 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.151 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.151 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long to retain device health metrics", 2024-03-23T23:13:42.151 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.151 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.151 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.151 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.151 INFO:teuthology.orchestra.run.smithi012.stdout: "scrape_frequency": { 2024-03-23T23:13:42.151 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "scrape_frequency", 2024-03-23T23:13:42.151 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:42.151 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.152 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.152 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "86400", 2024-03-23T23:13:42.152 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.152 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.152 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.152 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-23T23:13:42.152 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.152 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.152 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.152 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.152 INFO:teuthology.orchestra.run.smithi012.stdout: "self_heal": { 2024-03-23T23:13:42.152 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "self_heal", 2024-03-23T23:13:42.152 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.152 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.152 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.153 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.153 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.153 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.153 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.153 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-23T23:13:42.153 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.153 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.153 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.153 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.153 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-23T23:13:42.153 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-23T23:13:42.153 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:42.153 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.153 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.153 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-23T23:13:42.153 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.154 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.154 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.154 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to wake up and check device health", 2024-03-23T23:13:42.154 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.154 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.154 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.154 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.154 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_threshold": { 2024-03-23T23:13:42.154 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_threshold", 2024-03-23T23:13:42.154 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:42.154 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.154 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.154 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "7257600", 2024-03-23T23:13:42.154 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.154 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.155 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.155 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-23T23:13:42.155 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.155 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.155 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.155 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.155 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.155 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.155 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.155 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "diskprediction_local", 2024-03-23T23:13:42.155 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.155 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.155 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.156 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.156 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.156 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.156 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.156 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.156 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.156 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.156 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.156 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.156 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.156 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.156 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.156 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.156 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.156 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.157 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.157 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.157 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.157 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.157 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.157 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.157 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.157 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.157 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.157 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.157 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.157 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.157 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.157 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.158 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.158 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.158 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.158 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.158 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.158 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.158 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.158 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.158 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.158 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.158 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.158 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.158 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.158 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.158 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.159 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.159 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.159 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.159 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.159 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.159 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.159 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.159 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.159 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.159 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.159 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.159 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.159 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.159 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.159 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.159 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.160 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.160 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.160 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.160 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.160 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.160 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.160 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.160 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.160 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.160 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.160 INFO:teuthology.orchestra.run.smithi012.stdout: "predict_interval": { 2024-03-23T23:13:42.160 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "predict_interval", 2024-03-23T23:13:42.160 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.160 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.160 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.160 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "86400", 2024-03-23T23:13:42.161 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.161 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.161 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.161 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.161 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.161 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.161 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.161 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.161 INFO:teuthology.orchestra.run.smithi012.stdout: "predictor_model": { 2024-03-23T23:13:42.161 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "predictor_model", 2024-03-23T23:13:42.161 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.161 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.161 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.161 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "prophetstor", 2024-03-23T23:13:42.162 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.162 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.162 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.162 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.162 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.162 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.162 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.162 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.162 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-23T23:13:42.162 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-23T23:13:42.162 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.162 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.163 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.163 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-23T23:13:42.163 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.163 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.163 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.163 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.163 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.163 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.163 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.163 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.163 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.163 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.163 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.163 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "influx", 2024-03-23T23:13:42.163 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": false, 2024-03-23T23:13:42.164 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "influxdb python module not found", 2024-03-23T23:13:42.164 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.164 INFO:teuthology.orchestra.run.smithi012.stdout: "batch_size": { 2024-03-23T23:13:42.164 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "batch_size", 2024-03-23T23:13:42.164 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.164 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.164 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.164 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5000", 2024-03-23T23:13:42.164 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.164 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.164 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.164 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-23T23:13:42.164 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.164 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.164 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.164 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.165 INFO:teuthology.orchestra.run.smithi012.stdout: "database": { 2024-03-23T23:13:42.165 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "database", 2024-03-23T23:13:42.165 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.165 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.165 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.165 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "ceph", 2024-03-23T23:13:42.165 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.165 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.165 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.165 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-23T23:13:42.165 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.165 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.165 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.165 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.165 INFO:teuthology.orchestra.run.smithi012.stdout: "hostname": { 2024-03-23T23:13:42.165 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "hostname", 2024-03-23T23:13:42.166 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.166 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.166 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.166 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.166 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.166 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.166 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.166 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB server hostname", 2024-03-23T23:13:42.166 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.166 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.166 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.166 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.166 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-23T23:13:42.166 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-23T23:13:42.167 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:42.167 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.167 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.167 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "30", 2024-03-23T23:13:42.167 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "5", 2024-03-23T23:13:42.167 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.167 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.167 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-23T23:13:42.167 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.167 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.167 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.167 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.167 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.167 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.167 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.168 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.168 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.168 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.168 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.168 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.168 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.168 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.168 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.168 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.168 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.168 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.168 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.168 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.168 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.168 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.168 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.168 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.169 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.169 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.169 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.169 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.169 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.169 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.169 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.169 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.169 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.169 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.169 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.169 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.169 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.169 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.169 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.169 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.169 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.170 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.170 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.170 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.170 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.170 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.170 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.170 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.170 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.170 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.170 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.170 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.170 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.170 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.170 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.170 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.171 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.171 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.171 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.171 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.171 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.171 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.171 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.171 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.171 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.171 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.171 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.171 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.171 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.171 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.171 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.171 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.171 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.172 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.172 INFO:teuthology.orchestra.run.smithi012.stdout: "password": { 2024-03-23T23:13:42.172 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "password", 2024-03-23T23:13:42.172 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.172 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.172 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.172 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.172 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.172 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.172 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.172 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "password of InfluxDB server user", 2024-03-23T23:13:42.172 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.172 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.172 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.172 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.172 INFO:teuthology.orchestra.run.smithi012.stdout: "port": { 2024-03-23T23:13:42.172 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "port", 2024-03-23T23:13:42.173 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.173 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.173 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.173 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8086", 2024-03-23T23:13:42.173 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.173 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.173 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.173 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB server port", 2024-03-23T23:13:42.173 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.173 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.173 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.173 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.173 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl": { 2024-03-23T23:13:42.173 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl", 2024-03-23T23:13:42.173 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.174 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.174 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.174 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "false", 2024-03-23T23:13:42.174 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.174 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.174 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.174 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-23T23:13:42.174 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.174 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.174 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.174 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.174 INFO:teuthology.orchestra.run.smithi012.stdout: "threads": { 2024-03-23T23:13:42.174 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "threads", 2024-03-23T23:13:42.174 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.174 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.175 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.175 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-03-23T23:13:42.175 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-03-23T23:13:42.175 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "32", 2024-03-23T23:13:42.175 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.175 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-23T23:13:42.175 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.175 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.175 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.175 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.175 INFO:teuthology.orchestra.run.smithi012.stdout: "username": { 2024-03-23T23:13:42.175 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "username", 2024-03-23T23:13:42.175 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.175 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.175 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.175 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.176 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.176 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.176 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.176 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "username of InfluxDB server user", 2024-03-23T23:13:42.176 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.176 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.176 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.176 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.176 INFO:teuthology.orchestra.run.smithi012.stdout: "verify_ssl": { 2024-03-23T23:13:42.176 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "verify_ssl", 2024-03-23T23:13:42.176 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.176 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.176 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.176 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "true", 2024-03-23T23:13:42.176 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.176 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.176 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.177 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-23T23:13:42.177 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.177 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.177 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.177 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.177 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.177 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.177 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.177 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "insights", 2024-03-23T23:13:42.177 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.177 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.177 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.177 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.177 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.177 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.177 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.177 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.178 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.178 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.178 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.178 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.178 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.178 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.178 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.178 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.178 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.178 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.178 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.178 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.178 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.178 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.178 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.179 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.179 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.179 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.179 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.179 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.179 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.179 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.179 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.179 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.179 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.179 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.179 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.179 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.179 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.179 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.179 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.179 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.180 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.180 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.180 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.180 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.180 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.180 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.180 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.180 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.180 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.180 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.180 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.180 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.180 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.180 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.180 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.182 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.182 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.182 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.182 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.182 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.182 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.182 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "iostat", 2024-03-23T23:13:42.182 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.182 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.182 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.182 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.182 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.182 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.183 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.183 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.183 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.183 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.183 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.183 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.183 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.183 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.183 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.183 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.183 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.183 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.183 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.183 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.184 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.184 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.184 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.184 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.184 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.184 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.184 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.184 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.184 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.184 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.184 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.184 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.184 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.184 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.185 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.185 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.185 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.185 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.185 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.185 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.185 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.185 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.185 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.185 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.185 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.185 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.185 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.185 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.185 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.185 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.185 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.186 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.186 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.186 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.186 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.186 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.186 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.186 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.186 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.186 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.186 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.186 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.186 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.186 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.186 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.186 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.187 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.187 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.187 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.187 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.187 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.187 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.187 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.187 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.187 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.187 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.187 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "k8sevents", 2024-03-23T23:13:42.187 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.187 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.187 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.187 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_event_retention_days": { 2024-03-23T23:13:42.188 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ceph_event_retention_days", 2024-03-23T23:13:42.188 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.188 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.188 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.188 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "7", 2024-03-23T23:13:42.188 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.188 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.188 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.188 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-23T23:13:42.188 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.188 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.188 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.188 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.188 INFO:teuthology.orchestra.run.smithi012.stdout: "config_check_secs": { 2024-03-23T23:13:42.188 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_check_secs", 2024-03-23T23:13:42.188 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.188 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.189 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.189 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-23T23:13:42.189 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "10", 2024-03-23T23:13:42.189 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.189 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.189 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-23T23:13:42.189 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.189 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.189 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.189 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.189 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.189 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.189 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.189 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.189 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.189 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.190 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.190 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.190 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.190 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.190 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.190 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.190 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.190 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.190 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.190 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.190 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.190 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.190 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.190 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.191 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.191 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.191 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.191 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.191 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.191 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.191 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.191 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.191 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.191 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.191 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.191 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.191 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.191 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.191 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.192 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.192 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.192 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.192 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.192 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.192 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.192 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.192 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.192 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.192 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.192 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.192 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.192 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.192 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.192 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.192 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.194 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.194 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.194 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.194 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.194 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.194 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.194 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.194 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "localpool", 2024-03-23T23:13:42.194 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.194 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.194 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.194 INFO:teuthology.orchestra.run.smithi012.stdout: "failure_domain": { 2024-03-23T23:13:42.194 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "failure_domain", 2024-03-23T23:13:42.194 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "host", 2024-03-23T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "failure domain for any created local pool", 2024-03-23T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-23T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stdout: "min_size": { 2024-03-23T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "min_size", 2024-03-23T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default min_size for any created local pool", 2024-03-23T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-23T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stdout: "num_rep": { 2024-03-23T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "num_rep", 2024-03-23T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3", 2024-03-23T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default replica count for any created local pool", 2024-03-23T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_num": { 2024-03-23T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pg_num", 2024-03-23T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "128", 2024-03-23T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default pg_num for any created local pool", 2024-03-23T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stdout: "prefix": { 2024-03-23T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prefix", 2024-03-23T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "name prefix for any created local pool", 2024-03-23T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stdout: "subtree": { 2024-03-23T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "subtree", 2024-03-23T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "rack", 2024-03-23T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-23T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-23T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mds_autoscaler", 2024-03-23T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.210 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.210 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.210 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.210 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.210 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mirroring", 2024-03-23T23:13:42.210 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.210 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.210 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.210 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.210 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.210 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.210 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.210 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.212 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.212 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.212 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.212 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.212 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.212 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.212 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.212 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.212 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.212 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.212 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.212 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.212 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.213 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.213 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.213 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.213 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.213 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.213 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.213 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.213 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.213 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.213 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.213 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.213 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.213 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.214 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.214 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.214 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.214 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.214 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.214 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.214 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.214 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.214 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.214 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.214 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.215 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.215 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.215 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.215 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.215 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.215 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.215 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.215 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.215 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.215 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.215 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.215 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.215 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.215 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.216 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.216 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "nfs", 2024-03-23T23:13:42.216 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.216 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.216 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.216 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.216 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.216 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.216 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.216 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.216 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.216 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.216 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.216 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.217 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.217 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.217 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.217 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.217 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.217 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.217 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.217 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.217 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.217 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.217 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.217 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.217 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.217 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.218 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.218 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.218 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.218 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.218 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.218 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.218 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.218 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.218 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.218 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.218 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.218 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.218 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.218 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.219 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.219 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.219 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.219 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.219 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.219 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.219 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.219 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.219 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.219 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.219 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.219 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.219 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.219 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.219 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.220 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.220 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.220 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.220 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.220 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.220 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.220 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.220 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.220 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.220 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.220 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.220 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.220 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.220 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.221 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.221 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.221 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.221 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.221 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.221 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.221 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.221 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "orchestrator", 2024-03-23T23:13:42.221 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.221 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.221 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.221 INFO:teuthology.orchestra.run.smithi012.stdout: "fail_fs": { 2024-03-23T23:13:42.221 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "fail_fs", 2024-03-23T23:13:42.221 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.221 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.222 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.222 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.222 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.222 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.222 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.222 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-23T23:13:42.222 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.222 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.222 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.222 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.222 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.222 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.222 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.222 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.222 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.223 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.223 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.223 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.223 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.223 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.223 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.223 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.223 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.223 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.223 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.223 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.223 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.223 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.223 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.223 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.223 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.223 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.224 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.224 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.224 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.224 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.224 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.224 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.224 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.224 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.224 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.224 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.224 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.224 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.224 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.224 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.224 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.225 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.225 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.225 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.225 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.225 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.225 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.225 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.225 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.225 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.225 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.225 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.225 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.225 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.225 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.226 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.226 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.226 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.226 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.226 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.226 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.226 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.226 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.226 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.226 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.226 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.226 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.226 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.226 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.226 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.227 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.227 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.227 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.227 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.227 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator": { 2024-03-23T23:13:42.227 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "orchestrator", 2024-03-23T23:13:42.227 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.227 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.227 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.227 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.227 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.227 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.227 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.227 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm", 2024-03-23T23:13:42.227 INFO:teuthology.orchestra.run.smithi012.stdout: "rook", 2024-03-23T23:13:42.227 INFO:teuthology.orchestra.run.smithi012.stdout: "test_orchestrator" 2024-03-23T23:13:42.227 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.228 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Orchestrator backend", 2024-03-23T23:13:42.228 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.228 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.228 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.228 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.228 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.228 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.228 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.228 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "osd_perf_query", 2024-03-23T23:13:42.228 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.228 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.228 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.228 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.228 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.228 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.228 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.228 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.229 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.229 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.229 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.229 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.229 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.229 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.229 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.229 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.229 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.229 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.229 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.229 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.229 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.229 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.229 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.229 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.230 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.230 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.230 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.230 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.230 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.230 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.230 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.230 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.230 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.230 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.230 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.230 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.230 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.231 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.231 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.231 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.231 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.231 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.231 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.231 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.231 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.231 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.231 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.231 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.231 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.231 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.231 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.231 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.231 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.231 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.232 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.232 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.232 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.232 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.232 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.232 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.232 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.232 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.232 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.232 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.232 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.232 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.232 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.232 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.232 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.232 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.232 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.233 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.233 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.233 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.233 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.233 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.233 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "osd_support", 2024-03-23T23:13:42.233 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.233 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.233 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.233 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.233 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.233 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.233 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.233 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.233 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.234 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.234 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.234 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.234 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.234 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.234 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.234 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.234 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.234 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.234 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.234 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.234 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.234 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.234 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.234 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.234 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.235 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.235 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.235 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.235 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.235 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.235 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.235 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.235 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.235 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.235 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.235 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.235 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.235 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.235 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.235 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.235 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.236 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.236 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.236 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.236 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.236 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.236 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.236 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.236 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.236 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.236 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.236 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.236 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.236 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.236 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.236 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.236 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.237 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.237 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.237 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.237 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.237 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.237 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.237 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.237 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.237 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.237 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.237 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.237 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.237 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.237 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.237 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.237 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.238 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.238 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.238 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.238 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pg_autoscaler", 2024-03-23T23:13:42.238 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.238 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.238 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.238 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.238 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.238 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.238 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.238 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.238 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.238 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.238 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.239 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.239 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.239 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.239 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.239 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.239 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.239 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.239 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.239 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.239 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.239 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.239 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.239 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.239 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.239 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.239 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.239 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.239 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.240 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.240 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.240 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.240 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.240 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.240 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.240 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.240 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.240 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.240 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.240 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.240 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.240 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.240 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.240 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.240 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.241 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.241 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.241 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.241 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.241 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.241 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.241 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.241 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.241 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.241 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.241 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.241 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.241 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.241 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.241 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.241 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.242 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.242 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.242 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.242 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.242 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.242 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.242 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.242 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.242 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.242 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.242 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.242 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.242 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-23T23:13:42.242 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-23T23:13:42.242 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:42.243 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.243 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.243 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-23T23:13:42.243 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.243 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.243 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.243 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.243 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.243 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.243 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.243 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.243 INFO:teuthology.orchestra.run.smithi012.stdout: "threshold": { 2024-03-23T23:13:42.243 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "threshold", 2024-03-23T23:13:42.243 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-23T23:13:42.243 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.243 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.244 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3.0", 2024-03-23T23:13:42.244 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1.0", 2024-03-23T23:13:42.244 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.244 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.244 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "scaling threshold", 2024-03-23T23:13:42.244 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-23T23:13:42.245 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.245 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.245 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.245 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.245 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.245 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.245 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "progress", 2024-03-23T23:13:42.245 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.245 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.246 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.246 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_pg_recovery_event": { 2024-03-23T23:13:42.246 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_pg_recovery_event", 2024-03-23T23:13:42.246 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.246 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.246 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.246 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.246 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.246 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.246 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.246 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-23T23:13:42.246 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.246 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.246 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.246 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.246 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": { 2024-03-23T23:13:42.247 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enabled", 2024-03-23T23:13:42.247 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.247 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.247 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.247 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.247 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.247 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.247 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.247 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.247 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.247 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.247 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.247 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.247 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.247 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.248 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.248 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.248 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.248 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.248 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.248 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.248 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.248 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.248 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.248 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.248 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.248 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.248 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.248 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.248 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.249 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.249 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.249 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.249 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.249 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.249 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.249 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.249 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.249 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.249 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.249 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.249 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.249 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.249 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.249 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.250 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.250 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.250 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.250 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.250 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.250 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.250 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.250 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.250 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.250 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.250 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.250 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.250 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.250 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.251 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.251 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.251 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.251 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.251 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.251 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.251 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.251 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.251 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.251 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.251 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.251 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.251 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.251 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.251 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.252 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.252 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.252 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.252 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.252 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.252 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.252 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.252 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.252 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.252 INFO:teuthology.orchestra.run.smithi012.stdout: "max_completed_events": { 2024-03-23T23:13:42.252 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_completed_events", 2024-03-23T23:13:42.252 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.252 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.252 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.253 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "50", 2024-03-23T23:13:42.253 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.253 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.253 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.253 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "number of past completed events to remember", 2024-03-23T23:13:42.253 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.253 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.253 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.253 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.253 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-23T23:13:42.253 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-23T23:13:42.253 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:42.253 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.253 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.254 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-03-23T23:13:42.254 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.254 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.254 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.254 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long the module is going to sleep", 2024-03-23T23:13:42.254 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.254 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.254 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.254 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.254 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.254 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.254 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.254 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prometheus", 2024-03-23T23:13:42.254 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.255 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.255 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.255 INFO:teuthology.orchestra.run.smithi012.stdout: "cache": { 2024-03-23T23:13:42.255 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cache", 2024-03-23T23:13:42.255 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.255 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.255 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.255 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.255 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.255 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.255 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.255 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.255 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.255 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.255 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.255 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.256 INFO:teuthology.orchestra.run.smithi012.stdout: "exclude_perf_counters": { 2024-03-23T23:13:42.256 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "exclude_perf_counters", 2024-03-23T23:13:42.256 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.256 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.256 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.256 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.256 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.256 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.256 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.256 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-23T23:13:42.256 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-23T23:13:42.256 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.256 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.256 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.257 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.257 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.257 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.257 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.257 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.257 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.257 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.257 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.257 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.257 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.257 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.257 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.257 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.257 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.257 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.258 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.258 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.258 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.258 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.258 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.258 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.258 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.258 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.258 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.258 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.258 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.258 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.258 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.258 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.259 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.259 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.259 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.259 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.259 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.259 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.259 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.259 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.259 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.259 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.259 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.259 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.259 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.259 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.259 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.259 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.260 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.260 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.260 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.260 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.260 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.260 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.260 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.260 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.260 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.260 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.260 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.260 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.260 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.261 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.261 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.261 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.261 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.261 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.261 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.261 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.261 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.261 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.261 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.261 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.261 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_stats_pools": { 2024-03-23T23:13:42.261 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_stats_pools", 2024-03-23T23:13:42.262 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.262 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.262 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.262 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.262 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.262 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.262 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.262 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.262 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.262 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.262 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.262 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.262 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-23T23:13:42.263 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-23T23:13:42.263 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.263 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.263 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.263 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "300", 2024-03-23T23:13:42.263 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.263 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.263 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.263 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.263 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.263 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.263 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.263 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.264 INFO:teuthology.orchestra.run.smithi012.stdout: "scrape_interval": { 2024-03-23T23:13:42.264 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "scrape_interval", 2024-03-23T23:13:42.264 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-23T23:13:42.264 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.264 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.264 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15.0", 2024-03-23T23:13:42.264 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.264 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.264 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.264 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.264 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.264 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.264 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.264 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.264 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-03-23T23:13:42.265 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-03-23T23:13:42.265 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.265 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.265 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.265 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "::", 2024-03-23T23:13:42.265 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.265 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.265 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.265 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-23T23:13:42.265 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.265 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.265 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.265 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.265 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-03-23T23:13:42.266 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-03-23T23:13:42.266 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.266 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.266 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.266 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "9283", 2024-03-23T23:13:42.266 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.266 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.266 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.266 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-23T23:13:42.266 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.266 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.266 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.266 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.267 INFO:teuthology.orchestra.run.smithi012.stdout: "stale_cache_strategy": { 2024-03-23T23:13:42.267 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "stale_cache_strategy", 2024-03-23T23:13:42.267 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.267 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.267 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.267 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "log", 2024-03-23T23:13:42.267 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.267 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.267 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.267 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.267 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.267 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.267 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.268 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.268 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_behaviour": { 2024-03-23T23:13:42.268 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_behaviour", 2024-03-23T23:13:42.268 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.268 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.268 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.268 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "default", 2024-03-23T23:13:42.268 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.268 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.268 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.268 INFO:teuthology.orchestra.run.smithi012.stdout: "default", 2024-03-23T23:13:42.268 INFO:teuthology.orchestra.run.smithi012.stdout: "error" 2024-03-23T23:13:42.268 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.269 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.269 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.269 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.269 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.269 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.269 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_error_status_code": { 2024-03-23T23:13:42.269 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_error_status_code", 2024-03-23T23:13:42.269 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.269 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.269 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.269 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "500", 2024-03-23T23:13:42.269 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "400", 2024-03-23T23:13:42.269 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "599", 2024-03-23T23:13:42.269 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.269 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.270 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.270 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.270 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.270 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.270 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.270 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.270 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.270 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_support", 2024-03-23T23:13:42.270 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.270 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.270 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.270 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.270 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.271 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.271 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.271 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.271 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.271 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.271 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.271 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.271 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.271 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.271 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.271 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.271 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.271 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.271 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.271 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.271 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.272 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.272 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.272 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.272 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.272 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.272 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.272 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.272 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.272 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.272 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.272 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.272 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.272 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.273 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.273 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.273 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.273 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.273 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.273 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.273 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.273 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.273 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.273 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.273 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.273 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.273 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.274 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.274 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.274 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.274 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.274 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.274 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.274 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.274 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.274 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.274 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.274 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.274 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.274 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.274 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.274 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.275 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.275 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.275 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.275 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.275 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.275 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.275 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.275 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.275 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.275 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.275 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.275 INFO:teuthology.orchestra.run.smithi012.stdout: "max_concurrent_snap_create": { 2024-03-23T23:13:42.275 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_concurrent_snap_create", 2024-03-23T23:13:42.276 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.276 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.276 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.276 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-23T23:13:42.276 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.276 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.276 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.276 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.276 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.276 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.276 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.276 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.276 INFO:teuthology.orchestra.run.smithi012.stdout: "mirror_snapshot_schedule": { 2024-03-23T23:13:42.276 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mirror_snapshot_schedule", 2024-03-23T23:13:42.276 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.277 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.277 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.277 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.277 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.277 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.277 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.277 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.277 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.277 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.277 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.277 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.277 INFO:teuthology.orchestra.run.smithi012.stdout: "trash_purge_schedule": { 2024-03-23T23:13:42.277 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "trash_purge_schedule", 2024-03-23T23:13:42.277 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.278 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.278 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.278 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.278 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.278 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.278 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.278 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.278 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.278 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.278 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.278 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.278 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.278 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.278 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.278 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "restful", 2024-03-23T23:13:42.279 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.279 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.279 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.279 INFO:teuthology.orchestra.run.smithi012.stdout: "enable_auth": { 2024-03-23T23:13:42.279 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enable_auth", 2024-03-23T23:13:42.279 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.279 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.279 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.279 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.279 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.279 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.279 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.279 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.279 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.280 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.280 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.280 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.280 INFO:teuthology.orchestra.run.smithi012.stdout: "key_file": { 2024-03-23T23:13:42.280 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "key_file", 2024-03-23T23:13:42.280 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.280 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.280 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.280 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.280 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.280 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.280 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.280 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.280 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.280 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.281 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.281 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.281 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.281 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.281 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.281 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.281 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.281 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.281 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.281 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.281 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.281 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.281 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.281 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.281 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.282 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.282 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.282 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.282 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.282 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.282 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.282 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.282 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.282 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.282 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.282 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.282 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.282 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.282 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.282 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.282 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.283 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.283 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.283 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.283 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.283 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.283 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.283 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.283 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.283 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.283 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.283 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.283 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.283 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.284 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.284 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.284 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.284 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.284 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.284 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.284 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.284 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.284 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.284 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.284 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.284 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.284 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.284 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.285 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.285 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.285 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.285 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.285 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.285 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.285 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.285 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.285 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.285 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.285 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.285 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.285 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.285 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.285 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-03-23T23:13:42.286 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-03-23T23:13:42.286 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.286 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.286 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.286 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.286 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.286 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.286 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.286 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.286 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.286 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.286 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.286 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.286 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-03-23T23:13:42.286 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-03-23T23:13:42.287 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.287 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.287 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.287 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.287 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.287 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.287 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.287 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.287 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.287 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.287 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.287 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.287 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.287 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.287 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.287 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rgw", 2024-03-23T23:13:42.288 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.288 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.288 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.288 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.288 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.288 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.288 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.288 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.288 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.288 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.288 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.288 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.288 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.288 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.288 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.289 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.289 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.289 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.289 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.289 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.289 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.289 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.289 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.289 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.289 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.289 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.289 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.289 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.289 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.289 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.290 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.290 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.290 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.290 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.290 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.290 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.290 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.290 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.290 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.290 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.290 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.290 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.290 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.290 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.290 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.291 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.291 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.291 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.291 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.291 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.291 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.291 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.291 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.291 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.291 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.291 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.291 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.291 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.291 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.291 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.292 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.292 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.292 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.292 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.292 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.292 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.292 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.292 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.292 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.292 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.292 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.292 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.292 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.292 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.292 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.292 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.293 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rook", 2024-03-23T23:13:42.293 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.293 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.293 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.293 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.293 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.293 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.293 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.293 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.293 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.293 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.293 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.293 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.293 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.294 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.294 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.294 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.294 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.294 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.294 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.294 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.294 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.294 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.294 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.294 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.294 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.294 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.295 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.295 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.295 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.295 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.295 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.295 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.295 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.295 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.295 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.295 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.295 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.295 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.295 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.295 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.295 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.295 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.296 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.296 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.296 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.296 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.296 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.296 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.296 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.296 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.296 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.296 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.296 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.296 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.296 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.297 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.297 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.297 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.297 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.297 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.297 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.297 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.297 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.297 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.297 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.297 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.297 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.297 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.298 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.298 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.298 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.298 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.298 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.298 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_class": { 2024-03-23T23:13:42.298 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "storage_class", 2024-03-23T23:13:42.298 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.298 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.298 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.298 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "local", 2024-03-23T23:13:42.298 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.298 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.298 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.299 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-23T23:13:42.299 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.299 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.299 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.299 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.299 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.299 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.299 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.299 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "selftest", 2024-03-23T23:13:42.299 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.299 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.299 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.299 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.300 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.300 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.300 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.300 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.300 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.300 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.300 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.300 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.300 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.300 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.300 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.300 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.300 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.300 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.300 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.300 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.301 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.301 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.301 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.301 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.301 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.301 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.301 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.301 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.301 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.301 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.301 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.301 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.301 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.302 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.302 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.302 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.302 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.302 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.302 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.302 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.302 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.302 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.302 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.302 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.302 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.302 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.302 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.302 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.303 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.303 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.303 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.303 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.303 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.303 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.303 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.303 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.303 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.303 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.303 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.303 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.303 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.303 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.303 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.303 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.304 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.304 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.304 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.304 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.304 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.304 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.304 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.304 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.304 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.304 INFO:teuthology.orchestra.run.smithi012.stdout: "roption1": { 2024-03-23T23:13:42.304 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "roption1", 2024-03-23T23:13:42.304 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.304 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.304 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.304 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.304 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.304 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.305 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.305 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.305 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.305 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.305 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.305 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.305 INFO:teuthology.orchestra.run.smithi012.stdout: "roption2": { 2024-03-23T23:13:42.305 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "roption2", 2024-03-23T23:13:42.305 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.305 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.305 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.305 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "xyz", 2024-03-23T23:13:42.305 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.305 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.305 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.306 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.306 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.306 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.306 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.306 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.306 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption1": { 2024-03-23T23:13:42.306 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption1", 2024-03-23T23:13:42.306 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.306 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.306 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.306 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.306 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.306 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.306 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.307 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.307 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.307 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.307 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.307 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.307 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption2": { 2024-03-23T23:13:42.307 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption2", 2024-03-23T23:13:42.307 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.307 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.307 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.307 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.307 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.307 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.307 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.307 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.308 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.308 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.308 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.308 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.308 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption3": { 2024-03-23T23:13:42.308 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption3", 2024-03-23T23:13:42.308 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-23T23:13:42.308 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.308 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.308 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.308 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.308 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.308 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.308 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.308 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.308 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.309 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.309 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.309 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption4": { 2024-03-23T23:13:42.309 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption4", 2024-03-23T23:13:42.309 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.309 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.309 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.309 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.309 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.309 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.309 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.309 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.309 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.309 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.309 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.309 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.310 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption5": { 2024-03-23T23:13:42.310 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption5", 2024-03-23T23:13:42.310 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.310 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.310 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.310 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.310 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.310 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.310 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.310 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.310 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.310 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.310 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.310 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.311 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption6": { 2024-03-23T23:13:42.311 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption6", 2024-03-23T23:13:42.311 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.311 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.311 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.311 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.311 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.311 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.311 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.311 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.311 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.311 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.311 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.311 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.311 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption7": { 2024-03-23T23:13:42.311 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption7", 2024-03-23T23:13:42.312 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.312 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.312 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.312 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.312 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-03-23T23:13:42.312 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "42", 2024-03-23T23:13:42.312 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.312 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.312 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.312 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.312 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.312 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.312 INFO:teuthology.orchestra.run.smithi012.stdout: "testkey": { 2024-03-23T23:13:42.312 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testkey", 2024-03-23T23:13:42.312 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.312 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.313 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.313 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.313 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.313 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.313 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.313 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.313 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.313 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.313 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.313 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.313 INFO:teuthology.orchestra.run.smithi012.stdout: "testlkey": { 2024-03-23T23:13:42.313 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testlkey", 2024-03-23T23:13:42.313 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.313 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.313 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.314 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.314 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.314 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.314 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.314 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.314 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.314 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.314 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.314 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.314 INFO:teuthology.orchestra.run.smithi012.stdout: "testnewline": { 2024-03-23T23:13:42.314 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testnewline", 2024-03-23T23:13:42.314 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.315 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.315 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.315 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.315 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.315 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.315 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.315 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.315 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.315 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.315 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.315 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.315 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.315 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.315 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.315 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snap_schedule", 2024-03-23T23:13:42.315 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.316 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.316 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.316 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_m_granularity": { 2024-03-23T23:13:42.316 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_m_granularity", 2024-03-23T23:13:42.316 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.316 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.316 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.316 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.316 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.316 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.316 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.316 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow minute scheduled snapshots", 2024-03-23T23:13:42.316 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.316 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.316 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.317 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.317 INFO:teuthology.orchestra.run.smithi012.stdout: "dump_on_update": { 2024-03-23T23:13:42.317 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "dump_on_update", 2024-03-23T23:13:42.317 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.317 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.317 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.317 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.317 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.317 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.317 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.317 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "dump database to debug log on update", 2024-03-23T23:13:42.317 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.317 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.318 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.318 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.318 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.318 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.318 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.318 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.318 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.318 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.318 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.318 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.318 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.318 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.318 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.318 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.318 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.318 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.318 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.319 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.319 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.319 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.319 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.319 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.319 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.319 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.319 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.319 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.319 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.319 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.319 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.319 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.319 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.319 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.319 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.319 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.320 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.320 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.320 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.320 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.320 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.320 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.320 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.320 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.320 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.320 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.320 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.320 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.320 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.320 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.320 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.320 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.321 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.321 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.321 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.321 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.321 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.321 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.321 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.321 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.321 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.321 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.321 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.321 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.321 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.321 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.321 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.322 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.322 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.322 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.322 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.322 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.322 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.322 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.322 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.322 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.322 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.322 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "stats", 2024-03-23T23:13:42.322 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.322 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.322 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.322 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.323 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.323 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.323 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.323 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.323 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.323 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.323 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.323 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.323 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.323 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.323 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.323 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.323 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.323 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.323 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.323 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.324 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.324 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.324 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.324 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.324 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.324 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.324 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.324 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.324 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.324 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.324 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.324 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.324 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.324 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.324 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.324 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.324 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.325 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.325 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.325 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.325 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.325 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.325 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.325 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.325 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.325 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.325 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.325 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.325 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.325 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.325 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.325 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.325 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.326 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.326 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.326 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.326 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.326 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.326 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.326 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.326 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.326 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.326 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.326 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.326 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.326 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.326 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.327 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.327 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.327 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.327 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.327 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.327 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.327 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.327 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.327 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.327 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "status", 2024-03-23T23:13:42.327 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.327 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.327 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.327 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.327 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.327 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.328 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.328 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.328 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.328 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.328 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.328 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.328 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.328 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.328 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.328 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.328 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.328 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.328 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.328 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.328 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.328 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.328 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.329 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.329 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.329 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.329 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.329 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.329 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.329 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.329 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.329 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.329 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.329 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.329 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.329 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.329 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.329 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.329 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.330 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.330 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.330 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.330 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.330 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.330 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.330 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.330 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.330 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.330 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.330 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.330 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.330 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.330 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.331 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.331 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.331 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.331 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.331 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.331 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.331 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.331 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.331 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.331 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.331 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.331 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.331 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.331 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.331 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.331 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.332 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.332 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.332 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.332 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.332 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.332 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.332 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.332 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "telegraf", 2024-03-23T23:13:42.332 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.332 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.332 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.332 INFO:teuthology.orchestra.run.smithi012.stdout: "address": { 2024-03-23T23:13:42.332 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "address", 2024-03-23T23:13:42.332 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.332 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.332 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.332 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-23T23:13:42.333 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.333 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.333 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.333 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.333 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.333 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.333 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.333 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.333 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-23T23:13:42.333 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-23T23:13:42.333 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:42.333 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.333 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.333 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15", 2024-03-23T23:13:42.333 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.333 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.333 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.334 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.334 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.334 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.334 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.334 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.334 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.334 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.334 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.334 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.334 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.334 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.334 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.335 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.335 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.335 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.335 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.335 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.335 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.335 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.335 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.335 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.335 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.335 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.335 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.335 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.335 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.335 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.335 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.335 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.336 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.336 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.336 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.336 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.336 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.336 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.336 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.336 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.336 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.336 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.336 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.336 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.336 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.336 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.336 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.336 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.337 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.337 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.337 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.337 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.337 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.337 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.337 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.337 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.337 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.337 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.337 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.337 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.337 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.337 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.337 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.337 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.338 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.338 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.338 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.338 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.338 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.338 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.338 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.338 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.338 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.338 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.338 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.338 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.338 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.338 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.338 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.339 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.339 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.339 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "telemetry", 2024-03-23T23:13:42.339 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.339 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.339 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.339 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_basic": { 2024-03-23T23:13:42.339 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_basic", 2024-03-23T23:13:42.339 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.339 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.339 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.339 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.339 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.339 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.340 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.340 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-23T23:13:42.340 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.340 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.340 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.340 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.340 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_crash": { 2024-03-23T23:13:42.340 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_crash", 2024-03-23T23:13:42.340 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.340 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.340 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.340 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.340 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.340 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.341 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.341 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-23T23:13:42.341 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.341 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.341 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.341 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.341 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_device": { 2024-03-23T23:13:42.341 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_device", 2024-03-23T23:13:42.341 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.341 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.341 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.341 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.341 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.341 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.342 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.342 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-23T23:13:42.342 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.342 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.342 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.342 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.342 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_ident": { 2024-03-23T23:13:42.342 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_ident", 2024-03-23T23:13:42.342 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.342 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.342 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.342 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.342 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.342 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.342 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.343 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-23T23:13:42.343 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.343 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.343 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.343 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.343 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_perf": { 2024-03-23T23:13:42.343 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_perf", 2024-03-23T23:13:42.343 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.344 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.344 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.344 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.344 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.344 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.344 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.344 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-23T23:13:42.344 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.344 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.344 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.344 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.344 INFO:teuthology.orchestra.run.smithi012.stdout: "contact": { 2024-03-23T23:13:42.344 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "contact", 2024-03-23T23:13:42.344 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.344 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.344 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.345 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.345 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.345 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.345 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.345 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.345 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.345 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.345 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.345 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.345 INFO:teuthology.orchestra.run.smithi012.stdout: "description": { 2024-03-23T23:13:42.345 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "description", 2024-03-23T23:13:42.345 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.345 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.345 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.346 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.346 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.346 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.346 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.346 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.346 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.346 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.346 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.346 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.346 INFO:teuthology.orchestra.run.smithi012.stdout: "device_url": { 2024-03-23T23:13:42.346 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_url", 2024-03-23T23:13:42.346 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.346 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.346 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.347 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-23T23:13:42.347 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.347 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.347 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.347 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.347 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.347 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.347 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.347 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.347 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": { 2024-03-23T23:13:42.347 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enabled", 2024-03-23T23:13:42.347 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.347 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.347 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.347 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.348 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.348 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.348 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.348 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.348 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.348 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.348 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.348 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.348 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-23T23:13:42.348 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-23T23:13:42.348 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.348 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.348 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.348 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "24", 2024-03-23T23:13:42.349 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "8", 2024-03-23T23:13:42.349 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.349 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.349 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.349 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.349 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.349 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.349 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.349 INFO:teuthology.orchestra.run.smithi012.stdout: "last_opt_revision": { 2024-03-23T23:13:42.349 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "last_opt_revision", 2024-03-23T23:13:42.349 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.349 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.349 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.349 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1", 2024-03-23T23:13:42.349 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.350 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.350 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.350 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.350 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.350 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.350 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.350 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.350 INFO:teuthology.orchestra.run.smithi012.stdout: "leaderboard": { 2024-03-23T23:13:42.350 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "leaderboard", 2024-03-23T23:13:42.350 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.350 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.350 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.350 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.350 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.351 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.351 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.351 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.351 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.351 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.351 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.351 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.351 INFO:teuthology.orchestra.run.smithi012.stdout: "leaderboard_description": { 2024-03-23T23:13:42.351 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "leaderboard_description", 2024-03-23T23:13:42.351 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.351 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.351 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.351 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.351 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.351 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.352 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.352 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.352 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.352 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.352 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.352 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.352 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.352 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.352 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.352 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.352 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.352 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.352 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.352 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.353 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.353 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.353 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.353 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.353 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.353 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.353 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.353 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.353 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.353 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.353 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.353 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.353 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.353 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.354 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.354 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.354 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.354 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.354 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.354 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.354 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.354 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.354 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.354 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.354 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.354 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.354 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.354 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.355 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.355 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.355 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.355 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.355 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.355 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.355 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.355 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.355 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.355 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.355 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.355 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.355 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.355 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.356 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.356 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.356 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.356 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.356 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.356 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.356 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.356 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.356 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.356 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.356 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.356 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.356 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.357 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.357 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.357 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.357 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.357 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.357 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.357 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.357 INFO:teuthology.orchestra.run.smithi012.stdout: "organization": { 2024-03-23T23:13:42.357 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "organization", 2024-03-23T23:13:42.357 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.357 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.357 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.357 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.357 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.357 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.358 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.358 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.358 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.358 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.358 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.358 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.358 INFO:teuthology.orchestra.run.smithi012.stdout: "proxy": { 2024-03-23T23:13:42.358 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "proxy", 2024-03-23T23:13:42.358 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.358 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.358 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.358 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.358 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.358 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.358 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.358 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.359 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.359 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.359 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.359 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.359 INFO:teuthology.orchestra.run.smithi012.stdout: "url": { 2024-03-23T23:13:42.359 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "url", 2024-03-23T23:13:42.359 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.359 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.359 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.359 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-23T23:13:42.359 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.359 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.359 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.359 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.359 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.360 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.360 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.360 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.360 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.360 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.360 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.360 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test_orchestrator", 2024-03-23T23:13:42.360 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.360 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.360 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.360 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.360 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.360 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.360 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.360 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.361 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.361 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.361 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.361 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.361 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.361 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.361 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.361 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.361 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.361 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.361 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.361 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.361 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.361 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.362 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.362 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.362 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.362 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.362 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.362 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.362 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.362 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.362 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.362 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.362 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.362 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.362 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.362 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.362 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.363 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.363 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.363 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.363 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.363 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.363 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.363 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.363 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.363 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.363 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.363 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.363 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.363 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.363 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.363 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.364 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.364 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.364 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.364 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.364 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.364 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.364 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.364 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.364 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.364 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.364 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.364 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.364 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.364 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.364 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.365 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.365 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.365 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.365 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.365 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.365 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.365 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.365 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.365 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.365 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "volumes", 2024-03-23T23:13:42.365 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.365 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.365 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.365 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.365 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.366 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.366 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.366 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.366 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.366 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.366 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.366 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.366 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.366 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.366 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.366 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.366 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.366 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.366 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.366 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.366 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.367 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.367 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.367 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.367 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.367 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.367 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.367 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.367 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.367 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.367 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.367 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.367 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.367 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.367 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.368 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.368 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.368 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.368 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.368 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.368 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.368 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.368 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.368 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.368 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.368 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.368 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.368 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.368 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.368 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.368 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.369 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.369 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.369 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.369 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.369 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.369 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.369 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.369 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.369 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.369 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.369 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.369 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.369 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.369 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.370 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.370 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.370 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.370 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.370 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.370 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.370 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.370 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.370 INFO:teuthology.orchestra.run.smithi012.stdout: "max_concurrent_clones": { 2024-03-23T23:13:42.370 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_concurrent_clones", 2024-03-23T23:13:42.370 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.370 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.371 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.371 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "4", 2024-03-23T23:13:42.371 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.371 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.371 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.371 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-23T23:13:42.371 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.371 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.371 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.371 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.371 INFO:teuthology.orchestra.run.smithi012.stdout: "periodic_async_work": { 2024-03-23T23:13:42.371 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "periodic_async_work", 2024-03-23T23:13:42.371 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.371 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.371 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.371 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.372 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.372 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.372 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.372 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Periodically check for async work", 2024-03-23T23:13:42.372 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.372 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.372 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.372 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.372 INFO:teuthology.orchestra.run.smithi012.stdout: "snapshot_clone_delay": { 2024-03-23T23:13:42.372 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snapshot_clone_delay", 2024-03-23T23:13:42.372 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.372 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.372 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.372 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-23T23:13:42.373 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.373 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.373 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.373 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-23T23:13:42.373 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.373 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.373 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.373 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.373 INFO:teuthology.orchestra.run.smithi012.stdout: "snapshot_clone_no_wait": { 2024-03-23T23:13:42.373 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snapshot_clone_no_wait", 2024-03-23T23:13:42.373 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.373 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.373 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.373 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-23T23:13:42.373 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.374 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.374 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.374 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-23T23:13:42.374 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.374 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.374 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.374 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.374 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.374 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.374 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.374 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix", 2024-03-23T23:13:42.374 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-23T23:13:42.374 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-23T23:13:42.375 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-23T23:13:42.375 INFO:teuthology.orchestra.run.smithi012.stdout: "discovery_interval": { 2024-03-23T23:13:42.375 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "discovery_interval", 2024-03-23T23:13:42.375 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-23T23:13:42.375 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.375 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.375 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "100", 2024-03-23T23:13:42.375 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.375 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.375 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.375 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.375 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.375 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.375 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.375 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.376 INFO:teuthology.orchestra.run.smithi012.stdout: "identifier": { 2024-03-23T23:13:42.376 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "identifier", 2024-03-23T23:13:42.376 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.376 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.376 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.376 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.376 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.376 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.376 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.376 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.376 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.376 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.376 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.376 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.376 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-23T23:13:42.377 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-23T23:13:42.377 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-23T23:13:42.377 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.377 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.377 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-23T23:13:42.377 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.377 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.377 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.377 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.377 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.377 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.377 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.377 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.377 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-23T23:13:42.377 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-23T23:13:42.377 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.378 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.378 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.378 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.378 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.378 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.378 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.378 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.378 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.378 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.378 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.378 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.378 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.378 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.378 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.379 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.379 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.379 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.379 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.379 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-23T23:13:42.379 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-23T23:13:42.379 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.379 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.379 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.379 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.379 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.379 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.379 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.379 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.379 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.380 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.380 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.380 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.380 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-23T23:13:42.380 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-23T23:13:42.380 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.380 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.380 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.380 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-23T23:13:42.380 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.380 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.380 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-23T23:13:42.380 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-23T23:13:42.380 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-23T23:13:42.381 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-23T23:13:42.381 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-23T23:13:42.381 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-23T23:13:42.381 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-23T23:13:42.381 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.381 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.381 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.381 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.381 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.381 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.381 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-23T23:13:42.381 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-23T23:13:42.381 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-23T23:13:42.381 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.382 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-23T23:13:42.382 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-23T23:13:42.382 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.382 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.382 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.382 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.382 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.382 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.382 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.382 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.382 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_host": { 2024-03-23T23:13:42.382 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_host", 2024-03-23T23:13:42.382 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.382 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.382 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.383 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-23T23:13:42.383 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.383 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.383 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.383 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.383 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.383 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.383 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.383 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.383 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_port": { 2024-03-23T23:13:42.383 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_port", 2024-03-23T23:13:42.383 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-23T23:13:42.383 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.383 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.383 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10051", 2024-03-23T23:13:42.383 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.384 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.384 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.384 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.384 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.384 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.384 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.384 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.384 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_sender": { 2024-03-23T23:13:42.384 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_sender", 2024-03-23T23:13:42.384 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-23T23:13:42.384 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-23T23:13:42.384 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-23T23:13:42.384 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-23T23:13:42.384 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-23T23:13:42.384 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-23T23:13:42.384 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-23T23:13:42.384 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-23T23:13:42.385 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-23T23:13:42.385 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-23T23:13:42.385 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-23T23:13:42.385 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.385 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.385 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.385 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.385 INFO:teuthology.orchestra.run.smithi012.stdout: "services": { 2024-03-23T23:13:42.385 INFO:teuthology.orchestra.run.smithi012.stdout: "dashboard": "https://172.21.15.12:8443/" 2024-03-23T23:13:42.385 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.385 INFO:teuthology.orchestra.run.smithi012.stdout: "always_on_modules": { 2024-03-23T23:13:42.385 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus": [ 2024-03-23T23:13:42.385 INFO:teuthology.orchestra.run.smithi012.stdout: "balancer", 2024-03-23T23:13:42.385 INFO:teuthology.orchestra.run.smithi012.stdout: "crash", 2024-03-23T23:13:42.385 INFO:teuthology.orchestra.run.smithi012.stdout: "devicehealth", 2024-03-23T23:13:42.385 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator", 2024-03-23T23:13:42.386 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_autoscaler", 2024-03-23T23:13:42.386 INFO:teuthology.orchestra.run.smithi012.stdout: "progress", 2024-03-23T23:13:42.386 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_support", 2024-03-23T23:13:42.386 INFO:teuthology.orchestra.run.smithi012.stdout: "status", 2024-03-23T23:13:42.386 INFO:teuthology.orchestra.run.smithi012.stdout: "telemetry", 2024-03-23T23:13:42.386 INFO:teuthology.orchestra.run.smithi012.stdout: "volumes" 2024-03-23T23:13:42.386 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.386 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific": [ 2024-03-23T23:13:42.386 INFO:teuthology.orchestra.run.smithi012.stdout: "balancer", 2024-03-23T23:13:42.386 INFO:teuthology.orchestra.run.smithi012.stdout: "crash", 2024-03-23T23:13:42.386 INFO:teuthology.orchestra.run.smithi012.stdout: "devicehealth", 2024-03-23T23:13:42.386 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator", 2024-03-23T23:13:42.386 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_autoscaler", 2024-03-23T23:13:42.386 INFO:teuthology.orchestra.run.smithi012.stdout: "progress", 2024-03-23T23:13:42.386 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_support", 2024-03-23T23:13:42.387 INFO:teuthology.orchestra.run.smithi012.stdout: "status", 2024-03-23T23:13:42.387 INFO:teuthology.orchestra.run.smithi012.stdout: "telemetry", 2024-03-23T23:13:42.387 INFO:teuthology.orchestra.run.smithi012.stdout: "volumes" 2024-03-23T23:13:42.387 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.387 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy": [ 2024-03-23T23:13:42.387 INFO:teuthology.orchestra.run.smithi012.stdout: "balancer", 2024-03-23T23:13:42.387 INFO:teuthology.orchestra.run.smithi012.stdout: "crash", 2024-03-23T23:13:42.387 INFO:teuthology.orchestra.run.smithi012.stdout: "devicehealth", 2024-03-23T23:13:42.387 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator", 2024-03-23T23:13:42.387 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_autoscaler", 2024-03-23T23:13:42.387 INFO:teuthology.orchestra.run.smithi012.stdout: "progress", 2024-03-23T23:13:42.387 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_support", 2024-03-23T23:13:42.387 INFO:teuthology.orchestra.run.smithi012.stdout: "status", 2024-03-23T23:13:42.387 INFO:teuthology.orchestra.run.smithi012.stdout: "telemetry", 2024-03-23T23:13:42.387 INFO:teuthology.orchestra.run.smithi012.stdout: "volumes" 2024-03-23T23:13:42.387 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.387 INFO:teuthology.orchestra.run.smithi012.stdout: "reef": [ 2024-03-23T23:13:42.388 INFO:teuthology.orchestra.run.smithi012.stdout: "balancer", 2024-03-23T23:13:42.388 INFO:teuthology.orchestra.run.smithi012.stdout: "crash", 2024-03-23T23:13:42.388 INFO:teuthology.orchestra.run.smithi012.stdout: "devicehealth", 2024-03-23T23:13:42.388 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator", 2024-03-23T23:13:42.388 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_autoscaler", 2024-03-23T23:13:42.388 INFO:teuthology.orchestra.run.smithi012.stdout: "progress", 2024-03-23T23:13:42.388 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_support", 2024-03-23T23:13:42.388 INFO:teuthology.orchestra.run.smithi012.stdout: "status", 2024-03-23T23:13:42.388 INFO:teuthology.orchestra.run.smithi012.stdout: "telemetry", 2024-03-23T23:13:42.388 INFO:teuthology.orchestra.run.smithi012.stdout: "volumes" 2024-03-23T23:13:42.388 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-23T23:13:42.388 INFO:teuthology.orchestra.run.smithi012.stdout: "squid": [ 2024-03-23T23:13:42.388 INFO:teuthology.orchestra.run.smithi012.stdout: "balancer", 2024-03-23T23:13:42.388 INFO:teuthology.orchestra.run.smithi012.stdout: "crash", 2024-03-23T23:13:42.389 INFO:teuthology.orchestra.run.smithi012.stdout: "devicehealth", 2024-03-23T23:13:42.389 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator", 2024-03-23T23:13:42.389 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_autoscaler", 2024-03-23T23:13:42.389 INFO:teuthology.orchestra.run.smithi012.stdout: "progress", 2024-03-23T23:13:42.389 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_support", 2024-03-23T23:13:42.389 INFO:teuthology.orchestra.run.smithi012.stdout: "status", 2024-03-23T23:13:42.389 INFO:teuthology.orchestra.run.smithi012.stdout: "telemetry", 2024-03-23T23:13:42.389 INFO:teuthology.orchestra.run.smithi012.stdout: "volumes" 2024-03-23T23:13:42.389 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-23T23:13:42.389 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.389 INFO:teuthology.orchestra.run.smithi012.stdout: "last_failure_osd_epoch": 3, 2024-03-23T23:13:42.389 INFO:teuthology.orchestra.run.smithi012.stdout: "active_clients": [ 2024-03-23T23:13:42.389 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.389 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "devicehealth", 2024-03-23T23:13:42.389 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-03-23T23:13:42.390 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.390 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-03-23T23:13:42.390 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:0", 2024-03-23T23:13:42.390 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 3160893115 2024-03-23T23:13:42.390 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.390 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-23T23:13:42.390 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.390 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.390 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "libcephsqlite", 2024-03-23T23:13:42.390 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-03-23T23:13:42.390 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.390 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-03-23T23:13:42.390 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:0", 2024-03-23T23:13:42.390 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 2110872719 2024-03-23T23:13:42.391 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.391 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-23T23:13:42.391 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.391 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.391 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_support", 2024-03-23T23:13:42.391 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-03-23T23:13:42.391 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.391 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-03-23T23:13:42.391 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:0", 2024-03-23T23:13:42.391 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 3634400133 2024-03-23T23:13:42.391 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.391 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-23T23:13:42.391 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:42.391 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.391 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "volumes", 2024-03-23T23:13:42.391 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-03-23T23:13:42.392 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-23T23:13:42.392 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-03-23T23:13:42.392 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:0", 2024-03-23T23:13:42.392 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 1818821458 2024-03-23T23:13:42.392 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.392 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-23T23:13:42.392 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:42.392 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-23T23:13:42.392 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:13:42.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:41 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2245875883' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-23T23:13:42.393 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:41 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2245875883' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-23T23:13:42.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:41.189+0000 7fec8fe00640 1 -- 172.21.15.12:0/2245875883 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fec740a2800 msgr2=0x7fec740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:42.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:41.189+0000 7fec8fe00640 1 --2- 172.21.15.12:0/2245875883 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fec740a2800 0x7fec740a4cc0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fec9019afb0 tx=0x7fec780090e0 comp rx=0 tx=0).stop 2024-03-23T23:13:42.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:41.189+0000 7fec8fe00640 1 -- 172.21.15.12:0/2245875883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec901014e0 msgr2=0x7fec90199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:13:42.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:41.189+0000 7fec8fe00640 1 --2- 172.21.15.12:0/2245875883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec901014e0 0x7fec90199aa0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fec8002efd0 tx=0x7fec80005dd0 comp rx=0 tx=0).stop 2024-03-23T23:13:42.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:41.189+0000 7fec8fe00640 1 -- 172.21.15.12:0/2245875883 shutdown_connections 2024-03-23T23:13:42.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:41.189+0000 7fec8fe00640 1 --2- 172.21.15.12:0/2245875883 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fec740a2800 0x7fec740a4cc0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:42.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:41.189+0000 7fec8fe00640 1 --2- 172.21.15.12:0/2245875883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec90103090 0x7fec901a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:42.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:41.189+0000 7fec8fe00640 1 --2- 172.21.15.12:0/2245875883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec901026d0 0x7fec90199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:42.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:41.189+0000 7fec8fe00640 1 --2- 172.21.15.12:0/2245875883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec901014e0 0x7fec90199aa0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:13:42.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:41.189+0000 7fec8fe00640 1 -- 172.21.15.12:0/2245875883 >> 172.21.15.12:0/2245875883 conn(0x7fec900fca50 msgr2=0x7fec900738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:13:42.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:41.190+0000 7fec8fe00640 1 -- 172.21.15.12:0/2245875883 shutdown_connections 2024-03-23T23:13:42.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:13:41.190+0000 7fec8fe00640 1 -- 172.21.15.12:0/2245875883 wait complete. 2024-03-23T23:13:42.400 DEBUG:tasks.check_counter:Getting stats from a 2024-03-23T23:13:42.400 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-23T23:13:42.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:41 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2245875883' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-23T23:13:42.870 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:13:43.082 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:42 smithi012 ceph-mon[31695]: pgmap v747: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:13:43.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:42 smithi173 ceph-mon[36116]: pgmap v747: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:13:43.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:42 smithi203 ceph-mon[36693]: pgmap v747: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-23T23:13:44.984 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:44 smithi012 ceph-mon[31695]: pgmap v748: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.0 KiB/s wr, 9 op/s 2024-03-23T23:13:45.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:44 smithi173 ceph-mon[36116]: pgmap v748: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.0 KiB/s wr, 9 op/s 2024-03-23T23:13:45.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:44 smithi203 ceph-mon[36693]: pgmap v748: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.0 KiB/s wr, 9 op/s 2024-03-23T23:13:45.320 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:13:45.320 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-0": { 2024-03-23T23:13:45.320 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 0, 2024-03-23T23:13:45.320 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 2, 2024-03-23T23:13:45.320 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 0, 2024-03-23T23:13:45.321 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 0, 2024-03-23T23:13:45.321 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 4, 2024-03-23T23:13:45.321 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 0, 2024-03-23T23:13:45.321 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.003980749, 2024-03-23T23:13:45.321 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.000000000, 2024-03-23T23:13:45.321 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.000885100, 2024-03-23T23:13:45.321 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-23T23:13:45.321 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:13:45.321 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-23T23:13:45.321 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-23T23:13:45.321 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-23T23:13:45.321 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:45.322 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:13:45.322 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-23T23:13:45.322 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-23T23:13:45.322 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-23T23:13:45.322 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:45.322 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-23T23:13:45.322 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 0 2024-03-23T23:13:45.322 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:45.322 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-1": { 2024-03-23T23:13:45.322 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 1, 2024-03-23T23:13:45.322 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 177, 2024-03-23T23:13:45.322 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 160, 2024-03-23T23:13:45.322 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 108624, 2024-03-23T23:13:45.322 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 3, 2024-03-23T23:13:45.322 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 2, 2024-03-23T23:13:45.323 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.033586242, 2024-03-23T23:13:45.323 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.025730455, 2024-03-23T23:13:45.323 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.004044151, 2024-03-23T23:13:45.323 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-23T23:13:45.323 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:13:45.323 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 175, 2024-03-23T23:13:45.323 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.016184218, 2024-03-23T23:13:45.323 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000092481 2024-03-23T23:13:45.323 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:45.323 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:13:45.323 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-23T23:13:45.323 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-23T23:13:45.323 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-23T23:13:45.323 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:45.323 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-23T23:13:45.323 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 108624 2024-03-23T23:13:45.324 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:45.324 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-2": { 2024-03-23T23:13:45.324 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 238, 2024-03-23T23:13:45.324 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 233, 2024-03-23T23:13:45.324 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 483584, 2024-03-23T23:13:45.324 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 139584, 2024-03-23T23:13:45.324 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 2, 2024-03-23T23:13:45.324 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 1, 2024-03-23T23:13:45.324 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.085164786, 2024-03-23T23:13:45.324 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.035908337, 2024-03-23T23:13:45.324 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.041663629, 2024-03-23T23:13:45.324 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.010686896, 2024-03-23T23:13:45.324 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:13:45.324 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 233, 2024-03-23T23:13:45.324 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.022600063, 2024-03-23T23:13:45.324 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000096995 2024-03-23T23:13:45.325 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:45.325 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:13:45.325 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-23T23:13:45.325 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-23T23:13:45.325 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-23T23:13:45.325 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:45.325 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 483584, 2024-03-23T23:13:45.325 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 139584 2024-03-23T23:13:45.325 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:45.325 INFO:teuthology.orchestra.run.smithi012.stdout: "cct": { 2024-03-23T23:13:45.325 INFO:teuthology.orchestra.run.smithi012.stdout: "total_workers": 0, 2024-03-23T23:13:45.325 INFO:teuthology.orchestra.run.smithi012.stdout: "unhealthy_workers": 0 2024-03-23T23:13:45.325 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:45.325 INFO:teuthology.orchestra.run.smithi012.stdout: "mempool": { 2024-03-23T23:13:45.325 INFO:teuthology.orchestra.run.smithi012.stdout: "bloom_filter_bytes": 0, 2024-03-23T23:13:45.325 INFO:teuthology.orchestra.run.smithi012.stdout: "bloom_filter_items": 0, 2024-03-23T23:13:45.326 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_alloc_bytes": 0, 2024-03-23T23:13:45.326 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_alloc_items": 0, 2024-03-23T23:13:45.326 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_data_bytes": 0, 2024-03-23T23:13:45.326 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_data_items": 0, 2024-03-23T23:13:45.326 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-23T23:13:45.326 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_onode_items": 0, 2024-03-23T23:13:45.326 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-23T23:13:45.326 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_meta_items": 0, 2024-03-23T23:13:45.326 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_other_bytes": 0, 2024-03-23T23:13:45.326 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_other_items": 0, 2024-03-23T23:13:45.326 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-23T23:13:45.326 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_buffer_items": 0, 2024-03-23T23:13:45.326 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_extent_bytes": 0, 2024-03-23T23:13:45.326 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_extent_items": 0, 2024-03-23T23:13:45.327 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_blob_bytes": 0, 2024-03-23T23:13:45.327 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_blob_items": 0, 2024-03-23T23:13:45.327 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-23T23:13:45.327 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_shared_blob_items": 0, 2024-03-23T23:13:45.327 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-23T23:13:45.327 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_inline_bl_items": 0, 2024-03-23T23:13:45.327 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_fsck_bytes": 0, 2024-03-23T23:13:45.327 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_fsck_items": 0, 2024-03-23T23:13:45.327 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_txc_bytes": 0, 2024-03-23T23:13:45.327 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_txc_items": 0, 2024-03-23T23:13:45.327 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-23T23:13:45.327 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_deferred_items": 0, 2024-03-23T23:13:45.327 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_bytes": 0, 2024-03-23T23:13:45.327 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_items": 0, 2024-03-23T23:13:45.327 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_bytes": 0, 2024-03-23T23:13:45.327 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_items": 0, 2024-03-23T23:13:45.327 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_reader_bytes": 0, 2024-03-23T23:13:45.328 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_reader_items": 0, 2024-03-23T23:13:45.328 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_writer_bytes": 0, 2024-03-23T23:13:45.328 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_writer_items": 0, 2024-03-23T23:13:45.328 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_anon_bytes": 149007, 2024-03-23T23:13:45.328 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_anon_items": 30, 2024-03-23T23:13:45.328 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_meta_bytes": 0, 2024-03-23T23:13:45.328 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_meta_items": 0, 2024-03-23T23:13:45.328 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_bytes": 0, 2024-03-23T23:13:45.328 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_items": 0, 2024-03-23T23:13:45.328 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_mapbl_bytes": 0, 2024-03-23T23:13:45.328 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_mapbl_items": 0, 2024-03-23T23:13:45.328 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_pglog_bytes": 0, 2024-03-23T23:13:45.328 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_pglog_items": 0, 2024-03-23T23:13:45.328 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_bytes": 0, 2024-03-23T23:13:45.328 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_items": 0, 2024-03-23T23:13:45.328 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_mapping_bytes": 0, 2024-03-23T23:13:45.329 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_mapping_items": 0, 2024-03-23T23:13:45.329 INFO:teuthology.orchestra.run.smithi012.stdout: "pgmap_bytes": 0, 2024-03-23T23:13:45.329 INFO:teuthology.orchestra.run.smithi012.stdout: "pgmap_items": 0, 2024-03-23T23:13:45.329 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_co_bytes": 928, 2024-03-23T23:13:45.329 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_co_items": 2, 2024-03-23T23:13:45.329 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_1_bytes": 0, 2024-03-23T23:13:45.329 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_1_items": 0, 2024-03-23T23:13:45.329 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_2_bytes": 0, 2024-03-23T23:13:45.329 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_2_items": 0 2024-03-23T23:13:45.329 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:45.329 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-23T23:13:45.329 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-23T23:13:45.329 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 104857600, 2024-03-23T23:13:45.329 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-23T23:13:45.330 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 235, 2024-03-23T23:13:45.330 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 447515, 2024-03-23T23:13:45.330 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-23T23:13:45.330 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 235, 2024-03-23T23:13:45.330 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 0, 2024-03-23T23:13:45.330 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 0, 2024-03-23T23:13:45.330 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 235, 2024-03-23T23:13:45.330 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 447515, 2024-03-23T23:13:45.330 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-23T23:13:45.330 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-23T23:13:45.330 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-23T23:13:45.330 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-23T23:13:45.330 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:45.331 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:45.331 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:13:46.138 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-03-23T23:13:46.139 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-03-23T23:13:46.139 DEBUG:tasks.check_counter:Getting stats from d 2024-03-23T23:13:46.139 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-23T23:13:46.578 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:13:46.990 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:46 smithi012 ceph-mon[31695]: pgmap v749: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.9 KiB/s wr, 10 op/s 2024-03-23T23:13:47.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:46 smithi173 ceph-mon[36116]: pgmap v749: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.9 KiB/s wr, 10 op/s 2024-03-23T23:13:47.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:46 smithi203 ceph-mon[36693]: pgmap v749: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.9 KiB/s wr, 10 op/s 2024-03-23T23:13:48.809 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:13:48.809 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-0": { 2024-03-23T23:13:48.809 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 0, 2024-03-23T23:13:48.809 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 2, 2024-03-23T23:13:48.809 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 0, 2024-03-23T23:13:48.809 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 0, 2024-03-23T23:13:48.809 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 4, 2024-03-23T23:13:48.809 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 0, 2024-03-23T23:13:48.810 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.003929373, 2024-03-23T23:13:48.810 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.000000000, 2024-03-23T23:13:48.810 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.000997577, 2024-03-23T23:13:48.810 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-23T23:13:48.810 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:13:48.810 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-23T23:13:48.810 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-23T23:13:48.810 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-23T23:13:48.810 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:48.810 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:13:48.810 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-23T23:13:48.810 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-23T23:13:48.810 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-23T23:13:48.810 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:48.811 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-23T23:13:48.811 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 0 2024-03-23T23:13:48.811 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:48.811 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-1": { 2024-03-23T23:13:48.811 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 1, 2024-03-23T23:13:48.811 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 182, 2024-03-23T23:13:48.811 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 160, 2024-03-23T23:13:48.811 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 109504, 2024-03-23T23:13:48.811 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 3, 2024-03-23T23:13:48.811 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 2, 2024-03-23T23:13:48.811 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.034069660, 2024-03-23T23:13:48.811 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.026768736, 2024-03-23T23:13:48.811 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.003062750, 2024-03-23T23:13:48.811 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-23T23:13:48.812 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:13:48.812 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 180, 2024-03-23T23:13:48.812 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.017359216, 2024-03-23T23:13:48.812 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000096440 2024-03-23T23:13:48.812 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:48.812 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:13:48.812 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-23T23:13:48.812 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-23T23:13:48.812 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-23T23:13:48.812 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:48.812 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-23T23:13:48.812 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 109504 2024-03-23T23:13:48.812 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:48.812 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-2": { 2024-03-23T23:13:48.812 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 245, 2024-03-23T23:13:48.812 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 240, 2024-03-23T23:13:48.812 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 485488, 2024-03-23T23:13:48.813 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 143952, 2024-03-23T23:13:48.813 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 2, 2024-03-23T23:13:48.813 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 1, 2024-03-23T23:13:48.813 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.087145022, 2024-03-23T23:13:48.813 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.038537809, 2024-03-23T23:13:48.813 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.037779816, 2024-03-23T23:13:48.813 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.011162225, 2024-03-23T23:13:48.813 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:13:48.813 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 240, 2024-03-23T23:13:48.813 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.024858107, 2024-03-23T23:13:48.813 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000103575 2024-03-23T23:13:48.813 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:48.813 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:13:48.813 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-23T23:13:48.813 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-23T23:13:48.814 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-23T23:13:48.814 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:48.814 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 485488, 2024-03-23T23:13:48.814 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 143952 2024-03-23T23:13:48.814 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:48.814 INFO:teuthology.orchestra.run.smithi012.stdout: "cct": { 2024-03-23T23:13:48.814 INFO:teuthology.orchestra.run.smithi012.stdout: "total_workers": 0, 2024-03-23T23:13:48.814 INFO:teuthology.orchestra.run.smithi012.stdout: "unhealthy_workers": 0 2024-03-23T23:13:48.814 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:48.814 INFO:teuthology.orchestra.run.smithi012.stdout: "mempool": { 2024-03-23T23:13:48.814 INFO:teuthology.orchestra.run.smithi012.stdout: "bloom_filter_bytes": 0, 2024-03-23T23:13:48.814 INFO:teuthology.orchestra.run.smithi012.stdout: "bloom_filter_items": 0, 2024-03-23T23:13:48.814 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_alloc_bytes": 0, 2024-03-23T23:13:48.814 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_alloc_items": 0, 2024-03-23T23:13:48.815 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_data_bytes": 0, 2024-03-23T23:13:48.815 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_data_items": 0, 2024-03-23T23:13:48.815 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-23T23:13:48.815 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_onode_items": 0, 2024-03-23T23:13:48.815 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-23T23:13:48.815 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_meta_items": 0, 2024-03-23T23:13:48.815 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_other_bytes": 0, 2024-03-23T23:13:48.815 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_other_items": 0, 2024-03-23T23:13:48.815 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-23T23:13:48.815 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_buffer_items": 0, 2024-03-23T23:13:48.815 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_extent_bytes": 0, 2024-03-23T23:13:48.815 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_extent_items": 0, 2024-03-23T23:13:48.815 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_blob_bytes": 0, 2024-03-23T23:13:48.815 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_blob_items": 0, 2024-03-23T23:13:48.816 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-23T23:13:48.816 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_shared_blob_items": 0, 2024-03-23T23:13:48.816 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-23T23:13:48.816 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_inline_bl_items": 0, 2024-03-23T23:13:48.816 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_fsck_bytes": 0, 2024-03-23T23:13:48.816 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_fsck_items": 0, 2024-03-23T23:13:48.816 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_txc_bytes": 0, 2024-03-23T23:13:48.816 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_txc_items": 0, 2024-03-23T23:13:48.816 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-23T23:13:48.816 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_deferred_items": 0, 2024-03-23T23:13:48.816 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_bytes": 0, 2024-03-23T23:13:48.816 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_items": 0, 2024-03-23T23:13:48.816 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_bytes": 0, 2024-03-23T23:13:48.816 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_items": 0, 2024-03-23T23:13:48.817 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_reader_bytes": 0, 2024-03-23T23:13:48.817 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_reader_items": 0, 2024-03-23T23:13:48.817 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_writer_bytes": 0, 2024-03-23T23:13:48.817 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_writer_items": 0, 2024-03-23T23:13:48.817 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_anon_bytes": 149998, 2024-03-23T23:13:48.817 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_anon_items": 30, 2024-03-23T23:13:48.817 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_meta_bytes": 0, 2024-03-23T23:13:48.817 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_meta_items": 0, 2024-03-23T23:13:48.817 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_bytes": 0, 2024-03-23T23:13:48.817 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_items": 0, 2024-03-23T23:13:48.817 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_mapbl_bytes": 0, 2024-03-23T23:13:48.817 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_mapbl_items": 0, 2024-03-23T23:13:48.817 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_pglog_bytes": 0, 2024-03-23T23:13:48.818 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_pglog_items": 0, 2024-03-23T23:13:48.818 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_bytes": 0, 2024-03-23T23:13:48.818 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_items": 0, 2024-03-23T23:13:48.818 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_mapping_bytes": 0, 2024-03-23T23:13:48.818 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_mapping_items": 0, 2024-03-23T23:13:48.818 INFO:teuthology.orchestra.run.smithi012.stdout: "pgmap_bytes": 0, 2024-03-23T23:13:48.818 INFO:teuthology.orchestra.run.smithi012.stdout: "pgmap_items": 0, 2024-03-23T23:13:48.818 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_co_bytes": 928, 2024-03-23T23:13:48.818 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_co_items": 2, 2024-03-23T23:13:48.818 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_1_bytes": 0, 2024-03-23T23:13:48.818 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_1_items": 0, 2024-03-23T23:13:48.818 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_2_bytes": 0, 2024-03-23T23:13:48.818 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_2_items": 0 2024-03-23T23:13:48.818 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:48.818 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-23T23:13:48.819 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-23T23:13:48.819 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 104857600, 2024-03-23T23:13:48.819 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-23T23:13:48.819 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 242, 2024-03-23T23:13:48.819 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 448425, 2024-03-23T23:13:48.819 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-23T23:13:48.819 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 242, 2024-03-23T23:13:48.819 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 0, 2024-03-23T23:13:48.819 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 0, 2024-03-23T23:13:48.819 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 242, 2024-03-23T23:13:48.819 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 448425, 2024-03-23T23:13:48.819 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-23T23:13:48.819 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-23T23:13:48.819 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-23T23:13:48.820 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-23T23:13:48.820 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:48.820 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:48.820 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:13:49.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:48 smithi012 ceph-mon[31695]: pgmap v750: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 3.7 KiB/s wr, 8 op/s 2024-03-23T23:13:49.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:48 smithi173 ceph-mon[36116]: pgmap v750: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 3.7 KiB/s wr, 8 op/s 2024-03-23T23:13:49.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:48 smithi203 ceph-mon[36693]: pgmap v750: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 3.7 KiB/s wr, 8 op/s 2024-03-23T23:13:49.706 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-23T23:13:49.706 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-23T23:13:49.706 DEBUG:tasks.check_counter:Getting stats from g 2024-03-23T23:13:49.706 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-23T23:13:50.196 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:13:51.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:50 smithi012 ceph-mon[31695]: pgmap v751: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2024-03-23T23:13:51.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:50 smithi173 ceph-mon[36116]: pgmap v751: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2024-03-23T23:13:51.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:50 smithi203 ceph-mon[36693]: pgmap v751: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2024-03-23T23:13:52.375 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:13:52.375 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-0": { 2024-03-23T23:13:52.376 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 0, 2024-03-23T23:13:52.376 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 2, 2024-03-23T23:13:52.376 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 0, 2024-03-23T23:13:52.376 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 0, 2024-03-23T23:13:52.376 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 4, 2024-03-23T23:13:52.376 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 0, 2024-03-23T23:13:52.376 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.004377683, 2024-03-23T23:13:52.376 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.000000000, 2024-03-23T23:13:52.376 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.001056203, 2024-03-23T23:13:52.376 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-23T23:13:52.376 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:13:52.376 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-23T23:13:52.376 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-23T23:13:52.376 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-23T23:13:52.377 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:52.377 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:13:52.377 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-23T23:13:52.377 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-23T23:13:52.377 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-23T23:13:52.377 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:52.377 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-23T23:13:52.377 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 0 2024-03-23T23:13:52.377 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:52.377 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-1": { 2024-03-23T23:13:52.377 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 237, 2024-03-23T23:13:52.377 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 234, 2024-03-23T23:13:52.377 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 481408, 2024-03-23T23:13:52.377 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 141008, 2024-03-23T23:13:52.377 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 2, 2024-03-23T23:13:52.378 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 1, 2024-03-23T23:13:52.378 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.082768166, 2024-03-23T23:13:52.378 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.036177033, 2024-03-23T23:13:52.378 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.034907333, 2024-03-23T23:13:52.378 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.010967774, 2024-03-23T23:13:52.378 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:13:52.378 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 233, 2024-03-23T23:13:52.378 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.022951915, 2024-03-23T23:13:52.378 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000098506 2024-03-23T23:13:52.378 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:52.378 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:13:52.378 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-23T23:13:52.378 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-23T23:13:52.378 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-23T23:13:52.379 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:52.379 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 481408, 2024-03-23T23:13:52.379 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 141008 2024-03-23T23:13:52.379 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:52.379 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-2": { 2024-03-23T23:13:52.379 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 5, 2024-03-23T23:13:52.379 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 181, 2024-03-23T23:13:52.379 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 3152, 2024-03-23T23:13:52.379 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 109424, 2024-03-23T23:13:52.379 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 3, 2024-03-23T23:13:52.379 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 2, 2024-03-23T23:13:52.379 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.036465360, 2024-03-23T23:13:52.379 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.028098401, 2024-03-23T23:13:52.380 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.004352350, 2024-03-23T23:13:52.380 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-23T23:13:52.380 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:13:52.380 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 180, 2024-03-23T23:13:52.380 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.018457325, 2024-03-23T23:13:52.380 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000102540 2024-03-23T23:13:52.380 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:52.380 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:13:52.380 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-23T23:13:52.380 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-23T23:13:52.380 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-23T23:13:52.380 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:52.380 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-23T23:13:52.380 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 109424 2024-03-23T23:13:52.380 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:52.381 INFO:teuthology.orchestra.run.smithi012.stdout: "cct": { 2024-03-23T23:13:52.381 INFO:teuthology.orchestra.run.smithi012.stdout: "total_workers": 0, 2024-03-23T23:13:52.381 INFO:teuthology.orchestra.run.smithi012.stdout: "unhealthy_workers": 0 2024-03-23T23:13:52.381 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:52.381 INFO:teuthology.orchestra.run.smithi012.stdout: "mempool": { 2024-03-23T23:13:52.381 INFO:teuthology.orchestra.run.smithi012.stdout: "bloom_filter_bytes": 0, 2024-03-23T23:13:52.381 INFO:teuthology.orchestra.run.smithi012.stdout: "bloom_filter_items": 0, 2024-03-23T23:13:52.381 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_alloc_bytes": 0, 2024-03-23T23:13:52.381 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_alloc_items": 0, 2024-03-23T23:13:52.381 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_data_bytes": 0, 2024-03-23T23:13:52.381 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_data_items": 0, 2024-03-23T23:13:52.381 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-23T23:13:52.381 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_onode_items": 0, 2024-03-23T23:13:52.381 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-23T23:13:52.381 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_meta_items": 0, 2024-03-23T23:13:52.381 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_other_bytes": 0, 2024-03-23T23:13:52.382 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_other_items": 0, 2024-03-23T23:13:52.382 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-23T23:13:52.382 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_buffer_items": 0, 2024-03-23T23:13:52.382 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_extent_bytes": 0, 2024-03-23T23:13:52.382 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_extent_items": 0, 2024-03-23T23:13:52.382 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_blob_bytes": 0, 2024-03-23T23:13:52.382 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_blob_items": 0, 2024-03-23T23:13:52.382 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-23T23:13:52.382 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_shared_blob_items": 0, 2024-03-23T23:13:52.382 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-23T23:13:52.382 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_inline_bl_items": 0, 2024-03-23T23:13:52.382 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_fsck_bytes": 0, 2024-03-23T23:13:52.382 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_fsck_items": 0, 2024-03-23T23:13:52.382 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_txc_bytes": 0, 2024-03-23T23:13:52.383 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_txc_items": 0, 2024-03-23T23:13:52.383 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-23T23:13:52.383 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_deferred_items": 0, 2024-03-23T23:13:52.383 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_bytes": 0, 2024-03-23T23:13:52.383 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_items": 0, 2024-03-23T23:13:52.383 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_bytes": 0, 2024-03-23T23:13:52.383 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_items": 0, 2024-03-23T23:13:52.383 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_reader_bytes": 0, 2024-03-23T23:13:52.383 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_reader_items": 0, 2024-03-23T23:13:52.383 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_writer_bytes": 0, 2024-03-23T23:13:52.383 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_writer_items": 0, 2024-03-23T23:13:52.383 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_anon_bytes": 151029, 2024-03-23T23:13:52.383 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_anon_items": 31, 2024-03-23T23:13:52.383 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_meta_bytes": 0, 2024-03-23T23:13:52.383 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_meta_items": 0, 2024-03-23T23:13:52.384 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_bytes": 0, 2024-03-23T23:13:52.384 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_items": 0, 2024-03-23T23:13:52.384 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_mapbl_bytes": 0, 2024-03-23T23:13:52.384 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_mapbl_items": 0, 2024-03-23T23:13:52.384 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_pglog_bytes": 0, 2024-03-23T23:13:52.384 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_pglog_items": 0, 2024-03-23T23:13:52.384 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_bytes": 0, 2024-03-23T23:13:52.384 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_items": 0, 2024-03-23T23:13:52.384 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_mapping_bytes": 0, 2024-03-23T23:13:52.384 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_mapping_items": 0, 2024-03-23T23:13:52.384 INFO:teuthology.orchestra.run.smithi012.stdout: "pgmap_bytes": 0, 2024-03-23T23:13:52.384 INFO:teuthology.orchestra.run.smithi012.stdout: "pgmap_items": 0, 2024-03-23T23:13:52.384 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_co_bytes": 928, 2024-03-23T23:13:52.384 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_co_items": 2, 2024-03-23T23:13:52.384 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_1_bytes": 0, 2024-03-23T23:13:52.385 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_1_items": 0, 2024-03-23T23:13:52.385 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_2_bytes": 0, 2024-03-23T23:13:52.385 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_2_items": 0 2024-03-23T23:13:52.385 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:52.385 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-23T23:13:52.385 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-23T23:13:52.385 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 104857600, 2024-03-23T23:13:52.385 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-23T23:13:52.385 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 238, 2024-03-23T23:13:52.385 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 447905, 2024-03-23T23:13:52.385 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-23T23:13:52.385 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 238, 2024-03-23T23:13:52.385 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 0, 2024-03-23T23:13:52.385 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 0, 2024-03-23T23:13:52.385 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 238, 2024-03-23T23:13:52.386 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 447905, 2024-03-23T23:13:52.386 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-23T23:13:52.386 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-23T23:13:52.386 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-23T23:13:52.386 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-23T23:13:52.386 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:52.386 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:52.386 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:13:53.035 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:52 smithi012 ceph-mon[31695]: pgmap v752: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2024-03-23T23:13:53.107 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-03-23T23:13:53.107 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-03-23T23:13:53.107 DEBUG:tasks.check_counter:Getting stats from j 2024-03-23T23:13:53.107 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-23T23:13:53.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:52 smithi173 ceph-mon[36116]: pgmap v752: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2024-03-23T23:13:53.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:52 smithi203 ceph-mon[36693]: pgmap v752: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2024-03-23T23:13:53.601 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.a/config 2024-03-23T23:13:55.011 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:54 smithi012 ceph-mon[31695]: pgmap v753: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-23T23:13:55.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:54 smithi173 ceph-mon[36116]: pgmap v753: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-23T23:13:55.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:54 smithi203 ceph-mon[36693]: pgmap v753: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-23T23:13:56.048 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-23T23:13:56.048 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-0": { 2024-03-23T23:13:56.049 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 245, 2024-03-23T23:13:56.049 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 242, 2024-03-23T23:13:56.049 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 484208, 2024-03-23T23:13:56.049 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 145712, 2024-03-23T23:13:56.049 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 4, 2024-03-23T23:13:56.049 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 0, 2024-03-23T23:13:56.049 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.093973625, 2024-03-23T23:13:56.049 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.037640853, 2024-03-23T23:13:56.049 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.043957601, 2024-03-23T23:13:56.049 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.011944618, 2024-03-23T23:13:56.049 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:13:56.049 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 241, 2024-03-23T23:13:56.049 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.023403352, 2024-03-23T23:13:56.049 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000097109 2024-03-23T23:13:56.049 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:56.050 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:13:56.050 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-23T23:13:56.050 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-23T23:13:56.050 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-23T23:13:56.050 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:56.050 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 484208, 2024-03-23T23:13:56.050 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 145712 2024-03-23T23:13:56.050 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:56.050 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-1": { 2024-03-23T23:13:56.050 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 6, 2024-03-23T23:13:56.050 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 187, 2024-03-23T23:13:56.050 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 3312, 2024-03-23T23:13:56.050 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 188304, 2024-03-23T23:13:56.050 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 4, 2024-03-23T23:13:56.051 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 2, 2024-03-23T23:13:56.051 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.038988765, 2024-03-23T23:13:56.051 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.029170824, 2024-03-23T23:13:56.051 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.005041288, 2024-03-23T23:13:56.051 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-23T23:13:56.051 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:13:56.051 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 186, 2024-03-23T23:13:56.051 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.021948139, 2024-03-23T23:13:56.051 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000118000 2024-03-23T23:13:56.051 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:56.051 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:13:56.051 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-23T23:13:56.051 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-23T23:13:56.051 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-23T23:13:56.051 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:56.052 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-23T23:13:56.052 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 188304 2024-03-23T23:13:56.052 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:56.052 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-2": { 2024-03-23T23:13:56.052 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 0, 2024-03-23T23:13:56.052 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 2, 2024-03-23T23:13:56.052 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 0, 2024-03-23T23:13:56.052 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 0, 2024-03-23T23:13:56.052 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 2, 2024-03-23T23:13:56.052 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 1, 2024-03-23T23:13:56.052 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.001130062, 2024-03-23T23:13:56.052 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.000000000, 2024-03-23T23:13:56.052 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.000947040, 2024-03-23T23:13:56.052 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-23T23:13:56.053 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:13:56.053 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-23T23:13:56.053 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-23T23:13:56.053 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-23T23:13:56.053 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:56.053 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:13:56.053 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-23T23:13:56.053 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-23T23:13:56.053 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-23T23:13:56.053 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:56.053 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-23T23:13:56.053 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 0 2024-03-23T23:13:56.053 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:56.053 INFO:teuthology.orchestra.run.smithi012.stdout: "cct": { 2024-03-23T23:13:56.053 INFO:teuthology.orchestra.run.smithi012.stdout: "total_workers": 0, 2024-03-23T23:13:56.054 INFO:teuthology.orchestra.run.smithi012.stdout: "unhealthy_workers": 0 2024-03-23T23:13:56.054 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:56.054 INFO:teuthology.orchestra.run.smithi012.stdout: "mempool": { 2024-03-23T23:13:56.054 INFO:teuthology.orchestra.run.smithi012.stdout: "bloom_filter_bytes": 0, 2024-03-23T23:13:56.054 INFO:teuthology.orchestra.run.smithi012.stdout: "bloom_filter_items": 0, 2024-03-23T23:13:56.054 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_alloc_bytes": 0, 2024-03-23T23:13:56.054 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_alloc_items": 0, 2024-03-23T23:13:56.054 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_data_bytes": 0, 2024-03-23T23:13:56.054 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_data_items": 0, 2024-03-23T23:13:56.054 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-23T23:13:56.054 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_onode_items": 0, 2024-03-23T23:13:56.054 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-23T23:13:56.054 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_meta_items": 0, 2024-03-23T23:13:56.054 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_other_bytes": 0, 2024-03-23T23:13:56.055 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_other_items": 0, 2024-03-23T23:13:56.055 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-23T23:13:56.055 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_buffer_items": 0, 2024-03-23T23:13:56.055 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_extent_bytes": 0, 2024-03-23T23:13:56.055 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_extent_items": 0, 2024-03-23T23:13:56.055 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_blob_bytes": 0, 2024-03-23T23:13:56.055 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_blob_items": 0, 2024-03-23T23:13:56.055 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-23T23:13:56.055 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_shared_blob_items": 0, 2024-03-23T23:13:56.055 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-23T23:13:56.055 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_inline_bl_items": 0, 2024-03-23T23:13:56.055 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_fsck_bytes": 0, 2024-03-23T23:13:56.055 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_fsck_items": 0, 2024-03-23T23:13:56.055 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_txc_bytes": 0, 2024-03-23T23:13:56.055 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_txc_items": 0, 2024-03-23T23:13:56.055 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-23T23:13:56.056 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_deferred_items": 0, 2024-03-23T23:13:56.056 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_bytes": 0, 2024-03-23T23:13:56.056 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_items": 0, 2024-03-23T23:13:56.056 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_bytes": 0, 2024-03-23T23:13:56.056 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_items": 0, 2024-03-23T23:13:56.056 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_reader_bytes": 0, 2024-03-23T23:13:56.056 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_reader_items": 0, 2024-03-23T23:13:56.056 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_writer_bytes": 0, 2024-03-23T23:13:56.056 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_writer_items": 0, 2024-03-23T23:13:56.056 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_anon_bytes": 149982, 2024-03-23T23:13:56.056 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_anon_items": 29, 2024-03-23T23:13:56.056 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_meta_bytes": 0, 2024-03-23T23:13:56.056 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_meta_items": 0, 2024-03-23T23:13:56.056 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_bytes": 0, 2024-03-23T23:13:56.056 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_items": 0, 2024-03-23T23:13:56.056 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_mapbl_bytes": 0, 2024-03-23T23:13:56.057 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_mapbl_items": 0, 2024-03-23T23:13:56.057 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_pglog_bytes": 0, 2024-03-23T23:13:56.057 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_pglog_items": 0, 2024-03-23T23:13:56.057 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_bytes": 0, 2024-03-23T23:13:56.057 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_items": 0, 2024-03-23T23:13:56.057 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_mapping_bytes": 0, 2024-03-23T23:13:56.057 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_mapping_items": 0, 2024-03-23T23:13:56.057 INFO:teuthology.orchestra.run.smithi012.stdout: "pgmap_bytes": 0, 2024-03-23T23:13:56.057 INFO:teuthology.orchestra.run.smithi012.stdout: "pgmap_items": 0, 2024-03-23T23:13:56.057 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_co_bytes": 928, 2024-03-23T23:13:56.057 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_co_items": 2, 2024-03-23T23:13:56.057 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_1_bytes": 0, 2024-03-23T23:13:56.057 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_1_items": 0, 2024-03-23T23:13:56.057 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_2_bytes": 0, 2024-03-23T23:13:56.058 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_2_items": 0 2024-03-23T23:13:56.058 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-23T23:13:56.058 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-23T23:13:56.058 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-23T23:13:56.058 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 104857600, 2024-03-23T23:13:56.058 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-23T23:13:56.058 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 247, 2024-03-23T23:13:56.058 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 449609, 2024-03-23T23:13:56.058 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-23T23:13:56.058 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 247, 2024-03-23T23:13:56.058 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 0, 2024-03-23T23:13:56.058 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 0, 2024-03-23T23:13:56.058 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 247, 2024-03-23T23:13:56.058 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 449609, 2024-03-23T23:13:56.058 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-23T23:13:56.059 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-23T23:13:56.059 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-23T23:13:56.059 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-23T23:13:56.059 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:56.059 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-23T23:13:56.059 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-23T23:13:56.807 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-23T23:13:56.807 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-23T23:13:56.807 DEBUG:tasks.check_counter:Getting stats from b 2024-03-23T23:13:56.808 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-23T23:13:57.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:56 smithi012 ceph-mon[31695]: pgmap v754: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.4 KiB/s wr, 7 op/s 2024-03-23T23:13:57.074 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.b/config 2024-03-23T23:13:57.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:56 smithi173 ceph-mon[36116]: pgmap v754: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.4 KiB/s wr, 7 op/s 2024-03-23T23:13:57.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:56 smithi203 ceph-mon[36693]: pgmap v754: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.4 KiB/s wr, 7 op/s 2024-03-23T23:13:58.877 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-23T23:13:58.877 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2024-03-23T23:13:58.877 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 4823, 2024-03-23T23:13:58.877 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 5274, 2024-03-23T23:13:58.878 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 1280875, 2024-03-23T23:13:58.878 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 12511571, 2024-03-23T23:13:58.878 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 287, 2024-03-23T23:13:58.878 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 5, 2024-03-23T23:13:58.878 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.384962439, 2024-03-23T23:13:58.878 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.130568394, 2024-03-23T23:13:58.878 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.141568845, 2024-03-23T23:13:58.878 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.083790935, 2024-03-23T23:13:58.878 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:13:58.878 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 5276, 2024-03-23T23:13:58.878 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.386493847, 2024-03-23T23:13:58.878 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000073255 2024-03-23T23:13:58.878 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.878 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:13:58.878 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2444, 2024-03-23T23:13:58.879 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000491962, 2024-03-23T23:13:58.879 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000201 2024-03-23T23:13:58.879 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.879 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-23T23:13:58.879 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 0 2024-03-23T23:13:58.879 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.879 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2024-03-23T23:13:58.879 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 2991, 2024-03-23T23:13:58.879 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 4592, 2024-03-23T23:13:58.879 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 1490300, 2024-03-23T23:13:58.879 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 1859201, 2024-03-23T23:13:58.879 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 13, 2024-03-23T23:13:58.879 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 5, 2024-03-23T23:13:58.879 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.234457682, 2024-03-23T23:13:58.879 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.086208428, 2024-03-23T23:13:58.880 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.109995263, 2024-03-23T23:13:58.880 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.043005987, 2024-03-23T23:13:58.880 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:13:58.880 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 4592, 2024-03-23T23:13:58.880 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.734203479, 2024-03-23T23:13:58.880 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000159887 2024-03-23T23:13:58.880 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.880 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:13:58.880 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 4685, 2024-03-23T23:13:58.880 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000956197, 2024-03-23T23:13:58.880 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000204 2024-03-23T23:13:58.880 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.880 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 586544, 2024-03-23T23:13:58.880 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 209488 2024-03-23T23:13:58.880 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.881 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2024-03-23T23:13:58.881 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 1097, 2024-03-23T23:13:58.881 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 1173, 2024-03-23T23:13:58.881 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 235994, 2024-03-23T23:13:58.881 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 882449, 2024-03-23T23:13:58.881 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 10, 2024-03-23T23:13:58.881 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 7, 2024-03-23T23:13:58.881 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.082244328, 2024-03-23T23:13:58.881 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.031836668, 2024-03-23T23:13:58.881 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.027445724, 2024-03-23T23:13:58.881 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.019362445, 2024-03-23T23:13:58.881 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:13:58.881 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1166, 2024-03-23T23:13:58.882 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.254814948, 2024-03-23T23:13:58.882 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000218537 2024-03-23T23:13:58.882 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.882 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:13:58.882 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1134, 2024-03-23T23:13:58.882 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000197470, 2024-03-23T23:13:58.882 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000174 2024-03-23T23:13:58.882 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.882 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-23T23:13:58.882 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 479152 2024-03-23T23:13:58.882 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.882 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2024-03-23T23:13:58.882 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 1, 2024-03-23T23:13:58.882 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2024-03-23T23:13:58.883 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.883 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-MDSRank": { 2024-03-23T23:13:58.883 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-03-23T23:13:58.883 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-03-23T23:13:58.883 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 6215, 2024-03-23T23:13:58.883 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 1.870761361, 2024-03-23T23:13:58.883 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000301007 2024-03-23T23:13:58.883 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:13:58.883 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.883 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-PurgeQueue": { 2024-03-23T23:13:58.883 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-03-23T23:13:58.883 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-03-23T23:13:58.883 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 17, 2024-03-23T23:13:58.883 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000395708, 2024-03-23T23:13:58.884 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000023276 2024-03-23T23:13:58.884 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:13:58.884 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.884 INFO:teuthology.orchestra.run.smithi173.stdout: "mds": { 2024-03-23T23:13:58.884 INFO:teuthology.orchestra.run.smithi173.stdout: "request": 665, 2024-03-23T23:13:58.884 INFO:teuthology.orchestra.run.smithi173.stdout: "reply": 665, 2024-03-23T23:13:58.884 INFO:teuthology.orchestra.run.smithi173.stdout: "reply_latency": { 2024-03-23T23:13:58.884 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 665, 2024-03-23T23:13:58.884 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 2.893052155, 2024-03-23T23:13:58.884 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.004350454 2024-03-23T23:13:58.884 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.884 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_reply": 0, 2024-03-23T23:13:58.884 INFO:teuthology.orchestra.run.smithi173.stdout: "forward": 0, 2024-03-23T23:13:58.884 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_complete": 0, 2024-03-23T23:13:58.884 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_keys": 0, 2024-03-23T23:13:58.885 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_commit": 349, 2024-03-23T23:13:58.885 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_split": 0, 2024-03-23T23:13:58.885 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_merge": 0, 2024-03-23T23:13:58.885 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes": 25, 2024-03-23T23:13:58.885 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_top": 5, 2024-03-23T23:13:58.885 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_bottom": 20, 2024-03-23T23:13:58.885 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pin_tail": 0, 2024-03-23T23:13:58.885 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pinned": 17, 2024-03-23T23:13:58.885 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_expired": 0, 2024-03-23T23:13:58.885 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_with_caps": 2, 2024-03-23T23:13:58.885 INFO:teuthology.orchestra.run.smithi173.stdout: "caps": 2, 2024-03-23T23:13:58.885 INFO:teuthology.orchestra.run.smithi173.stdout: "subtrees": 2, 2024-03-23T23:13:58.885 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse": 729, 2024-03-23T23:13:58.885 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_hit": 684, 2024-03-23T23:13:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_forward": 0, 2024-03-23T23:13:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_discover": 0, 2024-03-23T23:13:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_dir_fetch": 0, 2024-03-23T23:13:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_remote_ino": 0, 2024-03-23T23:13:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_lock": 0, 2024-03-23T23:13:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: "load_cent": 57, 2024-03-23T23:13:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: "q": 0, 2024-03-23T23:13:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: "exported": 0, 2024-03-23T23:13:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: "exported_inodes": 0, 2024-03-23T23:13:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: "imported": 0, 2024-03-23T23:13:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: "imported_inodes": 0, 2024-03-23T23:13:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_dir_fetch": 0, 2024-03-23T23:13:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_backtrace_fetch": 302, 2024-03-23T23:13:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_peer_discover": 0, 2024-03-23T23:13:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rfiles": 4, 2024-03-23T23:13:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rbytes": 234, 2024-03-23T23:13:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rsnaps": 0, 2024-03-23T23:13:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_fetch": 302, 2024-03-23T23:13:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_set_tag": 0, 2024-03-23T23:13:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_repaired": 0, 2024-03-23T23:13:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_inotable_repaired": 0, 2024-03-23T23:13:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_inodes": 195, 2024-03-23T23:13:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_base_inodes": 22, 2024-03-23T23:13:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dirfrag_rstats": 173, 2024-03-23T23:13:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_file_inodes": 107, 2024-03-23T23:13:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_inode_file_caps": 1, 2024-03-23T23:13:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_revoke": 615, 2024-03-23T23:13:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_grant": 617, 2024-03-23T23:13:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_trunc": 599, 2024-03-23T23:13:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-23T23:13:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-23T23:13:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps": 724, 2024-03-23T23:13:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps_dirty": 702, 2024-03-23T23:13:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_cap_release": 2, 2024-03-23T23:13:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: "process_request_cap_release": 624 2024-03-23T23:13:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_cache": { 2024-03-23T23:13:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update": 0, 2024-03-23T23:13:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update_receipt": 0, 2024-03-23T23:13:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_try_discover": 0, 2024-03-23T23:13:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_send_discover": 0, 2024-03-23T23:13:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_handle_discover": 3, 2024-03-23T23:13:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays": 1, 2024-03-23T23:13:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_delayed": 0, 2024-03-23T23:13:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_enqueuing": 0, 2024-03-23T23:13:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_created": 3, 2024-03-23T23:13:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_enqueued": 2, 2024-03-23T23:13:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_reintegrated": 0, 2024-03-23T23:13:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_migrated": 0, 2024-03-23T23:13:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_processing": 0, 2024-03-23T23:13:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_enqueued": 0, 2024-03-23T23:13:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_prioritized": 0, 2024-03-23T23:13:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_started": 0, 2024-03-23T23:13:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_completed": 0, 2024-03-23T23:13:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_enqueue_scrub": 22, 2024-03-23T23:13:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_exportdir": 0, 2024-03-23T23:13:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_flush": 0, 2024-03-23T23:13:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragmentdir": 0, 2024-03-23T23:13:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragstats": 0, 2024-03-23T23:13:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_inodestats": 0 2024-03-23T23:13:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_log": { 2024-03-23T23:13:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: "evlrg": 0, 2024-03-23T23:13:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: "evadd": 2437, 2024-03-23T23:13:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: "evex": 2381, 2024-03-23T23:13:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: "evtrm": 2381, 2024-03-23T23:13:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: "ev": 56, 2024-03-23T23:13:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: "evexg": 0, 2024-03-23T23:13:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: "evexd": 51, 2024-03-23T23:13:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: "segadd": 55, 2024-03-23T23:13:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: "segex": 54, 2024-03-23T23:13:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: "segtrm": 54, 2024-03-23T23:13:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: "seg": 1, 2024-03-23T23:13:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: "segmjr": 1, 2024-03-23T23:13:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: "segexg": 0, 2024-03-23T23:13:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: "segexd": 1, 2024-03-23T23:13:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: "expos": 11905007, 2024-03-23T23:13:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: "wrpos": 12078591, 2024-03-23T23:13:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: "rdpos": 0, 2024-03-23T23:13:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: "jlat": { 2024-03-23T23:13:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2422, 2024-03-23T23:13:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 4.281079739, 2024-03-23T23:13:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001767580 2024-03-23T23:13:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: "replayed": 0 2024-03-23T23:13:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_mem": { 2024-03-23T23:13:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: "ino": 24, 2024-03-23T23:13:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: "ino+": 48, 2024-03-23T23:13:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: "ino-": 24, 2024-03-23T23:13:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: "dir": 19, 2024-03-23T23:13:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: "dir+": 20, 2024-03-23T23:13:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: "dir-": 1, 2024-03-23T23:13:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: "dn": 25, 2024-03-23T23:13:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: "dn+": 29, 2024-03-23T23:13:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: "dn-": 4, 2024-03-23T23:13:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: "cap": 2, 2024-03-23T23:13:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: "cap+": 16, 2024-03-23T23:13:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: "cap-": 14, 2024-03-23T23:13:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: "rss": 53984, 2024-03-23T23:13:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: "heap": 231804 2024-03-23T23:13:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_server": { 2024-03-23T23:13:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_client_request": 1885, 2024-03-23T23:13:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_server_request": 0, 2024-03-23T23:13:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_request": 665, 2024-03-23T23:13:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_session": 16, 2024-03-23T23:13:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_peer_request": 0, 2024-03-23T23:13:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: "req_create_latency": { 2024-03-23T23:13:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 5, 2024-03-23T23:13:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.006018432, 2024-03-23T23:13:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001203686 2024-03-23T23:13:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getattr_latency": { 2024-03-23T23:13:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 16, 2024-03-23T23:13:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.003528948, 2024-03-23T23:13:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000220559 2024-03-23T23:13:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getfilelock_latency": { 2024-03-23T23:13:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:13:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:13:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:13:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: "req_link_latency": { 2024-03-23T23:13:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:13:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:13:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:13:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookup_latency": { 2024-03-23T23:13:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 21, 2024-03-23T23:13:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.004066405, 2024-03-23T23:13:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000193638 2024-03-23T23:13:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookuphash_latency": { 2024-03-23T23:13:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:13:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:13:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:13:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupino_latency": { 2024-03-23T23:13:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1, 2024-03-23T23:13:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000213261, 2024-03-23T23:13:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000213261 2024-03-23T23:13:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupname_latency": { 2024-03-23T23:13:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:13:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:13:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:13:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupparent_latency": { 2024-03-23T23:13:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:13:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:13:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:13:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupsnap_latency": { 2024-03-23T23:13:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:13:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:13:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:13:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lssnap_latency": { 2024-03-23T23:13:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:13:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:13:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:13:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mkdir_latency": { 2024-03-23T23:13:58.897 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 8, 2024-03-23T23:13:58.897 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.014797146, 2024-03-23T23:13:58.897 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001849643 2024-03-23T23:13:58.897 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.897 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mknod_latency": { 2024-03-23T23:13:58.897 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:13:58.897 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:13:58.897 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:13:58.897 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.897 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mksnap_latency": { 2024-03-23T23:13:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:13:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:13:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:13:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: "req_open_latency": { 2024-03-23T23:13:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:13:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:13:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:13:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: "req_readdir_latency": { 2024-03-23T23:13:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2, 2024-03-23T23:13:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000914273, 2024-03-23T23:13:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000457136 2024-03-23T23:13:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.899 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rename_latency": { 2024-03-23T23:13:58.899 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2, 2024-03-23T23:13:58.899 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.002956645, 2024-03-23T23:13:58.899 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001478322 2024-03-23T23:13:58.899 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.899 INFO:teuthology.orchestra.run.smithi173.stdout: "req_renamesnap_latency": { 2024-03-23T23:13:58.899 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:13:58.899 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:13:58.899 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:13:58.899 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.899 INFO:teuthology.orchestra.run.smithi173.stdout: "req_snapdiff_latency": { 2024-03-23T23:13:58.899 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:13:58.899 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:13:58.899 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:13:58.899 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmdir_latency": { 2024-03-23T23:13:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2, 2024-03-23T23:13:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.012828128, 2024-03-23T23:13:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.006414064 2024-03-23T23:13:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmsnap_latency": { 2024-03-23T23:13:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:13:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:13:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:13:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmxattr_latency": { 2024-03-23T23:13:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:13:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:13:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:13:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setattr_latency": { 2024-03-23T23:13:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 600, 2024-03-23T23:13:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 2.824576129, 2024-03-23T23:13:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.004707626 2024-03-23T23:13:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setdirlayout_latency": { 2024-03-23T23:13:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:13:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:13:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:13:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setfilelock_latency": { 2024-03-23T23:13:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:13:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:13:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:13:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setlayout_latency": { 2024-03-23T23:13:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:13:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:13:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:13:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setxattr_latency": { 2024-03-23T23:13:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 7, 2024-03-23T23:13:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.018845889, 2024-03-23T23:13:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.002692269 2024-03-23T23:13:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: "req_symlink_latency": { 2024-03-23T23:13:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:13:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:13:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:13:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: "req_unlink_latency": { 2024-03-23T23:13:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1, 2024-03-23T23:13:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.004306899, 2024-03-23T23:13:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.004306899 2024-03-23T23:13:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_revoke_eviction": 0, 2024-03-23T23:13:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_acquisition_throttle": 0, 2024-03-23T23:13:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getvxattr_latency": { 2024-03-23T23:13:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:13:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:13:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:13:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:13:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_sessions": { 2024-03-23T23:13:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: "session_count": 1, 2024-03-23T23:13:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: "session_add": 2, 2024-03-23T23:13:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: "session_remove": 1, 2024-03-23T23:13:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_open": 1, 2024-03-23T23:13:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_stale": 0, 2024-03-23T23:13:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: "total_load": 89, 2024-03-23T23:13:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: "average_load": 89, 2024-03-23T23:13:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: "avg_session_uptime": 727, 2024-03-23T23:13:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: "mdthresh_evicted": 0 2024-03-23T23:13:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2024-03-23T23:13:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2024-03-23T23:13:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2024-03-23T23:13:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2024-03-23T23:13:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2024-03-23T23:13:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2024-03-23T23:13:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2024-03-23T23:13:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-23T23:13:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2024-03-23T23:13:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-23T23:13:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2024-03-23T23:13:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2024-03-23T23:13:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2024-03-23T23:13:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-23T23:13:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2024-03-23T23:13:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2024-03-23T23:13:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2024-03-23T23:13:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2024-03-23T23:13:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2024-03-23T23:13:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-23T23:13:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2024-03-23T23:13:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-23T23:13:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2024-03-23T23:13:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2024-03-23T23:13:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2024-03-23T23:13:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2024-03-23T23:13:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2024-03-23T23:13:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-23T23:13:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2024-03-23T23:13:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2024-03-23T23:13:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2024-03-23T23:13:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2024-03-23T23:13:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2024-03-23T23:13:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2024-03-23T23:13:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2024-03-23T23:13:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2024-03-23T23:13:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2024-03-23T23:13:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 299488, 2024-03-23T23:13:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 120, 2024-03-23T23:13:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 0, 2024-03-23T23:13:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 0, 2024-03-23T23:13:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2024-03-23T23:13:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2024-03-23T23:13:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2024-03-23T23:13:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2024-03-23T23:13:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2024-03-23T23:13:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2024-03-23T23:13:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 7536, 2024-03-23T23:13:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 159, 2024-03-23T23:13:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2024-03-23T23:13:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2024-03-23T23:13:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2024-03-23T23:13:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2024-03-23T23:13:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 97362, 2024-03-23T23:13:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 713, 2024-03-23T23:13:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2024-03-23T23:13:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2024-03-23T23:13:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2024-03-23T23:13:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2024-03-23T23:13:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: "objecter": { 2024-03-23T23:13:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: "op_active": 0, 2024-03-23T23:13:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: "op_laggy": 0, 2024-03-23T23:13:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send": 4546, 2024-03-23T23:13:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send_bytes": 8246145, 2024-03-23T23:13:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: "op_resend": 31, 2024-03-23T23:13:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: "op_reply": 4546, 2024-03-23T23:13:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: "op_latency": { 2024-03-23T23:13:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 4546, 2024-03-23T23:13:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 29.404352795, 2024-03-23T23:13:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.006468181 2024-03-23T23:13:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: "op_inflight": 0, 2024-03-23T23:13:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: "oplen_avg": { 2024-03-23T23:13:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 4546, 2024-03-23T23:13:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 5284 2024-03-23T23:13:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: "op": 4546, 2024-03-23T23:13:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: "op_r": 347, 2024-03-23T23:13:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: "op_w": 4199, 2024-03-23T23:13:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: "op_rmw": 0, 2024-03-23T23:13:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: "op_pg": 0, 2024-03-23T23:13:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_stat": 329, 2024-03-23T23:13:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_create": 25, 2024-03-23T23:13:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_read": 23, 2024-03-23T23:13:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_write": 2423, 2024-03-23T23:13:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writefull": 169, 2024-03-23T23:13:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writesame": 0, 2024-03-23T23:13:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_append": 0, 2024-03-23T23:13:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_zero": 0, 2024-03-23T23:13:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_truncate": 0, 2024-03-23T23:13:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_delete": 516, 2024-03-23T23:13:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_mapext": 0, 2024-03-23T23:13:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_sparse_read": 0, 2024-03-23T23:13:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_clonerange": 0, 2024-03-23T23:13:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_getxattr": 324, 2024-03-23T23:13:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_setxattr": 50, 2024-03-23T23:13:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_cmpxattr": 0, 2024-03-23T23:13:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_rmxattr": 0, 2024-03-23T23:13:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_resetxattrs": 0, 2024-03-23T23:13:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_call": 0, 2024-03-23T23:13:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_watch": 0, 2024-03-23T23:13:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_notify": 0, 2024-03-23T23:13:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_src_cmpxattr": 0, 2024-03-23T23:13:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls": 0, 2024-03-23T23:13:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls_filter": 0, 2024-03-23T23:13:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_other": 599, 2024-03-23T23:13:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_active": 0, 2024-03-23T23:13:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_send": 0, 2024-03-23T23:13:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_resend": 0, 2024-03-23T23:13:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_ping": 0, 2024-03-23T23:13:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_active": 0, 2024-03-23T23:13:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_send": 0, 2024-03-23T23:13:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_resend": 0, 2024-03-23T23:13:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_active": 0, 2024-03-23T23:13:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_send": 0, 2024-03-23T23:13:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_resend": 0, 2024-03-23T23:13:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_active": 0, 2024-03-23T23:13:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_send": 0, 2024-03-23T23:13:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_resend": 0, 2024-03-23T23:13:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: "command_active": 0, 2024-03-23T23:13:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: "command_send": 0, 2024-03-23T23:13:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: "command_resend": 0, 2024-03-23T23:13:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: "map_epoch": 88, 2024-03-23T23:13:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: "map_full": 0, 2024-03-23T23:13:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: "map_inc": 3, 2024-03-23T23:13:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_sessions": 12, 2024-03-23T23:13:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_open": 12, 2024-03-23T23:13:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_close": 0, 2024-03-23T23:13:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_laggy": 0, 2024-03-23T23:13:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_wr": 815, 2024-03-23T23:13:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_rd": 0, 2024-03-23T23:13:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_del": 11 2024-03-23T23:13:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: "oft": { 2024-03-23T23:13:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_objs": 1, 2024-03-23T23:13:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_kv_pairs": 0, 2024-03-23T23:13:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_updates": 0, 2024-03-23T23:13:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_removes": 0 2024-03-23T23:13:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: "osd-slow-ops": { 2024-03-23T23:13:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_ops_count": 0 2024-03-23T23:13:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: "purge_queue": { 2024-03-23T23:13:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops": 0, 2024-03-23T23:13:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops_high_water": 2, 2024-03-23T23:13:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing": 0, 2024-03-23T23:13:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_high_water": 2, 2024-03-23T23:13:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed_ops": 2, 2024-03-23T23:13:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed": 2, 2024-03-23T23:13:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_item_in_journal": 0 2024-03-23T23:13:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-23T23:13:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-23T23:13:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-03-23T23:13:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-23T23:13:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 8907, 2024-03-23T23:13:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 2179569, 2024-03-23T23:13:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-23T23:13:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 8907, 2024-03-23T23:13:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-03-23T23:13:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-03-23T23:13:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 8907, 2024-03-23T23:13:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 2179569, 2024-03-23T23:13:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-23T23:13:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:13:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:13:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:13:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:13:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_bytes": { 2024-03-23T23:13:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-23T23:13:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-03-23T23:13:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-23T23:13:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-03-23T23:13:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-03-23T23:13:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-23T23:13:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-03-23T23:13:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 4546, 2024-03-23T23:13:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 8246347, 2024-03-23T23:13:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 4546, 2024-03-23T23:13:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 8246347, 2024-03-23T23:13:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-23T23:13:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:13:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:13:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:13:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:13:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_ops": { 2024-03-23T23:13:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-23T23:13:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 1024, 2024-03-23T23:13:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-23T23:13:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-03-23T23:13:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-03-23T23:13:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-23T23:13:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-03-23T23:13:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 4546, 2024-03-23T23:13:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 4546, 2024-03-23T23:13:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 4546, 2024-03-23T23:13:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 4546, 2024-03-23T23:13:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-23T23:13:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:13:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:13:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:13:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:13:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle": { 2024-03-23T23:13:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-23T23:13:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-03-23T23:13:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-23T23:13:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 2, 2024-03-23T23:13:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 202, 2024-03-23T23:13:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-23T23:13:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 2, 2024-03-23T23:13:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-03-23T23:13:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-03-23T23:13:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 1, 2024-03-23T23:13:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 202, 2024-03-23T23:13:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-23T23:13:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:13:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:13:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:13:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:13:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:13:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle-0x555ade9f0500": { 2024-03-23T23:13:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-23T23:13:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-03-23T23:13:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-23T23:13:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 2437, 2024-03-23T23:13:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 7884287, 2024-03-23T23:13:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-23T23:13:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 2437, 2024-03-23T23:13:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-03-23T23:13:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-03-23T23:13:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 2422, 2024-03-23T23:13:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 7884287, 2024-03-23T23:13:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-23T23:13:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:13:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:13:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:13:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:13:58.920 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:13:58.920 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-23T23:13:59.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:13:58 smithi173 ceph-mon[36116]: pgmap v755: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-23T23:13:59.394 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:13:58 smithi012 ceph-mon[31695]: pgmap v755: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-23T23:13:59.394 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:13:58 smithi203 ceph-mon[36693]: pgmap v755: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-23T23:13:59.764 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-03-23T23:13:59.764 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-03-23T23:13:59.765 DEBUG:tasks.check_counter:Getting stats from e 2024-03-23T23:13:59.765 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-23T23:14:00.046 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.b/config 2024-03-23T23:14:01.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:00 smithi173 ceph-mon[36116]: pgmap v756: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-23T23:14:01.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:00 smithi203 ceph-mon[36693]: pgmap v756: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-23T23:14:01.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:00 smithi012 ceph-mon[31695]: pgmap v756: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-23T23:14:01.736 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-23T23:14:01.736 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2024-03-23T23:14:01.736 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 1121, 2024-03-23T23:14:01.736 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 1117, 2024-03-23T23:14:01.736 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 793904, 2024-03-23T23:14:01.736 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 422614, 2024-03-23T23:14:01.736 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 5, 2024-03-23T23:14:01.736 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 1, 2024-03-23T23:14:01.737 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.088557527, 2024-03-23T23:14:01.737 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.031170765, 2024-03-23T23:14:01.737 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.027519512, 2024-03-23T23:14:01.737 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.024340692, 2024-03-23T23:14:01.737 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:14:01.737 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1116, 2024-03-23T23:14:01.737 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.032513443, 2024-03-23T23:14:01.737 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000029133 2024-03-23T23:14:01.737 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.737 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:14:01.737 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.737 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.737 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.737 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.738 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 501920, 2024-03-23T23:14:01.738 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 146272 2024-03-23T23:14:01.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.738 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2024-03-23T23:14:01.738 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 4, 2024-03-23T23:14:01.738 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 188, 2024-03-23T23:14:01.738 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 876, 2024-03-23T23:14:01.738 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 563162, 2024-03-23T23:14:01.738 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 6, 2024-03-23T23:14:01.738 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 3, 2024-03-23T23:14:01.738 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.008663016, 2024-03-23T23:14:01.738 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.005894020, 2024-03-23T23:14:01.738 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.001465574, 2024-03-23T23:14:01.738 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.000041887, 2024-03-23T23:14:01.739 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:14:01.739 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 186, 2024-03-23T23:14:01.739 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.010551007, 2024-03-23T23:14:01.739 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000056725 2024-03-23T23:14:01.739 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.739 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:14:01.739 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.739 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.739 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.739 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.739 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-23T23:14:01.739 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 562464 2024-03-23T23:14:01.739 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.739 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2024-03-23T23:14:01.740 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 1610, 2024-03-23T23:14:01.740 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 1610, 2024-03-23T23:14:01.740 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 413511, 2024-03-23T23:14:01.740 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 511163, 2024-03-23T23:14:01.740 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 4, 2024-03-23T23:14:01.740 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 3, 2024-03-23T23:14:01.740 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.072238561, 2024-03-23T23:14:01.740 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.018922253, 2024-03-23T23:14:01.740 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.018559963, 2024-03-23T23:14:01.740 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.030186658, 2024-03-23T23:14:01.740 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:14:01.740 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1609, 2024-03-23T23:14:01.740 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.029200698, 2024-03-23T23:14:01.740 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000018148 2024-03-23T23:14:01.741 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.741 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:14:01.741 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.741 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.741 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.741 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.741 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-23T23:14:01.741 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 448 2024-03-23T23:14:01.741 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.741 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2024-03-23T23:14:01.741 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 1, 2024-03-23T23:14:01.741 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2024-03-23T23:14:01.741 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.741 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-MDSRank": { 2024-03-23T23:14:01.741 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-03-23T23:14:01.742 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-03-23T23:14:01.742 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 4212, 2024-03-23T23:14:01.742 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.080551141, 2024-03-23T23:14:01.742 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000019124 2024-03-23T23:14:01.742 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:14:01.742 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.742 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-PurgeQueue": { 2024-03-23T23:14:01.742 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-03-23T23:14:01.742 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-03-23T23:14:01.742 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.742 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.742 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.742 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:14:01.742 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.742 INFO:teuthology.orchestra.run.smithi173.stdout: "mds": { 2024-03-23T23:14:01.742 INFO:teuthology.orchestra.run.smithi173.stdout: "request": 0, 2024-03-23T23:14:01.743 INFO:teuthology.orchestra.run.smithi173.stdout: "reply": 0, 2024-03-23T23:14:01.743 INFO:teuthology.orchestra.run.smithi173.stdout: "reply_latency": { 2024-03-23T23:14:01.743 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.743 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.743 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.743 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.743 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_reply": 0, 2024-03-23T23:14:01.743 INFO:teuthology.orchestra.run.smithi173.stdout: "forward": 0, 2024-03-23T23:14:01.743 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_complete": 0, 2024-03-23T23:14:01.743 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_keys": 0, 2024-03-23T23:14:01.743 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_commit": 0, 2024-03-23T23:14:01.743 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_split": 0, 2024-03-23T23:14:01.743 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_merge": 0, 2024-03-23T23:14:01.743 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes": 0, 2024-03-23T23:14:01.743 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_top": 0, 2024-03-23T23:14:01.743 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_bottom": 0, 2024-03-23T23:14:01.743 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pin_tail": 0, 2024-03-23T23:14:01.744 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pinned": 0, 2024-03-23T23:14:01.744 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_expired": 0, 2024-03-23T23:14:01.744 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_with_caps": 0, 2024-03-23T23:14:01.744 INFO:teuthology.orchestra.run.smithi173.stdout: "caps": 0, 2024-03-23T23:14:01.744 INFO:teuthology.orchestra.run.smithi173.stdout: "subtrees": 1, 2024-03-23T23:14:01.744 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse": 0, 2024-03-23T23:14:01.744 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_hit": 0, 2024-03-23T23:14:01.744 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_forward": 0, 2024-03-23T23:14:01.744 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_discover": 0, 2024-03-23T23:14:01.744 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_dir_fetch": 0, 2024-03-23T23:14:01.744 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_remote_ino": 0, 2024-03-23T23:14:01.744 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_lock": 0, 2024-03-23T23:14:01.744 INFO:teuthology.orchestra.run.smithi173.stdout: "load_cent": 0, 2024-03-23T23:14:01.744 INFO:teuthology.orchestra.run.smithi173.stdout: "q": 0, 2024-03-23T23:14:01.744 INFO:teuthology.orchestra.run.smithi173.stdout: "exported": 0, 2024-03-23T23:14:01.745 INFO:teuthology.orchestra.run.smithi173.stdout: "exported_inodes": 0, 2024-03-23T23:14:01.745 INFO:teuthology.orchestra.run.smithi173.stdout: "imported": 0, 2024-03-23T23:14:01.745 INFO:teuthology.orchestra.run.smithi173.stdout: "imported_inodes": 0, 2024-03-23T23:14:01.745 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_dir_fetch": 0, 2024-03-23T23:14:01.745 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_backtrace_fetch": 0, 2024-03-23T23:14:01.745 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_peer_discover": 0, 2024-03-23T23:14:01.745 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rfiles": 0, 2024-03-23T23:14:01.745 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rbytes": 0, 2024-03-23T23:14:01.745 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rsnaps": 0, 2024-03-23T23:14:01.745 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_fetch": 0, 2024-03-23T23:14:01.745 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_set_tag": 0, 2024-03-23T23:14:01.745 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_repaired": 0, 2024-03-23T23:14:01.745 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_inotable_repaired": 0, 2024-03-23T23:14:01.745 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_inodes": 0, 2024-03-23T23:14:01.745 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_base_inodes": 0, 2024-03-23T23:14:01.745 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dirfrag_rstats": 0, 2024-03-23T23:14:01.745 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_file_inodes": 0, 2024-03-23T23:14:01.746 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_inode_file_caps": 0, 2024-03-23T23:14:01.746 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_revoke": 0, 2024-03-23T23:14:01.746 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_grant": 0, 2024-03-23T23:14:01.746 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_trunc": 0, 2024-03-23T23:14:01.746 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-23T23:14:01.746 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-23T23:14:01.746 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps": 0, 2024-03-23T23:14:01.746 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps_dirty": 0, 2024-03-23T23:14:01.746 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_cap_release": 0, 2024-03-23T23:14:01.746 INFO:teuthology.orchestra.run.smithi173.stdout: "process_request_cap_release": 0 2024-03-23T23:14:01.746 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.746 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_cache": { 2024-03-23T23:14:01.746 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update": 0, 2024-03-23T23:14:01.746 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update_receipt": 0, 2024-03-23T23:14:01.746 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_try_discover": 0, 2024-03-23T23:14:01.746 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_send_discover": 0, 2024-03-23T23:14:01.747 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_handle_discover": 0, 2024-03-23T23:14:01.747 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays": 0, 2024-03-23T23:14:01.747 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_delayed": 0, 2024-03-23T23:14:01.747 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_enqueuing": 0, 2024-03-23T23:14:01.747 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_created": 0, 2024-03-23T23:14:01.747 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_enqueued": 0, 2024-03-23T23:14:01.747 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_reintegrated": 0, 2024-03-23T23:14:01.747 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_migrated": 0, 2024-03-23T23:14:01.747 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_processing": 0, 2024-03-23T23:14:01.747 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_enqueued": 0, 2024-03-23T23:14:01.747 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_prioritized": 0, 2024-03-23T23:14:01.747 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_started": 0, 2024-03-23T23:14:01.747 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_completed": 0, 2024-03-23T23:14:01.747 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_enqueue_scrub": 0, 2024-03-23T23:14:01.747 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_exportdir": 0, 2024-03-23T23:14:01.747 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_flush": 0, 2024-03-23T23:14:01.748 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragmentdir": 0, 2024-03-23T23:14:01.748 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragstats": 0, 2024-03-23T23:14:01.748 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_inodestats": 0 2024-03-23T23:14:01.748 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.748 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_log": { 2024-03-23T23:14:01.748 INFO:teuthology.orchestra.run.smithi173.stdout: "evlrg": 0, 2024-03-23T23:14:01.748 INFO:teuthology.orchestra.run.smithi173.stdout: "evadd": 0, 2024-03-23T23:14:01.748 INFO:teuthology.orchestra.run.smithi173.stdout: "evex": 0, 2024-03-23T23:14:01.748 INFO:teuthology.orchestra.run.smithi173.stdout: "evtrm": 0, 2024-03-23T23:14:01.748 INFO:teuthology.orchestra.run.smithi173.stdout: "ev": 3, 2024-03-23T23:14:01.748 INFO:teuthology.orchestra.run.smithi173.stdout: "evexg": 0, 2024-03-23T23:14:01.748 INFO:teuthology.orchestra.run.smithi173.stdout: "evexd": 0, 2024-03-23T23:14:01.748 INFO:teuthology.orchestra.run.smithi173.stdout: "segadd": 0, 2024-03-23T23:14:01.749 INFO:teuthology.orchestra.run.smithi173.stdout: "segex": 0, 2024-03-23T23:14:01.749 INFO:teuthology.orchestra.run.smithi173.stdout: "segtrm": 0, 2024-03-23T23:14:01.749 INFO:teuthology.orchestra.run.smithi173.stdout: "seg": 2, 2024-03-23T23:14:01.749 INFO:teuthology.orchestra.run.smithi173.stdout: "segmjr": 2, 2024-03-23T23:14:01.749 INFO:teuthology.orchestra.run.smithi173.stdout: "segexg": 0, 2024-03-23T23:14:01.749 INFO:teuthology.orchestra.run.smithi173.stdout: "segexd": 0, 2024-03-23T23:14:01.749 INFO:teuthology.orchestra.run.smithi173.stdout: "expos": 4194304, 2024-03-23T23:14:01.749 INFO:teuthology.orchestra.run.smithi173.stdout: "wrpos": 4195198, 2024-03-23T23:14:01.749 INFO:teuthology.orchestra.run.smithi173.stdout: "rdpos": 4194892, 2024-03-23T23:14:01.749 INFO:teuthology.orchestra.run.smithi173.stdout: "jlat": { 2024-03-23T23:14:01.749 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.749 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.749 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.749 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.749 INFO:teuthology.orchestra.run.smithi173.stdout: "replayed": 3 2024-03-23T23:14:01.749 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.749 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_mem": { 2024-03-23T23:14:01.750 INFO:teuthology.orchestra.run.smithi173.stdout: "ino": 3, 2024-03-23T23:14:01.750 INFO:teuthology.orchestra.run.smithi173.stdout: "ino+": 3, 2024-03-23T23:14:01.750 INFO:teuthology.orchestra.run.smithi173.stdout: "ino-": 0, 2024-03-23T23:14:01.750 INFO:teuthology.orchestra.run.smithi173.stdout: "dir": 1, 2024-03-23T23:14:01.750 INFO:teuthology.orchestra.run.smithi173.stdout: "dir+": 1, 2024-03-23T23:14:01.750 INFO:teuthology.orchestra.run.smithi173.stdout: "dir-": 0, 2024-03-23T23:14:01.750 INFO:teuthology.orchestra.run.smithi173.stdout: "dn": 0, 2024-03-23T23:14:01.750 INFO:teuthology.orchestra.run.smithi173.stdout: "dn+": 0, 2024-03-23T23:14:01.750 INFO:teuthology.orchestra.run.smithi173.stdout: "dn-": 0, 2024-03-23T23:14:01.750 INFO:teuthology.orchestra.run.smithi173.stdout: "cap": 0, 2024-03-23T23:14:01.750 INFO:teuthology.orchestra.run.smithi173.stdout: "cap+": 0, 2024-03-23T23:14:01.750 INFO:teuthology.orchestra.run.smithi173.stdout: "cap-": 0, 2024-03-23T23:14:01.750 INFO:teuthology.orchestra.run.smithi173.stdout: "rss": 49620, 2024-03-23T23:14:01.750 INFO:teuthology.orchestra.run.smithi173.stdout: "heap": 215420 2024-03-23T23:14:01.750 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.750 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_server": { 2024-03-23T23:14:01.751 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_client_request": 0, 2024-03-23T23:14:01.751 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_server_request": 0, 2024-03-23T23:14:01.751 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_request": 0, 2024-03-23T23:14:01.751 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_session": 0, 2024-03-23T23:14:01.751 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_peer_request": 0, 2024-03-23T23:14:01.751 INFO:teuthology.orchestra.run.smithi173.stdout: "req_create_latency": { 2024-03-23T23:14:01.751 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.751 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.751 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.751 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.751 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getattr_latency": { 2024-03-23T23:14:01.751 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.751 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.751 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.751 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.751 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getfilelock_latency": { 2024-03-23T23:14:01.751 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.752 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.752 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.752 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.752 INFO:teuthology.orchestra.run.smithi173.stdout: "req_link_latency": { 2024-03-23T23:14:01.752 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.752 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.752 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.752 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.752 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookup_latency": { 2024-03-23T23:14:01.752 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.752 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.752 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.752 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.752 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookuphash_latency": { 2024-03-23T23:14:01.752 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.752 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.752 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.753 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.753 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupino_latency": { 2024-03-23T23:14:01.753 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.753 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.753 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.753 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.753 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupname_latency": { 2024-03-23T23:14:01.753 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.753 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.753 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.753 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.753 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupparent_latency": { 2024-03-23T23:14:01.753 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.753 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.753 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.754 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.754 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupsnap_latency": { 2024-03-23T23:14:01.754 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.754 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.754 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.754 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.754 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lssnap_latency": { 2024-03-23T23:14:01.754 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.754 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.754 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.754 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.754 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mkdir_latency": { 2024-03-23T23:14:01.754 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.754 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.754 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.754 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.755 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mknod_latency": { 2024-03-23T23:14:01.755 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.755 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.755 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.755 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.755 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mksnap_latency": { 2024-03-23T23:14:01.755 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.756 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.756 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.756 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.756 INFO:teuthology.orchestra.run.smithi173.stdout: "req_open_latency": { 2024-03-23T23:14:01.756 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.756 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.756 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.756 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.756 INFO:teuthology.orchestra.run.smithi173.stdout: "req_readdir_latency": { 2024-03-23T23:14:01.756 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.756 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.756 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.756 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.756 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rename_latency": { 2024-03-23T23:14:01.756 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.757 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.757 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.757 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.757 INFO:teuthology.orchestra.run.smithi173.stdout: "req_renamesnap_latency": { 2024-03-23T23:14:01.757 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.757 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.757 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.757 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.757 INFO:teuthology.orchestra.run.smithi173.stdout: "req_snapdiff_latency": { 2024-03-23T23:14:01.757 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.757 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.757 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.757 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.757 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmdir_latency": { 2024-03-23T23:14:01.757 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.757 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.757 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.758 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.758 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmsnap_latency": { 2024-03-23T23:14:01.758 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.758 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.758 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.758 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.758 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmxattr_latency": { 2024-03-23T23:14:01.758 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.758 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.758 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.758 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.758 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setattr_latency": { 2024-03-23T23:14:01.758 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.758 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.758 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.758 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.759 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setdirlayout_latency": { 2024-03-23T23:14:01.759 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.759 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.759 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.759 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.759 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setfilelock_latency": { 2024-03-23T23:14:01.759 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.759 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.759 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.759 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.759 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setlayout_latency": { 2024-03-23T23:14:01.759 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.759 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.759 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.759 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.759 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setxattr_latency": { 2024-03-23T23:14:01.760 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.760 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.760 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.760 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.760 INFO:teuthology.orchestra.run.smithi173.stdout: "req_symlink_latency": { 2024-03-23T23:14:01.760 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.760 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.760 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.760 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.760 INFO:teuthology.orchestra.run.smithi173.stdout: "req_unlink_latency": { 2024-03-23T23:14:01.760 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.760 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.760 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.760 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.760 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_revoke_eviction": 0, 2024-03-23T23:14:01.760 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_acquisition_throttle": 0, 2024-03-23T23:14:01.760 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getvxattr_latency": { 2024-03-23T23:14:01.761 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.761 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.761 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.761 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:14:01.761 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.761 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_sessions": { 2024-03-23T23:14:01.761 INFO:teuthology.orchestra.run.smithi173.stdout: "session_count": 1, 2024-03-23T23:14:01.761 INFO:teuthology.orchestra.run.smithi173.stdout: "session_add": 1, 2024-03-23T23:14:01.761 INFO:teuthology.orchestra.run.smithi173.stdout: "session_remove": 0, 2024-03-23T23:14:01.761 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_open": 1, 2024-03-23T23:14:01.761 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_stale": 0, 2024-03-23T23:14:01.761 INFO:teuthology.orchestra.run.smithi173.stdout: "total_load": 0, 2024-03-23T23:14:01.761 INFO:teuthology.orchestra.run.smithi173.stdout: "average_load": 0, 2024-03-23T23:14:01.761 INFO:teuthology.orchestra.run.smithi173.stdout: "avg_session_uptime": 2280, 2024-03-23T23:14:01.761 INFO:teuthology.orchestra.run.smithi173.stdout: "mdthresh_evicted": 0 2024-03-23T23:14:01.762 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.762 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2024-03-23T23:14:01.762 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2024-03-23T23:14:01.762 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2024-03-23T23:14:01.762 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2024-03-23T23:14:01.762 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2024-03-23T23:14:01.762 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2024-03-23T23:14:01.762 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2024-03-23T23:14:01.762 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-23T23:14:01.762 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2024-03-23T23:14:01.762 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-23T23:14:01.762 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2024-03-23T23:14:01.762 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2024-03-23T23:14:01.762 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2024-03-23T23:14:01.762 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-23T23:14:01.763 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2024-03-23T23:14:01.763 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2024-03-23T23:14:01.763 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2024-03-23T23:14:01.763 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2024-03-23T23:14:01.763 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2024-03-23T23:14:01.763 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-23T23:14:01.763 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2024-03-23T23:14:01.763 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-23T23:14:01.763 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2024-03-23T23:14:01.763 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2024-03-23T23:14:01.763 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2024-03-23T23:14:01.763 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2024-03-23T23:14:01.763 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2024-03-23T23:14:01.763 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-23T23:14:01.763 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2024-03-23T23:14:01.763 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2024-03-23T23:14:01.763 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2024-03-23T23:14:01.764 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2024-03-23T23:14:01.764 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2024-03-23T23:14:01.764 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2024-03-23T23:14:01.764 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2024-03-23T23:14:01.764 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2024-03-23T23:14:01.764 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2024-03-23T23:14:01.764 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 181281, 2024-03-23T23:14:01.764 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 46, 2024-03-23T23:14:01.764 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 0, 2024-03-23T23:14:01.764 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 0, 2024-03-23T23:14:01.764 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2024-03-23T23:14:01.764 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2024-03-23T23:14:01.764 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2024-03-23T23:14:01.764 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2024-03-23T23:14:01.764 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2024-03-23T23:14:01.764 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2024-03-23T23:14:01.764 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 7536, 2024-03-23T23:14:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 159, 2024-03-23T23:14:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2024-03-23T23:14:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2024-03-23T23:14:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2024-03-23T23:14:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2024-03-23T23:14:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 8320, 2024-03-23T23:14:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 13, 2024-03-23T23:14:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2024-03-23T23:14:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2024-03-23T23:14:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2024-03-23T23:14:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2024-03-23T23:14:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: "objecter": { 2024-03-23T23:14:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: "op_active": 0, 2024-03-23T23:14:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: "op_laggy": 0, 2024-03-23T23:14:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send": 2477, 2024-03-23T23:14:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send_bytes": 21, 2024-03-23T23:14:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: "op_resend": 3, 2024-03-23T23:14:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: "op_reply": 2477, 2024-03-23T23:14:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: "op_latency": { 2024-03-23T23:14:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2477, 2024-03-23T23:14:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 1.060977151, 2024-03-23T23:14:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000428331 2024-03-23T23:14:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: "op_inflight": 0, 2024-03-23T23:14:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: "oplen_avg": { 2024-03-23T23:14:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2477, 2024-03-23T23:14:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 2478 2024-03-23T23:14:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: "op": 2477, 2024-03-23T23:14:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: "op_r": 2477, 2024-03-23T23:14:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: "op_w": 0, 2024-03-23T23:14:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: "op_rmw": 0, 2024-03-23T23:14:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: "op_pg": 0, 2024-03-23T23:14:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_stat": 1601, 2024-03-23T23:14:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_create": 0, 2024-03-23T23:14:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_read": 874, 2024-03-23T23:14:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_write": 0, 2024-03-23T23:14:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writefull": 0, 2024-03-23T23:14:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writesame": 0, 2024-03-23T23:14:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_append": 0, 2024-03-23T23:14:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_zero": 0, 2024-03-23T23:14:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_truncate": 0, 2024-03-23T23:14:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_delete": 0, 2024-03-23T23:14:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_mapext": 0, 2024-03-23T23:14:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_sparse_read": 0, 2024-03-23T23:14:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_clonerange": 0, 2024-03-23T23:14:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_getxattr": 1, 2024-03-23T23:14:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_setxattr": 0, 2024-03-23T23:14:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_cmpxattr": 0, 2024-03-23T23:14:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_rmxattr": 0, 2024-03-23T23:14:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_resetxattrs": 0, 2024-03-23T23:14:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_call": 0, 2024-03-23T23:14:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_watch": 0, 2024-03-23T23:14:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_notify": 0, 2024-03-23T23:14:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_src_cmpxattr": 0, 2024-03-23T23:14:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls": 0, 2024-03-23T23:14:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls_filter": 0, 2024-03-23T23:14:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_other": 0, 2024-03-23T23:14:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_active": 0, 2024-03-23T23:14:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_send": 0, 2024-03-23T23:14:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_resend": 0, 2024-03-23T23:14:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_ping": 0, 2024-03-23T23:14:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_active": 0, 2024-03-23T23:14:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_send": 0, 2024-03-23T23:14:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_resend": 0, 2024-03-23T23:14:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_active": 0, 2024-03-23T23:14:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_send": 0, 2024-03-23T23:14:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_resend": 0, 2024-03-23T23:14:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_active": 0, 2024-03-23T23:14:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_send": 0, 2024-03-23T23:14:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_resend": 0, 2024-03-23T23:14:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: "command_active": 0, 2024-03-23T23:14:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: "command_send": 0, 2024-03-23T23:14:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: "command_resend": 0, 2024-03-23T23:14:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: "map_epoch": 88, 2024-03-23T23:14:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: "map_full": 0, 2024-03-23T23:14:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: "map_inc": 2, 2024-03-23T23:14:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_sessions": 4, 2024-03-23T23:14:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_open": 4, 2024-03-23T23:14:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_close": 0, 2024-03-23T23:14:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_laggy": 0, 2024-03-23T23:14:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_wr": 0, 2024-03-23T23:14:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_rd": 2, 2024-03-23T23:14:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_del": 0 2024-03-23T23:14:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: "oft": { 2024-03-23T23:14:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_objs": 0, 2024-03-23T23:14:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_kv_pairs": 0, 2024-03-23T23:14:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_updates": 0, 2024-03-23T23:14:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_removes": 0 2024-03-23T23:14:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: "osd-slow-ops": { 2024-03-23T23:14:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_ops_count": 0 2024-03-23T23:14:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: "purge_queue": { 2024-03-23T23:14:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops": 0, 2024-03-23T23:14:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops_high_water": 0, 2024-03-23T23:14:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing": 0, 2024-03-23T23:14:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_high_water": 0, 2024-03-23T23:14:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed_ops": 0, 2024-03-23T23:14:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed": 0, 2024-03-23T23:14:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_item_in_journal": 0 2024-03-23T23:14:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-23T23:14:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-23T23:14:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-03-23T23:14:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-23T23:14:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 2731, 2024-03-23T23:14:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 946529, 2024-03-23T23:14:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 2731, 2024-03-23T23:14:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-03-23T23:14:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-03-23T23:14:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 2731, 2024-03-23T23:14:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 946529, 2024-03-23T23:14:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-23T23:14:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:14:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_bytes": { 2024-03-23T23:14:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-23T23:14:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-03-23T23:14:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-23T23:14:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-03-23T23:14:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-03-23T23:14:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-03-23T23:14:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 2477, 2024-03-23T23:14:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 899, 2024-03-23T23:14:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 2477, 2024-03-23T23:14:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 899, 2024-03-23T23:14:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-23T23:14:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:14:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_ops": { 2024-03-23T23:14:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-23T23:14:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 1024, 2024-03-23T23:14:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-23T23:14:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-03-23T23:14:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-03-23T23:14:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-03-23T23:14:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 2477, 2024-03-23T23:14:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 2477, 2024-03-23T23:14:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 2477, 2024-03-23T23:14:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 2477, 2024-03-23T23:14:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-23T23:14:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:14:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle": { 2024-03-23T23:14:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-23T23:14:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-03-23T23:14:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-23T23:14:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-03-23T23:14:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-03-23T23:14:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-03-23T23:14:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-03-23T23:14:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-03-23T23:14:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2024-03-23T23:14:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2024-03-23T23:14:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-23T23:14:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:14:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle-0x55d7a91006e0": { 2024-03-23T23:14:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-23T23:14:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-03-23T23:14:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-23T23:14:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-03-23T23:14:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-03-23T23:14:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-03-23T23:14:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-03-23T23:14:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-03-23T23:14:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2024-03-23T23:14:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2024-03-23T23:14:01.777 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-23T23:14:01.777 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:01.777 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:01.777 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:01.777 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:14:01.777 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:14:01.777 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-23T23:14:02.614 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-23T23:14:02.615 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-23T23:14:02.615 DEBUG:tasks.check_counter:Getting stats from h 2024-03-23T23:14:02.615 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-23T23:14:02.885 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.b/config 2024-03-23T23:14:03.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:02 smithi173 ceph-mon[36116]: pgmap v757: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-23T23:14:03.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:02 smithi203 ceph-mon[36693]: pgmap v757: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-23T23:14:03.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:02 smithi012 ceph-mon[31695]: pgmap v757: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-23T23:14:04.619 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-23T23:14:04.620 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2024-03-23T23:14:04.620 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 1435, 2024-03-23T23:14:04.620 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 984, 2024-03-23T23:14:04.620 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 331421, 2024-03-23T23:14:04.620 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 786042, 2024-03-23T23:14:04.620 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 53, 2024-03-23T23:14:04.620 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 4, 2024-03-23T23:14:04.620 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.095544249, 2024-03-23T23:14:04.620 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.038622013, 2024-03-23T23:14:04.620 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.058767017, 2024-03-23T23:14:04.620 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.003348236, 2024-03-23T23:14:04.620 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:14:04.620 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 982, 2024-03-23T23:14:04.621 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.054645626, 2024-03-23T23:14:04.621 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000055647 2024-03-23T23:14:04.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.621 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:14:04.621 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2251, 2024-03-23T23:14:04.621 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000470180, 2024-03-23T23:14:04.621 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000208 2024-03-23T23:14:04.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.621 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-23T23:14:04.621 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 0 2024-03-23T23:14:04.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.621 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2024-03-23T23:14:04.621 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 34, 2024-03-23T23:14:04.622 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 218, 2024-03-23T23:14:04.622 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 11300, 2024-03-23T23:14:04.622 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 530776, 2024-03-23T23:14:04.622 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 13, 2024-03-23T23:14:04.622 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 4, 2024-03-23T23:14:04.622 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.015400946, 2024-03-23T23:14:04.622 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.007526659, 2024-03-23T23:14:04.622 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.006375235, 2024-03-23T23:14:04.622 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.000193621, 2024-03-23T23:14:04.622 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:14:04.622 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 217, 2024-03-23T23:14:04.622 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.027777338, 2024-03-23T23:14:04.622 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000128006 2024-03-23T23:14:04.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.622 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:14:04.623 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 34, 2024-03-23T23:14:04.623 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000005987, 2024-03-23T23:14:04.623 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000176 2024-03-23T23:14:04.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.623 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-23T23:14:04.623 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 477232 2024-03-23T23:14:04.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.623 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2024-03-23T23:14:04.623 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 349, 2024-03-23T23:14:04.623 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 342, 2024-03-23T23:14:04.623 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 534564, 2024-03-23T23:14:04.623 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 180225, 2024-03-23T23:14:04.623 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 6, 2024-03-23T23:14:04.624 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 5, 2024-03-23T23:14:04.624 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.031293549, 2024-03-23T23:14:04.624 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.010799504, 2024-03-23T23:14:04.624 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.019338916, 2024-03-23T23:14:04.624 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.004283760, 2024-03-23T23:14:04.624 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:14:04.624 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 341, 2024-03-23T23:14:04.624 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.017568161, 2024-03-23T23:14:04.624 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000051519 2024-03-23T23:14:04.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.624 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:14:04.624 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 91, 2024-03-23T23:14:04.624 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000014623, 2024-03-23T23:14:04.624 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000160 2024-03-23T23:14:04.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.625 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 511824, 2024-03-23T23:14:04.625 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 150112 2024-03-23T23:14:04.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.625 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2024-03-23T23:14:04.625 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 1, 2024-03-23T23:14:04.625 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2024-03-23T23:14:04.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.625 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-MDSRank": { 2024-03-23T23:14:04.625 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-03-23T23:14:04.625 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-03-23T23:14:04.625 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 53, 2024-03-23T23:14:04.625 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.001237729, 2024-03-23T23:14:04.625 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000023353 2024-03-23T23:14:04.625 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:14:04.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.626 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-PurgeQueue": { 2024-03-23T23:14:04.626 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-03-23T23:14:04.626 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-03-23T23:14:04.626 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2, 2024-03-23T23:14:04.626 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000023697, 2024-03-23T23:14:04.626 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000011848 2024-03-23T23:14:04.626 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:14:04.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.626 INFO:teuthology.orchestra.run.smithi173.stdout: "mds": { 2024-03-23T23:14:04.626 INFO:teuthology.orchestra.run.smithi173.stdout: "request": 3, 2024-03-23T23:14:04.626 INFO:teuthology.orchestra.run.smithi173.stdout: "reply": 2, 2024-03-23T23:14:04.626 INFO:teuthology.orchestra.run.smithi173.stdout: "reply_latency": { 2024-03-23T23:14:04.626 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2, 2024-03-23T23:14:04.627 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000959095, 2024-03-23T23:14:04.627 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000479547 2024-03-23T23:14:04.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.628 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_reply": 0, 2024-03-23T23:14:04.628 INFO:teuthology.orchestra.run.smithi173.stdout: "forward": 1, 2024-03-23T23:14:04.628 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_complete": 0, 2024-03-23T23:14:04.628 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_keys": 0, 2024-03-23T23:14:04.628 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_commit": 11, 2024-03-23T23:14:04.628 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_split": 0, 2024-03-23T23:14:04.628 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_merge": 0, 2024-03-23T23:14:04.628 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes": 14, 2024-03-23T23:14:04.628 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_top": 0, 2024-03-23T23:14:04.628 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_bottom": 14, 2024-03-23T23:14:04.628 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pin_tail": 0, 2024-03-23T23:14:04.628 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pinned": 14, 2024-03-23T23:14:04.628 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_expired": 0, 2024-03-23T23:14:04.629 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_with_caps": 2, 2024-03-23T23:14:04.629 INFO:teuthology.orchestra.run.smithi173.stdout: "caps": 2, 2024-03-23T23:14:04.629 INFO:teuthology.orchestra.run.smithi173.stdout: "subtrees": 2, 2024-03-23T23:14:04.629 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse": 3, 2024-03-23T23:14:04.629 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_hit": 1, 2024-03-23T23:14:04.629 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_forward": 0, 2024-03-23T23:14:04.629 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_discover": 1, 2024-03-23T23:14:04.629 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_dir_fetch": 0, 2024-03-23T23:14:04.629 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_remote_ino": 0, 2024-03-23T23:14:04.629 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_lock": 0, 2024-03-23T23:14:04.629 INFO:teuthology.orchestra.run.smithi173.stdout: "load_cent": 0, 2024-03-23T23:14:04.629 INFO:teuthology.orchestra.run.smithi173.stdout: "q": 0, 2024-03-23T23:14:04.629 INFO:teuthology.orchestra.run.smithi173.stdout: "exported": 0, 2024-03-23T23:14:04.629 INFO:teuthology.orchestra.run.smithi173.stdout: "exported_inodes": 0, 2024-03-23T23:14:04.629 INFO:teuthology.orchestra.run.smithi173.stdout: "imported": 0, 2024-03-23T23:14:04.630 INFO:teuthology.orchestra.run.smithi173.stdout: "imported_inodes": 0, 2024-03-23T23:14:04.630 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_dir_fetch": 0, 2024-03-23T23:14:04.630 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_backtrace_fetch": 0, 2024-03-23T23:14:04.630 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_peer_discover": 0, 2024-03-23T23:14:04.630 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rfiles": 5, 2024-03-23T23:14:04.630 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rbytes": 298, 2024-03-23T23:14:04.630 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rsnaps": 0, 2024-03-23T23:14:04.630 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_fetch": 0, 2024-03-23T23:14:04.630 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_set_tag": 0, 2024-03-23T23:14:04.630 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_repaired": 0, 2024-03-23T23:14:04.630 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_inotable_repaired": 0, 2024-03-23T23:14:04.630 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_inodes": 0, 2024-03-23T23:14:04.630 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_base_inodes": 0, 2024-03-23T23:14:04.630 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dirfrag_rstats": 0, 2024-03-23T23:14:04.631 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_file_inodes": 0, 2024-03-23T23:14:04.631 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_inode_file_caps": 0, 2024-03-23T23:14:04.631 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_revoke": 2, 2024-03-23T23:14:04.631 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_grant": 3, 2024-03-23T23:14:04.631 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_trunc": 0, 2024-03-23T23:14:04.631 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-23T23:14:04.631 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-23T23:14:04.631 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps": 2, 2024-03-23T23:14:04.631 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps_dirty": 0, 2024-03-23T23:14:04.631 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_cap_release": 0, 2024-03-23T23:14:04.631 INFO:teuthology.orchestra.run.smithi173.stdout: "process_request_cap_release": 1 2024-03-23T23:14:04.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.632 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_cache": { 2024-03-23T23:14:04.632 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update": 0, 2024-03-23T23:14:04.632 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update_receipt": 0, 2024-03-23T23:14:04.632 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_try_discover": 0, 2024-03-23T23:14:04.632 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_send_discover": 2, 2024-03-23T23:14:04.632 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_handle_discover": 0, 2024-03-23T23:14:04.632 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays": 0, 2024-03-23T23:14:04.632 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_delayed": 0, 2024-03-23T23:14:04.632 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_enqueuing": 0, 2024-03-23T23:14:04.632 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_created": 0, 2024-03-23T23:14:04.632 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_enqueued": 0, 2024-03-23T23:14:04.633 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_reintegrated": 0, 2024-03-23T23:14:04.633 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_migrated": 0, 2024-03-23T23:14:04.633 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_processing": 0, 2024-03-23T23:14:04.633 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_enqueued": 0, 2024-03-23T23:14:04.633 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_prioritized": 0, 2024-03-23T23:14:04.633 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_started": 0, 2024-03-23T23:14:04.633 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_completed": 0, 2024-03-23T23:14:04.633 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_enqueue_scrub": 0, 2024-03-23T23:14:04.633 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_exportdir": 0, 2024-03-23T23:14:04.633 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_flush": 0, 2024-03-23T23:14:04.633 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragmentdir": 0, 2024-03-23T23:14:04.633 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragstats": 0, 2024-03-23T23:14:04.633 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_inodestats": 0 2024-03-23T23:14:04.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.634 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_log": { 2024-03-23T23:14:04.634 INFO:teuthology.orchestra.run.smithi173.stdout: "evlrg": 0, 2024-03-23T23:14:04.634 INFO:teuthology.orchestra.run.smithi173.stdout: "evadd": 3, 2024-03-23T23:14:04.634 INFO:teuthology.orchestra.run.smithi173.stdout: "evex": 0, 2024-03-23T23:14:04.634 INFO:teuthology.orchestra.run.smithi173.stdout: "evtrm": 0, 2024-03-23T23:14:04.634 INFO:teuthology.orchestra.run.smithi173.stdout: "ev": 3, 2024-03-23T23:14:04.634 INFO:teuthology.orchestra.run.smithi173.stdout: "evexg": 0, 2024-03-23T23:14:04.634 INFO:teuthology.orchestra.run.smithi173.stdout: "evexd": 0, 2024-03-23T23:14:04.634 INFO:teuthology.orchestra.run.smithi173.stdout: "segadd": 2, 2024-03-23T23:14:04.634 INFO:teuthology.orchestra.run.smithi173.stdout: "segex": 0, 2024-03-23T23:14:04.634 INFO:teuthology.orchestra.run.smithi173.stdout: "segtrm": 0, 2024-03-23T23:14:04.634 INFO:teuthology.orchestra.run.smithi173.stdout: "seg": 2, 2024-03-23T23:14:04.634 INFO:teuthology.orchestra.run.smithi173.stdout: "segmjr": 2, 2024-03-23T23:14:04.634 INFO:teuthology.orchestra.run.smithi173.stdout: "segexg": 0, 2024-03-23T23:14:04.635 INFO:teuthology.orchestra.run.smithi173.stdout: "segexd": 0, 2024-03-23T23:14:04.635 INFO:teuthology.orchestra.run.smithi173.stdout: "expos": 4194304, 2024-03-23T23:14:04.635 INFO:teuthology.orchestra.run.smithi173.stdout: "wrpos": 4195198, 2024-03-23T23:14:04.635 INFO:teuthology.orchestra.run.smithi173.stdout: "rdpos": 0, 2024-03-23T23:14:04.635 INFO:teuthology.orchestra.run.smithi173.stdout: "jlat": { 2024-03-23T23:14:04.635 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2, 2024-03-23T23:14:04.635 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.002901472, 2024-03-23T23:14:04.635 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001450736 2024-03-23T23:14:04.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.635 INFO:teuthology.orchestra.run.smithi173.stdout: "replayed": 0 2024-03-23T23:14:04.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.635 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_mem": { 2024-03-23T23:14:04.635 INFO:teuthology.orchestra.run.smithi173.stdout: "ino": 17, 2024-03-23T23:14:04.635 INFO:teuthology.orchestra.run.smithi173.stdout: "ino+": 17, 2024-03-23T23:14:04.635 INFO:teuthology.orchestra.run.smithi173.stdout: "ino-": 0, 2024-03-23T23:14:04.635 INFO:teuthology.orchestra.run.smithi173.stdout: "dir": 15, 2024-03-23T23:14:04.636 INFO:teuthology.orchestra.run.smithi173.stdout: "dir+": 16, 2024-03-23T23:14:04.636 INFO:teuthology.orchestra.run.smithi173.stdout: "dir-": 1, 2024-03-23T23:14:04.636 INFO:teuthology.orchestra.run.smithi173.stdout: "dn": 14, 2024-03-23T23:14:04.636 INFO:teuthology.orchestra.run.smithi173.stdout: "dn+": 14, 2024-03-23T23:14:04.636 INFO:teuthology.orchestra.run.smithi173.stdout: "dn-": 0, 2024-03-23T23:14:04.636 INFO:teuthology.orchestra.run.smithi173.stdout: "cap": 2, 2024-03-23T23:14:04.636 INFO:teuthology.orchestra.run.smithi173.stdout: "cap+": 2, 2024-03-23T23:14:04.636 INFO:teuthology.orchestra.run.smithi173.stdout: "cap-": 0, 2024-03-23T23:14:04.636 INFO:teuthology.orchestra.run.smithi173.stdout: "rss": 49644, 2024-03-23T23:14:04.636 INFO:teuthology.orchestra.run.smithi173.stdout: "heap": 223612 2024-03-23T23:14:04.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.636 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_server": { 2024-03-23T23:14:04.636 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_client_request": 4, 2024-03-23T23:14:04.636 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_server_request": 0, 2024-03-23T23:14:04.636 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_request": 3, 2024-03-23T23:14:04.637 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_session": 10, 2024-03-23T23:14:04.637 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_peer_request": 0, 2024-03-23T23:14:04.637 INFO:teuthology.orchestra.run.smithi173.stdout: "req_create_latency": { 2024-03-23T23:14:04.637 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.637 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.637 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.637 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getattr_latency": { 2024-03-23T23:14:04.637 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1, 2024-03-23T23:14:04.637 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000820109, 2024-03-23T23:14:04.637 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000820109 2024-03-23T23:14:04.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.637 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getfilelock_latency": { 2024-03-23T23:14:04.637 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.638 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.638 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.638 INFO:teuthology.orchestra.run.smithi173.stdout: "req_link_latency": { 2024-03-23T23:14:04.638 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.638 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.638 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.638 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookup_latency": { 2024-03-23T23:14:04.638 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.638 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.638 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.639 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookuphash_latency": { 2024-03-23T23:14:04.639 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.639 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.639 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.639 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupino_latency": { 2024-03-23T23:14:04.639 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1, 2024-03-23T23:14:04.639 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000138986, 2024-03-23T23:14:04.639 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000138986 2024-03-23T23:14:04.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.639 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupname_latency": { 2024-03-23T23:14:04.639 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.639 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.639 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.639 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupparent_latency": { 2024-03-23T23:14:04.640 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.640 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.640 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.640 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupsnap_latency": { 2024-03-23T23:14:04.640 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.640 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.640 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.640 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lssnap_latency": { 2024-03-23T23:14:04.640 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.640 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.640 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.641 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mkdir_latency": { 2024-03-23T23:14:04.641 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.641 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.641 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.641 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mknod_latency": { 2024-03-23T23:14:04.642 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.642 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.642 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.642 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mksnap_latency": { 2024-03-23T23:14:04.642 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.642 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.642 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.642 INFO:teuthology.orchestra.run.smithi173.stdout: "req_open_latency": { 2024-03-23T23:14:04.642 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.642 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.642 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.643 INFO:teuthology.orchestra.run.smithi173.stdout: "req_readdir_latency": { 2024-03-23T23:14:04.643 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.643 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.643 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.643 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rename_latency": { 2024-03-23T23:14:04.643 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.643 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.643 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.644 INFO:teuthology.orchestra.run.smithi173.stdout: "req_renamesnap_latency": { 2024-03-23T23:14:04.644 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.644 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.644 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.644 INFO:teuthology.orchestra.run.smithi173.stdout: "req_snapdiff_latency": { 2024-03-23T23:14:04.644 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.644 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.644 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.644 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmdir_latency": { 2024-03-23T23:14:04.644 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.644 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.644 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.645 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmsnap_latency": { 2024-03-23T23:14:04.645 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.645 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.645 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.645 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmxattr_latency": { 2024-03-23T23:14:04.645 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.645 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.645 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.645 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setattr_latency": { 2024-03-23T23:14:04.645 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.645 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.646 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.646 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setdirlayout_latency": { 2024-03-23T23:14:04.646 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.646 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.646 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.646 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setfilelock_latency": { 2024-03-23T23:14:04.646 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.646 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.646 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.647 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setlayout_latency": { 2024-03-23T23:14:04.647 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.647 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.647 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.647 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setxattr_latency": { 2024-03-23T23:14:04.647 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.647 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.647 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.647 INFO:teuthology.orchestra.run.smithi173.stdout: "req_symlink_latency": { 2024-03-23T23:14:04.647 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.647 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.647 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.648 INFO:teuthology.orchestra.run.smithi173.stdout: "req_unlink_latency": { 2024-03-23T23:14:04.648 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.648 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.648 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.648 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_revoke_eviction": 0, 2024-03-23T23:14:04.648 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_acquisition_throttle": 0, 2024-03-23T23:14:04.648 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getvxattr_latency": { 2024-03-23T23:14:04.648 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.648 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.648 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.648 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:14:04.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.648 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_sessions": { 2024-03-23T23:14:04.648 INFO:teuthology.orchestra.run.smithi173.stdout: "session_count": 1, 2024-03-23T23:14:04.649 INFO:teuthology.orchestra.run.smithi173.stdout: "session_add": 1, 2024-03-23T23:14:04.649 INFO:teuthology.orchestra.run.smithi173.stdout: "session_remove": 0, 2024-03-23T23:14:04.649 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_open": 1, 2024-03-23T23:14:04.649 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_stale": 0, 2024-03-23T23:14:04.649 INFO:teuthology.orchestra.run.smithi173.stdout: "total_load": 1, 2024-03-23T23:14:04.649 INFO:teuthology.orchestra.run.smithi173.stdout: "average_load": 1, 2024-03-23T23:14:04.649 INFO:teuthology.orchestra.run.smithi173.stdout: "avg_session_uptime": 733, 2024-03-23T23:14:04.649 INFO:teuthology.orchestra.run.smithi173.stdout: "mdthresh_evicted": 0 2024-03-23T23:14:04.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.649 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2024-03-23T23:14:04.649 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2024-03-23T23:14:04.649 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2024-03-23T23:14:04.649 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2024-03-23T23:14:04.649 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2024-03-23T23:14:04.650 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2024-03-23T23:14:04.650 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2024-03-23T23:14:04.650 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-23T23:14:04.650 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2024-03-23T23:14:04.650 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-23T23:14:04.650 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2024-03-23T23:14:04.650 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2024-03-23T23:14:04.650 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2024-03-23T23:14:04.650 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-23T23:14:04.650 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2024-03-23T23:14:04.650 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2024-03-23T23:14:04.650 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2024-03-23T23:14:04.650 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2024-03-23T23:14:04.650 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2024-03-23T23:14:04.651 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-23T23:14:04.651 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2024-03-23T23:14:04.651 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-23T23:14:04.651 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2024-03-23T23:14:04.651 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2024-03-23T23:14:04.651 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2024-03-23T23:14:04.651 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2024-03-23T23:14:04.651 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2024-03-23T23:14:04.651 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-23T23:14:04.651 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2024-03-23T23:14:04.651 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2024-03-23T23:14:04.651 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2024-03-23T23:14:04.651 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2024-03-23T23:14:04.651 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2024-03-23T23:14:04.651 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2024-03-23T23:14:04.652 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2024-03-23T23:14:04.652 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2024-03-23T23:14:04.652 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2024-03-23T23:14:04.652 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 262935, 2024-03-23T23:14:04.652 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 89, 2024-03-23T23:14:04.652 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 0, 2024-03-23T23:14:04.652 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 0, 2024-03-23T23:14:04.652 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2024-03-23T23:14:04.652 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2024-03-23T23:14:04.652 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2024-03-23T23:14:04.652 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2024-03-23T23:14:04.652 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2024-03-23T23:14:04.652 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2024-03-23T23:14:04.652 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 7536, 2024-03-23T23:14:04.652 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 159, 2024-03-23T23:14:04.652 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2024-03-23T23:14:04.653 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2024-03-23T23:14:04.653 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2024-03-23T23:14:04.653 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2024-03-23T23:14:04.653 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 67325, 2024-03-23T23:14:04.653 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 291, 2024-03-23T23:14:04.653 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2024-03-23T23:14:04.653 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2024-03-23T23:14:04.653 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2024-03-23T23:14:04.653 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2024-03-23T23:14:04.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.653 INFO:teuthology.orchestra.run.smithi173.stdout: "objecter": { 2024-03-23T23:14:04.653 INFO:teuthology.orchestra.run.smithi173.stdout: "op_active": 0, 2024-03-23T23:14:04.653 INFO:teuthology.orchestra.run.smithi173.stdout: "op_laggy": 0, 2024-03-23T23:14:04.653 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send": 39, 2024-03-23T23:14:04.653 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send_bytes": 11511, 2024-03-23T23:14:04.654 INFO:teuthology.orchestra.run.smithi173.stdout: "op_resend": 33, 2024-03-23T23:14:04.654 INFO:teuthology.orchestra.run.smithi173.stdout: "op_reply": 39, 2024-03-23T23:14:04.654 INFO:teuthology.orchestra.run.smithi173.stdout: "op_latency": { 2024-03-23T23:14:04.654 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 39, 2024-03-23T23:14:04.654 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.282993715, 2024-03-23T23:14:04.654 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.007256249 2024-03-23T23:14:04.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.654 INFO:teuthology.orchestra.run.smithi173.stdout: "op_inflight": 0, 2024-03-23T23:14:04.654 INFO:teuthology.orchestra.run.smithi173.stdout: "oplen_avg": { 2024-03-23T23:14:04.654 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 39, 2024-03-23T23:14:04.654 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 63 2024-03-23T23:14:04.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.654 INFO:teuthology.orchestra.run.smithi173.stdout: "op": 39, 2024-03-23T23:14:04.654 INFO:teuthology.orchestra.run.smithi173.stdout: "op_r": 0, 2024-03-23T23:14:04.655 INFO:teuthology.orchestra.run.smithi173.stdout: "op_w": 39, 2024-03-23T23:14:04.655 INFO:teuthology.orchestra.run.smithi173.stdout: "op_rmw": 0, 2024-03-23T23:14:04.655 INFO:teuthology.orchestra.run.smithi173.stdout: "op_pg": 0, 2024-03-23T23:14:04.655 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_stat": 0, 2024-03-23T23:14:04.655 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_create": 10, 2024-03-23T23:14:04.655 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_read": 0, 2024-03-23T23:14:04.655 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_write": 2, 2024-03-23T23:14:04.655 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writefull": 6, 2024-03-23T23:14:04.655 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writesame": 0, 2024-03-23T23:14:04.655 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_append": 0, 2024-03-23T23:14:04.655 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_zero": 0, 2024-03-23T23:14:04.655 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_truncate": 0, 2024-03-23T23:14:04.655 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_delete": 6, 2024-03-23T23:14:04.655 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_mapext": 0, 2024-03-23T23:14:04.655 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_sparse_read": 0, 2024-03-23T23:14:04.655 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_clonerange": 0, 2024-03-23T23:14:04.656 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_getxattr": 0, 2024-03-23T23:14:04.656 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_setxattr": 20, 2024-03-23T23:14:04.656 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_cmpxattr": 0, 2024-03-23T23:14:04.656 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_rmxattr": 0, 2024-03-23T23:14:04.656 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_resetxattrs": 0, 2024-03-23T23:14:04.656 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_call": 0, 2024-03-23T23:14:04.656 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_watch": 0, 2024-03-23T23:14:04.656 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_notify": 0, 2024-03-23T23:14:04.656 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_src_cmpxattr": 0, 2024-03-23T23:14:04.656 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls": 0, 2024-03-23T23:14:04.656 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls_filter": 0, 2024-03-23T23:14:04.656 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_other": 0, 2024-03-23T23:14:04.656 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_active": 0, 2024-03-23T23:14:04.656 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_send": 0, 2024-03-23T23:14:04.656 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_resend": 0, 2024-03-23T23:14:04.657 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_ping": 0, 2024-03-23T23:14:04.657 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_active": 0, 2024-03-23T23:14:04.657 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_send": 0, 2024-03-23T23:14:04.657 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_resend": 0, 2024-03-23T23:14:04.657 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_active": 0, 2024-03-23T23:14:04.657 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_send": 0, 2024-03-23T23:14:04.657 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_resend": 0, 2024-03-23T23:14:04.657 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_active": 0, 2024-03-23T23:14:04.657 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_send": 0, 2024-03-23T23:14:04.657 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_resend": 0, 2024-03-23T23:14:04.657 INFO:teuthology.orchestra.run.smithi173.stdout: "command_active": 0, 2024-03-23T23:14:04.657 INFO:teuthology.orchestra.run.smithi173.stdout: "command_send": 0, 2024-03-23T23:14:04.657 INFO:teuthology.orchestra.run.smithi173.stdout: "command_resend": 0, 2024-03-23T23:14:04.657 INFO:teuthology.orchestra.run.smithi173.stdout: "map_epoch": 88, 2024-03-23T23:14:04.657 INFO:teuthology.orchestra.run.smithi173.stdout: "map_full": 0, 2024-03-23T23:14:04.657 INFO:teuthology.orchestra.run.smithi173.stdout: "map_inc": 2, 2024-03-23T23:14:04.658 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_sessions": 9, 2024-03-23T23:14:04.658 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_open": 9, 2024-03-23T23:14:04.658 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_close": 0, 2024-03-23T23:14:04.658 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_laggy": 0, 2024-03-23T23:14:04.658 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_wr": 18, 2024-03-23T23:14:04.658 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_rd": 0, 2024-03-23T23:14:04.658 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_del": 1 2024-03-23T23:14:04.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.658 INFO:teuthology.orchestra.run.smithi173.stdout: "oft": { 2024-03-23T23:14:04.658 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_objs": 1, 2024-03-23T23:14:04.658 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_kv_pairs": 0, 2024-03-23T23:14:04.658 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_updates": 0, 2024-03-23T23:14:04.658 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_removes": 0 2024-03-23T23:14:04.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.658 INFO:teuthology.orchestra.run.smithi173.stdout: "osd-slow-ops": { 2024-03-23T23:14:04.658 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_ops_count": 0 2024-03-23T23:14:04.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.659 INFO:teuthology.orchestra.run.smithi173.stdout: "purge_queue": { 2024-03-23T23:14:04.659 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops": 0, 2024-03-23T23:14:04.659 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops_high_water": 0, 2024-03-23T23:14:04.659 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing": 0, 2024-03-23T23:14:04.659 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_high_water": 0, 2024-03-23T23:14:04.659 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed_ops": 0, 2024-03-23T23:14:04.659 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed": 0, 2024-03-23T23:14:04.659 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_item_in_journal": 0 2024-03-23T23:14:04.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.659 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-23T23:14:04.659 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-23T23:14:04.659 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-03-23T23:14:04.659 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-23T23:14:04.659 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 1814, 2024-03-23T23:14:04.660 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 696661, 2024-03-23T23:14:04.660 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:04.660 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 1814, 2024-03-23T23:14:04.660 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-03-23T23:14:04.660 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-03-23T23:14:04.660 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 1814, 2024-03-23T23:14:04.660 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 696661, 2024-03-23T23:14:04.660 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-23T23:14:04.660 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.660 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.660 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.660 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:14:04.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.660 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_bytes": { 2024-03-23T23:14:04.661 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-23T23:14:04.661 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-03-23T23:14:04.661 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-23T23:14:04.661 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-03-23T23:14:04.661 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-03-23T23:14:04.661 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:04.661 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-03-23T23:14:04.661 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 39, 2024-03-23T23:14:04.661 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 11511, 2024-03-23T23:14:04.661 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 39, 2024-03-23T23:14:04.661 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 11511, 2024-03-23T23:14:04.661 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-23T23:14:04.661 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.661 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.661 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.661 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:14:04.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.662 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_ops": { 2024-03-23T23:14:04.662 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-23T23:14:04.662 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 1024, 2024-03-23T23:14:04.662 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-23T23:14:04.662 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-03-23T23:14:04.662 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-03-23T23:14:04.662 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:04.662 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-03-23T23:14:04.662 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 39, 2024-03-23T23:14:04.662 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 39, 2024-03-23T23:14:04.662 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 39, 2024-03-23T23:14:04.662 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 39, 2024-03-23T23:14:04.662 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-23T23:14:04.663 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.663 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.663 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.663 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:14:04.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.663 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle": { 2024-03-23T23:14:04.663 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-23T23:14:04.663 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-03-23T23:14:04.663 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-23T23:14:04.663 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-03-23T23:14:04.663 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-03-23T23:14:04.663 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:04.663 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-03-23T23:14:04.663 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-03-23T23:14:04.663 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-03-23T23:14:04.664 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2024-03-23T23:14:04.664 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2024-03-23T23:14:04.664 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-23T23:14:04.664 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.664 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.664 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.664 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:14:04.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:04.664 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle-0x561c563bc500": { 2024-03-23T23:14:04.664 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-23T23:14:04.664 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-03-23T23:14:04.664 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-23T23:14:04.664 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 3, 2024-03-23T23:14:04.664 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 894, 2024-03-23T23:14:04.664 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:04.664 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 3, 2024-03-23T23:14:04.664 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-03-23T23:14:04.664 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-03-23T23:14:04.665 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 2, 2024-03-23T23:14:04.665 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 894, 2024-03-23T23:14:04.665 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-23T23:14:04.665 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:04.665 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:04.665 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:04.665 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:14:04.665 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:14:04.665 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-23T23:14:05.100 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:04 smithi173 ceph-mon[36116]: pgmap v758: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-23T23:14:05.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:04 smithi203 ceph-mon[36693]: pgmap v758: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-23T23:14:05.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:04 smithi012 ceph-mon[31695]: pgmap v758: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-23T23:14:05.449 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-23T23:14:05.450 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-23T23:14:05.450 DEBUG:tasks.check_counter:Getting stats from k 2024-03-23T23:14:05.450 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-23T23:14:05.700 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.b/config 2024-03-23T23:14:07.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:06 smithi173 ceph-mon[36116]: pgmap v759: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 852 B/s wr, 6 op/s 2024-03-23T23:14:07.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:06 smithi203 ceph-mon[36693]: pgmap v759: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 852 B/s wr, 6 op/s 2024-03-23T23:14:07.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:06 smithi012 ceph-mon[31695]: pgmap v759: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 852 B/s wr, 6 op/s 2024-03-23T23:14:07.432 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-23T23:14:07.432 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2024-03-23T23:14:07.432 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 0, 2024-03-23T23:14:07.433 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 2, 2024-03-23T23:14:07.433 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 0, 2024-03-23T23:14:07.433 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 0, 2024-03-23T23:14:07.433 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 4, 2024-03-23T23:14:07.433 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 0, 2024-03-23T23:14:07.433 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.000890947, 2024-03-23T23:14:07.433 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.000000000, 2024-03-23T23:14:07.433 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.000104415, 2024-03-23T23:14:07.434 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-23T23:14:07.434 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:14:07.434 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:07.434 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:07.434 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:07.435 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:07.435 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:14:07.435 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:07.435 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:07.435 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:07.435 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:07.435 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-23T23:14:07.436 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 0 2024-03-23T23:14:07.436 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:07.436 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2024-03-23T23:14:07.436 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 6, 2024-03-23T23:14:07.436 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 191, 2024-03-23T23:14:07.436 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 3312, 2024-03-23T23:14:07.436 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 189008, 2024-03-23T23:14:07.436 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 4, 2024-03-23T23:14:07.436 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 2, 2024-03-23T23:14:07.436 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.007408382, 2024-03-23T23:14:07.436 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.005185747, 2024-03-23T23:14:07.436 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.001033299, 2024-03-23T23:14:07.436 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-23T23:14:07.437 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:14:07.437 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 190, 2024-03-23T23:14:07.437 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.009317763, 2024-03-23T23:14:07.437 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000049040 2024-03-23T23:14:07.437 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:07.437 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:14:07.437 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:07.437 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:07.437 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:07.437 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:07.437 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-23T23:14:07.437 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 189008 2024-03-23T23:14:07.437 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:07.438 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2024-03-23T23:14:07.438 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 249, 2024-03-23T23:14:07.438 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 246, 2024-03-23T23:14:07.438 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 485296, 2024-03-23T23:14:07.438 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 148208, 2024-03-23T23:14:07.439 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 2, 2024-03-23T23:14:07.439 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 1, 2024-03-23T23:14:07.439 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.026811663, 2024-03-23T23:14:07.439 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.009521307, 2024-03-23T23:14:07.439 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.013731739, 2024-03-23T23:14:07.439 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.004397878, 2024-03-23T23:14:07.439 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:14:07.439 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 245, 2024-03-23T23:14:07.439 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.007158721, 2024-03-23T23:14:07.439 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000029219 2024-03-23T23:14:07.439 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:07.439 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:14:07.439 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:07.439 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:07.439 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:07.439 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:07.440 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 485296, 2024-03-23T23:14:07.440 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 148208 2024-03-23T23:14:07.440 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:07.440 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2024-03-23T23:14:07.440 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 0, 2024-03-23T23:14:07.440 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2024-03-23T23:14:07.440 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:07.440 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2024-03-23T23:14:07.440 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2024-03-23T23:14:07.440 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2024-03-23T23:14:07.440 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2024-03-23T23:14:07.440 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2024-03-23T23:14:07.440 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2024-03-23T23:14:07.440 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2024-03-23T23:14:07.440 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-23T23:14:07.441 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2024-03-23T23:14:07.441 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-23T23:14:07.441 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2024-03-23T23:14:07.441 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2024-03-23T23:14:07.441 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2024-03-23T23:14:07.441 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-23T23:14:07.441 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2024-03-23T23:14:07.441 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2024-03-23T23:14:07.441 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2024-03-23T23:14:07.441 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2024-03-23T23:14:07.441 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2024-03-23T23:14:07.441 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-23T23:14:07.441 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2024-03-23T23:14:07.441 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-23T23:14:07.441 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2024-03-23T23:14:07.442 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2024-03-23T23:14:07.442 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2024-03-23T23:14:07.442 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2024-03-23T23:14:07.442 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2024-03-23T23:14:07.442 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-23T23:14:07.442 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2024-03-23T23:14:07.442 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2024-03-23T23:14:07.442 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2024-03-23T23:14:07.442 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2024-03-23T23:14:07.442 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2024-03-23T23:14:07.443 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2024-03-23T23:14:07.443 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2024-03-23T23:14:07.443 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2024-03-23T23:14:07.443 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2024-03-23T23:14:07.443 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 147898, 2024-03-23T23:14:07.443 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 27, 2024-03-23T23:14:07.443 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 0, 2024-03-23T23:14:07.443 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 0, 2024-03-23T23:14:07.443 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2024-03-23T23:14:07.443 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2024-03-23T23:14:07.443 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2024-03-23T23:14:07.443 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2024-03-23T23:14:07.443 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2024-03-23T23:14:07.443 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2024-03-23T23:14:07.444 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 0, 2024-03-23T23:14:07.444 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 0, 2024-03-23T23:14:07.444 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2024-03-23T23:14:07.444 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2024-03-23T23:14:07.444 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2024-03-23T23:14:07.444 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2024-03-23T23:14:07.444 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 928, 2024-03-23T23:14:07.444 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 2, 2024-03-23T23:14:07.444 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2024-03-23T23:14:07.444 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2024-03-23T23:14:07.444 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2024-03-23T23:14:07.444 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2024-03-23T23:14:07.444 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-23T23:14:07.444 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-23T23:14:07.444 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-23T23:14:07.445 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-03-23T23:14:07.445 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-23T23:14:07.445 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 251, 2024-03-23T23:14:07.445 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 450129, 2024-03-23T23:14:07.445 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:07.445 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 251, 2024-03-23T23:14:07.445 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-03-23T23:14:07.445 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-03-23T23:14:07.445 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 251, 2024-03-23T23:14:07.445 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 450129, 2024-03-23T23:14:07.445 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-23T23:14:07.445 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-23T23:14:07.445 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-23T23:14:07.445 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-23T23:14:07.446 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:14:07.446 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-23T23:14:07.446 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-23T23:14:08.142 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-03-23T23:14:08.143 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-03-23T23:14:08.143 DEBUG:tasks.check_counter:Getting stats from c 2024-03-23T23:14:08.143 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-23T23:14:08.398 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.c/config 2024-03-23T23:14:09.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:08 smithi203 ceph-mon[36693]: pgmap v760: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-23T23:14:09.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:08 smithi012 ceph-mon[31695]: pgmap v760: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-23T23:14:09.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:08 smithi173 ceph-mon[36116]: pgmap v760: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-23T23:14:10.344 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-23T23:14:10.344 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-03-23T23:14:10.344 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 1790, 2024-03-23T23:14:10.344 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 1790, 2024-03-23T23:14:10.344 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 4725291, 2024-03-23T23:14:10.345 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 568406, 2024-03-23T23:14:10.345 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 6, 2024-03-23T23:14:10.345 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 2, 2024-03-23T23:14:10.345 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.110802212, 2024-03-23T23:14:10.345 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.030614096, 2024-03-23T23:14:10.345 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.033231160, 2024-03-23T23:14:10.345 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.037311010, 2024-03-23T23:14:10.345 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:14:10.345 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1789, 2024-03-23T23:14:10.345 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.042035717, 2024-03-23T23:14:10.345 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000023496 2024-03-23T23:14:10.345 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.345 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:14:10.345 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.345 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.345 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.346 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.346 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-23T23:14:10.346 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 448 2024-03-23T23:14:10.346 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.346 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-03-23T23:14:10.346 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 315, 2024-03-23T23:14:10.346 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 500, 2024-03-23T23:14:10.346 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 77990, 2024-03-23T23:14:10.346 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 578512, 2024-03-23T23:14:10.346 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 6, 2024-03-23T23:14:10.346 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 4, 2024-03-23T23:14:10.346 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.029632793, 2024-03-23T23:14:10.346 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.012265064, 2024-03-23T23:14:10.346 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.006147185, 2024-03-23T23:14:10.346 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.008298346, 2024-03-23T23:14:10.346 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:14:10.347 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 498, 2024-03-23T23:14:10.347 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.020311992, 2024-03-23T23:14:10.347 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000040787 2024-03-23T23:14:10.347 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.347 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:14:10.347 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.347 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.347 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.347 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.347 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-23T23:14:10.347 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 478912 2024-03-23T23:14:10.347 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.347 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-03-23T23:14:10.347 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 1403, 2024-03-23T23:14:10.347 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 1399, 2024-03-23T23:14:10.347 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 4482677, 2024-03-23T23:14:10.348 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 512439, 2024-03-23T23:14:10.348 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 3, 2024-03-23T23:14:10.348 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 2, 2024-03-23T23:14:10.348 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.097866321, 2024-03-23T23:14:10.348 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.033798921, 2024-03-23T23:14:10.348 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.030969423, 2024-03-23T23:14:10.348 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.027651596, 2024-03-23T23:14:10.348 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:14:10.348 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1398, 2024-03-23T23:14:10.348 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.031786107, 2024-03-23T23:14:10.348 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000022736 2024-03-23T23:14:10.348 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.348 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:14:10.348 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.348 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.348 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.349 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.349 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 502352, 2024-03-23T23:14:10.349 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 147056 2024-03-23T23:14:10.349 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.349 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-03-23T23:14:10.349 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-03-23T23:14:10.349 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-03-23T23:14:10.349 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.349 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-03-23T23:14:10.349 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-03-23T23:14:10.349 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-03-23T23:14:10.349 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 5578, 2024-03-23T23:14:10.349 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.100250271, 2024-03-23T23:14:10.349 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000017972 2024-03-23T23:14:10.349 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:10.349 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.349 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-03-23T23:14:10.350 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-03-23T23:14:10.350 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-03-23T23:14:10.350 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.350 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.350 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.350 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:10.350 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.350 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-03-23T23:14:10.350 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 0, 2024-03-23T23:14:10.350 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 0, 2024-03-23T23:14:10.350 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-03-23T23:14:10.350 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.350 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.350 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.351 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.351 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-03-23T23:14:10.351 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 0, 2024-03-23T23:14:10.351 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 0, 2024-03-23T23:14:10.351 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 0, 2024-03-23T23:14:10.351 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 0, 2024-03-23T23:14:10.351 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 0, 2024-03-23T23:14:10.351 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-03-23T23:14:10.351 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 18, 2024-03-23T23:14:10.351 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 4, 2024-03-23T23:14:10.351 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 14, 2024-03-23T23:14:10.351 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 0, 2024-03-23T23:14:10.351 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 11, 2024-03-23T23:14:10.351 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 0, 2024-03-23T23:14:10.351 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 0, 2024-03-23T23:14:10.352 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 0, 2024-03-23T23:14:10.352 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 2, 2024-03-23T23:14:10.352 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 0, 2024-03-23T23:14:10.352 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 0, 2024-03-23T23:14:10.352 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 0, 2024-03-23T23:14:10.352 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 0, 2024-03-23T23:14:10.352 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 0, 2024-03-23T23:14:10.352 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-03-23T23:14:10.352 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 0, 2024-03-23T23:14:10.352 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 0, 2024-03-23T23:14:10.352 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-03-23T23:14:10.352 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 0, 2024-03-23T23:14:10.352 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 0, 2024-03-23T23:14:10.352 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 0, 2024-03-23T23:14:10.352 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 0, 2024-03-23T23:14:10.352 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-03-23T23:14:10.353 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 0, 2024-03-23T23:14:10.353 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 0, 2024-03-23T23:14:10.353 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 4, 2024-03-23T23:14:10.353 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 234, 2024-03-23T23:14:10.353 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 0, 2024-03-23T23:14:10.353 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 0, 2024-03-23T23:14:10.353 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-03-23T23:14:10.353 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-03-23T23:14:10.353 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-03-23T23:14:10.353 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 0, 2024-03-23T23:14:10.353 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-03-23T23:14:10.353 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 0, 2024-03-23T23:14:10.353 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 0, 2024-03-23T23:14:10.353 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 0, 2024-03-23T23:14:10.353 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 0, 2024-03-23T23:14:10.353 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 0, 2024-03-23T23:14:10.353 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 0, 2024-03-23T23:14:10.354 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-23T23:14:10.354 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-23T23:14:10.354 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 0, 2024-03-23T23:14:10.354 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 0, 2024-03-23T23:14:10.354 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 0, 2024-03-23T23:14:10.354 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 0 2024-03-23T23:14:10.354 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.354 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-03-23T23:14:10.354 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 0, 2024-03-23T23:14:10.354 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 0, 2024-03-23T23:14:10.354 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 0, 2024-03-23T23:14:10.354 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 0, 2024-03-23T23:14:10.354 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 0, 2024-03-23T23:14:10.354 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 0, 2024-03-23T23:14:10.354 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-03-23T23:14:10.354 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-03-23T23:14:10.355 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 0, 2024-03-23T23:14:10.355 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 0, 2024-03-23T23:14:10.355 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 0, 2024-03-23T23:14:10.355 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 0, 2024-03-23T23:14:10.355 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-03-23T23:14:10.355 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-03-23T23:14:10.355 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-03-23T23:14:10.355 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-03-23T23:14:10.355 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-03-23T23:14:10.355 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-03-23T23:14:10.355 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 0, 2024-03-23T23:14:10.355 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-03-23T23:14:10.355 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 0, 2024-03-23T23:14:10.355 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-03-23T23:14:10.355 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-03-23T23:14:10.356 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.356 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-03-23T23:14:10.356 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-03-23T23:14:10.356 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 0, 2024-03-23T23:14:10.356 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-03-23T23:14:10.356 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-03-23T23:14:10.356 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 2438, 2024-03-23T23:14:10.356 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-03-23T23:14:10.356 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-03-23T23:14:10.356 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 0, 2024-03-23T23:14:10.356 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-03-23T23:14:10.356 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-03-23T23:14:10.356 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 2, 2024-03-23T23:14:10.356 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 54, 2024-03-23T23:14:10.356 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-03-23T23:14:10.356 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-03-23T23:14:10.357 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 11905007, 2024-03-23T23:14:10.357 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 12081088, 2024-03-23T23:14:10.357 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 12078591, 2024-03-23T23:14:10.357 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-03-23T23:14:10.357 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.357 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.357 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.357 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.357 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 2438 2024-03-23T23:14:10.357 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.357 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-03-23T23:14:10.357 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 15, 2024-03-23T23:14:10.357 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 17, 2024-03-23T23:14:10.357 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 2, 2024-03-23T23:14:10.357 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 10, 2024-03-23T23:14:10.357 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 11, 2024-03-23T23:14:10.358 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 1, 2024-03-23T23:14:10.358 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 18, 2024-03-23T23:14:10.358 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 19, 2024-03-23T23:14:10.358 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 1, 2024-03-23T23:14:10.358 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 0, 2024-03-23T23:14:10.358 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 0, 2024-03-23T23:14:10.358 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 0, 2024-03-23T23:14:10.358 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 50496, 2024-03-23T23:14:10.358 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 215420 2024-03-23T23:14:10.358 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.358 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-03-23T23:14:10.358 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 0, 2024-03-23T23:14:10.358 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 0, 2024-03-23T23:14:10.358 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 0, 2024-03-23T23:14:10.358 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 0, 2024-03-23T23:14:10.359 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 0, 2024-03-23T23:14:10.359 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-03-23T23:14:10.359 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.359 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.359 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.359 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.359 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-03-23T23:14:10.359 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.359 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.359 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.359 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.359 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-03-23T23:14:10.359 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.359 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.359 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.360 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.360 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-03-23T23:14:10.360 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.360 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.360 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.360 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.360 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-03-23T23:14:10.360 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.360 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.360 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.360 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.360 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-03-23T23:14:10.360 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.360 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.360 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.360 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.361 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-03-23T23:14:10.361 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.361 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.361 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.361 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.361 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-03-23T23:14:10.361 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.361 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.361 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.361 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.361 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-03-23T23:14:10.361 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.361 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.361 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.361 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.361 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-03-23T23:14:10.361 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.361 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.362 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.362 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.362 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-03-23T23:14:10.362 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.362 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.362 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.362 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.362 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-03-23T23:14:10.362 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.362 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.363 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.363 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.363 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-03-23T23:14:10.363 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.363 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.363 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.363 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.363 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-03-23T23:14:10.363 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.363 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.363 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.363 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.363 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-03-23T23:14:10.363 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.363 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.363 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.364 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.364 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-03-23T23:14:10.364 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.364 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.364 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.364 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.364 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-03-23T23:14:10.364 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.364 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.364 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.364 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.364 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-03-23T23:14:10.364 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.364 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.364 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.364 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.365 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-03-23T23:14:10.365 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.365 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.365 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.365 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.365 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-03-23T23:14:10.365 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.365 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.365 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.365 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.365 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-03-23T23:14:10.365 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.365 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.365 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.365 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.365 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-03-23T23:14:10.365 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.366 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.366 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.366 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.366 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-03-23T23:14:10.366 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.366 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.366 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.366 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.366 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-03-23T23:14:10.366 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.366 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.366 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.366 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.366 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-03-23T23:14:10.366 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.366 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.367 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.367 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.367 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-03-23T23:14:10.367 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.367 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.367 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.367 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.367 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-03-23T23:14:10.367 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.367 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.367 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.367 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.367 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-03-23T23:14:10.367 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.367 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.367 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.367 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.368 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-03-23T23:14:10.368 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.368 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.368 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.368 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.368 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-03-23T23:14:10.368 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-03-23T23:14:10.368 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-03-23T23:14:10.368 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.368 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.368 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.368 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:10.368 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.368 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-03-23T23:14:10.368 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 1, 2024-03-23T23:14:10.368 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 2, 2024-03-23T23:14:10.368 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 1, 2024-03-23T23:14:10.369 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 1, 2024-03-23T23:14:10.369 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-03-23T23:14:10.369 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 0, 2024-03-23T23:14:10.369 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 0, 2024-03-23T23:14:10.369 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 2280, 2024-03-23T23:14:10.369 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-03-23T23:14:10.369 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.369 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-03-23T23:14:10.369 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-03-23T23:14:10.369 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-03-23T23:14:10.369 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-03-23T23:14:10.369 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-03-23T23:14:10.369 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-03-23T23:14:10.369 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-03-23T23:14:10.369 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-23T23:14:10.370 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-03-23T23:14:10.370 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-23T23:14:10.370 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-03-23T23:14:10.370 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-03-23T23:14:10.370 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-03-23T23:14:10.370 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-23T23:14:10.370 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-03-23T23:14:10.370 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-03-23T23:14:10.370 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-03-23T23:14:10.370 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-03-23T23:14:10.370 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-03-23T23:14:10.370 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-23T23:14:10.370 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-03-23T23:14:10.370 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-23T23:14:10.370 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-03-23T23:14:10.371 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-03-23T23:14:10.371 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-03-23T23:14:10.371 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-03-23T23:14:10.371 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-03-23T23:14:10.371 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-23T23:14:10.371 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-03-23T23:14:10.371 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-03-23T23:14:10.371 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-03-23T23:14:10.371 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-03-23T23:14:10.371 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-03-23T23:14:10.371 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-03-23T23:14:10.371 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-03-23T23:14:10.371 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-03-23T23:14:10.371 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-03-23T23:14:10.371 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 189415, 2024-03-23T23:14:10.371 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 52, 2024-03-23T23:14:10.372 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-03-23T23:14:10.372 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-03-23T23:14:10.372 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-03-23T23:14:10.372 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-03-23T23:14:10.372 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-03-23T23:14:10.372 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-03-23T23:14:10.372 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-03-23T23:14:10.372 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-03-23T23:14:10.372 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 7536, 2024-03-23T23:14:10.372 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 159, 2024-03-23T23:14:10.372 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-03-23T23:14:10.372 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-03-23T23:14:10.372 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-03-23T23:14:10.372 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-03-23T23:14:10.372 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 58338, 2024-03-23T23:14:10.372 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 354, 2024-03-23T23:14:10.373 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-03-23T23:14:10.373 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-03-23T23:14:10.373 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-03-23T23:14:10.373 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-03-23T23:14:10.373 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.373 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-03-23T23:14:10.373 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-03-23T23:14:10.373 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-03-23T23:14:10.373 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 3248, 2024-03-23T23:14:10.373 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 26, 2024-03-23T23:14:10.373 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 4, 2024-03-23T23:14:10.373 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 3248, 2024-03-23T23:14:10.373 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-03-23T23:14:10.373 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 3248, 2024-03-23T23:14:10.373 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 1.176971181, 2024-03-23T23:14:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000362367 2024-03-23T23:14:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-03-23T23:14:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-03-23T23:14:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 3248, 2024-03-23T23:14:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 3249 2024-03-23T23:14:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 3248, 2024-03-23T23:14:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 3248, 2024-03-23T23:14:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 0, 2024-03-23T23:14:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-03-23T23:14:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-03-23T23:14:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 1735, 2024-03-23T23:14:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 0, 2024-03-23T23:14:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 1510, 2024-03-23T23:14:10.375 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 0, 2024-03-23T23:14:10.375 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 0, 2024-03-23T23:14:10.375 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-03-23T23:14:10.375 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-03-23T23:14:10.375 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 0, 2024-03-23T23:14:10.375 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 0, 2024-03-23T23:14:10.375 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 0, 2024-03-23T23:14:10.375 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-03-23T23:14:10.375 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-03-23T23:14:10.375 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-03-23T23:14:10.375 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 2, 2024-03-23T23:14:10.375 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 0, 2024-03-23T23:14:10.375 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-03-23T23:14:10.375 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-03-23T23:14:10.375 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-03-23T23:14:10.375 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-03-23T23:14:10.376 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-03-23T23:14:10.376 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-03-23T23:14:10.376 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-03-23T23:14:10.376 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-03-23T23:14:10.376 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-03-23T23:14:10.376 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 0, 2024-03-23T23:14:10.376 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-03-23T23:14:10.376 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-03-23T23:14:10.376 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-03-23T23:14:10.376 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-03-23T23:14:10.376 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-03-23T23:14:10.376 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-03-23T23:14:10.376 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-03-23T23:14:10.376 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-03-23T23:14:10.376 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-03-23T23:14:10.376 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-03-23T23:14:10.377 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-03-23T23:14:10.377 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-03-23T23:14:10.377 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-03-23T23:14:10.377 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-03-23T23:14:10.377 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-03-23T23:14:10.377 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-03-23T23:14:10.377 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 88, 2024-03-23T23:14:10.377 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-03-23T23:14:10.377 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 2, 2024-03-23T23:14:10.377 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 5, 2024-03-23T23:14:10.377 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 5, 2024-03-23T23:14:10.377 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 0, 2024-03-23T23:14:10.377 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-03-23T23:14:10.377 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 0, 2024-03-23T23:14:10.377 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 2, 2024-03-23T23:14:10.377 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 0 2024-03-23T23:14:10.378 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.378 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-03-23T23:14:10.378 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 0, 2024-03-23T23:14:10.378 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-03-23T23:14:10.378 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-03-23T23:14:10.378 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-03-23T23:14:10.378 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.378 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-03-23T23:14:10.378 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 0 2024-03-23T23:14:10.378 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.378 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-03-23T23:14:10.378 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-03-23T23:14:10.378 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 0, 2024-03-23T23:14:10.378 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-03-23T23:14:10.378 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 0, 2024-03-23T23:14:10.379 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 0, 2024-03-23T23:14:10.379 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 0, 2024-03-23T23:14:10.379 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-03-23T23:14:10.379 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.379 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-23T23:14:10.379 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-23T23:14:10.379 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-03-23T23:14:10.379 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-23T23:14:10.379 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 3504, 2024-03-23T23:14:10.379 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 8954528, 2024-03-23T23:14:10.379 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:10.379 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 3504, 2024-03-23T23:14:10.379 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-03-23T23:14:10.379 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-03-23T23:14:10.379 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 3504, 2024-03-23T23:14:10.380 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 8954528, 2024-03-23T23:14:10.380 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-23T23:14:10.380 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.380 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.380 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.380 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:10.380 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.380 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-03-23T23:14:10.380 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-23T23:14:10.380 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-03-23T23:14:10.380 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-23T23:14:10.380 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-03-23T23:14:10.380 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-03-23T23:14:10.380 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:10.380 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-03-23T23:14:10.381 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 3248, 2024-03-23T23:14:10.381 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 7886794, 2024-03-23T23:14:10.381 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 3248, 2024-03-23T23:14:10.381 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 7886794, 2024-03-23T23:14:10.381 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-23T23:14:10.381 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.381 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.381 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.381 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:10.381 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.381 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-03-23T23:14:10.381 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-23T23:14:10.381 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-03-23T23:14:10.381 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-23T23:14:10.382 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-03-23T23:14:10.382 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-03-23T23:14:10.382 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:10.382 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-03-23T23:14:10.382 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 3248, 2024-03-23T23:14:10.382 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 3248, 2024-03-23T23:14:10.382 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 3248, 2024-03-23T23:14:10.382 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 3248, 2024-03-23T23:14:10.382 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-23T23:14:10.382 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.382 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.382 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.382 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:10.382 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.382 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-03-23T23:14:10.382 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-23T23:14:10.383 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-03-23T23:14:10.383 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-23T23:14:10.383 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-03-23T23:14:10.383 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-03-23T23:14:10.383 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:10.383 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-03-23T23:14:10.383 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-03-23T23:14:10.383 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-03-23T23:14:10.383 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-03-23T23:14:10.383 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-03-23T23:14:10.383 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-23T23:14:10.383 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.383 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.383 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.383 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:10.383 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:10.384 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x564dec1686e0": { 2024-03-23T23:14:10.384 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-23T23:14:10.384 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-03-23T23:14:10.384 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-23T23:14:10.384 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-03-23T23:14:10.384 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-03-23T23:14:10.384 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:10.384 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-03-23T23:14:10.384 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-03-23T23:14:10.384 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-03-23T23:14:10.384 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-03-23T23:14:10.384 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-03-23T23:14:10.384 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-23T23:14:10.384 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:10.384 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:10.384 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:10.385 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:10.385 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:10.385 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-23T23:14:11.055 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-23T23:14:11.055 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-23T23:14:11.055 DEBUG:tasks.check_counter:Getting stats from f 2024-03-23T23:14:11.056 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-23T23:14:11.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:10 smithi203 ceph-mon[36693]: pgmap v761: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-23T23:14:11.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:10 smithi012 ceph-mon[31695]: pgmap v761: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-23T23:14:11.358 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.c/config 2024-03-23T23:14:11.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:10 smithi173 ceph-mon[36116]: pgmap v761: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-23T23:14:13.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:12 smithi203 ceph-mon[36693]: pgmap v762: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-23T23:14:13.185 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-23T23:14:13.185 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-03-23T23:14:13.185 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 888, 2024-03-23T23:14:13.185 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 888, 2024-03-23T23:14:13.185 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 235811, 2024-03-23T23:14:13.185 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 281624, 2024-03-23T23:14:13.185 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 6, 2024-03-23T23:14:13.185 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 2, 2024-03-23T23:14:13.185 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.054005059, 2024-03-23T23:14:13.185 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.015083575, 2024-03-23T23:14:13.186 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.015833222, 2024-03-23T23:14:13.186 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.017593919, 2024-03-23T23:14:13.186 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:14:13.186 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 887, 2024-03-23T23:14:13.186 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.022093373, 2024-03-23T23:14:13.186 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000024907 2024-03-23T23:14:13.186 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.186 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:14:13.186 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.186 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.186 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.187 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.187 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-23T23:14:13.187 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 448 2024-03-23T23:14:13.187 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.187 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-03-23T23:14:13.187 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 257, 2024-03-23T23:14:13.187 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 253, 2024-03-23T23:14:13.187 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 503250, 2024-03-23T23:14:13.187 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 148785, 2024-03-23T23:14:13.187 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 3, 2024-03-23T23:14:13.187 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 2, 2024-03-23T23:14:13.187 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.025805284, 2024-03-23T23:14:13.187 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.008988596, 2024-03-23T23:14:13.187 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.012719529, 2024-03-23T23:14:13.188 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.004241757, 2024-03-23T23:14:13.188 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:14:13.188 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 252, 2024-03-23T23:14:13.188 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.009452424, 2024-03-23T23:14:13.188 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000037509 2024-03-23T23:14:13.188 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.188 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:14:13.188 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.188 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.188 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.188 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.188 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 502736, 2024-03-23T23:14:13.188 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 148144 2024-03-23T23:14:13.188 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.188 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-03-23T23:14:13.188 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 881, 2024-03-23T23:14:13.189 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 1066, 2024-03-23T23:14:13.189 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 295840, 2024-03-23T23:14:13.189 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 680752, 2024-03-23T23:14:13.189 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 4, 2024-03-23T23:14:13.189 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 3, 2024-03-23T23:14:13.189 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.066545302, 2024-03-23T23:14:13.189 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.022767521, 2024-03-23T23:14:13.189 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.016784141, 2024-03-23T23:14:13.189 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.019977792, 2024-03-23T23:14:13.189 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:14:13.189 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1064, 2024-03-23T23:14:13.189 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.041025619, 2024-03-23T23:14:13.189 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000038557 2024-03-23T23:14:13.189 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.190 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:14:13.190 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.190 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.190 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.190 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.190 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-23T23:14:13.190 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 400912 2024-03-23T23:14:13.190 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.190 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-03-23T23:14:13.190 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-03-23T23:14:13.191 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-03-23T23:14:13.191 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.191 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-03-23T23:14:13.191 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-03-23T23:14:13.191 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-03-23T23:14:13.191 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 3524, 2024-03-23T23:14:13.191 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.066053872, 2024-03-23T23:14:13.191 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000018744 2024-03-23T23:14:13.191 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:13.191 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.191 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-03-23T23:14:13.191 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-03-23T23:14:13.191 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-03-23T23:14:13.191 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.192 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.192 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.192 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:13.192 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.192 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-03-23T23:14:13.192 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 0, 2024-03-23T23:14:13.192 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 0, 2024-03-23T23:14:13.192 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-03-23T23:14:13.192 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.192 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.192 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.192 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.192 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-03-23T23:14:13.192 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 0, 2024-03-23T23:14:13.193 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 0, 2024-03-23T23:14:13.193 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 0, 2024-03-23T23:14:13.193 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 0, 2024-03-23T23:14:13.193 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 0, 2024-03-23T23:14:13.193 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-03-23T23:14:13.193 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 0, 2024-03-23T23:14:13.193 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 0, 2024-03-23T23:14:13.193 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 0, 2024-03-23T23:14:13.193 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 0, 2024-03-23T23:14:13.193 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 0, 2024-03-23T23:14:13.193 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 0, 2024-03-23T23:14:13.193 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 0, 2024-03-23T23:14:13.193 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 0, 2024-03-23T23:14:13.194 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 1, 2024-03-23T23:14:13.194 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 0, 2024-03-23T23:14:13.194 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 0, 2024-03-23T23:14:13.194 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 0, 2024-03-23T23:14:13.194 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 0, 2024-03-23T23:14:13.194 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 0, 2024-03-23T23:14:13.194 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-03-23T23:14:13.194 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 0, 2024-03-23T23:14:13.194 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 0, 2024-03-23T23:14:13.194 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-03-23T23:14:13.194 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 0, 2024-03-23T23:14:13.194 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 0, 2024-03-23T23:14:13.194 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 0, 2024-03-23T23:14:13.194 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 0, 2024-03-23T23:14:13.195 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-03-23T23:14:13.195 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 0, 2024-03-23T23:14:13.195 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 0, 2024-03-23T23:14:13.195 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 0, 2024-03-23T23:14:13.195 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 0, 2024-03-23T23:14:13.195 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 0, 2024-03-23T23:14:13.195 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 0, 2024-03-23T23:14:13.195 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-03-23T23:14:13.195 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-03-23T23:14:13.195 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-03-23T23:14:13.195 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 0, 2024-03-23T23:14:13.195 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-03-23T23:14:13.195 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 0, 2024-03-23T23:14:13.195 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 0, 2024-03-23T23:14:13.195 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 0, 2024-03-23T23:14:13.196 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 0, 2024-03-23T23:14:13.196 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 0, 2024-03-23T23:14:13.196 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 0, 2024-03-23T23:14:13.196 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-23T23:14:13.196 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-23T23:14:13.196 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 0, 2024-03-23T23:14:13.196 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 0, 2024-03-23T23:14:13.196 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 0, 2024-03-23T23:14:13.196 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 0 2024-03-23T23:14:13.196 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.196 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-03-23T23:14:13.196 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 0, 2024-03-23T23:14:13.196 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 0, 2024-03-23T23:14:13.196 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 0, 2024-03-23T23:14:13.197 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 0, 2024-03-23T23:14:13.197 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 0, 2024-03-23T23:14:13.197 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 0, 2024-03-23T23:14:13.197 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-03-23T23:14:13.197 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-03-23T23:14:13.197 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 0, 2024-03-23T23:14:13.197 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 0, 2024-03-23T23:14:13.197 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 0, 2024-03-23T23:14:13.197 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 0, 2024-03-23T23:14:13.197 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-03-23T23:14:13.197 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-03-23T23:14:13.197 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-03-23T23:14:13.197 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-03-23T23:14:13.197 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-03-23T23:14:13.197 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-03-23T23:14:13.197 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 0, 2024-03-23T23:14:13.198 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-03-23T23:14:13.198 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 0, 2024-03-23T23:14:13.198 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-03-23T23:14:13.198 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-03-23T23:14:13.198 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.198 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-03-23T23:14:13.198 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-03-23T23:14:13.198 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 0, 2024-03-23T23:14:13.198 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-03-23T23:14:13.198 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-03-23T23:14:13.198 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 2, 2024-03-23T23:14:13.198 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-03-23T23:14:13.198 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-03-23T23:14:13.198 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 0, 2024-03-23T23:14:13.199 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-03-23T23:14:13.199 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-03-23T23:14:13.199 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 2, 2024-03-23T23:14:13.199 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 2, 2024-03-23T23:14:13.199 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-03-23T23:14:13.199 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-03-23T23:14:13.199 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 4194304, 2024-03-23T23:14:13.199 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 4194892, 2024-03-23T23:14:13.199 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 4194352, 2024-03-23T23:14:13.199 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-03-23T23:14:13.199 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.199 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.199 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.200 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.200 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 2 2024-03-23T23:14:13.200 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.200 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-03-23T23:14:13.200 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 3, 2024-03-23T23:14:13.200 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 3, 2024-03-23T23:14:13.200 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 0, 2024-03-23T23:14:13.200 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 1, 2024-03-23T23:14:13.200 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 1, 2024-03-23T23:14:13.200 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 0, 2024-03-23T23:14:13.200 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 0, 2024-03-23T23:14:13.200 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 0, 2024-03-23T23:14:13.200 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 0, 2024-03-23T23:14:13.200 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 0, 2024-03-23T23:14:13.201 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 0, 2024-03-23T23:14:13.201 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 0, 2024-03-23T23:14:13.201 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 46428, 2024-03-23T23:14:13.201 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 215420 2024-03-23T23:14:13.201 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.201 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-03-23T23:14:13.201 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 0, 2024-03-23T23:14:13.201 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 0, 2024-03-23T23:14:13.201 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 0, 2024-03-23T23:14:13.201 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 0, 2024-03-23T23:14:13.201 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 0, 2024-03-23T23:14:13.201 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-03-23T23:14:13.201 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.201 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.201 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.202 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.202 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-03-23T23:14:13.202 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.202 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.202 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.202 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.202 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-03-23T23:14:13.202 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.202 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.202 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.202 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.202 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-03-23T23:14:13.202 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.202 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.203 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.203 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.203 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-03-23T23:14:13.203 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.203 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.203 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.203 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.203 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-03-23T23:14:13.203 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.203 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.203 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.203 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.203 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-03-23T23:14:13.203 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.203 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.204 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.204 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.204 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-03-23T23:14:13.204 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.204 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.204 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.204 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.204 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-03-23T23:14:13.204 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.204 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.204 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.204 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.204 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-03-23T23:14:13.204 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.204 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.205 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.205 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.205 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-03-23T23:14:13.205 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.205 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.205 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.205 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.205 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-03-23T23:14:13.205 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.205 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.205 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.205 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.206 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-03-23T23:14:13.206 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.206 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.206 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.206 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.206 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-03-23T23:14:13.206 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.207 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.207 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.207 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.207 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-03-23T23:14:13.207 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.207 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.207 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.207 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.207 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-03-23T23:14:13.207 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.207 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.207 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.207 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.207 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-03-23T23:14:13.208 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.208 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.208 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.208 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.208 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-03-23T23:14:13.208 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.208 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.208 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.208 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.208 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-03-23T23:14:13.208 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.208 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.208 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.208 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.208 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-03-23T23:14:13.209 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.209 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.209 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.209 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.209 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-03-23T23:14:13.209 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.209 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.209 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.209 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.209 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-03-23T23:14:13.209 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.209 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.209 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.209 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.209 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-03-23T23:14:13.210 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.210 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.210 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.210 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.210 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-03-23T23:14:13.210 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.210 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.210 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.210 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.210 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-03-23T23:14:13.210 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.210 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.210 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.211 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.211 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-03-23T23:14:13.211 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.211 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.211 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.211 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.211 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-03-23T23:14:13.211 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.211 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.211 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.211 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.211 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-03-23T23:14:13.211 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.211 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.211 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.212 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.212 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-03-23T23:14:13.212 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.212 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.212 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.212 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.212 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-03-23T23:14:13.212 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-03-23T23:14:13.212 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-03-23T23:14:13.212 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.212 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.212 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.212 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:13.212 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.212 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-03-23T23:14:13.213 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 0, 2024-03-23T23:14:13.213 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 0, 2024-03-23T23:14:13.213 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 0, 2024-03-23T23:14:13.213 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 0, 2024-03-23T23:14:13.213 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-03-23T23:14:13.213 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 0, 2024-03-23T23:14:13.213 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 0, 2024-03-23T23:14:13.213 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 0, 2024-03-23T23:14:13.213 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-03-23T23:14:13.213 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.213 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-03-23T23:14:13.213 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-03-23T23:14:13.213 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-03-23T23:14:13.213 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-03-23T23:14:13.213 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-03-23T23:14:13.214 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-03-23T23:14:13.214 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-03-23T23:14:13.214 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-23T23:14:13.214 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-03-23T23:14:13.214 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-23T23:14:13.214 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-03-23T23:14:13.214 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-03-23T23:14:13.214 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-03-23T23:14:13.214 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-23T23:14:13.214 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-03-23T23:14:13.214 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-03-23T23:14:13.214 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-03-23T23:14:13.214 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-03-23T23:14:13.214 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-03-23T23:14:13.215 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-23T23:14:13.215 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-03-23T23:14:13.215 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-23T23:14:13.215 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-03-23T23:14:13.215 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-03-23T23:14:13.215 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-03-23T23:14:13.215 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-03-23T23:14:13.215 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-03-23T23:14:13.215 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-23T23:14:13.215 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-03-23T23:14:13.215 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-03-23T23:14:13.215 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-03-23T23:14:13.215 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-03-23T23:14:13.215 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-03-23T23:14:13.215 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-03-23T23:14:13.216 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-03-23T23:14:13.216 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-03-23T23:14:13.216 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-03-23T23:14:13.216 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 181281, 2024-03-23T23:14:13.216 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 46, 2024-03-23T23:14:13.216 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-03-23T23:14:13.216 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-03-23T23:14:13.216 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-03-23T23:14:13.216 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-03-23T23:14:13.216 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-03-23T23:14:13.216 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-03-23T23:14:13.216 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-03-23T23:14:13.216 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-03-23T23:14:13.216 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 7536, 2024-03-23T23:14:13.216 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 159, 2024-03-23T23:14:13.216 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-03-23T23:14:13.217 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-03-23T23:14:13.217 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-03-23T23:14:13.217 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-03-23T23:14:13.217 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 8320, 2024-03-23T23:14:13.217 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 13, 2024-03-23T23:14:13.217 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-03-23T23:14:13.217 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-03-23T23:14:13.217 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-03-23T23:14:13.217 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-03-23T23:14:13.217 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.217 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-03-23T23:14:13.217 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-03-23T23:14:13.217 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-03-23T23:14:13.217 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 1766, 2024-03-23T23:14:13.218 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 21, 2024-03-23T23:14:13.218 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 3, 2024-03-23T23:14:13.218 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 1766, 2024-03-23T23:14:13.218 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-03-23T23:14:13.218 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1766, 2024-03-23T23:14:13.218 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 1.391965873, 2024-03-23T23:14:13.218 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000788202 2024-03-23T23:14:13.218 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.218 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-03-23T23:14:13.218 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-03-23T23:14:13.218 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1766, 2024-03-23T23:14:13.219 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 1767 2024-03-23T23:14:13.219 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.219 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 1766, 2024-03-23T23:14:13.219 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 1766, 2024-03-23T23:14:13.219 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 0, 2024-03-23T23:14:13.219 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-03-23T23:14:13.219 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-03-23T23:14:13.219 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 880, 2024-03-23T23:14:13.219 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 0, 2024-03-23T23:14:13.219 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 884, 2024-03-23T23:14:13.219 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 0, 2024-03-23T23:14:13.219 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 0, 2024-03-23T23:14:13.219 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-03-23T23:14:13.219 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-03-23T23:14:13.219 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 0, 2024-03-23T23:14:13.219 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 0, 2024-03-23T23:14:13.220 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 0, 2024-03-23T23:14:13.220 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-03-23T23:14:13.220 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-03-23T23:14:13.220 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-03-23T23:14:13.220 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 1, 2024-03-23T23:14:13.220 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 0, 2024-03-23T23:14:13.220 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-03-23T23:14:13.220 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-03-23T23:14:13.220 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-03-23T23:14:13.220 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-03-23T23:14:13.220 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-03-23T23:14:13.220 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-03-23T23:14:13.220 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-03-23T23:14:13.220 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-03-23T23:14:13.221 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-03-23T23:14:13.221 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 0, 2024-03-23T23:14:13.221 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-03-23T23:14:13.221 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-03-23T23:14:13.221 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-03-23T23:14:13.221 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-03-23T23:14:13.221 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-03-23T23:14:13.221 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-03-23T23:14:13.221 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-03-23T23:14:13.221 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-03-23T23:14:13.221 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-03-23T23:14:13.221 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-03-23T23:14:13.221 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-03-23T23:14:13.221 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-03-23T23:14:13.222 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-03-23T23:14:13.222 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-03-23T23:14:13.222 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-03-23T23:14:13.222 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-03-23T23:14:13.222 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 88, 2024-03-23T23:14:13.222 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-03-23T23:14:13.222 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 2, 2024-03-23T23:14:13.222 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 4, 2024-03-23T23:14:13.222 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 4, 2024-03-23T23:14:13.222 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 0, 2024-03-23T23:14:13.222 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-03-23T23:14:13.222 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 0, 2024-03-23T23:14:13.223 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 2, 2024-03-23T23:14:13.223 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 0 2024-03-23T23:14:13.223 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.223 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-03-23T23:14:13.223 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 0, 2024-03-23T23:14:13.223 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-03-23T23:14:13.223 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-03-23T23:14:13.223 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-03-23T23:14:13.223 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.223 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-03-23T23:14:13.223 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 0 2024-03-23T23:14:13.223 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.223 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-03-23T23:14:13.223 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-03-23T23:14:13.223 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 0, 2024-03-23T23:14:13.224 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-03-23T23:14:13.224 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 0, 2024-03-23T23:14:13.224 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 0, 2024-03-23T23:14:13.224 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 0, 2024-03-23T23:14:13.224 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-03-23T23:14:13.224 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.224 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-23T23:14:13.224 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-23T23:14:13.224 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-03-23T23:14:13.224 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-23T23:14:13.224 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 2022, 2024-03-23T23:14:13.224 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 836836, 2024-03-23T23:14:13.224 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:13.224 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 2022, 2024-03-23T23:14:13.224 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-03-23T23:14:13.224 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-03-23T23:14:13.225 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 2022, 2024-03-23T23:14:13.225 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 836836, 2024-03-23T23:14:13.225 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-23T23:14:13.225 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.225 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.225 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.225 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:13.225 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.225 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-03-23T23:14:13.225 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-23T23:14:13.225 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-03-23T23:14:13.225 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-23T23:14:13.225 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-03-23T23:14:13.225 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-03-23T23:14:13.225 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:13.226 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-03-23T23:14:13.226 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 1766, 2024-03-23T23:14:13.226 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 593, 2024-03-23T23:14:13.226 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 1766, 2024-03-23T23:14:13.226 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 593, 2024-03-23T23:14:13.226 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-23T23:14:13.226 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.226 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.226 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.226 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:13.226 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.226 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-03-23T23:14:13.226 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-23T23:14:13.226 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-03-23T23:14:13.226 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-23T23:14:13.227 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-03-23T23:14:13.227 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-03-23T23:14:13.227 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:13.227 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-03-23T23:14:13.227 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 1766, 2024-03-23T23:14:13.227 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 1766, 2024-03-23T23:14:13.227 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 1766, 2024-03-23T23:14:13.227 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 1766, 2024-03-23T23:14:13.227 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-23T23:14:13.227 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.227 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.227 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.227 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:13.227 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.227 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-03-23T23:14:13.227 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-23T23:14:13.228 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-03-23T23:14:13.228 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-23T23:14:13.228 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-03-23T23:14:13.228 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-03-23T23:14:13.228 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:13.228 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-03-23T23:14:13.228 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-03-23T23:14:13.228 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-03-23T23:14:13.228 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-03-23T23:14:13.228 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-03-23T23:14:13.228 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-23T23:14:13.228 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.228 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.228 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.229 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:13.229 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:13.229 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x562951e466e0": { 2024-03-23T23:14:13.229 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-23T23:14:13.229 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-03-23T23:14:13.229 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-23T23:14:13.229 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-03-23T23:14:13.229 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-03-23T23:14:13.229 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:13.229 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-03-23T23:14:13.229 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-03-23T23:14:13.229 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-03-23T23:14:13.229 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-03-23T23:14:13.229 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-03-23T23:14:13.229 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-23T23:14:13.229 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:13.229 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:13.230 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:13.230 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:13.230 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:13.230 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-23T23:14:13.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:12 smithi173 ceph-mon[36116]: pgmap v762: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-23T23:14:13.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:12 smithi012 ceph-mon[31695]: pgmap v762: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-23T23:14:14.015 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-23T23:14:14.015 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-23T23:14:14.015 DEBUG:tasks.check_counter:Getting stats from i 2024-03-23T23:14:14.016 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-23T23:14:14.259 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.c/config 2024-03-23T23:14:15.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:14 smithi203 ceph-mon[36693]: pgmap v763: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-23T23:14:15.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:14 smithi012 ceph-mon[31695]: pgmap v763: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-23T23:14:15.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:14 smithi173 ceph-mon[36116]: pgmap v763: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-23T23:14:16.020 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-23T23:14:16.020 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-03-23T23:14:16.020 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 69, 2024-03-23T23:14:16.020 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 77, 2024-03-23T23:14:16.020 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 14442, 2024-03-23T23:14:16.020 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 624466, 2024-03-23T23:14:16.020 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 58, 2024-03-23T23:14:16.020 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 7, 2024-03-23T23:14:16.020 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.018221645, 2024-03-23T23:14:16.020 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.001969621, 2024-03-23T23:14:16.020 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.012345185, 2024-03-23T23:14:16.021 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.000355671, 2024-03-23T23:14:16.021 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:14:16.021 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 70, 2024-03-23T23:14:16.021 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.032790026, 2024-03-23T23:14:16.021 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000468428 2024-03-23T23:14:16.021 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.021 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:14:16.021 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 48, 2024-03-23T23:14:16.021 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000005258, 2024-03-23T23:14:16.021 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000109 2024-03-23T23:14:16.021 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.021 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-23T23:14:16.021 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 91104 2024-03-23T23:14:16.021 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.022 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-03-23T23:14:16.022 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 72, 2024-03-23T23:14:16.022 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 252, 2024-03-23T23:14:16.022 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 24167, 2024-03-23T23:14:16.022 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 431105, 2024-03-23T23:14:16.022 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 7, 2024-03-23T23:14:16.022 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 3, 2024-03-23T23:14:16.022 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.011957052, 2024-03-23T23:14:16.022 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.006444981, 2024-03-23T23:14:16.022 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.004203058, 2024-03-23T23:14:16.022 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.000346706, 2024-03-23T23:14:16.022 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:14:16.022 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 252, 2024-03-23T23:14:16.023 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.026765670, 2024-03-23T23:14:16.023 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000106212 2024-03-23T23:14:16.023 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.023 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:14:16.023 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 109, 2024-03-23T23:14:16.023 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000021160, 2024-03-23T23:14:16.023 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000194 2024-03-23T23:14:16.023 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.023 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-23T23:14:16.023 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 398784 2024-03-23T23:14:16.023 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.023 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-03-23T23:14:16.023 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 811, 2024-03-23T23:14:16.023 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 914, 2024-03-23T23:14:16.024 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 621270, 2024-03-23T23:14:16.024 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 254860, 2024-03-23T23:14:16.024 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 5, 2024-03-23T23:14:16.024 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 3, 2024-03-23T23:14:16.024 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.066391517, 2024-03-23T23:14:16.024 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.024956511, 2024-03-23T23:14:16.024 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.037146454, 2024-03-23T23:14:16.024 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.007026491, 2024-03-23T23:14:16.024 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:14:16.024 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 917, 2024-03-23T23:14:16.024 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.027980380, 2024-03-23T23:14:16.024 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000030512 2024-03-23T23:14:16.024 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.024 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:14:16.024 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1149, 2024-03-23T23:14:16.025 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000203316, 2024-03-23T23:14:16.025 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000176 2024-03-23T23:14:16.025 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.025 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 509744, 2024-03-23T23:14:16.025 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 152320 2024-03-23T23:14:16.025 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.025 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-03-23T23:14:16.025 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-03-23T23:14:16.025 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-03-23T23:14:16.025 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.025 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-03-23T23:14:16.025 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-03-23T23:14:16.025 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-03-23T23:14:16.025 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 43, 2024-03-23T23:14:16.026 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.001808243, 2024-03-23T23:14:16.026 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000042052 2024-03-23T23:14:16.026 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:16.026 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.026 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-03-23T23:14:16.026 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-03-23T23:14:16.026 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-03-23T23:14:16.026 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2, 2024-03-23T23:14:16.026 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000007589, 2024-03-23T23:14:16.026 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000003794 2024-03-23T23:14:16.026 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:16.026 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.026 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-03-23T23:14:16.026 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 0, 2024-03-23T23:14:16.027 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 0, 2024-03-23T23:14:16.027 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-03-23T23:14:16.027 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.027 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.027 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.027 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.027 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-03-23T23:14:16.027 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 0, 2024-03-23T23:14:16.027 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 0, 2024-03-23T23:14:16.027 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 0, 2024-03-23T23:14:16.027 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 11, 2024-03-23T23:14:16.027 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 0, 2024-03-23T23:14:16.027 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-03-23T23:14:16.027 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 10, 2024-03-23T23:14:16.028 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 0, 2024-03-23T23:14:16.028 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 10, 2024-03-23T23:14:16.028 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 0, 2024-03-23T23:14:16.028 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 10, 2024-03-23T23:14:16.028 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 0, 2024-03-23T23:14:16.028 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 0, 2024-03-23T23:14:16.028 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 0, 2024-03-23T23:14:16.028 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 1, 2024-03-23T23:14:16.028 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 0, 2024-03-23T23:14:16.028 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 0, 2024-03-23T23:14:16.028 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 0, 2024-03-23T23:14:16.028 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 0, 2024-03-23T23:14:16.028 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 0, 2024-03-23T23:14:16.028 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-03-23T23:14:16.028 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 0, 2024-03-23T23:14:16.029 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 0, 2024-03-23T23:14:16.029 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-03-23T23:14:16.029 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 0, 2024-03-23T23:14:16.029 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 0, 2024-03-23T23:14:16.029 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 0, 2024-03-23T23:14:16.029 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 0, 2024-03-23T23:14:16.029 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-03-23T23:14:16.029 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 0, 2024-03-23T23:14:16.029 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 0, 2024-03-23T23:14:16.029 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 5, 2024-03-23T23:14:16.029 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 298, 2024-03-23T23:14:16.029 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 0, 2024-03-23T23:14:16.029 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 0, 2024-03-23T23:14:16.029 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-03-23T23:14:16.029 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-03-23T23:14:16.030 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-03-23T23:14:16.030 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 0, 2024-03-23T23:14:16.030 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-03-23T23:14:16.030 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 0, 2024-03-23T23:14:16.030 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 0, 2024-03-23T23:14:16.030 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 0, 2024-03-23T23:14:16.030 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 0, 2024-03-23T23:14:16.030 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 0, 2024-03-23T23:14:16.030 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 0, 2024-03-23T23:14:16.030 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-23T23:14:16.030 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-23T23:14:16.030 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 0, 2024-03-23T23:14:16.030 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 0, 2024-03-23T23:14:16.030 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 0, 2024-03-23T23:14:16.031 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 0 2024-03-23T23:14:16.031 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.031 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-03-23T23:14:16.031 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 0, 2024-03-23T23:14:16.031 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 0, 2024-03-23T23:14:16.031 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 0, 2024-03-23T23:14:16.031 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 1, 2024-03-23T23:14:16.031 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 0, 2024-03-23T23:14:16.031 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 0, 2024-03-23T23:14:16.031 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-03-23T23:14:16.031 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-03-23T23:14:16.031 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 0, 2024-03-23T23:14:16.031 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 0, 2024-03-23T23:14:16.032 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 0, 2024-03-23T23:14:16.032 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 0, 2024-03-23T23:14:16.032 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-03-23T23:14:16.032 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-03-23T23:14:16.032 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-03-23T23:14:16.032 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-03-23T23:14:16.032 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-03-23T23:14:16.032 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-03-23T23:14:16.032 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 0, 2024-03-23T23:14:16.032 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-03-23T23:14:16.032 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 0, 2024-03-23T23:14:16.032 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-03-23T23:14:16.032 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-03-23T23:14:16.032 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.033 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-03-23T23:14:16.033 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-03-23T23:14:16.033 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 2, 2024-03-23T23:14:16.033 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-03-23T23:14:16.033 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-03-23T23:14:16.033 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 2, 2024-03-23T23:14:16.033 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-03-23T23:14:16.033 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-03-23T23:14:16.033 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 2, 2024-03-23T23:14:16.033 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-03-23T23:14:16.033 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-03-23T23:14:16.033 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 2, 2024-03-23T23:14:16.033 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 2, 2024-03-23T23:14:16.033 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-03-23T23:14:16.033 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-03-23T23:14:16.034 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 4194304, 2024-03-23T23:14:16.034 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 4194892, 2024-03-23T23:14:16.034 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 0, 2024-03-23T23:14:16.034 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-03-23T23:14:16.034 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1, 2024-03-23T23:14:16.034 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.001319663, 2024-03-23T23:14:16.034 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.001319663 2024-03-23T23:14:16.034 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.034 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 0 2024-03-23T23:14:16.034 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.034 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-03-23T23:14:16.034 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 13, 2024-03-23T23:14:16.034 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 13, 2024-03-23T23:14:16.034 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 0, 2024-03-23T23:14:16.035 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 11, 2024-03-23T23:14:16.035 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 12, 2024-03-23T23:14:16.035 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 1, 2024-03-23T23:14:16.035 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 10, 2024-03-23T23:14:16.035 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 10, 2024-03-23T23:14:16.035 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 0, 2024-03-23T23:14:16.035 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 0, 2024-03-23T23:14:16.035 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 0, 2024-03-23T23:14:16.035 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 0, 2024-03-23T23:14:16.035 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 50136, 2024-03-23T23:14:16.035 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 223612 2024-03-23T23:14:16.035 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.035 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-03-23T23:14:16.035 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 0, 2024-03-23T23:14:16.036 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 0, 2024-03-23T23:14:16.036 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 0, 2024-03-23T23:14:16.036 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 0, 2024-03-23T23:14:16.036 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 0, 2024-03-23T23:14:16.036 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-03-23T23:14:16.036 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.036 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.036 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.036 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.036 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-03-23T23:14:16.036 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.036 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.036 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.036 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.036 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-03-23T23:14:16.036 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.037 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.037 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.037 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.037 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-03-23T23:14:16.037 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.037 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.037 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.037 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.037 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-03-23T23:14:16.037 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.037 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.037 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.037 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.037 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-03-23T23:14:16.037 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.038 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.038 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.038 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.038 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-03-23T23:14:16.038 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.038 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.038 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.038 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.038 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-03-23T23:14:16.038 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.038 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.038 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.038 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.038 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-03-23T23:14:16.039 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.039 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.039 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.039 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.039 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-03-23T23:14:16.039 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.039 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.039 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.039 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.039 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-03-23T23:14:16.039 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.039 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.039 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.039 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.039 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-03-23T23:14:16.040 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.040 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.040 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.040 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.040 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-03-23T23:14:16.040 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.041 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.041 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.041 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.041 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-03-23T23:14:16.041 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.041 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.041 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.041 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.041 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-03-23T23:14:16.041 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.041 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.041 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.041 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.041 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-03-23T23:14:16.042 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.042 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.042 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.042 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.042 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-03-23T23:14:16.042 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.042 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.042 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.042 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.042 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-03-23T23:14:16.042 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.042 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.042 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.042 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.043 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-03-23T23:14:16.043 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.043 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.043 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.043 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.043 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-03-23T23:14:16.043 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.043 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.043 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.043 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.043 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-03-23T23:14:16.043 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.043 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.043 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.043 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.044 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-03-23T23:14:16.044 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.044 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.044 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.044 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.044 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-03-23T23:14:16.044 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.044 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.044 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.044 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.044 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-03-23T23:14:16.044 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.044 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.044 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.044 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.045 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-03-23T23:14:16.045 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.045 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.045 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.045 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.045 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-03-23T23:14:16.045 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.045 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.045 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.045 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.045 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-03-23T23:14:16.045 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.045 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.045 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.045 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.045 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-03-23T23:14:16.046 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.046 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.046 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.046 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.046 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-03-23T23:14:16.046 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.046 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.046 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.046 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.046 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-03-23T23:14:16.046 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-03-23T23:14:16.046 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-03-23T23:14:16.046 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.047 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.047 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.047 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:16.047 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.047 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-03-23T23:14:16.047 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 0, 2024-03-23T23:14:16.047 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 0, 2024-03-23T23:14:16.047 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 0, 2024-03-23T23:14:16.047 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 0, 2024-03-23T23:14:16.047 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-03-23T23:14:16.047 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 0, 2024-03-23T23:14:16.047 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 0, 2024-03-23T23:14:16.047 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 0, 2024-03-23T23:14:16.047 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-03-23T23:14:16.047 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.048 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-03-23T23:14:16.048 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-03-23T23:14:16.048 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-03-23T23:14:16.048 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-03-23T23:14:16.048 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-03-23T23:14:16.048 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-03-23T23:14:16.048 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-03-23T23:14:16.048 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-23T23:14:16.048 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-03-23T23:14:16.048 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-23T23:14:16.048 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-03-23T23:14:16.048 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-03-23T23:14:16.048 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-03-23T23:14:16.048 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-23T23:14:16.049 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-03-23T23:14:16.049 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-03-23T23:14:16.049 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-03-23T23:14:16.049 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-03-23T23:14:16.049 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-03-23T23:14:16.049 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-23T23:14:16.049 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-03-23T23:14:16.049 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-23T23:14:16.049 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-03-23T23:14:16.049 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-03-23T23:14:16.049 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-03-23T23:14:16.049 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-03-23T23:14:16.049 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-03-23T23:14:16.049 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-23T23:14:16.050 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-03-23T23:14:16.050 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-03-23T23:14:16.050 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-03-23T23:14:16.050 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-03-23T23:14:16.050 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-03-23T23:14:16.050 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-03-23T23:14:16.050 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-03-23T23:14:16.050 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-03-23T23:14:16.050 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-03-23T23:14:16.050 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 221911, 2024-03-23T23:14:16.050 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 67, 2024-03-23T23:14:16.050 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-03-23T23:14:16.050 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-03-23T23:14:16.051 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-03-23T23:14:16.051 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-03-23T23:14:16.051 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-03-23T23:14:16.051 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-03-23T23:14:16.051 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-03-23T23:14:16.051 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-03-23T23:14:16.051 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 7536, 2024-03-23T23:14:16.051 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 159, 2024-03-23T23:14:16.051 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-03-23T23:14:16.051 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-03-23T23:14:16.051 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-03-23T23:14:16.051 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-03-23T23:14:16.051 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 50040, 2024-03-23T23:14:16.051 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 198, 2024-03-23T23:14:16.051 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-03-23T23:14:16.051 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-03-23T23:14:16.052 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-03-23T23:14:16.052 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-03-23T23:14:16.052 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.052 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-03-23T23:14:16.052 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-03-23T23:14:16.052 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-03-23T23:14:16.052 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 35, 2024-03-23T23:14:16.052 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 10710, 2024-03-23T23:14:16.052 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 33, 2024-03-23T23:14:16.052 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 35, 2024-03-23T23:14:16.052 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-03-23T23:14:16.052 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 35, 2024-03-23T23:14:16.052 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.315992100, 2024-03-23T23:14:16.052 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.009028345 2024-03-23T23:14:16.052 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.052 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-03-23T23:14:16.053 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-03-23T23:14:16.053 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 35, 2024-03-23T23:14:16.053 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 57 2024-03-23T23:14:16.053 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.053 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 35, 2024-03-23T23:14:16.053 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 0, 2024-03-23T23:14:16.053 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 35, 2024-03-23T23:14:16.053 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-03-23T23:14:16.053 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-03-23T23:14:16.053 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 0, 2024-03-23T23:14:16.053 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 10, 2024-03-23T23:14:16.053 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 0, 2024-03-23T23:14:16.053 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 1, 2024-03-23T23:14:16.053 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 5, 2024-03-23T23:14:16.053 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-03-23T23:14:16.054 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-03-23T23:14:16.054 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 0, 2024-03-23T23:14:16.054 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 0, 2024-03-23T23:14:16.054 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 6, 2024-03-23T23:14:16.054 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-03-23T23:14:16.054 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-03-23T23:14:16.054 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-03-23T23:14:16.054 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 0, 2024-03-23T23:14:16.054 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 20, 2024-03-23T23:14:16.055 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-03-23T23:14:16.055 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-03-23T23:14:16.055 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-03-23T23:14:16.055 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-03-23T23:14:16.055 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-03-23T23:14:16.055 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-03-23T23:14:16.055 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-03-23T23:14:16.055 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-03-23T23:14:16.055 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-03-23T23:14:16.055 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 0, 2024-03-23T23:14:16.055 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-03-23T23:14:16.055 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-03-23T23:14:16.055 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-03-23T23:14:16.055 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-03-23T23:14:16.055 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-03-23T23:14:16.055 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-03-23T23:14:16.056 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-03-23T23:14:16.056 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-03-23T23:14:16.056 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-03-23T23:14:16.056 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-03-23T23:14:16.056 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-03-23T23:14:16.056 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-03-23T23:14:16.056 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-03-23T23:14:16.056 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-03-23T23:14:16.056 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-03-23T23:14:16.056 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-03-23T23:14:16.056 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 88, 2024-03-23T23:14:16.056 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-03-23T23:14:16.056 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 2, 2024-03-23T23:14:16.056 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 8, 2024-03-23T23:14:16.056 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 8, 2024-03-23T23:14:16.057 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 0, 2024-03-23T23:14:16.057 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-03-23T23:14:16.057 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 14, 2024-03-23T23:14:16.057 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 0, 2024-03-23T23:14:16.057 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 1 2024-03-23T23:14:16.057 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.057 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-03-23T23:14:16.057 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 1, 2024-03-23T23:14:16.057 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-03-23T23:14:16.057 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-03-23T23:14:16.057 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-03-23T23:14:16.057 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.057 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-03-23T23:14:16.057 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 0 2024-03-23T23:14:16.058 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.058 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-03-23T23:14:16.058 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-03-23T23:14:16.058 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 0, 2024-03-23T23:14:16.058 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-03-23T23:14:16.058 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 0, 2024-03-23T23:14:16.058 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 0, 2024-03-23T23:14:16.058 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 0, 2024-03-23T23:14:16.058 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-03-23T23:14:16.058 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.058 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-23T23:14:16.058 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-23T23:14:16.058 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-03-23T23:14:16.058 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-23T23:14:16.059 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 948, 2024-03-23T23:14:16.059 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 557051, 2024-03-23T23:14:16.059 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:16.059 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 948, 2024-03-23T23:14:16.059 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-03-23T23:14:16.059 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-03-23T23:14:16.059 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 948, 2024-03-23T23:14:16.059 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 557051, 2024-03-23T23:14:16.059 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-23T23:14:16.059 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.059 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.059 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.059 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:16.059 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.059 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-03-23T23:14:16.060 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-23T23:14:16.060 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-03-23T23:14:16.060 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-23T23:14:16.060 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-03-23T23:14:16.060 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-03-23T23:14:16.060 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:16.060 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-03-23T23:14:16.060 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 35, 2024-03-23T23:14:16.060 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 10710, 2024-03-23T23:14:16.060 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 35, 2024-03-23T23:14:16.060 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 10710, 2024-03-23T23:14:16.060 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-23T23:14:16.060 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.060 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.061 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.061 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:16.061 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.061 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-03-23T23:14:16.061 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-23T23:14:16.061 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-03-23T23:14:16.061 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-23T23:14:16.061 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-03-23T23:14:16.061 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-03-23T23:14:16.061 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:16.061 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-03-23T23:14:16.061 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 35, 2024-03-23T23:14:16.061 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 35, 2024-03-23T23:14:16.062 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 35, 2024-03-23T23:14:16.062 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 35, 2024-03-23T23:14:16.062 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-23T23:14:16.062 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.062 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.062 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.062 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:16.062 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.062 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-03-23T23:14:16.062 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-23T23:14:16.062 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-03-23T23:14:16.062 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-23T23:14:16.062 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-03-23T23:14:16.062 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-03-23T23:14:16.063 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:16.063 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-03-23T23:14:16.063 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-03-23T23:14:16.063 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-03-23T23:14:16.063 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-03-23T23:14:16.063 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-03-23T23:14:16.063 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-23T23:14:16.063 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.063 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.063 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.063 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:16.064 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:16.064 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x56019f394500": { 2024-03-23T23:14:16.064 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-23T23:14:16.064 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-03-23T23:14:16.064 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-23T23:14:16.064 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 2, 2024-03-23T23:14:16.064 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 588, 2024-03-23T23:14:16.064 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:16.064 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 2, 2024-03-23T23:14:16.064 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-03-23T23:14:16.064 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-03-23T23:14:16.064 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 1, 2024-03-23T23:14:16.064 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 588, 2024-03-23T23:14:16.064 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-23T23:14:16.064 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:16.065 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:16.065 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:16.065 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:16.065 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:16.065 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-23T23:14:16.834 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-23T23:14:16.834 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-03-23T23:14:16.834 DEBUG:tasks.check_counter:Getting stats from l 2024-03-23T23:14:16.835 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 90f679ae-e967-11ee-95cd-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-23T23:14:17.087 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/mon.c/config 2024-03-23T23:14:17.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:16 smithi203 ceph-mon[36693]: pgmap v764: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:14:17.223 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:16 smithi012 ceph-mon[31695]: pgmap v764: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:14:17.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:16 smithi173 ceph-mon[36116]: pgmap v764: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-23T23:14:18.906 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-23T23:14:18.906 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-03-23T23:14:18.907 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 0, 2024-03-23T23:14:18.907 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 2, 2024-03-23T23:14:18.907 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 0, 2024-03-23T23:14:18.907 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 0, 2024-03-23T23:14:18.907 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 4, 2024-03-23T23:14:18.907 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 0, 2024-03-23T23:14:18.907 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.000879061, 2024-03-23T23:14:18.907 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.000000000, 2024-03-23T23:14:18.907 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.000090076, 2024-03-23T23:14:18.907 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-23T23:14:18.907 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:14:18.907 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:18.907 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:18.907 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:18.908 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:18.908 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:14:18.908 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:18.908 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:18.908 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:18.908 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:18.908 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-23T23:14:18.908 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 0 2024-03-23T23:14:18.908 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:18.908 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-03-23T23:14:18.908 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 256, 2024-03-23T23:14:18.908 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 251, 2024-03-23T23:14:18.908 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 489104, 2024-03-23T23:14:18.908 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 150528, 2024-03-23T23:14:18.908 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 2, 2024-03-23T23:14:18.908 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 1, 2024-03-23T23:14:18.909 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.026917415, 2024-03-23T23:14:18.909 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.009531117, 2024-03-23T23:14:18.909 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.013718847, 2024-03-23T23:14:18.909 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.004192769, 2024-03-23T23:14:18.909 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:14:18.909 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 251, 2024-03-23T23:14:18.909 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.009271634, 2024-03-23T23:14:18.909 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000036938 2024-03-23T23:14:18.909 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:18.909 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:14:18.909 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:18.909 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:18.909 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:18.909 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:18.910 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 489104, 2024-03-23T23:14:18.910 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 150528 2024-03-23T23:14:18.910 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:18.910 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-03-23T23:14:18.910 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 2, 2024-03-23T23:14:18.910 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 191, 2024-03-23T23:14:18.910 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 320, 2024-03-23T23:14:18.910 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 188912, 2024-03-23T23:14:18.910 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 4, 2024-03-23T23:14:18.910 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 2, 2024-03-23T23:14:18.910 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.007194161, 2024-03-23T23:14:18.910 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.005037284, 2024-03-23T23:14:18.910 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.000853681, 2024-03-23T23:14:18.911 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-23T23:14:18.911 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-03-23T23:14:18.911 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 189, 2024-03-23T23:14:18.911 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.011183912, 2024-03-23T23:14:18.911 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000059174 2024-03-23T23:14:18.911 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:18.911 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-03-23T23:14:18.911 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:18.911 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:18.911 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:18.911 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:18.911 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-23T23:14:18.911 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 188912 2024-03-23T23:14:18.911 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:18.912 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-03-23T23:14:18.912 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 0, 2024-03-23T23:14:18.912 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-03-23T23:14:18.912 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:18.912 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-03-23T23:14:18.912 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-03-23T23:14:18.912 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-03-23T23:14:18.912 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-03-23T23:14:18.912 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-03-23T23:14:18.912 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-03-23T23:14:18.912 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-03-23T23:14:18.912 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-23T23:14:18.912 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-03-23T23:14:18.912 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-23T23:14:18.913 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-03-23T23:14:18.913 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-03-23T23:14:18.913 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-03-23T23:14:18.913 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-23T23:14:18.913 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-03-23T23:14:18.913 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-03-23T23:14:18.913 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-03-23T23:14:18.913 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-03-23T23:14:18.913 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-03-23T23:14:18.913 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-23T23:14:18.913 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-03-23T23:14:18.913 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-23T23:14:18.913 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-03-23T23:14:18.913 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-03-23T23:14:18.914 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-03-23T23:14:18.914 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-03-23T23:14:18.914 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-03-23T23:14:18.914 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-23T23:14:18.914 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-03-23T23:14:18.914 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-03-23T23:14:18.914 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-03-23T23:14:18.914 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-03-23T23:14:18.914 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-03-23T23:14:18.914 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-03-23T23:14:18.914 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-03-23T23:14:18.914 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-03-23T23:14:18.914 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-03-23T23:14:18.914 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 148722, 2024-03-23T23:14:18.915 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 30, 2024-03-23T23:14:18.915 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-03-23T23:14:18.915 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-03-23T23:14:18.915 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-03-23T23:14:18.915 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-03-23T23:14:18.915 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-03-23T23:14:18.915 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-03-23T23:14:18.915 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-03-23T23:14:18.915 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-03-23T23:14:18.915 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 0, 2024-03-23T23:14:18.915 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 0, 2024-03-23T23:14:18.915 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-03-23T23:14:18.915 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-03-23T23:14:18.915 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-03-23T23:14:18.916 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-03-23T23:14:18.916 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 928, 2024-03-23T23:14:18.916 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 2, 2024-03-23T23:14:18.916 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-03-23T23:14:18.916 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-03-23T23:14:18.916 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-03-23T23:14:18.916 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-03-23T23:14:18.916 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-23T23:14:18.916 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-23T23:14:18.916 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-23T23:14:18.916 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-03-23T23:14:18.916 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-23T23:14:18.916 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 254, 2024-03-23T23:14:18.916 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 450519, 2024-03-23T23:14:18.917 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-23T23:14:18.917 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 254, 2024-03-23T23:14:18.917 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-03-23T23:14:18.917 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-03-23T23:14:18.917 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 254, 2024-03-23T23:14:18.917 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 450519, 2024-03-23T23:14:18.917 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-23T23:14:18.917 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-23T23:14:18.917 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-23T23:14:18.917 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-23T23:14:18.917 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:18.917 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-23T23:14:18.917 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-23T23:14:19.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:18 smithi203 ceph-mon[36693]: pgmap v765: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:14:19.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:18 smithi012 ceph-mon[31695]: pgmap v765: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:14:19.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:18 smithi173 ceph-mon[36116]: pgmap v765: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-23T23:14:19.611 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-03-23T23:14:19.877 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-03-23T23:14:19.877 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-23T23:14:19.990 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-23T23:14:19.990 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-23T23:14:19.990 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T23:14:20.057 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-23T23:14:20.057 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T23:14:20.112 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T23:14:20.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.345+0000 7f1bdd800640 1 -- 172.21.15.12:0/3928335252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bd8101f70 msgr2=0x7f1bd8104360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:20.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.345+0000 7f1bdd800640 1 --2- 172.21.15.12:0/3928335252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bd8101f70 0x7f1bd8104360 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8009900 tx=0x7f1bc802bc40 comp rx=0 tx=0).stop 2024-03-23T23:14:20.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.345+0000 7f1bdd800640 1 -- 172.21.15.12:0/3928335252 shutdown_connections 2024-03-23T23:14:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.345+0000 7f1bdd800640 1 --2- 172.21.15.12:0/3928335252 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1bd81071d0 0x7f1bd8109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.345+0000 7f1bdd800640 1 --2- 172.21.15.12:0/3928335252 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bd81048a0 0x7f1bd8106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.345+0000 7f1bdd800640 1 --2- 172.21.15.12:0/3928335252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bd8101f70 0x7f1bd8104360 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.345+0000 7f1bdd800640 1 -- 172.21.15.12:0/3928335252 >> 172.21.15.12:0/3928335252 conn(0x7f1bd80fb980 msgr2=0x7f1bd80fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:14:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.345+0000 7f1bdd800640 1 -- 172.21.15.12:0/3928335252 shutdown_connections 2024-03-23T23:14:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.345+0000 7f1bdd800640 1 -- 172.21.15.12:0/3928335252 wait complete. 2024-03-23T23:14:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.347+0000 7f1bdd800640 1 Processor -- start 2024-03-23T23:14:20.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.347+0000 7f1bdd800640 1 -- start start 2024-03-23T23:14:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.347+0000 7f1bdd800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bd8101f70 0x7f1bd8199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.347+0000 7f1bdd800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1bd81048a0 0x7f1bd8199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.347+0000 7f1bdd800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bd81071d0 0x7f1bd81a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.347+0000 7f1bdd800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd810d1d0 con 0x7f1bd8101f70 2024-03-23T23:14:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.347+0000 7f1bdd800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd810d050 con 0x7f1bd81048a0 2024-03-23T23:14:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.347+0000 7f1bdd800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd810d350 con 0x7f1bd81071d0 2024-03-23T23:14:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.348+0000 7f1bd7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1bd81048a0 0x7f1bd8199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.348+0000 7f1bd7e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bd8101f70 0x7f1bd8199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.348+0000 7f1bd7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1bd81048a0 0x7f1bd8199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:46954/0 (socket says 172.21.15.12:46954) 2024-03-23T23:14:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.348+0000 7f1bd7400640 1 -- 172.21.15.12:0/3099865187 learned_addr learned my addr 172.21.15.12:0/3099865187 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:14:20.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.348+0000 7f1bdce00640 1 --2- 172.21.15.12:0/3099865187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bd81071d0 0x7f1bd81a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:20.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.348+0000 7f1bd7400640 1 -- 172.21.15.12:0/3099865187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bd81071d0 msgr2=0x7f1bd81a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:20.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.348+0000 7f1bd7400640 1 --2- 172.21.15.12:0/3099865187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bd81071d0 0x7f1bd81a0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:20.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.348+0000 7f1bd7400640 1 -- 172.21.15.12:0/3099865187 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bd8101f70 msgr2=0x7f1bd8199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:20.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.348+0000 7f1bd7400640 1 --2- 172.21.15.12:0/3099865187 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bd8101f70 0x7f1bd8199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:20.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.348+0000 7f1bd7400640 1 -- 172.21.15.12:0/3099865187 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd81a1640 con 0x7f1bd81048a0 2024-03-23T23:14:20.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.348+0000 7f1bd7400640 1 --2- 172.21.15.12:0/3099865187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1bd81048a0 0x7f1bd8199f60 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f1bc400e9c0 tx=0x7f1bc400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:14:20.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.349+0000 7f1bd4c00640 1 -- 172.21.15.12:0/3099865187 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc400cd80 con 0x7f1bd81048a0 2024-03-23T23:14:20.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.349+0000 7f1bd4c00640 1 -- 172.21.15.12:0/3099865187 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1bc4005500 con 0x7f1bd81048a0 2024-03-23T23:14:20.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.349+0000 7f1bd4c00640 1 -- 172.21.15.12:0/3099865187 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc4010690 con 0x7f1bd81048a0 2024-03-23T23:14:20.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.349+0000 7f1bdd800640 1 -- 172.21.15.12:0/3099865187 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bd81a1930 con 0x7f1bd81048a0 2024-03-23T23:14:20.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.349+0000 7f1bdd800640 1 -- 172.21.15.12:0/3099865187 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bd81a1e30 con 0x7f1bd81048a0 2024-03-23T23:14:20.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.350+0000 7f1bdd800640 1 -- 172.21.15.12:0/3099865187 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bd8108d20 con 0x7f1bd81048a0 2024-03-23T23:14:20.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.353+0000 7f1bd4c00640 1 -- 172.21.15.12:0/3099865187 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1bc40046c0 con 0x7f1bd81048a0 2024-03-23T23:14:20.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.354+0000 7f1bd4c00640 1 --2- 172.21.15.12:0/3099865187 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1bbc0a2800 0x7f1bbc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:20.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.354+0000 7f1bd4c00640 1 -- 172.21.15.12:0/3099865187 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1bc40cee10 con 0x7f1bd81048a0 2024-03-23T23:14:20.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.354+0000 7f1bd7e00640 1 --2- 172.21.15.12:0/3099865187 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1bbc0a2800 0x7f1bbc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:20.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.355+0000 7f1bd7e00640 1 --2- 172.21.15.12:0/3099865187 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1bbc0a2800 0x7f1bbc0a4cc0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f1bc802c250 tx=0x7f1bc802cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:14:20.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.358+0000 7f1bd4c00640 1 -- 172.21.15.12:0/3099865187 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1bc40d2030 con 0x7f1bd81048a0 2024-03-23T23:14:20.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.563+0000 7f1bdd800640 1 -- 172.21.15.12:0/3099865187 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bd80622a0 con 0x7f1bd81048a0 2024-03-23T23:14:20.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.565+0000 7f1bd4c00640 1 -- 172.21.15.12:0/3099865187 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22695 (secure 0 0 0) 0x7f1bc409be60 con 0x7f1bd81048a0 2024-03-23T23:14:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:14:20.565 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":88,"fsid":"90f679ae-e967-11ee-95cd-87774f69a715","created":"2024-03-23T22:50:14.906952+0000","modified":"2024-03-23T23:00:24.349052+0000","last_up_change":"2024-03-23T22:58:28.907824+0000","last_in_change":"2024-03-23T22:58:08.342487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T22:54:11.831010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T22:59:25.350974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T22:59:26.390323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b70875d7-a23d-4251-9e8b-1a16d4edc2a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6803","nonce":1190079611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6805","nonce":1190079611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6809","nonce":1190079611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6807","nonce":1190079611}]},"public_addr":"172.21.15.12:6803/1190079611","cluster_addr":"172.21.15.12:6805/1190079611","heartbeat_back_addr":"172.21.15.12:6809/1190079611","heartbeat_front_addr":"172.21.15.12:6807/1190079611","state":["exists","up"]},{"osd":1,"uuid":"416e87af-db5d-41ce-84d9-731e3e07e3c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6801","nonce":1983639525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6803","nonce":1983639525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6807","nonce":1983639525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6805","nonce":1983639525}]},"public_addr":"172.21.15.173:6801/1983639525","cluster_addr":"172.21.15.173:6803/1983639525","heartbeat_back_addr":"172.21.15.173:6807/1983639525","heartbeat_front_addr":"172.21.15.173:6805/1983639525","state":["exists","up"]},{"osd":2,"uuid":"7a559c70-4c54-431b-8baf-4136b6e76493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6801","nonce":2253997387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6803","nonce":2253997387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6807","nonce":2253997387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6805","nonce":2253997387}]},"public_addr":"172.21.15.203:6801/2253997387","cluster_addr":"172.21.15.203:6803/2253997387","heartbeat_back_addr":"172.21.15.203:6807/2253997387","heartbeat_front_addr":"172.21.15.203:6805/2253997387","state":["exists","up"]},{"osd":3,"uuid":"c41cd287-a674-486c-8025-328cd36e5ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6811","nonce":1761446905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6813","nonce":1761446905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6817","nonce":1761446905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6815","nonce":1761446905}]},"public_addr":"172.21.15.12:6811/1761446905","cluster_addr":"172.21.15.12:6813/1761446905","heartbeat_back_addr":"172.21.15.12:6817/1761446905","heartbeat_front_addr":"172.21.15.12:6815/1761446905","state":["exists","up"]},{"osd":4,"uuid":"3c48f886-0b42-4a09-b31e-abccc200bb1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6809","nonce":242958272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6811","nonce":242958272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6815","nonce":242958272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6813","nonce":242958272}]},"public_addr":"172.21.15.173:6809/242958272","cluster_addr":"172.21.15.173:6811/242958272","heartbeat_back_addr":"172.21.15.173:6815/242958272","heartbeat_front_addr":"172.21.15.173:6813/242958272","state":["exists","up"]},{"osd":5,"uuid":"4a34f06f-8805-40c0-b1ab-23407b0276d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6809","nonce":4270966092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6811","nonce":4270966092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6815","nonce":4270966092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6813","nonce":4270966092}]},"public_addr":"172.21.15.203:6809/4270966092","cluster_addr":"172.21.15.203:6811/4270966092","heartbeat_back_addr":"172.21.15.203:6815/4270966092","heartbeat_front_addr":"172.21.15.203:6813/4270966092","state":["exists","up"]},{"osd":6,"uuid":"26ddc663-5623-48c9-80d1-f8b968765d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6819","nonce":3606839295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6821","nonce":3606839295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6825","nonce":3606839295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6823","nonce":3606839295}]},"public_addr":"172.21.15.12:6819/3606839295","cluster_addr":"172.21.15.12:6821/3606839295","heartbeat_back_addr":"172.21.15.12:6825/3606839295","heartbeat_front_addr":"172.21.15.12:6823/3606839295","state":["exists","up"]},{"osd":7,"uuid":"17dc9417-71cd-4576-aa08-933212ec55e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6817","nonce":1241504346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6819","nonce":1241504346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6823","nonce":1241504346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6821","nonce":1241504346}]},"public_addr":"172.21.15.173:6817/1241504346","cluster_addr":"172.21.15.173:6819/1241504346","heartbeat_back_addr":"172.21.15.173:6823/1241504346","heartbeat_front_addr":"172.21.15.173:6821/1241504346","state":["exists","up"]},{"osd":8,"uuid":"90a2b63a-471d-43b8-ac18-bb1f16778d18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6817","nonce":1516996192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6819","nonce":1516996192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6823","nonce":1516996192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6821","nonce":1516996192}]},"public_addr":"172.21.15.203:6817/1516996192","cluster_addr":"172.21.15.203:6819/1516996192","heartbeat_back_addr":"172.21.15.203:6823/1516996192","heartbeat_front_addr":"172.21.15.203:6821/1516996192","state":["exists","up"]},{"osd":9,"uuid":"6e1ca504-40b4-49d0-b245-7946181abfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6827","nonce":1175260774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6829","nonce":1175260774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6833","nonce":1175260774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6831","nonce":1175260774}]},"public_addr":"172.21.15.12:6827/1175260774","cluster_addr":"172.21.15.12:6829/1175260774","heartbeat_back_addr":"172.21.15.12:6833/1175260774","heartbeat_front_addr":"172.21.15.12:6831/1175260774","state":["exists","up"]},{"osd":10,"uuid":"09a595ed-5614-4386-8e61-0170af66e01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6825","nonce":3303506808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6827","nonce":3303506808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6831","nonce":3303506808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6829","nonce":3303506808}]},"public_addr":"172.21.15.173:6825/3303506808","cluster_addr":"172.21.15.173:6827/3303506808","heartbeat_back_addr":"172.21.15.173:6831/3303506808","heartbeat_front_addr":"172.21.15.173:6829/3303506808","state":["exists","up"]},{"osd":11,"uuid":"9ef1d595-a977-4ba7-9c6b-d2b3295f20b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6825","nonce":2323604506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6827","nonce":2323604506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6831","nonce":2323604506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6829","nonce":2323604506}]},"public_addr":"172.21.15.203:6825/2323604506","cluster_addr":"172.21.15.203:6827/2323604506","heartbeat_back_addr":"172.21.15.203:6831/2323604506","heartbeat_front_addr":"172.21.15.203:6829/2323604506","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:18.469038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:42.752628+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:07.069493+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:41.962478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:08.363104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:33.341336+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:07.286541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:34.243302+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:59.749699+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:57:33.822913+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:00.737269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:26.599854+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/4182783120":"2024-03-24T22:51:23.113201+0000","172.21.15.12:6800/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1900938637":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/3414861652":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6801/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/2580541477":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6800/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/501058839":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1192554716":"2024-03-24T22:50:48.401223+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T23:14:20.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.568+0000 7f1bdd800640 1 -- 172.21.15.12:0/3099865187 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1bbc0a2800 msgr2=0x7f1bbc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:20.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.568+0000 7f1bdd800640 1 --2- 172.21.15.12:0/3099865187 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1bbc0a2800 0x7f1bbc0a4cc0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f1bc802c250 tx=0x7f1bc802cf90 comp rx=0 tx=0).stop 2024-03-23T23:14:20.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.568+0000 7f1bdd800640 1 -- 172.21.15.12:0/3099865187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1bd81048a0 msgr2=0x7f1bd8199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:20.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.568+0000 7f1bdd800640 1 --2- 172.21.15.12:0/3099865187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1bd81048a0 0x7f1bd8199f60 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f1bc400e9c0 tx=0x7f1bc400ee80 comp rx=0 tx=0).stop 2024-03-23T23:14:20.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.568+0000 7f1bdd800640 1 -- 172.21.15.12:0/3099865187 shutdown_connections 2024-03-23T23:14:20.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.568+0000 7f1bdd800640 1 --2- 172.21.15.12:0/3099865187 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f1bbc0a2800 0x7f1bbc0a4cc0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:20.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.568+0000 7f1bdd800640 1 --2- 172.21.15.12:0/3099865187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bd81071d0 0x7f1bd81a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:20.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.568+0000 7f1bdd800640 1 --2- 172.21.15.12:0/3099865187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1bd81048a0 0x7f1bd8199f60 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:20.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.568+0000 7f1bdd800640 1 --2- 172.21.15.12:0/3099865187 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bd8101f70 0x7f1bd8199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:20.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.568+0000 7f1bdd800640 1 -- 172.21.15.12:0/3099865187 >> 172.21.15.12:0/3099865187 conn(0x7f1bd80fb980 msgr2=0x7f1bd8101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:14:20.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.568+0000 7f1bdd800640 1 -- 172.21.15.12:0/3099865187 shutdown_connections 2024-03-23T23:14:20.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.568+0000 7f1bdd800640 1 -- 172.21.15.12:0/3099865187 wait complete. 2024-03-23T23:14:20.587 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T22:54:11.831010+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T22:59:25.350974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T22:59:26.390323+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T23:14:20.587 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T23:14:20.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.804+0000 7ff04ce00640 1 -- 172.21.15.12:0/621092050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff048102030 msgr2=0x7ff0481088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:20.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.804+0000 7ff04ce00640 1 --2- 172.21.15.12:0/621092050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff048102030 0x7ff0481088c0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7ff040009ca0 tx=0x7ff04002c100 comp rx=0 tx=0).stop 2024-03-23T23:14:20.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.804+0000 7ff04ce00640 1 -- 172.21.15.12:0/621092050 shutdown_connections 2024-03-23T23:14:20.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.804+0000 7ff04ce00640 1 --2- 172.21.15.12:0/621092050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff048102030 0x7ff0481088c0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:20.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.804+0000 7ff04ce00640 1 --2- 172.21.15.12:0/621092050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff048101670 0x7ff048101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:20.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.804+0000 7ff04ce00640 1 --2- 172.21.15.12:0/621092050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff048100480 0x7ff0481008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:20.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.804+0000 7ff04ce00640 1 -- 172.21.15.12:0/621092050 >> 172.21.15.12:0/621092050 conn(0x7ff0480fba10 msgr2=0x7ff0480fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:14:20.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.804+0000 7ff04ce00640 1 -- 172.21.15.12:0/621092050 shutdown_connections 2024-03-23T23:14:20.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.804+0000 7ff04ce00640 1 -- 172.21.15.12:0/621092050 wait complete. 2024-03-23T23:14:20.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.805+0000 7ff04ce00640 1 Processor -- start 2024-03-23T23:14:20.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.805+0000 7ff04ce00640 1 -- start start 2024-03-23T23:14:20.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.805+0000 7ff04ce00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff048100480 0x7ff048104410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:20.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.805+0000 7ff04ce00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff048101670 0x7ff048102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:20.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.805+0000 7ff04ce00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff048102030 0x7ff048102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:20.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.805+0000 7ff04ce00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04810c110 con 0x7ff048100480 2024-03-23T23:14:20.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.805+0000 7ff04ce00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04810bf90 con 0x7ff048101670 2024-03-23T23:14:20.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.805+0000 7ff04ce00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04810c290 con 0x7ff048102030 2024-03-23T23:14:20.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.805+0000 7ff047400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff048100480 0x7ff048104410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:20.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.805+0000 7ff047e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff048102030 0x7ff048102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:20.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.806+0000 7ff047400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff048100480 0x7ff048104410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56434/0 (socket says 172.21.15.12:56434) 2024-03-23T23:14:20.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.806+0000 7ff047400640 1 -- 172.21.15.12:0/3811777201 learned_addr learned my addr 172.21.15.12:0/3811777201 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:14:20.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.806+0000 7ff046a00640 1 --2- 172.21.15.12:0/3811777201 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff048101670 0x7ff048102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:20.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.806+0000 7ff047e00640 1 -- 172.21.15.12:0/3811777201 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff048101670 msgr2=0x7ff048102a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:20.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.806+0000 7ff047e00640 1 --2- 172.21.15.12:0/3811777201 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff048101670 0x7ff048102a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:20.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.806+0000 7ff047e00640 1 -- 172.21.15.12:0/3811777201 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff048100480 msgr2=0x7ff048104410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:20.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.806+0000 7ff047e00640 1 --2- 172.21.15.12:0/3811777201 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff048100480 0x7ff048104410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:20.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.806+0000 7ff047e00640 1 -- 172.21.15.12:0/3811777201 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff048103820 con 0x7ff048102030 2024-03-23T23:14:20.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.806+0000 7ff047e00640 1 --2- 172.21.15.12:0/3811777201 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff048102030 0x7ff048102f90 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7ff040009bf0 tx=0x7ff040035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:14:20.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.806+0000 7ff03fe00640 1 -- 172.21.15.12:0/3811777201 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff040039040 con 0x7ff048102030 2024-03-23T23:14:20.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.807+0000 7ff03fe00640 1 -- 172.21.15.12:0/3811777201 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff040044070 con 0x7ff048102030 2024-03-23T23:14:20.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.807+0000 7ff04ce00640 1 -- 172.21.15.12:0/3811777201 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0481a19d0 con 0x7ff048102030 2024-03-23T23:14:20.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.807+0000 7ff03fe00640 1 -- 172.21.15.12:0/3811777201 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff040004480 con 0x7ff048102030 2024-03-23T23:14:20.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.807+0000 7ff04ce00640 1 -- 172.21.15.12:0/3811777201 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0481a1ef0 con 0x7ff048102030 2024-03-23T23:14:20.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.808+0000 7ff04ce00640 1 -- 172.21.15.12:0/3811777201 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0481a20c0 con 0x7ff048102030 2024-03-23T23:14:20.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.810+0000 7ff03fe00640 1 -- 172.21.15.12:0/3811777201 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff040004620 con 0x7ff048102030 2024-03-23T23:14:20.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.811+0000 7ff03fe00640 1 --2- 172.21.15.12:0/3811777201 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff02c0a2800 0x7ff02c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:20.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.811+0000 7ff03fe00640 1 -- 172.21.15.12:0/3811777201 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff0400f1080 con 0x7ff048102030 2024-03-23T23:14:20.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.812+0000 7ff047400640 1 --2- 172.21.15.12:0/3811777201 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff02c0a2800 0x7ff02c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:20.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.812+0000 7ff047400640 1 --2- 172.21.15.12:0/3811777201 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff02c0a2800 0x7ff02c0a4cc0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff0380058c0 tx=0x7ff038005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:14:20.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:20.815+0000 7ff03fe00640 1 -- 172.21.15.12:0/3811777201 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff0400be130 con 0x7ff048102030 2024-03-23T23:14:21.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.015+0000 7ff04ce00640 1 -- 172.21.15.12:0/3811777201 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff0481a2450 con 0x7ff048102030 2024-03-23T23:14:21.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.016+0000 7ff03fe00640 1 -- 172.21.15.12:0/3811777201 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff0400098b0 con 0x7ff048102030 2024-03-23T23:14:21.015 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-23T23:14:21.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.018+0000 7ff04ce00640 1 -- 172.21.15.12:0/3811777201 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff02c0a2800 msgr2=0x7ff02c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:21.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.018+0000 7ff04ce00640 1 --2- 172.21.15.12:0/3811777201 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff02c0a2800 0x7ff02c0a4cc0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff0380058c0 tx=0x7ff038005f90 comp rx=0 tx=0).stop 2024-03-23T23:14:21.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.018+0000 7ff04ce00640 1 -- 172.21.15.12:0/3811777201 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff048102030 msgr2=0x7ff048102f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:21.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.018+0000 7ff04ce00640 1 --2- 172.21.15.12:0/3811777201 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff048102030 0x7ff048102f90 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7ff040009bf0 tx=0x7ff040035d10 comp rx=0 tx=0).stop 2024-03-23T23:14:21.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.018+0000 7ff04ce00640 1 -- 172.21.15.12:0/3811777201 shutdown_connections 2024-03-23T23:14:21.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.018+0000 7ff04ce00640 1 --2- 172.21.15.12:0/3811777201 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7ff02c0a2800 0x7ff02c0a4cc0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:21.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.018+0000 7ff04ce00640 1 --2- 172.21.15.12:0/3811777201 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff048102030 0x7ff048102f90 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:21.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.018+0000 7ff04ce00640 1 --2- 172.21.15.12:0/3811777201 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff048101670 0x7ff048102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:21.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.018+0000 7ff04ce00640 1 --2- 172.21.15.12:0/3811777201 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff048100480 0x7ff048104410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:21.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.018+0000 7ff04ce00640 1 -- 172.21.15.12:0/3811777201 >> 172.21.15.12:0/3811777201 conn(0x7ff0480fba10 msgr2=0x7ff0480fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:14:21.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.019+0000 7ff04ce00640 1 -- 172.21.15.12:0/3811777201 shutdown_connections 2024-03-23T23:14:21.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.019+0000 7ff04ce00640 1 -- 172.21.15.12:0/3811777201 wait complete. 2024-03-23T23:14:21.036 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T23:14:21.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:20 smithi203 ceph-mon[36693]: pgmap v766: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T23:14:21.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:20 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3099865187' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:14:21.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:20 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:14:21.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.231+0000 7f9187c00640 1 -- 172.21.15.12:0/2600740345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91801016b0 msgr2=0x7f9180101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.231+0000 7f9187c00640 1 --2- 172.21.15.12:0/2600740345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91801016b0 0x7f9180101ad0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f9174009ca0 tx=0x7f917402c0f0 comp rx=0 tx=0).stop 2024-03-23T23:14:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.231+0000 7f9187c00640 1 -- 172.21.15.12:0/2600740345 shutdown_connections 2024-03-23T23:14:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.231+0000 7f9187c00640 1 --2- 172.21.15.12:0/2600740345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9180103260 0x7f9180109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.231+0000 7f9187c00640 1 --2- 172.21.15.12:0/2600740345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91801028a0 0x7f9180102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.231+0000 7f9187c00640 1 --2- 172.21.15.12:0/2600740345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91801016b0 0x7f9180101ad0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.231+0000 7f9187c00640 1 -- 172.21.15.12:0/2600740345 >> 172.21.15.12:0/2600740345 conn(0x7f91800fcc20 msgr2=0x7f91800ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:14:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.231+0000 7f9187c00640 1 -- 172.21.15.12:0/2600740345 shutdown_connections 2024-03-23T23:14:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.231+0000 7f9187c00640 1 -- 172.21.15.12:0/2600740345 wait complete. 2024-03-23T23:14:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.232+0000 7f9187c00640 1 Processor -- start 2024-03-23T23:14:21.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.232+0000 7f9187c00640 1 -- start start 2024-03-23T23:14:21.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.233+0000 7f9187c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91801016b0 0x7f9180075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:21.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.233+0000 7f9187c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91801028a0 0x7f9180073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:21.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.233+0000 7f9187c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9180103260 0x7f9180073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:21.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.233+0000 7f9187c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918010d300 con 0x7f91801028a0 2024-03-23T23:14:21.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.233+0000 7f9187c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918010d180 con 0x7f91801016b0 2024-03-23T23:14:21.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.233+0000 7f9187c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918010d480 con 0x7f9180103260 2024-03-23T23:14:21.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.233+0000 7f9185c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9180103260 0x7f9180073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:21.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.233+0000 7f9185200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91801016b0 0x7f9180075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:21.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.233+0000 7f917fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91801028a0 0x7f9180073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:21.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.233+0000 7f9185200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91801016b0 0x7f9180075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:46992/0 (socket says 172.21.15.12:46992) 2024-03-23T23:14:21.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.233+0000 7f9185200640 1 -- 172.21.15.12:0/2233618400 learned_addr learned my addr 172.21.15.12:0/2233618400 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:14:21.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.233+0000 7f9185c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9180103260 0x7f9180073fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:43390/0 (socket says 172.21.15.12:43390) 2024-03-23T23:14:21.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.234+0000 7f9185200640 1 -- 172.21.15.12:0/2233618400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9180103260 msgr2=0x7f9180073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.234+0000 7f9185200640 1 --2- 172.21.15.12:0/2233618400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9180103260 0x7f9180073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.234+0000 7f9185200640 1 -- 172.21.15.12:0/2233618400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91801028a0 msgr2=0x7f9180073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.234+0000 7f9185200640 1 --2- 172.21.15.12:0/2233618400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91801028a0 0x7f9180073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.234+0000 7f9185200640 1 -- 172.21.15.12:0/2233618400 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9180074860 con 0x7f91801016b0 2024-03-23T23:14:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.234+0000 7f917fe00640 1 --2- 172.21.15.12:0/2233618400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91801028a0 0x7f9180073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:14:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.234+0000 7f9185200640 1 --2- 172.21.15.12:0/2233618400 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91801016b0 0x7f9180075450 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f9174000f80 tx=0x7f9174032ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:14:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.234+0000 7f917d600640 1 -- 172.21.15.12:0/2233618400 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9174041020 con 0x7f91801016b0 2024-03-23T23:14:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.234+0000 7f9187c00640 1 -- 172.21.15.12:0/2233618400 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91801a1ac0 con 0x7f91801016b0 2024-03-23T23:14:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.234+0000 7f917d600640 1 -- 172.21.15.12:0/2233618400 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9174034cd0 con 0x7f91801016b0 2024-03-23T23:14:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.234+0000 7f917d600640 1 -- 172.21.15.12:0/2233618400 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9174039040 con 0x7f91801016b0 2024-03-23T23:14:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.234+0000 7f9187c00640 1 -- 172.21.15.12:0/2233618400 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91801a2030 con 0x7f91801016b0 2024-03-23T23:14:21.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.236+0000 7f9187c00640 1 -- 172.21.15.12:0/2233618400 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9180108a40 con 0x7f91801016b0 2024-03-23T23:14:21.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.239+0000 7f917d600640 1 -- 172.21.15.12:0/2233618400 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9174044020 con 0x7f91801016b0 2024-03-23T23:14:21.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.241+0000 7f917d600640 1 --2- 172.21.15.12:0/2233618400 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f91680a2800 0x7f91680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:21.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.241+0000 7f917d600640 1 -- 172.21.15.12:0/2233618400 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f91740f0810 con 0x7f91801016b0 2024-03-23T23:14:21.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.241+0000 7f917fe00640 1 --2- 172.21.15.12:0/2233618400 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f91680a2800 0x7f91680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:21.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.242+0000 7f917fe00640 1 --2- 172.21.15.12:0/2233618400 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f91680a2800 0x7f91680a4cc0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f91800751c0 tx=0x7f916c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:14:21.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.248+0000 7f917d600640 1 -- 172.21.15.12:0/2233618400 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f917404b020 con 0x7f91801016b0 2024-03-23T23:14:21.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:20 smithi012 ceph-mon[31695]: pgmap v766: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T23:14:21.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:20 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3099865187' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:14:21.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:20 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:14:21.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:20 smithi173 ceph-mon[36116]: pgmap v766: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T23:14:21.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:20 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3099865187' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:14:21.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:20 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T23:14:21.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.478+0000 7f9187c00640 1 -- 172.21.15.12:0/2233618400 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f91800622a0 con 0x7f91801016b0 2024-03-23T23:14:21.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.479+0000 7f917d600640 1 -- 172.21.15.12:0/2233618400 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f91740348f0 con 0x7f91801016b0 2024-03-23T23:14:21.478 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-23T23:14:21.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.481+0000 7f9187c00640 1 -- 172.21.15.12:0/2233618400 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f91680a2800 msgr2=0x7f91680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:21.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.481+0000 7f9187c00640 1 --2- 172.21.15.12:0/2233618400 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f91680a2800 0x7f91680a4cc0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f91800751c0 tx=0x7f916c005f90 comp rx=0 tx=0).stop 2024-03-23T23:14:21.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.481+0000 7f9187c00640 1 -- 172.21.15.12:0/2233618400 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91801016b0 msgr2=0x7f9180075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:21.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.481+0000 7f9187c00640 1 --2- 172.21.15.12:0/2233618400 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91801016b0 0x7f9180075450 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f9174000f80 tx=0x7f9174032ec0 comp rx=0 tx=0).stop 2024-03-23T23:14:21.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.481+0000 7f9187c00640 1 -- 172.21.15.12:0/2233618400 shutdown_connections 2024-03-23T23:14:21.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.481+0000 7f9187c00640 1 --2- 172.21.15.12:0/2233618400 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f91680a2800 0x7f91680a4cc0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:21.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.481+0000 7f9187c00640 1 --2- 172.21.15.12:0/2233618400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9180103260 0x7f9180073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:21.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.481+0000 7f9187c00640 1 --2- 172.21.15.12:0/2233618400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91801028a0 0x7f9180073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:21.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.481+0000 7f9187c00640 1 --2- 172.21.15.12:0/2233618400 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91801016b0 0x7f9180075450 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:21.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.481+0000 7f9187c00640 1 -- 172.21.15.12:0/2233618400 >> 172.21.15.12:0/2233618400 conn(0x7f91800fcc20 msgr2=0x7f9180105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:14:21.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.481+0000 7f9187c00640 1 -- 172.21.15.12:0/2233618400 shutdown_connections 2024-03-23T23:14:21.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.481+0000 7f9187c00640 1 -- 172.21.15.12:0/2233618400 wait complete. 2024-03-23T23:14:21.500 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T23:14:21.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.698+0000 7f7f5e600640 1 -- 172.21.15.12:0/3924456082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f58102000 msgr2=0x7f7f581043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:21.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.698+0000 7f7f5e600640 1 --2- 172.21.15.12:0/3924456082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f58102000 0x7f7f581043f0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009ca0 tx=0x7f7f4802c100 comp rx=0 tx=0).stop 2024-03-23T23:14:21.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.698+0000 7f7f5e600640 1 -- 172.21.15.12:0/3924456082 shutdown_connections 2024-03-23T23:14:21.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.698+0000 7f7f5e600640 1 --2- 172.21.15.12:0/3924456082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f58107260 0x7f7f581096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:21.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.698+0000 7f7f5e600640 1 --2- 172.21.15.12:0/3924456082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f58104930 0x7f7f58106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:21.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.698+0000 7f7f5e600640 1 --2- 172.21.15.12:0/3924456082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f58102000 0x7f7f581043f0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:21.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.698+0000 7f7f5e600640 1 -- 172.21.15.12:0/3924456082 >> 172.21.15.12:0/3924456082 conn(0x7f7f580fba10 msgr2=0x7f7f580fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:14:21.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.698+0000 7f7f5e600640 1 -- 172.21.15.12:0/3924456082 shutdown_connections 2024-03-23T23:14:21.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.698+0000 7f7f5e600640 1 -- 172.21.15.12:0/3924456082 wait complete. 2024-03-23T23:14:21.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.700+0000 7f7f5e600640 1 Processor -- start 2024-03-23T23:14:21.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.700+0000 7f7f5e600640 1 -- start start 2024-03-23T23:14:21.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.700+0000 7f7f5e600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f58102000 0x7f7f58199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:21.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.700+0000 7f7f5e600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f58104930 0x7f7f5819a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:21.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.700+0000 7f7f5e600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f58107260 0x7f7f581a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:21.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.700+0000 7f7f5e600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5810d210 con 0x7f7f58104930 2024-03-23T23:14:21.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.700+0000 7f7f5e600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5810d090 con 0x7f7f58102000 2024-03-23T23:14:21.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.700+0000 7f7f5e600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5810d390 con 0x7f7f58107260 2024-03-23T23:14:21.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.700+0000 7f7f4fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f58104930 0x7f7f5819a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:21.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.700+0000 7f7f4fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f58104930 0x7f7f5819a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56478/0 (socket says 172.21.15.12:56478) 2024-03-23T23:14:21.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.701+0000 7f7f4fe00640 1 -- 172.21.15.12:0/552862050 learned_addr learned my addr 172.21.15.12:0/552862050 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:14:21.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.701+0000 7f7f5dc00640 1 --2- 172.21.15.12:0/552862050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f58107260 0x7f7f581a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:21.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.701+0000 7f7f5d200640 1 --2- 172.21.15.12:0/552862050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f58102000 0x7f7f58199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:21.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.701+0000 7f7f4fe00640 1 -- 172.21.15.12:0/552862050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f58107260 msgr2=0x7f7f581a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:21.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.701+0000 7f7f4fe00640 1 --2- 172.21.15.12:0/552862050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f58107260 0x7f7f581a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:21.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.701+0000 7f7f4fe00640 1 -- 172.21.15.12:0/552862050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f58102000 msgr2=0x7f7f58199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:21.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.701+0000 7f7f4fe00640 1 --2- 172.21.15.12:0/552862050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f58102000 0x7f7f58199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:21.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.701+0000 7f7f4fe00640 1 -- 172.21.15.12:0/552862050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f581a1670 con 0x7f7f58104930 2024-03-23T23:14:21.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.701+0000 7f7f5dc00640 1 --2- 172.21.15.12:0/552862050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f58107260 0x7f7f581a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:14:21.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.701+0000 7f7f4fe00640 1 --2- 172.21.15.12:0/552862050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f58104930 0x7f7f5819a000 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7f4000e9c0 tx=0x7f7f4000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:14:21.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.701+0000 7f7f56000640 1 -- 172.21.15.12:0/552862050 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4000cd80 con 0x7f7f58104930 2024-03-23T23:14:21.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.702+0000 7f7f56000640 1 -- 172.21.15.12:0/552862050 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7f40005500 con 0x7f7f58104930 2024-03-23T23:14:21.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.702+0000 7f7f5e600640 1 -- 172.21.15.12:0/552862050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f581a1960 con 0x7f7f58104930 2024-03-23T23:14:21.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.702+0000 7f7f56000640 1 -- 172.21.15.12:0/552862050 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f40010690 con 0x7f7f58104930 2024-03-23T23:14:21.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.702+0000 7f7f5e600640 1 -- 172.21.15.12:0/552862050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f581a1eb0 con 0x7f7f58104930 2024-03-23T23:14:21.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.703+0000 7f7f5e600640 1 -- 172.21.15.12:0/552862050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f58108db0 con 0x7f7f58104930 2024-03-23T23:14:21.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.705+0000 7f7f56000640 1 -- 172.21.15.12:0/552862050 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7f400046c0 con 0x7f7f58104930 2024-03-23T23:14:21.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.706+0000 7f7f56000640 1 --2- 172.21.15.12:0/552862050 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7f3c0a2800 0x7f7f3c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:21.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.706+0000 7f7f56000640 1 -- 172.21.15.12:0/552862050 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7f400cee90 con 0x7f7f58104930 2024-03-23T23:14:21.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.707+0000 7f7f5d200640 1 --2- 172.21.15.12:0/552862050 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7f3c0a2800 0x7f7f3c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:21.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.707+0000 7f7f5d200640 1 --2- 172.21.15.12:0/552862050 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7f3c0a2800 0x7f7f3c0a4cc0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f7f4802c710 tx=0x7f7f48009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:14:21.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.711+0000 7f7f56000640 1 -- 172.21.15.12:0/552862050 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7f4009bee0 con 0x7f7f58104930 2024-03-23T23:14:21.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.909+0000 7f7f5e600640 1 -- 172.21.15.12:0/552862050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7f580622a0 con 0x7f7f58104930 2024-03-23T23:14:21.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.911+0000 7f7f56000640 1 -- 172.21.15.12:0/552862050 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7f4009b7e0 con 0x7f7f58104930 2024-03-23T23:14:21.911 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-23T23:14:21.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.913+0000 7f7f5e600640 1 -- 172.21.15.12:0/552862050 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7f3c0a2800 msgr2=0x7f7f3c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:21.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.913+0000 7f7f5e600640 1 --2- 172.21.15.12:0/552862050 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7f3c0a2800 0x7f7f3c0a4cc0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f7f4802c710 tx=0x7f7f48009920 comp rx=0 tx=0).stop 2024-03-23T23:14:21.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.913+0000 7f7f5e600640 1 -- 172.21.15.12:0/552862050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f58104930 msgr2=0x7f7f5819a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:21.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.913+0000 7f7f5e600640 1 --2- 172.21.15.12:0/552862050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f58104930 0x7f7f5819a000 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7f4000e9c0 tx=0x7f7f4000ee80 comp rx=0 tx=0).stop 2024-03-23T23:14:21.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.914+0000 7f7f5e600640 1 -- 172.21.15.12:0/552862050 shutdown_connections 2024-03-23T23:14:21.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.914+0000 7f7f5e600640 1 --2- 172.21.15.12:0/552862050 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7f3c0a2800 0x7f7f3c0a4cc0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:21.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.914+0000 7f7f5e600640 1 --2- 172.21.15.12:0/552862050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f58107260 0x7f7f581a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:21.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.914+0000 7f7f5e600640 1 --2- 172.21.15.12:0/552862050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f58104930 0x7f7f5819a000 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:21.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.914+0000 7f7f5e600640 1 --2- 172.21.15.12:0/552862050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f58102000 0x7f7f58199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:21.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.914+0000 7f7f5e600640 1 -- 172.21.15.12:0/552862050 >> 172.21.15.12:0/552862050 conn(0x7f7f580fba10 msgr2=0x7f7f58101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:14:21.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.914+0000 7f7f5e600640 1 -- 172.21.15.12:0/552862050 shutdown_connections 2024-03-23T23:14:21.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:21.914+0000 7f7f5e600640 1 -- 172.21.15.12:0/552862050 wait complete. 2024-03-23T23:14:21.931 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T23:14:22.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.122+0000 7fb3da200640 1 -- 172.21.15.12:0/392305946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3d4102790 msgr2=0x7fb3d4102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:22.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.122+0000 7fb3da200640 1 --2- 172.21.15.12:0/392305946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3d4102790 0x7fb3d4102c10 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0009900 tx=0x7fb3c002bc40 comp rx=0 tx=0).stop 2024-03-23T23:14:22.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.122+0000 7fb3da200640 1 -- 172.21.15.12:0/392305946 shutdown_connections 2024-03-23T23:14:22.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.122+0000 7fb3da200640 1 --2- 172.21.15.12:0/392305946 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d4103150 0x7fb3d41099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:22.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.122+0000 7fb3da200640 1 --2- 172.21.15.12:0/392305946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3d4102790 0x7fb3d4102c10 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:22.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.122+0000 7fb3da200640 1 --2- 172.21.15.12:0/392305946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3d41015a0 0x7fb3d41019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:22.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.122+0000 7fb3da200640 1 -- 172.21.15.12:0/392305946 >> 172.21.15.12:0/392305946 conn(0x7fb3d40fcb10 msgr2=0x7fb3d40fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:14:22.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.123+0000 7fb3da200640 1 -- 172.21.15.12:0/392305946 shutdown_connections 2024-03-23T23:14:22.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.123+0000 7fb3da200640 1 -- 172.21.15.12:0/392305946 wait complete. 2024-03-23T23:14:22.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.124+0000 7fb3da200640 1 Processor -- start 2024-03-23T23:14:22.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.124+0000 7fb3da200640 1 -- start start 2024-03-23T23:14:22.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.124+0000 7fb3da200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d41015a0 0x7fb3d41978a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:22.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.124+0000 7fb3da200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3d4102790 0x7fb3d4197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:22.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.124+0000 7fb3da200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3d4103150 0x7fb3d419ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:22.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.124+0000 7fb3da200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d410d250 con 0x7fb3d41015a0 2024-03-23T23:14:22.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.124+0000 7fb3da200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d410d0d0 con 0x7fb3d4102790 2024-03-23T23:14:22.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.124+0000 7fb3da200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d410d3d0 con 0x7fb3d4103150 2024-03-23T23:14:22.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.125+0000 7fb3d8e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d41015a0 0x7fb3d41978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:22.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.125+0000 7fb3d8e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d41015a0 0x7fb3d41978a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56494/0 (socket says 172.21.15.12:56494) 2024-03-23T23:14:22.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.125+0000 7fb3d8e00640 1 -- 172.21.15.12:0/2561358238 learned_addr learned my addr 172.21.15.12:0/2561358238 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:14:22.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.125+0000 7fb3d3e00640 1 --2- 172.21.15.12:0/2561358238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3d4102790 0x7fb3d4197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:22.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.125+0000 7fb3d9800640 1 --2- 172.21.15.12:0/2561358238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3d4103150 0x7fb3d419ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:22.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.125+0000 7fb3d8e00640 1 -- 172.21.15.12:0/2561358238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3d4103150 msgr2=0x7fb3d419ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:22.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.125+0000 7fb3d8e00640 1 --2- 172.21.15.12:0/2561358238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3d4103150 0x7fb3d419ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:22.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.125+0000 7fb3d8e00640 1 -- 172.21.15.12:0/2561358238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3d4102790 msgr2=0x7fb3d4197de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:22.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.125+0000 7fb3d8e00640 1 --2- 172.21.15.12:0/2561358238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3d4102790 0x7fb3d4197de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:22.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.125+0000 7fb3d8e00640 1 -- 172.21.15.12:0/2561358238 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d419f4c0 con 0x7fb3d41015a0 2024-03-23T23:14:22.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.125+0000 7fb3d8e00640 1 --2- 172.21.15.12:0/2561358238 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d41015a0 0x7fb3d41978a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb3c400ea80 tx=0x7fb3c400ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:14:22.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.126+0000 7fb3d1600640 1 -- 172.21.15.12:0/2561358238 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c400ce40 con 0x7fb3d41015a0 2024-03-23T23:14:22.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.126+0000 7fb3d1600640 1 -- 172.21.15.12:0/2561358238 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3c4011950 con 0x7fb3d41015a0 2024-03-23T23:14:22.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.126+0000 7fb3d1600640 1 -- 172.21.15.12:0/2561358238 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c401ab80 con 0x7fb3d41015a0 2024-03-23T23:14:22.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.126+0000 7fb3da200640 1 -- 172.21.15.12:0/2561358238 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3d419f7b0 con 0x7fb3d41015a0 2024-03-23T23:14:22.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.126+0000 7fb3da200640 1 -- 172.21.15.12:0/2561358238 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3d419fc80 con 0x7fb3d41015a0 2024-03-23T23:14:22.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.127+0000 7fb3da200640 1 -- 172.21.15.12:0/2561358238 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3d41088c0 con 0x7fb3d41015a0 2024-03-23T23:14:22.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.131+0000 7fb3d1600640 1 -- 172.21.15.12:0/2561358238 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb3c4011b00 con 0x7fb3d41015a0 2024-03-23T23:14:22.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.133+0000 7fb3d1600640 1 --2- 172.21.15.12:0/2561358238 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb3b80a2880 0x7fb3b80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:22.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.133+0000 7fb3d1600640 1 -- 172.21.15.12:0/2561358238 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb3c40cf1c0 con 0x7fb3d41015a0 2024-03-23T23:14:22.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.133+0000 7fb3d3e00640 1 --2- 172.21.15.12:0/2561358238 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb3b80a2880 0x7fb3b80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:22.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.133+0000 7fb3d1600640 1 -- 172.21.15.12:0/2561358238 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb3c4016030 con 0x7fb3d41015a0 2024-03-23T23:14:22.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.134+0000 7fb3d3e00640 1 --2- 172.21.15.12:0/2561358238 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb3b80a2880 0x7fb3b80a4d40 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fb3c002c250 tx=0x7fb3c002cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:14:22.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:21 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/3811777201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:14:22.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:21 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2233618400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:14:22.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:21 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:14:22.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:21 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:14:22.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:21 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:14:22.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.351+0000 7fb3da200640 1 -- 172.21.15.12:0/2561358238 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb3d402d070 con 0x7fb3d41015a0 2024-03-23T23:14:22.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.352+0000 7fb3d1600640 1 -- 172.21.15.12:0/2561358238 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v88) v1 ==== 81+0+591 (secure 0 0 0) 0x7fb3c409c290 con 0x7fb3d41015a0 2024-03-23T23:14:22.352 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6801/4127975083 2024-03-24T22:51:23.113201+0000 2024-03-23T23:14:22.353 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/4182783120 2024-03-24T22:51:23.113201+0000 2024-03-23T23:14:22.353 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6800/4127975083 2024-03-24T22:51:23.113201+0000 2024-03-23T23:14:22.353 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1900938637 2024-03-24T22:51:23.113201+0000 2024-03-23T23:14:22.353 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/3414861652 2024-03-24T22:50:48.401223+0000 2024-03-23T23:14:22.353 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6801/3270387158 2024-03-24T22:50:48.401223+0000 2024-03-23T23:14:22.353 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/2580541477 2024-03-24T22:50:48.401223+0000 2024-03-23T23:14:22.353 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6800/3270387158 2024-03-24T22:50:48.401223+0000 2024-03-23T23:14:22.353 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/501058839 2024-03-24T22:51:23.113201+0000 2024-03-23T23:14:22.353 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1192554716 2024-03-24T22:50:48.401223+0000 2024-03-23T23:14:22.353 INFO:teuthology.orchestra.run.smithi012.stderr:listed 10 entries 2024-03-23T23:14:22.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.355+0000 7fb3da200640 1 -- 172.21.15.12:0/2561358238 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb3b80a2880 msgr2=0x7fb3b80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:22.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.355+0000 7fb3da200640 1 --2- 172.21.15.12:0/2561358238 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb3b80a2880 0x7fb3b80a4d40 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fb3c002c250 tx=0x7fb3c002cf90 comp rx=0 tx=0).stop 2024-03-23T23:14:22.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.355+0000 7fb3da200640 1 -- 172.21.15.12:0/2561358238 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d41015a0 msgr2=0x7fb3d41978a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:22.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.355+0000 7fb3da200640 1 --2- 172.21.15.12:0/2561358238 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d41015a0 0x7fb3d41978a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb3c400ea80 tx=0x7fb3c400ef40 comp rx=0 tx=0).stop 2024-03-23T23:14:22.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.355+0000 7fb3da200640 1 -- 172.21.15.12:0/2561358238 shutdown_connections 2024-03-23T23:14:22.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.355+0000 7fb3da200640 1 --2- 172.21.15.12:0/2561358238 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fb3b80a2880 0x7fb3b80a4d40 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:22.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.355+0000 7fb3da200640 1 --2- 172.21.15.12:0/2561358238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3d4103150 0x7fb3d419ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:22.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.355+0000 7fb3da200640 1 --2- 172.21.15.12:0/2561358238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3d4102790 0x7fb3d4197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:22.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.355+0000 7fb3da200640 1 --2- 172.21.15.12:0/2561358238 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d41015a0 0x7fb3d41978a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:22.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.355+0000 7fb3da200640 1 -- 172.21.15.12:0/2561358238 >> 172.21.15.12:0/2561358238 conn(0x7fb3d40fcb10 msgr2=0x7fb3d4105a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:14:22.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.355+0000 7fb3da200640 1 -- 172.21.15.12:0/2561358238 shutdown_connections 2024-03-23T23:14:22.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:22.355+0000 7fb3da200640 1 -- 172.21.15.12:0/2561358238 wait complete. 2024-03-23T23:14:22.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:21 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/3811777201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:14:22.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:21 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2233618400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:14:22.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:21 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:14:22.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:21 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:14:22.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:21 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:14:22.375 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-23T23:14:22.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:14:22.375 DEBUG:teuthology.orchestra.run.smithi012:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-23T23:14:22.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:21 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/3811777201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T23:14:22.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:21 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2233618400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-23T23:14:22.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:21 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T23:14:22.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:21 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T23:14:22.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:21 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' 2024-03-23T23:14:22.511 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-03-23T23:14:22.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:14:22.512 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T23:14:22.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:14:22.567 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-23T23:14:22.604 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T23:14:22.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:14:22.605 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T23:14:22.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T23:14:22.720 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link delete ceph-brx 2024-03-23T23:14:22.832 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-23T23:14:22.842 INFO:tasks.cephadm:Teardown begin 2024-03-23T23:14:22.843 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-23T23:14:22.843 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:14:23.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.045+0000 7f3014600640 1 -- 172.21.15.12:0/1633295490 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f300c104ad0 msgr2=0x7f300c104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:23.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.045+0000 7f3014600640 1 --2- 172.21.15.12:0/1633295490 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f300c104ad0 0x7f300c104f50 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8009900 tx=0x7f2ff802bc30 comp rx=0 tx=0).stop 2024-03-23T23:14:23.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.045+0000 7f3014600640 1 -- 172.21.15.12:0/1633295490 shutdown_connections 2024-03-23T23:14:23.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.045+0000 7f3014600640 1 --2- 172.21.15.12:0/1633295490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f300c105490 0x7f300c10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:23.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.045+0000 7f3014600640 1 --2- 172.21.15.12:0/1633295490 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f300c104ad0 0x7f300c104f50 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:23.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.045+0000 7f3014600640 1 --2- 172.21.15.12:0/1633295490 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f300c1038e0 0x7f300c103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:23.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.045+0000 7f3014600640 1 -- 172.21.15.12:0/1633295490 >> 172.21.15.12:0/1633295490 conn(0x7f300c0fee50 msgr2=0x7f300c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:14:23.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.045+0000 7f3014600640 1 -- 172.21.15.12:0/1633295490 shutdown_connections 2024-03-23T23:14:23.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.045+0000 7f3014600640 1 -- 172.21.15.12:0/1633295490 wait complete. 2024-03-23T23:14:23.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.046+0000 7f3014600640 1 Processor -- start 2024-03-23T23:14:23.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.047+0000 7f3014600640 1 -- start start 2024-03-23T23:14:23.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.047+0000 7f3014600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f300c1038e0 0x7f300c199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:23.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.047+0000 7f3014600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f300c104ad0 0x7f300c19a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:23.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.047+0000 7f3014600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f300c105490 0x7f300c1a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:23.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.047+0000 7f3014600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300c10f5b0 con 0x7f300c104ad0 2024-03-23T23:14:23.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.047+0000 7f3014600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300c10f430 con 0x7f300c1038e0 2024-03-23T23:14:23.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.047+0000 7f3014600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300c10f730 con 0x7f300c105490 2024-03-23T23:14:23.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.047+0000 7f3011200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f300c104ad0 0x7f300c19a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:23.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.047+0000 7f3011c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f300c1038e0 0x7f300c199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:23.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.047+0000 7f3012600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f300c105490 0x7f300c1a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:23.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.048+0000 7f3011200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f300c104ad0 0x7f300c19a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56504/0 (socket says 172.21.15.12:56504) 2024-03-23T23:14:23.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.048+0000 7f3011200640 1 -- 172.21.15.12:0/966225966 learned_addr learned my addr 172.21.15.12:0/966225966 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:14:23.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.048+0000 7f3011c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f300c1038e0 0x7f300c199b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:47052/0 (socket says 172.21.15.12:47052) 2024-03-23T23:14:23.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.048+0000 7f3012600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f300c105490 0x7f300c1a1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:43458/0 (socket says 172.21.15.12:43458) 2024-03-23T23:14:23.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.048+0000 7f3011200640 1 -- 172.21.15.12:0/966225966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f300c105490 msgr2=0x7f300c1a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:23.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.048+0000 7f3011200640 1 --2- 172.21.15.12:0/966225966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f300c105490 0x7f300c1a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:23.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.048+0000 7f3011200640 1 -- 172.21.15.12:0/966225966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f300c1038e0 msgr2=0x7f300c199b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:23.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.048+0000 7f3011200640 1 --2- 172.21.15.12:0/966225966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f300c1038e0 0x7f300c199b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:23.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.048+0000 7f3011200640 1 -- 172.21.15.12:0/966225966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300c1a17b0 con 0x7f300c104ad0 2024-03-23T23:14:23.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.048+0000 7f3012600640 1 --2- 172.21.15.12:0/966225966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f300c105490 0x7f300c1a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:14:23.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.048+0000 7f3011200640 1 --2- 172.21.15.12:0/966225966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f300c104ad0 0x7f300c19a0d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8000c00 tx=0x7f2ff802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:14:23.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.049+0000 7f300a000640 1 -- 172.21.15.12:0/966225966 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8043020 con 0x7f300c104ad0 2024-03-23T23:14:23.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.049+0000 7f300a000640 1 -- 172.21.15.12:0/966225966 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ff80054e0 con 0x7f300c104ad0 2024-03-23T23:14:23.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.049+0000 7f3014600640 1 -- 172.21.15.12:0/966225966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f300c1a1a40 con 0x7f300c104ad0 2024-03-23T23:14:23.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.049+0000 7f300a000640 1 -- 172.21.15.12:0/966225966 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8003c40 con 0x7f300c104ad0 2024-03-23T23:14:23.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.049+0000 7f3014600640 1 -- 172.21.15.12:0/966225966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f300c1a1f90 con 0x7f300c104ad0 2024-03-23T23:14:23.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.050+0000 7f3014600640 1 -- 172.21.15.12:0/966225966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f300c10aba0 con 0x7f300c104ad0 2024-03-23T23:14:23.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.052+0000 7f300a000640 1 -- 172.21.15.12:0/966225966 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2ff8046020 con 0x7f300c104ad0 2024-03-23T23:14:23.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:22 smithi012 ceph-mon[31695]: pgmap v767: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-23T23:14:23.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:22 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/552862050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:14:23.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:22 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2561358238' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-23T23:14:23.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.054+0000 7f300a000640 1 --2- 172.21.15.12:0/966225966 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ff40a2800 0x7f2ff40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:23.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.054+0000 7f300a000640 1 -- 172.21.15.12:0/966225966 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2ff80f0af0 con 0x7f300c104ad0 2024-03-23T23:14:23.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.055+0000 7f3011c00640 1 --2- 172.21.15.12:0/966225966 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ff40a2800 0x7f2ff40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:23.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.055+0000 7f3011c00640 1 --2- 172.21.15.12:0/966225966 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ff40a2800 0x7f2ff40a4cc0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f300c104930 tx=0x7f3000005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:14:23.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.060+0000 7f300a000640 1 -- 172.21.15.12:0/966225966 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2ff804d020 con 0x7f300c104ad0 2024-03-23T23:14:23.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.310+0000 7f3014600640 1 -- 172.21.15.12:0/966225966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f300c02d070 con 0x7f300c104ad0 2024-03-23T23:14:23.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.312+0000 7f300a000640 1 -- 172.21.15.12:0/966225966 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f2ff80bdb40 con 0x7f300c104ad0 2024-03-23T23:14:23.312 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:14:23.312 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:14:23.312 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:14:23.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.314+0000 7f3014600640 1 -- 172.21.15.12:0/966225966 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ff40a2800 msgr2=0x7f2ff40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:23.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.314+0000 7f3014600640 1 --2- 172.21.15.12:0/966225966 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ff40a2800 0x7f2ff40a4cc0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f300c104930 tx=0x7f3000005f90 comp rx=0 tx=0).stop 2024-03-23T23:14:23.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.315+0000 7f3014600640 1 -- 172.21.15.12:0/966225966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f300c104ad0 msgr2=0x7f300c19a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:23.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.315+0000 7f3014600640 1 --2- 172.21.15.12:0/966225966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f300c104ad0 0x7f300c19a0d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8000c00 tx=0x7f2ff802c650 comp rx=0 tx=0).stop 2024-03-23T23:14:23.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.315+0000 7f3014600640 1 -- 172.21.15.12:0/966225966 shutdown_connections 2024-03-23T23:14:23.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.315+0000 7f3014600640 1 --2- 172.21.15.12:0/966225966 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f2ff40a2800 0x7f2ff40a4cc0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:23.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.315+0000 7f3014600640 1 --2- 172.21.15.12:0/966225966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f300c105490 0x7f300c1a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:23.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.315+0000 7f3014600640 1 --2- 172.21.15.12:0/966225966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f300c104ad0 0x7f300c19a0d0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:23.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.315+0000 7f3014600640 1 --2- 172.21.15.12:0/966225966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f300c1038e0 0x7f300c199b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:23.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.315+0000 7f3014600640 1 -- 172.21.15.12:0/966225966 >> 172.21.15.12:0/966225966 conn(0x7f300c0fee50 msgr2=0x7f300c107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:14:23.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.315+0000 7f3014600640 1 -- 172.21.15.12:0/966225966 shutdown_connections 2024-03-23T23:14:23.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.315+0000 7f3014600640 1 -- 172.21.15.12:0/966225966 wait complete. 2024-03-23T23:14:23.334 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T23:14:23.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:22 smithi173 ceph-mon[36116]: pgmap v767: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-23T23:14:23.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:22 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/552862050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:14:23.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:22 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2561358238' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-23T23:14:23.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:22 smithi203 ceph-mon[36693]: pgmap v767: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-23T23:14:23.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:22 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/552862050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-23T23:14:23.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:22 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2561358238' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-23T23:14:23.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.557+0000 7fbe3b600640 1 -- 172.21.15.12:0/663214535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe341028b0 msgr2=0x7fbe34102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:23.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.557+0000 7fbe3b600640 1 --2- 172.21.15.12:0/663214535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe341028b0 0x7fbe34102d30 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fbe20009900 tx=0x7fbe2002bc60 comp rx=0 tx=0).stop 2024-03-23T23:14:23.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.557+0000 7fbe3b600640 1 -- 172.21.15.12:0/663214535 shutdown_connections 2024-03-23T23:14:23.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.557+0000 7fbe3b600640 1 --2- 172.21.15.12:0/663214535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe34103270 0x7fbe34109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:23.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.557+0000 7fbe3b600640 1 --2- 172.21.15.12:0/663214535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe341028b0 0x7fbe34102d30 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:23.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.557+0000 7fbe3b600640 1 --2- 172.21.15.12:0/663214535 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe341016c0 0x7fbe34101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:23.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.557+0000 7fbe3b600640 1 -- 172.21.15.12:0/663214535 >> 172.21.15.12:0/663214535 conn(0x7fbe340fcc50 msgr2=0x7fbe340ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:14:23.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.557+0000 7fbe3b600640 1 -- 172.21.15.12:0/663214535 shutdown_connections 2024-03-23T23:14:23.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.557+0000 7fbe3b600640 1 -- 172.21.15.12:0/663214535 wait complete. 2024-03-23T23:14:23.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.558+0000 7fbe3b600640 1 Processor -- start 2024-03-23T23:14:23.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.559+0000 7fbe3b600640 1 -- start start 2024-03-23T23:14:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.559+0000 7fbe3b600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe341016c0 0x7fbe34075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.559+0000 7fbe3b600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe341028b0 0x7fbe34073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.559+0000 7fbe3b600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe34103270 0x7fbe34073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.559+0000 7fbe3b600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3410d400 con 0x7fbe341028b0 2024-03-23T23:14:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.559+0000 7fbe3b600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3410d280 con 0x7fbe341016c0 2024-03-23T23:14:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.559+0000 7fbe3b600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3410d580 con 0x7fbe34103270 2024-03-23T23:14:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.559+0000 7fbe33e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe341028b0 0x7fbe34073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.559+0000 7fbe38c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe341016c0 0x7fbe34075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.559+0000 7fbe38c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe341016c0 0x7fbe34075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:47070/0 (socket says 172.21.15.12:47070) 2024-03-23T23:14:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.559+0000 7fbe33e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe341028b0 0x7fbe34073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56516/0 (socket says 172.21.15.12:56516) 2024-03-23T23:14:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.559+0000 7fbe38c00640 1 -- 172.21.15.12:0/1390493136 learned_addr learned my addr 172.21.15.12:0/1390493136 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:14:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.559+0000 7fbe39600640 1 --2- 172.21.15.12:0/1390493136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe34103270 0x7fbe34073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.560+0000 7fbe33e00640 1 -- 172.21.15.12:0/1390493136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe34103270 msgr2=0x7fbe34073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:23.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.560+0000 7fbe33e00640 1 --2- 172.21.15.12:0/1390493136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe34103270 0x7fbe34073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:23.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.560+0000 7fbe33e00640 1 -- 172.21.15.12:0/1390493136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe341016c0 msgr2=0x7fbe34075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:23.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.560+0000 7fbe33e00640 1 --2- 172.21.15.12:0/1390493136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe341016c0 0x7fbe34075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:23.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.560+0000 7fbe33e00640 1 -- 172.21.15.12:0/1390493136 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe34074860 con 0x7fbe341028b0 2024-03-23T23:14:23.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.560+0000 7fbe33e00640 1 --2- 172.21.15.12:0/1390493136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe341028b0 0x7fbe34073a90 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fbe20036040 tx=0x7fbe20005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:14:23.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.560+0000 7fbe31600640 1 -- 172.21.15.12:0/1390493136 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe20041070 con 0x7fbe341028b0 2024-03-23T23:14:23.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.560+0000 7fbe31600640 1 -- 172.21.15.12:0/1390493136 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe200044d0 con 0x7fbe341028b0 2024-03-23T23:14:23.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.560+0000 7fbe3b600640 1 -- 172.21.15.12:0/1390493136 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe341a1b30 con 0x7fbe341028b0 2024-03-23T23:14:23.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.560+0000 7fbe31600640 1 -- 172.21.15.12:0/1390493136 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe20034440 con 0x7fbe341028b0 2024-03-23T23:14:23.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.561+0000 7fbe3b600640 1 -- 172.21.15.12:0/1390493136 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe341a2050 con 0x7fbe341028b0 2024-03-23T23:14:23.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.562+0000 7fbe3b600640 1 -- 172.21.15.12:0/1390493136 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe341089e0 con 0x7fbe341028b0 2024-03-23T23:14:23.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.563+0000 7fbe31600640 1 -- 172.21.15.12:0/1390493136 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbe20044070 con 0x7fbe341028b0 2024-03-23T23:14:23.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.565+0000 7fbe31600640 1 --2- 172.21.15.12:0/1390493136 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbe1c0a2800 0x7fbe1c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.565+0000 7fbe31600640 1 -- 172.21.15.12:0/1390493136 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fbe200f0970 con 0x7fbe341028b0 2024-03-23T23:14:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.565+0000 7fbe38c00640 1 --2- 172.21.15.12:0/1390493136 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbe1c0a2800 0x7fbe1c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:23.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.566+0000 7fbe38c00640 1 --2- 172.21.15.12:0/1390493136 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbe1c0a2800 0x7fbe1c0a4cc0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fbe28006fd0 tx=0x7fbe28009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:14:23.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.569+0000 7fbe31600640 1 -- 172.21.15.12:0/1390493136 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbe200bd370 con 0x7fbe341028b0 2024-03-23T23:14:23.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.807+0000 7fbe3b600640 1 -- 172.21.15.12:0/1390493136 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe3402d070 con 0x7fbe341028b0 2024-03-23T23:14:23.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.809+0000 7fbe31600640 1 -- 172.21.15.12:0/1390493136 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7fbe200c1410 con 0x7fbe341028b0 2024-03-23T23:14:23.809 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:14:23.809 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2201661812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2201661812},{"type":"v1","addr":"172.21.15.12:6835","nonce":2201661812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2696205127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2696205127},{"type":"v1","addr":"172.21.15.12:6837","nonce":2696205127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/771072116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":771072116},{"type":"v1","addr":"172.21.15.12:6839","nonce":771072116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/732195840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":732195840},{"type":"v1","addr":"172.21.15.12:6841","nonce":732195840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3254902484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3254902484},{"type":"v1","addr":"172.21.15.203:6833","nonce":3254902484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24473,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/4291212010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":4291212010},{"type":"v1","addr":"172.21.15.173:6833","nonce":4291212010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T22:59:26.906237+0000","modified":"2024-03-23T23:01:49.464235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24554,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/915277278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":915277278},{"type":"v1","addr":"172.21.15.203:6835","nonce":915277278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2648278843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2648278843},{"type":"v1","addr":"172.21.15.173:6835","nonce":2648278843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/438676939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":438676939},{"type":"v1","addr":"172.21.15.203:6837","nonce":438676939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.203:6839/852477167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":852477167},{"type":"v1","addr":"172.21.15.203:6839","nonce":852477167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24554":{"gid":24554,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/2648089141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2648089141},{"type":"v1","addr":"172.21.15.173:6837","nonce":2648089141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/1645433214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":1645433214},{"type":"v1","addr":"172.21.15.173:6839","nonce":1645433214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24554,24550,24599]},"id":1}]} 2024-03-23T23:14:23.809 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-03-23T23:14:23.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.811+0000 7fbe3b600640 1 -- 172.21.15.12:0/1390493136 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbe1c0a2800 msgr2=0x7fbe1c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:23.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.811+0000 7fbe3b600640 1 --2- 172.21.15.12:0/1390493136 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbe1c0a2800 0x7fbe1c0a4cc0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fbe28006fd0 tx=0x7fbe28009190 comp rx=0 tx=0).stop 2024-03-23T23:14:23.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.811+0000 7fbe3b600640 1 -- 172.21.15.12:0/1390493136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe341028b0 msgr2=0x7fbe34073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:23.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.811+0000 7fbe3b600640 1 --2- 172.21.15.12:0/1390493136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe341028b0 0x7fbe34073a90 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fbe20036040 tx=0x7fbe20005d50 comp rx=0 tx=0).stop 2024-03-23T23:14:23.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.811+0000 7fbe3b600640 1 -- 172.21.15.12:0/1390493136 shutdown_connections 2024-03-23T23:14:23.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.811+0000 7fbe3b600640 1 --2- 172.21.15.12:0/1390493136 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7fbe1c0a2800 0x7fbe1c0a4cc0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:23.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.811+0000 7fbe3b600640 1 --2- 172.21.15.12:0/1390493136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe34103270 0x7fbe34073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:23.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.811+0000 7fbe3b600640 1 --2- 172.21.15.12:0/1390493136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe341028b0 0x7fbe34073a90 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:23.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.811+0000 7fbe3b600640 1 --2- 172.21.15.12:0/1390493136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe341016c0 0x7fbe34075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:23.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.811+0000 7fbe3b600640 1 -- 172.21.15.12:0/1390493136 >> 172.21.15.12:0/1390493136 conn(0x7fbe340fcc50 msgr2=0x7fbe34105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:14:23.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.812+0000 7fbe3b600640 1 -- 172.21.15.12:0/1390493136 shutdown_connections 2024-03-23T23:14:23.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:23.812+0000 7fbe3b600640 1 -- 172.21.15.12:0/1390493136 wait complete. 2024-03-23T23:14:23.831 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T23:14:24.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.042+0000 7f7f21800640 1 -- 172.21.15.12:0/2853171114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f1c1036f0 msgr2=0x7f7f1c103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:24.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.042+0000 7f7f21800640 1 --2- 172.21.15.12:0/2853171114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f1c1036f0 0x7f7f1c103b10 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c009900 tx=0x7f7f0c02bc30 comp rx=0 tx=0).stop 2024-03-23T23:14:24.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.042+0000 7f7f21800640 1 -- 172.21.15.12:0/2853171114 shutdown_connections 2024-03-23T23:14:24.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.042+0000 7f7f21800640 1 --2- 172.21.15.12:0/2853171114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1c1052a0 0x7f7f1c10bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:24.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.042+0000 7f7f21800640 1 --2- 172.21.15.12:0/2853171114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f1c1048e0 0x7f7f1c104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:24.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.042+0000 7f7f21800640 1 --2- 172.21.15.12:0/2853171114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f1c1036f0 0x7f7f1c103b10 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:24.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.042+0000 7f7f21800640 1 -- 172.21.15.12:0/2853171114 >> 172.21.15.12:0/2853171114 conn(0x7f7f1c0fec80 msgr2=0x7f7f1c1010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:14:24.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.043+0000 7f7f21800640 1 -- 172.21.15.12:0/2853171114 shutdown_connections 2024-03-23T23:14:24.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.043+0000 7f7f21800640 1 -- 172.21.15.12:0/2853171114 wait complete. 2024-03-23T23:14:24.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.044+0000 7f7f21800640 1 Processor -- start 2024-03-23T23:14:24.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.044+0000 7f7f21800640 1 -- start start 2024-03-23T23:14:24.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.044+0000 7f7f21800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f1c1036f0 0x7f7f1c199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:24.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.044+0000 7f7f21800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f1c1048e0 0x7f7f1c199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:24.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.044+0000 7f7f21800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1c1052a0 0x7f7f1c1a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:24.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.044+0000 7f7f21800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c10f390 con 0x7f7f1c1052a0 2024-03-23T23:14:24.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.044+0000 7f7f21800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c10f210 con 0x7f7f1c1048e0 2024-03-23T23:14:24.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.044+0000 7f7f21800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c10f510 con 0x7f7f1c1036f0 2024-03-23T23:14:24.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.044+0000 7f7f1be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f1c1036f0 0x7f7f1c199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:24.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.045+0000 7f7f1be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f1c1036f0 0x7f7f1c199a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:50438/0 (socket says 172.21.15.12:50438) 2024-03-23T23:14:24.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.045+0000 7f7f1be00640 1 -- 172.21.15.12:0/1250468190 learned_addr learned my addr 172.21.15.12:0/1250468190 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:14:24.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.045+0000 7f7f13e00640 1 --2- 172.21.15.12:0/1250468190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f1c1048e0 0x7f7f1c199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:24.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.045+0000 7f7f20e00640 1 --2- 172.21.15.12:0/1250468190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1c1052a0 0x7f7f1c1a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:24.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.045+0000 7f7f1be00640 1 -- 172.21.15.12:0/1250468190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f1c1048e0 msgr2=0x7f7f1c199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:24.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.045+0000 7f7f1be00640 1 --2- 172.21.15.12:0/1250468190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f1c1048e0 0x7f7f1c199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:24.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.045+0000 7f7f1be00640 1 -- 172.21.15.12:0/1250468190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1c1052a0 msgr2=0x7f7f1c1a0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:24.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.045+0000 7f7f1be00640 1 --2- 172.21.15.12:0/1250468190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1c1052a0 0x7f7f1c1a0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:24.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.045+0000 7f7f1be00640 1 -- 172.21.15.12:0/1250468190 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f1c1a1620 con 0x7f7f1c1036f0 2024-03-23T23:14:24.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.045+0000 7f7f1be00640 1 --2- 172.21.15.12:0/1250468190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f1c1036f0 0x7f7f1c199a00 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c000c00 tx=0x7f7f0c02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:14:24.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.046+0000 7f7f19600640 1 -- 172.21.15.12:0/1250468190 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c043020 con 0x7f7f1c1036f0 2024-03-23T23:14:24.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.046+0000 7f7f19600640 1 -- 172.21.15.12:0/1250468190 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7f0c0054e0 con 0x7f7f1c1036f0 2024-03-23T23:14:24.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.046+0000 7f7f19600640 1 -- 172.21.15.12:0/1250468190 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c003c40 con 0x7f7f1c1036f0 2024-03-23T23:14:24.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.046+0000 7f7f21800640 1 -- 172.21.15.12:0/1250468190 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f1c1a18b0 con 0x7f7f1c1036f0 2024-03-23T23:14:24.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.046+0000 7f7f21800640 1 -- 172.21.15.12:0/1250468190 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f1c1a1e50 con 0x7f7f1c1036f0 2024-03-23T23:14:24.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.047+0000 7f7f21800640 1 -- 172.21.15.12:0/1250468190 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f1c10aa10 con 0x7f7f1c1036f0 2024-03-23T23:14:24.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.049+0000 7f7f19600640 1 -- 172.21.15.12:0/1250468190 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7f0c046020 con 0x7f7f1c1036f0 2024-03-23T23:14:24.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.050+0000 7f7f19600640 1 --2- 172.21.15.12:0/1250468190 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7f000a2880 0x7f7f000a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:24.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.051+0000 7f7f19600640 1 -- 172.21.15.12:0/1250468190 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7f0c0f0a40 con 0x7f7f1c1036f0 2024-03-23T23:14:24.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.051+0000 7f7f13e00640 1 --2- 172.21.15.12:0/1250468190 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7f000a2880 0x7f7f000a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:24.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.051+0000 7f7f13e00640 1 --2- 172.21.15.12:0/1250468190 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7f000a2880 0x7f7f000a4d40 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c19af10 tx=0x7f7f04005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:14:24.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.054+0000 7f7f19600640 1 -- 172.21.15.12:0/1250468190 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7f0c04d020 con 0x7f7f1c1036f0 2024-03-23T23:14:24.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.261+0000 7f7f21800640 1 -- 172.21.15.12:0/1250468190 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f1c0622a0 con 0x7f7f1c1036f0 2024-03-23T23:14:24.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.263+0000 7f7f19600640 1 -- 172.21.15.12:0/1250468190 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22695 (secure 0 0 0) 0x7f7f0c0bda90 con 0x7f7f1c1036f0 2024-03-23T23:14:24.264 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-23T23:14:24.264 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":88,"fsid":"90f679ae-e967-11ee-95cd-87774f69a715","created":"2024-03-23T22:50:14.906952+0000","modified":"2024-03-23T23:00:24.349052+0000","last_up_change":"2024-03-23T22:58:28.907824+0000","last_in_change":"2024-03-23T22:58:08.342487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T22:54:11.831010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T22:59:25.350974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T22:59:26.390323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b70875d7-a23d-4251-9e8b-1a16d4edc2a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6803","nonce":1190079611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6805","nonce":1190079611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6809","nonce":1190079611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":1190079611},{"type":"v1","addr":"172.21.15.12:6807","nonce":1190079611}]},"public_addr":"172.21.15.12:6803/1190079611","cluster_addr":"172.21.15.12:6805/1190079611","heartbeat_back_addr":"172.21.15.12:6809/1190079611","heartbeat_front_addr":"172.21.15.12:6807/1190079611","state":["exists","up"]},{"osd":1,"uuid":"416e87af-db5d-41ce-84d9-731e3e07e3c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6801","nonce":1983639525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6803","nonce":1983639525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6807","nonce":1983639525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1983639525},{"type":"v1","addr":"172.21.15.173:6805","nonce":1983639525}]},"public_addr":"172.21.15.173:6801/1983639525","cluster_addr":"172.21.15.173:6803/1983639525","heartbeat_back_addr":"172.21.15.173:6807/1983639525","heartbeat_front_addr":"172.21.15.173:6805/1983639525","state":["exists","up"]},{"osd":2,"uuid":"7a559c70-4c54-431b-8baf-4136b6e76493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6801","nonce":2253997387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6803","nonce":2253997387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6807","nonce":2253997387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2253997387},{"type":"v1","addr":"172.21.15.203:6805","nonce":2253997387}]},"public_addr":"172.21.15.203:6801/2253997387","cluster_addr":"172.21.15.203:6803/2253997387","heartbeat_back_addr":"172.21.15.203:6807/2253997387","heartbeat_front_addr":"172.21.15.203:6805/2253997387","state":["exists","up"]},{"osd":3,"uuid":"c41cd287-a674-486c-8025-328cd36e5ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6811","nonce":1761446905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6813","nonce":1761446905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6817","nonce":1761446905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":1761446905},{"type":"v1","addr":"172.21.15.12:6815","nonce":1761446905}]},"public_addr":"172.21.15.12:6811/1761446905","cluster_addr":"172.21.15.12:6813/1761446905","heartbeat_back_addr":"172.21.15.12:6817/1761446905","heartbeat_front_addr":"172.21.15.12:6815/1761446905","state":["exists","up"]},{"osd":4,"uuid":"3c48f886-0b42-4a09-b31e-abccc200bb1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6809","nonce":242958272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6811","nonce":242958272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6815","nonce":242958272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":242958272},{"type":"v1","addr":"172.21.15.173:6813","nonce":242958272}]},"public_addr":"172.21.15.173:6809/242958272","cluster_addr":"172.21.15.173:6811/242958272","heartbeat_back_addr":"172.21.15.173:6815/242958272","heartbeat_front_addr":"172.21.15.173:6813/242958272","state":["exists","up"]},{"osd":5,"uuid":"4a34f06f-8805-40c0-b1ab-23407b0276d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6809","nonce":4270966092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6811","nonce":4270966092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6815","nonce":4270966092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4270966092},{"type":"v1","addr":"172.21.15.203:6813","nonce":4270966092}]},"public_addr":"172.21.15.203:6809/4270966092","cluster_addr":"172.21.15.203:6811/4270966092","heartbeat_back_addr":"172.21.15.203:6815/4270966092","heartbeat_front_addr":"172.21.15.203:6813/4270966092","state":["exists","up"]},{"osd":6,"uuid":"26ddc663-5623-48c9-80d1-f8b968765d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6819","nonce":3606839295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6821","nonce":3606839295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6825","nonce":3606839295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3606839295},{"type":"v1","addr":"172.21.15.12:6823","nonce":3606839295}]},"public_addr":"172.21.15.12:6819/3606839295","cluster_addr":"172.21.15.12:6821/3606839295","heartbeat_back_addr":"172.21.15.12:6825/3606839295","heartbeat_front_addr":"172.21.15.12:6823/3606839295","state":["exists","up"]},{"osd":7,"uuid":"17dc9417-71cd-4576-aa08-933212ec55e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6817","nonce":1241504346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6819","nonce":1241504346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6823","nonce":1241504346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1241504346},{"type":"v1","addr":"172.21.15.173:6821","nonce":1241504346}]},"public_addr":"172.21.15.173:6817/1241504346","cluster_addr":"172.21.15.173:6819/1241504346","heartbeat_back_addr":"172.21.15.173:6823/1241504346","heartbeat_front_addr":"172.21.15.173:6821/1241504346","state":["exists","up"]},{"osd":8,"uuid":"90a2b63a-471d-43b8-ac18-bb1f16778d18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6817","nonce":1516996192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6819","nonce":1516996192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6823","nonce":1516996192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1516996192},{"type":"v1","addr":"172.21.15.203:6821","nonce":1516996192}]},"public_addr":"172.21.15.203:6817/1516996192","cluster_addr":"172.21.15.203:6819/1516996192","heartbeat_back_addr":"172.21.15.203:6823/1516996192","heartbeat_front_addr":"172.21.15.203:6821/1516996192","state":["exists","up"]},{"osd":9,"uuid":"6e1ca504-40b4-49d0-b245-7946181abfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6827","nonce":1175260774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6829","nonce":1175260774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6833","nonce":1175260774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":1175260774},{"type":"v1","addr":"172.21.15.12:6831","nonce":1175260774}]},"public_addr":"172.21.15.12:6827/1175260774","cluster_addr":"172.21.15.12:6829/1175260774","heartbeat_back_addr":"172.21.15.12:6833/1175260774","heartbeat_front_addr":"172.21.15.12:6831/1175260774","state":["exists","up"]},{"osd":10,"uuid":"09a595ed-5614-4386-8e61-0170af66e01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6825","nonce":3303506808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6827","nonce":3303506808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6831","nonce":3303506808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3303506808},{"type":"v1","addr":"172.21.15.173:6829","nonce":3303506808}]},"public_addr":"172.21.15.173:6825/3303506808","cluster_addr":"172.21.15.173:6827/3303506808","heartbeat_back_addr":"172.21.15.173:6831/3303506808","heartbeat_front_addr":"172.21.15.173:6829/3303506808","state":["exists","up"]},{"osd":11,"uuid":"9ef1d595-a977-4ba7-9c6b-d2b3295f20b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6825","nonce":2323604506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6827","nonce":2323604506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6831","nonce":2323604506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2323604506},{"type":"v1","addr":"172.21.15.203:6829","nonce":2323604506}]},"public_addr":"172.21.15.203:6825/2323604506","cluster_addr":"172.21.15.203:6827/2323604506","heartbeat_back_addr":"172.21.15.203:6831/2323604506","heartbeat_front_addr":"172.21.15.203:6829/2323604506","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:18.469038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:53:42.752628+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:07.069493+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:54:41.962478+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:08.363104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:55:33.341336+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:07.286541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:34.243302+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:56:59.749699+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:57:33.822913+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:00.737269+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T22:58:26.599854+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/4182783120":"2024-03-24T22:51:23.113201+0000","172.21.15.12:6800/4127975083":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1900938637":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/3414861652":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6801/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/2580541477":"2024-03-24T22:50:48.401223+0000","172.21.15.12:6800/3270387158":"2024-03-24T22:50:48.401223+0000","172.21.15.12:0/501058839":"2024-03-24T22:51:23.113201+0000","172.21.15.12:0/1192554716":"2024-03-24T22:50:48.401223+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T23:14:24.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.266+0000 7f7f21800640 1 -- 172.21.15.12:0/1250468190 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7f000a2880 msgr2=0x7f7f000a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:24.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.266+0000 7f7f21800640 1 --2- 172.21.15.12:0/1250468190 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7f000a2880 0x7f7f000a4d40 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c19af10 tx=0x7f7f04005f90 comp rx=0 tx=0).stop 2024-03-23T23:14:24.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.266+0000 7f7f21800640 1 -- 172.21.15.12:0/1250468190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f1c1036f0 msgr2=0x7f7f1c199a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:24.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.266+0000 7f7f21800640 1 --2- 172.21.15.12:0/1250468190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f1c1036f0 0x7f7f1c199a00 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c000c00 tx=0x7f7f0c02c650 comp rx=0 tx=0).stop 2024-03-23T23:14:24.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.266+0000 7f7f21800640 1 -- 172.21.15.12:0/1250468190 shutdown_connections 2024-03-23T23:14:24.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.266+0000 7f7f21800640 1 --2- 172.21.15.12:0/1250468190 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f7f000a2880 0x7f7f000a4d40 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:24.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.266+0000 7f7f21800640 1 --2- 172.21.15.12:0/1250468190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1c1052a0 0x7f7f1c1a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:24.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.266+0000 7f7f21800640 1 --2- 172.21.15.12:0/1250468190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f1c1048e0 0x7f7f1c199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:24.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.266+0000 7f7f21800640 1 --2- 172.21.15.12:0/1250468190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f1c1036f0 0x7f7f1c199a00 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:24.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.266+0000 7f7f21800640 1 -- 172.21.15.12:0/1250468190 >> 172.21.15.12:0/1250468190 conn(0x7f7f1c0fec80 msgr2=0x7f7f1c107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:14:24.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.267+0000 7f7f21800640 1 -- 172.21.15.12:0/1250468190 shutdown_connections 2024-03-23T23:14:24.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.267+0000 7f7f21800640 1 -- 172.21.15.12:0/1250468190 wait complete. 2024-03-23T23:14:24.283 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:23 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/966225966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:14:24.283 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:23 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1390493136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:14:24.287 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T23:14:24.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:23 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/966225966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:14:24.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:23 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1390493136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:14:24.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:23 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/966225966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:14:24.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:23 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1390493136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T23:14:24.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.498+0000 7f1279a00640 1 -- 172.21.15.12:0/2299557862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12741015a0 msgr2=0x7f12741019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:24.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.498+0000 7f1279a00640 1 --2- 172.21.15.12:0/2299557862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12741015a0 0x7f12741019c0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f1264009900 tx=0x7f126402bc40 comp rx=0 tx=0).stop 2024-03-23T23:14:24.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.498+0000 7f1279a00640 1 -- 172.21.15.12:0/2299557862 shutdown_connections 2024-03-23T23:14:24.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.498+0000 7f1279a00640 1 --2- 172.21.15.12:0/2299557862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1274103150 0x7f12741099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:24.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.498+0000 7f1279a00640 1 --2- 172.21.15.12:0/2299557862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1274102790 0x7f1274102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:24.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.498+0000 7f1279a00640 1 --2- 172.21.15.12:0/2299557862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12741015a0 0x7f12741019c0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:24.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.498+0000 7f1279a00640 1 -- 172.21.15.12:0/2299557862 >> 172.21.15.12:0/2299557862 conn(0x7f12740fcb10 msgr2=0x7f12740fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:14:24.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.498+0000 7f1279a00640 1 -- 172.21.15.12:0/2299557862 shutdown_connections 2024-03-23T23:14:24.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.498+0000 7f1279a00640 1 -- 172.21.15.12:0/2299557862 wait complete. 2024-03-23T23:14:24.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.499+0000 7f1279a00640 1 Processor -- start 2024-03-23T23:14:24.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.499+0000 7f1279a00640 1 -- start start 2024-03-23T23:14:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.499+0000 7f1279a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12741015a0 0x7f1274199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.499+0000 7f1279a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1274102790 0x7f127419a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.499+0000 7f1279a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1274103150 0x7f12741a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.499+0000 7f1279a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127410d240 con 0x7f1274102790 2024-03-23T23:14:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.499+0000 7f1279a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127410d0c0 con 0x7f1274103150 2024-03-23T23:14:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.499+0000 7f1279a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127410d3c0 con 0x7f12741015a0 2024-03-23T23:14:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.499+0000 7f1273e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12741015a0 0x7f1274199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.499+0000 7f1273e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12741015a0 0x7f1274199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.12:50464/0 (socket says 172.21.15.12:50464) 2024-03-23T23:14:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.499+0000 7f1273400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1274102790 0x7f127419a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.499+0000 7f1273e00640 1 -- 172.21.15.12:0/2004449438 learned_addr learned my addr 172.21.15.12:0/2004449438 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:14:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.499+0000 7f1279000640 1 --2- 172.21.15.12:0/2004449438 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1274103150 0x7f12741a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.500+0000 7f1273e00640 1 -- 172.21.15.12:0/2004449438 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1274103150 msgr2=0x7f12741a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.500+0000 7f1273e00640 1 --2- 172.21.15.12:0/2004449438 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1274103150 0x7f12741a1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.500+0000 7f1273e00640 1 -- 172.21.15.12:0/2004449438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1274102790 msgr2=0x7f127419a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.500+0000 7f1273e00640 1 --2- 172.21.15.12:0/2004449438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1274102790 0x7f127419a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:24.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.500+0000 7f1273e00640 1 -- 172.21.15.12:0/2004449438 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12741a16f0 con 0x7f12741015a0 2024-03-23T23:14:24.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.500+0000 7f1273400640 1 --2- 172.21.15.12:0/2004449438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1274102790 0x7f127419a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:14:24.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.500+0000 7f1273e00640 1 --2- 172.21.15.12:0/2004449438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12741015a0 0x7f1274199ad0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f126402efd0 tx=0x7f126402c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:14:24.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.501+0000 7f1270c00640 1 -- 172.21.15.12:0/2004449438 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1264041070 con 0x7f12741015a0 2024-03-23T23:14:24.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.501+0000 7f1270c00640 1 -- 172.21.15.12:0/2004449438 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1264004450 con 0x7f12741015a0 2024-03-23T23:14:24.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.501+0000 7f1270c00640 1 -- 172.21.15.12:0/2004449438 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1264003e50 con 0x7f12741015a0 2024-03-23T23:14:24.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.501+0000 7f1279a00640 1 -- 172.21.15.12:0/2004449438 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12741a1980 con 0x7f12741015a0 2024-03-23T23:14:24.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.501+0000 7f1279a00640 1 -- 172.21.15.12:0/2004449438 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12741a1e90 con 0x7f12741015a0 2024-03-23T23:14:24.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.502+0000 7f1279a00640 1 -- 172.21.15.12:0/2004449438 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12741a2060 con 0x7f12741015a0 2024-03-23T23:14:24.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.506+0000 7f1270c00640 1 -- 172.21.15.12:0/2004449438 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1264044070 con 0x7f12741015a0 2024-03-23T23:14:24.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.507+0000 7f1270c00640 1 --2- 172.21.15.12:0/2004449438 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f12580a2a20 0x7f12580a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:24.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.507+0000 7f1270c00640 1 -- 172.21.15.12:0/2004449438 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f12640f0920 con 0x7f12741015a0 2024-03-23T23:14:24.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.507+0000 7f1273400640 1 --2- 172.21.15.12:0/2004449438 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f12580a2a20 0x7f12580a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:24.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.508+0000 7f1270c00640 1 -- 172.21.15.12:0/2004449438 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f12640bd320 con 0x7f12741015a0 2024-03-23T23:14:24.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.508+0000 7f1273400640 1 --2- 172.21.15.12:0/2004449438 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f12580a2a20 0x7f12580a4ee0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f127419afe0 tx=0x7f1260005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:14:24.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.750+0000 7f1279a00640 1 -- 172.21.15.12:0/2004449438 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f12741a23f0 con 0x7f12741015a0 2024-03-23T23:14:24.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.758+0000 7f1270c00640 1 -- 172.21.15.12:0/2004449438 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v24) v1 ==== 161+0+0 (secure 0 0 0) 0x7f12741a23f0 con 0x7f12741015a0 2024-03-23T23:14:24.758 INFO:teuthology.orchestra.run.smithi012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T23:14:24.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.761+0000 7f1279a00640 1 -- 172.21.15.12:0/2004449438 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f12580a2a20 msgr2=0x7f12580a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:24.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.761+0000 7f1279a00640 1 --2- 172.21.15.12:0/2004449438 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f12580a2a20 0x7f12580a4ee0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f127419afe0 tx=0x7f1260005f90 comp rx=0 tx=0).stop 2024-03-23T23:14:24.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.761+0000 7f1279a00640 1 -- 172.21.15.12:0/2004449438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12741015a0 msgr2=0x7f1274199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:24.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.761+0000 7f1279a00640 1 --2- 172.21.15.12:0/2004449438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12741015a0 0x7f1274199ad0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f126402efd0 tx=0x7f126402c660 comp rx=0 tx=0).stop 2024-03-23T23:14:24.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.761+0000 7f1279a00640 1 -- 172.21.15.12:0/2004449438 shutdown_connections 2024-03-23T23:14:24.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.761+0000 7f1279a00640 1 --2- 172.21.15.12:0/2004449438 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f12580a2a20 0x7f12580a4ee0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:24.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.761+0000 7f1279a00640 1 --2- 172.21.15.12:0/2004449438 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1274103150 0x7f12741a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:24.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.761+0000 7f1279a00640 1 --2- 172.21.15.12:0/2004449438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1274102790 0x7f127419a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:24.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.761+0000 7f1279a00640 1 --2- 172.21.15.12:0/2004449438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12741015a0 0x7f1274199ad0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:24.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.761+0000 7f1279a00640 1 -- 172.21.15.12:0/2004449438 >> 172.21.15.12:0/2004449438 conn(0x7f12740fcb10 msgr2=0x7f12740738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:14:24.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.762+0000 7f1279a00640 1 -- 172.21.15.12:0/2004449438 shutdown_connections 2024-03-23T23:14:24.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.762+0000 7f1279a00640 1 -- 172.21.15.12:0/2004449438 wait complete. 2024-03-23T23:14:24.780 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-23T23:14:24.929 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:24 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-a[31691]: 2024-03-23T23:14:24.752+0000 7f73be400700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T23:14:24.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.971+0000 7f98ff200640 1 -- 172.21.15.12:0/1223176396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9900102080 msgr2=0x7f9900104470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:24.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.971+0000 7f98ff200640 1 --2- 172.21.15.12:0/1223176396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9900102080 0x7f9900104470 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f98f0009860 tx=0x7f98f002bb70 comp rx=0 tx=0).stop 2024-03-23T23:14:24.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.972+0000 7f98ff200640 1 -- 172.21.15.12:0/1223176396 shutdown_connections 2024-03-23T23:14:24.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.972+0000 7f98ff200640 1 --2- 172.21.15.12:0/1223176396 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99001072e0 0x7f99001096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:24.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.972+0000 7f98ff200640 1 --2- 172.21.15.12:0/1223176396 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99001049b0 0x7f9900106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:24.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.972+0000 7f98ff200640 1 --2- 172.21.15.12:0/1223176396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9900102080 0x7f9900104470 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:24.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.972+0000 7f98ff200640 1 -- 172.21.15.12:0/1223176396 >> 172.21.15.12:0/1223176396 conn(0x7f99000fba10 msgr2=0x7f99000fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:14:24.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.972+0000 7f98ff200640 1 -- 172.21.15.12:0/1223176396 shutdown_connections 2024-03-23T23:14:24.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.972+0000 7f98ff200640 1 -- 172.21.15.12:0/1223176396 wait complete. 2024-03-23T23:14:24.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.973+0000 7f98ff200640 1 Processor -- start 2024-03-23T23:14:24.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.973+0000 7f98ff200640 1 -- start start 2024-03-23T23:14:24.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.973+0000 7f98ff200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9900102080 0x7f9900199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:24.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.973+0000 7f98ff200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99001049b0 0x7f9900199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:24.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.973+0000 7f98ff200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99001072e0 0x7f99001a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:24.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.973+0000 7f98ff200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990010d210 con 0x7f9900102080 2024-03-23T23:14:24.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.973+0000 7f98ff200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990010d090 con 0x7f99001072e0 2024-03-23T23:14:24.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.973+0000 7f98ff200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990010d390 con 0x7f99001049b0 2024-03-23T23:14:24.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.973+0000 7f98fde00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9900102080 0x7f9900199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:24.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.973+0000 7f98fd400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99001049b0 0x7f9900199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:24.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.973+0000 7f98fde00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9900102080 0x7f9900199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47724/0 (socket says 172.21.15.12:47724) 2024-03-23T23:14:24.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.973+0000 7f98fde00640 1 -- 172.21.15.12:0/2948892241 learned_addr learned my addr 172.21.15.12:0/2948892241 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:14:24.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.973+0000 7f98fe800640 1 --2- 172.21.15.12:0/2948892241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99001072e0 0x7f99001a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:24.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.974+0000 7f98fde00640 1 -- 172.21.15.12:0/2948892241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99001049b0 msgr2=0x7f9900199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:24.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.974+0000 7f98fde00640 1 --2- 172.21.15.12:0/2948892241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99001049b0 0x7f9900199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:24.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.974+0000 7f98fde00640 1 -- 172.21.15.12:0/2948892241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99001072e0 msgr2=0x7f99001a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:24.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.974+0000 7f98fde00640 1 --2- 172.21.15.12:0/2948892241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99001072e0 0x7f99001a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:24.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.974+0000 7f98fde00640 1 -- 172.21.15.12:0/2948892241 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99001a16c0 con 0x7f9900102080 2024-03-23T23:14:24.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.974+0000 7f98fde00640 1 --2- 172.21.15.12:0/2948892241 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9900102080 0x7f9900199aa0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f98f00349e0 tx=0x7f98f0032450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:14:24.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.974+0000 7f98f6a00640 1 -- 172.21.15.12:0/2948892241 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f0041030 con 0x7f9900102080 2024-03-23T23:14:24.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.975+0000 7f98f6a00640 1 -- 172.21.15.12:0/2948892241 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f98f002c510 con 0x7f9900102080 2024-03-23T23:14:24.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.975+0000 7f98ff200640 1 -- 172.21.15.12:0/2948892241 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99001a1950 con 0x7f9900102080 2024-03-23T23:14:24.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.975+0000 7f98f6a00640 1 -- 172.21.15.12:0/2948892241 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f0039040 con 0x7f9900102080 2024-03-23T23:14:24.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.975+0000 7f98ff200640 1 -- 172.21.15.12:0/2948892241 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99001a1e90 con 0x7f9900102080 2024-03-23T23:14:24.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.976+0000 7f98ff200640 1 -- 172.21.15.12:0/2948892241 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9900108dd0 con 0x7f9900102080 2024-03-23T23:14:24.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.978+0000 7f98f6a00640 1 -- 172.21.15.12:0/2948892241 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f98f0044070 con 0x7f9900102080 2024-03-23T23:14:24.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.979+0000 7f98f6a00640 1 --2- 172.21.15.12:0/2948892241 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f98e40a2880 0x7f98e40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:24.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.979+0000 7f98f6a00640 1 -- 172.21.15.12:0/2948892241 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8477+0+0 (secure 0 0 0) 0x7f98f00f0080 con 0x7f9900102080 2024-03-23T23:14:24.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.979+0000 7f98fd400640 1 --2- 172.21.15.12:0/2948892241 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f98e40a2880 0x7f98e40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:24.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.980+0000 7f98fd400640 1 --2- 172.21.15.12:0/2948892241 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f98e40a2880 0x7f98e40a4d40 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f990019afb0 tx=0x7f98ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:14:24.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:24.983+0000 7f98f6a00640 1 -- 172.21.15.12:0/2948892241 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f98f00bcfd0 con 0x7f9900102080 2024-03-23T23:14:25.118 INFO:journalctl@ceph.mds.b.smithi173.stdout:Mar 23 23:14:24 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-b[66540]: ignoring --setuser ceph since I am not root 2024-03-23T23:14:25.118 INFO:journalctl@ceph.mds.b.smithi173.stdout:Mar 23 23:14:24 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-b[66540]: ignoring --setgroup ceph since I am not root 2024-03-23T23:14:25.118 INFO:journalctl@ceph.mds.b.smithi173.stdout:Mar 23 23:14:24 smithi173 ceph-mds[66544]: -- 172.21.15.173:0/2573389612 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56359db49a40 con 0x56359db1cc00 2024-03-23T23:14:25.118 INFO:journalctl@ceph.mds.b.smithi173.stdout:Mar 23 23:14:24 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-b[66540]: starting mds.b at 2024-03-23T23:14:25.119 INFO:journalctl@ceph.mds.e.smithi173.stdout:Mar 23 23:14:24 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-e[65912]: ignoring --setuser ceph since I am not root 2024-03-23T23:14:25.119 INFO:journalctl@ceph.mds.e.smithi173.stdout:Mar 23 23:14:24 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-e[65912]: ignoring --setgroup ceph since I am not root 2024-03-23T23:14:25.119 INFO:journalctl@ceph.mds.e.smithi173.stdout:Mar 23 23:14:24 smithi173 ceph-mds[65916]: -- 172.21.15.173:0/2493106111 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a26a3d7a40 con 0x55a26a3aa800 2024-03-23T23:14:25.119 INFO:journalctl@ceph.mds.e.smithi173.stdout:Mar 23 23:14:24 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-e[65912]: starting mds.e at 2024-03-23T23:14:25.119 INFO:journalctl@ceph.mds.h.smithi173.stdout:Mar 23 23:14:24 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-h[66230]: ignoring --setuser ceph since I am not root 2024-03-23T23:14:25.119 INFO:journalctl@ceph.mds.h.smithi173.stdout:Mar 23 23:14:24 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-h[66230]: ignoring --setgroup ceph since I am not root 2024-03-23T23:14:25.119 INFO:journalctl@ceph.mds.h.smithi173.stdout:Mar 23 23:14:24 smithi173 ceph-mds[66234]: -- 172.21.15.173:0/25762548 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a4743aba40 con 0x55a47437e800 2024-03-23T23:14:25.119 INFO:journalctl@ceph.mds.h.smithi173.stdout:Mar 23 23:14:24 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-h[66230]: starting mds.h at 2024-03-23T23:14:25.120 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:24 smithi173 ceph-mon[36116]: pgmap v768: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-23T23:14:25.120 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:24 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1250468190' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:14:25.120 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:24 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2004449438' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-23T23:14:25.120 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:24 smithi173 ceph-mon[36116]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-23T23:14:25.120 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:24 smithi173 ceph-mon[36116]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-23T23:14:25.120 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:24 smithi173 ceph-mon[36116]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T23:14:25.120 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:24 smithi173 ceph-mon[36116]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-23T23:14:25.120 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:24 smithi173 ceph-mon[36116]: osdmap e89: 12 total, 12 up, 12 in 2024-03-23T23:14:25.120 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:24 smithi173 ceph-mon[36116]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-23T23:14:25.157 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 23 23:14:24 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-i[68710]: ignoring --setuser ceph since I am not root 2024-03-23T23:14:25.157 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 23 23:14:24 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-i[68710]: ignoring --setgroup ceph since I am not root 2024-03-23T23:14:25.157 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 23 23:14:24 smithi203 ceph-mds[68714]: -- 172.21.15.203:0/2046183588 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559b820e3a40 con 0x559b820b7000 2024-03-23T23:14:25.158 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 23 23:14:24 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-i[68710]: starting mds.i at 2024-03-23T23:14:25.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:24 smithi203 ceph-mon[36693]: pgmap v768: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-23T23:14:25.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:24 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1250468190' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:14:25.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:24 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2004449438' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-23T23:14:25.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:24 smithi203 ceph-mon[36693]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-23T23:14:25.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:24 smithi203 ceph-mon[36693]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-23T23:14:25.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:24 smithi203 ceph-mon[36693]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T23:14:25.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:24 smithi203 ceph-mon[36693]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-23T23:14:25.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:24 smithi203 ceph-mon[36693]: osdmap e89: 12 total, 12 up, 12 in 2024-03-23T23:14:25.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:24 smithi203 ceph-mon[36693]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-23T23:14:25.159 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 23 23:14:24 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-f[68083]: ignoring --setuser ceph since I am not root 2024-03-23T23:14:25.159 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 23 23:14:24 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-f[68083]: ignoring --setgroup ceph since I am not root 2024-03-23T23:14:25.159 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 23 23:14:24 smithi203 ceph-mds[68087]: -- 172.21.15.203:0/2458976927 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5560dc3b7a40 con 0x5560dc38b000 2024-03-23T23:14:25.159 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 23 23:14:24 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-f[68083]: starting mds.f at 2024-03-23T23:14:25.159 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 23 23:14:24 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-c[68402]: ignoring --setuser ceph since I am not root 2024-03-23T23:14:25.159 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 23 23:14:24 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-c[68402]: ignoring --setgroup ceph since I am not root 2024-03-23T23:14:25.160 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 23 23:14:24 smithi203 ceph-mds[68408]: -- 172.21.15.203:0/617465290 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555c642e9a40 con 0x555c642bc800 2024-03-23T23:14:25.160 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 23 23:14:24 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-c[68402]: starting mds.c at 2024-03-23T23:14:25.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.217+0000 7f98ff200640 1 -- 172.21.15.12:0/2948892241 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f99000622a0 con 0x7f9900102080 2024-03-23T23:14:25.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:24 smithi012 ceph-mon[31695]: pgmap v768: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-23T23:14:25.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:24 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1250468190' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T23:14:25.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:24 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2004449438' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-23T23:14:25.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:24 smithi012 ceph-mon[31695]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-23T23:14:25.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:24 smithi012 ceph-mon[31695]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-23T23:14:25.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:24 smithi012 ceph-mon[31695]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T23:14:25.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:24 smithi012 ceph-mon[31695]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-23T23:14:25.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:24 smithi012 ceph-mon[31695]: osdmap e89: 12 total, 12 up, 12 in 2024-03-23T23:14:25.305 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:24 smithi012 ceph-mon[31695]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-23T23:14:25.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.758+0000 7f98f6a00640 1 -- 172.21.15.12:0/2948892241 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7f98f00bc8d0 con 0x7f9900102080 2024-03-23T23:14:25.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.761+0000 7f98ff200640 1 -- 172.21.15.12:0/2948892241 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f98e40a2880 msgr2=0x7f98e40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:25.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.761+0000 7f98ff200640 1 --2- 172.21.15.12:0/2948892241 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f98e40a2880 0x7f98e40a4d40 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f990019afb0 tx=0x7f98ec009260 comp rx=0 tx=0).stop 2024-03-23T23:14:25.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.761+0000 7f98ff200640 1 -- 172.21.15.12:0/2948892241 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9900102080 msgr2=0x7f9900199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:25.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.761+0000 7f98ff200640 1 --2- 172.21.15.12:0/2948892241 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9900102080 0x7f9900199aa0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f98f00349e0 tx=0x7f98f0032450 comp rx=0 tx=0).stop 2024-03-23T23:14:25.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.761+0000 7f98ff200640 1 -- 172.21.15.12:0/2948892241 shutdown_connections 2024-03-23T23:14:25.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.761+0000 7f98ff200640 1 --2- 172.21.15.12:0/2948892241 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f98e40a2880 0x7f98e40a4d40 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:25.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.761+0000 7f98ff200640 1 --2- 172.21.15.12:0/2948892241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99001072e0 0x7f99001a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:25.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.761+0000 7f98ff200640 1 --2- 172.21.15.12:0/2948892241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99001049b0 0x7f9900199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:25.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.761+0000 7f98ff200640 1 --2- 172.21.15.12:0/2948892241 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9900102080 0x7f9900199aa0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:25.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.761+0000 7f98ff200640 1 -- 172.21.15.12:0/2948892241 >> 172.21.15.12:0/2948892241 conn(0x7f99000fba10 msgr2=0x7f9900101810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:14:25.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.762+0000 7f98ff200640 1 -- 172.21.15.12:0/2948892241 shutdown_connections 2024-03-23T23:14:25.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.762+0000 7f98ff200640 1 -- 172.21.15.12:0/2948892241 wait complete. 2024-03-23T23:14:25.785 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-23T23:14:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.992+0000 7f12d6600640 1 -- 172.21.15.12:0/3408789853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12d0108db0 msgr2=0x7f12d0109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.992+0000 7f12d6600640 1 --2- 172.21.15.12:0/3408789853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12d0108db0 0x7f12d0109230 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f12bc009900 tx=0x7f12bc02bc30 comp rx=0 tx=0).stop 2024-03-23T23:14:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.993+0000 7f12d6600640 1 -- 172.21.15.12:0/3408789853 shutdown_connections 2024-03-23T23:14:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.993+0000 7f12d6600640 1 --2- 172.21.15.12:0/3408789853 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12d0109770 0x7f12d0110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.993+0000 7f12d6600640 1 --2- 172.21.15.12:0/3408789853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12d0108db0 0x7f12d0109230 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.993+0000 7f12d6600640 1 --2- 172.21.15.12:0/3408789853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12d0107bc0 0x7f12d0107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.993+0000 7f12d6600640 1 -- 172.21.15.12:0/3408789853 >> 172.21.15.12:0/3408789853 conn(0x7f12d0076b70 msgr2=0x7f12d0078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:14:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.993+0000 7f12d6600640 1 -- 172.21.15.12:0/3408789853 shutdown_connections 2024-03-23T23:14:25.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.993+0000 7f12d6600640 1 -- 172.21.15.12:0/3408789853 wait complete. 2024-03-23T23:14:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.994+0000 7f12d6600640 1 Processor -- start 2024-03-23T23:14:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.994+0000 7f12d6600640 1 -- start start 2024-03-23T23:14:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.994+0000 7f12d6600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12d0107bc0 0x7f12d019deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.994+0000 7f12d6600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12d0108db0 0x7f12d019e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.994+0000 7f12d6600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12d0109770 0x7f12d01a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.994+0000 7f12d6600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d0113890 con 0x7f12d0108db0 2024-03-23T23:14:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.994+0000 7f12d6600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d0113710 con 0x7f12d0109770 2024-03-23T23:14:25.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.994+0000 7f12d6600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d0113a10 con 0x7f12d0107bc0 2024-03-23T23:14:25.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.995+0000 7f12cfe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12d0108db0 0x7f12d019e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:25.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.995+0000 7f12d5c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12d0109770 0x7f12d01a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:25.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.995+0000 7f12d5200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12d0107bc0 0x7f12d019deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:25.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.995+0000 7f12d5c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12d0109770 0x7f12d01a5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.12:56154/0 (socket says 172.21.15.12:56154) 2024-03-23T23:14:25.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.995+0000 7f12d5c00640 1 -- 172.21.15.12:0/4089687156 learned_addr learned my addr 172.21.15.12:0/4089687156 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:14:25.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.995+0000 7f12cfe00640 1 -- 172.21.15.12:0/4089687156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12d0107bc0 msgr2=0x7f12d019deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:25.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.995+0000 7f12cfe00640 1 --2- 172.21.15.12:0/4089687156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12d0107bc0 0x7f12d019deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:25.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.995+0000 7f12cfe00640 1 -- 172.21.15.12:0/4089687156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12d0109770 msgr2=0x7f12d01a5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:25.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.995+0000 7f12cfe00640 1 --2- 172.21.15.12:0/4089687156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12d0109770 0x7f12d01a5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.995+0000 7f12cfe00640 1 -- 172.21.15.12:0/4089687156 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d01a5ad0 con 0x7f12d0108db0 2024-03-23T23:14:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.995+0000 7f12cfe00640 1 --2- 172.21.15.12:0/4089687156 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12d0108db0 0x7f12d019e3f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f12bc036040 tx=0x7f12bc02c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:14:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.996+0000 7f12cd600640 1 -- 172.21.15.12:0/4089687156 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12bc043020 con 0x7f12d0108db0 2024-03-23T23:14:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.996+0000 7f12cd600640 1 -- 172.21.15.12:0/4089687156 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12bc0054e0 con 0x7f12d0108db0 2024-03-23T23:14:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.996+0000 7f12d6600640 1 -- 172.21.15.12:0/4089687156 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12d01a5d60 con 0x7f12d0108db0 2024-03-23T23:14:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.996+0000 7f12cd600640 1 -- 172.21.15.12:0/4089687156 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12bc003ca0 con 0x7f12d0108db0 2024-03-23T23:14:26.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.996+0000 7f12d6600640 1 -- 172.21.15.12:0/4089687156 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12d01a6270 con 0x7f12d0108db0 2024-03-23T23:14:26.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.997+0000 7f12d6600640 1 -- 172.21.15.12:0/4089687156 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12d010eee0 con 0x7f12d0108db0 2024-03-23T23:14:26.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:25.999+0000 7f12cd600640 1 -- 172.21.15.12:0/4089687156 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f12bc046020 con 0x7f12d0108db0 2024-03-23T23:14:26.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:26.001+0000 7f12cd600640 1 --2- 172.21.15.12:0/4089687156 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f12b80a2880 0x7f12b80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:26.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:26.002+0000 7f12cd600640 1 -- 172.21.15.12:0/4089687156 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8477+0+0 (secure 0 0 0) 0x7f12bc0f0b80 con 0x7f12d0108db0 2024-03-23T23:14:26.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:26.002+0000 7f12d5200640 1 --2- 172.21.15.12:0/4089687156 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f12b80a2880 0x7f12b80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:26.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:26.002+0000 7f12d5200640 1 --2- 172.21.15.12:0/4089687156 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f12b80a2880 0x7f12b80a4d40 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f12d0108c10 tx=0x7f12c400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:14:26.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:26.006+0000 7f12cd600640 1 -- 172.21.15.12:0/4089687156 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f12bc04d020 con 0x7f12d0108db0 2024-03-23T23:14:26.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:25 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2948892241' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-23T23:14:26.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:25 smithi012 ceph-mon[31695]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-23T23:14:26.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:25 smithi012 ceph-mon[31695]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-23T23:14:26.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:25 smithi012 ceph-mon[31695]: Cluster is now healthy 2024-03-23T23:14:26.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:25 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/2948892241' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-23T23:14:26.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:25 smithi012 ceph-mon[31695]: mds.? [v2:172.21.15.173:6834/3725732400,v1:172.21.15.173:6835/3725732400] up:boot 2024-03-23T23:14:26.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:25 smithi012 ceph-mon[31695]: mds.? [v2:172.21.15.173:6836/708787681,v1:172.21.15.173:6837/708787681] up:boot 2024-03-23T23:14:26.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:25 smithi012 ceph-mon[31695]: mds.? [v2:172.21.15.203:6837/228282977,v1:172.21.15.203:6839/228282977] up:boot 2024-03-23T23:14:26.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:25 smithi012 ceph-mon[31695]: mds.? [v2:172.21.15.173:6838/1903079097,v1:172.21.15.173:6839/1903079097] up:boot 2024-03-23T23:14:26.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:25 smithi012 ceph-mon[31695]: mds.? [v2:172.21.15.203:6834/3996612453,v1:172.21.15.203:6835/3996612453] up:boot 2024-03-23T23:14:26.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:25 smithi012 ceph-mon[31695]: mds.? [v2:172.21.15.203:6836/4277125735,v1:172.21.15.203:6838/4277125735] up:boot 2024-03-23T23:14:26.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:25 smithi012 ceph-mon[31695]: fsmap 12 up:standby 2024-03-23T23:14:26.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:25 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-23T23:14:26.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:25 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-23T23:14:26.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:25 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-23T23:14:26.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:25 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-23T23:14:26.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:25 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-23T23:14:26.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:25 smithi012 ceph-mon[31695]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-23T23:14:26.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:26.207+0000 7f12d6600640 1 -- 172.21.15.12:0/4089687156 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f12d00622a0 con 0x7f12d0108db0 2024-03-23T23:14:26.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:25 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2948892241' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-23T23:14:26.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:25 smithi173 ceph-mon[36116]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-23T23:14:26.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:25 smithi173 ceph-mon[36116]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-23T23:14:26.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:25 smithi173 ceph-mon[36116]: Cluster is now healthy 2024-03-23T23:14:26.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:25 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/2948892241' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-23T23:14:26.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:25 smithi173 ceph-mon[36116]: mds.? [v2:172.21.15.173:6834/3725732400,v1:172.21.15.173:6835/3725732400] up:boot 2024-03-23T23:14:26.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:25 smithi173 ceph-mon[36116]: mds.? [v2:172.21.15.173:6836/708787681,v1:172.21.15.173:6837/708787681] up:boot 2024-03-23T23:14:26.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:25 smithi173 ceph-mon[36116]: mds.? [v2:172.21.15.203:6837/228282977,v1:172.21.15.203:6839/228282977] up:boot 2024-03-23T23:14:26.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:25 smithi173 ceph-mon[36116]: mds.? [v2:172.21.15.173:6838/1903079097,v1:172.21.15.173:6839/1903079097] up:boot 2024-03-23T23:14:26.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:25 smithi173 ceph-mon[36116]: mds.? [v2:172.21.15.203:6834/3996612453,v1:172.21.15.203:6835/3996612453] up:boot 2024-03-23T23:14:26.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:25 smithi173 ceph-mon[36116]: mds.? [v2:172.21.15.203:6836/4277125735,v1:172.21.15.203:6838/4277125735] up:boot 2024-03-23T23:14:26.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:25 smithi173 ceph-mon[36116]: fsmap 12 up:standby 2024-03-23T23:14:26.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:25 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-23T23:14:26.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:25 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-23T23:14:26.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:25 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-23T23:14:26.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:25 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-23T23:14:26.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:25 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-23T23:14:26.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:25 smithi173 ceph-mon[36116]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-23T23:14:26.410 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:25 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2948892241' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-23T23:14:26.410 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:25 smithi203 ceph-mon[36693]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-23T23:14:26.410 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:25 smithi203 ceph-mon[36693]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-23T23:14:26.410 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:25 smithi203 ceph-mon[36693]: Cluster is now healthy 2024-03-23T23:14:26.410 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:25 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/2948892241' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-23T23:14:26.410 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:25 smithi203 ceph-mon[36693]: mds.? [v2:172.21.15.173:6834/3725732400,v1:172.21.15.173:6835/3725732400] up:boot 2024-03-23T23:14:26.410 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:25 smithi203 ceph-mon[36693]: mds.? [v2:172.21.15.173:6836/708787681,v1:172.21.15.173:6837/708787681] up:boot 2024-03-23T23:14:26.411 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:25 smithi203 ceph-mon[36693]: mds.? [v2:172.21.15.203:6837/228282977,v1:172.21.15.203:6839/228282977] up:boot 2024-03-23T23:14:26.411 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:25 smithi203 ceph-mon[36693]: mds.? [v2:172.21.15.173:6838/1903079097,v1:172.21.15.173:6839/1903079097] up:boot 2024-03-23T23:14:26.411 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:25 smithi203 ceph-mon[36693]: mds.? [v2:172.21.15.203:6834/3996612453,v1:172.21.15.203:6835/3996612453] up:boot 2024-03-23T23:14:26.411 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:25 smithi203 ceph-mon[36693]: mds.? [v2:172.21.15.203:6836/4277125735,v1:172.21.15.203:6838/4277125735] up:boot 2024-03-23T23:14:26.411 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:25 smithi203 ceph-mon[36693]: fsmap 12 up:standby 2024-03-23T23:14:26.411 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:25 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-23T23:14:26.411 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:25 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-23T23:14:26.411 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:25 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-23T23:14:26.411 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:25 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-23T23:14:26.411 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:25 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-23T23:14:26.411 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:25 smithi203 ceph-mon[36693]: from='mgr.14152 172.21.15.12:0/3546349128' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-23T23:14:26.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:26.951+0000 7f12cd600640 1 -- 172.21.15.12:0/4089687156 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v90) v1 ==== 181+0+0 (secure 0 0 0) 0x7f12bc0bdad0 con 0x7f12d0108db0 2024-03-23T23:14:26.952 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' removed 2024-03-23T23:14:26.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:26.955+0000 7f12d6600640 1 -- 172.21.15.12:0/4089687156 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f12b80a2880 msgr2=0x7f12b80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:26.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:26.955+0000 7f12d6600640 1 --2- 172.21.15.12:0/4089687156 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f12b80a2880 0x7f12b80a4d40 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f12d0108c10 tx=0x7f12c400b390 comp rx=0 tx=0).stop 2024-03-23T23:14:26.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:26.955+0000 7f12d6600640 1 -- 172.21.15.12:0/4089687156 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12d0108db0 msgr2=0x7f12d019e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:26.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:26.955+0000 7f12d6600640 1 --2- 172.21.15.12:0/4089687156 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12d0108db0 0x7f12d019e3f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f12bc036040 tx=0x7f12bc02c6c0 comp rx=0 tx=0).stop 2024-03-23T23:14:26.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:26.956+0000 7f12d6600640 1 -- 172.21.15.12:0/4089687156 shutdown_connections 2024-03-23T23:14:26.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:26.956+0000 7f12d6600640 1 --2- 172.21.15.12:0/4089687156 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f12b80a2880 0x7f12b80a4d40 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:26.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:26.956+0000 7f12d6600640 1 --2- 172.21.15.12:0/4089687156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12d0109770 0x7f12d01a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:26.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:26.956+0000 7f12d6600640 1 --2- 172.21.15.12:0/4089687156 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12d0108db0 0x7f12d019e3f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:26.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:26.956+0000 7f12d6600640 1 --2- 172.21.15.12:0/4089687156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12d0107bc0 0x7f12d019deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:26.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:26.956+0000 7f12d6600640 1 -- 172.21.15.12:0/4089687156 >> 172.21.15.12:0/4089687156 conn(0x7f12d0076b70 msgr2=0x7f12d010c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:14:26.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:26.956+0000 7f12d6600640 1 -- 172.21.15.12:0/4089687156 shutdown_connections 2024-03-23T23:14:26.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:26.956+0000 7f12d6600640 1 -- 172.21.15.12:0/4089687156 wait complete. 2024-03-23T23:14:26.983 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-23T23:14:27.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.211+0000 7f5e3b200640 1 -- 172.21.15.12:0/3772792482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e3c105350 msgr2=0x7f5e3c10bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:27.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.211+0000 7f5e3b200640 1 --2- 172.21.15.12:0/3772792482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e3c105350 0x7f5e3c10bbe0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f5e34009be0 tx=0x7f5e3402bef0 comp rx=0 tx=0).stop 2024-03-23T23:14:27.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.211+0000 7f5e3b200640 1 -- 172.21.15.12:0/3772792482 shutdown_connections 2024-03-23T23:14:27.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.211+0000 7f5e3b200640 1 --2- 172.21.15.12:0/3772792482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e3c105350 0x7f5e3c10bbe0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:27.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.211+0000 7f5e3b200640 1 --2- 172.21.15.12:0/3772792482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e3c104990 0x7f5e3c104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:27.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.211+0000 7f5e3b200640 1 --2- 172.21.15.12:0/3772792482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e3c1037a0 0x7f5e3c103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:27.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.211+0000 7f5e3b200640 1 -- 172.21.15.12:0/3772792482 >> 172.21.15.12:0/3772792482 conn(0x7f5e3c0fed50 msgr2=0x7f5e3c101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:14:27.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.211+0000 7f5e3b200640 1 -- 172.21.15.12:0/3772792482 shutdown_connections 2024-03-23T23:14:27.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.211+0000 7f5e3b200640 1 -- 172.21.15.12:0/3772792482 wait complete. 2024-03-23T23:14:27.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.212+0000 7f5e3b200640 1 Processor -- start 2024-03-23T23:14:27.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.212+0000 7f5e3b200640 1 -- start start 2024-03-23T23:14:27.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.212+0000 7f5e3b200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e3c1037a0 0x7f5e3c199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:27.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.212+0000 7f5e3b200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e3c104990 0x7f5e3c199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:27.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.213+0000 7f5e3b200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e3c105350 0x7f5e3c1a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:27.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.213+0000 7f5e3b200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c10f420 con 0x7f5e3c1037a0 2024-03-23T23:14:27.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.213+0000 7f5e3b200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c10f2a0 con 0x7f5e3c105350 2024-03-23T23:14:27.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.213+0000 7f5e3b200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c10f5a0 con 0x7f5e3c104990 2024-03-23T23:14:27.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.213+0000 7f5e39e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e3c1037a0 0x7f5e3c199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:27.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.213+0000 7f5e39e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e3c1037a0 0x7f5e3c199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47756/0 (socket says 172.21.15.12:47756) 2024-03-23T23:14:27.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.213+0000 7f5e39e00640 1 -- 172.21.15.12:0/1915054025 learned_addr learned my addr 172.21.15.12:0/1915054025 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-23T23:14:27.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.213+0000 7f5e3a800640 1 --2- 172.21.15.12:0/1915054025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e3c105350 0x7f5e3c1a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:27.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.213+0000 7f5e31400640 1 --2- 172.21.15.12:0/1915054025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e3c104990 0x7f5e3c199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:27.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.213+0000 7f5e39e00640 1 -- 172.21.15.12:0/1915054025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e3c104990 msgr2=0x7f5e3c199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:27.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.213+0000 7f5e39e00640 1 --2- 172.21.15.12:0/1915054025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e3c104990 0x7f5e3c199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:27.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.213+0000 7f5e39e00640 1 -- 172.21.15.12:0/1915054025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e3c105350 msgr2=0x7f5e3c1a1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:27.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.213+0000 7f5e39e00640 1 --2- 172.21.15.12:0/1915054025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e3c105350 0x7f5e3c1a1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:27.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.213+0000 7f5e39e00640 1 -- 172.21.15.12:0/1915054025 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c1a1690 con 0x7f5e3c1037a0 2024-03-23T23:14:27.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.213+0000 7f5e31400640 1 --2- 172.21.15.12:0/1915054025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e3c104990 0x7f5e3c199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T23:14:27.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.213+0000 7f5e3a800640 1 --2- 172.21.15.12:0/1915054025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e3c105350 0x7f5e3c1a1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T23:14:27.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.214+0000 7f5e39e00640 1 --2- 172.21.15.12:0/1915054025 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e3c1037a0 0x7f5e3c199a70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c00e9c0 tx=0x7f5e2c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:14:27.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.214+0000 7f5e33400640 1 -- 172.21.15.12:0/1915054025 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c00cd80 con 0x7f5e3c1037a0 2024-03-23T23:14:27.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.214+0000 7f5e33400640 1 -- 172.21.15.12:0/1915054025 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e2c0035f0 con 0x7f5e3c1037a0 2024-03-23T23:14:27.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.214+0000 7f5e3b200640 1 -- 172.21.15.12:0/1915054025 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3c1a1980 con 0x7f5e3c1037a0 2024-03-23T23:14:27.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.214+0000 7f5e33400640 1 -- 172.21.15.12:0/1915054025 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c010690 con 0x7f5e3c1037a0 2024-03-23T23:14:27.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.214+0000 7f5e3b200640 1 -- 172.21.15.12:0/1915054025 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3c1a1e80 con 0x7f5e3c1037a0 2024-03-23T23:14:27.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.215+0000 7f5e3b200640 1 -- 172.21.15.12:0/1915054025 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e3c10aa60 con 0x7f5e3c1037a0 2024-03-23T23:14:27.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.217+0000 7f5e33400640 1 -- 172.21.15.12:0/1915054025 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5e2c0037a0 con 0x7f5e3c1037a0 2024-03-23T23:14:27.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.218+0000 7f5e33400640 1 --2- 172.21.15.12:0/1915054025 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5e200a2800 0x7f5e200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T23:14:27.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.219+0000 7f5e33400640 1 -- 172.21.15.12:0/1915054025 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 7944+0+0 (secure 0 0 0) 0x7f5e2c0ceda0 con 0x7f5e3c1037a0 2024-03-23T23:14:27.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.219+0000 7f5e31400640 1 --2- 172.21.15.12:0/1915054025 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5e200a2800 0x7f5e200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T23:14:27.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.219+0000 7f5e31400640 1 --2- 172.21.15.12:0/1915054025 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5e200a2800 0x7f5e200a4cc0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c19af80 tx=0x7f5e24009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T23:14:27.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.223+0000 7f5e33400640 1 -- 172.21.15.12:0/1915054025 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5e2c09bf10 con 0x7f5e3c1037a0 2024-03-23T23:14:27.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:26 smithi012 ceph-mon[31695]: pgmap v770: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-23T23:14:27.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:26 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/4089687156' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-23T23:14:27.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:26 smithi173 ceph-mon[36116]: pgmap v770: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-23T23:14:27.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:26 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/4089687156' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-23T23:14:27.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:26 smithi203 ceph-mon[36693]: pgmap v770: 97 pgs: 97 active+clean; 4.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-23T23:14:27.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:26 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/4089687156' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-23T23:14:27.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.423+0000 7f5e3b200640 1 -- 172.21.15.12:0/1915054025 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5e3c02d070 con 0x7f5e3c1037a0 2024-03-23T23:14:27.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.967+0000 7f5e33400640 1 -- 172.21.15.12:0/1915054025 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v91) v1 ==== 169+0+0 (secure 0 0 0) 0x7f5e2c09b810 con 0x7f5e3c1037a0 2024-03-23T23:14:27.967 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' removed 2024-03-23T23:14:27.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.971+0000 7f5e3b200640 1 -- 172.21.15.12:0/1915054025 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5e200a2800 msgr2=0x7f5e200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:27.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.971+0000 7f5e3b200640 1 --2- 172.21.15.12:0/1915054025 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5e200a2800 0x7f5e200a4cc0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c19af80 tx=0x7f5e24009260 comp rx=0 tx=0).stop 2024-03-23T23:14:27.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.971+0000 7f5e3b200640 1 -- 172.21.15.12:0/1915054025 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e3c1037a0 msgr2=0x7f5e3c199a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T23:14:27.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.971+0000 7f5e3b200640 1 --2- 172.21.15.12:0/1915054025 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e3c1037a0 0x7f5e3c199a70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c00e9c0 tx=0x7f5e2c00ee80 comp rx=0 tx=0).stop 2024-03-23T23:14:27.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.972+0000 7f5e3b200640 1 -- 172.21.15.12:0/1915054025 shutdown_connections 2024-03-23T23:14:27.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.972+0000 7f5e3b200640 1 --2- 172.21.15.12:0/1915054025 >> [v2:172.21.15.12:6800/1345467734,v1:172.21.15.12:6801/1345467734] conn(0x7f5e200a2800 0x7f5e200a4cc0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:27.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.972+0000 7f5e3b200640 1 --2- 172.21.15.12:0/1915054025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e3c105350 0x7f5e3c1a1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:27.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.972+0000 7f5e3b200640 1 --2- 172.21.15.12:0/1915054025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e3c104990 0x7f5e3c199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:27.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.972+0000 7f5e3b200640 1 --2- 172.21.15.12:0/1915054025 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e3c1037a0 0x7f5e3c199a70 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T23:14:27.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.972+0000 7f5e3b200640 1 -- 172.21.15.12:0/1915054025 >> 172.21.15.12:0/1915054025 conn(0x7f5e3c0fed50 msgr2=0x7f5e3c107c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T23:14:27.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.973+0000 7f5e3b200640 1 -- 172.21.15.12:0/1915054025 shutdown_connections 2024-03-23T23:14:27.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-23T23:14:27.973+0000 7f5e3b200640 1 -- 172.21.15.12:0/1915054025 wait complete. 2024-03-23T23:14:28.002 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-23T23:14:28.092 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-23T23:14:28.127 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-23T23:14:28.163 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-23T23:14:28.163 DEBUG:teuthology.orchestra.run.smithi012:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-23T23:14:28.223 DEBUG:teuthology.orchestra.run.smithi173:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-23T23:14:28.241 DEBUG:teuthology.orchestra.run.smithi203:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-23T23:14:28.258 INFO:tasks.cephadm:Stopping all daemons... 2024-03-23T23:14:28.258 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-23T23:14:28.258 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@mon.a 2024-03-23T23:14:28.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:27 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/4089687156' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-23T23:14:28.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:27 smithi012 ceph-mon[31695]: osdmap e90: 12 total, 12 up, 12 in 2024-03-23T23:14:28.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:27 smithi012 ceph-mon[31695]: from='client.? 172.21.15.12:0/1915054025' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-23T23:14:28.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:27 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/4089687156' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-23T23:14:28.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:27 smithi173 ceph-mon[36116]: osdmap e90: 12 total, 12 up, 12 in 2024-03-23T23:14:28.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:27 smithi173 ceph-mon[36116]: from='client.? 172.21.15.12:0/1915054025' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-23T23:14:28.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:27 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/4089687156' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-23T23:14:28.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:27 smithi203 ceph-mon[36693]: osdmap e90: 12 total, 12 up, 12 in 2024-03-23T23:14:28.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:27 smithi203 ceph-mon[36693]: from='client.? 172.21.15.12:0/1915054025' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-23T23:14:28.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:28 smithi012 systemd[1]: Stopping Ceph mon.a for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:14:29.296 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:28 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-a[31691]: 2024-03-23T23:14:28.914+0000 7f73c5200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:14:29.296 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:28 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-a[31691]: 2024-03-23T23:14:28.914+0000 7f73c5200700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-23T23:14:29.296 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:28 smithi012 podman[100463]: 2024-03-23 23:14:28.958089823 +0000 UTC m=+0.197731536 container died bdb2aed276e79f0d904555aa6918d03e2fd780229072949a875ee8254c37765b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-23T23:14:29.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:29 smithi012 podman[100463]: 2024-03-23 23:14:29.44057531 +0000 UTC m=+0.680217047 container cleanup bdb2aed276e79f0d904555aa6918d03e2fd780229072949a875ee8254c37765b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-23T23:14:29.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:29 smithi012 bash[100463]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-a 2024-03-23T23:14:30.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 23 23:14:29 smithi012 podman[100477]: 2024-03-23 23:14:29.700831361 +0000 UTC m=+0.729202867 container remove bdb2aed276e79f0d904555aa6918d03e2fd780229072949a875ee8254c37765b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True) 2024-03-23T23:14:30.241 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mon.a.service' 2024-03-23T23:14:30.450 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:14:30.450 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-23T23:14:30.451 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-23T23:14:30.451 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@mon.b 2024-03-23T23:14:30.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:30 smithi173 systemd[1]: Stopping Ceph mon.b for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:14:31.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:30 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-b[36112]: 2024-03-23T23:14:30.765+0000 7f2151c1c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:14:31.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:30 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-b[36112]: 2024-03-23T23:14:30.765+0000 7f2151c1c700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-23T23:14:31.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:31 smithi173 podman[71144]: 2024-03-23 23:14:31.010958439 +0000 UTC m=+0.363044015 container died 72286162409a4f9c012e1fe519afd4df1fd68da8a2593c2e35871bab01498d36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-23T23:14:31.725 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:31 smithi173 podman[71144]: 2024-03-23 23:14:31.483495467 +0000 UTC m=+0.835581034 container cleanup 72286162409a4f9c012e1fe519afd4df1fd68da8a2593c2e35871bab01498d36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-b, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-23T23:14:31.725 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 23 23:14:31 smithi173 bash[71144]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-b 2024-03-23T23:14:31.828 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mon.b.service' 2024-03-23T23:14:31.879 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:14:31.879 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-23T23:14:31.879 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-23T23:14:31.879 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@mon.c 2024-03-23T23:14:32.275 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:31 smithi203 systemd[1]: Stopping Ceph mon.c for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:14:32.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:32 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-c[36689]: 2024-03-23T23:14:32.275+0000 7f6faf805700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:14:32.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:32 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-c[36689]: 2024-03-23T23:14:32.275+0000 7f6faf805700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-23T23:14:32.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:32 smithi203 podman[73945]: 2024-03-23 23:14:32.398639903 +0000 UTC m=+0.321764985 container died f46e88dc1a11fc46375d0e892e4d6ebd408c6480a927ebc8cb0c8dfa47378640 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-23T23:14:33.060 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:32 smithi203 podman[73945]: 2024-03-23 23:14:32.759708804 +0000 UTC m=+0.682833874 container cleanup f46e88dc1a11fc46375d0e892e4d6ebd408c6480a927ebc8cb0c8dfa47378640 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-c, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-23T23:14:33.060 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 23 23:14:32 smithi203 bash[73945]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-mon-c 2024-03-23T23:14:33.224 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mon.c.service' 2024-03-23T23:14:33.274 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:14:33.274 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-23T23:14:33.274 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-23T23:14:33.274 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@mgr.x 2024-03-23T23:14:33.803 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 23:14:33 smithi012 systemd[1]: Stopping Ceph mgr.x for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:14:34.304 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 23:14:33 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T23:14:33.901+0000 7f7080400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:14:34.304 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 23:14:33 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x[31925]: 2024-03-23T23:14:33.901+0000 7f7080400700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-23T23:14:34.304 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 23:14:33 smithi012 podman[100584]: 2024-03-23 23:14:33.994701979 +0000 UTC m=+0.232949032 container died 1b29f43ea2ee610ca75b9576dcf67eda7f1fe9916c35cecf067898434254804e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-23T23:14:34.803 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 23:14:34 smithi012 podman[100584]: 2024-03-23 23:14:34.621605494 +0000 UTC m=+0.859852589 container cleanup 1b29f43ea2ee610ca75b9576dcf67eda7f1fe9916c35cecf067898434254804e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-23T23:14:34.804 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 23:14:34 smithi012 bash[100584]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x 2024-03-23T23:14:35.303 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 23 23:14:34 smithi012 podman[100598]: 2024-03-23 23:14:34.840262933 +0000 UTC m=+0.829918394 container remove 1b29f43ea2ee610ca75b9576dcf67eda7f1fe9916c35cecf067898434254804e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-x, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-23T23:14:35.347 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mgr.x.service' 2024-03-23T23:14:35.468 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:14:35.468 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-23T23:14:35.468 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-23T23:14:35.468 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@mgr.y 2024-03-23T23:14:35.808 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 23:14:35 smithi173 systemd[1]: Stopping Ceph mgr.y for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:14:36.117 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 23:14:35 smithi173 podman[71279]: 2024-03-23 23:14:35.809454332 +0000 UTC m=+0.147061423 container died 8fa22e2bce06b8119dbc6a585b9e58ac486d06df8b6333fe25e0b0eed935caab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-23T23:14:36.431 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 23:14:36 smithi173 podman[71279]: 2024-03-23 23:14:36.222836579 +0000 UTC m=+0.560443662 container cleanup 8fa22e2bce06b8119dbc6a585b9e58ac486d06df8b6333fe25e0b0eed935caab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS) 2024-03-23T23:14:36.431 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Mar 23 23:14:36 smithi173 bash[71279]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-y 2024-03-23T23:14:36.640 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mgr.y.service' 2024-03-23T23:14:36.678 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:14:36.678 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-23T23:14:36.678 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-23T23:14:36.678 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@mgr.z 2024-03-23T23:14:36.992 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 23:14:36 smithi203 systemd[1]: Stopping Ceph mgr.z for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:14:37.273 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 23:14:36 smithi203 podman[74068]: 2024-03-23 23:14:36.993758636 +0000 UTC m=+0.126887681 container died b6f6dfb7120a8d84226d0f3fa7ff5580ebb692d74e1d9baffa969de3b8028be2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-23T23:14:37.657 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 23:14:37 smithi203 podman[74068]: 2024-03-23 23:14:37.425668535 +0000 UTC m=+0.558797552 container cleanup b6f6dfb7120a8d84226d0f3fa7ff5580ebb692d74e1d9baffa969de3b8028be2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-23T23:14:37.657 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 23 23:14:37 smithi203 bash[74068]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-mgr-z 2024-03-23T23:14:37.891 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mgr.z.service' 2024-03-23T23:14:37.931 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:14:37.931 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-23T23:14:37.931 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-23T23:14:37.931 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.0 2024-03-23T23:14:38.304 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 23 23:14:38 smithi012 systemd[1]: Stopping Ceph osd.0 for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:14:38.803 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 23 23:14:38 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-0[46564]: 2024-03-23T23:14:38.498+0000 7fc80ec00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:14:38.804 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 23 23:14:38 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-0[46564]: 2024-03-23T23:14:38.498+0000 7fc80ec00700 -1 osd.0 91 *** Got signal Terminated *** 2024-03-23T23:14:38.804 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 23 23:14:38 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-0[46564]: 2024-03-23T23:14:38.498+0000 7fc80ec00700 -1 osd.0 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-23T23:14:44.304 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 23 23:14:44 smithi012 podman[100714]: 2024-03-23 23:14:44.008759111 +0000 UTC m=+5.651049109 container died 72a7dbc0a9b3b535389cf7f8534e16f12d27186453a4e01277b3d8cde878c7fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-23T23:14:44.752 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 23 23:14:44 smithi012 podman[100714]: 2024-03-23 23:14:44.501086685 +0000 UTC m=+6.143376676 container cleanup 72a7dbc0a9b3b535389cf7f8534e16f12d27186453a4e01277b3d8cde878c7fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-23T23:14:44.752 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 23 23:14:44 smithi012 bash[100714]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-0 2024-03-23T23:14:45.053 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 23 23:14:44 smithi012 podman[100735]: 2024-03-23 23:14:44.752532865 +0000 UTC m=+0.721909246 container remove 72a7dbc0a9b3b535389cf7f8534e16f12d27186453a4e01277b3d8cde878c7fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-03-23T23:14:45.973 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 23 23:14:45 smithi012 podman[100824]: 2024-03-23 23:14:45.532463475 +0000 UTC m=+0.040591011 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-23T23:14:45.973 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 23 23:14:45 smithi012 podman[100824]: 2024-03-23 23:14:45.661773737 +0000 UTC m=+0.169901279 container create 36016e22e238a4976e71b1d87f515bfa47ce1cdb8d2bc252dca7099ea6359b07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-23T23:14:45.974 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 23 23:14:45 smithi012 podman[100824]: 2024-03-23 23:14:45.974051713 +0000 UTC m=+0.482179261 container init 36016e22e238a4976e71b1d87f515bfa47ce1cdb8d2bc252dca7099ea6359b07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True) 2024-03-23T23:14:46.303 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 23 23:14:45 smithi012 podman[100824]: 2024-03-23 23:14:45.985871988 +0000 UTC m=+0.493999499 container start 36016e22e238a4976e71b1d87f515bfa47ce1cdb8d2bc252dca7099ea6359b07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-23T23:14:46.304 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 23 23:14:46 smithi012 podman[100824]: 2024-03-23 23:14:46.098154291 +0000 UTC m=+0.606281815 container attach 36016e22e238a4976e71b1d87f515bfa47ce1cdb8d2bc252dca7099ea6359b07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-0-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-23T23:14:46.799 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 23 23:14:46 smithi012 podman[100824]: 2024-03-23 23:14:46.43860698 +0000 UTC m=+0.946734556 container died 36016e22e238a4976e71b1d87f515bfa47ce1cdb8d2bc252dca7099ea6359b07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-23T23:14:47.181 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.0.service' 2024-03-23T23:14:47.311 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:14:47.311 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-23T23:14:47.311 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-23T23:14:47.311 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.1 2024-03-23T23:14:47.617 INFO:journalctl@ceph.osd.1.smithi173.stdout:Mar 23 23:14:47 smithi173 systemd[1]: Stopping Ceph osd.1 for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:14:48.118 INFO:journalctl@ceph.osd.1.smithi173.stdout:Mar 23 23:14:47 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-1[40706]: 2024-03-23T23:14:47.626+0000 7fcb4d4cb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:14:48.118 INFO:journalctl@ceph.osd.1.smithi173.stdout:Mar 23 23:14:47 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-1[40706]: 2024-03-23T23:14:47.626+0000 7fcb4d4cb700 -1 osd.1 91 *** Got signal Terminated *** 2024-03-23T23:14:48.118 INFO:journalctl@ceph.osd.1.smithi173.stdout:Mar 23 23:14:47 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-1[40706]: 2024-03-23T23:14:47.626+0000 7fcb4d4cb700 -1 osd.1 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-23T23:14:53.427 INFO:journalctl@ceph.osd.1.smithi173.stdout:Mar 23 23:14:53 smithi173 podman[71405]: 2024-03-23 23:14:53.126897272 +0000 UTC m=+5.621958395 container died 8ab51716323e58dffbaf720ac1872df3183c77517a610ff220ce27bea7e877a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-1, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-03-23T23:14:53.761 INFO:journalctl@ceph.osd.1.smithi173.stdout:Mar 23 23:14:53 smithi173 podman[71405]: 2024-03-23 23:14:53.552950353 +0000 UTC m=+6.048011512 container cleanup 8ab51716323e58dffbaf720ac1872df3183c77517a610ff220ce27bea7e877a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-1, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-23T23:14:53.761 INFO:journalctl@ceph.osd.1.smithi173.stdout:Mar 23 23:14:53 smithi173 bash[71405]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-1 2024-03-23T23:14:54.118 INFO:journalctl@ceph.osd.1.smithi173.stdout:Mar 23 23:14:53 smithi173 podman[71422]: 2024-03-23 23:14:53.762865766 +0000 UTC m=+0.629748278 container remove 8ab51716323e58dffbaf720ac1872df3183c77517a610ff220ce27bea7e877a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-1, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-23T23:14:54.457 INFO:journalctl@ceph.osd.1.smithi173.stdout:Mar 23 23:14:54 smithi173 podman[71514]: 2024-03-23 23:14:54.05787444 +0000 UTC m=+0.018000828 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-23T23:14:54.457 INFO:journalctl@ceph.osd.1.smithi173.stdout:Mar 23 23:14:54 smithi173 podman[71514]: 2024-03-23 23:14:54.18052977 +0000 UTC m=+0.140656134 container create 53c1cf6582159bc9bb0f563c6d736ff7a5829f25cf72502e6551e961600cca37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-23T23:14:54.748 INFO:journalctl@ceph.osd.1.smithi173.stdout:Mar 23 23:14:54 smithi173 podman[71514]: 2024-03-23 23:14:54.458492973 +0000 UTC m=+0.418619361 container init 53c1cf6582159bc9bb0f563c6d736ff7a5829f25cf72502e6551e961600cca37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-23T23:14:54.749 INFO:journalctl@ceph.osd.1.smithi173.stdout:Mar 23 23:14:54 smithi173 podman[71514]: 2024-03-23 23:14:54.461828182 +0000 UTC m=+0.421954556 container start 53c1cf6582159bc9bb0f563c6d736ff7a5829f25cf72502e6551e961600cca37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2) 2024-03-23T23:14:54.749 INFO:journalctl@ceph.osd.1.smithi173.stdout:Mar 23 23:14:54 smithi173 podman[71514]: 2024-03-23 23:14:54.550529389 +0000 UTC m=+0.510655778 container attach 53c1cf6582159bc9bb0f563c6d736ff7a5829f25cf72502e6551e961600cca37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-23T23:14:55.069 INFO:journalctl@ceph.osd.1.smithi173.stdout:Mar 23 23:14:54 smithi173 podman[71514]: 2024-03-23 23:14:54.749710653 +0000 UTC m=+0.709837033 container died 53c1cf6582159bc9bb0f563c6d736ff7a5829f25cf72502e6551e961600cca37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-23T23:14:55.383 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.1.service' 2024-03-23T23:14:55.426 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:14:55.427 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-23T23:14:55.427 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-23T23:14:55.427 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.2 2024-03-23T23:14:55.749 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 23 23:14:55 smithi203 systemd[1]: Stopping Ceph osd.2 for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:14:56.157 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 23 23:14:55 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-2[41863]: 2024-03-23T23:14:55.749+0000 7fafa57d4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:14:56.157 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 23 23:14:55 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-2[41863]: 2024-03-23T23:14:55.749+0000 7fafa57d4700 -1 osd.2 91 *** Got signal Terminated *** 2024-03-23T23:14:56.157 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 23 23:14:55 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-2[41863]: 2024-03-23T23:14:55.749+0000 7fafa57d4700 -1 osd.2 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-23T23:15:01.499 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 23 23:15:01 smithi203 podman[74205]: 2024-03-23 23:15:01.190860135 +0000 UTC m=+5.561362989 container died a4ee56e761e96400d084ec41b86584329ec27fbff95adc764949dad4577db52c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-23T23:15:01.867 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 23 23:15:01 smithi203 podman[74205]: 2024-03-23 23:15:01.591596752 +0000 UTC m=+5.962099606 container cleanup a4ee56e761e96400d084ec41b86584329ec27fbff95adc764949dad4577db52c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-23T23:15:01.867 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 23 23:15:01 smithi203 bash[74205]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-2 2024-03-23T23:15:02.157 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 23 23:15:01 smithi203 podman[74223]: 2024-03-23 23:15:01.868353648 +0000 UTC m=+0.671550749 container remove a4ee56e761e96400d084ec41b86584329ec27fbff95adc764949dad4577db52c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-23T23:15:02.446 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 23 23:15:02 smithi203 podman[74316]: 2024-03-23 23:15:02.071097105 +0000 UTC m=+0.017581780 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-23T23:15:02.446 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 23 23:15:02 smithi203 podman[74316]: 2024-03-23 23:15:02.194329409 +0000 UTC m=+0.140814082 container create fa17a367706ed41032e6e09bcea7c2bf6ab9497b5a7f79e215e7c0452e764ec1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5) 2024-03-23T23:15:02.736 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 23 23:15:02 smithi203 podman[74316]: 2024-03-23 23:15:02.447778998 +0000 UTC m=+0.394263671 container init fa17a367706ed41032e6e09bcea7c2bf6ab9497b5a7f79e215e7c0452e764ec1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-23T23:15:02.736 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 23 23:15:02 smithi203 podman[74316]: 2024-03-23 23:15:02.451053845 +0000 UTC m=+0.397538519 container start fa17a367706ed41032e6e09bcea7c2bf6ab9497b5a7f79e215e7c0452e764ec1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True) 2024-03-23T23:15:02.736 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 23 23:15:02 smithi203 podman[74316]: 2024-03-23 23:15:02.564752749 +0000 UTC m=+0.511237433 container attach fa17a367706ed41032e6e09bcea7c2bf6ab9497b5a7f79e215e7c0452e764ec1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-2-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-23T23:15:03.034 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 23 23:15:02 smithi203 podman[74316]: 2024-03-23 23:15:02.737803177 +0000 UTC m=+0.684287861 container died fa17a367706ed41032e6e09bcea7c2bf6ab9497b5a7f79e215e7c0452e764ec1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.vendor=CentOS) 2024-03-23T23:15:03.324 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.2.service' 2024-03-23T23:15:03.370 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:15:03.370 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-23T23:15:03.370 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-23T23:15:03.371 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.3 2024-03-23T23:15:03.805 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 23 23:15:03 smithi012 systemd[1]: Stopping Ceph osd.3 for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:15:04.304 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 23 23:15:04 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-3[53117]: 2024-03-23T23:15:04.026+0000 7fb76b200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:15:04.304 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 23 23:15:04 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-3[53117]: 2024-03-23T23:15:04.026+0000 7fb76b200700 -1 osd.3 91 *** Got signal Terminated *** 2024-03-23T23:15:04.304 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 23 23:15:04 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-3[53117]: 2024-03-23T23:15:04.026+0000 7fb76b200700 -1 osd.3 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-23T23:15:05.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:04 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:04.925+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:05.804 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 23 23:15:05 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-3[53117]: 2024-03-23T23:15:05.466+0000 7fb766a00700 -1 osd.3 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.206750+0000 front 2024-03-23T23:14:39.207022+0000 (oldest deadline 2024-03-23T23:15:04.506546+0000) 2024-03-23T23:15:05.804 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:05 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:05.692+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:05.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:05 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:05.509+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.152755+0000 front 2024-03-23T23:14:40.153115+0000 (oldest deadline 2024-03-23T23:15:05.452528+0000) 2024-03-23T23:15:06.367 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:05 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:05.917+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:06.803 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 23 23:15:06 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-3[53117]: 2024-03-23T23:15:06.496+0000 7fb766a00700 -1 osd.3 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.206750+0000 front 2024-03-23T23:14:39.207022+0000 (oldest deadline 2024-03-23T23:15:04.506546+0000) 2024-03-23T23:15:06.804 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:06 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:06.673+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:06.808 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:06 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:06.540+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.152755+0000 front 2024-03-23T23:14:40.153115+0000 (oldest deadline 2024-03-23T23:15:05.452528+0000) 2024-03-23T23:15:07.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:06 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:06.870+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:07.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:07 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:07.048+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:07.804 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:07 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:07.738+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:07.804 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 23 23:15:07 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-3[53117]: 2024-03-23T23:15:07.511+0000 7fb766a00700 -1 osd.3 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.206750+0000 front 2024-03-23T23:14:39.207022+0000 (oldest deadline 2024-03-23T23:15:04.506546+0000) 2024-03-23T23:15:07.804 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:07 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:07.701+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:07.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:07 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:07.581+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.152755+0000 front 2024-03-23T23:14:40.153115+0000 (oldest deadline 2024-03-23T23:15:05.452528+0000) 2024-03-23T23:15:08.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:07 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:07.859+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:08.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:08 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:08.018+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:08.803 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 23 23:15:08 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-3[53117]: 2024-03-23T23:15:08.508+0000 7fb766a00700 -1 osd.3 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.206750+0000 front 2024-03-23T23:14:39.207022+0000 (oldest deadline 2024-03-23T23:15:04.506546+0000) 2024-03-23T23:15:08.804 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:08 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:08.703+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:08.804 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:08 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:08.705+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:08.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:08 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:08.628+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.152755+0000 front 2024-03-23T23:14:40.153115+0000 (oldest deadline 2024-03-23T23:15:05.452528+0000) 2024-03-23T23:15:08.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:08 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:08.477+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:09.117 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:08 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:08.863+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:09.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:09 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:09.049+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:09.742 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:09 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:09.716+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:09.742 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 23 23:15:09 smithi012 podman[100929]: 2024-03-23 23:15:09.471998871 +0000 UTC m=+5.609272224 container died 2badf254fb62ee2b1f90a3f60a16e73d40788f91ee071e20e6b263d97635aa55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-3, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-23T23:15:09.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:09 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:09.641+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.152755+0000 front 2024-03-23T23:14:40.153115+0000 (oldest deadline 2024-03-23T23:15:05.452528+0000) 2024-03-23T23:15:09.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:09 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:09.516+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:10.053 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:09 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:09.742+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:10.054 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 23 23:15:09 smithi012 podman[100929]: 2024-03-23 23:15:09.917180375 +0000 UTC m=+6.054453747 container cleanup 2badf254fb62ee2b1f90a3f60a16e73d40788f91ee071e20e6b263d97635aa55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-3, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-23T23:15:10.054 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 23 23:15:09 smithi012 bash[100929]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-3 2024-03-23T23:15:10.117 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:09 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:09.817+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:10.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:10 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:10.049+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:10.553 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 23 23:15:10 smithi012 podman[100961]: 2024-03-23 23:15:10.227148482 +0000 UTC m=+0.744948892 container remove 2badf254fb62ee2b1f90a3f60a16e73d40788f91ee071e20e6b263d97635aa55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-3, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS) 2024-03-23T23:15:10.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:10 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:10.527+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:10.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:10 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:10.687+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.152755+0000 front 2024-03-23T23:14:40.153115+0000 (oldest deadline 2024-03-23T23:15:05.452528+0000) 2024-03-23T23:15:11.053 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:10 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:10.689+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:11.053 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:10 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:10.790+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:11.117 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:10 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:10.848+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:11.368 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 23 23:15:11 smithi012 podman[101050]: 2024-03-23 23:15:11.045517246 +0000 UTC m=+0.034419965 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-23T23:15:11.368 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 23 23:15:11 smithi012 podman[101050]: 2024-03-23 23:15:11.169870363 +0000 UTC m=+0.158773116 container create d5b7f2ae1ba79df838a5215e659faddb66d088ee3dfe86790150a1c773ebab0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-03-23T23:15:11.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:11 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:11.088+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:11.668 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 23 23:15:11 smithi012 podman[101050]: 2024-03-23 23:15:11.491078213 +0000 UTC m=+0.479980970 container init d5b7f2ae1ba79df838a5215e659faddb66d088ee3dfe86790150a1c773ebab0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-23T23:15:11.668 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 23 23:15:11 smithi012 podman[101050]: 2024-03-23 23:15:11.504773563 +0000 UTC m=+0.493676319 container start d5b7f2ae1ba79df838a5215e659faddb66d088ee3dfe86790150a1c773ebab0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-23T23:15:11.668 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 23 23:15:11 smithi012 podman[101050]: 2024-03-23 23:15:11.597388697 +0000 UTC m=+0.586291454 container attach d5b7f2ae1ba79df838a5215e659faddb66d088ee3dfe86790150a1c773ebab0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-3-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD) 2024-03-23T23:15:11.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:11 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:11.688+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.152755+0000 front 2024-03-23T23:14:40.153115+0000 (oldest deadline 2024-03-23T23:15:05.452528+0000) 2024-03-23T23:15:11.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:11 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:11.575+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:11.955 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:11 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:11.668+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:11.955 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:11 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:11.810+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:12.303 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 23 23:15:11 smithi012 podman[101050]: 2024-03-23 23:15:11.956244759 +0000 UTC m=+0.945147489 container died d5b7f2ae1ba79df838a5215e659faddb66d088ee3dfe86790150a1c773ebab0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-23T23:15:12.367 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:11 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:11.880+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:12.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:12 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:12.101+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:12.745 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.3.service' 2024-03-23T23:15:12.876 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:15:12.876 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-23T23:15:12.876 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-23T23:15:12.876 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.4 2024-03-23T23:15:12.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:12 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:12.557+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:12.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:12 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:12.731+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.152755+0000 front 2024-03-23T23:14:40.153115+0000 (oldest deadline 2024-03-23T23:15:05.452528+0000) 2024-03-23T23:15:13.054 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:12 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:12.656+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:13.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:12 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:12.769+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:13.117 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 23 23:15:12 smithi173 systemd[1]: Stopping Ceph osd.4 for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:15:13.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:12 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:12.836+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:13.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:13 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:13.086+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:13.618 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 23 23:15:13 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-4[47541]: 2024-03-23T23:15:13.198+0000 7f3fd17ea700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:15:13.618 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 23 23:15:13 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-4[47541]: 2024-03-23T23:15:13.198+0000 7f3fd17ea700 -1 osd.4 91 *** Got signal Terminated *** 2024-03-23T23:15:13.618 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 23 23:15:13 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-4[47541]: 2024-03-23T23:15:13.198+0000 7f3fd17ea700 -1 osd.4 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-23T23:15:13.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:13 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:13.758+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.152755+0000 front 2024-03-23T23:14:40.153115+0000 (oldest deadline 2024-03-23T23:15:05.452528+0000) 2024-03-23T23:15:13.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:13 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:13.556+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:14.053 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:13 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:13.642+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:14.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:13 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:13.747+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:14.117 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:13 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:13.850+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:14.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:14 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:14.076+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:14.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:14 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:14.724+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.152755+0000 front 2024-03-23T23:14:40.153115+0000 (oldest deadline 2024-03-23T23:15:05.452528+0000) 2024-03-23T23:15:14.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:14 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:14.558+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:15.053 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:14 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:14.679+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:15.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:14 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:14.788+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:15.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:14 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:14.816+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:15.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:14 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:14.816+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:15.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:15 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:15.120+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:15.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:15 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:15.554+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:15.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:15 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:15.719+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.152755+0000 front 2024-03-23T23:14:40.153115+0000 (oldest deadline 2024-03-23T23:15:05.452528+0000) 2024-03-23T23:15:15.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:15 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:15.719+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.354457+0000 front 2024-03-23T23:14:52.354514+0000 (oldest deadline 2024-03-23T23:15:15.254661+0000) 2024-03-23T23:15:16.053 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:15 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:15.710+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:16.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:15 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:15.748+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:16.118 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 23 23:15:15 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-4[47541]: 2024-03-23T23:15:15.903+0000 7f3fc9dd6700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:49.880793+0000 front 2024-03-23T23:14:49.880772+0000 (oldest deadline 2024-03-23T23:15:15.180931+0000) 2024-03-23T23:15:16.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:15 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:15.847+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:16.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:15 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:15.847+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:16.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:16 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:16.117+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:16.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:16 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:16.537+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:16.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:16 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:16.677+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.152755+0000 front 2024-03-23T23:14:40.153115+0000 (oldest deadline 2024-03-23T23:15:05.452528+0000) 2024-03-23T23:15:16.908 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:16 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:16.677+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.354457+0000 front 2024-03-23T23:14:52.354514+0000 (oldest deadline 2024-03-23T23:15:15.254661+0000) 2024-03-23T23:15:17.053 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:16 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:16.686+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:17.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:16 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:16.707+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:17.368 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 23 23:15:16 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-4[47541]: 2024-03-23T23:15:16.904+0000 7f3fc9dd6700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:49.880793+0000 front 2024-03-23T23:14:49.880772+0000 (oldest deadline 2024-03-23T23:15:15.180931+0000) 2024-03-23T23:15:17.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:16 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:16.875+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:17.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:16 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:16.875+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:17.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:17 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:17.068+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:17.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:17 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:17.554+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:17.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:17 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:17.683+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.152755+0000 front 2024-03-23T23:14:40.153115+0000 (oldest deadline 2024-03-23T23:15:05.452528+0000) 2024-03-23T23:15:17.908 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:17 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:17.683+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.354457+0000 front 2024-03-23T23:14:52.354514+0000 (oldest deadline 2024-03-23T23:15:15.254661+0000) 2024-03-23T23:15:18.053 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:17 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:17.709+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:18.053 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:17 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:17.709+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.824259+0000 front 2024-03-23T23:14:50.824337+0000 (oldest deadline 2024-03-23T23:15:16.724560+0000) 2024-03-23T23:15:18.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:17 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:17.690+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:18.117 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 23 23:15:17 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-4[47541]: 2024-03-23T23:15:17.899+0000 7f3fc9dd6700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:49.880793+0000 front 2024-03-23T23:14:49.880772+0000 (oldest deadline 2024-03-23T23:15:15.180931+0000) 2024-03-23T23:15:18.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:17 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:17.855+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:18.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:17 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:17.855+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:18.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:18 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:18.034+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:18.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:18 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:18.034+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:18.899 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 23 23:15:18 smithi173 podman[71631]: 2024-03-23 23:15:18.634796611 +0000 UTC m=+5.553615970 container died a1232df2ca07e3b39bae18c44d22cb7114d27788b7aabcdb6284f619f0da8414 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-4, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-23T23:15:18.899 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:18 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:18.881+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:18.899 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:18 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:18.881+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:18.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:18 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:18.587+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:18.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:18 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:18.698+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.152755+0000 front 2024-03-23T23:14:40.153115+0000 (oldest deadline 2024-03-23T23:15:05.452528+0000) 2024-03-23T23:15:18.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:18 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:18.698+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.354457+0000 front 2024-03-23T23:14:52.354514+0000 (oldest deadline 2024-03-23T23:15:15.254661+0000) 2024-03-23T23:15:19.053 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:18 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:18.733+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:19.053 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:18 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:18.733+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.824259+0000 front 2024-03-23T23:14:50.824337+0000 (oldest deadline 2024-03-23T23:15:16.724560+0000) 2024-03-23T23:15:19.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:18 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:18.715+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:19.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:18 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:18.715+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:19.266 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:18 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:18.988+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.485743+0000 front 2024-03-23T23:14:52.485777+0000 (oldest deadline 2024-03-23T23:15:18.286503+0000) 2024-03-23T23:15:19.266 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 23 23:15:18 smithi173 podman[71631]: 2024-03-23 23:15:18.999327051 +0000 UTC m=+5.918146401 container cleanup a1232df2ca07e3b39bae18c44d22cb7114d27788b7aabcdb6284f619f0da8414 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-4, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-23T23:15:19.266 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 23 23:15:18 smithi173 bash[71631]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-4 2024-03-23T23:15:19.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:19 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:19.017+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:19.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:19 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:19.017+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:19.615 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 23 23:15:19 smithi173 podman[71648]: 2024-03-23 23:15:19.267214823 +0000 UTC m=+0.623579103 container remove a1232df2ca07e3b39bae18c44d22cb7114d27788b7aabcdb6284f619f0da8414 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-4, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-23T23:15:19.867 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 23 23:15:19 smithi173 podman[71736]: 2024-03-23 23:15:19.519511004 +0000 UTC m=+0.017940208 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-23T23:15:19.868 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 23 23:15:19 smithi173 podman[71736]: 2024-03-23 23:15:19.6358908 +0000 UTC m=+0.134319990 container create c2c2d8b50549203f63af4ba8330a946e177a595e3f56af2c7149b9a8f1fcd873 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-23T23:15:19.868 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 23 23:15:19 smithi173 podman[71736]: 2024-03-23 23:15:19.854996322 +0000 UTC m=+0.353425521 container init c2c2d8b50549203f63af4ba8330a946e177a595e3f56af2c7149b9a8f1fcd873 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-23T23:15:19.868 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 23 23:15:19 smithi173 podman[71736]: 2024-03-23 23:15:19.858086744 +0000 UTC m=+0.356515934 container start c2c2d8b50549203f63af4ba8330a946e177a595e3f56af2c7149b9a8f1fcd873 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-23T23:15:19.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:19 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:19.552+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:19.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:19 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:19.739+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.152755+0000 front 2024-03-23T23:14:40.153115+0000 (oldest deadline 2024-03-23T23:15:05.452528+0000) 2024-03-23T23:15:19.908 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:19 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:19.739+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.354457+0000 front 2024-03-23T23:14:52.354514+0000 (oldest deadline 2024-03-23T23:15:15.254661+0000) 2024-03-23T23:15:20.053 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:19 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:19.781+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:20.054 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:19 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:19.781+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.824259+0000 front 2024-03-23T23:14:50.824337+0000 (oldest deadline 2024-03-23T23:15:16.724560+0000) 2024-03-23T23:15:20.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:19 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:19.685+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:20.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:19 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:19.685+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:20.141 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 23 23:15:19 smithi173 podman[71736]: 2024-03-23 23:15:19.954431858 +0000 UTC m=+0.452861060 container attach c2c2d8b50549203f63af4ba8330a946e177a595e3f56af2c7149b9a8f1fcd873 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-23T23:15:20.142 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:19 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:19.953+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.485743+0000 front 2024-03-23T23:14:52.485777+0000 (oldest deadline 2024-03-23T23:15:18.286503+0000) 2024-03-23T23:15:20.142 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:19 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:19.874+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:20.142 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:19 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:19.874+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:20.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:19 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:19.994+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:20.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:19 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:19.994+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:20.457 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 23 23:15:20 smithi173 podman[71736]: 2024-03-23 23:15:20.143293601 +0000 UTC m=+0.641722809 container died c2c2d8b50549203f63af4ba8330a946e177a595e3f56af2c7149b9a8f1fcd873 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-23T23:15:20.695 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.4.service' 2024-03-23T23:15:20.721 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 23 23:15:20 smithi173 podman[71762]: 2024-03-23 23:15:20.666957522 +0000 UTC m=+0.514865167 container remove c2c2d8b50549203f63af4ba8330a946e177a595e3f56af2c7149b9a8f1fcd873 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-23T23:15:20.721 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 23 23:15:20 smithi173 systemd[1]: ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.4.service: Deactivated successfully. 2024-03-23T23:15:20.721 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 23 23:15:20 smithi173 systemd[1]: Stopped Ceph osd.4 for 90f679ae-e967-11ee-95cd-87774f69a715. 2024-03-23T23:15:20.721 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 23 23:15:20 smithi173 systemd[1]: ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.4.service: Consumed 9.557s CPU time. 2024-03-23T23:15:20.737 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:15:20.737 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-23T23:15:20.737 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-23T23:15:20.737 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.5 2024-03-23T23:15:20.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:20 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:20.567+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:20.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:20 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:20.696+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.152755+0000 front 2024-03-23T23:14:40.153115+0000 (oldest deadline 2024-03-23T23:15:05.452528+0000) 2024-03-23T23:15:20.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:20 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:20.696+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.354457+0000 front 2024-03-23T23:14:52.354514+0000 (oldest deadline 2024-03-23T23:15:15.254661+0000) 2024-03-23T23:15:20.908 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:20 smithi203 systemd[1]: Stopping Ceph osd.5 for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:15:21.053 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:20 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:20.744+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:21.053 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:20 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:20.744+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.824259+0000 front 2024-03-23T23:14:50.824337+0000 (oldest deadline 2024-03-23T23:15:16.724560+0000) 2024-03-23T23:15:21.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:20 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:20.732+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:21.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:20 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:20.732+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:21.117 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:20 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:20.946+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.485743+0000 front 2024-03-23T23:14:52.485777+0000 (oldest deadline 2024-03-23T23:15:18.286503+0000) 2024-03-23T23:15:21.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:20 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:20.838+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:21.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:20 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:20.838+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:21.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:21 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:21.008+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:21.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:21 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:21.008+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:21.408 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:21 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:21.213+0000 7f09bebf0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:15:21.408 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:21 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:21.213+0000 7f09bebf0700 -1 osd.5 91 *** Got signal Terminated *** 2024-03-23T23:15:21.408 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:21 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:21.213+0000 7f09bebf0700 -1 osd.5 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-23T23:15:21.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:21 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:21.574+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:21.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:21 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:21.647+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.152755+0000 front 2024-03-23T23:14:40.153115+0000 (oldest deadline 2024-03-23T23:15:05.452528+0000) 2024-03-23T23:15:21.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:21 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:21.647+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.354457+0000 front 2024-03-23T23:14:52.354514+0000 (oldest deadline 2024-03-23T23:15:15.254661+0000) 2024-03-23T23:15:22.053 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:21 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:21.744+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:22.054 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:21 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:21.744+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.824259+0000 front 2024-03-23T23:14:50.824337+0000 (oldest deadline 2024-03-23T23:15:16.724560+0000) 2024-03-23T23:15:22.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:21 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:21.768+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:22.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:21 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:21.768+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:22.117 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:21 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:21.908+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.485743+0000 front 2024-03-23T23:14:52.485777+0000 (oldest deadline 2024-03-23T23:15:18.286503+0000) 2024-03-23T23:15:22.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:21 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:21.833+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:22.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:21 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:21.833+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:22.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:21 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:21.970+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:22.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:21 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:21.970+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:22.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:22 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:22.535+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:22.908 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:22 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:22.677+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.152755+0000 front 2024-03-23T23:14:40.153115+0000 (oldest deadline 2024-03-23T23:15:05.452528+0000) 2024-03-23T23:15:22.908 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:22 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:22.677+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.354457+0000 front 2024-03-23T23:14:52.354514+0000 (oldest deadline 2024-03-23T23:15:15.254661+0000) 2024-03-23T23:15:23.054 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:22 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:22.736+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:23.054 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:22 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:22.736+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.824259+0000 front 2024-03-23T23:14:50.824337+0000 (oldest deadline 2024-03-23T23:15:16.724560+0000) 2024-03-23T23:15:23.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:22 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:22.791+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:23.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:22 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:22.791+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:23.117 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:22 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:22.860+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.485743+0000 front 2024-03-23T23:14:52.485777+0000 (oldest deadline 2024-03-23T23:15:18.286503+0000) 2024-03-23T23:15:23.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:22 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:22.838+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:23.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:22 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:22.838+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:23.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:22 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:22.941+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:23.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:22 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:22.941+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:23.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:23 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:23.708+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.152755+0000 front 2024-03-23T23:14:40.153115+0000 (oldest deadline 2024-03-23T23:15:05.452528+0000) 2024-03-23T23:15:23.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:23 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:23.708+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.354457+0000 front 2024-03-23T23:14:52.354514+0000 (oldest deadline 2024-03-23T23:15:15.254661+0000) 2024-03-23T23:15:23.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:23 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:23.563+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:23.908 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:23 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:23.563+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:24.053 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:23 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:23.743+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:24.053 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:23 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:23.743+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.824259+0000 front 2024-03-23T23:14:50.824337+0000 (oldest deadline 2024-03-23T23:15:16.724560+0000) 2024-03-23T23:15:24.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:23 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:23.808+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:24.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:23 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:23.808+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:24.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:23 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:23.847+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.485743+0000 front 2024-03-23T23:14:52.485777+0000 (oldest deadline 2024-03-23T23:15:18.286503+0000) 2024-03-23T23:15:24.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:23 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:23.790+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:24.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:23 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:23.790+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:24.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:23 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:23.790+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:24.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:23 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:23.942+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:24.408 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:23 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:23.942+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:24.408 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:23 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:23.942+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:24.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:24 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:24.605+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:24.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:24 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:24.605+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:24.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:24 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:24.660+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.152755+0000 front 2024-03-23T23:14:40.153115+0000 (oldest deadline 2024-03-23T23:15:05.452528+0000) 2024-03-23T23:15:24.908 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:24 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:24.660+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.354457+0000 front 2024-03-23T23:14:52.354514+0000 (oldest deadline 2024-03-23T23:15:15.254661+0000) 2024-03-23T23:15:24.908 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:24 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:24.660+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:59.855987+0000 front 2024-03-23T23:14:59.856006+0000 (oldest deadline 2024-03-23T23:15:24.556157+0000) 2024-03-23T23:15:25.053 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:24 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:24.781+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:25.053 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:24 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:24.781+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.824259+0000 front 2024-03-23T23:14:50.824337+0000 (oldest deadline 2024-03-23T23:15:16.724560+0000) 2024-03-23T23:15:25.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:24 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:24.812+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:25.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:24 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:24.812+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:25.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:24 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:24.812+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:25.117 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:24 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:24.834+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.485743+0000 front 2024-03-23T23:14:52.485777+0000 (oldest deadline 2024-03-23T23:15:18.286503+0000) 2024-03-23T23:15:25.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:24 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:24.811+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:25.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:24 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:24.811+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:25.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:24 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:24.811+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:25.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:24 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:24.973+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:25.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:24 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:24.973+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:25.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:24 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:24.973+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:25.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:25 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:25.566+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:25.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:25 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:25.566+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:25.908 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:25 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:25.637+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.152755+0000 front 2024-03-23T23:14:40.153115+0000 (oldest deadline 2024-03-23T23:15:05.452528+0000) 2024-03-23T23:15:25.908 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:25 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:25.637+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.354457+0000 front 2024-03-23T23:14:52.354514+0000 (oldest deadline 2024-03-23T23:15:15.254661+0000) 2024-03-23T23:15:25.908 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:25 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5[48851]: 2024-03-23T23:15:25.637+0000 7f09b71dc700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:59.855987+0000 front 2024-03-23T23:14:59.856006+0000 (oldest deadline 2024-03-23T23:15:24.556157+0000) 2024-03-23T23:15:26.053 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:25 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:25.770+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:26.054 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:25 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:25.770+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.824259+0000 front 2024-03-23T23:14:50.824337+0000 (oldest deadline 2024-03-23T23:15:16.724560+0000) 2024-03-23T23:15:26.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:25 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:25.834+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:26.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:25 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:25.834+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:26.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:25 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:25.834+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:26.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:25 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:25.839+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.485743+0000 front 2024-03-23T23:14:52.485777+0000 (oldest deadline 2024-03-23T23:15:18.286503+0000) 2024-03-23T23:15:26.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:25 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:25.779+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:26.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:25 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:25.779+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:26.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:25 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:25.779+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:26.408 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:26 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:26.012+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:26.654 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:26 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:26.012+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:26.654 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:26 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:26.012+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:26.898 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:26 smithi203 podman[74434]: 2024-03-23 23:15:26.62668628 +0000 UTC m=+5.696191287 container died 24fbc3d1ce1e1181bcc7e3e26c2fceb6b18c88d328fac509c1f2630dad62a000 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True) 2024-03-23T23:15:26.899 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:26 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:26.604+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:26.899 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:26 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:26.604+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:27.053 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:26 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:26.747+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:27.054 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:26 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:26.747+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.824259+0000 front 2024-03-23T23:14:50.824337+0000 (oldest deadline 2024-03-23T23:15:16.724560+0000) 2024-03-23T23:15:27.054 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:26 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:26.747+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:15:00.728957+0000 front 2024-03-23T23:15:00.728783+0000 (oldest deadline 2024-03-23T23:15:26.629053+0000) 2024-03-23T23:15:27.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:26 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:26.864+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:27.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:26 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:26.864+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:27.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:26 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:26.864+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:27.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:26 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:26.865+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.485743+0000 front 2024-03-23T23:14:52.485777+0000 (oldest deadline 2024-03-23T23:15:18.286503+0000) 2024-03-23T23:15:27.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:26 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:26.801+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:27.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:26 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:26.801+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:27.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:26 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:26.801+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:27.157 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:26 smithi203 podman[74434]: 2024-03-23 23:15:26.988656465 +0000 UTC m=+6.058161455 container cleanup 24fbc3d1ce1e1181bcc7e3e26c2fceb6b18c88d328fac509c1f2630dad62a000 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-23T23:15:27.157 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:26 smithi203 bash[74434]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5 2024-03-23T23:15:27.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:26 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:26.979+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:27.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:26 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:26.979+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:27.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:26 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:26.979+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:27.577 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:27 smithi203 podman[74453]: 2024-03-23 23:15:27.290230982 +0000 UTC m=+0.660333775 container remove 24fbc3d1ce1e1181bcc7e3e26c2fceb6b18c88d328fac509c1f2630dad62a000 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-23T23:15:27.868 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:27 smithi203 podman[74545]: 2024-03-23 23:15:27.482296155 +0000 UTC m=+0.017884135 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-23T23:15:27.868 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:27 smithi203 podman[74545]: 2024-03-23 23:15:27.617799778 +0000 UTC m=+0.153387753 container create b2d1b79c8820ac627fd0cb26c6d9772cae02eacf3a82de0ca36bf58705c12624 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-23T23:15:27.869 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:27 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:27.648+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:27.869 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:27 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:27.648+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:28.053 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:27 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:27.731+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:28.053 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:27 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:27.731+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.824259+0000 front 2024-03-23T23:14:50.824337+0000 (oldest deadline 2024-03-23T23:15:16.724560+0000) 2024-03-23T23:15:28.054 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:27 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:27.731+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:15:00.728957+0000 front 2024-03-23T23:15:00.728783+0000 (oldest deadline 2024-03-23T23:15:26.629053+0000) 2024-03-23T23:15:28.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:27 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:27.867+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:28.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:27 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:27.867+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:28.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:27 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:27.867+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:28.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:27 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:27.845+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.485743+0000 front 2024-03-23T23:14:52.485777+0000 (oldest deadline 2024-03-23T23:15:18.286503+0000) 2024-03-23T23:15:28.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:27 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:27.845+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:15:00.587549+0000 front 2024-03-23T23:15:00.587647+0000 (oldest deadline 2024-03-23T23:15:26.987999+0000) 2024-03-23T23:15:28.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:27 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:27.845+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:28.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:27 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:27.845+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:28.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:27 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:27.845+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:28.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:27 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:27.968+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:28.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:27 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:27.968+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:28.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:27 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:27.968+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:28.158 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:27 smithi203 podman[74545]: 2024-03-23 23:15:27.869422162 +0000 UTC m=+0.405010139 container init b2d1b79c8820ac627fd0cb26c6d9772cae02eacf3a82de0ca36bf58705c12624 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-23T23:15:28.158 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:27 smithi203 podman[74545]: 2024-03-23 23:15:27.872744351 +0000 UTC m=+0.408332329 container start b2d1b79c8820ac627fd0cb26c6d9772cae02eacf3a82de0ca36bf58705c12624 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-23T23:15:28.158 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:27 smithi203 podman[74545]: 2024-03-23 23:15:27.969644397 +0000 UTC m=+0.505232375 container attach b2d1b79c8820ac627fd0cb26c6d9772cae02eacf3a82de0ca36bf58705c12624 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-23T23:15:28.522 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 23 23:15:28 smithi203 podman[74545]: 2024-03-23 23:15:28.16425139 +0000 UTC m=+0.699839361 container died b2d1b79c8820ac627fd0cb26c6d9772cae02eacf3a82de0ca36bf58705c12624 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-23T23:15:28.830 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:28 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:28.615+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:28.831 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:28 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:28.615+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:28.859 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.5.service' 2024-03-23T23:15:28.901 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:15:28.901 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-23T23:15:28.902 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-23T23:15:28.902 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.6 2024-03-23T23:15:29.035 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:28 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:28.755+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:29.035 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:28 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:28.755+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.824259+0000 front 2024-03-23T23:14:50.824337+0000 (oldest deadline 2024-03-23T23:15:16.724560+0000) 2024-03-23T23:15:29.035 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:28 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:28.755+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:15:00.728957+0000 front 2024-03-23T23:15:00.728783+0000 (oldest deadline 2024-03-23T23:15:26.629053+0000) 2024-03-23T23:15:29.035 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:28 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:28.862+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:29.035 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:28 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:28.862+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:29.035 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:28 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:28.862+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:29.117 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:28 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:28.807+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.485743+0000 front 2024-03-23T23:14:52.485777+0000 (oldest deadline 2024-03-23T23:15:18.286503+0000) 2024-03-23T23:15:29.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:28 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:28.807+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:15:00.587549+0000 front 2024-03-23T23:15:00.587647+0000 (oldest deadline 2024-03-23T23:15:26.987999+0000) 2024-03-23T23:15:29.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:28 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:28.797+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:29.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:28 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:28.797+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:29.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:28 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:28.797+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:29.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:28 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:28.959+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:29.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:28 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:28.959+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:29.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:28 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:28.959+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:29.303 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:29 smithi012 systemd[1]: Stopping Ceph osd.6 for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:15:29.804 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:29 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:29.509+0000 7fe652600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:15:29.804 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:29 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:29.509+0000 7fe652600700 -1 osd.6 91 *** Got signal Terminated *** 2024-03-23T23:15:29.804 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:29 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:29.509+0000 7fe652600700 -1 osd.6 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-23T23:15:29.804 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:29 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:29.745+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:29.804 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:29 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:29.745+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.824259+0000 front 2024-03-23T23:14:50.824337+0000 (oldest deadline 2024-03-23T23:15:16.724560+0000) 2024-03-23T23:15:29.804 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:29 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:29.745+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:15:00.728957+0000 front 2024-03-23T23:15:00.728783+0000 (oldest deadline 2024-03-23T23:15:26.629053+0000) 2024-03-23T23:15:29.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:29 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:29.625+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:29.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:29 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:29.625+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:30.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:29 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:29.842+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.485743+0000 front 2024-03-23T23:14:52.485777+0000 (oldest deadline 2024-03-23T23:15:18.286503+0000) 2024-03-23T23:15:30.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:29 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:29.842+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:15:00.587549+0000 front 2024-03-23T23:15:00.587647+0000 (oldest deadline 2024-03-23T23:15:26.987999+0000) 2024-03-23T23:15:30.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:29 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:29.844+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:30.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:29 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:29.844+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:30.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:29 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:29.844+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:30.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:29 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:29.844+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:30.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:29 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:29.861+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:30.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:29 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:29.861+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:30.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:29 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:29.861+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:30.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:29 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:29.974+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:30.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:29 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:29.974+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:30.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:29 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:29.974+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:30.942 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:30 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:30.674+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:30.943 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:30 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:30.674+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:31.054 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:30 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:30.785+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:31.054 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:30 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:30.785+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.824259+0000 front 2024-03-23T23:14:50.824337+0000 (oldest deadline 2024-03-23T23:15:16.724560+0000) 2024-03-23T23:15:31.054 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:30 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:30.785+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:15:00.728957+0000 front 2024-03-23T23:15:00.728783+0000 (oldest deadline 2024-03-23T23:15:26.629053+0000) 2024-03-23T23:15:31.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:30 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:30.872+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:31.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:30 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:30.872+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:31.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:30 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:30.872+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:31.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:30 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:30.892+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.485743+0000 front 2024-03-23T23:14:52.485777+0000 (oldest deadline 2024-03-23T23:15:18.286503+0000) 2024-03-23T23:15:31.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:30 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:30.892+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:15:00.587549+0000 front 2024-03-23T23:15:00.587647+0000 (oldest deadline 2024-03-23T23:15:26.987999+0000) 2024-03-23T23:15:31.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:30 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:30.857+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:31.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:30 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:30.857+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:31.119 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:30 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:30.857+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:31.119 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:30 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:30.857+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:31.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:30 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:30.943+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:31.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:30 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:30.943+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:31.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:30 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:30.943+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:32.054 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:31 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:31.745+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:32.054 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:31 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:31.745+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.824259+0000 front 2024-03-23T23:14:50.824337+0000 (oldest deadline 2024-03-23T23:15:16.724560+0000) 2024-03-23T23:15:32.054 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:31 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:31.745+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:15:00.728957+0000 front 2024-03-23T23:15:00.728783+0000 (oldest deadline 2024-03-23T23:15:26.629053+0000) 2024-03-23T23:15:32.055 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:31 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:31.871+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:32.055 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:31 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:31.871+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:32.055 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:31 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:31.871+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:32.055 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:31 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:31.871+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:32.157 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:31 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:31.709+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:32.158 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:31 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:31.709+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:32.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:31 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:31.914+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:32.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:31 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:31.914+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:32.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:31 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:31.914+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:32.368 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:31 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:31.926+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.485743+0000 front 2024-03-23T23:14:52.485777+0000 (oldest deadline 2024-03-23T23:15:18.286503+0000) 2024-03-23T23:15:32.369 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:31 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:31.926+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:15:00.587549+0000 front 2024-03-23T23:15:00.587647+0000 (oldest deadline 2024-03-23T23:15:26.987999+0000) 2024-03-23T23:15:32.369 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:31 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:31.926+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.988906+0000 front 2024-03-23T23:15:06.988773+0000 (oldest deadline 2024-03-23T23:15:31.688605+0000) 2024-03-23T23:15:32.369 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:31 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:31.870+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:32.369 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:31 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:31.870+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:32.369 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:31 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:31.870+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:32.369 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:31 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:31.870+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:33.054 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:32 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:32.780+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:33.054 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:32 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:32.780+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.824259+0000 front 2024-03-23T23:14:50.824337+0000 (oldest deadline 2024-03-23T23:15:16.724560+0000) 2024-03-23T23:15:33.054 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:32 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:32.780+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:15:00.728957+0000 front 2024-03-23T23:15:00.728783+0000 (oldest deadline 2024-03-23T23:15:26.629053+0000) 2024-03-23T23:15:33.054 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:32 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:32.780+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.630567+0000 front 2024-03-23T23:15:06.630344+0000 (oldest deadline 2024-03-23T23:15:32.529847+0000) 2024-03-23T23:15:33.055 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:32 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:32.846+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:33.055 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:32 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:32.846+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:33.055 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:32 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:32.846+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:33.055 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:32 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:32.846+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:33.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:32 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:32.867+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:33.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:32 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:32.867+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:33.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:32 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:32.867+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:33.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:32 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:32.867+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:33.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:32 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:32.949+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.485743+0000 front 2024-03-23T23:14:52.485777+0000 (oldest deadline 2024-03-23T23:15:18.286503+0000) 2024-03-23T23:15:33.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:32 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:32.949+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:15:00.587549+0000 front 2024-03-23T23:15:00.587647+0000 (oldest deadline 2024-03-23T23:15:26.987999+0000) 2024-03-23T23:15:33.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:32 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:32.949+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.988906+0000 front 2024-03-23T23:15:06.988773+0000 (oldest deadline 2024-03-23T23:15:31.688605+0000) 2024-03-23T23:15:33.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:32 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:32.870+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:33.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:32 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:32.870+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:33.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:32 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:32.870+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:33.158 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:32 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:32.732+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:33.158 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:32 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:32.732+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:33.159 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:32 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:32.732+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:08.466973+0000 front 2024-03-23T23:15:08.467207+0000 (oldest deadline 2024-03-23T23:15:31.966878+0000) 2024-03-23T23:15:34.054 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:33 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:33.747+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:42.120920+0000 front 2024-03-23T23:14:42.120991+0000 (oldest deadline 2024-03-23T23:15:07.420991+0000) 2024-03-23T23:15:34.054 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:33 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:33.747+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.824259+0000 front 2024-03-23T23:14:50.824337+0000 (oldest deadline 2024-03-23T23:15:16.724560+0000) 2024-03-23T23:15:34.054 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:33 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:33.747+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:15:00.728957+0000 front 2024-03-23T23:15:00.728783+0000 (oldest deadline 2024-03-23T23:15:26.629053+0000) 2024-03-23T23:15:34.054 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:33 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6[59987]: 2024-03-23T23:15:33.747+0000 7fe648e00700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.630567+0000 front 2024-03-23T23:15:06.630344+0000 (oldest deadline 2024-03-23T23:15:32.529847+0000) 2024-03-23T23:15:34.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:33 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:33.844+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:34.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:33 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:33.844+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:34.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:33 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:33.844+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:34.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:33 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:33.844+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:34.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:33 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:33.968+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.485743+0000 front 2024-03-23T23:14:52.485777+0000 (oldest deadline 2024-03-23T23:15:18.286503+0000) 2024-03-23T23:15:34.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:33 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:33.968+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:15:00.587549+0000 front 2024-03-23T23:15:00.587647+0000 (oldest deadline 2024-03-23T23:15:26.987999+0000) 2024-03-23T23:15:34.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:33 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:33.968+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.988906+0000 front 2024-03-23T23:15:06.988773+0000 (oldest deadline 2024-03-23T23:15:31.688605+0000) 2024-03-23T23:15:34.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:33 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:33.825+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:34.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:33 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:33.825+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:34.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:33 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:33.825+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:34.119 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:33 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:33.825+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:34.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:33 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:33.864+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:34.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:33 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:33.864+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:34.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:33 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:33.864+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:34.157 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:33 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:33.717+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:34.157 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:33 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:33.717+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:34.157 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:33 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:33.717+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:08.466973+0000 front 2024-03-23T23:15:08.467207+0000 (oldest deadline 2024-03-23T23:15:31.966878+0000) 2024-03-23T23:15:35.119 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:34 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:34.950+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.485743+0000 front 2024-03-23T23:14:52.485777+0000 (oldest deadline 2024-03-23T23:15:18.286503+0000) 2024-03-23T23:15:35.119 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:34 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:34.950+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:15:00.587549+0000 front 2024-03-23T23:15:00.587647+0000 (oldest deadline 2024-03-23T23:15:26.987999+0000) 2024-03-23T23:15:35.119 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:34 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:34.950+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.988906+0000 front 2024-03-23T23:15:06.988773+0000 (oldest deadline 2024-03-23T23:15:31.688605+0000) 2024-03-23T23:15:35.119 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:34 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:34.852+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:35.119 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:34 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:34.852+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:35.120 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:34 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:34.852+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:35.120 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:34 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:34.852+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:35.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:34 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:34.849+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:35.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:34 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:34.849+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:35.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:34 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:34.849+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:35.157 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:34 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:34.689+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:35.158 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:34 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:34.689+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:35.158 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:34 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:34.689+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:08.466973+0000 front 2024-03-23T23:15:08.467207+0000 (oldest deadline 2024-03-23T23:15:31.966878+0000) 2024-03-23T23:15:35.303 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:35 smithi012 podman[101155]: 2024-03-23 23:15:35.05880192 +0000 UTC m=+5.694712767 container died a2b25b14b545ef7886f7a570da16959e2188cd7b56353e8e17a985183f852ff4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-23T23:15:35.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:34 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:34.813+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:35.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:34 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:34.813+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:35.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:34 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:34.813+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:35.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:34 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:34.813+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:35.744 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:35 smithi012 podman[101155]: 2024-03-23 23:15:35.510255129 +0000 UTC m=+6.146166014 container cleanup a2b25b14b545ef7886f7a570da16959e2188cd7b56353e8e17a985183f852ff4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-23T23:15:35.744 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:35 smithi012 bash[101155]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6 2024-03-23T23:15:36.053 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:35 smithi012 podman[101174]: 2024-03-23 23:15:35.7448265 +0000 UTC m=+0.671155976 container remove a2b25b14b545ef7886f7a570da16959e2188cd7b56353e8e17a985183f852ff4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-23T23:15:36.053 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:35 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:35.835+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:36.053 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:35 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:35.835+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:36.053 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:35 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:35.835+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:36.054 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:35 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:35.835+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:36.058 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:35 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:35.881+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:36.059 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:35 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:35.881+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:36.059 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:35 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:35.881+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:36.059 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:35 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:35.700+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:36.059 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:35 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:35.700+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:36.059 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:35 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:35.700+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:08.466973+0000 front 2024-03-23T23:15:08.467207+0000 (oldest deadline 2024-03-23T23:15:31.966878+0000) 2024-03-23T23:15:36.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:35 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:35.966+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.485743+0000 front 2024-03-23T23:14:52.485777+0000 (oldest deadline 2024-03-23T23:15:18.286503+0000) 2024-03-23T23:15:36.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:35 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:35.966+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:15:00.587549+0000 front 2024-03-23T23:15:00.587647+0000 (oldest deadline 2024-03-23T23:15:26.987999+0000) 2024-03-23T23:15:36.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:35 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:35.966+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.988906+0000 front 2024-03-23T23:15:06.988773+0000 (oldest deadline 2024-03-23T23:15:31.688605+0000) 2024-03-23T23:15:36.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:35 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:35.846+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:36.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:35 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:35.846+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:36.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:35 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:35.846+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:36.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:35 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:35.846+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:36.857 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:36 smithi012 podman[101263]: 2024-03-23 23:15:36.504164461 +0000 UTC m=+0.036656491 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-23T23:15:36.858 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:36 smithi012 podman[101263]: 2024-03-23 23:15:36.620657325 +0000 UTC m=+0.153149381 container create 480e4ece0170689322fcbbb3c38d080a5bd0559e50e6f7added5b031b9963f6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-23T23:15:37.072 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:36 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:36.843+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:37.072 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:36 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:36.843+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:37.072 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:36 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:36.843+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:37.072 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:36 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:36.707+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:37.073 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:36 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:36.707+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:37.073 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:36 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:36.707+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:08.466973+0000 front 2024-03-23T23:15:08.467207+0000 (oldest deadline 2024-03-23T23:15:31.966878+0000) 2024-03-23T23:15:37.305 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:36 smithi012 podman[101263]: 2024-03-23 23:15:36.882585975 +0000 UTC m=+0.415078037 container init 480e4ece0170689322fcbbb3c38d080a5bd0559e50e6f7added5b031b9963f6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-23T23:15:37.305 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:36 smithi012 podman[101263]: 2024-03-23 23:15:36.894729862 +0000 UTC m=+0.427221924 container start 480e4ece0170689322fcbbb3c38d080a5bd0559e50e6f7added5b031b9963f6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-23T23:15:37.305 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:36 smithi012 podman[101263]: 2024-03-23 23:15:36.964571821 +0000 UTC m=+0.497063921 container attach 480e4ece0170689322fcbbb3c38d080a5bd0559e50e6f7added5b031b9963f6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-23T23:15:37.305 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:36 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:36.858+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:37.305 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:36 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:36.858+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:37.305 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:36 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:36.858+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:37.305 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:36 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:36.858+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:37.368 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:36 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:36.944+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.485743+0000 front 2024-03-23T23:14:52.485777+0000 (oldest deadline 2024-03-23T23:15:18.286503+0000) 2024-03-23T23:15:37.368 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:36 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:36.944+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:15:00.587549+0000 front 2024-03-23T23:15:00.587647+0000 (oldest deadline 2024-03-23T23:15:26.987999+0000) 2024-03-23T23:15:37.368 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:36 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:36.944+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.988906+0000 front 2024-03-23T23:15:06.988773+0000 (oldest deadline 2024-03-23T23:15:31.688605+0000) 2024-03-23T23:15:37.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:36 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:36.876+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:37.369 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:36 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:36.876+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:37.369 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:36 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:36.876+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:37.369 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:36 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:36.876+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:37.629 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 23 23:15:37 smithi012 podman[101263]: 2024-03-23 23:15:37.337406351 +0000 UTC m=+0.869898450 container died 480e4ece0170689322fcbbb3c38d080a5bd0559e50e6f7added5b031b9963f6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-23T23:15:37.983 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:37 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:37.823+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:37.984 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:37 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:37.823+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:37.984 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:37 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:37.823+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:37.984 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:37 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:37.823+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:38.025 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.6.service' 2024-03-23T23:15:38.119 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:15:38.120 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-23T23:15:38.120 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-23T23:15:38.120 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.7 2024-03-23T23:15:38.151 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:37 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:37.909+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.485743+0000 front 2024-03-23T23:14:52.485777+0000 (oldest deadline 2024-03-23T23:15:18.286503+0000) 2024-03-23T23:15:38.151 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:37 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:37.909+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:15:00.587549+0000 front 2024-03-23T23:15:00.587647+0000 (oldest deadline 2024-03-23T23:15:26.987999+0000) 2024-03-23T23:15:38.151 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:37 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:37.909+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.988906+0000 front 2024-03-23T23:15:06.988773+0000 (oldest deadline 2024-03-23T23:15:31.688605+0000) 2024-03-23T23:15:38.151 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:37 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:37.890+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:38.151 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:37 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:37.890+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:38.151 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:37 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:37.890+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:38.152 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:37 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:37.890+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:38.157 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:37 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:37.665+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:38.157 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:37 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:37.665+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:38.157 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:37 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:37.665+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:08.466973+0000 front 2024-03-23T23:15:08.467207+0000 (oldest deadline 2024-03-23T23:15:31.966878+0000) 2024-03-23T23:15:38.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:37 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:37.846+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:38.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:37 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:37.846+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:38.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:37 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:37.846+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:38.418 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:38 smithi173 systemd[1]: Stopping Ceph osd.7 for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:15:38.418 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:38 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:38.418+0000 7efcdde31700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:15:38.418 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:38 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:38.418+0000 7efcdde31700 -1 osd.7 91 *** Got signal Terminated *** 2024-03-23T23:15:38.418 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:38 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:38.418+0000 7efcdde31700 -1 osd.7 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-23T23:15:38.908 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:38 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:38.618+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:38.908 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:38 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:38.618+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:38.908 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:38 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:38.618+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:08.466973+0000 front 2024-03-23T23:15:08.467207+0000 (oldest deadline 2024-03-23T23:15:31.966878+0000) 2024-03-23T23:15:38.908 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:38 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:38.868+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:38.909 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:38 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:38.868+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:38.909 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:38 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:38.868+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:39.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:38 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:38.843+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:39.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:38 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:38.843+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:39.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:38 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:38.843+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:39.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:38 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:38.843+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:39.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:38 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:38.878+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.485743+0000 front 2024-03-23T23:14:52.485777+0000 (oldest deadline 2024-03-23T23:15:18.286503+0000) 2024-03-23T23:15:39.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:38 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:38.878+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:15:00.587549+0000 front 2024-03-23T23:15:00.587647+0000 (oldest deadline 2024-03-23T23:15:26.987999+0000) 2024-03-23T23:15:39.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:38 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:38.878+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.988906+0000 front 2024-03-23T23:15:06.988773+0000 (oldest deadline 2024-03-23T23:15:31.688605+0000) 2024-03-23T23:15:39.303 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:38 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:38.827+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:39.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:38 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:38.827+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:39.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:38 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:38.827+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:39.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:38 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:38.827+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:39.907 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:39 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:39.827+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:39.907 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:39 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:39.827+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:39.907 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:39 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:39.827+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:39.908 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:39 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:39.583+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:39.908 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:39 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:39.583+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:39.908 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:39 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:39.583+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:08.466973+0000 front 2024-03-23T23:15:08.467207+0000 (oldest deadline 2024-03-23T23:15:31.966878+0000) 2024-03-23T23:15:39.908 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:39 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:39.583+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.069470+0000 front 2024-03-23T23:15:17.069539+0000 (oldest deadline 2024-03-23T23:15:38.769385+0000) 2024-03-23T23:15:40.117 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:39 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:39.832+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.485743+0000 front 2024-03-23T23:14:52.485777+0000 (oldest deadline 2024-03-23T23:15:18.286503+0000) 2024-03-23T23:15:40.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:39 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:39.832+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:15:00.587549+0000 front 2024-03-23T23:15:00.587647+0000 (oldest deadline 2024-03-23T23:15:26.987999+0000) 2024-03-23T23:15:40.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:39 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:39.832+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.988906+0000 front 2024-03-23T23:15:06.988773+0000 (oldest deadline 2024-03-23T23:15:31.688605+0000) 2024-03-23T23:15:40.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:39 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:39.844+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:40.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:39 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:39.844+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:40.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:39 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:39.844+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:40.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:39 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:39.844+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:40.303 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:39 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:39.846+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:40.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:39 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:39.846+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:40.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:39 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:39.846+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:40.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:39 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:39.846+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:40.907 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:40 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:40.832+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:40.907 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:40 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:40.832+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:40.907 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:40 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:40.832+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:40.908 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:40 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:40.587+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:40.908 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:40 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:40.587+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:40.908 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:40 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:40.587+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:08.466973+0000 front 2024-03-23T23:15:08.467207+0000 (oldest deadline 2024-03-23T23:15:31.966878+0000) 2024-03-23T23:15:40.908 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:40 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:40.588+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.069470+0000 front 2024-03-23T23:15:17.069539+0000 (oldest deadline 2024-03-23T23:15:38.769385+0000) 2024-03-23T23:15:41.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:40 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:40.856+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.485743+0000 front 2024-03-23T23:14:52.485777+0000 (oldest deadline 2024-03-23T23:15:18.286503+0000) 2024-03-23T23:15:41.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:40 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:40.856+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:15:00.587549+0000 front 2024-03-23T23:15:00.587647+0000 (oldest deadline 2024-03-23T23:15:26.987999+0000) 2024-03-23T23:15:41.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:40 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:40.856+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.988906+0000 front 2024-03-23T23:15:06.988773+0000 (oldest deadline 2024-03-23T23:15:31.688605+0000) 2024-03-23T23:15:41.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:40 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:40.856+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:41.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:40 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:40.856+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:41.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:40 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:40.856+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:41.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:40 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:40.856+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:41.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:40 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:40.856+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:15:41.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:40 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:40.896+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:41.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:40 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:40.896+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:41.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:40 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:40.896+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:41.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:40 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:40.896+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:41.845 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:41 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:41.587+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:41.845 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:41 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:41.587+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:41.845 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:41 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:41.587+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:08.466973+0000 front 2024-03-23T23:15:08.467207+0000 (oldest deadline 2024-03-23T23:15:31.966878+0000) 2024-03-23T23:15:41.845 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:41 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:41.587+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.069470+0000 front 2024-03-23T23:15:17.069539+0000 (oldest deadline 2024-03-23T23:15:38.769385+0000) 2024-03-23T23:15:42.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:41 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:41.818+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.485743+0000 front 2024-03-23T23:14:52.485777+0000 (oldest deadline 2024-03-23T23:15:18.286503+0000) 2024-03-23T23:15:42.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:41 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:41.818+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:15:00.587549+0000 front 2024-03-23T23:15:00.587647+0000 (oldest deadline 2024-03-23T23:15:26.987999+0000) 2024-03-23T23:15:42.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:41 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:41.818+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.988906+0000 front 2024-03-23T23:15:06.988773+0000 (oldest deadline 2024-03-23T23:15:31.688605+0000) 2024-03-23T23:15:42.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:41 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:41.822+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:42.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:41 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:41.822+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:42.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:41 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:41.822+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:42.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:41 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:41.822+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:42.119 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:41 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:41.822+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:15:42.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:41 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:41.845+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:42.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:41 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:41.845+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:42.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:41 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:41.845+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:42.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:41 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:41.845+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:15:42.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:41 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:41.939+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:42.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:41 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:41.939+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:42.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:41 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:41.939+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:42.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:41 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:41.939+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:42.837 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:42 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:42.549+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:42.837 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:42 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:42.549+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:42.837 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:42 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:42.549+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:08.466973+0000 front 2024-03-23T23:15:08.467207+0000 (oldest deadline 2024-03-23T23:15:31.966878+0000) 2024-03-23T23:15:42.837 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:42 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:42.549+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.069470+0000 front 2024-03-23T23:15:17.069539+0000 (oldest deadline 2024-03-23T23:15:38.769385+0000) 2024-03-23T23:15:43.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:42 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:42.856+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.485743+0000 front 2024-03-23T23:14:52.485777+0000 (oldest deadline 2024-03-23T23:15:18.286503+0000) 2024-03-23T23:15:43.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:42 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:42.856+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:15:00.587549+0000 front 2024-03-23T23:15:00.587647+0000 (oldest deadline 2024-03-23T23:15:26.987999+0000) 2024-03-23T23:15:43.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:42 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7[54254]: 2024-03-23T23:15:42.856+0000 7efcd641d700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.988906+0000 front 2024-03-23T23:15:06.988773+0000 (oldest deadline 2024-03-23T23:15:31.688605+0000) 2024-03-23T23:15:43.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:42 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:42.832+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:43.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:42 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:42.832+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:43.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:42 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:42.832+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:43.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:42 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:42.832+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:43.119 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:42 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:42.832+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:15:43.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:42 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:42.837+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:43.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:42 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:42.837+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:43.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:42 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:42.837+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:43.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:42 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:42.837+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:15:43.303 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:42 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:42.981+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:43.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:42 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:42.981+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:43.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:42 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:42.981+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:43.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:42 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:42.981+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:43.870 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:43 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:43.541+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:43.870 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:43 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:43.541+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:43.870 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:43 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:43.541+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:08.466973+0000 front 2024-03-23T23:15:08.467207+0000 (oldest deadline 2024-03-23T23:15:31.966878+0000) 2024-03-23T23:15:43.871 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:43 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:43.541+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.069470+0000 front 2024-03-23T23:15:17.069539+0000 (oldest deadline 2024-03-23T23:15:38.769385+0000) 2024-03-23T23:15:44.118 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:43 smithi173 podman[71855]: 2024-03-23 23:15:43.842766712 +0000 UTC m=+5.527794180 container died 81d6f123480612b81bb0ceb63493119af04e4c23770c676fb3a6161e38147656 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-23T23:15:44.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:43 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:43.836+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:44.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:43 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:43.836+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:44.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:43 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:43.836+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:44.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:43 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:43.836+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:44.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:43 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:43.836+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:15:44.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:43 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:43.870+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:44.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:43 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:43.870+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:44.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:43 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:43.870+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:44.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:43 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:43.870+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:15:44.303 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:43 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:43.934+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:44.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:43 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:43.934+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:44.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:43 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:43.934+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:44.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:43 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:43.934+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:44.435 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:44 smithi173 podman[71855]: 2024-03-23 23:15:44.227149012 +0000 UTC m=+5.912176483 container cleanup 81d6f123480612b81bb0ceb63493119af04e4c23770c676fb3a6161e38147656 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-23T23:15:44.435 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:44 smithi173 bash[71855]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7 2024-03-23T23:15:44.753 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:44 smithi173 podman[71872]: 2024-03-23 23:15:44.436739828 +0000 UTC m=+0.592579540 container remove 81d6f123480612b81bb0ceb63493119af04e4c23770c676fb3a6161e38147656 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-23T23:15:44.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:44 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:44.546+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:44.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:44 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:44.546+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:44.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:44 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:44.546+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:08.466973+0000 front 2024-03-23T23:15:08.467207+0000 (oldest deadline 2024-03-23T23:15:31.966878+0000) 2024-03-23T23:15:44.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:44 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:44.546+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.069470+0000 front 2024-03-23T23:15:17.069539+0000 (oldest deadline 2024-03-23T23:15:38.769385+0000) 2024-03-23T23:15:45.072 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:44 smithi173 podman[71959]: 2024-03-23 23:15:44.656942969 +0000 UTC m=+0.017615783 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-23T23:15:45.072 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:44 smithi173 podman[71959]: 2024-03-23 23:15:44.762672862 +0000 UTC m=+0.123345706 container create d1b4dd6b6c4fd77430e8fe8c64765873dc11a401eb245ad4f901bd38cc56bcc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-23T23:15:45.072 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:44 smithi173 podman[71959]: 2024-03-23 23:15:44.981779779 +0000 UTC m=+0.342452599 container init d1b4dd6b6c4fd77430e8fe8c64765873dc11a401eb245ad4f901bd38cc56bcc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7-deactivate, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-23T23:15:45.072 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:44 smithi173 podman[71959]: 2024-03-23 23:15:44.984965966 +0000 UTC m=+0.345638774 container start d1b4dd6b6c4fd77430e8fe8c64765873dc11a401eb245ad4f901bd38cc56bcc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-23T23:15:45.073 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:44 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:44.799+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:45.073 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:44 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:44.799+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:45.073 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:44 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:44.799+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:45.073 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:44 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:44.799+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:45.073 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:44 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:44.800+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:15:45.303 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:44 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:44.965+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:45.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:44 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:44.965+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:45.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:44 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:44.965+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:45.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:44 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:44.965+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:45.368 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:45 smithi173 podman[71959]: 2024-03-23 23:15:45.073518274 +0000 UTC m=+0.434191086 container attach d1b4dd6b6c4fd77430e8fe8c64765873dc11a401eb245ad4f901bd38cc56bcc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319) 2024-03-23T23:15:45.368 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:45 smithi173 podman[71959]: 2024-03-23 23:15:45.270890991 +0000 UTC m=+0.631563787 container died d1b4dd6b6c4fd77430e8fe8c64765873dc11a401eb245ad4f901bd38cc56bcc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-23T23:15:45.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:44 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:44.912+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:45.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:44 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:44.912+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:45.408 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:44 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:44.912+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:45.408 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:44 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:44.912+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:15:45.866 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.7.service' 2024-03-23T23:15:45.867 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:45 smithi173 podman[71985]: 2024-03-23 23:15:45.844309704 +0000 UTC m=+0.568207917 container remove d1b4dd6b6c4fd77430e8fe8c64765873dc11a401eb245ad4f901bd38cc56bcc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-23T23:15:45.867 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:45 smithi173 systemd[1]: ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.7.service: Deactivated successfully. 2024-03-23T23:15:45.868 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:45 smithi173 systemd[1]: Stopped Ceph osd.7 for 90f679ae-e967-11ee-95cd-87774f69a715. 2024-03-23T23:15:45.868 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 23 23:15:45 smithi173 systemd[1]: ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.7.service: Consumed 9.802s CPU time. 2024-03-23T23:15:45.868 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:45 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:45.796+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:45.868 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:45 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:45.796+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:45.868 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:45 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:45.796+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:45.868 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:45 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:45.796+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:45.868 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:45 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:45.796+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:15:45.907 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:15:45.908 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-23T23:15:45.908 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-23T23:15:45.908 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.8 2024-03-23T23:15:45.909 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:45 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:45.498+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:45.909 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:45 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:45.498+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:45.909 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:45 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:45.498+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:08.466973+0000 front 2024-03-23T23:15:08.467207+0000 (oldest deadline 2024-03-23T23:15:31.966878+0000) 2024-03-23T23:15:45.909 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:45 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:45.498+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.069470+0000 front 2024-03-23T23:15:17.069539+0000 (oldest deadline 2024-03-23T23:15:38.769385+0000) 2024-03-23T23:15:46.226 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:45 smithi203 systemd[1]: Stopping Ceph osd.8 for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:15:46.226 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:45 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:45.940+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:46.226 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:45 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:45.940+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:46.226 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:45 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:45.940+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:46.226 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:45 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:45.940+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:15:46.303 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:45 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:45.937+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:46.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:45 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:45.937+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:46.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:45 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:45.937+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:46.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:45 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:45.937+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:46.514 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:46 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:46.226+0000 7f39cf71a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:15:46.515 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:46 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:46.226+0000 7f39cf71a700 -1 osd.8 91 *** Got signal Terminated *** 2024-03-23T23:15:46.515 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:46 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:46.226+0000 7f39cf71a700 -1 osd.8 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-23T23:15:46.515 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:46 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:46.515+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:46.515 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:46 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:46.515+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:46.515 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:46 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:46.515+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:08.466973+0000 front 2024-03-23T23:15:08.467207+0000 (oldest deadline 2024-03-23T23:15:31.966878+0000) 2024-03-23T23:15:46.515 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:46 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:46.515+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.069470+0000 front 2024-03-23T23:15:17.069539+0000 (oldest deadline 2024-03-23T23:15:38.769385+0000) 2024-03-23T23:15:46.868 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:46 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:46.763+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:46.868 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:46 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:46.763+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:46.868 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:46 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:46.763+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:46.868 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:46 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:46.763+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:46.868 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:46 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:46.763+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:15:47.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:46 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:46.895+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:47.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:46 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:46.895+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:47.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:46 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:46.895+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:47.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:46 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:46.895+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:15:47.303 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:46 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:46.898+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:47.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:46 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:46.898+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:47.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:46 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:46.898+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:47.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:46 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:46.898+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:47.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:47 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:47.542+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:47.908 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:47 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:47.542+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:47.908 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:47 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:47.542+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:08.466973+0000 front 2024-03-23T23:15:08.467207+0000 (oldest deadline 2024-03-23T23:15:31.966878+0000) 2024-03-23T23:15:47.908 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:47 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:47.542+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.069470+0000 front 2024-03-23T23:15:17.069539+0000 (oldest deadline 2024-03-23T23:15:38.769385+0000) 2024-03-23T23:15:47.908 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:47 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:47.542+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:24.571041+0000 front 2024-03-23T23:15:24.571053+0000 (oldest deadline 2024-03-23T23:15:47.471094+0000) 2024-03-23T23:15:48.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:47 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:47.801+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:48.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:47 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:47.801+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:48.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:47 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:47.801+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:48.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:47 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:47.801+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:48.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:47 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:47.801+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:15:48.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:47 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:47.801+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:22.349554+0000 front 2024-03-23T23:15:22.349467+0000 (oldest deadline 2024-03-23T23:15:47.049466+0000) 2024-03-23T23:15:48.303 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:47 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:47.894+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:48.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:47 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:47.894+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:48.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:47 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:47.894+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:48.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:47 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:47.894+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:48.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:47 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:47.894+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:23.626084+0000 front 2024-03-23T23:15:23.626019+0000 (oldest deadline 2024-03-23T23:15:47.726261+0000) 2024-03-23T23:15:48.330 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:47 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:47.916+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:48.330 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:47 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:47.916+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:48.330 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:47 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:47.916+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:48.330 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:47 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:47.916+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:15:48.657 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:48 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:48.564+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:48.657 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:48 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:48.564+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:48.657 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:48 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:48.564+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:08.466973+0000 front 2024-03-23T23:15:08.467207+0000 (oldest deadline 2024-03-23T23:15:31.966878+0000) 2024-03-23T23:15:48.658 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:48 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:48.564+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.069470+0000 front 2024-03-23T23:15:17.069539+0000 (oldest deadline 2024-03-23T23:15:38.769385+0000) 2024-03-23T23:15:48.658 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:48 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:48.564+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:24.571041+0000 front 2024-03-23T23:15:24.571053+0000 (oldest deadline 2024-03-23T23:15:47.471094+0000) 2024-03-23T23:15:49.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:48 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:48.850+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:49.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:48 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:48.850+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:49.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:48 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:48.850+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:49.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:48 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:48.850+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:49.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:48 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:48.850+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:15:49.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:48 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:48.850+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:22.349554+0000 front 2024-03-23T23:15:22.349467+0000 (oldest deadline 2024-03-23T23:15:47.049466+0000) 2024-03-23T23:15:49.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:48 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:48.898+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:49.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:48 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:48.898+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:49.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:48 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:48.898+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:49.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:48 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:48.898+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:49.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:48 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:48.898+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:23.626084+0000 front 2024-03-23T23:15:23.626019+0000 (oldest deadline 2024-03-23T23:15:47.726261+0000) 2024-03-23T23:15:49.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:48 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:48.957+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:49.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:48 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:48.957+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:49.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:48 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:48.957+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:49.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:48 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:48.957+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:15:49.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:49 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:49.552+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:49.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:49 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:49.552+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:49.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:49 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:49.552+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:08.466973+0000 front 2024-03-23T23:15:08.467207+0000 (oldest deadline 2024-03-23T23:15:31.966878+0000) 2024-03-23T23:15:49.907 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:49 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:49.552+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.069470+0000 front 2024-03-23T23:15:17.069539+0000 (oldest deadline 2024-03-23T23:15:38.769385+0000) 2024-03-23T23:15:49.908 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:49 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:49.552+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:24.571041+0000 front 2024-03-23T23:15:24.571053+0000 (oldest deadline 2024-03-23T23:15:47.471094+0000) 2024-03-23T23:15:50.117 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:49 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:49.848+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:50.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:49 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:49.848+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:50.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:49 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:49.848+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:50.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:49 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:49.848+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:50.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:49 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:49.848+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:15:50.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:49 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:49.848+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:22.349554+0000 front 2024-03-23T23:15:22.349467+0000 (oldest deadline 2024-03-23T23:15:47.049466+0000) 2024-03-23T23:15:50.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:49 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:49.919+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:50.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:49 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:49.919+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:50.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:49 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:49.919+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:50.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:49 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:49.919+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:50.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:49 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:49.919+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:23.626084+0000 front 2024-03-23T23:15:23.626019+0000 (oldest deadline 2024-03-23T23:15:47.726261+0000) 2024-03-23T23:15:50.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:49 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:49.908+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:50.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:49 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:49.908+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:50.408 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:49 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:49.908+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:50.408 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:49 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:49.908+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:15:50.881 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:50 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:50.564+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.463237+0000 front 2024-03-23T23:14:43.462912+0000 (oldest deadline 2024-03-23T23:15:08.162564+0000) 2024-03-23T23:15:50.881 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:50 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:50.564+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:56.765343+0000 front 2024-03-23T23:14:56.765499+0000 (oldest deadline 2024-03-23T23:15:22.665456+0000) 2024-03-23T23:15:50.881 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:50 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:50.564+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:08.466973+0000 front 2024-03-23T23:15:08.467207+0000 (oldest deadline 2024-03-23T23:15:31.966878+0000) 2024-03-23T23:15:50.881 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:50 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:50.564+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.069470+0000 front 2024-03-23T23:15:17.069539+0000 (oldest deadline 2024-03-23T23:15:38.769385+0000) 2024-03-23T23:15:50.881 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:50 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8[55843]: 2024-03-23T23:15:50.564+0000 7f39c7d06700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:24.571041+0000 front 2024-03-23T23:15:24.571053+0000 (oldest deadline 2024-03-23T23:15:47.471094+0000) 2024-03-23T23:15:51.117 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:50 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:50.848+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:51.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:50 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:50.848+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:51.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:50 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:50.848+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:51.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:50 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:50.848+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:51.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:50 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:50.848+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:15:51.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:50 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:50.848+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:22.349554+0000 front 2024-03-23T23:15:22.349467+0000 (oldest deadline 2024-03-23T23:15:47.049466+0000) 2024-03-23T23:15:51.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:50 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:50.881+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:51.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:50 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:50.881+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:51.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:50 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:50.881+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:51.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:50 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:50.881+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:15:51.303 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:50 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:50.922+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:51.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:50 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:50.922+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:51.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:50 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:50.922+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:51.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:50 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:50.922+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:51.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:50 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:50.922+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:23.626084+0000 front 2024-03-23T23:15:23.626019+0000 (oldest deadline 2024-03-23T23:15:47.726261+0000) 2024-03-23T23:15:51.874 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:51 smithi203 podman[74663]: 2024-03-23 23:15:51.560795405 +0000 UTC m=+5.459580197 container died 6df573fb668b23bd0e6c928261b0df0cd2c8c64837fd65c686ad09148459b21c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5) 2024-03-23T23:15:51.875 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:51 smithi203 podman[74663]: 2024-03-23 23:15:51.87634781 +0000 UTC m=+5.775132601 container cleanup 6df573fb668b23bd0e6c928261b0df0cd2c8c64837fd65c686ad09148459b21c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True) 2024-03-23T23:15:52.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:51 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:51.859+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:52.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:51 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:51.859+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:52.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:51 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:51.859+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:52.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:51 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:51.859+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:52.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:51 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:51.859+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:15:52.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:51 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:51.859+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:22.349554+0000 front 2024-03-23T23:15:22.349467+0000 (oldest deadline 2024-03-23T23:15:47.049466+0000) 2024-03-23T23:15:52.134 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:51 smithi203 bash[74663]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8 2024-03-23T23:15:52.134 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:51 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:51.928+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:52.135 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:51 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:51.928+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:52.135 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:51 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:51.928+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:52.135 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:51 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:51.928+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:15:52.135 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:51 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:51.928+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:25.370284+0000 front 2024-03-23T23:15:25.370438+0000 (oldest deadline 2024-03-23T23:15:51.270414+0000) 2024-03-23T23:15:52.303 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:51 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:51.878+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:52.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:51 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:51.878+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:52.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:51 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:51.878+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:52.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:51 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:51.878+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:52.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:51 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:51.878+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:23.626084+0000 front 2024-03-23T23:15:23.626019+0000 (oldest deadline 2024-03-23T23:15:47.726261+0000) 2024-03-23T23:15:52.407 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:52 smithi203 podman[74680]: 2024-03-23 23:15:52.13614441 +0000 UTC m=+0.576218776 container remove 6df573fb668b23bd0e6c928261b0df0cd2c8c64837fd65c686ad09148459b21c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True) 2024-03-23T23:15:52.847 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:52 smithi203 podman[74772]: 2024-03-23 23:15:52.44661395 +0000 UTC m=+0.017844510 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-23T23:15:52.847 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:52 smithi203 podman[74772]: 2024-03-23 23:15:52.54541966 +0000 UTC m=+0.116650205 container create 4f178beeab6be3ca4710013063cc95a3ce53f22d370433c4a7d93b93a08212b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True) 2024-03-23T23:15:52.847 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:52 smithi203 podman[74772]: 2024-03-23 23:15:52.748519803 +0000 UTC m=+0.319750363 container init 4f178beeab6be3ca4710013063cc95a3ce53f22d370433c4a7d93b93a08212b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-23T23:15:52.847 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:52 smithi203 podman[74772]: 2024-03-23 23:15:52.752087336 +0000 UTC m=+0.323317894 container start 4f178beeab6be3ca4710013063cc95a3ce53f22d370433c4a7d93b93a08212b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-23T23:15:53.157 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:52 smithi203 podman[74772]: 2024-03-23 23:15:52.848369077 +0000 UTC m=+0.419599649 container attach 4f178beeab6be3ca4710013063cc95a3ce53f22d370433c4a7d93b93a08212b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-23T23:15:53.157 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 23 23:15:53 smithi203 podman[74772]: 2024-03-23 23:15:53.048052087 +0000 UTC m=+0.619282649 container died 4f178beeab6be3ca4710013063cc95a3ce53f22d370433c4a7d93b93a08212b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-8-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-23T23:15:53.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:52 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:52.938+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:53.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:52 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:52.938+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:53.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:52 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:52.938+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:53.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:52 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:52.938+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:15:53.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:52 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:52.938+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:25.370284+0000 front 2024-03-23T23:15:25.370438+0000 (oldest deadline 2024-03-23T23:15:51.270414+0000) 2024-03-23T23:15:53.303 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:52 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:52.918+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:53.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:52 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:52.918+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:53.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:52 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:52.918+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:53.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:52 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:52.918+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:53.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:52 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:52.918+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:23.626084+0000 front 2024-03-23T23:15:23.626019+0000 (oldest deadline 2024-03-23T23:15:47.726261+0000) 2024-03-23T23:15:53.367 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:52 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:52.894+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:53.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:52 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:52.894+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:53.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:52 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:52.894+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:53.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:52 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:52.894+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:53.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:52 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:52.894+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:15:53.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:52 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:52.894+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:22.349554+0000 front 2024-03-23T23:15:22.349467+0000 (oldest deadline 2024-03-23T23:15:47.049466+0000) 2024-03-23T23:15:53.728 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.8.service' 2024-03-23T23:15:53.770 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:15:53.770 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-23T23:15:53.770 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-23T23:15:53.771 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.9 2024-03-23T23:15:54.271 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:53 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:53.977+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:54.271 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:53 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:53.977+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:54.271 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:53 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:53.977+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:54.271 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:53 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:53.977+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:15:54.271 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:53 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:53.977+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:25.370284+0000 front 2024-03-23T23:15:25.370438+0000 (oldest deadline 2024-03-23T23:15:51.270414+0000) 2024-03-23T23:15:54.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:53 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:53.885+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:54.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:53 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:53.885+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:54.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:53 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:53.885+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:54.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:53 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:53.885+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:54.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:53 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:53.885+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:23.626084+0000 front 2024-03-23T23:15:23.626019+0000 (oldest deadline 2024-03-23T23:15:47.726261+0000) 2024-03-23T23:15:54.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:53 smithi012 systemd[1]: Stopping Ceph osd.9 for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:15:54.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:53 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:53.911+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:54.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:53 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:53.911+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:54.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:53 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:53.911+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:54.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:53 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:53.911+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:54.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:53 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:53.911+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:15:54.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:53 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:53.911+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:22.349554+0000 front 2024-03-23T23:15:22.349467+0000 (oldest deadline 2024-03-23T23:15:47.049466+0000) 2024-03-23T23:15:54.803 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:54 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:54.429+0000 7f2a5f200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:15:54.804 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:54 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:54.429+0000 7f2a5f200700 -1 osd.9 91 *** Got signal Terminated *** 2024-03-23T23:15:54.804 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:54 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:54.429+0000 7f2a5f200700 -1 osd.9 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-23T23:15:55.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:54 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:54.869+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:55.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:54 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:54.869+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:55.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:54 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:54.869+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:55.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:54 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:54.869+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:55.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:54 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:54.869+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:23.626084+0000 front 2024-03-23T23:15:23.626019+0000 (oldest deadline 2024-03-23T23:15:47.726261+0000) 2024-03-23T23:15:55.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:54 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:54.934+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:55.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:54 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:54.934+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:55.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:54 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:54.934+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:55.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:54 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:54.934+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:55.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:54 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:54.934+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:15:55.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:54 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:54.934+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:22.349554+0000 front 2024-03-23T23:15:22.349467+0000 (oldest deadline 2024-03-23T23:15:47.049466+0000) 2024-03-23T23:15:55.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:54 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:54.951+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:55.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:54 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:54.951+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:55.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:54 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:54.951+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:55.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:54 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:54.951+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:15:55.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:54 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:54.951+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:25.370284+0000 front 2024-03-23T23:15:25.370438+0000 (oldest deadline 2024-03-23T23:15:51.270414+0000) 2024-03-23T23:15:56.303 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:55 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:55.888+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:56.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:55 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:55.888+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:56.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:55 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:55.888+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:56.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:55 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:55.888+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:56.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:55 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:55.888+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:23.626084+0000 front 2024-03-23T23:15:23.626019+0000 (oldest deadline 2024-03-23T23:15:47.726261+0000) 2024-03-23T23:15:56.367 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:55 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:55.961+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:56.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:55 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:55.961+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:56.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:55 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:55.961+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:56.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:55 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:55.961+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:56.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:55 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:55.961+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:15:56.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:55 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:55.961+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:22.349554+0000 front 2024-03-23T23:15:22.349467+0000 (oldest deadline 2024-03-23T23:15:47.049466+0000) 2024-03-23T23:15:56.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:55 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:55.961+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.750658+0000 front 2024-03-23T23:15:31.750777+0000 (oldest deadline 2024-03-23T23:15:55.250472+0000) 2024-03-23T23:15:56.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:55 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:55.973+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:56.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:55 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:55.973+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:56.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:55 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:55.973+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:56.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:55 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:55.973+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:15:56.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:55 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:55.973+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:25.370284+0000 front 2024-03-23T23:15:25.370438+0000 (oldest deadline 2024-03-23T23:15:51.270414+0000) 2024-03-23T23:15:56.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:55 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:55.973+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.271537+0000 front 2024-03-23T23:15:31.271450+0000 (oldest deadline 2024-03-23T23:15:55.371129+0000) 2024-03-23T23:15:57.303 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:56 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:56.890+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:57.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:56 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:56.890+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:57.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:56 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:56.890+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:57.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:56 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:56.890+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:57.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:56 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:56.890+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:23.626084+0000 front 2024-03-23T23:15:23.626019+0000 (oldest deadline 2024-03-23T23:15:47.726261+0000) 2024-03-23T23:15:57.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:56 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:56.994+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:57.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:56 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:56.995+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:57.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:56 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:56.995+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:57.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:56 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:56.995+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:57.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:56 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:56.995+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:15:57.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:56 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:56.995+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:22.349554+0000 front 2024-03-23T23:15:22.349467+0000 (oldest deadline 2024-03-23T23:15:47.049466+0000) 2024-03-23T23:15:57.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:56 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:56.995+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.750658+0000 front 2024-03-23T23:15:31.750777+0000 (oldest deadline 2024-03-23T23:15:55.250472+0000) 2024-03-23T23:15:57.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:56 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:56.957+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:57.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:56 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:56.957+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:57.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:56 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:56.957+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:57.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:56 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:56.957+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:15:57.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:56 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:56.958+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:25.370284+0000 front 2024-03-23T23:15:25.370438+0000 (oldest deadline 2024-03-23T23:15:51.270414+0000) 2024-03-23T23:15:57.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:56 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:56.958+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.271537+0000 front 2024-03-23T23:15:31.271450+0000 (oldest deadline 2024-03-23T23:15:55.371129+0000) 2024-03-23T23:15:58.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:57 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:57.860+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:58.583 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:57 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:57.860+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:58.583 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:57 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:57.860+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:58.583 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:57 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:57.860+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:58.583 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:57 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:57.860+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:23.626084+0000 front 2024-03-23T23:15:23.626019+0000 (oldest deadline 2024-03-23T23:15:47.726261+0000) 2024-03-23T23:15:58.583 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:57 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:57.860+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:34.031619+0000 front 2024-03-23T23:15:34.031078+0000 (oldest deadline 2024-03-23T23:15:56.931105+0000) 2024-03-23T23:15:58.584 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:57 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:57.968+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:58.584 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:57 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:57.968+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:58.584 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:57 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:57.968+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:58.584 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:57 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:57.968+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:15:58.584 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:57 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:57.968+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:25.370284+0000 front 2024-03-23T23:15:25.370438+0000 (oldest deadline 2024-03-23T23:15:51.270414+0000) 2024-03-23T23:15:58.584 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:57 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:57.968+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.271537+0000 front 2024-03-23T23:15:31.271450+0000 (oldest deadline 2024-03-23T23:15:55.371129+0000) 2024-03-23T23:15:58.585 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:58 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:58.044+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:58.585 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:58 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:58.044+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:58.585 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:58 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:58.044+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:58.585 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:58 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:58.044+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:58.585 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:58 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:58.044+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:15:58.585 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:58 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:58.044+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:22.349554+0000 front 2024-03-23T23:15:22.349467+0000 (oldest deadline 2024-03-23T23:15:47.049466+0000) 2024-03-23T23:15:58.585 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:58 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:58.044+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.750658+0000 front 2024-03-23T23:15:31.750777+0000 (oldest deadline 2024-03-23T23:15:55.250472+0000) 2024-03-23T23:15:59.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:58 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:58.878+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:43.315582+0000 front 2024-03-23T23:14:43.315534+0000 (oldest deadline 2024-03-23T23:15:05.014945+0000) 2024-03-23T23:15:59.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:58 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:58.878+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:52.519558+0000 front 2024-03-23T23:14:52.519458+0000 (oldest deadline 2024-03-23T23:15:18.419577+0000) 2024-03-23T23:15:59.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:58 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:58.878+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:58.420513+0000 front 2024-03-23T23:14:58.420575+0000 (oldest deadline 2024-03-23T23:15:24.320783+0000) 2024-03-23T23:15:59.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:58 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:58.878+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.623294+0000 front 2024-03-23T23:15:06.623346+0000 (oldest deadline 2024-03-23T23:15:31.322381+0000) 2024-03-23T23:15:59.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:58 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:58.878+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:23.626084+0000 front 2024-03-23T23:15:23.626019+0000 (oldest deadline 2024-03-23T23:15:47.726261+0000) 2024-03-23T23:15:59.304 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:58 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9[66847]: 2024-03-23T23:15:58.878+0000 7f2a5aa00700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:34.031619+0000 front 2024-03-23T23:15:34.031078+0000 (oldest deadline 2024-03-23T23:15:56.931105+0000) 2024-03-23T23:15:59.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:59 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:59.003+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:15:59.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:59 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:59.003+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:15:59.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:59 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:59.003+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:15:59.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:59 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:59.003+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:15:59.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:59 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:59.003+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:15:59.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:59 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:59.003+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:22.349554+0000 front 2024-03-23T23:15:22.349467+0000 (oldest deadline 2024-03-23T23:15:47.049466+0000) 2024-03-23T23:15:59.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:15:59 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:15:59.003+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.750658+0000 front 2024-03-23T23:15:31.750777+0000 (oldest deadline 2024-03-23T23:15:55.250472+0000) 2024-03-23T23:15:59.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:58 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:58.965+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:15:59.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:58 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:58.965+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:15:59.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:58 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:58.965+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:15:59.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:58 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:58.965+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:15:59.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:58 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:58.965+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:25.370284+0000 front 2024-03-23T23:15:25.370438+0000 (oldest deadline 2024-03-23T23:15:51.270414+0000) 2024-03-23T23:15:59.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:58 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:58.965+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.271537+0000 front 2024-03-23T23:15:31.271450+0000 (oldest deadline 2024-03-23T23:15:55.371129+0000) 2024-03-23T23:16:00.176 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:15:59 smithi012 podman[101379]: 2024-03-23 23:15:59.835618324 +0000 UTC m=+5.582858942 container died 3e97407675e6e2c32108e1b784ab236d2efeb65e2ae204003885ba9eb83062e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-23T23:16:00.176 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:16:00 smithi012 podman[101379]: 2024-03-23 23:16:00.175884189 +0000 UTC m=+5.923124847 container cleanup 3e97407675e6e2c32108e1b784ab236d2efeb65e2ae204003885ba9eb83062e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-23T23:16:00.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:00 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:00.036+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:16:00.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:00 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:00.036+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:16:00.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:00 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:00.036+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:16:00.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:00 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:00.036+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:16:00.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:00 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:00.036+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:16:00.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:00 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:00.036+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:22.349554+0000 front 2024-03-23T23:15:22.349467+0000 (oldest deadline 2024-03-23T23:15:47.049466+0000) 2024-03-23T23:16:00.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:00 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:00.036+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.750658+0000 front 2024-03-23T23:15:31.750777+0000 (oldest deadline 2024-03-23T23:15:55.250472+0000) 2024-03-23T23:16:00.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:59 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:59.924+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:16:00.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:59 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:59.924+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:16:00.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:59 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:59.924+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:16:00.408 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:59 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:59.924+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:16:00.408 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:59 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:59.924+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:25.370284+0000 front 2024-03-23T23:15:25.370438+0000 (oldest deadline 2024-03-23T23:15:51.270414+0000) 2024-03-23T23:16:00.408 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:15:59 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:15:59.924+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.271537+0000 front 2024-03-23T23:15:31.271450+0000 (oldest deadline 2024-03-23T23:15:55.371129+0000) 2024-03-23T23:16:00.553 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:16:00 smithi012 bash[101379]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9 2024-03-23T23:16:00.554 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:16:00 smithi012 podman[101397]: 2024-03-23 23:16:00.352406593 +0000 UTC m=+0.500907524 container remove 3e97407675e6e2c32108e1b784ab236d2efeb65e2ae204003885ba9eb83062e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-23T23:16:01.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:00 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:00.908+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:16:01.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:00 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:00.908+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:16:01.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:00 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:00.908+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:16:01.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:00 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:00.908+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:16:01.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:00 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:00.908+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:25.370284+0000 front 2024-03-23T23:15:25.370438+0000 (oldest deadline 2024-03-23T23:15:51.270414+0000) 2024-03-23T23:16:01.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:00 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:00.908+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.271537+0000 front 2024-03-23T23:15:31.271450+0000 (oldest deadline 2024-03-23T23:15:55.371129+0000) 2024-03-23T23:16:01.367 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:01 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:01.000+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:16:01.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:01 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:01.000+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:16:01.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:01 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:01.000+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:16:01.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:01 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:01.000+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:16:01.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:01 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:01.000+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:16:01.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:01 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:01.000+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:22.349554+0000 front 2024-03-23T23:15:22.349467+0000 (oldest deadline 2024-03-23T23:15:47.049466+0000) 2024-03-23T23:16:01.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:01 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:01.000+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.750658+0000 front 2024-03-23T23:15:31.750777+0000 (oldest deadline 2024-03-23T23:15:55.250472+0000) 2024-03-23T23:16:01.466 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:16:01 smithi012 podman[101484]: 2024-03-23 23:16:01.098803748 +0000 UTC m=+0.037388935 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-23T23:16:01.466 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:16:01 smithi012 podman[101484]: 2024-03-23 23:16:01.194835021 +0000 UTC m=+0.133420248 container create feec0d54c27996c0c998e88b7dcdee2728dc9041488800c362e92bcd6915d1b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-23T23:16:01.466 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:16:01 smithi012 podman[101484]: 2024-03-23 23:16:01.46574254 +0000 UTC m=+0.404327821 container init feec0d54c27996c0c998e88b7dcdee2728dc9041488800c362e92bcd6915d1b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-23T23:16:01.803 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:16:01 smithi012 podman[101484]: 2024-03-23 23:16:01.478169079 +0000 UTC m=+0.416754310 container start feec0d54c27996c0c998e88b7dcdee2728dc9041488800c362e92bcd6915d1b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-23T23:16:01.804 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:16:01 smithi012 podman[101484]: 2024-03-23 23:16:01.555625442 +0000 UTC m=+0.494210716 container attach feec0d54c27996c0c998e88b7dcdee2728dc9041488800c362e92bcd6915d1b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-23T23:16:02.270 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 23 23:16:01 smithi012 podman[101484]: 2024-03-23 23:16:01.921903064 +0000 UTC m=+0.860488251 container died feec0d54c27996c0c998e88b7dcdee2728dc9041488800c362e92bcd6915d1b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-9-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-23T23:16:02.367 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:02 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:02.007+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:16:02.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:02 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:02.007+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:16:02.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:02 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:02.007+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:16:02.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:02 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:02.007+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:16:02.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:02 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:02.007+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:16:02.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:02 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:02.007+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:22.349554+0000 front 2024-03-23T23:15:22.349467+0000 (oldest deadline 2024-03-23T23:15:47.049466+0000) 2024-03-23T23:16:02.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:02 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:02.007+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.750658+0000 front 2024-03-23T23:15:31.750777+0000 (oldest deadline 2024-03-23T23:15:55.250472+0000) 2024-03-23T23:16:02.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:01 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:01.925+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:16:02.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:01 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:01.925+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:16:02.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:01 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:01.925+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:16:02.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:01 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:01.925+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:16:02.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:01 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:01.925+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:25.370284+0000 front 2024-03-23T23:15:25.370438+0000 (oldest deadline 2024-03-23T23:15:51.270414+0000) 2024-03-23T23:16:02.408 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:01 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:01.925+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.271537+0000 front 2024-03-23T23:15:31.271450+0000 (oldest deadline 2024-03-23T23:15:55.371129+0000) 2024-03-23T23:16:02.641 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.9.service' 2024-03-23T23:16:02.758 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:16:02.758 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-23T23:16:02.758 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-23T23:16:02.758 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.10 2024-03-23T23:16:03.070 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:02 smithi173 systemd[1]: Stopping Ceph osd.10 for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:16:03.070 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:03 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:03.015+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:16:03.070 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:03 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:03.015+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:16:03.070 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:03 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:03.015+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:16:03.070 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:03 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:03.015+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:16:03.070 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:03 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:03.015+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:16:03.071 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:03 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:03.015+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:22.349554+0000 front 2024-03-23T23:15:22.349467+0000 (oldest deadline 2024-03-23T23:15:47.049466+0000) 2024-03-23T23:16:03.071 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:03 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:03.015+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.750658+0000 front 2024-03-23T23:15:31.750777+0000 (oldest deadline 2024-03-23T23:15:55.250472+0000) 2024-03-23T23:16:03.071 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:03 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:03.070+0000 7f7410ff4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:16:03.071 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:03 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:03.070+0000 7f7410ff4700 -1 osd.10 91 *** Got signal Terminated *** 2024-03-23T23:16:03.071 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:03 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:03.070+0000 7f7410ff4700 -1 osd.10 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-23T23:16:03.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:02 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:02.901+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:16:03.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:02 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:02.901+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:16:03.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:02 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:02.901+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:16:03.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:02 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:02.901+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:16:03.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:02 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:02.901+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:25.370284+0000 front 2024-03-23T23:15:25.370438+0000 (oldest deadline 2024-03-23T23:15:51.270414+0000) 2024-03-23T23:16:03.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:02 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:02.901+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.271537+0000 front 2024-03-23T23:15:31.271450+0000 (oldest deadline 2024-03-23T23:15:55.371129+0000) 2024-03-23T23:16:04.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:03 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:03.860+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:16:04.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:03 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:03.860+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:16:04.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:03 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:03.860+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:16:04.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:03 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:03.860+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:16:04.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:03 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:03.860+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:25.370284+0000 front 2024-03-23T23:15:25.370438+0000 (oldest deadline 2024-03-23T23:15:51.270414+0000) 2024-03-23T23:16:04.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:03 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:03.860+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.271537+0000 front 2024-03-23T23:15:31.271450+0000 (oldest deadline 2024-03-23T23:15:55.371129+0000) 2024-03-23T23:16:04.269 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:03 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:03.975+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:16:04.269 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:03 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:03.975+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:16:04.269 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:03 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:03.975+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:16:04.269 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:03 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:03.975+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:16:04.269 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:03 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:03.975+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:16:04.269 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:03 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:03.975+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:22.349554+0000 front 2024-03-23T23:15:22.349467+0000 (oldest deadline 2024-03-23T23:15:47.049466+0000) 2024-03-23T23:16:04.269 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:03 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:03.975+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.750658+0000 front 2024-03-23T23:15:31.750777+0000 (oldest deadline 2024-03-23T23:15:55.250472+0000) 2024-03-23T23:16:05.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:04 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:04.824+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:16:05.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:04 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:04.824+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:16:05.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:04 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:04.824+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:16:05.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:04 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:04.824+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:16:05.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:04 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:04.824+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:25.370284+0000 front 2024-03-23T23:15:25.370438+0000 (oldest deadline 2024-03-23T23:15:51.270414+0000) 2024-03-23T23:16:05.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:04 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:04.824+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.271537+0000 front 2024-03-23T23:15:31.271450+0000 (oldest deadline 2024-03-23T23:15:55.371129+0000) 2024-03-23T23:16:05.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:04 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:04.976+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:16:05.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:04 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:04.976+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:16:05.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:04 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:04.976+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:16:05.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:04 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:04.976+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:16:05.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:04 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:04.976+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:16:05.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:04 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:04.976+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:22.349554+0000 front 2024-03-23T23:15:22.349467+0000 (oldest deadline 2024-03-23T23:15:47.049466+0000) 2024-03-23T23:16:05.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:04 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:04.976+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.750658+0000 front 2024-03-23T23:15:31.750777+0000 (oldest deadline 2024-03-23T23:15:55.250472+0000) 2024-03-23T23:16:06.058 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:05 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:05.779+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:16:06.059 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:05 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:05.779+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:16:06.059 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:05 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:05.779+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:16:06.059 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:05 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:05.779+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:16:06.059 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:05 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:05.779+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:25.370284+0000 front 2024-03-23T23:15:25.370438+0000 (oldest deadline 2024-03-23T23:15:51.270414+0000) 2024-03-23T23:16:06.059 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:05 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:05.779+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.271537+0000 front 2024-03-23T23:15:31.271450+0000 (oldest deadline 2024-03-23T23:15:55.371129+0000) 2024-03-23T23:16:06.059 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:05 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:05.779+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-03-23T23:15:41.272075+0000 front 2024-03-23T23:15:41.272162+0000 (oldest deadline 2024-03-23T23:16:05.371919+0000) 2024-03-23T23:16:06.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:06 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:06.008+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:16:06.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:06 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:06.008+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:16:06.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:06 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:06.008+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:16:06.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:06 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:06.008+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:16:06.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:06 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:06.008+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:16:06.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:06 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:06.008+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:22.349554+0000 front 2024-03-23T23:15:22.349467+0000 (oldest deadline 2024-03-23T23:15:47.049466+0000) 2024-03-23T23:16:06.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:06 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:06.008+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.750658+0000 front 2024-03-23T23:15:31.750777+0000 (oldest deadline 2024-03-23T23:15:55.250472+0000) 2024-03-23T23:16:07.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:06 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:06.734+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:16:07.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:06 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:06.734+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:16:07.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:06 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:06.734+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:16:07.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:06 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:06.734+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:16:07.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:06 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:06.734+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:25.370284+0000 front 2024-03-23T23:15:25.370438+0000 (oldest deadline 2024-03-23T23:15:51.270414+0000) 2024-03-23T23:16:07.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:06 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:06.734+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.271537+0000 front 2024-03-23T23:15:31.271450+0000 (oldest deadline 2024-03-23T23:15:55.371129+0000) 2024-03-23T23:16:07.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:06 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:06.734+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-03-23T23:15:41.272075+0000 front 2024-03-23T23:15:41.272162+0000 (oldest deadline 2024-03-23T23:16:05.371919+0000) 2024-03-23T23:16:07.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:06 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:06.960+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:16:07.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:06 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:06.960+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:16:07.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:06 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:06.960+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:16:07.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:06 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:06.960+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:16:07.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:06 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:06.960+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:16:07.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:06 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:06.960+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:22.349554+0000 front 2024-03-23T23:15:22.349467+0000 (oldest deadline 2024-03-23T23:15:47.049466+0000) 2024-03-23T23:16:07.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:06 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:06.960+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.750658+0000 front 2024-03-23T23:15:31.750777+0000 (oldest deadline 2024-03-23T23:15:55.250472+0000) 2024-03-23T23:16:08.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:07 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:07.719+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:16:08.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:07 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:07.719+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:16:08.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:07 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:07.719+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:16:08.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:07 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:07.719+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:16:08.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:07 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:07.719+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:25.370284+0000 front 2024-03-23T23:15:25.370438+0000 (oldest deadline 2024-03-23T23:15:51.270414+0000) 2024-03-23T23:16:08.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:07 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:07.719+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.271537+0000 front 2024-03-23T23:15:31.271450+0000 (oldest deadline 2024-03-23T23:15:55.371129+0000) 2024-03-23T23:16:08.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:07 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:07.719+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-03-23T23:15:41.272075+0000 front 2024-03-23T23:15:41.272162+0000 (oldest deadline 2024-03-23T23:16:05.371919+0000) 2024-03-23T23:16:08.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:07 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:07.921+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:39.441892+0000 front 2024-03-23T23:14:39.442017+0000 (oldest deadline 2024-03-23T23:15:04.741836+0000) 2024-03-23T23:16:08.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:07 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:07.921+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:50.543286+0000 front 2024-03-23T23:14:50.543330+0000 (oldest deadline 2024-03-23T23:15:14.043366+0000) 2024-03-23T23:16:08.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:07 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:07.921+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.544536+0000 front 2024-03-23T23:14:57.544388+0000 (oldest deadline 2024-03-23T23:15:22.844410+0000) 2024-03-23T23:16:08.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:07 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:07.921+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-23T23:15:06.047138+0000 front 2024-03-23T23:15:06.047018+0000 (oldest deadline 2024-03-23T23:15:29.546852+0000) 2024-03-23T23:16:08.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:07 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:07.921+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:17.148234+0000 front 2024-03-23T23:15:17.148248+0000 (oldest deadline 2024-03-23T23:15:40.048304+0000) 2024-03-23T23:16:08.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:07 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:07.921+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:22.349554+0000 front 2024-03-23T23:15:22.349467+0000 (oldest deadline 2024-03-23T23:15:47.049466+0000) 2024-03-23T23:16:08.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:07 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10[61002]: 2024-03-23T23:16:07.921+0000 7f740d5e8700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.750658+0000 front 2024-03-23T23:15:31.750777+0000 (oldest deadline 2024-03-23T23:15:55.250472+0000) 2024-03-23T23:16:08.868 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:08 smithi173 podman[72076]: 2024-03-23 23:16:08.601342504 +0000 UTC m=+5.640358901 container died 8c751c7e3151b4aa58f1d2d7a91285f2c9afd4d75bae3ea1cf36c7e27bc25e5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True) 2024-03-23T23:16:09.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:08 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:08.713+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:16:09.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:08 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:08.713+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:16:09.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:08 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:08.713+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:16:09.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:08 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:08.713+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:16:09.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:08 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:08.713+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:25.370284+0000 front 2024-03-23T23:15:25.370438+0000 (oldest deadline 2024-03-23T23:15:51.270414+0000) 2024-03-23T23:16:09.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:08 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:08.713+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.271537+0000 front 2024-03-23T23:15:31.271450+0000 (oldest deadline 2024-03-23T23:15:55.371129+0000) 2024-03-23T23:16:09.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:08 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:08.713+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-03-23T23:15:41.272075+0000 front 2024-03-23T23:15:41.272162+0000 (oldest deadline 2024-03-23T23:16:05.371919+0000) 2024-03-23T23:16:09.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:09 smithi173 podman[72076]: 2024-03-23 23:16:09.037436857 +0000 UTC m=+6.076453251 container cleanup 8c751c7e3151b4aa58f1d2d7a91285f2c9afd4d75bae3ea1cf36c7e27bc25e5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-23T23:16:09.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:09 smithi173 bash[72076]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10 2024-03-23T23:16:09.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:09 smithi173 podman[72105]: 2024-03-23 23:16:09.205267211 +0000 UTC m=+0.600809893 container remove 8c751c7e3151b4aa58f1d2d7a91285f2c9afd4d75bae3ea1cf36c7e27bc25e5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-23T23:16:09.868 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:09 smithi173 podman[72197]: 2024-03-23 23:16:09.598096218 +0000 UTC m=+0.113817887 container create f2604e6d88df023993b5a30133e0b7f3f4c66273e47b936cf893ec3d11378678 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-23T23:16:09.868 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:09 smithi173 podman[72197]: 2024-03-23 23:16:09.501543974 +0000 UTC m=+0.017265618 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-23T23:16:09.868 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:09 smithi173 podman[72197]: 2024-03-23 23:16:09.808894999 +0000 UTC m=+0.324616645 container init f2604e6d88df023993b5a30133e0b7f3f4c66273e47b936cf893ec3d11378678 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10-deactivate, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-23T23:16:09.868 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:09 smithi173 podman[72197]: 2024-03-23 23:16:09.811886066 +0000 UTC m=+0.327607698 container start f2604e6d88df023993b5a30133e0b7f3f4c66273e47b936cf893ec3d11378678 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-23T23:16:10.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:09 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:09.707+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:16:10.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:09 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:09.707+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:16:10.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:09 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:09.707+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:16:10.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:09 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:09.707+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:16:10.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:09 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:09.707+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:25.370284+0000 front 2024-03-23T23:15:25.370438+0000 (oldest deadline 2024-03-23T23:15:51.270414+0000) 2024-03-23T23:16:10.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:09 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:09.707+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.271537+0000 front 2024-03-23T23:15:31.271450+0000 (oldest deadline 2024-03-23T23:15:55.371129+0000) 2024-03-23T23:16:10.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:09 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:09.707+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-03-23T23:15:41.272075+0000 front 2024-03-23T23:15:41.272162+0000 (oldest deadline 2024-03-23T23:16:05.371919+0000) 2024-03-23T23:16:10.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:09 smithi173 podman[72197]: 2024-03-23 23:16:09.875377381 +0000 UTC m=+0.391099022 container attach f2604e6d88df023993b5a30133e0b7f3f4c66273e47b936cf893ec3d11378678 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-23T23:16:10.368 INFO:journalctl@ceph.osd.10.smithi173.stdout:Mar 23 23:16:10 smithi173 podman[72197]: 2024-03-23 23:16:10.097648471 +0000 UTC m=+0.613370102 container died f2604e6d88df023993b5a30133e0b7f3f4c66273e47b936cf893ec3d11378678 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-10-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-23T23:16:10.675 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.10.service' 2024-03-23T23:16:10.718 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:16:10.718 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-23T23:16:10.718 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-23T23:16:10.718 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.11 2024-03-23T23:16:11.030 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:10 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:10.680+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:16:11.030 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:10 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:10.680+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:16:11.030 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:10 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:10.680+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:16:11.030 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:10 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:10.680+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:16:11.030 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:10 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:10.680+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:25.370284+0000 front 2024-03-23T23:15:25.370438+0000 (oldest deadline 2024-03-23T23:15:51.270414+0000) 2024-03-23T23:16:11.030 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:10 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:10.680+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.271537+0000 front 2024-03-23T23:15:31.271450+0000 (oldest deadline 2024-03-23T23:15:55.371129+0000) 2024-03-23T23:16:11.030 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:10 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:10.680+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-03-23T23:15:41.272075+0000 front 2024-03-23T23:15:41.272162+0000 (oldest deadline 2024-03-23T23:16:05.371919+0000) 2024-03-23T23:16:11.030 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:10 smithi203 systemd[1]: Stopping Ceph osd.11 for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:16:11.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:11 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:11.030+0000 7f3320036700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:16:11.407 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:11 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:11.030+0000 7f3320036700 -1 osd.11 91 *** Got signal Terminated *** 2024-03-23T23:16:11.408 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:11 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:11.030+0000 7f3320036700 -1 osd.11 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-23T23:16:11.907 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:11 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:11.647+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:16:11.907 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:11 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:11.647+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:16:11.907 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:11 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:11.647+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:16:11.907 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:11 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:11.647+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:16:11.908 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:11 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:11.647+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:25.370284+0000 front 2024-03-23T23:15:25.370438+0000 (oldest deadline 2024-03-23T23:15:51.270414+0000) 2024-03-23T23:16:11.908 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:11 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:11.647+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.271537+0000 front 2024-03-23T23:15:31.271450+0000 (oldest deadline 2024-03-23T23:15:55.371129+0000) 2024-03-23T23:16:11.908 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:11 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:11.647+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-03-23T23:15:41.272075+0000 front 2024-03-23T23:15:41.272162+0000 (oldest deadline 2024-03-23T23:16:05.371919+0000) 2024-03-23T23:16:13.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:12 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:12.687+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:16:13.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:12 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:12.687+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:16:13.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:12 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:12.687+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:16:13.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:12 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:12.687+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:16:13.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:12 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:12.687+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:25.370284+0000 front 2024-03-23T23:15:25.370438+0000 (oldest deadline 2024-03-23T23:15:51.270414+0000) 2024-03-23T23:16:13.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:12 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:12.687+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.271537+0000 front 2024-03-23T23:15:31.271450+0000 (oldest deadline 2024-03-23T23:15:55.371129+0000) 2024-03-23T23:16:13.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:12 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:12.687+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-03-23T23:15:41.272075+0000 front 2024-03-23T23:15:41.272162+0000 (oldest deadline 2024-03-23T23:16:05.371919+0000) 2024-03-23T23:16:14.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:13 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:13.666+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:16:14.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:13 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:13.666+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:16:14.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:13 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:13.666+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:16:14.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:13 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:13.666+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:16:14.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:13 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:13.666+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:25.370284+0000 front 2024-03-23T23:15:25.370438+0000 (oldest deadline 2024-03-23T23:15:51.270414+0000) 2024-03-23T23:16:14.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:13 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:13.666+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.271537+0000 front 2024-03-23T23:15:31.271450+0000 (oldest deadline 2024-03-23T23:15:55.371129+0000) 2024-03-23T23:16:14.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:13 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:13.666+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-03-23T23:15:41.272075+0000 front 2024-03-23T23:15:41.272162+0000 (oldest deadline 2024-03-23T23:16:05.371919+0000) 2024-03-23T23:16:14.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:13 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:13.666+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6820 osd.8 since back 2024-03-23T23:15:48.173587+0000 front 2024-03-23T23:15:48.173640+0000 (oldest deadline 2024-03-23T23:16:13.473758+0000) 2024-03-23T23:16:15.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:14 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:14.679+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:16:15.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:14 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:14.679+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:16:15.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:14 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:14.679+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:16:15.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:14 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:14.679+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:16:15.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:14 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:14.679+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:25.370284+0000 front 2024-03-23T23:15:25.370438+0000 (oldest deadline 2024-03-23T23:15:51.270414+0000) 2024-03-23T23:16:15.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:14 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:14.679+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.271537+0000 front 2024-03-23T23:15:31.271450+0000 (oldest deadline 2024-03-23T23:15:55.371129+0000) 2024-03-23T23:16:15.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:14 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:14.679+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-03-23T23:15:41.272075+0000 front 2024-03-23T23:15:41.272162+0000 (oldest deadline 2024-03-23T23:16:05.371919+0000) 2024-03-23T23:16:15.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:14 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:14.679+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6820 osd.8 since back 2024-03-23T23:15:48.173587+0000 front 2024-03-23T23:15:48.173640+0000 (oldest deadline 2024-03-23T23:16:13.473758+0000) 2024-03-23T23:16:16.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:15 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:15.662+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-23T23:14:40.464279+0000 front 2024-03-23T23:14:40.464196+0000 (oldest deadline 2024-03-23T23:15:06.864690+0000) 2024-03-23T23:16:16.157 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:15 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:15.662+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-03-23T23:14:51.466110+0000 front 2024-03-23T23:14:51.466248+0000 (oldest deadline 2024-03-23T23:15:17.366013+0000) 2024-03-23T23:16:16.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:15 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:15.662+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-23T23:14:57.366480+0000 front 2024-03-23T23:14:57.366454+0000 (oldest deadline 2024-03-23T23:15:23.266526+0000) 2024-03-23T23:16:16.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:15 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:15.662+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-03-23T23:15:15.468872+0000 front 2024-03-23T23:15:15.468900+0000 (oldest deadline 2024-03-23T23:15:41.369013+0000) 2024-03-23T23:16:16.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:15 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:15.662+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-03-23T23:15:25.370284+0000 front 2024-03-23T23:15:25.370438+0000 (oldest deadline 2024-03-23T23:15:51.270414+0000) 2024-03-23T23:16:16.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:15 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:15.662+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-23T23:15:31.271537+0000 front 2024-03-23T23:15:31.271450+0000 (oldest deadline 2024-03-23T23:15:55.371129+0000) 2024-03-23T23:16:16.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:15 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:15.662+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-03-23T23:15:41.272075+0000 front 2024-03-23T23:15:41.272162+0000 (oldest deadline 2024-03-23T23:16:05.371919+0000) 2024-03-23T23:16:16.158 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:15 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11[62927]: 2024-03-23T23:16:15.662+0000 7f3318622700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.203:6820 osd.8 since back 2024-03-23T23:15:48.173587+0000 front 2024-03-23T23:15:48.173640+0000 (oldest deadline 2024-03-23T23:16:13.473758+0000) 2024-03-23T23:16:16.731 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:16 smithi203 podman[74901]: 2024-03-23 23:16:16.480635833 +0000 UTC m=+5.566995820 container died 7f3bc901a4527034fd7c0a4c1d15cd2088926d3f869857c1569def9b9780a706 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-23T23:16:17.014 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:16 smithi203 podman[74901]: 2024-03-23 23:16:16.822307519 +0000 UTC m=+5.908667503 container cleanup 7f3bc901a4527034fd7c0a4c1d15cd2088926d3f869857c1569def9b9780a706 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD) 2024-03-23T23:16:17.014 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:16 smithi203 bash[74901]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11 2024-03-23T23:16:17.336 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:17 smithi203 podman[74918]: 2024-03-23 23:16:17.015675929 +0000 UTC m=+0.536095586 container remove 7f3bc901a4527034fd7c0a4c1d15cd2088926d3f869857c1569def9b9780a706 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-03-23T23:16:17.657 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:17 smithi203 podman[75012]: 2024-03-23 23:16:17.336063459 +0000 UTC m=+0.022132539 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-23T23:16:17.657 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:17 smithi203 podman[75012]: 2024-03-23 23:16:17.433252848 +0000 UTC m=+0.119321973 container create 9e913d1622c8c034d657cec2ab2ace04f8076059feaa979b00e82432e518ce4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-23T23:16:17.950 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:17 smithi203 podman[75012]: 2024-03-23 23:16:17.660706942 +0000 UTC m=+0.346776022 container init 9e913d1622c8c034d657cec2ab2ace04f8076059feaa979b00e82432e518ce4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-23T23:16:17.950 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:17 smithi203 podman[75012]: 2024-03-23 23:16:17.663733879 +0000 UTC m=+0.349802968 container start 9e913d1622c8c034d657cec2ab2ace04f8076059feaa979b00e82432e518ce4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-23T23:16:17.950 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:17 smithi203 podman[75012]: 2024-03-23 23:16:17.744152506 +0000 UTC m=+0.430221613 container attach 9e913d1622c8c034d657cec2ab2ace04f8076059feaa979b00e82432e518ce4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11-deactivate, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-23T23:16:18.262 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:17 smithi203 podman[75012]: 2024-03-23 23:16:17.951526765 +0000 UTC m=+0.637595864 container died 9e913d1622c8c034d657cec2ab2ace04f8076059feaa979b00e82432e518ce4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-23T23:16:18.498 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.11.service' 2024-03-23T23:16:18.523 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:18 smithi203 podman[75038]: 2024-03-23 23:16:18.472423032 +0000 UTC m=+0.515675809 container remove 9e913d1622c8c034d657cec2ab2ace04f8076059feaa979b00e82432e518ce4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-osd-11-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-23T23:16:18.523 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:18 smithi203 systemd[1]: ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.11.service: Deactivated successfully. 2024-03-23T23:16:18.523 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:18 smithi203 systemd[1]: Stopped Ceph osd.11 for 90f679ae-e967-11ee-95cd-87774f69a715. 2024-03-23T23:16:18.523 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 23 23:16:18 smithi203 systemd[1]: ceph-90f679ae-e967-11ee-95cd-87774f69a715@osd.11.service: Consumed 7.293s CPU time. 2024-03-23T23:16:18.537 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:16:18.538 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-23T23:16:18.538 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-23T23:16:18.538 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.a 2024-03-23T23:16:19.053 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 23 23:16:18 smithi012 systemd[1]: Stopping Ceph mds.a for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:16:19.553 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 23 23:16:19 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-a[73383]: 2024-03-23T23:16:19.132+0000 7f4e9da00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:16:19.554 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 23 23:16:19 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-a[73383]: 2024-03-23T23:16:19.132+0000 7f4e9da00700 -1 mds.a *** got signal Terminated *** 2024-03-23T23:16:22.304 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 23 23:16:22 smithi012 podman[101604]: 2024-03-23 23:16:22.012939812 +0000 UTC m=+3.028687131 container died 48a7fe6e2e688de9b2609ed0dcb9f229aa5a3b1aaca668b80000aaf350281e7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-23T23:16:22.709 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 23 23:16:22 smithi012 podman[101604]: 2024-03-23 23:16:22.433609221 +0000 UTC m=+3.449356588 container cleanup 48a7fe6e2e688de9b2609ed0dcb9f229aa5a3b1aaca668b80000aaf350281e7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-a, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-23T23:16:22.709 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 23 23:16:22 smithi012 bash[101604]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-a 2024-03-23T23:16:22.979 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 23 23:16:22 smithi012 podman[101617]: 2024-03-23 23:16:22.709676622 +0000 UTC m=+0.680410457 container remove 48a7fe6e2e688de9b2609ed0dcb9f229aa5a3b1aaca668b80000aaf350281e7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-a, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-23T23:16:23.197 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.a.service' 2024-03-23T23:16:23.287 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 23 23:16:23 smithi012 systemd[1]: ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.a.service: Deactivated successfully. 2024-03-23T23:16:23.287 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 23 23:16:23 smithi012 systemd[1]: Stopped Ceph mds.a for 90f679ae-e967-11ee-95cd-87774f69a715. 2024-03-23T23:16:23.287 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 23 23:16:23 smithi012 systemd[1]: ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.a.service: Consumed 2.470s CPU time. 2024-03-23T23:16:23.323 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:16:23.324 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-23T23:16:23.324 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-23T23:16:23.324 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.d 2024-03-23T23:16:23.554 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 23 23:16:23 smithi012 systemd[1]: Stopping Ceph mds.d for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:16:24.303 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 23 23:16:23 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-d[72375]: 2024-03-23T23:16:23.903+0000 7f8731000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:16:24.303 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 23 23:16:23 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-d[72375]: 2024-03-23T23:16:23.903+0000 7f8731000700 -1 mds.d *** got signal Terminated *** 2024-03-23T23:16:27.457 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 23 23:16:27 smithi012 podman[101723]: 2024-03-23 23:16:27.187049647 +0000 UTC m=+3.469591108 container died c3543e91c6973124fb066feeea8aae11a08085976248c01e0d90b586133cd836 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-d, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5) 2024-03-23T23:16:27.803 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 23 23:16:27 smithi012 podman[101723]: 2024-03-23 23:16:27.562065859 +0000 UTC m=+3.844607334 container cleanup c3543e91c6973124fb066feeea8aae11a08085976248c01e0d90b586133cd836 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-d, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-23T23:16:27.804 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 23 23:16:27 smithi012 bash[101723]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-d 2024-03-23T23:16:27.804 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 23 23:16:27 smithi012 podman[101737]: 2024-03-23 23:16:27.704720017 +0000 UTC m=+0.496111616 container remove c3543e91c6973124fb066feeea8aae11a08085976248c01e0d90b586133cd836 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-23T23:16:28.241 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.d.service' 2024-03-23T23:16:28.362 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:16:28.362 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-23T23:16:28.362 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-23T23:16:28.363 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.g 2024-03-23T23:16:28.803 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 23 23:16:28 smithi012 systemd[1]: Stopping Ceph mds.g for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:16:29.303 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 23 23:16:29 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-g[73000]: 2024-03-23T23:16:29.004+0000 7ff1a7e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:16:29.303 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 23 23:16:29 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-g[73000]: 2024-03-23T23:16:29.004+0000 7ff1a7e00700 -1 mds.g *** got signal Terminated *** 2024-03-23T23:16:33.303 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 23 23:16:33 smithi012 podman[101842]: 2024-03-23 23:16:33.018348191 +0000 UTC m=+4.180952102 container died b3ca6f1c01d5cbeefe4069601c8a0970d6063de86a7e8337aa65eb5259845e2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-g, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-23T23:16:33.756 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 23 23:16:33 smithi012 podman[101842]: 2024-03-23 23:16:33.431203809 +0000 UTC m=+4.593807717 container cleanup b3ca6f1c01d5cbeefe4069601c8a0970d6063de86a7e8337aa65eb5259845e2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-g, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-23T23:16:33.757 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 23 23:16:33 smithi012 bash[101842]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-g 2024-03-23T23:16:34.024 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 23 23:16:33 smithi012 podman[101856]: 2024-03-23 23:16:33.757411322 +0000 UTC m=+0.715962352 container remove b3ca6f1c01d5cbeefe4069601c8a0970d6063de86a7e8337aa65eb5259845e2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-g, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-23T23:16:34.051 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.g.service' 2024-03-23T23:16:34.166 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:16:34.166 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-23T23:16:34.166 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-23T23:16:34.166 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.j 2024-03-23T23:16:34.803 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 23 23:16:34 smithi012 systemd[1]: Stopping Ceph mds.j for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:16:35.303 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 23 23:16:34 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-j[72010]: 2024-03-23T23:16:34.849+0000 7f4774400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:16:35.304 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 23 23:16:34 smithi012 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-j[72010]: 2024-03-23T23:16:34.849+0000 7f4774400700 -1 mds.j *** got signal Terminated *** 2024-03-23T23:16:39.129 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 23 23:16:38 smithi012 podman[101962]: 2024-03-23 23:16:38.866695857 +0000 UTC m=+4.151089379 container died ff60b87f6b4183a1b8d352dc4aa688ea51122a4ed0dc3d8e026d401ef3f1b19e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-j, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_CLEAN=True) 2024-03-23T23:16:39.452 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 23 23:16:39 smithi012 podman[101962]: 2024-03-23 23:16:39.226069735 +0000 UTC m=+4.510463277 container cleanup ff60b87f6b4183a1b8d352dc4aa688ea51122a4ed0dc3d8e026d401ef3f1b19e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-j, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-23T23:16:39.453 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 23 23:16:39 smithi012 bash[101962]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-j 2024-03-23T23:16:39.803 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 23 23:16:39 smithi012 podman[101988]: 2024-03-23 23:16:39.451971674 +0000 UTC m=+0.562786188 container remove ff60b87f6b4183a1b8d352dc4aa688ea51122a4ed0dc3d8e026d401ef3f1b19e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-j, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-23T23:16:39.969 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.j.service' 2024-03-23T23:16:40.096 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:16:40.096 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-23T23:16:40.096 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-23T23:16:40.097 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.b 2024-03-23T23:16:40.378 INFO:journalctl@ceph.mds.b.smithi173.stdout:Mar 23 23:16:40 smithi173 systemd[1]: Stopping Ceph mds.b for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:16:40.867 INFO:journalctl@ceph.mds.b.smithi173.stdout:Mar 23 23:16:40 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-b[66540]: 2024-03-23T23:16:40.378+0000 7f55c96ac700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:16:40.868 INFO:journalctl@ceph.mds.b.smithi173.stdout:Mar 23 23:16:40 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-b[66540]: 2024-03-23T23:16:40.378+0000 7f55c96ac700 -1 mds.b *** got signal Terminated *** 2024-03-23T23:16:44.695 INFO:journalctl@ceph.mds.b.smithi173.stdout:Mar 23 23:16:44 smithi173 podman[72302]: 2024-03-23 23:16:44.386850513 +0000 UTC m=+4.093036143 container died 7d92fbafbf17eb5156105fe9749d3a5ec7d0ca0d869c4d4ef0fc43496d50ccf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-b, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-23T23:16:45.012 INFO:journalctl@ceph.mds.b.smithi173.stdout:Mar 23 23:16:44 smithi173 podman[72302]: 2024-03-23 23:16:44.753702978 +0000 UTC m=+4.459888590 container cleanup 7d92fbafbf17eb5156105fe9749d3a5ec7d0ca0d869c4d4ef0fc43496d50ccf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-23T23:16:45.012 INFO:journalctl@ceph.mds.b.smithi173.stdout:Mar 23 23:16:44 smithi173 bash[72302]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-b 2024-03-23T23:16:45.186 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.b.service' 2024-03-23T23:16:45.224 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:16:45.224 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-23T23:16:45.224 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-23T23:16:45.224 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.e 2024-03-23T23:16:45.618 INFO:journalctl@ceph.mds.e.smithi173.stdout:Mar 23 23:16:45 smithi173 systemd[1]: Stopping Ceph mds.e for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:16:45.619 INFO:journalctl@ceph.mds.e.smithi173.stdout:Mar 23 23:16:45 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-e[65912]: 2024-03-23T23:16:45.531+0000 7fce2eadd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:16:45.619 INFO:journalctl@ceph.mds.e.smithi173.stdout:Mar 23 23:16:45 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-e[65912]: 2024-03-23T23:16:45.531+0000 7fce2eadd700 -1 mds.e *** got signal Terminated *** 2024-03-23T23:16:49.117 INFO:journalctl@ceph.mds.e.smithi173.stdout:Mar 23 23:16:48 smithi173 podman[72435]: 2024-03-23 23:16:48.812822079 +0000 UTC m=+3.378071179 container died 8270dbda57bfa0eb45fe09f46a9731e8ee776337790f296917aad80cc7bba999 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-e, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True) 2024-03-23T23:16:49.449 INFO:journalctl@ceph.mds.e.smithi173.stdout:Mar 23 23:16:49 smithi173 podman[72435]: 2024-03-23 23:16:49.199446825 +0000 UTC m=+3.764695926 container cleanup 8270dbda57bfa0eb45fe09f46a9731e8ee776337790f296917aad80cc7bba999 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-23T23:16:49.449 INFO:journalctl@ceph.mds.e.smithi173.stdout:Mar 23 23:16:49 smithi173 bash[72435]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-e 2024-03-23T23:16:49.616 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.e.service' 2024-03-23T23:16:49.652 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:16:49.653 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-23T23:16:49.653 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-23T23:16:49.653 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.h 2024-03-23T23:16:49.977 INFO:journalctl@ceph.mds.h.smithi173.stdout:Mar 23 23:16:49 smithi173 systemd[1]: Stopping Ceph mds.h for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:16:50.367 INFO:journalctl@ceph.mds.h.smithi173.stdout:Mar 23 23:16:49 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-h[66230]: 2024-03-23T23:16:49.977+0000 7f8a0e167700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:16:50.368 INFO:journalctl@ceph.mds.h.smithi173.stdout:Mar 23 23:16:49 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-h[66230]: 2024-03-23T23:16:49.977+0000 7f8a0e167700 -1 mds.h *** got signal Terminated *** 2024-03-23T23:16:53.118 INFO:journalctl@ceph.mds.h.smithi173.stdout:Mar 23 23:16:52 smithi173 podman[72557]: 2024-03-23 23:16:52.824812876 +0000 UTC m=+2.953979399 container died e4b852201291c8fff7f085c94186afbee0393f16c3a32f50be20b3f60a2b8f8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-h, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-23T23:16:53.613 INFO:journalctl@ceph.mds.h.smithi173.stdout:Mar 23 23:16:53 smithi173 podman[72557]: 2024-03-23 23:16:53.262285962 +0000 UTC m=+3.391452458 container cleanup e4b852201291c8fff7f085c94186afbee0393f16c3a32f50be20b3f60a2b8f8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-h, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-23T23:16:53.613 INFO:journalctl@ceph.mds.h.smithi173.stdout:Mar 23 23:16:53 smithi173 bash[72557]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-h 2024-03-23T23:16:53.613 INFO:journalctl@ceph.mds.h.smithi173.stdout:Mar 23 23:16:53 smithi173 podman[72569]: 2024-03-23 23:16:53.423613857 +0000 UTC m=+0.597025252 container remove e4b852201291c8fff7f085c94186afbee0393f16c3a32f50be20b3f60a2b8f8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD) 2024-03-23T23:16:53.634 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.h.service' 2024-03-23T23:16:53.669 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:16:53.669 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-23T23:16:53.669 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-23T23:16:53.669 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.k 2024-03-23T23:16:53.992 INFO:journalctl@ceph.mds.k.smithi173.stdout:Mar 23 23:16:53 smithi173 systemd[1]: Stopping Ceph mds.k for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:16:54.367 INFO:journalctl@ceph.mds.k.smithi173.stdout:Mar 23 23:16:53 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-k[65589]: 2024-03-23T23:16:53.992+0000 7f62bc95c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:16:54.368 INFO:journalctl@ceph.mds.k.smithi173.stdout:Mar 23 23:16:53 smithi173 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-k[65589]: 2024-03-23T23:16:53.992+0000 7f62bc95c700 -1 mds.k *** got signal Terminated *** 2024-03-23T23:16:55.234 INFO:journalctl@ceph.mds.k.smithi173.stdout:Mar 23 23:16:54 smithi173 podman[72680]: 2024-03-23 23:16:54.95648025 +0000 UTC m=+1.071004551 container died 05355527fbafc9e98c2f88a6d1e3cd53f1370c212966536547f8f65f1aeaab69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-23T23:16:55.601 INFO:journalctl@ceph.mds.k.smithi173.stdout:Mar 23 23:16:55 smithi173 podman[72680]: 2024-03-23 23:16:55.318578186 +0000 UTC m=+1.433102491 container cleanup 05355527fbafc9e98c2f88a6d1e3cd53f1370c212966536547f8f65f1aeaab69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-k, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-23T23:16:55.602 INFO:journalctl@ceph.mds.k.smithi173.stdout:Mar 23 23:16:55 smithi173 bash[72680]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-k 2024-03-23T23:16:55.743 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.k.service' 2024-03-23T23:16:55.777 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:16:55.777 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-23T23:16:55.777 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-23T23:16:55.777 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.c 2024-03-23T23:16:56.079 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 23 23:16:55 smithi203 systemd[1]: Stopping Ceph mds.c for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:16:56.407 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 23 23:16:56 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-c[68402]: 2024-03-23T23:16:56.079+0000 7f7f9542f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:16:56.407 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 23 23:16:56 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-c[68402]: 2024-03-23T23:16:56.079+0000 7f7f9542f700 -1 mds.c *** got signal Terminated *** 2024-03-23T23:17:00.346 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 23 23:17:00 smithi203 podman[75118]: 2024-03-23 23:17:00.088216985 +0000 UTC m=+4.096232859 container died a9182ac7da09ceec89ee9484f594932e22a4b2bb2a250af084a7e37ce0015167 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-c, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0) 2024-03-23T23:17:00.612 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 23 23:17:00 smithi203 podman[75118]: 2024-03-23 23:17:00.42937679 +0000 UTC m=+4.437392628 container cleanup a9182ac7da09ceec89ee9484f594932e22a4b2bb2a250af084a7e37ce0015167 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-c, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-23T23:17:00.612 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 23 23:17:00 smithi203 bash[75118]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-c 2024-03-23T23:17:00.848 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.c.service' 2024-03-23T23:17:00.873 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 23 23:17:00 smithi203 podman[75143]: 2024-03-23 23:17:00.61410978 +0000 UTC m=+0.519268170 container remove a9182ac7da09ceec89ee9484f594932e22a4b2bb2a250af084a7e37ce0015167 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-23T23:17:00.873 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 23 23:17:00 smithi203 systemd[1]: ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.c.service: Deactivated successfully. 2024-03-23T23:17:00.873 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 23 23:17:00 smithi203 systemd[1]: Stopped Ceph mds.c for 90f679ae-e967-11ee-95cd-87774f69a715. 2024-03-23T23:17:00.873 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 23 23:17:00 smithi203 systemd[1]: ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.c.service: Consumed 2.497s CPU time. 2024-03-23T23:17:00.887 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:17:00.887 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-23T23:17:00.887 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-23T23:17:00.887 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.f 2024-03-23T23:17:01.157 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 23 23:17:00 smithi203 systemd[1]: Stopping Ceph mds.f for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:17:01.657 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 23 23:17:01 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-f[68083]: 2024-03-23T23:17:01.223+0000 7f8f0d454700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:17:01.657 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 23 23:17:01 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-f[68083]: 2024-03-23T23:17:01.223+0000 7f8f0d454700 -1 mds.f *** got signal Terminated *** 2024-03-23T23:17:05.091 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 23 23:17:04 smithi203 podman[75249]: 2024-03-23 23:17:04.832212776 +0000 UTC m=+3.727472083 container died 58f37a969a279d3b72419cc564b453853153f6b47b750d1c1c637e767826c389 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-f, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-23T23:17:05.407 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 23 23:17:05 smithi203 podman[75249]: 2024-03-23 23:17:05.183326832 +0000 UTC m=+4.078586163 container cleanup 58f37a969a279d3b72419cc564b453853153f6b47b750d1c1c637e767826c389 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-f, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-23T23:17:05.407 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 23 23:17:05 smithi203 bash[75249]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-f 2024-03-23T23:17:05.746 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.f.service' 2024-03-23T23:17:05.781 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:17:05.782 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-23T23:17:05.782 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-23T23:17:05.782 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.i 2024-03-23T23:17:06.119 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 23 23:17:05 smithi203 systemd[1]: Stopping Ceph mds.i for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:17:06.407 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 23 23:17:06 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-i[68710]: 2024-03-23T23:17:06.120+0000 7fc03e0f9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:17:06.407 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 23 23:17:06 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-i[68710]: 2024-03-23T23:17:06.120+0000 7fc03e0f9700 -1 mds.i *** got signal Terminated *** 2024-03-23T23:17:09.145 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 23 23:17:08 smithi203 podman[75371]: 2024-03-23 23:17:08.835782359 +0000 UTC m=+2.837861213 container died 92d4d344701bdee28505373753be3365f944490888263d351c5f4f4858d2b310 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-i, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux ) 2024-03-23T23:17:09.407 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 23 23:17:09 smithi203 podman[75371]: 2024-03-23 23:17:09.237859828 +0000 UTC m=+3.239938675 container cleanup 92d4d344701bdee28505373753be3365f944490888263d351c5f4f4858d2b310 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-i, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-23T23:17:09.407 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 23 23:17:09 smithi203 bash[75371]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-i 2024-03-23T23:17:09.597 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.i.service' 2024-03-23T23:17:09.631 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:17:09.631 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-23T23:17:09.632 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-23T23:17:09.632 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.l 2024-03-23T23:17:09.907 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 23 23:17:09 smithi203 systemd[1]: Stopping Ceph mds.l for 90f679ae-e967-11ee-95cd-87774f69a715... 2024-03-23T23:17:10.407 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 23 23:17:09 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-l[67776]: 2024-03-23T23:17:09.949+0000 7fb99b9b5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T23:17:10.407 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 23 23:17:09 smithi203 ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-l[67776]: 2024-03-23T23:17:09.949+0000 7fb99b9b5700 -1 mds.l *** got signal Terminated *** 2024-03-23T23:17:12.299 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 23 23:17:12 smithi203 podman[75492]: 2024-03-23 23:17:12.033919139 +0000 UTC m=+2.187727139 container stop 15b8198a9551323425dec04f47fa99f35d5b7d17c1450aa3c2d90f61c4e25b5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-l, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-23T23:17:12.299 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 23 23:17:12 smithi203 podman[75492]: 2024-03-23 23:17:12.059527793 +0000 UTC m=+2.213335793 container died 15b8198a9551323425dec04f47fa99f35d5b7d17c1450aa3c2d90f61c4e25b5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-l, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-23T23:17:12.657 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 23 23:17:12 smithi203 podman[75492]: 2024-03-23 23:17:12.358230799 +0000 UTC m=+2.512038797 container cleanup 15b8198a9551323425dec04f47fa99f35d5b7d17c1450aa3c2d90f61c4e25b5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-23T23:17:12.657 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 23 23:17:12 smithi203 bash[75492]: ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-l 2024-03-23T23:17:12.658 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 23 23:17:12 smithi203 podman[75506]: 2024-03-23 23:17:12.526084316 +0000 UTC m=+0.482092440 container remove 15b8198a9551323425dec04f47fa99f35d5b7d17c1450aa3c2d90f61c4e25b5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-90f679ae-e967-11ee-95cd-87774f69a715-mds-l, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-23T23:17:12.774 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-90f679ae-e967-11ee-95cd-87774f69a715@mds.l.service' 2024-03-23T23:17:12.808 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T23:17:12.808 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-23T23:17:12.808 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 90f679ae-e967-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-23T23:17:13.148 INFO:teuthology.orchestra.run.smithi012.stdout:Deleting cluster with fsid: 90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T23:17:18.238 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 90f679ae-e967-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-23T23:17:18.428 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: 90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T23:17:20.932 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 90f679ae-e967-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-23T23:17:21.124 INFO:teuthology.orchestra.run.smithi203.stdout:Deleting cluster with fsid: 90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T23:17:23.538 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-23T23:17:23.648 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-23T23:17:23.682 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-23T23:17:23.711 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-23T23:17:23.714 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616369/remote/smithi012/crash 2024-03-23T23:17:23.715 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/crash -- . 2024-03-23T23:17:23.824 INFO:teuthology.orchestra.run.smithi012.stderr:tar: /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-23T23:17:23.824 INFO:teuthology.orchestra.run.smithi012.stderr:tar: Error is not recoverable: exiting now 2024-03-23T23:17:23.830 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616369/remote/smithi173/crash 2024-03-23T23:17:23.830 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/crash -- . 2024-03-23T23:17:23.874 INFO:teuthology.orchestra.run.smithi173.stderr:tar: /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-23T23:17:23.874 INFO:teuthology.orchestra.run.smithi173.stderr:tar: Error is not recoverable: exiting now 2024-03-23T23:17:23.876 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616369/remote/smithi203/crash 2024-03-23T23:17:23.877 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/crash -- . 2024-03-23T23:17:23.914 INFO:teuthology.orchestra.run.smithi203.stderr:tar: /var/lib/ceph/90f679ae-e967-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-23T23:17:23.914 INFO:teuthology.orchestra.run.smithi203.stderr:tar: Error is not recoverable: exiting now 2024-03-23T23:17:23.915 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-23T23:17:23.916 DEBUG:teuthology.orchestra.run.smithi012:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-23T23:17:24.025 INFO:teuthology.orchestra.run.smithi012.stdout:2024-03-23T22:52:33.553104+0000 mon.a (mon.0) 188 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-23T23:17:24.026 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-23T23:17:24.027 DEBUG:teuthology.orchestra.run.smithi012:> sudo egrep '\[SEC\]' /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-23T23:17:24.124 DEBUG:teuthology.orchestra.run.smithi012:> sudo egrep '\[ERR\]' /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-23T23:17:24.218 DEBUG:teuthology.orchestra.run.smithi012:> sudo egrep '\[WRN\]' /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-23T23:17:24.308 INFO:teuthology.orchestra.run.smithi012.stdout:2024-03-23T22:52:33.553104+0000 mon.a (mon.0) 188 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-23T23:17:24.309 INFO:tasks.cephadm:Compressing logs... 2024-03-23T23:17:24.309 DEBUG:teuthology.orchestra.run.smithi012:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-23T23:17:24.314 DEBUG:teuthology.orchestra.run.smithi173:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-23T23:17:24.316 DEBUG:teuthology.orchestra.run.smithi203:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-23T23:17:24.348 INFO:teuthology.orchestra.run.smithi203.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-23T23:17:24.348 INFO:teuthology.orchestra.run.smithi203.stderr:: No such file or directory 2024-03-23T23:17:24.348 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.l.log 2024-03-23T23:17:24.348 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-23T23:17:24.348 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.i.log 2024-03-23T23:17:24.348 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.l.log: /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.11.log 2024-03-23T23:17:24.348 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mgr.z.log 2024-03-23T23:17:24.348 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.i.log: /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.8.log 2024-03-23T23:17:24.348 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mgr.z.log: gzip -5 92.4% --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.5.log 2024-03-23T23:17:24.348 INFO:teuthology.orchestra.run.smithi203.stderr: -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-23T23:17:24.348 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.f.log 2024-03-23T23:17:24.348 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-23T23:17:24.348 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.f.log: 90.4% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-23T23:17:24.349 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.log 2024-03-23T23:17:24.349 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.cephadm.log: 82.2% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-23T23:17:24.349 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-23T23:17:24.353 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.log: 94.7% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.l.log.gz 2024-03-23T23:17:24.353 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.c.log 2024-03-23T23:17:24.353 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.2.log: 87.9% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-23T23:17:24.353 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-23T23:17:24.354 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.c.log: 92.9% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mgr.z.log.gz 2024-03-23T23:17:24.354 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mon.c.log 2024-03-23T23:17:24.357 INFO:teuthology.orchestra.run.smithi173.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-23T23:17:24.357 INFO:teuthology.orchestra.run.smithi173.stderr:: No such file or directory 2024-03-23T23:17:24.357 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.10.log 2024-03-23T23:17:24.357 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-23T23:17:24.357 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.b.log 2024-03-23T23:17:24.357 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.4.log 2024-03-23T23:17:24.357 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-23T23:17:24.357 INFO:teuthology.orchestra.run.smithi173.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-23T23:17:24.357 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.k.log 2024-03-23T23:17:24.357 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.h.log 2024-03-23T23:17:24.358 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.7.log 2024-03-23T23:17:24.358 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.e.log 2024-03-23T23:17:24.358 INFO:teuthology.orchestra.run.smithi173.stderr: 90.4% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.audit.log.gz/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.7.log: 2024-03-23T23:17:24.358 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mgr.y.log 2024-03-23T23:17:24.358 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.e.log: 94.6% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.k.log.gz 2024-03-23T23:17:24.358 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-23T23:17:24.362 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mgr.y.log: 92.9% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mgr.y.log.gz 2024-03-23T23:17:24.362 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.log 2024-03-23T23:17:24.362 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.cephadm.log: 82.3% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-23T23:17:24.363 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-23T23:17:24.366 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mon.c.log: 90.9% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-23T23:17:24.367 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.log: 87.9% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-23T23:17:24.367 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mon.b.log 2024-03-23T23:17:24.381 INFO:teuthology.orchestra.run.smithi012.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-23T23:17:24.381 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-23T23:17:24.382 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mon.b.log: 90.9% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-23T23:17:24.382 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.6.log 2024-03-23T23:17:24.383 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mon.a.log 2024-03-23T23:17:24.384 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-23T23:17:24.385 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.g.log 2024-03-23T23:17:24.387 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.j.log 2024-03-23T23:17:24.388 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.3.log 2024-03-23T23:17:24.390 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mgr.x.log 2024-03-23T23:17:24.391 INFO:teuthology.orchestra.run.smithi012.stderr: 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-23T23:17:24.391 INFO:teuthology.orchestra.run.smithi012.stderr: 90.2% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-23T23:17:24.391 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-23T23:17:24.393 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-23T23:17:24.394 INFO:teuthology.orchestra.run.smithi012.stderr: 94.5% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.g.log.gz 2024-03-23T23:17:24.395 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.log 2024-03-23T23:17:24.397 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.cephadm.log: 82.6% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-23T23:17:24.397 INFO:teuthology.orchestra.run.smithi012.stderr: 94.5% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.j.log.gz 2024-03-23T23:17:24.397 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-23T23:17:24.398 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.9.log 2024-03-23T23:17:24.400 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.9.log: 87.8%gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.d.log 2024-03-23T23:17:24.400 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-23T23:17:24.402 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.a.log 2024-03-23T23:17:24.409 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.d.log: /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.a.log: 94.6% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.d.log.gz 2024-03-23T23:17:24.409 INFO:teuthology.orchestra.run.smithi203.stderr: 93.2% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.i.log.gz 2024-03-23T23:17:24.413 INFO:teuthology.orchestra.run.smithi203.stderr: 94.7% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.f.log.gz 2024-03-23T23:17:24.415 INFO:teuthology.orchestra.run.smithi012.stderr: 90.8% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-23T23:17:24.417 INFO:teuthology.orchestra.run.smithi012.stderr: 94.5% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.a.log.gz 2024-03-23T23:17:24.421 INFO:teuthology.orchestra.run.smithi173.stderr: 94.8% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.e.log.gz 2024-03-23T23:17:24.433 INFO:teuthology.orchestra.run.smithi173.stderr: 92.8% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.h.log.gz 2024-03-23T23:17:24.640 INFO:teuthology.orchestra.run.smithi203.stderr: 94.2% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.c.log.gz 2024-03-23T23:17:24.709 INFO:teuthology.orchestra.run.smithi012.stderr: 90.1% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mgr.x.log.gz 2024-03-23T23:17:24.747 INFO:teuthology.orchestra.run.smithi203.stderr: 91.7% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mon.c.log.gz 2024-03-23T23:17:24.791 INFO:teuthology.orchestra.run.smithi173.stderr: 91.7% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mon.b.log.gz 2024-03-23T23:17:25.019 INFO:teuthology.orchestra.run.smithi012.stderr: 92.2% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.9.log.gz 2024-03-23T23:17:25.269 INFO:teuthology.orchestra.run.smithi203.stderr: 93.4% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.11.log.gz 2024-03-23T23:17:25.297 INFO:teuthology.orchestra.run.smithi012.stderr: 92.4% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.3.log.gz 2024-03-23T23:17:25.416 INFO:teuthology.orchestra.run.smithi012.stderr: 90.3% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mon.a.log.gz 2024-03-23T23:17:25.499 INFO:teuthology.orchestra.run.smithi012.stderr: 93.1% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-23T23:17:25.552 INFO:teuthology.orchestra.run.smithi203.stderr: 93.0% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.8.log.gz 2024-03-23T23:17:25.630 INFO:teuthology.orchestra.run.smithi173.stderr: 92.2% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.4.log.gz 2024-03-23T23:17:25.757 INFO:teuthology.orchestra.run.smithi012.stderr: 92.9% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.6.log.gz 2024-03-23T23:17:25.760 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-03-23T23:17:25.760 INFO:teuthology.orchestra.run.smithi012.stderr:real 0m1.414s 2024-03-23T23:17:25.760 INFO:teuthology.orchestra.run.smithi012.stderr:user 0m5.024s 2024-03-23T23:17:25.760 INFO:teuthology.orchestra.run.smithi012.stderr:sys 0m0.485s 2024-03-23T23:17:25.820 INFO:teuthology.orchestra.run.smithi173.stderr: 92.8% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.7.log.gz 2024-03-23T23:17:25.917 INFO:teuthology.orchestra.run.smithi173.stderr: 91.6% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-23T23:17:26.134 INFO:teuthology.orchestra.run.smithi173.stderr: 92.9% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-mds.b.log.gz 2024-03-23T23:17:26.176 INFO:teuthology.orchestra.run.smithi203.stderr: 92.3% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-23T23:17:26.288 INFO:teuthology.orchestra.run.smithi173.stderr: 91.4% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.10.log.gz 2024-03-23T23:17:26.290 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-23T23:17:26.290 INFO:teuthology.orchestra.run.smithi173.stderr:real 0m1.959s 2024-03-23T23:17:26.290 INFO:teuthology.orchestra.run.smithi173.stderr:user 0m8.387s 2024-03-23T23:17:26.290 INFO:teuthology.orchestra.run.smithi173.stderr:sys 0m0.290s 2024-03-23T23:17:26.529 INFO:teuthology.orchestra.run.smithi203.stderr: 91.8% -- replaced with /var/log/ceph/90f679ae-e967-11ee-95cd-87774f69a715/ceph-osd.5.log.gz 2024-03-23T23:17:26.531 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-23T23:17:26.531 INFO:teuthology.orchestra.run.smithi203.stderr:real 0m2.200s 2024-03-23T23:17:26.531 INFO:teuthology.orchestra.run.smithi203.stderr:user 0m6.741s 2024-03-23T23:17:26.531 INFO:teuthology.orchestra.run.smithi203.stderr:sys 0m0.258s 2024-03-23T23:17:26.531 INFO:tasks.cephadm:Archiving logs... 2024-03-23T23:17:26.531 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616369/remote/smithi012/log 2024-03-23T23:17:26.532 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-23T23:17:27.338 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616369/remote/smithi173/log 2024-03-23T23:17:27.339 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-23T23:17:28.537 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616369/remote/smithi203/log 2024-03-23T23:17:28.538 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-23T23:17:29.618 INFO:tasks.cephadm:Removing cluster... 2024-03-23T23:17:29.618 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 90f679ae-e967-11ee-95cd-87774f69a715 --force 2024-03-23T23:17:29.963 INFO:teuthology.orchestra.run.smithi012.stdout:Deleting cluster with fsid: 90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T23:17:30.615 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 90f679ae-e967-11ee-95cd-87774f69a715 --force 2024-03-23T23:17:30.936 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: 90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T23:17:31.200 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 90f679ae-e967-11ee-95cd-87774f69a715 --force 2024-03-23T23:17:31.393 INFO:teuthology.orchestra.run.smithi203.stdout:Deleting cluster with fsid: 90f679ae-e967-11ee-95cd-87774f69a715 2024-03-23T23:17:31.656 INFO:tasks.cephadm:Removing cephadm ... 2024-03-23T23:17:31.656 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-23T23:17:31.716 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-23T23:17:31.733 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-23T23:17:31.748 INFO:tasks.cephadm:Teardown complete 2024-03-23T23:17:31.748 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-23T23:17:31.760 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-23T23:17:31.760 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-23T23:17:31.767 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-23T23:17:31.775 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-23T23:17:31.931 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-23T23:17:31.941 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-23T23:17:31.941 DEBUG:teuthology.orchestra.run.smithi012:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T23:17:31.949 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T23:17:31.952 DEBUG:teuthology.orchestra.run.smithi203:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T23:17:31.964 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-03-23T23:17:31.967 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-03-23T23:17:31.968 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T23:17:31.968 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-03-23T23:17:31.968 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 26 -621us[ -621us] +/- 139ms 2024-03-23T23:17:31.969 INFO:teuthology.orchestra.run.smithi173.stdout:^* hv02.front.sepia.ceph.com 3 6 377 30 -407us[ -378us] +/- 56ms 2024-03-23T23:17:31.969 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 +891us[ +891us] +/- 66ms 2024-03-23T23:17:31.969 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T23:17:31.971 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T23:17:31.971 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-03-23T23:17:31.971 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 26 -628us[ -590us] +/- 139ms 2024-03-23T23:17:31.971 INFO:teuthology.orchestra.run.smithi203.stdout:^* hv02.front.sepia.ceph.com 3 6 377 26 -434us[ -396us] +/- 56ms 2024-03-23T23:17:31.971 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 +857us[ +895us] +/- 66ms 2024-03-23T23:17:31.971 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T23:17:32.000 INFO:teuthology.orchestra.run.smithi012.stderr:bash: line 1: ntpq: command not found 2024-03-23T23:17:32.010 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T23:17:32.010 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2024-03-23T23:17:32.010 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 26 -629us[ -629us] +/- 139ms 2024-03-23T23:17:32.010 INFO:teuthology.orchestra.run.smithi012.stdout:^* hv02.front.sepia.ceph.com 3 6 377 26 -427us[ -391us] +/- 56ms 2024-03-23T23:17:32.010 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 +871us[ +907us] +/- 66ms 2024-03-23T23:17:32.010 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T23:17:32.012 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-23T23:17:32.022 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-23T23:17:32.023 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-23T23:17:32.033 DEBUG:teuthology.orchestra.run.smithi012:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-23T23:17:32.280 DEBUG:teuthology.orchestra.run.smithi173:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-23T23:17:32.358 DEBUG:teuthology.orchestra.run.smithi203:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-23T23:17:32.429 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-23T23:17:32.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T23:17:32.541 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-23T23:17:32.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T23:17:32.573 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-23T23:17:32.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T23:17:32.602 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-23T23:17:32.613 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-23T23:17:32.624 INFO:teuthology.task.internal:Duration was 2211.991498 seconds 2024-03-23T23:17:32.624 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-23T23:17:32.635 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-23T23:17:32.635 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-23T23:17:32.643 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-23T23:17:32.645 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-23T23:17:32.688 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T23:17:32.689 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T23:17:32.813 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T23:17:33.187 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-23T23:17:33.188 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi012.front.sepia.ceph.com 2024-03-23T23:17:33.188 DEBUG:teuthology.orchestra.run.smithi012:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-23T23:17:33.265 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2024-03-23T23:17:33.265 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-23T23:17:33.291 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi203.front.sepia.ceph.com 2024-03-23T23:17:33.292 DEBUG:teuthology.orchestra.run.smithi203:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-23T23:17:33.318 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-23T23:17:33.321 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-23T23:17:33.329 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-23T23:17:33.334 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-23T23:17:33.422 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-23T23:17:33.422 DEBUG:teuthology.orchestra.run.smithi012:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-23T23:17:33.428 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-23T23:17:33.430 DEBUG:teuthology.orchestra.run.smithi203:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-23T23:17:33.940 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-23T23:17:33.952 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-23T23:17:33.952 DEBUG:teuthology.orchestra.run.smithi012:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-23T23:17:34.050 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-23T23:17:34.078 DEBUG:teuthology.orchestra.run.smithi203:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-23T23:17:34.107 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-23T23:17:34.119 DEBUG:teuthology.orchestra.run.smithi012:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-23T23:17:34.127 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-23T23:17:34.129 DEBUG:teuthology.orchestra.run.smithi203:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-23T23:17:34.152 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2024-03-23T23:17:34.173 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = core 2024-03-23T23:17:34.236 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = core 2024-03-23T23:17:34.295 DEBUG:teuthology.orchestra.run.smithi012:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-23T23:17:34.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T23:17:34.346 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-23T23:17:34.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T23:17:34.361 DEBUG:teuthology.orchestra.run.smithi203:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-23T23:17:34.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T23:17:34.375 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-23T23:17:34.387 INFO:teuthology.task.internal:Transferring archived files... 2024-03-23T23:17:34.388 DEBUG:teuthology.misc:Transferring archived files from smithi012:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616369/remote/smithi012 2024-03-23T23:17:34.388 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-23T23:17:34.506 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616369/remote/smithi173 2024-03-23T23:17:34.506 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-23T23:17:34.546 DEBUG:teuthology.misc:Transferring archived files from smithi203:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616369/remote/smithi203 2024-03-23T23:17:34.546 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-23T23:17:34.585 INFO:teuthology.task.internal:Removing archive directory... 2024-03-23T23:17:34.585 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-23T23:17:34.593 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-23T23:17:34.596 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-23T23:17:34.646 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-23T23:17:34.660 INFO:teuthology.task.internal:Not uploading archives. 2024-03-23T23:17:34.660 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-23T23:17:34.672 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-23T23:17:34.672 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-23T23:17:34.679 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-23T23:17:34.682 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-23T23:17:34.694 INFO:teuthology.orchestra.run.smithi173.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 23 23:17 /home/ubuntu/cephtest 2024-03-23T23:17:34.697 INFO:teuthology.orchestra.run.smithi203.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 23 23:17 /home/ubuntu/cephtest 2024-03-23T23:17:34.733 INFO:teuthology.orchestra.run.smithi012.stdout: 393936 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 23 23:17 /home/ubuntu/cephtest 2024-03-23T23:17:34.736 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-23T23:17:34.747 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-23T23:17:34.775 INFO:teuthology.nuke:Checking targets against current locks 2024-03-23T23:17:34.800 DEBUG:teuthology.nuke:shortname: smithi012 2024-03-23T23:17:34.800 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-23T23:17:34.822 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616369', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 22:31:53.666787', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIODUQ7FvGYJLWYhbyXWV04x7zFmqpVbLTY6pZadtVVwqbWPx/tw67RhiroOV3Sm3+RL6c1FEedUmAdLsFfty9U='} 2024-03-23T23:17:34.826 DEBUG:teuthology.nuke:shortname: smithi173 2024-03-23T23:17:34.826 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-23T23:17:34.844 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616369', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 22:31:53.665044', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJGIytuhzMFSdGPBzLZNW38fnbqI7BKyIyhy4VYxsvoJy33yzGIYx+8Vmg7jQjy/DMHOx1w8+B3GPxZbNcmvK7o='} 2024-03-23T23:17:34.849 DEBUG:teuthology.nuke:shortname: smithi203 2024-03-23T23:17:34.849 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-23T23:17:34.869 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616369', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 22:31:53.663094', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL0DH2kuv4+0EuhMnWHVEO7RHu07yj9kLGKpbTXI1CJEPAewj0XPNn3gQUEfKbMwAsoyM1GMdcrnMd1plkCL0Rc='} 2024-03-23T23:17:34.877 INFO:teuthology.orchestra.console.smithi012:Power off 2024-03-23T23:17:34.877 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-23T23:17:34.893 INFO:teuthology.orchestra.console.smithi173:Power off 2024-03-23T23:17:34.893 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-23T23:17:34.907 DEBUG:teuthology.orchestra.console.smithi012:power off output: Chassis Power Control: Down/Off 2024-03-23T23:17:34.908 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T23:17:34.917 DEBUG:teuthology.orchestra.console.smithi173:power off output: Chassis Power Control: Down/Off 2024-03-23T23:17:34.917 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T23:17:34.929 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2024-03-23T23:17:34.940 INFO:teuthology.orchestra.console.smithi203:Power off 2024-03-23T23:17:34.940 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-23T23:17:34.950 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-03-23T23:17:34.967 DEBUG:teuthology.orchestra.console.smithi203:power off output: Chassis Power Control: Down/Off 2024-03-23T23:17:34.967 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T23:17:34.992 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-03-23T23:17:38.930 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T23:17:38.951 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T23:17:38.992 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T23:17:39.040 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2024-03-23T23:17:39.061 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-03-23T23:17:39.102 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-03-23T23:17:43.046 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T23:17:43.062 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T23:17:43.104 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T23:17:43.155 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2024-03-23T23:17:43.172 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is off 2024-03-23T23:17:43.213 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-03-23T23:17:43.274 INFO:teuthology.orchestra.console.smithi173:Power off completed 2024-03-23T23:17:47.157 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T23:17:47.214 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T23:17:47.267 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is off 2024-03-23T23:17:47.324 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-03-23T23:17:47.369 INFO:teuthology.orchestra.console.smithi012:Power off completed 2024-03-23T23:17:47.425 INFO:teuthology.orchestra.console.smithi203:Power off completed 2024-03-23T23:17:47.531 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} duration: 2211.9914979934692 failure_reason: '"2024-03-23T22:52:33.553104+0000 mon.a (mon.0) 188 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log' flavor: default owner: mchangir success: false 2024-03-23T23:17:47.531 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-23T23:17:47.631 INFO:teuthology.run:FAIL